Wrangling monitor files

Hey there 👋

It’s Robin from CFD Engine & I’ve got a little present for you 🎁

It’s a tool that makes working with OpenFOAM monitor files (like those produced by the forces & forceCoeffs function objects) a breeze.

I use it as a simple, flexible, scriptable way to extract the mean, standard deviation, variance & more from my force.dat files (& to do it superfast).

I used to use a couple of shell scripts for the same job, but this is waaay better. I reckon it’s worth a quick play, even if you already have a way to process your monitors.

Interested? Let’s go…

Quicksilver

Meet qsv (apparently it’s pronounced “quicksilver” 🤷‍♂️) – an open-source, command-line tool for wrangling comma-separated (CSV) data.

They have pre-built binaries for Mac, Windows & Linux & installing it is just a case of downloading the one for your system & unzipping it.

It has a whole bunch of features (many of which I’ll never use) which make CSV data almost as powerful as a database.

It’s a little light on examples, but you’ll see that it’s easy enough to use & simple to play with – I’m sure you’ll “get it”.

How I use it

qsv is intended to be used with Unix pipes, you run a command & pipe the output into the next command, repeating until you get what you need.

I’ve been using it to generate statistics from force monitors by adding a chain of commands like this to my run scripts…

Here’s what I’m doing with this chain:

1. I’m grabbing the last 250 rows of data;
2. I’m selecting only those columns with the pattern total_* in their name;
3. I’m calculating summary statistics for the data;
4. I’m keeping the mean, standard deviation, min, max, range & variance;
5. I’m printing them out as a table.

Here’s what the output looks like…

You could easily add or remove commands to tweak the output. You could report just a single value, you could do some more calculations, you could even save the output as an Excel file if you prefer.

Up to now I’ve been using shell scripts to do this stuff (with less stats), but I find this method easier to understand, easier to maintain, easier to extend across different file formats (& faster).

There’s just one problem…OpenFOAM output files aren’t comma-separated 🤫

Format wrangling

Let’s take the output of the forces function object as an example – it has a few header lines that we need to ignore, the column names have an extra # character at the start, the columns are grouped as vectors (surrounded by parentheses), plus it uses both space & tab delimiters 🤯

Great for readability, but not if we want to process it with other tools. We need to convert it to CSV.

It’s easily done, either with a few clicks in a text editor, or you could automate it with a couple of Unix tools.

I use sed to strip the header & the extra # character, I then use tr to remove the parenthesis & convert the whitespace to commas, but there are 100 other ways you could do it – shout if you want any pointers.

Once it’s in CSV format we can attack it with qsv (or a host of other tools).

Get your wrangle on

I’m guessing you already have a way to wrangle OpenFOAM monitor files. Maybe it’s a shell script (like me), a bit of Python, some Octave/Matlab, or even an Excel sheet.

Whatever you use, I’d recommend taking a look at qsv.

It’s fast, it’s actively developed, it’s permissively licensed, it’s multi-platform & it can do much more than my simple example showed.

In other words, it’s well worth a look 👀

I’m a fan, but let me know what you reckon, especially if you take it for a spin.

Alternatively, let me know how you handle OpenFOAM monitor files. I’m always keen to learn how you do stuff, drop me a note.

Until next week, stay safe,