Issue 030 – January 23, 2021

VMs all the way down

Happy Saturday,

It’s Robin from CFD Engine & this week I managed to grab a few minutes to start an experiment that’s been nagging at me for a while…

“How well does OpenFOAM run in Docker on AWS?”

Before you go. I get that you probably don’t use Docker & you probably don’t run on AWS, but don’t leave just yet…I’ll keep it short.

Some background

I’m lazy. So, wherever possible, I’ll use the Ubuntu binaries from OpenFOAM.com, instead of compiling the code myself. On AWS I run the binaries on an Ubuntu base image & locally, on my Mac, I run them in Docker.

My local jobs are typically small test cases for method development & case setup. All of my real simulations are run on AWS. Could my real simulations run in Docker too? It’s not exactly what it was built for but, might it work?

Running OpenFOAM in Docker has been great on my Mac. But it also opens up some interesting options for deploying and managing jobs in the cloud. Before I get tangled up with these new-to-me workflows, I needed to answer a question…

“How much slower would my simulations be in Docker on AWS?”

I’ve looked at this before & things looked good. But AWS instances are virtual machines & running Docker on them is like stacking virtual machine on virtual machine. That can’t be good, can it?

Luckily, AWS offer a few “bare metal” instances that aren’t virtualised, so we have a way to test a few different combinations & check out what happens when it’s virtual machines all the way down.

My little experiment

This was the plan…

  • Install OpenFOAM on a bare metal AWS instance, run a few test cases and collect the run times;
  • Install Docker on the same instance, build an OpenFOAM image, run the same test cases in Docker and collect the run times;
  • Repeat the above steps on a “normal” hardware virtualised (HVM) AWS instance;
  • Compare the run times ⏱

If you’re interested:

  • I chose c5.metal & c5.24xlarge instances – both sporting 48 cores & 192GB RAM – overkill for these tests, but there aren’t too many metal options available.
  • All machines/images ran Ubuntu 20.04 LTS & I installed the pre-compiled OpenFOAM.com v2012 Ubuntu binaries. I didn’t tweak or re-compile anything & you can find my simple Dockerfile here.
  • I inflated the motorBike tutorial to ~3.2 million cells by increasing the cell count in blockMeshDict – still too small really.

The headlines

I said I’d keep this short so I’m not going to labour over the results. Here are the headlines…

  • The bare metal & hardware virtual machines had comparable performance – for our purposes, these instance types are essentially equivalent.

  • On the hardware virtual machines, running the simulations in Docker did not affect their run times.

  • On the bare metal instances, running the simulations in Docker increased their run time by ~5%.

An oddity

I did have one odd finding though – I couldn’t use all of the available cores in Docker. Running on 48 cores failed, but 42 cores ran fine. This happened on both the bare metal & hardware virtual machines. Fine outside Docker, but not inside.

I need to check this out on other machines – is 48 a special number or can we just not use all of the available cores in Docker 🤔

More input needed

It was an interesting quick first look & OpenFOAM seems to run pretty well in Docker on AWS & it isn’t super-slow.

Perhaps there is an opportunity here? I reckon it deserves further investigation at the very least.

When I look again, I shan’t bother with the metal instances and I’ll use the larger test cases I made for my previous EC2 instance comparisons.

If it still looks promising, then I can go on to play with the deployment & management options I alluded to earlier – stay tuned.

Thanks for sticking with me on this one, I know Docker & AWS aren’t my most popular topics, but I appreciate the opportunity to document my doing & thinking 🙏

Drop me a note if you have any experience with the Docker+AWS combo – keen to hear where the traps are. Alternatively, let me know what you’d like to see in future issues – my inbox is always open.

Until next week, CFD safely,

Robin