Executing an OpenFOAM Motorbike Simulation

The Fuzzfile splits up the steps of the canonical OpenFOAM motorbike tutorial into individual Fuzzball jobs. The workflow calculates steady flow around a motorbike and its rider. It starts off by copying the example into a Fuzzball ephemeral volume. Then, it runs single process tasks surfaceFeatureExtract, blockMesh, and decomposePar. Next, meshing (using snappyHexMesh) and topoSet are run using 6 cores. Initial conditions for the simulation are set followed by another series of parallel tasks which include patchSummary, potentialFoam, checkMesh, and the simulation which executes solver simpleFoam. Finally, the mesh and partitions of the decomposed model are reconstructed using reconstructParMesh and reconstructPar respectively. The case directory with the simulation results is compressed into tarball motorbike-example-results.tar.gz which can be saved through data egress to a destination a path defined in an S3 URI.

All steps of the workflow use an OpenFOAM container on Dockerhub published by OpenCFD.

version: v1
volumes:
  openfoam-data-volume:
    name: openfoam-data-volume
    reference: volume://user/ephemeral
    # egress:
    #   - source:
    #       uri: file://motorbike-example-results.tar.gz
    #     destination:
    #       uri:
    #       secret:

jobs:

  prepare-motorbike-case:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
    command: ["/bin/bash", "-c", "cp -r $WM_PROJECT_DIR/tutorials/incompressible/simpleFoam/motorBike .; \
              cp motorBike/system/decomposeParDict.6 motorBike/system/decomposeParDict; \
              cat motorBike/system/decomposeParDict;"]
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GiB
    cwd: /data
    mounts:
      openfoam-data-volume:
        location: /data

  setup-blockmesh-decompose-par:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; \
               mkdir -p constant/triSurface && \
               cp -f $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/; \
               surfaceFeatureExtract; \
               blockMesh; \
               decomposePar;"]
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["prepare-motorbike-case"]

  snappy-hex-mesh:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; snappyHexMesh -overwrite -parallel;"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["setup-blockmesh-decompose-par"]

  toposet:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; topoSet -parallel;"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["snappy-hex-mesh"]


  set-initial-conditions:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; \
              source /usr/lib/openfoam/openfoam2212/bin/tools/RunFunctions; \
              restore0Dir -processor;"]
    resource:
      cpu:
        cores: 6
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["toposet"]

  patch-summary:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; patchSummary -parallel;"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["set-initial-conditions"]

  potential-foam:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; potentialFoam -writephi -parallel"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["patch-summary"]

  check-mesh:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; checkMesh -parallel -writeFields '(nonOrthoAngle)' -constant"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["potential-foam"]

  simple-foam:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
      - OMPI_ALLOW_RUN_AS_ROOT=1
      - OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
      - OMPI_MCA_pml=^ucx
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; simpleFoam -parallel"]
    multinode:
      nodes: 1 # Update nodes such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
      implementation: openmpi
    resource:
      cpu:
        cores: 6 # Update cores such that nodes * cores = numSubDomains if you are editing file decomposeParDict to run on more cores
        affinity: NUMA
      memory:
        size: 4GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["check-mesh"]

  reconstruct-mesh:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
    command: ["/bin/bash", "-c", "source $WM_PROJECT_DIR/etc/bashrc; \
               reconstructParMesh -constant; reconstructPar -latestTime;"]
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GiB
    cwd: /data/motorBike
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["simple-foam"]

  tar-results:
    image:
      uri: docker://opencfd/openfoam-default:2212
    env:
      - WM_PROJECT_DIR=/usr/lib/openfoam/openfoam2212
    command: ["/bin/bash", "-c", "tar -zcvf motorbike-example-results.tar.gz motorBike"]
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GiB
    cwd: /data
    mounts:
      openfoam-data-volume:
        location: /data
    requires: ["reconstruct-mesh"]

You can run this workflow either through the GUI or the CLI.

select

Please select either the GUI or CLI tab to see the appropriate instructions for your environment.

GUI

If you click “Workflow Editor” and “Create New”, you will see a blank page in the workflow editor.

Fuzzball create new workflow section

Now you can either click the ellipses (...) menu in the lower right and select “Edit YAML” or simply press e on your keyboard. An editor with a Fuzzfile stub will appear.

Fuzzball editor newly opened

You can delete the current contents and copy and paste the workflow definition of from above.

Fuzzball editor copied and pasted OpenFOAM workflow

Now pressing “save” will return you to the interactive workflow editor. You will now see the OpenFOAM workflow graph instead of a blank editor page. The Fuzzball GUI will automatically validate the yaml file for syntax errors.

Fuzzball workflow editor with OpenFOAM

Submitting your workflow to Fuzzball with the GUI is easy. Simply press the triangular “Start Workflow” button in the lower right corner of the workflow editor. You will be prompted to provide an optional descriptive name for your workflow.

Fuzzball name workflow and submit screen

Now you can click on “Start Workflow” in the lower right corner of the dialog box and your workflow will be submitted. If you click “Go to Status” you can view the workflow status page. The screenshot below shows the status page for a hello world workflow submission.

Fuzzball workflow status page

To retrieve logs produced by this workflow, select a job within the workflow such as prepare-motorbike-base, and click the “Logs” option on the right.

Fuzzball workflow dashboard showing logs from the prepare-motorbike-case

CLI

To run this workflow through the CLI you will need access to the Fuzzball CLI. You can install it using the Fuzzball CLI installation instructions.

First, you can create a Fuzzfile openfoam-motorbike-mpi.fz with the contents above using the text editor of your choice.

You can start start this workflow using the CLI by running the following command:

$ fuzzball workflow start openfoam-motorbike-mpi.fz
Workflow "9324e249-33d3-4424-8867-27d67b6f98a3" started.

You can monitor the workflow’s status by running the following command:

$ fuzzball workflow describe <workflow uuid>
Name:      openfoam-motorbike-mpi.fz
Email:     bphan@ciq.co
UserId:    e554e134-bd2d-455b-896e-bc24d8d9f81e
Status:    STAGE_STATUS_FINISHED
Created:   2024-05-21 03:22:52PM
Started:   2024-05-21 03:22:52PM
Finished:  2024-05-21 03:37:35PM
Error:


Stages:
KIND     | STATUS   | NAME                                    | STARTED               | FINISHED
Workflow | Finished | 9324e249-33d3-4424-8867-27d67b6f98a3    | 2024-06-20 04:04:16PM | 2024-06-20 04:20:28PM
Volume   | Finished | openfoam-data-volume                    | 2024-06-20 04:04:17PM | 2024-06-20 04:04:40PM
Image    | Finished | docker://opencfd/openfoam-default:2212  | 2024-06-20 04:04:17PM | 2024-06-20 04:04:34PM
Job      | Finished | prepare-motorbike-case                  | 2024-06-20 04:04:55PM | 2024-06-20 04:05:01PM
Job      | Finished | setup-blockmesh-decompose-par           | 2024-06-20 04:05:15PM | 2024-06-20 04:05:26PM
Job      | Finished | snappy-hex-mesh                         | 2024-06-20 04:06:22PM | 2024-06-20 04:07:39PM
Job      | Finished | toposet                                 | 2024-06-20 04:08:39PM | 2024-06-20 04:08:46PM
Job      | Finished | set-initial-conditions                  | 2024-06-20 04:09:42PM | 2024-06-20 04:09:49PM
Job      | Finished | patch-summary                           | 2024-06-20 04:10:55PM | 2024-06-20 04:11:03PM
Job      | Finished | potential-foam                          | 2024-06-20 04:12:15PM | 2024-06-20 04:12:23PM
Job      | Finished | check-mesh                              | 2024-06-20 04:13:19PM | 2024-06-20 04:13:29PM
Job      | Finished | simple-foam                             | 2024-06-20 04:14:25PM | 2024-06-20 04:17:40PM
Job      | Finished | reconstruct-mesh                        | 2024-06-20 04:17:55PM | 2024-06-20 04:18:11PM
Job      | Finished | tar-results                             | 2024-06-20 04:18:26PM | 2024-06-20 04:19:31PM
File     | Finished | file://motorbike-example-results.tar.gz | 2024-06-20 04:19:47PM | 2024-06-20 04:20:12PM
         |          | ->...                                   |                       |

You can view outputs logged by the workflow using the fuzzball workflow log command and provide the workflow UUID and job name. For example, executing the following command, will output logs from job set-initial-conditions in the workflow:

$ fuzzball workflow logs <workflow-uuid> set-initial-conditions
Restore 0/ from 0.orig/  [processor directories]