9) Success Criteria

In this tutorial, you will learn how to use success criteria. These will be created and applied to experiments for WRF, a free and open-source application for atmospheric research and operational forecasting applications.

This tutorial builds off of concepts introduced in previous tutorials. Please make sure you review those before starting with this tutorial’s content.

Create a Workspace

To begin with, you need a workspace to configure the experiments. This can be created with the following command:

$ ramble workspace create success_wrf

Activate the Workspace

Several of Ramble’s commands require an activated workspace to function properly. Activate the newly created workspace using the following command: (NOTE: you only need to run this if you do not currently have the workspace active).

$ ramble workspace activate success_wrf

Configure Experiment Definitions

To being with, you need to configure the workspace. The workspace’s root location can be seen under the Location output of:

$ ramble workspace info

Alternatively, the files can be edited directly with:

$ ramble workspace edit

Within the ramble.yaml file, write the following contents, which is the final configuration from a previous tutorial.

NOTE: This workspace utilizes the spack package manager. As a result, it requires spack is installed and available in your path. Modifications to the package_manager variant will change this behavior.

# Copyright 2022-2025 The Ramble Authors

# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

ramble:
  variants:
    package_manager: spack
  env_vars:
    set:
      OMP_NUM_THREADS: '{n_threads}'
  variables:
    processes_per_node: 16
    n_ranks: '{processes_per_node}*{n_nodes}'
    batch_submit: '{execute_experiment}'
    mpi_command: mpirun -n {n_ranks}
  applications:
    wrfv4:
      workloads:
        CONUS_12km:
          experiments:
            scaling_{n_nodes}:
              variables:
                n_nodes: [1, 2]
  software:
    packages:
      gcc9:
        pkg_spec: gcc@9.4.0
      intel-mpi:
        pkg_spec: intel-oneapi-mpi@2021.11.0
        compiler: gcc9
      wrfv4:
        pkg_spec: wrf@4.2 build_type=dm+sm compile_type=em_real nesting=basic ~chem
          ~pnetcdf
        compiler: gcc9
    environments:
      wrfv4:
        packages:
        - intel-mpi
        - wrfv4

The above configuration will execute 2 experiments, comprising a basic scaling study on 2 different sets of nodes. This is primarily defined by the use of vector experiments, which are documented in the vector logic portion of the workspace configuration file documentation. Vector experiments were also introduced in the vector and matrix tutorial.

Success Criteria

Ramble provides you with the ability to define success criteria within an application’s application.py file, or within the success_criteria configuration scope (which can also be defined in a workspace’s ramble.yaml configuration file). There are three supported types of success criteria, including:

  1. Regular Expression String Matching

  2. Figure of Merit Logic Based

  3. Arbitrary Python Based

This tutorial will focus on the first two of these, as the third can only be defined within the application.py file.

For more in-depth documentation about success criteria, see Success Criteria.

Regular Expression String Matching

The most common success criteria is the regular expression based string matching. These can be defined within either the application.py or the success_criteria configuration scope. Several application.py files already define this type of success criteria in order to make it easier for users to be sure their experiments completed successfully.

When WRF runs, it outputs the time each timestep takes. To begin with, you will define a new success criteria within your workspace configuration file that validates some timing data is present in the output of an experiment. In the experiment output, the timing data is prefixed with the string Timing for main.

This success criteria might look like the following:

success_criteria:
- name: 'timing-present'
  mode: 'string'
  match: 'Timing for main.*'
  file: '{experiment_run_dir}/rsl.out.0000'

Edit your workspace configuration file with:

$ ramble workspace edit

And add the example block within the wrfv4 application block. The resulting configuration file might look like the following:

# Copyright 2022-2025 The Ramble Authors

# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

ramble:
  variants:
    package_manager: spack
  variables:
    processes_per_node: 16
    n_ranks: '{processes_per_node}*{n_nodes}'
    batch_submit: '{execute_experiment}'
    mpi_command: mpirun -n {n_ranks}
  applications:
    wrfv4:
      success_criteria:
      - name: timing-present
        mode: string
        match: Timing for main.*
        file: '{experiment_run_dir}/rsl.out.0000'
      workloads:
        CONUS_12km:
          experiments:
            scaling_{n_nodes}:
              variables:
                n_nodes: [1, 2]
  software:
    packages:
      gcc9:
        pkg_spec: gcc@9.4.0
      intel-mpi:
        pkg_spec: intel-oneapi-mpi@2021.11.0
        compiler: gcc9
      wrfv4:
        pkg_spec: wrf@4.2 build_type=dm+sm compile_type=em_real nesting=basic ~chem
          ~pnetcdf
        compiler: gcc9
    environments:
      wrfv4:
        packages:
        - intel-mpi
        - wrfv4

Placing the success criteria definition here applies it to all of the experiments defined within the wrfv4 application.

Figure of Merit Logic Based

In addition to string matching, Ramble allows you to define a figure of merit based success criteria. These success criteria can be defined as equations that include placeholders for figures of merit the application will generate.

In this portion of the tutorial, you will define a new success criteria that ensures the experiments all have at least 50 timesteps to be considered successful. The figure of merit definition for this might look like the follow:

success_criteria:
- name: 'correct-timesteps'
  mode: 'fom_comparison'
  fom_name: 'Number of timesteps'
  formula: '{value} >= 50'

Edit your workspace configuration using:

$ ramble workspace edit

And add the success criteria within the CONUS_12km workload definition. The resulting configuration file might look like the following:

# Copyright 2022-2025 The Ramble Authors

# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

ramble:
  variants:
    package_manager: spack
  variables:
    processes_per_node: 16
    n_ranks: '{processes_per_node}*{n_nodes}'
    batch_submit: '{execute_experiment}'
    mpi_command: mpirun -n {n_ranks}
  applications:
    wrfv4:
      success_criteria:
      - name: timing-present
        mode: string
        match: Timing for main.*
        file: '{experiment_run_dir}/rsl.out.0000'
      workloads:
        CONUS_12km:
          success_criteria:
          - name: correct-timesteps
            mode: fom_comparison
            fom_name: Number of timesteps
            formula: '{value} >= 50'
          experiments:
            scaling_{n_nodes}:
              variables:
                n_nodes: [1, 2]
  software:
    packages:
      gcc9:
        pkg_spec: gcc@9.4.0
      intel-mpi:
        pkg_spec: intel-oneapi-mpi@2021.11.0
        compiler: gcc9
      wrfv4:
        pkg_spec: wrf@4.2 build_type=dm+sm compile_type=em_real nesting=basic ~chem
          ~pnetcdf
        compiler: gcc9
    environments:
      wrfv4:
        packages:
        - intel-mpi
        - wrfv4

This new success criteria will apply to all experiments within the CONUS_12km workload. This is because different workloads could have different numbers of timesteps.

It is important to note that the formula attribute of the success criteria definition can refer to other variables. As an example, one of the figures of merit output by the wrfv4 application definition is Average Timestep Time. If you had a single node value for this figure of merit and expected this to scale linerally you could define a success criteria as follows:

variables:
  single_node_value: '1.0'
success_criteria:
- name: 'valid-scaling'
  mode: 'fom_comparison'
  fom_name: 'Average Timestep Time'
  formula: '{value} <= {single_node_value} / {n_nodes}'

You won’t test this success criteria in this tutorial, as the single_node_value might not be correct for your system, but feel free to explore using this after you perform experiments.

Execute Experiments

Now that you have made the appropriate modifications, set up, execute, and analyze the new experiments using:

$ ramble workspace setup
$ ramble on
$ ramble workspace analyze

This creates a results file in the root of the workspace that contains extracted figures of merit. If the experiments were successful, this file will show the following results:

  • Average Timestep Time: Time (in seconds) on average each timestep takes

  • Cumulative Timestep Time: Time (in seconds) spent executing all timesteps

  • Minimum Timestep Time: Minimum time (in seconds) spent on any one timestep

  • Maximum Timestep Time: Maximum time (in seconds) spent on any one timestep

  • Number of timesteps: Count of total timesteps performed

  • Avg. Max Ratio Time: Ratio of Average Timestep Time and Maximum Timestep Time

To ensure the success criteria are checked and the experiments pass them, ensure SUCCESS is printed for the status of each experiment.

Running analyze in debug mode as:

$ ramble -d workspace analyze

Will print significnatly more output, but you should see where Ramble tests the timing-present and correct-timesteps success criteria in the output.

Clean the Workspace

Once you are finished with the tutorial content, make sure you deactivate your workspace:

$ ramble workspace deactivate

Additionally, you can remove the workspace and all of its content with:

$ ramble workspace remove success_wrf