2) Writing an HPL application definition

This tutorial will provide an introduction to writing a more complex application definition in Ramble. In this tutorial, you will create and test an application definition file to run the High-Performance Linpack (HPL) benchmark.

This tutorial builds on the concepts introduced in the Basic Application Tutorial and introduces new concepts such as software dependencies and input file templating.

It is a good idea to have a basic working understanding of how to create and use Ramble workspaces before starting this tutorial. You should at least be familiar with the content of the Hello World Tutorial.

Installation

To install Ramble, see the Getting Started guide.

NOTE: This tutorial requires a package manager to be installed and configured to build HPL. For the purposes of this tutorial, we will focus on Spack. You will need a working Spack installation and configuration on your system for this tutorial. For more information on Spack installation or usage, please refer to the official Spack documentation. Ramble will leverage your configured Spack instance during the workspace setup stage to build HPL.

Ramble Repositories

Before writing our application definition, we will create a repository to house the application definition.

$ ramble repo create tutorial-repo

This will create a new directory named tutorial-repo in your current directory. Inside, directories will exist for each of the object types.

Once your repository is created, you can register it with Ramble by issuing the following command:

$ ramble repo add tutorial-repo

HPL Application Definition

For the remainder of this tutorial, we will be writing and testing the contents of the HPL application definition.

Create Application Definition

To begin with, we will create an empty application definition file and a placeholder for our input file template. These will live in the same directory.

$ mkdir -p tutorial-repo/applications/hpl/templates
$ touch tutorial-repo/applications/hpl/application.py
$ touch tutorial-repo/applications/hpl/templates/HPL.dat.tpl

For the remainder of this tutorial, ramble edit hpl will open this file with the editor specified with your EDITOR environment variable.

Application Class

Following the description of the application class in the Basic Application Tutorial, the following can be used as the beginning of our HPL application definition.

from ramble.appkit import *

class Hpl(ExecutableApplication):
    name = 'hpl'

At this stage, our application should show up in the output of ramble list and ramble info hpl should show limited information about this application.

Software Specification

Unlike the hostname utility, HPL is not typically pre-installed. We need to tell Ramble how to build it. We can do this using the software_spec directive. This directive takes a Spack spec to define the software and its dependencies.

with when("package_manager_family=spack"):
    software_spec(name="hpl", pkg_spec="hpl@2.3")

This tells Ramble to use Spack to install HPL version 2.3. The with when(...) construct makes these directives conditional on the package manager being used.

NOTE: By default, Spack will be allowed to select any MPI implementation it wants for this installation. Application definitions can optionally define additional software_specs to further control the default software stack for applications.

Input File Template

HPL requires an HPL.dat input file to run. Ramble has directives for downloading input files from websites. However, given this is a basic text file, Ramble can generate this file from a template. We use the register_template directive to tell Ramble how to generate this file for us.

register_template(
    "hpl.dat.in",
    template_file="templates/HPL.dat.tpl",
    destination="HPL.dat"
)

Now, let’s populate the template file tutorial-repo/applications/hpl/templates/HPL.dat.tpl. We can refer to any workload variables we are going to add into the application definition using curly brace syntax. When Ramble renders this file into the experiment directory, these values will be expanded for us.

HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
          device out (6=stdout,7=stderr,file)
          # of problems sizes (N)
{Ns}         Ns
          # of NBs
{NBs}        NBs
          PMAP process mapping (0=Row-,1=Column-major)
          # of process grids (P x Q)
{Ps}         Ps
{Qs}         Qs
0         threshold
          # of panel fact
          PFACTs (0=left, 1=Crout, 2=Right)
          # of recursive stopping criterium
          NBMINs (>= 1)
          # of panels in recursion
          NDIVs
          # of recursive panel fact.
          RFACTs (0=left, 1=Crout, 2=Right)
          # of broadcast
          BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
          # of lookahead depth
          DEPTHs (>= 0)
          SWAP (0=bin-exch,1=long,2=mix)
         swapping threshold
          L1 in (0=transposed,1=no-transposed) form
          U  in (0=transposed,1=no-transposed) form
          Equilibration (0=no,1=yes)
          memory alignment in double (> 0)

Definition Experiment Constructs

Now we will define the executable and workload for HPL.

Application Executables

HPL is an MPI application. We will define an executable that runs xhpl, the HPL executable. We’ll set use_mpi=True to tell Ramble to prepend the workspace’s MPI command to this executable. This allows users to run it with an MPI launcher like mpirun or srun.

executable(
    "hpl-execute",
    "xhpl",
    use_mpi=True,
)

Application Workloads

Here, we’ll create a default workload that uses our executable. Given our input file is generated from a template, we do not need to list it in the workload specification.

workload(
    "default",
    executables=["hpl-execute"],
)

Workload Variables

Now that we have a workload defined, we can define the variables that we added into our template input file (e.g. {Ns}). We need to define them in our application definition using the workload_variable directive so users can set them.

workload_variable("Ns", default=1000, description="Problem size", workload="default")
workload_variable("NBs", default=256, description="Block size", workload="default")
workload_variable("Ps", default=1, description="Number of process rows", workload="default")
workload_variable("Qs", default=1, description="Number of process columns", workload="default")

Analysis of experiments

Ramble can analyze experiment output to extract performance metrics and determine success or failure.

Figure of merit

HPL itself provides a mechanism for running several problem sizes in an attempt to maximize the GFlops rating. To ensure we do not lose data when extracting GFlops rating from HPL, we can create a figure of merit context to relate the GFlops rating to the input parameters. We can then define the figure of merit to exist within this context. The context can look like the following:

figure_of_merit_context(
    "problem-name",
    regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
    output_format="N-NB-P-Q = {N}-{NB}-{P}-{Q}",
)

This creates a new figure of merit context named problem-name which figures of merit can be associated with. The context will have the N, NB, P, and Q values associated with all figures of merit within. Since HPL reports its performance in Gflops, we extract this value using the figure_of_merit directive with a regular expression.

figure_of_merit(
    "gflops",
    fom_regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
    group_name="gflops",
    units="Gflops",
    contexts=["problem-name"],
    fom_type=FomType.THROUGHPUT
)

Here we have associated the figure of merit with the problem-name and also added the fom_type attribute to convey to users (and Ramble) that this is a throughput metric.

HPL additionally outputs time for the experiments. We can track this as a separate figure of merit:

figure_of_merit(
    "Time",
    fom_regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
    group_name="time",
    units="s",
    contexts=["problem-name"],
    fom_type=FomType.TIME,
)

Putting it all together

Our complete application definition at this point is as follows:

from ramble.appkit import *

class Hpl(ExecutableApplication):
    name = 'hpl'

    with when("package_manager_family=spack"):
        software_spec(name="hpl", pkg_spec="hpl@2.3")

    register_template(
        "hpl.dat.in",
        template_file="templates/HPL.dat.tpl",
        destination="HPL.dat"
    )

    executable(
        "hpl-execute",
        "xhpl",
        use_mpi=True,
    )

    workload(
        "default",
        executables=["hpl-execute"],
    )

    workload_variable("Ns", default=1000, description="Problem size", workload="default")
    workload_variable("NBs", default=256, description="Block size", workload="default")
    workload_variable("Ps", default=1, description="Number of process rows", workload="default")
    workload_variable("Qs", default=1, description="Number of process columns", workload="default")

    figure_of_merit_context(
        "problem-name",
        regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
        output_format="N-NB-P-Q = {N}-{NB}-{P}-{Q}",
    )

    figure_of_merit(
        "gflops",
        fom_regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
        group_name="gflops",
        units="Gflops",
        contexts=["problem-name"],
        fom_type=FomType.THROUGHPUT
    )

    figure_of_merit(
        "Time",
        fom_regex=r".*?\s+(?P<N>[0-9]+)\s+(?P<NB>[0-9]+)\s+(?P<P>[0-9]+)\s+(?P<Q>[0-9]+)\s+(?P<time>[0-9]+\.[0-9]+)\s+(?P<gflops>\S+)",
        group_name="time",
        units="s",
        contexts=["problem-name"],
        fom_type=FomType.TIME,
    )

Final Tests

To complete this tutorial we will test the application.

First, create a workspace:

$ ramble workspace create -d tutorial-workspace -a

Now, add an experiment. We’ll need to provide values for the number of nodes and ranks. For HPL, the number of ranks should equal PxQ.

$ ramble workspace manage experiments hpl --overwrite \
    -v n_nodes=1 \
    -v n_ranks=4 \
    -v Ps=2 \
    -v Qs=2 \
    -v Ns=2000 \
    -v NBs=256

Now, to complete the test we can execute:

$ ramble workspace setup
$ ramble on
$ ramble workspace analyze

The ramble workspace setup command will build HPL using Spack (if not already installed), which can take some time. Then, the ramble on command will run the experiment. After analysis, the result of these commands should be the creation of a results.latest.txt file that contains the Gflops of your machine, and the time spent running HPL.

Summary and Final Cleanup

At this stage, you have now created a new application definition to build and run the HPL benchmark.

To clean up your system, make sure to deactivate your workspace before trying to remove it.

$ ramble workspace deactivate
$ rm -rf tutorial-workspace

The application definition you have now matches some of the basic functionality from Ramble’s HPL implementation. You should now be able to compare your definition from the one seen by using ramble edit --type base_applications -N builtin hpl.