Utility Definition Tutorial

This tutorial will guide you through creating a simple utility definition in Ramble. We will create a mock utility called my_tool that provides an executable and modifies the environment.

Installation

To install Ramble, see the Getting Started guide.

NOTE: This tutorial does not require a package manager to be installed or configured.

Ramble Repositories

Before writing our definitions, we will create a repository to house the application definition. Repositories in Ramble can house any object type, and are not limited to only those in this tutorial.

The ramble repo command is used to manage object repositories in Ramble. To create a new repository, execute the following:

$ ramble repo create tutorial-repo

This will create a new directory named tutorial-repo in your current directory. Inside, directories will exist for each of the object types. These will include things like applications and package_managers. Resulting object paths in this repo would look like:

tutorial-repo/applications/hostname/application.py

You can also create a repository without object subdirectories using:

$ ramble repo create tutorial-repo -d ""

In this case, the resulting structure looks like:

tutorial-repo/hostname/application.py

This latter layout allows objects with the same name but different types to coexist in the same directory. For example, if there was a package manager named hostname, it would exist in the following path:

tutorial-repo/hostname/package_manager.py

The remaining commands in this tutorial will assume your repository layout matches the first example, but you can feel free to use either layout. Just map any paths to the correct layout.

The actual object files within the repository are named based on the object they represent. Below is a mapping of some object types to file names:

Application - application.py

Base Application - base_application.py

Modifier - modifier.py

Package Manager - package_manager.py

Workflow Manager - workflow_manager.py

System - system.py

Platform - platform.py

As listed with the application.py file, each object also has a corresponding base version, that is mostly use to help inheritance into several concrete objects.

Once your repository is created, you can register it with Ramble by issuing the following command:

$ ramble repo add tutorial-repo

NOTE: Ramble comes with a default builtin repository. Adding new repositories gives them a higher precedence to other existing repositories. Ramble uses this precedence ordering to decide which object definition is used when multiple exist with the same name. Each repository has a namespace, and these namespaces can be used to refer to specific instances of each object definition.

Step 1: Setting up the file structure

First, we need to create a directory for our utility within the tutorial-repo repository we created above.

Navigate to the utilities directory:

$ mkdir -p tutorial-repo/utilities/my-tool
$ cd tutorial-repo/utilities/my-tool

Create a file named utility.py using your editor of choice.

Step 2: Defining the Class

Open utility.py and start by importing the necessary base classes and language directives.

from ramble.utility import UtilityBase
from ramble.language.utility_language import *

class MyTool(UtilityBase):
    """MyTool is a mock utility for this tutorial."""

    name = "my-tool"
    maintainers("your_username")

Step 3: Defining Variables and Bootstrapping

We want our utility to be able to be downloaded and installed by Ramble if the user doesn’t already have it.

bootstrappable(True)

Next, we define a variable to hold the path to where the utility is installed. We use scoped=True so that the variable name is automatically namespaced to {utility::my_tool::path}.

variable(
    name="path",
    default="system",
    description="Path to MyTool",
    scoped=True,
)

Step 4: Providing Executables

Let’s tell Ramble that this utility provides the my_tool_bin executable, and how Ramble can check the system to see if a valid version is already installed.

provides_executable(
    "my_tool_bin",
    version_cmd="my_tool_bin --version",
    version_regex=r"MyTool Version (\d+\.\d+\.\d+)"
)

If a user requests this utility, Ramble will run my_tool_bin –version. If the output matches the regex, Ramble can check the version against any min_version or max_version constraints requested by other objects.

Step 5: Modifying the Environment

Finally, we need to ensure that the experiments can actually use my_tool_bin. We do this by prepending the utility’s bin directory to the PATH environment variable.

env_prepend("PATH", "{utility::my-tool::path}/bin")

Step 6: Applying Utilities in Other Objects

Now that we have defined a utility, other objects (like applications or modifiers) can apply it. To do this, the object definition uses the requires_utility directive.

For example, an application definition could apply our new utility like this:

from ramble.appkit import *

class MyApplication(ExecutableApplication):
    name = "my-application"

    requires_utility("my-tool", min_version="2.0.0", max_version="3.0.0")

    executable(
        "run",
        "my-tool run {arguments}",
        use_mpi=False,
    )

When an experiment uses this application, the variables and environment modifications defined in MyTool will be applied to the experiment, and Ramble will ensure the utility is either found on the system or bootstrapped.

Putting it all together

Your complete utility.py should look like this:

from ramble.utility import UtilityBase
from ramble.language.utility_language import *

class MyTool(UtilityBase):
    """MyTool is a mock utility for this tutorial."""

    name = "my-tool"
    maintainers("your_username")

    bootstrappable(True)

    variable(
        name="path",
        default="system",
        description="Path to MyTool",
        scoped=True,
    )

    provides_executable(
        "my_tool_bin",
        version_cmd="my_tool_bin --version",
        version_regex=r"MyTool Version (\d+\.\d+\.\d+)"
    )

    env_prepend("PATH", "{utility::my-tool::path}/bin")

You’ve successfully created your first utility definition! You can now reference my-tool in your workspace configuration under the utilities: section.

Cleanup

If you are done with this tutorial and want to clean up the repository you created, you can execute the following:

$ ramble repo rm tutorial-repo
$ rm -rf tutorial-repo