ramble.language namespace

Submodules

ramble.language.application_language module

class ramble.language.application_language.ApplicationMeta(name, bases, attr_dict)

Bases: SharedMeta

ramble.language.application_language.environment_variable(name, value, description, workload=None, workloads=None, workload_group=None, **kwargs)

Define an environment variable to be used in experiments

Parameters:

• name (str) – Name of environment variable to define

• value (str) – Value to set env-var to

• description (str) – Description of the env-var

• workload (str) – Name of workload this env-var should be added to

• workloads (list(str)) – List of workload names this env-var should be added to

ramble.language.application_language.executable(name, template, **kwargs)

Adds an executable to this application

Defines a new executable that can be used to configure workloads and experiments with.

Executables may or may not use MPI.

Required Args:

name (str): Name of the executable template (list[str] or str): The template command this executable should generate from

Optional Args:

use_mpi or mpi (boolean): determines if this executable should be

wrapped with an mpirun like command or not.

variables (dict): Dictionary of variable definitions to use for this

executable only

redirect (str): Optional, sets the path for outputs to be written to.

defaults to {log_file}

output_capture (str): Optional, Declare which output (stdout, stderr,

both) to capture. Defaults to stdout

run_in_background (boolean): Optional, Declare if the command should

run in the background. Defaults to False

ramble.language.application_language.input_file(name, url, description, target_dir='{workload_input_dir}', sha256=None, extension=None, expand=True, **kwargs)

Adds an input file definition to this application

Defines a new input file. An input file must define it’s name, and a url where the input can be fetched from.

Parameters:

• url (str) – Path to the input file / archive

• description (str) – Description of this input file

• target_dir (str) – Optional, the directory where the archive will be expanded. Defaults to the ‘{workload_input_dir}’ + os.sep + ‘{input_name}’

• sha256 (str) – Optional, the expected sha256 checksum for the input file

• extension (str) – Optiona, the extension to use for the input, if it isn’t part of the file name.

• expand (boolean) – Optional. Whether the input should be expanded or not. Defaults to True

ramble.language.application_language.workload(name, executables=None, executable=None, input=None, inputs=None, tags=None, **kwargs)

Adds a workload to this application

Defines a new workload that can be used within the context of its application.

Parameters:

• executable (str) – The name of an executable to be used

• executables (str) – A list of executable names to be used

• input (str) – Optional, name of an input be used

• inputs (str) – Optional, A list of input names that will be used

One of executable, or executables is required as an input argument.

ramble.language.application_language.workload_group(name, workloads=None, mode=None, **kwargs)

Adds a workload group to this application

Defines a new workload group that can be used within the context of its application.

Parameters:

• name (str) – The name of the group

• workloads (list(str) | None) – A list of workloads to be grouped

ramble.language.application_language.workload_variable(name, default, description, values=None, workload=None, workloads=None, workload_group=None, expandable: bool = True, track_used: bool = True, **kwargs)

Define a new variable to be used in experiments

Defines a new variable that can be defined within the experiments.yaml config file, to control various aspects of an experiment.

These are specific to each workload.

Parameters:

• name (str) – Name of variable to define

• default – Default value of variable definition

• description (str) – Description of variable’s purpose

• values (list) – Optional list of suggested values for this variable

• workload (str) – Single workload this variable is used in

• workloads (list) – List of modes this variable is used in

• workload_group (str) – Name of workload group this variable is used in

• expandable (bool) – True if the variable should be expanded, False if not.

• track_used (bool) – True if the variable should be tracked as used, False if not. Can help with allowing lists without vecotizing experiments.

ramble.language.language_base module

This package contains the underlying implementation for the language directives, which are to allow functions to be invoked at class level

exception ramble.language.language_base.DirectiveError(message, long_message=None)

Bases: RambleError

This is raised when something is wrong with a language directive.

class ramble.language.language_base.DirectiveMeta(name, bases, attr_dict)

Bases: type

Flushes the directives that were temporarily stored in the staging area into the package.

classmethod directive(dicts=None)

Decorator for Ramble directives.

Ramble directives allow you to modify an object while it is being defined, e.g. to add version or dependency information. Directives are one of the key pieces of Ramble’s object “language”, which is embedded in python.

Here’s an example directive:

@directive(dicts='workloads')
workload('workload_name', ...):
    ...

This directive allows you write:

class Foo(ApplicationBase):
    workload(...)

The @directive decorator handles a couple things for you:

1. Adds the class scope (app) as an initial parameter when called, like a class method would. This allows you to modify a package from within a directive, while the package is still being defined.

2. It automatically adds a dictionary called “workloads” to the package so that you can refer to app.workloads.

The (dicts='workloads') part ensures that ALL applications in Ramble will have a workloads attribute after they’re constructed, and that if no directive actually modified it, it will just be an empty dict.

This is just a modular way to add storage attributes to the Application class, and it’s how Ramble gets information from the applications to the core.

ramble.language.language_helpers module

ramble.language.language_helpers.check_definition(single_type, multiple_type, single_arg_name, multiple_arg_name, directive_name)

Sanity check definitions before merging or require

Parameters:

• single_type – Single string for type name

• multiple_type – List of strings for type names, may contain wildcards

• multiple_pattern_match – List of strings to match against patterns in multiple_type

• single_arg_name – String name of the single_type argument in the directive

• multiple_arg_name – String name of the multiple_type argument in the directive

• directive_name – Name of the directive requiring a type

Returns:

List of all type names (Merged if both single_type and multiple_type definitions are valid)

ramble.language.language_helpers.expand_patterns(multiple_type: list, multiple_pattern_match: list)

Expand wildcard patterns within a list of names

This method takes an input list containing wildcard patterns and expands the wildcard with values matching a list of names. Returns a list containing matching names and any inputs with zero matches.

Parameters:

• multiple_types – List of strings for type names, may contain wildcards

• multiple_pattern_match – List of strings to match against patterns in multiple_type

Returns:

List of expanded patterns matching the names list plus patterns not found in the names list.

ramble.language.language_helpers.merge_definitions(single_type, multiple_type, multiple_pattern_match, single_arg_name, multiple_arg_name, directive_name)

Merge definitions of a type

This method will merge two optional definitions of single_type and multiple_type.

Parameters:

• single_type – Single string for type name

• multiple_type – List of strings for type names, may contain wildcards

• multiple_pattern_match – List of strings to match against patterns in multiple_type

• single_arg_name – String name of the single_type argument in the directive

• multiple_arg_name – String name of the multiple_type argument in the directive

• directive_name – Name of the directive requiring a type

Returns:

List of all type names (Merged if both single_type and multiple_type definitions are valid)

ramble.language.language_helpers.require_definition(single_type, multiple_type, multiple_pattern_match, single_arg_name, multiple_arg_name, directive_name)

Require at least one definition for a type in a directive

This method will validate that single_type / multiple_type are properly defined.

It will raise an error if at least one type is not defined, or if either are the incorrect type.

Parameters:

• single_type – Single string for type name

• multiple_type – List of strings for type names, may contain wildcards

• multiple_pattern_match – List of strings to match against patterns in multiple_type

• single_arg_name – String name of the single_type argument in the directive

• multiple_arg_name – String name of the multiple_type argument in the directive

• directive_name – Name of the directive requiring a type

Returns:

List of all type names (Merged if both single_type and multiple_type definitions are valid)

ramble.language.modifier_language module

class ramble.language.modifier_language.ModifierMeta(name, bases, attr_dict)

Bases: SharedMeta

ramble.language.modifier_language.default_mode(name, **kwargs)

Define a default mode for this modifier.

The default mode will be used if modifier mode is not specified in an experiment.

Parameters:

name (str) – Name of mode to be used as default

ramble.language.modifier_language.env_var_modification(name, modification=None, method='set', mode=None, modes=None, **kwargs)

Define an environment variable modifier

Environment variable modifications modify the values of environment variables within the application instance.

Parameters:

• name (str) – The name of the environment variable that will be modified

• modification (str) – The value of the modification

• method (str) – The method of the modification.

• mode (str) – Name of mode this env_var_modification should apply in

• modes (list(str)) – List of mode names this env_var_modification should apply in

Supported values for method are:

set: Defines the variable to equal the modification value unset: Removes any definition of the variable from the environment prepend: Prepends the modification to the beginning of the variable.

Always uses the separator ‘:’

append: Appends the modification value to the end of the value. Allows a

keyword argument of ‘separator’ to define the delimiter between values.

ramble.language.modifier_language.executable_modifier(name)

Register an executable modifier

Executable modifiers can modify various aspects of non-builtin application executable definitions.

These behave similarly to builtins, in that a python method defines the actual modifications

For example:

executable_modifier('write_exec_name')

def write_exec_name(self, executable_name, executable, app_inst=None):
  prepend_execs = []
  append_execs = [ExecutableCommand(
      template='echo "{executable_name}"',
      mpi=False,
      redirect='{log_file}',
      output_capture=OUTPUT_CAPTURE.DEFAULT
  )]

  return prepend_execs, append_execs

Would append the echo “{executable_name}” to every non-builtin executable in an experiment.

Executable modifiers are allowed to modify the input executable in place. Executable modifiers must return two lists of executables.

Parameters:

name (str) – Name of executable modifier to use. Should be the name of a class method.

Each executable modifier needs to return:

prepend_execs (list(CommandExecutable)): List of executables to inject

before the base executable

append_execs (list(CommandExecutable)): List of executables to inject

after the base executable

ramble.language.modifier_language.mode(name, description, **kwargs)

Define a new mode for this modifier.

Modes allow a modifier to bundle a set of modifications together.

NOTE: A mode ‘disabled’ is defined by default for all modifiers.

Parameters:

• name (str) – Name of the mode to define

• description (str) – Description of the new mode

ramble.language.modifier_language.modifier_variable(name: str, default, description: str, values: list | None = None, mode: str | None = None, modes: list | None = None, expandable: bool = True, track_used: bool = False, **kwargs)

Define a variable for this modifier

Parameters:

• name (str) – Name of variable to define

• default – Default value of variable definition

• description (str) – Description of variable’s purpose

• values (list) – Optional list of suggested values for this variable

• mode (str) – Single mode this variable is used in

• modes (list) – List of modes this variable is used in

• expandable (bool) – True if the variable should be expanded, False if not.

• track_used (bool) – True if the variable should be tracked as used, False if not. Can help with allowing lists without vecotizing experiments.

ramble.language.modifier_language.package_manager_requirement(command: str, validation_type: str, modes: list, regex=None, package_manager: str = '*', **kwargs)

Define a requirement when this modifier is used in an experiment with a package manager.

This allows modifiers to inject additional requirements on packages managers. These can be used to ensure package manager commands return specific values.

Parameters:

• command (str) – Package manager command to execute, when evaluating the requirement

• validation_type (str) – Type of validation to perform on output of command. Valid types are: ‘empty’, ‘not_empty’, ‘contains_regex’, ‘does_not_contain_regex’

• modes (list(str)) – List of usage modes this requirement should apply to

• regex (str) – Regular expression to use when validation_type is either ‘contains_regex’ or ‘does_no_contain_regex’

• package_manager (str) – Name of the package manager this requirement applies to

ramble.language.modifier_language.required_variable(var: str, results_level='variable', modes=None, description=None)

Mark a variable as being required by this modifier

Parameters:

• var (str) – Variable name to mark as required

• results_level (str) – ‘variable’ or ‘key’. If ‘key’, variable is promoted to a key within JSON or YAML formatted results.

• modes (list[str] | None) – modes that the required check should be applied. The default None means apply to all modes.

• description (str | None) – Description of the required variable.

ramble.language.modifier_language.variable_modification(name, modification, method='set', mode=None, modes=None, separator=' ', **kwargs)

Define a new variable modification for a mode in this modifier.

A variable modification will apply a change to a defined variable within an experiment.

Parameters:

• name (str) – The variable to modify

• modification (str) – The value to modify ‘name’ with

• method (str) – How the modification should be applied

• mode (str) – Single mode to group this modification into

• modes (str) – List of modes to group this modification into

• separator (str) – Optional separator to use when modifying with ‘append’ or ‘prepend’ methods.

Supported values are ‘append’, ‘prepend’, and ‘set’:

‘append’ will add the modification to the end of ‘name’ ‘prepend’ will add the modification to the beginning of ‘name’ ‘set’ (Default) will overwrite ‘name’ with the modification

ramble.language.package_manager_language module

class ramble.language.package_manager_language.PackageManagerMeta(name, bases, attr_dict)

Bases: SharedMeta

ramble.language.package_manager_language.package_manager_variable(name: str, default, description: str, values: list | None = None, expandable: bool = True, **kwargs)

Define a variable for this package manager

Parameters:

• name (str) – Name of variable to define

• default – Default value of variable definition

• description (str) – Description of variable’s purpose

• values (list) – Optional list of suggested values for this variable

• expandable (bool) – True if the variable should be expanded, False if not.

ramble.language.shared_language module

class ramble.language.shared_language.SharedMeta(name, bases, attr_dict)

Bases: DirectiveMeta

ramble.language.shared_language.archive_pattern(pattern)

Adds a file pattern to be archived in addition to figure of merit logs

Defines a new file pattern that will be archived during workspace archival. Archival will only happen for files that match the pattern when archival is being performed.

Parameters:

pattern (str) – Pattern that refers to files to archive

ramble.language.shared_language.define_compiler(name, pkg_spec, compiler_spec=None, compiler=None, package_manager='*')

Defines the compiler that will be used with this object

Adds a new compiler spec to this object. Software specs should reference a compiler that has been added.

Parameters:

• name (str) – Name of compiler package

• pkg_spec (str) – Package spec to install compiler

• compiler_spec (str) – Compiler spec (if different from pkg_spec)

• compiler (str) – Package name to use for compilation

• package_manager (str) – Glob supported pattern to match package managers this compiler applies to

ramble.language.shared_language.figure_of_merit(name, fom_regex, group_name, log_file='{log_file}', units='', contexts=None, fom_type: FomType = FomType.UNDEFINED)

Adds a figure of merit to track for this object

Defines a new figure of merit.

Parameters:

• name (str) – High level name of the figure of merit

• log_file (str) – File the figure of merit can be extracted from

• fom_regex (str) – A regular expression using named groups to extract the FOM

• group_name (str) – The name of the group that the FOM should be pulled from

• units (str) – The units associated with the FOM

• contexts (list(str) | None) – List of contexts (defined by figure_of_merit_context) this figure of merit should exist in.

• fom_type – The type of figure of merit

ramble.language.shared_language.figure_of_merit_context(name, regex, output_format)

Defines a context for figures of merit

Defines a new context to contain figures of merit.

Parameters:

• name (str) – High level name of the context. Can be referred to in the figure of merit

• regex (str) – Regular expression, using group names, to match a context.

• output_format (str) – String, using python keywords {group_name} to extract group names from context regular expression.

ramble.language.shared_language.maintainers(*names: str)

Add a new maintainer directive, to specify maintainers in a declarative way.

Parameters:

names (str(s)) – GitHub username(s) for the maintainer. Can provide multiple names as separate arguments.

ramble.language.shared_language.package_manager_config(name, config, package_manager='*', **kwargs)

Defines a config option to set within a package manager

Define a new config which will be passed to a package manager. The resulting experiment instance will pass the config to the package manager, which will control the logic of applying it.

Parameters:

• name (str) – Name of this configuration

• config (str) – Configuration option to set

• package_manager (str) – Name of the package manager this config should be used with

ramble.language.shared_language.register_builtin(name, required=True, injection_method='prepend', depends_on=None, dependents=None)

Register a builtin

Builtins are methods that return lists of strings. These methods represent a way to write python code to generate executables for building up workloads.

Manual injection of a builtins can be performed through modifying the execution order in the internals config section.

Modifier builtins are named: modifier_builtin::modifier_name::method_name.

Application modifiers are named: builtin::method_name.

Package manager builtins are named: package_manager_builtin::package_manager_name::method_name.

As an example, if the following builtin was defined:

register_builtin('example_builtin', required=True)
def example_builtin(self):
  ...

Its fully qualified name would be: * modifier_builtin::test-modifier::example_builtin when defined in a modifier named test-modifier * builtin::example_builtin when defined in an application

The ‘required’ attribute marks a builtin as required for all workloads. This will ensure the builtin is added to the workload if it is not explicitly added. If required builtins are not explicitly added to a workload, they are injected into the list of executables, based on the injection_method attribute.

The ‘injection_method’ attribute controls where the builtin will be injected into the executable list. Options are: - ‘prepend’ – This builtin will be injected at the beginning of the executable list - ‘append’ – This builtin will be injected at the end of the executable list

Parameters:

• name (str) – Name of builtin (should be the name of a class method) to register

• required (boolean) – Whether the builtin will be auto-injected or not

• injection_method (str) – The method of injecting the builtin. Can be ‘prepend’ or ‘append’

• depends_on (list(str) | None) – The names of builtins this builtin depends on (and must execute after).

• dependents (list(str) | None) – The names of builtins that should come after the current one.

ramble.language.shared_language.register_phase(name, pipeline=None, run_before=None, run_after=None)

Register a phase

Phases are portions of a pipeline that will execute when executing a full pipeline.

Registering a phase allows an object to know what the phases dependencies are, to ensure the execution order is correct.

If called multiple times, the dependencies are combined together. Only one instance of a phase will show up in the resulting dependency list for a phase.

Parameters:

• name (str) – The name of the phase. Phases are functions named ‘_<phase>’.

• pipeline (str) – The name of the pipeline this phase should be registered into.

• run_before (list(str) | None) – A list of phase names this phase should run before

• run_after (list(str) | None) – A list of phase names this phase should run after

ramble.language.shared_language.register_template(name: str, src_path: str, dest_path: str | None = None, define_var: bool = True, extra_vars: dict | None = None, extra_vars_func: str | None = None, output_perm=None)

Directive to define an object-specific template to be rendered into experiment run_dir.

For instance, register_template(name=”foo”, src_path=”foo.tpl”, dest_path=”foo.sh”) expects a “foo.tpl” template defined alongside the object source, and uses that to render a file under “{experiment_run_dir}/foo.sh”. The rendered path can also be referenced with the foo variable name.

Parameters:

• name – The name of the template. It is also used as the variable name that an experiment can use to reference the rendered path, if define_var is true.

• src_path – The location of the template. It can either point to an absolute or a relative path. It knows how to resolve workspace paths such as $workspace_shared. A relative path is relative to the containing directory of the object source.

• dest_path – If present, the location of the rendered output. It can either point to an absolute or a relative path. It knows how to resolve workspace paths such as $workspace_shared. A relative path is relative to the experiment_run_dir. If not given, it will use the same name as the template (optionally drop the .tpl extension) and placed under experiment_run_dir.

• define_var – Controls if a variable named name should be defined.

• extra_vars – If present, the variable dict is used as extra variables to render the template.

• extra_vars_func – If present, the name of the function to call to return a dict of extra variables used to render the template. This option is combined together with and takes precedence over extra_vars, if both are present.

• output_perm – The chmod mask for the rendered output file.

ramble.language.shared_language.required_package(name, package_manager='*')

Defines a new spack package that is required for this object to function properly.

Parameters:

• name (str) – Name of required package

• package_manager (str) – Glob package manager name to apply this required package to

ramble.language.shared_language.software_spec(name, pkg_spec, compiler_spec=None, compiler=None, package_manager='*')

Defines a new software spec needed for this object.

Adds a new software spec (for spack to use) that this object needs to execute properly.

Only adds specs to object that use spack.

Specs can be described as an mpi spec, which means they will depend on the MPI library within the resulting spack environment.

Parameters:

• name (str) – Name of package

• pkg_spec (str) – Package spec to install package

• compiler_spec (str) – Spec to use if this package will be used as a compiler for another package

• compiler (str) – Package name to use as compiler for compiling this package

• package_manager (str) – Glob supported pattern to match package managers this package applies to

ramble.language.shared_language.success_criteria(name, mode, match=None, file='{log_file}', fom_name=None, fom_context='null', formula=None, anti_match=None)

Defines a success criteria used by experiments of this object

Adds a new success criteria to this object definition.

These will be checked during the analyze step to see if a job exited properly.

Parameters:

• name (str) – The name of this success criteria

• mode (str) – The type of success criteria that will be validated Valid values are: ‘string’, ‘application_function’, and ‘fom_comparison’

• match (str) – For mode=’string’. Value to check indicate success (if found, it would mark success)

• file (str) – For mode=’string’. File success criteria should be located in

• fom_name (str) – For mode=’fom_comparison’. Name of fom for a criteria. Accepts globbing.

• fom_context (str) – For mode=’fom_comparison’. Context the fom is contained in. Accepts globbing.

• formula (str) – For mode=’fom_comparison’. Formula to use to evaluate success. ‘{value}’ keyword is set as the value of the FOM.

• anti_match (str) – For mode=’string’. Value to check indicate failure. This setting and match are mutually exclusive.

ramble.language.shared_language.tags(*values: str)

Add a new tag directive, to specify tags in a declarative way.

Parameters:

values (str(s)) – Values to mark as a tag. Can provide multiple values as separate arguments.

ramble.language.shared_language.target_shells(shell_support_pattern=None)

Directive to specify supported shells.

If not specified, i.e., not directly specified or inherited from the base, then it assumes all shells are supported.

Parameters:

shell_support_pattern (str) – The glob pattern that is used to match with the configured shell

ramble.language.workflow_manager_language module

class ramble.language.workflow_manager_language.WorkflowManagerMeta(name, bases, attr_dict)

Bases: SharedMeta

ramble.language.workflow_manager_language.workflow_manager_variable(name: str, default, description: str, values: list | None = None)

Define a variable for this wm :param name: Name of variable :param default: Default value if the variable is not defined :param description: Description of the variable :param values: Optional list of suggested values for this variable