ramble package

ramble.ramble_version = '0.6.0'

String containing Ramble version joined with .’s

ramble.ramble_version_info = (0, 6, 0)

major, minor, patch version for Ramble, in a tuple

Subpackages

Submodules

ramble.appkit module

ramble.application module

ramble.caches module

Caches used by Ramble to store data

class ramble.caches.MirrorCache(root)[source]

Bases: object

store(fetcher, relative_dest)[source]

Fetch and relocate the fetcher’s target into our mirror cache.

Symlink a human readable path in our mirror to the actual storage location.

ramble.caches.fetch_cache = <ramble.fetch_strategy.FsCache object>

Ramble’s local cache for downloaded source archives

ramble.caches.fetch_cache_location()[source]

Filesystem cache of downloaded archives.

This prevents Ramble from repeatedly fetch the same files when using them within multiple workspaces.

ramble.caches.misc_cache = <ramble.util.file_cache.FileCache object>

Ramble’s cache for small data

ramble.config module

This module implements Ramble’s configuration file handling.

This implements Ramble’s configuration system, which handles merging multiple scopes with different levels of precedence.

The scopes are:

  1. default

  2. system

  3. site

  4. user

Important functions in this module are:

get_config reads in YAML data for a particular scope and returns it. Callers can then modify the data and write it back with update_config.

When read in, Ramble validates configurations with jsonschemas. The schemas are in submodules of ramble.schema.

exception ramble.config.ConfigError(message, long_message=None)[source]

Bases: RambleError

Superclass for all ramble config related errors.

exception ramble.config.ConfigFileError(message, long_message=None)[source]

Bases: ConfigError

Issue reading or accessing a configuration file.

exception ramble.config.ConfigFormatError(validation_error, data, filename=None, line=None)[source]

Bases: ConfigError

Raised when a configuration format does not match its schema.

class ramble.config.ConfigScope(name, path)[source]

Bases: object

This class represents a configuration scope.

A scope is one directory containing named configuration files. Each file is a config “section” (e.g., mirrors, compilers, etc).

clear()[source]

Empty cached config information.

get_section(section)[source]
get_section_filename(section)[source]
property is_platform_dependent
exception ramble.config.ConfigSectionError(message, long_message=None)[source]

Bases: ConfigError

Error for referring to a bad config section name in a configuration.

class ramble.config.Configuration(*scopes)[source]

Bases: object

A full Ramble configuration, from a hierarchy of config files.

This class makes it easy to add a new scope on top of an existing one.

clear_caches()[source]

Clears the caches for configuration files,

This will cause files to be re-read upon the next request.

property file_scopes

List of writable scopes with an associated file.

get(path, default=None, scope=None)[source]

Get a config section or a single value from one.

Accepts a path syntax that allows us to grab nested config map entries. Getting the ‘config’ section would look like:

ramble.config.get('config')

and the dirty section in the config scope would be:

ramble.config.get('config:dirty')

We use : as the separator, like YAML objects.

get_config(section, scope=None)[source]

Get configuration settings for a section.

If scope is None or not provided, return the merged contents of all of ramble’s configuration scopes. If scope is provided, return only the configuration as specified in that scope.

This off the top-level name from the YAML section. That is, for a YAML config file that looks like this:

config:
  install_tree: $ramble/opt/ramble
  module_roots:
    lmod:   $ramble/share/ramble/lmod

get_config('config') will return:

{ 'install_tree': '$ramble/opt/ramble',
  'module_roots: {
      'lmod': '$ramble/share/ramble/lmod'
  }
}
get_config_filename(scope, section)[source]

For some scope and section, get the name of the configuration file.

highest_precedence_non_platform_scope()[source]

Non-internal non-platform scope with highest precedence

Platform-specific scopes are of the form scope/platform

highest_precedence_scope()[source]

Non-internal scope with highest precedence.

matching_scopes(reg_expr)[source]

List of all scopes whose names match the provided regular expression.

For example, matching_scopes(r’^command’) will return all scopes whose names begin with command.

pop_scope()[source]

Remove the highest precedence scope and return it.

print_section(section, blame=False)[source]

Print a configuration to stdout.

push_scope(scope)[source]

Add a higher precedence scope to the Configuration.

remove_scope(scope_name)[source]

Remove scope by name; has no effect when scope_name does not exist

set(path, value, scope=None)[source]

Convenience function for setting single values in config files.

Accepts the path syntax described in get().

update_config(section, update_data, scope=None, force=False)[source]

Update the configuration file for a particular scope.

Overwrites contents of a section in a scope with update_data, then writes out the config file.

update_data should have the top-level section name stripped off (it will be re-added). Data itself can be a list, dict, or any other yaml-ish structure.

Configuration scopes that are still written in an old schema format will fail to update unless force is True.

Parameters:

  • section (str) – section of the configuration to be updated

  • update_data (dict) – data to be used for the update

  • scope (str) – scope to be updated

  • force (str) – force the update

class ramble.config.ImmutableConfigScope(name, path)[source]

Bases: ConfigScope

A configuration scope that cannot be written to.

This is used for ConfigScopes passed on the command line.

class ramble.config.InternalConfigScope(name, data=None)[source]

Bases: ConfigScope

An internal configuration scope that is not persisted to a file.

This is for Ramble internal use so that command-line options and config file settings are accessed the same way, and Ramble can easily override settings from files.

clear()[source]

Empty cached config information.

get_section(section)[source]

Just reads from an internal dictionary.

get_section_filename(section)[source]
class ramble.config.SingleFileScope(name, path, schema, yaml_path=None)[source]

Bases: ConfigScope

This class represents a configuration scope in a single YAML file.

get_section(section)[source]
get_section_filename(section)[source]
property is_platform_dependent
ramble.config.add(fullpath, scope=None)[source]

Add the given configuration to the specified config scope. Add accepts a path. If you want to add from a filename, use add_from_file

ramble.config.add_from_file(filename, scope=None)[source]

Add updates to a config from a filename

ramble.config.command_line_scopes = []

configuration scopes added on the command line set by ramble.main.main().

ramble.config.config = <ramble.config.Configuration object>

This is the singleton configuration instance for ramble.

ramble.config.config_defaults = {'config': {'build_jobs': 2, 'build_stage': '$tempdir/ramble-stage', 'checksum': True, 'concretizer': 'clingo', 'connect_timeout': 10, 'debug': False, 'dirty': False, 'disable_passthrough': False, 'disable_progress_bar': False, 'input_cache': '$ramble/var/ramble/cache', 'license_dir': '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/etc/spack/licenses', 'n_repeats': '0', 'pip': {'install': {'flags': []}}, 'repeat_success_strict': True, 'report_dirs': '~/.ramble/reports', 'shell': 'bash', 'spack': {'flags': {'concretize': '--reuse', 'install': '--reuse'}}, 'upload': {'push_failed': True}, 'verify_ssl': True, 'workspace_dirs': '$ramble/var/ramble/workspaces'}}

Hard-coded default values for some key configuration options. This ensures that Ramble will still work even if config.yaml in the defaults scope is removed.

ramble.config.configuration_paths = (('defaults', '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/etc/ramble/defaults'), ('system', '/etc/ramble'), ('site', '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/etc/ramble'), ('user', '/home/docs/.ramble'))

Builtin paths to configuration files in ramble

ramble.config.default_list_scope()[source]

Return the config scope that is listed by default.

Commands that list configuration list all scopes (merged) by default.

ramble.config.default_modify_scope(section='config')[source]

Return the config scope that commands should modify by default.

Commands that modify configuration by default modify the highest priority scope.

Parameters:

section (bool) – Section for which to get the default scope. If this is not ‘experiments’, a general (non-workspace) scope is used.

ramble.config.ensure_latest_format_fn(section)[source]

Return a function that takes as input a dictionary read from a configuration file and update it to the latest format.

The function returns True if there was any update, False otherwise.

Parameters:

section (str) – section of the configuration e.g. “applications”, “config”, etc.

ramble.config.first_existing(dictionary, keys)[source]

Get the value of the first key in keys that is in the dictionary.

ramble.config.get(path, default=None, scope=None)[source]

Module-level wrapper for Configuration.get().

ramble.config.get_valid_type(path)[source]

Returns an instance of a type that will pass validation for path.

The instance is created by calling the constructor with no arguments. If multiple types will satisfy validation for data at the configuration path given, the priority order is list, dict, str, bool, int, float.

ramble.config.merge_yaml(dest, source)[source]

Merges source into dest; entries in source take precedence over dest.

This routine may modify dest and should be assigned to dest, in case dest was None to begin with, e.g.:

dest = merge_yaml(dest, source)

In the result, elements from lists from source will appear before elements of lists from dest. Likewise, when iterating over keys or items in merged OrderedDict objects, keys from source will appear before keys from dest.

Config file authors can optionally end any attribute in a dict with :: instead of :, and the key will override that of the parent instead of merging.

ramble.config.override(path_or_scope, value=None)[source]

Simple way to override config settings within a context.

Parameters:

  • path_or_scope (ConfigScope or str) – scope or single option to override

  • value (object or None) – value for the single option

Temporarily push a scope on the current configuration, then remove it after the context completes. If a single option is provided, create an internal config scope for it and push/pop that scope.

ramble.config.overrides_base_name = 'overrides-'

Base name for the (internal) overrides scope.

ramble.config.process_config_path(path)[source]
ramble.config.read_config_file(filename, schema=None)[source]

Read a YAML configuration file.

User can provide a schema for validation. If no schema is provided, we will infer the schema from the top-level key.

ramble.config.scopes()[source]

Convenience function to get list of configuration scopes.

ramble.config.scopes_metavar = '{defaults,system,site,user}'

metavar to use for commands that accept scopes this is shorter and more readable than listing all choices

ramble.config.section_schemas = {'applications': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'applications': {'additionalProperties': {'additionalProperties': False, 'default': '{}', 'properties': {'chained_experiments': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'command': {'type': 'string'}, 'inherit_variables': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}, 'order': {'type': 'string'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'type': 'array'}, 'env_vars': {'append': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'var-separator': {'type': 'string'}, 'vars': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'prepend': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'set': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'formatted_executables': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'commands': {'default': ['{unformatted_command}'], 'items': {'type': 'string'}, 'type': 'array'}, 'indentation': {'default': 0, 'type': 'number'}, 'join_separator': {'default': '\n', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'internals': {'additionalProperties': False, 'default': {}, 'properties': {'custom_executables': {'additionalProperties': {'default': {'output_capture': OUTPUT_CAPTURE.ALL, 'redirect': '{log_file}', 'template': [], 'use_mpi': False, 'variables': {}}, 'properties': {'redirect': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'template': {'anyOf': [{'type': 'string'}, {'type': 'number'}, {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}]}, 'use_mpi': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'executable_injection': {'default': [], 'items': {'additionalProperties': {'relative_to': {'type': 'string'}}, 'default': {}, 'properties': {'name': {'type': 'string'}, 'order': {'default': 'after', 'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'executables': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}, 'modifiers': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'mode': {'type': 'string'}, 'name': {'type': 'string'}, 'on_executable': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'type': 'array'}, 'success_criteria': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'anti_match': {'default': None, 'type': 'string'}, 'file': {'default': None, 'type': 'string'}, 'fom_context': {'default': None, 'type': 'string'}, 'fom_name': {'default': None, 'type': 'string'}, 'formula': {'default': None, 'type': 'string'}, 'match': {'default': None, 'type': 'string'}, 'mode': {'type': 'string'}, 'name': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'tags': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'template': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'variants': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'workloads': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'chained_experiments': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'command': {'type': 'string'}, 'inherit_variables': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}, 'order': {'type': 'string'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'type': 'array'}, 'env_vars': {'append': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'var-separator': {'type': 'string'}, 'vars': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'prepend': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'set': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'experiments': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'chained_experiments': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'command': {'type': 'string'}, 'inherit_variables': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}, 'order': {'type': 'string'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'type': 'array'}, 'env_vars': {'append': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'var-separator': {'type': 'string'}, 'vars': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'prepend': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'set': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'exclude': {'additionalProperties': False, 'default': {}, 'properties': {'matrices': {'default': [], 'items': {'anyOf': [{'default': [], 'items': {'type': 'string'}, 'type': 'array'}, {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}]}, 'type': 'array'}, 'matrix': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'where': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'zips': {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'type': 'object'}, 'formatted_executables': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'commands': {'default': ['{unformatted_command}'], 'items': {'type': 'string'}, 'type': 'array'}, 'indentation': {'default': 0, 'type': 'number'}, 'join_separator': {'default': '\n', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'internals': {'additionalProperties': False, 'default': {}, 'properties': {'custom_executables': {'additionalProperties': {'default': {'output_capture': OUTPUT_CAPTURE.ALL, 'redirect': '{log_file}', 'template': [], 'use_mpi': False, 'variables': {}}, 'properties': {'redirect': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'template': {'anyOf': [{'type': 'string'}, {'type': 'number'}, {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}]}, 'use_mpi': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'executable_injection': {'default': [], 'items': {'additionalProperties': {'relative_to': {'type': 'string'}}, 'default': {}, 'properties': {'name': {'type': 'string'}, 'order': {'default': 'after', 'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'executables': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}, 'matrices': {'default': [], 'items': {'anyOf': [{'default': [], 'items': {'type': 'string'}, 'type': 'array'}, {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}]}, 'type': 'array'}, 'matrix': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'modifiers': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'mode': {'type': 'string'}, 'name': {'type': 'string'}, 'on_executable': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'type': 'array'}, 'n_repeats': {'anyOf': [{'type': 'string'}, {'type': 'number'}], 'default': 0}, 'success_criteria': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'anti_match': {'default': None, 'type': 'string'}, 'file': {'default': None, 'type': 'string'}, 'fom_context': {'default': None, 'type': 'string'}, 'fom_name': {'default': None, 'type': 'string'}, 'formula': {'default': None, 'type': 'string'}, 'match': {'default': None, 'type': 'string'}, 'mode': {'type': 'string'}, 'name': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'tags': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'template': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'variants': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'zips': {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'formatted_executables': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'commands': {'default': ['{unformatted_command}'], 'items': {'type': 'string'}, 'type': 'array'}, 'indentation': {'default': 0, 'type': 'number'}, 'join_separator': {'default': '\n', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'internals': {'additionalProperties': False, 'default': {}, 'properties': {'custom_executables': {'additionalProperties': {'default': {'output_capture': OUTPUT_CAPTURE.ALL, 'redirect': '{log_file}', 'template': [], 'use_mpi': False, 'variables': {}}, 'properties': {'redirect': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'template': {'anyOf': [{'type': 'string'}, {'type': 'number'}, {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}]}, 'use_mpi': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'executable_injection': {'default': [], 'items': {'additionalProperties': {'relative_to': {'type': 'string'}}, 'default': {}, 'properties': {'name': {'type': 'string'}, 'order': {'default': 'after', 'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'executables': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}, 'modifiers': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'mode': {'type': 'string'}, 'name': {'type': 'string'}, 'on_executable': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'type': 'array'}, 'success_criteria': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'anti_match': {'default': None, 'type': 'string'}, 'file': {'default': None, 'type': 'string'}, 'fom_context': {'default': None, 'type': 'string'}, 'fom_name': {'default': None, 'type': 'string'}, 'formula': {'default': None, 'type': 'string'}, 'match': {'default': None, 'type': 'string'}, 'mode': {'type': 'string'}, 'name': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'tags': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'template': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'variants': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}, 'zips': {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'zips': {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'title': 'Ramble application configuration file schema', 'type': 'object'}, 'base_application_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'base_application_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble base application repository configuration file schema', 'type': 'object'}, 'base_modifier_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'base_modifier_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble base modifier repository configuration file schema', 'type': 'object'}, 'base_package_manager_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'base_package_manager_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble base package manager repository configuration file schema', 'type': 'object'}, 'base_workflow_manager_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'base_workflow_manager_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble base workflow manager repository configuration file schema', 'type': 'object'}, 'config': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'config': {'default': {}, 'deprecatedProperties': {'error': False, 'message': 'config:module_roots has been replaced by modules:[module set]:roots and is ignored', 'properties': ['module_roots']}, 'disable_logger': {'default': False, 'type': 'boolean'}, 'disable_passthrough': {'default': False, 'type': 'boolean'}, 'disable_progress_bar': {'default': False, 'type': 'boolean'}, 'input_cache': {'default': '$ramble/var/ramble/cache', 'type': 'string'}, 'n_repeats': {'default': '0', 'type': 'string'}, 'pip': {'additionalProperties': False, 'default': {'install': {'flags': []}}, 'properties': {'install': {'additionalProperties': False, 'properties': {'flags': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}, 'properties': {'additional_external_search_paths': {'items': {'type': 'string'}, 'type': 'array'}, 'allow_sgid': {'type': 'boolean'}, 'binary_index_root': {'type': 'string'}, 'build_jobs': {'minimum': 1, 'type': 'integer'}, 'build_language': {'type': 'string'}, 'build_stage': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'ccache': {'type': 'boolean'}, 'checksum': {'type': 'boolean'}, 'concretizer': {'enum': ['original', 'clingo'], 'type': 'string'}, 'connect_timeout': {'minimum': 0, 'type': 'integer'}, 'db_lock_timeout': {'minimum': 1, 'type': 'integer'}, 'debug': {'type': 'boolean'}, 'deprecated': {'type': 'boolean'}, 'dirty': {'type': 'boolean'}, 'extensions': {'items': {'type': 'string'}, 'type': 'array'}, 'install_hash_length': {'minimum': 1, 'type': 'integer'}, 'install_missing_compilers': {'type': 'boolean'}, 'install_path_scheme': {'type': 'string'}, 'install_tree': {'anyOf': [{'properties': {'padded_length': {'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]}, 'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'}, 'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'}, 'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'}, {'type': 'null'}]}, 'shared_linking': {'enum': ['rpath', 'runpath'], 'type': 'string'}, 'source_cache': {'type': 'string'}, 'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items': {'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'}, 'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'}, 'verify_ssl': {'type': 'boolean'}}, 'repeat_success_strict': {'default': True, 'type': 'boolean'}, 'report_dirs': {'default': '~/.ramble/reports', 'type': 'string'}, 'shell': {'enum': ['sh', 'bash', 'csh', 'tcsh', 'fish'], 'type': 'string'}, 'spack': {'additionalProperties': False, 'default': {'concretize': {'flags': '--reuse'}, 'install': {'flags': '--reuse'}}, 'flags': {}, 'properties': {'buildcache': {'additionalProperties': False, 'default': {'flags': '', 'prefix': ''}, 'properties': {'flags': {'default': '', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'compiler_find': {'default': {'flags': '', 'prefix': ''}, 'properties': {'flags': {'default': '', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'concretize': {'additionalProperties': False, 'default': {'flags': '--reuse', 'prefix': ''}, 'properties': {'flags': {'default': '--reuse', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'env_create': {'additionalProperties': False, 'default': {'flags': ''}, 'properties': {'flags': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'env_view': {'additionalProperties': False, 'default': {'link_type': 'symlink'}, 'properties': {'link_type': {'default': 'symlink', 'type': 'string'}}, 'type': 'object'}, 'global': {'additionalProperties': False, 'default': {'flags': ''}, 'properties': {'flags': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'install': {'additionalProperties': False, 'default': {'flags': '--reuse', 'prefix': ''}, 'properties': {'flags': {'default': '--reuse', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'object', 'upload': {'properties': {'push_failed': {'default': True, 'type': 'boolean'}, 'type': {'default': 'BigQuery', 'enum': ['BigQuery', 'PrintOnly'], 'type': 'string'}, 'uri': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'user': {'default': '', 'type': 'string'}, 'workspace_dirs': {'default': '$ramble/var/ramble/workspaces', 'type': 'string'}}}, 'title': 'Ramble core configuration file schema', 'type': 'object'}, 'env_vars': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'env_vars': {'append': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'var-separator': {'type': 'string'}, 'vars': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'prepend': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'set': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}}, 'title': 'Ramble environment variable configuration file schema', 'type': 'object'}, 'formatted_executables': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'formatted_executables': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'commands': {'default': ['{unformatted_command}'], 'items': {'type': 'string'}, 'type': 'array'}, 'indentation': {'default': 0, 'type': 'number'}, 'join_separator': {'default': '\n', 'type': 'string'}, 'prefix': {'default': '', 'type': 'string'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'title': 'Ramble formatted executables configuration file schema', 'type': 'object'}, 'internals': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'internals': {'additionalProperties': False, 'default': {}, 'properties': {'custom_executables': {'additionalProperties': {'default': {'output_capture': OUTPUT_CAPTURE.ALL, 'redirect': '{log_file}', 'template': [], 'use_mpi': False, 'variables': {}}, 'properties': {'redirect': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'template': {'anyOf': [{'type': 'string'}, {'type': 'number'}, {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}]}, 'use_mpi': {'type': 'boolean'}, 'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'executable_injection': {'default': [], 'items': {'additionalProperties': {'relative_to': {'type': 'string'}}, 'default': {}, 'properties': {'name': {'type': 'string'}, 'order': {'default': 'after', 'type': 'string'}}, 'type': 'object'}, 'type': 'array'}, 'executables': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}}, 'title': 'Ramble internals configuration file schema', 'type': 'object'}, 'licenses': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'licenses': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'append': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'var-separator': {'type': 'string'}, 'vars': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'prepend': {'items': {'additionalProperties': {}, 'properties': {'paths': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}}, 'type': 'object'}, 'type': 'array'}, 'set': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'title': 'Ramble licenses configuration file schema', 'type': 'object'}, 'mirrors': {'$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'properties': {'mirrors': {'additionalProperties': False, 'default': {}, 'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'properties': {'fetch': {'type': ['string', 'object']}, 'push': {'type': ['string', 'object']}}, 'required': ['fetch', 'push'], 'type': 'object'}]}}, 'type': 'object'}}, 'title': 'Ramble mirror configuration file schema', 'type': 'object'}, 'modifier_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'modifier_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble modifier repository configuration file schema', 'type': 'object'}, 'modifiers': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'modifiers': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'mode': {'type': 'string'}, 'name': {'type': 'string'}, 'on_executable': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'type': 'array'}}, 'title': 'Ramble application configuration file schema', 'type': 'object'}, 'package_manager_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'package_manager_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble package manager repository configuration file schema', 'type': 'object'}, 'repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble repository configuration file schema', 'type': 'object'}, 'software': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'software': {'additionalProperties': False, 'default': {}, 'properties': {'environments': {'additionalProperties': {'additionalProperties': {'type': 'string'}, 'default': {}, 'properties': {'external_env': {'default': None, 'type': 'string'}, 'packages': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'packages': {'additionalProperties': {'additionalProperties': True, 'default': {}, 'properties': {'compiler': {'default': None, 'type': 'string'}, 'compiler_spec': {'default': None, 'type': 'string'}, 'pkg_spec': {'type': 'string'}}, 'type': 'object'}, 'type': 'object'}}, 'type': 'object'}}, 'title': 'Software configuration file schema', 'type': 'object'}, 'spack': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'spack': {'additionalProperties': False, 'default': {}, 'properties': {'environments': {'additionalProperties': {'additionalProperties': False, 'default': {}, 'properties': {'external_env': {'default': None, 'type': 'string'}, 'external_spack_env': {'default': None, 'type': 'string'}, 'packages': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}, 'default': {}, 'properties': {}, 'type': 'object'}, 'packages': {'additionalProperties': {'additionalProperties': {'type': 'string'}, 'default': {}, 'properties': {'compiler': {'default': None, 'type': 'string'}, 'compiler_spec': {'default': None, 'type': 'string'}, 'pkg_spec': {'type': 'string'}}, 'type': 'object'}, 'type': 'object'}}, 'type': 'object'}}, 'title': 'Spack software configuration file schema', 'type': 'object'}, 'success_criteria': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'success_criteria': {'default': [], 'items': {'additionalProperties': False, 'default': {}, 'properties': {'anti_match': {'default': None, 'type': 'string'}, 'file': {'default': None, 'type': 'string'}, 'fom_context': {'default': None, 'type': 'string'}, 'fom_name': {'default': None, 'type': 'string'}, 'formula': {'default': None, 'type': 'string'}, 'match': {'default': None, 'type': 'string'}, 'mode': {'type': 'string'}, 'name': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}}, 'title': 'Ramble success criteria configuration file schema', 'type': 'object'}, 'variables': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'variables': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'title': 'Ramble variables configuration file schema', 'type': 'object'}, 'variants': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'variants': {'additionalProperties': True, 'default': {}, 'properties': {}, 'type': ['object', 'null']}}, 'title': 'Ramble variants configuration file schema', 'type': 'object'}, 'workflow_manager_repos': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'workflow_manager_repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Ramble workflow manager repository configuration file schema', 'type': 'object'}, 'zips': {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'zips': {'additionalProperties': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}, 'default': {}, 'properties': {}, 'type': 'object'}}, 'title': 'Ramble zips configuration file schema', 'type': 'object'}}

Dict from section names -> schema for that section

ramble.config.set(path, value, scope=None)[source]

Convenience function for setting single values in config files.

Accepts the path syntax described in get().

ramble.config.use_configuration(*scopes_or_paths)[source]

Use the configuration scopes passed as arguments within the context manager.

Parameters:

*scopes_or_paths – scope objects or paths to be used

Returns:

Configuration object associated with the scopes passed as arguments

ramble.config.validate(data, schema, filename=None)[source]

Validate data read in from a Ramble YAML file.

Parameters:

  • data (dict or list) – data read from a Ramble YAML file

  • schema (dict or list) – jsonschema to validate data

This leverages the line information (start_mark, end_mark) stored on Ramble YAML structures.

ramble.context module

class ramble.context.Context[source]

Bases: object

Class to represent a context

This class contains variable definitions to store any individual context (such as application, workload, or experiment) and logic to merge in additional contexts by order of precedence.

merge_context(in_context)[source]

Merges another Context into this Context.

ramble.context.create_context_from_dict(context_name, in_dict)[source]

Creates a new Context object from an input dictionary

Dictionaries should follow the below format:

in_dict = {

‘env_vars’: [], ‘variables’: {}, ‘variants’: {}, ‘internals’: {}, ‘template’: ‘’, ‘chained_experiments’: [], ‘modifiers’: [], ‘context_name’: ‘’, ‘exclude’: {}, ‘zips’: {}, ‘matrices’: {} or [], ‘tags’: [] ‘n_repeats’: ‘’,

Parameters:

  • context_name – The name of the context (e.g., application name)

  • in_dict – A dictionary representing the variable definitions

Returns:

Context(object)

ramble.error module

exception ramble.error.RambleCommandError[source]

Bases: Exception

Raised when RambleCommand execution fails.

exception ramble.error.RambleError(message, long_message=None)[source]

Bases: Exception

This is the superclass for all Ramble errors. Subclasses can be found in the modules they have to do with.

die()[source]
property long_message
print_context()[source]

Print extended debug information about this exception.

This is usually printed when the top-level Ramble error handler calls die(), but it can be called separately beforehand if a lower-level error handler needs to print error context and continue without raising the exception to the top level.

exception ramble.error.SpecError(message, long_message=None)[source]

Bases: RambleError

Superclass for all errors that occur while constructing specs.

ramble.error.debug = False

whether we should write stack traces or short error messages this is module-scoped because it needs to be set very early

ramble.expander module

exception ramble.expander.ApplicationNotDefinedError(message, long_message=None)[source]

Bases: ExpanderError

Raised when an application is not defined properly

class ramble.expander.Expander(variables, experiment_set, no_expand_vars=None)[source]

Bases: object

A class that will track and expand keyword arguments

This class will track variables and their definitions, to allow for expansion within string.

The variables can come from workspace variables, software stack variables, and experiment variables.

Additionally, math will be evaluated as part of expansion.

add_no_expand_var(var: str)[source]

Add a new variable to the no expand set

Parameters:

var (str) – Variable that should not expand

property application_input_dir
property application_name
property application_namespace
property application_run_dir
copy()[source]
property env_path
eval_math(node)[source]

Evaluate math from parsing the AST

Does not assume a specific type of operands. Some operators will generate floating point, while others will generate integers (if the inputs are integers).

evaluate_predicate(in_str, extra_vars=None, merge_used_stage: bool = True)[source]

Evaluate a predicate by expanding and evaluating math contained in a string

Parameters:

  • in_str – String representing predicate that should be evaluated

  • extra_vars – Variable definitions to use with highest precedence

Returns:

True or False, based on the evaluation of in_str

Return type:

boolean

expand_lists(var)[source]

Expand a variable into a list if possible

If expanding a variable would generate a list, this function will return a list. If any error case happens, this function will return the unmodified input value.

NOTE: This function is generally called early in the expansion. This allows lists to be generated before rendering experiments, but does not support pulling a list from a different experiment.

expand_var(var: str, extra_vars: Dict = None, allow_passthrough: bool = True, typed: bool = False, merge_used_stage: bool = True)[source]

Perform expansion of a string

Expand a string by building up a dict of all expansion variables.

Parameters:

  • var (str) – String variable to expand

  • extra_vars (dict) – Variable definitions to use with highest precedence

  • allow_passthrough (bool) – Whether the string is allowed to have keywords after expansion

  • typed (bool) – Whether the return type should be typed or not

  • merge_used_stage (bool) – Whether tracked variables are merged into the used variable set or not.

expand_var_name(var_name: str, extra_vars: Dict = None, allow_passthrough: bool = True, typed: bool = False, merge_used_stage: bool = True)[source]

Convert a variable name to an expansion string, and expand it

Take a variable name (var) and convert it to an expansion string by calling the expansion_str function. Pass the expansion string into expand_var, and return the result.

Parameters:

  • var_name (str) – String name of variable to expand

  • extra_vars (dict) – Variable definitions to use with highest precedence

  • allow_passthrough (bool) – Whether the string is allowed to have keywords after expansion

  • typed (bool) – Whether the return type should be typed or not

  • merge_used_stage (bool) – Whether tracked variables are merged into the used variable set or not.

static expansion_str(in_str)[source]
property experiment_name
property experiment_namespace
property experiment_run_dir
flush_used_variable_stage()[source]
property license_input_dir
merge_used_variable_stage()[source]
perform_math_eval(in_str)[source]

Attempt to evaluate in_str

Parameters:

in_str (str) – string representing math to attempt to evaluate

Returns:

(str) either the evaluation of in_str (if successful) or in_str unmodified (if unsuccessful)

set_no_expand_vars(no_expand_vars)[source]
property workload_input_dir
property workload_name
property workload_namespace
property workload_run_dir
exception ramble.expander.ExpanderError(message, long_message=None)[source]

Bases: RambleError

Raised when an error happens within an expander

class ramble.expander.ExpansionDelimiter[source]

Bases: object

Class representing the delimiters for ramble expansion strings

escape = '\\'
left = '{'
right = '}'
class ramble.expander.ExpansionDict[source]

Bases: dict

class ramble.expander.ExpansionGraph(in_str)[source]

Bases: object

Class representing a graph of ExpansionNodes

walk(in_node=None)[source]

Perform a DFS walk of the nodes in the graph

Parameters:

in_node (ExpansionNode) – node to begin the walk from, if not set uses self.root

Yields:

(ExpansionNode) – nodes following a DFS traversal of the graph

class ramble.expander.ExpansionNode(left_idx, right_idx)[source]

Bases: object

Class representing a node in a ramble expansion graph

add_children(children)[source]

Add children to this node

Parameters:

children (node, or list) – nodes to adds as children of self

define_value(expansion_dict, allow_passthrough=True, expansion_func=<class 'str'>, evaluation_func=<built-in function eval>, no_expand_vars=None, used_vars=None)[source]

Define the value for this node.

Construct the value of self. This builds up a string representation of self, and performs evaluation and formatting of the resulting string. This includes extracting the values of the children nodes, and replacing their values in the proper positions in self’s string.

Stores the resulting value in self.value

Parameters:

  • expansion_dict (dict) – variable definitions to use for expanding

  • matches (detected)

  • allow_passthrough (bool) – if true, expansion is allowed to fail. if

  • false

  • error. (failed expansion raises an)

  • expansion_func (func) – function to use for expansion of nested

  • definitions (variable)

  • evaluation_func (func) – function to use for evaluating math of strings

  • no_expand_vars (set) – set of variable names that should never be expanded

relative_indices(relative_to)[source]

Compute node indices relative to another node

Parameters:

relative_to (node) – node to shift current node’s indices relative to

Returns:

(tuple) indices of shifted match set

exception ramble.expander.ExperimentNotDefinedError(message, long_message=None)[source]

Bases: ExpanderError

Raised when an experiment is not defined properly

exception ramble.expander.MathEvaluationError(message, long_message=None)[source]

Bases: ExpanderError

Raised when an error happens while evaluating math during expansion

exception ramble.expander.RamblePassthroughError(message, long_message=None)[source]

Bases: ExpanderError

Raised when passthrough is disabled and variables fail to expand

exception ramble.expander.RambleSyntaxError(message, long_message=None)[source]

Bases: ExpanderError

Raised when a syntax error happens within variable definitions

class ramble.expander.VformatDelimiter[source]

Bases: object

Class representing the delimiters for the string.Formatter class

left = '{'
right = '}'
exception ramble.expander.WorkloadNotDefinedError(message, long_message=None)[source]

Bases: ExpanderError

Raised when a workload is not defined properly

ramble.expander.raise_passthrough_error(in_str, out_str)[source]

Raise an error when passthrough is disabled but variables are not all expanded

ramble.experiment_result module

class ramble.experiment_result.ExperimentResult(app_inst)[source]

Bases: object

Class containing results and related metadata of an experiment

to_dict()[source]

Generate a dict for encoders (json, yaml) and uploaders.

The generated dict preserves the existing serialized format so that previous result files work as expected.

ramble.experiment_set module

class ramble.experiment_set.ExperimentSet(workspace)[source]

Bases: object

Class to represent a full set of experiments

This class contains logic to take sets of variable definitions and generate experiments from the variable hierarchy.

Experiments are housed in the internal self.experiments dictionary. Keys of this dictionary are experiment names, while values are application instances.

add_chained_experiment(name, instance)[source]
all_experiment_tags()[source]

Aggregate all tags from experiments in this experiment set

Returns:

A set of all tags from the experiment set.

Return type:

(set)

all_experiments()[source]

Iterator over all experiments in this set

property application_namespace

Property to return application namespace (application name)

build_experiment_chains()[source]
property experiment_namespace

Property to return experiment namespace

Experiment namespaces are of the form: application_name.workload_name.experiment_name

filtered_experiments(filters)[source]

Return a filtered set of all experiments based on a logical expression

Exclusion takes overrides inclusion. If conflicting filters are provided which both include, and exclude the same experiment, the experiment will be excluded.

Parameters:

expression – A logical expression to evaluate, with each experiment

Yields:

exp – The name of the experiment, if expression results in True inst: An application instance representing the experiment

get_config_env_vars(workspace)[source]
get_config_vars(workspace)[source]
get_experiment(experiment)[source]
get_var_from_experiment(experiment, variable)[source]

Lookup a variable in a given experiment

Does not error if invalid values are passed in, to allow @ symbol to pass through to rendered content.

Parameters:

  • experiment – A fully qualified experiment name (application.workload.experiment)

  • variable – Name of variable to look up

keywords

Create experiment set class

num_experiments()[source]

Return the number of total experiments in this set

num_filtered_experiments(filters)[source]

Return the number of filtered experiments in this set

read_config_vars(workspace)[source]
search_primary_experiments(pattern)[source]

Search primary experiments using a glob syntax.

NOTE: This does not search experiments defined in an experiment chain

set_application_context(app_context)[source]

Set up current application context

set_base_var(var, val)[source]

Set a base variable definition

set_experiment_context(experiment_context)[source]

Set up current experiment context

set_required_var(var, val)[source]

Set a required variable definition

set_workload_context(workload_context)[source]

Set up current workload context

template_experiments()[source]

Iterator over template experiments in this set

property workload_namespace

Property to return workload namespace

Workload namespaces are of the form: application_name.workload_name

exception ramble.experiment_set.RambleExperimentSetError(message, long_message=None)[source]

Bases: RambleError

Super class for all experiment set errors

exception ramble.experiment_set.RambleVariableDefinitionError(message, long_message=None)[source]

Bases: RambleExperimentSetError

Error when a ramble variable definition is invalid

ramble.fetch_strategy module

Fetch strategies are used to download source code into a staging area in order to build it. They need to define the following methods:

  • fetch()

    This should attempt to download/check out source from somewhere.

  • check()

    Apply a checksum to the downloaded source code, e.g. for an archive. May not do anything if the fetch method was safe to begin with.

  • expand()

    Expand (e.g., an archive) downloaded file to source, with the standard stage source path as the destination directory.

  • reset()

    Restore original state of downloaded code. Used by clean commands. This may just remove the expanded source and re-expand an archive, or it may run something like git reset –hard.

  • archive()

    Archive a source directory, e.g. for creating a mirror.

class ramble.fetch_strategy.BundleFetchStrategy(**kwargs)[source]

Bases: FetchStrategy

Fetch strategy associated with bundle, or no-code, packages.

Having a basic fetch strategy is a requirement for executing post-install hooks. Consequently, this class provides the API but does little more than log messages.

TODO: Remove this class by refactoring resource handling and the link between composite stages and composite fetch strategies (see #11981).

property cachable

Report False as there is no code to cache.

fetch()[source]

Simply report success – there is no code to fetch.

mirror_id()[source]

BundlePackages don’t have a mirror id.

source_id()[source]

BundlePackages don’t have a source id.

url_attr = ''

There is no associated URL keyword in version() for no-code packages but this property is required for some strategy-related functions (e.g., check_pkg_attributes).

class ramble.fetch_strategy.CacheURLFetchStrategy(url=None, checksum=None, **kwargs)[source]

Bases: URLFetchStrategy

The resource associated with a cache URL may be out of date.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

exception ramble.fetch_strategy.ChecksumError(message, long_message=None)[source]

Bases: FetchError

Raised when archive fails to checksum.

class ramble.fetch_strategy.CvsFetchStrategy(**kwargs)[source]

Bases: VCSFetchStrategy

Fetch strategy that gets source code from a CVS repository.

Use like this in a package:

version(‘name’,

cvs=’:pserver:anonymous@www.example.com:/cvsroot%module=modulename’)

Optionally, you can provide a branch and/or a date for the URL:

version(‘name’,

cvs=’:pserver:anonymous@www.example.com:/cvsroot%module=modulename’, branch=’branchname’, date=’date’)

Repositories are checked out into the standard stage source path directory.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

property cvs
fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = ['branch', 'date']
reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

url_attr = 'cvs'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

exception ramble.fetch_strategy.ExtrapolationError(message, long_message=None)[source]

Bases: FetchError

Raised when we can’t extrapolate a version for a package.

exception ramble.fetch_strategy.FailedDownloadError(url, msg='')[source]

Bases: FetchError

Raised when a download fails.

exception ramble.fetch_strategy.FetchError(message, long_message=None)[source]

Bases: RambleError

Superclass for fetcher errors.

class ramble.fetch_strategy.FetchStrategy(**kwargs)[source]

Bases: object

Superclass of all fetch strategies.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

check()[source]

Checksum the archive fetched by this FetchStrategy.

expand()[source]

Expand the downloaded archive into the stage source path.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

classmethod matches(args)[source]

Predicate that matches fetch strategies to arguments of the version directive.

Parameters:

args – arguments of the version directive

mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = []
reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

url_attr = None

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.FetchStrategyComposite[source]

Bases: Composite

Composite for a FetchStrategy object.

classmethod matches(args)

Predicate that matches fetch strategies to arguments of the version directive.

Parameters:

args – arguments of the version directive

source_id()[source]
exception ramble.fetch_strategy.FetcherConflict(message, long_message=None)[source]

Bases: FetchError

Raised for packages with invalid fetch attributes.

class ramble.fetch_strategy.FsCache(root)[source]

Bases: object

destroy()[source]
fetcher(target_path, digest, **kwargs)[source]
store(fetcher, relative_dest)[source]
class ramble.fetch_strategy.GCSFetchStrategy(*args, **kwargs)[source]

Bases: URLFetchStrategy

FetchStrategy that pulls from a GCS bucket.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

url_attr = 'gs'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.GitFetchStrategy(**kwargs)[source]

Bases: VCSFetchStrategy

Fetch strategy that gets source code from a git repository. Use like this in a package:

version(‘name’, git=’https://github.com/project/repo.git’)

Optionally, you can provide a branch, or commit to check out, e.g.:

version(‘1.1’, git=’https://github.com/project/repo.git’, tag=’v1.1’)

You can use these three optional attributes in addition to git:

  • branch: Particular branch to build from (default is the

    repository’s default branch)

  • tag: Particular tag to check out

  • commit: Particular commit hash in the repo

Repositories are cloned into the standard stage source path directory.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

clone(dest=None, commit=None, branch=None, tag=None, bare=False)[source]

Clone a repository to a path.

This method handles cloning from git, but does not require a stage.

Parameters:

  • dest (str or None) – The path into which the code is cloned. If None, requires a stage and uses the stage’s source path.

  • commit (str or None) – A commit to fetch from the remote. Only one of commit, branch, and tag may be non-None.

  • branch (str or None) – A branch to fetch from the remote.

  • tag (str or None) – A tag to fetch from the remote.

  • bare (bool) – Execute a “bare” git clone (–bare option to git)

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

property git
property git_version
git_version_re = 'git version (\\S+)'
mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = ['tag', 'branch', 'commit', 'submodules', 'get_full_repo', 'submodules_delete']
protocol_supports_shallow_clone()[source]

Shallow clone operations (–depth #) are not supported by the basic HTTP protocol or by no-protocol file specifications. Use (e.g.) https:// or file:// instead.

reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

url_attr = 'git'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

static version_from_git(git_exe)[source]

Given a git executable, return the Version (this will fail if the output cannot be parsed into a valid Version).

class ramble.fetch_strategy.GoFetchStrategy(**kwargs)[source]

Bases: VCSFetchStrategy

Fetch strategy that employs the go get infrastructure.

Use like this in a package:

version(‘name’,

go=’github.com/monochromegane/the_platinum_searcher/…’)

Go get does not natively support versions, they can be faked with git.

The fetched source will be moved to the standard stage sourcepath directory during the expand step.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

expand()[source]

Expand the downloaded archive into the stage source path.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

property go
property go_version
reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

url_attr = 'go'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.HgFetchStrategy(**kwargs)[source]

Bases: VCSFetchStrategy

Fetch strategy that gets source code from a Mercurial repository. Use like this in a package:

version(‘name’, hg=’https://jay.grs.rwth-aachen.de/hg/lwm2’)

Optionally, you can provide a branch, or revision to check out, e.g.:

version(‘torus’,

hg=’https://jay.grs.rwth-aachen.de/hg/lwm2’, branch=’torus’)

You can use the optional ‘revision’ attribute to check out a branch, tag, or particular revision in hg. To prevent non-reproducible builds, using a moving target like a branch is discouraged.

  • revision: Particular revision, branch, or tag.

Repositories are cloned into the standard stage source path directory.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

property hg

Returns: Executable: the hg executable

mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = ['revision']
reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

url_attr = 'hg'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

exception ramble.fetch_strategy.InvalidArgsError(pkg=None, version=None, **args)[source]

Bases: FetchError

Raised when a version can’t be deduced from a set of arguments.

exception ramble.fetch_strategy.NoArchiveFileError(message, long_message=None)[source]

Bases: FetchError

“Raised when an archive file is expected but none exists.

exception ramble.fetch_strategy.NoCacheError(message, long_message=None)[source]

Bases: FetchError

Raised when there is no cached archive for a package.

exception ramble.fetch_strategy.NoDigestError(message, long_message=None)[source]

Bases: FetchError

Raised after attempt to checksum when URL has no digest.

exception ramble.fetch_strategy.NoStageError(method)[source]

Bases: FetchError

Raised when fetch operations are called before set_stage().

class ramble.fetch_strategy.S3FetchStrategy(*args, **kwargs)[source]

Bases: URLFetchStrategy

FetchStrategy that pulls from an S3 bucket.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

url_attr = 's3'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.SvnFetchStrategy(**kwargs)[source]

Bases: VCSFetchStrategy

Fetch strategy that gets source code from a subversion repository.

Use like this in a package:

version(‘name’, svn=’http://www.example.com/svn/trunk’)

Optionally, you can provide a revision for the URL:

version(‘name’, svn=’http://www.example.com/svn/trunk’,

revision=’1641’)

Repositories are checked out into the standard stage source path directory.

archive(destination)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = ['revision']
reset()[source]

Revert to freshly downloaded state.

For archive files, this may just re-expand the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

property svn
url_attr = 'svn'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.URLFetchStrategy(url=None, checksum=None, **kwargs)[source]

Bases: FetchStrategy

URLFetchStrategy pulls source code from a URL for an archive, check the archive against a checksum, and decompresses the archive.

The destination for the resulting file(s) is the standard stage path.

archive(destination)[source]

Just moves this archive to the destination.

property archive_file

Path to the source archive within this stage directory.

property cachable

Whether fetcher is capable of caching the resource it retrieves.

This generally is determined by whether the resource is identifiably associated with a specific package version.

Returns:

True if can cache, False otherwise.

Return type:

bool

property candidate_urls
check()[source]

Check the downloaded archive against a checksum digest. No-op if this stage checks code out of a repository.

property curl
expand()[source]

Expand the downloaded archive into the stage source path.

fetch()[source]

Fetch source code archive or repo.

Returns:

True on success, False on failure.

Return type:

bool

mirror_id()[source]

This is a unique ID for a source that is intended to help identify reuse of resources across packages.

It is unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.

optional_attrs = ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'checksum']
reset()[source]

Removes the source path if it exists, then re-expands the archive.

source_id()[source]

A unique ID for the source.

It is intended that a human could easily generate this themselves using the information available to them in the Spack package.

The returned value is added to the content which determines the full hash for a package using str().

url_attr = 'url'

The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.

class ramble.fetch_strategy.VCSFetchStrategy(**kwargs)[source]

Bases: FetchStrategy

Superclass for version control system fetch strategies.

Like all fetchers, VCS fetchers are identified by the attributes passed to the version directive. The optional_attrs for a VCS fetch strategy represent types of revisions, e.g. tags, branches, commits, etc.

The required attributes (git, svn, etc.) are used to specify the URL and to distinguish a VCS fetch strategy from a URL fetch strategy.

archive(destination, **kwargs)[source]

Create an archive of the downloaded data for a mirror.

For downloaded files, this should preserve the checksum of the original file. For repositories, it should just create an expandable tarball out of the downloaded repository.

check()[source]

Checksum the archive fetched by this FetchStrategy.

expand()[source]

Expand the downloaded archive into the stage source path.

ramble.fetch_strategy.all_strategies = [<class 'ramble.fetch_strategy.BundleFetchStrategy'>, <class 'ramble.fetch_strategy.URLFetchStrategy'>, <class 'ramble.fetch_strategy.CacheURLFetchStrategy'>, <class 'ramble.fetch_strategy.GoFetchStrategy'>, <class 'ramble.fetch_strategy.GitFetchStrategy'>, <class 'ramble.fetch_strategy.CvsFetchStrategy'>, <class 'ramble.fetch_strategy.SvnFetchStrategy'>, <class 'ramble.fetch_strategy.HgFetchStrategy'>, <class 'ramble.fetch_strategy.S3FetchStrategy'>, <class 'ramble.fetch_strategy.GCSFetchStrategy'>]

List of all fetch strategies, created by FetchStrategy metaclass.

ramble.fetch_strategy.check_pkg_attributes(pkg)[source]

Find ambiguous top-level fetch attributes in a package.

Currently this only ensures that two or more VCS fetch strategies are not specified at once.

ramble.fetch_strategy.fetcher(cls)[source]

Decorator used to register fetch strategies.

ramble.fetch_strategy.for_package_version(pkg, version)[source]

Determine a fetch strategy based on the arguments supplied to version() in the package description.

ramble.fetch_strategy.from_kwargs(**kwargs)[source]

Construct an appropriate FetchStrategy from the given keyword arguments.

Parameters:

**kwargs – dictionary of keyword arguments, e.g. from a version() directive in a package.

Returns:

The fetch strategy that matches the args, based

on attribute names (e.g., git, hg, etc.)

Return type:

FetchStrategy

Raises:

FetchError – If no fetch_strategy matches the args.

ramble.fetch_strategy.from_list_url(pkg)[source]

If a package provides a URL which lists URLs for resources by version, this can can create a fetcher for a URL discovered for the specified package’s version.

ramble.fetch_strategy.from_url(url)[source]

Given a URL, find an appropriate fetch strategy for it. Currently just gives you a URLFetchStrategy that uses curl.

TODO: make this return appropriate fetch strategies for other

types of URLs.

ramble.fetch_strategy.from_url_scheme(url, *args, **kwargs)[source]

Finds a suitable FetchStrategy by matching its url_attr with the scheme in the given url.

ramble.fetch_strategy.stable_target(fetcher)[source]

Returns whether the fetcher target is expected to have a stable checksum. This is only true if the target is a preexisting archive file.

ramble.fetch_strategy.warn_content_type_mismatch(subject, content_type='HTML')[source]

ramble.filters module

class ramble.filters.Filters(phase_filters=None, include_where_filters=None, exclude_where_filters=None, tags=None)[source]

Bases: object

Object containing filters for limiting various operations in Ramble

ramble.graphs module

class ramble.graphs.AttributeGraph(obj_inst)[source]

Bases: object

add_node(node)[source]

Add a node to the graph

Parameters:

node (GraphNode) – Node to add into graph

define_edges(node, dep_nodes=None, internal_order=False)[source]

Define graph edges

Process dependencies, and internal orderings (inside the node object) to define new graph edges.

Parameters:

  • node (GraphNode) – Node to inject or modify

  • dep_nodes (list(GraphNode) | None) – List of nodes that are dependencies

  • internal_order (Boolean) – True to process internal dependencies, False to skip

get_node(key)[source]

Given a key, return the node containing this key

Parameters:

key (str) – Name of key to find in the graph

Returns:

Node representing the key requested. Returns None if

the key isn’t found.

Return type:

(GraphNode)

node_type = 'object'
update_graph(node, dep_nodes=None, internal_order=False)[source]

Update the graph with a new node and / or new dependencies.

Given a node, and list of dependencies, define new edges in the graph. If the node is new, also construct a new phase node.

Parameters:

  • node (GraphNode) – Node to inject or modify

  • dep_nodes (list(GraphNode) | None) – List of nodes that are dependencies

  • internal_order (Boolean) – True to process internal dependencies, False to skip

walk()[source]

Walk the graph in topological ordering and yield each node.

Construct a topological ordering of the current graph, walk it, and yield each node one by one.

Yields:

node (GraphNode) – Each node in the graph

class ramble.graphs.ExecutableGraph(exec_order, executables, builtin_objects, builtin_groups, obj_inst)[source]

Bases: AttributeGraph

Graph that handles command executables and builtins

inject_executable(exec_name, injection_order, relative)[source]

Inject an executable into the graph

Parameters:

  • exec_name (str) – Name of executable to inject

  • injection_order (str) – Order for injection. Can be ‘before’ or ‘after’

  • relative (str) – Name of executable to inject relative to. Can be None to inject relative to the whole set of executables.

node_type = 'command executable'
class supported_injection_orders(*values)

Bases: Enum

after = 2
before = 1
exception ramble.graphs.GraphCycleError(message, long_message=None)[source]

Bases: GraphError

Exception raised when a cycle is detected in a graph

exception ramble.graphs.GraphError(message, long_message=None)[source]

Bases: RambleError

Exception raised with errors in a graph type

exception ramble.graphs.GraphNodeAmbiguousError(message, long_message=None)[source]

Bases: GraphError

Exception raised when the given name can be resolved to non-unique nodes

exception ramble.graphs.GraphNodeNotFoundError(message, long_message=None)[source]

Bases: GraphError

Exception raised when the given name cannot be resolved to a node

class ramble.graphs.PhaseGraph(phase_definitions, obj_inst)[source]

Bases: AttributeGraph

add_node(node, obj_inst=None)[source]

Add a new phase node to the graph

Extract the phase function from the object instance, and inject a new node into the graph.

Parameters:

  • node (GraphNode) – Phase node to add into graph

  • obj_inst (Object) – Object that owns the phase

node_type = 'phase'
update_graph(phase_name, dependencies=None, internal_order=False, obj_inst=None)[source]

Update the graph with a new phase and / or new dependencies.

Given a phase name, and list of dependencies, define new edges in the graph. If the phase is new, also construct a new phase node.

Parameters:

  • phase_name (str) – Name of the phase to inject or modify

  • dependencies (list(str) | None) – List of phase names to inject dependencies on

  • internal_order (Boolean) – True to process internal dependencies, False to skip

  • obj_inst (object) – Application or modifier instance to extract phase function from

ramble.keywords module

class ramble.keywords.Keywords(extra_keys=None)[source]

Bases: object

Class to represent known ramble keywords.

Each keyword contains a dictionary of its attributes. Currently, these include: - type - level

Valid types are identified by the ‘key_type’ variable as an enum. Valid levels are identified by the ‘output_level’.

Current key types are:

  • Reserved: Ramble defines these, and a user should not be allowed to define them

  • Optional: Ramble can function with a definition from the user but it isn’t required

  • Required: Ramble requires a definition for these. Ramble will try to set sensible defaults, but it might not be possible always.

Current levels are:
  • Key: Ramble defines this as a top level variable. When results are

    output, these are hoisted to a set of variables that are guaranteed to be in the output. These are non-application specific inputs that define a Ramble experiment.

  • Variable: These are considered standard variables. They might be

    derived from the values of entries with the level key. In results, they are presented in the variables section. These may include application specific inputs to further configure the experiment.

all_required_keys()[source]

Yield all required keys

Yields:

(str) – Key name

all_reserved_keys()[source]

Yield all reserved keys

Yields:

(str) – Key name

check_required_keys(definitions)[source]

Check a dictionary of variable definitions for all required keywords

check_reserved_keys(definitions)[source]

Check a dictionary of variable definitions for reserved keywords

copy()[source]
is_key_level(key)[source]

Check if key is part of the key level

is_optional(key)[source]

Check if a key is optional

is_required(key)[source]

Check if a key is required

is_reserved(key)[source]

Check if a key is reserved

is_valid(key)[source]

Check if a key is valid as a known keyword

is_variable_level(key)[source]

Check if key is part of the variable level

update_keys(extra_keys)[source]
exception ramble.keywords.RambleKeywordError(message, long_message=None)[source]

Bases: RambleError

Superclass for all errors to do with Ramble Keywords

ramble.keywords.key_type

alias of type

ramble.keywords.output_level

alias of level

ramble.main module

ramble.mirror module

This file contains code for creating ramble mirror directories. A mirror is an organized hierarchy containing specially named archive files. This enabled ramble to know where to find files in a mirror if the main server for a particular input is down. Or, if the computer where ramble is run is not connected to the internet, it allows ramble to download inputs directly from a mirror (e.g., on an intranet).

class ramble.mirror.Mirror(fetch_url, push_url=None, name=None)[source]

Bases: object

Represents a named location for storing input tarballs.

Mirrors have a fetch_url that indicate where and how artifacts are fetched from them, and a push_url that indicate where and how artifacts are pushed to them. These two URLs are usually the same.

display(max_len=0)[source]
property fetch_url
static from_dict(d, name=None)[source]
static from_json(stream, name=None)[source]
static from_yaml(stream, name=None)[source]
get_access_pair(url_type)[source]
get_access_token(url_type)[source]
get_endpoint_url(url_type)[source]
get_profile(url_type)[source]
property name
property push_url
set_access_pair(url_type, connection_tuple)[source]
set_access_token(url_type, connection_token)[source]
set_endpoint_url(url_type, url)[source]
set_profile(url_type, profile)[source]
to_dict()[source]
to_json(stream=None)[source]
to_yaml(stream=None)[source]
class ramble.mirror.MirrorCollection(mirrors=None, scope=None)[source]

Bases: Mapping

A mapping of mirror names to mirrors.

display()[source]
static from_dict(d)[source]
lookup(name_or_url)[source]

Looks up and returns a Mirror.

If this MirrorCollection contains a named Mirror under the name [name_or_url], then that mirror is returned. Otherwise, [name_or_url] is assumed to be a mirror URL, and an anonymous mirror with the given URL is returned.

to_dict(recursive=False)[source]
to_json(stream=None)[source]
to_yaml(stream=None)[source]
exception ramble.mirror.MirrorError(msg, long_msg=None)[source]

Bases: RambleError

Superclass of all mirror-creation related errors.

class ramble.mirror.MirrorReference(cosmetic_path, global_path=None)[source]

Bases: object

A MirrorReference stores the relative paths where you can store a resource in a mirror directory.

The appropriate storage location is given by storage_path. The cosmetic_path property provides a reference that a human could generate themselves based on reading the details of the input.

A user can iterate over a MirrorReference object to get all the possible names that might be used to refer to the resource in a mirror; this includes names generated by previous naming schemes that are no-longer reported by storage_path or cosmetic_path.

property storage_path
class ramble.mirror.MirrorStats[source]

Bases: object

added(resource)[source]
already_existed(resource)[source]
error(resource)[source]
next_spec(spec)[source]
stats()[source]
ramble.mirror.add(name, url, scope)[source]

Add a named mirror in the given scope

ramble.mirror.mirror_archive_paths(fetcher, per_input_ref)[source]

Returns a MirrorReference object which keeps track of the relative storage path of the resource associated with the specified fetcher.

ramble.mirror.push_url_from_directory(output_directory)[source]

Given a directory in the local filesystem, return the URL on which to push resources.

ramble.mirror.push_url_from_mirror_name(mirror_name)[source]

Given a mirror name, return the URL on which to push resources.

ramble.mirror.push_url_from_mirror_url(mirror_url)[source]

Given a mirror URL, return the URL on which to push resources.

ramble.mirror.remove(name, scope)[source]

Remove the named mirror in the given scope

ramble.modifier module

Define base classes for modifier definitions

exception ramble.modifier.InvalidModeError(message, long_message=None)[source]

Bases: ModifierError

Exception raised when an invalid mode is passed

class ramble.modifier.ModifierBase(file_path)[source]

Bases: object

all_env_var_modifications()[source]
all_package_manager_requirements()[source]
all_pipeline_phases(pipeline)[source]
applies_to_executable(executable)[source]
apply_executable_modifiers(executable_name, executable, app_inst=None)[source]
archive_patterns = {}
artifact_inventory(workspace, app_inst=None)[source]

Return an inventory of modifier artifacts

Artifact inventories are up to the individual modifier to define the format of.

This will then show up in an experiment inventory.

Returns:

(Any) Artifact inventory for this modifier

builtins = {}
compilers = {}
copy()[source]

Deep copy a modifier instance

define_variable(var_name, var_value)[source]

Define a variable within this modifier’s expander instance

disabled = False
env_var_modifications = {}
executable_modifiers = {}
figure_of_merit_contexts = {}
figures_of_merit = {}
format_doc(**kwargs)[source]

Wrap doc string at 72 characters and format nicely

get_required_variables()[source]

Get all the required variables based on the mode.

inherit_from_application(app)[source]
maintainers: List[str] = []

Lists of strings which contains GitHub usernames of attributes. Do not include @ here in order not to unnecessarily ping the users.

modded_variables(app, extra_vars=None)[source]
mode_variables()[source]

Return a dict of variables that should be defined for the current mode

modes = {'disabled': {'description': 'Mode to disable all modifier functionality'}}
modifier_class = 'ModifierBase'
modifier_variables = {}
modify_experiment(app)[source]

Stubbed method to allow modification of experiment variables before an experiment is completely defined.

This can be used to define things like n_ranks and have it influence the name of the resulting experiment.

name = None
no_expand_vars()[source]

Iterator over non-expandable variables in current mode

Yields:

(str) – Variable name

package_manager_configs = {}
package_manager_requirements = {}
phase_definitions = {}
required_packages = {}
required_vars = {}
run_phase_hook(workspace, pipeline, hook_name)[source]

Run a modifier hook.

Hooks are internal functions named _{hook_name}.

This is a wrapper to extract the hook function, and execute it properly.

Hooks are only executed if they are not defined as a phase from the modifier.

set_on_executables(on_executables)[source]

Set the executables this modifier applies to.

If given an empty list or a value of None, the default of: ‘*’ is usage.

set_usage_mode(mode)[source]

Set the usage mode for this modifier.

If not set, or given an empty string the modifier tries to auto-detect a mode.

If it cannot auto detect the usage mode, an error is raised.

software_specs = {}
success_criteria = {}
tags: List[str] = []
templates = {}
variable_modifications = {}
exception ramble.modifier.ModifierError(message, long_message=None)[source]

Bases: RambleError

Exception that is raised by modifiers

ramble.modkit module

modkit is a set of useful modules to import when writing modifiers

ramble.namespace module

class ramble.namespace.namespace[source]

Bases: object

Class of namespace variables

application = 'applications'
chained_experiments = 'chained_experiments'
command = 'command'
commands = 'commands'
compiler = 'compiler'
compiler_spec = 'compiler_spec'
custom_executables = 'custom_executables'
env_var = 'env_vars'
environments = 'environments'
exclude = 'exclude'
executable_injection = 'executable_injection'
executables = 'executables'
experiment = 'experiments'
external_env = 'external_env'
formatted_executables = 'formatted_executables'
include = 'include'
indentation = 'indentation'
inherit_variables = 'inherit_variables'
internals = 'internals'
join_separator = 'join_separator'
matrices = 'matrices'
matrix = 'matrix'
metadata = 'metadata'
modifiers = 'modifiers'
n_repeats = 'n_repeats'
package_manager = 'package_manager'
packages = 'packages'
pkg_spec = 'pkg_spec'
prefix = 'prefix'
ramble = 'ramble'
software = 'software'
success = 'success_criteria'
tags = 'tags'
template = 'template'
variables = 'variables'
variants = 'variants'
where = 'where'
workflow_manager = 'workflow_manager'
workload = 'workloads'
zips = 'zips'

ramble.package_manager module

ramble.paths module

Defines paths that are part of Ramble’s directory structure.

Do not import other ramble modules here. This module is used throughout Ramble and should bring in a minimal number of external dependencies.

ramble.paths.bin_path = '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/bin'

bin directory in the ramble prefix

ramble.paths.prefix = '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root'

This file lives in $prefix/lib/ramble/ramble/__file__

ramble.paths.ramble_root = '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root'

synonym for prefix

ramble.paths.ramble_script = '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/bin/ramble'

The ramble script itself

ramble.paths.sbang_script = '/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root/bin/sbang'

The sbang script in the ramble installation

ramble.paths.user_config_path = '/home/docs/.ramble'

User configuration location

ramble.pipeline module

ramble.pkgmankit module

ramble.renderer module

exception ramble.renderer.RambleRendererError(message, long_message=None)[source]

Bases: RambleError

Class for all renderer errors

class ramble.renderer.RenderGroup(obj_type=None, action='create')[source]

Bases: object

copy_contents(in_group)[source]

Copy contents of in_group into self

from_dict(name_template, in_dict)[source]

Extract RenderGroup definitions from a dictionary

Dictionaries should follow the below format:

in_dict = {

‘variables’: {}, ‘zips’: {}, ‘matrix’: [], ‘matrices’: {} or [],

}

Parameters:

  • name_template – The name template for the objects this group represents

  • in_dict – A dictionary representing the group definitions

Returns:

True if anything was extracted from the dictionary

Return type:

boolean

class ramble.renderer.Renderer[source]

Bases: object

render_objects(render_group, exclude_where=None, ignore_used=True, fatal=True)[source]

Render objects based on the input variables and matrices

Internally collects all matrix and vector variables.

First, zips are created. Zips extract vector variables, and group them into a higher level name.

Next, matrices are processed. Matrices consume vector variables, or explicit zips.

Vectors in the same matrix are crossed, sibling matrices are zipped. All matrices are required to result in the same number of elements, but not be the same shape.

Matrix elements are only allowed to be the names of variables. These variables are required to be vectors.

After matrices are processed, any remaining vectors are zipped together. All vectors are required to be of the same size.

The resulting zip of vectors is then crossed with all of the matrices to build a final list of objects.

After processing the expansion logic, this function yields a dictionary of variable definitions, one for each object that would be rendered.

If n_repeats is defined in input variables, this function yields one base and n copies of the rendered variable dictionary.

Yields:

  • a dictionary of variables for single object definition

  • a Repeats object indicating if rendered object is a repeat and its index

ramble.repeats module

class ramble.repeats.Repeats[source]

Bases: object

Class to represent configuration of experiment repeats

set_repeat_index(index)[source]

Set the index for an instance of a repeated experiment

set_repeats(is_repeat_base, n_repeats)[source]

Set if this instance will use repeats

Parameters:

  • is_repeat_base (Bool) – True if this is the base experiment of a repeat set, False for singletons and for repeats

  • n_repeats (Int) – 0 for singletons and repeats, positive integer for base experiment

ramble.reports module

ramble.repository module

exception ramble.repository.BadRepoError(message, long_message=None)[source]

Bases: RepoError

Raised when repo layout is invalid.

exception ramble.repository.FailedConstructorError(name, exc_type, exc_obj, exc_tb, object_type=None)[source]

Bases: RepoError

Raised when an object’s class constructor fails.

class ramble.repository.FastObjectChecker(objects_path, object_file_name, object_type)[source]

Bases: Mapping

Cache that maps object names to the stats obtained on the ‘.py’ files associated with them.

For each repository a cache is maintained at class level, and shared among all instances referring to it. Update of the global cache is done lazily during instance initialization.

invalidate()[source]

Regenerate cache for this checker.

last_mtime()[source]
exception ramble.repository.IndexError(message, long_message=None)[source]

Bases: RepoError

Raised when there’s an error with an index.

class ramble.repository.Indexer(object_type=ObjectTypes.applications)[source]

Bases: object

Adaptor for indexes that need to be generated when repos are updated.

create()[source]
needs_update(pkg)[source]

Whether an update is needed when the application file hasn’t changed.

Returns:

True if this application needs its index

updated, False otherwise.

Return type:

(bool)

We already automatically update indexes when object files change, but other files (like patches) may change underneath the object file. This method can be used to check additional object-specific files whenever they’re loaded, to tell the RepoIndex to update the index just for that object.

abstractmethod read(stream)[source]

Read this index from a provided file object.

set_object_type(object_type)[source]
abstractmethod update(obj_fullname)[source]

Update the index in memory with information about an object.

abstractmethod write(stream)[source]

Write the index to a file object.

exception ramble.repository.InvalidNamespaceError(message, long_message=None)[source]

Bases: RepoError

Raised when an invalid namespace is encountered.

ramble.repository.NOT_PROVIDED = <object object>

Guaranteed unused default value for some functions.

exception ramble.repository.NoRepoConfiguredError(message, long_message=None)[source]

Bases: RepoError

Raised when there are no repositories configured.

class ramble.repository.ObjectNamespace(namespace)[source]

Bases: ModuleType

Allow lazy loading of modules.

class ramble.repository.ObjectTypes(*values)

Bases: Enum

applications = 1
base_applications = 5
base_modifiers = 6
base_package_managers = 7
base_workflow_managers = 8
modifiers = 2
package_managers = 3
workflow_managers = 4
class ramble.repository.Repo(root, object_type=ObjectTypes.applications)[source]

Bases: object

Class representing a object repository in the filesystem.

Each object repository must have a top-level configuration file called repo.yaml.

Currently, repo.yaml this must define:

namespace:

A Python namespace where the repository’s objects should live.

all_object_classes()[source]

Iterator over all object classes in the repository.

Use this with care, because loading objects is slow.

all_object_names()[source]

Returns a sorted list of all object names in the Repo.

all_objects()[source]

Iterator over all objects in the repository.

Use this with care, because loading objects is slow.

dirname_for_object_name(obj_name)[source]

Get the directory name for a particular object. This is the directory that contains its object.py file.

dump_provenance(spec, path)[source]

Dump provenance information for a spec to a particular path.

This dumps the object file. Raises UnknownObjectError if not found.

exists(obj_name)[source]

Whether a object with the supplied name exists.

filename_for_object_name(obj_name)[source]

Get the filename for the module we should load for a particular object. objects for a Repo live in $root/<object_name>/<object_type>.py

This will return a proper <object_type>.py path even if the object doesn’t exist yet, so callers will need to ensure the object exists before importing.

find_module(fullname, path=None)[source]

Python find_module import hook.

Returns this Repo if it can load the module; None if not.

get(spec)[source]

Returns the object associated with the supplied spec.

get_obj_class(obj_name)[source]

Get the class for the object out of its module.

First loads (or fetches from cache) a module for the object. Then extracts the object class from the module according to Ramble’s naming convention.

property index

Construct the index for this repo lazily.

is_prefix(fullname)[source]

True if fullname is a prefix of this Repo’s namespace.

last_mtime()[source]

Time a object file in this repo was last updated.

load_module(fullname)[source]

Python importer load hook.

Tries to load the module; raises an ImportError if it can’t.

object_path(spec)[source]
objects_with_tags(*tags)[source]
purge()[source]

Clear entire object instance cache.

real_name(import_name)[source]

Allow users to import Ramble objects using Python identifiers.

A python identifier might map to many different Ramble object names due to hyphen/underscore ambiguity.

Easy example:

num3proxy -> 3proxy

Ambiguous:

foo_bar -> foo_bar, foo-bar

More ambiguous:

foo_bar_baz -> foo_bar_baz, foo-bar-baz, foo_bar-baz, foo-bar_baz

property tag_index

Index of tags and which objects they’re defined on.

exception ramble.repository.RepoError(message, long_message=None)[source]

Bases: RambleError

Superclass for repository-related errors.

class ramble.repository.RepoIndex(object_checker, namespace, object_type=ObjectTypes.applications)[source]

Bases: object

Container class that manages a set of Indexers for a Repo.

This class is responsible for checking objects in a repository for updates (using FastObjectChecker) and for regenerating indexes when they’re needed.

Indexers should be added to the RepoIndex using add_index(name, indexer), and they should support the interface defined by Indexer, so that the RepoIndex can read, generate, and update stored indices.

Generated indexes are accessed by name via __getitem__().

add_indexer(name, indexer)[source]

Add an indexer to the repo index.

Parameters:

  • name (str) – name of this indexer

  • indexer (object) – an object that supports create(), read(), write(), and get_index() operations

class ramble.repository.RepoLoader(fullname, repo, object_name)[source]

Bases: _PrependFileLoader

Loads a Python module associated with a object in specific repository

class ramble.repository.RepoPath(*repos, object_type=ObjectTypes.applications)[source]

Bases: object

A RepoPath is a list of repos that function as one.

It functions exactly like a Repo, but it operates on the combined results of the Repos in its list instead of on a single object repository.

Parameters:

repos (list) – list Repo objects or paths to put in this RepoPath

all_object_classes()[source]
all_object_names()[source]

Return all unique object names in all repositories.

all_objects()[source]
dirname_for_object_name(obj_name)[source]
dump_provenance(spec, path)[source]

Dump provenance information for a spec to a particular path.

This dumps the object file and any associated patch files. Raises UnknownObjectError if not found.

exists(obj_name)[source]

Whether object with the give name exists in the path’s repos.

Note that virtual objects do not “exist”.

filename_for_object_name(obj_name)[source]
find_module(fullname, path=None)[source]

Implements precedence for overlaid namespaces.

Loop checks each namespace in self.repos for objects, and also handles loading empty containing namespaces.

first_repo()[source]

Get the first repo in precedence order.

get(spec)[source]

Returns the object associated with the supplied spec.

get_full_namespace(namespace)[source]

Returns the full namespace of a repository, given its relative one.

get_obj_class(obj_name)[source]

Find a class for the spec’s object and return the class object.

get_repo(namespace, default=<object object>)[source]

Get a repository by namespace.

Parameters:

namespace – Look up this namespace in the RepoPath, and return it if found.

Optional Arguments:

default:

If default is provided, return it when the namespace isn’t found. If not, raise an UnknownNamespaceError.

in_path(maybe_obj_path)[source]

Whether the path belongs to any of the repos.

last_mtime()[source]

Time a object file in this repo was last updated.

load_module(fullname)[source]

Handles loading container namespaces when necessary.

See Repo for how actual object modules are loaded.

objects_with_tags(*tags)[source]
put_first(repo)[source]

Add repo first in the search path.

put_last(repo)[source]

Add repo last in the search path.

remove(repo)[source]

Remove a repo from the search path.

repo_for_obj(spec)[source]

Given a spec, get the repository for its object.

class ramble.repository.ReposFinder(object_type=ObjectTypes.applications)[source]

Bases: object

MetaPathFinder class that loads a Python module corresponding to an object

Return a loader based on the inspection of the current global repository list.

compute_loader(fullname)[source]
find_spec(fullname, python_path, target=None)[source]
class ramble.repository.RepositoryNamespace(namespace)[source]

Bases: ModuleType

Allow lazy loading of modules.

class ramble.repository.RepositoryNamespaceLoader[source]

Bases: object

create_module(spec)[source]
exec_module(module)[source]
class ramble.repository.TagIndex(object_type=ObjectTypes.applications)[source]

Bases: Mapping

Maps tags to list of applications.

static from_json(stream, object_type)[source]
to_json(stream)[source]
update_object(obj_name)[source]

Updates an object in the tag index.

Parameters:

  • obj_name (str) – name of the object to be removed from the

  • index

class ramble.repository.TagIndexer(object_type=ObjectTypes.applications)[source]

Bases: Indexer

Lifecycle methods for a TagIndex on a Repo.

read(stream)[source]

Read this index from a provided file object.

update(obj_fullname)[source]

Update the index in memory with information about an object.

write(stream)[source]

Write the index to a file object.

exception ramble.repository.UnknownEntityError(message, long_message=None)[source]

Bases: RepoError

Raised when we encounter a object ramble doesn’t have.

exception ramble.repository.UnknownNamespaceError(namespace)[source]

Bases: UnknownEntityError

Raised when we encounter an unknown namespace

exception ramble.repository.UnknownObjectError(name, repo=None, object_type='Object')[source]

Bases: UnknownEntityError

Raised when we encounter an object ramble doesn’t have.

ramble.repository.additional_repository(repository, object_type=ObjectTypes.applications)[source]

Adds temporarily a repository to the default one.

Parameters:

repository – repository to be added

ramble.repository.all_object_names(object_type=ObjectTypes.applications)[source]

Convenience wrapper around ramble.repository.all_object_names().

ramble.repository.autospec(function)[source]

Decorator that automatically converts the first argument of a function to a Spec.

ramble.repository.create(configuration, object_type=ObjectTypes.applications)[source]

Create a RepoPath from a configuration object.

Parameters:

configuration (ramble.config.Configuration) – configuration object

ramble.repository.create_or_construct(path, namespace=None)[source]

Create a repository, or just return a Repo if it already exists.

ramble.repository.create_repo(root, namespace=None, subdir='applications', object_type=ObjectTypes.applications, unified_repo=True)[source]

Create a new repository in root with the specified namespace.

If the namespace is not provided, use basename of root. Return the canonicalized path and namespace of the created repository.

ramble.repository.get(spec, object_type=ObjectTypes.applications)[source]

Convenience wrapper around ramble.repository.get().

ramble.repository.list_object_files(obj_inst, object_type)[source]

List object file paths of the given object along the inheritance chain.

This is currently used by ramble deployment to copy relevant files to create a self-contained repo.

ramble.repository.set_path(repo, object_type=ObjectTypes.applications)[source]

Set the path singleton to a specific value.

Overwrite path and register it as an importer in sys.meta_path if it is a Repo or RepoPath.

ramble.repository.use_repositories(*paths_and_repos, object_type=ObjectTypes.applications)[source]

Use the repositories passed as arguments within the context manager.

Parameters:

*paths_and_repos – paths to the repositories to be used, or already constructed Repo objects

Returns:

Corresponding RepoPath object

ramble.software_environments module

ramble.spec module

ramble.spec.AT = 2

These are possible token types in the spec grammar.

ramble.spec.COLON = 3

These are possible token types in the spec grammar.

ramble.spec.COMMA = 4

These are possible token types in the spec grammar.

ramble.spec.DEP = 1

These are possible token types in the spec grammar.

ramble.spec.EQ = 8

These are possible token types in the spec grammar.

ramble.spec.FILE = 11

These are possible token types in the spec grammar.

ramble.spec.HASH = 0

These are possible token types in the spec grammar.

ramble.spec.ID = 9

These are possible token types in the spec grammar.

ramble.spec.OFF = 6

These are possible token types in the spec grammar.

ramble.spec.ON = 5

These are possible token types in the spec grammar.

ramble.spec.PCT = 7

These are possible token types in the spec grammar.

class ramble.spec.Spec(spec_like=None)[source]

Bases: object

cformat(*args, **kwargs)[source]

Same as format, but color defaults to auto instead of False.

copy()[source]
format(format_string='{name}', **kwargs)[source]

Prints out particular pieces of a spec, depending on what is in the format string.

Using the {attribute} syntax, any field of the spec can be selected. Those attributes can be recursive.

Commonly used attributes of the Spec for format strings include:

name
workloads
Parameters:

format_string (str) – string containing the format to be expanded

Keyword Arguments:

  • color (bool) – True if returned string is colored

  • transform (dict) – maps full-string formats to a callable that accepts a string and returns another one

property fullname
exception ramble.spec.SpecFormatStringError(message, long_message=None)[source]

Bases: SpecError

Called for errors in Spec format strings.

class ramble.spec.SpecLexer[source]

Bases: Lexer

Parses tokens that make up spack specs.

exception ramble.spec.SpecParseError(parse_error)[source]

Bases: SpecError

Wrapper for ParseError for when we’re parsing specs.

class ramble.spec.SpecParser(initial_spec=None)[source]

Bases: Parser

check_identifier(id=None)[source]

The only identifiers that can contain ‘.’ are versions, but version ids are context-sensitive so we have to check on a case-by-case basis. Call this if we detect a version id where it shouldn’t be.

do_parse()[source]
workload(name=None)[source]

Return the name of the workload

ramble.spec.VAL = 10

These are possible token types in the spec grammar.

ramble.spec.parse(string)[source]

Returns a spec from an input string.

ramble.spec.spec_id_re = '\\w[\\w.-]*'

Regex for fully qualified spec names. (e.g., builtin.hdf5)

ramble.stage module

class ramble.stage.DIYStage(path)[source]

Bases: object

Simple class that allows any directory to be a ramble input stage. Consequently, it does not expect or require that the source path adhere to the standard directory naming convention.

cache_local()[source]
check()[source]
create()[source]
destroy()[source]
expand_archive()[source]
property expanded

Returns True since the source_path must exist.

fetch(*args, **kwargs)[source]
managed_by_ramble = False
restage()[source]
class ramble.stage.InputStage(url_or_fetch_strategy, name=None, path=None, mirror_paths=None, keep=True, lock=True, search_fn=None)[source]

Bases: object

Manages a stage directory for containing input files.

A Stage object is a context manager that handles a directory where some workload inputs are downloaded. It handles fetching inputs, either as an archive to be expanded or by checking it out of a repository. A stage’s lifecycle looks like this:

with InputStage() as stage:      # Context manager creates the stage
                            # directory
    stage.fetch()           # Fetch an input archive into the stage.
    stage.expand_archive()  # Expand the archive into input_path.

Because the input files are needed to execute various experiments, the stage is retained by default.

You can also use the stage’s create/destroy functions manually, like this:

stage = InputStage()
try:
    stage.create()          # Explicitly create the stage directory.
    stage.fetch()           # Fetch a source archive into the stage.
    stage.expand_archive()  # Expand the archive into source_path.
    <install>               # Build and install the archive.
                            # (handled by user of Stage)
finally:
    stage.destroy()         # Explicitly destroy the stage directory.

InputStages are required to be named by default. The name should match a workload namespace.

property archive_file

Path to the source archive within this stage directory.

cache_local()[source]
cache_mirror(mirror, stats)[source]

Perform a fetch if the resource is not already cached

Parameters:
check()[source]

Check the downloaded archive against a checksum digest. No-op if this stage checks code out of a repository.

create()[source]

Ensures the top-level (config:build_stage) directory exists.

destroy()[source]

Removes this stage directory.

expand_archive()[source]

Changes to the stage directory and attempt to expand the downloaded archive. Fail if the stage is not set up or if the archive is not yet downloaded.

property expanded

Returns True if source path expanded; else False.

property expected_archive_files

Possible archive file paths.

fetch(mirror_only=False, err_msg=None)[source]

Retrieves the code or archive

Parameters:

  • mirror_only (bool) – only fetch from a mirror

  • err_msg (str or None) – the error message to display if all fetchers fail or None for the default fetch failure message

managed_by_ramble = True
restage()[source]

Removes the expanded archive path if it exists, then re-expands the archive.

property save_filename
set_subdir(subdir_name)[source]
property source_path

Returns the well-known source directory path.

stage_locks = {}

Most staging is managed by Ramble. DIYStage is one exception.

steal_source(dest)[source]

Copy the source_path directory in its entirety to directory dest

This operation creates/fetches/expands the stage if it is not already, and destroys the stage when it is done.

class ramble.stage.ResourceStage(url_or_fetch_strategy, root, resource, **kwargs)[source]

Bases: InputStage

expand_archive()[source]

Changes to the stage directory and attempt to expand the downloaded archive. Fail if the stage is not set up or if the archive is not yet downloaded.

restage()[source]

Removes the expanded archive path if it exists, then re-expands the archive.

exception ramble.stage.RestageError(message, long_message=None)[source]

Bases: StageError

“Error encountered during restaging.

class ramble.stage.StageComposite[source]

Bases: Composite

Composite for Stage type objects. The first item in this composite is considered to be the root workload, and operations that return a value are forwarded to it.

property archive_file
property expanded
property path
property source_path
exception ramble.stage.StageError(message, long_message=None)[source]

Bases: RambleError

“Superclass for all errors encountered during staging.

exception ramble.stage.StagePathError(message, long_message=None)[source]

Bases: StageError

“Error encountered with stage path.

exception ramble.stage.VersionFetchError(message, long_message=None)[source]

Bases: StageError

Raised when we can’t determine a URL to fetch an input.

ramble.stage.create_stage_root(path)[source]

Create the stage root directory and ensure appropriate access perms.

ramble.stage.ensure_access(file)[source]

Ensure we can access a directory and die with an error if we can’t.

ramble.stage.get_checksums_for_versions(url_dict, name, first_stage_function=None, keep_stage=False, fetch_options=None, batch=False)[source]

Fetches and checksums archives from URLs.

This function is called by both ramble checksum and ramble create. The first_stage_function argument allows the caller to inspect the first downloaded archive, e.g., to determine the build system.

Parameters:

  • url_dict (dict) – A dictionary of the form: version -> URL

  • name (str) – The name of the input

  • first_stage_function (Callable) – function that takes a Stage and a URL; this is run on the stage of the first URL downloaded

  • keep_stage (bool) – whether to keep staging area when command completes

  • batch (bool) – whether to ask user how many versions to fetch (false) or fetch all versions (true)

  • fetch_options (dict) – Options used for the fetcher (such as timeout or cookies)

Returns:

A multi-line string containing versions and corresponding hashes

Return type:

(str)

ramble.success_criteria module

class ramble.success_criteria.ScopedCriteriaList[source]

Bases: object

A scoped list of success criteria

This object represents a list of success criteria. The criteria are scoped based on which portion of a workspace they are defined in.

Possible scopes are:

  • application_definition

  • application

  • workload

  • experiment

To see if success was met, all criteria will be checked and are AND-ed together.

add_criteria(scope, name, mode, *args, **kwargs)[source]
all_criteria()[source]
find_criteria(name)[source]
flush_scope(scope)[source]

Remove criteria within a scope, and lower level scopes

Scope to be purged are defined in self._flush_scopes.

passed()[source]
reset()[source]
validate_scope(scope)[source]
class ramble.success_criteria.SuccessCriteria(name, mode, match=None, file='{log_file}', fom_name=None, fom_context='null', formula=None, anti_match=None)[source]

Bases: object

A single success criteria object

This object represents a single criteria for success for a Ramble experiment.

anti_matched(test=None)[source]
mark_anti_found()[source]
mark_found()[source]
ok()[source]
passed(test=None, app_inst=None, fom_values=None)[source]
reset()[source]

ramble.wmkit module

wmkit is a set of useful modules to import when writing workflow managers

ramble.workflow_manager module

Define base classes for workflow manager definitions

class ramble.workflow_manager.WorkflowManagerBase(file_path)[source]

Bases: object

archive_patterns = {}
builtins = {}
compilers = {}
conditional_expand(templates)[source]

Return a (potentially empty) list of expanded strings

Parameters:

templates – A list of templates to expand. If the template cannot be fully expanded, it’s skipped.

Returns:

A list of expanded strings

copy()[source]

Deep copy a workflow manager instance

env_var_modifications = {}
executable_modifiers = {}
figure_of_merit_contexts = {}
figures_of_merit = {}
get_status(workspace)[source]

Return status of a given job

maintainers: List[str] = []
modes = {}
modifier_variables = {}
name = None
package_manager_configs = {}
package_manager_requirements = {}
phase_definitions = {}
required_packages = {}
required_vars = {}
set_application(app_inst)[source]

Set a reference to the associated app_inst

software_specs = {}
success_criteria = {}
tags: List[str] = []
template_render_vars()[source]

Define variables to be used in template rendering

templates = {}
variable_modifications = {}
wm_vars = {'hostfile': <ramble.workload.WorkloadVariable object>, 'workflow_banner': <ramble.workload.WorkloadVariable object>, 'workflow_hostfile_cmd': <ramble.workload.WorkloadVariable object>, 'workflow_pragmas': <ramble.workload.WorkloadVariable object>}

ramble.workload module

class ramble.workload.Workload(name: str, executables: List[str], inputs: List[str] | None = None, tags: List[str] | None = None)[source]

Bases: object

Class representing a single workload

add_environment_variable(env_var: WorkloadEnvironmentVariable)[source]

Add an environment variable to this workload

Parameters:

env_var (WorkloadEnvironmentVariable) – New environment variable to add to this workload

add_executable(executable: str)[source]

Add an executable to this workload

Parameters:

executable (str) – Name of executable to add to this workload

add_input(input: str)[source]

Add an input to this workload

Parameters:

input (str) – Name of input to add to this workload

add_tag(tag: str)[source]

Add a tag to this workload

Parameters:

tag (str) – Tag to add to this workload

add_variable(variable: WorkloadVariable)[source]

Add a variable to this workload

Parameters:

variable (WorkloadVariable) – New variable to add to this workload

as_str(n_indent: int = 0)[source]

String representation of this workload

Parameters:

n_indent (int) – Number of spaces to indent string with

Returns:

Representation of this workload

Return type:

(str)

find_environment_variable(name)[source]

Find an environment variable in this workload

Parameters:

env_var_name (str) – Name of environment variable to find

Returns:

Environment variable instance

if it exists, None if it is not found

Return type:

(WorkloadEnvironmentVariable / None)

find_executable(exec_name: str)[source]

Find an executable in this workload

Parameters:

exec_name (str) – Name of executable to find

Returns:

Name of executable if it exists, None if it is not found

Return type:

(str / None)

find_input(input_name)[source]

Find an input in this workload

Parameters:

input_name (str) – Name of input to find

Returns:

Name of input if it exists, None if it is not found

Return type:

(str / None)

find_variable(name)[source]

Find a variable in this workload

Parameters:

var_name (str) – Name of variable to find

Returns:

Variable instance if it exists, None if it is not found

Return type:

(WorkloadVariable / None)

is_valid()[source]

Test if this workload is considered valid

Returns:

True if workload is valid, False otherwise

Return type:

(bool)

class ramble.workload.WorkloadEnvironmentVariable(name: str, value=None, description: str = None)[source]

Bases: object

Class representing an environment variable in a workload

as_str(n_indent: int = 0)[source]

String representation of environment variable

Parameters:

n_indent (int) – Number of spaces to indent string representation by

Returns:

String representing this environment variable

Return type:

(str)

copy()[source]
class ramble.workload.WorkloadVariable(name: str, default=None, description: str = None, values=None, expandable: bool = True, track_used: bool = True, **kwargs)[source]

Bases: object

Class representing a variable definition

as_str(n_indent: int = 0)[source]

String representation of this variable

Parameters:

n_indent (int) – Number of spaces to indent string lines with

Returns:

Representation of this variable

Return type:

(str)

copy()[source]