ramble.util package

Subpackages

• ramble.util.imp package
  • Submodules
  • ramble.util.imp.imp_importer module
  • ramble.util.imp.importlib_importer module
    • PrependFileLoader
      • PrependFileLoader.get_data()
      • PrependFileLoader.path_stats()
    • load_source()

Submodules

ramble.util.class_attributes module

ramble.util.class_attributes.convert_class_attributes(obj)

Convert class attributes defined from directives to instance attributes Class attributes that are valid for conversion are stored in the _directive_names attribute.

Parameters:

obj (Object) – Input object instance to convert attributes in

ramble.util.colors module

ramble.util.colors.config_title(s)

ramble.util.colors.level_func(level)

ramble.util.colors.nested_1(s)

ramble.util.colors.nested_2(s)

ramble.util.colors.nested_3(s)

ramble.util.colors.nested_4(s)

ramble.util.colors.section_title(s)

ramble.util.command_runner module

class ramble.util.command_runner.CommandRunner(name=None, command=None, shell='bash', dry_run=False, path=None)

Bases: object

Runner for executing external commands

This class provides a generic wrapper on external commands, to provide a unified way to handle dry-run execution of external commands.

Can be inherited to construct custom command runners.

execute(executable, args: List[str], return_output: bool = False)

Wrapper around execution of a command

Handles execution of a command when the execution path is dependent on whether dry run is enabled or disabled.

Parameters:

• executable (spack.util.executable.Executable) – Executable to run with arguments

• args (list(str)) – List of string arguments to pass into executable

• return_output (bool) – Whether the output of the command should be returned or not

get_version()

Hook to get the version of the executable

Should return a string representation of the executable’s version.

set_dry_run(dry_run=False)

Set the dry_run state of this runner

exception ramble.util.command_runner.NoPathRunnerError

Bases: RunnerError

Raised when a runner is used that does not have a path set

exception ramble.util.command_runner.RunnerError

Bases: Exception

Raised when a problem occurs with a spack environment

exception ramble.util.command_runner.ValidationFailedError

Bases: RunnerError

Raised when a package manager requirement was not met

ramble.util.conversions module

ramble.util.conversions.list_str_to_list(in_str)

Convert a comma delimited list as a string into a python list

Parameters:

in_str (str) – Input string, comma delimited list of values

Returns:

(list) Each value from input string is a separate entry in the list.

ramble.util.directives module

ramble.util.directives.define_directive_methods(obj_inst)

Create class methods that execute directives

Wrap each directive, and inject it into this class instance as a class method.

This allows:

self.<directive_name>(<directive_args>) to be called. As in:

self.archive_pattern(’*.log’)

Which can be called within def __init__(self, file_path) instead of having to call archive_pattern(‘*.log’) at the class definition level.

This function requires the object instance to have internal attributes: - ‘_directive_classes’ - Dictionary mapping a directive to the class the

directive is defined for

• ‘_directive_functions’ - Dictionary mapping a directive to the decorated function that defines the directive

• ‘_language_classes’ - A list of classes that language features should be “imported” from

Both ‘_directive_classes’ and ‘_directive_functions’ are defined for all classes that use the DirectiveMeta meta-class.

The ‘_language_classes’ attribute is defined in ApplicationBase and ModifierBase.

ramble.util.directives.wrap_named_directive(obj_inst, name)

Wrap a directive to simplify execution

Create a wrapper method that executes a directive, to inject the (self) argument to simplify use of directives as class methods

ramble.util.editor module

Module for finding the user’s preferred text editor.

Defines one function, editor(), which invokes the editor defined by the user’s VISUAL environment variable if set. We fall back to the editor defined by the EDITOR environment variable if VISUAL is not set or the specified editor fails (e.g. no DISPLAY for a graphical editor). If neither variable is set, we fall back to one of several common editors, raising an EnvironmentError if we are unable to find one.

ramble.util.editor.editor(*args, **kwargs)

Invoke the user’s editor.

This will try to execute the following, in order:

1. $VISUAL <args> # the “visual” editor (per POSIX)

2. $EDITOR <args> # the regular editor (per POSIX)

3. some default editor (see _default_editors) with <args>

If an environment variable isn’t defined, it is skipped. If it points to something that can’t be executed, we’ll print a warning. And if we can’t find anything that can be executed after searching the full list above, we’ll raise an error.

Parameters:

args (list of str) – args to pass to editor

Optional Arguments:

_exec_func (function): invoke this function instead of os.execv()

ramble.util.env module

class ramble.util.env.RambleEnvModifications(other=None, traced=None)

Bases: EnvironmentModifications

shell_modifications(shell='sh', explicit=False, env=None)

Wrapper around spack’s shell_modifications

ramble.util.executable module

class ramble.util.executable.CommandExecutable(name, template, use_mpi=False, mpi=False, variables=None, redirect='{log_file}', output_capture=OUTPUT_CAPTURE.ALL, run_in_background=False, **kwargs)

Bases: object

CommandExecutable class This class is used to represent internal executables in Ramble.

These executables are portions of an experiment definition. They are generally used to group one or more commands together into an executable name.

copy()

Replicate a CommandExecutable instance

exception ramble.util.executable.CommandExecutableError(message, long_message=None)

Bases: RambleError

Class for errors when using command executable classes

class ramble.util.executable.PrefixedExecutable(name)

Bases: Executable

A version of spack.util.executable.Executable that allows command prefixes to be added

add_default_prefix(prefix)

Add a prefixed arg / cmd to the command

copy()

ramble.util.executable.which(*args, **kwargs)

Finds an executable in the path like command-line which.

If given multiple executables, returns the first one that is found. If no executables are found, returns None.

Parameters:

*args (str) – One or more executables to search for

Keyword Arguments:

• path (list or str) – The path to search. Defaults to PATH

• required (bool) – If set to True, raise an error if executable not found

Returns:

The first executable that is found in the path

Return type:

Executable

ramble.util.file_cache module

exception ramble.util.file_cache.CacheError(message, long_message=None)

Bases: RambleError

class ramble.util.file_cache.FileCache(root, timeout=120)

Bases: object

This class manages cached data in the filesystem.

• Cache files are fetched and stored by unique keys. Keys can be relative paths, so that there can be some hierarchy in the cache.

• The FileCache handles locking cache files for reading and writing, so client code need not manage locks for cache entries.

cache_path(key)

Path to the file in the cache for a particular key.

destroy()

Remove all files under the cache root.

init_entry(key)

Ensure we can access a cache file. Create a lock for it if needed.

Return whether the cache file exists yet or not.

mtime(key)

Return modification time of cache file, or 0 if it does not exist.

Time is in units returned by os.stat in the mtime field, which is platform-dependent.

read_transaction(key)

Get a read transaction on a file cache item.

Returns a ReadTransaction context manager and opens the cache file for reading. You can use it like this:

with file_cache_object.read_transaction(key) as cache_file:

cache_file.read()

remove(key)

write_transaction(key)

Get a write transaction on a file cache item.

Returns a WriteTransaction context manager that opens a temporary file for writing. Once the context manager finishes, if nothing went wrong, moves the file into place on top of the old file atomically.

ramble.util.file_util module

ramble.util.file_util.create_symlink(base, link)

Create symlink of a file to give a known and predictable path

ramble.util.file_util.get_file_path(path: str, workspace) → str

A wrapper for file paths used in a Ramble application, to facilitate testing.

Parameters:

• path (str) – A file path

• workspace (Workspace) – A ramble workspace

Returns:

A file path

Return type:

(str)

ramble.util.file_util.is_dry_run_path(path: str) → bool

Check if the path is already a dry_run path

ramble.util.foms module

class ramble.util.foms.BetterDirection(*values)

Bases: Enum

HIGHER = 1

INAPPLICABLE = 4

INDETERMINATE = 3

LOWER = 2

classmethod from_str(string)

class ramble.util.foms.FomType(*values)

Bases: Enum

CATEGORY = 4

INFO = 5

MEASURE = 3

THROUGHPUT = 2

TIME = 1

UNDEFINED = 6

better_direction()

copy()

classmethod from_str(string)

to_dict()

Converts the FomType enum member to a dictionary representation.

ramble.util.graph module

class ramble.util.graph.GraphNode(key, attribute=None, obj_inst=None)

Bases: object

Class representing a node of a graph, where the node can have an attribute attached to it.

This allows nodes to be added into a graph, have the topological order of the graph returned, and be able to refer to the attribute of the original node easily.

order_after(key)

Adds information that this node should come after another node

Parameters:

key (str) – Key of node that should come before this node.

order_before(key)

Adds information that this node should come before another node

Parameters:

key (str) – Key of node that should come after this node.

set_attribute(attr)

Sets the attribute of a graph node

Parameters:

attr – An arbitrary attribute to attach to this node.

ramble.util.hashing module

ramble.util.hashing.hash_file(file_path)

ramble.util.hashing.hash_json(in_json)

ramble.util.hashing.hash_string(string)

ramble.util.install_cache module

Provide data structures to assist with caching operations

class ramble.util.install_cache.SetCache

Bases: object

add(tupl)

contains(tupl)

ramble.util.lock module

Wrapper for llnl.util.lock allows locking to be enabled/disabled.

class ramble.util.lock.Lock(*args, **kwargs)

Bases: Lock

Lock that can be disabled.

This overrides the _lock() and _unlock() methods from llnl.util.lock so that all the lock API calls will succeed, but the actual locking mechanism can be disabled via _enable_locks.

cleanup(*args)

ramble.util.lock.check_lock_safety(path)

Do some extra checks to ensure disabling locks is safe.

This will raise an error if path can is group- or world-writable AND the current user can write to the directory (i.e., if this user AND others could write to the path).

This is intended to run on the Ramble prefix, but can be run on any path for testing.

ramble.util.logger module

class ramble.util.logger.Logger

Bases: object

Logger class

This class providers additional functionality on top of LLNL’s tty utility. Namely, this class provides a stack of log files, and allows errors to be printed to all log files instead of only one.

active_log()

Return the path for the active log

If any logs are in the log stack, return the filepath of the active log. Otherwise, return the string ‘stdout’

active_stream()

Return the stream for the active log

If any logs are in the log stack, return the stream object of the active log. Otherwise, return None to indicate the system is handling printing.

add_log(path)

Add a log to the current log stack

Opens (with ‘a+’ permissions) the file provided by the ‘path’ argument, and stores both the path, and the opened stream object in the current stack in the active position.

Parameters:

path – File path for the new log file

all_msg(*args, **kwargs)

Print a message to all logs

Pass all args and kwargs to tty.info (which will concatenate and print). Perform this action for all logs and the default log (to screen).

configure_colors(**kwargs)

debug(*args, **kwargs)

Print a debug message to the active log

Pass all args and kwargs to tty.debug (which will concatenate and print). Perform this action for the active log only.

die(*args, **kwargs)

Print an error message and terminate execution

Pass all args and kwargs to tty.error (which will concatenate and print). Perform this action all logs. After all logs are printed to, terminate execution (and error) using tty.die.

error(*args, **kwargs)

Print an error message

Pass all args and kwargs to tty.error (which will concatenate and print). Perform this action all logs, and the default stream (print to screen).

info(*args, **kwargs)

Print a message to the active log

Pass all args and kwargs to tty.info (which will concatenate and print). Perform this action for the active log only.

msg(*args, **kwargs)

Print a message to the active log

Pass all args and kwargs to tty.info (which will concatenate and print). Perform this action for the active log only.

remove_log()

Remove the active stack

Pop the active log from the log stack, and close the log stream.

verbose(*args, **kwargs)

Print a verbose message to the active log

Pass all args and kwargs to tty.verbose (which will concatenate and print). Perform this action for the active log only.

warn(*args, **kwargs)

Print a warning message to the active log

Pass all args and kwargs to tty.warn (which will concatenate and print). Perform this action for the active log only.

ramble.util.matrices module

ramble.util.matrices.extract_matrices(action, name, in_dict)

Extract matrix definitions from a dictionary

Parameters:

• action – The action of an object definition where matrices are being extracted from

• name – The name of the object

• in_dict – The dictionary containing definitions

Returns:

list of matrix definitions

ramble.util.naming module

class ramble.util.naming.NamespaceTrie(separator='.')

Bases: object

class Element(value)

Bases: object

has_value(namespace)

True if there is a value set for the given namespace.

is_leaf(namespace)

True if this namespace has no children in the trie.

is_prefix(namespace)

True if the namespace has a value, or if it’s the prefix of one that does.

ramble.util.naming.mod_to_class(mod_name)

Convert a name from module style to class name style. Ramble mostly follows PEP-8:

• Module and package names use lowercase_with_underscores.

• Class names use the CapWords convention.

Regular source code follows these conventions. Ramble is a bit more liberal with its Application names:

• They can contain ‘-’ as well as ‘_’, but cannot start with ‘-‘.

• They can start with numbers, e.g. “3proxy”.

This function converts from the module convention to the class convention by removing _ and - and converting surrounding lowercase text to CapWords. If mod_name starts with a number, the class name returned will be prepended with ‘_’ to make a valid Python identifier.

ramble.util.naming.possible_ramble_module_names(python_mod_name)

Given a Python module name, return a list of all possible ramble module names that could correspond to it.

ramble.util.naming.ramble_module_to_python_module(mod_name)

Given a Ramble module name, returns the name by which it can be imported in Python.

ramble.util.naming.simplify_name(name)

Simplify package name to only lowercase, digits, and dashes.

Simplifies a name which may include uppercase letters, periods, underscores, and pluses. In general, we want our package names to only contain lowercase letters, digits, and dashes.

Parameters:

name (str) – The original name of the package

Returns:

The new name of the package

Return type:

str

ramble.util.naming.valid_fully_qualified_module_name(mod_name)

Return whether mod_name is a valid namespaced module name.

ramble.util.naming.valid_module_name(mod_name)

Return whether mod_name is valid for use in Ramble.

ramble.util.naming.validate_fully_qualified_module_name(mod_name)

Raise an exception if mod_name is not a valid namespaced module name.

ramble.util.naming.validate_module_name(mod_name)

Raise an exception if mod_name is not valid.

ramble.util.output_capture module

class ramble.util.output_capture.OUTPUT_CAPTURE(*values)

Bases: Enum

ALL = 2

DEFAULT = 2

STDERR = 1

STDOUT = 0

class ramble.util.output_capture.output_mapper

Bases: object

generate_out_string(out_log, output_operator)

ramble.util.path module

Utilities for managing paths in ramble.

TODO: this is really part of ramble.config. Consolidate it.

ramble.util.path.canonicalize_path(path)

Same as substitute_path_variables, but also take absolute path.

ramble.util.path.substitute_config_variables(path, local_replacements)

Substitute placeholders into paths.

Ramble allows paths in configs to have some placeholders, as follows:

• $ramble The ramble instance’s prefix

• $user The current user’s username

• $tempdir Default temporary directory returned by tempfile.gettempdir()

These are substituted case-insensitively into the path, and users can use either $var or ${var} syntax for the variables.

ramble.util.path.substitute_path_variables(path, local_replacements=None)

Substitute config vars, expand environment vars, expand user home.

ramble.util.shell_utils module

Utils for handling shell specialization

exception ramble.util.shell_utils.UnsupportedError(message, long_message=None)

Bases: RambleError

Error when certain shell features are not supported.

ramble.util.shell_utils.cmd_sub_str(shell, cmd: str)

Get the command substitution string of the given shell

ramble.util.shell_utils.get_compatible_base_shell(shell)

Get the compatible base shell

For instance, given bash it returns sh. This is commonly used to translate shell name for calling spack utils, which only support such “base” shells.

Parameters:

shell (str) – Name of shell

Returns:

The base compatible shell.

Return type:

(str)

ramble.util.shell_utils.last_pid_var(shell: str) → str

ramble.util.shell_utils.source_str(shell)

Wrapper to return the source command for a given shell

Parameters:

shell (str) – Name of shell to get the source command for

Returns:

Source command for the requested shell.

Return type:

(str)

ramble.util.spec_utils module

ramble.util.spec_utils.specs_equiv(spec1, spec2)

ramble.util.stats module

class ramble.util.stats.StatsBase

Bases: object

compute(values)

get_unit(unit)

min_count = 1

name = ''

report(values, unit)

class ramble.util.stats.StatsCoefficientOfVariation

Bases: StatsBase

compute(values)

get_unit(unit)

min_count = 2

name = 'cv'

class ramble.util.stats.StatsMax

Bases: StatsBase

compute(values)

name = 'max'

class ramble.util.stats.StatsMean

Bases: StatsBase

compute(values)

name = 'mean'

class ramble.util.stats.StatsMedian

Bases: StatsBase

compute(values)

name = 'median'

class ramble.util.stats.StatsMin

Bases: StatsBase

compute(values)

name = 'min'

class ramble.util.stats.StatsStdev

Bases: StatsBase

compute(values)

min_count = 2

name = 'stdev'

class ramble.util.stats.StatsVar

Bases: StatsBase

compute(values)

get_unit(unit)

min_count = 2

name = 'variance'

ramble.util.stats.decimal_places(value)

Returns the number of decimal places of a value

ramble.util.stats.max_decimal_places(list)

Returns the max decimal places of a list of values

ramble.util.version module

ramble.util.version.get_git_hash(path='/home/docs/checkouts/readthedocs.org/user_builds/ramble/checkouts/v0.6.0/lib/ramble/docs/_ramble_root')

Get get hash from a path

Outputs ‘<git commit sha>’.

ramble.util.version.get_version()

Get a descriptive version of this instance of Ramble.

Outputs ‘<PEP440 version> (<git commit sha>)’.

The commit sha is only added when available.

ramble.util.web module

exception ramble.util.web.HTMLParseError

Bases: Exception

class ramble.util.web.LinkParser

Bases: HTMLParser

This parser just takes an HTML page and strips out the hrefs on the links. Good enough for a really simple spider.

handle_starttag(tag, attrs)

exception ramble.util.web.NoNetworkConnectionError(message, url)

Bases: SpackWebError

Raised when an operation can’t get an internet connection.

ramble.util.web.SPACK_USER_AGENT = 'Spackbot/0.18.0'

User-Agent used in Request objects

exception ramble.util.web.SpackWebError(message, long_message=None)

Bases: SpackError

Superclass for Spack web spidering errors.

ramble.util.web.find_versions_of_archive(archive_urls, list_url=None, list_depth=0, concurrency=32, reference_package=None)

Scrape web pages for new versions of a tarball. This function prefers URLs in the following order: links found on the scraped page that match a url generated by the reference package, found and in the archive_urls list, found and derived from those in the archive_urls list, and if none are found for a version then the item in the archive_urls list is included for the version.

Parameters:

• archive_urls (str or list or tuple) – URL or sequence of URLs for different versions of a package. Typically these are just the tarballs from the package file itself. By default, this searches the parent directories of archives.

• list_url (str or None) – URL for a listing of archives. Spack will scrape these pages for download links that look like the archive URL.

• list_depth (int) – max depth to follow links on list_url pages. Defaults to 0.

• concurrency (int) – maximum number of concurrent requests

• reference_package (spack.package.Package or None) – a spack package used as a reference for url detection. Uses the url_for_version method on the package to produce reference urls which, if found, are preferred.

ramble.util.web.get_header(headers, header_name)

Looks up a dict of headers for the given header value.

Looks up a dict of headers, [headers], for a header value given by [header_name]. Returns headers[header_name] if header_name is in headers. Otherwise, the first fuzzy match is returned, if any.

This fuzzy matching is performed by discarding word separators and capitalization, so that for example, “Content-length”, “content_length”, “conTENtLength”, etc., all match. In the case of multiple fuzzy-matches, the returned value is the “first” such match given the underlying mapping’s ordering, or unspecified if no such ordering is defined.

If header_name is not in headers, and no such fuzzy match exists, then a KeyError is raised.

ramble.util.web.list_url(url, recursive=False)

ramble.util.web.push_to_url(local_file_path, remote_path, keep_original=True, extra_args=None)

ramble.util.web.read_from_url(url, accept_content_type=None)

ramble.util.web.remove_url(url, recursive=False)

ramble.util.web.spider(root_urls, depth=0, concurrency=32)

Get web pages from root URLs.

If depth is specified (e.g., depth=2), then this will also follow up to <depth> levels of links from each root.

Parameters:

• root_urls (str or list) – root urls used as a starting point for spidering

• depth (int) – level of recursion into links

• concurrency (int) – number of simultaneous requests that can be sent

Returns:

A dict of pages visited (URL) mapped to their full text and the set of visited links.

ramble.util.web.url_exists(url)

ramble.util.web.uses_ssl(parsed_url)

ramble.util.web.warn_no_ssl_cert_checking()

ramble.util.yaml_generation module

Module representing utility functions for managing YAML configuration files within application definition files

These functions are intended to help read, write, and manipulate YAML based configuration files that an experiment might use as input.

Workload variables that represent configuration files should be defined using a ‘.’ delimiter between YAML object names. As an example:

``` foo:

bar:

baz: 1.0

```

Would translate to foo.bar.baz = 1.0 in Ramble syntax.

ramble.util.yaml_generation.all_config_options(config_data: Dict)

Extract all config options from config_data dictionary

Parameters:

config_data (dict) – A config dictionary representing data read from a YAML file.

Returns:

Set containing all detected fully qualified option names

Return type:

(set)

ramble.util.yaml_generation.apply_default_config_values(config_data, app_inst, default_config_string)

Apply default config values (from config_data) to an experiment

Process all workloads variables (for the current workload in app_inst). Any variable who’s expanded value is equal to default_config_string will have its value overwritten to the value in the config_dict dictionary.

Parameters:

• config_data (dict) – Dictionary of config data read from a YAML file

• app_inst (application) – Application instance representing an experiment

• default_config_string (str) – String that conveys the default config_data should be used in place of the current value.

ramble.util.yaml_generation.get_config_value(config_data: Dict, option_name: str)

Get a config option based on dictionary attribute syntax

Given an option_name of the format: attr1.attr2.attr3 return its value from config_data.

Parameters:

• config_data (dict) – A config dictionary representing data read from a YAML file.

• option_name (str) – Name of config option to get

Returns:

Value of config option

Return type:

(Any)

ramble.util.yaml_generation.read_config_file(conf_path: str)

Read an existing YAML file and return its data as a dictionary

Parameters:

conf_path (str) – Path to input configuration file to read

Returns:

Dictionary representation of the data contained in conf_path

Return type:

(dict)

ramble.util.yaml_generation.remove_config_value(config_data: Dict, option_name: str)

Remove a config option based on dictionary attribute syntax

Given an option_name of the format: attr1.attr2.attr3 remote it from config_data. Also, remove any parent scopes that are empty.

Parameters:

• config_data (dict) – A config dictionary representing data read from a YAML file.

• option_name (str) – Name of config option to set

ramble.util.yaml_generation.set_config_value(config_data: Dict, option_name: str, option_value: Any, force: bool = False)

Set a config option based on dictionary attribute syntax

Given an option_name of the format: attr1.attr2.attr3 set its value to option_value in config_data.

Parameters:

• config_data (dict) – A config dictionary representing data read from a YAML file.

• option_name (str) – Name of config option to set

• option_value (any) – Value to set config option to

• force (bool) – If true, all missing layers in the attribute list are created. If false, only sets existing attributes