diff options
| -rw-r--r-- | etc/spack/modules.yaml | 10 | ||||
| -rw-r--r-- | lib/spack/docs/basic_usage.rst | 272 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/module.py | 1 | ||||
| -rw-r--r-- | lib/spack/spack/config.py | 120 | ||||
| -rw-r--r-- | lib/spack/spack/environment.py | 16 | ||||
| -rw-r--r-- | lib/spack/spack/modules.py | 376 | ||||
| -rw-r--r-- | lib/spack/spack/package.py | 2 | ||||
| -rw-r--r-- | lib/spack/spack/test/__init__.py | 1 | ||||
| -rw-r--r-- | lib/spack/spack/test/modules.py | 143 | ||||
| -rwxr-xr-x | share/spack/setup-env.sh | 6 |
10 files changed, 827 insertions, 120 deletions
diff --git a/etc/spack/modules.yaml b/etc/spack/modules.yaml index aa2a2c3fe2..8f8f88e908 100644 --- a/etc/spack/modules.yaml +++ b/etc/spack/modules.yaml @@ -5,4 +5,14 @@ # although users can override these settings in their ~/.spack/modules.yaml. # ------------------------------------------------------------------------- modules: + prefix_inspections: { + bin: ['PATH'], + man: ['MANPATH'], + lib: ['LIBRARY_PATH', 'LD_LIBRARY_PATH'], + lib64: ['LIBRARY_PATH', 'LD_LIBRARY_PATH'], + include: ['CPATH'], + lib/pkgconfig: ['PKGCONFIG'], + lib64/pkgconfig: ['PKGCONFIG'], + '': ['CMAKE_PREFIX_PATH'] + } enable: ['tcl', 'dotkit'] diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index f08a2bb675..15db2f7a16 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -788,7 +788,7 @@ versions are now filtered out. .. _shell-support: -Environment modules +Integration with module systems ------------------------------- .. note:: @@ -798,42 +798,50 @@ Environment modules interface and/or generated module names may change in future versions. -Spack provides some limited integration with environment module -systems to make it easier to use the packages it provides. +Spack provides some integration with +`Environment Modules <http://modules.sourceforge.net/>`_ +and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_ to make +it easier to use the packages it installed. + Installing Environment Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to use Spack's generated environment modules, you must have installed the *Environment Modules* package. On many Linux -distributions, this can be installed from the vendor's repository. -For example: ```yum install environment-modules`` -(Fedora/RHEL/CentOS). If your Linux distribution does not have -Environment Modules, you can get it with Spack: +distributions, this can be installed from the vendor's repository: + +.. code-block:: sh -1. Install with:: + yum install environment-modules # (Fedora/RHEL/CentOS) + apt-get install environment-modules # (Ubuntu/Debian) + +If your Linux distribution does not have +Environment Modules, you can get it with Spack: .. code-block:: sh spack install environment-modules -2. Activate with:: -Add the following two lines to your ``.bashrc`` profile (or similar): +In this case to activate it automatically you need to add the following two +lines to your ``.bashrc`` profile (or similar): .. code-block:: sh MODULES_HOME=`spack location -i environment-modules` source ${MODULES_HOME}/Modules/init/bash -In case you use a Unix shell other than bash, substitute ``bash`` by -the appropriate file in ``${MODULES_HOME}/Modules/init/``. +If you use a Unix shell other than ``bash``, modify the commands above +accordingly and source the appropriate file in +``${MODULES_HOME}/Modules/init/``. -Spack and Environment Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. TODO : Add a similar section on how to install dotkit ? +Spack and module systems +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can enable shell support by sourcing some files in the ``/share/spack`` directory. @@ -841,7 +849,7 @@ For ``bash`` or ``ksh``, run: .. code-block:: sh - . $SPACK_ROOT/share/spack/setup-env.sh + . ${SPACK_ROOT}/share/spack/setup-env.sh For ``csh`` and ``tcsh`` run: @@ -853,17 +861,19 @@ For ``csh`` and ``tcsh`` run: You can put the above code in your ``.bashrc`` or ``.cshrc``, and Spack's shell support will be available on the command line. -When you install a package with Spack, it automatically generates an -environment module that lets you add the package to your environment. +When you install a package with Spack, it automatically generates a module file +that lets you add the package to your environment. -Currently, Spack supports the generation of `TCL Modules +Currently, Spack supports the generation of `Environment Modules <http://wiki.tcl.tk/12999>`_ and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_. Generated module files for each of these systems can be found in these directories: - * ``$SPACK_ROOT/share/spack/modules`` - * ``$SPACK_ROOT/share/spack/dotkit`` +.. code-block:: sh + + ${SPACK_ROOT}/share/spack/modules + ${SPACK_ROOT}/share/spack/dotkit The directories are automatically added to your ``MODULEPATH`` and ``DK_NODE`` environment variables when you enable Spack's `shell @@ -919,8 +929,7 @@ of installed packages. The names here should look familiar, they're the same ones from ``spack find``. You *can* use the names here directly. For example, -you could type either of these commands to load the callpath module -(assuming dotkit and modules are installed): +you could type either of these commands to load the callpath module: .. code-block:: sh @@ -935,7 +944,7 @@ easy to type. Luckily, Spack has its own interface for using modules and dotkits. You can use the same spec syntax you're used to: ========================= ========================== - Modules Dotkit + Environment Modules Dotkit ========================= ========================== ``spack load <spec>`` ``spack use <spec>`` ``spack unload <spec>`` ``spack unuse <spec>`` @@ -1002,15 +1011,216 @@ used ``gcc``. You could therefore just type: To identify just the one built with the Intel compiler. +Module files generation and customization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Environment Modules and Dotkit files are generated when packages are installed, +and are placed in the following directories under the Spack root: + +.. code-block:: sh + + ${SPACK_ROOT}/share/spack/modules + ${SPACK_ROOT}/share/spack/dotkit + +The content that gets written in each module file can be customized in two ways: + + 1. overriding part of the ``spack.Package`` API within a ``package.py`` + 2. writing dedicated configuration files + +Override ``Package`` API +^^^^^^^^^^^^^^^^^^^^^^^^ +There are currently two methods in ``spack.Package`` that may affect the content +of module files: + +.. code-block:: python + + def setup_environment(self, spack_env, run_env): + """Set up the compile and runtime environments for a package.""" + pass + +.. code-block:: python + + def setup_dependent_environment(self, spack_env, run_env, dependent_spec): + """Set up the environment of packages that depend on this one""" + pass + +As briefly stated in the comments, the first method lets you customize the +module file content for the package you are currently writing, the second +allows for modifications to your dependees module file. In both cases one +needs to fill ``run_env`` with the desired list of environment modifications. + +Example : ``builtin/packages/python/package.py`` +"""""""""""""""""""""""""""""""""""""""""""""""" + +The ``python`` package that comes with the ``builtin`` Spack repository +overrides ``setup_dependent_environment`` in the following way: + +.. code-block:: python + + def setup_dependent_environment(self, spack_env, run_env, extension_spec): + # ... + if extension_spec.package.extends(self.spec): + run_env.prepend_path('PYTHONPATH', os.path.join(extension_spec.prefix, self.site_packages_dir)) + +to insert the appropriate ``PYTHONPATH`` modifications in the module +files of python packages. + +Configuration files +^^^^^^^^^^^^^^^^^^^ + +Another way of modifying the content of module files is writing a +``modules.yaml`` configuration file. Following usual Spack conventions, this +file can be placed either at *site* or *user* scope. + +The default site configuration reads: + + .. literalinclude:: ../../../etc/spack/modules.yaml + :language: yaml + +It basically inspects the installation prefixes for the +existence of a few folders and, if they exist, it prepends a path to a given +list of environment variables. + +For each module system that can be enabled a finer configuration is possible: + +.. code-block:: yaml + + modules: + tcl: + # contains environment modules specific customizations + dotkit: + # contains dotkit specific customizations -Regenerating Module files -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The structure under the ``tcl`` and ``dotkit`` keys is almost equal, and will +be showcased in the following by some examples. -Module and dotkit files are generated when packages are installed, and -are placed in the following directories under the Spack root: +Select module files by spec constraints +""""""""""""""""""""""""""""""""""""""" +Using spec syntax it's possible to have different customizations for different +groups of module files. - * ``$SPACK_ROOT/share/spack/modules`` - * ``$SPACK_ROOT/share/spack/dotkit`` +Considering : + +.. code-block:: yaml + + modules: + tcl: + all: # Default addition for every package + environment: + set: + BAR: 'bar' + ^openmpi:: # A double ':' overrides previous rules + environment: + set: + BAR: 'baz' + zlib: + environment: + prepend_path: + LD_LIBRARY_PATH: 'foo' + zlib%gcc@4.8: + environment: + unset: + - FOOBAR + +what will happen is that: + + - every module file will set ``BAR=bar`` + - unless the associated spec satisfies ``^openmpi`` in which case ``BAR=baz`` + - any spec that satisfies ``zlib`` will additionally prepend ``foo`` to ``LD_LIBRARY_PATH`` + - any spec that satisfies ``zlib%gcc@4.8`` will additionally unset ``FOOBAR`` + +.. note:: + Order does matter + The modifications associated with the ``all`` keyword are always evaluated + first, no matter where they appear in the configuration file. All the other + spec constraints are instead evaluated top to bottom. + +Filter modifications out of module files +"""""""""""""""""""""""""""""""""""""""" + +Modifications to certain environment variables in module files are generated by +default. Suppose you would like to avoid having ``CPATH`` and ``LIBRARY_PATH`` +modified by your dotkit modules. Then : + +.. code-block:: yaml + + modules: + dotkit: + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] # Exclude changes to any of these variables + +will generate dotkit module files that will not contain modifications to either +``CPATH`` or ``LIBRARY_PATH`` and environment module files that instead will +contain those modifications. + +Autoload dependencies +""""""""""""""""""""" + +The following lines in ``modules.yaml``: + +.. code-block:: yaml + + modules: + tcl: + all: + autoload: 'direct' + +will produce environment module files that will automatically load their direct +dependencies. + +.. note:: + Allowed values for ``autoload`` statements + Allowed values for ``autoload`` statements are either ``none``, ``direct`` + or ``all``. In ``tcl`` configuration it is possible to use the option + ``prerequisites`` that accepts the same values and will add ``prereq`` + statements instead of automatically loading other modules. + +Blacklist or whitelist the generation of specific module files +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Sometimes it is desirable not to generate module files, a common use case being +not providing the users with software built using the system compiler. + +A configuration file like: + +.. code-block:: yaml + + modules: + tcl: + whitelist: ['gcc', 'llvm'] # Whitelist will have precedence over blacklist + blacklist: ['%gcc@4.4.7'] # Assuming gcc@4.4.7 is the system compiler + +will skip module file generation for anything that satisfies ``%gcc@4.4.7``, +with the exception of specs that satisfy ``gcc`` or ``llvm``. + +Customize the naming scheme and insert conflicts +"""""""""""""""""""""""""""""""""""""""""""""""" + +A configuration file like: + +.. code-block:: yaml + + modules: + tcl: + naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}' + all: + conflict: ['{name}', 'intel/14.0.1'] + +will create module files that will conflict with ``intel/14.0.1`` and with the +base directory of the same module, effectively preventing the possibility to +load two or more versions of the same software at the same time. + +.. note:: + Tokens available for the naming scheme + currently only the tokens shown in the example are available to construct + the naming scheme + +.. note:: + The ``conflict`` option is ``tcl`` specific + +Regenerating module files +^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes you may need to regenerate the modules files. For example, if newer, fancier module support is added to Spack at some later date, @@ -1020,7 +1230,7 @@ new features. .. _spack-module: ``spack module refresh`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +"""""""""""""""""""""""" Running ``spack module refresh`` will remove the ``share/spack/modules`` and ``share/spack/dotkit`` directories, then diff --git a/lib/spack/spack/cmd/module.py b/lib/spack/spack/cmd/module.py index a67f5c0c13..f996f4eb84 100644 --- a/lib/spack/spack/cmd/module.py +++ b/lib/spack/spack/cmd/module.py @@ -89,7 +89,6 @@ def module_refresh(): shutil.rmtree(cls.path, ignore_errors=False) mkdirp(cls.path) for spec in specs: - tty.debug(" Writing file for %s" % spec) cls(spec).write() diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py index 21e08ff666..34310077d7 100644 --- a/lib/spack/spack/config.py +++ b/lib/spack/spack/config.py @@ -118,21 +118,20 @@ Will make Spack take compilers *only* from the user configuration, and the site configuration will be ignored. """ +import copy import os import re import sys -import copy -import jsonschema -from jsonschema import Draft4Validator, validators -import yaml -from yaml.error import MarkedYAMLError -from ordereddict_backport import OrderedDict +import jsonschema import llnl.util.tty as tty -from llnl.util.filesystem import mkdirp - import spack +import yaml +from jsonschema import Draft4Validator, validators +from llnl.util.filesystem import mkdirp +from ordereddict_backport import OrderedDict from spack.error import SpackError +from yaml.error import MarkedYAMLError # Hacked yaml for configuration files preserves line numbers. import spack.util.spack_yaml as syaml @@ -146,7 +145,7 @@ section_schemas = { 'type': 'object', 'additionalProperties': False, 'patternProperties': { - 'compilers:?': { # optional colon for overriding site config. + 'compilers:?': { # optional colon for overriding site config. 'type': 'object', 'default': {}, 'additionalProperties': False, @@ -195,6 +194,7 @@ section_schemas = { 'default': [], 'items': { 'type': 'string'},},},}, + 'packages': { '$schema': 'http://json-schema.org/schema#', 'title': 'Spack package configuration file schema', @@ -238,24 +238,120 @@ section_schemas = { 'default' : {}, } },},},},},}, + 'modules': { '$schema': 'http://json-schema.org/schema#', 'title': 'Spack module file configuration file schema', 'type': 'object', 'additionalProperties': False, + 'definitions': { + 'array_of_strings': { + 'type': 'array', + 'default': [], + 'items': { + 'type': 'string' + } + }, + 'dictionary_of_strings': { + 'type': 'object', + 'patternProperties': { + r'\w[\w-]*': { # key + 'type': 'string' + } + } + }, + 'dependency_selection': { + 'type': 'string', + 'enum': ['none', 'direct', 'all'] + }, + 'module_file_configuration': { + 'type': 'object', + 'default': {}, + 'additionalProperties': False, + 'properties': { + 'filter': { + 'type': 'object', + 'default': {}, + 'additionalProperties': False, + 'properties': { + 'environment_blacklist': { + 'type': 'array', + 'default': [], + 'items': { + 'type': 'string' + } + } + } + }, + 'autoload': {'$ref': '#/definitions/dependency_selection'}, + 'prerequisites': {'$ref': '#/definitions/dependency_selection'}, + 'conflict': {'$ref': '#/definitions/array_of_strings'}, + 'environment': { + 'type': 'object', + 'default': {}, + 'additionalProperties': False, + 'properties': { + 'set': {'$ref': '#/definitions/dictionary_of_strings'}, + 'unset': {'$ref': '#/definitions/array_of_strings'}, + 'prepend_path': {'$ref': '#/definitions/dictionary_of_strings'}, + 'append_path': {'$ref': '#/definitions/dictionary_of_strings'} + } + } + } + }, + 'module_type_configuration': { + 'type': 'object', + 'default': {}, + 'anyOf': [ + { + 'properties': { + 'whitelist': {'$ref': '#/definitions/array_of_strings'}, + 'blacklist': {'$ref': '#/definitions/array_of_strings'}, + 'naming_scheme': { + 'type': 'string' # Can we be more specific here? + } + } + }, + { + 'patternProperties': {r'\w[\w-]*': {'$ref': '#/definitions/module_file_configuration'}} + } + ] + } + }, 'patternProperties': { r'modules:?': { 'type': 'object', 'default': {}, 'additionalProperties': False, 'properties': { + 'prefix_inspections': { + 'type': 'object', + 'patternProperties': { + r'\w[\w-]*': { # path to be inspected for existence (relative to prefix) + '$ref': '#/definitions/array_of_strings' + } + } + }, 'enable': { 'type': 'array', 'default': [], 'items': { - 'type': 'string' + 'type': 'string', + 'enum': ['tcl', 'dotkit'] } - } + }, + 'tcl': { + 'allOf': [ + {'$ref': '#/definitions/module_type_configuration'}, # Base configuration + {} # Specific tcl extensions + ] + }, + 'dotkit': { + 'allOf': [ + {'$ref': '#/definitions/module_type_configuration'}, # Base configuration + {} # Specific dotkit extensions + ] + }, } }, }, @@ -411,7 +507,7 @@ def _read_config_file(filename, schema): elif not os.path.isfile(filename): raise ConfigFileError( - "Invlaid configuration. %s exists but is not a file." % filename) + "Invalid configuration. %s exists but is not a file." % filename) elif not os.access(filename, os.R_OK): raise ConfigFileError("Config file is not readable: %s" % filename) diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py index 72aafa4e2d..92ab4e6bea 100644 --- a/lib/spack/spack/environment.py +++ b/lib/spack/spack/environment.py @@ -250,3 +250,19 @@ def validate(env, errstream): modifications = env.group_by_name() for variable, list_of_changes in sorted(modifications.items()): set_or_unset_not_first(variable, list_of_changes, errstream) + + +def filter_environment_blacklist(env, variables): + """ + Generator that filters out any change to environment variables present in the input list + + Args: + env: list of environment modifications + variables: list of variable names to be filtered + + Yields: + items in env if they are not in variables + """ + for item in env: + if item.name not in variables: + yield item diff --git a/lib/spack/spack/modules.py b/lib/spack/spack/modules.py index 61624fbd70..ffed469b20 100644 --- a/lib/spack/spack/modules.py +++ b/lib/spack/spack/modules.py @@ -40,16 +40,19 @@ various shell-support files in $SPACK/share/spack/setup-env.*. Each hook in hooks/ implements the logic for writing its specific type of module file. """ +import copy +import datetime import os import os.path import re -import shutil import textwrap +import string import llnl.util.tty as tty import spack import spack.config from llnl.util.filesystem import join_path, mkdirp +from spack.build_environment import parent_class_modules, set_module_variables_for_package from spack.environment import * __all__ = ['EnvModule', 'Dotkit', 'TclModule'] @@ -61,8 +64,9 @@ CONFIGURATION = spack.config.get_config('modules') def print_help(): - """For use by commands to tell user how to activate shell support.""" - + """ + For use by commands to tell user how to activate shell support. + """ tty.msg("This command requires spack's shell integration.", "", "To initialize spack's shell commands, you must run one of", @@ -72,7 +76,7 @@ def print_help(): " . %s/setup-env.sh" % spack.share_path, "", "For csh and tcsh:", - " setenv SPACK_ROOT %s" % spack.prefix, + " setenv SPACK_ROOT %s" % spack.prefix, " source %s/setup-env.csh" % spack.share_path, "") @@ -90,27 +94,138 @@ def inspect_path(prefix): """ env = EnvironmentModifications() # Inspect the prefix to check for the existence of common directories - prefix_inspections = { - 'bin': ('PATH',), - 'man': ('MANPATH',), - 'lib': ('LIBRARY_PATH', 'LD_LIBRARY_PATH'), - 'lib64': ('LIBRARY_PATH', 'LD_LIBRARY_PATH'), - 'include': ('CPATH',) - } - for attribute, variables in prefix_inspections.items(): - expected = getattr(prefix, attribute) + prefix_inspections = CONFIGURATION.get('prefix_inspections', {}) + for relative_path, variables in prefix_inspections.items(): + expected = join_path(prefix, relative_path) if os.path.isdir(expected): for variable in variables: env.prepend_path(variable, expected) - # PKGCONFIG - for expected in (join_path(prefix.lib, 'pkgconfig'), join_path(prefix.lib64, 'pkgconfig')): - if os.path.isdir(expected): - env.prepend_path('PKG_CONFIG_PATH', expected) - # CMake related variables - env.prepend_path('CMAKE_PREFIX_PATH', prefix) return env +def dependencies(spec, request='all'): + """ + Returns the list of dependent specs for a given spec, according to the given request + + Args: + spec: target spec + request: either 'none', 'direct' or 'all' + + Returns: + empty list if 'none', direct dependency list if 'direct', all dependencies if 'all' + """ + if request not in ('none', 'direct', 'all'): + raise tty.error("Wrong value for argument 'request' : should be one of ('none', 'direct', 'all') " + " [current value is '%s']" % request) + + if request == 'none': + return [] + + if request == 'direct': + return [xx for _, xx in spec.dependencies.items()] + + # FIXME : during module file creation nodes seem to be visited multiple times even if cover='nodes' + # FIXME : is given. This work around permits to get a unique list of spec anyhow. + # FIXME : Possibly we miss a merge step among nodes that refer to the same package. + seen = set() + seen_add = seen.add + l = [xx for xx in sorted(spec.traverse(order='post', depth=True, cover='nodes', root=False), reverse=True)] + return [xx for ii, xx in l if not (xx in seen or seen_add(xx))] + + +def update_dictionary_extending_lists(target, update): + for key in update: + value = target.get(key, None) + if isinstance(value, list): + target[key].extend(update[key]) + elif isinstance(value, dict): + update_dictionary_extending_lists(target[key], update[key]) + else: + target[key] = update[key] + + +def parse_config_options(module_generator): + """ + Parse the configuration file and returns a bunch of items that will be needed during module file generation + + Args: + module_generator: module generator for a given spec + + Returns: + autoloads: list of specs to be autoloaded + prerequisites: list of specs to be marked as prerequisite + filters: list of environment variables whose modification is blacklisted in module files + env: list of custom environment modifications to be applied in the module file + """ + # Get the configuration for this kind of generator + module_configuration = copy.deepcopy(CONFIGURATION.get(module_generator.name, {})) + + ##### + # Merge all the rules + ##### + module_file_actions = module_configuration.pop('all', {}) + for spec, conf in module_configuration.items(): + override = False + if spec.endswith(':'): + spec = spec.strip(':') + override = True + if module_generator.spec.satisfies(spec): + if override: + module_file_actions = {} + update_dictionary_extending_lists(module_file_actions, conf) + + ##### + # Process the common rules + ##### + + # Automatic loading loads + module_file_actions['autoload'] = dependencies(module_generator.spec, module_file_actions.get('autoload', 'none')) + # Prerequisites + module_file_actions['prerequisites'] = dependencies(module_generator.spec, module_file_actions.get('prerequisites', 'none')) + # Environment modifications + environment_actions = module_file_actions.pop('environment', {}) + env = EnvironmentModifications() + + def process_arglist(arglist): + if method == 'unset': + for x in arglist: + yield (x,) + else: + for x in arglist.iteritems(): + yield x + + for method, arglist in environment_actions.items(): + for args in process_arglist(arglist): + getattr(env, method)(*args) + + # for item in arglist: + # if method == 'unset': + # args = [item] + # else: + # args = item.split(',') + # getattr(env, method)(*args) + + return module_file_actions, env + + +def filter_blacklisted(specs, module_name): + """ + Given a sequence of specs, filters the ones that are blacklisted in the module configuration file. + + Args: + specs: sequence of spec instances + module_name: type of module file objects + + Yields: + non blacklisted specs + """ + for x in specs: + if module_types[module_name](x).blacklisted: + tty.debug('\tFILTER : %s' % x) + continue + yield x + + class EnvModule(object): name = 'env_module' formats = {} @@ -136,6 +251,36 @@ class EnvModule(object): if self.spec.package.__doc__: self.long_description = re.sub(r'\s+', ' ', self.spec.package.__doc__) + @property + def naming_scheme(self): + try: + naming_scheme = CONFIGURATION[self.name]['naming_scheme'] + except KeyError: + naming_scheme = self.default_naming_format + return naming_scheme + + @property + def tokens(self): + tokens = { + 'name': self.spec.name, + 'version': self.spec.version, + 'compiler': self.spec.compiler + } + return tokens + + @property + def use_name(self): + """ + Subclasses should implement this to return the name the module command uses to refer to the package. + """ + naming_tokens = self.tokens + naming_scheme = self.naming_scheme + name = naming_scheme.format(**naming_tokens) + name += '-' + self.spec.dag_hash() # Always append the hash to make the module file unique + # Not everybody is working on linux... + parts = name.split('/') + name = join_path(*parts) + return name @property def category(self): @@ -144,13 +289,46 @@ class EnvModule(object): return self.pkg.category # Extensions for extendee in self.pkg.extendees: - return '{extendee} extension'.format(extendee=extendee) + return '{extendee}_extension'.format(extendee=extendee) # Not very descriptive fallback - return 'spack installed package' + return 'spack' + @property + def blacklisted(self): + configuration = CONFIGURATION.get(self.name, {}) + whitelist_matches = [x for x in configuration.get('whitelist', []) if self.spec.satisfies(x)] + blacklist_matches = [x for x in configuration.get('blacklist', []) if self.spec.satisfies(x)] + if whitelist_matches: + message = '\tWHITELIST : %s [matches : ' % self.spec.cshort_spec + for rule in whitelist_matches: + message += '%s ' % rule + message += ' ]' + tty.debug(message) + + if blacklist_matches: + message = '\tBLACKLIST : %s [matches : ' % self.spec.cshort_spec + for rule in blacklist_matches: + message += '%s ' % rule + message += ' ]' + tty.debug(message) + + if not whitelist_matches and blacklist_matches: + return True + + return False def write(self): - """Write out a module file for this object.""" + """ + Writes out a module file for this object. + + This method employs a template pattern and expects derived classes to: + - override the header property + - provide formats for autoload, prerequisites and environment changes + """ + if self.blacklisted: + return + tty.debug("\tWRITE : %s [%s]" % (self.spec.cshort_spec, self.file_name)) + module_dir = os.path.dirname(self.file_name) if not os.path.exists(module_dir): mkdirp(module_dir) @@ -159,55 +337,72 @@ class EnvModule(object): # installation prefix env = inspect_path(self.spec.prefix) - # Let the extendee modify their extensions before asking for + # Let the extendee/dependency modify their extensions/dependencies before asking for # package-specific modifications spack_env = EnvironmentModifications() - for item in self.pkg.extendees: - try: - package = self.spec[item].package - package.setup_dependent_package(self.pkg.module, self.spec) - package.setup_dependent_environment(spack_env, env, self.spec) - except: - # The extends was conditional, so it doesn't count here - # eg: extends('python', when='+python') - pass + # TODO : the code down below is quite similar to build_environment.setup_package and needs to be + # TODO : factored out to a single place + for item in dependencies(self.spec, 'all'): + package = self.spec[item.name].package + modules = parent_class_modules(package.__class__) + for mod in modules: + set_module_variables_for_package(package, mod) + set_module_variables_for_package(package, package.module) + package.setup_dependent_package(self.pkg.module, self.spec) + package.setup_dependent_environment(spack_env, env, self.spec) # Package-specific environment modifications + set_module_variables_for_package(self.pkg, self.pkg.module) self.spec.package.setup_environment(spack_env, env) - # TODO : implement site-specific modifications and filters - if not env: - return - + # Parse configuration file + module_configuration, conf_env = parse_config_options(self) + env.extend(conf_env) + filters = module_configuration.get('filter', {}).get('environment_blacklist',{}) + # Build up the module file content + module_file_content = self.header + for x in filter_blacklisted(module_configuration.pop('autoload', []), self.name): + module_file_content += self.autoload(x) + for x in filter_blacklisted(module_configuration.pop('prerequisites', []), self.name): + module_file_content += self.prerequisite(x) + for line in self.process_environment_command(filter_environment_blacklist(env, filters)): + module_file_content += line + for line in self.module_specific_content(module_configuration): + module_file_content += line + + # Dump to file with open(self.file_name, 'w') as f: - self.write_header(f) - for line in self.process_environment_command(env): - f.write(line) + f.write(module_file_content) - def write_header(self, stream): + @property + def header(self): raise NotImplementedError() + def module_specific_content(self, configuration): + return tuple() + + def autoload(self, spec): + m = type(self)(spec) + return self.autoload_format.format(module_file=m.use_name) + + def prerequisite(self, spec): + m = type(self)(spec) + return self.prerequisite_format.format(module_file=m.use_name) + def process_environment_command(self, env): for command in env: try: - yield self.formats[type(command)].format(**command.args) + yield self.environment_modifications_formats[type(command)].format(**command.args) except KeyError: tty.warn('Cannot handle command of type {command} : skipping request'.format(command=type(command))) tty.warn('{context} at {filename}:{lineno}'.format(**command.args)) - @property def file_name(self): """Subclasses should implement this to return the name of the file where this module lives.""" raise NotImplementedError() - @property - def use_name(self): - """Subclasses should implement this to return the name the - module command uses to refer to the package.""" - raise NotImplementedError() - def remove(self): mod_file = self.file_name if os.path.exists(mod_file): @@ -222,42 +417,49 @@ class Dotkit(EnvModule): name = 'dotkit' path = join_path(spack.share_path, "dotkit") - formats = { + environment_modifications_formats = { PrependPath: 'dk_alter {name} {value}\n', SetEnv: 'dk_setenv {name} {value}\n' } + autoload_format = 'dk_op {module_file}\n' + + prerequisite_format = None # TODO : does something like prerequisite exist for dotkit? + + default_naming_format = '{name}-{version}-{compiler.name}-{compiler.version}' + @property def file_name(self): return join_path(Dotkit.path, self.spec.architecture, '%s.dk' % self.use_name) @property - def use_name(self): - return "%s-%s-%s-%s-%s" % (self.spec.name, self.spec.version, - self.spec.compiler.name, - self.spec.compiler.version, - self.spec.dag_hash()) - - def write_header(self, dk_file): + def header(self): # Category + header = '' if self.category: - dk_file.write('#c %s\n' % self.category) + header += '#c %s\n' % self.category # Short description if self.short_description: - dk_file.write('#d %s\n' % self.short_description) + header += '#d %s\n' % self.short_description # Long description if self.long_description: for line in textwrap.wrap(self.long_description, 72): - dk_file.write("#h %s\n" % line) + header += '#h %s\n' % line + return header + + def prerequisite(self, spec): + tty.warn('prerequisites: not supported by dotkit module files') + tty.warn('\tYou may want to check ~/.spack/modules.yaml') + return '' class TclModule(EnvModule): name = 'tcl' path = join_path(spack.share_path, "modules") - formats = { + environment_modifications_formats = { PrependPath: 'prepend-path {name} \"{value}\"\n', AppendPath: 'append-path {name} \"{value}\"\n', RemovePath: 'remove-path {name} \"{value}\"\n', @@ -265,28 +467,58 @@ class TclModule(EnvModule): UnsetEnv: 'unsetenv {name}\n' } + autoload_format = ('if ![ is-loaded {module_file} ] {{\n' + ' puts stderr "Autoloading {module_file}"\n' + ' module load {module_file}\n' + '}}\n\n') + + prerequisite_format = 'prereq {module_file}\n' + + default_naming_format = '{name}-{version}-{compiler.name}-{compiler.version}' + @property def file_name(self): return join_path(TclModule.path, self.spec.architecture, self.use_name) @property - def use_name(self): - return "%s-%s-%s-%s-%s" % (self.spec.name, self.spec.version, - self.spec.compiler.name, - self.spec.compiler.version, - self.spec.dag_hash()) - - def write_header(self, module_file): + def header(self): # TCL Modulefile header - module_file.write('#%Module1.0\n') + header = '#%Module1.0\n' + header += '## Module file created by spack (https://github.com/LLNL/spack) on %s\n' % datetime.datetime.now() + header += '##\n' + header += '## %s\n' % self.spec.short_spec + header += '##\n' + # TODO : category ? # Short description if self.short_description: - module_file.write('module-whatis \"%s\"\n\n' % self.short_description) + header += 'module-whatis \"%s\"\n\n' % self.short_description # Long description if self.long_description: - module_file.write('proc ModulesHelp { } {\n') + header += 'proc ModulesHelp { } {\n' for line in textwrap.wrap(self.long_description, 72): - module_file.write("puts stderr \"%s\"\n" % line) - module_file.write('}\n\n') + header += 'puts stderr "%s"\n' % line + header += '}\n\n' + return header + + def module_specific_content(self, configuration): + naming_tokens = self.tokens + # Conflict + conflict_format = configuration.get('conflict', []) + f = string.Formatter() + for item in conflict_format: + line = 'conflict ' + item + '\n' + if len([x for x in f.parse(line)]) > 1: # We do have placeholder to substitute + for naming_dir, conflict_dir in zip(self.naming_scheme.split('/'), item.split('/')): + if naming_dir != conflict_dir: + message = 'conflict scheme does not match naming scheme [{spec}]\n\n' + message += 'naming scheme : "{nformat}"\n' + message += 'conflict scheme : "{cformat}"\n\n' + message += '** You may want to check your `modules.yaml` configuration file **\n' + tty.error( + message.format(spec=self.spec, nformat=self.naming_scheme, cformat=item) + ) + raise SystemExit('Module generation aborted.') + line = line.format(**naming_tokens) + yield line diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index 4065553131..3626a574c8 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -1006,7 +1006,7 @@ class Package(object): fromlist=[self.__class__.__name__]) def setup_environment(self, spack_env, run_env): - """Set up the compile and runtime environemnts for a package. + """Set up the compile and runtime environments for a package. `spack_env` and `run_env` are `EnvironmentModifications` objects. Package authors can call methods on them to alter diff --git a/lib/spack/spack/test/__init__.py b/lib/spack/spack/test/__init__.py index 3c5edde66b..05f58ab7b1 100644 --- a/lib/spack/spack/test/__init__.py +++ b/lib/spack/spack/test/__init__.py @@ -54,6 +54,7 @@ test_names = ['versions', 'svn_fetch', 'hg_fetch', 'mirror', + 'modules', 'url_extrapolate', 'cc', 'link_tree', diff --git a/lib/spack/spack/test/modules.py b/lib/spack/spack/test/modules.py new file mode 100644 index 0000000000..b8b0d6fc6a --- /dev/null +++ b/lib/spack/spack/test/modules.py @@ -0,0 +1,143 @@ +import collections +from contextlib import contextmanager + +import StringIO + +FILE_REGISTRY = collections.defaultdict(StringIO.StringIO) + +# Monkey-patch open to write module files to a StringIO instance +@contextmanager +def mock_open(filename, mode): + if not mode == 'w': + raise RuntimeError('test.modules : unexpected opening mode for monkey-patched open') + + FILE_REGISTRY[filename] = StringIO.StringIO() + + try: + yield FILE_REGISTRY[filename] + finally: + handle = FILE_REGISTRY[filename] + FILE_REGISTRY[filename] = handle.getvalue() + handle.close() + +import spack.modules + +configuration_autoload_direct = { + 'enable': ['tcl'], + 'tcl': { + 'all': { + 'autoload': 'direct' + } + } +} + +configuration_autoload_all = { + 'enable': ['tcl'], + 'tcl': { + 'all': { + 'autoload': 'all' + } + } +} + +configuration_alter_environment = { + 'enable': ['tcl'], + 'tcl': { + 'all': { + 'filter': {'environment_blacklist': ['CMAKE_PREFIX_PATH']} + }, + '=x86-linux': { + 'environment': {'set': {'FOO': 'foo'}, 'unset': ['BAR']} + } + } +} + +configuration_blacklist = { + 'enable': ['tcl'], + 'tcl': { + 'blacklist': ['callpath'], + 'all': { + 'autoload': 'direct' + } + } +} + +configuration_conflicts = { + 'enable': ['tcl'], + 'tcl': { + 'naming_scheme': '{name}/{version}-{compiler.name}', + 'all': { + 'conflict': ['{name}', 'intel/14.0.1'] + } + } +} + +from spack.test.mock_packages_test import MockPackagesTest + + +class TclTests(MockPackagesTest): + def setUp(self): + super(TclTests, self).setUp() + self.configuration_obj = spack.modules.CONFIGURATION + spack.modules.open = mock_open + spack.modules.CONFIGURATION = None # Make sure that a non-mocked configuration will trigger an error + + def tearDown(self): + del spack.modules.open + spack.modules.CONFIGURATION = self.configuration_obj + super(TclTests, self).tearDown() + + def get_modulefile_content(self, spec): + spec.concretize() + generator = spack.modules.TclModule(spec) + generator.write() + content = FILE_REGISTRY[generator.file_name].split('\n') + return content + + def test_simple_case(self): + spack.modules.CONFIGURATION = configuration_autoload_direct + spec = spack.spec.Spec('mpich@3.0.4=x86-linux') + content = self.get_modulefile_content(spec) + self.assertTrue('module-whatis "mpich @3.0.4"' in content ) + + def test_autoload(self): + spack.modules.CONFIGURATION = configuration_autoload_direct + spec = spack.spec.Spec('mpileaks=x86-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if 'is-loaded' in x]), 2) + self.assertEqual(len([x for x in content if 'module load ' in x]), 2) + + spack.modules.CONFIGURATION = configuration_autoload_all + spec = spack.spec.Spec('mpileaks=x86-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if 'is-loaded' in x]), 5) + self.assertEqual(len([x for x in content if 'module load ' in x]), 5) + + def test_alter_environment(self): + spack.modules.CONFIGURATION = configuration_alter_environment + spec = spack.spec.Spec('mpileaks=x86-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if x.startswith('prepend-path CMAKE_PREFIX_PATH')]), 0) + self.assertEqual(len([x for x in content if 'setenv FOO "foo"' in x]), 1) + self.assertEqual(len([x for x in content if 'unsetenv BAR' in x]), 1) + + spec = spack.spec.Spec('libdwarf=x64-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if x.startswith('prepend-path CMAKE_PREFIX_PATH')]), 0) + self.assertEqual(len([x for x in content if 'setenv FOO "foo"' in x]), 0) + self.assertEqual(len([x for x in content if 'unsetenv BAR' in x]), 0) + + def test_blacklist(self): + spack.modules.CONFIGURATION = configuration_blacklist + spec = spack.spec.Spec('mpileaks=x86-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if 'is-loaded' in x]), 1) + self.assertEqual(len([x for x in content if 'module load ' in x]), 1) + + def test_conflicts(self): + spack.modules.CONFIGURATION = configuration_conflicts + spec = spack.spec.Spec('mpileaks=x86-linux') + content = self.get_modulefile_content(spec) + self.assertEqual(len([x for x in content if x.startswith('conflict')]), 2) + self.assertEqual(len([x for x in content if x == 'conflict mpileaks']), 1) + self.assertEqual(len([x for x in content if x == 'conflict intel/14.0.1']), 1) diff --git a/share/spack/setup-env.sh b/share/spack/setup-env.sh index 11a4c0a70c..dba6f1eff4 100755 --- a/share/spack/setup-env.sh +++ b/share/spack/setup-env.sh @@ -41,7 +41,7 @@ # commands. This allows the user to use packages without knowing all # their installation details. # -# e.g., rather than requring a full spec for libelf, the user can type: +# e.g., rather than requiring a full spec for libelf, the user can type: # # spack use libelf # @@ -113,11 +113,11 @@ function spack { unuse $_sp_module_args $_sp_full_spec fi ;; "load") - if _sp_full_spec=$(command spack $_sp_flags module find dotkit $_sp_spec); then + if _sp_full_spec=$(command spack $_sp_flags module find tcl $_sp_spec); then module load $_sp_module_args $_sp_full_spec fi ;; "unload") - if _sp_full_spec=$(command spack $_sp_flags module find dotkit $_sp_spec); then + if _sp_full_spec=$(command spack $_sp_flags module find tcl $_sp_spec); then module unload $_sp_module_args $_sp_full_spec fi ;; esac |