diff options
| -rw-r--r-- | lib/spack/docs/getting_started.rst | 101 | ||||
| -rw-r--r-- | lib/spack/docs/module_file_support.rst | 12 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/bootstrap.py | 125 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/clone.py | 107 | ||||
| -rwxr-xr-x | share/spack/setup-env.sh | 57 | ||||
| -rw-r--r-- | var/spack/repos/builtin/packages/environment-modules/package.py | 5 |
6 files changed, 292 insertions, 115 deletions
diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index ef0f25cb04..ffa9bf2414 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -52,7 +52,7 @@ For a richer experience, use Spack's shell support: .. code-block:: console - # For bash users + # For bash/zsh users $ export SPACK_ROOT=/path/to/spack $ . $SPACK_ROOT/share/spack/setup-env.sh @@ -60,10 +60,15 @@ For a richer experience, use Spack's shell support: $ setenv SPACK_ROOT /path/to/spack $ source $SPACK_ROOT/share/spack/setup-env.csh + This automatically adds Spack to your ``PATH`` and allows the ``spack`` -command to :ref:`load environment modules <shell-support>` and execute +command to be used to execute spack :ref:`commands <shell-support>` and :ref:`useful packaging commands <packaging-shell-support>`. +If :ref:`environment-modules or dotkit <InstallEnvironmentModules>` is +installed and available, the ``spack`` command can also load and unload +:ref:`modules <modules>`. + ^^^^^^^^^^^^^^^^^ Clean Environment ^^^^^^^^^^^^^^^^^ @@ -94,12 +99,12 @@ Optional: Alternate Prefix ^^^^^^^^^^^^^^^^^^^^^^^^^^ You may want to run Spack out of a prefix other than the git repository -you cloned. The ``spack bootstrap`` command provides this +you cloned. The ``spack clone`` command provides this functionality. To install spack in a new directory, simply type: .. code-block:: console - $ spack bootstrap /my/favorite/prefix + $ spack clone /my/favorite/prefix This will install a new spack script in ``/my/favorite/prefix/bin``, which you can use just like you would the regular spack script. Each @@ -849,6 +854,10 @@ well. They can generally be activated as in the ``curl`` example above; or some systems might already have an appropriate hand-built environment module that may be loaded. Either way works. +If you find that you are missing some of these programs, ``spack`` can +build some of them for you with ``spack bootstrap``. Currently supported +programs are ``environment-modules``. + A few notes on specific programs in this list: """""""""""""""""""""""""" @@ -885,52 +894,72 @@ Environment Modules In order to use Spack's generated environment modules, you must have installed one of *Environment Modules* or *Lmod*. On many Linux distributions, this can be installed from the vendor's repository. For -example: ``yum install environment-modules`` (Fedora/RHEL/CentOS). If -your Linux distribution does not have Environment Modules, you can get it -with Spack: +example: ``yum install environment-modules`` (Fedora/RHEL/CentOS). If +your Linux distribution does not have Environment Modules, Spack can +build it for you! -#. Consider using system tcl (as long as your system has Tcl version 8.0 or later): +What follows are three steps describing how to install and use environment-modules with spack. - #) Identify its location using ``which tclsh`` - #) Identify its version using ``echo 'puts $tcl_version;exit 0' | tclsh`` - #) Add to ``~/.spack/packages.yaml`` and modify as appropriate: +#. Install ``environment-modules``. - .. code-block:: yaml + * ``spack bootstrap`` will build ``environment-modules`` for you (and may build + other packages that are useful to the operation of Spack) - packages: - tcl: - paths: - tcl@8.5: /usr - buildable: False + * Install ``environment-modules`` using ``spack install`` with + ``spack install environment-modules~X`` (The ``~X`` variant builds without Xorg + dependencies, but ``environment-modules`` works fine too.) -#. Install with: +#. Add ``modulecmd`` to ``PATH`` and create a ``module`` command. - .. code-block:: console + * If you are using ``bash`` or ``ksh``, Spack can currently do this for you as well. + After installing ``environment-modules`` following the step + above, source Spack's shell integration script. This will automatically + detect the lack of ``modulecmd`` and ``module``, and use the installed + ``environment-modules`` from ``spack bootstrap`` or ``spack install``. + + .. code-block:: console - $ spack install environment-modules + # For bash/zsh users + $ export SPACK_ROOT=/path/to/spack + $ . $SPACK_ROOT/share/spack/setup-env.sh -#. Activate with the following script (or apply the updates to your - ``.bashrc`` file manually): - .. code-block:: sh + * If you prefer to do it manually, you can activate with the following + script (or apply the updates to your ``.bashrc`` file manually): - TMP=`tempfile` - echo >$TMP - MODULE_HOME=`spack location --install-dir environment-modules` - MODULE_VERSION=`ls -1 $MODULE_HOME/Modules | head -1` - ${MODULE_HOME}/Modules/${MODULE_VERSION}/bin/add.modules <$TMP - cp .bashrc $TMP - echo "MODULE_VERSION=${MODULE_VERSION}" > .bashrc - cat $TMP >>.bashrc + .. code-block:: sh -This adds to your ``.bashrc`` (or similar) files, enabling Environment -Modules when you log in. Re-load your .bashrc (or log out and in -again), and then test that the ``module`` command is found with: + TMP=`tempfile` + echo >$TMP + MODULE_HOME=`spack location --install-dir environment-modules` + MODULE_VERSION=`ls -1 $MODULE_HOME/Modules | head -1` + ${MODULE_HOME}/Modules/${MODULE_VERSION}/bin/add.modules <$TMP + cp .bashrc $TMP + echo "MODULE_VERSION=${MODULE_VERSION}" > .bashrc + cat $TMP >>.bashrc -.. code-block:: console + This is added to your ``.bashrc`` (or similar) files, enabling Environment + Modules when you log in. + +#. Test that the ``module`` command is found with: - $ module avail + .. code-block:: console + + $ module avail + + +If ``tcl`` 8.0 or later is installed on your system, you can prevent +spack from rebuilding ``tcl`` as part of the ``environment-modules`` dependency +stack by adding the following to your ``~/.spack/packages.yaml`` replacing +version 8.5 with whatever version is installed on your system: + + .. code-block:: yaml + packages: + tcl: + paths: + tcl@8.5: /usr + buildable: False ^^^^^^^^^^^^^^^^^ Package Utilities diff --git a/lib/spack/docs/module_file_support.rst b/lib/spack/docs/module_file_support.rst index 93c2ee33c6..d8f5e561a8 100644 --- a/lib/spack/docs/module_file_support.rst +++ b/lib/spack/docs/module_file_support.rst @@ -44,6 +44,7 @@ For ``csh`` and ``tcsh`` instead: $ source $SPACK_ROOT/share/spack/setup-env.csh +When ``bash`` and ``ksh`` users update their environment with ``setup-env.sh``, it will check for spack-installed environment modules and add the ``module`` command to their environment; This only occurs if the module command is not already available. You can install ``environment-modules`` with ``spack bootstrap`` as described in :ref:`InstallEnvironmentModules`. .. note:: You can put the source line in your ``.bashrc`` or ``.cshrc`` to @@ -54,10 +55,11 @@ For ``csh`` and ``tcsh`` instead: Using module files via Spack ---------------------------- -If you have shell support enabled you should be able to run either -``module avail`` or ``use -l spack`` to see what module/dotkit files have -been installed. Here is sample output of those programs, showing lots -of installed packages. +If you have installed a supported module system either manually or through +``spack bootstrap`` and have enabled shell support, you should be able to +run either ``module avail`` or ``use -l spack`` to see what module/dotkit +files have been installed. Here is sample output of those programs, +showing lots of installed packages. .. code-block:: console @@ -210,7 +212,7 @@ Scripts to load modules recursively may be made with the command: $ spack module loads --dependencies <spec> -An equivalent alternative is: +An equivalent alternative using `process substitution <http://tldp.org/LDP/abs/html/process-sub.html>`_ is: .. code-block :: console diff --git a/lib/spack/spack/cmd/bootstrap.py b/lib/spack/spack/cmd/bootstrap.py index b0550644f8..cd0a21b9e0 100644 --- a/lib/spack/spack/cmd/bootstrap.py +++ b/lib/spack/spack/cmd/bootstrap.py @@ -22,86 +22,69 @@ # License along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## -import os - import llnl.util.tty as tty -from llnl.util.filesystem import join_path, mkdirp - import spack -from spack.util.executable import ProcessError, which - -_SPACK_UPSTREAM = 'https://github.com/llnl/spack' +import spack.cmd +import spack.cmd.common.arguments as arguments -description = "create a new installation of spack in another prefix" +description = "Bootstrap packages needed for spack to run smoothly" section = "admin" level = "long" def setup_parser(subparser): subparser.add_argument( - '-r', '--remote', action='store', dest='remote', - help="name of the remote to bootstrap from", default='origin') + '-j', '--jobs', action='store', type=int, + help="explicitly set number of make jobs. default is #cpus") subparser.add_argument( - 'prefix', - help="names of prefix where we should install spack") - - -def get_origin_info(remote): - git_dir = join_path(spack.prefix, '.git') - git = which('git', required=True) - try: - branch = git('symbolic-ref', '--short', 'HEAD', output=str) - except ProcessError: - branch = 'develop' - tty.warn('No branch found; using default branch: %s' % branch) - if remote == 'origin' and \ - branch not in ('master', 'develop'): - branch = 'develop' - tty.warn('Unknown branch found; using default branch: %s' % branch) - try: - origin_url = git( - '--git-dir=%s' % git_dir, - 'config', '--get', 'remote.%s.url' % remote, - output=str) - except ProcessError: - origin_url = _SPACK_UPSTREAM - tty.warn('No git repository found; ' - 'using default upstream URL: %s' % origin_url) - return (origin_url.strip(), branch.strip()) - - -def bootstrap(parser, args): - origin_url, branch = get_origin_info(args.remote) - prefix = args.prefix - - tty.msg("Fetching spack from '%s': %s" % (args.remote, origin_url)) - - if os.path.isfile(prefix): - tty.die("There is already a file at %s" % prefix) - - mkdirp(prefix) - - if os.path.exists(join_path(prefix, '.git')): - tty.die("There already seems to be a git repository in %s" % prefix) - - files_in_the_way = os.listdir(prefix) - if files_in_the_way: - tty.die("There are already files there! " - "Delete these files before boostrapping spack.", - *files_in_the_way) - - tty.msg("Installing:", - "%s/bin/spack" % prefix, - "%s/lib/spack/..." % prefix) + '--keep-prefix', action='store_true', dest='keep_prefix', + help="don't remove the install prefix if installation fails") + subparser.add_argument( + '--keep-stage', action='store_true', dest='keep_stage', + help="don't remove the build stage if installation succeeds") + subparser.add_argument( + '-n', '--no-checksum', action='store_true', dest='no_checksum', + help="do not check packages against checksum") + subparser.add_argument( + '-v', '--verbose', action='store_true', dest='verbose', + help="display verbose build output while installing") - os.chdir(prefix) - git = which('git', required=True) - git('init', '--shared', '-q') - git('remote', 'add', 'origin', origin_url) - git('fetch', 'origin', '%s:refs/remotes/origin/%s' % (branch, branch), - '-n', '-q') - git('reset', '--hard', 'origin/%s' % branch, '-q') - git('checkout', '-B', branch, 'origin/%s' % branch, '-q') + cd_group = subparser.add_mutually_exclusive_group() + arguments.add_common_arguments(cd_group, ['clean', 'dirty']) - tty.msg("Successfully created a new spack in %s" % prefix, - "Run %s/bin/spack to use this installation." % prefix) + subparser.add_argument( + '--run-tests', action='store_true', dest='run_tests', + help="run package level tests during installation" + ) + + +def bootstrap(parser, args, **kwargs): + kwargs.update({ + 'keep_prefix': args.keep_prefix, + 'keep_stage': args.keep_stage, + 'install_deps': 'dependencies', + 'make_jobs': args.jobs, + 'run_tests': args.run_tests, + 'verbose': args.verbose, + 'dirty': args.dirty + }) + + # Define requirement dictionary defining general specs which need + # to be satisfied, and the specs to install when the general spec + # isn't satisfied. + requirement_dict = {'environment-modules': 'environment-modules~X'} + + for requirement in requirement_dict: + installed_specs = spack.store.db.query(requirement) + if(len(installed_specs) > 0): + tty.msg("Requirement %s is satisfied with installed " + "package %s" % (requirement, installed_specs[0])) + else: + # Install requirement + spec_to_install = spack.Spec(requirement_dict[requirement]) + spec_to_install.concretize() + tty.msg("Installing %s to satisfy requirement for %s" % + (spec_to_install, requirement)) + kwargs['explicit'] = True + package = spack.repo.get(spec_to_install) + package.do_install(**kwargs) diff --git a/lib/spack/spack/cmd/clone.py b/lib/spack/spack/cmd/clone.py new file mode 100644 index 0000000000..19201b1f86 --- /dev/null +++ b/lib/spack/spack/cmd/clone.py @@ -0,0 +1,107 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## +import os + +import llnl.util.tty as tty +from llnl.util.filesystem import join_path, mkdirp + +import spack +from spack.util.executable import ProcessError, which + +_SPACK_UPSTREAM = 'https://github.com/llnl/spack' + +description = "create a new installation of spack in another prefix" +section = "admin" +level = "long" + + +def setup_parser(subparser): + subparser.add_argument( + '-r', '--remote', action='store', dest='remote', + help="name of the remote to clone from", default='origin') + subparser.add_argument( + 'prefix', + help="names of prefix where we should install spack") + + +def get_origin_info(remote): + git_dir = join_path(spack.prefix, '.git') + git = which('git', required=True) + try: + branch = git('symbolic-ref', '--short', 'HEAD', output=str) + except ProcessError: + branch = 'develop' + tty.warn('No branch found; using default branch: %s' % branch) + if remote == 'origin' and \ + branch not in ('master', 'develop'): + branch = 'develop' + tty.warn('Unknown branch found; using default branch: %s' % branch) + try: + origin_url = git( + '--git-dir=%s' % git_dir, + 'config', '--get', 'remote.%s.url' % remote, + output=str) + except ProcessError: + origin_url = _SPACK_UPSTREAM + tty.warn('No git repository found; ' + 'using default upstream URL: %s' % origin_url) + return (origin_url.strip(), branch.strip()) + + +def clone(parser, args): + origin_url, branch = get_origin_info(args.remote) + prefix = args.prefix + + tty.msg("Fetching spack from '%s': %s" % (args.remote, origin_url)) + + if os.path.isfile(prefix): + tty.die("There is already a file at %s" % prefix) + + mkdirp(prefix) + + if os.path.exists(join_path(prefix, '.git')): + tty.die("There already seems to be a git repository in %s" % prefix) + + files_in_the_way = os.listdir(prefix) + if files_in_the_way: + tty.die("There are already files there! " + "Delete these files before boostrapping spack.", + *files_in_the_way) + + tty.msg("Installing:", + "%s/bin/spack" % prefix, + "%s/lib/spack/..." % prefix) + + os.chdir(prefix) + git = which('git', required=True) + git('init', '--shared', '-q') + git('remote', 'add', 'origin', origin_url) + git('fetch', 'origin', '%s:refs/remotes/origin/%s' % (branch, branch), + '-n', '-q') + git('reset', '--hard', 'origin/%s' % branch, '-q') + git('checkout', '-B', branch, 'origin/%s' % branch, '-q') + + tty.msg("Successfully created a new spack in %s" % prefix, + "Run %s/bin/spack to use this installation." % prefix) diff --git a/share/spack/setup-env.sh b/share/spack/setup-env.sh index c9a3858621..958689cab4 100755 --- a/share/spack/setup-env.sh +++ b/share/spack/setup-env.sh @@ -27,7 +27,9 @@ # # This file is part of Spack and sets up the spack environment for # bash and zsh. This includes dotkit support, module support, and -# it also puts spack in your path. Source it like this: +# it also puts spack in your path. The script also checks that +# at least module support exists, and provides suggestions if it +# doesn't. Source it like this: # # . /path/to/spack/share/spack/setup-env.sh # @@ -189,19 +191,68 @@ if [ -z "$_sp_source_file" ]; then fi # -# Set up modules and dotkit search paths in the user environment +# Find root directory and add bin to path. # _sp_share_dir=$(cd "$(dirname $_sp_source_file)" && pwd) _sp_prefix=$(cd "$(dirname $(dirname $_sp_share_dir))" && pwd) _spack_pathadd PATH "${_sp_prefix%/}/bin" +export SPACK_ROOT=${_sp_prefix} + +# +# Determine which shell is being used +# +function _spack_determine_shell() { + ps -p $$ | tail -n 1 | awk '{print $4}' | xargs basename +} +export SPACK_SHELL=$(_spack_determine_shell) + +# +# Check whether a shell function of the given name is defined +# +function _spack_fn_exists() { + type $1 2>&1 | grep -q 'shell function' +} +need_module="no" +if [ ! $(_spack_fn_exists use) ] && [ ! $(_spack_fn_exists module) ]; then + need_module="yes" +fi; + +# +# build and make available environment-modules +# +if [ "${need_module}" = "yes" ]; then + #check if environment-modules~X is installed + module_prefix="$(spack location -i "environment-modules" 2>&1 || echo "not_installed")" + module_prefix=$(echo ${module_prefix} | tail -n 1) + if [ "${module_prefix}" != "not_installed" ]; then + #activate it! + export MODULE_PREFIX=${module_prefix} + _spack_pathadd PATH "${MODULE_PREFIX}/Modules/bin" + module() { eval `${MODULE_PREFIX}/Modules/bin/modulecmd ${SPACK_SHELL} $*`; } + echo "INFO: Using spack managed module system." + else + echo "WARNING: A method for managing modules does not currently exist." + echo "" + echo "To resolve this you may either:" + echo "1. Allow spack to handle this by running 'spack boostrap'" + echo " and sourcing this script again." + echo "2. Install and activate a supported module managment engine manually" + echo " Supported engines include: environment-modules and lmod" + fi; +else + echo "INFO: Using system available module system." +fi; + +# +# Set up modules and dotkit search paths in the user environment +# _sp_sys_type=$(spack-python -c 'print(spack.architecture.sys_type())') _sp_dotkit_root=$(spack-python -c "print(spack.util.path.canonicalize_path(spack.config.get_config('config').get('module_roots', {}).get('dotkit')))") _sp_tcl_root=$(spack-python -c "print(spack.util.path.canonicalize_path(spack.config.get_config('config').get('module_roots', {}).get('tcl')))") _spack_pathadd DK_NODE "${_sp_dotkit_root%/}/$_sp_sys_type" _spack_pathadd MODULEPATH "${_sp_tcl_root%/}/$_sp_sys_type" -# # Add programmable tab completion for Bash # if [ -n "${BASH_VERSION:-}" ]; then diff --git a/var/spack/repos/builtin/packages/environment-modules/package.py b/var/spack/repos/builtin/packages/environment-modules/package.py index 2774402605..fa0ea90fcf 100644 --- a/var/spack/repos/builtin/packages/environment-modules/package.py +++ b/var/spack/repos/builtin/packages/environment-modules/package.py @@ -35,6 +35,8 @@ class EnvironmentModules(Package): version('3.2.10', '8b097fdcb90c514d7540bb55a3cb90fb') + variant('X', default=True, description='Build with X functionality') + # Dependencies: depends_on('tcl', type=('build', 'link', 'run')) @@ -75,6 +77,9 @@ class EnvironmentModules(Package): 'CPPFLAGS=' + ' '.join(cpp_flags) ] + if '~X' in spec: + config_args = ['--without-x'] + config_args + configure(*config_args) make() make('install') |