diff options
Diffstat (limited to 'lib')
53 files changed, 1893 insertions, 963 deletions
diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index dbad0a9f6d..df9a3901bf 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -183,7 +183,7 @@ To uninstall a package and every package that depends on it, you may give the spack uninstall --dependents mpich -will display a list of all the packages that depends on `mpich` and, upon confirmation, +will display a list of all the packages that depend on `mpich` and, upon confirmation, will uninstall them in the right order. A line like @@ -543,11 +543,12 @@ More formally, a spec consists of the following pieces: * ``+`` or ``-`` or ``~`` Optional variant specifiers (``+debug``, ``-qt``, or ``~qt``) for boolean variants * ``name=<value>`` Optional variant specifiers that are not restricted to -boolean variants + boolean variants * ``name=<value>`` Optional compiler flag specifiers. Valid flag names are -``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``, and ``ldlibs``. -* ``target=<value> os=<value>`` Optional architecture specifier -(``target=haswell os=CNL10``) * ``^`` Dependency specs (``^callpath@1.1``) + ``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``, and ``ldlibs``. +* ``target=<value> os=<value>`` Optional architecture specifier + (``target=haswell os=CNL10``) +* ``^`` Dependency specs (``^callpath@1.1``) There are two things to notice here. The first is that specs are recursively defined. That is, each dependency after ``^`` is a spec @@ -763,12 +764,12 @@ words ``target`` and/or ``os`` (``target=x86-64 os=debian7``). You can also use the triplet form of platform, operating system and processor. .. code-block:: sh - + spack install libelf arch=cray_xc-CNL10-haswell Users on non-Cray systems won't have to worry about specifying the architecture. Spack will autodetect what kind of operating system is on your machine as well -as the processor. For more information on how the architecture can be +as the processor. For more information on how the architecture can be used on Cray machines, check here :ref:`spack-cray` @@ -1146,11 +1147,12 @@ packages use RPATH to find their dependencies: this can be true in particular for Python extensions, which are currently *not* built with RPATH. -Modules may be loaded recursively with the command: +Modules may be loaded recursively with the ``load`` command's +``--dependencies`` or ``-r`` argument: .. code-block:: sh - $ module load `spack module tcl --dependencies <spec>... + $ spack load --dependencies <spec> ... More than one spec may be placed on the command line here. @@ -1792,36 +1794,36 @@ A nicer error message is TBD in future versions of Spack. Spack on Cray ----------------------------- -Spack differs slightly when used on a Cray system. The architecture spec +Spack differs slightly when used on a Cray system. The architecture spec can differentiate between the front-end and back-end processor and operating system. -For example, on Edison at NERSC, the back-end target processor +For example, on Edison at NERSC, the back-end target processor is \"Ivy Bridge\", so you can specify to use the back-end this way: .. code-block:: sh - + spack install zlib target=ivybridge You can also use the operating system to build against the back-end: .. code-block:: sh - + spack install zlib os=CNL10 -Notice that the name includes both the operating system name and the major +Notice that the name includes both the operating system name and the major version number concatenated together. -Alternatively, if you want to build something for the front-end, -you can specify the front-end target processor. The processor for a login node +Alternatively, if you want to build something for the front-end, +you can specify the front-end target processor. The processor for a login node on Edison is \"Sandy bridge\" so we specify on the command line like so: .. code-block:: sh - + spack install zlib target=sandybridge And the front-end operating system is: .. code-block:: sh - + spack install zlib os=SuSE11 @@ -1829,13 +1831,13 @@ And the front-end operating system is: Cray compiler detection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Spack can detect compilers using two methods. For the front-end, we treat -everything the same. The difference lies in back-end compiler detection. -Back-end compiler detection is made via the Tcl module avail command. -Once it detects the compiler it writes the appropriate PrgEnv and compiler -module name to compilers.yaml and sets the paths to each compiler with Cray\'s -compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load -the correct PrgEnv and compiler module and will call appropriate wrapper. +Spack can detect compilers using two methods. For the front-end, we treat +everything the same. The difference lies in back-end compiler detection. +Back-end compiler detection is made via the Tcl module avail command. +Once it detects the compiler it writes the appropriate PrgEnv and compiler +module name to compilers.yaml and sets the paths to each compiler with Cray\'s +compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load +the correct PrgEnv and compiler module and will call appropriate wrapper. The compilers.yaml config file will also differ. There is a modules section that is filled with the compiler\'s Programming Environment @@ -1848,8 +1850,8 @@ and module name. On other systems, this field is empty []:: - intel/15.0.109 ... -As mentioned earlier, the compiler paths will look different on a Cray system. -Since most compilers are invoked using cc, CC and ftn, the paths for each +As mentioned earlier, the compiler paths will look different on a Cray system. +Since most compilers are invoked using cc, CC and ftn, the paths for each compiler are replaced with their respective Cray compiler wrapper names:: ... @@ -1861,10 +1863,14 @@ compiler are replaced with their respective Cray compiler wrapper names:: ... As opposed to an explicit path to the compiler executable. This allows Spack -to call the Cray compiler wrappers during build time. +to call the Cray compiler wrappers during build time. For more on compiler configuration, check out :ref:`compiler-config`. +Spack sets the default Cray link type to dynamic, to better match other +other platforms. Individual packages can enable static linking (which is the +default outside of Spack on cray systems) using the -static flag. + Setting defaults and using Cray modules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1884,11 +1890,11 @@ Here\'s an example of an external configuration for cray modules: This tells Spack that for whatever package that depends on mpi, load the cray-mpich module into the environment. You can then be able to use whatever environment variables, libraries, etc, that are brought into the environment -via module load. +via module load. -You can set the default compiler that Spack can use for each compiler type. -If you want to use the Cray defaults, then set them under *all:* in packages.yaml. -In the compiler field, set the compiler specs in your order of preference. +You can set the default compiler that Spack can use for each compiler type. +If you want to use the Cray defaults, then set them under *all:* in packages.yaml. +In the compiler field, set the compiler specs in your order of preference. Whenever you build with that compiler type, Spack will concretize to that version. Here is an example of a full packages.yaml used at NERSC @@ -1916,11 +1922,11 @@ Here is an example of a full packages.yaml used at NERSC Here we tell spack that whenever we want to build with gcc use version 5.2.0 or if we want to build with intel compilers, use version 16.0.0.109. We add a spec -for each compiler type for each cray modules. This ensures that for each +for each compiler type for each cray modules. This ensures that for each compiler on our system we can use that external module. -For more on external packages check out the section :ref:`sec-external_packages`. +For more on external packages check out the section :ref:`sec-external_packages`. Getting Help ----------------------- diff --git a/lib/spack/docs/configuration.rst b/lib/spack/docs/configuration.rst index f2ffa07264..d39c932021 100644 --- a/lib/spack/docs/configuration.rst +++ b/lib/spack/docs/configuration.rst @@ -142,8 +142,9 @@ Here's an example packages.yaml file that sets preferred packages: .. code-block:: sh packages: - dyninst: + opencv: compiler: [gcc@4.9] + variants: +debug gperftools: version: [2.2, 2.4, 2.3] all: @@ -153,17 +154,17 @@ Here's an example packages.yaml file that sets preferred packages: At a high level, this example is specifying how packages should be -concretized. The dyninst package should prefer using gcc 4.9. -The gperftools package should prefer version +concretized. The opencv package should prefer using gcc 4.9 and +be built with debug options. The gperftools package should prefer version 2.2 over 2.4. Every package on the system should prefer mvapich for -its MPI and gcc 4.4.7 (except for Dyninst, which overrides this by preferring gcc 4.9). +its MPI and gcc 4.4.7 (except for opencv, which overrides this by preferring gcc 4.9). These options are used to fill in implicit defaults. Any of them can be overwritten on the command line if explicitly requested. Each packages.yaml file begins with the string ``packages:`` and package names are specified on the next level. The special string ``all`` applies settings to each package. Underneath each package name is -one or more components: ``compiler``, ``version``, +one or more components: ``compiler``, ``variants``, ``version``, or ``providers``. Each component has an ordered list of spec ``constraints``, with earlier entries in the list being preferred over later entries. diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 0f549e2957..70def5c39a 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -1286,6 +1286,31 @@ command line to find installed packages or to install packages with particular constraints, and package authors can use specs to describe relationships between packages. +Additionally, dependencies may be specified for specific use cases: + +.. code-block:: python + + depends_on("cmake", type="build") + depends_on("libelf", type=("build", "link")) + depends_on("python", type="run") + +The dependency types are: + + * **"build"**: made available during the project's build. The package will + be added to ``PATH``, the compiler include paths, and ``PYTHONPATH``. + Other projects which depend on this one will not have these modified + (building project X doesn't need project Y's build dependencies). + * **"link"**: the project is linked to by the project. The package will be + added to the current package's ``rpath``. + * **"run"**: the project is used by the project at runtime. The package will + be added to ``PATH`` and ``PYTHONPATH``. + +If not specified, ``type`` is assumed to be ``("build", "link")``. This is the +common case for compiled language usage. Also available are the aliases +``"alldeps"`` for all dependency types and ``"nolink"`` (``("build", "run")``) +for use by dependencies which are not expressed via a linker (e.g., Python or +Lua module loading). + .. _setup-dependent-environment: ``setup_dependent_environment()`` @@ -2602,14 +2627,14 @@ Spack packages with variants similar to already-existing Spack packages should use the same name for their variants. Standard variant names are: -======= ======== ======================== -Name Default Description -------- -------- ------------------------ -shared True Build shared libraries -static Build static libraries -mpi Use MPI -python Build Python extension -------- -------- ------------------------ + ======= ======== ======================== + Name Default Description + ======= ======== ======================== + shared True Build shared libraries + static Build static libraries + mpi Use MPI + python Build Python extension + ======= ======== ======================== If specified in this table, the corresponding default should be used when declaring a variant. diff --git a/lib/spack/env/cc b/lib/spack/env/cc index bf98b4c354..c6bb50d261 100755 --- a/lib/spack/env/cc +++ b/lib/spack/env/cc @@ -110,13 +110,13 @@ case "$command" in comp="CXX" lang_flags=CXX ;; - f90|fc|f95|gfortran|ifort|pgfortran|xlf90|nagfor) + ftn|f90|fc|f95|gfortran|ifort|pgfortran|xlf90|nagfor) command="$SPACK_FC" language="Fortran 90" comp="FC" lang_flags=F ;; - f77|gfortran|ifort|pgfortran|xlf|nagfor) + f77|gfortran|ifort|pgfortran|xlf|nagfor|ftn) command="$SPACK_F77" language="Fortran 77" comp="F77" diff --git a/lib/spack/env/cray/CC b/lib/spack/env/cray/CC new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/cray/CC @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/env/cray/cc b/lib/spack/env/cray/cc new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/cray/cc @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/env/cray/ftn b/lib/spack/env/cray/ftn new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/cray/ftn @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/env/craype/CC b/lib/spack/env/craype/CC new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/craype/CC @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/env/craype/cc b/lib/spack/env/craype/cc new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/craype/cc @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/env/craype/ftn b/lib/spack/env/craype/ftn new file mode 120000 index 0000000000..82c2b8e90a --- /dev/null +++ b/lib/spack/env/craype/ftn @@ -0,0 +1 @@ +../cc
\ No newline at end of file diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index e800c6717a..6e4cd338fe 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -42,7 +42,7 @@ __all__ = ['set_install_permissions', 'install', 'install_tree', 'FileFilter', 'change_sed_delimiter', 'is_exe', 'force_symlink', 'set_executable', 'copy_mode', 'unset_executable_mode', 'remove_dead_links', 'remove_linked_tree', 'find_library_path', - 'fix_darwin_install_name', 'to_link_flags'] + 'fix_darwin_install_name', 'to_link_flags', 'to_lib_name'] def filter_file(regex, repl, *filenames, **kwargs): @@ -431,6 +431,13 @@ def fix_darwin_install_name(path): break +def to_lib_name(library): + """Transforms a path to the library /path/to/lib<name>.xyz into <name> + """ + # Assume libXYZ.suffix + return os.path.basename(library)[3:].split(".")[0] + + def to_link_flags(library): """Transforms a path to a <library> into linking flags -L<dir> -l<name>. @@ -438,8 +445,7 @@ def to_link_flags(library): A string of linking flags. """ dir = os.path.dirname(library) - # Assume libXYZ.suffix - name = os.path.basename(library)[3:].split(".")[0] + name = to_lib_name(library) res = '-L%s -l%s' % (dir, name) return res diff --git a/lib/spack/spack/__init__.py b/lib/spack/spack/__init__.py index 20c9934704..d67585aac4 100644 --- a/lib/spack/spack/__init__.py +++ b/lib/spack/spack/__init__.py @@ -177,10 +177,11 @@ sys_type = None # should live. This file is overloaded for spack core vs. for packages. # __all__ = ['Package', 'StagedPackage', 'CMakePackage', \ - 'Version', 'when', 'ver'] + 'Version', 'when', 'ver', 'alldeps', 'nolink'] from spack.package import Package, ExtensionConflictError from spack.package import StagedPackage, CMakePackage from spack.version import Version, ver +from spack.spec import DependencySpec, alldeps, nolink from spack.multimethod import when import llnl.util.filesystem diff --git a/lib/spack/spack/architecture.py b/lib/spack/spack/architecture.py index a7cda2bf68..974505ee3a 100644 --- a/lib/spack/spack/architecture.py +++ b/lib/spack/spack/architecture.py @@ -76,7 +76,6 @@ attributes front_os and back_os. The operating system as described earlier, will be responsible for compiler detection. """ import os -import imp import inspect from llnl.util.lang import memoized, list_modules, key_ordering @@ -190,6 +189,12 @@ class Platform(object): return self.operating_sys.get(name, None) + @classmethod + def setup_platform_environment(self, pkg, env): + """ Subclass can override this method if it requires any + platform-specific build environment modifications. + """ + pass @classmethod def detect(self): @@ -200,15 +205,12 @@ class Platform(object): """ raise NotImplementedError() - def __repr__(self): return self.__str__() - def __str__(self): return self.name - def _cmp_key(self): t_keys = ''.join(str(t._cmp_key()) for t in sorted(self.targets.values())) @@ -279,7 +281,7 @@ class OperatingSystem(object): # ensure all the version calls we made are cached in the parent # process, as well. This speeds up Spack a lot. - clist = reduce(lambda x, y: x+y, compiler_lists) + clist = reduce(lambda x, y: x + y, compiler_lists) return clist def find_compiler(self, cmp_cls, *path): @@ -320,7 +322,7 @@ class OperatingSystem(object): # prefer the one with more compilers. prev_paths = [prev.cc, prev.cxx, prev.f77, prev.fc] - newcount = len([p for p in paths if p is not None]) + newcount = len([p for p in paths if p is not None]) prevcount = len([p for p in prev_paths if p is not None]) # Don't add if it's not an improvement over prev compiler. @@ -337,6 +339,7 @@ class OperatingSystem(object): d['version'] = self.version return d + @key_ordering class Arch(object): """Architecture is now a class to help with setting attributes. @@ -377,11 +380,9 @@ class Arch(object): else: return '' - def __contains__(self, string): return string in str(self) - def _cmp_key(self): if isinstance(self.platform, Platform): platform = self.platform.name @@ -424,7 +425,7 @@ def _operating_system_from_dict(os_name, plat=None): if isinstance(os_name, dict): name = os_name['name'] version = os_name['version'] - return plat.operating_system(name+version) + return plat.operating_system(name + version) else: return plat.operating_system(os_name) diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py index fe5186a7d7..5affd3c7c5 100644 --- a/lib/spack/spack/build_environment.py +++ b/lib/spack/spack/build_environment.py @@ -254,7 +254,8 @@ def set_build_environment_variables(pkg, env, dirty=False): env.set_path(SPACK_ENV_PATH, env_paths) # Prefixes of all of the package's dependencies go in SPACK_DEPENDENCIES - dep_prefixes = [d.prefix for d in pkg.spec.traverse(root=False)] + dep_prefixes = [d.prefix + for d in pkg.spec.traverse(root=False, deptype='build')] env.set_path(SPACK_DEPENDENCIES, dep_prefixes) # Add dependencies to CMAKE_PREFIX_PATH env.set_path('CMAKE_PREFIX_PATH', dep_prefixes) @@ -337,10 +338,6 @@ def set_module_variables_for_package(pkg, module): # Don't use which for this; we want to find it in the current dir. m.configure = Executable('./configure') - # TODO: shouldn't really use "which" here. Consider adding notion - # TODO: of build dependencies, as opposed to link dependencies. - # TODO: Currently, everything is a link dependency, but tools like - # TODO: this shouldn't be. m.cmake = Executable('cmake') m.ctest = Executable('ctest') @@ -388,9 +385,10 @@ def set_module_variables_for_package(pkg, module): def get_rpaths(pkg): """Get a list of all the rpaths for a package.""" rpaths = [pkg.prefix.lib, pkg.prefix.lib64] - rpaths.extend(d.prefix.lib for d in pkg.spec.dependencies.values() + deps = pkg.spec.dependencies(deptype='link') + rpaths.extend(d.prefix.lib for d in deps if os.path.isdir(d.prefix.lib)) - rpaths.extend(d.prefix.lib64 for d in pkg.spec.dependencies.values() + rpaths.extend(d.prefix.lib64 for d in deps if os.path.isdir(d.prefix.lib64)) # Second module is our compiler mod name. We use that to get rpaths from # module show output. @@ -446,10 +444,11 @@ def setup_package(pkg, dirty=False): set_compiler_environment_variables(pkg, spack_env) set_build_environment_variables(pkg, spack_env, dirty) + pkg.spec.architecture.platform.setup_platform_environment(pkg, spack_env) load_external_modules(pkg) # traverse in postorder so package can use vars from its dependencies spec = pkg.spec - for dspec in pkg.spec.traverse(order='post', root=False): + for dspec in pkg.spec.traverse(order='post', root=False, deptype='build'): # If a user makes their own package repo, e.g. # spack.repos.mystuff.libelf.Libelf, and they inherit from # an existing class like spack.repos.original.libelf.Libelf, diff --git a/lib/spack/spack/cmd/__init__.py b/lib/spack/spack/cmd/__init__.py index 672999159c..230115df50 100644 --- a/lib/spack/spack/cmd/__init__.py +++ b/lib/spack/spack/cmd/__init__.py @@ -27,16 +27,18 @@ import re import sys import llnl.util.tty as tty -from llnl.util.lang import attr_setdefault - import spack -import spack.spec import spack.config +import spack.spec +from llnl.util.lang import * +from llnl.util.tty.colify import * +from llnl.util.tty.color import * # # Settings for commands that modify configuration # -# Commands that modify confguration By default modify the *highest* priority scope. +# Commands that modify confguration By default modify the *highest* +# priority scope. default_modify_scope = spack.config.highest_precedence_scope().name # Commands that list confguration list *all* scopes by default. default_list_scope = None @@ -48,7 +50,7 @@ python_list = list ignore_files = r'^\.|^__init__.py$|^#' SETUP_PARSER = "setup_parser" -DESCRIPTION = "description" +DESCRIPTION = "description" command_path = os.path.join(spack.lib_path, "spack", "cmd") @@ -71,7 +73,7 @@ def get_module(name): module_name, fromlist=[name, SETUP_PARSER, DESCRIPTION], level=0) - attr_setdefault(module, SETUP_PARSER, lambda *args: None) # null-op + attr_setdefault(module, SETUP_PARSER, lambda *args: None) # null-op attr_setdefault(module, DESCRIPTION, "") fn_name = get_cmd_function_name(name) @@ -101,17 +103,17 @@ def parse_specs(args, **kwargs): specs = spack.spec.parse(args) for spec in specs: if concretize: - spec.concretize() # implies normalize + spec.concretize() # implies normalize elif normalize: spec.normalize() return specs - except spack.parse.ParseError, e: + except spack.parse.ParseError as e: tty.error(e.message, e.string, e.pos * " " + "^") sys.exit(1) - except spack.spec.SpecError, e: + except spack.spec.SpecError as e: tty.error(e.message) sys.exit(1) @@ -127,7 +129,7 @@ def elide_list(line_list, max_num=10): [1, 2, 3, '...', 6] """ if len(line_list) > max_num: - return line_list[:max_num-1] + ['...'] + line_list[-1:] + return line_list[:max_num - 1] + ['...'] + line_list[-1:] else: return line_list @@ -138,10 +140,104 @@ def disambiguate_spec(spec): tty.die("Spec '%s' matches no installed packages." % spec) elif len(matching_specs) > 1: - args = ["%s matches multiple packages." % spec, - "Matching packages:"] + args = ["%s matches multiple packages." % spec, + "Matching packages:"] args += [" " + str(s) for s in matching_specs] args += ["Use a more specific spec."] tty.die(*args) return matching_specs[0] + + +def ask_for_confirmation(message): + while True: + tty.msg(message + '[y/n]') + choice = raw_input().lower() + if choice == 'y': + break + elif choice == 'n': + raise SystemExit('Operation aborted') + tty.warn('Please reply either "y" or "n"') + + +def gray_hash(spec, length): + return colorize('@K{%s}' % spec.dag_hash(length)) + + +def display_specs(specs, **kwargs): + mode = kwargs.get('mode', 'short') + hashes = kwargs.get('long', False) + namespace = kwargs.get('namespace', False) + flags = kwargs.get('show_flags', False) + variants = kwargs.get('variants', False) + + hlen = 7 + if kwargs.get('very_long', False): + hashes = True + hlen = None + + nfmt = '.' if namespace else '_' + ffmt = '$%+' if flags else '' + vfmt = '$+' if variants else '' + format_string = '$%s$@%s%s' % (nfmt, ffmt, vfmt) + + # Make a dict with specs keyed by architecture and compiler. + index = index_by(specs, ('architecture', 'compiler')) + + # Traverse the index and print out each package + for i, (architecture, compiler) in enumerate(sorted(index)): + if i > 0: + print + + header = "%s{%s} / %s{%s}" % (spack.spec.architecture_color, + architecture, spack.spec.compiler_color, + compiler) + tty.hline(colorize(header), char='-') + + specs = index[(architecture, compiler)] + specs.sort() + + abbreviated = [s.format(format_string, color=True) for s in specs] + if mode == 'paths': + # Print one spec per line along with prefix path + width = max(len(s) for s in abbreviated) + width += 2 + format = " %%-%ds%%s" % width + + for abbrv, spec in zip(abbreviated, specs): + if hashes: + print(gray_hash(spec, hlen), ) + print(format % (abbrv, spec.prefix)) + + elif mode == 'deps': + for spec in specs: + print(spec.tree( + format=format_string, + color=True, + indent=4, + prefix=(lambda s: gray_hash(s, hlen)) if hashes else None)) + + elif mode == 'short': + # Print columns of output if not printing flags + if not flags: + + def fmt(s): + string = "" + if hashes: + string += gray_hash(s, hlen) + ' ' + string += s.format('$-%s$@%s' % (nfmt, vfmt), color=True) + + return string + + colify(fmt(s) for s in specs) + # Print one entry per line if including flags + else: + for spec in specs: + # Print the hash if necessary + hsh = gray_hash(spec, hlen) + ' ' if hashes else '' + print(hsh + spec.format(format_string, color=True) + '\n') + + else: + raise ValueError( + "Invalid mode for display_specs: %s. Must be one of (paths," + "deps, short)." % mode) # NOQA: ignore=E501 diff --git a/lib/spack/spack/cmd/bootstrap.py b/lib/spack/spack/cmd/bootstrap.py index bec11439b5..60e2bd3a11 100644 --- a/lib/spack/spack/cmd/bootstrap.py +++ b/lib/spack/spack/cmd/bootstrap.py @@ -23,7 +23,6 @@ # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## import os -from subprocess import check_call import llnl.util.tty as tty from llnl.util.filesystem import join_path, mkdirp @@ -31,26 +30,49 @@ from llnl.util.filesystem import join_path, mkdirp import spack from spack.util.executable import which +_SPACK_UPSTREAM = 'https://github.com/llnl/spack' + description = "Create a new installation of spack in another prefix" + def setup_parser(subparser): - subparser.add_argument('prefix', help="names of prefix where we should install spack") + subparser.add_argument( + '-r', '--remote', action='store', dest='remote', + help="name of the remote to bootstrap from", default='origin') + subparser.add_argument( + 'prefix', + help="names of prefix where we should install spack") -def get_origin_url(): +def get_origin_info(remote): git_dir = join_path(spack.prefix, '.git') git = which('git', required=True) - origin_url = git( - '--git-dir=%s' % git_dir, 'config', '--get', 'remote.origin.url', - output=str) - return origin_url.strip() + try: + branch = git('symbolic-ref', '--short', 'HEAD', output=str) + except ProcessError: + branch = 'develop' + tty.warn('No branch found; using default branch: %s' % branch) + if remote == 'origin' and \ + branch not in ('master', 'develop'): + branch = 'develop' + tty.warn('Unknown branch found; using default branch: %s' % branch) + try: + origin_url = git( + '--git-dir=%s' % git_dir, + 'config', '--get', 'remote.%s.url' % remote, + output=str) + except ProcessError: + origin_url = _SPACK_UPSTREAM + tty.warn('No git repository found; ' + 'using default upstream URL: %s' % origin_url) + return (origin_url.strip(), branch.strip()) def bootstrap(parser, args): - origin_url = get_origin_url() + origin_url, branch = get_origin_info(args.remote) prefix = args.prefix - tty.msg("Fetching spack from origin: %s" % origin_url) + tty.msg("Fetching spack from '%s': %s" % (args.remote, origin_url)) if os.path.isfile(prefix): tty.die("There is already a file at %s" % prefix) @@ -62,7 +84,8 @@ def bootstrap(parser, args): files_in_the_way = os.listdir(prefix) if files_in_the_way: - tty.die("There are already files there! Delete these files before boostrapping spack.", + tty.die("There are already files there! " + "Delete these files before boostrapping spack.", *files_in_the_way) tty.msg("Installing:", @@ -73,8 +96,10 @@ def bootstrap(parser, args): git = which('git', required=True) git('init', '--shared', '-q') git('remote', 'add', 'origin', origin_url) - git('fetch', 'origin', 'master:refs/remotes/origin/master', '-n', '-q') - git('reset', '--hard', 'origin/master', '-q') + git('fetch', 'origin', '%s:refs/remotes/origin/%s' % (branch, branch), + '-n', '-q') + git('reset', '--hard', 'origin/%s' % branch, '-q') + git('checkout', '-B', branch, 'origin/%s' % branch, '-q') tty.msg("Successfully created a new spack in %s" % prefix, "Run %s/bin/spack to use this installation." % prefix) diff --git a/lib/spack/spack/cmd/checksum.py b/lib/spack/spack/cmd/checksum.py index 95bd4771ed..aedb0fd99c 100644 --- a/lib/spack/spack/cmd/checksum.py +++ b/lib/spack/spack/cmd/checksum.py @@ -42,7 +42,8 @@ def setup_parser(subparser): '--keep-stage', action='store_true', dest='keep_stage', help="Don't clean up staging area when command completes.") subparser.add_argument( - 'versions', nargs=argparse.REMAINDER, help='Versions to generate checksums for') + 'versions', nargs=argparse.REMAINDER, + help='Versions to generate checksums for') def get_checksums(versions, urls, **kwargs): @@ -59,10 +60,10 @@ def get_checksums(versions, urls, **kwargs): with Stage(url, keep=keep_stage) as stage: stage.fetch() if i == 0 and first_stage_function: - first_stage_function(stage) + first_stage_function(stage, url) - hashes.append((version, - spack.util.crypto.checksum(hashlib.md5, stage.archive_file))) + hashes.append((version, spack.util.crypto.checksum( + hashlib.md5, stage.archive_file))) i += 1 except FailedDownloadError as e: tty.msg("Failed to fetch %s" % url) @@ -79,12 +80,12 @@ def checksum(parser, args): # If the user asked for specific versions, use those. if args.versions: versions = {} - for v in args.versions: - v = ver(v) - if not isinstance(v, Version): + for version in args.versions: + version = ver(version) + if not isinstance(version, Version): tty.die("Cannot generate checksums for version lists or " + "version ranges. Use unambiguous versions.") - versions[v] = pkg.url_for_version(v) + versions[version] = pkg.url_for_version(version) else: versions = pkg.fetch_remote_versions() if not versions: @@ -111,5 +112,7 @@ def checksum(parser, args): if not version_hashes: tty.die("Could not fetch any versions for %s" % pkg.name) - version_lines = [" version('%s', '%s')" % (v, h) for v, h in version_hashes] + version_lines = [ + " version('%s', '%s')" % (v, h) for v, h in version_hashes + ] tty.msg("Checksummed new versions of %s:" % pkg.name, *version_lines) diff --git a/lib/spack/spack/cmd/common/__init__.py b/lib/spack/spack/cmd/common/__init__.py new file mode 100644 index 0000000000..ed1ec23bca --- /dev/null +++ b/lib/spack/spack/cmd/common/__init__.py @@ -0,0 +1,24 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py new file mode 100644 index 0000000000..af04170824 --- /dev/null +++ b/lib/spack/spack/cmd/common/arguments.py @@ -0,0 +1,96 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## + +import argparse + +import spack.modules +from spack.util.pattern import Bunch +__all__ = ['add_common_arguments'] + +_arguments = {} + + +def add_common_arguments(parser, list_of_arguments): + for argument in list_of_arguments: + if argument not in _arguments: + message = 'Trying to add the non existing argument "{0}" to a command' # NOQA: ignore=E501 + raise KeyError(message.format(argument)) + x = _arguments[argument] + parser.add_argument(*x.flags, **x.kwargs) + + +class ConstraintAction(argparse.Action): + """Constructs a list of specs based on a constraint given on the command line + + An instance of this class is supposed to be used as an argument action + in a parser. It will read a constraint and will attach a list of matching + specs to the namespace + """ + qualifiers = {} + + def __call__(self, parser, namespace, values, option_string=None): + # Query specs from command line + d = self.qualifiers.get(namespace.subparser_name, {}) + specs = [s for s in spack.installed_db.query(**d)] + values = ' '.join(values) + if values: + specs = [x for x in specs if x.satisfies(values, strict=True)] + namespace.specs = specs + +parms = Bunch( + flags=('constraint',), + kwargs={ + 'nargs': '*', + 'help': 'Constraint to select a subset of installed packages', + 'action': ConstraintAction + }) +_arguments['constraint'] = parms + +parms = Bunch( + flags=('-m', '--module-type'), + kwargs={ + 'help': 'Type of module files', + 'default': 'tcl', + 'choices': spack.modules.module_types + }) +_arguments['module_type'] = parms + +parms = Bunch( + flags=('-y', '--yes-to-all'), + kwargs={ + 'action': 'store_true', + 'dest': 'yes_to_all', + 'help': 'Assume "yes" is the answer to every confirmation asked to the user.' # NOQA: ignore=E501 + }) +_arguments['yes_to_all'] = parms + +parms = Bunch( + flags=('-r', '--dependencies'), + kwargs={ + 'action': 'store_true', + 'dest': 'recurse_dependencies', + 'help': 'Recursively traverse spec dependencies' + }) +_arguments['recurse_dependencies'] = parms diff --git a/lib/spack/spack/cmd/create.py b/lib/spack/spack/cmd/create.py index 18c748e483..2c440096d1 100644 --- a/lib/spack/spack/cmd/create.py +++ b/lib/spack/spack/cmd/create.py @@ -103,6 +103,64 @@ ${versions} ${install} """) +# Build dependencies and extensions +dependencies_dict = { + 'autotools': "# depends_on('foo')", + 'cmake': "depends_on('cmake')", + 'scons': "depends_on('scons')", + 'python': "extends('python')", + 'R': "extends('R')", + 'octave': "extends('octave')", + 'unknown': "# depends_on('foo')" +} + +# Default installation instructions +install_dict = { + 'autotools': """\ + # FIXME: Modify the configure line to suit your build system here. + configure('--prefix={0}'.format(prefix)) + + # FIXME: Add logic to build and install here. + make() + make('install')""", + + 'cmake': """\ + with working_dir('spack-build', create=True): + # FIXME: Modify the cmake line to suit your build system here. + cmake('..', *std_cmake_args) + + # FIXME: Add logic to build and install here. + make() + make('install')""", + + 'scons': """\ + # FIXME: Add logic to build and install here. + scons('prefix={0}'.format(prefix)) + scons('install')""", + + 'python': """\ + # FIXME: Add logic to build and install here. + python('setup.py', 'install', '--prefix={0}'.format(prefix))""", + + 'R': """\ + # FIXME: Add logic to build and install here. + R('CMD', 'INSTALL', '--library={0}'.format(self.module.r_lib_dir), + self.stage.source_path)""", + + 'octave': """\ + # FIXME: Add logic to build and install here. + octave('--quiet', '--norc', + '--built-in-docstrings-file=/dev/null', + '--texi-macros-file=/dev/null', + '--eval', 'pkg prefix {0}; pkg install {1}'.format( + prefix, self.stage.archive_file))""", + + 'unknown': """\ + # FIXME: Unknown build system + make() + make('install')""" +} + def make_version_calls(ver_hash_tuples): """Adds a version() call to the package for each version found.""" @@ -133,60 +191,17 @@ def setup_parser(subparser): setup_parser.subparser = subparser -class ConfigureGuesser(object): - def __call__(self, stage): - """Try to guess the type of build system used by the project. - Set any necessary build dependencies or extensions. - Set the appropriate default installation instructions.""" - - # Build dependencies and extensions - dependenciesDict = { - 'autotools': "# depends_on('foo')", - 'cmake': "depends_on('cmake')", - 'scons': "depends_on('scons')", - 'python': "extends('python')", - 'R': "extends('R')", - 'unknown': "# depends_on('foo')" - } - - # Default installation instructions - installDict = { - 'autotools': """\ - # FIXME: Modify the configure line to suit your build system here. - configure('--prefix={0}'.format(prefix)) - - # FIXME: Add logic to build and install here. - make() - make('install')""", - - 'cmake': """\ - with working_dir('spack-build', create=True): - # FIXME: Modify the cmake line to suit your build system here. - cmake('..', *std_cmake_args) +class BuildSystemGuesser(object): + def __call__(self, stage, url): + """Try to guess the type of build system used by a project based on + the contents of its archive or the URL it was downloaded from.""" - # FIXME: Add logic to build and install here. - make() - make('install')""", - - 'scons': """\ - # FIXME: Add logic to build and install here. - scons('prefix={0}'.format(prefix)) - scons('install')""", - - 'python': """\ - # FIXME: Add logic to build and install here. - python('setup.py', 'install', '--prefix={0}'.format(prefix))""", - - 'R': """\ - # FIXME: Add logic to build and install here. - R('CMD', 'INSTALL', '--library={0}'.format(self.module.r_lib_dir), - self.stage.source_path)""", - - 'unknown': """\ - # FIXME: Unknown build system - make() - make('install')""" - } + # Most octave extensions are hosted on Octave-Forge: + # http://octave.sourceforge.net/index.html + # They all have the same base URL. + if 'downloads.sourceforge.net/octave/' in url: + self.build_system = 'octave' + return # A list of clues that give us an idea of the build system a package # uses. If the regular expression matches a file contained in the @@ -224,12 +239,6 @@ class ConfigureGuesser(object): self.build_system = build_system - # Set any necessary build dependencies or extensions. - self.dependencies = dependenciesDict[build_system] - - # Set the appropriate default installation instructions - self.install = installDict[build_system] - def guess_name_and_version(url, args): # Try to deduce name and version of the new package from the URL @@ -334,8 +343,8 @@ def create(parser, args): # Fetch tarballs (prompting user if necessary) versions, urls = fetch_tarballs(url, name, version) - # Try to guess what configure system is used. - guesser = ConfigureGuesser() + # Try to guess what build system is used. + guesser = BuildSystemGuesser() ver_hash_tuples = spack.cmd.checksum.get_checksums( versions, urls, first_stage_function=guesser, @@ -344,13 +353,13 @@ def create(parser, args): if not ver_hash_tuples: tty.die("Could not fetch any tarballs for %s" % name) - # Prepend 'py-' to python package names, by convention. + # Add prefix to package name if it is an extension. if guesser.build_system == 'python': - name = 'py-%s' % name - - # Prepend 'r-' to R package names, by convention. + name = 'py-{0}'.format(name) if guesser.build_system == 'R': - name = 'r-%s' % name + name = 'r-{0}'.format(name) + if guesser.build_system == 'octave': + name = 'octave-{0}'.format(name) # Create a directory for the new package. pkg_path = repo.filename_for_package_name(name) @@ -367,8 +376,8 @@ def create(parser, args): class_name=mod_to_class(name), url=url, versions=make_version_calls(ver_hash_tuples), - dependencies=guesser.dependencies, - install=guesser.install)) + dependencies=dependencies_dict[guesser.build_system], + install=install_dict[guesser.build_system])) # If everything checks out, go ahead and edit. spack.editor(pkg_path) diff --git a/lib/spack/spack/cmd/fetch.py b/lib/spack/spack/cmd/fetch.py index e40caaa234..1afc51d9fa 100644 --- a/lib/spack/spack/cmd/fetch.py +++ b/lib/spack/spack/cmd/fetch.py @@ -51,7 +51,7 @@ def fetch(parser, args): for spec in specs: if args.missing or args.dependencies: to_fetch = set() - for s in spec.traverse(): + for s in spec.traverse(deptype_query=spack.alldeps): package = spack.repo.get(s) if args.missing and package.installed: continue diff --git a/lib/spack/spack/cmd/find.py b/lib/spack/spack/cmd/find.py index 3ec671f93f..d3ea38c573 100644 --- a/lib/spack/spack/cmd/find.py +++ b/lib/spack/spack/cmd/find.py @@ -31,7 +31,7 @@ import spack.spec from llnl.util.lang import * from llnl.util.tty.colify import * from llnl.util.tty.color import * -from llnl.util.lang import * +from spack.cmd import display_specs description = "Find installed spack packages" @@ -104,89 +104,6 @@ def setup_parser(subparser): help='optional specs to filter results') -def gray_hash(spec, length): - return colorize('@K{%s}' % spec.dag_hash(length)) - - -def display_specs(specs, **kwargs): - mode = kwargs.get('mode', 'short') - hashes = kwargs.get('long', False) - namespace = kwargs.get('namespace', False) - flags = kwargs.get('show_flags', False) - variants = kwargs.get('variants', False) - - hlen = 7 - if kwargs.get('very_long', False): - hashes = True - hlen = None - - nfmt = '.' if namespace else '_' - ffmt = '$%+' if flags else '' - vfmt = '$+' if variants else '' - format_string = '$%s$@%s%s' % (nfmt, ffmt, vfmt) - - # Make a dict with specs keyed by architecture and compiler. - index = index_by(specs, ('architecture', 'compiler')) - - # Traverse the index and print out each package - for i, (architecture, compiler) in enumerate(sorted(index)): - if i > 0: - print - - header = "%s{%s} / %s{%s}" % (spack.spec.architecture_color, - architecture, spack.spec.compiler_color, - compiler) - tty.hline(colorize(header), char='-') - - specs = index[(architecture, compiler)] - specs.sort() - - abbreviated = [s.format(format_string, color=True) for s in specs] - if mode == 'paths': - # Print one spec per line along with prefix path - width = max(len(s) for s in abbreviated) - width += 2 - format = " %%-%ds%%s" % width - - for abbrv, spec in zip(abbreviated, specs): - if hashes: - print(gray_hash(spec, hlen), ) - print(format % (abbrv, spec.prefix)) - - elif mode == 'deps': - for spec in specs: - print(spec.tree( - format=format_string, - color=True, - indent=4, - prefix=(lambda s: gray_hash(s, hlen)) if hashes else None)) - - elif mode == 'short': - # Print columns of output if not printing flags - if not flags: - - def fmt(s): - string = "" - if hashes: - string += gray_hash(s, hlen) + ' ' - string += s.format('$-%s$@%s' % (nfmt, vfmt), color=True) - - return string - - colify(fmt(s) for s in specs) - # Print one entry per line if including flags - else: - for spec in specs: - # Print the hash if necessary - hsh = gray_hash(spec, hlen) + ' ' if hashes else '' - print(hsh + spec.format(format_string, color=True) + '\n') - - else: - raise ValueError( - "Invalid mode for display_specs: %s. Must be one of (paths," - "deps, short)." % mode) # NOQA: ignore=E501 - - def query_arguments(args): # Check arguments if args.explicit and args.implicit: diff --git a/lib/spack/spack/cmd/info.py b/lib/spack/spack/cmd/info.py index 64d0d20e24..498518057b 100644 --- a/lib/spack/spack/cmd/info.py +++ b/lib/spack/spack/cmd/info.py @@ -29,9 +29,11 @@ import spack.fetch_strategy as fs description = "Get detailed information on a particular package" + def padder(str_list, extra=0): """Return a function to pad elements of a list.""" length = max(len(str(s)) for s in str_list) + extra + def pad(string): string = str(string) padding = max(0, length - len(string)) @@ -40,7 +42,8 @@ def padder(str_list, extra=0): def setup_parser(subparser): - subparser.add_argument('name', metavar="PACKAGE", help="Name of package to get info for.") + subparser.add_argument( + 'name', metavar="PACKAGE", help="Name of package to get info for.") def print_text_info(pkg): @@ -81,12 +84,14 @@ def print_text_info(pkg): print " " + fmt % (name, default, desc) - print - print "Dependencies:" - if pkg.dependencies: - colify(pkg.dependencies, indent=4) - else: - print " None" + for deptype in ('build', 'link', 'run'): + print + print "%s Dependencies:" % deptype.capitalize() + deps = pkg.dependencies_of_type(deptype) + if deps: + colify(deps, indent=4) + else: + print " None" print print "Virtual packages: " diff --git a/lib/spack/spack/cmd/mirror.py b/lib/spack/spack/cmd/mirror.py index d5f7abe212..0cf682fc4f 100644 --- a/lib/spack/spack/cmd/mirror.py +++ b/lib/spack/spack/cmd/mirror.py @@ -179,7 +179,7 @@ def mirror_create(args): new_specs = set() for spec in specs: spec.concretize() - for s in spec.traverse(): + for s in spec.traverse(deptype_query=spack.alldeps): new_specs.add(s) specs = list(new_specs) diff --git a/lib/spack/spack/cmd/module.py b/lib/spack/spack/cmd/module.py index 70da689b67..a10e36e077 100644 --- a/lib/spack/spack/cmd/module.py +++ b/lib/spack/spack/cmd/module.py @@ -23,135 +23,233 @@ # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## from __future__ import print_function + +import collections import os import shutil import sys import llnl.util.tty as tty import spack.cmd -from llnl.util.filesystem import mkdirp +import spack.cmd.common.arguments as arguments +import llnl.util.filesystem as filesystem from spack.modules import module_types -from spack.util.string import * - -description = "Manipulate modules and dotkits." - - -def setup_parser(subparser): - sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='module_command') - - sp.add_parser('refresh', help='Regenerate all module files.') - find_parser = sp.add_parser('find', help='Find module files for packages.') +description = "Manipulate module files" - find_parser.add_argument( - 'module_type', - help="Type of module to find file for. [" + - '|'.join(module_types) + "]") +# Dictionary that will be populated with the list of sub-commands +# Each sub-command must be callable and accept 3 arguments : +# - mtype : the type of the module file +# - specs : the list of specs to be processed +# - args : namespace containing the parsed command line arguments +callbacks = {} - find_parser.add_argument( - '-r', '--dependencies', action='store_true', - dest='recurse_dependencies', - help='Recursively traverse dependencies for modules to load.') - find_parser.add_argument( - '-s', '--shell', action='store_true', dest='shell', - help='Generate shell script (instead of input for module command)') +def subcommand(subparser_name): + """Registers a function in the callbacks dictionary""" + def decorator(callback): + callbacks[subparser_name] = callback + return callback + return decorator - find_parser.add_argument( - '-p', '--prefix', dest='prefix', - help='Prepend to module names when issuing module load commands') - find_parser.add_argument( - 'spec', nargs='+', - help='spec to find a module file for.') - - -def module_find(mtype, flags, spec_array): - """Look at all installed packages and see if the spec provided - matches any. If it does, check whether there is a module file - of type <mtype> there, and print out the name that the user - should type to use that package's module. - prefix: - Prepend this to module names when issuing "module load" commands. - Some systems seem to need it. +def setup_parser(subparser): + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='subparser_name') + + # spack module refresh + refresh_parser = sp.add_parser('refresh', help='Regenerate module files') + refresh_parser.add_argument( + '--delete-tree', + help='Delete the module file tree before refresh', + action='store_true' + ) + arguments.add_common_arguments( + refresh_parser, ['constraint', 'module_type', 'yes_to_all'] + ) + + # spack module find + find_parser = sp.add_parser('find', help='Find module files for packages') + arguments.add_common_arguments(find_parser, ['constraint', 'module_type']) + + # spack module rm + rm_parser = sp.add_parser('rm', help='Remove module files') + arguments.add_common_arguments( + rm_parser, ['constraint', 'module_type', 'yes_to_all'] + ) + + # spack module loads + loads_parser = sp.add_parser( + 'loads', + help='Prompt the list of modules associated with a constraint' + ) + loads_parser.add_argument( + '--input-only', action='store_false', dest='shell', + help='Generate input for module command (instead of a shell script)' + ) + loads_parser.add_argument( + '-p', '--prefix', dest='prefix', default='', + help='Prepend to module names when issuing module load commands' + ) + arguments.add_common_arguments( + loads_parser, ['constraint', 'module_type', 'recurse_dependencies'] + ) + + +class MultipleMatches(Exception): + pass + + +class NoMatch(Exception): + pass + + +@subcommand('loads') +def loads(mtype, specs, args): + """Prompt the list of modules associated with a list of specs""" + # Get a comprehensive list of specs + if args.recurse_dependencies: + specs_from_user_constraint = specs[:] + specs = [] + # FIXME : during module file creation nodes seem to be visited + # FIXME : multiple times even if cover='nodes' is given. This + # FIXME : work around permits to get a unique list of spec anyhow. + # FIXME : (same problem as in spack/modules.py) + seen = set() + seen_add = seen.add + for spec in specs_from_user_constraint: + specs.extend( + [item for item in spec.traverse(order='post', cover='nodes') if not (item in seen or seen_add(item))] # NOQA: ignore=E501 + ) + + module_cls = module_types[mtype] + modules = [(spec, module_cls(spec).use_name) + for spec in specs if os.path.exists(module_cls(spec).file_name)] + + module_commands = { + 'tcl': 'module load ', + 'dotkit': 'dotkit use ' + } + + d = { + 'command': '' if not args.shell else module_commands[mtype], + 'prefix': args.prefix + } + + prompt_template = '{comment}{command}{prefix}{name}' + for spec, mod in modules: + d['comment'] = '' if not args.shell else '# {0}\n'.format( + spec.format()) + d['name'] = mod + print(prompt_template.format(**d)) + + +@subcommand('find') +def find(mtype, specs, args): """ - if mtype not in module_types: - tty.die("Invalid module type: '%s'. Options are %s" % - (mtype, comma_or(module_types))) - - # -------------------------------------- - def _find_modules(spec, modules_list): - """Finds all modules and sub-modules for a spec""" - if str(spec.version) == 'system': - # No Spack module for system-installed packages - return - - if flags.recurse_dependencies: - for dep in spec.dependencies.values(): - _find_modules(dep, modules_list) - - mod = module_types[mtype](spec) - if not os.path.isfile(mod.file_name): - tty.die("No %s module is installed for %s" % (mtype, spec)) - modules_list.append((spec, mod)) - - - # -------------------------------------- - raw_specs = spack.cmd.parse_specs(spec_array) - modules = set() # Modules we will load - seen = set() - for raw_spec in raw_specs: - - # ----------- Make sure the spec only resolves to ONE thing - specs = spack.installed_db.query(raw_spec) - if len(specs) == 0: - tty.die("No installed packages match spec %s" % raw_spec) - - if len(specs) > 1: - tty.error("Multiple matches for spec %s. Choose one:" % raw_spec) - for s in specs: - sys.stderr.write(s.tree(color=True)) - sys.exit(1) - spec = specs[0] - - # ----------- Chase down modules for it and all its dependencies - modules_dups = list() - _find_modules(spec, modules_dups) - - # Remove duplicates while keeping order - modules_unique = list() - for spec,mod in modules_dups: - if mod.use_name not in seen: - modules_unique.append((spec,mod)) - seen.add(mod.use_name) - - # Output... - if flags.shell: - module_cmd = {'tcl': 'module load', 'dotkit': 'dotkit use'}[mtype] - for spec,mod in modules_unique: - if flags.shell: - print('# %s' % spec.format()) - print('%s %s%s' % (module_cmd, flags.prefix, mod.use_name)) - else: - print(mod.use_name) - -def module_refresh(): - """Regenerate all module files for installed packages known to - spack (some packages may no longer exist).""" - specs = [s for s in spack.installed_db.query(installed=True, known=True)] - - for name, cls in module_types.items(): - tty.msg("Regenerating %s module files." % name) - if os.path.isdir(cls.path): - shutil.rmtree(cls.path, ignore_errors=False) - mkdirp(cls.path) - for spec in specs: - cls(spec).write() + Look at all installed packages and see if the spec provided + matches any. If it does, check whether there is a module file + of type <mtype> there, and print out the name that the user + should type to use that package's module. + """ + if len(specs) == 0: + raise NoMatch() + + if len(specs) > 1: + raise MultipleMatches() + + spec = specs.pop() + mod = module_types[mtype](spec) + if not os.path.isfile(mod.file_name): + tty.die("No %s module is installed for %s" % (mtype, spec)) + print(mod.use_name) + + +@subcommand('rm') +def rm(mtype, specs, args): + """Deletes module files associated with items in specs""" + module_cls = module_types[mtype] + specs_with_modules = [ + spec for spec in specs if os.path.exists(module_cls(spec).file_name)] + modules = [module_cls(spec) for spec in specs_with_modules] + + if not modules: + tty.msg('No module file matches your query') + raise SystemExit(1) + + # Ask for confirmation + if not args.yes_to_all: + tty.msg('You are about to remove {0} module files the following specs:\n'.format(mtype)) # NOQA: ignore=E501 + spack.cmd.display_specs(specs_with_modules, long=True) + print('') + spack.cmd.ask_for_confirmation('Do you want to proceed ? ') + + # Remove the module files + for s in modules: + s.remove() + + +@subcommand('refresh') +def refresh(mtype, specs, args): + """Regenerate module files for item in specs""" + # Prompt a message to the user about what is going to change + if not specs: + tty.msg('No package matches your query') + return + + if not args.yes_to_all: + tty.msg('You are about to regenerate {name} module files for the following specs:\n'.format(name=mtype)) # NOQA: ignore=E501 + spack.cmd.display_specs(specs, long=True) + print('') + spack.cmd.ask_for_confirmation('Do you want to proceed ? ') + + cls = module_types[mtype] + + # Detect name clashes + writers = [cls(spec) for spec in specs] + file2writer = collections.defaultdict(list) + for item in writers: + file2writer[item.file_name].append(item) + + if len(file2writer) != len(writers): + message = 'Name clashes detected in module files:\n' + for filename, writer_list in file2writer.items(): + if len(writer_list) > 1: + message += '\nfile : {0}\n'.format(filename) + for x in writer_list: + message += 'spec : {0}\n'.format(x.spec.format(color=True)) + tty.error(message) + tty.error('Operation aborted') + raise SystemExit(1) + + # Proceed regenerating module files + tty.msg('Regenerating {name} module files'.format(name=mtype)) + if os.path.isdir(cls.path) and args.delete_tree: + shutil.rmtree(cls.path, ignore_errors=False) + filesystem.mkdirp(cls.path) + for x in writers: + x.write(overwrite=True) def module(parser, args): - if args.module_command == 'refresh': - module_refresh() - - elif args.module_command == 'find': - module_find(args.module_type, args, args.spec) + # Qualifiers to be used when querying the db for specs + constraint_qualifiers = { + 'refresh': { + 'installed': True, + 'known': True + }, + } + arguments.ConstraintAction.qualifiers.update(constraint_qualifiers) + + module_type = args.module_type + constraint = args.constraint + try: + callbacks[args.subparser_name](module_type, args.specs, args) + except MultipleMatches: + message = 'the constraint \'{query}\' matches multiple packages, and this is not allowed in this context' # NOQA: ignore=E501 + tty.error(message.format(query=constraint)) + for s in args.specs: + sys.stderr.write(s.format(color=True) + '\n') + raise SystemExit(1) + except NoMatch: + message = 'the constraint \'{query}\' match no package, and this is not allowed in this context' # NOQA: ignore=E501 + tty.die(message.format(query=constraint)) diff --git a/lib/spack/spack/cmd/package-list.py b/lib/spack/spack/cmd/package-list.py index 6c5c4ae8c6..a27502d30e 100644 --- a/lib/spack/spack/cmd/package-list.py +++ b/lib/spack/spack/cmd/package-list.py @@ -22,10 +22,8 @@ # License along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## -import re import cgi from StringIO import StringIO -import llnl.util.tty as tty from llnl.util.tty.colify import * import spack @@ -34,21 +32,22 @@ description = "Print a list of all packages in reStructuredText." def github_url(pkg): """Link to a package file on github.""" - return ("https://github.com/llnl/spack/blob/master/var/spack/packages/%s/package.py" % - pkg.name) + url = "https://github.com/llnl/spack/blob/master/var/spack/packages/%s/package.py" # NOQA: ignore=E501 + return (url % pkg.name) def rst_table(elts): """Print out a RST-style table.""" cols = StringIO() ncol, widths = colify(elts, output=cols, tty=True) - header = " ".join("=" * (w-1) for w in widths) + header = " ".join("=" * (w - 1) for w in widths) return "%s\n%s%s" % (header, cols.getvalue(), header) def print_rst_package_list(): """Print out information on all packages in restructured text.""" - pkgs = sorted(spack.repo.all_packages(), key=lambda s:s.name.lower()) + pkgs = sorted(spack.repo.all_packages(), key=lambda s: s.name.lower()) + pkg_names = [p.name for p in pkgs] print ".. _package-list:" print @@ -62,7 +61,7 @@ def print_rst_package_list(): print "Spack currently has %d mainline packages:" % len(pkgs) print - print rst_table("`%s`_" % p.name for p in pkgs) + print rst_table("`%s`_" % p for p in pkg_names) print print "-----" @@ -79,12 +78,17 @@ def print_rst_package_list(): print if pkg.versions: print "Versions:" - print " " + ", ".join(str(v) for v in reversed(sorted(pkg.versions))) - if pkg.dependencies: - print "Dependencies" - print " " + ", ".join("`%s`_" % d if d != "mpi" else d - for d in pkg.dependencies) - print + print " " + ", ".join(str(v) for v in + reversed(sorted(pkg.versions))) + + for deptype in spack.alldeps: + deps = pkg.dependencies_of_type(deptype) + if deps: + print "%s Dependencies" % deptype.capitalize() + print " " + ", ".join("%s_" % d if d in pkg_names + else d for d in deps) + print + print "Description:" print pkg.format_doc(indent=2) print diff --git a/lib/spack/spack/cmd/setup.py b/lib/spack/spack/cmd/setup.py index 02e9bfd281..04f3d663df 100644 --- a/lib/spack/spack/cmd/setup.py +++ b/lib/spack/spack/cmd/setup.py @@ -44,7 +44,7 @@ def setup_parser(subparser): help="Display verbose build output while installing.") subparser.add_argument( 'spec', nargs=argparse.REMAINDER, - help="specs to use for install. Must contain package AND verison.") + help="specs to use for install. Must contain package AND version.") def setup(self, args): diff --git a/lib/spack/spack/cmd/test-install.py b/lib/spack/spack/cmd/test-install.py index 45592a7dda..14c06d136d 100644 --- a/lib/spack/spack/cmd/test-install.py +++ b/lib/spack/spack/cmd/test-install.py @@ -133,7 +133,12 @@ def fetch_log(path): def failed_dependencies(spec): - return set(item for item in spec.dependencies.itervalues() if not spack.repo.get(item).installed) + def get_deps(deptype): + return set(item for item in spec.dependencies(deptype) + if not spack.repo.get(item).installed) + link_deps = get_deps('link') + run_deps = get_deps('run') + return link_deps.union(run_deps) def get_top_spec_or_die(args): diff --git a/lib/spack/spack/cmd/uninstall.py b/lib/spack/spack/cmd/uninstall.py index a6f08d09ed..a17b7c685c 100644 --- a/lib/spack/spack/cmd/uninstall.py +++ b/lib/spack/spack/cmd/uninstall.py @@ -30,7 +30,6 @@ import llnl.util.tty as tty import spack import spack.cmd import spack.repository -from spack.cmd.find import display_specs description = "Remove an installed package" @@ -43,21 +42,10 @@ error_message = """You can either: display_args = { 'long': True, 'show_flags': True, - 'variants':True + 'variants': True } -def ask_for_confirmation(message): - while True: - tty.msg(message + '[y/n]') - choice = raw_input().lower() - if choice == 'y': - break - elif choice == 'n': - raise SystemExit('Operation aborted') - tty.warn('Please reply either "y" or "n"') - - def setup_parser(subparser): subparser.add_argument( '-f', '--force', action='store_true', dest='force', @@ -65,32 +53,37 @@ def setup_parser(subparser): subparser.add_argument( '-a', '--all', action='store_true', dest='all', help="USE CAREFULLY. Remove ALL installed packages that match each " + - "supplied spec. i.e., if you say uninstall libelf, ALL versions of " + - "libelf are uninstalled. This is both useful and dangerous, like rm -r.") + "supplied spec. i.e., if you say uninstall libelf, ALL versions of " + # NOQA: ignore=E501 + "libelf are uninstalled. This is both useful and dangerous, like rm -r.") # NOQA: ignore=E501 subparser.add_argument( '-d', '--dependents', action='store_true', dest='dependents', - help='Also uninstall any packages that depend on the ones given via command line.' + help='Also uninstall any packages that depend on the ones given via command line.' # NOQA: ignore=E501 ) subparser.add_argument( '-y', '--yes-to-all', action='store_true', dest='yes_to_all', - help='Assume "yes" is the answer to every confirmation asked to the user.' + help='Assume "yes" is the answer to every confirmation asked to the user.' # NOQA: ignore=E501 ) - subparser.add_argument('packages', nargs=argparse.REMAINDER, help="specs of packages to uninstall") + subparser.add_argument( + 'packages', + nargs=argparse.REMAINDER, + help="specs of packages to uninstall" + ) def concretize_specs(specs, allow_multiple_matches=False, force=False): - """ - Returns a list of specs matching the non necessarily concretized specs given from cli + """Returns a list of specs matching the non necessarily + concretized specs given from cli Args: specs: list of specs to be matched against installed packages - allow_multiple_matches : boolean (if True multiple matches for each item in specs are admitted) + allow_multiple_matches : if True multiple matches are admitted Return: list of specs """ - specs_from_cli = [] # List of specs that match expressions given via command line + # List of specs that match expressions given via command line + specs_from_cli = [] has_errors = False for spec in specs: matching = spack.installed_db.query(spec) @@ -99,7 +92,7 @@ def concretize_specs(specs, allow_multiple_matches=False, force=False): if not allow_multiple_matches and len(matching) > 1: tty.error("%s matches multiple packages:" % spec) print() - display_specs(matching, **display_args) + spack.cmd.display_specs(matching, **display_args) print() has_errors = True @@ -116,8 +109,8 @@ def concretize_specs(specs, allow_multiple_matches=False, force=False): def installed_dependents(specs): - """ - Returns a dictionary that maps a spec with a list of its installed dependents + """Returns a dictionary that maps a spec with a list of its + installed dependents Args: specs: list of specs to be checked for dependents @@ -147,7 +140,7 @@ def do_uninstall(specs, force): try: # should work if package is known to spack packages.append(item.package) - except spack.repository.UnknownPackageError as e: + except spack.repository.UnknownPackageError: # The package.py file has gone away -- but still # want to uninstall. spack.Package(item).do_uninstall(force=True) @@ -169,17 +162,20 @@ def uninstall(parser, args): with spack.installed_db.write_transaction(): specs = spack.cmd.parse_specs(args.packages) # Gets the list of installed specs that match the ones give via cli - uninstall_list = concretize_specs(specs, args.all, args.force) # takes care of '-a' is given in the cli - dependent_list = installed_dependents(uninstall_list) # takes care of '-d' + # takes care of '-a' is given in the cli + uninstall_list = concretize_specs(specs, args.all, args.force) + dependent_list = installed_dependents( + uninstall_list) # takes care of '-d' # Process dependent_list and update uninstall_list has_error = False if dependent_list and not args.dependents and not args.force: for spec, lst in dependent_list.items(): - tty.error("Will not uninstall %s" % spec.format("$_$@$%@$#", color=True)) + tty.error("Will not uninstall %s" % + spec.format("$_$@$%@$#", color=True)) print('') print("The following packages depend on it:") - display_specs(lst, **display_args) + spack.cmd.display_specs(lst, **display_args) print('') has_error = True elif args.dependents: @@ -188,14 +184,14 @@ def uninstall(parser, args): uninstall_list = list(set(uninstall_list)) if has_error: - tty.die('You can use spack uninstall --dependents to uninstall these dependencies as well') + tty.die('You can use spack uninstall --dependents to uninstall these dependencies as well') # NOQA: ignore=E501 if not args.yes_to_all: tty.msg("The following packages will be uninstalled : ") print('') - display_specs(uninstall_list, **display_args) + spack.cmd.display_specs(uninstall_list, **display_args) print('') - ask_for_confirmation('Do you want to proceed ? ') + spack.cmd.ask_for_confirmation('Do you want to proceed ? ') # Uninstall everything on the list do_uninstall(uninstall_list, args.force) diff --git a/lib/spack/spack/concretize.py b/lib/spack/spack/concretize.py index d9992a5680..5180f3cf04 100644 --- a/lib/spack/spack/concretize.py +++ b/lib/spack/spack/concretize.py @@ -40,12 +40,12 @@ import spack.architecture import spack.error from spack.version import * from functools import partial -from spec import DependencyMap from itertools import chain from spack.config import * class DefaultConcretizer(object): + """This class doesn't have any state, it just provides some methods for concretization. You can subclass it to override just some of the default concretization strategies, or you can override all of them. @@ -61,14 +61,17 @@ class DefaultConcretizer(object): if not providers: raise UnsatisfiableProviderSpecError(providers[0], spec) spec_w_preferred_providers = find_spec( - spec, lambda(x): spack.pkgsort.spec_has_preferred_provider(x.name, spec.name)) + spec, lambda x: spack.pkgsort.spec_has_preferred_provider(x.name, spec.name)) # NOQA: ignore=E501 if not spec_w_preferred_providers: spec_w_preferred_providers = spec - provider_cmp = partial(spack.pkgsort.provider_compare, spec_w_preferred_providers.name, spec.name) + provider_cmp = partial(spack.pkgsort.provider_compare, + spec_w_preferred_providers.name, + spec.name) candidates = sorted(providers, cmp=provider_cmp) - # For each candidate package, if it has externals, add those to the usable list. - # if it's not buildable, then *only* add the externals. + # For each candidate package, if it has externals, add those + # to the usable list. if it's not buildable, then *only* add + # the externals. usable = [] for cspec in candidates: if is_spec_buildable(cspec): @@ -85,7 +88,7 @@ class DefaultConcretizer(object): def cmp_externals(a, b): if a.name != b.name and (not a.external or a.external_module and - not b.external and b.external_module): + not b.external and b.external_module): # We're choosing between different providers, so # maintain order from provider sort return candidates.index(a) - candidates.index(b) @@ -103,7 +106,7 @@ class DefaultConcretizer(object): usable.sort(cmp=cmp_externals) return usable - + # XXX(deptypes): Look here. def choose_virtual_or_external(self, spec): """Given a list of candidate virtual and external packages, try to find one that is most ABI compatible. @@ -114,26 +117,26 @@ class DefaultConcretizer(object): # Find the nearest spec in the dag that has a compiler. We'll # use that spec to calibrate compiler compatibility. - abi_exemplar = find_spec(spec, lambda(x): x.compiler) + abi_exemplar = find_spec(spec, lambda x: x.compiler) if not abi_exemplar: abi_exemplar = spec.root # Make a list including ABI compatibility of specs with the exemplar. strict = [spack.abi.compatible(c, abi_exemplar) for c in candidates] - loose = [spack.abi.compatible(c, abi_exemplar, loose=True) for c in candidates] + loose = [spack.abi.compatible(c, abi_exemplar, loose=True) + for c in candidates] keys = zip(strict, loose, candidates) # Sort candidates from most to least compatibility. # Note: # 1. We reverse because True > False. # 2. Sort is stable, so c's keep their order. - keys.sort(key=lambda k:k[:2], reverse=True) + keys.sort(key=lambda k: k[:2], reverse=True) # Pull the candidates back out and return them in order - candidates = [c for s,l,c in keys] + candidates = [c for s, l, c in keys] return candidates - def concretize_version(self, spec): """If the spec is already concrete, return. Otherwise take the preferred version from spackconfig, and default to the package's @@ -167,7 +170,8 @@ class DefaultConcretizer(object): if valid_versions: # Disregard @develop and take the next valid version - if ver(valid_versions[0]) == ver('develop') and len(valid_versions) > 1: + if ver(valid_versions[0]) == ver('develop') and \ + len(valid_versions) > 1: spec.versions = ver([valid_versions[1]]) else: spec.versions = ver([valid_versions[0]]) @@ -193,40 +197,45 @@ class DefaultConcretizer(object): return True # Things changed def _concretize_operating_system(self, spec): - platform = spec.architecture.platform if spec.architecture.platform_os is not None and isinstance( - spec.architecture.platform_os,spack.architecture.OperatingSystem): + spec.architecture.platform_os, + spack.architecture.OperatingSystem): return False if spec.root.architecture and spec.root.architecture.platform_os: - if isinstance(spec.root.architecture.platform_os,spack.architecture.OperatingSystem): - spec.architecture.platform_os = spec.root.architecture.platform_os + if isinstance(spec.root.architecture.platform_os, + spack.architecture.OperatingSystem): + spec.architecture.platform_os = \ + spec.root.architecture.platform_os else: - spec.architecture.platform_os = spec.architecture.platform.operating_system('default_os') - return True #changed + spec.architecture.platform_os = \ + spec.architecture.platform.operating_system('default_os') + return True # changed def _concretize_target(self, spec): - platform = spec.architecture.platform if spec.architecture.target is not None and isinstance( spec.architecture.target, spack.architecture.Target): return False if spec.root.architecture and spec.root.architecture.target: - if isinstance(spec.root.architecture.target,spack.architecture.Target): + if isinstance(spec.root.architecture.target, + spack.architecture.Target): spec.architecture.target = spec.root.architecture.target else: - spec.architecture.target = spec.architecture.platform.target('default_target') - return True #changed + spec.architecture.target = spec.architecture.platform.target( + 'default_target') + return True # changed def _concretize_platform(self, spec): if spec.architecture.platform is not None and isinstance( spec.architecture.platform, spack.architecture.Platform): return False if spec.root.architecture and spec.root.architecture.platform: - if isinstance(spec.root.architecture.platform,spack.architecture.Platform): + if isinstance(spec.root.architecture.platform, + spack.architecture.Platform): spec.architecture.platform = spec.root.architecture.platform else: spec.architecture.platform = spack.architecture.platform() - return True #changed? + return True # changed? def concretize_architecture(self, spec): """If the spec is empty provide the defaults of the platform. If the @@ -245,25 +254,29 @@ class DefaultConcretizer(object): return True # Concretize the operating_system and target based of the spec - ret = any((self._concretize_platform(spec), - self._concretize_operating_system(spec), - self._concretize_target(spec))) + ret = any((self._concretize_platform(spec), + self._concretize_operating_system(spec), + self._concretize_target(spec))) return ret - - def concretize_variants(self, spec): """If the spec already has variants filled in, return. Otherwise, add - the default variants from the package specification. + the user preferences from packages.yaml or the default variants from + the package specification. """ changed = False + preferred_variants = spack.pkgsort.spec_preferred_variants( + spec.package_class.name) for name, variant in spec.package_class.variants.items(): if name not in spec.variants: - spec.variants[name] = spack.spec.VariantSpec(name, variant.default) changed = True + if name in preferred_variants: + spec.variants[name] = preferred_variants.get(name) + else: + spec.variants[name] = \ + spack.spec.VariantSpec(name, variant.default) return changed - def concretize_compiler(self, spec): """If the spec already has a compiler, we're done. If not, then take the compiler used for the nearest ancestor with a compiler @@ -278,30 +291,32 @@ class DefaultConcretizer(object): """ # Pass on concretizing the compiler if the target is not yet determined if not spec.architecture.platform_os: - #Although this usually means changed, this means awaiting other changes + # Although this usually means changed, this means awaiting other + # changes return True # Only use a matching compiler if it is of the proper style - # Takes advantage of the proper logic already existing in compiler_for_spec - # Should think whether this can be more efficient + # Takes advantage of the proper logic already existing in + # compiler_for_spec Should think whether this can be more + # efficient def _proper_compiler_style(cspec, arch): platform = arch.platform compilers = spack.compilers.compilers_for_spec(cspec, platform=platform) return filter(lambda c: c.operating_system == - arch.platform_os, compilers) - #return compilers - + arch.platform_os, compilers) + # return compilers all_compilers = spack.compilers.all_compilers() if (spec.compiler and spec.compiler.concrete and - spec.compiler in all_compilers): + spec.compiler in all_compilers): return False - #Find the another spec that has a compiler, or the root if none do - other_spec = spec if spec.compiler else find_spec(spec, lambda(x) : x.compiler) + # Find the another spec that has a compiler, or the root if none do + other_spec = spec if spec.compiler else find_spec( + spec, lambda x: x.compiler) if not other_spec: other_spec = spec.root @@ -313,9 +328,12 @@ class DefaultConcretizer(object): spec.compiler = other_compiler.copy() return True - # Filter the compilers into a sorted list based on the compiler_order from spackconfig - compiler_list = all_compilers if not other_compiler else spack.compilers.find(other_compiler) - cmp_compilers = partial(spack.pkgsort.compiler_compare, other_spec.name) + # Filter the compilers into a sorted list based on the compiler_order + # from spackconfig + compiler_list = all_compilers if not other_compiler else \ + spack.compilers.find(other_compiler) + cmp_compilers = partial( + spack.pkgsort.compiler_compare, other_spec.name) matches = sorted(compiler_list, cmp=cmp_compilers) if not matches: raise UnavailableCompilerVersionError(other_compiler) @@ -330,7 +348,6 @@ class DefaultConcretizer(object): assert(spec.compiler.concrete) return True # things changed. - def concretize_compiler_flags(self, spec): """ The compiler flags are updated to match those of the spec whose @@ -338,54 +355,66 @@ class DefaultConcretizer(object): Default specs set at the compiler level will still be added later. """ - if not spec.architecture.platform_os: - #Although this usually means changed, this means awaiting other changes + # Although this usually means changed, this means awaiting other + # changes return True ret = False for flag in spack.spec.FlagMap.valid_compiler_flags(): try: nearest = next(p for p in spec.traverse(direction='parents') - if ((p.compiler == spec.compiler and p is not spec) - and flag in p.compiler_flags)) - if not flag in spec.compiler_flags or \ - not (sorted(spec.compiler_flags[flag]) >= sorted(nearest.compiler_flags[flag])): + if ((p.compiler == spec.compiler and + p is not spec) and + flag in p.compiler_flags)) + if flag not in spec.compiler_flags or \ + not (sorted(spec.compiler_flags[flag]) >= + sorted(nearest.compiler_flags[flag])): if flag in spec.compiler_flags: - spec.compiler_flags[flag] = list(set(spec.compiler_flags[flag]) | - set(nearest.compiler_flags[flag])) + spec.compiler_flags[flag] = list( + set(spec.compiler_flags[flag]) | + set(nearest.compiler_flags[flag])) else: - spec.compiler_flags[flag] = nearest.compiler_flags[flag] + spec.compiler_flags[ + flag] = nearest.compiler_flags[flag] ret = True except StopIteration: - if (flag in spec.root.compiler_flags and ((not flag in spec.compiler_flags) or - sorted(spec.compiler_flags[flag]) != sorted(spec.root.compiler_flags[flag]))): + if (flag in spec.root.compiler_flags and + ((flag not in spec.compiler_flags) or + sorted(spec.compiler_flags[flag]) != + sorted(spec.root.compiler_flags[flag]))): if flag in spec.compiler_flags: - spec.compiler_flags[flag] = list(set(spec.compiler_flags[flag]) | - set(spec.root.compiler_flags[flag])) + spec.compiler_flags[flag] = list( + set(spec.compiler_flags[flag]) | + set(spec.root.compiler_flags[flag])) else: - spec.compiler_flags[flag] = spec.root.compiler_flags[flag] + spec.compiler_flags[ + flag] = spec.root.compiler_flags[flag] ret = True else: - if not flag in spec.compiler_flags: + if flag not in spec.compiler_flags: spec.compiler_flags[flag] = [] # Include the compiler flag defaults from the config files # This ensures that spack will detect conflicts that stem from a change # in default compiler flags. - compiler = spack.compilers.compiler_for_spec(spec.compiler, spec.architecture) + compiler = spack.compilers.compiler_for_spec( + spec.compiler, spec.architecture) for flag in compiler.flags: if flag not in spec.compiler_flags: spec.compiler_flags[flag] = compiler.flags[flag] if compiler.flags[flag] != []: ret = True else: - if ((sorted(spec.compiler_flags[flag]) != sorted(compiler.flags[flag])) and - (not set(spec.compiler_flags[flag]) >= set(compiler.flags[flag]))): + if ((sorted(spec.compiler_flags[flag]) != + sorted(compiler.flags[flag])) and + (not set(spec.compiler_flags[flag]) >= + set(compiler.flags[flag]))): ret = True - spec.compiler_flags[flag] = list(set(spec.compiler_flags[flag]) | - set(compiler.flags[flag])) + spec.compiler_flags[flag] = list( + set(spec.compiler_flags[flag]) | + set(compiler.flags[flag])) return ret @@ -394,8 +423,10 @@ def find_spec(spec, condition): """Searches the dag from spec in an intelligent order and looks for a spec that matches a condition""" # First search parents, then search children - dagiter = chain(spec.traverse(direction='parents', root=False), - spec.traverse(direction='children', root=False)) + deptype = ('build', 'link') + dagiter = chain( + spec.traverse(direction='parents', deptype=deptype, root=False), + spec.traverse(direction='children', deptype=deptype, root=False)) visited = set() for relative in dagiter: if condition(relative): @@ -403,9 +434,11 @@ def find_spec(spec, condition): visited.add(id(relative)) # Then search all other relatives in the DAG *except* spec - for relative in spec.root.traverse(): - if relative is spec: continue - if id(relative) in visited: continue + for relative in spec.root.traverse(deptypes=spack.alldeps): + if relative is spec: + continue + if id(relative) in visited: + continue if condition(relative): return relative @@ -452,8 +485,10 @@ def cmp_specs(lhs, rhs): class UnavailableCompilerVersionError(spack.error.SpackError): + """Raised when there is no available compiler that satisfies a compiler spec.""" + def __init__(self, compiler_spec): super(UnavailableCompilerVersionError, self).__init__( "No available compiler version matches '%s'" % compiler_spec, @@ -461,16 +496,20 @@ class UnavailableCompilerVersionError(spack.error.SpackError): class NoValidVersionError(spack.error.SpackError): + """Raised when there is no way to have a concrete version for a particular spec.""" + def __init__(self, spec): super(NoValidVersionError, self).__init__( - "There are no valid versions for %s that match '%s'" % (spec.name, spec.versions)) + "There are no valid versions for %s that match '%s'" % (spec.name, spec.versions)) # NOQA: ignore=E501 class NoBuildError(spack.error.SpackError): + """Raised when a package is configured with the buildable option False, but no satisfactory external versions can be found""" + def __init__(self, spec): super(NoBuildError, self).__init__( - "The spec '%s' is configured as not buildable, and no matching external installs were found" % spec.name) + "The spec '%s' is configured as not buildable,and no matching external installs were found" % spec.name) # NOQA: ignore=E501 diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py index 84179e1469..e2e7dbc0ee 100644 --- a/lib/spack/spack/config.py +++ b/lib/spack/spack/config.py @@ -257,7 +257,13 @@ section_schemas = { 'paths': { 'type' : 'object', 'default' : {}, - } + }, + 'variants': { + 'oneOf' : [ + { 'type' : 'string' }, + { 'type' : 'array', + 'items' : { 'type' : 'string' } }, + ], }, },},},},},}, 'modules': { @@ -328,6 +334,11 @@ section_schemas = { 'anyOf': [ { 'properties': { + 'hash_length': { + 'type': 'integer', + 'minimum': 0, + 'default': 7 + }, 'whitelist': {'$ref': '#/definitions/array_of_strings'}, 'blacklist': {'$ref': '#/definitions/array_of_strings'}, 'naming_scheme': { @@ -492,8 +503,15 @@ class ConfigScope(object): """Empty cached config information.""" self.sections = {} +"""Default configuration scope is the lowest-level scope. These are + versioned with Spack and can be overridden by sites or users.""" +ConfigScope('defaults', os.path.join(spack.etc_path, 'spack', 'defaults')) + +"""Site configuration is per spack instance, for sites or projects. + No site-level configs should be checked into spack by default.""" +ConfigScope('site', os.path.join(spack.etc_path, 'spack')) -ConfigScope('site', os.path.join(spack.etc_path, 'spack')), +"""User configuration can override both spack defaults and site config.""" ConfigScope('user', os.path.expanduser('~/.spack')) diff --git a/lib/spack/spack/database.py b/lib/spack/spack/database.py index a4bbff3d5a..317b0d5784 100644 --- a/lib/spack/spack/database.py +++ b/lib/spack/spack/database.py @@ -60,7 +60,7 @@ from spack.repository import UnknownPackageError _db_dirname = '.spack-db' # DB version. This is stuck in the DB file to track changes in format. -_db_version = Version('0.9.1') +_db_version = Version('0.9.2') # Default timeout for spack database locks is 5 min. _db_lock_timeout = 60 @@ -215,9 +215,10 @@ class Database(object): # Add dependencies from other records in the install DB to # form a full spec. if 'dependencies' in spec_dict[spec.name]: - for dep_hash in spec_dict[spec.name]['dependencies'].values(): - child = self._read_spec_from_yaml(dep_hash, installs, hash_key) - spec._add_dependency(child) + yaml_deps = spec_dict[spec.name]['dependencies'] + for dname, dhash, dtypes in Spec.read_yaml_dep_specs(yaml_deps): + child = self._read_spec_from_yaml(dhash, installs, hash_key) + spec._add_dependency(child, dtypes) # Specs from the database need to be marked concrete because # they represent actual installations. @@ -334,7 +335,10 @@ class Database(object): counts = {} for key, rec in self._data.items(): counts.setdefault(key, 0) - for dep in rec.spec.dependencies.values(): + # XXX(deptype): This checks all dependencies, but build + # dependencies might be able to be dropped in the + # future. + for dep in rec.spec.dependencies(): dep_key = dep.dag_hash() counts.setdefault(dep_key, 0) counts[dep_key] += 1 @@ -406,7 +410,7 @@ class Database(object): else: self._data[key] = InstallRecord(spec, path, True, explicit=explicit) - for dep in spec.dependencies.values(): + for dep in spec.dependencies(('link', 'run')): self._increment_ref_count(dep, directory_layout) def _increment_ref_count(self, spec, directory_layout=None): @@ -421,7 +425,7 @@ class Database(object): self._data[key] = InstallRecord(spec.copy(), path, installed) - for dep in spec.dependencies.values(): + for dep in spec.dependencies('link'): self._increment_ref_count(dep) self._data[key].ref_count += 1 @@ -466,7 +470,7 @@ class Database(object): if rec.ref_count == 0 and not rec.installed: del self._data[key] - for dep in spec.dependencies.values(): + for dep in spec.dependencies('link'): self._decrement_ref_count(dep) def _remove(self, spec): @@ -480,7 +484,7 @@ class Database(object): return rec.spec del self._data[key] - for dep in rec.spec.dependencies.values(): + for dep in rec.spec.dependencies('link'): self._decrement_ref_count(dep) # Returns the concrete spec so we know it in the case where a @@ -631,13 +635,14 @@ class WriteTransaction(_Transaction): class CorruptDatabaseError(SpackError): def __init__(self, path, msg=''): super(CorruptDatabaseError, self).__init__( - "Spack database is corrupt: %s. %s." + \ - "Try running `spack reindex` to fix." % (path, msg)) + "Spack database is corrupt: %s. %s." % (path, msg), + "Try running `spack reindex` to fix.") class InvalidDatabaseVersionError(SpackError): def __init__(self, expected, found): super(InvalidDatabaseVersionError, self).__init__( - "Expected database version %s but found version %s." + \ - "Try running `spack reindex` to fix." % - (expected, found)) + "Expected database version %s but found version %s." + % (expected, found), + "`spack reindex` may fix this, or you may need a newer " + "Spack version.") diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index ca8f21dc08..e92dd6fb67 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -171,7 +171,7 @@ def version(pkg, ver, checksum=None, **kwargs): pkg.versions[Version(ver)] = kwargs -def _depends_on(pkg, spec, when=None): +def _depends_on(pkg, spec, when=None, type=None): # If when is False do nothing if when is False: return @@ -180,10 +180,29 @@ def _depends_on(pkg, spec, when=None): when = pkg.name when_spec = parse_anonymous_spec(when, pkg.name) + if type is None: + # The default deptype is build and link because the common case is to + # build against a library which then turns into a runtime dependency + # due to the linker. + # XXX(deptype): Add 'run' to this? It's an uncommon dependency type, + # but is most backwards-compatible. + type = ('build', 'link') + + if isinstance(type, str): + type = spack.spec.special_types.get(type, (type,)) + + for deptype in type: + if deptype not in spack.spec.alldeps: + raise UnknownDependencyTypeError('depends_on', pkg.name, deptype) + dep_spec = Spec(spec) if pkg.name == dep_spec.name: raise CircularReferenceError('depends_on', pkg.name) + pkg_deptypes = pkg._deptypes.setdefault(dep_spec.name, set()) + for deptype in type: + pkg_deptypes.add(deptype) + conditions = pkg.dependencies.setdefault(dep_spec.name, {}) if when_spec in conditions: conditions[when_spec].constrain(dep_spec, deps=False) @@ -191,13 +210,13 @@ def _depends_on(pkg, spec, when=None): conditions[when_spec] = dep_spec -@directive('dependencies') -def depends_on(pkg, spec, when=None): +@directive(('dependencies', '_deptypes')) +def depends_on(pkg, spec, when=None, type=None): """Creates a dict of deps with specs defining when they apply.""" - _depends_on(pkg, spec, when=when) + _depends_on(pkg, spec, when=when, type=type) -@directive(('extendees', 'dependencies')) +@directive(('extendees', 'dependencies', '_deptypes')) def extends(pkg, spec, **kwargs): """Same as depends_on, but dependency is symlinked into parent prefix. @@ -326,3 +345,13 @@ class CircularReferenceError(DirectiveError): directive, "Package '%s' cannot pass itself to %s" % (package, directive)) self.package = package + + +class UnknownDependencyTypeError(DirectiveError): + """This is raised when a dependency is of an unknown type.""" + def __init__(self, directive, package, deptype): + super(UnknownDependencyTypeError, self).__init__( + directive, + "Package '%s' cannot depend on a package via %s." % + (package, deptype)) + self.package = package diff --git a/lib/spack/spack/directory_layout.py b/lib/spack/spack/directory_layout.py index ee13e2dcbc..8150a6da2b 100644 --- a/lib/spack/spack/directory_layout.py +++ b/lib/spack/spack/directory_layout.py @@ -34,6 +34,7 @@ import yaml import llnl.util.tty as tty from llnl.util.filesystem import join_path, mkdirp +import spack from spack.spec import Spec from spack.error import SpackError @@ -223,8 +224,14 @@ class YamlDirectoryLayout(DirectoryLayout): def read_spec(self, path): """Read the contents of a file and parse them as a spec""" - with open(path) as f: - spec = Spec.from_yaml(f) + try: + with open(path) as f: + spec = Spec.from_yaml(f) + except Exception as e: + if spack.debug: + raise + raise SpecReadError( + 'Unable to read file: %s' % path, 'Cause: ' + str(e)) # Specs read from actual installations are always concrete spec._mark_concrete() @@ -285,7 +292,7 @@ class YamlDirectoryLayout(DirectoryLayout): return path if spec.dag_hash() == installed_spec.dag_hash(): - raise SpecHashCollisionError(installed_hash, spec_hash) + raise SpecHashCollisionError(spec, installed_spec) else: raise InconsistentInstallDirectoryError( 'Spec file in %s does not match hash!' % spec_file_path) @@ -431,7 +438,7 @@ class SpecHashCollisionError(DirectoryLayoutError): def __init__(self, installed_spec, new_spec): super(SpecHashCollisionError, self).__init__( 'Specs %s and %s have the same SHA-1 prefix!' - % installed_spec, new_spec) + % (installed_spec, new_spec)) class RemoveFailedError(DirectoryLayoutError): @@ -456,10 +463,12 @@ class InstallDirectoryAlreadyExistsError(DirectoryLayoutError): "Install path %s already exists!") +class SpecReadError(DirectoryLayoutError): + """Raised when directory layout can't read a spec.""" + + class InvalidExtensionSpecError(DirectoryLayoutError): """Raised when an extension file has a bad spec in it.""" - def __init__(self, message): - super(InvalidExtensionSpecError, self).__init__(message) class ExtensionAlreadyInstalledError(DirectoryLayoutError): diff --git a/lib/spack/spack/graph.py b/lib/spack/spack/graph.py index 22058d41d8..063e4647b6 100644 --- a/lib/spack/spack/graph.py +++ b/lib/spack/spack/graph.py @@ -80,12 +80,14 @@ def topological_sort(spec, **kwargs): """ reverse = kwargs.get('reverse', False) + # XXX(deptype): iterate over a certain kind of dependency. Maybe color + # edges based on the type of dependency? if not reverse: - parents = lambda s: s.dependents - children = lambda s: s.dependencies + parents = lambda s: s.dependents() + children = lambda s: s.dependencies() else: - parents = lambda s: s.dependencies - children = lambda s: s.dependents + parents = lambda s: s.dependencies() + children = lambda s: s.dependents() # Work on a copy so this is nondestructive. spec = spec.copy() diff --git a/lib/spack/spack/modules.py b/lib/spack/spack/modules.py index ce46047fa3..db8b20ae42 100644 --- a/lib/spack/spack/modules.py +++ b/lib/spack/spack/modules.py @@ -120,7 +120,7 @@ def dependencies(spec, request='all'): return [] if request == 'direct': - return [xx for _, xx in spec.dependencies.items()] + return spec.dependencies() # FIXME : during module file creation nodes seem to be visited multiple # FIXME : times even if cover='nodes' is given. This work around permits @@ -188,6 +188,8 @@ def parse_config_options(module_generator): ##### # Automatic loading loads + module_file_actions['hash_length'] = module_configuration.get( + 'hash_length', 7) module_file_actions['autoload'] = dependencies( module_generator.spec, module_file_actions.get('autoload', 'none')) # Prerequisites @@ -237,6 +239,7 @@ class EnvModule(object): formats = {} class __metaclass__(type): + def __init__(cls, name, bases, dict): type.__init__(cls, name, bases, dict) if cls.name != 'env_module' and cls.name in CONFIGURATION[ @@ -295,7 +298,9 @@ class EnvModule(object): if constraint in self.spec: suffixes.append(suffix) # Always append the hash to make the module file unique - suffixes.append(self.spec.dag_hash()) + hash_length = configuration.pop('hash_length', 7) + if hash_length != 0: + suffixes.append(self.spec.dag_hash(length=hash_length)) name = '-'.join(suffixes) return name @@ -338,7 +343,7 @@ class EnvModule(object): return False - def write(self): + def write(self, overwrite=False): """ Writes out a module file for this object. @@ -399,6 +404,15 @@ class EnvModule(object): for line in self.module_specific_content(module_configuration): module_file_content += line + # Print a warning in case I am accidentally overwriting + # a module file that is already there (name clash) + if not overwrite and os.path.exists(self.file_name): + message = 'Module file already exists : skipping creation\n' + message += 'file : {0.file_name}\n' + message += 'spec : {0.spec}' + tty.warn(message.format(self)) + return + # Dump to file with open(self.file_name, 'w') as f: f.write(module_file_content) @@ -454,9 +468,10 @@ class EnvModule(object): class Dotkit(EnvModule): name = 'dotkit' - + path = join_path(spack.share_path, 'dotkit') environment_modifications_formats = { PrependPath: 'dk_alter {name} {value}\n', + RemovePath: 'dk_unalter {name} {value}\n', SetEnv: 'dk_setenv {name} {value}\n' } @@ -466,7 +481,7 @@ class Dotkit(EnvModule): @property def file_name(self): - return join_path(spack.share_path, "dotkit", self.spec.architecture, + return join_path(self.path, self.spec.architecture, '%s.dk' % self.use_name) @property @@ -494,7 +509,7 @@ class Dotkit(EnvModule): class TclModule(EnvModule): name = 'tcl' - + path = join_path(spack.share_path, "modules") environment_modifications_formats = { PrependPath: 'prepend-path --delim "{delim}" {name} \"{value}\"\n', AppendPath: 'append-path --delim "{delim}" {name} \"{value}\"\n', @@ -514,7 +529,7 @@ class TclModule(EnvModule): @property def file_name(self): - return join_path(spack.share_path, "modules", self.spec.architecture, self.use_name) + return join_path(self.path, self.spec.architecture, self.use_name) @property def header(self): diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index 74008c4dd9..c916bfaaa2 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -37,7 +37,6 @@ import os import re import textwrap import time -import glob import string import llnl.util.tty as tty @@ -62,10 +61,10 @@ from llnl.util.tty.log import log_output from spack.stage import Stage, ResourceStage, StageComposite from spack.util.compression import allowed_archive from spack.util.environment import dump_environment -from spack.util.executable import ProcessError, Executable, which +from spack.util.executable import ProcessError, which from spack.version import * from spack import directory_layout -from urlparse import urlparse + """Allowed URL schemes for spack packages.""" _ALLOWED_URL_SCHEMES = ["http", "https", "ftp", "file", "git"] @@ -410,7 +409,6 @@ class Package(object): """Return the directory where the package.py file lives.""" return os.path.dirname(self.module.__file__) - @property def global_license_dir(self): """Returns the directory where global license files for all @@ -565,6 +563,11 @@ class Package(object): def fetcher(self, f): self._fetcher = f + def dependencies_of_type(self, *deptypes): + """Get subset of the dependencies with certain types.""" + return dict((name, conds) for name, conds in self.dependencies.items() + if any(d in self._deptypes[name] for d in deptypes)) + @property def extendee_spec(self): """ @@ -577,7 +580,7 @@ class Package(object): name = next(iter(self.extendees)) # If the extendee is in the spec's deps already, return that. - for dep in self.spec.traverse(): + for dep in self.spec.traverse(deptypes=('link', 'run')): if name == dep.name: return dep @@ -642,12 +645,13 @@ class Package(object): yield self for name in sorted(self.dependencies.keys()): - spec = self.dependencies[name] + dep_spec = self.get_dependency(name) + spec = dep_spec.spec - # currently, we do not descend into virtual dependencies, as this + # Currently, we do not descend into virtual dependencies, as this # makes doing a sensible traversal much harder. We just assume # that ANY of the virtual deps will work, which might not be true - # (due to conflicts or unsatisfiable specs). For now this is ok + # (due to conflicts or unsatisfiable specs). For now this is ok, # but we might want to reinvestigate if we start using a lot of # complicated virtual dependencies # TODO: reinvestigate this. @@ -685,7 +689,9 @@ class Package(object): for spec in spack.installed_db.query(): if self.name == spec.name: continue - for dep in spec.traverse(): + # XXX(deptype): Should build dependencies not count here? + # for dep in spec.traverse(deptype=('run')): + for dep in spec.traverse(deptype=spack.alldeps): if self.spec == dep: dependents.append(spec) return dependents @@ -696,13 +702,13 @@ class Package(object): return self.spec.prefix @property - #TODO: Change this to architecture + # TODO: Change this to architecture def compiler(self): """Get the spack.compiler.Compiler object used to build this package""" if not self.spec.concrete: raise ValueError("Can only get a compiler for a concrete package.") return spack.compilers.compiler_for_spec(self.spec.compiler, - self.spec.architecture) + self.spec.architecture) def url_version(self, version): """ @@ -758,7 +764,6 @@ class Package(object): self.stage.cache_local() - def do_stage(self, mirror_only=False): """Unpacks the fetched tarball, then changes into the expanded tarball directory.""" @@ -876,6 +881,7 @@ class Package(object): return resource_stage_folder install_phases = set(['configure', 'build', 'install', 'provenance']) + def do_install(self, keep_prefix=False, keep_stage=False, @@ -887,7 +893,7 @@ class Package(object): fake=False, explicit=False, dirty=False, - install_phases = install_phases): + install_phases=install_phases): """Called by commands to install a package and its dependencies. Package implementations should override install() to describe @@ -908,7 +914,8 @@ class Package(object): run_tests -- Runn tests within the package's install() """ if not self.spec.concrete: - raise ValueError("Can only install concrete packages.") + raise ValueError("Can only install concrete packages: %s." + % self.spec.name) # No installation needed if package is external if self.spec.external: @@ -917,7 +924,8 @@ class Package(object): return # Ensure package is not already installed - if 'install' in install_phases and spack.install_layout.check_installed(self.spec): + layout = spack.install_layout + if 'install' in install_phases and layout.check_installed(self.spec): tty.msg("%s is already installed in %s" % (self.name, self.prefix)) rec = spack.installed_db.get_record(self.spec) if (not rec.explicit) and explicit: @@ -998,20 +1006,17 @@ class Package(object): if 'install' in self.install_phases: self.sanity_check_prefix() - # Copy provenance into the install directory on success if 'provenance' in self.install_phases: - log_install_path = spack.install_layout.build_log_path( - self.spec) - env_install_path = spack.install_layout.build_env_path( - self.spec) - packages_dir = spack.install_layout.build_packages_path( - self.spec) + log_install_path = layout.build_log_path(self.spec) + env_install_path = layout.build_env_path(self.spec) + packages_dir = layout.build_packages_path(self.spec) # Remove first if we're overwriting another build # (can happen with spack setup) try: - shutil.rmtree(packages_dir) # log_install_path and env_install_path are inside this + # log_install_path and env_install_path are here + shutil.rmtree(packages_dir) except: pass @@ -1038,7 +1043,7 @@ class Package(object): except directory_layout.InstallDirectoryAlreadyExistsError: if 'install' in install_phases: # Abort install if install directory exists. - # But do NOT remove it (you'd be overwriting someon else's stuff) + # But do NOT remove it (you'd be overwriting someone's data) tty.warn("Keeping existing install prefix in place.") raise else: @@ -1089,7 +1094,7 @@ class Package(object): def do_install_dependencies(self, **kwargs): # Pass along paths of dependencies here - for dep in self.spec.dependencies.values(): + for dep in self.spec.dependencies(): dep.package.do_install(**kwargs) @property @@ -1270,7 +1275,7 @@ class Package(object): (self.name, self.extendee.name)) def do_activate(self, force=False): - """Called on an etension to invoke the extendee's activate method. + """Called on an extension to invoke the extendee's activate method. Commands should call this routine, and should not call activate() directly. @@ -1282,7 +1287,7 @@ class Package(object): # Activate any package dependencies that are also extensions. if not force: - for spec in self.spec.traverse(root=False): + for spec in self.spec.traverse(root=False, deptype='run'): if spec.package.extends(self.extendee_spec): if not spec.package.activated: spec.package.do_activate(force=force) @@ -1328,7 +1333,7 @@ class Package(object): for name, aspec in activated.items(): if aspec == self.spec: continue - for dep in aspec.traverse(): + for dep in aspec.traverse(deptype='run'): if self.spec == dep: raise ActivationError( "Cannot deactivate %s because %s is activated and depends on it." # NOQA: ignore=E501 @@ -1414,9 +1419,10 @@ class Package(object): def rpath(self): """Get the rpath this package links with, as a list of paths.""" rpaths = [self.prefix.lib, self.prefix.lib64] - rpaths.extend(d.prefix.lib for d in self.spec.traverse(root=False) + deps = self.spec.dependencies(deptype='link') + rpaths.extend(d.prefix.lib for d in deps if os.path.isdir(d.prefix.lib)) - rpaths.extend(d.prefix.lib64 for d in self.spec.traverse(root=False) + rpaths.extend(d.prefix.lib64 for d in deps if os.path.isdir(d.prefix.lib64)) return rpaths @@ -1433,6 +1439,13 @@ def install_dependency_symlinks(pkg, spec, prefix): flatten_dependencies(spec, prefix) +def use_cray_compiler_names(): + """Compiler names for builds that rely on cray compiler names.""" + os.environ['CC'] = 'cc' + os.environ['CXX'] = 'CC' + os.environ['FC'] = 'ftn' + os.environ['F77'] = 'ftn' + def flatten_dependencies(spec, flat_dir): """Make each dependency of spec present in dir via symlink.""" for dep in spec.traverse(root=False): @@ -1529,24 +1542,29 @@ def _hms(seconds): parts.append("%.2fs" % s) return ' '.join(parts) + class StagedPackage(Package): """A Package subclass where the install() is split up into stages.""" def install_setup(self): - """Creates an spack_setup.py script to configure the package later if we like.""" - raise InstallError("Package %s provides no install_setup() method!" % self.name) + """Creates a spack_setup.py script to configure the package later.""" + raise InstallError( + "Package %s provides no install_setup() method!" % self.name) def install_configure(self): """Runs the configure process.""" - raise InstallError("Package %s provides no install_configure() method!" % self.name) + raise InstallError( + "Package %s provides no install_configure() method!" % self.name) def install_build(self): """Runs the build process.""" - raise InstallError("Package %s provides no install_build() method!" % self.name) + raise InstallError( + "Package %s provides no install_build() method!" % self.name) def install_install(self): """Runs the install process.""" - raise InstallError("Package %s provides no install_install() method!" % self.name) + raise InstallError( + "Package %s provides no install_install() method!" % self.name) def install(self, spec, prefix): if 'setup' in self.install_phases: @@ -1563,9 +1581,10 @@ class StagedPackage(Package): else: # Create a dummy file so the build doesn't fail. # That way, the module file will also be created. - with open(os.path.join(prefix, 'dummy'), 'w') as fout: + with open(os.path.join(prefix, 'dummy'), 'w'): pass + # stackoverflow.com/questions/12791997/how-do-you-do-a-simple-chmod-x-from-within-python def make_executable(path): mode = os.stat(path).st_mode @@ -1573,9 +1592,7 @@ def make_executable(path): os.chmod(path, mode) - class CMakePackage(StagedPackage): - def make_make(self): import multiprocessing # number of jobs spack will to build with. @@ -1589,37 +1606,41 @@ class CMakePackage(StagedPackage): return make def configure_args(self): - """Returns package-specific arguments to be provided to the configure command.""" + """Returns package-specific arguments to be provided to + the configure command. + """ return list() def configure_env(self): - """Returns package-specific environment under which the configure command should be run.""" + """Returns package-specific environment under which the + configure command should be run. + """ return dict() - def spack_transitive_include_path(self): + def transitive_inc_path(self): return ';'.join( os.path.join(dep, 'include') for dep in os.environ['SPACK_DEPENDENCIES'].split(os.pathsep) ) def install_setup(self): - cmd = [str(which('cmake'))] + \ - spack.build_environment.get_std_cmake_args(self) + \ - ['-DCMAKE_INSTALL_PREFIX=%s' % os.environ['SPACK_PREFIX'], - '-DCMAKE_C_COMPILER=%s' % os.environ['SPACK_CC'], - '-DCMAKE_CXX_COMPILER=%s' % os.environ['SPACK_CXX'], - '-DCMAKE_Fortran_COMPILER=%s' % os.environ['SPACK_FC']] + \ - self.configure_args() - - env = dict() - env['PATH'] = os.environ['PATH'] - env['SPACK_TRANSITIVE_INCLUDE_PATH'] = self.spack_transitive_include_path() - env['CMAKE_PREFIX_PATH'] = os.environ['CMAKE_PREFIX_PATH'] + cmd = [str(which('cmake'))] + cmd += spack.build_environment.get_std_cmake_args(self) + cmd += ['-DCMAKE_INSTALL_PREFIX=%s' % os.environ['SPACK_PREFIX'], + '-DCMAKE_C_COMPILER=%s' % os.environ['SPACK_CC'], + '-DCMAKE_CXX_COMPILER=%s' % os.environ['SPACK_CXX'], + '-DCMAKE_Fortran_COMPILER=%s' % os.environ['SPACK_FC']] + cmd += self.configure_args() + + env = { + 'PATH': os.environ['PATH'], + 'SPACK_TRANSITIVE_INCLUDE_PATH': self.transitive_inc_path(), + 'CMAKE_PREFIX_PATH': os.environ['CMAKE_PREFIX_PATH'] + } setup_fname = 'spconfig.py' with open(setup_fname, 'w') as fout: - fout.write(\ -r"""#!%s + fout.write(r"""#!%s # import sys @@ -1627,7 +1648,7 @@ import os import subprocess def cmdlist(str): - return list(x.strip().replace("'",'') for x in str.split('\n') if x) + return list(x.strip().replace("'",'') for x in str.split('\n') if x) env = dict(os.environ) """ % sys.executable) @@ -1635,34 +1656,39 @@ env = dict(os.environ) for name in env_vars: val = env[name] if string.find(name, 'PATH') < 0: - fout.write('env[%s] = %s\n' % (repr(name),repr(val))) + fout.write('env[%s] = %s\n' % (repr(name), repr(val))) else: if name == 'SPACK_TRANSITIVE_INCLUDE_PATH': sep = ';' else: sep = ':' - fout.write('env[%s] = "%s".join(cmdlist("""\n' % (repr(name),sep)) + fout.write('env[%s] = "%s".join(cmdlist("""\n' + % (repr(name), sep)) for part in string.split(val, sep): fout.write(' %s\n' % part) fout.write('"""))\n') - fout.write("env['CMAKE_TRANSITIVE_INCLUDE_PATH'] = env['SPACK_TRANSITIVE_INCLUDE_PATH'] # Deprecated\n") + fout.write("env['CMAKE_TRANSITIVE_INCLUDE_PATH'] = " + "env['SPACK_TRANSITIVE_INCLUDE_PATH'] # Deprecated\n") fout.write('\ncmd = cmdlist("""\n') fout.write('%s\n' % cmd[0]) for arg in cmd[1:]: fout.write(' %s\n' % arg) fout.write('""") + sys.argv[1:]\n') - fout.write('\nproc = subprocess.Popen(cmd, env=env)\nproc.wait()\n') + fout.write('\nproc = subprocess.Popen(cmd, env=env)\n') + fout.write('proc.wait()\n') make_executable(setup_fname) - def install_configure(self): cmake = which('cmake') with working_dir(self.build_directory, create=True): - os.environ.update(self.configure_env()) - os.environ['SPACK_TRANSITIVE_INCLUDE_PATH'] = self.spack_transitive_include_path() - options = self.configure_args() + spack.build_environment.get_std_cmake_args(self) + env = os.environ + env.update(self.configure_env()) + env['SPACK_TRANSITIVE_INCLUDE_PATH'] = self.transitive_inc_path() + + options = self.configure_args() + options += spack.build_environment.get_std_cmake_args(self) cmake(self.source_directory, *options) def install_build(self): diff --git a/lib/spack/spack/platforms/cray_xc.py b/lib/spack/spack/platforms/cray_xc.py index e710303e23..03d0383cc5 100644 --- a/lib/spack/spack/platforms/cray_xc.py +++ b/lib/spack/spack/platforms/cray_xc.py @@ -1,7 +1,11 @@ import os +import spack from spack.architecture import Platform, Target from spack.operating_systems.linux_distro import LinuxDistro from spack.operating_systems.cnl import Cnl +from spack.util.executable import which +from llnl.util.filesystem import join_path + class CrayXc(Platform): priority = 20 @@ -9,9 +13,8 @@ class CrayXc(Platform): back_end = 'ivybridge' default = 'ivybridge' - front_os = "SuSE11" back_os = "CNL10" - default_os = "CNL10" + default_os = "CNL10" def __init__(self): ''' Since cori doesn't have ivybridge as a front end it's better @@ -32,15 +35,38 @@ class CrayXc(Platform): # Could switch to use modules and fe targets for front end # Currently using compilers by path for front end. self.add_target('sandybridge', Target('sandybridge')) - self.add_target('ivybridge', + self.add_target('ivybridge', Target('ivybridge', 'craype-ivybridge')) - self.add_target('haswell', - Target('haswell','craype-haswell')) + self.add_target('haswell', + Target('haswell', 'craype-haswell')) - self.add_operating_system('SuSE11', LinuxDistro()) + # Front end of the cray platform is a linux distro. + linux_dist = LinuxDistro() + self.front_os = str(linux_dist) + self.add_operating_system(str(linux_dist), linux_dist) self.add_operating_system('CNL10', Cnl()) @classmethod - def detect(self): - return os.path.exists('/opt/cray/craype') + def setup_platform_environment(self, pkg, env): + """ Change the linker to default dynamic to be more + similar to linux/standard linker behavior + """ + env.set('CRAYPE_LINK_TYPE', 'dynamic') + cray_wrapper_names = join_path(spack.build_env_path, 'cray') + if os.path.isdir(cray_wrapper_names): + env.prepend_path('PATH', cray_wrapper_names) + env.prepend_path('SPACK_ENV_PATHS', cray_wrapper_names) + @classmethod + def detect(self): + try: + cc_verbose = which('ftn') + text = cc_verbose('-craype-verbose', + output=str, error=str, + ignore_errors=True).split() + if '-D__CRAYXC' in text: + return True + else: + return False + except: + return False diff --git a/lib/spack/spack/preferred_packages.py b/lib/spack/spack/preferred_packages.py index 4820584150..5f18e212b6 100644 --- a/lib/spack/spack/preferred_packages.py +++ b/lib/spack/spack/preferred_packages.py @@ -26,8 +26,10 @@ import spack from spack.version import * + class PreferredPackages(object): - _default_order = {'compiler' : [ 'gcc', 'intel', 'clang', 'pgi', 'xlc' ] } # Arbitrary, but consistent + # Arbitrary, but consistent + _default_order = {'compiler': ['gcc', 'intel', 'clang', 'pgi', 'xlc']} def __init__(self): self.preferred = spack.config.get_config('packages') @@ -35,24 +37,25 @@ class PreferredPackages(object): # Given a package name, sort component (e.g, version, compiler, ...), and # a second_key (used by providers), return the list - def _order_for_package(self, pkgname, component, second_key, test_all=True): + def _order_for_package(self, pkgname, component, second_key, + test_all=True): pkglist = [pkgname] if test_all: pkglist.append('all') for pkg in pkglist: order = self.preferred.get(pkg, {}).get(component, {}) - if type(order) is dict: + if isinstance(order, dict) and second_key: order = order.get(second_key, {}) if not order: continue return [str(s).strip() for s in order] return [] - # A generic sorting function. Given a package name and sort # component, return less-than-0, 0, or greater-than-0 if # a is respectively less-than, equal to, or greater than b. - def _component_compare(self, pkgname, component, a, b, reverse_natural_compare, second_key): + def _component_compare(self, pkgname, component, a, b, + reverse_natural_compare, second_key): if a is None: return -1 if b is None: @@ -84,92 +87,109 @@ class PreferredPackages(object): else: return 0 - # A sorting function for specs. Similar to component_compare, but # a and b are considered to match entries in the sorting list if they # satisfy the list component. - def _spec_compare(self, pkgname, component, a, b, reverse_natural_compare, second_key): - if not a or not a.concrete: + def _spec_compare(self, pkgname, component, a, b, + reverse_natural_compare, second_key): + if not a or (not a.concrete and not second_key): return -1 - if not b or not b.concrete: + if not b or (not b.concrete and not second_key): return 1 specs = self._spec_for_pkgname(pkgname, component, second_key) a_index = None b_index = None reverse = -1 if reverse_natural_compare else 1 for i, cspec in enumerate(specs): - if a_index == None and (cspec.satisfies(a) or a.satisfies(cspec)): + if a_index is None and (cspec.satisfies(a) or a.satisfies(cspec)): a_index = i if b_index: break - if b_index == None and (cspec.satisfies(b) or b.satisfies(cspec)): + if b_index is None and (cspec.satisfies(b) or b.satisfies(cspec)): b_index = i if a_index: break - if a_index != None and b_index == None: return -1 - elif a_index == None and b_index != None: return 1 - elif a_index != None and b_index == a_index: return -1 * cmp(a, b) - elif a_index != None and b_index != None and a_index != b_index: return cmp(a_index, b_index) - else: return cmp(a, b) * reverse - - + if a_index is not None and b_index is None: + return -1 + elif a_index is None and b_index is not None: + return 1 + elif a_index is not None and b_index == a_index: + return -1 * cmp(a, b) + elif (a_index is not None and b_index is not None and + a_index != b_index): + return cmp(a_index, b_index) + else: + return cmp(a, b) * reverse # Given a sort order specified by the pkgname/component/second_key, return # a list of CompilerSpecs, VersionLists, or Specs for that sorting list. def _spec_for_pkgname(self, pkgname, component, second_key): key = (pkgname, component, second_key) - if not key in self._spec_for_pkgname_cache: + if key not in self._spec_for_pkgname_cache: pkglist = self._order_for_package(pkgname, component, second_key) if not pkglist: if component in self._default_order: pkglist = self._default_order[component] if component == 'compiler': - self._spec_for_pkgname_cache[key] = [spack.spec.CompilerSpec(s) for s in pkglist] + self._spec_for_pkgname_cache[key] = \ + [spack.spec.CompilerSpec(s) for s in pkglist] elif component == 'version': - self._spec_for_pkgname_cache[key] = [VersionList(s) for s in pkglist] + self._spec_for_pkgname_cache[key] = \ + [VersionList(s) for s in pkglist] else: - self._spec_for_pkgname_cache[key] = [spack.spec.Spec(s) for s in pkglist] + self._spec_for_pkgname_cache[key] = \ + [spack.spec.Spec(s) for s in pkglist] return self._spec_for_pkgname_cache[key] - def provider_compare(self, pkgname, provider_str, a, b): - """Return less-than-0, 0, or greater than 0 if a is respecively less-than, equal-to, or - greater-than b. A and b are possible implementations of provider_str. - One provider is less-than another if it is preferred over the other. - For example, provider_compare('scorep', 'mpi', 'mvapich', 'openmpi') would return -1 if - mvapich should be preferred over openmpi for scorep.""" - return self._spec_compare(pkgname, 'providers', a, b, False, provider_str) - + """Return less-than-0, 0, or greater than 0 if a is respecively + less-than, equal-to, or greater-than b. A and b are possible + implementations of provider_str. One provider is less-than another + if it is preferred over the other. For example, + provider_compare('scorep', 'mpi', 'mvapich', 'openmpi') would + return -1 if mvapich should be preferred over openmpi for scorep.""" + return self._spec_compare(pkgname, 'providers', a, b, False, + provider_str) def spec_has_preferred_provider(self, pkgname, provider_str): - """Return True iff the named package has a list of preferred provider""" - return bool(self._order_for_package(pkgname, 'providers', provider_str, False)) - + """Return True iff the named package has a list of preferred + providers""" + return bool(self._order_for_package(pkgname, 'providers', + provider_str, False)) + + def spec_preferred_variants(self, pkgname): + """Return a VariantMap of preferred variants and their values""" + variants = self.preferred.get(pkgname, {}).get('variants', '') + if not isinstance(variants, basestring): + variants = "".join(variants) + return spack.spec.Spec(pkgname + variants).variants def version_compare(self, pkgname, a, b): """Return less-than-0, 0, or greater than 0 if version a of pkgname is - respecively less-than, equal-to, or greater-than version b of pkgname. - One version is less-than another if it is preferred over the other.""" + respectively less-than, equal-to, or greater-than version b of + pkgname. One version is less-than another if it is preferred over + the other.""" return self._spec_compare(pkgname, 'version', a, b, True, None) - def variant_compare(self, pkgname, a, b): """Return less-than-0, 0, or greater than 0 if variant a of pkgname is - respecively less-than, equal-to, or greater-than variant b of pkgname. - One variant is less-than another if it is preferred over the other.""" + respectively less-than, equal-to, or greater-than variant b of + pkgname. One variant is less-than another if it is preferred over + the other.""" return self._component_compare(pkgname, 'variant', a, b, False, None) - def architecture_compare(self, pkgname, a, b): - """Return less-than-0, 0, or greater than 0 if architecture a of pkgname is - respecively less-than, equal-to, or greater-than architecture b of pkgname. - One architecture is less-than another if it is preferred over the other.""" - return self._component_compare(pkgname, 'architecture', a, b, False, None) - + """Return less-than-0, 0, or greater than 0 if architecture a of pkgname + is respectively less-than, equal-to, or greater-than architecture b + of pkgname. One architecture is less-than another if it is preferred + over the other.""" + return self._component_compare(pkgname, 'architecture', a, b, + False, None) def compiler_compare(self, pkgname, a, b): """Return less-than-0, 0, or greater than 0 if compiler a of pkgname is - respecively less-than, equal-to, or greater-than compiler b of pkgname. - One compiler is less-than another if it is preferred over the other.""" + respecively less-than, equal-to, or greater-than compiler b of + pkgname. One compiler is less-than another if it is preferred over + the other.""" return self._spec_compare(pkgname, 'compiler', a, b, False, None) diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 77bc57147d..8bdae0445e 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -96,7 +96,6 @@ specs to avoid ambiguity. Both are provided because ~ can cause shell expansion when it is the first character in an id typed on the command line. """ import sys -import itertools import hashlib import base64 import imp @@ -116,8 +115,6 @@ import spack.parse import spack.error import spack.compilers as compilers -# TODO: move display_specs to some other location. -from spack.cmd.find import display_specs from spack.version import * from spack.util.string import * from spack.util.prefix import Prefix @@ -155,6 +152,14 @@ _separators = '[%s]' % ''.join(color_formats.keys()) every time we call str()""" _any_version = VersionList([':']) +# Special types of dependencies. +alldeps = ('build', 'link', 'run') +nolink = ('build', 'run') +special_types = { + 'alldeps': alldeps, + 'nolink': nolink, +} + def index_specs(specs): """Take a list of specs and return a dict of lists. Dict is @@ -292,6 +297,32 @@ class CompilerSpec(object): @key_ordering +class DependencySpec(object): + """Dependencies can be one (or more) of several types: + + - build: needs to be in the PATH at build time. + - link: is linked to and added to compiler flags. + - run: needs to be in the PATH for the package to run. + + Fields: + - spec: the spack.spec.Spec description of a dependency. + - deptypes: strings representing the type of dependency this is. + """ + def __init__(self, spec, deptypes): + self.spec = spec + self.deptypes = deptypes + + def _cmp_key(self): + return self.spec + + def copy(self): + return DependencySpec(self.spec.copy(), self.deptype) + + def __str__(self): + return str(self.spec) + + +@key_ordering class VariantSpec(object): """Variants are named, build-time options for a package. Names depend @@ -440,11 +471,11 @@ class DependencyMap(HashableMap): The DependencyMap is keyed by name. """ @property def concrete(self): - return all(d.concrete for d in self.values()) + return all(d.spec.concrete for d in self.values()) def __str__(self): return ''.join( - ["^" + str(self[name]) for name in sorted(self.keys())]) + ["^" + str(self[name].spec) for name in sorted(self.keys())]) @key_ordering @@ -472,13 +503,13 @@ class Spec(object): # writes directly into this Spec object. other = spec_list[0] self.name = other.name - self.dependents = other.dependents self.versions = other.versions self.architecture = other.architecture self.compiler = other.compiler self.compiler_flags = other.compiler_flags self.compiler_flags.spec = self - self.dependencies = other.dependencies + self._dependencies = other._dependencies + self._dependents = other._dependents self.variants = other.variants self.variants.spec = self self.namespace = other.namespace @@ -500,7 +531,50 @@ class Spec(object): # Spec(a, b) will copy a but just add b as a dep. for dep in dep_like: spec = dep if isinstance(dep, Spec) else Spec(dep) - self._add_dependency(spec) + # XXX(deptype): default deptypes + self._add_dependency(spec, ('build', 'link')) + + def get_dependency(self, name): + dep = self._dependencies.get(name) + if dep is not None: + return dep + raise InvalidDependencyException( + self.name + " does not depend on " + comma_or(name)) + + def _deptype_norm(self, deptype): + if deptype is None: + return alldeps + # Force deptype to be a set object so that we can do set intersections. + if isinstance(deptype, str): + # Support special deptypes. + return special_types.get(deptype, (deptype,)) + return deptype + + def _find_deps(self, where, deptype): + deptype = self._deptype_norm(deptype) + + return [dep.spec + for dep in where.values() + if deptype and any(d in deptype for d in dep.deptypes)] + + def dependencies(self, deptype=None): + return self._find_deps(self._dependencies, deptype) + + def dependents(self, deptype=None): + return self._find_deps(self._dependents, deptype) + + def _find_deps_dict(self, where, deptype): + deptype = self._deptype_norm(deptype) + + return dict((dep.spec.name, dep) + for dep in where.values() + if deptype and any(d in deptype for d in dep.deptypes)) + + def dependencies_dict(self, deptype=None): + return self._find_deps_dict(self._dependencies, deptype) + + def dependents_dict(self, deptype=None): + return self._find_deps_dict(self._dependents, deptype) # # Private routines here are called by the parser when building a spec. @@ -578,7 +652,8 @@ class Spec(object): mod = imp.load_source(mod_name, path) class_name = mod_to_class(value) if not hasattr(mod, class_name): - tty.die('No class %s defined in %s' % (class_name, mod_name)) + tty.die( + 'No class %s defined in %s' % (class_name, mod_name)) cls = getattr(mod, class_name) if not inspect.isclass(cls): tty.die('%s.%s is not a class' % (mod_name, class_name)) @@ -601,29 +676,32 @@ class Spec(object): def _set_os(self, value): """Called by the parser to set the architecture operating system""" - if self.architecture.platform: - self.architecture.platform_os = self.architecture.platform.operating_system(value) + arch = self.architecture + if arch.platform: + arch.platform_os = arch.platform.operating_system(value) def _set_target(self, value): """Called by the parser to set the architecture target""" - if self.architecture.platform: - self.architecture.target = self.architecture.platform.target(value) + arch = self.architecture + if arch.platform: + arch.target = arch.platform.target(value) - def _add_dependency(self, spec): + def _add_dependency(self, spec, deptypes): """Called by the parser to add another spec as a dependency.""" - if spec.name in self.dependencies: + if spec.name in self._dependencies: raise DuplicateDependencyError( "Cannot depend on '%s' twice" % spec) - self.dependencies[spec.name] = spec - spec.dependents[self.name] = self + self._dependencies[spec.name] = DependencySpec(spec, deptypes) + spec._dependents[self.name] = DependencySpec(self, deptypes) # # Public interface # @property def fullname(self): - return (('%s.%s' % (self.namespace, self.name)) if self.namespace else - (self.name if self.name else '')) + return ( + ('%s.%s' % (self.namespace, self.name)) if self.namespace else + (self.name if self.name else '')) @property def root(self): @@ -632,15 +710,15 @@ class Spec(object): installed). This will throw an assertion error if that is not the case. """ - if not self.dependents: + if not self._dependents: return self # If the spec has multiple dependents, ensure that they all # lead to the same place. Spack shouldn't deal with any DAGs # with multiple roots, so something's wrong if we find one. - depiter = iter(self.dependents.values()) - first_root = next(depiter).root - assert(all(first_root is d.root for d in depiter)) + depiter = iter(self._dependents.values()) + first_root = next(depiter).spec.root + assert(all(first_root is d.spec.root for d in depiter)) return first_root @property @@ -679,18 +757,29 @@ class Spec(object): if self._concrete: return True - self._concrete = bool(not self.virtual - and self.namespace is not None - and self.versions.concrete - and self.variants.concrete - and self.architecture - and self.architecture.concrete - and self.compiler and self.compiler.concrete - and self.compiler_flags.concrete - and self.dependencies.concrete) + self._concrete = bool(not self.virtual and + self.namespace is not None and + self.versions.concrete and + self.variants.concrete and + self.architecture and + self.architecture.concrete and + self.compiler and self.compiler.concrete and + self.compiler_flags.concrete and + self._dependencies.concrete) return self._concrete - def traverse(self, visited=None, d=0, **kwargs): + def traverse(self, visited=None, deptype=None, **kwargs): + traversal = self.traverse_with_deptype(visited=visited, + deptype=deptype, + **kwargs) + if kwargs.get('depth', False): + return [(s[0], s[1].spec) for s in traversal] + else: + return [s.spec for s in traversal] + + def traverse_with_deptype(self, visited=None, d=0, deptype=None, + deptype_query=None, _self_deptype=None, + **kwargs): """Generic traversal of the DAG represented by this spec. This will yield each node in the spec. Options: @@ -742,6 +831,12 @@ class Spec(object): direction = kwargs.get('direction', 'children') order = kwargs.get('order', 'pre') + if deptype is None: + deptype = alldeps + + if deptype_query is None: + deptype_query = ('link', 'run') + # Make sure kwargs have legal values; raise ValueError if not. def validate(name, val, allowed_values): if val not in allowed_values: @@ -759,30 +854,37 @@ class Spec(object): if key in visited and cover == 'nodes': return - # Determine whether and what to yield for this node. + def return_val(res): + return (d, res) if depth else res + yield_me = yield_root or d > 0 - result = (d, self) if depth else self # Preorder traversal yields before successors if yield_me and order == 'pre': - yield result + yield return_val(DependencySpec(self, _self_deptype)) + + deps = self.dependencies_dict(deptype) # Edge traversal yields but skips children of visited nodes if not (key in visited and cover == 'edges'): # This code determines direction and yields the children/parents - successors = self.dependencies + successors = deps if direction == 'parents': - successors = self.dependents + successors = self.dependents_dict() visited.add(key) for name in sorted(successors): child = successors[name] - for elt in child.traverse(visited, d + 1, **kwargs): + children = child.spec.traverse_with_deptype( + visited, d=d + 1, deptype=deptype_query, + deptype_query=deptype_query, + _self_deptype=child.deptypes, **kwargs) + for elt in children: yield elt # Postorder traversal yields after successors if yield_me and order == 'post': - yield result + yield return_val(DependencySpec(self, _self_deptype)) @property def short_spec(self): @@ -807,6 +909,7 @@ class Spec(object): if self._hash: return self._hash[:length] else: + # XXX(deptype): ignore 'build' dependencies here yaml_text = yaml.dump( self.to_node_dict(), default_flow_style=True, width=sys.maxint) sha = hashlib.sha1(yaml_text) @@ -819,11 +922,15 @@ class Spec(object): params = dict((name, v.value) for name, v in self.variants.items()) params.update(dict((name, value) for name, value in self.compiler_flags.items())) + deps = self.dependencies_dict(deptype=('link', 'run')) d = { - 'parameters' : params, - 'arch' : self.architecture, - 'dependencies' : dict((d, self.dependencies[d].dag_hash()) - for d in sorted(self.dependencies)) + 'parameters': params, + 'arch': self.architecture, + 'dependencies': dict( + (name, { + 'hash': dspec.spec.dag_hash(), + 'type': [str(s) for s in dspec.deptypes]}) + for name, dspec in deps.items()) } # Older concrete specs do not have a namespace. Omit for @@ -848,7 +955,7 @@ class Spec(object): def to_yaml(self, stream=None): node_list = [] - for s in self.traverse(order='pre'): + for s in self.traverse(order='pre', deptype=('link', 'run')): node = s.to_node_dict() node[s.name]['hash'] = s.dag_hash() node_list.append(node) @@ -889,9 +996,34 @@ class Spec(object): raise SpackRecordError( "Did not find a valid format for variants in YAML file") + # Don't read dependencies here; from_node_dict() is used by + # from_yaml() to read the root *and* each dependency spec. + return spec @staticmethod + def read_yaml_dep_specs(dependency_dict): + """Read the DependencySpec portion of a YAML-formatted Spec. + + This needs to be backward-compatible with older spack spec + formats so that reindex will work on old specs/databases. + """ + for dep_name, elt in dependency_dict.items(): + if isinstance(elt, basestring): + # original format, elt is just the dependency hash. + dag_hash, deptypes = elt, ['build', 'link'] + elif isinstance(elt, tuple): + # original deptypes format: (used tuples, not future-proof) + dag_hash, deptypes = elt + elif isinstance(elt, dict): + # new format: elements of dependency spec are keyed. + dag_hash, deptypes = elt['hash'], elt['type'] + else: + raise SpecError("Couldn't parse dependency types in spec.") + + yield dep_name, dag_hash, list(deptypes) + + @staticmethod def from_yaml(stream): """Construct a spec from YAML. @@ -902,25 +1034,30 @@ class Spec(object): represent more than the DAG does. """ - deps = {} - spec = None - try: yfile = yaml.load(stream) except MarkedYAMLError, e: raise SpackYAMLError("error parsing YAML spec:", str(e)) - for node in yfile['spec']: - name = next(iter(node)) - dep = Spec.from_node_dict(node) - if not spec: - spec = dep - deps[dep.name] = dep + nodes = yfile['spec'] + + # Read nodes out of list. Root spec is the first element; + # dependencies are the following elements. + dep_list = [Spec.from_node_dict(node) for node in nodes] + if not dep_list: + raise SpecError("YAML spec contains no nodes.") + deps = dict((spec.name, spec) for spec in dep_list) + spec = dep_list[0] - for node in yfile['spec']: + for node in nodes: + # get dependency dict from the node. name = next(iter(node)) - for dep_name in node[name]['dependencies']: - deps[name].dependencies[dep_name] = deps[dep_name] + yaml_deps = node[name]['dependencies'] + for dname, dhash, dtypes in Spec.read_yaml_dep_specs(yaml_deps): + # Fill in dependencies by looking them up by name in deps dict + deps[name]._dependencies[dname] = DependencySpec( + deps[dname], set(dtypes)) + return spec def _concretize_helper(self, presets=None, visited=None): @@ -940,8 +1077,9 @@ class Spec(object): changed = False # Concretize deps first -- this is a bottom-up process. - for name in sorted(self.dependencies.keys()): - changed |= self.dependencies[name]._concretize_helper(presets, visited) + for name in sorted(self._dependencies.keys()): + changed |= self._dependencies[ + name].spec._concretize_helper(presets, visited) if self.name in presets: changed |= self.constrain(presets[self.name]) @@ -965,13 +1103,16 @@ class Spec(object): def _replace_with(self, concrete): """Replace this virtual spec with a concrete spec.""" assert(self.virtual) - for name, dependent in self.dependents.items(): + for name, dep_spec in self._dependents.items(): + dependent = dep_spec.spec + deptypes = dep_spec.deptypes + # remove self from all dependents. - del dependent.dependencies[self.name] + del dependent._dependencies[self.name] # add the replacement, unless it is already a dep of dependent. - if concrete.name not in dependent.dependencies: - dependent._add_dependency(concrete) + if concrete.name not in dependent._dependencies: + dependent._add_dependency(concrete, deptypes) def _replace_node(self, replacement): """Replace this spec with another. @@ -982,13 +1123,15 @@ class Spec(object): to be normalized. """ - for name, dependent in self.dependents.items(): - del dependent.dependencies[self.name] - dependent._add_dependency(replacement) + for name, dep_spec in self._dependents.items(): + dependent = dep_spec.spec + deptypes = dep_spec.deptypes + del dependent._dependencies[self.name] + dependent._add_dependency(replacement, deptypes) - for name, dep in self.dependencies.items(): - del dep.dependents[self.name] - del self.dependencies[dep.name] + for name, dep_spec in self._dependencies.items(): + del dep_spec.spec.dependents[self.name] + del self._dependencies[dep.name] def _expand_virtual_packages(self): """Find virtual packages in this spec, replace them with providers, @@ -1008,13 +1151,14 @@ class Spec(object): a problem. """ # Make an index of stuff this spec already provides + # XXX(deptype): 'link' and 'run'? self_index = ProviderIndex(self.traverse(), restrict=True) changed = False done = False while not done: done = True - + # XXX(deptype): 'link' and 'run'? for spec in list(self.traverse()): replacement = None if spec.virtual: @@ -1054,24 +1198,26 @@ class Spec(object): # If replacement is external then trim the dependencies if replacement.external or replacement.external_module: - if (spec.dependencies): + if (spec._dependencies): changed = True - spec.dependencies = DependencyMap() - replacement.dependencies = DependencyMap() + spec._dependencies = DependencyMap() + replacement._dependencies = DependencyMap() replacement.architecture = self.architecture # TODO: could this and the stuff in _dup be cleaned up? def feq(cfield, sfield): return (not cfield) or (cfield == sfield) - if replacement is spec or (feq(replacement.name, spec.name) and - feq(replacement.versions, spec.versions) and - feq(replacement.compiler, spec.compiler) and - feq(replacement.architecture, spec.architecture) and - feq(replacement.dependencies, spec.dependencies) and - feq(replacement.variants, spec.variants) and - feq(replacement.external, spec.external) and - feq(replacement.external_module, spec.external_module)): + if replacement is spec or ( + feq(replacement.name, spec.name) and + feq(replacement.versions, spec.versions) and + feq(replacement.compiler, spec.compiler) and + feq(replacement.architecture, spec.architecture) and + feq(replacement._dependencies, spec._dependencies) and + feq(replacement.variants, spec.variants) and + feq(replacement.external, spec.external) and + feq(replacement.external_module, + spec.external_module)): continue # Refine this spec to the candidate. This uses # replace_with AND dup so that it can work in @@ -1116,7 +1262,7 @@ class Spec(object): changed = any(changes) force = True - for s in self.traverse(): + for s in self.traverse(deptype_query=alldeps): # After concretizing, assign namespaces to anything left. # Note that this doesn't count as a "change". The repository # configuration is constant throughout a spack run, and @@ -1128,10 +1274,10 @@ class Spec(object): if s.namespace is None: s.namespace = spack.repo.repo_for_pkg(s.name).namespace - for s in self.traverse(root=False): if s.external_module: - compiler = spack.compilers.compiler_for_spec(s.compiler, s.architecture) + compiler = spack.compilers.compiler_for_spec( + s.compiler, s.architecture) for mod in compiler.modules: load_module(mod) @@ -1146,7 +1292,7 @@ class Spec(object): Only for internal use -- client code should use "concretize" unless there is a need to force a spec to be concrete. """ - for s in self.traverse(): + for s in self.traverse(deptype_query=alldeps): s._normal = True s._concrete = True @@ -1159,6 +1305,13 @@ class Spec(object): return clone def flat_dependencies(self, **kwargs): + flat_deps = DependencyMap() + flat_deps_deptypes = self.flat_dependencies_with_deptype(**kwargs) + for name, depspec in flat_deps_deptypes.items(): + flat_deps[name] = depspec.spec + return flat_deps + + def flat_dependencies_with_deptype(self, **kwargs): """Return a DependencyMap containing all of this spec's dependencies with their constraints merged. @@ -1169,23 +1322,31 @@ class Spec(object): returns them. """ copy = kwargs.get('copy', True) + deptype_query = kwargs.get('deptype_query') flat_deps = DependencyMap() try: - for spec in self.traverse(root=False): + deptree = self.traverse_with_deptype(root=False, + deptype_query=deptype_query) + for depspec in deptree: + spec = depspec.spec + deptypes = depspec.deptypes + if spec.name not in flat_deps: if copy: - flat_deps[spec.name] = spec.copy(deps=False) + dep_spec = DependencySpec(spec.copy(deps=False), + deptypes) else: - flat_deps[spec.name] = spec + dep_spec = DependencySpec(spec, deptypes) + flat_deps[spec.name] = dep_spec else: - flat_deps[spec.name].constrain(spec) + flat_deps[spec.name].spec.constrain(spec) if not copy: - for dep in flat_deps.values(): - dep.dependencies.clear() - dep.dependents.clear() - self.dependencies.clear() + for depspec in flat_deps.values(): + depspec.spec._dependencies.clear() + depspec.spec._dependents.clear() + self._dependencies.clear() return flat_deps @@ -1200,17 +1361,11 @@ class Spec(object): """Return DependencyMap that points to all the dependencies in this spec.""" dm = DependencyMap() + # XXX(deptype): use a deptype kwarg. for spec in self.traverse(): dm[spec.name] = spec return dm - def flatten(self): - """Pull all dependencies up to the root (this spec). - Merge constraints for dependencies with the same name, and if they - conflict, throw an exception. """ - for dep in self.flat_dependencies(copy=False): - self._add_dependency(dep) - def _evaluate_dependency_conditions(self, name): """Evaluate all the conditions on a dependency with this name. @@ -1267,7 +1422,8 @@ class Spec(object): elif required: raise UnsatisfiableProviderSpecError(required[0], vdep) - def _merge_dependency(self, dep, visited, spec_deps, provider_index): + def _merge_dependency(self, dep, deptypes, visited, spec_deps, + provider_index): """Merge the dependency into this spec. This is the core of normalize(). There are some basic steps: @@ -1294,7 +1450,9 @@ class Spec(object): dep = provider else: index = ProviderIndex([dep], restrict=True) - for vspec in (v for v in spec_deps.values() if v.virtual): + for vspec in (v.spec + for v in spec_deps.values() + if v.spec.virtual): if index.providers_for(vspec): vspec._replace_with(dep) del spec_deps[vspec.name] @@ -1307,25 +1465,25 @@ class Spec(object): # If the spec isn't already in the set of dependencies, clone # it from the package description. if dep.name not in spec_deps: - spec_deps[dep.name] = dep.copy() + spec_deps[dep.name] = DependencySpec(dep.copy(), deptypes) changed = True # Constrain package information with spec info try: - changed |= spec_deps[dep.name].constrain(dep) + changed |= spec_deps[dep.name].spec.constrain(dep) except UnsatisfiableSpecError, e: e.message = "Invalid spec: '%s'. " e.message += "Package %s requires %s %s, but spec asked for %s" - e.message %= (spec_deps[dep.name], dep.name, e.constraint_type, - e.required, e.provided) + e.message %= (spec_deps[dep.name].spec, dep.name, + e.constraint_type, e.required, e.provided) raise e # Add merged spec to my deps and recurse dependency = spec_deps[dep.name] - if dep.name not in self.dependencies: - self._add_dependency(dependency) + if dep.name not in self._dependencies: + self._add_dependency(dependency.spec, dependency.deptypes) - changed |= dependency._normalize_helper( + changed |= dependency.spec._normalize_helper( visited, spec_deps, provider_index) return changed @@ -1351,10 +1509,11 @@ class Spec(object): for dep_name in pkg.dependencies: # Do we depend on dep_name? If so pkg_dep is not None. pkg_dep = self._evaluate_dependency_conditions(dep_name) + deptypes = pkg._deptypes[dep_name] # If pkg_dep is a dependency, merge it. if pkg_dep: changed |= self._merge_dependency( - pkg_dep, visited, spec_deps, provider_index) + pkg_dep, deptypes, visited, spec_deps, provider_index) any_change |= changed return any_change @@ -1385,11 +1544,13 @@ class Spec(object): # Ensure first that all packages & compilers in the DAG exist. self.validate_names() # Get all the dependencies into one DependencyMap - spec_deps = self.flat_dependencies(copy=False) + spec_deps = self.flat_dependencies_with_deptype( + copy=False, deptype_query=alldeps) # Initialize index of virtual dependency providers if # concretize didn't pass us one already - provider_index = ProviderIndex(spec_deps.values(), restrict=True) + provider_index = ProviderIndex( + [s.spec for s in spec_deps.values()], restrict=True) # traverse the package DAG and fill out dependencies according # to package files & their 'when' specs @@ -1462,20 +1623,17 @@ class Spec(object): other.variants[v]) # TODO: Check out the logic here - if self.architecture is not None and other.architecture is not None: - if self.architecture.platform is not None and other.architecture.platform is not None: - if self.architecture.platform != other.architecture.platform: - raise UnsatisfiableArchitectureSpecError(self.architecture, - other.architecture) - if self.architecture.platform_os is not None and other.architecture.platform_os is not None: - if self.architecture.platform_os != other.architecture.platform_os: - raise UnsatisfiableArchitectureSpecError(self.architecture, - other.architecture) - if self.architecture.target is not None and other.architecture.target is not None: - if self.architecture.target != other.architecture.target: - raise UnsatisfiableArchitectureSpecError(self.architecture, - other.architecture) - + sarch, oarch = self.architecture, other.architecture + if sarch is not None and oarch is not None: + if sarch.platform is not None and oarch.platform is not None: + if sarch.platform != oarch.platform: + raise UnsatisfiableArchitectureSpecError(sarch, oarch) + if sarch.platform_os is not None and oarch.platform_os is not None: + if sarch.platform_os != oarch.platform_os: + raise UnsatisfiableArchitectureSpecError(sarch, oarch) + if sarch.target is not None and oarch.target is not None: + if sarch.target != oarch.target: + raise UnsatisfiableArchitectureSpecError(sarch, oarch) changed = False if self.compiler is not None and other.compiler is not None: @@ -1490,15 +1648,16 @@ class Spec(object): changed |= self.compiler_flags.constrain(other.compiler_flags) old = str(self.architecture) - if self.architecture is None or other.architecture is None: - self.architecture = self.architecture or other.architecture + sarch, oarch = self.architecture, other.architecture + if sarch is None or other.architecture is None: + self.architecture = sarch or oarch else: - if self.architecture.platform is None or other.architecture.platform is None: - self.architecture.platform = self.architecture.platform or other.architecture.platform - if self.architecture.platform_os is None or other.architecture.platform_os is None: - self.architecture.platform_os = self.architecture.platform_os or other.architecture.platform_os - if self.architecture.target is None or other.architecture.target is None: - self.architecture.target = self.architecture.target or other.architecture.target + if sarch.platform is None or oarch.platform is None: + self.architecture.platform = sarch.platform or oarch.platform + if sarch.platform_os is None or oarch.platform_os is None: + sarch.platform_os = sarch.platform_os or oarch.platform_os + if sarch.target is None or oarch.target is None: + sarch.target = sarch.target or oarch.target changed |= (str(self.architecture) != old) if deps: @@ -1510,7 +1669,7 @@ class Spec(object): """Apply constraints of other spec's dependencies to this spec.""" other = self._autospec(other) - if not self.dependencies or not other.dependencies: + if not self._dependencies or not other._dependencies: return False # TODO: might want more detail than this, e.g. specific deps @@ -1526,13 +1685,17 @@ class Spec(object): # Update with additional constraints from other spec for name in other.dep_difference(self): - self._add_dependency(other[name].copy()) + dep_spec_copy = other.get_dependency(name) + dep_copy = dep_spec_copy.spec + deptypes = dep_spec_copy.deptypes + self._add_dependency(dep_copy.copy(), deptypes) changed = True return changed def common_dependencies(self, other): """Return names of dependencies that self an other have in common.""" + # XXX(deptype): handle deptypes via deptype kwarg. common = set( s.name for s in self.traverse(root=False)) common.intersection_update( @@ -1625,15 +1788,25 @@ class Spec(object): # Architecture satisfaction is currently just string equality. # If not strict, None means unconstrained. - if self.architecture and other.architecture: - if ((self.architecture.platform and other.architecture.platform and self.architecture.platform != other.architecture.platform) or - (self.architecture.platform_os and other.architecture.platform_os and self.architecture.platform_os != other.architecture.platform_os) or - (self.architecture.target and other.architecture.target and self.architecture.target != other.architecture.target)): + sarch, oarch = self.architecture, other.architecture + if sarch and oarch: + if ((sarch.platform and + oarch.platform and + sarch.platform != oarch.platform) or + + (sarch.platform_os and + oarch.platform_os and + sarch.platform_os != oarch.platform_os) or + + (sarch.target and + oarch.target and + sarch.target != oarch.target)): return False - elif strict and ((other.architecture and not self.architecture) or - (other.architecture.platform and not self.architecture.platform) or - (other.architecture.platform_os and not self.architecture.platform_os) or - (other.architecture.target and not self.architecture.target)): + + elif strict and ((oarch and not sarch) or + (oarch.platform and not sarch.platform) or + (oarch.platform_os and not sarch.platform_os) or + (oarch.target and not sarch.target)): return False if not self.compiler_flags.satisfies( @@ -1657,13 +1830,14 @@ class Spec(object): other = self._autospec(other) if strict: - if other.dependencies and not self.dependencies: + if other._dependencies and not self._dependencies: return False - if not all(dep in self.dependencies for dep in other.dependencies): + if not all(dep in self._dependencies + for dep in other._dependencies): return False - elif not self.dependencies or not other.dependencies: + elif not self._dependencies or not other._dependencies: # if either spec doesn't restrict dependencies then both are # compatible. return True @@ -1714,11 +1888,16 @@ class Spec(object): # We don't count dependencies as changes here changed = True if hasattr(self, 'name'): - changed = (self.name != other.name and self.versions != other.versions and \ - self.architecture != other.architecture and self.compiler != other.compiler and \ - self.variants != other.variants and self._normal != other._normal and \ - self.concrete != other.concrete and self.external != other.external and \ - self.external_module != other.external_module and self.compiler_flags != other.compiler_flags) + changed = (self.name != other.name and + self.versions != other.versions and + self.architecture != other.architecture and + self.compiler != other.compiler and + self.variants != other.variants and + self._normal != other._normal and + self.concrete != other.concrete and + self.external != other.external and + self.external_module != other.external_module and + self.compiler_flags != other.compiler_flags) # Local node attributes get copied first. self.name = other.name @@ -1726,8 +1905,8 @@ class Spec(object): self.architecture = other.architecture self.compiler = other.compiler.copy() if other.compiler else None if kwargs.get('cleardeps', True): - self.dependents = DependencyMap() - self.dependencies = DependencyMap() + self._dependents = DependencyMap() + self._dependencies = DependencyMap() self.compiler_flags = other.compiler_flags.copy() self.variants = other.variants.copy() self.variants.spec = self @@ -1739,15 +1918,30 @@ class Spec(object): # If we copy dependencies, preserve DAG structure in the new spec if kwargs.get('deps', True): # This copies the deps from other using _dup(deps=False) - new_nodes = other.flat_dependencies() + # XXX(deptype): We can keep different instances of specs here iff + # it is only a 'build' dependency (from its parent). + # All other instances must be shared (due to symbol + # and PATH contention). These should probably search + # for any existing installation which can satisfy the + # build and latch onto that because if 3 things need + # the same build dependency and it is *not* + # available, we only want to build it once. + new_nodes = other.flat_dependencies(deptype_query=alldeps) new_nodes[self.name] = self - # Hook everything up properly here by traversing. - for spec in other.traverse(cover='nodes'): - parent = new_nodes[spec.name] - for child in spec.dependencies: - if child not in parent.dependencies: - parent._add_dependency(new_nodes[child]) + stack = [other] + while stack: + cur_spec = stack.pop(0) + new_spec = new_nodes[cur_spec.name] + + for depspec in cur_spec._dependencies.values(): + stack.append(depspec.spec) + + # XXX(deptype): add any new deptypes that may have appeared + # here. + if depspec.spec.name not in new_spec._dependencies: + new_spec._add_dependency( + new_nodes[depspec.spec.name], depspec.deptypes) # Since we preserved structure, we can copy _normal safely. self._normal = other._normal @@ -1790,7 +1984,7 @@ class Spec(object): raise KeyError("No spec with name %s in %s" % (name, self)) def __contains__(self, spec): - """True if this spec satisfis the provided spec, or if any dependency + """True if this spec satisfies the provided spec, or if any dependency does. If the spec has no name, then we parse this one first. """ spec = self._autospec(spec) @@ -1814,13 +2008,13 @@ class Spec(object): if self.ne_node(other): return False - if len(self.dependencies) != len(other.dependencies): + if len(self._dependencies) != len(other._dependencies): return False - ssorted = [self.dependencies[name] - for name in sorted(self.dependencies)] - osorted = [other.dependencies[name] - for name in sorted(other.dependencies)] + ssorted = [self._dependencies[name].spec + for name in sorted(self._dependencies)] + osorted = [other._dependencies[name].spec + for name in sorted(other._dependencies)] for s, o in zip(ssorted, osorted): visited_s = id(s) in vs @@ -1858,7 +2052,6 @@ class Spec(object): self.compiler, self.compiler_flags) - def eq_node(self, other): """Equality with another spec, not including dependencies.""" return self._cmp_node() == other._cmp_node() @@ -1874,9 +2067,10 @@ class Spec(object): 1. A tuple describing this node in the DAG. 2. The hash of each of this node's dependencies' cmp_keys. """ + dep_dict = self.dependencies_dict(deptype=('link', 'run')) return self._cmp_node() + ( - tuple(hash(self.dependencies[name]) - for name in sorted(self.dependencies)),) + tuple(hash(dep_dict[name]) + for name in sorted(dep_dict)),) def colorized(self): return colorize_spec(self) @@ -2053,41 +2247,39 @@ class Spec(object): def dep_string(self): return ''.join("^" + dep.format() for dep in self.sorted_deps()) - def __cmp__(self, other): - #Package name sort order is not configurable, always goes alphabetical + # Package name sort order is not configurable, always goes alphabetical if self.name != other.name: return cmp(self.name, other.name) - #Package version is second in compare order + # Package version is second in compare order pkgname = self.name if self.versions != other.versions: - return spack.pkgsort.version_compare(pkgname, - self.versions, other.versions) + return spack.pkgsort.version_compare( + pkgname, self.versions, other.versions) - #Compiler is third + # Compiler is third if self.compiler != other.compiler: - return spack.pkgsort.compiler_compare(pkgname, - self.compiler, other.compiler) + return spack.pkgsort.compiler_compare( + pkgname, self.compiler, other.compiler) - #Variants + # Variants if self.variants != other.variants: - return spack.pkgsort.variant_compare(pkgname, - self.variants, other.variants) + return spack.pkgsort.variant_compare( + pkgname, self.variants, other.variants) - #Target + # Target if self.architecture != other.architecture: - return spack.pkgsort.architecture_compare(pkgname, - self.architecture, other.architecture) + return spack.pkgsort.architecture_compare( + pkgname, self.architecture, other.architecture) - #Dependency is not configurable - if self.dependencies != other.dependencies: - return -1 if self.dependencies < other.dependencies else 1 + # Dependency is not configurable + if self._dependencies != other._dependencies: + return -1 if self._dependencies < other._dependencies else 1 - #Equal specs + # Equal specs return 0 - def __str__(self): return self.format() + self.dep_string() @@ -2101,12 +2293,14 @@ class Spec(object): indent = kwargs.pop('indent', 0) fmt = kwargs.pop('format', '$_$@$%@+$+$=') prefix = kwargs.pop('prefix', None) + deptypes = kwargs.pop('deptypes', ('build', 'link')) check_kwargs(kwargs, self.tree) out = "" cur_id = 0 ids = {} - for d, node in self.traverse(order='pre', cover=cover, depth=True): + for d, node in self.traverse( + order='pre', cover=cover, depth=True, deptypes=deptypes): if prefix is not None: out += prefix(node) out += " " * indent @@ -2160,8 +2354,8 @@ class SpecLexer(spack.parse.Lexer): # Lexer is always the same for every parser. _lexer = SpecLexer() -class SpecParser(spack.parse.Parser): +class SpecParser(spack.parse.Parser): def __init__(self): super(SpecParser, self).__init__(_lexer) self.previous = None @@ -2196,10 +2390,13 @@ class SpecParser(spack.parse.Parser): specs.append(self.spec(None)) self.previous = None if self.accept(HASH): - specs[-1]._add_dependency(self.spec_by_hash()) + dep = self.spec_by_hash() else: self.expect(ID) - specs[-1]._add_dependency(self.spec(self.token.value)) + dep = self.spec(self.token.value) + # XXX(deptype): default deptypes + def_deptypes = ('build', 'link') + specs[-1]._add_dependency(dep, def_deptypes) else: # Attempt to construct an anonymous spec, but check that @@ -2211,8 +2408,8 @@ class SpecParser(spack.parse.Parser): except spack.parse.ParseError, e: raise SpecParseError(e) - - # If the spec has an os or a target and no platform, give it the default platform + # If the spec has an os or a target and no platform, give it + # the default platform for spec in specs: for s in spec.traverse(): if s.architecture.os_string or s.architecture.target_string: @@ -2263,8 +2460,8 @@ class SpecParser(spack.parse.Parser): spec.external = None spec.external_module = None spec.compiler_flags = FlagMap(spec) - spec.dependents = DependencyMap() - spec.dependencies = DependencyMap() + spec._dependents = DependencyMap() + spec._dependencies = DependencyMap() spec.namespace = spec_namespace spec._hash = None diff --git a/lib/spack/spack/test/__init__.py b/lib/spack/spack/test/__init__.py index fb91f24721..a849d5f350 100644 --- a/lib/spack/spack/test/__init__.py +++ b/lib/spack/spack/test/__init__.py @@ -32,15 +32,17 @@ from llnl.util.tty.colify import colify from spack.test.tally_plugin import Tally """Names of tests to be included in Spack's test suite""" -test_names = ['architecture', 'versions', 'url_parse', 'url_substitution', 'packages', 'stage', - 'spec_syntax', 'spec_semantics', 'spec_dag', 'concretize', - 'multimethod', 'install', 'package_sanity', 'config', - 'directory_layout', 'pattern', 'python_version', 'git_fetch', - 'svn_fetch', 'hg_fetch', 'mirror', 'modules', 'url_extrapolate', - 'cc', 'link_tree', 'spec_yaml', 'optional_deps', - 'make_executable', 'configure_guess', 'lock', 'database', - 'namespace_trie', 'yaml', 'sbang', 'environment', 'cmd.find', - 'cmd.uninstall', 'cmd.test_install', 'cmd.test_compiler_cmd'] +test_names = [ + 'architecture', 'versions', 'url_parse', 'url_substitution', 'packages', + 'stage', 'spec_syntax', 'spec_semantics', 'spec_dag', 'concretize', + 'multimethod', 'install', 'package_sanity', 'config', 'directory_layout', + 'pattern', 'python_version', 'git_fetch', 'svn_fetch', 'hg_fetch', + 'mirror', 'modules', 'url_extrapolate', 'cc', 'link_tree', 'spec_yaml', + 'optional_deps', 'make_executable', 'build_system_guess', 'lock', + 'database', 'namespace_trie', 'yaml', 'sbang', 'environment', 'cmd.find', + 'cmd.uninstall', 'cmd.test_install', 'cmd.test_compiler_cmd', + 'cmd.module' +] def list_tests(): diff --git a/lib/spack/spack/test/architecture.py b/lib/spack/spack/test/architecture.py index ae3f08deed..09bdb021af 100644 --- a/lib/spack/spack/test/architecture.py +++ b/lib/spack/spack/test/architecture.py @@ -1,7 +1,31 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## """ Test checks if the architecture class is created correctly and also that the functions are looking for the correct architecture name """ -import unittest +import itertools import os import platform as py_platform import spack @@ -14,9 +38,8 @@ from spack.platforms.darwin import Darwin from spack.test.mock_packages_test import * -#class ArchitectureTest(unittest.TestCase): -class ArchitectureTest(MockPackagesTest): +class ArchitectureTest(MockPackagesTest): def setUp(self): super(ArchitectureTest, self).setUp() self.platform = spack.architecture.platform() @@ -36,24 +59,22 @@ class ArchitectureTest(MockPackagesTest): self.assertEqual(arch, new_arch) - self.assertTrue( isinstance(arch, spack.architecture.Arch) ) - self.assertTrue( isinstance(arch.platform, spack.architecture.Platform) ) - self.assertTrue( isinstance(arch.platform_os, - spack.architecture.OperatingSystem) ) - self.assertTrue( isinstance(arch.target, - spack.architecture.Target) ) - self.assertTrue( isinstance(new_arch, spack.architecture.Arch) ) - self.assertTrue( isinstance(new_arch.platform, - spack.architecture.Platform) ) - self.assertTrue( isinstance(new_arch.platform_os, - spack.architecture.OperatingSystem) ) - self.assertTrue( isinstance(new_arch.target, - spack.architecture.Target) ) - + self.assertTrue(isinstance(arch, spack.architecture.Arch)) + self.assertTrue(isinstance(arch.platform, spack.architecture.Platform)) + self.assertTrue(isinstance(arch.platform_os, + spack.architecture.OperatingSystem)) + self.assertTrue(isinstance(arch.target, + spack.architecture.Target)) + self.assertTrue(isinstance(new_arch, spack.architecture.Arch)) + self.assertTrue(isinstance(new_arch.platform, + spack.architecture.Platform)) + self.assertTrue(isinstance(new_arch.platform_os, + spack.architecture.OperatingSystem)) + self.assertTrue(isinstance(new_arch.target, + spack.architecture.Target)) def test_platform(self): output_platform_class = spack.architecture.platform() - my_arch_class = None if os.path.exists('/opt/cray/craype'): my_platform_class = CrayXc() elif os.path.exists('/bgsys'): @@ -91,7 +112,7 @@ class ArchitectureTest(MockPackagesTest): default_os = self.platform.operating_system("default_os") default_target = self.platform.target("default_target") - default_spec = Spec("libelf") # default is no args + default_spec = Spec("libelf") # default is no args default_spec.concretize() self.assertEqual(default_os, default_spec.architecture.platform_os) self.assertEqual(default_target, default_spec.architecture.target) @@ -107,10 +128,11 @@ class ArchitectureTest(MockPackagesTest): combinations = itertools.product(os_list, target_list) results = [] for arch in combinations: - o,t = arch + o, t = arch spec = Spec("libelf os=%s target=%s" % (o, t)) spec.concretize() - results.append(spec.architecture.platform_os == self.platform.operating_system(o)) + results.append(spec.architecture.platform_os == + self.platform.operating_system(o)) results.append(spec.architecture.target == self.platform.target(t)) res = all(results) diff --git a/lib/spack/spack/test/configure_guess.py b/lib/spack/spack/test/build_system_guess.py index bad3673e7a..e728a47cf4 100644 --- a/lib/spack/spack/test/configure_guess.py +++ b/lib/spack/spack/test/build_system_guess.py @@ -28,14 +28,14 @@ import tempfile import unittest from llnl.util.filesystem import * -from spack.cmd.create import ConfigureGuesser +from spack.cmd.create import BuildSystemGuesser from spack.stage import Stage from spack.test.mock_packages_test import * from spack.util.executable import which class InstallTest(unittest.TestCase): - """Tests the configure guesser in spack create""" + """Tests the build system guesser in spack create""" def setUp(self): self.tar = which('tar') @@ -44,12 +44,10 @@ class InstallTest(unittest.TestCase): os.chdir(self.tmpdir) self.stage = None - def tearDown(self): shutil.rmtree(self.tmpdir, ignore_errors=True) os.chdir(self.orig_dir) - def check_archive(self, filename, system): mkdirp('archive') touch(join_path('archive', filename)) @@ -60,24 +58,24 @@ class InstallTest(unittest.TestCase): with Stage(url) as stage: stage.fetch() - guesser = ConfigureGuesser() - guesser(stage) + guesser = BuildSystemGuesser() + guesser(stage, url) self.assertEqual(system, guesser.build_system) - - def test_python(self): - self.check_archive('setup.py', 'python') - - def test_autotools(self): self.check_archive('configure', 'autotools') - def test_cmake(self): self.check_archive('CMakeLists.txt', 'cmake') + def test_scons(self): + self.check_archive('SConstruct', 'scons') - def test_unknown(self): - self.check_archive('foobar', 'unknown') + def test_python(self): + self.check_archive('setup.py', 'python') + def test_R(self): + self.check_archive('NAMESPACE', 'R') + def test_unknown(self): + self.check_archive('foobar', 'unknown') diff --git a/lib/spack/spack/test/cmd/find.py b/lib/spack/spack/test/cmd/find.py index 371e9650e0..fa82db7733 100644 --- a/lib/spack/spack/test/cmd/find.py +++ b/lib/spack/spack/test/cmd/find.py @@ -27,11 +27,7 @@ import spack.cmd.find import unittest - -class Bunch(object): - - def __init__(self, **kwargs): - self.__dict__.update(kwargs) +from spack.util.pattern import Bunch class FindTest(unittest.TestCase): diff --git a/lib/spack/spack/test/cmd/module.py b/lib/spack/spack/test/cmd/module.py new file mode 100644 index 0000000000..36a4a73fe6 --- /dev/null +++ b/lib/spack/spack/test/cmd/module.py @@ -0,0 +1,83 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## +import argparse +import os.path + +import spack.cmd.module as module +import spack.modules as modules +import spack.test.mock_database + + +class TestModule(spack.test.mock_database.MockDatabase): + + def _get_module_files(self, args): + return [ + modules.module_types[args.module_type](spec).file_name for spec in args.specs # NOQA: ignore=E501 + ] + + def test_module_common_operations(self): + parser = argparse.ArgumentParser() + module.setup_parser(parser) + # Try to remove a non existing module [tcl] + args = parser.parse_args(['rm', 'doesnotexist']) + self.assertRaises(SystemExit, module.module, parser, args) + # Remove existing modules [tcl] + args = parser.parse_args(['rm', '-y', 'mpileaks']) + module_files = self._get_module_files(args) + for item in module_files: + self.assertTrue(os.path.exists(item)) + module.module(parser, args) + for item in module_files: + self.assertFalse(os.path.exists(item)) + # Add them back [tcl] + args = parser.parse_args(['refresh', '-y', 'mpileaks']) + module.module(parser, args) + for item in module_files: + self.assertTrue(os.path.exists(item)) + # TODO : test the --delete-tree option + # TODO : this requires having a separate directory for test modules + # Try to find a module with multiple matches + args = parser.parse_args(['find', 'mpileaks']) + self.assertRaises(SystemExit, module.module, parser, args) + # Try to find a module with no matches + args = parser.parse_args(['find', 'doesnotexist']) + self.assertRaises(SystemExit, module.module, parser, args) + # Try to find a module + args = parser.parse_args(['find', 'libelf']) + module.module(parser, args) + # Remove existing modules [dotkit] + args = parser.parse_args(['rm', '-y', '-m', 'dotkit', 'mpileaks']) + module_files = self._get_module_files(args) + for item in module_files: + self.assertTrue(os.path.exists(item)) + module.module(parser, args) + for item in module_files: + self.assertFalse(os.path.exists(item)) + # Add them back [dotkit] + args = parser.parse_args(['refresh', '-y', '-m', 'dotkit', 'mpileaks']) + module.module(parser, args) + for item in module_files: + self.assertTrue(os.path.exists(item)) + # TODO : add tests for loads and find to check the prompt format diff --git a/lib/spack/spack/test/cmd/test_install.py b/lib/spack/spack/test/cmd/test_install.py index d17e013ed2..a94d3c8bba 100644 --- a/lib/spack/spack/test/cmd/test_install.py +++ b/lib/spack/spack/test/cmd/test_install.py @@ -58,16 +58,39 @@ test_install = __import__("spack.cmd.test-install", fromlist=['test_install']) class MockSpec(object): def __init__(self, name, version, hashStr=None): - self.dependencies = {} + self._dependencies = {} self.name = name self.version = version self.hash = hashStr if hashStr else hash((name, version)) + def _deptype_norm(self, deptype): + if deptype is None: + return spack.alldeps + # Force deptype to be a tuple so that we can do set intersections. + if isinstance(deptype, str): + return (deptype,) + return deptype + + def _find_deps(self, where, deptype): + deptype = self._deptype_norm(deptype) + + return [dep.spec + for dep in where.values() + if deptype and any(d in deptype for d in dep.deptypes)] + + def dependencies(self, deptype=None): + return self._find_deps(self._dependencies, deptype) + + def dependents(self, deptype=None): + return self._find_deps(self._dependents, deptype) + def traverse(self, order=None): - for _, spec in self.dependencies.items(): - yield spec + for _, spec in self._dependencies.items(): + yield spec.spec yield self - #allDeps = itertools.chain.from_iterable(i.traverse() for i in self.dependencies.itervalues()) + #from_iterable = itertools.chain.from_iterable + #allDeps = from_iterable(i.traverse() + # for i in self.dependencies()) #return set(itertools.chain([self], allDeps)) def dag_hash(self): @@ -104,7 +127,7 @@ def mock_fetch_log(path): specX = MockSpec('X', "1.2.0") specY = MockSpec('Y', "2.3.8") -specX.dependencies['Y'] = specY +specX._dependencies['Y'] = spack.DependencySpec(specY, spack.alldeps) pkgX = MockPackage(specX, 'logX') pkgY = MockPackage(specY, 'logY') diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py index ce02b08bc3..ae3ceecfc8 100644 --- a/lib/spack/spack/test/concretize.py +++ b/lib/spack/spack/test/concretize.py @@ -197,32 +197,36 @@ class ConcretizeTest(MockPackagesTest): def test_virtual_is_fully_expanded_for_callpath(self): # force dependence on fake "zmpi" by asking for MPI 10.0 spec = Spec('callpath ^mpi@10.0') - self.assertTrue('mpi' in spec.dependencies) + self.assertTrue('mpi' in spec._dependencies) self.assertFalse('fake' in spec) spec.concretize() - self.assertTrue('zmpi' in spec.dependencies) - self.assertTrue(all(not 'mpi' in d.dependencies for d in spec.traverse())) + self.assertTrue('zmpi' in spec._dependencies) + self.assertTrue(all('mpi' not in d._dependencies + for d in spec.traverse())) self.assertTrue('zmpi' in spec) self.assertTrue('mpi' in spec) - self.assertTrue('fake' in spec.dependencies['zmpi']) + self.assertTrue('fake' in spec._dependencies['zmpi'].spec) def test_virtual_is_fully_expanded_for_mpileaks(self): spec = Spec('mpileaks ^mpi@10.0') - self.assertTrue('mpi' in spec.dependencies) + self.assertTrue('mpi' in spec._dependencies) self.assertFalse('fake' in spec) spec.concretize() - self.assertTrue('zmpi' in spec.dependencies) - self.assertTrue('callpath' in spec.dependencies) - self.assertTrue('zmpi' in spec.dependencies['callpath'].dependencies) - self.assertTrue('fake' in spec.dependencies['callpath'].dependencies['zmpi'].dependencies) + self.assertTrue('zmpi' in spec._dependencies) + self.assertTrue('callpath' in spec._dependencies) + self.assertTrue('zmpi' in spec._dependencies['callpath']. + spec._dependencies) + self.assertTrue('fake' in spec._dependencies['callpath']. + spec._dependencies['zmpi']. + spec._dependencies) - self.assertTrue(all(not 'mpi' in d.dependencies for d in spec.traverse())) + self.assertTrue(all(not 'mpi' in d._dependencies for d in spec.traverse())) self.assertTrue('zmpi' in spec) self.assertTrue('mpi' in spec) diff --git a/lib/spack/spack/test/mock_packages_test.py b/lib/spack/spack/test/mock_packages_test.py index c8b06cd7d7..9d96622a6e 100644 --- a/lib/spack/spack/test/mock_packages_test.py +++ b/lib/spack/spack/test/mock_packages_test.py @@ -191,8 +191,7 @@ class MockPackagesTest(unittest.TestCase): # restore later. self.saved_deps = {} - - def set_pkg_dep(self, pkg_name, spec): + def set_pkg_dep(self, pkg_name, spec, deptypes=spack.alldeps): """Alters dependence information for a package. Adds a dependency on <spec> to pkg. @@ -206,7 +205,9 @@ class MockPackagesTest(unittest.TestCase): self.saved_deps[pkg_name] = (pkg, pkg.dependencies.copy()) # Change dep spec - pkg.dependencies[spec.name] = { Spec(pkg_name) : spec } + # XXX(deptype): handle deptypes. + pkg.dependencies[spec.name] = {Spec(pkg_name): spec} + pkg._deptypes[spec.name] = set(deptypes) def cleanmock(self): @@ -216,6 +217,7 @@ class MockPackagesTest(unittest.TestCase): shutil.rmtree(self.temp_config, ignore_errors=True) spack.config.clear_config_caches() + # XXX(deptype): handle deptypes. # Restore dependency changes that happened during the test for pkg_name, (pkg, deps) in self.saved_deps.items(): pkg.dependencies.clear() diff --git a/lib/spack/spack/test/mock_repo.py b/lib/spack/spack/test/mock_repo.py index a8098b8eec..386af282e7 100644 --- a/lib/spack/spack/test/mock_repo.py +++ b/lib/spack/spack/test/mock_repo.py @@ -103,6 +103,8 @@ class MockGitRepo(MockVCSRepo): def __init__(self): super(MockGitRepo, self).__init__('mock-git-stage', 'mock-git-repo') + self.url = 'file://' + self.path + with working_dir(self.path): git('init') @@ -140,8 +142,6 @@ class MockGitRepo(MockVCSRepo): self.r1 = self.rev_hash(self.branch) self.r1_file = self.branch_file - self.url = self.path - def rev_hash(self, rev): return git('rev-parse', rev, output=str).strip() diff --git a/lib/spack/spack/test/modules.py b/lib/spack/spack/test/modules.py index 6d2e3705bd..135cd028e3 100644 --- a/lib/spack/spack/test/modules.py +++ b/lib/spack/spack/test/modules.py @@ -27,7 +27,6 @@ from contextlib import contextmanager import StringIO import spack.modules -import unittest from spack.test.mock_packages_test import MockPackagesTest FILE_REGISTRY = collections.defaultdict(StringIO.StringIO) @@ -266,7 +265,7 @@ class TclTests(MockPackagesTest): def test_blacklist(self): spack.modules.CONFIGURATION = configuration_blacklist - spec = spack.spec.Spec('mpileaks') + spec = spack.spec.Spec('mpileaks ^zmpi') content = self.get_modulefile_content(spec) self.assertEqual(len([x for x in content if 'is-loaded' in x]), 1) self.assertEqual(len([x for x in content if 'module load ' in x]), 1) diff --git a/lib/spack/spack/test/spec_dag.py b/lib/spack/spack/test/spec_dag.py index c56c70b1fe..972e79aa20 100644 --- a/lib/spack/spack/test/spec_dag.py +++ b/lib/spack/spack/test/spec_dag.py @@ -148,10 +148,12 @@ class SpecDagTest(MockPackagesTest): # Normalize then add conflicting constraints to the DAG (this is an # extremely unlikely scenario, but we test for it anyway) mpileaks.normalize() - mpileaks.dependencies['mpich'] = Spec('mpich@1.0') - mpileaks.dependencies['callpath'].dependencies['mpich'] = Spec('mpich@2.0') + mpileaks._dependencies['mpich'].spec = Spec('mpich@1.0') + mpileaks._dependencies['callpath']. \ + spec._dependencies['mpich'].spec = Spec('mpich@2.0') - self.assertRaises(spack.spec.InconsistentSpecError, mpileaks.flatten) + self.assertRaises(spack.spec.InconsistentSpecError, + lambda: mpileaks.flat_dependencies(copy=False)) def test_normalize_twice(self): @@ -197,15 +199,17 @@ class SpecDagTest(MockPackagesTest): def check_links(self, spec_to_check): for spec in spec_to_check.traverse(): - for dependent in spec.dependents.values(): + for dependent in spec.dependents(): self.assertTrue( - spec.name in dependent.dependencies, - "%s not in dependencies of %s" % (spec.name, dependent.name)) + spec.name in dependent.dependencies_dict(), + "%s not in dependencies of %s" % + (spec.name, dependent.name)) - for dependency in spec.dependencies.values(): + for dependency in spec.dependencies(): self.assertTrue( - spec.name in dependency.dependents, - "%s not in dependents of %s" % (spec.name, dependency.name)) + spec.name in dependency.dependents_dict(), + "%s not in dependents of %s" % + (spec.name, dependency.name)) def test_dependents_and_dependencies_are_correct(self): @@ -442,3 +446,69 @@ class SpecDagTest(MockPackagesTest): orig_ids = set(id(s) for s in orig.traverse()) copy_ids = set(id(s) for s in copy.traverse()) self.assertFalse(orig_ids.intersection(copy_ids)) + + """ + Here is the graph with deptypes labeled (assume all packages have a 'dt' + prefix). Arrows are marked with the deptypes ('b' for 'build', 'l' for + 'link', 'r' for 'run'). + + use -bl-> top + + top -b-> build1 + top -bl-> link1 + top -r-> run1 + + build1 -b-> build2 + build1 -bl-> link2 + build1 -r-> run2 + + link1 -bl-> link3 + + run1 -bl-> link5 + run1 -r-> run3 + + link3 -b-> build2 + link3 -bl-> link4 + + run3 -b-> build3 + """ + def test_deptype_traversal(self): + dag = Spec('dtuse') + dag.normalize() + + names = ['dtuse', 'dttop', 'dtlink1', 'dtlink3', 'dtlink4', + 'dtrun1', 'dtlink5', 'dtrun3'] + + traversal = dag.traverse() + self.assertEqual([x.name for x in traversal], names) + + def test_deptype_traversal_with_builddeps(self): + dag = Spec('dttop') + dag.normalize() + + names = ['dttop', 'dtbuild1', 'dtlink2', 'dtrun2', 'dtlink1', + 'dtlink3', 'dtlink4', 'dtrun1', 'dtlink5', 'dtrun3'] + + traversal = dag.traverse() + self.assertEqual([x.name for x in traversal], names) + + def test_deptype_traversal_full(self): + dag = Spec('dttop') + dag.normalize() + + names = ['dttop', 'dtbuild1', 'dtbuild2', 'dtlink2', 'dtrun2', + 'dtlink1', 'dtlink3', 'dtlink4', 'dtrun1', 'dtlink5', + 'dtrun3', 'dtbuild3'] + + traversal = dag.traverse(deptype_query=spack.alldeps) + self.assertEqual([x.name for x in traversal], names) + + def test_deptype_traversal_pythonpath(self): + dag = Spec('dttop') + dag.normalize() + + names = ['dttop', 'dtbuild1', 'dtrun2', 'dtlink1', 'dtrun1', + 'dtrun3'] + + traversal = dag.traverse(deptype=spack.nolink, deptype_query='run') + self.assertEqual([x.name for x in traversal], names) diff --git a/lib/spack/spack/test/stage.py b/lib/spack/spack/test/stage.py index 6d8c3ac67c..d3e3bf1383 100644 --- a/lib/spack/spack/test/stage.py +++ b/lib/spack/spack/test/stage.py @@ -35,8 +35,8 @@ from llnl.util.filesystem import * from spack.stage import Stage from spack.util.executable import which -test_files_dir = join_path(spack.stage_path, '.test') -test_tmp_path = join_path(test_files_dir, 'tmp') +test_files_dir = os.path.realpath(join_path(spack.stage_path, '.test')) +test_tmp_path = os.path.realpath(join_path(test_files_dir, 'tmp')) archive_dir = 'test-files' archive_name = archive_dir + '.tar.gz' diff --git a/lib/spack/spack/util/pattern.py b/lib/spack/spack/util/pattern.py index 6d4bcb1039..bc5e9d2ffe 100644 --- a/lib/spack/spack/util/pattern.py +++ b/lib/spack/spack/util/pattern.py @@ -28,42 +28,50 @@ import functools def composite(interface=None, method_list=None, container=list): - """ - Returns a class decorator that patches a class adding all the methods it needs to be a composite for a given - interface. + """Returns a class decorator that patches a class adding all the methods + it needs to be a composite for a given interface. - :param interface: class exposing the interface to which the composite object must conform. Only non-private and - non-special methods will be taken into account + :param interface: class exposing the interface to which the composite + object must conform. Only non-private and non-special methods will be + taken into account :param method_list: names of methods that should be part of the composite - :param container: container for the composite object (default = list). Must fulfill the MutableSequence contract. - The composite class will expose the container API to manage object composition + :param container: container for the composite object (default = list). + Must fulfill the MutableSequence contract. The composite class will expose + the container API to manage object composition :return: class decorator """ - # Check if container fulfills the MutableSequence contract and raise an exception if it doesn't - # The patched class returned by the decorator will inherit from the container class to expose the - # interface needed to manage objects composition + # Check if container fulfills the MutableSequence contract and raise an + # exception if it doesn't. The patched class returned by the decorator will + # inherit from the container class to expose the interface needed to manage + # objects composition if not issubclass(container, collections.MutableSequence): raise TypeError("Container must fulfill the MutableSequence contract") - # Check if at least one of the 'interface' or the 'method_list' arguments are defined + # Check if at least one of the 'interface' or the 'method_list' arguments + # are defined if interface is None and method_list is None: - raise TypeError("Either 'interface' or 'method_list' must be defined on a call to composite") + raise TypeError("Either 'interface' or 'method_list' must be defined on a call to composite") # NOQA : ignore=E501 def cls_decorator(cls): - # Retrieve the base class of the composite. Inspect its methods and decide which ones will be overridden + # Retrieve the base class of the composite. Inspect its methods and + # decide which ones will be overridden def no_special_no_private(x): return inspect.ismethod(x) and not x.__name__.startswith('_') - # Patch the behavior of each of the methods in the previous list. This is done associating an instance of the - # descriptor below to any method that needs to be patched. + # Patch the behavior of each of the methods in the previous list. + # This is done associating an instance of the descriptor below to + # any method that needs to be patched. class IterateOver(object): + """Decorator used to patch methods in a composite. + + It iterates over all the items in the instance containing the + associated attribute and calls for each of them an attribute + with the same name """ - Decorator used to patch methods in a composite. It iterates over all the items in the instance containing the - associated attribute and calls for each of them an attribute with the same name - """ + def __init__(self, name, func=None): self.name = name self.func = func @@ -72,8 +80,9 @@ def composite(interface=None, method_list=None, container=list): def getter(*args, **kwargs): for item in instance: getattr(item, self.name)(*args, **kwargs) - # If we are using this descriptor to wrap a method from an interface, then we must conditionally - # use the `functools.wraps` decorator to set the appropriate fields. + # If we are using this descriptor to wrap a method from an + # interface, then we must conditionally use the + # `functools.wraps` decorator to set the appropriate fields if self.func is not None: getter = functools.wraps(self.func)(getter) return getter @@ -81,7 +90,8 @@ def composite(interface=None, method_list=None, container=list): dictionary_for_type_call = {} # Construct a dictionary with the methods explicitly passed as name if method_list is not None: - # python@2.7: method_list_dict = {name: IterateOver(name) for name in method_list} + # python@2.7: method_list_dict = {name: IterateOver(name) for name + # in method_list} method_list_dict = {} for name in method_list: method_list_dict[name] = IterateOver(name) @@ -89,28 +99,40 @@ def composite(interface=None, method_list=None, container=list): # Construct a dictionary with the methods inspected from the interface if interface is not None: ########## - # python@2.7: interface_methods = {name: method for name, method in inspect.getmembers(interface, predicate=no_special_no_private)} + # python@2.7: interface_methods = {name: method for name, method in + # inspect.getmembers(interface, predicate=no_special_no_private)} interface_methods = {} - for name, method in inspect.getmembers(interface, predicate=no_special_no_private): + for name, method in inspect.getmembers(interface, predicate=no_special_no_private): # NOQA: ignore=E501 interface_methods[name] = method ########## - # python@2.7: interface_methods_dict = {name: IterateOver(name, method) for name, method in interface_methods.iteritems()} + # python@2.7: interface_methods_dict = {name: IterateOver(name, + # method) for name, method in interface_methods.iteritems()} interface_methods_dict = {} for name, method in interface_methods.iteritems(): interface_methods_dict[name] = IterateOver(name, method) ########## dictionary_for_type_call.update(interface_methods_dict) - # Get the methods that are defined in the scope of the composite class and override any previous definition + # Get the methods that are defined in the scope of the composite + # class and override any previous definition ########## - # python@2.7: cls_method = {name: method for name, method in inspect.getmembers(cls, predicate=inspect.ismethod)} + # python@2.7: cls_method = {name: method for name, method in + # inspect.getmembers(cls, predicate=inspect.ismethod)} cls_method = {} - for name, method in inspect.getmembers(cls, predicate=inspect.ismethod): + for name, method in inspect.getmembers(cls, predicate=inspect.ismethod): # NOQA: ignore=E501 cls_method[name] = method ########## dictionary_for_type_call.update(cls_method) # Generate the new class on the fly and return it # FIXME : inherit from interface if we start to use ABC classes? - wrapper_class = type(cls.__name__, (cls, container), dictionary_for_type_call) + wrapper_class = type(cls.__name__, (cls, container), + dictionary_for_type_call) return wrapper_class return cls_decorator + + +class Bunch(object): + """Carries a bunch of named attributes (from Alex Martelli bunch)""" + + def __init__(self, **kwargs): + self.__dict__.update(kwargs) |