summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2019-04-11Updated Sphinx configuration (#11165)Massimiliano Culpo1-3/+3
2019-04-11intel: optionally take gcc executable from cflags (#11136)Denis Davydov1-3/+15
2019-04-10Maintain a view for an environment (#10017)Peter Scheibel7-114/+569
Environments are nowm by default, created with views. When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell. Example: ``` spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env spack env create e1 --with-view=/abs/path/to/anywhere spack env create e1 --without-view ``` The `spack.yaml` manifest file now looks like this: ``` spack: specs: - python view: true #or false, or a string ``` These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file: ``` spack env view enable spack env view envable /abs/path/to/anywhere spack env view disable ``` Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`: ``` spack env activate e1 spack add X spack install Y # immediately concretizes and installs Y and Z' spack install # concretizes X and Z ``` In this case `Z'` will be favored over `Z`. Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order. Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
2019-04-10Unload altd and darshan (#11162)Mario Melara1-1/+1
2019-04-10Add documentation for chaining Spack instances (#11152)Peter Scheibel3-0/+102
Add documentation for the Spack chain feature added in #8772
2019-04-09Typo: Unkown -> Unknown (#11150)George Hartzell1-1/+1
2019-04-09intel: consolidate libs() in the base class (#11143)Denis Davydov1-3/+10
* intel: consolidate libs() in the base class * white space * flake8
2019-04-06Allow spack install to take either cdash stamp or track (#11106)Scott Wittenburg3-4/+44
When providing a track, the cdash reporter will format the stamp itself, as it has always done, and register the build during the package installation process. When providing a stamp, it should first be formatted as cdash expects, and then cdash will be sure to report results to same build id which was registered manually elsewhere.
2019-04-05Fix reading externals from old databases (#11118)Greg Becker4-2/+36
* Update Spec.prefix to have special case for 'None' in database path; regression test * Update in database reader rather than spec * Change assertion to conditional + raise * Added test for concrete check in Spec.prefix
2019-04-05Fix module_parsing test (#11087)Michael Kuhn1-0/+3
The module_parsing test checks whether the module function is available by looking for the string 'not found'. If the user has set a different locale, the test can assume that the module function is available when it actually is not.
2019-04-02Fix directory layout using namespace (#11076)Tim Fuller2-1/+8
Directory layouts using ${NAMESPACE} were broken. This addresses that and adds a test to prevent regression.
2019-04-02spack repo add: fix error message when `packages' directory is missing (#11031)Tristan Carel1-1/+1
2019-04-02Added tests on compiler's version detection (#10931)Massimiliano Culpo12-258/+200
* Split get_compiler_version into two functions: get_compiler_version_output runs the compiler with the relevant option to print the version; extract_version_from_output determines the version by examining this output. This makes it easier to test the customized version detection for each compiler. Users can customize this by overriding the following: * version_argument: this is the argument that tells the compiler to print its version. It assumes that the compiler will report its version if invoked with a single option (like "--version") * version_regex: the regular expression used to extract the version from the compiler argument. This assumes that a regular expression is sufficient to extract the version, and that the version can be extracted from a single capture group (Spack uses the first capture group) * default_version: allows you to completely override all version detection logic * get_compiler_version_output: if getting the compiler to report its version is more complex than invoking it with a single arg * extract_version_from_output: if it is difficult to define a regex that can be used to extract the version from the output * Added tests for version detection of most compilers * Removed redundant code from xl_r compiler class (by inheriting from xl compiler definition)
2019-03-29Update llnl.util.lang.memoized so that Sphinx can extract signature (#11055)Massimiliano Culpo1-18/+12
Replace the original implementation of the "memoized" decorator with an implementation that exposes the docstring and arguments of the wrapped function. This is achieved using functools.wraps.
2019-03-29add libs property to IntelMkl and IntelParallelStudio (#10993)Denis Davydov1-3/+4
* add libs property to IntelMkl and IntelParallelStudio * fix scalapack_libs when MPI is provided by intel-parallel-studio
2019-03-28Spack can be extended with external commands (#8612)Massimiliano Culpo8-19/+225
This provides a mechanism to implement a new Spack command in a separate directory, and with a small configuration change point Spack to the new command. To register the command, the directory must be added to the "extensions" section of config.yaml. The command directory name must have the prefix "spack-", and have the following layout: spack-X/ pytest.ini #optional, for testing X/ cmd/ name-of-command1.py name-of-command2.py ... tests/ #optional conftest.py test_name-of-command1.py templates/ #optional jinja templates, if needed And in config.yaml: config: extensions: - /path/to/spack-X If the extension includes tests, you can run them via spack by adding the --extension option, like "spack test --extension=X"
2019-03-28Consistent patch ordering (#10879)Peter Scheibel4-23/+76
* preserve the order in which patches are applied by packages (in spite of grouping them by 'when') * add tests confirming patch order
2019-03-27spack chain (#8772)Peter Scheibel18-165/+769
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed * need to query upstream dbs when adding new record to local db * prevent reindexing upstream DBs * set prefix on specs read from DB based on path stored in install record * check that Spack does not install packages that are recorded as installed in an upstream db * externals do not add their path to install records - need to use 'external_path' to get path of upstream externals * views need to check for upstream installations when linking metadata * package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated * only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream * when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it * support recursive upstream databases (allow upstream databases to use their own upstream databases) * separate upstream config into separate file with its own schema; each entry now also includes a name * metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout * treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
2019-03-27Replace previous bash command to new one (#7005)Mario Melara1-16/+5
Prevents infinite recursion caused by sourcing `setup-env.sh` in bashrc
2019-03-26Move CMakePackage build directory to base stage directory (#8431)Chris Green4-32/+30
Change the location of the CMake build area from the staged source directory to the stage base directory. This change allows CMake packages to refer to the build directory in setup_environment (e.g. if tests need to have a directory in PATH): Staging happens after the call to setup_environment(), and if the stage area does not exist, then spec.stage.source_path returns None. To accommodate this change, archived files (like config.log for Autotools packages) are archived relative to the stage base directory rather than the expanded source directory. Other packages (those not using CMake) will still use the staged source directory as the default working directory for builds (and will still be unable to reference this directory in setup_environment())
2019-03-26Retrieve environment-modules prefix based on architecture (#10975)ajw19801-1/+2
When multiple instances of environment-modules were installed with different architectures, Spack was not retrieving the installation appropriate for the current architecture when finding the module prefix.
2019-03-26Use 'shlex' to split default Executable arguments (#10929)Seth R. Johnson1-1/+2
This allows shell commands for `spack edit` to be executed correctly if they have quoted arguments.
2019-03-26Add regression marker to pytest.ini (#11011)Massimiliano Culpo1-0/+1
2019-03-26Fixed some issues with CUDA-Intel compiler conflicts. (#10924)Nichols A. Romero1-6/+13
* Fixed some issues with CUDA-Intel compiler conflicts. * Comment about expressing CUDA-compiler conflicts. * More precise conflicts and also add support for Intel 19.0
2019-03-21typo in documentation (#10976)Shahzeb Siddiqui1-3/+3
2019-03-21Minor commentary correction, TCL -> lmod (#10951)George Hartzell1-1/+1
2019-03-21prevent UnboundLocalError when sourcing files (#10950)Stephen McDowell1-0/+1
Patch extracted from #7536 courtesy of @mgsternberg
2019-03-19Fix help text for --no-checksum. (#10932)Chris Green1-1/+1
2019-03-14Ensure that every package has a description (#10896)Adam J. Stewart1-0/+8
2019-03-14Support VISUAL environment variable for editing. (#10898)Chris Green1-20/+26
If the user has set the environment variable VISUAL, it will be used in preference to EDITOR for all Spack editing activities. If VISUAL is not set or fails (perhaps due to a lack of graphical editing capabilities),EDITOR will be used instead. We fall back to one of several common editors if neither bears fruit. This feature has been tailored to: * Provide identical behavior to the previous implementation in the case that VISUAL is not set. * Not require any change to code utilizing the editor feature. * Follow usual UNIX behavior concerning VISUAL and EDITOR.
2019-03-12Do not use `string` module to be compatible with python 3 (#248) (#10667)Tristan Carel1-3/+2
`string.find` is not part of Python 3 anymore.
2019-03-12find_versions_of_archive: expand link search (#10758)Justin Stanley1-1/+2
2019-03-12Fix clearing EnvironmentModifications with python2 (#10791)Nick Forrington2-1/+8
* Fix clearing EnvironmentModifications with python2 * Add EnvironmentModifications::clear unit test Use re-assignment rather than del to clear array * Fix flake issues
2019-03-11format change and typo in doc (#10848)Shahzeb Siddiqui1-2/+2
2019-03-11Improved detection of Clang versions (#10316)Massimiliano Culpo2-12/+47
Fixes #10191 * Add more regular expressions to detect clang versions that were not being picked up * Add a test for parsing versions from the output of Clang (this does not run Clang, but rather uses example outputs from Clang) * Separate Clang version parsing into its own method (to make it easier to test)
2019-03-08Fix find_headers to also look for C++ headers and Fortran modules (#10798)Michael Kuhn1-3/+6
Currently, only C headers are considered, causing build failures for packages depending on, e.g., netcdf-fortran and xerces-c. Additionally, the regex used to look for the include path component did not consider word boundaries, causing false matches.
2019-03-07Create option to build compilers as needed (#10761)Greg Becker6-65/+131
* Create option to build missing compilers and add them to config before installing packages that use them * Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
2019-03-06Make is a required dependency of Spack (#10386)Adam J. Stewart1-4/+5
Update documentation on getting started to mention that a "make" executable is required to build packages with Spack.
2019-03-01This fixes a problem where the placeholder path was not in the first rpath ↵Patrick Gartung3-105/+76
entry. * Rework of buildcache creation and install prefix checking using the functions introduced in https://github.com/spack/spack/pull/9199 Instead of replacing rpaths with placeholder and then checking strings, make use of the functions relocate.is_recocatable and relocate.is_file_relocatable to decide if a package needs the allow-root option. This fixes a problem where the placeholder path was not in the first rpath entry. This was seen in c++ libraries and binaries because the compiler was outside the spack install base path and always appears first in the rpath. Instead of checking the first rpath entry, all rpaths have the placeholder path and the old install path (if it exists) replaced with the new install path. * flake8
2019-02-28Added a sub-command to show if packages are relocatable (#9199)Massimiliano Culpo10-25/+309
* Added the `spack buildcache preview` sub-command This is similar to `spack spec -I` but highlights which nodes in a DAG are relocatable and which are not. spec.tree has been generalized a little to accept a status function, instead of always showing the install status The current implementation works only for ELF, and needs to be generalized to other platforms. * Added a test to check if an executable is relocatable or not This test requires a few commands to be present in the environment. Currently it will run only under python 3.7 (which uses Xenial instead of Trusty). * Added tests for the 'buildcache preview' command. * Fixed codebase after rebase * Fixed the list of apt addons for Python 3.7 in travis.yaml * Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229) * Fixed flake8 issues * Add handling for macOS mach binaries (#231)
2019-02-28Fix shell integration with environment-modules@4 (#10736)Michael Kuhn1-1/+1
2019-02-27Fix 'make test' detection when LANG is not in English (#10499)Adam J. Stewart1-1/+5
2019-02-27Update environment-modules package (#10717)Massimiliano Culpo1-1/+1
The environment modules package has been updated to include versions up to 4.0.0. The url of the package and the homepage have been updated accordingly. The `spack bootstrap` command now builds version 3.2.10 of the environment-modules package, and will do until #10708 is fixed.
2019-02-26Use Package.headers for -I options (#10623)Massimiliano Culpo6-25/+167
This restores the use of Package.headers when computing -I options for building a package that was added in #8136 and reverted in #10604. #8136 used utility logic that located all header files in an installation prefix, and calculated the -I options as the immediate roots containing those header files. In some cases, for a package containing a directory structure like prefix/ include/ ex1.h subdir/ ex2.h dependents may expect to include ex2.h relative to 'include', and adding 'prefix/include/subdir' as a -I was causing errors, in particular if ex2.h has the same name as a system header. This updates header utility logic to by default return the base "include" directory when it exists, rather than subdirectories. It also makes it possible for package implementers to override Package.headers to return the subdirectory when it is required (for example with libxml2).
2019-02-26Add libhio v1.4.1.3, 1.4.1.1 (#10699)Daniel Topa1-1/+1
- Switch all libhio tarball listings to sha256 checksums - Correct typo in alert message in /lib/spack/spack/util/module_cmd.py: fucntion -> function https://github.com/hpc/libhio/releases Released 2019-02-01 Verification builds on LANL Darwin: **Intel Xeon** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 40 On-line CPU(s) list: 0-39 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz Stepping: 2 CPU MHz: 1198.779 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 5193.70 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-9,20-29 NUMA node1 CPU(s): 10-19,30-39 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts ` [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.3-s4fnmesfp65trhks5qi3it5p73ssfpsp [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.2-fkgh5vqpijvwqywffmokgmsglqxwfrtl ` **Arm** Architecture: aarch64 Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 4 Core(s) per socket: 32 Socket(s): 2 NUMA node(s): 2 Model: 0 BogoMIPS: 400.00 L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 32768K NUMA node0 CPU(s): 0-127 NUMA node1 CPU(s): 128-255 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm ` [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.3-q6nnwiy6bi7ktnghdsngwamom23zpmgy [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.2-y6nwovff3qbdy242zc4x2toloz6xpcvm ` 2019-02-25 Signed-off-by: Daniel Topa <dantopa@lanl.gov>
2019-02-25Add /hash to spack help --spec (#10691)Adam J. Stewart1-1/+5
* Add /hash to spack help --spec * Make /hash gray, add ^/hash
2019-02-21CPATH warning: downgrade to debug message (#10675)Peter Scheibel1-3/+3
Spack warns users when a dependency package updates CPATH. This warning message is generating bug reports and alarm in cases where there is no problem. For now this downgrades the warning message to the debug level, so it only shows up if something goes wrong for the user and they ask for more information from Spack.
2019-02-21release workflow: Add build scripts for jobs and means to upload pkgsScott Wittenburg1-0/+212
2019-02-21release workflow: Add spack command for generating the .gitlab-ci.ymlScott Wittenburg4-0/+823
This spack command adds a new schema for a file which describes the builder containers available, along with the compilers availabe on each builder. The release-jobs command then generates the .gitlab-ci.yml file by first expanding the release spec set, concretizing each spec (in an appropriate docker container if --this-machine-only argument is not provided on command line), and then combining and staging all the concrete specs as jobs to be run by gitlab.
2019-02-21buildcache: Add sub-commands needed by release workflowScott Wittenburg5-79/+599
Adds four new sub-commands to the buildcache command: 1. save-yaml: Takes a root spec and a list of dependent spec names, along with a directory in which to save yaml files, and writes out the full spec.yaml for each of the dependent specs. This only needs to concretize the root spec once, then indexes it with the names of the dependent specs. 2. check: Checks a spec (via either an abstract spec or via a full spec.yaml) against remote mirror to see if it needs to be rebuilt. Comparies full_hash stored on remote mirror with full_hash computed locally to determine whether spec needs to be rebuilt. Can also generate list of specs to check against remote mirror by expanding the set of release specs expressed in etc/spack/defaults/release.yaml. 3. get-buildcache-name: Makes it possible to attempt to read directly the spec.yaml file on a remote or local mirror by providing the path where the file should live based on concretizing the spec. 4. download: Downloads all buildcache files associated with a spec on a remote mirror, including any .spack, .spec, and .cdashid files that might exist. Puts the files into the local path provided on the command line, and organizes them in the same hierarchy found on the remote mirror This commit also refactors lib/spack/spack/util/web.py to expose functionality allowing other modules to read data from a url.