summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2019-03-28Consistent patch ordering (#10879)Peter Scheibel4-23/+76
* preserve the order in which patches are applied by packages (in spite of grouping them by 'when') * add tests confirming patch order
2019-03-27spack chain (#8772)Peter Scheibel18-165/+769
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed * need to query upstream dbs when adding new record to local db * prevent reindexing upstream DBs * set prefix on specs read from DB based on path stored in install record * check that Spack does not install packages that are recorded as installed in an upstream db * externals do not add their path to install records - need to use 'external_path' to get path of upstream externals * views need to check for upstream installations when linking metadata * package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated * only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream * when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it * support recursive upstream databases (allow upstream databases to use their own upstream databases) * separate upstream config into separate file with its own schema; each entry now also includes a name * metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout * treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
2019-03-27Replace previous bash command to new one (#7005)Mario Melara1-16/+5
Prevents infinite recursion caused by sourcing `setup-env.sh` in bashrc
2019-03-26Move CMakePackage build directory to base stage directory (#8431)Chris Green4-32/+30
Change the location of the CMake build area from the staged source directory to the stage base directory. This change allows CMake packages to refer to the build directory in setup_environment (e.g. if tests need to have a directory in PATH): Staging happens after the call to setup_environment(), and if the stage area does not exist, then spec.stage.source_path returns None. To accommodate this change, archived files (like config.log for Autotools packages) are archived relative to the stage base directory rather than the expanded source directory. Other packages (those not using CMake) will still use the staged source directory as the default working directory for builds (and will still be unable to reference this directory in setup_environment())
2019-03-26Retrieve environment-modules prefix based on architecture (#10975)ajw19801-1/+2
When multiple instances of environment-modules were installed with different architectures, Spack was not retrieving the installation appropriate for the current architecture when finding the module prefix.
2019-03-26Use 'shlex' to split default Executable arguments (#10929)Seth R. Johnson1-1/+2
This allows shell commands for `spack edit` to be executed correctly if they have quoted arguments.
2019-03-26Add regression marker to pytest.ini (#11011)Massimiliano Culpo1-0/+1
2019-03-26Fixed some issues with CUDA-Intel compiler conflicts. (#10924)Nichols A. Romero1-6/+13
* Fixed some issues with CUDA-Intel compiler conflicts. * Comment about expressing CUDA-compiler conflicts. * More precise conflicts and also add support for Intel 19.0
2019-03-21typo in documentation (#10976)Shahzeb Siddiqui1-3/+3
2019-03-21Minor commentary correction, TCL -> lmod (#10951)George Hartzell1-1/+1
2019-03-21prevent UnboundLocalError when sourcing files (#10950)Stephen McDowell1-0/+1
Patch extracted from #7536 courtesy of @mgsternberg
2019-03-19Fix help text for --no-checksum. (#10932)Chris Green1-1/+1
2019-03-14Ensure that every package has a description (#10896)Adam J. Stewart1-0/+8
2019-03-14Support VISUAL environment variable for editing. (#10898)Chris Green1-20/+26
If the user has set the environment variable VISUAL, it will be used in preference to EDITOR for all Spack editing activities. If VISUAL is not set or fails (perhaps due to a lack of graphical editing capabilities),EDITOR will be used instead. We fall back to one of several common editors if neither bears fruit. This feature has been tailored to: * Provide identical behavior to the previous implementation in the case that VISUAL is not set. * Not require any change to code utilizing the editor feature. * Follow usual UNIX behavior concerning VISUAL and EDITOR.
2019-03-12Do not use `string` module to be compatible with python 3 (#248) (#10667)Tristan Carel1-3/+2
`string.find` is not part of Python 3 anymore.
2019-03-12find_versions_of_archive: expand link search (#10758)Justin Stanley1-1/+2
2019-03-12Fix clearing EnvironmentModifications with python2 (#10791)Nick Forrington2-1/+8
* Fix clearing EnvironmentModifications with python2 * Add EnvironmentModifications::clear unit test Use re-assignment rather than del to clear array * Fix flake issues
2019-03-11format change and typo in doc (#10848)Shahzeb Siddiqui1-2/+2
2019-03-11Improved detection of Clang versions (#10316)Massimiliano Culpo2-12/+47
Fixes #10191 * Add more regular expressions to detect clang versions that were not being picked up * Add a test for parsing versions from the output of Clang (this does not run Clang, but rather uses example outputs from Clang) * Separate Clang version parsing into its own method (to make it easier to test)
2019-03-08Fix find_headers to also look for C++ headers and Fortran modules (#10798)Michael Kuhn1-3/+6
Currently, only C headers are considered, causing build failures for packages depending on, e.g., netcdf-fortran and xerces-c. Additionally, the regex used to look for the include path component did not consider word boundaries, causing false matches.
2019-03-07Create option to build compilers as needed (#10761)Greg Becker6-65/+131
* Create option to build missing compilers and add them to config before installing packages that use them * Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
2019-03-06Make is a required dependency of Spack (#10386)Adam J. Stewart1-4/+5
Update documentation on getting started to mention that a "make" executable is required to build packages with Spack.
2019-03-01This fixes a problem where the placeholder path was not in the first rpath ↵Patrick Gartung3-105/+76
entry. * Rework of buildcache creation and install prefix checking using the functions introduced in https://github.com/spack/spack/pull/9199 Instead of replacing rpaths with placeholder and then checking strings, make use of the functions relocate.is_recocatable and relocate.is_file_relocatable to decide if a package needs the allow-root option. This fixes a problem where the placeholder path was not in the first rpath entry. This was seen in c++ libraries and binaries because the compiler was outside the spack install base path and always appears first in the rpath. Instead of checking the first rpath entry, all rpaths have the placeholder path and the old install path (if it exists) replaced with the new install path. * flake8
2019-02-28Added a sub-command to show if packages are relocatable (#9199)Massimiliano Culpo10-25/+309
* Added the `spack buildcache preview` sub-command This is similar to `spack spec -I` but highlights which nodes in a DAG are relocatable and which are not. spec.tree has been generalized a little to accept a status function, instead of always showing the install status The current implementation works only for ELF, and needs to be generalized to other platforms. * Added a test to check if an executable is relocatable or not This test requires a few commands to be present in the environment. Currently it will run only under python 3.7 (which uses Xenial instead of Trusty). * Added tests for the 'buildcache preview' command. * Fixed codebase after rebase * Fixed the list of apt addons for Python 3.7 in travis.yaml * Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229) * Fixed flake8 issues * Add handling for macOS mach binaries (#231)
2019-02-28Fix shell integration with environment-modules@4 (#10736)Michael Kuhn1-1/+1
2019-02-27Fix 'make test' detection when LANG is not in English (#10499)Adam J. Stewart1-1/+5
2019-02-27Update environment-modules package (#10717)Massimiliano Culpo1-1/+1
The environment modules package has been updated to include versions up to 4.0.0. The url of the package and the homepage have been updated accordingly. The `spack bootstrap` command now builds version 3.2.10 of the environment-modules package, and will do until #10708 is fixed.
2019-02-26Use Package.headers for -I options (#10623)Massimiliano Culpo6-25/+167
This restores the use of Package.headers when computing -I options for building a package that was added in #8136 and reverted in #10604. #8136 used utility logic that located all header files in an installation prefix, and calculated the -I options as the immediate roots containing those header files. In some cases, for a package containing a directory structure like prefix/ include/ ex1.h subdir/ ex2.h dependents may expect to include ex2.h relative to 'include', and adding 'prefix/include/subdir' as a -I was causing errors, in particular if ex2.h has the same name as a system header. This updates header utility logic to by default return the base "include" directory when it exists, rather than subdirectories. It also makes it possible for package implementers to override Package.headers to return the subdirectory when it is required (for example with libxml2).
2019-02-26Add libhio v1.4.1.3, 1.4.1.1 (#10699)Daniel Topa1-1/+1
- Switch all libhio tarball listings to sha256 checksums - Correct typo in alert message in /lib/spack/spack/util/module_cmd.py: fucntion -> function https://github.com/hpc/libhio/releases Released 2019-02-01 Verification builds on LANL Darwin: **Intel Xeon** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 40 On-line CPU(s) list: 0-39 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz Stepping: 2 CPU MHz: 1198.779 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 5193.70 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-9,20-29 NUMA node1 CPU(s): 10-19,30-39 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts ` [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.3-s4fnmesfp65trhks5qi3it5p73ssfpsp [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.2-fkgh5vqpijvwqywffmokgmsglqxwfrtl ` **Arm** Architecture: aarch64 Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 4 Core(s) per socket: 32 Socket(s): 2 NUMA node(s): 2 Model: 0 BogoMIPS: 400.00 L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 32768K NUMA node0 CPU(s): 0-127 NUMA node1 CPU(s): 128-255 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm ` [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.3-q6nnwiy6bi7ktnghdsngwamom23zpmgy [+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.2-y6nwovff3qbdy242zc4x2toloz6xpcvm ` 2019-02-25 Signed-off-by: Daniel Topa <dantopa@lanl.gov>
2019-02-25Add /hash to spack help --spec (#10691)Adam J. Stewart1-1/+5
* Add /hash to spack help --spec * Make /hash gray, add ^/hash
2019-02-21CPATH warning: downgrade to debug message (#10675)Peter Scheibel1-3/+3
Spack warns users when a dependency package updates CPATH. This warning message is generating bug reports and alarm in cases where there is no problem. For now this downgrades the warning message to the debug level, so it only shows up if something goes wrong for the user and they ask for more information from Spack.
2019-02-21release workflow: Add build scripts for jobs and means to upload pkgsScott Wittenburg1-0/+212
2019-02-21release workflow: Add spack command for generating the .gitlab-ci.ymlScott Wittenburg4-0/+823
This spack command adds a new schema for a file which describes the builder containers available, along with the compilers availabe on each builder. The release-jobs command then generates the .gitlab-ci.yml file by first expanding the release spec set, concretizing each spec (in an appropriate docker container if --this-machine-only argument is not provided on command line), and then combining and staging all the concrete specs as jobs to be run by gitlab.
2019-02-21buildcache: Add sub-commands needed by release workflowScott Wittenburg5-79/+599
Adds four new sub-commands to the buildcache command: 1. save-yaml: Takes a root spec and a list of dependent spec names, along with a directory in which to save yaml files, and writes out the full spec.yaml for each of the dependent specs. This only needs to concretize the root spec once, then indexes it with the names of the dependent specs. 2. check: Checks a spec (via either an abstract spec or via a full spec.yaml) against remote mirror to see if it needs to be rebuilt. Comparies full_hash stored on remote mirror with full_hash computed locally to determine whether spec needs to be rebuilt. Can also generate list of specs to check against remote mirror by expanding the set of release specs expressed in etc/spack/defaults/release.yaml. 3. get-buildcache-name: Makes it possible to attempt to read directly the spec.yaml file on a remote or local mirror by providing the path where the file should live based on concretizing the spec. 4. download: Downloads all buildcache files associated with a spec on a remote mirror, including any .spack, .spec, and .cdashid files that might exist. Puts the files into the local path provided on the command line, and organizes them in the same hierarchy found on the remote mirror This commit also refactors lib/spack/spack/util/web.py to expose functionality allowing other modules to read data from a url.
2019-02-21add CombinatorialSpecSet class for taking cross-products of Specs.Todd Gamblin5-0/+623
- add CombinatorialSpecSet in spack.util.spec_set module. - class is iterable and encaspulated YAML parsing and validation. - Adjust YAML format to be more generic - YAML spec-set format now has a `matrix` section, which can contain multiple lists of specs, generated different ways. Including: - specs: a raw list of specs. - packages: a list of package names and versions - compilers: a list of compiler names and versions - All of the elements of `matrix` are dimensions for the build matrix; we take the cartesian product of these lists of specs to generate a build matrix. This means we can add things like [^mpich, ^openmpi] to get builds with different MPI versions. It also means we can multiply the build matrix out with lots of different parameters. - Add a schema format for spec-sets
2019-02-15Dependency libs: filter system paths and always add lib dir (#10622)Peter Scheibel1-7/+16
Fixes #10617 Fixes #10624 Closes: #10619 #8136 dependended entirely on spec.libs to retrieve library directories from dependencies. By default this function only retrieves libraries if their name is something like lib<package> (e.g. "libfoo.so" for a package called "Foo"). This unconditionally adds lib/lib64 directories for each dependency as link/rpath directories. This also filters system paths from link/rpaths/include directories and removes duplicated paths that #8136 could add.
2019-02-15Fix bug getting specs from build caches (#9600)Javier Cervantes1-2/+2
2019-02-14CDash: allow installing from spec.yaml (#10565)Scott Wittenburg2-1/+48
If the -f <specyamlfile> argument to install is used (rather than providing package specs on the command line), CDash throws an exception due to missing the installation command (the packages targeted for install). This fixes that behavior so CDash reporting succeeds in either case.
2019-02-14Removed pkg.headers.directories from the include list (#10604)Massimiliano Culpo3-15/+13
fixes #10601 Due to a bug this attribute is wrong for packages that use directories as namespaces. For instance it will add "<boost-prefix>/include/boost" instead of "<boost-prefix>/include" to the include path. As a minor addition a few loops in the compiler wrappers have been simplified.
2019-02-13Dynamic library/include paths (#8136)Peter Scheibel8-248/+261
Fixes #7855 Closes #8070 Closes #2645 When searching for library directories (e.g. to add "-L" arguments to the compiler wrapper) Spack was only trying the "lib/" and "lib64/" directories for each dependency install prefix; this missed cases where packages would install libraries to subdirectories and also was not customizable. This PR makes use of the ".headers" and ".libs" properties for more-advanced location of header/library directories. Since packages can override the default behavior of ".headers" and ".libs", it also allows package writers to customize. The following environment variables which used to be set by Spack for a package build have been removed: * Remove SPACK_PREFIX and SPACK_DEPENDENCIES environment variables as they are no-longer used * Remove SPACK_INSTALL environment variable: it was not used before this PR
2019-02-13Bug Fix in permission setter (#10584)abernede1-2/+3
* fix permission setter Fix a typo in islink test when applied to files. * os.walk explicitly set not to follow links The algorithm strongly rely on not following links.
2019-02-13Allow tty output to be timestamped (#10554)sknigh5-6/+103
* Spack debug output now includes microsecond-granularity timestamps. * Timestamps can also be enabled with the `--timestamp` command line argument.
2019-02-13enh: allow time like HH:MM in date strings. (#10034)Matthias Wolf2-1/+8
2019-02-12Enhancement to module autoload documentation (#10310)George Hartzell1-1/+15
* Note that `none` is the default for lmod autoload Save a bit of confusion by *explicitly* pointing out that `none` is the default value for autoload in the lmod module file generator. * Add a tip re building software externally Add a tip about using `autoload: all` when building packages outside of the tree that use artifacts (e.g. libraries, includes) within the tree.
2019-02-12Allow secondary generators when building with CMake. (#9324)Chris Green2-4/+50
CMake supports the notion of secondary generators which provide extra information to (e.g.) IDEs over and above that normally provided by the primary generator. Spack only supports the 'Unix Makefiles' and 'Ninja' primary generators but was not parsing out the primary generator when a secondary generator was also included (e.g. for a generator attribute like 'Codeblocks - Ninja'). This adds a regex for extracting the primary generator for validation. Since the secondary generator is irrelevant to a Spack build, it is passed on to CMake without further validation.
2019-02-08CudaPackage: fix wrong version range (#10551)Denis Davydov1-1/+1
2019-02-07log-parse: fix error message when no error lines are found (#10543)Massimiliano Culpo1-1/+1
2019-02-06Add x86_64 as target to cray platform (#10369)Mario Melara1-0/+4
Add x86_64 as a target for the Cray platform, and also designate it as the default front_end target.
2019-02-06CUDA compiler conflicts for Linux (#10460)Nichols A. Romero1-14/+69
* CUDA compiler conflicts for Linux. * Add Volta and Turing GPUs. * Add mandatory conflict for Volta and Turing GPUs. * Revert "CUDA compiler conflicts for Linux." This reverts commit 7d4ff654ac53aad272c59e9f7f8bb3fbb32bcec4. * Compiler conflicts introduced from previous commit into CUDA packaged moved and integrated into CUDA build system. * More conversative with compiler conflicts for cuda 10.0.130, since I don't know what will happen with future cuda 10.x releases. * Correct off-by-one errors in clang conflicts for x86_64 Linux. * No restrictions on Apple Clang compiler until we are able to distinguish Xcode clang from github clang more easily. Note to fix this in the future. * Change comment to clarify that github clang refers to LLVM clang. * Fix and simplify index range. * Fix overlapping conflicts for CUDA 10.0.130 * Removed extra ^cuda from conflict.
2019-01-29debug output includes modulecmd output (#9476)Peter Scheibel1-7/+28
Debug output now includes the output of modulecmd executions. Only output module content when a failure occurs; always report when a module is loaded/unloaded.