summaryrefslogtreecommitdiff
path: root/etc
AgeCommit message (Collapse)AuthorFilesLines
2020-03-20multiprocessing: allow Spack to run uninterrupted in background (#14682)Greg Becker1-0/+1
Spack currently cannot run as a background process uninterrupted because some of the logging functions used in the install method (especially to create the dynamic verbosity toggle with the v key) cause the OS to issue a SIGTTOU to Spack when it's backgrounded. This PR puts the necessary gatekeeping in place so that Spack doesn't do anything that will cause a signal to stop the process when operating as a background process.
2020-02-19Distributed builds (#13100)Tamara Dahlgren1-1/+1
Fixes #9394 Closes #13217. ## Background Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`. This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.). The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method. Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`). The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`). This PR adds support for distributed (single- and multi-node) parallel builds. The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages. ## Approach ### File System Locks Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks. The runs can be a combination of interactive and batch processes affecting the same file system. Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed. Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes. The resulting file contains the failing spec to facilitate manual debugging. ### Priority Queue Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies. Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix. Consequently, packages can no longer get away with an install method consisting of `pass`, for example. ## Caveats - This still only parallelizes a single-rooted build. Multi-rooted installs (e.g., for environments) are TBD in a future PR. Tasks: - [x] Adjust package lock timeout to correspond to value used in the demo - [x] Adjust database lock timeout to reduce contention on startup of concurrent `spack install <spec>` calls - [x] Replace (test) package's `install: pass` methods with file creation since post-install `sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!` - [x] Resolve remaining existing test failures - [x] Respond to alalazo's initial feedback - [x] Remove `bin/demo-locks.py` - [x] Add new tests to address new coverage issues - [x] Replace built-in package's `def install(..): pass` to "install" something (i.e., only `apple-libunwind`) - [x] Increase code coverage
2020-02-13hipsycl: new package and new 'sycl' virtual package (#14051)Federico Ficarelli1-0/+1
2019-11-14Config option to allow gpg warning suppression (#13743)Greg Becker1-0/+8
Add a configuration option to suppress gpg warnings during binary package verification. This only suppresses warnings: a gpg failure will still fail the install. This allows users who have already explicitly trusted the gpg key they are using to avoid seeing repeated warnings that it is self-signed.
2019-10-23Users can configure use of RPATH or RUNPATH (#9168)Massimiliano Culpo1-0/+5
Add a new entry in `config.yaml`: config: shared_linking: 'rpath' If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH. Details: * Spack cc wrapper explicitly adds `--disable-new-dtags` when linking * cc wrapper also strips `--enable-new-dtags` from the compile line when disabling (and vice versa) * We specifically do *not* add any dtags flags on macOS, which uses Mach-O binaries, not ELF, so there's no RUNPATH)
2019-10-05Consistently support pkg-config files in share subdirectory (#12838)Michael Kuhn1-0/+2
While the build environment already takes share/pkgconfig into account, the generated module files etc. only consider lib/pkgconfig and lib64/pkgconfig.
2019-10-02Remove support for generating dotkit files (#11986)Massimiliano Culpo2-2/+0
Dotkit is being used only at a few sites and has been deprecated on new machines. This commit removes all the code that provide support for the generation of dotkit module files. A new validator named "deprecatedProperties" has been added to the jsonschema validators. It permits to prompt a warning message or exit with an error if a property that has been marked as deprecated is encountered. * Removed references to dotkit in the docs * Removed references to dotkit in setup-env-test.sh * Added a unit test for the 'deprecatedProperties' schema validator
2019-09-13Remove CombinatorialSpecSet in favor of environments + stacksScott Wittenburg1-16/+0
2019-09-03Use spack/user-specific stage root by default; stage cleaning (#12516)Tamara Dahlgren1-7/+12
* When cleaning the stage root, only remove directories that appear to be used for staging Spack packages. Previously Spack was clearing all directories in the stage root, which could remove content not related to Spack if the user chose a staging root which contains files/directories not managed by Spack. * The documentation is updated with warnings about choosing a stage directory that is only managed by Spack (although generally the check added in this PR for "spack clean" should avoid removing content that was not created by Spack) * The default stage directory (in config.yaml) is now $tempdir/$user/spack-stage and the logic is updated to omit the $user portion of this path if $tempdir already contains a $user directory. * When creating stage root assign user read/write permissions to all directories in the path under $user. Previously Spack was assigning the permissions of the first existing parent directory
2019-08-19features: Remove stage symlinks (#12072)Tamara Dahlgren1-8/+18
Fixes #11163 The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used. Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
2019-07-25Make OpenJDK the default (#12106)Elizabeth Fischer1-1/+1
2019-07-01openfoam: rename openfoam-com to openfoam (#11876)Mark Olesen1-1/+0
The original packaging, with openfoam as a virtual and with openfoam-com, openfoam-org and openfoam-extend as providers, adds an obfuscation layer without any benefit. - Rename `openfoam-com` to `openfoam`, and remove the `openfoam` virtual package.
2019-06-10ninja{-fortran}: make pkgs nonvirtual (fix #11628) (#11629)Geoffrey Oxberry1-2/+0
2019-05-28Cap the maximum number of build jobs (#11373)Massimiliano Culpo1-4/+6
* config:build_jobs now controls the number of parallel jobs to spawn during builds, but cannot ever exceed the number of cores on the machine. * The default is set to 16 or the number of available cores, whatever is lowest. * Updated docs to reflect the changes done to limit parallel builds
2019-05-21Add Fujitsu compiler to Spack. (#11287)t-karatsu1-1/+1
* Add Fujitsu compiler to Spack. * Fixes for flake8 * Chenges location of FCC to subdirectory called case-insensitive * Add compiler tests for Fujitsu compiler * Modify the logic of taking compiler version for new version of Fujitsu compiler
2019-05-07ninja: drop python as a run dep (#11347)Chuck Atkins1-0/+2
* ninja: Make python a build-only dependency * ninja, ninja-fortran: allow the packages to provide eachother
2019-05-01mesa: re-written to new meson build system (#10482)Chuck Atkins1-1/+2
* mesa: re-writen to new meson build system * xorg-server: Remove unneeded mesa dependency * glx: provide the virtual GLX package * Convert most "mesa" dependencies to use virtual gl and glx dependencies * mesa: Switch to always building from the git repo instead of a tarball * Use gl and glx version ranges instead of exact versions * glx: provide a default package * add auto to swr variant * Revert "add auto to swr variant" This reverts commit 2e7cd8a8aca07767c00657d0d09985ddd6306883. * update mesa package to version 19.0 * set +glx by default only when running on linux * fix style issues in geant4 package * remove virtual package "egl"
2019-03-26mysql: Support client-only, cxxstd and more versions (#10911)Chris Green1-1/+1
* add cxxstd variant * add CMake constraints based on platform/version * add older versions * update boost dependency version constraints (which are closely tied to mysql version) and update boost cxxstd choice to be the same as the cxxstd chosen for mysql * add client-only support (including a patch for 5.5.x) * record the mysql package as a provider of the mysql-client virtual
2019-03-15New package: mariadb-c-client (#7729)Chris Green1-0/+2
* mariadb-c-client is a new package using the distinct, LGPL, MySQL-compatible client library from mariadb.com. It provides the virtual package mariadb-client * mariadb is recorded as a provider of the mariadb-client virtual * The mysql-client virtual package is also added, and mariadb-c-client is recorded as a provider for it
2019-03-12New package: ibm-java (#10777)Mark W. Krentel1-1/+1
* Add binary package of the IBM Java SDK for big and little-endian powerpc (power7, 8 and 9). The jdk and openjdk packages only install on x86_64. * Add ibm-java as a java provider * The jdk and openjdk packages only install on x86_64. Add conflicts for ppc64 and ppc64le to jdk and openjdk.
2019-03-07Create option to build compilers as needed (#10761)Greg Becker1-0/+6
* Create option to build missing compilers and add them to config before installing packages that use them * Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
2019-02-21release workflow: Add spack command for generating the .gitlab-ci.ymlScott Wittenburg1-0/+16
This spack command adds a new schema for a file which describes the builder containers available, along with the compilers availabe on each builder. The release-jobs command then generates the .gitlab-ci.yml file by first expanding the release spec set, concretizing each spec (in an appropriate docker container if --this-machine-only argument is not provided on command line), and then combining and staging all the concrete specs as jobs to be run by gitlab.
2018-12-21Remove /nfs/tmp2 from default build_stage locations (#10170)Elsa Gonsiorowski, PhD1-1/+0
* Remove /nfs/tmp2 from default configuration * /nfs/tmp2 is going away from LC... and doesn’t exist for the rest of the world. * update documentation to remove /nfs/tmp2 as well
2018-11-11Minor changes to Basic Settings docs for SC18 (#9809)Adam J. Stewart1-4/+6
Grammar/spelling issues
2018-11-09Introduce fftw-api virtual package for Intel-MKL and FFTW (#9618)Nichols A. Romero1-0/+1
* Introduce FFTW2 and FFT3 providers for Intel-MKL and FFTW Spack packages. * make fftw default package for fftw-api virtual package * virtual package test assertion now provides location of default virtual packages. * Change name of virtual package to fftw-api and used versioned interface.
2018-10-18Allow setting language of compiler messages (#9486)Michael Kuhn1-0/+8
* Add a build_language config.yaml option which controls the language of compiler messages * build_language defaults to "C", in which case the compiler messages will be in English. This allows Spack log parsing to detect and highlight error messages (since the regular expressions to find error messages are in English) * The user can use the default language in their environment by setting the build_language config variable to null or ''
2018-10-11permissions: add permission configuration to packages.yaml (#8773)Greg Becker1-1/+3
Spack can now be configured to assign permissions to the files installed by a package. In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence. The `read` and `write` attributes take one of `user`, `group`, and `world`. packages: all: permissions: write: group group: spack my_app: permissions: read: group group: my_team
2018-10-10Correct name for install_path_scheme in defaults config.yaml (#9482)Greg Becker1-1/+1
Default config.yaml mis-named `install_path_scheme` as `directory_layout`. This commit corrects the error.
2018-09-25Increase and customize lock timeouts (#9219)Peter Scheibel1-0/+13
Fixes #9166 This is intended to reduce errors related to lock timeouts by making the following changes: * Improves error reporting when acquiring a lock fails (addressing #9166) - there is no longer an attempt to release the lock if an acquire fails * By default locks taken on individual packages no longer have a timeout. This allows multiple spack instances to install overlapping dependency DAGs. For debugging purposes, a timeout can be added by setting 'package_lock_timeout' in config.yaml * Reduces the polling frequency when trying to acquire a lock, to reduce impact in the case where NFS is overtaxed. A simple adaptive strategy is implemented, which starts with a polling interval of .1 seconds and quickly increases to .5 seconds (originally it would poll up to 10^5 times per second). A test is added to check the polling interval generation logic. * The timeout for Spack's whole-database lock (e.g. for managing information about installed packages) is increased from 60s to 120s * Users can configure the whole-database lock timeout using the 'db_lock_timout' setting in config.yaml Generally, Spack locks (those created using spack.llnl.util.lock.Lock) now have no timeout by default This does not address implementations of NFS that do not support file locking, or detect cases where services that may be required (nfslock/statd) aren't running. Users may want to be able to more-aggressively release locks when they know they are the only one using their Spack instance, and they encounter lock errors after a crash (e.g. a remote terminal disconnect mentioned in #8915).
2018-08-23Set DYLD_LIBRARY_PATH on macOS (#9073)Adam J. Stewart3-2/+44
2018-08-06spack general packages.yaml defaults: add unwindGeoffrey Malcolm Oxberry1-0/+2
This commit adds 'libunwind' as the default 'unwind' provider in spack. This preference should be overridden by the darwin/packages.yaml file on macOS. Continues the changes discussed in
2018-08-06darwin packages.yaml defaults: add unwind providerGeoffrey Malcolm Oxberry1-0/+7
This commit adds default unwind providers to the default packages.yaml for darwin. Compiler versions are supplied with the apple-unwind package so that apple-unwind is only used with Apple's clang fork, and not with LLVM's clang.
2018-07-09Add ccache support (#3761)Christoph Junghans1-0/+4
If the user sets "ccache: true" in spack's config.yaml, Spack will use an available ccache executable when compiling c/c++ code. This feature is disabled by default (i.e. "ccache: false") and the documentation is updated with how to enable ccache support
2018-06-24refactor: move templates from root to share/spackTodd Gamblin1-1/+1
- This complies with the unix directory hierarchy standard (which Spack attempts to follow) - Also unclutters the repo root directory.
2018-05-18locks: add configuration and command-line options to enable/disable locks ↵Todd Gamblin1-0/+12
(#7692) - spack.util.lock behaves the same as llnl.util.lock, but Lock._lock and Lock._unlock do nothing. - can be disabled with a control variable. - configuration options can enable/disable locking: - `locks` option in spack configuration controls whether Spack will use filesystem locks or not. - `-l` and `-L` command-line options can force-disable or force-enable locking. - Spack will check for group- and world-writability before disabling locks, and it will not allow a group- or world-writable instance to have locks disabled. - update documentation
2018-05-05Change default elf provider on macOS (#7939)Adam J. Stewart1-0/+2
2018-05-05Set default provider for every virtual package (#7784)Adam J. Stewart1-1/+2
* Set default provider for every virtual package * Add unit test to ensure that every virtual package has a default provider explicitly set
2018-04-12Specify default providers for the glu virtual package. (#7733)Veselin Dobrev1-0/+1
2018-03-25fix outdated comments in config.yaml (#7594)healther1-3/+3
* `spack purge` now is `spack clean`
2018-03-23Add the ability to separate architecture in path scheme (#4329)Mario Melara1-0/+3
* Add format to separate target and os for path spec format can now handle separations of target and os for setting up the path. * Added ${PLATFORM} et al to spec.format() ${PLATFORM}, ${OS}, ${TARGET} * Update tests Updated tests and got rid of unnecessary code. * Also update documentation to reflect this new ability. * Add default path scheme to config.yaml Added default path scheme to config.yaml. Users can overwrite this section if they want.
2018-01-22Update gl/qt options for vtk package (#6551)Scott Wittenburg1-0/+1
This provides options for hardware and software rendering with the gl API and updates the vtk package to make use of those options: * Create new "gl" virtual package, provided by mesa * Add external-only "opengl" package, which represents a system install that provides gl * For vtk: prefer system gl implementation by default to get speed up from hardware rendering. When software rendering is specifically requested (+osmesa), try to use the llvmpipe approach, as it is much faster. * Make qt dependency optional for vtk * Add basic support for VTK on osx * qt: Depend on virtual gl package, rather than mesa impl
2017-11-23Introduce virtual dependency pkgconfig (#5198)Michael Kuhn1-0/+1
There are two providers, pkgconf and pkg-config, with the former being the default provider.
2017-09-19Modulefiles generated with a template engine (#3183)Massimiliano Culpo2-0/+7
* Module files now are generated using a template engine refers #2902 #3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies. * Improved the support for templates in module files. Added an entry in `config.yaml` (`template_dirs`) to list all the directories where Spack could find templates for `jinja2`. Module file generators have a simple override mechanism to override template selection ('modules.yaml' beats 'package.py' beats 'default'). * Added jinja2 and MarkupSafe to vendored packages. * Spec.concretize() sets mutual spec-package references The correct place to set the mutual references between spec and package objects at the end of concretization. After a call to concretize we should now be ensured that spec is the same object as spec.package.spec. Code in `build_environment.py` that was performing the same operation has been turned into an assertion to be defensive on the new behavior. * Improved code and data layout for modules and related tests. Common fixtures related to module file generation have been extracted in `conftest.py`. All the mock configurations for module files have been extracted from python code and have been put into their own yaml file. Added a `context_property` decorator for the template engine, to make it easy to define dictionaries out of properties. The default for `verbose` in `modules.yaml` is now False instead of True. * Extendable module file contexts + short description from docstring The contexts that are used in conjunction with `jinja2` templates to generate module files can now be extended from package.py and modules.yaml. Module files generators now infer the short description from package.py docstring (and as you may expect it's the first paragraph) * 'module refresh' regenerates all modules by default `module refresh` without `--module-type` specified tries to regenerate all known module types. The same holds true for `module rm` Configure options used at build time are extracted and written into the module files where possible. * Fixed python3 compatibility, tests for Lmod and Tcl. Added test for exceptional paths of execution when generating Lmod module files. Fixed a few compatibility issues with python3. Fixed a bug in Tcl with naming_scheme and autoload + unit tests * Updated module file tutorial docs. Fixed a few typos in docstrings. The reference section for module files has been reorganized. The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml Also: - Licenses have been updated to include NOTICE and extended to 2017 - docstrings have been reformatted according to Google style * Removed redundant arguments to RPackage and WafPackage. All the callbacks in `RPackage` and `WafPackage` that are not build phases have been modified not to accept a `spec` and a `prefix` argument. This permits to leverage the common `configure_args` signature to insert by default the configuration arguments into the generated module files. I think it's preferable to handling those packages differently than `AutotoolsPackage`. Besides only one package seems to override one of these methods. * Fixed broken indentation + improved resiliency of refresh Fixed broken indentation in `spack module refresh` (probably a rebase gone silently wrong?). Filter the writers for blacklisted specs before searching for name clashes. An error with a single writer will not stop regeneration, but instead will print a warning and continue the command.
2017-08-24Make jpeg a virtual dependency. (#5190)Sergey Kosukhin1-0/+1
* Make jpeg a virtual dependency. * Make 'libjpeg-turbo' the default implementation of 'jpeg'.
2017-08-16Refactor IntelInstaller into IntelPackage base class (#4300)Adam J. Stewart1-3/+4
* Refactor IntelInstaller into IntelPackage base class * Move license attributes from __init__ to class-level * Flake8 fixes: remove unused imports * Fix logic that writes the silent.cfg file * More specific version numbers for Intel MPI * Rework logic that selects components to install * Final changes necessary to get intel package working * Various updates to intel-parallel-studio * Add latest version of every Intel package * Add environment variables for Intel packages * Update env vars for intel package * Finalize components for intel-parallel-studio package Adds a +tbb variant to intel-parallel-studio. The tbb package was renamed to intel-tbb. Now both intel-tbb and intel-parallel-studio+tbb provide tbb. * Overhaul environment variables set by intel-parallel-studio * Point dependent packages to the correct MPI wrappers * Never default to intel-parallel-studio * Gather env vars by sourcing setup scripts * Use mpiicc instead of mpicc when using Intel compiler * Undo change to ARCH * Add changes from intel-mpi to intel-parallel-studio * Add comment explaining mpicc vs mpiicc * Prepend env vars containing 'PATH' or separators * Flake8 fix * Fix bugs in from_sourcing_file * Indentation fix * Prepend, not set if contains separator * Fix license symlinking broken by changes to intel-parallel-studio * Use comments instead of docstrings to document attributes * Flake8 fixes * Use a set instead of a list to prevent duplicate components * Fix MKL and MPI library linking directories * Remove +all variant from intel-parallel-studio * It is not possible to build with MKL, GCC, and OpenMP at this time * Found a workaround for locating GCC libraries * Typos and variable names * Fix initialization of empty LibraryList
2017-08-05Add a new package for IcedTea (#4804)snehring1-0/+1
* alsa-lib: new package * cups: new package * icedtea: new package * icedtea: adding JAVA_HOME variable * alsa: removing placeholder * icedtea: typo fix * alsa: fixing formattting * icedtea: formatting fixes and cleanup * icedtea: additional cleanup * jdk: adding provides('java') to jdk, setting jdk to default java. * icedtea: adding jdk as a dependency * java: changing depends_on('jdk') and similar entries to depends_on('java') * icedtea: removing unused imports, trying to placate flake8. * fastqc: fixing flake8 error * bazel: fixing typo in java dependency * jdk: changing provides java to a range * icedtea: cleaning up. Splitting up checksums. * icedtea: jdk dependency is build only
2017-08-04New package libaec: a BSD-licensed replacement for the szip library. (#4437)Sergey Kosukhin1-0/+1
* Added package for 'libaec'. * New version for package 'libaec'.
2017-06-21refactor openfoam packages (#3669)Mark Olesen1-1/+1
* Several improvements for the openfoam packages -- Refactor openfoam packages by adding an OpenfoamArch class Use separate configure, build, install phases. Provide FOAM_PROJECT_DIR dependent env for openfoam packages - easier way to locate Eliminate intermediate installation directories - unneeded clutter. - makes it less than easy to find the etc/bashrc file Add versioning for all openfoam patches - no certainty which parts (if any) will be needed in future versions, especially if we strive to ensure that the upstream version builds well with spack to begin with. Support build of develop branches - helps track build regressions for future openfoam releases STYLE: use common/ and assets/ to provide additional (build) resources ... * - adjust OpenFOAM provider Move openfoam-com up front since this is the one being used as a base for the others
2017-04-15Allow users to set parallel jobs in config.yaml (#3812)Adam J. Stewart1-0/+6
* Allow users to set parallel jobs in config.yaml * Undo change from endash to emdash * Remove parallel config, rename jobs to build_jobs
2017-04-02Set default providers for all virtual dependencies (#3634)Adam J. Stewart1-2/+11
* Set default providers for everything * Add default OpenFOAM provider