summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2020-09-25Streamline key management for build caches (#17792)Omar Padron1-1/+10
* Rework spack.util.web.list_url() list_url() now accepts an optional recursive argument (default: False) for controlling whether to only return files within the prefix url or to return all files whose path starts with the prefix url. Allows for the most effecient implementation for the given prefix url scheme. For example, only recursive queries are supported for S3 prefixes, so the returned list is trimmed down if recursive == False, but the native search is returned as-is when recursive == True. Suitable implementations for each case are also used for file system URLs. * Switch to using an explicit index for public keys Switches to maintaining a build cache's keys under build_cache/_pgp. Within this directory is an index.json file listing all the available keys and a <fingerprint>.pub file for each such key. - Adds spack.binary_distribution.generate_key_index() - (re)generates a build cache's key index - Modifies spack.binary_distribution.build_tarball() - if tarball is signed, automatically pushes the key used for signing along with the tarball - if regenerate_index == True, automatically (re)generates the build cache's key index along with the build cache's package index; as in spack.binary_distribution.generate_key_index() - Modifies spack.binary_distribution.get_keys() - a build cache's key index is now used instead of programmatic listing - Adds spack.binary_distribution.push_keys() - publishes keys from Spack's keyring to a given list of mirrors - Adds new spack subcommand: spack gpg publish - publishes keys from Spack's keyring to a given list of mirrors - Modifies spack.util.gpg.Gpg.signing_keys() - Accepts optional positional arguments for filtering the set of keys returned - Adds spack.util.gpg.Gpg.public_keys() - As spack.util.gpg.Gpg.signing_keys(), except public keys are returned - Modifies spack.util.gpg.Gpg.export_keys() - Fixes an issue where GnuPG would prompt for user input if trying to overwrite an existing file - Modifies spack.util.gpg.Gpg.untrust() - Fixes an issue where GnuPG would fail for input that were not key fingerprints - Modifies spack.util.web.url_exists() - Fixes an issue where url_exists() would throw instead of returning False * rework gpg module/fix error with very long GNUPGHOME dir * add a shim for functools.cached_property * handle permission denied error in gpg util * fix tests/make gpgconf optional if no socket dir is available
2020-09-23OLCF Ascent gitlab ci trigger: pass SPACK_REF (#18875)eugeneswalker1-0/+3
2020-09-18trigger ascent e4s pipeline on merge to spack develop (#18655)Shahzeb Siddiqui1-0/+6
* trigger ascent e4s pipeline on merge to spack develop * change pipeline name ecpcitest/e4s is the pipeline that will be triggered for merge on develop its the E4S use-case.
2020-09-14Make sure each develop pipeline tests associated commitScott Wittenburg1-1/+1
2020-09-14Provide your own script, before_script, and after_scriptScott Wittenburg1-1/+1
2020-09-10Bugfix for fish support: overly zealous arg matching (#18528)Johannes Blaschke1-18/+59
* bugfix for issue 18369 * fix typo Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-09-08commands: update help for `spack install --yes-to-all` (#18367)Richarda Butler1-1/+1
`spack install --yes-to-all` doesn't actually make the build non-interactive, but that is why people typically use it. This documents that you must also specify `--no-checksum` for a fully non-interactive build.
2020-09-05Make spack environment configurations writable from spack external and spack ↵Robert Blake1-1/+1
compiler find (#18165) * spack config: default modification scope can be an environment The previous model was that environments are the highest priority config scope for config reading operations, but were not considered for config writing operations. Now, the active environment is the highest priority config scope for both reading and writing operations. Now spack config add, spack external find and spack compiler set environment configuration in the environment by default if an environment is active. This is a change in default behavior for these routines, but better matches the mental model for an environment taking precedence over the user's default config file. * add scope argument to 'spack external find' to choose non-default scope * Increase testing for config modifications on environments Co-authored-by: Gregory Becker <becker33@llnl.gov>
2020-09-04Rely on E4S project variable for SPACK_REPOScott Wittenburg1-2/+0
2020-09-02Add new RubyPackage build system base class (#18199)Adam J. Stewart1-1/+5
* Add new RubyPackage build system base class * Ruby: add spack external find support * Add build tests for RubyPackage
2020-08-10Simplify the detection protocol for packagesMassimiliano Culpo1-1/+5
Packages can implement “detect_version” to support detection of external instances of a package. This is generally easier than implementing “determine_spec_details”. The API for determine_version is similar: for example you can return “None” to indicate that an executable is not an instance of a package. Users may implement a “determine_variants” method for a package. When doing external detection, executables are grouped by version and each group results in a single invocation of “determine_variants” for the associated spec. The method returns a string specifying the variants for the package. The method may additionally return a dictionary representing extra attributes for the package. These will be stored in the spec yaml and can be retrieved from self.spec.extra_attributes The Spack GCC package has been updated with an implementation of “determine_variants” which adds the following extra attributes to the package: c, cxx, fortran
2020-08-10Update packages.yaml format and support configuration updatesMassimiliano Culpo2-14/+56
The YAML config for paths and modules of external packages has changed: the new format allows a single spec to load multiple modules. Spack will automatically convert from the old format when reading the configs (the updates do not add new essential properties, so this change in Spack is backwards-compatible). With this update, Spack cannot modify existing configs/environments without updating them (e.g. “spack config add” will fail if the configuration is in a format that predates this PR). The user is prompted to do this explicitly and commands are provided. All config scopes can be updated at once. Each environment must be updated one at a time.
2020-07-31Move Python 2.6 unit tests to Github Actions (#17279)Massimiliano Culpo1-22/+0
* Run Python2.6 unit tests on Github Actions * Skip url tests on Python 2.6 to reduce waiting times * Skip foreground background tests on Python 2.6 to reduce waiting times * Removed references to Travis in the documentation * Deleted install_patchelf.sh (can be installed from repo on CentOS 6)
2020-07-27tutorial: Add boto3 installation to setup script (#17722)Todd Gamblin1-0/+6
2020-07-27add tutorial setup script to share/spack (#17705)Greg Becker1-0/+123
* add tutorial setup script to share/spack * Add check for Ubuntu 18, fix xvda check, fix apt-get errors - now works on t2.micro, t2.small, and m instances - apt-get needs retries around it to work Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-07-23add tutorial public key to share/spack/keys dir (#17684)Greg Becker1-0/+38
2020-07-16Make the largest layer of the docker image cacheable (#17553)Harmen Stoppels4-56/+56
2020-07-15spack containerize: added --fail-fast argument to containerize install. (#17533)Paul2-2/+2
2020-07-08Buildcache: bindist test without invoking spack compiler wrappers. (#15687)Patrick Gartung1-0/+22
* Buildcache: * Try mocking an install of quux, corge and garply using prebuilt binaries * Put patchelf install after ccache restore * Add script to install patchelf from source so it can be used on Ubuntu:Trusty which does not have a patchelf pat package. The script will skip building on macOS * Remove mirror at end of bindist test * Add patchelf to Ubuntu build env * Revert mock patchelf package to allow other tests to run. * Remove depends_on('patchelf', type='build') relying instead on * Test fixture to ensure patchelf is available. * Call g++ command to build libraries directly during test build * Flake8 * Install patchelf in before_install stage using apt unless on Trusty where a build is done. * Add some symbolic links between packages * Flake8 * Flake8: * Update mock packages to write their own source files * Create the stage because spec search does not create it any longer * updates after change of list command arguments * cleanup after merge * flake8
2020-07-08spack create: ask how many to download (#17373)Adam J. Stewart1-1/+1
2020-07-06bugfix: no infinite recursion in setup-env.sh on CrayTodd Gamblin3-0/+29
On Cray platforms, we rely heavily on the module system to figure out what targets, compilers, etc. are available. This unfortunately means that we shell out to the `module` command as part of platform initialization. Because we run subcommands in a shell, we can get infinite recursion if `setup-env.sh` and friends are in some init script like `.bashrc`. This fixes the infinite loop by adding guards around `setup-env.sh`, `setup-env.csh`, and `setup-env.fish`, to prevent recursive initializations of Spack. This is safe because Spack never shells out to itself, so we do not need it to be initialized in subshells. - [x] add recursion guard around `setup-env.sh` - [x] add recursion guard around `setup-env.csh` - [x] add recursion guard around `setup-env.fish`
2020-07-01Moved flake8, shell and documentation tests to Github Action (#17328)Massimiliano Culpo3-14/+10
* Move flake8 tests on Github Actions * Move shell test to Github Actions * Moved documentation build to Github Action * Don't run coverage on Python 2.6 Since we get connection errors consistently on Travis when trying to upload coverage results for Python 2.6, avoid computing coverage entirely to speed-up tests.
2020-06-30Activate environment in container file (#17316)Glenn Johnson2-2/+4
* Activate environment in container file This PR will ensure that the container recipes will build the spack environment by first activating the environment. * Deactivate environment before environment collection For Singularity, the environment must be deactivated before running the command to collect the environment variables. This is because the environment collection uses `spack env activate`.
2020-06-30Add fish shell support (#9279)Johannes Blaschke4-4/+1128
* share/spack/setup-env.fish file to setup environment in fish shell * setup-env.fish testing script * Update share/spack/setup-env.fish Co-Authored-By: Elsa Gonsiorowski, PhD <gonsie@me.com> * Update share/spack/qa/setup-env-test.fish Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com> * updates completions using `spack commands --update-completion` * added stderr-nocaret warning * added fish shell tests to CI system Co-authored-by: becker33 <becker33@llnl.gov> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
2020-06-29remove three commands that have been deprecated since v0.13.0 (#17291)Greg Becker1-28/+1
* remove three commands that have been deprecated since v0.13.0
2020-06-26Use json for buildcache index (#15002)Scott Wittenburg1-20/+3
* Start moving toward a json buildcache index * Add spec and database index schemas * Add a schema for buildcache spec.yaml files * Provide a mode for database class to generate buildcache index * Update db and ci tests to validate object w/ new schema * Remove unused temporary upload-s3 command * Use database class to generate buildcache index * Do not generate index with each buildcache creation * Make buildcache index mode into a couple of constructor args to Database class * Use keyword args for _createtarball * Parse new json index when we get specs from buildcache Now that only one index file per mirror needs to be fetched in order to have all the concrete specs for binaries available on the mirror, we can just fetch and refresh the cached specs every time instead of needing to use the '-f' flag to force re-reading.
2020-06-25add workaround for gitlab ci needs limit (#17219)Omar Padron1-1/+1
* add workaround for gitlab ci needs limit * fix style/address review comments * convert filter obj to list * update command completion * remove dict comprehension * add workaround tests * fix sorting issue between disparate types * add indeces to format
2020-06-25spack config: new subcommands add/remove (#13920)Greg Becker1-1/+28
spack config add <value>: add nested value value to the configuration scope specified spack config remove/rm: remove specified configuration from the relevant scope
2020-06-24features: Add install failure tracking removal through `spack clean` (#15314)Tamara Dahlgren1-1/+1
* Add ability to force removal of install failure tracking data through spack clean * Add clean failures option to packaging guide
2020-06-23Added support for --fail-fast install option to terminate on first failureTamara Dahlgren1-1/+1
2020-06-23Added unit tests to Github Actions (#16610)Massimiliano Culpo1-12/+3
* Added unit tests to Github Actions * Set user e-mail and name for git tests to succeed * Simplify setup.sh logic * Replicate Travis script on Github Actions * Update flags since '.' is not allowed * Added badge, simplified workflow * Remove pinning of coverage * Remove unit tests run on Github Actions from Travis
2020-06-22Pre ci optimization (#16372)Omar Padron1-1/+1
* add initial optimization script * integrate optimization in spack ci * make optimization opt-in * fix import error * flake8 fixes * update command completion * work around vermin errors * fix sphynx errors
2020-06-18Explicitly install setuptools in docker images (#17143)Omar Padron4-0/+4
2020-06-16fix docker image entrypoints (#17105)Omar Padron7-436/+269
Also removes extraneous prompt and ssh handling logic.
2020-06-05commands: use a single ThreadPool for `spack versions` (#16749)Massimiliano Culpo1-1/+1
This fixes a fork bomb in `spack versions`. Recursive generation of pools to scrape URLs in `_spider` was creating large numbers of processes. Instead of recursively creating process pools, we now use a single `ThreadPool` with a concurrency limit. More on the issue: having ~10 users running at the same time spack versions on front-end nodes caused kernel lockup due to the high number of sockets opened (sys-admin reports ~210k distributed over 3 nodes). Users were internal, so they had ulimit -n set to ~70k. The forking behavior could be observed by just running: $ spack versions boost and checking the number of processes spawned. Number of processes per se was not the issue, but each one of them opens a socket which can stress `iptables`. In the original issue the kernel watchdog was reporting: Message from syslogd@login03 at May 19 12:01:30 ... kernel:Watchdog CPU:110 Hard LOCKUP Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#110 stuck for 23s! [python3:2756] Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#94 stuck for 22s! [iptables:5603]
2020-06-03Mirrors: add option to exclude packages from "mirror create" (#14154)Peter Scheibel1-1/+1
* add an --exclude-file option to 'spack mirror create' which allows a user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option * allow specifying number of versions when mirroring all packages * when mirroring all specs within an environment, include dependencies of root specs * add '--exclude-specs' option to allow user to specify that specs should be excluded on the command line * add test for excluding specs
2020-06-03Feature: add option to create view by copying/relocating files (#16480)Greg Becker1-1/+19
* add subcommand `spack view copy/relocate` * update bash completions * add copy/relocate commands to view tests * allow copied views to be removed
2020-05-14Pipelines: Support DAG scheduling and dynamic child pipelinesScott Wittenburg2-9/+21
This change also adds a code path through the spack ci pipelines infrastructure which supports PR testing on the Spack repository. Gitlab pipelines run as a result of a PR (either creation or pushing to a PR branch) will only verify that the packages in the environment build without error. When the PR branch is merged to develop, another pipeline will run which results in the generated binaries getting pushed to the binary mirror.
2020-05-13Added alias and bash completion for spacktivate (#16472)Ben Bergen4-0/+17
2020-05-12travis: use bionic as default for Linux (#16521)Massimiliano Culpo2-25/+43
Modifications: - [x] Travis now uses `bionic` as a default (`xenial` used for Python 3.5, `trusty` for Python 2.6) - [x] Shell unit tests have been factored into their own run - [x] `kcov` is built only for tests that upload coverage results Overall with this we shave 3-4 mins. on each run and add an additional run of about 3 min. For some reason `kcov` 38 fails forwarding output when used with Python unit tests, so I used v34 for that and v38 (latest) for shell testing. Previously we were using v25.
2020-05-11Remove 'spack bootstrap' and associated docs (#15179)Massimiliano Culpo2-42/+1
fixes #15145 This commit removes the outdated `spack bootstrap` command and any reference to it in the documentation and unit tests.
2020-05-07"spack checksum" QoL (#14311)iarspider1-1/+1
* Non-interactive mode for spack checksum; allow passing 'package@version' to spack checksum * Flake8 fixes * Update checksum.py Fix typo * Update spack-completion script * Automatically set non-interactive mode if more than one version passed * Update lib/spack/spack/cmd/checksum.py Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com> * Add documentation and update spack-completion * Flake8 * Rename option * Update spack-completion * Update lib/spack/spack/cmd/checksum.py Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com> * Update checksum.py * Update stage.py * Update create.py Use batch mode when adding a new package Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-05-05Automatically find externals (#15158)Peter Scheibel1-1/+19
Add a `spack external find` command that tries to populate `packages.yaml` with external packages from the user's `$PATH`. This focuses on finding build dependencies. Currently, support has only been added for `cmake`. For a package to be discoverable with `spack external find`, it must define: * an `executables` class attribute containing a list of regular expressions that match executable names. * a `determine_spec_details(prefix, specs_in_prefix)` method Spack will call `determine_spec_details()` once for each prefix where executables are found, passing in the path to the prefix and the path to all found executables. The package is responsible for invoking the executables and figuring out what type of installation(s) are in the prefix, and returning one or more specs (each with version, variants or whatever else the user decides to include in the spec). The found specs and prefixes will be added to the user's `packages.yaml` file. Providing the `--not-buildable` option will mark all generated entries in `packages.yaml` as `buildable: False`
2020-05-01dev-build: --drop-in <shell> (#14887)Axel Huebl1-2/+2
* dev-build: --drop-in <shell> Add a `--drop-in <shell>` option to `spack dev-build`. This option will automatically run a `spack build-env <spec> -- <shell>` at the end of a `dev-build`, e.g. to quickly drop-and-devel into a build phase of a package. Example usage: ``` spack dev-build --before cmake --drop-in bash openpmd-api@develop ``` * build_env: drop in unit test Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-04-28dev-build: stop before phase (#14699)Axel Huebl1-2/+2
Add `-b,--before` option to dev-build command to stop before the phase in question.
2020-04-23Travis: change the spec we concretize on MacOS (#16256)Massimiliano Culpo1-1/+5
Since CMake can't build with GCC on MacOS, choose a spec that doesn't have CMake in the DAG.
2020-04-20autotools: remove FORCE_UNSAFE_CONFIGURE from docs, commands, code (#16145)Todd Gamblin5-11/+4
Since #16132, we've consolidated the setting of FORCE_UNSAFE_CONFIGURE to `autotools.py`, so we don't need to use it in packages like `coreutils`, in our commands, or in our container recipes. - [x] Remove FORCE_UNSAFE_CONFIGURE from packages - [x] Remove FORCE_UNSAFE_CONFIGURE from container recipes - [x] Remove FORCE_UNSAFE_CONFIGURE from `spack ci` command
2020-04-16macos: use DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATHTodd Gamblin2-8/+8
`DYLD_LIBRARY_PATH` can frequently break builtin macOS software when pointed at Spack libraries. This is because it takes *higher* precedence than the default library search paths, which are used by system software. `DYLD_FALLBACK_LIBRARY_PATH`, on the other hand, takes lower precedence. At first glance, this might seem bad, because the software installed by Spack in an environment needs to find *its* libraries, and it should not use the defaults. However, Spack's isntallations are always `RPATH`'d, so they do not have this problem. `DYLD_FALLBACK_LIBRARY_PATH` is thus useful for things built in an environment that need to use Spack's libraries, that don't set *their* RPATHs correctly for whatever reason. We now prefer it to `DYLD_LIBRARY_PATH` in modules and in environments because it helps a little bit, and it is much less intrusive.
2020-04-03Add option "--first" for "spack load" (#15622)G-Ragghianti1-1/+1
* Implemented --first option for "spack load" * added test for "spack load --first" Co-authored-by: gragghia <gragghia@localhost.localdomain>
2020-04-02Add commands to facilitate Spack/Python/OS reporting (#15834)Adam J. Stewart1-2/+6
* Add --version arg to spack python command * Add `spack debug report` command
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-09 08:08:57 +0000