summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-06-03Pipelines: Fix generation when dep and pkg arch differ (#24089)Scott Wittenburg1-1/+1
2021-06-03Extend cuda conflicts to cray platform (#24057)Harmen Stoppels1-61/+52
The CUDA compiler conflicts are valid on Cray too, and likely on Darwin x86_64 with %gcc and %clang too, so drop platform=linux
2021-06-02Pipelines: pass relative artifact paths to child jobs (#24085)Scott Wittenburg4-34/+104
Passing absolute paths from pipeline generate job to downstream rebuild jobs causes problems when the CI_PROJECT_DIR is not the same for the generate and rebuild jobs. This has happened, for example, when gitlab checks out the project into a runner-specific directory and different runners are chosen for the generate and rebuild jobs.
2021-06-02ensure the staging dir exists for `spack stage -p <PATH>` (#23963)Danny McClanahan4-20/+43
* ensure that the stage root exists for `spack stage -p <PATH>` * add test to verify `spack stage -p <PATH>` works! * move out shared tmp staging path setup to a fixture to fix the test
2021-06-01Fix bug where cmake prefix path on the command line does not include ↵Harmen Stoppels2-18/+20
transitive deps (#23965)
2021-06-01Simplified the spack.util.gpg implementation (#23889)Massimiliano Culpo10-368/+304
* Simplified the spack.util.gpg implementation All the classes defined in this Python module, which were previously used to construct singleton instances, have been removed in favor of four global variables. These variables are initialized lazily, like before. The API of the module has been unchanged for the most part. A few tests have been modified to use the new global names.
2021-06-01Fix leading / during spack buildcache -f ... (#24028)Harmen Stoppels1-1/+1
For me the buildcache force overwrite option does not work. It tries to delete a file, but errors with a key error, apparently because the leading / has to be removed.
2021-05-31Log performance improvement (#23925)Tom Scogland1-22/+37
* util.tty.log: read up to 100 lines if ready Rework to read up to 100 lines from the captured stdin as long as data is ready to be read immediately. Adds a helper function to poll with `select` for ready data. This showed a roughly 5-10x perf improvement for high-rate writes through the logger with relatively short lines. * util.tty.log: Defer flushes to end of ready reads Rather than flush per line, flush per set of reads. Since this is a non-blocking loop, the total perceived wait is short. * util.tty.log: only scan each line once, usually Rather than always find all control characters then substitute them all, use `subn` to count the number of control characters replaced. Only if control characters exist find out what they are. This could be made truly single pass with sub with a function, but it's a more intrusive change and this got 99%ish of the performance improvement (roughly another 2x in some cases). * util.tty.log: remove check for `readable` Python < 3 does not support a readable check on streams, should not be necessary here since we control the only use and it's explicitly a stream to be read.
2021-05-28adding support for export of private gpg key (#22557)Vanessasaurus5-12/+115
This PR allows users to `--export`, `--export-secret`, or both to export GPG keys from Spack. The docs are updated that include a warning that this usually does not need to be done. This addresses an issue brought up in slack, and also represented in #14721. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-28Cache compiler lookup per package (#23988)Harmen Stoppels1-1/+3
Before: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.593 s ± 0.016 s [User: 1.468 s, System: 0.126 s] Range (min … max): 1.575 s … 1.628 s 10 runs ``` After: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.407 s ± 0.020 s [User: 1.280 s, System: 0.127 s] Range (min … max): 1.393 s … 1.455 s 10 runs ```
2021-05-28Separable module configuration -- without the bugs this time (#23703)Greg Becker31-200/+662
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment). This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention. As part of this change, the module roots configuration moved from the config section to inside each module configuration. Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
2021-05-28Pipelines: reproducible builds (#22887)Scott Wittenburg12-548/+1656
### Overview The goal of this PR is to make gitlab pipeline builds (especially build failures) more reproducible outside of the pipeline environment. The two key changes here which aim to improve reproducibility are: 1. Produce a `spack.lock` during pipeline generation which is passed to child jobs via artifacts. This concretized environment is used both by generated child jobs as well as uploaded as an artifact to be used when reproducing the build locally. 2. In the `spack ci rebuild` command, if a spec needs to be rebuilt from source, do this by generating and running an `install.sh` shell script which is then also uploaded as a job artifact to be run during local reproduction. To make it easier to take advantage of improved build reproducibility, this PR also adds a new subcommand, `spack ci reproduce-build`, which, given a url to job artifacts: - fetches and unzips the job artifacts to a local directory - looks for the generated pipeline yaml and parses it to find details about the job to reproduce - attempts to provide a copy of the same version of spack used in the ci build - if the ci build used a docker image, the command prints a `docker run` command you can run to get an interactive shell for reproducing the build #### Some highlights One consequence of this change will be much smaller pipeline yaml files. By encoding the concrete environment in a `spack.lock` and passing to child jobs via artifacts, we will no longer need to encode the concrete root of each spec and write it into the job variables, greatly reducing the size of the generated pipeline yaml. Additionally `spack ci rebuild` output (stdout/stderr) is no longer internally redirected to a log file, so job output will appear directly in the gitlab job trace. With debug logging turned on, this often results in log files getting truncated because they exceed the maximum amount of log output gitlab allows. If this is a problem, you still have the option to `tee` command output to a file in the within the artifacts directory, as now each generated job exposes a `user_data` directory as an artifact, which you can fill with whatever you want in your custom job scripts. There are some changes to be aware of in how pipelines should be set up after this PR: #### Pipeline generation Because the pipeline generation job now writes a `spack.lock` artifact to be consumed by generated downstream jobs, `spack ci generate` takes a new option `--artifacts-root`, inside which it creates a `concrete_env` directory to place the lockfile. This artifacts root directory is also where the `user_data` directory will live, in case you want to generate any custom artifacts. If you do not provide `--artifacts-root`, the default is for it to create a `jobs_scratch_dir` within your `CI_PROJECT_DIR` (a gitlab predefined environment variable) or whatever is your current working directory if that variable isn't set. Here's the diff of the PR testing `.gitlab-ci.yml` taking advantage of the new option: ``` $ git diff develop..pipelines-reproducible-builds share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml diff --git a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml index 579d7b56f3..0247803a30 100644 --- a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml +++ b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml @@ -28,10 +28,11 @@ default: - cd share/spack/gitlab/cloud_pipelines/stacks/${SPACK_CI_STACK_NAME} - spack env activate --without-view . - spack ci generate --check-index-only + --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir" --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml" artifacts: paths: - - "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml" + - "${CI_PROJECT_DIR}/jobs_scratch_dir" tags: ["spack", "public", "medium", "x86_64"] interruptible: true ``` Notice how we replaced the specific pointer to the generated pipeline file with its containing folder, the same folder we passed as `--artifacts-root`. This way anything in that directory (the generated pipeline yaml, as well as the concrete environment directory containing the `spack.lock`) will be uploaded as an artifact and available to the downstream jobs. #### Rebuild jobs Rebuild jobs now must activate the concrete environment created by `spack ci generate` and provided via artifacts. When the pipeline is generated, a directory called `concrete_environment` is created within the artifacts root directory, and this is where the `spack.lock` file is written to be passed to the generated rebuild jobs. The artifacts root directory can be specified using the `--artifacts-root` option to `spack ci generate`, otherwise, it is assumed to be `$CI_PROJECT_DIR`. The directory containing the concrete environment files (`spack.yaml` and `spack.lock`) is then passed to generated child jobs via the `SPACK_CONCRETE_ENV_DIR` variable in the generated pipeline yaml file. When you don't provide custom `script` sections in your `mappings` within the `gitlab-ci` section of your `spack.yaml`, the default behavior of rebuild jobs is now to change into `SPACK_CONCRETE_ENV_DIR` and activate that environment. If you do provide custom rebuild scripts in your `spack.yaml`, be aware those scripts should do the same thing: assume `SPACK_CONCRETE_ENV_DIR` contains the concretized environment to activate. No other changes to existing custom rebuild scripts should be required as a result of this PR. As mentioned above, one key change made in this PR is the generation of the `install.sh` script by the rebuild jobs, as that same script is both run by the CI rebuild job as well as exported as an artifact to aid in subsequent attempts to reproduce the build outside of CI. The generated `install.sh` script contains only a single `spack install` command with arguments computed by `spack ci rebuild`. If the install fails, the job trace in gitlab will contain instructions on how to reproduce the build locally: ``` To reproduce this build locally, run: spack ci reproduce-build https://gitlab.next.spack.io/api/v4/projects/7/jobs/240607/artifacts [--working-dir <dir>] If this project does not have public pipelines, you will need to first: export GITLAB_PRIVATE_TOKEN=<generated_token> ... then follow the printed instructions. ``` When run locally, the `spack ci reproduce-build` command shown above will download and process the job artifacts from gitlab, then print out instructions you can copy-paste to run a local reproducer of the CI job. This PR includes a few other changes to the way pipelines work, see the documentation on pipelines for more details. This PR erelies on ~- [ ] #23194 to be able to refer to uninstalled specs by DAG hash~ EDIT: that is going to take longer to come to fruition, so for now, we will continue to install specs represented by a concrete `spec.yaml` file on disk. - [x] #22657 to support install a single spec already present in the active, concrete environment
2021-05-28aocc version detection (#23907)Greg Becker1-1/+4
2021-05-27bugfix: mirror index shows missing packages (#23939)Greg Becker3-4/+66
- [x] add `in_buildcache` field to DB records to indicate what parts of an index, which includes roots and dependencies, are in the buildcache. - [x] add `mark()` method to DB for setting values on single nodes of the DAG.
2021-05-27only readlink on links (#23948)Greg Becker1-1/+1
2021-05-26allow whitespace formatting in variant descriptions (#23853)Greg Becker1-4/+6
2021-05-25adding json export for spack blame (#23417)Vanessasaurus3-14/+121
I would like to be able to export (and save and then load programatically) spack blame metadata, so this commit adds a spack blame --json argument, along with developer docs for it Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-25first set of work to allow for saving local results with spack monitor (#23804)Vanessasaurus5-22/+154
This work will come in two phases. The first here is to allow saving of a local result with spack monitor, and the second will add a spack monitor command so the user can do spack monitor upload. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-25Fix cross references in inteloneapipackage doc (#23744)Robert Cohn1-7/+7
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-05-25Stand-alone/Smoke tests: copy cached test sources to test stage (#23713)Tamara Dahlgren8-26/+104
2021-05-25MesonPackage: make "default_library" a multi-valued variant (#23540)Harmen Stoppels1-6/+7
Currently if one package does `depends_on('pkg default_library=shared')` and another does `depends_on('pkg default_library=both')`, you'd get a concretization error. With this PR one package can do `depends_on('pkg default_library=shared')` and another depends_on('default_library=static'), and it would concretize to `pkg default_library=shared,static` Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-05-25Fix packaging guide table's build system links (#23879)Tamara Dahlgren1-9/+9
2021-05-25Fix hyperlink formatting in docs (#23846)HDF-EOS Tools Information Center1-1/+1
2021-05-22Fix makefile filter suggestions (#23856)Seth R. Johnson2-6/+8
Bash has a builtin `fc` that will override the compiler if you use "fc", so it's better to use the full spack-supplied compiler path. Additionally, the filter regex in the docs was wrong: it replaced the entire assignment operation with the RHS.
2021-05-21Modification to R environment (#23623)Glenn Johnson1-2/+1
* Modification to R environment This PR modifies how the R environmnet is presented, and fixes installing the standalone Rmath library. - The Rmath build and install methods are combined into one - Set parallel=False when installing Rmath - remove the run environment that set up variables for libraries and headers that are not really needed, and pollute the environment. * Add setup_run_environment back - Add back the setup_run_environment with LD_LIBRARY_PATH and PKG_CONFIG_PATH. - Adjust documentation to reflect the current code.
2021-05-20spack: update archspecMassimiliano Culpo3-7/+172
This fixes the detection of Apple M1 and adds virtual levels for x86_64 architectures
2021-05-20versions: do not drop 2.0.X if 2.0 is a declared version in the package (#23217)Valentin Volkl2-6/+30
2021-05-20Cray: fix extracting paths from module files (#23472)Harmen Stoppels2-3/+13
Co-authored-by: Tiziano Müller <tm@dev-zero.ch>
2021-05-19Write junit-report to reports directory to allow installation from ↵Valentin Volkl2-1/+3
read-only spack (#20158)
2021-05-19adding support to tag a buildvsoch3-5/+26
This will be useful to run multiple build experiments and organize by name Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-05-19Add caret/hat to spack spec help documentation (#23758)Jamie Finney1-0/+1
Co-authored-by: Jamie Finney <finneyjm@ornl.gov>
2021-05-19monitor: fix issue with attribute lookup (#23773)Vanessasaurus1-1/+1
Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-18docs/packaging guide: Reference test stage directory (#23707)Tamara Dahlgren1-3/+15
2021-05-18Bugfix: fetching oddly-named resources from local mirrors (#23300)Michael Kuhn1-1/+9
Spack uses curl to fetch URL resources. For locally-stored resources it uses curl's file protocol; when using this protocol, curl expects that the URL encoding conforms to RFC 3986 (which reserves characters like '?' and '=' for special use). We were not performing this encoding, and found a resource where curl was interpreting this in an unfavorable way (succeeding, but producing an empty file). This commit properly encodes URLs when using curl's file protocol. This error did not likely come up before because in most contexts Spack was either fetching via http or it was using URLs without offending characters (for example, the sha-based URLs in mirrors never contain these characters).
2021-05-17Revert "Separable module configurations (#22588)" (#23674)Harmen Stoppels29-491/+183
This reverts commit cefbe48c89209dc3df654795644973b1885cdea4.
2021-05-17performance: speed up existence checks in packages (#23661)Todd Gamblin3-24/+37
Spack doesn't require users to manually index their repos; it reindexes the indexes automatically when things change. To determine when to do this, it has to `stat()` all package files in each repository to make sure that indexes up to date with packages. We currently index virtual providers, patches by sha256, and tags on packages. When this was originally implemented, we ran the checker all the time, at startup, but that was slow (see #7587). But we didn't go far enough -- it still consults the checker and does all the stat operations just to see if a package exists (`Repo.exists()`). That might've been a wash in 2018, but as the number of packages has grown, it's gotten slower -- checking 5k packages is expensive and users see this for small operations. It's a win now to make `Repo.exists()` check files directly. **Fix:** This PR does a number of things to speed up `spack load`, `spack info`, and other commands: - [x] Make `Repo.exists()` check files directly again with `os.path.exists()` (this is the big one) - [x] Refactor `Spec.satisfies()` so that a checking for virtual packages only happens if needed (avoids some calls to exists()) - [x] Avoid calling `Repo.exists(spec)` in `Repo.get()`. `Repo.get()` will ultimately try to load a `package.py` file anyway; we can let the failure to load it indicate that the package doesn't exist, and avoid another call to exists(). - [x] Fix up some comments in spec parsing - [x] Call `UnknownPackageError` more consistently in `repo.py`
2021-05-15some fixes for command help strings (#23658)Todd Gamblin3-8/+5
- [x] `analyze` isn't commonly used; move it to long help (`spack -H` vs `spack -h`). Give it its own section. - [x] make it clear from `spack -h` that `spack module` can generate module files - [x] shorten help for `spack style`
2021-05-15do not sort projections alphabetically (#23649)Greg Becker1-3/+7
* do not sort projections alphabetically * add assertion for ordered dict
2021-05-14Separable module configurations (#22588)Greg Becker29-183/+491
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment). This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention. As part of this change, the module roots configuration moved from the `config` section to inside each module configuration. Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec. TODO: - [x] code changes to support multiple module sets - [x] code changes to support modules relative to a view - [x] Tests for multiple module configurations - [x] Tests for modules relative to a view - [x] Backwards compatibility for module roots from config section - [x] Backwards compatibility for default module set without the name specified - [x] Tests for backwards compatibility
2021-05-13spec: simplify __str__ implementation (#23593)Massimiliano Culpo2-13/+10
The implementation for __str__ has been simplified to traverse the spec directly, and doesn't call anymore the flat_dependencies method. Dead code has been removed.
2021-05-13cc: change mode to ccld for loopopt edit (#23482)Frank Willmore2-4/+13
For configure (e.g. for hdf5) to pass, this option needs to be pulled out when invoked in ccld mode. I thought it had fixed the issue but I still saw it after that. After some digging, my guess is that I was able to get hdf5 to build with ifort instead of ifx. Lot of overlapping changes occurring at the time, as it were. There are still outstanding issues building hdf5 with ifx, and Intel is looking into what appears to be a compiler bug, but this manifests during build and is likely a separate issue. I have verified that the making the edit in 'ccld' mode removes the -loopopt=0 and enables hdf5 to pass configure. It should be fine to make the edit in 'ld' mode as well, but I have not tested that and didn't include an -or- condition for it.
2021-05-13config key error: fix format string (#23610)Greg Becker1-1/+1
2021-05-13env views: make view updates atomic (#23476)Greg Becker6-72/+172
Currently, environment views blink out of existence during the view regeneration, and are slowly built back up to their new and improved state. This is not good if other processes attempt to access the view -- they can see it in an inconsistent state. This PR fixes makes environment view updates atomic. This requires a level of indirection (via symlink, similar to nix or guix) from the view root to the underlying implementation on the filesystem. Now, an environment view at `/path/to/foo` is a symlink to `/path/to/._foo/<hash>`, where `<hash>` is a hash of the contents of the view. We construct the view in its content-keyed hash directory, create a new symlink to this directory, and atomically replace the symlink with one to the new view. This PR has a couple of other benefits: * It future-proofs environment views so that we can implement rollback. * It ensures that we don't leave users in an inconsistent state if building a new view fails for some reason. For background: * there is no atomic operation in posix that allows for a non-empty directory to be replaced. * There is an atomic `renameat2` in the linux kernel starting in version 3.15, but many filesystems don't support the system call, including NFS3 and NFS4, which makes it a poor implementation choice for an HPC tool, so we use the symlink approach that others tools like nix and guix have used successfully.
2021-05-12ASP-based solver: account for deprecated versions (#23491)Massimiliano Culpo4-1/+43
fixes #22351 The ASP-based solver now accounts for the presence in the DAG of deprecated versions and tries to minimize their number at highest priority.
2021-05-11Environments: add run deps to shell modifications (#23485)Peter Scheibel2-12/+42
When adding an Environment to a user's shell, Spack was only adding root specs. This now includes run dependencies of root specs.
2021-05-11ASP-based solver: variants set from cli are considered as defaults (#23542)Massimiliano Culpo3-7/+40
Variants explicitly set in an abstract root spec are considered as defaults for the package they refer to, and they override what is in packages.yaml and in package.py. This is relevant only for multi-valued variants, where a constraint may extend an already default value.
2021-05-11cray: fix parsing of module list (#23566)Howard Pritchard1-0/+1
The code for guessing cpu archtype based on craype modules names got confused, at least on LLNL RZ prototype systems. In particular a (L) or (D) at the end of a craype-x86-xxx or other cpu architecture module was geting the logic confused. With this patch, any white space + remaining characters in the moduel name are removed. Signed-off-by: Howard Pritchard <howardp@lanl.gov>
2021-05-11Updates and format tweaks to the release documentation (#22053)Tamara Dahlgren1-93/+131
2021-05-11Documentation: Refinement of "Checking an installation" (#22210)Tamara Dahlgren3-164/+703
There have been a lot of questions and some confusion recently surrounding Spack installation test capabilities so this PR is intended to clean up and refine the documentation for "Checking an installation". It aims to better distinguish between checks that are performed during an installation (i.e., build-time tests) and those that can be done days and weeks after the software has been installed (i.e., install (or smoke) tests).
2021-05-10bugfix: correct force_autoreconf method syntax (#23546)Tamara Dahlgren1-1/+1