summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-06-15adding spack upload command (#24321)Vanessasaurus3-8/+121
this will first support uploads for spack monitor, and eventually could be used for other kinds of spack uploads Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-14extending example for buildcaches (#22504)Vanessasaurus1-6/+78
* extending example for buildcaches I was attempting to create a local build cache from a directory, and I found the docs for both buildcaches and mirrors, but did not connect the docs that the url variable could be the local filesystem variable. I am extending the docs for buildcaches with an example of creating and interacting with one on the filesystem because I suspect other users will run into this need and possibly not find what they are looking for. Signed-off-by: vsoch <vsoch@users.noreply.github.com> * adding as follows to spack mirror list Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-06-14adding more description to binary caches (#23934)Vanessasaurus1-0/+23
It is currently kind of confusing to the reader to distinguish spack buildcache install and spack install, and it is not clear how to use a build cache once a mirror is added. Hopefully this little big of description can help (and I hope I got it right!) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-14oneAPI packages: fix install for python2 (#24296)Robert Cohn1-2/+2
Fix platform detection logic to work for Python 2 and 3
2021-06-12IntelPackage: use 'version_yearlike' in check for libfabrics RPATH. (#16700)Dominik Gresch1-1/+1
Use the 'version_yearlike' attribute instead of 'version' to check if the SPACK_COMPILER_EXTRA_RPATHS should be set to include the built-in 'libfabrics'. When using the bare 'version', the comparison is wrong when building with 'intel-parallel-studio', which has the version format '<edition>.YYYY.Nupdate', due to the leading '<edition>'.
2021-06-12Ensure all roots of an installed environment are marked explicit in db (#24277)Greg Becker2-0/+28
2021-06-11setup-env: allow users to skip module function setup (#24236)Adam J. Stewart1-3/+9
* setup-env: allow users to skip module function setup * Add documentation on SPACK_SKIP_MODULES
2021-06-11Display proper message when patch checksum doesn't match (#24229)iarspider1-1/+1
2021-06-08Pipelines: Fix default generated rebuild job script (#24185)Scott Wittenburg1-2/+1
2021-06-08ASP-based solver: permit to use virtual specs in environments (#24199)Massimiliano Culpo4-17/+50
Extracting specs for the result of a solve has been factored as a method into the asp.Result class. The method account for virtual specs being passed as initial requests.
2021-06-08ASP-based solver: reordered low priority optimization criteria (#24184)Massimiliano Culpo2-16/+25
Minimizing compiler mismatches in the DAG and preferring newer versions of packages are now higher priority than trying to use as many default values as possible in multi-valued variants.
2021-06-08macOS: add monterey as macOS version 12. (#24192)Todd Gamblin1-1/+2
2021-06-07Fix brittle unit-tests on providers (#24186)Massimiliano Culpo1-2/+0
These tests were broken by #24176
2021-06-07build_systems: Make autotools builds verbose (#24161)Michael Kuhn1-1/+4
This is also what our other build systems are doing.
2021-06-07Docs: fix missing backtick in Environments docs (#24109)Adam J. Stewart1-1/+1
2021-06-05Fix git_version on macOS (#24163)Adam J. Stewart1-2/+3
2021-06-04main, modules: fix module roots not being found (#24135)Michael Kuhn2-10/+5
Since the module roots were removed from the config file, `--print-shell-vars` cannot find the module roots anymore. Fix it by using the new `root_path` function. Moreover, the roots for lmod and modules seems to have been flipped by accident.
2021-06-04enable std c++14 (#24127)Joe Heaton1-0/+6
2021-06-04cmd/stage: print stage path (#24019)Michael Kuhn1-0/+3
This is a small quality of life improvement so that users can easily copy and paste the stage path after executing `spack stage spec`.
2021-06-04bugfix: modules relative to view use top-level view root, not implementation ↵Greg Becker3-4/+4
root (#24124)
2021-06-04Don't warn about missing source id for external packages (#24125)Adam J. Stewart1-2/+3
2021-06-04Speed-up version parsing and comparison (#22973)Tom Scogland2-52/+84
The VALID_VERSION regex didn't check that the version string was completely valid, only that a prefix of it was. This version ensures the entire string represents a valid version. This makes a few related changes. 1. Make the SEGMENT_REGEX identify *which* arm it matches by what groups are populated, including whether it's a string or int component or a separator all at once. 2. Use the updated regex to parse the input once with a findall rather than twice, once with findall and once with split, since the version components and separators can be distinguished by their group status. 3. Rather than "convert to int, on exception stay string," if the int group is set then convert to int, if not then construct an instance of the VersionStrComponent class 4. VersionStrComponent now implements all of the special string comparison logic as part of its __lt__ and __eq__ methods to deal with infinity versions and also overloads comparison with integers. 5. Version now uses direct tuple comparison since it has no per-element special logic outside the VersionStrComponent class. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-06-03Pipelines: Fix generation when dep and pkg arch differ (#24089)Scott Wittenburg1-1/+1
2021-06-03Extend cuda conflicts to cray platform (#24057)Harmen Stoppels1-61/+52
The CUDA compiler conflicts are valid on Cray too, and likely on Darwin x86_64 with %gcc and %clang too, so drop platform=linux
2021-06-02Pipelines: pass relative artifact paths to child jobs (#24085)Scott Wittenburg4-34/+104
Passing absolute paths from pipeline generate job to downstream rebuild jobs causes problems when the CI_PROJECT_DIR is not the same for the generate and rebuild jobs. This has happened, for example, when gitlab checks out the project into a runner-specific directory and different runners are chosen for the generate and rebuild jobs.
2021-06-02ensure the staging dir exists for `spack stage -p <PATH>` (#23963)Danny McClanahan4-20/+43
* ensure that the stage root exists for `spack stage -p <PATH>` * add test to verify `spack stage -p <PATH>` works! * move out shared tmp staging path setup to a fixture to fix the test
2021-06-01Fix bug where cmake prefix path on the command line does not include ↵Harmen Stoppels2-18/+20
transitive deps (#23965)
2021-06-01Simplified the spack.util.gpg implementation (#23889)Massimiliano Culpo10-368/+304
* Simplified the spack.util.gpg implementation All the classes defined in this Python module, which were previously used to construct singleton instances, have been removed in favor of four global variables. These variables are initialized lazily, like before. The API of the module has been unchanged for the most part. A few tests have been modified to use the new global names.
2021-06-01Fix leading / during spack buildcache -f ... (#24028)Harmen Stoppels1-1/+1
For me the buildcache force overwrite option does not work. It tries to delete a file, but errors with a key error, apparently because the leading / has to be removed.
2021-05-31Log performance improvement (#23925)Tom Scogland1-22/+37
* util.tty.log: read up to 100 lines if ready Rework to read up to 100 lines from the captured stdin as long as data is ready to be read immediately. Adds a helper function to poll with `select` for ready data. This showed a roughly 5-10x perf improvement for high-rate writes through the logger with relatively short lines. * util.tty.log: Defer flushes to end of ready reads Rather than flush per line, flush per set of reads. Since this is a non-blocking loop, the total perceived wait is short. * util.tty.log: only scan each line once, usually Rather than always find all control characters then substitute them all, use `subn` to count the number of control characters replaced. Only if control characters exist find out what they are. This could be made truly single pass with sub with a function, but it's a more intrusive change and this got 99%ish of the performance improvement (roughly another 2x in some cases). * util.tty.log: remove check for `readable` Python < 3 does not support a readable check on streams, should not be necessary here since we control the only use and it's explicitly a stream to be read.
2021-05-28adding support for export of private gpg key (#22557)Vanessasaurus5-12/+115
This PR allows users to `--export`, `--export-secret`, or both to export GPG keys from Spack. The docs are updated that include a warning that this usually does not need to be done. This addresses an issue brought up in slack, and also represented in #14721. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-28Cache compiler lookup per package (#23988)Harmen Stoppels1-1/+3
Before: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.593 s ± 0.016 s [User: 1.468 s, System: 0.126 s] Range (min … max): 1.575 s … 1.628 s 10 runs ``` After: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.407 s ± 0.020 s [User: 1.280 s, System: 0.127 s] Range (min … max): 1.393 s … 1.455 s 10 runs ```
2021-05-28Separable module configuration -- without the bugs this time (#23703)Greg Becker31-200/+662
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment). This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention. As part of this change, the module roots configuration moved from the config section to inside each module configuration. Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
2021-05-28Pipelines: reproducible builds (#22887)Scott Wittenburg12-548/+1656
### Overview The goal of this PR is to make gitlab pipeline builds (especially build failures) more reproducible outside of the pipeline environment. The two key changes here which aim to improve reproducibility are: 1. Produce a `spack.lock` during pipeline generation which is passed to child jobs via artifacts. This concretized environment is used both by generated child jobs as well as uploaded as an artifact to be used when reproducing the build locally. 2. In the `spack ci rebuild` command, if a spec needs to be rebuilt from source, do this by generating and running an `install.sh` shell script which is then also uploaded as a job artifact to be run during local reproduction. To make it easier to take advantage of improved build reproducibility, this PR also adds a new subcommand, `spack ci reproduce-build`, which, given a url to job artifacts: - fetches and unzips the job artifacts to a local directory - looks for the generated pipeline yaml and parses it to find details about the job to reproduce - attempts to provide a copy of the same version of spack used in the ci build - if the ci build used a docker image, the command prints a `docker run` command you can run to get an interactive shell for reproducing the build #### Some highlights One consequence of this change will be much smaller pipeline yaml files. By encoding the concrete environment in a `spack.lock` and passing to child jobs via artifacts, we will no longer need to encode the concrete root of each spec and write it into the job variables, greatly reducing the size of the generated pipeline yaml. Additionally `spack ci rebuild` output (stdout/stderr) is no longer internally redirected to a log file, so job output will appear directly in the gitlab job trace. With debug logging turned on, this often results in log files getting truncated because they exceed the maximum amount of log output gitlab allows. If this is a problem, you still have the option to `tee` command output to a file in the within the artifacts directory, as now each generated job exposes a `user_data` directory as an artifact, which you can fill with whatever you want in your custom job scripts. There are some changes to be aware of in how pipelines should be set up after this PR: #### Pipeline generation Because the pipeline generation job now writes a `spack.lock` artifact to be consumed by generated downstream jobs, `spack ci generate` takes a new option `--artifacts-root`, inside which it creates a `concrete_env` directory to place the lockfile. This artifacts root directory is also where the `user_data` directory will live, in case you want to generate any custom artifacts. If you do not provide `--artifacts-root`, the default is for it to create a `jobs_scratch_dir` within your `CI_PROJECT_DIR` (a gitlab predefined environment variable) or whatever is your current working directory if that variable isn't set. Here's the diff of the PR testing `.gitlab-ci.yml` taking advantage of the new option: ``` $ git diff develop..pipelines-reproducible-builds share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml diff --git a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml index 579d7b56f3..0247803a30 100644 --- a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml +++ b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml @@ -28,10 +28,11 @@ default: - cd share/spack/gitlab/cloud_pipelines/stacks/${SPACK_CI_STACK_NAME} - spack env activate --without-view . - spack ci generate --check-index-only + --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir" --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml" artifacts: paths: - - "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml" + - "${CI_PROJECT_DIR}/jobs_scratch_dir" tags: ["spack", "public", "medium", "x86_64"] interruptible: true ``` Notice how we replaced the specific pointer to the generated pipeline file with its containing folder, the same folder we passed as `--artifacts-root`. This way anything in that directory (the generated pipeline yaml, as well as the concrete environment directory containing the `spack.lock`) will be uploaded as an artifact and available to the downstream jobs. #### Rebuild jobs Rebuild jobs now must activate the concrete environment created by `spack ci generate` and provided via artifacts. When the pipeline is generated, a directory called `concrete_environment` is created within the artifacts root directory, and this is where the `spack.lock` file is written to be passed to the generated rebuild jobs. The artifacts root directory can be specified using the `--artifacts-root` option to `spack ci generate`, otherwise, it is assumed to be `$CI_PROJECT_DIR`. The directory containing the concrete environment files (`spack.yaml` and `spack.lock`) is then passed to generated child jobs via the `SPACK_CONCRETE_ENV_DIR` variable in the generated pipeline yaml file. When you don't provide custom `script` sections in your `mappings` within the `gitlab-ci` section of your `spack.yaml`, the default behavior of rebuild jobs is now to change into `SPACK_CONCRETE_ENV_DIR` and activate that environment. If you do provide custom rebuild scripts in your `spack.yaml`, be aware those scripts should do the same thing: assume `SPACK_CONCRETE_ENV_DIR` contains the concretized environment to activate. No other changes to existing custom rebuild scripts should be required as a result of this PR. As mentioned above, one key change made in this PR is the generation of the `install.sh` script by the rebuild jobs, as that same script is both run by the CI rebuild job as well as exported as an artifact to aid in subsequent attempts to reproduce the build outside of CI. The generated `install.sh` script contains only a single `spack install` command with arguments computed by `spack ci rebuild`. If the install fails, the job trace in gitlab will contain instructions on how to reproduce the build locally: ``` To reproduce this build locally, run: spack ci reproduce-build https://gitlab.next.spack.io/api/v4/projects/7/jobs/240607/artifacts [--working-dir <dir>] If this project does not have public pipelines, you will need to first: export GITLAB_PRIVATE_TOKEN=<generated_token> ... then follow the printed instructions. ``` When run locally, the `spack ci reproduce-build` command shown above will download and process the job artifacts from gitlab, then print out instructions you can copy-paste to run a local reproducer of the CI job. This PR includes a few other changes to the way pipelines work, see the documentation on pipelines for more details. This PR erelies on ~- [ ] #23194 to be able to refer to uninstalled specs by DAG hash~ EDIT: that is going to take longer to come to fruition, so for now, we will continue to install specs represented by a concrete `spec.yaml` file on disk. - [x] #22657 to support install a single spec already present in the active, concrete environment
2021-05-28aocc version detection (#23907)Greg Becker1-1/+4
2021-05-27bugfix: mirror index shows missing packages (#23939)Greg Becker3-4/+66
- [x] add `in_buildcache` field to DB records to indicate what parts of an index, which includes roots and dependencies, are in the buildcache. - [x] add `mark()` method to DB for setting values on single nodes of the DAG.
2021-05-27only readlink on links (#23948)Greg Becker1-1/+1
2021-05-26allow whitespace formatting in variant descriptions (#23853)Greg Becker1-4/+6
2021-05-25adding json export for spack blame (#23417)Vanessasaurus3-14/+121
I would like to be able to export (and save and then load programatically) spack blame metadata, so this commit adds a spack blame --json argument, along with developer docs for it Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-25first set of work to allow for saving local results with spack monitor (#23804)Vanessasaurus5-22/+154
This work will come in two phases. The first here is to allow saving of a local result with spack monitor, and the second will add a spack monitor command so the user can do spack monitor upload. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-05-25Fix cross references in inteloneapipackage doc (#23744)Robert Cohn1-7/+7
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-05-25Stand-alone/Smoke tests: copy cached test sources to test stage (#23713)Tamara Dahlgren8-26/+104
2021-05-25MesonPackage: make "default_library" a multi-valued variant (#23540)Harmen Stoppels1-6/+7
Currently if one package does `depends_on('pkg default_library=shared')` and another does `depends_on('pkg default_library=both')`, you'd get a concretization error. With this PR one package can do `depends_on('pkg default_library=shared')` and another depends_on('default_library=static'), and it would concretize to `pkg default_library=shared,static` Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-05-25Fix packaging guide table's build system links (#23879)Tamara Dahlgren1-9/+9
2021-05-25Fix hyperlink formatting in docs (#23846)HDF-EOS Tools Information Center1-1/+1
2021-05-22Fix makefile filter suggestions (#23856)Seth R. Johnson2-6/+8
Bash has a builtin `fc` that will override the compiler if you use "fc", so it's better to use the full spack-supplied compiler path. Additionally, the filter regex in the docs was wrong: it replaced the entire assignment operation with the RHS.
2021-05-21Modification to R environment (#23623)Glenn Johnson1-2/+1
* Modification to R environment This PR modifies how the R environmnet is presented, and fixes installing the standalone Rmath library. - The Rmath build and install methods are combined into one - Set parallel=False when installing Rmath - remove the run environment that set up variables for libraries and headers that are not really needed, and pollute the environment. * Add setup_run_environment back - Add back the setup_run_environment with LD_LIBRARY_PATH and PKG_CONFIG_PATH. - Adjust documentation to reflect the current code.
2021-05-20spack: update archspecMassimiliano Culpo3-7/+172
This fixes the detection of Apple M1 and adds virtual levels for x86_64 architectures
2021-05-20versions: do not drop 2.0.X if 2.0 is a declared version in the package (#23217)Valentin Volkl2-6/+30
2021-05-20Cray: fix extracting paths from module files (#23472)Harmen Stoppels2-3/+13
Co-authored-by: Tiziano Müller <tm@dev-zero.ch>