summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2023-10-23Adios2: add kokkos variant (#40623)Vicente Bolea3-1/+5
* adios2: update variants and dependencies * adios2: add kokkos rocm|cuda|sycl variant * e4s oneapi ci stack: add adios2 +sycl * e4s ci stack: add adios2 +rocm * [@spackbot] updating style on behalf of vicentebolea * Apply suggestions from code review * adios2: fixed cuda variant * update ecp-data-vis-sdk * Update share/spack/gitlab/cloud_pipelines/stacks/e4s-power/spack.yaml --------- Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com> Co-authored-by: vicentebolea <vicentebolea@users.noreply.github.com>
2023-10-20concretize separately: show concretization time per spec as they concretize ↵Harmen Stoppels1-2/+2
when verbose (#40634)
2023-10-20gromacs +cp2k: build in CI (#40494)Harmen Stoppels3-0/+3
* gromacs +cp2k: build in CI * libxsmm: x86 only * attempt to fix dbcsr + new mpich * use c11 standard * gromacs: does not depend on dbcsr * cp2k: build with cmake in CI, s.t. dbcsr is a separate package * cp2k: cmake patches for config files and C/C++ std * cp2k: remove unnecessary constraints due to patch
2023-10-19Improve setup build / run / test environment (#35737)Harmen Stoppels2-7/+5
This adds a `SetupContext` class which is responsible for setting package.py module globals, and computing the changes to environment variables for the build, test or run context. The class uses `effective_deptypes` which takes a list of specs (e.g. single item of a spec to build, or a list of environment roots) and a context (build, run, test), and outputs a flat list of specs that affect the environment together with a flag in what way they do so. This list is topologically ordered from root to leaf, so that one can be assured that dependents override variables set by dependencies, not the other way around. This is used to replace the logic in `modifications_from_dependencies`, which has several issues: missing calls to `setup_run_environment`, and the order in which operations are applied. Further, it should improve performance a bit in certain cases, since `effective_deptypes` run in O(v + e) time, whereas `spack env activate` currently can take up to O(v^2 + e) time due to loops over roots. Each edge in the DAG is visited once by calling `effective_deptypes` with `env.concrete_roots()`. By marking and propagating flags through the DAG, this commit also fixes a bug where Spack wouldn't call `setup_run_environment` for runtime dependencies of link dependencies. And this PR ensures that Spack correctly sets up the runtime environment of direct build dependencies. Regarding test dependencies: in a build context they are are build-time test deps, whereas in a test context they are install-time test deps. Since there are no means to distinguish the build/install type test deps, they're both. Further changes: - all `package.py` module globals are guaranteed to be set before any of the `setup_(dependent)_(run|build)_env` functions is called - traversal order during setup: first the group of externals, then the group of non-externals, with specs in each group traversed topological (dependencies are setup before dependents) - modules: only ever call `setup_dependent_run_environment` of *direct* link/run type deps - the marker in `set_module_variables_for_package` is dropped, since we should call the method once per spec. This allows us to set only a cheap subset of globals on the module: for example it's not necessary to compute the expensive `cmake_args` and w/e if the spec under consideration is not the root node to be built. - `spack load`'s `--only` is deprecated (it has no effect now), and `spack load x` now means: do everything that's required for `x` to work at runtime, which requires runtime deps to be setup -- just like `spack env activate`. - `spack load` no longer loads build deps (of build deps) ... - `spack env activate` on partially installed or broken environments: this is all or nothing now. If some spec errors during setup of its runtime env, you'll only get the unconditional variables + a warning that says the runtime changes for specs couldn't be applied. - Remove traversal in upward direction from `setup_dependent_*` in packages. Upward traversal may iterate to specs that aren't children of the roots (e.g. zlib / python have hundreds of dependents, only a small fraction is reachable from the roots. Packages should only modify the direct dependent they receive as an argument)
2023-10-19gitlab ci: Rework how mirrors are configured (#39939)Scott Wittenburg30-81/+77
Improve how mirrors are used in gitlab ci, where we have until now thought of them as only a string. By configuring ci mirrors ahead of time using the proposed mirror templates, and by taking advantage of the expressiveness that spack now has for mirrors, this PR will allow us to easily switch the protocol/url we use for fetching binary dependencies. This change also deprecates some gitlab functionality and marks it for removal in Spack 0.23: - arguments to "spack ci generate": * --buildcache-destination * --copy-to - gitlab configuration options: * enable-artifacts-buildcache * temporary-storage-url-prefix
2023-10-19ci: remove incorrect compilers.yaml (#40610)Harmen Stoppels1-27/+0
2023-10-18AutotoolsPackage / MakefilePackage: add gmake build dependency (#40380)Harmen Stoppels2-36/+36
2023-10-17Support `spack env activate --with-view <name> <env>` (#40549)Harmen Stoppels2-6/+6
Currently `spack env activate --with-view` exists, but is a no-op. So, it is not too much of a breaking change to make this redundant flag accept a value `spack env activate --with-view <name>` which activates a particular view by name. The view name is stored in `SPACK_ENV_VIEW`. This also fixes an issue where deactivating a view that was activated with `--without-view` possibly removes entries from PATH, since now we keep track of whether the default view was "enabled" or not.
2023-10-12Remove deprecated "extra_instructions" option for containers (#40365)Massimiliano Culpo2-13/+0
2023-10-11Update bootstrap buildcache to support Python 3.12 (#40404)Massimiliano Culpo6-479/+650
* Add support for Python 3.12 * Use optimized build of clingo
2023-10-10e4s arm stack: duplicate and target neoverse v1 (#40369)eugeneswalker2-13/+17
* e4s arm stack: duplicate and target both neoverse n1, v1 * remove neoverse_n1 target until issue #40397 is resolved
2023-10-06Change 'exit' to 'return' in setup-env.sh (#36137)Alex Richert1-1/+1
* Change 'exit' to 'return' in `setup-env.sh` to avoid losing shell in some cases when sourcing twice.
2023-10-06Make "minimal" the default duplicate strategy (#39621)Massimiliano Culpo1-10/+6
* Allow branching out of the "generic build" unification set For cases like the one in https://github.com/spack/spack/pull/39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
2023-10-05Revert "cray rhel: disable due to runner issues (#40324)" (#40335)eugeneswalker1-16/+16
This reverts commit bf7f54449ba8ed157c9ee258007e0a7a509600cf.
2023-10-05cray rhel: disable due to runner issues (#40324)Harmen Stoppels1-16/+16
2023-10-04ci: pull E4S images from github instead of dockerhub (#40307)Scott Wittenburg6-21/+21
2023-10-04e4s arm: disable bricks due to target=aarch64 not being respected (#40308)eugeneswalker1-6/+6
2023-10-03e4s ci stacks: sync with e4s-23.08 (#40263)eugeneswalker6-289/+1212
* e4s amd64 gcc ci stack: sync with e4s-23.08 * e4s amd64 oneapi ci stack: sync with e4s-23.08 * e4s ppc64 gcc ci stack: sync with e4s-23.08 * add new ci stack: e4s amd64 gcc w/ external rocm * add new ci stack: e4s arm gcc ci * updates * py-scipy: -fvisibility issue is resolved in 2023.1.0: #39464 * paraview oneapi fails * comment out pkgs that fail to build on power * fix arm stack name * fix cabana +cuda specification * comment out failing spces * visit fails build on arm * comment out slepc arm builds due to make issue * comment out failing dealii arm builds
2023-10-03Deactivate Cray sles, due to unavailable runner (#40291)Massimiliano Culpo1-12/+12
This reverts commit 027409120408eb4d41d8bdd82a13c34b3ac7fea9.
2023-10-02Implemented +sycl for slate (#39927)G-Ragghianti1-0/+1
* Implemented +sycl for slate * style * style * add slate +sycl to ci * slate +sycl: explicitly specify +mpi * comment out slate+sycl+mpi in e4s oneapi stack * removing requirement of intel-oneapi-mpi * Added slate+sycl to e4s oneapi stack * Removing obsolete comment --------- Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2023-10-01py-scipy: -fvisibility issue is resolved in 2023.1.0: (#39464)eugeneswalker1-0/+3
* py-scipy: -fvisibility issue is resolved in 2023.1.0: * e4s oneapi ci: add py-scipy
2023-09-30e4s cray sles ci: expand spec list (#40162)eugeneswalker1-13/+127
* e4s cray sles stack: expand spec list * remove unnecessary packages:trilinos:one_of
2023-09-30e4s ci: add packages: drishti, dxt-explorer (#39597)eugeneswalker3-0/+6
* e4s ci: add packages: drishti, dxt-explorer * e4s oneapi ci: comment out dxt-explorer until r%oneapi issue #40257 is resolved
2023-09-29Test package detection in a systematic way (#18175)Massimiliano Culpo2-1/+19
This PR adds a new audit sub-command to check that detection of relevant packages is performed correctly in a few scenarios mocking real use-cases. The data for each package being tested is in a YAML file called detection_test.yaml alongside the corresponding package.py file. This is to allow encoding detection tests for compilers and other widely used tools, in preparation for compilers as dependencies.
2023-09-29Revert "e4s cray sles: suspend ci until machine issues resolved (#40219)" ↵eugeneswalker1-12/+12
(#40238) This reverts commit 5165524ca6eb9cb0b8c2e225352e4729b5a04062.
2023-09-27e4s cray sles: suspend ci until machine issues resolved (#40219)eugeneswalker1-12/+12
2023-09-24trilinos +shylu: patch to resolve trilinos issue #12048 (#40169)eugeneswalker1-1/+1
* trilinos +shylu: patch to resolve trilinos issue #12048 * e4s ci: build xyce ^trilinos+shylu --------- Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2023-09-22e4s cray rhel stack: expand to full spec list (#40111)eugeneswalker1-13/+122
* e4s cray rhel stack: expand to full spec list * comment out gasnet; require %gcc for unzip * require openblas@0.3.20 to get around %cce error; follow up with issue report * comment out failing specs; * comment out axom and xyce due to errors * improve clarity of failing specs
2023-09-21Revert "e4s-cray: drop in ci as there are no runners (#40122)" (#40144)eugeneswalker1-12/+12
This reverts commit bd8d0324a9a06c7acd81485b265ec2a48b579cec.
2023-09-21e4s: drop python 3.8 preference (#40123)Harmen Stoppels2-4/+0
2023-09-21e4s-cray: drop in ci as there are no runners (#40122)Harmen Stoppels1-12/+12
2023-09-19ci: dont use nightly image tag (#40070)Harmen Stoppels4-6/+6
2023-09-19Use process pool executors for web-crawling and retrieving archives (#39888)Massimiliano Culpo2-6/+8
Fix a race condition when searching urls, and updating a shared set '_visited'.
2023-09-19ci: drop redundant packages.yaml (#40068)Harmen Stoppels1-14/+0
2023-09-18Revert "gitlab ci: temporarily disable darwin aarch64 (#39974)" (#40052)eugeneswalker1-18/+17
This reverts commit 2c13361b09c552427623f9e8a728fad94e3de4e9.
2023-09-15gitlab ci: submit the correct audience for protected publish jobs (#40046)Scott Wittenburg1-1/+1
2023-09-14gitlab ci: Put Cray binaries in public mirrors (#39938)Scott Wittenburg3-34/+7
2023-09-14Add OIDC tokens to gitlab-ci jobs (#39813)Dan LaManna2-0/+23
* Add OIDC tokens to gitlab-ci jobs This should allow us to start issuing just-in-time generated credentials for CI jobs that need to modify binary mirrors. The "aud" claim of the token describes what the token is allowed to do. The claim is verified against a set of rules on the IAM role using signed information from GitLab. See spack-infrastructure for the claim verification logic. --------- Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-09-14Add efficient `deptype` flag and `spack.deptypes` module (#39472)Harmen Stoppels1-2/+2
This commit replaces the internal representation of deptypes with `int`, which is more compact and faster to operate with. Double loops like: ``` any(x in ys for x in xs) ``` are replaced by constant operations bool(xs & ys), where xs and ys are dependency types. Global constants are exposed for convenience in `spack.deptypes`
2023-09-14data-vis-sdk: make llvm~libomptarget the default (#40007)Massimiliano Culpo1-1/+1
Currently the configuration of the pipeline is such that there are multiple "optimal" solutions. This is due to the pipeline making ~lld the default for LLVM, but leaving +libomptarget from package.py Since LLVM recipe has a: conflicts("~lld", "+libomptarget") clingo is forced to change one of the two defaults, and the penalty for doing so is the same for each. Hence, it is random whether we'll get +libomptarget+lld or ~libomptarget~lld. This fixes it by changing also the default for libomptarget.
2023-09-13gitlab ci: temporarily disable darwin aarch64 (#39974)Harmen Stoppels1-17/+18
as there are no or few runners available
2023-09-12ci: Tag reindex job to match the image architecture (#39958)Scott Wittenburg1-1/+1
2023-09-12CI/Add superlu-dist to broken stand-alone tests due to hang (#38908)Tamara Dahlgren1-0/+1
* Add superlu-dist to broken stand-alone CI tests * Revert "disable superlu test (#38894)" This reverts commit e16397b5d865d402c749482d05bb2bdc09b96661.
2023-09-11gitlab ci: Use image with awscli for rebuild-index job (#39636)Scott Wittenburg5-9/+10
Instead of pointing to the image on DockerHub, which rate limits us and causes pipeline failures durying busy times, use the version at ghcr. And we might as well use the ghcr version everywhere else too.
2023-09-08bugfix: only complete aliases for potential aliases (#39887)Todd Gamblin3-4/+11
Smart alias completion introduced in #39499 wasn't as smart as it needed to be, and would complete any invalid command prefix and some env names with alias names. - [x] don't complete aliases if there are no potential completions e.g., don't convert `spack isnotacommand` -> `spack concretize` - [x] don't complete with an aliases if we're not looking at a top-level subcommand.
2023-09-08Speed-up `spack external find` execution (#39843)Massimiliano Culpo2-2/+4
* Perform external spec detection with multiple workers The logic to perform external spec detection has been refactored into classes. These classes use the GoF "template" pattern to account for the small differences between searching for "executables" and for "libraries", while unifying the larger part of the algorithm. A ProcessPoolExecutor is used to parallelize the work. * Speed-up external find by tagging detectable packages automatically Querying packages by tag is much faster than inspecting the repository, since tags are cached. This commit adds a "detectable" tag to every package that implements the detection protocol, and external detection uses it to search for packages. * Pass package names instead of package classes to workers The slowest part of the search is importing the Python modules associated with candidate packages. The import is done serially before we distribute the work to the pool of executors. This commit pushes the import of the Python module to the job performed by the workers, and passes just the name of the packages to the executors. In this way imports can be done in parallel. * Rework unit-tests for Windows Some unit tests were doing a full e2e run of a command just to check a input handling. Make the test more focused by just stressing a specific function. Mark as xfailed 2 tests on Windows, that will be fixed by a PR in the queue. The tests are failing because we monkeypatch internals in the parent process, but the monkeypatching is not done in the "spawned" child process.
2023-09-07CI Timing Statistics (#38598)kwryankrattiger2-1/+43
* Write timing information for installs from cache * CI: aggregate and upload install_times.json to artifacts * CI: Don't change root directory for artifact generation * Flat event based timer variation Event based timer allows for easily starting and stopping timers without wiping sub-timer data. It also requires less branching logic when tracking time. The json output is non-hierarchical in this version and hierarchy is less rigidly enforced between starting and stopping. * Add and write timers for top level install * Update completion * remove unused subtimer api * Fix unit tests * Suppress timing summary option * Save timers summaries to user_data artifacts * Remove completion from fish * Move spack python to script section * Write timer correctly for non-cache installs * Re-add hash to timer file * Fish completion updates * Fix null timer yield value * fix type hints * Remove timer-summary-file option * Add "." in front of non-package timer name --------- Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-06gitlab: set cache and config dir in top-level file (#39829)Harmen Stoppels1-2/+4
2023-09-05Set backtrace in ci (#39737)Harmen Stoppels1-0/+1
2023-09-01Windows: fix pwsh env activate/deactivate; load/unload (#39118)John W. Parent2-6/+12
These commands are currently broken on powershell (Windows) due to improper use of the InvokeCommand commandlet and a lack of direct support for the `--pwsh` argument in `spack load`, `spack unload`, and `spack env deactivate`.