summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2023-04-17CI: update Linux images in ML pipelines (#36766)Adam J. Stewart4-12/+18
Add missing openssl/curl/pkgconfig deps to py-tokenizers Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-04-14gitlab ci: Better tagging of "service" jobs (#36846)Scott Wittenburg1-0/+21
- Tag non-rebuild jobs to target a cheaper (and more highly available) subset of runners. - Add missing resource requests to these jobs as well.
2023-04-13remove x86_64_v3 tags (#36828)eugeneswalker1-4/+4
2023-04-13containerize: strip binaries in a less aggressive way (#36683)Massimiliano Culpo2-2/+2
2023-04-13CI: Update Data Vis SDK image (#36761)kwryankrattiger2-3/+3
Supersedes #34224
2023-04-10ci: gpu-tests stack: swap x86_64-{cuda,rocm} for x86_64 (#36759)eugeneswalker1-2/+6
2023-04-10Ci backwards compat (#36045)kwryankrattiger2-0/+167
* CI: Fixup docs for bootstrap. * CI: Add compatibility shim * Add an update method for CI Update requires manually renaming section to `ci`. After this patch, updating and using the deprecated `gitlab-ci` section should be possible. * Fix typos in generate warnings * Fixup CI schema validation * Add unit tests for legacy CI * Add deprecated CI stack for continuous testing * Allow updating gitlab-ci section directly with env update * Make warning give good advice for updating gitlab-ci * Fix typo in CI name * Remove white space * Remove unneeded component of deprected-ci
2023-04-06Ecp hdf5 vol (#35195)kwryankrattiger1-0/+1
* ECP-SDK: enable hdf5 VOL adapters - When +hdf5, enable VOL adapters suitable for the SDK. - Each VOL package must prepend to the HDF5_PLUGIN_PATH. - hdf5: 1.13.3 will break existing VOL packages, constrain VOLs related to SDK and add note to keep 1.13.2 available. - hdf5-vol-async: - Do not set HDF5_VOL_CONNECTOR, consumers must opt-in. - Enforce DAG constraints on MPI to require threaded version. - Depend on an explicit version of argbots to relax concretization issues in other spack environments. - paraview: fix compiler flag usage for the 110 ABI (followup to #33617). * ECP Data and ViS: Add constraits for HDF5 VOLS * CI: HDF5 1.14 builds without VisIt * hdf5-vol-async: Update docs string --------- Co-authored-by: Stephen McDowell <stephen.mcdowell@kitware.com>
2023-04-03Improve Dockerfile recipe generation (#35187)Massimiliano Culpo2-4/+21
- Update default image to Ubuntu 22.04 (previously was still Ubuntu 18.04) - Optionally use depfiles to install the environment within the container - Allow extending Dockerfile Jinja2 template - Allow extending Singularity definition file Jinja2 template - Deprecate previous options to add extra instructions
2023-03-31gitlab ci: request more memory for publish job (#36560)Scott Wittenburg1-0/+2
2023-03-29ci: require x86_64_v3 everywhere (#36158)Harmen Stoppels13-241/+211
2023-03-28e4s power ci: ecp-data-vis-sdk: disable visit due to build issues (#36475)eugeneswalker1-1/+1
2023-03-27modules: enhance help message (#36410)Xavier Delaruelle2-3/+11
Update tcl and lmod modulefile template to provide more information on help message (name, version and target) like done on whatis for lmod modulefiles.
2023-03-24modules: add support for append_flags/remove_flags (#36402)Xavier Delaruelle2-4/+4
Adapt tcl and lmod modulefile templates to generate append-path or remove-path commands in modulefile when respectively append_flags or remove_flags commands are defined in package for run environment. Fixes #10299.
2023-03-24modules tcl: simplify env modification block in template (#36334)Xavier Delaruelle1-16/+0
Simplify environment modification block in modulefile Tcl template by always setting a path delimiter to the prepend-path, append-path and remove-path commands. Remove --delim option to the setenv command as this command does not allow such option. Update test_prepend_path_separator test to explicitly check the 6 path-like commands that should be present in generated modulefile.
2023-03-20spack install: simplify behavior when inside environments (#35206)Harmen Stoppels1-2/+2
Example one: ``` spack install --add x y z ``` is equivalent to ``` spack add x y z spack concretize spack install --only-concrete ``` where `--only-concrete` installs without modifying spack.yaml/spack.lock Example two: ``` spack install ``` concretizes current spack.yaml if outdated and installs all specs. Example three: ``` spack install x y z ``` concretizes current spack.yaml if outdated and installs *only* concrete specs in the environment that match abstract specs `x`, `y`, or `z`.
2023-03-20modules tcl: fix autoload mechanism in template (#36237)Xavier Delaruelle1-6/+1
Adapt tcl modulefile template to call "module load" on autoload dependency without testing if this dependency is already loaded or not. The is-loaded test is not necessary, as module commands know how to cope with an already loaded module. With environment-modules 4.2+ (released in 2018) it is also important to have this "module load" command even if dependency is already loaded in order to record that the modulefile declares such dependency. This is important if you want to keep a consistent environment when a dependent module is unloaded. The "Autoloading" verbose message is also removed as recent module commands will report such information to the user (depending on the verbosity configured for the module command). Such change has been test successfully with Modules 3.2 (EL7), 4.5 (EL8) and 5.2 (latest) and also with Lmod 7 and 8 (as it is mentionned in Spack docs that Lmod can be used along with tcl modules). Dependencies are correctly loaded or unloaded, whether they are loaded/unloaded or not. This change fixes Tcl quoting issue introduced in #32853. Fixes #19155.
2023-03-19Fix case spelling for Lmod and Tcl (#36215)Xavier Delaruelle2-2/+2
2023-03-17Update the PyTorch ecosystem (#36132)Adam J. Stewart3-0/+3
* py-pytorch-lightning: add v2.0.0 * py-lightning-utilities: add v0.8.0 * Update all PyTorch packages * Open-CE does not yet have patches for PyTorch 2 on ppc64le
2023-03-15Hotfix: CI: Add CI target for gpu-test stack (#36136)kwryankrattiger1-0/+1
2023-03-14Fix typo affecting Gitlab CI (#36103)Harmen Stoppels1-1/+1
Introduced in #35944
2023-03-14concretizer: add mode to reuse dependencies only (#30990)Michael Kuhn1-9/+9
This adds a new mode for `concretizer:reuse` called `dependencies`, which only reuses dependencies. Currently, `spack install foo` will reuse older versions of `foo`, which might be surprising to users.
2023-03-13Set build_jobs dynamically in CI to avoid oversubscription (#35996)Harmen Stoppels4-2/+7
Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com> Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
2023-03-10get --dev and drop set -x (#36010)Harmen Stoppels1-1/+1
2023-03-10curl flag is not universally supported (#36009)Harmen Stoppels3-3/+3
2023-03-10hotfix: fix double double quotes (#36005)Harmen Stoppels1-1/+1
2023-03-10ci: version bump for ghcr.io/spack/e4s-amazonlinux-2 (#35976)Zack Galbreath10-15/+15
* ci: version bump for ghcr.io/spack/e4s-amazonlinux-2 This new image comes with GnuPG v2.4.0 * py-cython: upperbounds for Python versions * fix py-gevent nonsense --------- Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-03-10CI boilerplate reduction (#34272)kwryankrattiger30-2527/+573
* CI configuration boilerplate reduction and refactor Configuration: - New notation for list concatenation (prepend/append) - New notation for string concatenation (prepend/append) - Break out configuration files for: ci.yaml, cdash.yaml, view.yaml - Spack CI section refactored to improve self-consistency and composability - Scripts are now lists of lists and/or lists of strings - Job attributes are now listed under precedence ordered list that are composed/merged using Spack config merge rules. - "service-jobs" are identified explicitly rather than as a batch CI: - Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files - Make padding consistent across all stacks (256) - Merge all package -> runner mappings to be consistent across all stacks Unit Test: - Refactor CI module unit-tests for refactor configuration Docs: - Add docs for new notations in configuration.rst - Rewrite docs on CI pipelines to be consistent with refactored CI workflow * Script verbose environ, dev bootstrap * Port #35409
2023-03-10ci: Increase the amount of pruning possible for PR pipelines (#35944)Scott Wittenburg1-0/+1
By setting the traversal depth to 1, only specs matching the changed package and direct dependents of those (and of course all dependencies of that set) are removed from pruning candidacy.
2023-03-03CI: Fix timeout for VisIt and ParaView (#35787)kwryankrattiger1-0/+2
ref. #35400
2023-03-01Remove handling of deprecated target names (graviton) (#35537)Massimiliano Culpo3-3/+3
* Update target names for Gitlab pipelines * Remove handling of deprecated names for graviton
2023-02-16Style: black 23, skip magic trailing comma (#35351)Adam J. Stewart1-11/+4
* Style: black 23, skip magic trailing commas * isort should use same line length as black * Fix unused import * Update version of black used in CI * Update new packages * Update new packages
2023-02-13ci: gpu test stack: turn off cuda@11.7 builds (#35467)eugeneswalker1-10/+0
2023-02-09gpu test stack: add test w/ ^cuda@12 builds on A100 w/ newer driver (#35375)eugeneswalker1-8/+13
* gpu test stack: test cuda@12 builds on A100 w/ newer driver * get gpu info via nvidia-smi; * kokkos+cuda^cuda@12 has genuine failure
2023-02-03ci: add minimal gpu testing stack (#35251)eugeneswalker3-0/+368
* ci: add minimal gpu testing stack * kokkos +cuda requires +wrapper... * require pass * add raja+cuda
2023-01-30e4s: add and prefer boost +python +filesystem +iostreams +system (#35180)eugeneswalker1-0/+3
2023-01-29e4s oneapi: use oneapi@2023.0.0 (#34634)eugeneswalker2-83/+98
2023-01-29e4s: add power stack (#32710)eugeneswalker3-317/+477
* e4s: restore builds builds * gitlab ci: allow UO to build protected binaries for signing * use newer image; comment out failing builds * gitlab-ci: Some tweaks for e4s power builds - fix tags (no longer require generate jobs to run on aws) - fix resource requests for generation jobs resource requests - remove SPACK_SIGNING_KEY from protected power build jobs - update UO signing key path - change the CDash build group to reflect stack name - retry pipeline generation jobs *always* * correct double packages: section * gitlab-ci:script: modernize * remove new gnu make, not for ppc64le --------- Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-01-23e4s: add cusz +cuda (#35004)eugeneswalker1-0/+1
2023-01-21Add --exclude option to 'spack external find' (#35013)Alex Richert1-1/+1
* Add --exclude option to 'spack external find' to ignore user-specified external packages * Update bash completion arg order for external find Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-01-20Allocate more memory for generate jobs in all stacks (#35040)Zack Galbreath1-11/+2
* Allocate more memory for generate jobs in all stacks * Add a 60 minute timeout on generate jobs
2023-01-20e4s: ecp-data-vis-sdk +rocm: remove redundant amdgpu_target specification ↵eugeneswalker1-1/+1
(#35022)
2023-01-19depfile: --make-target-prefix -> --make-prefix (#35009)Harmen Stoppels1-1/+1
Since SPACK_PACKAGE_IDS is now also "namespaced" with <prefix>, it makes more sense to call the flag `--make-prefix` and alias the old flag `--make-target-prefix` to it.
2023-01-18e4s ci: clean up unnecessary constraints (#35006)eugeneswalker1-17/+11
2023-01-18e4s: add sz3 (#35007)eugeneswalker1-0/+1
2023-01-18e4s ci: add veloc (#35003)eugeneswalker1-0/+1
2023-01-18license year bump (#34921)Harmen Stoppels23-23/+23
* license bump year * fix black issues of modified files * mypy * fix 2021 -> 2023
2023-01-18e4s: add py-h5py ~mpi variant (#34998)eugeneswalker1-1/+2
2023-01-18depfile: variable with all identifiers (#34678)Harmen Stoppels1-0/+3
With the new variable [prefix/]SPACK_PACKAGE_IDS you can conveniently execute things after each successful install. For example push just-built packages to a buildcache ``` SPACK ?= spack export SPACK_COLOR = always MAKEFLAGS += -Orecurse MY_BUILDCACHE := $(CURDIR)/cache .PHONY: all clean all: push ifeq (,$(filter clean,$(MAKECMDGOALS))) include env.mk endif # the relevant part: push has *all* example/push/<pkg identifier> as prereqs push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS)) $(SPACK) -e . buildcache update-index --directory $(MY_BUILDCACHE) $(info Pushed everything, yay!) # and each example/push/<pkg identifier> has the install target as prereq, # and the body can use target local $(HASH) and $(SPEC) variables to do # things, such as pushing to a build cache example/push/%: example/install/% @mkdir -p $(dir $@) $(SPACK) -e . buildcache create --allow-root --only=package --unsigned --directory $(MY_BUILDCACHE) /$(HASH) # push $(SPEC) @touch $@ spack.lock: spack.yaml $(SPACK) -e . concretize -f env.mk: spack.lock $(SPACK) -e . env depfile -o $@ --make-target-prefix example clean: rm -rf spack.lock env.mk example/ ``
2023-01-16Refer to mirrors by name, path, or url (#34891)Harmen Stoppels1-3/+13
With this change we get the invariant that `mirror.fetch_url` and `mirror.push_url` return valid URLs, even when the backing config file is actually using (relative) paths with potentially `$spack` and `$env` like variables. Secondly it avoids expanding mirror path / URLs too early, so if I say `spack mirror add name ./path`, it stays `./path` in my config. When it's retrieved through MirrorCollection() we exand it to say `file://<env dir>/path` if `./path` was set in an environment scope. Thirdly, the interface is simplified for the relevant buildcache commands, so it's more like `git push`: ``` spack buildcache create [mirror] [specs...] ``` `mirror` is either a mirror name, a path, or a URL. Resolving the relevant mirror goes as follows: - If it contains either / or \ it is used as an anonymous mirror with path or url. - Otherwise, it's interpreted as a named mirror, which must exist. This helps to guard against typos, e.g. typing `my-mirror` when there is no such named mirror now errors with: ``` $ spack -e . buildcache create my-mirror ==> Error: no mirror named "my-mirror". Did you mean ./my-mirror? ``` instead of creating a directory in the current working directory. I think this is reasonable, as the alternative (requiring that a local dir exists) feels a bit pendantic in the general case -- spack is happy to create the build cache dir when needed, saving a `mkdir`. The old (now deprecated) format will still be available in Spack 0.20, but is scheduled to be removed in 0.21: ``` spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...] ``` This PR also touches `tmp_scope` in tests, because it didn't really work for me, since spack fixes the possible --scope values once and for all across tests, so tests failed when run out of order.