summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2022-05-05Makefile generator for parallel spack install of environments (#30254)Harmen Stoppels5-5/+322
`make` solves a lot of headaches that would otherwise have to be implemented in Spack: 1. Parallelism over packages through multiple `spack install` processes 2. Orderly output of parallel package installs thanks to `make --sync-output=recurse` or `make -Orecurse` (works well in GNU Make 4.3; macOS is unfortunately on a 16 years old 3.x version, but it's one `spack install gmake` away...) 3. Shared jobserver across packages, which means a single `-j` to rule them all, instead of manually finding a balance between `#spack install processes` & `#jobs per package` (See #30302). This pr adds the `spack env depfile` command that generates a Makefile with dag hashes as targets, and dag hashes of dependencies as prerequisites, and a command along the lines of `spack install --only=packages /hash` to just install a single package. It exposes two convenient phony targets: `all`, `fetch-all`. The former installs the environment, the latter just fetches all sources. So one can either use `make all -j16` directly or run `make fetch-all -j16` on a login node and `make all -j16` on a compute node. Example: ```yaml spack: specs: [perl] view: false ``` running ``` $ spack -e . env depfile --make-target-prefix env | tee Makefile ``` generates ```Makefile SPACK ?= spack .PHONY: env/all env/fetch-all env/clean env/all: env/env env/fetch-all: env/fetch env/env: env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww @touch $@ env/fetch: env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc @touch $@ env/dirs: @mkdir -p env/.fetch env/.install env/.fetch/%: | env/dirs $(info Fetching $(SPEC)) $(SPACK) -e '/tmp/tmp.7PHPSIRACv' fetch $(SPACK_FETCH_FLAGS) /$(notdir $@) && touch $@ env/.install/%: env/.fetch/% $(info Installing $(SPEC)) +$(SPACK) -e '/tmp/tmp.7PHPSIRACv' install $(SPACK_INSTALL_FLAGS) --only-concrete --only=package --no-add /$(notdir $@) && touch $@ # Set the human-readable spec for each target env/%/cdqldivylyxocqymwnfzmzc5sx2zwvww: SPEC = perl@5.34.1%gcc@10.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-zen2 env/%/gv5kin2xnn33uxyfte6k4a3bynhmtxze: SPEC = berkeley-db@18.1.40%gcc@10.3.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu20.04-zen2 env/%/cuymc7e5gupwyu7vza5d4vrbuslk277p: SPEC = bzip2@1.0.8%gcc@10.3.0~debug~pic+shared arch=linux-ubuntu20.04-zen2 env/%/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: SPEC = diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2 env/%/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws: SPEC = libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2 env/%/yfz2agazed7ohevqvnrmm7jfkmsgwjao: SPEC = gdbm@1.19%gcc@10.3.0 arch=linux-ubuntu20.04-zen2 env/%/73t7ndb5w72hrat5hsax4caox2sgumzu: SPEC = readline@8.1%gcc@10.3.0 arch=linux-ubuntu20.04-zen2 env/%/trvdyncxzfozxofpm3cwgq4vecpxixzs: SPEC = ncurses@6.2%gcc@10.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-zen2 env/%/sbzszb7v557ohyd6c2ekirx2t3ctxfxp: SPEC = pkgconf@1.8.0%gcc@10.3.0 arch=linux-ubuntu20.04-zen2 env/%/c4go4gxlcznh5p5nklpjm644epuh3pzc: SPEC = zlib@1.2.12%gcc@10.3.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu20.04-zen2 # Install dependencies env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww: env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p: env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao: env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu: env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs: env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/clean: rm -f -- env/env env/fetch env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc ``` Then with `make -O` you get very nice orderly output when packages are built in parallel: ```console $ make -Orecurse -j16 spack -e . install --only-concrete --only=package /c4go4gxlcznh5p5nklpjm644epuh3pzc && touch c4go4gxlcznh5p5nklpjm644epuh3pzc ==> Installing zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc ... Fetch: 0.00s. Build: 0.88s. Total: 0.88s. [+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc spack -e . install --only-concrete --only=package /sbzszb7v557ohyd6c2ekirx2t3ctxfxp && touch sbzszb7v557ohyd6c2ekirx2t3ctxfxp ==> Installing pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp ... Fetch: 0.00s. Build: 3.96s. Total: 3.96s. [+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp ``` For Perl, at least for me, using `make -j16` versus `spack -e . install -j16` speeds up the builds from 3m32.623s to 2m22.775s, as some configure scripts run in parallel. Another nice feature is you can do Makefile "metaprogramming" and depend on packages built by Spack. This example fetches all sources (in parallel) first, print a message, and only then build packages (in parallel). ```Makefile SPACK ?= spack .PHONY: env all: env spack.lock: spack.yaml $(SPACK) -e . concretize -f env.mk: spack.lock $(SPACK) -e . env depfile -o $@ --make-target-prefix spack fetch: spack/fetch @echo Fetched all packages && touch $@ env: fetch spack/env @echo This executes after the environment has been installed clean: rm -rf spack/ env.mk spack.lock ifeq (,$(filter clean,$(MAKECMDGOALS))) include env.mk endif ```
2022-05-05Enable PyTorch 1.10.x to build on Power systems (#30494)Brian Van Essen1-0/+18
* Use patches from IBM's Open CE project to enable PyTorch to build on Power systems. Cherry-pick a patch to allow earlier versions of PyTorch to build with CUDA 11.4. * Update var/spack/repos/builtin/packages/py-torch/package.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-05-05package audit: ensure stand-alone test method not include in build-phase ↵Tamara Dahlgren3-2/+22
testing (#30352)
2022-05-05clingo: add v5.5.2 (#30495)Massimiliano Culpo1-12/+14
* clingo add v5.5.2 * Unify dependencies for @spack and @master
2022-05-05Update totalview package.py with new version and maintainer (#30341)David1-3/+22
* Update totalview package.py with new version and maintainer * Update totalview/package.py version order Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-05-05Bug fix for mesa-glu: if gl is provided by mesa, need to specify argument to ↵Dom Heinzeller1-0/+6
./configure (#30082)
2022-05-05octopus: update to 11.4 and add tests (#30321)Hans Fangohr3-30/+245
* octopus: adding versions up to 11.4 * octopus: add smoke tests * octopus: add necessary flags for gcc@10 * octopus: update to compilation and dependencies * octopus: adding new variants * octopus: remove 'poke' (as this poke is not in spack [yet]) * octopus: allow compilation from git repo develop branch * octopus: adapt to spack style requirements * octopus: add maintainer * octopus: make tests after install optional Thank you @tldahlgren * octopus: follow recommended practice for test input data Move the two configuration files we use for smoke tests into `test` subdirectory. Thanks @tldahlgren. * Adding maintainer with their agreement by email * octopus: reduce duplication of flags - part of code review * octopus: https is preferred over http * octopus: remove .99 from versioning information Thanks to https://github.com/spack/spack/pull/26402, we can drop the "2:3.99" notation when we mean all versions 2.x and 3.x Examples: https://github.com/spack/spack/commit/b9e72557e8b52b63244500f226afda0efe798df2#diff-b8373d30b3a141c495c2281273ee6184fc513413142afaf2adac1f406cd6b0d7 (from review) * octopus: args.extend([x]) -> args.append(x) (hint from review)
2022-05-05yaksa: Add support for cuda_arch variants (#30471)Ken Raffenetti1-0/+5
2022-05-05ucx: overhaul package recipe (#30215)Harmen Stoppels1-138/+108
- hyphens to underscore in variant names (e.g. `~ib_hw_tm` instead of `~ib-hw-tm`) - disable docs, disable go, drop the unused `doc` variant - `+static +shared` => `libs=static,shared` - `vfs` support (add libfuse@3: dep) - fix `backtrace_detail` for UCX 1.12+ (`--enable-backtrace-detail` => `--with-bfd`) - group variants and sort
2022-05-05baurmc: add new package (#30245)Valentin Volkl1-0/+43
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-05git: adding subtree support, post-install step, reordered deps (#30400)Jennifer Green1-6/+18
2022-05-05LAPACK: add API compatibility versions (#29828)Adam J. Stewart6-1/+16
2022-05-05rnaz: add version 2.1.1 and git repository to newest version. (#30349)dlkuehn1-0/+7
Co-authored-by: David Kuehn <las_dkuehn@iastate.edu> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-05trimmomatic: adding more flexible java dep (#30488)snehring1-1/+2
2022-05-05flatbufffers: patches are in v2.0.0 (#30492)Wouter Deconinck1-2/+2
2022-05-05spectre: add v2022.05.05 (#30493)SXS Bot1-0/+1
Co-authored-by: sxs-bot <sxs-bot@users.noreply.github.com>
2022-05-05spack external find: add search path customization (#30479)Greg Becker2-3/+8
2022-05-04qt: fix ~opengl and use_spack_dep (#30487)Seth R. Johnson1-13/+16
2022-05-04UPC++: minor packaging improvements (#30482)Dan Bonachea1-0/+36
* upcxx: Add `spack external find` support * upcxx: Minor tweaks to Shasta support
2022-05-04Add scitokens-cpp recipe and also as a variant for xrootd (#30362)Andrea Valenzuela2-0/+33
* Add scitokens-cpp recipe * Add scitokens-cpp variant in xrootd * Fix typo * Fix flake8 style errors * Update license date Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-05-04Fix professor recipe (#30002)iarspider1-1/+13
* [WIP] Cleanup professor dependencies * Add dependency on ROOT * Remove direct dependency on wxwidgets * `prof-I` tool requires matplotlib with wxwidgets backend, made that optional * Update package.py * Update package.py (#38) * Update package.py * Update package.py
2022-05-04Update gtk-doc depends for tce findings (#30337)David1-1/+2
gtk-doc can call pkg-config when used at runtime, so pkgconfig needs to be a runtime requirement. itstool needs to be a build-time requirement.
2022-05-04gxsview: vtk9 no longer builds libvtkjpeg... (#30340)Olivier Cessenat2-0/+28
2022-05-04Add checksum for py-prompt-toolkit@3.0.29 (#30330)Andrea Valenzuela1-0/+1
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
2022-05-04curl: tls=mbedtls requires fPIC (#30447)Carlos Bederián1-2/+2
2022-05-04gatk: updating and expanding java dependency (#30476)snehring1-1/+4
2022-05-04Python package: add backup BINDIR setting (#30408)Adam J. Stewart1-0/+1
2022-05-04qt: Assimp dependency specificationa is more nuanced (#30460)Chuck Atkins2-18/+29
libassimp has been a dependency for all of 5.x but expressing that has varied significantly throughout the 5.x lifecycle: v5.0: qt3d uses internal-only libassimp v5.5: external-only libassimp v5.6: either internal or external libassimp via autodetection v5.9: user-selectable internal-vs-external via -assimp v5.14: additional qtquick3d module uses -assimp v5.15: qtquick3d switches to the -quick3d-assimp option * current bug where the incorrect target is setup
2022-05-04bugfix: fix filter_compiler_wrappers for Cray compiler (#29786)Sergey Kosukhin1-3/+22
2022-05-04GDAL: add v3.4.3 (#30474)Adam J. Stewart1-0/+1
2022-05-04add package: abacus (#30416)Bitllion1-0/+108
* add pacakge: abacus * rename * fix some style bugs * update package:abacus * fix some style bugs * format code * Delete a line of useless comments * updatee abacus
2022-05-04Add mimalloc package with use in HPX and pika (#30457)Mikael Simberg3-5/+32
* Add mimalloc package * Add mimalloc as allocator option to pika * Add mimalloc as allocator option to hpx * Set git property globally instead of per-version in pika, hpx, and mimalloc packages Co-authored-by: Mikael Simberg <mikael.simberg@iki.if>
2022-05-04gpi-2_fix_deps: setup 'gawk' and 'sed' dependencies (#29740)acastanedam1-1/+3
Strictly, `sed` is a `build` and `run` dependency in all gpi-2 versions, whereas `gawk` is a `run` (`build` and `run`) dependency for gpi-2 versions greater or equal (less) than 1.4.0
2022-05-04gitlab: Remove temporary storage url from all stacks (#29949)Scott Wittenburg7-10/+2
Gitlab pipelines run for spack already have other S3 storage locations configured for storage of binaries, so this PR removes the redundant per-pipeline mirror. As a result, the "cleanup" jobs will no longer be generated at the end of each pipeline, removing one possible point of pipeline failure.
2022-05-04Add checksum for version 1.1.13 and update dependencies (#30470)Andrea Valenzuela1-1/+3
2022-05-04chameleon: add maintainers information (#30472)fpruvost1-0/+1
2022-05-04pika: Add 0.4.0 release (#30469)Auriane R1-0/+1
2022-05-04Use gccgo to bootstrap go on aarch64 (#30350)Ross Miller2-3/+17
The go-bootstrap package doesn't work on aarch64 platforms, so the only way to build Go is to use gccgo. Also, some versions of gccgo have a bug that prevents them from compiling go (see golang/go#47771), so this patch limits gcc to versions newer than 10.4.0 or 11.3.0. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-04chameleon: add new package (#30368)fpruvost1-0/+97
2022-05-04Add checksum for py-plac@1.3.5 (#30327)Andrea Valenzuela1-0/+1
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
2022-05-04Add checksum for py-networkx@2.7.1 and fix python dependency (#30326)Andrea Valenzuela1-0/+2
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
2022-05-04mesa: intel needs c99 standard specified under intel. (#30334)Paul Ferrell1-0/+6
2022-05-04maq: adding perl dep, addressing gcc errors (#30312)snehring1-1/+13
2022-05-04namd: place an upper bound on the charmpp dependency version (#30397)lpoirel1-1/+1
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-04WarpX: add v22.05 (#30467)Axel Huebl2-3/+5
Update `warpx` & `py-warpx` to the latest release, `22.05`.
2022-05-04igv: changing to 'all platforms' version. Adding igvtools variant. (#30379)snehring1-11/+18
2022-05-04py-shapely: add v1.8.2 (#30466)Adam J. Stewart1-0/+1
2022-05-04Remove deprecated "--run-tests" option of "spack install" (#30461)Massimiliano Culpo3-10/+2
2022-05-03boost: fix v1.78 build with oneapi (#30463)eugeneswalker2-0/+18
2022-05-04autodock-vina: add versions up to v1.2.3, fixes (#30443)Carlos Bederián3-74/+10
* C++ package, use spack_cxx * download from github, add 1.2.x versions * remove patches already present in 1.1.2-boost-new release * specify boost dependencies
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-16 23:21:18 +0000