summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-07-07cvs tests: don't use dateutil at allTodd Gamblin1-18/+21
`dateutil.parser` was an optional dependency for CVS tests. It was failing on macOS beacuse the dateutil types were not being installed, and mypy was failing *even when the CVS tests were skipped*. This seems like it was an oversight on macOS -- `types-dateutil-parser` was not installed there, though it was on Linux unit tests. It takes 6 lines of YAML and some weird test-skipping logic to get `python-dateutil` and `types-python-dateutil` installed in all the tests where we need them, but it only takes 4 lines of code to write the date parser we need for CVS, so I just did that instead. Note that CVS date format can vary from system to system, but it seems like it's always pretty similar for the parts we care about. - [x] Replace dateutil.parser with a simpler date regex - [x] Lose the dependency on `dateutil.parser`
2021-07-07style: get close to full coverage of `spack style`Todd Gamblin3-95/+182
Previous tests of `spack style` didn't really run the tools -- they just ensure that the commands worked enough to get coverage. This adds several real tests and ensures that we hit the corner cases in `spack style`. This also tests sucess as well as failure cases.
2021-07-07style: clean up and restructure `spack style` commandTodd Gamblin1-93/+77
This consolidates code across tools in `spack style` so that each `run_<tool>` function can be called indirecty through a dictionary of handlers, and os that checks like finding the executable for the tool can be shared across commands. - [x] rework `spack style` to use decorators to register tools - [x] define tool order in one place in `spack style` - [x] fix python 2/3 issues to Get `isort` checks working - [x] make isort error regex more robust across versions - [x] remove unused output option - [x] change vestigial `TRAVIS_BRANCH` to `GITHUB_BASE_REF` - [x] update completion
2021-07-07style: Move isort configuration to pyproject.tomlTodd Gamblin1-4/+0
- [x] Remove flake8-import-order checks, as we only need isort for this - [x] Clean up configuration and requirements
2021-07-07style: add support for `isort` and `--fix`Danny McClanahan1-9/+55
2021-07-06oneAPI packages: add 2021.3 release (#24617)Vasily Danilin2-12/+35
2021-07-06fix buffered download (#24623)Harmen Stoppels1-8/+8
* Use shutil to do a buffered copy from http response to file * Fix flake8... * Somehow flake8 still complains about unrelated files
2021-07-06installer: fix double print of exception (#24697)eugeneswalker1-0/+1
2021-07-02Fix fetching for Python 3.9.6 (#24686)Adam J. Stewart1-6/+6
When using Python 3.9.6, Spack is no longer able to fetch anything. Commands like `spack fetch` and `spack install` all break. Python 3.9.6 includes a [new change](https://github.com/python/cpython/pull/25853/files#diff-b3712475a413ec972134c0260c8f1eb1deefb66184f740ef00c37b4487ef873eR462) that means that `scheme` must be a string, it cannot be None. The solution is to use an empty string like the method default. Fixes #24644. Also see https://github.com/Homebrew/homebrew-core/pull/80175 where this issue was discovered by CI. Thanks @branchvincent for reporting such a serious issue before any actual users encountered it! Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-02Pipelines: Improve broken specs check (#24643)Scott Wittenburg2-21/+25
We should not fail the generate stage simply due to the presence of a broken-spec somewhere in the DAG. Only fail if the known broken spec needs to be rebuilt.
2021-07-02Add `when` context manager to group common constraints in packages (#24650)Massimiliano Culpo4-63/+183
This PR adds a context manager that permit to group the common part of a `when=` argument and add that to the context: ```python class Gcc(AutotoolsPackage): with when('+nvptx'): depends_on('cuda') conflicts('@:6', msg='NVPTX only supported in gcc 7 and above') conflicts('languages=ada') conflicts('languages=brig') conflicts('languages=go') ``` The above snippet is equivalent to: ```python class Gcc(AutotoolsPackage): depends_on('cuda', when='+nvptx') conflicts('@:6', when='+nvptx', msg='NVPTX only supported in gcc 7 and above') conflicts('languages=ada', when='+nvptx') conflicts('languages=brig', when='+nvptx') conflicts('languages=go', when='+nvptx') ``` which needs a repetition of the `when='+nvptx'` argument. The context manager might help improving readability and permits to group together directives related to the same semantic aspect (e.g. all the directives needed to model the behavior of `gcc` when `+nvptx` is active). Modifications: - [x] Added a `when` context manager to be used with package directives - [x] Add unit tests and documentation for the new feature - [x] Modified `cp2k` and `gcc` to show the use of the context manager
2021-07-01meson: fix typo (libs instead of default_library) (#24649)Harmen Stoppels1-2/+2
2021-06-30ci: only write to broken-specs list on SpackError (#24618)Zack Galbreath2-18/+20
ci: only write to broken-specs list on SpackError Only write to the broken-specs list when `spack install` raises a SpackError, instead of writing to this list unnecessarily when infrastructure-related problems prevent a develop job from completing successfully.
2021-06-30ci: Don't raise an exception if we fail to relate CDash builds (#24299)Zack Galbreath4-73/+96
ci: Do not allow cdash-related errors to cause pipeline failure
2021-06-29spack style: warn if flake8-import-order is missing (#24590)Adam J. Stewart1-4/+7
2021-06-29Fix prefix-collision detection for projections (#24049)Harmen Stoppels4-32/+80
If two Specs have the same hash (and prefix) but are not equal, Spack originally had logic to detect this and raise an error (since both cannot be installed in the same place). Recently this has eroded and the check no-longer works; moreover, when defining projections (which may truncate the hash or other distinguishing properties from the prefix) Spack was also failing to detect collisions (in both of these cases, Spack would overwrite the old prefix with the new Spec). This PR maintains a list of all "taken" prefixes: if a hash is not registered (i.e. recorded as installed in the database) but the prefix is occupied, that is a collision. This can detect collisions created by defining projections (specifically when they omit the hash). The PR does not detect collisions where specs have the same hash (and prefix) but are not equal.
2021-06-28Use flake8-import-order to enforce PEP-8 compliance (#23947)Adam J. Stewart1-9/+9
2021-06-28CUDA: 11.0.3 (11.0 "Update 1") (#24481)Axel Huebl1-4/+4
Add CUDA 11.0.3. This release adds new features such as NVCC flags `--forward-unknown-to-host-compiler` and `--forward-unknown-to-host-linker`
2021-06-27docs: link projections docs to spec format (#24478)Dylan Simon1-1/+2
2021-06-26Update archspec to support arm compiler on a64fx (#24524)Massimiliano Culpo2-1/+13
2021-06-25Doc: Note on required changes after merge of reproducible builds (#24347)Adrien Bernede1-1/+18
* Suggestion of a note for conversion of existing pipelines. * Wording * Fix format in .rst note * Wording
2021-06-25Add support for .tbz file extensions (#24479)Adam J. Stewart2-1/+3
2021-06-24Environment modifications: de-prioritize external packages (#23824)Peter Scheibel4-170/+270
Prior to any Spack build, Spack modifies PATH etc. to help the build find the dependencies it needs. It also allows any package to define custom environment modifications (and furthermore a package can specify environment modifications to apply when it is used as a dependency). If an external package defines custom environment modifications that alter PATH, and the external package is in a merged or system prefix, then that prefix could "override" the Spack-built packages. This commit reorders environment modifications so that PrependPath actions which expose Spack-built packages override PrependPath actions for custom environment modifications of external packages. In more detail, the original order of environment modifications is: * Modules * Compiler flag variables * PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH for dependencies * Custom package.py modifications in the following order: * dependencies * root This commit changes the order: * Modules * Compiler flag variables * For each external dependency * PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH modifications * Custom modifications * For each Spack-built dependency * PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH modifications * Custom modifications
2021-06-24Pipelines: Set a pipeline type variable (#24505)Scott Wittenburg3-13/+17
Spack pipelines need to take specific actions internally that depend on whether the pipeline is being run on a PR to spack or a merge to the develop branch. Pipelines can also run in other repositories, which represents other possible use cases than just the two mentioned above. This PR creates a "SPACK_PIPELINE_TYPE" gitlab variable which is propagated to rebuild jobs, and is also used internally to determine which pipeline-specific tasks to run. One goal of the PR is fix an issue where rebuild jobs which failed on develop pipelines did not properly report the broken full hash to the "broken-specs-url".
2021-06-24Add Externally Findable section to info command (#24503)Asher Mancinelli2-0/+37
* Add Externally Findable section to info command * Use comma delimited detection attributes in addition to boolean value * Unit test externally detectable part of spack info
2021-06-23Update command to setup tutorial (#24488)Massimiliano Culpo1-2/+2
2021-06-23spack ci: use return codes to signal exit status (#24090)Scott Wittenburg1-3/+3
2021-06-23Fix broken CI for package only PRs, make dateutil not strictly required (#24484)Massimiliano Culpo1-6/+5
* Force the Python interpreter with an env variable This commit forces the Python interpreter with an environment variable, to ensure that the Python set by the "setup-python" action is the one being used. Due to the policy adopted by Spack to prefer python3 over python we may end up picking a Python 3.X interpreter where Python 2.7 was meant to be used. * Revert "Update conftest.py (#24473)" This reverts commit 477c8ce8205ec149fa897c9d83e530815c978d8b. * Make python-dateutil a soft dependency for unit tests Before #23212 people could clone spack and run ``` spack unit-tests ``` while now this is not possible, since python-dateutil is a required but not vendored dependency. This change makes it not a hard requirement, i.e. it will be used if found in the current interpreter. * Workaround mypy complaint
2021-06-22filter_compiler_wrappers: include realpath of compiler wrappers (#24456)eugeneswalker1-1/+5
2021-06-22Add config option to use urllib to fetch if curl missing (#21398)loulawrence5-117/+227
* Use Python module urllib to fetch in the case that curl is missing
2021-06-22Update conftest.py (#24473)Peter Scheibel1-2/+2
2021-06-22ASP-based solver: fix provider logic (#24351)Massimiliano Culpo2-10/+31
This commit fixes a subtle bug that may occur when a package is a "possible_provider" of a virtual but no "provides_virtual" can be deduced. In that case the cardinality constraint on "provides_virtual" may arbitrarily assign a package the role of provider even if the constraints for it to be one are not fulfilled. The fix reworks the logic around three concepts: - "possible_provider": a package may provide a virtual if some constraints are met - "provides_virtual": a package meet the constraints to provide a virtual - "provider": a package selected to provide a virtual
2021-06-22ASP-based solver: fix facts for default providers (#24380)Massimiliano Culpo1-1/+2
Facts used to compute weights for providers only need the package name, since the other attributes are computed as part of the solve.
2021-06-22Implement CVS fetcher (#23212)Erik Schnetter4-2/+423
Spack packages can now fetch versions from CVS repositories. Note this fetch mechanism is unsafe unless using :extssh:. Most public CVS repositories use an insecure protocol implemented as part of CVS.
2021-06-22adding save of build times on install (#24350)Vanessasaurus6-44/+123
Here we are adding an install_times.json into the spack install metadata folder. We record a total, global time, along with the times for each phase. The type of phase or install start / end is included (e.g., build or fail) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-21Fetching: git on Mac OS (#24247)Peter Scheibel3-5/+15
Extend the changes in #24163 to unit tests.
2021-06-18Add an audit system to Spack (#23053)Massimiliano Culpo5-0/+625
Add a new "spack audit" command. This command can check for issues with configuration or with packages and is intended to help a user debug a failed Spack build. In some cases the reported issues are always errors but are too costly to check for (e.g. packages that specify missing variants on dependencies). In other cases the issues may be legitimate but uncommon usage of Spack and we want to be sure the user intended the behavior (e.g. duplicate compiler definitions). Audits are grouped by theme, and for now the two themes are packages and configuration. For example you can run all available audits on packages with "spack audit packages". It is intended that in the future users will be able to define their own audits. The package audits are good candidates for running in package_sanity (i.e. they could catch bugs in user-submitted packages before they are merged) but that is left for a later PR.
2021-06-17oneAPI compiler: update openmp flag (#23771)Frank Willmore2-2/+2
2021-06-17Adding support for spack monitor with containerize (#23777)Vanessasaurus6-1/+417
This should get us most of the way there to support using monitor during a spack container build, for both Singularity and Docker. Some quick notes: ### Docker Docker works by way of BUILDKIT and being able to specify --secret. What this means is that you can prefix a line with a mount of type secret as follows: ```bash # Install the software, remove unnecessary deps RUN --mount=type=secret,id=su --mount=type=secret,id=st cd /opt/spack-environment && spack env activate . && export SPACKMON_USER=$(cat /run/secrets/su) && export SPACKMON_TOKEN=$(cat /run/secrets/st) && spack install --monitor --fail-fast && spack gc -y ``` Where the id for one or more secrets corresponds to the file mounted at `/run/secrets/<name>`. So, for example, to build this container with su (spackmon user) and sv (spackmon token) defined I would export them on my host and do: ```bash $ DOCKER_BUILDKIT=1 docker build --network="host" --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container . ``` And when we add `env` to the secret definition that tells the build to look for the secret with id "st" in the environment variable `SPACKMON_TOKEN` for example. If the user is building locally with a local spack monitor, we also need to set the `--network` to be the host, otherwise you can't connect to it (a la isolation of course.) ## Singularity Singularity doesn't have as nice an ability to clearly specify secrets, so (hoping this eventually gets implemented) what I'm doing now is providing the user instructions to write the credentials to a file, add it to the container to source, and remove when done. ## Tags Note that the tags PR https://github.com/spack/spack/pull/23712 will need to be merged before `--monitor-tags` will actually work because I'm checking for the attribute (that doesn't exist yet): ```bash "tags": getattr(args, "monitor_tags", None) ``` So when that PR is merged to update the argument group, it will work here, and I can either update the PR here to not check if the attribute is there (it will be) or open another one in the case this PR is already merged. Finally, I added a bunch of documetation for how to use monitor with containerize. I say "mostly working" because I can't do a full test run with this new version until the container base is built with the updated spack (the request to the monitor server for an env install was missing so I had to add it here). Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-17ci: add all locally computed hashes as job variables (#24359)Scott Wittenburg1-4/+9
2021-06-17Unset LD_PRELOAD and DYLD_INSERT_LIBRARIES (#24177)Harmen Stoppels1-0/+4
When running executables from build dependencies, we want to avoid that `LD_PRELOAD` and `DYLD_INSERT_LIBRARIES` any of their shared libs build by spack with system libraries.
2021-06-15adding spack upload command (#24321)Vanessasaurus3-8/+121
this will first support uploads for spack monitor, and eventually could be used for other kinds of spack uploads Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-14extending example for buildcaches (#22504)Vanessasaurus1-6/+78
* extending example for buildcaches I was attempting to create a local build cache from a directory, and I found the docs for both buildcaches and mirrors, but did not connect the docs that the url variable could be the local filesystem variable. I am extending the docs for buildcaches with an example of creating and interacting with one on the filesystem because I suspect other users will run into this need and possibly not find what they are looking for. Signed-off-by: vsoch <vsoch@users.noreply.github.com> * adding as follows to spack mirror list Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-06-14adding more description to binary caches (#23934)Vanessasaurus1-0/+23
It is currently kind of confusing to the reader to distinguish spack buildcache install and spack install, and it is not clear how to use a build cache once a mirror is added. Hopefully this little big of description can help (and I hope I got it right!) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-14oneAPI packages: fix install for python2 (#24296)Robert Cohn1-2/+2
Fix platform detection logic to work for Python 2 and 3
2021-06-12IntelPackage: use 'version_yearlike' in check for libfabrics RPATH. (#16700)Dominik Gresch1-1/+1
Use the 'version_yearlike' attribute instead of 'version' to check if the SPACK_COMPILER_EXTRA_RPATHS should be set to include the built-in 'libfabrics'. When using the bare 'version', the comparison is wrong when building with 'intel-parallel-studio', which has the version format '<edition>.YYYY.Nupdate', due to the leading '<edition>'.
2021-06-12Ensure all roots of an installed environment are marked explicit in db (#24277)Greg Becker2-0/+28
2021-06-11setup-env: allow users to skip module function setup (#24236)Adam J. Stewart1-3/+9
* setup-env: allow users to skip module function setup * Add documentation on SPACK_SKIP_MODULES
2021-06-11Display proper message when patch checksum doesn't match (#24229)iarspider1-1/+1
2021-06-08Pipelines: Fix default generated rebuild job script (#24185)Scott Wittenburg1-2/+1