summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-07-17Decompression: fix naming issues (#37749)Dan Lipsa2-35/+76
* When using system tools to unpack a .gz file, the input file needs a different name than the output file. Normally, we generate this new name by stripping off the .gz extension off of the file name. This was not sufficient if the file name did not have an extension, so we temporarily rename the file in that case. * When using system tar utility to untar on Windows, we were (erroneously) skipping the actual untar step if the filename was lacking a .tar extension * For foo.txz, we were not changing the extension of the decompressed file (i.e. we would decompress foo.txz to foo.txz). This did not cause any problems, but is confusing, so has been updated such that the output filename reflects its decompressed state (i.e. foo.tar). * Added test for strip_compression_extension * Update test_native_unpacking to test each archive type with and without an extension as part of the file name (i.e. we test "foo.tar.gz", but also make sure we decompress properly if it is named "foo").
2023-07-17Fix wrong StageComposite keep override (#38938)Harmen Stoppels2-2/+38
`Stage(keep=True)` was ignored when put in a composite that doesn't override the value.
2023-07-17Fix python import tests (#38928)Manuela Kuhn1-1/+1
Running `spack test run <python package>` resulted in the error ``` 'str' object is not callable ``` because the python executable was not set correctly.
2023-07-17Lock, database and store don't need global configuration on construction ↵Massimiliano Culpo24-337/+431
(#33495) Lock objects can now be instantiated independently, without being tied to the global configuration. The same is true for database and store objects. The database __init__ method has been simplified to take a single lock configuration object. Some common lock configurations (e.g. NO_LOCK or NO_TIMEOUT) have been named and are provided as globals. The use_store context manager keeps the configuration consistent by pushing and popping an internal scope. It can also be tuned by passing extra data to set up e.g. upstreams or anything else that might be related to the store.
2023-07-17Update new PythonPackage template to prefer --config-settings (#38918)Adam J. Stewart3-17/+12
2023-07-17Environment/depfile: fix bug with Git hash versions (attempt #2) (#37560)Peter Scheibel2-12/+91
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-07-17Minimal cleanup of a few tests in `test/packaging.py` (#38880)Massimiliano Culpo1-162/+125
* Minimal cleanup of a few tests in packaging.py * Use f-strings
2023-07-14Remove unused context manager (#38897)Harmen Stoppels1-8/+0
2023-07-14Fix issue on cray with super call (#38895)Rocco Meli1-1/+1
2023-07-13Bugfix/spack spec: read and use the environment concretizer:unification ↵Tamara Dahlgren2-4/+62
option (#38248) * Bugfix: spack.yaml concretizer:unify needs to be read and used * Optional: add environment test to ensure configuration scheme is used * Activate environment in unit tests A more proper solution would be to keep an environment instance configuration as an attribute, but that is a bigger refactor * Delay evaluation of Environment.unify * Slightly simplify unit tests --------- Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-07-13spack buildcache push: improve argparse (#38876)Harmen Stoppels1-8/+11
2023-07-13mirrors: distinguish between source/binary mirror; simplify schema (#34523)Harmen Stoppels11-298/+469
Allow the following formats: ```yaml mirrors: name: <url> ``` ```yaml mirrors: name: url: s3://xyz access_pair: [x, y] ``` ```yaml mirrors: name: fetch: http://xyz push: url: s3://xyz access_pair: [x, y] ``` And reserve two new properties to indicate the mirror type (e.g. mirror.spack.io is a source mirror, not a binary cache) ```yaml mirrors: spack-public: source: true binary: false url: https://mirror.spack.io ```
2023-07-13spack audit: allow skipping version checks from package.py (#28372)Massimiliano Culpo1-1/+12
A few packages have version directives evaluated within if statements, conditional on the value of `platform.platform()`. Sometimes there are no cases for e.g. platform=darwin and that causes a lot of spurious failures with version existence audits. This PR allows expressing conditions to skip version existence checks in audits and avoid these spurious reports.
2023-07-13Add type hints to spack.installer (#38872)Harmen Stoppels4-623/+670
2023-07-13Add missing space in error msg (#38863)Adam J. Stewart1-1/+1
2023-07-13Standardize subcommand help strings (#38804)Adam J. Stewart42-348/+295
### Rationale While working on #29549, I noticed a lot of inconsistencies in our argparse help messages. This is important for fish where these help messages end up as descriptions in the tab completion menu. See https://github.com/spack/spack/pull/29549#issuecomment-1627596477 for some examples of longer or more stylized help messages. ### Implementation This PR makes the following changes: - [x] help messages start with a lowercase letter. - [x] Help messages do not end with a period - [x] the first line of a help message is short and simple longer text is separated by an empty line - [x] "help messages do not use triple quotes" """(except docstrings)""" - [x] Parentheses not needed for string concatenation inside function call - [x] Remove "..." "..." string concatenation leftover from black reformatting - [x] Remove Sphinx argument docs from help messages The first 2 choices aren't very controversial, and are designed to match the syntax of the `--help` flag automatically added by argparse. The 3rd choice is more up for debate, and is designed to match our package/module docstrings. The 4th choice is designed to avoid excessive newline characters and indentation. We may actually want to go even further and disallow docstrings altogether. ### Alternatives Choice 3 in particular has a lot of alternatives. My goal is solely to ensure that fish tab completion looks reasonable. Alternatives include: 1. Get rid of long help messages, only allow short simple messages 2. Move longer help messages to epilog 3. Separate by 2 newline characters instead of 1 4. Separate by period instead of newline. First sentence goes into tab completion description The number of commands with long help text is actually rather small, and is mostly relegated to `spack ci` and `spack buildcache`. So 1 isn't actually as ridiculous as it sounds. Let me know if there are any other standardizations or alternatives you would like to suggest.
2023-07-12containers: retain shallow git data (#37734)Jonathon Anderson2-7/+13
2023-07-12Add path to MPI executables to ^mpi dependents (#35758)Michael Fink1-3/+7
2023-07-12vtk-m: correct cuda_arch variant behavior (#38697)Vicente Bolea2-0/+48
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2023-07-12installer: Improve status reporting (#37903)Michael Kuhn4-48/+64
Refactor `TermTitle` into `InstallStatus` and use it to show progress information both in the terminal title as well as inline. This also turns on the terminal title status by default. The inline output will look like the following after this change: ``` ==> Installing m4-1.4.19-w2fxrpuz64zdq63woprqfxxzc3tzu7p3 [4/4] ```
2023-07-10find: add `--hashes` shortcut for piping to other commands (#38663)Todd Gamblin1-0/+8
People frequently ask us how to pipe `spack find` output to other commands, and we tell them to do things like this: ```console $ spack find --format "/{hash}" | spack uninstall -ay ``` Sometimes users don't know about hash references and come up with potentially ambiguous formulations like this: ```console spack find --format {name}@{version}%{compiler} | spack uninstall -ay ``` Since this is a common enough thing to want to do, and to make it more obvious how, this PR adds a `-H` / `--hashes` as a shortcut, so you can now just do: ```console spack find -H | spack uninstall -ay ```
2023-07-09ci: Fix broken SPACK_CHECKOUT_VERSION (#38778)Jonathon Anderson2-17/+13
2023-07-09Don’t call spec.format in Database._get_matching_spec_key (#38792)Harmen Stoppels1-1/+15
`"%s" % spec` formats the spec with deps included, which produces sometimes KBs of data and is slow to run in pure Python. It can delay otherwise very short-lived read/write locks on the database. Discovered in #38762 where profile output showed about 2 seconds is spent in `spec.format`, which is significant overhead when using multiprocessing to install from binary cache in parallel (installation often takes <5s for small packages). With this change, `spec.format` no longer shows up in profile output. (This line hasn't changed since Spack v0.9 ;p) * move format() call to custom NoSuchSpecError exception * add a comment saying why, so we can eventually change `Spec.__str__`
2023-07-07Remove "node_compiler" from the list of unknown atoms (#38753)Massimiliano Culpo1-0/+1
2023-07-07Fix multiple quadratic complexity issues in environments (#38771)Harmen Stoppels3-63/+57
1. Fix O(n^2) iteration in `_get_overwrite_specs` 2. Early exit `get_by_hash` on full hash 3. Fix O(n^2) double lookup in `all_matching_specs` with hashes 4. Fix some legibility issues
2023-07-07Prevent "spack external find" to error out on wrong permissions (#38755)Massimiliano Culpo2-4/+36
fixes #38733
2023-07-07add info about spack env from spack-configs for oneAPI build tools (#38751)Andrey Parfenov1-0/+47
Signed-off-by: Andrey Parfenov <andrey.parfenov@intel.com>
2023-07-05Drop Python 2 super syntax (#38718)Adam J. Stewart62-245/+199
2023-07-05Drop Python 2 object subclassing (#38720)Adam J. Stewart103-226/+226
2023-07-05Installations: don't set group permissions when they match what is desired ↵Peter Scheibel2-0/+36
(#38036) * When installing a package Spack will attempt to set group permissions on the install prefix even when the configuration does not specify a group. Co-authored-by: David Gomez <dvdgomez@users.noreply.github.com>
2023-07-04spack commands: add type hints and docstrings (#38705)Adam J. Stewart3-159/+371
2023-07-04macos sip: apply on macos only, dont store LD_LIBRARY_PATH (#38687)Harmen Stoppels1-12/+20
2023-07-04spack make-installer: deterministic choice order (#38706)Adam J. Stewart1-3/+2
2023-07-04Remove from __future__ imports (#38703)Adam J. Stewart61-125/+1
2023-07-02SIPPackage: documentation fix (#38672)Adam J. Stewart1-1/+1
2023-06-30gitlab ci: reorganize when we check for specs on mirrors (#38626)Scott Wittenburg1-86/+85
Move the logic checking which mirrors have the specs we need closer to where that information is needed. Also update the staging summary to contain a brief description of why we scheduled or pruned each job. If a spec was found on any mirrors, regardless of whether we scheduled a job for it, print those mirrors.
2023-06-30tests/pythons: convert to new stand-alone test process (#38340)Tamara Dahlgren1-6/+9
2023-06-27Windows: Add PowerShell env support (#37951)John W. Parent7-13/+59
PowerShell requires explicit shell and env support in Spack. This is due to the distinct differences in shell interactions between cmd and pwsh. Add a doskey in pwsh piping 'spack' commands to a powershell script similar to the sh function 'spack'. Add support for PowerShell-specific shell interactions from Spack (set/unset shell variables).
2023-06-27ci: remove compiler bootstrapping code (#38543)Scott Wittenburg5-742/+193
2023-06-27Adjust cc90 to require at least CUDA 12 (#38572)Wileam Y. Phan1-3/+2
2023-06-27Revert "Updates to symlinking for Windows (#34701)" (#38578)Harmen Stoppels13-817/+191
This reverts commit 66f75407d136c0c06ed01774d5815728afba35ca.
2023-06-26CachedCMakePackage: set build type (#38502)Brian Van Essen1-0/+1
Fixed the cached CMake package so that the build_type field is saved in the cached configuration file.
2023-06-26Updates to symlinking for Windows (#34701)David Alexander13-191/+817
* Support hardlinks/junctions on Windows systems without developer mode enabled * Generally, use of llnl.util.symlink.symlink is preferred over os.symlink since it handles this automatically * Generally an error is now reported if a user attempts to create a symlink to a file that does not exist (this was previously allowed on Linux/Mac). * One exception to this: when Spack installs files from the source into their final prefix, dangling symlinks are allowed (on Linux/Mac - Windows does not allow this in any circumstance). The intent behind this is to avoid generating failures for installations on Linux/Mac that were succeeding before. * Because Windows is strict about forbidding dangling symlinks, `traverse_tree` has been updated to skip creating symlinks if they would point to a file that is ignored. This check is not transitive (i.e., a symlink to a symlink to an ignored file would not be caught appropriately) * Relocate function: resolve_link_target_relative_to_the_link (this is not otherwise modified) Co-authored-by: jamessmillie <smillie@txcorp.com>
2023-06-26modules: ignore more Modules variables in from_sourcing_file (#38455)Xavier Delaruelle3-6/+51
Update list of excluded variables in `from_sourcing_file` function to cover all variables specific to Environment Modules or Lmod. Add specifically variables relative to the definition of `module()`, `ml()` and `_module_raw()` Bash functions. Fixes #13504
2023-06-26Add raw attribute to env.set command (#38465)Xavier Delaruelle4-9/+42
Update `env.set` command and underlying `SetEnv` object to add the `raw` boolean attribute. `raw` is optional and set to False by default. When set to True, value format is skipped for object when generating environment modifications. With this change it is now possible to define environment variable whose value contains variable reference syntax (like `{foo}` or `{}`) that should be set as-is. Fixes #29578
2023-06-23spack buildcache sync (--manifest-glob <glob> | <src> <dest>) are mutually ↵Harmen Stoppels1-3/+6
exclusive (#38533)
2023-06-18cc: Ensure that user-specified flags take precedence over others (#37376)Andrey Parfenov2-195/+256
Spack flags supplied by users should supersede flags from package build systems and other places in Spack. However, Spack currently adds user-supplied flags to the beginning of the compile line, which means that in some cases build system flags will supersede user-supplied ones. The right place to add a flag to ensure it has highest precedence for the compiler really depends on the type of flag. For example, search paths like `-L` and `-I` are examined in order, so adding them first is highest precedence. Compilers take the *last* occurrence of optimization flags like `-O2`, so those should be placed *after* other such flags. Shim libraries with `-l` should go *before* other libraries on the command line, so we want user-supplied libs to go first, etc. `lib/spack/env/cc` already knows how to split arguments into categories like `libs_list`, `rpath_dirs_list`, etc., so we can leverage that functionality to merge user flags into the arg list correctly. The general rules for injected flags are: 1. All `-L`, `-I`, `-isystem`, `-l`, and `*-rpath` flags from `spack_flags_*` to appear before their regular counterparts. 2. All other flags ordered with the ones from flags after their regular counterparts, i.e. `other_flags` before `spack_flags_other_flags` - [x] Generalize argument categorization into its own function in the `cc` shell script - [x] Apply the same splitting logic to injected flags and flags from the original compile line. - [x] Use the resulting flag lists to merge user- and build-system-supplied flags by category. - [x] Add tests. Signed-off-by: Andrey Parfenov <andrey.parfenov@intel.com> Co-authored-by: iermolae <igor.ermolaev@intel.com>
2023-06-18fix oneapi modules (#38400)simonleary-umass-edu1-1/+1
2023-06-16unparser: drop Python 2, fix testing bugs with newer Pythons (#38424)Greg Becker2-129/+20
The `unparser` that Spack uses for package hashing had several tweaks to ensure compatibility with Python 2.7: 1. Currently, the unparser automatically moves `*` and `**` args to the end to preserve compatibility with `python@:3.4` 2. `print a, b, c` statements and single-tuple `print((a, b, c))` function calls were remapped to `print(a, b, c)` in the unparsed output for consistency across versions. (1) is causing issues in our tests because a recent patch to the Python source code (https://github.com/python/cpython/pull/102953/files#diff-7972dffec6674d5f09410c71766ac6caacb95b9bccbf032061806ae304519c9bR813-R823) has a `**` arg before an named argument, and we round-trip the core python source code as a test of our unparser. This isn't actually a break with our consistent unpausing -- it's still consistent, the python source just doesn't unparse to the same thing anymore. It does makes it harder to test, so it's not worth maintaining the Python2-specific stuff anymore. Since we only support `python@3.6:`, this PR removes (1) and (2) from the unparser, but keeps one last tweak for unicode AST inconsistencies, as it's still needed for Python 3.5-3.7. This fixes the CI error we've been seeing on `python@3.11.4` and `python@3.10.12`. Again, that bug exists only in the test system and doesn't affect our canonical hashing of Python code.
2023-06-16show external status as [e] (#33792)Greg Becker2-14/+34