| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This feature generates a verification manifest for each installed
package and provides a command, "spack verify", which can be used to
compare the current file checksums/permissions with those calculated
at installed time.
Verification includes
* Checksums of files
* File permissions
* Modification time
* File size
Packages installed before this PR will be skipped during verification.
To verify such a package you must reinstall it.
The spack verify command has three modes.
* With the -a,--all option it will check every installed package.
* With the -f,--files option, it will check some specific files,
determine which package they belong to, and confirm that they have
not been changed.
* With the -s,--specs option or by default, it will check some
specific packages that no files havae changed.
|
|
fixes #13005
This commit fixes an issue with the name of the root directory for
module file hierarchies. Since #3206 the root folder was named after
the microarchitecture used for the spec, which is too specific and
not backward compatible for lmod hierarchies. Here we compute the
root folder name using the target family instead of the target name
itself and we add target information in the 'whatis' portion of the
module file.
|
|
From Python docs:
--
'surrogateescape' will represent any incorrect bytes as code points in
the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These
private code points will then be turned back into the same bytes when
the surrogateescape error handler is used when writing data. This is
useful for processing files in an unknown encoding.
--
This will allow us to process files with unknown encodings.
To accommodate the case of self-extracting bash scripts, filter_file
can now stop filtering text input if a certain marker is found. The
marker must be passed at call time via the "stop_at" function argument.
At that point the file will be reopened in binary mode and copied
verbatim.
* use "surrogateescape" error handling to ignore unknown chars
* permit to stop filtering if a marker is found
* add unit tests for non-ASCII and mixed text/binary files
|
|
* bugfix: install --only dependents works in env
includes regression testing
|
|
`spack url stats` now also looks at packages' resources when outputting
statistics.
Example:
```
$ spack url stats
==> URL stats for 3531 packages:
--------------------------------------------------------------
stat versions % resources %
--------------------------------------------------------------
url 8335 89.3% 339 89.0%
schemes
https 6489 69.5% 93 24.4%
ftp 32 0.3% 8 2.1%
http 1763 18.9% 237 62.2%
file 51 0.5% 1 0.3%
checksums
md5 26 0.3% 0 0.0%
sha256 8306 89.0% 336 88.2%
no checksum 3 0.0% 3 0.8%
--------------------------------------------------------------
go 1 0.0% 0 0.0%
--------------------------------------------------------------
hg 7 0.1% 0 0.0%
--------------------------------------------------------------
no code 4 0.0% 0 0.0%
--------------------------------------------------------------
svn 4 0.0% 16 4.2%
--------------------------------------------------------------
git 981 10.5% 26 6.8%
branch 442 4.7% 4 1.0%
commit 362 3.9% 14 3.7%
no ref 36 0.4% 2 0.5%
tag 141 1.5% 6 1.6%
--------------------------------------------------------------
```
|
|
|
|
* add `--cache-only` option to install
* testing for `--cache-only`
* remove extraneous stage creation at stage destroy time
|
|
- Add a test that verifies checksums on all packages
- Also add an attribute to packages that indicates whether they need a
manual download or not, and add an exception in the tests for these
packages until we can verify them.
|
|
|
|
|
|
|
|
|
|
|
|
Both floating-point and NEON are required in all standard ARMv8
implementations. Theoretically though specialized markets can support
no NEON or floating-point at all. Source:
https://developer.arm.com/docs/den0024/latest/aarch64-floating-point-and-neon
On the other hand the base procedure call standard for Aarch64
"assumes the availability of the vector registers for passing
floating-point and SIMD arguments". Further "the Arm 64-bit
architecture defines two mandatory register banks: a general-purpose
register bank which can be used for scalar integer processing and
pointer arithmetic; and a SIMD and Floating-Point register bank".
Source:
https://developer.arm.com/docs/ihi0055/latest/procedure-call-standard-for-the-arm-64-bit-architecture
This makes customization of Aarch64 with no NEON instruction set
available so unlikely that we can consider them a feature of the
generic family.
|
|
fixes #13111
Due to a missing case we were treating a single target that was not
equal to the one we were comparing to as a range open on the right.
|
|
replace_prefix_bin. (#13114)
This should fix a Python3 error from concatenating strings and bytes.
|
|
fixes #12010
|
|
|
|
This PR adds a 'concretize' entry to an environment's spec.yaml file
which controls how user specs are concretized. By default it is
set to 'separately' which means that each spec added by the user is
concretized separately (the behavior of environments before this PR).
If set to 'together', the environment will concretize all of the
added user specs together; this means that all specs and their
dependencies will be consistent with each other (for example, a
user could develop code linked against the set of libraries in the
environment without conflicts).
If the environment was previously concretized, this will re-concretize
all specs, in which case previously-installed specs may no longer be
used by the environment (in this sense, adding a new spec to an
environment with 'concretize: together' can be significantly more
expensive).
The 'concretize: together' setting is not compatible with Spec
matrices; this PR adds a check to look for multiple instances of the
same package added to the environment and fails early when
'concretize: together' is set (to avoid confusing messages about
conflicts later on).
|
|
Applying accepted fix from spack/spack.io#4
|
|
While the build environment already takes share/pkgconfig into account,
the generated module files etc. only consider lib/pkgconfig and
lib64/pkgconfig.
|
|
When removing support for dotkit in #11986 the code trying to set the
paths of the various module files was not updated to skip it. This
results in a failure because of a key error after the deprecation
warning is displayed to user.
This commit fixes the issue and adds a unit test for regression.
Note that code for Spack chains has been updated accordingly but
no unit test has been added for that case.
|
|
|
|
fixes #12928
|
|
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.
A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.
* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
|
|
|
|
|
|
* binary cache: show all packages for compatible differing targets
* Don't restrict spack buildcache list to arch or os
* Fix from merge conflict
|
|
fixes #12915
closes #12916
Since Spack has support for specific targets it might happen that
software is built for targets that are not exactly the host because
it was either an explicit user request or the compiler being used is
too old to support the host.
Modules for different targets are written into different directories
and by default Spack was adding to MODULEPATH only the directory
corresponding to the current host. This PR modifies this behavior to
add all the directories that are **compatible** with the current host.
|
|
|
|
Sometimes when remove_file is called on a link, that link is missing
(perhaps ctrl-C happened halfway through a previous action). As
removing a non-existent file is no problem, this patch changes the
behavior so Spack continues rather than stopping with an error.
Currently you would see
ValueError: /path/to/dir is not a link tree!
and now it continues with a warning.
|
|
bin/spack now needs to have a "-*- python -*-" line after the shebang, so
that emacs will interpret it as a python file instead of as a shell
script. Add one line to the license check limit to accommodate this.
|
|
LLVM, mesa and other packages check for these generic
microarchitectures. One solution is to let Spack know they exist.
|
|
This fixes #12852 where perl builds that use Build.PL will fail when the
shebang of the Build script produced from the configure step is too
long.
|
|
Changes deps and rpaths for bins and libs, changes id for libs.
|
|
|
|
Linux (#12909)
|
|
The output of subprocess.check_output is a byte string in Python 3. This causes dictionary lookup to fail later on.
A try-except around this function prevented this error from being noticed. Removed this so that more errors can propagate out.
|
|
|
|
|
|
Preferred targets were failing because we were looking them up by
Microarchitecture object, not by string.
- [x] Add a call to `str()` to fix target lookup.
- [x] Add a test to exercise this part of concretization.
- [x] Add documentation for setting `target` in `packages.yaml`
|
|
* microarchitectures: zen starts from x86_64, not from excavator
* Unit tests: fixed a test that is wrong with the new modeling
* microarchitectures: fixed features and inheritance for 15h family
bulldozer doesn't inherit from barcelona (10h) + added xop, lwp and tbm
instruction sets to the 15h family (it distinguish the family from 17h)
|
|
|
|
Addresses #12804
This PR adds the creation of the remaining (16) templates to ensure we can create them with expected content. The goal is to facilitate catching during testing.
|
|
|
|
Spack doesn't need `requests`, and neither does `jsonschema`, but
`jsonschema` tries to import it, and it'll succeed if `requests` is on
your machine (which is likely, given how popular it is). This commit
removes the import to improve Spack's startup time a bit.
On a mac with SSD, the import of requests is ~28% of Spack's startup time
when run as `spack --print-shell-vars sh,modules` (.069 / .25 seconds),
which is what `setup-env.sh` runs.
On a Linux cluster where Python is mounted from NFS, this reduces
`setup-env.sh` source time from ~1s to .75s.
Note: This issue will be eliminated if we upgrade to a newer `jsonschema`
(we'd need to drop Python 2.6 for that). See
https://github.com/Julian/jsonschema/pull/388.
|
|
|
|
- This is needed to support Cray machines -- we need an architecture
mic_knl > x86_64
- We used Cray's naming scheme for this target to make it work seamlessly
with the module-based detection sccheme on Cray. mic_knl is pretty
much dead, so this will be the last succh target. We will need to work
wtih Cray and other vendors in the future.
|
|
Seamless translation from 'target=<generic>' to either
- target.family == <generic> (in methods)
- 'target=<generic>:' (in directives)
Also updated docs to show ranges in directives.
|
|
Spack can now:
- label ppc64, ppc64le, x86_64, etc. builds with specific
microarchitecture-specific names, like 'haswell', 'skylake' or
'icelake'.
- detect the host architecture of a machine from /proc/cpuinfo or similar
tools.
- Understand which microarchitectures are compatible with which (for
binary reuse)
- Understand which compiler flags are needed (for GCC, so far) to build
binaries for particular microarchitectures.
All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets. The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.
The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines. Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.
Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.
Specifics:
- Add microarchitectures.json with descriptions of architectures
- Relaxed semantic of compiler's "target" attribute. Before this change
the semantic to check if a compiler could be viable for a given target
was exact match. This made sense as the finest granularity of targets
was architecture families. As now we can target micro-architectures,
this commit changes the semantic by interpreting as the architecture
family what is stored in the compiler's "target" attribute. A compiler
is then a viable choice if the target being concretized belongs to the
same family. Similarly when a new compiler is detected the architecture
family is stored in the "target" attribute.
- Make Spack's `cc` compiler wrapper inject target-specific flags on the
command line
- Architecture concretization updated to use the same algorithm as
compiler concretization
- Micro-architecture features, vendor, generation etc. are included in
the package hash. Generic architectures, such as x86_64 or ppc64, are
still dumped using the name only.
- If the compiler for a target is not supported exit with an intelligible
error message. If the compiler support is unknown don't try to use
optimization flags.
- Support and define feature aliases (e.g., sse3 -> ssse3) in
microarchitectures.json and on Microarchitecture objects. Feature
aliases are defined in targets.json and map a name (the "alias") to a
list of rules that must be met for the test to be successful. The rules
that are available can be extended later using a decorator.
- Implement subset semantics for comparing microarchitectures (treat
microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
a) can all be false.
- Implement logic to automatically demote the default target if the
compiler being used is too old to optimize for it. Updated docs to make
this behavior explicit. This avoids surprising the user if the default
compiler is older than the host architecture.
This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type. A more careful
refactor that takes into account the type system might be due later.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
|
