diff options
| author | Omar Padron <omar.padron@kitware.com> | 2019-10-22 03:32:04 -0400 |
|---|---|---|
| committer | Todd Gamblin <tgamblin@llnl.gov> | 2019-10-22 00:32:04 -0700 |
| commit | fd58c98b0edd198e2b1bc8c9f81208d7d6010178 (patch) | |
| tree | 826bf9531541343d71ea4fa7fc28f36bc346135c /.gitlab-ci.yml | |
| parent | 6cb972a9d25cee6ed8b92195cccd8ad2255e98a9 (diff) | |
| download | spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.gz spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.bz2 spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.xz spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.zip | |
fetching: S3 upload and download (#11117)
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket.
High level to-do list:
- [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs.
- [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs.
- [x] Implement S3 source fetching
- [x] Implement S3 binary mirror indexing
- [x] Implement S3 binary package fetching
- [x] Implement S3 source pushing
- [x] Implement S3 binary package pushing
Important details:
* refactor URL handling to handle S3 URLs and mirror URLs more gracefully.
- updated parse() to accept already-parsed URL objects. an equivalent object
is returned with any extra s3-related attributes intact. Objects created with
urllib can also be passed, and the additional s3 handling logic will still be applied.
* update mirror schema/parsing (mirror can have separate fetch/push URLs)
* implement s3_fetch_strategy/several utility changes
* provide more feature-complete S3 fetching
* update buildcache create command to support S3
* Move the core logic for reading data from S3 out of the s3 fetch strategy and into
the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since
read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's
not clear whether the ideal design is to have S3 fetching functionality in a fetch
strategy, directly implemented in read_from_url, or both.
* expanded what can be passed to `spack buildcache` via the -d flag: In addition
to a directory on the local filesystem, the name of a configured mirror can be
passed, or a push URL can be passed directly.
Diffstat (limited to '.gitlab-ci.yml')
| -rw-r--r-- | .gitlab-ci.yml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index c4d6bcaab6..f0cfd456ff 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,8 @@ +# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + generate ci jobs: script: - "./bin/generate-gitlab-ci-yml.sh" |