summaryrefslogtreecommitdiff
path: root/.gitlab-ci.yml
diff options
context:
space:
mode:
authorOmar Padron <omar.padron@kitware.com>2019-10-22 03:32:04 -0400
committerTodd Gamblin <tgamblin@llnl.gov>2019-10-22 00:32:04 -0700
commitfd58c98b0edd198e2b1bc8c9f81208d7d6010178 (patch)
tree826bf9531541343d71ea4fa7fc28f36bc346135c /.gitlab-ci.yml
parent6cb972a9d25cee6ed8b92195cccd8ad2255e98a9 (diff)
downloadspack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.gz
spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.bz2
spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.tar.xz
spack-fd58c98b0edd198e2b1bc8c9f81208d7d6010178.zip
fetching: S3 upload and download (#11117)
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket. High level to-do list: - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs. - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs. - [x] Implement S3 source fetching - [x] Implement S3 binary mirror indexing - [x] Implement S3 binary package fetching - [x] Implement S3 source pushing - [x] Implement S3 binary package pushing Important details: * refactor URL handling to handle S3 URLs and mirror URLs more gracefully. - updated parse() to accept already-parsed URL objects. an equivalent object is returned with any extra s3-related attributes intact. Objects created with urllib can also be passed, and the additional s3 handling logic will still be applied. * update mirror schema/parsing (mirror can have separate fetch/push URLs) * implement s3_fetch_strategy/several utility changes * provide more feature-complete S3 fetching * update buildcache create command to support S3 * Move the core logic for reading data from S3 out of the s3 fetch strategy and into the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's not clear whether the ideal design is to have S3 fetching functionality in a fetch strategy, directly implemented in read_from_url, or both. * expanded what can be passed to `spack buildcache` via the -d flag: In addition to a directory on the local filesystem, the name of a configured mirror can be passed, or a push URL can be passed directly.
Diffstat (limited to '.gitlab-ci.yml')
-rw-r--r--.gitlab-ci.yml5
1 files changed, 5 insertions, 0 deletions
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c4d6bcaab6..f0cfd456ff 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,8 @@
+# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
generate ci jobs:
script:
- "./bin/generate-gitlab-ci-yml.sh"