Legion package overhaul (#23252)

Complete overhaul of the Legion package to better capture a more
up-to-date set of configuration options and variants.  This update
adds additional flexibility and features that were requested by
users.

* Add version 21.03.0 and "stable" branch
* Remove all older numeric versions
* Add support for CUDA, Python, PAPI support and more
* Add maintainer
* This no longer uses the Spack `gasnet` package: it defaults to
  using an embedded gasnet or can be pointed to an external

2 files changed, 450 insertions, 59 deletions

diff --git a/var/spack/repos/builtin/packages/legion/README.md b/var/spack/repos/builtin/packages/legion/README.md
new file mode 100644
index 0000000000..f8cac2e5d4
--- /dev/null
+++ b/var/spack/repos/builtin/packages/legion/README.md
@@ -0,0 +1,153 @@
+# Legion Spack Package Notes
+
+The Legion Spack package follows the underlying approach used in Legion's CMake-based build infrastructure.  If you are not familiar with using Spack, we encourage you to start with the **[Spack 101 Tutorial](https://spack-tutorial.readthedocs.io/en/latest/)**.  For more background on Legion please visit the project [website](https://legion.stanford.edu). There you can find information on getting started, tutorials, and supporting documentation.
+
+## Versions
+
+We strongly recommend the majority of users start with the latest **stable** branch of Legion.  This will guarantee the most regularly tested and debugged version of the code.  We also strive to have quarterly tagged release throughout the calendar year starting March (e.g., legion-YY.MM.0 where MM starts at *'03'* and runs through *'12'* for any calendar year).  These versioned downloads are captured via the [Legion GitHub page](https://github.com/StanfordLegion/legion) and captured in the Spack package for individual use via the naming convention above.  The details for each release are captured in the CHANGES.txt file at the top-level of the repository.
+
+## Spack Usage and Details
+
+**NOTE: This version of the Legion *spackage* no longer depends upon an external GASNet spackage.  Instead, as supported and preferred by the GASNet developers, Legion internalizes the GASNet configuration and build process tailored to Legion's needs.  At present this approach should minimize both building and configuration issues that have occurred in the past.**
+
+With a working installation of Spack following variants can be used to download, configure, and build legion.  As of its latest release Legion uses features from C++11 and can be build using either GCC or Clang (and likely any other compilers with full C++11 support).  For example,
+
+    spack install legion %clang@10.0
+
+will build and install Legion using version 10.0 of the clang compiler.  Additional compiler flags (beyond those automatically enabled by the underlying Legion CMake configuration) may be specified by using the `cppflags` option to Spack:
+
+    spack install legion %clang@10.0 cppflags=-Wall
+
+You can find more details about customizing the compiler with Spack [here](https://spack-tutorial.readthedocs.io/en/latest/tutorial_basics.html#customizing-compilers).
+
+There are a number of configuration parameters (variants) for Legion that can be specified on the command line.  To see a complete listing of these variants you can use the following command:
+
+    spack info legion
+
+The default build and install of Legion is suitable for laptop/desktop development needs.  Additional variants will need to be specified to build for distributed memory systems, GPU support, and other features (e.g., debugging, profiling, etc.).  A full list of the currently supported variants is provided below.
+
+These variants may then be used on the Spack command line to customize the build of Legion you would like to install.  For boolean-based variants the `+`, `-`, or `~` *sigils* can be used. In this case, the `+` specifies a `True` value and either `-` or `~` can be used for the `False` setting (two values for `False` are provided to avoid conflicts with shell command line parameters. For example,
+
+    spack install legion@stable~cuda+hdf5%clang10.0
+
+will build/install the stable version of Legion without CUDA and with HDF5 support; using Clang 10.0 as the compiler.  See below for more examples of the various package options. 
+
+## Overview of Legion's Spack Variants
+
+
+This section provides an overview of the variants that are available for builds and installations of Legion and the underlying Realm runtime libraries.  These variants range from important configuration parameters for many users, to more involved and low-level details more likely to be leveraged when debugging correctness and performance issues.  For these last set of use cases we encourage users to visit the Legion [Debugging](https://legion.stanford.edu/debugging/) and [Profiling](https://legion.stanford.edu/profiling/) pages.
+
+Note that any variants with a default of `on` or `off` are boolean values and may be configured on the Spack command line using the `+`, or `-` (or `~`) sigils.
+
+Finally, to build the highest performing installation of Legion requires an appropriate configuration and installation of GASNet-Ex (while we have support for an MPI transport layer it is still in testing and evaluation for achieving the best possible performance).  See the following section for details on the steps needed to build a high-performance install of Legion using Spack.
+
+### Debugging & Internal Runtime Configurations
+
+* **`build_type`**: This variant exposes the CMake build type setting.  Following directly from CMake these available values for this variant are [`Debug`, `RelWithDebInfo`, and `Release`]. `default=RelWithDebInfo`
+
+* **`bounds_checks`**:  This variant supports `on` or `off` values and enables, or disables bounds checking within the runtime's data accessors.  This is helpful for debugging but  does have the potential to degrade performance (therefore it is disabled in the default build).  `default=off`
+
+* **`privilege_checks`**: This variant enables the runtime checking of data privileges in Legion's data accessors. This can helpful in debugging applications. `default=off`
+
+* **`spy`**: This variant enables applications to produce detailed logging information for debugging with the [Legion Spy](https://legion.stanford.edu/debugging/#legion-spy) tool.  `default=off`
+
+* **`max_dims`**: This variant provides an integral value for the maximum number of dimensions in a logical region that are supported by the build/install.  `default=3`  *Note: Currently supported values range from 1 to 9.*
+
+* **`max_fields`**: This variant provides an integral value for the maximum number of fields that may be stored within a logical region.  `default=512` *Note: Internally this requires a value that is a power-of-two -- if a non power-of-two value is provided the next largest power-of-two will be used.*
+
+* **`output_level`**: This variant enables the (dynamic) debugging level for the runtime.  It can be one of the following values: [`spew`, `debug`, `info`, `print`, `warning`, `error`, `fatal`, or `none`].  `default=warning`
+
+* **`papi`**: This variant enables PAPI performance measurements.  `default=off`
+
+* **`redop_complex`** This variant enables support for the reduction of complex types. `default=off`  Note: will be enabled when `+bindings` is set.
+
+* **`shared_libs`**: Build shared libraries for Legion and Realm. `default=off`
+
+* **`enable_tls`**: This variant supports `on` or `off` and enables thread-local-storage of the Legion runtime context.  `default=off`
+
+### Bindings & Interoperability
+
+* **`bindings`**: This variant supports `on` or `off` and will build the language bindings for Legion.  In addition to the required C++ interface, this currently includes C, Python, and Fortran. `default=off`  Note you can use `+python` or `+fortran` to select a subset of these bindings.
+
+* **`python`**: This variant enables Python support in terms of both bindings and runtime support (where instance(s) of the Python interpreter must be available.) `default=off`
+
+* **`fortran`**: This variant supports `on` or `off` and enables building of Fortran language bindings for Legion. `default=off`
+
+[//]: <> (TOOD: More details here on Kokkos interop?)
+
+* **`kokkos`**: Enable support for interoperability with [Kokkos](https://github.com/kokkos) use in Legion tasks. `default=off`
+
+[//]: <> (TOOD: More details here on OpenMP interop?)
+
+* **`openmp`**: This variant enables OpenMP support within Legion tasks (and within the Realm runtime). Please note that the full OpenMP feature set (e.g. OpenMP 5.0) is not fully supported when enabling this feature.  `default=off`
+
+* **`libdl`**: Enable support for dynamic object loading (via "libdl").  `default=on`
+
+### Processor Architecture Support (e.g., GPUs)
+
+* **`cuda`**: This variant supports `on` or `off` and enables CUDA support within Legion.  `default=off`
+
+* **`cuda_arch`**: This variant specifics the specific CUDA architecture to support within the Legion build/installation.  Currently this variant must be one of [`60`, `70`, `75`, or `80`].  Where `60` is the Pascal architecture, `70` is for Volta, `75` is for Turing, and `80` is for `Volta`. `default=70`
+
+* **`cuda_hijack`**:  This variant supports `on` or `off` and determines if the build enables performance enhancements by "*hijacking* entry points into CUDA's runtime API; thus, it obviously implies `+cuda`.This is a performance enhancement and not necessary but suggested for production use cases on NVIDIA-based systems.  `default=off`
+
+### External Library Support
+
+* **`hdf5`**: This variant supports `on` or `off` and enables building of HDF5 support within the runtime. `default=off`
+
+* **`hwloc`**: Build using "libhwloc" support for numa-aware topology support within Realm (Legion's low-level runtime layer).  `default=off`
+
+* **`zlib`**: This variant enables support for zlib. `default=on`
+
+### Distributed Memory/Network Transport/Interconnect Support
+
+* **`network`**: This variant specifies what network transport layer Legion/Realm should use for moving data across the distributed memories of a system.  The options are currently: [`gasnet`, `mpi`, or `none`].  The most tested and reliable interface for distributed memory systems is `gasnet`.  Via this package GASNet will be automatically configured and built. If you want to use a pre-installed version of GASNet you can use the `gasnet_root` variant; this will disable the automatic/internal installation of GASNet. We strongly encourage the use of the automatic configuration and build mechanism as it is tailored to Legion's use cases. If you would like to read more about please visit their [web page](https://gasnet.lbl.gov).  Note that the `mpi` transport layer is new and still being tuned and debugged.  Finally, if you want to run on a single system (e.g., a desktop or laptop) for development activities you may drop this parameter or specify `network=none`.  `default=none`
+
+* **`conduit`** requires `(network=gasnet)` **only**: This variant selects the GASNet conduit to use and must be provided when `network=gasnet`.  The current choices are: [`aries`, `ibv`, `mpi`, `ucx`, `udp`, or `none`]:  defaults to `none`.
+  * **aries**: Aries for Cray XC series (see [documentation](https://gasnet.lbl.gov/dist-ex/aries-conduit/README)).
+  * **ibv**: OpenIB/OpenFabrics Verbs for InfiniBand (see [documentation](https://gasnet.lbl.gov/dist-ex/ibv-conduit/README)).
+  * **mpi**: Portable conduit, works on any network with MPI 1.1 or newer (see [documentation](https://gasnet.lbl.gov/dist-ex/mpi-conduit/README)).
+  * **ucx**: *(NOTE: EXPERIMENTAL)*  Unified Communication X framework (see [documentation](https://gasnet.lbl.gov/dist-ex/ucx-conduit/README)).
+  * **udp**: Portable conduit, works on any network with a TCP/IP stack (see [documentation](https://gasnet.lbl.gov/dist-ex/udp-conduit/README)).
+
+* **`gasnet_root`**: This variant points the package at a pre-installed version of GASNet for use when building Legion.  This skips the default behavior of embedding the GASNet build within that of Legion. It requires that `conduit` be set to match that used by the installed version of GASNet (*note that incorrectly setting `conduit` could result in failures that are not caught by the Spack package as failures will only occur during the build of Legion*). `default=/usr/local`
+
+## Examples
+
+This section highlights a few common configurations/installations for various common installations of Legion. 
+
+### Laptop/Desktop Development
+
+The default configuration will build a version of Legion without network support that is often helpful in laptop or desktop installs for development using Legion. The first example is the most straightforward and enables a CPU-only configuration of Legion:
+
+`$ spack install legion@stable`
+
+Given that many uses cases for laptop/desktop development involve debugging Legion you might want to consider enabling [Legion Spy](https://legion.stanford.edu/debugging/#legion-spy), bounds and privilege checks, and optionally enable debugging level output diagnostics:
+
+`$ spack install legion@stable +spy +bounds_checks +privilege_checks [output=debug]`
+
+Note that by specifying `+spy` the package will also install the Legion profiler.  Both will be available in the corresponding `bin` directory created by Spack's install mechanisms.
+
+#### Adding GPU Support
+
+To enable support for NVIDIA GPUs with Legion you can add the `+cuda` variant to the examples provided above.  In addition, you can also use the `cuda_arch` flag to enable specific GPU architectures.  For example, the follow command line installs the stable version of Legion with CUDA support for NVIDIA's Volta architecture:
+
+`$ spack install legion@stable +cuda cuda_arch=70`
+
+If you wanted to program tasks using Kokkos support for GPUs you can simply add the `+kokkos` option to the previous command line: 
+
+`$ spack install legion@stable +cuda cuda_arch=70 +kokkos`
+
+this will enable Kokkos interoperability in Legion and also build and configure a version of Kokkos with cuda and Volta GPU support.
+
+#### Networking/Distributed Memory Support 
+
+In order to add modify the above configuration to support a distributed memory architecture you need to add a `network` target.  For example, to build a version of Legion that uses GASNet for the data transport layer you would specify:
+
+`$ spack install legion@stable +cuda cuda_arch=70 +kokkos network=gasnet`
+
+Without the addition of a specific GASNet *conduit* this command will use the UDP conduit. To change this to support a Infniband-based cluster you would add:
+
+`$ spack install legion@stable +cuda cuda_arch=70 +kokkos network=gasnet conduit=ibv`
+
+For a pre-existing installation of GASNet you can point the package at the root directory of the installation (also often know as the installation *prefix*). For example, to point the package at a version installed under `/opt/local` (*i.e., the include files are stored under `/opt/local/include` and libraries are in `/opt/local/lib`*) you would add `gasnet_root=/opt/local` to the command line.
diff --git a/var/spack/repos/builtin/packages/legion/package.py b/var/spack/repos/builtin/packages/legion/package.py
index 105d3c231d..d278a918f6 100644
--- a/var/spack/repos/builtin/packages/legion/package.py
+++ b/var/spack/repos/builtin/packages/legion/package.py
@@ -3,8 +3,8 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-
 from spack import *
+import os
 
 
 class Legion(CMakePackage):
@@ -19,79 +19,317 @@ class Legion(CMakePackage):
        interface provides explicit programmer controlled placement of data in
        the memory hierarchy and assignment of tasks to processors in a way
        that is orthogonal to correctness, thereby enabling easy porting and
-       tuning of Legion applications to new architectures.
-    """
+       tuning of Legion applications to new architectures."""
+
     homepage = "http://legion.stanford.edu/"
-    url      = "https://github.com/StanfordLegion/legion/tarball/legion-17.02.0"
-    git      = "https://github.com/StanfordLegion/legion.git"
+    git = "https://github.com/StanfordLegion/legion.git"
 
-    maintainers = ['streichler']
+    maintainers = ['pmccormick', 'streichler']
 
+    version('21.03.0', tag='legion-21.03.0')
+    version('stable', branch='stable')
     version('master', branch='master')
-    version('ctrl-rep', branch='control_replication')
-    version('20.03.0', sha256='ae5feedb5ed9f357b56424b9d73cea4f224a61e291e022556f796d1ff24d1244')
-    version('19.12.0', sha256='ea517638de7256723bb9c119796d4d9d4ef662c52d0151ad24af5288e5a72e7d')
-    version('19.09.1', sha256='c507133fb9dce16b7fcccd7eb2933d13cce96ecf835da60a27c0f66840cabf51')
-    version('19.09.0', sha256='a01c3e3c6698cafb64b77a66341cc06d039faed4fa31b764159f021b94ce13e8')
-    version('19.06.0', sha256='31cd97e9264c510ab83b1f9e8e1e6bf72021a0c6ee4a028966fce08736e39fbf')
-    version('19.04.0', sha256='279bbc8dcdab4c75be570318989a9fc9821178143e9db9c3f62e58bf9070b5ac')
-    version('18.12.0', sha256='71f2c409722975c0ad92f2caffcc9eaa9260f7035e2b55b731d819eb6a94016c')
-    version('18.09.0', sha256='58c5a3072d2b5086225982563c23524692ca5758cbfda8d0f0a4f00ef17b3b8d')
-    version('18.05.0', sha256='4c3cef548b3a459827e4c36b5963c06b6fcf0a4ca1800fbb0f73e6ba3b1cced4')
-    version('18.02.0', sha256='e08aeef98003593391a56f11a99d9d65af49647fe87a2a5e8837c8682a337a60')
-    version('17.10.0', sha256='af4f1e9215e57c4aac4805ae2bf53defe13eeaf192576bf5a702978f43171b1e')
-    version('17.08.0', sha256='20aabdb0fabb1e32aa713cd5fa406525093f8dad33fca5d23046408d42d3c7b3')
-    version('17.02.0', sha256='423d8699729b0e7fef663740e239aa722cca544f6bda8c9f782eaba4274bf60a')
-
-    variant('mpi', default=True,
-            description='Build on top of mpi conduit for mpi inoperability')
-    variant('ibv', default=False,
-            description='Build on top of ibv conduit for InfiniBand support')
-    variant('shared', default=True, description='Build shared libraries')
-    variant('hdf5', default=True, description='Enable HDF5 support')
+    version('cr', branch='control_replication')
+
+    depends_on("cmake@3.16:", type='build')
+    # TODO: Need to spec version of MPI v3 for use of the low-level MPI transport
+    # layer. At present the MPI layer is still experimental and we discourge its
+    # use for general (not legion development) use cases.
+    depends_on('mpi', when='network=mpi')
+    depends_on('mpi', when='network=gasnet')  # MPI is required to build gasnet (needs mpicc).
+    depends_on('ucx', when='conduit=ucx')
+    depends_on('mpi', when='conduit=mpi')
+    depends_on('cuda@10.0:11.9', when='+cuda_unsupported_compiler')
+    depends_on('cuda@10.0:11.9', when='+cuda')
+    depends_on('hdf5', when='+hdf5')
+    depends_on('hwloc', when='+hwloc')
+
+    # cuda-centric
+    # reminder for arch numbers to names: 60=pascal, 70=volta, 75=turing, 80=ampere
+    # TODO: we could use a map here to clean up and use naming vs. numbers.
+    cuda_arch_list = ('60', '70', '75', '80')
+    for nvarch in cuda_arch_list:
+        depends_on('kokkos@3.3.01+cuda+cuda_lambda+wrapper cuda_arch={0}'.format(nvarch),
+                   when='%gcc+kokkos+cuda cuda_arch={0}'.format(nvarch))
+        depends_on("kokkos@3.3.01+cuda+cuda_lambda~wrapper cuda_arch={0}".format(nvarch),
+                   when="%clang+kokkos+cuda cuda_arch={0}".format(nvarch))
+
+    depends_on('kokkos@3.3.01~cuda', when='+kokkos~cuda')
+    depends_on("kokkos@3.3.01~cuda+openmp", when='kokkos+openmp')
+
+    depends_on('python@3', when='+python')
+    depends_on('papi', when='+papi')
+    depends_on('zlib', when='+zlib')
+
+    # TODO: Need a AMD/HIP variant to match support landing in 21.03.0.
+
+    # Network transport layer: the underlying data transport API should be used for
+    # distributed data movement.  For Legion, gasnet is the currently the most
+    # mature.  We have many users that default to using no network layer for
+    # day-to-day development thus we default to 'none'.  MPI support is new and
+    # should be considered as a beta release.
+    variant('network', default='none',
+            values=('gasnet', 'mpi', 'none'),
+            description="The network communications/transport layer to use.",
+            multi=False)
+
+    # We default to automatically embedding a gasnet build. To override this
+    # point the package a pre-installed version of GASNet-Ex via the gasnet_root
+    # variant.
+    #
+    # make sure we have a valid directory provided for gasnet_root...
+    def validate_gasnet_root(value):
+        if value == 'none':
+            return True
+
+        if not os.path.isdir(value):
+            print("gasnet_root:", value, "-- no such directory.")
+            return False
+        else:
+            return True
+
+    variant('gasnet_root',
+            default='none',
+            values=validate_gasnet_root,
+            description="Path to a pre-installed version of GASNet (prefix directory).",
+            multi=False)
+    conflicts('gasnet_root', when="network=mpi")
+
+    variant('conduit', default='none',
+            values=('aries', 'ibv', 'udp', 'mpi', 'ucx', 'none'),
+            description="The gasnet conduit(s) to enable.",
+            multi=False)
+
+    conflicts('conduit=none', when='network=gasnet',
+              msg="a conduit must be selected when 'network=gasnet'")
+
+    gasnet_conduits = ('aries', 'ibv', 'udp', 'mpi', 'ucx')
+    for c in gasnet_conduits:
+        conflict_str = 'conduit=%s' % c
+        conflicts(conflict_str, when='network=mpi',
+                  msg="conduit attribute requires 'network=gasnet'.")
+        conflicts(conflict_str, when='network=none',
+                  msg="conduit attribute requires 'network=gasnet'.")
+
+    variant('gasnet_debug', default=False,
+            description="Build gasnet with debugging enabled.")
+    conflicts('+gasnet_debug', when='network=mpi')
+    conflicts('+gasnet_debug', when='network=none')
+
+    variant('shared_libs', default=False,
+            description="Build shared libraries.")
+
+    variant('bounds_checks', default=False,
+            description="Enable bounds checking in Legion accessors.")
+
+    variant('privilege_checks', default=False,
+            description="Enable runtime privildge checks in Legion accessors.")
+
+    variant('enable_tls', default=False,
+            description="Enable thread-local-storage of the Legion context.")
+
+    variant('output_level', default='warning',
+            # Note: these values are dependent upon those used in the cmake config.
+            values=("spew", "debug", "info", "print", "warning", "error", "fatal",
+                    "none"),
+            description="Set the compile-time logging level.",
+            multi=False)
+
     variant('spy', default=False,
-            description='Enable detailed logging for Legion Spy')
-    variant('build_type', default='Release',
-            values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'),
-            description='The build type to build', multi=False)
+            description="Enable detailed logging for Legion Spy debugging.")
+
+    # note: we will be dependent upon spack's latest-and-greatest cuda version...
+    variant('cuda', default=False,
+            description="Enable CUDA support.")
+    variant('cuda_hijack', default=False,
+            description="Hijack application calls into the CUDA runtime (+cuda).")
+    variant('cuda_arch', default='70',
+            values=cuda_arch_list,
+            description="GPU/CUDA architecture to build for.",
+            multi=False)
+    variant('cuda_unsupported_compiler', default=False,
+            description="Disable nvcc version check (--allow-unsupported-compiler).")
+    conflicts('+cuda_hijack', when='~cuda')
+
+    variant('fortran', default=False,
+            description="Enable Fortran bindings.")
+
+    variant('hdf5', default=False,
+            description="Enable support for HDF5.")
+
+    variant('hwloc', default=False,
+            description="Use hwloc for topology awareness.")
+
+    variant('kokkos', default=False,
+            description="Enable support for interoperability with Kokkos.")
+
+    variant('bindings', default=False,
+            description="Build runtime language bindings (excl. Fortran).")
+
+    variant('libdl', default=True,
+            description="Enable support for dynamic object/library loading.")
+
+    variant('openmp', default=False,
+            description="Enable support for OpenMP within Legion tasks.")
+
+    variant('papi', default=False,
+            description="Enable PAPI performance measurements.")
+
+    variant('python', default=False,
+            description="Enable Python support.")
 
-    depends_on("cmake@3.1:", type='build')
-    depends_on("gasnet~aligned-segments~pshm segment-mmap-max='16GB'", when='~mpi')
-    depends_on("gasnet~aligned-segments~pshm segment-mmap-max='16GB' +mpi", when='+mpi')
-    depends_on("gasnet~aligned-segments~pshm segment-mmap-max='16GB' +ibv", when='+ibv')
-    depends_on("hdf5", when='+hdf5')
+    variant('zlib', default=True,
+            description="Enable zlib support.")
+
+    variant('redop_complex', default=False,
+            description="Use reduction operators for complex types.")
+
+    variant('max_dims', values=int, default=3,
+            description="Set max number of dimensions for logical regions.")
+    variant('max_fields', values=int, default=512,
+            description="Maximum number of fields allowed in a logical region.")
+
+    variant('native', default=False,
+            description="Enable native/host processor optimizaton target.")
 
     def cmake_args(self):
-        cmake_cxx_flags = [
-            '-DPRIVILEGE_CHECKS',
-            '-DBOUNDS_CHECKS',
-            '-DENABLE_LEGION_TLS']
-
-        options = [
-            '-DLegion_USE_GASNet=ON',
-            '-DLEGION_USE_CUDA=OFF',
-            '-DLEGION_USE_OPENMP=OFF',
-            '-DLegion_BUILD_EXAMPLES=ON',
-            '-DBUILD_SHARED_LIBS=%s' % ('+shared' in self.spec)]
-
-        if self.spec.variants['build_type'].value == 'Debug':
+        spec = self.spec
+        cmake_cxx_flags = []
+        options = []
+
+        if 'network=gasnet' in spec:
+            options.append('-DLegion_NETWORKS=gasnetex')
+            if spec.variants['gasnet_root'].value != 'none':
+                gasnet_dir = spec.variants['gasnet_root'].value
+                options.append('-DGASNet_ROOT_DIR=%s' % gasnet_dir)
+            else:
+                options.append('-DLegion_EMBED_GASNet=ON')
+                gasnet_dir = join_path(self.stage.source_path, 'gasnet')
+                options.append('-DLegion_EMBED_GASNet_LOCALSRC=%s' % gasnet_dir)
+
+            gasnet_conduit = spec.variants['conduit'].value
+            options.append('-DGASNet_CONDUIT=%s' % gasnet_conduit)
+
+            if '+gasnet_debug' in spec:
+                options.append('-DLegion_EMBED_GASNet_CONFIGURE_ARGS=--enable-debug')
+        elif 'network=mpi' in spec:
+            options.append('-DLegion_NETWORKS=mpi')
+            if spec.variants['gasnet_root'].value != 'none':
+                raise InstallError("'gasnet_root' is only valid when 'network=gasnet'.")
+        else:
+            if spec.variants['gasnet_root'].value != 'none':
+                raise InstallError("'gasnet_root' is only valid when 'network=gasnet'.")
+            options.append('-DLegion_EMBED_GASNet=OFF')
+
+        if '+shared_libs' in spec:
+            options.append('-DBUILD_SHARED_LIBS=ON')
+        else:
+            options.append('-DBUILD_SHARED_LIBS=OFF')
+
+        if '+bounds_checks' in spec:
+            # default is off.
+            options.append('-DLegion_BOUNDS_CHECKS=ON')
+        if '+privilege_checks' in spec:
+            # default is off.
+            options.append('-DLegion_PRIVILEGE_CHECKS=ON')
+        if '+enable_tls' in spec:
+            # default is off.
+            options.append('-DLegion_ENABLE_TLS=ON')
+        if 'output_level' in spec:
+            level = str.upper(spec.variants['output_level'].value)
+            options.append('-DLegion_OUTPUT_LEVEL=%s' % level)
+        if '+spy' in spec:
+            # default is off.
+            options.append('-DLegion_SPY=ON')
+
+        if '+cuda' in spec:
+            cuda_arch = spec.variants['cuda_arch'].value
+            options.append('-DLegion_USE_CUDA=ON')
+            options.append('-DLegion_GPU_REDUCTIONS=ON')
+            options.append('-DLegion_CUDA_ARCH=%s' % cuda_arch)
+            if '+cuda_hijack' in spec:
+                options.append('-DLegion_HIJACK_CUDART=ON')
+            else:
+                options.append('-DLegion_HIJACK_CUDART=OFF')
+
+            if '+cuda_unsupported_compiler' in spec:
+                options.append('-DCUDA_NVCC_FLAGS:STRING=--allow-unsupported-compiler')
+
+        if '+fortran' in spec:
+            # default is off.
+            options.append('-DLegion_USE_Fortran=ON')
+
+        if '+hdf5' in spec:
+            # default is off.
+            options.append('-DLegion_USE_HDF5=ON')
+
+        if '+hwloc' in spec:
+            # default is off.
+            options.append('-DLegion_USE_HWLOC=ON')
+
+        if '+kokkos' in spec:
+            # default is off.
+            options.append('-DLegion_USE_Kokkos=ON')
+            os.environ['KOKKOS_CXX_COMPILER'] = spec['kokkos'].kokkos_cxx
+
+        if '+libdl' in spec:
+            # default is on.
+            options.append('-DLegion_USE_LIBDL=ON')
+        else:
+            options.append('-DLegion_USE_LIBDL=OFF')
+
+        if '+openmp' in spec:
+            # default is off.
+            options.append('-DLegion_USE_OpenMP=ON')
+
+        if '+papi' in spec:
+            # default is off.
+            options.append('-DLegion_USE_PAPI=ON')
+
+        if '+python' in spec:
+            # default is off.
+            options.append('-DLegion_USE_Python=ON')
+
+        if '+zlib' in spec:
+            # default is on.
+            options.append('-DLegion_USE_ZLIB=ON')
+        else:
+            options.append('-DLegion_USE_ZLIB=OFF')
+
+        if '+redop_complex' in spec:
+            # default is off.
+            options.append('-DLegion_REDOP_COMPLEX=ON')
+
+        if '+bindings' in spec:
+            # default is off.
+            options.append('-DLegion_BUILD_BINDINGS=ON')
+            options.append('-DLegion_REDOP_COMPLEX=ON')  # required for bindings
+            options.append('-DLegion_USE_Fortran=ON')
+
+        if spec.variants['build_type'].value == 'Debug':
             cmake_cxx_flags.extend([
                 '-DDEBUG_REALM',
                 '-DDEBUG_LEGION',
                 '-ggdb',
             ])
 
-        options.append('-DCMAKE_CXX_FLAGS=%s' % (" ".join(cmake_cxx_flags)))
+        maxdims = int(spec.variants['max_dims'].value)
+        # TODO: sanity check if maxdims < 0 || > 9???
+        options.append('-DLegion_MAX_DIM=%d' % maxdims)
 
-        if '+mpi' in self.spec:
-            options.append('-DGASNet_CONDUIT=mpi')
+        maxfields = int(spec.variants['max_fields'].value)
+        if (maxfields <= 0):
+            maxfields = 512
+        # make sure maxfields is a power of two.  if not,
+        # find the next largest power of two and use that...
+        if (maxfields & (maxfields - 1) != 0):
+            while maxfields & maxfields - 1:
+                maxfields = maxfields & maxfields - 1
+            maxfields = maxfields << 1
+        options.append('-DLegion_MAX_FIELDS=%d' % maxfields)
 
-        if '+hdf5' in self.spec:
-            options.append('-DLegion_USE_HDF5=ON')
-        else:
-            options.append('-DLegion_USE_HDF5=OFF')
-
-        if '+spy' in self.spec:
-            options.append('-DLegion_SPY=ON')
+        if '+native' in spec:
+            # default is off.
+            options.append('-DBUILD_MARCH:STRING=native')
 
         return options