Packaging of netlib-lapack for windows (#24993)

MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml

Saving progress on packaging zlib for Windows

Fixing the shared CMake flag

* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py

* Giving CMake a Fortran compiler argument

* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows

* Have netlib-lapack depend  on ninja@1.10

Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>

Making a default config.yaml for Windows

Small path length for build_stage

Provide more prerequisite details, mention default config.yaml

Killing an unnecessary setvars call

Replacing some lost changes, proofreading, updating windows-supported package list

Co-authored-by: John Parent <john.parent@kitware.com>

7 files changed, 190 insertions, 103 deletions

diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst
index 353d97bf3f..3022c27dba 100644
--- a/lib/spack/docs/getting_started.rst
+++ b/lib/spack/docs/getting_started.rst
@@ -1523,11 +1523,9 @@ linux distro.
 Spack On Windows
 ----------------
 
-Windows support for Spack is currently under development. While this work is
-still in an early stage, it is currently possible to set up Spack and
-perform a few operations on Windows. This section will guide
-you through the steps needed to install Spack and start running it on a
-fresh Windows machine.
+Windows support for Spack is currently under development.  While this work is still in an early stage,
+it is currently possible to set up Spack and perform a few operations on Windows.  This section will guide
+you through the steps needed to install Spack and start running it on a fresh Windows machine. 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Step 1: Install prerequisites
@@ -1536,48 +1534,82 @@ Step 1: Install prerequisites
 To use Spack on Windows, you will need the following packages:
 
 * Microsoft Visual Studio
-* Python
+* Intel Fortran (needed for some packages)
+* Python 
 * Git
+* Perl (needed for some packages)
+* NASM (needed for some packages)
+* CMake
 
 """""""""""""""""""""""
 Microsoft Visual Studio
 """""""""""""""""""""""
 
-Microsoft Visual Studio provides the Windows C/C++ compiler that is
-currently supported by Spack.
+Microsoft Visual Studio provides the Windows C/C++ compiler that is currently supported by Spack.
 
-We require several specific components to be included in the Visual Studio
-installation. One is the C/C++ toolset, which can be selected as "Desktop
-development with C++" or "C++ build tools," depending on installation type
-(Professional, Build Tools, etc.)  The other required component is
-"C++ CMake tools for Windows," which can be selected from among the optional
-packages. This provides CMake and Ninja for use during Spack configuration.
+We require several specific components to be included in the Visual Studio installation.
+One is the C/C++ toolset, which can be selected as "Desktop development with C++" or "C++ build tools,"
+depending on installation type (Professional, Build Tools, etc.)  The other required component is
+"C++ CMake tools for Windows," which can be selected from among the optional packages.
+This provides CMake and Ninja for use during Spack configuration.
 
-If you already have Visual Studio installed, you can make sure these
-components are installed by rerunning the installer. Next to your
-installation, select "Modify" and look at the "Installation details" pane on the right.
+If you already have Visual Studio installed, you can make sure these components are installed by
+rerunning the installer.  Next to your installation, select "Modify" and look at the
+"Installation details" pane on the right.
+
+"""""""""""""
+Intel Fortran
+"""""""""""""
+
+For Fortran-based packages on Windows, we strongly recommend Intel's oneAPI Fortran compilers.
+The suite is free to download from Intel's website, located at 
+https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/fortran-compiler.html#gs.70t5tw.
+The executable of choice for Spack will be Intel's Beta Compiler, ifx, which supports the classic
+compiler's (ifort's) frontend and runtime libraries by using LLVM.
 
 """"""
 Python
 """"""
 
-As Spack is a Python-based package, an installation of Python will be needed
-to run it. Python 3 can be downloaded and installed from the Windows Store,
-and will be automatically added to your ``PATH`` in this case.
+As Spack is a Python-based package, an installation of Python will be needed to run it.
+Python 3 can be downloaded and installed from the Windows Store, and will be automatically added
+to your ``PATH`` in this case.
 
 """
 Git
 """
 
 A bash console and GUI can be downloaded from https://git-scm.com/downloads.
+If you are unfamiliar with Git, there are a myriad of resources online to help
+guide you through checking out repositories and switching development branches.
 
 When given the option of adjusting your ``PATH``, choose the ``Git from the
 command line and also from 3rd-party software`` option. This will automatically
 update your ``PATH`` variable to include the ``git`` command.
 
-If you are unfamiliar with Git, there are a myriad of resources online to help
-guide you through checking out repositories and switching development
-branches.
+""""
+Perl
+""""
+
+Perl is a flexible and feature-rich programming language that comes built-in
+on Unix boxes but needs to be installed externally for Windows users. Fortunately,
+you can find the most recent release at https://www.perl.org/get.html.
+
+""""
+NASM
+""""
+
+The Netwide Assembler (NASM) is a x86-64 assembler that some Windows packages
+will use to create binaries and can be found at https://www.nasm.us.
+
+"""""
+CMake
+"""""
+
+While the CMake provided by your Microsoft Visual Studio installation should
+suffice for most packages, we still recommend downloading and installing the
+most recent version of the software at https://cmake.org/download/ in case
+of version restrictions.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Step 2: Install and setup Spack
@@ -1629,31 +1661,19 @@ To configure Spack, first run the following command inside the Spack console:
 
    spack compiler find
 
-This creates a ``.spack`` directory in our home directory, along with a
-``windows`` subdirectory containing a ``compilers.yaml`` file. On a fresh
-Windows install, the only compiler that should be found is your installation
-of Microsoft Visual Studio.
-
-We need to provide the ``config.yaml`` configuration by ourselves. This goes
-in the ``.spack\windows`` directory in your home directory. Open your text
-editor of choice and enter the following lines for ``config.yaml``:
-
-.. code-block:: yaml
+This creates a ``.spack`` directory in our home directory, along with a ``windows`` subdirectory
+containing a ``compilers.yaml`` file. On a fresh Windows install with the above packages
+installed, this command should only detect Microsoft Visual Studio and the Intel Fortran
+compiler will be integrated within the first version of MSVC present in the ``compilers.yaml``
+output.
 
-   config:
-     locks: false
-     install_tree:
-       root: $spack\opt\spack
-       projections:
-         all: '${ARCHITECTURE}\${COMPILERNAME}-${COMPILERVER}\${PACKAGE}-${VERSION}-${HASH}'
-     build_stage:
-       - ~/.spack/stage
-
-(These settings are identical to those in the default ``config.yaml``
-provided with your Spack checkout, except with forward slashes replaced by
-backslashes for Windows compatibility.) It is important that all indentions
-in .yaml files are done with spaces and not tabs, so take care when editing
-one by hand.
+Spack provides a default ``config.yaml`` file for Windows that it will use unless overridden.
+This file is located at ``etc\spack\defaults\windows\config.yaml``. You can read more on how to
+do this and write your own configuration files in the :ref:`Configuration Files<configuration>` section of our
+documentation. If you do this, pay particular attention to the ``build_stage`` block of the file
+as this specifies the directory that will temporarily hold the source code for the packages to
+be installed. This path name must be sufficiently short for compliance with cmd, otherwise you
+will see build errors during installation (particularly with CMake) tied to long path names.
 
 For the ``packages.yaml`` file, there are two options. The first
 and easiest choice is to use Spack to find installation on your system. In
@@ -1668,10 +1688,9 @@ The ``spack external find <name>`` will find executables on your system
 with the same name given. The command will store the items found in
 ``packages.yaml`` in the ``.spack\`` directory.
 
-Assuming the Spack found CMake and Ninja executables in the previous
-step, continue to Step 4. If no executables were found, we will need to
-direct spack towards the CMake and Ninja installations we set up with
-Visual Studio. Therefore, your ``packages.yaml`` file will look something
+Assuming that the command found CMake and Ninja executables in the previous
+step, continue to Step 4. If no executables were found, we need to manually direct spack towards the CMake
+and Ninja installations we set up with Visual Studio. Therefore, your ``packages.yaml`` file will look something
 like this, with possibly slight variants in the paths to CMake and Ninja:
 
 .. code-block:: yaml
@@ -1688,27 +1707,18 @@ like this, with possibly slight variants in the paths to CMake and Ninja:
          prefix: 'c:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja'
        buildable: False
 
-It is important to note that the version of your Ninja and CMake could
-be different than what is shown here. If there is a difference, make sure
-to use your version instead of the version listed above. Similiarly, if
-you use a different version of Visual Studio ("Community" for example),
-make sure the Professional part of the location is changed to your version.
-
-The ``packages.yaml`` file should be placed inside either the ``.spack``
-directory or the ``.spack\windows`` directory.
-
 You can also use an separate installation of CMake if you have one and prefer
-to use it. If you don't have a path to Ninja analogous to the above, then
-you can obtain it by running the Visual Studio Installer and following the
-instructions at the start of this section.
-
+to use it. If you don't have a path to Ninja analogous to the above, then you can
+obtain it by running the Visual Studio Installer and following the instructions
+at the start of this section. Also note that .yaml files use spaces for indentation
+and not tabs, so ensure that this is the case when editing one directly.
 
 ^^^^^^^^^^^^^^^^^
 Step 4: Use Spack
 ^^^^^^^^^^^^^^^^^
 
-Once the configuration is complete, it is time to give the installation a
-test. Install a basic package through the Spack console via:
+Once the configuration is complete, it is time to give the installation a test.  Install a basic package though the
+Spack console via:
 
 .. code-block:: console
 
@@ -1725,6 +1735,9 @@ packages known to work on Windows:
 * abseil-cpp
 * cpuinfo
 * glm
+* netlib-lapack (requires Intel Fortran)
+* openssl
+* zlib
 
 ^^^^^^^^^^^^^^
 For developers
diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py
index 02f32766bd..979a6bb024 100644
--- a/lib/spack/spack/build_systems/cmake.py
+++ b/lib/spack/spack/build_systems/cmake.py
@@ -174,6 +174,10 @@ class CMakePackage(PackageBase):
             define('CMAKE_CXX_COMPILER:FILEPATH', pkg.compiler.cxx.replace('\\', '/'))
         ]
 
+        if pkg.compiler.fc is not None:
+            args.append(define('CMAKE_Fortran_COMPILER:FILEPATH',
+                               pkg.compiler.fc.replace('\\', '/')))
+
         # CMAKE_INTERPROCEDURAL_OPTIMIZATION only exists for CMake >= 3.9
         if pkg.spec.satisfies('^cmake@3.9:'):
             args.append(define('CMAKE_INTERPROCEDURAL_OPTIMIZATION', ipo))
diff --git a/lib/spack/spack/compilers/__init__.py b/lib/spack/spack/compilers/__init__.py
index 5aad6b2b98..8e9917488c 100644
--- a/lib/spack/spack/compilers/__init__.py
+++ b/lib/spack/spack/compilers/__init__.py
@@ -211,7 +211,6 @@ def find_compilers(path_hints=None):
     for o in all_os_classes():
         search_paths = getattr(o, 'compiler_search_paths', default_paths)
         arguments.extend(arguments_to_detect_version_fn(o, search_paths))
-
     # Here we map the function arguments to the corresponding calls
     tp = multiprocessing.pool.ThreadPool()
     try:
diff --git a/lib/spack/spack/compilers/intel.py b/lib/spack/spack/compilers/intel.py
index 83ea87ccd7..d52feebb4d 100644
--- a/lib/spack/spack/compilers/intel.py
+++ b/lib/spack/spack/compilers/intel.py
@@ -3,6 +3,8 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import sys
+
 from spack.compiler import Compiler, UnsupportedCompilerFlag
 from spack.version import ver
 
@@ -29,8 +31,15 @@ class Intel(Compiler):
     PrgEnv = 'PrgEnv-intel'
     PrgEnv_compiler = 'intel'
 
-    version_argument = '--version'
-    version_regex = r'\((?:IFORT|ICC)\) ([^ ]+)'
+    if sys.platform == 'win32':
+        version_argument = '/QV'
+    else:
+        version_argument = '--version'
+
+    if sys.platform == 'win32':
+        version_regex = r'([1-9][0-9]*\.[0-9]*\.[0-9]*)'
+    else:
+        version_regex = r'\((?:IFORT|ICC)\) ([^ ]+)'
 
     @property
     def verbose_flag(self):
diff --git a/lib/spack/spack/compilers/msvc.py b/lib/spack/spack/compilers/msvc.py
index a613d02d36..d0ae6921ea 100644
--- a/lib/spack/spack/compilers/msvc.py
+++ b/lib/spack/spack/compilers/msvc.py
@@ -8,6 +8,8 @@ import subprocess
 import sys
 from typing import List  # novm
 
+import spack.operating_systems.windows_os
+import spack.util.executable
 from spack.compiler import Compiler
 
 
@@ -19,20 +21,24 @@ class Msvc(Compiler):
     cxx_names = ['cl.exe']
 
     # Subclasses use possible names of Fortran 77 compiler
-    f77_names = []  # type: List[str]
+    f77_names = ['ifx.exe']  # type: List[str]
 
     # Subclasses use possible names of Fortran 90 compiler
-    fc_names = []  # type: List[str]
+    fc_names = ['ifx.exe']  # type: List[str]
 
     # Named wrapper links within build_env_path
-    link_paths = {'cc': 'msvc/cl.exe',
-                  'cxx': 'msvc/cl.exe',
+    link_paths = {'cc': '',
+                  'cxx': '',
                   'f77': '',
                   'fc': ''}
 
     #: Compiler argument that produces version information
     version_argument = ''
 
+    # For getting ifx's version, call it with version_argument
+    # and ignore the error code
+    ignore_version_errors = [1]
+
     #: Regex used to extract version from compiler's output
     version_regex = r'([1-9][0-9]*\.[0-9]*\.[0-9]*)'
 
@@ -41,10 +47,15 @@ class Msvc(Compiler):
 
     def __init__(self, *args, **kwargs):
         super(Msvc, self).__init__(*args, **kwargs)
-        self.vcvarsallfile = os.path.abspath(
-            os.path.join(self.cc, '../../../../../../..'))
-        self.vcvarsallfile = os.path.join(
-            self.vcvarsallfile, 'Auxiliary', 'Build', 'vcvarsall.bat')
+        if os.getenv("ONEAPI_ROOT"):
+            # If this found, it sets all the vars
+            self.setvarsfile = os.path.join(
+                os.getenv("ONEAPI_ROOT"), "setvars.bat")
+        else:
+            self.setvarsfile = os.path.abspath(
+                os.path.join(self.cc, '../../../../../../..'))
+            self.setvarsfile = os.path.join(
+                self.setvarsfile, 'Auxiliary', 'Build', 'vcvars64.bat')
 
     @property
     def verbose_flag(self):
@@ -55,28 +66,60 @@ class Msvc(Compiler):
         return ""
 
     def setup_custom_environment(self, pkg, env):
-        """Set environment variables for MSVC using the Microsoft-provided
-        script."""
+        """Set environment variables for MSVC using the
+        Microsoft-provided script."""
         if sys.version_info[:2] > (2, 6):
-            # Capture output from batch script and DOS environment dump
+            # Set the build environment variables for spack. Just using
+            # subprocess.call() doesn't work since that operates in its own
+            # environment which is destroyed (along with the adjusted variables)
+            # once the process terminates. So go the long way around: examine
+            # output, sort into dictionary, use that to make the build
+            # environment.
             out = subprocess.check_output(  # novermin
-                'cmd /u /c "{0}" {1} && set'.format(self.vcvarsallfile, 'amd64'),
+                'cmd /u /c "{}" {} && set'.format(self.setvarsfile, 'amd64'),
                 stderr=subprocess.STDOUT)
             if sys.version_info[0] >= 3:
-                out = out.decode('utf-16le', errors='replace')
+                out = out.decode('utf-16le', errors='replace')  # novermin
+
+            int_env = {  # novermin
+                key.lower(): value
+                for key, _, value in
+                (line.partition('=') for line in out.splitlines())
+                if key and value
+            }
+
+            if 'path' in int_env:
+                env.set_path('PATH', int_env['path'].split(';'))
+            env.set_path('INCLUDE', int_env.get('include', '').split(';'))
+            env.set_path('LIB', int_env.get('lib', '').split(';'))
         else:
+            # Should not this be an exception?
             print("Cannot pull msvc compiler information in Python 2.6 or below")
 
-        # Process in to nice Python dictionary
-        vc_env = {  # novermin
-            key.lower(): value
-            for key, _, value in
-            (line.partition('=') for line in out.splitlines())
-            if key and value
-        }
-
-        # Request setting environment variables
-        if 'path' in vc_env:
-            env.set_path('PATH', vc_env['path'].split(';'))
-        env.set_path('INCLUDE', vc_env.get('include', '').split(';'))
-        env.set_path('LIB', vc_env.get('lib', '').split(';'))
+    # fc_version only loads the ifx compiler into the first MSVC stanza;
+    # if there are other versions of Microsoft VS installed and detected, they
+    # will only have cl.exe as the C/C++ compiler
+
+    @classmethod
+    def fc_version(cls, fc):
+        # We're using intel for the Fortran compilers, which exist if
+        # ONEAPI_ROOT is a meaningful variable
+        if os.getenv("ONEAPI_ROOT"):
+            try:
+                sps = spack.operating_systems.windows_os.WindowsOs.compiler_search_paths
+            except Exception:
+                print("sps not found.")
+                raise
+            try:
+                clp = spack.util.executable.which_string("cl", path=sps)
+            except Exception:
+                print("cl not found.")
+                raise
+            ver = cls.default_version(clp)
+            return ver
+        else:
+            return cls.default_version(fc)
+
+    @classmethod
+    def f77_version(cls, f77):
+        return cls.fc_version(f77)
diff --git a/lib/spack/spack/detection/path.py b/lib/spack/spack/detection/path.py
index 6f6e7c32a7..8db209f009 100644
--- a/lib/spack/spack/detection/path.py
+++ b/lib/spack/spack/detection/path.py
@@ -16,6 +16,7 @@ import warnings
 import llnl.util.filesystem
 import llnl.util.tty
 
+import spack.operating_systems.windows_os as winOs
 import spack.util.environment
 
 from .common import (
@@ -40,7 +41,18 @@ def executables_in_path(path_hints=None):
         path_hints (list): list of paths to be searched. If None the list will be
             constructed based on the PATH environment variable.
     """
+    # build_environment.py::1013: If we're on a Windows box, run vswhere, steal the installationPath using
+    # windows_os.py logic, construct paths to CMake and Ninja, add to PATH
     path_hints = path_hints or spack.util.environment.get_path('PATH')
+    if sys.platform == 'win32':
+      msvcPaths = winOs.WindowsOs.vsInstallPaths
+      msvcCMakePaths = [os.path.join(path, "Common7", "IDE", "CommonExtensions", "Microsoft", "CMake", "CMake", "bin")
+                   for path in msvcPaths]
+      [path_hints.insert(0, path) for path in msvcCMakePaths]
+      msvcNinjaPaths = [os.path.join(path, "Common7", "IDE", "CommonExtensions", "Microsoft", "CMake", "Ninja")
+                   for path in msvcPaths]
+      [path_hints.insert(0, path) for path in msvcNinjaPaths]
+
     search_paths = llnl.util.filesystem.search_paths_for_executables(*path_hints)
 
     path_to_exe = {}
diff --git a/lib/spack/spack/operating_systems/windows_os.py b/lib/spack/spack/operating_systems/windows_os.py
index a9132c7bce..91b7ddb678 100755
--- a/lib/spack/spack/operating_systems/windows_os.py
+++ b/lib/spack/spack/operating_systems/windows_os.py
@@ -10,7 +10,6 @@ import glob
 from spack.architecture import OperatingSystem
 from spack.version import Version
 
-
 # FIXME: To get the actual Windows version, we need a python that runs
 # natively on Windows, not Cygwin.
 def windows_version():
@@ -22,21 +21,23 @@ def windows_version():
 class WindowsOs(OperatingSystem):
     """This class represents the Windows operating system.  This will be
     auto detected using the python platform.win32_ver() once we have a
-    python setup that runs natively.  The Windows platform will be represented
-    using the major version operating system number, e.g. 10.
+    python setup that runs natively.  The Windows platform will be
+    represented using the major version operating system number, e.g.
+    10.
     """
 
     # Find MSVC directories using vswhere
     compSearchPaths = []
+    vsInstallPaths = []
     root = os.environ.get('ProgramFiles(x86)') or os.environ.get('ProgramFiles')
     if root:
         try:
             extra_args = {}
             if sys.version_info[:3] >= (3, 6, 0):
                 extra_args = {'encoding': 'mbcs', 'errors': 'strict'}
-            paths = subprocess.check_output([
-                os.path.join(root, "Microsoft Visual Studio", "Installer",
-                             "vswhere.exe"),
+            paths = subprocess.check_output([  # novermin
+                os.path.join(root, "Microsoft Visual Studio",
+                             "Installer", "vswhere.exe"),
                 "-prerelease",
                 "-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
                 "-property", "installationPath",
@@ -44,16 +45,22 @@ class WindowsOs(OperatingSystem):
             ], **extra_args).strip()
             if (3, 0) <= sys.version_info[:2] <= (3, 5):
                 paths = paths.decode()
-            msvcPaths = paths.split('\n')
+            vsInstallPaths = paths.split('\n')
             msvcPaths = [os.path.join(path, "VC", "Tools", "MSVC")
-                         for path in msvcPaths]
+                         for path in vsInstallPaths]
             for p in msvcPaths:
                 compSearchPaths.extend(
                     glob.glob(os.path.join(p, '*', 'bin', 'Hostx64', 'x64')))
+            if os.getenv("ONEAPI_ROOT"):
+                comp_search_paths.extend(glob.glob(os.path.join(
+                    str(os.getenv("ONEAPI_ROOT")),
+                    'compiler', '*',
+                    'windows', 'bin')))
         except (subprocess.CalledProcessError, OSError, UnicodeDecodeError):
             pass
     if compSearchPaths:
         compiler_search_paths = compSearchPaths
+    # print(vsInstallPaths)
 
     def __init__(self):
         super(WindowsOs, self).__init__('Windows10', '10')