diff options
Diffstat (limited to 'lib/spack/docs/build_systems/pythonpackage.rst')
| -rw-r--r-- | lib/spack/docs/build_systems/pythonpackage.rst | 777 |
1 files changed, 275 insertions, 502 deletions
diff --git a/lib/spack/docs/build_systems/pythonpackage.rst b/lib/spack/docs/build_systems/pythonpackage.rst index 365c5d7bce..6fdd3ee9b0 100644 --- a/lib/spack/docs/build_systems/pythonpackage.rst +++ b/lib/spack/docs/build_systems/pythonpackage.rst @@ -9,216 +9,80 @@ PythonPackage ------------- -Python packages and modules have their own special build system. +Python packages and modules have their own special build system. This +documentation covers everything you'll need to know in order to write +a Spack build recipe for a Python library. -^^^^^^ -Phases -^^^^^^ - -The ``PythonPackage`` base class provides the following phases that -can be overridden: - -* ``build`` -* ``build_py`` -* ``build_ext`` -* ``build_clib`` -* ``build_scripts`` -* ``install`` -* ``install_lib`` -* ``install_headers`` -* ``install_scripts`` -* ``install_data`` - -These are all standard ``setup.py`` commands and can be found by running: - -.. code-block:: console - - $ python setup.py --help-commands - - -By default, only the ``build`` and ``install`` phases are run: - -#. ``build`` - build everything needed to install -#. ``install`` - install everything from build directory - -If for whatever reason you need to run more phases, simply modify your -``phases`` list like so: - -.. code-block:: python - - phases = ['build_ext', 'install'] - - -Each phase provides a function ``<phase>`` that runs: - -.. code-block:: console - - $ python -s setup.py --no-user-cfg <phase> - - -Each phase also has a ``<phase_args>`` function that can pass arguments to -this call. All of these functions are empty except for the ``install_args`` -function, which passes ``--prefix=/path/to/installation/prefix``. There is -also some additional logic specific to setuptools and eggs. - -If you need to run a phase that is not a standard ``setup.py`` command, -you'll need to define a function for it like so: - -.. code-block:: python - - phases = ['configure', 'build', 'install'] - - def configure(self, spec, prefix): - self.setup_py('configure') - - -^^^^^^ -Wheels -^^^^^^ - -Some Python packages are closed-source and distributed as wheels. -Instead of using the ``PythonPackage`` base class, you should extend -the ``Package`` base class and implement the following custom installation -procedure: - -.. code-block:: python - - def install(self, spec, prefix): - pip = which('pip') - pip('install', self.stage.archive_file, '--prefix={0}'.format(prefix)) - - -This will require a dependency on pip, as mentioned below. - -^^^^^^^^^^^^^^^ -Important files -^^^^^^^^^^^^^^^ - -Python packages can be identified by the presence of a ``setup.py`` file. -This file is used by package managers like ``pip`` to determine a -package's dependencies and the version of dependencies required, so if -the ``setup.py`` file is not accurate, the package will not build properly. -For this reason, the ``setup.py`` file should be fairly reliable. If the -documentation and ``setup.py`` disagree on something, the ``setup.py`` -file should be considered to be the truth. As dependencies are added or -removed, the documentation is much more likely to become outdated than -the ``setup.py``. - -The Python ecosystem has evolved significantly over the years. Before -setuptools became popular, most packages listed their dependencies in a -``requirements.txt`` file. Once setuptools took over, these dependencies -were listed directly in the ``setup.py``. Newer PEPs introduced additional -files, like ``setup.cfg`` and ``pyproject.toml``. You should look out for -all of these files, as they may all contain important information about -package dependencies. - -Some Python packages are closed-source and are distributed as Python -wheels. For example, ``py-azureml-sdk`` downloads a ``.whl`` file. This -file is simply a zip file, and can be extracted using: - -.. code-block:: console - - $ unzip *.whl - - -The zip file will not contain a ``setup.py``, but it will contain a -``METADATA`` file which contains all the information you need to -write a ``package.py`` build recipe. - -.. _pypi: - -^^^^ -PyPI -^^^^ +^^^^^^^^^^^ +Terminology +^^^^^^^^^^^ -The vast majority of Python packages are hosted on PyPI (The Python -Package Index), which is :ref:`preferred over GitHub <pypi-vs-github>` -for downloading packages. ``pip`` only supports packages hosted on PyPI, making -it the only option for developers who want a simple installation. -Search for "PyPI <package-name>" to find the download page. Note that -some pages are versioned, and the first result may not be the newest -version. Click on the "Latest Version" button to the top right to see -if a newer version is available. The download page is usually at:: +In the Python ecosystem, there are a number of terms that are +important to understand. - https://pypi.org/project/<package-name> +**PyPI** + The `Python Package Index <https://pypi.org/>`_, where most Python + libraries are hosted. +**sdist** + Source distributions, distributed as tarballs (.tar.gz) and zip + files (.zip). Contain the source code of the package. -Since PyPI is so common, the ``PythonPackage`` base class has a -``pypi`` attribute that can be set. Once set, ``pypi`` will be used -to define the ``homepage``, ``url``, and ``list_url``. For example, -the following: +**bdist** + Built distributions, distributed as wheels (.whl). Contain the + pre-built library. -.. code-block:: python - - homepage = 'https://pypi.org/project/setuptools/' - url = 'https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip' - list_url = 'https://pypi.org/simple/setuptools/' - - -is equivalent to: - -.. code-block:: python +**wheel** + A binary distribution format common in the Python ecosystem. This + file is actually just a zip file containing specific metadata and + code. See the + `documentation <https://packaging.python.org/en/latest/specifications/binary-distribution-format/>`_ + for more details. - pypi = 'setuptools/setuptools-49.2.0.zip' +**build frontend** + Command-line tools used to build and install wheels. Examples + include `pip <https://pip.pypa.io/>`_, + `build <https://pypa-build.readthedocs.io/>`_, and + `installer <https://installer.readthedocs.io/>`_. +**build backend** + Libraries used to define how to build a wheel. Examples + include `setuptools <https://setuptools.pypa.io/>`__, + `flit <https://flit.readthedocs.io/>`_, and + `poetry <https://python-poetry.org/>`_. ^^^^^^^^^^^ -Description +Downloading ^^^^^^^^^^^ -The top of the PyPI downloads page contains a description of the -package. The first line is usually a short description, while there -may be a several line "Project Description" that follows. Choose whichever -is more useful. You can also get these descriptions on the command-line -using: - -.. code-block:: console +The first step in packaging a Python library is to figure out where +to download it from. The vast majority of Python packages are hosted +on `PyPI <https://pypi.org/>`_, which is +:ref:`preferred over GitHub <pypi-vs-github>` for downloading +packages. Search for the package name on PyPI to find the project +page. The project page is usually located at:: - $ python setup.py --description - $ python setup.py --long-description + https://pypi.org/project/<package-name> +On the project page, there is a "Download files" tab containing +download URLs. Whenever possible, we prefer to build Spack packages +from source. If PyPI only has wheels, check to see if the project is +hosted on GitHub and see if GitHub has source distributions. The +project page usually has a "Homepage" and/or "Source code" link for +this. If the project is closed-source, it may only have wheels +available. For example, ``py-azureml-sdk`` is closed-source and can +be downloaded from:: -^^^^^^^^ -Homepage -^^^^^^^^ + https://pypi.io/packages/py3/a/azureml_sdk/azureml_sdk-1.11.0-py3-none-any.whl -Package developers use ``setup.py`` to upload new versions to PyPI. -The ``setup`` method often passes metadata like ``homepage`` to PyPI. -This metadata is displayed on the left side of the download page. -Search for the text "Homepage" under "Project links" to find it. You -should use this page instead of the PyPI page if they differ. You can -also get the homepage on the command-line by running: +Once you've found a URL to download the package from, run: .. code-block:: console - $ python setup.py --url - - -^^^ -URL -^^^ - -If ``pypi`` is set as mentioned above, ``url`` and ``list_url`` will -be automatically set for you. If both ``.tar.gz`` and ``.zip`` versions -are available, ``.tar.gz`` is preferred. If some releases offer both -``.tar.gz`` and ``.zip`` versions, but some only offer ``.zip`` versions, -use ``.zip``. - -Some Python packages are closed-source and do not ship ``.tar.gz`` or ``.zip`` -files on either PyPI or GitHub. If this is the case, you can still download -and install a Python wheel. For example, ``py-azureml-sdk`` is closed source -and can be downloaded from:: - - https://pypi.io/packages/py3/a/azureml_sdk/azureml_sdk-1.11.0-py3-none-any.whl - + $ spack create <url> -You may see Python-specific or OS-specific URLs. Note that when you add a -``.whl`` URL, you should add ``expand=False`` to ensure that Spack doesn't -try to extract the wheel: -.. code-block:: python - - version('1.11.0', sha256='d8c9d24ea90457214d798b0d922489863dad518adde3638e08ef62de28fb183a', expand=False) +to create a new package template. .. _pypi-vs-github: @@ -226,11 +90,13 @@ try to extract the wheel: PyPI vs. GitHub """"""""""""""" -Many packages are hosted on PyPI, but are developed on GitHub or another -version control systems. The tarball can be downloaded from either -location, but PyPI is preferred for the following reasons: +Many packages are hosted on PyPI, but are developed on GitHub or +another version control system hosting service. The source code can +be downloaded from either location, but PyPI is preferred for the +following reasons: -#. PyPI contains the bare minimum number of files needed to install the package. +#. PyPI contains the bare minimum number of files needed to install + the package. You may notice that the tarball you download from PyPI does not have the same checksum as the tarball you download from GitHub. @@ -267,300 +133,217 @@ location, but PyPI is preferred for the following reasons: PyPI is nice because it makes it physically impossible to re-release the same version of a package with a different checksum. -Use the :ref:`pypi attribute <pypi>` to facilitate construction of PyPI package -references. - -^^^^^^^^^^^^^^^^^^^^^^^^^ -Build system dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are a few dependencies common to the ``PythonPackage`` build system. +The only reason to use GitHub instead of PyPI is if PyPI only has +wheels or if the PyPI sdist is missing a file needed to build the +package. If this is the case, please add a comment above the ``url`` +explaining this. -"""""" -Python -"""""" +^^^^ +PyPI +^^^^ -Obviously, every ``PythonPackage`` needs Python at build-time to run -``python setup.py build && python setup.py install``. Python is also -needed at run-time if you want to import the module. Due to backwards -incompatible changes between Python 2 and 3, it is very important to -specify which versions of Python are supported. If the documentation -mentions that Python 3 is required, this can be specified as: +Since PyPI is so commonly used to host Python libraries, the +``PythonPackage`` base class has a ``pypi`` attribute that can be +set. Once set, ``pypi`` will be used to define the ``homepage``, +``url``, and ``list_url``. For example, the following: .. code-block:: python - depends_on('python@3:', type=('build', 'run')) + homepage = 'https://pypi.org/project/setuptools/' + url = 'https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip' + list_url = 'https://pypi.org/simple/setuptools/' -If Python 2 is required, this would look like: +is equivalent to: .. code-block:: python - depends_on('python@:2', type=('build', 'run')) - - -If Python 2.7 is the only version that works, you can use: - -.. code-block:: python + pypi = 'setuptools/setuptools-49.2.0.zip' - depends_on('python@2.7:2.8', type=('build', 'run')) +If a package has a different homepage listed on PyPI, you can +override it by setting your own ``homepage``. -The documentation may not always specify supported Python versions. -Another place to check is in the ``setup.py`` or ``setup.cfg`` file. -Look for a line containing ``python_requires``. An example from -`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_ -looks like: +^^^^^^^^^^^ +Description +^^^^^^^^^^^ -.. code-block:: python +The top of the PyPI project page contains a short description of the +package. The "Project description" tab may also contain a longer +description of the package. Either of these can be used to populate +the package docstring. - python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*' +^^^^^^^^^^^^^ +Build backend +^^^^^^^^^^^^^ +Once you've determined the basic metadata for a package, the next +step is to determine the build backend. ``PythonPackage`` uses +`pip <https://pip.pypa.io/>`_ to install the package, but pip +requires a backend to actually build the package. -You may also find a version check at the top of the ``setup.py``: +To determine the build backend, look for a ``pyproject.toml`` file. +If there is no ``pyproject.toml`` file and only a ``setup.py`` or +``setup.cfg`` file, you can assume that the project uses +:ref:`setuptools`. If there is a ``pyproject.toml`` file, see if it +contains a ``[build-system]`` section. For example: -.. code-block:: python +.. code-block:: toml - if sys.version_info[:2] < (2, 7) or (3, 0) <= sys.version_info[:2] < (3, 4): - raise RuntimeError("Python version 2.7 or >= 3.4 required.") + [build-system] + requires = [ + "setuptools>=42", + "wheel", + ] + build-backend = "setuptools.build_meta" -This can be converted to Spack's spec notation like so: +This section does two things: the ``requires`` key lists build +dependencies of the project, and the ``build-backend`` key defines +the build backend. All of these build dependencies should be added as +dependencies to your package: .. code-block:: python - depends_on('python@2.7:2.8,3.4:', type=('build', 'run')) + depends_on('py-setuptools@42:', type='build') -If you are writing a recipe for a package that only distributes -wheels, look for a section in the ``METADATA`` file that looks like:: - - Requires-Python: >=3.5,<4 +Note that ``py-wheel`` is already listed as a build dependency in the +``PythonPackage`` base class, so you don't need to add it unless you +need to specify a specific version requirement or change the +dependency type. +See `PEP 517 <https://www.python.org/dev/peps/pep-0517/>`_ and +`PEP 518 <https://www.python.org/dev/peps/pep-0518/>`_ for more +information on the design of ``pyproject.toml``. -This would be translated to: +Depending on which build backend a project uses, there are various +places that run-time dependencies can be listed. -.. code-block:: python +""""""""" +distutils +""""""""" - extends('python') - depends_on('python@3.5:3', type=('build', 'run')) +Before the introduction of setuptools and other build backends, +Python packages had to rely on the built-in distutils library. +Distutils is missing many of the features that setuptools and other +build backends offer, and users are encouraged to use setuptools +instead. In fact, distutils was deprecated in Python 3.10 and will be +removed in Python 3.12. Because of this, pip actually replaces all +imports of distutils with setuptools. If a package uses distutils, +you should instead add a build dependency on setuptools. Check for a +``requirements.txt`` file that may list dependencies of the project. - -Many ``setup.py`` or ``setup.cfg`` files also contain information like:: - - Programming Language :: Python :: 2 - Programming Language :: Python :: 2.6 - Programming Language :: Python :: 2.7 - Programming Language :: Python :: 3 - Programming Language :: Python :: 3.3 - Programming Language :: Python :: 3.4 - Programming Language :: Python :: 3.5 - Programming Language :: Python :: 3.6 - - -This is a list of versions of Python that the developer likely tests. -However, you should not use this to restrict the versions of Python -the package uses unless one of the two former methods (``python_requires`` -or ``sys.version_info``) is used. There is no logic in setuptools -that prevents the package from building for Python versions not in -this list, and often new releases like Python 3.7 or 3.8 work just fine. +.. _setuptools: """""""""" setuptools """""""""" -Originally, the Python language had a single build system called -distutils, which is built into Python. Distutils provided a common -framework for package authors to describe their project and how it -should be built. However, distutils was not without limitations. -Most notably, there was no way to list a project's dependencies -with distutils. Along came setuptools, a non-builtin build system -designed to overcome the limitations of distutils. Both projects -use a similar API, making the transition easy while adding much -needed functionality. Today, setuptools is used in around 90% of -the Python packages in Spack. - -Since setuptools isn't built-in to Python, you need to add it as a -dependency. To determine whether or not a package uses setuptools, -search the file for an import statement like: +If the ``pyproject.toml`` lists ``setuptools.build_meta`` as a +``build-backend``, or if the package has a ``setup.py`` that imports +``setuptools``, or if the package has a ``setup.cfg`` file, then it +uses setuptools to build. Setuptools is a replacement for the +distutils library, and has almost the exact same API. Dependencies +can be listed in the ``setup.py`` or ``setup.cfg`` file. Look for the +following arguments: -.. code-block:: python - - import setuptools +* ``python_requires`` + This specifies the version of Python that is required. -or: +* ``setup_requires`` -.. code-block:: python + These packages are usually only needed at build-time, so you can + add them with ``type='build'``. - from setuptools import setup +* ``install_requires`` + These packages are required for building and installation. You can + add them with ``type=('build', 'run')``. -Some packages are designed to work with both setuptools and distutils, -so you may find something like: +* ``extras_require`` -.. code-block:: python + These packages are optional dependencies that enable additional + functionality. You should add a variant that optionally adds these + dependencies. This variant should be False by default. - try: - from setuptools import setup - except ImportError: - from distutils.core import setup +* ``tests_require`` + These are packages that are required to run the unit tests for the + package. These dependencies can be specified using the + ``type='test'`` dependency type. However, the PyPI tarballs rarely + contain unit tests, so there is usually no reason to add these. -This uses setuptools if available, and falls back to distutils if not. -In this case, you would still want to add a setuptools dependency, as -it offers us more control over the installation. +See https://setuptools.pypa.io/en/latest/userguide/dependency_management.html +for more information on how setuptools handles dependency management. +See `PEP 440 <https://www.python.org/dev/peps/pep-0440/#version-specifiers>`_ +for documentation on version specifiers in setuptools. -Unless specified otherwise, setuptools is usually a build-only dependency. -That is, it is needed to install the software, but is not needed at -run-time. This can be specified as: +"""" +flit +"""" -.. code-block:: python +There are actually two possible ``build-backend`` for flit, ``flit`` +and ``flit_core``. If you see these in the ``pyproject.toml``, add a +build dependency to your package. With flit, all dependencies are +listed directly in the ``pyproject.toml`` file. Older versions of +flit used to store this info in a ``flit.ini`` file, so check for +this too. - depends_on('py-setuptools', type='build') +Either of these files may contain keys like: +* ``requires-python`` -""" -pip -""" + This specifies the version of Python that is required -Packages distributed as Python wheels will require an extra dependency -on pip: +* ``dependencies`` or ``requires`` -.. code-block:: python + These packages are required for building and installation. You can + add them with ``type=('build', 'run')``. - depends_on('py-pip', type='build') +* ``project.optional-dependencies`` or ``requires-extra`` + This section includes keys with lists of optional dependencies + needed to enable those features. You should add a variant that + optionally adds these dependencies. This variant should be False + by default. -We will use pip to install the actual wheel. +See https://flit.readthedocs.io/en/latest/pyproject_toml.html for +more information. """""" -cython +poetry """""" -Compared to compiled languages, interpreted languages like Python can -be quite a bit slower. To work around this, some Python developers -rewrite computationally demanding sections of code in C, a process -referred to as "cythonizing". In order to build these package, you -need to add a build dependency on cython: - -.. code-block:: python - - depends_on('py-cython', type='build') - - -Look for references to "cython" in the ``setup.py`` to determine -whether or not this is necessary. Cython may be optional, but -even then you should list it as a required dependency. Spack is -designed to compile software, and is meant for HPC facilities -where speed is crucial. There is no reason why someone would not -want an optimized version of a library instead of the pure-Python -version. - -Note that some release tarballs come pre-cythonized, and cython is -not needed as a dependency. However, this is becoming less common -as Python continues to evolve and developers discover that cythonized -sources are no longer compatible with newer versions of Python and -need to be re-cythonized. - -^^^^^^^^^^^^^^^^^^^ -Python dependencies -^^^^^^^^^^^^^^^^^^^ - -When you install a package with ``pip``, it reads the ``setup.py`` -file in order to determine the dependencies of the package. -If the dependencies are not yet installed, ``pip`` downloads them -and installs them for you. This may sound convenient, but Spack -cannot rely on this behavior for two reasons: - -#. Spack needs to be able to install packages on air-gapped networks. - - If there is no internet connection, ``pip`` can't download the - package dependencies. By explicitly listing every dependency in - the ``package.py``, Spack knows what to download ahead of time. - -#. Duplicate installations of the same dependency may occur. - - Spack supports *activation* of Python extensions, which involves - symlinking the package installation prefix to the Python installation - prefix. If your package is missing a dependency, that dependency - will be installed to the installation directory of the same package. - If you try to activate the package + dependency, it may cause a - problem if that package has already been activated. - -For these reasons, you must always explicitly list all dependencies. -Although the documentation may list the package's dependencies, -often the developers assume people will use ``pip`` and won't have to -worry about it. Always check the ``setup.py`` to find the true -dependencies. - -If the package relies on ``distutils``, it may not explicitly list its -dependencies. Check for statements like: - -.. code-block:: python - - try: - import numpy - except ImportError: - raise ImportError("numpy must be installed prior to installation") - - -Obviously, this means that ``py-numpy`` is a dependency. - -If the package uses ``setuptools``, check for the following clues: - -* ``python_requires`` - - As mentioned above, this specifies which versions of Python are - required. - -* ``setup_requires`` - - These packages are usually only needed at build-time, so you can - add them with ``type='build'``. - -* ``install_requires`` - - These packages are required for building and installation. You can - add them with ``type=('build', 'run')``. - -* ``extra_requires`` - - These packages are optional dependencies that enable additional - functionality. You should add a variant that optionally adds these - dependencies. This variant should be False by default. - -* ``test_requires`` +Like flit, poetry also has two possible ``build-backend``, ``poetry`` +and ``poetry_core``. If you see these in the ``pyproject.toml``, add +a build dependency to your package. With poetry, all dependencies are +listed directly in the ``pyproject.toml`` file. Dependencies are +listed in a ``[tool.poetry.dependencies]`` section, and use a +`custom syntax <https://python-poetry.org/docs/dependency-specification/#version-constraints>`_ +for specifying the version requirements. Note that ``~=`` works +differently in poetry than in setuptools and flit for versions that +start with a zero. - These are packages that are required to run the unit tests for the - package. These dependencies can be specified using the - ``type='test'`` dependency type. However, the PyPI tarballs rarely - contain unit tests, so there is usually no reason to add these. +"""""" +wheels +"""""" -In the root directory of the package, you may notice a -``requirements.txt`` file. It may look like this file contains a list -of all of the package's dependencies. Don't be fooled. This file is -used by tools like Travis to install the pre-requisites for the -package... and a whole bunch of other things. It often contains -dependencies only needed for unit tests, like: +Some Python packages are closed-source and are distributed as Python +wheels. For example, ``py-azureml-sdk`` downloads a ``.whl`` file. This +file is simply a zip file, and can be extracted using: -* mock -* nose -* pytest +.. code-block:: console -It can also contain dependencies for building the documentation, like -sphinx. If you can't find any information about the package's -dependencies, you can take a look in ``requirements.txt``, but be sure -not to add test or documentation dependencies. + $ unzip *.whl -Newer PEPs have added alternative ways to specify a package's dependencies. -If you don't see any dependencies listed in the ``setup.py``, look for a -``setup.cfg`` or ``pyproject.toml``. These files can be used to store the -same ``install_requires`` information that ``setup.py`` used to use. -If you are write a recipe for a package that only distributes wheels, -check the ``METADATA`` file for lines like:: +The zip file will not contain a ``setup.py``, but it will contain a +``METADATA`` file which contains all the information you need to +write a ``package.py`` build recipe. Check for lines like:: + Requires-Python: >=3.5,<4 Requires-Dist: azureml-core (~=1.11.0) Requires-Dist: azureml-dataset-runtime[fuse] (~=1.11.0) Requires-Dist: azureml-train (~=1.11.0) @@ -572,62 +355,58 @@ check the ``METADATA`` file for lines like:: Requires-Dist: azureml-train-automl (~=1.11.0); extra == 'automl' -Lines that use ``Requires-Dist`` are similar to ``install_requires``. -Lines that use ``Provides-Extra`` are similar to ``extra_requires``, -and you can add a variant for those dependencies. The ``~=1.11.0`` -syntax is equivalent to ``1.11.0:1.11``. - -"""""""""" -setuptools -"""""""""" - -Setuptools is a bit of a special case. If a package requires setuptools -at run-time, how do they express this? They could add it to -``install_requires``, but setuptools is imported long before this and is -needed to read this line. And since you can't install the package -without setuptools, the developers assume that setuptools will already -be there, so they never mention when it is required. We don't want to -add run-time dependencies if they aren't needed, so you need to -determine whether or not setuptools is needed. Grep the installation -directory for any files containing a reference to ``setuptools`` or -``pkg_resources``. Both modules come from ``py-setuptools``. -``pkg_resources`` is particularly common in scripts found in -``prefix/bin``. +``Requires-Python`` is equivalent to ``python_requires`` and +``Requires-Dist`` is equivalent to ``install_requires``. +``Provides-Extra`` is used to name optional features (variants) and +a ``Requires-Dist`` with ``extra == 'foo'`` will list any +dependencies needed for that feature. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Passing arguments to setup.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The default build and install phases should be sufficient to install -most packages. However, you may want to pass additional flags to -either phase. +The default install phase should be sufficient to install most +packages. However, the installation instructions for a package may +suggest passing certain flags to the ``setup.py`` call. The +``PythonPackage`` class has two techniques for doing this. -You can view the available options for a particular phase with: +"""""""""""""" +Global options +"""""""""""""" -.. code-block:: console +These flags are added directly after ``setup.py`` when pip runs +``python setup.py install``. For example, the ``py-pyyaml`` package +has an optional dependency on ``libyaml`` that can be enabled like so: - $ python setup.py <phase> --help +.. code-block:: python + def global_options(self, spec, prefix): + options = [] + if '+libyaml' in spec: + options.append('--with-libyaml') + else: + options.append('--without-libyaml') + return options -Each phase provides a ``<phase_args>`` function that can be used to -pass arguments to that phase. For example, -`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_ -adds: -.. code-block:: python +""""""""""""""" +Install options +""""""""""""""" - def build_args(self, spec, prefix): - args = [] +These flags are added directly after ``install`` when pip runs +``python setup.py install``. For example, the ``py-pyyaml`` package +allows you to specify the directories to search for ``libyaml``: - # From NumPy 1.10.0 on it's possible to do a parallel build. - if self.version >= Version('1.10.0'): - # But Parallel build in Python 3.5+ is broken. See: - # https://github.com/spack/spack/issues/7927 - # https://github.com/scipy/scipy/issues/7112 - if spec['python'].version < Version('3.5'): - args = ['-j', str(make_jobs)] +.. code-block:: python - return args + def install_options(self, spec, prefix): + options = [] + if '+libyaml' in spec: + options.extend([ + spec['libyaml'].libs.search_flags, + spec['libyaml'].headers.include_flags, + ]) + return options ^^^^^^^ @@ -669,9 +448,9 @@ a "package" is a directory containing files like: whereas a "module" is a single Python file. -The ``PythonPackage`` base class automatically detects these module -names for you. If, for whatever reason, the module names detected -are wrong, you can provide the names yourself by overriding +The ``PythonPackage`` base class automatically detects these package +and module names for you. If, for whatever reason, the module names +detected are wrong, you can provide the names yourself by overriding ``import_modules`` like so: .. code-block:: python @@ -692,10 +471,8 @@ This can be expressed like so: @property def import_modules(self): modules = ['yaml'] - if '+libyaml' in self.spec: modules.append('yaml.cyaml') - return modules @@ -713,8 +490,8 @@ Unit tests """""""""" The package may have its own unit or regression tests. Spack can -run these tests during the installation by adding phase-appropriate -test methods. +run these tests during the installation by adding test methods after +installation. For example, ``py-numpy`` adds the following as a check to run after the ``install`` phase: @@ -740,34 +517,14 @@ when testing is enabled during the installation (i.e., ``spack install Setup file in a sub-directory ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In order to be compatible with package managers like ``pip``, the package -is required to place its ``setup.py`` in the root of the tarball. However, -not every Python package cares about ``pip`` or PyPI. If you are installing -a package that is not hosted on PyPI, you may find that it places its -``setup.py`` in a sub-directory. To handle this, add the directory containing -``setup.py`` to the package like so: - -.. code-block:: python - - build_directory = 'source' - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Alternate names for setup.py -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -As previously mentioned, packages need to call their setup script ``setup.py`` -in order to be compatible with package managers like ``pip``. However, some -packages like -`py-meep <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-meep/package.py>`_ and -`py-adios <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-adios/package.py>`_ -come with multiple setup scripts, one for a serial build and another for a -parallel build. You can override the default name to use like so: +Many C/C++ libraries provide optional Python bindings in a +subdirectory. To tell pip which directory to build from, you can +override the ``build_directory`` attribute. For example, if a package +provides Python bindings in a ``python`` directory, you can use: .. code-block:: python - def setup_file(self): - return 'setup-mpi.py' if '+mpi' in self.spec else 'setup.py' + build_directory = 'python' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -781,10 +538,14 @@ on Python are not necessarily ``PythonPackage``'s. Choosing a build system """"""""""""""""""""""" -First of all, you need to select a build system. ``spack create`` usually -does this for you, but if for whatever reason you need to do this manually, -choose ``PythonPackage`` if and only if the package contains a ``setup.py`` -file. +First of all, you need to select a build system. ``spack create`` +usually does this for you, but if for whatever reason you need to do +this manually, choose ``PythonPackage`` if and only if the package +contains one of the following files: + +* ``pyproject.toml`` +* ``setup.py`` +* ``setup.cfg`` """"""""""""""""""""""" Choosing a package name @@ -857,10 +618,9 @@ having to add that module to ``PYTHONPATH``. When deciding between ``extends`` and ``depends_on``, the best rule of thumb is to check the installation prefix. If Python libraries are -installed to ``prefix/lib/python2.7/site-packages`` (where 2.7 is the -MAJOR.MINOR version of Python you used to install the package), then -you should use ``extends``. If Python libraries are installed elsewhere -or the only files that get installed reside in ``prefix/bin``, then +installed to ``<prefix>/lib/pythonX.Y/site-packages``, then you +should use ``extends``. If Python libraries are installed elsewhere +or the only files that get installed reside in ``<prefix>/bin``, then don't use ``extends``, as symlinking the package wouldn't be useful. ^^^^^^^^^^^^^^^^^^^^^ @@ -893,4 +653,17 @@ External documentation ^^^^^^^^^^^^^^^^^^^^^^ For more information on Python packaging, see: -https://packaging.python.org/ + +* https://packaging.python.org/ + +For more information on build and installation frontend tools, see: + +* pip: https://pip.pypa.io/ +* build: https://pypa-build.readthedocs.io/ +* installer: https://installer.readthedocs.io/ + +For more information on build backend tools, see: + +* setuptools: https://setuptools.pypa.io/ +* flit: https://flit.readthedocs.io/ +* poetry: https://python-poetry.org/ |