Doc: Add guide for adding extension modules to CPython

Adding a new C extension module to CPython's standard library
requires updating multiple files across different build systems,
which is not well documented. This adds a comprehensive guide that
covers:

- Prerequisites and design decisions (built-in vs shared)
- Step-by-step process for all platforms
- Unix/Linux: configure.ac and Setup file changes
- Windows: MSBuild .vcxproj file creation
- Testing, documentation, and cross-platform considerations
- Troubleshooting common issues with solutions
- Complete pre-submission checklist

The guide is created in a new Doc/build_system/ directory for
build-system-specific documentation that doesn't fit in the
existing extending/ or c-api/ sections.

This significantly reduces the barrier to adding new stdlib modules
and provides a template for contributors to follow.

Doc/build_system/adding_modules.rst

@@ -0,0 +1,464 @@
+.. _adding-stdlib-modules:
+
+*************************************
+Adding Extension Modules to CPython
+*************************************
+
+This guide explains how to add a new C extension module to the CPython standard
+library. It covers the complete process from initial setup through testing and
+cross-platform support.
+
+.. note::
+
+   This guide is for adding modules to CPython's standard library (stdlib).
+   For building third-party extension modules, see :ref:`extending-index`.
+
+Overview
+========
+
+Adding a new extension module to CPython requires updating multiple files across
+the build system. The exact files depend on your target platforms, but typically
+include:
+
+* Module source code (``.c`` and ``.h`` files in ``Modules/``)
+* Build configuration for Unix-like systems (``configure.ac``, Setup files)
+* Build configuration for Windows (MSBuild ``.vcxproj`` files)
+* Python interface and tests
+* Documentation
+
+This guide focuses on the build system aspects. For guidance on writing the C
+code itself, see :ref:`extending-index` and :ref:`c-api-index`.
+
+Prerequisites
+=============
+
+Before adding a new module, you should:
+
+* Be familiar with Python's C API and extension module development
+* Have a working CPython development environment
+* Understand your module's external dependencies (if any)
+* Know whether the module should be built-in (static) or shared (dynamic)
+
+**Built-in vs. Shared Modules:**
+
+* **Built-in (static)**: Linked directly into the Python interpreter. Used for
+  essential modules like ``sys``, ``_io``, and ``_thread`` that are needed
+  during interpreter startup.
+* **Shared (dynamic)**: Compiled as separate ``.so``/``.dylib``/``.pyd`` files.
+  Most stdlib extension modules are shared. This is the recommended choice for
+  new modules unless there's a specific reason to make them built-in.
+
+Step-by-Step Guide
+===================
+
+Step 1: Create Module Source Files
+-----------------------------------
+
+Place your module source in the ``Modules/`` directory:
+
+``Modules/mymodule.c``
+  The main module implementation. Must include:
+
+  * A ``PyInit_mymodule`` function that creates and returns the module object.
+  * Module methods, types, and other objects.
+  * Module documentation string.
+
+  Minimal example::
+
+    #define PY_SSIZE_T_CLEAN
+    #include "Python.h"
+
+    PyDoc_STRVAR(module_doc, "My new module.");
+
+    static PyMethodDef mymodule_methods[] = {
+        {NULL, NULL, 0, NULL}  /* Sentinel */
+    };
+
+    static struct PyModuleDef mymodule = {
+        PyModuleDef_HEAD_INIT,
+        "mymodule",        /* m_name */
+        module_doc,        /* m_doc */
+        -1,                /* m_size */
+        mymodule_methods,  /* m_methods */
+        NULL,              /* m_slots */
+        NULL,              /* m_traverse */
+        NULL,              /* m_clear */
+        NULL,              /* m_free */
+    };
+
+    PyMODINIT_FUNC
+    PyInit_mymodule(void)
+    {
+        return PyModule_Create(&mymodule);
+    }
+
+``Modules/mymodule.h`` (optional)
+  Header file for internal declarations if the module is complex.
+
+``Modules/clinic/mymodule.c.h`` (generated)
+  If using Argument Clinic for parsing function arguments, this file will be
+  generated. See :ref:`clinic-howto` for details.
+
+Step 2: Configure for Unix-like Systems
+----------------------------------------
+
+For Unix-like systems (Linux, macOS, BSD), you need to update autoconf
+configuration and Setup files.
+
+2a. Update configure.ac
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Add dependency detection logic to ``configure.ac`` if your module requires
+external libraries. Find the section with other ``PY_STDLIB_MOD`` calls
+(around line 7800) and add::
+
+    # Example for a module with no dependencies
+    PY_STDLIB_MOD([mymodule])
+
+    # Example for a module that requires libfoo
+    PY_CHECK_LIB([foo], [foo_init], [LIBFOO_LIBS], [libfoo])
+    PY_STDLIB_MOD([mymodule],
+                  [], [test "$LIBFOO_LIBS" != ""],
+                  [], [$LIBFOO_LIBS])
+
+The ``PY_STDLIB_MOD`` macro parameters are:
+
+1. Module name (as used in ``import mymodule``)
+2. Required dependencies (empty for none)
+3. Condition for building (shell test expression, leave empty for "always")
+4. Additional CFLAGS
+5. Additional LDFLAGS/LIBS
+
+See existing modules in ``configure.ac`` for more examples.
+
+After modifying ``configure.ac``, regenerate the configure script::
+
+    ./Tools/build/regen-configure.sh
+
+Or locally::
+
+    autoreconf -ivf -Werror
+
+.. note::
+
+   The ``regen-configure.sh`` script uses a Docker container to ensure
+   reproducible output with specific autoconf versions. The generated
+   ``configure`` script should be committed along with your changes to
+   ``configure.ac``.
+
+2b. Update Modules/Setup.stdlib.in
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Add your module to ``Modules/Setup.stdlib.in`` in the appropriate section.
+For most new modules without dependencies::
+
+    @MODULE_MYMODULE_TRUE@mymodule mymodule.c
+
+For modules with dependencies::
+
+    @MODULE_MYMODULE_TRUE@mymodule mymodule.c $(LIBFOO_CFLAGS) $(LIBFOO_LIBS)
+
+The ``@MODULE_MYMODULE_TRUE@`` marker is substituted by the configure script:
+
+* If dependencies are met, it becomes empty (module is included)
+* If dependencies are not met, it becomes ``#`` (module is commented out)
+
+**Placement matters:**
+
+* Modules before ``*static*`` are built-in (linked into the interpreter)
+* Modules after ``*shared*`` are shared libraries (typical for new modules)
+
+For most new modules, add them in the shared section after the ``*shared*``
+marker.
+
+Step 3: Configure for Windows
+------------------------------
+
+Windows uses MSBuild and Visual Studio project files instead of autotools.
+
+3a. Create a Visual Studio Project File
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Create ``PCbuild/mymodule.vcxproj`` based on an existing simple module's project
+file. For example, copy ``PCbuild/_csv.vcxproj`` and modify:
+
+1. Update ``<ProjectName>`` to your module name
+2. Update source file paths in ``<ClCompile Include="...">`` sections
+3. Update dependencies in ``<ProjectReference>`` if needed
+4. Update preprocessor definitions if needed
+
+Example structure::
+
+    <?xml version="1.0" encoding="utf-8"?>
+    <Project DefaultTargets="Build" ToolsVersion="4.0">
+      <ItemGroup Label="ProjectConfigurations">
+        <!-- Configuration entries (don't modify) -->
+      </ItemGroup>
+      <PropertyGroup Label="Globals">
+        <ProjectGuid>{GENERATE-NEW-GUID-HERE}</ProjectGuid>
+        <RootNamespace>mymodule</RootNamespace>
+      </PropertyGroup>
+      <Import Project="python.props" />
+      <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+      <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+      <ItemGroup>
+        <ClCompile Include="..\Modules\mymodule.c" />
+      </ItemGroup>
+      <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+    </Project>
+
+Generate a new GUID for your project::
+
+    python -c "import uuid; print('{' + str(uuid.uuid4()).upper() + '}')"
+
+3b. Add to PCbuild/pcbuild.proj
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Add a reference to your new project in ``PCbuild/pcbuild.proj``::
+
+    <Projects Include="mymodule.vcxproj" />
+
+This ensures your module is built as part of the standard Windows build.
+
+3c. Update Windows Dependency Handling (if needed)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If your module depends on external libraries:
+
+1. Check if the library is already in ``PCbuild/get_externals.bat``
+2. If not, add it to the external dependencies script
+3. Create appropriate property sheets (``.props`` files) for compiler/linker flags
+4. Reference the property sheets in your ``.vcxproj`` file
+
+Step 4: Add Python-level Interface (Optional)
+----------------------------------------------
+
+Many C extension modules have a Python wrapper in ``Lib/`` that provides a
+more Pythonic interface. For example:
+
+* ``Modules/_json.c`` has ``Lib/json/``
+* ``Modules/_sqlite/`` has ``Lib/sqlite3/``
+* ``Modules/_csv.c`` has ``Lib/csv.py``
+
+If your module provides a low-level C API, consider adding a Python wrapper
+in ``Lib/mymodule.py`` or ``Lib/mymodule/``.
+
+Step 5: Add Tests
+-----------------
+
+Create comprehensive tests in ``Lib/test/test_mymodule.py``::
+
+    import unittest
+    import mymodule
+
+    class MyModuleTests(unittest.TestCase):
+        def test_basic_functionality(self):
+            # Test your module's functionality
+            pass
+
+        def test_error_handling(self):
+            # Test error cases
+            pass
+
+    if __name__ == '__main__':
+        unittest.main()
+
+Ensure tests cover:
+
+* Basic functionality
+* Error cases and exceptions
+* Edge cases
+* Platform-specific behavior (if any)
+* Memory management (no leaks)
+
+Run the test suite to verify::
+
+    ./python -m test test_mymodule
+
+Step 6: Update Documentation
+-----------------------------
+
+Add or update documentation for your module:
+
+``Doc/library/mymodule.rst``
+  Full module documentation following the CPython documentation style.
+  See existing module documentation for templates.
+
+``Doc/library/index.rst``
+  Add your module to the appropriate section of the library index.
+
+``Doc/whatsnew/3.XX.rst``
+  Document your new module in the "What's New" document for the next release.
+
+Step 7: Build and Test
+-----------------------
+
+Build CPython with your new module:
+
+On Unix-like systems::
+
+    ./configure
+    make -j
+    ./python -c "import mymodule; print(mymodule.__doc__)"
+
+On Windows::
+
+    PCbuild\build.bat
+    PCbuild\amd64\python.exe -c "import mymodule; print(mymodule.__doc__)"
+
+Run the full test suite::
+
+    make test
+
+Or on Windows::
+
+    PCbuild\build.bat -t Test
+
+Step 8: Handle Cross-platform Issues
+-------------------------------------
+
+Testing on Multiple Platforms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Your module should work on all supported platforms:
+
+* **Linux**: Ubuntu, Debian, Fedora, etc. (different architectures: x86_64, ARM, etc.)
+* **macOS**: x86_64 and ARM64 (Apple Silicon)
+* **Windows**: x86_64 and ARM64
+* **WASM**: Emscripten and WASI (if applicable)
+* **Mobile**: iOS and Android (if applicable)
+
+Platform-specific Code
+^^^^^^^^^^^^^^^^^^^^^^
+
+Use ``#ifdef`` guards for platform-specific code::
+
+    #ifdef MS_WINDOWS
+        /* Windows-specific implementation */
+    #elif defined(__APPLE__)
+        /* macOS-specific implementation */
+    #else
+        /* Linux/BSD/other Unix */
+    #endif
+
+Or use ``Py_BUILD_CORE`` macros and features from ``pyconfig.h``.
+
+Handling Missing Dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If your module's dependencies aren't available on all platforms:
+
+1. Make it an **optional module** (not required for basic Python operation)
+2. Use ``PY_STDLIB_MOD`` conditions in ``configure.ac`` to disable on platforms
+   without dependencies
+3. On Windows, use ``Condition`` attributes in ``.vcxproj`` to control building
+4. Document platform availability in module documentation
+
+Checklist
+=========
+
+Before submitting your module for review, verify:
+
+Build System Integration:
+  ☐ Source files in ``Modules/``
+
+  ☐ ``configure.ac`` updated with ``PY_STDLIB_MOD`` entry
+
+  ☐ ``Modules/Setup.stdlib.in`` updated
+
+  ☐ ``configure`` regenerated (``./Tools/build/regen-configure.sh``)
+
+  ☐ Windows ``.vcxproj`` file created in ``PCbuild/``
+
+  ☐ ``PCbuild/pcbuild.proj`` updated
+
+Testing:
+  ☐ Test file created in ``Lib/test/``
+
+  ☐ Tests pass on Linux
+
+  ☐ Tests pass on macOS (if possible)
+
+  ☐ Tests pass on Windows (if possible)
+
+  ☐ No memory leaks (run with ``valgrind`` or ASAN)
+
+Documentation:
+  ☐ Module documentation in ``Doc/library/``
+
+  ☐ Library index updated
+
+  ☐ What's New document updated
+
+  ☐ Docstrings complete
+
+Code Quality:
+  ☐ Follows :pep:`7` (C style guide)
+
+  ☐ No compiler warnings
+
+  ☐ Proper error handling (all error paths covered)
+
+  ☐ Reference counting correct (no leaks or crashes)
+
+Common Issues and Solutions
+============================
+
+Module doesn't build on Linux
+------------------------------
+
+**Problem**: After adding to ``Setup.stdlib.in``, module doesn't appear in build.
+
+**Solution**:
+
+1. Check that ``PY_STDLIB_MOD`` condition in ``configure.ac`` is satisfied
+2. Run ``./configure`` again to regenerate Makefile
+3. Check ``Modules/Setup.local`` doesn't override your module
+4. Examine configure output for your module's status
+5. Run ``make clean && make`` to force rebuild
+
+Module builds but import fails
+-------------------------------
+
+**Problem**: ``ImportError: dynamic module does not define module export function``
+
+**Solution**:
+
+Ensure your module initialization function is named exactly ``PyInit_modulename``
+where ``modulename`` matches the first argument to ``PyModule_Create`` and the
+filename.
+
+Windows build fails
+-------------------
+
+**Problem**: MSBuild can't find your module project or dependencies.
+
+**Solution**:
+
+1. Verify ``.vcxproj`` file XML is well-formed
+2. Ensure GUID is unique (not copied from another project)
+3. Check that ``PCbuild/pcbuild.proj`` includes your project
+4. Verify paths to source files are correct (use ``..`` to go up from ``PCbuild/``)
+5. Check that external dependencies are in ``PCbuild/`` after running ``get_externals.bat``
+
+Module initialization crashes
+------------------------------
+
+**Problem**: Python crashes during module import.
+
+**Solution**:
+
+1. Use a debugger (``gdb`` on Linux/macOS, Visual Studio debugger on Windows)
+2. Check reference counting (use ``--with-pydebug`` build)
+3. Verify all ``PyObject *`` pointers are checked for ``NULL``
+4. Ensure ``PyMODINIT_FUNC`` is used for the init function
+5. Build with ``--with-address-sanitizer`` to detect memory errors
+
+See Also
+========
+
+* :ref:`extending-index` - Extending Python with C or C++
+* :ref:`c-api-index` - Python/C API Reference
+* :pep:`7` - Style Guide for C Code
+* :ref:`clinic-howto` - Using Argument Clinic
+* `CPython Developer's Guide <https://devguide.python.org/>`_ - Full development workflow

Doc/build_system/index.rst

@@ -0,0 +1,16 @@
+.. _build-system-guides:
+
+####################
+Build System Guides
+####################
+
+These guides explain CPython's build system internals and how to work with it.
+
+For basic build instructions, see :ref:`setup-building` in the `CPython Developer's Guide <https://devguide.python.org/>`_.
+
+For configure options and build system overview, see :ref:`configure-options`.
+
+.. toctree::
+   :maxdepth: 2
+
+   adding_modules.rst

Doc/contents.rst

@@ -10,6 +10,7 @@
    reference/index.rst
    library/index.rst
    extending/index.rst
+   build_system/index.rst
    c-api/index.rst
    installing/index.rst
    howto/index.rst