From ae606bb600d3503e330cdfc0fcde39ecc4ff548d Mon Sep 17 00:00:00 2001 From: Greg Shuflin Date: Sat, 15 Nov 2025 00:35:55 -0800 Subject: [PATCH] Doc: Add guide for adding extension modules to CPython Adding a new C extension module to CPython's standard library requires updating multiple files across different build systems, which is not well documented. This adds a comprehensive guide that covers: - Prerequisites and design decisions (built-in vs shared) - Step-by-step process for all platforms - Unix/Linux: configure.ac and Setup file changes - Windows: MSBuild .vcxproj file creation - Testing, documentation, and cross-platform considerations - Troubleshooting common issues with solutions - Complete pre-submission checklist The guide is created in a new Doc/build_system/ directory for build-system-specific documentation that doesn't fit in the existing extending/ or c-api/ sections. This significantly reduces the barrier to adding new stdlib modules and provides a template for contributors to follow. --- Doc/build_system/adding_modules.rst | 464 ++++++++++++++++++++++++++++ Doc/build_system/index.rst | 16 + Doc/contents.rst | 1 + 3 files changed, 481 insertions(+) create mode 100644 Doc/build_system/adding_modules.rst create mode 100644 Doc/build_system/index.rst diff --git a/Doc/build_system/adding_modules.rst b/Doc/build_system/adding_modules.rst new file mode 100644 index 0000000000..225550f6e9 --- /dev/null +++ b/Doc/build_system/adding_modules.rst @@ -0,0 +1,464 @@ +.. _adding-stdlib-modules: + +************************************* +Adding Extension Modules to CPython +************************************* + +This guide explains how to add a new C extension module to the CPython standard +library. It covers the complete process from initial setup through testing and +cross-platform support. + +.. note:: + + This guide is for adding modules to CPython's standard library (stdlib). + For building third-party extension modules, see :ref:`extending-index`. + +Overview +======== + +Adding a new extension module to CPython requires updating multiple files across +the build system. The exact files depend on your target platforms, but typically +include: + +* Module source code (``.c`` and ``.h`` files in ``Modules/``) +* Build configuration for Unix-like systems (``configure.ac``, Setup files) +* Build configuration for Windows (MSBuild ``.vcxproj`` files) +* Python interface and tests +* Documentation + +This guide focuses on the build system aspects. For guidance on writing the C +code itself, see :ref:`extending-index` and :ref:`c-api-index`. + +Prerequisites +============= + +Before adding a new module, you should: + +* Be familiar with Python's C API and extension module development +* Have a working CPython development environment +* Understand your module's external dependencies (if any) +* Know whether the module should be built-in (static) or shared (dynamic) + +**Built-in vs. Shared Modules:** + +* **Built-in (static)**: Linked directly into the Python interpreter. Used for + essential modules like ``sys``, ``_io``, and ``_thread`` that are needed + during interpreter startup. +* **Shared (dynamic)**: Compiled as separate ``.so``/``.dylib``/``.pyd`` files. + Most stdlib extension modules are shared. This is the recommended choice for + new modules unless there's a specific reason to make them built-in. + +Step-by-Step Guide +=================== + +Step 1: Create Module Source Files +----------------------------------- + +Place your module source in the ``Modules/`` directory: + +``Modules/mymodule.c`` + The main module implementation. Must include: + + * A ``PyInit_mymodule`` function that creates and returns the module object. + * Module methods, types, and other objects. + * Module documentation string. + + Minimal example:: + + #define PY_SSIZE_T_CLEAN + #include "Python.h" + + PyDoc_STRVAR(module_doc, "My new module."); + + static PyMethodDef mymodule_methods[] = { + {NULL, NULL, 0, NULL} /* Sentinel */ + }; + + static struct PyModuleDef mymodule = { + PyModuleDef_HEAD_INIT, + "mymodule", /* m_name */ + module_doc, /* m_doc */ + -1, /* m_size */ + mymodule_methods, /* m_methods */ + NULL, /* m_slots */ + NULL, /* m_traverse */ + NULL, /* m_clear */ + NULL, /* m_free */ + }; + + PyMODINIT_FUNC + PyInit_mymodule(void) + { + return PyModule_Create(&mymodule); + } + +``Modules/mymodule.h`` (optional) + Header file for internal declarations if the module is complex. + +``Modules/clinic/mymodule.c.h`` (generated) + If using Argument Clinic for parsing function arguments, this file will be + generated. See :ref:`clinic-howto` for details. + +Step 2: Configure for Unix-like Systems +---------------------------------------- + +For Unix-like systems (Linux, macOS, BSD), you need to update autoconf +configuration and Setup files. + +2a. Update configure.ac +^^^^^^^^^^^^^^^^^^^^^^^ + +Add dependency detection logic to ``configure.ac`` if your module requires +external libraries. Find the section with other ``PY_STDLIB_MOD`` calls +(around line 7800) and add:: + + # Example for a module with no dependencies + PY_STDLIB_MOD([mymodule]) + + # Example for a module that requires libfoo + PY_CHECK_LIB([foo], [foo_init], [LIBFOO_LIBS], [libfoo]) + PY_STDLIB_MOD([mymodule], + [], [test "$LIBFOO_LIBS" != ""], + [], [$LIBFOO_LIBS]) + +The ``PY_STDLIB_MOD`` macro parameters are: + +1. Module name (as used in ``import mymodule``) +2. Required dependencies (empty for none) +3. Condition for building (shell test expression, leave empty for "always") +4. Additional CFLAGS +5. Additional LDFLAGS/LIBS + +See existing modules in ``configure.ac`` for more examples. + +After modifying ``configure.ac``, regenerate the configure script:: + + ./Tools/build/regen-configure.sh + +Or locally:: + + autoreconf -ivf -Werror + +.. note:: + + The ``regen-configure.sh`` script uses a Docker container to ensure + reproducible output with specific autoconf versions. The generated + ``configure`` script should be committed along with your changes to + ``configure.ac``. + +2b. Update Modules/Setup.stdlib.in +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Add your module to ``Modules/Setup.stdlib.in`` in the appropriate section. +For most new modules without dependencies:: + + @MODULE_MYMODULE_TRUE@mymodule mymodule.c + +For modules with dependencies:: + + @MODULE_MYMODULE_TRUE@mymodule mymodule.c $(LIBFOO_CFLAGS) $(LIBFOO_LIBS) + +The ``@MODULE_MYMODULE_TRUE@`` marker is substituted by the configure script: + +* If dependencies are met, it becomes empty (module is included) +* If dependencies are not met, it becomes ``#`` (module is commented out) + +**Placement matters:** + +* Modules before ``*static*`` are built-in (linked into the interpreter) +* Modules after ``*shared*`` are shared libraries (typical for new modules) + +For most new modules, add them in the shared section after the ``*shared*`` +marker. + +Step 3: Configure for Windows +------------------------------ + +Windows uses MSBuild and Visual Studio project files instead of autotools. + +3a. Create a Visual Studio Project File +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create ``PCbuild/mymodule.vcxproj`` based on an existing simple module's project +file. For example, copy ``PCbuild/_csv.vcxproj`` and modify: + +1. Update ```` to your module name +2. Update source file paths in ```` sections +3. Update dependencies in ```` if needed +4. Update preprocessor definitions if needed + +Example structure:: + + + + + + + + {GENERATE-NEW-GUID-HERE} + mymodule + + + + + + + + + + +Generate a new GUID for your project:: + + python -c "import uuid; print('{' + str(uuid.uuid4()).upper() + '}')" + +3b. Add to PCbuild/pcbuild.proj +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Add a reference to your new project in ``PCbuild/pcbuild.proj``:: + + + +This ensures your module is built as part of the standard Windows build. + +3c. Update Windows Dependency Handling (if needed) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If your module depends on external libraries: + +1. Check if the library is already in ``PCbuild/get_externals.bat`` +2. If not, add it to the external dependencies script +3. Create appropriate property sheets (``.props`` files) for compiler/linker flags +4. Reference the property sheets in your ``.vcxproj`` file + +Step 4: Add Python-level Interface (Optional) +---------------------------------------------- + +Many C extension modules have a Python wrapper in ``Lib/`` that provides a +more Pythonic interface. For example: + +* ``Modules/_json.c`` has ``Lib/json/`` +* ``Modules/_sqlite/`` has ``Lib/sqlite3/`` +* ``Modules/_csv.c`` has ``Lib/csv.py`` + +If your module provides a low-level C API, consider adding a Python wrapper +in ``Lib/mymodule.py`` or ``Lib/mymodule/``. + +Step 5: Add Tests +----------------- + +Create comprehensive tests in ``Lib/test/test_mymodule.py``:: + + import unittest + import mymodule + + class MyModuleTests(unittest.TestCase): + def test_basic_functionality(self): + # Test your module's functionality + pass + + def test_error_handling(self): + # Test error cases + pass + + if __name__ == '__main__': + unittest.main() + +Ensure tests cover: + +* Basic functionality +* Error cases and exceptions +* Edge cases +* Platform-specific behavior (if any) +* Memory management (no leaks) + +Run the test suite to verify:: + + ./python -m test test_mymodule + +Step 6: Update Documentation +----------------------------- + +Add or update documentation for your module: + +``Doc/library/mymodule.rst`` + Full module documentation following the CPython documentation style. + See existing module documentation for templates. + +``Doc/library/index.rst`` + Add your module to the appropriate section of the library index. + +``Doc/whatsnew/3.XX.rst`` + Document your new module in the "What's New" document for the next release. + +Step 7: Build and Test +----------------------- + +Build CPython with your new module: + +On Unix-like systems:: + + ./configure + make -j + ./python -c "import mymodule; print(mymodule.__doc__)" + +On Windows:: + + PCbuild\build.bat + PCbuild\amd64\python.exe -c "import mymodule; print(mymodule.__doc__)" + +Run the full test suite:: + + make test + +Or on Windows:: + + PCbuild\build.bat -t Test + +Step 8: Handle Cross-platform Issues +------------------------------------- + +Testing on Multiple Platforms +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Your module should work on all supported platforms: + +* **Linux**: Ubuntu, Debian, Fedora, etc. (different architectures: x86_64, ARM, etc.) +* **macOS**: x86_64 and ARM64 (Apple Silicon) +* **Windows**: x86_64 and ARM64 +* **WASM**: Emscripten and WASI (if applicable) +* **Mobile**: iOS and Android (if applicable) + +Platform-specific Code +^^^^^^^^^^^^^^^^^^^^^^ + +Use ``#ifdef`` guards for platform-specific code:: + + #ifdef MS_WINDOWS + /* Windows-specific implementation */ + #elif defined(__APPLE__) + /* macOS-specific implementation */ + #else + /* Linux/BSD/other Unix */ + #endif + +Or use ``Py_BUILD_CORE`` macros and features from ``pyconfig.h``. + +Handling Missing Dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If your module's dependencies aren't available on all platforms: + +1. Make it an **optional module** (not required for basic Python operation) +2. Use ``PY_STDLIB_MOD`` conditions in ``configure.ac`` to disable on platforms + without dependencies +3. On Windows, use ``Condition`` attributes in ``.vcxproj`` to control building +4. Document platform availability in module documentation + +Checklist +========= + +Before submitting your module for review, verify: + +Build System Integration: + ☐ Source files in ``Modules/`` + + ☐ ``configure.ac`` updated with ``PY_STDLIB_MOD`` entry + + ☐ ``Modules/Setup.stdlib.in`` updated + + ☐ ``configure`` regenerated (``./Tools/build/regen-configure.sh``) + + ☐ Windows ``.vcxproj`` file created in ``PCbuild/`` + + ☐ ``PCbuild/pcbuild.proj`` updated + +Testing: + ☐ Test file created in ``Lib/test/`` + + ☐ Tests pass on Linux + + ☐ Tests pass on macOS (if possible) + + ☐ Tests pass on Windows (if possible) + + ☐ No memory leaks (run with ``valgrind`` or ASAN) + +Documentation: + ☐ Module documentation in ``Doc/library/`` + + ☐ Library index updated + + ☐ What's New document updated + + ☐ Docstrings complete + +Code Quality: + ☐ Follows :pep:`7` (C style guide) + + ☐ No compiler warnings + + ☐ Proper error handling (all error paths covered) + + ☐ Reference counting correct (no leaks or crashes) + +Common Issues and Solutions +============================ + +Module doesn't build on Linux +------------------------------ + +**Problem**: After adding to ``Setup.stdlib.in``, module doesn't appear in build. + +**Solution**: + +1. Check that ``PY_STDLIB_MOD`` condition in ``configure.ac`` is satisfied +2. Run ``./configure`` again to regenerate Makefile +3. Check ``Modules/Setup.local`` doesn't override your module +4. Examine configure output for your module's status +5. Run ``make clean && make`` to force rebuild + +Module builds but import fails +------------------------------- + +**Problem**: ``ImportError: dynamic module does not define module export function`` + +**Solution**: + +Ensure your module initialization function is named exactly ``PyInit_modulename`` +where ``modulename`` matches the first argument to ``PyModule_Create`` and the +filename. + +Windows build fails +------------------- + +**Problem**: MSBuild can't find your module project or dependencies. + +**Solution**: + +1. Verify ``.vcxproj`` file XML is well-formed +2. Ensure GUID is unique (not copied from another project) +3. Check that ``PCbuild/pcbuild.proj`` includes your project +4. Verify paths to source files are correct (use ``..`` to go up from ``PCbuild/``) +5. Check that external dependencies are in ``PCbuild/`` after running ``get_externals.bat`` + +Module initialization crashes +------------------------------ + +**Problem**: Python crashes during module import. + +**Solution**: + +1. Use a debugger (``gdb`` on Linux/macOS, Visual Studio debugger on Windows) +2. Check reference counting (use ``--with-pydebug`` build) +3. Verify all ``PyObject *`` pointers are checked for ``NULL`` +4. Ensure ``PyMODINIT_FUNC`` is used for the init function +5. Build with ``--with-address-sanitizer`` to detect memory errors + +See Also +======== + +* :ref:`extending-index` - Extending Python with C or C++ +* :ref:`c-api-index` - Python/C API Reference +* :pep:`7` - Style Guide for C Code +* :ref:`clinic-howto` - Using Argument Clinic +* `CPython Developer's Guide `_ - Full development workflow diff --git a/Doc/build_system/index.rst b/Doc/build_system/index.rst new file mode 100644 index 0000000000..00c2757355 --- /dev/null +++ b/Doc/build_system/index.rst @@ -0,0 +1,16 @@ +.. _build-system-guides: + +#################### +Build System Guides +#################### + +These guides explain CPython's build system internals and how to work with it. + +For basic build instructions, see :ref:`setup-building` in the `CPython Developer's Guide `_. + +For configure options and build system overview, see :ref:`configure-options`. + +.. toctree:: + :maxdepth: 2 + + adding_modules.rst diff --git a/Doc/contents.rst b/Doc/contents.rst index b57f4b09a5..46b1e247da 100644 --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -10,6 +10,7 @@ reference/index.rst library/index.rst extending/index.rst + build_system/index.rst c-api/index.rst installing/index.rst howto/index.rst