Build: Add build system improvements tracking document

This document tracks the incremental improvements being made to
CPython's build system. It includes:

- Summary of completed work (3 documentation commits)
- Submission strategy and review checklists
- Analysis of build system complexity
- Remaining improvement opportunities
- Testing procedures

This is a working document to coordinate build system simplification
efforts and is not intended for inclusion in CPython upstream
(should be in .gitignore or kept in a fork).

BUILD_SYSTEM_IMPROVEMENTS.md

@@ -0,0 +1,266 @@
+# CPython Build System Simplification - Progress Report
+
+## Overview
+
+This document tracks incremental improvements to CPython's build system aimed at making it simpler and easier to understand. The approach focuses on small, reviewable commits that are likely to be accepted by the CPython core team.
+
+## Completed Work (Ready for Submission)
+
+### Commit 1: Enhanced Build System Architecture Documentation ✅
+
+**File Modified:** `Doc/using/configure.rst`
+
+**Changes:**
+- Added comprehensive "Overview" section explaining build system phases
+- Documented "Build System Components" for different platforms
+- Added detailed "Bootstrap Process" explanation (4-stage process)
+- Created "Module Configuration System" documentation explaining the three-layer system
+- Enhanced "Main build steps" with detailed phase-by-phase breakdown
+- Added visual build flow diagram
+
+**Impact:** Answers the question "how does CPython build?" for new contributors
+
+**Review Checklist:**
+- [ ] Documentation builds without warnings
+- [ ] Content is technically accurate
+- [ ] Cross-references to other docs work
+- [ ] Formatting follows CPython doc style
+
+### Commit 2: Module Addition Guide ✅
+
+**Files Created:**
+- `Doc/build_system/adding_modules.rst` - Comprehensive guide for adding new extension modules
+- `Doc/build_system/index.rst` - Build system guides index
+
+**File Modified:**
+- `Doc/contents.rst` - Added build_system section to main documentation
+
+**Changes:**
+The guide includes:
+- Step-by-step process for adding modules to CPython stdlib
+- Unix/Linux configuration (configure.ac, Setup files)
+- Windows configuration (MSBuild .vcxproj files)
+- Testing, documentation, and cross-platform considerations
+- Troubleshooting section for common issues
+- Complete checklist for module authors
+
+**Impact:** Reduces friction for adding new modules, provides template to follow
+
+**Review Checklist:**
+- [ ] Documentation builds without warnings
+- [ ] Examples are accurate
+- [ ] Guide tested against actual module addition
+- [ ] Links to other docs work
+
+### Commit 3: configure.ac Section Documentation ✅
+
+**File Modified:** `configure.ac`
+
+**Changes:**
+Added 5 major section headers to organize the 8,200+ line file:
+
+1. **Section 1 (Line 137):** Platform and Build Configuration Detection
+   - Platform triplet detection
+   - Cross-compilation setup
+   - Python-for-build configuration
+
+2. **Section 2 (Line 1070):** Compiler Detection and Configuration
+   - C/C++ compiler detection
+   - Compiler characteristics
+   - Compilation environment setup
+
+3. **Section 3 (Line 1731):** Python Build Feature Flags
+   - --disable-gil (free-threading)
+   - --with-pydebug
+   - --enable-optimizations
+   - --enable-experimental-jit
+   - Other build variants
+
+4. **Section 4 (Line 7840):** Standard Library Extension Module Configuration
+   - PY_STDLIB_MOD macro definition
+   - Module state tracking
+   - Dependency detection for all stdlib modules
+
+5. **Section 5 (Line 4054):** External Library Dependencies
+   - System libraries (sockets, threads)
+   - Third-party libraries (OpenSSL, zlib, libffi, SQLite, etc.)
+
+**Impact:** Makes the massive configure.ac file navigable, helps developers find relevant sections
+
+**Review Checklist:**
+- [ ] Comments don't affect generated configure script
+- [ ] Section boundaries are logical
+- [ ] No typos in section headers
+- [ ] Run `./Tools/build/regen-configure.sh` to verify no breakage
+
+## Submission Strategy
+
+### Order of Submission
+
+Submit commits in order (1, 2, 3) as they build on each other:
+
+1. **Start with Commit 1** - Safest, pure documentation addition
+2. **Follow with Commit 2** - New files, no modification of existing code
+3. **Submit Commit 3 last** - Modifies configure.ac (comments only, but needs verification)
+
+### Before Submitting Each Commit
+
+1. **Create a GitHub Issue** at https://github.com/python/cpython/issues
+   - Title: "[Documentation] Add build system architecture documentation" (or similar)
+   - Describe the problem (missing documentation) and proposed solution
+   - Reference this issue in your PR
+
+2. **Test Documentation Build**
+   ```bash
+   cd Doc
+   make html  # Verify no Sphinx warnings
+   ```
+
+3. **Review Your Changes**
+   ```bash
+   git diff  # Check what you're submitting
+   git log -p  # Review your commit message
+   ```
+
+4. **Create Pull Request**
+   - Follow CPython PR guidelines
+   - Reference the issue number
+   - Explain motivation clearly
+   - Be responsive to reviewer feedback
+
+### Example Commit Messages
+
+```
+Doc: Add comprehensive build system architecture documentation
+
+The CPython build system is complex, but lacks comprehensive
+documentation explaining how the pieces fit together. This adds:
+
+- Overview of build phases (configure, bootstrap, build, install)
+- Platform-specific build system details
+- Explanation of the 4-stage bootstrap process
+- Module configuration system documentation
+
+This helps new contributors understand the build system and reduces
+questions about "how does CPython build?"
+
+Related to: gh-XXXXX
+```
+
+## Remaining Work (For Future Consideration)
+
+These commits require more extensive testing and are better suited for contributors with more CPython experience:
+
+### Commit 4: Extract Platform Detection Macros
+- **Risk:** Medium
+- **Effort:** ~2 weeks
+- **Benefit:** Reduces configure.ac by ~400 lines
+
+### Commit 5: Standardize Dependency Detection Pattern
+- **Risk:** Low-Medium
+- **Effort:** ~1 week
+- **Benefit:** Template for future dependencies
+
+### Commit 6: Module Configuration Data File (Proof of Concept)
+- **Risk:** Medium
+- **Effort:** ~2-3 weeks
+- **Benefit:** Demonstrates path toward reducing duplication
+
+### Commit 7: Add Verbose Mode to makesetup
+- **Risk:** Low
+- **Effort:** ~3 days
+- **Benefit:** Better debugging for module build issues
+
+### Commit 8: Python makesetup Prototype
+- **Risk:** Medium-High
+- **Effort:** ~2 weeks
+- **Benefit:** Path toward replacing 345-line shell script
+
+## Key Insights from Analysis
+
+### What Makes CPython Build System Complex
+
+1. **Platform Diversity**: Unix, Windows, macOS, iOS, Android, WASM all use different build systems
+2. **Historical Accumulation**: 30+ years of changes with backward compatibility requirements
+3. **Module System**: 100+ optional modules with varying external dependencies
+4. **Bootstrap Paradox**: Need Python to build Python (solved with 4-stage bootstrap)
+5. **Optimization Layers**: PGO, BOLT, LTO add build complexity
+
+### Most Complex Components
+
+1. `configure.ac` (8,288 lines) - Platform/dependency detection
+2. `Makefile.pre.in` (3,418 lines) - 27+ build phases
+3. `PCbuild/*.vcxproj` (55 files) - Windows duplicates Unix logic
+4. Module configuration - Setup system + makesetup script
+5. Bootstrap/freezing - 4-stage build process
+
+### Simplification Opportunities
+
+**Low-Hanging Fruit (Done):**
+- ✅ Documentation improvements
+- ✅ Section organization
+- ✅ Module addition guides
+
+**Medium-Term (TODO):**
+- Replace makesetup shell script with Python
+- Modularize configure.ac
+- Generate Windows .vcxproj from metadata
+- Consolidate module configuration
+
+**Long-Term (Ambitious):**
+- Evaluate modern build systems (Meson, CMake)
+- Create build system abstraction layer
+- Improve cross-platform consistency
+
+## Testing Your Changes
+
+### Documentation Changes
+
+```bash
+# Build documentation
+cd Doc
+make clean
+make html
+
+# Check for warnings
+make html 2>&1 | grep -i warning
+
+# View in browser
+open build/html/index.html  # macOS
+xdg-open build/html/index.html  # Linux
+```
+
+### configure.ac Changes
+
+```bash
+# Regenerate configure script
+./Tools/build/regen-configure.sh
+
+# Verify no unexpected changes
+git diff configure
+
+# Test configure still works
+./configure
+make -j
+./python -c "import sys; print(sys.version)"
+```
+
+## Resources
+
+- **CPython Developer's Guide**: https://devguide.python.org/
+- **PEP 7 (C Style Guide)**: https://www.python.org/dev/peps/pep-0007/
+- **Issue Tracker**: https://github.com/python/cpython/issues
+- **Discourse**: https://discuss.python.org/c/core-dev
+
+## Questions?
+
+If you have questions about these changes or the build system:
+1. Check the documentation you've created!
+2. Ask on python-dev or discourse.python.org
+3. Reference this document in discussions
+
+---
+
+**Created:** 2025-11-15
+**Status:** Commits 1-3 ready for submission
+**Next Step:** Create GitHub issue for Commit 1