Merge branch 'main' into check-callable-callmethodformat

Author: johnslavik

Doc/deprecations/pending-removal-in-3.20.rst

@@ -7,6 +7,7 @@ Pending removal in Python 3.20
 
   - :mod:`argparse`
   - :mod:`csv`
+  - :mod:`ctypes`
   - :mod:`!ctypes.macholib`
   - :mod:`decimal` (use :data:`decimal.SPEC_VERSION` instead)
   - :mod:`http.server`

Doc/library/profiling.sampling.rst

@@ -295,21 +295,23 @@ The default configuration works well for most use cases:
    :widths: 25 75
 
    * - Option
-     - Default behavior
-   * - ``--interval`` / ``-i``
+     - Default
+   * - Default for ``--interval`` / ``-i``
      - 100 µs between samples (~10,000 samples/sec)
-   * - ``--duration`` / ``-d``
-     - Profile for 10 seconds
-   * - ``--all-threads`` / ``-a``
-     - Sample main thread only
-   * - ``--native``
+   * - Default for ``--duration`` / ``-d``
+     - 10 seconds
+   * - Default for ``--all-threads`` / ``-a``
+     - Main thread only
+   * - Default for ``--native``
      - No ``<native>`` frames (C code time attributed to caller)
-   * - ``--no-gc``
-     - Include ``<GC>`` frames when garbage collection is active
-   * - ``--mode``
+   * - Default for ``--no-gc``
+     - ``<GC>`` frames included when garbage collection is active
+   * - Default for ``--mode``
      - Wall-clock mode (all samples recorded)
-   * - ``--realtime-stats``
-     - No live statistics display during profiling
+   * - Default for ``--realtime-stats``
+     - Disabled
+   * - Default for ``--subprocesses``
+     - Disabled
 
 
 Sampling interval and duration
@@ -442,6 +444,78 @@ working correctly and that sufficient samples are being collected. See
 :ref:`sampling-efficiency` for details on interpreting these metrics.
 
 
+Subprocess profiling
+--------------------
+
+The :option:`--subprocesses` option enables automatic profiling of subprocesses
+spawned by the target::
+
+   python -m profiling.sampling run --subprocesses script.py
+   python -m profiling.sampling attach --subprocesses 12345
+
+When enabled, the profiler monitors the target process for child process
+creation. When a new Python child process is detected, a separate profiler
+instance is automatically spawned to profile it. This is useful for
+applications that use :mod:`multiprocessing`, :mod:`subprocess`,
+:mod:`concurrent.futures` with :class:`~concurrent.futures.ProcessPoolExecutor`,
+or other process spawning mechanisms.
+
+.. code-block:: python
+   :caption: worker_pool.py
+
+   from concurrent.futures import ProcessPoolExecutor
+   import math
+
+   def compute_factorial(n):
+       total = 0
+       for i in range(50):
+           total += math.factorial(n)
+       return total
+
+   if __name__ == "__main__":
+       numbers = [5000 + i * 100 for i in range(50)]
+       with ProcessPoolExecutor(max_workers=4) as executor:
+           results = list(executor.map(compute_factorial, numbers))
+       print(f"Computed {len(results)} factorials")
+
+::
+
+   python -m profiling.sampling run --subprocesses --flamegraph worker_pool.py
+
+This produces separate flame graphs for the main process and each worker
+process: ``flamegraph_<main_pid>.html``, ``flamegraph_<worker1_pid>.html``,
+and so on.
+
+Each subprocess receives its own output file. The filename is derived from
+the specified output path (or the default) with the subprocess's process ID
+appended:
+
+- If you specify ``-o profile.html``, subprocesses produce ``profile_12345.html``,
+  ``profile_12346.html``, and so on
+- With default output, subprocesses produce files like ``flamegraph_12345.html``
+  or directories like ``heatmap_12345``
+- For pstats format (which defaults to stdout), subprocesses produce files like
+  ``profile_12345.pstats``
+
+The subprocess profilers inherit most sampling options from the parent (interval,
+duration, thread selection, native frames, GC frames, async-aware mode, and
+output format). All Python descendant processes are profiled recursively,
+including grandchildren and further descendants.
+
+Subprocess detection works by periodically scanning for new descendants of
+the target process and checking whether each new process is a Python process
+by probing the process memory for Python runtime structures. Non-Python
+subprocesses (such as shell commands or external tools) are ignored.
+
+There is a limit of 100 concurrent subprocess profilers to prevent resource
+exhaustion in programs that spawn many processes. If this limit is reached,
+additional subprocesses are not profiled and a warning is printed.
+
+The :option:`--subprocesses` option is incompatible with :option:`--live` mode
+because live mode uses an interactive terminal interface that cannot
+accommodate multiple concurrent profiler displays.
+
+
 .. _sampling-efficiency:
 
 Sampling efficiency
@@ -1217,6 +1291,11 @@ Sampling options
    Compatible with ``--live``, ``--flamegraph``, ``--heatmap``, and ``--gecko``
    formats only.
 
+.. option:: --subprocesses
+
+   Also profile subprocesses. Each subprocess gets its own profiler
+   instance and output file. Incompatible with ``--live``.
+
 
 Mode options
 ------------

Doc/whatsnew/3.15.rst

@@ -74,6 +74,8 @@ Summary -- Release highlights
 * :pep:`782`: :ref:`A new PyBytesWriter C API to create a Python bytes object
   <whatsnew315-pep782>`
 * :ref:`Improved error messages <whatsnew315-improved-error-messages>`
+* :ref:`__pycache__ directories now contain a .gitignore file
+  <whatsnew315-pycache-gitignore>`
 
 
 New features
@@ -397,6 +399,12 @@ Other language changes
   for any class.
   (Contributed by Serhiy Storchaka in :gh:`41779`.)
 
+.. _whatsnew315-pycache-gitignore:
+
+* :file:`__pycache__` directories now contain a :file:`.gitignore` file for Git
+  that ignores their contents.
+  (Contributed by Stan Ulbrych in :gh:`141081`.)
+
 
 New modules
 ===========
@@ -1024,6 +1032,7 @@ New deprecations
 
     - :mod:`argparse`
     - :mod:`csv`
+    - :mod:`ctypes`
     - :mod:`!ctypes.macholib`
     - :mod:`decimal` (use :data:`decimal.SPEC_VERSION` instead)
     - :mod:`http.server`

Include/internal/pycore_context.h

@@ -55,5 +55,8 @@ struct _pycontexttokenobject {
 // Export for '_testcapi' shared extension
 PyAPI_FUNC(PyObject*) _PyContext_NewHamtForTests(void);
 
+PyAPI_FUNC(int) _PyContext_Enter(PyThreadState *ts, PyObject *octx);
+PyAPI_FUNC(int) _PyContext_Exit(PyThreadState *ts, PyObject *octx);
+
 
 #endif /* !Py_INTERNAL_CONTEXT_H */

Include/internal/pycore_global_objects_fini_generated.h

@@ -717,6 +717,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(readline)
         STRUCT_FOR_ID(readonly)
         STRUCT_FOR_ID(real)
+        STRUCT_FOR_ID(recursive)
         STRUCT_FOR_ID(reducer_override)
         STRUCT_FOR_ID(registry)
         STRUCT_FOR_ID(rel_tol)

Include/internal/pycore_global_strings.h

@@ -5,22 +5,16 @@
 import sysconfig as _sysconfig
 import types as _types
 
-__version__ = "1.1.0"
-
 from _ctypes import Union, Structure, Array
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
-from _ctypes import __version__ as _ctypes_version
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 from _ctypes import SIZEOF_TIME_T
 from _ctypes import CField
 
 from struct import calcsize as _calcsize
 
-if __version__ != _ctypes_version:
-    raise Exception("Version number mismatch", __version__, _ctypes_version)
-
 if _os.name == "nt":
     from _ctypes import COMError, CopyComPointer, FormatError
 
@@ -673,3 +667,12 @@ def DllCanUnloadNow():
     raise SystemError(f"Unexpected sizeof(time_t): {SIZEOF_TIME_T=}")
 
 _reset_cache()
+
+
+def __getattr__(name):
+    if name == "__version__":
+        from warnings import _deprecated
+
+        _deprecated("__version__", remove=(3, 20))
+        return "1.1.0"  # Do not change
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Include/internal/pycore_runtime_init_generated.h

@@ -967,6 +967,19 @@ def set_data(self, path, data, *, _mode=0o666):
                 _bootstrap._verbose_message('could not create {!r}: {!r}',
                                             parent, exc)
                 return
+
+            if part == _PYCACHE:
+                gitignore = _path_join(parent, '.gitignore')
+                try:
+                    _path_stat(gitignore)
+                except FileNotFoundError:
+                    gitignore_content = b'# Created by CPython\n*\n'
+                    try:
+                        _write_atomic(gitignore, gitignore_content, _mode)
+                    except OSError:
+                        pass
+                except OSError:
+                    pass
         try:
             _write_atomic(path, data, _mode)
             _bootstrap._verbose_message('created {!r}', path)