gh-75572: Speed up test_xpickle #144393

serhiy-storchaka · 2026-02-02T11:04:07Z

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

Issue: Forward-port test_xpickle from 2.7 to 3.x #75572

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

vstinner · 2026-02-02T12:49:26Z

Lib/test/test_xpickle.py

+        worker = cls.worker
+        if worker is None:
+            target = os.path.join(os.path.dirname(__file__), 'xpickle_worker.py')
+            worker = subprocess.Popen([*python, target],


Why not calling start_worker() here?

Oh, this is a remnant. I was not sure whether it is worth to move this code out into a method.

vstinner · 2026-02-02T12:54:23Z

Lib/test/test_xpickle.py

+        cls.worker = worker
+
+    @classmethod
+    def close_worker(cls):


"stop_worker()" name would be better to have start/stop_worker methods.

vstinner · 2026-02-02T13:00:32Z

Lib/test/test_xpickle.py

+                                  stdout=subprocess.PIPE,
+                                  stderr=subprocess.PIPE,
+                                  # For windows bpo-17023.
+                                  shell=is_windows)


You should use text=True to fix this code below:

_, stderr = worker.communicate() raise RuntimeError(stderr)

But we do binary IO for stdin/stdout in normal case.

This code is only for troubleshooting. It is not executed when the tests pass.

vstinner

./python -m test test_xpickle -u xpickle,cpu -v command takes:

main branch: 24 min 38 sec
this PR: 1 min 24 sec

Well, it's much faster with this PR :-)

Note: I measured timings on a debug build (gcc -Og).

vstinner

LGTM.

serhiy-storchaka · 2026-02-02T17:59:32Z

Note that you can run it only for the selected version, e.g. xpython=3.13. In debug build it is about 50% slower.

miss-islington-app · 2026-02-02T18:00:17Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request. (cherry picked from commit 29acc08) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app · 2026-02-02T18:00:32Z

GH-144403 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2026-02-02T18:00:37Z

GH-144404 is a backport of this pull request to the 3.13 branch.

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request. (cherry picked from commit 29acc08) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-bot · 2026-02-03T11:19:10Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows Server 2022 NoGIL 3.13 (tier-1) has failed when building commit 9d76689.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1488/builds/1904) and take a look at the build logs.
Check if the failure is related to this commit (9d76689) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1488/builds/1904

Failed tests:

test_launcher

Failed subtests:

test_search_major_2 - test.test_launcher.TestLauncher.test_search_major_2
test_search_major_3 - test.test_launcher.TestLauncher.test_search_major_3

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "C:\Users\Administrator\buildarea\3.13.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\test_launcher.py", line 445, in test_search_major_3
    self.assertEqual("PythonCore", data["env.company"])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'PythonCore' != 'PythonTestSuite'
- PythonCore
+ PythonTestSuite


Traceback (most recent call last):
  File "C:\Users\Administrator\buildarea\3.13.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\test_launcher.py", line 465, in test_search_major_2
    self.assertEqual("PythonCore", data["env.company"])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'PythonCore' != 'PythonTestSuite'
- PythonCore
+ PythonTestSuite

bedevere-bot · 2026-02-03T11:29:57Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.13 (tier-1) has failed when building commit 9d76689.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1441/builds/1761) and take a look at the build logs.
Check if the failure is related to this commit (9d76689) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1441/builds/1761

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 856, in gc_collect
    gc.collect()
ResourceWarning: unclosed file <_io.FileIO name=13 mode='wb' closefd=True>

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

pythongh-75572: Speed up test_xpickle

c7bd567

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

serhiy-storchaka added tests Tests in the Lib/test dir skip news needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Feb 2, 2026

bedevere-app bot added the awaiting core review label Feb 2, 2026

bedevere-app bot mentioned this pull request Feb 2, 2026

Forward-port test_xpickle from 2.7 to 3.x #75572

Closed

vstinner reviewed Feb 2, 2026

View reviewed changes

Cleanup.

1081bff

vstinner reviewed Feb 2, 2026

View reviewed changes

vstinner approved these changes Feb 2, 2026

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Feb 2, 2026

serhiy-storchaka merged commit 29acc08 into python:main Feb 2, 2026
49 checks passed

bedevere-app bot removed the awaiting merge label Feb 2, 2026

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Feb 2, 2026

serhiy-storchaka deleted the test_xpickle-speedup branch February 2, 2026 18:00

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Feb 2, 2026

Aniketsy pushed a commit to Aniketsy/cpython that referenced this pull request Feb 3, 2026

pythongh-75572: Speed up test_xpickle (pythonGH-144393)

20535dc

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

Uh oh!

gh-75572: Speed up test_xpickle #144393

gh-75572: Speed up test_xpickle #144393

Conversation

serhiy-storchaka commented Feb 2, 2026 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka commented Feb 2, 2026

Uh oh!

Uh oh!

miss-islington-app bot commented Feb 2, 2026

Uh oh!

bedevere-app bot commented Feb 2, 2026

Uh oh!

bedevere-app bot commented Feb 2, 2026

Uh oh!

bedevere-bot commented Feb 3, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Feb 3, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

serhiy-storchaka commented Feb 2, 2026 •

edited by bedevere-app bot

Loading