Fork safe ConcurrentMultiSpanProcessor

[gregoiredx]

Description

ConcurrentMultiSpanProcessor is not "fork safe". This MR tries to solve this problem.

Forking a threaded process is not a good idea in the first place, citing the Python doc, see https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Note that safely forking a multithreaded process is problematic.

Also, the OTel Python SDK suggest to only initialize the SDK post fork, see https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html

Nevertheless, on an existing code base, it's not always easy to avoid all forks. So it looks like a good idea to try and make the SDK as fork safe as possible.

Note: BatchProcessor is already relying on register_at_fork used here, see

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/_shared_internal/__init__.py

Line 125 in 102fec2

os.register_at_fork(after_in_child=lambda: weak_reinit()()) # pyright: ignore[reportOptionalCall] pylint: disable=unnecessary-lambda

Fixes #2349

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

See unit test and discussions below.

An equivalent patch is currently running in our stack.

Does This PR Require a Contrib Repo Change?

?

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: xrmx / name: Riccardo Magliocchetti (7771136)

[gregoiredx]

@DylanRussell thanks a lot for the review.
The first pass of the CI wasn't successful.
I applied two fixes:

• multiprocessing.set_start_method was already started in another test and it can be called only once, see https://github.com/open-telemetry/opentelemetry-python/actions/runs/20662185206/job/59826482487#step:5:778. I fixed it by switching to multiprocessing.get_context
• changelog was missing, I added a line to CHANGELOG.md

There's another test which failed but it doesn't look like it's related to my change, see https://github.com/open-telemetry/opentelemetry-python/actions/runs/20662185256/job/59826483790#step:6:39

Can you relaunch the CI please?

[gregoiredx]

There's still a CI step failing but I have no idea how I can fix this. It looks like a bug/flakiness in the CI workflow
https://github.com/open-telemetry/opentelemetry-python/actions/runs/20852250476/job/59913535963#step:6:22

Updating https://github.com/open-telemetry/opentelemetry-python.git (2b1e6540d049eb4abcf09d445cec06ae389aa1e4)
  × Failed to download and build `opentelemetry-api @
  │ git+[https://github.com/open-telemetry/opentelemetry-python.git@2b1e6540d049eb4abcf09d445cec06ae389aa1e4#egg=opentelemetry-api&subdirectory=opentelemetry-api`](https://github.com/open-telemetry/opentelemetry-python.git@2b1e6540d049eb4abcf09d445cec06ae389aa1e4#egg=opentelemetry-api&subdirectory=opentelemetry-api%60)
  ├─▶ Git operation failed
  ├─▶ failed to find branch, tag, or commit
  │   `2b1e6540d049eb4abcf09d445cec06ae389aa1e4`

[DylanRussell]

Yeah don't worry about that it's a very common flake that randomly occurs on tests, some sort of race condition I think

[gregoiredx]

can we relaunch the failing tests? Or merge? the failures look like flaky tests