feat(http): add error handling for exporting #4709

pafi-code · 2025-08-01T14:21:34Z

Description

When using the OTEL http exporters the std.out gets polluted by unhandled error traces. I don't think that this should be the desired result. This happens when the endpoint is not available when exporting telemetry. This can be cause by different events but in general I think it should be better handled.
Therefore I added error handling with more specific and shorted error messages.

Type of change

I'm not sure if the change type is rather fix or feature, as the service was still running.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Don't provide a collector at the configured endpoint

Does This PR Require a Contrib Repo Change?

Yes. - Link to PR:
No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

linux-foundation-easycla · 2025-08-01T14:21:37Z

The committers listed above are authorized under a signed CLA.

✅ login: pafi-code / name: Paul (397b040, 80a5f87, 8d9e42e, 943634f, afc67a3, b36fb79, b8cea5c, bcc20fd, cc3f2f7, ef2ff34, f501b94, f63e810, fb0684e)

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py

pafi-code · 2025-10-21T13:42:46Z

Okay so I know that the pipeline is failing and I assume this PR should be rebased, but for me there is still the question if my proposed fix is actually how it should be handled. Would appreciate some more feedback here. Also I've created an issue related to this PR: #4712

emdneto

@pafi-code can we have tests for this?

pafi-code · 2025-11-06T11:04:42Z

@emdneto I had a look into the existing tests and too be honest I have no clue what's going on. I'm not used to the sort of testing that you guys are doing. Would appreciate if you could help regarding the tests.

DylanRussell · 2025-11-06T17:17:36Z

...orter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py

+                    return MetricExportResult.SUCCESS
+            except requests.exceptions.RequestException as error:
+                reason = str(error)
+                retryable = True


Maybe only mark it as retryable if it's a connection error to start out ? Otherwise this looks good to me. Also can we add a test that exercises this branch in the code...

So you want me to catch the ConnectionError? :D

For testing as I already said I have a really hard time understanding whats going on :D So I would appreciate some help.

No I meant only make Connection error retryable, but leave the broad requests exception catch as you have it..

For testing you just need to add 1 test (maybe 2 if you special case Connection error as retryable to exercise that logic). Here is the span exporter test: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-http/tests/test_proto_span_exporter.py#L281-L304 -- you'll want to add basically duplicate tests to each of the 3 http exporters.. You can set up the mock post call to raise an exception (mock_post.side_effect = RequestsException), see (https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-http/tests/test_proto_span_exporter.py#L288C9-L288C38)

IN order to run the tests make sure you have uv installed.. Then run uv sync at the top level directory (opentelemetry-python). Then activate the venv that uv creates (source .venv/bin/activate in the same directory).. Then you can run the unit tests via uv via tox -e py312-test-opentelemetry-exporter-otlp-proto-http -- you can run them for a different version of python too (run tox -l | grep http to see all python versions the tests can run again).. You can also have uv install python version if you haven't done that check out the docs for UV tool.

@DylanRussell is this what you had in mind?

herin049 · 2025-11-16T02:32:26Z

Have we considered just using urllib3.util.Retry instead of rolling our own retry-backoff loop?
Docs: https://requests.readthedocs.io/en/latest/user/advanced/#example-automatic-retries

pafi-code · 2025-11-17T13:30:10Z

@herin049 should this be part of this PR as this is a refactor and not a fix?

pafi-code · 2025-11-18T11:22:41Z

Regarding the two failing checks I'm not really sure what I should do.
For the public_symbols_check I did not introduce those changes for which it is failing.
For the changelog: Should I update anything there?

emdneto

@pafi-code, please add the changelog. public-symbols-check should be ok by just rebasing the branch before merging.

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py

CHANGELOG.md

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py

...orter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py

...porter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py

CHANGELOG.md

pafi-code · 2025-12-04T10:55:47Z

I will rebase the PR once again, after we got the approvals. Is there any time limit regarding how long we wait for approval or do we just wait? (not trying to stress here, just curious about the process)

exporter/opentelemetry-exporter-otlp-proto-http/tests/metrics/test_otlp_metrics_exporter.py

emdneto · 2025-12-17T23:44:03Z

CHANGELOG.md

 > [!IMPORTANT]
 > We are working on stabilizing the Log signal that would require making deprecations and breaking changes. We will try to reduce the releases that may require an update to your code, especially for instrumentations or for sdk developers.

 ## Unreleased


Remind to fix changelog

Hmm? Remind for whom?

For you before we merge

aabmass · 2026-01-15T17:11:33Z

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py

+                if resp.ok:
+                    return LogRecordExportResult.SUCCESS
+            except requests.exceptions.RequestException as error:
+                reason = str(error)


nit I don't think you need to convert to a string here, the logger.error() call should handle it lazily and then handlers/formatters can read the exception directly.

aabmass · 2026-01-15T17:12:22Z

...orter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py

+                if resp.ok:
+                    return MetricExportResult.SUCCESS
+            except requests.exceptions.RequestException as error:
+                reason = str(error)


same comment

aabmass · 2026-01-15T17:12:32Z

...porter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py

+                if resp.ok:
+                    return SpanExportResult.SUCCESS
+            except requests.exceptions.RequestException as error:
+                reason = str(error)


emdneto

@pafi-code can you please fix the comments and changelog?

pafi-code requested a review from a team as a code owner August 1, 2025 14:21

pafi-code force-pushed the feature/add_error_handling_for_otlp_exporters branch from d3d4031 to afb76f2 Compare August 1, 2025 14:45

aabmass reviewed Aug 1, 2025

View reviewed changes

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py Outdated Show resolved Hide resolved

pafi-code mentioned this pull request Aug 5, 2025

Retry loop never executed in OTLP HTTP exporters #4712

Open

xrmx added this to @xrmx's Python PR digest Aug 18, 2025

xrmx moved this to Reviewed PR that needs fixing in @xrmx's Python PR digest Aug 18, 2025

pafi-code force-pushed the feature/add_error_handling_for_otlp_exporters branch from afb76f2 to f581f22 Compare October 24, 2025 10:02

emdneto reviewed Oct 29, 2025

View reviewed changes

DylanRussell reviewed Nov 6, 2025

View reviewed changes

emdneto reviewed Nov 18, 2025

View reviewed changes

...xporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py Outdated Show resolved Hide resolved

pafi-code force-pushed the feature/add_error_handling_for_otlp_exporters branch 2 times, most recently from 6b5c920 to f33957f Compare November 24, 2025 13:31

emdneto reviewed Nov 28, 2025

View reviewed changes

emdneto requested review from DylanRussell and aabmass November 28, 2025 19:51

pafi-code force-pushed the feature/add_error_handling_for_otlp_exporters branch from a6c93d1 to 4a6fec8 Compare December 1, 2025 09:56

emdneto approved these changes Dec 1, 2025

View reviewed changes

emdneto moved this from Reviewed PRs that need fixes to Ready for review in @xrmx's Python PR digest Dec 2, 2025

DylanRussell reviewed Dec 4, 2025

View reviewed changes

exporter/opentelemetry-exporter-otlp-proto-http/tests/metrics/test_otlp_metrics_exporter.py Outdated Show resolved Hide resolved

exporter/opentelemetry-exporter-otlp-proto-http/tests/metrics/test_otlp_metrics_exporter.py Outdated Show resolved Hide resolved

pafi-code added 3 commits December 9, 2025 11:03

feat(http): add error handling for exporting

cc3f2f7

feat(http_exporter): allow to run retry loop on connection errors

bcc20fd

feat(http): change error types that are caught

b8cea5c

pafi-code added 10 commits December 9, 2025 11:03

refactor(http): introduce variables to unify logging

fb0684e

feat(http_exporter): only retry on connection error

80a5f87

test(http_exporter): add test case for connection errors while exporting

943634f

refactor(http): simplify if statements

397b040

docs(changelog): add changes

afc67a3

fix(http_exporter): use correct class after rebase

8d9e42e

docs(changelog): update changelog

b36fb79

refactor(http_exporter): add empty space in logs

f63e810

docs(tests): remove comments

f501b94

refactor(tests): simplify tests

ef2ff34

pafi-code force-pushed the feature/add_error_handling_for_otlp_exporters branch from dd052a7 to ef2ff34 Compare December 9, 2025 10:05

emdneto reviewed Dec 17, 2025

View reviewed changes

aabmass approved these changes Jan 15, 2026

View reviewed changes

DylanRussell approved these changes Jan 15, 2026

View reviewed changes

emdneto reviewed Jan 16, 2026

View reviewed changes

feat(http): add error handling for exporting #4709

Are you sure you want to change the base?

feat(http): add error handling for exporting #4709

Conversation

pafi-code commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Uh oh!

linux-foundation-easycla bot commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pafi-code commented Oct 21, 2025

Uh oh!

emdneto left a comment

Choose a reason for hiding this comment

Uh oh!

pafi-code commented Nov 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DylanRussell Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herin049 commented Nov 16, 2025

Uh oh!

pafi-code commented Nov 17, 2025

Uh oh!

pafi-code commented Nov 18, 2025

Uh oh!

emdneto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pafi-code commented Dec 4, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emdneto left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pafi-code commented Aug 1, 2025 •

edited

Loading

linux-foundation-easycla bot commented Aug 1, 2025 •

edited

Loading

DylanRussell Nov 11, 2025 •

edited

Loading