Connection Error Mid Training

[Jgmedina95]

Everything was doing great on training, for at least 90 steps, and then this error appeared, any idea on how I can prevent complete failure when theres connection issues? This training was on an 8xH100 instance from PrimeIntellect.

Generating rollouts (train): 75%|███████▌ | 96/128 [01:31<00:12, 2.63it/s]2025-12-02 03:58:17 - verifiers.envs.CrystalRelaxationMultiTurnEnv - ERROR - Error getting model response: Connection error.
2025-12-01 22:58:17

2025-12-01 22:58:17
Exiting...
2025-12-01 22:58:18
2025-12-02 03:58:17.856 | ERROR | asyncio.events:_run:88 - An error has been caught in function '_run', process 'MainProcess' (11969), thread 'MainThread' (139832729312128):
2025-12-01 22:58:18
Traceback (most recent call last):
2025-12-01 22:58:18

2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
2025-12-01 22:58:18
yield
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
2025-12-01 22:58:18
resp = await self._pool.handle_async_request(req)
2025-12-01 22:58:18
│ │ │ └ <Request [b'POST']>
2025-12-01 22:58:18
│ │ └ <function AsyncConnectionPool.handle_async_request at 0x7f2b63f58540>
2025-12-01 22:58:18
│ └ <AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>
2025-12-01 22:58:18
└ <httpx.AsyncHTTPTransport object at 0x7f2b5685e3f0>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
2025-12-01 22:58:18
raise exc from None
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
2025-12-01 22:58:18
response = await connection.handle_async_request(
2025-12-01 22:58:18
│ └ <function AsyncHTTPConnection.handle_async_request at 0x7f2b63f4b740>
2025-12-01 22:58:18
└ <AsyncHTTPConnection [CONNECTION FAILED]>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
2025-12-01 22:58:18
raise exc
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
2025-12-01 22:58:18
stream = await self._connect(request)
2025-12-01 22:58:18
│ │ └ <Request [b'POST']>
2025-12-01 22:58:18
│ └ <function AsyncHTTPConnection._connect at 0x7f2b63f4b7e0>
2025-12-01 22:58:18
└ <AsyncHTTPConnection [CONNECTION FAILED]>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 124, in _connect
2025-12-01 22:58:18
stream = await self._network_backend.connect_tcp(**kwargs)
2025-12-01 22:58:18
│ │ │ └ {'host': 'localhost', 'port': 8000, 'local_address': None, 'timeout': 1200, 'socket_options': None}
2025-12-01 22:58:18
│ │ └ <function AutoBackend.connect_tcp at 0x7f2b63f498a0>
2025-12-01 22:58:18
│ └ <httpcore._backends.auto.AutoBackend object at 0x7f2b5685e2d0>
2025-12-01 22:58:18
└ <AsyncHTTPConnection [CONNECTION FAILED]>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
2025-12-01 22:58:18
return await self._backend.connect_tcp(
2025-12-01 22:58:18
│ │ └ <function AnyIOBackend.connect_tcp at 0x7f2b63f5b2e0>
2025-12-01 22:58:18
│ └ <httpcore.AnyIOBackend object at 0x7f2a87ad3110>
2025-12-01 22:58:18
└ <httpcore._backends.auto.AutoBackend object at 0x7f2b5685e2d0>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
2025-12-01 22:58:18
with map_exceptions(exc_map):
2025-12-01 22:58:18
│ └ {<class 'TimeoutError'>: <class 'httpcore.ConnectTimeout'>, <class 'OSError'>: <class 'httpcore.ConnectError'>, <class 'anyio...
2025-12-01 22:58:18
└ <function map_exceptions at 0x7f2b640c6b60>
2025-12-01 22:58:18
File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit
2025-12-01 22:58:18
self.gen.throw(value)
2025-12-01 22:58:18
│ │ │ └ OSError('All connection attempts failed')
2025-12-01 22:58:18
│ │ └ <method 'throw' of 'generator' objects>
2025-12-01 22:58:18
│ └ <generator object map_exceptions at 0x7f26c33f7c40>
2025-12-01 22:58:18
└ <contextlib._GeneratorContextManager object at 0x7f26e0f529c0>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2025-12-01 22:58:18
raise to_exc(exc) from exc
2025-12-01 22:58:18
└ <class 'httpcore.ConnectError'>
2025-12-01 22:58:18

2025-12-01 22:58:18
httpcore.ConnectError: All connection attempts failed
2025-12-01 22:58:18

2025-12-01 22:58:18

2025-12-01 22:58:18
The above exception was the direct cause of the following exception:
2025-12-01 22:58:18

2025-12-01 22:58:18

2025-12-01 22:58:18
Traceback (most recent call last):
2025-12-01 22:58:18

2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1529, in request
2025-12-01 22:58:18
response = await self._client.send(
2025-12-01 22:58:18
│ │ └ <function AsyncClient.send at 0x7f2ba5d2f060>
2025-12-01 22:58:18
│ └ <httpx.AsyncClient object at 0x7f2b56a18440>
2025-12-01 22:58:18
└ <openai.AsyncOpenAI object at 0x7f2b56a183b0>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
2025-12-01 22:58:18
response = await self._send_handling_auth(
2025-12-01 22:58:18
│ └ <function AsyncClient._send_handling_auth at 0x7f2ba5d2f100>
2025-12-01 22:58:18
└ <httpx.AsyncClient object at 0x7f2b56a18440>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
2025-12-01 22:58:18
response = await self._send_handling_redirects(
2025-12-01 22:58:18
│ └ <function AsyncClient._send_handling_redirects at 0x7f2ba5d2f1a0>
2025-12-01 22:58:18
└ <httpx.AsyncClient object at 0x7f2b56a18440>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
2025-12-01 22:58:18
response = await self._send_single_request(request)
2025-12-01 22:58:18
│ │ └ <Request('POST', 'http://localhost:8000/v1/chat/completions')>
2025-12-01 22:58:18
│ └ <function AsyncClient._send_single_request at 0x7f2ba5d2f240>
2025-12-01 22:58:18
└ <httpx.AsyncClient object at 0x7f2b56a18440>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
2025-12-01 22:58:18
response = await transport.handle_async_request(request)
2025-12-01 22:58:18
│ │ └ <Request('POST', 'http://localhost:8000/v1/chat/completions')>
2025-12-01 22:58:18
│ └ <function AsyncHTTPTransport.handle_async_request at 0x7f2ba5d23ec0>
2025-12-01 22:58:18
└ <httpx.AsyncHTTPTransport object at 0x7f2b5685e3f0>
2025-12-01 22:58:18
File "/alloc/prime-rl/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
2025-12-01 22:58:18
with map_httpcore_exceptions():

[Jgmedina95]

Hi! after talking with @willccbb, we hypothesized that maybe this was occurring because of memory issues with rubrics, so he suggested to host the model Im using in my rubric and call that instead (using fastapi quickly). I did that but I still got the error. Although this time I did save the logs of the inference pane.

Here is also the toml file im using for this run when the error arised, and as you can see im using only a 4B model here (mainly for testing), because 8H100 should be enough to handle any issues with memory. So im unsure to why this is happening.
I did have to initialize my server separately to start training otherwise, LLM inference would failed automatically cause I think it assumed I covered that on my own.

inference_gpu_ids = [0,1,2,3,4,5]
trainer_gpu_ids = [6,7]

max_steps = 500
max_async_level = 4

[model]
name = "Qwen/Qwen3-4B-Thinking-2507"

[wandb]
project = "crystal"
name = "crystal-4b"

[trainer.optim]
lr = 1e-5
weight_decay = 0.0

[trainer.model.experimental.lora]
rank = 8
alpha = 32
dropout = 0.0
target_modules = [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
]

[orchestrator]
batch_size = 128
rollouts_per_example = 8
seq_len = 6000


[orchestrator.sampling]
max_tokens = 4096

[orchestrator.buffer]
online_difficulty_filtering = true

[[orchestrator.env]]
id = "jmedina9/crystal_relaxation"
args={port="2222",dataset_path="/data/jmedina9-crystal_relaxation-latest/data/mp_20/mp_20_relaxation_generation_dataset",max_turns=3}
[inference.model]
enable_auto_tool_choice = true
tool_call_parser = "hermes"

error_inf.log