Lots of failed requests in end-to-end tests

[krzema12]

See e.g. https://github.com/typesafegithub/github-workflows-kt/actions/runs/20230697144/job/58075564463

Run rm '.github/workflows/end-to-end-tests.yaml' && '.github/workflows/end-to-end-tests.main.kts'
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://bindings.krzeminski.it:443: Connection reset
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://bindings.krzeminski.it:443: Connection reset
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://bindings.krzeminski.it:443: Connection reset
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://bindings.krzeminski.it:443: Connection reset
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->[https://bindings.krzeminski.it:443](https://bindings.krzeminski.it/)
.github/workflows/end-to-end-tests.main.kts: warning: the following artifacts could not be resolved: actions:github-script:jar:v8 (absent): Could not transfer artifact actions:github-script:jar:v8 from/to https___bindings.krzeminski.it (https://bindings.krzeminski.it/): transfer failed for https://bindings.krzeminski.it/actions/github-script/v8/github-script-v8.jar
.github/workflows/end-to-end-tests.main.kts:7:1: warning: file 'actions:github-script:v8' not found
Error: Process completed with exit code 1.

I noticed this problem occurs mainly, if not only, in https://github.com/typesafegithub/github-workflows-kt/blob/main/.github/workflows/end-to-end-tests.main.kts. Reproduces locally. When I don't use the library published locally, things work fine.

[krzema12]

I checked that when I use a local instance of the service, the end-to-end workflow compiles without failed requests. Can it be that there's too many requests at the same time for packages where the service returns HTTP 404? Maybe some load balancer has a spillover enabled?

[krzema12]

Testing with:

for i in $(seq 1 1000);
do
    echo "Request $i"
    curl https://bindings.krzeminski.it/org/jetbrains/kotlin/kotlin-stdlib/2.1.0/kotlin-stdlib-2.1.0.pom 
done 2>&1 | tee out.txt

And getting some curl: (35) Recv failure: Connection reset by peer.

[krzema12]

Looks like #2183 helped - got green runs in:

• https://github.com/typesafegithub/github-workflows-kt/actions/runs/20353149963
• https://github.com/typesafegithub/github-workflows-kt/actions/runs/20353149970

Let's wait a few days to see how it works.

[krzema12]

It still fails sometimes.

Let me start an investigation doc: Troubleshoot requests without response from github-workflows-kt's binding server.