| Skia | doko: oh, not again.... last time it required a lot of kicking to finally pass. iirc it's partially because the machines are slow, and partially because we have a proxy | 08:02 |
|---|---|---|
| Skia | I think the root issue is that the test's connection is going through the proxy, while the test is not aware of that, and the proxy connection times out, but with the wrong manner, just because the machine is too slow, or something like that | 08:03 |
| Skia | thinking back a bit more, I think it's even the contrary: when the machines are too slow, the connection going to the proxy is timing out, thus the tests succeed, but when it's fast enough, the connection is able to succeed and the timeout never occurs, making the test to fail | 08:12 |
| doko | Skia: please could you document that in https://bugs.launchpad.net/ubuntu/+source/rust-reqwest/+bug/2080346 ? | 08:35 |
| doko | ginggs: ^^^ how should we handle that, apparently these are infrastructure issues | 08:35 |
| doko | and the tmpfail failure rate on armhf was awful last night | 08:37 |
| Skia | doko: yes, I'm on it | 08:56 |
| Skia | doko: there, I've commented on the bug, and along the way, thought of maybe one possible solution that I'm going to experiment right away | 09:19 |
| Skia | doko: I've finally found something that works: `iptables -A OUTPUT -d 10.255.255.1/32 -p tcp -j DROP` so that the IP is really unreachable. I don't know what side effect this could have, but I don't expect many, so it's already cowboyed on one of the two workers. I'll discuss quickly with the team to see about deploying that more broadly, and then let you know | 13:01 |
| doko | https://autopkgtest.ubuntu.com/packages/rust-gst-plugin-gtk4/oracular/arm64 shows some long running tests, +10h. please could you check if these are dead? | 13:31 |
| Skia | doko: the `iptables` change is deployed everywhere, so you should see some improvements with `rust-reqwest`, but you might be interested to read the concern raised in the commit log: https://git.launchpad.net/autopkgtest-cloud/commit/?id=5ca81efbda620af37267105615a8ec9d8637d59c | 13:42 |
| Skia | please provide feedback if you observe something unrelated to break that might be related to this change | 13:43 |
| Skia | I'll finish documenting that fix and will look at your possibly stuck jobs | 13:43 |
| Skia | doko: I've terminated the jobs that were running for more than 10h, to give them a chance to run on a faster machine | 14:54 |
| Skia | the one that's been running for more than 5 hours is still running fine, so I'll leave it | 14:54 |
| Skia | and for information, we don't see anything in the logs because the last line of logs is more than 2k characters (a big cargo command), and autopkgtest-cloud doesn't handle that well | 14:56 |
| doko | ahh, I see you are already retrying | 14:56 |
| Skia | rust-reqwest only, not the one with the simingly stuck jobs | 15:00 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!