| andersson123 | test | 09:48 |
|---|---|---|
| bluca | hi andersson123 - may I annoy you with autopkgtest trigger problems? :-) | 09:57 |
| andersson123 | of course, what's happening | 09:58 |
| bluca | been getting error 500 for 2 days | 09:58 |
| bluca | error 500s with: A server error has occured, please contact a member of the Ubuntu QA team. You can contact them via the ubuntu-quality mailing list, or via the #ubuntu-quality IRC channel on irc.libera.chat. Traceback: <urlopen error Tunnel connection failed: 403 Forbidden> | 09:58 |
| andersson123 | we have recently migrated our bastion, workers and webservers (the webserver that you request a test via) to a new data centre. I hadn't thought about it until now, but my guess is that the auth creds you have set up haven't been migrated. Apologies, we haven't migrated the bastion etc before and this was unforeseen | 10:00 |
| bluca | it started tuesday late afternoon | 10:00 |
| bluca | ah I see - do we need new creds? | 10:00 |
| andersson123 | yep. Our service was down for a little while whilst we migrated and then since it came back up (yesterday) I guess it's your creds giving you 500 | 10:01 |
| andersson123 | not sure just yet, we might be able to save them from the old bastion, I'll look soon | 10:01 |
| bluca | ok, thanks, let me know | 10:02 |
| andersson123 | okay, I was lying (involuntarily). The secrets are the same. I can also see your test requests | 10:05 |
| andersson123 | precisely when are you gitting this 403? | 10:05 |
| andersson123 | when you are trying to get the results, or when you request the test? | 10:05 |
| andersson123 | because we are still working on the storage of the results - the logs etc are not currently publically accessible | 10:06 |
| andersson123 | and will give you 403 unauthorised | 10:07 |
| andersson123 | https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/s/systemd/20231108_234427_774dc@/log.gz | 10:07 |
| andersson123 | like this | 10:07 |
| bluca | when the test is requested as far as I can see | 10:07 |
| andersson123 | do you have a link to your runner? I'd like to look | 10:07 |
| bluca | when a PR is opened or when I trigger manually | 10:07 |
| andersson123 | or a log from your runner | 10:08 |
| bluca | so there's a retry-github-test on the autopkgtest repo that we lifted - that gives the error too, can you use that to repro it? | 10:09 |
| bluca | the webhook is in the github repo and it's not visible | 10:09 |
| bluca | I can fish out logs from there if it can help | 10:09 |
| bluca | an example URL req is: | 10:10 |
| bluca | https://autopkgtest.ubuntu.com/request.cgi?release=jammy&arch=amd64&package=systemd-upstream&build-git=https%3A%2F%2Fsalsa.debian.org%2Fsystemd-team%2Fsystemd.git%23upstream-ci&ppa=upstream-systemd-ci/systemd-ci&env=CFLAGS=-O0%3BDEB_BUILD_PROFILES=noudeb% | 10:10 |
| bluca | this is the script I am talking about: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/retry-github-test | 10:12 |
| bluca | called through this script: https://paste.debian.net/1297646/ | 10:13 |
| andersson123 | I'm confused because I can see your tests in our logs returning 200 (at least at the request stage) so if you have a link to any logs from the runner, that'd be helpful | 10:13 |
| bluca | eg: retry-gh-systemd-test 29924 amd64 gives me HTTP Error 500: INTERNAL SERVER ERROR | 10:13 |
| bluca | let me check on a webhook failed invocation, need to make sure there's no secret | 10:15 |
| andersson123 | thank you | 10:15 |
| bluca | https://paste.debian.net/1297647/ | 10:17 |
| bluca | that's everyting I see for a req/resp on a pr open event | 10:17 |
| bluca | from an hour ago or so | 10:18 |
| bluca | if you want to see things happen "live" let me know and I can click the redeliver button | 10:18 |
| bluca | this broke at around 5pm UTC on tuesday btw if it helps | 10:18 |
| andersson123 | looking now! | 10:22 |
| andersson123 | I changed something in the code that should make it easier to debug what's going on. When you have a free moment, could you retrigger a test or something please? | 10:26 |
| bluca | done | 10:27 |
| bluca | body of response: A server error occurred. Please contact the administrator. | 10:27 |
| andersson123 | perfect ta | 10:29 |
| andersson123 | You can (hopefully) expect this fixed a little later today | 10:35 |
| andersson123 | bluca: I've figured out what the issue is, fix on the way :D will probably be a little while though. | 12:12 |
| bluca | thank you! | 12:30 |
| bluca | let me know when it's ready to test | 12:30 |
| andersson123 | I have tried a quick (hopeful) fix, pretty sure it won't work but if you could give it a try now and let me know if it works that'd be helpful | 12:46 |
| bluca | got a 404 now | 12:48 |
| bluca | sorry, 504 | 12:48 |
| bluca | but no header nor body | 12:48 |
| andersson123 | ok ty | 12:48 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!