| === marcustomlinson_ is now known as marcustomlinson | ||
| === synapt is now known as nate | ||
| === leo is now known as Guest4913 | ||
| Guest4913 | hey folks! just out of curiosity is there something going on with ppa.launchpad.net at the moment? we are looking at some timeouts starting from 07:00 UTC today connecting to ppa.launchpad.net (91.189.95.85) from AWS us-east-1 (Northern Virginia/USA), this seems to be a transient issue and we have assembled a lot of different signals to debug this on our side at this point. happy to put some relevant information wherever it can be useful to y'all, | 13:08 |
|---|---|---|
| cjwatson | Guest4913: #canonical-sysadmin might know better. From this point of view ppa.launchpad.net is basically just an Apache server and the Launchpad team doesn't manage the networking. I don't know of any issues though | 13:13 |
| Guest4913 | cjwatson: thank you for the answer!! i will ask over there if something's going on | 13:15 |
| === sarnold_ is now known as sarnold | ||
| === bdmurray_ is now known as bdmurray | ||
| NIXKnight | Hi | 15:09 |
| NIXKnight | is there a problem with launchpad right nmow | 15:10 |
| NIXKnight | we are having a lot of timeouts | 15:10 |
| NIXKnight | for some repos | 15:11 |
| NIXKnight | that we are pulling from on AWS. | 15:11 |
| === Guest4913 is now known as leo | ||
| === leo is now known as Chiron | ||
| === Chiron is now known as leoc | ||
| cjwatson | NIXKnight: Need to be more specific. Are you talking about ppa.launchpad.net? | 15:16 |
| NIXKnight | cjwatson yes | 15:16 |
| cjwatson | NIXKnight: #canonical-sysadmin manages that - from LP staff's point of view it's just an Apache server and we don't control the networking. Somebody else reported some problems from AWS earlier today as well and filed a ticket | 15:17 |
| adzuci | Thanks cjwatson, I saw the message from https://irclogs.ubuntu.com/2021/08/10/%23launchpad.txt but don't have the history from @canonical-sysadmin, do you have the link to the ticket handy? | 15:25 |
| cjwatson | adzuci: Well, I mean I have https://rt.ubuntu.com/Ticket/Display.html?id=36786 but even I can't see it at the moment | 15:26 |
| cjwatson | So not sure how helpful that is | 15:26 |
| cjwatson | The only reports I've seen so far are from AWS | 15:26 |
| NIXKnight | Thanks cjwatson | 15:28 |
| adzuci | Thanks for the ticket, NIXKnight and I have also only seen this issue in AWS and were wondering if it could be related to rate limiting logic. | 15:28 |
| cjwatson | I'm not aware of rate-limiting at our end | 15:28 |
| cjwatson | Which is not to say there isn't any | 15:29 |
| cjwatson | If it were rate-limiting then you'd normally expect an immediate 503 rather than a timeout, though | 15:29 |
| === lan3y is now known as Laney | ||
| timo_ | Hello all, I'm having issues uploading some files. Both through the web interface and Python launchpadlib. At first glance it might be a filesize issue, a 4.6 tar.gz file uploads fine, while a 36 mb zip and 74 mb dmg return a 502 error. Full error is here: https://pastebin.com/W4MinBRA | 17:08 |
| timo_ | Note that similar files uploaded fine in the past, this is just a version update of my application. Am I hitting a size limit for my project perhaps? | 17:09 |
| cjwatson | timo_: There are no project-specific size limits; I don't immediately know what the problem might be here, and am about to finish up for the day. Could you please file a support ticket using the link in the topic? | 17:10 |
| timo_ | Sure, what's the right place? https://answers.launchpad.net/launchpad ? | 17:11 |
| cjwatson | timo_: Yes | 17:12 |
| cjwatson | I haven't so far found any obvious diagnostics on our side | 17:12 |
| cjwatson | Is it consistently reproducible? | 17:12 |
| timo_ | Yes, over the last 4-5 days. | 17:13 |
| cjwatson | timo_: Details of the URL you're attempting to upload to would likely be helpful (for searching logs) | 17:15 |
| timo_ | Here: https://launchpad.net/pigeonplanner/4.2/4.1.1 | 17:16 |
| cjwatson | 2021-08-10 15:22:48.295Z INFO talisker.wsgi "POST /1.0/pigeonplanner/4.2/4.1.1" method=POST path=/1.0/pigeonplanner/4.2/4.1.1 status=500 duration_ms=29827.591 ip=91.189.89.30 proto=HTTP/1.1 length=14 request_length=36708708 request_type="[...]" forwarded=[...] ua="[...]" logging_count=6 logging_time_ms=0.951 nonpython_actions=1 scheme=https host=api.launchpad.net ... | 17:20 |
| cjwatson | ... request_id=75401959-8d56-42e9-9ff1-27f580f4c5d6 | 17:20 |
| cjwatson | Looks like an application timeout, not yet sure why | 17:20 |
| cjwatson | https://oops.canonical.com/oops/?oopsid=OOPS-0206f5d07d58d3b361bd01f8fb50e617 (Canonical-internal link, just posting here for reference) | 17:22 |
| timo_ | Thanks for checking. | 17:23 |
| cjwatson | https://docs.gunicorn.org/en/stable/settings.html#timeout defaults to 30 seconds and we don't override that. I wonder if we need to bump that so that it's larger than all the per-page hard timeouts we've explicitly set | 17:24 |
| cjwatson | timo_: Could you post the launchpadlib code you're using (feel free to redact anything private), just so that I can make sure I'm looking in the right place at our end? | 17:25 |
| cjwatson | We switched to gunicorn fairly recently, so this may be correlated with that | 17:26 |
| cjwatson | I don't know why small files would be a problem though, that certainly seems backwards | 17:26 |
| cjwatson | Anyway, dinnertime | 17:26 |
| timo_ | Note that the small 4.1mb file uploaded just fine, it's the two others that fail. Also through the web interface, so it's not limited to using launchpadlib. | 17:29 |
| timo_ | It looks like the last time I uploadsed files was a year ago, so recent changes could make sense. | 17:30 |
| timo_ | Anyway, thanks for looking at it last minute. Highly appreciated. | 17:30 |
| === ernstp_ is now known as ernstp | ||
| cjwatson | timo_: Oh, I misread your "4.6 tar.gz" line as "4.6G tar.gz" for some reason | 20:19 |
| cjwatson | No wonder I was confused | 20:20 |
| cjwatson | timo_: (Having that support ticket would still be good though - we have a lot going on and chances are I'll forget otherwise.) | 20:23 |
| timo_ | cjwatson: my bad, I didn't realise I missed the "mb" bit. | 20:57 |
| timo_ | I'll create a ticket tomorrow with all details. Thanks again. | 20:58 |
| === cjwatson changed the topic of #launchpad to: Known issues: bos02-* builders broken | Help contact: cristiangsp (08:00-17:00 UTC) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | ||
| cjwatson | The nova API service in bos02 has fallen over, so we can't reset builder VMs. I'm going to bed shortly but have alerted IS | 21:46 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!