=== marcustomlinson_ is now known as marcustomlinson === synapt is now known as nate === leo is now known as Guest4913 [13:08] hey folks! just out of curiosity is there something going on with ppa.launchpad.net at the moment? we are looking at some timeouts starting from 07:00 UTC today connecting to ppa.launchpad.net (91.189.95.85) from AWS us-east-1 (Northern Virginia/USA), this seems to be a transient issue and we have assembled a lot of different signals to debug this on our side at this point. happy to put some relevant information wherever it can be useful to y'all, [13:13] Guest4913: #canonical-sysadmin might know better. From this point of view ppa.launchpad.net is basically just an Apache server and the Launchpad team doesn't manage the networking. I don't know of any issues though [13:15] cjwatson: thank you for the answer!! i will ask over there if something's going on === sarnold_ is now known as sarnold === bdmurray_ is now known as bdmurray [15:09] Hi [15:10] is there a problem with launchpad right nmow [15:10] we are having a lot of timeouts [15:11] for some repos [15:11] that we are pulling from on AWS. === Guest4913 is now known as leo === leo is now known as Chiron === Chiron is now known as leoc [15:16] NIXKnight: Need to be more specific. Are you talking about ppa.launchpad.net? [15:16] cjwatson yes [15:17] NIXKnight: #canonical-sysadmin manages that - from LP staff's point of view it's just an Apache server and we don't control the networking. Somebody else reported some problems from AWS earlier today as well and filed a ticket [15:25] Thanks cjwatson, I saw the message from https://irclogs.ubuntu.com/2021/08/10/%23launchpad.txt but don't have the history from @canonical-sysadmin, do you have the link to the ticket handy? [15:26] adzuci: Well, I mean I have https://rt.ubuntu.com/Ticket/Display.html?id=36786 but even I can't see it at the moment [15:26] So not sure how helpful that is [15:26] The only reports I've seen so far are from AWS [15:28] Thanks cjwatson [15:28] Thanks for the ticket, NIXKnight and I have also only seen this issue in AWS and were wondering if it could be related to rate limiting logic. [15:28] I'm not aware of rate-limiting at our end [15:29] Which is not to say there isn't any [15:29] If it were rate-limiting then you'd normally expect an immediate 503 rather than a timeout, though === lan3y is now known as Laney [17:08] Hello all, I'm having issues uploading some files. Both through the web interface and Python launchpadlib. At first glance it might be a filesize issue, a 4.6 tar.gz file uploads fine, while a 36 mb zip and 74 mb dmg return a 502 error. Full error is here: https://pastebin.com/W4MinBRA [17:09] Note that similar files uploaded fine in the past, this is just a version update of my application. Am I hitting a size limit for my project perhaps? [17:10] timo_: There are no project-specific size limits; I don't immediately know what the problem might be here, and am about to finish up for the day. Could you please file a support ticket using the link in the topic? [17:11] Sure, what's the right place? https://answers.launchpad.net/launchpad ? [17:12] timo_: Yes [17:12] I haven't so far found any obvious diagnostics on our side [17:12] Is it consistently reproducible? [17:13] Yes, over the last 4-5 days. [17:15] timo_: Details of the URL you're attempting to upload to would likely be helpful (for searching logs) [17:16] Here: https://launchpad.net/pigeonplanner/4.2/4.1.1 [17:20] 2021-08-10 15:22:48.295Z INFO talisker.wsgi "POST /1.0/pigeonplanner/4.2/4.1.1" method=POST path=/1.0/pigeonplanner/4.2/4.1.1 status=500 duration_ms=29827.591 ip=91.189.89.30 proto=HTTP/1.1 length=14 request_length=36708708 request_type="[...]" forwarded=[...] ua="[...]" logging_count=6 logging_time_ms=0.951 nonpython_actions=1 scheme=https host=api.launchpad.net ... [17:20] ... request_id=75401959-8d56-42e9-9ff1-27f580f4c5d6 [17:20] Looks like an application timeout, not yet sure why [17:22] https://oops.canonical.com/oops/?oopsid=OOPS-0206f5d07d58d3b361bd01f8fb50e617 (Canonical-internal link, just posting here for reference) [17:23] Thanks for checking. [17:24] https://docs.gunicorn.org/en/stable/settings.html#timeout defaults to 30 seconds and we don't override that. I wonder if we need to bump that so that it's larger than all the per-page hard timeouts we've explicitly set [17:25] timo_: Could you post the launchpadlib code you're using (feel free to redact anything private), just so that I can make sure I'm looking in the right place at our end? [17:26] We switched to gunicorn fairly recently, so this may be correlated with that [17:26] I don't know why small files would be a problem though, that certainly seems backwards [17:26] Anyway, dinnertime [17:29] Note that the small 4.1mb file uploaded just fine, it's the two others that fail. Also through the web interface, so it's not limited to using launchpadlib. [17:30] It looks like the last time I uploadsed files was a year ago, so recent changes could make sense. [17:30] Anyway, thanks for looking at it last minute. Highly appreciated. === ernstp_ is now known as ernstp [20:19] timo_: Oh, I misread your "4.6 tar.gz" line as "4.6G tar.gz" for some reason [20:20] No wonder I was confused [20:23] timo_: (Having that support ticket would still be good though - we have a lot going on and chances are I'll forget otherwise.) [20:57] cjwatson: my bad, I didn't realise I missed the "mb" bit. [20:58] I'll create a ticket tomorrow with all details. Thanks again. === cjwatson changed the topic of #launchpad to: Known issues: bos02-* builders broken | Help contact: cristiangsp (08:00-17:00 UTC) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad [21:46] The nova API service in bos02 has fallen over, so we can't reset builder VMs. I'm going to bed shortly but have alerted IS