=== marcustomlinson_ is now known as marcustomlinson
=== synapt is now known as nate
=== leo is now known as Guest4913
Guest4913hey folks! just out of curiosity is there something going on with ppa.launchpad.net at the moment?  we are looking at some timeouts starting from 07:00 UTC today connecting to ppa.launchpad.net ( from AWS us-east-1 (Northern Virginia/USA), this seems to be a transient issue and we have assembled a lot of different signals to debug this on our side at this point. happy to put some relevant information wherever it can be useful to y'all, 13:08
cjwatsonGuest4913: #canonical-sysadmin might know better.  From this point of view ppa.launchpad.net is basically just an Apache server and the Launchpad team doesn't manage the networking.  I don't know of any issues though13:13
Guest4913cjwatson: thank you for the answer!! i will ask over there if something's going on13:15
=== sarnold_ is now known as sarnold
=== bdmurray_ is now known as bdmurray
NIXKnightis there a problem with launchpad right nmow15:10
NIXKnightwe are having a lot of timeouts15:10
NIXKnightfor some repos 15:11
NIXKnightthat we are pulling from on AWS.15:11
=== Guest4913 is now known as leo
=== leo is now known as Chiron
=== Chiron is now known as leoc
cjwatsonNIXKnight: Need to be more specific.  Are you talking about ppa.launchpad.net?15:16
NIXKnightcjwatson yes15:16
cjwatsonNIXKnight: #canonical-sysadmin manages that - from LP staff's point of view it's just an Apache server and we don't control the networking.  Somebody else reported some problems from AWS earlier today as well and filed a ticket15:17
adzuciThanks cjwatson, I saw the message from https://irclogs.ubuntu.com/2021/08/10/%23launchpad.txt but don't have the history from @canonical-sysadmin, do you have the link to the ticket handy?15:25
cjwatsonadzuci: Well, I mean I have https://rt.ubuntu.com/Ticket/Display.html?id=36786 but even I can't see it at the moment15:26
cjwatsonSo not sure how helpful that is15:26
cjwatsonThe only reports I've seen so far are from AWS15:26
NIXKnightThanks cjwatson15:28
adzuciThanks for the ticket, NIXKnight and I have also only seen this issue in AWS and were wondering if it could be related to rate limiting logic.15:28
cjwatsonI'm not aware of rate-limiting at our end15:28
cjwatsonWhich is not to say there isn't any15:29
cjwatsonIf it were rate-limiting then you'd normally expect an immediate 503 rather than a timeout, though15:29
=== lan3y is now known as Laney
timo_Hello all, I'm having issues uploading some files. Both through the web interface and Python launchpadlib. At first glance it might be a filesize issue, a 4.6 tar.gz file uploads fine, while a 36 mb zip and 74 mb dmg return a 502 error. Full error is here: https://pastebin.com/W4MinBRA17:08
timo_Note that similar files uploaded fine in the past, this is just a version update of my application. Am I hitting a size limit for my project perhaps?17:09
cjwatsontimo_: There are no project-specific size limits; I don't immediately know what the problem might be here, and am about to finish up for the day.  Could you please file a support ticket using the link in the topic?17:10
timo_Sure, what's the right place? https://answers.launchpad.net/launchpad ?17:11
cjwatsontimo_: Yes17:12
cjwatsonI haven't so far found any obvious diagnostics on our side17:12
cjwatsonIs it consistently reproducible?17:12
timo_Yes, over the last 4-5 days.17:13
cjwatsontimo_: Details of the URL you're attempting to upload to would likely be helpful (for searching logs)17:15
timo_Here: https://launchpad.net/pigeonplanner/4.2/4.1.117:16
cjwatson2021-08-10 15:22:48.295Z INFO talisker.wsgi "POST /1.0/pigeonplanner/4.2/4.1.1" method=POST path=/1.0/pigeonplanner/4.2/4.1.1 status=500 duration_ms=29827.591 ip= proto=HTTP/1.1 length=14 request_length=36708708 request_type="[...]" forwarded=[...] ua="[...]" logging_count=6 logging_time_ms=0.951 nonpython_actions=1 scheme=https host=api.launchpad.net ...17:20
cjwatson... request_id=75401959-8d56-42e9-9ff1-27f580f4c5d617:20
cjwatsonLooks like an application timeout, not yet sure why17:20
cjwatsonhttps://oops.canonical.com/oops/?oopsid=OOPS-0206f5d07d58d3b361bd01f8fb50e617 (Canonical-internal link, just posting here for reference)17:22
timo_Thanks for checking.17:23
cjwatsonhttps://docs.gunicorn.org/en/stable/settings.html#timeout defaults to 30 seconds and we don't override that.  I wonder if we need to bump that so that it's larger than all the per-page hard timeouts we've explicitly set17:24
cjwatsontimo_: Could you post the launchpadlib code you're using (feel free to redact anything private), just so that I can make sure I'm looking in the right place at our end?17:25
cjwatsonWe switched to gunicorn fairly recently, so this may be correlated with that17:26
cjwatsonI don't know why small files would be a problem though, that certainly seems backwards17:26
cjwatsonAnyway, dinnertime17:26
timo_Note that the small 4.1mb file uploaded just fine, it's the two others that fail. Also through the web interface, so it's not limited to using launchpadlib.17:29
timo_It looks like the last time I uploadsed files was a year ago, so recent changes could make sense.17:30
timo_Anyway, thanks for looking at it last minute. Highly appreciated.17:30
=== ernstp_ is now known as ernstp
cjwatsontimo_: Oh, I misread your "4.6 tar.gz" line as "4.6G tar.gz" for some reason20:19
cjwatsonNo wonder I was confused20:20
cjwatsontimo_: (Having that support ticket would still be good though - we have a lot going on and chances are I'll forget otherwise.)20:23
timo_cjwatson: my bad, I didn't realise I missed the "mb" bit.20:57
timo_I'll create a ticket tomorrow with all details. Thanks again.20:58
=== cjwatson changed the topic of #launchpad to: Known issues: bos02-* builders broken | Help contact: cristiangsp (08:00-17:00 UTC) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad
cjwatsonThe nova API service in bos02 has fallen over, so we can't reset builder VMs.  I'm going to bed shortly but have alerted IS21:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!