[00:42] <tsimonq2> aha! that's what's been going on... :)
[00:42] <tsimonq2> wxl: do you have access to https://twitter.com/launchpadstatus ?
[00:43] <tsimonq2> whoops
[00:43] <wgrant> I'm getting there.
[00:43] <wgrant> Lots of other things to do.
[00:43] <tsimonq2> meant wgrant :)
[00:43] <tsimonq2> alright :)
[00:43] <c_smith> hey, is something going on with Launchpad?
[00:43] <tsimonq2> wxl: sorry, accident :)
[00:43] <tsimonq2> c_smith: topic :)
[00:43] <c_smith> ah, I see. didn't notice. XD
[00:44] <c_smith> and wxl is here, eh? nice to see some fellow Oregonians here.
[00:44] <tsimonq2> lol yeah I'm pretty good friends with him FWIW :)
[00:44] <c_smith> don't know him all that well myself, only real contact I've had with him is through the LoCo
[00:45] <tsimonq2> I see :)
[00:46] <c_smith> now if only I could figure out why cairo dock flat out refuses to load the notification area in Arch...
[19:49] <oparoz> Hello, is there a snap package builder status page to know when to avoid requesting builds?
[19:50] <dobey> https://launchpad.net/builders ?
[19:52] <oparoz> PErfect, thanks dobey :)
[19:54] <oparoz> And where do we report issues?
[19:55] <oparoz> When pulling a dependency: EOF occurred in violation of protocol (_ssl.c:645)
[19:57] <dobey> for snappy builds? i presume bugs against launchpad itself (https://launchpad.net/launchpad)
[19:57] <oparoz> Thanks you dobey
[20:00] <oparoz> https://bugs.launchpad.net/launchpad/+bug/1569023
[20:00] <ubot5`> Launchpad bug 1569023 in Launchpad itself "Snap builder fails to pull files (violation of protocol)" [Undecided,New]
[20:15] <cjwatson> oparoz: Is this repeatable?
[20:15] <cjwatson> It looks potentially transient to me
[20:16] <oparoz> cjwatson: yes. Sometimes it fails elsewhere, when doing a git clone
[20:16] <oparoz> cjwatson: This new build seems stuck: https://code.launchpad.net/~olivier-interfasys/+snap/owncloud-snap/+build/535
[20:17] <cjwatson> OK, I've heard others muttering about the snap proxy being not totally reliable, so perhaps the former problem is that
[20:17] <oparoz> cjwatson: Looksl ike some networking issue, but the dashboard shows everything as green
[20:17] <cjwatson> NTP> wtf
[20:17] <cjwatson> /builders won't tell you about this sort of thing
[20:18] <cjwatson> oparoz: I would suggest cancelling that one and re-requesting
[20:19] <cjwatson> we might get a more complete log after cancellation
[20:19] <cjwatson> let me try cancelling it, in fact
[20:20] <oparoz> cjwatson: OK, I'll let you do it
[20:20] <cjwatson> mm, this isn't cancelling very quickly, so I suspect that the VM crashed early on
[20:22] <oparoz> So, I 've tried 4 build and it used 3 different VMs, all failed at some point because of the network
[20:22] <cjwatson> I think you've misdiagnosed the last one as a network issue
[20:23] <cjwatson> looks much more like catastrophic guest crash to me
[20:23] <cjwatson> yeah, after cancellation we didn't get a build log out of it, that generally indicates that the VM was toast
[20:23] <cjwatson> try once more for luck?
[20:24] <cjwatson> the builder name basically doesn't matter, it's a fresh VM every time
[20:24] <cjwatson> so you actually got 4 different VMs
[20:25] <cjwatson> https://launchpadlibrarian.net/253330034/buildlog_snap_ubuntu_xenial_armhf_owncloud-snap_BUILDING.txt.gz is your problem to fix, not a network problem
[20:25] <cjwatson> you are not permitted network access after the pull phase has completed
[20:25] <cjwatson> hm, wait, no
[20:25] <cjwatson> ignore that last :)
[20:26] <oparoz> cjwatson: :)
[20:26] <cjwatson> two of them seem likely to be proxy bugs
[20:27] <oparoz> Is it all from the same machine or do these hosts belong to a cluster?
[20:27] <cjwatson> cluster
[20:27] <cjwatson> or rather, a private cloud
[20:27] <cjwatson> it's an openstack cloud running on a pile of HP Proliant m400 cartridges plugged into a single chassis
[20:28] <oparoz> Nice pile ;)
[20:28] <oparoz> The new build failed again, since it all fails at the same step, I suspect a proxy problem bwtween that cloud and Sourceforge
[20:28] <cjwatson> the git.l.n resolution failure is odd, I would suspect that of being a problem with the infrastructure since there's some slightly exotic stuff going on in this cloud region to avoid long-fat-pipe latency problems
[20:29] <cjwatson> oparoz: what's the URL it's trying to pull there?
[20:29] <oparoz> The URL given as a source is HTTP, maybe it doesn't like that
[20:29] <oparoz> source: http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz
[20:29] <cjwatson> HTTP should work
[20:30] <cjwatson> SF redirects it to HTTPS anyway
[20:30] <oparoz> When I tried to get the file using wget, I do get lots of redirections
[20:30] <oparoz> 301, 302, 302, 302, 200
[20:42] <cjwatson> I really must arrange to get access to the production squid logs, this is difficult to debug otherwise
[20:45]  * cjwatson adds a card for that
[20:45] <oparoz> cjwatson: I've launched a new build with another URL to see if it helps. Seems stuck, but maybe it's still downloading the 80MB file
[20:47] <cjwatson> Silly question but have you tried it locally?
[20:47] <oparoz> cjwatson: Not recently. I was afraid it would rebuild MySQL...
[20:47] <cjwatson> I'm trying it here but my ADSL is very slow
[20:47] <oparoz> But I will if it fails with the new URL
[20:50] <oparoz> cjwatson: Just tried and no problem to pull the file
[20:54] <cjwatson> oparoz: OK.  I walked five miles this morning carrying a cat back and forward to the vet so I'm pretty tired, I'll pick this up again tomorrow if none of my colleagues beat me to it
[20:55] <oparoz> No worries, thank you cjwatson
[20:58] <cjwatson> and that one got 504 (gateway timeout).  definitely need to have a look at squid logs
[20:59] <cjwatson> hmm
[21:00] <cjwatson> from a staging instance: http://paste.ubuntu.com/15767707/
[21:01] <oparoz> cjwatson: Locally it bounces to a non-SSL URL
[21:01] <cjwatson> With --debug I see it bouncing around a bit and then hanging
[21:02] <cjwatson> or sometimes just failing earlier
[21:10] <cjwatson> oparoz: looks like network-level trouble, I've punted to sysadmin
[21:10] <oparoz> Thank you cjwatson!
[23:04] <SpamapS> o/ ... login.launchpad.net is still up/down ... can't use my launchpad openid today. Anybody else experiencing that?
[23:18] <wgrant> SpamapS: Can you explain the problem?
[23:21] <SpamapS> wgrant: I'm using openid from launchpad, and from time to time login.launchpad.net times out.
[23:21] <SpamapS> wgrant: it has also worked a few times today, so it's not 100% down.
[23:21] <SpamapS> wgrant: also o/
[23:22] <wgrant> SpamapS: So the problem you're seeing is a TCP connection timeout?
[23:23] <SpamapS> wgrant: I can't 100% say for sure. My browser never shows login.launchpad.net, it just shows me the openid consumer's redirect page.
[23:23] <SpamapS> wgrant: but then a refresh sometimes does work.
[23:25] <SpamapS> "The server at login.launchpad.net is taking too long to respond."
[23:26] <SpamapS> wgrant: it is working perfectly now.. so it's entirely possible the problem was internet storms. ;)
[23:27] <wgrant> Right, but it's a network-level timeout, not an application-level one.
[23:27] <wgrant> Thanks.
[23:46] <wgrant> SpamapS: Which network are you connecting from?
[23:46] <wgrant> We're seeing weird packet loss to particular external networks, so Internet storms seem reasonable.
[23:56] <SpamapS> wgrant: http://paste.ubuntu.com/15769308/
[23:58] <wgrant> SpamapS: Thanks. Any additional data is valuable, as everything's quite confusing.