=== rvba` is now known as rvba [12:49] wgrant, hey, is there a way to grab all comments I made on merge proposals within a given time frame? [12:59] cjwatson, about 80% of my snap builds fail with "Received HTTP code 407 from proxy after CONNECT" [13:06] kyrofa: Still today? We had such reports yesterday, but I thought it might just be load from the 16.04 release in some way. [13:06] cjwatson, ah, no, it was yesterday [13:06] cjwatson, so that sounds reasonable [13:07] cjwatson, I'll come back if it continues today :) [13:14] launchpadlib FTW [13:45] cjwatson, just got it again [13:48] kyrofa: links please [13:48] cjwatson, https://launchpadlibrarian.net/255181413/buildlog_snap_ubuntu_xenial_armhf_owncloud-next_BUILDING.txt.gz . It isn't the same error message, but still an HTTP code 407 === cmars` is now known as cmars [14:01] cjwatson, again: https://launchpadlibrarian.net/255185283/buildlog_snap_ubuntu_xenial_armhf_owncloud-next_BUILDING.txt.gz [14:03] kyrofa: on a call right now [14:03] I will get back to you [14:03] Sounds good :) [15:18] * cjwatson attempts to clone owncloud-snap onto dogfood in order to try it out there [15:38] kyrofa: https://dogfood.launchpadlibrarian.net/245107951/buildlog_snap_ubuntu_xenial_amd64_owncloud_BUILDING.txt.gz may be an artifact of building on dogfood and in any case isn't the error you were asking about, but do you recognise it? [15:38] "module 'snapcraft.common' has no attribute 'get_parallel_build_count' [15:38] " [15:39] cjwatson, looks like it's building an old version of the package [15:40] kyrofa: hm, looks like parts/plugins/x-apache.py in your code is calling snapcraft.common.get_parallel_build_count but it's now in snapcraft.internal.common [15:40] cjwatson, right, the plugin has changed in more recent code though [15:40] kyrofa: it's building the master branch of lp:owncloud-snap [15:40] kyrofa: should I build develop instead? [15:40] cjwatson, yeah that's old [15:40] cjwatson, indeed [15:40] cjwatson, master is release-- haven't released the new version yet [15:41] ok [15:42] cjwatson, yeah I've not been able to get any builds in develop completed (armhf is all I've tested though). Hopefully you'll be able to duplicate with it [15:43] right. I just ideally want to replicate it somewhere where I can just ssh into the proxy and poke around rather than having to go through ops [15:43] if I'm unlucky it's somehow load-related and I won't be able to, but we'll see [16:01] hmm. I'm certainly seeing some 407s here, but it seems like git is backing off and retrying, which muddies the waters [16:02] cjwatson, yeah load-related would be tough. It wouldn't be time-related, right? [16:03] cjwatson, like some proxy token expiring? [16:03] cjwatson, I can provide many more logs of the problem. I've even opened a LP issue for it. [16:03] kyrofa: when I've seen this in logs so far, it's been followed by successful requests at least sometimes [16:03] oparoz_: yes, I saw, thanks [16:03] oparoz_: no need for more logs at this time [16:04] cjwatson, It does work from time to time, but seems to be pure luck [16:04] e.g. I see this kind of pattern: [16:04] 10.222.80.42 - SNAPBUILD-341-1461339733 [22/Apr/2016:15:50:15 +0000] "GET http://ftpmaster.internal/ubuntu/pool/main/libx/libxml2/libxml2_2.9.3%2bdfsg1-1_amd64.deb HTTP/1.1" 200 696070 "-" "Debian APT-HTTP/1.3 (1.2.10)" TCP_HIT:HIER_NONE [16:04] 10.222.80.42 - - [22/Apr/2016:15:50:17 +0000] "CONNECT github.com:443 HTTP/1.1" 407 2234 "-" "git/2.7.4" TCP_DENIED:HIER_NONE [16:04] 10.222.80.42 - SNAPBUILD-341-1461339733 [22/Apr/2016:15:50:36 +0000] "CONNECT github.com:443 HTTP/1.1" 200 28725357 "-" "git/2.7.4" TCP_MISS:HIER_DIRECT [16:05] Hmm [16:06] I wonder if it's related to the low credentialsttl [16:06] We deliberately set a low TTL so that we notice token expiry quickly, but perhaps that's not quite retrying in the right way [16:11] cjwatson, is that something ops does, or something you can tweak? [16:11] the latter [16:12] please don't bother ops about this :) [16:12] trying to reproduce with higher debug_options first so that I can see exactly what squid's doing (after these builds have finished, anyway) [16:12] cjwatson, oh I wasn't planning on it. I appreciate your poking at it-- please let me know if I can help in any way :) [16:12] (well, I can't tweak it directly in production, but the service is maintained by us and ops just deploy what we tell them to) [17:03] Oh, of course, 407/200 pairs are perfectly normal because it'll be requesting proxy auth for each request. hmm. [17:04] * cjwatson reminds self how HTTP works