[12:49] <Saviq> wgrant, hey, is there a way to grab all comments I made on merge proposals within a given time frame?
[12:59] <kyrofa> cjwatson, about 80% of my snap builds fail with "Received HTTP code 407 from proxy after CONNECT"
[13:06] <cjwatson> kyrofa: Still today?  We had such reports yesterday, but I thought it might just be load from the 16.04 release in some way.
[13:06] <kyrofa> cjwatson, ah, no, it was yesterday
[13:06] <kyrofa> cjwatson, so that sounds reasonable
[13:07] <kyrofa> cjwatson, I'll come back if it continues today :)
[13:14] <Saviq> launchpadlib FTW
[13:45] <kyrofa> cjwatson, just got it again
[13:48] <cjwatson> kyrofa: links please
[13:48] <kyrofa> cjwatson, https://launchpadlibrarian.net/255181413/buildlog_snap_ubuntu_xenial_armhf_owncloud-next_BUILDING.txt.gz . It isn't the same error message, but still an HTTP code 407
[14:01] <kyrofa> cjwatson, again: https://launchpadlibrarian.net/255185283/buildlog_snap_ubuntu_xenial_armhf_owncloud-next_BUILDING.txt.gz
[14:03] <cjwatson> kyrofa: on a call right now
[14:03] <cjwatson> I will get back to you
[14:03] <kyrofa> Sounds good :)
[15:18]  * cjwatson attempts to clone owncloud-snap onto dogfood in order to try it out there
[15:38] <cjwatson> kyrofa: https://dogfood.launchpadlibrarian.net/245107951/buildlog_snap_ubuntu_xenial_amd64_owncloud_BUILDING.txt.gz may be an artifact of building on dogfood and in any case isn't the error you were asking about, but do you recognise it?
[15:38] <cjwatson> "module 'snapcraft.common' has no attribute 'get_parallel_build_count'
[15:38] <cjwatson> "
[15:39] <kyrofa> cjwatson, looks like it's building an old version of the package
[15:40] <cjwatson> kyrofa: hm, looks like parts/plugins/x-apache.py in your code is calling snapcraft.common.get_parallel_build_count but it's now in snapcraft.internal.common
[15:40] <kyrofa> cjwatson, right, the plugin has changed in more recent code though
[15:40] <cjwatson> kyrofa: it's building the master branch of lp:owncloud-snap
[15:40] <cjwatson> kyrofa: should I build develop instead?
[15:40] <kyrofa> cjwatson, yeah that's old
[15:40] <kyrofa> cjwatson, indeed
[15:40] <kyrofa> cjwatson, master is release-- haven't released the new version yet
[15:41] <cjwatson> ok
[15:42] <kyrofa> cjwatson, yeah I've not been able to get any builds in develop completed (armhf is all I've tested though). Hopefully you'll be able to duplicate with it
[15:43] <cjwatson> right.  I just ideally want to replicate it somewhere where I can just ssh into the proxy and poke around rather than having to go through ops
[15:43] <cjwatson> if I'm unlucky it's somehow load-related and I won't be able to, but we'll see
[16:01] <cjwatson> hmm.  I'm certainly seeing some 407s here, but it seems like git is backing off and retrying, which muddies the waters
[16:02] <kyrofa> cjwatson, yeah load-related would be tough. It wouldn't be time-related, right?
[16:03] <kyrofa> cjwatson, like some proxy token expiring?
[16:03] <oparoz_> cjwatson, I can provide many more logs of the problem. I've even opened a LP issue for it.
[16:03] <cjwatson> kyrofa: when I've seen this in logs so far, it's been followed by successful requests at least sometimes
[16:03] <cjwatson> oparoz_: yes, I saw, thanks
[16:03] <cjwatson> oparoz_: no need for more logs at this time
[16:04] <oparoz_> cjwatson, It does work from time to time, but seems to be pure luck
[16:04] <cjwatson> e.g. I see this kind of pattern:
[16:04] <cjwatson> 10.222.80.42 - SNAPBUILD-341-1461339733 [22/Apr/2016:15:50:15 +0000] "GET http://ftpmaster.internal/ubuntu/pool/main/libx/libxml2/libxml2_2.9.3%2bdfsg1-1_amd64.deb HTTP/1.1" 200 696070 "-" "Debian APT-HTTP/1.3 (1.2.10)" TCP_HIT:HIER_NONE
[16:04] <cjwatson> 10.222.80.42 - - [22/Apr/2016:15:50:17 +0000] "CONNECT github.com:443 HTTP/1.1" 407 2234 "-" "git/2.7.4" TCP_DENIED:HIER_NONE
[16:04] <cjwatson> 10.222.80.42 - SNAPBUILD-341-1461339733 [22/Apr/2016:15:50:36 +0000] "CONNECT github.com:443 HTTP/1.1" 200 28725357 "-" "git/2.7.4" TCP_MISS:HIER_DIRECT
[16:05] <kyrofa> Hmm
[16:06] <cjwatson> I wonder if it's related to the low credentialsttl
[16:06] <cjwatson> We deliberately set a low TTL so that we notice token expiry quickly, but perhaps that's not quite retrying in the right way
[16:11] <kyrofa> cjwatson, is that something ops does, or something you can tweak?
[16:11] <cjwatson> the latter
[16:12] <cjwatson> please don't bother ops about this :)
[16:12] <cjwatson> trying to reproduce with higher debug_options first so that I can see exactly what squid's doing (after these builds have finished, anyway)
[16:12] <kyrofa> cjwatson, oh I wasn't planning on it. I appreciate your poking at it-- please let me know if I can help in any way :)
[16:12] <cjwatson> (well, I can't tweak it directly in production, but the service is maintained by us and ops just deploy what we tell them to)
[17:03] <cjwatson> Oh, of course, 407/200 pairs are perfectly normal because it'll be requesting proxy auth for each request.  hmm.
[17:04]  * cjwatson reminds self how HTTP works