[04:41] <haseeb> hi
[04:42] <haseeb> i am trying to load the current trunk of gtg, but getting error as
[04:42] <haseeb> [haseeb@localhost ~]$ bzr launchpad-login haseeb
[04:42] <haseeb> [haseeb@localhost ~]$ bzr branch lp:gtg
[04:42] <haseeb> Permission denied (publickey).
[04:42] <haseeb> bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
[04:42] <StevenK> haseeb: It looks like Launchpad doesn't know about your SSH key.
[04:43] <haseeb> StevenK,  i have uploaded the key on launchpad
[07:33] <stewart> known current issue? "bzr: ERROR: Invalid http response for http://bazaar.launchpad.net/%2Bbranch-id/41661/.bzr/repository/packs/d12c3b2f38af0cdab110a25c1f50dd7d.pack: Unable to handle http code 503: Service Temporarily Unavailable"
[07:35] <wgrant> stewart: Yeah, we're working on it. bzr+ssh should be working fine.
[07:36] <stewart> wgrant, yeah, but my jenkins slaves don't do bzr+ssh :(
[07:42] <wgrant> stewart: It's been stable for a few minutes now.
[07:42] <stewart> wgrant, great, i'll give it a go :)
[10:03] <l3on> Hi all... does somebody know how much data (in term of MBytes) I can upload in a team branch ?
[10:05] <wgrant> l3on: There's no hard limit.
[10:07] <czajkowski> l3on: I just explained this to Gwaihir :)
[10:08] <l3on> czajkowski, wgrant: thank you! :
[10:08] <l3on> :)
[11:09] <fabo> hello, the merge proposal https://code.launchpad.net/~deeptik/linaro-license-protection/publish-to-snapshots/+register-merge never succeed
[11:10] <fabo> is it a known issue?
[11:15] <wgrant> fabo: Sounds like it might be bug #961126. That causes the page to appear to hang if there's a validation error on the form.
[11:15] <wgrant> Are you sure you specified eg. the reviewer correctly?
[11:17] <fabo> wgrant: reviewer is optional. I try to set the default one
[11:17] <fabo> same result
[11:18] <fabo> wgrant: yes, seems the same issue.
[11:25] <wgrant> fabo: "There is already a branch merge proposal registered for branch lp:~deeptik/linaro-license-protection/publish-to-snapshots to land on lp:linaro-license-protection that is still active."
[11:25] <wgrant> fabo: That's the error.
[11:27] <fabo> it explains
[11:30] <fabo> wgrant: thanks
[12:12] <ugoc> Hi
[12:13] <ugoc> is there a blacklist for accessing bzr.launchpad.net?
[12:13] <ugoc> Since this morning we can't get to our branches on launchpad servers :S
[12:14] <jelmer> hi ugoc
[12:14] <jelmer> ugoc: I think I saw reports of a host hitting our connection limits this morning
[12:14] <jelmer> wgrant: still there?
[12:14] <ugoc> ho
[12:15] <wgrant> jelmer: Hi
[12:15] <jelmer> wgrant: ^
[12:15] <ugoc> jelmer: do you have it's ip?
[12:15] <wgrant> ugoc: You were making about 1000 requests over 10 seconds every 5 minutes, which seems slightly abnormal.
[12:15] <wgrant> It was you :)
[12:15] <ugoc> Are you sure?
[12:16] <wgrant> They were ~shadowrobot branches, and that's a team owned by you, so it seems likely.
[12:16] <ugoc> yup indeed
[12:16] <ugoc> damn
[12:16] <wgrant> ugoc: So, can you scale that robot back a bit?
[12:16] <ugoc> damn.... We're setting up a test server to test our branches :S May be something went wrong there
[12:16] <wgrant> It's checking lots of branches at the same moment, every 5 minutes.
[12:17] <markpitchless> hi wgrant (it was me setting up the test server)
[12:17] <markpitchless> it could be but surprising it was making that many requests
[12:18] <wgrant>    1033 2012-03-26 09:01
[12:18] <wgrant>    1033 2012-03-26 09:06
[12:18] <wgrant>    1033 2012-03-26 09:11
[12:18] <wgrant> HTTP requests to ~shadowrobot branches from that IP, by minute
[12:18] <wgrant> And the requests are over ~10s in each of those minutes.
[12:19] <ugoc> If we disable our test server for now
[12:19] <wgrant> We can unblock you immediately, sure.
[12:19] <markpitchless> wow! I'll scale the checks back to once per hour. would have still only expected about 100 requests each check though
[12:19] <markpitchless> ok, will stop all the jobs
[12:19] <ugoc> Ok great.
[12:20] <wgrant> What command are you running? bzr info?
[12:20] <markpitchless> not sure. it is jenkins
[12:20] <markpitchless> what ever its bzr plugin uses
[12:20] <wgrant> Ah
[12:21] <wgrant> 'cause it's checking for some pretty strange stuff.
[12:22] <markpitchless> can you mail me some logs and I'll investigate? It is only supposed to checking if the repo has changed or not.
[12:22] <wgrant> markpitchless: So, scaling back is good. If you can stagger the checks to not be 100 requests per second, even better.
[12:22] <wgrant> Sure, let me see.
[12:23] <markpitchless> wgrant: will do
[12:27] <markpitchless> wgrant: i have disabled all our jenkins jobs, so we will be back to normal, human generated, traffic levels. can you unblock us please?
[12:29] <wgrant> markpitchless: A sysadmin is on it. Should be bac kshortly.
[12:30] <markpitchless> wgrant: many thanks. sorry for the hassle.
[12:31] <ugoc> wgrant: thanks a lot, we'll probably setup an internal repo for our test server...
[12:32] <gnuoy`> wgrant, markpitchless: the ip is unblocked
[12:32] <wgrant> markpitchless: I've just emailed you a minute of the relevant logs.
[12:32] <wgrant> gnuoy`: Thanks.
[12:32] <gnuoy`> np
[12:32] <markpitchless> cheers
[12:33] <wgrant> markpitchless: I'll probably look at the jenkins-bazaar plugin tomorrow and see if there's anything obvious that can be done.
[12:33] <wgrant> Unless jelmer beats me to it.
[12:36] <markpitchless> wgrant: Cheers. Too many requests was partly my fault, used a script to generate 130 jobs, which were checking every 5mins. Plus 2 active builds that would also do a branch. But thats doesn't add up to 1000.
[12:39] <Laney> looks like caph (the buildd) is busted
[12:39] <Laney> https://launchpad.net/ubuntu/+source/haskell-crypto-conduit/0.1.3.1-1/+build/3319209
[12:40] <Laney> oh, maybe not, ...
[12:41] <wgrant> Laney: Yeah, the chroot file in caph's cache seemed to be randomly corrupted (correct length, incorrect content). We forced it to refetch and it's happy now.
[12:41] <Laney> wgrant: good news, thanks
[12:42] <wgrant> Any builds that were hit will need to be manually retried, though.
[12:49] <Laney> hrm, they don't show up as build failures on +synchronised-packages
[13:03] <Laney> wgrant: are you aware that it's happening on other buildds too? (chort)
[13:07] <yann_> hello! are there problems on launchpad today?
[13:07] <yann_> getting 503 on 503
[13:10] <yann_> trying to access that page http://bazaar.launchpad.net/~yannh/kt-replication-plugin/trunk/revision/5
[13:15] <czajkowski> yann_: I can access it
[13:15] <czajkowski> granted it's a bit slow but it does resolve
[15:02] <Laney> perhaps someone else can take a look at (ping the ops about) chort?
[15:02] <czajkowski> Laney: link ?
[15:03] <Laney> czajkowski: https://launchpad.net/builders/chort/+history
[15:03] <Laney> sounds like they did the same fix for caph a while back
[15:04] <Laney> i just gave back my builds until another buildd got them
[15:04] <Laney> which is why you don't see them on that list
[15:12] <czajkowski> Laney: being looked into
[15:12] <Laney> okies
[15:35] <czajkowski> Laney: can you explain to thedac what the issue is exactly please.
[15:36] <Laney> I don't know what it is, just that the chroot on chort is broken in the same way that it was on caph. wgrant seemed to know about it earlier.
[15:36] <Laney> I guess you can fix it by getting it to refetch its chroot
[15:46] <czajkowski> Laney: it's been cleaned out and refreshed so we'll wait and see
[15:46] <Laney> cool! good work chaps
[15:46] <czajkowski> :)