[00:01] <arand> When I'm running a script from hydrazine which wants to authorize access I'm getting "Please input your password for the keyring" which password does this refer to?
[00:02] <arand> (after I've authorized access via the web browser)
[00:21] <arand> ^ Hmm, it appears I worked ym way around it by installing python-gnomekeyring...
[02:44] <tsmith> I cannot for the life of me figure out how to create a release for a project of mine!! I can't eeven find the place to create milestones! I've googled for about 3 hours now. Please please please someone help.
[02:48] <lifeless> tsmith: whats the project?
[02:49] <wgrant> tsmith: You create milestones and releases inside a series of your project. https://help.launchpad.net/Projects/SeriesMilestonesReleases might be a helpful overview of the model.
[02:49] <tsmith> https://launchpad.net/phpu-training
[02:49] <wgrant> (if you don't know what a series is, you probably want trunk)
[02:49] <tsmith> i want the release off of trunk.
[02:49] <lifeless> tsmith: click on 'trunk series  ' on the left hand side of https://launchpad.net/phpu-training
[02:50] <lifeless> tsmith: then click on 'create release' on the mid-right side of the page.
[02:50] <tsmith> oh oh my god
[02:50] <tsmith> lifeless, thank you so very very very much.
[02:51] <tsmith> lifeless, what is a milestone, exactly?
[02:51] <tsmith> LIke the reason I'm releasing this is b/c i finished the multiauth library..
[02:51] <tsmith> is that the milestone?
[02:52] <lifeless> a milestone is an arbitrary marker
[02:52] <lifeless> project downloads hang off of releases, which hang off of milestones.
[02:53] <tsmith> do milestone names have to be unique across all of launchpad?
[02:53] <lifeless> they have to be unique in your project
[02:54] <tsmith> lifeless, you're much much much better than the existing documentation (http://blog.launchpad.net/coming-features/linking-project-releases-in-launchpad-to-milestones)
[07:57] <IlikeMoose> how long after submitting a bug to launchpad should you expect to see a response email containing the number of the bug if it has been accepted?
[08:34] <czajkowski> IlikeMoose: straight away
[08:44] <mgedmin> any ideas why 'curl https://bugs.launchpad.net/bugs/1031642/+attachment/3244425/+files/logs2html.py.diff' produces no output?
[08:44] <mgedmin> wget -O - on the same URL works fine
[08:45] <wgrant> mgedmin: curl doesn't follow redirects by default, probably.
[08:45] <mgedmin> oh, really? wow
[08:45] <Peng> Yeah, curl -v looks like it isn't following the redirect.
[08:45] <wgrant> The -L option may be handy
[08:46] <mgedmin> yeah, curl -L works
[08:47] <mgedmin> any reason why launchpad doesn't follow the HTTP recommendation of returning a human-readable body content that explains the redirect and mentions the new location??
[08:47] <wgrant> Probably because every other client in the world follows redirects by default :)
[08:48] <wgrant> But we could.
[08:49] <mgedmin> but, wow, not following redirects by default?  wow, curl, you amaze me
[08:49]  * mgedmin headdesks
[08:49] <wgrant> Heh
[08:52] <jml> hi. https://code.launchpad.net/~james-w/pkgme-service/log-downloads/+merge/117819 has been up for 8 hours with no diff.
[08:53] <wgrant> jml: You'll see that the branch failed to scan.
[08:53] <jml> wgrant: Will I?
[08:54] <mgedmin> in case anybody cares: "Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s)." -- http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3
[08:54] <wgrant> jml: Note the lack of revisions on https://code.launchpad.net/~james-w/pkgme-service/log-downloads
[08:54] <mgedmin> personally, I feel curl is more in the wrong here
[08:55] <wgrant> [2012-08-01 23:50:45,994: INFO/PoolWorker-2] Retrieving history from bzrlib.
[08:55] <wgrant> [2012-08-01 23:50:46,036: INFO/PoolWorker-2] Job resulted in OOPS: OOPS-ed69e1bac8822e89d44b42851ddd520a
[08:56] <wgrant> Bad bot.
[08:56] <wgrant> jml: That branch looks pretty broken.
[08:56] <jml> wgrant: works just fine for me via bzr
[08:56] <wgrant> RevisionNotPresent: Revision {ubuntu@server-13843-20120726154335-qbvhsrci6g53odpe} not present in "Graph(StackedParentsProvider(bzrlib.repository._LazyListJoin(([CachingParentsProvider(CallableToParentsProviderAdapter(<bound method CHKInventoryRepository._get_parent_map_no_fallbacks of CHKInventoryRepository('lp-internal:///~james-w/pkgme-service/log-downloads/.bzr/repository')>))], []))))".
[08:58] <wgrant> Um
[08:58] <wgrant> http://paste.ubuntu.com/1124876/
[08:59] <wgrant> revno: 1
[08:59] <wgrant> modified: src/djpkgme/tasks.py
[08:59] <wgrant> Something is a little off.
[08:59] <wgrant> jml: How does it look for you?
[08:59] <jml> wgrant: http://paste.ubuntu.com/1124878/
[09:00] <wgrant> last-revision looks correct...
[09:00] <wgrant> Right, that's a bit more plausible.
[09:00] <wgrant> But you cheated by using an existing repo :)
[09:02] <wgrant> jml: bzr check on the lp: URL crashes
[09:02] <jml> wgrant: ah ok.
[09:03] <wgrant> wgrant@lamuella:/tmp$ bzr branch lp:~james-w/pkgme-service/log-downloads
[09:03] <wgrant> bzr: ERROR: No such file: 'homepage.html-20120313222053-nn7riwtsskyte0l2-1'
[10:05] <om26er> hey! wasn't the dowtime supposed to start at 11am UTC ?
[10:06] <om26er> i see this http://imagebin.org/index.php?mode=image&id=223007
[10:07] <mgz> om26er: this is the normal fastdowntime... plus some scrambling
[10:08] <om26er> mgz, when is it expected to get back ?
[10:08] <wgrant> 7 minutes ago :)
[10:10] <om26er> ah working now :)
[10:14] <wgrant> om26er: Yeah, sorry, a bit of a glitch with the DB patching process.
[10:14] <wgrant> Something caused replication lag during the patch, which glued everything up.
[10:15] <om26er> wgrant, all fixed now ;-)
[10:15] <wgrant> Yep
[10:15] <wgrant> Note that we still have PPA build downtime at 11UTC
[13:22] <cjohnston> Outside of events such as UDS, is 'participation essential' for blueprints really used for anything/capable of anything, or is it no different than subscribing to a blueprint?
[13:24] <czajkowski> cjohnston: sprints use them
[13:25] <cjohnston> How do sprints use the participation essential? do things get scheduled around that?
[13:26] <czajkowski> cjohnston: not sure about that perhaps to track them for their work
[13:26] <czajkowski> cjohnston: why do you ask ?
[13:27] <cjohnston> changes to summit that I want to discuss the wording on participation essential
[13:28] <czajkowski> cjohnston: post to lp users
[13:29] <czajkowski> cjohnston: or https://launchpad.net/~launchpad-dev  might make more sense
[17:44] <SpamapS> Something broken on code hosting?
[17:44] <SpamapS>      6kB     4kB/s | Fetching revisions:Get stream source
[17:44] <SpamapS> bzr pull's keep stopping there
[18:45] <SpamapS> !losaping
[18:47] <lifeless> SpamapS: thats not watched for on this channel
[18:47] <lifeless> SpamapS: or anywhere these days :)
[18:47] <SpamapS> ah
[18:47] <SpamapS> well, I cannot push or pull lp branches
[18:47] <lifeless> a) its webops now, and b) they are insulated from random public queries ;)
[18:48] <lifeless> SpamapS: testing myself
[18:49] <lifeless> SpamapS: I see similar symptoms; escalating
[18:54] <lifeless> SpamapS: try having more patience
[18:54] <lifeless> SpamapS: took 6 minutes for my test; clearly something bong
[19:09] <SpamapS> Write failed: Broken pipe
[19:09] <SpamapS> lifeless: not going so well for me.. been waiting about 5 minutes
[19:10] <czajkowski> SpamapS: well they are working on it atm
[19:14] <SpamapS> czajkowski: thanks, I'll just wait for a status update :)
[19:15] <cielak> hello everyone
[19:15] <cielak> I have a problem pushing bzr branches to launchpad
[19:15] <cielak> can anyone give me a hand?
[19:15] <czajkowski> cielak: yes we curretnly have some code hosting isues
[19:16] <cielak> alright, then I will wait patiently, thanks czajkowski!
[19:18] <dobey> i wonder if my missing UDD branch update is due to similar/same issue
[19:19] <czajkowski> dobey: I suspect so
[19:19] <mmrazik> thedac: srry. I was on different server
[19:20] <thedac> mmrazik: czajkowski lifeless: so mmrazik's script is polling sequentially but I suspect may be the cause of the codehosting degredation
[19:20] <dobey> czajkowski: did the issues start yesterdayish?
[19:20] <czajkowski> dobey: it's intermittent
[19:20] <dobey> ah
[19:21] <thedac> mmrazik: can the script be stopped for a few minutes to see if things clear up and we can confirm that is the problem?
[19:23]  * mmrazik is thinking
[19:23] <thedac> mmrazik: how long has this script been running?
[19:24] <mmrazik> thedac: months
[19:24] <thedac> oh, interesting
[19:25] <mmrazik> thedac: but we are adding projects so the number is probably growing
[19:26] <thedac> possibley we hit a tipping point
[19:26] <thedac> possibly
[19:27] <dobey> is it constantly polling bzr branches directly or something?
[19:27] <czajkowski> thedac: could be the two scripts combined from the QA folks doing this :/
[19:27] <mmrazik> there is jenkins polling the bzr branches
[19:28] <mmrazik> and the other script is using launchpadlib to check for new merge proposals
[19:28] <thedac> yeah, these are the top connections https://pastebin.canonical.com/71404/
[19:29] <thedac> mmrazik: you can see yours and victors are far and away the greatest number
[19:29] <mmrazik> right
[19:29] <mmrazik> let me shutdown our jenkins to see if it helps
[19:29] <mmrazik> ok?
[19:29] <czajkowski> mmrazik: thanks
[19:30] <thedac> sounds good
[19:30] <mmrazik> thedac, czajkowski: done
[19:31] <mmrazik> but TBH if we are then problem then we are probably screwed
[19:31] <mmrazik> and essentially I have no idea how we can do continuous integration
[19:32] <thedac> I still see victor's connections comming in. but let's test as is
[19:32] <james_w> mmrazik, why do you have to poll the branches if you are polling the merge proposals?
[19:33] <dobey> oh wow. that's a lot more connections than u1 even
[19:34] <mmrazik> james_w: right. When thinking about it the bzr polling is relatively small part of it and you are right we don't need it
[19:34] <mmrazik> in most cases the polling is on a daily basis anyway
[19:34] <mmrazik> so it shouldn't be responsible for huge numbers
[19:34] <mmrazik> I mean the bzr polling
[19:35] <james_w> mmrazik, those connection numbers don't suggest daily polling
[19:35] <james_w> unless you have 30000 projects
[19:36] <mmrazik> james_w: right. What I'm saying is that that most of the projects are just watching for new merge proposals (every 5 minutes)
[19:36] <mmrazik> instead of polling daily on bzr trunk
[19:36] <james_w> mmrazik, the merge proposal polling isn't included in those numbers
[19:36] <mmrazik> oh
[19:36] <mmrazik> so its just via bzr?
[19:36] <james_w> those numbers are only when a bzr branch is actuall fetched/checked
[19:37] <james_w> anything using launchpadlib goes over http rather than bzr
[19:37]  * mmrazik is thinking
[19:38] <mmrazik> so it means we are (polling or branching) bzr 30+14k/day
[19:39] <mmrazik> right?
[19:40] <james_w> yeah
[19:40] <james_w> bzr lp:something calls
[19:40] <james_w> or bzrlib.Branch.open() or whatever if you are using bzrlib
[19:41] <mmrazik> right
[19:44] <mmrazik> ok. there is probably something weird happening
[19:44] <mmrazik> as the numbers really don't add up to 30k
[19:44] <mmrazik> let me check once our jenkins is up again
[19:47] <lifeless> who is still having issues ?
[19:47] <james_w> mmrazik, are you using bzrlib at all, or just calling out to bzr?
[19:49] <mmrazik> james_w: I think both. I believe the jenkins bzr plugin is using bzr directly. I start to suspect there might be something wrong in this part.
[19:50] <mmrazik> james_w: we are using bzrlib to do some merging. But that is not happening that often
[19:50] <lifeless> the bzr plugin shells out to be
[19:50] <lifeless> bzr
[19:50] <czajkowski> SpamapS: are you still having issues?
[19:50] <mmrazik> we are essentially using tarmac
[19:50] <james_w> mmrazik, it's sometimes possible to accidentally open a lot of connections with bzrlib
[19:50] <james_w> but if you're using the tarmac codebase it's likely ok
[19:50] <mmrazik> yeah. I'm importing a class from tarmac
[19:50] <mmrazik> and using that
[19:51] <SpamapS> czajkowski: checking
[19:53] <james_w> it's possible that it's that script
[19:53] <SpamapS> still having issues
[19:53] <SpamapS> trying to bzr push to lp:~clint-fewbar/charms/precise/nagios/add-monitors-2
[19:54] <SpamapS> and trying to pull lp:ubuntu/gearmand
[19:54] <mmrazik> james_w: which script. The one using the Branch stuff from tarmac?
[19:54] <james_w> mmrazik, yeah, I
[19:54] <james_w> 'd suspect that over the jenkins bzr plugin
[19:54] <james_w> unless you have the bzr plugin set to poll the branch on * * * * *
[19:55] <mmrazik> I'm just looking at the configs
[19:55] <mmrazik> there are several projects with that
[19:55] <mmrazik> like ~10 or so
[19:55] <mmrazik> which is then ~14k connections
[19:55] <mmrazik> a day
[19:55]  * mmrazik is just changing those
[19:57] <james_w> mmrazik, do you have the poll log from one of those you could put in a pastebin?
[19:57] <james_w> it's possible it calls bzr twice per poll?
[19:58] <mmrazik> let me check
[19:59] <mmrazik> james_w: https://pastebin.canonical.com/71406/
[20:01] <james_w> right, so that's just one connection
[20:01] <mmrazik> I will be changing the configuration now in a batch for all the projects
[20:01] <mmrazik> to poll */15 * * * *
[20:01] <mmrazik> can we check the numbers in two days or so?
[20:02] <mmrazik> I guess the numbers for today will be still high
[20:03] <mmrazik> can I turn on our jenkins now?
[20:03] <mmrazik> (the polling has been changed)
[20:05] <lifeless> SpamapS: please try with -Dhpss and paste the log from that invocation of bzr
[20:07] <SpamapS> its worth noting that bzr break-lock works fine
[20:08] <SpamapS> lifeless: nothing prints out until I ctrl-C the waiting push...
[20:08] <SpamapS> lifeless: then I just get one extra line
[20:08] <lifeless> SpamapS: it goes to ~/.bzr.log
[20:08] <SpamapS> HPSS calls: 16 (0 vfs) SmartSSHClientMedium(bzr+ssh://clint-fewbar@bazaar.launchpad.net/)
[20:09] <SpamapS> Oh
[20:11] <lifeless> SpamapS: can you get a hung process going all the time please (e.g. put that in a loop or something)
[20:11] <lifeless> Want to see if we can spot it server side
[20:12] <SpamapS> lifeless: sure its hung now, should I leave that and give you the log as-is w/ the hung process?
[20:12] <lifeless> bac: SpamapS: You guy aren't on the same ISP or anything ?
[20:12] <lifeless> SpamapS: yes please
[20:12] <bac> lifeless: i'm on road runner.  SpamapS?
[20:12] <lifeless> bac: SpamapS: this has roughly the same symptoms I would expect from a pMTUd blackhole
[20:13] <lifeless> SpamapS: can you try pastebinning > 1K of text to paste.ubuntu.com ?
[20:13] <SpamapS> Time Warner
[20:13] <bac> hmm
[20:13] <SpamapS> lifeless: I am trying actually
[20:13] <SpamapS> same failure
[20:13] <lifeless> ok
[20:13] <SpamapS> trying to pastebin bzr.log
[20:13] <lifeless> so, not codehosting per se.
[20:13] <SpamapS> MTU issues?
[20:13] <lifeless> Dis be network biatches.
[20:14] <bac> lifeless: this started happening around 15:30UTC
[20:14] <lifeless> both paste.u.c and b.l.n are in the same network
[20:14] <SpamapS> cpe-76-94-208-1.socal.res.rr.com
[20:14] <lifeless> thedac: ^
[20:14] <SpamapS> thats me btw
[20:14] <SpamapS> 21:  canonical-2.mpr1.bsn004.pnap.net                     95.851ms pmtu 1476
[20:15] <thedac> interesting, a rounting issue perhaps?
[20:15] <thedac> SpamapS: can you do an mtr or traceroute and pastebin it?
[20:17] <SpamapS> http://pastebin.com/GxhHDJ00
[20:19] <SpamapS> dropping my MTU to 1476 makes everything work
[20:20] <bac> SpamapS: adjusted mtu on your router?
[20:20] <SpamapS> anyway, I have to run to lunch. bbiab
[20:20] <thedac> interesting
[20:21] <SpamapS> bac: no I only dropped MTU on this one machine behind my router
[20:21] <SpamapS> so I suspect the others would have the same issue
[20:22] <elmo> there's a currently a GRE tunnel in your path to the UK
[20:22] <elmo> so you (and anyone else with broken PMTUd) is going to be seeing brokeness
[20:22] <elmo> we're changing the routing announcements to work around this (i.e. avoid the GRE tunnel)
[20:24] <lifeless> elmo: we're sure we're not breaking ptmud ourselves - e.g. the tunnel acting as a blackhole [usually if an endpoint is unroutable or firewalled...]
[20:24] <lifeless> ?
[20:25] <bac> lifeless: verified i can push with mtu lowered
[20:25] <elmo> bac: the routing's already been changed; please confirm, you should be coming in via datahop or L3, not internap?
[20:30] <bac> elmo: i cannot confirm.  traceroute is timing out
[20:30] <elmo> bac: try mtr
[20:35] <bac> elmo: http://paste.ubuntu.com/1125869/
[20:35] <bac> elmo: so, yes, l3
[20:35] <elmo> bac: cool, thanks
[21:14]  * SpamapS is back
[21:20] <SpamapS> elmo: just to confirm, I am also good with MTU of 1500 now ... and tracepath does not show any MTU drop
[21:51] <lifeless> mmrazik: your job wasn't the issue, but turning down your probes to be less frequent is useful :)