[00:00] <elmo> I still think the fact it's not running Add DistroArchSeres is key
[00:00] <wgrant> I don't think so.
[00:00] <wgrant> I think it only does that once.
[00:00] <wgrant> Let's see...
[00:00] <elmo> it seems to do it every time
[00:01] <wgrant> Once each cycle (~5 seconds), and only for ubuntu/hardy/amd64?
[00:01] <elmo> no, let me paste more of the log
[00:02] <wgrant> Oh. BuilderSet calls addDistroArchSeries. I see.
[00:04] <elmo> wgrant: http://paste.ubuntu.com/303910/
[00:04]  * elmo cries a little at the mentions of hoary in there
[00:04] <wgrant> Aha.
[00:05] <wgrant> OK, this is useful.
[00:07] <wgrant> WTF are BuilderGroup and BuilderSet separate arrrrgh
[00:18] <wgrant> elmo: cesium can actually talk to the slaves, right?
[00:18] <wgrant> What if you run an XML-RPC status() on them?
[00:19] <wgrant> I can't see why else they would be unavailable!
[00:19] <elmo> wgrant: checking
[00:20] <wgrant> 'import xmlrpclib; xmlrpclib.ServerProxy("http://osmium.ppa:8221/rpc").status()' should do it.
[00:21] <elmo> AAAAAAAAAAAAAAAAHA
[00:21] <elmo> good call
[00:21] <elmo> ['BuilderStatus.ABORTING', '1288257-2696293']
[00:21] <elmo> wgrant: ^--
[00:21] <wgrant> Aha.
[00:21] <wgrant> Now.
[00:21] <wgrant> Oh dear.
[00:22] <wgrant> lp-buildd has a bug where it won't remove the old build directories.
[00:22] <wgrant> So if you restart them, they might die again.
[00:22] <wgrant> But we'll see.
[00:22] <elmo> oh right
[00:22] <elmo> and that just happened on a massive scale
[00:22] <wgrant> Does bouncing osmium's lp-buildd do anything at all?
[00:22] <elmo> awesome
[00:22] <wgrant> Hmm.
[00:22] <wgrant> I guess that the virtual builders should be OK, since they get reset each time.
[00:23] <elmo> they only get reset for a new build
[00:23] <wgrant> But any non-virt *might* need manual cleanup. You'll see soon.
[00:23] <wgrant> I forget exact at which stage they get handed back.
[00:23] <elmo> in this case we're not getting far enough to reset them
[00:23] <elmo> but I can reset them by hand easy enough
[00:23] <wgrant> That might be a good idea.
[00:23] <wgrant> I guess you'll probably need to then re-OK them all, but that's easy enough.
[00:24] <wgrant> Urgh, apologies for not thinking to check the status() first. I got distracted by other parts of the log fragments.
[00:25] <wgrant> It might be an idea to grab the lp-buildd logs on a couple of failed builders to see why they got stuck in ABORTING.
[00:26] <wgrant> Since they will of course vanish once you reset the guests...
[00:26] <wgrant> I presume it's the residual build directory, but cannot be sure.
[00:33] <wgrant> sbuild must have got really really wedged (it should have been SIGKILLed 10 seconds after the abort), or Twisted is being bad.
[00:34] <elmo> wait what
[00:34] <elmo> we knocked the buildds off line at 17:00
[00:35] <elmo> palmer was building at 20:00
[00:35] <wgrant> palmer is fine.
[00:35] <wgrant> The queue is just empty.
[00:36] <elmo> hmm
[00:36] <wgrant> (if you look at the logs, it will tell you that there are no candidates for palmer)
[00:36] <elmo> right
[00:36] <elmo> I'm not sure it always was though
[00:36] <elmo> or maybe I'm losing my mind
[00:36] <elmo> the problem is i can't actually check on the virtual builders
[00:36] <wgrant> Why not?
[00:36] <elmo> we lock ourselves out of the guests for reasons of paranoia that are probably not sane
[00:36] <wgrant> Ah.
[00:36] <wgrant> Well.
[00:36] <wgrant> We can at least get the logtail.
[00:37] <elmo> ok, how about sparc
[00:37] <elmo> that's certainly got shit to do :)
[00:37] <elmo> wgrant: we can?
[00:38] <elmo> interesting
[00:38] <elmo> AFAICT, artigas was mid-build when cesium lost contact with it
[00:39] <elmo> so I suspect this is readily reproducible
[00:42] <wgrant> elmo: We can, through RPC. I would find the call, but I've had to move to another machine; it seems my WAP is unhappy.
[00:44] <wgrant> Ah, no, we can't.
[00:44] <wgrant> We can only get it while it's BUILDING :(
[00:50] <wgrant> elmo: What became of that slave that you bounced before the issue was properly diagnosed?
[00:50] <wgrant> Did it recover?
[00:51] <elmo> that was palmer?
[00:52] <elmo> I've reset all the virtual ones now
[00:52] <elmo> artigas is still stuck and is:
[00:52] <elmo> >>> import xmlrpclib; xmlrpclib.ServerProxy("http://artigas.buildd:8221/rpc").status()
[00:52] <elmo> ['BuilderStatus.WAITING', 'BuildStatus.OK', '1312047-2740484',
[00:52] <wgrant> Ahhh.
[00:52] <elmo> I suspect bouncing it's slave will recover it
[00:52] <wgrant> So palmer wasn't actually broken.
[00:52] <wgrant> Hm.
[00:52] <wgrant> That means it's waiting for the upload to happen.
[00:52] <wgrant> What does buildd-manager's log say about it?
[00:53] <elmo> whatever was in the pastebin I guess?  but checking
[00:53] <wgrant> If you bounce it, it will have to build again, and you might have to manually clean up the build directory. I would have thought buildd-manager should have dealt with that one.
[00:55] <wgrant> Did it recover, or did you kill it?
[00:56] <elmo> artigas?  I bounced the slave
[00:56] <wgrant> Aha.
[00:56] <elmo> and am cleaning up byhand
[00:56] <wgrant> Great.
[01:01] <elmo> wgrant: thanks a lot for your help
[01:02] <wgrant> elmo: np.
[01:02] <wgrant> I can reproduce the artigas situation.
[01:03] <wgrant> buildd-manager just sits there and ignores it.
[01:17]  * mwhudson afk for a short while
[01:19] <wgrant> elmo: OK, I think I see why artigas was special.
[01:19] <elmo> oh?
[01:19] <wgrant> elmo: There is code to rescue lost builders.
[01:20] <wgrant> That is invoked if they report that they are currently building a (build ID, queue ID) pair that doesn't exist any more.
[01:20] <wgrant> It is that code which aborts the build.
[01:20] <wgrant> However...
[01:20] <wgrant> there is a flaw!
[01:20] <wgrant> It doesn't check that the build is still assigned to the builder.
[01:21] <wgrant> So when the builder times out, it is unassigned from the buildqueue, and the build goes back to 'Needs Building'. But when the slave is scanned, the master sees that the referenced buildqueue still exists, so doesn't attempt a rescue.
[01:21] <wgrant> For the PPA builders, that will resolve pretty quickly -- another builder will grab the build, complete it, and the buildqueue will disappear.
[01:21] <wgrant> At that point, the builder will reference something that doesn't exist, and will be aborted.
[01:22] <wgrant> But for sparc, there are only two buildds. I guess artigas's current build wasn't picked up by the other buildd, so it never got rescued.
[01:22] <wgrant> But there are at least two bugs here, since the abort on at least some of the virt builders hung.
[01:30] <wgrant> elmo: I have reproduced all pieces of the problem locally.
[01:30] <wgrant> Including the ABORTING hang.
[01:32] <elmo> wgrant: great
[01:55] <wgrant> elmo: The pile of rescue bugs is growing...
[02:35] <poolie> rockstar: or other, could you approve my list request for https://edge.launchpad.net/~ubuntu-distributed-development
[02:36] <rockstar> poolie, now that's a question that could have gone to #launchpad - You're all turned around today.  :)
[02:36] <poolie> i am :)
[02:36] <poolie> this is not helped by pidgin showing this channel as "launchpad..." :)
[02:36] <rockstar> poolie, :)
[02:38] <rockstar> poolie, generally, anything Ubuntu related is passed to jorge
[02:38] <poolie> ok
[02:39] <poolie> and he has the (technical) right to approve it?
[02:40] <thumper> poolie: normally it is because ubuntu lists are on ubuntu.com not launchpad.net
[02:40] <poolie> mm
[02:40] <poolie> is there a technical reason for that?
[02:40] <thumper> that I don't know
[02:43] <rockstar> poolie, no, I think it's so that he's aware of what's going on, and so that we don't have a bunch of ubuntu lists.
[02:45] <poolie> ok, i'll mail him
[02:46] <lifeless> hah
[02:46] <lifeless> new list notice still points at launchpad.dev
[02:47] <lifeless> poolie: did you undo the udd membership?
[02:47] <poolie> i don't think so
[02:47] <poolie> what do you mean?
[02:48] <lifeless> well I got the notice that canonical-bazaar is in the team udd, and that we can enable list delivery
[02:48] <lifeless> but I can't see the list on https://edge.launchpad.net/~lifeless/+editemails
[02:49] <poolie> i presume it's because the list is in this half-created stnate
[02:49] <lifeless> oh
[02:50] <lifeless> have you pung spm to whack it on?
[02:52] <poolie> read the scrollback
[02:52] <poolie> mwhudson: did you get that mail about graphs?  do you think you could do it reasonably easily, not necessarily today
[02:53] <mwhudson> poolie: yes, just writing up the last one now
[02:53] <poolie> you legend
[02:55] <lifeless> spm: ping; whats the mail list approval ul?
[02:55] <lifeless> URL
[02:56] <poolie> lifeless: i sent mail to jorge, i'll see what he thinks
[02:56] <poolie> i don't want to force it today
[02:57] <spm> lifeless: +mailinglists
[02:57] <lifeless> poolie: hmm? if you've applied for the list, the CHR will probably just do it when they notice
[02:57] <lifeless> oh, it has the word ubuntu, it may need the special treatment.
[02:57] <lifeless> spm: thanks, but ignore that ;)
[02:58] <poolie> no further action is needed til jorge replies
[02:58]  * spm adds lifeless back to /ignore :-P
[02:58] <poolie> :)
[02:58] <lifeless> ooh quick, while hes not listening...
[03:04] <spm> heh
[03:05] <mwhudson> poolie: sent
[03:11] <mwhudson> so where has inline editing of merge proposal status gone?
[03:29] <thumper> mwhudson: it isn't there yet
[03:29] <thumper> mwhudson: it is branch status that has been done
[03:29] <thumper> mwhudson: it is on my todo list
[03:29] <mwhudson> thumper: oh right
[03:58] <poolie> mwhudson: could you add lpstats for failing imports by type?
[03:59] <mwhudson> poolie: yes
[03:59]  * mwhudson writes an RT
[04:00] <thumper> mwhudson: want to add to that for one for succeeding imports by type?
[04:00] <mwhudson> thumper: yeah
[04:00] <mwhudson> suspended/invalid probably not all that interesting
[04:19] <cody-somerville> Isn't there instructions somewhere on how to setup launchpad development instance without running rocketfuel-setup?
[04:19] <thumper> cody-somerville: yes (ish)
[04:19] <thumper> cody-somerville: I did it recently
[04:19] <thumper> cody-somerville: as I don't use rocketfuel scripts
[04:20] <thumper> cody-somerville: read the script and do the bits that make sense :)
[04:20] <thumper> I don't use the 'blessed' directory or repo layout
[04:42]  * thumper EODs for now
[05:07] <wgrant> I see somebody renamed samarium to 'samariumd' fairly recently.
[05:07] <wgrant> Somebody might want to fix that.
[05:07] <elmo> we can't
[05:07] <elmo> LP breaks when you try
[05:08] <wgrant> elmo: Oh, it's always been like that? Hmm.
[05:08] <elmo> wgrant: no it's recent
[05:08] <elmo> we've no idea how it changed, but the web ui won't let you change it back
[05:08] <rockstar> mwhudson, have you landed your branches I've reviewed.  pqm threw up on me.
[05:08] <wgrant> Niiice.
[05:08]  * wgrant tries to rename one locally.
[05:09] <wgrant> Hm, WFM
[05:09] <wgrant> elmo: How does it complain when you try?
[05:09] <wgrant> (and are you going to sleep at some point?)
[05:10] <mwhudson> rockstar: yes, i have a success mail from PQM
[05:10] <mwhudson> rockstar: i just did ec2 land fwiw
[05:11] <elmo> wgrant: it just times out, I think - I haven't had a chance to dig into the oops, I only noticed it myself earlier this evening
[05:11] <wgrant> elmo: Urgh. That's not too nice.
[05:20] <rockstar> mwhudson, hrm.
[05:22]  * mwhudson EODs
[07:04] <cody-somerville> wgrant, whats the url to your blog?
[08:24] <adeuring> good morning
[08:27] <henninge> Moin adeuring!
[08:27] <adeuring> hi henninge!
[08:27] <henninge> Hallo jtv!
[08:27] <jtv> hi henninge!
[08:28] <henninge> jtv: your memory footprint is about to land! cool!
[08:28] <henninge> work
[08:29] <jtv> henninge: yes... there's still a lot of buildup in the final phase, but the first phase has been tamed.
[08:30] <wgrant> cody-somerville: I have none. Why?
[08:30] <cody-somerville> wgrant, I thought you wrote a blog post on setting up soyuz.
[08:31] <wgrant> cody-somerville: http://williamgrant.id.au/f/1/2009/running-soyuz.html
[08:31] <wgrant> Should probably throw that on the wiki at some point.
[08:32] <wgrant> lp-buildd is the troublesome bit, particularly on Karmic.
[08:32] <jtv> henninge: I'm about to go for lunch—ttyl!
[08:32] <henninge> jtv: ok, enjoy it!
[08:33] <wgrant> But even on pre-Karmic it requires some hacking if you grab official Ubuntu buildd chroots.
[08:33] <wekt> will trade a bug triage.  you triage 1 mine, i'll try to triage yours.  (I have not triaged for Ubuntu yet) https://bugs.launchpad.net/launchpad/+bug/453804
[08:33] <mup> Bug #453804: absolute font sizing and size smaller than default creates accessibility and usability difficulties <Launchpad itself:New> <https://launchpad.net/bugs/453804>
[08:57]  * wgrant plots an evil PPA package which grabs stuff from the librarian, and uses a launchpadlib script to silently inject arbitrary untraceable binaries into the resultant packages.
[08:57] <wgrant> I think that would work.
[08:57] <noodles775> yeah, thanks wgrant.
[08:58] <wgrant> noodles775: Hm?
[09:01] <henninge> hey danilo_, danilos !
[09:01] <noodles775> hm... is anyone else having trouble accessing dev.launchpad.net?
[09:02] <wgrant> I couldn't last night, but that was massive packet loss in L3.
[09:02] <wgrant> It's slow, but working.
[09:02] <noodles775> ok, it's working again now... :)
[09:02] <noodles775> yep, thanks wgrant.
[09:06] <wgrant> al-maisan, noodles775: I suppose neither of you know too much about buildd-manager? I've fixed one of the pieces of this morning's build farm disaster, but I'm not sure about how to fix the other.
[09:06]  * al-maisan looks at wgrant's emails again
[09:07] <noodles775> wgrant: I've had to take a look at it recently and al-maisan is familiar with it too... so shoot (it'll be good for us) :)
[09:08] <wgrant> Bug #463041 was easily fixed, but bug #463046 is rather less obvious.
[09:08] <mup> Bug #463041: Lost builder detection is insufficiently aggressive <Soyuz:New> <https://launchpad.net/bugs/463041>
[09:08] <mup> Bug #463046: Rescuing a BUILDING builder just makes things worse <Soyuz:New> <https://launchpad.net/bugs/463046>
[09:08] <noodles775> wgrant: were you able to setup a test that shows the first issue that you mentioned in that bug?
[09:08] <noodles775> (463041?)
[09:08] <noodles775> ok.
[09:09] <wgrant> noodles775: 463041 is fixed and tested in a branch here.
[09:09] <noodles775> Great!
[09:09] <wgrant> Landing that now would just make everything worse, however.
[09:12] <noodles775> wgrant: have you got time for a call? (skype: absoludity)
[09:13] <wgrant> noodles775: Sure. Let me just install Skype...
[09:13] <noodles775> al-maisan: we can do a conf?
[09:14] <al-maisan> noodles775: nice :)
[09:14] <wekt> ekiga works well instead of skype as long as it gets ALSA directly.  pasuspender can be used for that
[09:15] <wgrant> Hopefully Empathy will be usable for this sort of thing soonish.
[09:15] <wekt> empathy uses all plugins through dbus.  I'd rather use pidgin and libpurple
[09:16] <noodles775> I'm happy to use ekiga ... (although not sure of a conf line I can use).
[09:16] <al-maisan> ekiga works here as well
[09:16] <wekt> it has a conference room, but i have not tried it
[09:17] <noodles775> ok, let's try that first then :)
[09:17] <al-maisan> what's the sip address?
[09:17] <al-maisan> 501@ekiga.net
[09:18] <wekt> I'm there
[09:19] <noodles775> I'm not... I can call other conf lines in ekiga, but that's just dropping for me :/
[09:20]  * noodles775 registers at ekiga.
[09:20] <wekt> is your account @ekiga.net?
[09:20] <al-maisan> wekt: al-maisan
[09:21] <wekt> al-maisan & I are chatting in the conference room
[09:24] <wekt> i hear a third party.  with much humm
[09:24]  * wgrant awards Ekiga the "even worse than Skype" award.
[09:24] <al-maisan> nice award :)
[09:24] <wekt> you might turn on silence detection.
[09:24] <wgrant> (I have Skype working, but cannot get Ekiga to hear me :()
[09:24] <wekt> i hear you
[09:25] <al-maisan> we did hear somebody say "hello"
[09:25] <wgrant> Ah.
[09:25] <wgrant> I see.
[09:25] <wgrant> Let's try that again.
[09:25] <wekt> you might turn down your mic boost you were quite loud.
[09:25] <wekt> & turn on silence detection
[09:26] <wekt> I leave now
[09:26] <al-maisan> bye wekt
[09:27] <al-maisan> wgrant: I do hear you
[09:27] <wgrant> al-maisan: The only thing I've heard from Ekiga is a very broken up introduction to the conference room :(
[09:27] <wgrant> Skype works fine; shall we use that?
[09:28] <al-maisan> hmm ..
[09:28] <al-maisan> wgrant: yup
[09:29] <wgrant> I'm william.a.grant.
[09:29] <al-maisan> wgrant: what's your skype id? Mine is 'mhrnjad'.
[09:29] <al-maisan> OK
[09:29] <wgrant> I've added both you and noodles775.
[09:29] <al-maisan> wgrant: thanks
[09:30]  * wgrant has very little idea how to use Skype.
[09:30] <al-maisan> noodles775 seems busy with a code review ..
[09:30] <al-maisan> wgrant: let me call you
[09:31] <elmo> wgrant: did you file any bugs about that drama last night btw?  (if not, I can, just don't want to duplicate)
[09:31] <wgrant> elmo: I did.
[09:31] <elmo> wgrant: neato; thanks
[09:31] <wgrant> al-maisan: https://code.edge.launchpad.net/~wgrant/launchpad/rescue-robbed-builders
[09:42] <wgrant> Well, that is a nice simple solution.
[09:44] <wgrant> elmo: Just fixing and writing tests for the second master issue now. However, there still remains the issue of the ABORTING slave hang, which I don't know how best to resolve. I'm thinking that if lp-buildd can't be easily fixed, the master could be taught to just fire the reset trigger if it hasn't aborted after a minute or so.
[09:51] <wgrant> Is there some script around to dispatch builds to manual builders?
[09:51] <wgrant> I've occasionally seen it happen on production, and it would handy for testing sometimes.
[10:03] <deryck> Morning, all.
[10:03] <_thumper_> deryck: hey, sent you an email :)
[10:17] <deryck> thumper, I saw.  Was going to reply here in a sec.
[10:26]  * al-maisan needs a coffee
[11:09] <maxb> I have an MP targeted at ~launchpad-dev//ztk-2.5 if someone has a moment: https://code.edge.launchpad.net/~maxb/launchpad/pymig-remove-lsprof/+merge/14091
[11:13] <salgado> maxb, r=me
[11:53] <allenap> jml: I have a branch that moves devscripts out of lib/. I've updated the Makefile to run the devscripts tests with zope.testrunner and trial, and I wondered which I should actually go for? See http://pastebin.ubuntu.com/303684/. Trial doesn't do subunit does it? zope.testrunner doesn't either, but there is (your) code to make it work already in the tree.
[12:15] <jtv> BjornT_: You're at least one step ahead of me when it comes to submitting lp-production-configs branches.  Can you help me get to that point?
[12:30] <jtv> salgado, you've got one of those branches in pqm... can you explain to me how to submit to lp-production-configs?
[12:30] <jtv> I'm documenting this stuff, too.
[12:31] <jtv> ah, it wasn't salgado, it was matsubara-afk
[12:31] <salgado> jtv, I don't think I have any branches on PQM
[12:31] <jtv> salgado: I was wrong, sorry.  I always knew I'd confuse you to someday
[12:31] <jtv> you *two
[12:40] <BjornT_> jtv: how far have you come so far?
[12:40] <jtv> BjornT_: I'm trying to pqm-submit, but I don't know how to set up bzr for it.
[12:44] <BjornT_> jtv: this is in my locations.conf: http://pastebin.ubuntu.com/304261/
[12:44] <jtv> BjornT_: cool, thanks.  If I turn this into wiki documentation, would you mind casting a glance at it and seeing if it makes sense?
[12:46] <BjornT_> jtv: sure. it's similar to all the other pqm-managed lp branches; it's just different product names and branches. there should be some documentation already...
[12:47] <jtv> BjornT_: I searched all wikis for lp-production-configs without any luck
[12:48] <BjornT_> jtv: well, i was thinking about pqm in general. i can't find anything either, except for some really old instructions
[12:49] <jtv> BjornT_: cool, then we have a more or less clean slate.  :-)
[13:23] <maxb> salgado: You did review:approve on my ztk-2.5 MP but not merge:approve - just oversight?
[13:23] <salgado> maxb, yeah, I always forget that
[13:23] <salgado> will do now
[13:24] <maxb> I should try my hand at ajaxifying that
[13:26] <BjornT_> EdwinGrubbs: ping?
[13:27] <EdwinGrubbs> BjornT_: pong
[13:27] <BjornT_> EdwinGrubbs: i think you had problems with windmill, in that asserting a property/element sometimes failed, after having waited for it, right?
[13:28] <EdwinGrubbs> BjornT_: right
[13:30] <BjornT_> EdwinGrubbs: so, i'm not sure why that is, i have to take a closer look. but i'm interested, why do you have to use an assert at all? why not do like this? http://pastebin.ubuntu.com/304288/
[13:32] <EdwinGrubbs> BjornT_: basically, I don't believe theat waits.forElementProperty() works correctly if the assert after it fails. I would have removed it if I didn't start getting spurious errors.
[13:33] <BjornT_> EdwinGrubbs: does it fail if you make sure the element isn't there?
[13:34] <BjornT_> EdwinGrubbs: also, an occasional spurious success is better than an occsional spurious failure :)
[13:36] <BjornT_> gary_poster: would you know anything about this error when running utilities/paste? http://pastebin.ubuntu.com/304293/
[13:37] <BjornT_> gary_poster: the only thing before the _pythonpath import is whitespace and comments
[13:42] <gary_poster> BjornT_: how are you starting paste?
[13:43] <BjornT_> gary_poster: in that instance, by invoking /devel/launchpad/rocketfuel/utilities/paste
[13:44] <BjornT_> gary_poster: hmm. i noticed that it's only there. it works in another branch, which should be just as up-to-date
[13:44] <gary_poster> BjornT_: OK. Is ascii the usual Python encoding, and your system is mildly unusually specifying utf8?  If so, I would suggest modifying buildout-templates/_pythonpath.py.in to include encodings.utf_8 as hard-coded in the clean-modules list and committing with rs=gary.  There may be a better way to handle it, but that is a reasonable start IMO.
[13:45] <EdwinGrubbs> BjornT_: yes, I believe it normally fails normally if I assert that the node doesn't have the style. I have not tested if there would be spurious successes with that. I'm not sure how that makes the test any better.
[13:46] <BjornT_> EdwinGrubbs: because a spurious success means that we would miss catching a test failure in some rare instances; it would probably never happen. a spurious failure means that the we enter testfix mode, which is quite bad.
[13:47] <maxb> LP auto-sets my MP to "Merged" when it is merged. Is it intentional that it doesn't also update the branch status to "Merged" ?
[13:48] <BjornT_> EdwinGrubbs: we should be able to trust waiting for an element. my guess is that waiting for an element gets notified that the element gets added slightly before it actually gets added to the dom, which asserting it checks the dom directly, so you have a race condition there
[13:49] <jtv> BjornT_: I seem to have pqm-submit working for my configs branch... would you care to review the documentation byproduct?  https://dev.launchpad.net/LoggingOopses
[13:49] <jtv> BjornT_: if you don't see anything horribly wrong with it, I'll start linking to it and asking for more feedback from mailing lists.
[13:50] <EdwinGrubbs> BjornT_: I assume what you are saying is that we catch that error and just ignore it. The waits.forElementProperty() also causes this spurious error, and that situation doesn't involve the timing of adding an element to the dom.
[13:51] <EdwinGrubbs> BjornT_: why check the node if we are going to ignore the outcome?
[13:52] <BjornT_> jtv: well, the documentaion looks good, but it seems misplaced. i would never have thought of looking for that documentation in a document called LoggingOopses
[13:53] <BjornT_> EdwinGrubbs: s/element gets added/dom gets modified/
[13:54] <BjornT_> EdwinGrubbs: i'm saying that waiting for an element/property probably works, and we should trust it. we don't have to assert that the wait worked everytime we wait for something. so we don't ignore the outcome. if the wait fails, the test fails.
[13:54] <mars> Has anyone had a problem building ctypes on Karmic?
[13:55] <mars> The egg build step complains about a missing Python.h file... missing the Python2.5 version maybe?
[13:56] <BjornT_> mars: does the build fail? sometimes errors are produced for other pythons than 2.4, but they can be ignored.
[13:56] <mars> BjornT_, yep, build fails: An error occured when trying to install ctypes 1.0.2.Look above this message for any errors thatwere output by easy_install.
[13:56] <mars> make: *** [/home/mars/canonical/lp-branches/trunk/bin/py] Error 1
[13:57] <jtv> BjornT_: the meat is to be about the code part, really.
[13:57] <mars> might have to do a make clean
[13:59] <BjornT_> mars: does it help if you install python-dev?
[13:59] <mars> BjornT_, it is already installed for Python2.6.  Are we running on 2.6 yet?
[14:00] <BjornT_> mars: is python2.4-dev installed?
[14:00] <mars> BjornT_, nope
[14:01] <mars> it must have been cleaned up during the upgrade
[14:02] <EdwinGrubbs> BjornT_: I see what you are saying. I think I've just become pessimistic about windmill doing what I expect.
[14:03] <BjornT_> mars: then you probably don't have launchpad-dependencies install either. the lp ppa does get disabled during upgrades
[14:03] <mars> BjornT_, looking at the make output - looks like we are still on python2.4.  thanks for the pointer
[14:03] <mars> yep, have to re-enable that next
[14:03] <BjornT_> EdwinGrubbs: i'll certainly look into this, to see what is really going on, but until then i'm going to change the tests to be more robus
[14:04] <EdwinGrubbs> ok
[14:05] <maxb> I just pushed a branch: https://code.edge.launchpad.net/~maxb/launchpad/remove-stray-ref-to-sourcecode-lazr-js and it has a weird error
[14:05] <maxb> No such file: u'/srv/bazaar.launchpad.net/mirrors/00/03/eb/bb/.bzr/repository/upload/28c77e113c8b6a60a9181c736b1c6ddf.pack': [Errno 2] No such file or directory: u'/srv/bazaar.launchpad.net/mirrors/00/03/eb/bb/.bzr/repository/upload/28c77e113c8b6a60a9181c736b1c6ddf.pack'
[14:11] <mars> BjornT_, yep, the Karmic upgrade is the culprit.  I thought it was strange when the cleanup step told me ¨You don´t need all these developer tools any more¨
[14:43] <Ursinha> a translations person is missing, so I think I'll be the double agent in the meeting
[14:43] <Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting in 18 mins
[14:45] <allenap> Ursinha: Google Calendar says in 1h15m...
[14:45] <Ursinha> allenap, google calendar is wrong :)
[14:46] <sinzui> google calendar has difficulty with UTC
[14:46] <allenap> Ursinha: Cool.
[14:46] <Ursinha> allenap, unless UTC time changed...
[14:46]  * sinzui only uses the shite...site under duress
[14:46] <allenap> sinzui: I agree on both points.
[14:51] <Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting in 9 mins
[14:56] <Ursinha> henninge, ha! :) are you joining the prod. meeting in 5 mins?
[14:56] <henninge> Ursinha: I can "me" there, if you like ... ;)
[14:57] <Ursinha> henninge, I can wear both hats, if you like
[14:57] <Ursinha> I don't mind
[14:57] <henninge> Ursinha: Please do that, there is no special information that I have
[14:57] <henninge> at least, not that I can think of.
[14:58] <Ursinha> henninge, except that I'll ask someone to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891
[14:58] <mup> Bug #462891: TraversalError on +export <oops> <Launchpad Translations:New> <https://launchpad.net/bugs/462891>
[14:58] <Ursinha> :)
[14:58]  * henninge looks at that
[15:00] <Ursinha> thaaaanks henninge :)
[15:01] <Ursinha> stub, Chex, gary_poster, rockstar, bigjools, sinzui, allenap, Ursinha -> production meeting @ #launchpad-meeting now :)
[16:06] <maxb> I have a branch which has somehow broken - should I file a launchpad-code bug for someone to investigate? https://code.edge.launchpad.net/~maxb/launchpad/remove-stray-ref-to-sourcecode-lazr-js
[16:07] <maxb> broken, as in, I pushed it, and it ended up in that stuck state
[16:11] <Ursinha> rockstar, is there something maxb can do to dig the problem a bit more?
[16:12] <rockstar> maxb, have you tried pushing again?
[16:13] <maxb> rockstar: For my own use, I re-pushed the branch under a different name. I've left that one alone in case anyone wants to debug its state. If no one does, I shall simply delete it
[16:14] <rockstar> maxb, yeah, I'd suggest just deleting it.  It looks like there was some issue on push.
[17:26]  * sinzui plants face in plam
[17:27] <sinzui> abentley: thank you for your bug report. launchpad.dev is hardcoded in a template
[17:27] <abentley> sinzui: No problem.
[17:33] <danilos> BjornT_, hi, still around?
[18:21] <mrevell> Night all!
[18:21] <rockstar> sinzui, jam is reporting many timeouts with https://edge.launchpad.net/bzr/+download
[19:02] <mwhudson> good morning
[19:36]  * mwhudson looks at oops-prune failure mails
[19:56] <thumper> mwhudson: we should grab spm later
[19:56] <thumper> oh and good morning
[19:59] <mwhudson> thumper: good morning
[20:08] <mwhudson> thumper: to run branch-distro.py ?
[20:08] <mwhudson> thumper: or something else?
[20:08] <thumper> mwhudson: yes to run branch-distro
[20:09] <mwhudson> thumper: we could get mbarnett to start it now i guess?
[20:09] <mwhudson> or at least do some prep
[20:09] <thumper> mwhudson: what prep is needed?
[20:09] <mwhudson> thumper: well getting a newer launchpad tree onto crowberry
[20:10] <mwhudson> not much else i admit
[20:16] <thumper> rockstar-afk: ping
[20:16] <thumper> abentley: skype?
[20:47] <thumper> sinzui: we are due to talk right?
[20:47] <sinzui> I think so
[20:48] <thumper> sinzui: when?
[20:48] <thumper> sinzui: I think my local calendar is in local time
[20:48] <sinzui> 40 minutes
[20:48] <sinzui> if we are keeping UTC
[20:48] <thumper> sinzui: can you do earlier, or shall we do it in 40?
[20:49] <sinzui> I can do it earlier. We can talk now if you like
[20:49] <thumper> sinzui: ok, lets do it now
[20:56] <poolie> interesting responses re imports hey?
[20:56] <poolie> but perhaps not reaching many people we don't already know
[21:21] <rockstar-afk> thumper, pong
[21:48] <awilkins> Offtopic but you guys are the most likely guys to know stuff about servers at Canonical I know of... anyone know if keyserver.ubuntu.com is being thrashed or just down?
[21:48] <rockstar> awilkins, yeah, I don't think many people would know the direct answer to that.
[21:49] <awilkins> I thikn just thrashed... it just served me a key
[21:49] <awilkins> But it's very slow
[21:50] <wgrant> Well, consider what day it is.
[21:50] <awilkins> Indeed
[21:50]  * rockstar awards wgrant a gold star
[21:51] <wgrant> rockstar: What have I done now?
[21:51] <rockstar> wgrant, pointing out that it's release day.
[21:52] <wgrant> Ah.
[21:52] <awilkins> Bah, my ISP still hasn't mirrored it
[21:52] <wgrant> awilkins: Torrents!
[21:52] <awilkins> Oh, I had the ISO before it became a release ISO
[21:52] <awilkins> Three of them
[21:53] <wgrant> awilkins: (also, #canonical-sysadmin is the right place)
[21:53] <awilkins> Via torrents
[21:53] <rockstar> wgrant, torrenting is illegal.
[21:53] <wgrant> Yes, I know. Please don't announce it like that next time.
[21:53] <wgrant> awilkins: Or we will kick you more aggressively.
[21:53] <wgrant> rockstar: Naturally. Evil nasty Ubuntu pirates.
[21:55] <awilkins> Hmm. I really don't see how uploading files to a world-visible webserver doesn't constitute an announcement in itself... but que sera
[21:56] <wgrant> awilkins: Mirroring has to be done before the release is widely known, or mirroring becomes impossible.
[23:12]  * thumper is looking for a nice unicode character to test with
[23:12] <thumper> any suggestions?
[23:13] <spm> thumper: ☕
[23:14] <spm> cofee mug. how could you go wrong.
[23:14] <thumper> spm: which character is that
[23:14] <lifeless> ☕
[23:14] <lifeless> obviosuly
[23:14] <thumper> haha
[23:14] <spm> braille is another useful to try
[23:15] <thumper> 2615 apparently
[23:20] <wgrant> thumper: You know you can use literals like u'\N{INTERROBANG}'?
[23:21] <thumper> wgrant: and what does an interrobang look like?
[23:21] <thumper> wgrant: and what is the name of the coffee cup?
[23:21] <wgrant> thumper: It's a combo exclamation mark question mark.
[23:21] <thumper> wgrant: I had vague recollections, thanks for reminding me
[23:34] <thumper> abentley: still around?
[23:49] <thumper> abentley: nm, fixed it
[23:55] <thumper> mwhudson: had to fix a bug in attachment encoding to write the test for attachment viewing :-|