[00:03] mwhudson: [00:03] lp://staging/~mwhudson/linux/trunk Started: 48 minutes ago Last heartbeat: 14 seconds ago [00:03] 2010-04-15 22:54:11 INFO 49969129 bytes transferred | Fetching revisions:Inserting stream [00:03] 2010-04-15 22:55:11 INFO 43484334 bytes transferred | Fetching revisions:Inserting stream [00:03] 2010-04-15 22:56:13 INFO 35991244 bytes transferred | Fetching revisions:Inserting stream [00:03] 2010-04-15 22:57:14 INFO 31731071 bytes transferred | Fetching revisions:Inserting stream [00:03] mbarnett: i meant like "log in and run bzr st", sorry [00:04] mwhudson: hah, ok [00:05] mwhudson: [00:05] importd@strawberry:/srv/importd.staging.launchpad.net/staging/launchpad$ bzr st [00:05] unknown: [00:05] lib/canonical/launchpad/apidoc/wadl-staging-1.0.xml [00:05] lib/canonical/launchpad/apidoc/wadl-staging-beta.xml [00:05] lib/canonical/launchpad/apidoc/wadl-staging-devel.xml [00:05] mbarnett: oh ok [00:05] revno 9233, if that is at all interesting [00:06] mbarnett: can you run bzr merge http://bazaar.launchpad.net/~mwhudson/launchpad/bzr-git-improvements ? [00:06] if it has conflicts, please revert it [00:08] mwhudson: Warning: criss-cross merge encountered. [00:08] mbarnett: hmm [00:08] still going though [00:08] ah, 3 conflicts [00:09] i shall revert [00:09] thanks [00:09] welcoem [00:09] i guess i'll just wait until all the changes make their way through to db-stable [00:26] 10721 Launchpad Patch Queue Manager 2010-04-15 [merge] [00:26] [testfix][rs=me][ui=None] Fix some tests broken in the branch that [00:26] changes the default PPA quota [00:26] but not all of them! [00:26] guys! [00:44] mwhudson: Ew. What's left? [00:47] wgrant: lib/lp/soyuz/browser/tests/archive-views.txt [00:47] i have a branch in pqm fixing it [00:47] Ah. [00:58] * mwhudson lunches early [01:54] lifeless: can you please tag any merge queue bugs with merge-queues? [02:03] thumper: sure [02:04] ta [02:14] mwhudson: you said my 'cannot set status to 'code-failed-to-merge' bug was trivial' [02:14] mwhudson: care to point me at it ? [02:16] lifeless: i think that was thumper [02:16] ? [02:16] BranchMergeProposal.set_status [02:16] mwhudson: well, the plural 'you' [02:16] thumper: yeah [02:16] or setStatus [02:17] bug 561160 [02:17] Bug #561160: API: 'Code failed to merge' setting doesn't work [02:18] I pinged mwhudson because thumper was going out, I thought ;P [02:19] I'd just like a pointer at what to hit on, given its marked trvial [02:20] trivial for me [02:20] perhaps not trivial for you :) [02:32] thumper: so, gimme a couple pointers, and we'll see [02:33] lifeless: setStatus needs a conditional to set mergeFailed [02:37] where do the tests live ? [02:39] lp/code/model/tests [02:39] test_branchmergeproposal.py [02:45] thumper: Looks to me like setStatus has a bug, if you do setStatus(Rejected) when it was Queued, it won't dequeue [02:46] lifeless: it may well do, we haven't extensively tested queues [02:52] spm: hey, could you trigger an update of tellurium and the staging code import slaves pls? [02:52] mwhudson: sure, gimme a bit... [03:00] spm: so, we haz keyfile ? :) [03:00] spm: I'm keep to get to the point that we find new rt tickets to file [03:01] spm: at which point stopping won't block things [03:18] spm: update in progress? if you don't have time feel free to tell me to go away :-) [03:19] not yet.... [03:20] thumper: you said the tests were in lp/code/tests/test_branchmergeproposal [03:20] thumper: that seems very small? [03:20] lifeless: maybe it's lp/code/model/tests/test_branchmergeproposal.py [03:21] mwhudson: aha! [03:21] mwhudson: tellurium == codehost only? [03:22] mwhudson: and to what revno were you expecting? is currtently 9250 [03:23] spm: oh, 9250 is probably enough [03:23] spm: what is strawberry at? [03:23] cool - that's tellurium btw; haven't checked the slaves yet [03:23] snap... [03:24] 9233. OLD as the hills... or me. [03:25] spm: can you bully 9250 or newer on there? [03:25] mwhudson: tho if I update that - is that going to break if the DB isn't in sync? [03:25] spm: it doesn't talk to the database any more [03:25] okis [03:25] spm: do you have notes on why edge is so far out of date? [03:26] thumper: there's a bug somewhere..... [03:26] spm: heh [03:27] ha. No. I mean a real one. not generally/amusing. hahaha :-D [03:27] There's the CSS bug, the lp.testing bug... any others? [03:28] https://bugs.edge.launchpad.net/launchpad-foundations/+bug/559128 <== basically auto updates are suspended as they've been breaking. We even a "Be Ware!" note doing CP's that they too can go spectacularly south. [03:28] Bug #559128: edge rollouts broke CSS [03:30] plumbing emergency [03:30] AFK [03:33] ok, and how do I run a single test ? [03:35] wgrant: ^ [03:39] lifeless: I generally use bin/test -cvvt nameoftest [03:39] wgrant: what generates bin/py ? [03:40] lifeless: buildout, probably. Just run make. [03:40] ok [03:40] whats the magic to fix [03:40] Getting distribution for 'zope.testing==3.9.4'. [03:40] Error: Couldn't find a distribution for 'zope.testing==3.9.4'. [03:40] ? [03:40] Update your download-cache. [03:41] Or just rocketfuel-get [03:41] make update-your-download-cache ? [03:42] bzr up ~/launchpad/lp-sourcedeps/download-cache [03:42] rocketfuel-get will do that for you, though. [03:49] wgrant: hmm, not finding the test [03:54] lifeless: Which test, and what are you running? [03:55] its a single test in a unittest test file, not a doctest [03:55] with -ct it seems to find it [03:55] I'm running 'testr run -- -ct testname' now [04:11] spm: so are the code import slaves on a newer branch yet? [04:12] mwhudson: alas no... [04:18] wgrant: i pushed a fix for the serveOverHTTP hang to ~launchpad-committers/launchpad/python2.6 [04:22] mwhudson: did your emacs change font when upgrading to lucid? [04:23] thumper: maybe [04:24] it certainly has done on upgrade before, don't remember if the lucid upgrade did [04:25] mwhudson: because I don't like the current font but I can't remember what I had :( [04:25] thumper: heh [04:26] thumper: i seem to be using "DejaVu Sans Mono" [04:27] seems good enough I suppose [04:27] I don't feel happy with any of them [04:31] I used to use one called 'console' for terminals; bu tthe latest version of 'konsole' makes a *horrible* mess of that and a few other fonts; so switched to dejavu sans mono, which is the least worst... [04:31] mwhudson: how is your afternoon going? [04:31] thumper: ok i guess [04:32] mwhudson: need to talk or are you chugging along nicely? [04:32] thumper: still working on the no-hosted-area puller, i deleted a few too many tests [04:32] mwhudson: I can't do the qa I wanted to do [04:32] heh [04:32] thumper: argh [04:33] I could fix the issue that sinzui assigned to 10.04 [04:33] for us [05:00] thumper: mwhudson: for one of you; argh [05:00] timeout proposing the branch [05:00] https://code.edge.launchpad.net/~lifeless/launchpad/merge [05:00] ok got it in on 4th try [05:00] https://code.edge.launchpad.net/~lifeless/launchpad/merge/+merge/23521 [05:01] mwhudson: Only 'slightly horrible'? :P [05:02] lifeless: There seems to be something really really wrong with that diff. [05:02] the timeouts suck [05:02] I wonder why you are getting those [05:02] I guess because you proposed to db-devel rather than devel. [05:02] wgrant: I proposed to the thing lp suggested I should [05:02] Yeah, the development focus is not actually the focus of development, in Launchpad's case [05:03] wasn't there a thread to fix that ? [05:03] well... [05:03] the development focus is the default stacking target [05:04] wgrant: there's no monkey patching [05:04] at least [05:04] mwhudson: True. [05:05] lifeless: the timeouts are probably because the branch was still being scanned [05:05] (yay branchrevision!) [05:05] maxb: You appear to have fixed bug #526826; you might want to assign it to yourself and mark it as Fix Committed. [05:05] Bug #526826: test_ftparchive failing on Lucid [05:06] mwhudson: Is BranchRevision going to become less stupid some time soon? === Ursinha is now known as Ursinha-afk [05:11] stub: Is trusted.sql tested anywhere? [05:11] wgrant: hopefully [05:12] make schema ensures it is syntactically correct. [05:13] Individual bits get tested when those application layer bits get tested. [05:13] such as the triggers etc. [05:13] check constraints etc. not so much, as they are just a safety net (the application needs to handle validation nicely, rather than have the database throw back exceptions to the end user) [05:16] wgrant: should I repropose against devel ? [05:16] lifeless: Probably. Otherwise ec2 land won't DTRT, and the diff will make people scream. [05:17] why doesn't devel turn up in the web form ? [05:19] done [05:23] thumper: your trivial fix is done, I think. [05:29] grr fricking slow acceptance tests :( [05:30] mwhudson: Slow, or hanging? [05:31] just slow [05:36] do we have official redacted text? [05:47] thumper: Private teams used . [05:47] But there's already special text for redacted import URLs, isn't there? [05:48] * thumper shrugs [05:48] mwhudson: I'll do the merge conflict if you like [05:49] thumper: that'd be cool [06:05] fix submitted [06:08] what was the solution to SFTPError: Garbage packet received ? [06:09] using ec2 land [06:09] _ensure_ec2test_user_has_keys is in the stacktrace [06:11] thumper: merge devel [06:11] thumper: but that was quite a while ago, is this an old branch? [06:11] mwhudson: no, newly created from devel about an hour ago [06:11] or if it's branched of production-devel i guess you can run ./utilities/ec2 from devel; [06:11] ah [06:11] thumper: don't know then [06:11] no it isn't [06:11] it uses the LCA from devel and production-devel [06:12] * thumper pqm submits [06:13] thumper: I know its near EOD; if you could tell me if I did roughly the right thing - even if its not a full review - that would be awesome [06:13] which url? [06:14] https://code.edge.launchpad.net/~lifeless/launchpad/merge/+merge/23523 [06:16] thumper: ^ [06:18] commented [06:20] thanks! [06:21] thumper: why does mergeFailed not dequeue ? [06:34] * mwhudson submits no-hosted-area to ec2 test -- should be good for a laugh -- and EOWs [06:34] thumper: in fact, I'm deleting mergeFailed, because its cleaner. [06:34] thumper: if its not deleted, dequeue has to be a special case [07:03] thumper: if you haven't eod'd, I've replied. [07:25] wgrant: thanks, didn't realize there was a bug for that [07:30] good morning [07:34] maxb: What were the 2.6 pygpgme failures that you mentioned earlier? [07:34] I don't see any. [07:35] Who prepared out Python2.5 packages? I'm missing setuptools and easy_install isn't there either for the quick fix (catch-22 there I guess...) [07:36] wgrant: I submitted a branch for bumping to the existing r49 of the sourcecode/pygpgme branch, assuming it would have been tested before it got landed there. But actually, it breaks things [07:36] stub: that would be me and wgrant [07:36] maxb: Ah, so it didn't actually land? [07:37] Is it quick to get the packages made (eg. copying the packages from hardy into the PPA)? Or should I just install setuptools from source? [07:37] c/hardy/karmic [07:37] Why do you need setuptools? [07:37] I need to generate 2.5 eggs [07:37] Ah. [07:38] A no-change rebuild should do it, I think. [07:38] wgrant: specifically, pygpgme seems to be returning unicode strings instead of byte strings all over the place now, leading to doctests puking all over the place - plus some other fails :-( [07:38] Lucid? [07:38] Yup [07:38] maxb: Hm. Damn. We'd best fix that. [07:39] So I just find the Karmic package and copy it into the Launchpad PPA, with the rebuild option selected? [07:39] stub: no! [07:40] why do we need this, btw? [07:40] There is an open bug in distutils, the end result being we need to use eggs of pytz rather than tarballs. [07:41] I can't generate a new 2.5 egg of pytz at the moment because setuptools isn't installed on my box. [07:41] oh, yes. I really need to got back to upstream on that [07:41] I can just install from source if it isn't a quick fix [07:41] s/distutils/setuptools and distribute/ [07:42] Yer - first part of installing from source is working out what needs to be installed ;) [07:42] stub: Lucid has switched from setuptools to distribute. Installing setuptools on lucid would be.... potentially chaotic [07:45] I wonder if I should do a no-change upload of distribute to the ppa [07:46] Worth a try. [07:46] It won't be a dependency, so it won't make anything explode [07:47] Uh... well... [07:48] I might point out that *buildout* is involved here. [07:48] :-) [07:49] * stub lunches [07:50] * maxb lpnochange distribute [08:05] ta. I'll give that a spin when I'm back [09:00] Morning === henninge_ is now known as henninge === daniloff is now known as danilos [10:17] jml: so, how do i use subunit? [10:18] i have this test results mail and i'd like to have a list of failing tests [10:19] mwhudson, you could open it in tribunal, use subunit2pyunit or use subunit-filter [10:19] mwhudson, is there no list of failing tests in the email? [10:20] not that i can see [10:20] also subunit-filter is breaking [10:20] mwhudson, can you please forward me the email? [10:21] mwhudson, how is it breaking? [10:22] jml: http://pastebin.ubuntu.com/415439/ [10:23] jml: mail sent [10:44] mwhudson: Your hackery regarding test hangs under python2.6 is.... intriguing :-) [10:44] I am rather mystified, as I can't pin down the underlying cause of the problem appearing [10:44] maxb: yeah, i couldn't reproduce the problem in a small example [10:45] but my socket knowledge has bit rotted a bit [10:45] oh, really? I did [10:45] h=HTTPServer(); h.start_server(); h.stop_server(); h._http_thread.join() <<< the issue is that in 2.5 socket.shutdown() gets called when another thread is in accept, and accept returns when this happens [10:46] maxb: i meant 'just using the socket module' [10:46] in 2.6 the other thread is in select(), not accept() [10:46] oh, really? I didn't pursue the shutdown angle. I went straight to 'why doesn't closing the socket kick it out of select()' [10:47] and the answer seems to be 'Python has a LAAAAME wrapper that attempts to implement platform-agnostic socket.dup() which doesn't actually close the socket' [10:48] I think I might have a try at simplifying your changes - I postulate that adding a server._http_thread._sock.close() cleanup will do the job [10:48] ah [10:49] i wasn't very awake when i was digging :) [10:50] hmm. oh gosh. I wonder if the very fact we're inside a select on the socket is what keeps alive the refcount which stops the socket being closed [11:01] well, in 2.5 we're in an accept() when stop_server is called [11:08] Morning, all. [11:23] mwhudson, I'm going to spend a little time digging around your subunit failure [11:23] * jml is off the phone now [11:31] mwhudson, fuck [11:31] mwhudson, I think this is a \r\n problme [11:34] mwhudson: thanks for fixing my fsckup [11:36] jml: I responded to some of your questions on the pools LEP [11:36] bigjools, thanks. [11:36] some new mockups done [11:36] well, changed [11:38] bigjools: How inappropriate would it be to rSP bugs when closing them from a changelog? [11:38] wgrant: rSP? [11:38] removeSecurityProxy? [11:38] bigjools: removeSecurityProxy. It's already pretty much done in the normal case, where process-upload.py uses a permissive security policy. [11:38] right [11:39] my initial reaction is: nofuckingwayareyoumad? [11:39] :) [11:39] the right approach is to get the security model correct [11:39] rSP is just papering over turd-smeared walls [11:39] What is 'correct' here? [11:40] It is. [11:40] But switching to a permissive policy during a webapp request sounds approximately infinitely worse. [11:40] we need to define "correct" [11:40] When do you close bugs from a changelog from a webapp request? [11:40] queue accept [11:41] ah [11:41] deryck: around? [11:41] Why doesn't the accepting user have permission to close the bug anyway? [11:41] maxb: The bug is private. [11:41] hmm [11:41] bigjools, hi [11:42] hey deryck, can you check scrollback and give your opinion please? [11:42] (a similar thing can also now happen if a public bug is Won't Fix, although I believe all of ~ubuntu-archive has sufficient privs to avoid that problem) [11:42] * deryck looks [11:42] deryck: just 3 mins back [11:42] Bug 564491 [11:42] Bug #564491: Cannot accept package which closes inaccessible bug [11:43] wgrant, bigjools: Hi! [11:43] ;) [11:43] guten tag henninge [11:43] bigjools, do you mean how do I feel about using removeSecurityProxy? [11:43] Morning henninge. [11:43] Is bug 539185 valid? [11:43] Bug #539185: BuildBase implementation for build farm jobs [11:43] That sounds like a noodles question. [11:44] deryck: that's a no-no AFAIAC, but it would be good to see if you have any good ideas on how to deal with this situation "properly" [11:44] AIUI he is reworking that all. [11:44] he is [11:44] so I should not touch that atm? [11:44] henninge: That sounds sane. [11:44] henninge: have a chat with him, you might be able to help :) [11:45] or that ;-) [11:45] mwhudson, it's definitely a \r\n bug [11:45] bigjools, naughty, naughty boy... ;) [11:45] danilos: I know, I suck [11:45] assumptions are the mother of all fuckups, as they say [11:45] bigjools, but sure, if it needs helping, and henninge can do it while making sure it works for us as well, excellent :) [11:45] bigjools, what is the user when trying to close a bug from a changelog? I'm afraid I need to understand better. [11:45] danilos: I would have expected nothing else [11:46] danilos: well that would be beneficial to us all, yes :) [11:46] noodles775: Are you out to lunch? ;-) [11:46] deryck: let's have a call [11:46] I can explain [11:46] Speaking of the buildfarm... can we have a launchpad-buildfarm project so we can stop randomly assigning bugs between Code, Soyuz and Translations? [11:46] henninge, you are in the same country, just drop by him [11:46] ;-) [11:46] wgrant: yeah, I'll try and set that up [11:46] danilos: true, it's my turn. Last time he came by me. [11:46] wgrant, no, let's just assign them all to soyuz [11:46] bigjools, sure. skype or mumble? [11:47] * danilos puts on an innocent smile [11:47] deryck: given that I wasn't around when mumble accounts got handed out, skype :) [11:47] hopefully they'll be openid today or monday [11:47] * wgrant hastily uses launchpadlib to reassign all soyuz-build bugs to danilos. [11:47] lol [11:47] heh [11:48] bigjools, give me 2 minutes to close something here. [11:48] that API thing was a big mistake! [11:48] henninge: nope, I'm here. [11:48] deryck: ok call when you're ready, you can guess my skype id ... [11:49] mmm can't see any ash in the sky yet [11:50] I wonder when air traffic will resume. [11:50] this evening they say [11:50] but they said this morning, yesterday :) [11:50] Qantas here is saying Sunday. [11:50] depends on where each airline's planes are located I guess, they need to catch up [11:51] I guess, yeah. [11:51] But it probably depends more on what the eruption does... [11:56] we really should have "components" in the tracker [11:56] mwhudson, I guess you're beyond caring right now, but I can't track down why the critical '\r' character is being dropped [11:56] henninge, danilos, wgrant: bug updated (well, commented). [11:56] jml: +inf [11:57] noodles775, thanks [11:57] noodles775: great thanks! [11:57] * henninge goes to lunch now ... [11:57] ;-) [11:58] noodles775, that does sound great as well, thanks :) [12:00] np. [12:01] * noodles775 follows henninge === stub1 is now known as stub [12:04] noodles775: Hi. Has the ec2 of lp:~maxb/launchpad/use-pygpgme-r49 hung, or has it died and just not sent me an email? [12:12] maxb: it would've died, that's what's happened to me twice now so I expect it to not be a coincidence [12:12] I thought we fixed that issue [12:13] It's a different one probably [12:13] I have no problems with other branches, just with this specific one [12:20] jml: you might like, in your 'advocating apis sub-hat', the recent commits to lp:pqm [12:20] lifeless, thanks for the heads up [12:21] jelmer: Hi. I have run into test failures when I try to update pygpgme to the update you landed on the sourcecode branch. Do you know anything about that? [12:39] jml: do you know much about twisted's TimeoutMixin? [12:40] bigjools, I know a bit. [12:40] we appear to see it completely ignoring the timeout that was set [12:40] bigjools, show me the cdoe [12:41] and an exception ensues [12:41] jml: top of lib/lp/buildmaster/manager.py [12:41] bigjools, today's stable recent enough? [12:41] oh yes [12:41] bigjools, can you show me the error too? [12:41] yup one sec [12:42] jml: http://pastebin.ubuntu.com/415490/ [12:42] Could someone please subscribe me to bug 441039? [12:42] Bug #441039: Ubuntu One crashed on launch [12:45] bigjools, have you reproduced this on a development environment? [12:45] jml: hahaha [12:46] bigjools, I can help you write a unit test for this. [12:46] sounds good [12:46] bigjools, couple more questions first though [12:46] wgrant: you have a local builder setup, can you re-produce this? [12:46] bigjools, what is config.builddmaster.socket_timeout set to on production? [12:47] bigjools: It doesn't actually need a builder at all. But I will try it. [12:47] Oh, wait, that one does. [12:47] Not the one I thought. [12:47] it's where it ignores the timeout when scanning it [12:47] * wgrant firewalls. [12:48] jml: 180 [12:48] bigjools, how did you figure that out? [12:48] jml: I looked in the production configs [12:49] bigjools, are we sure they are the ones being used in the deployment with this error? [12:50] jml: reasonably, why? [12:50] I'll get lamont to check [12:50] bigjools, well, if it's not set, or if it's set to a value higher than the system socket timeout, then no amount of debugging will help. [12:51] system is like an hour, I thought? [12:51] but maybe it's not the default any morie [12:51] bigjools, well, as far as I can tell from that log you pasted, it is timing out roughly three minutes after the process starts [12:52] which is 180 seconds as the config says [12:52] right. [12:52] adeuring, can you subscribe me to bug 441039 please? [12:52] Bug #441039: Ubuntu One crashed on launch [12:52] so what's happening that differs from what you are expecting? [12:53] I wasn't expecting a traceback [12:53] deryck, done [12:53] bigjools: So, I get a connection timeout after 20 seconds here. It looks like the dev timeout should be 40 seconds. [12:53] but I don't know twistd well enough to know what it's supposed to do when the timeout is set [12:53] deryck: "The following errors were encountered:Deryck Hodge has already been subscribed" [12:53] jml: is there a callback from the setTimeout() call if it does time out? [12:54] adeuring, ah, ok. someone must have already. thanks. [12:54] wgrant: yeah dev is 40. ummm wtf [12:54] bigjools, yeah, timeoutConnection [12:54] bigjools, http://paste.ubuntu.com/415493/ [12:55] jml: which is not defined in the buildd-manager .... === mrevell is now known as mrevell-lunch [12:55] ho ho ho [12:55] bigjools, the default implementation disconnects [12:56] jml: ok I'm trying to work out what effect that would have on b-m [12:56] http://twistedmatrix.com/documents/current/api/twisted.protocols.policies.TimeoutMixin.html fwiw [12:56] bigjools: To be clear, I get a traceback like that, except with a normal socket exception. [12:56] ok [12:57] so I can't figure out if that's normal for the timeout that was set or if something else causes it [12:58] well, it's not an error that TimeoutMixin generates [12:58] but it's reasonable to expect xmlrpclib to explode if its socket is disconnected from under it. [12:59] we don't get the "Scanning failed with:" message so it's not hitting the scanFailed callback [12:59] I would have expected twisted to catch that and call the right callback in the Deferred [12:59] hmm [13:00] which is what we normally see [13:00] so I still wonder if it's a twistd bug [13:00] bigjools, what kind of object is "self.slave" in the traceback you pasted [13:01] heh [13:02] bigjools, nvm [13:02] this stuff is like black magic [13:02] its not that clean [13:02] ha [13:03] I know that it's an xmlrpc thingy but I can't find exactly where it's instantiated [13:03] bigjools, because it looks like it's using a non-Twisted xmlrpc client [13:03] because this code is so obtuse [13:04] I also can't find where the 'echo' method that is called in the traceback is defined [13:04] xmlrpclib.ServerProxy [13:04] in there ^ [13:04] no it's not. [13:05] ServerProxy has magic __getattr__ evil [13:05] but where's the server-side implementation that it's trying to call? [13:07] jml: see lib/canonical/buildd/slave.py [13:07] it's magic, like I sid [13:07] said [13:07] ok. [13:07] :) [13:07] XMLRPCBuildDSlave specifically [13:07] bigjools, so the thing is, we hate transparent RPC in Twisted [13:08] s/in Twisted// [13:08] bigjools, well, we hate it enough to do something about it [13:08] This is why we have the even more transparent launchpadlib...? [13:08] bigjools, your hatred is like a candle to our sun [13:08] rofl [13:09] self.slave.callRemote('echo', ...) is the way it's normally done in Twisted [13:09] maxb: hey. I only received emails about trivial-bad-httpcaller..., py2.6-warnings and tolerate-lucid-apt... It was definitely running as I checked it's status when you asked yesterday, so I'd say it died :/ [13:09] wgrant: launchpadlib makes me think of the movie Mystery Men [13:09] jml: basically this won;t work then until we make both sides use twisted's xmlrpc gubbins? [13:10] noodles775: ok. no need to resumbit, it has further problems that need addressing anyway. [13:10] bigjools, so, I _think_ what's happening here is that the twisted client with the timeout mixin is doing its job and pulling the rug out from under the non-twisted client [13:10] bigjools, but that's only a guess [13:10] bigjools, yeah, I think so. [13:10] jml: I concur with you [13:10] Why does the slave need to? [13:10] jml: I wonder if there's anything we can do on the client side to handle this better in the timeout callback [13:11] where the slave == the server side (confusing I know) [13:11] bigjools, you mean, as a work-around? [13:11] yes [13:11] bigjools, nothing leaps to mind [13:11] I have no desire to start re-writing the xmlrpc innards :( [13:12] bigjools, and as for adding another obscure workaround to this code... the phrase "house of cards" leaps to mind. [13:12] :) [13:12] the stuff at the top of lib/lp/buildmaster/model/builder.py also deals with timeouts [13:13] bigjools, you could wrap the non-twisted xmlrpc calls in a try/except block -- that'd be your quickest way out, I think. [13:13] bigjools, yeah, that'll be what's triggering the error you see. [13:13] * bigjools idly flicks the bottom card away [13:13] gah [13:13] I have no idea how this code works at all [13:14] is there anyone here in Sao Paulo who's prepared to do some kidnapping? [13:14] We really really need to refactor things a bit so RecordingSlave can DIE. [13:14] Hahah. [13:15] I know how it mostly works... but it's mostly evil. [13:15] every time I think I understand it, I promptly get confused again [13:16] RecordingSlave was a quick way to avoid doing what I hinted at earlier - re-writing the xmlrpc gubbins to be twisted [13:16] hmm [13:16] are there any bugs filed about cleaning it up? [13:16] It's not *that* hard. Just sometimes you'll get a RecordingSlave, sometimes you'll get a BuilderSlave, and sometimes it depends on who called this particular bit of code this time. And then you get fake responses which are fricking confusing and aaarrrrgh. [13:16] I might add them to my someday/maybe list. [13:16] it's basically proxying what's in BuilderSlave [13:16] you know [13:17] after I fix subunit, land the ssh server changes, make pretty graphs for bugs and make our test suite runnable more than once on a computer [13:17] can I have a flying car [13:17] Only if it's asynchronous. [13:18] * jml returns bigjools a Deferred that will fire a flying car on success [13:18] ha [13:18] actually, maybe before the runnable-more-than-once thing [13:21] jml: I wonder if we just need to remove that timeout in builder.py and wait for the twisted one to work instead [13:21] it seems like they will compete [13:21] bigjools, they'll definitely compete [13:23] zoinks [13:23] jml: this is not using twisted xmlrpc at all, RecordingSlave is just setting up a load of Deferreds [13:23] the mixin seems like a total waste of time? [13:24] bigjools, well, dispatchBuild and checkDispatch use it [13:24] I think [13:27] maxb: That distribute package just seems to have 2.6 stuff in it. No 2.5. [13:27] ugh [13:27] unfortunately, I've got a bucket load of stuff to do today. :( [13:28] It must not respond to the general python version selections presented in python-defaults and python-support [13:28] I am about to lunch. I'll take a look afterwards [13:29] jml: no worries, thanks for your time [13:30] bigjools, np. my pleasure. [13:31] jml: I think I can throw an extra catch in here to get that timeout but I've no idea how to test it! [13:37] allenap, Do you have time for a call about bug 530113? [13:37] Bug #530113: Information on a bugwatch error is obscure [13:37] gmb: In about 3 minutes (ordering computer parts, woot). [13:37] allenap, Cool, I'll sit around on t'mumble. [13:38] bigjools, I know how to test timeouts with pure Twisted systems [13:39] boo yah [13:39] the first of my four SSH branches just passed ec2 test [13:39] (they failed yesterday because devel was broken) [13:39] jml: I'll think of something hackish no doubt [13:39] jml: I apologise for breaking devel [13:40] bigjools, np. the real problem is the system that allows a simple human error to screw up other people's workflows [13:42] aye [13:50] jml: if you have a sec can you look at lib/lp/buildmaster/model/builder.py - updateBuilderStatus() [13:51] and tell me what else it needs to catch to get that timeout? [13:52] bigjools, that ought to be it... [13:53] you'd think :/ [13:53] bigjools, easy enough to figure out though. Make a TimeoutHTTPConnection work-a-like, give it a low timeout, see what error gets raised [13:54] right [13:54] shame it's not in the log :( [13:54] mmm this makes me hungry [13:54] I think it's more that socket module has crap errors. === mrevell-lunch is now known as mrevell [14:15] yay, gnome-bugs bugs are updating! [14:15] gmb, error message question for bug watch update activity.... [14:16] gmb, see bug 44082 [14:16] Bug #44082: GNOME Panel icons (on right side) move apparently randomly on session start in some situations [14:16] Urr. [14:16] deryck, its more like "oh no! mail flood!" [14:16] nigelb, for you. :-) For me, it's "oh happy day, praise be to the internet almighty." :-) [14:17] deryck, haha. [14:17] deryck, Whatsyerquestion? [14:18] gmb, so expand the first task row.... that error there? Is that from bug watch activity log? It seems out of date, since the bug is in fact updated now. [14:18] deryck, It's not from the activity log, no, it's from the last_error_type property of BugWatch. Let me just have a poke around and see if I can find out what's oging on. [14:19] gmb, cool, thanks. I want this expandable form to go away. It causes no end of grief since it gets out of sync with the rest of the app. [14:22] Is there any mechanism by which a ~launchpad-pqm sourcecode branch can be push --overwritten? [14:24] deryck, allenap: AHAHAHAHA [14:24] oh, do tell [14:24] deryck, So, that error is a real error. The status sync succeeded but the backlinking failed (and our errors aren't smart enough to handle that yet; will fix that this pm). [14:24] deryck, OOPS-1567CCW1152 [14:25] (Another win for BWA, though) [14:25] indeed! This logging is working out well. [14:25] deryck, allenap: Looks like we don't have permission to backlink on the remote system. [14:25] gmb: Oh, jolly good. [14:26] allenap, I think it might be because we (used to?) backlink for every watch, whether the LP bug was relevant or not. [14:26] So the gnome-bugs admins disabled backlinking. [14:26] gmb: That sounds right. [14:27] allenap, Is there an easy way to disable backlinking on our end to prevent the errors or is it not fine-grained enough for that? ICR. [14:28] gmb: Not fine-grained enough. [14:29] gmb: We could quite easily disable all back-linking by removing the ISupportsBackLinking interface from BugzillaAPI, but that could/would affect other trackers. [14:30] allenap, Yeah. [14:30] allenap, Though, hang on, if we can't sync comments why are we trying to backlink anyway? [14:30] That seems like huffing on the bong pipe to me. [14:31] * allenap looks [14:31] gmb: It's not conditional on sync_comments. [14:32] allenap, I think it should be. That would solve this problem for a start. [14:33] gmb: The condition, in full, is: (bug not dupe and linked to at least one bugtask and supports back-linking) [14:33] noodles775, hi, do you have any suggestions how could I QA https://code.edge.launchpad.net/~jtv/launchpad/bug-553077/+merge/22599? [14:33] * noodles775 looks [14:34] allenap, Right. Do you think that sync_comments should be used there too? I mean, I know it's a misnomer if we do that, but... [14:34] maxb: np. I worked around by installing from pypi. I then uninstalled it after generating the eggs, as buildout exploded ( https://bugs.edge.launchpad.net/launchpad-foundations/+bug/564680 ). I'm not sure if your package will cause the same issue, but maybe it is best not bothering (I can survive, and we should be on 2.6 sometime soonish) [14:34] Bug #564680: buildout explodes when distribute is installed [14:34] danilos: nope. [14:35] danilos: I mean, if the buildd's are still running, it's effectively qa'd. [14:35] But that code is not on production at the moment. [14:35] danilos: so, ensuring dogfo.. [14:36] noodles775, so, how do I get that tested on dogfood? [14:36] danilos: so, ensuring the buildd's used by dogfood are updated, and then that it still works. [14:37] danilos: as long as we ensure the buildd's are updated, we can just push a build through on df. [14:39] * noodles775 checks with lamont to get the buildd's updated on dogfood. [14:39] noodles775, cool, thanks (I totally haven't done anything about this, so I don't even know where to start) [14:42] danilos, looks like you found a bug in paramiko: line 163, "str(data[:chunk])" should be unicode(data...)? [14:43] danilos, don't know why it would start failing now though [14:45] mars, it might not be recent, I haven't used ec2 for a while [14:47] hey, paramiko uses Launchpad! :) https://edge.launchpad.net/paramiko [14:50] mars, I can file a bug, if you think that's the thing to do :) [14:59] danilos, not sure if switching to unicode would help. I just tried an experiement in ipython, and reproduced that error. It is the str() call. [15:00] str(u'Māris') raises the error [15:02] mars, right, it'd be worth seeing if it makes sense to .encode('utf-8') before passing it out to paramiko: it might be that unicode there won't help either and it'll barf later if no UTF-8 locale is used [15:04] danilos, worth a shot. Looking at the docs, and sftp should support utf-8 encoding for filenames [15:05] mars, well, UTF-8 wasn't called UTF for filenames for nothing on Plan9 I think :) [15:06] hehe [15:06] anyway, me goes get some late lunch, will be back later for a short while === danilos is now known as daniloff [15:35] rocketfuel-branch is the wrong command to use when making a one-line patch :/ [16:09] sinzui: do you have time in the next 0.5hr for a brief call regarding the style.css work? [16:09] (no problem if not). [16:09] I do [16:27] gary_poster, have a sec to review a one line patch? https://code.edge.launchpad.net/~mars/launchpad/fix-ec2-email-encoding/+merge/23557 [16:28] mars, looking [16:29] mars, I'm guessing you have empirical proof it works? :-) [16:29] gary_poster, just a traceback from danilos sent to lp-dev. This same fix is present throughout the ec2test.py file. [16:31] gary_poster, I have have danilos try it out before submitting to PQM [16:32] mars, alright. I have vague memories of pain with UTF in email headers and the stdlib email packages, but even if things don't get better, this shouldn't make anything worse (and hopefully it is a fix, of course!). Yes, would be happy if someone gave it a try before merging. Will approve with that condition, since you already have it planned. [16:33] gary_poster, ok, thanks. I'll send a mail to danilos saying as much. [16:33] cool [16:33] approved [16:58] jml`: still busy? [16:59] bigjools, now is an ok interrupt time [16:59] jml`: ok should be quick :) [16:59] see lib/lp/buildmaster/model/builder.py "resumeSlaveHost" [16:59] versus [16:59] lib/lp/buildmaster/manager.py "checkResume" [17:00] the former is called when b-m starts up and reports the stdout/stderr if it fails [17:00] the latter is asynchronous and I don't know how to do the same [17:00] * jml` pulls up the code [17:00] is stdout/err buried in response somewhere? [17:02] bigjools, is "slave" a different type of object in the two methods? [17:02] I don't think so [17:02] unless one's the recording slave and the other's the BuilderSlave [17:02] who knows :/ [17:03] so uhh [17:03] resume is what then? [17:03] actually yes checkResume is dealing with a RecordingSlave [17:03] resume is when we restart the VM [17:03] it ssh'es to the slave and calls a script [17:03] the slave's host I should say [17:04] bigjools, where is it defined? [17:04] in config [17:04] ppa_reset_command or something [17:04] anyway [17:04] my head hurts [17:04] I need to get at the output of running that - it uses run_process_with_timeout [17:05] ok let's keep it simple [17:05] checkResume gets a response from run_process_with_timeout [17:05] if the process failed I want its stdout/err [17:05] oh, it's a callback attached to run_process_with_timeout [17:05] ok, I'm on it. [17:08] bigjools, run_process_with_timeout isn't very good. [17:08] bigjools, or at least, it gives you know way of getting at stdout/stderr [17:08] * jml` sketches up some code [17:09] \o/ [17:09] :/ [17:10] if it wasn't Friday afternoon I would be sprawled on my desk in despair [17:11] run_process_with_timeout is using ProcessMonitorProtocolWithTimeout, which is actually intended for code that wants to monitor a process as it's running and report that information to some external system [17:12] i.e. not at all what you want === deryck is now known as deryck[lunch] [17:12] I guess it was used because it has a timeout [17:12] and the name seemed right [17:12] *shrug* [17:14] is this nearer the mark? http://twistedmatrix.com/documents/current/api/twisted.internet.process.Process.html [17:14] doesn't seem to handle timeouts [17:14] no... http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ProcessProtocol.html [17:15] make a subclass that accumulates stdout & stderr, fires a deferred on disconnect and mixes in TimeoutMixin [17:15] * jml` is doing a rough sketch right now [17:15] Two minutes! [17:16] you da man [17:20] bigjools, http://paste.ubuntu.com/415642/ [17:21] lookng [17:22] bigjools, ignore that function at the top [17:22] ok [17:22] the clock business is so you can write unit tests that don't rely on the system clock [17:23] jml`: looks great, thanks [17:23] you may want to inherit from ProcessProtocolWithTwoStageKill instead of ProcessProtocol, to get the improved killing behaviour [17:23] "improved"? :) [17:24] bigjools, also, you could actually define "outReceived" and "errReceived", and have them reset the timeout. that'd give a different timeout behaviour which may be more appropriate for what you're doing. [17:25] right [17:25] bigjools, "improved" as in, "send SIGINT, if it's not dead after a little while, send SIGKILL" [17:25] don't think it is, the script works in ~6 seconds [17:25] ah ok, don't really need that sophistication here I think [17:25] bigjools, lucky you :) [17:26] it's an ssh - it can die die die :) [17:26] bigjools, for the puller, we reset the timeout on activity -- doesn't matter how long a branch takes overall, just as long as we can reasonably expect it to keep making progress. [17:26] yer [17:27] * bigjools hacks [17:30] maxb: still there? [17:34] jml`: in timeoutConnection you're using error and it's not defined [17:34] * bigjools loves the vim pyflakes plugin === deryck[lunch] is now known as deryck === beuno is now known as beuno-lunch [18:01] Guten nacht [18:05] jelmer: back now === gary_poster is now known as gary-lunch === beuno-lunch is now known as beuno === dpm is now known as dpm-afk [20:06] g'night all === gary-lunch is now known as gary_poster