/srv/irclogs.ubuntu.com/2010/04/16/#launchpad-dev.txt

mbarnettmwhudson:00:03
mbarnettlp://staging/~mwhudson/linux/trunk  Started: 48 minutes ago  Last heartbeat: 14 seconds ago00:03
mbarnett2010-04-15 22:54:11 INFO    49969129 bytes transferred | Fetching revisions:Inserting stream00:03
mbarnett2010-04-15 22:55:11 INFO    43484334 bytes transferred | Fetching revisions:Inserting stream00:03
mbarnett2010-04-15 22:56:13 INFO    35991244 bytes transferred | Fetching revisions:Inserting stream00:03
mbarnett2010-04-15 22:57:14 INFO    31731071 bytes transferred | Fetching revisions:Inserting stream00:03
mwhudsonmbarnett: i meant like "log in and run bzr st", sorry00:03
mbarnettmwhudson: hah, ok00:04
mbarnettmwhudson:00:05
mbarnettimportd@strawberry:/srv/importd.staging.launchpad.net/staging/launchpad$ bzr st00:05
mbarnettunknown:00:05
mbarnett  lib/canonical/launchpad/apidoc/wadl-staging-1.0.xml00:05
mbarnett  lib/canonical/launchpad/apidoc/wadl-staging-beta.xml00:05
mbarnett  lib/canonical/launchpad/apidoc/wadl-staging-devel.xml00:05
mwhudsonmbarnett: oh ok00:05
mbarnettrevno 9233, if that is at all interesting00:05
mwhudsonmbarnett: can you run bzr merge  http://bazaar.launchpad.net/~mwhudson/launchpad/bzr-git-improvements ?00:06
mwhudsonif it has conflicts, please revert it00:06
mbarnettmwhudson: Warning: criss-cross merge encountered.00:08
mwhudsonmbarnett: hmm00:08
mbarnettstill going though00:08
mbarnettah, 3 conflicts00:08
mbarnetti shall revert00:09
mwhudsonthanks00:09
mbarnettwelcoem00:09
mwhudsoni guess i'll just wait until all the changes make their way through to db-stable00:09
mwhudson10721 Launchpad Patch Queue Manager2010-04-15 [merge]00:26
mwhudson      [testfix][rs=me][ui=None] Fix some tests broken in the branch that00:26
mwhudson      changes the default PPA quota00:26
mwhudsonbut not all of them!00:26
mwhudsonguys!00:26
wgrantmwhudson: Ew. What's left?00:44
mwhudsonwgrant:  lib/lp/soyuz/browser/tests/archive-views.txt00:47
mwhudsoni have a branch in pqm fixing it00:47
wgrantAh.00:47
* mwhudson lunches early00:58
thumperlifeless: can you please tag any merge queue bugs with merge-queues?01:54
lifelessthumper: sure02:03
thumperta02:04
lifelessmwhudson: you said my 'cannot set status to 'code-failed-to-merge' bug was trivial'02:14
lifelessmwhudson: care to point me at it ?02:14
mwhudsonlifeless: i think that was thumper02:16
thumper?02:16
thumperBranchMergeProposal.set_status02:16
lifelessmwhudson: well, the plural 'you'02:16
lifelessthumper: yeah02:16
thumperor setStatus02:16
lifelessbug 56116002:17
mupBug #561160: API: 'Code failed to merge' setting doesn't work <code-review> <merge-queues> <trivial> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/561160>02:17
lifelessI pinged mwhudson because thumper was going out, I thought ;P02:18
lifelessI'd just like a pointer at what to hit on, given its marked trvial02:19
thumpertrivial for me02:20
thumperperhaps not trivial for you :)02:20
lifelessthumper: so, gimme a couple pointers, and we'll see02:32
thumperlifeless: setStatus needs a conditional to set mergeFailed02:33
lifelesswhere do the tests live ?02:37
thumperlp/code/model/tests02:39
thumpertest_branchmergeproposal.py02:39
lifelessthumper: Looks to me like setStatus has a bug, if you do setStatus(Rejected) when it was Queued, it won't dequeue02:45
thumperlifeless: it may well do, we haven't extensively tested queues02:46
mwhudsonspm: hey, could you trigger an update of tellurium and the staging code import slaves pls?02:52
spmmwhudson: sure, gimme a bit...02:52
lifelessspm: so, we haz keyfile ? :)03:00
lifelessspm: I'm keep to get to the point that we find new rt tickets to file03:00
lifelessspm: at which point stopping won't block things03:01
mwhudsonspm: update in progress? if you don't have time feel free to tell me to go away :-)03:18
spmnot yet....03:19
lifelessthumper: you said the tests were in lp/code/tests/test_branchmergeproposal03:20
lifelessthumper: that seems very small?03:20
mwhudsonlifeless: maybe it's lp/code/model/tests/test_branchmergeproposal.py03:20
lifelessmwhudson: aha!03:21
spmmwhudson: tellurium == codehost only?03:21
spmmwhudson: and to what revno were you expecting? is currtently 925003:22
mwhudsonspm: oh, 9250 is probably enough03:23
mwhudsonspm: what is strawberry at?03:23
spmcool - that's tellurium btw; haven't checked the slaves yet03:23
spmsnap...03:23
spm9233. OLD as the hills... or me.03:24
mwhudsonspm: can you bully 9250 or newer on there?03:25
spmmwhudson: tho if I update that - is that going to break if the DB isn't in sync?03:25
mwhudsonspm: it doesn't talk to the database any more03:25
spmokis03:25
thumperspm: do you have notes on why edge is so far out of date?03:25
spmthumper: there's a bug somewhere.....03:26
thumperspm: heh03:26
spmha. No. I mean a real one. not generally/amusing. hahaha :-D03:27
wgrantThere's the CSS bug, the lp.testing bug... any others?03:27
spmhttps://bugs.edge.launchpad.net/launchpad-foundations/+bug/559128 <== basically auto updates are suspended as they've been breaking. We even a "Be Ware!" note doing CP's that they too can go spectacularly south.03:28
mupBug #559128: edge rollouts broke CSS <Launchpad Foundations:In Progress by flacoste> <https://launchpad.net/bugs/559128>03:28
thumperplumbing emergency03:30
thumperAFK03:30
lifelessok, and how do I run a single test ?03:33
lifelesswgrant: ^03:35
wgrantlifeless: I generally use bin/test -cvvt nameoftest03:39
lifelesswgrant: what generates bin/py ?03:39
wgrantlifeless: buildout, probably. Just run make.03:40
lifelessok03:40
lifelesswhats the magic to fix03:40
lifeless  Getting distribution for 'zope.testing==3.9.4'.03:40
lifelessError: Couldn't find a distribution for 'zope.testing==3.9.4'.03:40
lifeless?03:40
wgrantUpdate your download-cache.03:40
wgrantOr just rocketfuel-get03:41
lifelessmake update-your-download-cache ?03:41
wgrantbzr up ~/launchpad/lp-sourcedeps/download-cache03:42
wgrantrocketfuel-get will do that for you, though.03:42
lifelesswgrant: hmm, not finding the test03:49
wgrantlifeless: Which test, and what are you running?03:54
lifelessits a single test in a unittest test file, not a doctest03:55
lifelesswith -ct it seems to find it03:55
lifelessI'm running 'testr run -- -ct testname' now03:55
mwhudsonspm: so are the code import slaves on a newer branch yet?04:11
spmmwhudson: alas no...04:12
mwhudsonwgrant: i pushed a fix for the serveOverHTTP hang to ~launchpad-committers/launchpad/python2.604:18
thumpermwhudson: did your emacs change font when upgrading to lucid?04:22
mwhudsonthumper: maybe04:23
mwhudsonit certainly has done on upgrade before, don't remember if the lucid upgrade did04:24
thumpermwhudson: because I don't like the current font but I can't remember what I had :(04:25
mwhudsonthumper: heh04:25
mwhudsonthumper: i seem to be using "DejaVu Sans Mono"04:26
thumperseems good enough I suppose04:27
thumperI don't feel happy with any of them04:27
spmI used to use one called 'console' for terminals; bu tthe latest version of 'konsole' makes a *horrible* mess of that and a few other fonts; so switched to dejavu sans mono, which is the least worst...04:31
thumpermwhudson: how is your afternoon going?04:31
mwhudsonthumper: ok i guess04:31
thumpermwhudson: need to talk or are you chugging along nicely?04:32
mwhudsonthumper: still working on the no-hosted-area puller, i deleted a few too many tests04:32
thumpermwhudson: I can't do the qa I wanted to do04:32
thumperheh04:32
mwhudsonthumper: argh04:32
thumperI could fix the issue that sinzui assigned to 10.0404:33
thumperfor us04:33
lifelessthumper: mwhudson: for one of you; argh05:00
lifelesstimeout proposing the branch05:00
lifelesshttps://code.edge.launchpad.net/~lifeless/launchpad/merge05:00
lifelessok got it in on 4th try05:00
lifelesshttps://code.edge.launchpad.net/~lifeless/launchpad/merge/+merge/2352105:00
wgrantmwhudson: Only 'slightly horrible'? :P05:01
wgrantlifeless: There seems to be something really really wrong with that diff.05:02
thumperthe timeouts suck05:02
thumperI wonder why you are getting those05:02
wgrantI guess because you proposed to db-devel rather than devel.05:02
lifelesswgrant: I proposed to the thing lp suggested I should05:02
wgrantYeah, the development focus is not actually the focus of development, in Launchpad's case05:02
lifelesswasn't there a thread to fix that ?05:03
thumperwell...05:03
thumperthe development focus is the default stacking target05:03
mwhudsonwgrant: there's no monkey patching05:04
mwhudsonat least05:04
wgrantmwhudson: True.05:04
mwhudsonlifeless: the timeouts are probably because the branch was still being scanned05:05
mwhudson(yay branchrevision!)05:05
wgrantmaxb: You appear to have fixed bug #526826; you might want to assign it to yourself and mark it as Fix Committed.05:05
mupBug #526826: test_ftparchive failing on Lucid <soyuz-publish> <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/526826>05:05
wgrantmwhudson: Is BranchRevision going to become less stupid some time soon?05:06
=== Ursinha is now known as Ursinha-afk
wgrantstub: Is trusted.sql tested anywhere?05:11
mwhudsonwgrant: hopefully05:11
stubmake schema ensures it is syntactically correct.05:12
stubIndividual bits get tested when those application layer bits get tested.05:13
stubsuch as the triggers etc.05:13
stubcheck constraints etc. not so much, as they are just a safety net (the application needs to handle validation nicely, rather than have the database throw back exceptions to the end user)05:13
lifelesswgrant: should I repropose against devel ?05:16
wgrantlifeless: Probably. Otherwise ec2 land won't DTRT, and the diff will make people scream.05:16
lifelesswhy doesn't devel turn up in the web form ?05:17
lifelessdone05:19
lifelessthumper: your trivial fix is done, I think.05:23
mwhudsongrr fricking slow acceptance tests :(05:29
wgrantmwhudson: Slow, or hanging?05:30
mwhudsonjust slow05:31
thumperdo we have official redacted text?05:36
wgrantthumper: Private teams used <hidden>.05:47
wgrantBut there's already special text for redacted import URLs, isn't there?05:47
* thumper shrugs05:48
thumpermwhudson: I'll do the merge conflict if you like05:48
mwhudsonthumper: that'd be cool05:49
thumperfix submitted06:05
thumperwhat was the solution to SFTPError: Garbage packet received ?06:08
thumperusing ec2 land06:09
thumper_ensure_ec2test_user_has_keys is in the stacktrace06:09
mwhudsonthumper: merge devel06:11
mwhudsonthumper: but that was quite a while ago, is this an old branch?06:11
thumpermwhudson: no, newly created from devel about an hour ago06:11
mwhudsonor if it's branched of production-devel i guess you can run ./utilities/ec2 from devel;06:11
thumperah06:11
mwhudsonthumper: don't know then06:11
thumperno it isn't06:11
thumperit uses the LCA from devel and production-devel06:11
* thumper pqm submits06:12
lifelessthumper: I know its near EOD; if you could tell me if I did roughly the right thing - even if its not a full review - that would be awesome06:13
thumperwhich url?06:13
lifelesshttps://code.edge.launchpad.net/~lifeless/launchpad/merge/+merge/2352306:14
lifelessthumper: ^06:16
thumpercommented06:18
lifelessthanks!06:20
lifelessthumper: why does mergeFailed not dequeue ?06:21
* mwhudson submits no-hosted-area to ec2 test -- should be good for a laugh -- and EOWs06:34
lifelessthumper: in fact, I'm deleting mergeFailed, because its cleaner.06:34
lifelessthumper: if its not deleted, dequeue has to be a special case06:34
lifelessthumper: if you haven't eod'd, I've replied.07:03
maxbwgrant: thanks, didn't realize there was a bug for that07:25
adeuringgood morning07:30
wgrantmaxb: What were the 2.6 pygpgme failures that you mentioned earlier?07:34
wgrantI don't see any.07:34
stubWho prepared out Python2.5 packages? I'm missing setuptools and easy_install isn't there either for the quick fix (catch-22 there I guess...)07:35
maxbwgrant: I submitted a branch for bumping to the existing r49 of the sourcecode/pygpgme branch, assuming it would have been tested before it got landed there. But actually, it breaks things07:36
maxbstub: that would be me and wgrant07:36
wgrantmaxb: Ah, so it didn't actually land?07:36
stubIs it quick to get the packages made (eg. copying the packages from hardy into the PPA)? Or should I just install setuptools from source?07:37
stubc/hardy/karmic07:37
wgrantWhy do you need setuptools?07:37
stubI need to generate 2.5 eggs07:37
wgrantAh.07:37
wgrantA no-change rebuild should do it, I think.07:38
maxbwgrant: specifically, pygpgme seems to be returning unicode strings instead of byte strings all over the place now, leading to doctests puking all over the place - plus some other fails :-(07:38
wgrantLucid?07:38
stubYup07:38
wgrantmaxb: Hm. Damn. We'd best fix that.07:38
stubSo I just find the Karmic package and copy it into the Launchpad PPA, with the rebuild option selected?07:39
maxbstub: no!07:39
maxbwhy do we need this, btw?07:40
stubThere is an open bug in distutils, the end result being we need to use eggs of pytz rather than tarballs.07:40
stubI can't generate a new 2.5 egg of pytz at the moment because setuptools isn't installed on my box.07:41
maxboh, yes. I really need to got back to upstream on that07:41
stubI can just install from source if it isn't a quick fix07:41
maxbs/distutils/setuptools and distribute/07:41
stubYer - first part of installing from source is working out what needs to be installed ;)07:42
maxbstub: Lucid has switched from setuptools to distribute. Installing setuptools on lucid would be.... potentially chaotic07:42
maxbI wonder if I should do a no-change upload of distribute to the ppa07:45
wgrantWorth a try.07:46
stubIt won't be a dependency, so it won't make anything explode07:46
wgrantUh... well...07:47
wgrantI might point out that *buildout* is involved here.07:48
maxb:-)07:48
* stub lunches07:49
* maxb lpnochange distribute07:50
stubta. I'll give that a spin when I'm back08:05
mrevellMorning09:00
=== henninge_ is now known as henninge
=== daniloff is now known as danilos
mwhudsonjml: so, how do i use subunit?10:17
mwhudsoni have this test results mail and i'd like to have a list of failing tests10:18
jmlmwhudson, you could open it in tribunal, use subunit2pyunit or use subunit-filter10:19
jmlmwhudson, is there no list of failing tests in the email?10:19
mwhudsonnot that i can see10:20
mwhudsonalso subunit-filter is breaking10:20
jmlmwhudson, can you please forward me the email?10:20
jmlmwhudson, how is it breaking?10:21
mwhudsonjml: http://pastebin.ubuntu.com/415439/10:22
mwhudsonjml: mail sent10:23
maxbmwhudson: Your hackery regarding test hangs under python2.6 is.... intriguing :-)10:44
maxbI am rather mystified, as I can't pin down the underlying cause of the problem appearing10:44
mwhudsonmaxb: yeah, i couldn't reproduce the problem in a small example10:44
mwhudsonbut my socket knowledge has bit rotted a bit10:45
maxboh, really? I did10:45
maxbh=HTTPServer(); h.start_server(); h.stop_server(); h._http_thread.join()  <<<<HANG10:45
mwhudsonthe issue is that in 2.5 socket.shutdown() gets called when another thread is in accept, and accept returns when this happens10:45
mwhudsonmaxb: i meant 'just using the socket module'10:46
mwhudsonin 2.6 the other thread is in select(), not accept()10:46
maxboh, really? I didn't pursue the shutdown angle. I went straight to 'why doesn't closing the socket kick it out of select()'10:46
maxband the answer seems to be 'Python has a LAAAAME wrapper that attempts to implement platform-agnostic socket.dup() which doesn't actually close the socket'10:47
maxbI think I might have a try at simplifying your changes - I postulate that adding a server._http_thread._sock.close() cleanup will do the job10:48
mwhudsonah10:48
mwhudsoni wasn't very awake when i was digging :)10:49
maxbhmm. oh gosh. I wonder if the very fact we're inside a select on the socket is what keeps alive the refcount which stops the socket being closed10:50
mwhudsonwell, in 2.5 we're in an accept() when stop_server is called11:01
deryckMorning, all.11:08
jmlmwhudson, I'm going to spend a little time digging around your subunit failure11:23
* jml is off the phone now11:23
jmlmwhudson, fuck11:31
jmlmwhudson, I think this is a \r\n problme11:31
bigjoolsmwhudson: thanks for fixing my fsckup11:34
bigjoolsjml: I responded to some of your questions on the pools LEP11:36
jmlbigjools, thanks.11:36
bigjoolssome new mockups done11:36
bigjoolswell, changed11:36
wgrantbigjools: How inappropriate would it be to rSP bugs when closing them from a changelog?11:38
bigjoolswgrant: rSP?11:38
maxbremoveSecurityProxy?11:38
wgrantbigjools: removeSecurityProxy. It's already pretty much done in the normal case, where process-upload.py uses a permissive security policy.11:38
bigjoolsright11:38
bigjoolsmy initial reaction is: nofuckingwayareyoumad?11:39
bigjools:)11:39
bigjoolsthe right approach is to get the security model correct11:39
bigjoolsrSP is just papering over turd-smeared walls11:39
wgrantWhat is 'correct' here?11:39
wgrantIt is.11:40
wgrantBut switching to a permissive policy during a webapp request sounds approximately infinitely worse.11:40
bigjoolswe need to define "correct"11:40
maxbWhen do you close bugs from a changelog from a webapp request?11:40
bigjoolsqueue accept11:40
maxbah11:41
bigjoolsderyck: around?11:41
maxbWhy doesn't the accepting user have permission to close the bug anyway?11:41
wgrantmaxb: The bug is private.11:41
maxbhmm11:41
deryckbigjools, hi11:41
bigjoolshey deryck, can you check scrollback and give your opinion please?11:42
wgrant(a similar thing can also now happen if a public bug is Won't Fix, although I believe all of ~ubuntu-archive has sufficient privs to avoid that problem)11:42
* deryck looks11:42
bigjoolsderyck: just 3 mins back11:42
wgrantBug 56449111:42
mupBug #564491: Cannot accept package which closes inaccessible bug <oops> <queue-page> <ui> <Soyuz:Triaged> <https://launchpad.net/bugs/564491>11:42
henningewgrant, bigjools: Hi!11:43
henninge;)11:43
bigjoolsguten tag henninge11:43
deryckbigjools, do you mean how do I feel about using  removeSecurityProxy?11:43
wgrantMorning henninge.11:43
henningeIs bug 539185 valid?11:43
mupBug #539185: BuildBase implementation for build farm jobs <buildfarm> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/539185>11:43
wgrantThat sounds like a noodles question.11:43
bigjoolsderyck: that's a no-no AFAIAC, but it would be good to see if you have any good ideas on how to deal with this situation "properly"11:44
wgrantAIUI he is reworking that all.11:44
bigjoolshe is11:44
henningeso I should not touch that atm?11:44
wgranthenninge: That sounds sane.11:44
bigjoolshenninge: have a chat with him, you might be able to help :)11:44
henningeor that ;-)11:45
jmlmwhudson, it's definitely a \r\n bug11:45
danilosbigjools, naughty, naughty boy... ;)11:45
bigjoolsdanilos: I know, I suck11:45
bigjoolsassumptions are the mother of all fuckups, as they say11:45
danilosbigjools, but sure, if it needs helping, and henninge can do it while making sure it works for us as well, excellent :)11:45
deryckbigjools, what is the user when trying to close a bug from a changelog?  I'm afraid I need to understand better.11:45
henningedanilos: I would have expected nothing else11:45
bigjoolsdanilos: well that would be beneficial to us all, yes :)11:46
henningenoodles775: Are you out to lunch? ;-)11:46
bigjoolsderyck: let's have a call11:46
bigjoolsI can explain11:46
wgrantSpeaking of the buildfarm... can we have a launchpad-buildfarm project so we can stop randomly assigning bugs between Code, Soyuz and Translations?11:46
daniloshenninge, you are in the same country, just drop by him11:46
henninge;-)11:46
bigjoolswgrant: yeah, I'll try and set that up11:46
henningedanilos: true, it's my turn. Last time he came by me.11:46
daniloswgrant, no, let's just assign them all to soyuz11:46
deryckbigjools, sure.  skype or mumble?11:46
* danilos puts on an innocent smile11:47
bigjoolsderyck: given that I wasn't around when mumble accounts got handed out, skype :)11:47
bigjoolshopefully they'll be openid today or monday11:47
* wgrant hastily uses launchpadlib to reassign all soyuz-build bugs to danilos.11:47
bigjoolslol11:47
danilosheh11:47
deryckbigjools, give me 2 minutes to close something here.11:48
danilosthat API thing was a big mistake!11:48
noodles775henninge: nope, I'm here.11:48
bigjoolsderyck: ok call when you're ready, you can guess my skype id ...11:48
bigjoolsmmm can't see any ash in the sky yet11:49
wgrantI wonder when air traffic will resume.11:50
bigjoolsthis evening they say11:50
bigjoolsbut they said this morning, yesterday :)11:50
wgrantQantas here is saying Sunday.11:50
bigjoolsdepends on where each airline's planes are located I guess, they need to catch up11:50
wgrantI guess, yeah.11:51
wgrantBut it probably depends more on what the eruption does...11:51
jmlwe really should have "components" in the tracker11:56
jmlmwhudson, I guess you're beyond caring right now, but I can't track down why the critical '\r' character is being dropped11:56
noodles775henninge, danilos, wgrant: bug updated (well, commented).11:56
wgrantjml: +inf11:56
danilosnoodles775, thanks11:57
henningenoodles775: great thanks!11:57
* henninge goes to lunch now ...11:57
henninge;-)11:57
danilosnoodles775, that does sound great as well, thanks :)11:58
noodles775np.12:00
* noodles775 follows henninge 12:01
=== stub1 is now known as stub
maxbnoodles775: Hi. Has the ec2 of lp:~maxb/launchpad/use-pygpgme-r49 hung, or has it died and just not sent me an email?12:04
jelmermaxb: it would've died, that's what's happened to me twice now so I expect it to not be a coincidence12:12
jmlI thought we fixed that issue12:12
jelmerIt's a different one probably12:13
jelmerI have no problems with other branches, just with this specific one12:13
lifelessjml: you might like, in your 'advocating apis sub-hat', the recent commits to lp:pqm12:20
jmllifeless, thanks for the heads up12:20
maxbjelmer: Hi. I have run into test failures when I try to update pygpgme to the update you landed on the sourcecode branch. Do you know anything about that?12:21
bigjoolsjml: do you know much about twisted's TimeoutMixin?12:39
jmlbigjools, I know a bit.12:40
bigjoolswe appear to see it completely ignoring the timeout that was set12:40
jmlbigjools, show me the cdoe12:40
bigjoolsand an exception ensues12:41
bigjoolsjml: top of lib/lp/buildmaster/manager.py12:41
jmlbigjools, today's stable recent enough?12:41
bigjoolsoh yes12:41
jmlbigjools, can you show me the error too?12:41
bigjoolsyup one sec12:41
bigjoolsjml: http://pastebin.ubuntu.com/415490/12:42
deryckCould someone please subscribe me to bug 441039?12:42
mupBug #441039: Ubuntu One crashed on launch <apport-crash> <i386> <ubuntuone-karmic> <Android's Fortune:Invalid> <NULL Project:Invalid> <Ubuntu One Client:Fix Released by dobey> <ubuntuone-client (Ubuntu):Fix Committed by dobey> <https://launchpad.net/bugs/441039>12:42
jmlbigjools, have you reproduced this on a development environment?12:45
bigjoolsjml: hahaha12:45
jmlbigjools, I can help you write a unit test for this.12:46
bigjoolssounds good12:46
jmlbigjools, couple more questions first though12:46
bigjoolswgrant: you have a local builder setup, can you re-produce this?12:46
jmlbigjools, what is config.builddmaster.socket_timeout set to on production?12:46
wgrantbigjools: It doesn't actually need a builder at all. But I will try it.12:47
wgrantOh, wait, that one does.12:47
wgrantNot the one I thought.12:47
bigjoolsit's where it ignores the timeout when scanning it12:47
* wgrant firewalls.12:47
bigjoolsjml: 18012:48
jmlbigjools, how did you figure that out?12:48
bigjoolsjml: I looked in the production configs12:48
jmlbigjools, are we sure they are the ones being used in the deployment with this error?12:49
bigjoolsjml: reasonably, why?12:50
bigjoolsI'll get lamont to check12:50
jmlbigjools, well, if it's not set, or if it's set to a value higher than the system socket timeout, then no amount of debugging will help.12:50
bigjoolssystem is like an hour, I thought?12:51
bigjoolsbut maybe it's not the default any morie12:51
jmlbigjools, well, as far as I can tell from that log you pasted, it is timing out roughly three minutes after the process starts12:51
bigjoolswhich is 180 seconds as the config says12:52
jmlright.12:52
deryckadeuring, can you subscribe me to bug 441039 please?12:52
mupBug #441039: Ubuntu One crashed on launch <apport-crash> <i386> <ubuntuone-karmic> <Android's Fortune:Invalid> <NULL Project:Invalid> <Ubuntu One Client:Fix Released by dobey> <ubuntuone-client (Ubuntu):Fix Committed by dobey> <https://launchpad.net/bugs/441039>12:52
jmlso what's happening that differs from what you are expecting?12:52
bigjoolsI wasn't expecting a traceback12:53
jmlderyck, done12:53
wgrantbigjools: So, I get a connection timeout after 20 seconds here. It looks like the dev timeout should be 40 seconds.12:53
bigjoolsbut I don't know twistd well enough to know what it's supposed to do when the timeout is set12:53
adeuringderyck: "The following errors were encountered:Deryck Hodge has already been subscribed"12:53
bigjoolsjml: is there a callback from the setTimeout() call if it does time out?12:53
deryckadeuring, ah, ok.  someone must have already.  thanks.12:54
bigjoolswgrant: yeah dev is 40. ummm wtf12:54
jmlbigjools, yeah, timeoutConnection12:54
jmlbigjools, http://paste.ubuntu.com/415493/12:54
bigjoolsjml: which is not defined in the buildd-manager ....12:55
=== mrevell is now known as mrevell-lunch
bigjoolsho ho ho12:55
jmlbigjools, the default implementation disconnects12:55
bigjoolsjml: ok I'm trying to work out what effect that would have on b-m12:56
jmlhttp://twistedmatrix.com/documents/current/api/twisted.protocols.policies.TimeoutMixin.html fwiw12:56
wgrantbigjools: To be clear, I get a traceback like that, except with a normal socket exception.12:56
bigjoolsok12:56
bigjoolsso I can't figure out if that's normal for the timeout that was set or if something else causes it12:57
jmlwell, it's not an error that TimeoutMixin generates12:58
jmlbut it's reasonable to expect xmlrpclib to explode if its socket is disconnected from under it.12:58
bigjoolswe don't get the "Scanning failed with:" message so it's not hitting the scanFailed callback12:59
bigjoolsI would have expected twisted to catch that and call the right callback in the Deferred12:59
jmlhmm12:59
bigjoolswhich is what we normally see13:00
bigjoolsso I still wonder if it's a twistd bug13:00
jmlbigjools, what kind of object is "self.slave" in the traceback you pasted13:00
bigjoolsheh13:01
jmlbigjools, nvm13:02
bigjoolsthis stuff is like black magic13:02
lifelessits not that clean13:02
bigjoolsha13:02
bigjoolsI know that it's an xmlrpc thingy but I can't find exactly where it's instantiated13:03
jmlbigjools, because it looks like it's using a non-Twisted xmlrpc client13:03
bigjoolsbecause this code is so obtuse13:03
jmlI also can't find where the 'echo' method that is called in the traceback is defined13:04
bigjoolsxmlrpclib.ServerProxy13:04
bigjoolsin there ^13:04
jmlno it's not.13:04
jmlServerProxy has magic __getattr__ evil13:05
jmlbut where's the server-side implementation that it's trying to call?13:05
bigjoolsjml: see lib/canonical/buildd/slave.py13:07
bigjoolsit's magic, like I sid13:07
bigjoolssaid13:07
jmlok.13:07
bigjools:)13:07
bigjoolsXMLRPCBuildDSlave specifically13:07
jmlbigjools, so the thing is, we hate transparent RPC in Twisted13:07
bigjoolss/in Twisted//13:08
jmlbigjools, well, we hate it enough to do something about it13:08
wgrantThis is why we have the even more transparent launchpadlib...?13:08
jmlbigjools, your hatred is like a candle to our sun13:08
bigjoolsrofl13:08
jmlself.slave.callRemote('echo', ...) is the way it's normally done in Twisted13:09
noodles775maxb: hey. I only received emails about trivial-bad-httpcaller..., py2.6-warnings and tolerate-lucid-apt... It was definitely running as I checked it's status when you asked yesterday, so I'd say it died :/13:09
lifelesswgrant: launchpadlib makes me think of the movie Mystery Men13:09
bigjoolsjml: basically this won;t work then until we make both sides use twisted's xmlrpc gubbins?13:09
maxbnoodles775: ok. no need to resumbit, it has further problems that need addressing anyway.13:10
jmlbigjools, so, I _think_ what's happening here is that the twisted client with the timeout mixin is doing its job and pulling the rug out from under the non-twisted client13:10
jmlbigjools, but that's only a guess13:10
jmlbigjools, yeah, I think so.13:10
bigjoolsjml: I concur with you13:10
wgrantWhy does the slave need to?13:10
bigjoolsjml: I wonder if there's anything we can do on the client side to handle this better in the timeout callback13:10
bigjoolswhere the slave == the server side (confusing I know)13:11
jmlbigjools, you mean, as a work-around?13:11
bigjoolsyes13:11
jmlbigjools, nothing leaps to mind13:11
bigjoolsI have no desire to start re-writing the xmlrpc innards :(13:11
jmlbigjools, and as for adding another obscure workaround to this code... the phrase "house of cards" leaps to mind.13:12
bigjools:)13:12
bigjoolsthe stuff at the top of lib/lp/buildmaster/model/builder.py also deals with timeouts13:12
jmlbigjools, you could wrap the non-twisted xmlrpc calls in a try/except block -- that'd be your quickest way out, I think.13:13
jmlbigjools, yeah, that'll be what's triggering the error you see.13:13
* bigjools idly flicks the bottom card away13:13
jmlgah13:13
jmlI have no idea how this code works at all13:13
bigjoolsis there anyone here in Sao Paulo who's prepared to do some kidnapping?13:14
wgrantWe really really need to refactor things a bit so RecordingSlave can DIE.13:14
wgrantHahah.13:14
wgrantI know how it mostly works... but it's mostly evil.13:15
bigjoolsevery time I think I understand it, I promptly get confused again13:15
bigjoolsRecordingSlave was a quick way to avoid doing what I hinted at earlier - re-writing the xmlrpc gubbins to be twisted13:16
jmlhmm13:16
jmlare there any bugs filed about cleaning it up?13:16
wgrantIt's not *that* hard. Just sometimes you'll get a RecordingSlave, sometimes you'll get a BuilderSlave, and sometimes it depends on who called this particular bit of code this time. And then you get fake responses which are fricking confusing and aaarrrrgh.13:16
jmlI might add them to my someday/maybe list.13:16
bigjoolsit's basically proxying what's in BuilderSlave13:16
jmlyou know13:16
jmlafter I fix subunit, land the ssh server changes, make pretty graphs for bugs and make our test suite runnable more than once on a computer13:17
bigjoolscan I have a flying car13:17
wgrantOnly if it's asynchronous.13:17
* jml returns bigjools a Deferred that will fire a flying car on success13:18
bigjoolsha13:18
jmlactually, maybe before the runnable-more-than-once thing13:18
bigjoolsjml: I wonder if we just need to remove that timeout in builder.py and wait for the twisted one to work instead13:21
bigjoolsit seems like they will compete13:21
jmlbigjools, they'll definitely compete13:21
jmlzoinks13:23
bigjoolsjml: this is not using twisted xmlrpc at all, RecordingSlave is just setting up a load of Deferreds13:23
bigjoolsthe mixin seems like a total waste of time?13:23
jmlbigjools, well, dispatchBuild and checkDispatch use it13:24
jmlI think13:24
stubmaxb: That distribute package just seems to have 2.6 stuff in it. No 2.5.13:27
maxbugh13:27
jmlunfortunately, I've got a bucket load of stuff to do today. :(13:27
maxbIt must not respond to the general python version selections presented in python-defaults and python-support13:28
maxbI am about to lunch. I'll take a look afterwards13:28
bigjoolsjml: no worries, thanks for your time13:29
jmlbigjools, np. my pleasure.13:30
bigjoolsjml: I think I can throw an extra catch in here to get that timeout but I've no idea how to test it!13:31
gmballenap, Do you have time for a call about bug 530113?13:37
mupBug #530113: Information on a bugwatch error is obscure <story-reliable-bug-syncing> <Launchpad Bugs:In Progress by gmb> <https://launchpad.net/bugs/530113>13:37
allenapgmb: In about 3 minutes (ordering computer parts, woot).13:37
gmballenap, Cool, I'll sit around on t'mumble.13:37
jmlbigjools, I know how to test timeouts with pure Twisted systems13:38
jmlboo yah13:39
jmlthe first of my four SSH branches just passed ec2 test13:39
jml(they failed yesterday because devel was broken)13:39
bigjoolsjml: I'll think of something hackish no doubt13:39
bigjoolsjml: I apologise for breaking devel13:39
jmlbigjools, np. the real problem is the system that allows a simple human error to screw up other people's workflows13:40
bigjoolsaye13:42
bigjoolsjml: if you have a sec can you look at lib/lp/buildmaster/model/builder.py -  updateBuilderStatus()13:50
bigjoolsand tell me what else it needs to catch to get that timeout?13:51
jmlbigjools, that ought to be it...13:52
bigjoolsyou'd think :/13:53
jmlbigjools, easy enough to figure out though. Make a TimeoutHTTPConnection work-a-like, give it a low timeout, see what error gets raised13:53
bigjoolsright13:54
bigjoolsshame it's not in the log :(13:54
bigjoolsmmm this makes me hungry13:54
jmlI think it's more that socket module has crap errors.13:54
=== mrevell-lunch is now known as mrevell
deryckyay, gnome-bugs bugs are updating!14:15
deryckgmb, error message question for bug watch update activity....14:15
deryckgmb, see bug 4408214:16
mupBug #44082: GNOME Panel icons (on right side) move apparently randomly on session start in some situations <gloam> <qa-hardy-desktop> <qa-jaunty-desktop> <qa-karmic-desktop> <GNOME Panel:Fix Released> <One Hundred Paper Cuts:Confirmed for ryan.maki> <gnome-panel (Ubuntu):Confirmed for desktop-bugs> <gnome-panel (Ubuntu Hardy):Triaged by desktop-bugs> <https://launchpad.net/bugs/44082>14:16
gmbUrr.14:16
nigelbderyck, its more like "oh no! mail flood!"14:16
derycknigelb, for you. :-)  For me, it's "oh happy day, praise be to the internet almighty." :-)14:16
nigelbderyck, haha.14:17
gmbderyck, Whatsyerquestion?14:17
deryckgmb, so expand the first task row.... that error there?  Is that from bug watch activity log?  It seems out of date, since the bug is in fact updated now.14:18
gmbderyck, It's not from the activity log, no, it's from the last_error_type property of BugWatch. Let me just have a poke around and see if I can find out what's oging on.14:18
deryckgmb, cool, thanks.  I want this expandable form to go away.  It causes no end of grief since it gets out of sync with the rest of the app.14:19
maxbIs there any mechanism by which a ~launchpad-pqm sourcecode branch can be push --overwritten?14:22
gmbderyck, allenap: AHAHAHAHA14:24
deryckoh, do tell14:24
gmbderyck, So, that error is a real error. The status sync succeeded but the backlinking failed (and our errors aren't smart enough to handle that yet; will fix that this pm).14:24
gmbderyck, OOPS-1567CCW115214:24
gmb(Another win for BWA, though)14:25
deryckindeed!  This logging is working out well.14:25
gmbderyck, allenap: Looks like we don't have permission to backlink on the remote system.14:25
allenapgmb: Oh, jolly good.14:25
gmballenap, I think it might be because we (used to?) backlink for every watch, whether the LP bug was relevant or not.14:26
gmbSo the gnome-bugs admins disabled backlinking.14:26
allenapgmb: That sounds right.14:26
gmballenap, Is there an easy way to disable backlinking on our end to prevent the errors or is it not fine-grained enough for that? ICR.14:27
allenapgmb: Not fine-grained enough.14:28
allenapgmb: We could quite easily disable all back-linking by removing the ISupportsBackLinking interface from BugzillaAPI, but that could/would affect other trackers.14:29
gmballenap, Yeah.14:30
gmballenap, Though, hang on, if we can't sync comments why are we trying to backlink anyway?14:30
gmbThat seems like huffing on the bong pipe to me.14:30
* allenap looks14:31
allenapgmb: It's not conditional on sync_comments.14:31
gmballenap, I think it should be. That would solve this problem for a start.14:32
allenapgmb: The condition, in full, is: (bug not dupe and linked to at least one bugtask and supports back-linking)14:33
danilosnoodles775, hi, do you have any suggestions how could I QA https://code.edge.launchpad.net/~jtv/launchpad/bug-553077/+merge/22599?14:33
* noodles775 looks14:33
gmballenap, Right. Do you think that sync_comments should be used there too? I mean, I know it's a misnomer if we do that, but...14:34
stubmaxb: np. I worked around by installing from pypi. I then uninstalled it after generating the eggs, as buildout exploded ( https://bugs.edge.launchpad.net/launchpad-foundations/+bug/564680 ). I'm not sure if your package will cause the same issue, but maybe it is best not bothering (I can survive, and we should be on 2.6 sometime soonish)14:34
mupBug #564680: buildout explodes when distribute is installed <Launchpad Foundations:New> <https://launchpad.net/bugs/564680>14:34
noodles775danilos: nope.14:34
noodles775danilos: I mean, if the buildd's are still running, it's effectively qa'd.14:35
wgrantBut that code is not on production at the moment.14:35
noodles775danilos: so, ensuring dogfo..14:35
danilosnoodles775, so, how do I get that tested on dogfood?14:36
noodles775danilos: so, ensuring the buildd's used by dogfood are updated, and then that it still works.14:36
noodles775danilos: as long as we ensure the buildd's are updated, we can just push a build through on df.14:37
* noodles775 checks with lamont to get the buildd's updated on dogfood.14:39
danilosnoodles775, cool, thanks (I totally haven't done anything about this, so I don't even know where to start)14:39
marsdanilos, looks like you found a bug in paramiko: line 163, "str(data[:chunk])" should be unicode(data...)?14:42
marsdanilos, don't know why it would start failing now though14:43
danilosmars, it might not be recent, I haven't used ec2 for a while14:45
marshey, paramiko uses Launchpad! :)  https://edge.launchpad.net/paramiko14:47
danilosmars, I can file a bug, if you think that's the thing to do :)14:50
marsdanilos, not sure if switching to unicode would help.  I just tried an experiement in ipython, and reproduced that error.  It is the str() call.14:59
marsstr(u'Māris')  raises the error15:00
danilosmars, right, it'd be worth seeing if it makes sense to .encode('utf-8') before passing it out to paramiko: it might be that unicode there won't help either and it'll barf later if no UTF-8 locale is used15:02
marsdanilos, worth a shot.  Looking at the docs, and sftp should support utf-8 encoding for filenames15:04
danilosmars, well, UTF-8 wasn't called UTF for filenames for nothing on Plan9 I think :)15:05
marshehe15:06
danilosanyway, me goes get some late lunch, will be back later for a short while15:06
=== danilos is now known as daniloff
marsrocketfuel-branch is the wrong command to use when making a one-line patch :/15:35
noodles775sinzui: do you have time in the next 0.5hr for a brief call regarding the style.css work?16:09
noodles775(no problem if not).16:09
sinzuiI do16:09
marsgary_poster, have a sec to review a one line patch?  https://code.edge.launchpad.net/~mars/launchpad/fix-ec2-email-encoding/+merge/2355716:27
gary_postermars, looking16:28
gary_postermars, I'm guessing you have empirical proof it works? :-)16:29
marsgary_poster, just a traceback from danilos sent to lp-dev.  This same fix is present throughout the ec2test.py file.16:29
marsgary_poster, I have have danilos try it out before submitting to PQM16:31
gary_postermars, alright.  I have vague memories of pain with UTF in email headers and the stdlib email packages, but even if things don't get better, this shouldn't make anything worse (and hopefully it is a fix, of course!).  Yes, would be happy if someone gave it a try before merging.  Will approve with that condition, since you already have it planned.16:32
marsgary_poster, ok, thanks.  I'll send a mail to danilos saying as much.16:33
gary_postercool16:33
gary_posterapproved16:33
bigjoolsjml`: still busy?16:58
jml`bigjools, now is an ok interrupt time16:59
bigjoolsjml`: ok should be quick :)16:59
bigjoolssee lib/lp/buildmaster/model/builder.py "resumeSlaveHost"16:59
bigjoolsversus16:59
bigjoolslib/lp/buildmaster/manager.py "checkResume"16:59
bigjoolsthe former is called when b-m starts up and reports the stdout/stderr if it fails17:00
bigjoolsthe latter is asynchronous and I don't know how to do the same17:00
* jml` pulls up the code17:00
bigjoolsis stdout/err buried in response somewhere?17:00
jml`bigjools, is "slave" a different type of object in the two methods?17:02
bigjoolsI don't think so17:02
bigjoolsunless one's the recording slave and the other's the BuilderSlave17:02
bigjoolswho knows :/17:02
jml`so uhh17:03
jml`resume is what then?17:03
bigjoolsactually yes checkResume is dealing with a RecordingSlave17:03
bigjoolsresume is when we restart the VM17:03
bigjoolsit ssh'es to the slave and calls a script17:03
bigjoolsthe slave's host I should say17:03
jml`bigjools, where is it defined?17:04
bigjoolsin config17:04
bigjoolsppa_reset_command or something17:04
bigjoolsanyway17:04
jml`my head hurts17:04
bigjoolsI need to get at the output of running that - it uses run_process_with_timeout17:04
bigjoolsok let's keep it simple17:05
bigjoolscheckResume gets a response from run_process_with_timeout17:05
bigjoolsif the process failed I want its stdout/err17:05
jml`oh, it's a callback attached to run_process_with_timeout17:05
jml`ok, I'm on it.17:05
jml`bigjools, run_process_with_timeout isn't very good.17:08
jml`bigjools, or at least, it gives you know way of getting at stdout/stderr17:08
* jml` sketches up some code17:08
bigjools\o/17:09
bigjools:/17:09
bigjoolsif it wasn't Friday afternoon I would be sprawled on my desk in despair17:10
jml`run_process_with_timeout is using ProcessMonitorProtocolWithTimeout, which is actually intended for code that wants to monitor a process as it's running and report that information to some external system17:11
jml`i.e. not at all what you want17:12
=== deryck is now known as deryck[lunch]
bigjoolsI guess it was used because it has a timeout17:12
bigjoolsand the name seemed right17:12
bigjools*shrug*17:12
bigjoolsis this nearer the mark? http://twistedmatrix.com/documents/current/api/twisted.internet.process.Process.html17:14
bigjoolsdoesn't seem to handle timeouts17:14
jml`no... http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ProcessProtocol.html17:14
jml`make a subclass that accumulates stdout & stderr, fires a deferred on disconnect and mixes in TimeoutMixin17:15
* jml` is doing a rough sketch right now17:15
jml`Two minutes!17:15
bigjoolsyou da man17:16
jml`bigjools, http://paste.ubuntu.com/415642/17:20
bigjoolslookng17:21
jml`bigjools, ignore that function at the top17:22
bigjoolsok17:22
jml`the clock business is so you can write unit tests that don't rely on the system clock17:22
bigjoolsjml`: looks great, thanks17:23
jml`you may want to inherit from ProcessProtocolWithTwoStageKill instead of ProcessProtocol, to get the improved killing behaviour17:23
bigjools"improved"? :)17:23
jml`bigjools, also, you could actually define "outReceived" and "errReceived", and have them reset the timeout. that'd give a different timeout behaviour which may be more appropriate for what you're doing.17:24
bigjoolsright17:25
jml`bigjools, "improved" as in, "send SIGINT, if it's not dead after a little while, send SIGKILL"17:25
bigjoolsdon't think it is, the script works in ~6 seconds17:25
bigjoolsah ok, don't really need that sophistication here I think17:25
jml`bigjools, lucky you :)17:25
bigjoolsit's an ssh - it can die die die :)17:26
jml`bigjools, for the puller, we reset the timeout on activity -- doesn't matter how long a branch takes overall, just as long as we can reasonably expect it to keep making progress.17:26
bigjoolsyer17:26
* bigjools hacks17:27
jelmermaxb: still there?17:30
bigjoolsjml`: in timeoutConnection you're using error and it's not defined17:34
* bigjools loves the vim pyflakes plugin17:34
=== deryck[lunch] is now known as deryck
=== beuno is now known as beuno-lunch
mrevellGuten nacht18:01
maxbjelmer: back now18:05
=== gary_poster is now known as gary-lunch
=== beuno-lunch is now known as beuno
=== dpm is now known as dpm-afk
jml`g'night all20:06
=== gary-lunch is now known as gary_poster

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!