=== cinerama_ is now known as cinerama [00:27] jcsackett: Around? [00:28] lifeless: Hi. [00:28] hola [00:29] lifeless: Given the utterly disastrous state of QA over the past few days, I am tempted to revert the bad picker fixes. [00:31] did jons branch fail ? [00:31] I agree we despearately need to get stable [00:31] Jon's branch hasn't even been reviewed [00:32] Jon's branch is not reviewed, not ec2'd, and not necessarily going to work, and we've already got three rollbacks in the queue. [00:32] And 3 more revs to QA [00:32] A fourth to get us back to a reasonable state cannot hurt. [00:32] * StevenK finds his rope. [00:33] And it's not even easy QA. [00:37] I like rollbacks :) [00:40] Ooh, it still reverts cleanly. [00:40] * wgrant reverts cleanly. [00:41] wgrant: Are you going to self-review it? [00:41] Yes. [00:45] It is done. [00:51] wgrant: hi; rabbit [00:51] wgrant: did you have some success ? [00:51] Not really. [00:51] with it in vms specifically [00:51] k [00:52] I suspect I just need to poke /etc/hosts harder. [00:53] I'm really disappointed with rabbit ease-of-use in this area [00:54] Indeed. It's entirely unnecessary, too. [00:54] Yay, create-lxc-aufs still works. [00:57] OK, got it working. [00:57] 127.0.0.1 localhost lucid-lp-temp-aufs-7YHq [00:57] Doesn't work [00:57] 127.0.0.1 localhost [00:57] 127.0.1.1 lucid-lp-temp-aufs-7YHq [00:57] Does [00:58] butwhy [00:58] Because it is hideous and likes to resolve its name both ways :) [00:58] wgrant, hey, cody-somerville said you rang [00:58] It's like kerberos, except worse. [00:58] wgrant: what do you mean? [00:59] Hmm, I wonder if it's our fixture, actually. [01:00] well, u1 had the same ugly [01:00] and we got our fixture's core from u1 [01:00] but I'd like to understand what you mean [01:00] does it manually parse /etc/hosts ? [01:00] or is it binding to 127.0.1.1 for some bizarre reason? [01:00] NAFAIK [01:00] Need to experiment. [01:01] my lxc doesn't have 127.0.1.1 in 'ip addr' [01:01] It's not going to. [01:01] It's 127.0.0.0/8 [01:01] inet 127.0.0.1/8 scope host lo [01:01] Well, that. [01:01] the /8 only affects routing [01:01] Yes... [01:01] And it will resolve your hostname. [01:02] 127.0.1.1 isn't usable unless the address is added somewhere [01:02] Yes it is. [01:02] or something daft is going on [01:02] ping 127.99.99.99 [01:03] 1: lo: mtu 16436 qdisc noqueue state UNKNOWN [01:03] link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 [01:03] inet 127.0.0.1/8 scope host lo [01:03] inet6 ::1/128 scope host [01:03] valid_lft forever preferred_lft forever [01:03] 127.0.0.1/8 is on the interface. [01:03] Hmm. [01:04] Anyway. [01:04] lo takes the whole /8 [01:04] Always has. [01:04] And rabbit or the fixture somehow need to be able to resolve it both ways. [01:07] hah [01:08] searching for lo address gets a cee lo news clip ;P [01:08] Heh [01:09] I'm not sure lo always bound to all the the 127 addresses, but meh [01:10] wgrant: whats odd/confusing is why it *wants* 127.0.1.1 [01:10] 127.0.1.1 has been in the default installation forever. [01:10] wgrant: e.g. why its not happy with 127.0.0.1 [01:10] wgrant: no, it hasn't [01:10] lifeless: 127.0.0.1 reverse resolves to localhost first. [01:10] wgrant: feisty/etch brought it in. [01:11] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=316099 [01:11] ah, so its whinging because it gets a different name back ? [01:12] I presume so. [01:12] It's possibly the fixture being crap. [01:12] *However*, Feisty only added 127.0.1.1 on new installs [01:16] Not even 12000 OOPSes from yesterday's amusement. [01:16] How boring. [01:17] Project devel build #865: STILL FAILING in 5 hr 50 min: https://lpci.wedontsleep.org/job/devel/865/ [01:24] totally irrelevant, but apparently the 127.0.0.1/8 thing is different on os x === poolie_ is now known as poolie [02:12] lifeless: Hi. [02:14] lifeless: The problem goes away if we pass fq_nodename instead of just nodename in as RABBITMQ_NODENAME. [02:15] lifeless: rabbitmq will use nodename@hostname if it's specified, otherwise it will do a reverse lookup to guess hostname. [02:15] So it's a very small patch? [02:15] That's how the system one works: /usr/lib/rabbitmq/bin/rabbitmq-env constructs rabbit@hostname [02:15] Yes. [02:15] Three characters. [02:19] I think rabbitfixture also leaks tmpdirs. [02:19] Need to check that out. [02:19] Heh [02:20] Like, my laptop had several thousand by the end of last week. [02:21] hi StevenK, wgrant, lifeless [02:21] O hai! [02:21] Hi poolie. [02:29] Mm, doesn't seem to leak normally. [02:29] I guess I just killed it too aggressively. [02:29] A thousand times or more [02:39] poolie: hi [02:39] wgrant: cool [03:08] Orsum. bzrlib.lockable_files._LockWarner is now affecting ~1,500 tests on Jenkins [03:12] Upgrade your bzrlib :P [03:14] spiv: That's with bzr 2.3.3-0~bazaar1~lucid2 [03:15] StevenK: The relevant change landed on trunk about a week ago [03:15] I think I'm blind [03:16] spiv: we run releases in lp though [03:16] I don't think I want to run LP tests against bzr trunk :-) [03:16] spiv: so I think 'release 2.4 please' [03:16] spiv: is a prerequisite [03:16] We often run betas. [03:16] the more bzrlib changes things the less I think that that is a good idea [03:17] what I mean is that the old one-line-of-releases, each month is as stable as possible, deprecations are always graceful : well that was more amenable to using recent developments. [03:17] its now harder to upgrade bzrlibs [03:18] and we can't be as confident about e.g. wire level support issues till the final release in the series [03:19] I'm not saying this is a problem per se [03:19] but its a consequence of the changed policies [03:19] zope.testing tests make my eyes bleed [03:20] What are you doing to it? [03:20] see my last comment in https://code.launchpad.net/~lifeless/launchpad/storylayers/+merge/66738 [03:20] I am teaching it that if it gets a test with no layer, it should not do layer per-test teardown and setup, regardless of whatever layers are setup. [03:21] its that [03:21] Hah [03:21] or I can insert an adapting TestResult [03:21] between the DocFileTests in a StoryPageTestCase and the zope.testing runner [03:22] but the adapter would need to merge all the failures in all the elements of the story [03:22] I think this is probably a lesser evil [03:22] anyhow, we have 6 failures in our fork already [03:22] and no idea if they are bitrot, changes, or what have you [03:22] so I'm going to close my eyes and JFDI [03:31] Question: What exactly is the Launchpad Community dev team? https://edge.launchpad.net/~launchpad-dev [03:31] its a mailing list [03:31] Just a list? [03:31] you are welcome to join if you want to discuss the development of launchpad [03:32] I'm alreaady a member on it. [03:33] lifeless, it would be nice if there was some clearer "what is this team for" concept [03:34] some of them you're welcome to join if you're just interested; some are only for trusted peolpe [03:34] perhaps they shouldn't all be presented as teams === almaisan-away is now known as al-maisan [03:34] if it just a list they should just have a list section [03:35] there is a section about it being a list [03:35] biab [03:37] What does biab mean? [03:38] 'Back In A Bit' [03:51] spiv: Out of interest, can you link me the fix for that? [03:52] StevenK: http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/revision/6002 [03:53] spiv: That sounds like a good plan. [03:54] Yep. [03:54] Probably makes the GC a bit happier. [03:55] Hmm, probably not so much. I doubt our lock objects were participating in reference cycles. [03:57] I suspect we have bugs causing us to see the warnings [03:58] Probably. [03:59] ok, so this route isn't going to work [03:59] I'll do a big-arse adapter instead. [04:00] To adapt it to a smaller arse? [04:01] if only [04:01] this is z.testing [04:23] * StevenK attempts to QA stuff [04:27] wgrant: Can haz blessing to update DF? [04:28] StevenK: doit [04:29] wgrant: I think bug 740208 shouldn't be to bad to QA [04:29] It's not very clear from the code if it's an e-mail that's in a title or something [04:30] I checked it for descriptions earlier. [04:30] It seems to be OK. [04:30] But eeeh. [04:30] Heh [04:38] biab [04:54] wgrant: So it looks fine, you're just nervous about it? [04:56] StevenK: Yes. [04:56] Also considering tracking down whoever decided that erlang extensions should automatically download their dependencies from git/hg. [04:57] blink [04:57] My eyes [04:57] Haha [04:58] wgrant: Do you think it's fine to deploy if we keep on an eye on it, or not? [04:58] StevenK: I think it's OK. [04:58] git clone http://github.com/mochi/mochiweb.git [04:58] Check out the 'rebar' file. [04:58] rebar seems to be some kind of extension build system. [04:58] But look at what the file is. [05:00] I'd rather not touch a git repo :-P [05:00] wgrant: And we can probably deploy jtv's change, since it's FF'd [05:00] rvba's change is running glaciall^Won mawson [05:01] wgrant: it looks lovely in github's browser, that file [05:01] mwhudson: URL? [05:01] Project db-devel build #700: STILL FAILING in 5 hr 40 min: https://lpci.wedontsleep.org/job/db-devel/700/ [05:01] StevenK: https://github.com/mochi/mochiweb/blob/master/rebar [05:01] 'blob' has rarely been more appropriate [05:02] Right. Now to review my lunch. [05:02] that blob looks like a .zip file [05:02] It is. [05:02] it's something erlang related isn't it? [05:02] What else... [05:04] wgrant: So, the IDSJ runner just finished on mawson. It looks like it got past the problematic bit, so I'll qa-ok it, but there's another bug there. [05:04] * StevenK files it [05:06] StevenK: k [05:10] Firefox, how you annoy me [05:10] Who broke click handling between Firefox 4 and 5, seriously? [05:11] We are probably deployable in 2 hours [05:14] zomg [05:14] Almost there. === al-maisan is now known as almaisan-away [05:32] Bah, bzr-hg fail. [05:32] Hm? [05:32] qa? [05:33] Ha [05:33] When r13376 hits qas, we can deploy 42 revisions [05:35] bzr-hg fails to work with rabbitmq-mochiweb. [05:35] It works with the rest :( [05:35] file a bug, please? [05:35] or, find one? [05:36] which branch? [05:37] http://hg.rabbitmq.com/rabbitmq-mochiweb/ [05:37] Can't branch it directly. [05:37] But if I try to branch a clone, it does a bit and then: [05:37] bzr: ERROR: An inconsistent delta was supplied involving '', 'hg:test' [05:37] reason: Parent is not present in resulting inventory. [05:43] Right, buildbot done, waiting for qas [05:44] wgrant: I have prepared a bug list, too [05:44] Excellent. [05:44] 42 revisions, 28 bugs [05:46] Less excellent. [05:49] wgrant, this rings a bell [05:50] wgrant, https://bugs.launchpad.net/bzr-hg/+bug/513368 [05:50] <_mup_> Bug #513368: inconsistent delta during fetch < https://launchpad.net/bugs/513368 > [05:50] poolie: Thanks. [05:57] * StevenK whimpers at qastaging [05:58] How did you miss 13376?! HOW? [06:01] buildbot-poller being slow, maybe? === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 204 - 0:[######=_]:256 [06:03] wgrant: But surely it pulls from stable? [06:04] But, perhaps [06:04] it does [06:05] but perhaps you've forgotten the 30 minute build of the tree on carob [06:07] Why does stable need to build on carob? [06:07] Since the first thing qas is run buildouyt [06:07] *buildout [06:09] StevenK: buildbot-poller pushes stable. [06:09] StevenK: So buildbot completing is not enough to update stable. [06:10] You have to hope that buildbot-poller sees it finish in a timely manner. [06:10] Right, so moving to Jenkins might actually solve this -- since my plan is that when devel builds successfully it will push to stable/db-devel as the last step [06:11] Jenkins shouldn't have permission to do that, I don't think. [06:11] But maybe. [06:11] Well, Jenkins tell tarmac to do it [06:11] Handwave [06:12] Utterly kill buildbot-poller [06:16] 3/4 deps packaged... [06:17] All of which originally grabbed $VCS checkouts of all of their dependencies [06:17] WTF [07:09] Project devel build #866: STILL FAILING in 5 hr 52 min: https://lpci.wedontsleep.org/job/devel/866/ [07:13] Ahah [07:13] Packaged rabbitmq-management stack working. [07:25] nice [07:25] good night all [07:26] Night. [07:40] morning [07:40] Morning bigjools. [07:44] wgrant: did you see the QueueInconsistentStateError oops? [07:45] bigjools: Yes, I identified the bug and Steve got the PU rejected. [07:45] wgrant: good. is it a new bug? [07:46] looked like it was PPA though [07:46] bigjools: Yes. OEM copied a package into a new PPA before it was private (the setup script hadn't called lp_save()), which triggered a delayed copy. It was then rerun with lp_save() added, so it was made private and recopied directly. [07:47] The delayed copy processed a couple of minutes later, but the direct copy had already been done. [07:47] awesome :/ [07:47] Yay [07:47] the story is .... [07:47] delayed copies need to die [07:48] Yes [07:49] In other news, I've packaged rabbitmq-management. It's a plugin that provides an HTTP API which will allow the test suite to quickly reset rabbit. [07:49] So we can have test isolation. [07:49] Which will be nice. [07:49] splendid [08:04] Morning all. [08:05] o/ [08:08] Hi [08:09] lifeless: Hi, I modified the code you rejected earlier today to throw a new exception with an additional message instead of using "print" (https://code.launchpad.net/~rvb/launchpad/rabbit-error-plugins/+merge/67000) ... if you think it's still clunky, I'll just forget about it but allenap and I though it would be nice to get a warning about potentially conflicting rabbit plugins. [08:10] good morning [08:10] rvba: I think there is a better way to tackle this; I will describe it in the proposal [08:10] rvba: We're going to depend on rabbitmq-management soon, so I'll be working out how to run multiple instances tomorrow, hopefully. [08:10] But there may be others. [08:11] lifeless: ok thanks. [08:11] rvba: O hai. I QA'd your change, but hit https://bugs.launchpad.net/launchpad/+bug/806307 [08:11] <_mup_> Bug #806307: IDS has no access for DistributionSourcePackage < https://launchpad.net/bugs/806307 > [08:11] wgrant: I suppose the 'conflict' is caused by multiple rabbit instances trying to use the very same port. [08:12] rvba: Right, I need to work out how to configure that. [08:12] Not sure if we can do it through an envvar like with the AMQP listener. [08:12] rvba: Are you running natty? [08:13] wgrant: yes. [08:13] Not sure if I want to backport natty's rabbitmq 2.3.1 to everywhere, or take 2.5.0 from oneiric. [08:13] 2.5.0 has some major plugin build system changes. [08:13] 2.3.1 has everything we need. [08:14] rvba: done [08:14] wgrant: think CATS [08:14] s/S// [08:15] I forget how to check what's there. [08:15] Is there a madison around somewhere? [08:15] presumably apt-cache search on carob [08:15] Or must I FTP internally? [08:15] carob only sees stock lucid rabbitmq-server. [08:15] lifeless: thanks ... I have used getDetails too ;) ... I guess I'll wait to see what wgrant does about this problem, if he manages to fix this in a generic fashion, the additional warning will be useless. [08:15] Perhaps it's in some other pocket. [08:16] * StevenK kills parallel-test and disables the job again [08:16] Running for 26 hours makes me SAD [08:16] StevenK: sob, thanks. [08:16] Project parallel-test build #85: ABORTED in 1 day 2 hr: https://lpci.wedontsleep.org/job/parallel-test/85/ [08:16] StevenK: any evidence of how it failed/ [08:16] StevenK: it's a good sign that the initialisation started. Thanks for QAing my change. [08:17] Project parallel-test build #86: ABORTED in 29 sec: https://lpci.wedontsleep.org/job/parallel-test/86/ [08:17] * Launchpad Patch Queue Manager: [r=wgrant][rollback=13334] Revert r13334 due to bug #806179. [08:17] * Launchpad Patch Queue Manager: [r=gmb][no-qa] New lp.app.longpoll package. [08:17] * Launchpad Patch Queue Manager: [r=gmb][bug=805547] Fix packageset's copy in initialise_distroseries. [08:17] <_mup_> Bug #806179: PersonPickers broken from bad passed in config < https://launchpad.net/bugs/806179 > [08:17] * Launchpad Patch Queue Manager: [r=deryck][bug=740208] obfuscated email addresses for anonymous [08:17] webservice requests [08:17] * Launchpad Patch Queue Manager: [r=gmb][bug=802840] Ignore None component/section on sync upload [08:17] source override. [08:17] * Launchpad Patch Queue Manager: [r=stevenk][rollback=13356] Rollback revision 13356. [08:17] Bah, sorry. [08:17] * StevenK smacks LPCIBot. [08:17] lifeless: Possibly interesting is that one of my parallel LXC workers hangs most times. [08:18] lifeless: Only one. [08:18] wgrant: fun [08:18] lifeless: And the whole suite passes if not run with --parallel [08:18] wgrant: you're doing aufs isolation right ? [08:18] lifeless: This is with aufs, yes. [08:18] wgrant: that jenkins job isn't, so its much more fragile [08:19] Sure. [08:19] rvba: You installed rabbitmq-management by grabbing the .ez files from the website and installing them manually? [08:20] wgrant: Yes. [08:21] Installing=moving the files to /usr/lib/rabbitmq/lib/rabbitmq_server-version/plugins [08:24] Yeah. === almaisan-away is now known as al-maisan [08:35] rvba: How is your JS branch going? [08:36] wgrant: the generic part is up for review. https://code.launchpad.net/~rvb/launchpad/lp-app-longpoll-js/+merge/66936 [08:36] Ah, excellent. [08:36] The remaining parts (that I shall address after the DD work is finished) are: [08:37] - fix up a service for the async frontend [08:37] - use the generic stuff in the MP page (this will be just cleaning up the prototype that we've done in Dublin) [08:44] lifeless: Only 2.2.0 is in lucid-cat-landscape. That's just new enough that I can probably get this plugin stack backported to it. But natty's 2.3.1 backports to lucid fine without even a rebuild, and I'm not sure if 2.2.0 will work for us. [08:44] I guess I should test. [08:51] Anyone mind if I kick dogfood? bigjools, StevenK, wgrant? [08:51] Not I [08:51] jtv: are you updating it? [08:51] Yes [08:52] cool, was gonna do that anyway [08:52] I updated it a few hours ago [08:52] So you may not need to [08:52] jtv: remember which user you're supposed to use ;) [08:52] I made doing it right easier than doing it wrong, remember? [08:52] jtv: hi [08:52] hi lifeless — saw your note, thanks for that. [08:52] jtv: I'd love to know what I missed [08:53] jtv: (I'm assuming you tried that and it didn't work...) [08:53] Oh, no. My query was just a quick scribble on a late-night train. :) [08:54] I have an observation about postgresql - if there is an ORDER BY matching an index, pg will almost exclusively use it [08:54] Which makes a lot of sense. [08:54] so if you order by id with a date constraint, it will use the primary key index [08:55] Yes, I was simply too focused on getting to the id side of things when I wrote that version. [08:55] jtv: I think the heuristic is too strong, but I haven't had the time to dig into what/why is going on [08:55] Well... it can save you some very painful plan nodes such as hash joins. [08:55] jtv: anyhow, it means that I now have a spinal reflex: to force an index, order by it === jtv is now known as jtv-eat [10:43] Project db-devel build #701: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/701/ === jtv-eat is now known as jtv === al-maisan is now known as almaisan-away [11:23] lifeless: I think I just remembered why I wrote that query the way I did… it performs reasonably for reasonable inputs even without the new index. The simpler version is abysmal without the index no matter what the input is. I think I'll land it my way in devel and then do it your way in db-devel after adding the index. [11:25] No, that wasn't it. :( [11:27] It's been said so many times: if only the statistics would recognize monotonic columns. :( [11:28] jtv: fancy doing a review? [11:29] bigjools: if it's not too big [11:29] ~280 lines [11:29] but a lot of refactoring [11:30] https://code.launchpad.net/~julian-edwards/launchpad/set-previous-series-bug-805913/+merge/66942 [11:30] * jtv has at it [11:30] jtv: the previous_series stuff that I mentioned earlier [11:30] ta [11:30] ah yes [11:30] jtv: we can add the index live [11:30] oh that'd be great [11:31] do we have a Procedure for that yet? [11:31] jtv: land it on devel with a non -0 patch number [11:31] lifeless: slony copes with index changes now? [11:31] normal review (stub + me for db, launchpad-reviewers for the code changes), and stub or a losa can create the index live at any point [11:31] So the -0 patches go out as db-devel rollouts? [11:32] yeah [11:32] danilos: I see https://code.launchpad.net/~danilo/launchpad/expander-anim isn't proposed for merging yet... what's the delay? https://code.launchpad.net/~spiv/launchpad/bmp-inline-diffs/+merge/66634 depends on it. [11:32] so the plan is to make -0 verboten [11:32] danilos: I'd like to get my shiny diffs landed :) [11:32] jtv: https://dev.launchpad.net/Database/LivePatching [11:32] spiv, I'll be getting it landed today, some tests missing still [11:32] lifeless: ta [11:32] bigjools: slony doesn't see the index [11:32] danilos: woo! [11:32] bigjools: what we do is CREATE INDEX CONCURRENTLY on all the replicates [11:32] s/tes/s/ [11:33] Cool, I'll nag people to review mine tomorrow then. [11:33] ah ok [11:33] master first if its an index that can fail (e.g. UNIQUE), then the readonly replicas [11:33] bigjools: Slony doesn't care about indexes. it does care about constraints, which are sometimes tied up with the indexes. [11:33] * lifeless hands over to stub [11:33] I get up in 6 hours :( [11:34] furry muff [11:34] jtv: anyhow, moral of the story is the index can be made live right now if needed. [11:34] great news [11:34] halt() [11:34] lifeless: night, see you at 6am :) [11:35]