[00:25] thumper: :) [00:52] can anyone remember how to get local mail delivery working? [00:52] for the dev environment [00:53] apt-get install postfix isn't it? [00:53] i don't think there's anything special to do [00:54] I just use postfix with its stock local config. [00:54] Then redirect root's mail to my user. [00:55] And point Evo at my mbox. [00:57] wgrant: do you use a .forward file? [00:57] * thumper can't recall [01:00] best not to install new software while a distro upgrade is in progress [01:00] you know [01:00] parallel dpkg would be fun [01:01] hmm postfix is installed [01:01] as would fine grained locks [01:01] thumper: I just use /etc/aliases === jelmer_ is now known as jelmer [01:03] hmm, my local postfix install didn't generate a main.cf [01:03] dpkg-reconfigure postfix [01:04] postconf -n >> main.cf ;-) [01:09] * thumper thinks of something that will generate an email to test [01:10] mail -s foo [01:10] also [01:10] exim! [01:10] telnet localhost 25 [01:11] spm: ooh hardcore [01:11] heh, more like being on hardened firewalls... :-) [01:19] spm: do you know about postfix setup? [01:20] thumper: that'd be a yes. getting rusty; but yes. [01:21] in 2001, spm updated ~ 25+ firewalls and rather horrible mail funnel from sendmail to postfix. only 1 (minor) mess in the entire process. whee! [01:21] spm: I'm trying to get my local postfix to route email through my servers config using the submission port [01:22] my email client sends email through penhey.net:587 [01:22] hmm... [01:22] perhaps I'll try something else [01:24] thumper: Why do you want it sent externally, to pollute your real mailbox? [01:24] I was just thinking about that [01:24] perhaps just relay into local user mail [01:24] It's what I do. [01:25] thumper: exim can do it trivially [01:25] :P [01:25] * lifeless flogs the horse some more [01:25] wgrant: what do you read your local email with? [01:26] thumper: Evolution. [01:26] I use it for my normal mail too, though. [01:26] * mwhudson reads launchpad mail with mail(1) [01:30] spm: how to I remove a deferred email from the postfix queue? [01:31] thumper: sorry - mid debugging a U1 problem. one sec. [01:32] thumper: postsuper -d [01:33] where the hell is the default mailbox? [01:33] /var/mail ? [01:33] * thumper looks [01:33] thumper: /var/spool/mail/root [01:33] Also /var/mail. [01:36] gah, kontact still won't see it [01:36] I can see the mail in the file [01:38] * thumper shakes his head [01:39] haha [01:39] spam caught it [01:40] rofl [01:40] PEBKA.... spam filter [01:44] grar, another stable->db-devel conflict already [01:49] hmm... I wonder if I should boot into gnome rather than kde when lucid starts here... === ]reed[ is now known as [reed] [02:30] hmm [02:30] lucid decided to uninstall postgre 8.3 and python 2.5 [02:31] Yeah, they're gone. [02:32] reenable the launchpad ppa and install launchpad-developer-dependencies [02:32] Reactivate the ~launchpad PPA, and install lp-dev-deps again. [02:32] Damn. [02:32] :) [02:34] Is ReviewerSchedule out of date? [02:34] I don't think jml is in APAC any more. [02:34] well... [02:34] the default font for my irc client is much smaller :) [02:36] And al-maisan is no longer around. [02:36] And there have been no European reviewers since Monday night, although that might be special for this week. [02:50] thumper: interesting discovery [02:50] thumper: the kernel import has imported about 140 megs of packs so far, which sounds about right [02:50] thumper: the git.db file though is a hair under 6 GIGABYTES [02:51] which suggests something is going a bit wrong [02:51] Is that why it is so damn slow? [02:51] who knows [02:51] it doesn't sound very helpful though [02:51] 'taking 26 hours' <- niiiiice [02:52] Hm. Isn't the topic out of date, and it's in fact week 3 of 10.03? [02:52] Even pear looks like it's going to take about 14 hours. [02:52] StevenK: I think so. === mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.03 | PQM is in open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | [02:52] mwhudson: :-) [02:52] StevenK: don't know what you're talking about!! [02:54] mwhudson: OMG [02:54] thumper: indeed [02:55] mwhudson: where is jelmer_ when you need to smack him around a bit [02:55] well gnome wasn't pleasent [02:55] thumper: he knows, at least [02:55] it killed x [02:55] gwibber wouldn't start [02:55] and made gdm take over instead of kdm [02:56] * thumper has punted the little blighter back into his corner [02:56] back into kde now [03:01] Hm. What do I do if I see a typo in a branch which I didn't notice until it landed? :-( [03:05] The typo is in a debug print, but I suspect it may throw an exception when it's run. [03:06] StevenK: submit a branch to fix it [03:10] * thumper is reinstalling launchpad-developer-dependancies [03:10] somehow it got lost [03:11] thumper: Perhaps it got lost when the upgrade was removing obselete packages? [03:11] StevenK: probably [03:11] * thumper shurgs [03:11] Hm. How does one shurg? [03:11] it is like shrugging, but different :) [03:12] :-) [03:14] * thumper afk [03:15] mwhudson: Would you mind reviewing the change, it's a one-line patch [03:15] StevenK: sure [03:15] mwhudson: https://code.edge.launchpad.net/~stevenk/launchpad/fix-typo-for-bug-503258/+merge/22098 ; but you'll need to wait for the diff [03:21] Which has turned up [03:21] StevenK: is this covered by tests? [03:21] mwhudson: I could have sworn it was -- which is why I'm a little concerned how it got landed [03:21] hmm [03:23] mwhudson: The original branch (fixes-bug-503258) changes the test to make sure that custom files are not private [03:24] StevenK: ah [03:24] StevenK: want to add a test then? [03:25] Is debug printing on when running under a test? [03:25] StevenK: it doesn't matter does it? [03:25] debug(1, foo.bar) will blow up if foo doesn't have a bar attribute [03:25] no matter what debug does [03:26] mwhudson: It's a string subst, though [03:27] StevenK: even more so then [03:34] Dear bin/test, plz output debug output [03:38] mwhudson: Ah ha! It just prints the wrong filename, but doesn't error [03:38] (Un)lucky. [03:39] StevenK: heh heh [03:39] AttributeError: 'PackageUploadCustom' object has no attribute 'filename' [03:39] * StevenK grumbles [03:39] So even my fixed branch is broken [03:40] Which does fail the tests, excellent [03:44] * wgrant applies an axe to buildd-slavescanner.txt. [03:46] mwhudson: Do you need to re-rubber-stamp the branch now that I've fixed it so that tests actually pass, or shall I land it? [03:47] StevenK: i'll have another look quickly [03:47] mwhudson: It's pushed, so you should be able to see it [03:49] StevenK: +1 [03:49] land it [03:49] Aye, doing so === flacoste is now known as flacoste_afk [05:46] wheeee vm installs fly [06:53] stub: ping [06:53] thumper: pong [06:53] stub: my lucid upgrade installed postgres8.4 and removed 8.3 [06:53] stub: I have 8.3 back from lp dev deps [06:53] stub: but I've lost all my setup [06:54] stub: should I remove 8.4? [06:54] It did? Mine didn't, but I had both 8.3 and 8.4 installed. [06:54] stub: also, I can't find the database setup code [06:54] You can leave it. You can even use it - the tree passes and I would have loved to switch this cycle but not enough time. I just haven't announced it. [06:55] ah, ok [06:55] stub: where are the setup instructions now? [06:55] I looked in rocketfuel-setup but didn't see them [06:55] Dunno :-) Looking in the wiki now [06:56] https://dev.launchpad.net/Running says there is a script in utilities, although I've always done things manually (per the linked DatabaseSetup page) [06:56] ./utilities/launchpad-database-setup [07:02] stub: will that work with 8.3 and 8.4 installed? [07:03] IIRC it will prefer 8.4 if it finds it. [07:03] I have no idea. I've never run it myself. Should be fine though - unless someone tried hard to break things it will use whichever PG is running on port 5432. [07:04] I believe it finds the latest version and hopes that it's running on 5432. [07:05] hmm, I have 8.3 on 5432 and 8.4 on 5433 [07:06] The script looks like it should die in that case. [07:09] well that was a big fail [07:10] What died? [07:11] the setup script [07:12] failed in applying hunks [07:12] failed to create the user [07:12] failed on other bits [07:12] then said... [07:12] Looks like everything went ok. [07:12] bullshit [07:15] Hahhaa. [07:17] stub: it is all fucked! [07:20] thumper: https://dev.launchpad.net/DatabaseSetup still seems relevant and should work for 8.3. It should also work for 8.4 if you don't add 'max_fsm_relations = 2000' to postgresql.conf [07:22] stub: well, 8.4 won't start at all says it has invalid directories et al [07:23] stub: so I decided to go with 8.3 for now [07:23] however [07:23] http://pastebin.ubuntu.com/401000/ [07:24] why can't it get the working directory? [07:24] Because you're in the dropped cluster. [07:24] Because that script is destroying the directory you are running it from? [07:25] oh arse [07:26] I'll update that script for 8.4 soon anyway. I think it is how the ec2 instances get setup. [07:29] http://pastebin.ubuntu.com/401003/ [07:29] stub: ^^ [07:30] postgres8.3 says it is running [07:30] How does it say that? [07:31] :( [07:31] It might be running, but not on port 5432 [07:31] * Restarting PostgreSQL 8.3 database server [ OK ] [07:31] however [07:31] $ /etc/init.d/postgresql-8.3 status [07:31] Error: Invalid data directory [07:31] WTF? [07:31] Why not purge both, install 8.4, and run the script? [07:31] will that work? [07:32] It has for me. [07:32] Did you run step 4? [07:32] Except I installed 8.3, because LP didn't support 8.4 yet. [07:32] dropcluster,createcluster? [07:32] jtv updated the script to work with 8.4 IIRC [07:32] stub: yeah [07:33] I'd go with wgrant's suggestion as it will probably work. [07:33] ok [07:33] And even if it doesn't work, it won't get you into any worse a situation than you're in now. [07:36] ahh [07:36] 8.4 can't be removed because it fails to stop [07:36] because it has an invalid data directory [07:37] how can I force it harder? [07:37] kill it and see if it still tries to stop. [07:37] it isn't running [07:37] * thumper wonders [07:37] stub: ec2 test uses the script yeah [07:37] Make the data directory uninvalid, or hack the script to not whinge. [07:39] how do I make the data directory uninvalid [07:39] I don't know where it is looking [07:39] 'valid' [07:39] the word is 'valid' [07:39] lifeless: Well, I didn't want to take it that far. [07:39] It probably doesn't need to be a valid postgres data dir. It just needs to not look completely invalid. [07:40] wgrant: so, appear valid :P [07:40] Pfft. [07:43] stub: any idea how I can get 8.4 unfucked? [07:43] I'm losing patience rapidly [07:43] killall postgres ? [07:44] done [07:44] but the --remove triggers a stop, which fails [07:44] which stops the --remove [07:44] --remove? [07:45] sorry [07:45] sudo aptitude remove --purge postgresql-8.4 [07:45] that is what fails [07:46] Removing postgresql-8.4 ... [07:46] * Stopping PostgreSQL 8.4 database server * Error: Invalid data directory [07:46] Kill the processes, nuke /etc/postgresql, nuke /var/lib/postgresql [07:46] And file a bug if you can be arsed [07:46] stub: do you think the uninstall will work then? [07:47] Should. There are no config files to know what to shutdown. [07:52] * thumper thrashes the HDD [07:54] stub: aptitude tells me that postgresql-8.3 is installed, but there is no /etc/postgresql [07:54] I nuked it before [07:55] Sounds like aptitude didn't do some setup step. Or maybe its the 8.3 packages on lucid - I thought you where going to use 8.4? [07:56] Hmm. If you purged (not just removed) postgresql-8.3, postgresql-8.4 (and maybe postgresql-common), that directory should have been recreated on installation. [07:57] if I try to remove postgresql-8.3 it removes all the launchpad developer deps [07:58] With synaptic, it would remove the launchpad-developer-deps packages sure. But not the rest of the deps. [07:59] aptitude did, apt-get won't [07:59] stub: Regarding that EmailAddress issue, USSO was creating them until recently. Bug #523346. But I thought they fixed the data too. [08:01] stub: so if I install the 8.4 packages for slony, contrib, and plpython I should be good? [08:02] stub: 8.4 didn't add a /etc/postgresql directory either :( [08:03] stub: before when you said nuke, I just moved them to /junk [08:03] stub: could that matter? [08:03] You can run pg_createcluster yourself per the wiki docs, but something is screwed [08:03] I don't see how that could matter [08:04] it is creating the 8.4 cluster now [08:04] the launchpad database setup script needs to be run as root? [08:05] I expect so, unless it uses sudo internally [08:05] it uses sudo internally [08:05] just checked [08:06] I think I'm getting somewhere [08:06] making schema [08:06] on 8.4 [08:07] ok [08:07] work done for me [08:07] I think I'm up to a working launchpad environment on lucid [08:07] stub: thanks for your help [08:07] and wgrant [08:08] Hopefully there's still only that one test failure. I might run the whole suite again tonight. [08:11] wgrant: Which test failure? [08:11] StevenK: test_generateConfig in Soyuz. [08:11] apt-ftparchive outputs extra hashes. [08:11] s/Soyuz/archivepublisher/ [08:12] Ah. I just had test_mark_duplicate_form_overlay fail for me, thought it was related [08:13] There was a commit related to that this morning. [08:17] * wgrant WTFs at builder.txt. [08:53] Ah. [08:53] I was wondering why the start of builder.txt was so shocking. [08:53] annotate shows that it hasn't been changed significantly since around r6000. [08:53] Er. [08:53] r600. [08:54] :D [08:55] * wgrant throws a quick rewrite into another branch that touches that file. [09:09] bigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree. [09:09] The worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support. [09:09] wgrant: sounds fine then [09:11] * bigjools considers firewalling all IPs from China, Russia and South America [09:11] bigjools: Should I poke a LOSA with instructions, or ...? [09:11] bigjools: Good idea. [09:12] bastards DoSing me trying to crack my ssh [09:12] denyhosts FTW. [09:12] yarp [09:13] 235 jobs (30 hours) [09:13] WOW. [09:13] I am glad I haven't tried to use a PPA lately. [09:13] I auto-drop after 3 failed attempts but it still screws my connection [09:13] builders removed, it'll come back [09:14] Um, 30 hours is really quite ridiculous and calls for drastic action. [09:14] it's not ideal [09:15] Waiting more than a day for a build is well beyond not ideal. [09:18] wgrant: umm anyway yes if you prepare some SQL I'm sure we can get a losa to poke it in [09:18] bigjools: OK. Thanks. [09:18] * wgrant works it out. [09:26] did anyone see https://bugs.launchpad.net/bugs/546017 ? [09:26] Bug #546017: can't add linux-meta-ti-omap to kernel/lucid package set [09:29] LOSAs: Can you please run this query on production to enable 3.0 (quilt) and 3.0 (native) in Karmic? [09:29] INSERT INTO sourcepackageformatselection (distroseries, format) SELECT id, 1 FROM distroseries WHERE name='karmic' UNION SELECT id, 2 FROM distroseries WHERE name='karmic'; [09:29] bigjools: Yeah, that's an odd one. [09:30] Although I don't see that source. [09:30] wgrant: can you pastebin that query, I need to get permssion from da management [09:30] wgrant: I'd need approval from someone (BjornT?) to run that on production [09:30] Ah, of course. [09:30] * wgrant pastes. [09:31] http://pastebin.ubuntu.com/401051/ [09:31] ta [09:31] I ran it on dogfood anyway [09:34] bigjools, wgrant: why is that query needed? [09:34] BjornT: I just updated LPS [09:34] BjornT: See scrollback starting from 25 minutes ago [09:36] wgrant: i prefer short summaries :) [09:36] Well, the relevant scrollback is two lines. [09:36] 20:09:24 < wgrant> bigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree. [09:36] 20:09:40 < wgrant> The worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support. [09:37] BjornT: there's a short summary in LPS ... [09:39] wgrant: well, finding two lines in 25 minutes of scrollback isn't that easy, if you don't know which lines to look for [09:40] thanks bigjools [09:40] True. [09:42] approved [09:42] Thanks. [09:49] thanks BjornT [09:49] owdy [10:07] Morning, all. [10:08] buildd-slavescanner.txt creates builder1-11, sticks them in buildergroup.builders, and then does nothing further with buildergroup. [10:09] Before my branch it does touch buildergroup, but only to call rescueBuilderIfLost, which doesn't touch that .builders at all. [10:09] So there's a whole lot of dead tests code there that looks potentially important. [10:10] wgrant: is it possible that the actual tests have been gradually moved out over time, leaving just the test setup there? [10:10] noodles775: I hope so. [10:11] It looks like it might now create other equivalent builders as it goes. [10:12] The functionality does seem to be tested elsewhere, so I guess the creations and assignments can go. [10:13] Great. [10:13] (I'm removing buildergroup, so needed to do something with them) === dpm is now known as dpm-bbl === magcius_ is now known as magcius === magcius is now known as magcius_ === magcius_ is now known as magcius [10:59] stub: Still around? [10:59] yo [11:01] stub: Ohai. Can you cast your weather eye over this query and tell us if it's completely mad?: http://pastebin.ubuntu.com/401091/ [11:01] stub: We want to use it to schedule the next_check of a bug watch based on the number of errors it has had over the past 5 attempts. [11:03] The coalesce(array( stuff seems needlessly confusing over 'count(*)' [11:04] stub: I tried with a count but it didn't work after a couple of minutes and the array stuff did, so I didn't spend any more time on it. [11:05] or sum [11:06] But fine. It actually works? [11:06] stub: Yes. In my head, I imbued that question with a tone of surprise. [11:07] Sticking a subquery in the FROM clause of an UPDATE that references the table being updated - can't recall having tried that [11:07] nm,.. not doing that [11:08] Your missing 'BugWatch.id = bug_watch.id' in the WHERE clause [11:08] sorry... BugWatch.id = counts.id [11:08] Ah yes, thanks. [11:09] You don't *have* to do it in a single query though ;) [11:10] stub: It made allenap happy so I was loathe to stop him. [11:12] stub: Another version of the array stuff with count(*) instead: http://pastebin.ubuntu.com/401099/ [11:19] * stub wonders why next_check is ever null [11:19] Wouldn't it just default to NOW and be updated after every check, successful or not? [11:21] stub: I don't completely understand. Do you mean that we should calculated at the same time the watch is checked? [11:22] stub: next_check is set to NULL by checkwatches and then updated by a garbo process. We wanted to keep the scheduling logic out of checkwatches since it's complex enough already. [11:22] ok [11:22] Not sure if that is a win, but I'll believe you :) [11:23] From our point of view, anything that makes checkwatches do less is a win :) [11:23] So how does that query work in the context of a DBLoopTuner loop? It is updating all of them at once. [11:25] That could be tens of thousands if things don't run for a few days. === jelmer_ is now known as jelmer [11:26] stub: We can run that query against a batch of BugWatch IDs. [11:27] That would fit nicely with the LoopTuner way of doing things. [11:27] ok [11:28] If you do that though, you are still materializing the recent activity list for all bugwatches each time though, even though you are only updating a batch. [11:29] Maybe PG can optimize that, but I doubt it. [11:29] Oh... you put the list of ids in the subquery [11:29] that would work [11:30] stub: I was just about to say that ;) [11:30] wgrant: karmic now accepts quilt [11:30] bigjools: Thanks. [11:30] $LOSA: Also thanks. [11:32] wgrant: I just thought, we've never pushed a source package branch through dogfood [11:32] bigjools: True. [11:32] can you think of anything that would prevent that now (apart from lack of a UI) [11:32] It should all work now. [11:32] As long as the buildd is OK. [11:32] All relevant work is long-merged. [11:32] cool - have you got a script I can steal to poke something in? [11:33] I have somewhere.. [11:33] * wgrant hunts. [11:34] bigjools: I used http://pastebin.ubuntu.com/401111/ [11:34] It probably still works. [11:35] * wgrant checks the DF configs for the right branch prefix. [11:35] ok [11:37] Hm, so plain lp:whatever should work fine. [11:41] ok I'll poke it through now [11:41] I suppose you can just use any old production package branch. [11:42] And hope that dogfood doesn't explode... [11:43] * bigjools starts ripping chunks of hair out [11:43] AttributeError: 'NoneType' object has no attribute 'utf_8_decode' [11:43] ARGH [11:43] That reappears every couple of months. [11:43] Yay. === noodles785 is now known as noodles775 [11:43] I always forget how to fix it by the next occurrence. [11:44] and every f***ing day on dogfood [11:44] make clean [11:44] Oh. [11:45] well it's only doing it on iharness atm [11:45] I guess I'll have to put up with plain harness === matsubara-afk is now known as matsubara [11:48] Is it reasonably up to date, so the UI won't burn up once the build gets dispatched? [11:50] wgrant: http://pastebin.ubuntu.com/401120/ [11:50] seen that? [11:50] bigjools: The branch you referred to probably doesn't exist on DF. [11:50] What was your recipe? [11:50] ok that makes sense, I blindly copied yours :) [11:51] lp:ubuntu/hello should work [11:51] yeah [11:53] bigjools: How old is the DF archive? [11:53] old [11:54] If it lacks a reasonably recent bzr-builder, I guess the target archive will need external dependency on production primary. [12:00] wgrant: ran the script, b-m not dispatching :/ [12:03] bigjools: The builders are all disabled... [12:03] cobblers, why didn't I check that [12:05] You probably also need to twiddle scores, but we'll see. [12:06] which I really should have done before watching gcc get dispatched. [12:06] cock [12:06] Oops. [12:09] You can't abort it? [12:09] not easily [12:10] anyway, there's no sourcepackagerecipebuild [12:10] so something is borked [12:10] transaction.commit()? [12:11] * bigjools weeps [12:11] that really shouldn't be required unless I started a txn >:( [12:14] bigjools, hello, are there plans to add more builders cause there is a huge package queue waiting to be built? [12:14] ricotz: the missing ones will return at some point, I don't know when [12:14] 33 hours and counting... [12:15] bigjools, ok, so this is out your hands then [12:15] bigjools: Is concordia the one that took 2 hours to do a 2 minute build? [12:16] yes, the machines that are missing are not exclusively PPA builders, they get loaned when not in use for other stuff [12:16] wgrant: dunno - but I started rubidium up and got a traceback when b-m tried to dispatch the sprb to it [12:16] ok, thanks [12:17] ricotz: we just have to be patient I'm afraid [12:17] bigjools: What was the TB? [12:17] DB permissions [12:17] 2010-03-25 12:16:42+0000 [-] psycopg2.ProgrammingError: permission denied for re [12:17] lation seriessourcepackagebranch [12:17] Ah. Easy fix. [12:17] yep [12:18] I thought I fixed that. [12:18] unless you don't regularly run security.py on upgrades. [12:18] Hm, maybe not. [12:19] I do run it every time [12:19] Yeah, i forgot about official source package branches when adding perms. [12:19] Oops. === salgado_ is now known as salgado [12:27] so it's building now - I guess we need to fix: https://dogfood.launchpad.net/builders/rubidium [12:27] :) [12:27] I thought that was meant to work now. [12:27] What's the exception? [12:27] No canonical URL for the build? [12:27] ForbiddenAttribute: ('title', )
[12:28] Argh. [12:28] I wish we had these exposed on the API. [12:31] still building... slow... [12:31] how long should that take? [12:31] Heh. It depends, mainly on disk cache. [12:31] On my laptop the time-consuming part is extracting and upgrading the chroot, and installing the build-deps. [12:32] But that's because the disk is slow and RAM is eaten by the rest of LP. [12:32] It normally takes a couple of minutes here. [12:32] k [12:35] What's it say it's doing now? [12:35] Error accessing Librarian: HTTP Error 404: Not Found [12:35] wak wak oops [12:35] Fail. [12:35] No chroot? [12:35] prob the source expiry script [12:36] Why would it be getting sources? [12:36] nfi [12:36] the input to a recipe build is a chroot and recipe text. [12:36] you'd think [12:37] Hm, that LFA is still OK. [12:39] No idea which file it is? [12:40] nup [12:40] trying to get the log out [12:40] The only thing the behavior tries to cache is the chroot. [12:40] And the URL I get through the API still works. [12:41] it got as far as "Processing triggers for python-support" [12:41] no more log available [12:41] wut. [12:41] Is it still going? [12:42] It shouldn't be failing with a librarian error at that point... [12:42] seems so [12:42] the log must be OOD [12:43] OOD? [12:43] out of date [12:43] Yeah. [12:43] oh wait [12:43] my bad - I didn't clear the failnotes from last time [12:43] I was about to ask that. [12:43] builderok is True [12:43] So it's still going? [12:43] Great [12:43] but not doing anything [12:44] Can you SSH into the builder? [12:44] I think I'll poke another one in shortly on a non-virtual ppa so I can ssh into it [12:44] not into virtual ones. no [12:44] Or just flip the virtualized flag on the BQ. [12:44] yar [12:44] need to un-dispatch it though [12:44] Mark the builder not-OK. [12:49] grrrr [12:49] I really really hate the million damn predicates for dispatching [12:50] It's somehow excluded? [12:50] I'm making a new build, much much easier [12:51] I don't think we build SPRBs non-virt even for non-virt PPAs. [12:51] it'll get queued and then I can flip the bit before it's dispatched [12:52] True. [12:52] ffs [12:52] IntegrityError: duplicate key value violates unique constraint "sourcepackagerecipe__owner__distroseries__sourcepackagename__na" [12:53] Just change any character in the SPR name. [12:53] * wgrant loves constraint name length limits. [12:55] 2010-03-25 12:55:36+0000 [-] exceptions.AttributeError: 'SourcePackageRecipeBuild' object has no attribute 'is_virtualized' [12:55] awesome [12:56] Wha? How did it work last time? [12:56] I'm dispatching to a non-virt builder now [12:56] that's the only difference [12:56] Heh. [12:56] My cursor was already over the right line. How odd. [12:56] assert not (not self._builder.virtualized and build.is_virtualized), ( [12:56] that's the badger [12:57] grrr [12:57] Yes. [12:58] Couldn't you have just turned one of the non-virt builders into a virtual one? [13:02] ferraz is building it now [13:03] Excellent. [13:04] buildrecipe is running [13:06] Ah, much better. [13:07] Targetting my PPA? [13:07] yes [13:07] it's still running and not doing anything :/ [13:07] idle box [13:07] Ew. [13:07] What's the latest in the log? [13:07] and I can't get to the log >:( [13:08] rw only for the buildd user [13:08] Maybe it's checking out the branch? [13:08] 0.0%wa [13:09] logtail shows bzr running with the usual locale warning [13:09] and that's it [13:09] OK, so it's probably checking out. [13:09] Or being blocked by a firewall. [13:09] it's running bzr dailydeb [13:09] Can you telnet to bazaar.launchpad.net:80? [13:09] Aha. [13:09] Hmm. [13:10] it's not that big a package. But let's give it a while. [13:10] yes I can telnet there [13:10] :( [13:10] ummm the command line has: [13:10] /usr/bin/python /usr/bin/bzr dailydeb --no-build /home/buildd/work/recipe /home/buildd/work/tree --manifest /home/buildd/work/tree/manifest [13:10] but /home/buildd/work doesn't exist [13:12] I think that's fine. [13:12] buildrecipe constructs the path but never mkdirs it, and it has worked locally. [13:12] k [13:13] I need to wait for lamont then I can see what's going on [13:13] Can you check the buildd-manager log and grab the recipe text? [13:13] * bigjools needs food in the meantime [13:13] and I can't get to the log >:( [13:13] rw only for the buildd user [13:13] on the master. [13:13] oh that [13:13] sorry [13:14] # bzr-builder format 0.2 deb-version 2.4-3+${rev [13:14] no}\nlp:ubuntu/hello\n [13:14] That's the text that was sent to the slave? [13:14] If so, try connecting to xmlrpc.launchpad.net on 443. [13:14] ha [13:14] ha ha ha [13:15] I was assuming it would change it to an http URL. [13:15] But I guess not. [13:15] :/ [13:15] The connection is hanging? [13:15] yip === flacoste_afk is now known as flacoste [13:15] Excellent. [13:15] So it's not actually broken. [13:16] I'm glad we weren't trying to do this on Friday afternoon. [13:17] 2010-03-25 13:16:21+0000 [-] Scanning failed with: 'NoneType' object has no attribute 'utf_8_decode' [13:17] *weep* [13:18] Hahah. [13:19] wgrant, "I'm glad we weren't trying to do this on Friday afternoon" - you are a wise man :) [13:20] wgrant: are you doing a security.cfg fix? [13:21] bigjools: Since we've identified no other issues there, I might. [13:21] And I think that's the only other relevant table. [13:21] and the other UI issues need sorting [13:21] I forget -- can that go to devel? [13:21] no, db-devel [13:22] strictly it can, but the regex will kill the landing IIRC [13:22] Grmph. [13:22] OK. [13:24] * wgrant looks at the UI issue. [13:25] I thought jtv had landed something to fix that actually [13:25] He fixed it for translations builds. [13:26] ah right [13:26] SPRBs are broken for other reasons. [13:26] SELECT on seriessourcepackagebranch for fiera is all that you added? [13:26] we need an interface [13:26] yep [13:26] did I miss something [13:26] ? [13:27] We have one. The view code just doesn't respect it. [13:27] jtv: nope - we were just testing recipe builds on dogfood and the ui blew up [13:27] * jtv looks for quick exit [13:27] something I did? [13:27] jtv: nope :) [13:27] phew [13:28] bigjools: How quickly can you get a sysadmin to apply a crowbar to the firewall? [13:28] * bigjools looks which direction the wind is blowing in [13:36] buildmailman.py REALLY doesn't like being run concurrently. === dpm-bbl is now known as dpm [13:57] noodles775: Do we ave a title scheme for source package recipe builds yet? [13:59] wgrant: I'll send another email summarising, but the conversation the other day lead to (1) the build url will sit under the recipe's url, and (2) we need to list the builds in multiple places (ie. obviously under the recipe, but also for the PPA or Builder contexts). [13:59] Is (1) what you were after? [14:00] Not quite, but I might need that in a minute. [14:00] I'm wanting a string to use in the builder UI. [14:00] To identify the recipe build. [14:01] But I guess the recipe schema in the designs is different, so I'll just make something up for now. [14:01] jml: Can you give me your thoughts on my suggestion for ttps://bugs.edge.launchpad.net/bugs/538024 [14:01] Bug #538024: No way of saying "This project is not packaged" [14:02] "You are here because the Launchpad Edge environment uses the Launchpad Login Service service." [14:03] wgrant: ok... so it looks like there is no title yet on the SPRBuild model. Just add one... something that mentions the recipe name and the target archive and the.... hmm, base-branch short name? Not sure. [14:04] noodles775: I won't mention the target archive at this point, since that's handled by other parts of the UI in somewhat revolting ways. [14:04] wgrant: huh? isn't it a relevant part of what the SPRBuild represents? [14:05] noodles775: Yes, but Builds don't do it, and the builder-related pages include the archive regardless. [14:06] wgrant: I think the only reason builds don't do it is because they are always in their context (PPA or a distroseries), it is on the build detail page of course (https://edge.launchpad.net/ubuntu/+source/bash/4.1-2ubuntu1/+build/1580941) [14:07] s/don't do it/don't include it in the title/ [14:07] noodles775: Those pages don't exist for SPRBs yet. If you look at https://launchpad.net/builders, you'll see that the archive name is included outside the link text (which is the title). [14:07] The same thing will occur for any BuildBase. [14:08] This is all going to have to be revised once the schema is finalised; this is just to make it stop OOPSing. [14:08] True. === matsubara is now known as matsubara-lunch [14:28] bigjools: hello [14:28] bigjools: why did you remove the qa-needstesting tag from bug 541807? [14:29] Bug #541807: Archive should have a status column [14:31] I wonder if I have a weird internet problem that makes me visible only to the rosetta people [14:31] danilos: hi [14:31] Ursinha, I don't see you, sorry [14:31] * wgrant neither. [14:31] danilos: so you don't have internet problems, you're blind === pgquiles__ is now known as pgquiles [14:49] bigjools: Merging lp:~wgrant/launchpad/fix-sprb-ui should unbreak the UI, making logtails far less invisible. [15:04] wgrant: https://dogfood.launchpad.net/builders/ferraz [15:04] bigjools: *that's* a bit more encouraging. [15:05] we'll see! [15:07] bzr is taking its time... [15:09] I wonder what its problem is now. [15:11] ARGH [15:11] DEFAULT_INSTANCE = 'edge' [15:11] lp: uses xmlrpc.edge.launchpad.net by default. [15:11] Not xmlrpc.launchpad.net. [15:12] bigjools: ^^ :( [15:14] wgrant: fuck sake [15:14] whyyyyyyy [15:14] # We use edge as the default because: [15:14] # Beta users get redirected to it [15:14] # All users can use it [15:14] # There is a bug in the launchpad side where redirection causes an OOPS. [15:15] "Beta users get redirected to it" makes no sense, though, since bzr won't possess the cookie. [15:25] indeed [15:25] check this out: https://edge.launchpad.net/~ghostonline/+archive/personal/+packages [15:25] packages stuck at pending :/ [15:26] Yay. What's the error? [15:26] can't find a mention of that ppa in the logs :/ [15:26] ... [15:27] oooo [15:27] its publish flag is "f" [15:27] wtf [15:27] Haha. [15:27] how! [15:28] noodles775: do you know if anyone was disabling publishing on any archvies and might have got the wrong one? [15:28] I didn't think there was any code that actually set that. [15:28] bigjools: in production? Nope. [15:28] It was a couple of weeks ago. [15:29] we've got 115 archives where publish=false [15:29] 110 are ppas [15:30] What are the rest? Debian primary and a copy archive or four? [15:30] copy === matsubara-lunch is now known as matsubara [15:31] barry: ping [15:35] bigjools: hey [15:37] Ursinha: oi [15:37] tudo bem? [15:37] bigjools: tudo bem, e vocĂȘ? [15:38] Ursinha: wet [15:38] bigjools: danilos could join this conversation, he's even able to read brazilian newspapers [15:38] otherwise great :) [15:38] :P [15:38] bigjools: it's raining here as well [15:38] Ursinha, bigjools: cuzao [15:38] Ursinha: let me guess, they're all complaining about your president? [15:39] danilos: rofl [15:39] lol [15:40] man [15:40] hauhau [15:40] so, bigjools, why are you removing qa-needstesting tags from your bugs? [15:40] wgrant: *cough* publish is set in the +edit page.... *cough* [15:40] don't you like them [15:40] Ursinha: are you stalking me? [15:41] they're untestable bugs [15:41] bigjools: you can mark them as qa-untestable then :) [15:41] refactoring and suchlike [15:41] DETAILS! [15:41] :) [15:41] lol [15:41] bigjools: W...why? [15:41] bigjools: I can do that for you [15:42] wgrant: the question is, why not [15:42] * bigjools hugs Ursinha [15:42] bigjools: done [15:42] :) [15:43] Ursinha: how's your sprint? [15:43] bigjools: sorry about the stalking, I just needed to know if you're removing because of a weird use case I haven't thought about [15:43] you guys need a dogfood lesson [15:43] bigjools: it's great [15:43] ah I was only joking about the stalking :) [15:43] bigjools: :) [15:43] is it hot there? [15:48] hi abentley [15:48] bac, hi. [15:49] bac, sure. [15:49] ok [15:50] bigjools: it's always hot in here :) [15:50] * wgrant sleeps. [15:50] night wgrant [15:50] abentley: i've got a branch A based on devel. i've got another branch B that depends on A. [15:50] abentley: both A and B have merged from devel and have all recent changes [15:51] You should not merge B from devel directly. You should merge devel into A, and A into B. [15:51] This is one of the things bzr-pipeline gives you. [15:51] abentley: if i run 'bzr diff -r ancestor:../A' while in B i see diffs that don't exist. it shows a diff in the Makefile even though the Makefile is the same in A, B, and devel [15:53] bac, you have a criss-cross merge, which means that the revision selected by -r ancestor: will be relatively old. [15:53] abentley: ok [15:54] bac, it should go away if you merge A into B. This will warn you that it is a criss-cross merge. I suggest using merge --lca, because it is a criss-cross merge. [15:54] abentley: ok, i just merged again from A and now the phantom Makefile diff is gone [15:54] abentley: thanks [15:54] * bac vows to start using pipelines [15:54] bac, no problem. === salgado is now known as salgado-lunch [15:55] matsubara, ping, do you have IE installed for some simple QA work? I would like to do QA on staging *before* landing my changes. [15:56] bac, of course, you can just manually make sure not to merge the same branch into A and a branch derived from A. [15:57] abentley: right. i just had a temporary lapse. [15:57] If class A has implements(ISomeInterface) then should ISomeInterface.providedBy(A) be True? [15:57] mars, nope. there's the windows EC2 machine though. I can use that. I can help you after the meeting. [15:57] bac, ah, okay. [15:57] .providedBy(instance of A) I mean [15:57] james_w: yes [16:45] anyone have any experience with the python memcached client? === Ursinha is now known as Ursinha-lunch [17:01] sinzui, sure. [17:01] sinzui, thanks for poking me === deryck is now known as deryck[lunch] === salgado-lunch is now known as salgado === beuno is now known as beuno-lunch [17:28] matsubara, instructions are on the wiki for testing with the EC2 instance? [17:30] mars, I think so. let me find them [17:33] mars, https://wiki.canonical.com/Launchpad/LPAWSAccounts for the win AMI and password. you basically start it up using elasticfox or amazon console [17:34] matsubara, great, thanks [17:34] magcius, then $ ec2-authorize default -p 3389 -s your_ip_address/32 [17:34] mars, and then $ rdesktop -u administrator :3398 [17:34] magcius, sorry, above message was for mars, not you [17:34] I wonder if the ec2 toolkit survived the Lucid upgrade? [17:35] now is a great time to find out :-) [17:59] matsubara, should the rdesktop port be 3389 or 3398? [18:00] mars, 3389 [18:00] matsubara, thanks. 3389 works, guess the second command had the typo [18:01] mars, yep, sorry about that === deryck[lunch] is now known as deryck === beuno-lunch is now known as beuno [18:18] *described* === Ursinha-lunch is now known as Ursinha === Ursinha is now known as Ursinha-sprintin === Ursinha-sprintin is now known as Ursinha-busy === dpm is now known as dpm-afk [20:02] morning [20:24] hey thumper how's the ui work going for recipe builds? [20:24] bigjools: you should ask rockstar and abentley, but they are pretty happy [20:26] bigjools, I've spent today wrangling branches to make sure we have all our db patches in this cycle. [20:27] I tried putting a recipe build through dogfood today and found quite a few bugs [20:27] bigjools, eep. Well, we should surface all of those in the next few weeks. [20:28] yeah Michael is going to work more on the soyuz ui side [20:28] it seems as though the builders will need access to xmlrpc.edge.l.n as well [20:46] sinzui: hi, thanks for the offer, my branch is at https://code.edge.launchpad.net/~james-w/launchpad/export-code-import/+merge/22175 [20:46] * sinzui looks [21:02] james_w: what kind of test can we write to verify this kind of mistake does not happen again. I agree you fixed these issues, but none of the other webservice tests changed. I think the problem is that the webservice/xx-distroseries.txt test reports that driver_link is correct, but we know that is not true [21:03] it is true [21:03] I am working on a fix for lazr.restful that will stop this from ever happening again [21:03] but that's not going to be ready in the next 18 hours [21:04] understood. [21:04] so it depends if you want to land the fixes with the tests that will show the issue [21:07] james_w: I understand your point. and agree it is a separate issue. Lets land this now. I glad to know there is a comprehensive fix for this issue. [21:07] thanks [21:08] it's not pretty, but it throws a big error in your face if you make this mistake, meaning the current tests we do for the webservice will be sufficient and we won't start having to write tests that assert that the wadl contains the right thing for everything we export as well [21:09] fab [21:09] maybe someone that knows zope better can come up with a more elegant solution [21:10] that would be gary_poster, flacoste, and leonardr. [21:10] * gary_poster waves [21:11] james_w: do you need me to land this for you? [21:11] and I can't work out how to write the test in lazr.restful as things don't seem to behave the same way there [21:11] sinzui: yes please [21:12] gary_poster: hi, I'm confused by all the adapters and indirection that is going in the two codebases. [21:12] tests in lazr are different since we do not assume launchpad exists. We build a lot of hypothetical objects and explicitly wire them. launchpad hides that a lot [21:12] if you look at the merge proposal above you can see the problem is triggered by using a zope.schema.Choice where a lazr.restful.fields.ReferenceChoice is intended [21:13] so I'm trying to do the same thing in lazr.restful [21:15] to use a Choice I need to provide a vocabulary, the only one currently defined in lazr.restful is an EnumeratedType, but that seems to give different behaviour, as the test uses getMultiAdapter, and picks a different Marshaller to what we get when the bug occurs in LP [21:15] bigjools: Did the test build end up working? [21:15] so I'm trying to understand how the getMultiAdapter call decides to pick an ObjectLookupFieldMarshaller in LP. [21:16] wgrant: nope, I gave up waiting for someone to fix the firewall [21:16] Yay. [21:16] But it seems to mostly work. [21:16] yeah I think it will be fine [21:16] I could change the recipe I suppose so it uses http [21:17] If you just hack getRecipeData to use getPublicURL instead, yeah. [21:17] You can't change the recipe data itself, since the branches are stored as references and URLs are regenerated. [21:17] oh :/ [21:19] bigjools: In lib/lp/code/models/sourcepackagerecipedata.py, s/bzr_identity/composePublicURL()/ [21:20] aha! lib/canonical/launchpad/zcml/webservice.zcml defines an adapter for SQLObjectVocabularyBase === salgado is now known as salgado-afk [21:23] Hmmm. Are we have silent test failures again? [21:23] I have three successful ec2test emails, with around 3 failures each. [21:23] But they get ignored. [21:24] Ah, no, just: lib/canonical/launchpad/pagetests/standalone/xx-form-layout.txt, lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt [21:27] Can somebody please re-land https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model/+merge/22016? It got hit by a spurious failure a few hours ago. === dpm-afk is now known as dpm [21:28] did pqm reject it? [21:29] No, it died from the buildd-slave.txt ProtocolError that sometimes appears. [21:29] oh in ec2 [21:30] Yeah. [21:31] I've just confirmed that it still passes locally. === fjlacoste is now known as flacoste [21:55] bigjools: Also, those two DF fixes from last night landed. [22:01] wgrant: coolio [22:03] bigjools: Since the only unique failure appears to be spurious, can you please land rescueiflost-and-updatestatus-to-model? [22:04] The non-unique failures are more concerning :/ [22:05] wgrant: it's best if you grab someone else, I am about to go to bed and if it fails again my name will be mud :) [22:05] maybe a friendly antipodean can help [22:05] bigjools: OK, thanks. [22:05] mwhudson, while not an antipodean, is very helpful :) [22:06] mm what's this? shoving things into ec2 land? [22:06] yarp [22:06] mwhudson: Well, I have a branch that had two real failures and one spurious failure. [22:06] can you land something for wgrant please [22:06] The two real failures are in trunk, so they don't matter. [22:07] (for that branch, at least -- in wider matters they are rather concerning indeed) [22:07] https://login.ubuntu.com/+openid -> http 500 [22:07] yay!! [22:07] and now it works [22:08] wgrant: are you trying to convince me to just pqm-submit it? [22:08] mwhudson: I might as well try. [22:08] wgrant: i would rather not, tbh [22:08] OK. [22:08] Can you EC2 it, then? [22:09] sure [22:09] Thanks. [22:09] right, glass of single malt and then bedtime, g'night folks [22:09] wgrant: https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model [22:09] Night bigjools. [22:09] bigjools: g'night [22:09] mwhudson: That's the one. [22:10] mwhudson: Have a look at http://paste.ubuntu.com/401411/. It's a successful test from last night, but with failures. [22:10] All the other runs had the same issues, but also ignored them. [22:11] wgrant: that's rather odd [22:11] Yes. It's not a feature I really look for in my test infrastructure, personally. [22:12] does it happen locally? [22:12] Trying. [22:13] i wonder if there's a way to stop thunderbird 3 tying up a cpu indexing all my mail [22:14] ah, yes [22:15] wgrant: the test fails locally, but also fails the test run [22:15] which is nice [22:15] I recall a similar issue a few months ago. [22:16] But with a specific type of test case. [22:16] there was a problem with twisted tests [22:16] That was it, yeah. [22:16] i think that was somewhat different [22:16] :( [22:18] yes [22:19] oh [22:19] hm [22:19] errr [22:19] * mwhudson swivels to point at zope.testing [22:20] wgrant: i am extremely suspicious about the Exception in thread Thread-9:/NameError: global name 'suberr' is not defined traceback [22:22] mwhudson: Ah, yes, true. [22:23] the rage is building [22:25] we last changed our zope.testing version in 2009-08-12 though [22:25] oh right, i think it's because the subprocess is dying in a particular way [22:26] zope.testing-3.8.1-py2.5.egg/zope/testing/testrunner/runner.py contains a hilarious number of potential name errors :( [22:27] Heh. [22:27] So jml's new zope.testing hasn't landed yet? [22:28] no [22:28] though i'm not sure what the blocker is now [22:28] hardy packages of subunit maybe? [22:29] if e.errno == errno.EINTR: [22:29] # If the subprocess dies before we finish reading its [22:29] # output, a SIGCHLD signal can interrupt the reading. [22:29] # The correct thing to to in that case is to retry. [22:29] continue [22:29] Probably buildbot. [22:29] YES BUT ONLY IF YOU'VE IMPORTED THE errno MODULE [22:29] Hahah. [22:30] otherwise this won't work so well [22:33] wgrant: the good news is that the newer zope.testing has these bugs fixed [22:33] Aha. [22:33] Interesting timing. [22:34] it's another thing that will make it harder to land jml's branch [22:36] Indeed. [22:36] rage rage rage [22:39] wgrant: ./bin/test -vvct test_dir_construction_and_trivial_running goes bang rather stunningly [22:41] and rather slowly too: it takes about a minute [22:42] * wgrant is waiting... [22:42] The missing DB issue? [22:43] yes [22:43] of course the last change to the file containing that test was the addition of the AGPL copyright line [22:43] YAY [22:46] trying to run it in production-devel gets the [22:46] exceptions.ImportError: No module named psycopg2 [22:46] thing that plagued buildbot for a while [22:46] Hmm, I regret using allcaps before, since I can't make it an any bigger 'yay' now. [22:47] wgrant: Use the figlet. [22:47] * mwhudson does binary chop on buildbot logs [22:48] actually it's rather easy to find [22:48] Indeed. [22:48] because it starts where the number of tests reported as being run drops by 2000 [22:49] er [22:49] 7000 [22:49] i wonder how many of these 7000 tests we're no longer running fail now! [22:49] Oh, it's that many!? [22:49] Ouch. === Peng___ is now known as Peng_ [22:50] build 716 -> test 26193 tests 26193 passed [22:50] build 718 -> test 19516 tests 19516 passed [22:50] (buildbot vomited on 717) [22:50] .... ouch. [22:51] That wasn't long ago, was it? [22:51] so between 10574 and 10580 [22:51] no, not very long ago [22:51] Phew. [22:51] yeah [22:52] i'm suprised noone's noticed the test suite going faster [22:52] http://pastebin.ubuntu.com/401425/ [22:53] Yeah, I just looked. Nothing there looks suspicious, except for maybe the launchpadlib thing. [22:53] it's hard to see which of these commits could have done it [22:53] so next thought: check the ami versions [22:53] Indeed. Didn't jml push his new image a day or two ago? [22:54] 716: ami-296f8340 [22:54] 718: ami-296f8340 [22:54] not that then [22:54] * wgrant reverts 10575. [22:55] * mwhudson reverts back to 10574 [22:55] 7K missing tests? [22:56] "optimisation" [22:56] Reverting 10575 fixes the missing DB error. [22:56] WTF [22:56] there's some quote in the company quotes page from jamesh about making the test suite faster by deleting all the tests [22:57] this is just the sneakier version [22:57] wgrant: !! [22:57] * wgrant goes hunk by hunk. [22:58] Yeah, it's the least unobvious change. [22:58] The Launchpad.login call. [22:58] wgrant: the test is faster when it passes than when it fails [22:58] Much, yes. [22:59] wgrant: uhhhhhhhhh [22:59] (and no, lifeless, the word is not 'obvious' :P) [22:59] mwhudson: Uh? [22:59] wgrant: just thinking about what might be going on here [23:00] LEONARD!!!! [23:00] It is an odd one. [23:00] Although I guess that probably makes requests for WADL and stuff. [23:00] yeah i guess so [23:00] And is going to be horrifyingly slow. [23:01] if that's being done for each page test... [23:01] I really hope not. [23:01] It takes a while to even import. [23:02] Ah, not too bad with a hot cache. [23:02] i guess it's the recursive call to suite.run() in test_dir_construction_and_trivial_running that's special about that test [23:02] So basically we now have to look at all of launchpadlib to work out what the problem is. [23:02] i dunno [23:02] And lazr.restfulclient and wadllib. [23:03] the page tests that test_dir_construction_and_trivial_running runs are exceedingly trivial: empty files [23:03] it's possible that if they made any requests at all, we'd have this effect already [23:03] Does that not set up a DB, maybe? [23:03] The opstats failures make perfect sense, but this one does not. [23:05] i don't really know [23:05] Oh. [23:06] PageTestLayer doesn't inherit DatabaseLayer at all. [23:07] hm? [23:07] running PageTestLayer tests certainly sets up the database [23:07] Oh, yes it does. [23:07] Confusing inheritance hierarchy is too complex. [23:09] wgrant: hah yes [23:09] wgrant: if i change the page tests that test_dir_construction_and_trivial_running runs to actually make a request [23:09] everything goes bang [23:09] Given how close we are to freeze, do we want to just revert the two problematic hunks (the Launchpad.login, and the single test that uses launchpadlib)? [23:09] Aha. [23:10] So somehow the the DB is getting deconstructed. Yay. [23:11] um [23:11] i'm not sure what the best course is [23:12] having a launchpad object in the test globs that's authenticated as a sample data admin is bonkers anyway [23:12] for so many reasons [23:15] wgrant: http://pastebin.ubuntu.com/401435/ ? [23:16] mwhudson: Yeah, that. [23:16] wgrant: do you know what the failures we've seen in the log actually are? [23:16] the regular looking page test failures [23:17] mwhudson: The opstats one is because of the queries that Launchpad.login makes. I haven't looked at the other one. [23:19] * wgrant runs with Launchpad.login removed. [23:19] Still fails. Probably legit. [23:20] Ah, field description change. [23:21] well that was nice [23:21] my keyboard stopped working! [23:21] oh [23:22] it was probably seahorse asking for my passphrase :( [23:23] mwhudson: The pagetest failure is a team description change yesterday. [23:23] * wgrant fixes. [23:23] wgrant: ah, cool [23:24] i wonder how many other test failures this fix will shake out of the woodwork [23:25] It's only been a couple of days. [23:25] * mwhudson is tempted to pqm-submit it so everyone gets to suffer [23:25] yeah true [23:25] wgrant: want to push/pastebin your fix and i'll merge it? [23:26] thumper: want to review a branch? [23:26] mwhudson: Just rerunning the test now. [23:27] cool [23:28] http://pastebin.ubuntu.com/401444/ [23:29] So that's all known failures fixed. [23:30] yeah [23:32] * mwhudson shoves it into ec2, starts writing email [23:35] Great. [23:42] ah balls, forgot --headless on the ec2 test [23:43] Oops. [23:46] well that wasn't what i said i would be doing in the standup [23:48] Heh. [23:48] Thank goodness for spurious failure, or that would have gone unnoticed for a while longer. [23:52] yeah [23:54] mwhudson: sure [23:55] mwhudson: Regarding point 3), it basically calls DatabaseLayer.testTearDown twice in succession. [23:55] No idea why. [23:55] Hmm. [23:55] I guess it tears down in the sub-suite run, then at the end of the test itself. [23:55] wgrant: but why doesn't that always fall over in a heap? [23:56] Doesn't it do some optimisation about not tearing down the DB if it's not dirty? [23:56] maaaybe [23:58] https://code.edge.launchpad.net/~mwhudson/launchpad/aiee-everything-is-broken/+merge/22185 [23:58] thumper: ^ [23:58] mwhudson: is your branch one that reverts the launchpad change? [23:58] thumper: it's a little more delicate than that [23:58] mwhudson: I just read your email thread [23:58] if (ConnectionWrapper.committed and ConnectionWrapper.dirty): [23:58] PgTestSetup._reset_db = True [23:58] ConnectionWrapper.committed = False [23:58] ConnectionWrapper.dirty = False [23:58] if PgTestSetup._reset_db: [23:59] self.dropDb() [23:59] PgTestSetup._reset_db = True [23:59] Looks like if you never commit, it probably won't drop the DB. [23:59] though i think my diff still says "XXX outraged XXX goes here" [23:59] # XXX Outraged XXX goes here. [23:59] yeah