[00:13] * mwhudson lunches [00:48] * NCommander is amazed at how long the LP test suite takes to run [00:49] NCommander: welcome to our world [00:51] mwhudson: how long does a fullrun take? [00:51] NCommander: about 4 hours, i think [00:51] mwhudson: (I'm seeing some codehosting failures which I'm concerend about, since I didn't touch that bit of the code) [00:51] mwhudson: this has been going for at least 6-7 [00:51] NCommander: what hardware? [00:51] that's pretty slow though [00:52] mwhudson: 2.2Ghz dual core intel dual core processor [00:52] 8 GiB of RAM running on an SSD [00:52] Got a few weird failures [00:52] mwhudson: I'm running lucid versus karmic though, which may explain some of the failures [00:53] ah possibly [00:53] mwhudson: I know its possible to run the test suit eon EC2; can you shed some light into that? [00:54] NCommander: have you seen https://dev.launchpad.net/EC2Test ? [00:55] mwhudson: how much does EC2 cost for something like this? [00:55] a few $ i guess [00:55] it uses a c1.xlarge instance for 4-5 hours [00:56] mwhudson: might be worth it although I can't expense it as I'm not on the LP team :-/ [00:56] NCommander: i can run it for you if you like [00:56] * mwhudson is currently trying to find stuff to delete so i can upgrade to lucid without scaring myself [00:56] mwhudson: that may be worthwhile, although I'm concerned I may have actually broken part of LP (first db changes I ever made) [00:57] NCommander: i don't see why that means you shouldn't run it under ec2 though? [00:57] mwhudson: oh, no, that was just a side comment, not a blocker :-) [00:57] :) [00:57] NCommander: where is the branch on lp? [00:58] mwhudson: its at ~mcasadevall/launchpad/raw_source_changelog, but I'm thinking I should upload this last round of changes as a new branch [00:58] I rewrote the functionality to use librarian rather than storing in teh DB [00:58] NCommander: well, ok, i'm not going anywhere, so if you want me to test a branch, just ping me [00:58] * NCommander also feeds a second code review may be a good idea [00:58] mwhudson: give me a sec [01:02] mwhudson: lp:~mcasadevall/launchpad/raw_source_changelog_librarian [01:03] NCommander: what email address do you want results sent to? [01:03] (/msg if you like...) [01:27] mwhudson: mcasadevall@ubuntu.com [01:28] mwhudson: (sorry for the delay in answering) [01:28] NCommander: ok, starting up ec2 test [01:33] mwhudson: thanks. I'm thinking I should probably turn my old laptop into running UEC and then run ec2test on it :-) [01:33] heh heh [01:34] not if you want results quickly i suspect [01:34] mwhudson: the old laptop is 2.2Ghz 4GiB of RAM [01:34] its not that slow [01:35] oh ok [01:35] no :) [01:39] mwhudson: Total: 25575 tests, 16 failures, 19 errors in 510 minutes 32.100 seconds. [01:39] irony, it just finished [01:40] I tee'ed the output, but I don't even know if the failures are due to lucid, or due to something in my environment :-/ [01:42] NCommander: well the results from ec2 will tell you some of that [01:42] mwhudson: thanks :-/ [02:17] NCommander: The only known Lucid failure is in apt-ftparchive. [02:17] NCommander: Lucid's adds SHA1 and SHA256 hashes in addition to pre-Lucid's MD5s. [02:17] Any other failures are probably real problems. [02:18] wgrant: I think my postgresql server dumped out towards the end, I got a bunch of *strange* errors with write errors :-/ [02:19] Heh. [02:19] Does anybody happen to have an idea of where the time is taken in the primary archive publisher? [02:24] wgrant: well, at least then I know which tests to check [02:25] NCommander: Right. [04:16] wgrant: not an accurate one; its a problem I'd like to look at. If you choose to look at it I'd be very happy to be a sounding board. [04:21] lifeless: From what I can tell, publish-distro.py takes about 20-25 minutes currently. [04:21] I've cut two or so minutes out of that locally by optimising domination. [04:22] And I think most of the rest mus tbe in the apt-ftparchive integration. But I'd like to know for sure. [04:23] (apt-ftparchive integration is the one bit I can't sanely simulate locally, since it requires lots and lots of real packages, not just fake contentless ones) [04:24] wgrant: You could simulate by mirroring karmic and "publishing" bits of current lucid. [04:25] persia: But that requires a local copy of the archive, which has a sane Internet connection as a prereq. [04:27] The pacific ocean clearly needs a denser net of high-capacity optical fibre [04:28] But anyway, I have just reduced native index generation time for largely-unchanged Ubuntu-sized binary pockets to 2-3 seconds. [04:28] or electrical flows that exceed the speed of light [04:29] It would be very interesting to know how much longer than 20 seconds the apt-ftparchive integration code takes to run. [04:29] spm: yeee-haw! [04:29] Can anybody see cron.publish-ftpmaster logs? [04:29] Is there some test I can run locally that doesn't require me to setup/install launchpad first? [04:29] mwhudson: indeed [04:30] persia: No. [04:30] But the times are all in the logs that presumably somebody can see. [04:30] dammit, now i want to watch that movie again, and i only watched it last week or so [04:30] Which movie? [04:31] dr strangelove [04:31] Ah. [04:31] spm posted http://davideubank.files.wordpress.com/2008/09/slim-pickens_riding-the-bomb_enh-lores-720p.jpg with the comment "the life of a losa" or something like that [04:32] Ahhh. [04:32] it' an accurate description. [04:34] spm: Does cron.publish-ftpmaster log anywhere that you can check quickly? I am interested in the time taken by index generation (starting with '* Step C', ending with '* Step D') [04:34] looks... [04:37] urg. we don't rotate the poppy log. /me hads a MUST FIX. [04:41] wgrant: http://paste.ubuntu.com/399107/ [04:41] Aha. [04:42] So we can cut 17 minutes of the publisher run-time to probably well under one. [04:42] Thanks spm. [04:42] heh; if you can cut the time down? we'll thank you! :-) [04:46] wgrant: This isn't all part of some master plan to release 48 times a day, is it? [04:46] * NCommander waves to persia [04:46] persia: ??? there's more minutes than 48 in a single day! :-P [04:48] spm: Sure, but one has to run the entire publisher for each release, so it would need lots more optimisation to release every minute. [04:48] persia: To at least make it possible to drive it that fast if necessary, yes [04:49] Actually, with the "if necessary" bit, that's a good thing. I'm just not sure I'd like it that fast by default: sometimes folk catch stuff within a run, which gets hard if it gets smaller. [04:49] (this maaaay not be unrelated to Saturday's events) [04:49] heh [04:50] persia: well, given a fast enough machine, it should be possible to publish once a minute [04:51] NCommander: s/fast enough machine/less pathetic publishing code/ [04:51] NCommander: You'd also need a faster network, faster mirrors, etc. [04:51] persia: or a longer minute [04:51] feh. [04:51] I mean, if you redefine a minute ... [04:51] Microsoft redefined stable after all, it could be done :-) [04:52] My timeline has a 19 minute gap :( [04:52] My minute is 551,557,906,200 vibrations of cesium-133 at STP. [04:53] My minute is 0.06 kiloseconds [04:54] My second however is define as 0.0864 standard seconds. What sorta second am I? [04:54] wgrant: Which bit is missing? [04:54] a 2nd rater ? [04:55] NCommander: you really left yourself open to that one :-) [04:55] er [04:55] *0.864 seconds [04:55] spm: its a decimal second :-) [04:56] persia: There is a 19 minute hole containing dsync's link-dupes run, cron.germinate and the cocoplum->syncproxy sync. The last probably takes three minutes. [04:56] So one of the first two is taking an awful long time. [04:56] link-dupes [04:57] I'd suspect so, yes. [04:57] I'm not quite sure why it's run. [04:57] It takes > 2 hours to run link-dupes against my mirror set. [04:57] Do you know? [04:57] Massively saves space. [04:57] Even though we symlink matching files anyway? [04:57] Erm. [04:58] Then no, doesn't do anything at all. [04:58] I'd rather see matching files hardlinked, because it mirrors better, but that's just preference. [04:58] So even though it has no effect, it still takes forever to run fdupes. [04:58] I mean, when we need a file published in multiple locations, we symlink the non-master copy as any good archive software should. [04:59] This is well before the archive is split into archive/ports/security, so I don't see why there would be any dupes. [04:59] If it's pre-mirror-split, there should be no dupes. [04:59] Right. [04:59] Or rather, only known multi-publication. [05:01] I'm guessing you use link-dupes because you maintain mirrors of both ports and non-ports (or portions thereof)? [05:02] Actually, I use my own hacked fdupes-based script, and also mirror debian, but yes. [05:05] So if there's little point running it, and we fix NMAF to support extra-overrides and migrate the primary archive to use it, *and* my two optimisation branches actually work properly, we can cut more than half an hour off the 40-minute-ish publisher. [05:06] Which could shorten the time-to-fix for something like Saturday's incident down to ~90 minutes with manual publisher runs. [05:06] s/90/30 [05:06] what happened on saturday? [05:06] wgrant: consider build times too. [05:06] Upload CDBS, wait for it to build in a few minutes, publish. Upload two broken packages, built in 10 minutes, publish. [05:07] mwhudson: Lucid broke right beta 1. [05:07] mwhudson: some updates were published to lucid that broke gnome-panel autostart on login. [05:07] gnome-panel and nautilus broke so they wouldn't autostart. [05:07] Leaving all the willing testers with no desktop at all. [05:07] wooo [05:08] (or at least everyone who logged out and logged in after the updates) [05:08] i see.... [05:08] persia: Which lots of people did, after they couldn't open folders any more :/ [05:08] speaking as someone who's 7 mins from downloading for the u/g. is it all fixed on latest updates? [05:08] Oh, yeah. non-logout/non-reboot is the significant minority. [05:08] just out of totally idle curiosity, what versions of things contain the fixes? [05:08] spm: zing! [05:08] It's fixed. [05:08] mwhudson: -0ubuntu2 of both of those is broken. [05:09] -0ubuntu3 is safe. [05:09] It's been fixed since Saturday night. [05:09] It just took a lot longer than it could have if the publisher didn't take an hour. [05:09] mmm [05:09] i have -0ubuntu1 of those [05:09] i guess my mirror aint so fresh [05:09] having my main workstation cactus is a pain; but not unsurvivable. karmic forced me onto my laptop for a week or 2; totally broke my workstation's startup. [05:10] Getting the publisher down to ~10 minutes makes it something people will be more willing to babysit too. [05:10] Right. [05:12] Plus it'd be less of a pain around releases, when you see release managers tossing up whether to run it now or not. [05:13] Absolutely. [05:13] ha, ha [05:13] 255 updates [05:55] mwhudson: thanks for ec2ing my branch. TIme to fix some failures [05:55] mwhudson: you got less failrues and errors than I do it [05:58] what can cause this: ProgrammingError: permission denied for relation sourcepackagerelease [05:58] NCommander: have you encountered database/schema/security.cfg yet? [06:00] mwhudson: nope [06:01] NCommander: in that case "ProgrammingError: permission denied for relation sourcepackagerelease" means you need to make friends with this particular file [06:01] NCommander: different bits of launchpad connect to the database as different database users [06:01] and the different users have permissions to access the various tables in different ways [06:03] mwhudson: ugh. I have no idea how to fix this, since there is a SELECT permisison already in place [06:04] NCommander: for which user though? [06:05] mwhudson: librariangc. Looking at the code, its trying to DELETE, but I'm kinda confused on how objects leave librarian [06:07] mwhudson: it seems to me its trying to delete rows, but I'm not understanding why librariangc isn't happy [06:08] mwhudson: oh wait ... [06:09] mwhudson: alright, I think I figured out why the librariangc tests are failing :-) [06:12] NCommander: grats :) [06:18] mwhudson: I assume I need to re-make schema after editing security.cf [06:18] *cfg [06:19] NCommander: i think you can just run python database/schema/security.py launchpad_ftest_template [06:19] or something [06:19] but i usually just run make schema, yeah [06:19] mwhudson: thanks, I'm still somewhat new at all this LP hackery. I'll probably ask you to re-run ec2test for me since I got a lot mor efailures on 127.0.0.1 than ec2 did :-/ [06:20] * NCommander will probably setup EC2 sometime in the future but can't ATM :-/ [06:20] NCommander: It's pretty easy, it took me about 10 minutes [06:21] StevenK: how much did it cost per run? [06:21] Roughly $3.50 [06:22] But they don't charge until the end of the month [06:22] StevenK: that can add up fast -/ [06:22] i spent a lot on ec2 in february [06:22] luckily i can expense it... [06:32] mwhudson: woo, librarian fixed [06:32] * NCommander now works on registery failures [06:33] NCommander: i'm going to be going afk soon, but europe should be getting up soon ish :) [06:47] mwhudson: hrm, is it possible for a test to be failing in db-devel? [06:47] * NCommander can't see how this failure is possible [06:47] er [06:47] or how is impossible [06:48] NCommander: Stuff doesn't land if the tests fail [06:48] StevenK: not true any more, sadly [06:48] mwhudson: that was what I was afraid of [06:48] Awww [06:48] NCommander: which tests fail for you? [06:48] mwhudson: lp.registry.tests.test_distroseries.TestDistroSeriesPackaging [06:48] oh [06:49] that doesn't seem to be failing in buildbot [06:49] mwhudson: NameError: global name 'gpg_key' is not defined [06:49] Looking at the file in question though, there is only one refernece to gpg_key in the whole thing [06:49] ! [06:49] is that in devel or db-devel? [06:50] mwhudson: db-devel, but hold on [06:50] * NCommander might have done something stupid [06:51] Damn. Turns out my 20 second index generation run had broken bz2 compression. Fixing that blows it out to more than a minute :( [06:51] wgrant: A minute versus 24, though? [06:51] NCommander: i don't see gpg_key in that file [06:51] (on either branch) [06:51] 17-18, but yes. [06:52] The rest of the 24 is being cut away in other places. [06:52] mwhudson: I dunno, I think I went a little merge happy at some point, and lack of sleep is not helping [06:53] lack of sleep often doesn't help with hacking [06:57] mwhudson: no, but my girlfriend went ona cleaning spree [06:57] and my bed currently isn't where I left it [06:58] NCommander: haha [06:58] Are we week 2 or 3 at the moment? [06:58] wgrant: just starting week 3 [06:58] Aha. [06:59] * wgrant had better get his last PPA download stats branch in order, then. [07:00] * NCommander broke something bad [07:00] very bad [07:25] anyone around who can run the ec2test script OR run the test suite locally without getting unrelated failures [07:25] * NCommander has it fixed === mthaddon` is now known as mthaddon [08:57] bigjools: Morning. [08:57] g'day [08:58] What are the blockers for NMAFing primary? Just extra overrides? [08:59] yep, barring any hidden bugs [08:59] and possibly performance [08:59] Performance is definitely an issue. [08:59] But with a bit of caching I can publish all indices for a larger-than-Ubuntu archive in 20 seconds. [09:00] (one series thereof, that is, like most normal runs) [09:02] wgrant: using NMAF? [09:02] bigjools: Yeah. [09:03] it would be interesting to do like for like comparisons [09:03] Hello [09:03] hey mrevell [09:03] wgrant: did you see Bug 543162 BTW? [09:03] Bug #543162: launchpad-buildd exception: Try to register an already registered process. [09:03] bigjools: I was going to, but a-f requires real sources and binaries, whereas NMAF doesn't care if they're garbage. [09:04] wgrant: dogfood FTW :) [09:04] bigjools: I have, yeah. lamont poked me about it at the time to check if there was anything obvious. [09:04] it's an odd one - I can't work out if twisted blew up or the slave code [09:05] bigjools: Is dogfood a good performance indicator? [09:05] wgrant: not against anything else, but relative to itself of course it is [09:05] bigjools: if it's related to tearing down twisted code, that behaviour changed recently. [09:05] I mean, a-f is going to hit the DB in different ways to NMAF. [09:05] wgrant: indeed [09:06] noodles775: was that in lucid though? [09:06] I tried to use memcache, but it was far too slow :( [09:06] bigjools: nm, not on cesium. [09:07] ironic :) [09:07] noodles775: It's the buildds themselves, which are in fact pre-Wellington. [09:08] wgrant: Right (just reading the bug). [09:11] It's only been seen on armel, but it happened on two of them (at a reasonably inconvenient, almost simultaneous time) [09:13] bigjools: Also, do you know the purpose of the dsync link-dupes call in cron.publish-ftpmaster? There shouldn't be any duplicates... [09:14] no idea, it's always been there [09:14] (and it probably takes forever and then a little while longer (or just 15 minutes) [09:18] bigjools: Who's likely to know? [09:19] 'cause slow publishing makes me sad and breaks lots of users' systems. [09:22] wgrant: probably people who have left :/ [09:23] I can check with IS [09:23] It's from dak, so elmo probably does. [09:50] Morning, all. === stub1 is now known as stub [09:53] good morning Launchpadders [10:25] What do I need to do to get the PPA download stats script running on production? [10:25] wgrant: is it a cron job? [10:26] where is it? [10:27] bigjools: It's a cron job. cronscripts/parse-ppa-apache-access-logs.py [10:27] ok [10:27] wgrant: we need to run it on dogfood first [10:27] Right. [10:27] Or staging. [10:28] And it needs production configs. [10:28] not staging [10:28] It doesn't need a real archive. Just logs to be put somewhere. [10:28] I'll work on the configs and test it on DF [10:28] Thanks. [10:29] hmm busy day [10:34] bigjools: Ew, history deletion. [10:35] wgrant: ? [10:35] bigjools: Deleting archives and EVERYTHING with them. [10:35] wgrant: yep [10:35] like deleting a branch [10:35] Yes, both are repulsive. [10:36] deletion is deletion, not some half-assed attempt [10:36] if you want history, keep the f***ing thing [10:36] or it can be disabled [10:36] If you want malicious or misled users to be able to inappropriately destroy things irrevocably, let them actually delete thousands of records in one click. [10:37] If you want users to be able to remove the only records of their malicious activities on the buildds... [10:39] Also, how do binary copies work? [10:39] If I delete an archive that has had binaries copied from it, the builds are more than a little awkward. [10:40] the binaries won't be deleted [10:40] What happens to the builds? [10:40] We cannot delete them. [10:40] neither those [10:40] So we lie about their archive? [10:41] no, we change the code to say their original archive was deleted [10:41] did you read the bug? [10:42] It didn't cover builds in particular. I guess stuff might not break if the archive disappears. [10:43] that's what the test suite is for === elmo_ is now known as elmo [11:46] ok, seriously, this sucks [11:46] I've turned referer back on, and I still can't add attachments to bugs [11:47] What's the error? Still 'Unexpected Form Data' with a 500? [11:47] oh, lala [11:47] sorry, never mind [11:48] it's my fascist firefox apparmor profile [11:48] \o/ [11:48] Heh. [12:02] adeuring: PQM is behaving like it has not seen my submissions (http://paste.ubuntu.com/399248/). Am I missing something? Can it have built so fast that I missed it? :-) [12:03] adeuring: (oh, I made a cut-and-paste error in that paste -- the submissions actually do match their directories, don't worry) [12:03] kfogel: ;) [12:04] adeuring: https://pqm.launchpad.net/ shows nothing though [12:05] adeuring: bbiab [12:10] wgrant: ok I have dogfood updated, what configs does that cronscript need? it looks generic [12:11] bigjools: [12:11] [ppa_apache_log_parser] [12:11] logs_root: /srv/ppa.launchpad.net-logs [12:11] Paths will vary. [12:11] wgrant: what about where it puts results? [12:12] bigjools: The DB? [12:12] ah nm the config is ppa specific [12:12] how does it know to read that config? [12:14] bigjools: It's used in cronscripts/parse-ppa-apache-access-logs.py [12:14] * bigjools slaps face [12:14] I was looking at the wrong script, sorry [12:14] Heh. [12:14] I thought something might have been a bit odd. [12:14] yeah :) [12:22] wgrant: so does this script depend on only ppa logs being in the logs_root ? [12:22] bigjools: It'll ignore them if they don't match a PPA binary for any reason. [12:22] So it shouldn't matter. [12:23] mmkay [12:24] wgrant: well it does if one of the files is not readable :/ [12:24] Yeah. [12:24] Not my code :P [12:24] * bigjools watches the buck flying overhead [12:25] It will also do strange things if you run both the librarian and PPA log parsers over the same directory, but that's unlikely to happen. [12:33] kfogel: did you perhaps receive a mail from pqm with an error message? [12:33] wgrant: it doesn't log much does it :) [12:33] and 2.1g resident :/ [12:33] * bigjools gets coffee while waiting for it [12:39] Can someone fix the grammar fail in the PPA description? https://edge.launchpad.net/~launchpad/+archive/ppa [12:40] bigjools: Did you see the warning in the docstring (lovingly copied from the librarian one) [12:43] (I didn't think there'd be many logs on dogfood -- sorry) [12:47] maxb: done [12:47] thanks [12:48] wgrant: I fiddled the log so that it only appears as a symlink in another dir [12:50] m_o_d: Did you run rocketfuel-setup? [12:50] wgrant: yes [12:50] m_o_d: Did you see it installing lots of packages? [12:51] wgrant: yes, my error is when i run make schema [12:51] m_o_d: I know. But it's complaining about something that rocketfuel-setup would have instaled. [12:51] Do you have python-tickcount installed? [12:51] bigjools, any recommendations for coffee from hasbean? [12:51] yes, 0.1-0ubuntu10launchpad1 [12:52] m_o_d: Which version of Ubuntu are you using? [12:52] jml: oh yes :) === mrevell is now known as mrevell-lunch [12:53] wgrant: lucid [12:53] jml: http://www.hasbean.co.uk/products/Guatemala-Finca-Puerta-Verde-Antigua-2009%252d2010.html [12:53] you had that last week [12:53] well, on Friday when the other stuff ran out [12:55] m_o_d: Try to apt-get upgrade. Does that propose anything python-related? [12:55] bigjools, thanks. [12:55] jml: also check out the Bolivia Machacamarca [12:56] jml: that one is simply awesome [12:56] bigjools: Is the script still doing stuff? [12:56] wgrant: no it finished some time ago, I got sidetracked [12:57] binarypackagereleasedownloadcount has got 4736 rows now [12:57] Excellent. [12:57] bigjools, ta. I'll order the Guatemala one and see how I go. [12:58] bigjools: Do you have my latest branch? [12:58] wgrant: I'm using db-devel [12:58] Ah, the webapp is not up. [12:58] bigjools, I'll have to do a taste test. [12:58] jml: most stuff from central America is great [12:59] wgrant: noodles775 is playing with iut [13:04] adeuring: yeah -- testfix/rc [13:04] adeuring: :-( [13:09] bigjools, or maybe anybody else running on lucid+py2.6. have you seen this error? [13:09] http://paste.ubuntu.com/399288/ [13:10] barry: no not seen that [13:10] maxb: Arrrrgh it's back. ^^ [13:10] barry: but then I'm not really running 2.6 :) [13:10] barry: That one has been plaguing me on and off since I started using Lucid a month in. [13:10] maxb has seen it too. [13:10] wgrant, bigjools, maxb arg! :) [13:10] *blink* [13:11] UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages) <--- that's a new one [13:11] wgrant, maxb, bigjools i really don't want to install python2.5 if i can help it. [13:11] barry: it's part of lp-dev-deps isn't it? [13:11] barry: You have to. Launchpad doesn't run on 2.6. [13:11] maxb: i do have python-setuptools installed, which should provide distribute [13:12] wot wgrant said [13:12] barry: You want to install python2.5. Unless you want to dedicate your entire time to massive porting project for the forseeable future, you want to install python2.5 [13:12] (this is a bug that really needs to be fixed soon) [13:12] and I thought barry was fixing it actually :) [13:12] dang. [13:12] it's gotta run on 2.6 pretty soon, though, right? [13:13] adeuring, wgrant: have a sec to chat about https://bugs.edge.launchpad.net/malone/+bug/78565/comments/16 (ironically, you'll have to navigate from there to the bug for context, becuase the bug isn't fixed yet). [13:13]