[00:01] wgrant: there's some stuff from 23rd Nov and also 29th: http://paste.ubuntu.com/538152/ [00:04] wallyworld: Any references in the p-u log to the directory mentioned in that b-m line? [00:04] I wonder if this is the race where p-u will have moved it to failed. [00:04] Because it ran before the status was set. [00:05] does anyone else get a 404 on this - http://ppa.launchpad.net/testing-cabal/ppa/ubuntu/dists/maverick/main/source/Sources.gz ? [00:06] wgrant: you mean this directory? 20101129-213106-2078946-PACKAGEBUILD-2081126 [00:06] wallyworld: That one. [00:06] lifeless: Yes. Should it be there? [00:06] I assume so [00:06] there are packages... [00:06] http://ppa.launchpad.net/testing-cabal/ppa/ubuntu/dists/maverick/main/binary-amd64/Packages.gz [00:06] is 404ing for me too [00:07] https://code.launchpad.net/~testing-cabal/+archive/archive [00:07] reckons theres a bunch of stuf in the ppa [00:08] Ugh, indeed. [00:09] I'd like to use this stuff [00:09] lifeless: er, the PPA is named 'archive', not 'ppa'. [00:09] Your URL is wrong. [00:09] oh right [00:09] apt-add-repository fail [00:09] Hm, it let you do that? [00:09] yes [00:10] wgrant: don't appear to be any references to that exact directory. there are references to 213106 from earlier dates eg 20101115 [00:10] wallyworld: That's just a timestamp. [00:11] wgrant: yeah but i was thinking earlier uploads may have worked [00:12] wgrant: so you think it may be a race condition? can we kick off a manual upload? [00:13] spm: Do you have a moment? [00:19] wgrant: sure [00:19] spm: Can you see if there's a 20101129-213106-2078946-PACKAGEBUILD-2081126 directory in cesium's buildd-manager upload root somewhere? [00:19] /srv/launchpad.net/builddmaster [00:21] ./failed/20101129-213106-2078946-PACKAGEBUILD-2081126 [00:21] * jelmer is here but was about to get some sleep [00:21] jelmer: You've been working on this, right? [00:21] I wonder if we should just throw it back into incoming. [00:21] wgrant: I haven't read the context - what's going on exactly? [00:22] jelmer: Binary upload stuck UPLOADING, with its directory in 'failed'. [00:22] jelmer: No logs. [00:23] wgrant: Yeah, moving it back to failed should be ok in that case. I did a fix recently to not move things to failed as quickly but keep them in incoming until we've committed the change to the build status. [00:23] wgrant: That hasn't been deployed yet though. [00:24] spm: Could you move that dir from failed back to incoming? [00:26] sure, one sec (was encoffeenating) [00:26] done [00:28] spm: Thanks [00:28] spm: Thanks. [00:28] And now we have an upload log. [00:28] np, and heyo jelmer [00:28] Hah, it's been superseded already. [00:28] wgrant: spm: thanks for helping with this. i'll let the user know the upload is retrying [00:29] wgrant: at what point should a ppa package become installable on client machines [00:30] lifeless: After the binaries are built and published. [00:31] grrr [00:31] why isn't apt seeing it [00:31] ahh [00:31] http://ppa.launchpad.net/testing-cabal/archive/ubuntu/pool/main/p/python-testtools/ [00:31] i386 for all. [00:31] https://code.launchpad.net/~testing-cabal/+archive/archive/+packages [00:31] wgrant: expand ython-testtools - 0.9.7+bzr146~ppa116~maverick1 [00:31] Publishing details [00:31] * Published 15 minutes ago [00:32] lifeless: That be the source. [00:32] Note how it has a green flashing gear. [00:32] no [00:32] And it says the binaries aren't pblished. [00:32] is that what it is? [00:33] lifeless: Which series? [00:33] wgrant: colour means approximately nothing to me in UIs [00:33] Natty is still building, and Maverick and Lucid should have a warning about unpublished binaries when you expand their row. [00:33] wgrant: i see a dark octagon, which might be greed. [00:33] thumper: ping [00:33] *might be green* [00:33] wallyworld: aye [00:34] someone needs to do user testing on this [00:34] lifeless: Right, there's that, but you'll get more details if you expand the row. [00:34] wgrant: i did [00:34] lifeless: Launchpad does not do UI. [00:34] wgrant: Launchpad does. [00:34] thumper: unknown dailydeb implies out of date bzrlib? [00:34] some things have more room to improve than others. [00:34] wgrant: is there a 'this needs work' bug ? [00:35] wallyworld: eh? [00:35] lifeless: I don't think so. [00:35] thumper: https://pastebin.canonical.com/40235/ see the error at the end. i seem to recall dailydeb is a recent addition to bzrlib? [00:36] Unable to load plugin 'builder' from '/usr/lib/python2.6/dist-packages/bzrlib/plugins' [00:36] line 434 [00:37] * wallyworld slaps forehead [00:37] which hints at a bigger problem [00:39] thumper: the log came from https://launchpadlibrarian.net/59824676/buildlog.txt.gz [00:40] thumper: does that mean we have something to look at on our end? [00:40] certainly looks like it [00:40] I would have thought that this would have come up in QA testing [00:42] thumper: yeah. so is it a matter of updating some servers to get the necessary plugin installed? [00:42] wallyworld: I'm not entirely sure [00:42] wallyworld: check with abentley or lamont [00:43] thumper: thanks, will do [00:48] GRRR!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1111 [00:49] * thumper runs bzr blame [00:49] thumper: ? [00:49] wallyworld: I think I've found "my" bug [00:49] thumper: you mean the test isolation one [00:50] ? [00:50] yep [00:50] a monkeypatch [00:50] Hierarchy.makeBreadcrumbForRequestedPage = lambda self: None [00:50] what was it put there for? [00:51] # Monkey patch Hierarchy.makeBreadcrumbForRequestedPage so that we don't [00:51] # have to create fake views and other stuff to test breadcrumbs here. The [00:51] # functionality provided by that method is tested in [00:51] # webapp/tests/test_breadcrumbs.py. [00:51] but the method is never put back [00:53] doh! [00:54] aye [00:55] lifeless: do you know if we have a way to nicely run something at the end of a doctest? [00:55] can we use addCleanup? [00:55] no [00:55] :( [00:55] you could add an empty fixture to globs [00:55] e.g. 'test_fixture' : Fixture() [00:56] then in the setUp set it up, and in the teardown do cleanUp [00:56] and in the test do test_fixture.addCleanup(...) [00:56] rockstar: ping! [00:56] that sounds like a good idea... [00:56] I may well do it [00:57] I'm running all the tests in the layer again to see if my monkey patch reversal now makes it work [00:57] I think my test failed due to adding more tests caused some re-ordering [00:57] which caused the failure to be shown [00:57] the problem was there before [01:01] thumper: you know the rules: last person to touch it broke it :-P [01:01] :P [01:06] ugh. does anyone happen to know which table project groups live in? need to rename a new one. [01:06] project [01:06] projects are products [01:06] project groups are projects [01:06] so projects are in product; and project groups are in project? logical :-) [01:06] snap! [01:06] echangingourmindisn'teasy [01:06] :-) [01:06] spm: so how about that proxypass rule [01:06] slowly atm [01:18] why do I not have memcached working? [01:19] anyone got any ideas? [01:24] wallyworld, hi [01:25] rockstar: just wondering if you wanted to review the product merge queue branch :-) [01:25] rockstar: also, you have a qa task on our board [01:25] rockstar: can you move it / do it? [01:25] thumper, sure. [01:26] ta [01:29] wallyworld: It looks like bzr-builder is there, but apt_pkg is not. [01:31] abentley: so this is on one of our servers I think? wonder how that could have happened? [01:31] wallyworld: This would be on one of our builders. [01:31] wallyworld: Please post a link to the actual build-- the build log doesn't provide navigation to it. [01:32] abentley: i'll have to get it. i got the log from someone on #launchpad. i'll see if i can ping him [01:39] wallyworld: I believe it is this one: https://code.launchpad.net/~audacity-team/+recipe/daily.karmic/+build/9502 [01:40] abentley: right you are [01:41] abentley: so, who do i talk to to get the issue looked at? [01:42] wallyworld: james_w could tell you what's apt_pkg, because it doesn't look like it's bzr-builder directly. [01:42] \o/ [01:42] wallyworld: lamont can install updated packages on the builders. [01:42] finally landing the binary build change [01:43] I emailed the list with my problem resolution [01:43] abentley: thanks. so it seems we just need to install and/or update apt_pkg on that server? [01:44] thumper: most importantly, did you figure out who to blame? :-) [01:44] wallyworld: No, there's detective work needed. What is apt_pkg used by? Why didn't apt install it when it installed whatever requires it? [01:44] yes [01:44] but I won't publicly shame [01:45] wallyworld: Possibly we need to fix the dependencies on some packages. [01:45] thumper: didn't expect you to, just having the satisfaction of knowing it wasn't you is enough :-) [01:45] :) [01:45] yes, it was not me [01:46] abentley: thanks. will pass on the details. does it seem like it may only be an issue with one server? should I tell the user try again? [01:47] wallyworld: it seems likely this is a single-server issue. We also had another bug with the wrong version of something being installed: https://bugs.launchpad.net/bugs/682941 [01:47] <_mup_> Bug #682941: no such option --append-version in recipe build [01:47] abentley: cool. many thanks for your help. [01:50] wallyworld: It appears that a few servers were recently added to the build farm. That may be at the root of this: https://lpstats.canonical.com/graphs/BuildersActiveVirtual/ [01:51] abentley: I'm smiling at the daily build spikes :) [01:51] abentley: could be [01:54] thumper: They're nice, but we may want to look into staggering them to make the build farm's responsiveness more consistent. [01:54] abentley: yeah. I've heard bigjools request as much [01:54] * thumper goes to collect kids [01:55] wgrant: fyi. patched, zapped, unpatched. looks good. thanks! [01:58] wallyworld: in the meantime, see if any new reports were built on sandpaperfig or lansones, because maybe only those two are problematic. [01:59] abentley: ok [02:00] === Top 10 Time Out Counts by Page ID === [02:00] Hard / Soft Page ID [02:00] 287 / 6525 Archive:+index [02:00] 170 / 390 BugTask:+index [02:00] 43 / 260 Distribution:+bugs [02:00] 27 / 140 ProjectGroupSet:CollectionResource:#project_groups [02:00] 18 / 1 Archive:EntryResource:syncSource [02:00] 14 / 35 DistroSeries:+queue [02:00] 13 / 259 Distribution:+bugtarget-portlet-bugfilters-stats [02:00] 12 / 38 Milestone:+index [02:00] 12 / 28 MailingListApplication:MailingListAPIView [02:00] 12 / 4 Person:+bugs [02:02] spm: Yay, thanks. [02:16] wgrant: I can has Archive fix. [02:16] wgrant: iz getting worse. [02:48] \o/ db-devel builder passed third time lucky [03:17] wow: http://odata.stackexchange.com/ubuntu/s/665/too-many-comments [03:17] > Stack Exchange Data Explorer allows you to run arbitrary SQL queries on the Stack Exchange family of sites, share those queries with your friends and explore interesting queries. [03:23] I imagine it's not really SQL. [03:23] Well, at least not direct. [03:26] i think it is [03:26] but it's against a warehouse db, not the real one [03:27] Ah. [03:27] may still be filtered [03:27] i think they strip private stuff on the way to the warehouse [03:28] they may have much less private stuff, and a simpler delineation of it, than lp would have even now [03:28] Yeah, true. [03:38] thumper, these new incremental diffs in merge proposals are the best thing since slide bread [03:39] beuno: I'm glad you like them, abentley is fixing a bug in them :) [03:39] I lovez em! [03:39] They are really great. [03:40] Now all we need is comments on lines, somehow. [03:40] yes we do [03:40] beuno, thumper: they're disabled until we can get that bug sorted, which unfortunately is a bzr bug, but I've submitted a fix. [03:41] abentley, that's fine, I already got a taste of them, will keep me warm and fuzzy for a good week [03:45] abentley, thumper: yes they're nice, thanks for adding them! [03:46] spiv: you're welcome. [03:46] Are they restricted to betatesters? [03:47] abentley, thumper: I was wondering if it might be nice to be able to have a javascript thing to collapse them all for times when you want to just skim the conversation [03:47] thumper: why does self.assertEqual(x, y) fail unless i removeSecurityProxy(x) and removeSecurityProxy(y)? i'm logged in as the owner of x and y [03:47] +1 [03:47] e.g. if you are a patch pilot and want to quickly get a sense of where the discussion is at [03:48] wallyworld: probably a buggy __eq__ [03:48] spiv: Could be. [03:48] wallyworld: sp(x).__eq__(sp(y)) -> x.__eq__(sp(y)) [03:48] lifeless: Or __eq__ falling back to is. [03:49] abentley: sure, which in the absense of sp's is reasonable. In our context its arguably buggy though [03:49] in this case i have two sets of branches, and i have to iterate over the contents of each set and remove the security proxies [03:50] for now, i would like to do the workaround and file a bug, does that sound ok? [03:56] StevenK: [03:56] StevenK: Morning. [06:05] publicrestrictedlibrarian default 0 on <== re-enabled [06:06] wgrant: give it a spin ? [06:08] lifeless: It's unbroken! Well done. [06:08] spm: looks good to me too [06:08] \o/ [06:12] lifeless: Is it intentional that bzr won't stack if you terminate the initial push? [06:14] wgrant: there is a bug [06:15] Ugh. Why do project bug pages now advertise Ubuntu? [06:16] the downstream link? [06:16] Yes. [06:16] they should show any distro packages, not just ubuntu [06:16] The last thing Launchpad needs is more Ubuntu-specific stuff. [06:16] I suspect the links are only in place for Ubuntu. [06:16] Big upstreams *do not want that sort of thing*. [06:17] mmmm [06:18] wgrant: I think the placement needs tuning [06:18] wgrant: but I don't buy your assertion - I think its overstated. [06:18] wgrant: many upstreams *want* easier querying of bugs in their stuff in Ubuntu. [06:18] lifeless: Hm? I thought it was an undeniable fact that Ubuntu-specificity was hurting Launchpad adoption. [06:19] wgrant: including bug ones. [06:19] s/bug/big/ [06:19] Sure. [06:19] poolie: http://rbtcollins.wordpress.com/2010/11/30/testrepository-iteration-for-python-projects/ fyi [06:19] But it needs to not just scream about Ubuntu on every page. [06:20] wgrant: file bugs on making it more subtle - point out that it needs to scale to baltix, linaro etc. [06:20] Ah yes, I keep forgetting that Linaro might force people to be less insane. [06:20] wgrant: Ubuntu specificity is a mixed blessing [06:21] wgrant: its totally unclear whether its a net win or negative [06:21] lifeless: It will get you lots of nice little projects. [06:21] But scare away any big ones. [06:21] wgrant: !cite [06:21] wgrant: seriously, hyperbole doesn't help analyse this. [06:22] wgrant: Ubuntu specific stuff can be harmful in at least two distinct ways [06:22] there may be defaults or hardcoded behaviour that make less sense for upstreams [06:23] which is a special case of 'less than the best of breed $tool than LP could be' [06:25] wgrant: secondly theres stuff that presents Ubuntu where it isn't welcome. [06:25] wgrant: its my opinion that the former has been a frequent problem in the past [06:25] wgrant: but I don't think this falls into that category. [06:26] lifeless: There have been both progressions and regressions around the former over the last few months, but it's probably not too bad now apart from the portlet on Product:+index. [06:27] And maybe this bug listing thing, but that may just be an artifact of the portlet. [06:28] on the positive side [06:28] Ubuntu is a very demanding and large customer [06:29] Indeed. [06:29] many things that Ubuntu experiences problems with other large users experience problems with. [06:29] lifeless: what do i do about bug 615788? it's been fixed but the user has reopened it against launchpad because they don't agree with the resolution. should i change to Invalid and ask them to file a new bug against ubuntu-website? [06:29] <_mup_> Bug #615788: gpg should use port 443 by default in order to work from behind firewalls [06:29] But the Ubuntu-specificity reduces Launchpad adoption, which reduces bzr adoption, which reduces Launchpad adoption, which... [06:29] wallyworld: file an rt ticket asking for a gpg server on port 80 [06:30] wgrant: Ubuntu may sometimes ask for things to solve their issues that don't work well for other users, but its *our job* to find a way to solve their problem creatively, in a way other folk also benefit from. [06:30] lifeless: ok. but we don't want the bug to stay assigned to launchpad do we? [06:30] wallyworld: why not? [06:30] lifeless, because it's not launchpad's issue? [06:30] "(I don't have permissions to reopen the original bug for ubuntu-website, so I reopened for launchpad instead - sorry for the hassle)" [06:31] Project devel build (260): STILL FAILING in 4 hr 5 min: https://hudson.wedontsleep.org/job/devel/260/ [06:31] lifeless: the bug when first filed was tiraged to ubuntu-website so i assumed that's where we wanted it to go [06:31] * lifeless shrugs [06:31] I'm on leave. [06:31] I'm telling you what I'd do. [06:31] but I really don't want to go deep on the discussion. [06:31] lifeless: ok. i wasn't going to bother you but you keep popping up :-) [06:31] wallyworld, I'd close it as "Won't Fix" for launchpad. [06:31] wallyworld: It doesn't need an RT. [06:31] It's Invalid in Launchpad. [06:31] and reading the bug, its already fixed. [06:32] he wants a bug open on gpg upstream. [06:32] that will need some clear communication. [06:32] It needs a fix in python-software-properties, and possibly Soyuz. [06:32] Er, yeah, Invalid. What wgrant said. [06:32] And maybe gnupg. [06:32] gnupg could certainly try 80 if the default fails epically. [06:33] wgrant: back to reputation [06:33] wgrant: Ubuntu may ask for crack, and we may do it occasionally, in response to its [large] scaling problems. [06:33] lifeless: Sure. [06:33] rockstar: so should the user file his own bug upstream? [06:34] wgrant: its up to us to help them solve their issues in such a way that our other users are helped too. [06:35] wgrant: the pressure to solve these issues is a positive pressure [06:35] wallyworld, well, he certainly shouldn't just re-open the bug on another project just because he doesn't like the fix. [06:36] rockstar: Well, what he did was more correct than what he initially tried to do. [06:36] wgrant: and secondly many upstreams care passionately about the quality of their *delivered to users* software. [06:36] It needs an LP config change and possibly an Ubuntu gnupg or python-software-properties change. [06:36] The sysadmin side is done. [06:36] wgrant, I guess I'm confused on how this issue relates to Launchpad at all. [06:36] wgrant: tighter reporting and easier analysis of their bugs in Ubuntu is desired and positive [06:36] It's a keyserver/client issue. [06:36] rockstar: it doesn't, he's confused. [06:37] rockstar: Launchpad produces links to the keyserver. It uses port 11371 [06:37] wgrant: we should change those links. [06:37] lifeless: Hence the config change. [06:37] I'd file a new bug for that though. [06:37] rockstar: so it does apply, though he is confused about how it all hangs together. [06:37] wallyworld, ^^ Close the bug as invaled, reference a new bug about ports for 11371 [06:37] * rockstar goes to bed [06:37] rockstar: thanks [06:38] wallyworld: Also Invalidate the synaptic task, while you're there. [06:38] wgrant: what should i ask for the 11371 port number to be changed to in the new bug? [06:39] 80 [06:39] lifeless: thanks. just wanted to make sure so i didn't enter something stupid in the bug report [06:42] gpghandler.port is the relevant config key. [06:43] wgrant: thanks to you too [06:44] wgrant: anyhow, think on those points [06:44] wgrant: I think its very much undecided whether the net influence is win or lose [06:44] Hmmm. [06:52] and we have plenty of room to make great decisions going forward to make the most of having a huge customer [07:43] Project db-devel build (175): FAILURE in 4 hr 16 min: https://hudson.wedontsleep.org/job/db-devel/175/ === almaisan-away is now known as al-maisan [08:29] losa ping [08:31] wgrant: hi [08:32] mthaddon: germanium's publisher has issues. [08:32] Hasn't published anything for a couple of hours, AFAICT. [08:33] ok, what stage are we at in terms of troubleshooting it? [08:33] mthaddon: 0 [08:33] Well. [08:33] I know that process-accepted.py isn't finishing. [08:33] I'm not sure if it's starting. [08:33] am? [08:33] 19:35 < lifeless> wgrant: its up to us to help them solve their issues in such a way that our other users are helped too. [08:33] 19:36 < lifeless> wgrant: the pressure to solve these issues is a positive pressure [08:33] bah, copy paste faillll [08:42] wgrant: would that be cron.publish-ppa or cron.daily-ppa (I'm assuming the former)? [08:42] mthaddon: The former. [08:43] wgrant: it seems to be doing stuff - the logfile is being written to fine with non-error messages (things like: 2010-11-30 08:42:53 INFO Processing http://ppa.launchpad.net/some/url) [08:43] mthaddon: What time does cron.daily-ppa run? [08:44] wgrant: 5:59 UTC [08:44] Hah. [08:44] er, 05:59 UTC, that is [08:44] Then I bet it takes a little over two hours to run, blocking publication. [08:45] Since everything woke up about 10 minutes ago. [08:45] As it did last time I poked you about this. [08:45] interesting - they write to the same log file [08:45] They share a lock too. [08:45] yeah [08:45] but it does look like it's cron.publish-ppa running now [08:45] Yeah, it's all good now. [08:45] k [08:45] I just have awful timing. [08:45] Again. [08:46] hello [08:46] s/Again/Still/ [08:46] Morning bigjools. [08:46] I love the smell of burning Soyuz in the morning [08:47] bigjools: It's been a pretty boring day, apart from the usual hardy linux cowboy. [08:47] And a b-m race. [08:48] Oooh, I missed the race [08:48] ? [08:49] Uploads stuck in UPLOADING, fixed by moving them back from failed to incoming. [08:49] But I believe that bug is fixed? [08:50] the race is not fixed [08:50] :( [08:50] *cough* jelmer [08:55] wgrant: I can't fathom why cron.daily-ppa takes so long [08:55] bigjools: It shouldn't. [08:55] The tree isn't that huge. [08:56] 2.5 hours .... [08:56] I guess we should make DiskPool take care of it. [08:57] and why does it need to share a lock [08:57] Because otherwise it'll crash the publisher. [08:57] how [08:57] It could remove a dir just after the publisher creates it. [08:57] Boom. [08:58] grar [08:58] Jar. [08:58] this Is Bad [08:58] Barelt. [08:58] DiskPool can do it easily. [08:58] Hmm. [08:58] But not easily all the way up. [08:58] I guess we could. [08:58] I want to know why this is so slow first [08:59] and, I am sick to death of PFOE [08:59] bigjools: Did you get anywhere with that logparser error? Did my theory about postgres 32-bit integer limits being exceeded hold any water? [08:59] wgrant: yes, that is it [08:59]