[01:29] bryceh: just fyi, your blog has some trouble with openid ;) [01:29] anyway, thats a neat new feature :) [02:07] nigelb, it does? howso? [02:07] I didn't even know it could do openid [02:07] nigelb, although I did notice nobody wrote comments :-/ [02:11] bryceh: heh, I tried to login with LP [02:11] failed [02:11] you want me to grab the error? [02:13] yeah [02:13] I upgraded my web server to lucid, and drupal got upgraded, however it seems there is some bug that causes upgraded drupals to break in lucid [02:13] (which is why I haven't been blogging for a long time) [02:14] maybe that bug is why user registration doesn't work either [02:17] yeah [02:17] hold on, doing it again [02:19] is the drupal issue that it's using deprecated functions? [02:19] it probbaly is [02:19] lucid only has 5.3 [02:19] http://pastebin.com/r3fTKwmQ [02:20] ajmitch, no it looks like just a bug in the upgrade script, doesn't set fields to autoincrement [02:20] lucid does have a drupal6 package, fwiw, but it's even OT for here :) [02:21] yeah, autoincrement thing :) [02:21] nigelb, ok yeah I've reproduced that error [02:21] that explains the comments :) [02:22] why don't you move to something like wordpress for ease of use? [02:29] nigelb, why? [02:30] drupal is nice, but just "complicated" ;) [02:31] nigelb, I develop on launchpad and x.org, I shouldn't be put off by "complicated" ;-) [02:31] nigelb, is wordpress open source? [02:31] bryceh: hahahaaha [02:32] and yes, its open source [02:32] also complicated for visitors is what I meant :) [02:36] well, if I wanted something simple for visitors I'd just post to facebook [02:37] hahah [02:37] but that still needs signup [02:38] btw, you're still in Prague? [02:38] nope [02:38] got home last night [02:41] :) [02:41] someday I gotta start hacking on LP. [02:44] nigelb: JOIN US! [02:44] wgrant: I'm just worried [02:44] I use this system for work [02:44] nigelb, it might be interesting for the LP folks to know what is holding you back, so they can work to eliminate that as an issue [02:44] so, can't have my apache dying on me [02:45] nigelb: LP doesn't usually eat systems alive, and if you're really worried you can use a VM. [02:45] the instructions say that the settings used for working on LP means usual web development cannot happen [02:45] nigelb, do you not have another system you could use for development on? [02:45] rocketfuel-setup will try to remove PHP (because it installs apache2-mpm-worker), but that can be worked around. [02:46] launchpad-database-setup obliterates all PostgreSQL databases, but that can also be worked around if required. [02:46] bryceh: not yet. My own system is in the shop awaiting motherboard replacement [02:46] But apart from that, it coexists fine. [02:46] Just adds some vhosts to Apache. [02:46] * nigelb developes on PHP :( [02:47] VM seems like a good idea. [02:47] nigelb: that just means you have to tweak rocketfuel-setup to not install apache2-mpm-worker. [02:47] It's not actually required. [02:47] nigelb, well, like wgrant says, running in a VM works too. that's how brian does it [02:47] But yes, a VM works too. [02:47] I'll let a VM run rocketfuel overnight today :) [02:48] * nigelb gets sucked in LP [02:51] wgrant, maybe the newbie intro docs should just be written from the start to nudge the user towards setting up launchpad in a VM? it'd sidestep so many issues and worries [02:52] bryceh: Possibly. I'm not sure who manages those docs these days, though, since Karl disappeared. [02:53] disappeared? [02:53] there was someone who'd stepped up to that role I think [03:20] there is a volunteer, yes. [03:29] who? [03:31] it was announced to the list [03:31] heh this is a bad sign if no one knows [03:31] public I think. [03:32] I know, it just not paged in. [03:32] lajjr [03:32] also, I present, another victem of the heat. [03:35] nigelb, anyway I think you may be right I should just use wordpress. I had visions of using drupal for some more sophisticated website stuff, but these days I hardly have time to mess around with updating my website anyway [03:35] bryceh: heh :) [03:35] just be sure to update regularly [03:36] unfortunately that means losing all my old blog posts [03:36] nigelb, does something bad happen if you don't update regularly? [03:37] grrr jmkuhn - thanks mwhudson. [03:37] bryceh: vulnerabilities get reported often and get security updates [03:37] if you're lazy, you get owned :D [03:38] that sounds bad [03:38] bryceh: wordpress.com ++ [03:38] nigelb, is updating as simple as apt-get upgrade wordpress or is more to it? [03:38] there would be a zip file i guess which you would use [03:38] ick [03:38] I've never done it. I just use wordpress.com ;) [03:38] there is also a bzr branch you can just pull [03:39] yay, bzr++ [03:39] lifeless: that should be a snap then :) [03:40] lifeless: I just coped the planet.ubuntu.com idea of launchpad team + bzr branch of config for adding to blog syndication [03:40] lifeless: total win - should be a launchpad feature... [03:40] cool [03:40] hmm [03:40] interesting folk are here [03:40] here is a crazy idea I had. [03:41] what if every object was [crudely] available as a 1-item-long rss feed (using accept-encoding negotiation) [03:41] interesting [03:41] To what end? [03:41] pubsubhubbub [03:42] * mtaylor hates it that lifeless answer word is real and that I know what it is [03:42] we would pingthesemanticweb on a change to any object [excluding private in the first instance] [03:42] that is monitored by at least one hub [03:42] Hmm. [03:42] and you can get callbacks when the object has changed. [03:43] would make it easier to monitor things like lists-of-available-merge-requests [03:43] want to know when a build has finished on arm + i386? subscribe to the build [03:43] any thing that gets me callbacks will make me happy as far as those things go [03:44] bryceh: I stand corrected. There is a press-button upgrade feature. [03:45] nigelb, mm [04:16] mwhudson: Hmmm, so you're actually going to fix LP and modularise it (and the model classes)? [04:57] wgrant: well [04:57] wgrant: to some extent or other [05:06] rewrite from scratch. in php. [05:07] ewwww [05:07] * wgrant stabs spm repeatedly. [05:07] * mtaylor pokes spm until he stops [05:07] spm: that's the official word from what the LOSAs want then? [05:07] ajmitch: um.... [05:08] next he'll be saying that PHP 5 is too new, let's go for 4 [05:08] tho I will concede my troll has been matched, and the trolling stacks raised higher than I'm prepared to go. [05:08] Oooo! didn't think if that! ta! yes! php4 ftw. [05:10] * ajmitch shouldn't continue down this path [05:10] fuck that... cobol forms are much more powerful [05:11] spm: I just got a message that said that ~swift-bugs is subscribed to openstack - but I cannot for the life of me see _why_ that would be the case [05:12] mtaylor: hrm. looking [05:12] spm: also, I think we're clear to delete all of those -private things [05:12] mtaylor: ahh. sweet. I'll do that first. it's possible one is causing the other. [05:12] mmm. good point [05:13] openstack-private gone [05:14] woot [05:14] swift-bugs-private swift-core-private gone [05:15] gah. swift-private is being recalcitrant, per that answers request you have. been looking at that this morning to no joy [05:15] glad to know I haven't stopped causing you grief :) [05:16] actually.. thinking about that - I suspect it may have been self inflicted, by myself.one I need to chase. [05:17] mtaylor: I assume the ozone ones can go too? [05:18] spm: yes. all ozone can die [05:18] kk [05:31] Is there some DB load issue at the moment? [05:31] For the past 24-36 hours, my script that heavily uses getBuildRecords has timed out every time. [05:31] It previously only timed out one every few hours. [05:31] that's not cool... [05:32] 1 min load is higher than usual, but 5 and 15 is normal. light even. [05:34] wgrant: you wouldn't happen to know what table that references? [05:34] spm: BinaryPackageBuild, PackageBuild and BuildFarmJob, mainly. [05:36] is that talking to edge? I'd suggest not but... ? [05:37] It may be... let's see. [05:37] It moves around depending on where the API is least broken. [05:37] heh [05:37] It's currently using edge. [05:38] I'm seeing a few timeouts for queries with BinaryPackageBuild in them, all via edge [05:38] These should be 12-15 minutes past the hour. [05:39] ta! that helps hugely. [05:40] hrm. these are all almost uniformly on the hour; so possibly not you. [05:41] Hmm. [05:42] wine, firefox, kdm,? [05:42] ubiquity is another [05:42] My script requests everything with a build failure [05:42] nigelb, http://www2.bryceharrington.org:8080/wordpress/?p=3 [05:45] wgrant: are you getting an oops from these? the commonish query that somewhat matches is taking around 10secs based on a quick random grab. cop enough of those... [05:46] spm: I'm getting OOPSes, but this old launchpadlib doesn't display them. [05:46] Maybe I should upgrade that host to Lucid at some point. [05:46] bleh [05:48] Hm, wow, nearly a year since the open sourcing. === mtaylor is now known as mtaylor|beach [06:06] Argh. [06:06] https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177 is redispatching constantly. [06:06] Presumably because there is no longer a warty/i386 chroot. [06:06] Ah, and there are hoary and breezy builds too. [06:08] bryceh: w00t w00t! [08:15] wgrant: timeouts ? [08:16] lifeless: Yes. spm reminded me that you reduced the timeout last week. [08:17] there is a bug script hammering one bug on prod with trouble [08:17] the hourly edge graph is clean though [08:17] what issue are you having? [08:18] wgrant: ^ [08:18] lifeless: distroseries.getBuildRecords is frequently timing out for me. [08:18] got an oops id ? [08:19] This old launchpadlib doesn't display them, so no :/ [08:19] can you get one, please ? [08:19] OK. You can't easily find OOPSes for a particular method? [08:20] * wgrant turns httplib2 debugging on. [08:21] i wonder... if oopses record the remote IP, lifeless, you should be able to fgrep -R on devpad and hunt thru them all? brute force, but... [08:22] It's hopefully reproducible. [08:22] wgrant: no, not yet. [08:22] It normally happens within the first five minutes of the script, and it's running with debug logging on now... [08:22] I'll file a bug on finding oops by method [08:23] spm: seeing ip addressess is a bug [08:23] we need to fix that - and seeing private urls : both should be limited to losa and gsa [08:24] https://bugs.edge.launchpad.net/oops-tools/+bug/607087 [08:24] <_mup_> Bug #607087: enable 'search by method' [08:30] wgrant: any sign ? [08:31] lifeless: It seems to be an excellent Heisenbug. [08:31] arrrgh. [08:31] Just died now. [08:31] cool [08:31] But there's no X-Lazr-Oops header... [08:31] its not in the oops report either FWIW [08:33] wgrant: :( [08:33] what is there ? [08:33] pastebin ? [08:34] Hm. Maybe timeouts don't create an X-Lazr-Oops header :( [08:34] wgrant: this is what edge was last night - [08:34] === Top 10 Time Out Counts by Page ID === [08:34] Hard / Soft Page ID [08:34] 63 / 618 DistroSeries:EntryResource [08:34] 12 / 71 Person:+participation [08:34] 9 / 13 ScopedCollection:CollectionResource [08:34] 8 / 29 BugTask:+index [08:34] 5 / 50 Sprint:+temp-meeting-export [08:34] 5 / 6 Archive:+copy-packages [08:34] 4 / 2 DistroSeries:+index [08:34] 3 / 3 BugTask:+editstatus-page [08:34] 2 / 3 Milestone:+index [08:34] 2 / 1 Person:+map [08:34] wgrant: we do get timeout oops on edge apis [08:34] It's probably the first one. [08:34] OOPS-1660EB2408 [08:34] It would come under the first one, that is. [08:35] The request URL should have ws.op=getBuildRecords [08:35] !pastebin [08:35] QUERY_STRING: build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=200&ws.size=50 [08:35] StevenK: !sanity [08:35] That's the right kind of request. [08:35] GET /1.0/ubuntu/maverick?build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=1750&ws.size=50 [08:35] That's the one that failed just now. [08:36] lifeless: You should know better. :-P [08:36] I'd only expect 24 hard OOPSes a day from this, though. [08:36] StevenK: 12 lines in a totally dedicated channel is not a problem [08:36] StevenK: when there are no other discussions taking place. [08:36] StevenK: we are, after all, here to *communicate* [08:37] lifeless: I can has that OOPS? [08:37] lifeless: Ah, thanks for the review. [08:38] SQL time: 12978 ms [08:38] Non-sql time: 1499 ms [08:38] Aha. [08:38] What're the big queries? [08:38] Or are there lots? [08:39] http://pastebin.com/DsDmLqLk [08:39] one repeated query is the culprit I think [08:39] Forgive me -- I've no idea what those columns mean. [08:41] I'd really like to know where that query is coming from. [08:41] Hmm. [08:41] Maybe current_source_publication, I guess. [08:41] Although that query should be pretty quick. [08:43] lifeless: Do you have sufficient powers yet to land that branch? [08:47] wgrant: probably but EBUSY [08:47] wgrant: please grab someone else [08:47] lifeless: OK. [08:47] wgrant: oh, the 37 is repetitions [08:48] row reps time avg time-avg db-id query [08:48] there are also two very slow queries [08:49] 12 seconds total [08:49] http://pastebin.com/VsQpzqUq [08:49] wgrant: ^ [08:49] wgrant: thats actually the issue, probably. [08:49] WTF is it doing a count for. [08:49] the rest is inefficient but broadly noise. [08:50] Thanks. [08:50] Those are indeed the problems. [08:50] wgrant: I'd like to lower the timeout again today; is that ok with you, given you're about to propose a fix ? :) [08:50] lifeless: My other scripts are reasonably happy, so go ahead. [08:52] I fully support any measures to make LP less slow. === almaisan-away is now known as al-maisan [09:01] Hey up [09:01] Can I get an EXPLAIN ANALYZE executed, or is that a stub thing? [09:03] wgrant: any of the losas can do it for you. fire away. [09:04] spm: Can you EXPLAIN ANALYZE the second query on http://pastebin.com/VsQpzqUq, please? [09:09] wgrant: http://paste.ubuntu.com/465800/ [09:10] spm: Which DB was that? main master? [09:11] one of the slaves [09:11] Ah. Thanks. [09:41] spm: it was running on main [09:42] spm: you might want to check its the same? [09:42] actually nevermind [09:42] 7.7secs is 7.7secs [09:42] It's about a second slower, so it's pretty simialr. [09:43] heh, i'd be impressed if that made a difference, and no, looks near as identical. [09:43] 7.9 vs 7.7; within the bounds of load/server abuse [09:58] There are some recipe builds redispatching constantly, since they are targetted to series without chroots. [09:59] https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx [10:04] rockstar: can you fix that? [10:05] * bigjools sighs at people making builds for breezy, hoary and warty [10:05] the UI should have blocked that [10:05] It does. [10:05] I bet the API doesn't, though. [10:06] wgrant, why are they targetted to series without chroots? [10:06] But we should fail builds if the chroot's missing, I think. [10:06] rockstar: see the series I mentioned [10:06] rockstar: Because Ted thought it was fun, I guess. [10:06] bigjools, the UI does block that, so I guess it was an API thing? [10:06] validation in browser code is largely useless now we have the API :/ [10:07] yay for fat models [10:07] rockstar: yeah it must be [10:07] bigjools: Also, we have a whole lot of impossible builds being created in the primary archive when security updates with failed builds are copied. [10:07] bigjools, truthfully, that should probably take place in the model anyway. [10:07] wgrant, can you file a bug? [10:07] They get marked as failed immediately, but they clutter up build listings and probably shouldn't exist at all. [10:08] * rockstar does not have a shortage of recipe bugs to fix [10:08] heh [10:08] rockstar: One for preventing creation, and another for dealing with broken ones? [10:08] (since they can become broken after creation) [10:08] wgrant, I think the broken ones is probably a buildmaster issue, right? [10:09] rockstar: No. You pick the chroot in lp.code somewhere, and that'd be what's crashing. [10:09] Also, what's going on with porting SPRBs and TTBJs to the new model? [10:09] it needs to fail the build [10:10] in progress [10:10] Excellent. [10:10] the latter is pronounced as "Titty Bee Jay" BTW [10:11] Heh. [10:11] rockstar: so those builds that won't ever complete, you should poke some SQL at the LOSAs to remove them [10:11] and add a "cancel" button to the UI [10:11] raising the tone [10:11] There's a cancel branch in progress, right? [10:11] bigjools, there is a cancel button already for admins. [10:11] I thought I saw it floating around last week. [10:11] ummm [10:11] losa ping [10:12] rockstar: why only admins? [10:12] rockstar: hi [10:12] should be the user, and buildd-admins [10:12] bigjools, because that's what I wrote the code to do. [10:12] :) [10:12] What does it do when the build is in progress or finished? [10:12] wgrant, where are these builds? [10:12] wgrant, i thought might please you [10:12] rockstar: the three pending builds on https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx [10:13] poolie: Indeed, I was pleased to see that. [10:13] mthaddon, I have some builds that probably need to be cancelled. [10:13] rockstar: what does the cancel button do BTW? mark them with a new status or delete them? [10:14] mthaddon, https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/175 https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/176 and https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177 [10:14] * rockstar makes note to head punch ted [10:14] I'm somewhat concerned that Code is doing things that are impossible. [10:14] bigjools, I originally was deleting the build, but abentley thumped me, so I'm just changing the status now and deleting the buildqueue record if there is one. [10:14] Like deleting branches related to recipes related to builds. [10:14] rockstar: what is "wrong" with what ted has done here? [10:14] And cancelling builds that are in progress. [10:15] mthaddon, requesting builds against unsupported distros. [10:15] rockstar: great, sounds fine, with the caveat that wgrant is raising [10:15] rockstar: should he not be allowed to do that? if so is that a bug that we let him do that? [10:16] mthaddon: Yes, bug #607125 [10:16] <_mup_> Bug #607125: Forbid creation of recipe builds for obsolete series through the API [10:16] I see your caveat, and raise a "well, it's either that, or assume the buildmaster will fix it, which has been a bad assumption so far" [10:16] thx wgrant [10:16] * mthaddon adds to losa watched bugs [10:16] rockstar: Well, cancelling an in progress build will at the moment either not work, or not work and knock the builder out until somebody notices that it has been idle for weeks. [10:16] wgrant, there should be a cancel button on there now though. [10:17] wgrant, this is why we left it up to the admins. The reason it's here is because builds keep getting re-tried even though they are always failing. [10:17] we have some work to do to cancel in-progress builds [10:17] rockstar: those have been cancelled [10:17] rockstar: can you change that permission to buildd-admins please [10:17] mthaddon, great, thanks. [10:17] bigjools, you can... :) [10:18] rockstar: I would but we have all these buildd-manager bugs you keep exposing ;) [10:18] * rockstar refrains from "it's open source" comment. [10:18] * bigjools might do the same with the next buildd-manager bug [10:18] bigjools, this is a hell of a pissing match... "My code is more broken than yours!" "Nuh uh!" [10:19] lol [10:19] Ah, it already raises a CannotBuild when the chroot is missing. [10:19] But that just causes the dispatch to do nothing. [10:19] So it's going to be retried on the next run. [10:20] wgrant, ...unless we've cancelled the build. [10:20] rockstar: Right. [10:20] wgrant, "Cancel build" is there so that we don't have to make the losas run SQL. [10:20] I speak of the general case. [10:20] It shouldn't have to exist at all. [10:20] one of my b-m bugs is to detect failures and work out whether to fail the job or the builder [10:20] Luckily, spr is only available on edge. [10:21] * bigjools cringes at the 2nd meaning of SPR now :( [10:21] bigjools, maybe just see if the builder failed the last 5 builds before the builder commits seppuku. [10:21] rockstar: that's the exact plan [10:21] bigjools, my use of SPR is more important now. [10:21] it can see if it's the same builder or the same job [10:21] * rockstar only works on important things [10:21] bigjools: We also need to fire the reset trigger if a builder is being stupid. [10:21] haha [10:21] bigjools: eg. during an abort. [10:21] Or when we get a [10:22] I have a branch that does that, but I backed it out [10:22] :( [10:22] wgrant, also, recipe builds are about to be able to be re-scored as well. [10:22] wgrant: it disabled builders that were not resetting quick enough :/ [10:22] bigjools: Oh, right. [10:24] bigjools: Once we no longer need to preserve ddebs on the buildd filesystem, is there any need for the virt/non-virt distinction? I think the architecture restriction magic can handle it all. [10:25] well yes [10:25] It would also mean we wouldn't have to lie about ivy and kaylaberry. [10:26] we still need a split between non-virt and virtual i386/and64 [10:26] amd* [10:26] Why? [10:26] Once our queue discipline doesn't suck, there's no reason to. [10:26] bigjools, it'd be nice if we could return the spr builds to be "anything but arm" instead of i386 only. [10:26] archive restriction won't work there [10:26] bigjools: Why not? [10:26] rockstar: agreed [10:27] wgrant: because we don't want PPA builds ending up on distro builders [10:27] bigjools: But the distro builders don't need to be special. [10:27] yes, they do [10:27] Whyso? [10:27] they're not PPA builders [10:27] distro has its own hardware [10:28] Ah, so it is political. Yay. [10:28] besides, not resetting the builder shaves 30 seconds off the build time [10:30] Anyway, regarding the "anything but arm" thing: we could just leave them on i386 and let amd64 builders build i386 and lpia as well, as has been planned for years. [10:31] yeah there's a bug about that [10:32] it would need a bit of work server-side :) [10:32] Not really. [10:32] The slave side is done, but not merged. [10:32] And the master is simple enough. [10:37] Although it might make /builders a little odd. [10:45] mthaddon: hi [10:45] lifeless: hi [10:45] I has a new config CP on the wiki page [10:46] lifeless: you've added it to ones already done rather than requested ones [10:46] oh darn [10:46]