=== Moot2 is now known as MootBot === mrevell is now known as mrevell-lunch === salgado-afk is now known as salgado === kiko-afk is now known as kiko === mrevell-lunch is now known as mrevell === kiko is now known as kiko-fud === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado === salgado is now known as salgado-afk === mthaddon_ is now known as mthaddon === salgado-afk is now known as salgado [18:56] hello MootBot! [18:57] is mootbot back? [18:57] it is [18:58] it was working yesterday [18:58] so we shall see today [18:58] very cool [18:58] the re-write from eggdrop to ubottu hasn't happened yet but it planned [18:58] mwhudson_: I'm loving the new import system, I get email notifications as new revisions come in! [18:59] meee [18:59] yep, his ntp server is fixed [19:00] * Rinchen crosses his fingers [19:00] #startmeeting [19:00] Meeting started at 13:01. The chair is Rinchen. [19:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [19:00] woot! [19:00] [TOPIC] Roll Call [19:00] New Topic: Roll Call [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] mrevell ? [19:00] me [19:00] me [19:00] me [19:00] me [19:00] thumper? [19:01] flacoste, leonardr? [19:01] meeee [19:01] me === vednis is now known as mars [19:01] me [19:01] me [19:01] me again [19:01] me === kiko-fud is now known as kiko [19:01] Rinchen: foundations is complete [19:01] I'm the most me a me can be [19:01] thanks flacoste, releases is here too [19:01] allenap? [19:02] me [19:02] Rinchen: lpcomm is all here [19:02] me [19:02] great thanks [19:02] abel is missing. [19:02] no, he's not [19:02] no, he's adeuring, i think [19:02] no, I'm here [19:02] me [19:02] oh yeah, sorry :) [19:02] bigjools? [19:02] ah gotcha [19:02] thanks [19:02] ok. [19:02] [TOPIC] Agenda [19:02] New Topic: Agenda [19:03] me [19:03] * Next meeting [19:03] * Actions from last meeting [19:03] * Oops report (Matsubara) [19:03] * Critical Bugs (Rinchen) [19:03] * Bug tags [19:03] * Operations report (mthaddon/herb) [19:03] * DBA report (stub) [19:03] * Sysadmin requests (Rinchen) [19:03] * New packages required (salgado) [19:03] * A top user-affecting issue (mrevell) [19:03] me [19:03] * Doc Team report (mrevell) [19:03] [TOPIC] Next meeting [19:03] New Topic: Next meeting [19:03] btw apologies received from SteveA [19:03] same time same day next week? [19:04] anyone know they will not be able to attend? [19:04] I won't be here for the meeting next week [19:04] [AGREED] same time next week. Apologies from statik [19:04] AGREED received: same time next week. Apologies from statik [19:04] [TOPIC] Actions from last meeting [19:04] New Topic: Actions from last meeting [19:05] one, and it was completed [19:05] [TOPIC] Oops report (Matsubara) [19:05] New Topic: Oops report (Matsubara) [19:05] Today's oops report is about bugs 236425, 228307 [19:05] matsubara: Bug 236425 on http://launchpad.net/bugs/236425 is private [19:05] sinzui, how's #236425 fix coming? [19:05] schwuk, any news about 228307? [19:05] matsubara: it is in PQM [19:06] flacost, I've seen a couple of CyclicalTeamMembershipError. isn't that one fixed? [19:06] matsubara: not yet [19:06] cool. thanks sinzui [19:06] matsubara: about 18 hours form landing :( [19:06] matsubara: but the hwdb oops are continuing [19:07] it must be coming from New Zealand [19:07] matsubara, CyclicalTeamMembershipError? where did you see them? [19:07] schwuk: yeah, I wonder if the UploadFailed errors are related [19:07] salgado: https://devpad.canonical.com/~matsubara/oops.cgi/2008-05-31/F1254 [19:07] oh, when reassigning? [19:07] matsubara: most likely [19:08] schwuk: hmm I'll add those errors to the timeout report then. thanks [19:08] matsubara, hadn't seen that one, but I was expecting them to show up [19:08] salgado: +editproposedmember, I'll follow up with you after the meeting then [19:09] Rinchen: I'm done. Thanks [19:09] ok [19:09] thanks everyone [19:09] thanks matsubara [19:09] [TOPIC] Critical Bugs (Rinchen) [19:09] New Topic: Critical Bugs (Rinchen) [19:09] [LINK] https://bugs.edge.launchpad.net/launchpad/+bug/237086 [19:09] Stale PID - mars, how are you coming with this? [19:09] LINK received: https://bugs.edge.launchpad.net/launchpad/+bug/237086 [19:09] Rinchen: Error: This bug is private [19:09] MootBot: Error: This bug is private [19:09] Rinchen, the fix should be running right now [19:09] excellente [19:09] thanks [19:09] [TOPIC] Bug tags [19:09] New Topic: Bug tags [19:10] we have no new requests [19:10] [TOPIC] Operations report (mthaddon/herb) [19:10] New Topic: Operations report (mthaddon/herb) [19:10] Friday (2008-05-30) we did the re-roll. Since then, we've had one CP for r6407 on Monday (2008-06-02). We have one CP, r6410, waiting for approval. [19:10] A branch with configs for the restricted librarian is in the queue at the moment. The directory has been set up on the server. We're just waiting for the branch to land. [19:10] We've started work on the Debian import machine. [19:10] We had an issue occur Tuesday (2008-06-02). A spike in load on the production DB server caused performance problems with LP. The site became so slow it was virtually unresponsive. Seems similar to the bug filed back in April: Bug #224623. [19:10] herb: Bug 224623 on http://launchpad.net/bugs/224623 is private [19:10] [LINK] http://launchpad.net/bugs/224623 [19:10] LINK received: http://launchpad.net/bugs/224623 [19:10] Rinchen: Error: This bug is private [19:11] MootBot: Error: This bug is private [19:11] that's it from Tom and me unless there are questions. [19:12] herb, was there any more data from Tuesday that can be added to the bug report? [19:12] herb, without more definition I don't think stub can narrow it down [19:12] Rinchen: there are the debug logs that we typically get. I can throw them in there. [19:13] herb, ok, please do. stub can evaluate if it need further detail after he views them [19:13] I'm happy to try to collect more info if the problem occurs again. Just would need some direction as to what would be helpful. [19:13] btw, herb and mthaddon, I really like the format you use for your report. [19:13] stub, any words of wisdom here? [19:13] No - I haven't found where the debug logs are yet. [19:14] ok. thanks [19:14] herb: thanks for start working on the gina-machine. [19:14] stub: I'll attach it to the bug above. [19:14] stub, on hackberry in ~postgres/logs [19:14] Any questions for herb and mthaddon? [19:14] cprov: thank mthaddon, he's been heading that up. [19:14] thanks herb [19:14] mthaddon: thanks ;) [19:14] [TOPIC] DBA report (stub) [19:14] New Topic: DBA report (stub) [19:14] what about the restricted librarian ? [19:14] cprov, see above [19:15] mthaddon: duh, sorry, I missed that. [19:15] :) [19:15] https://bugs.edge.launchpad.net/ubuntu/+source/slony1/+bug/236749 is blocking me testing a low downtime PG 8.3 upgrade procedure. If we can't get a package built, we either need to do a manual install (with the crappy workarounds for hardy mentioned in the bug report) or take the downtime hit. [19:15] Both Abel and Danilo are having similar Unicode issues at the moment, where the DB is running in the C locale so lower() doesn't always give correct results. [19:15] Launchpad bug 236749 in slony1 "No PostgreSQL 8.2 (or earlier) packages" [Undecided,New] [19:15] [LINK] https://bugs.edge.launchpad.net/ubuntu/+source/slony1/+bug/236749 [19:15] LINK received: https://bugs.edge.launchpad.net/ubuntu/+source/slony1/+bug/236749 [19:16] I need to investigate if there are still functional index bugs in PostgreSQL that will stop us using them as the solution to their problems. [19:16] stub, why don't we use slony on 8.3? [19:16] Because that doesn't help us migrating from 8.2 -> 8.3. [19:17] stub, how much time is the upgrade going to take if we do it cold turkey? [19:17] PG 8.2 needs stored procedures built and installed for it, and so does PG 8.3 to do the migration between versions [19:17] 3 or 4 hours [19:17] But we will have the same issue next major release too probably, and by then we will have high availability systems. [19:18] stub, sure, but we /will/ have slony for 8.3 right? [19:19] I suppose. [19:19] stub, I will talk to pitti and doko and see if either of them can help. if not, we should just take the downtime. [19:19] it's no use waiting for too long on this. [19:19] I'm thinking for the 8.3->8.4 migration. We need slony for two versions of PG installed to do the migration. [19:19] sure [19:19] once you have 8.3 running, you can install slony for it [19:20] Yes - it isn't blocking the migration. Just blocking being able to do it without much downtime. [19:20] stub, if we do decide to upgrade postgresql without slony, why don't we do it at the same time mthaddon and herb schedule the upgrade of our systems to hardy? [19:20] We already have PG 8.3 with slony installed. [19:20] cool [19:20] kiko, that's a lot of moving parts, but a possibility... [19:20] mthaddon, herb: any news on the hardy updates, btw? that just rang a bell! [19:20] That timing would be fine. The main DB server is already running hardy. [19:20] kiko, we're working through the low impact ones, and once we're ready to move ahead with the others will get a schedule together [19:21] mthaddon, well, yes, but moving parts that are pretty distinct -- i.e. no LP code will change, just the DB and the frontends [19:21] true [19:21] Just more to debug if it all falls in a heap ;) [19:21] that is true [19:21] But that would never happen [19:21] anything else for stub? [19:21] mthaddon, so, any ETA at all for that downtime? [19:21] stub, yes! [19:21] person-auth-split, for 1.2.6, yes/no? [19:22] kiko, not yet - I'll try and get you something by the end of the week (a time scheduled, I mean) [19:22] gone through review and ready to land. yes. [19:22] stub, I'm asking you because I think you're not aware of our likely post-1.2.6-rollout process [19:22] neither is anyone else outside of people on yesterday's call of course [19:22] Its only the first section though - the whole thing has huge scope. But the data model will all be in place. [19:22] but we're unlikely to do any DB patches for 1.2.7 [19:22] that's cool and what I was hoping for [19:22] thanks stub [19:23] anything else for stub? [19:23] mthaddon, thanks. it needs to be within the next 3 weekends I guess [19:23] popular guy today [19:23] mthaddon, or then only in august [19:23] kiko, yep [19:23] nah [19:23] toda [19:23] [TOPIC] Sysadmin requests (Rinchen) [19:23] Is anyone blocked on an RT or have any that are becoming urgent? [19:23] New Topic: Sysadmin requests (Rinchen) [19:24] Rinchen, I reminded RT of one to remove some old bazaar repos still public [19:24] it's an old one poolie asked for [19:24] thumper, will we need any RTs for upgrading bzr? [19:25] kiko: I think it is just a normal rollout for LP, or are you referring to other machines? [19:25] I saw the thread. If you get no reply to it soon I'll go make myself more visible to IS [19:25] both soyuz-related requests are in-progress (restricted-librarian and gina), thank you. [19:25] we have 1 RT for the bzr-beta already [19:25] thumper, both [19:26] kiko: as Rinchen says, there is one for bzr-beta already, and I don't think we'll need one for LP [19:26] okay, thanks. [19:26] nifty. Anything else for this topic? [19:26] [TOPIC] New packages required (salgado) [19:26] New Topic: New packages required (salgado) [19:26] anything to add this week? [19:27] salgado, maybe something to remove. do we currently depend on bzrtools? [19:27] isn't shelf in bzrtools? [19:27] kiko, thumper submitted an rt to remove that [19:27] iirc [19:27] I don't think we do depend on it [19:27] but many have it installed [19:27] there's no launchpad-d* packages which depend on it, btw [19:27] (remove from source that is) [19:28] I have launchpad-developer-dependencies but not bzrtools [19:28] salgado, yes, it can be part of -developer-dependencies, but not of -dependencies, because it has a bug in the shelf test suite [19:28] PG8.3 has been done now, which is good. [19:28] okay, cool [19:28] thumper, is there a bug filed for the bzrtools bug, btw? would be nice to get that fixed for intrepid.. [19:28] kiko: abentley is aware, but not sure if there is a bug filed [19:29] anything else for salgado? [19:29] kiko: A fix has already been committed to trunk. [19:30] abentley, like the rest of the bzr team, fixes things almost before we notice them! [19:30] heh [19:30] Rinchen, no, thanks [19:30] ok, I think it's time for us to kick off the Matt Revell show....staring... mrevell [19:30] [TOPIC] A top user-affecting issue (mrevell) [19:30] New Topic: A top user-affecting issue (mrevell) [19:30] heh :) [19:30] On Sunday, there was an issue with Bazaar branch hosting for branches using the knit format. [19:30] This led to a discussion about the best way to communicate service-affecting issues to people who use and rely on Launchpad. [19:30] The outcome is that we have set up a launchpad-announce list. This will be very low volume - service-affecting issues and release announcements only. All mail to the list is held for moderation and only appropriate messages will get through. [19:31] I shall be announcing the, err, announcements list shortly. [19:31] launchpad-meta-announce? [19:31] If you have something that must be announced to our users please ping me or, in my absence, RInchen [19:31] mrevell, we should make it so /everybody technical/ at canonical subscribes [19:31] mrevell, I'll talk to mdz about this [19:31] kiko: Okay, thanks. [19:32] a bit taboo but we could subscribe the distro-team list :-) [19:32] Rinchen: You'd wanna get some flame-retardent gear on first :) [19:32] eww [19:32] Rinchen: Unless anybody has a comment on that, or the Bazaar hosting outage itself, that's that for me. [19:32] thanks mrevell [19:32] [TOPIC] Doc Team report (mrevell) [19:32] New Topic: Doc Team report (mrevell) [19:32] And the limelight calls again [19:32] I'm pleased to say that community contributions to the doc team and growing slowly in number. We have a couple of community members who are now actively filing bugs for things they find in the help wiki and who are diving right in and making changes themselves. [19:33] s/and growing/are growing [19:33] Other docs news: tour is progressing and I have another meeting with Mark and the designers next week. I have mailed team leads to ask for confirmation of the items I've selected for 1.2.6 what's new and release announcement. Thanks to those who've already responded. [19:33] Thanks Rinchen [19:33] awesome [19:33] woo! [19:33] good stuff, thanks mrevell [19:33] When is the tour due to land? [19:34] mpt: it might be easier to answer that after my meeting with SEM and Mark on Wednesday [19:34] ok [19:34] we have a recent addition to the agenda.... [19:34] [TOPIC] Storm Integration [19:34] New Topic: Storm Integration [19:34] kiko! [19:34] heh [19:35] first, I want to thank BjornT for knocking off 100 minutes from PQM test run times [19:35] (otherwise known as "the gathering storm") [19:35] (nothing to do with storm of course) [19:35] * Rinchen cheers. [19:35] err [19:35] extra zero there, and then some [19:35] 100? [19:35] but YKWIM [19:35] a lot [19:35] https://devpad.canonical.com/~mthaddon/pqm-durations.png [19:35] etc [19:35] Hurray for BjornT! [19:35] Yeah, it now takes PQM -20 minutes to run. [19:36] nice drop off :) [19:36] thumper, funny you speak up, because I wanted to ask you if you are getting rid of buildbot anytime soon -- it currently is tested against launchpad because it adds lib/ to its path [19:36] kiko: yes we are [19:36] thumper, alternatively, do you know if we can avoid running buildbot tests when LP lands? [19:36] BjornT: what happened, you turned off all bugs tests? [19:37] oh [19:37] the sourcecode dependancy... [19:37] that took 20mins, amazing... [19:37] 10 mins... [19:37] my joke missed the audience [19:38] Rinchen: I got it [19:38] thumper, well, ok. when you remove that we'll knock a few minutes off, so that's great to hear. [19:38] flacoste didn't :) [19:38] okay [19:38] now for the real storm topic [19:38] thumper, flacoste, jtv, statik: did you guys ack that list I sent out? [19:38] kiko: seems sane to me [19:38] kiko: didn't ack it yet, will do when I wake up [19:38] already knocked quite a few off [19:38] I haven't seen any email back on that topic so I want to make sure something is happening [19:38] thumper, you are the man [19:38] There where some pillarname ones I would have thought I would have, but if statik wants them he is welcome to them :) [19:39] kiko: yep, edwin started looking at it already. funny, i get a lot more tests failing locally than in the report, but we're diggin ito it [19:39] there is a test dependancy issue thought [19:39] jtv, please have a call with jamesh when you get up to sort yours out, I'm unhappy you didn't start on that today [19:39] stub: we might send you mail later :) [19:39] many run fine by themselves [19:39] statik, very cool, thanks so much [19:39] but fail when run with everything [19:39] something is stomping on the db configs [19:39] thumper, yeah, need to isolate which one is fscking us over! [19:39] * thumper looks at sinzui for some strange reason [19:39] hey, sinzui isn't responsible for all the config problems [19:39] just the API [19:39] thumper: et al. mutating the config is bad for testing [19:40] thumper, the problem is tests that write to the configs don't reset them [19:40] yeah, I know [19:40] We have an API for config problems? [19:40] heh [19:40] we have APIs for everything [19:40] okay [19:40] jtv, ack? [19:40] I think we need to switch to the push() and pop() form of the config and raise an error when someone tries to create a new attribute on the conifg. [19:40] kiko: sure [19:40] pickle the config in the layer testSetUp, pickle it in the layer testTearDown and compare. [19:40] thanks vm [19:41] sinzui: i think that's a little drastic change for our schedule [19:41] i say only plug the hole, not redo the whole plumbing [19:41] * sinzui checks the weekend schedule. [19:41] lol [19:41] what flacoste says, sinzui -- don't get carried away!! :) [19:41] okay, that was all [19:41] Rinchen, take over before it's too late! [19:41] Thank you all for attending this week's Launchpad Developer Meeting. [19:42] sinzui: did they open that crystal meth lab in you 'hood lately? [19:42] #endmeeting [19:42] Meeting finished at 13:43. [19:42] thanks Rinchen [19:42] the crystal [19:42] thanks Rinchen [19:42] tx Rinchen [19:42] hehe [19:42] flacoste: It went up in smoke when the try knocked out the power lines on my street yeterday [19:42] Rinchen: thank for another one on time [19:42] now i Just need to find out where mootbot put the logs :-) [19:42] I am relying on an infant to keep me awake [19:43] sinzui, that's probably a little healthier === mars is now known as vednis === vednis is now known as mars [21:32] statik: oh good :) === kiko is now known as kiko-afk === thumper-office is now known as thumper === salgado is now known as salgado-afk === mwhudson_ is now known as mwhudson