=== cprov is now known as cprov-afk === cprov-afk is now known as cprov === mrevell is now known as mrevell-lunch === salgado-afk is now known as salgado === mrevell-lunch is now known as mrevell === EdwinGrub is now known as EdwinGrubbs === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado [18:57] * Rinchen sobs for mootbot [18:58] be brave Rinchen [18:58] WTF is mootbot [18:58] It's been eaten by a grue. [18:58] kiko: It was Rinchen's horse. We shot it when it broke its leg. [18:59] it is no more [18:59] It is an ex bot [18:59] it sunk with seveas [18:59] It has ceased to be [18:59] it has shuffled of this mortal coil [18:59] there's a meeting coming up to talk about the future of all the bots [18:59] like ubotu [18:59] 'e 'as gone to meet 'is maker [18:59] Rinchen, really? [18:59] dinnaknothat [18:59] It is an ex-bot [19:00] well, mootbot is offline. Dennis is still around. :-) [19:00] just not on irc [19:00] sinzui: gmb beat you to it === EdwinGrub is now known as EdwinGrubbs [19:00] I could pretend to be MootBot [19:00] scribes team has had a few offers but they are not sure what to do because mootbot is eggdrop code. They were talking about building it into ubotu [19:00] damn [19:01] Let's move on to the cheese shop sketch then [19:01] (just without the semi-permanent-record part) [19:01] eggdrop makes swiss cheese look like high-density concrete [19:01] go! [19:01] Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development. [19:01] sinzui: I'll fetch my bourzouki, hang on. [19:01] Roll Call [19:01] me! [19:01] me [19:01] me [19:01] me [19:01] mÄ› [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] ,e [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:01] me [19:02] me [19:02] me [19:02] me [19:02] jt1 = jtv in disguise [19:02] Releases is here === jt1 is now known as jtv [19:02] foundations, bugs, etc? [19:02] Rinchen: no, that was an impostor. Here I am. [19:02] not a very good disguise it seems [19:02] lpcomm is here [19:03] me [19:03] ok there is soyuz [19:03] me [19:03] soyuz was here ages ago :) [19:03] allenap_, BjornT ? [19:03] Rinchen: foundations all there [19:03] and I see thumper...ok, let's go [19:03] Agenda [19:03] me [19:03] * Next meeting [19:03] * Actions from last meeting [19:03] * Oops report (Matsubara) [19:03] * Critical Bugs (Rinchen) [19:03] * Bug tags [19:03] * Operations report (mthaddon/herb) [19:03] * DBA report (stub) [19:03] * Sysadmin requests (Rinchen) [19:03] * New packages required (salgado) [19:04] * A top user-affecting issue (mrevell) [19:04] * Doc Team report (mrevell) [19:04] Next meeting [19:04] well, there was not much consensus on the rotation option email [19:04] Kiko, SteveA - you're still against two meetings, correct? [19:05] I'm not against anything per se, but we lose a lot by having separate meetings. [19:05] me, sorry I'm late [19:05] so I find the drawbacks kinda high [19:05] I'm against split meetings [19:05] that leaves us to pick one, or do thumper's 6 hour incrementing meeting [19:05] I'm fine with having two meetings, provided they're approximately one week apart [19:05] lol [19:05] incrementing meetings are a bit scary in that everybody gets confused [19:05] SteveA: :) [19:05] kiko: people are smart enough [19:05] and we have a team calendar [19:06] thumper, speaking from experience on the Ubuntu side, not really. [19:06] Also, we set the next meeting date in the meeting... [19:06] so I'm kinda -1 on rotating meetings [19:06] but +1 on two meeting times alternating [19:06] sounds like nobody liked the times there, though [19:06] could we propose other times? [19:06] I will be at both meetings [19:06] midnight UTC is good for me :) [19:07] worksforme [19:07] gah [19:07] I can go back and look for other slots [19:07] the 11am UTC one is the one with the most chance more highest attendance [19:07] anyway, if no decision comes in through email this week, same time next week. [19:07] what time is that for jtv? [19:07] ok, I'll set up everything for 18:00 UTC for next week and we can change it [19:07] moving on [19:08] kiko, 1am-ish [19:08] kiko: 07:00, or 6 hours before my working day [19:08] jtv is in thailand. surely he can get drugs to make the current meeting time palettable [19:08] * mpt doesn't know what he's talking about [19:08] SteveA: to me, yes. To you? Funny maybe, but... [19:08] Rinchen: if so then i'm afraid i'll have to miss next week. sorry! [19:08] right then, moving on [19:08] oh, thanks intellectronica [19:08] Actions from last meeting [19:08] none [19:08] Oops report (Matsubara) [19:09] Today's oops report is about bugs 228305, OOPS-855EB78, 228307. [19:09] Launchpad bug 228305 in malone "OOPS accessing contextless bug url" [High,New] https://launchpad.net/bugs/228305 [19:09] intellectronica, can you take 228305. I think it's related to the email interface. It needs further investigation. [19:09] cprov, wasn't OOPS-855EB78 RCFIXED? Or is that a new one? [19:09] jtv, 7pm? [19:09] matsubara: sure, i'll investigate and fix as appropriate [19:09] kiko: midnight UTC is 7am for me. [19:09] I've noticed some timeouts in +storeblob and +hwdb/+submit pages. I've asked about it to flacoste last Friday. More eyes on the issue would be appreciated(#228307) [19:09] jtv, it's 11am utc. [19:09] matsubara: bigjools fixed it last friday, IIRC [19:10] my hypothesis was some libarian connection dalys [19:10] kiko: 11am utc is just dandy for me. [19:10] cprov: the oops is from sunday, on edge [19:10] but that's just a wild guess [19:10] matsubara, so hmmm something else is amiss? [19:10] matsubara: uhmm ... let me check it again then [19:11] do we record times taken talking to the HTTP librarian? [19:11] if not, we should do [19:11] and these should be available in OOPS reports [19:11] as a general principle, any time we do something that talks to another process or another computer, we should record the time taken [19:11] matsubara: no, that's a new oops, needs a bug. [19:12] cprov: I'll file one after the meeting. thanks for checking [19:12] SteveA, for the upload? we do not. [19:12] SteveA: I don't think we do. [19:12] matsubara: np, thank you [19:12] SteveA: I mean, I don't think we do the logging [19:12] that's too bad. it would help us out in knowing what's going on. [19:13] librarian logging sucks also, but that's probably another issue [19:14] matsubara, anything else? [19:14] Rinchen: that's all from me. [19:14] thanks [19:14] Critical Bugs (Rinchen) [19:14] thanks all. I'll update the bug [19:14] Memory issue. flacoste, can we reopen the bug since it's still a problem, or create a new one? How is the current investigation going? [19:15] matsubara: would you file a bug on the librarian client that it should record times for HTTP calls [19:15] Rinchen: file a new one [19:15] SteveA: sure. doing it now [19:15] and SteveA thinks it might be related to batch size [19:15] thanks [19:15] i'm going to put a hard limit on it [19:15] 300 [19:15] (watch out for the exception handler that uploads the exception to the librarian - might end up in a loop) [19:15] Rinchen: earlier, tom, jtv and I did some experiments on staging [19:15] I am okay with that. would be even happier with 500 but anyway.. :) [19:15] i'm going to file a bug about the batch size issue [19:15] flacoste: I just discussed the issue with sinzui as well, so [19:15] thanks flacoste [19:16] flacoste: can we get together and compare notes? [19:16] https://bugs.edge.launchpad.net/rosetta/+bug/224617 [19:16] jtv - is this really critical or simply high? [19:16] Launchpad bug 224617 in rosetta "XPI import stumbles over malformed or email-less contributor entries." [Critical,In progress] [19:16] flacoste: (I have a bug for the Translations side) [19:16] and we could see a large-ish memory increase for translation pages with large batch sizes [19:16] Rinchen: that's critical to the Firefox people. [19:17] Rinchen: the fix landed today, and I'm negotiating a CP [19:17] jtv, k, thanks. [19:17] Rinchen, jtv: I'm going to chat with asac; meanwhile jtv will test on staging. [19:17] https://bugs.edge.launchpad.net/launchpad/+bug/224623 [19:17] DB load. stub, mthaddon - this hasn't been worked on. Therefore I submit that it's not a critical bug but rather high. Do you agree? [19:17] Rinchen: Error: This bug is private [19:17] :-) [19:17] agreed. [19:17] jtv: sure [19:17] jtv: grab me after meeting [19:17] flacoste: not now though; it's deep night and I've been at it for 15 hours straight [19:18] herb, if you could pass that on [19:18] will do [19:18] jtv: ok, i'll probably fix it later today though [19:18] I'll go ahead and lower this. If it happens again, herb if you could get Tom to update the details in the bug report please. thanks [19:18] I'll flag Bug 224623 as incomplete - there is no way to progress the bug report. [19:18] flacoste: okay, then we'll have to do it right after [19:18] Rinchen: got it. [19:18] Rinchen: we should adapt some code that gustavo wrote for landscape [19:18] stub: Bug 224623 on http://launchpad.net/bugs/224623 is private [19:18] ok, thanks stub [19:19] Rinchen: to limit the connections that are accepted from the network into the webapp [19:19] Rinchen: based on the size of the queue of connections waiting for app threads. [19:19] Sounds like a job for spec circumstances or foundations. [19:19] Rinchen: that will keep launchpad responsive even in situations that look like the one described by tom in the bug report [19:19] it's a foundations job, for post 2.0 [19:20] which we can look at moving forward if it happens again [19:20] ok, I'll add that to the laundry list. :-) [19:20] That's it from me. [19:20] Operations report (mthaddon/herb) [19:20] Highlights from the last week: [19:20] - 05/02: Production re-roll. [19:20] - 05/05: lpnet6 (running with the debug config) stopped responding and was restarted. [19:20] - 05/06: lpnet4 died and was restarted. [19:20] - 05/06: Cherry pick r6222 to lpnet* [19:20] - 05/07: Restarted librarian after a 13 minute outage. [19:20] New edge server running hardy should be up and in the rotation starting tomorrow. [19:20] No update from us on the memory issue. The debug instance (lpnet6) doesn't seem to be growing in the same way that the non-debug instances are. The typical lpnet process seems to grow to 800-1000MB RSS (3x-5x it's initial RSS) within a few hours and stay there idefinitely. [19:20] That's it from Tom and me unless there are any questions. [19:21] So we should fix the memory issue by running in debug mode? :-/ [19:21] herb: did you get anyone to look at lp6 when it stopped responding? [19:21] what's the overhead in running debug mode? [19:21] kiko: lots [19:21] kiko: we run single threaded [19:22] and it doesn't fix the memory issues [19:22] SteveA: Tom handled it. so I don't know. I know he copied the ref* files over in case there was something of value in there. [19:22] "Less damage per second" [19:22] it makes them less likely to occur, because the server handles fewer requests [19:22] maybe 800-1000Mb is what LP needs as working memory to process our requests that perform badly [19:22] 1/16th the number of requests. [19:22] herb: if an app server becomes unresponsive and it's only one, please take it out of rotation in the load balancer [19:22] and then find someone ifrastructural to look into it [19:23] if two go down in the same way, then restart the second one, leaving the first one hung [19:23] please add this to the LOSA operational manual, or whatever :-) [19:23] SteveA: ok. [19:23] we can usually afford to lose one app server [19:24] anything else for herb? [19:24] SteveA: when you say someone infrastructural, who should we be looking for? [19:24] and it's good to have the opportunity to diagnose it [19:24] a member of the foundations team [19:24] or SC [19:24] or me [19:24] SteveA: ok [19:24] herb, you can ping steve, kiko, or I on -code if it happens and we can help you find someone [19:24] Rinchen: sounds good. [19:24] thanks herb [19:24] off we go.... [19:24] DBA report (stub) [19:25] The production DB server is being upgraded to hardy as soon as IS can schedule it. This involves downtime. [19:25] Hopefully we get the PQM box running Hardy at around the same time. This means we can switch to PG 8.3 for development. [19:25] If things go to plan, we can upgrade production to PG 8.3 later with minimal downtime. I need to test using Slony-I to perform the migration with real data on Carbon once it has been Hardified. [19:25] Devs will need to switch to PG 8.3 when PQM switches. You are welcome to switch earlier if you want. The docs are already updated on the Wiki. [19:25] Had a good discussion on the person/auth split with jamesh and refactored the model again. The diagram on the wiki page has been updated (and no longer matches the text). [19:25] OOT. [19:26] Better watch out kiko, I'm going to hardify you [19:26] oot? [19:26] lol [19:26] over and out [19:26] out of time [19:26] ah [19:26] Rinchen, I'm on hardy!! [19:26] our obstinate technologist [19:26] jtv, put down that GTA list right now and step away from the computer [19:26] Rinchen: it's GTF [19:26] our other thailander [19:26] Thanks stub [19:26] Rinchen: and you're too late: it's already uploading [19:27] Sysadmin requests (Rinchen) [19:27] Is anyone blocked on an RT or have any that are becoming urgent? [19:27] our original Thailander? [19:27] and bigjools, I didn't look, are you done? [19:27] mrevell: nice [19:27] Rinchen: yes! \o/ [19:27] yippee [19:28] Rinchen: having said that I need to talk to flacoste about the restricted librarian rollout, which will need another RT [19:28] i suck [19:28] sorry, completely forgot that one [19:28] jesus guys [19:28] Pass it my way and I'll do the priority magic on it [19:28] this was due last CENTURY [19:29] I'll get SteveA to do a rain dance too [19:29] flacoste: are you free after the meeting sometime? [19:29] kiko, yeah, bigjools ppa fought the Hardy release and Hardy won [19:30] New packages required (salgado) [19:30] bigjools: tomorrow might be better, i have three persons already in line :-( [19:30] anything to add to launchpad-dependencies this week? [19:30] beer? [19:30] so... [19:30] not that I can think of [19:30] if launchpad-dependencies were managed in /sourcecode [19:30] thanks salgado! [19:30] then we could have a check on 'make run' [19:30] that warned if launchpad-dependencies is not of the appropriate version for this LP tree [19:31] SteveA: we can have a check even if it's not managed in sourcecode [19:31] sorry to bring up this old chestnut, but I miss xmas [19:31] flacoste: it means we need to record the version number somewhere, and keep it up to date [19:31] flacoste: rather than just work off what's in /sourcecode [19:32] SteveA, how about you file a bug for that and tell flacoste about it? [19:32] SteveA, well, depends if sourcecode is linked out of somewhere or not [19:32] i.e. if it is shared [19:32] Won't fix [19:32] we won't lose your chestnut then [19:33] flacoste, well, you can negotiate that with your boss ;-) [19:34] going once [19:34] we need more value out of it to offset the additional process cost [19:34] ok, sounds like an offline discussion is in order. ACTION: Steve and Francis to chat about dependencies and make check [19:35] thanks, but it's fine [19:35] A top user-affecting issue (mrevell) [19:35] howdy [19:35] I will go with flacoste's judgement on this issue. I'm sure we'll revisit it later, once we move more of launchpad code into launchpad. [19:35] A common theme over the past week - although we haven't been inundated with requests - has been requests to either report spam or edit existing bug comments. [19:35] As we've discussed this in meetings before, I'd be interested in hearing from other people who have either dealt with or seen an interesting user-affecting issue. [19:36] SteveA: actually, once servers are upgraded to hardy, the cost will lessen, so we should revisit at that time [19:36] one thing i encountered is users wanting to edit or delete their own comments [19:36] we need to have "add a 'report this comment' link on comments" and the same for other user-submitted content [19:36] for example, because they disclosed information they would rather not, accidentally [19:36] and that should be on the post-2.0 list [19:37] services like facebook do this well [19:37] intellectronica: Yes, I've seen that come up too. There's a problem of editing history there, though, isn't there? [19:37] Why is that a problem? [19:37] "other user-submitted content"? almost all of it is... [19:37] mrevell: what do you mean by "editing history"? [19:37] intellectronica, changing what I said in the past. [19:37] mrevell, not if it's within five minutes or before anyone else comments. [19:37] intellectronica: Someone could edit their comment to cast themselves in a better light, perhaps. [19:38] mpt: Hmm, fair point. [19:38] (5 minutes being when the mail notification goes out) [19:38] There is a general topic of spam handling that I've already added on behalf of the OSAs [19:38] Why is that a problem? We are writing a bug tracker and things - not a banking system. [19:38] on the post-2.0 list [19:38] Do we have a bug report? I didn't see one. [19:38] We have bug reports on both those issues [19:38] spam is bug 45419 [19:38] Launchpad bug 45419 in launchpad "Launchpad needs a way of easily flagging spam" [High,Confirmed] https://launchpad.net/bugs/45419 [19:38] I think we should always keep the comments [19:39] I don't remember the editing one offhand [19:39] mpt: Yeah, I saw the spam but not the edit history. I'll look again, thanks. [19:39] and hide them, and have a link saying "N comments removed" [19:39] openness is important [19:39] bug 80895 [19:39] Launchpad bug 80895 in malone "Give people five minutes to edit/delete their comment" [Undecided,Confirmed] https://launchpad.net/bugs/80895 [19:39] thanks mpt [19:39] np [19:40] As we have a plan, I'm done. [19:40] yep [19:40] Doc Team report (mrevell) [19:40] it's post 2.0 unfo [19:40] mpt: i'm not sure 5 minutes is enough. i realise that after that we've sent an email anyway, but i suspect that in some cases users will still want to remove a comment [19:40] SteveA: a db patch was landed last cycle, which will allow us to hide comment. we just need to implement it. [19:40] I'd like to give the doc team members a list of bite-sized items that would be outside the scope of our normal docs work. If you have ideas for such work, that involves your part of LP, please let me know. [19:40] On a docs-related note: Rinchen, Statik and I are recording a new episode of the Launchpad podcast next week. All ideas, such as suggestions of CC-licensed music for the theme and a name are welcome :) [19:40] intellectronica, then they can put themselves into the "Inappropriate" queue just like anyone marking spam [19:40] and be moderated accordingly [19:41] mrevell: a new episode? [19:41] mrevell: we have episodes? what's the RSS feed? I'll add it to my reader [19:41] mpt: maybe you're right. anyway, there's a bug, so let's continue there [19:41] sure [19:41] 4-minute warning... [19:41] SteveA: I think there was one too many adjectives in mrevell's sentence. [19:41] SteveA: Well, let's say episode 1.0, to the version 0 that I posted last year. The feed is... [19:42] http://news.launchpad.net/category/podcast/feed [19:42] thanks [19:42] Thanks Rinchen [19:42] Thanks. [19:42] Thank you all for attending this week's Launchpad Developer Meeting. [19:42] The end! [19:42] thanks Rinchen [19:43] thanks Rinchen ! [19:43] thanks Rinchen! [19:43] * Rinchen laments the loss of Mootbot. [19:43] mootbot, I mourns it [19:43] I'll just have to get the scissors out and cut up this log. [19:43] woo [19:43] matsubara, pass me the glue eh? [19:44] gmb, can I use your scissors? Yours seem awfully sharp. [19:44] did anybody say "glue"? [19:44] * intellectronica sniffles [19:44] Rinchen: Alas, I only use scalpels. [19:45] gmb, ok, I'll check with sinzui then. Last I knew his where pink [19:45] or maybe they were pinking shears.... === salgado is now known as salgado-brb === salgado-brb is now known as salgado === salgado is now known as salgado-brb === EdwinGrub is now known as EdwinGrubbs