[00:02] i predict this new will make jml unhappy [00:03] Which new? [00:03] The restructure? [00:07] spm it turns out, just fyi, that qastaging has no mx record [00:07] i will send a mail to the list [00:07] ah. k [00:25] flacoste: Thanks for forwarding that to the list. [00:30] wgrant: hey [00:31] wgrant: so, I don't know if you noticed, but the archive:+index timeouts are going up daily. I don't think this is random. [00:32] lifeless: That's odd. [00:32] (sorry, I've been distracted with other stuff the last couple of days) [00:32] doesn't deeply worry me [00:32] its a little concerning, and I want to be worry free on my holiday [00:33] 21st: 387 / 6695 Archive:+index [00:33] Hmmm. [00:33] bah [00:33] 1st: 387 / 6695 Archive:+index [00:33] 30th: 324 / 7217 Archive:+index [00:33] 29th: [00:34] 287 / 6525 Archive:+index [00:34] The other pageids aren't showing a similar trend? [00:34] That's a pretty massive escalation. [00:34] 29th: [00:34] 287 / 6525 Archive:+index [00:34] 170 / 390 BugTask:+index [00:34] 43 / 260 Distribution:+bugs [00:34] If it's the only pageid doing it, that's more 'deeply concerning' than 'a little concerning'. [00:34] 28th: [00:34] 260 / 6568 Archive:+index [00:34] 142 / 301 BugTask:+index [00:34] 26 / 290 Distribution:+bugs [00:34] 27th: [00:35] 115 / 5419 Archive:+index [00:35] 92 / 233 BugTask:+index [00:35] 23 / 244 Distribution:+bugs [00:35] Uh. [00:35] I think we have a general problem :) [00:35] possibly [00:35] 30th: [00:35] 324 / 7217 Archive:+index [00:35] 148 / 451 BugTask:+index [00:35] 36 / 150 ProjectGroupSet:CollectionResource:#project_groups [00:36] 19 / 296 Distribution:+bugs [00:36] 1st: [00:36] 387 / 6695 Archive:+index [00:36] 155 / 482 BugTask:+index [00:36] 126 / 226 Question:+index [00:36] 49 / 104 RootObject:+login [00:36] 39 / 247 Distribution:+bugs [00:36] These are all genuinely problematic pages [00:36] so I'm not inclined to shout fire yet [00:36] rather fix the pages. [00:37] and the worst.. well you know the worst. [00:37] it is, on its own, 25% of the timeouts on the site. [00:38] besides which [00:38] I know flacoste will be looking at these reports and biting his teeth in my absence ;) [00:38] Heh. [00:39] Ah, I remember why I didn't get to it last time. download-cache took forever to upgrade. [00:39] It's done now, so the appserver might finally start... [00:39] pillarname || 963021.70 tuples/sec [00:39] branch || 443538.40 tuples/sec [00:39] WTF? [00:39] archive || 41660.33 tuples/sec [00:40] 28th [00:40] pillarname || 825961.90 tuples/sec [00:40] branch || 463744.27 tuples/sec [00:40] sl_confirm || 133113.62 tuples/sec [00:40] binarypackagepublishinghistory || 49345.86 tuples/sec [00:40] archive || 33620.98 tuples/sec [00:40] shippingrequest || 24934.80 tuples/sec [00:40] 1st: [00:40] pillarname || 827065.04 tuples/sec [00:40] branch || 469628.31 tuples/sec [00:40] sl_confirm || 130679.92 tuples/sec [00:40] binarypackagepublishinghistory || 53159.91 tuples/sec [00:40] archive || 40228.91 tuples/sec [00:40] sourcepackagename || 24256.23 tuples/sec [00:41] not hugely different on a day to day basis [00:41] it may be that traffic spike that I called out on the list causing trouble [00:41] 'the list' being the internal one, or have I missed something? [00:48] wgrant: uhm [00:48] checking [00:54] lifeless: Aw, is that really my first? :( [00:54] wgrant: no :) [01:09] something must be sequentially scanning branch to get numbers like that [01:09] i'd love to know what [01:10] well not 'must' i guess [01:10] seems very likely though [01:13] And Julian doesn't think I'm crazy :D [01:34] poolie: Any progress with QA for r12008? [01:34] abentley: hi, is that the dkim and gpg bugfixes? [01:34] poolie: yes. [01:34] i presume yes, as that's the only one i landed [01:35] it turns out qastaging doesn't receive mail [01:35] it doesn't even have an mx [01:35] that's where i'm up to [01:35] my next thing is to send mail about perhaps turning it on [01:35] it may take too long to do it [01:35] i mean, it may take a while to turn it on and get it working, and that could cause rollouts to stall for too long [01:36] poolie: does your patch make anything worse, do you think? [01:36] perhaps there's a better approach? [01:36] poolie: normally, I'd suggest using staging instead, then. But unfortunately, we are doing a full db restore in order to ensure we don't run overtime. [01:36] poolie: and if so, can it be disabled easily? [01:36] i thought that qastaging would be like staging in this regard (ie reads mail, doesn't send it) [01:36] abentley: Will my subunit db patch be in the rollout, do you think? [01:36] but it's not [01:36] lifeless: it changes existing code [01:36] lifeless: Yes, that's the plan. [01:36] so, it's possible it will cause a regression [01:36] abentley: \o/ [01:36] and it's not isolated by a flag [01:37] poolie: I think qastaging is still being configured. [01:38] so if it does turn out to break anything that was previously working, i think we would have to roll back or reverse-cherrypick or i guess live with it until a new fix is landed :/ [01:38] poolie: It's not on staging yet either? [01:38] poolie: db deploy - we can't roll those back. [01:38] poolie: like it doesn't have import slaves or a build farm at the moment. [01:39] wgrant: It would be, but staging is doing a full restore ATM. [01:39] abentley: Yay :( [01:40] i'll send mail first at least [01:41] should we/do we have a bug asking for real-time log feeds? [01:42] poolie: do you mind testing it in ~24h? [01:42] do you mean <24h or >24h? [01:43] i mean, do you mean "will you please test it on saturday au/sydney?" [01:43] poolie: that is what I mean. If that's okay with you, it's probably the simplest solution. [01:44] and you mean test it on staging? [01:44] poolie: yes. [01:44] that's ok with me [01:44] poolie: Thanks. [01:45] is there any more specific estimate when it'll be live there? [01:45] just trying to work out if i can do it first thing saturday or something (like 20h from now) [01:47] Chex: Do you have an estimate when the staging update will complete? [01:50] abentley: i might also work out/document/improve testing it locally [01:50] manually testing it locally, i mean [01:50] but i think checking it in the dc would be really good too [01:51] there are enough different moving parts [01:52] poolie: Yes. We don't normally consider it QAed until it's tested in the datacentre. And I know our mail handling can be a bit odd, so I think testing it on staging is really worthwhile. [01:53] me too [01:53] thanks for following up on it [01:53] poolie: np [01:56] wgrant: so, please look at that branch :) [01:56] ciao [02:24] hey, can someone help me work out how to land max's loggerhead changes [02:24] is it enough to just merge them to the relevant branch? [02:24] spm? wallyworld? [02:54] poolie: I'm not sure at all tbh [03:04] poolie: Don't you just PQM them onto lp:~launchpad-pqm/loggerhead/devel, and then update utilities/sourcedeps.conf? [03:05] i think so; i'll try it later [03:15] are one-letter project names banned? [03:17] Hmm, are the appservers overloaded? [03:17] I'm getting truncated responses again. [03:17] Less frequently than last time, though. [03:21] a couple are, yeah [03:23] https://lpstats.canonical.com/graphs/AppServerRequestLpnet/ wheeee [03:47] wgrant: with any luck, things should be better. looks like a buggy script was being rude. [03:54] spm: Thanks. [03:55] yeah the # requests graph is dropping off due to our heavy hand of stabbing application [04:24] wgrant, Do you know much about SQL indexes? [04:24] cody-somerville: Some. What's the issue? [04:26] wgrant, I'm wondering if an index could be used to make finding the latest of something that relates to something else faster - ie. lets say finding the latest build for a project in a CI. [04:27] cody-somerville: Of course. [04:28] Would just creating an index on say the finished_at field of the buildresult table be sufficient? [04:28] or would an index on the foreign key to the project and the finished_at field be better? [04:29] Probably the latter. [04:30] be aware that adding indexes can have major negative performance impacts. if you're looking to modify LP stuff, I'd strongly urge you to check with stub, and at least get some explain's happening to show if an index *may* help. [06:00] launchpad.dev seems to be just about unusable for me. [06:00] It has to grab several hundred YUI files on every page load... [06:02] O_O [06:02] Yeah, more than 400 YUI JS files are included in each page's . This seems questionable. [06:04] wgrant: Whereas, usually, launchpad.dev is *fast* [06:04] StevenK: I think it would be almost acceptably fast if the JS was all in one file like it is on prod. [06:04] I thought the Makefile did combine-css? [06:04] The main issue is that only a single request can be processed at a time. [06:04] Oh, wait [06:04] I'm a bozo [06:05] Doesn't lazr-js do that for us? [06:05] The dev appserver doesn't use the combined launchpad.js, even if it has been built. [06:06] I wonder which config key it is. [06:06] Ah, devmode. [06:07] OMG so fast [06:11] There's also a disturbing number of 404 OOPSes that have just shown up in the last few minutes. [06:11] Like 3000 of them. [06:12] I think something might be wrong with the lazr-js upgrade :/ [06:17] StevenK: Could you pastebin the query log from OOPS-1780G1265 for me? [06:17] No, you can bloody well wait 10 days [06:17] Heh. [06:17] But IS probably won't get around to creating my account for years :P [06:18] I doubt it [06:18] wgrant: http://pastebin.ubuntu.com/539296/ [06:18] Oh sigh, that's the wrong bit, isn't it [06:19] wgrant: http://pastebin.ubuntu.com/539297/ [06:21] StevenK: Thanks. === almaisan-away is now known as al-maisan [07:03] === Top 10 Time Out Counts by Page ID === [07:03] Hard / Soft Page ID [07:03] 80 / 5210 Archive:+index [07:03] 62 / 271 BugTask:+index [07:03] 21 / 248 Distribution:+bugs [07:03] 18 / 0 BinaryPackageBuild:+retry [07:03] lifeless: Yes, I'm working on it right now :P [07:03] 15 / 3 ProjectGroup:+milestones [07:03] 12 / 276 Distribution:+bugtarget-portlet-bugfilters-stats [07:03] But the factory is being shitty. [07:03] 12 / 19 Archive:+copy-packages [07:03] 10 / 123 ProjectGroupSet:CollectionResource:#project_groups [07:03] 9 / 146 POFile:+translate [07:03] 8 / 2 Person:+bugs [07:03] *interesting* [07:03] The 300 or so missing ones? [07:03] cool [07:13] wgrant: factory fixin time? [07:14] lifeless: I think it's OK now. [07:15] I was confused for a while after I fixed it, since the baseline assertion had started failing without me noticing. [07:15] But I think it's sorted now. [07:31] kk [08:48] good morning [09:20] Morning Launchpad devvers [10:00] So, StevenK... about that failure in db-devel. Is anyone looking at why it happened? === al-maisan is now known as almaisan-away === jkakar_ is now known as jkakar [11:01] jml: Given the nature of the current db-devel problem (i.e. ongoing) is there any reason not to force the build so that we don't block on it? I'll add a comment to the bug, obviously. [11:01] Hudson's happy. Let's use Hudson. [11:04] * gmb can't stop thinking people are referring to mwhudson when they say "Hudson." [11:04] My statement may still stand, even in that case. [11:04] Hudson, old boy, merge this branch would you? There's a good chap. [11:04] True. [11:15] jml: Ah, ignore me; it's obviously not all that intermittent; 3 out of the last 4 builds failed because of this, if I'm reading buildbot right. [11:56] How easy is it to find a timeout by its pageid? [12:02] Morning, all. [12:11] gmb: I utterly forgot to look and got distracted by work [12:11] gmb: But if I'm reading your last statement rightly, it's not really not my fault, it's buildbot being crap? [12:11] When did it start? [12:11] StevenK: Yeah, it looks like it's an annoying but ongoing problem. I haven't had time to dig further. [12:12] Around the time of the databasefixture landing? [12:12] wgrant: The bug was filed on the 29th, FWIW: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/682861 [12:12] <_mup_> Bug #682861: db-devel librarian hung with '[Errno socket error] timed out' [12:13] That's, er, slightly suspicious timing. [12:13] The databasefixture stuff landed on the 28th. [12:14] * StevenK is going to bang the Hudson drum in the new year [12:14] I might actually *gasp* do some work on it while I'm off next week [12:15] Damn, then who am I going to harass during my last week without ~launchpad superpowers? [12:15] wgrant: Go play in the traffic [12:15] :( [12:19] wgrant: well done on passing [12:19] bigjools: Thanks! [12:20] did you get graded? [12:22] Overall for the degree? No. [12:22] when do you know? or don't you? [12:23] I'm not sure if they do it... but let's see what it would be. [12:23] Oh good, H1. [12:24] * bigjools has no idea what that means [12:24] First-Class Honours, ie >=80%. [12:24] I did get a first class honours in hindsight from the university of life [12:24] Haha. [12:25] well done, anyway! [12:25] Thanks. [12:25] It is great to be done. [12:30] jml, i sent email about this, but what do you mean when you refer to the set-based design we discussed 3 years ago? === mrevell is now known as mrevell-lunch [12:55] deryck: Hi. [12:56] hi wgrant [12:57] deryck: I'm seeing around 450 YUI JS files included in when it devmode. It makes page loads sort of slow. Is this likely to be related to your recent lazr-js upgrade? [12:58] wgrant, this is definitely related. After I do qa today on the production changes, then I'll turn my attention to making that better. [12:58] deryck: Thanks. [12:58] I should have sent mail about that already, sorry. [12:58] Turning of devmode makes it fast again, since it just uses launchpad.js. [12:58] I cannot type tonight, apparently. [12:59] heh [12:59] yeah, that is definitely a work around if you're not hacking on js itself. [13:09] flacoste: with db-devel being broken because of this ongoing but not-all-the-time timeout problem, does it make sense to force the build so that people can land things in the interim (and who knows, maybe it'll work this time) [13:09] ? [13:12] * gmb JFDoesIt. === vednis is now known as mars === mrevell-lunch is now known as mrevell [13:41] flacoste, ping === almaisan-away is now known as al-maisan [13:56] gmb, +1 on forcing the build, +1000 on finding a possible clue as to its cause [13:56] mars: Yeah. I'll take a look later if I get the chance. [13:59] mars: It's probably the librarian fixture changes. [13:59] In db-devel r10013 [14:02] wgrant: awake enough for a very quick chat about removing non-PPA pockets from their indexes? [14:02] bigjools: 'course. [14:02] bigjools: They shouldn't be there, though... [14:03] wgrant: I think it's an easy case of fixing the loops in C_writeIndexes and D_writeReleaseFiles [14:03] I was going to abstract the pockets to a call on IArchive [14:03] bigjools: They're not actually ever written at the moment. [14:03] Because they're never dirty. [14:03] right [14:03] but it loops them nonetheless [14:03] We should skip them in A and B as well. [14:03] ah yes [14:05] bigjools: So, I'd just replace all the 'for pocket in PackagePublishingPocket.items' calls with 'for pocket in self.archive.getPocketsForSeries(distroseries)' === matsubara is now known as matsubara-lunch [14:06] wgrant: why by series? [14:07] bigjools: Because you never know what Ubuntu will want to do next. But I guess for now we can skip that bit. [14:07] Since it'll only be used in a couple of places. [14:16] Anyway, it is getting late. [14:29] hi mars [14:45] aww jml you painted over my green bikeshed! [14:53] foundations guys: I think I need to add lazr.enum._enum.EnumItems in lp_sitecustomize.py so that it has no security proxy, any thoughts? === flacoste changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.12 | PQM is open | firefighting: db-devel broken | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting [15:12] flacoste, pong === matsubara-lunch is now known as matsubara [15:17] mars: what can I do for you? [15:18] bigjools, I think Gary would be best positioned to answer that question, but unfortunately he is out today [15:18] mars: yeah, he's blowing out a lot of candles [15:18] :) [15:19] I can't talk, I've got the same number in 3 months... [15:25] how often is the data on qastaging restored? [15:34] * bigjools shakes fist at zope testrunner [15:35] jcsackett: one a week [15:35] once* [15:35] thanks. [15:36] so, if I use an argument to bin/test like "-t soyuz -t !windmill" it ends up running every single test except windmill ones rather than all the soyuz tests except windmill. [15:41] -t is unfortunately || rather than && [15:42] * jelmer often just uses a file with a list of tests to run and calls ./bin/test --load-list=FILE [15:55] jelmer: yeah, && would be more useful === beuno is now known as beuno-lunch [16:31] jtv: are you disappearing soon? [16:31] bac: from the channel, yes. [16:31] jtv: rats. i'd like to work with you on a problem but i realize it is really late for you. [16:32] bac: oh, you're back on the West Pole already? [16:32] yep, for a week [16:32] A week. [16:33] bac: but hey, see if you can grab my attention. What is it you wanted to work on? === deryck is now known as deryck[lunch] === abentley changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.12 | PQM in RC mode | firefighting: - | Get the code: https://dev.launchpad.net/Getting === abentley changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.12 | PQM in RC mode | firefighting: - | Get the code: https://dev.launchpad.net/Getting === deryck[lunch] is now known as deryck === benji is now known as benji-lunch === beuno-lunch is now known as beuno [17:44] libmozjs2s from LP-PPA-Launchpad is trying to uninstall my xulrunner :( === benji-lunch is now known as benji [18:55] sinzui, ping [18:58] Is there an icon that would mean something like "give me the raw content of this" in Launchpad that I should re-use for loggerhead? [19:09] Not that I have seen [19:09] hi deryck [19:10] Also you'd have to jump through lots of licensing hoops if you wanted to take an icon from LP into Loggerhead [19:11] hi sinzui. Is the calendar widget used on registry pages in js or python? I can't find tests or a js file for it. [19:11] deryck, the calendar is yui2. I think it was moved to lp.app [19:12] deryck, we want the widget to be used every time there is a date or datetime field, but we do not seem to know how to ensure the js loads because the field is present [19:13] sinzui, ok, cool. That helps then. The lazr-js upgrade has borked the style of this widget, but I suspect it's an easy fix via class names. [19:13] maxb: Why, are the LP icons not GPL? [19:14] mkanat: No, they are copyright Canonical, restricted rights for use in development only [19:14] maxb: Oh, okay. [19:15] deryck, I thought there was a windmill test for the calendar, but I do not see it [19:16] sinzui, yeah, it doesn't appear to have any test coverage, as far as I can see. [19:17] but no biggie. I'll fix it all up next week. [19:17] I there a yui3 calendar widget yet? [19:17] hmmm, not sure. [19:17] I can look into that a do a replacement. === al-maisan is now known as almaisan-away [19:31] Is there a design document (or something similar) that describes the librarian? [19:31] yes [19:32] librarian.txt, docstrings. [19:32] k, thanks [19:51] jkakar: [19:51] jkakar: hey [19:52] Hi! :) [19:52] jkakar: wondered what you think of my object graph to model query proposal. [19:52] I'm sure its not new, but none of the python ORM's I've seen do it ... [19:53] lifeless: It's still marked as "read" in my mail... I haven't made the time to read through it properly. :( [19:53] lifeless: I want to, just been pressed with lots of other things. [19:53] lifeless: Will try to get to it this weekend. === matsubara is now known as matsubara-afk [20:54] flacoste: Do you really think that Soyuz can be simplified such that domain expertise is no longer required? [20:54] hi william! [20:54] the whole pqm/release thing now largely has to do with stuff in db-devel, right? things done in devel will still roll out on a fairly regular interval, so RCs are less important, aren't they? or have i completely misunderstood something? [20:54] Heh, morning. [20:56] wgrant: i do believe that we can make it so that many changes can be done without requiring vetting by somebody who "understands the whole system" and without the risk of breaking everything [20:56] wgrant: but people will have to understand the basics of package building and archive management [20:56] jcsackett: No, that's right. [20:56] but i don't think it requires a year of dedicated Soyuz hacking to get that [20:56] jelmer: thanks. [20:57] * jcsackett relaxes a touch. [20:58] jcsackett: you should still make sure that all your revisions before 12019 are qa-okl [20:58] or get a rc to fix any qa-bad in that range [20:58] jcsackett: but yeah, we'll continue nodowntime deployment afterwards from devel [20:58] flacoste: i may need to do that. it's not a qa-bad, per se, but a qa-ok that's an incomplete solution to an inprogress bug. but i think i'm about to have the full solution, which would be nice to release all together. [20:59] jcsackett: right, you can negotiate that with abentley [20:59] flacoste: right. first though i have to know if have anything to negotiate. was just checking on time tables. :-P [21:03] jcsackett: are any of your changes going to introduce regressions? [21:03] in the thing i'm working on? [21:04] abentley: in the thing i'm working on, no. in the branch that's qa-oked but we've discovered is incomplete, it wouldn't appear so. nothing is worse than it was, or we'd qa-bad it. [21:05] wgrant: required? no. God Like Knowledge Of Every Line Of Code - that can be fixed. [21:05] jcsackett: Okay, so it won't need to block deployment. [21:05] wgrant: for instance, concurrent uploads to the uploader, what unique keys are :) [21:05] abentley: yeah, the branch already landed isn't a blocker. [21:06] jcsackett: We can give it an RC if you think it's necessary to deploy your changes at the first available opportunity. [21:06] lifeless: It's not quite that simple to fix, I'm afraid. [21:06] jcsackett: That would be Wednesday next week at 10:00 UTC. [21:06] abentley: dig. [21:06] wgrant: is it worth fixing? [21:07] wgrant: (rhetorical) [21:07] jcsackett: Otherwise, land it normally, and it can be part of the first nodowntime deployment of the next cycle. [21:07] abentley: okay. thanks. [21:07] wgrant: I think its worth fixing because the current set of folk that can work on soyuz is the intersect of 'knows zope well', 'knows twisted well', 'knows debian packaging well', 'knows debian archive mgmt well' [21:07] wgrant: ... very small set. [21:08] lifeless: Sure, that sucks. [21:08] every one of those that we can relax by better layering, clearer code, safer system, the larger the set of folk that can contribute sensible. [21:08] But the new model means that the last two sets will vanish. [21:09] uhm [21:09] And after that it's just a matter of time until someone mistakenly qa-ok's something and tens of millions of machines notices. [21:09] wgrant: I think you're mistaken. [21:09] rockstar: chat? [21:10] wgrant: right now, the review policy does not require domain reviews of changes. [21:10] wgrant: the reorg asks folk to take on the challenge of a wider view of the system. It doesn't expect them to suddenly know things they didn't. [21:10] wgrant: learning will be involved. [21:10] lifeless: No, but few outside the team make Soyuz changes. And I know that at least I review everything Soyuzy. [21:10] wgrant: what will stop you doing that? [21:11] lifeless: Trusting everyone to make changes to such a fragile, critical piece of Launchpad. Probably starting with myself. [21:12] wgrant: do you mean 'there will be so many soyuz patches wgrant cannot look at them all' ? [21:14] lifeless: No. I mean what I said. When I can be reasonably confident that people aren't able to accidentally break things, I will have a reason to stop watching. [21:15] Soyuz is fragile. Some of this fragility is implied by the domain. And the consequences of a mistake can be immense. [21:16] abentley, sure. Mumble? [21:16] rockstar: sure. [21:18] wgrant: you're talking about breaking the primary archive in some fashion, right ? [21:18] lifeless: That's the worst case, but sure. [21:18] so, some of the fragility is due to apt disk layout, sure. [21:18] wgrant: i really don't get 'fragility implied by the domain' [21:18] but a lot of it is due to our code. [21:18] i think that's my main point of disagreement [21:18] what is so fragile in archive management [21:19] lifeless: A lot of the code is atrocious, sure. [21:19] wgrant: I'll buy 'expected cost of a qa failure is huge' [21:19] I'll also throw in "QAing completely is just about impossible" [21:19]