[00:04] atm [00:04] losa pin [00:04] lifeless: heya [00:04] ^ [00:04] thumper: is it on staging ? [00:04] lifeless: staging is down [00:05] thumper: well, spmdo works miracles [00:05] anyhow [00:05] the answer is: [00:05] - to profile [00:05] a) get it on staging [00:05] b) losa ping and ask them to enable profiling. hit the page. ask them to disable [00:05] and the profile ends up on devpad 3 minutes later [00:05] thumper: if its on edge, do you have a user-oops to look at perhaps ? [00:06] no, but I can create one with ++oops++ [00:06] :) [00:07] right, I'd do that - until we have a staging-daily environment its the best you can do until it has percolated to staging [00:08] * thumper nods === Ursinha is now known as Ursinha-afk [00:21] wgrant: ping [00:39] thumper: Hi. [00:39] wgrant: hi [00:39] wgrant: IBuildFarmBuildJob says that build can't be None [00:39] does that fit with your understanding? [00:39] because the database doesn't have that constraint [00:40] thumper: Indeed it doesn't. That's a bug. [00:40] But those tables are scheduled for removal as soon as Translations gets their job fixed. [00:41] ah... [00:41] what? [00:41] Hm? [00:41] I know there was a migration from some tables to some other tables [00:41] but I thought that this is the one we migrated to not from [00:41] because the recipe stuff is using it [00:42] The migration is half done. [00:42] (at least on production) [00:42] was the recipe stuff landed this cycle? [00:42] The remainder cannot be done until Translations does the first half. [00:42] which might be on db-devel only [00:42] Last cycle. [00:42] Production has the current build farm schema. [00:43] if this is a table that needs to go, and recipe stuff should not be using it, and it should have landed last cycle [00:43] why are we getting oops 1705EB2401 [00:43] Recipe stuff is using it. [00:44] but it shouldn't be? [00:44] It is part of the old queueing model. [00:44] SPRBs and BPBs have been ported to the new one. [00:44] But they still use the old one too. [00:45] Since TTBJs use the old one. [00:45] rockstar: btw, I'd really like us to keep the approval means 'human says ok' and use Queued for tarmac. [00:45] Once TTBJs use the new one, we can quickly stop using and remove the old one. [00:45] rockstar: really really really like us to do that. [00:47] thumper: So, the table is still in use, but is redundant. [00:49] wgrant: do you know if the TTBJs have been moved? [00:50] They haven't. [00:50] I was hoping it would happen this cycle. [00:50] But apparently not. [00:50] * thumper adds reminder to poke danilos [01:07] wgrant: whats DistroSeries:+templates all about ? [01:07] reminder to self; nag stub about PPR reliability, staging. [01:09] lifeless: Isn't that Translations? [01:09] ah, that would make sense [01:09] 29.25 - 99% completion time [01:10] How is that possible? [01:10] How can the 99% be above the timeout? [01:10] the magic of science [01:10] easily === Ursinha-afk is now known as Ursinha [01:28] hm [01:28] does someone remember where that dead easy graph generator was? [01:29] where you could graph any number on any website? [01:29] webnumbr? [01:29] poolie: yes, thanks [01:50] wgrant: around ? [01:52] wgrant: were you going to poke at https://bugs.edge.launchpad.net/soyuz/+bug/618372 or is it my imagination ? [01:52] <_mup_> Bug #618372: Distribution:+search slow 50% of requests [01:54] lifeless: I don't think I was doing anything about that. [01:55] I did look and cringe at the code and views a few weeks ago, but that was for other reasons. [01:55] wgrant: care to be enticed ? [01:58] I have four projects due tomorrow... so no, sorry. [01:58] wgrant: meep. get on them :) [01:58] Heh, maybe. [01:58] wgrant: Did you see this one - 19997.01launchpad-main-masterSELECT COUNT(*) FROM Bug WHERE Bug.private = FALSE [01:58] bah [01:58] thats 1 9997 [01:58] on https://api.staging.launchpad.net/1.0/bugs?assignee=xxx [01:59] Hah. [01:59] Awesome. [01:59] Any idea what's causing that? [01:59] not yet. [02:00] but I LoLed [02:00] wgrant: also, https://dev.launchpad.net/ArchitectureGuide may be interesting for you; as always I seek feedback on these things. [02:05] Hmm. [02:06] wgrant: as I say in the presentation, the metrics are noddy [02:06] but until we try, we won't have any [02:07] Launchpad's never really done the whole metric thing. [02:07] So it's a good start. [02:16] wgrant: so, distro:+search [02:16] it was the top timeout yesterday [02:16] Hm, that's good, considering how many other timeouts there used to be. [02:16] 57 / 13 Distribution:+search [02:17] 44 / 114 CodeImportSchedulerApplication:CodeImportSchedulerAPI [02:17] anyhow [02:17] you said it was terrible and should use <<>>> [02:17] did you have some advice for what it should use ? [02:18] wgrant: yes, we're getting the first rung of omg out of the way [02:18] wgrant: I'm nearly ready to lower the timeout again [02:19] in fact, I'll throw that up now [02:20] What are they sitting at now? [02:20] lpnet is 17 [02:21] I'll leave edge for now [02:21] hmm edge had 29 timeouts [02:21] drop both [02:21] Yup. [02:22] It would be nice to see how much the two cache tables actually win us. [02:23] Because the query there isn't really using them. [02:25] I can't see what's generating that query. [02:25] I'll poke some [02:25] perhaps do-from-scratch [02:25] Oh, right, there. [02:26] Distribution.searchBinaryPackages, with exact_match=True. [02:26] At that point it's really not using the only beneficial part of the cache. [02:26] (binpkgnames) [02:27] IIRC that search is just about the only user of the tables... so if you can make it work without them we can delete lots. [02:33] thumper: ping [02:34] spm: ping [02:34] lifeless: heyo [02:34] hey [02:35] that config change didn't merge [02:35] or something [02:35] and I have more now [02:35] also the diff updater is slow? [02:35] by which I mean, please check it doth not need shooting. [02:35] yeah. :-) started looking then got an alert that calamansi has gone awol - just chsing that down atm. I've decied I'm going to relearn forth. the stack based thing is more in tune with my reality/ [02:36] you could go the musical route [02:36] learn fifth [02:36] * lifeless boom tishes [02:36] * spm stares blankly at the screen and conceeds the point to lifeless [02:37] spm: I'm pretty sure the MP daemon is ded [02:37] 10 mins and 5 deep in interrupts. le sigh :-) [02:37] :< [02:37] are they all LP ? [02:38] alas no === Ursinha is now known as Ursinha-afk [02:39] yeah. looks wedged. about 30 mins. killing... [02:40] 20 mins. not 30. [02:40] lifeless: What are your performance targets? [02:40] For timeout and 99%. [02:40] wgrant: the same, and both were in the vision document ;) [02:41] wgrant: the immediate target is 5 seconds for 99% of requests [02:41] the long term one is 1 second, with hard timeout still at 5 seconds [02:41] (so right now they are the same, once we get below 5 seconds across the board it will be different) [02:41] lifeless: I didn't read the presentation. [02:42] wgrant: :P [02:42] Right, I was wanting the long term one. [02:42] wgrant: not even 2 months back ? [02:42] (I read the presentation just after the Epic) [02:42] Yeah. [02:42] yeah [02:42] lifeless: well you'll be pleased to know that it was either you or maxb that caused this problem <== science-less accusation [02:42] spm: ^^ [02:42] but those mp's should now be processed :-) [02:43] thanks [02:44] spm: ok, https://code.edge.launchpad.net/~lifeless/lp-production-configs/timeouts/+merge/34042 is updated; can you please (I know you're busy) eyeball the updated diff, so i can start the 1 hour lead-in to make the change happen ? [02:44] sure sure [02:45] lifeless: +1'd [02:46] lifeless: actually - I assumed that we have already dropped to equivalent values on staging as a trial? [02:46] staging is set to 10 seconds [02:46] the data I am going off is the last few days of oops reports for lpnet and edge [02:46] sweet; re-confirming the +1. ta. [02:46] because not enough people hammer staging for it to mean anything [02:47] :-) [02:47] I have a selection of hammers here I'm happy to lend? [02:47] small sledge, ball pein, couple of reular claws; japanese chisel (my personal fave); few others.... [02:48] rubber ducky, you're the one === Ursinha-afk is now known as Ursinha [02:51] lifeless: pong [02:52] thumper: I want your +1 on lowering the timeouts [02:52] thumper: since the new policy seems to say noone can JFDI even with sysadmin agreement [02:53] thumper: the branch which will need to be deployed is running through now (its not an LP branch) [02:53] lifeless: lowering which timeouts to what? [02:53] lpnet and edge hard timeouts, by 1 second each [02:53] do it [02:53] +1 [02:54] thank you [02:54] spm: that is in pqm now, all going well in ~ 60 minutes I'll be asking for a deployment to all appservers. [02:54] spm: if that is convenient [02:55] * spm consults the diary [02:55] spm: I can use the magic words if you want [02:55] "NOW"? [02:56] spm: NOW, Knave! [02:56] Knave. I like it. [02:56] but actually, NOW+60 :P [02:56] ${NOW}+60? [02:56] :> [03:07] what happened between 1500 and 1600 graph time yesterday ? [03:07] https://lpstats.canonical.com/graphs/OopsEdgeHourly/ [03:58] spm: help [03:58] zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/script.zcml", line 7.4-7.35 [03:58] ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/lib/canonical/configure.zcml", line 80.4-86.10 [03:59] ImportError: No module named debian [03:59] I got that merging the config change [03:59] make: *** [lib/canonical/launchpad/apidoc/index.html] Error 1 [03:59] blink [03:59] It means the pqm chroot's packages are out of date [03:59] * spm holds head in hands and cries a little. on the inside at any rate. [03:59] spm: can you smack or arrange a smack, of apt-get update; apt-get-upgrade in it ? [04:00] yah, one sec === Ursinha is now known as Ursinha-zzz [05:05] spm: hi [05:05] spm: https://bugs.edge.launchpad.net/soyuz/+bug/618372/comments/6 [05:05] <_mup_> Bug #618372: Distribution:+search slow 50% of requests [05:05] can you please rnu the query in that link on a prod slave [05:05] I've checked staging already, its 10x slower than stub found [05:06] I just need the \timing for it [05:09] db tuning: funrollloops for adults [05:11] wgrant: whould would searchBinaryPackages use the dspc ? [05:16] wgrant: also Distribution.searchBinaryPackages interface docstring lies! [05:30] lifeless: Why wouldn't it? [05:31] Ah. [05:31] It returns DSPCs, rather than DSPs? [05:31] yes [05:31] so its not as simple as 'make a DSP returning function' [05:31] lifeless: ew. fun. [05:32] spm: pweese [05:32] wgrant: distroarchseries searchBinaryPackages doesn't, it does something different [05:33] wgrant: I don't understand why they have or need different code [05:33] wgrant: though you may hate das - it uses BPPH, or is that ok ? [05:34] lifeless: Well, it might be that simple if you change the callsites to not be insane. [05:34] "may hate das"? [05:34] Oh. [05:34] DAS [05:34] Right. [05:35] lifeless: If DAS's implementation is fast, it's fine. [05:35] its not on the top candidates page [05:35] I'll shove it into D [05:36] DSPC is basically just a precalculated version of that join. [05:36] lifeless: prod1. Time: 173.932 ms [05:36] spm: grahfuck [05:36] repeats are 50ms... [05:36] spm: what pg version is staging running ? [05:36] 8.3 [05:36] ok [05:37] so there is something really quite different there >< [05:37] spm: that was the one in my comment, comment 6 ? [05:38] yup [05:39] thanks, appreciated. [05:39] * lifeless subscribes stub again [05:44] wgrant: also, DAS has no code reuse at all. sob. [05:45] lifeless: is it worse that foo.specifications ? [05:45] *than [05:46] yes [05:46] specs at least has a mix in I could pull code into [05:46] impressive [05:46] DAS was stuck between Soyuz and Registry for a while. [05:46] It's not well-loved. [05:46] it appears positively hated [05:46] but fast (or unused) [05:47] https://edge.launchpad.net/ubuntu/+search?text=mplayer is still boom on prod [05:48] oh and thats interesting [05:48] distroseries search is totally differnet [05:48] \o/ [05:48] * lifeless needs a drink [05:51] wgrant: is that a bug? [05:52] anyhow, there would be a different if I drop DSPC - rather than one row per source package it would be N rows - one per binary [05:52] so, I'm going to assume thats actually the desired win for now [05:52] and try and ask bigjools about this later [05:53] lifeless: Can't you just get distinct DSPs? [05:54] wgrant: *think* [05:54] lifeless: Hm? [05:55] wgrant: oh, hmm. [05:55] yes, should be doable [05:55] DAS.sBP returns BinaryPackageRelease, [05:56] wgrant: you asked about 99% stuff and timeouts before [05:56] wgrant: I forgot to answer [05:56] the time at which 99% of requests complete can be over the timeout if either: [05:56] - enough requests fail (e.g. 2%) [05:57] - the requests are completing with high durations as soft time outs due to bugs in the timeout trapping code (or no queries happening after the timeout) [05:57] stub: hi! [05:57] stub: got a few minutes? [05:57] lifeless: I don't see how it's possible unless the timeout stuff is buggy. [05:57] Since nothing can finish after the timeout -- it would have timed out. [05:58] wgrant: the timeout code works by checking for a timeout at predefined points. [05:58] currently that predefined point is 'sql queries within publication' [05:58] yo [05:58] lifeless: Ah, so an SQL query won't be terminated? [05:58] wgrant: it will, because we tell the server the remaining time [05:58] Ah. [05:58] wgrant: but there are caveats - pg doesn't cancel queries until you get the locks yoiu're waiting on [05:59] wgrant: and if stop querying and go into python, or librarian, or email time, you won't time out. [05:59] these are bugs. they will be fixed. [05:59] Ew. [06:00] stub: so, I have a bug you looked at that puzzles me on staging; https://bugs.edge.launchpad.net/soyuz/+bug/618372 and... [06:00] <_mup_> Bug #618372: Distribution:+search slow 50% of requests [06:00] stub: PPR is failing very often - it seems to run sodium into swap, which triggers a meltdown [06:00] and [06:00] stub: staging is failing waiting for a slony timeout, so its now not updated since sunday. [06:01] stub: I have no idea how these slot into your priorities [06:01] I got the PPR email. I haven't seen it go that big. We can fix that but it will be slower generating reports (how much? Dunno until it is done) [06:02] wgrant: so its not hugely ew; unless we raise a signal right on the timeout its always going to be a little fuzzy [06:02] Staging probably means the slon daemons died. Restarting the process should sort things, but I was going to have a look now. [06:02] wgrant: (and doing that has *serious* hair on it) [06:02] stub: its died the same way three times in a row [06:03] wgrant: so if its approximate, even without massive-overruns, all you need is 2% timing out and the 99% point will be outside the timeout. [06:03] ok. no idea what happened then - the email I had wasn't enough to diagnose. [06:04] stub: I say this because I've seen the same pastebin contents +- the list of exact what is attached, on tuesd morn, wed morn & today :) [06:08] That is odd, because the only log on the system is from the pastebin I got emailed yesterday. [06:08] So the restore has been attempted once and it failed, or the logs are disappearing or this was from manual runs [06:12] is https://pastebin.canonical.com/36551/ what you got yesteday ? [06:12] that happened overnight from my perspective [06:13] oic. This is a code only update, and the slon daemons have died for whatever reason so it can't apply new patches to the existing database. [06:14] So the code only update needs to bouce the replication daemons to ensure they are running [06:14] jtv: bug 618393 - you say it would be hard to do for individual translators - could you expand why? it might be like the one I did for Bug.userCanView (which grants EDIT) [06:14] <_mup_> Bug #618393: TranslationGroup:+index slow 1-2% of requests timing out [06:14] brb [06:15] spm: /srv/staging.launchpad.net/staging/launchpad/database/replication; LPCONFIG=staging ./slon_ctl.py stop; LPCONFIG=staging ./slon_ctl.py start [06:16] spm: I've run that so the code update should work now. [06:16] stub: context? that needs to be in the staging restore script? [06:16] oh right. ta. I'll rm the lock [06:16] spm: So we must not have a nagios check on the staging replication lag [06:17] cat /etc/nagios/nrpe.cfg :-) [06:17] nope === jamesh_ is now known as jamesh [07:04] spm: Do you know if there is movement on the PG 8.4 on staging request? [07:05] stub: that's stuck on me atm, and no is the short answer; I plan on going into an irc hole of /ignore * tomorrow to make it happen - get the packages ready as in. [07:05] Ta [07:06] ... not helped by codebounce wanting all the swap it can get it's grubby mitts on [07:06] spm, lifeless: One of the things that is improved every PG release is the query planner. Some of the issues we are looking at the moment might just evaporate. [07:06] ^^^ boundless optimism on display from the DBA [07:06] (Oooo! it rhymes!) [07:07] If I can keep 'that should be fixed in the next release' up until retirement, I'll be sweet. [07:07] stub + sweet. the mind rebels. === henninge_ is now known as henninge [08:52] good morning [09:03] spm: have you EOD'd ? [09:03] I'm still getting ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/lib/canonical/configure.zcml", line 80.4-86.10 [09:03] ImportError: No module named debian [09:03] landing the config change [09:03] losa ping ^ [09:05] '/home/---trunk/' [09:05] ---? [09:05] pqm glue [09:05] Yay. [09:06] wgrant: do you recall the cause of that error ? [09:06] wgrant: its stale package versions right ? [09:06] mthaddon: ping [09:06] one sec - in a meeting [09:06] mthaddon: I'll write here, take it up when you get free :) [09:07] the above error is occuring trying to land a change to launchpad-production-config; AIUI landing those configs does a full buildout, so the remaining cause is a stale apt package for (IIRC) python-apt, or something like that. [09:08] There should be new packages in CAT; spm had updated this earlier today, we thought. [09:08] lifeless: Or stale sourcedeps. [09:08] I would deeply deeply appreciate it if we can fix this so that the production-configs change can land [09:08] Hm, actually, python-debian is a package now. [09:09] So, yeah, old version of the package. [09:09] so it may be that PQM is pulling in an old sourcedep [09:09] e.g. if for some reason its using a config-manager config rather than buildout [09:09] (I suspect it may be) [09:09] then, that config may be stale and need to be updated to use the python-debian package rather than the sourcedep. [09:09] lifeless: sorta - was on the weekly with tom [09:10] I will pop back later to offer what assistance I can [09:10] spm: mthaddon: sorry for interrupting the meeting [09:10] so what update to launchpad-dependencies do we think will fix this? [09:10] oh right. so we updated the packages in the chroot for pqm on prasé that should have pulled in the latest hotness of everything [09:10] so, its not the packages [09:11] its probably the manner in which it sets up 'sourcecode' ? [09:11] mthaddon: https://pastebin.canonical.com/36602/ is the list of what was updated [09:12] I don't see python-debian there [09:13] (pqm-hardy)pqm@praseodymium:~$ dpkg -l | grep python-apt [09:13] ii python-apt 0.7.4ubuntu7.5 Python interface to libapt-pkg [09:13] er... [09:13] no [09:13] python-debian [09:13] sorry, wrong package [09:13] ii python-debian 0.1.9 python modules to work with Debian-related d [09:13] whats that launchpad-dependencies package version ? [09:14] 0.72~0.IS.8.04 [09:14] current is 0.81 [09:15] that's not been installed on any of the servers yet [09:15] but nothing in the changelog looks relevant [09:15] and if this works on the servers its clearly not the issue [09:15] mthaddon: can you pastebin me the pqm stanza for production-configs ? [09:15] hrm. that bb amis rt I'm working on may be relevant to that version.... [09:15] yeah, just finding it now [09:16] hrm. no. 72 should be fine. [09:18] wgrant: can you check your sourcecode dir [09:18] tell me, do you have a python-debian subdir ? === danilo_ is now known as danilos [09:20] lifeless: I don' [09:20] t [09:20] wgrant: ah, you're on a recent release [09:21] wgrant: look in lib, it has deb822 and debian symlinks [09:21] lifeless: Yeah, I'm not sure they're used any more [09:21] they are new [09:21] rev 11324 [09:22] Ah. [09:28] mthaddon: spm: Thanks for the help. [09:28] sure [09:28] stub: when is pg8.4 realistically happening; I don't want to wait another cycle to fix the batch of issues. [09:29] stub: but I could; I *really* don't want to wait another 2 cycles. [09:30] lifeless: It will take 1 week after we are happy with staging - swap over one box per day. [09:30] stub: Rather than waiting for the query planner to magically be fixed, shouldn't we be reporting problems? [09:30] lifeless: It is on a separate cycle to rollouts [09:30] lifeless: np [09:30] stub: ok, thats good to know. [09:32] wgrant: I haven't actually seen problems with the query planner. I have seen issues where it is not smart enough. [09:32] wgrant: if the planner isn't smart enough in 8.4 they would reasonably say 'try 9' [09:33] wgrant: we should look at bringing up a trunk pg box to do such evaluations on [09:33] but not just yet [09:33]