| lifeless | atm | 00:04 |
|---|---|---|
| lifeless | losa pin | 00:04 |
| spm | lifeless: heya | 00:04 |
| lifeless | ^ | 00:04 |
| lifeless | thumper: is it on staging ? | 00:04 |
| thumper | lifeless: staging is down | 00:04 |
| lifeless | thumper: well, spmdo works miracles | 00:05 |
| lifeless | anyhow | 00:05 |
| lifeless | the answer is: | 00:05 |
| lifeless | - to profile | 00:05 |
| lifeless | a) get it on staging | 00:05 |
| lifeless | b) losa ping and ask them to enable profiling. hit the page. ask them to disable | 00:05 |
| lifeless | and the profile ends up on devpad 3 minutes later | 00:05 |
| lifeless | thumper: if its on edge, do you have a user-oops to look at perhaps ? | 00:05 |
| thumper | no, but I can create one with ++oops++ | 00:06 |
| thumper | :) | 00:06 |
| lifeless | right, I'd do that - until we have a staging-daily environment its the best you can do until it has percolated to staging | 00:07 |
| * thumper nods | 00:08 | |
| === Ursinha is now known as Ursinha-afk | ||
| thumper | wgrant: ping | 00:21 |
| wgrant | thumper: Hi. | 00:39 |
| thumper | wgrant: hi | 00:39 |
| thumper | wgrant: IBuildFarmBuildJob says that build can't be None | 00:39 |
| thumper | does that fit with your understanding? | 00:39 |
| thumper | because the database doesn't have that constraint | 00:39 |
| wgrant | thumper: Indeed it doesn't. That's a bug. | 00:40 |
| wgrant | But those tables are scheduled for removal as soon as Translations gets their job fixed. | 00:40 |
| thumper | ah... | 00:41 |
| thumper | what? | 00:41 |
| wgrant | Hm? | 00:41 |
| thumper | I know there was a migration from some tables to some other tables | 00:41 |
| thumper | but I thought that this is the one we migrated to not from | 00:41 |
| thumper | because the recipe stuff is using it | 00:41 |
| wgrant | The migration is half done. | 00:42 |
| thumper | (at least on production) | 00:42 |
| thumper | was the recipe stuff landed this cycle? | 00:42 |
| wgrant | The remainder cannot be done until Translations does the first half. | 00:42 |
| thumper | which might be on db-devel only | 00:42 |
| wgrant | Last cycle. | 00:42 |
| wgrant | Production has the current build farm schema. | 00:42 |
| thumper | if this is a table that needs to go, and recipe stuff should not be using it, and it should have landed last cycle | 00:43 |
| thumper | why are we getting oops 1705EB2401 | 00:43 |
| wgrant | Recipe stuff is using it. | 00:43 |
| thumper | but it shouldn't be? | 00:44 |
| wgrant | It is part of the old queueing model. | 00:44 |
| wgrant | SPRBs and BPBs have been ported to the new one. | 00:44 |
| wgrant | But they still use the old one too. | 00:44 |
| wgrant | Since TTBJs use the old one. | 00:45 |
| lifeless | rockstar: btw, I'd really like us to keep the approval means 'human says ok' and use Queued for tarmac. | 00:45 |
| wgrant | Once TTBJs use the new one, we can quickly stop using and remove the old one. | 00:45 |
| lifeless | rockstar: really really really like us to do that. | 00:45 |
| wgrant | thumper: So, the table is still in use, but is redundant. | 00:47 |
| thumper | wgrant: do you know if the TTBJs have been moved? | 00:49 |
| wgrant | They haven't. | 00:50 |
| wgrant | I was hoping it would happen this cycle. | 00:50 |
| wgrant | But apparently not. | 00:50 |
| * thumper adds reminder to poke danilos | 00:50 | |
| lifeless | wgrant: whats DistroSeries:+templates all about ? | 01:07 |
| lifeless | reminder to self; nag stub about PPR reliability, staging. | 01:07 |
| wgrant | lifeless: Isn't that Translations? | 01:09 |
| lifeless | ah, that would make sense | 01:09 |
| lifeless | 29.25 - 99% completion time | 01:09 |
| wgrant | How is that possible? | 01:10 |
| wgrant | How can the 99% be above the timeout? | 01:10 |
| lifeless | the magic of science | 01:10 |
| lifeless | easily | 01:10 |
| === Ursinha-afk is now known as Ursinha | ||
| mwhudson | hm | 01:28 |
| mwhudson | does someone remember where that dead easy graph generator was? | 01:28 |
| mwhudson | where you could graph any number on any website? | 01:29 |
| poolie | webnumbr? | 01:29 |
| mwhudson | poolie: yes, thanks | 01:29 |
| lifeless | wgrant: around ? | 01:50 |
| lifeless | wgrant: were you going to poke at https://bugs.edge.launchpad.net/soyuz/+bug/618372 or is it my imagination ? | 01:52 |
| _mup_ | Bug #618372: Distribution:+search slow 50% of requests <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/618372> | 01:52 |
| wgrant | lifeless: I don't think I was doing anything about that. | 01:54 |
| wgrant | I did look and cringe at the code and views a few weeks ago, but that was for other reasons. | 01:55 |
| lifeless | wgrant: care to be enticed ? | 01:55 |
| wgrant | I have four projects due tomorrow... so no, sorry. | 01:58 |
| lifeless | wgrant: meep. get on them :) | 01:58 |
| wgrant | Heh, maybe. | 01:58 |
| lifeless | wgrant: Did you see this one - 19997.01launchpad-main-masterSELECT COUNT(*) FROM Bug WHERE Bug.private = FALSE | 01:58 |
| lifeless | bah | 01:58 |
| lifeless | thats 1 9997 | 01:58 |
| lifeless | on https://api.staging.launchpad.net/1.0/bugs?assignee=xxx | 01:58 |
| wgrant | Hah. | 01:59 |
| wgrant | Awesome. | 01:59 |
| wgrant | Any idea what's causing that? | 01:59 |
| lifeless | not yet. | 01:59 |
| lifeless | but I LoLed | 02:00 |
| lifeless | wgrant: also, https://dev.launchpad.net/ArchitectureGuide may be interesting for you; as always I seek feedback on these things. | 02:00 |
| wgrant | Hmm. | 02:05 |
| lifeless | wgrant: as I say in the presentation, the metrics are noddy | 02:06 |
| lifeless | but until we try, we won't have any | 02:06 |
| wgrant | Launchpad's never really done the whole metric thing. | 02:07 |
| wgrant | So it's a good start. | 02:07 |
| lifeless | wgrant: so, distro:+search | 02:16 |
| lifeless | it was the top timeout yesterday | 02:16 |
| wgrant | Hm, that's good, considering how many other timeouts there used to be. | 02:16 |
| lifeless | 57 / 13 Distribution:+search | 02:16 |
| lifeless | 44 / 114 CodeImportSchedulerApplication:CodeImportSchedulerAPI | 02:17 |
| lifeless | anyhow | 02:17 |
| lifeless | you said it was terrible and should use <<>>> | 02:17 |
| lifeless | did you have some advice for what it should use ? | 02:17 |
| lifeless | wgrant: yes, we're getting the first rung of omg out of the way | 02:18 |
| lifeless | wgrant: I'm nearly ready to lower the timeout again | 02:18 |
| lifeless | in fact, I'll throw that up now | 02:19 |
| wgrant | What are they sitting at now? | 02:20 |
| lifeless | lpnet is 17 | 02:20 |
| lifeless | I'll leave edge for now | 02:21 |
| lifeless | hmm edge had 29 timeouts | 02:21 |
| lifeless | drop both | 02:21 |
| wgrant | Yup. | 02:21 |
| wgrant | It would be nice to see how much the two cache tables actually win us. | 02:22 |
| wgrant | Because the query there isn't really using them. | 02:23 |
| wgrant | I can't see what's generating that query. | 02:25 |
| lifeless | I'll poke some | 02:25 |
| lifeless | perhaps do-from-scratch | 02:25 |
| wgrant | Oh, right, there. | 02:25 |
| wgrant | Distribution.searchBinaryPackages, with exact_match=True. | 02:26 |
| wgrant | At that point it's really not using the only beneficial part of the cache. | 02:26 |
| wgrant | (binpkgnames) | 02:26 |
| wgrant | IIRC that search is just about the only user of the tables... so if you can make it work without them we can delete lots. | 02:27 |
| lifeless | thumper: ping | 02:33 |
| lifeless | spm: ping | 02:34 |
| spm | lifeless: heyo | 02:34 |
| lifeless | hey | 02:34 |
| lifeless | that config change didn't merge | 02:35 |
| lifeless | or something | 02:35 |
| lifeless | and I have more now | 02:35 |
| lifeless | also the diff updater is slow? | 02:35 |
| lifeless | by which I mean, please check it doth not need shooting. | 02:35 |
| spm | yeah. :-) started looking then got an alert that calamansi has gone awol - just chsing that down atm. I've decied I'm going to relearn forth. the stack based thing is more in tune with my reality/ | 02:35 |
| lifeless | you could go the musical route | 02:36 |
| lifeless | learn fifth | 02:36 |
| * lifeless boom tishes | 02:36 | |
| * spm stares blankly at the screen and conceeds the point to lifeless | 02:36 | |
| lifeless | spm: I'm pretty sure the MP daemon is ded | 02:37 |
| spm | 10 mins and 5 deep in interrupts. le sigh :-) | 02:37 |
| lifeless | :< | 02:37 |
| lifeless | are they all LP ? | 02:37 |
| spm | alas no | 02:38 |
| === Ursinha is now known as Ursinha-afk | ||
| spm | yeah. looks wedged. about 30 mins. killing... | 02:39 |
| spm | 20 mins. not 30. | 02:40 |
| wgrant | lifeless: What are your performance targets? | 02:40 |
| wgrant | For timeout and 99%. | 02:40 |
| lifeless | wgrant: the same, and both were in the vision document ;) | 02:40 |
| lifeless | wgrant: the immediate target is 5 seconds for 99% of requests | 02:41 |
| lifeless | the long term one is 1 second, with hard timeout still at 5 seconds | 02:41 |
| lifeless | (so right now they are the same, once we get below 5 seconds across the board it will be different) | 02:41 |
| wgrant | lifeless: I didn't read the presentation. | 02:41 |
| lifeless | wgrant: :P | 02:42 |
| wgrant | Right, I was wanting the long term one. | 02:42 |
| lifeless | wgrant: not even 2 months back ? | 02:42 |
| wgrant | (I read the presentation just after the Epic) | 02:42 |
| wgrant | Yeah. | 02:42 |
| lifeless | yeah | 02:42 |
| spm | lifeless: well you'll be pleased to know that it was either you or maxb that caused this problem <== science-less accusation | 02:42 |
| lifeless | spm: ^^ | 02:42 |
| spm | but those mp's should now be processed :-) | 02:42 |
| lifeless | thanks | 02:43 |
| lifeless | spm: ok, https://code.edge.launchpad.net/~lifeless/lp-production-configs/timeouts/+merge/34042 is updated; can you please (I know you're busy) eyeball the updated diff, so i can start the 1 hour lead-in to make the change happen ? | 02:44 |
| spm | sure sure | 02:44 |
| spm | lifeless: +1'd | 02:45 |
| spm | lifeless: actually - I assumed that we have already dropped to equivalent values on staging as a trial? | 02:46 |
| lifeless | staging is set to 10 seconds | 02:46 |
| lifeless | the data I am going off is the last few days of oops reports for lpnet and edge | 02:46 |
| spm | sweet; re-confirming the +1. ta. | 02:46 |
| lifeless | because not enough people hammer staging for it to mean anything | 02:46 |
| spm | :-) | 02:47 |
| spm | I have a selection of hammers here I'm happy to lend? | 02:47 |
| spm | small sledge, ball pein, couple of reular claws; japanese chisel (my personal fave); few others.... | 02:47 |
| lifeless | rubber ducky, you're the one | 02:48 |
| === Ursinha-afk is now known as Ursinha | ||
| thumper | lifeless: pong | 02:51 |
| lifeless | thumper: I want your +1 on lowering the timeouts | 02:52 |
| lifeless | thumper: since the new policy seems to say noone can JFDI even with sysadmin agreement | 02:52 |
| lifeless | thumper: the branch which will need to be deployed is running through now (its not an LP branch) | 02:53 |
| thumper | lifeless: lowering which timeouts to what? | 02:53 |
| lifeless | lpnet and edge hard timeouts, by 1 second each | 02:53 |
| thumper | do it | 02:53 |
| thumper | +1 | 02:53 |
| lifeless | thank you | 02:54 |
| lifeless | spm: that is in pqm now, all going well in ~ 60 minutes I'll be asking for a deployment to all appservers. | 02:54 |
| lifeless | spm: if that is convenient | 02:54 |
| * spm consults the diary | 02:55 | |
| lifeless | spm: I can use the magic words if you want | 02:55 |
| spm | "NOW"? | 02:55 |
| lifeless | spm: NOW, Knave! | 02:56 |
| spm | Knave. I like it. | 02:56 |
| lifeless | but actually, NOW+60 :P | 02:56 |
| spm | ${NOW}+60? | 02:56 |
| lifeless | :> | 02:56 |
| lifeless | what happened between 1500 and 1600 graph time yesterday ? | 03:07 |
| lifeless | https://lpstats.canonical.com/graphs/OopsEdgeHourly/ | 03:07 |
| lifeless | spm: help | 03:58 |
| lifeless | zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/script.zcml", line 7.4-7.35 | 03:58 |
| lifeless | ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/lib/canonical/configure.zcml", line 80.4-86.10 | 03:58 |
| lifeless | ImportError: No module named debian | 03:59 |
| lifeless | I got that merging the config change | 03:59 |
| lifeless | make: *** [lib/canonical/launchpad/apidoc/index.html] Error 1 | 03:59 |
| spm | blink | 03:59 |
| lifeless | It means the pqm chroot's packages are out of date | 03:59 |
| * spm holds head in hands and cries a little. on the inside at any rate. | 03:59 | |
| lifeless | spm: can you smack or arrange a smack, of apt-get update; apt-get-upgrade in it ? | 03:59 |
| spm | yah, one sec | 04:00 |
| === Ursinha is now known as Ursinha-zzz | ||
| lifeless | spm: hi | 05:05 |
| lifeless | spm: https://bugs.edge.launchpad.net/soyuz/+bug/618372/comments/6 | 05:05 |
| _mup_ | Bug #618372: Distribution:+search slow 50% of requests <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/618372> | 05:05 |
| lifeless | can you please rnu the query in that link on a prod slave | 05:05 |
| lifeless | I've checked staging already, its 10x slower than stub found | 05:05 |
| lifeless | I just need the \timing for it | 05:06 |
| lifeless | db tuning: funrollloops for adults | 05:09 |
| lifeless | wgrant: whould would searchBinaryPackages use the dspc ? | 05:11 |
| lifeless | wgrant: also Distribution.searchBinaryPackages interface docstring lies! | 05:16 |
| wgrant | lifeless: Why wouldn't it? | 05:30 |
| wgrant | Ah. | 05:31 |
| wgrant | It returns DSPCs, rather than DSPs? | 05:31 |
| lifeless | yes | 05:31 |
| lifeless | so its not as simple as 'make a DSP returning function' | 05:31 |
| spm | lifeless: ew. fun. | 05:31 |
| lifeless | spm: pweese | 05:32 |
| lifeless | wgrant: distroarchseries searchBinaryPackages doesn't, it does something different | 05:32 |
| lifeless | wgrant: I don't understand why they have or need different code | 05:33 |
| lifeless | wgrant: though you may hate das - it uses BPPH, or is that ok ? | 05:33 |
| wgrant | lifeless: Well, it might be that simple if you change the callsites to not be insane. | 05:34 |
| wgrant | "may hate das"? | 05:34 |
| wgrant | Oh. | 05:34 |
| wgrant | DAS | 05:34 |
| wgrant | Right. | 05:34 |
| wgrant | lifeless: If DAS's implementation is fast, it's fine. | 05:35 |
| lifeless | its not on the top candidates page | 05:35 |
| lifeless | I'll shove it into D | 05:35 |
| wgrant | DSPC is basically just a precalculated version of that join. | 05:36 |
| spm | lifeless: prod1. Time: 173.932 ms | 05:36 |
| lifeless | spm: grahfuck | 05:36 |
| spm | repeats are 50ms... | 05:36 |
| lifeless | spm: what pg version is staging running ? | 05:36 |
| spm | 8.3 | 05:36 |
| lifeless | ok | 05:36 |
| lifeless | so there is something really quite different there >< | 05:37 |
| lifeless | spm: that was the one in my comment, comment 6 ? | 05:37 |
| spm | yup | 05:38 |
| lifeless | thanks, appreciated. | 05:39 |
| * lifeless subscribes stub again | 05:39 | |
| lifeless | wgrant: also, DAS has no code reuse at all. sob. | 05:44 |
| mwhudson | lifeless: is it worse that foo.specifications ? | 05:45 |
| mwhudson | *than | 05:45 |
| lifeless | yes | 05:46 |
| lifeless | specs at least has a mix in I could pull code into | 05:46 |
| mwhudson | impressive | 05:46 |
| wgrant | DAS was stuck between Soyuz and Registry for a while. | 05:46 |
| wgrant | It's not well-loved. | 05:46 |
| lifeless | it appears positively hated | 05:46 |
| lifeless | but fast (or unused) | 05:46 |
| lifeless | https://edge.launchpad.net/ubuntu/+search?text=mplayer is still boom on prod | 05:47 |
| lifeless | oh and thats interesting | 05:48 |
| lifeless | distroseries search is totally differnet | 05:48 |
| lifeless | \o/ | 05:48 |
| * lifeless needs a drink | 05:48 | |
| lifeless | wgrant: is that a bug? | 05:51 |
| lifeless | anyhow, there would be a different if I drop DSPC - rather than one row per source package it would be N rows - one per binary | 05:52 |
| lifeless | so, I'm going to assume thats actually the desired win for now | 05:52 |
| lifeless | and try and ask bigjools about this later | 05:52 |
| wgrant | lifeless: Can't you just get distinct DSPs? | 05:53 |
| lifeless | wgrant: *think* | 05:54 |
| wgrant | lifeless: Hm? | 05:54 |
| lifeless | wgrant: oh, hmm. | 05:55 |
| lifeless | yes, should be doable | 05:55 |
| lifeless | DAS.sBP returns BinaryPackageRelease, | 05:55 |
| lifeless | wgrant: you asked about 99% stuff and timeouts before | 05:56 |
| lifeless | wgrant: I forgot to answer | 05:56 |
| lifeless | the time at which 99% of requests complete can be over the timeout if either: | 05:56 |
| lifeless | - enough requests fail (e.g. 2%) | 05:56 |
| lifeless | - the requests are completing with high durations as soft time outs due to bugs in the timeout trapping code (or no queries happening after the timeout) | 05:57 |
| lifeless | stub: hi! | 05:57 |
| lifeless | stub: got a few minutes? | 05:57 |
| wgrant | lifeless: I don't see how it's possible unless the timeout stuff is buggy. | 05:57 |
| wgrant | Since nothing can finish after the timeout -- it would have timed out. | 05:57 |
| lifeless | wgrant: the timeout code works by checking for a timeout at predefined points. | 05:58 |
| lifeless | currently that predefined point is 'sql queries within publication' | 05:58 |
| stub | yo | 05:58 |
| wgrant | lifeless: Ah, so an SQL query won't be terminated? | 05:58 |
| lifeless | wgrant: it will, because we tell the server the remaining time | 05:58 |
| wgrant | Ah. | 05:58 |
| lifeless | wgrant: but there are caveats - pg doesn't cancel queries until you get the locks yoiu're waiting on | 05:58 |
| lifeless | wgrant: and if stop querying and go into python, or librarian, or email time, you won't time out. | 05:59 |
| lifeless | these are bugs. they will be fixed. | 05:59 |
| wgrant | Ew. | 05:59 |
| lifeless | stub: so, I have a bug you looked at that puzzles me on staging; https://bugs.edge.launchpad.net/soyuz/+bug/618372 and... | 06:00 |
| _mup_ | Bug #618372: Distribution:+search slow 50% of requests <dba> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/618372> | 06:00 |
| lifeless | stub: PPR is failing very often - it seems to run sodium into swap, which triggers a meltdown | 06:00 |
| lifeless | and | 06:00 |
| lifeless | stub: staging is failing waiting for a slony timeout, so its now not updated since sunday. | 06:00 |
| lifeless | stub: I have no idea how these slot into your priorities | 06:01 |
| stub | I got the PPR email. I haven't seen it go that big. We can fix that but it will be slower generating reports (how much? Dunno until it is done) | 06:01 |
| lifeless | wgrant: so its not hugely ew; unless we raise a signal right on the timeout its always going to be a little fuzzy | 06:02 |
| stub | Staging probably means the slon daemons died. Restarting the process should sort things, but I was going to have a look now. | 06:02 |
| lifeless | wgrant: (and doing that has *serious* hair on it) | 06:02 |
| lifeless | stub: its died the same way three times in a row | 06:02 |
| lifeless | wgrant: so if its approximate, even without massive-overruns, all you need is 2% timing out and the 99% point will be outside the timeout. | 06:03 |
| stub | ok. no idea what happened then - the email I had wasn't enough to diagnose. | 06:03 |
| lifeless | stub: I say this because I've seen the same pastebin contents +- the list of exact what is attached, on tuesd morn, wed morn & today :) | 06:04 |
| stub | That is odd, because the only log on the system is from the pastebin I got emailed yesterday. | 06:08 |
| stub | So the restore has been attempted once and it failed, or the logs are disappearing or this was from manual runs | 06:08 |
| lifeless | is https://pastebin.canonical.com/36551/ what you got yesteday ? | 06:12 |
| lifeless | that happened overnight from my perspective | 06:12 |
| stub | oic. This is a code only update, and the slon daemons have died for whatever reason so it can't apply new patches to the existing database. | 06:13 |
| stub | So the code only update needs to bouce the replication daemons to ensure they are running | 06:14 |
| lifeless | jtv: bug 618393 - you say it would be hard to do for individual translators - could you expand why? it might be like the one I did for Bug.userCanView (which grants EDIT) | 06:14 |
| _mup_ | Bug #618393: TranslationGroup:+index slow 1-2% of requests timing out <timeout> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/618393> | 06:14 |
| lifeless | brb | 06:14 |
| stub | spm: /srv/staging.launchpad.net/staging/launchpad/database/replication; LPCONFIG=staging ./slon_ctl.py stop; LPCONFIG=staging ./slon_ctl.py start | 06:15 |
| stub | spm: I've run that so the code update should work now. | 06:16 |
| spm | stub: context? that needs to be in the staging restore script? | 06:16 |
| spm | oh right. ta. I'll rm the lock | 06:16 |
| stub | spm: So we must not have a nagios check on the staging replication lag | 06:16 |
| spm | cat /etc/nagios/nrpe.cfg :-) | 06:17 |
| spm | nope | 06:17 |
| === jamesh_ is now known as jamesh | ||
| stub | spm: Do you know if there is movement on the PG 8.4 on staging request? | 07:04 |
| spm | stub: that's stuck on me atm, and no is the short answer; I plan on going into an irc hole of /ignore * tomorrow to make it happen - get the packages ready as in. | 07:05 |
| stub | Ta | 07:05 |
| spm | ... not helped by codebounce wanting all the swap it can get it's grubby mitts on | 07:06 |
| stub | spm, lifeless: One of the things that is improved every PG release is the query planner. Some of the issues we are looking at the moment might just evaporate. | 07:06 |
| spm | ^^^ boundless optimism on display from the DBA | 07:06 |
| spm | (Oooo! it rhymes!) | 07:06 |
| stub | If I can keep 'that should be fixed in the next release' up until retirement, I'll be sweet. | 07:07 |
| spm | stub + sweet. the mind rebels. | 07:07 |
| === henninge_ is now known as henninge | ||
| adeuring | good morning | 08:52 |
| lifeless | spm: have you EOD'd ? | 09:03 |
| lifeless | I'm still getting ZopeXMLConfigurationError: File "/home/pqm/pqm-workdir/home/---trunk/launchpad/lib/canonical/configure.zcml", line 80.4-86.10 | 09:03 |
| lifeless | ImportError: No module named debian | 09:03 |
| lifeless | landing the config change | 09:03 |
| lifeless | losa ping ^ | 09:03 |
| wgrant | '/home/---trunk/' | 09:05 |
| wgrant | ---? | 09:05 |
| lifeless | pqm glue | 09:05 |
| wgrant | Yay. | 09:05 |
| lifeless | wgrant: do you recall the cause of that error ? | 09:06 |
| lifeless | wgrant: its stale package versions right ? | 09:06 |
| lifeless | mthaddon: ping | 09:06 |
| mthaddon | one sec - in a meeting | 09:06 |
| lifeless | mthaddon: I'll write here, take it up when you get free :) | 09:06 |
| lifeless | the above error is occuring trying to land a change to launchpad-production-config; AIUI landing those configs does a full buildout, so the remaining cause is a stale apt package for (IIRC) python-apt, or something like that. | 09:07 |
| lifeless | There should be new packages in CAT; spm had updated this earlier today, we thought. | 09:08 |
| wgrant | lifeless: Or stale sourcedeps. | 09:08 |
| lifeless | I would deeply deeply appreciate it if we can fix this so that the production-configs change can land | 09:08 |
| wgrant | Hm, actually, python-debian is a package now. | 09:08 |
| wgrant | So, yeah, old version of the package. | 09:09 |
| lifeless | so it may be that PQM is pulling in an old sourcedep | 09:09 |
| lifeless | e.g. if for some reason its using a config-manager config rather than buildout | 09:09 |
| lifeless | (I suspect it may be) | 09:09 |
| lifeless | then, that config may be stale and need to be updated to use the python-debian package rather than the sourcedep. | 09:09 |
| spm | lifeless: sorta - was on the weekly with tom | 09:09 |
| lifeless | I will pop back later to offer what assistance I can | 09:10 |
| lifeless | spm: mthaddon: sorry for interrupting the meeting | 09:10 |
| mthaddon | so what update to launchpad-dependencies do we think will fix this? | 09:10 |
| spm | oh right. so we updated the packages in the chroot for pqm on prasé that should have pulled in the latest hotness of everything | 09:10 |
| lifeless | so, its not the packages | 09:10 |
| lifeless | its probably the manner in which it sets up 'sourcecode' ? | 09:11 |
| spm | mthaddon: https://pastebin.canonical.com/36602/ is the list of what was updated | 09:11 |
| lifeless | I don't see python-debian there | 09:12 |
| mthaddon | (pqm-hardy)pqm@praseodymium:~$ dpkg -l | grep python-apt | 09:13 |
| mthaddon | ii python-apt 0.7.4ubuntu7.5 Python interface to libapt-pkg | 09:13 |
| mthaddon | er... | 09:13 |
| lifeless | no | 09:13 |
| lifeless | python-debian | 09:13 |
| mthaddon | sorry, wrong package | 09:13 |
| mthaddon | ii python-debian 0.1.9 python modules to work with Debian-related d | 09:13 |
| lifeless | whats that launchpad-dependencies package version ? | 09:13 |
| mthaddon | 0.72~0.IS.8.04 | 09:14 |
| lifeless | current is 0.81 | 09:14 |
| mthaddon | that's not been installed on any of the servers yet | 09:15 |
| lifeless | but nothing in the changelog looks relevant | 09:15 |
| lifeless | and if this works on the servers its clearly not the issue | 09:15 |
| lifeless | mthaddon: can you pastebin me the pqm stanza for production-configs ? | 09:15 |
| spm | hrm. that bb amis rt I'm working on may be relevant to that version.... | 09:15 |
| mthaddon | yeah, just finding it now | 09:15 |
| spm | hrm. no. 72 should be fine. | 09:16 |
| lifeless | wgrant: can you check your sourcecode dir | 09:18 |
| lifeless | tell me, do you have a python-debian subdir ? | 09:18 |
| === danilo_ is now known as danilos | ||
| wgrant | lifeless: I don' | 09:20 |
| wgrant | t | 09:20 |
| lifeless | wgrant: ah, you're on a recent release | 09:20 |
| lifeless | wgrant: look in lib, it has deb822 and debian symlinks | 09:21 |
| wgrant | lifeless: Yeah, I'm not sure they're used any more | 09:21 |
| lifeless | they are new | 09:21 |
| lifeless | rev 11324 | 09:21 |
| wgrant | Ah. | 09:22 |
| lifeless | mthaddon: spm: Thanks for the help. | 09:28 |
| mthaddon | sure | 09:28 |
| lifeless | stub: when is pg8.4 realistically happening; I don't want to wait another cycle to fix the batch of issues. | 09:28 |
| lifeless | stub: but I could; I *really* don't want to wait another 2 cycles. | 09:29 |
| stub | lifeless: It will take 1 week after we are happy with staging - swap over one box per day. | 09:30 |
| wgrant | stub: Rather than waiting for the query planner to magically be fixed, shouldn't we be reporting problems? | 09:30 |
| stub | lifeless: It is on a separate cycle to rollouts | 09:30 |
| spm | lifeless: np | 09:30 |
| lifeless | stub: ok, thats good to know. | 09:30 |
| stub | wgrant: I haven't actually seen problems with the query planner. I have seen issues where it is not smart enough. | 09:32 |
| lifeless | wgrant: if the planner isn't smart enough in 8.4 they would reasonably say 'try 9' | 09:32 |
| lifeless | wgrant: we should look at bringing up a trunk pg box to do such evaluations on | 09:33 |
| lifeless | but not just yet | 09:33 |
| wgrant | stub: Well, one could argue that a completely dumb planner is just not smart enough. | 09:33 |
| lifeless | ok, pqm is now trying that -again- | 09:34 |
| lifeless | we'll know in ~ an hour | 09:34 |
| stub | wgrant: Yes. But it isn't like upstream is doing nothing - I mentioned PG 8.4 before *because* of the insane speed upstream moves at. | 09:34 |
| stub | wgrant: Its been rare to trip over a PG issue that hasn't already been fixed. | 09:34 |
| wgrant | stub: Ah, I see. | 09:35 |
| lifeless | ... twitter: When you click on these links from Twitter.com or a Twitter application, Twitter will log that click. We hope to use this data to provide better and more relevant content to you over time. ... | 09:35 |
| lifeless | not evil. no, not at all. | 09:36 |
| stub | I think the issue with GIN indexes is one that hasn't been dealt with that will help us | 09:36 |
| stub | And there is an open bug on that. | 09:36 |
| wgrant | lifeless: Doesn't just about everybody do that now? | 09:37 |
| wgrant | Google, Facebook... | 09:37 |
| stub | lifeless: Which is pretty stupid, as every twitter client I have seen goes through a URL shortening service. | 09:37 |
| wgrant | noodles775: Bug 628427 looks relevant. | 09:38 |
| _mup_ | Bug #628427: oops on /builders - LocationError(SourcePackageRecipeBuildJob, 'build') - BuilderSet:+index <oops> <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/628427> | 09:38 |
| wgrant | (morning) | 09:38 |
| * noodles775 looks. Hi :) | 09:39 | |
| lifeless | stub: they are going to unwrap it | 09:39 |
| lifeless | stub: store it, and rewrap it, AFAICT | 09:39 |
| noodles775 | wgrant: Indeed, I've updated 628239. | 09:42 |
| stub | lifeless: Assuming URL shortening services play ball. | 09:45 |
| lifeless | stub: indeed, but they just announced this so I assume their ducks are lined up | 09:45 |
| stub | I guess difficult not too | 09:45 |
| stub | I heard some time ago they were planning this? | 09:46 |
| lifeless | yeah | 09:47 |
| lifeless | I mean they've done their 'mail all users' anouncement | 09:48 |
| stub | I recall discussion on using this to bypass character limits, so http://someservice.com/lets/have/our/tweet/in/a/url would... erm... I forget why. | 09:48 |
| wgrant | noodles775: The expandy bit for PPA source packages lists binaries (without links), then later links to all of the files. Is there any particular reason that it doesn't instead have "(i386) (amd64) (lpia)" links next to each binary package? | 09:53 |
| wgrant | It would save space and probably be more understandable. | 09:53 |
| allenap | bigjools: Do you know much about lp.archivepublisher, specifically its test suite? | 10:06 |
| bigjools | allenap: lots. But I am OTP, I can talk in 30m | 10:06 |
| allenap | bigjools: Cool, thanks :) | 10:06 |
| bigjools | allenap: or you can grab one of my esteemed team | 10:07 |
| allenap | bigjools: Sure, anyone in particular for lp.ap? My questions are about test layers and possible weird interactions with zope.component. | 10:09 |
| allenap | jelmer: Do you have time to help me with a problem in lp.archiveuploader? | 10:19 |
| jelmer | allenap, sure | 10:19 |
| allenap | jelmer: In 4 tests of test_pool.TestPool, I get "Could not adapt" errors for some new cachedproperty code I've done. | 10:20 |
| allenap | jelmer: http://pastebin.ubuntu.com/487147/ | 10:21 |
| wgrant | It's a plain TestCase. | 10:21 |
| wgrant | It has no layer. | 10:21 |
| wgrant | So no Zope. | 10:21 |
| allenap | wgrant: Yeah, but the adapters are also registered with the global site manager when propertycache is imported. | 10:21 |
| wgrant | allenap: Is there a global site manager? | 10:21 |
| allenap | wgrant: Always. | 10:22 |
| allenap | wgrant: Well, zope.component.getGlobalSiteManager() will always return something :) | 10:22 |
| wgrant | Hmm. | 10:23 |
| allenap | wgrant, jelmer: One oddity is that these tests run fine locally. On EC2 they fail. I think there's something weird going on when the full suite is run. | 10:23 |
| lifeless | allenap: you could consider nuking the zope aspect of this | 10:24 |
| lifeless | allenap: I don't think it would make any difference to the spelling | 10:24 |
| allenap | lifeless: Of propertycache? Yeah, that's on my mind :-/ I'd like to try and understand why it's not working if I can. | 10:24 |
| allenap | lifeless: Yeah, agreed. It's just very frustrating and confusing :) | 10:25 |
| jelmer | allenap: I'm at a loss so far, my thought would also be that it's related to layers but then it shouldn't work locally either... | 10:25 |
| allenap | jelmer: Yeah, bloody odd. Okay, I'll email the list to see if anyone has inspiration, and remove the Zope-ish bits so this can land. Thank you :) | 10:27 |
| wgrant | Graaaaah. | 10:28 |
| wgrant | archiveuploader, your lack of test coverage and corresponding missed brokenness distress me. | 10:29 |
| wgrant | The relevant test elides the important bit. | 10:31 |
| wgrant | Gah. | 10:31 |
| allenap | wgrant: Gah indeed :) | 10:32 |
| lifeless | wgrant: 'gahk' | 10:33 |
| lifeless | \o/ | 10:35 |
| wgrant | lifeless: That too, if it's the sound of choking on doctests. | 10:35 |
| lifeless | https://bugs.edge.launchpad.net/ubuntu/+bug/1 didn't time out | 10:35 |
| _mup_ | Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali | 10:35 |
| lifeless | showing all 1327 pages did | 10:35 |
| wgrant | Which pages? | 10:36 |
| lifeless | comments | 10:37 |
| lifeless | blah | 10:37 |
| lifeless | I knew what I meant | 10:37 |
| wgrant | Ah. | 10:37 |
| wgrant | They should be paginated :( | 10:37 |
| lifeless | lets see what the query count is down to | 10:37 |
| lifeless | hmm | 10:39 |
| lifeless | 1327 comments in the summary | 10:39 |
| lifeless | comment 1350 is the end one shown | 10:39 |
| lifeless | methinks there is a bug | 10:39 |
| lifeless | anyhow, at least that is down to 408 queries :P | 10:40 |
| lifeless | hmm, assignments still is right on the edge | 10:41 |
| bigjools | allenap: hi, did someone answer your question? | 10:41 |
| lifeless | At least 76 queries issued in 13.06 seconds | 10:41 |
| allenap | bigjools: Yeah, jelmer helped out. It baffled him too, so I'm taking lifeless's advice, which is to remove the Zopeish bits for now. | 10:42 |
| lifeless | ooh thats nice, bug index in 1.17 seconds now | 10:42 |
| bigjools | allenap: I didn't really pay attention to what you were talking about, so that doesn't mean much to me :) | 10:42 |
| lifeless | for a more regular bug | 10:42 |
| allenap | lifeless: There may be "deleted" comments... I /think/ they're still counted so that urls are stable. | 10:43 |
| lifeless | allenap: ahh | 10:43 |
| lifeless | anyone else feel bug pages are a bit snappier today ? | 10:43 |
| lifeless | on edge, since it deployed | 10:43 |
| allenap | bigjools: I'll explain if you're interested? Part of my plan was also to write to the list so you can wait for that if you want. | 10:44 |
| bigjools | allenap: oh ok I'll wait, don't explain twice | 10:44 |
| allenap | bigjools: Cool :) | 10:45 |
| lifeless | all of bug 1's comments can be retrieved in 2.5 seconds | 10:47 |
| _mup_ | Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali | 10:47 |
| lifeless | wgrant: ^ | 10:47 |
| wgrant | lifeless: Wow! | 10:47 |
| lifeless | wgrant: theres not reason to paginate, we should be able to spit it out easily | 10:47 |
| lifeless | SQL time: 2550 ms | 10:47 |
| lifeless | Non-sql time: 14016 ms | 10:47 |
| lifeless | Total time: 16566 ms | 10:47 |
| lifeless | Statement Count: 408 | 10:47 |
| wgrant | Impressive. | 10:47 |
| lifeless | with some more tuning | 10:48 |
| lifeless | I reckon we can take that under 2 seconds sql | 10:48 |
| lifeless | next step is a profile on staging, which will be tomorrow - need the patch in db-stable | 10:48 |
| lifeless | it also does some daft stuff - grabs all the official bug tags for all related things | 10:49 |
| lifeless | rather than joining and getting the relevant ones | 10:49 |
| lifeless | thats trivial, but still linear vs log | 10:49 |
| lifeless | mthaddon: we run the default threads per appserver right? | 10:52 |
| lifeless | mthaddon: what would be in the impact on you & deployment if I asked for a change to 1 thread per appserver instance and 4 times the appserver instances. | 10:52 |
| lifeless | mthaddon: same hardware. | 10:52 |
| mthaddon | lifeless: it'd be a real maintenance headache (4 times as many initscripts), but it'd be doable | 10:53 |
| lifeless | mthaddon: there is mounting evidence that we're having cross thread interaction | 10:54 |
| lifeless | not bugs, just python being python | 10:54 |
| mthaddon | you're saying python can't do threads? | 10:54 |
| lifeless | python can't do threads | 10:54 |
| lifeless | well known fact | 10:54 |
| * mthaddon gets a poster made up | 10:54 | |
| wgrant | Well, CPython can't do threads. | 10:55 |
| lifeless | if we were primarily waiting on db, it wouldn't matter | 10:55 |
| lifeless | but reality is we're getting that sorted fairly well | 10:55 |
| noodles775 | wgrant: I'm not sure I'm visualising what you mean (re. the build/package links on the package details expander) | 10:55 |
| lifeless | i expect db load to go down over the next few months | 10:55 |
| wgrant | noodles775: I want to remove the binaries from the file list, replacing them with links in the 'Built packages' section. | 10:56 |
| lifeless | wgrant: yes, thats true. But its what everyone means when they don't say 'python the specification' | 10:56 |
| lifeless | except perhaps 4 or 5 people | 10:56 |
| wgrant | Heh. | 10:57 |
| elmo | lifeless: we benched this in montreal as part of the splitit project, and certainly at the time, there was no evidence that we were being hurt by gil contention | 10:57 |
| wgrant | That will hopefully change soon. | 10:57 |
| lifeless | elmo: I didn't say anything about gil contention | 10:57 |
| elmo | while, I'm sure a lot's changed since then, this is not the kind of change I want to run in without some evidence not only that it won't hurt but that it'll help | 10:57 |
| lifeless | elmo: I'm talkin about gil serialisation which is the primary gil issue | 10:57 |
| elmo | lifeless: then s/contention/problems with python and threads/ | 10:57 |
| lifeless | elmo: naturally. I'd want an oops-per-instance report; then try one server with the different config and see if the oops rate changes more than due to the splitting of the instances | 10:58 |
| lifeless | plus PPR stats of the same | 10:58 |
| elmo | what I mean is, specifically, we messed around with number of app servers, number of threads per app server quite a lot | 10:58 |
| lifeless | elmo: what application | 10:58 |
| lifeless | elmo: because if it was the SSO the results are meaningless for this | 10:59 |
| elmo | it was SSO and bugs | 10:59 |
| lifeless | bugs? | 10:59 |
| elmo | bugs are problems in software, but that's not important right now | 10:59 |
| lifeless | ok | 10:59 |
| elmo | lifeless: no, seriously, I mean malone/bugs.launchpad.net - whatever you want to call it | 10:59 |
| lifeless | so the reason the SSO stuff isn't relevant is that it isn't pushing 15K responses around | 11:00 |
| lifeless | elmo: I didn't know you benched malone | 11:00 |
| lifeless | elmo: do you have the data -raw or massaged- for me to look at? | 11:00 |
| lifeless | elmo: when a request takes 5 seconds to render, most of that time is pure bytecode interpretation | 11:01 |
| lifeless | which is primarily serialised | 11:01 |
| lifeless | so a 7 second request with 2 seconds db and 5 seconds render [unloaded] | 11:01 |
| lifeless | means, effectively, ~4 seconds during which the other threads can do nothing - but its not in one big block because the GIL gets released | 11:02 |
| elmo | lifeless: I'm sorry, i don't have any of the data to hand - what there is may be on the wiki if you search for 'splitit' | 11:02 |
| elmo | right | 11:02 |
| elmo | so wait, how is this not about contention then? | 11:02 |
| lifeless | if the timeout is near the requests actual render time then, two of these at once will time each other out. | 11:02 |
| elmo | i.e. what are you trying to solve by having one app server? | 11:03 |
| lifeless | parallelism | 11:03 |
| lifeless | responsiveness more precisely | 11:03 |
| elmo | so, the problem with all of this is that app servers are not exactly slim and svelte | 11:04 |
| elmo | 8056 launchpa 20 0 814m 510m 9512 S 19 8.5 729:46.99 /usr/bin/python -S bin/run -i lpnet7 | 11:05 |
| lifeless | elmo: much of the footprint is per-thread, in theory. | 11:05 |
| elmo | we can't actually have 16 of those | 11:05 |
| lifeless | elmo: of course not. | 11:05 |
| elmo | well we'd want 16? | 11:05 |
| lifeless | if you run up a single thread appserver and put it under load, you can see how big it will be | 11:05 |
| lifeless | we don't want 16 of *those*, we want N of *those/N* where N is the current threadcount (4 for launchpad) | 11:06 |
| lifeless | they won't drop quite linearly in size | 11:06 |
| lifeless | elmo: https://wiki.canonical.com/TaskForce/SplitIt/PerformanceSprint doesn't talk about bugs at all, only shipit and sso :( | 11:08 |
| lifeless | elmo: a key finding: but the average response time increase as the number of concurrent users increase. | 11:08 |
| noodles775 | wgrant: I don't see how you could move all the "Package files" (ie. 3 per build) next to the one built package? (can you scribble on a screenshot? it might make this easier. I've confidence that what you want will make sense ;) ). | 11:09 |
| lifeless | elmo: thats exactly what I'm worried is strongly affecting LP | 11:09 |
| wgrant | noodles775: My though was to stick '(i386) (amd64) (lpia)' links next to each package name. | 11:09 |
| noodles775 | wgrant: OK, what about when the package files are no longer published (ie. we'd want the builds to still be linked)? | 11:10 |
| lifeless | elmo: ok thanks I read the wrapup | 11:12 |
| lifeless | elmo: what I read correlates with the theory I'm putting forward | 11:12 |
| lifeless | elmo: if I'm right, even halving the thread count and doubling the appserver count would be an improvement | 11:12 |
| lifeless | elmo: there is no suggestion, in the benchmarks, that you tried 1 or 2 threads per appserver. | 11:13 |
| wgrant | noodles775: http://williamgrant.id.au/f/1/2010/listing-archive-extra.png | 11:15 |
| wgrant | noodles775: They're not the build links. | 11:16 |
| noodles775 | wgrant: Yes, I find that more intuitive (thinking as a user after the binary package). | 11:18 |
| wgrant | noodles775: Right. I'm looking at this since a friend complained how confusing it was. | 11:21 |
| noodles775 | wgrant: as a user, I'd want those links (to the debs) to be available from the main PPA page. There's certainly room for a column with the built debs (when available) at https://edge.launchpad.net/~wgrant/+archive/experimental | 11:24 |
| wgrant | noodles775: Well, PPA +index needs a bit of a rethink. | 11:25 |
| noodles775 | wgrant: indeed... there was some thought put into it about 6 months ago, but never the time to actualise it :/ | 11:26 |
| wgrant | Can I change PPA key generation to always use 'Launchpad PPA for <person.displayname>'? At the moment it will be named after the display name of your first PPA and then shared with the rest, which is almost always not what you want. | 11:34 |
| bigjools | +1 | 11:34 |
| lifeless | bigjools: briefly, before I crash | 11:34 |
| lifeless | bigjools: distro/+search | 11:34 |
| bigjools | yarp? | 11:34 |
| lifeless | distro/series/+search | 11:34 |
| lifeless | distro/series/arch/+search | 11:35 |
| lifeless | all return different objects | 11:35 |
| lifeless | the /arch/ one I can understand | 11:35 |
| lifeless | distro/+search is performing very badly | 11:35 |
| lifeless | near-top timeout | 11:35 |
| lifeless | by volume, yesterday | 11:35 |
| lifeless | I'm wondering if you want them harmonised, and if so which objet - DSP or DSPCache is appropriate (assuming that the performance is acceptable either way) | 11:36 |
| bigjools | lifeless: I'd need to think about it. I don't recall the use cases for /arch and /series and it might be useful to talk to Ubuntu folks first. | 11:37 |
| wgrant | Do we log who uses pages? | 11:38 |
| lifeless | bigjools: so the thing I really care about is DSP/DSPCache, and AFAICT they are meant to be approx equivalent in UI | 11:38 |
| bigjools | lifeless: I think DSPCache was invented for performance reasons, if we can do without it I'd be ecstatic since the script that generates the cache only runs daily. | 11:38 |
| lifeless | wgrant: effectively, yes. | 11:38 |
| lifeless | bigjools: ok, I will have a fiddle on staging tomorrow | 11:38 |
| lifeless | bigjools: also I have a favour to ask | 11:38 |
| bigjools | lifeless: can I breathe yet? :) | 11:41 |
| lifeless | bigjools: the favour is this: have a look at | 11:41 |
| lifeless | https://lpstats.canonical.com/graphs/OopsEdgeHourly/ | 11:42 |
| lifeless | and | 11:42 |
| lifeless | https://lpstats.canonical.com/graphs/OopsLpnetHourly | 11:42 |
| lifeless | about 70 minutes after mthaddon finishes the timeout CP | 11:42 |
| lifeless | I was going to monitor this myself, but an orthogonal issue prevented the CP until now. | 11:42 |
| bigjools | what are you CPing? | 11:42 |
| lifeless | if it has gone ballistic, roll it back | 11:42 |
| lifeless | bigjools: 1 second timeout drop | 11:42 |
| bigjools | /o\ | 11:43 |
| bigjools | ok | 11:43 |
| bigjools | on soft or hard or both? | 11:43 |
| lifeless | hard | 11:43 |
| lifeless | soft gets tweaked a little to match other reporting tools | 11:43 |
| lifeless | lpnet and edge | 11:43 |
| lifeless | I don't expect a disaster, or I wouldn't be doing it. | 11:44 |
| lifeless | but if it is terrible, we have the tools to undo it rapidly. | 11:44 |
| lifeless | mthaddon: can assist rolling back the timeouts *if* needed. | 11:44 |
| lifeless | bigjools: are you up for this ? | 11:44 |
| bigjools | lifeless: I'll monitor it for an hour or so then I have to go out | 11:45 |
| bigjools | I can pass the baton to someone else | 11:45 |
| lifeless | bigjools: oh, and as an expectation - 50 hard oopses an hour is tolerable | 11:45 |
| lifeless | 200 hard oopses an hour - rollback | 11:46 |
| bigjools | good data point, thanks | 11:46 |
| lifeless | thank you! | 11:46 |
| * lifeless crashes | 11:46 | |
| bigjools | sleep well | 11:46 |
| wgrant | Night lifeless. | 11:46 |
| deryck | Morning, all. | 12:03 |
| allenap | Morning deryck :) | 12:08 |
| === almaisan-away is now known as al-maisan | ||
| jtv | gmb: I have a faint recollection that ProductWithLicenses was one of yours. Is that right? | 12:40 |
| gmb | jtv, No, mot I. | 12:42 |
| gmb | *not | 12:42 |
| jtv | I'll ask bzr then.. | 12:42 |
| * gmb ;unches | 12:42 | |
| * gmb learns to type | 12:42 | |
| jtv | ah, 'twas EdwinGrubbs | 12:43 |
| jtv | EdwinGrubbs: very minor bug in ProductWithLicenses. | 12:44 |
| wgrant | bigjools: http://bazaar.launchpad.net/~wgrant/launchpad/ppa-key-name/revision/11486 | 12:51 |
| bigjools | heh ""Not Celso Providelo"" | 12:55 |
| wgrant | bigjools: So, does it look reasonable? | 13:06 |
| bigjools | I can't think of a reason why it should not land | 13:06 |
| wgrant | OK, I'll propose it then. | 13:06 |
| wgrant | Thanks. | 13:06 |
| bigjools | cheers | 13:06 |
| * bigjools -> lunch | 13:07 | |
| benji | wgrant: I got your script working in a virtualenv with the new launchpadlib; I did have to make some small changes: http://pastebin.ubuntu.com/487228/ | 14:05 |
| benji | I'm going to see if the changes I made indicate bugs in the latest launchpadlib or if they were intentional changes | 14:05 |
| wgrant | Ah, great. | 14:06 |
| benji | speed-wise, in unscientific testing I got no meaningful difference in total run time | 14:06 |
| stub | Is sourcepackage one word or two? I keep forgetting what we prefer? | 14:30 |
| wgrant | It's capitalised as two. | 14:31 |
| wgrant | There have been a few debates around this. | 14:31 |
| === Ursinha-zzz is now known as Ursinha | ||
| === kiko` is now known as kiko | ||
| deryck | adeuring, almost done on your review. Some questions, just to be clear, though.... | 15:02 |
| deryck | adeuring, there is no public api change here, right? | 15:02 |
| adeuring | deryck: yes, the api did not change | 15:02 |
| deryck | adeuring, and concerning the bits of the webservice test you added, the main reason for that test is to verify the app server is serving the librarian file? | 15:03 |
| adeuring | deryck: right | 15:03 |
| deryck | adeuring, so the bits that would fail if this were not true are the X-Powered-By header line? The rest is just setup? | 15:04 |
| adeuring | deryck: yes; thoguh the X-powered-by header is not really important; the point is that "websevice.get(...)" returns the file data | 15:06 |
| adeuring | deryck: a corresponding test some lines up in that test connect to the Librarian in a completely different way to read a publci file | 15:06 |
| deryck | gotcha | 15:07 |
| deryck | adeuring, so what is the final webservice.patch for in the diff? | 15:07 |
| adeuring | deryck: some other tests later in that file need a public bug | 15:08 |
| deryck | adeuring, ah, because it's sample data based. So I think we could expand the doc portion of this doctest to make this clear. | 15:09 |
| adeuring | deryck: i'll add a note | 15:10 |
| deryck | I do realize the webservice tests are kind of a weird page/doc test amalgam, but it wasn't obvious to me. | 15:10 |
| deryck | adeuring, so lifeless concern about this was that by serving files from the app server, we will most certainly have timeout issues now with private attachments via the API? | 15:17 |
| deryck | that was a question, i.e. is that ^^ correct? | 15:18 |
| adeuring | deryck: it could be. But I am not 100% sure: IIRC, the timeout exceptions are raised somewhere "near" the storm layer -- but once we have the LFA ready, the request does not touch DB objects anymore | 15:18 |
| deryck | adeuring, so this wasn't lifeless concern then? | 15:22 |
| deryck | what was? | 15:22 |
| adeuring | deryck: no, as i understood him, these timeouts were his concern | 15:22 |
| adeuring | ..perhaps aside from additional load for the app servers | 15:23 |
| deryck | adeuring, and you're suggesting that you're not sure you agree that timeouts will be an issue? | 15:24 |
| adeuring | deryck: well, we should test that ;) Should be easy: We just need to upload a sufficiemtly large file to the staging librarian and then to download it again via a sufficiently slow connection | 15:24 |
| deryck | adeuring, at any rate, we really don't have another option, short of the "enable retracers in dc" option that we cannot do, right? | 15:25 |
| adeuring | right. and regarind testing. there is even this linux kernel feature to throttle connections, about which I always forget how to use it | 15:26 |
| deryck | adeuring, ok. I'm about ready to r=me this. I asked gary_poster to take a look over my shoulder, just to make sure I'm not missing something. | 15:32 |
| adeuring | deryck: ok, thanks! | 15:32 |
| deryck | sorry for paranoia, but I want this to go in without further problems. | 15:32 |
| deryck | adeuring, you could go ahead and ping losa to cowboy the patch to staging and performance test it, since that is a primary concern from lifeless. | 15:33 |
| adeuring | deryck: ok, though I think testing it tomorrow should fine | 15:33 |
| deryck | adeuring, well, ubuntu beta is today and they need the retracers running. I'd like to end today telling them to point the retracers at edge and go for it, but I want to have confidence this will work. | 15:35 |
| adeuring | deryck: ok | 15:35 |
| adeuring | deryck: could you please add your review to to mp? | 15:48 |
| deryck | adeuring, yes, sorry. | 15:48 |
| adeuring | npmp ;) | 15:48 |
| deryck | adeuring, you're ready to go now. See my comments/reminder about the staging test. | 15:56 |
| adeuring | deryck: thanks! | 15:56 |
| deryck | np! | 15:56 |
| === matsubara is now known as matsubara-lunch | ||
| === Ursinha is now known as Ursinha-lunch | ||
| === salgado is now known as salgado-lunch | ||
| abentley | rockstar, here are the permission changes that are giving me grief: http://pastebin.ubuntu.com/487320/ | 17:03 |
| abentley | rockstar, or lp:~abentley/launchpad/recipe-interfaces | 17:04 |
| === benji is now known as benji-lunch | ||
| === Ursinha-lunch is now known as Ursinha | ||
| === matsubara-lunch is now known as matsubara | ||
| === beuno is now known as beuno-lunch | ||
| === salgado-lunch is now known as salgado | ||
| === benji-lunch is now known as benji | ||
| === deryck is now known as deryck[lunch] | ||
| === gary_poster changed the topic of #launchpad-dev to: Code hosting offline 8.00-9.30 UTC on Friday 3rd September for unexpected hardware maintenance. http://is.gd/eRMxF | Launchpad Development Channel | Performance Tuesday | Week 3 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | ||
| jcsackett | lifeless, you around today yet? | 18:59 |
| === al-maisan is now known as almaisan-away | ||
| === beuno-lunch is now known as beuno | ||
| === deryck[lunch] is now known as deryck | ||
| lifeless | jcsackett: hi | 19:53 |
| jcsackett | lifeless: heya. | 19:53 |
| jcsackett | so, EdwinGrubbs and I have looked into what was plaguing me yesterday. Edwin may have found a fix for the test: http://pastebin.ubuntu.com/487390/ | 19:54 |
| jcsackett | so, we may now be good. when i pinged you earlier, still no luck. | 19:54 |
| jcsackett | i do have the one extra query isolated, if you're interested. | 19:55 |
| lifeless | what is it | 19:55 |
| lifeless | looking at your patch | 19:55 |
| lifeless | the flush+reset would need an explanation at a minimum | 19:55 |
| jcsackett | https://pastebin.canonical.com/36645/ | 19:56 |
| jcsackett | lifeless ^ | 19:56 |
| jcsackett | and yeah, this is without comments; just seeing if it passes in the full suite. | 19:56 |
| lifeless | that looks like _init to me | 19:58 |
| lifeless | what happens is that _init - a storm hook - is called *before* the variables are assigned | 20:01 |
| lifeless | Person._init checks self.teamownerID | 20:01 |
| lifeless | and storm does an on-demand query to populate the attributes | 20:02 |
| * jcsackett nods. | 20:03 | |
| jcsackett | so this problem is already fix-committed, right? | 20:03 |
| lifeless | the storm bug is yes | 20:04 |
| lifeless | fixing it though, uncovered a nastier one | 20:04 |
| lifeless | https://bugs.edge.launchpad.net/storm/+bug/620615 | 20:04 |
| _mup_ | Bug #620615: please contact the developers <Storm:In Progress by niemeyer> <https://launchpad.net/bugs/620615> | 20:04 |
| lifeless | bug 628762 | 20:05 |
| _mup_ | Bug #628762: propertycache adaption failures in test suite <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/628762> | 20:05 |
| lifeless | allenap: jfdi ;) | 20:05 |
| lifeless | oops graphs are looking good | 20:09 |
| jcsackett | lifeless: that does look unpleasant. | 20:12 |
| lifeless | slight increase on prod, edge seems unaffected \o/ | 20:12 |
| lifeless | james_w`: ping | 20:28 |
| james_w` | hi lifeless | 20:29 |
| lifeless | hiya | 20:29 |
| lifeless | james_w`: so, swallowing eh | 20:32 |
| james_w` | yeah, I don't want to trigger exceptions | 20:33 |
| lifeless | I was starting to reply | 20:33 |
| lifeless | I figured I'd chat | 20:33 |
| james_w` | usually means that something has gone wrong, and so I don't want my tests to pass | 20:33 |
| lifeless | cleanUp is used from two locations | 20:34 |
| lifeless | as a cleanUp | 20:34 |
| lifeless | and from reset | 20:35 |
| lifeless | now, cleanUp isn't nestable | 20:35 |
| lifeless | I mean TestCase.addCleanUp - | 20:35 |
| lifeless | cleanups are regular work-or-throw functions | 20:35 |
| lifeless | they can't represent 2 exceptions | 20:35 |
| lifeless | what you have to do is to have two cleanups | 20:36 |
| lifeless | so, if TestCase.addCleanUp could in some way handle functions that want to blow up many times | 20:37 |
| lifeless | we could change the fixture protocol to match | 20:37 |
| lifeless | hmm | 20:38 |
| james_w` | why not mandate that cleanups can be run multiple times? | 20:38 |
| lifeless | lets say we make TestCase.addCleanUpExt | 20:38 |
| lifeless | it takes things that yield exceptions | 20:38 |
| lifeless | and has an adapter that is used by addCleanUp | 20:38 |
| lifeless | to turn a regular function into a yields-exc-info | 20:38 |
| lifeless | james_w`: because for existing trivial cleanups its undefined whether thats safe or not | 20:39 |
| lifeless | james_w`: and for many, it won't be, or will raise the same way every time | 20:39 |
| lifeless | james_w`: its also less general: it only helps with cleanUps that happen to be a loop-on-a-list | 20:40 |
| lifeless | mmm, now what about reset | 20:40 |
| lifeless | it will need to signal exceptions in the same way as cleanup | 20:41 |
| lifeless | (whether thats raise a MultiException with a list of exc_infos, or return an iterable, I dunno yet) | 20:42 |
| lifeless | it also needs to signal 'setUp failed' | 20:42 |
| lifeless | probably any different exception would be sufficient for that - e.g. user caused exceptions are fine. | 20:43 |
| lifeless | testresources optimisation should stop I guess if it sees reset fail, and skip all the tests with that fixture | 20:44 |
| leonardr | lifeless, can you tell me if http://pastebin.ubuntu.com/487402/ is ec2 being flaky or a real problem? i can't duplicate it locally and it seems to have nothing to do with my branch | 20:44 |
| lifeless | leonardr: line 18 is that storm bug I was discussing with jcsackett | 20:45 |
| lifeless | Person._init being called to early | 20:45 |
| leonardr | i see | 20:45 |
| lifeless | the fingerprint is the id=%s limit 1 call | 20:45 |
| lifeless | james_w`: what do you think of my sketching ? | 20:45 |
| sabdfl | hi folks | 20:46 |
| james_w` | lifeless: sounds reasonable to me | 20:46 |
| lifeless | hi sabdfl | 20:46 |
| sabdfl | does the bug description deliberately specify a different font-face, or is that an bug? | 20:46 |
| sabdfl | for example, https://bugs.edge.launchpad.net/indicator-network/+bug/621168 | 20:46 |
| _mup_ | Bug #621168: Connection to encrypted hidden network fails <Connection Manager:Confirmed> <Network Menu:Confirmed> <https://launchpad.net/bugs/621168> | 20:46 |
| sabdfl | new font is looking pretty good generally, mono version on the way | 20:46 |
| lifeless | deryck: this ones for you I think? | 20:47 |
| deryck | ah, yes. | 20:47 |
| deryck | sabdfl, it's a bug. It's falling back to the YUI css for some non-obvious reason. | 20:48 |
| deryck | sinzui and I worked on this a bit already. | 20:48 |
| beuno | deryck, the YUI CSS is more specific than the one in LP, so it wins | 20:48 |
| sabdfl | aha. low importance, i was just curious, but please do file it so it doesn't get lost | 20:48 |
| deryck | sabdfl, will do. | 20:48 |
| lifeless | jcsackett: with your query count bug | 20:49 |
| sabdfl | thanks deryck | 20:49 |
| lifeless | jcsackett: please use | 20:49 |
| lifeless | Equals(count), Equals(count + 1) | 20:49 |
| lifeless | jcsackett: so that we don't have to simulaneously change the tests when we get storm 0.18 | 20:49 |
| lifeless | leonardr: changing the test you're hitting to be similar to the pastebin changes jcsackett pasted before - with a MatchesAny(Equals(n), Equals(n+1)) and a flush + reset should work around it. | 20:51 |
| lifeless | leonardr: I will be very happy when storm 0.18 arrives ;)( | 20:51 |
| deryck | beuno, thanks for the "more specific" pointer | 20:51 |
| beuno | deryck, also, a different font may break the widget layout. You had worked on making them fluid rather than fixed, may of fixed the font-face problem | 20:52 |
| beuno | deryck, we may want to make lazr-js be less specific about CSS, so it doesn't end up being a battle between the projects' CSS and the widget | 20:53 |
| deryck | beuno, IIRC, the font had to be specified to make it re-usable. Changing the font will force changes to the widget, whether it happens in lazr-js on lp's use of the widget. | 20:56 |
| beuno | deryck, right. I was thinking of just specifying the font more generically on the DOM elements | 20:57 |
| beuno | vaguely enough that it applies by default, but can still be easily overriden | 20:57 |
| deryck | ah, so it does win in the specificity battle. | 20:58 |
| beuno | right | 20:58 |
| beuno | otherwise we end up having to slap !important on | 20:59 |
| beuno | or have crazy CSS rules | 20:59 |
| beuno | either way, LP still needs to catch-up with YUI 3.1.2 | 20:59 |
| beuno | so you will probably have to hack it in LP first | 20:59 |
| beuno | this was more of a for-the-future thought | 21:00 |
| deryck | right | 21:01 |
| deryck | thanks, I appreciate it. | 21:01 |
| deryck | and now we have bug 629063 | 21:02 |
| _mup_ | Bug #629063: Description editing widget should use the Ubuntu font <javascript> <ui> <Launchpad Bazaar Integration:New> <LAZR Javascript Library:New> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/629063> | 21:02 |
| leonardr | lifeless: do you want me to make that change in this branch, or is it ok to land my branch as is? ie. are you just telling me how to work around it for now? | 21:02 |
| lifeless | your branch won't land because you're tickling the storm cache enough that you'll see that extra query. I don't know what your branch changes :) | 21:03 |
| lifeless | so you need to make that change in your branch, so you can land it. I'll happily review the workaround for you | 21:04 |
| === dobey_ is now known as dobey | ||
| === abentley_ is now known as abentley | ||
| leonardr | ok | 21:35 |
| leonardr | lifeless: all right, see how http://pastebin.ubuntu.com/487469/ grabs you | 22:04 |
| lifeless | leonardr: that should be enough, yes. give it a sping - with XXX around the flush and reset saying 'bug 619017' please | 22:08 |
| _mup_ | Bug #619017: __storm_loaded__ called on empty object <Storm:Fix Committed by therve> <https://launchpad.net/bugs/619017> | 22:08 |
| === matsubara is now known as matsubara-afk | ||
| === salgado is now known as salgado-afk | ||
| bdmurray | Is there anyway to change an attribute in a zope macro? | 23:14 |
| lifeless | bdmurray: what do you mean? | 23:20 |
| bdmurray | lifeless: lib/lp/bugs/templates/bug-portlet-dupe-subscribers-content.pt uses subscriber-row from bug-portlet-subscribers-content and I want to chagne the title attribute in subscriber-row | 23:20 |
| lifeless | hmmm | 23:21 |
| lifeless | I don't tihnk so | 23:21 |
| lifeless | it also feels weird to me | 23:21 |
| lifeless | losa ping | 23:23 |
| lifeless | https://staging.launchpad.net/successful-updates.txt claims 9737 | 23:23 |
| lifeless | but the running instance is 9710 | 23:23 |
| lifeless | spm: when you start, I know you want to crawl into a hole and hide; I have two small things batched up so that they don't interupt. | 23:44 |
| lifeless | spm: one is the staging thing above | 23:44 |
| lifeless | spm: the other is can you please put the canonical losa tag whatever it is onto https://bugs.edge.launchpad.net/launchpad-foundations/+bug/629139 | 23:45 |
| _mup_ | Bug #629139: cannot make production config changes <Launchpad Foundations:New> <https://launchpad.net/bugs/629139> | 23:45 |
| Ursinha | jcsackett, hello | 23:45 |
| Ursinha | jcsackett, just saw your comment on bug 623428 about the bug status change, I'm investigating | 23:45 |
| _mup_ | Bug #623428: Don't get ZODB 3.10 by installing Zope2.13a <Zope 2:Triaged> <https://launchpad.net/bugs/623428> | 23:45 |
| Ursinha | hmmm | 23:46 |
| Ursinha | bug 623408 | 23:46 |
| _mup_ | Bug #623408: Offiical_* booleans must be deprecated in favor of usage enums <bridging-the-gap> <Launchpad Registry:In Progress by jcsackett> <https://launchpad.net/bugs/623408> | 23:46 |
| Ursinha | there you go :) | 23:46 |
| Ursinha | it's not supposed to change the bug status, just tag it as untestable | 23:47 |
| Ursinha | because of the new merge workflow | 23:47 |
| Ursinha | I'm taking care of it | 23:48 |
| jcsackett | Ursinha: thanks. | 23:50 |
| mbarnett | lifeless: spm: i have tagged that bug. | 23:51 |
| lifeless | mbarnett: oh awesome, thanks | 23:51 |
| lifeless | mbarnett: I figured you'd EOD when the ping fell into silence :) | 23:51 |
| mbarnett | lifeless: we had an austin meetup this afternoon.. was in transit home | 23:51 |
| lifeless | meetup? | 23:52 |
| mbarnett | a bunch of canonical employees who live in the area worked from a coffee shop together for a couple hours. | 23:52 |
| lifeless | nice | 23:52 |
| mbarnett | yeah, i hadn't put a face on a couple people yet, so it was great. | 23:52 |
| lifeless | I met the closest canonical staffer to me, in prague :P | 23:53 |
| lifeless | (we both live in NZ) | 23:54 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
