[05:48] <stub> lifeless: So my branch had timeout as the timeout, and your branch has timeout as the goal time for the 99% calculations, and histogram_width as the timeout
[05:49] <lifeless> yeah, my first branhc, as you said, was in poor shape
[05:49] <lifeless> the incremental changes are twofold
[05:49] <lifeless> firstly, there is an existing bug in the report: the stddev is being mangled by the clipping of the data
[05:49] <lifeless> likewise the variange
[05:50] <lifeless> so I changed that, and having fixed that I then had the abilitity to do the histogram columns independently, and I figures that seeing the shape beyond the timeout would be useful
[05:54] <stub> The logic was that if a page takes longer than the hard timeout, that is irrelevant. It doesn't render so ends up in the 'fail' column at the hard timeout, giving a nice 'kick' at the end.
[05:55] <stub> I would have labelled the last column 'timeout' if the graphing tool let me.
[05:56] <stub> I suspect the only correct answer though would be to report a timeout%, and have the stats calculated non-timeout pages.
[05:58] <lifeless> stub: well, what I've done willso have a kick
[05:58] <lifeless> and - we could do a kick at the exact timeout column if you wanted
[05:58] <lifeless> but I wanted the stddev to be accurate, because that tells us the 99% figure
[05:59] <lifeless> stub: perhaps you could give it a spin and we can see what it looks like ?
[06:03] <stub> lifeless: I don't like that we are growing a separate list for capped request times. I think we want to create this on the fly using a generator, and use numpy.fromiter instead of numpy.asarray to construct the array.
[06:04] <lifeless> stub: ok
[06:04] <lifeless> stub: is that a memory overhead concern ?
[06:04] <stub> We have nearly 500 of these things in ram containing over 10 million 32bit objects, so yeah
[06:05] <stub> But I pulled out the spool-to-disk version, as it seemed to be better to hit swap than open 500 temporary files. That was implemented as I thought I was crashing devpad, but it was just hardware issues.
[06:06] <stub> (monthly reports will get lots of requests, when devpad stays up long enough to actually generate one...)
[06:07] <stub> I'll give it a run as is - memory shouldn't be an issue for a daily run.
[06:09] <lifeless> I'm happy to change that for you
[06:09] <lifeless> I considered doing it on the fly but it seemed slower ;P
[06:19] <stub> It is slower, but that is a minor part of our runtime. Parsing the logs takes time. Generating the stats is pretty much instantaneous.
[06:39] <lifeless> ok
[06:39] <lifeless> so - lets see how it looks, and I'll look up fromiter
[06:46] <stub> fromiter(iterable, numpy.float32, len(self.request_times)) - we know the length so can use the third parameter
[06:47] <lifeless> and iterable is an adapter to call min ?
[06:48] <stub> https://devpad.canonical.com/~stub/ppr/edge/latest-daily-timeout-candidates.html is done. lpnet still generating
[06:48] <stub> lifeless: I imagine it is a generator that returns the time capped to the timeout
[06:49] <lifeless> RIGHT
[06:49] <lifeless> bah
[06:49] <lifeless> Right
[06:49] <lifeless> thats what I thought too
[06:53] <stub> Bwaha... LibraryFileAlias:StreamOrRedirectLibraryFileAliasView is a mess without the cap ;)
[06:53] <stub> But I guess that is the truth.
[06:53] <lifeless> yup
[06:54] <lifeless> do think these changes are ok then ?
[06:56] <lifeless> I've pushed up a (hopefully working) fromiter version
[07:04] <stub> That all looks good. I've kicked off another run with the new version
[07:05] <lifeless> thanks
[10:56] <jelmer> noodles775: Could I add a branch to your queue?
[10:56] <noodles775> jelmer: please do :)
[10:57] <jelmer> noodles775: The MP is https://code.edge.launchpad.net/~jelmer/launchpad/613468-xb-ppa-db/+merge/32307
[10:57] <jelmer> noodles775, thanks!
[11:00] <StevenK> noodles775: O hai, can you re-re-review https://code.edge.launchpad.net/~stevenk/launchpad/switch-ifp-to-unittests/+merge/32073 ?
[11:01] <noodles775> StevenK: indeed :)
[11:33] <noodles775> StevenK: I'm not trying to make your life hard, and hate dragging this review on, but I don't understand why test_create_builds does so much more than what the name implies... do you have time to mumble and discuss it briefly?
[11:44] <StevenK> noodles775: Yes, doing so in a sec
[11:45] <StevenK> noodles775: But it doesn't? It asserts the *source* for 0.1-2 exists, and then creates a build for it?
[12:15] <StevenK> noodles775: The diff does not appear to have been updated yet, but the MP is ready for your re-re-re-review.
[12:15] <noodles775> Thanks StevenK
[12:38] <StevenK> noodles775: No news is good news, or I'm in the queue?
[12:42] <noodles775> StevenK: looking now (had to merge as the MP hadn't updated)
[12:42] <StevenK> Yes, I'm not sure what is going on there
[12:50] <jelmer> noodles775: Thanks for the review!
[12:51] <noodles775> np
[12:57] <noodles775> StevenK: looks good. I've just highlighted what seem to be a few inconsistencies in the comments, but otherwise r=me: http://pastebin.ubuntu.com/476903/
[13:07] <wgrant> jelmer: Hm, why are the extra fields JSON?
[13:07] <wgrant> They're known to be string pairs.
[13:08] <wgrant> It seems like an extra table would do fine, leave the table smaller, and make it possible to query them in future.
[13:08] <jelmer> wgrant: consistency with other fields in the database, where we also use JSON.
[13:08] <wgrant> Are there any, apart from on Job?
[13:08] <StevenK> noodles775: Planning to commit http://pastebin.ubuntu.com/476907/
[13:09] <jelmer> wgrant: No, just job actually.
[13:09] <wgrant> jelmer: And it's like that so that custom job types can store different data there.
[13:09] <wgrant> I don't see why JSON should be used here, and not a separate table.
[13:17] <jelmer> wgrant: simplicity; I don't see why we would need to query that field. if we would go the way of a separate table we might want to use it for some of the other control fields as well
[13:17] <jelmer> which might not be a bad idea, just out of scope for me at the moment.
[13:31] <StevenK> gmb: O hai! Given noodles775's has Approved the MP at https://code.edge.launchpad.net/~stevenk/launchpad/switch-ifp-to-unittests/+merge/32073; can haz moar review?
[13:39] <gmb> StevenK, If you stop asking in lolspeak, yes. ;)
[13:40] <gmb> In fact, I think noodles775's review is far more comprehensive than mine (hurrah for context)
[13:40] <gmb> So I'll just mark it approved.
[13:40] <StevenK> gmb: Okay, cool
[15:19] <james_w> jml: hi, would you be willing to do an incremental review of https://code.edge.launchpad.net/~james-w/launchpad/no-more-sampledata-2/+merge/31891 ?
[15:19] <jml> james_w, yes. I'll look at it shortly
[15:19] <james_w> thanks
[16:41] <leonardr> can i get someone to review my really simple launchpadlib branch? i've been in the queue since yesterday
[16:41] <bac> hey salgado, can you do a shipit review for me?
[16:42] <leonardr> https://code.edge.launchpad.net/~leonardr/launchpadlib/616055/+merge/32352
[16:42] <salgado> bac, sure!
[16:42] <bac> salgado:  https://code.edge.launchpad.net/~bac/shipit/bug-616059/+merge/32457
[16:42] <salgado> bac, not allowed
[16:43] <bac> salgado:  try now
[16:43] <salgado> bac, while you subscribe me to your branch, subscribe launchpad-pqm as well, or else you won't be able to land it later
[16:44] <bac> salgado:  thanks for the tip!
[16:44] <leonardr> bac, while you wait, will you review my branch please?
[16:44] <bac> leonardr:  sure
[16:44]  * bac looks
[16:45] <leonardr> thanks
[16:47] <salgado> bac, did you consider doing that for canonical.* instead of canonical.launchpad.*?
[16:48] <bac> salgado:  yeah, sort of.  i thought lp was sufficient.  you think i chose wrong?
[16:49] <salgado> bac, I'd have done for canonical.* but lp.* and canonical.launchpad.* is certainly where we do most of our changes/refactorings, so it's no big deal
[16:57] <bac> salgado:  yeah, you would've...but you're smarter than me!
[17:02] <salgado> bac, If that was the case I'd have done this change long ago. ;)
[17:02] <bac> salgado:  yeah, if you weren't such a slacker
[17:10] <salgado> that is certainly the case
[17:11] <bac> leonardr:  r=bac.
[17:11] <salgado> bac, did you get my review?  you might want to get someone from ISD to have a look as well, as they own shipit now
[17:13] <bac> salgado:  i see it now.  did you mean to have it just be a comment?
[17:13] <salgado> no, just approved using the web UI, but it'd be good to at least let them know about the change before it lands
[17:14] <bac> salgado:  i'll talk to achuni or someone else
[17:14] <bac> salgado:  thanks for catching those issues
[17:14] <bac> i renamed to sd_setUp b/c there is another setUp being imported in that file
[17:15] <salgado> np.  thank you for doing this change
[17:15] <salgado> fair enough
[17:15] <bac> sd is cryptic for 'systemdocs'
[17:44] <leonardr> benji-lunch, it looks like we're missing a review for lazr.restfulclient/total_size_link
[18:00] <leonardr> noodles775, the branch i'd like reviewed is https://code.edge.launchpad.net/~benji/lazr.restfulclient/total_size_link/+merge/32472
[18:01] <leonardr> it's benji's branch but most of the code is mine
[18:01] <leonardr> just fyi, when you get to me
[18:07] <jml> james_w, done
[18:25] <james_w> thanks jml
[18:26] <jml> np
[18:42] <leonardr> noodles775, i'd also like to add this to the queue: https://code.edge.launchpad.net/~leonardr/launchpad/new-lazr.restful-and-friends/+merge/32474
[18:48] <jelmer> leonardr: I think Michael has signed off for the day.
[18:48] <leonardr> well, damn
[18:48] <leonardr> bac, are you in the mood to do more reviews?
[18:49] <bac> leonardr:  i can do one.
[18:49] <jelmer> rockstar should be around later, or is he on leave?
[18:49]  * bac has a full day of reviews tomorrow
[18:49] <rockstar> Oh snap, forgot it was Thursday.
[18:50] <leonardr> bac: ok, rockstar will take care of these
[18:51] <bac> thanks leonardr and rockstar
[18:59] <rockstar> leonardr, apparently there are two review to do for you.  Where's the other one?
[18:59] <rockstar> (I already did new-lazr.restful-and-friends)
[18:59] <leonardr> rockstar: ok, the other one is https://code.edge.launchpad.net/~benji/lazr.restfulclient/total_size_link/+merge/32472
[19:05] <bac> rockstar may i add a small branch to your queue?
[19:05] <rockstar> bac, sure.
[19:07] <rockstar> bac, ftr, I'm going to clear out all the +activereviews today, so you won't have a huge queue tomorrow.
[19:12] <bac> rockstar:  that would be swell.  do leave me some, though!
[19:30] <bac> leonardr:  what is the status of this branch: https://code.edge.launchpad.net/~leonardr/launchpad/grant-permissions-oauth/+merge/29425
[19:30] <leonardr> bac: i got sidetracked by an emergency project before i could land it, and i don't want to land it on its own because it's part of another large project
[19:31] <leonardr> so, the status is "wait and see"
[19:31] <bac> leonardr:  ok
[21:33] <bac> thanks for the review rockstar
[22:25] <jelmer> rockstar: I've fixed the conflicts in my branch; any chance I could add it back into your queue?
[22:26] <rockstar> jelmer, if it's in +activereviews, I'll get to it today.
[22:26]  * rockstar is firefighting currently.
[22:26] <jelmer> rockstar: great, thanks
[22:26] <jelmer> rockstar: oh, ok
[22:43] <thumper> jelmer: are you going to propose the bzr-git update branch?
[22:44] <jelmer> thumper: yes, it should fix quite a few of the broken imports
[22:44] <thumper> cool
[22:44] <thumper> there is someone trying the chicken import every few days
[22:45] <jelmer> thumper: yeah, he pinged me on IRC today about it.
[22:45] <jelmer> thumper: How hard it is to get the new bzr-git deployed without waiting for the next lp release?
[22:45] <thumper> jelmer: very trivial
[22:45] <thumper> jelmer: we land it, test it on staging
[22:46] <thumper> jelmer: then CP it to the production branch
[22:46] <thumper> jelmer: and get it rolled out
[22:46] <thumper> so lag time is 3 x ec2 test run + staging rollout lag + production rollout lag
[22:47] <thumper> add in test time, and getting approval, both of which should be pretty short
[22:47] <jelmer> ok
[23:20] <jelmer> thumper: https://code.edge.launchpad.net/~jelmer/launchpad/update-bzr-git/+merge/32527
[23:35] <thumper> jelmer: done
[23:36] <jelmer> thumper: bug 497428 appears to be fixed, scilab's import is working. Can I mark it fixreleased in launchpad-code?
[23:36] <_mup_> Bug #497428: Import on a large git repository is failing <code-import> <Bazaar Git Plugin:Fix Released> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/497428>
[23:36] <thumper> jelmer: yep
[23:47] <jelmer> thumper: perhaps we should increase the number of revisions imported per step with bzr-svn
[23:47] <thumper> jelmer: I'm up for that
[23:47] <thumper> jelmer: it is just a config change I think
[23:47] <jelmer> thumper: the KDE branches appear to be working now, but they're spending 15 minutes trying to figure out what 1k revisions to fetch and then 2 minutes doing the actual work.
[23:48] <thumper> jelmer: 10k seem reasonable?
[23:48] <jelmer> thumper: Yeah, definitely.
[23:49] <jelmer> thumper: Could the git batch size also be increased perhaps?
[23:49] <thumper> jelmer: yep
[23:49] <jelmer> I think it's at 10k now but we should be able to handle 30k at least
[23:49] <thumper> jelmer: sure, sounds reasonable
[23:57] <mwhudson> i still wonder if it should be time-limited not rev-count limited
[23:59] <jelmer> mwhudson: time and memory limited in that case, but yeah, that sounds reasonable.