[00:02] <mwhudson> i predict this new will make jml unhappy
[00:03] <wgrant> Which new?
[00:03] <wgrant> The restructure?
[00:07] <poolie> spm it turns out, just fyi, that qastaging has no mx record
[00:07] <poolie> i will send a mail to the list
[00:07] <spm> ah. k
[00:25] <wgrant> flacoste: Thanks for forwarding that to the list.
[00:30] <lifeless> wgrant: hey
[00:31] <lifeless> wgrant: so, I don't know if you noticed, but the archive:+index timeouts are going up daily. I don't think this is random.
[00:32] <wgrant> lifeless: That's odd.
[00:32] <wgrant> (sorry, I've been distracted with other stuff the last couple of days)
[00:32] <lifeless> doesn't deeply worry me
[00:32] <lifeless> its a little concerning, and I want to be worry free on my holiday
[00:33] <lifeless> 21st:      387 / 6695  Archive:+index
[00:33] <wgrant> Hmmm.
[00:33] <lifeless> bah
[00:33] <lifeless> 1st:      387 / 6695  Archive:+index
[00:33] <lifeless> 30th:      324 / 7217  Archive:+index
[00:33] <lifeless> 29th:
[00:34] <lifeless>      287 / 6525  Archive:+index
[00:34] <wgrant> The other pageids aren't showing a similar trend?
[00:34] <wgrant> That's a pretty massive escalation.
[00:34] <lifeless> 29th:
[00:34] <lifeless>      287 / 6525  Archive:+index
[00:34] <lifeless>      170 /  390  BugTask:+index
[00:34] <lifeless>       43 /  260  Distribution:+bugs
[00:34] <wgrant> If it's the only pageid doing it, that's more 'deeply concerning' than 'a little concerning'.
[00:34] <lifeless> 28th:
[00:34] <lifeless>      260 / 6568  Archive:+index
[00:34] <lifeless>      142 /  301  BugTask:+index
[00:34] <lifeless>       26 /  290  Distribution:+bugs
[00:34] <lifeless> 27th:
[00:35] <lifeless>      115 / 5419  Archive:+index
[00:35] <lifeless>       92 /  233  BugTask:+index
[00:35] <lifeless>       23 /  244  Distribution:+bugs
[00:35] <wgrant> Uh.
[00:35] <wgrant> I think we have a general problem :)
[00:35] <lifeless> possibly
[00:35] <lifeless> 30th:
[00:35] <lifeless>      324 / 7217  Archive:+index
[00:35] <lifeless>      148 /  451  BugTask:+index
[00:35] <lifeless>       36 /  150  ProjectGroupSet:CollectionResource:#project_groups
[00:36] <lifeless>      19 /  296  Distribution:+bugs
[00:36] <lifeless>   1st:
[00:36] <lifeless>      387 / 6695  Archive:+index
[00:36] <lifeless>      155 /  482  BugTask:+index
[00:36] <lifeless>      126 /  226  Question:+index
[00:36] <lifeless>       49 /  104  RootObject:+login
[00:36] <lifeless>       39 /  247  Distribution:+bugs
[00:36] <lifeless> These are all genuinely problematic pages
[00:36] <lifeless> so I'm not inclined to shout fire yet
[00:36] <lifeless> rather fix the pages.
[00:37] <lifeless> and the worst.. well you know the worst.
[00:37] <lifeless> it is, on its own, 25% of the timeouts on the site.
[00:38] <lifeless> besides which
[00:38] <lifeless> I know flacoste will be looking at these reports and biting his teeth in my absence ;)
[00:38] <wgrant> Heh.
[00:39] <wgrant> Ah, I remember why I didn't get to it last time. download-cache took forever to upgrade.
[00:39] <wgrant> It's done now, so the appserver might finally start...
[00:39] <lifeless> pillarname ||  963021.70 tuples/sec
[00:39] <lifeless>                                   branch ||  443538.40 tuples/sec
[00:39] <wgrant> WTF?
[00:39] <lifeless> archive ||   41660.33 tuples/sec
[00:40] <lifeless> 28th
[00:40] <lifeless>                               pillarname ||  825961.90 tuples/sec
[00:40] <lifeless>                                   branch ||  463744.27 tuples/sec
[00:40] <lifeless>                               sl_confirm ||  133113.62 tuples/sec
[00:40] <lifeless>           binarypackagepublishinghistory ||   49345.86 tuples/sec
[00:40] <lifeless>                                  archive ||   33620.98 tuples/sec
[00:40] <lifeless>                          shippingrequest ||   24934.80 tuples/sec
[00:40] <lifeless> 1st:
[00:40] <lifeless>                            pillarname ||  827065.04 tuples/sec
[00:40] <lifeless>                                   branch ||  469628.31 tuples/sec
[00:40] <lifeless>                               sl_confirm ||  130679.92 tuples/sec
[00:40] <lifeless>           binarypackagepublishinghistory ||   53159.91 tuples/sec
[00:40] <lifeless>                                  archive ||   40228.91 tuples/sec
[00:40] <lifeless>                        sourcepackagename ||   24256.23 tuples/sec
[00:41] <lifeless> not hugely different on a day to day basis
[00:41] <lifeless> it may be that traffic spike that I called out on the list causing trouble
[00:41] <wgrant> 'the list' being the internal one, or have I missed something?
[00:48] <lifeless> wgrant: uhm
[00:48] <lifeless> checking
[00:54] <wgrant> lifeless: Aw, is that really my first? :(
[00:54] <lifeless> wgrant: no :)
[01:09] <mwhudson> something must be sequentially scanning branch to get numbers like that
[01:09] <mwhudson> i'd love to know what
[01:10] <mwhudson> well not 'must' i guess
[01:10] <mwhudson> seems very likely though
[01:13] <wgrant> And Julian doesn't think I'm crazy :D
[01:34] <abentley> poolie: Any progress with QA for r12008?
[01:34] <poolie> abentley: hi, is that the dkim and gpg bugfixes?
[01:34] <abentley> poolie: yes.
[01:34] <poolie> i presume yes, as that's the only one i landed
[01:35] <poolie> it turns out qastaging doesn't receive mail
[01:35] <poolie> it doesn't even have an mx
[01:35] <poolie> that's where i'm up to
[01:35] <poolie> my next thing is to send mail about perhaps turning it on
[01:35] <poolie> it may take too long to do it
[01:35] <poolie> i mean, it may take a while to turn it on and get it working, and that could cause rollouts to stall for too long
[01:36] <lifeless> poolie: does your patch make anything worse, do you think?
[01:36] <poolie> perhaps there's a better approach?
[01:36] <abentley> poolie: normally, I'd suggest using staging instead, then.  But unfortunately, we are doing a full db restore in order to ensure we don't run overtime.
[01:36] <lifeless> poolie: and if so, can it be disabled easily?
[01:36] <poolie> i thought that qastaging would be like staging in this regard (ie reads mail, doesn't send it)
[01:36] <lifeless> abentley: Will my subunit db patch be in the rollout, do you think?
[01:36] <poolie> but it's not
[01:36] <poolie> lifeless: it changes existing code
[01:36] <abentley> lifeless: Yes, that's the plan.
[01:36] <poolie> so, it's possible it will cause a regression
[01:36] <lifeless> abentley: \o/
[01:36] <poolie> and it's not isolated by a flag
[01:37] <abentley> poolie: I think qastaging is still being configured.
[01:38] <poolie> so if it does turn out to break anything that was previously working, i think we would have to roll back or reverse-cherrypick or i guess live with it until a new fix is landed :/
[01:38] <wgrant> poolie: It's not on staging yet either?
[01:38] <lifeless> poolie: db deploy - we can't roll those back.
[01:38] <abentley> poolie: like it doesn't have import slaves or a build farm at the moment.
[01:39] <abentley> wgrant: It would be, but staging is doing a full restore ATM.
[01:39] <wgrant> abentley: Yay :(
[01:40] <poolie> i'll send mail first at least
[01:41] <poolie> should we/do we have a bug asking for real-time log feeds?
[01:42] <abentley> poolie: do you mind testing it in ~24h?
[01:42] <poolie> do you mean <24h or >24h?
[01:43] <poolie> i mean, do you mean "will you please test it on saturday au/sydney?"
[01:43] <abentley> poolie: that is what I mean.  If that's okay with you, it's probably the simplest solution.
[01:44] <poolie> and you mean test it on staging?
[01:44] <abentley> poolie: yes.
[01:44] <poolie> that's ok with me
[01:44] <abentley> poolie: Thanks.
[01:45] <poolie> is there any more specific estimate when it'll be live there?
[01:45] <poolie> just trying to work out if i can do it first thing saturday or something (like 20h from now)
[01:47] <abentley> Chex: Do you have an estimate when the staging update will complete?
[01:50] <poolie> abentley: i might also work out/document/improve testing it locally
[01:50] <poolie> manually testing it locally, i mean
[01:50] <poolie> but i think checking it in the dc would be really good too
[01:51] <poolie> there are enough different moving parts
[01:52] <abentley> poolie: Yes.  We don't normally consider it QAed until it's tested in the datacentre.  And I know our mail handling can be a bit odd, so I think testing it on staging is really worthwhile.
[01:53] <poolie> me too
[01:53] <poolie> thanks for following up on it
[01:53] <abentley> poolie: np
[01:56] <lifeless> wgrant: so, please look at that branch :)
[01:56] <lifeless> ciao
[02:24] <poolie> hey, can someone help me work out how to land max's loggerhead changes
[02:24] <poolie> is it enough to just merge them to the relevant branch?
[02:24] <poolie> spm? wallyworld?
[02:54] <spm> poolie: I'm not sure at all tbh
[03:04] <maxb> poolie: Don't you just PQM them onto lp:~launchpad-pqm/loggerhead/devel, and then update utilities/sourcedeps.conf?
[03:05] <poolie> i think so; i'll try it later
[03:15] <poolie> are one-letter project names banned?
[03:17] <wgrant> Hmm, are the appservers overloaded?
[03:17] <wgrant> I'm getting truncated responses again.
[03:17] <wgrant> Less frequently than last time, though.
[03:21] <spm> a couple are, yeah
[03:23] <spm> https://lpstats.canonical.com/graphs/AppServerRequestLpnet/ wheeee
[03:47] <spm> wgrant: with any luck, things should be better. looks like a buggy script was being rude.
[03:54] <wgrant> spm: Thanks.
[03:55] <spm> yeah the # requests graph is dropping off due to our heavy hand of stabbing application
[04:24] <cody-somerville> wgrant, Do you know much about SQL indexes?
[04:24] <wgrant> cody-somerville: Some. What's the issue?
[04:26] <cody-somerville> wgrant, I'm wondering if an index could be used to make finding the latest of something that relates to something else faster - ie. lets say finding the latest build for a project in a CI.
[04:27] <wgrant> cody-somerville: Of course.
[04:28] <cody-somerville> Would just creating an index on say the finished_at field of the buildresult table be sufficient?
[04:28] <cody-somerville> or would an index on the foreign key to the project and the finished_at field be better?
[04:29] <wgrant> Probably the latter.
[04:30] <spm> be aware that adding indexes can have major negative performance impacts. if you're looking to modify LP stuff, I'd strongly urge you to check with stub, and at least get some explain's happening to show if an index *may* help.
[06:00] <wgrant> launchpad.dev seems to be just about unusable for me.
[06:00] <wgrant> It has to grab several hundred YUI files on every page load...
[06:02] <spm> O_O
[06:02] <wgrant> Yeah, more than 400 YUI JS files are included in each page's <head>. This seems questionable.
[06:04] <StevenK> wgrant: Whereas, usually, launchpad.dev is *fast*
[06:04] <wgrant> StevenK: I think it would be almost acceptably fast if the JS was all in one file like it is on prod.
[06:04] <StevenK> I thought the Makefile did combine-css?
[06:04] <wgrant> The main issue is that only a single request can be processed at a time.
[06:04] <StevenK> Oh, wait
[06:04] <StevenK> I'm a bozo
[06:05] <StevenK> Doesn't lazr-js do that for us?
[06:05] <wgrant> The dev appserver doesn't use the combined launchpad.js, even if it has been built.
[06:06] <wgrant> I wonder which config key it is.
[06:06] <wgrant> Ah, devmode.
[06:07] <wgrant> OMG so fast
[06:11] <wgrant> There's also a disturbing number of 404 OOPSes that have just shown up in the last few minutes.
[06:11] <wgrant> Like 3000 of them.
[06:12] <wgrant> I think something might be wrong with the lazr-js upgrade :/
[06:17] <wgrant> StevenK: Could you pastebin the query log from OOPS-1780G1265 for me?
[06:17] <StevenK> No, you can bloody well wait 10 days
[06:17] <wgrant> Heh.
[06:17] <wgrant> But IS probably won't get around to creating my account for years :P
[06:18] <StevenK> I doubt it
[06:18] <StevenK> wgrant: http://pastebin.ubuntu.com/539296/
[06:18] <StevenK> Oh sigh, that's the wrong bit, isn't it
[06:19] <StevenK> wgrant: http://pastebin.ubuntu.com/539297/
[06:21] <wgrant> StevenK: Thanks.
[07:03] <lifeless> [07:03] <lifeless>     Hard / Soft  Page ID
[07:03] <lifeless>       80 / 5210  Archive:+index
[07:03] <lifeless>       62 /  271  BugTask:+index
[07:03] <lifeless>       21 /  248  Distribution:+bugs
[07:03] <lifeless>       18 /    0  BinaryPackageBuild:+retry
[07:03] <wgrant> lifeless: Yes, I'm working on it right now :P
[07:03] <lifeless>       15 /    3  ProjectGroup:+milestones
[07:03] <lifeless>       12 /  276  Distribution:+bugtarget-portlet-bugfilters-stats
[07:03] <wgrant> But the factory is being shitty.
[07:03] <lifeless>       12 /   19  Archive:+copy-packages
[07:03] <lifeless>       10 /  123  ProjectGroupSet:CollectionResource:#project_groups
[07:03] <lifeless>        9 /  146  POFile:+translate
[07:03] <lifeless>        8 /    2  Person:+bugs
[07:03] <lifeless> *interesting*
[07:03] <wgrant> The 300 or so missing ones?
[07:03] <lifeless> cool
[07:13] <lifeless> wgrant: factory fixin time?
[07:14] <wgrant> lifeless: I think it's OK now.
[07:15] <wgrant> I was confused for a while after I fixed it, since the baseline assertion had started failing without me noticing.
[07:15] <wgrant> But I think it's sorted now.
[07:31] <lifeless> kk
[08:48] <adeuring> good morning
[09:20] <mrevell> Morning Launchpad devvers
[10:00] <gmb> So, StevenK... about that failure in db-devel. Is anyone looking at why it happened?
[11:01] <gmb> jml: Given the nature of the current db-devel problem (i.e. ongoing) is there any reason not to force the build so that we don't block on it? I'll add a comment to the bug, obviously.
[11:01] <wgrant> Hudson's happy. Let's use Hudson.
[11:04]  * gmb can't stop thinking people are referring to mwhudson when they say "Hudson."
[11:04] <wgrant> My statement may still stand, even in that case.
[11:04] <gmb> Hudson, old boy, merge this branch would you? There's a good chap.
[11:04] <gmb> True.
[11:15] <gmb> jml: Ah, ignore me; it's obviously not all that intermittent; 3 out of the last 4 builds failed because of this, if I'm reading buildbot right.
[11:56] <wgrant> How easy is it to find a timeout by its pageid?
[12:02] <deryck> Morning, all.
[12:11] <StevenK> gmb: I utterly forgot to look and got distracted by work
[12:11] <StevenK> gmb: But if I'm reading your last statement rightly, it's not really not my fault, it's buildbot being crap?
[12:11] <wgrant> When did it start?
[12:11] <gmb> StevenK: Yeah, it looks like it's an annoying but ongoing problem. I haven't had time to dig further.
[12:12] <wgrant> Around the time of the databasefixture landing?
[12:12] <gmb> wgrant: The bug was filed on the 29th, FWIW: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/682861
[12:12] <_mup_> Bug #682861: db-devel librarian hung with '[Errno socket error] timed out' <build-infrastructure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/682861>
[12:13] <wgrant> That's, er, slightly suspicious timing.
[12:13] <wgrant> The databasefixture stuff landed on the 28th.
[12:14]  * StevenK is going to bang the Hudson drum in the new year
[12:14] <StevenK> I might actually *gasp* do some work on it while I'm off next week
[12:15] <wgrant> Damn, then who am I going to harass during my last week without ~launchpad superpowers?
[12:15] <StevenK> wgrant: Go play in the traffic
[12:15] <wgrant> :(
[12:19] <bigjools> wgrant: well done on passing
[12:19] <wgrant> bigjools: Thanks!
[12:20] <bigjools> did you get graded?
[12:22] <wgrant> Overall for the degree? No.
[12:22] <bigjools> when do you know?  or don't you?
[12:23] <wgrant> I'm not sure if they do it... but let's see what it would be.
[12:23] <wgrant> Oh good, H1.
[12:24]  * bigjools has no idea what that means
[12:24] <wgrant> First-Class Honours, ie >=80%.
[12:24] <bigjools> I did get a first class honours in hindsight from the university of life
[12:24] <wgrant> Haha.
[12:25] <bigjools> well done, anyway!
[12:25] <wgrant> Thanks.
[12:25] <wgrant> It is great to be done.
[12:30] <leonardr> jml, i sent email about this, but what do you mean when you refer to the set-based design we discussed 3 years ago?
[12:55] <wgrant> deryck: Hi.
[12:56] <deryck> hi wgrant
[12:57] <wgrant> deryck: I'm seeing around 450 YUI JS files included in <head> when it devmode. It makes page loads sort of slow. Is this likely to be related to your recent lazr-js upgrade?
[12:58] <deryck> wgrant, this is definitely related.  After I do qa today on the production changes, then I'll turn my attention to making that better.
[12:58] <wgrant> deryck: Thanks.
[12:58] <deryck> I should have sent mail about that already, sorry.
[12:58] <wgrant> Turning of devmode makes it fast again, since it just uses launchpad.js.
[12:58] <wgrant> I cannot type tonight, apparently.
[12:59] <deryck> heh
[12:59] <deryck> yeah, that is definitely a work around if you're not hacking on js itself.
[13:09] <gmb> flacoste: with db-devel being broken because of this ongoing but not-all-the-time timeout problem, does it make sense to force the build so that people can land things in the interim (and who knows, maybe it'll work this time)
[13:09] <gmb> ?
[13:12]  * gmb JFDoesIt.
[13:41] <mars> flacoste, ping
[13:56] <mars> gmb, +1 on forcing the build, +1000 on finding a possible clue as to its cause
[13:56] <gmb> mars: Yeah. I'll take a look later if I get the chance.
[13:59] <wgrant> mars: It's probably the librarian fixture changes.
[13:59] <wgrant> In db-devel r10013
[14:02] <bigjools> wgrant: awake enough for a very quick chat about removing non-PPA pockets from their indexes?
[14:02] <wgrant> bigjools: 'course.
[14:02] <wgrant> bigjools: They shouldn't be there, though...
[14:03] <bigjools> wgrant: I think it's an easy case of fixing the loops in C_writeIndexes and  D_writeReleaseFiles
[14:03] <bigjools> I was going to abstract the pockets to a call on IArchive
[14:03] <wgrant> bigjools: They're not actually ever written at the moment.
[14:03] <wgrant> Because they're never dirty.
[14:03] <bigjools> right
[14:03] <bigjools> but it loops them nonetheless
[14:03] <wgrant> We should skip them in A and B as well.
[14:03] <bigjools> ah yes
[14:05] <wgrant> bigjools: So, I'd just replace all the 'for pocket in PackagePublishingPocket.items' calls with 'for pocket in self.archive.getPocketsForSeries(distroseries)'
[14:06] <bigjools> wgrant: why by series?
[14:07] <wgrant> bigjools: Because you never know what Ubuntu will want to do next. But I guess for now we can skip that bit.
[14:07] <wgrant> Since it'll only be used in a couple of places.
[14:16] <wgrant> Anyway, it is getting late.
[14:29] <flacoste> hi mars
[14:45] <bigjools> aww jml you painted over my green bikeshed!
[14:53] <bigjools> foundations guys: I think I need to add lazr.enum._enum.EnumItems in lp_sitecustomize.py so that it has no security proxy, any thoughts?
[15:12] <mars> flacoste, pong
[15:17] <flacoste> mars: what can I do for you?
[15:18] <mars> bigjools, I think Gary would be best positioned to answer that question, but unfortunately he is out today
[15:18] <bigjools> mars: yeah, he's blowing out a lot of candles
[15:18] <mars> :)
[15:19] <bigjools> I can't talk, I've got the same number in 3 months...
[15:25] <jcsackett> how often is the data on qastaging restored?
[15:34]  * bigjools shakes fist at zope testrunner
[15:35] <bigjools> jcsackett: one a week
[15:35] <bigjools> once*
[15:35] <jcsackett> thanks.
[15:36] <bigjools> so, if I use an argument to bin/test like "-t soyuz -t !windmill" it ends up running every single test except windmill ones rather than all the soyuz  tests except windmill.
[15:41] <jelmer> -t is unfortunately || rather than &&
[15:42]  * jelmer often just uses a file with a list of tests to run and calls ./bin/test --load-list=FILE
[15:55] <bigjools> jelmer: yeah, && would be more useful
[16:31] <bac> jtv: are you disappearing soon?
[16:31] <jtv> bac: from the channel, yes.
[16:31] <bac> jtv: rats.  i'd like to work with you on a problem but i realize it is really late for you.
[16:32] <jtv> bac: oh, you're back on the West Pole already?
[16:32] <bac> yep, for a week
[16:32] <jtv> A week.
[16:33] <jtv> bac: but hey, see if you can grab my attention.  What is it you wanted to work on?
[17:44] <cody-somerville> libmozjs2s from LP-PPA-Launchpad is trying to uninstall my xulrunner :(
[18:55] <deryck> sinzui, ping
[18:58] <mkanat> Is there an icon that would mean something like "give me the raw content of this" in Launchpad that I should re-use for loggerhead?
[19:09] <maxb> Not that I have seen
[19:09] <sinzui> hi deryck
[19:10] <maxb> Also you'd have to jump through lots of licensing hoops if you wanted to take an icon from LP into Loggerhead
[19:11] <deryck> hi sinzui.  Is the calendar widget used on registry pages in js or python?  I can't find tests or a js file for it.
[19:11] <sinzui> deryck, the calendar is yui2. I think it was moved to lp.app
[19:12] <sinzui> deryck, we want the widget to be used every time there is a date or datetime field, but we do not seem to know how to ensure the js loads because the field is present
[19:13] <deryck> sinzui, ok, cool.  That helps then.  The lazr-js upgrade has borked the style of this widget, but I suspect it's an easy fix via class names.
[19:13] <mkanat> maxb: Why, are the LP icons not GPL?
[19:14] <maxb> mkanat: No, they are copyright Canonical, restricted rights for use in development only
[19:14] <mkanat> maxb: Oh, okay.
[19:15] <sinzui> deryck, I thought there was a windmill test for the calendar, but I do not see it
[19:16] <deryck> sinzui, yeah, it doesn't appear to have any test coverage, as far as I can see.
[19:17] <deryck> but no biggie.  I'll fix it all up next week.
[19:17] <sinzui> I there a yui3 calendar widget yet?
[19:17] <deryck> hmmm, not sure.
[19:17] <deryck> I can look into that a do a replacement.
[19:31] <benji> Is there a design document (or something similar) that describes the librarian?
[19:31] <lifeless> yes
[19:32] <lifeless> librarian.txt, docstrings.
[19:32] <benji> k, thanks
[19:51] <lifeless> jkakar:
[19:51] <lifeless> jkakar: hey
[19:52] <jkakar> Hi! :)
[19:52] <lifeless> jkakar: wondered what you think of my object graph to model query proposal.
[19:52] <lifeless> I'm sure its not new, but none of the python ORM's I've seen do it ...
[19:53] <jkakar> lifeless: It's still marked as "read" in my mail... I haven't made the time to read through it properly. :(
[19:53] <jkakar> lifeless: I want to, just been pressed with lots of other things.
[19:53] <jkakar> lifeless: Will try to get to it this weekend.
[20:54] <wgrant> flacoste: Do you really think that Soyuz can be simplified such that domain expertise is no longer required?
[20:54] <flacoste> hi william!
[20:54] <jcsackett> the whole pqm/release thing now largely has to do with stuff in db-devel, right? things done in devel will still roll out on a fairly regular interval, so RCs are less important, aren't they? or have i completely misunderstood something?
[20:54] <wgrant> Heh, morning.
[20:56] <flacoste> wgrant: i do believe that we can make it so that many changes can be done without requiring vetting by somebody who "understands the whole system" and without the risk of breaking everything
[20:56] <flacoste> wgrant: but people will have to understand the basics of package building and archive management
[20:56] <jelmer> jcsackett: No, that's right.
[20:56] <flacoste> but i don't think it requires a year of dedicated Soyuz hacking to get that
[20:56] <jcsackett> jelmer: thanks.
[20:57]  * jcsackett relaxes a touch.
[20:58] <flacoste> jcsackett: you should still make sure that all your revisions before 12019 are qa-okl
[20:58] <flacoste> or get a rc to fix any qa-bad in that range
[20:58] <flacoste> jcsackett: but yeah, we'll continue nodowntime deployment afterwards from devel
[20:58] <jcsackett> flacoste: i may need to do that. it's not a qa-bad, per se, but a qa-ok that's an incomplete solution to an inprogress bug. but i think i'm about to have the full solution, which would be nice to release all together.
[20:59] <flacoste> jcsackett: right, you can negotiate that with abentley
[20:59] <jcsackett> flacoste: right. first though i have to know if have anything to negotiate. was just checking on time tables. :-P
[21:03] <abentley> jcsackett: are any of your changes going to introduce regressions?
[21:03] <jcsackett> in the thing i'm working on?
[21:04] <jcsackett> abentley: in the thing i'm working on, no. in the branch that's qa-oked but we've discovered is incomplete, it wouldn't appear so. nothing is worse than it was, or we'd qa-bad it.
[21:05] <lifeless> wgrant: required? no. God Like Knowledge Of Every Line Of Code - that can be fixed.
[21:05] <abentley> jcsackett: Okay, so it won't need to block deployment.
[21:05] <lifeless> wgrant: for instance, concurrent uploads to the uploader, what unique keys are :)
[21:05] <jcsackett> abentley: yeah, the branch already landed isn't a blocker.
[21:06] <abentley> jcsackett: We can give it an RC if you think it's necessary to deploy your changes at the first available opportunity.
[21:06] <wgrant> lifeless: It's not quite that simple to fix, I'm afraid.
[21:06] <abentley> jcsackett: That would be Wednesday next week at 10:00 UTC.
[21:06] <jcsackett> abentley: dig.
[21:06] <lifeless> wgrant: is it worth fixing?
[21:07] <lifeless> wgrant: (rhetorical)
[21:07] <abentley> jcsackett: Otherwise, land it normally, and it can be part of the first nodowntime deployment of the next cycle.
[21:07] <jcsackett> abentley: okay. thanks.
[21:07] <lifeless> wgrant: I think its worth fixing because the current set of folk that can work on soyuz is the intersect of 'knows zope well', 'knows twisted well', 'knows debian packaging well', 'knows debian archive mgmt well'
[21:07] <lifeless> wgrant: ... very small set.
[21:08] <wgrant> lifeless: Sure, that sucks.
[21:08] <lifeless> every one of those that we can relax by better layering, clearer code, safer system, the larger the set of folk that can contribute sensible.
[21:08] <wgrant> But the new model means that the last two sets will vanish.
[21:09] <lifeless> uhm
[21:09] <wgrant> And after that it's just a matter of time until someone mistakenly qa-ok's something and tens of millions of machines notices.
[21:09] <lifeless> wgrant: I think you're mistaken.
[21:09] <abentley> rockstar: chat?
[21:10] <lifeless> wgrant: right now, the review policy does not require domain reviews of changes.
[21:10] <lifeless> wgrant: the reorg asks folk to take on the challenge of a wider view of the system. It doesn't expect them to suddenly know things they didn't.
[21:10] <lifeless> wgrant: learning will be involved.
[21:10] <wgrant> lifeless: No, but few outside the team make Soyuz changes. And I know that at least I review everything Soyuzy.
[21:10] <lifeless> wgrant: what will stop you doing that?
[21:11] <wgrant> lifeless: Trusting everyone to make changes to such a fragile, critical piece of Launchpad. Probably starting with myself.
[21:12] <lifeless> wgrant: do you mean 'there will be so many soyuz patches wgrant cannot look at them all' ?
[21:14] <wgrant> lifeless: No. I mean what I said. When I can be reasonably confident that people aren't able to accidentally break things, I will have a reason to stop watching.
[21:15] <wgrant> Soyuz is fragile. Some of this fragility is implied by the domain. And the consequences of a mistake can be immense.
[21:16] <rockstar> abentley, sure.  Mumble?
[21:16] <abentley> rockstar: sure.
[21:18] <lifeless> wgrant: you're talking about breaking the primary archive in some fashion, right ?
[21:18] <wgrant> lifeless: That's the worst case, but sure.
[21:18] <lifeless> so, some of the fragility is due to apt disk layout, sure.
[21:18] <flacoste> wgrant: i really don't get 'fragility implied by the domain'
[21:18] <lifeless> but a lot of it is due to our code.
[21:18] <flacoste> i think that's my main point of disagreement
[21:18] <flacoste> what is so fragile in archive management
[21:19] <wgrant> lifeless: A lot of the code is atrocious, sure.
[21:19] <lifeless> wgrant: I'll buy 'expected cost of a qa failure is huge'
[21:19] <wgrant> I'll also throw in "QAing completely is just about impossible"
[21:19] <lifeless> wgrant: but I don't see a reason for a qa failure to be more or less likely simply because its archive mgmt
[21:19] <flacoste> the fact that we relied on poor perl code for stuff isn't something implied by the domain
[21:19] <lifeless> wgrant: builds are already more-than-soyuz-team today.
[21:20] <wgrant> lifeless: Um, not really, but OK.
[21:20] <lifeless> wgrant: buildd manager isn't, but it breaking isn't catastrophic (compared to the main web UI going down, APIs going down, or codehosting going down)
[21:21] <lifeless> wgrant: its not less catastrophic either.
[21:21] <wgrant> lifeless: buildd-manager problems are always handled by Soyuz.
[21:21] <wgrant> Still.
[21:21] <lifeless> yes, I know
[21:22] <wgrant> Despite it in theory being shared for 11 months now.
[21:22] <lifeless> well
[21:22] <lifeless> buildd manager wasn't ever shared yet AFAIK
[21:22] <lifeless> slaves are
[21:22] <wgrant> Hm?
[21:22] <lifeless> and they get worked on
[21:23]  * jelmer waves
[21:24] <jelmer> flacoste: I don't think the fragility is implied by the domain. Even if it is, it's certainly not why things are fragile at the moment.
[21:24] <wgrant> Some of it is. A lot of it isn't.
[21:27] <flacoste> jelmer: thanks! i was starting to think i was blind to something
[21:30] <wgrant> Yes, Soyuz is far worse than it needs to be.
[21:33] <jelmer> flacoste: I do agree with Julian's concerns about Soyuz' fragility in the shorter term though.
[21:37] <jelmer> perhaps I should follow up on the ml
[21:37] <cody-somerville> Sounds like there is an interesting discussion going on somewhere that I haven't seen yet.
[21:38] <wgrant> On canonical-launchpad@l.c.c
[21:38] <flacoste> cody-somerville: 'Announcing Launchpad Squads' on canonical-launchpad mailing list
[21:39] <lifeless> flacoste: perhaps its time to take it to lp-dev ?
[21:39] <flacoste> perhaps
[21:39] <flacoste> or start a different thread there
[21:40] <lifeless> the latter is what I meant
[21:41] <elmo> "Grave mistakes caused by soyuz rookies setting off booby traps is a future problem that might not ever actually happen.", i.e. "this risk may not happen, therefore it does not exist"
[21:41] <elmo> this thread is really beginnning to depress me
[21:43] <cody-somerville> :-(
[21:48] <wgrant> lifeless: On a lighter note, my Archive:+index and Archive:+packages fix is reviewed and tested.
[21:49] <lifeless> wgrant: sweet.
[21:49] <lifeless> wgrant: land it!
[21:49] <wgrant> lifeless: Aren't we still r-c?
[21:50]  * lifeless twitches
[21:50] <wgrant> The topic suggests it, although db-stable was merged recently.
[21:50] <lifeless> yes
[21:50] <lifeless> we need to qa *past* the db merge
[21:50] <lifeless> and then we can unlock
[21:50] <wgrant> Ah.
[21:50] <lifeless> thats the critical section
[21:51] <lifeless> currently waiting for buildbot
[22:21] <flacoste> elmo: i think i understand your concerns, iow, fuck ups in Soyuz can have a high-impact, we shouldn't thread this lightly
[22:23] <flacoste> elmo: it does mean that we should be careful when reviewing changes and QA there, but not sure that keeping the people who can make changes there is a solution
[22:23] <flacoste> elmo: it didn't really got us far now
[23:01] <EdwinGrubbs> lifeless: if a page explicitely gets the master store, doesn't it mean that every *Set class will load objects with the wrong store since they usually use IStore to get it? It seems like passing a store object into methods on *Set classes is a an inefficient way to handle this.
[23:01] <lifeless> EdwinGrubbs: there may be many cases where its a problem.
[23:01] <lifeless> EdwinGrubbs: my primary point was you're adding a new instance of the problem :)
[23:02] <lifeless> EdwinGrubbs: we had timeout/performance issues on some pages due to this with LibraryFileAlias, for instance.
[23:02] <lifeless> EdwinGrubbs: I don't mind if you want to search for a cool elegant way to handle it, but supplying the store is a fairly obvious and will-work approach.
[23:03] <lifeless> EdwinGrubbs: essentially, IStore should be used as a matter of last resort, because its a global lookup rather than related-state lookup.
[23:04] <wgrant> The Sets traditionally use I(Master|Slave)?Store.
[23:04] <wgrant> Since they don't have an object on which they can call Store.of.
[23:10] <lifeless> right
[23:10] <lifeless> its a bug
[23:12] <lifeless> the whole set structure is a bit buggy
[23:12] <lifeless> they are exposed as singletones
[23:12] <lifeless> s/es/s/
[23:12] <wgrant> Right.
[23:12] <lifeless> but correct behaviour for them depends on non-global state
[23:12] <wgrant> I'm hoping that your next piece of work will make this a bit more fixable.
[23:32] <wgrant> cody-somerville: What do I have to do to convince you not to use PPAs for obsolete series?
[23:33] <lifeless> wgrant: customers which we have contracts with need them.
[23:33] <wgrant> Well that's stupid.
[23:34] <wgrant> s/stupid/insecure/, if you must.
[23:34] <lifeless> s/.*/reality/
[23:34] <wgrant> Shh.
[23:34] <lifeless> insecure?
[23:35] <wgrant> Or does OEM maintain security updates for obsolete series?
[23:35] <lifeless> a per device thing, I believe
[23:35] <lifeless> (so yes)
[23:35] <wgrant> :(
[23:36] <jpds> Ew.
[23:57] <lifeless> jpds: well, its essentially a LTS for a specific customer, AIUI. cody-somerville will know all the gory details.
[23:58] <lifeless> point is, *some* PPA's today need to keep publishing non-regularsupported suites.