[02:33] <nigelb> I still can't figure out why I started hacking on LP before I read about Zope 3.
[02:33] <nigelb> Finally I'm reading Zope 3 intros :P
[04:53]  * StevenK purges 3GiB of postgres logs so that /var actually has some free space.
[04:57]  * nigelb twitches at GiB
[05:00] <StevenK> nigelb: You don't like {K,M,G,T}iB?
[05:00] <StevenK> Can't remember what comes after T. E?
[05:21] <lifeless> Eca
[05:21] <lifeless> bah
[05:21] <lifeless> Exa
[05:22] <StevenK> Right, so it is E
[05:22] <lifeless> or is it P
[05:22] <StevenK> Then, P, then Z, and then OMGBBQWTFBIG
[05:22] <lifeless> it might be P E
[05:22] <lifeless> ah E P
[05:24] <lifeless> wgrant: we don't build-in a structure of partitioned namespaces (partly because teams have no project affiliation etc
[05:24] <lifeless> wgrant: so folk need to remember
[05:32] <lifeless> wgrant: however, this one has me confused too ;P
[06:20] <nigelb> StevenK: My dislike is for the 'i'
[06:21] <lifeless> nigelb: you're a no i kinda guy ?
[06:22] <nigelb> lifeless: Bwahaha. No.
[06:26] <nigelb> So, I noticed something weird today.
[06:26] <nigelb> Potentialy a mintor security thing.
[06:27] <nigelb> When I go to https://launchpad.net/~canonical-sysadmins, I see a <hidden> under "Subteam of"
[06:27] <nigelb> Isn't our policy to to show "its not even there" when something isn't supposed to be seen?
[06:28] <nigelb> s/show/do'g
[06:28] <nigelb> s/show/do/g
[06:35] <lifeless> thats an adapter kicking in and preventing disclosure
[06:35] <lifeless> yes, it shouldn't show at all, but its a cosmetic, not a security, concern
[06:35] <lifeless> please do file a bug
[06:36] <nigelb> cool, will do
[06:49] <wgrant> lifeless: It is actually a security concern, slightly.
[06:49] <wgrant> I filed it a couple of years ago.
[06:49] <wgrant> It's in alphabetical order.
[06:49] <wgrant> So you can determine the display name.
[06:50] <nigelb> wgrant: there's already a bug?
[06:50] <wgrant> I think so. Trying to find it...
[06:51] <lifeless> wgrant: you can guess at possible display names ;)
[06:51] <wgrant> lifeless: Then you can create teams on staging to do a search.
[06:51] <nigelb> wgrant: bug 329132
[06:51] <nigelb> ?
[06:51] <_mup_> Bug #329132: Strange string under "Subteam of" in LOSAs' team page <confusing-ui> <lp-registry> <registry-people> <Launchpad itself:Fix Released by bac> < https://launchpad.net/bugs/329132 >
[06:52] <wgrant> Hah
[06:52] <wgrant> What a fix.
[06:52] <wgrant> I thought I filed a similar one, though...
[06:53]  * nigelb stabs Zope 3
[06:54] <nigelb> Trying to get it to work. Refuses to work.
[06:54] <wgrant> What are you doing to the poor creature?
[06:54] <nigelb> Using Benji's quick start guide
[06:54] <wgrant> nigelb: I guess you should file a new one rather than reopen the old one.
[06:55] <nigelb> wgrant: Filing now
[06:55] <wgrant> nigelb: You may have some issues with using that.
[06:55] <nigelb> Oh.
[06:55] <wgrant> nigelb: Since Zope 3 doesn't exist any more.
[06:55] <nigelb> wtf.
[06:55] <wgrant> It's... complicated.
[06:55] <wgrant> Most of the underlying libraries now make up what is known as the Zope Toolkit, or ZTK.
[06:56] <wgrant> The UI (ZMI and other stuff that's mentioned there) is not part of the ZTK, and was abandoned for a time.
[06:56] <wgrant> But then was made into its own project atop the ZTK, BlueBream.
[06:56] <nigelb> All I wanted was to get a hands on feel for zope minus the complications of Launchpad so I can figure out Launchpad faster.
[06:56] <wgrant> Not sure if that is still active.
[06:56] <wgrant> http://bluebream.zope.org/screencasts.html may be helpful.
[06:57] <wgrant> http://bluebream.zope.org/doc/1.0/index.html too
[06:57] <wgrant> Looks like the docs don't make me want to die any more. Which is a good start.
[06:57] <nigelb> Oh, cool!
[06:57] <nigelb> Thank you
[06:57] <nigelb> Is that closer to what LP is using?
[06:58] <wgrant> LP doesn't use BlueBream, but it uses many components of the ZTK.
[06:58] <wgrant> Not all the latest versions, but mostly within the last 18 months.
[06:58] <wgrant> It also uses some stuff that BlueBream uses but that is outside the ZTK.
[06:58] <nigelb> Ok, so it should give me a good basic idea.
[06:58] <wgrant> Yup.
[06:59] <nigelb> Right now, I'm coding on LP with extensing grepping and asking here.
[06:59] <nigelb> And grepping my logs of asking here.
[06:59] <wgrant> Previously I would have suggested that you obtain, by some means, a copy of Web Component Development with Zope 3. But it was already obsolete when I started, and it's going to not be practically usable now.
[07:00] <wgrant> Good in-depth teaching of the concepts, though.
[07:00] <wgrant> Most LP engineers do what you do, I think :)
[07:00] <wgrant> They learn Zope 3 by drowning in LP.
[07:00] <nigelb> Hah
[07:01] <wgrant> "It is recommended to use a custom-built Python for working with BlueBream."
[07:01] <wgrant> Gaaaaah stupid Python people.
[07:01] <nigelb> heh
[07:02] <wgrant> (if following the bluebream getting started guide, I would use virtualenv to install bluebream without polluting your system)
[07:03] <nigelb> I always use virtualenv for this kinda stuff
[07:03] <lifeless> wgrant: ah, nifty
[07:03] <wgrant> From Wikipedia: "BlueBream is in active development and is now considered a stable framework, used on production projects worldwide, most notably Launchpad."
[07:04] <nigelb> WHAT?
[07:04] <wgrant> But we don't use the whole thing :(
[07:04] <wgrant> Maybe they've altered the scope of BlueBream to be everything that was in Zope 3 that is not in the ZTK.
[07:04] <wgrant> Last I heard it was primarily the ZMI.
[07:04] <lifeless> nigelb: my preferred solution for 'LP is hard to understand' is to clean things up
[07:05] <lifeless> nigelb: e.g. by deleting lots o fcode
[07:05] <nigelb> lifeless: My problem is my web development experience is in Django. I'm trying to figure out how is the sort of right way to to do things in zope.
[07:05] <lifeless> nigelb: ah
[07:05] <nigelb> Its probably a day or two of reading up
[07:05] <lifeless> nigelb: so do you want the good news or the bad nes ?
[07:05]  * StevenK deleted something like 2,800 lines of code yesterday
[07:05] <nigelb> I figured its time I invested that
[07:05] <nigelb> lifeless: I'll take the bad news first
[07:05] <lifeless> nigelb: the right way of doing things is the wrong way for LP
[07:06] <nigelb> Expected.
[07:06] <nigelb> Yay!
[07:06] <lifeless> sorry, that was mistyped. But hilarious.
[07:06] <lifeless> nigelb: the right way of doing things in zope is the wrong way for LP
[07:06] <nigelb> StevenK: When I meet at some point, I'll buy you a drink for that.
[07:06] <lifeless> is what I meant to say.
[07:06] <nigelb> Small difference :)
[07:06] <nigelb> StevenK: *meet you
[07:07] <lifeless> two words, but significant impact :>
[07:07] <wgrant> lifeless: Well, everything except persistence is not too bad.
[07:07] <lifeless> wgrant: except that *everything* is driven by persistence.
[07:07] <lifeless> wgrant: also you forgot configuration.
[07:07] <nigelb> Well, so I'm probably looking at bare basic concepts. Like tal thingy or zcml.
[07:08] <nigelb> Both of these I still don't really understand, but I can read and guess
[07:08] <wgrant> lifeless: Except ZCA and views and TAL and ZCML and the general architecture.
[07:08] <lifeless> wgrant: tal and views are impacted by persistence; zcml is because of the model:interfaces interactions.
[07:09] <wgrant> Only slightly.
[07:09] <wgrant> lifeless: Also, do we want to roll back that spaces in URLs thing?
[07:09] <lifeless> wgrant: zca also interacts with persistence in the same way, fwiw
[07:09] <lifeless> wgrant: yes, we do.
[07:10] <wgrant> Shouldn't have to roll back production, since we're almost deployable.
[07:10] <wgrant> Also there's also an additional 1000 Loggerhead OOPSes that we might care about.
[07:10] <lifeless> wgrant: are you still on leave ?
[07:10] <wgrant> Yes.
[07:10] <lifeless> heh
[07:10] <lifeless> wgrant: I was going to ask if you felt like doing the rollback
[07:10] <nigelb> If wgrant is *this* productive when he's on leave...
[07:10] <wgrant> qemu-kvm's broken gdbstub is broken enough that I will look at work, but not do it :)
[07:11] <nigelb> lifeless: I saw the wiki LEP. Has it gone stale or is it still alive?
[07:11] <lifeless> nigelb: its confused
[07:11] <nigelb> Because I'd like to propose extracting stuff from readthedocs and bring it into LP.
[07:11] <StevenK> lifeless: Oh, the loggerhead rollback?
[07:11] <lifeless> StevenK: not loggerhead, url ascii rules
[07:12] <wgrant> loggerhead's broken too, but I think that might be two rollouts ago.
[07:12] <wgrant> Not quite sure.
[07:12] <lifeless> 993 RuntimeError: dictionary changed size during iteration ?
[07:12] <lifeless> probably bzrlib
[07:13] <wgrant> We didn't upgrade bzrlib, did we?
[07:13] <wgrant> Still 2.3.3
[07:13] <wgrant> We did upgrade loggerhead.
[07:15] <lifeless> we need a bug at the least
[07:15] <lifeless> and probably a rollback of loggerhead too
[07:18]  * wgrant curses lifeless a bit.
[07:18] <lifeless> wgrant: ?
[07:18] <wgrant> The only thing left writing to BugTask's target key attributes directly is the bugsummary tests.
[07:19] <wgrant> removeSecurityProxy ftw.
[07:19] <lifeless> wgrant: hah. I didn't write those :P
[07:19] <wgrant> GUILTY
[07:20] <wgrant> We might be able to put NULL Project to sleep on Monday...
[07:21] <StevenK> Oh, nice
[07:21] <StevenK> wgrant: Can I land populate-bprc now?
[07:22] <lifeless> StevenK: why wouldn't you ?
[07:22] <wgrant> Well, it almost has a veto.
[07:22] <wgrant> (not from me!)
[07:22] <StevenK> lifeless: From a certain GSA who won't be named saying it sounded like an epicly bad idea
[07:23] <lifeless> how so ?
[07:23] <lifeless> what was the concern ?
[07:23] <lifeless> https://bugs.launchpad.net/loggerhead/+bug/826136
[07:23] <_mup_> Bug #826136: dictionary changed size during iteration <oops> <loggerhead:Triaged> < https://launchpad.net/bugs/826136 >
[07:24] <StevenK> lifeless: That the data does not and should not go into the main DB
[07:24] <lifeless> StevenK: thats not a concern
[07:24] <lifeless> StevenK: thats a conclusion
[07:24] <lifeless> :)
[07:25] <StevenK> lifeless: After wgrant and I talked about it, we decided to wait for a bit, since if we kick off the migration, it's quite difficult to migrate the data back out again
[07:25] <lifeless> StevenK: I don't follow.
[07:25] <wgrant> It's slightly awkward.
[07:25] <wgrant> Not quite difficult :)
[07:27] <StevenK> lifeless: It's a large amount of data that will land into launchpad_main -- it will sit like a red wine stain. Once it's there, it will require persistance to move
[07:28] <lifeless> StevenK: so the concern is size?
[07:29] <lifeless> StevenK: why isn't deleting it as simple as a loop doing delete LIMIT 50 ?
[07:29] <StevenK> lifeless: No. My small concern is migration pain
[07:29] <StevenK> My other concern is I'm not clever enough to implement step 3 and 4 of the grand plan
[07:30] <lifeless> they being?
[07:30] <StevenK> 3. Hook into the publisher to create and delete rows in BPRC and BPP when it publishes or superseded a BPR.
[07:31] <StevenK> 4. Hook into the publisher to generate contents file, and turn off the current contents generation script.
[07:31] <lifeless> 3 seems bogus
[07:31] <wgrant> Both are.
[07:31] <StevenK> Eh?
[07:31] <wgrant> Neither should be in the publisher, I don't think.
[07:32] <wgrant> BPRC lifetime is the same as file lifetime.
[07:32] <lifeless> 4 could be in the publisher, or separate
[07:32] <wgrant> So they should certainly not be removed by the publisher.
[07:32] <wgrant> 4 is possibly OK, but not required, and there's already enough in the publisher IMO :)
[07:32] <lifeless> 3 is bogus because there can be N published entries, and trying to add just-in-time or delete just-after-use will be massively racy.
[07:34] <StevenK> The garbo job should certainly keep up with new BPRs, but I'm concerned about bloat in BPRC/BPP
[07:34] <wgrant> The BPF expiry script will remove them.
[07:34] <StevenK> wgrant: Ah, that reminds me.
[07:35] <StevenK> wgrant: The populate-bprc garbo job returns BPRs that do not have any BPFs. Do not want.
[07:35] <wgrant> Indeed, although in production there shouldn't be any.
[07:36] <StevenK> wgrant: Hah, maybe.
[07:36] <lifeless> StevenK: so here is a question folk might like to answer
[07:36] <StevenK> lifeless, wgrant: So, land the garbo population job and then think about the next step while it populates?
[07:36] <lifeless> 'when was /usr/bin/foo introducted to the archive'
[07:37] <lifeless> StevenK: we might want to gc things for space considerations
[07:37] <lifeless> but having enough data to ssatisfy the use case is the first step
[07:37] <StevenK> We certainly do want to GC things
[07:37] <lifeless> StevenK: what was the gsa's concern ?
[07:37] <StevenK> lifeless: That it doesn't belong in the main DB
[07:37] <lifeless> StevenK: You can fit *every* file in *every* arch in *every* published deb, ever, in < 5GB
[07:38] <lifeless> [excluding PPAs, but they are (for now) still minimal)
[07:38] <StevenK> lifeless: Modulo bugs :-(
[07:42] <lifeless> StevenK: yes, but having extra rows would be fairly obvious
[07:42] <lifeless> StevenK: I don't think gc is a given.
[07:44]  * StevenK fixes the conflicts the branch has with devel
[08:17] <lifeless> rev 13687 is the rollback
[08:18] <StevenK> lifeless: We don't have much QA to do, wallyworld_, bigjools, 2 * danilo
[08:18] <StevenK> We might be able to deploy on Monday afternoon UK time
[08:26] <lifeless> not morning ?
[08:30] <StevenK> lifeless: Depends on how quick the QA is done -- I was being pessimistic.
[15:00] <nigelb> gah, qastaging is timing out, known?
[15:12] <jelmer> nigelb, hey
[15:12] <jelmer> nigelb, I don't think it is - perhaps it's just updating?
[16:05] <nigelb> jelmer: hm, maybe.
[16:05] <nigelb> I'll just try a while later :)
[20:08] <lifeless> nigelb: timing out on all pages or just one ?
[20:08] <lifeless> nigelb: some timeouts are (unfortunately) 'normal' for [qa]staging
[23:30] <lifeless> aieeeee
[23:30] <lifeless> IUnloggedException. Sob.