[00:07] <lvh> hi
[00:07] <lvh> I'm trying to configure my lp instance for remote access, and I'm in the /etc/hosts file
[00:08] <lvh> is this rule supposed to remain unchanged?  127.0.0.77      vostok.dev archive.vostok.dev
[00:08] <lifeless> have you looked on the wiki
[00:08] <lifeless> Running/VM
[00:08] <lifeless> I think
[00:09] <lifeless> covers some sutff
[00:11] <lvh> I hadn't, because I'm running this on a physical machine :-)
[00:11] <lifeless> has the same issues though
[00:11] <lifeless> note that a dev environment isn't suitable for production use (in case thats what you're trying to do)
[00:12] <lvh> I'm experimenting, but that was the eventual plan, yes
[00:12] <lifeless> hosts files are local - you'll need actual DNS, ssl certificates, front end servers etc
[00:13] <lvh> Unless I use ssh forwarding, I still get unstyled main pages that say: Lost something?
[00:13] <lvh> There’s no page with this address in Launchpad.
[00:13] <lvh> (For the front page.)
[00:14] <lifeless> you'll need a production config, lazr schema, slave db etc too
[00:15] <lifeless> we don't really document how to run production LP - its not something that was envisioned when we open sourced it (note too that you /must/ rebrand it - the look n feel is not opened)
[00:15] <lifeless> you may want to record the things you learn on the wiki somewhere.
[00:16] <lifeless> I don't know why you'd get Lost something pages, unless you're bypassing your front-end apache
[00:16] <lvh> I'm not trying to kick any shins, I'm fine just using ye olde bzr + roundup...
[00:17] <lifeless> sorry, I can't parse that
[00:17] <lvh> (Is there a canonical acronym for LAMP, but for developing software instead of serving webapps?)
[00:18] <lvh> lifeless: I'm not trying to piss people off by running my own instances of lp that aren't for developing lp.
[00:18] <lvh> lifeless: If that's something you don't want people doing, that's fine -- unfortunate, since that means I have to set up like twenty things instead of one, but I'll survive :-)
[00:19] <lifeless> its fine to run your own instance, thats clearly something the AGPL permits; What you can't do is use the launchpad branching - the stock icons, widget set - because thats (IIRC) trademarked.
[00:19] <lifeless> lvh: I am curious why you want to run your own instance rather than use launchpad.net (but its not really my business :))
[00:19] <persia> Doesn't that just prohibit use to the public?
[00:19] <persia> (but ask counsel for detailed use restrictions on trademarks)
[00:20] <lifeless> persia: https://dev.launchpad.net/LaunchpadLicense
[00:20] <lvh> lifeless: IANAL but I did the relevant parts of a law degree
[00:20] <lvh> (Enough to know nobody really understands the AGPL ;-))
[00:21] <lifeless> I wasn't part of the open sourcing process; its a bit neither-fish-nor-fowl at the moment.
[00:21] <lifeless> it would be nice to move the theming into a dedicated tree and just say 'whats in tip, is fully agpl, go free'.
[00:21] <lifeless> but there's some implications for test and development
[00:21] <lvh> lifeless: Mostly because I'm researching development stacks at the moment
[00:21] <lvh> lifeless: for closed-source stuff
[00:22] <lifeless> lvh: sure. Are you aware that launchpad.net supports closed-source stuff ?
[00:22] <lvh> lifeless: Yes
[00:22] <lifeless> great
[00:22] <lvh> lifeless: I think their pricing system is wrong ;-)
[00:22] <lifeless> anyhow, I hope I've helped with the issue you're seeing
[00:22] <lifeless> lvh: have you given that feedback to mrevell ?
[00:22] <persia> lifeless, Makes sense.  I simply don't know enough of the relevant precedent to know if it's enforceable in cases where it's undiscoverable.  Given that it's an explcit restriction on license, rather than just a trademark claim, I'm no longer tempted to believe it's a public/private thing (which is the common trademark model)
[00:22] <lvh> lifeless: Yeah.
[00:22] <lifeless> lvh: ok, cool
[00:23] <lvh> lifeless: Something along the lines of "I see your point, but it works for some people, and we're not changing it anyway"
[00:23] <lifeless> mmm
[00:23] <lifeless> nothing is unchangable
[00:23] <lvh> lifeless: (I believe you should limit *people*, not *projects*)
[00:24] <lvh> Or at least not primarily projects.
[00:24] <lifeless> (because?)
[00:24] <lvh> (And about a week after I blogged about that, Atlassian went out and did it with Bitbucket)
[00:24] <lvh> lifeless: Well, practical example
[00:25] <lvh> lifeless: Small interdependent pieces of code: good, big monolithic pieces of code: bad, right?
[00:25] <lifeless> bitbucket does closed source by user, not product ?
[00:25] <lvh> lifeless: bitbucket lets you make infinite repos
[00:25] <lvh> closed or open
[00:25] <lifeless> yeah, I can see the argument.
[00:25] <lifeless> it makes sense to me.
[00:25] <lvh> lifeless: however, if you want to *share* those repositories, that's when it costs you
[00:25] <lvh> for 5 or below, it's 0, etc
[00:26] <lifeless> afterall users are what incur costs, not small bits of metadata.
[00:26] <lvh> right
[00:26] <lvh> plus everything else pretty much bills you per user
[00:26] <lvh> the cutoff point is pretty high too
[00:27] <lvh> 250/yr, let's say 20/month
[00:27] <lvh> that gets you 25 users at bitbucket, which isn't a trivial two-bit operation anymore
[00:27] <lvh> and they get infinite projects
[00:28] <lvh> (I realise my comparison is flawed because bitbucket doesn't have merge proposals and their bugtracker is useless, but you get the basic idea)
[00:28] <lifeless> yeah
[00:29] <lifeless> we're primarily here for open source though
[00:29] <lifeless> somewhat different concerns
[00:29] <lvh> well, that's true, but
[00:29] <lvh> the closed source stuff is so I *can* work on open source stuff for free
[00:29] <lvh> I still gotta eat at some point in time
[00:29] <lifeless> sure
[00:30] <lvh> (it actually says that explicitly in the notary ... not sure what the correct english term is. deed? document-that-starts-a-company. One of the goals of the comapny is to contribute to open source projects)
[00:31] <lifeless> cool
[00:31] <lifeless> articles of incorporation
[00:31] <lifeless> constitution
[00:32] <lifeless> depends on the country
[00:32] <lvh> yeah, I'm from belgium, country-ish things are vague and fleeting
[00:32] <lifeless> hmm, the quake a wee bit back was a 5.0
[00:35] <lifeless> http://www.geonet.org.nz/earthquake/drums/
 Doesn't that just prohibit use to the public? (but ask counsel for detailed use restrictions on trademarks)
[00:35] <poolie> this is kind of fud-ish
[00:35] <poolie> i know it's not meant maliciously
[00:37] <persia> poolie, It's structurally built into things: it's published that anyone providing legal advice who is not so licensed (where I live, where you live, and many other places) can be (successfully) sued by any counsel without such comments.
[00:37] <poolie> well, don't provide legal advice then
[00:38] <poolie> if i say "how do i set up gpg" and you tell me "ask counsel for detailed use restrictions" it may be technically true but it's not very welcoming
[00:40] <poolie> do you see what i mean?
[00:40] <persia> That's kinda different.  I may be overcareful when it comes to disclaimers when talking about licenses, trademarks, patents, etc., but I don't think it's better to not talk about them.
[00:41] <poolie> i think it's better not to say something vaguely worrying
[00:41] <persia> Which is the worrying bit?
[00:44] <poolie> the "ask counsel" bit
[00:49] <persia> I guess.  Similar to IANAL.  Anyway, off-topic, and taking to /query
[00:56] <poolie> yeah, i guess it is just "ianal"
[00:56] <poolie> to me, it's a more alarming way of saying it, but i see what you mean
[01:14] <mwhudson> lifeless: hey, did you figure out your testtools error?
[01:14] <lifeless> no
[01:14] <lifeless> no idea at all
[01:14] <mwhudson> lifeless: i mean "Subject: [Launchpad-dev] Fwd: Test results: testtools => devel: FAILURE"
[01:14] <mwhudson> lifeless: i just had a WAG
[01:15] <lifeless> \o/
[01:15] <mwhudson> basically <storm expr> == anything is true?
[01:15] <mwhudson> so perhaps some change in testtools now tests types, or something?
[01:15] <mwhudson> i.e. the tests were already bong
[01:15] <lifeless> could be
[01:15] <lifeless> I'll look more closely; thanks.
[01:16] <mwhudson> lifeless: hmm, doesn't look likely actually
[01:16] <mwhudson> but it's still worth checking i guess
[01:30] <mwhudson> lifeless: i was right, the test is broken
[01:30] <mwhudson> lifeless: and the reason your branch exposes it is that some change in testtools flips the order of the comparison
[01:31] <mwhudson> (Pdb) p operator.eq(matchee, matcher.expected)
[01:31] <mwhudson> False
[01:31] <mwhudson> (Pdb) p operator.eq(matcher.expected, matchee)
[01:31] <mwhudson> <storm.expr.Eq object at 0xcc17d90>
[01:31] <mwhudson> \o/ ?
[01:31] <lifeless> hah
[01:32] <wgrant> Heh.
[01:32] <mwhudson>         self.assertEqual(UTC_NOW, branch.next_mirror_time)
[01:32] <mwhudson> sigh
[01:33] <mwhudson> should be
[01:33] <mwhudson> self.assertSqlAttributeEqualsDate(branch, 'next_mirror_time', UTC_NOW)
[01:33] <mwhudson> i guess
[01:33] <mwhudson> lifeless: i guess you can fix that in your branch?
[01:34] <lifeless> otp
[01:34] <mwhudson> k
[01:34] <mwhudson> the other failure is probably something similar
[01:35] <thumper> are we still importing the with statement from the future?
[01:38] <lifeless> no need
[01:38] <thumper> so we are 2.6 everywhere now
[01:38] <thumper> ?
[01:38] <lifeless> yes
[01:38] <lifeless> no
[01:38] <thumper> w00t
[01:38] <lifeless> I lied
[01:38] <thumper> was there an email?
[01:38] <lifeless> future still
[01:38] <lifeless> yes
[01:38] <wgrant> There were a few emails, all contradicting each other.
[01:38] <lifeless> thread with mars and me
[01:39] <thumper> so...
[01:39] <lifeless> from __future__ is still needed
[01:39] <thumper> still future
[01:39] <thumper> ok
[01:39] <wgrant> The PQM chroot?
[01:39] <wgrant> Or is there something else?
[02:18] <LPCIBot> Project db-devel build (82): STILL FAILING in 3 hr 44 min: https://hudson.wedontsleep.org/job/db-devel/82/
[02:18] <LPCIBot> * Launchpad Patch Queue Manager: [r=abentley][ui=None] Add database patch for branch merge queues
[02:18] <LPCIBot> * Launchpad Patch Queue Manager: [r=rockstar][ui=none][bug=662664] Ensure recipe builds run only under
[02:18] <LPCIBot> Xen
[02:30] <lifeless> mwhudson: thanks for looking into that
[02:31] <mwhudson> lifeless: s'ok -- i hate mysteries :)
[02:31] <thumper> wild stab in the dark, but I thought "bin/test -m lp.code.windmill" would only run the code windmill tests
[02:32]  * thumper thinks
[02:32] <thumper> actually I think there is a bug layer one there
[02:32] <thumper> canonical.testing.layers.LayerIsolationError: Test didn't reset the socket default timeout.
[02:32] <thumper> anyone seen that running windmill tests?
[02:34] <lifeless> http://i.imgur.com/IOBPq.jpg
[02:37] <thumper> I wonder how many page loads were needed to get the right ad :)
[02:38] <mwhudson> :)
[02:38] <thumper> ERROR DISABLED: Test left new live threads: [<Thread(Thread-3076, started daemon 140448791578384)>] ?!?
[02:41] <wallyworld_> thumper: bin/test -vvt code.windmill.tests has been working for me
[02:41] <wallyworld_> but yours should have worked too i guess
[02:41] <wallyworld_> i'm also seeing the live threads stuff
[02:53] <lifeless> bombs away
[02:56] <wgrant> An interesting proposal.
[02:56] <lifeless> 3 minutes, you're running slow
[02:57] <wgrant> Shh.
[02:57] <lifeless> ;)
[03:00] <lifeless> wgrant: any more feedback than that ?
[03:01] <wgrant> lifeless: I think it's a good idea to try it.
[03:01] <wgrant> As long as the scope is clear.
[03:01] <wgrant> But I haven't really got time to think about it too much at the moment.
[04:35] <mars> wgrant, if you are talking about lifeless' reviews experiment: the scope is 'anything that requires craftsmanship', which basically means even a single line of code: SQL, Python, or otherwise.
[04:50] <lifeless> mars: hi
[04:51] <lifeless> mars: I think you're projecting
[04:51] <lifeless> mars: I realise my examples can be categorised that way, but its not what I meant.
[04:51] <lifeless> s/project/something/
[04:59] <mars> lifeless, yes, but as wgrant pointed out, drawing the line can be difficult.  I just pointed out a minimum point at which the line may be drawn.  Hopefully discussion will take us beyond that.
[05:00]  * mars signs off for the night - bye!
[05:59] <rockstar> wallyworld_, I'm tempted to just say "forget the test, just land the code"
[06:03]  * persia suspects that is a recipe for summoning for certain well-meaning over-knowledgeable folk
[06:11] <rockstar> persia, then I punt to them.  :)
[06:26] <lifeless> rockstar: whats the question ?
[06:28] <LPCIBot> Project devel build (130): STILL FAILING in 3 hr 56 min: https://hudson.wedontsleep.org/job/devel/130/
[06:29] <wallyworld_> rockstar: i'd still like to know what's going on
[06:29] <wallyworld_> lifeless: the issue is that some ajax stuff works fine but fails under windmill
[06:32] <lifeless> ugh
[06:32] <lifeless> that sucks
[06:33] <wallyworld_> yep. basically Y.io(...) is just plain failing
[06:33] <lifeless> wallyworld_: btw
[06:33] <lifeless> bug 663068
[06:33] <_mup_> Bug #663068: Remove hard coded test urls <launchpad-foundations> <Launchpad itself:New for wallyworld> <https://launchpad.net/bugs/663068>
[06:33] <lifeless> the tag is unusual
[06:33] <lifeless> normally bugs are put on the launchpad-foundations *project* with separate tags ;)
[06:34] <wallyworld_> bugger. i stuffed that one up. sorry
[06:34] <lifeless> no worries
[06:34] <lifeless> I've fixed
[06:34] <wallyworld_> thanks
[06:34] <lifeless> but I thought something might have got lost in translation
[06:34] <lifeless> so letting you know
[06:34] <lifeless> bug 663068
[06:34] <_mup_> Bug #663068: Remove hard coded test urls <paralleltest> <Launchpad Foundations:New for wallyworld> <https://launchpad.net/bugs/663068>
[06:34] <lifeless> see^
[06:35]  * wallyworld_ files mental note
[06:35] <lifeless> I think it would be nice to have all bugs on the launchpad project
[06:35] <lifeless> but thats not the current practice ;)
[06:36] <wallyworld_> well, it would make it easier perhaps
[08:18] <lifeless> mwhudson: looks interesting vis a vis our discussion earlier - http://pypi.python.org/pypi/withrestart/0.2.7
[08:19] <lifeless> not sure its tasteful though ;)
[08:36] <adeuring> good morning
[08:54] <lifeless> jelmer: hi
[08:55] <lifeless> jelmer: the thing you CP'd. what rev of trunk is it? (by CPing you've kind of blocked all deployments till we catch up, now that we're on the new process)
[08:55] <lifeless> jelmer: I want to assess what we should do now
[08:56] <lifeless> hmmm
[08:56] <lifeless> 9pm
[08:56] <lifeless> time to write some code
[08:56] <lifeless> [one of those days]
[09:23] <jml> good morning
[09:24] <bigjools> morning
[09:28] <lifeless> hmmm, leaked config dirs
[09:30] <jml> lifeless: any traction on your testtools => devel fail?
[09:30] <lifeless> some
[09:30] <lifeless> its a storm expr being used in the test
[09:30] <lifeless> so its dependent on the magic ordering of ==
[09:31] <jml> lifeless: oh wow.
[09:31] <lifeless> we need to rewrite those tests, shallowly, or thoroughly
[09:31] <jml> lifeless: ok
[09:31] <lifeless> mwhudson dug into it, and phrased it as broken.
[09:31] <lifeless> :)
[09:31] <jml> lifeless: I retract any offer of help that my enquiry might have implied :P
[09:31] <lifeless> hah!
[09:34] <lifeless> jml: did you see https://bugs.edge.launchpad.net/launchpad-foundations/+bug/662519 ?
[09:34] <_mup_> Bug #662519: lp test startup exhausts dev/random <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/662519>
[09:34] <jml> no
[09:34] <jml> but I love that title :)
[09:34] <lifeless> it will add some hilarity to your day
[09:35] <jml> of all of the problems that I would have guessed we had...
[09:54] <lifeless> jml: well, the thing about surprises
[09:57] <bigjools> wgrant: hello
[09:59] <wgrant> bigjools: Hi.
[09:59] <bigjools> wgrant: evening.  I'm trying to test your fix for bug 629921
[09:59] <_mup_> Bug #629921: Archive:+packages with empty name search does like '%%' search. <pg83> <qa-needstesting> <timeout> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/629921>
[10:00] <wgrant> bigjools: Navigate to the page ++oops++, filter to Superseded packages, verify that the query isn't insane.
[10:00] <bigjools> wgrant: it's timing out
[10:00] <wgrant> ... hah.
[10:00] <bigjools> I don't need a ++oops++ :)
[10:01] <wgrant> Declivitous?
[10:02] <bigjools> yes
[10:03] <wgrant> It sounds negative.
[10:03] <bigjools> it means "sloping down steeply"
[10:04] <wgrant> Ah.
[10:04] <bigjools> we're half way through the development
[10:04] <wgrant> Optimistic!
[10:05] <bigjools> not really
[10:13] <wgrant> bigjools: I have the first three lucilleconfig-elimination branches proposed, if you'd like to eyeball them.
[10:13] <wgrant> I still need to work out what to do about the germanium temproot issue, though.
[10:14] <bigjools> wgrant: I don't have the time right now - I can look probably Thursday if you can sit on it a while
[10:14] <wgrant> bigjools: Sure.
[10:14] <wgrant> No urgency.
[10:14] <bigjools> yeah :)
[10:15] <bigjools> I am doing QA on your other publisher change at the moment
[10:15] <wgrant> Ah, thanks. Sorry for leaving that to you, but I can't really do it myself yet...
[10:15] <bigjools> indeed
[10:16] <bigjools> right now you do increase my workload quite a lot :)
[10:19] <jml> https://code.edge.launchpad.net/~jml/launchpad/codebrowse-port-config/+merge/38812 – need a review
[10:19] <jml> it's part of getting qastaging up
[10:19] <lifeless> +1
[10:19] <jml> lifeless: heh heh
[10:19] <jml> lifeless: thanks
[10:20] <jml> lifeless: can you please rubber stamp for landing tools.
[10:20] <lifeless> I did
[10:21] <bigjools> wgrant: the next thing you might want to look at that would be most beneficial if you're feeling like doing more work is to do that query speedup in the domination
[10:22] <bigjools> the PPA publishing cycle is slow :/
[10:22] <wgrant> bigjools: Yeah, I need to do that. But I want to try to fix the actual query.
[10:22] <bigjools> wgrant: fix?
[10:22] <wgrant> bigjools: To not use a temp table.
[10:22] <jml> lifeless: ta
[10:22] <bigjools> wgrant: there's nothing wrong with that
[10:22] <wgrant> bigjools: A big part of the publisher slowness (at least locally) is the regular cache purging.
[10:23] <wgrant> Removing that cuts a primary run from >1min to <10s.
[10:23] <bigjools> are you talking about the storm cache?
[10:23] <wgrant> (every (distroarchseries, pocket) domination clears the Storm cache twice)
[10:23] <bigjools> right
[10:24] <bigjools> the first question is - why does it need to flush it
[10:24] <wgrant> I'm not sure hwo dumb our cache eviction strategy is.
[10:24] <wgrant> It may not evict at all.
[10:25] <bigjools> I'm sure lifeless will have something useful to contribute :^)
[10:26] <lifeless> beware the tides, wait oh what ?
[10:26] <lifeless> storm cache depends on the cache thats installed
[10:27] <lifeless> we have a custom one
[10:27] <lifeless> I don't recall the details about when its installed
[10:27] <lifeless> how does the sql temp table relate to storm cache flushing?
[10:27] <bigjools> I no approximately nothing
[10:27] <bigjools> jeez
[10:27] <bigjools> know*
[10:27] <wgrant> lifeless: I don't think it should relate, besides being in roughly the same piece of code.
[10:27] <bigjools> lifeless: this is something that cprov may have talked to you about in the past when we started using storem
[10:28] <lifeless> ok, so I was confused ;)
[10:28] <lifeless> bigjools: which this do you refer to ?
[10:28] <lifeless> wgrant: so are we talking storm performance or the query ?
[10:28] <bigjools> lifeless: caching on  the publisher
[10:28] <wgrant> The publisher drops caches.
[10:28] <wgrant> Frequently.
[10:29] <lifeless> ok, as a correctness thing I guess? you're changing values in the db
[10:29] <bigjools> lifeless: see judgeAndDominate() inside lib/lp/archivepublisher/domination.py
[10:29] <bigjools> which calls clear_cache()
[10:29] <wgrant> That code isn't *too* painful any more.
[10:29] <wgrant> So you won't go insane from reading it.
[10:30] <wgrant> I don't think it's correctness.
[10:30] <wgrant> We create a temp table, read in a list of publications from it, then use Storm operations on those.
[10:30] <bigjools> lifeless: I've no idea why it needs to flush_database_updates() then clear_current_connection_cache() then gc.collect()
[10:31] <bigjools> I have a vague recollection of it blowing memory into small pieces
[10:31] <wgrant> bigjools: Because it wants to take 20 minutes :D :D :D
[10:31] <bigjools> yeah
[10:31] <wgrant> I don't see why it would be dealing with a particularly huge number of objects.
[10:32] <lifeless> the conection cache so cccc is store.invalidate
[10:32] <wgrant> At worst it should be a few thousand during an autosync run.
[10:32] <lifeless> blah late night typing
[10:32] <lifeless> so you could do this in a more direct form
[10:32] <lifeless> drop the store object
[10:32] <lifeless> put a new one in its place
[10:33] <lifeless> wgrant: I agree - but - numbers will help.
[10:33] <lifeless> lets get peak memory utilisation logged (sys.utimes or whatever)
[10:33] <bigjools> it is logged
[10:33] <lifeless> great
[10:34] <lifeless> hows it look at the moment?
[10:34] <bigjools> actually - maybe it's not logged for domination
[10:35] <bigjools> ah it's deathrow that does it
[10:35] <bigjools> it uses process_in_batchers
[10:35] <lifeless> bah
[10:35] <bigjools> sigh
[10:35] <lifeless> librarian hates
[10:35] <bigjools>  process_in_batches
[10:36] <lifeless> psycopg2.OperationalError: FATAL:  database "launchpad_ftest" does not exist
[10:36] <wgrant> bigjools: Doesn't lots of publisher stuff use that?
[10:36] <wgrant> Or is PublishingTunableLoop a separate thing?
[10:36] <wgrant> There are a few things like that.
[10:36] <bigjools> wgrant: just deathrow and ftparchive it seems
[10:36] <wgrant> Ah, yeah.
[10:36] <bigjools> when overriding, for the latter
[10:36] <wgrant> Not for generating file lists too?
[10:37] <bigjools> ah yes
[10:37] <wgrant> Yeah.
[10:37] <wgrant> I've been in that code a bit lately...
[10:37] <bigjools> :)
[10:37] <bigjools> lucky you
[10:38] <wgrant> Hey, deleting crappy code makes a good break from finishing my last two projects evaar.
[10:38] <bigjools> I am running the ubuntu publisher on DF at the moment, when it finishes in about an hour or so we'll see if your other fix worked
[10:38] <bigjools> if the publisher code is a break, I'd hate to see what your projects look like
[10:39] <wgrant> bigjools: The file list generation really takes most of that time?
[10:39] <wgrant> Not a-f itself?
[10:39] <wgrant> I do not understand :(
[10:41] <bigjools> wgrant: domination is SLOW
[10:42] <wgrant> bigjools: The three minutes it takes on production is also slow. But I guess your definition is different.
[10:42] <bigjools> dogfood is spethial
[10:44] <lifeless> jml: hey, whats 'stnadard' to log something in twistd
[10:44] <bigjools> 2010-10-19 09:38:42 DEBUG   Sorting packages...
[10:44] <bigjools> 2010-10-19 09:43:39 DEBUG   Dominating packages...
[10:44] <bigjools> wgrant: ^
[10:44] <wgrant> bigjools: Ah, so only twice production's slowness.
[10:44] <wgrant> Not so bad.
[10:44] <jml> lifeless: log.msg(), where log = twisted.python.log
[10:54] <lifeless> jml: nothing gets logged
[10:55] <lifeless> jml: I guess the log only shows after the tac is parsed?
[10:55] <jml> lifeless: that sounds vaguely familiar
[10:55] <jml> lifeless: but I don't know for sure.
[10:56] <lifeless> I wonder how I can easily make this easier to debug
[10:56] <jml> lifeless: also, our tachandler nonsense might interfere with early logging
[11:17] <bigjools> wgrant: filelist generation is the slowest part on dogfood it seems.  Probably production too.
[11:18] <wgrant> bigjools: Better than before cprov fixed it a year ago...
[11:20] <wgrant> bigjools: Prod file list calculation seems to take just a couple of minutes.
[11:21] <wgrant> Then a-f takes ~15 minutes.
[11:22] <wgrant> Debian recently parallelised a-f calls.
[11:49] <LPCIBot> Project db-devel build (83): STILL FAILING in 3 hr 54 min: https://hudson.wedontsleep.org/job/db-devel/83/
[12:04] <deryck> Morning, all
[12:23] <jml> bigjools: btw, now that the first cut of testtools twisted support has landed, I'm blocking on the mega-branch for converting all our trial tests & deleting the twisted layers
[12:23] <bigjools> jml: sweet
[12:23] <bigjools> jml: are you going to make the change so I can merge it?
[12:24] <jml> bigjools: no, I'm going to wait until your branch lands before I try to land on trunk
[12:24] <bigjools> ok
[12:24] <jml> bigjools: Launchpad doesn't have the testools twisted support yet, it's only landed in testtools trunk.
[12:24] <wgrant> bigjools: Has dogfood blessed me yet?
[12:25] <bigjools> wgrant: hahaha no
[12:25] <wgrant> :(
[12:25] <bigjools> 2010-10-19 09:50:16 DEBUG   Calculating binary filelist.
[12:25] <bigjools> 2010-10-19 11:16:49 DEBUG   Iteration 9 (size 10000.0): 518.693 seconds
[12:25] <jml> bigjools: I'm working on the existing twisted tests in LP in a branch now, working out the kinks with my perfect-in-theory twisted support code
[12:25] <bigjools> 2010-10-19 11:25:25 DEBUG   Iteration 10 (size 10000.0): 516.320 seconds
[12:25] <wgrant> ...
[12:25] <wgrant> That is utterly pathetic.
[12:25] <bigjools> dogfood is SLOW
[12:26] <bigjools> jml: coolio
[12:26] <wgrant> Maybe qastaging will be better.
[12:26] <lifeless> wgrant: qastaging is on the staging machine, for now.
[12:26] <lifeless> same db server, same appserver
[12:26] <wgrant> Well, I don't know how fast asuka is these days.
[12:26] <lifeless> 450 milliparsecs
[12:27] <wgrant> But it can't be quite mawson-slow.
[12:27] <bigjools> lifeless: less than twelve parsecs, surely.  :)
[12:39] <lifeless> indeed
[12:39] <lifeless> hhmm, nearly 1am
[12:39] <lifeless> psql -l | grep launchpad_ftest_[\[:digit:\]] | awk '{ print $1 }' | xargs -n1 dropdb
[12:39] <lifeless> ^ folk will need that till we shake the bugs out of the parallel stuff :<
[12:45] <jml> lifeless: bah, kiwis are nocturnal anyway
[12:45]  * jml is off to lunch & errands
[12:48] <lifeless> I'm amazed some of this stuff ever worked
[13:03] <lifeless> I hate globals.
[13:03] <lifeless> there, I said it.
[13:04] <persia> Needed to be said, really.
[13:04] <lifeless> lib/canonical/lp/__init__.py takes the database name found at process startup and caches it indefinitely.
[13:04] <persia> *choke*
[13:05] <lifeless> that interacts badly with changing it in the test runner.
[13:09] <wgrant> There are a lot of similar pieces of evil.
[13:10] <lifeless> wgrant: check this out
[13:10] <wgrant> And then you check their history and find they were added in early 2005 and probably haven't even been looked at since.
[13:10] <lifeless> sqlbase.connect_string
[13:10] <lifeless> config.rw_main_master
[13:10] <lifeless> then lp.dbname
[13:10] <lifeless> which is always set...
[13:10] <lifeless> to dbconfig.main_master
[13:10] <wgrant> lifeless: There used to be a non-main store.
[13:10] <lifeless> yes
[13:10] <lifeless> but its circular
[13:10] <lifeless> scripts options stash it there
[13:10] <lifeless> and then its read back
[13:11] <wgrant> Ah.
[13:11] <lifeless> so a dynamic config system becomes a static variable
[13:11] <lifeless> facepalm
[13:12] <wgrant> I'm glad you've found rationale sufficient to justify destruction of this madness.
[13:12] <wgrant> Albeit at 1am.
[13:17] <wgrant> bigjools: https://edge.launchpad.net/ubuntu/+source/icecc/0.9.6-1/+build/2000412 is FAILEDTOUPLOAD but has no upload log.
[13:17] <wgrant> Is this meant to happen?
[13:17] <wgrant> ('tis breaking scripts)
[13:18] <bigjools> wgrant: yes - jelmer fixed stuff that was stuck in UPLOADING
[13:18] <lifeless> \o/ appserver running up against launchpad_ftet_22437
[13:18] <wgrant> bigjools: Ah, so the old stuff got set to FAILEDTOUPLOAD without a log?
[13:18]  * wgrant makes the scripts cope.
[13:18] <bigjools> yes - there was no log available.
[13:18] <wgrant> :(
[13:18] <bigjools> it was only 200 or so
[13:19] <bigjools> annoying, but unavoidable because of the bug
[13:19] <wgrant> Yup.
[13:27]  * bigjools puts the finishing touches to the new buildd-manager and rejoices
[13:28] <bigjools> assuming jml likes my changes of course :)
[13:28] <lifeless> jml: when you return
[13:29] <wgrant> What does it actually change?
[13:29] <lifeless> my databasefixture branch has a couple of somewhat nontrivial alterations added to it
[13:29] <wgrant> Makes dispatching properly asynchronous?
[13:29] <lifeless> jml: ^ - would appreciate another glance please.
[13:29] <bigjools> wgrant: yes
[13:29] <bigjools> wgrant: I still need to make file fetching asynchronous but that's for another branch and day
[13:30] <bigjools> the most important thing is that the code is now vastly simplified
[13:30] <wgrant> Right.
[13:30] <bigjools> I'm particularly pleased that all the error handling is done in one place now
[13:39] <gmb> mars: Did you land that test helper for setting / unsetting feature flags?
[13:40] <gmb> ISTR you mentioning it in an email thread ages ago but I can't find any details.
[14:02] <LPCIBot> Project devel build (131): STILL FAILING in 3 hr 51 min: https://hudson.wedontsleep.org/job/devel/131/
[14:31] <cr3> hi folks, is there a convention for separating names consisting of more than one word in launchpad urls? lets say I have a new target called "foo bar", should the url be "foo-bar", "foo_bar", "foobar" or is the convention to find whatever way to just have one word?
[14:33] <jml> lifeless: will do.
[14:33] <jml> bigjools: I'll take a look after looking at lifeless's branch
[14:34] <bigjools> jml: I found some test failures caused by the most recent revision, ping me before you look as I'm fixing them now
[14:35] <jml> bigjools: will do.
[14:43] <bigjools> jml: ok it was a trivial piece of pebkac.  However I've got a weird error when it tears down the layers, have you got any idea what's wrong here? http://pastebin.ubuntu.com/516258/
[14:48] <jml> bigjools: something is screwing with the layers. otp. things to try include checking for recent changes to devel that aren't in stable.
[14:48] <bigjools> righto
[14:48] <jml> (or bad revisions you've merged in from devel)
[14:48] <bigjools> oh the shame
[14:55] <mars> is the use of the rubber-stamp 'rs' review documented anywhere?
[15:00] <jml> mars: if you can't find it, then the answer is "functionally, no"
[15:02] <mars> jml, I did not even know what it meant until Deryck mentioned it in his listmail.  I thought it was just some funny typo people used from time to time.
[15:02] <jml> bigjools: maybe r11734 is a culprit?
[15:02]  * jml otp again
[15:03] <bigjools> jml: I'd lay a wager on it
[15:15] <jam> abentley: I think you have some private lp branches available. I ran your script, and only see 4.3k branches
[15:16] <abentley> jam: quite possible.  As a member of the code team, I have read/write access to everything.
[15:48] <LPCIBot> Project parallel-test build (6): STILL FAILING in 2 hr 7 min: https://hudson.wedontsleep.org/job/parallel-test/6/
[15:49] <bigjools> jml: latest devel made it go away.  The branch is ready for your perusal.
[16:01] <gmb> mars: Did you land that test helper for setting / unsetting feature flags?
[16:01] <gmb> ISTR you mentioning it in an email thread ages ago but I can't find any details.
[16:02] <gmb> I'm writing tests at the moment and the whole ignore = getFeatureStore().add(...) dance is horrendous.
[16:02] <mars> gmb, no, the branch has two small failing tests in it.  I have to make time to fix them
[16:02] <mars> gmb, :)
[16:03] <gmb> mars: Which branch is it? I can try to take a look at the failures if you'd like.
[16:04] <mars> gmb, sure!  lp:~mars/launchpad/add-profiling-feature-flag
[16:04] <mars> gmb, the failing tests are: bin/test -cv lp.services.profile
[16:05] <mars> gmb, just two simple ones IIRC
[16:06] <mars> gmb, I changed the way that the profiling instruments know that they should be turned on and measuring the request.  The old code had some tests for 'instruments on' and 'instruments off', and they broke when I changed the way that the on/off switch works.
[16:07] <gmb> mars: Okay, I'll take a look.
[16:07] <mars> gmb, the new code for the flags helper and fixture is in lp.services.features.helpers
[16:07] <gmb> Cool.
[16:13] <mars> make that lp.services.features.testing
[16:23] <abentley> jam, I've updated https://dev.launchpad.net/Code/BranchRevisions but it doesn't list you as an email recipient.
[16:24] <jam> abentley: thanks. I had added "BranchRevisions" but maybe it needs a full match rather than a subset
[16:26] <abentley> jam, anyhow, I'd really appreciate your feedback on it.
[16:37] <jml> bigjools: are you intending anything more for this branch?
[16:37] <bigjools> jml: nothing planned
[16:40] <jml> bigjools: reviewed
[16:41] <bigjools> jml: thanks, will look shortly
[16:42] <jam> abentley: I'll have some more interesting numbers for you in a couple minutes
[16:42] <jam> I ran the script and grabbed the revisions for all those branches
[16:42] <abentley> jam, cool.  Hard data is good.
[16:42] <jam> so at least for the ~4k branches I can see, I can tell you how the db will hold it
[16:55] <jam> abentley: so the import is done, how would you like the numbers?
[16:56] <jml> huh
[16:56] <jml> how did that happen
[16:56] <abentley> jam,  maybe the best place is the BranchRevision page?
[16:56] <jml> bigjools: fwiw, I'm going to be off IRC all tomorrow
[16:57] <jam> abentley: sure
[17:19] <jam> abentley: updated, and you weren't listed as notified
[17:19] <jam> (neither was I, but I'm hoping that is because it is me)
[17:21] <abentley> jam, perhaps I should subscribe :-)
[17:24] <abentley> jam, to calculate the BranchRevisions, take the distance-to-null of the branch tips and sum them.
[17:24] <jam> abentley: it is the full ancestry
[17:24] <jam> not the mainline
[17:24] <jam> gdfo would be a decent approximation, but still low
[17:24] <abentley> jam, my bad.
[17:24] <jam> I'm just going to do a pass across the data, find the ancestry of each rev
[17:24] <jam> and then sum those
[17:26] <jam> dang, I have to do set operations because 2 parents don't "add" cleanly. but I'll get there
[17:32] <abentley> jam, so it looks like the ratio of branches to dotted_revno_rows improves as the number of branches increases.
[17:33] <jam> abentley: possibly. I think it depends a lot of the branches themselves
[17:33] <jam> A long-lived branch will be 'bigger'
[17:33] <jam> and can certainly skew the results more when there are fewer total branches
[17:33] <abentley> jam, true.
[17:35] <abentley> I get 410:1 with merged, 1030:1 with abandoned, 1219:1 with experimental.
[17:36] <abentley> jam, it would be interesting to sort by the revision date.  That should give a rough idea how it scales over time.
[17:37] <abentley> jam, I'd like to hear whether you think my algorithms under "use case analysis" make sense.
[17:57] <jam> abentley: so by my numbers, BranchRevision for those 4k branches is 328M rows
[17:57] <jam> your logic seems ok
[17:58] <jam> I'm not sure that walking 'backwards' using mainline_parent is the best thing to do
[17:58] <jam> specifically, as a branch changes over-time more entries will end up in the mainline_parent table
[17:58] <jam> we'll need a way to remove old entries, etc.
[17:58] <jam> that table wasn't very big, so I wasn't worried about it accumulating cruft
[17:59] <abentley> jam, walking backwards when implementing a ParentsProvider, for example?
[17:59] <jam> Well I say "backwards" I should be saying "forwards" (in the opposite direction of normal)
[17:59] <jam> the ParentsProvider seems good
[17:59] <abentley> jam, okay.
[17:59] <jam> the 'most relevant branch' is tricky
[18:00] <abentley> jam, yeah.  We might be looking at changing it so we cache the most relevant branch on the revision, and only need to calculate it if the branch goes away.
[18:01] <jam> abentley: I think that is a good start, though AIUI it can change
[18:01] <jam> specifically, if I push up a merge of your branch before you push up the branch itself
[18:02] <abentley> jam, true.
[18:02] <abentley> jam, the whole thing is a bit questionable.  The most important thing we need it for is calculating revision karma, which is only done once.
[18:03] <jam> if you are willing to start with "Branches owned by the user in this given project" then you can start with the known tips and walk backwards
[18:03] <abentley> jam, I have to meet someone for  lunch now.  TTYL.
[18:03] <jam> later
[18:35] <rockstar> Someone from foundations, I could really use your help.
[18:35] <rockstar> Er, help figuring out what's going on with the librarian.
[18:35] <mars> gary_poster, ^ ?
[18:38] <jml> rockstar: what's the problem?
[18:39] <rockstar> jml, http://pastebin.ubuntu.com/516392/ - that's usually what I expect when the librarian is still up.
[18:39] <rockstar> But I don't see it in ps, and bin/kill-test-services appears to be broken.
[18:40] <jml> rockstar: that message seems to indicate that the librarian is *not* running.
[18:41] <jml> "Librarian has been killed or has hung." and then later "[Errno socket error] [Errno 111] Connection refused" … looking at the layer code, that's coming from a failed urlopen(foo).read()
[18:41] <rockstar> jml, yeah, we don't seem to be very good at error messages.
[18:41] <sinzui> flacoste, mumble?
[18:43] <rockstar> jml, nevermind.  It seems fixing bin/kill-test-services and then running it has fixed it.
[18:43] <rockstar> jml, meaning that there was probably crap left over from the last librarian instance.
[18:43] <jml> rockstar: I think there's a bug filed somewhere about it leaving its pid file around after dying
[18:44] <rockstar> jml, ah, okay.
[18:44] <rockstar> jml, I think that bug's been around for a while, and we've been using bin/kill-test-services to workaround it.
[18:45] <rockstar> But when that script is broken, then it's a little harder to sort out.  :)
[18:50] <LPCIBot> Project devel build (132): STILL FAILING in 3 hr 54 min: https://hudson.wedontsleep.org/job/devel/132/
[18:52] <cr3> hi folks, sorry for asking again, is there a convention for separating names consisting of more than one word in launchpad urls? lets say I have a new target called "foo bar", should the url be "foo-bar", "foo_bar", "foobar" or is the convention to find whatever way to just have one word?
[18:53] <rockstar> cr3, we mostly use foo-bar
[18:53] <deryck> All my ec2 mails complain of failure when it's windmill errors due to threads.  Should I open a bug about this?
[18:54] <jml> deryck: probably yes, but I'd also check StevenK's recent emails about windmill threading errors on the list
[18:54] <deryck> jml, ok, will do.
[18:54] <lamont> jml: so if someone claims something has landed on launchpad/db-stable (which I'm gathering == launchpad/staging?), then is it fair to assert that it should also exist on lp:launchpad/devel?
[18:54] <jml> lamont: no, it's not.
[18:54] <lamont> that explains why it fails that assertion
[18:56] <lamont> jml: historically, launchpad-buildd has developed/released directly from /devel, since it's not really part of launchpad, etc, etc.  I'm wondering what else would break in breaking from that tradition, or if I should go have the person who landed his branch on /staging also land it on /devel
[18:56] <jml> lamont: branches that require db changes land on db-devel→db-stable, after we've rolled out a db update, db-stable gets merged into devel.
[18:57] <lamont> ah, interesting.  given that we tested this without db changes, I expect that he should have landed it on /devel instead then, yes?
[18:57] <jml> lamont: probably
[18:57] <lamont> or additionally, or whatever
[18:57] <jml> lamont: but I couldn't say for sure
[18:57] <lamont> can I propose someone elses branch for merging?
[18:57] <jml> lamont: you can indeed
[18:57] <cr3> rockstar: thanks, might you have an example I could use as precedent?
[18:57]  * lamont goes to dig around a bit
[18:58] <jml> lamont: fwiw, splitting buildd into its own branch & project is one of my many Launchpad someday/maybes.
[18:58] <rockstar> cr3, not off the top of my head, no. Sorry.
[18:58] <jml> anyway, I have to go now. have a great evening all.
[19:01] <lamont> https://code.edge.launchpad.net/~abentley/launchpad/detect-xen/+merge/38867 <-- rockstar, wanna approve that beasty?
[19:03] <rockstar> lamont, I reviewed that already. It looks like abentley landed it into db-devel (which is lp:launchpad)
[19:03] <lamont> right.  and I need it landed on /devel
[19:07] <rockstar> lamont, why?
[19:08] <lamont> because that's where launchpad-buildd releases from, and since it's an as-released thing when it merges, merge conflicts absolutely suck, since it means that the source in lp is wrong (relative to what's actually running)
[19:10] <lamont> rockstar: given that the two branches appear to be divergent, it's less work for me to just keep releasing from /devel rather than migrating the releases over to db-stable
[19:10] <rockstar> lamont, okay.
[19:11] <lamont> esp since the release process comes down to me changing debian/changelog and telling bigjools/whoever to "hey please land this, it's live, kthx"
[19:11] <rockstar> lamont, I'm going to go on the assumption that abentley landed it in db-devel on accident, and approve this.  If this assumption is wrong, is quite likely that abentley will thump me.
[19:12] <rockstar> (I just want you to know the risk I'm taking)
[19:13] <lamont> rockstar: understood.  tell him I made you do it.
[19:13] <lamont> 'twas tested without db changes, so I'm gonna assert that it doesn't need to be on db-stable
[19:13] <rockstar> lamont, I know abentley well enough to know he won't accept excuses.  :)
[19:14] <lamont> of course, the descriptions of the branches make even less sense, since db-stable claims to be "development" and /devel makes no such claim
[19:14] <lamont> rockstar: my fall back is to just tell him to land it there and let him wait another day to get his code rolled out.
[19:14] <lamont> > Please land your branch - after the merge pain of times past, I've learned
[19:14] <lamont> > not to deploy from anywhere other than mainline launchpad/devel.
[19:14] <lamont> The branch has landed on db-stable.
[19:15] <rockstar> lamont, devel is to db-devel as stable is to db-stable
[19:16] <lamont> so stable is for the existing db, and devel/db-devel is for the "we need a db upgrade" stuff?
[19:17] <rockstar> lamont, no, things get landed into devel, and when buildbot blesses them, they get pulled into stable.  db-devel then merges stable, and when things get blessed by buildbot, they get pulled into db-stable
[19:17] <rockstar> lamont, so, if the buildd code ever gets tests, you'll want to at least deploy from stable, since you know all the tests pass there.
[19:18] <lamont> ah, yeah. tests would be cool
[19:18] <lamont> we've been doing the "develop and test on dogfood, smack it into devel" process.
[19:18] <lamont> sadly, sometimes devel moves during that process, and then I cry a little
[19:19] <rockstar> abentley, could you please review https://code.edge.launchpad.net/~rockstar/launchpad/fix-kill-test-services/+merge/38870
[19:19] <lamont> rockstar: and good to know, thanks.  I've mostly been flying blind wrt branches
[19:37] <rockstar> abentley, I can chat whenever you can.
[19:37] <lifeless> morning
[19:45] <LPCIBot> Project db-devel build (84): STILL FAILING in 3 hr 57 min: https://hudson.wedontsleep.org/job/db-devel/84/
[19:51] <lifeless> deryck: ping
[19:51] <deryck> hi lifeless
[19:55] <lifeless> henninge: hi
[19:56] <lifeless> deryck: will come back to you :)
[19:56] <lifeless> henninge: I haven't checked all my mail yet, but as we have only a brief overlap
[19:57] <lifeless> henninge: ah, echannel
[19:57] <deryck> lifeless, ack :-)  I'll be here awhile longer.
[19:58] <mwhudson> lifeless: re http://pypi.python.org/pypi/withrestart/0.2.7, yeah, i've done something like that too, but as it's not part of the normal protocol in python programs it's kinda useless
[19:58] <lifeless> mwhudson: yeah
[19:59] <mwhudson> i gave a talk at .... accu? about this once
[19:59]  * mwhudson hunts for slides
[19:59] <mwhudson> http://www.nhplace.com/kent/Papers/Condition-Handling-2001.html <- the paper i was talking about
[20:11] <lvh> is there a reasonable way of uninstalling laucnhpad after having been installed with rocketfuel-setup
[20:20] <mwhudson> i guess reading the script and undoing what it does
[20:20] <mwhudson> it's not _that_ bad for system pollution is it?  some /etc/hosts changes, an apache site, a ppa + some packages installed
[20:23] <abentley> rockstar: chat?
[20:23] <rockstar> abentley, yes please!
[20:24] <lifeless> lvh: rm -rf / ?
[20:24] <jam> abentley: would you be interested in a copy of the sqlite db? It is about 33MB after bzip2
[20:24] <lvh> mwhudson: Yeah, no, it's not very terrible
[20:24] <abentley> jam: sure.
[20:24] <lvh> mwhudson: I was just hoping there was a rocketfuel-unsetup
[20:24] <mwhudson> lvh: nope
[20:25] <abentley> jam: I don't actually understand what to do to make history-db work locally.
[20:25] <rockstar> lvh, launchpad is like the Hotel California. You can never leave.
[20:25] <lvh> Yaay.
[20:25] <lvh> at least I'm glad I didn't do it on my development box then
[20:27] <jam> abentley: #1 install it as a bzr plugin ("bzr branch lp:bzr-history-db ~/.bazaar/plugins/history_db)
[20:32] <lifeless> deryck: ok, hey
[20:32] <deryck> lifeless, hey hey
[20:33] <lifeless> I'm wondering if you'd be up for a voice chat
[20:34] <lifeless> deryck: ^
[20:35] <deryck> lifeless, I don't mind a voice chat.  but having to work and watch my youngest while Wendy picks up my oldest.
[20:35] <deryck> so it's loud here. ;-)  Working with headphones and music while I eye the little on watching Sponge Bob :-)
[20:35] <lifeless> if that means some interrupts during a call, its fine by me
[20:35] <deryck> lifeless, ok.  it's kind of loud, too, and I can't leave the room.
[20:35] <deryck> but I'm game if you are.
[20:36] <lifeless> lets give it a shot
[20:36] <lifeless> you have skype?
[20:36] <rockstar> lvh, for context, I run launchpad in a chroot.
[20:36] <lifeless> ah yes, you do :)
[20:36] <deryck> lifeless, yup.  firing it up now.
[20:37] <lvh> rockstar: I ran it in a VM but it was dead slow
[20:37] <lvh> rockstar: turns out the reason why wasn't really related to it being ran in a VM
[20:37] <lvh> so I ran it on a spare physical machine
[20:38] <abentley> rockstar: https://dev.launchpad.net/Code/BranchRevisions
[20:38] <mwhudson> wow, https://dev.launchpad.net/Code/BranchRevisions suggests that half of the rowns in the BranchRevision table are from Launchpad
[20:44] <mbarnett> wgrant: ping.
[20:53] <gary_poster> hey sinzui, is there a way to add members to teams if the member AJAX thing is timing out too much?
[20:53] <sinzui> gary_poster, yes, the underlying view
[20:54] <jkakar> Hi. :)
[20:54] <jkakar> I continue to be basically unable to use any popup dialog in Launchpad.  They all timeout for me, almost all the time.
[20:54] <jkakar> I've tried edge and not-edge, same result.
[20:54] <jkakar> I've tried Firefox and Chrome, same result.
[20:55] <jkakar> Hopefully all my attempts are creating timeout-juice that will bring these issues up on some report or another. :)
[20:55] <sinzui> gary_poster, append +addmember to the team url
[20:55] <jkakar> As a workaround I've been using /+request-review on merge proposals and the bug task disclosure triangle thing on bug reports.
[20:55] <jkakar> sinzui: Ah, thanks!
[20:56] <gary_poster> wunderbar sinzui, thank you
[20:56] <jkakar> sinzui: I should have looked at the status bar, d'oh.  Anyway, it works, thanks a lot.
[20:56] <sinzui> jkakar, gary_poster. EdwinGrubbs is doing SQL analysis right now to fix the issue
[20:56] <jkakar> sinzui: Wicked, thanks!
[20:56] <gary_poster> yay sinzui, thank you :-)
[20:56] <gary_poster> and yay EdwinGrubbs, I should say
[20:58] <sinzui> jkakar, gary_poster, last week I used an API script to set a user.
[20:59]  * jkakar hugs sinzui
[20:59] <gary_poster> :-)
[21:00] <thumper> morning
[21:12] <EdwinGrubbs> sinzui: on that topic, do you know why there is a Person.account and an EmailAddress.account foreign key?
[21:13] <sinzui> EdwinGrubbs, : gary_poster:  historical baggage. I believe we intend to remove both
[21:14] <gary_poster> EdwinGrubbs: step 5 of https://dev.launchpad.net/LEP/OpenIdRoadmap
[21:14] <EdwinGrubbs> sinzui: if those are both removed, how can I tell if a person is active? Do I just check that Person.merged is false?
[21:15] <sinzui> EdwinGrubbs, that is one reason why person.account still exists
[21:16] <gary_poster> EdwinGrubbs: maybe I misunderstand the question, but part of # 5 is "Remove all Account and https://dev.launchpad.net/EmailAddress records not linked to a Person, reducing the number of Account records on the system by 90%."
[21:16] <gary_poster> So I would think that the answer is that, when we drop the EmailAddress.account, it will no longer be an issue.
[21:16] <sinzui> EdwinGrubbs, 18 months ago we wanted to separate account and person and host SSO. 6 months ago it was decided that we do not want SS0 and account separation cause too many bugs
[21:17] <gary_poster> If I'm wrong, we should probably clarify that in that wiki page, and the associated bug
[21:18] <EdwinGrubbs> sinzui, gary_poster: so for the time being can I assume that I want to check whether an person is active by looking at Person.account.status as opposed to Person.email.account.status, which the vocab is using now.
[21:18] <sinzui> EdwinGrubbs, excellent question!
[21:18] <EdwinGrubbs> I'm waiting for an excellent answer.
[21:19] <sinzui> Edwin. before Account email address status=4 was the way we knew a user was active. SSO wrongly lets users with any email status to login
[21:20] <gary_poster> um
[21:20] <sinzui> EdwinGrubbs, ^ So account is the only way to check, but the who authentication system is untrust worthy see the daily oopses and weep
[21:21] <gary_poster> sinzui, is this one of the balls I have dropped?  If so, please take this opportunity to kick me in an appropriate direction
[21:21] <gary_poster> appropriately helpful to getting something fixed, is what I had in mind :-)
[21:22] <sinzui> EdwinGrubbs, It is *still* our policy that all users must have a preferred email address and we want that check to suffice. We have dozens of checks in the code that require the email address.
[21:22] <sinzui> gary_poster, yes, the ball dropped
[21:24] <sinzui> EdwinGrubbs, Most cases where we pick out a person is to send notification...email. We know the user must have a preferred email. The mailing code does not check account. it knows not to send email to any user without a preferred email address
[21:24] <rockstar> wallyworld__, thumper, stand up?
[21:26] <thumper> rockstar: wallyworld__ won't be up yet
[21:26] <thumper> rockstar: did you want to talk to me?
[21:28] <gary_poster> sinzui, not clear yet, but I have call.  will ping you later to make sure I understand
[21:29] <sinzui> gary_poster, I said this in the user question I answered two weeks ago for you. No user should be permitted to authenticate if there is no email address
[21:31] <sinzui> The None error in +editemails and openid url errors that we see every day in oopses are when authentication falls over
[21:31] <EdwinGrubbs> sinzui: when a person is deactivated, is the preferred email removed as well as the account being deactivated?
[21:31] <sinzui> EdwinGrubbs, yes
[21:31] <sinzui> Edwin ResetPassword used to fix the address. That does not happen any more.
[21:33] <EdwinGrubbs> sinzui: ok, so it should be safe for me to check if a person has a preferred email to determine whether it is a valid person in the vocabulary. It would only cause problems for users that have active accounts but no preferred email address.
[21:33] <sinzui> EdwinGrubbs, I think another way of saying that is that the other users are broken and there is not point in pretending that they belong in ValidPersonVocabulary
[21:34] <EdwinGrubbs> ok
[21:34] <sinzui> EdwinGrubbs, I assume teams are handled differently
[21:35] <sinzui> EdwinGrubbs, is this issue about ValidPersonVocabulary?
[21:35]  * sinzui wonders if that object really exists
[21:37] <lifeless> gary_poster: hi
[21:37] <gary_poster> lifeless on call, ping
[21:37] <gary_poster> I mean will ping you
[21:37] <lifeless> gary_poster: wondering if you wanted a catchup
[21:37] <lifeless> yeah sure
[21:37] <lifeless> I'll be here ;)
[21:38] <sinzui> EdwinGrubbs, ValidPersonVocabulary must only contain users with preferred email addresses. Deactivated and Suspended users have NEW addresses. Unactivated users have NEW addresses too.
[21:39] <wgrant> mbarnett: Hi.
[21:40] <mbarnett> wgrant: heya.  i have a question for you, but probably better in pm.
[21:44] <rockstar> thumper, I was talking with abentley, but my mental clock was an hour ahead.
[21:45] <thumper> ok
[21:45] <deryck> Ok, I'm out now.  Later on, everyone.
[21:45] <abentley> jam, yes, I had already done step 1.
[21:46] <lifeless> StevenK: it would be awesome to gather diskio and cpu use
[21:46] <lifeless> I wonder if there is a plugin
[21:47] <jam> abentley: after that you can do something like "bzr history-db-create --db foo.db -d branch"
[21:47] <poolie> lifeless: hi there
[21:47] <poolie> hi thumper, abentley, deryck
[21:47] <poolie> and jam
[21:47] <abentley> poolie: hi.
[21:48] <thumper> hi poolie
[21:48] <jam> hi poolie
[21:48] <poolie> lifeless: it occurred to me this morning that filing RTs for things we don't intend to do soon is a kind of lean waste...
[21:50] <StevenK> lifeless: Over the life of the test run? ENOIDEA
[21:51] <abentley> jam, ISTM that we would get the best performance from mainline_parent_range if we added to existing ranges where possible.  Do you agree?
[21:51] <jam> abentley: if you add to an existing one, what about the old tip?
[21:51] <jam> the code as written, does try to always fill out full groups
[21:51] <jam> so if it would chain a 3 + 70 it creates a new 73 length group
[21:52]  * StevenK goes back for a lie down, his headache creeping back
[21:52] <abentley> jam, the old tip is still part of the range, but no longer the tip.
[21:52] <lifeless> poolie: well, we want to do it now
[21:52] <lifeless> poolie: we're resource constrained
[21:52] <lifeless> poolie: so I agree, and disagree.
[21:53] <jam> abentley: right, but you can't find the range if it is in the middle (well, not easily, or without indexing all of them)
[21:53] <poolie> i think keeping track of things you want but don't have time to start is good
[21:53] <abentley> jam, you convert the revision-id to a revision primary key, you look up the primary key in the mainline_parent table.
[21:54] <poolie> on the whole i think starting them just a little bit is normally bad, unless it's going to facilitate back-burner percolation
[21:54] <EdwinGrubbs> sinzui: yes, this is about the ValidPersonOrTeamVocabulary and the email address is ignored for teams.
[21:55] <sinzui> EdwinGrubbs, I assume is uses merged is not null
[21:56] <abentley> jam, you could even use the revision primary key as the mainline_parent primary key.
[21:56] <abentley> jam, because in this scenario each revision appears in only one revision range.
[21:57] <abentley> jam, and THAT implies that we could combine the revision table with the mainline_parent table.
[21:57] <thumper> mwhudson: know what's happening with https://code.edge.launchpad.net/~jameinel/launchpad/lp-service/+merge/37531 ?
[21:57] <EdwinGrubbs> sinzui: it checks that merged is null. Not null would mean it is merged.
[21:58] <jam> abentley: the mainline parent table intentionally de-normalizes groups into 100-ranges. I don't really see how you can put that in the revision table
[21:58] <mwhudson> thumper: it's bouncing off ec2
[21:58] <thumper> ah
[21:58] <sinzui> EdwinGrubbs, I guess I know enough to write such a vocab :)
[22:00] <abentley> jam, this would mean that recent groups would have 1-100 revisions, but they would expand until full.
[22:00] <EdwinGrubbs> sinzui: can you run these queries. Hopefully, this will be the last set. http://pastebin.ubuntu.com/516472/
[22:01] <jam> abentley: I had originally designed it so that any branch tip would find itself in the mainline_parent_range table. you're probably right that it could be done differently
[22:01] <jam> I think the index would be much bigger
[22:02] <abentley> jam, you're probably right about the index being bigger, but I don't usually worry about that.
[22:02] <sinzui> EdwinGrubbs, https://pastebin.canonical.com/38825/
[22:03] <abentley> jam: btw, I successfully ran bzr history-db-create
[22:04] <wallyworld__> morning
[22:04] <abentley> jam, at least I now understand your objection to the "relevant branch" algorithm better.
[22:07] <abentley> wallyworld__: morning
[22:08] <wallyworld__> abentley: ready for standup whenever you guys are
[22:08] <abentley> thumper: ready for standup?
[22:08] <thumper> abentley: aye
[22:11] <sinzui> \o/ I traced an oops to a spam attack.
[22:12] <lifeless> :)
[22:21] <LPCIBot> Project devel build (133): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/133/
[22:21] <LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=653122] BugSubscriptionSubscribeSelfView is
[22:21] <LPCIBot> now a LaunchpadFormView.
[22:21] <LPCIBot> * Launchpad Patch Queue Manager: [r=bac][ui=none][no-qa] Only set LPCONFIG if it wasn't set in the
[22:21] <LPCIBot> environment.
[22:21] <LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Codebrowse host and port are now
[22:21] <LPCIBot> configured with codebrowse.listen_host and codebrowse.port.
[22:24] <gary_poster> thumper: about to get on phone again, but would love your thoughts on https://bugs.edge.launchpad.net/launchpad-foundations/+bug/662912 .  Is this a known issue--am I flailing uselessly? :-)
[22:24] <gary_poster> rockstar: any chance to get a reply to mars' mail before EoD?
[22:24] <gary_poster> lifeless: off phone, ready when you are
[22:24] <_mup_> Bug #662912: staging librarian is broken for merge proposals <Launchpad Bazaar Integration:New> <Launchpad Foundations:Triaged by gary> <https://launchpad.net/bugs/662912>
[22:24] <rockstar> gary_poster, uh, I sent it this morning.  Did you not get it?
[22:24] <gary_poster> rockstar, no
[22:24] <thumper> gary_poster: I'll take a look
[22:24] <gary_poster> thanks thumper
[22:25] <rockstar> thumper, okay, I'll send again.
[22:33] <rockstar> wallyworld__, so, I didn't want to ask on the standup for fear of getting off on a tangent, but did you say your Y.io call worked in windmill when you used a post instead of get?
[22:34] <flacoste> thumper: call?
[22:34] <thumper> flacoste: yep
[22:34] <wallyworld__> rockstar: i'm still gathering evidence
[22:34] <wallyworld__> rockstar: it *appears* at this point that it *may* make a difference
[22:35] <rockstar> wallyworld__, I frakkin' hate windmill.
[22:35] <wallyworld__> rockstar:  i put it aside to work on the windmill issue - i'll let you know as soon as i make some more progress on it
[22:35] <wallyworld__> i should say - the *other* windmill issue ;-(
[22:36] <rockstar> wallyworld__, what's the other windmill issue?  I heard you talking about something with setUp.
[22:36] <wallyworld__> rockstar: yes - some seemingly innocuous refactoring has broken one and only one windmill test
[22:36] <rockstar> wallyworld__, wanna skype about it?  I'm curious.
[22:37] <wallyworld__> rockstar: ok
[23:01]  * mwhudson finds another way one of the tests that broke lifeless' testtools branch is broken
[23:01] <mwhudson>         future_time = datetime.now(pytz.UTC) - timedelta(days=1)
[23:01] <lifeless> rockstar: what do you hate ?
[23:01] <mwhudson> spot the mistake!
[23:01] <lifeless> mwhudson: -
[23:02] <beuno> or rather, the delta needs an actual delta?
[23:02] <lifeless> no
[23:02] <lifeless> thats one day in the past
[23:02] <mwhudson> yeah, such an innocent little character
[23:03]  * beuno sits back down
[23:04] <jcsackett> mwhudson: was that passing for some cases? i would think a day in the past when you expect a day in the future would screw everything up.
[23:04] <lifeless> jcsackett: storm :)
[23:05] <mwhudson> jcsackett: yes, but then the test was written in a way that couldn't fail
[23:05] <mwhudson> which was the first thing i fixed :)
[23:05] <jcsackett> mwhudson: ah, fun times. :-)
[23:05] <jcsackett> lifeless: sadly, i haven't had enough experience with storm where just referencing it explains away things. :-)
[23:06] <jcsackett> though i
[23:06] <mwhudson> jcsackett: if X is a storm SQL expression, X == anything is another storm SQL expression
[23:06] <jcsackett> i've had this exchange often enough to accept it as gospel. :-P
[23:06] <jcsackett> aaah.
[23:06] <jcsackett> dig.
[23:06] <mwhudson> which is true when evaluated in a boolean context
[23:06] <mwhudson> aka "operator overloading is bad"
[23:07] <wgrant> aka wgrant broke Soyuz :/
[23:07] <jcsackett> mwhudson: okay, and thus everything was passing.
[23:07] <mwhudson> yeah
[23:07] <mwhudson> jcsackett: lifeless has a branch which has the sideffect of swapping the order the comparison is done in in assertEquals
[23:08] <mwhudson> this found a few cases where == was not symmetric...
[23:08] <lifeless> maybe we should check both directions for Equals
[23:08] <lifeless> what do you think
[23:08] <lifeless> a == b and b ==a
[23:08] <lifeless> a <b and b>=a
[23:08] <lifeless> etc
[23:09] <lifeless> might be a little surprising
[23:09] <lifeless> well, with the boolean logic done right
[23:10] <mwhudson> i'd be careful of mixing ordering in
[23:10] <jcsackett> lifeless: at least one fork of rspec (the ruby testing system) does just that. catches some surprising things when people start monkeypatching.
[23:10] <mwhudson> but checking a == b and b == a and a != b and b != a are all consistent might be good
[23:15] <mwhudson> ah hah, zope proxies
[23:17] <lifeless> say its not so
[23:19] <mwhudson> bug 331919
[23:19] <_mup_> Bug #331919: delegates() should provide __eq__  and __ne__ operators. <tech-debt> <lazr.delegates:Triaged> <https://launchpad.net/bugs/331919>
[23:22] <mwhudson> lifeless: https://code.edge.launchpad.net/~mwhudson/launchpad/testtools/+merge/38896
[23:24] <lifeless> wow, fixtures is getting 30-40 downloads over the course of a few adys
[23:24] <lifeless> http://pypi.python.org/pypi/fixtures/0.3.1
[23:24] <lifeless> mwhudson: lp reckons there are conflicts
[23:25] <mwhudson> lifeless: odd, i guess i'll merge trunk
[23:25] <wallyworld__> rockstar: for when you get back from class - got the windmill test working, had to move login from test method to setup phase
[23:25] <LPCIBot> Project db-devel build (85): STILL FAILING in 3 hr 39 min: https://hudson.wedontsleep.org/job/db-devel/85/
[23:25] <LPCIBot> Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=617699] Add getter/setter property for
[23:25] <LPCIBot> linking distribution/source_package to BugTrackerComponents
[23:25] <mwhudson> lifeless: the conflicts must be in your branch though :p
[23:26] <lifeless> probably ;)
[23:30] <mwhudson> yeah, a Contains matcher was added
[23:30] <mwhudson> which should be swiftly shuffled off to testtools too i guess
[23:30] <lifeless> hah
[23:30] <lifeless> yes
[23:30] <mwhudson> anyways, pushing the merged branch now
[23:30] <lifeless> sweet. bombs away
[23:34] <lifeless> I think this is starting to look pretty nice as a test:
[23:34] <lifeless> http://pastebin.com/tq43kxhW
[23:35] <wgrant> I think you could use some more context managers.
[23:35] <lifeless> wgrant: don't be a hater ;)
[23:36] <lifeless> mars: BaseWindmillLayer looks like it tries to disable the new thread check anyhow
[23:36] <mars> lifeless, ? looking
[23:36] <lifeless> look for disable_thread_check
[23:37] <lifeless> in layers.py
[23:38] <mars> yep
[23:38] <mars> already had it open
[23:38] <mars> reading
[23:38] <mars> hmm
[23:38] <mars> lifeless, could it be it is working, but subunit is eating the "ERROR DISABLED" message?
[23:38] <mars> lifeless, or it could be that the disabling is not working any more
[23:39] <lifeless> yes
[23:39] <mars> lifeless, is this functionality tested?
[23:39] <lifeless> if error disabled is being 'print' output, subunit will a) munch it and b) get corrupted
[23:39] <lifeless> mars: I think it has some tests, yes. Dunno how many
[23:41] <mars> lifeless, ./testing/tests/test_layers.py has a 'test_slow_thread' test
[23:42] <mars> no test verification though - I guess it relies on the testrunner running the test and blowing up if it doesn't work
[23:44] <wgrant> test_slow_thread is one that often has a LayerIsolationError.
[23:45] <lifeless> mmm, I suspect all these parallel test branches of mine will land as a clump :(
[23:46] <mars> ok, frustrating - if the 'wait for live threads to exit' should be working.  It gives /lots/ of time for them to shut down
[23:46] <lifeless> mars: how long
[23:46] <mars> 10 seconds
[23:46] <mars> 100 * 0.1 seconds
[23:46] <lifeless> mars: stuff dealing with tcp can timeout after 5 minutes
[23:46] <mars> so_linger - yeah :(
[23:46] <lifeless> (or more, but 300 seconds is the common one)
[23:46] <mars> thought of that
[23:47] <mars> that would make sense with StevenK's stale request hypothesis
[23:49] <mars> you know, if it is always roughly the same windmill test that has issues
[23:49] <mars> then we should be able to pick apart the test and see what is generating the request
[23:49] <mars> it would not be the first time an stale thread issue was solved this way
[23:50] <mars> bugs had the same problem
[23:51] <lifeless> mars: fwiw gary has agreed to have the check disabled entirely until the windmill issue is addressed
[23:51] <lifeless> I need to file a bug and make the change.
[23:51] <mars> bug 383615
[23:51] <_mup_> Bug #383615: Spurious test failure in emailaddress.txt. <spurious-test-failure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/383615>
[23:51] <mars> ack, no
[23:51] <mars> wrong bug
[23:52] <mars> bug 516781
[23:52] <_mup_> Bug #516781: Intermittant failure of test_inline_subscriber <qa-ok> <Launchpad Bugs:Fix Released by deryck> <https://launchpad.net/bugs/516781>
[23:53] <jam> mwhudson: Pushed up a new revision which should fix those things, I still get an atexit warning, but I think that is the existing code.
[23:53] <mwhudson> jam: cool, i'll throw it at ec2 again
[23:54] <lifeless> the atexit is transitional
[23:54] <lifeless> while we fix layers