#launchpad-dev 2009-08-03
 * mwhudson FINALLY has an automated test case for bug 120977
<mup> Bug #120977: cscvs breaks when a cvs merge creates a file <Launchpad CSCVS:In Progress by mwhudson> <NUnit 2.5:New> <https://launchpad.net/bugs/120977>
<mwhudson> unfortunately it has a sleep 60, a timezone dependence and i have to hack the cscvs source...
<mwhudson> find . -name '*,v' -print0 | xargs -0 sed -ie 's/^date\t[0-9]\{4\}\.[0-9]\{2\}\.[0-9]\{2\}\./date\t2007.01.01./'
<mwhudson> :)
<thumper> ew
<thumper> mwhudson: at least we won't be running cscvs tests when landing LP branches
<mwhudson> yeah
<mwhudson> thumper: i have a faster testcase now
<thumper> :)
<mwhudson> that's what that regexp is about though
<ajmitch> ugly but readable, why do you need to rewrite the dates?
<mwhudson> ajmitch: oh god
<ajmitch> that much pain? :)
 * ajmitch won't ask
<mwhudson> ajmitch: basically because cscvs looks a week in the past because cscvs
<mwhudson> because cvs is a bit vague
<mwhudson> but i need stuff to have happened longer ago than that
<thumper> OSError: [Errno 13] Permission denied: '/var/tmp/bazaar.launchpad.dev/mirrors'
<thumper> :(
<thumper> grr
<thumper> owned by root again!
<thumper> WTF??!?
<lifeless> mwhudson: cool
<spm> mwhudson: hmmm. interesting. just looking at the numbers of requests in the xmlrpc log - was a delay of ~ 15 mins before the service was restarted onto a new log; and can clearly see a drop in the # of requests. ~ 1/4 ish. Ouchy.
<mwhudson> spm: :( !
<mwhudson> spm: make build suckiness?
<spm> mwhudson: I guess....
<spm> mwhudson: ahh. I'd put money on the compression phase holding things up.
<mwhudson> spm: oh
<mwhudson> spm: well, i have this change that will reduce the number of requests by 75% or so ...
<spm> be nice!
<spm> as in, would be :-)
<thumper> spm: can you bounce the buildbot? the db_lp builder seems fubared
<spm> thumper: sure
<spm> bleh. looks like the slave is borked
<spm> thumper: right that seems to have kicked it along.
<thumper> spm: ta
<JamalFanaian> So, I think the make schema step of the launchpad setup used to ask me what I wanted the psql username to be, but it isn't on this computer and I keep getting the following error:
<JamalFanaian> psql: FATAL:  Ident authentication failed for user "jamal"
<JamalFanaian> Could I be doing something wrong?
<wgrant> JamalFanaian: make sure you've run utilities/launchpad-database-setup
<JamalFanaian> wgrant: ah ok! thanks :)
<JamalFanaian> wow, i even went back to the wiki to make sure i didn't miss a step.. bah! lol
<JamalFanaian> wgrant: thank you
<JamalFanaian> ok so now my next question is, how can i configure it to listen to requests to all ips, or at least the external ip, instead of 127.0.0.88...
<JamalFanaian> i'm running it in a vm and copied the hosts directives but changed it to point to the ip of the vm (instead of the local ip), but the request is being delivered to apache's default Vhost
<wgrant> JamalFanaian: You want to poke at /etc/apache2/sites-enabled/local-launchpad
<JamalFanaian> wgrant: thanks again :)
<JamalFanaian> awesome, that works perfectly.. thank you!
<wgrant> Excellent.
<mars> YUI3 beta 1 adds the YUI test suite! \o/
<mars> something good to build on
<wgrant> So make jscheck can take even longer?
<mars> depends if we add our unit test suite to the acceptance test suite
<mars> which we might do, for convenience sake
<jtv> spm, g'day!  I'm trying to get the translations-export-to-branch output in my error-reports, but I can't seem to find the right topic or something.  Do you know where these logs go?
<mwhudson> i get them
<mwhudson> so it's probably some code related topic
<thumper> mwhudson: I'm going to get a branch to disable spark lines for now
<beuno> thumper, I hate you
<wgrant> thumper: I love you.
<beuno> it
<beuno> it's the wrong solution
<thumper> beuno: nothing personal
<beuno> I know, but only a handful of people complain, and for the wrong reasons
<thumper> beuno: it can be fixed...
<thumper> beuno: I'm not deleting the code
<thumper> beuno: just disabling for now
<lifeless> beuno: I don't think looking ugly is a wrong reason :)
<thumper> beuno: we can fix it before 3.0
<thumper> beuno: but until it is fixed, I'm making the executive decision
<beuno> thumper, I'm very against removing a feature under the users, which benefit them
<mwhudson> thumper: ok
<lifeless> beuno: we're users too though
<mwhudson> thumper: i've got that branch of yours mostly reviewed
<thumper> beuno: I don't see the benefit that isn't obvious elsewhere
<beuno> thumper, activity
<beuno> a lot of users use it
<thumper> beuno: commit count shows activity
<lifeless> beuno: its not showing what they probably think it is
<beuno> thumper, not as well
<beuno> again, this is personal opinion, versus people who actually use it
<lifeless> beuno: I feel like you want to keep something substandard because its graphical
<lifeless> beuno: we should probably gather some real data on this. After all, so far we've only got the complaints - there is measurement error possibility there
<thumper> beuno: lets work to fix it then... but right now, and for the next few days at least, I'll be gone on edge
<beuno> lifeless, if we remove all substandard things in launchpad, we'll remain with a very think software
<lifeless> beuno: but it would be beautiful
<beuno> thumper, fix what Paul broke?
<thumper> beuno: there are more issues with it though, rendering being a big one for lifeless and poolie
<thumper> beuno: there are many bugs for the sparklines
<thumper> beuno: perhaps we should look to address them when we bring them back
<thumper> beuno: but right now, they don't look good
<beuno> right *now* they don't. They where good enough a week ago
<thumper> no
<thumper> they weren't
<thumper> there are a lot of bugs with them
<thumper> I don't think you've been looking at them
<thumper> there are issues with the sparklines on production
<lifeless> beuno: they have *never* been good for me.
<beuno> I have. They where wonky on the person page (they shouldn't of showed up)
<lifeless> beuno: I'm not being difficult.
<beuno> lifeless, I know. They have been great for other people.
<beuno> anyway, I've said what I think. It your app, so it's all I can do.
<lifeless> I think the problem with both our arguments is !citation
<lifeless> I'd like to see sparklines that look good and expose useful information
<beuno> I can dig up IRC conversations and blog posts where people mention them as what they use to measure activity
<beuno> but it feels like a waste of time
<beuno> meh
<lifeless> I don't want you to waste your time
<lifeless> and its thumpers call about this
<beuno> yeah, I'm really absolutely off to bed now  :)
<beuno> night guys
<lifeless> sleep well
<thumper> I like the idea of sparklines
<thumper> I think they could be really good
<thumper> lets make them better
<lifeless> yes
<lifeless> I would in fact like them on every row, once fixed
<lifeless> but unlike data in (say) stocks, our branches are tightly coupled
<lifeless> so we need to consider that
<thumper> right
<thumper> bug 408207
<mup> Bug #408207: Branch sparklines are disabled <ui> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/408207>
<thumper> I've targetted 3.0 to fix this
<stub> What does 'max commits' on the sparklines actually mean?
<stub> I just opened https://bugs.edge.launchpad.net/launchpad-code/+bug/408211 for giggles - the colour choices violate our UI standards.
<mup> Bug #408211: sparklines using error red and link blue when it shouldn't <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/408211>
<thumper> stub: bug 390645
<mup> Bug #390645: Sparkline text doesn't make sense <confusing-ui> <Launchpad Bazaar Integration:Triaged by thumper> <https://launchpad.net/bugs/390645>
<thumper> mwhudson: there isn't any point landing the fix now as it won't get in before the edge rollout anyway
<thumper> mwhudson: I'll check the windmill with rockstar tomorrow
<mwhudson> k
<mwhudson> thumper: finally reviewed your branch
<thumper> mwhudson: thanks very much
<mwhudson> thumper: https://code.edge.launchpad.net/~mwhudson/launchpad-cscvs/merge-creates-file-bug-120977-attempt-2/+merge/9573
<mwhudson> lifeless: you might like to look at that too ^^
<lifeless> mwhudson: remind mem tomorrow
<mwhudson> fair enough
 * mwhudson is no longer here
<thumper> mwhudson: replied, and I'm EOD now
<noodles775> Morning
<jtv> hi noodles775
<noodles775> Hiya jtv :)
<jtv> mthaddon, are you here?
<stub> Killing translationstobranch(30855), 2009-08-03 03:30:08.209901+00:00, 2009-08-03 03:30:08.304575+00:00
<stub> jtv: Doing something in a long transaction? or perhaps a hang.
<jtv> stub: have you done that before?  I noticed the export for ddtp-ubuntu got interrupted a few days ago.
<jtv> It's just a very large export job.
<stub> Killing translationstobranch(21076), 2009-08-02 03:30:09.157374+00:00, 2009-08-02 03:30:09.289792+00:00
<stub> Killing translationstobranch(11525), 2009-08-01 03:30:10.466531+00:00, 2009-08-01 03:30:10.590209+00:00
<stub> Is there any reason to do that inside the transaction?
<jtv> Yes, that matches.  The ddtp-ubuntu exports are taking too long & getting killed off.
<jtv> We should be able to break it down into smaller transactions.  That'd benefit exports in general.
<stub> Yup. Autocommit mode could be best if you don't need the transactional integrity.
<stub> If you do need transactional integrity, it might need refactoring to dump information into a temporary table in one transaction and then export from the holding area in autocommit mode.
<mrevell> Howdy Launchpadders
<jtv> hi mrevell!
<jtv> stub: I don't think we need it that badly.
<jtv> mrevell: hey, the What's New list on the front page is still 2.2.6!
<jtv> (a little Serbian bird just told me)
<mrevell> jtv: I'm afraid I wasn't here for the release (paternity leave) so not everything happened exactly as normal. I need to see if there's a way to get an updated What's New onto production without a roll-out.
<danilos> jtv: mrevell was on leave at the time, kfogel was supposed to update it
<jtv> danilos: I know, but he's not here now.
<danilos> mrevell: sure there is, it's called a cherrypick :)
<danilos> jtv: that's even better: blame the one who's not here :)
<jtv> danilos: it does mean I can't tab-complete the nick, so that's why I went with mrevell
<mrevell> danilos: to be fair to kfogel, I was in a rush to get out the door to tend to my increasingly pained wife, so I may have left the "what's new" requirement out of my "oh my God, can you please do these things" email.
<mrevell> heh
<danilos> mrevell: you might have, but I remember reminding him about it and sending my items along with it :)
<jtv> danilos: he did email them out.
<mrevell> Anyway, let's not worry about what happened in the past, let's just make the future a great place to be.
<mrevell> haha
<mrevell> :)
<mrevell> Right, anyway, I'll prepare a branch now and get it through testing and review.
<danilos> mrevell: sure, so if you can get a branch ready, I am sure we can have it cherrypicked
<mrevell> cool
<danilos> mrevell: I'd be happy to restart my work with a review as well :)
<mrevell> danilos: superb :) Gimme a few mins to prepare this branch.
<danilos> mrevell: np :)
<carlos> danilos: hey, you are alive!
<carlos> btw, hi ;-)
<danilos> carlos: hey hey
<danilos> carlos: just back from my holidays :)
<danilos> carlos: how's it going? have you tried the new Launchpad? :)
<carlos> danilos: I supposed that :-P
<carlos> danilos: I didn't have time yet, I only checked it out
<danilos> carlos: heh, cool :)
<carlos> danilos: btw, congratulations for the release ;-
<carlos> ;-)
<danilos> carlos: thanks, really happy about it myself :)
<deryck> Morning, folks.
<noodles775> Hi deryck :)
<danilos> jtv, henninge: hey hey, I'd really like to hear your voices, it's been so long!
<jtv> danilos: should I sing a bit?
<danilos> jtv: I'd love that :)
<henninge> danilos: I am sorry, I forgot my headset at home ... :(
<jtv> So who's going to do backup vocals?
<danilos> henninge: a shame, a shame
<henninge> danilos: I can go and grab it if you are willing to wait 10 minutes for my voice ...
 * henninge runs off
<danilos> henninge: sure, sounds fine :)
<jtv> danilos: did you _have_ to say "sounds fine"?
<danilos> jtv: of course I did, the sound was splendid :)
 * jtv sighs
<henninge> danilos, jtv: ready!
<jtv> go
<danilos> jtv, henninge: calling
* mrevell changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 2.2.8 | https://dev.launchpad.net/ |  Please use #launchpad for support. | https://launchpad.net/~launchpad-dev | Get it: https://dev.launchpad.net/Getting | http://people.canonical.com/~herb/ | http://paste.ubuntu.com/
 * henninge lunches
<jtv> herb, hi!  I sent two requests to the LOSAs list.  Have to leave now, but the message-sharing one is the more important one.
<jtv> herb, to reiterate: apply https://pastebin.canonical.com/20693/ on staging, then run scripts/rosetta/message-sharing-merge.py -vvv -T -p elisa
<jtv> tia :)
<jtv> Ah, that should be the -P option (the -T one isn't necessary here)
 * jtv afks
<mars> sinzui, around?
<sinzui> I am
<mars> hi sinzui, just wondering if my argument for saving 50% of CHR's project review time swayed you to give the experiment a shot :)
<sinzui> I think we need to fix the test/translation/junk problem first
<sinzui> The test and translation problems are a regression from the old design
<kfogel> jml: mmm, I envy you the real Guinness
<jml> kfogel, and an interesting thing came up in discussion
<kfogel> jml: hit me :-)
<jml> kfogel, heh heh
<jml> kfogel, I reckon a lot of the potential Launchpad hackers out there are people who are actually more interested in Ubuntu.
<jml> kfogel, there's an internal Canonical wiki page called UbuntuInfrastructureNeeds which has a bunch of stuff that we need to address
 * kfogel goes to look at that page
<jml> kfogel, we should move much of that page to the dev.lp.net wiki
<kfogel> jml: agreed.  Since I'm working in the wiki right now, I'll do that too.
<jml> kfogel, and I think, more generally, it'd be worth making it really obvious for people to answer the question "how can I help Ubuntu by patching Launchpad?"
<kfogel> jml: I'm torn on that one.  On the one hand, Ubuntu devs are big Launchpad users.  On the other hand, by making Launchpad (including Launchpad development) more attractive to non-Ubuntu-specific upstreams, we can get them using Launchpad more... which would be good for Ubuntu in the long run of course.  So I'm not sure how Ubuntu-pandery to be.
<jml> kfogel, fair enough.
<kfogel> jml: the answer might be to talk about Launchpad in Ubuntu forums, not in Launchpad forums.
<jml> kfogel, perhaps.
<jml> kfogel, I'm going to mull over this one, I think.
<kfogel> jml: me too.  Some of the answers may become obvious as I'm editing the wiki today.
<jml> kfogel, cool :)
<mars> gary_poster, btw, do you have any ideas for stub's suggestion of forking buildout for python2.6?
<mars> does buildout let you keep a light-weight configuration fork like that?
<stub> I was just thinking of a seperate branch with automatic merges like devel -> db-devel
<stub> Although I guess running 'make check PYTHON=python2.5' might work just as well
<gary_poster> mars, I'm afraid I didn't see that.  Looking.  In any case, the typical approach for that sort of thing is to have a separate .cfg file that extends the normal one (or some other similar variant).  You can specify the cfg to use with the -c option to bin/buildout
<mars> gary_poster, lp-dev, maxb's "Launchpad on Python2.5" thread
<gary_poster> Yeah I see it
<mars> you know
<gary_poster> mars, not sure if that's what he meant.  If we have a separate branch then the branch can just modify buildout.cfg as appropriate
<mars> it would be nice to have a clear, visual buildbot display from running the suite on 2.6
<mars> so we could start running it right now, see the errors
<mars> and anyone can start fixing them
<mars> *without* running the entire suite
<gary_poster> That's a good idea.  I think I'd be more keen on that once I have the zope buildout branch done.
<mars> ok
<jml> hello :)
<mars> hi jml
<jml> I'm getting james_w set up with a Launchpad development environment...
<jml> and he's getting errors to do with the storm egg
<mars> stub or gary_poster, ^ ?
<gary_poster> jml, what sort of errors? :-)
<stub> Refresh download-cache
<stub> (and make sure the power cord is plugged in)
<gary_poster> lol
<jml> stub, 'bzr up download-cache' is up to date
 * jml gets a stack trace
<james_w> http://pastebin.ubuntu.com/246303/
<jml> Tree is up to date at revision 57.
<james_w> # ls /tmp/buildd/launchpad/devel/eggs/storm-0.14trunk_321-py2.4-linux-i686.egg/storm/ | grep cext
<james_w> cextensions.c
<james_w> cextensions.py
<james_w> cextensions.pyc
<stub> Any build errors though?
<stub> The storm egg is broken, so what happened.
<gary_poster> right
<stub> storm update happened last commit actually (move from custom branch to trunk), so if this is a genuine problem revert r9024. I have to head off in a few mins though.
<stub> make clean; make build
<gary_poster> james_w: if you have lost build errors in your terminal, you can remove /tmp/buildd/launchpad/devel/eggs/storm-0.14trunk_321-py2.4-linux-i686.egg and rerun make (or simply run bin/buildout) to try to get the error
<stub> Or maybe make clean; rm -rf eggs/*; make build
<JamalFanaian> hi, what is the proper way to run the python unit tests? i was trying bin/test using the --test parameter, which runs but results in 0 tests executed
<noodles775> JamalFanaian: what's the exact command that you're running? (I generally use -vvt, but not sure why you're getting 0 tests?)
<jml> ok.
<JamalFanaian> noodles775: bin/test --test=test_person.py
<jml> so that seems to have addressed the problem
<gary_poster> huh
<jml> it seems a bit suboptimal that 'make clean' doesn't actually clean the build
<gary_poster> well, good, and, huh??
<bigjools> JamalFanaian: use -t test_person
<stub> 'bin/test test_person' is what you want.
<JamalFanaian> bigjools: stub ok thx :)
<jml> gary_poster, yeah, that's where I am :)
<stub> --test is a regexp that doesn't match the filename you are specifying
<JamalFanaian> stub: oh ok
<stub> Because the filename is not what is being matched - the module and testname is
<noodles775> JamalFanaian: so for unittests, just leave of the .py ... yes, as bigjools said
<JamalFanaian> makes sense, it's running now :)
<JamalFanaian> noodles775: yeah it's working now, thank you :)
<gary_poster> jml: make clean does clean the build.  it doesn't delete all the eggs, which are shared resources.  I suppose we could have a make clean_eggs but I'm not even in favor of that, because it would affect other branches (at least in the current set up, which has speed advantages)
<mars> jml, noted, we need 'make distclean'.  'make clean' already does some stuff to clean up the app server state, but not the initial artifacts
<gary_poster> jml: your launchpad build was fine.  an egg was broken.  That's "not supposed to happen" in such a way that rebuilding the egg makes everything happy
<jml> gary_poster, I see.
<stub> gary_poster: Should buildout trash the egg it just tried to create if the creation fails?
<jml> I wonder how 'make clean' interacts with sourcecode/
<gary_poster> stub: yeah, that seems like it would be good behavior
<jml> answer: it cleans it up incorrectly by accident
<gary_poster> jml: good question.  so...not quite sure what the answer means to the buildout story (given the "incorrectly by accident" part)
<jml> gary_poster, well, it's hard to determine intent
<gary_poster> gotcha :-)
<jml> it uses find to recursively delete things that are likely to be build artifacts
<gary_poster> jml: huh.  ...I wonder if it therefore would be the cause of this symptom, if it walks into eggs...
<jml> I only ever really use 'make clean' when I'm suspecting an error in the build process and want to try again...
<jml> gary_poster, that's probably it, actually.
 * jml experiments
 * gary_poster thinks he's used make clean plenty since buildout-ification and not had a problem like this to his knowledge...
<jml> oh, btw, I'm making a branch that removes the 'apidoc' dependency from 'make build'
<jml> ahh, got it.
<jml> james_w has download-cache in the devel directory
<jml> not as a symlink
<jml> so find is recursing down into it, whereas it doesn't recurse into symlinks
<gary_poster> jml: (apidoc dependency) oh, I thought we needed that for lazr.restful-based tests.  Are you sure that's not the case?
<jml> (so my earlier statement about it cleaning up sourcecode was untrue)
<gary_poster> jml: "download-cache" just has .tgzs, but I suspect your analysis is true for "eggs" as well, which would in fact cause this symptom
<gary_poster> jml: ah ok
<jml> gary_poster, right, I meant to say eggs.
<gary_poster> cool
<jml> gary_poster, so this would explain why we have never seen it.
<gary_poster> right exactly.  ...I suppose we could try to make the ``make clean`` target avoid "eggs"?
<gary_poster> for this kind of situation:
<jml> either that or get it to delete them :)
<jml> but it should be one or the other.
<gary_poster> sorry, s/:/./
<gary_poster> jml: yeah.  ok, unless you want to argue for the deleting the eggs, I'll add a bug for making "make clean" not remove .sos and so on from eggs
<jml> gary_poster, +1
<gary_poster> coool
<jml> so, on the apidocs
<jml> there are two things. the first is that we should make the actual step more resilient to failure
<jml> the second is that if it's not actually needed to run launchpad locally, then it shouldn't be in 'make build'
<rockstar> jml, why in the world are you awake?
<jml> rockstar, because it's 3:33pm where I am.
<rockstar> jml, I do not think you are where I think you are.
<jml> rockstar, I'm in Dublin
<rockstar> jml, ah, okay then.
<gary_poster> jml: make the actual step more resilient to failure, sure, makes sense.
<kfogel> Hey folks, new page: https://dev.launchpad.net/Help .  I'll post/write about these new pages later, as I'm revamping a bunch of stuff, but thought you'd want to know about that one early.
<gary_poster> jml: not needed to run launchpad locally: if you are right (don't know) and there's a target that the losas use for building that is not make build (don't know) then makes sense
<jml> gary_poster, do you know if it's needed to run launchpad on production?
<gary_poster> jml: flacoste would know if it is necessary for development, and leonardr might know.  Have you already determined this?  If not, we should consult leonardr; if he is not sure, I suspect we should consult with flacoste when he is available later this week.  (I do believe this is necessary for launchpad on production, at least because we link to it from the wiki)
<jml> gary_poster, I haven't already determined it, but I figure that if I just do some experiments & grepping, I can find out enough.
<leonardr> jml, gary_poster: it's necessary for development in the sense that there's a test that verifies its existence
<leonardr> which i think is why we put it in make build
<jml> leonardr, hi, long time no see :)
<leonardr> hi
<jml> leonardr, but why is the test there?
<leonardr> to make sure that the generation code works?
<jml> leonardr, why can't the test run the generation code itself?
<gary_poster> jml, can we just address the first problem then (make it more resilient to failure)?  Not clear on what the problems are there
<jml> gary_poster, yes. that's easy enough.
<jml> the problem is that the way the make step works, if it fails, there's an empty file at $API_INDEX
<gary_poster> ah yes
<jml> so the next time you run 'make build', you get a horrid syntax error
 * gary_poster looks mildly guilty
<leonardr> jml: well, right now it's a simple pagetest. changing it to run the generation code would mean making it more like a doctest
<leonardr> there's no reason why we couldn't do it
<mrevell> kfogel: Help page looks good to me
<jml> leonardr, ok. I think I'd like to do it then.
<gary_poster> jml, leonardr, but I'm not sure what the win is, since we do need it for production, unless there is a different make target that the losas use
<kfogel> mrevell: do you know how to color text (not in a table) in moin?
<kfogel> mrevell: without installing the separate Color() macro, that is :-).
<jml> gary_poster, the win is shaving 20 seconds off every make build run.
<mrevell> kfogel: the only way, of which I know, without the macro is using tables
<gary_poster> jml: does it still run after an initial build?  I thought I had fixed that.
<kfogel> mrevell: ah, okay.  thanks
<gary_poster> jml (shaving 20 seconds off every new branch's make build run is nothing to sneeze at; I'm clarifying for my own interests, I suppose)
<jml> gary_poster, no, it doesn't, I think.
<gary_poster> ok, good, at least.
 * jml verifies experimentally
<gary_poster> leonardr, jml, am I right that shaving 20 seconds at build time means we regain 20 seconds within test time, given the current proposed change?
<leonardr> gary_poster: yeah
<gary_poster> mm.
<jml> gary_poster, just in time :)
<gary_poster> and jml, I apologize for repeating, but do you know if losas have a different build target?
<herb> gary_poster: what are you trying to figure out?
<jml> gary_poster, no, we don't. I'd definitely want to confirm that before landing the change
<jml> herb, I'd like to know which make targets are run as part of the deployment process
<herb> jml: confirming. one moment.
<jml> herb, thanks.
<gary_poster> jml: again feeling mildly guilty, this time for putting brakes on your efforts.  I raised my concerns; now will get out of your way. :-)
<jml> gary_poster, no worries at all :)
<herb> jml: so, we're kinda all over the place on this.
<gary_poster> :-)
<herb> jml: at the very minimum we run make build
<jml> gary_poster, friendly concern is always appreciated :)
<kfogel> mrevell: https://dev.launchpad.net/Running (just created, review welcome)
<herb> jml: actually
<gary_poster> heh, cool
<kfogel> mrevell: I also modified the top-level Getting page to refer out to Running and Help now.
<herb> jml: let me approach it this way
<herb> jml: in one way or another we use the following targets in production: build, static, start, initscript-start, stop, shutdown
<jml> herb, and that list is complete?
<mrevell> kfogel: makes sense
<mrevell> kfogel: will look at running
<herb> jml: don't hold me to that.
<beuno_> mars, intellectronica, rockstar, noodles775, AJAX in 10?
<jml> herb, fair enough :)
<rockstar> beuno, yessir.
<noodles775> beuno: yup.
<herb> jml: I would say that covers 99.9% of the cases.
<jml> herb, certainly, knowing what the public interface of our Makefile is helps a lot :)
<herb> jml: but I didn't go out and check every server.
<jml> herb, np. thanks for checking :)
<herb> jml: welcome
<intellectronica> beuno: yes
<kfogel> mrevell: is there a single, canonical page for Karmic anywhere?  I see www.ubuntu.com/testing/karmic/alpha2 and .../alpha3, and various other pages, but I don't see a single "this is what Karmic is" page
<mars> beuno, yep
<mrevell> kfogel: https://wiki.ubuntu.com/KarmicKoala ?
<kfogel> mrevell: perfect, thank you
<AdamDH> Hey, I did a bzr co lp:launchpad but I cannot see the Soyuz source or Codehosting source code? Any ideas where this is?
<noodles775> AdamDH: ./lib/lp/soyuz ?
<jml> ./lib/lp/codehosting
<intellectronica> beuno: are you leading?
<beuno> intellectronica, I am. trying to hang up another call
<mrevell> kfogel: I'll PM you some comments.
<intellectronica> beuno: just offer whoever is on the line to listen to some really great, relaxing music for a bit ;)
<beuno> heh
<kfogel> mrevell, jml: a thought -- I'm editing https://dev.launchpad.net/PatchSubmission right now, and it says for pre-implementation discussion, contact the on-call help person in #launchpad.  That doesn't seem quite right; better to just come into #launchpad-dev and talk to someone here, no?
<mrevell> kfogel: That certainly seems to make better sense to me. I think we're providing pretty good coverage in here.
<intellectronica> kfogel: nor neither. the best thing to do is to come to #launchpad-reviews and talk to the ocr
<jml> kfogel, agreed.
<kfogel> intellectronica, jml: even better, thank you
<kfogel> mrevell: ^^ intellectronica and jml told us where to go, sha-ZZAM
<mrevell> aha
<mrevell> :)
<AdamDH> noodles775: thanks did not see it in there
<maxb> PatchSubmission could also use some clarification about what to do for small obvious fixes - e.g. a three line fix for buildmailman.py - you can't have a pre-imp discussion when you don't know the change you'll be making until you've researched a bug
<mars> beuno, still around?
<beuno> mars, almost dialing in, sorry
<beuno> dialing
<allenap> BjornT: stub has commented on bug 253242 that he's planning on pruning bugnotification soon. wgrant pointed out to him in the bug referenced that perhaps we (bugs) want to use that data first. Do you have an idea what the value of this data is? Should we ask him for a dump or an archived copy before it's gone for good?
<mup> Bug #253242: Fill in the gaps in the activity log using data from bug notifications <bughistory> <feature> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/253242>
<mars> Guest36657, AJAX call?
<mars> EdwinGrubbs, joining the call?
<BjornT> allenap: yeah, dumping the data would be a good idea.
<allenap> BjornT: Okay.
<allenap> BjornT: I'll comment on the bug.
<BjornT> thanks
<mars> beuno, intellectronica, ah!  Forgot to settle next steps for the UI/QA experiment
<mars> we really should close that out formally
<mars> is was successful, right? :)
<intellectronica> mars: it will be especially interesting in light of our qa team changing. it might mean the qa people can be even more on hand
<intellectronica> mars: yes, from my experience i can say it was successful
<mars> intellectronica, would you mind writing a mail to the list saying "It works!  Here's how we did it!"
<mars> ?
<mars> It would lend closure
<EdwinGrubbs> mars: I'll call in now. I thought I had already missed the meeting entirely.
<mars> and let us start talking about the next steps
<intellectronica> mars: sure, i can do that
<mars> EdwinGrubbs, nm, we wrapped it up at 11:35
<intellectronica> mars: b.t.w one thing we still haven't figured out is how to track the results. we currently use the team's test plan, but we find it very inconvenient
<mars> intellectronica, you should include that problem at the end of the experiment summary.  It might start some discussion.
<mars> "It works!", "Here's what we did", "Here's what didn't work so well"
<mars> :)
<kfogel> I want to do a search for all open-and-unstarted bugs that affect Launchpad that have the 'trivial' tag.  But Launchpad isn't just one component; it's many components.  How to do this search?
<kfogel> I tried https://bugs.edge.launchpad.net/launchpad/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&field.bug_commenter=&field.subscriber=&field.omit_dupes.used=&field.omit_dupes=on&field.h
<kfogel> as_patch.used=&field.has_cve.used=&field.tag=trivial&field.tags_combinator=ANY&search=Search
<kfogel> but that brought zero results.
<kfogel> jml: ^^, and note if I do the same without the 'trivial' tag, I get 58 results.
<jml> launchpad-project
<jml> kfogel, launchpad-project is the conglomeration of all launchpad components.
<jml> (sorry)
<kfogel> jml: thanks
<kfogel> jml: how did we make that magic happen?  Is it one of those special encapsulating projects ("project groups") that some admin has to create/
<kfogel> ?
<jml> kfogel, yes, that.
<kfogel> jml: thanks
<kfogel> mrevell: please see front page and critique away
 * mrevell looks
<mrevell> kfogel: Looks good! Is the line "This wiki is new and we're still moving information to it from other places:" misplaced?
<beuno> mrevell, I promise to reply to your email today
<mrevell> beuno: thanks man :)
<kfogel> mrevell: oh wow, forgot about that line
<kfogel> mrevell: killing it now
<beuno> mrevell, don't thank me until I follow through  ;)
<kfogel> mrevell: since it will always be true, there is no need to say it
<mrevell> kfogel: joey was keen to see a section where we link to an overview page for each team. I've actually got that as a bug assigned to me. I guess we should leave it to each TL as to what they want on their team overview page.
<kfogel> mrevell: that's the part right under the top table now
<kfogel> mrevell: I was just noticing how some of the pages need to be filled in.
<mrevell> kfogel: Is it, though? I mean, we don't have a Blueprint team and that's listed there. I suppose the question is: is it worth distinguishing, on the front page, between "parts" of Launchpad the software and the make-up of the Canonical development team?
<kfogel> mrevell: IMHO we should concentrate on parts of Launchpad -- that's how it looks to the community.  Our internal division into teams, while not completely irrelevant to the dev community, is much less important than the architecture of Launchpad itself (particularly since in practice, people from one team often help out with bugs in some other area).
<mrevell> kfogel: Perhaps could you take a look at the description of bug 391666 ?
<mup> Bug #391666: Please list Launchpad Teams on dev.lp.net's front page <cleanup> <launchpad-team> <trivial> <Launchpad Documentation:Triaged by matthew.revell> <https://launchpad.net/bugs/391666>
<kfogel> mrevell: looking
<jtv> herb: hi!  Can I just badger you some more?
<herb> jtv: sure.
<kfogel> mrevell: commented there
<mrevell> kfogel: Perhaps a CanonicalTeams page. Let me try something.
<kfogel> jml: do you have any idea what that project group is called "launchpad-project" instead of "launchpad"?  The latter would be more natural.
<kfogel> mrevell: should this be a high priority, anyway?  It doesn't seem that important to me, compared with the other things we need in the dev wiki.
<jml> kfogel, historical reasons, I think. ISTR mpt doing a lot of thinking about this a while back
<kfogel> If it's fast to do -- i.e., a simple page with links to the https://launchpad.net/~TEAM pages -- then sure.
<kfogel> jml: *nod*
<kfogel> jml: uh, I'll add that to my stack of questions to ask on list
 * kfogel notices the bottom of the stack starting to spontaneously combust from the pressure
<jtv> herb: did you get my requests for script runs on staging?
<herb> jtv: yes. saw the emails to losas@. haven't had a chance to jump on the requests yet. will soon.
<jtv> herb: great, thanks.  It's late night here... could you email me with the output?  A bit impersonal, but...
<herb> jtv: will do
 * jtv à¹à¸§à¹s herb
<kfogel> beuno: https://bugs.edge.launchpad.net/launchpad/+bug/408470
<mup> Bug #408470: "offer to mentor this work" line repeated on mentoring page <Launchpad itself:New> <https://launchpad.net/bugs/408470>
 * beuno looks
<mpt> kfogel, jml: In the past projects used to be called "products", and project groups were called "projects". (They still have the old names in the database schema.)
<jtv> herb: if what I just wrote doesn't make sense to you, it's because I mis-spelled it.  I meant à¹à¸«à¸§à¹, not à¹à¸§à¹
<mpt> kfogel, I think kiko gave launchpad-project its ID.
<kfogel> beuno: do you see it too?
<mpt> kfogel, I would much prefer that bug tagging was improved enough that we didn't need separate Launchpad projects at all.
<kfogel> mpt: well, before I go too far down this road, here's what I tried to do: I was searching for a bug (or maybe reporting one and in the dup-search stage?) and I put "launchpad-project" as the group, and that turned out to be invalid.  :-(
<kfogel> mpt: ah, we were typing at the same time.  Yes, that's exactly the problem.
<beuno> kfogel, yes
<mpt> yes, you can't report a bug on a project group
<mpt> though you can search for them
<kfogel> beuno: I'll bet you can fix that in thirty seconds, man :-).
<kfogel> (beuno: whereas it would take me all of two minutes!)
<mpt> If you click "Report a bug" on bugs.launchpad.net/launchpad-project the first thing you'll be asked is to choose which project
<kfogel> mpt: :-(
<beuno> kfogel, yes, I agree that sinzui can fix it in 15 seconds
<kfogel> mpt: so in one context, we should type "launchpad-project", and in another we should type "launchpad", even though they're semantically basically the same thing.
<kfogel> beuno: niiiiiice
<kfogel> mpt: "they" being the two contexts, I mean
 * sinzui reads scrollback
<kfogel> sinzui: bug #408470
<mup> Bug #408470: "offer to mentor this work" line repeated on mentoring page <Launchpad itself:New> <https://launchpad.net/bugs/408470>
<mpt> kfogel, yes, it's all a workaround for Launchpad having poor categorization of bug reports in large projects
<kfogel> mpt: I see and am not shocked.
<sinzui> beuno: this is one of this issues I reported in my email several weeks ago. The form (view) and the template are both setting headings. This will be fixed the bug team need to fix this since 2008-03-05
<beuno> kfogel, there you go
<beuno> under 15 seconds
<sinzui> beuno: kfogel: The bug's team will have a change to fix this when they update the template to main_only
<kfogel> sinzui: was there already a bug on this?
<sinzui> kfogel: search /malone
<sinzui> kfogel: I only understand the problem, I just uses blame to see when the error was introduces
<kfogel> sinzui: bug #317589
<mup> Bug #317589: "Offer to mentor this work" heading is repeated <trivial> <ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/317589>
<kfogel> sinzui: and I failed to find it because of precisely the problem mpt and I were just discussing :-).
<sinzui> kfogel: when I search for a bug that I do not know the domain, I search launchpad-project which is the project group for everything we can change: https://bugs.edge.launchpad.net/launchpad-project?field.searchtext=mentoring+heading&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%
<jml> I can't submit via ec2test :(
<sinzui> kfogel: beuno-lunch: mpt: matsubara-lunch: Do we want to ask an admin to: 1) rename launchpad => launchpad-old. 2) rename launchpad-project => launchpad 3) add the launchpad-project alias to launchpad. QA will then triage all bugs that belong to the project.
<jml> socket.gaierror: (-2, 'Name or service not known') during delete_previous_key_pair
<jml> http://paste.ubuntu.com/246454/
<jml> gary_poster, is there anything I can do to work around this, or at least get more info?
<matsubara> sinzui, why do you want to do that?
<mpt> sinzui, I'm not sure that would solve kfogel's problem. Depends on whether he searched first, or relied on the dup-finder.
<mpt> Anyway, it wouldn't solve the problem for people who *do* rely on the dup-finder.
<sinzui> matsubara: I think kfogel would have not files a duplicate bug if he could have found the original bug. launchpad only exists to triage bugs, so users probably never find the bug they want investigate unless they know how launchpad is structured
 * sinzui is really upset the the 'd' key keeps moving on his keyboard
<gary_poster> jml: never seen this before.  I assume you agree that it looks vaguely like a network connection to AWS is broken or flaky somewhere along the line.  I would just need to dig into boto myself.
<matsubara> sinzui, I think we need to fix bug 174443 instead of doing the rename
<mup> Bug #174443:  	Better duplicate searching in Launchpad's own projects <dupefinder> <feature> <Launchpad Answers:Triaged> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/174443>
<jml> gary_poster, james_w just diagnosed it for me
<gary_poster> jml: and digging into boto would probably just verify that the broken socket is to AWS.  I'd hopefully see precisely what the address was, but not sure where I'd go from there
<jml> gary_poster, my hardy chroot has a dodgy resolv.conf.
<gary_poster> jml: ah, ok.  cool
<jml> (and the socket module raises shitty errors, making it harder to diagnose.)
<jml> gary_poster, thanks.
<gary_poster> np
<gary_poster> agree on socket module errors :-/
<sinzui> matsubara: but the duplicate work in this case. kfogel did not know that launchapd is NOT a PROJECT.
<jml> it's a general Python problem, saying the error but not providing enough data about the objects involved
<sinzui> We suck for creating a project that we would NEVER approve if it was made by someone else.
<gary_poster> agree
<rockstar> beuno-lunch, can I have you take a look at: https://code.edge.launchpad.net/~rockstar/launchpad/codereview-js/+merge/9587
<sinzui> I am so very tempted to spend my weekends to put wiki markup in in blueprints so that I never have to use moin again.
<kfogel> sinzui: I if one could a) file a bug against 'launchpad' even when it's a project group, and b) have 'launchpad' work in the dup finder when filing, then your plan would be good.  The fact that the dup finder does not search the same domains as search is very problematic.
<sinzui> kfogel: as the link I pasted showed, in your case it did work You did not know that 'launchpad' is QA's bug holding project. The real project is named launchpad-project
<kfogel> sinzui: I'm on a call, back in a sec (had trouble understanding what you're telling me, but unable to formulate the right question right now)
<beuno-lunch> rockstar, yes, right after lunch, I'll take that on
<beuno-lunch> I think today is UI review day, I have like 6 of them in queue
<mrevell> see you tomorrow guys.
<gary_poster> BjornT: ping.  I have someone who is getting timeouts when trying to use https://bugs.edge.launchpad.net/bugs/+filebug (specifically, after pressing Continue).  I tried it, and while it didn't give me a timeout error, it was pretty slow if you just type in "Testing" in the Summary, and press Continue without filling anything else out.  Is this a known bug I can report?
<gary_poster> BjornT: On the same page, when I try to use the Package chooser and search for "bug" then I get "Loading results failed".
<JamalFanaian> i'm trying to create a unittest but i'm having issues.. i have created a person object using the LaunchpadObjectFactory makePerson method, but I can't figure out a decent way to create karma records for that person using the factory.. would someone be willing to help me a bit with that?
<ia> hello. since LP has been "open sourced", should 3rd party users/developers count on packaging debs of LP (by Canonical Launchpad Engineering team, i guess) in the near future?
<jml> ia, no. :)
<elmo> (I like the "scare" quotes, btw)
<jml> ia, we're all busy making Launchpad better.
<BjornT> gary_poster: we already have a bug for it. the search for duplicates sometimes times out, and it's not easy to fix.
<gary_poster> BjornT: ok thank you
<gary_poster> BjornT: do you happen to know anything about the ubuntu-bug command line application?  I have someone on IRC who wants to report a bug about it, but neither he nor I can find it in Launchpad.
<gary_poster> leonardr: ^^
<elmo> gary_poster: https://launchpad.net/ubuntu/+source/apport
<gary_poster> elmo: thank you!
<rockstar> sinzui, hi
<sinzui> hello
<rockstar> sinzui, do you think I could have a pre-imp call with you re: javascript error handling and 3.0?
<sinzui> sure. I know nothing about either of these issues, but I have opinions if you are willing to listen to me
<rockstar> sinzui, well, you know most about 3.0, and mars and I have already had a few discussions about this.
<mars> rockstar, sinzui has been doing web work for over a decade, so I trust his opinions :)
<sinzui> rockstar: That is a common misunderstanding. I am trying to make the pictures that beuno shows me to work in the code. I do not know what any page should look like. I personally want to know what can go in the side portlets and what they look like
<sinzui> rockstar: I am ready for a call when ever you are ready
<rockstar> sinzui, what is your skype id?
<sinzui> rockstar: curtis.hovey
<salgado> kfogel, I think https://answers.edge.launchpad.net/launchpad/+faq/195 needs to be updated. ;)
<kfogel> salgado: done; thanks
<gary_poster> deryck: you are on the bugs team, right? :-)  A user put some primate information (his email signature) in a bug comment.  Is the only way to change this to go to a losa, or is there some other approach?
<gary_poster> lol, s/primate/private/
<gary_poster> <ooka ooka>
<deryck> heh
<deryck> monkeys in the system, no!
<gary_poster> :-)
<deryck> gary_poster, yeah, if it's in the comment, there's no way that I know of except to edit the comment or have a LOSA hide the comment.
<deryck> gary_poster, anyone with admin privileged on lp can hide the comment because that is exposed in the API now.
<deryck> gary_poster, can hide it with a script, rather.
<gary_poster> deryck: so the only way out is database skuduggery?  oh, ok.  I'll ask the user if that's what they want then.  thank you!
<mars> or curl, perhaps? :)
<deryck> gary_poster, yeah, two options -- db work or run a script to hide the comment.
<gary_poster> deryck: ok cool thanks.
<sinzui> "There are monkeys in the palace" was a call sign at Time Life when someone misconfigured products in the database.
<deryck> heh
<gary_poster> :-)
<deryck> gary_poster, I wonder if that isn't an exercise in futility ultimately, though -- Google cache or any other web archive may have picked up the person's data and make cleaning LP pointless, no?
<gary_poster> deryck: perhaps so, but using that as a customer service reason for not doing anything strikes me as unfriendly at best, don't you think?
<deryck> gary_poster, oh, yeah, I'm not against helping. :)  I think the user should be warned, though, that this data has been compromised and not believe that simply hiding the comment fixes the potential problem.
<gary_poster> deryck: cool, agree
<deryck> gary_poster, cool :)
 * rockstar goes to lunch
<mars> sinzui, btw, something I thought might enjoy reading, when you have a moment: http://www.inc.com/magazine/20090701/joel-spolsky-the-day-my-industry-died.html
<sinzui> mars: you fiend! How can I ignore this article now that you have shown it to me? The new project layout cannot be finished until my reading is complete.
<mars> well, it's a short read...
 * beuno stabs mars 
<beuno> don't distract sinzui!
<mars> ow!
<beuno> :)
<sinzui> beuno: <as I look up from the article> is there are final picture of what side portlets should look like? Should we be showing the translation's involvement link if the project does not use translations.
<mars> lol
<beuno> sinzui, we should not
<sinzui> good, I already coded it that way
<beuno> sinzui, if only you weren't already married...
<sinzui> my wife is very jealous woman.
<beuno> can't have everything in life I guess...
<sinzui> beuno: I have two answers sections to complete then I can take some screen shots. I am also struggling with the YUI first class. it is does not behave well when it is used around a portlet that the template call.
 * sinzui is looking for a recipie to make the template and th portlet simple
<beuno> sinzui, can you cheat using firebug to send me the screenshots?
<beuno> I'm currently reviewing rockstar's branch
<beuno> but I won't leave my desk today until I've done all UI reviews on my plate
<sinzui> beuno: the first class must be on the portlet div, which means all templates that use it must always use it as first (left) portlet. The solution is to only use yui layout classes in the template that makes the page.
 * sinzui play with this a bit more before declaring defeat
<sinzui> beuno: we use the term "mugshot" for user images. Should we use that term instead of "branding" in links to update the small, medium, and large images?
<beuno> sinzui, I don't think that's a very well understood term
<beuno> "update pictures"?
<sinzui> okay. I agree that mugshot is not clear. (nor was hackergotchi)
<sinzui> beuno: I propose that I update the form to state where we will use the images.
<beuno> sinzui, I saw, and it's a fantastic idea
<sinzui> beuno: Is this the "Get involved" presentation you expect to see?
<sinzui> https://devpad.canonical.com/~curtis/LP_projectdetail.png
<beuno> sinzui, yes. The icons weren't a great idea
<sinzui> thanks
<BjornT> beuno: what is the "bug root page"? (talking about pages to be redesigned for 3.0)
<JamalFanaian> salgado: hey, do you have another min? i'm terribly sorry that i've been such a pester today.. just trying to get my head around the way factories work :\
<salgado> JamalFanaian, sure, what's up?
<beuno> BjornT, bugs.lp.net/project
<beuno> BjornT, I had no idea how to refer to it  :)
<JamalFanaian> salgado: well i'm trying to figure out how to create a karmacache record using the factory.. i've got a unittest that is able to create a person object and set up the PersonView
<JamalFanaian> I tried defining a makeKarmaCache method in the factory that just creates a KarmaCache object that I can then pull up when person.getProjectsAndCategoriesContributedTo() gets called
<JamalFanaian> but I can't actually get the KarmaCache record to be created correctly..
<JamalFanaian> i'm guessing i'm just doing it wrong, so i just wanted to see if you could give me a few pointers of where to look
<BjornT> beuno: ok, i suspected it was that one :)
<salgado> JamalFanaian, sure, do you get any errors when creating the KarmaCache record?  Can I have a look at the diff?
<JamalFanaian> salgado: let me create a diff, but i get an error when i try to call transaction.commit() after creating the karmacache record, not sure if i need to commit or not
<salgado> JamalFanaian, can you paste the error too?
<JamalFanaian> salgado: sure thing
<JamalFanaian> http://pastebin.com/f3627c1e6 -- this is my changes.. i created a simplified sql query to test if the data was even in there..
<JamalFanaian> salgado: here is the test execution with errors http://pastebin.com/f1f8fbe6
<JamalFanaian> salgado: woops ignore that last link ;0
<JamalFanaian> ;)*
<JamalFanaian> salgado: http://pastebin.com/f31c00a8 this would be the correct one
<salgado> JamalFanaian, that's because the db user used on the web app (the same one that's used for tests) is not allowed to insert rows on the KarmaCache table
<salgado> that's becasue that table is maintained by a script, so only that script's DB user can insert on that table
<JamalFanaian> salgado: ah, ok i understand..
<JamalFanaian> salgado: so is my option to try to get that insert query to run as the cron DB user? or am i taking a completely wrong approach at this?
<salgado> JamalFanaian, I think it's OK to change the new factory method to do the insert as a dbuser that can write to that table
<salgado> that dbuser would be config.karmacacheupdater.dbuser
<salgado> but I'll have to look up how to get the existing connection to use that user
<JamalFanaian> salgado: that's beyond my comprehension of the system so far lol
<salgado> JamalFanaian, I don't think we have good infrastructure for switching the db user in factory methods, so I suggest you do the following
<salgado> 1. move your new factory method as a helper method of your test class
<salgado> 2. there you can use LaunchpadZopelessLayer.switchDbUser(dbuser) before creating the KarmaCache entries, and change it back afterwards
<JamalFanaian> salgado: let me try that then, thank you so much for your time and assistance
<salgado> 3. email launchpad-dev@lists.launchpad.net asking if there's any way to turn that test helper into a factory method
<salgado> JamalFanaian, don't forget the 3rd step, that's the most important one. ;)
<JamalFanaian> salgado: I won't :).. Should I do that after I commit the branch so I can show them? Or just post the diff in the email
<salgado> JamalFanaian, just the diff is fine
#launchpad-dev 2009-08-04
<Makis> it hasn't
<jtv> Heaven help me, one of the ladyboys next door is singing.  Badly.
<jtv> Makis: unless they just changed, I don't think you read the instructions right.  :-)
<Makis> ok, i was able to download the file successfully from your link
<jtv> (I'm trying to reproduce from the instructions now.  I'll probably end up with egg on my face.  :-)
<Makis> this command however didn't work: bzr --no-plugins cat http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/utilities/rocketfuel-setup > rocketfuel-setup
<Makis> i don't know if something was broken because i tried that yesterday. i did clean /etc/hosts, but maybe something else was changed as well
<Makis> i had to stop the script because download speed for the trunk was 0kb/s, i had it running from yesterday afternoon to this morning and it had downloaded less than a megabyte, but i think i now know why
<jtv> Makis: quite possibly...  aiui the lp: protocol was redirected to go over http for the duration.
<mrevell> Salut les Launchpaddeurs :)
<jtv> mrevell: et toi, le plus grand launchpaddeur de tous
<mrevell> jtv: pas de tous, mon ami de l'est
<jtv> mrevell: je ne suis pas de l'est, c'est seulement que je vive ici
<jtv> Makis: funny...  I can "bzr branch" off lp:~launchpad-pqm/launchpad-devel but I can't "bzr cat" a file out of it.
<jtv> Makis: and the _really_ funny part is that "bzr cat" seems to insist that the branch must be in my home directory.
<mrevell> jtv: Mais, je me demande combien des ans il fault que tu vives la devant tu peux dire que tu veins de l'est. Okay, enough :)
<jtv> mrevell: quite.
<jtv> mrevell: it's only a small step from there to "how many roads must a man walk down"
<jtv> (the answer is zero, by the way: the hypothetical man is already a man by the assumptions made in the question)
<mrevell> :)
 * mrevell goes to create #launchpad-philosophy
<jtv> mrevell: it'll be only you, so a nice exercise in solipsism.
<mrevell> :)
<jtv> And if you leave, will the channel be nihilist?
<jtv> Then again, existentialism says you have to do it in order to validate your existence
<jtv> Personally I prefer Meatloaf.
<jtv> Makis: don't let this babbling fool you, I'm still experimenting
<jtv> kfogel, when you get here (*much* later I hope), Makis' problem will interest you: errno -2 while trying to obtain rocketfuel-setup with "bzr cat" as documented on https://dev.launchpad.net/Getting
<stub> jtv: Maybe a glitch - we have only seen the spikes today. I don't see any previous spikes like this looking at the monthly or yearly graphs.
<stub> jtv: Are aborted poimport jobs retried automatically?
<jtv> stub: nope
<jtv> they fail, and if they fail this dramatically, there'll be no error output.
<stub> jtv: Can we manually retry the jobs that failed today?
<jtv> What surprises me is that I haven't seen it show up on error-reports either.
<stub> Hmm
<jtv> stub: we don't know exactly which ones will have failed today, but we can reset the status on recent FAILED uploads with error_output IS NULL
<stub> Will that annoy users?
<jtv> stub: I don't think so.  As long as we do this only for recent uploads, these are cases where the uploader should have been notified but wasn't.
<stub> If 'no' or 'not much', it sounds worth a go to see if we can reproduce the problem.
<jtv> So set status back to 1 where it was 4, and where dateimported (a blatant misnomer) is not much older than the 24th, and error_output is null.
<jtv> spm, still here?  You shouldn't be, but if you are, you wouldn't know of anything changed with our setup for users getting RF in the past day or so would you?
<thumper> who decided that TAGS over site-packges was a good idea?
<stub> UPDATE TranslationImportQueueEntry SET status=1
<jtv> thumper: I don't know, but let's just blame Bill gates like we always do.
<stub> WHERE status = 4 and error_output is null and dateimported > date '2009-08-03'
<stub> Sorry - 24th
<jtv> stub: and month 07, and day 23 :-)
<stub> UPDATE TranslationImportQueueEntry SET status=1
<stub> WHERE status = 4 and error_output is null and dateimported > date '2009-07-23'
<jtv> stub: then, plug your ears, crouch behind a solid object, and watch the importer go to work.
<jtv> yup, that looks right
<stub> 778
 * thumper gets the desire to smack someone around
<thumper> I'll avoid looking at bzr blame
<jtv> thumper: no, do look at bzr blame.  At least that way I'm reasonably sure I'll be safe personally.
<jtv> stub: that's quite a lot.
<jtv> stub: I should have seen lots of complaints on error-reports about that...
<jtv> stub: useful piece of trivia here: dateimported is the date of the original upload.  It does not get updated during import, or subsequent upload of a newer version of the same file, or anywhere else.
<stub> sounds like that needs expanding
<jtv> stub: dateimported should actually be called date_uploaded.
<jtv> stub: it is the creation date of the import queue entry.
<bigjools> hey wgrant, thanks for taking on bug 385129
<mup> Bug #385129: add PPA dependencies information to the api <api> <ppa> <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/385129>
<stub> I'd have date_created and date_uploaded if you can upload a newer version, or something like that
<jtv> stub: we gave it a better name in the web service API, but haven't bothered to rename the attribute itself.
<stub> Hmm... we have two poimport connections
<jtv> There can be only one.
<jtv> Unless one is for the slave.
<stub> Nah - one for the lpmain replication set, one for the authdb replication set
<stub> Actually, scrub that
<jtv> does it make sense for a script like this to talk to the authdb?
<danilos> jtv: poimport emails don't go to error-reports anymore, don't they?
<jtv> danilos: oh, that must be it.  Don't know why, but I thought that'd been reverted.
<jtv> Hi btw :)
<danilos> jtv: yeah, hi :)
<danilos> jtv: we've got stuff like http://launchpadlibrarian.net/29834164/uK8pt7QInv178EM0SyyuKEuaZai.txt
<jtv> danilos: henning's still sick :/
<stub> jtv: https://pastebin.canonical.com/20726/ -- same database, same db user, both tables only available in the lpmain replication set.
<danilos> jtv: yeah, noticed
<jml> hello everyone
<jtv> hi jml
<danilos> jml: hello
<jtv> stub: iirc the script is supposed to lock...
<stub> jtv: Looks like approve imports is connecting as the poimport database user?
<jml> wuu, all the hip .eu people are around :)
<danilos> jtv: poimport logs on devpad have a lot of tracebacks like this
<jtv> oh, yes, that could be it.
<danilos> jml: all the hip .eu people *and* me too :)
<jtv> danilos: where did those logs live again?
<danilos> jtv: let me check, I've got a symlink in my devpad $HOME
<danilos> jtv: devpad:/srv/launchpad.net-logs/scripts/loganberry
<jtv> danilos: thanks
<stub> https://bugs.edge.launchpad.net/rosetta/+bug/118633
<mup> Bug #118633: pofile import approver should connect as a different database user <fix-it-friday> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/118633>
<Makistos> jtv: thanks for your help, it seems our corporate network is blocking ssh so that's why the download of the trunk didn't work
<danilos> jtv: many, many failures like this since the rollout, I can take a look myself
<jtv> Makistos: but the "bzr cat" went over http...
<jtv> danilos: thanks, then I'll go back to the message-sharing.
<Makistos> yeah, but at least i got the thing working by downloading the setup script from your link and using a different network for the rest
<jtv> Makistos: glad to hear it.
<jtv> stub: so we're getting a lot of nasty import failures that aren't visible on error-reports anymore.  :-(
<jml> I'm creating a celebrity team
<jml> and I need to put it into the sample data
<jml> what should be the PersonCreationRationale for the team owner?
<thumper> jml: what celebrity team are you creating?
<jml> thumper, techboard
<jml> thumper, it's actually already referred to in the code, but not as a celebrity.
<thumper> ew
<jml> exactly
<jml> so I'm fixing that before I proceed to write some tests that actually need to use it
<jml> I'm going with PersonCreationRationale.UNKNOWN for the time being
<thumper> jml: did you see that mwhudson fixed the setBranch API bug?
<jml> thumper, yes, I saw that.
<jml> very embarrassing
<wgrant> bigjools: cprov suggested it. I did the read-only export hours ago, but do you think exposing {add,remove}ArchiveDependency is useful?
 * thumper shrugs
<thumper> jml: sometimes it just takes someone else to look at it
<thumper> jml: people can be too close to the code sometimes
<thumper> I'm about to propose a branch for merging that _may_ be causing our segfaults
<bigjools> wgrant: it depends on who would use it, nobody I know needs that right now, so if you don't need it, leave it for later.
<thumper> anyone interested in reviewing it?
 * thumper should go to #launchpad-reviews
<wgrant> bigjools: It'd also need some shuffling of the security checks from browser code to model code, so it's not just a trivial export. I'll leave it.
<bigjools> wgrant: don't put security checks in model code
<bigjools> bad bad idea
<bigjools> security checks go in the ... gasp.... security code :)
<wgrant> bigjools: Well, yes, but I'm not sure how that would work in this case. The restriction in question is that the user must be able to view the PPA to add it as a dependency.
<wgrant> Although that's already done in syncSource, so I might look there just to see how it's done.
<wgrant> Oh, that just works because getPublishedSources is secured.
<bigjools> wgrant: it's done in the security adapter
<wgrant> bigjools: But it depends on an argument, not the context or method.
<wgrant> addArchiveDependency doesn't need to do anything to the dependency argument, so no checks are ever done.
<wgrant> So you get nasty nasty bugs like that one in January.
<bigjools> if the security adapter is set up properly, it will be okay
<bigjools> it's not always possible to make the decision there though
<wgrant> I don't think it's possible.
<bigjools> upload ACLs being one example
<wgrant> Right.
<wgrant> How would a security adapter catch this case?
<jml> bigjools, :)
<bigjools> jml: :D
<bigjools> wgrant: you would need lp.Edit on the archive
<wgrant> bigjools: You need lp.Edit on the dependent archive to call the function, but not the dependency archive.
<wgrant> bigjools: Unless you explicitly check for the permission in addArchiveDependency, which is probably bad.
<bigjools> right, you need lp.view on it
<bigjools> which will get checked in the sec adapter as soon as you try to reference it
<bigjools> write a doc test, you will see (I hope)
<thumper> night all
<noodles775> Enjoy your evening thumper
<jml> thumper, g'night
<wgrant> bigjools: Why would the security adapter fail an attempt to reference the archive?
<wgrant> Everybody can get the distro of a P3A.
<wgrant> And other stuff.
<bigjools> hmph
<wgrant> And can't I pass a securityproxy'd object around as much as I want?
<wgrant> It only cares if I try to access attributes.
<bigjools> pesky kids
<jml> wgrant, you can pass around a securityproxy'd object as much as you want.
<bigjools> that's what he said
<bigjools> we could do with views for API calls
<wgrant> So if I don't try to do anything to it, I will never die.
<wgrant> bigjools: Right.
<bigjools> some of the model code takes a user parameter.... ewww
<jml> bigjools, it's not the worst thing in the world
<jml> bigjools, granted, it's not ideal.
<bigjools> well it means that the content class becomes tied to a request
<jml> there's a way of saying 'this parameter is the logged-in user' in the API permissions.
<jml> bigjools, not necessarily
<bigjools> yes, I hate that
<bigjools> our content classes are already huge :(
<jml> bigjools, ISourcePackage.setBranch for example, the user parameter is the registrant. That's very much a model-level detail.
<stub> jtv: Perhaps create a new rationale?
<jtv> stub: did you mean jml?
<bigjools> jml: that case is fine
<stub> I guess the existing ones would be lies then
<stub> Yup - jml
<jml> stub, UNKNOWN is only a white lie :)
<wgrant> bigjools: Anyway, I will go with read-only for now.
<bigjools> wgrant: good plan
<bigjools> baby steps, and all that
<bigjools> jml: btw, the extra info on the pqm landing emails is top drawer
<jml> stub, 'make newsampledata' generates a file without a copyright statement, which leads to spurious diffs -- how would I fix that?
<jml> bigjools, thanks, but I'm pretty sure that wasn't me :)
<bigjools> jml: you're the only representative from the code team who's awake right now :)
<jml> :)
<jml> stub, I'm thinking that I should change the 'build_new_sampledata' definition in database/schema/Makefile to echo the copyright statement to the file.
 * jml just does that.
<stub> sounds sane
<jml> hmm. a lot of targets in this makefile don't actually create files named after the target, and aren't included in phony.
<jtv> danilos, that diff vous wanted: https://code.edge.launchpad.net/~jtv/launchpad/bug-408206/+merge/9579
<danilos> jtv: looking
<jml> how do I determine which config is being used?
<jml> stub, can I get 'make harness' working from launchpad_ftest_playground? I want to use it to add some test sampledata.
<wgrant> bigjools: One questionable thing I did to get the export of IArchiveDependency working was replace IAD.archive's Choice over the PPAVocabulary with a plain Reference, or lazr.restful ignores it in the WADL export. Should I instead create a PPAChoice?
<bigjools> let me check
<stub> jml: make harness LP_DBNAME=launchpad_ftest_playground might work
<danilos> jtv: it is simple enough, but without checking that it really helps, I'd really like to avoid asking for a cherrypick
<jml> oh huh... 'make harness LPCONFIG=test-playground' is actually mentioned in the README.
<stub> My method doesn't work
<jml> stub, I'm just verifying my method.
 * jml might tweak the README to make this more obvious.
<jtv> danilos: we can be pretty sure it helps with the "commit more often" problem.  :-)
<jtv> (Which is actually what triggered all the other problems)
<danilos> jtv: yeah, but "making stub happier" is usually not a good enough argument for a CP, even though we can argue if it should be :)
<bigjools> wgrant: I don't know what the right thing to do is here, and I don't know why the existing choice would not get exported
<wgrant> bigjools: I think it's because it has no explicit interface.
<jtv> danilos: "make mvo happier" maybe?  :)
<danilos> jtv: heh, that is, but only if we can show that it will make him happier :)
<bigjools> wgrant: could be, stuff like PublicPersonChoice inherits from ReferenceChoice
<jtv> danilos: alternatively, we can tell mvo to STAY AWAY FROM OUR SHINY DATABASE for now.  :-)
<wgrant> bigjools: PublicPersonChoice is able to be exported, and it basicall just has the interface set.
<wgrant> bigjools: Ah, true, that too.
<jtv> (I tried to contact him yesterday, but no luck)
<bigjools> wgrant: ok, give PPAChoice a try, but call it ArchiveChoice
<danilos> jtv: no, alternatively, we can try this out on staging, and then see if it will fix stuff for mvo, and then ask or not for a CP
<bigjools> wgrant: actually no, PPAChoice
<wgrant> bigjools: Doesn't it actually want to be restricted to PPAs, though?
<bigjools> yeah
<jtv> danilos: once staging is working again, of course.  Absolutely.  I'm talking only about what we do until then, if it takes longer.
<wgrant> Right.
<danilos> jtv: well, do you know what's up with staging restores? they are not working either
<jtv> danilos: no, they haven't been working lately but I think there was no errorâwhich would probably mean there's a deliberate lock stopping them.  I mentioned it in my email to the LOSAs, so presumably they know about it.
<jml> make schema takes way too long :(
<stub> danilos: The long running transactions are victimizing other parts of the system. We either need to cherry pick the fix, or I need to be more aggressive about killing long transactions (and killing your jobs).
<jml> RAOF, hello
<danilos> stub: certainly, but that doesn't mean that we should cherrypick something without testing it on staging first (which is what jtv and I were discussing); fwiw, this might still be using long transactions, and we have symptoms which can tell us if they are
<wgrant> bigjools: Some grepping around showed I just needed to replace the Choice with a lazr.restful.fields.ReferenceChoice and specify a schema. Now it all works.
<bigjools> ah!
<stub> danilos: Sure.
<bigjools> wgrant: I am shocked that you didn't have to deal with circular imports
<bigjools> circular import DEATH HELL
<wgrant> bigjools: I did.
<wgrant> bigjools: But they were easily patched.
<jtv> danilos, stub: for now at least, the problems aren't happening because that one large export has been disabled.
<bigjools> ah good, sometimes they are not.  I hope you liked my patch_ funcs :)
<wgrant> bigjools: They're very useful.
<danilos> jtv: right, but we need to re-enable it on staging and test it
<RAOF> jml: Hello there.
<danilos> jtv: I'll email mvo about it
<jtv> danilos: absolutely.  But again, talking about the very short term until we can do that.
<bigjools> inline bug commenting ROCKS
<danilos> jtv: we've got a workaround for the time being: email mvo and disable ddtp bzr exports; a week of waiting won't kill him, I believe
<jtv> danilos: I had already disabled them, so we're in the clear for now.  The last run succeeded.
<danilos> jtv: right, but mvo is unaware :) I'll email him (CCing you) and mention the bug we are working on
<jtv> danilos: thanks.
<jml> bigjools, hi
<bigjools> helleau
<jml> bigjools, who owns the 'build source packages from branches' task?
<bigjools> I don't think anyone does yet
<bigjools> Tim and I talked about it
<jml> bigjools, cprov, james_w & I specced out a solution at UDS, and I saw you mention that you & Tim talked about it
<jml> & mwh @ UDS
<bigjools> yeah, it's not as trivial as first thought
<bigjools> we need to have a job farm like PPA builders that run VMs that will do the bzr builddeb stage
<jml> yes.
<jml> that's what we've got on our spec
<bigjools> good, so far :)
<jml> (which currently lives on a big sheet of butcher's paper in my backpack)
 * jml is trying to think of the next step here
<jml> it's a very important feature, someone ought to own it, and I don't think I can reasonably volunteer.
<jml> although I'd rather like to.
<bigjools> jml: I can suggest transferring your notes to a blueprint
<jml> bigjools, yeah, that will definitely happen
<bigjools> :)
<bigjools> jml: the first step is to break the work up into chunks, I think "someone" will be >1 person
<jml> bigjools, the project still needs a single person to own it though
<bigjools> jml: not necessarily, but yes it does need a single person up front to do the initial breakdown
<bigjools> I think most of the work is on the Soyuz side
<bigjools> and IS
<bigjools> so it might make sense for me to own it
<jml> bigjools, fair enough. I think the main idea is that someone needs to be able to answer the question "how are we going with the build a branch from source package work?"
<jml> bigjools, I'll put the spec into a blueprint & send a message to the dev list & your good self.
<bigjools> okiedokie, thanks
<jml> bigjools, btw, liw just sent a spec / rfc to the launchpad-dev list that you might be interested in.
<bigjools> jml: ok, it's not arrived yet but I'll take a butchers in a bit
 * jml gains rhyming slang xp
<jtv> herb: nag time again.  See my email to the losa list... could you run it with LP_DEBUG_SQL_EXTRA enabled so it creates a full log of what's going on?
<stub> Why does checkwatches chew up so much CPU on the DB server. I would have thought the database side was trivial with all the time spent talking to the remote sites.
<stub> BjornT: Do you think this is expected, or should I open a bug and try to get some statement logs to see what it is doing. Might be missing an important index or something.
<BjornT> stub: i was going to look into that. checkwatches take much longer to run before, but i don't know why yet. the database side should be trivial
<stub> It might be obvious from the statement logs.
<sinzui> noodles775: ping
<sinzui> jtv: beuno. I have been thinking about application pages (translations.launchpad.net) and collection pages (/people). I suspect they are locationless, but unlike the other location pages we have, these are prominent.
<sinzui> jtv: beuno: this is like the global search issue. There is no pillar or person to create branding and tabs for. What are we showing at the top of the page?
<jtv> sinzui: I don't think I'd like these to be locationless.  That top tabs bar on the front page is such a nice way to get a quick top-level overview.
<sinzui> jtv: I think you are right
<jtv> In fact I get a reasonable approximation of the front pages we know and love by using main_side and moving the buttons list into the side slot.
<sinzui> jtv: beuno: I wonder if we need an alternate layout (and heading) for application pages (which are roots) and /people /bugtrackers with was collections in rooted apps
<beuno> sinzui, on a call for another hour
<beuno> but I'll think about it
<sinzui> beuno: understood
<beuno> sinzui, I suspect we could just find a good context instead
<beuno> users have a different perception of context than we do
<jtv> sinzui: another question I had... we still have a few isolated actions menu items <boo!> here and there.  If we are to move those inline, do we replicate the permissions requirements in the TAL conditions?
<slytherin> does launchpad go through any load testing?
<sinzui> jtv: put the links in a menu so that the permissions and text are consistent
<jtv> slytherin: mostly by actual use.  Why?
<jtv> sinzui: what kind of menu?
<slytherin> jtv: just curious. I recently packaged jakarta jmeter for Ubuntu. It is a good application to do load/stress testing of web applications.
<sinzui> jtv: If the links act on the context in a global way (change details) then it may go side portlets
<jtv> herb, got time for my script run today?  Nag, nag. :-)
<jml> any vim users about?
<jml> I have an emacs thing that makes it really easy to run 'bzr ls -VR --kind=file --null | xargs -0 grep -In <foo>' in the editor
<jtv> mars: see slytherin'
<sinzui> jtv: as for menus, I have placed links in application menus (overview for my app) and in navigation menus (for views that share menus)
<jml> lots of our emacs users love it, since it makes it really easy to find obscure code in launchpad
<jtv> s note
<jml> how can this be bundled to provide similar joy for vim users
<herb> jtv: yes. in a few minutes.
<jtv> herb: cool thx
<bigjools> jml: what does that do?
<jml> bigjools, it greps all of the version controlled files in a branch that aren't binary files
<bigjools> jml: I have an LP vim hacking page on the old lp wiki
<sinzui> jtv: While we thought we were getting rid of navigation menus, this was wrong. The new design encourages them in fact. I do not think we will have navigation menus on model objects though, or sub menus. We just need to define sets of links that are common to a set of pages
<bigjools> jml: I will migrate it and add your special bzr-fu command
<mars> Ursinha, ping, ^ regarding slytherin's question, do you know how far matsubara got with the stress testing tools?
<jml> bigjools, cool, thanks.
<jml> bigjools, it's generically useful for bzr. you might want to look at the .el file linked from https://dev.launchpad.net/EmacsTips for how it works in practice
<jtv> sinzui: will the existing app menus still show up in the new macros then?
<jtv> sinzui: sorry, nav menus.
<sinzui> No, beuno hates that design.
 * jtv wishes UI design would stop affecting the code :-)
<sinzui> jtv: I dismantled the product edit nav menu, then realised that I put the same set of links on 5 pages. So I restored the navigation menu and used it to define the canonical set of links for edit pages.
<beuno> jtv, we all do  ;)
<jtv> beuno: stay out of this.  You're the one causing this pain.  :-P
<jtv> sinzui: so what did you do to make the nav menu show up in the actual page again?
<bigjools> jml: I'd rather not look at *any* .el file ;)
<sinzui> jtv: I don't think navigation menus on context objects will be needed. The ones on the view still have value. Move the links into the main content area. If you have a set of "related pages" that you want that should be shown use the view/@@/+related-pages call to generate a portlet of related pages
<jml> bigjools, :)
<jml> gary_poster, I just filed https://bugs.edge.launchpad.net/launchpad-foundations/+bug/408897 -- maybe it's not appropriate
<mup> Bug #408897: Use latest zope.testing <Launchpad Foundations:New> <https://launchpad.net/bugs/408897>
<bigjools> for reference: https://dev.launchpad.net/UltimateVimPythonSetup
<Ursinha> mars: actually that was me, I was working with FunkLoad
<jtv> sinzui: ah, I just read about that transformation.
<Ursinha> me and stub
<sinzui> jtv: look at lib/lp/registry/templates/product-edit.py for an example of how a nav menu went from being implicitly rendered by the layout to explicitly rendered by the template.
<jml> (but I just came across a corker of a testrunner bug while helping cjwatson with his python2.5-based Launchpad development)
<jtv> sinzui: cool, examples are good.  Thanks.
<bigjools> jml: have you seen ack-grep BTW?
<jml> bigjools, the name rings a bell.
<bigjools> it's written in perl but I won't hold that against it (too much)
<jtv> sinzui: I don't mind (well, not very much) the menu as a whole needing some work, but I didn't want to have to find a new home for every link in it and make sure it shows up at the right times.  :)
<mars> Ursinha, slytherin was just wondering if we wanted to try out jmeter, as he has packaged it for the base distro
<sinzui> jtv: I did a small refactoring the the ProductEditNavigationMenu. I moved the links it uses and the overview menu uses to a mixin to ensure they links had the same implementation...they were different in the menus
<sinzui> jtv: if the information does not naturally fit in the content, but users will realise after reading the page that they want to know more about the subject, use a navigation menu and the +related-pages. call
<gary_poster> jml: ok, cool.  sorry for the pain.  the zope buildout branch is next in line for me to return to it. :-) :-/
<jtv> sinzui: a mixin sounds like a nice solution.
 * bigjools curses at the miserably crap GTK file-open dialog corrupting the perfection of KDE4
<jml> gary_poster, no worries at all. just wanted to know, since a lot of this python upgrade stuff is still fairly opaque.
<bigjools> jml: I have no idea what that dot hell file is doing, I will need help to convert to a vim macro :(
<gary_poster> jml: you mean, wanted to know if we planned to move to a newer version of zope.testing?  or where we were in the process?
<jtv> sinzui: none of this is particularly good for our single-option admin-only "edit this object" links, but maybe there mentioning the privilege in TAL isn't so bad.
<andrea-bs> hijacker_, sinzui. I wanted to work on bug #315858. It's marked Fix Released, but I can't understand why. Also looking at rev 7974 I don't find anything useful. Could you help me?
<mup> Bug #315858: It's not obvious how to subscribe to a mailing list <mailing-lists> <registry-people> <ui> <Launchpad Registry:Fix Released by sinzui> <https://launchpad.net/bugs/315858>
<cjwatson> jml: http://paste.ubuntu.com/247277/ is the "wcgrep" thing I use; I believe I nicked it from the Subversion tree years ago and hacked on it as needed
<bigjools> jml: does emacs have python bindings?
<sinzui> andrea-bs: I think we should avoid working on this issue until launchpad engineers stop giving teams and subscriptions dual roles of communication and permissions. The situation is completely buggerd
<maxb> jml: How is cjwatson doing his python2.5-based Launchpad development, ooi? Using what I've noted on http://dev.launchpad.net/LaunchpadOnKarmic or an independent effort?
<sinzui> andrea-bs: It is never clear how to get a mailing-list even if you can read the code and see in the database. The fix we did was better. We can try to improve it in the team redesign
<cjwatson> maxb: I'm using your stuff
<maxb> yay :-)
<cjwatson> well, more or less, it's been a little while since I synced
<cjwatson> *please* get that merged if at all possible :-)
<maxb> I still haven't managed to convince the testsuite to run all the way through
<cjwatson> I'm not even trying
<gary_poster> maxb: you don't have to get the test suite to run all the way through in Py 2.5 to commit
<gary_poster> maxb: you have to get it to run in 2.4
<gary_poster> so maybe that would make it easier to make incremental checkins?
<gary_poster> (s/commit/merge)
<sinzui> andrea-bs: The problem is that if a team creates a mailing list, the team is clearly intended for communication, so every member should get the list mail. Instead of addressing the use case and rethinking the purpose of teams, we introduced a maze of rules for the team owner and team member to negotiate.
<maxb> I really ought to get myself a working 2.4 environment. I don't have one at the moment.
<gary_poster> maxb: sounds like a plan ;-)
<cjwatson> it's a right pain to do in karmic. I tried briefly but life is too short.
<herb> jtv: devpad:/home/herb/jtv.out
<cjwatson> (chroots are of course possible)
<jtv> herb: great, thanks.
<gary_poster> cjwatson: granted :-/
<cjwatson> also, OMG distroseries.createUploadedSourcePackageRelease is a painful interface
<gary_poster> heh
<jtv> herb: this time I caught the bugger with an explicit sync, just what I needed to pinpoint it.
<andrea-bs> sinzui, I see that the situation is a bit complex :) Thank you for the clarification.
<sinzui> andrea-bs: I really want it to be simple. I have a bug that is blocked by this situation. As a team owner, I want to send an important message to each member. I cannot because user can be unsubscribed from the list, they may not even know they are unsubscribed.
<andrea-bs> sinzui, I know: I have reported a bug describing the same situation
<sinzui> andrea-bs: there was a proposal to add a bool to the team model to indicate that the team is intended for communication. I am not sure it is needed. We just need to guard the situation where the team can be placed in the dual roles of communications and ACL. We just need to ask the owner to confirm his choice, and give him an option to create another team to fill the needed role.
<salgado> beuno, you don't have a mockup of the project/distro page, like the one you have for the person page?
<andrea-bs> sinzui, As a team and project owner, I think that having separate teams for communications and ACL is a bad idea, because one can end up with too many teams. Instead I think that mailing lists should be less team-depended: for example anyone should be able to subscribe to a mailing list, also without being a member of the team (because often a mailing list is not just for members, but also for people who would like to
<andrea-bs> become a member).
<sinzui> andrea-bs: that has beend discussed too. When we list all the features user expect it to have, see it behaves like a team. IT has to support the same rules of aggrigation of membership
<sinzui> andrea-bs: There is another rule that any person of good launchapd standing should be able to post a message to any public list
<sinzui> andrea-bs: and any person should be able to receive list mail even if he is not a member of a team
<andrea-bs> sinzui, I agree that, in a sense, mailing list should behave like teams. But I think that having two real teams (one for ACL and one for communications) is really a bad idea because makes the work really harder: when someone wants to register a team, (s)he has to register two teams; when a member wants to join/leave a team, (s)he has to join/leave two teams; when an admin wants to change the icon of the team he has to
<andrea-bs> do this twice and so on...
<andrea-bs> sinzui, Also, and this is for me the most important point, the team/project owner has to tell the community to join two teams and why they should do this. Explaining such things to someone who has never used Launchpad before can be really difficult.
<mars> hmm.  so enforcing the communications team split would in effect be encoding policy into Launchpad
<mars> and project teams will ask questions when that policy is foisted upon them
<mars> andrea-bs, I agree with that last point.  It would be really odd.
<mars> and onerous to explain
<sinzui> andrea-bs: In the past we said DO NOT JOIN A TEAM. being a member of team should never be required to contribute. Teams are meant to be ACL. This was stupid. Every person on the planet knows that a team is a group of people with a common goal and they communicate.
 * jml is back
<jml> gary_poster, whether we plan to do so, I guess.
<mars> sinzui, :)
<jml> gary_poster, and also, I think that in an ideal world, there'd be a tag called python-upgrade and a lot of bugs filed against it.
<sinzui> andrea-bs: So team owers encourages users to join teams, but then discovered the members had more power than they should
<jml> bigjools, emacs has an extension that lets you use python bindings
<jml> bigjools, by default, it only has elisp bindings :)
<sinzui> andrea-bs: It might be helpful if we rethink what we did. Launchpad created control groups, and called them teams. A few years later, it created mailing lists, and called the teams. Maybe we do not need teams
<mars> sinzui, why not go with "teams to communicate", "capabilities for rights"?
<jml> maxb, I started this patch, btw, http://paste.ubuntu.com/247305/
<jml> maxb, I had to do a similar thing to get 'trial' working on python 2.[3456]
<mars> sinzui, so a team is a social structure
<maxb> jml: aha, I didn't attempt a [456] solution on that one yet
<jml> maxb, if my calculations are correct, it'll let TestCase work on Python 2.4 and 2.5.
<mars> sinzui, and capabilities (or groups with roles) codify the governance structure
<maxb> It's a bit worrying that we have to dig into double-underscore attributes at all
<sinzui> mars: That has been discussed. I want the team to be social, I want it to behave as users think of teams. I want to extract the ACL rules to a separate object to that is clear what the purpose is
<jml> (but I've bumped into some zope.testing bugs now)
<mars> sinzui, big +1
<jml> maxb, it's not our fault the unittest module sucks.
<maxb> jml: Why are you using getattr? Wouldn't direct attribute access work just the same?
<mars> sinzui, you'll have to answer the inevitable question, "Can't I give every other member the capability to do X?"
<danilos> rockstar, abentley, jml: where can we see if there are any problems with rosetta upload bzr job?
<mars> sinzui, but that is easy if you have the two objects clearly separated
<mars> s/objects/concepts/
<abentley> danilos: What kind of problems?  Oopses?
<danilos> abentley: I don't know, I am suspecting some OOPSes or exceptions, nothing is happening when it should, and I want to see why
<beuno> salgado, I do for project
<beuno> it's what sinzui is working on AFAIK
<jml> maxb, because the patch isn't finished yet :)
<maxb> heh
<sinzui> mars: I have recently been thinking about implementing roles in launchpad (owner, driver) as an implicit team, you are always adding to it (not assigning it). you cannot see it. It is there. Just like every project automatically gets a trunk series.
<abentley> danilos: I would expect to see them at devpad:/x/launchpad.net-logs/scripts/crowberry but there aren't any.
<jml> maxb, anyway, I'd personally be very happy to review & land patches of about that size that help with python 2.5 support
<gary_poster> jml: we do plan to do it--I'm in the middle of it but have had umpteen million things inbetween start and finish.  And +1 on tag, sounds reasonable.
<jml> since I'm running karmic & developing inside a chroot atm
<jml> gary_poster, I know the feeling :(
<mars> sinzui, ah, interesting.  Nice abstraction.
<sinzui> mars: two objects is my desire. I want to include it in my 4.0 goals, I also want subscriptions simplified. It should not be used  for ACL
<jml> gary_poster, sorry for the hassling. I'll make the tag now and encourage cjwatson, maxb & others to start filing bugs :)
<gary_poster> jml: no apologies needed :-)  cool thanks
<danilos> abentley: how often is that refreshed, if you know?
<maxb> jml: I need to get myself a chroot set up so I can validate that I haven't broken 2.4 - is it just a plain debootstrap of jaunty, apt-get install launchpad-dependencies, and rocketfuel-setup inside the chroot?
<abentley> danilos: Every 10 minutes, I believe.  Is this on production or staging?
<danilos> abentley: production
<salgado> beuno, right, I just had a call with him and he'll send me some screenshots. thanks
<jml> maxb, I'm running a hardy chroot, pretty much a plain debootstrap
<jml> maxb, but I guess jaunty would work fine
<abentley> danilos: Best to ask a LOSA, I think.
<beuno> sinzui, let me know if you need a call  to unblock you on anything
<beuno> noodles775, ditto
<danilos> abentley: sure, thanks
<beuno> bigjools, looking at your mockups
<jml> maxb, I think I'm going to abandon that patch I pasted you, since I really really want to get the official package branch permission thing up and running
<maxb> jml: ok then, I'll merge it into my branch
<jml> maxb, cool
<bigjools> beuno: jolly good
<jml> maxb, again, let me encourage you to land lots of small changes that get us incrementally closer to 2.5.
<gary_poster> +1
<jml> maxb, I'll review them faster than you can say Jack Robinson
<gary_poster> Jack Robinson
<maxb> I think you have to say it after a review branch exists :-)
<jml> yeah
<gary_poster> oh darn
<jml> also, you have to say it, and then mail me a cassette with the recording of you saying it.
<gary_poster> lol
<andrea-bs> sinzui, I like your idea, but without a team, how can somebody *easily* have permissions, for example, to push to a branch? I mean: clicking on "Join" and asking for a team admin to approve your membership is quite simple. If the team admin also have to give you the access to a branch, things get more complicated, don't you think?
<sinzui1> Sorry beuno, andrea-bs, mars: Firefox is killing X. I cannot see your messages
 * sinzui1 is trying a different computer
<jml> leonardr, hi
<jml> leonardr, I have a cred-cache branch for launchpadlib that we talked about some time ago. You suggested that I ping you about it after a while.
 * jml loves being in a convenient timezone
<leonardr> jml: there's a slight chance someone else wrote and landed your branch
<leonardr> how long ago are we talking about?
<jml> maybe two months
<jml> nope
<jml> the merge proposal was requested on  2009-06-18
<jml> leonardr, https://code.edge.launchpad.net/~jml/launchpadlib/cred-cache/+merge/7608
<leonardr> jml, take a look at Launchpad.login_with()?
<leonardr> no, that already exists in your branch, so you must be doing something different...
<jml> leonardr, I've added a convenience function that does this:
<jml> - look for cached credentials in a deterministic location
<jml> - if they exist, log in with those
<jml> - if they don't exist, get a new set of credentials from the user and store them in a deterministic location
<beuno> sinzui, I just said to let me know if you need a call to unblock you on anything
<sinzui2> beuno: I am preparing screencaps of the project page now
<beuno> sounds good
<leonardr> jml: i believe login_with() does the same thing. take a look and see if you agree. there were a lot of reviewed but unmerged launchpadlib branches the last couple months due to the lazr refactoring
<andrea-bs> sinzui, here's my last message: I like your idea, but without a team, how can somebody *easily* have permissions, for example, to push to a branch? I mean: clicking on "Join" and asking for a team admin to approve your membership is quite simple. If the team admin also have to give you the access to a branch, things get more complicated, don't you think?
<sinzui> andrea-bs: right. This is great example of why teams are really needed
<sinzui> andrea-bs: This kind of team may also want a list, in which case we the owner should be warned that  all members will get mail. if this is what the owner intends, continue, otherwise create a list.
<jml> leonardr, the code looks the same. I'll double check with a simple experimental script
<leonardr> jml, great, sorry for the duplication
<jml> leonardr, no worries at all. :)
<jml> cool. I'll abandon the branch and delete an item or so from my todo lists :)
<jml> leonardr, while I have you on the line, so to speak, can you take a look at bug 376284
<mup> Bug #376284: Ability to remove branches using the API <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/376284>
<jml> leonardr, I made an extremely vague comment at the bottom. ISTR that DELETE support was only recently added to launchpadlib?
<leonardr> jml, i'll look, but i think the only thing you can delete is file uploads
<jml> leonardr, but could we expose a DELETE thing for branches?
<jml> in fact, I'll let you just comment on the bug, rather than reply to my rambling questions.
<leonardr> jml, sure
<sinzui> beuno: can you comment on https://bugs.edge.launchpad.net/launchpad-registry/+bug/405916 today. I have screencaps and issues for you to think about
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
<leonardr> jml, commented
<beuno> sinzui, sure, right after lunch
<jml> leonardr, thanks.
<beuno-lunch> bigjools, the second option is looking better
<beuno-lunch> bigjools, didn't we find a different way to display in what archs it was built?
<bigjools> beuno-lunch: not that I know of
<beuno-lunch> columns seems...  odd and not very scalable?
<beuno-lunch> bigjools, could we hide the bits of information that don't have a value?
<beuno-lunch> "Suggests:", for example
<bigjools> beuno-lunch: yes, I planned on that, it's just there for the sake of the mockup right now
<bigjools> beuno-lunch: but the table seems perfect to show what we need, although it have a lot of columns in production
<bigjools> s/it/it will/
<mrevell> see you tomorrow guys
<EdwinGrubbs> sinzui: ping
<EdwinGrubbs> beuno-lunch: ping
<beuno> Edwin-lunch, hi
<salgado> sinzui, what's the rule for the usage of the top-portlet class?  I thought there should be only one element with that class in a page but I see 3 in the new project page on your branch
<sinzui> salgado: It i the top of a column. It just defines whitespace rules and does not have a top-border. Lots of simple pages like edit forms do not require it
<dobey> hey all
<dobey> is there a public project on lp that would be an appropriate place to stick some useful client tools that use the API to do things?
<salgado> sinzui, I see, thanks
<beuno> leonardr, ^
<leonardr> dobey: we usually put those scripts in launchpadlib/samples
<EdwinGrubbs> beuno: you mentioned that the latest design has a place for global actions. Is that above the sidebar as shown in the mockups. It seems kinda strange to not be in the sidebar.
<EdwinGrubbs> sinzui: ping
<sinzui> Hi EdwinGrubbs
<EdwinGrubbs> sinzui: meeting?
<beuno> EdwinGrubbs, sinzui has the mockup
<beuno> it's in a portlet on the right
<sinzui> EdwinGrubbs: yes. we should talk. And I have two images for you
<sinzui> EdwinGrubbs: You have mail
<salgado> sinzui, is the 'Mentoring' link gone from the new project page?  if so, should it not exist in the new project-group page as well?
<sinzui> salgado: I am infavour of trying to make mentoring work. Recent discussion by others imply we should abandon the feature
<sinzui> salgado:  I did not realise that I lost mentoring in the product-index draft I did
 * sinzui thinks beuno would approve of that
<salgado> heh
<salgado> consider it gone from the projectgroup page too, then. :)
<beuno> \o/
<sinzui> beuno: salgado: if we drop mentoring without trying to fix it, users will point to it as an example of something we built, never dogfooded, and never cared about.
 * beuno high-fives salgado 
<beuno> sinzui, and it will be true
<beuno> I'm all for fixing it, but I'd only leave it on there if we have a concrete plan to do so
<sinzui> well I agree with that, and the best that can be said is that it is a 4.0 candidate.
<sinzui> beuno: salgado: let's ignore the feature
<beuno> :)
<sinzui> As I think about this. we have ignored bounties long enough. When are we removing the code?
<beuno> I don't like missleading users into thinking they can do something they really can't
 * sinzui knows there are hacks in the test suite to ensure the rotting code does not break other tests
<salgado> sinzui, I see some portlets with class="yui-u portlet" and others with just class="yui-g" in product-index.pt, but I can't see any difference between them... are both meant to be used? if so, when?
<sinzui> hmm
<sinzui> I think I need to update the wiki with your observation
<salgado> actually, it's only the series-and-milestones portlet that uses "yui-u portlet"
<sinzui> yui-g created columns, yui-u is a column. The first column is 'yui-u first'
<beuno> salgado, has someone told you about the navigation work we're going to be doing together?
<sinzui> salgado: 'yui-u portlet' makes a column and presents it as a portlet, but I discovered that yui-u and first must always be on the same div, which does not work for how we embed portlets
<salgado> sinzui, sorry, I meant that some have "yui-u portlet" and others have just "yui-u".  your email mentions the distinction between yui-u and yui-g
<sinzui> salgado: so I decided that our portlet will have the portlet class, and the calling template will wrap the portlet in a yui-u div (optionally defining the first one)
<salgado> (man, how can it be so painful to type yui.  is it just me?)
<beuno> sinzui, the ffedback I need to give you is in bug 405916, right?
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
<salgado> beuno, nope, nobody mentioned it to me. when are we starting?
<sinzui> salgado: I think I need to revised my product-index page to never mix 'portlet' with 'yui-u' since it often breaks
 * sinzui will try to explain the problem better on the wiki page
<beuno> salgado, as soon as I manage to focus for 1 hour straight. The gist of it making breadcrumbs more detailed, so we need specific pages to be able to generate a path we decided on
<sinzui> beuno: yes, that one.
<beuno> sinzui, ok, it's almost next on my list then
<beuno> I'm loosing track of the balls I have in the air  :)
 * sinzui is a day behind on email because he needs to get UI 3.0 example to everyone, which mean actually creating what ahs never been done before
<beuno> sinzui, email is for whimps
<beuno> if people really want to talk to you, they will go meet you in person, like I did
<beuno> sinzui, also, the new project pages look sweet
<beuno> even more so when they're busy projects
<beuno> what do you think about not showing the sections that have no data?
<beuno> I talked about this via email with EdwinGrubbs
<sinzui> I have mixed feelings. I am torn between the bugs reported that something is not obvious because we not not clearly state: There is not mailing list, there are not branches, there are no bugs, and the desire to show only what I want to know
<beuno> sinzui, in some cases, sure
<beuno> but top contributors, latest questions, events, etc
<beuno> they add no value
<sinzui> beuno: We need to be consistent so that I can mark bugs WONTFIX
<sinzui> beuno: and without those sections users do not know what will happen when they appear
<beuno> sinzui, only show "this does not exist" for objects that can be added/created?
<sinzui> I can add a create FAQ
<sinzui> considering there is only one link on the site to do this...this would double the number of places
 * sinzui like the rule and will use it to close some bugs
 * beuno is practicing his create-rules-on-the-fly skills
<beuno> BjornT, inline comenting is SO much nicer
<beuno> requests seem to take a long time to get back
<beuno> I wonde rwhy
<mars> beuno, how many MS does it take?
<mars> milliseconds
<beuno> mars, I probably waited for 5 or 6 seconds
<sinzui> beuno: I think yui-u and yui-u first will break with optional content
<mars> I'm curious, because one of the big negatives to adopting AJAX was the server response time
<mars> 5 to 6 seconds?  :(
<beuno> yeah, maybe we're under some stress in teh servers, I don't know. Will continue to observe
<mars> 5000ms ! <= 450ms
<beuno> mars, try it out, let me know how it goes  :)
<sinzui> beuno: when sprints appeared on it own in my page, it was positioned to the right. I think every portlet that has a first must be guaranteed to render
<beuno> sinzui, so we can't float them to the left?
<sinzui> well that is what first is doing. but the way YUI-grid is working the calling template and the portlet do not know what they are doing. This was a big layout problem for me
 * beuno tries to decide if he will let technology take over UI or not
<mars> beuno, which bug were you using?
<sinzui> beuno: I considered the option of laying out the page from right to left. Then I do not need first, I just need to to know the page portlet calls are the reverse of what you see in the page
<beuno> mars, https://launchpad.net/bugs/405916
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
<beuno> sinzui, that sounds like a solution as well
<sinzui> beuno: the approach is awkward, but guaranteed to scale with a bug page with lots of shared portlets
<beuno> sinzui, it solves my problem, so that's as far as I can approve of that
<sinzui> just remember that the order you see in the page is not what you see in the code
<beuno> that's going to cause a few WTFs
<beuno> but just once  :)
<mars> beuno, have inline comments landed on staging yet?
<beuno> mars, no  :*
<beuno> :(
<beuno> didn't mean to kiss you  :)
<mars> urgh
<mars> Have to keep in mind to test the response times later on
<beuno> if time where free
<beuno> we would of gotten that profile information coded into the widgets
<beuno> and would have these amazing graphs
<beuno> and flying cars
<dobey> we have flying cars
<rockstar> dobey, just because you drive fast doesn't mean your car is flying.
<dobey> rockstar: actually, at speed one of my cars builds up enough pressure under the hood, that it creates lift
<dobey> rockstar: but i was talking about http://www.terrafugia.com
<dobey> or http://www.moller.com/
<beuno> mars, see #launchpad-cde
<rockstar> beuno, http://devpad.canonical.com/~rockstar/answers-portlets.png - http://devpad.canonical.com/~rockstar/answers-portlets-after.png
 * beuno looks
<rockstar> beuno, does that look better?  I think it does, but I'm also not the UI dude.
<rockstar> I think we need to do our best to not cram stuff together.
<beuno> rockstar, it does. It makes me wonder 2 things:
 * rockstar listens intently
<beuno> 1) Should we be cramming that in a portlet?
<beuno> 2) Should the information be bold instead of the labels?
<rockstar> To 1), I say yes, because it's meta information, and has no bearing on the actual page.  I'd like the branch page to take note from it.
<rockstar> I think the answer pages are about questions and answers, and everything else is extra.
<rockstar> To 2), I think it's good the way it is.  Originally I thought of changing font sizes, but I read a book that says that's bad, and I believe everything I read.
<beuno> rockstar, it's better, but it feels crammed, and hard to read
<beuno> I'm trying to think of what would improve that
<beuno> when you say it has no bearing with the actual page
<rockstar> Yeah, it does feel crammed.  That's why I spaced it out a bit.
<beuno> what page is it?
<rockstar> The answers index page.
<beuno> rockstar, some of it does feel as part of the page
<beuno> you could split this out
<beuno> you could display who filed it and when
<beuno> just like we do on bugs/blueprints/code
<beuno> that removes 2 items from there
<rockstar> Well, now that I'm looking at it, that's already there.
<beuno> status and asignee seem important, as who solved it and when
<rockstar> Also, why is there a "Last query" field.
<beuno> so that leaves a small portlet, which shouldn't feel crammed anymore
<beuno> sounds crazy
<beuno> probably "last time something happened"?
<rockstar> Wait, lemme screenshot the whole page for you really quick.
<rockstar> http://devpad.canonical.com/~rockstar/answers-page.png
<rockstar> We might do better on the phone for this actually.  Is that okay with youL
<rockstar> beuno, ^^
 * beuno looks
<beuno> sure
<beuno> let me find headphones
<beuno> rockstar, almost ready
<Ursinha> bigjools, are you there?
<thumper> morning
<thumper> I have to drop the car off for new brake pads
<thumper> I shouldn't be too long
<thumper> bbs
<mwhudson> good morning
<al-maisan_> Hello mwhudson and thumper!
<rockstar> al-maisan_, go to bed.  :)
<al-maisan_> rockstar: aye! :)
<jml> mwhudson, thumper: good morning
<mwhudson> jml: yo
<jml> mwhudson, mind if I skype you quickly to catch up on some stuff I missed last week?
<mwhudson> jml: go for it
<thumper> back
<rockstar> sinzui, I have a CSS question for you.
<thumper> jml: you areound for the standup?
<sinzui> rockstar: sorry, I wasn't ignoring you, I thought you were going to ask the question in the next minute
<rockstar> sinzui, oh, I was trying to do a content ping there.  :)
<jml> thumper: sure, I guess.
<rockstar> sinzui, so, I'm messing around with the subscribers portlet, and I need to make the h2 in the portlets have the underline and bold, like the subscribers portlets for bugs and branches.
<thumper> rockstar: skype doesn't like you this morning
<rockstar> sinzui, my understanding is that you are currently the CSS gatekeeper, so I wondered if you had done something already, or could suggest a place for me to put it.
<rockstar> thumper, really?
<jml> thumper: do it!
<thumper> jml: I'm trying
<sinzui> rockstar: I was not aware that I am, but I am happy to advise. My principle concern is reusablity. So I hope that every class added will be used by more than one app.
<jml> thumper: the call drop
<jml> fate hates us.
<thumper> jml: np
<jml> I'll read the minutes :)
<barry> gary_poster: ping
<thumper> jml: ok
<gary_poster> barry: swish!
<rockstar> sinzui, okay, so maybe I should coordinate with thumper, since he'll be the next fool to mess with subscribers.
<sinzui> rockstar: <h2> in the side portlets are grey in the design. why is it underlined?
<barry> gary_poster: hi.  i'm not really here <wink> but i have a quick question for you.  i'm getting a test failure in lazr.restful and was wondering if you could verify.
<rockstar> Well, beuno said he liked the way they were, and the branch page has them underlined.
<sinzui> rockstar: what happens when they are clicked? How do other developer know that their <h2> must be underlined.
<sinzui> rockstar: I think I can ask this differently. If this is special, what is that reason and how do we ensure eveything like it do also treated the same way so that engineers and users understand the rule
<rockstar> sinzui, well, this is strictly for subscribers.
<sinzui> rockstar: because there are direct and indirect subscriptions?
<rockstar> sinzui, kinda, but not really.
<sinzui> oh!
<rockstar> If there's a better way we can standardize, I'm all for it.  I was just instructed to make answers subscriptions look like everyone else's subscriptions.
<sinzui> sorry, I just looked at the current page. it has a rule under the heading,
<sinzui> I think the grey <h2> should have the same bold ass all side portlet headings.
<sinzui> rockstar: I think all need the rule or none do. I like the rule
<rockstar> sinzui, I do as well.  Where should I put the css for that?
<wgrant> Can somebody please ec2test lp:~wgrant/launchpad/export-archive-dependencies for me?
<rockstar> wgrant, I can.
<sinzui> There is already a css rule for .side h2. We can add an bottom border. I think that will look odd in the design that beuno approved for the project page though
 * sinzui tries it right now
<wgrant> rockstar: Thanks.
<rockstar> wgrant, is it approved for landing?
<wgrant> rockstar: It's not. Should it be?
<wgrant> None of those with whom I discussed it are around now.
<rockstar> wgrant, no, it doesn't need to be.
<rockstar> wgrant, it's just possible for me to have ec2 submit to pqm when done.
<wgrant> rockstar: Ah.
<sinzui> rockstar: beuno: compare https://devpad.canonical.com/~curtis/product/firefox-3.0.png with https://devpad.canonical.com/~curtis/product/h2-test.png All I did was add the old side portlet <h2> rule to the new CSS. This does not look very good. What can he change in the project page to make all content look good with this <h2>?
<thumper> phew
<thumper> finally
<thumper> inbox has no unread emails
#launchpad-dev 2009-08-05
<mwhudson> boo to not being able to restart the buildmaster ever :(
<thumper> huh?
<thumper> what changed?
<mwhudson> well, there's always a build running
<mwhudson> if i bounced it now, we'd lose the progress so far on the tests currently running
<mwhudson> maybe it's worth it though
<mwhudson> (it would be nice if the bzr builder ran today)
<mwhudson> thumper: opinion?
<thumper> hmm
 * thumper wants a stop.patch
<thumper> mwhudson: will we need to manually kick the builds to get them going again?
<thumper> mwhudson: if we won't, do it now
<mwhudson> thumper: i don't think so
<mwhudson> thumper: let's find out!
 * mwhudson restarts the wrong bit, oops
<wgrant> What's the bzr builder?
<mwhudson> builds seem to be going now
<mwhudson> wgrant: it runs the launchpad tests against bzr.dev
<wgrant> mwhudson: Ah.
<thumper> mwhudson: we have a new image for bzr?
<mwhudson> thumper: no, it was master only changes
<mwhudson> (a new image wouldn't require a master restart)
<thumper> mwhudson: ah crap
<thumper> mwhudson: my tests run fine locally
<thumper> mwhudson: but there are changes on devel that cause it to fail
<mwhudson> thumper: wheeee!
<thumper> mwhudson: but I really want this cherry pickable...#
<thumper> mwhudson: if I fix it to land, it won't cherry pick
<mwhudson> thumper: land it on devel first, then prepare a branch off production containing the fix?
<thumper> the branch I have is prepared off production
<mwhudson> oh right
<thumper> mwhudson: I think I'll push my current branch to a new branch on lp
<thumper> then fix to land on devel
<mwhudson> right, two branches is the answer i think
<thumper> ah fuck
 * thumper walks away for coffee
<thumper> I'll fix the problem later...
 * mwhudson afk for a bit again
<mwhudson> (well, for lunch)
 * wgrant cringes a bit at IAnnouncement.
<wgrant> And related interfaces..
<meaton2veggies_> wgrant: installed on jaunty now
<wgrant> meaton2veggies_: i386?
<thumper> wgrant: do a bzr blame on that one
<wgrant> thumper: I did see some comments around which indicate what I think you might be suggesting.
 * wgrant tries.
<meaton2veggies_> wgrant: yes, VM
<wgrant> Indeed.
<meaton2veggies_> wgrant: any idea how i fix self-signed cert with feeds.launchpad.dev:443?
<wgrant> meaton2veggies_: Tell your browser to trust it.
<meaton2veggies_> yep, doing that now
<meaton2veggies_> in firefox 3.5 it comes up on the front page launchpad.dev error alert
<wgrant> That's correct.
<wgrant> You need to tell Firefox to trust it.
<meaton2veggies_> do i have to create a proper cert for the local site
<wgrant> You could, but there's little point.
<meaton2veggies_> ok
<wgrant> thumper: How aggressively should method name conventions be fixed?
<thumper> wgrant: what are you wanting to fix?
<wgrant> thumper: IAnnouncement.set_published_date and IHasAnnouncements.announcements
<wgrant> (to setPublishedDate and getAnnouncements respectively)
<thumper> wgrant: are they methods?
<wgrant> thumper: They are.
<wgrant> And they both want to be exported.
<thumper> announcements is a method not an attribute?
<wgrant> thumper: Correct.
<thumper> wgrant: I'd review and approve those changes
<wgrant> thumper: It takes some arguments to customise the result. Maybe it used to be an attribute.
<wgrant> thumper: I shall start another branch and do the renames. Thanks.
<wgrant> Seems much nicer than using exported_as.
<thumper> wgrant: agreed
<wgrant> And they're not used much, so it's easy.
 * mwhudson reappears
<thumper> wb
<thumper> :)
<mwhudson> mars: "- Node.queryAll()'s signature changed.  It now returns an empty sentinel object with size() == 0 if no results were found, instead of returning 'null'. "
<mwhudson> hoo-bleeping-ray!!
<mwhudson> that was a particularly stupid change in pr2
<mars> hehe
<mars> well, it makes the test clearer, and universal
<mars> test for length: done!
<mars> I don't know what Y.all() returns though
<mwhudson> oh right
<mwhudson> in loggerhead i had to change a few Y.query(selector).each(...) to nodes = Y.query(selector); if (nodes) nodes.each(...)
<mwhudson> s/query/all/ i guess
<mwhudson> which just made me a bit unhappy
<wgrant> The sparklines really encourage me to read the project branch listing very quickly, before the horrible virus slowly infects all of the rows.
<lifeless> wgrant: thumper is landing a fix, or you could put one up :)
<wgrant> lifeless: They should be gone in a few hours, I think.
<wgrant> I'm surprised they missed the edge update last night.
<thumper> wgrant: heh
<thumper> wgrant: missed the cut off by about 2.5 h
<lifeless> thumper: thats ok, have a beer for commiseration. Or something.
 * wgrant defeats the stupid uni firewall, and successfully pushes that IAnnouncement branch.
<wgrant> thumper: Shall I request a review from you?
<thumper> wgrant: sure
<wgrant> Hmmm.
<wgrant> I attempted to search for 'devel' as the branch to propose the merge to, and the picker gives me one option: "undefined"
<wgrant> Oh. "Too many matches" is the error, "undefined" is the option.
<thumper> ew
<thumper> wgrant: try "lp:launchpad/devel"
<thumper> wgrant: once you've done it once, it'll remember
<thumper> and offer it next time#
<wgrant> thumper: Ah, useful.
<wgrant> thumper: Thanks. I'll file a bug.
<thumper> ok
 * thumper takes a break as 4 ec2's do their thing
<wgrant> So, a friend of mine just totally failed to realise that status and importance are AJAXy, because they have no icons.
<lifeless> wgrant: I filed a bug on that
<lifeless> search for mystery meat
<lifeless> bad web design since 1995
<wgrant> I thought I remembered seeing them.
<wgrant> s/them/one/
<thumper> WTF?
<thumper> got an invalid docstring :(
<mwhudson> apidocs are processed from docstings
<thumper> api doc compilation thingy
<thumper> where can I find out what exactly is invalid?
<mwhudson> thumper: i'd start with the diff against trunk :/
<mwhudson> woo bzr builder ran to completion today
<thumper> its a new docstring for a new method
<thumper> mwhudson: 8 failures
<thumper> mwhudson: but yes, it ran :)
<thumper> oh 3 failures, 10 errors
<mwhudson> thumper: almost all uifactory failures
<mwhudson> i think i have a branch that fixes that actually...
<mwhudson> hm, no
<mwhudson> i have a branch that was created to fix that :)
<thumper> heh
<thumper> mwhudson: add it to the todo list :)
<mwhudson> https://bugs.edge.launchpad.net/bugs/405052
<mup> Bug #405052: bzr 1.18dev breaks our uifactory <Launchpad Bazaar Integration:In Progress by mwhudson> <https://launchpad.net/bugs/405052>
<mwhudson> filed a while ago
<mwhudson> i'm sure i fixed this, must have lost the changes somehow
<thumper> :(
<mwhudson> it's minor
<thumper> off to get the car now
<thumper> bbs
 * thumper is no $806 poorer
<thumper> s/no/now
<thumper> d'oh
<spm> thumper: I'd be more worried about the $806 vs the no/now
<thumper> spm: well, the car needed brake pads and discs
<spm> what do you need brakes for? don't NZer's just run into the nearest sheep to stop?
<lifeless> no way, waste of a good sheep
<thumper> spm: it is the lack of sheep at the town lights that cause problems
<spm> you'd think they'd do something about that. what do you pay your taxes for eh!
<thumper> spm: to pay for the politicians Wellington houses apparently
<spm> ah. some things dno't ever change....
<thumper> spm: how goes disk space on codehosting?
<thumper> spm: also, what is the RT for more disk space?
<mars> wgrant, was your friend using Safari?  Be sure to note that in the bug report.  Safari doesn't show the icons, IIRC.
 * mars goes off to sleep for real now
<thumper> mars: I have noticed that sometimes they don't show at all
<thumper> mars: on FF2.0 on Windows
<wgrant> mars: Firefox 3.0. It's intended that they don't show up unless hovered.
<wgrant> That they don't show up even if hovered over is another issue.
<thumper> wgrant: even on hover they weren't showing
<lifeless> mystery meat sucks
<wgrant> lazr.restful seems to think that my zope.schema.URI is a hosted file :(
<thumper> lifeless: there was an Indian resturant near where I lived once in London that had "meat chilli chilli" as a menu item :)
<thumper> I was never game to ask exactly what the meat was
<thumper> it was a mystery :)
<lifeless> :)
<lifeless> http://en.wikipedia.org/wiki/Mystery_meat_navigation is what I meant though
<thumper> lifeless: found the community review tag bug
<lifeless> cool
<thumper> 1 line fix, 50 lines of tests to write
<mwhudson> concurrency is hard :(
<mwhudson> !!
<mwhudson> emacs just died on me?
<mwhudson> ah no
<mwhudson> just hiding
<thumper> I'm done for now
<thumper> will finish off these tests later
<thumper> night
<mwhudson> night
 * mwhudson very nearly finished too
<mwhudson> thumper: what was the problem for community reviews?
<thumper> branch.reviewer vs. branch.code_reviewer
<thumper> reviewer can be None
<thumper> code_reviewer == branch.reviewer or branch.owner
<thumper> I'm changing the view logic and renaming a method and adding tests :)
<mwhudson> argh
<noodles775> Morning ppl
<jml> morning
<noodles775> morning jml
<spm> hey jml, noodles775; and g'night
<noodles775> :)
<jml> spm, g'night
<mrevell> Morning!
<allenap> mrevell: Morning :)
<mrevell> hey allenap
<allenap> Does anyone know why PQM is throwing out my branches? It seems to be claiming we're in testfix mode for lp/devel, but the waterfall is green a long way back.
<danilos> allenap: we get into testfix even if db-devel is red
<danilos> allenap: it's "stop the line" approach
<allenap> danilos: Ah ha, thank you.
<BjornT> allenap: we should get out of testfix mode soon, though. it depends on when the script that turn it off sees the testfix commit.
<allenap> BjornT: Fantastic :)
<jml> hey
<jml> liwii sent an email to the launchpad-dev mailing list
 * wgrant hasn't seen it come through.
<jml> and it doesn't seem to be in the archive, nor in the moderation queue
<jml> wgrant, sorry I got distracted for a while
<wgrant> jml: Not even in the moderation queue? Huh.
<mrevell> jtv: ping
<jtv> mrevell: pong
<jtv> mrevell: saw your message about the getting-started guide, though haven't looked at the guide yet
<jtv> mrevell: just what we need though, so definitely let's set a time.
<jtv> (that is what this is about, isn't it?)
<mrevell> jtv: No worries. This is on a slightly different subject. Should the help wiki present bzr imports and exports as the default method?
<jml> but I don't know how the moderation queue works.
<jml> do we have an admin around who can take a look?
<deryck> Hi, all.
<jtv> mrevell: for exports I'd present both options side by side depending on use case (and it should be clear that it's for maintainers, reviewers etc., not for end-users)
<jtv> mrevell: for imports, I would like bzr to be seen as the default source.  Also makes it less inviting to set up proprietary projects for translation without contacting us.
<jml> deryck, g'day
<jtv> hi deryck, hi jml
<mrevell> jtv: Great. I'm uncertain what you mean by " makes it less inviting to set up proprietary projects for translation without contacting us."
<mrevell> yo dezza
<jml> jtv, hi
 * jtv feels an O(nÂ²) process
<danilos> jtv, henninge_: let's have a call
<mrevell> danilos: hands off, I'm talking to jtv
<danilos> mrevell: no way, it's in the calendar ;)
<mrevell> heh
<jtv> mrevell: with bzr imports, you share your source first and the translation imports are then easy.  With manual uploads, it can seem to make sense not to share source.
<jtv> danilos: call
<jtv> So flattering to have people fight about you...
<mrevell> jtv: I see, although of course people could just have a bzr branch with only their templates. But I get the point. Cool, will update das wiki now.
<mrevell> henninge: die, der or das for wiki?
<jtv> mrevell: cool, thanks.
<henninge> mrevell: das
<henninge> danilos: call
<mrevell> lucky guess on my part, thanks henninge :)
<danilos> das wiki ist mein wiki, henning get online on skype
<henninge> danilos: I am
<danilos> henninge: just seen you, calling
<jtv> "ring"
<jtv> "ring"
<stub> Now I have Abba in my head. Thanks :-P
<danilos> jtv, henninge: https://lpstats.canonical.com/graphs/TranslationsCommitToBranches/
<jml> al-maisan, who has launchpad.Edit permissions on a package set?
<jml> al-maisan, I can't find any security adapter for it
<al-maisan> jml: it should be "techboard"..
<al-maisan> jml: just a minute ..
<jml> al-maisan, techboard is the person with IPackagesetSet.new permissions.
 * wgrant hopes there is never a need to access groups of package sets.
<jml> BjornT, do you know who has launchpad.Edit permissions to an object in the absence of a security adapter?
<jml> my reading says no one has permissions
<mrevell> bigjools: Nice change on the Packaging/PPA help page, thanks.
<bigjools> mrevell: yeah, it kinda helps to know how to upload :)
<BjornT> jml: right. if there is no adapter, no one should have permission. although we have some general adapters, for example for IHasOwner, which might get used.
<jml> BjornT, hmm, thanks. Maybe that was being used in this case...
<jml> BjornT, are there any tools to find out which security adapter is being used?
<jml> currently I just use grep
<BjornT> jml: queryAdapter(ob, IAuthorization, 'launchpad.Edit')
<jml> BjornT, of course, thanks.
<kfogel> BjornT: does bugs.l.n offer full text search via the APIs?  If not, are there plans to make it do so?  (Someone from the Emacs project is asking.)
<kfogel> leonardr: ^^  (do you know answer to above?)
 * kfogel goes to look
<leonardr> kfogel: i don't know the answer
<intellectronica> kfogel: sort of. we offer the same search you get in the advanced bugs search
<BjornT> kfogel: yes, it does, through the searchTasks() method
<BjornT> kfogel: as intellectronica said, it's the same as the advanced search in the ui.
<kfogel> leonardr, intellectronica, BjornT: thanks.  Yes, looking at advanced search, it seems to meet the description "full text search".  I mean, it searches on the body as well as the description and summary of the bug, right?
<intellectronica> anyone knows what might be the problem with PQM? my lazr-js branch is sitting there for quite some time with no progress
<BjornT> kfogel: yes
<intellectronica> nm. it just got merged :)
<kfogel> BjornT: searchTasks() is not in launchpadlib, it is part of the ReST API, right?
<intellectronica> kfogel: that's true for everything in the API. launchpadlib is just an automatically generated wrapper
<intellectronica> kfogel: see https://launchpad.net/+apidoc/ for the complete reference
<kfogel> intellectronica: sure, but he could be saying a method name in launchpadlib
<intellectronica> kfogel: the point is, the method will be there when you use launchpadlib, but you won't find it grepping the source of launchpadlib
<kfogel> intellectronica: oh, it dynamically generates the method names by querying the site?
<kfogel> at run time?
<intellectronica> kfogel: exactly
<kfogel> intellectronica: inspired.  I didn't know that.  Thank you.
<intellectronica> kfogel: plus or minus a few modifications to make the interface a bit nicer
<kfogel> leonardr: we generate HTML id="foo" for each section name in the API docs; if we generated title="foo" attribute as well, then hovering over a given section would show the name, and one could write deep link URLs without having to View Source first.  For example, view source around https://launchpad.net/+apidoc/#bug_target-searchTasks to see what I mean.
<intellectronica> kfogel: that should be quite easy to modify. the documentation is transformed into html from wadl, which is an xml interface description language
<leonardr> kfogel: yeah, file a bug for it or jfdi by modifying the xslt file
<intellectronica> that's done using xslt, which is heaps of fun, if you haven't tried it yet :)
<kfogel> leonardr: just wanted to make sure you thought it was a good idea; I'll have a look
<mwhudson> jml: did you get my mail?
<jml> mwhudson, yes, I did.
<jml> mwhudson, I guess I should actually read it in detail.
<mwhudson> jml: i would appreciate it
<mwhudson> jml: no hurry though, i should be asleep :)
<jml> mwhudson, sure thing.
<jml> mwhudson, I've been puzzling through security & soyuz stuff
<mwhudson> jml: so concurrency in the puller should be a nice relaxing change for your brain
<mwhudson> jml: or something
<jml> mwhudson, yeah, for sure.
<bigjools> soyuz gets such a bad rap ...
<jml> bigjools, there's a fair bit of scope for cleaning it up
<bigjools> yeah, it's the oldest code in LP
<jml> and the problem domain itself is very complex
<jml> which compounds the problem
<bigjools> I personally never thought it was that complex, there's just a lot to know
<jml> bigjools, that's the sense I meant, I guess :)
<jml> bigjools, so, while I have you distracted...
<bigjools> I knew you'd say that :)
<jml> bigjools, why doesn't nascentupload check pocket permissions?
<bigjools> because they don't exist? :)
<jml> bigjools, what's IDistroSeries.canUploadToPocket all about then?
<jml> oh, it's in uploadpolicy
<bigjools> oh that's a state check
<bigjools> right
<bigjools> it's not a permission as such
<jml> what's the guiding principle for a check being in uploadpolicy rather than nascentupload?
<bigjools> I guess that makes your job interesting
<bigjools> hysterical raisins?
<bigjools> honestly, no idea, it doesn't seem that sane does it
<jml> bigjools, that's fine. I'm happy to clean things up as I go along.
<jml> just as long as I'm justified in doing so :)
 * bigjools owes jml a lot of whisky
<jml> it's also nice doing this in the same room as the ubuntu foundations team :)
<bigjools> ha :)
 * bigjools waves to cjwatson
<jml> bigjools, btw, this is what the verify_upload function looks like now: http://paste.ubuntu.com/247942/
<jml> it's still missing component checking & pocket checking
<jml> actually, the tests might be a little more interesting: http://paste.ubuntu.com/247943/
<bigjools> jml: is that a new file?
<jml> bigjools, yeah. the new file is at archiveuploader/permission.py -- http://paste.ubuntu.com/247946/ for the full thing
<bigjools> it seems to duplicate checks done elsewhere
<jml> bigjools, I'm happy to move it, but I figured that was as good a starting place as any
<jml> bigjools, it's extracting out verify_acl
<bigjools> right - so we can remove some other tests
<bigjools> I doubt they're as efficient as these
<jml> bigjools, yeah, that's probably a good idea.
<bigjools> since they're probably trying to upload a package
<jml> finding tests to delete is a lot harder than finding code to delete though.
<jml> heh heh.
<bigjools> lib/lp/soyuz/doc/nascentupload*
<bigjools> and
<bigjools> lib/lp/archiveuploader/tests/*
<jml> and I've been keeping the existing tests as guarantees that my refactoring maintains the existing behaviour.
<bigjools> good call
 * bigjools is looking forward to seeing your changes
<jml> bigjools, do the interim versions look ok so far?
<bigjools> jml: I think so, I am still looking
<jml> cool.
<jml> well, I'm off to lunch now, but I'll be happy to read your comments in the backlog :)
<jml> ciao
<bigjools> jml: enjoy, I am eating now too
<bigjools> hmmm the new AJAX-y "request another review" for an MP doesn't go quite right
<kfogel> intellectronica, leonardr: http://lists.gnu.org/archive/html/emacs-devel/2009-08/msg00219.html
<leonardr> kfogel, col
<intellectronica> kfogel: so what, emacs is moving or thinking of moving to LP?
<kfogel> intellectronica: Emacs is switching to Bazaar (but trunk not hosted on LP).  I'm proposing that it switch to launchpad bugs, because right now only launchpad bugs and debbugs meet Emacs' criteria for a bug tracker: that it be free software, and that it be entirely operable by email.
<intellectronica> very nice
<kfogel> leonardr: find . -name "*.xslt" -print  ==> no results in Launchpad tree, same with "*.wadl".  Give me hint as to what to search for?
<kfogel> leonardr: ./lib/canonical/launchpad/apidoc/wadl-development.xml
<kfogel> ?
<leonardr> kfogel: no, that's a test file
<leonardr> the xslt file is in launchpadlib
<leonardr> now that launchpad is open source we could move it back into launchpad
<leonardr> but some people wanted to modify it
<kfogel> leonardr: cd ~/src/launchpadlib && find . -name "*xslt*"  ==> no results
<NCommander> kfogel, debbugs could also do it
<NCommander> oh, doh
<kfogel> NCommander: :-)
<NCommander> kfogel, I thought bugzilla was adding support for that htough
<kfogel> leonardr: ah, found it I think
<NCommander> Does anyone know if the sync packages script can copy into a component
<kfogel> leonardr: wadl-to-refhtml.xsl, right?
<leonardr> y
<kfogel> leonardr: where is the process that drives the XSL conversion?  I don't see any other file (like a Makefile or setup.py) that refers to wadl-to-refhtml.xsl.  I.e., how will I test my changes?
<leonardr> kfogel: i believe there is something in the launchpad makefile that does the conversion and puts it in apidoc/
<kfogel> leonardr: oh, launchpadlib is a dependency of launchpad?  I didn't realize that.  Thanks.
<kfogel> (I thought we drove the ReST APIs directly... maybe we do, and the dep is only for the apidocs.  dunno)
<leonardr> kfogel: i'll lay it down for you
<kfogel> leonardr: lay it on the line for me, brother
<leonardr> there are five pieces of software: lazr.restful, launchpad, wadllib, lazr.restfulclient, and launchpadlib
<leonardr> data flows out of the launchpad database from left to right
<leonardr> lazr.restful and launchpad are on the server. wadllib, lazr.restfulclient, and launchpadlib are on the client
<leonardr> lazr.restful has self-contained tests that use fake web services
<leonardr> wadllib has self-contained tests that use an old static copy of launchpad's wadl
<leonardr> lazr.restfulclient has tests where it acts as a client for lazr.restful's fake web service
<leonardr> launchpadlib contains launchpad-specific code and must be tested against the actual launchpad web service
<leonardr> we integrated launchpadlib tests into launchpad because changes to launchpad can break launchpadlib
<leonardr> but all the other pieces of software -- lazr.restful, wadllib, and lazr.restfulclient -- are tested standalone
<kfogel> leonardr: *nod*  thank you.  (please take a look at http://paste.ubuntu.com/247977/ to see if I'm on the right track, when you get a chance)
<leonardr> kfogel: that looks good to me, but ask sinzui, he's the xslt expert
 * sinzui looks
<kfogel> sinzui: thanks.  Not tested yet (i.e., I haven't generated docs from it yet)
<sinzui> kfogel: what is the value of having a title in every heading?
<kfogel> sinzui: right now, if you're reading the API docs online (say, visit https://launchpad.net/+apidoc/#bug_target-searchTasks), you cannot know the name of the section you're in without doing View Source.  With this change, hovering the mouse pointer over a section would show its name.  That way people could more easily refer to specific sections within the documentation.
<kfogel> s/refer/refer other people to/
<kfogel> heh, I mean s/refer to/refer other people to/
<kfogel> In other words, I had to do View Source to come up with the "#bug_target-searchTasks" part of that URL.  After this change, I could just hover and grab it.
<sinzui> okay. This change look good.
<kfogel> sinzui: do you know off the top of your had what command I should run to re-generate the HTML to test this?  (If not, no worries, I'll poke around.)
<sinzui> make clean; make build
<sinzui> will do it
<kfogel> heh
<kfogel> thnks
<kfogel> thanks
<flacoste> morning everyone
<kfogel> flacoste: morning
<mrevell> hey flacoste, good morning. I'll let you settle in but I have a CP request when you have time to talk about it :)
<kfogel> sinzui: http://paste.ubuntu.com/247987/    Not sure how to interpret that error message -- there are 9 files named "_pythonpath.py" in the tree, and they are all under version control or a symlink to a file under version control, so just removing the file would be weird...
<sinzui> I wonder if this relates the the make distclean that was requested
<flacoste> kfogel: we want to get rid of _pythonpath, buildout allows it to a certain extent, but it's not complete yet
<sinzui> kfogel: I'm not sure what is wrong
<kfogel> sinzui: thanks
<kfogel> flacoste: ok.  In the meantime, do you know what I should do to make my tree buildable?
<kfogel> sinzui: (yeah, I was wondering the same thing re distclean)
<flacoste> kfogel: i don't know what is the problem, and my tree is two weeks old here so...
<kfogel> flacoste: np
<gary_poster> kfogel: top level _pythonpath
<kfogel> gary_poster: remove it?  (it's versioned...)
<gary_poster> it is?  :-/  hm, checking further
<gary_poster> uh, it shouldn't be.  checking further.
<gary_poster> kfogel: it is not versioned.  it is in .bzrignore.  buildout-templates/_pythonpath.py.in is versioned
<kfogel> gary_poster: sigh.  So when you do 'bzr status -v' on an unversioned-but-ignored file, it does not say "unknown file", it just returns silently.
<salgado> JamalFanaian, ping. can you set a commit message on https://code.edge.launchpad.net/~jamalfanaian/launchpad/bug-337582/+merge/9524 so that I can use it when landing your branch?
<kfogel> gary_poster: Unfortunately, this is the same behavior as for a versioned-but-unmodified file!  So it is difficult in bzr to ask the question "Is this file under version control?"
<kfogel> gary_poster: (I just learned this now by testing.)
<gary_poster> kfogel: looks like it.  I was wondering about how to ask that question too.  abentley ^^^ ?
<abentley> kfogel: ls -V
<gary_poster> kfogel: I hope to get the answer to this bzr question also.  oh, thank you abentley.  but kfogel, IRT the problem you started with, is the change I described sufficient? it should be.  and not removing _pythonpath.py in make clean is an oversight.  If you have a branch you are preparing for merge, it would be great to have you add that.  otherwise let me know and I will.
<kfogel> gary_poster: my branch is of launchpadlib; it would be great if you could take care of the 'make clean' issue with _pythonpath.py.
<kfogel> gary_poster: rebuilding now, after 'rm _pythonpath.py' in the top level
<kfogel> abentley: thanks
<gary_poster> kfogel: cool, ok
<kfogel> gary_poster: sigh, 'make run' failing, seems I might be out-of-date with some deps: http://paste.ubuntu.com/248001/
<gary_poster> kfogel: no, you just need to run make schema
<kfogel> gary_poster: and launchpad-dependencies failing to update in system manager, hmm: http://paste.ubuntu.com/248002/
<kfogel> gary_poster: I tried that
<gary_poster> kfogel: uh.
<kfogel> gary_poster: http://paste.ubuntu.com/248003/
<kfogel> gary_poster: though 'make schema' seems to have failed only with perm denied errors, let me see...
<gary_poster> kfogel: for your apt problems, I'm not your man.  I would try uninstalling and reinstalling, or seeing if there's some force-reinstall
<kfogel> http://paste.ubuntu.com/248004/
<kfogel> gary_poster: hmmm, manually doing the rm -rf seems to have fixed things, fingers crossed
<gary_poster> kfogel: I've encountered those Permission denied thing too.  Don't know why they appear (seemingly bzr-related, obviously).  didn't investigate. but just removed them by hand as you did
<kfogel> gary_poster: :-(   I'll bet the same is true of all of us.
<kfogel> gary_poster: okay, 'make schema' completed apparently successfully.  trying 'make run' now
<kfogel> Ha.  Ha hah hah HAH.  HAAAAAAAAAAAAAAh.  Of course, I can't test my change if my FIREFOX IS GOING TO FREEZE UP ON ME, NOW CAN I???
<gary_poster> kfogel: lol, hang in there
<kfogel> gary_poster: :-)
<kfogel> gary_poster: just need to rant.  btw, killed ffox, now it won't start up at all
<gary_poster> urg :-(
<mars> check the network meter
<mars> sometimes FF will spin in the background before rendering if there is a bunch of JS or Flash to load onto the pages
<bigjools> FF is rapidly losing my friendship
<mars> bigjools, noodles775 uses Chrome....
<kfogel> gary_poster: (all good now)
<gary_poster> kfogel: \o/ :-)
<bigjools> can I get noscript and adblock for it?
<bigjools> and firebug
<kfogel> sinzui: so my change appears to have had no effect.  Here's what I did: manually took my patch and 'patch'ed it into sourcecode/launchpadlib in the new tree I created with rocketfuel-branch, built that tree, make clean && make build && make schema && make run.
<sinzui> I puzzled then
<sinzui> api/index.html had no title?
<sinzui> kfogel: does this work when called by itself?
<sinzui> bin/apiindex lib/canonical/launchpad/apidoc/wadl-development.xml
<gary_poster> kfogel: we are using launchpadlib from egg AFAIK (on call)
<kfogel> gary_poster: thank you
<kfogel> gary_poster: but do we re-unpack that egg every time?  I mean, I patched the source tree on disk.
<kfogel> gary_poster: and my "title=" mods are still in it...
<kfogel> sinzui: yes, let me look at teh output
<gary_poster> kfogel: sourcecode is not consulted.  I'm checking if we still update launchpadib in sourcecode
<kfogel> gary_poster: hunh.  shouldn't we get rid of the one in sourcecode then?
<gary_poster> kfogel: +1
<kfogel> sinzui: ran that command -- the output does not contain "title=" anywhere
 * sinzui look at the xslt again
<bac> kfogel: do you want to join the reviewers meeting at the top of the hour to discuss your fine work on the wiki update?
<sinzui> kfogel: your changes must be fine, I think something else is going the generation here. Break the XSLT, do something to verify that we get errors if this file is insane
<kfogel> bac: might be a good idea
<bac> kfogel: if so, join #launchpad-meeting in three minutes
<gary_poster> kfogel: looks like it has been removed from utilities/sourcedeps.conf so it must be still in the rsynced directory,  I'll send something to RT...
<kfogel> gary_poster: rsync'd from where?  devpad?  We shouldn't have *any* of those left...
<gary_poster> kfogel: no?  I'm a bit behind the times on what we do for sourcecode I guess.  I thought we were still doing that.  So, maybe you and I just have old sourcecodes?
<kfogel> gary_poster: apparently.
<kfogel> gary_poster: nothing can come from devpad because devpad is not publicly accessible :-).
<kfogel> gary_poster: we open-sourced.  Here, let me find you the press release... :-)
 * kfogel kidding
<kfogel> bac: there
<gary_poster> kfogel: :-P :-) so, if this is only controlled by sourcedeps, then we are ok already afaik
<rockstar> abentley, #launchpad-meeting
<abentley> me
<abentley> Doh.
<gary_poster> :-) you!
<noodles775> beuno: I'm looking at the code that creates the breadcrumbs (c.lp.browser.launchpad.Hierarchy) - and the presentation is actually in the code rather than a separate template. Are you going to be working on that with salgado? I'd like to add something like render_ul(), but don't want to waste time if it'll be done elsewhere.
<JamalFanaian> salgado: hey, sorry i was away.. what do you mean exactly?
<kfogel> sinzui: due to a typo, I removed bin/apiindex.  it's not versioned, so I can't revert.  'make build' and 'make run' don't regenerate it.  Where does it come from, and how do I get mine back?
<salgado> JamalFanaian, I'm going to request a merge of your branch into our trunk branch, and to do that I need a commit message -- something that describes (in one or two sentences) what you've done in your branch
<kfogel> sinzui: ah, lib/lp/scripts/utilities/apiindex.py
<sinzui> kfogel: the first make build
<sinzui> kfogel: the scripts to no regenerate if bin is there
<JamalFanaian> salgado: would you like me to just post it here?
<salgado> JamalFanaian, yeah, here would be fine, but you could also enter it on the merge-proposal page
<JamalFanaian> salgado: ok
<JamalFanaian> salgado: Renamed the sort option in the branch list page from "by registrant" to "by owner".
<JamalFanaian> is that good?
<salgado> JamalFanaian, it is great, thanks
<beuno> noodles775, I will
<JamalFanaian> salgado: thank you :)
<beuno> noodles775, we can tackle this with salgado then
<noodles775> beuno: yeah, it wouldn't make sense to do the css until the html is sorted (a simple ul :) ).
<beuno> noodles775, agreed. Thanks.
<beuno> noodles775, replying
<noodles775> thanks beuno
<joshuahoover> sorry if this is off topic, but are there plans to ever provide launchpad as a package vs. what we have today?
<beuno> joshuahoover, no, we want people to contribute, not set up instances  :)
<joshuahoover> beuno: got ya...i was asked by a former co-worker who is interested in launchpad for a project he's working on for the us army
<kfogel> https://dev.launchpad.net/PatchSubmission   review on new wording welcome
<kfogel> bac: ^^
 * bac looks
<beuno> noodles775, done
<beuno> approved
<noodles775> Cheers :)
<bac> kfogel: i think rocketfuel-setup makes the .bazaar/locations.conf work so that you don't have to name the destination for pushes to launchpad.  so step 3 would just be 'bzr push'
<jml> anyone here know xsl?
<kfogel> jml: heh, working on some xsl right now.  doesn't mean I know it, though.  What do you need?
<jml> kfogel, I just filed bug 409360 and I was going to try to fix it
<mup> Bug #409360: API docs need a table of contents <Launchpad Foundations:New> <https://launchpad.net/bugs/409360>
<kfogel> jml: we're working in the same file, it seems.
<jml> kfogel, wadl-to-refhtml.xsl?
<kfogel> jml: so, I don't know how to generate TOC in xsl.  I may be able to help with regenerating the HTML, once you get to that point.
<kfogel> jml: right, that file
<jml> kfogel, hmm. me neither.
<jml> google to the rescue
<kfogel> sinzui: so, I replaced the contents of wadl-to-refhtml.xsl with the word "fish" and re-ran bin/apiindex, and it still succeeded!  Obviously, sourcecode/launchpadlib/src/launchpadlib/wadl-to-refhtml.xsl is not the file apiindex is actually using...
<kfogel> sinzui: but, sys.path in apiindex does contain "/home/kfogel/private/work/canonical/lp/lp-sourcedeps/eggs/launchpadlib-1.0.2-py2.4.egg"
<kfogel> sinzui: is it getting launchpadlib's .xsl file directly from inside the egg?
<sinzui> ah
<sinzui> kfogel: you need to hack on launchpadlib, not launchpad.
<jml> kfogel, almost certainly it is.
<kfogel> sinzui: I know.  That's what I'm trying to do.
<kfogel> sinzui: my patch is to launchpadlib; the question is, how do I get the patch into a launchpadlib that launchpad is paying attention to?
<kfogel> sourcecode/launchpadlib is apparently not the one!
<kfogel> sinzui: it seems I need to regenerate the egg, yah?
<beuno> sinzui, noodles775, do you guys want to have a call about the remaining navigation questions?
<noodles775> Sure.
<sinzui> kfogel: you are hacking on the launchpad code, you need to be hacking on the launchpadlib code it seems. https://code.edge.launchpad.net/launchpadlib
<kfogel> sinzui: please read above
<sinzui> kfogel: a different checkout
<kfogel> sinzui:  I'm already there.  Long done :-).  I'm hacking on launchpadlib.
<kfogel> sinzui:  the question here is "How do get Launchpad's bin/apiindex to use the changed launchpadlib?"
<sinzui> I think the next step is to place your new egg into develop-eggs/ gary_poster will no for certain. I have never done egg development
<sinzui> beuno: I am preparing for a meeting. I expect to be available in 1 hour
<gary_poster> kfogel: yeah, what sinzui says is probably the easiest.  I think I put it in the docs, checking
<beuno> noodles775, are you around in an hour?
<leonardr> gary_poster: aren't we still far away from having launchpad use an up-to-date (post-lazr.restfulclient) launchpadlib?
<leonardr> the best thing to do might be to move that file back into launchpad now that launchpad is open source
<noodles775> beuno: for 15mins yes (I've got to run in 1hr 15mins)
<beuno> sinzui, noodles775, ok, ping me as soon as you guys are available
<gary_poster> leonardr: oh is that what this is?  yes, it's the next big thing to which I plan to return, having almost gotten one off my plate, but I have not proven to be a speed demon for this particular task :-/
<noodles775> beuno: I'm available any time so, I guess sinzui can ping us when he's available :)
<gary_poster> kfogel: fwiw, the way to do a develop egg is described in doc/buldout.txt, albeit in passing.  grep for "develop another dependency"
<kfogel> gary_poster: thanks
<beuno> noodles775, so am I
<kfogel> gary_poster: will move that to dev wiki as appropriate
<gary_poster> kfogel: but what leonardr brings up is a legitimate concern
<kfogel> gary_poster: leonardr is just talking about the wadl file, right, not all of launchpadlib?
<gary_poster> kfogel: I'm not clear on dev wiki vs. doc directory.  kiko seemed to agree with my reviewers that this should go in the doc directory, although we are supposed to clean up that directory.
<kfogel> gary_poster: btw, is most of buildout.txt up-to-date?  I.e., I could transfer to dev wiki with impunity?
<kfogel> gary_poster: I'm not sure I see any reason to have anything in the doc/ directory other than a single README saying "Please see https://dev.launchpad.net/."
<gary_poster> kfogel: yes, but the doc directory version is supposed to be the source, according to my last discussions.
<kfogel> gary_poster: there can only be one master, but it can be anywhere we want.  The only complication is when we auto-generate some other stuff from the source -- is that happening here?
<gary_poster> kfogel: wiki vs. doc directory: that is a discussion you need to have on the list.  It was already started, with kiko somewhat agreeing that some stuff belongs in the doc directory.  the policy is not clear, though I believe he gave a general direction dor a distinction of what goes where.  jml, for instance, may have a strong opinion about docs going in the doc directory
<kfogel> gary_poster: I understand that there is a larger discussion here, & will raise it.  In general, I don't see any good reason to split our documentation between in-tree and in-wiki.  That way lies madness not only for contributors but for (*cough*) Canonical employees like me.
<kfogel> And you :-).
<gary_poster> kfogel: :-) (I don't have a horse in the race really, but a (new?) decision ITR should at least be aware of the history and the agreements.)
<gary_poster> I mean past disagreements
<gary_poster> the only horse in the race I have is that I want to be able to use ReST, not moin, wherever I have to write docs :-)
<jml> hello what.
<gary_poster> what's on first base
<jml> I think there should be one place
<kfogel> jml++
<jml> and that should be in the tree
<jml> and it should be in rest
<kfogel> jml: sigh
<kfogel> jml: why not the dev wiki?
<jml> (in that order of conviction)
<kfogel> jml: that is, why is some of our documentation in the wiki and other docs in tree, making every dev search two places?
<jml> kfogel, because I think I hate textarea more than I hate waiting for our test suite
<jml> kfogel, but it's a close call.
<kfogel> jml: get the ViewSourceWith plugin.  This is an easily solved problem.
<kfogel> I never edit in my browsre.
<abentley> beuno: Could you please review https://code.edge.launchpad.net/~abentley/launchpad/reassign/+merge/9463 ?
<kfogel> jml: that is one of *many* plugins that solve this problem.
<beuno> abentley, absolutely. On my queue
<jml> kfogel, there's also greppability and ease of making changes to multiple docs.
<abentley> beuno: Thanks.
<kfogel> jml: er, well, yes, but that kind of argues against the wiki entirely.
 * jtv goes awl
<kfogel> jml: we don't have a perfect solution.  Having better user experience (the wiki) means a harder writer experience.  That may just be life...
<jml> kfogel, hmm, I think that might be true
<kfogel> jml: wiki search should answer the greppability question (and if it's broken, we should fix it, obviously)
<jml> kfogel, also, someone should pull their finger out and allow 'bzr branch http://dev.launchpad.net/'
<kfogel> jml: I don't have a good answer for the "similar changes to multiple docs" problem, but we deal with the wiki every day and we manage.
<abentley> kfogel: Until our wiki plans come to fruition...
<kfogel> jml: that would be *fantastic*
<kfogel> jml: but: Let not the perfect be the enemy of the cliche.  Or something like that.
<jml> kfogel, to me, the most important values I have are 'one place' and 'navigable'
<kfogel> jml: what about "easy for newcomers and lightweight visitors"?
<jml> kfogel, and I guess putting everything on the wiki gives us that.
<kfogel> jml: oh, by navigable, you meant by readers?
<kfogel> (I thought you meant for editing.  It's funny how each system supplies one of those and not the other.)
<jml> kfogel, yeah, by people trying to figure things out :)
<jml> kfogel, I think in-tree would provide navigation for readers
<kfogel> jml: those are my two priorities too.  I think the wiki is the answer then, despite the fact that it is harder to edit.
<jml> we'd just have to change the way we store docs in tree.
<kfogel> jml: uh... hyperlinks?
<jml> kfogel, documentation build step
<kfogel> jml: oh, from .txt to .html?  We could do that.
<gary_poster> Sphinx
<gary_poster> my pref
<jml> kfogel, that's what Python does, and it's pretty nice
<jml> kfogel, we can also make sure that the docs for trunk are always available on the web
<kfogel> jml: ideal would be if launchpad-top/docs/  ==  dev.launchpad.net/
<jml> kfogel, bzr does this already
<kfogel> bzr branch and edit away
<kfogel> jml: all these are things for the future.  in the meantime, our docs are split across two places; I'd like to get docs/* into the wiki.
<jml> e.g. http://doc.bazaar-vcs.org/bzr.dev/en/developer-guide/HACKING.html
<jml> kfogel, I think that makes sense.
<kfogel> jml: being editable in the wiki is a huge advantage too.  Really huge, actually.  If fixing a doc bug means getting a branch merged, nothing will ever get fixed by contributors.
<jml> kfogel, yeah. that makes me sad though.
<gary_poster> m etoo
<kfogel> jml: another process problem that could use solving ;-)
<kfogel> jml: but, if the wiki were backed by a bzr branch with auto-commit-after-identifying-through-OpenID, then we'd be golden.
<jml> someone's already done this for hg.
<gary_poster> jml: with Sphinx?
<jml> which is embarrassing because we've been talking about it for bzr for years
<jml> gary_poster, no. just a custom wiki that is a view on an hg branch.
<jml> gary_poster, some sort of wiki syntax.
<kfogel> jml: we get trapped by our own shadow too often
<gary_poster> jml: gotcha
<kfogel> gary_poster: how to build an egg?  (I could find this on the Net, but if it's two easy steps and you know them, please share)
<gary_poster> kfogel: mm, context?  you may not need to
<kfogel> gary_poster: I've modified my buildout.cfg to point to my new 'develop' location, I just need to put a new eg there
<kfogel> gary_poster: from the beginning, for clarity:
<gary_poster> kfogel: ah, you don't need egg
<kfogel> ah
 * kfogel listens
<gary_poster> kfogel: just symlink your branch into the lp tree using the name you put in the develop section
<gary_poster> then rerun bin/buildout
<gary_poster> you may have to change the version
<gary_poster> in versions.cfg
<gary_poster> note that leonardr's warning was about the package, not about the output
<gary_poster> so if this is from trunk, your change will break
<gary_poster> releasing a 0.1.1 with your change (branching from the 0.1 source, if we were good package maintainers) would be the right thing to do
<beuno> intellectronica, when the project doesn't have milestone, it lets you target, and shows you an empty overlay
<kfogel> gary_poster: reading your instructions makes me realize how badly I need breakfast.  Back soon.
<gary_poster> or doing what leonard said, which is to temporarily give up on eggs because I Haven't finished eggifying everything, which is what is blocking this process
<gary_poster> ack
<beuno> deryck, hey dude
<deryck> beuno, hey man
<beuno> deryck, got any news for me on your awesome inline editing?
<deryck> beuno, very, very close. :)
<beuno> deryck, is it up for review yet?
<deryck> beuno, no, not yet.  I need the rest of today to finish.  So in review late today or early tomorrow.
<beuno> deryck, perfect
<deryck> beuno, I had CHR and OCR two of the last three work days, so today really is the first I've had to just sit and work on it since we last talked.
<beuno> deryck, no worries. I'm just anxious  :)
<deryck> beuno, yeah, me too actually.  I like this feature a lot.
<intellectronica> beuno: oh, right. that doesn't make much sense. fixing
<intellectronica> beuno: fixed and pushed
<intellectronica> allenap: https://pastebin.canonical.com/20793/ is the incremental
<allenap> intellectronica: Cool.
<beuno> intellectronica, super, thanks. Just submitted the review
<intellectronica> beuno: i don't know why you don't see the spinner. i always do. i remember you've had this before, though
<beuno> intellectronica, I'm happy to trust you, I just don't. As well as I don't see any feedback after clicking until the green flash
<beuno> the overlay is gone, and I don't see anything until it flashes
<intellectronica> that's weird, and it needs investigation. looks fine here
<intellectronica> beuno: as for the location of the overlay - it's just centered around where you clicked. we can compensate and make sure it doesn't go beyond the current page dimensions. that's a lazr fix, so i'll do that in another branch - i don't think it should block this one
<beuno> intellectronica, agreed. If you schedule that change to lazr-js, it's fine to have it this way for a little while
<intellectronica> beuno: finally, where do you want to see the milestone icon? we didn't display it before. i'm not sure it makes sense to repeat it for each milestone what do you think?
<beuno> intellectronica, on the bugtask, no?
<beuno> we don't display it now?
<intellectronica> beuno: all other comments are clear and easy to address
<beuno> cool
<intellectronica> beuno: no, we just display the milestone title
<beuno> intellectronica, I swear I saw the milestone icon. Maybe I broke it with my sprites branch
<beuno> anyway, if you can show it, super
<beuno> otherwise, it's a separate bug
<beuno> I think it will make it clearer
<intellectronica> beuno: you mean, next to the milestone title?
<beuno> intellectronica, yes
<intellectronica> i think it doesn't make much sense to repeat it for every row, but it's easy to give it a try and see
<beuno> intellectronica, well, you usually don't have a milestone per every row?
<intellectronica> beuno: why not?
<beuno> intellectronica, don't know, doesn't seem to be the common case
<beuno> you *can*
<beuno> it's just rare that things line up so much  :)
<intellectronica> right, i see what you mean. it is the less common case
<intellectronica> in general, most bugs have only one task anyway
<beuno> xzactly
<bigjools> beuno: did you have any more comments on my mockups?
<beuno> bigjools, I've had it open in a tab for days
<beuno> all I can think of is "it's confusing"
<bigjools> beuno: welcome to soyuz
<bigjools> I can take you through it on a call later if you like
<beuno> bigjools, I think I understand what each element is, it's just that the layout is confusing. Maybe we should be mocking this up on a more wireframe level?
<beuno> sinzui, noodles775, how are we coming along with that call?  postpone til tomorrow?
<bigjools> beuno: which page are you talking about?
 * sinzui almost ready
<bigjools> I made 2
<beuno> bigjools, I know I'm not being helpful, I need to sit down and dedicate a few hours to this. I just haven't been able to.
<beuno> bigjools, looking at: http://people.canonical.com/~ed/dspr_mockup2.png
<beuno> which is an improvment ovr the previous one
<bigjools> beuno: they are pretty much exactly as we discussed in our sprint, apart from the second where I took the liberty of moving stuff to the sidebar
<bigjools> ah, I did three then :)
<beuno> bigjools, I sucked at solving the problem then!
<bigjools> beuno: it was a team suck^Weffort
<bigjools> beuno: it would be best if we had a call, I can explain the nuances
<beuno> bigjools, agreed. Can we have it tomorrow, so I can clear my schedule to work on it after the call if needed?
<bigjools> beuno: +1
<noodles775> beuno, sinzui: I've replied to sinzui's post on the MP with some more screenshots, just to help visualize things if/when we call.
<sinzui> beuno: noodles775: I am ready
 * noodles775 waits for beuno's skype conference call :)
 * beuno fire sup skype
<beuno> noodles775, sinzui, http://people.canonical.com/~michaeln/menu_3-0/10-project-index.png
<noodles775> sinzui: http://people.canonical.com/~michaeln/menu_3-0/11-sprint.png
<noodles775> sinzui: http://people.canonical.com/~michaeln/menu_3-0/6-no-location-default-launchpad.png
<beuno-lunch> salgado-lunch, are you available to have a call around 20UTC today?
<bac> intellectronica: you indicated this bug was similar to others you've fixed in the past and should be easy.  https://bugs.edge.launchpad.net/malone/+bug/397457
<mup> Bug #397457: Bug privacy edit icon is not visible <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/397457>
<bac> intellectronica: any chance of it getting bumped in priority?  the inability to change bug privacy is quite annoying
<intellectronica> bac: i don't know if it's similar. why the urgency? i can definitely spend a bit of time investigating, at the very least
<bac> intellectronica: the urgency is just broken functionality for all of our fine users of webkit-based browsers make their (my) experience less than rewarding.
<intellectronica> bac: well, if it's like the other one than talking about it takes longer than fixing. i'll try to have a look later today
<bac> intellectronica: great!
<ara> intellectronica, hey tom, I am trying to break a lock and I get an error
<ara> bzr: ERROR: Unsupported protocol for url "lp-45197200://~mago-contributors/mago/mago/.bzr/branch/lock
<salgado> beuno-lunch, can we make it 21UTC?
<ara> intellectronica, any ideas?
<ara> bzr: ERROR: Unsupported protocol for url "lp-45197200:///~mago-contributors/mago/mago/.bzr/branch/lock
<intellectronica> ara: right, just replace the bogus stuff in the beginning of the url to bzr+ssh://bazaar.launchpad.net and it should work
<intellectronica> i don't know why bzr spits this weird stuff
<ara> intellectronica, ta! (btw, you're missed in the Platform sprint...)
<intellectronica> maybe rockstar or abentley know? ^^^^^
<intellectronica> ara: and i'm missing you. have fun!
<beuno-lunch> salgado, even better
<beuno-lunch> salgado, just ping me when you're ready
<beuno-lunch> BjornT, do you know why adding bug comments is so slow? (4+ seconds)
<beuno-lunch> is there a bug filed about it?
<mrevell> Night all!
<kfogel> gary-lunch: ping when you're back
<BjornT> beuno-lunch: well, i'm guessing it's because lp is slow in general. loading the front page also took 4+s seconds just now
<BjornT> beuno-lunch: but we certainly can speed it up a bit. we're currently doing at least two requests, where one should be enough
<beuno> BjornT, do you want me to file a bug about it?
<beuno> I *think* most ajax interactions aren't that slow, like, say, subscribing
<BjornT> beuno: sure. i don't think it's not easy to fix, though; it requires changes to the api infrastructure
<beuno> BjornT, gotha. Will file it for tracking purposes
<gary_poster> kfogel: pong
<kfogel> gary_poster: hey.  So I read what you wrote to me a while back (right before I ate).  I confess I understood it not :-(.  You might have to say it again with more words?  My starting point is: https://code.edge.launchpad.net/~kfogel/launchpadlib/apidoc-html-title-attrs .  I'm trying to test r53 there.
<kfogel> gary_poster: you said I don't have to build a new egg?  But then how do I make Launchpad's bin/apiindex use the new wadl file?
<gary_poster> kfogel: ok, this is complicated, no doubt, and you are also going to be sad as to where we are with this particular package and your goals.
<kfogel> gary_poster: thank you for preparing me :-)
<kfogel> gary_poster: (btw, I have a conf call in 30min, could take this up after that's over if you want)
<gary_poster> kfogel: :-) so, first (forget about eggs and such for now), we are not using launchpadlib trunk in launchpad.  We are using 0.1 (notice that you are editing something > 1.5.  We will not be able to move forward until I get launchpad to use zope eggs.  That branch has been following me around for probably months now.
<gary_poster> Everybody wants it to be done, including myself.  I hope to get it done next, like, within the next week.  I know flacoste wants this to have been done long ago.  So, you might want to consider waiting to fix whatever it is you want to fix in launchpad for a week.  Then again, you might not.
<gary_poster> sorry, I write novels
<kfogel> gary_poster: no, this is great, keep it flowing
<gary_poster> kfogel: second, there are two kinds of ways to work with new revisions.  If you are testing something tricky, sometimes you want to use a develop egg.  That technique is almost never something we actually want to commit.
 * kfogel absorbs hungrily
<gary_poster> kfogel: the other way is to make a new egg.  it can be temporary (you don't check it in) or permanent (you check it in).  This is described as thoroughly as I could in the buildout doc to which I referred you earlier
<gary_poster> kfogel: if we were in a good place with launchpadlib, this is what I would have suggested to you:
<gary_poster> (If the wind blew from a southeasterly direction:) Make an egg and put it in download-cache/dist as described in the document.  Upgrade your versions.txt in a launchpad branch and see if it works.  If it does, ideally arrange socially and technically to make a real release of launchpadlib, put the release's egg in download-cache/dist (discarding your temporary one), and put your launchpad branch up for review.
<gary_poster> (if the wind blew from a northwesterly direction, I would explain how develop eggs works.  their cleanup-after-test story is somewhat better)
<gary_poster> kfogel: so what to do *now* is a unclear.  I'd say you have three options, on the face of it.
<gary_poster> (where "now" recognizes that our launchpadlib + launchpad story is unfortunate)
<gary_poster> 1) see if I do what I want to do and get the zope eggs in by next week.  then see if you can add launchpadlib in the usual way.
<gary_poster> 2) figure out a branch of whatever we used for launchpadlib 0.1.  make a branch of that, cherrypick your change, and make a 0.1.1 release, doing thinsg the normal way
<gary_poster> 3) remove launchpadlib from the whole buildout story and add it back to the old sourcecode story.  hack away as needed.
<gary_poster> kfogel, neither 2 or 3 are easy and quick IMO.  #1 is easy but slower than 2 & 3.
<kfogel> gary_poster: (1) it is.  Thank you for explaining!
<gary_poster> (because you are waiting on me; the process itself is much easier)
<kfogel> gary_poster: what bug do I subscribe to to know when your work is done?
<gary_poster> kfogel: welcome, and sorry our current state has foiled your dastardly plans ;-)
<gary_poster> kfogel: the branch is https://code.edge.launchpad.net/~gary/launchpad/zbuildout
<gary_poster> I'm not aware of a bug for the task.  There probably should be one.  bugs, and the py 2.5/2.6 effort, are waiting on it.  here's an example of a waiting bug I happen to have lying around: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/408897
<mup> Bug #408897: Use latest zope.testing <python-upgrade> <Launchpad Foundations:New> <https://launchpad.net/bugs/408897>
<kfogel> gary_poster: want to create a bug about the actual task and then I'll make my branch/bug dependent on it?
<gary_poster> kfogel: ok.  lemme doublecheck in various places I forget about that there is not one already...
<kfogel> gary_poster: np
<maxb> Is it worth filing more "Use latest zope.foo" bugs, every time a specific need to upgrade is discovered?
<maxb> We need newer zope.proxy and zope.security for python 2.5
<jml> maxb, yes.
<beuno> kiko-fud, TL call?
<gary_poster> kfogel: bug 409480
<mup> Bug #409480: Use source distributions (via zc.buildout) for zope dependencies <Launchpad Foundations:In Progress by gary> <https://launchpad.net/bugs/409480>
<gary_poster> maxb: that bug is the one you are waiting on too
<maxb> Ah, it has a bug as well as a branch
<kfogel> gary_poster: thank you
<gary_poster> maxb: brand new fresh bug right out of the oven :-)
<james_w> thanks mthaddon
<mthaddon> er, no problem (I think) :)
<jml> :)
 * jml is off to dinner
<rockstar> sinzui, so, I think I might be missing a class assignment somewhere that makes the h2 of portlets gray.
<sinzui> rockstar: h2 just works if you are using it in a class="portlet"
<rockstar> class="yui-u portlet"
<maxb> I have a testsuite question - are the names of the tests printed when the start, or finish?
<maxb> i.e. if a test times out and is killed, is it the last test shown, or the next in sequence after that?
<sinzui> rockstar: That is correct for presentation. I just updated a bug for beuno to comment on I think we need to avoid using yui-u and portlet on the same div. I have two proposals that I need feedback on
<sinzui> rockstar: can you paste the page so that I can see the whole context?
<beuno> sinzui, point me to the bug!
<rockstar> sinzui, yea, one sec.
<sinzui> beuno: https://bugs.edge.launchpad.net/launchpad-registry/+bug/405916
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
<salgado> maxb, they're printed after the test finishes
<rockstar> sinzui, https://pastebin.canonical.com/20818/
<rockstar> The only thing I have left on this portlet is getting the h2 styled right.
<sinzui> rockstar: and the calling page? it sets main_side
<rockstar> sinzui, one sec.
<sinzui> rockstar: the yui-u is unneed in the side portlets...it is not in a yui-g
<rockstar> sinzui, https://pastebin.canonical.com/20819/
<sinzui> rockstar: your <tal:side metal:fill-slot="side"> looks perfect
<rockstar> sinzui, yeah, I thought so too, but I keep getting: http://devpad.canonical.com/~rockstar/answers-subscribers.png
<rockstar> I can't for the life of me, get that "Subscribers" h2 to be gray, thought you might have some insight.
<sinzui> I wonder if the id is imposing style. <h2> is always color: #5a5a5a; in maincontent and .side
<beuno> rockstar, h2's seem to be tweaked for blueprints?
<rockstar> beuno, maybe.  Sleuthing now.
<beuno> intellectronica, BjornT, is there any reason why we show inactive people in the subscribers portlets?
<beuno> rockstar, what does firebug say?
<beuno> it should tell you where it's coming from
<rockstar> Yup, firebug says there's some special rule overriding the color.
<intellectronica> beuno: really? that sounds pointless
<sinzui> beuno: rockstar I see that h2 is resent in style-3-0.css, but maybe it is wrong:
<sinzui> body.tab-answers #mainarea #portlets .portlet.active h2
<BjornT> beuno: i don't think so. we simply haven't cared about it, and that's the way they are rendered, i think
<beuno> intellectronica, Bjorn, I will file a but then. It will de-clutter it a bit
<intellectronica> cool
<sinzui> beuno: intellectronica: wee need a process that remove deactivated and suspended users from teams and subscriptions. There is a bug for it
<rockstar> beuno, sinzui, .tab-answers h2 is defined on line 2210 of style.css as color: 3840BE
<sinzui> we do not want to automatically remove the user in the case of suspended because we may want to restore the user in a few days
<beuno> yes, bug 238493
<mup> Bug #238493: Deactivated account still appear in 'subscribers'-portlet <registry> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/238493>
<beuno> rockstar, bad css! bad!
<rockstar> beuno, I think the big thing is that the 3.0 styles should override the style.css stuff.
<beuno> rockstar, it should kill style.css, actually
<rockstar> beuno, so I can eated that bad style then?
<sinzui> rockstar: can we tweak line 141 of style-3-0.css to fix the issue
<rockstar> sinzui, I don't see a id="portlets" in my current mockup.
<EdwinGrubbs> sinzui: you mentioned putting polls in a portlet. Is that for the team, or some other object?
<sinzui> hmm
<rockstar> Er, s/mockup/markup
<sinzui> rockstar: I think the rule I wrote was bad then We need a definitive rule to make h2 grey
<sinzui> EdwinGrubbs: yes, polls belong to teams
<rockstar> sinzui, why not just .portlet h2 { color: grey }
<sinzui> rockstar: okay, use the real colour though
<rockstar> sinzui, yeah, I will.
 * rockstar was being lazy
<rockstar> sinzui, so, like I said, the real issue is that no matter what I put in the style-3.0 css, the style.css rule overrides it.  :/
<sinzui> firebug confirms this?
<rockstar> sinzui, yup, the color is properly set in the 3.0 styles, but then is crossed out for line 2210 of style.css
<rockstar> Basically, h2 is set globally in style-3-0.css, and overridden in style.css to change the colors.
<sinzui> rockstar: but 3.0 is loaded after style, and 3.0 sis after reset
<rockstar> :/
<sinzui> If I make a change 3 times and never see anything work, I assume I am editing or viewing the wrong thing
<sinzui> rockstar: if you copy that crack selector from style to 3.0 and set the colour, is the issue fixed?
<rockstar> sinzui, trying now.
<beuno> thumper, sinzui, BjornT, could you guys fill in: https://dev.launchpad.net/UI/ThreeDotOPages
<thumper> yes
<sinzui> I will
<rockstar> sinzui, I slapped my browser a bit, and it's working now.
<sinzui> control+shift+r ?
<rockstar> sinzui, I restarted my dev instance, and then refreshed, and everything was hunky dory.  No idea...
<maxb> Is there any way to disable shhh.py without having to remember to say SHHH= on every make invocation and without having a local uncommitted modification that blocks merging?
<gary_poster> maxb: pretty sure not.  I suppose we could add some sort of a ~/.NAME scheme for a directory or a file if it made sense
<maxb> I filed bug 408564
<mup> Bug #408564: Make shhh.py not the default <Launchpad itself:New> <https://launchpad.net/bugs/408564>
<rockstar> sinzui, actions go above portlets, but in the same slot, correct?
<gary_poster> maxb: cool, I see your point.  I tend to run the internal commands by hand when I want detail, but I see what you mean.
<JamalFanaian> do you have to restart the launchpad daemon if you make any changes to the py files?
<JamalFanaian> so far it seems like that is the case for me
<rockstar> JamalFanaian, yes.
<JamalFanaian> rockstar: ah ok, thanks
<sinzui> rockstar: every block of content is a portlet. I am preparing a branch that creates an action portlet from a NavigationMenu that can be placed as the first portlet in the side
<rockstar> sinzui, great, that would be helpful for what I'm currently doing.
<sinzui> rockstar: I created (well reused) a navigation menu. I just invented
<sinzui> <tal:menu replace="structure view/@@+global-actions" /> a few hours ago
<rockstar> sinzui, ah, so it's nothing magic, just a new way to do it?
<beuno> salgado, call?
<sinzui> rockstar: nothing magic. I am reusing the all the nav menu exception for the black bar template (we can remove it in a few weeks)
<salgado> beuno, haven't we moved it to 21UTC? I need to go out in a few minutes to pick up my new passport
<beuno> salgado, ah, yes. Got my UTCs wrong
<beuno> just ping me  :)
<salgado> will do
<sinzui> rockstar: beuno is looking at bug 405916, it has some screen shots showing what can happen in yui. The nicest solution is the trickiest to implement
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
 * jml looks around
<sinzui> rockstar:  you may want to take a look at the firefox-3.0-missing-rows.png because that is what will happen with the yui classes that are documented in the conversion page
<JamalFanaian> Sorry if this is a retarded question, but what does foaf stand for?
<gary_poster> sinzui: I'm getting a test failure on my zope-egg branch that is odd, and I wonder if you have any insights.  Running "./bin/test -vt test_script_default_error_dir" fails in my branch and on trunk.  The error is that "/var/tmp/lperr.test is not a directory".  I actually can't see where that directory should have been made in the test's setUp, and I wonder if it has been passing because of test ordering "luck".  How is that s
<gary_poster> JamalFanaian: friend of a friend.  it is an rdf spec, IIRC
<sinzui> gary_poster: on my two computers it is /var/tmp
<JamalFanaian> gary_poster: hm... ok i guess that makes sense
<sinzui> gary_poster: I I recall needing to root to make that directory mine during some big config change
<gary_poster> sinzui: I have /var/tmp.  I don't have /var/tmp/lperr.test .
<mwhudson> morning
<gary_poster> morning
<gary_poster> or something :-)
<jml> hi
<sinzui> gary_poster: I have that a lots of other dirs
<gary_poster> sinzui: right, but presumably the tests are responsible for making that dir?
<sinzui> gary_poster: do an ls -l and see if you lost ownership
<sinzui> correct
<gary_poster> sinzui: no.  I have ownership, and can add directories there.  So, on both machines, if you remove /var/tmp/lperr.test and then run that test in isolation, it passes?  If so, it's clearly something on my box.  However, I'm not convinced of that yet
<beuno> sinzui, trying to decide
<beuno> and understand  :)
 * sinzui makes a quick drawing of the underlying implementation
<salgado> beuno, if you're free, we could have the call now. (gf is late so we won't be able to get the passport today)
<beuno> salgado, sure
<beuno> salgado, I'm on skype: martin.albisetti
<salgado> I think I have you
<gary_poster> sinzui: when you have a moment, then, I think that this https://pastebin.canonical.com/20830/ is necessary to make ``./bin/test -vt test_script_default_error_dir`` pass in isolation, without a primed /var/tmp.  It works for me.  You and (mostly) Stuart seem to be the ones who have touched this, so if you can verify that what I am proposing seems reasonable, I would appreciate it.  (If you don't have time, understood.)
<beuno> jml, FWIW, edit away on the wiki!   :)
<jml> beuno, thanks :)
<sinzui> beuno: I added a comment that explains what is making the pictures
<sinzui> gary_poster: I get the same error after I delete the dir
<gary_poster> sinzui: cool
<gary_poster> sinzui: ok good enough, unless you want to look at the patch.  I think it is right. :-)
<sinzui> I wonder id it fails when the layer is run
<sinzui> I bet something else creates it
<beuno> sinzui, does my inclination towards #4 align with your expectations?
<sinzui> yes, that is what I have ready to commit because that is what I want to do. I think I should not recommend it to all engineers since it requires more though
<sinzui> t
<beuno> sinzui, I commented on your preferred option as well
<gary_poster> the layer is run when you run this in isolation.  the test may be in the wrong layer, no idea.  but the test's setUp is already creating an oops directory at another location and populating it, so this is at least not out of character
<sinzui> beuno: I think messing with yui-* definitions is a bit dangerous. These problems we are seeing as the reason yui-grid is deprecated. New browser can do this better, so the next layout solution for YUI will use CSS 3
<beuno> sinzui, agreed. Still, we need to make sure the end result is good, no matter what we use
<sinzui> I think this will be a lot easier when I compile this experience into sample templates
<sinzui> rockstar: I am now splitting my work into 3 branches. The first is an infrastructure branch that will help with the menus and provide layout ideas. My branch also has some fixes for FAQs
<rockstar> sinzui, yeah, I've been keeping my branches in pipes so that I don't have to split it up later.
<rockstar> sinzui, I'm currently auditing the answers pages to make sure I didn't forget anything.  That's how I found I had forgot action menus.
<sinzui> rockstar: I have learned never to mix yui-* with portlet
<rockstar> I don't use answers enough to know what might be missing.
<bac> sinzui: quick question, the portlets i have on the side don't have rounded corners where the examples i've gotten from you and bigjools do.  what does that indicate?
<sinzui> I added links to the faqcollection menu
<rockstar> sinzui, why?  I though yui-* wasn't doing anything currently.
<thumper> sinzui: one thing that I found when starting the conversions, is that I didn't have any expectations of what the changes would be
<thumper> sinzui: is it expected that the text gets bigger?
<sinzui> bac CSS .side .portlet. I think you are missing the portlet class on you content divs, or you are not using the side slot
<bac> sinzui: i know i am using the side slot
<rockstar> thumper, yes, in some places.
<rockstar> thumper, generally, I just went with the flow of what happened with CSS, unless there was something totally out of whack (like most the answers CSS)
<sinzui> rockstar: yui-* does positioning, not decoration. since our content often has rules to not render if the portlet is empty, you get wholes in your layout: https://bugs.edge.launchpad.net/launchpad-registry/+bug/405916.
<mup> Bug #405916: Update project page to UI 3.0 <story-ui-3> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/405916>
<sinzui> rockstar: think of yui-g as a tr and yui-u as td, and the 'first' class a a col rule. They all assume the content is always rendered and demands that it control which is first.
<rockstar> sinzui, Ah, I see. I wondered how first would be used if that portlet wasn't rendered.
<sinzui> rockstar: so I am being very strategic with yui-g. never put an optional portlet next to an essential portlet. I will update the conversion doc with my experience and suggest 3 different combinations of yui-* classes to make the page display well
<rockstar> sinzui, when you say "Global object menu" do you mean action menu?
<rockstar> sinzui, cool.
<rockstar> sinzui, I've also been trying to update it as I go along.
<sinzui> yes, the action menu what I mean
<rockstar> sinzui, so are you planning on fixing that bug soon?
<rockstar> I was just going to do what you proposed earlier.
<sinzui> thumper: the bigger text is a side-effect of the backward compatible fonts rules that were added last release. I am tempted to remove them in my next branch
<sinzui> rockstar: I am making that branch now.
<beuno> sinzui, do it. I keep on promising I will and never do
<rockstar> sinzui, okay.  Do you have an ETA on that?
<sinzui> goody. I get to see if my page really looks like the mockup now
<sinzui> Landing tomorrow. I'll give you the branch URL when it is ready
<jml> g'night all.
<beuno> night jml
<bac> sinzui: could you take a peek at this snippet:  http://paste.ubuntu.com/248246/
<bac> sinzui: re: square corners
<sinzui> bac You do not need yui-u or first because the classes are not in a yui-g
<sinzui> bac: crtl+shift+r if you are not seeing rounded corners. The rule in the css is .side .portlet {
<sinzui>     -moz-border-radius: 3%;
<sinzui>     -webkit-border-radius: 3%;
<sinzui> }
<sinzui> I see a     border-radius: 3%; property too
<bac> sinzui: the corners are roundy in ff but not safari
<sinzui> bac: That is interesting
<sinzui> bac: note that we have both the webkit and official properties in the rule. Can you try to fix the error?
<bac> sinzui: yeah, can you give me an idea where to start?  just poking at the CSS?
<sinzui> bac:-webkit-border-radius is safari 1-3, border-radius is safari 4. I expected that to just work
<sinzui> bac: start with this reference: http://en.wikipedia.org/wiki/Comparison_of_layout_engines_(CSS)
<bac> sinzui: changed -webkit-border-radius to 5px; and it works
<sinzui> interesting
<sinzui> bac: I suppose that border-radius is not supported as I thought. It is listed as experimental
<bac> mwhudson: ping
<mwhudson> bac: hello
 * rockstar standses up
<maxb> "Send the email to contributor-agreement@canonical.com and to the project lead for the project in question (usually, that's the person who asked you to send in the form)."   -- does that mean I'm supposed to talk to someone before filing the contributor agreement?
<lifeless> is it for launchpad?
<maxb> yes
<lifeless> kiko
<lifeless> just mail him the agreement
<lifeless> that prose seems to assume much more centralised communication than we're seeing
<thumper> rockstar: http://kiwicon.org/default
<lifeless> classic
<mwhudson> thumper: i hear rumours of a 2009 version of that, know anything about it?
<thumper> mwhudson: look at the twitter feed
<thumper> mwhudson: Nov 28th 29th
<mwhudson> ah, another weekend thing
<thumper> sinzui: got a few minutes tot alk?
<sinzui> I do
<sinzui> thumper: https://code.edge.launchpad.net/~michael.nelson/launchpad/3-0-menu/+merge/9577
 * thumper goes to cook bacon - http://xkcd.com/418/
#launchpad-dev 2009-08-06
<maxb> gah. /me carefully follows the instructions on what to write in the email to contributor-agreement@canonical.com.... and then forgets to actually attach the pdf
<mwhudson> thumper: when i was an undergrad, we had a "rashers of bacon consumed" league table in our house for a term
<thumper> mwhudson: heh
<mwhudson> it got a bit silly
 * thumper is making a quadruple bacon egg and cheese muffin, well, two muffins
<thumper> the bacon had to be used
<thumper> it would be a shame to throw it out
<thumper> I only realised after it went in the pan that I _could_ have frozen some
<ajmitch> such a thing to regret, having too much bacon
<lifeless> Bacon Explosion!
<lifeless> DoIt
<thumper> wgrant: your announcement method rename fix is landing now
<wgrant> thumper: Thanks.
<maxb> Is there any good way to produce a summary list of failing tests?
<mwhudson> man concurrency is hard :(
 * mwhudson gives up on it for now and has lunch instead
<wgrant> thumper: Does PQM hate me, or is it still just really slow?
<spiv_> PQM is an equal opportunity hater.
<spm> wgrant: you've got a branch waiting to land via pqm?
<spm> yes.
<wgrant> spm: I presume so.
<spm> wgrant: pqm is currently stopped due to a CP in progress, sorry. please to be waiting ~ 4 hours.
<wgrant> But it's all still private, along with buildbot.
<wgrant> spm: Ah, right. Thanks.
<maxb> CP ?
<wgrant> Cherrypick.
<spm> maxb: basically taking a revno that has already landed on one of the trees; merging it into the existing current build; run the full test suite. push that live where appropriate. Is only done for critically broken things.
<wgrant> Or forgotten "What's new?" updates. Ehem.
<maxb> How can I ask the test system for a list of tests in the order they will actually be executed? I tried --list and it gave me a list, but they don't seem to be executing in that order
<maxb> --list-tests
<lifeless> maxb: they get sorted to reduce churn on expensive test fixtures
<lifeless> maxb: there isn't a defined order, as such.
 * maxb crosses fingers and hopes this testrun actually manages to finish
<maxb> It's not looking too bad actually (Python 2.5)
<sidnei> maxb: glad to hear that!
<maxb> 26 failures 19 errors out of some ~17000 tests so far
<sidnei> not bad at all. reminds me of my templating branch. which, by the way, i need to bring up again. hopefully tomorrow :)
<jtv> Hi folks, everything still running?
<wgrant> sidnei: Which templating branch? The Chameleon one?
<thumper> crap
<thumper> ec2test I started before I ran away failued due to a trunk conflict
<thumper> grr
<jtv> thumper: buy a faster cloud
<thumper> jtv: and why would that help?
<jtv> thumper: so you can find out before you run away.
 * thumper smacks jtv around for a stupid idea
 * thumper is not in a good mood today
 * jtv tries to look chipper and supportive, but can't keep the hurt look out of his eyes
<thumper> jtv: I have no sense of humour today
 * jtv chokes back obvious bad joke about Americans
<jtv> mwhudson: the translations-to-bzr script is doing quite nicely...  it just exported translations for 18 productseries, skipped 1, and failed 4 because their branches weren't set up, all in 2 minutes 10 seconds.
<jtv> I really ought to start skipping unchanged files, for that order-of-magnitude speed boost.
<mwhudson> jtv: hooray
<jtv> mwhudson: the dirty little secret there is that ddtp-ubuntu still breaks it because the transaction runs for too long and the database watchdogs kill it.  Fix has landed, but we won't CP it unless it becomes unavoidable.
<jtv> mwhudson: would it be very hard, or evil, to create branches that give me "not a branch" errors?
<thumper> anyone remember the view attribute for the new page titles?
<jtv> thumper: not off the top of my head... but wasn't pagetitles.py also supposed to work?
<thumper> yes, but... I'm going to add a comment to the top of pagetitles.py saying "stop bloody using this file!!!"
<thumper> and I'd like to be nice and tell them what to use instead
 * thumper goes hunting
<thumper> hah
<sidnei> wgrant: xactly
<thumper> page_title
<thumper> surprise!
<jtv> *gasp*
<sidnei> wgrant: looking for someone to pick that up and push forward (hint hint)
<mwhudson> jtv-afk: no, not at all
<mwhudson> hard, i mean
<jtv> mwhudson: I guess though that the user may still have a branch sitting around locally, waiting to be pushed...
<jtv> If I created the branch and populated it, and then the user pushed a different branch altogether, would the user's branch "win"?
<lifeless> if they use --overwrite yes
<jtv> So not an unsafe idea at all to create the branches... How would I do it?  Would be nice if DirectBranchCommit could do it.
<mwhudson> jtv: you could create another branch type
<mwhudson> jtv: and have them pulled from some known location
<mwhudson> (a bit like import branches are now)
<jtv> mwhudson: I do require that it be Hosted.  If it's another type, it won't be referenced in this field.
<mwhudson> and not let anyone write to them on disk
<mwhudson> (don't know if this would make sense)
 * wgrant curses replication lag to death.
<wgrant> It's particularly bad now that the bug page is AJAXy, as AJAX POSTs and PATCHes don't force further browser requests to the master.
<lifeless> wgrant: file a bug
<wgrant> There is a bug.
<lifeless> jtv: I would hesitate to write to the users branch directl.y
<lifeless> jtv: writing to your own branch the user can pull/merge from is conceptually clearer.
<lifeless> Personally, I'd model it as: there is a user ~rosetta
<jtv> lifeless: I already write to the user's branch, as per the user's wishes.  Question is whether I should create the bzr branch behind the user's "DB branch" if there isn't one.
<lifeless> ~rosetta can write to any branch that the user has been given access to
<lifeless> well, as long as they tell you the branch to branch from
<jtv> They have already created the DB branch and configured it as the one I should write to.  But there isn't necessarily a real branch behind it yet.
<lifeless> jtv: if there isn't a real branch, you would have to branch from somewhere
<lifeless> jtv: thus my comment
<jtv> lifeless: oic... basically an empty "prototype branch"?
<lifeless> jtv: that said, I'm strongly inclined to consider a db branch without bzr data in it a configuration error
<lifeless> jtv: no, nothing empty
<jtv> Okay, near-empty?
<jtv> Anyway, isn't a db branch without any bzr data what you get when you register a Hosted branch in LP?  It seems a bit pointless to require that users populate the branch.
<jtv> (In this case)
<jtv> lifeless: actually, why would I have to branch from somewhere?
<lifeless> to get their content
<lifeless> jtv: registering 'hosted branches in lp' is a bug, it causes lots of questions and confusion
<jtv> In this case it's just what I want though, and if I could just create the bzr directory on the fly if needed, that'd be perfect for me.
<lifeless> so whats the use case.
<lifeless> I'm not understanding the desire to have a branch the user can't merge from
<jtv> I don't understand... Why would the user not be able to merge from it?
<lifeless> try this
<lifeless> cd /tmp
<lifeless> actually
<lifeless> cd ~lp-repo
<lifeless> bzr init --2a foo
<lifeless> cd foo
<lifeless> bzr commit -m "first post"
<lifeless> bzr merge ../trunk
<lifeless> [adjust paths for your environment]
<jtv> The commit fails, of course.
<lifeless> it shouldn't
<jtv> There's no data in there yet.
<lifeless> oh, add --unchanged there
<jtv> Anyway, what you're trying to tell me is I can't merge branches without common ancestors?
<lifeless> yes
<lifeless> it goes deeper
<lifeless> you can't merge files unless the files started at a common place
<Ursinha> jtv, hi! :)
<jtv> hi Ursinha!
<Ursinha> jtv, do you have a few minutes after your discussion with lifeless?
<jtv> Ursinha: very very few.  :)
<jtv> Ursinha: if it's about the approval script failures, we're on it.
<Ursinha> jtv, :P
<Ursinha> jtv, actually no, it's about some weird timeouts
<jtv> lifeless: so it'd be good for the user to choose the branch wisely... but in this case afaics having a branch that lives in its own universe is still a lot better than having one that produces nothing at all.
<lifeless> jtv: I still don't understand the case
<lifeless> what is the use case
<jtv> lifeless: this is for daily translations exports.  "Pick a Launchpad-hosted branch and I'll commit daily snapshots of your translations to it."
<jtv> lifeless: minimally, bzr is just used as an asynchronous transport here.
<jtv> An alternative to ftp, almost.  Merging is slightly more advanced usage, but not everyone will even want that.
<lifeless> jtv: thats fine; just check when they select the branch that it has been successfully mirrored once
<lifeless> jtv: if they want an empty branch, they do 'bzr init lp:~person/project/trunk
<jtv> lifeless: mirrored?  or pulled?  this works for Hosted branches only.
<lifeless> jtv: copied from private hosting to http
<lifeless> jtv: what I'm saying is: don't special case this in your code. Depend on them making the branch appropriately.
<jtv> lifeless: sorry to be thick, but... private to whom?  And to what over http?
<lifeless> it is, at worst, one command.
<lifeless> jtv: hosted branches get scanned and published inside the server
<lifeless> there is a flag written when this succeeds.
<lifeless> I'm saying check that this flag has been set
<lifeless> as its being set indicate the branch has been real for at least one puller cycle
<jtv> lifeless: now I see...  That's good, thanks.  It's another way of avoiding these quiet failures.  I'll file a bug.
<jtv> lifeless: I think they call the http-accessible hosting facility the "read-only area."
<Ursinha> hi thumper
<jtv> lifeless: bug 409686
<mup> Bug #409686: Export branch should have real bzr branch attached <Launchpad Translations:Triaged> <https://launchpad.net/bugs/409686>
<jtv> thumper: about those timeouts...  we see weirdness all over the place atm and suspect foundational problems.
<jtv> sorry, not thumper, Ursinha
<jtv> (I wonder how I could've made _that_ mistake...)
<Ursinha> jtv, I've asked flacoste earlier today about those, questioning that that could be due to the storm update, but he said that was unlikely, and that we had changes in messaging sharing
<Ursinha> and if I asked danilo about that... I don't, so I'm asking you :)
<Ursinha> let me find some of the oopses
<Ursinha> jtv, OOPS-1309F1091,OOPS-1312C1554,OOPS-1309F1091
<Ursinha> the slowest query is select replication_lag(), that according to flacoste should be fast
 * Ursinha curses the bot
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1312H1982
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1312C1554
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1309F1091
<spm> wgrant: fyi. pqm reopened, so your landing should pass shortly
<wgrant> spm: Thanks.
<wgrant> spm: It has indeed just landed.
<spm> \o/
<thumper> I need to go make dinner
<thumper> TTFN people
<jml> hello
<lifeless> hai
<lifeless> jml: http://advogato.org/person/robertc/diary/108.html
<jml> lifeless, \o/
<lifeless> hudson is pretty sext
<mrevell> Howdy
 * maxb rejoices at running the testsuite all the way through for the first time on python 2.5
 * maxb doesn't rejoice about it taking 5 hours though
<wgrant> maxb: What are the stats?
<noodles775> Great stuff maxb !
<maxb> https://dev.launchpad.net/LaunchpadOnKarmic
<maxb> wgrant: I've added a test summary ^
<maxb> in brief - there's work to be done, but it could have been a lot worse
<noodles775> maxb: 1328 tests ran?
<noodles775> Hmm... It should be closer to 23k?
 * noodles775 checks latest buildbot run.
<wgrant> buildbot is still in the exclusive realm of Canonical employees :(
<noodles775> 23741 tests.
<bigjools> make check will run 23k tests
<bigjools> if it doesn't, there's a problem with the tests
<wgrant> That's getting close.
<jml> wgrant: as in, buildbot visibility?
<maxb> 23117 if you sum all the categories
<wgrant> jml: Yes.
<jml> :(
<wgrant> maxb: That's what I got too.
<wgrant> So, not too bad.
<wgrant> I'm very surprised so many passed.
<wgrant> how much of Zope did you have to poke?
<maxb> I cherrypicked some upstream changes for zope.proxy and zope.security, and did an almightly bodge in zope.sendmail
<wgrant> Ah, I shall check the branch.
<maxb> oh, and a cherrypick for zope.app.security too
<noodles775> maxb: sounds close (the buildbot run that ran 23741 tests was r9053).
<maxb> r9053@devel ?
<noodles775> Yep.
 * maxb curses bzr's weird idea of revnos
<wgrant> What's weird about it?
<wgrant> apart from presumably being different to hg's.
<jml> BjornT, you around?
<BjornT> jml: yes
<maxb> The way it bears no meaning whatsoever cross-branch
<jml> BjornT, I was wondering about bug-branch linking appearing in the activity log.
<jml> BjornT, ISTR you or someone on the bugs team fixing something in the scanner to make the linking appear inline with the bug comments, but that doesn't seem to be what's happening.
<maxb> Ah right, so by happy coincidence my testrun was based on devel@r9053 too
<BjornT> jml: no, we didn't fix it to appear inline, we just fixed it to actually appear in the activity log.
<jml> BjornT, ahh ok.
<jml> BjornT, and one doesn't imply the other, I take it.
<BjornT> jml: right. we don't want all activity to appear inline, for example subscriptions. however, there are some things that definitely should appear inline, but we didn't have time to implement it.
<jml> BjornT, ok, cool.
<BjornT> jml: you think it would be useful to have branch linking being inline? there's always the question whether it's actually useful, or if it simply adds noise
<noodles775> maxb: but that's your local branch of devel, you can easily check the log to see where you diverged from devel (or did you mean that you'd merged devel@r9053 before doing running tests?)
<jml> BjornT, I think so.
<jml> BjornT, it doesn't happen often, and means that someone has done some work on the bug.
<jml> (a secondary bug is that the bug should probably be marked as in-progress)
<maxb> noodles775: I mean I used bzr log -n0 --show-ids to find my last merged rev and then looked up the revid in devel to map it back to a revno
<noodles775> maxb: in which case, it's strange that you had a different number of tests. Was that using 'make check'?
<maxb> yes, make check
<jml> BjornT, we have a bug somewhere on launchpad-code about making it appear in line. I'd like to add some notes to the bug about how to actually do it.
<bigjools> maxb: `bzr misssing` not work for you?
<wgrant> noodles775: It's not incredible that some test modules would fail to import.
<maxb> bigjools: that'd tell me what I didn't have - I wanted to know what I didn't have - oh.... so I should have run bzr missing --this in devel and subtracted one from the lowest shown revno
<noodles775> wgrant: no not at all, I'm just surprised that it didn't cause an error if that's the case.
<maxb> Is it possible that some of the errors noted in that summary could abort the entire test module, skipping later tests?
<jml> BjornT, where should I start looking to add bug/branch linking events to the inline view
<wgrant> noodles775: The tests don't exist if the module failed to import.
<wgrant> noodles775: There'll just be an exception somewhere back in the log.
<bigjools> maxb: yes, that can happen
<noodles775> maxb: good question.
<bigjools> particularly in doctests
<wgrant> I forget exactly how doctests count. Is it one test per block?
<BjornT> jml: you should start at BugTaskView.activity_by_date, and adjust the interesting_changes bit to include branch linking
<wgrant> Not as in Python syntax block, but doctest block.
<BjornT> jml: with some luck, that's all that's needed
<jml> BjornT, thanks.
<bigjools> wgrant: no idea, it seems random to me :/
<noodles775> wgrant: Hmm... if that's the case, then I'd say it's a bug - if 'all tests pass' then it lands (at least, I don't scroll back if all tests passed)
<bigjools> if he altered tests then the number of tests could have changed
<noodles775> wgrant: so hopefully the difference is due to the failures or altered tests, rather than import failures.
<wgrant> noodles775: Hopefully.
<jml> BjornT, the string should match the 'whatchanged' key of the activity dict, I assume.
<BjornT> jml: yeah. if it doesn't look good, you have to change BugActivityItem to do special things for branch links
<lifeless> jml: http://gorf.tangent.org/hudson/job/Drizzle-subunit/ has a test graph now
<jml> BjornT, thanks.
<maxb> zero altered tests
<maxb> (so far)
<wgrant> maxb: Did you collect a list of the failed tests to start attacking?
<maxb> I have the entire log courtesy of screen, I need to work my way through it turning it into a list of test names and initial notes of probably causes
<maxb> *probable
<maxb> When I say zero altered tests....
<maxb> I did have to alter the harness to wait longer for the librarian to start up
<lifeless> I wish lp output in subunit at this point
<lifeless> cause then you could do 'subunit-filter log | subunit-ls' and have that list instantly ;)
<wgrant> Does a newer zope.testing do subunit?
<bigjools> buildbot seems to parse it ok!
<lifeless> wgrant: not per se; jml and I did a patch to launchpads test runner use arbitrary TestResult objects
<lifeless> which permits subunit; jml was landing that work but I dont know if he had
<jml> lifeless, I think I have
<lifeless> cool
<jml> but it might have rotted a little with recent buildout shenanigans.
<lifeless> its a shame buildout != dpkg
<wgrant> Indeed...
<wgrant> eggs always seemed like a bad direction.
<lifeless> its a bit of an awkward halfway-house
<lifeless> wgrant: I loathe them. With a passion.
<maxb> Does anyone not loathe eggs? :-)
<wgrant> lifeless: I noticed! I'm not a fan either.
<lifeless> anyhoo... internet dragons to kill
<lifeless> jml: theres an xfail support patch up you might like to review; if you have a minute
<jml> lifeless, sure. I'd like to dive back into actual Launchpad hacking now that my email kneejerks are done
<jml> lifeless, what's the url?
<mwhudson> jml: hello, btw
<lifeless> https://code.edge.launchpad.net/~lifeless/subunit/xfail/+merge/9748
<jml> mwhudson, hi
<mwhudson> jml: want to review a branch?
<jml> lifeless, ta
<jml> mwhudson, is this your race condition branch?
<mwhudson> jml: i'm much happier than i was with this stuff than i was yesterday :)
<mwhudson> jml: yes
<mwhudson> jml: the cover letter will include alan perlis' epigram no. 58
<jml> mwhudson, beautiful. I'd love to look at it.
<lifeless> jml: have a good day
 * lifeless goes to entertain himself
<jml> lifeless, good evening
<mwhudson> jml: https://code.edge.launchpad.net/~mwhudson/launchpad/more-task-scheduled-bug-408638/+merge/9749
<jml> mwhudson, thanks.
<jml> I'm wondering if it's worth having a bug tag for "I'll bend over backwards to help you land a patch that fixes this bug"
<mwhudson> jml: thanks in advance
<jml> mwhudson, np
<mwhudson> jml: if you want to ask things about it, sooner is better than later :)
<jml> mwhudson, sure.
<jml> mwhudson, in context switch mode right nowm.
<mwhudson> k
<jml> 'if' should be an expression in Python
<mwhudson> it is in 2.5!
<mwhudson> ish
<deryck> Morning.
<jml> mwhudson, review done.
<jml> but I guess it's pretty late
<wgrant> A couple of days ago I did a branch to export IArchiveDependency on the webservice, but I'm not entirely sure about the security bits.
<wgrant> At the moment read access is allowed unconditionally, and write access is completely forbidden, but you can't get hold of one with having launchpad.View on the Archive. Should a security adapter be written for IArchiveDependency, or is that sufficient?
<jml> I'm not sure.
<jml> wgrant: I'd guess 'no', but I don't really have enough domain knowledge to give a full answer.
<jml> bigjools, ^^ ?
<wgrant> jml: 'no' to which?
<bigjools> wgrant: if you're not sure then you haven't written enough tests
<wgrant> bigjools: I know I can't get hold of an object, but I wonder if I should make it safer in case another exported method leaks them.
<jml> wgrant: "no you don't need to write an adapter"
<wgrant> jml: OK.
<wgrant> But I think I probably do.
<jml> ok.
<jml> you'd know better than me, at this point :)
<bigjools> wgrant: how do you know write access is forbidden?
<wgrant> bigjools: I haven't a test for that yet, but lib/lp/soyuz/configure.zcml strongly suggests that it is.
<bigjools> wgrant: if it's not tested then you don't know, by definition :)
<wgrant> Right.
<wgrant> Although similar tests (eg. ArchivePermission in stories/webservice/xx-archive.txt) don't seem to test that.
<bigjools> then they suck
<bigjools> :(
<bigjools> which is probably my fault
<mwhudson> jml: thanks
<wgrant> Can somebody point me at a good example?
<wgrant> I'm not clear on exactly how thorough the tests should be.
 * bigjools looks
<bigjools> wgrant: lib/lp/soyuz/stories/webservice/xx-archive.txt "Modifying privacy"
<jml> wgrant: lp.code.tests.test_branch has unit tests for branch security
<bigjools> jml: I think he explicitly needs webservice tests though
<jml> oh right.
<bigjools> the existing object is fine as it is, it's not directly exposed until wgrant exports it
<jml> bigjools, about verify_acl
<bigjools> yep?
<jml> bigjools, it doesn't currently check component privileges for existing packages.
<jml> is this a feature or a bug?
<bigjools> feature
<bigjools> deliberate
<wgrant> I don't know to what depth I must go -- do I just verify that people can't write to a couple of the fields?
<jml> bigjools, what's the reasoning?
 * bigjools is trying to remember
<bigjools> wgrant: test important fields
<bigjools> where it would compromise functionality
<jml> because BugTask.canApprove consults component privileges, and IIRC you implied I should check it for the package branch stuff.
<wgrant> bigjools: OK, sounds good.
<bigjools> jml: there's a comment in the middle of verify_acl that explains it.  I remember putting it there because I kept forgetting why :)
<bigjools> it's because of overrides, basically
<jml> bigjools, I'm looking at verify_acl & don't see anything that answers my question.
<bigjools> oh wait the comment talks about new packages.  in any case, it's because of overrides
<jml> ok.
<bigjools> so we make sure that the person has an ACL to at least one component
<bigjools> then let the overrides work after that
<bigjools> jml: ummm wait
 * bigjools engages brain
<bigjools> why do you think that?
<jml> bigjools, why do I think what?
<bigjools> jml: "bigjools, it doesn't currently check component privileges for existing packages."
<jml> bigjools, it checks it based on the dsc, but not the other thing... gimme a sec to find it.
<bigjools> yeah it's at the bottom of verify_acl
<jml> latest_published_component
<jml> right. so it checks the dsc rather than latest_published_component. why's that?
 * bigjools thinks
<wgrant> It's already overridden, isn't it?
<bigjools> yes, on the files
<wgrant> find_and_apply_overrides() is called a few lines before verify_acl
<bigjools> I can think of a situation where it might help
<bigjools> but I am not sure if it's valid
<bigjools> if a package goes universe->main, then a motu can continue to upload it if they target universe
<wgrant> That's not reasonable, and I don't see why that would work?
<bigjools> because it checks the dsc and not the current component
<wgrant> But the dsc component is already overridden with the latest ancestry.
<bigjools> ah so it os
<bigjools> is
<wgrant> That's pretty damn strange.
<wgrant> But it works.
<bigjools> christ I need more coffee
<bigjools> jml: ok so it checks based on the override, just in a roundabout way
<wgrant> I smell a missing comment.
<jml> and AIUI, the override is not the same as the dsc file is not the same as latest_published_component
<jml> is this correct?
<bigjools> distroseries.getPublishedReleases()[0] is used
<bigjools> which should be the same as latest_published_component
<henninge> danilo, jtv: Help!
<bigjools> well, its publishing record
<jtv> henninge: ?
<wgrant> (with the archive, too)
<wgrant> latest_published_component doesn't know about the archive.
<jtv> henninge: seems danilo's network conked out
<wgrant> But I don't see why that would matter in any current situation.
<henninge> jtv: oh
<bigjools> jml: so, it applies the override first, which changes the component on the DSC
<henninge> jtv: I thought it was me
<jtv> henninge: and me
<jtv> and him :)
<henninge> jtv: I don't see you on skype either, though.
<jtv> henninge: then I'll just restart to make sure
<bigjools> wgrant: it does matter, that's not good
<danilo> jtv: ok, back, but don't see you on skype
<jml> BugNomination takes the latest_published_component and asks archive.canUpload(person, component)
<jtv> danilo: I was just restarting it for luck
<jml> is that any different, in effect, from what nascentupload is doing?
<danilo> jtv: it's ringing
<bigjools> jml: eeek that's wrong, because if a PPA has a more recent publication in main it will use that
<wgrant> bigjools: When does it matter?
<bigjools> all of these queries need to be archive specific
<wgrant> bigjools: Doesn't latest_published_component restrict to distribution_archives?
 * wgrant checks.
<wgrant> I'm fairly sure it did.
<bigjools> ah it does
<bigjools> phew
<wgrant> It uses _getFirstPublishing or something like that.
<bigjools> ok
<bigjools> it's kinda gross
<wgrant> A little.
<wgrant> I imagine it was all better before archive-rework.
<wgrant> Nice and simple back then.
<bigjools> when we added PPAs, we found a lot of problems in our data model that needed hacks like that
<jml> bigjools, so it ought to be 'get_latest_published_component(archive)'?
<bigjools> jml: not needed (yet)
<bigjools> PPAs are always main
<jml> ok.
<bigjools> the nascentupload stuff is a remnant of when we used components in PPAs
<jml> SEP.
<mwhudson> jml: replied
<bigjools> jml: so did I answer your question yet? :)
<jml> the question is, "is BugNominations component permission check different from nascentupload's in any way that matters?"
<jml> the answer so far seems to be "no, and everything is terrible"
<jml> mwhudson, thanks.
 * mwhudson giggles at wgrant correcting bigjools on soyuz arcana
<wgrant> mwhudson: My mind is just freshly scarred with DSP and NascentUpload strangeness.
<bigjools> I stepped on too many soyuz mines and lost half of my brain
<wgrant> Easy to do.
<jml> so, was my answer correct?
<bigjools> pretty much
<jml> bigjools, ok, thanks.
<bigjools> jml: np, sorry it took so long to get there
<jml> it's ok. my goal here is to make this kind of conversation unnecessary in future. :)
 * bigjools hi-fives jml
<wgrant> I have to wonder why the files are overridden, rather than the ACL code explicitly checking the overrides.
<danilo> mrevell: hey, btw, what happened to frontpage updates?
<mrevell> danilo: I was waiting for flacoste to return to get the CP and didn't want to overwhelm him on his day back, so I'll be asking him later on today.
<danilo> mrevell: heh, ok, but kiko-afk is still able to do that for us as well :) fwiw, I already had one CP done
<kiko-afk> I am indeed
<mrevell> kiko-afk: Hey there. I was hoping to get a CP to update the what's new. I have an MP: https://code.edge.launchpad.net/~matthew.revell/launchpad/whatsnew227/+merge/9576
<jml> sidnei, hi
<sidnei> hey jml
<jml> sidnei, so, in answer to your question, the release finder stuff isn't exposed via the api afaict
<jml> sidnei, if you wanted to do so, you'd just have to expose IProductSeries.releasefileglob, I think.
<sidnei> jml: sounds easy when you say it that way *wink*
<sidnei> jml: i will give it a try
<kiko-afk> mrevell, rc=kiko, though normally you'd just ask for a release-critical review on the MP
<mrevell> ah thanks kiko-afk
<mrevell> noted for future
<sidnei> jml: what im trying to figure out is if i could automate setting up the import globs to suck in all the existing zope releases, which there maybe 50 of
<jml> sidnei, cool. just ping me if you need help with the patch, or want a review.
<sidnei> s/maybe/may be
<sidnei> jml: thank you!
<sidnei> jml: seems like there's no indication at all in the UI about the status of crawling for new releases
<jml> sidnei, that's quite possible.
<sidnei> jml: to boot, when you create the series it doesnt save the glob, you have to go back and fill the glob field again
 * sidnei files a bug
<danilo> beuno: hey
<bigjools> beuno: did you knock up any CSS for "Choose a task" ?
<noodles775> The CSS should "just work", I'll have a go at adding the template snippet to include it in the sidebar if it's not there already.
<jml> hi
<flacoste> hi
<jml> flacoste, hi
<jml> I just filed bug 409846, fwiw,
<mup> Bug #409846: Contributors file <Launchpad itself:New> <https://launchpad.net/bugs/409846>
<jml> not exactly sure what to put in such a file, right now
<flacoste> jml: contributors should list contributors?
<maxb> I don't think it necessarily follows that all OSS projects should have a contributors file
<jml> maxb, no it doesn't. I think we should have one though.
 * jml was a bit lazy filing that bug
<jml> flacoste, yeah, but I don't have a complete list to hand
<jml> maxb, a Changelog or a NEWS file would satisfy me.
<maxb> As a contributor, I don't care. As a committer, I don't want to bother maintaining one or judging how big a contribution gains entry
<maxb> A NEWS or CHANGES file is useful, but serves a completely different purpose to a Contributors file
<maxb> A Changelog.... hmm, well if everyone just writes decent commit messages
<mrevell> sinzui: Does this look like a question for the registry team? Or the code team? https://answers.edge.launchpad.net/launchpad/+question/79261
<sinzui> mrevell: code. I think their may be an faq about this. I recall helping someone with the same issues 1 month ago
<mrevell> Ah thanks sinzui
<mrevell> cprov: Can you help with this PPA question please? https://answers.edge.launchpad.net/launchpad/+question/79246
<beuno> danilo, hi
<beuno> bigjools, I don't think so
<cprov> mrevell: sure, assign it to me, I will reply in a sec
<mrevell> thanks cprov
<danilo> beuno: anyway, I just wanted to say that I am considering putting henninge on +translate page for the following two months, and we'd like some private time with you to discuss ultimate +translate page next week (to give all of us time to think about what'd be cool to do)
<jml> maxb, personally, I like to be credited when I contribute to projects, and I feel a bit awkward committing someone else's contributions without crediting them
<sinzui> mrevell: This question looks a lot like a question asked by https://answers.edge.launchpad.net/~basicxp but I think that was a feedback email.
<maxb> jml: Yes, but I'd address that in the log msg or bzr commit --author
<beuno> danilo, sure. Can we try tomorrow?  today's kind of crazy
<danilo> beuno: as I said, next week is best
<jml> maxb, I don't think that's enough glory :)
<beuno> danilo, even better
<danilo> beuno: and if my timezones are correct, we should be in a call in about 2 minutes :)
<beuno> danilo, yes, I have my headphones on
<mrevell> danilo: Could you take a look at this question please? https://answers.edge.launchpad.net/launchpad/+question/79233
<danilo> mrevell: sure I could, but I don't want to
<mrevell> heh
<mrevell> but you're going to, so that's ok
<leonardr> gary_poster: i might as well take care of bug 397903 while i'm touching every lazr package. is creating LICENSE.ZPL the right thing to do?
<mup> Bug #397903: ZPL LICENSE not included in source <lazr.restful-client:New> <lazr.uri:New> <https://launchpad.net/bugs/397903>
<gary_poster> leonardr: look in _bootstrap.  I believe that bug report is in error
<leonardr> gary_poster: indeed
<gary_poster> leonardr: so my opinion is mark as Invalid.  You can ask James if the names we have for the files are OK for Ubuntu, I guess, if you want to.  Or just leave it be, maybe, woud be better.
<james_w> I don't see a _bootstrap in the branch?
<gary_poster> james_w: no?  it is in one I have around (at top of package).  will look at trunk.
<leonardr> james_w: what branch are you using?
<james_w> lazr.uri at least
<james_w> lp:~launchpad-pqm/lazr.uri/trunk/
<gary_poster> I have one in my copy of that too.  Actually getting trunk now as promised
<leonardr> gary: i do not have _bootstrap in my copy of trunk
<gary_poster> I may have been bad and not merged, way back when when I made a PyPI release.  checking.
<gary_poster> james_w, leonardr: bzr+ssh://bazaar.launchpad.net/%7Elaunchpad/lazr.uri/1.0.1/ .  We (largely, I) didn't do a great job with getting all of this right in lp at the start.  1.0.1 should be merged with trunk, whatever that is
<leonardr> gary_poster: ok, do you want to fix the path while you're at it? then i'll review
<jml> cprov, ping
<cprov> jml: pong
<gary_poster> james_w: lazr.restful does have _bootstrap; it is now part of the lazr.* templates.  I don't think we have any actual releases without the license
<gary_poster> fwiw
<jml> cprov, I want to write a test for 'if a package is in a component and you have upload rights for that component in the archive then you can upload to that package'
<jml> cprov, where 'package in a component' is probably defined by sourcepackage.latest_published_component
<gary_poster> leonardr: 1.0.1 is an old branch of what was released as 1.0.1.  I'll branch it, make the path change, and push it to someplace or other...
<jml> cprov, how do I populate latest_published_component without actually uploading a package?
<matsubara> stub, herb, flacoste, rockstar, bigjools, henninge, sinzui, intellectronica: production meeting in #launchpad-meeting in 9min
<leonardr> gary_poster: so we did the release but didn't merge with trunk
<cprov> jml: SP.latest_published_component doesn't cope with PPA sources, IIRC
<gary_poster> leonardr: right.
<jml> cprov, it doesn't have to be PPA
<cprov> jml: right, you can create publications with SoyuzTestPublisher()
 * jml greps
<jml> cprov, I was kind of hoping for something a little more object factory friendly
<james_w> gary_poster, leonardr: thanks, if we can get fresh releases with these changes for lazr.restfulclient and lazr.uri that would be great
<leonardr> james_w: you should have for lazr.restfulclient now
<bigjools> jml: it's a sort of factory itself, but keeps the db relations intact
<james_w> thanks leonardr
<cprov> jml: we can wrap STP in the factory, but it requires a lot of assumption that are not good for the current callsites.
<jml> bigjools, the factory keeps db relations intact though, doesn't it?
<gary_poster> leonardr: it would also be fantastic to remove any remaining use of the setuptools_bzr stuff in these packages.  There's a bug for that someplace: it is annoying for *NIX, and apparently outright broken for Windows
<gary_poster> leonardr: I'm merging 1.0.1 to trunk now.  (At the time, merging required the whole pqm dance).  That does not include the new path stuff
<gary_poster> leonardr: this is the diff of 1.0.1: https://pastebin.canonical.com/20872/
<cprov> jml: it's not that difficult to create Factory.makeSourcePublication(...) using STP, but requires some cleanup.
<bigjools> it's possible but I question the value
<leonardr> gary_poster: r=me on that diff
<gary_poster> leonardr: cool thanks.  nevermind about me submitting to trunk.  I thought this was no longer managed by pqm.  Do we still have to trigger a launchpad pqm test in order to land on this branch?  If so, could I toss this back to you?  I'd really, really like to be working on the zope buildout branch
<leonardr> gary_poster: it looks like lazr.uri trunk is managed by pqm, but i think i can just change the development focus, like i did for lazr.restful and wadllib. flacoste, is this okay?
<flacoste> leonardr: why not simply submit through PQM?
<flacoste> we want all of trunks managed by PQM eventually
<flacoste> the reason it's not is because it's not set-up to run only the packages tests yet
<flacoste> but for lazr.uri, i think it should be fine
<flacoste> unless it runs the whole test suite
<flacoste> where whole = launchpad
<leonardr> why wouldn't it?
<leonardr> all right, i'll do lazr.uri after lazr.yourpkg
<flacoste> jtv: are you done with your lock on LPS?
<flacoste> jtv: wiki lock on LaunchpadProductionStatus
<jtv> flacoste: no, I've only had it a minute!
<jtv> flacoste: released it for now
<rockstar> abentley, ready to chat
<abentley> rockstar: 1 sec
<rockstar> abentley, just call when you're good to go
<allenap> Does anyone know why https://bugs.staging.launchpad.net/openfiler might keep timing out?
<allenap> It has consistently timed out on me all day.
<jml> it needs the librarian :(
<sinzui> Should I publicise  how to register a menu without touching ZCML?
<allenap> jml: Was that a response to my question? If so, is the librarian in staging borkened?
<allenap> Argh. I started writing borked, decided to use broken instead and somehow ended up with borkened.
<Ursinha> hey bigjools, you're using kde, right?
<bigjools> yarp
<Ursinha> bigjools, my kde acts up sometimes and doesn't allow me to do a rf-get or commit to a branch without asking the the passphrase of a rsa key
<Ursinha> but the key has no passphrase,so I'm not able to do what I want
<bigjools> that's down to the gpg-agent settings, not KDE
<Ursinha> it seems ssh-agent or something is borked.. don't know
<bigjools> or ssh-agent!
<Ursinha> bigjools, why does it work fine on gnome?
<bigjools> no idea :/
<Ursinha> bigjools, did you have this problem
<Ursinha> ?
<bigjools> no, nothing like that
<Ursinha> :(
<Ursinha> thanks anyway bigjools
<bigjools> Ursinha: can you narrow it down to what command causes it?
<Ursinha> the error messages or ssh-agent to work on kde?
<bigjools> well, you say bzr commit causes it?
<Ursinha> yes
<Ursinha> and a rf-get
<bigjools> have you got sign-commits turned on?
<Ursinha> trying to push branches to lp
<Ursinha> yes
<Ursinha> I do
<bigjools> turn that off and commit again, see if it happens
<bigjools> wait, committing does it, or pushing?
<jml> allenap, no, it wasn't
<jml> allenap, The SoyuzTestPublisher needs the librarian :(
<allenap> jml: Okay, thanks :)
<bigjools> jml: that can be bypassed with a code change if you want
<jml> bigjools, ooh, how?
<bigjools> jml: add an option so that addPackageUpload isn't called
<jml> ok, thanks.
<bigjools> that's the part that inserts librarian files
<bigjools> but make it default to true :)
<danilo> jtv: hey, are you around?
<jtv> danilo: barely... get my ping?
<danilo> jtv: if it's on internal IRC, no (you get on and off there)
<jtv> yeah :(
<jtv> "danilo: CP requests are up, all merge cleanly into 2.2.7, and all Translations tests pass (apart from that test_args thing that buildbot doesn't complain about; I didn't touch that).  And I'm done.  :-)"
<jml> bigjools, naturally :)
<danilo> jtv: just a note about https://code.edge.launchpad.net/~jtv/launchpad/bug-409330/+merge/9760 : it's really ugly to have this option by ID, and I don't want us to schedule a job like that ever
<jtv> danilo: a bit late to tell me that!
<danilo> jtv: can you prepare a simpler patch to just remove it from the nightly instead?
<jtv> danilo: the patch is the same, but the request to the LOSAs is different.
<jtv> I Cc'ed you on the email, and there's another link to the same request on LaunchpadProductionStatus.  You can follow up & change if you like.
<danilo> jtv: why'd you need to add start-id parameter if we just want to remove it from the nightly?
<danilo> jtv: ok, sure
<jtv> danilo: this was exactly the solution I described on the call.
<jml> :(
<danilo> jtv: yeah, sorry, I guess we misunderstood each other :)
<danilo> jtv: anyway, don't worry about it, I'll figure something out
<danilo> jtv: go home already :)
<jml> soyuz folks: why is this test failing?
<jtv> danilo: ok, ok!
 * bigjools awaits the paste
<jml> *ahem* http://paste.ubuntu.com/248735/
<barry> leonardr: ping
<leonardr> barry, welcome back
<bigjools> jml: what is makeComponent?
<bigjools> slightly rhetorical Q
<jml> bigjools, http://paste.ubuntu.com/248738/
<bigjools> jml: check the publishing status
<jml> bigjools, it's something I added to the factory, because it seemed useful.
<jml> bigjools, ahh, it'll be PENDING
<bigjools> jml: I think it's a bit bogus, you should stick to known components
<bigjools> jml: exactly :)
<bigjools> jml: return "main" by default maybe?
<jml> bigjools, well, the point is more to stop tests making implicit assumptions about components
<jml> bigjools, if it returned 'main' by default, then that would lead to accidentally tying tests to properties of main.
<jml> but it's a small thing.
<bigjools> okay
<jml> woot. I now have the failing test :)
<mrevell> Night all
<salgado> beuno!
<beuno> salgado!
<salgado> I was wondering, what will the breadcrumbs for bugs.lp.net/shipit/+bug/315110 look like?
<beuno> salgado, I need to sit down and nail it, but something like "Ship It > Bug Listing > Bug title blah bl..."
<salgado> beuno, right, that's what I was expecting. just wanted to make sure
<salgado> beuno, what about, say,   https://code.edge.launchpad.net/~spiv/bzr/inventory-delta. do you have any idea what the breadcrumbs will look like?
<beuno> salgado, someething line "Bazaar > Branch listing > inventory-delta"
<beuno> last breadcrumb is always bold, and not a link
<salgado> right
<jml> cprov, the code currently in nascentuploader's verify_acl checks permissions for ppa's before doing the binary upload shortcut
<jml> cprov, is that meaningful in any way?
<salgado> beuno, so, would it be fair to generalize the breadcrumbs for app.launchpad.net/something/sth-else to "lp.net > lp.net/something > app.lp.net/something > app.lp.net/something/sth-else (unlinked)", for a start?
<beuno> salgado, lp.net is never the root
<beuno> projects are the root
<beuno> well, maybe with the few exceptions of launchpad-wide search and such
<beuno> so the root is always lp.net/foo
<cprov> jml: let me check the code
<jml> cprov, thanks.
<beuno> salgado, or lp.net/~foo if it's not linked to a project
<salgado> beuno, oh, right, I included lp.net because the link to the home page seems to be part of the breadcrumbs
<salgado> but it's not
<cprov> jml: no, 'if self.binaryful: return' can happen before the ppa check.
<beuno> salgado, right. We want to emphazise projects rather than Launchpad
<jml> cprov, cool. thanks.
<salgado> beuno, so, is that link to the home page going away?
<beuno> salgado, yes, unless you're in a context-less page
<cprov> jml: as in "binary uploads don't need permission checks"
<jml> cprov, sure. but I don't trust comments implicitly :)
<rockstar> Holy poo poo, I just used the inline "Request a review" and it's awesome.
<cprov> jml: the comment isn't any clear about this aspect, you can amend it if you have a chance.
<jml> cprov, will do.
<salgado> beuno, ok. I now know enough about breadcrumbs to have an idea of how much work will be involved to tweak them to our needs, so whenever you want to talk more, just let me know
<beuno> salgado, how's in 30-40 minutes?
<salgado> beuno, wfm
<jml> cprov, verify_acl doesn't have any checks to do with the pocket -- why?
<beuno> cool
<jml> (apologies if I've asked this before)
<cprov> jml: because it's done in UploadPolicy.check_upload(), IRC
<jml> oh right.
 * jml remembers some more stuff
<cprov> jml: it would be nice to consolidate both check in one place
<jml> cprov, yeah, that's what I'll do.
<jml> cprov, but I've got a tested function that does what most of verify_acl does, so I'll slot that in place first before expanding its features
<cprov> jml: nice move.
 * jml is off for a bit
<sinzui> gary_poster: is the https://lpbuildbot.canonical.com/ failure one of your mad-science projects?
<gary_poster> sinzui: mthaddon's, with my assistance as possible.  sorry, we are disabling the emails.  this will become the "real" buildbot hopefully within a week though.
<beuno> salgado, ready when you are
<salgado> beuno, I'm ready
<salgado> beuno, calling now
<beuno> salgado, https://bugs.edge.launchpad.net/bzr/+bugs?search=Search&field.status=New&field.status=Incomplete&field.status=Confirmed&field.status=Triaged&field.status=In+Progress&field.status=Fix+Committed
<beuno> https://bugs.edge.launchpad.net/bzr/
<beuno> salgado, https://devpad.canonical.com/~beuno/LP_new_design_Bugs_v3.1.png
<beuno> sinzui, did your font size fix land?
<beuno> barry!!!
<sinzui> beuno: it is in the review queue
<beuno> yay
<sinzui> beuno: the fix is in the branch that adds the action menu and involvement portlet.
<beuno> sinzui, I feel everything coming together
<sinzui> beuno: me too. my project page is two branches. I'll have the view changes in review in a few hours
<beuno> I'm happy
<beuno> salgado, the project group is great
<beuno> I don't see any icons to edit things inline
<beuno> is that because you can't
<beuno> ?
<beuno> also, maybe "change details" isn't the best text
<beuno> and that should probably be the first option on that list
<salgado> beuno, I wasn't sure if the links to +reassign and +edit-maintainer should be inlined there or be the related pages from +edit.  maybe they should be in both?
<beuno> salgado, where are those now?
<salgado> nowhere
 * beuno logs in as the owner of the prokject
<salgado> oh, right now they're in the actions menu
<beuno> salgado, also, the maintainer doesn't have a person icon
<sinzui> salgado: I used Change details to get to one edit page. It has a releated-pages menu of all the other pages. The design of project and project-group should be the same
<salgado> beuno, good catch
<beuno> salgado, I don't see the reassign link at all
<salgado> sinzui, agreed, but should the links to +reassign and +driver be inlined in the project-group's +index page?
<salgado> beuno, that's the Change maintainer
<salgado> the other is Change driver
<beuno> salgado, I don't see that link on the page
<salgado> appoint driver
<salgado> really?
<sinzui> salgado: beuno: this is the same issue that EdwinGrubbs had. These can be dangerous to change. should it be that easy?
<salgado> beuno, are you looking at the new page?  in the new page it's not shown
<beuno> salgado, https://devpad.canonical.com/~beuno/Screenshot-1.png
<beuno> sinzui, ^
<salgado> beuno, I thought you were looking at the page on mainline
<salgado> a mainline branch, that is
<beuno> no no, your branch
<salgado> ok, in there it's not shown anywhere
<salgado> because I thought they should be the related pages on +edit
<sinzui> beuno: salgado: The project summary should be bigger. The .summary class in the the CSS that is in review
<beuno> YES
<beuno> agreed
<beuno> I think they should be inline
<beuno> edit icon next to the name
<salgado> right
<sinzui> salgado: beuno: this is the project screenshot that is going into review https://devpad.canonical.com/~curtis/firefox.png
<salgado> beuno, I think it's be nice to have them on the related pages of +edit as well, any objections?
<EdwinGrubbs> sinzui, salgado, beuno: I think that it would be nice to have some consistency, so we don't have some edit links on the +index page, some on the +edit page, and some inline on other pages. How do we know where to find an edit link when it is not next to the field we're editing?
<sinzui> salgado: I think we want the same titles for external links and move the rdf into the side portlet
<beuno> salgado, I think no objections, but we want to aim so that people don't need to go to that page for anything
<beuno> EdwinGrubbs, agreed
<salgado> beuno, absolutely
<sinzui> EdwinGrubbs: We can do that when YUI and lazr.js supports all browsers
<beuno> sinzui, we need to chop off long names in the download field
<sinzui> beuno: The design does not match reality
<EdwinGrubbs> sinzui: you don't need ajax to put an edit link next to a field.
<sinzui> beuno: I also do not open my browser to the size of my desktop because I know that is crack
<sinzui> EdwinGrubbs: correct that is why we do not need lots of links in the action menu. The reality is that many Change details links will go to a form that has something that is not inline.
<sinzui> EdwinGrubbs: we are encourages to move everything in an edit form into the page
<sinzui> EdwinGrubbs: Beuno: There is still a usability issue with the edit icon next to an editable value. When I place it next to a title, do I mean I am editing the title or the object?
<beuno> sinzui, I think that if we put an edit icon on the title, we should mean you can edit the title
<beuno> which means, the branch page needs changing
<sinzui> beuno: EdwinGrubbs. So every model object gets a Change details link if has just one filed that is no shown inline.
 * sinzui really has to expose the releasefileglob on series.
<beuno> salgado, also, I think we should drop the milestone portlet
<beuno> and use EdwinGrubbs's graph
<beuno> sinzui, yes
<sinzui> beuno: that is a rule that I will add to the conversion page
<sinzui> beuno: read the bugs about project milestones. Users want them, we need to fix them
<salgado> beuno, EdwinGrubbs, can I render it easily for a project group as well?  or do I need to do some work?
<beuno> sinzui, I know all about them, they want inheritance
 * beuno deflects to EdwinGrubbs 
<sinzui> They want to see them and know which project or series they pertain too
<beuno> sinzui, yeah, but that portlet is very close to useless
<beuno> so we should drop it
<beuno> and figure out what people need
<sinzui> salgado: look at launchpad project. The list is large. Every project group that I have seen in the past 3 months has more than 5  projects to list. I think we need to recognise the list will be long
<EdwinGrubbs> salgado: you would have to add a method like model.Product.getTimeline() to the project group and export it.
<sinzui> beuno: Let's separate projects from milestones and relases. The portlets have different users. I think a developer needs to see where development is happening in the project-group. We can show a portlet that represents the +milestones view that summarises the latest five mielstones and releases across all projects
<sinzui> Oh bugger, I don't have a structural subscription on the project page.
<beuno> sinzui, +1
<mwhudson> does someone want to slap the db_lp buider?
<mwhudson> (morning, btw)
<sinzui> salgado: you can add an edit link to any field value in the template using TALES menu:<facet>/<link_name>/fmt:icon after you render the value.
<barry> flacoste: hi!  been talking to leonardr about @error_status in lazr.restful.  it seems like @error_status (or webservice_error()) should register the exception, but i don't think it does
<sinzui> We can add them after driver and maintainer. I jut do not want them editable inline until we have a "shoot your self in the foot warning" for the
<sinzui> m
<salgado> oh, right.  I thought all we were talking about was having the links in the main content area
<EdwinGrubbs> sinzui: right now the $team/+edit page has edit links at the bottom of the page that are not available on the +index page. It doesn't seem like I can create an ActionMenu class for it, since it only differentiates based on facet and context, so it would show the same action menu on +edit and +index. Is it right that I should just shove that into the side slot?
<flacoste> barry: it simply annotates the exception with the status, it doesn't register anything
<salgado> sinzui, anyway, agreed on non-inline editing for these things for now
<barry> flacoste: ah.  then maybe this comment in configure.zcml is confusing me:     XXX flacoste 2008/05/09 bug=185958:
<barry>     Ideally, we would use webservice_error() to register the view,
<barry>     but InvalidBatchSizeError is in lazr.batchnavigator and webservice_error()
<barry>     can only be called from within a class definition.
<barry> flacoste: so, basically, i need to copy that chunk for my exception i guess
<flacoste> barry: hmm, hang on
<flacoste> barry: you are right, it does register the exception view
<flacoste> barry: so there are two possiblities here
<flacoste> 1) webservice_error isn't being processed (is it in a module that is introspected for annotations)
<flacoste> 2) (more likely) you're zope integration publication doesn't provide an adequate handleException implementation
<flacoste> in Launchpad, we rely on the default ZopePubliation implementation
<barry> 1) definitely is because it's in an interface module that has other annotations
<flacoste> if you look at our test publiaction, you'll see that we provide a lightweight implementation of handleException that looks up the view
<barry> 2) is a possibility.  i should try ripping out my specialized stuff and use leonardr's new wsgi stuff
<flacoste> leonardr told me he's having the same problem
<flacoste> so it probably wasn't copied over to the wsgi base stuff
<barry> interesting
<sinzui> salgado: I think we need some easy way to see that pillars used the same portlets. I see you have blueprints and I do not. I have answers, I do not think you do. I created FAQs
<barry> flacoste: yep, i see it in lazr/restful/publisher.py.  my publisher is definitely not doing the view lookup.  i'll bet that's it
 * barry hacks
<EdwinGrubbs> sinzui: the action links on the +edit, +reassign, +branding, +edithomepage, etc. are currently displayed on all the edit pages, which is similar to how it was with the second level tab bar. I assume only the +edit page should have these links from now on.
<flacoste> barry: can you reply to your post? sorry i didn't get to it, but i was away from it all
<flacoste> for the past two weeks
<barry> flacoste: sure; if i get it working :)
<sinzui> EdwinGrubbs: right, That is how I know I had to put the NavigationMenu back. Only the edit pages use the menu. I used view/+related-pages to render it. Also I used a mixin for the nav menu and the facet menu to ensure shared links have the same title and icon.
<sinzui> EdwinGrubbs: salgado: beuno: We might want a rule where all the links are defined in a mixin and the implementing menu classes just select the links they need.
 * sinzui used that rule as the example he sent in the navigation email.
<salgado> sinzui, that sounds like a good idea
<sinzui> salgado: My next controversial suggest is to create browser/pillar.py  and define common menus and and views.
<EdwinGrubbs> sinzui: is the @@+global-actions and @@+related-pages framework in trunk? I assume that those views need to be registered somehow that your example doesn't show.
<sinzui> @@+related landed last week. @@+global-actions is in review now. You can review them if you like
<sinzui> EdwinGrubbs: My navigation menu email gives more than examples, I divulge how to register a menu without ZCML.
 * sinzui has kept that a secret for 14 months
<EdwinGrubbs> sinzui: I saw how you registered the menus without using <browser:menus> but doesn't there still need to be a <browser:page name="+global-actions"> somewhere?
<sinzui> EdwinGrubbs: it is in the branch in review.
<EdwinGrubbs> cool
 * sinzui looks for branch
<thumper> morning
<barry> leonardr, flacoste i just sent an update.  see the two little nits i still have.  but it's pretty much working.  thanks for your help, and i'm off irc now! :)  if you have any insights please respond to the thread.
<leonardr> barry, looking forward to it
<flacoste> ack
<mwhudson> thumper: skype?
<thumper> yep
<mwhudson> cool
<thumper> just getting off with curtis
<mwhudson> ah cool
<thumper> abentley, rockstar: pign
<rockstar> thumper, I see you calling, but I can't answer.
<thumper> rockstar: why?
<rockstar> thumper, skype sucks?
<rockstar> I said can't, not won't.  :)
<rockstar> thumper, try again please.
<abentley> thumper: Sorry.  Here now.
<wgrant> Huh. Somebody just suggested that all the apps should be removed except Bugs, I think.
<mwhudson> would make my life easier
<mwhudson> maybe?
<abentley> thumper: bzr+ssh://bazaar.launchpad.net/~abentley/launchpad/related-bugs
<mwhudson> spm: hi, can i get you to cowboy a branch on to tellurium for me?
<spm> mwhudson: sure
<mwhudson> spm: i really want to test this today, i'll have entirely forgotten how this works in a fortnight :)
<mwhudson> spm: what's easiest for you?  a branch url, a diff ?
<spm> mwhudson: :-) patch - by far
<spm> mwhudson: codehost? or browse?
<mwhudson> spm: codehost
<mwhudson> spm: http://pastebin.ubuntu.com/248903/
<mwhudson> spm: i guess i'd like you to disable the update that would otherwise wipe this out in 30 minutes too :)
<spm> mwhudson: aye, done.
<mwhudson> spm: thanks
<spm> mwhudson: patched and restarted; rewrite hup'd?
<mwhudson> spm: nah, it's not in that area
<spm> cool
 * mwhudson bounces buildbot
<mwhudson> spm: can you sync the puller.log file to devpad?
<mwhudson> but things seem to be working \o/!
<spm> mwhudson: is syncing atm
<mwhudson> spm: ta
<spm> done
<mwhudson> grr, why isn't buildbot building !?
<rockstar> thumper, call?
<mwhudson> spm: can you see if there's a mirror-branch.py process on tellurium?
#launchpad-dev 2009-08-07
<thumper> rockstar: 2 minutes
<wgrant> Who is this Green Dragon character, and what is with their bugs?
<rockstar> wgrant, link?
<wgrant> rockstar: Lots of bugs against launchpad.
<wgrant> Some with a further 20 comments by this same person.
<beuno> yeah, made me feel stabby
<wgrant> It's not exactly coherent either.
<ajmitch> I see what you mean, additional comments every 2-3 minutes on some of those bugs
<wgrant> 10 comments on bug #410054, eg.
<mup> Bug #410054: From "bugs.edge.launchpad.net" their is no back link to "launchpad.net" <Launchpad itself:New> <https://launchpad.net/bugs/410054>
<ajmitch> https://bugs.edge.launchpad.net/~great-wyrm-green-dragon
<ajmitch> bug 410076 is a bit odd
<mup> Bug #410076: Maybe one could add a wiki in the code section so the code could be edited more easily? <Launchpad itself:New> <https://launchpad.net/bugs/410076>
<wgrant> Yes.
<wgrant> That's the one I was referring to when I said that somebody had recommended that all apps except Bugs be dropped.
<wgrant> Oddly, the account is nearly two years old.
<spm> mwhudson: no mirror-branch process running atm; wasn't earlier either if that helps?
<mwhudson> spm: can you sync the logs again?
<spm> mwhudson: syncing...
<spm> done
<mwhudson> spm: can you mirror oopses too?
<mwhudson> spm: also from asuka?
<spm> mwhudson: that was everything, so yes; asuka, will do.
<maxb> A conundrum.... what to do with doctests which will output different styles of output run with python2.4 vs. python2.5 ?
<mwhudson> spm: can you sync the access log for asuka too?
<wgrant> A bit of ellipsis hopefully solves that.
<wgrant> maxb: How different?
<mwhudson> (that seems rather out of date, maybe i'm looking in the wrong place though)
<spm> mwhudson: ah. i did bzrsyncd, not lp
<mwhudson> err
<mwhudson> has sodium just fallen over?
<spm> yup
<mwhudson> hooray
<wgrant> sodium is devpad?
<mwhudson> yes
<mwhudson> spm: we disable mirrored branches somehow on staging
<mwhudson> spm: do you remember how?
<spm> mwhudson: update set branchs == no mirroring; was moderately simple istr.
<spm> mwhudson: it's in the staging update for codehost... errr. one sec.
<mwhudson> spm: i'm reasonably sure that's not valid sql :)
<spm> it wasn't meant to be :-)
<spm> mwhudson: part of staging restore: UPDATE branch SET next_mirror_time = NULL WHERE branch_type = 2 AND id NOT IN (SELECT branch.id FROM branch, product WHERE branch.product = product.id AND product.name = 'bzrtools');
<mwhudson> spm: ta
<mwhudson> spm: can you sync the puller.log again?
<mwhudson> spm: preferably without killing devpad again :)
<spm> oh you developers and your pony requirements
<spm> is done
<maxb> wgrant: py2.4 tracebacks call anonymous modules '?', py2.5 says '<module>' that's just one of many similar issues
<spm> mwhudson: btw, i've re-enabled that update for codehost - not that I suspect you'll care... ;-)
<mwhudson> spm: i hope not
 * mwhudson curses
<wgrant> maxb: Ellipsis should solve that one easily.
<mwhudson> hooray for QA
<sidnei> wgrant, maxb: there's a way to use regexes for normalizing tracebacks, search for 'RENormalizing'
<wgrant> sidnei: Ah, useful.
 * wgrant will fix some of the failing tests later.
<sidnei> http://svn.zope.org/zope.testing/trunk/src/zope/testing/renormalizing.py?rev=84340&view=markup
<sidnei> maxb: there's an example here that deals with exceptions differences: http://svn.zope.org/zope.testing/trunk/src/zope/testing/testrunner/tests.py?rev=100712&view=auto
<maxb> yikes, how involved. ok then.
<maxb> The other thing I'm rather concerned about is the amounts of complaints about tests leaving garbage I'm seeing
<maxb> also, can anyone explain a bit about LayerIsolationErrors complaining about garbage?
<mwhudson> spm: can you apply another patch on top of the one i gave you earlier?
<spm> sure
<mwhudson> spm: http://pastebin.ubuntu.com/248930/
<spm> mwhudson: patched and restarted
<mwhudson> spm: thanks
<mwhudson> (restart wasn't necessary, can't hurt though :)
<spm> oh? oki.
<spm> being in twisted I just assumed...
<mwhudson> woo, works!
<mwhudson> spm: can you sync the log again?
<mwhudson> puller.log that is
<spm> and I suppose you wnat a crashless sync again?
<mwhudson> if poss
 * spm rolls eyes at the impossible demands some folks have
<spm> mwhudson: done
<mwhudson> thanks a lot
<thumper> edge seems extremely sluggish
<wgrant> thumper: Didn't I see something in the production meeting about high load and lots of timeouts?
<thumper> yeah
 * mwhudson off -- please don't send me too much email while i'm gone :)
<Ursinha> mwhudson, good luck :)
<noodles775> Hi
<mrevell> Hi
<deryck> Morning, all.
<jtv> hi deryck, and hi mrevell
<mrevell> Hi hi
 * jml waves
<mrevell> jtv: the term "translation domain" is confusing me a tad.
<mrevell> jtv: wrt naming templates.
<jtv> mrevell: "name" is the name we use in URLs etc.  "domain" is what's used in filenames on the end-user's system
<henninge> mrevell: translation domain is a term used by the gettext system
<mrevell> jtv: Is that terminology that we've invented or will be apparent to most readers of the user guide?
<mrevell> ah right thanks
<jtv> mrevell: translation domain is a standard term
<mrevell> cool, thanks chaps
<danilos> jtv: ping when you are alive again
<henninge> danilos: I have 30 min now or how ever much you like at abt. 13:45
<henninge> our time
<danilos> henninge: let's do it now then
<danilos> henninge: ok, I'll do a separate call with jtv, and then see how quickly we're done
<henninge> danilos: np
<jml> http://paste.ubuntu.com/249155/ -- anyone know what this error actually means?
<wgrant> lazr.restful doesn't treat ForbiddenAttributes like Unauthorizeds -- it OOPSes instead. Intentional?
<jml> couldn't say.
<wgrant> Quite a few existing exports suffer from that.
<danilos> henninge: https://pastebin.canonical.com/20914/
<sinzui> I think PQM hates BjÃ¶rk.
<intellectronica> for once, i feel empathic to PQM :)
<jml> allenap, thanks for commenting!
<allenap> jml: No worries :) It's interesting.
<jml> anyone have any idea what might cause this error message: http://paste.ubuntu.com/249155/
<mwhudson> wgrant: ForbiddenAttribute is more like AttributeError than Unathorized
<wgrant> mwhudson: True.
<noodles775> jml: I've got vague memories of seeing that before... can you paste your test if it's new?
<wgrant> (I hate that)
<jml> noodles775, it's not new :(
 * noodles775 checks...
<jml> noodles775, but I've changed the underlying code substantially
<jml> noodles775, actually, I think I've got a little closer to figuring it out
<noodles775> jml: yeah? What was it? (I just saw the big XXX in store.py
<jml> noodles775, it's an ordering thing...
<jml> noodles775, don't have the exact details, I'm in a talk now and can't type as loudly as I need to think right
<noodles775> :) np.
<flacoste> is it just me or edge is slow?
<beuno> flacoste, it's been slow for me since yesterday
<flacoste> beuno: well, there is slow and slow
<beuno> I get the feeling it's not that pages are slow to generate, but the connection being slow
<beuno> flacoste, "extra slow"
<flacoste> "Connecting to bugs.edge.launchpad.net"
<flacoste> sitting there for 3 minutes now
<beuno> sometimes I've had to hit a page twice to actually get it
<flacoste> and counting
<bigjools> slashdotted?
<flacoste> same thing for wiki.canonical.com
<flacoste> or dev.launchpad.net
<flacoste> herb, mthaddon: any idea of the above?
<herb> flacoste: no idea. everything seems to be operating properly
<flacoste> glad to hear!
<flacoste> it might be a connectivity issue with my provider, let me traceroute
<flacoste> what is the greatest tool in term of tracerouting nowadays?
<herb> mtr is still the way to go
<flacoste> hmm
<flacoste> no loss
<flacoste> 14 hops
<flacoste> 120ms to vanadium
<flacoste> hmm
<flacoste> works fine from wget / openssl
 * flacoste restarts firefox
<flacoste> seems to be a problem with all SSL sites on my side :-(
<flacoste> don't you love system upgrades
<bigjools> jml: thanks for the feedback emails, I'm thinking about them
<jml> bigjools, np. I hope they help.
<bigjools> they will
<sinzui> beuno: ping
<beuno> sinzui, hi
<sinzui> Upgrading portlets is awkward. for example to make Top Contributors work for 3.0 on the project page, the distro page gets a floating link
<beuno> sinzui, floating link?
<jml> ok
<jml> have deciphered our Branch -> Archive diagram, time to wikify it.
<jml> if I refer to the sun god at any point, it probably indicates that I've misinterpreted some of the runes
<sinzui> beuno: The see _More contributors_ link is in the header, in a 2.0 page. So my dilema is A) ignore the problem because the distro will be updated next week, B) write code to run two implementations at the same time. C) add 2,0 and 3,0 CSS hacks that hide eachother's special layout rules.
<beuno> sinzui, this is just to drop that link for a week?
<sinzui> right
 * sinzui makes screencap
<beuno> sinzui, drop it for a week, be happy
<sinzui> beuno: https://devpad.canonical.com/~curtis/crack.png
<beuno> sinzui, that's 2.0 now?
<sinzui> beuno: that is what will happen on the distro page if I land the project page changes.
<beuno> sinzui, drop that link for distro until next week
<beuno> don't sweat it
<beuno> BjornT, the dupe finder hates me and doesn't want me to file bugs
<beuno> is that issue on your plate?
<sinzui> beuno: sorry. I do not understand. The portlet is shared. Either the link is always there or it is never there
<beuno> sinzui, I see. Drop the link for a week, don't wasate time hacking around it for so little time
<sinzui> okay
<sinzui> EdwinGrubbs: I am removing the latestquestions portlet from the team page. It broke in my branch, then I realised is never worked...team cannot ask questions.
<EdwinGrubbs> ok
<jml> I've just requested a mailing list on staging, can I please get someone to approve it?
<jml> oh wait, I can.
<beuno> sinzui, could you point me at a project that the product release finder imports automatically?
 * sinzui thinks
<sinzui> sidnei: ping
<sinzui> sweet. I still have a log
<sinzui> beuno: You can see when the prf got fixed in this project:https://edge.launchpad.net/vpnc
<sinzui> beuno: This project uses the prf, but since it did not work, they have manually made the releases: https://edge.launchpad.net/netrek-archive
<beuno> sinzui, thanks
<sinzui> beuno: the log lists the projects first, then starts downloading. it is hard to see which download is for which project and series
<beuno> sinzui, right, I'm trying to see if this is good enough for us to know if we're shipping a version in ubuntu which isn't upstream's latest
<sidnei> sinzui: yo
<bigjools> see y'all, have a nice weekend
<abentley> flacoste: do you have a few minutes to chat?
<mrevell> Good weekend all
<sinzui> Hi sidnei I believe you have projects that set the filereleaseglob to automatically download releases from another site
<sidnei> sinzui: i gave it a try on staging, but had trouble with it, two bugs were filed
<sinzui> sidnei: I think you still have it setup in production. I think that because I think I saw you listed in the log.
<sinzui> sidnei: I will try to fix a few of the reporting bugs related to in before 3.0
<sidnei> sinzui: uhm, that would be weird, i don't remember doing anything in production
<sinzui> It may have been a coincidence that your were testing on staging in June/July when I was running tests
<maxb> Hi, can anyone explain some background about LayerIsolationExceptions complaining about garbage?
<intellectronica> have a lovely weekend, everyone!
<sidnei> sinzui: so... do you need any feedback/help debugging or anything?
<flacoste> abentley: i'm back, skype?
<salgado> maxb, sure, what's up
<sinzui> sidnei: I was curious if it found any new files for you.
<maxb> In my Python 2.5 testing I'm getting lots of messages about garbage from individual tests, and the first test in many layers dies with a LayerIsolationError complaining about garbage
<maxb> I'm basically looking for a nudge in the right direction of what to look for.
<sidnei> sinzui: so. i tried it in staging for the zope2 project yesterday (or two days ago), there was no visible indication that it found any files or that it failed in anyway, this is one of the bugs that i reported
<sidnei> sinzui: whatever i did in june/july completely escapes my memory right now :)
<sinzui> sidnei: me to.
<salgado> maxb, IME that usually means a layer has been setUp() but not torn down in the correct order
<maxb> mhm
<maxb> LayerIsolationError: Test left uncollectable garbage
<maxb> [<bzrlib.smart.medium.SmartSimplePipesClientMedium object at 0x10de8ad0>] (referenced from [(<bzrlib.smart.medium.SmartSimplePipesClientMedium object at 0x10de8ad0>,), [<bzrlib.smart.medium.SmartSimplePipesClientMedium object at 0x10de8ad0>], {'_state': 'reading', '_medium': <bzrlib.smart.medium.SmartSimplePipesClientMedium object at 0x10de8ad0>}])
<abentley> flacoste: back too :-)
<beuno> deryck_!
<deryck_> beuno!
<beuno> deryck, what's my daily update on awesome inline description editing?
<beuno> landed?  :)
<deryck> beuno, :)  Just waiting for you to ask :)
<deryck> beuno, very close... which I guess is my standard response now. :)
<deryck> beuno, I have three minor issues on the lazr-js branch to finish and I'm ready to put it up for code review.
<beuno> deryck, so I shouldn't make any fun plans for the weekend, as I will be able to play with descriptions all night long?
 * beuno increases the pressure
<deryck> heh, I think you should enjoy your weekend :)
<beuno> well dodged
<deryck> beuno, I have 3-4 hours of work to be done on the lazr-js branch, and I'll keep at it until it's done....
<beuno> deryck, thanks man
<rockstar> abentley, let's do that standup thing
<abentley> rockstar: roger
#launchpad-dev 2009-08-08
<wgrant> Oh dear.
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/export-archive-dependencies
<wgrant> I broke something.
<wgrant> I think I did start a push and then quickly Ctrl+C, but still...
<wgrant> (a subsequent push succeeded, but it's still broken)
<mthaddon> wgrant: push --overwrite doesn't fix it?
<wgrant> mthaddon: I doubt it, but I might try. The hosted copy of the branch is fine; the mirrored one is broken.
<wgrant> --overwrite shouldn't do anything useful here.
<mthaddon> you probably know way more about it all than I do, so I'll just leave you to it :)
<wgrant> I'm not sure how to force it to do a full remirror except by a format change, and that's not possible with 2a.
#launchpad-dev 2009-08-09
<dobey> how do i make launchpad not mail me about bugs/branches/etc... that i am subscribed to?
<beuno> dobey, you can't
<dobey> :(
<thumper> morning
<cszikszoy> Is this the right channel to ask a question about building and running LP?
<thumper> yes
<cszikszoy> I've followed the guide, but am getting stuck at $ make schema
<cszikszoy> It asks me to run utilities/link-external-sourcecode, but that errors saying the parent branch wasn't specified and couldn't be discovered
<thumper> hmm..
<thumper> I've not read our new guide
<thumper> can you point me to the page you've been following
<thumper> ?
<cszikszoy> https://dev.launchpad.net/Running
<cszikszoy> I'm doing this on a clean VM of jaunty, fully updated, so I don't think there should be any sort of conflicting configuration issues.
<thumper> can you pastebin the following: du -h --max-depth=2 ~/launchpad
<cszikszoy> sure, http://paste.pocoo.org/show/133342/
<thumper> cszikszoy: which directory are you running make schema in?
<cszikszoy> devel
<thumper> cszikszoy: which directory were you running link-external-sourcecode from?
<cszikszoy> same dir
 * thumper looks at the script
<thumper> cszikszoy: try this: ./utilities/link-external-sourcecode ../../lp-sourcedeps
<cszikszoy>   Wanted to link ../../lp-sourcedeps/download-cache to ./download-cache but source does not exist
<cszikszoy> I tried giving a couple of different paths to link-external-sourcecode, but still no luck
<thumper> damn
 * thumper thinks some more
<thumper> btw, I don't actually use any of these scripts
<thumper> too set in my old ways
<cszikszoy> :)
<thumper> ok
<thumper> you are missing the download cache in the lp-sourcedeps
<thumper> but the rocketfuel-get script puts it there
<thumper> so
<thumper> ./utilities/rocketfuel-get - from the devel directory
<cszikszoy> I get a permission denied (publickey) error, but I think I remember reading in the script that that was OK
<thumper> um
<thumper> does lp have your public ssh key?
<cszikszoy> no
<thumper> it needs to
<thumper> you should care if you get  permission denied (publickey)
<cszikszoy> hm, I figured I didn't need it because I wasn't writing anything to LP, just branching, and I saw this: http://paste.pocoo.org/show/133347/
<beuno> hi thumper
<thumper> hi beuno
<thumper> cszikszoy: hmm
<thumper> cszikszoy: have you done a bzr launchpad-login before?
<cszikszoy> thumper, no, but rocketfuel-setup asks for it
<thumper> cszikszoy: if you run `bzr launchpad-login`, what does it say?
<cszikszoy> just tried, it shows my lp name, same one I gave to rocketfule-setup
<thumper> right
<thumper> this is why it is complaining
<thumper> if you've told bzr your launchpad login, it tries to get things over bzr+ssh
<thumper> and since it doesn't know your ssh public key it fails
<thumper> you have to either:
<thumper>  * tell LP your ssh public key
<thumper>  * tell bzr to forget your launchpad-login
<cszikszoy> I see.  Well, I've made a ssh key for this vm, and given LP this pub key for my lp-login
<cszikszoy> Rerunning rocketfuel-get right now, and it looks like it's branching something
<cszikszoy> great.  Looks like it's creating the "download-cache" directory in lp-sourcedeps.
<cszikszoy> thumper, thanks for the help.  It appears to be downloading the sourcedeps right now, so hopefully everything will work this time.
<thumper> cool
<thumper> spm: ping
<spm> thumper: heyo
<thumper> spm: what's the status of the svn patch for the imports?
<wgrant> Ahhhh.
<wgrant> I wondered why the import machines were gone.
<thumper> spm: we have halted the git imports for no reason that makes sense to them
<wgrant> I should have realised.
<thumper> wgrant: :)
<thumper> spm: you looking or ignoring?
<elmo> thumper: he's hating
<elmo> thumper: on you
<thumper> elmo: :)
<thumper> elmo: how you doin?
<elmo> thumper: frazzled
<spm> thumper: was looking :-) am starting up the importds now
<thumper> spm: fantastic
<spm> tho... code.aundpad is a nice fail...
<thumper> ok, with that, I'm taking the car to get a warrent#
<spm> thumper: should be all systems go; am logging in to same to verify
<thumper> spm: cool
<spm> thumper: all looks green
#launchpad-dev 2010-08-09
<thumper> wgrant: you just have to poke don't you?
<thumper> wgrant: messing with whiteboards
<lifeless> wgrant: what awas that oops you mentioned earlier ?
<wgrant> thumper: That was my "Ahem, oops.", yes. Forgot I wasn't on staging.
<wgrant> (also, I really shouldn't be able to write to that whiteboard at all)
<wgrant> lifeless: OOPS-1681S1580?
<wgrant> Hm. No (LO|G)SAs during release week. Great.
<thumper> :)
<thumper> there should be someone interruptable
<thumper> but you are right
<thumper> no one on hand right now
<thumper> mwhudson: can you have member functions be function decorators?
 * thumper thinks
<mwhudson> thumper: what does that mean?
<thumper> I'm trying to clean up some test code
<thumper> but with a function decorator, I don't need the self.addCleanup I guess
<lifeless> thumper: self.addCleanup is generally a good thing
<lifeless> thumper: can you describe a bit more, or pastebin an example ?
<thumper> sure
<thumper> liefhttp://pastebin.ubuntu.com/475182/
<thumper> lifeless: http://pastebin.ubuntu.com/475182/
<lifeless> stab stab stab opening
<lifeless> openind
<thumper> lifeless: we repeatedly have the lines 4-9 in a number of tests
<thumper> I wanted to have a simple test function decorator to do the config push and pop
<lifeless> you could
<lifeless> but I'd put it in a fixture
<lifeless> using installfixture
<thumper> example somewhere?
<lifeless> class BuilderConfig:
<lifeless>    def setUp(self):
<lifeless>     bzr_builder_config = """
<lifeless> ...
<lifeless> def tearDown(self):
<thumper> does the fixture need a tearDown?
<lifeless>     config.pop("bzr_builder_config")
<lifeless> self.installFixture(BuilderConfig())
<thumper> now a question would be, is it worth making a fixture, or just have a method?
<lifeless> fixtures can be used by different test classes
<lifeless> so yes, its worth it
<lifeless> wgrant:
<lifeless> Module lp.code.model.diff, line 73, in text
<lifeless>     self.diff_text.open()
<lifeless>   Module canonical.launchpad.database.librarian, line 110, in open
<lifeless>     self._datafile = self.client.getFileByAlias(self.id, timeout)
<lifeless>   Module canonical.librarian.client, line 349, in getFileByAlias
<lifeless>     raise LookupError, aliasID
<lifeless> LookupError: 52704649
<wgrant> lifeless: Yeah, that makes sense. Thanks.
<lifeless> \o/ cleanup part 2 done.
<lifeless> poolie: around ? Would like to land your flags branch - have you looked at the failures from friday, or should I do that ?
<thumper> wgrant: what do we do with stuck builders?
<thumper> what sort of privledges do you need?
<lifeless> take em out back and shoot em
<mwhudson> thumper: isn't there a pushConfig on lp.testing.TestCase already?
<mwhudson> i know i've written it somewhere...
<thumper> mwhudson: you know, I think you might be right
<thumper> lifeless: do you have a rifle?
<thumper> WTF?
<thumper> admins don't have launchpad.Special for IPerson?
<wgrant> thumper: It needs a lamont.
<wgrant> thumper: Which builder?
<wgrant> thumper: That's the point of .Special. Admins don't have it.
<wgrant> That is its entire purpose.
<wgrant> Why it's used, I don't know.
<thumper> wgrant: see #launchpad for stuck builders
<wgrant> Only for SSH keys, AIUI.
<thumper> wgrant: and account deactivation
<wgrant> Craaap, not terranova.
<wgrant> It's not meant to die.
<wgrant> Nothing that can be done without a GSA.
<wgrant> And lamont, normally.
<thumper> wgrant: are the build jobs actually instantiated by the build master before dispatching to a builder?
<thumper> wgrant: I'm wanting to oops on a recipe for a deactivated person before it gets to the actual builder
<wgrant> thumper: Yes -- the builder can't talk to the DB.
<wgrant> The master has to do everything.
<thumper> ah ha
<thumper> so the recipes are taking out the build master?
<wgrant> Yes.
<wgrant> -> the entire farm.
<thumper> and it can't handle exceptions?
<wgrant> Anything outside lib/canonical/buildd runs on the master.
<thumper> that's a bit poor
<wgrant> It is somewhat.
<wgrant> But it's not that bad.
<thumper> ORLY?
<thumper> wgrant: so where is the code in the buildmaster that is dying?
<thumper> when the job is barfing on no email?
<wgrant> thumper: In recipebuilder.py, where I showed you before. Or do you mean what calls taht?
<thumper> I've fixed the recipe owned by team with no email part
<wgrant> How have you fixed that?
<thumper> now looking at the possibility of a person deactivating themselves after a recipe job has been created
<thumper> I use the recipe registrant instead
<thumper> and say email sent on behalf of the team
<thumper> in that edge case
<thumper> I want the job to blowup earlier though in the situation where we can't get a preferred email for the team or registrant
<thumper> which means an oops
<wgrant> Erm.
<wgrant> Sent on behalf of the team?
<thumper> which means that the code has to handle it
<thumper> as the sender
<wgrant> The crash is not sending an email.
<wgrant> It is producing an email address to be used in the changelog.
<wgrant> brb
<thumper> oh...
<thumper> hmm...
<thumper> best fix that then...
<thumper> well, setting the author_name to "Eric the Viking (on behalf of Vikings team)"
<thumper> still seems to make sense to me
<thumper> for a build author
<thumper> as it should be the team, but there is no email address set
<thumper> although, I'm tempted to walk up the team.owner chain until we hit something with a preferred email
<thumper> as we know we'll get someone eventually
<thumper> as disabled accounts aren't part of teams
<lifeless> thumper: there is a failure mode there ;)
<thumper> lifeless: which is?
<lifeless> thumper: two in fact - do we permit circular team ownership
<thumper> a team with one disabled member?
<lifeless> team A owns team B owns team A
<thumper> I'm pretty sure we have DAGs for owners
<lifeless> secondly, you may run up a team list until you find mark :)
<thumper> lifeless: so your solution would be?
<lifeless> what you're doing
<lifeless> check that there is a DAG constraint is all
<thumper> what I'm doing is to use the registrant of the recipe
<lifeless> an alternative would be to say 'source-recipe-builder@launchpad.net'
<thumper> which could be a disabled person
<lifeless> I'm pretty sure that soyuz does something like that for the actual buildd
<thumper> hmm... I reckon we use the recipe owner first, and if that isn't valid, use the @lp.net email
<wgrant> Back.
<wgrant> Using the owner or registrant seems wrong.
<wgrant> I'd construct some special email address.
<wgrant> Like Bugs does.
<wgrant> Lying about the email address is less bad than lying about the name.
<wgrant> s/owner/team owner/
<lifeless> I think having a dedicated email to represent the things lp did makes sense
<lifeless> its a little odd what ppa signing keys do, which is a similar situation
<wgrant> How's what they do odd?
<lifeless> they claim to be me
<wgrant> They say 'Launchpad PPA for Somebody'
<wgrant> No email address.
<lifeless> hmmm
<lifeless> I remember a bug from somewhere
<wgrant> There is the great Bugs discussion.
<wgrant> But that seems to have been resolved fairly reasonably.
<lifeless> wow, I'm nearly at the end of my code queye
<lifeless> bwah
<lifeless> queue
<wgrant> Oh?
<lifeless> two branches to go
<lifeless>  /participants timeouts
<wgrant> Ah, right.
<lifeless> and ffeature fflags
<lifeless> I've got a tagger that alerts about deployable revisions
<lifeless> we need to do something to make that releasable
<wgrant> Is the new system going to be open source?
<lifeless> yes
<lifeless> the sticking point here is a filein the history that maps emails <-> lp ids
<lifeless> which maybe indexing is a bit nuts
<wgrant> Hmm.
<lifeless> so we may need to restart the history with a blank file, and have the deployed version have that file.
<lifeless> or something.
<wgrant> It can't use the API?
<lifeless> ENOIDEA
<lifeless> anyhow, if we get it open
<lifeless> with a blank file
<lifeless> you can fix it to not need the file ;)
<wgrant> Heh.
<lifeless> [I don't know any of the history or design of it - I spent a day hacking and rearranging, so I know some of it, but that bit I didn't touch]
<wgrant> If it's anything like Launchpad, we probably don't want to know the history....
<lifeless> well
<thumper> wgrant: do you thing the build author should be a special LP email all the time?
<wgrant> thumper: No.
<thumper> or just when the owner doesn't have an email set
<wgrant> thumper: Yes.
 * thumper nods
<wgrant> Or possibly when it's private.
<wgrant> Like Bugs now does.
<wgrant> But that's not clear.
<wgrant> brb
 * thumper afk for kid collection
<lifeless> jkakar: around ? I has a storm question on left outer joins in tuple-result sets
<lifeless> the question is, are NULL matches represented as None, or Undef, or something else ?
<lifeless> also
<lifeless> is there a result set decorator already existing ?
<lifeless> if not, any suggestions where I'd put it
<wgrant> lifeless: You mean DecoratedResultSet?
<lifeless> possibly
<lifeless> where is that
<wgrant> c.l.components.decoratedresultset
<lifeless> wgrant: I want a thing that will handle a tuple-query, answer count() and the like, but call into some adaption code to support first(), any() and iteration
<lifeless> plus copy config etce tc
<wgrant> I believe that does what you want.
<lifeless> ok cool
<lifeless> brilliant
<lifeless> thumper: ^
<lifeless> thumper: your code query-tuning stuff that uses a list should use that
<lifeless> wgrant: yup, exactly what I needed
<thumper> lifeless: why?
<lifeless> thumper: listification starts to perform badly when you exceed batch_size
<thumper> I don't listify anything bigger than the batch
<lifeless> I may be misremembering
<lifeless> sec
<lifeless> thumper: garh, lost the reference; what was the thing you tuned ?
<thumper> lifeless: I've tuned lots of things :)
<lifeless> heh
<lifeless> ok
<thumper> I'm a tuning machine
<lifeless> so you had something that was doing potato queries
<lifeless> and you pointed me at it
<lifeless> it was, perhaps, an API
<lifeless> ah no
<lifeless> it was the thing you were created a cached decorator for
<lifeless> ah, linked_bugs is one example
<lifeless> it listifies, it would be better if it was built on DecoratedResultSet, though its also got the challenge that it does permission checks late
<lifeless> _getRevisionsSinceReviewStart is another candidate for pushing more work to the db
<lifeless> thumper: I'm just noting places we can do better - the code pages perform well, as you know :)
<thumper> lifeless: not as well as I'd like
<lifeless> thumper: well, tomorrow is performance day! if you want we can pair on analysing/tuning
<thumper> I've got two^Wthree urgent things to get landed
<lifeless> ah well
<lifeless> how does one do subqueries in storm ?
<lifeless> [want to get coc signing with a person query
<lifeless> also
<lifeless> can anyone confirm the Class.q.column syntax is unneeded now, with storm ?
<thumper> lifeless: Select(...)
<thumper> store.find(..., self.some_id.in.(Select(...)))
<thumper> actually, it is 'is_in'
<thumper> I think Class.q.column isn't needed
<thumper> personally I never used it
<lifeless> ok, so thats a subquery as a condition
<lifeless> to use it as a result
<lifeless> store.find((Person, Select()), conditions) ?
<lifeless> select person.id, exists (select * from signedcodeofconduct where signedcodeofconduct.owner = person.id) as signed_coc from person <- what I want
<thumper> Exists is a subquery I think
 * thumper looks at some code
<thumper> hmm...
<lifeless> right, nearly got it
<lifeless> just need to get the exists into the where clause
<thumper> lifeless: are you wanting it in the where clause
<thumper> or the select cols?
<lifeless> >>> print [(p[0].id, p[1]) for p in store.using((Person,)).find((Person, Alias(Exists(Select(SignedCodeOfConduct, SignedCodeOfConduct.owner == Person.id, SignedCodeOfConduct)), name="signed")), And(Person.id >= 16, Person.id <= 17))]
<lifeless> [(16, True), (17, False)]
<lifeless> now to make it pretty
<lifeless> thumper: select cols, I knew what I meant :P -sorry for the confusion I'm sure I engendered
<thumper> lifeless: yeah, I just couldn't think of anything else...
<poolie> thumper: this sounds interesting
<poolie> is this for a graph or for a feature in lp?
<lifeless> poolie: hi, see up above re ff
<poolie> hi lifeless, i saw the features
<poolie> i spent friday afternoon trying to work out why my lp dev instance was failing, and then getting a chroot up
<poolie> i'm going to have another go at it now
<lifeless> poolie: ok cool.
<lifeless> poolie: if I can help, please shout.
 * lifeless goes back to bashing up storm stuff
<thumper> lifeless: you can go find((Person.id, ...), ...)
<thumper> lifeless: to stop getting all of the people columns returned too (and chewing up non-sql time)
<poolie> so lifeless, what are you doing with coc signing?
<lifeless> thumper: we want all the people columns
<lifeless> poolie: improving the performance of a registry API which is the top OOPS at the moment
<poolie> cool :)
<lifeless> thumper: thanks for the hint though - will keep it in the toolkit
<lifeless> poolie: this is is the go-back-and-do-single-query pathology I was talking about
<lifeless> down to 23 queries to generate the page for a team with 3 members ><
<wgrant> lifeless: What's trying to find CoC sigs?
<lifeless>  /participants
<wgrant> Hm.
<lifeless> because is_ubuntu_coc_signer is a property
<lifeless> just don't say that out loud
<wgrant> Ahh.
<lifeless> wgrant: you can look at my 'registry' branch if you want to see the arc
<wgrant> That name concerns me.
<wgrant> thumper: What's the error in the #lp OOPS?
<mtaylor> lifeless: how ludicrous of a feature request would it be to add "pastebin url" to the list of metadata info kept for a project? (you know, in there in the laundry list of all the other urls it keeps)
<thumper> AssertionError: Wrong account! 2401541, 3228204
<thumper> wgrant: ^^
<lifeless> mtaylor: patches considered
<wgrant> thumper: Ah, he merged his person, most probably.
<mtaylor> lifeless: ok
<wgrant> I believe that's scheduled for the next release or two.
<lifeless> so, API question
<thumper> mtaylor: add the description to the project
<lifeless> there is a property
<lifeless> 'archive'
<lifeless> which is an exported Reference
<lifeless> the API outputs a _link for it
<lifeless> but it - and this is the nuts bit - creates the object to do that
<wgrant> We should probably kill that (it's been deprecated for ages), but that's another story.
<mtaylor> thumper: so - the pastebin I'm working with has a remote api... so I was thinking that a "bzr paste" command would be a nice way to spit a diff into a pastebin ... but it would be even cooler if bzr could look up _where_ to paste on the project via launchpadlib
<lifeless> wgrant: so, if its in 1.0, we're screwed ?
<wgrant> lifeless: Of course. It needs to know the name to get the canonical_url.
<wgrant> Well.
<wgrant> I think we should take API versioning with a grain of salt.
<lifeless> by needs you mean 'its a constant pattern wtf'
<lifeless> ?
<mtaylor> thumper: pretty low on the list of things I want to get done -but I just wanted to see if was a waste of time to even put it on the todo list
<mtaylor>  :)
<wgrant> How else can the URL be created?
<lifeless> wgrant: we have the person, the url is a constant suffix to the person canonical url
<lifeless> wgrant: we don't need a DB lookup to do that
<wgrant> lifeless: Erm, it shouldn't be constant.
<wgrant> It should be /+archive/NAME
<lifeless> wgrant: which for the first archive is _always_ ppa
<wgrant> lifeless: No.
<wgrant> lifeless: It's merely the default.
<wgrant> it's been changeable for quite a while now.
<lifeless> wgrant: unless its been changed, the form that you might be thinking of to let you change it actually ignores it on the first ppa
<wgrant> It changed within the last year.
<wgrant> (And broke a few things in the process.)
<lifeless> wgrant: why does my branch name concern you?
<wgrant> lifeless: Well, it suggests that you have one branch per app.
<wgrant> Rather than one per piece of work.
<lifeless> wgrant: I keep my queue very shallow
<lifeless> I've done 2 separate landings from my registry branch so far
<lifeless> both targeted and small
<wgrant> True.
<lifeless> of course, the thirds a doozy
<lifeless> but its still one thing
<lifeless> whats ValidPersonCache all about ?
<lifeless> also, \o/ 14
<thumper> lifeless: I have no idea
<thumper> lifeless: probably histerical raisins
<thumper> lifeless: worth asking the list though
<thumper> someone might have a better answer
<spiv> lifeless: "A materialized view listing the Person.ids of all valid people (but not teams)." :P
<wgrant> It doesn't seem to be materialised.
<wgrant> But that might just be the dev schema.
<wgrant> I ran into it this morning.
<wgrant> And WTFed a bit.
<lifeless> spiv: thanks :P
<lifeless> \o/ 11
<lifeless> spm: what tz is your sprint in/
<wgrant> lifeless: Is that including the auth[nz] and traversal queries?
<wgrant> They number quite a few themselves.
<lifeless> wgrant: yes
<lifeless> wgrant: its now constant at 11 for 2 members or 3 members
<wgrant> Excellent.
<lifeless> I'd appreciate an eyeball on the branch, revno 11316
<lifeless> I'm merging devel into it now to find out about conflicts
<lifeless> and excellent, no conflicts.
<wgrant> lifeless: re. r11309, conjunction is the default. You can just use a sequence.
<lifeless> you mean the conditions variable ?
<wgrant> Yes.
<wgrant> Also, why do you get a whole KarmaTotalCache object, then switch on its .karma_total? Why not COALESCE the column directly in the query?
<wgrant> Saves you an if and some object construction, which must be good.
<wgrant> However.
<wgrant> Adding all this caching at the model layer does not sit well with me.
<lifeless> wgrant: so, coalesce - yes thats possible.
<lifeless> OTOH the pain we're feeling is primarily driven by latency here
<wgrant> It is.
<lifeless> and vertical bouncing
<wgrant> But let's make the DB server work for its dinner.
<lifeless> I'd like to see how it works on staging before tuning further
<lifeless> it may be so fast we don't care
<lifeless> or it may be sql work thats needed, or python in which case absolutely focus on that
<lifeless> so, caching.
<lifeless> is in invalidation that concerns you
<lifeless> s/in/it/?
<wgrant> It is.
<lifeless> are model objects persisted across transactions ?
<lifeless> and does cachedproperty caches get cleared (the storm cache has some stuff done to it)
<wgrant> No.
<wgrant> I believe it gets cleared, yes.
<lifeless> if the objects are dumped after a transaction thats suffiicent
<lifeless> its only if they are reused that the cachedproperty stuff would have a *huge glaring hole*
<wgrant> Oh yes.
<wgrant> But I'm worried about what happens if this is used outside a webservice request.
<lifeless> so, do you have alternative approach
<wgrant> There's no alternative approach possible at the moment.
<lifeless> so, this has the following properties:
<lifeless>  - the caching will happen automatically
<lifeless>  - precaching will require explicit action
<lifeless>  - only the all_members_prepopulated takes that explicit action
<wgrant> But calling that will taint people throughout the transaction.
<lifeless> yes
<lifeless> the first point is more worrying to me
<lifeless> which is that Person.is_valid_person; change EmailAddress table; Person.is_valid_person won't change
<lifeless> but!
<lifeless> its no worse than having to call store.flush()
<lifeless> which is also needed.
<lifeless> Essentially we have a missing concept in our DSL
<lifeless> which is 'the database is slow'
<wgrant> Erm, store.flush() won't do this AFAIK.
<lifeless> wgrant: its an analagous, overlapping problem.
<lifeless> if you make a person valid, and don't call store.flush, is_valid will continue to return False
<lifeless> without my patch
<wgrant> Will it?
<lifeless> yes
<wgrant> Doesn't it make a query?
<lifeless> because its a view
<wgrant> Ah.
<wgrant> But... no.
<lifeless> it will query
<wgrant> It will cause a flush.
<lifeless> I didn't think every query caused a flush
<wgrant> It will flush before it makes a query, so the view will update.
<wgrant> I believe it does, or everything would be pretty broken.
<lifeless> ok
<lifeless> yet-another-reason to have an explicit step for
<lifeless> please talk to the storage layer
<lifeless> rather than things-that-look-cheap.
<wgrant> It would be really nice if we could statically analyse the code and just say 'prejoin everything kthxbye'
<lifeless> yes
<lifeless> however
<lifeless> in the absence of that, I have a few ideas.
<lifeless> replacing cachedproperty with static derived attributes is one - with an explicit talk-to-the-db api in there
<lifeless> its on my list to write up a RFC about this
<lifeless> I wanted to get this done, as concrete experience within the current code, before shooting my mouth off
<lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/32067
<lifeless> wgrant: also prejoining has exactly the same side effects as this, except its even more magic.
<wgrant> lifeless: prejoining only caches references, and is perfectly safe.
<lifeless> mmm
<lifeless> I'll have to revisit it then
<lifeless> it may have changed
<wgrant> It requests the extra objects from the DB, then throws them away.
<wgrant> But that gets them into Storm's cache.
<wgrant> So when it looks at EmailAddress.personID and sees '1', it checks its cache and finds (Person, 1), and doesn't bother to grab the object again.
<lifeless> storm, because of sql's definition, really doesn't get much mileage from the cache
<wgrant> AIUI, at least.
<lifeless> for collections defined on a column, yes
<lifeless> but not for any of this stuff I was doing
<wgrant> Right.
<wgrant> it's safe, but limited.
<lifeless> wgrant: so, please do review and criticise
<lifeless> poolie: how goes it ?
<poolie> nb, still fiddling with vms
<lifeless> I might have a look at the failures then
<lifeless> my API tuning arc is complete
<poolie> ok
<poolie> it looks like the base template i put the 'features' macro into is not actually used by all the tests
<lifeless> ah
<poolie> do you know any better place to define it?
<lifeless> all the tests you wrote, or some other tests ?
<poolie> other tests
<poolie> not sure if it's all or only some subclass
<lifeless> ok
<lifeless> and what causes the failure then ?
<lifeless> oh, you use the features/thing macro in a different .pt file which those others tests do use ?
<lifeless> wgrant: do you perhaps have any thoughts here?
<lifeless> or thumper, though I know its late for you
<poolie> i have it in i think base-template.pt
<poolie> i wondered if that would work :)
<poolie> apparently not
<poolie> perhaps it can get stuffed into the tal context by the publication code?
<poolie> actually i'd like to try to fix this with you rather than have you just fix it
<lifeless> it may need to be
<lifeless> great
<lifeless> I will just dive around the code
<lifeless> and provide entertaining chatter
<wgrant> lifeless: Not sure, sorry.
<lifeless> poolie: so you put it in what, base-layout-macros ?
<lifeless> poolie: also, did you merge-back my tweak ?
<poolie> i did
<lifeless> ah, simple pull - cool
<lifeless> poolie: I was thinking of doing a skin like the private-page skin
<lifeless> for showing to lp developers only
<lifeless> red-when-50-queries
<lifeless> with a big SLOW as a watermark
<poolie> yes, that's what i was thinknig too
<lifeless> \o/
<lifeless> anyhow, this bug
<poolie> i was trying to communicate that by vocie the other day :)
<lifeless> no wonder the idea felt familiar ;)
<lifeless> poolie: so perhaps try lib/lp/app/templates/base-layout-macros.py
<lifeless> bah
<lifeless> .pt
<poolie> mm i thought i did
<lifeless> :!bzr st -r ancestor:../db-devel
<lifeless>   lib/lp/app/templates/base-layout.pt
<lifeless> is all thats mentioned as a .pt file
<lifeless> but it seems to be the parent of base-layout
<poolie> wow doctest knockon failures suck :)
<lifeless> and some portlets use base-layout-macros stuff directly
<lifeless> I am of course totally guessing
<poolie> seems like a reasonable guess
<poolie> multicore machines are nice
<StevenK> poolie: I've been dealing with those for the past week. I am now getting revenge and replacing the doctest with unit tests ...
<wgrant> StevenK: Who is the victim?
<poolie> it's building one vm and running loggerhead-setup in another and there's no perceptible slowdown for interactive use
<poolie> probably would be if i did something disk-heavy
<StevenK> wgrant: lib/lp/soyuz/doc/initialise-from-parent.txt
<wgrant> Ah, that lovely one.
<lifeless> poolie: so, lets use science on this thing
<lifeless> this isn't part of the stack I've dug into
<poolie> i have apport crash warnings all over my screen :(
<lifeless> the first failure is oauth-pages
<poolie> science may have to wait...
<lifeless> oops!
<lifeless> ok, well I'll journal here
<poolie> i am certainly noticing less suggestions from the dupefinder :)
<poolie> i guess it'll be up to ~ubuntu whether that's better or not
<lifeless> poolie: if you find its dropping *relevant* dupes, shout at me
<wgrant> It's still in one-term-missing mode, right?
<poolie> well, for instance "gnome-settings-daemon crashed with SIGSEGV" shows nothing
<poolie> which is a bit surprising
<lifeless> on /ubuntu/ ?
<poolie> finding dupes based on that string is a bit problematic anyhow
<poolie> https://bugs.edge.launchpad.net/ubuntu/+source/gnome-settings-daemon/+filebug/.....
<poolie> ah, my mistake
<poolie> bad example
<lifeless> how so ?
<poolie> because that does find dupes
<poolie> "update-alternatives crashed with SIGSEGV in __strcmp_sse2" in dpkg is a better one
<poolie> also you can see why it worried me :)
<lifeless> well, lets see what the oracle of prod says
<lifeless> it says ...
<lifeless> timeout
<poolie> win
<poolie> so at least the instructions in https://dev.launchpad.net/Running/VirtualMachine#Alternatively seem to work
<poolie> which is nice
<poolie> so when you say b-l-macros is the parent of b-l.pt
<poolie> where is that expressed?
<poolie> i can see it refers to some specfici macros
<lifeless> I did a grep for b-l-macros
<lifeless> and base-layout uses it
<lifeless> as well as portlets
<lifeless> lib/canonical/launchpad/webapp/tales.py provides a hint
<lifeless> base = ViewPageTemplateFile('../../../lp/app/templates/base-layout.pt')
<lifeless> so it does look like thats the core
<lifeless> anyhow, going back to science
<lifeless> oauth test failure
<lifeless> PathExpr standard:u'request/features'>
<lifeless> is whats blowing up
<poolie> mm
<poolie> i defined 'features' as a macro evaluating to that
<poolie> so it has the request object
<poolie> it looks like the request object probably doesn't have a features attribute
<lifeless> raise LocationError(subject, name)
<lifeless>        - __traceback_info__: (<canonical.launchpad.webapp.servers.LaunchpadTestRequest instance URL=http://127.0.0.1>, 'features', [])
<lifeless> yes
<poolie> perhaps the place i hooked in isn't reached by in-process requests for testing
<lifeless> poolie: perhaps!
<lifeless> so, you're hooked into start_request and end_request
<lifeless> there are two possibilities that leap out
<lifeless> three possibilities that leap out
<lifeless> firstly, the test environment might not be raising startrequest properly
<lifeless> its a new things I had to add, so this is possible
<lifeless> secondly, perhaps the request object is being shanghaied or something
<lifeless> lastly, perhaps doing a singleton and not indirecting via request would be better; I-don't-know-its-just-a-thought
<lifeless> oh, also perhaps its got a proxy on it that makes your setattr work but getattr die a silent death
<poolie> we have a singleton
<lifeless> poolie: does the requests/features macro work for you in interactive, 'make run' style testing ?
 * poolie makes schema, and fails
<poolie> yes, it does
<poolie> so we could check the singleton
<poolie> it's the same object
<lifeless> ok, thats good
<poolie> it seemed slightly cleaner to go via 'request'
<poolie> mm the fact it works interactively is why i think that
<poolie> it's something about the request object used here
<poolie> or the way things are set up
<lifeless> so, setting stuff on request seems idiomatic enough, from what I'm seeing
<poolie> mm it looked like it
<lifeless> ok, and there is only one beforeTraversal in our code base
<lifeless> so we should definitely be raising the signal
<lifeless> that leaves possibly only being registered for the signal some of the time
<lifeless> ah! layers
<lifeless> what test layer do ftests run in...
<poolie> mm, how would layers interact with this?
<lifeless> different layers active more of the zope machinery
<lifeless> I get the impression that the DB level stuff is available before all the security proxies (and I'm gyessing events) are available
<poolie> oh, you think the "fire an event when a request starts" stuff may depend on a particular layer being running?
<poolie> possible
<lifeless> stub: ping
<poolie> yay, vm lp is running
<lifeless> another possibility
<lifeless> is that oauth-pages is calling stuff under the covers
<lifeless> and in fact, it appears to be doing that
<lifeless> its invoking the view directly
<poolie> right, because it's not every pagetest failing
<lifeless> and webservice-error.txt is doing the same thing
<lifeless> using a custom RequestExpired
<lifeless> so
<lifeless> I get the feeling that doing what the other similar things do - accessing the singleton directly - is a lot less work
<poolie> some of these failing tests, like test_base_layout, are not even in my branch yet
<poolie> only in devel
<poolie> ok so that sounds good
<poolie> i guess seeing if that fails will at least help us triangulate
<lifeless> define features as modules/lp.services.features.xxx/get_singleton
<lifeless> or whatever the helper-that-looksup-in-a-thread-and-sets-if-not-set is called
<poolie> hm
<poolie> but
<poolie> if it's not set on the request, i wonder if it will be set at all?
<poolie> i don't think it makes sense to have "just some random flags" set
<poolie> iow it has to be created for a request
<poolie> i don't think we want a ground-state value
<lifeless> so
<lifeless> a few thoughts
<lifeless> the NullObject pattern may help - you can say 'here is a features thing' that does nothing
<lifeless> this is really a tests-do-not-match-reality problem its nothing to do with reality
<poolie> mm
<lifeless> in reality, we'll only look things up in the context of the publication,
<poolie> i think if the tests explicitly opt in to "i'm not in a request but i want to pretend"
<poolie> that would be ok
<lifeless> but I'd like to get the feature first and cleanup the testing second, because of the timeliness fo rthis
<poolie> but there's some risk of hiding bugs
<lifeless> well they are doing that, but its implicit
<poolie> so i see in some of these cases that we have a LaunchpadTestRequest instance
<poolie> maybe that's not created in the usual way
<lifeless> as in, we can only systematically tell that they haven't used the publication by the fact that they haven't used the publication
<poolie> if we wanted to give it a null object, that might be a reasonable place to put it
<lifeless> you could, if you are particularly concerned, use the hook you have to replace the null object with a real one, and have folk coming in the back door get the real object
<lifeless> OTOH
<lifeless> we want features in cron scripts and so on too
<lifeless> so 'request' is really too tight a scope anyhow
<poolie> sure
<poolie> i think basically, someone should have an opinion on what flags ought to be active
<poolie> that could be publication, or cron framework
<poolie> or a test saying "just notihng"
<lifeless> Ideally, I completely agree.
<lifeless> We could change the doctest loader to state this as a setUp for all doctests.
<lifeless> but that has the risk of fallout across the board impacting things
<poolie> :/ i should add the tribunal "hide tests containing string $x" fetaure :)
<lifeless> so, I might sleep on this.
<poolie> i think i will add a null object created by the test request
<poolie> to start with
<lifeless> I think we are running into an accomodation of the test suite to the performance constraint
<poolie> perhaps just actually hardcoded to null
<poolie> or more precisely getitem(x)->None
<lifeless> and its going to cause a lot of friction like this
<poolie> oh, which constraint?
<lifeless> 'things are slow' -> developers avoid calling the full stack :)
<lifeless> its also a stack thing - folk want to (reasonably) exercise the views directly
<lifeless> but the views implicitly depend on the full publication happening
<lifeless> (I can name other things like this that are only by-chance not causing headaches)
<lifeless> such as profiling
<lifeless> or the request summary information which is reset by the publisher.
<lifeless> or the storm cache
<poolie> heh
<lifeless> so the thing that is unique about your code, is that you are indirecting via the request
<lifeless> rathe than being sloppy and tolerating these tests getting silly things back
<lifeless> This is why I'm encouraging you to be sloppy (but not careless) here
<lifeless> anyhow, 9pm; time for a break, I'll be needing to chat to mars in a few hours
<lifeless> avoir
<poolie> let's try this: http://pastebin.ubuntu.com/475347/
<wgrant> [A22
<wgrant> Gah.
<poolie> sure, see you later
<stub> lifeless: pong
<lifeless> stub: nevermind, sorry. 'unpong' :P
<lifeless> poolie: I think thats a good start
<lifeless> poolie: also RequestExpired will need it
<lifeless> (note that it is RuntimeError :P)
<poolie> hm
<lifeless> (look at the webservice-error.txt file)
<lifeless> poolie: as for the tests that are not in your branch and failing, merge db-stable to your branch, and you should get them all
<poolie> ?
<lifeless> or db-devel to your branch
<poolie> no that's the same Request tiype
<poolie> sure
<poolie> is this now targeted at devel or db-devel?
<poolie> i guess db-devel now merged to devel?
<poolie> since my db changes?
<lifeless> not yet, late this week
<lifeless> class RequestExpired(RuntimeError):
<lifeless>     """Request has timed out."""
<lifeless>     implements(IRequestExpired)
<lifeless> thats in adapter.py
<lifeless> anyhow, gl, chat with you tomorrow
<lifeless> by 'thats a good start' I meant your pastebin, in case it wasn't clear.
<poolie> good night
<poolie> understood
<poolie> i don't see what i need to do with requestexpired
<poolie> that test is using the same request class
<lifeless> oh
<lifeless> interesting
<lifeless> ok, cool.
<poolie> is it just me with launchpad tests failing in
<poolie> with a deprecationwarning in bzr builder?
<poolie> maybe update-sourcecode will fix it
<bigjools> wgrant: evening
<wgrant> bigjools: Hi.
<bigjools> wgrant: what part of soyuz uses guessPackageNames>
<bigjools> ?
<wgrant> I have to disappear in a few minutes.
<wgrant> bigjools: Nothing.
<bigjools> ok
 * bigjools wonders why it's a soyuz bug then :)
<wgrant> bigjools: Someone said so. I don't recall who.
<wgrant> And it is a Soyuzy query!
<bigjools> curtis probably
<bigjools> sigh
<wgrant> It's mostly used by the bug target picker.
<bigjools> I've no idea how to QA it
<bigjools> which most likely means it's not a soyuz bug
<wgrant> It's one of those things that's in the void, and nobody has touched for years.
<wgrant> I will QA it when I return later this evening.
<bigjools> ok
<bigjools> cheers
<poolie> night all
<wgrant> [A22
<wgrant> GAH.
<wgrant> Why has this started happening?
<bigjools> thumper: around?
<thumper> bigjools: no
<thumper> bigjools: you have 3 minutes
<thumper> whazzup?
<bigjools> reviewed your branch
<bigjools> gotta dash myself
<jml> hello
<thumper> bigjools: I had taken lifeless's rc as ok and landed it already, the original diff was updated to use config.launchpad.no_reply_email
<thumper> bigjools: if you like we can file another bug and look at it for the start of next cycle
<lifeless> leonardr: hi
<leonardr> hi lifeless
<deryck> Morning, all
<lifeless> morning deryck
<lifeless> have a good weekend ?
<wgrant> bigjools: Hi. Am I likely to be granted an RC for bug #615286?
<_mup_> Bug #615286: DEPWAIT not recognized from build log <Launchpad Auto Build System:New> <https://launchpad.net/bugs/615286>
<wgrant> Simple lp-buildd regex change.
<deryck> lifeless, indeed.  Busy but relaxing.  Thanks!  And you?
<bigjools> wgrant: no, it's not RC
<bigjools> they can manually retry
<lifeless> deryck: pretty good, house stuff + hacked on qa-tagger sunday
<deryck> cool
<lifeless> wgrant: cherrypick it IMO
<lifeless> we're getting rid of the whole release thing this cycle, if everything comes out well
<bigjools> lifeless: it's a buildd change, it's not affected by the release at all
<bigjools> IS releases buildd changes
<lifeless> bigjools: yah
<bigjools> so it's not a CP :)
<lifeless> well, in the sense of 'deploy without waiting a month'
<lifeless> we dont have a good phrase for that atm
<deryck> heh
<spiv> "deploy now"
<deryck> my "heh" was because I read lifeless as "deploy without *wasting* a month."  Clearly I need more coffee.
<lifeless> heh
<lifeless> nice misread there
<lifeless> grah
<lifeless> when we are moving to lucid ec2 test images ?
<jml> after we move to lucid everything else?
 * jml isn't sure.
<lifeless> neither am I
<lifeless> AIUI bb looks at lucid now for blessing
<lifeless> so it would make sense to move the ec2 images to lucid
<jml> agreed
<lifeless> have I mentioned I fear and loath doctest failures
<jml> lifeless, once or twice
<jml> (also, "loathe" is a verb, "loath" is an adjective)
<lifeless> +e
<StevenK> lifeless: O hai.
<StevenK> lifeless: You mentioned a transform that Storm could do for me, WRT _copy_lucille_config. Could I have an example?
<lifeless> uhm
<lifeless> show me your diff again :P
<StevenK> lifeless: https://code.edge.launchpad.net/~stevenk/launchpad/move-ifp-from-idistroseries/+merge/31520
<StevenK> lifeless: I didn't want to change the current code, but I have more work in the same area, and I could use what you were suggesting
<lifeless> right
<lifeless> you create a result set
<lifeless> store.find(DistroSeries, DistroSeries.id == self.distroseries.id)
<lifeless> then you call set on it
<lifeless> pdr = Alias(DistroSeries,
<lifeless> name='pdr')
<lifeless> resultset.set(DistroSeries.lucilleconfig=Select(pdr.lucilleconfig, pdr.id==self.parentid))
<lifeless> I think thats it - check the sql it emits of course
<StevenK> lifeless: Right, but to do a copy like that in general, .set() is my friend?
<lifeless> to do an UPDATE, .set() on a result set is your friend
<lifeless> that 'copy' is actually a two-stage thing, it duplicates, then it changes
<lifeless> or something like that - the one method is really 'trash the config using that one over there'
 * bigjools actually thinks that SQL syntax is better for once
 * lifeless doesn't really care
<lifeless> with the registry stuff, doing it in raw sql would be worse I think
<bigjools> probably, but why?
<bigjools> Storm syntax is generally pretty unintelligible for me
<lifeless> for the registry stuff I was doing, because it is combining 6-7 different clauses, constraints and tables into one aggregate query
<lifeless> bigjools: I'm saying 'consider on a case by case basis' :)
<bigjools> I completely agree :)
<lifeless> storm is so tied to sql that if we wanted to do things differently we're no better-or-worse off
<gmb> lifeless, Can we consider https://bugs.edge.launchpad.net/malone/+bug/607776 qa-ok?
<_mup_> Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>
<deryck> lifeless, I think we can mark bug 607776 as qa-ok.  Would you agree?
<_mup_> Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>
<deryck> heh
<lifeless> jinx
<lifeless> yes, absolutely.
<lifeless> mea culpa
<deryck> no worries.
<gmb> DOne
<deryck> gmb, I'll leave it to you to mark, since I stepped on you once. ;)
<lifeless> I've been watching like a hawk for issue
<lifeless> *issues*
<gmb> deryck, Heh. I wonder what happens if two people try to make the same change. Probably the world ends or something.
<deryck> me, too.  Seems solid to me.
<deryck> the AJAX world ends
<lifeless> its still possible we'll find an important common use case it doesn't handle well, but for now, its happy
<deryck> gmb, I'm still working on all the ones for my dupe work.  Staging doesn't have the change yet, so I'm poking around bugs that have moved duplicates to make sure things look good.
<gmb> Right
<lifeless> _mup_: poke mars
* bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<deryck> Ok, bugs team is down to 4 bugs needing qa now.  One requires a chat with bdmurray.  I think gmb is aware and working on qa for bug 594211.
<_mup_> Bug #594211: Add a RecipientReason class to bugnotificationrecipients.py <qa-needstesting> <story-better-bug-notification> <story-refactor-bugnotifications> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/594211>
<deryck> The other two are me and need my changes on staging to qa.
<deryck> bigjools, ^^
<bigjools> deryck: rock n roll, thanks
<gmb> deryck, bigjools That's qa-ok; marking it as such now.
<deryck> np, sorry it took us so long this time.
<deryck> gmb, excellent, thanks!
 * gmb lunches
 * bigjools -> lunch
<lifeless> gmb: are you sure bug 606914 is fixed ?
<_mup_> Bug #606914: Bug.initial_message fetches all of the bug's messages <qa-ok> <timeout> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/606914>
<lifeless> gmb: unless there is a test checking the sql count, I suspect we'll find its still a problem after the rollout
<lifeless> gnight all
<jtv-afk> oh, yeah, nick.
<gmb> lifeless, I'll check
<gmb> lifeless, So, that particular bug - and the problem described in it (that initial_message was stupid) is fixed. It doesn't fix the timeout problem, but there are other bugs covering that (which may or may not need consolidating into one sensible report, or at least clarifying so that the distinctions between them become more obvious). I'm happy for this particular one to be qa-ok, though.
<salgado> sinzui, do you know where's the code that allows traversal from a distro to a distroseries? I thought it'd be in DistributionNavigation but it's not
<salgado> oh, it must come from GetitemNavigation,
<salgado> indeed, that's it
<cperrin88> Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?
<cperrin88> the whole root wadl file
<cperrin88> The sounds like no
<mars> cperrin88, leonardr is working on that very problem
<mars> cperrin88, IIRC older versions of wadllib grabbed the whole thing.  Newer versions cache it.
<leonardr> mars, cperrin88, i was off irc during the crucial moment. can you recap?
<mars> leonardr, <cperrin88> Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?
<mars> <cperrin88> the whole root wadl file
<leonardr> cperrin88: yes, it will get the wadl file on first startup and about once a month afterwards (whenever launchpad is upgraded)
<cperrin88> okay .... well ... I'm doind a Java port of the lib ...
<leonardr> cperrin88: ok, you can implement the same behavior. the wadl file is served with cache-control headers and an etag
<cperrin88> Yeas
<leonardr> so you can know how long to cache it for, and once the cache expires, you can avoid getting the whole document unless launchpad has been upgraded
<cperrin88> that's what I wanted to do
<cperrin88> I just wanted to make sure that you really grab the whole thing
<cperrin88> it's rather big
<leonardr> cperrin88: if you send Accept-Encoding: gzip it's only about 100k
<cperrin88> hey
<cperrin88> your right
<cperrin88> that's cool
<cperrin88> without it's more than 10 times the size
<leonardr> xml compresses very well
<cperrin88> seems like .... that helps a lot
<leonardr> so if you implement all these optimizations your client will be making 1 http request once a week and once a month that http request will result in a 100k download
<cperrin88> I was concerned about mobile users
<cperrin88> yep
<leonardr> note that there is also a json version of the service root, which has the same rules, but it's much smaller--only about 1k uncompressed
<cperrin88> The lib is a result of my development of a Launchpad android client
<cperrin88> Yeah
<cperrin88> I know
<leonardr> ok
<cperrin88> but I think I will handle it more like launchpadlib does now
<cperrin88> thank you for your help
<leonardr> sure
<rockstar> benji, ping
<benji> rockstar: hey
<rockstar> benji, do you know much about View permissions?
<benji> a bit, what's up?
<cperrin88> leonardr: I think I found a small error in the WADL file. Two doc parts have invalied entitiys ... At least my parser says that
<leonardr> cperrin: send me the doc output. we validate the wadl before deploying, but it's possible we have access to some external doc you don't
<cperrin88> it's   `     (Apostrophe)
<cperrin88> lemme see
<cperrin88> I take it back
<cperrin88> I guess it was my fault
<leonardr> ok
<benji___> rockstar: sorry, had some technical difficulties; will you repeat your question?
<rockstar> benji, permissions on View don't care what the request type is, correct?  It's the same for GET and POST?
<benji> rockstar: correct
<benji> permissions don't assume "correct" use of HTTP verbs
<lifeless> moin
<lifeless> mars: hi
<mars> Hi lifeless
<mwhudson> morning
<lifeless> mars: Ursinha: I just got up, so haven't trawled email yet; I'm going to look for review feedback now
<mars> lifeless, Ursinha and I have both looked over your changes.  They look good - thanks for doing that.  She is just finishing her review now.
<Ursinha> yes sir
<lifeless> cool, I hope its helpful
<mars> lifeless, very.  Good hooks for the parts we still need to write, too
<Ursinha> lifeless, do you intend to write more code there?
<lifeless> Ursinha: maybe :) - AFAICT theres enough there now that we could switch to the new workflow and iterate on polish ?
<Ursinha> lifeless, and yes, that was pretty much what I thought to be a way to reuse qa-tagger structure, very helpful
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Performance Day! | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<mars> lifeless, Ursinha may have to double-check the qa-sort algorithm.  But there should be enough to start integration with whatever downstream component there will be.
<Ursinha> lifeless, hm, not sure, I'm not that familiar with the whole changes to put that in prod now
<mars> lifeless, assuming Ursinha thinks the script output is still sane for our current daily production use :)
<lifeless> so
<lifeless> if more work is needed to make this usable for the new merge workflow where we qa edge and request deployments from the losas at up-to revision granularity
<lifeless> (usable, not 'perfect' :P)
<mars> of course
<lifeless> then - yes, I'll write more code, while you guys sleep tonight :)
<lifeless> If, OTOH, its sufficient to get us started, then I've some other things to help along to get that workflow transition to happen
<mars> Ursinha, would it be possible to run this tagger code beside the existing stable tagger code?  So we don't interrupt our production line?
<mars> lifeless, Ursinha is in the process of reviewing the code to make sure that the already in-place QA process and tagger are not interrupted.
<Ursinha> mars, well, yes, we can do that and tag bugs twice or change one of them a bit so we wouldn't have the two fighting over bugs
<mars> Ursinha, perhaps we could *not* run the tagger part?  By adding a CLI switch or something?
<Ursinha> lifeless, btw, waking up having code to review is nice
<Ursinha> mars, that can be done
<lifeless> mars: we don't need to run them in parallel so much as be sure that we have *enough* so that when we switch we can use it
<lifeless> Ursinha: my pleasure
<lifeless> mars: although in parallel with a smooth transition is nice, obviously
<Ursinha> we still need to consider the rollback commits
<lifeless> ok
<Ursinha> not much of a big deal but needs a bit of code there
<lifeless> they are tagged qa-rolledback or something ?
<Ursinha> qa-rollback
<mars> lifeless, well, I believe Ursinha's goal is not to get enough working, it is 'can be guaranteed to still work like it did on Friday' :)
<Ursinha> pretty much :)
<lifeless> does it mean 'needs to be rolled back', or 'it has been rolled back'
<Ursinha> lifeless, it was rolled back
<lifeless> kk
<lifeless> so, if 'qa-ok' in tags or 'qa-rollback' in tags :)
<lifeless> should be a 12-character fix :)
<mars> + test
<lifeless> of course
<Ursinha> lifeless, https://dev.launchpad.net/QAProcessContinuousRollouts
<Ursinha> lifeless, it also needs to add the tag, when appropriated
<lifeless> mars: so the overall -goal- here is to change the way we work; keeping things unaltered is insufficient ;)
<lifeless> Ursinha: ok, based on the new lp-land switches ?
<Ursinha> the commit should have a [rollback] clause, same as [testfix]
<Ursinha> so we know that if a new commit is a rollback, we know the bug and know the last revision that's blocked waiting for this rollback
<lifeless> so the tagger would tag the old revisions bug as rollback
<Ursinha> a rollback is a reversion of a qa-bad or qa-needstesting previous one
<Ursinha> yes sir
<Ursinha> and let it go
<lifeless> righto
<lifeless> so 'rollback' is 'unqaable' for the reporter, but still taggable for the tagger
<Ursinha> lifeless, yes
<lifeless> is that the total of the remaining must-have features ?
<Ursinha> lifeless, why are you pinging me? :)
<lifeless> bah, fat fingers
<Ursinha> lifeless, unqaable: untestable, orphaned, testfix, buildbot, and now rollback
<Ursinha> therefore blessed
<lifeless> I was meaning to do 'TIME'
<mars> lifeless, I understand.  I think Ursinha should make the call on when the new qa-tagger code can replace what is already running.  Parallel deployments is just an idea.  We don't have to take it.
<lifeless> mars: I'm happy either way
<Ursinha> mars, I like the idea of running them in parallel
<lifeless> ok, phone time for me
<lifeless> Ursinha: Its your 5pm now right?
<Ursinha> lifeless, yes
<lifeless> I'll look at adding rollback now
<Ursinha> lifeless, cool
<Ursinha> thanks
<lifeless> if there is more than that needed, let me know please before you retire for the night.
<Ursinha> sure
<Ursinha> lifeless, that wiki page has details, maybe that helps a bit
<mars> lifeless, btw, are you on the Kanban board, and can't edit?  Or do you not have an account at all?
<lifeless> couldn't edit your board
<mars> ok
<mars> no admin privs - I can't change it.  Oh well.
<lifeless> flacoste: https://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/32067
<sinzui> lifeless, I started this. I got distracted by my sprint
<lifeless> sinzui: sprinting is good
<sinzui> lifeless, I forsee a few state issues in this branch
<lifeless> sinzui: how do you feel if it lands without your review? or could you give it enough cycles to say 'thats ok we can tweak later' or 'lifeless, you need to look at x,y,z'
<lifeless> sinzui: ok, can you roughly describe them ?
<sinzui> I can deactivate a person and the page will still show active...I need to refresh my browser I think
<sinzui> lifeless, Person.preferredemail knows when it is mutated to manually purge its _*_cached property
<sinzui> lifeless, But i am speculating on these scenarios  for coc and archive
<sinzui> lifeless, I am fine with this landing. We can fix issues as we discover them
<lifeless> sinzui: thanks!
<lifeless> flacoste: line 680
<flacoste> lifeless: lp/registry/model/product.py:470
<EdwinGrubbs> jelmer: ping
<jelmer> EdwinGrubbs, pong
<EdwinGrubbs> jelmer: it looks like you can set the upstream project differently on each distroseries? Is that a good idea.
<jelmer> EdwinGrubbs: there are some (very rare) situations in which that is necessary
<jelmer> EdwinGrubbs: e.g. I think that the chromium package used to be for the chromium bsu game but now is for the chromium browser
<EdwinGrubbs> ok
<mars> sinzui, does pocketlint not catch tab and indentation errors?  (check person.py line 3603-5)
<mars> tabnanny.py yelled at me after finding that
<mars> sinzui, lib/lp/registry/browser/person.py:3603
<sinzui> mars, I think it yells twice in some cases
<mars> I'm kind of weirded out by the interpreter not stopping on that.  I would figure the change of indentation type in the middle of a method would cause something bad to happen.
<benji> mars: the interpreter equates tabs with eight spaces
<sinzui> Someone landed tabs in registry code 6 weeks ago. jtv reported it and I added tab checking
<sinzui> mars, yes, pocketlint hates the file
<sinzui> mars, should it call bzr blame for those lines too?
<sinzui> mars, actually, I see bac's emacs is set to correct tabs to spaces and his diff in review accidentally started removing the tabs :)
<mars> sinzui, pocketlint on dev systems?  It ignores it on mine.
<mars> sinzui, should we ditch tabnanny then?
<mars> It would make test-on-merge faster
<sinzui> hmm, maybe the version is wrong in developer-deps
<sinzui> mars, this is what I see http://pastebin.ubuntu.com/475630/
<mars> sinzui, exit code zero?
<mars> sinzui, ok, PEBCAK here on the output - but the exit code is a bit odd.
<mars> sinzui, so I'll remove tabnanny from test_on_merge.py then.  That is what lint is for, and that file has no need of the additional complexity.
<sinzui> As I said above, developer-deps is behind
<sinzui> I stopped publishing pocket-lint because hardy had build issues
<sinzui> You can get the latest version from my ppa, or wait till next week (after the release) where I can push a version that will play in the test runner
<mars> done - tabnanny in test_on_merge is no more
<sinzui> lint also screams at pdbs
<sinzui> maybe we want a test that checks the code for pdbs
<mars> don't know - might be a good idea, but it would take a while to run.  If most people run lint, then it may not be necessary.
<mars> lint catches far more things
<mars> lint is already a 'best effort'
<benji> leonardr: I started to write the MP for ~benji/lazr.restfulclient/total_size_link and realized that you might be the better one to write it because you did almost all the work :)
<benji> oops, wrong chan
<lifeless> sinzui: thanks for approving that; will land as soon as pqm opens :)
<bdmurray> sinzui: I'm having an issue with return Link() in lib/lp/registry/browser/structuralsubscription.py and StructuralSubscriptionMenuMixin.  deryck thought I might check with you about it.
<lifeless> leonardr: ping
<mwhudson> rockstar: are you confusing TAL and ZCML in your mail to launchpad-dev?
<rockstar> mwhudson, yes, because it is all ugly...
<wgrant> mwhudson: ShipIt sadly uses TeamParticipation and Person.
<wgrant> So it's not that easy.
<mwhudson> wgrant: ah
<mwhudson> oh well
<lifeless> web services
<wgrant> Right.
<wgrant> The former can be done from SSO.
<wgrant> And the latter is just for karma.
<wgrant> So it's not *that* hard.
 * lifeless hates code freeze
<lifeless> do we have an unfrozen branch for folk to work on this week ?
 * ajmitch hates seeing edge timeouts :)
<wgrant> lifeless: No :(
<lifeless> or do we all just pile up and conflict on friday?
<wgrant> Precisely.
<wgrant> I believe you have a fix for this, though :)
<lifeless> ajmitch: whats the oops id ?
<ajmitch> OOPS-1682EC4078
<ajmitch> it was for a bug link that had comments=all
<ajmitch> so no surprise that it'd take a little while
<lifeless> well
<lifeless> lets see
<lifeless> Statement Count: 303
<ajmitch> performance tuesday for you?
<lifeless> thats going to be a large driver
<lifeless> ajmitch: yup
<lifeless> yeah, lots of repeated things
<lifeless> 285.	6825	24ms	launchpad-main-slave	SELECT LibraryFileContent.datecreated, LibraryFileContent.filesize, LibraryFileContent.id, LibraryFileContent.md5, LibraryFileContent.sha1 FROM LibraryFileContent WHERE LibraryFileContent.id = %s LIMIT 1
<lifeless> 286.	11541	46ms	launchpad-main-slave	SELECT LibraryFileAlias.content, LibraryFileAlias.date_created, LibraryFileAlias.expires, LibraryFileAlias.filename, LibraryFileAlias.hits, LibraryFileAlias.id, LibraryFileAlias.last_accessed, LibraryFileAlias.mimetype, LibraryFileAlias.restricted FROM LibraryFileAlias WHERE LibraryFileAlias.id = %s LIMIT 1
<lifeless> note that there is 3 seconds of time between those two db calls - thats going to be python
<lifeless> not 'lots of bytecode', rather 'some other thread screwed us'
<lifeless> 120 subscription lookups
<lifeless> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/131094/+index
<_mup_> Bug #131094: Heavy Disk I/O harms desktop responsiveness <cft-2.6.27> <Linux:Invalid> <linux (Ubuntu):Confirmed for ubuntu-kernel-team> <linux-source-2.6.22 (Ubuntu):Won't Fix by ben-collins> <https://launchpad.net/bugs/131094>
<poolie> spm: hey, awake yet?
<wgrant> poolie: IS sprint...
<james_w> hi poolie
<james_w> hi wgrant
<wgrant> Morning james_w.
<poolie> hi james_w; thanks wgrant
<lifeless> poolie: all the losas are in UTC+2 this week
<lifeless> Ursinha: / matsubara-afk: how can I reset the bug infestation for an oops ?
<Ursinha> lifeless, let me find the tomboy note that explains that
<leonardr> lifeless, hi
<lifeless> \o/
<lifeless> leonardr: hi
<lifeless> leonardr: uhm, uhm, I've forgotten my thread
<Ursinha> lifeless, wait, do you want to know just to know or you just need something to be changed?
<lifeless> Ursinha: I keep finding wrong ones when you guys are asleep
<lifeless> so I want to know how to do it myself
<Ursinha> lifeless, ah sure
<leonardr> lifeless, just type it if you remember, i'll come back later
<Ursinha> lifeless, I'll send one email
<lifeless> Ursinha: thanks!
<lifeless> leonardr: no worries!
<Ursinha> no problem :)
<james_w> wgrant: want to tell me where https://code.edge.launchpad.net/~james-w/launchpad/move-soyuz-test-publisher/+merge/32157 falls on a scale of 1 to heinous?
<wgrant> That scrollbar scares me.
<wgrant> And those are some serious docstrings.
<james_w> yeah, I think that's where a lot of the vertical height comes from
<mars> bzr question: I am using sandboxes with 'bzr switch', I have pushed my branch, bzr says 'Created new branch', but Launchpad insists "This branch has not been pushed to yet.".  What gives?
<mars> On top of that, 'bzr push --overwrite' says "No new revisions to push"
<wgrant> mars: LP may be being a little slow. Replication lag, maybe?
<lifeless> mars: did the url it reported make sense to you?
<mars> lifeless, which URL from which command?
<lifeless> when you push, bzr tells you where it is pushing
<lifeless> I think, or is that commit.
<lifeless> anyhow, where di dyou push to ?
<lifeless> precisely
<mars> lifeless, ah, got it: incompatible repo formats
<mars> lifeless, and bzr doesn't re-raise the error when I try again
<wgrant> james_w: Hm, so test_publishing has a special STP derivative which uses sampledata?
<mars> lifeless, how did you push your qa-tagger code without it barfing?
<lifeless> mars: I didn't put it in a shared repo
<lifeless> its odd that bzr would only whine once
<mars> lifeless, ok - using 'upgrade this branch' on the UI is harmless? >:)
<lifeless> mars: you need to coordinate with everyone
<mars> yarg
<lifeless> mars: e.g. with the losas because we roll this stuff out, unless qa have direct control over where its rolled out to and can do the upgrade on sodium-or-whatever
<mars> lifeless, this is under qa's control
<lifeless> so, coordinate with them :)
<mars> lifeless, so I hit 'upgrade this branch', then what happens?  They try to pull new revisions and it blows up or something?
<lifeless> well
<lifeless> tell me what 'this' is bound to
<lifeless> because I don't have the full picture
<mars> https://code.edge.launchpad.net/~launchpad-qa/qa-tagger/devel
<lifeless> ok
<lifeless> so if you upgrade that
<lifeless> then yes, everyone merging or pulling from it will blow up
<lifeless> they then need to upgrade and can then pull/merge appropriately
<mars> lifeless, will you still be able to land your unmerged work if I do this?
<wgrant> james_w: So, looks heinous, but it's probably the right thing anyway.
<lifeless> mars: yes, I just need to upgrade my branch too - as I said, you need to coordinate with everyone :)
<wgrant> Until we can port everything away from the old one, which could be a while...
<lifeless> mars: e.g. send a mail
<mars> lifeless, ok, will do
 * mars presses the Big Yellow Button
<thumper> yellow?
<wgrant> The drunken exclamation mark.
<mars> thumper, to be honest, I think the +upgrade page needs some work :/
<thumper> mars: probably
<mars> I could just copy and paste this IRC conversation :)
<mars> "What does this button do?"  "Well, this conversation describes it well: ..."
<lifeless> hmm, I could have sworn there was a bug open for oops' miscategorising storm timeouts
<mars> lifeless, so besides all the branch format shenanigans, I have some code that is built on top of yours.  You may want to merge it: lp:~mars/qa-tagger/lp-service-switch
<mars> (not pushed yet, need the upgrade to finish first)
<lifeless> what does it do /
<lifeless> ?
<lifeless> does anyone remember the oops-doesn't-include-storm-timeouts-as-sql-time bug ?
<mars> lifeless, adds a --lp-service command-line switch.  That switch tells the script which lp instance to run against.  We want to use this for smoke-testing
<mars> lifeless, failsafes to 'staging'.  Any developer should be able to check out the code and safely do a smoketest this way.
#launchpad-dev 2010-08-10
<mars> lifeless, I'm signing off for now, will push the branch once the upgrade finishes
<mars> (which would be now)
<lifeless> mars: cool
<wgrant> lifeless: I think basic auth is still enabled, just using the old passwords.
<wgrant> Which is probably useless and a security risk.
<lifeless> oh, hahaha.
<lifeless> I just realised another interaction/issue with lowering hard timeouts
 * lifeless sighs
<wgrant> Oh?
<thumper> lifeless: ?
<lifeless> a single rogue page that renders slowly can cause 3 other requests to timeout
<lifeless> because python + threads is epic fail.
<lifeless> so the lower the hard timeout, the more boundary-case slow pages will cause *other* pages to have inexplicable timeouts like - https://lp-oops.canonical.com/oops.py/?oopsid=1681EB3871
<lifeless> OTOH this may just be a genuinely slow template
<wgrant> lifeless: Maybe you should use that as an excuse to spawn a push to minimise Python time throughout the application.
<lifeless> wgrant: we're doing that by lowering the timeout :P
<wgrant> Is the performance report public again yet?
<lifeless> any losa around that can't sleep? I need a kcachegrind profile from staging please.
<lifeless> wgrant: no, it needs log filtering applied to strip private urls.
<lifeless> wgrant: technically even staff can't look at it atm.
<wgrant> Ahh.
<lifeless> well, thats a slight exaggeration, but you know what I mean: there *can* be stuff turn up that normally you'd need DB access to see
<lifeless> which is undesirable at best
<ajmitch> OOPS reports don't expose quite as much?
<lifeless> they are also staff only
<lifeless> and also need to have this issue addressed
<lifeless> with the issue addressed we could start publicising them, which I would dearly love to do
 * thumper is frustrated
<thumper> damn code didn't work as I expected
<lifeless> which one?
<thumper> QAing the fix I did for processing merge directives
<lifeless> wgrant: publishedpackage is what you nuked yes?
<lifeless> wgrant: what revision did it land in? has it landed?
<thumper> I fixed one bug, but now it isn't actually pulling in the directive (it seems)
<thumper> how often are the staging logs synced to devpad?
<lifeless> 7 minutes I think
<lifeless> perhaps 6
<thumper> :( known 2a and MD bug check failed: Cannot add revision(s) to repository: missing text keys:
 * thumper thinks
<thumper> only way to confirm fix is not to use 2a
<wgrant> lifeless: db-devel r9628
<lifeless> damn
<wgrant> Not on staging yet?
<lifeless> wgrant: I wish you had done that as two separate patches, db + other
<lifeless> wgrant: no, 10 timeouts from it in sundays edge report
<wgrant> :(
<wgrant> Which views?
<wgrant> +distrotask?
<lifeless> +filebug
<wgrant> Ah.
<lifeless> tell me about SPPPH
<lifeless> sorry, SPPH
<lifeless> also, ugh, pagination *hate*
<thumper> lifeless: got a stutter there?
<thumper> can anyone push to staging right now?
<wgrant> lifeless: What about SPPH?
<wgrant> SSPPH died a few months ago.
<lifeless> wgrant: timeouts
<wgrant> lifeless: Ah, the getBuildRecords timeout?
<james_w> wgrant: yeah, it's a separate class so we can port a bit at a time. It has a different API as I think this is better than the old one.
<wgrant> james_w: It looks like it, yes.
<wgrant> STP is pretty old and was put together over a long time.
<james_w> yeah, it's understandable. IMO it needs to mature now that it is a central part of Soyuz's testing
<wgrant> Definitely.
<wgrant> And bonus points for eliminating sample data while we're at it.
<james_w> yeah, that's my main aim, but as I started working with it I wanted to clean up the API
<wgrant> 0*$ bash                                                                                                                                                                                                                           Menu:<F9>
<james_w> some changes are gratuitous, but others will help stop being the tests so tightly bound to the implementation, which is one of the problems that we have with sampledata, so I wouldn't want to replicate it at a different level
<wgrant> Indeed.
<wgrant> We have model code working around bad sampledata :(
<wgrant> So it will be good to eliminate it.
<lifeless> grah
<lifeless> hmm
<lifeless> this is nonoptimal
<lifeless> lpmain_staging=> SELECT Bug.heat FROM Bug, Bugtask, DistroSeries WHERE Bugtask.bug = Bug.id AND Bugtask.distroseries = DistroSeries.id AND DistroSeries.distribution = 3 ORDER BY Bug.heat DESC LIMIT 1;
<lifeless>  heat
<lifeless> ------
<lifeless> (0 rows)
<lifeless> Time: 235347.492 ms
<ajmitch> ...
<ajmitch> that's awfully long
<lifeless> you think?
<wgrant> It is staging, though.
<wgrant> But yeah, just slightly slow.
<lifeless> 14 seconds 'hot'
<lifeless> which matches production timing
<lifeless> http://pastebin.com/G8sctDVF
<wgrant> Hmm.
<lifeless> If I read it correctly its scanning every bug ever in descending heat and one by one comparing their distroseries distribution
<lifeless> what we'd really rather it do is expand the set of distroseries ids to be 'IN select distroseries.id from distroseries where distroseries.distribution = 3'
<mwhudson> i don't suppose it matters here, but can there really not be an index on distroseries.distribution?
<lifeless> of course, also to do that magically.
<mwhudson> maybe it's just such a small table that seqscan is quicker
<lifeless> mwhudson: it may be that that is needed.
<wgrant> It's a tiny table at the moment.
<lifeless> yes, but query analyser magic.
 * wgrant wishes for an import fixer.
<wgrant> sed's handy for moving stuff, but then you have to reorder and merge lines :(
<lifeless> edwin has a vim script for it
<wgrant> I have his set of vim scripts.
<wgrant> Maybe I have it but just don't know about it.
 * wgrant looks.
<mwhudson> i want to write something that does all my import statement editing for me
<mwhudson> (based on tags or something)
<mwhudson> rope apparently has something to do this, but i never got ropemacs to work
<wgrant> Hm, so, yes, it has Ctrl+L which uses tags and adds the magical import.
<wgrant> And Ctrl+P supposedly reformats all the imports.
<lifeless> wgrant: if you're interested in performance stuff
<lifeless> DistroSeries:EntryResource:getBuildRecords
<lifeless> https://bugs.launchpad.net/soyuz/+bug/590708
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Launchpad Foundations:Triaged by benji> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<lifeless> wgrant: ^ the soyuz task on it needs to change to a better query in the short term
<lifeless> wgrant: danilo has crafted one, but its not been transcribed; should be very low hanging fruit, and its the top timeout from mondahy
<lifeless> poolie: shall we continue with that bug ?
<lifeless> hmm, when do we toggle from 'fix committed' to 'fix released' ?
<lifeless> sinzui: hi, btw Ursinha requested that we use oops and timeout tags as mutually exclusive things
<Ursinha> lifeless, after the rollout?
<lifeless> Ursinha: no, a couple of weeks ago
<lifeless> Ursinha: you pinged me about filing bugs with oops & timeout tags, as said 'one or other'
<Ursinha> lifeless, I answered your first question
<Ursinha> <lifeless> hmm, when do we toggle from 'fix committed' to 'fix released' ?
<lifeless> Ursinha: ah, cool, ECONFUSED :)
<Ursinha> hehe :)
<sinzui> lifeless, akc
<sinzui> or ack
<sinzui> Ursinha, When I see the release announcement, and I see the registry change is indeed on lpnet, I run the close_developer_bugs for the registry team. When I was RM, I ran it for all launchpad-project
<Ursinha> sinzui, that's cool
<Ursinha> lifeless, ^
<lifeless> cool
<lifeless> I was asking because I wanted to fit in ;)
<sinzui> I know I sent it to the list a few months back. you can pass a list of launchpad ids to update, or ALL to DTRT
<lifeless> sinzui: the portlet thing
<sinzui> yes that thing
<lifeless> when I looked at an oops for the timeout
<lifeless> it shows a very long non-sql gap taking place
<lifeless> do you think that that is just the parsing and construction of the objects returned by the earlier query ?
<sinzui> lifeless, yes. EdwinGrubbs can to that conclusion
<sinzui> we only want a number too
<lifeless> right
<lifeless> I'm with you on that, just wanting to be sure we fix the core issue :)
<lifeless> sinzui: the thing is, that there are 2155 rows
<lifeless> sinzui: I find it hard to believe that creating 2155 objects takes 18 seconds of python time
<lifeless> sorry, 16 seconds
<lifeless> sinzui: I'm looking at https://lp-oops.canonical.com/oops.py/?oopsid=1681EB2995
<lifeless> the last query that completed is at 960ms into the request, and has no limit - its the one you want to convert to get a count.
<lifeless> the distroseries is 103 based on another place in that page that has the literal
<sinzui> That is the one I was looking at
<sinzui> I think it is getting packaging, spn, product_series, product) because in most cases, you want to see all the items...but most cases also make the request with a batch navigatot
<sinzui> lifeless, ^
<lifeless> so, that makes sense to me
<sinzui> lifeless, The property must use a decorated resultset too. It is iterating over *all* the items
<lifeless> it does, or it should ?
<sinzui> It should I think
<lifeless> makes sense to me
<lifeless> whats the property called ?
<sinzui> distroseries.packaging
<sinzui> lifeless, last year this property only had to content with a few hundred packages. It is time to review all the callsites and decide wha the model really needs to provide
<sinzui> s/content/contend/
<lifeless> thanks
<lifeless> yes, I agree
<lifeless> scaling  - this is part of my design guidelines, and its past time I wrote them up on the wiki
<sinzui> I almost got rid of distroseries.packaging 6 months ago too :(
<lifeless> I wanted to get some actual concrete fingers-dirty things done before making real suggestions
<lifeless> sinzui: :)
<sinzui> lifeless,  we are down to 4 callsite, 3 of which is the distroseries model itself
<lifeless> so is the attribute the issue
<lifeless> or the table joins etc?
<lifeless> I guess I mean, whichi thing are yo uworking to reduce-the-use-of-and-delete?
<Ursinha> lifeless, email sent
<sinzui> I am wrong, one callsite, just the one getting the count! we can replace it without packaging_count
<Ursinha> (about oops/bug change)
<lifeless> Ursinha: sweet, thanks.
 * sinzui sees EdwinGrubbs move the other callsites to use _all_packagings via specialised menthods
<cody-somerville> is '++profile++' the name of soom tool?
<cody-somerville> *some
<spiv> It's a magic URL path segment you'll be able to add to Launchpad URLs to make it generate profiling data.
<spiv> (once the relevant work lands, if it hasn't already)
<poolie> lifeless: hi, yes, let's
<poolie> scutwork of expenses and sysadmin is done
<poolie> i  wonder if paused kvms survive reboots?
<poolie> probably not
<lifeless> The defn does
<lifeless> the instance doesn't
<lifeless> spiv: everything but the glue has landed
<cody-somerville> Are there any examples in Launchpad where the privacy of an object is determined by an 'owning' object?
<poolie> lifeless, as of last night, i put that null object into the testrequest and that had a trivial failure
<poolie> so let's try again
<lifeless> cody-somerville: not polished ones, no.
<lifeless> cody-somerville: but private bug attachments will be kindof this soon
<lifeless> cody-somerville: and branches already are via the 'branch policy' stuff
<lifeless> poolie: cool; I has caffeine and I has toast cooking
<cody-somerville> Thats not really what I meant
<cody-somerville> although branch policy describes the default privacy for a new branch, privacy checks aren't actually delegated to it
 * poolie tries again
<poolie> have you seen any deprecation warning failures in bzr builder to do with 'debian' vs 'debian_bundle'?
<lifeless> cody-somerville: sinzui's team will be working on something very much like this soon
<wgrant> cody-somerville: P3As do that sort of thing.
<poolie> or is that just out of date sourcecode or something?
<wgrant> All the publications and builds within the P3A inherit their privacy from it.
<lifeless> poolie: jelmer did something related to that recently
<cody-somerville> wgrant, Is this a feature inherent to zope or something the soyuz folks bolted on?
<wgrant> The security adapters just delegate to the archive.
<wgrant> You could write a security adapter which just by default delegates to the parent, but that's not the default.
<wgrant> In a normal Zope app that would work fine, since everything has a parent.
<wgrant> Not so much in LP.
<cody-somerville> wgrant, Do you actually write a security adapter or is it all done via zcml?
<lifeless> god no
<wgrant> cody-somerville: One just needs to throw an adapter in lib/canonical/launchpad/security.py. No ZCML required.
 * ajmitch is reminded too much of acquisition at the moment
<wgrant> ajmitch: Hm?
<lifeless> cody-somerville: its a four letter word around here :P
<ajmitch> though that's more the containing object
<lifeless> ajmitch: its more uncles
<poolie> how do you typically run tests?
<poolie> make check?
<poolie> or bin/test?
<wgrant> poolie: The former runs everything, so you can't use it.
<poolie> ?
<wgrant> You want to use bin/test, unless you want to run the whole test suite locally, in which case you might be crazy.
<lifeless> poolie: testr run -- -t PATTERN
<poolie> cos.. it'll take a long time?
<wgrant> poolie: ~4 hours on a fast machine.
<poolie> that's ok
<wgrant> I just 'bin/test -1cvvt [sometestspec]'
<wgrant> Ah.
<poolie> i have a decently fast machine
<poolie> i don't see a good reason to run it on ec2 and leave this idle
<poolie> but also, i want to run specific tests too
<wgrant> If you want to run the whole thing, do use 'make check'. It wraps things in xvfb-run so you don't get Firefox windows popping up everywhere.
<poolie> on a separate vm
<lifeless> wgrant: so does testr run :)
<wgrant> lifeless: Hm, so it does.
<wgrant> Handy.
<jtv> The good thing about waking up at decent times for my timezone is the first thing I see online is my antipodean friends.  The bad thing is: there is a risk that they may be talking windmill.
<lifeless> wgrant: so what do you think about doing that query change?
<wgrant> lifeless: I was hoping that stub could come up with something better.
<wgrant> But apparently not.
<lifeless> poolie: ok, so where does that leave us - are you making headway, or fighting friction?
<lifeless> wgrant: I'm about to start hacking on another branch; you're better equiped to do that one, but I don't want to ignore the high-oopser
<lifeless> wgrant: which is why I'm looking for either 'I'm on it' or 'not me' from you :)
<poolie> still moving forward
<wgrant> lifeless: I'm on it.
<lifeless> wgrant: \o/
<poolie> hm apparently 2GB is not enough to run the tests
<wgrant> lifeless: Can you please explain analyze these two queries a few times on staging? http://paste.ubuntu.com/475721/
<lifeless> http://pastebin.com/AT04PS1P
<wgrant> WTF
<sinzui> poolie, I run the whole suite in 2G on a dual core 1.8Gh
<lifeless> sinzui: so this is the problem line ?
<wgrant> lifeless: http://paste.ubuntu.com/475724/
<lifeless>    <dt id="recently-linked" tal:condition="view/recently_linked">
<lifeless> ?
<sinzui> before that
<lifeless> sinzui: its my first time (in many years) debugging from the pt down, so please excuse my fail :)
<sinzui> lifeless, count view/num_linked_packages >> len(self.context.packagings)
<lifeless> ok
<lifeless> and that would be found by looking at the view class
<lifeless> yes?
<lifeless> (just so I know how the dots are joined)
<sinzui> lifeless, the line you were looking at is intended to return the 5 latest packagings...I noted that we may need an index on datecreated too
<sinzui> lifeless, yes, I see the view class doing len()
<lifeless> wgrant: http://pastebin.com/1mdW53Yv
<lifeless> sinzui: ok, cool.
<lifeless> sinzui: I'll have a poke
<wgrant> lifeless: Aha, thanks.
<wgrant> The query planner is stupid.
<lifeless> wgrant: remember that staging stats can be different to prod too, but yes - thats what that bug seemed to say, to me.
<lifeless> wgrant: do you think you've got a better offering than danilos?
<wgrant> lifeless: I'm trimming a couple of other bits out of the query that don't make sense.
<lifeless> ah, ye old holistic approach
<lifeless> sinzui: this looks odd to me
<poolie> sinzui: my mistake, the vm only had 1gb
<poolie> and no swap
<lifeless> results = results.order_by(Desc(Packaging.datecreated), SourcePackageName.name)[:5]
<lifeless> oh foo, ignore me.
<lifeless> ECAFFEINE>
<poolie> would it make any sense to share ~/launchpad across VMs over nfs?
<lifeless> no
<lifeless> test isolation is - and I use the word advisedly - terrible in Launchpad
<mwhudson> bzr doesn't work across nfs, remember!
 * mwhudson hides
<lifeless> mwhudson: :P
<poolie> :-P
<lifeless> might make sense to share your repo and use a checkout over nfs
<lifeless> the sourcedeps caches are pretty small and I wouldn't stress about them
<mwhudson> well
<mwhudson> my download-cache is twice the size of my launchpad repo
<lifeless> lightweight checkout time ;)
<mwhudson> and the eggs dir about 1.5x the size
<lifeless> its actually a case where that works well
<mwhudson> i guess that would save about 50% yar
<lifeless> the eggs dir I have to nuke regularly
<poolie> i have a 1.5GB download cache
<poolie> how do you clean it up? just randomly delete stuff?
<mwhudson> that seems very large
<mwhudson> mine is 415 megs and is a full branch i think
<lifeless> rm -rf eggs/*; make build (but of course be sure to delete a couple of things I never remember because the Makefile + buildout combo is ... traumatic
<mwhudson> oh eggs can get huge
<poolie> 611MB eggs; 470MB sourcecode; 416MB download-cache (of which 217MB .bzr)
<lifeless> sinzui: whats a good test case to run to be sure I haven't broken this ?
 * sinzui thinks
<sinzui> lifeless, I do not see a single test for num_linked_packages
<lifeless> sinzui: well, anything that uses 'packagings' will be fine.
<sinzui> lifeless,  lp/registry/doc/distroseries.txt has 3 tests
<lifeless> thanks
<lifeless> http://pastebin.com/xUwxKGC1
<lifeless> hah, forbidden attribute len()
<poolie> ok, so with all eggs deleted, i get 'no module ZConfig'
<poolie> i guess it depends on actually having some eggs present?
<lifeless> yes
<poolie> :/
<lifeless> as I said - traumatic
<lifeless> you need to delete the generated files from bin/
<lifeless> (bzr can tell you those)
<lifeless> and then do python ./bootstrap
<lifeless> and then 'make build
<poolie> bin in my branch, or elsewhere?
<lifeless> in your working tree
<sinzui> lifeless, I do not think we need .packagings any more. No callsite wants the data returned. The only callsite wants count
<lifeless> sinzui: \o/ tests pass
<lifeless> sinzui: whats the page that shows it ?
<lifeless> just any distroseries ?
<lifeless> Upstream packaging
<lifeless> 3  source packages are linked to registered upstream projects. 1  need linking.
<lifeless> \o/
<sinzui> any ubuntu or debian series
<lifeless> sinzui: well, this was very shallow, so I'll push it up and we can see ;)
<sinzui> fab
<lifeless> sinzui: I don't know enough to know what the right/wrong fix is here
<lifeless> other than that this should help
<poolie> ok, re-bootstrapping
<lifeless> sinzui: https://code.edge.launchpad.net/~lifeless/launchpad/registry-packagings/+merge/32164
<sinzui> lifeless, r=me
<poolie> lifeless: so if 'testr run' mentions no failures, there were no failures?
<lifeless> sinzui: cool
<lifeless> poolie: thats right
<lifeless> sinzui: do you think this is rc ?
<poolie> what's the invocation to re-run failures?
<lifeless> testr run --faling
<lifeless> bah
<lifeless> testr run --failing
<sinzui> lifeless, yes.I think it is RC. I expect it is also a CP candidate since we really want users seeing the distroseries page
<lifeless> well as rc I can shoot it off now and it should be in the rollout ?
<poolie> ok now http://pastebin.ubuntu.com/475731/ is strange
<lifeless> poolie: how so ? the fact its failing?
<poolie> yes
<poolie> maybe it's not my fault
<poolie> how would i re-run just a doctest? -t views_txt doesn't seem to work
<poolie> maybe with a dot?
<lifeless> teset
<lifeless> yes
<lifeless> e.g. -t doc/views.txt
<lifeless> or whatever
<poolie> do you have any guesses about the memcached test failure?
<lifeless> none sorry
<lifeless> it looks like it uses a subclassed test or something ?
<lifeless> sinzui: throwing it at ec2
<lifeless> subclassed view I mean
<sinzui> woot!
<poolie> maybe memcached is sad? i'm also getting "WARNING:root:Memcache set failed"...
<poolie> just thinking aloud here
<sinzui> poolie, I think that means the test needs to run on the LaunchpadFunctionalLayer
<sinzui> Oh, it is on the correct layer
<sinzui> poolie, Did you change something on the distroseries-needs-packaging.pt page?
<lifeless> sinzui: he changed base-layout
<sinzui> I do not think that should have any impact on the test. The test does not care about layout/markup.
<sinzui> poolie, when you saw ""WARNING:root:Memcache" did you see the the memcache layer was brought up before the test?
<lifeless> poolie: you can do 'subunit-ls<.testrepository/$testid' to answer that question
<jtv> lifeless: would it be easy to implement the fake librarian as a test resource?  I could install it from setUp/tearDown but it'd be such a chore.
<lifeless> jtv: we don't have testresources glued in with layers yet
<lifeless> jtv: so, setUp/tearDown will be easier, for now.
<lifeless> afk for a bit
<jtv> bummer
<poolie> sinzui: it did say the layer had been started up
<sinzui> poolie, I see the test requires LaunchpadFunctionalLayer. I have see the warning reported when authoring a test on the wrong layer
<sinzui> poolie, memcache layer is brought up just before the LaunchpadFunctionalLayer...
<mwhudson> do we need another scummy loop to make sure memcache has started accepting connections?
<sinzui> poolie, I think (speculate) that I have seen memcached startup/shutdown contentions between tests. I think I saw memcached not come up because it was not fully torn down in a previous test
<sinzui> I saw it only once, and the issue was only on my machine when I was refactoring
<poolie> so maybe it's something about the specific tests i was asking for
<sinzui> :(
<sinzui> poolie, you may want to try --layer=LaunchpadFunctional to restrict tests to a certain layer
<poolie> that runs only the tests in that layer?
<sinzui> correct
<poolie> ok
<poolie> i'm now trying to just run everything
<poolie> everything seems to be passing
<sinzui> :)
<poolie> though who knows what will fail later
<poolie> and i found another tribunal bug :)
<poolie> it should set stdin into nonblocking mode to cope better with slowly-fed pipes
<lifeless> jtv: did you get the mail 'Translation template import - typewriter in pidgin-typewriter trunk' just now ?
<jtv> lifeless: I don't think I did.
<lifeless> forwarded
<lifeless> its a little worrying that it exposes all the email addresses
<lifeless> and
<lifeless> that we're getting it at all :)
<jtv> That was a little side job I didn't quite get around to doing at Epic.
<jtv> We want to disable these success notices for Ubuntu imports at the least.
<poolie> i get a lot of these
<poolie> i don't know about yours in particular
<jtv> lifeless: I believe the exposure of co-recipients happens in simple_sendmail
<poolie> jtv: i just happened to notice that there's what seems to be translations exported metadata inside 'virsh help domname'
<poolie> actually i'll just file, or look for, a bug
<jtv> poolie: sorry what is "virsh help domname"?
<lifeless> virsh is a tool for managing kvm stuff
<lifeless> domname is a help topic in the tool
<jtv> and who is domname?
<poolie> i think it's a bug in either their use of gettext or our packaging
<jtv> Then that would probably be a bug in libvirt-bin
<lifeless> we're down to 52 timeout bugs
<lifeless> < 2 per developer!
<jtv> (or rather, whatever source package it came from)
<cody-somerville> lifeless, you rock! :)
<lifeless> cody-somerville: not I
<lifeless> cody-somerville: I've done 2 I think
<cody-somerville> lifeless, I was just commenting on how you rock :P
<lifeless> cody-somerville: aw shucks, well thanks!
<poolie> or 52 per lifeless
<poolie> so with all of the tests running, i don't seem to see that memcached failure
<lifeless> poolie: cool
<poolie> lifeless: with that null object added, 0 failures out of 2280 so far
<lifeless> awesome
<poolie> want to give me an incremental review?
<lifeless> yes
<lifeless> r=lifeless
<lifeless> except as its not release-critical, it can't be landed for the moment ><
<poolie> ok
<poolie> i might take a break and let this keep grinding
<lifeless> kk
<lifeless> wgrant: around/
<wgrant> lifeless: Just wandered into uni.
<wgrant> So, yes.
<lifeless> hey
<lifeless> so
<lifeless> I mailed you an oops-extract
<lifeless> I was wondering if you could quickly eyeball me and tell me if its still relevant, or if something done in soyuz recently has 'made it all better'
<wgrant> My eyes are bleeding.
<wgrant> That... that query.
<wgrant> So, I'd say that's still problematic.
<wgrant> And probably related to the model changes a couple of months ago.
<wgrant> Nothing there has changed recently.
<lifeless> the query at the top isn't particularly bad
<lifeless> its death by a thousand cuts territory
<wgrant> It's always been a problematic page.
<lifeless> do you remember the template name ?
<lifeless> nvm I see it
<lifeless> more relevantly, are their unit tests of it ?
<wgrant> Of the view?
<wgrant> Very unlikely.
<wgrant> Only Code tends to do those, AFAIK.
<lifeless> s/their/there/
<lifeless> wgrant: many places do fwiw
<lifeless> not directly, but via publication, using unit test frameork though
<wgrant> Hm, so they do.
<lifeless> in fact, test_archive_packages == soyuz one
<wgrant> Indeed, there are a few.
<wgrant> But it's tiny.
<lifeless> also, if they drive the view without publication its an invalid test
<lifeless> which is higher up the stack
<lifeless> LPFunctionalLayer
<lifeless> or DBFunctionalLayer ?
<wgrant> lifeless: So, the whole getBuildRecords stack makes me somewhat sad. Code duplication, manual SQL construction, etc. I've a quick and ugly fix for the slow query now, but I'll rewrite it all next cycle.
<lifeless> wgrant: \o/
<wgrant> lifeless: LP == DB + librarian + memcached
<wgrant> functional = Zope
<wgrant> zopeless = Zopeless Zope
<wgrant> I think those are the main ones.
<spiv> I should have called "zopeless" "zope on a rope".  It would still be inaccurate, but at least it would make me laugh.
<wgrant> spiv: Was it ever actually Zopeless?
<lifeless> wgrant: it is zope--
<spiv> Kinda.
<spiv> Once upon a time it avoided loading all the config and zcml and whatnot.
<wgrant> Right. That's what I thought it would do, when I first came upon the name.
<lifeless> whats the thing to get a test browser again ?
<spiv> I think that was back when the paint in cave paintings was still wet.
<wgrant> lifeless: self.getUserBrowser
<lifeless> spiv: so, yesterday then ?
<lifeless> spiv: :P
<lifeless> wgrant: any tips on test helpers to populate a ppa?
<lifeless> wgrant: I want to trigger the repeated queries seen in https://edge.launchpad.net/~kalon33/+archive/experimental-stuff/+packages
<wgrant> lifeless: See SoyuzTestPublisher. james_w is in the middle of an API change, but the old one will be around for a while.
<wgrant> Although there might be enough stuff directly in the factory now.
<wgrant> Let's see.
<wgrant> lifeless: STP.getPubBinaries will be more useful if you want sources, binaries and builds (which is probably the case). If you just want sources, LOF.makeSourcePackagePublishingHistory would work.
<lifeless> thanks, I'll see if I can get a failing test up with those hints
<lifeless> object.canonical_url is the way to get a canonical url, right ?
<wgrant> canonical_url(object)
<lifeless> blah
 * mtaylor decides to just be annoying and complain about zcml for no reason
<lifeless> mtaylor: I think you need to actually make a complaint for that to make any sense at all
<mtaylor> lifeless: oh - I was just complaining about the existence of zcml in the first place
<lifeless> mtaylor: it solves a problem
<mtaylor> lifeless: yes. it does.
<lifeless> mtaylor: you didn't have enough things to complain about!
<mtaylor> lifeless: this is a strong, but potentially correct assertion :)
<mtaylor> gah! python-software-properties should have been in base
<wgrant> mtaylor: Yes :(
<wgrant> add-apt-repository is handy.
<mtaylor> it's fantastic
<mtaylor> of course ... don't even get me started that to install add-apt-repository one has to install python-software-properties...
<lifeless> wgrant: self.getPubBinaries ?
<wgrant> lifeless: You need a SoyuzTestPublisher.
<wgrant> Which is in lp.soyuz.tests.test_publishing
<lifeless> ay
<lifeless> is making a new series feasible
<lifeless> or should I just use hoary?
<wgrant> lifeless: http://bazaar.launchpad.net/~wgrant/launchpad/bug-590708-getBuildsByArchIds-timeout/revision/11317 is my quick fix. It does the same thing we do in most of the rest of the Soyuz queries -- precalculate the archive IDs (making use of the distribution/purpose index), and stick them into the query, preventing the planner from fucking us over.
<lifeless> I'm cloning from the top of xx-archive.txt which only makes me slightly dirty
<wgrant> lifeless: Use hoary for now.
<wgrant> james_w is fixing everything to let us not use sampledata, but it's not quite there yet.
<StevenK> My problem with that is that almost all of the Soyuz tests use sampledata :-(
<wgrant> Yes :(
<lifeless> wgrant: +1
<lifeless> wgrant: is the fakeChroots thing etc needed?
<lifeless> wgrant: I see 5 lines of boilerplate in xx-archive.txt and and don't want to copy unneeded boilerplate
<wgrant> lifeless: I wonder if you might not get any builds if you don't add them.
<wgrant> But try without.
<lifeless> so, just construct and use ?
<wgrant> In order to get the right binaries, you may have to prepareBreezyAutotest and use breezy-autotest instead.
<wgrant> Different tests do different things for different reasons :(
<lifeless> wgrant: I'm totally list
<lifeless> lost
<wgrant> lifeless: STP.prepareBreezyAutotest does 'stuff'.
<wgrant> The 'stuff' may be necessary to get the right binaries.
<wgrant> I'm not quite sure.
<lifeless> ok, I'll just copy cprovs stuff :)
<wgrant> lifeless: So, I'd like to confirm that the query still performs sanely. For that I need the archive IDs.
<lifeless> shoot
<wgrant> SELECT archive.id, archive.name FROM archive JOIN distribution ON archive.distribution = distribution.id WHERE distribution.name = 'ubuntu' AND archive.purpose IN (1, 4);
<lifeless>    1 | primary
<lifeless>  534 | partner
<wgrant> Thanks.
<lifeless> you could do this with a subselect and an IN clause, just saying.
<wgrant> The query planner disagrees.
<lifeless> oh!
<lifeless> thats fail
<lifeless> though I'm a natural born sceptic here
<lifeless> I would suspect a correlated subquery-by-mistake or something like that
<wgrant> That's the whole problem with the query.
<wgrant> lifeless: http://paste.ubuntu.com/475767/
<wgrant> analyze plz.
<lifeless> wgrant: cold cache this one is a bitch
<wgrant> Of course.
<lifeless> suggests we have more work to do is all
<lifeless> Index Scan Backward using buildfarmjob__date_finished__idx on buildfarmjob  (cost=0.00..71787.87 rows=103051 width=12) (actual time=0.118..246.771 rows=778 loops=1)
<lifeless> is the one that seems to be painful
<wgrant> Still, timing-wise it's 30 times better than the current one.
<wgrant> And I may be able to remove the date_finished thing later.
<wgrant> Its purpose is limited.
<cody-somerville> how do you interpret the cost= bit?
<lifeless> cody-somerville: its a scale-less estimate
<lifeless> all the costs are on the same scale, but the scale can't be compared outside the explain
<StevenK> It looked at over 103k rows?
<wgrant> That's the expected.
<lifeless> StevenK: well, the stats made it thought it would
<poolie> one more failure, a very specific test about base-layout
<poolie> should be shallow
<lifeless> poolie: \o/
<wgrant> First parenthesised section is what the planner thinks, second is what actually happened.
<poolie> with 6168 run
<lifeless> wgrant:
<lifeless> ERROR: lp.soyuz.browser.tests.test_archive_packages.TestPPAPackages.test_ppa_packages_query_limit
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/soyuz/browser/tests/test_archive_packages.py", line 111, in test_ppa_packages_query_limit
<lifeless>     test_publisher.getPubBinaries('binary1', distroseries=hoary, archive=ppa)
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/soyuz/tests/test_publishing.py", line 303, in getPubBinaries
<lifeless>     build.builder = builder
<lifeless> Unauthorized: (<BinaryPackageBuild at 0xda2c250>, 'builder', 'launchpad.Edit')
<lifeless> ------------
<lifeless> should I login as the owner of the ppa first ?
<wgrant> I hope not.
<lifeless> http://pastebin.com/ciEsMcd6
<lifeless> the with_person_logged_in bit was not there when it errored
<StevenK> I didn't think .builder was settable except by celebrity?
<lifeless> so how are you meant to use this test 'helper'
<wgrant> lifeless: See how others use it, I guess.
<wgrant> I suspect the other users might mostly run Zopeless :/
<lifeless> deathrow doesn't even login
<wgrant> Zopeless doesn't need login.
<lifeless> ok, so -who- should be logged in to use this ?
<wgrant> I guess you could just use an admin.
<wgrant> That's what lots of tests seem to use.
<lifeless> how is that spelt ?
<wgrant> login('admin@canonical.com')
<wgrant> There may be a constant for that now.
<lifeless> adding one
<StevenK> I thought one had been added
<mwhudson> there is one
<mwhudson> of course i can't find it now
<mwhudson> login_team('admins') also works i guess
<lifeless>     elif person_or_team.is_team:
<lifeless> AttributeError: 'str' object has no attribute 'is_team'
<lifeless> sadness, person_logged_in -> fail on str
<StevenK> Then obviously it isn't an IPersonRole
<StevenK> I think I'm misremembering that
<poolie> jtv, lifeless, bug 615708 is the one i mentioned before
<_mup_> Bug #615708: help for domname contains Launchpad translations metadata? <amd64> <apport-bug> <maverick> <Launchpad itself:New> <libvirt (Ubuntu):New> <https://launchpad.net/bugs/615708>
<poolie> StevenK: do you have any idea how that^^ could happen in soyuz
<wgrant> Sounds like Rosetta's langpack stuff to me.
<StevenK> Indeed
<wgrant> Soyuz has just about nothing to do with that.
<jtv> No, that's the po file header (which would typically be present anyway, Launchpad involvement or not) being allowed to sneak into somewhere where it's not supposed to be.
<wgrant> Rosetta just sends the files through it.
<jtv> As I said yesterday, looks more like a bug in the source packageâsomething in how it's built.
<jtv> Maybe upstream uses gettext files without headers, and came to rely on that.  Maybe they run the empty string through gettext somewhere and so get the header as a translation.
<lifeless> in short, its a software bug in virsh ? :)
<jtv> lifeless: that was my impression yesterday, and I stick by it.
<jtv> stand by it, rather.
 * StevenK grumbles at initialise-from-parent.txt
<wgrant> StevenK: bzr rm
<StevenK> I did
<StevenK> wgrant: Code from it in a unittest fails
<wgrant> StevenK: Functional vs Zopeless, I bet.
<lifeless> wgrant: thanks, I have enough now to try hooking in HasQueryCount
<wgrant> Or possibly just being logged in as a different user.
<StevenK> wgrant: The line that is failing is hoary['i386'].getPublishedReleases(). But IDistroArchSeries doesn't have a .getPublishedReleases()? WTF?
<wgrant> StevenK: i-f-p.txt doesn't have ['i386'].
<wgrant> Also, that method name is broken.
<wgrant> Kill it.
<StevenK> - >>> hoary_i386_pmount_pubs = hoary['i386'].getReleasedPackages('pmount')
<lifeless> haha
<lifeless> hahaha
<lifeless> hahahahahahaha
<wgrant> lib/lp/soyuz/doc/initialise-from-parent.txt:  >>> hoary_pmount_pubs = hoary.getPublishedReleases('pmount')
<lifeless> Difference: queries do not match: 12 is >= 51
<wgrant> lib/lp/soyuz/doc/initialise-from-parent.txt:  >>> foobuntu_pmount_pubs = foobuntu.getPublishedReleases('pmount')
<lifeless> to show a PPA with 2 binary publications.
<wgrant> StevenK: getPublishedReleases != getReleasedPackages
<StevenK> wgrant: A few lines under that
<wgrant> StevenK: You initially spoke of getPublishedReleases. But the line you quote mentions getReleasedPackages.
<StevenK> Right. wgrant correctly tells me that I need to learn to read.
<wgrant> I stand by my assertion that the method names are fucked.
<wgrant> This may mean that I need to go on a rampage next week and destroy them all.
<lifeless> and 55 for 3 packages
<lifeless> this is going to end badly:)
<wgrant> Indeed it will.
<poolie> omg doctests
<poolie> failing on whitespace differences
<lifeless> actually
<lifeless> what it probably means is that there is -one- key diff
<lifeless> and the whitespace stuff just telling you about all differences
<lifeless> perhaps not, but I found this behaviour very confusion
<lifeless> s/ion/ing/
<poolie> right, that is it
<StevenK> poolie: "Feature"
<lifeless> wgrant: if you're still around ?
<wgrant> lifeless: Of course.
<wgrant> In a shitty tute.
<wgrant> So please, distract me :)
<lifeless> I'd like to do roughly what I did with _all_members for +packaging
<lifeless> which is to make the request count constant for 2 and 3 build pages by populating related data
<lifeless> If you could point me at a couple of entry points in soyuz, that would be great.
<wgrant> +packaging is more Registry.
<lifeless> I don't believe in those lines
<wgrant> So there probably aren't really any Soyuz entrypoints, as such.
<lifeless> the data is generated by soyuz
<wgrant> So I'm not quite sure what you're asking.
<lifeless> hmm
<lifeless> so
<lifeless> I'll follow my noise
<lifeless> *nose*, from the template
 * StevenK wonders if he can get the exception text with self.assertRaises()
<lifeless> it should do that by default
<lifeless> or you could grab my matchesException , mmm, I wonder if thats in testtools yet
<poolie> lifeless: i wonder if we should convert doctests one by one to just use matchers or something
<poolie> just in line, to make it easier
<poolie> it would have made this one easier
<poolie> however
<poolie> my test run failed i guess due to the lack of a $display to run windmill things
<poolie> i'm going to retry with make check and then i think that's it
<lifeless> testr run should have done the display thing for you
<lifeless> be sure to log in with X forwarding in your ssh
<lifeless> it lets it work regardless
<poolie> i didn't use it because i wanted to pipe it out to tribunal
<lifeless> ah right - to get incremental ?
<poolie> incremental progress and errors yes
<lifeless> stub: btw, I've tagged a couple of things dba
<stub> k
<lifeless>  /launchpad-project/+bugs?field.tag=dba or something like that
<lifeless> stub: I need to chat with you to make sure your life won't become a living hell with the RFWTAD LEP
<lifeless> francis pointed out that db patching is a bit labour intensive at the moment, or something
<StevenK> ubuntu['breezy-autotest']
<StevenK> And ubuntutest['breezy-autotest']
<StevenK> Sampledata, YOU (&(^ING SUCK
<lifeless> how does this hang together
<lifeless> archive-packages.pt does
<lifeless> <metal:package-list use-macro="context/@@+macros/source-package-list" />
<lifeless> oh, I see, non_selectable is a bit magic
<lifeless> and -fun- it has a template per row
<StevenK> O hai, I can haz review?
<StevenK> O hai, I can haz review?
<StevenK> Whoops.
<StevenK> And obviously not
<lifeless> heh
<lifeless> whats up?
<StevenK> lifeless: Tis simple-ish: https://code.edge.launchpad.net/~stevenk/launchpad/switch-ifp-to-unittests/+merge/32073
<lifeless> Its a bit late for me to be confident in checking that
<lifeless> sorry
<lifeless> at a first glance though
<lifeless> you have lots of literal sample data there
<StevenK> So the original doctest :-(
<lifeless> might want to add constants to sampledata.py
<StevenK> Er, so did
<lifeless> remind me
<lifeless> how does storm and __init__ interact
<jtv> danilos: another nice effect of your big variants cleanup is that we're one step closer to not passing pofiles around in the new "recife" methods.
<lifeless> \o/ registry-packagings passed ec2
<lifeless> stub: ping
 * lifeless wtfs class SourcePackagePublishingHistory(SQLBase, ArchivePublisherBase):
<StevenK> Yes, SPPH is ... special
<lifeless> StevenK: btw I really really wish you wouldn't propose things that you're only just working on
<StevenK> lifeless: Why not?
<lifeless> terrible diff gets mailed out
<lifeless> by which I mean useless
<StevenK> lifeless: The last MP I did as WIP?
<lifeless> yes
<StevenK> Does that still mail out a diff?
<lifeless> its not how the system is meant to be used
<StevenK> lifeless: If I want Julian to look at a branch, I need a WIP MP :-(
<lifeless> why?
<lifeless> (obvious answer, don't worry about Julian looking at the branch :P)
<StevenK> Because he wants to view a nice diff in a web page, I guess
<jtv> danilos, henninge_: setCurrentTranslation doesn't really need a TranslationSide argument, so I'm removing that.  Also, IPOTemplate now exposes its TranslationSide but not the traits that go with it.
<StevenK> I'm doing it because Julian asks me, not to annoy you
<jtv> danilos, henninge_: at least, when I get this branch reviewed & landed.  :)
<lifeless> StevenK: you're the only person doing it, its pretty weird. I know you're not trying to annoy; What do we need to change to fix this?
<StevenK> lifeless: Have Julian be less lazy?
<StevenK> I guess? :-)
<henninge> jtv: yeah, try finding a reviewer willing to approve *that*!
<jtv> uh-oh
<henninge> ;)
<jtv> henninge: well no worries, guess who's ocr tomorrow :)
<henninge> no jtv, you can *not* approve your own branches!
<jtv> Â«damnâhow did he guess!?Â»
<lifeless> is there a utility to do what cachedproperty does, but in a function ?
<adeuring> good morninig
<poolie> https://bugs.edge.launchpad.net/launchpad/+bug/615740 sigh
<_mup_> Bug #615740: test_on_merge.py doesn't handle eintry <Launchpad itself:New> <https://launchpad.net/bugs/615740>
<lifeless> are there ppas in the sample data?
<StevenK> yes
<StevenK> cprov has one, for example
<lifeless> ok, cool, I've found code that runs when the packages list is shown :)
<lifeless> except, not on +packages, only on the ppa page itself. wtf
<stub> lifeless: pong
<lifeless> stub: hey
<lifeless> stub: can you do a brief skype call, or is irc more convenient? topic: agile db patches
<lifeless> oh man, thats so annoying; my timeout fix for sinzui missed edge by 1 hour
<stub> I can do either. Typing is generally better for me.
<lifeless> ok, lets go with typing
<lifeless> so, in the new workflow that should start in ~ a week
<lifeless> we'll get rid of edge
<lifeless> and have nondb patches flowing straight through with in-line qa
<lifeless> you know all this ;)
<lifeless> to start doing db patches in a similar manner, I've been assuming it was primarily a matter of figuring out a series of non-downtime db patches and background scripts interspersed with code changes to match
<lifeless> e.g. db, code, db, code etc until the transition is done
<lifeless> not all db patches can be done like this, and so far I've been figuring that its up to the developer to choose between their development time vs doing a single patch in the monthly downtime
<stub> Unless we plan to be cowboying all our DB changes, the main issue is reworking our processes and tools to support that.
<lifeless> so - what things are likely to cause friction or failure
<stub> With replication, non-downtime means data migration and adding/dropping indexes.
<lifeless> could you expand a little, so I think I understand :)
<poolie> how do i fix this:
<poolie>   Getting distribution for 'testtools==0.9.6dev91'.
<poolie> Error: Couldn't find a distribution for 'testtools==0.9.6dev91'.
<lifeless> poolie: update your download-cache and run make build
<stub> Data migration are just python or sql scripts that get run on the master - just data changes like any other script. No real problem, except we need to ensure they get run and tested on staging automatically.
<poolie> lifeless: with update-sourcecode?
<stub> Adding and dropping indexes may well have to be a manual process. We need to run CREATE INDEX CONCURRENTLY on each node, outside of a transaction, and be around to cleanup if things messup. Also need to manage long running transactions as the index build will block on long running transactions.
<lifeless> poolie: I use 'bzr update' in the download-cache directory
<stub> For pretty much all other DB patches, we need to go via the slony tools. Slony locks all the tables, removes all the replication triggers, applies the patches to the nodes one at a time, rebuilds all the replication triggers (which might be different now there might be new columns etc.), and unlocks.
<lifeless> stub: does that involve perceived downtime ?
<poolie> oh for deb dependencies :)
<stub> lifeless: Yes. The lock will never be granted while the appservers are active.
<stub> locks I should say (exclusive lock on every replicated table in the replication set)
<lifeless> stub: ok, so that I understand - we can't e.g. add a column live
<lifeless> stub: but we could create a whole new table, migrate data to it, then start it replicating ?
<stub> Yup. So goal is to minimize that time. There are at least two improvements we can make to our existing tools to lower times, and that is the first step I think.
<stub> Not invoking comments.sql on production (no real need so it is waste), and making security.py more intelligent.
<stub> Yes, we can create a new table live and add it to replication.
<lifeless> ok
<lifeless> so a long term theme will be increasing the refactorings we can do live
<poolie> spiv i was going to try something like https://bugs.edge.launchpad.net/launchpad-foundations/+bug/615740/comments/1
<_mup_> Bug #615740: test_on_merge.py doesn't handle eintr <Launchpad Foundations:In Progress by mbp> <https://launchpad.net/bugs/615740>
<stub> Although it usually would be create table, add to replication, populate rather than create table, populate, replicate
<poolie> oh ffs launchpad
<lifeless> poolie: ?
<poolie> whitespace folding in bug comments is not the kindest thing for python programmers
<lifeless> heh
<lifeless> masochists all
<poolie> anyhow spiv http://pastebin.ubuntu.com/475807/
<lifeless> stub: thanks; so - in summary - its doable for a small set of cases now, and we can start building them out as we go
 * spiv moves to this channel
<lifeless> stub: there is expected to be a high risk of pear-shapedness, so it won't be part of the fast-qa path
<lifeless> stub: and thus we also need some reporting for folk that are doing staged-refactorings so they know when w particular db change has gone live and they can move to their next stage.
<spiv> poolie: looks reasonable to me
<lifeless> stub: is that about the size of it ? [glossing over the details about what we can/can't do today]
<poolie> i'll clarify that comment and interactively test it
<spiv> poolie: although I wonder why make check needs a SIGWINCH handler?
<lifeless> spiv: make check runs the pqm merge check
<lifeless> spiv: which ~ 0 devs run - it has a supervisor process etc
<stub> lifeless: Pretty much. We already do indexes live as necessary, normally without too much grief. The biggest fallout from live changes is occasionally timeout OOPSES when I underestimate how long a lock will need to be held, and breaking staging (eg. creating new DB users and forgetting to replicate that work there).
<spiv> lifeless: this only makes me more perplexed
<lifeless> stub: thanks heaps; I'll write this up somewhere. How would you feel if I encourage people to start exercising this ?
<stub> lifeless: New tables we can do with the existing infrastructure for process and staging. And if the patch is suitable to be applied live, I can do so. I'll need to do it manually, but I don't see any risk on the db side.
<poolie> spiv i can't see where it gets one from
<lifeless> stub: do you think the additional work will be a burden ?
<lifeless> stub: (work for you I mean)
<stub> We can start now, sure. I'm not sure how popular it will be - suck it and see I guess.
<poolie> maybe as a side effect of psycopg2 or something?
<lifeless> stub: cool, thanks!
<spiv> poolie: possibly something like that :/
<lifeless> poolie: +1
<poolie> lifeless: to which bit? the patch?
<lifeless> yes
 * poolie is glad he bought a new desktop cpu/mb :)
<mrevell> Good morning
<stub> lifeless: So do we want a short term 'I'm busy' system? If the appserver is in read-only mode, all requests pop up a 'I'm busy. This page will automatically reload in 60 seconds' screen. This could give us some agility for, say, <5 min db outages without annoying users too much (for some values of 'too much').
<lifeless> stub: that sounds like an improvement on the current read-only thing regardless
<poolie> why auto reload?
<stub> So people can walk away, have a coffee, and their posts have been completed when they get back with no need for retyping.
<lifeless> except for POSTs
<lifeless> :)
<stub> It would need to cope with POSTs. I'm not sure of the JS magic required.
<lifeless> coping with POSTs will run into browser issues
<lifeless> an interaction between the HTTP spec for POST, and browser security - I
<wgrant> stub: Hi.
<lifeless> 'm fairly sure it would be a path to pain
<stub> yo
<stub> If it doesn't cope with posts, then there is no improvement. GET requests already work just fine in read-only mode.
<wgrant> stub: Could you please EXPLAIN ANALYZE http://paste.ubuntu.com/475767/ on a production slave?
<lifeless> gotcha
<stub> I think we already do something similar for OpenID - not sure.
<stub> wgrant: http://paste.ubuntu.com/475813/
<wgrant> stub: Eeexcellent. Thanks.
<bigjools> lifeless: helleau
<lifeless> bigjools: hiya
<bigjools> lifeless: a new storm seems to be in our sourcecode, which is nice, coz it's got some improvements we should be using.  Thought you might like to be aware :)
<lifeless> bigjools: I'm trying to find the code that runs to generate the table of packages on ppa/+packages
<lifeless> bigjools: any tips?
<bigjools> I'm going to go on a rampage to remove unnecessary .count() with .any()
<bigjools> s/remove/replace/
<bigjools> since the latter now strips ordering
 * noodles775 looks
<lifeless> bigjools: thanks for highlighting that; as it happens I knew because I let deryck know he needed it to land his branch - his branch was how we found the bug with the storm cache that is fixed in 0.17
<bigjools> and bool() on SQLObjResultSet now works
<bigjools> \o/
<stub> wgrant: That might start performing bad when there are a large number of unfinished jobs
<stub> wgrant: I think we can make an index to cope if that happens
<wgrant> stub: There are already lots with date_finished NULL.
<bigjools> lifeless: yeah +packages can get slow.
<lifeless> bigjools: right, I'm pulling on it - it goes up by 4 queries per package, more or less
<lifeless> bigjools: but having trouble figuring out what damn methods run :)
<noodles775> lifeless: is it the source-package-list macro that you're looking for (in archive-macros.pt), which uses the view's batched_sources property?
<lifeless> noodles775: no, its down at the model layer
<lifeless> I navigated the templates ok with a little elbow grease
<stub> wgrant:  (((status <> 1) OR (date_finished IS NOT NULL)) AND (status = 2))
<stub> wgrant: Doesn't that just simplify to 'status=2' ?
<stub> Or am I lacking coffee again?
<wgrant> stub: It does, yes.
<wgrant> But fixing that requires a bit more of a rework.
<poolie> lifeless (if any): i think flags-webapp is good now, can you try it again?
<noodles775> lifeless: so you've been through the ArchiveSourcePublications iterator that's used by the batched_sources property then? Hrm. ok.
<wgrant> There's duplicated code and string-based query construction everywhere. I plan to rewrite it.
<stub> Cool. If we can simplify that sort of thing, we can improve things with more specialized indexes if needed.
<lifeless> poolie: try again ? If you mean land it - no, I can't - we're frozen for non-rc till, I think, friday
<lifeless> noodles775: I'm currently looking for getBuildSummaryStatus
<lifeless> poolie: I can do a test-only run, but that should be precisely equivalent to what you just did
<lifeless> noodles775: I found what I needed - ArchiveSourcePublication in the adapters
<noodles775> Great.
<lifeless> ok, so it causes 2 of those per-row queries itself
<lifeless> and man is it indirect, how many getUtility things?
<lifeless> I'll unravel it a bit tomorrow. Objects are great.
<bigjools> lifeless: dude, you can't RC approve your own branches!
<StevenK> Teehee
<StevenK> That's cheating
<thumper> heh
<bigjools> thumper: did you see my comment on the bug you marked qa-untestable?
<thumper> yes
<thumper> but I didn't see how that tested it
<thumper> it didn't start
<thumper> and staging was down
<bigjools> the problem was in the creation - as soon as you hit the button it oopsed
<bigjools> it didn't need to start building
<thumper> I didn't think so
<thumper> I thought it was the fact that it could have a date finished, but not a date started
<thumper> that is what the branch fixed
<bigjools> hmmm actually, I might be mistaken
<lifeless> bigjools: can't rc-review, can rc-approve ;)
<bigjools> lifeless: dude, no!
<bigjools> not if it's your own branch
<lifeless> no? my bad then; where are the docs, I'll get the policy straight for future
<lifeless> rc analysis seems to be totally separate from code review to me
<bigjools> in this case it's fine I would have said it was ok, but generally it's frowned on
<thumper> lifeless: but you can't rc analyse your own
<lifeless> thumper: *why not*
<lifeless> its not like code review
<thumper> why can't you review your own code?
<thumper> yeah, it is
<lifeless> no, its not
<thumper> in this regard it is
<lifeless> *why*
<thumper> judgement is skewed when looking at your own stuff
<bigjools> conflict of interest
<thumper> that is why we have other people look at things
<lifeless> thats a very adversarial way of thinking about both code review and release-critical assessement
<bigjools> if it's a genuine RC you won't have any problems getting it in
<bigjools> there is no rush at this stage
<lifeless> huh? I think you've just segued to a different concern
<bigjools> I would not think of doing a branch, getting someone to review it, and then landing it with my own RC
<bigjools> not really, I'm pointing out that you had plenty of time to get approval from someone else
<lifeless> I didn't think about time-till-release at all in doing this
<bigjools> the new approval process says that TLs must get approval from Francis or other TLs
<lifeless> what new approval process? Only one that I recall was db-query approvals wasn't it?
<bigjools> all out of band prod changes
<lifeless> which was (francis, stub, me: can approve anything, TL's can approve each other)
<bigjools> and you can't approve your own stuff
<lifeless> taking a step back, I can see you feel I didn't something totally wrong
<lifeless> and I don't want to be doing that
<lifeless> however I don't *understand* the assertion you are making
<lifeless> so I want to drill into that
<bigjools> it boils down to the fact that we don't like unilateral changes and rc analysing your own code is a potential recipe for problems.
<lifeless> so there are two things there that concern
<bigjools> you're the first person I know who's RCed his own branch :)
<lifeless> what risks do you see in assessing a branch for rc status at this stage, and how are they affected by who wrote the branch vs who assesses it.
<bigjools> it's the same principle as reviewing your own code, you *will* miss problems that other people see
<lifeless> I object to it being the same principle
<lifeless> I don't think its even vaguely similar
<bigjools> 2 of us disagree with you so far
<bigjools> anyway I don't want to argue with you about this, perhaps you'd like to raise it in tomorrow's TL call
<lifeless> You've both asserted a cultural norm, not made any attempt to expand other than to say 'judgment is in question'
<lifeless> I'm going to read up the process etc
<lifeless> and get well across that
<lifeless> I'll probably send it to the list, I think the more intellects assessing this the better
<thumper> I think it is a matter of culture and policy rather than us saying "in this particular respect"
<bigjools> I am not asserting a cultural norm, I am telling you that it's the same as reviewing your own code
<lifeless> bigjools: And I'm disagreeing with the assertion that it is the same.
<thumper> but I agree with bigjools and the "same as reviewing your own code"
<thumper> two to one, we win
 * thumper leaves
<bigjools> lol
<lifeless> rc analysis is about risk assessment: chance of throwing out the release vs delaying the fix for a separate QA round
<lifeless> code review is about making the code better, understanding if its clear enough for another person to understand it and maintain it
<bigjools> lifeless: in that case, why don't we let all developers RC their own branches?
<bigjools> what makes you special?
<lifeless> because RC assessment is delegated to the rotating RM, plus jml me & flacoste, AIUI
<bigjools> ah, so it's by convention
<lifeless> yes
<bigjools> or should I say "culture and policy"
<bigjools> which is what you were arguing against earlier
<lifeless> I didn't argue against it, I said I seem to have violated a cultural norm
<lifeless> that in itself doesn't mean its good or bad
<lifeless> just different
<lifeless> and
<lifeless> Would I be happy with every developer being able to make an RC assessment? Actually, I think I would.
<lifeless> however, entirely separately, I must apologise for doing *any* RC approvals
<lifeless> according to https://dev.launchpad.net/PolicyAndProcess/ReleaseManagerRotation its not delegated to *anyone* except the RM
<lifeless> I'd like to understand why
<lifeless> (because it doesn't fit what I've observed happening here in the not-very-distant-before-I-became-TA-past
<lifeless> bigjools: I shall however follow it now that I've reread it and raise it for reevaluation
<bigjools> lifeless: usually the RM and Francis dole out RCs, because they are best placed to see the bigger picture
<lifeless> bigjools: and this makes sense to me, and was why I felt comfortable both handing out one the other day (though I put a caveat saying 'If I can' - which you and the other people on the MP seemed to think was appropiate)
<lifeless>  - and handing out one today, because I have been doing little but watching the bigger picture
<bigjools> lifeless: as long as it's not your own branch I don't mind, but the RM should know about it regardless
<lifeless> bigjools: of course - note that the one I did of mine I had you as an explicit rc-review requested
<lifeless> so that you would
<bigjools> which was good :)
<lifeless> so concretely, you're happy that that particular branch was rc worthy, you're happy that you were in the loop, you're unhappy that I made that assessment *and* wrote the code.
<bigjools> I think it would be safe to say that most if not all of us would be uncomfortable with people giving RCs on their own code.
<lifeless> why? Given what RC /means/
<thumper> lifeless: the whole point is to have more than one set of eyes on the problem
<lifeless> thumper: not all problems are equal. Perhaps a definition of RC would help.
<lifeless> 'worth adding to the QA queue while the trunk is frozen'
<lifeless> thats it. Thats /all/ that rc means.
<bigjools> no, that's completely incorrect
<lifeless> ok.
<lifeless> please help me understand, because thats what the process documentation says it is
<bigjools> RC means we need to land something that is either fixing a critical breakage if we release with it, or adding something that is business critical
<lifeless> I think you have conflated two concepts: why we rc-approve, and what rc-approve *implies*
<lifeless> we rc-approve to fix critical breakage or address a business critical concern
<lifeless> rc approval *implies* that we'll have more QA coming in rather than just reducing the QA so that the release can happen
<lifeless> reasons *why* we do something are motivation. The impact and effect of doing it are risk and rewards.
<lifeless> assessing whether something should be rc approved is balancing the why - the desired reward - against the risk and the expected actual reward
<bigjools> yes, but the motivations to do it are as I stated.  The person giving RC needs to assess how risky the change is.
<bigjools> and now, I am done talking about this.
<lifeless> ok
<lifeless> its annoying how chromiums search box is on top of the moin search box on the wiki
<lifeless> gnight y'all
 * thumper throws in the towel
<bigjools> see y'all
<deryck> Morning, all.
<jtv1> Yay!  staging is back.  Let's Q/A, people!
<wgrant> Finally!
<wgrant> Of course that doesn't mean the code isn't ancient.
<wgrant> OOPS-1683S339
<wgrant> Can I have a statement log for that?
<wgrant> It's really quick now. I guess I could just blame a cold cache.
<jelmer> wgrant: Did somebody get the statement log for you for that OOPS yet?
<wgrant> jelmer: Not as yet.
<wgrant> OOPS-1683S348 also concerns me, and is still happening.
<wgrant> (I'm trying to QA one of my revs)
<jelmer> wgrant: I'll get them to you in a sec.
<wgrant> jelmer: Thanks.
<bigjools> stub: still around?  I have problems with the PG 8.4 upgrade
<stub> wazzup?
<bigjools> stub: I installed the 8.4 packages, did the config changes then ran utilites/launchpad-database-setup
<bigjools> and I get:
<bigjools> createuser: could not connect to database postgres:
<bigjools> after it's done the config change
<bigjools> the server is running AFAICS
<stub> There is another error message after the one you quoted (the reason it couldn't connect)
<bigjools> could not connect to server: No such file or directory
<stub> That makes it sound like it isn't running. You shut it down, edited the configs, restarted yes?
<bigjools> /var/run/postgresql/.s.PGSQL.5432 is not there, but /var/run/postgresql/.s.PGSQL.5433 is
<bigjools> yep
<bigjools> your port instructions on the wiki page conflict with the ones in that email BTW
<stub> So look in /var/log/postgresql/postgresql-8.4-main.log - should be an error message in there saying why it isn't running.
<bigjools> it's running ok
<bigjools> just using port 5433 from the looks of it
<bigjools> but the client expects 5422
<stub> Yes, which is why we edited the config files to make 8.4 listen on 5432
<bigjools> gar
<bigjools> missed that bit :(
<stub> ;)
<bigjools> https://dev.launchpad.net/DatabaseSetup needs updating then
<bigjools> it says to change 5432 into 5433
<bigjools> stub: aha, I see what's going on
<bigjools> launchpad-database-setup is changing the port to 5433
<stub> bigjools: Look closer - that section is about upgrading, so matches my email (edit pg8.3's config to use a different port, so 8.4 can use 5432)
<bigjools> so I hadn't missed it :)
<bigjools> ah ok
<bigjools> launchpad-database-setup is buggering things up though
<stub> So the wiki page documents reconfiguring the old version before the installing the new packages, and my email documents reconfiguring after installing the new packages.
<stub> Dunno why launchpad-database-setup would be doing that :-P
<jelmer> maxb, ping
<maxb> jelmer: pong, but about to disappear for lunch
<stub> bigjools: Did you edit your 8.3 postgresql.conf too?
<maxb> oh, was just replying to bigjools email
<jelmer> maxb: ah, ok
<stub> bigjools: And bounce 8.3 per the email?
<jelmer> maxb: I was just wondering if you might be able to have a look at my open meta-lp-deps reviews
<bigjools> stub: yes, I took it down
<bigjools> entirely
<stub> And the config has been edited to say port=5433 ?
<jelmer> maxb: in preparation of uploading a new version of meta-lp-deps, but from what I read now that process already seems to be in motion..
<stub> bigjools: The 8.3 config I mean
<bigjools> stub: no, I left it alone since I didn't intend to start it up (I wanted to purge the configs but the package dependencies foiled me)
<stub> Because it looks like launchpad-database-setup rebuilds the 8.4 cluster, and if the postgresql tools see another database configured to use a port it will choose another.
<bigjools> yay :/
<maxb> jelmer: I'll look. AFAIK the current blocker is unresolved questions about what to do w.r.t. pocket-lint and rabbitmq on hardy
 * maxb gone
<jelmer> maxb: pocket-lint should no longer be an issue (we've uploaded a package for hardy), not sure about rabbitmq
<maxb> bigjools: Are you all settled on pg8.4 now?
<bigjools> maxb: I am :)
<bigjools> 8.3 and configs is history
<wgrant> When's production moving to Lucid?
<bigjools> RSN
<maxb> What's the ordering here? => pg 8.4, then later => Lucid?
<maxb> jelmer: you saw my needsinfo review of https://code.edge.launchpad.net/~jelmer/meta-lp-deps/chardet-dep/+merge/31260, right?
<jelmer> maxb: Yep, thanks
<maxb> Also, in the spirit of Robert's email, I should point out that no one has said I can review meta-lp-deps merges. Though in this case I _know_ that the process is that there is no defined process.
<jelmer> maxb: That's a good point. You had been doing reviews (and are the registrant of lp:meta-lp-deps) so I assumed that was the proper procedure.
<maxb> https://dev.launchpad.net/LaunchpadPpa says "Exercise personal judgment on whether your change merits a merge proposal, or is sufficiently trivial to just be committed directly." -- and I wrote that
<jelmer> maxb: That doesn't say anything about who should do the review :-)
<maxb> That seemed in keeping with the fact that I wasn't given any requirements to seek review of any kind when I was issued with ~launchpad/+archive/ppa upload permissions
<wgrant> It doesn't even say there should be a review, just an MP :P
<jelmer> maxb: Anyway, these changes are quite simple but I would prefer if somebody else could give them a quick glance regardless. There are two MPs left.
<jelmer> wgrant: Well, a proposal implies that there is going to be acceptance or a declination somehow.
<henninge> danilos, jtv: POTMsgSet.submitSuggestion does not sanitize translations. Is that intended?
 * henninge vaguely remembers something like that.
<danilos> henninge, what does "sanitize" translations mean? calling old "sanitizeTranslations"? :)
<jtv> henninge: for setCurrentTranslation it was a documented assumption, but I don't think we discussed it for this one.
<henninge> danilos: No, calling the new code.
<danilos> henninge, oh, right, we've split that out already, right?
<danilos> henninge, jtv: I don't think it should sanitize input
<henninge> danilos: yes, I did that in a hotel room in Recife ;-)
<danilos> henninge, jtv: it's something that callsites should do
<danilos> henninge, when it was four of us squeezing together? oh yes, I remember
<henninge> danilos: ok, so in tests I should make sure to pass sanitized input.
<danilos> henninge, put that into a factory method ;)
<jtv> The problem IIRC was that "sanitized" was actually relative to the input mediumâdifferent de-indentation requirements for browser and import, etc.
<danilos> henninge, or simply into translations test helpers
<danilos> jtv, exactly
<jtv> Then, there's validation.
<danilos> jtv, also, we store invalid translations as suggestions as well
<danilos> :)
<jtv> But we don't make them current, right?
<jtv> Although that can still happen if a template update adds a flag, I guess.
<danilos> jtv, yeah, that's an old bug
<danilos> jtv, but unrelated to this :)
<jtv> just saying :)
 * bigjools -> lunch
<abentley> bigjools, jelmer: Does soyuz.browser.build.CompleteBuild serve any purpose nowadays?  AFAICT, we don't use its self._buildqueue_record attribute at all.
<jelmer> abentley: I'm not sure, I
<jelmer> 'm not very familiar with that class.
<jelmer> noodles775: Do you happen to know perhaps?
<maxb> https://code.edge.launchpad.net/~maxb/meta-lp-deps/pg8.4/+merge/32200  <-- if someone has a moment
<noodles775> abentley, jelmer: It's caching the buildqueue_record with the batched builds, and seems to be used in soyuz/templates/builds-list.pt:96?
<jelmer> maxb: Reviewing
<abentley> noodles775, ah.  I missed that buildqueue_record method.  (Which surely ought to be a property?)
<noodles775> Looks like it certainly should, but zope templates apparently don't care.
<abentley> noodles775, it was causing me grief because it delegates to IBinaryPackageBuild, but setupCompleteBuilds was wrapping it onto SourcePackageRecipeBuilds.
<maxb> Who is the best person to ask to push for getting the launchpad ppa / hardy / rabbitmq situation resolved? Basically it needs packages copying from the internal "cat" archive
<abentley> maxb, copied *to* the cat archive?
<maxb> *from*
<abentley> maxb, to what?
<maxb> to the public launchpad PPA -  the problem is that launchpad-dependencies depends on rabbitmq, which is satisfied via the cat archive, but makes launchpad-dependencies uninstallable unless you have access to cat
<abentley> maxb, I assume you mean on older distros, because I don't use the cat archive and I have launchpad-dependencies installed.
<maxb> Correct, this is only an issue on hardy
<abentley> maxb, elmo is the usual guardian of that archive.  mars might also be interested.
<mars> maxb, I can file an RT for you.  It is rabbitmq for hardy, right?  Do we still support hardy in the PPA?  I thought we skipped to the next LTS?
<maxb> I believe there's an email from flacoste stating that we ought to support what production is running in the PPA
<jelmer> maxb: FWIW the landscape folks have a backport of rabbitmq to hardy in their PPA.
<maxb> lifeless asked mthaddon about this on launchpad-dev@, but the thread died
<mthaddon> maxb: the request has been filed, we're just working on other stuff (based on priorities that we've agreed with the LP devs)
<mars> mthaddon is usually swamped, and rarely checks mail unless it is an RT
<mthaddon> iow, it should get done at some magical time in the future when we have time to do it :)
<maxb> mthaddon: ok. That bit of info didn't make it into the public dev list :-)
<maxb> Meanwhile, we can't/shouldn't update the launchpad-dependencies version in hardy in the PPA, unless we make a hardy branch that omits rabbit
<salgado> sinzui, do you know what external services/scripts use xmlrpc.launchpad.net?
<sinzui> salgado, I do not
<salgado> flacoste, do you? (^)
<james_w> salgado: bazaar at least
<salgado> yeah, I thought bazaar would be one of its users.  I'm trying to find all of its users to see if we'd need it for vostok, in order to make sure https://code.edge.launchpad.net/~salgado/launchpad/layer-specific-navigation/+merge/32124 is not going to be a problem for us
<salgado> james_w, btw, it'd be great if you could have a look at the cover letter there and tell me what you think
<jml> salgado, maybe your best bet is to ask a losa for the logs?
<salgado> indeed, that should work.  thanks jml
<james_w> salgado: I think that if we do want xmlrpc to support any existing clients then we likely won't need to change the navigation
<james_w> salgado: would it be feasible to have a vostokxmlrpc layer if we needed?#
<salgado> I think so
<jml> salgado, how do I look at the apidocs?
<salgado> jml, apidocs.launchpad.dev
<jml> salgado, and that starts from "make run"?
<salgado> yep
<jml> is there a way to do a static build of them?
<salgado> jml, yes, gary has one at https://devpad.canonical.com/~gary/apidoc.launchpad.dev/++apidoc++/static.html
<jml> salgado, do you know how to make one of those?
<jml> hmm. it looks much better from the live site.
<salgado> jml, I think gary pointed something like wget at apidocs.lp.dev/static.html to generate that
<jml> heh, although clicking "Book" makes an oops.
<jml> salgado, ok, thanks.
<james_w> do I need to do anything special to do LFA.read() in the testsuite?
<jml> james_w, maybe be in the librarian layer?
<jml> (I honestly have no clue)
<james_w> It appears to be a problem with something about http_proxy, which is then masked in to an error "raise LookupError, aliasID"
<james_w> jml: yep, already hit that one
<jml> james_w, perhaps you could find out what PGPSignatureNotPreserved in test_uploadprocessor does and copy that
<james_w> jml: doesn't seem to do anything special. I suspect an environment issue.
<jml> james_w, good luck with that. sorry  I couldn't help more.
 * jml goes to purchase some consolation chocolate
<maxb> jelmer: Don't forget to 'bzr mark-uploaded' and push
<leonardr> up-to-date launchpad branch is getting deprecation warning from bzrplugins/builder ("please use 'debian' instead of 'debian_bundle'). anyone know what's up?
<maxb> leonardr: I'll guess that that's been triggered by upgrading to the newer python-debian in the launchpad ppa
<james_w> hmm, no it's nothing to do with http_proxy it seems. It seems like the librarian just doesn't have the file. Maybe there is a commit missing somewhere or something
<bigjools> james_w: you need to commit
<bigjools> or it doesn't write the file
 * bigjools thinks commits are evil in tests :(
<james_w> thanks bigjools
<leonardr> salgado, when i integrate the version of lazr.restful with your change into launchpad i get some test failures. can you take al ook?
<salgado> leonardr, sure, send them my way
* bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<leonardr> salgado, http://paste.ubuntu.com/476022/
<leonardr> any clues?
 * salgado checks
<salgado> leonardr, is that the only failure you got?
<leonardr> salgado: there were a bunch of failures afterward but they were all caused by spec not being defined
<salgado> leonardr, right, so only this test failed, then?
<leonardr> salgado: there was also a failure in cache.txt but i don't think that was your fault
<salgado> leonardr, apparently that's a bug which is only exposed when we export an entry with no fields/operations.  I should be able to get it fixed quickly
<leonardr> salgado: lazr.restful bug? ok
<salgado> yes
<salgado> leonardr, got http://paste.ubuntu.com/476025/ when running buildout on a new lazr.restful branch.  any idea why?
<leonardr> salgado: yeah, benji changed buildout to disallow 'picked' versions of the dependencies. they have to be explicitly named in versions.cfg
<leonardr> i must have had that one installed globally or something, so i missed it
<leonardr> add distributed = 0.6.14 to versions.cfg in the appropriate place and rerun
<salgado> leonardr, I'm seeing a failure on webservice-declarations.txt on trunk.  is that known?
<salgado> leonardr, and http://paste.ubuntu.com/476043/ has a fix for the problem you encountered when running LP's test suite
<mars> rockstar, ping, Tarmac question for you
<rockstar> mars, pong.  Also, #tarmac if I'm not around.
<mars> oh, cool
<rockstar> Er, if I'm ever not around.
<leonardr> salgado: give me the test failure?
<salgado> leonardr, http://paste.ubuntu.com/476065/
<salgado> leonardr, does the fix look ok to you?  would you like me to create a merge-proposal for it?
<jml> o/` Mammal, mammal. Their names are called, they raise a paw, the bat, the cat, dolphin and dog, koala bear and hog! o/`
<leonardr> salgado, i think start should really be 0, so i don't know what's up
<salgado> leonardr, doesn't it fail for you on a trunk branch?
<leonardr> i don't think so, let me try
<leonardr> salgado: no, i get no failures
<leonardr> you are at revno 137?
<salgado> yes, 137.  using python2.6
<leonardr> weird
 * salgado starts a new trunk branch from scratch, just in case
<jml> someone might be interested in reviewing https://code.launchpad.net/~jml/launchpad/readme/+merge/32238
<jml> (not that I can land it)
<salgado> leonardr, it doesn't fail on this branch I started from scratch
<salgado> s/branch/tree
<jml> flacoste, you might actually be interested ^^
<flacoste> jml: reviewed
<flacoste> jml: not that i have any comments apart awesome!
<jml> flacoste, thanks. I'll land that first thing Tuesday :)
<flacoste> jml: that's today or next week?
<jml> whenever PQM opens.
 * rockstar heads to lunch
<james_w> thanks jml
<lifeless> matsubara: I'm here
<rockstar> Is there a way to tell what user I'm logged in as in the tests?  I'm pretty sure I have the right permissions to get to these attributes, but for some reason the test is saying no.
<lifeless> getCurrentInteraction or something
<lifeless> its the 'interaction' that defines 'who is logged in'
<lifeless> look in lp.testing._login
<rockstar> lifeless, great, thanks.
<mwhudson> there's get_current_principal somewhere in canonical.launchpad.webapp
<mwhudson> rockstar: there's also getUtility(ILaunchBag).user
<mwhudson> if the two disagree, confusion can certainly result
<rockstar> mwhudson, yeah, I had your suggestion, and it seems to not be working.
 * mwhudson afk
<lifeless> grah I remember what I wanted leonardr for
<lifeless> nope, forgotten again
<jml> james_w, any time
<lifeless> when is the next staging update due ?
<wgrant> lifeless: It looks like one might be in progress now.
<wgrant> Although it hasn't been trying to update regularly since it returned last night...
<wgrant> So the break in logging may not be indicative of an in-progress update, but just that someone hasn't switched the cron job on properly.
<lifeless> losa ping ^
<lifeless> nice, edge is looking way better than prod now
<lifeless> mean 0.37 stddev 0.62
<lifeless> mean 0.36 stddev 0.94
<lifeless> wow, Public XML-RPC is terrible
<wgrant> Is that over all requests?
<lifeless> 8.36 4.48
<lifeless> mwhudson ^
<wgrant> What's public XML-RPC, except for lp: lookups?
<lifeless> wgrant: I'm not sure
<lifeless> but 8 seconds of db time ain't brilliant
<lifeless> db avg stddev
<lifeless> 7.49 2.61
<wgrant> Ew.
<lifeless> for that group
<lifeless> translations are also up there
<lifeless> 1.42 1.89
<lifeless> compared to the overall app
<lifeless> problem pages
<lifeless> Archive:+copy-packages
<lifeless> Archive:+delete-packages
<lifeless> Archive:+index
<lifeless> Archive:+packages
<wgrant> I knew +delete-packages would be up there, but i didn't think it was that bad.
<wgrant> Also, is there a reason that the OOPS summaries are private?
<lifeless> BazaarApplication:+index
<wgrant> Don't they just contain page IDs, not URLs?
<lifeless> wgrant: they contain urls
<lifeless> but we can fix
<wgrant> :(
<lifeless> please file a bug
<wgrant> Against what?
<wgrant> oops-tools?
<lifeless> yeah
<lifeless> BranchMergeProposal:+index
<lifeless> BranchSet:CollectionResource
<lifeless> BugTask:+create-question is -terrible-
<lifeless> 11.03 8.97
<lifeless> only two hits, but *boy* where they doozies
<wgrant> I don't know how it's so bad, given that it's normally used on new bugs.
<lifeless> no, its not
<lifeless> can't be given the hit count
<wgrant> Hm.
<lifeless> BugTask:+distrotask
<lifeless> BugTask:+editstatus-page
<wgrant> I've hopefully fixed those.
<wgrant> Well, that and the heat fix.
<lifeless> BugTrackerSet:+index wow
<lifeless> 14.63 5.58
<wgrant> It's unbatched and going to be selecting lots of tasks, I guess.
<lifeless> Builder:+history
<lifeless> 8.59 2.42
<wgrant> It probably needs some prejoining for the new schema.
<lifeless> Distribution:+addquestion
<lifeless> thats probably search performance, which is fixed
<lifeless> Distribution:+archivemirrors
<lifeless> 7.76 4.18
<lifeless> 99% in 19 seconds
<lifeless> bet you its on the lpnet timeout summary
<lifeless> Distribution:+assignments
<lifeless> 15.98 7.77
<lifeless> Distribution:+branches 5.44 2.76
<lifeless> Distribution:+bugs 3.24 2.98
<lifeless> Distribution:+bugtarget-portlet-bugfilters-stats 6.79 1.64
<lifeless> Distribution:+bugtarget-portlet-tags-content 6.35 2.00
<lifeless> Distribution:+cdmirrors
<lifeless> Distribution:+filebug - again likely search and thus fixed.
<lifeless> Distribution:+filebug-show-similar ditto
<lifeless> Distribution:+patches
<lifeless> Distribution:+ppas
<lifeless> Distribution:+questions
<lifeless> Distribution:+search
<lifeless> DistributionSourcePackage:+addquestion - again, hopefully fixed
<lifeless> DistributionSourcePackage:+filebug-show-similar ditto
<lifeless> DistributionSourcePackage:+questions ditto
<lifeless> DistroArchSeries:+builds 3.50 4.04
<lifeless> DistroArchSeries:+index 2.39 3.10
<lifeless> mwhudson: hi
<lifeless> 10:22 < lifeless> mean 0.36 stddev 0.94
<lifeless> 10:23 < lifeless> wow, Public XML-RPC is terrible
<lifeless> 10:23 < wgrant> Is that over all requests?
<lifeless> 10:23 < lifeless> 8.36 4.48
<lifeless> mwhudson: ^ page performance stats
<lifeless> DistroSeries:+builds 6.47 4.50
<lifeless> DistroSeries:+cve 19.64 0.00
<mwhudson> lifeless: got a link?
<lifeless> https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-combined.html
<mwhudson> lifeless: i suspect that's actually private xml-rpc stuff
<mwhudson> getJobForMachine does a good job of contending with itslef
<lifeless> DistroSeries:+needs-packaging 12.93 7.63
<lifeless> DistroSeries:+templates 18.53 0.73
<lifeless> LibraryFileAlias:StreamOrRedirectLibraryFileAliasView 12.45 9.55
<mwhudson> yeah, regex for private xml-rpc looks wrong
<lifeless> MaloneApplication:+bugs 7.09 3.19
<lifeless> Person:+editproposedmembers 12.22 6.93
<lifeless> its a little worrying how fast I'm getting at eyeballing 99% on this thing
<lifeless> Person:+patches 17.61 0.00
<lifeless> Product:+filebug-show-similar search again
<lifeless> ProductSet:+project-cloud 6.22 4.31
<lifeless> ProjectGroup:+milestones 4.87 5.94
<mwhudson> project-cloud _really_ should be in memcache
<mwhudson> or something
<lifeless> Question:+confirm 4.87 6.75
<lifeless> Question:+edit 6.66 6.87
<lifeless> mwhudson: we need the _miss_ time to be subsecond
<lifeless> mwhudson: after that we can talk caching
<lifeless> if we need to memoise it to make it subsecond we should do that, but it should be persistent, no ?
<mwhudson> lifeless: maybe generated daily and stuffed somewhere that the website can access then?
<mwhudson> doing it 'in-line' in the appserver is fairly bonkers
<lifeless> mwhudson: agreed
<lifeless> Question:+makebug 4.61 7.73 - probably search
<_mup_> Bug #4: Importing finished po doesn't change progressbar <Launchpad Translations:Fix Released by carlos> <Ubuntu:Invalid> <https://launchpad.net/bugs/4>
<lifeless> RootObject:+recently-changed-branches 6.68 3.05
<lifeless> RootObject:+recently-registered-branches 5.08 2.30
<lifeless> TranslationGroup:+index 6.82 5.10
<lifeless> -fin-
<lifeless> we fix those to lower their 99% point, we can lower the timeout some more
<thumper> lifeless: the branch ones shouldn't be too hard
<thumper> lifeless: I've not looked at them
<lifeless> thumper: sure, I'm looking forward now that we've got edge OOPS really under control
#launchpad-dev 2010-08-11
<wgrant> Hm, so staging just slept for 3 hours.
<thumper> wgrant: everyone needs a break now and then :)
<lifeless> I'm chatting to elmo about it. Or maybe just depressing him, I dunno.
<lifeless> thumper: https://lpstats.canonical.com/graphs/OopsEdgeHourly/20100712/20100811/
<lifeless> https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100712/20100811/
<lifeless> wgrant: so, staging did a 5.5 hour update
<wgrant> lifeless: Looks like it did a 2.5 hour update, as usual.
<wgrant> It normally goes right back to checking for updates...
<wgrant> Or was this case special, in that it logged completion then did more stuff?
<lifeless> wgrant: that log doesn't log start + finish
<lifeless> http://www.newscientist.com/article/dn19287-p--np-its-bad-news-for-the-power-of-computing.html
<wgrant> lifeless: No, but look what it does normally.
<wgrant> lifeless: It disappears for 2.5 hours, then logs the update.
<wgrant> Then makes the next 14/44 check.
<lifeless> how does one subscribe on the dev wiki? I get an error about permissions
<lifeless> when I do 'subscribeUser'
<wgrant> lifeless: The subscribe action isn't shown.
<wgrant> You have to hack the URL.
<wgrant> ?action=subscribe
<lifeless> so, we expose a subcribe action that doesn't work, and don't show the one that does?
<lifeless>  /win
<wgrant> That's correct.
<lifeless> could you please file a bug on that ?
<wgrant> I think there might be on already.
<wgrant> Let's see.
<wgrant> ('Subscribe User' is for admins to subscribe others)
<wgrant> Bug #586601
<_mup_> Bug #586601: dev wiki toolbar has 'subscribe user' but not plain 'subscribe' <Launchpad Development Wiki Moin theme:Triaged> <https://launchpad.net/bugs/586601>
<thumper> lifeless: I just go to the user preferences and edit the subscribed pages list
 * thumper looks for mwh
<lifeless> hmm, prose is hard.
<lifeless> I can has code?
<lifeless> anyone want to read a draft? https://dev.launchpad.net/Performance
<lifeless> hmm
<lifeless> staging go boom ?
 * ajmitch reads up on performance
<wgrant> Interesting.
<wgrant> I wouldn't have expected the update to finish for a while yet.
<wgrant> Oh, wait, yes it should have.
<lifeless> wgrant: I'm not sure it has :P
<lifeless> it still says 9644
<wgrant> It's been updating to 9646 for nearly two hours.
<lifeless> where are you looking
<wgrant> successful-updates.txt. 9646 was in db-stable at the time it should have started the last update, at 00:14 BST.
<wgrant> It would normally have finished the update a few minutes ago.
<wgrant> But it seems to be running slightly late. Or has failed during the restart. One of them.
<wgrant> Ah, it's back.
<wgrant> r9646.
<lifeless> \o/
<lifeless> time to see if the distroseries page is fixifies
<wgrant> Oddly late.
<lifeless> \o/
<lifeless> didnae timeout
<wgrant> Also, why is it that staging takes longer to update than a full production rollout?
<lifeless> no idea
<lifeless> check the script
<lifeless> ... oh :P
<wgrant> Heh.
 * wgrant tries getBuildRecords.
<wgrant> Hm, and both +distrotask fixes should be active now./
 * wgrant tries that too.
<wgrant> Ah, crap.
<lifeless> nada?
<wgrant> No, it's nice and fast.
<lifeless> \o/
<wgrant> Just with a bug.
<lifeless>  /o\
<lifeless> top oops yesterdat
<lifeless> on edge
<lifeless> 32 https://api.edge.launchpad.net/1.0/ubuntu/maverick (DistroSeries:EntryResource:getBuildRecords)
<wgrant> That should be fixed now.
<lifeless> 63 cases
<lifeless>  https://edge.launchpad.net/ubuntu/maverick/+index (DistroSeries:+index
<lifeless> second at 11
<wgrant> Also, I hate doctests, particularly when they're not comprehensive.
<lifeless> s/sive/sable/ ?
<wgrant> No.
<wgrant> The only tests for this method are doctests. And they don't cover the case that I missed.
<lifeless> whats the bug
<wgrant> Distribution.guessPackageNames takes a package name, and tries to resolve it to a (sourcepackagename, binarypackagename) pair. It first tries to look up a source with the given name, then tries to find a binary within that source with the given name. If it can't find such a binary, it returns (sourcepackagename, None). If it can't find any published source with that name, it looks for a binary under any source. If it finds such a binary, ...
<wgrant> ... it then takes the binary's source name.
<wgrant> Now, for that last case, order is critical, since binaries can move between sources.
<wgrant> So now 'libgcc1' maps to the 'gcc-3.3' source, rather than 'gcc-4.5' as it should.
<wgrant> I dropped the (imperfect, but better than nothing) order from the old query.
<wgrant> Oops.
<lifeless> ah
<lifeless> and perhaps the order is what made it work ?:)
<lifeless> thumper: you have staging access right ?
<wgrant> Yes :(
<wgrant> But there are no tests.
<wgrant> Nasty thing.
<lifeless> thumper: You might need to do explain analyses for wgrant, I've got to pop out @3:39
<lifeless> bah 3:39
<lifeless> bah 3:30
<thumper> lifeless: I have school parent-teacher interviews this afternoon
<lifeless> ah, well poolie_ has turned up so he can do that too :)
<poolie_> hi lifeless
<poolie_> teacher-parent interviews? ok
<thumper> mwhudson_: got a few minutes?
<lifeless> poolie_: no, explain analyze's for wgrant on staging
<mwhudson_> thumper: yeah
<thumper> mwhudson: can you skype?
<mwhudson> thumper: perhaps :-)
<thumper> mwhudson: I have a problem with the sftp acceptance tests
<wgrant> I can has statement log from OOPS-1684S39?
<thumper> and trying to work out where to start
<poolie_> oops
<poolie_> so lifeless, to update you on feature flags
<poolie_> i think they passed all tests last night, aside from one to do with twisted test runners that looks timing dependent
<mwhudson> thumper: it appears i can skype
<poolie_> i can't seem to get into my home server from spiv's house
<poolie_> which is a bit annoying
<lifeless> wgrant: 27 statements
<poolie_> can we put wgrant into a team that lets him see oopses?
<lifeless> no
<poolie_> wgrant, if you want anything run, just ak
<poolie_> *ask
<wgrant> lifeless: I need to see one of the statements.
<lifeless> the problem is urls that reference private teams
<wgrant> Well, a couple of them.
<lifeless> and in future other things
<lifeless> wgrant: any in particular, or do you want the full 27
<wgrant> lifeless: Ideally the full 27.
<lifeless> enjoy
<lifeless> poolie_: until we have automated stripping/identifying of those, we can't share it
<thumper> mwhudson: found it
<thumper> mwhudson: I was checking for path.startswith('.bzr/')
<thumper> mwhudson: and the sftp server steps through '.bzr'
<thumper> so no trailing slash
<poolie_> lifeless,  can you try to re-send the feature flags branch, if you're happy with the later changes?
<mwhudson> thumper: :-)
<poolie_> if you didn't do that last night
<lifeless> poolie_: 'release-critical' mode - launchpad has a 1 week stall
<lifeless> poolie_: no, we can't till friday
<lifeless> poolie_: I am happy with your changs though
<lifeless> poolie_: but as we want this for development going forward, not for something on lpnet immediately, I think its non-rc
<lifeless> poolie_: once its landed we can use it on edge immediately, which will be sufficient till edge->lpnet, and at that point it will be universally available
<lifeless> spiv: https://bugs.edge.launchpad.net/launchpad-code/+bug/84838
<_mup_> Bug #84838: code browser should use oops system <Launchpad Bazaar Integration:In Progress by spiv> <https://launchpad.net/bugs/84838>
<lifeless> spiv: is the bug status stale ?
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/bug-616154-guessPackageNames-order-fix/+merge/32285 is RC. db-devel closes soon. What do I do?
<thumper> :(
<lifeless> wgrant: please add it to https://dev.launchpad.net/CurrentRolloutBlockers
<wgrant> Ah.
<thumper> mwhudson: it seems that faults raised in the xmlrpc server *DO NOT* cause a transaction.abort
<lifeless> I'm sending it to ec2 land now
<mwhudson> thumper: raised or returned?
<mwhudson> thumper: but either way :(
<mwhudson> thumper: i can probably tell you where to fix this, btw
<thumper> mwhudson: I'm tempted to use the @return_fault adapter and have a try except block
<thumper> mwhudson: fix the cause perhaps?
 * thumper shakes head
<thumper> not what I ment
<thumper> fix at the root?
<thumper> ENOTENOUGHSLEEP
<lifeless> thumper: can you 'approve' that merge above, ec2land is whinging
<mwhudson> thumper: PublicXMLRPCPublication.endPublication in servers.py?
<lifeless> thumper: we need release-critical *and* a normal approve, or it blows up
<thumper> lifeless: --force
<lifeless> no
<lifeless> different bug
<lifeless> humour me
<thumper> ah, which?
<lifeless> ec2 land https://code.edge.launchpad.net/~wgrant/launchpad/bug-616154-guessPackageNames-order-fix/+merge/32285 --force
<lifeless> ec2: ERROR: Cannot land branches that haven't got approved code reviews. Get an 'Approved' vote so we can fill in the [r=REVIEWER] section.
<lifeless> it has an approval from me
<lifeless> but its labelled 'release-critical' (which it has to be to get past pqm)
<wgrant> I suppose I need to mark the original bug qa-bad, too.
<wgrant> Is there anything special involved in that?
<thumper> doen
<lifeless> thumper: thanks
<thumper> mwhudson: how do we know whether to commit or abort?
<mwhudson> thumper: i'm now confused
<mwhudson> thumper: an excellent question
<lifeless> thumper: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/616164
<_mup_> Bug #616164: ec2 land refuses to land approved rc patch <Launchpad Foundations:New> <https://launchpad.net/bugs/616164>
<thumper> mwhudson: publication afterCall
<lifeless> transaction.doom() ?
<mwhudson> thumper: but it only seems to abort for a read-only request or for a doomed transaction
<mwhudson> ah right
<thumper> doom?
<lifeless> when something goes wrong we doom the transaction
<mwhudson> it doesn't seem that simply raising an error causes an abort
<lifeless> if we try to commit a doom transaction storm blows up as a safety measure
<thumper> mwhudson: no...
<mwhudson> unless there's a doom() in there somewhere
 * mwhudson greps
<lifeless> whats up?
<thumper> lifeless: I have a method in the xmlrpc server that tries to create a branch and link it
<thumper> lifeless: the branch is created, the link fails due to permission checks
<thumper> I expected the transaction to be aborted
<thumper> but it isn't
<lifeless> fun!
<lifeless> what sort of permission check
<thumper> launchpad.edit on the product
<thumper> I can manually abort it
<thumper> but it seems weird that I'd have to
<lifeless> do you catch it ?
<thumper> I would have thought that if the xmlrpc server is returning a fault, that it would be aborted for me
<lifeless> or rather how far up is it propogating
<thumper> lifeless: I catch it to return a fault
<lifeless> then you have to abort
<lifeless> I'm pretty sure we don't glue returned objects to db aborts
<thumper> I can.
<thumper> ok
<lifeless> not saying we shouldn't, just that thats my understanding
<thumper> faults are special though aren't they?
<mwhudson> really
<mwhudson> we should fix the problem that makes raising a fault cause an oops in the logs
<lifeless> thumper: a raised object would trigger an abort I think, not a returned one.
<mwhudson> maybe it should just log an informational oops
<lifeless> mwhudson: why shouldn't it have the same rules as web server oops
<lifeless> mwhudson: that is, we apply logic to it ;)
<mwhudson> lifeless: not sure quite what you're asking, suspect only having square tuits though
<lifeless> I mean we don't file OOPS for permission issues on the main appserver
<mwhudson> right
<lifeless> if you try something naughty, tsk, your problem.
<thumper> clucking bells
 * thumper afk for school stuff
<spiv> lifeless: no, in that I still haven't spoken to the losas about the production config for it
<lifeless> poolie_: lynne has bought red dead redemption for ps3 :) (and we've bought a ps3)
<lifeless> spiv: can you please shoot that mail off to losas @ c.c then ? that should be trivial to do...
<spiv> lifeless: ok, although I feel a bit like I'm drowning in distractions from the bzr bug I'm working on :/
<lifeless> So, the reason I'm nagging is that its not clear in the bug what needs to happen next; noone else can move it forward without chatting with you.
<lifeless> If you make it clear there, someone (me?) will move it forward sometime, otherwise its going to stay in your pile indefinintely.
<lifeless> And I must go, builder to give instructions to.
<lifeless> BBIAW
<poolie_> lifeless, i think that means i can't shoot lynne :-(
<spiv> lifeless: that's fine, just grumbling to the world in general rather than you
<poolie_> being on a ps3
<spiv> lifeless: thanks for the nag
<poolie_> lifeless, nice performance doc, though i have to confess the "3*SD + mean" really squicks me for some pedantic reason
<poolie_> i wish it just computed the 99% value
<poolie_> lifeless>> caches (unlike memos) are populated by the first request for the data
<poolie_> isn't that exactly what a memo does?
<lifeless> spiv: thanks
<poolie_> <poolie_> lifeless>> caches (unlike memos) are populated by the first request for the data
<poolie_> <poolie_> isn't that exactly what a memo does?
<poolie_> also
<poolie_> <poolie_> lifeless, nice performance doc, though i have to confess the "3*SD + mean" really squicks me for some pedantic reason
<poolie_> <poolie_> i wish it just computed the 99% value
<lifeless> poolie_: so do I
<lifeless> poolie_: I plan to hack on it some shortly
<poolie_> that would be nice
<poolie_> in particular if we have just a few outlier results (because they do hard io, or they miss a normally-reliable cache) there might be a big difference
<poolie_> briellant
<lifeless> poolie_: you might like the email I just sent off
<lifeless> wgrant: your branch has landed
<lifeless> wgrant: sorry, it has ec2 passed, pqm time now
<lifeless> poolie_: I'm not sure that 99% != mean+3SD here. I know it *might not*, but actually its pretty accurate so far when I have compared
<lifeless> poolie_: as for memoisation vs caching; I'm trying to distinguish things that are redundant that we store in advance vs things that are redundant that we calculate just-in-time-and-remember
<lifeless> poolie_: better terms appreciated
<wgrant> lifeless: Thanks.
<jtv> lifeless: just made some superficial changes to your new Performance wiki pageâ¦  to be clear: are the cases of "its" where I'd expect "it's" mistakes (as I believe they probably are), or is there a rationale behind it?
<jtv> lifeless: also, it may be worth mentioning the distinction between responsiveness and completion speed earlier on.
<stub> The report is two wide already, so +1 replacing columns with something more generally useful.
<jtv> lifeless: then, where it says the bugs database is write-only, do you mean append-only?
<jtv> stub: wrong window?
<lifeless> jtv: thanks
<lifeless> jtv: I'm terrible on my its'
<lifeless> jtv: what bit of the distinction would you put earlier? just the idea of responding quickly, doing the work, then completing ?
<jtv> lifeless: no, the point about getting a response not meaning that you're all doneâe.g. "loading the page a bit faster at the cost of having parts follow a bit more slowly is a win in responsiveness whenever the user doesn't feel held up by the slower bits."
<jtv> I believe you mention this under memoisation, where the relevance isn't immediately clear to me tbh
<jtv> (look for "red flag")
<jtv> BTW don't worry about the "it's"; I went through all the itses on the page.
<lifeless> wgrant: its through pqm now
<jtv> it's!
<jtv> it's through pqm!
<lifeless> see, terrible
<jtv> :-)
<lifeless> jtv: ah yes
<lifeless> doing background processing in an interactive context
<lifeless> the caveat being that sometimes we have to do it to be 'complete'
<jtv> lifeless: saw a good point about responsiveness back in the 20th century: the important thing is that you know that things are going on and can sympathizeâat the time a flurry of disk seeks was very audible but an http connection to (as I believe the poster put it) Outer Mongolia was not.
<lifeless> lol! I love it!
<lifeless> did they have a wav file of chattering disks ?
<jtv> So waiting 10s for disk seeks was fine, but waiting 7s for a page to load was not.
<jtv> lifeless: this was in the days when you didn't just add some huge binary file just for the hell of it.
<jtv> It was also in the days when anyone likely to read the message could easily produce the noise for themselves!
<henninge> jtv: https://code.launchpad.net/~henninge/launchpad/bug-595925-second-attempt
<henninge> jtv: The last revision makes the Windmill test fail in a strange way and I don't have a clue.
<jtv> henninge: diff generation is broken todayâ¦ do you have a diff somewhere?
<henninge> jtv: shure. Hang on...
 * henninge turns on speakers for a start
<henninge> jtv: http://paste.ubuntu.com/476320/
 * jtv looks
<henninge> jtv: I included the first part from the factory so you can see what makeSuggestion does.
<henninge> jtv: The windmill test produces an OOPS with this change.
<henninge> lemme do that again
<adeuring> good morning
<jtv> hi adeuring
<henninge> adeuring: Hallo!
<jtv> henninge: what kind of oops does it give you?
<henninge> jtv: here is the test command and the oops that appears in the windmill browser window:
<henninge> http://paste.ubuntu.com/476322/
<jtv> henninge: nasty traceback.
<henninge> what is a "LocationError" about, anyway?
<jtv> damned if I know
<jtv> But I would guess that the problem is that we're creating a TM without a pofile.
<henninge> We do?
<jtv> In this case, yes.
<jtv> But we never completed the new +translate page, so we have some hacked-in workarounds that rely on the field being set.
<jtv> Again theoretically, we set those completely in-memory without bothering the database, but as I found in my experiments a few months back, nulling the database field will produce oopses.
<lifeless> stub: remind me where the ppr code is again ?
<henninge> jtv: so I'd have to check the view code to see where it uses tm.pofile?
<stub> utilities/page-performance-report.py (half are reports are in utilities and half in scripts)
<jtv> henninge: it might be anywhere
<henninge> ouch
<lifeless> stub: thanks
<jtv> henninge: otp
<jtv> henninge: and so should you be :)
<lifeless> stub: now, I'm thinking of doing a 'timeout-candidates summary
<lifeless> stub: showing just page ids and 99%
<lifeless> stub: is that controlled entirely in the source, or is it going to be partly how its deployed?
<lifeless> stub: and, do you think such a summary can be published publically ?
<stub> consider dropping the varience and stddev columns
<stub> I can't see what we consider private on the existing report.
<lifeless> stub: top-urls
<stub> Oh, right.
<lifeless> the urls might be for a private team like ~vendor-supplier or something
<lifeless> stub: 'just page ids and 99%' includes dropping the mean and stddev :)
<stub> page-performance-report-daily.sh is the script I run each day to generate the reports in the relevant locations
<lifeless> ok
<stub> lifeless: I mean that the variance and stddev might be irrelevant if we output the 99% on the existing reports
<lifeless> stub: ack
<lifeless> stub: uhm, I think they are useful to have somewhere
<stub> Or maybe JS love to hide unwanted columns might be better
<lifeless> otoh the graph is a pretty good visual aid for same
<lifeless> hmm thats interesting
<lifeless> what happened on the 29th https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100712/20100811/
<lifeless> something causing both exceptions and timeouts got changed
<lifeless> mrevell: I'm happy with your changes to the oops wiki page btw
<mrevell> Hello
<lifeless> mrevell: thank you for doing them!
<stub> lifeless: maybe me rebuilding the bug full text indexes?
<lifeless> stub: could be it
<lifeless> nothing leaps out at me from the production-stable branch changelog
<mrevell> Thanks for writing it lifeless!
<bigjools> morning all
<lifeless> morning bigjools
 * bigjools loves seeing last minute RCs
<wgrant> bigjools: Sorry :(
<bigjools> c'est la vie
<lifeless> !oops
<lifeless> stub: am I correct - no tests for the script ?
<stub> lifeless: Yes.
<lifeless> stub: how can I be sure I haven't stuffed it up ?
<lifeless> stub: or will you do that if I get you something approximate ?
<stub> I've run it against a log and looked at the output.
<stub> We could write a test for it, but it would be incredibly fragile.
<stub> So put if off until it stabalized
<lifeless> lp:~lifeless/launchpad/foundations - I added a column (sorry!) but it should give us one report most devs can focus on.
<lifeless> which will be pretty short
<stub> So that will need a merge proposal
<lifeless> firing one up
<lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/foundations/+merge/32299
<stub> numpy doesn't have a magic 99th percentile helper?
<stub> doesn't look like it
<lifeless> http://docs.scipy.org/doc/numpy/reference/routines.statistics.html is less than completely helpful
<elmo> https://pastebin.canonical.com/35707/ <-- it may be trivial/obvious, but there's one I had to hand, FWIW
<lifeless> elmo: hah! thanks.
<lifeless> I'm using stddev*3 + mean atm
<lifeless> which may be more useful
<lifeless> as grabbing the Nth can be much higher/lower than the curve fitted approach
<stub> Beyond my recollections of high school anyway.
<stub> Need food.
<bigjools> mrevell: when logging in to the dev wiki, it doesn't redirect back to the page you were on, do you know if that's fixable?
<mrevell> bigjools, Yeah, that's a pain. I don't know if it's fixable but I can certainly ask.
<bigjools> ok cheers
<wgrant> bigjools: I would have loved to rewrite the doctest, but I opted for a less invasive change as a last-minute RC!
<bigjools> wgrant: I know, it's the right thing in this case
<bigjools> doesn't stop me being sad about it though :)
<wgrant> Heh.
<lifeless> gmb: hi
<lifeless> gmb: I think it was a bug of yours I touched today - about +filebug timing out doing large blobs.
<lifeless> gmb: I wanted to check that that is out of the webapp request now ?
<gmb> lifeless, Err, let me check...
<gmb> lifeless, In that case it's just that the dependencies are done. At the moment the polling (such as it is) is by way of a page refresh.
<gmb> I'll note that on the bug.
<lifeless> gmb: there were two
<lifeless> gmb: thanks for adding a note to the one about the polling
<lifeless> the other bug I closed off which was tagged timeout because of those blos
<lifeless> blobs
<lifeless> I figure that that is fixed, no ?
<gmb> lifeless, Can you give me a number for the other bug?
 * gmb might have marked it as read by mistake
<lifeless> hmm
<lifeless> gmb: https://bugs.launchpad.net/malone/+bug/357907
<_mup_> Bug #357907: +filebug is timing out when processing large blobs <timeout> <Launchpad Bugs:Fix Released> <https://launchpad.net/bugs/357907>
<gmb> lifeless, Yes, that's fixed.
<lifeless> mrevell: hey, so the changes to the +filebug and answers search  - I documented them reasonably well, but there isn't a blog post ready-to-roll with the release.
<lifeless> mrevell: how do you feel about writing that up, so that in the release folk aren't taken aback ?
<lifeless> mrevell: failing that, I'll be up for the team leads meeting tomorrow am and can make writing a blog post a priority then, but I may need a hand connecting to the lp blog etc etc, if you could email me the necessary that would be grand.
<mrevell> lifeless, Heh, I had it on my list for today to email you about those. I can write the posts. To summarise, rather than getting 10 or so results in the dupe-finder you're more likely to get 3 or 4 and the results may be slightly different to what you'd have seen before. Is that right?
<lifeless> mrevell: user visible behaviour - yes, thats right.
<lifeless> slightly different *should* be 'more relevant'
<lifeless> as in, yes its different, but we hope its better too.
<lifeless> its /also/ much faster.
<lifeless> like 15 seconds faster for /ubuntu/+filebug
<mrevell> lifeless, Cool :) What would be really helpful to me, if you have time, would be a quick email with very rough notes on this. Or just paste them here are you are now :)
<lifeless> for many searches
<lifeless> so the arch is
<lifeless> arc
<lifeless> +filebug and answers were both performing terribly, with the search engine at the core
<lifeless> we're going to replace the search engine, but that takes time - and by horribly I mean timing out *a lot*. Nearly unusable.
<lifeless> So as a band aid we've made the searches with the current engine narrower.
<lifeless> This makes it faster, and - due to how we've changed it - slightly more relevant at the same time.
<lifeless> We *can* switch this off easily if we have to, so we *do* want feedback about how people find this.
<mrevell> Cool. This is very helpful, thanks. Can you give me some detail of how you've changed it? Also, what's the best way for people to provide feedback to you?
<lifeless> We *want* to stick with the band aid for 5-6 months while we replace the search engine.
<lifeless> The change specifics
<lifeless> the old search did a pre-pass over *every possible hit*, which is 400000 items for ubuntu - thats very slow to do, and then did a search matching *any document* which had a rare search term in it.
<lifeless> Where rare == 'turns up < 50% of the possible hits'
<lifeless> so if you searched for firefox crashes on <website> in flash
<lifeless> on /ubuntu/+filebug
<lifeless> it would search for any bug with *any of* 'firefox' (< 50% of bugs are firefox), 'crash' (<50% of bugs say crash), '<website>' (<50%...), 'flash' (< 50%..)
<lifeless> however, *many many* bugs mention firefox, *and* many many bugs mention crash and many many mention flash
<lifeless> so the total the search return could be 10000 or 100000 quite easily
<lifeless> and - unlike other search engines - the more terms you typed in, to make it more precise, the *less precise* it became.
<lifeless> because it started bring back bugs from anywhere that happened to mention any search term
<lifeless> and the relevance weighting - well, it added confusion to it.
<lifeless> What we do now is:
<lifeless> if you sesarch for firefox crashes on <website> in flash
<lifeless> we search for any bug continaing 3 of the 4 non stopwords
<lifeless> that is firefix,crashes,website,flash - if a bug mentions any 3, it will be returned.
<mrevell> Thanks Rob.
<mrevell> lifeless, As for the LP blog ... it's hooked into LP's OpenID stuff, so you need to log in at https://blog.launchpad.net/wp-admin (igoring the invalid cert) and it'll generate an account for you. I can then upgrade that to author status.
<lifeless> contacting us about this? launchpad-users list, tweet to launchpadstatus, faccebook page, irc
<lifeless> I'll take point on any heat
<lifeless> but I think we'll all be interested in feedback
<lifeless> mrevell: I've logged in with openid
<deryck> Morning, all.
<mrevell> lifeless, Okay, you should now be able to makes posts on the blog.
<mrevell> Howdy deryck.
<lifeless> thanks mrevell
<lifeless> I'll give that a spin on the next bit I redefine from ground up :)
<lifeless> gnight
<deryck> gmb, your heat timeout patch is on staging now?
<mrevell> :) Night Rob.
<lifeless> deryck: I qa'd the heat timeout patch
 * lifeless finds leaving the keyboard tricky
<deryck> heh, ok, thanks.
<deryck> Was just curious since the timeout was blocking other qa for me.
<lifeless> deryck: (I mentioned it in the patch :P)
<lifeless> s/patch/bug/
<bigjools> wgrant: still around?
<wgrant> bigjools: Sure.
<bigjools> wgrant: your fix got through buildbot, I'm going to see if I can get staging refreshed so you can QA it
<wgrant> bigjools: That would be excellent. Thanks.
<bigjools> wgrant: losas are at lunch I think, are you around much more today?
<wgrant> bigjools: I'll be around for another 2-3 hours at least.
<bigjools> great, thanks
<deryck> I *really* wish we didn't do subscriber notifications in app.
<bigjools> totally
<bigjools> deryck: we have the same problem, indirectly.  When accepting packages that close bugs it calls your notification code.
<deryck> yeah, it bites everyone.  I think that's what killed sinzui's one-click release work.
<wgrant> What's the time spent on? Creating BugNotificationRecipients?
<deryck> wgrant, yes, mostly there
<sabdfl> hi folks, do the project aliases enable redirects? if i rename a project from foo to bar, with foo as an alias, will stuff just work? only care about bug links
<maxb> I know they do for https://launchpad.net/<foo> links
<jelmer> sabdfl: hi. Yes, they enable redirects. https://bugs.launchpad.net/alias/+bug/42 will redirect to https://bugs.launchpad.net/name/+bug/42
<_mup_> Bug #42: Bug description listed in task is not the correct description <Launchpad Bugs:Fix Released by bradb> <https://launchpad.net/bugs/42>
<_mup_> Bug #42: Bug description listed in task is not the correct description <Launchpad Bugs:Fix Released by bradb> <https://launchpad.net/bugs/42>
<sabdfl> brilliant, thanks jelmer
<sabdfl> now that we've solved bug #42, everything else should be easy :-)
<_mup_> Bug #42: Bug description listed in task is not the correct description <Launchpad Bugs:Fix Released by bradb> <https://launchpad.net/bugs/42>
<wgrant> bigjools: Hm, it's not there yet.
<bigjools> :/
<wgrant> I need r9648
<wgrant> It's r9647 now.
<bigjools> wgrant: check again in a while, it'll update soon
<wgrant> bigjools: ... and take 2.5 hours to do so :/
<bigjools> wgrant: 20 minutes
<wgrant> Ah, that's a little better.
<stub> sabdfl: Yes. https://bugs.edge.launchpad.net/launchpad-translations/+bug/615673 for an example.
<_mup_> Bug #615673: IPOTemplate.path says it's not required <oops> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/615673>
<wgrant> bigjools: staging's been saying 'Code Update In Progress' for nearly half an hour now...
<jelmer> stub: hi. Can you allocate a database patch id for me?
<stub> jelmer: lp:~jelmer/launchpad/613468-xb-ppa-db ?
<jelmer> stub: yep
<stub> Reviewed. Suspect that won't be landing this cycle though.
<mars> wgrant, staging code updates take an average of 100 minutes
<mars> we have a graph of it
<wgrant> mars: It's always down for the whole time?
<mars> That I do not know
<bigjools> wgrant: ok I'm chasing it, thanks
<jelmer> stub: No, that wasn't the intention. Thanks.
<bigjools> wgrant: can you QA that bug on dogfood?  it's up to date
<bigjools> stub: did you get any ideas with that query that's timing out?
<wgrant> bigjools: DF still says r9644.
<bigjools> wgrant: ignore it
<bigjools> I didn't rebuild the revision file, it takes too long
<wgrant> Ah, heh.
<bigjools> (as it rebuilds the wadl too)_
<wgrant> The fix is good.
<bigjools> yay
<salgado> EdwinGrubbs, IIUC, you'd then run that query once for every superteam, is that right?
<EdwinGrubbs> salgado: oh, I didn't think about that. If the superteam is a member of another super-super-team, you have to update its teamparticipation also. bleh.
<EdwinGrubbs> salgado: it might be possible to get that to run in a single query if I just join teamparticipation table with itself. It would be a joining a list of all the new members with all the new superteams.
<EdwinGrubbs> of course, the performance impact of that could be surprisingly high.
<wgrant> bigjools: Anything else before I disappear?
<bigjools> wgrant: no, thanks for hanging around
<bigjools> sleep well
<wgrant> Thanks.
<wgrant> Night.
<stub> bigjools: Can you run on dogfood: explain analyze SELECT COUNT(*) FROM BuildFarmJob
<stub> JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id
<stub> JOIN Archive ON PackageBuild.archive = Archive.id
<stub> WHERE BuildFarmJob.builder = 106;
<bigjools> stub: yep, one sec
<bigjools> stub: http://pastebin.ubuntu.com/476455/
<stub> bigjools: So the weird way of doing the join was slowing things down
<bigjools> figures
<bigjools> noodles775: ^
<bigjools> thanks stub
<noodles775> stub, bigjools: erm, doesn't it need to be a left join? (not all BuildFarmJobs have a related PackageBuild)
<noodles775> s/have/will have
 * noodles775 looks for the code.
<stub> noodles775: Hmm
<noodles775> bigjools, stub: see comment in lp/buildmaster/model/buildfarmjob.py:BuildFarmJobSet.getBuildsForBuilder
<stub> So the actual query should just be 'select count(*) from buildfarmjob where builder=106;'
<jtv> bigjools: then why did you ask in -code?
<bigjools> jtv: because I suck
<jtv> stub: hehâ¦ we both converted the join but you converted to inner-join whereas I left out Archive.  Together we should rule.
<noodles775> stub: if we updated the count to include private builds (and just display them as 'Private build' on the history, yes. But currently (we don't know the history) it only displays the builds you are allowed to see... hence the joins.
<stub> I never use implicit outer join so misparse 'left join'
<jtv> stub: doesn't your version change the zero case though?
<stub> noodles775: But that isn't the query we are looking at
<jtv> And it often pays to keep "private, visible" queries completely separate from "public" queries.
<noodles775> stub: I think it is..., if you check the code, you'll see why what you're expecting is missing (the oops was generated by a df admin, so it doesn't bother adding the extra where clauses)
<stub> I can't really optimize a partial query
<noodles775> I'll generate one as a non-admin.
<danilos> stub, you are lame, I can optimize even bits like "FROM Prod" ;)
<jtv> danilos: go play with mysqlâthe /real/ men are talking here
 * danilos blinks and powers up mysql
<jtv> danilos: you did realize I was joking, right?  I wouldn't do that to you.
<danilos> :)
<jtv> (So once it's started up, please just shut it down again)
<danilos> (aptitude is still installing it, how do I break that?)
<bigjools> don't use aptitude? :)
<jtv> Pull all plugs and cables out, then wait for battery to drain if applicable.
<noodles775> stub: I've added https://pastebin.canonical.com/35735/ to the bug - an oops for the same page when logged in as a non-admin.
<stub> explain analyze SELECT COUNT(*) FROM BuildFarmJob
<stub> LEFT OUTER JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id
<stub> LEFT OUTER JOIN Archive ON PackageBuild.archive = Archive.id
<stub> WHERE BuildFarmJob.builder = 106;
<stub> That has a much nicer explain on production
<jtv> stub: I have a feeling I was responsible for the COALESCEâ¦
<stub> noodles775: Do you know what the constants would be in the COALESCE(Archive.private, ???) = ???
<noodles775> stub: nope, I just followed jtv's review.
<stub> They wouldn't have been in jtv's review...
<jtv> stub: it was a boolean
<stub> True or False?
<stub> I'm guessing False
<jtv> bigjools would know
<bigjools> can point me at the code for context?
<jtv> https://pastebin.canonical.com/35735/ I believe
 * bigjools tries to work out wtf that is in the code
<jtv> bigjools: this is the bit you asked me about isn't it, where I suggested COALESCE?
<stub> Yes, false
<bigjools> jtv: different query I think
<jtv> ah
<stub> lib/lp/buildmaster/model/buildfarmjob.py
<jtv> stub: so in that case, Archive.private IS NOT FALSE would do as well?
<stub> So we are just trying to return the number of buildfarm jobs that are visible?
<bigjools> yes
<bigjools> we should be returning all of them and displaying private ones differently really, but that's another matter
<jtv> stub: you're right, the %s must be false
<jtv> In which case, leaving out that join can still make sense here
<stub> explain analyze
<stub> SELECT COUNT(DISTINCT BuildFarmJob.id)
<stub> FROM BuildFarmJob
<stub> LEFT OUTER JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id
<stub> LEFT OUTER JOIN Archive ON PackageBuild.archive = Archive.id
<stub> LEFT OUTER JOIN TeamParticipation ON TeamParticipation.team = Archive.owner
<stub> WHERE
<stub>     BuildFarmJob.builder = 106
<stub>     AND (Archive.private IS FALSE OR TeamParticipation.person = 1)
<stub> (I'm not sure of who a suitable owner would be, so using '1')
<jtv> (note to self: I was talking nonsense earlier)
<bigjools> noodles775: I am guessing that we won't be able to make a branch for this in the next few minutes.
<noodles775> Thanks stub: I'll try the storm equivalent.
<noodles775> bigjools: no.
<bigjools> noodles775: ok, we'll have to release with this broken, and do a re-roll or CP
<bigjools> how it got broken is a good question though!
<jtv> bigjools: I don't suppose you could dig up an older oops with the pre-broken query?
<stub> It might just be the data changing things. My query could suck for instance if we end up with archives owned by a team with a large membership.
<noodles775> As per the bug, the data that we know changed on dogfood/staging is that we now have other types of BuildFarmJob's... but there could be other data changes too of course.
<bigjools> ah of course, new job types trying to get displayed
<jtv> I don't see job types playing here, at least not directly
<jtv> bigjools, stub, noodles775: actually, looks to me like the "flattening" of that Archive join at least is very easy to do.  It's explicit in the code.
<jtv> left_join_pkg_buildsâ¦ defined then used immediately, and never again.  Contains the whole drama.
<stub> Might be worth someone running that explain on dogfood though... the plan on production looks ok, but that isn't terribly useful since the problem is on dogfood.
<noodles775> jtv: easy to do in storm? Great... can you paste an example? And what do you mean never again? (isn't it required for the later conditions on archive.private?
<noodles775> stub: doing so now.
<jtv> noodles775: just saying there are no boundaries of encapsulation, responsibility etc. to complicate changing this.  Hang on.
<noodles775> stub: https://pastebin.canonical.com/35741/
<stub> ok - that seems good for dogfood.
<jtv> hmm this rain is _cooling_ my laptop, but is it good for the laptop _overall_?
<stub> There is an alternative version where we remove the DISTINCT and do the teamparticipation check in a subquery
<stub> you have rain? Its hot outside here.
<jtv> stub: I'm North of you, but not far
<jtv> Not a lot of rain, mind you; just enough to make the screen a bit harder to read
<stub> Yup. Dry here. I think you have a leak.
<jtv> (I'm in front of the house, under an awning and further shielded by those big umbrellas we used to have out on the balcony in Din Daeng)
<jtv> noodles775: this is definitely not my first litre of beer, so I may not be the right person to rewrite your storm query right now.  But if you look at BuildFarmJobSet.getBuildsForBuilder, you see essentially the whole problem query spelled out, inner-join-nested-in-outer-join and all.  If you can rephrase that, you've got stub's change.
<noodles775> jtv: yes, I'm in the process of converting it now based on stub's query above.
<jtv> delightful
<jtv> noodles775: what sort of row counts would you expect out of this?
<noodles775> jtv: if it helps, the review discussion we had is here: http://irclogs.ubuntu.com/2010/06/16/%23launchpad-reviews.html#t15:26
<jtv> So that *was* the Coalesce I suggested, just not to bigjools!
<noodles775> Yep, that's what I meant earlier when I referred to the review.
<jtv> rightâgot it now.
<jtv> My God, did I really spell it as FULLYBUILD?
<leonardr> james_w, question about your launchpadlib test failures
<james_w> hi leonardr
<leonardr> are you also getting test failures in toplevel.txt where httplib2 prints out debug messages when it's not expected
<leonardr> ?
<leonardr> wgrant, would you try running the tests with lp:~leonardr/launchpadlib/616055 and see if the test failures all go away?
<james_w> leonardr: yes, that was one of the bug reports I filed I thought
<james_w> leonardr: it appears to now GET earlier than the tests are expecting
<leonardr> james_w: ok, try the new branch and everything should work
<james_w> leonardr: your cover letter should probably explain how to get launchpad to use that branch when running the tests. Just running the command there will run the old version.
<james_w> leonardr: anyway, they are running, I'll let you know after lunch
<leonardr> james_w: great
<james_w> leonardr: you missed one it seems: http://paste.ubuntu.com/476544/
<leonardr> james_w: ah, that's an error in launchpad, not launchpadlib. that's why i didn't see it
<leonardr> was that the only one?
<james_w> leonardr: with -r launchpadlib, yes
<leonardr> james_w: oh, actually the launchpad branch i'm working on already fixed that error, _that's_ why i didn't see it
<lifeless> https://devpad.canonical.com/~stub/ppr/edge/latest-daily-timeout-candidates.html
 * rockstar goes for a walk
<bryceh> $ make lint-verbose
<bryceh> ...
<bryceh> ./bin/lint.sh: line 163: pocketlint: command not found
<bryceh> aha nevermind
<jml> lifeless, please review my testr branches
<lifeless> flacoste: elmo: if you want to move the foundations call up, fine with me - as elmo is sprinting it might be easier/worse
<lifeless> jml: thanks!
<flacoste> elmo: i'm free, so it's your call
 * jml off
<lifeless> night jml
<jml> g'night.
<jml> I might be back later to land some patches for some testing tools. :)
<lifeless> Ursinha: I'm sorrry, I haven't written the rollback patch yet
<lifeless> Ursinha: if its not in your inbox your tomorrow am, please assume I'm swamped and won't get to it
<Ursinha> lifeless, I can do that, if you don't have plans to do that soon
<Ursinha> ah, sure
<Ursinha> :)
<lifeless> Ursinha: well, if you wanted to do it today that would be awesome
<Ursinha> lifeless, I can do that
<lifeless> \o/ thanks
<lifeless> mthaddon: I kind of wish that https://lpstats.canonical.com/graphs/OopsEdgeHourly/20100712/20100811/ included the request count, to make me feel better :)
<lifeless> but I know it would make the graph much less useful
<flacoste> lifeless: given that elmo is in Madrid on a sprint, i'm not sure he'll make the conf call
<lifeless> ok
<lifeless> well, we can ring his mobile ;)
<flacoste> lifeless: hmm, do you have an agenda?
<lifeless> for the first meeting ? ;)
<lifeless> rfwtad; lucene;
<lifeless> the first implies an 'are we blocked on all lucid stuff, or just per machine', 'are we there yet' style annoying question, which I know he'd just -love-
<lifeless> :P
<flacoste> lifeless: unless you insist, i'd skip this one :-)
<lifeless> its fine
<lifeless> skipped, I'll be back in a bit
<lifeless> ok
<lifeless> who really understands the prejoin code
<elmo> flacoste/lifeless: sorry I didn't intend to make the call but failed at timezone math and was afk at the scheduled time
<flacoste> lifeless: jamesh does
<lifeless> elmo: do you mean 'did intend' ?
<lifeless> flacoste: I shall stalk him!
<elmo> lifeless: yes
<lifeless> elmo: no worries
<elmo> apparently I also fail at English
<lifeless> all I had for the meeting I put in the channel :)
<elmo> AFAIK, lucid stuff is at the 'can do machines any time we want', but the LOSAs want to wait till they aren't sprinting for me to do the upgrades
<elmo> the lucene one was a little dense for me
<lifeless> elmo: I want to move forward on evaluating it
<elmo> lifeless: ok - what do you need from me/us?
<lifeless> need to coordinate temporary use of a machine or two with the grunt to reindex all of LP's content, to see how it performance and evaluate search results
<elmo> *choke*
<lifeless> may need a slave replica along the lines of staging to read the content to reindex from
<elmo> is that all? :-P
<lifeless> elmo: or we need to figure out a way to do a sensible evaluation
<elmo> what's the grunt?  disk, memory or both?
<elmo> if you know
<lifeless> elmo: I'm raising this now so that we can figure out how to do it without panicing ;)
<lifeless> Well, we don't /really/ know yet.
<lifeless> Disk wise we need a copy of LP's DB - thats 250GB IIRC? And we'll need ~ that again for the index (to allow learning space)
<elmo> ok
<elmo> I'll let my brain background on WTF to do that
<lifeless> memory I have no idea
<lifeless> it might be so damn great that 1GB will do
<elmo> and pigs might fly given sufficient thrust
<lifeless> porcine aviators unite!
<lifeless> ... pigs in space
<lifeless> anyhow :)
<lifeless> I suspect we'll need several GB to get the content out of pg efficiently, and lucene will want a few hundred MB, maybe a GB, to index as we go
<lifeless> when we move onto test querying, we'll find out how good the index locality is.
<lifeless> there is no schedule for this yet - as I say, I'm raising it to give you plenty of warning
<lifeless> the broad plan I have is: evaluate it, and if its as good as all the feedback I've been getting, get a performance profile for it; then we will have some idea about how big a machine we need, and can start to talk deployment/migration/staging etc etc etc
<lifeless> the goal is to take a lot of load off pgsql doing this
<lifeless> so we might even free up a slave box or something : we don't know yet though.
<lifeless> s/the goal/a goal/
<elmo> sure, ok
<lifeless> I'd be happy to evaluate on a really small memory/cpu box, and scale up if its not enough - but that may imply a bunch of fiddling and so on that isn't particularly helpful.
<lifeless> I will be totally guided by you
<wgrant> leonardr: Did you really mean me?
<lifeless> wgrant: was the ppr useful ?
<wgrant> lifeless: I was just reading it when #launchpad-meeting took my attention, so I forgot.
 * wgrant looks.
<lifeless> the stats are a little bong - I've sent stub a mp to fix em
<wgrant> That is a wiide table.
<lifeless> yeha
<lifeless> sort by the 99% column
<wgrant> There is no sorting. I guess I'll grab the JS manually.
<lifeless> hmm
<lifeless> it should be coming from people.canonical.com
<wgrant> Ah, yes, it is, but Evo doesn't follow external links.
<wgrant> Works fine in a web browser.
<wgrant> That is an interesting page, certainly.
<jml> good night all
<mwhudson> night jml
<leonardr> wgrant: sorry, i meant james_w
<wgrant> Ah :)
<poolie> lifeless: back to feature flags for a bit today
<lifeless> cool
#launchpad-dev 2010-08-12
<lifeless> is there a wiki page listing the performance problem-and-resolutions folk have run into before ?
<jelmer> lifeless, there should be, but I don't think there is.
<lifeless> jelmer: https://dev.launchpad.net/Performance has one now
<jelmer> lifeless: Yeah, I saw it. Thanks, also for that summary of improvements you had made and how/why earlier.
<lifeless> which summary?
<jelmer> the one you sent at the end of your day yesterday
<jelmer> well, it wasn't really a summary I guess
<jelmer> English fail.
<lifeless> :)
<lifeless> the the 'some thoughts on performance' ?
<jelmer> lifeless: I meant "Performance tuesday - a few notes"
<jelmer> which I pretty much read as the summary of your day
<jelmer> and now I'm getting sleep before I do more gibbering :-)
<lifeless> :)
<lifeless> I'm glad its useful
<lifeless> that was two days ago :>
<jelmer> lifeless: Heh, timezone fail. Tuesday was yesterday. >-)
<mwhudson> lies
<lifeless> jelmer: 36 hours then :P
<lifeless> wgrant: so
<lifeless> wgrant: I'm thinking of writing an extension to cachedproperty
<lifeless> to register the caches in __cachedproperties__
<lifeless> and a SQLBase base class storm invalidation hook to clear that out.
<lifeless> wgrant: ignoring performance, does that raise any red flags for you?
<lifeless> mwhudson: ^ also
<thumper> lifeless: a bit of advice if you can...
<thumper> I want my tests to all flush the store to make sure database constraints are triggered
<thumper> should I just add a cleanup from the test setup?
<thumper> is that a reasonable thing to do?
<thumper> mwhudson_: comments on the above?
<wgrant> lifeless: On what would it be invalidated?
<thumper> is an error in cleanup execution still considered a failure?
<wgrant> Isn't the entire object already invalidated on the transaction boundary?
<thumper> wgrant: removed from the storm cache, but not invalidated
<thumper> AFAIK
<wgrant> thumper: So if I hold a reference to an object, it will stay around?
<wgrant> That seems bad.
<thumper> used in many places though
<thumper> like scripts
<wgrant> Hmm, true.
<lifeless> thumper: an error in cleanups is an error in the test, yes.
<lifeless> wgrant: it will be invalidated when storm invalidates the object
<lifeless> wgrant: I need to check exactly what triggers that
<lifeless> the storm cache is flushed on transactions but only in the webapps
<lifeless> scripts and tests don't do that
<lifeless> storm invalidates objects before reusing them
<wgrant> lifeless: I'd really like the cache to not be populated by an attribute access.
<wgrant> Because that makes it happen *everywhere*.
<lifeless> the reason it has an object cache is so that scripts and the like with long running references don't end up with aliasing problems
<lifeless> wgrant: I'd like attribute accesses to always be cheap. Because they happen everywhere.
<lifeless> wgrant: we're /dying/ from this today.
<lifeless> wgrant: Pages are doing thousands of queries
<lifeless> and prejoining is effectively useless
<lifeless> because:
<lifeless>  - its too greedy (pay the cost everywhere)
<wgrant> lifeless: In most cases it won't help.
<lifeless>  - its too specific (fails on many non-collection lookups)
<lifeless> wgrant: its a tool when used with tuple queries that works very well
<wgrant> lifeless: @cachedproperty normally only significantly saves queries when you do a prejoin beforehand.
<lifeless> thumper: yes, a store flush seems reasonable where you are doing updates/inserts
<wgrant> If you're doing a prejoin and you want this behaviour, *then* populate the cache.
<lifeless> wgrant: that requires the cache to exist
<wgrant> Doing it implicitly without checking every callsite is a recipe for disaster.
<lifeless> wgrant: Can you quantify that at all?
<lifeless> I don't like the idea of a 'sometimes its cached' decorator - it will be confusing as hell when looking at code
<lifeless> 'is this cached? I don't know? maybe?' bwah
<lifeless> wgrant: let me add some data here:
<lifeless>  - we *already* have cachedproperty on model objects
<wgrant> We do.
<lifeless>  - we *already* have adhoc hooks of that into storm
<lifeless> I'm proposing to systematise it so that we don't have accidental failures to flush those caches when the backing data is recalculated
<lifeless> you seem to be objecting to an improvement, which confuses me
 * thumper afk for lunch
<wgrant> Well, you also a couple of days ago proposed to immediately start caching tonnes of attributes.
<wgrant> I don't object to your invalidation improvement.
<lifeless> yes, which is btw no worse that prejoins AFAICT
<lifeless> we've been around that discussion
<wgrant> Assuming that DB triggers/views don't play tricks, prejoining is perfectly safe, since it only works for references.
<lifeless> if you're talking risk, thats a fairly big one :)
<wgrant> Hm?
<lifeless> we had a storm bug with triggers just this cycle
<lifeless> which took a day to debug
<lifeless> more-or-less
<wgrant> Right. But triggers are well-known and few.
<wgrant> They are part of the model.
<lifeless> wgrant: here's another way to think about it
<lifeless> on one hand we have death by sql
<lifeless> on the other we have a) cachedproperty and b) some huge handwaving I've done with nothing concrete and c) nil, zip, nada, nothing
<lifeless> I'm -very- interested in alternative approaches without the potential downside of cachedproperty.
<wgrant> Or d) cachedproperty populated by prejoins only when we explicitly want it
<lifeless> prejoins don't help.
<wgrant> How are you going to reduce the query count without a prejoin?
<lifeless> So please stop talking about them - I'm starting to think you don't understand the problem or something, which I *know* is wrong, because you do.
<wgrant> An uncached cachedproperty will still cause a query.
<lifeless> wgrant: lets define prejoin. I mean 'lp.services.database.prejoin' when I say prejoin.
<lifeless> I don't mean the general tactic of prepopulating things.
<wgrant> Ah.
<wgrant> That's the problem, then. I mean what you did with /participations or whatever it was.
<lifeless> ok
<wgrant> Like prejoining (in that we only return the first element), but also populating caches behind the scenes.
<lifeless> lets call that prepopulating
<wgrant> OK.
<wgrant> That technique is good. I need it for some publisher optimisations.
<wgrant> But having to turn everything into a cachedproperty for that sucks.
<wgrant> There are some parts of the publisher where I cannot safely do that.
<wgrant> I need them to only be populated when I ask. Not when I just request the attribute.
<lifeless> why
<lifeless> I'm asking for details here
<lifeless> not to be silly
<wgrant> eg. SPR.files. The uploader needs to add things to that, then iterate over them later.
<wgrant> But the publisher needs to get them all in one query.
<wgrant> If I make it a cachedproperty, I can prepopulate it in the publisher.
<lifeless> and you can update the cache when you add things to it
<wgrant> But why?
<wgrant> In the uploader case, I don't want it cached!
<wgrant> I shouldn't have to invalidate a cache that serves no purpose. The cache shouldn't exist.
<lifeless> no? You *want* to go back to the DB when you iterate?
<wgrant> Possibly not. But imposing a cache invalidation problem on every object modification in the entire application is... not good.
<lifeless> so there are a few broad approaches here
<wgrant> if we were writing this from scratch, perhaps.
<lifeless> *) different attributes - some cached, some not. Make them visually distinctive
<wgrant> But so much code has been written that assumes that attributes behave sanely.
<lifeless> *) single attributes which are sometimes cached and sometimes not, with a switch to toggle.
<wgrant> We should not be immediately breaking assumptions made by a few hundred thousand lines of code.
<lifeless> wgrant: I understand you feel strongly about this, but you're inflating your argument with hyperbole; its not helping us design a solution that addresses what you want and what I want.
<lifeless> any one attribute can be looked at and assesed.
<lifeless> the surface area for any one change is not 'the entire code base'
<wgrant> But we want a solution that can be widely applied.
<lifeless> and the entire code base is < 200K *anyway*
<lifeless> wgrant: sure, so lets get back to talking about it.
<lifeless> I'm just saying, going off the deep end shuts down conversation and brainstorming
<wgrant> lifeless: There's like 400KLOC of Python.
<lifeless> 270K including tests.
<lifeless> and for assessing the risk tests are not relevant: they are a problem for whomever puts a specific branch together, its the headache when we silently break production that really worries me.
<wgrant> My "find -name '*.py' | xargs wc -l" says 511K, but anyway.
<wgrant> Right.
<lifeless> I used sloccount
<lifeless> on lib/lp
<lifeless> and then lib/canonical
<lifeless> because including all of subvertpy etc didn't make sense to me
<lifeless> anyhow
<wgrant> Well, this is a clean branch without sourcecode linked.
<lifeless> theres code
<wgrant> But yes, anwyay.
<lifeless> it does stuff
<lifeless> and many many things it does are terribly slow because we do incremental DB access
<lifeless> So back to approaches
<wgrant> Certainly.
<lifeless> excluding radical ones - I want to do them, just *after* we are in a decent place vis-a-vis performance.
<lifeless> *) different attributes - some cached, some not. Make them visually distinctive
<lifeless> *) single attributes which are sometimes cached and sometimes not, with a switch to toggle.
<lifeless> *) attributes are never cached, add cached methods
<lifeless> *) attributes are always cached, add noncached methods
<lifeless> *) Alternatives go here <...>
<lifeless> what you appear to be arguing 8for* is the second one.
<wgrant> None are pretty.
<lifeless> right
<lifeless> I *totally* agree.
<wgrant> I'd prefer it if we could add a caching wrapper around objects.
<wgrant> But that is impossible.
<lifeless> well, its possible, but I think it would give us two problems :)
<lifeless> (its python, with sufficient will most anything is possible)
<wgrant> Heh.
<wgrant> But anyway, that leaves us with mutating the global Storm objects.
<lifeless> yes
<lifeless> or a lookaside weak dict
<lifeless> which is equivalent and slower.
<lifeless> so lets exclude that
<wgrant> Well, regardless we have to modify the global objects' behaviour.
<lifeless> sure
<lifeless> so here are some constraints I have
<lifeless> I want the *default* to be fast.
<lifeless> or if I can't get that, that when someone makes something fast, it behaves decently in a very broad scope.
<lifeless> I want this because most of our pages are slow, and most of them do not mutate things so most of the time any of the implementations we've thought of should be ok.
<lifeless> at *worst* I want it to be very easy to take something from slow to fast
<lifeless> secondly, I do not want heisenbugs.
<mwhudson_> thumper: yeah, sounds basically ok
<mwhudson_> thumper: errors in tearDown or cleanups fail the test
<lifeless> Folk debugging the code should not need to see object state to tell if something is or isn't caching.
<lifeless> Its ok that they need to read the code (but not ideal)
<lifeless> its not ok if they need a memory dump from the live server
<wgrant> Thaat's true.
<lifeless> A criticism I have of the solution you appear to favour is that its likely to need that memory dump, if - e.g. a kept storm object has the 'cache this attr now' turned on and not reset (due to a bug)
<lifeless> its vulnerable to cascade failures
<lifeless> always on
<mwhudson_> lifeless: are you sure storm doesn't flush it's per-thread cache on transacton boundaries btw?
<lifeless> or always off
<wgrant> It is.
<lifeless> mwhudson_: check publication/webapp.py or whatever.
<lifeless> mwhudson_: we do it
<wgrant> But it means we need to audit *lots* of code.
<lifeless> line 699
<mwhudson_> lifeless: you're probably right, but just because we do something by hand doesn't mean it doesn't happen automatically too :-)
<lifeless> mwhudson_: oh, so yes, I am sure.
<lifeless> mwhudson_: or all our doctests would fail.
<mwhudson_> ok
<mwhudson_> um
<lifeless> because the object outside userbrowser lookups would be different
<lifeless> and hilarity would ensue with aliasing issues and the like
<lifeless> if thread_name != 'MainThread' or name.endswith('-slave'):
<lifeless>     store.reset()
<lifeless> is what we do, for exactly that reason.
<mwhudson> aaah
<lifeless> yes, yes you should run screaming.
<mwhudson> hang on a sec
<mwhudson> presumably we somehow don't call transaction.begin() (or moral equiv) at the start of testbrowser requests either
<mwhudson> or we would have similar hilarity
<lifeless> I haven't looked at exactly what testbrowser does
<lifeless> it scares me that we avoid the publication machinery, given what is hooked into publication
 * mwhudson confuzzles
<wgrant> lifeless: So, OK, your approach is OK if we are going to move to it throughout the application.
<wgrant> But it's a fair bit of work.
<lifeless> what do you think of having foo_uncached attributes where you need them ?
<lifeless> or perhaps
<lifeless> a dynamically generated getter
<lifeless> @cachedproperty('bwah')
<lifeless> def foo
<lifeless> -> _bwah_cached cache attribute
<lifeless> -> bwah_uncached lookup bypassing the cache
<lifeless> it won't layer well
<lifeless> if you have Foo referencing Bar and Bar has a thing you need uncached from Foo, you're ok.
<lifeless> but if you have Quux->Foo->Bar, you're screwed if you try .foo.bar.thing
<lifeless> OTOH you shouldn't do that anyhow :P
<wgrant> Well. Let's think about the cases where we are going to need something uncached.
<wgrant> 1) We start a new transaction.
<wgrant> 2) We make a change. The method making the change should invalidate or update the cache.
<wgrant> 3) DB triggers or views play tricks on us.
<wgrant> Any others?
<lifeless> 1) isn't strictly true, but its a damn sensible default
<lifeless> 2) yes, agreed.
<lifeless> 3) yes, agreed (and thats where storm broke this release too)
<wgrant> 2) is difficult, particularly if we're doing mass-updates as it seems we might.
<lifeless> if we're doing mass updates we shouldn't realise the objects in the first place
<lifeless> the existing storm objects will get broken by that anyway, so its not a unique problem to cachedproperty decorators
<wgrant> I guess that case is pretty much the same as 3)
<wgrant> So 3) should be revised to changes in the DB, not going through the objects.
<lifeless> functionally equivalent
<lifeless> yes
<lifeless> is DistroSeries:EntryResource:getBuildRecords fixed?
<lifeless> wgrant: so, thats when we need uncached things
<lifeless> wgrant: do we then say 'if you have a cache, and you invalidate it,the owning object invalidates the cache'
<lifeless> e.g. Foo.newBar() -> Foo's Barcache is Foo's job to invalidate
<wgrant> lifeless: It should be fixed, yes.
<wgrant> lifeless: And I think that example is correct.
<lifeless> I wonder if we can hook in to make a simple del attrname invalidate without wiping the cache logic
<lifeless> we have about 40 uses of cachedproperty, in model code, in a simple check today
<lifeless> including sourcepackagerelease
<lifeless> wgrant: did you fix DistroSeries:EntryResource:getBuildRecords or is it my imagination ?
<wgrant> lifeless: I'm pretty sure it's fixed.
<wgrant> I mean, the query it executes now sucks 30x less.
<jamesh> if you're doing mass updates via Storm's ResultSet.set() API, Storm should keep its own caches coherent
<jamesh> if that helps.
<lifeless> jamesh: does it trigger invalidate() ?
<jamesh> lifeless: no, but it marks the columns you're updating as AutoReload (or the new values if the query is simple enough)
<lifeless> ah
<lifeless> simple would have to be darn simple ;)
<jamesh> see the compile_python code in storm.expr for an idea of what counts as simple
<lifeless> I just mean it has to either be explicit primary key values
<lifeless> or all rows
<lifeless> no?
<jamesh> it can do more than that.
<lifeless> I didn't think UPDATE returned the row ids
<jamesh> if you do store.find(Foo, name="blah").set(x=42), it will update any cached Foo object with name=="blah"
<lifeless> ok, thats nice
<jamesh> it tries to create a python function based on the where conditions.  If it can't, then it will invalidate the column on all cached Foo objects
<lifeless> so
<lifeless> we want to cache derived data
<lifeless> like store.find(Foo, And(Foo.id==self.foo, Foo.status=="finished"))
<lifeless> on Bar
<lifeless> to avoid death-by-sql
<jamesh> this sounds like you want a constrained version of Reference()
<lifeless> + eager loading
<lifeless> but only when we want it
<jamesh> or are you after something more general?
<lifeless> and very fine control
<lifeless> jamesh: does storm have constrained references ?
<jamesh> no
<lifeless> heh
<lifeless> https://dev.launchpad.net/Database/Performance
<lifeless> is a first sketch starting to write up some of this
<lifeless> also
<lifeless> https://dev.launchpad.net/Performance may be interesting-though-I-bet-you-know-it
<jamesh> IIRC, when I was optimising some soyuz views way back, I did one query to get all the pairs of objects and used lazr decorates(...) code to override the slow attribute in the view.
<lifeless> yeah
<lifeless> the problem with that is that the default is 'behave badly' :)
<jamesh> well, the definition of good behaviour often depended on what the view was trying to do.
<lifeless> thats true
<lifeless> you don't want unnecessary columns included
<lifeless> s/columns/tables/
<lifeless> what I meant is that when you have any model methods
<lifeless> a delegated approach to fix things doesn't help the core object
<jamesh> right.
<lifeless> so you end up going through contortions to achieve that
<jamesh> if you are working on this, don't treat storm as a black box though: improving Storm may be a better way of fixing your problem
<jamesh> (we can always use more help)
<lifeless> jamesh: oh, I'm not
<lifeless> I spent some time talking with jkakar about this
<jamesh> awesome
<lifeless> my rough ideas there are:
<lifeless>  - stop subclassing to get objects - use a separate compile step, so we have clear separation of responsibilities
<lifeless>  - define a formal user accessible caching layer which can handle more complex algebras
<jamesh> "subclassing to get objects"?
<lifeless> we have classes that represent what storm gets from the DB *and* what we write code on
<lifeless> thats confusing responsibilities
<lifeless> its confusing because it becomes unclear whether foo.bar implies database access
<lifeless> or local data only access
<lifeless> so I want to structure things such that when you're in python land, there is no ambiguity
<lifeless> put data access and updates alongside rather than mixed-in
<lifeless> what we have is awesome for interactive use, ok for batch processing, and poor for web serving - IMNSHO
<lifeless> I don't see value, from a performance perspective, in the attribute lookup facilities of Reference, ReferenceSet
<lifeless> I may being overly cautious here.
<lifeless> Its early days now
<lifeless> but I'm thinking about how to move LP to a staged model where no DB access /can/ happen during template rendering
<lifeless> and we have really really clear 'this is doing db access' now behaviour
<lifeless> because as developers, its really hard to reason about performance when things are sometimes fast and sometimes slow.
<jamesh> I know therve's twisted-integration branch tried to get things to a point where database access only happened at defined points.
<lifeless> I think if you make all the objects readonly and don't use Reference or ReferenceSet, that it will be fine.
<lifeless> push updates over to ResultSet
<jamesh> that work has never really gotten to a point where it can be used for real work though.
<lifeless> anyhow - this is long term stuff
<lifeless> we can get a long way with cachedproperty and DecoratedResultSet
<lifeless> but when we're down solidly under 5 seconds hard timeout, then I want to start looking at this.
<lifeless> jamesh: https://lists.launchpad.net/launchpad-dev/msg04228.html shows where we're at - and most of these pages are driven by that sort of problem IME
<lifeless> which is admitedly limited :)
<lifeless> elmo: when you get up
<lifeless> https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100712/20100811/
<lifeless> you can see a oops spike on a weekly basis
<lifeless> elmo: can we determine whats going on there ? Perhaps its that backup issue we identified ?
<lifeless> rockstar: hey
<rockstar> lifeless, hi.
<lifeless> rockstar: you might like to file a bug that our email stack *lets* us send emails without rationales
<lifeless> rockstar: separately from the rosetta instance
<spiv> +1
<rockstar> lifeless, hadn't thought about that.
<lifeless> rockstar: we should at least have to say explicitly that we don't want a rationale
<rockstar> lifeless, I can't imagine why we would ever not want a rationale.
<lifeless> right
<lifeless> but at the moment we don't have one in N places
<lifeless> so that would provide a migration strategy
<lifeless> the folk searching lp bugs for porn amuse me.
<lifeless> anyhow, its just a thought
<lifeless> we have a class of bugs that can be easily made explicit rather than accidental
<rockstar> lifeless, people actually type "porn" in the bug search box?
 * rockstar goes to do just that
<rockstar> Oh man, these bugs are hilarious.
<lifeless> rockstar: check the oops report from lpnet today
<lifeless> look at the exceptions
<rockstar> lifeless, oh man, that's awesome!
<lifeless> its possibly an attack by a spammer trying to inject that
<lifeless> I haven't looked closely
<lifeless> (pun intended)
<ajmitch> someone doing a bit of security testing on LP? :)
<lifeless> not with those queries ;)
<ajmitch> now you've made me curious
<lifeless> fairly scummy porn terms, in permutations, with some verging on illegality most places
<ajmitch> heh
<poolie> lifeless/whoever, my last run of flags-webapp had one failure in the twistedjobrunner, which i'm confident is spurious
<poolie> and one in    lp.registry.windmill.tests.test_team_index.TestTeamIndex.test_addmember
<poolie> which is probably spurious - do you want to reassure me it is?
<lifeless> poolie: hi
<lifeless> its in windmill
<lifeless> its probably spurious.
<lifeless> is it repeatable ?
<poolie> just finished, and it didn't fail the second time
<poolie> \o/
<poolie> what a saga
<poolie> so i guess i can't land this atm anyhow? not anywhere?
<lifeless> thats right
<lifeless> not the PQM is release-critical message above is cleared
<poolie> ok
<lifeless> poolie: I'm really glad you've gotten it solid
<lifeless> poolie: If you want to keep momentum up, I'd start a new branch building on it
<thumper> poolie: or you could fix the commit to a stacked branch bug ;-)
<poolie> lifeless: yes i think i'll do that; thumper john is working on commit-to-stacked when he flushes his queue
<thumper> ok
<lifeless> morning stub
<lifeless> stub: I has a pressie for you  - more code to review :)
<stub> yo
<lifeless> Ursinha-bbl: hi
<lifeless> Ursinha-bbl: your script has gone nuts ;)
<lifeless> https://bugs.edge.launchpad.net/launchpad-registry/+bug/615237
<_mup_> Bug #615237: /participants API timing out <timeout> <Launchpad Registry:In Progress by lifeless> <https://launchpad.net/bugs/615237>
<StevenK> lifeless: You can't QA that one before it lands?
<lifeless> StevenK: its not landed
<lifeless> StevenK: the qa bot went nuts
<lifeless> also bug 84838
<_mup_> Bug #84838: code browser should use oops system <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by spiv> <https://launchpad.net/bugs/84838>
<lifeless> although that one is merged, so ignore me on this point
<Ursinha> wgrant, hi
<Ursinha> hmm
<Ursinha> thumper, hi
<lifeless> hi Ursinha
<Ursinha> hi lifeless
<spiv> lifeless: co-incidentally I'm just about to hit "submit" on the merge proposal for the production configs change for codebrowse OOPSes.
<lifeless> spiv: \o/
<lifeless> spiv: after you do that
<poolie> nice
<lifeless> spiv: you need to file a bug on oopstools to have matsubara update the config so the back end finds them
<poolie> next meliaes?
<spiv> lifeless: yes, and do an RT to make sure those OOPSes will be synced or whatever is supposed to happen to them, I think.
<lifeless> spiv: yes please
<spiv> doing now :P
<lifeless> Ursinha: I'm curious why tagger script takked 615237 as fix committed etc
<wgrant> Ursinha: Hi.
<lifeless> Ursinha: I think there was some confusion
<Ursinha> bug 615237
<_mup_> Bug #615237: /participants API timing out <timeout> <Launchpad Registry:In Progress by lifeless> <https://launchpad.net/bugs/615237>
<lifeless> Ursinha: you'll need to look at it to see the activity
<Ursinha> lifeless, I'm looking, wanted the link :)
<lifeless> :>
<lifeless> Ursinha: pad.lv/615237 - nearly as easy
<Ursinha> lifeless, actually I'm kinda smoke testing the refactored tagger
<lifeless> Ursinha: cool!
<Ursinha> so there's the confusion
<poolie> lifeless: can you think of any good examples i should crib from for the flag editor ui?
<poolie> either for code or tests or both
<lifeless> of course, I hope I didn't add a bug to it ;)
<Ursinha> lifeless, no, the refactoring was really cool :)
<lifeless> poolie: hmm, you know, I'm not sure.
<Ursinha> lifeless, it's working ok, only think that's a PITA is that devpad is hardy and runs old version of lplib
<lifeless> thumper: hey ^ poolies question.
<Ursinha> lifeless, is there anything you could do about it?
<Ursinha> lifeless, so stuff works locally but not there
<Ursinha> that sucks
<lifeless> Ursinha: about hardy on devpad ?
<Ursinha> lifeless, yes, or maybe upgrading lplib
<lifeless> the losas don't want to break things while they are sprinting
<Ursinha> wgrant, do you know if all of your soyuz changes have been tested?
<Ursinha> lifeless, argh
<lifeless> but we can request that devpad be near/at the front of the list
<Ursinha> really?
<Ursinha> yay
<lifeless> for next week
<Ursinha> cool!
<lifeless> I don't see why not
<lifeless> staging is lucid already
<wgrant> Ursinha: I don't know how non-bug QA works.
<wgrant> So I can't really tell.
<lifeless> Ursinha: I will mail the losas and flacoste now
<Ursinha> lifeless, thanks muchly
<Ursinha> wgrant, for those that have bugs
<wgrant> Most of them are untestable, anyway.
<Ursinha> wgrant, I'll give you the list and you tell me
 * wgrant checks.
<Ursinha> please
<wgrant> Sure.
<lifeless> Ursinha: is that soon enough ?
<Ursinha> lifeless, for what?
<lifeless> devpad to lucid
<Ursinha> lifeless, I guess
<lifeless> if you need something urgently there may be a backported lplib in CAT which we can use
<lifeless> Ursinha: you tell me :)
<Ursinha> lifeless, well, I need to port the script to run newer lplib
<Ursinha> that doesn't have transitionToMilestone and the like
<lifeless> I thought that was just the LP API version
<lifeless> if you specify that you want 1.0, it will use .milestone + lp_save()
<Ursinha> lifeless, iirc I tried and didn't work
<Ursinha> don't remember why now, too late for my brain
<poolie> should the ui code be in say registry, or is it ok in services?
<lifeless> Ursinha: ok
<lifeless> poolie: services - keep it with the component
<lifeless> poolie: features/browser/*
<lifeless> poolie: and features/templates/*
<Ursinha> wgrant, Bug 590708, Bug 241129, Bug 491418, Bug 605002, Bug 612157
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <qa-ok> <soyuz-build> <timeout> <Launchpad Foundations:Triaged by benji> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/590708>
<_mup_> Bug #241129: 'queue override' command scales at O(n^2) with the number of packages on the commandline <qa-needstesting> <soyuz-ftpmaster-tools> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/241129>
<_mup_> Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418>
<_mup_> Bug #605002: Soyuz doesn't accept upload with "Architecture: linux-any" <qa-needstesting> <soyuz-build> <soyuz-upload> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/605002>
<_mup_> Bug #612157: PPA size calculation should no longer exclude DDEBs <qa-untestable> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/612157>
<Ursinha> sorry the flood
<poolie> maybe featuredproject is a good comparable simple example?
<wgrant> Ursinha: The first and last are already OK and untestable respectively. 241129 needs DF console access, 491418 requires a buildd rollout (so isn't really relevant to the release anyway), and 605002 requires DF access too.
<wgrant> So, 2 and 4 should really be QAd, but I can't do it.
<Ursinha> wgrant, right. I'll talk to bigjools tomorrow
<wgrant> Thanks.
<Ursinha> wgrant, are those changes likely to explode everything else if qa-bad?
<wgrant> Ursinha: 4 could be pretty terrible, but that code has been exercised reasonably well on DF lately.
<wgrant> Just not the new case.
<Ursinha> wgrant, ah, right, so almost QAed :)
<lifeless> poolie: actually, I know an example that might be good
<lifeless> poolie: 'jabber ids' on people
<lifeless> thats something that is list-list
<lifeless> list-like
<Ursinha> wgrant, thanks for the info
<lifeless> https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.tag=qa-needstesting
<lifeless> hmm
<lifeless> kindof want to see assignee there
<Ursinha> lifeless, in the list?
<lifeless> yeah
<lifeless> not always
<lifeless> just now :P
<Ursinha> hehe
<lifeless> Ursinha: so, how do we say 'xxx is not relevant to the release, even though its not qa tested yet'
<lifeless> https://bugs.edge.launchpad.net/launchpad-code/+bug/84838
<_mup_> Bug #84838: code browser should use oops system <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by spiv> <https://launchpad.net/bugs/84838>
<lifeless> is fix committed
<lifeless> and is not tested
<lifeless> but its got a pending related branch for the prod configs
<lifeless> and then rsync + backend oops change
<lifeless> so it sshouldn't affect the release
<lifeless> Ursinha: would changing the milestone be enough ?
<Ursinha> lifeless, if that's going to be released next cycle, assign to the milestone in which that should be released
<Ursinha> lifeless, yes
<lifeless> done
<Ursinha> thanks
<lifeless> and to say 'does not need qa' ?
<lifeless> qa-untestable, or is there something more precise ?
<lifeless> bug https://bugs.edge.launchpad.net/launchpad-registry/+bug/611853
<_mup_> Bug #611853: remove remove_security_proxy_and_shout_at_engineer calls <qa-needstesting> <tech-debt> <Launchpad Registry:Fix Committed by sinzui> <https://launchpad.net/bugs/611853>
 * lifeless goes with untestable for now
<lifeless> wgrant: can you aid with the qa of the other three soyuz things?
<Ursinha> lifeless, qa-untestable
<lifeless> rockstar: don't support you're around ? https://bugs.edge.launchpad.net/launchpad-code/+bug/592937
<Ursinha> lifeless, I had this question on epic, but I couldn't convince myself that another kind of untestable tag was justified
<_mup_> Bug #592937: "View source package recipes" oddly located on Person index <confusing-ui> <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by rockstar> <https://launchpad.net/bugs/592937>
<lifeless> Ursinha: its fine by me as it is; just wanting to do the right thing
<wgrant> lifeless: The three Ursinha gave me above?
<Ursinha> lifeless, ok :)
<Ursinha> lifeless, I guess wgrant already gave his take on those three
<Ursinha> he needs access he doesn't have to test that
<lifeless> ah
<wgrant> Two of them are untested, one of them won't be rolled out tonight anyway. Of the two that are untested, one has been mostly tested, and the other is in a rarely-used archive admin script.
<wgrant> So, not terribly critical.
<lifeless> wgrant: there are 4 ;>
<lifeless> 3 soyuz one buildd
<wgrant> What are the three Soyuz?
<wgrant> There's one Soyuz/buildd.
 * lifeless refreshes the page
<lifeless> https://bugs.edge.launchpad.net/soyuz/+bug/241129
<_mup_> Bug #241129: 'queue override' command scales at O(n^2) with the number of packages on the commandline <qa-needstesting> <soyuz-ftpmaster-tools> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/241129>
<lifeless> https://bugs.edge.launchpad.net/soyuz/+bug/491418
<Ursinha> lifeless, I need to poke bigjools tomorrow about those
<_mup_> Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418>
<lifeless> https://bugs.edge.launchpad.net/soyuz/+bug/605002
<_mup_> Bug #605002: Soyuz doesn't accept upload with "Architecture: linux-any" <qa-needstesting> <soyuz-build> <soyuz-upload> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/605002>
<lifeless> and the buildd one
<lifeless> https://bugs.edge.launchpad.net/launchpad-buildd/+bug/491418
<_mup_> Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418>
<wgrant> lifeless: 491418 is the buildd/Soyuz one.
<lifeless> right
<lifeless> so 491418 we can ignore - lamonts baby
<wgrant> RIght.
<lifeless> the queue override one is the console one
<lifeless> I'm not sure why the other two are untestable
<lifeless> can't you toss something at stagings upload queue and see what happens ?
<wgrant> staging doesn't have an upload queue.
<lifeless> !
<wgrant> DF does, but it needs someone to run everything.
<lifeless> I'll RT that now.
<wgrant> It doesn't have an archive setup. That's why we have DF.
<lifeless> wgrant: staging /exists/ for QA
<lifeless> if it can't QA everything, its a bug.
<wgrant> And DF exists for Soyuz QA!
<lifeless> DF is intended to be rather broader than that.
<lifeless> wgrant: do you think we should /not/ be able to QA this on staging?
<wgrant> lifeless: No, but I think it's a fair bit of work to get it set up there.
<lifeless> wgrant: longest journey, smallest step
<lifeless> done
<lifeless> wgrant: the goal of the QA environment is to QA changes; dogfood is great for experimenting and playing with stuff, but its not generally accessible nor maintained in the same strict way staging is.
<lifeless> wgrant: *if* staging ran an upload queue and had a virtualised builder, we could qa recipes, and both of your fixes.
<lifeless> rt 40837 for reference
<thumper> poolie: I'd suggest playing with balsamiq a bit to design the interface
<poolie> thumper: how do i get an account or access?
<Ursinha> thumper, do you know about QA of bug 614858 and bug 592937?
<_mup_> Bug #614858: Recipe builds break the world when requester does not have a preferred email address <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by thumper> <https://launchpad.net/bugs/614858>
<_mup_> Bug #592937: "View source package recipes" oddly located on Person index <confusing-ui> <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by rockstar> <https://launchpad.net/bugs/592937>
<thumper> poolie: canonical has a license to use it
<thumper> poolie: and you can download and trial easily
<thumper> Ursinha: 614858 requires a daily build to run on staging which requires a losa to run the script
<thumper> 592937 not so much
<Ursinha> thumper, anyone of those could destroy the world if qa-bad?
<thumper> Ursinha: given my unittests, I'm pretty certain it is fine
<thumper> Ursinha: but I can't qa it right now
<thumper> 592937 isn't that important
<thumper> it just moves a link around
<wgrant> And it's on edge -- it's fine.
<Ursinha> thumper, right
<Ursinha> thumper, when could you QA it?
<lifeless> stub: I tried to capture some common things - https://dev.launchpad.net/Database/Performance - would love it if you were to garden that page :>
<lifeless> stub: I didn't get into db design issues at all, I think that with you reviewing every db patch we don't need to try to and pre-communicate about that at this stage
<stub> I'll look after fooding
<lifeless> yeah no rush
<poolie> what's the tasteful way to define the page title? make it a property of the view object?
<lifeless> theres a slot that is automatically filled
<wgrant> There are page_title and label properties, IIRC.
<wgrant> I forget which one is which.
<wgrant> I think page_title is used in the <title> and breadcrumbs, while label is used in the h1.
* spm changed the topic of #launchpad-dev to: Launchpad down/read-only from 0800-0930 UTC for a code update | Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
<lifeless> spm: we're releasing ?
<spm> lifeless: yes; tho that you're questioning implies a level of surprise?
<lifeless> excitement
<lifeless> I plan to watch the hourly oops report like a hawk
<maxb> buildbot/pqm has been switched to watching the *lucid* builders  <--- exciting too
<lifeless> stub: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-timeout-candidates.html is now updated
<lifeless> stub: and yeah, LFA is terribad :)
<maxb> That's a nice new word :-)
<kb9vqf> Any idea what might suddenly cause this?
<kb9vqf> http://pastebin.ubuntu.com/476804/
<lifeless> kb9vqf: looks like an API has changed
<kb9vqf> This is on all buildd systems now
<kb9vqf> Uh oh
<lifeless> is this on the launchpad.net deployment ?
<kb9vqf> A local copy of Launchpad
<lifeless> ok
<kb9vqf> I did just upgrade some packages on the master server
<lifeless> so make sure you're running the latest launchpad-buildd and launchpad code bases
<kb9vqf> lifeless: I don't think it's an API change--I just looked at the latest slave.py file in bazaar and the API for the failing function is identical
<kb9vqf> (to my version, that is)
<kb9vqf> This all started after I tried to add a new distribution series to the database
<lifeless> wgrant: ^ ring any bells?
<wgrant> kb9vqf: I haven't seen that in ages. You've restarted buildd-manager?
<wgrant> It just seems to... happen... sometimes.
<wgrant> I think I knew the reaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaspoint.
<wgrant> Woah.
<kb9vqf> ???
<wgrant> "reason at some point", that was.
<kb9vqf> Yeah, me too
<kb9vqf> But I can't pull it up in my notes
<kb9vqf> Seems really familiar
 * kb9vqf goes to restart buildd manager
<adeuring> good morning
<kb9vqf> wgrant: That was it...a buildd-manager restart cured the problem
<kb9vqf> I'm writing that down somewhere for reference ;-)
<wgrant> Hmm.
<kb9vqf> wgrant: I don't know if this will help, but I inserted a debug statement printing the RPC function call arguments
<kb9vqf> The results are here: http://pastebin.ubuntu.com/476811/
<kb9vqf> It looks like something sends an invalid set of arguments beginning with "urlbase" when the failure occurs
* henninge changed the topic of #launchpad-dev to: Launchpad is currently down/read-only until 0930 UTC for a code update | Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad
<mrevell> Hey up
<poolie> lifeless, if any, i have a form showing current rules
<poolie> just trying to work out how to make it cope nicely with items with no single integer primary key
<lifeless> poolie: nice
<lifeless> mthaddon: hey, when you have a second: what time does tuolumne do the hourly oops graph updates?
* spm changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad
<wgrant> buildd-manager seems to be doing reasonably well.
<mwhudson_> oh, we haz rollout?
<lifeless> yep
<bigjools> wgrant: it's not doing too bad is it, although there's a couple of problems
<bigjools> not seen them again since it start though
<wgrant> bigjools: It's still slow.
<wgrant> bigjools: But it's at least staggered.
<wgrant> So it should be less prone to getting itself into spirals of slow, slow death.
<bigjools> it will be slightly better than before
<bigjools> in terms of throughput
<wgrant> Right.
<bigjools> because we're interleaving more stuff now
<wgrant> The main problem before was resumes, right?
<wgrant> They'd block everything for a couple of minutes.
<bigjools> although I am getting "error: (4, 'Interrupted system call')" a lot :/
<jml> is Launchpad writable yet?
<bigjools> and it ends up disabling the builder
<bigjools> jml: yes, since 30 mins ago
<jml> yay
<wgrant> bigjools: Hm, is that a slave exception?
<bigjools> wgrant: no
<wgrant> Also, what's with the db_lp failure?
<jml> it normally means that a signal happened during a system call
<jml> and *that* normally means that you should try again
<bigjools> yes - I suspect it's getting sigchld since I am interleaving Popen with data comms to builders
<jml> that would make sense
<bigjools> jml: since it's used Twisted XMLRPC I would have hope that it would have taken care of that for me :/
<jml> bigjools, what? no.
<jml> bigjools, Twisted XMLRPC doesn't do magic process handling
<jml> bigjools, reactor.spawnProcess, otoh.
<bigjools> surely it should catch EINTR and retry
<bigjools> http://pastebin.ubuntu.com/476868/
<jml> the reactor does that.
<poolie> bigjools: strangely enough i just fixed a bug like that in non-twisted lp code
<jml> bigjools, you are not using Twisted XMLRPC there.
<poolie> i wonder if something we import hooks a signal so we see it
<bigjools> ah you know what, that bit's not twisted
<jml> :)
<poolie> see eintr when we normally won't
<bigjools> poolie: ah hmmm
<jml> poolie, aiui, what we're seeing is the normal behaviour of Python's libraries
<lifeless> there's a regular sigchld bug
<lifeless> hmm, actually. ESLEEP.
<lifeless> I shall read backchat tomorrow :)
<poolie> you shouldn't get eintry from sigchld unless you've registered for sigchld
<jml> heh
<poolie> i think
<poolie> it's complicated
<poolie> anyhow i should go too
<poolie> good night
<jml> testr branches will be reviewed maÃ±ana
<bigjools> we have registered it I think
 * jml stops talking about programming and instead reads resumes
<bigjools> that so needs an accented e
<mwhudson_> risumi
<bigjools> rezumey
<bigjools> I thought he was resuming reading
<mwhudson_> i is what you get if you send e-with-acute-in-latin-1 via an email system that strips the high bit
<bigjools> it would be so much easier if everyone spoke the Queen's English
<lifeless> mthaddon: https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100811/20100812/ seems to be missing about 10 hours of samples ?
<lifeless> mthaddon: ah thanks, dunno what you did but it just updated
<bigjools> losa: is everything looking ok from your PoV?
<jml> bigjools, the compose key needs to be easier to use
 * jml wishes again that Linux just used Mac-style keybindings
<bigjools> prob more of a desktop environment issue
<jml> I'm a muppet user. It's all Linux to me.
<bigjools> which muppet character would you be?
<wgrant> jml: How could the compose key be easier? RAlt+' e
<jml> wgrant, for a start, I can never remember if it's RAlt or the menu key (it's the menu key on my laptop)
<jml> wgrant, secondly, discovering key combinations is hard
<wgrant> jml: Well, it's configurable.
<jml> wgrant, thirdly, some of the key combinations in system compose files just don't work
<wgrant> Hah.
<jml> wgrant, I know that. I don't want to configure it.
<jml> fourthly, some of the compose combinations are silly, especially when compared to the Mac equivalent
<jml> e.g. en dash is Compose+- - . on Linux, and Opt+- on Mac
<jml> em dash is Compose + - - - vs Opt + _
<wgrant> Hmm.
<stub> The first problem is having to enable that crap, which usually involves configuring it, which means everyone uses a different setup, so documentation becomes non-existent and people think it just doesn't work.
<jelmer> AltGr seems to work pretty consistently across Linux desktops, I don't remember having to configure it
<jelmer> jml: What keyboard layout are you using?
<jml> jelmer, dvorak
<wgrant> jelmer: Most of us don't have an AltGr.
<jelmer> jml: What country ?
<jml> US
<jelmer> hmm, same here
<deryck> Morning, all.
<lifeless> deryck: we'll find out if the new search gets widespread +1 or -1 today ;)
<deryck> lifeless, indeed!  Fingers crossed. :-)
<bigjools> wgrant: the buildd-manager suddenly decided to start dispatching a build for a disabled rebuild archive :/
<wgrant> bigjools: Not again... :/
<wgrant> bigjools: How fatal is that now?
<wgrant> There's no cycle to abort every time.
<wgrant> Although I guess that just means every builder tries to dispatch it.
<bigjools> wgrant: so it's pretty fatal as yes, every build tries in turn to dispatch it :)
<bigjools> builder*
<bigjools> wgrant: what I am more confused about is why is the disabled flag getting ignored now
<wgrant> bigjools: archive.disabled hasn't been honoured for a while.
<wgrant> job.status == SUSPENDED is how it's done.
<bigjools> wgrant: I beg to differ
<bigjools> oh ok, well same shit
<wgrant> Well, the latter could possibly be inconsistent with archive.disabled, so the distinction is important....
<bigjools> this means the jobs got re-enabled somehow
<bigjools> yes
<wgrant> Possibly a queue-builder run?
<bigjools> I'm butchered them now anyway
<bigjools> q-b hasn't run in months
<bigjools> now I need to fix this eintr issue, we're getting regular disabled builders :(
* bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08 | 10.08 is released, PQM is OPEN | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad
 * bigjools wonders how to test something that handles eintr
<jelmer> Would make sense for ./utilities/ec2 land to CC the MP in question with the test results? I currently find myself going through failure emails that all have the same subject, it would be nicer if the results could just be found in the MP.
<jml> jelmer, hmm.
<jml> jelmer, first up, it would make sense to have the MP name in the subject.
<jml> jelmer, secondly, you are reminding me about how poor the email is and how I ought to fix it.
<jml> jelmer, I don't think the MP should have test results in the case of failure (although I wouldn't be opposed to attaching the subunit log)
<jml> jelmer, what actual data would you like to see on the MP?
<jelmer> jml: Personally what I'm interested in is the list of test ids that is failing, so I can run just those tests on my local machine.
<jelmer> Having the full log attached would also be useful in case of e.g. spurious test failures, but is less important imo
<jelmer> It might be different for other people though
<jelmer> jml: Why should the MP not have the test results in case of failure?
<jml> jelmer, well, it depends on what you mean by test results. the current email is awful.
<jml> jelmer, a list of failing tests would be fine by me.
<jml> jelmer, and I don't think MPs have attachments, sadly.
<jelmer> jml: Ah, right. I was confusing them with bugs. :-/
<jelmer> What would be totally awesome is if Launchpad could support attachments for MPs in the same places as comments, and then display summaries when it encounters subunit files.
 * jelmer dreams on
<jml> jelmer, yeah, that would be good. :)
<sinzui> bigjools, ping
<sinzui> bigjools, I can make all bugs fix released. all lp, or just my team
<sinzui> flacoste, I can make all bugs fix released. all lp, or just my team
<flacoste> sinzui: i think diogo has a script hat did the same, no?
<sinzui> I have never seen it close all the fixcommitted bugs
<sinzui> I have done all lp for the last two months and I have always done it for the projects I manag
<matsubara> sinzui, flacoste: https://code.edge.launchpad.net/~launchpad/lp-qa-tools/bug-editor
<bigjools> sinzui, matsubara: ya, please release them
<bigjools> the script I committed to do that ages ago got moved to launchpadlib contrib and now I can't find that either
<jml> jcsackett, as a new developer, I'd like your opinion on the README that just landed. (no rush though)
<bigjools> matsubara: your script doesn't work any more
<bigjools> sinzui: where's yours?
<matsubara> bigjools, it's been a long time since I used it. should I fix it or are there any other newer tools that does the same thing?
 * bigjools has no idea
<sinzui> I just closed the registry bugs. I can start on the whole team
<bigjools> what replaced transitionToStatus?
<sinzui> bigjools, ``bug_task.status = u'Fix Released'`` it is exported as a mutator in the api
<bigjools> ah
<candrea> sinzui: on the registry bugs I'm subscribed to I've seen the comment "Fix released in...", but no status changes (e.g. bug 70613)
<_mup_> Bug #70613: Encourage projects to configure all services (eg. bug tracker) <bridging-the-gap> <qa-ok> <Launchpad Registry:Fix Committed by bac> <https://launchpad.net/bugs/70613>
<sinzui> candrea, :( Yes. I see that to. I am missing an lp_save
<jcsackett> jml: sure; i've got a suite running now so i can take a look. anything in particular you would like comments on?
<jml> jcsackett, nothing in particular. does it help you understand Launchpad, do you wish you had it a week ago, is it confusing, etc
<jcsackett> jml: dig.
<jcsackett> jml: definitely helpful, it covers a fair bit i was dumped into in the first days.
<jcsackett> jml: assuming you don't mind a bit of crossover with dev.launchpad, a teeny bit about the rocketfuel utils might be nice.
<jml> jcsackett, good idea
<jml> jcsackett, though I personally _never_ use the rocketfuel utils
<jcsackett> jml: i'm still wavering between using them and straight bzr. i find rocketfuel a tad confusing, but the auto run of makebuild &c when you branch isn't half bad.
<jcsackett> jml: i also agree with flacoste. you hit the mark on tone.
<jml> jcsackett, thanks :)
<jcsackett> jml: you're welcome.
<deryck> gmb, hi.  Is bug 424849 done and out in 10.08?  The branches are listed as merged, but I thought this work was still in progress.
<_mup_> Bug #424849: Launchpad should batch attachment notification emails <story-better-bug-notification> <story-better-notification-sending> <Launchpad Bugs:In Progress by gmb> <apport (Ubuntu):Invalid> <https://launchpad.net/bugs/424849>
<gmb> deryck, Argh, it's listed as done because I did part of the work, not all of it. It isn't finished, so I'll mark it as Triaged for now since we won't be coming back to it for a bit.
<deryck> gmb, great, thanks.  Remove the milestone, too, please.
<gmb> Done
<jml> james_w, are your html matchers available in Launchpad?
<jml> as in, for Launchpad development
<james_w> jml: rockstar might have landed them, I'm not sure.
<jml> james_w, thanks.
<james_w> jml: what would you like to use them for? (use case gathering)
<jml> james_w, testing link formatters
<rockstar> jml, I have a branch that lands them, but that was before james_w added his matcher stuff too.  :)
<rockstar> (I did it on the plane back from Prague)
<jml> james_w, testing this code: http://paste.ubuntu.com/476985/
<jml> rockstar, cool.
<james_w> jml: cool, probably well suited
<rockstar> jml, yeah, abentley and I have been trying hard to use unit tests instead of doctests for pagetesting.
<sinzui> gmb, deryck: I would like a mid implementation review for https://bugs.edge.launchpad.net/launchpad-registry/+bug/613610 I have a paste of what I thought would fix the issue. Can either of you look at the bug and the paste in the comment and explain my incompetence.
<_mup_> Bug #613610: +needspackaging bug counts are wrong <bridging-the-gap> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/613610>
<gmb> sinzui, I'll take a look. No promise that I'll actually be any help, though :)
<sinzui> thanks
<gmb> Hrm
<matsubara> bigjools, I fixed the bugeditor script and release 18 foundations bugs
<bigjools> matsubara: can you do launchpad-project !
<bigjools> or I can do it
 * bigjools does it
<matsubara> bigjools, ok, rev 16 is the one you want with the fix
<bigjools> matsubara: yeah I have it running
<matsubara> cool
<gmb> sinzui, Sodomy non sapiens, I'm afraid.
<gmb> It *should* work; I don't understand why it doesn't.
<sinzui> I suspect that the trigger fires while the bug is open
<gmb> sinzui, Yes, that might be the case. Gah. Death to triggers. Might be worth roping stub in to see whether he can shed any light on proceedings.
<sinzui> thanks gmb
<gmb> np; sorry I couldn't be more helpful.
 * jml gone
<EdwinGrubbs> leonardr: ping
<nigelb> Oh, I like the new format for "reported by $foo" :)
<nigelb> for bugs that is.
<leonardr> edwingrubbs, hi
<EdwinGrubbs> leonardr: I just found operation_for_version(). I assume that will let me specify two different @operation_parameters lists, so I can add a parameter without getting the error:  getTimeline" doesn't have the following exported parameters in version "(earliest version)": max_releases.
<leonardr> Edwin: kind of. you will need to have _something_ happen with max_releases in the earliest version, or lazr.restful won't know what value to pass into it
<leonardr> but you can give it a default value or None or something
<EdwinGrubbs> leonardr: do I reference the earliest version as '1.0'?
<leonardr> edwin: you shouldn't need to reference it explicitly, but it is 'beta'
<leonardr> the code at the bottom of the pile of annotations applies to the earliest version
<lifeless> MOIN
<lifeless> that ism good morning
<lifeless> flacoste: morning
<lifeless> flacoste: whats the upload address for staging ?
<flacoste> lifeless: you mean for sftp / poppy?
<lifeless> yeah
<flacoste> li have no idea
<lifeless> ahh :)
<flacoste> not sure if such a service runs there
<lifeless> that may be the missing services then
<lifeless> losa ping
<lifeless> ^
<flacoste> lifeless: they are EOD
<lifeless> flacoste: I know
<lifeless> flacoste: thanks :
<lifeless> )
<flacoste> i see that you have a lot of faith in scrollback :-)
<lifeless> for some things :)
<lifeless> if they don't notice, its not a big deal :)
<lifeless> so hows its looking, 10 hours after release?
<lifeless> flacoste: do you know where the 10.08 release notes are?
<flacoste> lifeless: looks good i think
<flacoste> lifeless: do you mean the roll-out report?
<lifeless> no
<lifeless> I saw that
<flacoste> that's all, we don't produce release notes anymore
<lifeless> I mean mrevell's user facing explanation of whats new
<flacoste> we don't do these anymore
<flacoste> we only publish blog posts
<lifeless> the losa template needs a change then :)
<lifeless> There are separate messages with release notes, so not duplicated here.
<flacoste> lol
<flacoste> yeah
<flacoste> indeed
<flacoste> blog posts will be published tomorrow
<lifeless> ok cool
<lifeless> hmm
<lifeless> it would be nice to do that a little closer to the change
<lifeless> rfwtad will help
<lifeless> danilos: still around ?
<flacoste> lifeless: it usually is, mrevell decided to play it cautiaus this time around
<flacoste> lifeless: i think it was because he was in London and travelling back
<flacoste> wanted to make sure we didn't have to pull stuff off
<lifeless> can other people help with it ?
<lifeless> e.g. put it as a draft in wordpress, allow any editor to tweak-and-publish ?
<flacoste> pushing the buttons to publish them? i guess
<flacoste> i know kfogel used to have access, jml probably also has
<lifeless> mrevell has given me access to blog on the blog
<lifeless> and I would blog about search (yay!) but I'm sure his release blog includes because we were chatting about that
<lifeless> anyhow, minor issue
<lifeless> now, I'm landing webapp-flags
<lifeless> martin fixed all the test fallout and we were waiting for pqm to be open ;)
<lifeless> Launchpad Development Channel | Week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
<thumper> lifeless: it isn't week 1 yet
<lifeless> thumper: it isn't ?
<thumper> lifeless: not until next week
<lifeless> ><
<thumper> :)
<lifeless> if the release is the end of the process? :)
<thumper> we do calendar weeks
<thumper> don't confuse our poor wee brains
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08; I hate mondays -> week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
<ajmitch> so the 'I hate mondays' is just a general statement then?
<lifeless> :P
<mars> Is there some way to tell LP to re-scan a merge proposal diff?  Mine has conflicts that have since been resolved.
<wgrant> It should automatically update within a couple of minutes of the source branch push.
<jelmer> mars: the diff generator was down earlier
<jelmer> rockstar: is that the fire you're fighting perhaps?
<mars> ok.  Thanks guys
<rockstar> jelmer, no, I'm fighting with maverick recipe builds failing.
<thumper> jelmer: no the diff generator is a twisted bug that should be impossible
<thumper> jelmer: we know it exists, but haven't yet been able to work out where
<jelmer> ah, ok
<wgrant> Do imports stack yet?
<mwhudson> wgrant: no
<mwhudson> jelmer: hi, did you get a mail i sent you about importing non-master branches yet?
<mwhudson> -yet
#launchpad-dev 2010-08-13
<jelmer> mwhudson: no - what address did you send it to?
<mwhudson> firstname.lastname@canonical.com
<mwhudson> i think
<mwhudson> maybe i typoed
<mwhudson> mm can't find it now
<jelmer> typos are not uncommon with my surname :)
<thumper> mwhudson: I'm sure jelmer@c.c would get to jelmer as well :)
<mwhudson> Aug 11 14:47:37 grond postfix/local[5352]: 4973011603A: to=<jelmer.vernooij@canonical.com>, relay=local, delay=0.53, delays=0.21/0.26/0/0.07, dsn=5.1.1, status=bounced (unknown user: "jelmer.vernooij")
<mwhudson> jelmer: that's right though isn't it?
<mwhudson> maybe this was when the sender was wrong
<jelmer> mwhudson, yeah
<mwhudson> oh heh, the bounce got rejected with the same message
<jelmer> mwhudson: if it's the sender that is the problem the error message is whacky
<mwhudson> yeah
<mwhudson> jelmer: anyway
<mwhudson> jelmer: what does remain to be done to import non-master branches from git on launchpad?
<jelmer> mwhudson: Basically, everything is there. The one thing that is missing is the UI in Bazaar to parse branch names in URLs
<lifeless> which you just need to rework - push down to transport ?
<lifeless> which will make a small patch too :)
<jelmer> lifeless: yes
<mwhudson> jelmer: ok, that was roughtly what i thought
<mwhudson> thanks
<jelmer> lifeless: and then it needs to be used in the BzrDir as well
<thumper> lifeless: ping
<lifeless> hey
<lifeless> poolie: your webapp patch is landed
<thumper> lifeless: I have some bzr related questions if you have a few minutes
<lifeless> poolie: you might like to send a quick howto to the list or something.
<lifeless> thumper: shoot
<thumper> skype would be easier for me, but just finishing something off
<lifeless> kk
<lifeless> I'm signed in
<lifeless> just connect when you're ready
<poolie> hi lifeless, thanks  for that
<poolie> lifeless: is it live on edge or staging yet?
<poolie> good idea
<lifeless> poolie: nope, just hit devel
<poolie> apparently not live
<lifeless> in ~ 6 hours it will get through buildbot and be deployed to edge at the next edge update
<lifeless> and ~ 14 hours should see it on staging
<lifeless> I mean
<lifeless> from bzrlib import ui
<lifeless> foo(ui)
<thumper> I'm running everything in the launchpadzopelesslayer
 * spiv misread "running" as "ruining"
<lifeless_> thumper: so
<lifeless_> internets fail
<thumper> :)
<thumper> spiv: I'd like to ruin most of them as they are probably doctests
<lifeless_> error-report | subunit-filter -s --no-passthrough | subunit-ls > all-tests.klist
<lifeless_> where error-report is the log file from ec2 (you might need a gunzip -c in there)
<lifeless> then
<lifeless> copy that file
<lifeless> remove all the stuff *after* the failing test
<lifeless> and remove 1/2 of the stuff *before* the failing test
<lifeless> and then
<lifeless> bin/test --load-list the-edited-file
<spiv> thumper: yeah, I thought "he's ruining launchpadzopelesslayer?  fair enough..." ;)
<adeuring> good morning
<mrevell> Hi
<lifeless> hiya
<lifeless> have we have any feedback on the release?
<deryck> Morning, all.
<jml> hey
<jml> I get a stack trace in Launchpad when I mark a bug as a duplicate of itself
<jml> is that because I'm special?
<wgrant> lazr.restful notices that you're in ~launchpad, so it gives you one. lazr-js responds by rendering the whole several hundred lines of it in the widget.
<wgrant> Isn't it great?
<jml> really want my testr branches reviewed.
<jelmer> hi jml
<jml> jelmer, hi
<jelmer> jml: Is this for testr itself?
<jml> jelmer, yes.
<jelmer> jml: I would review, but I'm not a committer...
<jelmer> (yet..)
<jml> jelmer, thanks.
<jml> jelmer, I'm hoping lifeless will make me a committer soon.
<jml> jelmer, actually, some of my branches conflict with each other :)
<al-maisan> jtv: thanks for pointing out the forthcoming bytea array syntax changes in postgres9 :)
<jml> I can't believe we have to run our tests under xvfb
<mars> well, only one suite needs that
<jml> and yet all pay the price for it.
<mars> the price being?
<mars> I didn't think it took that much to start Xvfb
<jml> I didn't say it was a high price :P
<daker> hi
<jml> daker, hi
<daker> i have a proposal for the LP homepage
<daker> http://ubuntuone.com/p/Bg6/
<jml> daker, cool.
<jml> daker, would the big images under "Get more contributors" etc. be links?
<jml> daker, if so, where to?
<daker> yes links
<daker> and the rounded box should icons
<daker> be
<barry> EdwinGrubbs: ping
<jml> on a completely different topic, the importfascist and the warninghandler work together so as to prevent one from getting a useful warning if something warns on import.
<EdwinGrubbs> barry: hi, Barry. do you want to talk about mailman now?
<jml> daker, where would they link to?
<barry> EdwinGrubbs: sure! irc would work for right now.  i'm not set up for skype or mumble atm, but could be with a few minutes work
<daker> jml, i don't konw may be a HELP pages, that shows the users how they can use LP
<jml> daker, fair enough.
<jml> daker, what would be different for people who are already logged in?
<EdwinGrubbs> barry: ok, my first question is: can I look up the email that caused that oops so that I can understand how the decoding error was triggered?
<barry> EdwinGrubbs: the losas will have to help.  the message should be in the shunt queue.  the trick is going to be finding it.  everything in the shunt queue is a python pickle of the message object and the metadata dictionary.  the file names do not encode the message-id
<barry> and i think the shunt file name is not encoded in the oops
<barry> EdwinGrubbs: i'm not even sure the message-id is available in the oops report
<EdwinGrubbs> barry: is that too large for me to ask the losas for the whole thing?
<mrevell> Thanks daker, that's very cool. The box in the centre at the top, I'm not sure what that is.
 * jml is tempted to delete both the warning handler & the import fascist.
<EdwinGrubbs> barry: does it have a date column I could limit it by?
<barry> EdwinGrubbs: probably best would be to ask them to sync the shunted messages from the date of the oops
<barry> yep
<EdwinGrubbs> ok
<barry> EdwinGrubbs: in a built launchpad lib/mailman/bin/dumpdb <messagefile> can be used to dump out the shunt messages
<barry> EdwinGrubbs: it shouldn't be too hard to find the offender from that
<jelmer> jml: it would be nice to have the import fascist as part of 'make lint' but not as something that breaks everything.
<barry> EdwinGrubbs: i'm around all day so please do ping me with any questions
<jml> jelmer, yeah. landscape have an "import guardian" which is nicer in many ways
<jml> jelmer, I once tried to split it out, make it a lazr project & re-use it
<jml> jelmer, but got blocked on the "make a lazr project" part.
<jml> maybe I should have another try now that I know a little more about buildout & friends.
<daker> mrevell, the rounded boxes should take you to a some kind of help, and it should shows how to use the chosen functionality
<mrevell> daker, what about the box at the top of the page that looks a little like an envelope?
<EdwinGrubbs> barry: do you think that I should provide any different handling for a bad rfc822msgid right now, for example just bouncing the email back to the sender? If it should just raise an oops like it does now, I was thinking that the xmlrpc args in the oops should be improved to introspect the xmlrpc.Binary object so that we don't have to download the shunt messages in the future.
<daker> mrevell, ah that's the logo of LP :D
<mrevell> heh, ah right :)
<mars> jml, I can probably do the lazr conversion for you pretty quickly.  I've lightened the lazr spec a bit, it just has not been formalized yet
<barry> EdwinGrubbs: we could possibly be rejecting bad message-ids at the source, either in the mta, or earlier in mailman, but it's also possible this is a shallow bug.
<jml> mars, that'd be great, thanks
<mars> jml, I'm running the qa-tagger through a lazr conversion right now.  We'll see how long it takes.
<jml> mars, although I'd also love to be able to do it quickly for myself.
<jml> mars, I used lazr.newproject or something last time
<jml> and all I got was incomprehensible errors :\
<mars> jml, yep.  That is a bit heavier than what I am doing for new projects now.
<jml> mars, should it be updated to be more like what you do now?
<barry> EdwinGrubbs: i need to reacquaint myself with that code, but i don't have a launchpad build right now ;)  does lp build on maverick these days?  can i just do a rocketfuel-setup still?
<mars> jml, it should, and I plan to.
<jml> mars, cool. :)
<EdwinGrubbs> barry: I think somebody is running launchpad on maverick, but I can't find the email to confirm that it works fine.
<jelmer> EdwinGrubbs: Yes, I am.
<jelmer> EdwinGrubbs: I am running with a backported version of python-psycopg2 though
<jelmer> s/backported/older/
<barry> i can bring up a lucid vm if necessary
<EdwinGrubbs> barry: jelmer says it works for him.
<barry> jelmer: cool.  i'll give it a shot
<barry> EdwinGrubbs: cool.  i'll move upstairs in a bit and give it a try
<jelmer> barry: you'll need the python-psycopg2 from lucid as 2.2 (which is in maverick) is a lot more pedantic and breaks lp
<EdwinGrubbs> barry: will the losas know what to do if I just ask for "shunt messages" between specific dates, or do I need to point them at some docs?
<barry> jelmer: k, thx
<EdwinGrubbs> barry: or does that dumpdb thing take a date arg?
<barry> EdwinGrubbs: queue/shunt should be enough i think
<barry> EdwinGrubbs: no, it only takes a file name
<barry> EdwinGrubbs: but the date is encoded in the file name (along with a sha1 hash of the file contents, but you don't care about that :)
<daker> mrevell, i should be able to make it real
<mrevell> daker, I'd love to see more. We're not going to change the home page right now but we plan to in the next few months. I'm about to start researching what LP users want from the home page, so that we have a basis for any changes we make.
<daker> brb
<barry> EdwinGrubbs: brb
<jtv> al-maisan: it sure gave me enough trouble for me to notice itâ¦  I still run daily tests of e.g. postgres 9.0 using libpq 7.3
<daker> mrevell, good
<al-maisan> jtv: ahm
<al-maisan> s/ahm/ah, I see :)/
<EdwinGrubbs> barry: mbarnett doesn't know what the shunt messages are? Do you know what server or directory they are in?
<deryck> gmb, hey.  So we don't currently have plans among the work we divided up between you and Abel for the "view all my subscriptions page," right?
<deryck> gmb, I chatted with bdmurray about taking this on, which is why I ask.
<gmb> deryck, Nope no plans (I responded to your ping last night but you'd already left)
<gmb> deryck, It would be great if someone could tackle it this cycle.
<deryck> gmb, yeah, no worries.  I loose track of the TZ sometimes when I ping. :-)
<gmb> deryck, Maybe I should make my proxy respond with the local time when I've disconnected :)
<deryck> gmb, bdmurray is definitely interested, so I suggested he start the first part of it -- hooking up a view, displaying what we can for now -- and then when he's blocked waiting on your work and Abel's, then he can do individual bugs again.
<gmb> deryck, That sounds like an excellent idea.
<deryck> cool
<mrevell> Hey, I have a failing test that I don't know how to fix. The failure message doesn't look like something I've seen before. It is:
<mrevell> FAILURE: canonical.launchpad.webapp.tests.test_login.TestOpenIDReplayAttack.test_replay_attacks_do_not_succeed (subunit.RemotedTestCase)
<mrevell> Anyone able to lend a hand?
<benji> mrevell: I can take a look at it in a few minutes.
<mrevell> Thanks benji :)
<bac> hi mrevell
<mrevell> hey there bac
<bac> mrevell:  where did you see that failure?  i just ran that test in isolation and it passes
<mrevell> bac, On my branch: https://code.edge.launchpad.net/~matthew.revell/launchpad/top-right-name-bug134957
<bac> mrevell:  on ec2?
<mrevell> bac, yeah
<bac> mrevell:  did you consider displaying both the name and displayname?
<bac> e.g. 'Brad Crittenden (bac)'?
<mrevell> bac, Yeah, but decided that it could get very long ... I'd Matthew Revell (matthew.revell) for example. Would you particularly miss seeing the displayname?
<bac> dunno.
<bac> mrevell:  i think that test failure is valid
<bac> mrevell:  It includes self.assertIn('Sample Person', login_status)
<benji> mrevell: ok I'm available now, still looking for an extra pair of eyes?
<bac> mrevell:  so i think you need to update the test for your new expected output
<bac> mrevell:  have you tried running it locally?
<bac> bin/test -vvt TestOpenIDReplayAttack
<mrevell> Ahhhh, I see, that's the name of the test? Right. Thanks bac, that's what I was missing. I didn't know which test was failing.
<mrevell> Thanks benji but I think bac has helped me.
<benji> cool
<jml> is there any particular reason for ec2 test to go out of its way to nicely format the html that is served from instance?
<jml> g'night all
<jelmer> have a good weekend jml
<EdwinGrubbs> rockstar: ping
<EdwinGrubbs> matsubara-afk: ping
<rockstar> EdwinGrubbs, pong
<EdwinGrubbs> rockstar: I was wondering if you were familiar with the job system and whether we could have a pre-impl call on that.
<rockstar> EdwinGrubbs, sure, lemme fire up skype.
<rockstar> EdwinGrubbs, I don't see you on skype.
<EdwinGrubbs> rockstar: it doesn't seem to want to connect.
<rockstar> EdwinGrubbs, :(
<EdwinGrubbs> rockstar: I haven't used it in a while since I just use mumble.
<rockstar> EdwinGrubbs, yeah, my headset that liked mumble has given up the ghost.
<lifeless> hi EdwinGrubbs
<EdwinGrubbs> salgado: ping
<salgado> hi EdwinGrubbs
<EdwinGrubbs> salgado: do you know if it is possible to include more information in oopses than just adding individual lines to the Request Variables section?
<lifeless> EdwinGrubbs: it is
<salgado> EdwinGrubbs, not really, but Ursinha or matsubara-afk should know
<lifeless> EdwinGrubbs: we may need to tweak the api - but thats easy
<Ursinha> EdwinGrubbs, matsubara-afk knows it best
<EdwinGrubbs> oh, so you pass the buck to the guy who isn't around. I see out it works.
<lifeless> EdwinGrubbs: what do you want to record?
<EdwinGrubbs> lifeless: the email that mailman sends over via xmlrpc. Currently, the losas have to go look for a files in one directory based on the unix timestamp that is in the filename. The files themselves are pickles, so searching it isn't fun.
<lifeless> I'm not familiar with the mailman integration
<EdwinGrubbs> lifeless: I've found the part of the code where stuff gets added to the oops report, but I don't know if I'll blow up the oops.py.
<lifeless> for my curiousity, could you give me a little extra context?
<EdwinGrubbs> here's the bug with the oops: https://bugs.edge.launchpad.net/launchpad-registry/+bug/615655
<_mup_> Bug #615655: UnicodeDecodeError in xmlrpc holdMessage <mailing-lists> <oops> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/615655>
<EdwinGrubbs> lifeless: as you'll see in the oops, it just tells you that the xmlrpc args contains an <xmlrpclib.Binary> object. not very helpful.
<lifeless> so, check oops-tools
<lifeless> to see if it will go boom
<lifeless> it shouldn't, but you'll need to tweak it anyhow to show new data
<lifeless> unless you put it in req_vars or db-statements which would be a bit of a hack
<lifeless> ok, so mailmain makes an xmlrpc call, giving us the email for <some reason> ?
<EdwinGrubbs> I'll look at that.
<lifeless> so, I'd like to have a free form 'extra detail' section
<lifeless> oops-tools needs to change to show it in the lp-oops web service
<lifeless> and we need an api to add stuff to it
<lifeless> if we had that, would it be sufficient for your needs?
<EdwinGrubbs> right, it looks like there is a bug open to make the oopsMessage() context method do that instead of sticking things in the req vars.
<EdwinGrubbs> a free form extra detail area would definitely work for including email messages.
<lifeless> \o/
<lifeless> so why does mailman xmlrpc call us ?
<EdwinGrubbs> lifeless: for moderating emails, which is done in a Launchpad page.
<lifeless> thanks
<lifeless> EdwinGrubbs: are you planning on doing the extension work on oops to make this happen ?
<EdwinGrubbs> lifeless: well, assuming that matsubara-afk doesn't tell me any reason it shouldn't be done on Monday. I definitely want to do it, and it doesn't look like it could possibly take more than a day of work on my side.
<lifeless> \o/
#launchpad-dev 2010-08-14
<wgrant> Yay for having IS all in the same sleeping timezone.
<MTecknology> wgrant: Hi!
<cody-somerville> It looks like borium is lost and the abort is failing with Fault 8002: 'error' when xmlrpclib tries to cleanup after parsing the response.
<wgrant> cody-somerville: Hm, and that kills everything?
<wgrant> It shouldn't.
<cody-somerville> appears so as the log keeps saying the same thing over and over and over
<cody-somerville> *bohrium
<cody-somerville> in fact, it appears caught in a loop trying to do this
<wgrant> Ahh.
<wgrant> Can you paste a full loop?
<cody-somerville> wgrant, http://pastebin.ubuntu.com/477761/
<wgrant> cody-somerville: Hmm, and it's doing that constantly, with no other log entries? How often?
<wgrant> We probably just have to disable bohrium to get things running again, but there's nobody around who can do that :/
<cody-somerville> wgrant, yes
<cody-somerville> wgrant, multiple times per second
<wgrant> Aha.
<wgrant> So, yes, disabling it will fix it.
<wgrant> StevenK: ^^?
<wgrant> Interesting that it keeps retrying that one, though...
<cody-somerville> probably has something to do with twisted
<wgrant> Oh yes, it's all a nice Twisted mess.
<wgrant> You know...
<wgrant> It wouldn't surprise me if it was aborting the transaction because of the failed scan.
<wgrant> So it sets to the builder as not-OK, then aborts before it commits it.
<cody-somerville> lol
<cody-somerville> Wouldn't surprise me either
<cody-somerville> Soyuz has a habit of making mistakes like that
<wgrant> Hm, no, it should be committing.
<wgrant> This is, of course, brand new code :/
<wgrant> Oh, damn, it's in requestAbort instead.
<wgrant> Ah, no, but the whole thing is wrapped.
<wgrant> So it does commit immediately afterwards.
<lifeless_> wgrant: we can always escalate
<lifeless_> whats up
<cody-somerville> lifeless, buildd-manager is hung up
<lifeless> whats the impat
<lifeless> impact
<cody-somerville> lifeless, nothing is getting built
<cody-somerville> lifeless, not  for the Ubuntu archive or for any PPAs
<lifeless> will it recover on its own?
<cody-somerville> It doesn't appear so.
<cody-somerville> lifeless, the log is filled with this: http://pastebin.ubuntu.com/477761/
<cody-somerville> Might be able to fix it by disabling the bohrium builder (which a buildd admin can do) but no guarantee.
<lifeless> ok
<lifeless> its only bohrium showing like that ?
<cody-somerville> looks that way, yea
<lifeless> ok
<lifeless> have you considered escalating to IS ?
<wgrant> IS should be almost awake now...
<lifeless> 9am for those that are sprinting
<lifeless> which isn't everyone
<lifeless> (AFAIK)
<lifeless> cody-somerville: ^
<cody-somerville> I considered it, yes. Probably should have but haven't since I didn't have a pressing reason to do so personally.
<cody-somerville> plus I'm tired of writing incident reports for Launchpad downtime :P
<lifeless> cody-somerville: heh
<lifeless> so I think we should escalate
<lifeless> because otherwise its going to stay down all weekend
<cody-somerville> lifeless, agreed
<wgrant> Right.
<wgrant> It *probably* just needs a buildd admin to disable bohrium. But it may be more broken than that...
<lifeless> wgrant: cody-somerville: its being looked at
<wgrant> lifeless: Thanks.
<lifeless> wgrant: can you file a bug please
<lifeless> wgrant: the builder row was deadlocked
<wgrant> Builder row?
<wgrant> Wait, in the DB?
<lifeless> yes
<wgrant> Wow.
<wgrant> I've not seen that before.
<lifeless> wgrant: so, airlock was apparently doing something to the builder
<lifeless> and hung waiting on a lock
<elmo> whee
<lifeless> so lp then was timing out trying to disable the builder
<elmo> it's broken again
<lifeless> elmo: have we bounced the builddmanager?
<wgrant> Airlock?
<lifeless> wgrant: the thing that steals buildds and gives them back
<wgrant> Ah.
<lifeless> it predates API's and writes to the DB
<elmo> lifeless: yes; I'm going to try the update SQL, if that's locked, face stab the buildd-manager and try again
<lifeless> is there an API to disable a builder and enable it again ?
<elmo> update SQL to get the fuck rid of bohrium
<wgrant> lifeless: Not at the moment.
<lifeless> wgrant: if you were to make one, it would help with this
<wgrant> I've considered it. It's not hard.
<lifeless> because we have timeouts set in the webapp ;)
<wgrant> But we've not run into this contention before.
<lifeless> wgrant: -please-
<elmo> ok, so I can't run the SQL again
<wgrant> buildd-manager's transaction usage changed massively a couple of days ago. I'd suspect there's something a little wrong with it.
<elmo> I think it's because b-m is in a tight loop failing on bohrium
<lifeless> elmo seemed to think its occured before but perhaps not as violently
<lifeless> elmo: yeah. Take the b-m down as gracefully as possible.
<elmo> haha, gracefully
<elmo> the init script tries TERM which always fails
<elmo> then it KILLs
<wgrant> TERM normally works.
<wgrant> It can take a few seconds, though.
<elmo> 'always' may be slightly hyperbolic; but I haven't seen TERM work for me since the latest round of implosions started happening
<wgrant> Ew.
<wgrant> Anyway, I also don't see how a DB deadlock could result in this loop.. unless the commit is failing, and this isn't logged?
<lifeless> oh
<elmo> ok, bohrium disabled; b-m back up
<lifeless> so we found an interesting xmlrpc thing the other day
<lifeless> returning a Fault -> doesn't abort transactions
<lifeless> raising one does.
<lifeless> probably not the thing here, but a good thing to remember until we fix it
<lifeless> wgrant: in general don't we structure things so that 'unhandled exception -> rollback' ?
<wgrant> lifeless: Yes. But the code here catches the Fault, disables the builder, then commits.
<elmo> I have to go and pack, but I'll leave my laptop up as late as I can and keep an eye on the b-m log
<lifeless> wgrant: given that for the last 90 minutes there was a db backend waiting for a lock
<lifeless> wgrant: I highly doubt that its working as advertised
<wgrant> lifeless: The codepath is really short and clear.
<wgrant> Anyway, dinner.
<lifeless> elmo: thanks heaps
<lifeless> night all
<lifeless> grah rosetta is unhappy
<lifeless> hmm, time for incident report about lsat nights soyuuz thing
<jelmer> lifeless, there was another incident, or is this the EINTR one?
<lifeless> jelmer: there was another one
<lifeless> IncidentReports/2010-08-14-Soyuz-Airlock-Deadlock
<lifeless> jelmer: ^
<jelmer> thanks, reading
<lifeless> jkakar: https://bugs.edge.launchpad.net/storm/+bug/617973 btw
<_mup_> Bug #617973: timeouterror could be more clear about the implications <Storm:New> <https://launchpad.net/bugs/617973>
<lifeless> bbiab
<lifeless> jml: https://devpad.canonical.com/~jml/lp-doc/index.html might be better as wiki pages
#launchpad-dev 2010-08-15
<jkakar> lifeless: Branch attached, please review.
<lifeless> jkakar: oh hai
<lifeless> jkakar: thanks
<lifeless> jkakar: if you're still around, theres another thing that might be useful :)
<lifeless> jkakar: would like you're input on https://bugs.edge.launchpad.net/launchpad-foundations/+bug/618019
<_mup_> Bug #618019: OOPS may be underrepresenting storm/sql time <Launchpad Foundations:New> <https://launchpad.net/bugs/618019>
<jml> lifeless, I partly disagree, but you're welcome to explain why some other time.
 * jml off to bed.
<lifeless> jml: bed! wow, little late :)
<lifeless> moin moin
<lifeless> hmm
<lifeless> where to file 'sprint' bugs ?
<jelmer> lifeless: the same place as blueprint bugs, though I can't remember where that was...
<jelmer> lifeless: 'blueprint'
<lifeless> thanks
<lifeless> I'
<lifeless> ll go back and find it in a minute
<lifeless> can LP API's be used in GAE ?
<jelmer> I can't think of an obvious reason why they shouldn't be, but I don't know.
<lifeless> jelmer: https://bugs.edge.launchpad.net/soyuz/+bugs?field.tag=timeout
<lifeless> :)
<jelmer> that's quite a few..
<lifeless> I've been filing the next set - timeouts by % of request rather than volume per day
<thumper> lifeless: I'm looking at that binary chop thingy
<lifeless> \o/
<lifeless> it will be a bit fiddly I suspect :(
<thumper> if I just run the LaunchpadZopelessLayer tests I get the failure
<lifeless> if its any consolation its trivial in bzr :)
<lifeless> thumper: \o/ ok thats a good starting point
<thumper> so instead of starting with everything...
<lifeless> yeah
<lifeless> do the ls
<thumper> so how do I run the test suite with the right output?
<lifeless> it will include the layer setup and teardowns
<thumper> my default test run isn't subunit
<lifeless> so you can use that to shrink things
<lifeless> --subunit
<lifeless> cat .testr.conf - may give you some hints :)
<thumper> running now
<thumper> takes about 40 - 45 mintues
<thumper> also I my first chop is going to take most of the tests out, as I think it is an interaction between the code import tests and the translation tests
 * thumper goes to make first coffee
<lifeless> jelmer: wth does Distribution:+archivemirrors do that it ever times out :)
<wgrant> It might be the freshness queries.
<wgrant> In fact, that's almost definitely the case.
<lifeless> ...
<wgrant> It needs to aggregate freshness data for all series into a single value for each mirror.
<lifeless> in the webapp transaction ?
<wgrant> IIRC.
<lifeless> we only have what 10 series
<wgrant> * lots of archs and pockets
<jpds> It's most likely the freshness stuff.
<wgrant> And I bet the queries aren't efficient...
<lifeless> ok
<lifeless> well, if you want a ++oops++ on it, shout
<lifeless> wgrant: distroseries:+builds is still pain
<al-maisan>  /msg nickserv identify  OafNem1Glewg:Ter
<wgrant> lifeless: :(
<lifeless> al-maisan: time to change your password
<al-maisan> yack
<lifeless> wgrant: nearly flat spread between 0 and 5 seconds
<lifeless> sorry
<lifeless> 0 and 7 seconds
<wgrant> Hm :/
<lifeless> and then a 9% spike @  15 seconds
<al-maisan> lifeless: thanks for telling me anyway :P
<lifeless> al-maisan: might have been a failed script
<lifeless> al-maisan: wanted to be sure you knew :)
<lifeless> wgrant: https://bugs.edge.launchpad.net/soyuz/+bug/618405
<_mup_> Bug #618405: DistroSeries:+builds timing out in ~9% of requests by edge timeout <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/618405>
<lifeless> jelmer: you might like lp:python-fixtures
<lifeless> I'd value a second opinion on bug 618019
<_mup_> Bug #618019: OOPS may be underrepresenting storm/sql time <Launchpad Foundations:New> <https://launchpad.net/bugs/618019>
<jelmer> lifeless: looks interesting
<jelmer> lifeless, where do you find the time to think of, implement and release a new project each weekend ? :-)
<mwhudson> morning
<lifeless> jelmer: that one has been percolating for ~ a year, I think.
<lifeless> hi mwhudson
<jelmer> 'morning mwhudson
<jelmer> lifeless, heh, ok
<lifeless> jelmer: I just hadn't written any code or anything
<lifeless> it simply needed actually getting a sunday with no tasks, no chores, no crises, no interrupts.
<lifeless> and voila
<jelmer> lifeless: did your recent experiences with launchpad layers have anything to do with it?
<lifeless> no
<lifeless> testresources has some api tensions
<lifeless> which I've talked with poolie and jml extensively about
<lifeless> in the 'wish it was better' sense :)
<lifeless> I concluded (dunno when ;P) that it was because it was doing too much
<lifeless> it should be a manager for an existing concept
<lifeless> rather than a whole concept + manager
<lifeless> so this project is targeted at that core concept
<jelmer> ah, ok
<jelmer> anyway, it looks interesting, I've put it on my list of stuff to look at :-)
<lifeless> :)
<poolie> hi there
<jelmer> 'morning poolie
 * mwhudson finds http://www.kernel.org/pub/software/scm/git/docs/git-notes.html
<mwhudson> jelmer: something for bzr-git roundtripping?
#launchpad-dev 2011-08-08
<wallyworld_> wgrant: some already filed a bug http://youtrack.jetbrains.net/issue/PY-2741
<wallyworld_> someone
<wallyworld_> actually, it seems like it pertains to code completion but it will be related
<lifeless> wallyworld_: this is the whole 'our packaging can only support one version at a time' issue.
<wallyworld_> lifeless: thanks. btw, a gentle reminder about https://code.launchpad.net/~xaav/loggerhead/export-tarball/+merge/66408 :-)
<lifeless> thanks ;P
<wallyworld_> an adaptor question - i think we should be able to specify a class as being an adaptor for more than one interface, wither via @adaptor(foo, bar) or in zcml for="foo bar"? it appears not since it doesn't work. and so i have to have one zcml adaptor element per interface, which the same adapter class/factory specfified. is there something i'm missing?
<wgrant> wallyworld_: I'm not sure... but it's probably not an adapter for more than one interface.
<wgrant> Those interfaces probably share a subinterface.
<wallyworld_> wgrant: in this case it is the same adapter implementation for more than one interface that i would like to specify. they do share a subinterface (in fact > 1) and the adaptor makes use of those >1 subinterfaces. you'd think it would be possible
<wallyworld_> in this case, i want to say "this specified class adapts both IDistribution and IProduct"
<wallyworld_> i can do it but need to have multiple adapter declarations with just a different for="xxx" which seems dumb
<wallyworld_> and i can't user @adapter() so am forced to use zcml
<wgrant> StevenK: QA!
<StevenK> wgrant: Yeah yeah :-P
<StevenK> lifeless: Can haz bazaar-experts team deleted?
<lifeless> StevenK: we've deployed everywhere since?
<lifeless> StevenK: including crannies like soyuz ?
<StevenK> I know cocobanana has been deployed to since, and one of germanium's trees is in NDT. Not sure about the other tree
<wgrant> poppy doesn't use bazaar-experts, so meh.
<StevenK> I'd be a little concerned if it did ...
<nigelb> Good morning
<wgrant> lifeless: Do you think the timeouts are regressions, or bloat?
<wgrant> We went from 300 to 1100 in a week.
<StevenK> Wheee
<wgrant> I suspect the subscription changes.
<wgrant> But that doesn't explain Person:EntryResource:searchTasks
<wgrant> I don't think.
<lifeless> wgrant: the soft counts last week would have been large if it was creeping bloatism
<StevenK> lifeless: So, can haz bazaar-experts deleted, or do you want to wait until every single tree doesn't mention the celebrity?
<lifeless> I'd like to wait
<lifeless> easy to do, reduces risk, doesn't slow anything down.
<StevenK> lifeless: Until when? :-)
<StevenK> If you say the heat death of the universe, I'm booking a flight to NZ :-P
<lifeless> no live trees with the celeb in them
<StevenK> The only services I can think of are germanium's poppy and probably forster
<StevenK> Oh, and the librarian?
<lifeless> likely, yes.
<lifeless> wallyworld_: hi
<wallyworld_> hello
<lifeless> wallyworld_: your affiliation work; it looks like its going to do lots of eager loading.
<wallyworld_> i put in tests for the query count (for bugtask) - it's only 4 quesies soince stuff is already loaded
<lifeless> per person
<wallyworld_> yes, that is true
<lifeless> s/eager loading/lazy loading/
<wallyworld_> so i probably need to fix that
<lifeless> wallyworld_: anytime you end up writing adapters, and object orientated code that does data access, its probably going to involve late evaluation/lazy loading problems.
<lifeless> wallyworld_: the -only- case I know where such code is fine is when it happens /once/ on a page/http request as part of establishing the context (rather than potentially in inner loops etc)
<wallyworld_> i'm starting to see that. i still haven't got it hard wired in my brain that zope object navigation = queries
<lifeless> also inTeam() for instance.
<lifeless> userA.inTeam(teamT)
<wallyworld_> i'm used to popo/pojo type domain models
<lifeless> userB.inTeam(teamT)
<lifeless> will do 2 queries, even if userA, userB and teamT are all already loaded.
<wallyworld_> to access the join table?
<lifeless> yup
<wallyworld_> ah, well it's behind a feature flag at least. i'll do a branch to optimise. that's for pointing it out
<wallyworld_> except i think we turn on the feature flag recently, must check
<wgrant> picker_enhancements is on.
<wallyworld_> lifeless: the issue with the implementation is that the call to the affiliation adapter is done inside getPickerEntry(), which itself is an adapter method which is done per picker row
<wallyworld_> so there's potentially a bit of work to do to change all that
<wallyworld_> still, must be done
<wallyworld_> not quite so straightforward, since each item in a batch might not use the same picker entry adaptor
<lifeless> right
<lifeless> It's common to run into friction like this
<wallyworld_> yeah, so now picker entry adapters and HugeVocabularyJSONView are getting a bit of rework
<wallyworld_> lifeless: the current affiliation adaptor implmentation may not be so bad. the inTeam checks will be cached and the 4 queries i mentioned before are not per person i don't believe - eg once the pillar.bug_supervisor  is fetched, it is not required to hit the db again for each person to check
<wallyworld_> actually, the inteam caching assertion is bogus
<lifeless> the current caching implementation is (team, member) specific
<lifeless> what you need is an interface for populating many such cache tuples at once
<wallyworld_> there's lots of other places where we do inTeam(xxx) or inTeam(yyy) eg canBeUnsubscribedBy()
<lifeless> and to call that with all the teams, for all the rows, and all the row person-or-team objects at once.
<wallyworld_> that is a solution but it severely breaks encapulation of the adapter implementation
<lifeless> yes
<lifeless> such encapasulation is harmful in our environment TBH
<wallyworld_> it may be easier to load the team members inside the adapter implementatiin and cache those
<wallyworld_> encapsulation is preserved and cachiing can still occur
<lifeless> is the adapter per row ?
<wallyworld_> the adapter is per row but the teams are static for a given adaper instance
<lifeless> to solve the problem you need to have set based interfaces.
<lifeless> anything that is object based or row based will have late evaluation symptoms
<wallyworld_> or load the team members the adapter knows it needs up front
<lifeless> no
<lifeless> that won't work because the adapter is per row
<wallyworld_> it will work - each row is checking for emebrship in  fixed set of teams, known in advance
<lifeless> uhm
<wallyworld_> so if those team members are loaded and cached, no extra db access is needed
<wgrant> Bug #822500
<_mup_> Bug #822500: select <Launchpad itself:New> < https://launchpad.net/bugs/822500 >
<lifeless> wallyworld_: This doesn't fit with my understanding of the code.
<StevenK> Nice title
<lifeless> wallyworld_: calling personA.inTeam(teamT) will not seed the cache for calling personB.inTeam(teamT)
<StevenK> In fact, I found something odd last night -- https://code.launchpad.net/~aaacavazos/+recipe/launchpad-daily-1
<wallyworld_> lifeless: yes, that's why i am saying we need to load and cache these relationships
<wallyworld_> do it once up front for the known, fixed teams
<lifeless> wallyworld_: which requires knowing all the personX's and all the teamY's, doesn't it ?
<StevenK> And there are seven recipes for stable ...
<wallyworld_> lifeless: a person (row) is affiliated with the context (eg bug task) if they are a member of (eg) the security contacts team or the bug supervisors team
<wallyworld_> so we can load and cache the team membership of those 2 teams
<wallyworld_> then as each row is checked, no extra db acces sis needed
<wallyworld_> i guess that doesn't account for indirect membership
<wallyworld_> but it could
<lifeless> wallyworld_: this seems both more work, and more code, than accumulating the set of (person, team) pairs we need an doing one call to check them all.
<wallyworld_> but it preserves encapulation and the inheritance model used but the adapter implenmentation
<lifeless> by more work I mean more runtime work.
<lifeless> You have to be sure what you're encapsulating; if you're encapsulting the wrong things its a pessimisation.
<wallyworld_> hmm. not sure. maybe.
<wallyworld_> what's being encapulated is the method to find out a person's affiliation with a given context
<lifeless> yes, (I've read the patches)
<lifeless> this is *exactly* the sort of thing I've had to rip out to fix performance issues up and down the LP stack. bugs. soyuz. questions. blueprints.
<wallyworld_> and as dicussed, it hooks into the existing workflow of one call per picker row
<lifeless> productseries has its own driver
<lifeless> a product with 15 series will have 15 series teams to check.
<wallyworld_> i don't think that sort of check is done currently
<lifeless> (if you honour them all)
<wallyworld_> if the context is a product, then series doesn't come into it
<lifeless> anyhow, I can see this isn't making sense to you
<lifeless> so we can drop it and be evidence based.
<wallyworld_> it is, i'm just not quite agreeing with the suggested approach (yet)
<lifeless> All I can say, is that everything I've seen over the last year predisposes me to consider *any* per-single-object interface flawed by design.
<wallyworld_> sure, but there's a balance between that and adhering to good oo design principals
<lifeless> not for code involved in batches/iteration/queries
<wallyworld_> yes, i can see the merit in that statement
<lifeless> (oh and for loops, which is implicit in iteration)
<lifeless> OO design hasn't scaled to non-main-memory datasets IMO
<wallyworld_> there's two ways of solving it - use set based interfaces or provide a caching implementation that prevents single calls from hitting the db
<lifeless> I agree there are two approaches, but the latter isn't one of the two - it doesn't work.
<wallyworld_> ie for a given use case, load the domain model known to be required to satisfy the request
<lifeless> (the second approach I'd layout is the object query language approach)
<wallyworld_> i've used that in combination with my second approach above
<lifeless> there are two disjoint  problems with caching implementations
<lifeless> if you avoid one you get the other
<lifeless> the first problem is bog standard late evaluation
<lifeless> even if you cache each single call, when you iterate you make a new call for each person, and thats a cache miss
<wallyworld_> but if the domain model that is being reference for each call is the same...
<lifeless> the answer to that is (broadly) readahead: when you get two similar lookups your cache can decide to load everything related
<lifeless> wallyworld_: its not in your case, but we'll come back to that :)
<lifeless> the problem with readahead is that it blows out in size
<lifeless> e.g. readahead in the inTeam case would be to select person from teamparticipation where team=self.id, in inTeam
<wallyworld_> readahead is suboptimal - i normally don't like tyo use it except ib very narrow, specific circumstances
<lifeless> rather than looking up single members.
<lifeless> for LP, we have some very big teams:
<lifeless> select count(*) from teamparticipation group by team order by count(*) desc limit 20;
<lifeless>  count
<lifeless> -------
<lifeless>  18490
<lifeless> (dropping to 855 at the 20th)
<lifeless> but still, those big teams, in a naive implementation, will be a bit of a headache.
<lifeless> (for instance, loading 18K rows from disk, if we had a cold cache situation, is ~ 40 seconds on its own)
<wallyworld_> what's that large team called?
<lifeless> that one is locoteams
<lifeless>  18490 | 2968414 | locoteams
<lifeless>  11026 | 2979678 | locoteams-approved
<lifeless>   3842 | 2388044 | launchpad-users
<lifeless>   2936 | 1121852 | ubuntu-us
<lifeless> wallyworld_: but I only brought that up to illustrate the solution-is-worse-than-the-problem nature of using on demand caching strategies in this sort of problem
<wallyworld_> given the size of the largest teams, i can see the issue.
<wallyworld_> although i do wonder whether such team would ever be a product owner or bug supervisor etc
<wallyworld_> i had the feeling such attributes would be individuals or small teams
<lifeless> for loco team issue tracking, sure ;)
<wallyworld_> fair enough. one of my underlying assumptions was that the team sizes involved in affiliation would be smallish
<wallyworld_> thus allowing other interfaces than a set based one
<lifeless> ubuntu-bugs has 416 people in it
<lifeless> openstack has 1300
<lifeless> ubuntu-marketing 1400
<wallyworld_> that's not too bad - caching 1300 id tuples
<wallyworld_> but anyway, that seems like suboptimal approach now
<wallyworld_> it also means that the adaptor based approach needs to be rethought i suspect
<wallyworld_> actually, maybe not
<lifeless> if you expose the things that need looking up
<wallyworld_> just the getAffiliation interface
<lifeless> a single call can seed the caches of the various teams with the results for all the queries that may be made
<lifeless> this is a set based interface
<lifeless> set based doesn't mean 'no abstraction' :)
<wgrant> I think we probably need to do away with adapters at some point.
<wgrant> It is always tempting to use them because they are nice and we use them already, but they make it awkward to setify.
<wallyworld_> i think i just need to change getAffiliationBadge(self, person) to getAffiliationBadges(self, people)
<wgrant> Oh, if the API is like that, that's not too bad.
<wallyworld_> or something like that, plus all the upstream changes for hooking it into the vocab
<wgrant> As long as there aren't many affiliation contexts.
<wallyworld_> well, atm, each batch of picker results is for the same context, so there's only one
<wallyworld_> lifeless: i think the TeamParticipation table is good for indirect memberships too, right?
<wgrant> Yes.
<wgrant> TeamParticipation includes indirect memberships.
<wallyworld_> right. makes it a bit easier.
<lifeless> wallyworld_: yes, I would encourage any implementation to start with refactoring inTeam() to have a single multi-lookup method it goes through.
<wallyworld_> lifeless: agreed. inTeam will likely not be used now
<lifeless> wallyworld_: cool; I suggest a single method so that we don't have duplication
<wallyworld_> lifeless: depending on how many usages of inTeam there are, I may optimise the affiliation stuff separately
<wallyworld_> i want to get that done asap without the baggage associated with an inTeam refactoring
<wallyworld_> across the entire product
<lifeless> wallyworld_: you shouldn't need to touch other callers of inTeam
<lifeless> they are either fine, or separate problems.
<lifeless> I'm just talking 'extract Method'
<wgrant> And leave an inTeam wrapper around the new method, for compatibility.
<wallyworld_> lifeless: oh, i see what you are saying i think. as if i would cut'n'paste code everywhere :-/
<lifeless> wgrant: compat/convenience
<lifeless> wgrant: all the searchTasks timeouts are on one person
<wgrant> Aha.
<wgrant> I didn't look that far.
<lifeless> could well be patching
<lifeless>     276 https://api.launchpad.net/1.0/%7Eubuntu-security-sponsors (Person:EntryResource:searchTasks)
<lifeless>        OOPS-2045A18, OOPS-2045A33, OOPS-2045A34, OOPS-2045A39, OOPS-2045A52
<lifeless> 276 SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_close ...  = FALSE)) AS BugTask JOIN Bug ON BugTask.bug = Bug.id ORDER BY BugTask.id LIMIT $INT OFFSET $INT:
<lifeless> I suspect its the high offset issue
<wgrant> So adeuring will save the world.
* henninge changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 247 - 0:[#######=]:256
<bigjools> morning
<adeuring> good morning
<mrevell> Hi
<bigjools> wgrant or StevenK, care to talk about bug 820941 for a moment?
<_mup_> Bug #820941: Distro uploads always email XXXX_derivatives@packages.qa.debian.org <derivation> <Launchpad itself:Triaged> < https://launchpad.net/bugs/820941 >
<wgrant> bigjools: New field on distribution to provide a template?
<bigjools> yeah that's my first reaction too
<bigjools> well, since it's not going to change it can be fixed, we just need to decide whether a distro is going to email or not
<wgrant> Nobody is ever going to want to email anybody except Debian?
<bigjools> there's currently zero requirement to do that
<bigjools> and I don't want to fix nonexistent bugs
<wgrant> We can almost freely remove a hard-coded Debian-specific string.
<bigjools> it should move to config
<wgrant> Why?
<wgrant> It's distribution-specific.
<bigjools> no, it's Debian specific
<wgrant> And not everybody wants to send stuff to Debian.
<bigjools> there is no requirement to do this for any other LP-hosted distro
<wgrant> They may want to send it elsewhere.
<bigjools> "may"
<wgrant> We need either add a bool or a string field.
<bigjools> but they don't, right now.  I am not interested in fixing nonexistent bugs
<wgrant> If we add a bool, we get to keep the hardcodedness for another 5 years.
<wgrant> If we add a string field instead, we get an extra line of code and remove the hardcoded string.
<wgrant> The overhead of making it customisable is probably somewhere around two minutes.
<bigjools> loving your optimism
<bigjools> you're usually pessimistic ;)(
<bigjools> where's lifeless's wiki page on the DB change process?
<wgrant> At the moment, don't.
<lifeless> the usual place
<wgrant> There's https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess
<wgrant> But it will probably be at least two weeks before we can do it.
<bigjools> lifeless: that's not helpful :)
<lifeless> bigjools: :P I was looking fo rit
<bigjools> PolicyAndProcess isn't linked from the front page
<bigjools> lifeless: are we blocked on DB changes until it's implemeted?
<lifeless> bigjools: fastdowntime? not blocked no. *If* we reach the end of the monthly cycle without being ready to use it, we can manually schedule a 90 minute legacy downtime.
<bigjools> lifeless: ok thanks
<jml> lifeless: so that announcement about something being the last 90 minute downtime was optimistic?
<lifeless> jml: slightly
<lifeless> jml: its certainly the last as-regular-schedule
<lifeless> jml: I would consider us having to do another as a big deal.
<bigjools> stub: where do you keep your list of db patch numbers?  I need one and I can't remember where it lives. :(
<lifeless> bigjools: its linked on that wiki page
<jml> lifeless: ok. fair enough. I sort of thought from skimming it that fast db rollouts were a done thing.
<stub> lp:~launchpad/+junk/dbpatches
<jml> but that was a reckless assumption.
<lifeless> jml: we're in pretty good shape
<bigjools> stub: ta
<lifeless> jml: code is written, staging is updating using it
<jml> lifeless: that's pretty good.
<wgrant> We really have no idea how long it will take on production, though :(
<lifeless> well
<lifeless> the first one will be nerve racking
<lifeless> and will change nothing.
<wgrant> Yes.
<lifeless> Stuart is investigating slony 1.2 as a possible major improvement
<wgrant> Well.
<wgrant> We *hope* it will change nothing.
<lifeless> wgrant: -har-, yes.
<wgrant> It's slony1 v2, not slony 1.2 :)
<lifeless> blah :P
<wgrant> (the product is named slony-i or slony1)
 * lifeless dons the not-a-details-job hat
<wgrant> Heh
<StevenK> bigjools: I agree with William, make it a string
<stub> My attention span:  |---|
<stub> Packaging learning curve: |-------------------------------------------------------------|
<wgrant> stub: What are you packaging?
<wgrant> Not Slony-I 2?
<stub> Will need a 2.0.7 package, but basic packaging has been on my todo list for 6 years now...
<wgrant> That's easy.
<stub> :-p
<wgrant> Assuming none of the patches need changing, apt-get source slony1-2; cd slony1-2-2.0.6; uupdate /path/to/new/tarball, dch -e and edit the changelog as fit, debuild -S -sa, dput ppa:what/ever ../slony1-2*_source.changes
<stub> Its arcane. Hunger is winning out over this introductory documentation atm.
<bigjools> StevenK: I am doing that.  I just dislike only having one solution discussed.
<StevenK> bigjools: So, you can either kill it, or make it generic.
<jtv> henninge: could I beg for a review?  https://code.launchpad.net/~jtv/launchpad/bug-820796/+merge/70701
<StevenK> Killing it requires a discussion with distro and Debian
<StevenK> Making it generic with a string is a my vote.
<henninge> jtv: sure, you may beg ;)
<jtv> Good, that's step 1.
<jtv> "henninge, please, _please_ review my branch for me!"
<jtv> That's step 2.
<henninge> jtv: I will do that.
<jtv> Thanks!  That's step 3.
<henninge> jtv: welcome
<henninge> baby steps? ;-)
<StevenK> bigjools: There. Two solutions discussed. :-P
<henninge> why is this diff taking so long ... Is it long?
<jtv> henninge: nopeâlooks like the differ broke.  :(
<jtv> It's a very simple branch.
<henninge> cool, I'll pull
<henninge> branch, I mean
<henninge> whatever
<bigjools> StevenK: you've been taking lessons in diplomacy from your wife
<jtv> cjwatson, are you back?
<StevenK> He is back today
<adeuring> lifeless: fancy a review of a StormRangeFactory MP? https://code.launchpad.net/~adeuring/launchpad/bug-739052-4/+merge/70717
<danilos> mrevell, hi
<wgrant> stub: Any luck?
<lifeless> adeuring: conflicts
<stub> eh?
<lifeless> adeuring:
<lifeless> Diff against target:	1338 lines (+879/-25) 21 files modified (has conflicts)
<lifeless> Text conflict in lib/canonical/launchpad/webapp/batching.py
<wgrant> stub: With the evils of packaging.
<lifeless> etc
<bigjools> stub: db MP winging its way to you
<adeuring> lifeless: gahhh
<stub> wgrant: your cheat sheet + common sense + google got me as far as a debsign failure. Docs bored me. Still hungry.
<wgrant> stub: Heh. Probably just your name + email in debian/changelog not matching your OpenPGP identity exactly.
<wgrant> stub: You can use debsign's -k option to specify a different key ID.
<stub> Yer  - need a (Work) in there
<adeuring> lifeless: devel merged; new revision pushed
<stub> wgrant: But that would involve me knowing what I'm doing and where to edit that :)
<wgrant> stub: Should be reasonably obvious in debian/changelog.
<wgrant> Although this is Debian-obvious, not classical obvious.
<stub> wgrant: I meant getting different options passed to debsign
<wgrant> stub: Should just be able to give it to debuild.
<stub> wgrant: debbuild done now. Now to learn about ppas and uploading
<wgrant> http://help.launchpad.net/Packaging/PPA/Uploading
<stub> wgrant: At what point to I get to specify 'lucid and above'?
<wgrant> stub: You have to upload to one, and copy to the rest later.
<wgrant> Normally one would upload to the earliest, as that is what it will build for.
<wgrant> Then the binaries and source will be copied to maverick/natty/oneiric.
<wgrant> AFAICT this should build fine on Lucid, but I guess we're about to find out.
<wgrant> stub: The upload series is specified on the top line in debian/changelog, in case you hadn't seen that.
<wgrant> Defaults to 'unstable'.
<wgrant> Which obviously won't work too well for Ubuntu.
<stub> ta
<stub> wgrant: version number 2.0.7-0~ppa1~lucid1 or 2.0.7-0~ppa1 ?
<wgrant> stub: That's one of the great unanswered questions of the universe.
<wgrant> stub: I'd probably use the latter.
<wgrant> I actually tend to use -0ppa1
<stub> what I have atm. docs are trying to confuse me :)
<stub> Ahh... what I had earlier before something told me to put in a twiddly
<wgrant> stub: It depends.
<wgrant> We often backport stuff, in which case we'll normally use $(original version)~ppa1
<wgrant> But -0ppa1 < -0ubuntu1 < -1, so it's cleaner and still works fine.
<stub> its in there now... lets see what lp does with it while I get fud
<wgrant> And building already.
<lifeless> jml: can we unsubscribe testtools-dev from the bug tracker ?
<lifeless> https://bugs.launchpad.net/testtools/+subscriptions
<jml> lifeless: you just want to be on the mailing list?
<lifeless> jml: I think having the mailing list get a copy of every bug change makes it useless for discussion
<jml> lifeless: oh huh. I'm not getting those mails.
<lifeless> jml: see the page I linked; you may need to look under 'other subscriptions' or something
<jml> lifeless: I'm not getting those mails.
<lifeless> jml: thats because they have the same message id
<lifeless> jml: and you are also subscribed directly
<jml> ah ok.
<lifeless> jml: and you are using a MUA that dedupes on message id.
<lifeless> https://lists.launchpad.net/testtools-dev/
<lifeless> jml: its also getting merge proposals
<lifeless> jml: have you, by chance, combined a discussion-group and a permission-granting-team ?
<jml> sigh
<jml> no.
<jml> I uncombined it.
<jtv> henninge: any questions about the review so far?  I'm hoping to scoot off soon.
<henninge> jtv: no, sorry, did not get that far yet.
<jtv> henninge: in the meantime, I replaced the core of the Ubuntu distribution infrastructure and it performed its first-ever production run.  I think I need to peddle out the tension on my bicycle soon.  :-)
<henninge> jtv: you mean, you pushed a new revisionÃ
<henninge> jtv: but go ahead and peddle
<henninge> s/Ã/?/
<jtv> henninge: sort of â Tom rolled out the necessary cron change for me.  And then we all sat around worrying until the script ran.
<henninge> ah that
<lifeless> adeuring: reviewed; couple of tiny things
<adeuring> lifeless: thanks!
<adeuring> lifeless: >>> def f(a,b):
<adeuring> ...  return a+b
<adeuring> ...
<adeuring> >>> reduce (f, 1, 2, 3)
<adeuring> Traceback (most recent call last):
<adeuring>   File "<stdin>", line 1, in <module>
<adeuring> TypeError: reduce expected at most 3 arguments, got 4
<adeuring> so, reduce(And, clauses) is right, I think
<lifeless> adeuring: huh? that seems unrelated ;)
<adeuring> lifeless: argh, sorry, got your comment wrong
<lifeless> And is a CompoundExpr
<lifeless>     def __init__(self, *exprs):
<lifeless>         self.exprs = exprs
<adeuring> right
<henninge> jtv: no questions, r=me
<jtv> thanks henninge!  That's stepâ¦ 4?
<henninge> jtv: wasn't that 5 already?
<jtv> Could be, could be.
<wgrant> stub: So, looks like slony1-2 all went well.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge, benji | Critical bugs: 247 - 0:[#######=]:256
<stub> Need to copy it into natty so I can test with it
<wgrant> You should see a handy https://launchpad.net/~stub/+archive/launchpad/+copy-packages
<wgrant> Make sure to select to copy with binaries, and to the right archive.
<stub> Not rebuild?
<stub> Guess better if I test the actual lucid binaries
<stub> What are p-series and q-series?
<wgrant> stub: Ubuntu 11.10 and 12.04 LTS, which should not be shown on +copy-packages.
<wgrant> But we don't really have a way to model that properly yet.
<wgrant> Er, 12.04 LTS and 12.10.
<wgrant> I am 6 months behind, apparently.
<bigjools> wgrant: should be very easy to filter in the vocab - check for len(supported archs ) == 0
<wgrant> bigjools: Yeah, but what if it's partially initialised and oh god everything comes back to this doesn't it.
<bigjools> argh
<bigjools> wgrant: we're going to have to suck that bullet and just fix this
<wgrant> Probably.
<StevenK> But we have a field in DSP for this purpose?
<wgrant> DSP is not relevant here; not everything has a parent.
<bigjools> needs to be a bool on the series I expect
<StevenK> Which we did have ...
<StevenK> Which hooked into DSP, and was a horrid hack, IIRC
<StevenK> benji: You haz QA to do!
<StevenK> gary_poster: QA for you as well :-P
<benji> thanks, StevenK; I'll look now
<gary_poster> StevenK, looking thanks
<jtv> Who will review some security.cfg extensions?  benji, I've already bothered your co-OCR today so I'm angling for you: https://code.launchpad.net/~jtv/launchpad/bug-822640/+merge/70730
<benji> jtv: sure, I'll take a look now
<jtv> thanks
<benji> jtv: done, looks fine
<jtv> Great, thanks!
<deryck> Morning, all.
<abentley> deryck: welcome back.
<deryck> abentley: thanks!
<danilos> abentley, hi, I was wondering about bug 820511, where it would be most useful to have the job ID logged? I was thinking of doing it in the existing debug statements in the runner, but I wonder if that'd be enough for the circumstances we were in?
<_mup_> Bug #820511: Job scripts don't report job ids <jobs> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/820511 >
<abentley> danilos: on the phone
<abentley> danilos: The normal logging should be fine.  I think it should be "info", so it still shows when the script is run non-verbosely.
<danilos> abentley, right, that's what I was thinking as well
<danilos> abentley, thanks
<abentley> danilos: cool.
<bigjools> jml: is it wrong that when I saw the sperm whale stranded on the Australian coast earlier, I thought "Beached as!"
<jml> bigjools: not at all :)
<jml> although I do believe that was a kiwi whale in the yt video
<bigjools> it may have crossed the Tasman
<jml> heh
<henninge> benji: Hi
<henninge> benji: are you currently reviewing something?
<benji> hi, henninge
<benji> henninge: not at the moment
<henninge> benji: ok, I'll take jelmer's big one, then.
<jcsackett> sinzui: can you mumble?
<sinzui> yes
<jcsackett> can you hear me, sinzui?
<henninge> jelmer: Hi!
<bigjools> hi henninge or benji, can I bother one of you for a review please?
<benji> bigjools: I can after I finish this one.
<bigjools> benji: thank you
<bigjools> https://code.launchpad.net/~julian-edwards/launchpad/code-bug-820941/+merge/70753
<henninge> benji: which one are you doing?
<benji> henninge: gary asked me to do https://code.launchpad.net/~gary/launchpad/bug821531/+merge/70738
<jelmer> henninge: hi!
<henninge> benji: ah, ok
<henninge> jelmer: Hi!
<henninge> jelmer: I just wanted to ask if your safe-branch-open branch is strictly refactoring.
<henninge> jelmer: I assumed so, just wanted to double-check.
<jelmer> henninge: yes, it should not change any of the existing behaviour
<henninge> jelmer: cool, I already approved it.
<jelmer> henninge: Great, thanks for the review :)
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 247 - 0:[#######=]:256
<henninge> jelmer: I'll do your other one before I finish.
<jelmer> henninge: Cool
<henninge> jelmer: Line 120 of the diff: I think you wanted to use the "port" parameter here.
<benji> bigjools: I'm done with the review.  I had one small question.
<bigjools> ok
<bigjools> benji: In the pre-requisite schema branch the SQL updates Ubunbu address.  Nothing else needs one right now.
<bigjools> Ubuntu, even!
<benji> bigjools: cool, I figred it was something like that
<bigjools> benji: thanks for reviewing
<benji> my pleasure
<jelmer> henninge, that's a good point. or potentially, we could get rid of the port parameter altogether
<henninge> or that
<gary_poster> bac, someone wants is to "_REALLY_ DELETE ACCOUNT" (https://answers.launchpad.net/launchpad/+question/167045).  "please delete my account permanently. i still can sign in even when i use the deactivate procedure recommended."  Are they out of luck?
<gary_poster> I'm going to say "no" and point them to https://answers.launchpad.net/launchpad/+faq/968
<sinzui> gary_poster, SSO reenables the user profile. Deactivate works, just do not sign in again to make it active
<gary_poster> thanks sinzui, yes, that's where I was going.  While I have your attention, someone wants to convert a person to a team (~lintian-maint fwiw).  Am I right that they cannot do this, and that they should create a new team and then fiddle with the names if absolutely necessary?
 * gary_poster going to lunch; will be back later
<sinzui> gary_poster, a person can be converted (claimed?) to a team in one case, but I cannot remember how that is done.
<sinzui> gary_poster, see lp/registry/stories/team/xx-team-claim.txt
<gary_poster> thank you sinzui
<benji> allenap: I think there's a prblem with the Python-dict-in-JS branch, my thoughts are in the MP: https://code.launchpad.net/~allenap/launchpad/javascript-utils-2/+merge/70760
<gary_poster> abentley, would you mind if I assigned you a code import question?  https://answers.launchpad.net/launchpad/+question/167251  Alternatively, I'm happy to formulate a reply if you give me the gist, or do other necessary grunt work.
<abentley> gary_poster: looking
<gary_poster> (that is, I'm happy to do other necessary grunt work if directed)
<gary_poster> thank you
<abentley> gary_poster: I'll take it.  I looks like the answer is "this is a bug in bzr-hg".
<gary_poster> ok thanks abentley
<j-johan-edwards> Hello, is there any way to get a dump for soyuz packaging records in the Ubuntu archives?
<j-johan-edwards> The sample data is pretty sparse...
<j-johan-edwards> Hey, sorry to repeat (won't again) but is there any way to get a dump for soyuz packaging records in the Ubuntu archive?
<abentley> benji: could you please review https://code.launchpad.net/~abentley/launchpad/ubuntu_merges/+merge/70772 ?
<benji> abentley: sure
<benji> I appreciate the courtesy space after the URL. ;)
<benji> abentley: done
<lifeless> morning y'all
<lifeless> sinzui: hi; re bug 298152
<_mup_> Bug #298152: privacy portlet is not visible enough for indicating private objects <disclosure> <lp-web> <oem-services> <privacy> <qa-ok> <ui> <Launchpad itself:Fix Released by jcsackett> < https://launchpad.net/bugs/298152 >
<sinzui> yes
 * jcsackett perks up.
<lifeless> sinzui: 2 questions;. A) there was some user feedback that the new ribbon is less noticable than the old border thing, particularly when someone has been switching tabs and is no longer at the top of page.
<lifeless> and B) have we enabled the feature flag to show it to everyone ?
<sinzui> lifeless, the privacy ribbon cannot be scrolled off.
<lifeless> sinzui: it used to have a close button
<jcsackett> lifeless: sinzui and i were going to add it to beta-testers. i forgot to ping a losa on that point.
<sinzui> I asked jcsackett to enable the ribbon for all beta testers. We will blog about it to encourage feedback
<jcsackett> i'll correct that now.
<lifeless> sinzui: so I guess the scenario is 'close, scroll, switch windows, come back and forget'.
<sinzui> lifeless, I question the hide button because it is not obvious how to unhide it
<jcsackett> sinzui, lifeless: i'm plus one on removing the hide part of it.
<lifeless> sinzui: anyhow, my question (A) is, has that user reaction been taken into account? Perhaps the bug should stay open, or a new follow on be opened, if not.
<thedac> jcsackett: did you need something done now?
<sinzui> lifeless, https://bugs.launchpad.net/launchpad/+bug/789566 < scroll and confirm it is fixed at the top of your screen
<jcsackett> thedac: yeah, i'm looking up a feature flag rule i need. :-)
<thedac> ok
<sinzui> lifeless, the ribbon is fixed using css2. if your browser does not support it, please update to a browser released in the last 10 years
<lifeless> sinzui: I don't actually have an opinion about it myself, just noticed the feedback in https://bugs.launchpad.net/launchpad/+bug/298152/comments/35 and no apparent response
<_mup_> Bug #298152: privacy portlet is not visible enough for indicating private objects <disclosure> <lp-web> <oem-services> <privacy> <qa-ok> <ui> <Launchpad itself:Fix Released by jcsackett> < https://launchpad.net/bugs/298152 >
<lifeless> sinzui: yes, its fixed when not hidden.
<lifeless> sinzui: I have no concerns about the implementation :)
<sinzui> sorry 'fixed' is a css position property. I did not mean this bug is fixed when the user can see the ribbon
<lifeless> sinzui: I meant 'pinned to the top of the visible area' when I said fixed.
<sinzui> lifeless, We had positive reaction, but NO reaction from non Canonical staff so I asked that we broaden the test
<lifeless> sinzui: that was because only canonical staff could see it :P
<sinzui> indeed that is why I asked
<lifeless> sinzui: sounds like its all under control... cool
<lifeless> thanks
<sinzui> lifeless, I wanted to remove the hide action, but that would be me second guessing that users do not want to shoot themselves in the foot
<lifeless> sinzui: could we tell how many people hide the ribbon somehow?
<sinzui> no I cannot, nor did I agree with the reason for that addition
<jcsackett> sinzui, lifeless: maybe we should just create a mockup with hide and without hide on some page and throw it to user testing?
<sinzui> I think there was confusion between the desire to show state and the need for a universal notification presentation that did not suck
<sinzui> jcsackett, I am hoping for someone to be reasonable a realise that the cost of a leak could be millions of dollars and individuals should not be given an opportunity to be the source of the leak
<jcsackett> sinzui: fair point.
<sinzui> jcsackett, as we discussed last week. The privacy portlet is not on subordinate pages. So the hide code that changes that portlet will not work
<jcsackett> sinzui: right, the hide code doesn't do the highlight portlet thing if there is no portlet.
<jcsackett> the original code already had a toggle for hiding without highlighting; based on our conversation i thought we elected to not remove "hide" and simply go with "only try to highlight the portlet if it exists".
<sinzui> jcsackett, mpt reported a bug where the ribbon separates the state from when you change the state. The privacy portlet is largely an artefact of the 2/3x designs. Moving the control to change privacy to the ribbon would 1 Remove the need for the hide button's effect.
<abentley> lifeless: AIUI, Launchpad URLs should never contain whitespace or non-ascii characters in the path portion of the URL.  Bug 819841 and bug 819859 would be solved if such URLs automatically 404ed.  Do you think it makes sense to apply that at the URL parsing level?
<_mup_> Bug #819841: OOPs rendering page with non-ascii URL <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/819841 >
<lifeless> sinzui: what stops you removing the hide button ?
<_mup_> Bug #819859: launchpad urls permit spurious whitespace <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/819859 >
<lifeless> abentley: do these urls currently work sometimes, or not at all ?
<jcsackett> sinzui: interesting. so we get rid of the privacy portlet (or at least the edit part of it) and move that into the ribbon, simultaneously killing the "hide" button?
<sinzui> understanding of why it was added. As I wrote above. I think it was a misunderstanding. Privacy is not a notification, it should not be dismissable
<lifeless> abentley: and by parsing level, do you mean in the incoming url routing area, or in our 'regex to figure out what is a url in a page' ?
<abentley> lifeless: urls with ascii whitespace work.  urls with non-ascii whitespate oops.
<abentley> s/whitespate/whitespace/
<abentley> lifeless: I mean in the incoming url routing area.
<sinzui> lifeless, yet one requirement may have been that the notice cannot take up vertical space that is needed on small screens
<lifeless> abentley: ok. You probably don't have this at hand, but do we emit such urls-with-ascii-whitespace from our regex linkifier code? E.g. would we break some in-site links from user content doing this?
<lifeless> sinzui: you could have a button that moves it to the left hand side ?
<sinzui> That was my original plan that was scraped by Huw's testing/design/work
<lifeless> sinzui: FWIW I accept your assertion that being able to hide it seems liable to increase the risk of accidental disclosure
<lifeless> abentley: I think 404ing on urls that cannot possibly resolve to urls we would generate makes a great deal of sense.
<abentley> lifeless: I can test whether we linkify them.
<lifeless> abentley: and even if we do break some things we linkify today, thats ok - we can fix the linkifier
<lifeless> abentley: I would ask, if we expect nontrivial fallout, that we do both at once; if the fallout is going to be minor, 2 landings, or even 1 with the second maybe-or-maybe-not happening would be fine with me
<sinzui> lifeless, the ribbon is officially in customer acceptance as of Wednesday. I expect to get feedback about the design and bugs about what is or is not displayed private
<abentley> lifeless: urls with ascii whitespace can and do resolve today.  So you would forbid only the non-ascii ones?
<lifeless> abentley: I would be ok with either all or only non-ascii as you choose; my thinking is that if we don't emit it, there is no reason to accept it, and room for interesting bugs if we do.
<lifeless> abentley: s/don't emit it/don't emit it or explicitly expect it [e.g. search form submission]/
<abentley> lifeless: Right.  And with with search form submission I believe we use the query, not the path.
<lifeless> abentley: yes
<lifeless> abentley: (you said 'urls' rather than 'url paths' before, and I was trying to avoid being pedantic while still giving useful accurate answers)
<abentley> lifeless: I started with "path portion of the url".  Sorry if I was unclear.
<lifeless> abentley: ah, I failed to parse!
<lifeless> abentley: so, we're synced :)
<abentley> lifeless: We do linkify URLs with whitespace: https://bugs.qastaging.launchpad.net/bzrtools/+bug/760435
<_mup_> Bug #760435: failUnless & co cause PendingDeprecationWarning <easy> <selftest> <verification-done> <Bazaar:Fix Released by mbp> <Bazaar 2.3:Fix Released by vila> <bzr-pipeline:Fix Committed> <BzrTools:Fix Committed> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Natty):Fix Released by jelmer> < https://launchpad.net/bugs/760435 >
<lifeless> abentley: that particular case I think is fine, cause its already hex escaped
<lifeless> I meant something like <https://foo bar/>
<abentley> lifeless: I meant something like %20 or %A0
<lifeless> abentley: in the web request yes; I was worried about someone finding a way to cause that from what looks like whitespace
<lifeless> abentley: e.g. the example I've put into that same qastaging bug
<sinzui> I am having network difficulties. Maybe no one will see this. I am restarting.
<abentley> lifeless: So to give examples, I think https://bugs.qastaging.launchpad.net/bzrtools/+bug/760435 should work, https://bugs.qastaging.launchpad.net/bzrtools/+bug/760435%20 and https://bugs.qastaging.launchpad.net/bzrtools/+bug/760435%C2%A0 should 404.
<_mup_> Bug #760435: failUnless & co cause PendingDeprecationWarning <easy> <selftest> <verification-done> <Bazaar:Fix Released by mbp> <Bazaar 2.3:Fix Released by vila> <bzr-pipeline:Fix Committed> <BzrTools:Fix Committed> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Natty):Fix Released by jelmer> < https://launchpad.net/bugs/760435 >
<_mup_> Bug #760435: failUnless & co cause PendingDeprecationWarning <easy> <selftest> <verification-done> <Bazaar:Fix Released by mbp> <Bazaar 2.3:Fix Released by vila> <bzr-pipeline:Fix Committed> <BzrTools:Fix Committed> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Natty):Fix Released by jelmer> < https://launchpad.net/bugs/760435 >
<_mup_> Bug #760435: failUnless & co cause PendingDeprecationWarning <easy> <selftest> <verification-done> <Bazaar:Fix Released by mbp> <Bazaar 2.3:Fix Released by vila> <bzr-pipeline:Fix Committed> <BzrTools:Fix Committed> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Natty):Fix Released by jelmer> < https://launchpad.net/bugs/760435 >
<abentley> lifeless: Right now, the second is equivalent to the first, and the third oopses.
<jcsackett> ?
 * jcsackett fails at keyboards again.
<lifeless> abentley: I agree
<abentley> lifeless: excellent.
<lifeless> abentley: I don't think the linkifier needs touching
<abentley> lifeless: cool.
<lifeless> abentley: based on this discussion + experiment
<lifeless> if someone hex encodes whitespace into a url in a page, we should linkify that
<abentley> lifeless: do you know where we parse/validate incoming urls?
<lifeless> abentley: A reasonable place to start would be lib/canonical/launchpad/webapp/publisher.py
<abentley> lifeless: Yes, I'll have a look there.
<lifeless> abentley: I don't know precisely
<thedac> jcsackett: to confirm feature flag change is on production, correct?
<abentley> gary_poster: do you happen to know where would be a good place to add extra validation rules for incoming URLs (i.e. when a browser does a GET or POST on us)?
<jcsackett> thedac: yes.
<thedac> jcsackett: and do you have a one liner log message I can put in?
<gary_poster> abentley, so, at the very beginning of processing a URL?
<abentley> gary_poster: Yes.  I want to verify that the "path" portion does not contain any whitespace or non-ascii characters.
<abentley> gary_poster: And by "whitespace", I mean %20 as well as " ".
<gary_poster> abentley, and I assume the goal is to not produce an OOPS.  It would relatively easy to do this if we didn't mind an OOPS with a IStartRequestEvent handler... I wonder if we can somehow short-circuit anything there.  alternatively...maybe we have overridden enough of the publisher that you can do it in there.  Lemme dig for a sec abentley
<abentley> gary_poster: okay, thanks.
<gary_poster> abentley, so, I suggest that you work after the request has been made, probably in beforeTraverse.  Take a look at eggs/zope.publisher-3.12.0-py2.6.egg/zope/publisher/publish.py and in lib/canonical/launchpad/webapp/publication.py .
<gary_poster> In publish.py, you'll see a publish function.  That's the basics of how Zope publishes a request.  It uses a publication object, which we have customized in the second file I listed.  As you can see in publish, if you raise an error in beforeTraverse, then the publication's handleException will be called.
<gary_poster> You'll want to raise an exception that has a view registered for it that gives the user the information they will want about the error.
<gary_poster> There are examples of times we do that in the LP code now; see zope.app.publication.browser.BrowserPublication.handleException (we override it but the meat you want is elsewhere) for how that works if you want the details.
<gary_poster> You might want to advertise this to devs after you are done; I wonder if there are any other "quit now" shortcut rules we might want to implement this way.
<abentley> gary_poster: Okay, thanks for the pointers.
<gary_poster> welcome
<gary_poster> lifeless, thanks for review!  I agree and normally self-review those, but I'm asking benji to make sure everything works for him in Gnome.  It's kind of a double-check for Oneiric
<lifeless> ah cool
<lifeless> I didn't realise
<deryck> abentley: hey, was talking and didn't realize I was on mute. ;)
<abentley> deryck: no worries.  I didn't mean to be in Mumble anyhow :-)
<deryck> abentley: heh.  Was coming back to my normal home from another call :)
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 247 - 0:[#######=]:256
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 247 - 0:[#######=]:256
<lifeless> bah, missed gary
<lifeless> benji: still here, perchance ?
<lifeless> abentley: btw - def publishTraverse(self, request, name):
<lifeless>  in publisher.py
<lifeless> seems to do the ascii only enforcement, so is close to what you were looking at
#launchpad-dev 2011-08-09
<lifeless> ahhhhhh
<lifeless>  select count(*) from bugnomination, distroseries where bugnomination.distroseries=distroseries.id and name='maverick';
<lifeless>  count
<lifeless> -------
<lifeless>   3329
<lifeless> 6K for lucid. Hmm
<wgrant> lifeless: What's enlightening about that?
<lifeless> wgrant: I had a scary moment
<lifeless> wgrant: I thought the +nominations page might not be batched
<lifeless> also we have nominations on debian sid
<lifeless> which is more than a little bong
<StevenK> lifeless: You know I found eight recipes over the weekend that reference LP branches, which is just odd
<lifeless> StevenK: interesting. Do you think they are intentional or confused ?
<StevenK> I think it's confusion
<lifeless> win
<StevenK> lifeless: https://code.launchpad.net/~launchpad-pqm/launchpad/stable/+recipes for example
<wgrant> LP's policy on trusting people to not do nasty things (eg. change bug details) seems to be flawed, because an awful lot of people seem to default to clicking on everything until something happens.
<lifeless> I would love a earnt-trust graduated facility
<wgrant> Create recipe on a Launchpad branch -> ISP banned forever
<StevenK> Haha
<StevenK> They keep attempting to build, too
<spm> if I may take the contrary position - if people are clicking (apparently) randomly to achieve something; that sounds like a failure to make it easy for them to figure out what they want and need; vs any issue with trust. I'd suggest further that raising barriers to doing something (trust) will only exacerbate the problem.
<lifeless> spm: stackexchange is an example of graduated trust
<spm> still sounds like targetting symptoms; not fixing the problem.
<lifeless> spm: sometimes the problem is lack of knowledge
<lifeless> spm: like driving a car
<lifeless> spm: making it easier to release the brakes and accelerate to 100kmph isn't fixing the problem
<lifeless> spm: I take your point, but I don't think this is an either-or situation
<LPCIBot> Project db-devel build #794: FAILURE in 3 hr 20 min: https://lpci.wedontsleep.org/job/db-devel/794/
 * StevenK blinks
<StevenK> That's a little ... quick
<lifeless> biab
<lifeless> now, 1062 /  129  BugTask:+index may be a regression
<lifeless> that, or something had a lock out on Bug / BugTask
<jtv> hi wgrantâdid I miss any further disasters?  There were some distressing messages from the script after we fixed the permissions issues, but it looks like most of them had been coming out of the old script since 2006.  :)
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 235 - 0:[#######=]:256
<lifeless> jtv: it was last night you deployed the new script right ?
<jtv> yup
<lifeless> jtv: does it write to the DB ?
<jtv> Yes
<lifeless> mmm
<jtv> (Note that the only thing "deployed" in the technical sense was a cron change)
<jtv> Trouble at the mill?
<jtv> (Please say no)
<lifeless> does it alter bug or bugtask at all ?
<lifeless> https://bugs.launchpad.net/launchpad/+bug/823028
<_mup_> Bug #823028: sudden contention on Bug/BugTask tables <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/823028 >
<jtv> Can do, yes.
<lifeless> jtv: yes trouble, 2000 timeouts in todays oops report
<lifeless> all on bug/bugtask selects
<jtv> And it's not even doing anything the old script wasn't doing.  :(
<lifeless> it may not be it
<jtv> Well one thing it does there is close bugs.
<lifeless> I'm simply starting to round up 'things thare are different'
<lifeless> it may not be contention, I've retitled to remove that assumption
<jtv> If transactions have become longer (which frankly I don't expect, but who knows) and the bugtask selects involve status checksâ¦
<lifeless> we've either tipped over a index bloat threshold causing a plan change to poor plans, changed a query in a poor way, or run into contention
<lifeless> I think thats about the size of the option-set
<jtv> Excuse me while I delete more bug spam that hit me _right after_ deleting the overnight batch, and then update my spam filters and _then_ go back to debugging akismet.  :(
<jtv> Nothing significant _should_ have changed w.r.t. these queries, but anything _could_.
<lifeless> 2000 yesterday, 1138 the day before, 1188 the day before, 998 the day before
<lifeless> ok, so its a big jump, but not as big as I thought
<jtv> Back.
<jtv> Still, good thing you're paying attention to it.  I'm just reading up on the bug.
<jtv> Count queries are nasty: lock-sensitive.
<lifeless> oh?
<lifeless> I didn't realise they were more lock sensitive than other selects
<jtv> Well it's more that looking at them, it's so natural to expect them to be less lock-sensitive.
<lifeless> they'll need to touch every page
<lifeless> so if there are lots of writes I'd expect contention on the page-access-lock
<jtv> That too, though thankfully postgres does no page locking.  :)
<lifeless> but they should be reading their mvcc-version of the pages
<jtv> Oh, there's a page access lock?  Didn't know that.
<lifeless> http://www.postgresql.org/docs/current/static/explicit-locking.html 13.3.2
<lifeless> 'In addition to table and row locks, page-level share/exclusive locks are used to control read/write access to table pages in the shared buffer pool. These locks are released immediately after a row is fetched or updated. Application developers normally need not be concerned with page-level locks, but they are mentioned here for completeness.'
<jtv> FFS SSO you know what oops I want, don't give me that stupid search page.
<jtv> Well that doesn't sound like there can be any contention for them as such.
<jtv> All that trouble loading up an oops page and then it renders the referrer string all across the oops text.  How depressing.
<lifeless> win
 * jtv starts up another browser to do the same dance with
<wgrant> lifeless: The new script only appeared 11 hours into yesterday.
<wgrant> When did the OOPSes start?
 * wgrant checks appserver graphs.
<lifeless> it looks like midnight precisely
<lifeless> and then 1200
<jtv> Ah, SSO login page, confirm a few certificates, and the unwanted search page again.
<jtv> Progress.
<wgrant> lifeless: Note that it only runs for ~25 minutes of each hour.
<jtv> Starting at 3 minutes past the hour.
<wgrant> The second half of each hour should have no publish-ftpmaster.
<lifeless> sure
<lifeless> the graph I'm looking at is hour granularity
<wgrant> https://lpstats.canonical.com/graphs/AppServer5xxsLpnetNoRobot/20110803/20110810/ is extremely troubling.
<lifeless> we'd need one that was 15m granularity to rule-out the publisher
<wgrant> 2/5 through 2011-08-07, things started going bad.
<jtv> Most of these seem to have happened at 22 past the hour.
<jtv> Although a lot of the oops pages are giving me "500" errors.
<jtv> Oh, here's one at 5:41.  So that's probably not our doing.
<lifeless> jtv: heh, poor liitle oops service
<lifeless> and one at Aug. 8, 2011, 12:42 p.m.
<jtv> It's oopsing.
<jtv> Definitely a big spike at 6:22 though.
<wgrant> lifeless: https://lpstats.canonical.com/graphs/AppServer5xxsLpnetNoRobot/20110809/20110810/ has the sort of granularity you are looking for.
<jtv> wgrant: I don't suppose it's simply BPPH scans pushing bugtasks out of a cache?
<wgrant> Not very likely.
<jtv> How come?
<wgrant> That hasn't changed.
<jtv> Well something's changed, and we're exploring the hypothesis that it may be something in this script.
<lifeless> jtv: bugtask is spectacularly hot, the linux vmcache would keep it in
<wgrant> jtv: Have you looked at https://lpstats.canonical.com/graphs/AppServer5xxsLpnetNoRobot/20110803/20110810/?
<wgrant> Look at the second half of the 7th.
<wgrant> Something started going wrong then.
<jtv> So not me then?
<lifeless> I think the graph is in BST
<lifeless> or vice versa
<lifeless> because it has a growth at 0722
<wgrant> BST is in the graph?
<wgrant> devpad is still BST, but ewww.
<wgrant> Argh.
<wgrant> I wish we could just get a graph of BugTask:+index timeouts.
<wgrant> Let's see...
<lifeless> log into the oopsdb
<wgrant> Mortals cannot.
<wgrant> We are restricted to obscene pipelines.
<lifeless> meh, remind me tomorrow and I will rt access up for you
<lifeless> we have no reason not to do that for anyone in the team that shows an interest
<wgrant> Maybe.
<jtv> wgrant: something elseâ¦ you said the new script didn't "ls -lR" the archive.  Are you quite sure?
<wgrant> jtv: No, it's there. I just expected it to be in a run-parts.
<jtv> Ah.  No, IIRC it wasn't in quite the right place to join either of the existing run-parts dirs.
<wgrant> Right, I recqall that discussion.
<jtv> BTW lifeless, I notice that we still have tons of fields on bugtask that look highly mutableâ¦ do we still update those directly on the bugtask or have you changed how that works?
<jtv> Heat in particular.
<lifeless> jtv: no, we still have liveness headaches there
<lifeless> its something we need to address
<jtv> And let's be honest: I just want to see if there's something to the normal forms past Boyce-Codd after all.  âº
<lifeless> :P
<jtv> In Translations I think it could make a real difference as well, not copying translationmessages around all the time just because some status flags move around.
<lifeless> jtv: you mean having a separate table ?
<jtv> Yes.
<jtv> Would save a lot of vacuuming, I suspect.
<lifeless> perhaps; plus index rewrites too
<jtv> Yup.
<jtv> Pretty much all the moving parts involved in updating rows.
<lifeless> OTOH we can solve contention by queuing updates to non-latency-sensitive fields
<jtv> This is latency-sensitive.
<jtv> Also, we have a bunch of partial indexes covering those flags.
<jtv> So updating one flag also searches for and updates another TM that currently holds the flag, and each of them gets taken out of one partial index and into another.
<lifeless> yah
<lifeless> would want to avoid hitting larger row counts of both constant-and-mutable-tables
<StevenK> lifeless: Like the query of death yesterday ...
<jtv> QoD?
<StevenK> jtv: http://paste.ubuntu.com/660924/ and related
<jtv> Oh that one
<lifeless> jtv: I got it down from 25 minutes to 6 seconds
<jtv> Nothing particularly mutable in there, is there?
<lifeless> jtv: but not reliably
<jtv> Ah, BPPH
<jtv> Oh
<lifeless> jtv: no, and there are bad stats involved
<lifeless> jtv: the bpn->bpr relation estimates are out by a factor of 10 for %linux%
<jtv> Not a dramatic difference, really, compared to what can happen.
<StevenK> You underestimate just how many binary packages the linux source package builds :-)
<jtv> Don't "LIKE" matches use some completely arbitrary guess?
<jtv> One of those cases where a table scan is probably best...
<jtv> lifeless: did you lift the BPN search out of the join and materialize it?
<lifeless> jtv: yes
<jtv> Good chap.
<lifeless> jtv: I tried a subselect, subselect with offset 0, CTE and temp table
<jtv> CTE probably does the same as temp table, perhaps with a bit less overhead?
<lifeless> jtv: all the gory details are in the -ops backlog
<lifeless> jtv: CTE performed much work - different plan
<lifeless> bah
<lifeless> temp table was < CTE was < others
<jtv> Oh, the optimizer can transform through CTEs already?
<lifeless> yeah
<lifeless> even in 8.4
<lifeless> the temp table meant it was planning on more accurate stats
<jtv> So not reliable as an optimization barrier then.  Does "offset 0" still work for that?
<lifeless> not sure
<jtv> Anyway, I hope that we can use separation of model and storage layers as a way to free the hottest parts of our schema from the OO-style bundling of static information with mutable status.
<jtv> Or look at an old pet peeve of mine: POTemplate.  Quite costly to retrieve, queried all over the place for lots of reasons, and the main performance suspect is a Unicode header column that we only ever need in 2 places.
<jtv> In fact maybe we should see these performance knells as a trigger for building that separate storage layer.
<wgrant> Is the expense loading them, or in row width?
<wgrant> If the former, we have a solution already.
<wgrant> If the latter, can we force them to TOAST or something?
<jtv> I'm not sure.  I've never had the time to investigate it.  IIRC I suspected a bit of both; certainly row width could be much much less and this is a pretty hot table.
<jtv> What is the solution we have for the former?  I've been angling for one for ages.
<lifeless> jtv: what do you mean (for clarity) by separate storage layer ?
<jtv> One thing I had in mind was "demand-loaded properties" in Storm.
<wgrant> jtv: See SPR.copyright.
<jtv> lifeless: what wallyworld brought up at the time â DAO
<wgrant> It's a manual implementation of on-demand column loading.
<wgrant> It's a bit ugly, but was a huge performance improvement.
<jtv> wgrant: ah, I started doing that exact same thing with POTemplate at one point, but time pressure didn't allow me to get very far with it.
<jtv> (I think TranslationMessage also has a copyright field, but it's unused)
<jtv> I think lifting that sort of thing out of the table into a "rarely needed static background information" table could be even better because it also improves locality of search queries etc.
<jtv> At the cost of the few places where you need the data, of course, but we have load_related/load_referencing now.
<wgrant> jtv: SPR.copyright is often dozens of lines long.
<wgrant> Not sure POT headers are that large.
<jtv> SPR is another one of those tables that on the one hand we use all the time as a waystation in joins and the subject of searches, but on the other hand it holds lots of detailed information.  And I suspect it'd break down quite neatly into a lean, hot dude and a cool, fat bloke.
<jtv> Dozens of lines?  You kids today have it easy.
<wgrant> Most of the time all you need to know about the SPR is the ID.
<wgrant> You can normally avoid joining through it at all.
<wgrant> Go directly from SPPH to BPB, for example.
<jtv> I don't notice those cases much, because I try not to join with tables in that way in the first place.  :)  But think of the cases where you need just id and spn.
<jtv> In fact I was discussing this with Simon Riggs a few weeks back.
<wgrant> Ah, true, often need SPN.
<wgrant> But we're going to denorm that onto SPPH shortly.
<jtv> Nice.
<jtv> Funny how denormalizing like that is no longer a dirty word.
<jtv> wgrant: while we're hereâ¦ one problem we keep running into is "find the latest SPPH (or BPPH, I suppose) for a given package in a given distroseries."  Would a separate cache for those buy us anything?
<wgrant> jtv: Denorming SPN and BPN onto SPPH and BPPH makes that pretty much free.
<wgrant> As we can have an (archive, distroseries, sourcepackagename, status) index.
<wgrant> Possibly even a partial index.
<jtv> But no index-only scans.
<wgrant> Unless we just do (archive, distroseries, sourcepackagename, status) WHERE status in (1, 2)
<jtv> But no index-only scans.
<wgrant> Why not?
<jtv> Because postgres doesn't do index-only scans.
<lifeless> jtv: I think having an explicit mapping layer would be interesting; I also think that we'll get more bang for buck by the SOA project
<wgrant> Oh, of course.
<lifeless> still, having a small number of rows to actually consider is a good thing
<jtv> lifeless: I suspect they're just different ways of looking at what in this specific case would be very much the same thing: properly isolate responsibility for querying and retrieving these objects, then use the elbow room afforded by that isolation to optimize storage for use.  I wasn't thinking so much of a very formal layer as of a gradual extension of our development patterns to suit that optimization.
<lifeless> jtv: sure
<lifeless> I would, for risk management and cycle time, do either one thing : optimise storage or change the way we query
<StevenK> wgrant, lifeless: I fail to see how denormalising SPN and BPN helps in this case. I need to reach for the SPR and then the BPR via the BPB anyway?
<wgrant> StevenK: You can get the SPR ID from SPPH, then join directly to BPB.
<wgrant> No need to join across SPR; all you need is the ID and name.
<wgrant> Both of which can be on SPPH.
<wgrant> SPR is pretty boring otherwise.
<wgrant> A few things need the version, but that is about it.
<wgrant> And by the time you want the version, you're probably already fairly selective.
<wgrant> So it's OK to venture into SPR.
<wgrant> As most queries are based on SPPH.status, rather than sorting by version.
<lifeless> the big thing for me is that these tables have live and historic data
<lifeless> I'd really like to see that partitioned
<wgrant> Really partial indices and clustering should solve everything, but I guess postgres isn't quite there.
<lifeless> well
<lifeless> I have to disagree there
<wgrant> It probably also relies on either better stats or customised plans.
<lifeless> after 5 years, 10 releases, 3 current - 30% of thedata *at most* is live, 70% dead.
<wgrant> And?
<lifeless> 10 years, 20 releases, 3 current - 15% live
<lifeless> etc
<lifeless> the stats will degrade linearly
<wgrant> There's no reason that has to be bad.
<wgrant> It's only bad because we are relying on random stats.
<wgrant> If we could dictate parts of the plan, it would be easy.
<jtv> *If* we can make sure that everything we do has good locality, so that the "dead" data can sit on disk undisturbed.
<wgrant> Right, we'd need clustering too.
<jtv> Not necessarily, but we'd need good locality.
<wgrant> All the partitioning does is adds barriers because postgres tries to be too smart.
<lifeless> this is a case for temporal normal form basically - but not a date based partition, its a status based partition
<lifeless> 4th?5th?6th? I forget the label.
<StevenK> I wonder if SPRs exist without SPPHs
<wgrant> -D  cronscripts/publishing/cron.publish-ftpmaster
<wgrant> Yay.
<wgrant> StevenK: Yes.
<wgrant> StevenK: eg. stuff that was uploaded and then rejected.
<wgrant> And stuff that was the victim of DB mangling grrr.
<StevenK> I'm just wondering if I work on the SPN/BPN denormalisation if it make my query not suck
<wgrant> Do you have a paste of the latest version?
<wgrant> Preferably with an explain analyze.
<lifeless> I can grab you one from staging
<StevenK> It's going to be faster than dogfood
<StevenK> I'd prefer no temp tables, since this query will be used by the evilness that is pickers
<wgrant> Now, the plan was to denorm the string name onto ?PPH.
<lifeless> http://paste.ubuntu.com/661655/
<wgrant> Which probably means you'd need to determine the set of candidate strings from BPN, then look them up on BPPH.
<wgrant> As otherwise you have a seq scan on BPPH, unless we have awesome trigram indices.
<lifeless> bah, thats xringd
<lifeless> I'll run linux now
<StevenK> But the whole point of the SPN was so stuff isn't duplicated?
<wgrant> StevenK: Is that a concern?
<StevenK> It might be
<wgrant> Lack of duplication without rationale is hardly something to fight to retain.
<lifeless> so a %LIKE% on SPN is fast
<wgrant> Sure, because the table is tiny.
<lifeless> it is unlikely to be fast if the strings themselves are denormed into ?PPH
<StevenK> wgrant: I daresay space saving is 'sans rationale'
<lifeless> but if you put the FK onto ?PPH that should be fine
<jtv> And worse than just not fast, it's impossible to estimate.
<jtv> With "I'm looking for these BPNs" at least the statistics have a chance.
<wgrant> lifeless: Not so long ago you were arguing thatn BPN and SPN should be abolished.
<StevenK> lifeless: What about a %LIKE% across BPN?
<wgrant> If you are OK with keeping FKs there, that's good.
<lifeless> wgrant: yes, with trigrams or fti they should
<lifeless> wgrant: these are separate discussions
<wgrant> lifeless: This denorm is reasonably expensive to implement.
<wgrant> We probably want some idea of where we are going to go.
<lifeless> wgrant: how is it expensive?
<wgrant> Mm, I guess it's not so bad if we have fastdowntime soon.
<StevenK> fti across SPN simultaneously makes me happy since we can rank matches, and scared since fti is a pox
<LPCIBot> Project devel build #958: FAILURE in 5 hr 40 min: https://lpci.wedontsleep.org/job/devel/958/
<jtv> FTI is a hard problem.  But does it make any sense at all for package names!?
<StevenK> Sure, why not?
<lifeless> no stemming rules for packages
<lifeless> and no substrings in the tsearch2 fti implementation
<wgrant> It would be nice if we could cheaply set up a copy of a few tables from the DB to test stuff on.
<StevenK> I was thinking that
<lifeless> We can test on [qa]staging
<lifeless> e.g. with temp tables in a transaction
<lifeless> or even on the main tables if its compatible
<lifeless> http://paste.ubuntu.com/661660/
<lifeless> StevenK: ^ wgrant ^
<wgrant> lifeless: What if you try a temp table of BPPHs with matching BPNs?
<lifeless> boostrapping that now
<lifeless> select * into temporary table test_bpph from binarypackagepublishinghistory;
<wgrant> Oh, I was thinking you could just get a temp table of BPPH IDs that match the BPN query.
<wgrant> Rather than duplicating a 15M row table.
 * lifeless shrugs
<lifeless> may as well understand the row width impact
<wgrant> I was hoping to cheaply and roughly emulate the BPPH.name index.
<lifeless> EODish, I'll be doin ghouse things for a bit etc, will pop back to do the next step
<wgrant> k, thanks.
<lifeless> heh thats done
<lifeless> now to add the column
<wgrant> I guess sourcherry is a bit of a monster, even if it isn't wildcherry.
<StevenK> Haha
<wgrant> Particularly since it's a small table.
<wgrant> s/small/narrow/
<lifeless>  update test_bpph set name=binarypackagerelease.binarypackagename  from binarypackagerelease where test_bpph.binarypackagerelease=binarypackagerelease.id;
<lifeless> running now
<lifeless> (and yes, I know I could have done this as one step
<lifeless> but this has a smaller footprint - I'm betting faster overall
<wgrant> In case you've not done it already, I think http://paste.ubuntu.com/661666/ is about as good as it gets.
<wgrant> ie. not very, but we'll see.
<wgrant> As long as it goes the right way we should be OK.
<StevenK> wgrant: Didn't you want to skip over SPR entirely?
<wgrant> StevenK: Can't here.
<wgrant> Since SPPH isn't involved in the second query.
<StevenK> Pity
<wgrant> We have to go through one of them.
<StevenK> wgrant: Where did 'JOIN _1 AS' come from?
<wgrant> s/_1 AS //
<wgrant> miscopied from lifeless'
<StevenK> Sigh, I can't run it anyway
<wgrant> http://paste.ubuntu.com/661675/ has the added bonus of working and being tested.
<StevenK> Oh mawson?
<StevenK> s/h/n/
<StevenK> Tempted to do the test_bpph shuffle there too
<wgrant> On dev.
<wgrant> I'm not hugely tempted to rewrite 18M rows on mawson, but maybe.
<StevenK> lifeless: I wonder if that UPDATE is done yet.
<StevenK> jtv: QA!
<lifeless>  update test_bpph set name=binarypackagerelease.binarypackagename  from binarypackagerelease where test_bpph.binarypackagerelease=binarypackagerelease.id;
<lifeless> UPDATE 15886099
<lifeless> Time: 884894.389 ms
<lifeless> there is another approach to ths btw
<lifeless> a dedicated query schema for the use case we are solving
<lifeless> e.g. a fact table with spn bpn, archive
<lifeless> I suspect that that would fly insanely fast and be tiny
<lifeless> even if populated for the world
<lifeless> will come back to that
<lifeless> we will probably want some indices
<StevenK> Might need distribution too, but that can probably be deduced from archive
<lifeless> for your specific case we don't, but yes, we might want the schema to have it
<lifeless> Seq Scan on test_bpph binarypackagepublishinghistory  (cost=0.00..882508.82 rows=1272 width=12)
<lifeless> so, I'll add a partial index on status, archive
<wgrant> Not name too?
<lifeless> sure
<lifeless> create index test_bpph_archive_status on test_bpph (archive, name) where status in (1,2);
<lifeless> this may take a little time
<wgrant> Probably also want DAS, but we'll see.
<lifeless> 30 seconds
<lifeless> estimates 4000 cost now; we'll see
<lifeless> wgrant: so, the reason we expec this to be better is that the active-filter is status,archive on bpph ?
<jtv> StevenK: are you trying to tell me in your very personal way that one of my branches has been touched by the qa tagger?
<wgrant> lifeless: Were I the planner, I would find candidate BPNs, then look up BPPHs by (archive, name, status)
<StevenK> jtv: I'm trying to emulate wgrant's nagging
<jtv> I could tell the difference easily.
<lifeless> crap stats
<lifeless>                                              ->  Bitmap Index Scan on test_bpph_archive_status  (cost=0.00..1250.23 rows=792 width=0)
<lifeless>                                                    Index Cond: (archive = 1)
<lifeless> analyzing
<lifeless> ok, now we get a sensible cost out - 900K
<lifeless> its doing a hash join
<jtv> Rarely good.
<lifeless> forcing that off, lets see
<lifeless> instant
<StevenK> Hm?
<lifeless> 500ms
<StevenK> Nice
<lifeless> thats with hash joins forced off
<lifeless> I think we may have bong tuning parameters somewhere
<lifeless> 12 seconds with it off, run 1
<lifeless> 11 seconds for run 2
<wgrant> Or possibly I'm better at devising plans than postgres, so it should listen :)
<lifeless> wgrant: we all are when we use domain knowledge
<wgrant> Exactly.
<wgrant> But we can't tell it to listen.
<lifeless> the trick is to figure out why pg thinks the nested loop is going to be so expensive
<lifeless> the %linux% may be a cause
<jtv> Well that'd do it.
<lifeless> it can eliminate all names shorter than linux
<jtv> As I said, there's no way to get conservative statistics on selectivity for a LIKE.
<lifeless> yeah :)
<wgrant> lifeless: What if you take the BPN query out into a CTE and hope it doesn't optimise through it?
<lifeless> won't work
<lifeless> temp table might, I'm just trying that now
<jtv> Pulling the BPN ids into the client may help.
<lifeless> jtv: this is roughly the same as a temp table
<lifeless> jtv: though the temp table is better
<wgrant> set i_am_right=true;
<wgrant> Surely that will work.
<StevenK> Haha
<jtv> lifeless: there may be a sharp change in performance as the size of your temp table exceeds the number of common values we keep in our stats.
<lifeless> jtv: yes, there can be - and passing in a big IN clause has a similar kneee
<lifeless> jtv: but the knee for IN clauses is lower than that for temp tables.
<lifeless> jtv: I haven't checked the code to see the why of this
<jtv> That's remarkable.  I would have expected the kneeee for the temp table to be dominated by our statistics config (which admittedly we've got higher than normal) and the one for the IN by projected seek costs.
<lifeless> you may be right; I haven't tried to model this on different pg instances
<lifeless> jtv: but IN seems to just bail somewhere in the K's of entries region
<jtv> Well that may just be the sensible thing to do based on random_page_cost.
<lifeless> ok, first cut with temp table was 46 ms but wrong
<jtv> I think I can beat that number, if the answer doesn't have to be right.  :)
<jtv> TDD implementation of factorial: if n <= 1: return 1 ; else: return 6
<lifeless> ok, this works well enough
<lifeless> 1341ms total time
<lifeless> http://paste.ubuntu.com/661687/
<lifeless> Indexes:
<lifeless>     "test_bpph_archive_status" btree (archive, name) WHERE status = ANY (ARRAY[1, 2])
<lifeless> I'd like to try a fact table
<StevenK> That 844 is hot?
<lifeless> yes
<lifeless> I was iterating on the bpn side
<lifeless> so some of the spn side dropped out of cache
<wgrant> (archive, distroseries, sourcepackagename, binarypackagename)?
<wgrant> Might also be interesting to have overrides in there, possibly.
<StevenK> lifeless: Why the OFFSET 0?
<lifeless> StevenK: hacks the optimiser
<jtv> Or rather, stops it from moving bits into and out of that query.
<jtv> Optimization barrier.
<lifeless> StevenK: forces it to optimise the subplan separately
<StevenK> lifeless: You could also try the SPPH.name bit too
<lifeless> the denorm? sure, after fact table + dinner
<lifeless> meh, not going to do archive.purpose
<lifeless> oh wow, DISTINCT + LIMIT 1 -> pointless work.
<lifeless> wgrant: StevenK: this is my proposed fact table building query:
<lifeless> http://paste.ubuntu.com/661693/
<StevenK> Your unbuilt assumptions are a little wonky
<StevenK> A BPB will exist, it just won't link to a BPR
<wgrant> Not necessarily.
<StevenK> SourcePackagePublishingHistory.status IN (1, 2) -- active ; s/active/published and pending source packages/
<lifeless> sure
<wgrant> active is the recognised term for that.
<lifeless> close enough :)
<lifeless> I ask because I get 7 rows (without the distinct) that are the same for a LIMIT 10 run of the query.
<lifeless>  id | archive | ds |  spn  |  bpn
<lifeless> ----+---------+----+-------+-------
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21221 | 61531
<lifeless>     |       1 | 29 | 21238 | 61565
<wgrant> lifeless: One for each DAS?
<StevenK> Per DAS would be my guess
<lifeless> makes sense
<lifeless> running the full mccoy : insert into spn_lookup SELECT DISTINCT sourcepackagepublishinghistory.archive,...
<lifeless> bah, needed a column statement or null::int at the start. Iz off again
<jtv> cjwatson: sorry for haunting you likeâ¦ something thatâ¦ hauntsâ¦ proverbiallyâ¦ (examples escape me) but if you have a moment to talk about bug 659769 I may be able to resolve it for you very soon.
<_mup_> Bug #659769: should copy custom uploads to newly-initialised release <derivation> <lp-soyuz> <new-release-cycle-process> <soyuz-publish> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/659769 >
<lifeless> ok, so done
<lifeless> 460K rows
<lifeless> 1400ms with a scan
<lifeless> 550ms union with a couple of naive indices
<lifeless> ah, 440 with a non broken query
<lifeless> jtv: is pg smart enough to split 'foo or bar' into two separate index walks ?
<jtv> lifeless: AFAIK they start out as separate nodes, and it's the optimizer that sees if they should be joined into one.  So yes.
<lifeless> 350ms with OFFSET 0 optimiser hacks
<jtv> Why bother optimising?  Just use a NoSQL database.  It's web scale.
<lifeless> because we want it fast now ?
<jtv> Hmm I need to work on that irony.  It evidently does not come across.
<lifeless> 114ms in bpn scan
<lifeless> 25ms in spn scan
<lifeless> 16ms in spn scan to get the names out
<lifeless> http://explain.depesz.com/s/P9n and http://paste.ubuntu.com/661734/
<lifeless> StevenK: ^ wgrant: ^
<stub> what is the source query you are trying to fix?
<jtv> Gaaaaahhhh!  Kill InitializeDistroSeries tests!  Kill!  Kill!
<lifeless> stub: an 8? 9? table monster - sourcepackagename -> release -> publication -> distroseries + the same on the binary side
<lifeless> stub: 'what source package names are currently actively published for Ubuntu and match %linux% for either the source package name or the binary package name'
<stub> so select spn.name from sourcepackagename as spn, sourcepackagerelease as spr, sourcepackagepublicationhistory as spph, distroseries where spn.name like '%foo%', spr.sourcepackagename=spn.id,spph.release=spr.id,distroseries.id=spph.distroseries union the binary side?
<lifeless> http://paste.ubuntu.com/661660/
<lifeless> has an experiment using a temp table as part of optimising things
<lifeless> I don't have the full baseline query handy, sorry.
<stub> yer - I'm after the simple (slow) form of the query.
<lifeless> substitute _1 into the second query on that pastebin, and you'll have it
<stub> lifeless: the distinct is unnecessary with the UNION, or perhaps leaving them in and using UNION ALL. Might help a little.
<lifeless> stub: both sides can return the same SPN, so UNION is needed; the DISTINCT improves the side query even though its conceptually not needed - drops one of the intermediary table scans down by 60%
<StevenK> lifeless: I quite liked the BPPH refactor results, too
<lifeless> StevenK: yeh; they /might/ be generally applicable, OTOH they make that table wider and perhaps slower at other things
<adeuring> good morning
<mrevell> Morning
<lifeless> StevenK: 320ms
<lifeless> StevenK: using distinct outside and union all inside
<StevenK> Is that SPPH.name, or the fact table?
<lifeless> fact table
<lifeless> and at that, 150ms of that time is just looking up the spn and bpn in their tables
<StevenK> My concern with a fact table is how to keep the bugger up to date
<lifeless> there are a few strategies; in the appserver, triggers
<lifeless> it may not be worth it. It was spectacular with bugtask though
<lifeless> so I thought worth testing here.
<StevenK> So, based on the numbers, I think the refactoring of name into {B,S}PPH is a good deal
<lifeless> there are 18000 linux package names in bpn
<lifeless> only 125 are relevant
<wgrant> Howso?
<lifeless> if we had trigrams I'd be keen to inline it
<wgrant> There should be way more than 125 relevant.
<wgrant> There are thousands of Linux binary names in the primary archive.
<lifeless> wgrant: the final result with all these queries has been 125
<wgrant> ah.
<wgrant> Right, that's just distinct sources, of course.
<lifeless> StevenK: these options aren't exclusive; can do both, or neither.
<StevenK> lifeless: Sure
<lifeless> StevenK: either way you have a redundancy to cater for
<StevenK> lifeless: Given the numbers for queries without either, I think it's clear we have to do one or both of them
<lifeless> agreed
<lifeless> its probably least engineering to do just the bpph denorm
<lifeless> spph isn't needed - we can get through to the spn in 300ms or so IIRC
<wgrant> We can probably sensibly do the fact table in triggers, unlike bugsummary, but the name denorm is still far simpler.
<lifeless> I'd consider maintain this one in appserver code - you know when you are deleting something much more accurately than a trigger would; will be faster.
<wgrant> Ahahaha no.
<lifeless> :)
<lifeless> which reminds me
<lifeless> we seem to have some skew built up or building pu in bugsummary
<StevenK> bigjools: O hai -- didn't you say you were moving DF to devel?
<wgrant> lifeless: That's amusing.
<wgrant> And entirely unexpected, of course...
<bigjools> StevenK: I'm waiting until FDT is deployed and working
<lifeless> wgrant: was a risk, which we did wwhat we could to address up front
<wgrant> Yay, oneiric.
<wgrant> lightdm appears to not start automatically any more.
<lifeless> no, thats the new version
<lifeless> ultralightdm
 * nigelb waves
<nigelb> hello!
<wgrant> Morning nigelb.
<rvba> lifeless: Hi, I'm having a look at your latest-monthly-pageids.html thing, could you send me a large zserver trace logs file for me to be able to run the script locally. (I'm sure I have access to logs somewhere but I need your help to find them)
<nigelb> hello wgrant
<nigelb> Working on my slow moving bug fix for tooltips today
<wgrant> rvba: carob:/srv/launchpad.net-logs/lpnet/*/launchpad-trace*.log
<wgrant> nigelb: How's that going?
<wgrant> rvba: Today's are a gigabyte so far, however :)
<nigelb> wgrant: slow, due to lack of time.
<wgrant> nigelb: That's probably the best reason for it to be slow.
<rvba> wgrant: Thanks!
<stub> lifeless: we can get trigrams - its in pg_contrib, but we have not used it yet (tied up in all our search story).
<nigelb> wgrant: I keep having to spend 5 to 10 minutes figuring out what I was trying the last time :)
<wgrant> lifeless: Ah, no, it's not lightdm... someone broke i915, so it's even more amusing.
<nigelb> Intel graphics is always fun.
<jtv> StevenK, still here?
<StevenK> jtv: Hm?
<jtv> I was just wondering about may_require_job vs. multi-parent.
<jtv> AFAICT may_require_job currently returns False if the derived series has any parent that's in the same distro, on the grounds that we don't track DSDs within the same distro.
<jtv> But ISTM that's just a surviving workaround from the time when it didn't get passed an explicit parent series.
<StevenK> I doubt that is a common case
<jtv> Now that it receives both derived_series and parent_series, I suspect the check should be simply for derived_series.distribution == parent_series.distribution.
<lifeless> stub: it would be interesting to see if trigrams got useful statistics for query plans
<stub> where were you thinking of using them? scan of the spn/bpn didn't seem that bad compared to the rest of the query you were looking at. Thinking of keeping redundant copies of the packagename so the queries don't need to go that deep?
<lifeless> stub: the query planner can't assess row counts sensibly for like '%...' queries
<stub> fwiw, WITH cte's should let you avoid temporary tables.
<lifeless> stub: different plans, WITH cte was 2-3 times slower than the temp table
<stub> because you could create an index?
<stub> I don't think trigrams will help assess row counts better.
<lifeless> no, because the temp table gets fully evaluated before the plan for the next query is done
<lifeless> so rather than seeing 1M rows, it sees 60K
<lifeless> when it thinks the row count is higher than it will be (by factor of 10, for instance), it will switch to hash joins, or seq scans
<lifeless> stub: I don't know if they do/don't. I can imagine a stats gatherer for them that will
<lifeless> so, one place to use them would be for that.
<lifeless> the other place, is, as you say, to inline the search term to a middle table in the query
<stub> yer, but this is in contrib so I doubt that exists atm.
<lifeless> however, we did a simple denorm experiment putting the bpn id onto bpph
<bigjools> lifeless or stub, is my db patch ok to land and if so I presume db-devel for now?
<lifeless> and that got us a 650ms query
<lifeless> which is probably fine
<lifeless> we can get 300ms without trigrams with a fact table
<lifeless> with trigrams we could shave 100ms off that easy, I think.
<poolie> did lp just lose its session cookies?
<stub> no
<lifeless> (because 100ms of that 300ms is the bpn scan)
<poolie> oh for some reason i got an edge url from history, nm
<jml> (also, LP still sends out edge URLs)
<stub> lifeless: I'm just putting up a branch turning off all the launchpaddatabaserevision checks. I can't think of any sane runtime checks that will work with fastdowntime that won't bite us in the arse.
<StevenK> jml: In which cases?
<wgrant> stub: Isn't the current one OK?
<wgrant> stub: That is, refusing to start if patches are unapplied.
<stub> wgrant: Only if we never use -0 patches.
<lifeless> jml: in email? yes, if someone uses edge to do an action.
<wgrant> We should stop special-casing -0 patches.
<wgrant> But I think the -[^0] patch rules are fine.
<stub> wgrant: we have optional patches that might not be live by the time the code is updated.
<lifeless> stub: all things considered, I'm +1
<wgrant> Perhaps -0 patches get the -[^0] rules, and -[^0] get no rules?
<stub> wgrant: But when would you ever use -0? You can't have a required patch because the code needs to run with the currently deployed schema and the HEAD schema.
<lifeless> stub: a required patch would be one that is deployed
<wgrant> stub: Note the code after the patch.
<wgrant> s/Note/Not/
<wgrant> A rev after the patch could reasonably expect to be running on the patch, couldn't it?
<lifeless> stub: e.g. develop it in a pipeline, land the db patch to db-devel, get it deployed, then land the rest to devel
<stub> So you get no benefit of the run time check, as you couldn't get that far without managing that yourself.
<lifeless> sure
<lifeless> which is why I'm +1, all things considered ;) just describing a scenario where it could save some headache, if someone landed on devel too soon without the db component.
<lifeless> of course, merging a pipe would merge the db patch to unless special care it taken
<stub> Your code can't land on devel until your patch has been merged from db-devel or your tests will fail (and if they pass, it isn't a required patch...). So it would catch someone merging db-devel without a deployment having happened.
<lifeless> rvba: legend!
<nigelb> stub: that's going to go into a T-shirt ;)
<rvba> lifeless: ;)
<lifeless> night all
<nigelb> night lifeless
<henninge> jtv: Hi! Do you have a few minutes for pre-imp chat?
<jtv> henninge: not really, sorry!
<henninge> jtv: np
<allenap> gmb: Thanks for the review :)
<gmb> allenap: No problem :)
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 235 - 0:[#######=]:256
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #795: FIXED in 6 hr 8 min: https://lpci.wedontsleep.org/job/db-devel/795/
<stub> gmb: An all-red MP for you: https://code.launchpad.net/~stub/launchpad/trivial/+merge/70840
<gmb> stub: \0/
<gmb> stub: Approved.
<stub> ta
 * gmb -> food
<danilos> mrevell, "You've got mail!"
 * danilos -> food
<mrevell> thanks danilos
<gary_poster> allenap, for your Dict, I imagine you already thought about wrapping an object (so that the dict API operates on the inner, "clean" object).  Just sharing a thought I had while reading mail. :-)
 * henninge lunches
<deryck> Morning, all.
<allenap> gary_poster: I did think about that. Unfortunately the convenience of dict.name notation is lost. Also, I think JavaScript is not quite rich enough (yet) to absorb all Python idioms, and maybe I just ought to get used to it.
<gary_poster> allenap, heh, ok cool.  You can also try a JS language compiler ;-)
<allenap> gary_poster: Yeah, that would be pretty cool. I'd need to learn how to see the Matrix to use a debugger with it, that's the only downside :)
<gary_poster> :-)
<rvba> gmb: Hi and thanks for the review! Do you think IÂ should land it as is or, like in suggest in the MP, get stub to host the jquery plugin file alongside the other stuff that the report page uses?
<gmb> rvba: I got as far as "uses?" and then a little message that says "incompatible encoding". My computer may be being all English and refusing to display something...
<rvba> gmb: weird ... "uses?" is the last word of my sentence ...
<gmb> Huh.
<gmb> Odd
<StevenK> "think Iï¿½should" is what I saw, so something odd is there
<gmb> rvba: Anyway, I think getting stub to host the plugin is probably the canonical best-way-of-doing-things here, so let's do that.
<rvba> gmb: all right.
<rvba> Thanks.
<gmb> np
<gary_poster> allenap, I found it interesting and amusing given our earlier banter that I just heard of this in a mailing list: http://www.infoq.com/news/2011/08/debug-languages-on-javascript-vm (would allow mapping debugging through compiled languages, and compressed JS, in Firefox and WebKit)
<allenap> gary_poster: Hehe, that is awesome :)
<gary_poster> :-)
<cr3> who created the images under ./lib/canonical/launchpad/images/? I'd like to ask for a similar image :)
 * deryck switches offices, back online soon
<gary_poster> bac, we have this bug: "My Ubuntu Member membership recently expired. I was expecting to get a notification by email when it came due, but I didn't. I have checked my spam trap, but there's nothing there."  Is that a regression or a feature request?
<bac> gary_poster: i know in the past i have gotten expiration reminders.  is it configurable?
<gary_poster> bac, no idea, looking
<bac> gary_poster: me too.
<bac> gary_poster: forwarded you an email
<bac> gary_poster: teammembership-email-notification.txt is a good start
<bac> gary_poster: team renewal policy must be ONDEMAND
<bac> gary_poster: so it is either a regression or the team is not configured as he expectes
 * bac suspects the latter
<gary_poster> bac, got it, thanks.  it is the ubuntu team AIUI but maybe it is loco.  Do you know where I can find the ONDEMAND setting for the team in the web?  I'm looking for ~yellow and have not found yet
<gary_poster> Actually bac
<gary_poster> I see option on +edit
<gary_poster> but option starts with text "When someone's membership is about to expire, notify them and:"
<gary_poster> so notification should always be sent according to that text
<bac> gary_poster: yes, i suspect it is the second of that group
<gary_poster> bac, you mean loco?
<bac> gary_poster: no, i meant ONDEMAND probably corresponds to the second selection
<bac> but you're right, it should notify based on that description in the UI
<gary_poster> bac OIC.  But we don't actually need ONDEMAND for the email to be sent out, right?  yeah ok. So I'll maybe try to find some logs for this and see if I can find the user there...and generally emails being sent out
<gary_poster> thank you bac
<bac> gary_poster: yes, looking through that doc test i see the other states should also send email
<gary_poster> ack
<sinzui> jcsackett, do you have time to mumble?
<jcsackett> sinzui: sure, just a moment.
<jcsackett> sinzui: can you hear me?
<jcsackett> sinzui: i just got dropped; one second.
<sinzui> I saw
<adeuring> gary_poster: do you know how I can get the permission to register a new release of laze.batchnavigator on PyPI?
<gary_poster> adeuring probably by asking me :-) what is your PyPI user name--the same?
<adeuring> gary_poster: yes
<benji> gmb: can I get https://code.launchpad.net/~benji/launchpad/bug-798945/+merge/70909 into your review queue?
<gmb> benji: Sure.
<benji> thanks
<gary_poster> adeuring, done (sorry for slow turn around, other things going on simultaneously)
<adeuring> gary_poster: thanks!
<gary_poster> yw
<gmb> benji: approved
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 235 - 0:[#######=]:256
<benji> gmb: thanks
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #959: FIXED in 5 hr 41 min: https://lpci.wedontsleep.org/job/devel/959/
<lifeless> morning
<deryck> morning, lifeless
<bac> lifeless: regarding bug 34086 and the new assignee i think you should check out https://answers.launchpad.net/launchpad/+question/167455, also by that person.
<_mup_> Bug #34086: removal of arch-all packages while there are arch-specific packages dependent on it results in uninstallable binaries <escalated> <feature> <lp-soyuz> <soyuz-publish> <Launchpad itself:In Progress by rreynoso45> < https://launchpad.net/bugs/34086 >
<lifeless> bac: heh
<lifeless> bac: I considered just toggling it back, but decided to be optimisting
<bac> lifeless: i think with the additional data it would be best to toggle it back
<lifeless> yeah, doing so
<bac> chr, ftw
<lifeless> thanks
<henninge> Hello!
<henninge> Can somebody please review this for me?
<henninge> https://code.launchpad.net/~henninge/launchpad/bug-823164-remove-translations-by/+merge/70961
<lifeless> henninge: wouldn't it be simpler for the user option parser to just store the string ?
<lifeless> henninge: thats what all the other scripts do, don't they ?
<henninge> lifeless: AFAIUI the reason behind this construct is that the option is validated during initialzation.
<lifeless> henninge: but its means you're in a transaction before you've taken out the script lock
<lifeless> henninge: this seems unwise
<henninge> lifeless: hm, true
<lifeless> probably harmless in this case
<lifeless> but I can imagine things that do lock in the db having trouble
<lifeless> anyhow, I'm fine if you want to land this
<lifeless> mmm, perhaos
<lifeless> acutally no
<lifeless> it conflicts with the inifile-shutdown thing for scripts
<lifeless> here is why: option parsing is in order supplied; until we consult the ini file (found via the config) to see whether the script can run, which means we need to know the config to be using...
<lifeless> we can't be sure that the DB will even be available
<lifeless> perhaps I'm wrong
<henninge> lifeless: yes, it seems wiser to change that construct.
<henninge> even if that is a bit more work
 * henninge regrets not having had a proper pre-imp discussion as he had meant to ...
<henninge> lifeless: thanks! ;-)
<lifeless> no probs!
<lifeless> I'll paste this in the review for education of anyone looking at it
<henninge> lifeless: cool
<LPCIBot> Project db-devel build #796: FAILURE in 5 hr 43 min: https://lpci.wedontsleep.org/job/db-devel/796/
<sinzui> wallyworld_, http://pastebin.ubuntu.com/662245/
<StevenK> sinzui: select count(*) from sourcepackagepublishinghistory as spph join distroseries as ds on spph.distroseries = ds.id where status in (1, 2) and archive = 1 and ds.distribution = 1; => 207k
<sinzui> StevenK, https://launchpad.net/ubuntu/oneiric shows only 18542 published in development. U think you should consider using distinct
<lifeless> StevenK: thats all series ever
<lifeless> sinzui: distinct isn't relevant, you're comparing apples and oranges.
#launchpad-dev 2011-08-10
<StevenK> lifeless: So, sinzui told me about another table that I had forgotten about on the call this morning.
<StevenK> launchpad_dogfood=> SELECT sourcepackagename from distributionsourcepackage as dsp join sourcepackagename as spn on dsp.sourcepackagename = spn.id where dsp.distribution = 1 and spn.name like '%linux%';
<StevenK> Time: 37.420 ms
<lifeless> is that populated now ?
<StevenK> Seems to be
<StevenK> 30k rows, a whole bunch for Ubuntu and Debian
<StevenK> lifeless: Try that query on staging for me?
<lifeless> 53ms
<lifeless> 136 rows
<lifeless> which suggests it includes stale packages.
<lifeless> is linux 2.6.8 still valid, for instance ?
<lifeless> StevenK: spn was never a problem anyhow ;)
<StevenK> It's a published source package, at least
<StevenK> lifeless: Well, no, but if we can fetch a good list of spns in 50 ms, it gives us a bit more time for the binary side
<wgrant> StevenK: You can't use DSP.
<StevenK> :-(
<wgrant> StevenK: It's used for storing bug reporting guidelines.
<wgrant> That is its only meaning.
<wgrant> Some uploads and gina create them.
<wgrant> But you cannot derive any meaning from them.
<StevenK> Curtis was of the opinion that it was used for upstream linking too
<lifeless> it is
<lifeless> it could be used, if you extend it to know active/inactive
<lifeless> and make sure its kept up to date
<StevenK> TBH, if wgrant says the data in it is not trustworthy I'm inclined to not use it
<lifeless> StevenK: oh, and for the vocab, don't forget to also look for published source package branches with no SPR
<StevenK> lifeless: Right, and that 26 minute query can take even longer :-P
<lifeless> StevenK: there is a escalated bug for ensemble requesting this
<StevenK> lifeless: Curtis and I have discussed it a few times. I want to get to the vocab working and then work on extending it
<lifeless> sure
<wgrant> Aha!
<wgrant> Unity has taken minimalism a step further.
<wgrant> It is now Nautilus.
<wgrant> The launcher and dash and indicators are gone... all there is is the Nautilus root window and its menu bar.
<wallyworld_> does anyone know how to create a project or distro with an icon for testing purposes?
<lifeless> set the branding
<lifeless> click on 'edit details' scroll to the bottom then 'edit branding', I think
<lifeless> its a very clunky bit of ui
<wallyworld_> lifeless: i mean in code
<lifeless> not offhand - does the factory have an option ?
<wallyworld_> not that i can see :-(
<StevenK> factory.makeDistribution() and then you probably have to twiddle something in it
<StevenK> Let's see
<lifeless> look for tests for branding to see how
<wallyworld_> ok, thanks. my other searches have failed to find anything
<StevenK> wallyworld_: Create an LFA and then set IDistribution.icon to it
<StevenK> You also have IDistribution.logo
<wallyworld_> LFA?
<StevenK> LibraryFileAlias
<wallyworld_> ok, there is a factory or something for that?
<StevenK> factory.makeLibraryFileAlias(content=<foo>) ; transaction.commit()
<wallyworld_> excellent, thanks
<wallyworld_> wgrant: StevenK: want a small review (one line change plus tests)? https://code.launchpad.net/~wallyworld/launchpad/person-affiliation-breakage-823644/+merge/70981
<StevenK> wallyworld_: It will cost you. What do you have?
<StevenK> Oh, right. You're married with kids, so nothing.
<wallyworld_> my eternal gratitude?
<wallyworld_> hah
<wallyworld_> i keep photos in my wallet where my money used to be :-/
<j-johan-edwards> > from lazr.restful.interfaces import IRepresentationCache
<j-johan-edwards> ImportError: cannot import name IRepresentationCache
<j-johan-edwards> Any idea what might be going on? (I have lucid's python-lazr.restful installed)
<StevenK> wallyworld_: r=me with a comment
<wallyworld_> StevenK: thanks
<wallyworld_> StevenK: with the commit, there's a number of other usages of LFA in tests without a commit. modt of them seem to not have one
<StevenK> j-johan-edwards: The version of python-lazr.restful in lucid is about 18 months old
<StevenK> wallyworld_: Right, so I'm not saying thou must add thy commit, or no +1 from me
<StevenK> wallyworld_: I'm just saying it's something to watch for
<wallyworld_> StevenK: np. thanks
<j-johan-edwards> StevenK: Ah. The wiki Running page said Lucid was the only support version
<StevenK> j-johan-edwards: What are you trying to do with lazr.restful?
<j-johan-edwards> Trying to make heads or tails of the source tree. Imported lp.soyuz.model.binarypackagerelease, which asks for that module.
<StevenK> j-johan-edwards: I would strongly suggest you use rocketfuel-setup to check out LP's tree
<j-johan-edwards> I did. It claimed everything was okay, so I ran `make schema`, which also worked
<j-johan-edwards> `make run` also works, oddly
<j-johan-edwards> I actually tried building on Natty too, but had a similar import error for lazr.config
<StevenK> Oh! Are you running 'python' and trying to import stuff?
<j-johan-edwards> Yup
<StevenK> Yes, that won't work
<StevenK> Use 'bin/py' in the source tree, or 'make iharness'
<j-johan-edwards> Thanks a ton! Just tried it again, and that works great.
<wgrant> j-johan-edwards: LP uses lots of dependencies that aren't in Lucid. You can find them in eggs/ within the LP branch.
<lifeless> nesting project groups would be mega useful.
<wgrant> lifeless: No.
<wgrant> lifeless: Nesting projects :)
<lifeless> wgrant: well, if project groups died, sure
<wgrant> They are going to die before they get any significant changes.
<LPCIBot> Project devel build #960: FAILURE in 5 hr 42 min: https://lpci.wedontsleep.org/job/devel/960/
<lifeless> is it intentional that LP shows a /!\ in the involvement portlet when you select 'N/A' for translations ?
<StevenK> It's probably supposed to be an icon
<lifeless> well it is an icon
<lifeless> I'm just confused
<lifeless> if you say 'unknown', a warning with 'lp needs to know' is sensible
<lifeless> but its set to N/A
<lifeless> shouldn't that just hide the line ?
<j-johan-edwards> Hello, sorry to come crying for help again so soon
<j-johan-edwards> Every getUtility(Iwhatever) call I make results in a zope.component.interfaces.ComponentLookupError
<wgrant> j-johan-edwards: bin/py doesn't set up utilities -- use bin/harness for that.
<j-johan-edwards> thanks!
<j-johan-edwards> Is there some piece of documentation I'm missing on the wiki? Or are some sections of the Hacking page (eg #Storm) just incomplete?
 * j-johan-edwards ignores the possibility he simply fails to grasp the obvious
<lifeless> j-johan-edwards: theres a vast learning curve around lp development
<lifeless> we want to improve that
<lifeless> (by lowering the curve, not documenting the curve :P)
<wgrant> j-johan-edwards: Well, Hacking doesn't say it should be run in bin/py.
<StevenK> It isn't a curve, it's a bleeding cliff!
<j-johan-edwards> haha
<lifeless> j-johan-edwards: so what are you hacking on ?
<j-johan-edwards> lp:archive-index (I might give up if I can't hack in though)
<lifeless> j-johan-edwards: integration with LP ?
<j-johan-edwards> lifeless: yeah, for the past few months I've been working on an archive crawler for mvo
<j-johan-edwards> but it's becoming obvious it won't scale, so an automated solution is important
<lifeless> what sort of data does it extract?
<j-johan-edwards> app-install-data-ubuntu, basically
<StevenK> Right, so it's as I feared. You'd like an easier way to get at contents rather than parsing Contents-<arch>.gz?
<j-johan-edwards> Yup, I saw your populate-bprc branch, which boosted my hopes of making that possible
 * StevenK had no idea he had a stalker ... :-P
<j-johan-edwards> haha, lifeless actually directed me to you, so you've got two
<StevenK> lifeless just stalks all LP devs, so that's nothing new
<lifeless> StevenK: and now its my job!
 * StevenK waits for "And I'm watching you ..."
<stub> StevenK: that will arrive by a postcard slipped under your door.
<StevenK> Haha
<wgrant> No, he'll find it in his pocket.
<lifeless> ... of his pyjamas
<StevenK> Along with a video called "The Branch" ?
<StevenK> j-johan-edwards: Keep in mind the BPRC work is in my spare time, and I'm not really pushing it with any priority
<nigelb> StevenK: "It isn't a curve, it's a bleeding cliff!" +!
<nigelb> +1
<wgrant> It's not that bad!
<wgrant> I submitted my first branch after like a day.
<StevenK> wgrant: Yes, but 1) You're insane. 2) You already knew Zope.
<wgrant> And it even changed tests.
<nigelb> haha
<nigelb> I wsa about tos ay that.
<nigelb> I wrote a one line patch.
<nigelb> Then I realized I had to write 25 lines of tests :)
<nigelb> so, for someone starting out, you open the code. and you need to grep to find what you're looking for. Then you a *lot* xml files.
<StevenK> My first branch for LP changed 2 lines. Sadly, it then broke 200 tests
<nigelb> fun!
<nigelb> I redid a branch 2 times I think.
<StevenK> nigelb: Why? You can get away with ignoring the ZCML
<StevenK> And bzr grep is *love*
<nigelb> StevenK: of course you can. but its scary to see a lot of xml :)
<StevenK> nigelb: Lots of XML is solved by shovelling more XML at the problem
<StevenK> After a while you forget what you were trying to solve.
<nigelb> is that like regular expresssions?
<nigelb> ;)
<StevenK> Oh hey, I wrote one of those yesterday
 * ajmitch is suddenly scared
<StevenK> And then ran it over devel ...
<nigelb> "if you have a problem and you use regular expressions to solve it, then you have 2 problems"
<StevenK> % bzr di -r submit: | wc -l
<StevenK> Using submit branch /home/steven/launchpad/lp-branches/devel
<StevenK> 6501
<nigelb> also of note, now I'm more comfortable with lp.
<StevenK> nigelb: You need to submit more branches!
<nigelb> StevenK: working on one, need more hours in a day.
<StevenK> Make a branch for that too?
<nigelb> against what project?
<StevenK> lp:universe
<nigelb> aha
 * StevenK ponders how to write a db patch, given this whole FDT thing
<nigelb> I'm going through rtfd that jml set up some time back.
<nigelb> I wonder if lp will ever get a feature like rtfd/github pages
<lifeless> if someone does it well, sure
<lifeless> though there is merit in just integrating with rtfd well, particularly if that site is open source (is it?)
<nigelb> it is
<nigelb> its django based
<lifeless> nigelb: and its source is available ?
<nigelb> https://github.com/rtfd/readthedocs.org
<lifeless> cool
<lifeless> now just need to get them hosted on an open site
<nigelb> so, out of the box, rtfd already can scan bzr branches.
<StevenK> That is one thing that annoys me. How much flak did LP cop for not being open, and you don't see people complaining about github
<nigelb> heh
<wgrant> Nobody is going to clamour for GitHub to be open, because GitHub is not crap.
<lifeless> that seems spurious :)
<wgrant> Oh?
<wgrant> People wanted LP to be open partly because LP was a piece of shit :)
<lifeless> I would expect the reverse correlatio
<lifeless> more good -> more desire for it to be open
<nigelb> everyone seems to have a github.
<wgrant_> jtv: QA!
<nigelb> haha
<wgrant> That was a real WTF moment for a while there.
<jtv> wgrant: FO!
<nigelb> lol
<nigelb> WIN
<nigelb> StevenK: That was awesome.
<wgrant> I must need to crank up the protection on wgrant_...
<StevenK> Now to look up my Nickserv password again
<nigelb> can't protect it for these less that 30 sec bursts.
<nigelb> unless you're signed into it.
<nigelb> in which case, StevenK can still d wgrant__
<nigelb> or wgrant`
<StevenK> It was for comedic value only
<nigelb> and it was sucessfull run!
<wgrant> Yeah, I only have wgrant and wgrant_, and ENFORCE only works after a certain time.
<wgrant> Sad.
<StevenK> I told jtv he had QA to do and he told me I could have been better at impersonating wgrant.
<StevenK> So I was.
<nigelb> So, hacking on Lp can still be counted as rest when I'm sick right? :)
<wgrant> I hope so, or I'm a bad person.
<lifeless> if it counts as rest, thats proof you are sick ?
<nigelb> hah
<lifeless> pop quiz
 * wgrant blames python-oops.
<lifeless> Would defining an oops report as a 'dict suitable for bson serialisation' be problematic ?
<lifeless> wgrant: :P
<wgrant> lifeless: That's what has always made a lot of sense to me.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[#######=]:256
<j-johan-edwards> I just retrieved a record from a table
<j-johan-edwards> Most beautiful moment of my life...
 * wallyworld__ doing qa, and finds an interesting user on qas:
<wallyworld__> Cost Of Viagra - Online Pharmacy - No Prescription Drugs, Health and Beauty, plus more in Launchpad in Launchpad (dalefyoboardinghouse)
<wallyworld__> <email address hidden>
<wallyworld__> Member since 2009-12-17
<wgrant> Hah.
<nigelb> lol
<wallyworld__> i'm too young for viagra
<wgrant> We get a lot of them, sadly.
<wallyworld__> just
<wallyworld__> do we have permission to deactivate such users ourselves?
<wgrant> Administer account
<wgrant> Set status to suspended.
<wgrant> Declare victory.
<wgrant> There are instructions on the wiki.
<wgrant> On dealing with spam.
<wallyworld__> np. was just wondering out loud if we had permission to do that
<lifeless> oh wow
<lifeless> this is going to break LP :P
<lifeless> we've been parsing iso8601 timestamps as TZ unaware.
<wgrant> Python's good at that.
<lifeless> ah well, extracted code can be good.
<lifeless> LP can deal.
<lifeless> woot
<nigelb> extracted? abstracted?
<lifeless> extracted
<nigelb> what does that mean?
<lifeless> to take out of
<lifeless> well, 'taken out of' 'removed'
<nigelb> ah
<lifeless> lp:python-oops
<lifeless> can has review? https://code.launchpad.net/~lifeless/python-oops/extraction/+merge/70993
<StevenK> Suppose
<lifeless> original code is in lib/canonical/launchpad/webapp/errorlog (and tests/test_errorlog)
<lifeless> StevenK: thanks!
<lifeless> stub: how long does the 'ALTER TABLE BugMessage ALTER COLUMN owner SET NOT NULL;' take to apply ?
<lifeless> stub: (I'm wondering if an index could do it instead, and be done CONCURRENTLY
<nigelb> who's the postgres expert?
 * nigelb points them to https://schemaverse.com
<lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/useoops/+merge/70995
<lifeless> nigelb: thats evil
<nigelb> heh
<rvba> lifeless: Hi, I'd be happy to move all the js files used to ~lpqateam but I'm afraid you will have to tell me how to do that (if I can do it myself) because I don't seem to have the right permissions.
<lifeless> I can probably do that later tonight; or losa could;
<lifeless> or francis or the qa team :)
<rvba> lifeless: ok thanks, if you don't have to do it before you EOD, I'll grab a losa later today then.
<jtv> Thanks for the review, StevenK
<rvba> don't have *time* even
<jtv> StevenK: The extra check you're asking for is not under the purview of the method that the test is for; it's DSD.update's job and presumably you tested for it!
<jtv> The fact that update() is called on the DSD is also already tested.
<adeuring> good morning
<stub> lifeless: as long as it takes to do a full table scan of bugmessage
<stub> lifeless: You can't duplicate that constraint with an index.
<stub> lifeless: it might use the index on the column to avoid the scan, but I wouldn't count on it. Staging timing is what we want to see.
<stub> full scan is < 10 secs
<lifeless> stub: cool
<lifeless> stub: we're going to need to think up ways to handle big tables ;)
<stub> lifeless: Apart from bypassing PG's protections and attacking the data dictionary directly, you can't. Theoretically CONCURRENTLY could be added to this stuff, but I doubt anyone will bother.
<stub> lifeless: Best way of handling big tables is to not have them :-)
<allenap> Thanks for the review StevenK :)
<lifeless> heh :P
<lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/useoops/+merge/70995
<stub> lifeless: Not totally joking. Archiving old data, partitioning
<lifeless> stub: yeah, I know
<stub> lifeless: reviewed
<lifeless> thanks!
<StevenK> allenap: I'm not so happy with JS-only reviews, but it looked pretty good.
<allenap> StevenK: Didn't you hear? The SOA is going to be based on node.js.
<lifeless> allenap: you can troll so much better if you want to
<allenap> StevenK: Didn't you hear? The SOA is going to be based on Drupal.
<StevenK> allenap: Are you just working down a list? Is PHP the next item? :-P
<wgrant> Drupal *is* PHP.
<allenap> StevenK: I thought I'd gone past PHP with Drupal.
<lifeless> stub: so poking at the internals and reindexing seems plausible as a future recipe
<stub> lifeless: I'd consider not adding the constraint before considering internal hacking. That could screw us, and we not know for quite a while.
<lifeless> sure
<lifeless> not something to do lightly
<stub> Yay standardization. Landscape just got pgbouncer installed on port 5434 - assume 5433 was already taken ;)
<lifeless> \o/
<wgrant> stub: Your slony1-2 packages didn't burn down your machine?
<mrevell> Morning
<stub> wgrant: Working fine. And all the staging stuff worked out of the box like you said - thought I would have to rewrite parts of the automation but looks unnecessary!
<wgrant> stub: Yeah, I was a bit wary about trying it when I did, thinking I was going down a bit of a rabbit hole... but nope.
<wgrant> All worked.
<wgrant> Python exceptions are slow :(
<jtv> Yes.
<wgrant> For dicts, it seems that these relations hold: LBYL with 'in' < get() < catching KeyError
<wgrant> I would not have expected get() to be slower than a manual check.
<lifeless> mmm
<lifeless> 'in' + get will be slower than just get, if you know you want the item
<lifeless> in itself is fast, yes
<wgrant> Nope.
<lifeless> really?
<lifeless> !cite
<wgrant> Well, if there are lots of misses then checking 'in' first is faster.
<wgrant> Saves 50ms in privilege delta generation.
<wgrant> Not significant, but rather odd.
<lifeless> strange:
<lifeless> http://paste.ubuntu.com/662462/
<wgrant> Odd.
<lifeless> get on hit or miss is ~constant
<lifeless> in is also approx constant, and 1/3 a get
<jtv> how sensitive are they to what's in the dict?
<lifeless> the hash() call on the object is crucial
<jtv> This is a pretty big difference already.
<lifeless> beyond that, its dict size - and they scale very well
<jtv> btree, right?
<jtv> Just curious about irregularities w.r.t. how deep in the tree the element is, and how branch prediction factors in.
<lifeless> its a hash
<jtv> Open or closed?
<lifeless> open
<lifeless> http://permalink.gmane.org/gmane.comp.python.devel/29960
<lifeless> mrevell: hey
<lifeless> mrevell: I'd like to catch up with you sometime; when would be good?
<jtv> No linear probing?  I would've expected that to work pretty well with good access predictors.
<mrevell> lifeless, Howdy. How does after the TL call sound?
<lifeless> mrevell: sweet
<wgrant> stub: So, I have table privilege resetting down from 5.5s to 60ms.
<wgrant> Which should shave 40s off the production outage time.
<wgrant> Which is possibly handy.
<stub> wgrant: \o/
<wgrant> The whole thing still takes 450ms per DB, but the first 400ms is as-yet unoptimised and probably small enough that we don't care too much.
<LPCIBot> Project db-devel build #797: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/797/
<stub> wgrant: Yup. We are in the good enough zone for now. We need to test this on staging first of course - if we land it now on db-devel it should get a run before the first possible fastdowntime deploy.
<stub> wgrant: or we can leave it until after and have synced db-devel->devel etc.
<rvba> matsubara: Hi Diogo, may IÂ bother you with a small "moving files around" problem? I'd like your help to put a few files in https://devpad.canonical.com/~lpqateam/.
<rvba> matsubara: Here is the whole story https://code.launchpad.net/~rvb/launchpad/trace-report/+merge/70839
<matsubara> hi rvba. sure thing. let me take a look
<matsubara> rvba, I need to figure out how the ppr is deployed. I'll do that today and let you know what's needed
<rvba> matsubara: Ok great. (It's just a matter of moving the 3 or 4 Javascript file that are needed by reports like https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-monthly-pageids.html but I'll obviously have to change the paths in the python code that generates the reports once you've decided where to put them)
<rvba> files*
<allenap> Okay, who removed check-db-revision.py?!
<StevenK> allenap: "Come on, own up! I can wait all night ..."
<matsubara> rvba, I put all the js files here: https://devpad.canonical.com/~lpqateam/ppr/js/
<matsubara> would that work for you?
<rvba> matsubara: absolutely. Thanks a lot. I'll modify the script.
<matsubara> rvba, cool. you're welcome. let me know when that lands so I can update the launchpad branch on devpad
<rvba> matsubara: I'll make the changes right away and throw that at ec2.
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilos | Critical bugs: 237 - 0:[#######=]:256
<danilos> allenap, re check-db-revision, could it be related to our hot/cold db patches stuff? iow, 2208-*-0 patches are checked, and all the others aren't (at least that's how I understood it)
<wgrant> danilos, allenap`: We needed a way to have optional patches.
<wgrant> danilos, allenap`: And it was decided to just remove the checks entirely.
<allenap> wgrant: The problem is that check_schema in Makefile is broken. I wondered if anyone had noticed. If not, I'll kill it.
<wgrant> Hm, why didn't that hit anyone earlier, I wonder.
<wgrant> Anyway, yeah, should be killed.
<jml> is there a syntax highlighting mode for build failures, or something?
<wgrant> jml: For binary package builds? No.
<wgrant> Given that they are in whatever format the package chooses, that would be... non-trivial.
<jml> well
<jml> I guess I don't really know enough about what canonical.buildd does, but I could imagine it inserting some kind of markup to make failures easier to read.
<jml> or, you know, *find*
<wgrant> jml: We have no control over the output between the lines starting 'dpkg-buildpackage: ' and the '************************************************' right before the 'FAILED [dpkg-buildpackage died]'
<wgrant> We like to treat everything between 'RUN: /usr/share/launchpad-buildd/slavebin/sbuild-package' and 'RUN: /usr/share/launchpad-buildd/slavebin/scan-for-processes' as not ours, too, since that would bring us further away from real sbuild, when we hope to use the package soon.
<jml> for my build failure, those are lines 524-597 out of 720
<jml> which is a pretty small fraction
 * jml looks again
<jml> and would probably be interesting to call out in some way
<jml> starting from sbuild-package expands it quite a lot.
<jml> wgrant: are there plans to switch to real sbuild?
<wgrant> jml: There's even a branch.
<jml> zowie
<wgrant> But it's not quite landable.
<wgrant> It's been dormant for around a year.
<wgrant> May get to it when I'm on maintenance again.
<wgrant> But it needs to be done this year some time.
<wgrant> Or Ubuntu will explode.
<jml> well more advanced than 'stop using commercial admin' plans
<jml> ubuntu has already exploded a little
<wgrant> Heh.
<jml> apt-file doesn't work any more. Debian has its Contents files in different places now.
<wgrant> They seem to be in the same place as usual...
<wgrant> http://ftp.debian.org/debian/dists/sid/Contents-i386.gz
<wgrant> Huh.
<wgrant> launchpad_main holds only DELETE on packagebuild.
<wgrant> Not SELECT or anything.
<jml> wgrant: apt-file is looking for them under the component dirs.
<jml> e.g. http://ftp.debian.org/debian/dists/sid/main/
<wgrant> jml: They're there too (Ubuntu doesn't have component-specific ones, though)
<jml> yeah, so apt-file is broken on Ubuntu.
<wgrant> Ohh, I see what you mean.
<wgrant> Right.
<wgrant> I thought you meant our apt-file was broken for Debian.
<wgrant> Right.
<wgrant> We'll be writing out Contents files natively soon (StevenK has a branch for big bits of that), so it will be easier.
<wgrant> We still use apt-ftparchive to do it, which Debian stopped doing a year or so ago.
<jml> what does Debian use now?
<wgrant> They write it out directly from dak's DB.
<elmo> wait what?
<elmo> they put package file listings into the DB?
<wgrant> Yes.
<elmo> wow, that's insane
<wgrant> Howso?
<wgrant> a-f is not precisely fast.
<elmo> the answer is not to bloat out the DB with something that's you don't use in any sort of relational way
<elmo> I mean if it works for Debian, all power to them
<elmo> but for us, it would be crazy
<wgrant> It should ideally be in some separate database, yes.
<elmo> it's a cache, not a DB
<wgrant> A DB cache.
<elmo> *shrug* sure - putting that into launchpad_prod would be, IMNSHO, worse than silly
<elmo> (and that's me trying very hard to be polite)
<wgrant> The hottest part of the table would be ~300MB that is accessed hourly, or possibly only once a day.
<wgrant> It's not *that* terrible.
<elmo> a single Contents file for a single architecture/series is 300Mb
<elmo> so I question your maths
<wgrant> You have a good point there.
<elmo> and quite frankly, even if it only ends up adding tens of gigabytes, it's still completely the wrong thing to do
<wgrant> I'd forgotten about the whole multiple archs thing there for a moment.
<wgrant> Maybe in a few years we'll have multiple DBs.
<elmo> and until then we'll keep adding stuff to the single one we have that we don't need to?
<allenap> danilos: Do you fancy reviewing a 1910 line diff? 90% of it is mechanical change that you can skip. https://code.launchpad.net/~allenap/launchpad/lazr-anim-stuff/+merge/71016
<danilos> allenap, "fancy" is the wrong word, but sure :)
<allenap> danilos: Thanks :)
<danilos> allenap, ok, mostly good, I wonder if a check Y.Lang.isValue(to) would work better for the "to" variable (line 248 of the diff)
<allenap> danilos: Yeah, that would keep in spirit with the rest of the changes.
<danilos> allenap, right, that's all I could find to complain about, so r=me :)
<allenap> danilos: Thanks!
<deryck> Morning, all.
<allenap> Morning deryck :)
 * danilos -> food
<allenap> danilos: Got time for a *much* shorter JavaScript branch? https://code.launchpad.net/~allenap/launchpad/phantom-overlay-bug-820828/+merge/71044
<danilos> allenap, you've got a bunch of conflicts there
<danilos> allenap, though, that seems to be from the anim branch
<allenap> danilos: Yeah. I'm working on them now.
<danilos> allenap, r=me, but I wonder about the var declaration and assignment split-up: is there a reason to prefer one over the other? or is that just your coding style that you hope becomes LP JS style?
<allenap> danilos: Ah, that's to silence lint as much as anything. It complains that PersonPicker is not fully defined in PersonPicker.
<allenap> I have no preference for that style, but I don't mind it either.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilos, abentley | Critical bugs: 237 - 0:[#######=]:256
<bac> matsubara: great job on the dashboard!
<allenap> danilos: Thanks for the review :)
<matsubara> thanks bac :-)
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugs: 237 - 0:[#######=]:256
<gary_poster> abentley, are you up for answering an importd question from spm, or should I try to get an answer from thumper or mwhudson?  https://answers.launchpad.net/launchpad/+question/167471
<abentley> gary_poster: best to ask thumper or mwhudson.
<gary_poster> abentley, ack will do thanks
<gary_poster> bigjools, would you be willing to field https://answers.launchpad.net/launchpad/+question/167457?  If I had to guess, I would guess that this is a "convert question to bug and mark WONTFIX" but it's really well out of my league.
<bigjools> gary_poster: I'l answer it.
<gary_poster> Thank you bigjools
<bigjools> gary_poster: answered.  He probably won't like it... I hope you grok what I said
<gary_poster> bigjools, I read it and understood, thank you (other than not being familiar specifically with quilt).
<bigjools> gary_poster: I am not that familiar with it either :)
<gary_poster> heh ok :-)
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #961: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/devel/961/
<allenap> abentley: Hi there, got time for a little bug fix in browser code? https://code.launchpad.net/~allenap/launchpad/dsd-batch-position-reset-bug-822781/+merge/71069
<abentley> allenap: sure.
<allenap> Thanks.
<abentley> allenap: what is the full_url for?
<abentley> allenap: I think "The forms should post to themselves, including GET params" is internally inconsistent.
<allenap> abentley: The full_url is to show that the URL is not just for the action, which it was before. It's just a naming exercise to show that the action_url and next_url are both essentially the URL with query parameters.
<allenap> abentley: I'll fix the docstring, oops.
<abentley> allenap: It would be nice if  BasicLaunchpadRequest.get_url could produce URLs that included the query string.
<abentley> s/get_url/getURL/
<abentley> allenap: r=me.
<allenap> abentley: I could add a getFullURL or getURLWithQuery method to LaunchpadBrowserRequestMixin, which would then sort out LaunchpadTestRequest at the same time as BasicLaunchpadRequest.
<abentley> allenap: Or getURL(include_query=False).  Any of those would be nice.
<allenap> abentley: Okay, I'll do that, shouldn't take long... <days later>...
<abentley> allenap: -)
<sinzui> jcsackett, do you have time to mumble?
<jcsackett> sinzui: sure.
<allenap> abentley-lunch: I've made that change to getURL. Would you be able to take another look at the mp? It's still reasonably short. The only problem is that I have to go now, but I can check back in ~2 hours.
<abentley-lunch> allenap: sure.
<allenap> Thanks!
<abentley> allenap: is it actually true that the form is POSTed to a URL with GET parameters?
<abentley> allenap: it is pretty unusual to post to a URL with a query string, although I'm not clear whether it violates the spec.
<henninge> abentley: Hi! Can you please review my branch? https://code.launchpad.net/~henninge/launchpad/bug-823164-remove-translations-by/+merge/70961
<abentley> henninge: sure.
<henninge> abentley: cool, thanks
<abentley> henninge: r=me
<henninge> abentley: thank you! ;-)
<lifeless> morning
<nigelb> Good morning.
<allenap> abentley: I don't think there is anything against POSTing to a URL with a query string. I think the earlier docstring was misleading: a query string is part of the URL, but we've become accustomed to thinking of them as GET parameters.
<abentley> allenap: I guess that makes sense.
<abentley> allenap: thanks for making the change.
<allenap> abentley: It was a good idea.
<sixstring> I'm digging into the canonical-identity-provider code. I know I'm going to need some help plugging my stuff into it. Is this the right IRC channel for asking for help with canonical-identity-provider?
<sixstring> bots?
<allenap> sixstring: I don't think many of us hack on the central SSO parts, but I can't find a better place for it. sinzui, do you know any more?
<allenap> sixstring: Email https://launchpad.net/~stuartmetcalfe; if anyone will know where the best place to hang out for c-i-p is, it's him.
<sixstring> allenap: Thanks. Will do.
<allenap> abentley: I forgot to say, thanks for the (double) review.
<abentley> allenap: no problem.
<sinzui> sixstring, allenap SSO is 100% ISD. We did not write any of the code
<sixstring> ISD?
<sixstring> Sorry, I don't know that abbrev, sinzui.
<sinzui> sixstring, https://launchpad.net/canonical-identity-provider The team that maintain the app
<sixstring> OK, sinzui. That makes sense now.
 * sixstring is still figuring out how Canonical works.
<nigelb> sinzui: hey
<sixstring> sinzui, allenap: stuart pointed me to #canonical-isd.
<nigelb> sinzui: was my post unclear that sso is a separate service?
<sinzui> nigelb, I did not see your post. I know very well it is a separate service
<nigelb> hm, wonder who commented then.
<sinzui> I advocate turning of login.launchpad.net to stop confusing people about what Lp knows about users
<nigelb> well, except there are other comlications.
<nigelb> login.u.c is more stricter in config than login.l.c
<nigelb> and there's some slight schema change
<nigelb> but, yeah. Eventually, we'll all switch.
<bdmurray> gary_poster: do you have any plans to follow up with the reporter of bug 823367?
<_mup_> Bug #823367: Natty installation failed <installer-crash> <natty> <ubiquity-2.6.10> <ubiquity (Ubuntu):New> < https://launchpad.net/bugs/823367 >
<gary_poster> bdmurray, I did not, but I can if desired.  I think I can get her email again
<bdmurray> gary_poster: well its a common error if installing behind a proxy server with a specific behavior so working around it may be complicated for her.  I'll just set to invalid rather than finding the master bug.
<gary_poster> bdmurray, ok.  If you had a page that might help her, I'd certainly be happy to pass it along.
<bdmurray> gary_poster: I don't think we have a really good workaround documented
<gary_poster> fair enough bdmurray.  OK thanks for heads up.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[#######=]:256
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 236 - 0:[#######=]:256
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #798: FIXED in 5 hr 43 min: https://lpci.wedontsleep.org/job/db-devel/798/
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 234 - 0:[#######=]:256
<wgrant> nigelb: There are no schema differences (they're the same DB), and I believe both are equally unstrict now.
<nigelb> oh. WIN.
<wgrant> lifeless: Bug #820796 didn't make it to coco, did it?
<_mup_> Bug #820796: Lock before publishing <derivation> <qa-ok> <Launchpad itself:Fix Released by jtv> < https://launchpad.net/bugs/820796 >
<lifeless> wgrant: blah
<lifeless> deploy request is still up for it
<wgrant> The logical response is to create a new Fix Release Requested status.'
<lifeless> wgrant: !cite
<wgrant> :(
<lifeless> or we could have different code bases for different services
<lifeless> I know thats heretical and all
<wgrant> You are a horrible person.
<lifeless> I'm shocked!
<lifeless> here, let me delete more code from LP for us.
<wgrant> You used ... though.
<wgrant> That increases your evil rank significantly.
<lifeless> where? oh in the bug ?
<wgrant> from ...oops import
<lifeless> oh yeah
<lifeless> well, thats cause of absolute imports needing __future__
<wgrant> Yeah.
<wgrant> Not sure why implicit relative imports ever existed.
<lifeless> python1
<wgrant> Before my time.
<lifeless> yeah
<lifeless> python started without packages :)
<sinzui> wallyworld, http://pastebin.ubuntu.com/663028/
<wallyworld> sinzui: awesome. thank you!
<sinzui> sorry about the surprise in there
<wallyworld> yea, np :-(
<lifeless> wgrant: I thought bug 824227 wasn't needed with the rendering improvements ?
<_mup_> Bug #824227: Prevent changing bugtask pillars to/from a distribution with series tasks <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/824227 >
<wgrant> lifeless: For project it's fine... for distributions it renders fine, but everything past that is more complicated.
<wgrant> lifeless: If you approve a new distroseries nomination, all distribution/distributionsourcepackage tasks get a new task.
<wgrant> If you then create a new distribution/distributionsourcepackage task, it also creates a task for all approved nominations.
<wgrant> This gets amusing if you retarget to/from a DSP task, as you'll have partial sets of tasks which are unable to be fixed.
<lifeless> those seem like bugs
<wgrant> It requires entirely reworking how distroseries nominations work.
<lifeless> which reminds me
<lifeless> I was going to mail -users and u-devel about that
<lifeless> I'd like to drop the table
<wgrant> Oh?
<lifeless> replace with a 'nominated' status
<wgrant> Yeah, probably.
<lifeless> use 'invalid' for rejected; the separate issue with making invalid be hidden would take care of hiding rejected ones
<wgrant> Particularly since nominations are restricted now, they're pretty pointless.
<lifeless> well, they have the same scope as tasks
<lifeless> (once all the known bugs are fixed I mean)
<lifeless> I also need to write up a perf tuesday this week; I'm thinking concurrency/taskswitching modelling.
<wgrant> Oh?
<lifeless> which line are you ohing?
<wgrant> Where does concurrency/taskswitching come in?
<lifeless> couple of places
<lifeless> some other canonical projects are having significant perf issues, and running hundreds of concurrent queries on single pg instances
<lifeless> not deliberately - starts with thousands of clients, mostly idle, and then stats kicks in when a spike comes through the system
<wgrant> Erm.
<wgrant> https://code.launchpad.net/~gary/launchpad/bug724025/+merge/71076
<wgrant> Doesn't that make large bugs just about uneditable?
<wgrant> Rather than simply non-AJAX.
<lifeless> heh, yes. So perhaps not good enough
#launchpad-dev 2011-08-11
<lifeless> can has review ?https://code.launchpad.net/~lifeless/python-oops/extraction/+merge/71128
<lifeless> diff is there
<lifeless> bwha, whereforeartthou, reviewer?
<j-johan-edwards> <pedantry>wherefore means why</pedantry>
<lifeless> heh
<lifeless> when going for comic effect, one ignores detail :P
 * ajmitch 28
<nigelb> 28?
<ajmitch> was meant to be /win 28
<nigelb> typing /me instead of /win sounds... fail :P
<ajmitch> where the merits of concrete were being discussed
<ajmitch> oh it was :)
<StevenK> I wonder how many times ajmitch uses /win instead of /me
<ajmitch> not often
<wallyworld> anyone know why we aren't using yui3 grids? they look nice, i have a use for them, and i would  like to include them in our yui dependencies
<wgrant> I thought 3.0 was YUI3 grids.
<wgrant> Isn't that was does the layout most modern pages?
<wgrant> eg. Product:+index?
<wallyworld> wgrant: there's a greds.css we are not including in launchpad.js http://developer.yahoo.com/yui/3/cssgrids/
<wallyworld> very easy to use grids with just a css class or two
<wallyworld> not sure what Product:+index uses
<j-johan-edwards> Does any Soyuz guru have time to look over a rough ER draft of some schema changes? ( http://www.mediafire.com/?l6rvdmn237c28ku )
<j-johan-edwards> I realize reviewers are in short supply at the moment.
<nigelb> wallyworld: aha, brilliant. You're here. I'm off work sick. Perfect day to finish off that branch.
<wgrant> j-johan-edwards: Why should this be in LP?
<wgrant> j-johan-edwards: LP is mostly full of things that shouldn't be in LP.
<wallyworld> nigelb: hello, too bad you're sick
<wgrant> Like Soyuz.
<j-johan-edwards> Indexing of application data in PPAs, and for the main archive
<j-johan-edwards> LEP Archive-Index, basically
<StevenK> nigelb: Actually sick, or you wanted to spend the day working on LP? :-P
<nigelb> StevenK: Taking a day off to work on LP would be sick. BUt I'm also actually sick.
<nigelb> If I wwanted a day off, next Monday is off anyway.
<wgrant> j-johan-edwards: So, binarypackagename shouldn't be in there.
<wgrant> j-johan-edwards: A BinaryPackageRelease contains files, while a BinaryPackageName is related to potentially hundreds of sets of files over many years.
<StevenK> lifeless: So, are we going to grow another database in the next few months?
<j-johan-edwards> wgrant: Unfortunately, applications often request a particular package name
<StevenK> j-johan-edwards: You can still link to the BPN from the BPR
<wgrant> j-johan-edwards: What's an application?
<j-johan-edwards> wgrant: So 'wesnoth-common's desktop file may be requesting 'wesnoth'
<wgrant> Ohh, so it's in addition to the BPR link, right.
<wgrant> Not as the source of the ApplicationRelease, but a dependency.
<j-johan-edwards> Yes. Usually it will just be the same package name as the release
<wgrant> j-johan-edwards: Does LP need the data broken down like this?
<wgrant> Or could we just store a JSON dict of metadata to publish?
<wgrant> I'm not sure if LP really needs to be able to inspect this much.
<j-johan-edwards> That's an interesting idea
<lifeless> StevenK: in what sense?
<wgrant> wallyworld: lib/canonical/launchpad/icing/cssgrids?
<wgrant> wallyworld: (nauseatingly included, but there)
<wallyworld> wgrant: yes. i've just updated yui-deps.py to process it, otherwise it's not in launchpad.js
<wallyworld> ah hold on
<StevenK> lifeless: In the sense that we talk to multiple databases, I guess?
<wgrant> wallyworld: How have we been using it for more than two years, then? :/
<wallyworld> wgrant: there's one in lib/canonical/launchpad/icing/yui/cssgrids
<wgrant> brb, dying.
<wallyworld> that's one one i'm talking about
<wgrant> I guess we're probably still using the external one.
<wgrant> Should delete that and move to the one in YUI?
<lifeless> StevenK: no, never.
<lifeless> StevenK: (assuming you don't just mean sharding)
<wallyworld> wgrant: the one in  lib/canonical/launchpad/icing/cssgrids is the *old* grid implementation i think
<StevenK> lifeless: So, I'm mostly parroting for wgrant
<wallyworld> the new > yui3.2 grids are much better
<lifeless> StevenK: we can grow a microservice whenever, if there is a good definition for the service
<wgrant> "whenever"
<StevenK> lifeless: "In 18 months once we have multiple DBs we can devise a migration strategy."
<wgrant> Yes.
<wgrant> And the means of having multiple DBs is by using microservices.
<lifeless> StevenK: unless and until we actually split domains out of the main service, we should put things for those domains in the main service [modulo things that we can define good microservices for, for those we should bring up the microservice and use it]
<wgrant> j-johan-edwards: It's a fair bit lighter if we basically need one table which fairly opaquely stores the contents of a .desktop file.
<wgrant> j-johan-edwards: Easier to implement initially, easier to change later.
<j-johan-edwards> wgrant: Could Launchpad handle storing a JSON file for every single application BPR? While designing the ER I was worried about excess string duplication.
<lifeless> j-johan-edwards: it would be a lot easier to sensibly discuss this if we had a LEP: a document descring the use cases and expected behaviour
<lifeless> j-johan-edwards: questions like 'do we need to search the data', 'how often does it change' and 'where will it be presented'
<wgrant> Potentially evolving ArchiveIndex into a more modern form.
<j-johan-edwards> lifeless: Actually, we already do. https://dev.launchpad.net/ArchiveIndex ;)
<lifeless> gnhh unusual place
<StevenK> lifeless: Says you, who regularly misspells wiki URLs :-P
<nigelb> wallyworld: I add a CSS class to every bug, and there was the existing work that does it for branches. How would I test this JS code?
<nigelb> Rather, what am I looking to assert
<wallyworld> nigelb: you would write a yui test
<j-johan-edwards> wgrant: Does LibraryFile transparently handle file-deduplication with checksums? That would probably prevent the disk requirements from ballooning too much
<wgrant> j-johan-edwards: It does. But it's not clear whether we'd want to use the librarian here.
<wgrant> j-johan-edwards: As lifeless suggests, I'd check over ArchiveIndex and bring it up to date.
<wallyworld>  for examplenigelb: there are several examples you could look at - anything under lp/app/javascript/xxx/tests
<lifeless> j-johan-edwards: it seems like a document containing conclusions, and there isn't enough logic there to check them.
<wallyworld> nigelb: there are several examples you could look at - anything under lp/app/javascript/xxx/tests, for example
<nigelb> ah.
<nigelb> this is nontrivial.
<lifeless> j-johan-edwards: for instance, doing a web service search api would be very different to generating an index.
<nigelb> As usual.
<wallyworld> you need to mock the io call and return some samlple data
<lifeless> j-johan-edwards: and we have the goal, for software centre, of scaling to 10's of thousands of apps all with these icons.
<lifeless> j-johan-edwards: why is this going in the archive ?
<wallyworld> nigelb: give me a few minutes and i can point you at an example if i can find a simple one
<nigelb> wallyworld: sure, let me see what I can wwrite in the meanwhile.
<j-johan-edwards> lifeless: Because Software-center doesn't have direct access to the LP databases. For the same reason that apt uses Packages.gz from the archive.
<lifeless> j-johan-edwards: those are two totally different things
<j-johan-edwards> lifeless: If we aren't indexing application data in the archive, where will we index it?
<wgrant> I see two related things here.
<wgrant> 1) SC should list applications from enabled PPAs.
<wgrant> 2) SC should be able to list commercial apps that are in PPAs without them being enabled yet.
<lifeless> j-johan-edwards: software center can consume LP as an API
<wgrant> This solves 1)
<wgrant> It does not solve 2)
<lifeless> j-johan-edwards: very very easily
<wgrant> lifeless: Bye bye LP?
<lifeless> j-johan-edwards: it already does!
<lifeless> wgrant: hmm?
<wgrant> lifeless: It only consumes LP indirectly via SCA now.
<lifeless> wgrant: right
<wgrant> The load from regular data retrieval would be... non-trivial.
<lifeless> wgrant: there are a few approaches there
<lifeless> wgrant: I really wish the LP/SCA/SC integration was clearer
<lifeless> wgrant: but thats a different discussion :)
<wgrant> Ooh yes.
<wgrant> I'm not sure it's a different discussion.
<lifeless> well
<wgrant> I'm pretty sure this depends on the outcome of that.
<wgrant> So I guess it is different.
<wgrant> But not separate.
<lifeless> if we want to help j-johan-edwards get his thing done without huge scope creep, it is a different discussion
<lifeless> its related
<j-johan-edwards> wgrant: Doesn't binary package release publishing happen seperately of BinaryPackageRelease build?
<wgrant> And if we want to get this thing done without supporting it for 5 years, we need some idea of what SC is.
<lifeless> what I'm concerned about is an integration story where we solve a problem in A by making B and C do fugly things
<j-johan-edwards> wgrant: If so LP could just refuse to return records for unpublished packages
<wallyworld> nigelb: test_personpicker has a reasonable example (we have about 4 ways of doing it in various tests sadly). search for "We patch Launchpad client to return some fake data for the patch"
<wgrant> j-johan-edwards: Refuse to return records?
<lifeless> j-johan-edwards: wgrants point is that SC needs to know about metadata for an app you have *not bought yet*
<wgrant> Oh.
<lifeless> j-johan-edwards: which is very closely related to my questions
<wgrant> I see what you're getting at now.
<wgrant> What lifeless said.
<lifeless> j-johan-edwards: putting stuff in the archive only works for archives that are available a priori, and not all all for stuff that you need to do something special to enable.
<nigelb> wallyworld: is there a commandline way to run the tests?
<lifeless> s/all all/at all/
<j-johan-edwards> lifeless: Do app-install-data-commercial packages exist as LP BPRs?
<wallyworld> nigelb: yes, but easier when just running one test case to openin  your browser the html page correspondingn to the test.js file
<j-johan-edwards> lifeless: If so I don't see the problem...
<lifeless> j-johan-edwards: not in the main archive
<lifeless> j-johan-edwards: commercial packages metadata is only for... the archive the commercial package is in
<wgrant> j-johan-edwards: They exist in LP. But they can't just be in an ArchiveIndex style index, because clients don't have those archives enabled.
<nigelb> wallyworld: okay, so as I understand, first create some sample html and let YUI loose on it to run the tests :)
<lifeless> there is a very fundamental disconnect here :)
<wallyworld> nigelb: copy an existing html and js file pair - the html file is loaded in the browser and this causes the tests to run
<nigelb> Already did that.
<lifeless> j-johan-edwards: so, there are two disjoint use cases for SC: find apps in the mirrorable-archives (ubuntu / linaro / public ppas / private team-based ppas)
<wallyworld> nigelb: more info at https://dev.launchpad.net/JavascriptUnitTesting
<lifeless> j-johan-edwards: and secondly, find apps in unlockable archives (the pay-for-an-app story)
<nigelb> wallyworld: aha, thanks!
<nigelb> After this I need to write the server side bits too.
<lifeless> j-johan-edwards: the commercial metadata package stores metadata for the unlockable archives: its a cross-archive data structure.
<j-johan-edwards> lifeless: Oh... I get it now. There's no way to provide AppInfo.gz for an archive that a non-purchaser client can't access. Is that it?
<lifeless> j-johan-edwards: *by definition* an archive [technically a suite in this case]-specific data structure won't include data for things from outside that archive
<lifeless> j-johan-edwards: right
<lifeless> look at the lep, see the example file does not suggest a url / ppa / anything - its just 'package'
<lifeless> now
<lifeless> perhaps SCA /already/ solves this ?
<wgrant> Well, there was an idea that we would not restrict the metadata. Indeed, I think it may already be unrestricted.
<wgrant> But we don't want clients having to grab it from *every* archive.
<lifeless> if SCA already solves this for commercial ppas, which we expect to dwarf the primary archive over time, it raises the question for me, of why not have SCA do it for the main archive too
<lifeless> or why not put these fields into Packages.gz directly? its where they belong AFAICT
<lifeless> (sorry that last is a jump...
<lifeless> I mean, 'if SCA handles the unlockable-PPA case, then we can think much more narrowly about the other case, and wouldn't it be easier to ...'
<j-johan-edwards> lifeless: how does LP provide private apt archives currently? `apt` usually uses publically accessible URIs to collect repo metadata...
<lifeless> j-johan-edwards: a sources.list.d entry with an https url containing a username and password
<j-johan-edwards> Okay. I see two potential solutions:
<j-johan-edwards> 1) Use permissions to allow access to only AppInfo.gz in private url repos
<j-johan-edwards> That's the dirty solution
<j-johan-edwards> 2) Come up with an API for clients to request a comprehensive view of all private PPAs and their associated metadata.
<j-johan-edwards> Which would be hard
<lifeless> j-johan-edwards: or have a search api ?
<lifeless> j-johan-edwards: but!
<lifeless> j-johan-edwards: private ppa != commercial ppa
<lifeless> we don't really distinguish them at the moment, but many private ppas would be shocked to have their existence documented, let alone a list of packages present in them exposed.
<j-johan-edwards> lifeless: Perhaps we can have a MetaArchive table...
<lifeless> j-johan-edwards: I think perhaps when you say potential solutions, you are thinking 'how can I achieve a file in the archive with this content'
<lifeless> j-johan-edwards: but! thats not the goal. Thats a way to achieve the goal. The goal is something fuzzier around software centre searches.
<j-johan-edwards> lifeless: But how can we provide a search API when we have no data structures to determine what data is and is not publically searchable?
<lifeless> j-johan-edwards: software centre agent is where said api would probably live, its what software centre talks to already
<lifeless> j-johan-edwards: I need to stop thinking about this now, or I'll be leaving a messy tree to wake up to tomorrow :) - somehow LP and SCA co-operate at the moment on purchase-a-package, so the metadata exists.
<lifeless> j-johan-edwards: we have a big open question: is this a solved problem for commercial already? If not (and i suspect not if that package contains the index) then design is needed
<jtv> StevenK: I'd like to create a new derived series on dogfoodâ¦ mind if I add one to deribuntu?
<lifeless> (not ER design, architecture design)
<LPCIBot> Project devel build #962: FAILURE in 5 hr 44 min: https://lpci.wedontsleep.org/job/devel/962/
<j-johan-edwards> lifeless: I believe the only metadata currently existing is hand-crated for app-install-data-commercial... so I don't think so
<lifeless> I suggest an email thread or conference call with (probably minimally) mvo, mpt, and michael nelson or whomever is doing SCA these days
<lifeless> I would like to participate, and we probably want one of bigjools or wgrant or stevenk as a preferred soyuz afficionado
<j-johan-edwards> On a side note, Is this chatroom logged (I think my client trashed some conversation above)
<StevenK> I believe so
<wgrant> Should be on irclogs.ubuntu.com
<j-johan-edwards> wgrant: thanks
<lifeless> wgrant: you will probably enjoy qaing r r13658
<wgrant> lifeless: Indeed.
<lifeless> 230350 |   159
<lifeless> bug, count
<wgrant> I was about to ask.
<wgrant> Thanks.
<lifeless> bug 1 has no expanders
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <GenOS:In Progress by gen-os> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux Mint:In Progress> <The Linux OS Project:In Pr
<lifeless> and does have ajax widgets
<wgrant> Hm, it should be over the AJAX  threshold.
<wgrant> Sure they're AJAX widgets and not just links?
<lifeless> ah you are right
<wgrant> So you can still get to editstatus.
<wgrant> So this is possibly not too bad.
<lifeless> anyhow, 230350 still doesn't render
<wgrant> There we go.
<wgrant> Rendered after three refreshed.
<wgrant> 800 queries, 8.61s.
<lifeless>  OOPS-2049DZ7)
<lifeless> oh
<lifeless> duh me
<wgrant> ?
<lifeless> what server was I on?
<wgrant> Hahaha
<lifeless> assignment will be undiscoverable
<lifeless> we should file a (high) bug about that, and about the UI downgrading
<wgrant> Yeah.
<wgrant> We really should just remove the expanders globally ASAP.
<wgrant> Looks much nicer.
<wgrant> lifeless: Can you check how many bugs have >=10 tasks? I guess lots will, due to nominations.
<lifeless> https://bugs.qastaging.launchpad.net/ubuntu/+source/afflib/+bug/230350?comments=all still bwarfs
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<lifeless> wgrant: 578
<wgrant> Fewer than I thought.
<lifeless> you can grab off dogfood if you want the ids - select bug, count(bug) from bugtask group by bug  having count(bug) >= 10;
<wgrant> Yeah.
<wgrant> But DF is slow.
<lifeless> ah, worked on refresh
<wgrant> And I'm not SSHed in at the moment.
<wgrant> And you are here.
<StevenK> GRRRRR. *So* tempted to make a release of storm to fix this import policy violation
<wgrant> Yes :(
<StevenK> It is beginning to *really* annoy me
<lifeless> snapshot I tihnk you mean ?we're running a (minor) fork
<lifeless> but yes, do that - or nag abel  :)
<StevenK> So lp:storm is not the right place?
<lifeless> see versions.cfg
<StevenK> Yes, but that tells me a version string, *not* the branch I should be using
<wgrant> There isn't a comment about the branch?
<wgrant> No.
<StevenK> There is not
<wgrant> Anyway, there is probably a ~launchpad or ~launchpad-committers storm branch.
<lifeless> review please? https://code.launchpad.net/~lifeless/python-oops/extraction/+merge/71128
<StevenK> wgrant: https://code.launchpad.net/~launchpad-commiters no workies, and I can't see a storm branch in https://code.launchpad.net/~launchpad
<lifeless> one sec and I'll dig it up
<wgrant> StevenK: You are unable to spell "committers"
<wgrant> https://code.launchpad.net/~launchpad-committers/storm/with-without-datetime
<lifeless> hah, there go
<lifeless> review #2 please? https://code.launchpad.net/~lifeless/launchpad/useoops/+merge/71139
<wgrant> Sorry, I've got stuff I need to finish up today :/
<lifeless> no worries
<lifeless> perhaps StevenK can take mercy? they are small branches
<StevenK> Tell you what, if you make me lunch, I'll review your branches
<lifeless> Can I eat it too ?
<lifeless> (I would if I could)
<lifeless> if you can review them after lunch, I would really appreciate that
<lifeless> speaking of which, I should have lunch myself
<StevenK> Can I just push to this storm branch, or do I need to go through the branch and review dance?
<lifeless> the best thing to do is make a fresh branch off of trunk for your change
<lifeless> put that up for review by the storm folk
<lifeless> and separately land the change into the branch we run
<lifeless> keep the delta minimal
<StevenK> It's a one line diff!
<StevenK> But I knew this was going to be a pain in the bum.
<lifeless> put it in debian/patches :P
<lifeless> FWIW I've done precisely this for several of the patches we have - and got them merged to storm
 * StevenK marks the first of lifeless's branches as "Needs Fixing"
<StevenK> Just for that :-P
<lifeless> hah, thanks
<StevenK> "The author of this branch needs better material"
<lifeless> so - go eat
<lifeless> I'll be back in ~60 myself
<jtv> StevenK: need some help with IDSâ¦ I think I'm skipping a step because I'm getting no DSDs for my new series.
<StevenK> IDS doesn't create DSDs?
<jtv> Argh.  Of course.  Well it does, but not in this case.
<StevenK> Can haz review? https://code.launchpad.net/~stevenk/launchpad/storm-394/+merge/71146
<lifeless> StevenK: 404s
<StevenK> lifeless: Yes, make dies horribly
<StevenK> Hm
<StevenK> r393's __init__.py says 0.18.0.99
<StevenK> But r394 causes make to blow up
<lifeless> make or buildout?
<StevenK> buildout, run by make
<lifeless> you have to invoke setup.py in a slightly non-default way
<StevenK> I just ran make release
<StevenK> Obviously that was silly, so let me delete the tarball from the download-cache
<wgrant> python setup.py egg_info -bsomesuffix sdist
<wgrant> I forget was suffix we normally use
<lifeless> the one in versions.cfg
<wgrant> python setup.py egg_info -b-lpwithnodatetime-r394 sdist
<lifeless> no leading hypen
<lifeless> -blpwithnodatetime-r394
<lifeless> I think
<lifeless> IMBW(tm)
<wgrant> I don't think so.
<wgrant> But we'll see soon.
<StevenK> storm-0.18.0.99-lpwithnodatetime-r394.tar.bz2
<StevenK> Looks right to me
<wgrant> I thought I usually kept the leading -.
<wgrant> Perhaps it accepts both.
<StevenK>   File "/home/steven/launchpad/lp-branches/storm-394/lib/canonical/launchpad/scripts/oops.py", line 21, in <module>
<StevenK>     from ...oops import uniquefileallocator
<StevenK> ImportError: No module named oops
<StevenK> LIFELESS!
<jtv> Hey, he does that to other people too?
<wgrant> StevenK: make
<wgrant> StevenK: Or upgrade from Python 2.4.
<jtv> Not necessarily in that order.
 * StevenK runs 'make clean && make'
<jtv> While we're here: isn't initialize-from-parent.py supposed to run InitializeDistroSeriesJobs?
<StevenK> NO
<lifeless> StevenK: yes ?
<lifeless> StevenK: we're running python2.6 ...
<jtv> Well what does run InitializeDistroSeriesJobs?
<lifeless> StevenK: do you have an oops egg in your eggs directory?
<lifeless> jobsource probably
<lifeless> jtv: ^
<StevenK> run_jobs, IIRC
<StevenK> lifeless: I have eggs/oops-0.0.1-py2.6.egg
<jtv> Ahhhâ¦ this is from before we started using utility names as section names.  That's why that didn't work earlier.
<StevenK> And after a make clean and make I still get the traceback
<jtv> Did you re-link the external source code thingamajig?
<wgrant> Unnecessary.
<lifeless> StevenK: what python version is being run ?
<lifeless> StevenK: can you do bin/py -c 'import oops'
<lifeless> StevenK: I landed the useoops branch for 0.0.1 through ec2, precisely to avoid WTF moments.
<StevenK> lifeless: That works. But http://pastebin.ubuntu.com/663179/
<lifeless> bin/py -c 'import canonical.launchpad.scripts.oops'
<lifeless> and
<lifeless> bin/py --version
<StevenK> 2.6.6
<StevenK> bin/py -c 'import canonical.launchpad.scripts.oops'
<StevenK> Traceback (most recent call last):
<StevenK> ...
<StevenK> (As before)
<lifeless> urgll
<lifeless> e
<lifeless> I wonder if its an import fascist bug
<lifeless> what happens if you do
<lifeless> bin/py -c 'import oops.uniquefileallocator; import canonical.launchpad.scripts.oops'
<StevenK> Uhhh
<StevenK> steven@liquified:~/launchpad/lp-branches/storm-394% head -n 21 lib/canonical/launchpad/scripts/oops.py | tail -n 1
<StevenK> from ...oops import uniquefileallocator
<wgrant> That's fine.
<wgrant> New in 2.6, but fine.
<wgrant> ../../../oops, basically.
<StevenK> If that is what it translates to:
<StevenK> steven@liquified:~/launchpad/lp-branches/storm-394% ls -lh ../../../oops
<StevenK> ls: cannot access ../../../oops: No such file or directory
<wgrant> I mean in the Python import sense.
<lifeless> StevenK: what does my most recent request do ?
<StevenK> lifeless: Sorry. Give the same traceback
<lifeless> ok
<lifeless> can you, before that import
<lifeless> do
<lifeless> import sys
<lifeless> print sys.modules['oops']
<lifeless> (edit the scripts/oops.py)
<StevenK> Huh?
<StevenK> sys.modules['oops'] gives a KeyError
<lifeless> even when you do bin/py -c 'import oops.uniquefileallocator; import canonical.launchpad.scripts.oops' ?
<lifeless> so, I know htis works on lucid, its where I tested it ><
<lifeless> with our PPA and all.
<StevenK> >>> print sys.modules['oops']
<StevenK> <module 'oops' from '/home/steven/launchpad/lp-sourcedeps/eggs/oops-0.0.1-py2.6.egg/oops/__init__.pyc'>
<lifeless> ok, and then the next line barfs
<StevenK> Right
<lifeless> what happens if you remove one of the .'s ? and/or add one?
 * lifeless speculates about subtle changes in python minor versions messing lifeup
<lifeless> you've got 2.6.6, surely its not a pre-rc1 build or something booong ?
<wgrant> StevenK: Is your new DSP widget meant to contain something like "ubuntu/mozilla-firefox"?
<StevenK> wgrant: That should work, yes
<wgrant> :/ I don't want it to.
<wgrant> Makes it look like you can change the distribution.
<StevenK> Why not?
<wgrant> Which is forbidden for SourcePackage tasks.
<StevenK> Curtis did that
<StevenK> lifeless: It hasn't changed?
<wgrant> I might remove the flag bit, then.
<lifeless> StevenK: what hasn't changed ?
<wgrant> (it needs to be reworked for my LaunchpadTargetWidget branch)
<lifeless> I have to go, need to grab groceries before lynne goes to pilates
<StevenK> lifeless: My python setup?
<lifeless> I will drop in later; the options I can see are: this is a bong python build (fix, move on). Switch to absolute imports (requires a __future__ statement). Switch to __import__ hackery (fugly).
<lifeless> StevenK: given that oops is new from yesterday, and the only explicit relative import in the tree, your python setup need not have chnaged to still be at fault
<StevenK> wgrant: Can haz review? https://code.launchpad.net/~stevenk/launchpad/storm-394/+merge/71151
<wgrant> Huh.
<wgrant> Just got the same oops error myself.
<wgrant> So you might not be insane.
<wgrant> StevenK: Done.
<StevenK> wgrant: Objections to just lp-landing?
<wgrant> Given what happened with the egg before, yes.
<StevenK> Bleh
 * StevenK tosses it at ec2
<wgrant> :)
<poolie> automatic branch suggestions, awesome
<jtv> StevenK: about bug 812667â¦
<_mup_> Bug #812667: https://launchpad.net/ubuntu/+series mentions derives from <derivation> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/812667 >
<jtv> â¦how about "Successor to"?
<StevenK> Hmmm
<jtv> Instead of "Derives from"?
<StevenK> Based on
<poolie> quick review of https://code.launchpad.net/~mbp/udd/819910-service-root/+merge/71154 anyone?
<jtv> StevenK: that makes the distinction from derivation a little obscure to my taste.
<jtv> "Derived from" is, to me, merely one specific form of "based on."
<StevenK> jtv: 'Successor to' doesn't really do it for me
<StevenK> Meh, 'Previous release' ?
<jtv> Makes it sound a bit as if the previous release is the previous release _of_ the distroseries.
<StevenK> Well, then, ask Julian
<jtv> Who is not currently here.
<jtv> I would say "supersedes," but I don't know if that is always the case (e.g. did etch "supersede" sid?)
<StevenK> Mu
<StevenK> sid and etch were utterly distinct, and sid will never release
<wgrant> etch also doesn't have sid as previous_series, I hope.
<jtv> That's the point: how will previous_series fit in with the various possible structures?
<jtv> Anyway, if there's nothing more specific against "successor to" than "doesn't really do it for me" then that makes it the best option so far.  Thanks.
<stub> wgrant: Did you find somewhere we are actually using WITH GRANT OPTION, or are you just being completionist?
<lifeless> StevenK: (passing through) - still having grief with ... ?
<adeuring> good morning
<wgrant> stub: I hope we're not using it.
<wgrant> stub: Just decided I should handle it, since it won't be revoked automatically.
<wgrant> stub: And you never know what might happen, given, you know, LP.
<stub> wgrant: We are not. I was just idly wondering if we should raise errors or log warnings if we see it.
<wgrant> lifeless: It's broken for me too.
<wgrant> stub: Crashing fastdowntime sounds like a bad idea.
<wgrant> lifeless: Might debug later.
<wgrant> stub: As you can see, the aclitem parsing is mildly evil, but the rest isn't too terrible.
<stub> yup. There might have been another view or function buried in information_schema or similar that might have made it easier, but how you are dealing with the raw data is fine.
<wgrant> In 2004 some functions were added to get the grantor/privs/grantee out, but they've since been removed.
<wgrant> And I couldn't find anything to map the priv chars to names.'
<wgrant> It differs between versions, but the breakage should be obvious when we try to upgrade.
<stub> This stuff is actually pretty stable (you know this as you occasionally trip over cruft).
<stub> Need a better way of handling connections that should be read only in the test suite - this magic _ro stuff is horrible.
<wgrant> stub: It is really bad, yeah.
<wgrant> Was wondering if we still needed it at all.
<wgrant> All the magic roles are slightly bad, but at least admin/read/public have obvious purposes.
<stub> wgrant: r=stub
<wgrant> stub: Thanks!
<wgrant> stub: Were the production clusters rebuilt, or just the DBs?
<wgrant> During the recent switcheroos, that is.
<stub> wgrant: IIRC we are using them to ensure that exceptions get raised if you update stuff in a slave store. I think there was an issue with just setting the connection to read_only (such as exceptions not getting raised if you rolled back? I don't recall the details)
<wgrant> Ahh.
<wgrant> So it's even more magical than I guessed.
<stub> yer, but it was also assembled for PG 8.2
<stub> Might be a better way now.
<stub> We could drop the admin role too if we ensure there is a superuser the testsuite can use.
<wgrant> Indeed.
<wgrant> It's only a little bit more evil than read, though.
<wgrant> And the evil is only 6 lines long.
<stub> Yer. Not sure if we are using read any more.
<wgrant> Isn't that what people have on staging/qastaging?
<stub> Probably using read to grant devs access to staging
<stub> yer\
<wgrant> And what LOSAs use for read-only?
<wgrant> Right.
<stub> They all log in as postgres I think
<stub> Want read only, connect to a slave.
<wgrant> Ah, of course.
<wgrant> We should probably also write tests for some of this database/ stuff eventually.
<wgrant> As it's becoming less basic.
<mrevell> Hello
<stub> Traditionally, the staging update has been the test suite. But yeah, now it is becoming more complex we could have issues non fatal and not detected on staging (eg. revoke not revoking).
<stub> Smoketest after staging rollout might be appropriate
<stub> Or splitting this out standalone with a real test suite.
<wgrant> Yes.
<wgrant> It just needs PermissionGatherer and DbSchema mocked out, and it's pretty easily testable.
<stub> Mock? You want a real db there to prove it works with particular PG releases.
<wgrant> For some stuff, yes.
<henninge> rvba: Hi! ;-)
<rvba> henninge: Hi!
<henninge> rvba: are you doing the QA for bug 798522?
<_mup_> Bug #798522: https://launchpad.net/ubuntu/oneiric/+localpackagediffs should require a comment when blacklisting a package <derivation> <qa-needstesting> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/798522 >
<rvba> henninge: OTP right now but just after the call, I'll do it.
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 234 - 0:[#######=]:256
<henninge> rvba: great, thanks! ;-)
<henninge> allenap: Hi! ;-)
<allenap> henninge: Good morning :)
<henninge> allenap: you have a bunch of revisions ready for QA ;-)
<henninge> allenap: do you need help with any of that?
<allenap> henninge: I'm looking at them now. Thanks for the offer! I think they'll be quite quick to check, but if I get stuck I'll seek you out.
<henninge> allenap: cool, thanks
<wgrant> henninge: Thanks!
<rvba> henninge: QA done :)
<henninge> rvba: cool, thanks ;)
<rvba> np
<henninge> wgrant: having my own revision way down in the queue helps as a motivator ;)
<wgrant> henninge: Ah, of course.
<wgrant> jelmer: Can you QA bug #797915? We'll hopefully be deploying up to the rev before soon, and it'd be nice to be able to have yours too.
<_mup_> Bug #797915: large bzr-svn imports fail with branch.fetch_tags enabled <code-import> <qa-needstesting> <regression> <Bazaar Subversion Plugin:Fix Released by jelmer> <Launchpad itself:Fix Committed by jelmer> <gcc-4.6 (Ubuntu):Invalid> <gcc-4.6 (Ubuntu Oneiric):Invalid> < https://launchpad.net/bugs/797915 >
<jelmer> wgrant: I started an import earlier on qastaging to see if it works now, but it'll take a while before we know
<wgrant> jelmer: Ah, great, thanks.
 * henninge was going to ask that, too. ;-)
<henninge> wgrant: I heard there was a helper now to list all the bugs fixed by a deployment on the LPS page.
<henninge> Do you know what and where that is?
<wgrant> henninge: It was on lpqateam.canonical.com, but the redesign removed it.
<henninge> oh :(
<wgrant> henninge: matsubara was going to look at replacing it, but it's not back yet.
<henninge> yes, that's where I was looking for it.
<henninge> wgrant: thanks, I'll just do it manually then
<wgrant> Not too many, fortunately, since we deployed 12 hours ago.
<henninge> yes, thanks
<henninge> jelmer: how much is "a while before we know" ? ;-)
<jelmer> henninge: my guess is about an hour
<jelmer> the bug is about large branches not importing properly, so it'll take a while before we know if they work now
<jelmer> otoh small branches still work ok, so there's no regression here
<wgrant> henninge: We could just deploy that with the next stuff tomorrow morning :)
<wgrant> Well, lifeless can.
<henninge> jelmer: would it be ok to not include your revision now or is it rather urgent to get it deployed
<henninge> if so, I am happy to wait another hour.
<jelmer> henninge: it would be nice to deploy with it, as it's a critical bug that's affecting some of the platform and Linaro folks
<henninge> jelmer: np, ping me when you are done.
<jelmer> henninge: will do, thanks
<allenap> wgrant: "Bug tasks can now be freely retargeted between products, distributions and distribution source packages." \o/
<wgrant> allenap: 10 branches later...
<allenap> wgrant: Do you know the process for landing to lp:loggerhead?
<wgrant> allenap: bzr ci
<allenap> wgrant: Ta.
<wgrant> It was going to be PQM-managed, but nobody ever set that up.
<wgrant> So just manual merge+ci.
<lifeless> wgrant: urgh, that sucks
<wgrant> lifeless: loggerhead?
<lifeless> ...
<wgrant> What sucks?
<wgrant> I was thinking you mean the loggerhead PQM thing.
<wgrant> But maybe not.
<lifeless> wgrant: the ... thing
<lifeless> wgrant: being broken for you too
<wgrant> Oh.
<wgrant> I thought you were ...ing at me.
<lifeless> no, I was ...ing at you :P
<bigjools> ...
<rvba> ?...?
<bigjools> lifeless: what are my best options right now to get a decent OOPS-like query analysis from scripts?
<lifeless> one of or systematic ?
<lifeless> s/of/off/
<lifeless> bigjools: anyhow, calling ErrorReportingUtility.handling(...) would be the easiest way
<lifeless> the dbstatements should already be hooked up for most scripts
<bigjools> lifeless: systematic would be nice, but in this case I need a one-off
<wgrant> Most scripts have librarian tracking, but not DB statement logging.
<bigjools> exactly, I want that
<wgrant> eg. process-accepted, merge proposal stuff.
<lifeless> thats a small matter of glueing in the storm tracer
<wgrant> Yup.
<wgrant> Just something to watch out for.
<lifeless> I would offer to do it for you now but 11pm + need to leave here at 8am for a hospital visit
<bigjools> lifeless: baby coming soon?
<wgrant> Bleeeergh.
<lifeless> bigjools: lib/canonical/launchpad/webapp/adapter.py has the storm tracer thats used
<wgrant> And it installs them.
<lifeless> bigjools: yeah, we're 37w5d now
<wgrant> At module load time.
<lifeless> wgrant: I was fairly sure scripts had dbstatements
<bigjools> lifeless v2 on its way :)
<lifeless> wgrant: I seem to recall checkwatches reporting like 10000 sql statements
<wgrant> It's been a while, though.
<wgrant> Needs fastdowntime.
<wgrant> lifeless: Yes, but checkwatches is the most special script we have.
<lifeless> bigjools: so step 1 - locally - add a call to getUtility(ErrorReportingUtility).handling(...)
<bigjools> I was thinking that we go to a lot of effort to optimise page loads but I've not seen much done for scripts
<lifeless> bigjools: look in the oops that is written, and check it has db statements; I think it will. IMBW.
<lifeless> add that call at the end of the script
<bigjools> ok
<bigjools> thanks, sleep well
<lifeless> it returns the oops object
<lifeless> so thats (in devel *now* but not tomorrow :P)
<lifeless> oops = ...handling(..)
<lifeless> logging.info("logged oops %s", oops.id)
<lifeless> tomorrow it becomes
<lifeless> logging.info("logged oops %s", oops['id'])
<lifeless> if I'm right, its that one line and you are set
<lifeless> each run will generate and report in its log the oops with the db queries
<lifeless> if I'm wrong, you'll also need to activate the adapter.py storm tracer
<lifeless> righto, gnight
<lifeless> good luck bigjools
<bigjools> lifeless: splendid, ta
 * StevenK peers at http://pastebin.ubuntu.com/663335/
<bigjools> what I need next is to create the report so I can see repeated queries
<bigjools> repeated/long
<rvba> matsubara: Hi, the branch which improves the js of the reports has landed.
<matsubara> rvba, cool. I'll update on devpad.
<rvba> matsubara: great, can you run the reports generation or should we wait for the cron to kick in? (I'd like to QA that soon if it's possible)
<rvba> allenap: Can I add this MP to you queue? (I've fixed the conflicts) https://code.launchpad.net/~rvb/launchpad/add-comment-bug-823777/+merge/71171
<matsubara> rvba, the branch used on devpad is db-stable. can we wait until the fix reach db-stable? I could switch but I don't know what other scripts are using that branch.
<rvba> matsubara: sure, no problem.
<matsubara> rvba, thanks. I'll let you know once it's updated
<rvba> matsubara: great.
<allenap> rvba: Sure.
<henninge> which one's correct? "you are not logged in into Launchpad" or "you are not logged in to Launchpad" or "you are not logged into Launchpad" ???
<rvba> allenap: Thanks.
<allenap> henninge: The latter I think.
<henninge> allenap: are you sure? ;-)
<allenap> henninge: The former is definitely wrong.
<henninge> allenap: ok, that helps. thanks
<henninge> this is n ot for ui, btw, just an email
<henninge> jelmer: hey, any news?
<wgrant> Looks good to me, although the svn import caused a hopefully unrelated appserver OOPS.
<henninge> wgrant: if you feel comfortable, can you please qa-ok the bug?
<wgrant> Once the OOPS syncs.
<henninge> ok
<jelmer> wgrant: It seems the import was interrupted too, which worried me
<wgrant> Slightly.
<wgrant> jelmer: But isn't that KeyboardInterrupt likely to be the workermonitor killing the worker because of the OOPS?
<wgrant> In the logs it's the other way around, but that's all that makes sense.
<jelmer> wgrant: yeah, but why would the workermonitor be oopsing?
<wgrant> jelmer: It's nearly synced...
<wgrant> 00001-00003@SQL-launchpad-main-master SELECT FeatureFlag.date_modified, FeatureFlag.flag, FeatureFlag.priority, FeatureFlag.scope, FeatureFlag."value" FROM FeatureFlag ORDER BY FeatureFlag.flag, FeatureFlag.priority DESC
<wgrant> 14224-14224@SQL-launchpad-main-master SELECT CodeImportJob.code_import, CodeImportJob.date_created, CodeImportJob.date_due, CodeImportJob.date_started, CodeImportJob.heartbeat, CodeImportJob.id, CodeImportJob.logtail, CodeImportJob.machine, CodeImportJob.ordering, CodeImportJob.requesting_user, CodeImportJob.state FROM CodeImportJob WHERE CodeImportJob.id = 7716186 LIMIT 1
<wgrant> Python sucks at threads.
<wgrant> It was a GIL timeout.
<wgrant> Nothing to be concerned about.
<jelmer> s/Python/*/
<jelmer> cool
<wgrant> If smaller imports still worked, I think we should probably just deploy.
<wgrant> We could retry this, but it takes forever and seemed to be happy enough.
<wgrant> (the call that failed was just updating the logtail)
<wgrant> So it was indeed before the job died.
<wgrant> So probably triggered the workermonitor to kill the job.
<jelmer> great - the job was doing a lot better overall anyway
<wgrant> qa-ok it is.
<wgrant> henninge: ^^
<wgrant> Hmm.
<wgrant> I bet gary_poster was QAing his bug at the time that OOPS happened.
<wgrant> Let's see.
<wgrant> :( no.
<wgrant> Something else must have been murdering the appserver.
<jelmer> I'm having problems getting through to the appserver now
<jelmer> (the usual "Sorry, there was a problem connecting to the Launchpad server. ")
<henninge> wgrant, jelmer: very cool, thanks! ;-)
<wgrant> gary_poster: FWIW, that huge bug (230550 or whatever it is) was consistently rendering on qastaging in 8.6s earlier.
<wgrant> So it can render when things aren't broken.
<wgrant> But only just.
<gary_poster> wgrant, oh ok
<gary_poster> wgrant, thanks for letting me know.
<gary_poster> are you all discussing a problem we need to jump on?  qastaging is loading fine for me right now
<gary_poster> as is launchpad.net
<gary_poster> allenap, https://code.launchpad.net/~gary/launchpad/profiler/+merge/71192 is ready if you are.  The diff is 907 lines, though.
<allenap> gary_poster: I'll give it a go. Nearly done with my current one.
<gary_poster> thanks allenap
<wgrant> gary_poster: staging's been blocking for 14s and otherwise timing out a bit. Certainly not something that needs to be jumped on -- it's staging.
<gary_poster> wgrant, ok.  Thanks for the explanation.
<mwhudson> jelmer: re https://answers.launchpad.net/launchpad/+question/167471, last time i looked the svn caches compresses veeeery well :)
<deryck> Morning, all.
<jelmer> mwhudson: It'd be annoying to compress/uncompress it every time we're checking for new revisions in e.g. a small branch in the apache repo
<mwhudson> jelmer: yes, i guess so
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugs: 234 - 0:[#######=]:256
<mwhudson> jelmer: a cache backend that supports compression?
<mwhudson> not sure what that means
<jcsackett> allenap: can i throw https://code.launchpad.net/~jcsackett/launchpad/fix-picker-entry-adapters/+merge/71204 into your queue?
<allenap> jcsackett: Sure. I'm doing a big review for Gary right now, but yours can be next.
<jcsackett> allenap: awesome, thanks. :-)
<jelmer> mwhudson: the plan is still to use bzr pack files to store cache data, but that requires some refactoring of the pack code
<mwhudson> jelmer: ah yes, that would be great i suspect
<LPCIBot> Project devel build #963: STILL FAILING in 4 hr 52 min: https://lpci.wedontsleep.org/job/devel/963/
<jelmer> mwhudson: btw, I've now updated https://code.launchpad.net/~jelmer/launchpad/bzr-code-imports/+merge/70684 to use the safe branch opener from the puller. If you have a chance to have another look, that'd be great.
<jelmer> the safe branch opener has been extracted from BranchMirrorer and merged with the one from bzrutils
<mwhudson> jelmer: i saw that change go by (happily)
<rvba> allenap: Thanks for the review! I've followed your suggestions ... and this is the end of this week's work on this Javascript code: \o/.
<allenap> rvba: Awesome, I too shall \o/
<rvba> allenap: we even shall _o/\o_
<allenap> Cool :)
<allenap> gary_poster: I'm having trouble opening callgrind files with kcachegrind. It says "Check it exists and you have enough permissions to read it."
<allenap> Fwiw, I have checked the permissions :) Have you seen that problem before?
<gary_poster> allenap, no...
<gary_poster> allenap, I've used files from qastaging.  Let me see if I can dupe with locally produced files.  I haven't changed the internal aspect of how those files are created, but I've touched the periphery, so maybe I messed something up.
<gary_poster> (allenap, I've been using pstats stuff for my actual analysis locally, fwiw)
<gary_poster> allenap, wfm
<allenap> gary_poster: Okay, maybe it's a problem with kcachegrind. Can you try with https://devpad.canonical.com/~gavin/callgrind.out.2011-08-11_15:05:50.484-RootObject:index.html-OOPS-2049X3-Dummy-6
<allenap> ?
<gary_poster> sure allenap, trying
<gary_poster> allenap, fails for me too.  Looking at file, not that I know much about the syntax...
<gary_poster> allenap, look at the file :-)
<gary_poster> allenap, oh, no, my download error.  lemme try again
<henninge> adeuring: did you get my sms
<henninge> ?
<adeuring> henninge: no... reminds me that I should update my mobile phone number in the directtory...
<henninge> adeuring: I used the one I had on my phone - probably the old one.
<henninge> deryck: Hi, sorry I missed the call! ;)
<adeuring> henninge: right
<deryck> henninge: no worries.  adeuring warned you might be away.
<sinzui> bigjools, ping
<bigjools> sinzui: you punged
<gary_poster> allenap, I got the right file this time.  The problem is that it is a pstats file, not a callgrind file.  You can verify by using python -m pstats <file name> and then "stats 40" or something.  This is a bug in my branch.  Fixing it ought to be easy, but I'm not sure at all how to test it.Mm, I have an idea on something that is barely ok.  I'll ping you when I have something in.  Thanks for catching this, and sorry yo
<sinzui> bigjools, I am struggling to remember the name of the method that can check if a SPN or DSP was ever published. You mentioned it in a bug about telling users reporting bug that the package dies not exist.
<bigjools> sinzui: IArchive.getPublishedSource()
<sinzui> thanks
<bigjools> np.  That's probably the most important soyuz method to know about. :)
<allenap> gary_poster: Ah, interesting. I'm glad I bumped into it then :)
<gary_poster> :-)
<gary_poster> allenap, fixed: http://pastebin.ubuntu.com/663445/
<gary_poster> (change pushed)
<allenap> gary_poster: Cool.
<jcsackett> allenap: sinzui looked at my branch, so don't worry about it.
<allenap> jcsackett: Cool, okay.
<gary_poster> matsubara, hey.  could you make https://launchpad.net/lp-devops-dashboard maintained by ~launchpad please?
<matsubara> gary_poster, done
<gary_poster> Thanks matsubara :-)
<matsubara> np
<gary_poster> thank you allenap!  for #1, answer is because I was working from an original source that did it that way.  It is convenient for subclasses that want to share the lock, which we were doing, but we're not doing that anymore with this branch.  I'll simplify as you suggest.  I'll also do your #2: good idea.  Thanks again
<allenap> gary_poster: Cool, and you're welcome.
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 234 - 0:[#######=]:256
<matsubara> bigjools, hi
<bigjools> matsubara: hey
<matsubara> bigjools, you said in a reply to the dashboard email: "Possibly make it show more pending revisions like the deployment report does". What did you mean?
<matsubara> bigjools, taking https://devpad.canonical.com/~lpqateam/qa_reports/deployment-db-stable.html as an example, what info there you'd like to see in the dashboard?
<bigjools> matsubara: it only shows one; it would be nicer if it showed more, if not all
<matsubara> bigjools, you mean all the needstesting ones or everything after the blocking revision?
<bigjools> matsubara: mainly the needs testing ones
<bigjools> hopefully it won't be too big a list since we're all awesome at doing QA
<matsubara> bigjools, ok, thanks for the info. I'm filing bugs from your feedback email.
<bigjools> matsubara: cool, thanks for taking it on board
<matsubara> you're welcome
<benji> jcsackett: when you get a chance, I have a little JS branch for you: https://code.launchpad.net/~benji/launchpad/bug-823321/+merge/71229
<jcsackett> benji: looking at it now.
<benji> cool
<jcsackett> benji: i'm a little confused about the logic of always doing checks like (user_has_search === automated_search). given your stated goal, isn't it clearer to just check if user_has_searched is true? or is there something here i'm missing (always very possible)?
<benji> jcsackett: if user_has_searched and *this* result is from the user searching (!automated) then we want to display the results, that's why we have to check both
<jcsackett> benji: ah, okay.
<sinzui> jcsackett, do you have time to mumble?
<jcsackett> sinzui: sure.
<jcsackett> sinzui: hm. mumble will not start for me. give me a few moments.
<sinzui> okay
 * deryck goes offline for lunch, back very soon
<sixstring> sinzui: hey, I just saw pocketlint. pretty cool.
<henninge> gary_poster: Hi!
<sinzui> sixstring, thanks. Beware of js linting on Oneric. seed segfaults. I may switch to a different js backend if there is no commitment to support it
<abentley_> sinzui: I would like SourcePackageNameSet.new to raise InvalidName if the name is invalid.  Also, I'd move InvalidName to registry.errors.  Thoughts?
<gary_poster> henninge, hey.  was lunching, and may need to take half day
<jcsackett> benji: sorry i forgot to update the MP and tell you earlier. r=me on that js branch.
<benji> jcsackett: no problem; thanks
<LPCIBot> Project db-devel build #800: FAILURE in 6 hr 58 min: https://lpci.wedontsleep.org/job/db-devel/800/
<mtaylor> lifeless: ping
<thumper> hey mtaylor
<mtaylor> hey thumper !
<mtaylor> thumper: I was just brainstorming about whether or not we could make linked branches in bugs point to external branches or not
<thumper> mtaylor: not very easily
<thumper> mtaylor: the link points to a branch id
<mtaylor> thumper: ah.
<thumper> although...
<thumper> I believe there is still the concept of a remote branch
<thumper> but we were wanting to kill that
<mtaylor> thumper: I was asking because we were hacking it in bug comments right now: https://bugs.launchpad.net/openstack-ci/+bug/809479
<_mup_> Bug #809479: gerrit integration test <OpenStack Core Infrastructure:Fix Committed> < https://launchpad.net/bugs/809479 >
<mtaylor> thumper: well, and I'd _thought_ about having something that would make an imported branch and then link to that - but that feels _REALLY_ heavy
<thumper> mtaylor: doable
<mtaylor> thumper: I mean, right now I'm just having idle imaginings... I think as we do more with gerrit we'll need to understand what the workflow _actually_ is and what is helpful
<LPCIBot> Project devel build #964: STILL FAILING in 7 hr 19 min: https://lpci.wedontsleep.org/job/devel/964/
<wallyworld_> sinzui_: we are here
<wallyworld_> sinzui_: we can here you
<jcsackett> sinzui: we both heard you.
<wallyworld_> hear
<jcsackett> we've been chatting.
<sinzui> looks like I had network issues
<StevenK> http://pastebin.ubuntu.com/663335/
#launchpad-dev 2011-08-12
<lifeless> mtaylor: hi ?
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 234 - 0:[#######=]:256
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 238 - 0:[#######=]:256
<mtaylor> hi lifeless
<lifeless> mtaylor: what can I do for yo u?
<mtaylor> lifeless: well, I was going to chat with you about the possibility of having branch links in bugs be able to point to external branches
<mtaylor> lifeless: talked with thumper about it a little bit
<mtaylor> lifeless: he indicated that it would not be trivial, which I kindof figured
<lifeless> well
<lifeless> so the key thing is that we currently have two sorts of remote branch
<lifeless> 'imports' and 'mirrors'
<lifeless> jelmer is -right now- nuking 'mirrors' so we will only have 'imports', which includs just-copy-a-bzr-branch
<lifeless> I don't think we want 'a branch we know about but cannot read the data from'
<mtaylor> are imports one-time imports?
<lifeless> so it should be trivial - register the branch from your gerrit integration scripts, and reference it thereafter
<lifeless> mtaylor: no
<mtaylor> ok. cool
<lifeless> mtaylor: but tuning the import system to deal with branches that get deleted, and or have a way to massively dial back the probing when a branch goes quiet - those are good things in their own right
<mtaylor> yes. I imagine that they are - also probably dealing with rebases of those too, if you don't already
<lifeless> I need to pop out for a quick grocery trip; will respond to anything more upon my return :)
<mtaylor> k. I'm good for now
 * wallyworld_ has to go and pick up dog. back soon
<lifeless> rebasing should be fine
<lifeless> -> gone
<lifeless> back
<lifeless> mtaylor: oh there is something
<mtaylor> lifeless: mmyesss?
<lifeless> mtaylor: can you pull in the fd fix for the jenkins bazaar plugin ? I haven't gotten setup on git for that yet...
<lifeless> bazaar-plugin/pull/6
<lifeless> https://github.com/jenkinsci/bazaar-plugin/pull/6
<mtaylor> lifeless: it's already merged - I just need to release it
<lifeless> mtaylor: ... Do....It....Please...?
<mtaylor> lifeless: a) do you need me to give you push access on that? and b) have you tested the tree sufficiently so that you are satified with me releasing it?
<lifeless> mtaylor: there are a couple others there too I think. Or maybe I'm thinking of platformlabeler
<mtaylor> lifeless: I also added lightweight checkout support in the tree ...
<mtaylor> but haven't tested extensively
<lifeless> mtaylor: I don't know if I do/don't have push access there. If I don't, I think I would probably be more useful with it...
<lifeless> rbtcollins on github
<spm> rbtcollins ?? Robert ... Bloody Terrific Collins?
<lifeless> yes, yes indeed
<LPCIBot> Project db-devel build #801: STILL FAILING in 6 hr 12 min: https://lpci.wedontsleep.org/job/db-devel/801/
<LPCIBot> Project devel build #965: STILL FAILING in 6 hr 29 min: https://lpci.wedontsleep.org/job/devel/965/
<StevenK> lifeless: Still getting that oops rubbish
<StevenK> lifeless: And wgrant hit it last night too
<StevenK> However, my Storm release hit devel and it's stopped whinging about that at least
<lifeless> cool
<lifeless> the oops thing is bizarre
<lifeless> it should be temporary - I hope to have that oops stuff fully removed earliy next week
<StevenK> lifeless: Can haz DB patch number?
<lifeless> self service!
<StevenK> Eh?
<StevenK> I can do that?
<lifeless> https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess
<lifeless> you might like to refresh your memory, cause it has been changed ;)
<lifeless> (and announced to tl's who were meant to point everyone at it :P)
<lifeless> https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#Patch ids
<lifeless> is the specific bit you need
<StevenK> lifeless: Oh, so, should I add some indinces to this patch?
<StevenK> lifeless: (I'm adding SPN and BPN to SPPH and BPPH)
<lifeless> you'll need two patches
<lifeless> see https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#Background
<lifeless> (yes, we will almost certainly need some indices :P)
<StevenK> I see that I can't do ALTER TABLE live
<lifeless> adding the columns is a cold patch
<lifeless> it will go in fastdowntime
<lifeless> adding the indices is hot, so will get deployed after the downtime
<StevenK> So, I want -1 or something with the ADD COLUMN patch?
<lifeless> https://dev.launchpad.net/Database/LivePatching has various bits of gory detail, but is still a work in progress
<lifeless> yup, exactly.
<lifeless> then we deploy that, then a garbo job to migrate
<StevenK> Yes
<lifeless> (ok, I'm telling you how to suck eggs... sorry :))
<StevenK> I'll worry about indexes later
<lifeless> yup
<lifeless> meep
<lifeless> testPartnerUploadToNonReleaseOrProposedPocket
<lifeless> is not generating an oops for me ><
<StevenK> Then you broke it
<StevenK> :-P
<lifeless> prethumably, but I have no idea how
<lifeless> I hate how it fails by telling you a completely unrelated oops happened
<lifeless> StevenK: this is for my useoops branch - ec2 found some tests [just tests so far] that I'd missed updating
<StevenK> lifeless: https://code.launchpad.net/~stevenk/launchpad/denorm-bspph/+merge/71303
<lifeless> we didn't test spph - things seemed fast on the source side anyhow
<lifeless> (do we need it?)
<wgrant> It would be handy for overrides.
<wallyworld_> StevenK: do you need to specify that those int columns are foreign keys?
<StevenK> *shrug* It could go either way
<wallyworld_> StevenK: imho, they should be specified as foreign keys for referential integrity but i'm sure a dba can provide definitive input
<wgrant> No reason for them not to be foreign keys, and they should be.
<wallyworld_> StevenK: i won't claim the review since there's already 2 highly qualified people on the review list :-)
<lifeless> wallyworld_: you couldn't be authoritative anyhow
<lifeless> wallyworld_: db patches have a different review rule
<lifeless> https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#Reviews
<lifeless> yes, they should be FK's
<StevenK> lifeless: Done. So I need to wait until Monday?
<StevenK> I suppose it can't land until FDT anyway
<lifeless> yes
<lifeless> you can land it when its reviewed (on db-devel as all schema patches do)
<lifeless> we'll deploy it as soon as we can
<StevenK> And then how it does flow back into devel?
<lifeless> depends
<lifeless> if things are merged up, cherrypick merge.
<lifeless> bah
<lifeless> are *not* merged up
<lifeless> cherrypick merge
<lifeless> if they are merged up, a regular merge
<lifeless> to be merged up, all that is on db-devel before the thing being deployed must be also deployed.
<jelmer> g'morning
<spm> heya jelmer
<lifeless> ok, tests fixed. \o/
<lifeless> StevenK: want to do an incremental review of test fixups ? I would say 'no' if I were you ;)
<StevenK> Fine, no :-P
<lifeless> woot, lp-land time!
<StevenK> Ask wallyworld_ :-P
<lifeless> meh, its all the same mechanical stuff you saw before
<lifeless> with some do-it-the-easy way stuff for extra credit
<lifeless> (actually, I'm ec2 landing, cause i don't want a broken trunk on friday evening, but bah)
<lifeless> argh, I want bandwidth. sob.
<lifeless> Using saved parent location: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/
<lifeless>    983kB     2kB/s / Fetching revisions:Inserting stream:Estimate 688/1174
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[#######=]:256
<adeuring> good morning
<jelmer> hi adeuring
<adeuring> hi jelmer!
* adeuring changed the topic of #launchpad-dev to: Das Thema fÃ¼r #launchpad-dev ist: https://dev.launchpad.net/ | On call reviewer: adeuring | Critical bugs: 238 - 0:[#######=]:256
<wgrant> 2011-08-11 10:38:53 WARNING public.spn_lookup not in calculated lpmain replication set. Update *_SEED in replication/helpers.py
<wgrant> lifeless: ^^ is that you?
<wgrant> lifeless: Caused the staging full-update to blow up, although the appserver came back OK.
<LPCIBot> Project db-devel build #802: STILL FAILING in 5 hr 17 min: https://lpci.wedontsleep.org/job/db-devel/802/
<StevenK> That would be lifeless, yes
<lifeless> wgrant: heh, ><
<lifeless> wgrant: today?
<lifeless> that temp table had hung around
<wgrant> lifeless: Around 10am.
<wgrant> Maybe more 11am.
<wgrant> My time.
<lifeless> dropped it now
<wgrant> Thanks.
<wgrant> We'll see in 16 hours if it worked.
<wgrant> Really need to turn off those jobs.
<wgrant> The topic is a bit more German than usual.
<jelmer> :-)
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #803: FIXED in 16 min: https://lpci.wedontsleep.org/job/db-devel/803/
<StevenK> WHAT?
 * StevenK kicks Jenkins. In the face.
<lifeless> StevenK: \o/
<StevenK> successful: lp.buildmaster.tests.test_manager.TestBuilddManagerScript.testBuilddManagerRuns
<StevenK> Killed
<StevenK> Hmmmmm
<StevenK> [67994.667699] Out of memory: kill process 13032 (xvfb-run) score 198282 or a child
<StevenK> Whee
<StevenK> Not much I can do about the OOM killer
<wgrant> StevenK: How big is the instance?
<mrevell> Morning
<StevenK> c1.xlarge, 2Gb of RAM
<wgrant> jelmer: Did you see the odd failure on https://code.launchpad.net/~doko/gcc/4.6?
<jelmer> wgrant, the database one?
<wgrant> http://launchpadlibrarian.net/77073020/doko-gcc-4.6.log
<wgrant> Says it took an hour, but there's only two minutes and a timeout in the log.
<jelmer> hmm, that's odd indeed
<bigjools> StevenK, wgrant: did one of you upgrade the schema on DF?
<wgrant> Not I.
<bigjools> I wanted to do my new patch myself so I could note how long it took.  I guess I can;t now.  >:(
<wgrant> DF is invalid. Use staging.
<wgrant> Patch number?
<bigjools> well also if staging actually told me how long it took that'd be nice
<bigjools> DF is not invalid.  It's always slower.
<bigjools> 80-1
<bigjools> ah it's in the l-d-r table
<wgrant> It is.
<bigjools> forgot about that, I was hunting logs
<wgrant> It's also not on staging yet.
<wgrant> Will be in 10ish hours.
<wgrant> The last few updates failed because lifeless was a bad person.
<bigjools> it is on staging
<wgrant> Huh.
<bigjools> was on staging yesterday
<wgrant> Oh, right, it applies the patches before it crashes.
<wgrant> Just doesn't print the times.
<bigjools> right
<bigjools> which is.... not  useful
<wgrant> But you can still get the timing from there, since you have access.
<bigjools> yes, but most people can't
<bigjools> "Bad magic number" ... I hate pyc files
<lifeless> wgrant: was not
<lifeless> wgrant: the script considers *temporary* tables
<lifeless> wgrant: thats a bug :)
<wgrant> lifeless: That was a temp table?
<wgrant> Oh, right, I remember we had this problem a while ago.
<wgrant> Guess we should check relistemp.
<lifeless> wgrant: yes, 'create temp table spn...'
<wgrant> Bah, oops fail.
<wgrant> Ah, no, unrelated.
<wgrant> lifeless: Ah, temp tables were fixed in security.py, but replication.helpers says this:
<wgrant>     # Ignore any tables and sequences starting with temp_. These are
<wgrant>     # transient and not to be replicated per Bug #778338.
<_mup_> Bug #778338: restartable multitablecopier interacts badly with replication and upgrade operations <qa-untestable> <Launchpad itself:Fix Released by stub> < https://launchpad.net/bugs/778338 >
<wgrant> It doesn't check properly.
<lifeless> yeah
<lifeless> win
<wgrant> Should probably use the same pg_table_is_visible check that security.py has.
<gmb> allenap: Is bug 812583 QAable?
<_mup_> Bug #812583: IndexError in add_template_values in loggerhead annotate <loggerhead> <oops> <qa-needstesting> <Launchpad itself:Fix Committed by allenap> <loggerhead:Fix Committed by pr0gg3d> < https://launchpad.net/bugs/812583 >
 * allenap looks
<allenap> gmb: Yeah, probably :) I'll give it a go now.
<gmb> allenap: Thanks
<LPCIBot> Project devel build #966: STILL FAILING in 6 hr 2 min: https://lpci.wedontsleep.org/job/devel/966/
<allenap> gmb: Do you know how I get a branch into qastaging?
<StevenK> allenap: You beg a LOSA
<gmb> allenap: Sodomy non sapiens, I'm afraid.
<allenap> FFS.
<StevenK> Or land it via stable :-)
<allenap> StevenK: Ah, oops, I was unclear. I mean, I want a branch to be available in codehosting in qastaging.
<StevenK> Oh, that use.
<gmb> allenap: Also, does bug 796669 actually need QAing since it's a testing-related issue?
<_mup_> Bug #796669: The lp.registry.distroseriesdifferences_details module (javascript) is very hard to test. <derivation> <qa-needstesting> <tech-debt> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/796669 >
<allenap> rvba: ^
<rvba> allenap: gmb I'm on it.
<gmb> allenap, rvba: Ah, sorry, yes. I forgot that "person who filed the bug" != "person who fixed it"
<allenap> Hehe.
<danilos> matsubara, hi, I find that OOPS from bug 690568 doesn't really match up well with what the bug describes â do you have any idea what might be going on?
<allenap> Does anyone know how I can create a revision without a commit message? bzr forbids.
<StevenK> allenap: -m " " ?
<allenap> StevenK: I needed a properly empty revision for QA.
<allenap> s/revision/message/
<allenap> StevenK: I've done it now; wgrant suggesting bzrlib directly, which was quite straightforward with the help of Google.
<StevenK> allenap: Indeed, I saw, since IRC is good like that. :-)
<bigjools> StevenK, wgrant: IArchive.getPublishedSources() doesn't use a default status.  This seems dangerous to me, I am tempted to make it default to PUBLISHED.  Thoughts?
<danilos> adeuring, hi, I've got a short branch up for review https://code.launchpad.net/~danilo/launchpad/bug-728220/+merge/71343
<wgrant> bigjools: PUBLISHED is dangerous.
<wgrant> bigjools: [PUBLISHED, PENDING] is necessary but dangerous unless you're sure that you're expecting it.
<wgrant> Nothing is safe.
<wgrant> So there is no sane default.
<bigjools> well with a method called getPublishedSources, you'd expect published sources, no? :)
<bigjools> pending is assumed to be returned anyway
<wgrant> Published, published, or published?
<wgrant> There are three definitions.
<wgrant> Currently published.
<wgrant> Published.
<wgrant> And having been published.
<bigjools> I don't expect superseded sources
<bigjools> that's the dangerous part
<wgrant> Let me guess.
<bigjools> heh
<wgrant> You initialised a distroseries with the parents' entire history?
<bigjools> define "you"
<bigjools> but yes
<wgrant> Heh.
<bigjools> and does explain the IDS script memory usage
<bigjools> to some extent anyway
<wgrant> Not entirely, but it goes some way.
<wgrant> Right.
<bigjools> I still think defaulting to (pending, published) is right
<wgrant> Possibly.
<bigjools> casual users will get a big surprise
<adeuring> danilos: I'll look
<danilos> adeuring, thanks
* bac changed the topic of #launchpad-dev to: Das Thema fÃ¼r #launchpad-dev ist: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugs: 238 - 0:[#######=]:256
<bac> hi adeuring -- how goes the battle?
<adeuring> morning bac, I must admit that I did not do many reviews this morning
<wgrant> henninge-lunch: Considering a deployment today?
<jelmer> wow, the deployments are daily now?
<wgrant> jelmer: We've been doing around 1.5 a day lately.
<jelmer> that's really exciting.
<wgrant> We can even deploy codehosting without downtime now, although it's slow so we don't do it normally.
<wgrant> This leaves librarian, ftpmaster and poppy as the only downtime services.
<wgrant> And librarian will be fixed in a week or two.
<wgrant> And we can fix ftpmaster now.
<wgrant> So we are just about fully nodowntime.
<adeuring> danilos: the branch looks good. Just one question: did you consider to treat something like SELECT 'a''b'? Not really important to fix the bug, I assume, but anyway...
<bac> adeuring: i'm starting on jtv's branch
<adeuring> bac: ok
<danilos> adeuring, I didn't bother spending too much time considering the queries in OOPSes are all very "standard"
<danilos> adeuring, I was also thinking about escaped quotes, but they can't show up in what we care about most, so no big deal
<adeuring> danilos: right, so r=me.
<danilos> adeuring, thanks
<danilos> bigjools, wgrant: how do I make a PPA private on staging? (I want to do it for https://qastaging.launchpad.net/~danilo/+archive/epiphany so I can get a fresh OOPS for bug 690568)
<henninge> wgrant: had not thought of that yet but sure ;-)
<bigjools> danilos: you need to be a commercial admin and you need an empty PPA
<henninge> wgrant: requested ;)
<bigjools> I can do it for you if you give me an empty one
<danilos> bigjools, oh, so a new one needs to be created? sure thing, just a minute
<danilos> bigjools, https://qastaging.launchpad.net/~danilo/+archive/empty
<bigjools> danilos: done
<danilos> bigjools, thanks
<StevenK> adeuring, bac: Which of you two wants to review a massive-oversized mostly-scripted branch?
<bac> StevenK: wow, you're a salesman at heart
<StevenK> Haha
<adeuring> StevenK: I'll look
<bac> StevenK: i'm in the middle of a mildly oversized branch ATM
<adeuring> but let me finish another review first
<StevenK> adeuring: I will owe you at least one beer, possibly more. It's 4,701 lines.
<bac> StevenK: so you'll be next whichever one of us getst there first
 * adeuring runs awy
<adeuring> nah, it's fine
<bac> StevenK: that is 6 times the limit, so i think you'd owe him at least five beers
<StevenK> Haha
<StevenK> It's only 5.9 times the limit, so 4.9 :-P
<gmb> wgrant: If you're still around: is your fix for bug 824368 QAable?
<_mup_> Bug #824368: security.py takes 6 seconds per DB <qa-needstesting> <Launchpad itself:Fix Committed by wgrant> < https://launchpad.net/bugs/824368 >
<wgrant> gmb: The next (not current) staging update will do it.
<gmb> wgrant: Okay, thanks.
<wgrant> But it's on db-devel, so nobody should care...
<gmb> Ah, fair point.
<gmb> We should have a qa-yeahwhatever tag.
<StevenK> You tag it like that and see what the tagger does.
<StevenK> It would be amusing.
<danilos> wow, we've got 73 bugs that are fix committed in launchpad-project
<danilos> (well, bug tasks, but still)
<gmb> danilos: Indeed. I'm trying to drive down the QA list to the point where we can roll a bunch of those out.
<bigjools> did someone break devel?
<matsubara> danilos, for some time we had a duplicationa on oops prefixes and then got oopses with the same oops id
<bigjools> s/devel/db-devel/
<danilos> bigjools, I believe check-db-revisions is not running anymore, try 'make schema' first
<danilos> matsubara, oh, so you think this is just an old one (i.e. no need to worry about them still being overwritten like this)?
<matsubara> danilos, I'll double check, but I think it's been fixed and this is one of the old cases
<danilos> matsubara, excellent, thanks
<bigjools> danilos: it failed in ec2
<bigjools> but thanks for reminding me about that :)
<LPCIBot> Project devel build #967: STILL FAILING in 4 hr 5 min: https://lpci.wedontsleep.org/job/devel/967/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #968: FIXED in 12 min: https://lpci.wedontsleep.org/job/devel/968/
<abentley> adeuring or bac: could you please review https://code.launchpad.net/~abentley/launchpad/push-creates-package/+merge/71366 ?
<adeuring> abentley: I need to finish another review; when that is finished, I'll have a look
<abentley> adeuring: thanks.
<matsubara> danilos, just checked and we have only one config with oops_prefix: C in the configs branch. If I remember correctly that's the generic prefix used and have been changed to the REPORTIFSEEN on loganberry
<matsubara> if you spot an oops redirecting to the wrong oops report, let me know
<bac> adeuring: i can do abentley's review unless you've already started
<adeuring> bac: I was just now going to start it :) I'll do it
<bac> adeuring: ok
<rvba> matsubara: Hi, looks like my branch (about improving the JS of the reports) made it into db-stable. Just to give you a heads up :).
<matsubara> thanks rvba
<rvba> np
<matsubara> rvba, the ppr branch is now on r10890.
<rvba> matsubara: great, do you know how often the report generation script runs?
<matsubara> rvba, we have a couple of lines in the crontab. did you change only the ppr or the ppr and the dbr?
<matsubara> in any case here's the crontab lines:
<matsubara> rvba, https://pastebin.canonical.com/51172/
<rvba> Only the ppr.
<matsubara> so basically, once a day for the ppr and more often for the dbr
<matsubara> I'll kick a run of the ppr script now so you can try things. give a sec
<rvba> Thanks a lot!
<adeuring> abentley: overall, a nice branch. The only thing that puzzled me was that you had to add the valid_name() call to SourcePackageNameSet.new(). I understand that it is necessary -- but where and how was the validity check done before?
<abentley> adeuring: There's a check constraint on the database, but I wanted to get a more specific exception than the database one.
<adeuring> abentley: ah, ok. thanks. So, no conflicts, I assume. r=me
<abentley> adeuring: thanks.
<matsubara> rvba, hey, I didn't forget about you. script is still running. I'll ping you when it finishes
<rvba> thanks matsubara
<cjwatson> adeuring,bac: can either of you review https://code.launchpad.net/~cjwatson/meta-lp-deps/new-apt/+merge/71386 ?
<adeuring> cjwatson: sure, I'll look
<cjwatson> I don't know the process for actually getting that into the PPA, although it seems to involve recipes
<adeuring> cjwatson: the diff against devel is huge...
<adeuring> cjwatson: never mind
<adeuring> cjwatson: r=me.
<cjwatson> adeuring: thanks.  can you land it?  I don't have access
<adeuring> cjwatson: sure
<cjwatson> (it'll need to have UNRELEASED replaced with the appropriate series in debian/changelog, of course)
<cjwatson> (if you'd like me to do that, tell me which series to use)
<bigjools> allenap: ta for the tracer advice
<bigjools> I guessed it might be pebkac
<allenap> bigjools: That wasn't pebkac. It's hairy code in there; the stuff in checkwatches took a long time to get right.
<bigjools> allenap: that points to a bigger problem then
<adeuring> cjwatson: i think natty would appropriate
<allenap> bigjools: Indeed. But I am not going down there until our feet start punching holes in the floorboards.
<bigjools> allenap: or fists in walls :)
<allenap> Hehe :)
<danilos> adeuring, bac: I've put a branch up for review, I'm leaving but would appreciate a review: https://code.launchpad.net/~danilo/launchpad/bug-690568/+merge/71394
<danilos> thanks :)
<bigjools> allenap: hah, it's timing out the txn now of course.
<adeuring> danilos: I'm quite close to EOD/EOW too ;)
 * allenap experiences schadenfreude
* adeuring changed the topic of #launchpad-dev to: Das Thema fÃ¼r #launchpad-dev ist: https://dev.launchpad.net/ | On call reviewer: bac | Critical bugs: 238 - 0:[#######=]:256
<mtaylor> how come I have to have direct team membership to have PPA access?
<mtaylor> I'd think that being a member of a team that owned a team would give me access to the PPA of that team
<mtaylor> team team team
<bigjools> mtaylor: that should work, what error are you getting?
<bigjools> allenap: yay, trace at last
<mtaylor> bigjools: Signer has no upload rights to this PPA.
<allenap> bigjools: Woohoo \o/
<bigjools> mtaylor: can you confirm it's signed with the right key?
<mtaylor> bigjools: to be specific - I am a member of the owning team, and I was also a direct member, and then we deactivated my direct membership  - just in case that has something to do with it
<mtaylor> bigjools: yes - when I reactivated my direct membership, it worked again
<bigjools> ok
<bigjools> you'd better file a bug then
<mtaylor> bigjools: okie!
<bigjools> matsubara: I have got an OOPS file on dogfood, how do I turn it into a lovely web page that analyses repeated and long queries?
<bigjools> mtaylor: sorry man :(
<matsubara> bigjools, it should appear in the lp-oops.c.c. isn't that working?
<bigjools> matsubara: it's a dogfood oops, so I doubt that :)
<matsubara> it's supposed to work. what's the oops id?
<SpamapS> Hi! So, I need to have a distribution renamed... apparently this is not doable through LP's admin interface...
<matsubara> bigjools, I think we need to ask the losas to sync dogfood oopses to devpad
<bigjools> matsubara: or can I just poke the tools on DF and run them there?
<bigjools> I only want it for  a single oops
<bigjools> or is it not that easy?
<bigjools> I don't mind syncing them to devpad
<bigjools> SpamapS: it's not
<nigelb> German topic?
<matsubara> bigjools, not that easy. syncing would be easier and would set things up for the same in the future
<bigjools> matsubara: ok, I guess we need an RT
<bigjools> matsubara: can you copy this single oops for me for now?
<matsubara> bigjools, I'll file the rt. yep, could you copy it to devpad and I'll try to load it into lp-oops.c.c
<cjwatson> bac: I've done as adeuring asked above for https://code.launchpad.net/~cjwatson/meta-lp-deps/new-apt/+merge/71386 - would you be able to land it?
<bac> cjwatson: sure
<cjwatson> excellent, thanks
<bac> cjwatson: in just a bit
<cjwatson> (and presumably kick off the recipe build or whatever it is)
<cjwatson> https://dev.launchpad.net/LaunchpadPpa#launchpad-dependencies looks wrong to me, seeing as recent builds seem to have been done by way of a recipe
<cjwatson> https://code.launchpad.net/~launchpad/+recipe/meta-lp-deps-on-demand looks relevant
<bac> cjwatson: i'll get the branch landed...the recipe stuff i'll have to figure out
<cjwatson> I guess that ought to be triggered on commit?
<cjwatson> or daily, anyway
<bac> cjwatson: did abel just ask for some changes on IRC?
<cjwatson> 16:56 <cjwatson> (it'll need to have UNRELEASED replaced with the appropriate series in debian/changelog, of course)
<cjwatson> 16:56 <cjwatson> (if you'd like me to do that, tell me which series to use)
<cjwatson> 17:02 <adeuring> cjwatson: i think natty would appropriate
<cjwatson> so yeah, I did that
<cjwatson> otherwise he just said 16:55 <adeuring> cjwatson: r=me.
<bac> cjwatson: ok, thanks, i wasn't sure if that was what you were referring to
<bac> cjwatson: were you suggesting the instructions around step 11 of https://dev.launchpad.net/LaunchpadPpa#launchpad-dependencies are incorrect?  with the recipe build, the branch only needs to be pushed up?
<bac> hi sinzui, could you tell me if the instructions for uploading to meta-lp-deps as shown here are still correct:  https://dev.launchpad.net/LaunchpadPpa#launchpad-dependencies ?  i see you've done commits to that branch lately.
<sinzui> bac: Those instructions look correct, but I see a recipe is setup to build it https://code.launchpad.net/~launchpad/+recipe/meta-lp-deps-on-demand
<bac> sinzui: correct.  so due to the recipe, which steps can i skip?
<sinzui> bac: 1-12
<sinzui> Error you would have seen in steps 1-12 will be sent to you via email
<bac> sinzui: so i just push the branch then?  and it gets signed by LP?
<bac> sinzui: is it an issue that i'm pushing up a branch for cjwatson and he is listed as the changer in the changelog?
<sinzui> bac, no it is not an issue
<bac> thanks sinzui.  i'll update that wiki page now.
<sinzui> bac: the package is signed by you I believe. I may be mistaken and it is signed by the owner of the ppa
<bac> cjwatson: done.  waiting for it to build.
<timrc> anyone know off hand how you can express " Use all Ubuntu components available." with the addArchiveDependency API call? it seems that the default behavior (no specifying a component or specifying None) is the equivalent to "Use the same components used for each source in the Ubuntu primary archive."
<bac> cjwatson: it *looks* happy: https://code.launchpad.net/~launchpad/+recipe/meta-lp-deps-on-demand
<cjwatson> bac: excellent, thank you - I'll update the corresponding RT ticket in a bit
<cjwatson> (done)
<lifeless> grr race conditions on landing :(
<lifeless> anyone around ?
<lifeless> I need a sanity check, I think we have a subtley broken test.
 * lifeless wishes for a highlight-all facility to ping folk :P
<lifeless> sinzui: are you still around ?
<sinzui> lifeless, yes
<lifeless> sinzui: can you please follow the reproduction instructions on https://bugs.launchpad.net/launchpad/+bug/825486
<_mup_> Bug #825486: testPartnerUploadToNonReleaseOrProposedPocket is incorrectly passing <Launchpad itself:New> < https://launchpad.net/bugs/825486 >
* bac changed the topic of #launchpad-dev to: Das Thema fÃ¼r #launchpad-dev ist: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[#######=]:256
<sinzui> lifeless, my error is very similar: AttributeError: 'NoneType' object has no attribute 'write'
<lifeless> sinzui: great, thanks.
<lifeless> test disabling will happen with the landing of my useoops branch (2nd landing)
<lifeless> StevenK: you know that test I asked about which you said I must have broken? Nope. Broken in trunk.
<abentley> lifeless: I have a change to codehosting that I'll be landing soon.  What the current deployment story?
<lifeless> we can do a nearly-no-downtime
<lifeless> but its on request, not part of the default nodowntime set
<lifeless> because of losa overhead basically - got to wait an hour or so for connections to drop off, and usually have to manually kill the daemon (because of tarmac and similar htings that are holding open persistent connections)
<lifeless> bug 819604: reconnect when connection drops
<lifeless> bug 795025: allow admins to ask the server to close the connection
<lifeless> after it finishes the current request
<lifeless> bug 824797: drop connections that have been idle for a while
<_mup_> Bug #795025: no way to gracefully disconnect clients and shut down the bzr server <canonical-losa-lp> <hpss> <launchpad> <ssh> <Bazaar:Confirmed> <Launchpad itself:Triaged> < https://launchpad.net/bugs/795025 >
<_mup_> Bug #824797: bzr serve doesn't drop idle/dead connections <hpss> <serve> <Bazaar:Confirmed> < https://launchpad.net/bugs/824797 >
<_mup_> Bug #819604: when an idle ssh transport is interrupted, bzrlib errors; should reconnect instead <error-reporting> <hpss> <launchpad> <ssh> <Bazaar:Confirmed for mbp> <Bazaar 2.1:Confirmed> <Bazaar 2.2:Confirmed> <Bazaar 2.3:Confirmed> <Bazaar 2.4:Confirmed> < https://launchpad.net/bugs/819604 >
<lifeless> abentley: we do have two instances though, so short lived connections - normal operations - are not inconvenienced at all
<lifeless> abentley: once a server is in quiesce mode haproxy no longer sends traffic to it
<abentley> lifeless: I've updated the internal xmlrpc server to emit a new type of fault in rare circumstances.  Do you think that would break a nodowntime deployment?
<lifeless> abentley: I don't know :). What happens if the existing codehosting server runs against the new xmlrpc server ?
<lifeless> (and that fault gets emitted)
<abentley> lifeless: I think you'll get an ugly error about an unexpected fault.
<lifeless> abentley: the new fault only occurs in a subset of situations that would previously fault differently though, right ?
<lifeless> abentley: its not 'something that worked now doesn't', its 'something that didn't work fails differently'
<abentley> lifeless: Yes.  Specifically, when you try to push to a sourcepackage branch that doesn't exist, using an invalid name.
<lifeless> yeah, I thought that was the branch you were referring to
<lifeless> so, I wouldn't halt the nodowntime deployment to appservers for this.
<lifeless> but I would expedite the deploy of the updated fault handling code to codehosting
<abentley> lifeless: okay, so after landing, we can do a low-downtime deployment.
<lifeless> yes - land, nodowntime as normal, then a codehosting specific deploy.
<abentley> lifeless: btw, the relative import works fine on Canonistack's lucid instance.
<lifeless> good to know
<lifeless> Got to be some subtle difference between python versions :(
<LPCIBot> Project devel build #969: FAILURE in 6 hr 13 min: https://lpci.wedontsleep.org/job/devel/969/
#launchpad-dev 2011-08-13
<wgrant> lifeless: "- Issue #7902: When using explicit relative import syntax, don't try implicit relative import semantics."
<wgrant> lifeless: Changed in 2.6.6.
<wgrant> lifeless: I presume that's it, but it doesn't really make sense...
<wgrant> sinzui: Oh, nice.
<wgrant> sinzui: I'm glad someone finally did that.
<sinzui> wgrant, I am fixing the private-by-default case when bugs are created by email
<wgrant> Ah
<StevenK> wgrant: Did you see my branch of evil?
<wgrant> StevenK: I did.
<wgrant> lifeless: You fail at dropping that table.
<StevenK> wgrant: And I don't get called a horrible person? :-)
<wgrant> Only if you don't check the test counts afterwards.
<StevenK> Ah yes, I will ec2 *test* it and compare the numbers to buildbot
<wgrant> Compare the numbers with a previous ec2 run, preferably of the same rev. I don't trust buildbot to report the same numbers.
<StevenK> Same base rev?
<wgrant> Right.
<wgrant> ie start an ec2 run of the branch and not the branch at the same time.
<wgrant> compare numbers.
<wgrant> profit
<wgrant> It's going to be fairly easy for a single test to be missed here.
<StevenK> Was exactly the plan
 * StevenK pulls out r13677 of devel into a branch
 * StevenK races ec2test
<StevenK> Other than bin/ec ls lying, I have two instances running
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #970: FIXED in 5 hr 51 min: https://lpci.wedontsleep.org/job/devel/970/
<nigelb> Morning!
<wgrant> nigelb: Finished Launchpad yet?
<nigelb> wgrant: Wha?
<nigelb> You mean finished the launchpad patch I was working on?
<StevenK> Dear ec2. Please finish quicker.
<wgrant> No, finished lots of Launchpad patches that make Launchpad finished.
<nigelb> Hah. I'm not you. :D
<StevenK> wgrant: And then what will you do when you're back from holidays? :-P
<nigelb> Its a 3-day weekend for me, so I intend to finish what I was working on.
<nigelb> I didn't realize being new to YUI would be slightly complicated :)
<StevenK> nigelb: Did you see my mega-branch of evil?
<wgrant> StevenK: Another three branches of bugtask cleanup!
<wgrant> Then that saga should be over.
<wgrant> After almost 15 branches.
<StevenK> Heh
<StevenK> wgrant: But, it's utterly awesome now!
<StevenK> wgrant: Next, remove conjoinment
<wgrant> Die.
<StevenK> :-D
<nigelb> StevenK: No! What does it do? Kick wgrant away? :P
<StevenK> nigelb: https://code.launchpad.net/~stevenk/launchpad/kill-test-suite-dead/+merge/71353
<nigelb> StevenK: All I have to say is what adeuring said "Blessed be those who remove obsolete code.
<nigelb> "
<nigelb> StevenK: Also, there can be no greater pleasure than seeing code removed
 * StevenK tries to page his LaTeX skills back in
<kb9vqf> Quick question: My poppy/ directory keeps filling up with files that are never purged
<kb9vqf> I figured I could safely delete poppy/failed/* and poppy/rejected/*, but what about the files in poppy/accepted/*?
<kb9vqf> can I delete them?
<wgrant> kb9vqf: You can, yes.
<kb9vqf> ok
 * kb9vqf doesn't want to mess up his nicely running QuickBuild system :)
<kb9vqf> wgrant: Just to make sure, I can completely purge poppy/rejected/* without affecting librarian or any other Launchpad systems?
<wgrant> We delete everything after a week, just in case.
<wgrant> Everything in the poppy upload dirs is unrequired once process-upload runs.
<kb9vqf> darn copy + paste
<kb9vqf> ok
<kb9vqf> (obviously I meant poppy/accepted/* above)
<wgrant> Yup.
<kb9vqf> thanks!
<kb9vqf> I was wondering where all my disk space was wandering off to
<StevenK> Poppy keeps looging "OM NOM NOM"
<StevenK> Sigh. scriptactivity for request_daily_build is still daily
<StevenK> And still on loganberry
<wgrant> StevenK: Is that a problem?
<wgrant> It's probably the ackeeness that is a problem.
<wgrant> I'm not sure if it was moved, though...
<StevenK> It's part of bzrsyncd, I thought?
<wgrant> Shouldn't be, but maybe.
<StevenK> It runs every 15 minutes, so perhaps the checking should be more often
<wgrant> So it is.
<wgrant> It shouldn't need to be bzrsyncd, though.
<StevenK> 15
<StevenK> 15? 10? I don't remember.
<wgrant> */15, yes.
<StevenK> wgrant: I should get my answer from the two ec2 instances in the next few minutes ...
<wgrant> You could also diff test lists without ec2, but I like to see if ec2 works too :)
<StevenK> I have at least four test failures, too
<StevenK> But it was better than the previous incantation of this branch, which had 500.
<wgrant> Hah
<StevenK> r13685 of devel => 15,607 tests.
<StevenK> I am disappoint. 15,575 tests.
<StevenK> But only 8 errors.
<wgrant> StevenK: Try bin/test --list-tests and diff?
<StevenK> I threw the subunit streams through subunit-ls
<wgrant> Or that.
<StevenK> Looks like test_bzrutils needs reverting, it repeats tests
<StevenK> Before revert: Ran 10 tests with 0 failures and 4 errors in 0.909 seconds.
<StevenK> After: Ran 46 tests with 0 failures and 0 errors in 4.804 seconds.
<wgrant> Which is interesting, because that's 4 more tests than were missing.
<StevenK> Heh, yes.
<StevenK> The subunit streams didn't show tests which errored, either.
<StevenK> wgrant: Right, another file reverted and it said 102 before, and 99 after.
<StevenK> So now we're at -1
<wgrant> --list-tests diff
<StevenK> Which is canonical.launchpad.database.tests.test_oauth.BaseOAuthTestCase.test__get_store_should_return_the_main_master_store
<StevenK> wgrant: http://pastebin.ubuntu.com/664825/
<StevenK> wgrant: I think those tests are skipped/disabled by fiddling with the test loader.
<StevenK> wgrant: Using --list-tests and diffing: http://pastebin.ubuntu.com/664832/
<StevenK> The first hunk is a red herring
<wgrant> Ah, and the rest are too.
<StevenK> Really?
<wgrant> They're all testrunner* config name differences, aren't they?
<wgrant> Remember that each testrunner process creates a new testrunner and testrunner-appserver config with the PID in the name.
<StevenK> Except that my branch has more of them
<StevenK> Double, in fact
<wgrant> Indeed, but that may just mean you ran and killed the testrunner badly.
<StevenK> Which I probably did
<StevenK> wgrant: So shall I commit, push and lp-land this monstrosity?
<wgrant> If that's the whole diff now, I suspect so.
<StevenK> That's the whole diff from --list-tests
<wgrant> StevenK: You never said there was Perl involved!
<wgrant> StevenK: This changes everything.
<StevenK> wgrant: You didn't read the MP?
<nigelb> heh
<nigelb> All the more reasons to be "extra" evil
<StevenK> nigelb: I pasted the *Perl script* in the description of the MP!
<nigelb> StevenK: I saw :)
<nigelb> StevenK: This shows how much attention wgrant paid to the MP.
<StevenK> Haha
<nigelb> Btw, you both are up late. On a Saturday! Why!
<StevenK> I was entertaining people for dinner, had to clean up, and are now winding down
<nigelb> :)
<lifeless> wgrant: select * from pg_class where relname like '%lookup%';
<lifeless>  entitlement_lookup_idx |         2200
<lifeless> (1 row)
<wgrant> StevenK: I don't tend to read MP descriptions until the end of the review, because it may influence how I treat the diff.
<wgrant> lifeless: :( why is Ensemble treating LP like Ubuntu does -- its own namespace to stomp all over?
 * jelmer waves to wgrant
<wgrant> Morning jelmer.
<jelmer> wgrant: How is Ubuntu stomping over Launchpad namespaces? Most things either seem to live under /ubuntu/ or have a ubuntu- prefix?
<wgrant> jelmer: Nowadays they're pretty good, yeah.
<wgrant> They used to not be :)
<jelmer> hah, ok
<wgrant> https://launchpad.net/~motu https://launchpad.net/~techboard are old examples.
<wgrant> https://launchpad.net/~developer-membership-board is a new onew.
<wgrant> https://launchpad.net/~mono
<wgrant> That sort of thing.
<jelmer> gotcha
#launchpad-dev 2011-08-14
<nigelb> I still can't figure out why I started hacking on LP before I read about Zope 3.
<nigelb> Finally I'm reading Zope 3 intros :P
 * StevenK purges 3GiB of postgres logs so that /var actually has some free space.
 * nigelb twitches at GiB
<StevenK> nigelb: You don't like {K,M,G,T}iB?
<StevenK> Can't remember what comes after T. E?
<lifeless> Eca
<lifeless> bah
<lifeless> Exa
<StevenK> Right, so it is E
<lifeless> or is it P
<StevenK> Then, P, then Z, and then OMGBBQWTFBIG
<lifeless> it might be P E
<lifeless> ah E P
<lifeless> wgrant: we don't build-in a structure of partitioned namespaces (partly because teams have no project affiliation etc
<lifeless> wgrant: so folk need to remember
<lifeless> wgrant: however, this one has me confused too ;P
<nigelb> StevenK: My dislike is for the 'i'
<lifeless> nigelb: you're a no i kinda guy ?
<nigelb> lifeless: Bwahaha. No.
<nigelb> So, I noticed something weird today.
<nigelb> Potentialy a mintor security thing.
<nigelb> When I go to https://launchpad.net/~canonical-sysadmins, I see a <hidden> under "Subteam of"
<nigelb> Isn't our policy to to show "its not even there" when something isn't supposed to be seen?
<nigelb> s/show/do'g
<nigelb> s/show/do/g
<lifeless> thats an adapter kicking in and preventing disclosure
<lifeless> yes, it shouldn't show at all, but its a cosmetic, not a security, concern
<lifeless> please do file a bug
<nigelb> cool, will do
<wgrant> lifeless: It is actually a security concern, slightly.
<wgrant> I filed it a couple of years ago.
<wgrant> It's in alphabetical order.
<wgrant> So you can determine the display name.
<nigelb> wgrant: there's already a bug?
<wgrant> I think so. Trying to find it...
<lifeless> wgrant: you can guess at possible display names ;)
<wgrant> lifeless: Then you can create teams on staging to do a search.
<nigelb> wgrant: bug 329132
<nigelb> ?
<_mup_> Bug #329132: Strange string under "Subteam of" in LOSAs' team page <confusing-ui> <lp-registry> <registry-people> <Launchpad itself:Fix Released by bac> < https://launchpad.net/bugs/329132 >
<wgrant> Hah
<wgrant> What a fix.
<wgrant> I thought I filed a similar one, though...
 * nigelb stabs Zope 3
<nigelb> Trying to get it to work. Refuses to work.
<wgrant> What are you doing to the poor creature?
<nigelb> Using Benji's quick start guide
<wgrant> nigelb: I guess you should file a new one rather than reopen the old one.
<nigelb> wgrant: Filing now
<wgrant> nigelb: You may have some issues with using that.
<nigelb> Oh.
<wgrant> nigelb: Since Zope 3 doesn't exist any more.
<nigelb> wtf.
<wgrant> It's... complicated.
<wgrant> Most of the underlying libraries now make up what is known as the Zope Toolkit, or ZTK.
<wgrant> The UI (ZMI and other stuff that's mentioned there) is not part of the ZTK, and was abandoned for a time.
<wgrant> But then was made into its own project atop the ZTK, BlueBream.
<nigelb> All I wanted was to get a hands on feel for zope minus the complications of Launchpad so I can figure out Launchpad faster.
<wgrant> Not sure if that is still active.
<wgrant> http://bluebream.zope.org/screencasts.html may be helpful.
<wgrant> http://bluebream.zope.org/doc/1.0/index.html too
<wgrant> Looks like the docs don't make me want to die any more. Which is a good start.
<nigelb> Oh, cool!
<nigelb> Thank you
<nigelb> Is that closer to what LP is using?
<wgrant> LP doesn't use BlueBream, but it uses many components of the ZTK.
<wgrant> Not all the latest versions, but mostly within the last 18 months.
<wgrant> It also uses some stuff that BlueBream uses but that is outside the ZTK.
<nigelb> Ok, so it should give me a good basic idea.
<wgrant> Yup.
<nigelb> Right now, I'm coding on LP with extensing grepping and asking here.
<nigelb> And grepping my logs of asking here.
<wgrant> Previously I would have suggested that you obtain, by some means, a copy of Web Component Development with Zope 3. But it was already obsolete when I started, and it's going to not be practically usable now.
<wgrant> Good in-depth teaching of the concepts, though.
<wgrant> Most LP engineers do what you do, I think :)
<wgrant> They learn Zope 3 by drowning in LP.
<nigelb> Hah
<wgrant> "It is recommended to use a custom-built Python for working with BlueBream."
<wgrant> Gaaaaah stupid Python people.
<nigelb> heh
<wgrant> (if following the bluebream getting started guide, I would use virtualenv to install bluebream without polluting your system)
<nigelb> I always use virtualenv for this kinda stuff
<lifeless> wgrant: ah, nifty
<wgrant> From Wikipedia: "BlueBream is in active development and is now considered a stable framework, used on production projects worldwide, most notably Launchpad."
<nigelb> WHAT?
<wgrant> But we don't use the whole thing :(
<wgrant> Maybe they've altered the scope of BlueBream to be everything that was in Zope 3 that is not in the ZTK.
<wgrant> Last I heard it was primarily the ZMI.
<lifeless> nigelb: my preferred solution for 'LP is hard to understand' is to clean things up
<lifeless> nigelb: e.g. by deleting lots o fcode
<nigelb> lifeless: My problem is my web development experience is in Django. I'm trying to figure out how is the sort of right way to to do things in zope.
<lifeless> nigelb: ah
<nigelb> Its probably a day or two of reading up
<lifeless> nigelb: so do you want the good news or the bad nes ?
 * StevenK deleted something like 2,800 lines of code yesterday
<nigelb> I figured its time I invested that
<nigelb> lifeless: I'll take the bad news first
<lifeless> nigelb: the right way of doing things is the wrong way for LP
<nigelb> Expected.
<nigelb> Yay!
<lifeless> sorry, that was mistyped. But hilarious.
<lifeless> nigelb: the right way of doing things in zope is the wrong way for LP
<nigelb> StevenK: When I meet at some point, I'll buy you a drink for that.
<lifeless> is what I meant to say.
<nigelb> Small difference :)
<nigelb> StevenK: *meet you
<lifeless> two words, but significant impact :>
<wgrant> lifeless: Well, everything except persistence is not too bad.
<lifeless> wgrant: except that *everything* is driven by persistence.
<lifeless> wgrant: also you forgot configuration.
<nigelb> Well, so I'm probably looking at bare basic concepts. Like tal thingy or zcml.
<nigelb> Both of these I still don't really understand, but I can read and guess
<wgrant> lifeless: Except ZCA and views and TAL and ZCML and the general architecture.
<lifeless> wgrant: tal and views are impacted by persistence; zcml is because of the model:interfaces interactions.
<wgrant> Only slightly.
<wgrant> lifeless: Also, do we want to roll back that spaces in URLs thing?
<lifeless> wgrant: zca also interacts with persistence in the same way, fwiw
<lifeless> wgrant: yes, we do.
<wgrant> Shouldn't have to roll back production, since we're almost deployable.
<wgrant> Also there's also an additional 1000 Loggerhead OOPSes that we might care about.
<lifeless> wgrant: are you still on leave ?
<wgrant> Yes.
<lifeless> heh
<lifeless> wgrant: I was going to ask if you felt like doing the rollback
<nigelb> If wgrant is *this* productive when he's on leave...
<wgrant> qemu-kvm's broken gdbstub is broken enough that I will look at work, but not do it :)
<nigelb> lifeless: I saw the wiki LEP. Has it gone stale or is it still alive?
<lifeless> nigelb: its confused
<nigelb> Because I'd like to propose extracting stuff from readthedocs and bring it into LP.
<StevenK> lifeless: Oh, the loggerhead rollback?
<lifeless> StevenK: not loggerhead, url ascii rules
<wgrant> loggerhead's broken too, but I think that might be two rollouts ago.
<wgrant> Not quite sure.
<lifeless> 993 RuntimeError: dictionary changed size during iteration ?
<lifeless> probably bzrlib
<wgrant> We didn't upgrade bzrlib, did we?
<wgrant> Still 2.3.3
<wgrant> We did upgrade loggerhead.
<lifeless> we need a bug at the least
<lifeless> and probably a rollback of loggerhead too
 * wgrant curses lifeless a bit.
<lifeless> wgrant: ?
<wgrant> The only thing left writing to BugTask's target key attributes directly is the bugsummary tests.
<wgrant> removeSecurityProxy ftw.
<lifeless> wgrant: hah. I didn't write those :P
<wgrant> GUILTY
<wgrant> We might be able to put NULL Project to sleep on Monday...
<StevenK> Oh, nice
<StevenK> wgrant: Can I land populate-bprc now?
<lifeless> StevenK: why wouldn't you ?
<wgrant> Well, it almost has a veto.
<wgrant> (not from me!)
<StevenK> lifeless: From a certain GSA who won't be named saying it sounded like an epicly bad idea
<lifeless> how so ?
<lifeless> what was the concern ?
<lifeless> https://bugs.launchpad.net/loggerhead/+bug/826136
<_mup_> Bug #826136: dictionary changed size during iteration <oops> <loggerhead:Triaged> < https://launchpad.net/bugs/826136 >
<StevenK> lifeless: That the data does not and should not go into the main DB
<lifeless> StevenK: thats not a concern
<lifeless> StevenK: thats a conclusion
<lifeless> :)
<StevenK> lifeless: After wgrant and I talked about it, we decided to wait for a bit, since if we kick off the migration, it's quite difficult to migrate the data back out again
<lifeless> StevenK: I don't follow.
<wgrant> It's slightly awkward.
<wgrant> Not quite difficult :)
<StevenK> lifeless: It's a large amount of data that will land into launchpad_main -- it will sit like a red wine stain. Once it's there, it will require persistance to move
<lifeless> StevenK: so the concern is size?
<lifeless> StevenK: why isn't deleting it as simple as a loop doing delete LIMIT 50 ?
<StevenK> lifeless: No. My small concern is migration pain
<StevenK> My other concern is I'm not clever enough to implement step 3 and 4 of the grand plan
<lifeless> they being?
<StevenK> 3. Hook into the publisher to create and delete rows in BPRC and BPP when it publishes or superseded a BPR.
<StevenK> 4. Hook into the publisher to generate contents file, and turn off the current contents generation script.
<lifeless> 3 seems bogus
<wgrant> Both are.
<StevenK> Eh?
<wgrant> Neither should be in the publisher, I don't think.
<wgrant> BPRC lifetime is the same as file lifetime.
<lifeless> 4 could be in the publisher, or separate
<wgrant> So they should certainly not be removed by the publisher.
<wgrant> 4 is possibly OK, but not required, and there's already enough in the publisher IMO :)
<lifeless> 3 is bogus because there can be N published entries, and trying to add just-in-time or delete just-after-use will be massively racy.
<StevenK> The garbo job should certainly keep up with new BPRs, but I'm concerned about bloat in BPRC/BPP
<wgrant> The BPF expiry script will remove them.
<StevenK> wgrant: Ah, that reminds me.
<StevenK> wgrant: The populate-bprc garbo job returns BPRs that do not have any BPFs. Do not want.
<wgrant> Indeed, although in production there shouldn't be any.
<StevenK> wgrant: Hah, maybe.
<lifeless> StevenK: so here is a question folk might like to answer
<StevenK> lifeless, wgrant: So, land the garbo population job and then think about the next step while it populates?
<lifeless> 'when was /usr/bin/foo introducted to the archive'
<lifeless> StevenK: we might want to gc things for space considerations
<lifeless> but having enough data to ssatisfy the use case is the first step
<StevenK> We certainly do want to GC things
<lifeless> StevenK: what was the gsa's concern ?
<StevenK> lifeless: That it doesn't belong in the main DB
<lifeless> StevenK: You can fit *every* file in *every* arch in *every* published deb, ever, in < 5GB
<lifeless> [excluding PPAs, but they are (for now) still minimal)
<StevenK> lifeless: Modulo bugs :-(
<lifeless> StevenK: yes, but having extra rows would be fairly obvious
<lifeless> StevenK: I don't think gc is a given.
 * StevenK fixes the conflicts the branch has with devel
<lifeless> rev 13687 is the rollback
<StevenK> lifeless: We don't have much QA to do, wallyworld_, bigjools, 2 * danilo
<StevenK> We might be able to deploy on Monday afternoon UK time
<lifeless> not morning ?
<StevenK> lifeless: Depends on how quick the QA is done -- I was being pessimistic.
<nigelb> gah, qastaging is timing out, known?
<jelmer> nigelb, hey
<jelmer> nigelb, I don't think it is - perhaps it's just updating?
<nigelb> jelmer: hm, maybe.
<nigelb> I'll just try a while later :)
<lifeless> nigelb: timing out on all pages or just one ?
<lifeless> nigelb: some timeouts are (unfortunately) 'normal' for [qa]staging
<lifeless> aieeeee
<lifeless> IUnloggedException. Sob.
#launchpad-dev 2012-08-06
<StevenK> So, you have a dastardly plan?
<StevenK> wgrant: &
 * StevenK grumbles at his fingers.
<wgrant> StevenK: Yeah, testing it on DF
 * StevenK resurrects the redux branch
<wgrant> StevenK: I have an equivalent that is 1ms rather than 1000ms, but I'm not sure that it is truly equivalent yet.
<StevenK> wgrant: That was the problem? The query was taking 1 second per notification?
<wgrant> StevenK: Calculating the structural subscribers for a private bug took 600-1200ms
<wgrant> StevenK: And because the notification code is crap, it ran that calculation several times for a single operation.
<wgrant> I have a query which fixes the slowness
<wgrant> And I might refactor it all later to remove the duplication
<wgrant> Since four separate bugnotificationrecipient timeouts plagued my last week
<StevenK> wgrant: Ah, so the real smoking gun is the change in model/structuralsubscription.py
<wgrant> StevenK: I believe so
<wgrant> StevenK: Though dupe/etc subs may be similarly afflicted, they weren't important in this case.
<wgrant> (probably since private bugs with dupes are relatively rare)
<wgrant> StevenK: I've added the full suite of APGs on DF, so performance testing there should be more accurate.
 * StevenK stabs django for lying
<lifeless> StevenK: oh?
<StevenK> lifeless: Figured it out. The docs could be clearer about lots of things.
 * lifeless is no wiser
<StevenK> lifeless: Adding commands that manage.py can execute
<wgrant> Oh
<wgrant> EXPLAIN ANALYZE gets a bit confused when dealing with UNIONed subqueries
<wgrant>            ->  Index Scan using teamparticipation_team_key on public.teamparticipation  (cost=0.00..30.16 rows=8 width=8) (actual time=0.016..0.173 rows=56 loops=5)
<wgrant>                  Output: public.teamparticipation.id, public.teamparticipation.team, public.teamparticipation.person
<wgrant>                  Index Cond: (public.teamparticipation.team = (2794))
<wgrant> What it actually means is "Index Cond: (public.teamparticipation.team = [result of UNIONed subquery that happens to start with a literal 2794])" :(
<StevenK> Hah, helpfl
<StevenK> *helpful
<lifeless> wgrant: really ?
<lifeless> wgrant: can I see the full explain analyze + original query ?
<wgrant> lifeless: https://pastebin.canonical.com/71501/
<wgrant> Observe the last couple of lines of the plan
<wgrant> And how they make no sense unless it actually means the full UNIONed result
<wgrant> Note also that the output of the HashAgg is '(2794)' because of that
<wgrant> (ignore the Hash Semi Join... I'm trying to work out what it's doing there)
<lifeless> What happens if you replace the select constant as grantee
<lifeless> with select grantee from (select constant as grantee)
<lifeless> I wonder if its misplanning entirely
<lifeless> -> worried
<wgrant> Nah, it just optimises that away
<wgrant> The plan seems correct apart from the hash semi join.
<wgrant> Which duplicates the other subplan, which is largely identical and more efficient
<wgrant> I don't understand why it thinks it also needs the hash semi join
<lifeless> 11294 Time Outs
<lifeless> 6415 /    0  Branch:EntryResource:getMergeProposals
<lifeless> ^ regression methinks
<wgrant> lifeless: That's the bug you triaged a week ago
<lifeless> ah'
<wgrant> (and yes)
<lifeless> i'm getting old in my young age.
<spm> you have a beard. bearded people are old. qed.
<spm> a 9yo expert in old people told me that.
<wgrant> spm: He forgot that already
<wgrant> Due to his age.
<spm> clearly
<StevenK> spm: '9yo expert' is a tautology
<spm> not from the 9yo's perspective.
<spm> ^^ relativity
<lifeless> thats special
<wgrant> StevenK: So, mind if I take over the structsubs branch?
<StevenK> wgrant: ... to do what?
<wgrant> StevenK: Make it fast.
<wgrant> StevenK: I mean the private structsubs branch
<StevenK> wgrant: If you like
<wgrant> Not the information type one
<StevenK> The model code has landed already.
<StevenK> The UI will happen tomorrow now that get_bug_tags() no longer exists
<wgrant> Right
<StevenK> I'm still trying to bend django to my will
<StevenK> It acts utterly and completly stuffed with :memory: sqlite
<lifeless> StevenK: :(
<lifeless> StevenK: it possibly is dropping all the handles to it
<lifeless> StevenK: if thats the case, use a random /tmp path for the db
<StevenK> lifeless: Yeah, I'm getting to that point
<StevenK> lifeless: I thought it was a process thing last Monday -- as in, if I could do syncdb and runserver in one process, it should be fine. It isn't.
<lifeless> StevenK: sorry for making your life hard
<lifeless> StevenK: /tmp will be the go
<StevenK> lifeless: Yeah, I have two choices there. Hard code /tmp/auditor.sql in auditor's config, or forcibly import auditor's settings in the fixture and mktemp
<lifeless> StevenK: you don't want a hard coded path :)
<lifeless> StevenK: If I can make a (little awkward but tolerable I think) suggestion.
<StevenK> Yes I do, it will be EASY
<StevenK> As opposed to that last thing you suggested
<lifeless> it will fail horribly when you have concurrency.
<lifeless> so, in your settings.py, consult an environment variable for the path
<lifeless> if its not set, use whatever path you were using before the :memory: experinment.
<lifeless> in the fixture, set a unique path using TemporaryDirectory to get a throwawy subtree.
<lifeless> and export it via EnvironmentVariable
<lifeless> (both are fixtures from fixtures)
<StevenK> The fixture already creates a temp directory for the logfile, so that should be easy
<lifeless> cool
<StevenK> lifeless: http://pastebin.ubuntu.com/1131833/
<lifeless> cool
<StevenK> Now to hack the fixture
<wgrant> There'd better not be a default of a well-known path in /tmp, or I will be unhappy :)
<StevenK> wgrant: BLEH
<StevenK> Because debugging via connecting to the DB is for chumps or something.
<lifeless> wgrant: there will be, but the fixture will override it
<lifeless> wgrant: and prod deploys will use postgresql.
<lifeless> wgrant: so suck it up white boy :)
<wgrant> lifeless: The fact that /tmp exposes a "create item by name" interface is a historical mistake.
<wgrant> So a path of /tmp/auditor.sql doesn't make sense
<wgrant> As it's not possible by sensible means to create a file with that path
<StevenK> wgrant: So I'm supposed to make use of tempfile.mkstemp and just guess?
<wgrant> Guess?
<StevenK> Even though the fixture is going to override it and it's just for people wanting to run auditor's tests locally?
<wgrant> Is any callsite not going to override the default?
<wgrant> The test suite surely has to
<wgrant> If nothing uses the default, then the default doesn't need to be there.
<StevenK> There are three testsuites here -- auditor's, auditorfixture's and LP's.
<StevenK> The fixture will override the default, so it's fine for the fixture itself and LP.
<lifeless> the risk is that someone will attack your system by having a symlink to something you care about at that path
<lifeless> but they can't generate arbitrary contents
<wgrant> It's not a significant risk here.
<wgrant> But it's something that people will copy.
<wgrant> And it provides no value here
<lifeless> so put big warning messages around it.
<wgrant> => it should not exist
<lifeless> wgrant: whats your alternative ?
<lifeless> wgrant: whenever you find yourself saying 'has no value' please provide an alternative, as an existence proof, for something functionally equivalent.
<StevenK> wgrant: It provides value for people being able to run auditor's test suite
<lifeless> wgrant: and yes, I realise the irony that I didn't do that for ORM discussion.
<lifeless> wgrant: I should have.
<StevenK> Without having to set environment variable
<wgrant> lifeless: Well, in order to use the default, I'd have to copy auditor.sql to /tmp/auditor.sql before running auditor. It's surely just as easy to set AUDITOR_SQL before running auditor.
<wgrant> StevenK: Can't the test suite poke it in other ways?
<lifeless> wgrant: huh?!
<wgrant> Having the test suite use a single well-known commonly-conflicted path is not a good idea.
<StevenK> wgrant: Like I said before, which test suite?
<lifeless> wgrant: tell you what, to get StevenK moving along.
<StevenK> There are three.
<wgrant> StevenK: The one that would use /tmp/auditor.sql
<wgrant> Any test suite using that is non-concurrent
<lifeless> wgrant: understand we're *improving* the situation, it *used* to be just /tmp/auditor.sql always.
<lifeless> wgrant: so file a bug, and StevenK can look at it later.
<StevenK> Actually, it used to be /home/steven/auditor.sql ... :-(
<lifeless> StevenK: hah, same issue though.
<StevenK> Right
<wgrant> Ah, if it's already worse, then OK :)
<StevenK> Although, thinking about it, the testsuite doesn't use that, so the default can probably die
<lifeless> StevenK: one step at a time
<lifeless> StevenK: you may be right, but why yak shave when you don't need to.
<StevenK> wgrant: Are you winning at making private structsubs fast?
<wgrant> StevenK: Yes.
<wgrant> Trying to also win at not making the queries horrid
<adeuring> good morning
<rick_h_> jcsackett: ping when you get in, quick ? for you
<jcsackett> rick_h_: ping.
<rick_h_> jcsackett: pong
<deryck> adeuring, here's my branch, if you don't mind:  https://code.launchpad.net/~deryck/launchpad/support-pape-max-auth-age/+merge/117781
<rick_h_> jcsackett: nvm about earilier, don't think it'll work
<adeuring> deryck: ok
<jcsackett> rick_h_: ok.
<jcsackett> rick_h_: hangout still planned for now?
<rick_h_> jcsackett: yea, just hangouts not playing nice and starting up for me atm
<rick_h_> 5th try to get it to load/start
<deryck> rick_h_, are we using our url, or you starting a random one?
<rick_h_> deryck: I'll try our url next I think
<rick_h_> was trying to start a new one with no success
<deryck> worked for me.
<adeuring> deryck: r=me
<rick_h_> jcsackett: so I'm going to go think for a bit around lunch. Do you want to get together for a call this afternoon and kind of run through some points?
<jcsackett> rick_h_: sounds like a plan. have a particular time in mind?
<rick_h_> jcsackett: shoot for 3:00?
<jcsackett> sounds good.
<rick_h_> cool, thanks
<deryck> adeuring, thanks for the review
<jcsackett> sinzui: ping.
<sinzui> hello
<jcsackett> sinzui: free to chat a bit?
<sinzui> yes
<jcsackett> fantastic.
<jcsackett> sinzui: hangout invite sent.
<rick_h_> jcsackett: up for hangout?
<rick_h_> deryck: will want to put you on deck perhaps after this one as I try to get stuff talked over pre-EOD (heads up)
<jcsackett> rick_h_: i'm free.
<rick_h_> jcsackett: awesome
<deryck> rick_h_, sure.
<deryck> rick_h_, I'm done in an hour, since I started early and am at the beach.  Just to give you a heads up too ;)
<rick_h_> deryck: ok, will try to go quick
<jcsackett> rick_h_: we can keep ours short so you can talk to deryck.
<jcsackett> rick_h_: can you send me the hangout invite? there are two of you in my G+ list and i'm not sure which to send the invite too. :P
<lifeless> deryck: o/
<nigelb> mwhudson: welcome back, post-new family member arrival :)
<mwhudson> nigelb: thanks!
<lifeless> mwhudson: wb
<lifeless> mwhudson: everything go smoothly ?
<mwhudson> lifeless: ended up with a c-section due to baby getting stuck on the way out, but aside from that everything is good
<lifeless> mwhudson: woo (we did too)
<mwhudson> ah right
<mwhudson> it's a pretty intense 45 mins or so between them saying "we need to do a caesar" and getting it done :)
<mwhudson> especially if everyone's been awake for 24 hours by that point
<lifeless> mwhudson: I have /no idea/ how long it took
<lifeless> tis all a blur
<lifeless> I remember the operating room vividly
<mwhudson> heh yeah
<mwhudson> i remember being taken away and forcefully fed some carbs
<lifeless> before, after or during ?
<mwhudson> before the op
<lifeless> interesting
<lifeless> they didn't do that for me
<lifeless> perhaps my weight dissuaded them :P
<rick_h_> deryck: still got 10min?
<deryck> rick_h_ sure
<rick_h_> deryck: cool, normal hangout url?
<lifeless> deryck: want to pick a time for a call?
<deryck> lifeless, yeah.  let's do my Thursday afternoon, your Friday morning.  if that works for you.
<lifeless> its clear
<lifeless> sinzui: did my logic make sense w.r.t. mutually invisible teams in a project?
<sinzui> lifeless: yes
<sinzui> I asked Cody if they really use mutually invisible teams in a project?
<lifeless> maxb: *headdesk* ** 100
<maxb> indeed :-/
<StevenK> wgrant: http://pastebin.ubuntu.com/1133382/ is the traceback
<wgrant> StevenK: Something under related_projects is throwing an AttributeError, probably.
<wgrant> You may want to call it manually
<wgrant> in a test or harness
<wgrant> And see what the actual error s
<wgrant> is
<StevenK> wgrant: Sorted it out. Did you want to review the MP I just put up?
<wgrant> StevenK: Am doing so
<wgrant> StevenK: Just a few things that need fixing.
<StevenK> wgrant: Thanks. I've never quite figured out how IStoreSelector maps to I{,Master,Slave}Store
<wgrant> StevenK: DEFAULT_FLAVOR == IStore
<wgrant> MASTER_FLAVOR = IMasterStore
<wgrant> SLAVE_FLAVOR = ISlaveStore
<wgrant> Webapp code should very very very very very rarely use anything other than IStore
<lifeless> cjwatson: nominations seem like overkill to me
<lifeless> cjwatson: we had to cripple *who* could nominate because folk just nominated everything.
<cjwatson> I'm aware of that
<lifeless> good good
<cjwatson> Nevertheless our defect analysts etc. need to be able to target bugs, but we don't want them to have full driver privs
<cjwatson> note target not nominate
<lifeless> so target is a superset of nominate.
<cjwatson> I know
<cjwatson> BugNomination.canApprove => target
<wgrant> cjwatson: What undesirable privilege does being a series driver convey?
<lifeless> I am confused; perhaps you can describe the things you /don't/ want the defect analysts to do, and how often they do them today.
#launchpad-dev 2012-08-07
<cjwatson> Sorry, could we please do this on the bug?  This is a 4.5 year old sprawling bug with months-long hiatuses; switching back and forward between bug mail and IRC increases the probability of us forgetting stuff; plus I have to go to bed now and can't really spend the time to give a proper answer to these questions right now.
<cjwatson> And you may be right that it's unnecessary but if so I'd really like both the question and the answer - i.e. answer including motivation for it - to be recorded
<wgrant> Hm
<wgrant> I wonder if I can mangle format-imports into a generic import mover
<wgrant> (BugTaskSearchParams is moving out of bugtask.py)
<michaelh> Hey, I'm getting a corruption error from zlib while doing an export
<michaelh> Should two packs with the same hash have the same content?
<wgrant> From bzr?
<wgrant> That's probably more of a #bzr question
<michaelh> Heh, wrong Canonical-ish channel, sorry
<lifeless> wgrant: michaelh: yes, same hash - same content.
<StevenK> OMGWTFBBQ
<StevenK> structural subscriptions model is under bugs, but the JS for it is in registry?
<wgrant> StevenK: It was all initially in Registry
<wgrant> StevenK: Then the bug subscriptions work moved it into Bugs
<wgrant> For reasons unclear
<StevenK> Maybe they were going to impact branches too
<wgrant> I mean the reason for the move was unclear
<wgrant> They were going to work for blueprints etc. in the initial design
<StevenK> Ah
<lifeless> shiny
<lifeless> https://status.heroku.com/
 * StevenK stabs the SS JS
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/bugsubscriptionfilter-itype-ui/+merge/118472
<wallyworld_> StevenK: i'm just about to run and and get the kid (i'm a bit late), i can look when i get back
<wallyworld_> StevenK: the _english_condition() method could be improved - you could pass in the list to extend as a parameter
<wallyworld_> it would avoid the unnecessary list creation
<wgrant> Also, Proprietary isn't spelt like that.
<StevenK> Bleh
<StevenK> Those JS tests suck
<wallyworld_> StevenK: will this actually hook up to the back end and do something?
<wallyworld_> 342	+ 'are of information type: '+filter.information_types.join(', ')));  <-- you need a space around the + operator
<StevenK> wallyworld_: Notice the existing pattern above, shall I fix those too?
<wallyworld_> yes please
<wallyworld_> while you are there
<StevenK> wallyworld_: And yes, it hooks up to the model
<wallyworld_> cool, and it works? ie you tried it out?
<wallyworld_> normally i'd ask for a screenshot to be attached to the mp
<StevenK> Yeah, I have tried it out.
<StevenK> I can make a few screenshots, if you wish
<StevenK> wallyworld_: ^
<wallyworld_> StevenK: just one really - to show the new rendering of the info types in the popup, ie the expanded accordian
<StevenK> Ah
<wallyworld_> it's hard to visualise
<StevenK> Let me do that
<wallyworld_> thanks :-)
<wallyworld_> if there's anything else you fell would add value, feel free to include it :-)
<StevenK> wallyworld_: MP description changed to include screenshot
<wallyworld_> .me looks
<StevenK> Might change the 4 columns to 3. Not sure.
<wallyworld_> StevenK: looks good. i think 3 columns is better. it's on my todo list to get rid of those godaweful little lazr buttons from formoverlays
<StevenK> wallyworld_: http://people.canonical.com/~stevenk/information_type_filter_3.jpg
<wallyworld_> much nicer!
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1133830/
<wgrant> wallyworld_: But those buttons are so nice and fiddly to click :(
<wallyworld_> StevenK: i'd rename the _english_condition method to _add_english_condition or something. my personal preference would be for the conditions list to go as the first argument but i'm not sure if we have a standard for that
<StevenK> wallyworld_: You say as I push those changes ....
<wallyworld_> sorry :-(
<wallyworld_> wgrant: i'll do the change so that if the traversed path of the current content includes wgrant, you'll still get the little buttons
<StevenK> Hahaha
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1133836/
<wallyworld_> StevenK: looks, good, thanks. i'm surprised lint didn't pick up the spaces around the + in the js. you should include a == Lint == section in your mp
<StevenK> I always check for lint before proposing
<wallyworld_> maybe the linter needs updating. i can't recall if it complains at me when I forget the spcaes. i know curtis picks it up. maybe we should hook him up as the linter to use
<StevenK> wallyworld_: Sounds like https://wiki.canonical.com/Quotes/2011 -- search for 'Automation and Service Status'
<wallyworld_> lol
<wgrant> StevenK: dogfood's running the fixed code, and seems pretty fast
<wgrant> I've only specifically profiled the structsub queries, trying to track down the others now.
<wgrant> But please throw rocks at it.
<adeuring> good morning
<bigjools> StevenK: I'm glad that most of my humour is too foul to get Quoted
<mgz> so... there are no tests for debbugs import, but I'm not allowed to say that right?
<wgrant> mgz: You care about the debbugs externalbugtracker, not debbugs-import.
<mgz> I'm looking at bug 1029294 which is tripping in lp.bugs.scripts.debbugs
<mgz> fix looks trivial but I don't want to break the world.
<wgrant> Ah
<wgrant> Yeah
<wgrant> That code is a bit old
<wgrant> It may even be from Mark's original stuff in 2004.
<wgrant> Not sure how much it's evolved, but looking at the code... probably not much
<wgrant> Yeah, was introduced in Nov 2004, largely untouched since.
<cjwatson> mgz: Have you checked the debbugs source to see what other changes are associated with summary version 3?
<cjwatson> When I looked at that I couldn't actually find a record of such a version ...
<cjwatson> So I was going to look on the master system when I got a chance
<mgz> I have, the only one is a set of fields are no longer rfc 1522 encoded
<cjwatson> That's summary version 2.
<cjwatson> Isn't it?
<mgz> summary version 2 encodes subject and so on
<mgz> summary version 3 does not
<cjwatson> Oh, yeah, I see it now.  OK.
<mgz> the launchpad code doesn't seem to handle any encoding regardless
<mgz> so, we might get byte/unicode mix exceptions rather than harmless =?manging? without some extra handling
<wgrant> mgz: We only do bugwatches nowadays, not imports. So we really only care about status/importance, probably.
<mgz> that would simplify things, will grep around a bit
<mgz> what is this nonsense...
<mgz> ImportError: No module named auditorclient.client
<mgz> why does launchpad want to import whatever that is when it's not in deps or pulled in by rocketfuel...
<mgz> will branch the darn thing for now, but this seems borked
<wgrant> mgz: Have you rerun buildout?
<wgrant> 'make build' will do it
<mgz> I ran 'make run' which depends on build....
<mgz> it's possible I hit a window where I somehow got the code change but not the dep update, but that seems backwards
<wgrant> mgz: Possibly it didn't automatically buildout because of some timestamp thing. But the code and dep changes were in the same rev.
<mgz> so, rerunning build picks it up... but this really screws up my nice automation when launchpad periodically needs manual intervention to fix things
<mgz> especially as the fallout is non-obvious
<wgrant> It's pretty obvious
<wgrant> Missing dep
<wgrant> => deps are probably out of date
<wgrant> => buildout is out of date, since buildout manages deps
<mgz> wgrant: so, I have the log from the original update, and that pulls in the source, but left lp in a borked state, want to take a quick look?
<wgrant> mgz: Sure
<mgz> okay, pastebin was a bad idea, slow for big things...
<mgz> ...and then it timed out anyway
<mgz> wgrant: on devpad.canonical.com:/home/gz/lp.log
<mgz> so, that succeeded, but when I then went to run the tests later, auditorclient wasn't importable
<wgrant> mgz: Looking
<wgrant> mgz: Are you sure you were running the tests from the same tree?
<wgrant> Because the webapp is unlikely to start in the situation that you described.
<wgrant> And buildout was clearly run
<mgz> wgrant: doh, yup, that's it, rocketfuel not getting lightweight checkouts paining me again
<rick_h_> adeuring: ping, you get the email on the skype call?
<adeuring> rick_h_saw it a minute ago, online on skyke in a minute...
<rick_h_> jcsackett: stand up is over whenever you're ready to chat
<abentley> abel: Since rick is busy, do you mind reviewing this?  https://code.launchpad.net/~abentley/launchpad/same-queue-query-fmr/+merge/118428
<sinzui> jcsackett: rick_h_: While I know we are in an awkward be-ready-to-blue-sky sharing phase there are a few bugs in the current UI that would be nice to fix/avoid bug 1023427
<_mup_> Bug #1023427: Sharing implies you can edit <disclosure> <javascript> <sharing> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1023427 >
<rick_h_> sinzui: thanks, will check it out
<cjwatson> Any chance of a review of https://code.launchpad.net/~cjwatson/launchpad/report-pcj-oops/+merge/117601?  I'll need to get that QAed before pushing up the fix for the second of that pair of critical bugs.
<abentley> adeuring: Since rick is busy, do you mind reviewing this?  https://code.launchpad.net/~abentley/launchpad/same-queue-query-fmr/+merge/118428
<adeuring> abentley: sure, I'
<adeuring> ll look
<abentley> adeuring: thanks.
<abentley> cjwatson: I'll review it if you promise to update sampledata next time you land a DB patch :-)
<cjwatson> Did I screw something up before?
<abentley> cjwatson: You didn't update the sampledata to include distroseries on archivepermission.
<cjwatson> Oops.  Sorry about that.  Duly promised.
<cjwatson> Has that been done now?
<cjwatson> (I mean, does it need me to do a follow-up branch?)
<abentley> cjwatson: Yes, I landed it as part of my db patch.
<cjwatson> Good, thanks.
<abentley> cjwatson: r=me.
<cjwatson> thank you.
<abentley> cjwatson: np
<adeuring> abentley: r=me, but I find it still scary that this fix is necessary...
<abentley> adeuring: Thanks.
<cjwatson> Oh, since you two are around: is celery in a state where new jobs could reasonably be deployed using it?
<cjwatson> PackageCopyJobs might benefit.  (Also, I was wondering if either of you knew if there's a reason why PlainPackageCopyJob.createMultiple doesn't poke celery.)
<abentley> cjwatson: No, not yet.  There was a problem with queues accumulating that we believe is now fixed, but is related to this branch I'm landing.
<cjwatson> OK.  Useful to know to calibrate my expectations ...
<abentley> cjwatson: Yeah, hopefully with this landed, we can try to deploy celery job handling again.
<jcsackett> sinzui: i can grab bug 1023427; this is just the on production w/o write enabled issue, correct?
<_mup_> Bug #1023427: Sharing implies you can edit <disclosure> <javascript> <sharing> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1023427 >
<sinzui> no
<sinzui> jcsackett: the choicesource widget adds the action/edit link to both the value and the EDITICON. +sharing's UI knows to hide the EDITICON node so we do not render an icon that claims you can edit, but the values are still linked.
<sinzui> jcsackett: I think we need to consider changing choicesource to not linkify values as we need, or we rethink how +sharing renders all the choices
<sinzui> jcsackett: The issue relates to https://bugs.launchpad.net/launchpad/+bug/568768 which has a lovely picture showing that Lp's inconsistency with what activates a choice causes madness in 8 out of 10 people
<_mup_> Bug #568768: Bug UI Inconsistancy <confusing-ui> <javascript> <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/568768 >
<sinzui> jcsackett: I think wallyworld_ had pondered a flag to choicesource config to no link the value. That might solve my immediate concern. I still hope to stop linking the bug status and importance to the popup
<jcsackett> sinzui: bug status/importance? you mean on the bugs page instead of on the +sharing page?
<sinzui> jcsackett: yes. A good plan will allow us to fix both pages
<sinzui> jcsackett: on this picture: https://launchpadlibrarian.net/45059419/bug-incon.png the bug status and importance were linked to the popup. They should not be linked, or if they are, they are linked to search for more bugs of that status or importance
<jcsackett> sinzui: got it.
<jcsackett> sinzui: ok, i got you. was just confused as i was thinking in terms of +sharing, not in terms of choicesource.
<sinzui> jcsackett: I am not asking for you to fix the bug page, I just want to you to check that a fix for +sharing could be reused on the other page to verify the design is good
<jcsackett> sinzui: dig.
<sinzui> Sorry for writing in cryptic curtisese
<jcsackett> sinzui: all good, i can usually translate.:-P
<deryck> rick_h_, you reviewing at all now, or still UI-pairing?
<rick_h_> deryck: just losing all my UI work atm :/
<rick_h_> deryck: what's up? Need a review?
<deryck> rick_h_, that doesn't sound fun.  And yeah, if you have the bandwidth.
<rick_h_> deryck: sure thing, linky please and I'll take a break being angry at the service
<deryck> rick_h_, thanks!  https://code.launchpad.net/~deryck/launchpad/reauth-for-email-363916/+merge/118612
<lifeless> o/
<rick_h_> deryck: r=me with couple comments
<deryck> rick_h_, thanks!
<Ergo^> evening
<lifeless> hi
<Ergo^> gary_poster, hi just replied to your email
<rick_h_> man missed him
#launchpad-dev 2012-08-08
<StevenK> wallyworld_: Haha, did you miss buildbot on purpose?
<wallyworld_> StevenK: yes. i didn't want to risk any failure
<wallyworld_> you never know...
<StevenK> Heh, fair enough
 * StevenK re-reads the feature flag destruction diff, trying to figure out how to QA it
<StevenK> wgrant: Hm, neither https://qastaging.launchpad.net/ubuntu/+source/evolution/+sharing-details or https://qastaging.launchpad.net/ubuntu/precise/+source/evolution/+sharing-details works, and I'd expect them too
<wgrant> StevenK: You fail at vhosts
<StevenK> Oh, bloody translations!
<StevenK> wgrant: You need to land your branch to remove them all already
<wgrant> Heh
<StevenK> wgrant: I've done my QA, the deployment report is full green.
<wgrant> StevenK: Great.
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/no-ibug-private-default/+merge/118661
<StevenK> wgrant: r=me
<wgrant> StevenK: Thanks
<cody-somerville> wgrant, is that going to break PES scripts again? ;)
<StevenK> It is not.
<lifeless> cody-somerville: you know we have 'break PES today!' printed out above our desks...
<wgrant> It's the proper fix to the quick fix that sinzui did this morning.
<wgrant> It doesn't delete the tests he added, fortunately.
<cody-somerville> :-)
<cody-somerville> love you guys
<StevenK> Oh, you're just saying that.
<wgrant> In related news, Launchpad is not software; it's a minefield of fail-open, non-defensive, untested hacks. :)
<wgrant> lib/lp/bugs/tests/test_bugtarget2.py
<wgrant> wut
<wgrant> test_bugtarget2?
<wgrant> StevenK: Mildly oversized (due to largely mechanical test changes) cleanup MP at https://code.launchpad.net/~wgrant/launchpad/createBugParams-target/+merge/118673, if you're still reviewing
<StevenK> wgrant: Swap you, https://code.launchpad.net/~stevenk/launchpad/format-imports-once-more/+merge/118674
<wgrant> About half of those are my fault
<wgrant> But I ran it...
<wgrant> oh, not after I moved that last thing from model to interfaces
<wgrant> r=me, anyway
<StevenK> WTF, I see what you mean about test_bugtarget2
<wgrant> I might merge them later.
<StevenK> I'm not really happy about IBugTarget.createBug(), but bloody heck, it's tons better than what was there.
<wgrant> StevenK: Well, compare to the product/distribution/distributionsourcepackage implementations
<wgrant> Which all reduce to that in the new API
<StevenK> Yes, that's exactly what I mean.
<StevenK> Why not move IllegalTarget out to bugs.errors? Branch too large already?
<wgrant> StevenK: The other similar bugtask edit errors are in interfaces.bugtask
<wgrant> They should probably all be moved eventually.
<wgrant> But not now.
<StevenK> wgrant: Not fair that you keep finding problems with my branches, and I can't with yours. :-)  r=me
<wgrant> Heh
<wgrant> Thanks.
<wgrant> Next I'm going to fix the factory method that I mentioned (which has irked me for years -- makeBugTask takes target=, makeBug doesn't), and I hate to think how big the branch will be...
<StevenK> wgrant: QA, and then we can deploy again.
<wgrant> StevenK: Indeed.
<adeuring> good morning
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 4.0*10^2
<lifeless> what are the exact cross-domain limits on JS based POSTs?
<lifeless> (for error reporting via JS) - doing massive gets is unappealing.
 * lifeless goes - will look for replies in morning
<rick_h_> lifeless: proxy on the server in the end: http://stackoverflow.com/questions/6812331/cross-domain-ajax-post-call has most of the details but browser support for the headers are limited
<rick_h_> jcsackett: ping morning, got questions for you when you get a sec
<jcsackett> rick_h_: Pong. PM, or G+?
<rick_h_> jcsackett: I'll G+ you in a sec, trying to get these rollbacks going
<jcsackett> Ok.
<rick_h_> meh, diffs taking forever to update, should be quick.
<rick_h_> jcsackett: invite sent
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba), abentley | Firefighting: - | Critical bugs: 4.0*10^2
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: 4.0*10^2
<rick_h_> sinzui: ping
<sinzui> hi rick_h_
<sinzui> hi rick_h_
<rick_h_> sinzui: hey, PM
<sinzui> I have no meeting tomorrow so PM is fine
<rick_h_> sinzui: ok cool thanks
<rick_h_> sinzui: deryck is afk so can't verify, but told him to expect to set aside some time so should be good
<lifeless> rick_h_: thanks
<rick_h_> lifeless: np, something I've battled and just a pita :/
<rick_h_> wish we were a true middleware shop and could just build a wsgi middleware to include that could catch/handle all JS error/debug requests and do the proxy for you
<lifeless> rick_h_: well, we can trivially
<lifeless> rick_h_: but I'd like to offer a single instance forall canonical sites by preference
<rick_h_> yea, looked at building something like that at my last job using a wsgi middleware on a pylons stack that required a js library and wsgi middleware hookup to function
<Ergo^> rick_h_, sounds a bit like errormator i need to work on JS client though ;-)
<rick_h_> Ergo^: yea, to an extent
<rick_h_> http://www.olark.com/spw/2012/03/dont-break-the-internet-with-your-javascript/ is an interesting read along the lines
<lifeless> rick_h_: this is for lp:js-oopsd btw, considering tweaks + we need the JS side of it
<rick_h_> lifeless: cool, assumed it was oops related. Will peek at the project
<jam> abentley: quick question about the branch scanner (the new celery based code
<jam> are they long-lived processes?
<abentley> jam: all ears.
<abentley> jam: Not generally.  Sometimes branch scans are long, but otherwise no.
<abentley> jam: Well, actually, the worker threads and daemon are long lived.
<abentley> jam: But they're managed by celeryd and we can control how often they expire.
<jam> abentley: https://pastebin.canonical.com/71417/
<jam> I'm confused by the '-idle-'
<jam> I was trying to figure out some of the recent 'things are swapping' issues. It claimed to be swapping at 1GB, but it seems ilke it 'idles' at 500MB
<abentley> jam: I'm confused that they're running at all.
<jam> those might be celeryd processes vs branch scanner processes?
<abentley> jam: I talked to a webops like a week ago and said we should kill them until we are ready to do branch scanning via celeryd
<abentley> jam: It looks like a worker process that hasn't done anything yet.
<jam> abentley: so we aren't actually doing branch scanner in celery, we just have a celeryd worker group that is sort of sitting around?
<jam> I guess running: CeleryRunJobIgnoreResult
<abentley> jam: That's right.
<abentley> jam: Yeah, I suppose that dates from the last time we tried enabling it.
<abentley> jam: Actually, the Feature Flag still shows branch scan jobs enabled.  I don't understand why.  (jobs.celery.enabled_classes)
<abentley> jcsackett: ping
<lifeless> rick_h_: wow, the w3c have really forgotten how to write specs.
<lifeless> http://www.w3.org/TR/cors/#preflight-request is giving me a headache
<Ergo^> lifeless, yeah cors is PITA and doesnt work in every browser as advertised ;-)
<lifeless> Ergo^: its the green and blue presentation that is breaking me
<lifeless> and clauses like "If the field name of each header in author request headers is not an ASCII case-insensitive match for one of the header field names in headers and the header is not a simple header, apply the cache and network error steps."
<lifeless> Would be much better done as:
<lifeless> * header field names are compared insensitively between CORS supplied references and actual HTTP headers.
<lifeless> * If a complex header is named in the author request headers but not supplied in the response, apply cache and network error handling.
<lifeless> or something
<lifeless> 'if not A and not B =>' <==> 'not (A OR B)'
<lifeless> I shouldn't need to do boolean rearrangements to their rules to make them understandable! :)
<sinzui> jcsackett: about out branch in review. the issue is not about read-only
<sinzui> jcsackett: as a driver of a project, I can see who the project shares with, but I cannot edit
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
<jcsackett> sinzui: the "clickable_content" on +sharing is the little tag thing that shows the information.
<jcsackett> with this change, it doesn't look editable, which it shouldn't, per the reported bug.
<jcsackett> clickable_content is how our flag works, which is why i mentioned writable in the MP.
<sinzui> jcsackett: yep, and I see that is what the sharing table used. I think we have another bug. drivers should be seeing that page and they cannot edit. I suspect our code does not know what to do with drivers at all
<jcsackett> sinzui: they can't edit on production, or at all?
<sinzui> qastaging
<sinzui> permission denied getting to the page.
<sinzui> I see lots of things wrong with drivers that our squad needs to discuss. We will also discuss going to maintenance instead of working on disclosure
<jcsackett> sinzui: dig. so we actually have new issues, not an issue with this branch.
<wgrant> Um
<wgrant> what
<wgrant> jam: To clarify, we've been running branch scan jobs through celery on production for at least several weeks.
<wgrant> jam: The processes are meant to be long-running.
<wgrant> They're daemons.
<wallyworld_> sinzui: wgrant: jcsackett:     @enabled_with_permission('launchpad.Driver')
<wallyworld_>     def sharing(self):
<wallyworld_>         text = 'Sharing'
<wallyworld_>         enabled_readonly_flag = 'disclosure.enhanced_sharing.enabled'
<wallyworld_>         enabled_writable_flag = 'disclosure.enhanced_sharing.writable'
<wallyworld_>         enabled = (bool(getFeatureFlag(enabled_readonly_flag))
<wallyworld_>             or bool(getFeatureFlag(enabled_writable_flag)))
<wallyworld_>         return Link('+sharing', text, icon='edit', enabled=enabled)
<jam> lifeless: ^^ see wgrant's comments. So should we a) change them from long-lived because they waste the memory, or b) just shove more RAM into the box?
<jam> the long-livedness means as time => infinity, we're likely to see all the processes get pretty darn big
<jam> (if we still have the ulimit set, I guess that kills them off eventually)
 * jam sleeps
<wgrant> jam, lifeless: Killing the processes is probably stupid; we need a few spare workers to avoid paying the startup time for every scan. But they shouldn't grow infinitely, and it should probably only keep a few around.
<wgrant> But I believe it handles that automatically.
<wgrant> (I'll talk to Aaron and Abel tonight about this, since they were the ones that set it up on prod, but seem to not know that it's set up on prod)
<cgoldberg> hello.  I am trying to run the tests for pageperformancereport.  looks like I need to run buildout first?  all I've done so far is an initial checkout/branch.. any tips on how to run buildout/bootstrap so I can run `bin/test`?
<cjwatson> 'make' should do it, if you've used rocketfuel-get or an equivalent.
<wgrant> cgoldberg: If you haven't used rocketfuel-setup, you'll need to 'bzr branch lp:lp-source-dependencies download-cache', utilities/update-sourcecode, make compile
<cgoldberg> wgrant, thanks.. will give that a try..
<cgoldberg> wgrant, cjwatson, what's rocketfuel-* ?
<wgrant> cgoldberg: The LP dev setup script that most people use
<wgrant> As described on https://dev.launchpad.net/Running
<cgoldberg> wgrant, should i go that route, or just branch and compile?
<cgoldberg> wgrant, thanks.. *thats* the page i was looking for
<wgrant> As the page says, it's recommended to run LP in a Lucid LXC container on a Precise system, using rocketfuel-setup.
<cgoldberg> i'm just working with pageperformance reports.  it's a utility, so not sure I need a running LP.  thanks though.
<lifeless> cgoldberg: btw this is why I said to move it to lp-dev-utils first :)
<lifeless> cgoldberg: gets you out of the heinous LP environment
<cgoldberg> lifeless... what does bin/test do?  how can I port the tests so it works in lp-dev-utils?  what kinda test runner will I need?
<lifeless> lp-dev-utils should have a test runner already
<lifeless> but if it doesn't, just use python -m testtools.run
<cgoldberg> lifeless, ah perfect.. I'm gonna go that route (move to lp-dev-utils)
<lifeless> cgoldberg: if you have trouble, I can do the initial move for you if you want.
<cgoldberg> lifeless, doubt I'll have trouble.. but that would certainly be helpful, as I'm not quite sure where you want them
<lifeless> cgoldberg: ok, I'll do it later today
<cgoldberg> lifeless.. sweet.. thanks much
#launchpad-dev 2012-08-09
<StevenK> wgrant: factory.makeAccessPolicyGrant() is giving me ClassInfoError: <type 'zope.security._proxy._Proxy'>.__storm_table__ missing
<wgrant> StevenK: How're you calling it?
<StevenK>         self.factory.makeAccessPolicyGrant(
<StevenK>             policy=ap, grantee=subscriber, grantor=product.owner)
<wgrant> StevenK: Can I see the whole test?
<StevenK> wgrant: http://pastebin.ubuntu.com/1137210/
<wgrant> StevenK: line 22
<wgrant> [ap] =
<wgrant> Rather than ap =
<wgrant> find returns a resultset.
<StevenK> Ah!
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/publishinghistory-show-copier/+merge/118223 is worth looking at again now, I think.
<cjwatson> It's a bit longer in terms of LoC, but I have more credit now anyway :)
<lifeless> mm
<lifeless> you know the goal is overall reduction not stasis
<cjwatson> I know, but I'm well over 4000 down, I'm making considerably more net progress than most Launchpad developers ;-)
<cjwatson> (At least on this metric)
<lifeless> \o/
<lifeless> thats excellent
<lifeless> we're what, 1% of the way there
<cjwatson> About -0.4%.
<cjwatson> http://people.canonical.com/~cjwatson/tmp/loc-cum.png
<wgrant> cjwatson: Looking at your MP
<cjwatson> Blast.  Anyone want to hazard a guess as to why r15769 hasn't produced an improved oops notification on https://qastaging.launchpad.net/~cjwatson/+archive/ppa/+packages
<cjwatson> ?
<wgrant> It may be wise to confirm that gandwana's up to date
<wgrant> I believe that it sometimes lags an update behind asuka
<wgrant> But I've never been able to confirm it
<wgrant> cjwatson: ^^
<cjwatson> I was going to look at https://devpad.canonical.com/~wgrant/production-revnos but it's showing a Python traceback.
<wgrant> cjwatson: That also doesn't show staging stuff
<wgrant> Since they're deployed by an entirely separate system, for not very good reasons
<wgrant> In fact, by two entirely separate systems.
<cjwatson> Yeah, I thought it might not, but also that you might want to know about the traceback :)
<wgrant> Indeed, thanks.
<wgrant> Ah
<wgrant> cron-control on the frontends is breaking it
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/redirect-unsub-private-branch/+merge/118859
<wallyworld_> StevenK: sorry, was having lunch, looking now
<wallyworld_> r=me
<StevenK> wallyworld_: Thanks!
 * StevenK tosses it at ec2.
<StevenK> wgrant: Can you make sense of bug 834370?
<_mup_> Bug #834370: factory.makeSPPHForBPPH is incomplete <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/834370 >
<wgrant> StevenK: I am confuse.
<StevenK> wgrant: Oh?
<wgrant> StevenK: It doesn't make sense.
<wgrant> SPPHs don't have arches...
<StevenK> Right.
<wgrant> Maybe bigjools knows.
<bigjools> it wasn't me.
<StevenK> I'm tempted to just rip out the method. It has exactly one callsite.
<bigjools> what was the question
<StevenK> bigjools: bug 834370
<_mup_> Bug #834370: factory.makeSPPHForBPPH is incomplete <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/834370 >
<bigjools> ummm
<bigjools> without looking at code, s/architectures/archives/ ?
<StevenK> It respects bpph.archive
<bigjools> bug description fail
<bigjools> I'm sure it was valid at the time but no memory is being jogged
<bigjools> builds?
<StevenK> It references bpr.build.spr
<bigjools> I dunno then
<wgrant> Yeah
<wgrant> It all looks pretty sensible to me.
<wgrant> So I don't get it.
<bigjools> maybe it got fixed
<StevenK> Still tempted to rip it out
<bigjools> and the bug was not updated
 * bigjools continues with maas bugs
<wgrant> StevenK, wallyworld_: https://code.launchpad.net/~wgrant/launchpad/noro/+merge/118867 â 1369 lines (+19/-889) 22 files modified
<StevenK> I already approve
<StevenK> wgrant: r=me
<wgrant> Thanks.
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/noro-config/+merge/118869
<lifeless> https://code.launchpad.net/~lifeless/lp-dev-utils/ppr/+merge/118870
<lifeless> wgrant: ^ while you wait for StevenK  :)
<StevenK> 114	- database_name = ConnectionString(dbconfig.rw_main_master).dbname
<StevenK> 115	+ database_name = ConnectionString(dbconfig.main_master).dbname
<StevenK> wgrant: ^ Que?
<wgrant> StevenK: Observe the comment above it.
<wgrant> StevenK: It was avoiding the usual main_master property, since it always wanted RW
<wgrant> lifeless: Did you just like make lp-dev-utils depend on all of ZTK?
<lifeless> wgrant: pretty much
<wgrant> lifeless: And turn what was previously an innocent little piece of code into a horrid Zope egg?
<lifeless> wgrant: zservertracereport is used to pull out stuff from the log files, and it pulls in ZOMG
<lifeless> diminishing returns to avoid that.
<wgrant> lifeless: Right
<wgrant> That probably means we shouldn't use zservertracereport to pull out stuff from the log files :)
<lifeless> but I didn't migrate anything else, so most folk can just ignore it.
<wgrant> Since if we do then it won't be useful to anyone else.
<lifeless> its no less useful to anyone.
<wgrant> lifeless: Hmm?
<StevenK>     ImportError: No module named testbrowser
<StevenK> :-(
<wgrant> lifeless: You're adding an insane amount of bloat that will probably be removed the first time someone touches it
<wgrant> lifeless: Since I doubt $person wants to use it for Zope stuff
<lifeless> wgrant: they're going to make it more pluggable
<lifeless> we still need it for our logs.
<lifeless> I don't know what your point is.
<wgrant> My point is that lp-dev-utils is presently sane and usable.
<lifeless> And its no less so.
<wgrant> And doesn't have this bootstrap.py lets-copy-it-everywhere-because-screw-good-ideas madness
<lifeless> All the existing scripts have exactly the same overheads they had before.
<wgrant> It looks sensible, apart from the code and build infrastructure being trainwrecks.
<wgrant> So OK, I guess.
<lifeless> the existing code is the existing code; I copied in our custom option parser as a stopgap.
<lifeless> The build infrastructure is no worse than what we use elsewhere, which for all its flaws has the massive advantage of actually working.
<lifeless> wgrant: do I get a vote on the MP from you ?
<wgrant> And the massive disadvantage of having all the vileness of Python upstream's distribution attitude combined with the vileness of Zope's.
<wgrant> Sure
<wgrant> But I would like to register my extreme distaste :)
<lifeless> Sure, I feel the same way; but can't affort to spit the dummy - we'd end up having to write a stupidly large stack.
<lifeless> starting with the way packages are imported from disk and working out from there.
<wgrant> Right.
<wgrant> But I was hoping to sort of keep the infection contained.
<lifeless> it is contained, it doesn't affect php or ruby.
<wgrant> lp-dev-utils is one of the last bastions of sanity.
<wgrant> Ruby has their own mess
<wgrant> And PHP is their own mess
<StevenK> % bzr di versions.cfg | grep '^\+\w'
<StevenK> +mechanize = 0.2.5
<StevenK> +zope.app.testing = 3.10.0
<StevenK> +zope.testbrowser = 4.0.2
<StevenK> FEAR
<wgrant> StevenK: What have you done to who and why?
<StevenK> wgrant: http://pastebin.ubuntu.com/1137400/
<wgrant> StevenK: There's a nuclear winter in our future.
<StevenK> wgrant: That good?
<wgrant> So much fallout.
<StevenK> 'make' works, so WCPGW, right?
<wgrant> Shipit!
<StevenK> Haha
<StevenK> HAHAHAHA
<wgrant> Is that an evil mad nuclear physicist cackle?
<StevenK> Yes, at your ShipIt reference.
<StevenK> I was going to toss it at ec2 and sob like wallyworld_ and propose if ec2 doesn't send hate mail.
<wgrant> Sounds good
<wgrant> But the big zope.app.testing bump is a bit scary
<StevenK> Although I'm still trying to get the last reference of bug=98437 out
<lifeless> wgrant: or StevenK: I suck, I haven't ec2 enabled myself again yet. Could you land https://code.launchpad.net/~lifeless/launchpad/ppr-move/+merge/118873 ?
<lifeless> per favor
<StevenK> Fail.
<wgrant> Hm
<lifeless> StevenK: is that 'sure thing' ?
<StevenK> It is not.
<StevenK> ec2: ERROR: Branch doesn't have linked bugs and doesn't have no-qa option set. Use --no-qa, or link the related bugs to the branch.
<StevenK> lifeless: ^ no-qa is fine?
<lifeless> StevenK: yeah
<lifeless> stub: hey; can we do a catchup @ uhm, 2h from now?
<stub> lifeless: ok
<lifeless> cool,t hanks
<wgrant> wallyworld_: How do I see the bug title?
<wallyworld_> ECONTEXT
<wgrant> If it's ellipsised in the only place it's displayed, how do I see it?
<wallyworld_> if it's too long then too bad
<wallyworld_> or words to that effect
<wgrant> But the font is huge and you can fit like nothing in one line
<wgrant> In terms of bug title
<wgrant> For projects it might be reasonable.
<wallyworld_> the bugtitle issue was the one mentioned as needing fixing. perhaps a longer allowed title is required
<wallyworld_> maybe we allow 2 lines or something
<adeuring> good morning
<mgz> okay, some let's play writing some debbugs tests
<StevenK> mgz: Hahahahaha
<dpm> Hi launchpadders. Right now source packages from the extras.ubuntu.com archives are not available in Launchpad through https://launchpad.net/ubuntu/+source/<package> - could someone tell me (roughly) what would be needed to get source packages from extras show up in Launchpad?
<cjwatson> From Launchpad's point of view, extras.ubuntu.com is just another PPA.
<cjwatson> So they do show up there, but only under the "untrusted archives" expander (e.g. https://launchpad.net/ubuntu/+source/alg3py).
<wgrant> Hm
<wgrant> Why should they show up under Ubuntu?
<wgrant> They're not part of or maintained by Ubuntu.
<dpm> that's a good point. I was thinking in terms of were to report bugs, but hadn't considered the fact that they're not really part of Ubuntu
<dpm> wgrant, but you can still report bugs against e.g. https://bugs.launchpad.net/ubuntu/+source/alg3py right? So what do you think could be the best approach to report bugs against extras applications in LP?
<wgrant> dpm: Hm, I thought we used to hide the link. It nowadays seems to let you into the form, but won't let you submit.
<wgrant> So no, you can't actually file bugs.
<wgrant> It just tempts you to do it
<wgrant> I'm not sure what the best solution is
<wgrant> But forcing them under Ubuntu like partner is probably not it.
<mgz> where do the debbugs things live on disk? could do with looking at some real data to see if my tests are valid
<mgz> config has /srv/bugs-mirror.debian.org/ but don't know what server, or the process that creates the files
<dpm> ok, thanks wgrant
<cjwatson> mgz: http://www.debian.org/Bugs/Access "Accessing the raw bug data"
<mgz> ah, so we have a full mirror rsynced?
<mgz> do you know where that lives?
<cjwatson> No
<cjwatson> But it's easy enough to rsync at least a fragment of it for yourself
<mgz> right, I just need to try and get a reasonable fragment without killing this http connection :)
<mgz> (specifically I want some bugs with non-ascii characters in headers, just to check)
<cjwatson> I suspect that these days any 1% of the active spool would do, so pick a subdirectory ...
<wgrant> It used to be on macquarie then synced to loganberry, but the former copy moved recently, AIUI
<lifeless> mgz: the network diagram ops have lists the machine name I believe.
<mgz> loganberry works for me
<wgrant> lifeless: Have you fixed the crontab for the PPR move?
<lifeless> wgrant: no, I didn't think it was on autodeploy
<lifeless> wgrant: fairly sure its manually updated.
<wgrant> Ah, probably true,.
<lifeless> so I was going to completely ignore it and wait for screams, if any.
<jam> anyone have advice for https://answers.launchpad.net/launchpad/+question/205338
<jam> I can see that https://launchpad.net/ubuntu/+source/openconnect is definitely not the same project as https://launchpad.net/openconnect
<jam> However, I don't see the former as being imported anywhere.
<jam> i'm also hesitant to rename an existing project and import something new in its place.
<jam> Thoughts?
<mgz> jam: you have a reasonable fix now, from the question?
<jam> mgz: I think so
<mgz> seems unexpected success in the lp test suite doesn't count as a failure?
<mgz> gmb: what did you do previously to make the launchpad test runner understand other test results?
<gmb> mgz, I'm sprinting today so I can't give you a good answer right now... Is this under Python 2.6 or 2.7?
<mgz> er, I'm testing in precise, so 2.7, but that's a good point
<wgrant> mgz: LP's test suite doesn't have any expected failures.
<wgrant> How're you trying it?
<mgz> ah, so I shouldn't use that feature at all...
<mgz> it derives from testtools so appears to work, but the zope runner is ignorant
<wgrant> We make tests pass instead :)
<mgz> I'm (sometimes) in the habit of committing tests, then fixing the bug and flipping the expectedFailure clause
<mgz> but I can just not do that.
<mgz> also have some assertions that are not currently unused code that I guess can just be deleted, like
<mgz> +        name, email = tracker.getBugReporter("1")
<mgz> +        self.expectFailure("No decoding on headers done",
<mgz> +            self.assertThat, name, Not(Equals("=?UTF-8?Q?Jes=C3=BAs?=")))
<mgz> +        self.assertThat((name, email),
<mgz> +            Equals((u"Jes\xfas", "jesus@example.com")))
<cgoldberg> morning... I am having a problem bootstrapping lp-dev-utils.  "pkg_resources.DistributionNotFound: zc.buildout==1.5.2" (I have python-zc.buildout 1.52 installed globally) .... stack trace of make output:  http://paste.ubuntu.com/1137808/
<cgoldberg> anyone have an idea?
<jam> I'm trying to go to: https://launchpad.net/projects/+review-licenses but now it is generally timing out
<jam> I think it succeeded 1 time in about 8
<jam> (It worked a bit ago, but then I reviewed a bunch of projects, so maybe there is a data-bug where one of the new projects to review is causing the timeout?)
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: 4.0*10^2
<deryck> cgoldberg, hey.  Does a plain `make` not work?  I expect it's something to do with lp-dev-utils expectations about there being a download-cache.
<cgoldberg> deryck, plain `make` does nothing (Nothing to be done for `all'.)
<deryck> hmmm
<cgoldberg> `make bin/buildout` and calling bootstrap.py directly  gives same error as my paste
<cgoldberg> i've tried with, and without zc.buildout installed system-wide
<deryck> cgoldberg, ok, looking at my copy now.....
<cgoldberg> deryck, kk thanks
<cgoldberg> i've also tried running bootstrap.py with -t and -d options
<deryck> cgoldberg, what are you trying to do?  I didn't even realize we needed a Makefile in there.  It used to just be a collection of scripts.
<cgoldberg> deryck, right... so lifeless moved pageperformancereport (prr) into there from lp:launchpad.  i'm trying to run the tests:  test_pageperformancereport.py
<cgoldberg> deryck, the README is updated... look at the last part of it
<cgoldberg> i'm not sure if he added the bootstrap/make stuff or that was there already.. but seems it's needed
<deryck> cgoldberg, ah.  reading the makefile, I suspect you need `make download-cache && make eggs` before `make bin/buildout`.
<cgoldberg> deryck, i'll try
<deryck> ok
<cgoldberg> deryck, "make: `download-cache' is up to date."  ... same for eggs
<deryck> cgoldberg, do you have a download-cache directory in the tree?
<cgoldberg> deryck, yup
<deryck> cgoldberg, hmmm, I'm not sure then.  Maybe check with lifeless when he is around.
<cgoldberg> with a ton of tarballs/zips under /dist
<cgoldberg> deryck, yea.. fraid i'll have to.. was hoping to get through this before he arrives later
<deryck> cgoldberg, I would think if you had buildout in the download cache, or system wide, either one.  it should work.
<cgoldberg> deryck, me too :)
<deryck> cgoldberg, assuming the version is indeed 1.5.2
<cgoldberg> right.. which it is
<cgoldberg> deryck,  well.. thanks for looking  anyway!
<deryck> cgoldberg, np!  sorry I couldn't help.
<james_w> cgoldberg, either remove the global installation, or run the bootstrap in a --no-site-packages virtualenv (which can then be deleted I think)
<james_w> buildout bootstrap fails if there is a system-wide install of the same version as the desired one
<cgoldberg> james_w, ahh..i tried removing my system-wide buildout.. didn't help.  I'll try in a virtualenv
<deryck> adeuring, let's use the standup hangout.
<mgz> ocr: please take a look at  <https://code.launchpad.net/~gz/launchpad/format_version_3_debbugs_1029294/+merge/118989>
<deryck> rick_h_, no hurry, but let's use the standup hangout again when you're ready
<rick_h_> ok, jumping in
<lifeless> sinzui: o/
<lifeless> deryck: hi
<sinzui> both of you just got mail
<lifeless> deryck: we were going to do a call this morning, but neither of us put it into google calendar.
<deryck> lifeless, dang, dude.  I completely forgot.  *sigh*
<deryck> lifeless, I need to go pick up the dog too.
<deryck> lifeless, shall we calendar something for my monday afternoon, your Tuesday morning?
<lifeless> sure
<deryck> thanks, really sorry about dropping that again.
<deryck> lifeless, I'm not sure what time for me works for you, so feel free to calendar me for a time that works for you.
<deryck> i'll be around whenever then.
<deryck> sinzui, thanks for the mail.  I'll need some time tonight to get it all in my head, but it's nice to have a good list like that.
<sinzui> deryck: I cannot say that is in my head. That might be a fiction about what was done
<deryck> well, I don't mean it will stay in my head. :)
<deryck> Get my head around it, might be a better way to say it.
<Ergo^> gary_poster, ping
<gary_poster> Hey Ergo^
<lifeless> sinzui: your 12 step program is short a step
<sinzui> So is my head
<lifeless> :>
<jcsackett> wgrant: online?
<lifeless> jcsackett: 0724 for him.
 * lifeless expresses doubt about onlineness
<jcsackett> lifeless: fair. we do standup in 30 min, wondered if he might be online early.
<lifeless> oh definitely worth checking
<lifeless> never know your luck in the big penal colony
 * jcsackett laughs
<jcsackett> well, without wgrant, sinzui. i am *finally* able to focus on that banner branch today and i think discussions from last night are wrong.
<jcsackett> sinzui: IPrivacy, in this instance at least, would only be used for display of the banner...which is a very different thing from who all can *see* it.
<sinzui> yes.
<jcsackett> and the rules we agreed to adapt are all about seeing. i think the archive being private is enough to say show the banner.
<sinzui> jcsackett: it is a start certainly. But is two people look at the same build. One gets a 403, so ask the other to look and he sees it. the page might as well show the privacy banner so that user to can tell person one that is the case
<sinzui> I think the ambiguity of the bug is that user know that not everyone can see the page and have a reason in their head that something might be private...
<sinzui> and per you decision, the archive is going to leap to the user's mind
<jcsackett> what i'm saying is that i think that's the case. the archive being private is the first check for 'View'. so if it's private, the build is private. so someone else goes to look at it and gets in b/c they match another rule. but they'll see it's private.
<jcsackett> e.g. I don't think there's a reason you get a 403 looking for it when the archive is public.
<sinzui> okay
<sinzui> I think that fixes jml's concern
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
<jcsackett> ok.
<wgrant> jcsackett: The build archive being private is required, but not sufficient.
<wgrant> I can copy the source+binaries to the Ubuntu primary archive
<wgrant> Then it is no longer private.
<lifeless> rick_h_: http://paste.ubuntu.com/1138589/ + http://paste.ubuntu.com/1138605/ - a WIP
<lifeless> rick_h_: imbrandon is hacking the js bits together
<jcsackett> wgrant: but in terms of viewing the build, shouldn't we still display the banner? b/c there is still private info on the build (e.g. the existence of the archive), right?
<wgrant> jcsackett: But the build is publicly available, therefore the existence of the archive is disclosed.
<StevenK> wallyworld_: My X201 is 1280x800
<wallyworld_> wow, that's sad
<wallyworld_> in 2000 i would expect that, not 2012
<StevenK> wallyworld_: It also weighs 1.4kg and has 8 hours battery life?
<wallyworld_> mine weighs about the same, battery is only 4h
<wallyworld_> but i hardly ever run on battery
<rick_h_> lifeless: why do the iframe if it's a GET request anyway?
<rick_h_> and are we thinking more than one per page? because the instances can clobber in this plain object fashion vs creating a new instance of an oops
<rick_h_> but I guess there's not much to change atm since things like action/etc aren't populated
<lifeless> rick_h_: its existing code imbrandon is using
<rick_h_> ah gotcha
<StevenK> wgrant: An awful lot of the failures are NoReferrerError: No value for REFERER header
<lifeless> rick_h_: #juju is where we are talking about this
<wgrant> StevenK: Ah
<wgrant> StevenK: I guess people might be posting forms directly
<StevenK> Right
<wgrant> StevenK: Rather than properly submitting them
<wgrant> In which case you'll need to either properly submit them or forge a referer
<StevenK> wgrant: browser.submit() , isn't it?
<wgrant> Right
<wgrant> Maybe
<wgrant> It may be on the form
<wgrant> Can't remember.
<StevenK> admin_browser.getControl('Delete').click()
<StevenK> That's wrong?
<wgrant> That should work.
<StevenK> wgrant: http://pastebin.ubuntu.com/1138729/
<wgrant> StevenK: Do other things work?
<wgrant> Perhaps the new testbrowser doesn't send a referer at all
#launchpad-dev 2012-08-10
<wgrant> StevenK:   File "/srv/buildbot/slaves/launchpad/lucid-devel/build/orig_sourcecode/eggs/auditorfixture-0.0.3-py2.6.egg/auditorfixture/server.py", line 100, in _start
<wgrant>     raise Exception("Timeout waiting for auditor to start.")
<wgrant> Exception: Timeout waiting for auditor to start.
<StevenK> wgrant: I don't think the new testbrowser does send a referer.
<StevenK> wgrant: Hmmmmm. What? That's passed a few other buildbot runs.
<wgrant> StevenK: Yes. It's a nice new intermittent failure.
<StevenK> Bleh
<StevenK> First one, I think
<StevenK> wgrant: It looks like most of the test failures will be sorted by fixing NoReferrerError.
<wgrant> StevenK: There are surely more than 21 tests that use testbrowser to submit forms, so there must be something special about yours.
<StevenK> wgrant: http://pastebin.ubuntu.com/1138745/ is the diff. 17481 tests run in 4:24:53.047311, 21 failures, 2 errors
<wgrant> StevenK: Sure, but what's special about the tests that failed?
<wgrant> StevenK: How goes the QA?
<StevenK> wgrant: You offered a project I could push to
<StevenK> wgrant: All of the failing tests are doctests.
<StevenK> So maybe the problem is in the browser objects we toss into the doctests
<wgrant> StevenK: But there are hundreds of doctests that work.
<wgrant> StevenK: You now have APG for python-oops-tools/private, and branches default to private
<wgrant> (using a sharing policy, not BVPs)
<wgrant> Bah, actually, that won't work
<wgrant> Need to use a BVP
<wgrant> sec
<wgrant> There
<StevenK> wgrant: So, there are 675 doctests. Only 11 use getControl(..).click()
<wgrant> StevenK: No
<wgrant> $ bzr grep -l 'getControl.*click()' | wc -l
<wgrant> 331
<wgrant> $ bzr grep -l 'getControl.*click()' | grep txt$ | wc -l
<wgrant> 311
<StevenK> for i in $(find . -name '*.txt' | grep -E '(doc|tests)') ; do grep -l '.click()' $i; done | wc -l
<StevenK> 11
<wgrant> pagetests live in stories
<wgrant> not doc or tests
<StevenK> Ah
<StevenK> I thought I might be missing one
<StevenK> wgrant:  bzr push lp://qastaging/~stevenk/python-oops-tools/foo-1
<StevenK> It's been too long since I had to push a branch to qas.
<wgrant> Hm, I can't see it...
<wallyworld_> sinzui: sadly there is no native css support for multi-line text truncation but i have found a really neat little yui module which works perfectly and does all the internal calcs to simply allow a requested number of lines to be specified. it also falls back to native support if only one line is required.
<StevenK> Sigh, pasted it but didn't hit enter
<sinzui> wallyworld_: go on?
<StevenK> wgrant: Which gives me an error, so there has to be something wrong with that URL
<wgrant> StevenK: What's the error? That /+branch-id/foo is not a branch?
<wallyworld_> sinzui: i'd like to use it. the js then is Y.all('.ellipsis').ellipsis({'lines': 2})
<StevenK> bzr: ERROR: Server sent an unexpected error: ('error', 'NotBranchError', 'Not a branch: "chroot-67089488:///+branch-id/519259/".')
<wgrant> StevenK: Right, that's fine. The stacked-on branch doesn't exist on qas.
<wallyworld_> sinzui: for example
<wgrant> StevenK: The branch is created now. You can test.
<wallyworld_> sinzui: i haven't looked, but the tooltip could also display all the text as well i think
<wallyworld_> sinzui:  so it seems like a nice solution for a few hundred lines of 3rd part code, much like we do for sottable
<sinzui> wallyworld_: since you are adding it to things that use the title editor, I don't see a reason why you wouldn't hesitate
<wallyworld_> sinzui: extra code, so thought i would check
<sinzui> wallyworld_: add it I can take a look if you wnt
<sinzui> want
<StevenK> wgrant: Okay, unsubscribed and no redirect.
<StevenK> wgrant: Remove the BVP?
<wgrant> StevenK: You mean APG?
<wallyworld_> sinzui: ok, will tidy up the prototype and add it properly, then update the mp. no rush, since it won't be deployed till next week anyway
<StevenK> wgrant: Er, yeah.
<wgrant> You still want it to be private, don't you?
<wgrant> Right
<wgrant> Gone
<StevenK> wgrant: Right, so now we create foo-2 which should have me with an AAG?
<wgrant> StevenK: Or I subscribe you to foo-1 again
<wgrant> That's probably better
<wgrant> StevenK: You are subscribed
<StevenK> Too late, I pushed foo-2, but I get Forbidden on foo-1
 * StevenK refreshes
<wgrant> And +sharing confirms you have access
<StevenK> Okay, unsubscribing
<StevenK> wgrant: Redirected to https://code.qastaging.launchpad.net/python-oops-tools
<wgrant> Excellent
 * StevenK marks as qa-ok
 * wgrant deploys
<StevenK> wgrant: Oh, with the notification too, so it's excellent
<StevenK> wgrant: Sigh, it looks like the poll doctests make use of POST
<StevenK> wgrant: I bet that auditor failure was the usual omg-port-is-in-use-panic-and-catch-fire failure
<wgrant> StevenK: If you haven't worked out what's broken, throw me a list of errors and I'll try to find out
<StevenK> wgrant: I've fixed a few.
<wgrant> StevenK: Any that were clicking submit?
<wgrant> Or are those still proving troublesome?
<StevenK> wgrant: Nope, I've dropped that for now.
<StevenK> wgrant: I can scp the subunit stream to lillypilly if you wish
<wgrant> StevenK: testr failing | utilities/paste
<StevenK> wgrant: http://pastebin.ubuntu.com/1138836/
<wgrant> StevenK: A lot of them seem to be posting manually
<wgrant> Another one calls goBack just before the failure, which is possibly relevant
<StevenK> wgrant: Yeah, I've started on xx-productseries.txt converting it to browser.post
<wgrant> It looks like the actual change must be in zope.app.testing, rather than mechanize or testbrowser
<wgrant> I think
<wgrant> Since http() uses zope.app.testing's HTTPCaller directly.
<wgrant> In fact
<wgrant> I think all the browser-based failures are immediately after a goBack
<lifeless> anyone up for a review - https://code.launchpad.net/~lifeless/python-oops-amqp/misc/+merge/119076 ?
 * sinzui awards wgrant a gold â
<wgrant> sinzui: What have I done?
<wgrant> lifeless: Looking
<sinzui> made bugs public
<wgrant> sinzui: Ah, yeah
<wgrant> sinzui: I also removed ~launchpad-security's subscriptions to about 400 bugs
<wgrant> That were public
<wgrant> We can hopefully do away with the team soon.
<sinzui> I just made qastaging laucnhapd/+sharing look more like I expect production to be.
<wgrant> Great.
<sinzui> We have a lot of bots that I left in place.
<wgrant> I'm trying to clean things up so that we can use sharing as we intend sharing to be used :)
<sinzui> I left many teams in place to keeps subscriptions, then I looked at the bugs, saw they were closed, so I unsubscribed a lot of teams
<wgrant> lifeless: Do you deliberately depend on both bsons?
<lifeless> wgrant: james_w is doing a migration to the 'real' one across everything, incrementally.
<lifeless> wgrant: so this is just dealing with that more or less
<lifeless> wgrant: I've ushed up the fixes james_w asked for
<wgrant> lifeless: Looks good, then.
<wgrant> Thanks.
 * StevenK stabs these tests
<wgrant> StevenK: 'sup?
<StevenK> wgrant: My switch from POST to browser.post() is not going well.
<wgrant> Ah
<StevenK> browser.post() is also triggering NoReferrerError
<wgrant> Is there a referer?
<StevenK> browser.open() ... browser.post() should set one?
<StevenK> Or is my understanding of zope.testbrowser bonkers?
<wgrant> I'm not sure that post uses the current context
<wgrant> It's probably similar to open in that respect
<StevenK> I'm not sure that sprinkling 'browser.addHeader('Referrer: ...')' into the test is the right behaviour either
<wgrant> There's hopefully a prettier way to do that.
<StevenK> ValueError: line 123 of the docstring for xx-productseries.txt lacks blank after ...: '  ...Support E4X in EcmaScript...'
<StevenK> Bleh
<StevenK> What the heck does 'lacks blank' even mean, anyway. :-(
<wgrant> StevenK: It thinks it's a continuation of the previous statement
<StevenK> ... how
<wgrant> Like that, yes.
<StevenK>  lib/lp/blueprints/stories/blueprints/xx-productseries.txt
<StevenK>   Ran 1 tests with 0 failures and 0 errors in 4.433 seconds.
<StevenK> However, that was by adding .addHeader('Referer', ...)
<lifeless> wgrant: stub is curious about the sso link removal project status
<wgrant> lifeless: I have an SSO branch. It works. It has no tests, and IIRC it doesn't handle failure very well, and due to SSO's view structure it makes several XML-RPC requests
<wgrant> So the whole thing needs a lot of refactoring
<wgrant> Some of which landed three months after I proposed the branch
<wgrant> But the LP side works fine, and fundamentally the SSO side is fairly easily doable.
<lifeless> ok so some moderate work to do
<wgrant> Yes
<wgrant> And I think elmo will cry less if LP is never down :)
<stub> Is the LP side a separate service or just the appserver?
<stub> Now I think of it, if we tear out the slony code from the appserver then I think it will happily respond to read only requests when the master is down, because it doesn't need the master to calculate lag.
<wgrant> stub: xmlrpc-private is always master-only at present, but indeed
<stub> Might need a little polish, like ignoring the last write timestamp in the cookie, and no master only mode if lag > 2 minutes
<wgrant> stub: Well
<wgrant> Maybe
<wgrant> Slave-capable things should use the slave if possible. If the slave is lagging too much, it should fall back to the master.
<wgrant> If the master is unavailable, does it want to fail the request, or use the lagging slave?
<stub> Use the lagging slave
<wgrant> Right.
<wgrant> That's my suspicion.
<wgrant> It means we need to tweak things to only request master if they really need it
<stub> We are interested in using up to date master. If the master is down, the lagged slave is still by definition the most up to date data we have
<wgrant> Most XML-RPC requests are probably OK with a slave
<stub> c/using up to date master/using up to date data/
<wgrant> Actually
<wgrant> Hm
<wgrant> It might be similar to the API
<wgrant> Where we want to use the master if at all possible
<wgrant> Because why not be consistent and up to date
<stub> Well, that is a bug
<wgrant> So we want really up to date data, but we don't want to fail if we can avoid it
<wgrant> So MasterPleaseIfAtAllPossibleDatabasePolicy :)
<stub> By the time you receive your data, it might be out of date. We can never guarantee consistency, even from the master
<wgrant> True.
<wgrant> And I guess slave lag should be pretty minimal nowadays.
<wgrant> "Nowadays" being since Monday.
<spm> if the master is down, will the lag result actually show as lagged? I assume yes, but...
<stub> So instead we just give data that is 'recent enough', which can come from a slave.
<wgrant> spm: Yes
<wgrant> spm: The "lag" is the age of the last WAL replayed from the master.
<wgrant> stub: Right.
<stub> The cutoff we care about is 'is the lag greater than my last write', which means we need a session identifier.
<spm> cool. just conscious that the process on the master is what does the laggy updates, aiui
<wgrant> spm: In the old world, yeah
<spm> oh this is new shiny? nm then.
<wgrant> spm: (although the old stuff also wouldn't break in this case: it stored lag plus a last update time, IIRC)
<stub> spm: Its designed that the master isn't particularly aware of the slaves, who they are or what they are doing.
<spm> right
<wgrant> So if the master goes away, the last update time lags, and clients can notice
<wgrant> stub: So I think there's a place for a MasterPlease policy, which is used for eg. recently-POSTed web sessions and xmlrpc-private and all API sessions (until we have a reliable session identifier for API clients), which uses the master unless it's disappeared.
<wgrant> Real write requests would still use the classical Master policy
<wgrant> So would fail during fdt.
<stub> yeah, sounds about right.
<StevenK> MasterIfPossibleDatabasePolicy ?
<StevenK> MasterWithFallbackDatabasePolicy perhaps
<stub> We might be able to just tweak the existing LP policy.
<stub> If the master is unavailable but asked for, give out a slave. If the POST or XML-RPC request or whatever attempts to UPDATE, it will fail with a read only violation. Put some lipstick on that, make it a 503 status code and we might be good.
<lifeless> stub: btw, do we set the feedback setting on the slaves?
<stub> lifeless: yes, I didn't want to mess with behaviour too much just yet
<stub> lifeless: it is probably how we will keep it too.
<lifeless> hot_standby_feedback is the one I mean; defaults off but looks like we may want it on
<stub> it is on for us, yes
<lifeless> cool cool
<adeuring> good morning
<ev> is there anyone I need to notify if I'm going to do a large number of lplib API calls in a test?
<wgrant> ev: How many is 'large', what sort of calls, and can you do it on (qa)staging instead/first?
<ev> wgrant: apols, I just realized that was hopelessly vague. So I have 81,455 crashes. I'm going to get the package for each and all of the relevant dependency packages. I then need to call getPublishedBinaries for each of those (but I'll cache calls on a key of package + series)
<ev> and yes, I can do it on staging first. Is that woefully slow by comparsion?
<ev> comparison*
<cjwatson> Is this a one-off or in a frequently-run test suite?
<wgrant> I forget whether getPublishedBinaries is terrible or not
<wgrant> It should be reasonably fast even on staging if it doesn't do any stupid substring matching
<ev> cjwatson: it will be run daily, but for right this moment it's just a one off to get some basic data
<ev> wgrant: I have exact_match set, though I do realize there could be substring matches elsewhere
<wgrant> I think that should be most of it
<wgrant> So, I'd try it on staging first.
<wgrant> It should be fairly quick once it's warmed up
<ev> excellent
<wgrant> Although
<ev> wgrant: should I notify webops as well?
<wgrant> At 82k crashes
<cjwatson> getPublishedBinaries - which publication statuses?
<wgrant> Presumably there's lots of deps?
<wgrant> For each?
<wgrant> Ah, but if you cache...
<lifeless> ev: no need to notify webops if you are doing these alls serially.
<lifeless> ev: its a less than 1% increase in traffic.
<ev> lifeless: it is serially, and hi :)
<lifeless> ev: if you're doing it in parallel, thats another matter :)
<lifeless> ev: oh hi :)
<cjwatson> If it's just "Published", it would be better to just get the relevant Packages files from a mirror and parse locally ...
<wgrant> It's not going to be a problem, but we can probably make it faster :)
<wgrant> Yeah
<wgrant> That's the thing
<lifeless> ev: btw, I did get you commit access to lp:python-oopsrepository right?
<wgrant> I don't see why you don't just use the normal indices
<cjwatson> If you're trying to get historical publication information for some reason, that would be different
<ev> lifeless: yes, I've been terrible and having merged back yet. Will do today.
<cjwatson> Like when they were superseded or something
<lifeless> ev: we've got webops, u1, ca and LP all using one python-oops-tools system now
<lifeless> ev: so the interest in migrating to a cassandra backend is growing.
<ev> lifeless: though I'll throw it up as a MP first, just so people have a chance to tell me no before I merge it in
<ev> lifeless: excellent!
<ev> lifeless: yeah, I've had brief conversations with james_w about it, and you I believe :)
<ev> cjwatson: historical information. It's about creating an "ideal" crash line
<ev> that is, crashes where every package in the dependency chain that apport lists was up to date at the time of the crash
<ev> cjwatson: I've forwarded you a mail I sent to lifeless explaining the basic idea
<ev> the code will be something akin to this http://paste.ubuntu.com/1139323/  (at least for the test)
<cjwatson> Can you use created_since_date
<cjwatson> ?
<ev> since we now have to calculate the unique users seen in the past 90 day period for the denominator, and that's not a calculation that can be done quickly, the whole thing will be calculated once a day for the day that's passed
<cjwatson> Consider ordered=False too, since you don't appear to need ordered results
<ev> (with the "actual" line being total crashes divided by unique users in 90 days and the "ideal" line being total crashes that were on up to date systems divided by unique users in 90 days)
<ev> cjwatson: created_since_date doesn't work as far as I can tell for the reason mentioned in the code comment. But maybe I'm wrong?
<ev> cjwatson: ordered=False> excellent, will do
<wgrant> ev: Would you be better served by maintaining a full set of when each (name, version, arch) first appeared?
<wgrant> Rather than querying most of Ubuntu's history every day
<cjwatson> Yeah, surely there's some kind of inter-run caching possible here
<ev> wgrant: so cache the package name, version, and arch tuple into cassandra?
<cjwatson> It's not like binary_package_version or date_published on past publications are going to change
<wgrant> Right
<wgrant> The history won't change
<lifeless> ev: is that 81K distinct crash signatures?
<lifeless> ev: or 81K reports ?
<ev> yeah, sure
<cjwatson> date_superseded might of course, but you aren't looking at that
<wgrant> You can easily keep a local copy of the relevant bits of history
<ev> lifeless: 81K reports for a day period
<ev> which seems about average
<lifeless> so few
<wgrant> And use created_since_date to just bring in all the new records every $interval
<cjwatson> It might even be worth doing one getPublishedBinaries call with created_since_date for the whole interval, rather than one per binary name?
<ev> cjwatson: right, just when it was published
<wgrant> cjwatson: Exactly.
<wgrant> You keep a local database of (name, version, arch, date_published/date_created), then every $interval ask for all the new publications since the last time you asked - a bit
<cjwatson> It's per-series so the initial setup will have some giant returned result set, but only back <six months
<ev> where interval is the daily run of this code to generate the totals for the ideal line for the day past, right?
<mpt> lifeless, once this publishing history discussion ^ is sorted, we have a fun question about calculation of that "ideal" line
<wgrant> ev: Well, it doesn't really have to be this code
<wgrant> ev: The update process is separate.
<lifeless> mpt: cool
<wgrant> ev: You can rapidly query your local cache of the relevant info whenever.
<lifeless> FWIW I don't care whether ev caches the data or not.
<lifeless> datastores are data stores.
<ev> wgrant: okay, but still iterating over the same data set, right? My point is that it's not building a cache for packages it's not going to care about. Just ones for the oopses and their dependencies that we've seen day by day
<ev> lifeless: you'd argue for talking directly to LP without a cache?
<lifeless> LP can trivially handle the load; the current API may be inefficient, but its got no intrinsic reason to be so.
<wgrant> lifeless: Datastores are datastores, but the LP API is about as inefficient as it gets.
<lifeless> ev: I would start with the simplest thing possible.
<wgrant> Cache locally => hundreds of times faster
<lifeless> ev: and add complexity only when I had to.
<wgrant> ev: How many packages is that? The closure of dependencies could be fairly large.
<ev> wgrant: I can calculate an approximation based off a days run
<lifeless> ev: e.g. start by just talking to LP; then the next step either make the LP API faster (often easy, lots of unoptimised stuff) or add a local store.
<ev> I'll just add that to the set of things to count in this sample
<ev> lifeless: okay
<lifeless> wgrant and cjwatson may be entirely correct that doing it via LP will be terrible (and the hidden HTTP requests launchpadlib does are likely to prove them right :P)
<lifeless> but its still better to deliver something soon and then iterate IMNSHO
<ev> absolutely
<cjwatson> I wouldn't be making these suggestions if I thought they were hard to implement :)
<cjwatson> FWIW
<lifeless> cjwatson: sure, and I don't think they are necessarily wrong.
<cjwatson> as in, it's what I'd do and I expect writing the code for it would be quicker than waiting for the initial "easy" but slow version to complete
<lifeless> I'm just rather aware about the political side of getting this data as soon as possible, due to that u-r thread of doom
<cjwatson> so I think in this case the "easy" version is a false economy
<lifeless> cjwatson: I'm not suggesting using launchpadlib directly because its easier, but because it has less moving parts.
<cjwatson> even so
<ev> I suspect I'm going to lose the thread of doom. There's no way I can get the changes to apport for a single pair of dialogs done by the end of the day. Well, I can probably have the code done, but then there's getting pitti to magically appear and review it, and it's quite deep.
<lifeless> FWIW my initial suggestion to ev was a dedicated API to do the heavy lifting in LP.
<ev> indeed, I wasn't expecting to get to such optimization just yet as this conversation started off discussing an initial test
<ev> so, mpt. Maths
<ev> gah, NOT A PLURAL WORD
<mpt> Road works
<lifeless> Mathematics
<ev> :)
<mpt> lifeless, so. The graph aims to show the average number of crashes per calendar day. (Making it per 24 hours of uptime, to eliminate the spike during weekends, is a problem we've tabled for now.)
<mpt> lifeless, to do that we take the number of errors reported each day, and divides it by an estimate of the number of machines from which errors would be reported if they happened.
<mpt> (Now I'm the one adding extra Ses.)
<mpt> As an estimate of "the number of machines from which errors would be reported", we use "the number of machines that reported at least one error any time in the 90 days up to that day".
<mpt> That slightly under-counts because of machines that were active but lucky enough not to have any errors. And it slightly over-counts because of machines that were destroyed or had Ubuntu removed from them during that 90-day period.
<mpt> Hopefully the under-count and over-count cancel each other out.
<mpt> Anyway.
<lifeless> uh,
<lifeless> it massively undercounts
<lifeless> but thats a different point
<mpt> ok, why does it massively undercount?
<lifeless> you want 'size of the population of machines with error reporting turned on and users that don't always hit no'
<mpt> "users that would usually hit yes", but yes.
<lifeless> you are getting '90 sliding observation of [machines with error reporting turned on and users that don't always hit no] that encountered 1 or more errors and reported them'
<lifeless> mpt: how often does the error reporting message come up for you
<mpt> lifeless, about three or four times a week.
<lifeless> mpt: so for me it comes up -maybe- once a month. I think twice since precise released.
<mpt> lifeless, if it turns out that the average is anywhere close to 1/90, then we'll need to increase the 90-day period to more than that.
<lifeless> mpt: the underreport is due to all the machines that don't encounter errors at all
<lifeless> and you can't tell how big the under report is because the sample you have is only from reporting machines.
<mpt> So? So is the numerator.
<lifeless> I mean, machines that are biased to report at a frequency of 1 in 90 days or great.
<lifeless> *greater*
<lifeless> mpt: I don't follow how that matters
<lifeless> mpt: you said "As an estimate of "the number of machines from which errors would be reported"
<mpt> yes
<lifeless> mpt: I'm saying that its seems likely to me that your estimate is very low. We can test this theory.
<lifeless> ev: whats the current unique reporting machine count for the last 90 days ?
<mpt> We're assuming the number of errors/day is a unimodal distribution (probably Poisson), and that there aren't a lot of machines that have zero errors in a 90-day period
<ev> lifeless: I don't have that yet. The query was taking more than 12 hours to back-populate so I need to come up with a quicker approach.
<ev> but
<mpt> (where "aren't a lot" = "are fewer than 1.1%"
<mpt> )
<ev> oh, nevermind. I thought I had a quick way to get the unique machines for all releases for the past 90 days
<lifeless> mpt: For a single individual, the distribution should be poisson, unless the way they use their machine influences crash rates, in which case it won't be.
<ev> but it's not as quick as I thought
<lifeless> mpt: this is a distraction; we can investigate whether we have a underestimate or not separately.
<mpt> lifeless, I think if we are massively under-counting, then the next rollout of the graph will probably show an average of close to 0.01 errors/day, because it was dominated by machines that reported only one error in that 90-day period.
<mpt> Anyway, distraction, yes.
<mpt> For the ideal line, we want to show the effect of people installing updates or not.
<lifeless> mpt: that doesn't necessarily follow. We know our estimate should match some fraction of the precise userbase, where that fraction is the number of users that leave the tick box on and click continue.
<mpt> Not how quickly they are, but how much their promptness/tardiness affects Ubuntu's reliability in the wild.
<lifeless> mpt: we can independently estimate that fraction, multiple by the separate estimate of precise desktop users, and compare to the errors.ubuntu.com estimate.
<lifeless> mpt: if they different substantially, one or more of the estimators is wrong.
<ev> (so I can quickly get the number of unique users that have ever report crashes, so something just over 120 days, and that's 1,975,010)
<lifeless> mpt: 81K*90 = 7.2M, which is too low I believe.
<lifeless> ev: great.
<lifeless> That means we're massively underestimating :)
<mpt> lifeless, it's much less than 81K*90, because many of those machines are the same in multiple days
<lifeless> mpt: sure, I used 81K*90 as an upper bound
<lifeless> mpt: because if it was still too low, there is no way that any answer ev gave could be higher.
<mpt> sure
<lifeless> ok, so ideal line.
<lifeless> so you want to show the number of crashes per day that would be saved if users updated ?
<mpt> If we calculate it right, the "ideal" line will be like a smooth + lagged version of the "actual" line.
<mpt> wait, no, other way around.
<mpt> The "actual" line will be like a smooth + lagged version of the "ideal" line.
<mpt> If we issue a fix for an error that's causing 50% of the errors reported, the "ideal" line will drop down to half its previous level immediately, and the "actual" line will drift down slowly to meet it.
<mpt> Conversely, if something goes wrong and we issue a really crashy update, the (now-misnamed) "ideal" line will spike up, and the "actual" line will drift up to meet it.
<lifeless> Sure
<lifeless> s/ideal/projected/
<lifeless> potential
<lifeless> possible
<mpt> something like that.
<mpt> "If all updates were installed", we call it in the page currently
<mpt> just for clarity :-)
<mpt> Now, for this we do the same kind of division as before
<lifeless> yeah
<mpt> The numerator is the number of error reports on that day, for which that package and all its dependencies were up to date
<mpt> But we're not sure what the denominator should be.
<mpt> I thought that it should be the same denominator as the "actual" line, the estimate of all machines that would typically report errors if they encountered them.
<mpt> That passes the sanity test that if every Ubuntu machine was perfectly up to date, the "actual" and "potential" lines would be exactly the same.
<lifeless> uhm
<lifeless> why calculate it from scratch ?
<lifeless> I mean, the way you expressed it: 'issue a fix for an error that's causing 50% of the errors reported, the "ideal" line will drop down to half its previous level'
<mpt> If you mean the denominator, I'm not suggesting calculating it from scratch
<mpt> oh
<lifeless> when everything is up to date / there are no fixes available then ideal == actual
<mpt> Because errors are not evenly distributed over updates. For example, there are a bunch of machines out there (we don't know how many) that install only security updates, not other updates, and security updates may be more or less likely than average to fix reportable errors.
<mpt> s/over updates/over updated packages/
<jml> any buildout folks around?
<lifeless> jml: passingly familiar
<lifeless> mpt: ideally all machines install all updates right?
<mpt> lifeless, anyway, I *think* (though I'm not sure) that using the total number of 90-day-active machines may cause the "projected" line to be too low. The slower people install updates, the lower the number of errors will be from up-to-date packages, but that doesn't mean those up-to-date packages are more reliable.
<mpt> lifeless, ideally, yes.
<jml> we have a Python project that comes with executable files. buildout (and pretty much anything that installs from eggs) loses the executable bit. afaict, this is due to a bug in stdlib zipfile where the external_attr part of the ZipInfo is ignored on extraction.
<mpt> lifeless, but remember, we're not trying to measure the proportion of machines that are all up to date, we're trying to measure how much reliability is affected by packages being out of date.
<lifeless> ok, so lets talk reliability estimators
<mpt> If a package is way out of date but doesn't generate any error reports, that's fine as far as this graph is concerned.
<jml> my immediate plan is to carry a temporary fork of distribute that corrects extract_zipfile in setuptools.archive_utils to chmod after extraction.
<lifeless> jml: executable files in the package? or scripts that should be executable after install
<jml> lifeless: the first one.
<lifeless> jml: thats frowned upon. lintian will whinge, for instance.
<lifeless> jml: why do you want that ?
<jml> lifeless: I am not changing it.
<lifeless> jml: the question is too abstract.
<jml> lifeless: but, since you asked, it's because pkgme's interface to backends is by spawning sub-processes
<jml> a backend is just a couple of executables
<jml> if those backends happen to be written in Python, then distributing them is very tedious, thanks to this bug in zipfile
<lifeless> jml: I don't think its a bug.
<jml> lifeless: tarfile preserves permissions
<cjwatson> lifeless: lintian> that rather depends on where the files in question are installed, and whether they include #!
<jml> lifeless: why should zipfile not?
<lifeless> jml: the interface for running things from a python package is python -m foo.bar
<lifeless> jml: not /usr/lib/pythonx.y/dist-packages/foo/bar.py
<jml> lifeless: it's a script
<lifeless> jml: if its a script, the installation of it should be putting it in the right bin directory, updating the interpreter and making it executable for you.
<lifeless> cjwatson: 'in a python package' is well defined, and #! in those files also warns IIRC.
<cjwatson> Oh, you meant Python package, right, you just said package :)
<jml> lifeless: pkgme works by searching a list of paths that contain backends and then running the executables it finds there.
<cjwatson> Though I'm not sure I believe your IIRC without proof, as I don't immediately see evidence of that in lintian
<lifeless> cjwatson: I may well be horribly mistaken
<lifeless> jml: so, with buildout, that won't work unless those scripts have no dependencies that buildout is supplying.
<lifeless> jml: you need to run them via bin/py <path to python file> or bin/py -m <python module path>
<lifeless> jml: otherwise you will get the system interpreter path.
<jml> lifeless: yes, this is true of virtualenv too
<lifeless> jml: To me, this makes the issue you are actually facing irrelevant.
<lifeless> jml: Have I missed something ?
<jml> we have some work-around for that atm
<jml> which I'm having trouble locating
<jml> ah yes. it's hideous and won't work with buildout.
<lifeless> so, we can talk about the mode bg
<jml> but I know something worse that will
<lifeless> but I don't think it will help you will it ?
<lifeless> jml: its late, I need to go. I'd be happy to design something simple that will work for you, just not now.
<jml> lifeless: ok, thanks.
<jam> trying to go to: https://launchpad.net/projects/+review-licenses is timing out for me. It worked for a bit yesterday (until I reviewed a bunch of projects and then reloaded)
<jam> I submitted a bug, is there much else to do (I'm trying to take care of the review queue, etc)
<wgrant> jam: It's working for me. Tried refreshing a couple of times?
<wgrant> My superpowers may be causing permission checks to be skipped though
<rick_h_> yea, 6 loads here all timeouts
<rick_h_> timeout backs to the is_valid_person ValidPersonCache.get and all storm from there on out
<StevenK> Sounds like preloading is in order then
<jam> rick_h_: yeah, my OOPS shows 1.7s run in a single query, though running it on staging completes in 17ms...
<wgrant> jam: OOPS ID?
<jam> wgrant: https://oops.canonical.com/oops/?oopsid=OOPS-3abca09f555663402bbd26a37805e0a0
<wgrant> Ah
<wgrant> I bet it's the private team privacy adapter
<wgrant> But sooooo many queries
<wgrant> Indeed
<wgrant> The insane private team privacy rules
<wgrant> jam: Can you see it now?
<wgrant> I've removed the only obvious private team from the listing
<jam> wgrant: timeout again
<jam> wgrant: new oops: https://oops.canonical.com/oops/?oopsid=OOPS-5c67f643d64f10768d671842a80680e0
<wgrant> Ah, there's another one
<wgrant> Try now?
<rick_h_> loads now
<wgrant> Yeah
<wgrant> There were two private teams on Canonical projects
<jam> yay
<jam> though there still seems to be death-by-thousand-cuts on that page
<rick_h_> good to know, shold we get a hard timeout exception for the review page for now?
<rick_h_> so it's usable for maint?
<jam> If you look at the new oops, it has one query repeated 56 times.
<wgrant> jam: Yeah, fortunately they're pretty small cuts in this case.
<wgrant> rick_h_: Maybe. But AIUI czajkowski is back next week, so it's not maintenance's responsibility any more
<wgrant> And she is immune to these issues
<rick_h_> ah, wasn't aware she was immune
<wgrant> This is useful ammunition in my war against lifeless' overcomplicated private team visibility rules :)
<jam> wgrant: weird, I see the helensburgh project in there, even though it succeeded in loading... :)
<wgrant> jam: It's driven by a private team, not owned by one
<wgrant> That page only shows owners
<wgrant> (well, and registrantsa)
<rick_h_> huwshimi: ping, do we have a login for the web balsmiq?
<rick_h_> huwshimi: or did you just get the air version running?
<huwshimi> rick_h_: I think so, I'll forward the details.
<rick_h_> huwshimi: ty
<cjwatson> wgrant: Care to re-review https://code.launchpad.net/~cjwatson/launchpad/archive-getallpermissions/+merge/117606 ?  I'm becoming good friends with StormStatementRecorder.
<wgrant> cjwatson: Looks good, thanks.
<wgrant> cjwatson: (although you might want to rename allPermissions to getPermissionsForArchive or permissionsForArchive or something)
<cjwatson> Mm, yeah.  The almost-but-not-quite-the-same names between Archive and ArchivePermission are a tad confusing.
<wgrant> Yeah
<wgrant> ArchivePermissionSet's are wrong
<wgrant> Because method names are meant to be verbs
<wgrant> But consistency might be best there for now
<deryck> rick_h_, hey, how about at 15 after?  in roughly 5 minutes, for our call?
<rick_h_> sure thing
<lifeless> wgrant: unoptimisde thing is slow isn't a surprise
<Ergo^> evening
#launchpad-dev 2012-08-11
<wgz> do I need a l-osa to qa a bug watch change, to manually run the update script on qastaging?
<wgrant> wgz: Given how trivial that change is, I'd probably avoid QAing it. staging is somewhat designed to not run checkwatches at all, since it can easily DoS remote bugtrackers.
<wgrant> And it doesn't have a local debbugs mirror
<wgrant> (but yes, if you really want to QA it, you'll need a sysadmin)
<wgz> okay, I'll wimp out on your advice then :)
<wgrant> :)
<cody-somerville> is staging down?
<cody-somerville> 'Technically, this is a 503 error and has been caused by our database being temporarily offline. '
<cody-somerville> also, https://api.dogfood.launchpad.net/ seems to have an invalid ssl cert
<lifeless> cody-somerville: dogfood is expected, its crap
<cody-somerville> lifeless, indeed. I'm testing error handling code
<cody-somerville> lifeless, so thought it would be perfect :P
<cody-somerville> (error handling code in api script that is)
<lifeless> this is about the time a staging restore should be taking place, so thats possible
<lifeless> however, its sunday, so meh :)
<wgrant> lifeless: (yes, staging's meant to be down, as it's restoring and/or failing to restore around now)
#launchpad-dev 2012-08-12
<cjwatson> Hmph.  Have I managed to curse https://code.launchpad.net/~cjwatson/launchpad/sync-source-nonexistent, or is the branch scanner not running?
<wgrant> cjwatson: It's cursed. Rename it away and push again
<wgrant> [2012-08-12 09:23:19,656: INFO/PoolWorker-2] Job resulted in OOPS: OOPS-cb085746330124527e92bb196b6ed7e4
<cjwatson> I can't delete it?
<wgrant> cjwatson: Probably not, no.
<wgrant> cjwatson: It's likely already have all the branchrevisions, just no last_scanned_id
<wgrant> And it's not possible to delete 125k branchrevisions within the 5s timeout.
<cjwatson> wgrant: Did you ever get round to giving me a patch number for bug 745799?  http://irclogs.ubuntu.com/2012/08/04/%23launchpad-dev.html#t01:49
<_mup_> Bug #745799: DistroSeries:+queue Timeout accepting packages (bug structural subscriptions) <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/745799 >
<cjwatson> I don't see it in http://bazaar.launchpad.net/~launchpad/+junk/dbpatches/view/head:/allocated.txt
<wgrant> cjwatson: Sorry, 2209-27-2 it is.
#launchpad-dev 2013-08-05
<cjwatson> wgrant: Have you had a chance to look at any of my buildd branches?
<wgrant> cjwatson: Vaguely. Was pretty sick and avoiding nasty state machines all last week, should get to them properly in a day or two.
<cjwatson> OK, thanks
<cjwatson> ubuflu?
<wgrant> Probably
<wgrant> It struck about two hours before I landed in Melbourne
#launchpad-dev 2013-08-06
<stub> lifeless: Right. I was copying from juju...
<lifeless> stub: juju's client specifically, I imagine?
<stub> lifeless: yeah, was wondering how to stuff in settings and it is the other tool I use that needs to talk to openstack stuff.
<stub> 'cause I didn't want to create yet more config files in Launchpad or put secrets in the existing ones.
<lifeless> stub: well we have secrets in the existing ones already
<lifeless> stub: so I would just do that, and solve that existing problem another time.
<wgrant> cjwatson: What happens in the case that a reset doesn't fix the builder?
<wgrant> Ideally it'd fail eventually, but I'm not sure that's implemented atm.
<wgrant> Trying to work out in which cases the new behaviour is worse than the current one.
#launchpad-dev 2013-08-07
<cjwatson> wgrant: I increment the builder failure count before calling resetOrFail, so won't the slave scanner eventually notice that it's gone over the failure threshold and call failBuilder?
<cjwatson> (I didn't touch assessFailureCounts in this pass; figured the upheaval was large enough as it was.)
#launchpad-dev 2013-08-08
<wgrant> StevenK: Bug #1208866 might be of interest
<_mup_> Bug #1208866: PackageDiff.requester not set properly for recipe build uploads <package-diff> <recipe> <soyuz-upload> <Launchpad itself:Triaged> <https://launchpad.net/bugs/1208866>
<StevenK> wgrant: Because the uploads are unsigned
<wgrant> StevenK: Yes
<wgrant> signing_key is none; you need to go through SPR.SPRB.requester instead
<wgrant> Not sure of the best place to do that
<StevenK> wgrant: http://pastebin.ubuntu.com/5961229/
<StevenK> wgrant: I was also pondering destroying IPerson.isUploader() and subsuming it into its remaining callsite
<wgrant> StevenK: That would seem reasonable; it never belonged on Person anyway.
<wgrant> Distribution, potentially
<wgrant> But if there's only one callsite then inline it.
<StevenK> Which was my plan, yeah
<StevenK> lib/lp/registry/doc/person.txt, lib/lp/registry/interfaces/person.py, lib/lp/registry/model/person.py, lib/lp/soyuz/adapters/notification.py
<StevenK> So inline it goes
<jtv> Hi StevenK, hi wgrant.  It's a shame we never meet up any more.  I'm rapidly going through a kilo bag of drop bears and mint crosses.
<wgrant> Heh
<wgrant> Hi
<wgrant> jtv: What's a mint cross?
<jtv> Similar, but more sweet than salty (still sugar-free in this case, like the drop bears) and with mint in it.
<lifeless> jtv: o/
<jtv> Hi there lifeless!
<jtv> How are the various kittens?
<lifeless> they are cats now; and great fun
<jtv> Too many yappy dogs around here.  Don't see the cats much any more.  :(
<lifeless> it's winter here, so we're waking up at 5:30/6am to cats climbing in bed for warmth
<jtv> That's always sweet.
<jtv> I knew these two sisters in the Northeast (Siamese) who would do that.  We had 9 degrees one winter night.
<stub> Squirrels and cats here
<stub> And rats and cockroaches :)
<jtv> You have squirrels too?  Bastards steal my mangoes.
<jtv> Eat 'em right off the tree.
<jtv> I have a bat, too.
<stub> I cook my squirrels first
<jtv> What was that joke I saw the other day...  Bat father putting his son to bed.  His son makes him check that Ozzie Osbourne isn't under the bed.
<stub> Looking at feature flags, to discover lib/lp/services/features/doc/features.txt is 0 bytes...
<StevenK> Haha
<StevenK> Tempted to drive-by delete that in the branch I have up
<stub> Should I be attempting to use a feature flag from twisted, or just stick with a config value?
<wgrant> Should work fine from the librarian
<lifeless> stub: there was docs for the feature flags; no idea why that file is empty
<stub> StevenK: Or pull the docs in from lib/lp/services/features/__init__.py - seems to be in there
<stub> wgrant: I'll need to ensure it is all populated somehow before starting the reactor, or we will end up blocking on db calls.
<wgrant> stub: But only once.
<wgrant> So it's not that bad.
<lifeless> wgrant: eh
<lifeless> once per flag context
<lifeless> you're not proposing to have to restart to activate changes?
<wgrant> We've not much choice. We can't sanely query every time.
<lifeless> loopingcall once a minute? call into db from a thread?
<wgrant> If we want to do that, yeah.
<lifeless> well, not my problem :)
<lifeless> but, restarting services on config change is a bit of a PITA
<lifeless> you're not putting creds in the feature flags are you?
<lifeless> cause that would be bad :)
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/sprb-findpersontonotify/+merge/179093
<stub> lifeless: it needs to be restarted to reload flags, or it will block.
<stub> Unless I do a largish refactor to allow me to make the flag lookup deferToThread, but since the purpose of the flag would be to avoid using the new code path and instead stick to the antique and stable one...
<stub> ioc, yeah, separate task to refresh the flags
<stub> Do we care if the Librarian supports range queries? It should be irrelevant now all access is via squid?
<lifeless> stub: squid will pass on range queries if it is a miss
<stub> 'Cause I can't support range queries without cargo culting a pile of code from twisted.web
<lifeless> stub: you can reply with the full content, but if that's not cachable for some reason you'll waste a bunch of in-dc bandwidth
<lifeless> I think it's probably fine to not support them
<stub> lifeless: yeah, but if squid passes on the range query and receives the entire file, will it chop it up and send only the requested bits to the client?
<lifeless> it will try
<lifeless> bastard clients can cause issues,.
<lifeless> like, if they ask for the last byte, then the first byte, then the middle, then the second last, then ...
<lifeless> in one request
<stub> I'd trust squid's implementation over the twisted one in any case...
<lifeless> anyhow, if you give squid the whole file, and it's cachable, squid can cache it and then satisfy from the cache
<stub> especially as I'm not going to allow rewinding my swift stream ;)
<lifeless> I *think* that still happens on misses, but if it doesn't, the client is required to deal anyhow.
<lifeless> since swift data can be compressed I think that's very wise.
#launchpad-dev 2014-08-04
<waj> I'm trying to create ubuntu packages for a new language that we're developing but the problem is that the language is written on the language itself so I need a precompiled compiler to create the package. I created then a second package with just the binary.
<waj> So the "crystal" package has a build dep to "crystal-zero"
<waj> but crystal-zero has only the binary for amd64
<waj> and seems like during the generation of the source packages for "crystal" the process is run on a 32 bits machine and that makes the process to fail
<waj> https://launchpadlibrarian.net/181474577/buildlog.txt.gz
<waj> is there any way to avoid this issue with the automated recipes?
#launchpad-dev 2014-08-06
<lifeless> wgrant: you applied to manage lp on ohloh right?
<lifeless> wgrant: (vs I just got trolled by someone random :P)
<wgrant> lifeless: I did, thanks.
<wgrant> cjwatson: Don't you need to remove --archive=ubuntu from the config file in https://code.launchpad.net/~cjwatson/launchpad-buildd/unhardcode-distribution/+merge/228114?
<wgrant> Or are you relying on the second option override the first?
<cjwatson> The latter.  Although that strategy is no longer necessary now that the master-side change has been rolled out everywhere.
<wgrant> Yeah.
<cjwatson> At the time I wasn't sure what the deployment order would be.
<cjwatson> Any objection to me nuking the Xen bits of BuilddFixing (i.e. most of it)?
<wgrant> cjwatson: Murder.
<cjwatson> Looks like builddfizttzer is still marginally useful; it has a small number of non-virt-suitable actions.
<cjwatson> builddfitzer, indeed.
<wgrant> jelmer: Do you recall if the side-effect that I mention in comment #1 of https://bugs.launchpad.net/launchpad-buildd/+bug/1242796 is deliberate?
<jelmer> wgrant: including the epoch in the build directory name was not an intentional change
<wgrant> jelmer: It's not quite that. The tree is initially built as TEMPLATE, but then just before the package build it gets renamed to PACKAGE-VERSION as normal.
<wgrant> --no-build used to exit after the rename to PACKAGE-VERSION, so it never got renamed back to TEMPLATE.
<wgrant> Which meant that 'bzr dailydeb --no-build' used to leave a sensibly named directory.
<wgrant> But r158 causes --no-build to just skip the package build -- it still does the rename back to TEMPLATE.
<jelmer> oh, right
<jelmer> yeah, that's not intentional
<wgrant> I didn't think so. Thanks.
#launchpad-dev 2014-08-10
<fully_human> Hello, everyone. I'm developing an application that uses the launchpad API. Is there a test username I can use so that anyone who finds their username in the code won't be PO'd?
<fully_human> Github uses 'octocat.' Is there one similar for Launchpad?
<xnox> fully_human: most api is accessible read only without authentication... and there is staging server where you can operate against your own accounts without affecting real data.
<fully_human> Right. I'm just wanting to get user data based on an id without any log-in. I'm hoping on making my code, which involves unit tests, public.
<fully_human> I just don't want anyone reviewing the code to see their username and get upset.
<xnox> fully_human: i'm not sure what you are asking for, but currently logged-in user can be accessed via +me alias.
<fully_human> This is a question only about respect towards other developers. Many API's have a "test username" so that you're not picking on some random user of the  site (octocat, for github, for example).
<fully_human> Does launchpad have a "test username"?
<xnox> fully_human: there are plenty of bots in launchpad, but they are all real... e.g. https://launchpad.net/~janitor closes all bug reports from ubuntu uploads.
<fully_human> Ah, bots...that's the term I was looking for...okay, thanks.
<xnox> fully_human: but your unit tests should not be hitting api.launchpad.net, instead you should mock, use your own launchpad instance, or use https://staging.launchpad.net/
<fully_human> Okay, thank you. I'll look into it. :-)
#launchpad-dev 2015-08-03
<blr> wgrant: so it looks like $LANG is unset in a hook execution context
<blr> which can lead to utf-8 weirdness in some libraries (envdir uses codecs.open, rather than open for some reason)
<blr> as a workaround I'm forcing en_US.UTF-8
<wgrant> blr: What does it break?
<blr> wgrant: codecs.open drops back to ASCII, even in 3.4 weirdly
<wgrant> As it should.
<wgrant> But does that break anything?
<blr> wgrant: it breaks envdir
<blr> setting LANG in the charm works, it just seems like an odd thing to have to do
<blr> wgrant: is it reasonable to expect LANG in the juju context?
<wgrant> blr: https://github.com/juju/juju/issues/133
<wgrant> blr: But why does envdir care? Is it writing out non-ASCII?
<blr> wgrant: it builds it's setuptools description from a few different files, presumably one of them contains non ascii characters
<blr> https://github.com/jezdez/envdir/blob/master/setup.py
<blr> not sure why he's used codecs.open there
<mwhudson> today's lesson: cross compiling go from pcc64le to amd64 doesn't work
<cjwatson> blr: C.UTF-8 is a better choice; unlike en_US.UTF-8, it's available on all Ubuntu systems, even the most minimal ones, without needing to install anything extra
<blr> cjwatson: excellent, thanks I'll use that.
<cjwatson> wgrant,blr: I'm rather blocked on reviews, if either of you get some time.  I'm going to sidestep that today by having a go at some more of the BaseMailer stuff.
<wgrant> cjwatson: I have snapbuild-basic-model pretty much done, just wanted to give it a few hours to percolate before I actually hit the r=me. Will look over again now.
<wgrant> Meant to do it earlier but rabbitmq ate me.
<wgrant> The whole privacy thing worries me.
<wgrant> But it is little more intractable than eg. recipes.
<cjwatson> Hopefully a good bit better since it doesn't involve descending into Soyuz.
<cjwatson> Converting team notifications to BaseMailer looks pretty tractable and likely to be an improvement in terms of consistency/filterability, but is involving a lot of very careful thinking!
<cjwatson> (Not least because each notification has potentially multiple teams involved, so I'm having to think carefully about which one gets the @ notation in X-Launchpad-Message-Rationale.)
<wgrant> I don't actually recall if we send two if you're on both ends.
<wgrant> I'd think not, but I don't know.
<cjwatson>         # The member's email address may be in admin_addrs too; let's remove
<cjwatson>         # it so the member don't get two notifications.
<cjwatson> Which is just as well as otherwise it's a pain to BaseMailerify.
<cjwatson> Though, hmm, I wonder how to handle the multiple templates.
<wgrant> It'll have to be two separate mailer invocations, I suspect.
<cjwatson> Yeah, I don't think I can convert notify_team_join into a single mailer instance.
<blr> morning
#launchpad-dev 2015-08-04
<wgrant> blr: How's it looking?
<wgrant> cjwatson: Did you work out your team mail stuff?
<cjwatson> wgrant: No serious blockers but I got distracted by landing all the things yesterday.
<cjwatson> Hm.  I just found RecipeBuildBehaviour._extraBuildArgs and config.builddmaster.bzr_builder_sources_list.  I wonder if I should replace my tools_archive stuff with that?
<wgrant> Oh, I thought that had been removed.
<wgrant> Oops.
<wgrant> We haven't used it in years.
<cjwatson> tools_archive is in one way a bit more elegant, but it doesn't support specifying a different series, nor does it support specifying a sources.list somewhere that isn't the same LP instance.
<cjwatson> I assumed it had been removed ages ago and hadn't even looked; I only ran across it by accident
<cjwatson> I'm inclined to push the bzr_builder_sources_list stuff down to get_sources_list_for_building and remove tools_archive.  It's basically the same as the deprecated OEM external dependencies stuff.
<cjwatson> Except per-build-behaviour rather than per-archive.
<wgrant> Right, sounds reasonable to me.
<wgrant> Sorry, I hadn't realised it was still around.
<cjwatson> NP
<cjwatson> RecipeBuildBehaviour puts it after the primary archive in sources.list, but I think just before (as tools_archive) is conceptually slightly better.
<wgrant> Agreed.
<cjwatson> Little difference in practice unless the same version ends up in the primary archive.
<wgrant> One day I am going to hack Zope to allow deep views.
<wgrant> +webhooks, +webhook/foo, +new-webhook
<wgrant> Madness
<cjwatson> wgrant: Your celery tests seem very unreliable in buildbot ...
<wgrant> cjwatson: Yes, I've relaxed the two unreliable ones in my latest branch.
<wgrant> buildbot really can be very slow at running code :(
<cjwatson> wgrant: Were you thinking that I should have a straight-through test that there's failure-counting if the snap source has been deleted, or is it enough to trust the general scan/scanFailed behaviour of builddmaster and check for the AssertionError on dispatch?
<wgrant> cjwatson: I think the latter is fine.
<wgrant> buildd-manager tests are awkward, and that behaviour is already reasonably well tested.
<cjwatson> Good, that's what I thought/hoped.
<wgrant> I'm not that cruel.
<cjwatson> Oh, fixing this will imply making branch/repository deletion hook into snaps, though.  Which was on my to-do, just a bit further down.
<cjwatson> I guess it's not hard.
<wgrant> Ah, fair.
<wgrant> It's not a blocker for landing the branch.
<wgrant> But I'd quite like to have some assertion that the behaviour is sane so when we rework something in three years we don't burn alphecca down.
<cjwatson> I think I'll just propose a separate branch first to fix deletion.  Easier that way round than some kind of weird assertion about what happens if a reference points into the void ...
<wgrant> Yep
#launchpad-dev 2015-08-05
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/snap-source-deletion/+merge/266866 updated as you requested, I hope
<cjwatson> I appear to be the first user of that Update Storm class in LP, which is a bit surprising
<blr> cjwatson: that's a little scary :)
<cjwatson> It seems to DTRT based on checking LP_DEBUG_SQL=1 output.
<cjwatson> (And tests of course.)
<wgrant> cjwatson: People mostly use ResultSet.set instead.
<wgrant> Which would work here, in fact.
<wgrant> IStore(Snap).find(Snap, git_repository=repository).set(git_repository=None, git_path=None) ish
<cjwatson> wgrant: Ah, nice, will transform to that tomorrow.
<cjwatson> wgrant: Does http://paste.ubuntu.com/12007951/ look like a half-way reasonable approach to this translations madness?
#launchpad-dev 2015-08-06
<wgrant> cjwatson: The comment is misleading (it skips whole sources, not single templates), but other than that it looks good.
<blr> wgrant: was just going to make an mp for the squid-forwardproxy settings, but noted your branch is still for precise, worth renaming to charms/trusty/squid-forwardproxy?
<wgrant> blr: There's no trusty charm branch today, and there's no need for the charm to differ between series, so we might as well keep it in precise for now.
<wgrant> AFAIK the Ecosystem people don't have good policies for sharing charms between series.
<blr> wgrant: okay fair enough, no issue deploying it to trusty at any rate, it was more cosmetic.
<cjwatson> wgrant: Working on tests for it and struggling to see how this makes sense ... http://paste.ubuntu.com/12012366/
<cjwatson> it just dropped that table, how can it already exist?
<wgrant> cjwatson: hee hee
<wgrant> cjwatson: Check the postgres log for the query that failed, but I think you'll find it's an amusing and confusing truncation bug.
<wgrant> Relation names are limited.
<wgrant> The query is creating an index called temp_translationtemplateitem_holding_distribution-100000_target_id, but the last three characters are too long, so it gets truncated to what happens to be the same name as the table.
<cjwatson> argh
<wgrant> >>> len('temp_translationtemplateitem_holding_distribution-100000_target')
<wgrant> 63
<cjwatson> hatred
<cjwatson> all right, makeDistribution(name='short') it is
<wgrant> Let's just hope we don't need ubuntu-rtm-seriously-this-is-the-last-fork-never-again-i-swear.
<cjwatson> oh and indeed there's a comment in another test about that
<wgrant> Heh, is there?
<wgrant> Ah.
<wgrant> That might be why I knew it immediately, I added that comment in August last year.
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/send-bug-notifications-oops/+merge/264814, the actual notification that generated the message being sent is ... well, it's available somewhere, but construct_email_notifications loses the mapping pretty thoroughly.  Is it worth the hassle to fix that, or should I just do something like dumping out the IDs of all notifications in the current batch?
<wgrant> cjwatson: Ah, right, the loop references *a* BugNotification, but it's not exactly the relevant one. Carry on, then.
<cjwatson> OK :)
<cjwatson> wgrant: Trying to think of where to put the +new-snap view.  It could go on /+snaps/+new-snap, but then I don't see how anyone would be able to find it.  It could go on <branch or ref>/+new-snap, which seems better in that it's discoverable and would save having to put a branch/ref picker on the +new-snap page.  But then I'd kind of want an IHasSnaps to attach the browser:page tag to, and you just asked me to remove that from an ...
<cjwatson> ... earlier branch :-)
<cjwatson> (And you just attached webhooks to repositories in a reasonably similar way ...)
<wgrant> cjwatson: A webhook is strictly subordinate to a git repository, and is managed by the people who manage the repository. It is basically an extension of the repository's configuration.
<wgrant> IHasSnaps doesn't have to die, but I think the API-accessible collection on the branch/repo should.
<wgrant> It's a relationship which crosses application domains and has no clear hierarchy.
<wgrant> I am loathe to clutter up branch and git repo pages with snap bits (think how useless the recipe is on most of them), but I'm not sure how better to do it.
<cjwatson> Right, I'm not suggesting bringing back the webservice collection.
<cjwatson> Snaps themselves are currently subordinate only to their owner.  I suppose they could be discoverable just by putting a link on Person:+index, but that seems little better in terms of clutter, and still leaves the picker issue.
<cjwatson> (And Person:+index is already massive)
<wgrant> IMO private interfaces may violate layering if it is significantly beneficial, but public APIs should not as they're very difficult to change later.
<wgrant> Indeed.
<wgrant> Branch:+index is also fairly large but it's also already mostly useless.
<cjwatson> So I think cross-linking is a bit different from subordinate relationships.
<wgrant> So adding another useless link is probably not a big problem.
<cjwatson> i.e. having Branch:+new-snap doesn't mean that the resulting object is subordinate to Branch
<wgrant> Oh sure, I think that might be the most reasonable path forward.
<cjwatson> The other possibility I can think of is a link on Branch:+index to /+snaps/+new-snap?branch=foo, but that seems like complication for little benefit.
<cjwatson> The snap source might as well be the context object there.
<wgrant> Right, adding a new view under Branch isn't a serious problem.
<wgrant> Cluttered view namespaces aren't an issue.
<wgrant> Cluttered API namespaces are permanent.
<cjwatson> Yep.
<wgrant> Cluttered UI is annoying.
<cjwatson> Perhaps we can minimise the clutter by renaming the "Related source package recipes" section somehow and putting the +new-snap link in there.
<wgrant> Indeed, I think that makes sense.
<cjwatson> They're not dissimilar.
<wgrant> They are very similar indeed.
<cjwatson> (Maybe I should have put Snap* under code ...)
<wgrant> OTOH a branch/repo picker really wouldn't be that bad.
<wgrant> Meh, lib/lp isn't too cluttered.
<wgrant> lib/lp/services is.
<wgrant> This is always going to be a pretty separate component.
<cjwatson> No, but I don't think it really makes sense.  You have a branch and you want to make a snap package out of it.
<wgrant> It is more closely tied to buildmaster than it is code.
<wgrant> That is true, but in most cases you have a branch and you *don't* want to make a snap package out of it.
<wgrant> Reworking the Branch:+index recipes section is consistent with what we have done previously, and it is probably what we should do here.
<cjwatson> Mm, still leaves the discoverability question though.
<wgrant> But "X (where X is 0.1% of the population) want to Y from Z, therefore Z must prominently show Y" is how the UI got into this state, so I like to think up ways to avoid it in future.
<cjwatson> (If it's not linked to from branch/ref, that is; and if it is linked to from there then there's no reason not to fill in the context.)
<wgrant> Right.
<cjwatson> Part of the problem is a paucity of alternatives for things that are otherwise quite isolated.
<wgrant> Exactly the issue.
<cjwatson> livefs we could get away with being largely undiscoverable because only a dozen people care.
<wgrant> There may also be the argument that building a snap in isolation on LP is a relatively rare case, and that the workflow would mostly be entered from external services.
<wgrant> I don't know if that's a reasonable assumption.
<cjwatson> Perhaps /+code-imports/+new is interesting precedent too.
<cjwatson> The only links to it are external (e.g. help.l.n) and the weird big green link button from the top of code.
<wgrant> It is also a rarely used page.
<wgrant> https://code.launchpad.net/launchpad/+new-import is more common.
<cjwatson> That sort of points towards <context>/+new-snap without a picker rather than /+snaps/+new with
<wgrant> All precedent undoubtedly points to that, indeed.
<cjwatson> Maybe I'm wrong in looking for precedent :-)
<cjwatson> I could also just do /+snaps/+new with a picker on the grounds that it's easy to add <context>/+new without a picker later
<cjwatson> Er <context>/+new-snap
 * cjwatson argues self into knots
<wgrant> cjwatson: Any revelations?
<cjwatson> I'm quite tempted to do both in the way we do for new code imports. :-P  The one with a context object would be behind a feature flag anyway for some time ...
<cjwatson> But I can see developer.ubuntu.com usefully linking to /+snaps/+new
<cjwatson> (e.g.)
<wgrant> Right.
<wgrant> I'm not sure it necessarily makes sense to have a separate $CONTEXT/+new-snap rather than the link just prefilling the field on /+snaps/+new, but we can try.
<cjwatson> Can certainly try that.
<blr> morning
#launchpad-dev 2015-08-07
<costello> Dear Sirs, I'm developer for a package that is coming to ubuntu via debian. When checking package details in https://launchpad.net/ubuntu/+source/... it seems it would be possible to set up upstream project link in there too -> but does it give any additional value to do so? if someone uses reportbug, it gets mailed to correct maintainer address already..?
<costello> And if I'd like to do so, I would need to "crete project" inside launchpad website, even tough the development actually happens elsewhere..?
<cjwatson> costello: It doesn't affect notifications, but it may be useful to give people obvious links.  And yes, many LP projects exist to provide code imports or links or what-have-you, they aren't all necessarily full hosted projects.
<cjwatson> But you don't have to.
<costello> ok, I'll look into creating one..
#launchpad-dev 2016-08-09
<cjwatson> wgrant: Have you had a chance to consider https://code.launchpad.net/~cjwatson/launchpad/git-update-related-bugs/+merge/298369 ?
<wgrant> cjwatson: Are you deliberately bypassing the logic in linkBug and unlinkBug (eg. bug activity logs)?
<wgrant> I'd also be tempted to have it not try to link 100,000 bugs.
<cjwatson> wgrant: I was trying to make it not be ridiculously inefficient, but I did indeed forget about the notifies.  Maybe it isn't worth trying to do bulk XRef creations when it's going to have to do notifies and all the stuff hung off them anyway?
<cjwatson> wgrant: 100,000> YM an arbitrary upper limit of some kind?
<cjwatson> Well, configurable I suppose, but still arbitrary.
<cjwatson> We don't have a limit for Bazaar.  Which I appreciate isn't a guarantee of making sense, but it does suggest that it maybe isn't a huge problem.
<cjwatson> wgrant: I've at least switched it over to linkBug/unlinkBug now.
#launchpad-dev 2016-08-12
<xnox> I guess no fix for bzrlib.plugins.git.errors.UnknownCommitExtra: Unknown extra fields in <Commit 34cc75327f52718ce19f64740264c2b7f4d46a35>: ['gpgsig'] still?
<cjwatson> xnox: unlikely to happen
<cjwatson> xnox: git-to-git imports are fairly close to the top of our to-do list, after some snappy stuff
<xnox> cool \o/
#launchpad-dev 2017-08-07
<ShillaSaebi> Hi Everyone! Wondering if anyone is around? I have an issue with my launchpad account after trying to file a bug in OpenStack neutron.
<cjwatson> ShillaSaebi: Can we have details?
<cjwatson> ShillaSaebi: If there's an OOPS, please tell us the OOPS ID.
<ShillaSaebi> sure, I tried to file a bug at this link: https://launchpad.net/neutron once i filed the bug, it was reported and here is the link: https://bugs.launchpad.net/neutron/+bug/1709102
<ShillaSaebi> immediately after i tried to login to launchpad and my account was suspended.
<cjwatson> Ah, must have been a false positive in the spam-fighting tools.  wgrant ^^
<ShillaSaebi> The same thing happened to my colleague
<cjwatson> ShillaSaebi: I've reinstated your account (please sign in to complete the process) and reopened the bug.
<ShillaSaebi> Thank you! Can you also reinstate my colleagues account? aju@iraw.net
<ShillaSaebi> He tried the same thing and his account was also suspended
<ShillaSaebi> thanks for reopening the bug. I think my account was suspended again
<cjwatson> Sigh.  Yes it was.  I can't fix that - needs wgrant who operates the spam-fighting tools
<cjwatson> So I'm afraid you'll need to wait until .au time when they're awake
<ShillaSaebi> alright np
<cjwatson> wgrant: https://bugs.launchpad.net/bugs/1709102 and https://bugs.launchpad.net/bugs/1708731 FWIW
<mup> Bug #1708731:  ovs-fw does not reinstate GRE conntrack entry . <in-stable-newton> <ovs-fw> <sg-fw> <neutron:New> <https://launchpad.net/bugs/1708731>
<cjwatson> wgrant: (also https://support.one.ubuntu.com/Ticket/Display.html?id=68552 and https://support.one.ubuntu.com/Ticket/Display.html?id=68553)
<mwhudson> i don't suppose anyone wants to talk about releasing snaps to multiple channels on bild
<mwhudson> *build
#launchpad-dev 2017-08-08
<wgrant> mwhudson: What's your use case?
<wgrant> Bah
<wgrant> Just missed the user
<wgrant> cjwatson: Fixed the normalisation bug, will email the user.
<wgrant> Thanks for the pointers.
<mwhudson> wgrant: wanting to upload go1.9rc2 to latest/candidate and 1.9/candidate
#launchpad-dev 2019-08-07
<SpecialK|Canon> cjwatson: can't figure out how to link to specific diff lines, but the copyright year updates in https://code.launchpad.net/~cjwatson/launchpad/team-members-ordering/+merge/371035 - do you have some funky automation for that or do you Just Remember?
<cjwatson> SpecialK|Canon: utilities/update-copyright
<SpecialK|Canon> cjwatson: nice!
<cjwatson> I've been known to run it from the LP tree when making changes to other Canonical projects on occasion since it's often close enough :)
<SpecialK|Canon> cjwatson: do you have some kind of funky automation to run _that_ or do you Just Remember? ;)
<SpecialK|Canon> (bzr/editor hooks?)
<cjwatson> I Just Remember
<cjwatson> (usually)
<SpecialK|Canon> hah ok neat :)
<cjwatson> My fingers type   utilities/update-copyright; utilities/format-new-and-modified-imports; lp-lint   more or less autonomously (with tab-completion)
<cjwatson> (lp-lint is my own wrapper that runs 'make lint' in a suitable container)
<SpecialK|Canon> nodnod, nice
#launchpad-dev 2019-08-09
<SpecialK|Canon> Before I go read through them all, to what extent is the list of Projects in https://launchpad.net/launchpad-project likely to be current/relevant/complete?
<cjwatson> SpecialK|Canon: There are certainly some stale ones, but it should be pretty complete
<SpecialK|Canon> cjwatson: fab, thanks
