#launchpad-dev 2009-07-13
<jtv> hey noodles775
<noodles775> Hi jtv :)
<noodles775> bigjools: have you heard or do you know why edge has not been updated since Friday?
<bigjools> yes
<bigjools> updates were stopped for a security patch
<bigjools> but didn't get re-enabled
<noodles775> bigjools: so there's no reason why it can't be re-enabled now, or even have edge updated right now?
<noodles775> If not I'll ask on IS...
<bigjools> noodles775: I'm on it
<wgrant> spm suggested a few hours ago that the XSS fixes weren't in stable yet.
<bigjools> they are in stable
#launchpad-dev 2009-07-14
<joey> I just started the stub for 4.0.  https://dev.launchpad.net/VersionFourDotO
<joey> for the community members on channel
<wgrant> Aha.
<wgrant> Is there a definite timeline for 3.0 yet?
<joey> kfogel's been handling that (I've been off on temporary assignment)
<joey> I saw something today in passing which seems to indicate it's gotten more solid but I haven't seen anything yet
<joey> i.e. official
<wgrant> OK.
<joey> The anticipation is building though so I am hopeful we'll see something really soon now. ;-)
<wgrant> I think the anticipation is more for the open-sourcing, though.
<joey> and we're running out of July :-)
<joey> yes! That's what I meant.
<joey> anticipation and excitement
<wgrant> July is no deadline any more...
<wgrant> Why aren't the team hierarchies on the dev wiki consistent? Registry's test plans live at 'Registry Team/Registry Test Plans', Soyuz at 'Soyuz Test Plan', Code at 'Code Team Test Plans', Translations at 'Translations/Test Plans', and a few other variants for the other teams.
<joey> wgrant: they were organic I think.  I certainly didn't prescribe what it should be, only that there should be test plans. :-)
<joey> wgrant: I'll pass it on to matsubara and ursinha. Maybe they might tackle the inconsistency.
<joey> ok, I'm going to bugger off for the day
<joey> have a good one
<wgrant> joey: Thanks.
<wgrant> You too.
<jtv> danilos: https://pastebin.canonical.com/19696/
<danilos> jtv: thanks
<danilos> jtv: I've emailed you the RT number if it doesn't get done by the end of the week
<jtv> danilos: as if I don't have enough trouble getting losa attention :)
<jtv> danilos: branch just landed on devel btw
<danilos> jtv: to be honest, who doesn't? note that Tom is on vacation this week, which means it's harder for everyone, especially with him being the most responsive of them all IME :)
<danilos> jtv: this is for bzr commit stuff, right?
<jtv> danilos: I wouldn't know, I rarely start work that early.  :)
<jtv> danilos: for that and other QA.
<danilos> well, Tom would usually show up in a few hours from now
<jtv> danilos: in my language you're saying "just before midnight."  :)
<danilos> jtv: so, have you started on the bzr commit QA? (you'll get other QA done I know, but this one probably requires a few hours of constant presence with a LOSA which is harder for you)
<jtv> danilos: just did a bit of preparation.  Not so sure it requires constant LOSA attention though.
<danilos> jtv: I didn't say constant LOSA attention, I said it requires you being constantly there when you've got LOSA at hand
<jtv> ah ok
<jtv> well, I'm working on it.  :)
<danilos> jtv: btw, has bug 383610 landed?
<ubot3> Malone bug 383610 in rosetta "Message-sharing migration script allows duplicate TranslationMessages" [High,In progress] https://launchpad.net/bugs/383610
<jtv> danilos: yes
<danilos> jtv: I don't see it on devel
<jtv> danilos: it's on db-devel, because it depends on a trigger change.
<danilos> jtv: is it only on db-devel?
<jtv> danilos: hang on, working with herb
<danilos> jtv: ah, ok, so that's not something we can CP
<danilos> jtv: sure, go on, that's fine, thanks
<henninge> danilos, jtv: Does this mean that staging's data is as of Friday, 16:25 (whatever TZ)?
<henninge> https://staging.launchpad.net/successful-updates.txt
<danilos> henninge: most likely, yes
<jtv> henninge: yes, it's Friday's data with later code.
<henninge> thanks
<danilos> henninge: there are enough import failures from before that date as well, so it shouldn't be hard to reproduce :)
<henninge> Actually, there aren't that many in the logs on devpad, just 5.
<danilos> henninge: right, it seems logrotate has messed it up, let me look up older ones and how they behave
<henninge> danilos: don't bother right now
<henninge> danilos: I will try the oldest I find there, first.
<sinzui> flacoste: I am confused about where I need to /also/ merge the DateMarshaller fix. I think you said lp:lazr.restful and to update the changelog, but this tree does not have a changelog.
<flacoste> sinzui: it should
<flacoste> sinzui: in src/lazr/restful/CHANGES.txt i think
<sinzui> flacoste: lp:lazr.restful AKA lp:~lazr-developers/lazr.restful/trunk does not. I can create one
<flacoste> sinzui: hang on, i'm sure there is on
<flacoste> sinzui: src/lazr/restful/NEWS.txt
<sinzui> I see that
<sinzui> thanks
#launchpad-dev 2009-07-15
<wgrant> sinzui: I don't think bug #397398 is a duplicate of the usual one - in that bug the user wanted it to be in Russia, but the timezone is wrong.
<ubot3> Malone bug 397398 in launchpad-registry "Wrong user timezone (Saratov) (dup-of: 387738)" [Undecided,New] https://launchpad.net/bugs/397398
<ubot3> Malone bug 387738 in launchpad-registry "Launchpad refuses to believe that I don't live in Russia, near the Kazakhstan border." [High,Fix committed] https://launchpad.net/bugs/387738
<sinzui> wgrant: I think it is because smara is were others were ending up
<wgrant> sinzui: Ah.
<sinzui> However we did just change the pytz lib, I should double check the date. There were two days of wierdness during the transition
<sinzui> I wonder if people will complain is I create the tag "ml-archive-sucks"
<lifeless> sinzui: mars says to talk to you about +series
<sinzui> yes
<lifeless> this is me, talking to you:)
<lifeless> I've filed a bug
<lifeless> I don't mean to be overly negative, but its nearly unusable at the moment.
<sinzui> Wow for me it went from a wasted click to at least a report that something is happening
<lifeless> wasted click?
<lifeless> we may have very different understandings of what I'm intending to ask about
<sinzui> Any time I was taken to a series page, I had to find a link to a page that would let me perform an action or learn about what was happening. The series page was not a good tool.
<sinzui> How can I make the series page a good tool?
<wgrant> sinzui: I would love a ml-archive-sucks tag, if it means the archives might eventually be less utterly sucky.
<lifeless> so there are two things, both similar. Perhaps I should file two bugs
<lifeless> firstly, can I ask you to have a look at
<lifeless> https://edge.launchpad.net/bzr/
<lifeless> there is a funny stick figure about half way down labelled 'bzr.dev'
<sinzui> oh yes the timeline that never shows more than one line of development
<sinzui> It gets bigger in the new page design
<wgrant> lifeless: Bug #398705?
<ubot3> Malone bug 398705 in launchpad-registry "Unobvious that timeline graph is scrollable" [Low,Triaged] https://launchpad.net/bugs/398705
<lifeless> I know that its scrollable. its just _painful_ to scroll on a thumbpad
<sinzui> I asked edwin to try putting borders on it that beuno would like
<lifeless> on a 125x125 dpi screen
<lifeless> click, drag about an inch
<lifeless> release go back, repeat
<lifeless> personally, I'd much rather not have to scroll at all
<sinzui> That sucks
<lifeless> that list of series seems truncated anyway, so why not truncate a little more and at the same time make the canvas taller
<lifeless> vertical space is cheap
<sinzui> Well it will be taller. I think we need to convey that the widget is a viewport
<lifeless> should I file a bug about this aspect? I haven't so far
<sinzui> definitely!
<sinzui> Edwin is work on a bug to improve the UI. As asked edwin to take wgrant's bug into consideration.
<lifeless> bug filed
<lifeless> secondly
<sinzui> thanks
<lifeless> have a look at https://edge.launchpad.net/bzr/+series
<lifeless> remember - trackpad user
<lifeless> I do have cheap vertical scrolling (of the whole page)
<sinzui> ha ha
<lifeless> my window is about 900 pixels wide
<sinzui> I will show that to Edwin and Martin
<sinzui> mine is 800
<wgrant> Does that one show inactive milestones too?
<wgrant> The one on +index doesn't.
<sinzui> This one show obsolete series at 50% opacity
<sinzui> I think I need to restart the discussion about sane zoom on load. I seeing this makes me think I was right
<lifeless> I'd like to note that *I don't care about the graph*
<lifeless> it can jump off a cliff and I'll never regret its passing
 * _thumper_ wonders if the william grant showing up on my Facebook suggestions is wgrant
<sinzui> lifeless: I can create a [X] Display timelines check box. That is how I got the maps off my pages
<wgrant> thumper: Could be. I have a few Launchpadders.
<lifeless> sinzui: I love timelines. That isn't a timeline :)
<lifeless> sinzui: or should I say, I love vertically presented regular text and markup timelines
<sinzui> understood
<thumper> wgrant: University of Melbourne '10
<thumper> ?
<wgrant> thumper: That would be me indeed.
<lifeless> ok, so two bugs filed. Let me know if you need more details on either.
<thumper> :)
<wgrant> The current one also isn't a timeline, since it doesn't show things relative to time.
<wgrant> Milestones, releases and series are equally spaced.
<wgrant> With no regard to time.
<wgrant> So bzr.dev seems to have all the latest milestones.
<sinzui> wgrant: and you can image why we did that given bzr's series
<wgrant> sinzui: If everything was positioned by time, it could be a very wide but fairly short graph that was able to be zoomed sanely.
<sinzui> oh, kfogel filed one of the the ml-archive bugs. I'm definitely using the new tag
<wgrant> I've been wondering for a while if you've been wanting to minimise adoption of LP mailing lists.
<wgrant> The archives are... suboptimal.
<wgrant> Non-members cannot subscribe.
<sinzui> wgrant: \o/
<wgrant> Message acceptance policies are unconfigurable.
<wgrant> Subjects are mangled.
<wgrant> (unconfigurably)
<sinzui> That is one of my points why it sucks, along with it does not integrate lp objects so I cannot link in and out of it
<wgrant> They could easily be very awesome, but they are not!
<sinzui> I cannot forward a message to my inbox so that I can reply (even if I am not a member of the list because I have good standing)
<wgrant> This standing thing also isn't documented anywhere except bugs, is it?
<sinzui> wgrant: We were going to create our own, or improve pipermail to rock. but we had two days, so we used MHonarc
<wgrant> sinzui: But MHonarc can do better than that...
<sinzui> kfogel: It can.
<sinzui> wgrant: kfogel looked into making some improvement. It can do better.
<wgrant> sinzui: This is good.
<wgrant> The lp-users archives are pretty terrible to use.
<wgrant> What is changing in the 3.0 UI?
<sinzui> We are reintroducing side portlets to emphasise activity. There are not many concrete examples of this
<sinzui> We want to show that things are happening on launchpad, commits, messages, subscribers, changes. And the side portlets can have actions that create these activities
<wgrant> But didn't you just try to kill all of the portlets?
<wgrant> Ah, good. That is one area in which Launchpad is really lacking.
<sinzui> We did, beuno likes them. He thinks they failed because they did not have strong rules of what they could contain or set users expectation about what they could do
<sinzui> links are still moving into the page
<sinzui> We are putting all the developers on updating all the pages to ensure we don't have any more 0, 1, and 2 ui pages left
<sinzui> wgrant: We want to improve the breadcrumbs and tabs, but we do not have any design and sope set for that yet. I think many pages will look a lot like the 2.0 pages.
<wgrant> Even completing the 2.0 UI would make things much better...
<wgrant> It's all pretty inconsistent now.
<sinzui> yep. I need to send an email asking for consistent heading/title rules.
<wgrant> Since you're moving launchpad-loggerhead into the launchpad tree, are you not removing codehosting at all?
<kfogel> wgrant: there's a bug and a branch for the ml improvements
<kfogel> when I get off the phone I'll find the bug number
<joey> hey matsubara-lunch, maybe you might get wgrant to review https://dev.launchpad.net/BugTriage/Draft and provide input?
<joey> Ursinha probably should have a run through too if she hasn't already
 * Ursinha looks
<matsubara> good idea joey
<joey> matsubara: only in that he's had some recent experience :-)
<joey> matsubara: you did good work on both the original and now this version so... might as well share!
#launchpad-dev 2009-07-16
<mwhudson> if you ever ever ever think that you should design a publishing system where AttributeError turns into some kind of not found error, please stop right there
<mwhudson> </vent>
<wgrant> mwhudson: Be thankful you have an object publishing system, and not some horrid URL-regex-matcher.
<mwhudson> wgrant: this is true
<mwhudson> but then at least programming errors would look like programming errors
<wgrant> True.
<rockstar> mwhudson, are you talking about TraversalError?
<mwhudson> rockstar: why, yes i am!
<rockstar> mwhudson, the benefit of obscure exceptions like that is that curse words get SO creative, it's almost worth having the obscurity.  :)
<mwhudson> yes, it's all an experiment to see if my rage can get hot enough to burn all the way through the earth's crust
<mwhudson> (not this time; must try harder)
<rockstar> Well, if you manage to put a whole through the earth, it would might make travelling to sprints a little easier.
<jml> Quick! Fetch me my unobtanium armor!
<lifeless> One word.
<lifeless> "Acquisition".
<thumper> AArrgg!
<thumper> not acquisition!
<mwhudson> lifeless: well yes, i would hope we'd have learnt the lesson a mere 10 years later
<rockstar> I just realized that I misspelled "hole"  I used to not have these issues until we published that writing style guide.  I think it broke me.
<mwhudson> (about, what, 15% we've been programming _at all_ as a race?)
<lifeless> style guides are overrated
<lifeless> :P
<flacoste> me
#launchpad-dev 2010-07-19
<nigelb> bryceh: just fyi, your blog has some trouble with openid ;)
<nigelb> anyway, thats a neat new feature :)
<bryceh> nigelb, it does?  howso?
<bryceh> I didn't even know it could do openid
<bryceh> nigelb, although I did notice nobody wrote comments :-/
<nigelb> bryceh: heh, I tried to login with LP
<nigelb> failed
<nigelb> you want me to grab the error?
<bryceh> yeah
<bryceh> I upgraded my web server to lucid, and drupal got upgraded, however it seems there is some bug that causes upgraded drupals to break in lucid
<bryceh> (which is why I haven't been blogging for a long time)
<bryceh> maybe that bug is why user registration doesn't work either
<nigelb> yeah
<nigelb> hold on, doing it again
<ajmitch> is the drupal issue that it's using deprecated functions?
<nigelb> it probbaly is
<nigelb> lucid only has 5.3
<ajmitch> </3 php
<nigelb> http://pastebin.com/r3fTKwmQ
<bryceh> ajmitch, no it looks like just a bug in the upgrade script, doesn't set fields to autoincrement
<ajmitch> lucid does have a drupal6 package, fwiw, but it's even OT for here :)
<nigelb> yeah, autoincrement thing :)
<bryceh> nigelb, ok yeah I've reproduced that error
<nigelb> that explains the comments :)
<nigelb> why don't you move to something like wordpress for ease of use?
<bryceh> nigelb, why?
<nigelb> drupal is nice, but just "complicated" ;)
<bryceh> nigelb, I develop on launchpad and x.org, I shouldn't be put off by "complicated" ;-)
<bryceh> nigelb, is wordpress open source?
<nigelb> bryceh: hahahaaha
<nigelb> and yes, its open source
<nigelb> also complicated for visitors is what I meant :)
<bryceh> well, if I wanted something simple for visitors I'd just post to facebook
<nigelb> hahah
<nigelb> but that still needs signup
<nigelb> btw, you're still in Prague?
<bryceh> nope
<bryceh> got home last night
<nigelb> :)
<nigelb> someday I gotta start hacking on LP.
<wgrant> nigelb: JOIN US!
<nigelb> wgrant: I'm just worried
<nigelb> I use this system for work
<bryceh> nigelb, it might be interesting for the LP folks to know what is holding you back, so they can work to eliminate that as an issue
<nigelb> so, can't have my apache dying on me
<wgrant> nigelb: LP doesn't usually eat systems alive, and if you're really worried you can use a VM.
<nigelb> the instructions say that the settings used for working on LP means usual web development cannot happen
<bryceh> nigelb, do you not have another system you could use for development on?
<wgrant> rocketfuel-setup will try to remove PHP (because it installs apache2-mpm-worker), but that can be worked around.
<wgrant> launchpad-database-setup obliterates all PostgreSQL databases, but that can also be worked around if required.
<nigelb> bryceh: not yet.  My own system is in the shop awaiting motherboard replacement
<wgrant> But apart from that, it coexists fine.
<wgrant> Just adds some vhosts to Apache.
 * nigelb developes on PHP :(
<nigelb> VM seems like a good idea.
<wgrant> nigelb: that just means you have to tweak rocketfuel-setup to not install apache2-mpm-worker.
<wgrant> It's not actually required.
<bryceh> nigelb, well, like wgrant says, running in a VM works too.  that's how brian does it
<wgrant> But yes, a VM works too.
<nigelb> I'll let a VM run rocketfuel overnight today :)
 * nigelb gets sucked in LP
<bryceh> wgrant, maybe the newbie intro docs should just be written from the start to nudge the user towards setting up launchpad in a VM?  it'd sidestep so many issues and worries
<wgrant> bryceh: Possibly. I'm not sure who manages those docs these days, though, since Karl disappeared.
<nigelb> disappeared?
<ajmitch> there was someone who'd stepped up to that role I think
<lifeless> there is a volunteer, yes.
<bryceh> who?
<lifeless> it was announced to the list
<bryceh> heh this is a bad sign if no one knows
<lifeless> public I think.
<lifeless> I know, it just not paged in.
<wgrant> lajjr
<lifeless> also, I present, another victem of the heat.
<bryceh> nigelb, anyway I think you may be right I should just use wordpress.  I had visions of using drupal for some more sophisticated website stuff, but these days I hardly have time to mess around with updating my website anyway
<nigelb> bryceh: heh :)
<nigelb> just be sure to update regularly
<bryceh> unfortunately that means losing all my old blog posts
<bryceh> nigelb, does something bad happen if you don't update regularly?
<lifeless> grrr jmkuhn - thanks mwhudson.
<nigelb> bryceh: vulnerabilities get reported often and get security updates
<nigelb> if you're lazy, you get owned :D
<bryceh> that sounds bad
<lifeless> bryceh: wordpress.com ++
<bryceh> nigelb, is updating as simple as apt-get upgrade wordpress or is more to it?
<nigelb> there would be a zip file i guess which you would use
<bryceh> ick
<nigelb> I've never done it.  I just use wordpress.com ;)
<lifeless> there is also a bzr branch you can just pull
<nigelb> yay, bzr++
<nigelb> lifeless: that should be a snap then :)
<mtaylor> lifeless: I just coped the planet.ubuntu.com idea of launchpad team + bzr branch of config for adding to blog syndication
<mtaylor> lifeless: total win - should be a launchpad feature...
<lifeless> cool
<lifeless> hmm
<lifeless> interesting folk are here
<lifeless> here is a crazy idea I had.
<lifeless> what if every object was [crudely] available as a 1-item-long rss feed (using accept-encoding negotiation)
<mtaylor> interesting
<wgrant> To what end?
<lifeless> pubsubhubbub
 * mtaylor hates it that lifeless answer word is real and that I know what it is
<lifeless> we would pingthesemanticweb on a change to any object [excluding private in the first instance]
<lifeless> that is monitored by at least one hub
<wgrant> Hmm.
<lifeless> and you can get callbacks when the object has changed.
<mtaylor> would make it easier to monitor things like lists-of-available-merge-requests
<lifeless> want to know when a build has finished on arm + i386? subscribe to the build
<mtaylor> any thing that gets me callbacks will make me happy as far as those things go
<nigelb> bryceh: I stand corrected.  There is a press-button upgrade feature.
<bryceh> nigelb, mm
<wgrant> mwhudson: Hmmm, so you're actually going to fix LP and modularise it (and the model classes)?
<mwhudson> wgrant: well
<mwhudson> wgrant: to some extent or other
<spm> rewrite from scratch. in php.
<mtaylor> ewwww
 * wgrant stabs spm repeatedly.
 * mtaylor pokes spm until he stops
<ajmitch> spm: that's the official word from what the LOSAs want then?
<spm> ajmitch: um....
<ajmitch> next he'll be saying that PHP 5 is too new, let's go for 4
<spm> tho I will concede my troll has been matched, and the trolling stacks raised higher than I'm prepared to go.
<spm> Oooo! didn't think if that! ta! yes! php4 ftw.
 * ajmitch shouldn't continue down this path
<mtaylor> fuck that... cobol forms are much more powerful
<mtaylor> spm: I just got a message that said that ~swift-bugs is subscribed to openstack - but I cannot for the life of me see _why_ that would be the case
<spm> mtaylor: hrm. looking
<mtaylor> spm: also, I think we're clear to delete all of those -private things
<spm> mtaylor: ahh. sweet. I'll do that first. it's possible one is causing the other.
<mtaylor> mmm. good point
<spm> openstack-private gone
<mtaylor> woot
<spm> swift-bugs-private swift-core-private gone
<spm> gah. swift-private is being recalcitrant, per that answers request you have. been looking at that this morning to no joy
<mtaylor> glad to know I haven't stopped causing you grief :)
<spm> actually.. thinking about that - I suspect it may have been self inflicted, by myself.one I need to chase.
<spm> mtaylor: I assume the ozone ones can go too?
<mtaylor> spm: yes. all ozone can die
<spm> kk
<wgrant> Is there some DB load issue at the moment?
<wgrant> For the past 24-36 hours, my script that heavily uses getBuildRecords has timed out every time.
<wgrant> It previously only timed out one every few hours.
<spm> that's not cool...
<spm> 1 min load is higher than usual, but 5 and 15 is normal. light even.
<spm> wgrant: you wouldn't happen to know what table that references?
<wgrant> spm: BinaryPackageBuild, PackageBuild and BuildFarmJob, mainly.
<spm> is that talking to edge? I'd suggest not but... ?
<wgrant> It may be... let's see.
<wgrant> It moves around depending on where the API is least broken.
<spm> heh
<wgrant> It's currently using edge.
<spm> I'm seeing a few timeouts for queries with BinaryPackageBuild in them, all via edge
<wgrant> These should be 12-15 minutes past the hour.
<spm> ta! that helps hugely.
<spm> hrm. these are all almost uniformly on the hour; so possibly not you.
<wgrant> Hmm.
<spm> wine, firefox, kdm,?
<spm> ubiquity is another
<wgrant> My script requests everything with a build failure
<bryceh> nigelb, http://www2.bryceharrington.org:8080/wordpress/?p=3
<spm> wgrant: are you getting an oops from these? the commonish query that somewhat matches is taking around 10secs based on a quick random grab. cop enough of those...
<wgrant> spm: I'm getting OOPSes, but this old launchpadlib doesn't display them.
<wgrant> Maybe I should upgrade that host to Lucid at some point.
<spm> bleh
<wgrant> Hm, wow, nearly a year since the open sourcing.
<wgrant> Argh.
<wgrant> https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177 is redispatching constantly.
<wgrant> Presumably because there is no longer a warty/i386 chroot.
<wgrant> Ah, and there are hoary and breezy builds too.
<nigelb> bryceh: w00t w00t!
<lifeless> wgrant: timeouts ?
<wgrant> lifeless: Yes. spm reminded me that you reduced the timeout last week.
<lifeless> there is a bug script hammering one bug on prod with trouble
<lifeless> the hourly edge graph is clean though
<lifeless> what issue are you having?
<lifeless> wgrant: ^
<wgrant> lifeless: distroseries.getBuildRecords is frequently timing out for me.
<lifeless> got an oops id ?
<wgrant> This old launchpadlib doesn't display them, so no :/
<lifeless> can you get one, please ?
<wgrant> OK. You can't easily find OOPSes for a particular method?
 * wgrant turns httplib2 debugging on.
<spm> i wonder... if oopses record the remote IP, lifeless, you should be able to fgrep -R on devpad and hunt thru them all? brute force, but...
<wgrant> It's hopefully reproducible.
<lifeless> wgrant: no, not yet.
<wgrant> It normally happens within the first five minutes of the script, and it's running with debug logging on now...
<lifeless> I'll file a bug on finding oops by method
<lifeless> spm: seeing ip addressess is a bug
<lifeless> we need to fix that - and seeing private urls : both should be limited to losa and gsa
<lifeless> https://bugs.edge.launchpad.net/oops-tools/+bug/607087
<_mup_> Bug #607087: enable 'search by method' <OOPS Tools:New> <https://launchpad.net/bugs/607087>
<lifeless> wgrant: any sign ?
<wgrant> lifeless: It seems to be an excellent Heisenbug.
<wgrant> arrrgh.
<wgrant> Just died now.
<lifeless> cool
<wgrant> But there's no X-Lazr-Oops header...
<lifeless> its not in the oops report either FWIW
<lifeless> wgrant: :(
<lifeless> what is there ?
<lifeless> pastebin ?
<wgrant> Hm. Maybe timeouts don't create an X-Lazr-Oops header :(
<lifeless> wgrant: this is what edge was last night -
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       63 /  618  DistroSeries:EntryResource
<lifeless>       12 /   71  Person:+participation
<lifeless>        9 /   13  ScopedCollection:CollectionResource
<lifeless>        8 /   29  BugTask:+index
<lifeless>        5 /   50  Sprint:+temp-meeting-export
<lifeless>        5 /    6  Archive:+copy-packages
<lifeless>        4 /    2  DistroSeries:+index
<lifeless>        3 /    3  BugTask:+editstatus-page
<lifeless>        2 /    3  Milestone:+index
<lifeless>        2 /    1  Person:+map
<lifeless> wgrant: we do get timeout oops on edge apis
<wgrant> It's probably the first one.
<lifeless> OOPS-1660EB2408
<wgrant> It would come under the first one, that is.
<wgrant> The request URL should have ws.op=getBuildRecords
<StevenK> !pastebin
<lifeless> QUERY_STRING: build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=200&ws.size=50
<lifeless> StevenK: !sanity
<wgrant> That's the right kind of request.
<wgrant> GET /1.0/ubuntu/maverick?build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=1750&ws.size=50
<wgrant> That's the one that failed just now.
<StevenK> lifeless: You should know better. :-P
<wgrant> I'd only expect 24 hard OOPSes a day from this, though.
<lifeless> StevenK: 12 lines in a totally dedicated channel is not a problem
<lifeless> StevenK: when there are no other discussions taking place.
<lifeless> StevenK: we are, after all, here to *communicate*
<wgrant> lifeless: I can has that OOPS?
<wgrant> lifeless: Ah, thanks for the review.
<lifeless> SQL time: 12978 ms
<lifeless> Non-sql time: 1499 ms
<wgrant> Aha.
<wgrant> What're the big queries?
<wgrant> Or are there lots?
<lifeless> http://pastebin.com/DsDmLqLk
<lifeless> one repeated query is the culprit I think
<wgrant> Forgive me -- I've no idea what those columns mean.
<wgrant> I'd really like to know where that query is coming from.
<wgrant> Hmm.
<wgrant> Maybe current_source_publication, I guess.
<wgrant> Although that query should be pretty quick.
<wgrant> lifeless: Do you have sufficient powers yet to land that branch?
<lifeless> wgrant: probably but EBUSY
<lifeless> wgrant: please grab someone else
<wgrant> lifeless: OK.
<lifeless> wgrant: oh, the 37 is repetitions
<lifeless> row reps time avg time-avg db-id query
<lifeless> there are also two very slow queries
<lifeless> 12 seconds total
<lifeless> http://pastebin.com/VsQpzqUq
<lifeless> wgrant: ^
<lifeless> wgrant: thats actually the issue, probably.
<wgrant> WTF is it doing a count for.
<lifeless> the rest is inefficient but broadly noise.
<wgrant> Thanks.
<wgrant> Those are indeed the problems.
<lifeless> wgrant: I'd like to lower the timeout again today; is that ok with you, given you're about to propose a fix ? :)
<wgrant> lifeless: My other scripts are reasonably happy, so go ahead.
<wgrant> I fully support any measures to make LP less slow.
<mrevell> Hey up
<wgrant> Can I get an EXPLAIN ANALYZE executed, or is that a stub thing?
<spm> wgrant: any of the losas can do it for you. fire away.
<wgrant> spm: Can you EXPLAIN ANALYZE the second query on http://pastebin.com/VsQpzqUq, please?
<spm> wgrant: http://paste.ubuntu.com/465800/
<wgrant> spm: Which DB was that? main master?
<spm> one of the slaves
<wgrant> Ah. Thanks.
<lifeless> spm: it was running on main
<lifeless> spm: you might want to check its the same?
<lifeless> actually nevermind
<lifeless> 7.7secs is 7.7secs
<wgrant> It's about a second slower, so it's pretty simialr.
<spm> heh, i'd be impressed if that made a difference, and no, looks near as identical.
<spm> 7.9 vs 7.7; within the bounds of load/server abuse
<wgrant> There are some recipe builds redispatching constantly, since they are targetted to series without chroots.
<wgrant> https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx
<bigjools> rockstar: can you fix that?
 * bigjools sighs at people making builds for breezy, hoary and warty
<bigjools> the UI should have blocked that
<wgrant> It does.
<wgrant> I bet the API doesn't, though.
<rockstar> wgrant, why are they targetted to series without chroots?
<wgrant> But we should fail builds if the chroot's missing, I think.
<bigjools> rockstar: see the series I mentioned
<wgrant> rockstar: Because Ted thought it was fun, I guess.
<rockstar> bigjools, the UI does block that, so I guess it was an API thing?
<bigjools> validation in browser code is largely useless now we have the API :/
<bigjools> yay for fat models
<bigjools> rockstar: yeah it must be
<wgrant> bigjools: Also, we have a whole lot of impossible builds being created in the primary archive when security updates with failed builds are copied.
<rockstar> bigjools, truthfully, that should probably take place in the model anyway.
<rockstar> wgrant, can you file a bug?
<wgrant> They get marked as failed immediately, but they clutter up build listings and probably shouldn't exist at all.
 * rockstar does not have a shortage of recipe bugs to fix
<bigjools> heh
<wgrant> rockstar: One for preventing creation, and another for dealing with broken ones?
<wgrant> (since they can become broken after creation)
<rockstar> wgrant, I think the broken ones is probably a buildmaster issue, right?
<wgrant> rockstar: No. You pick the chroot in lp.code somewhere, and that'd be what's crashing.
<wgrant> Also, what's going on with porting SPRBs and TTBJs to the new model?
<bigjools> it needs to fail the build
<bigjools> in progress
<wgrant> Excellent.
<bigjools> the latter is pronounced as "Titty Bee Jay" BTW
<wgrant> Heh.
<bigjools> rockstar: so those builds that won't ever complete, you should poke some SQL at the LOSAs to remove them
<bigjools> and add a "cancel" button to the UI
<poolie> raising the tone
<wgrant> There's a cancel branch in progress, right?
<rockstar> bigjools, there is a cancel button already for admins.
<wgrant> I thought I saw it floating around last week.
<bigjools> ummm
<rockstar> losa ping
<bigjools> rockstar: why only admins?
<mthaddon> rockstar: hi
<bigjools> should be the user, and buildd-admins
<rockstar> bigjools, because that's what I wrote the code to do.
<bigjools> :)
<wgrant> What does it do when the build is in progress or finished?
<rockstar> wgrant, where are these builds?
<poolie> wgrant, i thought <https://code.edge.launchpad.net/~leonardr/launchpad/true-anonymous-access/+merge/30088> might please you
<wgrant> rockstar: the three pending builds on https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx
<wgrant> poolie: Indeed, I was pleased to see that.
<rockstar> mthaddon, I have some builds that probably need to be cancelled.
<bigjools> rockstar: what does the cancel button do BTW?  mark them with a new status or delete them?
<rockstar> mthaddon, https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/175 https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/176 and https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177
 * rockstar makes note to head punch ted
<wgrant> I'm somewhat concerned that Code is doing things that are impossible.
<rockstar> bigjools, I originally was deleting the build, but abentley thumped me, so I'm just changing the status now and deleting the buildqueue record if there is one.
<wgrant> Like deleting branches related to recipes related to builds.
<mthaddon> rockstar: what is "wrong" with what ted has done here?
<wgrant> And cancelling builds that are in progress.
<rockstar> mthaddon, requesting builds against unsupported distros.
<bigjools> rockstar: great, sounds fine, with the caveat that wgrant is raising
<mthaddon> rockstar: should he not be allowed to do that? if so is that a bug that we let him do that?
<wgrant> mthaddon: Yes, bug #607125
<_mup_> Bug #607125: Forbid creation of recipe builds for obsolete series through the API <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/607125>
<rockstar> I see your caveat, and raise a "well, it's either that, or assume the buildmaster will fix it, which has been a bad assumption so far"
<mthaddon> thx wgrant
 * mthaddon adds to losa watched bugs
<wgrant> rockstar: Well, cancelling an in progress build will at the moment either not work, or not work and knock the builder out until somebody notices that it has been idle for weeks.
<rockstar> wgrant, there should be a cancel button on there now though.
<rockstar> wgrant, this is why we left it up to the admins.  The reason it's here is because builds keep getting re-tried even though they are always failing.
<bigjools> we have some work to do to cancel in-progress builds
<mthaddon> rockstar: those have been cancelled
<bigjools> rockstar: can you change that permission to buildd-admins please
<rockstar> mthaddon, great, thanks.
<rockstar> bigjools, you can...  :)
<bigjools> rockstar: I would but we have all these buildd-manager bugs you keep exposing ;)
 * rockstar refrains from "it's open source" comment.
 * bigjools might do the same with the next buildd-manager bug
<rockstar> bigjools, this is a hell of a pissing match...  "My code is more broken than yours!"  "Nuh uh!"
<bigjools> lol
<wgrant> Ah, it already raises a CannotBuild when the chroot is missing.
<wgrant> But that just causes the dispatch to do nothing.
<wgrant> So it's going to be retried on the next run.
<rockstar> wgrant, ...unless we've cancelled the build.
<wgrant> rockstar: Right.
<rockstar> wgrant, "Cancel build" is there so that we don't have to make the losas run SQL.
<wgrant> I speak of the general case.
<rockstar> It shouldn't have to exist at all.
<bigjools> one of my b-m bugs is to detect failures and work out whether to fail the job or the builder
<rockstar> Luckily, spr is only available on edge.
 * bigjools cringes at the 2nd meaning of SPR now :(
<rockstar> bigjools, maybe just see if the builder failed the last 5 builds before the builder commits seppuku.
<bigjools> rockstar: that's the exact plan
<rockstar> bigjools, my use of SPR is more important now.
<bigjools> it can see if it's the same builder or the same job
 * rockstar only works on important things
<wgrant> bigjools: We also need to fire the reset trigger if a builder is being stupid.
<bigjools> haha
<wgrant> bigjools: eg. during an abort.
<wgrant> Or when we get a <Fault 8002: error>
<bigjools> I have a branch that does that, but I backed it out
<wgrant> :(
<rockstar> wgrant, also, recipe builds are about to be able to be re-scored as well.
<bigjools> wgrant: it disabled builders that were not resetting quick enough :/
<wgrant> bigjools: Oh, right.
<wgrant> bigjools: Once we no longer need to preserve ddebs on the buildd filesystem, is there any need for the virt/non-virt distinction? I think the architecture restriction magic can handle it all.
<bigjools> well yes
<wgrant> It would also mean we wouldn't have to lie about ivy and kaylaberry.
<bigjools> we still need a split between non-virt and virtual i386/and64
<bigjools> amd*
<wgrant> Why?
<wgrant> Once our queue discipline doesn't suck, there's no reason to.
<rockstar> bigjools, it'd be nice if we could return the spr builds to be "anything but arm" instead of i386 only.
<bigjools> archive restriction won't work there
<wgrant> bigjools: Why not?
<bigjools> rockstar: agreed
<bigjools> wgrant: because we don't want PPA builds ending up on distro builders
<wgrant> bigjools: But the distro builders don't need to be special.
<bigjools> yes, they do
<wgrant> Whyso?
<bigjools> they're not PPA builders
<bigjools> distro has its own hardware
<wgrant> Ah, so it is political. Yay.
<bigjools> besides, not resetting the builder shaves 30 seconds off the build time
<wgrant> Anyway, regarding the "anything but arm" thing: we could just leave them on i386 and let amd64 builders build i386 and lpia as well, as has been planned for years.
<bigjools> yeah there's a bug about that
<bigjools> it would need a bit of work server-side :)
<wgrant> Not really.
<wgrant> The slave side is done, but not merged.
<wgrant> And the master is simple enough.
<wgrant> Although it might make /builders a little odd.
<lifeless> mthaddon: hi
<mthaddon> lifeless: hi
<lifeless> I has a new config CP on the wiki page
<mthaddon> lifeless: you've added it to ones already done rather than requested ones
<lifeless> oh darn
<lifeless> fixing
<mthaddon> lifeless: and since this is an edge one, it's not really a cherry pick - just a request for an edge rollout
<lifeless> mthaddon: ok
<lifeless> mthaddon: can you please rollout edge then? Sorry about not getting the process right :(
<bigjools> wgrant: the hard part is the build dispatch ETA
<wgrant> bigjools: Yeah. But arch-agnostic builds completely break that too. I think we need a new way to do things.
<mthaddon> lifeless: ugh, we don't have a new commit to "stable" since the last edge rollout, so it'd make a rollout currently a very manual process
<bigjools> wgrant: they don't break it if they end up in the same arch-indep builders each time
<bigjools> it's when the arch is indeterminate, mixed with determinate builds that you have an issue
<bigjools> queueing theory is a bitch
<wgrant> Yes :(
<lifeless> mthaddon: here's what I'd like to do: I want to drop the edge timeout to 12 seconds. The ppr suggests 10 might be safe, but I don't totallly trust it.
<lifeless> mthaddon: I'd like to find out during our work day if that change is an issue so we can rollback in a controlled fashion.
<mthaddon> lifeless: the next auto rollout would be tomorrow morning (assuming there's a new landing to stable between now and then), so how about that?
<lifeless> mthaddon: it would make me sad. But if thats the best we can do, its the best we can do.
<lifeless> mthaddon: also, its a bit risky.
<lifeless> mthaddon: because if the auto rollout happens, and its bad, we're in the poo, and I explicitly want to avoid that.
<mthaddon> lifeless: there are LOSAs around during the auto-rollout, or if someone lands a change to stable between now and then, kicking off a rollout is fairly trivial
<lifeless> mthaddon: could you just do an empty commit then, from praseodymium ?
<lifeless> checkout lp:launchpad/stable; bzr commit --allow-unchanged ?
<mthaddon> lifeless: I *could* but this is more "working around our process rather than fixing our process" which makes me sad
<lifeless> mthaddon: it makes me sad too
<lifeless> mthaddon: I don't understand why our process has this limitation.
<lifeless> Can you tell me about the limitation ?
<mthaddon> lifeless: the limitation is that we want to be able to push and build code without taking down the current app server - we do that by pushing to a new directory
<mthaddon> lifeless: the name of the new directory is determined by the revno of the top level bzr dir in the code tree
<mthaddon> lifeless: so unless we have a commit to that, we'd be pushing over the top of the existing directory
<mthaddon> lifeless: which our deployment scripts would stop from allowing
<lifeless> right
<mthaddon> s/would stop from allowing/wouldn't allow/
<lifeless> that would be bad.
<lifeless> so, a few things occur
<lifeless> we could make the top level number be a composite - lprevno,configrevno
<lifeless> we could use a datestamp - lprevno,$time
<lifeless> or even $time
<lifeless> do you have any other ideas?
<mthaddon> lifeless: that would be problematic in terms of how the deployment scripts handle other projects (we don't just use it for LP), but the lprevno,$time would work
<lifeless> ok, where is the script, so that I can put a patch together
<mthaddon> I mean the lprevno,configrevno wouldn't work, but lprevno,$time would work
<mthaddon> lifeless: lp:~canonical-losas/deployment-manager/trunk
<lifeless> mthaddon: RepositoryFormatKnitPack1 - you could benefit by upgrading that :)
<wgrant> bigjools: Is there a reason that external_dependencies are between the archive itself and the other archive dependencies in sources.list? Would the world explode if I moved it to the end (it would make some refactoring simpler)?
<bigjools> I think so, let me remember
<wgrant> It's possible that people are relying on it. But that's a bit crazy :/
 * mthaddon does it through the UI
<bigjools> I think they are - it was for OEM's migration
<wgrant> I guess I could just insert them after the first line.
<bigjools> mthaddon: is that a bumper sticker? :)
<bigjools> wgrant: what are you changing?
<mthaddon> bigjools: no, that's "LOSAs do it through the UI" :)
<wgrant> bigjools: https://code.edge.launchpad.net/~wgrant/launchpad/bug-598345-restrict-dep-contexts/+merge/30203
<bigjools> mthaddon: :D
<bigjools> wgrant: yay!
<bigjools> btw I have a new buildd-manager that I finished (sorta)} last week
<lifeless> wgrant: so, please tell me you're going to fix the soyuz api :)
<wgrant> Yeah, I saw that. Looks good.
<wgrant> lifeless: Hm? The timeouts?
<bigjools> wgrant: are you stalking me?  I didn't even make an MP yet!
<lifeless> yes, in particular
<wgrant> bigjools: I guess I'll just change it to insert the external deps after the first real dep entry.
<wgrant> bigjools: I stalked lots of LP branches over the weekend :P
<bigjools> lifeless: we need a way of jobbing copy-packages before we get rid of the major source of timeouts ;)
<lifeless> bigjools: you could use the jobs system
<lifeless> no ?
<bigjools> yes - but it would be a lot of work
<lifeless> how is it done now ?
<bigjools> all synchronously in the appserver
<lifeless> aiieeee
<bigjools> it issues a shitload of queries
<bigjools> the copy checking is very complicated
<wgrant> lifeless: This is Launchpad. Launchpad's motto is "Let's do everything in the appserver until it times out, then ignore it for a while after it starts to."
<bigjools> but it is well-factored so it's just a case of UI changes
<wgrant> bigjools: But isn't a lot of the time taken in verifying that the copy can actually happen successfully?
<bigjools> wgrant: yes, that's exactly what I am talking about
<lifeless> wgrant: that was last month.
<bigjools> the copy itself is trivial
<bigjools> the checking - not so much
<lifeless> rabbit is now available.
<wgrant> Hmm. Doing that asynchronously sounds a bit awkward.
<lifeless> the jobs system is pretty good even without rabbit.
<bigjools> lifeless: I hope you saw my comment on that bug about more daemons
<wgrant> lifeless: There isn't really a jobs "system"...
<lifeless> bigjools: I did, but I don't have an answer other than 'yes, we should fix it so we don't monkey with apache, pgsql, etc/hosts, etc etc etc'
<bigjools> lifeless: the problem is endemic
<lifeless> agreed
<lifeless> this is why I develop lp in a VM
<bigjools> that's not a cure, it's pain relief
<lifeless> If you have suggestions for how to develop software using rabbit without having rabbit present, I'm all ears.
<bigjools> rabbit ears?
<StevenK> Bwahaha
<bigjools> we just need a way of selectively starting this stuff when needed
<bigjools> during make run, or during the tests
<lifeless> in this case, perhaps file a bug on rabbitmq in debian
<wgrant> bigjools: Is buildd-manager really so slow at the moment because of process-upload.py, or is the candidate selection query also taking a while?
<lifeless> the package being how it is isn't really my fault :)
<bigjools> I am really fed up of more daemons running in the background :(
<lifeless> bigjools: we shouldn't/can't use the system rabbit for tests anyhow.
<bigjools> lifeless: it's not unreasonable that it starts when you install it
<lifeless> bigjools: please don't think I'm advocating having it running all the time: I'm not.
<bigjools> but the lp packages should turn it off
<bigjools> wgrant: most of it is upload
<lifeless> bigjools: that would worry me; too much chance of turning it off on prod.
<bigjools> the next slowest part is dispatching (it takes 10 seconds to send all the files to the builders)
<bigjools> candidate selection is very fast
<wgrant> bigjools: Hmm. But there are only 59 builders... and it gets out to 15 minutes sometimes.
<bigjools> lifeless: make start should start it
<lifeless> no
<bigjools> .......
<lifeless> we should not use the system instance at all
<lifeless> make start should run a private instance using the system binary
<bigjools> wgrant: yeah that's upload time
<wgrant> Hm, I guess that is Hm, I guess that only means 15s of upload time, yeah.
<lifeless> which may be what you mean
<wgrant> Crazy.
<bigjools> lifeless: yep
<lifeless> althought, again, make start on prod - we have to be very careful not to mess up there.
<wgrant> lifeless: But we do the same sort of thing with the librarian, so there's precedent.
<lifeless> wgrant: the way we use the librarian causes lots of headaches, testing included.
<wgrant> lifeless: True.
<lifeless> mthaddon: I am admiring of your unit tests :P
<mthaddon> lifeless: er.... yeah
<bigjools> I'd really like "make start" to start everything it should do, given the LPCONFIG
<jml> I'd like to not start Launchpad using make targets.
<lifeless> jml: what would you like to start using make targets?
<jml> lifeless, are you trolling?
<lifeless> only a little
<jml> lifeless, well, I don't understand the question. But to elaborate on what I meant, I'd like to not have make targets for running Launchpad or its bits, and instead use actual scripts.
<jml> maybe even the same scripts mthaddon uses!
<lifeless> we should talk about this later
<jml> lifeless, yes.
<jml> lifeless, after you review my branch, for example :P
<bigjools> my point was that I don't want to think about what to start
<lifeless> I was thinking after LP is fast
<jml> lifeless, please review my branch before then.
<lifeless> haha
<lifeless> seriously though, if someone else reviews it its all good too.
<lifeless> uhm, those logouts shouldn't be needed now though, should they ?
<bigjools> lifeless: are you lowering hard or soft timeouts on edge?  or both?
<bigjools> lifeless: also have you notified launchpad-users?
<bryceh> lifeless, LP is fast???
<james_w> what they did for couchdb was make a couchdb-bin package with all the binaries, and a couchdb package with the init script.
<james_w> doing a similar thing would be l-dev-deps could depend on rabbitmq-bin
<bigjools> +1
<bigjools> james_w: while you're there!  can you check out the LEP for derived distros please?
<bigjools> in particular please make sure we've captured all your requirements
<wgrant> Can someone please land lp:~wgrant/launchpad/remove-obsolete-buildd-junk?
<jml> wgrant, sure, if it's approved.
<wgrant> jml: It is indeed approved.
<lifeless> bigjools: prod can't go lower until we deploy edge to prod
<lifeless> bigjools: because its got many more timeouts as a baseline
<jml> wgrant, running ec2 land now.
<wgrant> jml: Thanks.
<jml> np
<bigjools> lifeless: that's not what I asked! :)  you know soft vs hard?
<lifeless> yes
<lifeless> sorry, got ELOCAL for a sec
<lifeless> anyhow, edge - dropping both
<lifeless> bryceh: that is the goal.
<lifeless> deryck: hai
<deryck> lifeless, hi
<lifeless> deryck: If you guys have any slack, or the ability to deal with oops and near-oops, I have a few things for you :)
<deryck> lifeless, yes, we can deal with oops and near-oops.
<lifeless> deryck: so, I sent you a mail listing 4 pages that are near to timing out on prod
<lifeless> but also
<wgrant> Is stub around this week?
<lifeless> on friday night there was a big boom
<lifeless> wgrant: once he gets home, yes.
<wgrant> lifeless: Thanks.
<lifeless> deryck: on friday night, one API call went 8boom8
<deryck> lifeless, re: mail.  I marked it to look at today, and will respond with comments.  But generally, making those a priority is welcome and not a problem for me.
<lifeless> 4000 times
<deryck> lifeless, I saw that.  The message stuff, right?  Same API stuff bryceh was asking about at the epic, right?
<lifeless> poolie and jtv have done some analysis
<lifeless> I guess, yes.
<deryck> lifeless, yeah.  gmb is testing a fix as we type. :-)
<lifeless> awesome
<bryceh> lifeless, haha
<lifeless> bryceh: everyone wants it to be fast.
<lifeless> so we shall be fast.
<lifeless> deryck: https://lp-oops.canonical.com/oops.py/?oopsid=1659B1002 for instance
<lifeless> deryck: poolie reckons its just the calculation of the canonical url per item
<deryck> hey, there's bryceh :-)  go to bed. ;)
 * deryck is reading the OOPS
<bryceh> deryck, yeah, 4am == well past bedtime
<deryck> lifeless, so having processed all the info now, I think gmb and I can chase up a fix today for this.  And I'll respond in depth to the timeout email before EOD for me.
<lifeless> awesome
<lifeless> we're getting up some momentum on performance
<lifeless> if we can just get a bit of a cadence going we'll be at 5 seconds in no time
<gmb> deryck, lifeless: Have fix will travel: http://pastebin.ubuntu.com/465875/
<wgrant> lifeless: Crank the timeout down to 1s.
<wgrant> Then see how quickly things get fixed :P
<lifeless> wgrant: that would just fuck everyone over
<wgrant> :(
<gmb> There's already tonnes of test coverage.
<lifeless> gmb: isn't messages.first() or something even cheaper?
<gmb> lifeless, It might be; I can never remember the Storm runes. Let me try it.
<lifeless> its not first, I don't remember them either
<lifeless> food time
<wgrant> bigjools: Am I OK to go through with https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667? It seems to be a good cleanup, and is somewhat necessary for ddeb stuff.
<bigjools> my eyes
<bigjools> what have you changed?
<wgrant> Oh, right, I haven't added a description yet. Oops.
<wgrant> You know the atomic arch-indep domination stuff?
<bigjools> vaguely
<bigjools> I've never done anything in that code
<gmb> lifeless, FTR, first() appears to shave ~2s off test run time, though that could be the weather or the proximity of chickens to the developer for all I know.
<bigjools> it's headfuck territory
<wgrant> bigjools: Yes. Exactly. I'm trying to unheadfuck it.
<wgrant> At the moment, publications have a supersede() method.
<wgrant> It basically just sets datesuperseded and the status.
<bigjools> wgrant: in tandem with my b-m changes, we might start unheadfucking soyuz
<wgrant> The Dominator handles setting supersededby, and it also handles atomic domination rules.
<wgrant> (atomic domination means that if an arch-indep publication is superseded, other publications of the same binary immediately get superseded too)
<wgrant> This makes for exceedingly ugly code, and even uglier tests.
<wgrant> So I've moved all of that into the model.
<deryck> gmb, so like lifeless I think use .first, but otherwise this looks good to me.
<wgrant> So Dominator just calls publication.supersede(dominant), and the model works out what else needs to die, sets supersededby, etc. Dominator just finds the candidates and tells them to do their stuff.
<gmb> deryck, Cool, preparing a branch now.
<wgrant> A subsequent branch replaces the tests with some that suck less.
<gmb> s/branch/mp
<wgrant> *Then* i can extend it to dominate ddebs properly, which is the whole aim of this.
<deryck> gmb, I wonder if we should look at bug 606911 while we're here, and then cowboy to staging and hit with an api script to see if we're better?
<_mup_> Bug #606911: Bug.indexed_messages unused? <Launchpad Bugs:New> <https://launchpad.net/bugs/606911>
<deryck> gmb, especially if that is just dropping unused code.
<bigjools> wgrant: ok seems like a rational thing to do
<bigjools> wgrant: my concerns are: performance and performance
<wgrant> bigjools: No performance changes here.
<wgrant> It just shuffles methods into the model and wraps them up.
<bigjools> ok - I'd still like to do testing pre and post-branch on DF to make sure
<gmb> deryck, Yeah, I'm thinking that too.
 * bigjools doesn't apologise for being paranoid
<wgrant> Heh, OK.
<bigjools> I hate finding my own code when searching on google for stuff
<wgrant> Haha.
<gmb> deryck, Ahaa. Except that indexed_messages is exported as 'messages'.
<gmb> So that would be a big bag of fail.
<wgrant> I hate finding all this Soyuz code that hasn't been touched by anyone who's still around, and is revolting.
<gmb> wgrant, That's *why* they're not still around.
<wgrant> gmb: Heh.
<gmb> deryck, I suggest we just try cowboying the initial_message fix for now.
<deryck> gmb, that's fine with me.
<deryck> gmb, I can investigate the other bug today after I settle in.  Then, you can go back to subscriptions hacking after this fix.
<gmb> deryck, Okay. I'll work up an mp for the performance fix anyway.
<deryck> gmb, yes, do.  I wasn't suggesting dropping this fix.  Just that you don't have to worry about the other bug for now.
<gmb> deryck, Right. I managed to put the word "anyway" in there without meaning to. What I meant was "now"
<deryck> heh, fair enough
<lifeless> gmb: is it first () ?
<gmb> lifeless, Yes
<lifeless> hah, who knew :)
<gmb> lifeless, Well intuited :)
<lifeless> gmb: you win some, you win some more
<jml> wgrant, oops. forgot to update download-cache. trying again.
<jml> wait. no.
<jml> ec2 land has been broken by recent changes to Launchpad.
<lifeless> gmb: https://code.edge.launchpad.net/~gmb/launchpad/bug-606914/+merge/30259
<wgrant> jml: Awesomeness.
<gmb> lifeless, Thanks. Will remove the whitespace; bad habit.
<jml> wgrant, I'm going to fix those before landing your branch.
<lifeless> gmb: my overly general rule of thumb is 'if it needs vws in a function, it probably needs two functions'
<lifeless> jml: what changes broke ec2land?
<gmb> lifeless, Good rule :)
<jml> lifeless, a change to the webservice.
<lifeless> jml: .next()
<jml> lifeless, AttributeError: 'Entry' object has no attribute 'queue_status'
<lifeless> grah
<lifeless> thanks
<jml> lifeless, I think jkakar has a patch for better testing support with launchpadlib
<lifeless> jml: a bit, yes.
<lifeless> he's trying to get it landed
<lifeless> but launchpadlib is a bit single-writer-gated atm
<jml> hmm.
<poolie> jam OOPS-1661ED1057
<poolie> hi there deryck
<poolie> thanks for looking at the attachment bug
<jml> queue_status is still being exported :\
<jml> benji, good morning.
<benji> good morning, jml; how goes it?
<lifeless> gmb: when you finish with that, I have a question on your reset watches stuff
<jml> benji, alright.
<jml> fixing bugs. not taking prisoners.
<deryck> hi poolie.  np.  glad the fix was easy.
<deryck> gmb did the heavy lifting :-)
<jml> benji, there are a few pending patches from Launchpadders to zope.testing. would you be interested in reviewing & landing them?
<jml> (if not, I'll try to find sidnei)
<gmb> lifeless, Sure, ask.
<benji> glad to take a look
<jml> benji, https://code.edge.launchpad.net/zope.testing/+activereviews
<lifeless> who will be looking for bad attempts to reset bug watches
<jml> lifeless, wgrant: false alarm. it's my bug.
<gmb> lifeless, I'm not sure what you mean. Do you mean "who will be looking to see which bug trackers need their watches resetting?" or "who will be looking to make sure no-one maliciously resets watches?"
<gmb> Or something else?
<lifeless> the second
<poolie> deryck, i'd be interested to review the result when it's up for review, for my own education
<gmb> lifeless, Any UserCannotResetWatchesErrors should appear in the OOPS report, shouldn't they? So we should spot them fairly quickly.
<deryck> poolie, ok, it's up.  let me find the link.
<gmb> lifeless, Only LP Devs and Admins are able to do the resetting or for that matter see the reset button at all.
<deryck> poolie, https://code.edge.launchpad.net/~gmb/launchpad/bug-606914/+merge/30259
<poolie> got it thanks
<lifeless> gmb: who is going to be *looking* for that though
<lifeless> its going to show up in 'exceptions'
<lifeless> gmb: and how is it different to any attempt to circumvent permissions - e.g. creating a new private branch, viewing a private bug etc.
<gmb> lifeless, I'm not sure that it is different... Ah, but, instead of the custom Exception it would make more sense to raise the same kind of exception as the other attempts to circumvent permissions do.
<gmb> I don't know if anyone actively looks for those errors, but I think it's a safe bet that someone does because they get reported fairly quickly.
<gmb> Let me do some digging.
<poolie> gmb then i suspect bug 424671 won't be fixed by this
<_mup_> Bug #424671: attachments: accessing attachment's message is very slow <api> <Launchpad Foundations:Triaged by leonardr> <https://launchpad.net/bugs/424671>
<poolie> but it's similar
<poolie> as you probably already know
<lifeless> gmb: I think the normal permission denied behaviour is really all that is needed
<lifeless> gmb: unless you have some particularly special horror story we're expecting
<gmb> lifeless, No, I just wrote that stuff about half an hour before getting on a plane and didn't think about it very hard :)
<lifeless> :)
<gmb> I'll change it appropriately.
<gmb> poolie, I agree that they're similar but I think they should be fixed separately (if only because I knew exactly how to fix the first bug but wasn't sure I grokked a solution for the second).
<lifeless> if it fixes the timeout, I'm happy
<lifeless> there are 4 other near-timeouts that can queue between 'better' and 'perfect' for this API, IMHO :)
<jml> vocabularies.txt is 1428 lines long.
<wgrant> jml: Don't look at archive.txt...
<jml> I probably will eventually.
<jml> I'm on a quest to remove all the imports in canonical.launchpad.interfaces.__init__
<wgrant> ahahah.
<jml> my current plan is to do it one import at a time
<jml> but to fix all of the imports in every file I touch
<jml> so that it gets easier as I go along
<wgrant> Right.
<lifeless> wgrant: your VWS in supersede doesn't really gel for me - please remove or comment
 * gmb -> lunch
<wgrant> lifeless: What's the issue with it?
<lifeless> its not clear why these lines need special attention drawn to them
<wgrant> Should I remove all blank lines from both methods?
<lifeless> I generally follow pep8 here
<lifeless> by which I mean 'yes, unless you really want special attention'
<wgrant> Hm, OK.
<lifeless> http://www.python.org/dev/peps/pep-0008/
<lifeless> "  Use blank lines in functions, sparingly, to indicate logical sections.
<lifeless> "
<wgrant> They are, in my opinion, indicating logical sections.
<wgrant> It's unfortunate that one of those sections is a super() call, and that it has another section on top of it.
<lifeless> thus my request to add a comment :)
<wgrant> Like what? "# Set date and status."?
<lifeless> wgrant: if its self evident, I don't understand why its really a different section; if its not self evident, surely it should have a comment?
<lifeless> wgrant: however, its up to you, this isn't a blocking issue.
<wgrant> lifeless: I don't think it deserves its own section. But the segments of code above and below it do.
<wgrant> lifeless: Thanks for the reviews.
<lifeless> de nada
<wgrant> lifeless: Are you landing those, or must I coerce others?
<lifeless> coerce
<wgrant> k
<wgrant> Can someone please ec2 land lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs?
<lifeless> deryck: I like your team strat
<deryck> lifeless, for testing?
<lifeless> and cycle time
<deryck> ah, right.  Cool.
<lifeless> jml: busy?
<lifeless> jpds: if you need a hand on https://code.edge.launchpad.net/~jpds/launchpad/fix_116279/+merge/17940 gimme a shout
<mars> james_w, ping
<jml> lifeless, hi
<lifeless> jml: where could I find you?
<jml> lifeless, community room
<jml> gary_poster, mars: hello. there's a doctest issue?
<mars> jml, looking
<gary_poster> jml: hey.  Did you see my discussion about the number of doctests in #launchpad-code
<gary_poster> With mars
<jml> gary_poster, it's gone down, right?
<gary_poster> That would be the fastest way to catch up
<gary_poster> right
<jml> ok.
<gary_poster> and I think I know why, and that it is ok
<jml> oh good. :)
<gary_poster> but would love confirmation
<mars> looking now
<gary_poster> yeah, cool
<mars> simple summary compare
<jml> gary_poster, can you tell me why you think why?
<jml> rockstar, I'll catch up w/ you re private branch lookup after this.
<gary_poster> jml: "buildbot went from reporting 26777 tests to 8490 tests when jml landed the branch to use stdlib doctest.  It looks like it might be OK: the amount of time didn't change, so I think this is because Zope's doctest counts each block as a test, while the normal doctest code counts the whole thing as one. "
<jml> yeah, that sounds right.
<jml> gary_poster, mars, I don't really know how I'd confirm that, other than comparing the fulltext buildbot output.
<mars> yep, doing so
<mars> test lines before: 8941
<mars> test lines after: 8976
<mars> so yes, a) zope counts doctests multiple times, and b) our test suite runtime sucks three times more than I thought it did
<benji> it's not quite that zope's doctest counts tests multiple times, its that it counted tests slightly differently than the stdlib doctest
<mars> yep
<mars> didn't notice that before
<mars> by the way, I found that a good sized part of our testrunner time is being spent parsing doctests
<mars> which is a pity because it happens in the test discovery phase, and after the doctest is discovered, then the new layer's subprocess means that the work has to be done again
<benji> that surprises me; doctests should be very cheap to parse
<mars> I tried profiling the layer setup, and doctest parsing was a big part of it
<mars> it was volume more than anything
<lifeless> \o/
<lifeless> benji: 'parse' + 'cheap' does not exist in python.
<jml> rockstar, I've commented on the bug for private branches. I figured that was the best way. I'm available for face-to-face convo if you'd like one.
<mars> lifeless, that is part of that mysterious uncounted timesink when you first fire up the runner.  I couldn't isolate .pyc cleanup due to disk caching issues.
<benji> btw, mars: remember last week when we were wondering if the lower-memory EC2 instances would work for ec2 test?  I watched one of the instances when running the tests and it ended up using 2.6G of RAM, of which 1.3G was cache/buffers
<lifeless> mars: nice
<mars> I also suspect the importfascist may be sinking some time, again due to volume of use
<mars> benji, what does 50% memory use as cache/buffer percent mean?
<benji> that's primarily disk cache
<StevenK> That the kernel is using it as a disk buffer
<poolie> salgado, ping for review of https://code.launchpad.net/~leonardr/launchpad/true-anonymous-access/+merge/30088
<leonardr> poolie, salgado says it's ok
<poolie> sweet
<leonardr> i just need to change a comment and i'll land it
<poolie> fire one!
<poolie> thanks for doing it
<salgado> poolie, yeah, did that over IRC some time ago.  just updated the MP, though
<lifeless> jtv1: hi
<jtv1> hi lifeless
<lifeless> jtv1: https://code.edge.launchpad.net/~ursinha/launchpad/add-ec2land-rules-orphaned-branches/+merge/29255 needs a follow up review
<lifeless> per your request to urshina
<jtv> lifeless: cool, thanks for letting me know
<lifeless> bah
<Ursinha> lifeless, I haven't finished that yet
<Ursinha> doing that today
<lifeless> irsinha
<jtv> oh
<lifeless> Ursinha: oh, ok.
<lifeless> I'll put it back to wip for now hten
<lifeless> \o/
<lifeless> no outstanding reviews
<lifeless> -> chemist
<Ursinha> lifeless, ok then, I'm finishing details to resubmit, what I couldn't finish last week :)
<Ursinha> and I have a hilight for urshina here, the mispelling happens quite often I consider myself that too
<lifeless> Ursinha: I'm sorry ;)
<Ursinha> no problem :)
<lifeless> bigjools: hi
<bigjools> hello
<lifeless> bigjools: if you feel particularly strongly either way about admins & PPA creation via the API, please do what you think is right.
<lifeless> bigjools: I'm trying to express a logic chain in the review not to mandate that it go either way.
<bigjools> lifeless: well I don't feel strongly enough to cry if it's not done, as I said on the review
<lifeless> bigjools: the underlying reason why I am arguing against a special case here is that the admins have said they want *less* interrupts.
<lifeless> and given that any user can make a PPA via the web ui already, I don't understand why we'd need an admin override facility here.
<lifeless> bigjools: sure; I'm being explicit that its your choice.
<lifeless> I *feel* that we should default one way; however its always changable, and its in your part of the product :)
<bigjools> lifeless: I understand your concern.  Mine is that one day we'll need it and the facility is not there.  I don't expect LOSAs to routinely get asked to make PPAs!
<lifeless> I'm also still feeling moderately crap and I was communicating badly in the review which is why I've upgraded to real-time ;)
 * bigjools hears ya
<lifeless> bigjools: thinking through it more I have another reason/condition around admins making arbitrary PPA's
<lifeless> we require a CoC signature from PPA users.
<lifeless> would that be done on the admin, or on the user the admin is making the PPA on [in the current code]? If on the admin, then that would be a bug ?
<bigjools> interesting point
<maxb> lifeless: When you're done with this discussion, I have a question for you - In https://answers.launchpad.net/launchpad-code/+question/117193 you asked me to file a bug for the "class of problem" of requiring LOSA intervention to make code imports work. I don't understand why. Shouldn't bugs be about concrete issues that we can fix?
<lifeless> gary_poster: hi
<gary_poster> hey lifeless.
<lifeless> gary_poster: I want to talk cronscripts with you in a bit
<lifeless> I've just realised I should answer maxb, and get dinner first.
<lifeless> gary_poster: the theme will be 'making them safer to upgrade'
<maxb> I will be around later, I would hate to keep someone from their dinner :-)
<lifeless> maxb: the class of problem is 'needing to run ssh <details> and accept the key'
<gary_poster> lifeless: BTW, I intend to write up the PQM/tarmac requirements; I realized this weekend that there are some bits that I still need to clarify
<lifeless> gary_poster: awesome
<gary_poster> lifeless: cronscripts: cool
<lifeless> gary_poster: I saw the split to a subspec
<maxb> lifeless: oh, ok I see what you meant now
<lifeless> maxb: that class of issue is actionable and automatable.
<lifeless> maxb: :)
<lifeless> ok, awol for a little bit
<jml> by "actionable", presumably you don't mean that we can sue.
<lifeless> jml: who knows.... the phantom knows
<jml> shadow.
<lifeless> jml: I'm grabbing a club sandwich if you want to join me
<jml> phantom doesn't know anything.
<jml> lifeless, sounds good
<jml> lifeless, once this ec2 land detaches :)
<lifeless> jml: I has a table
<jml> lifeless, k. still detaching.
<jml> done
<lifeless> gary_poster: hi, so if you're still @ work
<gary_poster> Hey lifeless.  Yeah
<lifeless> to change to the 'release features when they are ready' workflow
<lifeless> we need to drastically simplify and reduce the downtime windows involved in rollouts
<lifeless> james troup and I did a mental audit
<gary_poster> lifeless: we release edge every night
<lifeless> and I'm going to file a bunch of bugs related to this
<gary_poster> isn't that the model we want to pursue?
<lifeless> gary_poster: we do, but - and this is one of the problems with the current setup - that is only a small fraction of the lp instances around.
<gary_poster> ah, ok
<lifeless> gary_poster: so we need to do what we've done for edge and solve the issues
<lifeless> for instance
<gary_poster> ok, fair enough
<lifeless> we have a lot of cronscripts
<gary_poster> (BTW: https://dev.launchpad.net/Foundations/Proposals/SimplifyMergeMachinery)
<lifeless> great, queued it up to read
<gary_poster> cool
<lifeless> so in particular, we have a lot of cronscripts
<lifeless> which run 'prod'
<lifeless> so we will be updating their code more frequently
<gary_poster> run 'prod': that means, there is no "edge" equivalent
<lifeless> I'm wondering if there is something we can do to make it safer to do
<gary_poster> ?
<lifeless> right
<lifeless> so at the moment their 'deploy frequency' is 1/month
<gary_poster> Eh, yeah, I hate the cronscript story.  Touching them is terrifying.
<gary_poster> My long term preference is that they disappear
<lifeless> agreed
<lifeless> however we have to break the deadlock somewhere ;)
<gary_poster> right
<lifeless> 170 cronscripts! [estimated]
<gary_poster> yeah
<gary_poster> so, why is touching them terrifyingL
<gary_poster> :
<lifeless> so the bug is going to be a bit generic
<gary_poster> it is terrifying because the tests for them largely suck, AFAIK
<lifeless> but in general we want to aim at being able to automatically upgrade them
<gary_poster> Is there a problem other than the one I perceive?
<lifeless> in the short term that could be manual but smoother than today.
<lifeless> gary_poster: yes, several.
<gary_poster> cool, should I wait for the bug report? :-)
<lifeless> - they run from cron, so either they try to run while we're upgrading, or they may still be running while we upgrade
<gary_poster> ah
<lifeless> - they run from the 'current' symlink, not the 'active revno dir', so they could in principle do late buggy imports (but this is rare, we can ignore for now)
<lifeless> - some of them have terrible knock on effects if interrupted, and take 15-20 minutes every hour.
 * gary_poster tries to look at the cron scripts so they feel bad about themselves
<gary_poster> ...not much effect...
<gary_poster> I have some vague ideas on how to address this, that have to do with trying to come up with cheap machinery and/or patterns for cronscript devs to follow
<gary_poster> I feel constrained both because I don't have any knowledge or even oversight of the cronscripts, and because I don't want to spend too much time on them, knowing that I want them to die.
<gary_poster> Perhaps it would be reasonable to discuss the machinery that we want to have to replace them in the long term
<lifeless> perhaps
<gary_poster> to see if it would inform the short term stepe we might take
<lifeless> Its late for me right now
<gary_poster> Fair enough
 * benji wonders if gary_poster is thinking of zc.async
<lifeless> I wanted to frame the discussion
<lifeless> broadly, I think we need something to wrap them
<lifeless> to quiesce them when we're preparing to deploy
<gary_poster> benji, sure, but we won't be using that here, I don't think :-)
<lifeless> to alert us when important ones are running still
<lifeless> and to insulate us mid-upgrade
<gary_poster> I assume some sort of LOSA gesture like "I plan to upgrade things in half an hour so let's start quieting things down" would be reasonable
<lifeless> https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadRollout
<lifeless> for now, absolutely
<lifeless> https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadRollout#Comment out Cronjobs and Kill DB Backups (approx 30-40 mins before rollout)
<lifeless> thats terrifying
<lifeless> we have a tool to comment out cronjobs!
<gary_poster> sigh
<gary_poster> Another aspect to this is that the cronscripts themselves have had no constraints on them to my knowledge
<lifeless> indeed
<gary_poster> But again, I suspect that we want to retain that freedom for now
<lifeless> so I plan to conduct an audit
<gary_poster> Or else we will be bogged down
<lifeless> etc
<lifeless> but yeah
<gary_poster> yeah, cool
<lifeless> my basic thought is to have a single constant wrapper around all the scripts
<lifeless> so that rather than commenting them out and hoping they stop etc
<lifeless> the wrapper can check a flag somewhere
<gary_poster> you make some flag to make it a no-op
<lifeless> and not run
<gary_poster> yeah
<lifeless> right
<lifeless> for now, I'm going to file a bug on each area of concern for the new workflow
<gary_poster> OK, sounds reasonable
<lifeless> this one is the least well formed
<lifeless> which is why I wanted to kick off some neuron
<lifeless> s
<gary_poster> Cool
<gary_poster> Oh, I'll just choose one :-)
<lifeless> :)
<lifeless> \o/ lsprof has landed
<lifeless> thanks mars
<gary_poster> great news, mars
<gary_poster> and thanks to you too lifeless
<lifeless> de nada
<mars> cool, one step closer
<gary_poster> ...so, yeah, such a cron script wrapper could be fairly complex.  I think the trick will be to come up with a plan for one that is as simple and failsafe as possible.  I'l think about it, and I'll hope to brainstorm with you about it later.
<lifeless> great
<lifeless> elmo: around?
<lifeless> elmo: I'm trying to remember whats on loganberry that is rollout-sensitive
<benji> what's the normal turn-around for a (relatively simple) RT?
<lifeless> yes
<lifeless> more precisely, its either in the priority queue, or 'sometime over the horizon' - like us, they have more incoming than outgoing.
<lifeless> so you should talk to francis
<mtaylor> rockstar: what's the reasoning behind the need for a merge request to have a commit message?
<mtaylor> lifeless: not sure it's relevant, but on my openstack dev deployment, I'm using a hudson master and scheduled jobs instead of cron to do repetitive tasks
<lifeless> losa ping
<lifeless> abentley: hey, can I ask you a question about the importd's
<lifeless> flacoste: rt 40477
<flacoste> lifeless: before or after staging PQM deployment?
<flacoste> (assessing priority)
<flacoste> RT 39150
<lifeless> yeah
<lifeless> staging pqm will only be useful when I get some cycles to do the thing for gary
<lifeless> so before I think
<lifeless> I was crook in the weekend, so didn't get anything interesting done.
<abentley> lifeless, pong
<lifeless> hey abentley
<flacoste> lifeless: ok
<flacoste> leonardr: is RT 39922 still relevant?
<lifeless> I wanted to know about the interactions between upgrades and the importd service
<leonardr> flacoste, humor me and tell me the summary while i look that rt up
<leonardr> flacoste: got it
<leonardr> flacoste: no, the performance test has been run and there is no need to do anything else right now
<lifeless> flacoste: also rt 40480
<lifeless> abentley: so if I wanted to upgrade the code on the importds, does it imply downtime, and interruption to long running tasks, and if so how much downtime and how much interruption.
<flacoste> leonardr: thanks, i'll make it as obsolete or invalid watherver their status is
<flacoste> lifeless: while I look at that other RT, mind looking at my follow-up to your review :-)
<lifeless> sure
<flacoste> lifeless: bumbed above PQM staging deployment
<lifeless> thanks
<lifeless> gnight
<abentley> lifeless, I don't know the deployment story of the importds.
<lifeless> abentley: who would? [that knows the code too, so we can talk about how to make it slick]
<abentley> lifeless, mwhudson.
<lifeless> thanks
<flacoste> gary_poster: do you remember if l-d-d is updated every buildbot run?
<gary_poster> flacoste: I think so, but things may have changed.  want me to look?
<flacoste> gary_poster: i don't think it would have changed, i remember not being the case
<flacoste> so if we fixed that at some point, i'm pretty sure it works now
<m4n1sh> is the staging server getting an update?
<gary_poster> looking'
<m4n1sh> it breaks when using API
<gary_poster> flacoste: ah, no, it does *not* update l-d-d
<lifeless> Ursinha: have you or matsubara asked for the daily staging setup yet ?
<lifeless> in RT I mean, if not, I will.
<gary_poster> (matsubara is out of office)
<gary_poster> m4n1sh: yes, getting update
<m4n1sh> gary_poster, thanks. does it happen daily?
<gary_poster> yes, m4n1sh
<m4n1sh> gary_poster, Thanks for the info
<gary_poster> np
<mtaylor> rockstar: tags don't seem to be propgating through tarmac
<flacoste> lifeless: can you remove your Needs fixing?
<flacoste> vote
<flacoste> so that ec2 land doesn't stomp me on the fingers
<mwhudson> morning
<m4n1sh> gary_poster, is there any attribute on launchpad API's me object which is private to all?
<Ursinha> me
<Ursinha> lifeless, no
<m4n1sh> gary_poster, I mean, I want to use it as an auto password... that attribute should not be visible in any case on launchpad.net/~foo
<lifeless> Ursinha: ok, I've filed RT 40482
<lifeless> flacoste: that seems buggy
<lifeless> flacoste: like, it should trust that when I say approve, I'm not being schizotic.
<lifeless> flacoste: I've fixed it but perhaps you could make a sane bug report up
<lifeless> ?
<flacoste> i will
<lifeless> thanks!
<lifeless> I'm really going to sleep now.
<deryck> Later on, all....
<flacoste> the bug is that it looks at the approver to determine the r=
<flacoste> and since there are no reviewers approving...
<lifeless> flacoste: right, but *my* vote as a reviewer has to be treated as approve
<lifeless> flacoste: given I approved the merge.
<lifeless> ciao
<flacoste> chao, sleep weel
<flacoste> well
<gary_poster> m4n1sh: mmm, I stared at your questions and I think I understand sort of.  You want to hack/abuse an attribute in some way, right?
<m4n1sh> gary_poster, actually i want to use launchpadlib in a django app
<gary_poster> cool
<m4n1sh> but django needs a password in authentication module
<m4n1sh> but i want to use launchpad OAuth
<m4n1sh> so when the user returns after giving the permissions
<m4n1sh> i dont want him to ask the password
<m4n1sh> but there should be some field which is private to the user
<gary_poster> oh!  so...if the person is authenticated by launchpadlib, he's ok by you?
<m4n1sh> which i can use as the implicit password
<m4n1sh> yes
<m4n1sh> it is
<m4n1sh> everything is done..
<gary_poster> OK
<m4n1sh> but just to satisfy django's authentication module i need a password
<gary_poster> You can't override it?  In the abstract, that feels like the right thing to do, but I don't know Django's auth stuff so I'm just waving my hands.
<m4n1sh> same here. I thought there should be way around
<m4n1sh> django has a openid module
<m4n1sh> gary_poster, have a look here
<m4n1sh> http://docs.djangoproject.com/en/1.2/topics/auth/#authentication-in-web-requests
<m4n1sh> esp in authenticate() method
<gary_poster> Well, it sounds like you are in hack land
<gary_poster> If you are in hack land...
<m4n1sh> the thing wont keep quiet without a password
<m4n1sh> http_etag would suffice?
<gary_poster> no
<m4n1sh> if i am in a hack land then?
<m4n1sh> gary_poster, i am not able to think any way to overcome this password part. Something needs to be done
<gary_poster> can you just say that the password for *everyone*is an empty string, or some other hardcoded blather, but make sure that it is not actually usable?  If you have a launchpadlib object for the person, then you stuff the hardcoded password in; otherwise, you make sure that the machinery doesn't work?
<gary_poster> To be clear, this sounds absolutely horrible to me :-)
<m4n1sh> gary_poster, same here. and i have no idea about the security aspect too
<m4n1sh> if i go this way
<gary_poster> If I were you I'd be banging around on Django docs or code or mailing lists or something to figure out a way to subclass the auth machinery
<gary_poster> stashing the password...
<gary_poster> ...is hackland. :-/
<gary_poster> I'll look in a little bit if you really want me to (I don't know of an answer offhand)
<gary_poster> but I have to hope Django lets you substitute this bit
<m4n1sh> gary_poster, I think so
<m4n1sh> I found django-oauth too, but launchpadlib is so darn easy that I don't want more stacked up modules
<gary_poster> sure
<m4n1sh> probably I would go by the hack you suggested. Let's see.
<m4n1sh> probably I might shoot off a mail to django mailing lists. I know lots of flames might return, but still I will learn something new
<m4n1sh> gary_poster, well looks like you are busy. I don't want to disturb you anymore
<m4n1sh> Thanks a lot for your help
<gary_poster> m4n1sh: sorry it wasn't more, but yeah, busy. :-/  http://docs.djangoproject.com/en/dev/topics/auth/ seems to imply that looking at set_unusable_password might help; and that trying to see how LDAP support works might give you a pattern to follow
<m4n1sh> gary_poster, I think you got exactly what I was looking for.
<m4n1sh> I am an idiot. Looked at the whole page, but couldn't find out that very line
<gary_poster> :-)
<m4n1sh> it is like a fine print :)
<gary_poster> takes another person sometimes
<m4n1sh> gary_poster, I think the last hurdle also solved
<gary_poster> awesome!
<m4n1sh> now I can go ahead and start working on the app
<m4n1sh> thank you very much (no this is not sarcasm, I mean it)
<gary_poster> lol
<gary_poster> np glad I could help
<flacoste> mars, gary_poster: i got errors compiling numpy!?!
<flacoste> on ec2 land
<mars> flacoste, that's odd.  I have not heard of that.
<flacoste> mars: actually, that's not the real error
<flacoste> never mind
<mars> ok
<flacoste> i forgot to add commit the dep in download-cache
<wgrant> Can someone please ec2 land lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs?
<maxb> urgh. Do we really need a rabbitmq server just to develop any bits of launchpad?
<wgrant> You will soon.
<wgrant> But it will probably be started by make run soon.
<wgrant> mwhudson: Could you be convinced to land the three branches I referenced above?
#launchpad-dev 2010-07-20
<wgrant> spm: Hi. I'm trying to QA my fix for bug #592573. The differences between the right panels on https://launchpad.net/builders and https://edge.launchpad.net/builders show that there are ~146 builds in some sort of limbo. Do you have a moment to do a bit of DB poking to work out what they are and why?
<_mup_> Bug #592573: BuilderSet.getBuildQueueSizes doesn't consider non-binary builds <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/592573>
<spm> wgrant: hrm. curious. sure. I guess the ones you have via the bug is a good starter?
<wgrant> spm: SELECT id, builder, lastscore, job, job_type, processor FROM buildqueue WHERE virtualized=true;
<wgrant> At the moment that query *should* be empty.
<wgrant> But it will probably return around 61 rows
<spm> (23461 rows) ho ho ho ho
<wgrant> Oh, oops. Forgot a join.
<spm> heh, np
<wgrant> SELECT id, builder, lastscore, job, job_type, processor FROM buildqueue JOIN job ON job.id = buildqueue.job WHERE virtualized=true AND job.status = 0;
<spm> you did get the '61', just missed the 23,400 as well. so.. not too shabby?
<wgrant> SELECT buildqueue.id, builder, lastscore, job, job_type, processor FROM buildqueue JOIN job ON job.id = buildqueue.job WHERE virtualized=true AND job.status = 0;
<spm> yarp. 61 rows
<spm> one looks suspiciously old. very low job# compared to the rest
<wgrant> There should be nothing sensitive there (I excluded logtail). Can you pastebin, please?
<spm> http://paste.ubuntu.com/466237/
<wgrant> Oh wow :(
<spm> hrm? in what sense?
<wgrant> Can you 'SELECT MAX(id) FROM buildqueue;' just to see how old those are?
<spm> 3708959 eek
<wgrant> Ah, so nothing new. Good.
<wgrant> Just from the early days of the move to the job system, I suspect.
<wgrant> (those are all queued builds that are being ignored, so are somehow corrupt)
<spm> ahh I see. is this something that should be cleaned up? or can be happily ignored?
<wgrant> We need to clean it up. I guess I'll talk to Julian about it.
<spm> oki, ta
<wgrant> SELECT * FROM buildpackagejob WHERE job=2691238;
<wgrant> I suspect it's the BuildPackageJobs that are missing.
<spm>    id   |   job   |  build
<spm> --------+---------+---------
<spm>  495749 | 2691238 | 1684155
<wgrant> Damn.
<mtaylor> spm: ola amigo. como estas
<mtaylor> spm: or, should I say - Â¿como estas?
<spm> mtaylor: alas, my spanish is limited to that picked up via overhearing Dora the Explorer. So beyond Hola and Gracios(sp?) you've lost me :-)
<mtaylor> spm: Â¿como estas? means "what's up?" - although I honestly don't speak much myself
<spm> heh
<mtaylor> but I'm in cozumel, mx, so I'm trying to make an effort past "uno mas magarita, por favor"
<wgrant> spm: One last one:
<wgrant> SELECT buildqueue.id, builder, lastscore, buildqueue.job, job_type, processor, build FROM buildqueue JOIN job ON job.id = buildqueue.job JOIN buildpackagejob ON buildpackagejob.job = job.id WHERE virtualized=true AND job.status = 0;
<wgrant> Then I can examine the builds myself.
<spm> mtaylor: ha!
 * mtaylor cringes that you're implementing queues in the database... then decides to shut his mouth before he's asked to fix it
<spm> wgrant: http://paste.ubuntu.com/466245/
<spm> what do you mean *implementing*?!? implemented. :-)
<mtaylor> *shudder* :)
<wgrant> mtaylor: Before my time :(
<wgrant> This is five years old :(
<spm> I believe plans are to move to something like rabbit, or whatever. but nfi.
<mtaylor> wgrant: it's ok - in my days as a mysql consultant, I saw _many_ _MANY_ thing implemented inside of an RDBMS that didn't belong there
<mtaylor> but my favorite things people mis-use dbs for are queues and email
<mtaylor> free tip #1 from formerly over-paid db consultant - if your app needs some sort of message system that's similar to email ... USE EMAIL SERVERS .. don't half-way implement private broken email in database tables
<mtaylor> spm: :)
<wgrant> spm: Ah, it's not a bug after all. Most of those are builds that were cancelled by manual SQL to mark them superseded.
<spm> ah
<wgrant> Not all, but most. I'll work out how to clean them up.
<wgrant> Thanks.
<spm> mtaylor: that sounds suspiciously like heresy. email for email!?!? I mean, srsly!?!?
<spm> wgrant: np
<mtaylor> spm: I know - right? you should have seen the look on the client's face the first time I suggested that as the fix for their performance isues
<mtaylor> issues
<mtaylor> "um, hai! you've implemented email in php ... try learning the internet"
<spm> heh
<spm> NIH. Alive and well.
<mtaylor> yup. also "I don't want to learn anything"
<lifeless> morning
 * wgrant is still looking for someone to land three branches.
<mwhudson> here's two fun lines to have next to each other:
<mwhudson> from canonical.launchpad.layers import WebServiceLayer
<mwhudson> from canonical.testing.layers import FunctionalLayer
<lifeless> mwhudson: can you help wgrant out?
<lifeless> pretty please?
<mwhudson> ah right
<mwhudson> wgrant: url me up
<wgrant> mwhudson: lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs
 * mwhudson grrs at not being able to give branch urls to ec2 land
<wgrant> Can't you?
<wgrant> I thought you could now.
<mwhudson> well i didn't work for these
<wgrant> Does anyone happen to know if BFB works for Hardy yet?
<mwhudson> wgrant: ok, three instances started
<mwhudson> wgrant: did it not work for hardy at some point?
<wgrant> mwhudson: Oh, right, I remember now.
<wgrant> Its bzr was old, so it didn't work with 2a.
<wgrant> Thanks for sending those off.
<wgrant> mwhudson: Ah, no, it is really broken like this: http://launchpadlibrarian.net/52193804/buildlog.txt.gz
<wgrant>   bzr-builder: Depends: python-debian but it is not going to be installed
<mwhudson> wgrant: weee
 * wgrant builds it the old way :(
<lifeless> wgrant: hi
<wgrant> lifeless: Hi.
<rockstar> mtaylor, re: Tarmac needing commit message set... The reasoning is because Tarmac doesn't know what to set the commit message to when it commits if you don't specify it.
<rockstar> mtaylor, I realize that this is rather awkward. Merge queues will fix that.
 * rockstar keeps rolling the squeaky wheel
<poolie> what advice do we normally get to people who don't receive the account confirmation mail?
<lifeless> check spam
<wgrant> SSO does hate some domains, though.
<poolie> or they hate us?
<poolie> is there any escape?
<wgrant> Well, OK, one of my email aliases that is handled by a Canonical machine does not get LP or SSO email.
<lifeless> it becomes a losa ping
<wgrant> Everything else works.
<wgrant> So there's something not quite right about some email setup somewhere.
<lifeless> wgrant: orly? have you filed an rt?
<wgrant> lifeless: The address isn't important to me, so I never really bothered.
<wgrant> lifeless: What was that ping before about?
<lifeless> buildds
<lifeless> seen the thread on deployment?
<wgrant> Yes.
<wgrant> They don't care if the connection dies.
<lifeless> can you confirm or deny slave behaviour ?
<wgrant> You can kill buildd-manager at any time, and it will be fine.
<lifeless> \o/
<lifeless> elmo: ^
<lifeless> wgrant: what other things can go wrong
<wgrant> buildd-manager and the protocol are pretty stateless.
<wgrant> (apart from DB build state)
<lifeless> sure
<lifeless> wgrant: is there anything that can be done to make the publisher runs able to be interrupted without causing havoc ?
<wgrant> lifeless: Um.
<wgrant> They shouldn't be tooo bad at the moment.
<lifeless> any manual recovery == too bad
<wgrant> For PPAs it would be really bad.
<wgrant> For primary it should just about work.
<wgrant> (primary has atomic dists/ update; PPAs do not)
<wgrant> Phase A (file publishing) is fine, since it doesn't matter if we publish something again on the next run.
<wgrant> B (domination) is all in one transaction, and doesn't touch the filesystem at all.
<wgrant> But C and D touch indices, so will leave the archive inconsistent.
<wgrant> Hm, actually, it's slightly worse.
<wgrant> If we get through A fine, commit, then get terminated, there'll be no dirty pockets to force index regeneration when everything is rerun.
<lifeless> 2pc needed ?
<wgrant> That would mean a half-hour transaction.
<lifeless> ugh
<wgrant> (alternatively we could make the publisher less goddamn slow)
<lifeless> or start domination later ?
<wgrant> So, there are two problems:
<lifeless> there are two sorts of people in the world ....
<wgrant> 1) We rely on state internal to the publisher process to work out whether we need to regenerate indices.
<wgrant> 2) Termination during index generation will leave the archive inconsistent for at least a few minutes.
<lifeless> seems like they are both addressed by making indice regeneration into a separate task
<lifeless> pipeline-like
<wgrant> Applying the atomic dists/ update system to all archives would solve #2 simply.
<wgrant> For #1... we may need to store dirty pockets in the DB.
<lifeless> also less code variation - ++
<wgrant> Perhaps.
<wgrant> But the way it's done now is not exactly... acceptable.
<mrevell> GUten morgen.
<lifeless> wgrant: are bugs filed to fix it?
<wgrant> lifeless: It's cron.publish-ftpmaster. Everyone is terrified.
<lifeless> is it risk ?
<wgrant> It's evil and does some things that nobody understands any more.
<wgrant> Well, I guess that's just dsync.
<wgrant> Morning bigjools.
<wgrant> You may be, er, interested to know that we have somewhere around 146 inconsistent builds on production.
<wgrant> They have BuildQueues and Jobs, but are not actually pending.
<bigjools> sigh
<bigjools> givebacks?
<wgrant> Some are SUPERSEDED, others are FULLYBUILT.
<bigjools> did I just catch lifeless looking at cron.publish-ftpmaster?
<wgrant> The former can be explained by LOSAs manually cancelling builds, I suppose.
<bigjools> I expect so - doing it wrong.  Are they package builds?
<bigjools> binary that is
<wgrant> They're all BPBs, yes.
<bigjools> are they rebuilds?
<wgrant> http://paste.ubuntu.com/466245/ are the virtualized ones. Although there are a couple of legitimate builds that snuck in there.
<bigjools> how was this discovered?
<wgrant> The ones with score == 0 are the buggy ones.
<wgrant> I was QAing my getBuildQueueSizes change.
<wgrant> Compare the build queue sizes on edge and production.
<bigjools> sigh
<bigjools> lastscore=0 indicates a retry
<wgrant> Ah, true. I guess it would be NULL if they'd just finished naturally.
<wgrant> They're fortunately all fairly old.
<bigjools> I need to clean up the rebuild anyway
<wgrant> There are 80 or so non-virt builds which I didn't query for, since the non-virt build queues are large.
<bigjools> how old?
<wgrant> Well, we're up to BQ 3700000 or so.
<wgrant> Most of them are around 3330000
<bigjools> did you get any dates?
<wgrant> They should be in Job, but I didn't query for them, no.
<wgrant> I decided I'd bothered people enough.
<bigjools> it's so much harder to query across jobs now :(
<wgrant> http://paste.ubuntu.com/466245/  has the query
<bigjools> I have some pre-potted ones
<wgrant> Just throw a couple of extra columns like BPB.status and Job.date_created in, I suppose.
<bigjools> argh, I can't even do this on staging any more
<wgrant> No perms?
<bigjools> no, we wipe the queues when restoring
<bigjools> to stop staging collecting build from production
<wgrant> Oh, true.
<wgrant> So, the few builds on there from before 3300000 are all SUPERSEDED.
<wgrant> Then there are a couple around 3300000
<wgrant> And the rest around 3330000
<wgrant> I haven't checked the statuses for the last two categories, though
<bigjools> wgrant: there's one (!) like that on dogfood
<wgrant> bigjools: What's its status?
<wgrant> Build status, not Job status.
 * bigjools rides SQL wild horses
<lifeless> bigjools: asking about things that will make deployment hard
<lifeless> bigjools: easy deployment is a really important thing for iterating on performance at faster than monthly cycles.
<bigjools> lifeless: do you know how it will make it hard?
<lifeless> bigjools: no idea, wgrant mentioned it was all
<bigjools> other than being a PoS bash script, I don't see the problem
<lifeless> bigjools: for the publisher specifically I'm told that interrupting it causes damaged PPA archives until its run again, and in some cases until the PPA has a new build and the publisher runs again.
<bigjools> yes it's possible, but easy to fix
<wgrant> bigjools: If we die during !primary index generation, the indices will be inconsistent until the next run.
<wgrant> And if we die before index generation, there'll be no dirty pockets, so no index regeneration will occur.
<bigjools> we need to write to a tmp dir like ubuntu does
<wgrant> That's what I suggested.
<wgrant> But it's not trivial to port, given how it's done now.
<bigjools> and remove the stupid partial commits
<wgrant> We can't do that until we publish everything really quickly.
<lifeless> bigjools: the impact of that on the deployment story is that we have to be careful about when we start deployments, shutting down the task well in advance, which adds latency and that adds perceived downtime.
<wgrant> I think a half-hour transaction over the whole primary archive would make stub cry a bit.
<lifeless> none of this is intractable, but the more steps that are needed, the more coordination, the slower the process is.
<bigjools> yeah, whoever wrote the publisher knew nothing about recoverability
<lifeless> yup
<wgrant> bigjools: Alternatively, we could write indices atomically like the primary archive, and store dirty pockets somewhere persistent.
<wgrant> Everything else should work.
<lifeless> wgrant: everything you do to make the system more resilient, j'adore
<bigjools> me likey atomic
<wgrant> Or we make the publisher complete in a few seconds, and do it all in one transaction.
<wgrant> But I can't get it much below two minutes.
<lifeless> can you make it pipeline / incremental ?
<wgrant> The thing that takes all the time (after optimisation) is serialising the indices.
<lifeless> does the db know the indices ?
<bigjools> pipelining is possible but has ramifications
<lifeless> could you, just commit, then do the indices as a read-back from the db ?
<lifeless> (in principle)
<bigjools> lifeless: you mean lazy generation?
<bigjools> not sure I follow
<jml> I'm fixing the conflict, btw
<jml> is there any meaningful difference between IStore(self) and Store.of(self)?
<jml> IStore uses c.l.webapp.adapter.get_store
<lifeless> I think the second reads more easily
<wgrant> Does the former respect the master/slave policy?
<lifeless> bigjools: I mean doing the transaction commit as soon as possible
<lifeless> bigjools: and generating the indices from the result of the commit, not within the commit
<wgrant> lifeless: The problem is that if we commit before doing indices, we don't know that we need to regenerate them later.
<bigjools> lifeless: what wgrant said
<lifeless> wgrant: that seems fixable
<bigjools> hehe - take a look at the publisher :)
<wgrant> lifeless: Right, by storing dirty pockets in the DB, which fixes it all.
<wgrant> But is very ugly.
<lifeless> compared to 30 minute db transactions with uninterruptable cron scripts that prevent rollouts for 30% of the day.
<lifeless> not ugly at all.
<wgrant> Shhhhh.
<lifeless> I'm very keen to see things here improved.
<lifeless> The general principle of 'do small amounts of work, often' and 'delay till outside of transactions things that don't need to be in the transaction' are very dear to me.
<jml> wgrant, the former goes straight to the master, I think.
<wgrant> jml: Ah.
<bigjools> wgrant: does this look sane to display the affected builds? http://pastebin.ubuntu.com/466361/
<mwhudson> wgrant: your branches were 1 of 3, did you get the emails?
<wgrant> mwhudson: Yeah, saw that. Thanks.
<wgrant> bigjools: I would have said buildfarmjob.status NOT IN (0, 6), but looks OK.
<bigjools> wgrant: using NOT IN makes queries slow
<wgrant> bigjools: Ah, I guess so.
<wgrant> bigjools: Also, grab buildqueue.virtualized.
<bigjools> why?
<wgrant> Why not?
<wgrant> Might as well get all the categorisation information.
<wgrant> I only omitted it from the initial query because I was restricting to virt jobs.
 * wgrant grumbles about the incomplete Registry<->Soyuz split.
<wgrant> mwhudson: Tests fixed. Can you please rerun, if you're still around?
<wgrant> I guess not.
<wgrant> Can someone else please re-EC2 lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts?
<bigjools> wgrant: problematic queue rows blitzed, how's it looking?
<wgrant> bigjools: So none of them were actually pending?
<bigjools> some where but I left those alone ;)
<bigjools> sigh
<bigjools> some *were*
<wgrant> That looks OK.
<wgrant> The numbers match now.
<bigjools> \\o/
<wgrant> I will be really glad when we get the model rework done, and such inconsistency becomes impossible.
<bigjools> yarp
<wgrant> Because we will be able to avoid storing the status in four or five places...
<bigjools> although notice that all of those were from march/april
<wgrant> Yeah.
<bigjools> wgrant: https://edge.launchpad.net/~oem-archive/+archive/budapest/+build/1880257
<bigjools> can you see that?
<wgrant> bigjools: No. Is that the one that failed to upload this morning?
<bigjools> yes
<wgrant> I considered going hunting for the upload log, but decided the search space was slightly too big.
<bigjools> only it says it's built properly so it looks like it dispatched twice :/
<wgrant> What was the upload error?
<bigjools> PM
<wgrant> Or is it FULLYBUILT now?
<wgrant> ?
<wgrant> oH.
<wgrant> Right.
<bigjools> it's built
<wgrant> So it was built four times.
<wgrant> Succeeded the first and last.
<wgrant> But somehow was retried after the first.
<wgrant> Yay.
<wgrant> I thought we'd weeded out all of that :(
<bigjools> wgrant: I suspect double clicking on UI buttons
<wgrant> bigjools: But... transactions.
<bigjools> wgrant: that already causes havok on copy packages
<wgrant> For copies, sure.
<bigjools> wgrant: they don't help if the db constraints don't catch the problem
<wgrant> bigjools: But a retry resets the BFJ status.
<wgrant> They both have to update the same row.
<lifeless> I have a suspicion something is double-forwarding every now and then
<lifeless> or something
<bigjools> to the same thing
<lifeless> see the bug I filed about getting two bugs
<wgrant> bigjools: I really hope that Postgres doesn't accept that.
<bigjools> wgrant: unless there's a constraint, it will
<bigjools> lifeless: really? ugh
<lifeless> elmo: https://bugs.edge.launchpad.net/soyuz/+bug/607397
<_mup_> Bug #607397: buildds need to survive the buildd master being upgraded <Soyuz:Incomplete> <https://launchpad.net/bugs/607397>
<lifeless> elmo: can you please describe there the build farm issue you related to me - or perhaps its no longer an issue ?
<wgrant> bigjools: With SERIALIZABLE as the isolation level, a concurrent update like that is prevented.
<wgrant> I just checked.
<wgrant> I wonder what Storm uses, though.
<lifeless> elmo: nvm
<wgrant> It default to serializable, as I'd hoped.
<wgrant> Ahh.
<wgrant> But LP overrides it to READ COMMITTED, and that allows it.
<wgrant> Damn.
<lifeless> bwah
<lifeless> all of lp  is read committed ?
<wgrant> I think so. Need to check Postgres logs harder.
<bigjools> that's.... not good
 * wgrant looks harder.
<wgrant> At least the appserver transactions immediately set it to READ COMMITTED, or so the postgresql logs show.
<lifeless> bah
<lifeless> wgrant: you broke our builds
<lifeless> :P
<wgrant> lifeless: It was too quick to be my fault :(
<lifeless> Build Reason:
 * wgrant blames the build system.
<lifeless> Build Source Stamp: [branch bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel] HEAD
<lifeless> Blamelist: William Grant <me@williamgrant.id.au>
<lifeless> BUILD FAILED: failed shell_6 compile
<wgrant> Oh yes, I got the email.
<wgrant> But the test suite doesn't take an hour.
<wgrant> So either I broke things really badly, or the build system is broken as usual.
<wgrant> Or someone turned on parallelisation while I wasn't looking.
<lifeless> no
<lifeless> not done yet
<wgrant> bigjools: Ah, wait, there's a UNIQUE on buildpackagejob.build. So you can't queue the build twice regardless of how screwed LP's default transaction level may or may not be.
<bigjools> that's the kind of constraint I like
<wgrant> So we have more hunting to do.
<bigjools> grar
<wgrant> Anything in the logs yet, or must we go librarian diving?
<bigjools> hang on
<wgrant> Hmm. The librarian GC is rather aggressive :/
<lifeless> mpt: hai
<lifeless> mpt: lunch is @ 1
<lifeless> mpt: are you planning to starve me?
<wgrant> It will immediately kill anything unreferenced and with no expiry date set.
<mpt> lifeless, clearly, by "1" I meant "2"
<lifeless> awesome
<lifeless> Ursinha-afk: how does the oops <-> bug stuff work ?
<lifeless> wgrant: are you working on 78 SELECT COUNT(*) FROM Archive, BinaryPackageBuild, BuildFarmJob, PackageBuild WHERE distro_arch_se ... tus=$INT AND Archive.purpose IN ($INT,$INT) AND Archive.id = PackageBuild.archive AND ($INT=$INT):
<lifeless>  ?
<wgrant> jml: Um, canonical.uuid has been gone for more than a hundred devel revisions...
<wgrant> lifeless: I don't really know how to fix it.
<wgrant> There's nothing obviously wrong with it that I can see.
<wgrant> http://paste.ubuntu.com/465800/ is the EXPLAIN ANALYZE of the other slow query, which is just about the same.
<lifeless> right
<lifeless> so dropping the count * separate query will save 5 seconds
<lifeless> sorry 7.7
<wgrant> But that's lazr.restful.
<wgrant> Not sure we can do much about that.
<wgrant> And the query shouldn't take long at all anyway.
<lifeless> sure we can
<lifeless> is there a bug for it ?
<wgrant> The slow queries?
<jml> wgrant, I'm just the messenger.
<lifeless> the problem
<wgrant> I filed one a month or two ago.
<lifeless> whats the number
 * wgrant is hunting.
<wgrant> If bug search was a little better...
<lifeless> hush
<wgrant> Bug #590708
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<wgrant> jml: Well, since I have no recent shipit, I can do nothing.
<lifeless> bigjools: hi - https://bugs.edge.launchpad.net/soyuz/+bug/590708
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
 * wgrant just commented with the paste.
<wgrant> lifeless: I wonder if we should test kernel delayed copies and acceptance from +queue before taking the timeout down permanently. Those are done infrequently, take ages, and it's pretty bad if they stop working.
<lifeless> wgrant: can you add a test plan for testing them on staging ?
<lifeless> wgrant: staging is at 12 seconds.
<wgrant> lifeless: Um, I'm not sure if testing on staging is valid.
<lifeless> wgrant: why wouldn't it be ?
<wgrant> lifeless: It sucks performance-wise.
<lifeless> right
<lifeless> so if it works on staging, we're set for prod.
<wgrant> True.
<wgrant> bigjools: Any luck? It'd be nice to get onto it before librarian-gc deletes the evidence.
<bigjools> wgrant: no, best start diving
<wgrant> bigjools: I guess you could just look for any recent restricted 'uploader.log's...
<bigjools> urh
<wgrant> Since buildd-manager's log doesn't seem to be much help in this sort of situation.
<lifeless> wgrant: so, I've commented and escalated this
<wgrant> lifeless: Thanks.
<lifeless> wgrant: I think lazr restful is hurting us here and we may want to change it.
<wgrant> lifeless: Possibly, that probably breaks all clients in the wild.
<lifeless> separately fixing up the query to not do table scans - +1
<lifeless> flacoste: http://people.canonical.com/~flacoste/tags-burndown-report.html is not updating ?
<lifeless> wgrant: on this url, they are already broken.
<deryck> Morning, all.
<lifeless> it was timing out regularly on prod before
<lifeless> now its just -clear-  :)
<lifeless> hey deryck
<wgrant> Can someone please re-EC2 lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts?
<bigjools> wgrant: I can do it locally if nobody volunteers ec2
<bigjools> btw I haz logs
<wgrant> Ooh, logs.
<wgrant> Are the logs useful?
<bigjools> somewhat, I'm PM you
<poolie> lifeless, hi?
<danilos> adiroiban, hi, it seems there's a conflict in +templates fix of yours now (I'm trying to land it); can you please take a look and fix it :)
 * wgrant still needs someone to land those two branches.
<adiroiban> danilos: Hi. Looking...
<danilos> wgrant, got MPs for me that I can just pass into "ec2 land"? (fwiw, I had some problems with ec2 land lately, so I am not promising it will work)
<wgrant> danilos: Thanks. https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667 and https://code.edge.launchpad.net/~wgrant/launchpad/bug-598345-restrict-dep-contexts/+merge/30203 are the MPs.
<jml> mars, hi
<mars> Hi jml, what's up?
<jml> mars, I don't understand your recent email.
<mars> jml, the "Hurray for failing fast" one?
<jml> mars, yes. the build *did* go on to fail with indecipherable errors.
<mars> build 1066, right?
<mars> According to the waterfall, I see "pull new revisions [failed]", and "compile [failed]", and that's it
<mars> According to this it never ran the test suite
<jml> mars, it still tried to.
<jml> mars, and the error the compile fails with is: zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/srv/buildbot/slaves/launchpad/devel/build/script.zcml", line 7.4-7.35
<jml>     ZopeXMLConfigurationError: File "/srv/buildbot/slaves/launchpad/devel/build/lib/canonical/configure.zcml", line 157.4-158.42
<jml>     ZopeXMLConfigurationError: File "/srv/buildbot/slaves/launchpad/devel/build/lib/canonical/shipit/configure.zcml", line 55.4
<jml>     ImportError: No module named uuid
<jml> mars, I mean, it's definitely better than running the whole test suite.
<jml> mars, which is wonderful :)
<mars> hmm
<mars> I don't know why it moved on to compile_6.  Just a sec, checking the config
<jml> but if a dependent branch had changed in a subtler way, it still would have gone on to run the whole suite
<mars> interesting
<mars> my fix did *not* land
<mars> the compile steps are supposed to halt the build by default
<mars> perhaps that is what caught it
<jml> what caught it was that it's an import error
<mars> mthaddon, ping, was there any word on landing my buildbot "fail fast" config change?
<jml> and the apidoc generation has to import everything
<mthaddon> mars: we landed it but didn't restart the builder - want me to do that now?
<jml> oh I see, there were no steps beyond the compile one.
<jml> actually, no, there were
<mars> mthaddon, sure, there are three branches in there, but we have to do it sometime :/
<mars> jml, yes, but you are right about the subtle changes things.  If it misses pulling GPG or something, then it happily goes onward into the suite.  You are right, we are lucky it happened to fail and halt on the compile step.
<adiroiban> danilos: conflict solved and I have pushed the changes
<adiroiban> danilos: the branch should be ready for ec2-test and landing
<jml> mars, ok cool. I'll gladly watch my two branches get delayed if your fix for that gets deployed & works.
<jml> (will I have to resubmit?)
<jml> actually, I guess it's just force another rebuild
<mars> jml, nope, it will be pulled into the next build
<mars> right
 * jml moves on to his next problem
<jml> how can I run pyflakes on doctests?
<jml> I seem to remember being able to do so
<Ursinha> or, lifeless, hi
<mars> jml, if you don't find something in the list archive, it might be on the Hacking pages
<mars> jml, check your "Sent" folder, dated 13/7/2009, "[EMACS] Another flymake trick"
<lifeless> Ursinha: hi! ;)
<Ursinha> lifeless, you asked about the oops <-> bug link, I assume you're talking about the bug link in the oopses?
<jml> mars, :) thanks.
<Ursinha> lifeless, it was too early in this timezone when you pinged me :)
<lifeless> Ursinha: yeah
<lifeless> Ursinha: and yes, I asked for your backscroll :)
<lifeless> Ursinha: I also filed a bug
<jml> mars, actually, nothing useful in there :\
<Ursinha> lifeless, let me see
<barry> losa ping: bazaar.lp.net seems unhappy. can i haz restart?
<lifeless> barry: hi
<lifeless> barry: whats your bug # ?
<mars> jml, Ah, sorry, thought that mail was right on target.  Maybe the great Warsaw knows
<mthaddon> barry: unhappy in what way?
<barry> mthaddon, can't connect
<mthaddon> barry: via bzr+ssh?
<lifeless> barry: its happy for me
<barry> lifeless, bug #?
<lifeless> barry: try turning on your network
<barry> http://bazaar.launchpad.net/~ubuntu-dev/ubuntu-dev-tools/trunk/files/head:/doc/
<lifeless> ?
<lifeless> barry: you said you had a page timing out
<mthaddon> barry: that's codebrowse - was just restarted
<mars> barry, works for me
<barry> lifeless, ah, yes
<lifeless> barry: also if that page comes up
<barry> lifeless, https://edge.launchpad.net/~pythoneers/+archive/py27stack4/+packages?start=0&batch=200
<Ursinha> lifeless, bug 607087 ?
<_mup_> Bug #607087: enable 'search by method' <OOPS Tools:New> <https://launchpad.net/bugs/607087>
<lifeless> barry: wait 60 seconds and hit ctrl-R
<barry> mthaddon, thanks! works now
<lifeless> Ursinha: no, a new one ;)
<lifeless> Ursinha: I'd love that one fixed too, of course ;)
<Ursinha> hehe
<lifeless> Ursinha: I filed the other one in launchpad-foundations
<Ursinha> ah
<Ursinha> lifeless, do you have the #:
<lifeless> because its about our docs
<lifeless> uhm
<wgrant> barry: Is this another of those 2700-build monsters that gets deleted a couple of hours after it finishes using days of build farm time?
<lifeless> sec
<Ursinha> lifeless, I can find it here, no prob
<barry> wgrant, ;) no
<lifeless> barry: so is that your hacked url
<barry> wgrant, just 150-ish packages
<mars> mthaddon, btw, can we update the buildbot configs trunk with my 'fail fast' patch?  Or do you want to wait for it to run successfully first?
<lifeless> barry: or the original
<barry> lifeless, it is hacked
<lifeless> barry: what url fails
<Ursinha> lifeless, bug 607680
<_mup_> Bug #607680: documentation needed on oops<->bugs linking <Launchpad Foundations:New for matsubara> <https://launchpad.net/bugs/607680>
<wgrant> barry: Ah, good. Sanity.
<lifeless> Ursinha: yes!
<barry> lifeless, that url.  iow.  normally you get batches of 50 but i want to see all packages in one page
<Ursinha> lifeless, what links oopses to bugs is part of the oops-tools
<jml> barry, do you have a copy of pyflakes-doctest, or now where I can find one?
<barry> jml, atm, i don't
<jml> barry, thanks anyway.
<barry> jml, yeah, sorry
<lifeless> Ursinha: ok, thats cool. However most devs don't have that on their disk :)
 * barry -> reboots.  hopefully will brb
<Ursinha> lifeless, exactly :) afaiu it's a simple mechanism that associates the "oops signature" to a bug number
<jml> ahh
<jml> it's in old versions of the tree
<Ursinha> that's why we have incorrect links sometimes
<Ursinha> lifeless, what exactly are you aiming with that?
<flacoste> lifeless: that graph was moved to lpstats
<danilos> adiroiban, wgrant: all your branches are on ec2 now
<flacoste> lifeless: https://lpstats.canonical.com/graphs/LPQA/
<flacoste> lifeless: https://lpstats.canonical.com/graphs/LPQAByTeam/
<danilos> lifeless, I have a suggestion for bug 590708, I've added it to the bug and emailed some reasoning to the list as well
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<wgrant> danilos: Thanks.
<danilos> bigjools, ^
<danilos> lifeless, ignore the timing differences on the bug (that's with explain analyze which usually doubles the times) and in the email though ;)
<wgrant> danilos: Ooh, that's good.
<danilos> wgrant, the traditional translations tricks fwiw :)
<danilos> wgrant, hopefully the queries are compatible :)
<danilos> or, let's say equivalent
<mthaddon> mars: which branch is that again?
<lifeless> Ursinha: I want to be able to make the associations, so I want the way I should do that documented :)
<lifeless> flacoste: ok cool, its linked from https://dev.launchpad.net/PolicyAndProcess/ZeroOOPSPolicy
<lifeless> flacoste: how does oops tie into that graph ?
<lifeless> danilos: actually the explain in the bug case adds about 1000ms if you compare to the oopses
<danilos> lifeless, well, even explain analyze runs quickly (300ms) on an optimized limit 50 query for me
<Ursinha> lifeless, ah, that's bloody simple :) to make the association when there's no association, I mean, because we still cannot edit that other that using sql directly on oops-tools application db
<Ursinha> *other than
<Ursinha> lifeless, I'll make sure that's documented somewhere
<danilos> lifeless, but anyway, I was just trying to point out that the times I recorded are not really correct (I've ran it a number of times with both explain analyze and without), but my conclusions should generally hold for these queries
<lifeless> danilos: yeah
<lifeless> danilos: want to make a patch ?
<lifeless> Ursinha: ok, so how does one do it ?
<danilos> lifeless, not really, I've got a few branches in my queue already :)
<danilos> lifeless, it'd be good to test queries on production DB first as well
<lifeless> flacoste: is zerooopspolicy still active? or died-a-quiet-death ?
<lifeless> mthaddon: could you try danilos query on a slave please? from the bug https://launchpad.net/bugs/590708
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<mthaddon> lifeless: am in the middle of some ISD deployments at the moment - can it wait a little?
<lifeless> mthaddon: of course
<mthaddon> thx
<flacoste> lifeless: it's kind of a not fully implemented policy
<lifeless> ok
<flacoste> lifeless: the intent is still there, but we lack the tools to fully back it up
<Ursinha> lifeless, if an oops doesn't have a bug associated, add the bug number to the text box where the number usually is, click "Bug #", and it's done
<lifeless> but in principle folk have the goahead to shelve stuff to work on oops
<lifeless> Ursinha: oh, on the web ui ?
<flacoste> lifeless: the OOPS report don't make it easy to get the "list of things" that need fixing
<Ursinha> lifeless, as you can see in this one, for example: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1661CCW1
<Ursinha> lifeless, yes sir
<lifeless> ah doh! I see
<Ursinha> :)
<flacoste> lifeless: yes
<Ursinha> lifeless, it clearly needs to be documented! :)
<lifeless> Ursinha: nah, i'm still a little tired here, its been busy ;)
<lifeless> Ursinha: close with 'lifeless is blind', if you like.
<lifeless> flacoste: ok, so lets talk about making it easy to see what needs to be fixed.
<Ursinha> lifeless, hehe, no, it really needs to be somewhere, so we can make easy to see what needs to be done
<lifeless> flacoste: As I get the policy, every oops in the oops report - say just the top 15 timeouts for now - should be a high/critical bug right ?
<lifeless> Ursinha: does the oops report sent to the list list the bugs when there is one ?
<Ursinha> lifeless, yes, right below the oops signature/count
<Ursinha> lifeless, as you can see in the oops reports
<Ursinha> I guess I just repeated what you said, but nevermind :)
<flacoste> lifeless: there should be clear list of things to fix for each team
<flacoste> currently, they have to look for that list across several reports and several questions
<flacoste> s/questions/sections/
<Ursinha> flacoste, I've discussed that with diogo a few times, and we need to improve the oops signature in order to better group the oopses
<Ursinha> done that, I guess the per team's reports will be more accurate
<Ursinha> I also would like to know from the teams which oopses are real ones and which ones are only tainting the summaries
<Ursinha> like the checkwatches one, for example
<flacoste> Ursinha: best way to do that is to book a call with each TL and go over a couple of reports with them
<Ursinha> like bug 592345
<_mup_> Bug #592345: Checkwatches produces a lot of OOPSes that aren't real LP failures <oops> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/592345>
<Ursinha> flacoste, I started with bugs team
<Ursinha> but I'll have to do that with every other one, yes
<lifeless> Ursinha: they're all real.
<lifeless> In my simple simple opinion
<flacoste> lifeless: not really
<flacoste> for example a 404 isn't a real OOPS
<flacoste> and a lot of checkwatches failure are of that sort
<lifeless> flacoste: here - <gmb> Ursinha, We already have a backoff mechanism in place. However, now that we're tracking BugWatchActivity we can probably stop recording OOPSes for some things.
<flacoste> right
<lifeless> flacoste: I agree a 404 isn't a real oops, but the oops system already knows about that one and could trivially skip it
<lifeless> flacoste: (except where we have an internal ref that makes a 404)
<lifeless> barry: so what url was timing out again? not the hacked one, the real one.
<lifeless> Ursinha: help me out with the oops reports here -
<lifeless> the 19th report, edge errors
<lifeless> === Top 15 Time Outs (total of 108 unique items) ===
<lifeless>  876 SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_close ... ON BugTask.bugwatch = "_prejoin5".id
<lifeless> when I open the first oops up
<lifeless> it has a bug #
<lifeless> but I didn't see the bug reference on the page
<Ursinha> lifeless, it's wrong
<Ursinha> hm
<barry> lifeless, i don't think the unhacked one was timing out :/
<Ursinha> lifeless, it has?
<lifeless> barry: ah. Don't hack the url
<Ursinha> lifeless, let me just do my daily call with foundations, I'll get back to you in a moment
<barry> >:-#
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1661A1010
<barry> :)
<lifeless> Ursinha: ^
<lifeless> Ursinha: but it thinks it is a translations problem, yet its a bugs issue
<Ursinha> lifeless, I'll explain
<lifeless> after your call :)
<lifeless> gmb: are you back ?
<Ursinha> lifeless, so
<Ursinha> lifeless, this bug is linked incorrectly because of the "oops signature" oops-tools uses to identify the uniqueness of a problem, and link it to the bug
<Ursinha> lifeless, we're aware that the way it is now isn't good because it doesn't work for timeouts
<lifeless> Ursinha: can we purge the old one then ? I mean, we need a improvement in that eventually.
<Ursinha> lifeless, it's a known issue
<lifeless> but right now its steering folk wrong
<Ursinha> lifeless, sure, I can do that
<lifeless> sweet
<lifeless> Ursinha: do you think we'll be ready to switch to the new workflow this cycle ?
<Ursinha> lifeless, that will require changes in the scripts we use today
<Ursinha> I'm not sure
<Ursinha> lifeless, when the new cycle starts? in three weeks?
<lifeless> 1 week I think, we're in 4 of 10.07 according to topic.
<lifeless> of course, topic could be lying
<lifeless> Ursinha: if you could have a set of bugs tagged release-features-when-they-are-done, I might try to help out on the change
<wgrant> This is 10.08 week 1.
* Ursinha changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<mars> lifeless, do you mean switch to the new merge workflow at the end of this cycle?
<lifeless> mars: thats the question
<Ursinha> lifeless, we still need to create the blesser, and integrate that with the merger infrastructure
<Ursinha> and mars know about the second part better
<lifeless> given its non trivial, its not jfdi-able
<lifeless> I'd like to be able to see where I should help make this happen
<mars> yes, I can think of three separate code changes (including one entirely new script or application)
<lifeless> either by begging some time from team leads, or by helping out and doing, as appropriate.
<lifeless> its a crucial change to make our production story cleaner and better
<mars> well, it will happen - as to an alpha of the new cycle, I think getting that for 10.08 is unknown (I don't know our velocity), but I would think 10.09 for an alpha or even a beta could definitely happen
<Ursinha> I agree with mars
<mars> lifeless, I'll keep this moving forward, and I will let you know how things are going, so you can jump in where you think it's best
<lifeless> if you make it visible to me
<lifeless> I'll help in some fashion
<lifeless> better deployment is a key enabler for overall velocity
<lifeless> and this is a necessary condition for better deployment
<mars> lifeless, it should be on the Foundations Kanban board - I'll make a lane now
<lifeless> wow
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1661C183
<lifeless> mars: so kanban is great too - my main point is if *I* can figure out what is still todo, I can help somehow :)
<lifeless> 14 seconds of non-sql time
<mars> heh, ok
<jml> just to be clear, are our buildbots running Python 2.6?
<jml> (because ec2 test certainly isn't)
<mars> jml, only the lucid_db_lp one is
<jml> mars, thanks.
<jml> next question
<mars> jml, ok, so 'lucid ec2 image' goes on my ToDo list
<jml> how do I use "with" statements in doctests in Python 2.5?
<mars> benji might know?
<gary_poster> from __future__ in a >>> line doesn't work?
 * benji reads the scrollback
<jml> gary_poster, apparently not.
<lifeless> gary_poster: from __future__ is magic IIRC
<lifeless> gary_poster: it changes the parser
<lifeless> IIRC to do that with a non .py compile you need to pass a compile flag in
<benji> jml: I assume you tried "from __future__ import with_statement" and it didn't work.
<jml> benji, that's correct.
<lifeless> deryck: whats 'null bug task' all about
<benji> hmm, there is some support for future in doctests, let me look at the source real quick
<jml> my experimentation cycle is quite long, since I don't know how to build Launchpad locally for Python 2.5
<deryck> lifeless, null is a workaround for not being able to delete a task, since marking it invalid means you continue to receive mail
<lifeless> not the null project
<lifeless> the code path
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1661C183
<lifeless> Page-id: NullBugTask:+index
<lifeless> time: 4388 ms
<lifeless> non-sql time: 14274 ms
<lifeless> Statement Count: 499
<wgrant> erm, shouldn't NullBugTask just redirect now?
<gary_poster> lifeless and bigjools, leonardr and I are discussing his reply to https://bugs.edge.launchpad.net/soyuz/+bug/590708 .
<gary_poster> We are considering the backwards compatibility issues of what he described, because we feel we're the ones who are most likely to care about that, and we are responsible for it.
<gary_poster> If we decide that leonardr's proposal is acceptable, I have the understanding that you are calling this a critical issue and that we should proceed to work on it, pushing our other tasks aside per the usual "critical" behavior.
<gary_poster> (1) Do I understand correctly?  (2) If so, feedback on his reply would be appreciated, particularly if you have concerns.
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<bigjools> gary_poster: sounds good to me - AIUI we are supposed to treat OOPSes as "stop the line"
 * bigjools reading the reply
<gary_poster> I don't think that policy is practically acceptable for Foundations on a global basis, but that's a different conversation for a different forum
<lifeless> mthaddon: https://bugs.edge.launchpad.net/soyuz/+bug/590708 - another for the queue, equal basis with doing the queries from danilo on a slave
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<lifeless> mthaddon: leonardr needs to correspond with the folk having trouble
<lifeless> leonardr: wgrant is one of those people
<wgrant> Hi.
<mthaddon> lifeless: can you ping losa, I've passed the baton to another losa for interrupt queries
<leonardr> wgrant, i'd like to see the program that's getting the timeouts
<leonardr> so that i can see if your program will break if we apply the fix i've proposed
<lifeless> mthaddon: sure, sorry I should have in that case too.
<lifeless> losa ping
<Chex> lifeless: morning
<wgrant> leonardr: http://qa.ubuntuwire.org/ftbfs/source/build_status.py
<Chex> lifeless: or evening in your case?!
<lifeless> Chex: currently mid avo
<lifeless> Chex: as I'm in prague
<Chex> lifeless: ah yes, your sprinting, cool.
<lifeless> Chex: if you look up a bit
<lifeless> there is a bug - one of several - high frequency timeouts
<wgrant> leonardr: It's currently timing out every time it runs.
<leonardr> gary: wgrant's script doesn't use len(), it just iterates over the resultset
<gary_poster> jml: I wanted to abstract the Python selection so that it wouldn't have to always be the system's Python, but it was regarded as unnecessary.  The "test with another version of Python" story is another way in which that feature would be valuable, though.  Maybe it will come to life at some point.
<jml> gary_poster, *nod*
<lifeless> Chex: we'd like two bits of losa assistance in the short term on this - a) danilo has a faster proposed query, we'd like to validate its performance on a production slave.
<jml> gary_poster, I guess it's only really necessary during interim phases like this one.
<gary_poster> right
<lifeless> Chex: b) we may need to get leonardr in contact with some users via api keys
<lifeless> Chex: a) should be cheap - can we do that please; b) ask leonardr in a minute :)
<benji> jml: any futures included in the test globs are respected
<leonardr> wgrant: did it ever work?
<wgrant> leonardr: Until the edge timeout reduction, it worked aboutu 75% of the time.
<wgrant> leonardr: Until 10.05, it worked 100% of the time.
<bigjools> wgrant: it broke with the build farm model changes?
<jml> benji, cool. how would I add a future to a test glob?
<wgrant> bigjools: Somewhat, yes.
<lifeless> gary_poster: I don't have an position-specific opinion on who should do the work; LEAN dictates team accountability (all of LP devs being a single team), not smaller granularity; but we don't seem to do that at the moment : and I'm finding my feet.
<wgrant> bigjools: I suspect it was sitting just under the threshold, or something has gone really wrong plan-wise.
<benji> I'll figure out an example.
<jml> benji, thank you.
<bigjools> ideally we should make the query very very fast
<gary_poster> lifeless: sure, fair enough on all counts
<wgrant> Certainly.
<lifeless> we want to both make it fast, and avoid unnecessary work.
<Chex> lifeless: ok, looking at the bug, and your request
<lifeless> the count(*) there seems to be unnecessary for many cases.
<lifeless> Chex: thanks
<bigjools> unless we make the query(ies) faster, LP will never get faster
<lifeless> right
<Chex> lifeless: im a little confused what you would like me to do for you?
<lifeless> Chex: run the query in https://bugs.edge.launchpad.net/soyuz/+bug/590708/comments/8
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<bigjools> so I think the count(*) is a red herring
<lifeless> Chex: on a slave
<lifeless> bigjools: its not the root cause here, but its not a non-issue.
<bigjools> agreed
<wgrant> COUNT is always going to be fairly slow.
<Chex> lifeless: ok, so one of the DB slaves, seems ok, and the SQL seems to be a SELECT only, so thats ok, too
<wgrant> And it
<lifeless> fixing *either* the query, or the count(*) will fix this issue.
<wgrant> it's unnecessary.
<Chex> lifeless: any idea on the performance hit for the query?
<lifeless> Chex: by ok, can you paste the output please :)
<lifeless> Chex: its being hit by bots at least once an hour
<bigjools> count should be as quick as the query itself
<lifeless> bigjools: hell no
<bigjools> ?
<Chex> lifeless: you mean the bots are generating that query once an hour?
<lifeless> Chex: something very like it, but causing table scans.
<lifeless> bigjools: sec, let me get the query tested first.
<Chex> lifeless: understood, ok, will run on one of the Db slaves, chokecherry
<lifeless> Chex: thanks
<lifeless> bigjools: ok, so count(*) has to complete the entire thing, ignoring offsets and limits
<lifeless> bigjools: this is always more work except the special case when the limit of the first chunk matches the total work
<bigjools> lifeless: if the query has an order, that doesn't apply
<lifeless> bigjools: why do you say that?
<bigjools> it has to complete the query to order it!
<wgrant> Even with indices?
<lifeless> that depends on the query
<lifeless> very very much depends on the query
<bigjools> yes
 * bigjools goes back to buildd-manager hacking
<lifeless> anyhow, my point is just - don't assume count(*) is effectively free: its not. :)
<bigjools> lifeless: hell no
<bigjools> but if the original query is quick, and it bloody well should be, then the count should not matter in the bigger picture
<Chex> lifeless: bigjools: https://pastebin.canonical.com/34829/
<lifeless> I agree it would be lost in noise today
<lifeless> so, we may have stale statistics or something
<lifeless> Chex: thanks!
<bigjools> lifeless: "date_finished IS NOT NULL" kills that query
<lifeless> bigjools: is date_finished indexed ?
<bigjools> yes
<bigjools> but it does an index scan over 103k rows
<Chex> lifeless: your welcome
<bigjools> when you use NOT IN it has no choice
<bigjools> Chex: thanks from me too :)
<lifeless> bigjools: not in - yes, I get that. I don't see 'is not null' being == to 'not in', but I may be missing a specific technicality.
<bigjools> my tiny brain conflated the two, my bad
<lifeless> hehe no worries.
<benji> jml: import __future__
<lifeless> is not null can be badly affected by index statistics and index selectivity though
<benji> and then in your test setUp, add a global like...
<bigjools> hmmm status
<benji> test.globs['with_statement'] = __future__.with_statement
<lifeless> leonardr: so, I have a small separate idea for you.
<leonardr> lifeless, ok
<lifeless> leonardr: what if, when the result set is < pagination size
<lifeless> leonardr: lazr restful simply *does not* call len() on the result set
<jml> benji, sweet. thanks.
<benji> jml: I don't have a Python 2.5 so I can't test it, so hopefully it'll work as-is
<lifeless> leonardr: in this particular case we have 19 results
<leonardr> lifeless, makes sense
<bigjools> lifeless: we can improve it with an index btree(date_finished, status)
<jml> benji, well, I'll try that and resubmit via ec2
<bigjools> lifeless: it's scanning over status
<lifeless> leonardr: I know it won't help in the greater case
<wgrant> lifeless: Only 19? There should be more than that...
<jml> benji, in ~3hrs I'll know if it worked.
<benji> heh
<benji> it might be quicker for one of us to install 2.5
<lifeless> wgrant: privmsg
<lifeless> leonardr: how big a fix would that short hack I'm proposing be ?
<leonardr> lifeless: very easy
<lifeless> fix/workaround
<lifeless> leonardr: I propose, if you think its sensible, that we:
<lifeless>  - do this tiny hack; get that cowboyed to prod
<lifeless>  - do the larger one you proposed, if you think its tolerable
<lifeless>  - add the index bigjools is proposing, after evaluating it on staging
<wgrant> lifeless: The 'Failed to build' query that normally times out should return 2614 results.
<lifeless> wgrant: thats odd :P
<wgrant> That's larger than the batch size.
<lifeless> wgrant: yes, we have to deal with bigger things
<leonardr> lifeless et al: a quick analysis of the oopses shows that we seem to have three users
<leonardr> 1. leann ogasawara
<leonardr> 2. someone at ubuntuwire.org
<bigjools> lifeless: one thing I think we need to do is to document the queries we're using and the indexes that they need.  They're currently disjoint and we have no idea what needs what and what indexes are obsolete (which are a waste of processing time)
<leonardr> 3. someone at cranberry.canonical.com (wgrant?)
<bigjools> lifeless: a bit like the prejoins too
<wgrant> leonardr: I have access to no Canonical machines. I manage the script on ubuntuwire.org.
<leonardr> no, sorry, leann ogasawara is the person at cranberry.canonical.com
<leonardr> our third client is someone from optusnet.com.au
<wgrant> That's me.
<lifeless> bigjools: I think thats a great idea. Start doing it however, we can iterate to make it structured later.
<jml> wgrant, not internode?
<wgrant> jml: No. Too far for reasonable ADSL performance.
<bigjools> lifeless: one thing that really scares me is changing prejoins - we've simply got no idea what it will affect.
<wgrant> jml: And Optus' caps are about an order of magnitude larger now than they were 6 months ago.
<wgrant> So it's not that bad.
<lifeless> bigjools: we'll be a lot safer once staging's timeout limit is down to 5 seconds
<leonardr> wgrant: ok, so we have two users, you and leann
<lifeless> we can rev leann's launchpadlib pretty easily
<lifeless> let me go ask her
<jml> wgrant, fair enough :)
<jml> leann is being asked now in person
<leonardr> ah, i just asked her on irc, but ok
<lifeless> she was walking past the door to a meeting
<lifeless> she is running in the dc on the platform lp lib - hardies I suspect
<lifeless> but she can RT an upgrade anytime.
<leonardr> lifeless, have we run the queries that oopsed to see how many results they actually return?
<lifeless> there are 800 or so
<lifeless> I'd rather not do that by hand
<lifeless> wgrant says one in particular he does routinely returns 2.7K
<james_w> isn't backwards compatibility not really an issue as len() didn't work for a long time?
<lifeless> leonardr: when doing 'approve' on an MP please approve the overall proposal too - so that the queue is representative of the work reviewers have left to do
<leonardr> sorry
 * wgrant sleeps.
<wgrant> Thanks for looking at this.
<lifeless> leonardr: no probs
<lifeless> leonardr: its a small thing, but it helps the flow.
<leonardr> james_w: it's somewhat unlikely, but there was a published workaround for a long time, so i want to make sure
<lifeless> flacoste: is there a burndown chart of oops and timeout bugs ?
<lifeless> flacoste: or should I perhaps ask jml with his mad graphing skills to write one
<jml> I only do graphs with wobbly lines that go upwards and to the right.
<lifeless> don't worry, we can make this one do that
<lifeless> jml: how hard would it be for you to do this?
<lifeless> hmm meeting time
<jml> lifeless, about as hard as it would be for you.
<jml> maybe a little bit less if I used lpstats.
 * bigjools chuckles
<lifeless> jml: I think you underestimate startup cost/activation energy
<jml> lifeless, maybe.
<jml> lifeless, if you email me with exactly what you want, I can give it a go.
<jml> lifeless, but if you want it today or tomorrow, you're genuinely better off finding someone else.
<lifeless>  have mailed you
<mars> rockstar, ping
<rockstar> mars, distracted pong
<mars> rockstar, just wondering what the progress was on your YUI 3.1 upgrade.
<rockstar> mars, ah, very close.  Dealing with a bug where YUI 3.1 doesn't like the lp.client.
<mars> ok
<rockstar> mars, I'll send it to you for review.
<mars> I'm going to get someone to test 1.0, then we can look at rippling the change upward through the lazr-js tree
<mars> rockstar, sure
<rockstar> mars, 1.0?
<leonardr> lifeless et al: ogasawara is indeed accessing the total_size (using the workaround since len() doesn't work in old wadllib)
<leonardr> in fact, that's all she's doing with the data
<mars> rockstar, yes, there are three lines of development right now: trunk (dev), a 1.0 release branch that will become 1.0-dev, and the 2.0-dev line
<rockstar> mars, is there anyone else outside of Canonical using lazr-js?
<mars> rockstar, not to my knowledge
<rockstar> mars, I guess I'm asking "is there much point in the overhead required for coordinating various lines of development" ?
<rockstar> I think there should be trunk, and then the project using it can maintain their own branch of that.
<mars> yes, because we have projects on 1.0, and also projects on 2.0
<rockstar> mars, what projects are those?
<mars> ISD and LP are on 1.0, U1 and Landscape are on 2.0
<rockstar> mars, afaik, LP is on 0.9.2, which means very little, since we're not really on a branch at all.
<mars> rockstar, I'm working to get it down to two branches: 1.0 dev, and 2.0 dev
<mars> yes, the fact we are not on a branch is a problem as well
<rockstar> mars, what I'm saying is "the branch we're based off has little to do with what we're actually running"
<rockstar> mars, landscape, for instance, maintains their own branch.
<leonardr> ogasawara says that when it worked, her script found a total_size of 2614. given that neither of our users will benefit from the small-batch optimization, i think i should be doing something else (if we have a consensus about what else should be done)
<rockstar> If we land something on the 1.0 branch, we shouldn't be risking the breakage of a bunch of other projects.
<mars> rockstar, yes, and that leads to everything being one big hairball :)
<rockstar> mars, how's that?
<mars> because changes like the YUI 3.0/3.1 split or the distribute debacle force other projects to maintain branches
<rockstar> mars, if we make changes in trunk and then the projects then pull when they're ready, then we really only have one branch to worry about as lazr-js (trunk), and two branches to worry about as LP (trunk and whatever we're running)
<mars> 4 projects with 4 branches and mainline is 5 times the maintenance work needed
<rockstar> mars, the YUI 3.0/3.1 debacle wouldn't have happened if sidnei had landed in trunk.
<mars> and no one else would have been able to use or patch trunk
<mars> you either have everyone maintaining a private fork, or you consolidate
<mars> I want to get everything consolidated
<rockstar> mars, everyone needs to maintain a fork anyway.
<mars> rockstar, why?
<rockstar> Hopefully it's a "pull only" fork.
<rockstar> mars, because if we update 1.0, we can break other projects.
<rockstar> Allowing the other projects to pull in changes when they are ready is (IMHO) the best option.  We only have one line of development, and when they're ready to upgrade, they do.
<mars> that leads to real problems with versioning and contribution
<rockstar> mars, howso?
<mars> you have to write two patches (one for you, one for mainline), and you can't just pull trunk to get some new feature - there could be massive changes, meaning you have to backport and maintain yet another patch for your private fork
<mars> People should just be able to skip between releases
<mars> and releases should be documented in the changes they perform
<rockstar> mars, I think you ought to propose this to a mailing list somewhere, and find out what other projects are doing.  I have suspicions whether or not it needs to be this complicated.
<rockstar> mars, having two lines of development might mean that I need to patch my private fork, 1.0, and 2.0.
<mars> rockstar, I don't see the complexity - we'll have one mainline (2.0), and one legacy line (1.0)
<mars> rockstar, then you should drop your private fork, and fix mainline
<rockstar> mars, I think this is better for the mailing list.  I suspect that the private fork is a feature other projects are a bit attached to (Launchpad historically is)
<mars> rockstar, would you be willing to test the 1.0 branch to see if it builds on your system?  I would like to make lazr-js hackable again.
<rockstar> mars, do you need it right now, or can it be in 1 hour?
<mars> rockstar, better make that ~3 hours then, I'll be taking lunch around 12:30
<mars> err, "I'll be taking lunch in 1 hour"
<rockstar> mars, okay, that will work better, because I need to eat dinner soon.  I'm happy to test it though.
<mars> cool
<rockstar> bigjools, are you still around?
<bigjools> yep
<lifeless> leonardr: lol!
<lifeless> leonardr: so just exposing the size only thing would help her
<leonardr> lifeless: yes, if we implemented the full solution she would have to change her script but we could get it to work
<lifeless> \o/
<rockstar> bigjools, I'm having a pretty hard time changing the status of a SPRBuild...  The security proxy is only slightly the problem.
<rockstar> bigjools, do you have methods for flipping the switches?  All I see is handleStatus, and then things like "_handleStatus_OK" etc.
<bigjools> rockstar: what are you trying to do?
<leonardr> lifeless, i posted an update to the bug. i think we should shelve the 'don't run the count(*)' optimization since it's somewhat difficult and it won't solve the big problems. do you want me to work on the annotation-based solution?
<leonardr> (shelve for purposes of this problem, not permanently)
<rockstar> bigjools, make a SourcePackageRelease tied to a recipe
<rockstar> (it's a hole in our testing currently)
<bigjools> rockstar: in a test or in the code?  if the latter, at what point in the pipeline?
<rockstar> bigjools, in a test.
<bigjools> ok
<rockstar> bigjools, it won't let me set ISourcePackageRecipeBuild.source_package_release
<bigjools> ok let me check
<bigjools> rockstar: dude, ISourcePackageRecipeBuild.source_package_release is a property so I'm not surprised :)
<bigjools> rockstar: set SourcePackageRelease.source_package_recipe_build
<rockstar> bigjools, *facepalm* I was looking at the interface thinking it'd give me all I needed...
<rockstar> :)
<bigjools> :)
<bigjools> rockstar: if it makes you feel any better, I've done this exact same thing myself
<rockstar> bigjools, it's a sign that we should delete all interfaces.
<poolie> how do i do an api query for bugs containing a particular tag (and maybe other tags)?
<lifeless> leonardr: please
<lifeless> bigjools: BuildFarmJob.status <> 1 - thats an issue
<lifeless> losa ping
<mthaddon> lifeless: hi
<lifeless> hi, we'd like to run an analyze on packagebuild after checking how big the table is
<lifeless> on each db server
<lifeless> and then check explain analyze SELECT BinaryPackageBuild.distro_arch_series, BinaryPackageBuild.id, BinaryPackageBuild.package_build, BinaryPackageBuild.source_package_release FROM Archive, BinaryPackageBuild, BuildFarmJob, PackageBuild WHERE distro_arch_series IN (109, 110, 111, 112, 113, 114) AND BinaryPackageBuild.package_build = PackageBuild.id AND PackageBuild.build_farm_job = BuildFarmJob.id AND (BuildFarmJob.status <> 1
<lifeless> again
<lifeless> if the table is -huge- we obviously don't want to wedge things.
<mthaddon> lifeless: erm, why do we need to do this?
<lifeless> also I'd like to know the range of values in BuildFarmJob.status - select status from buildfarmjob unique;
<lifeless> mthaddon: because we have an API call timing out - taking 18 seconds - and the query plan suggests a mismatch between statistics and actual data.
<lifeless> http://paste.ubuntu.com/465800/
<lifeless> we've identified a few issues all at once related to this:
<lifeless>  - the api is doubling the db load by one of its bits of magic
<lifeless>  - the query is extremely slow itself
<mthaddon> lifeless: I'd like to get stub's input on that (particularly why it's out of whack in the first place) ideally
<lifeless>  - the query contains an exclude - status <>1  rather than status in (2,3,4,5,6) or whatever it should be
<lifeless> mthaddon: hes on a plane.
<lifeless> mthaddon: I tried to ring him just before :(
<mthaddon> lifeless: sure - is this a critical issue now?
<lifeless> its timing out for ogsawara and wgrant every time; they aren't able to retry to fix it.
<lifeless> if its simply stale statistics, that would be a very easy bandaid.
<lifeless> we can also raise the timeout again
<lifeless> by which I mean, 'retries also time out'
<mthaddon> I think increasing the timeout is a better short term fix - what should we increase it to?
<mthaddon> lifeless: and it times out for them on both edge and lpnet?
<lifeless> yes
<lifeless> thats my [weak] understanding
<lifeless> edge is doing a timeout every 120 seconds
<lifeless> prod is a lot more unhappy, but that is primarily the bug attachment oops which gmb has been working on.
<lifeless> which reminds me
<mthaddon> lifeless: so which one are we changing? edge or lpnet? and to what?
<lifeless> mthaddon: do you know how to generate a manual oops report for the oops since this morning on edge and lpnet?
<lifeless> mthaddon: that would help me answer your question
<lifeless> because I know what fixes are in-progress
<mthaddon> lifeless: I don't, no :(
<lifeless> mthaddon: then I'd say lets raise it back to 14 seconds on edge
<lifeless> I know that most of the prod ones are the bug attachment script
<lifeless> and it is in progress
<mthaddon> lifeless: like this? https://pastebin.canonical.com/34844/
<lifeless> mthaddon: could we run an analyze on staging at least, see how long it takes, and if it improves the query ?
<lifeless> mthaddon: yes, that patch will raise the edge timeout.
<mthaddon> lifeless: it's hard to say if that will match production since the load on the DBs is so different though (having never been asked to do this before for LP is throwing up a minor red flag as "doing it wrong" as well)
<lifeless> mthaddon: I'm positive stub has done analyze's to fix statistics many times. A greb of the lp-code logs will probably find some ;)
<mthaddon> in any case, I'm pushing out the cowboy to edge with the higher timeout now
<lifeless> thanks
<lifeless> mthaddon: I'm curious how, since its the same revno ...
<mthaddon> lifeless: I landed the branch that allows me to specify a revno
<Ursinha> lifeless, lpnet oopses since 00utc: https://devpad.canonical.com/~lpqateam/lpnet-oops.html#time-outs
<lifeless> mthaddon: \o/
<lifeless> mthaddon: thats awesome
<mthaddon> s/specify a revno/specify a custom directory name/
<lifeless> Ursinha: thanks
<Ursinha> lifeless, same for edge and staging: https://devpad.canonical.com/~lpqateam/edge-oops.html#time-outs https://devpad.canonical.com/~lpqateam/staging-oops.html#time-outs
<nigelb> bryceh: where is the code for it?
<nigelb> (I wish bluprints had a comments area too for each action item)
<bryceh> nigelb, hang on I'm composing an email
<nigelb> heh :D
<bryceh> nigelb, damn you're quick ;-)
<nigelb> haha
<mthaddon> lifeless: timeout increased on edge
<mthaddon> lifeless: although the config change hasn't been landed, so it'll be overwritten on next rollout unless that happens
<lifeless> mthaddon: ok, can you do that too - or should I just land an r=mthaddon to increase it ?
<lifeless> Ursinha: thanks
<mthaddon> lifeless: r=mthaddon would be great, thx
<mthaddon> lifeless: has it fixed the issue?
<lifeless> mthaddon: don't know
<lifeless> mthaddon: wgrant may have gone to sleep
<lifeless> mthaddon: yes its fixed
<mthaddon> cool
<lifeless> at least for leanne
<lifeless> but I think they're looking at the same think
<lifeless> *thing*
<lifeless> sinzui: bug 607879 - if you want to discuss with me, gimme a shout
<_mup_> Bug #607879: https://bugs.edge.launchpad.net/~person/+participation timeouts <oops> <timeout> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/607879>
<lifeless> losa ping
<bryceh> nigelb, ok finally got that email out
<Chex> lifeless: hi there
<lifeless> Chex: hi, uhm channel confusion - query plan tweaking on staging
<Chex> lifeless: ok, run that query on staging DB, then?
<lifeless> Chex: so an analyze of packagebuild on staging
<lifeless> and then
<lifeless> # explain analyze SELECT BinaryPackageBuild.distro_arch_series, BinaryPackageBuild.id, BinaryPackageBuild.package_build, BinaryPackageBuild.source_package_release FROM Archive, BinaryPackageBuild, BuildFarmJob, PackageBuild WHERE distro_arch_series IN (109, 110, 111, 112, 113, 114) AND BinaryPackageBuild.package_build = PackageBuild.id AND PackageBuild.build_farm_job = BuildFarmJob.id AND (BuildFarmJob.status <> 1 OR BuildFarm
<lifeless> on staging
<danilos> lifeless, you don't mind me doing the query I suggested two times on production slave? (i.e. you still have reasons to believe it would hurt us)
<danilos> lifeless, fwiw, your query above was cut-off
<Chex> lifeless: ERROR:  column "buildfarm" does not exist
<Chex> LINE 5: ... BuildFarmJob.id AND (BuildFarmJob.status <> 1 OR BuildFarm)...
<lifeless> Chex: http://paste.ubuntu.com/465800/
<lifeless> Chex: top line
<Chex> lifeless: oops, yeah pastebin is better, thanks
<Chex> lifeless: http://pastebin.ubuntu.com/466583/
<lifeless> Chex: and you analyzed packagebuild first ?
<Chex> lifeless: sorry, no I did not
<danilos> lifeless, fwiw, I wasn't thinking of doing analyze on production DB :)
<lifeless> Chex: please do :) - there is a mismatch between rows=702 and rows=28253 in the middle of the explain
<lifeless> that jtv pointed out
<jtv> danilos: this is "analyze," not "explain analyze"
<danilos> jtv, right, lifeless stopped me from doing explain analyze on production slave because it would "mess up caches on production DBs"
<lifeless> danilos: well no, you were saying something that I interpreted to mean 'drop caches'
<lifeless> danilos: which is rather different from 'run twice to eliminate cold cache effects'
<danilos> lifeless, well, I was saying exactly this: "losa quick ping: hi, can you please check how caches on DB server affect executing a query at https://bugs.edge.launchpad.net/soyuz/+bug/590708/comments/8 (i.e. do it a few times on the same production slave DB)"; I'd never interpret it the way you did, but that's not up for debate :)
<_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out <api> <oops> <soyuz-build> <timeout> <Soyuz:Triaged by michael.nelson> <https://launchpad.net/bugs/590708>
<lifeless> danilos: crossed wires happen :)
<danilos> lifeless, yeah, you were doing something very similar here so I guess that's where the confusion comes from ;)
<danilos> anyway, Chex, can you please try the query above twice on a single production slave to compare the results?
<Chex> lifeless: http://pastebin.ubuntu.com/466585/
<Chex> lifeless: this look any better?
<jtv> Nope
<lifeless> well, 2 seconds better
<lifeless> Chex: so, to check - you did 'analyze packagebuild' then the query from the pastebin I linked earlier ?
<lifeless> jtv: Nested Loop  (cost=0.00..11792.41 rows=2783 width=8) (actual time=0.068..2155.895 rows=904092 loops=1)
<lifeless> jtv: thats another row expectation mismatch ?
<Chex> lifeless: that is correct
<Chex> the analyze packagebuild then the query you pasted.
<lifeless> cool
<Chex> danilos: ok, sure, hang on
<lifeless> can you please do analyze archive and analyze binarypackagebuild too
<Chex> then analyze packagebuild, then the query?
<lifeless> analyze archive; analyze binarypackagebuild; the query
<jtv> lifeless: that looks like a mismatch, yes...  but that looks like a definite but
<jtv> bug
<Chex> lifeless: ok.
<jtv> I mean, why a million rows there?
<Chex> lifeless: http://pastebin.ubuntu.com/466592/
<lifeless> oh!
<jtv> Ah, it's a highly unfortunate thing... those million rows are the Archive Ã PackageBuild records
<lifeless> Chex: lets do this differently.
<lifeless> Chex: analyze packagebuild; analyze archive; analyze binarypackagebuild; - outside of a transaction
<lifeless> Chex: we don't want to rollback, we want the analyzes committed
<lifeless> I'm not 100% sure about the impact of analyze + rollback :-
<Chex> lifeless: oh, fair enough, hang on
<Chex> lifeless: done
<Chex> lifeless: now try your query?
<lifeless> now, the explain query please :)
<Chex> lifeless: http://pastebin.ubuntu.com/466597/
<lifeless> ok, well that fairly definitely answers that
<lifeless> thanks
<lifeless> danilos: I've finished monopolising Chex for now :)
<lifeless> jtv: I agree that that 900K loop finding 0 rows is an issue
<jtv> It was only expecting 2.0358 iterations there I guess.
<bryceh> nigelb, had a chance to try it out?  thoughts so far?
<jtv> Sorry, 20,358
<lifeless> so there are lots of bpb records
<lifeless> and pb records
<lifeless> I guess
<lifeless> please tell me we haven't split out a common table to join that has 1:1 mapping to the table we filter on ?
<Chex> danilos: now _your_ query..
<danilos> Chex, mine should be quick ;)
<lifeless> bigjools: are you still around ?
<Chex> danilos: http://pastebin.ubuntu.com/466604/
<Chex> danilos: note the much quicker run the 2nd time
<danilos> Chex, excellent, just what I suspected :)
<danilos> lifeless, jtv: the times with the above query seem much better this time, see http://pastebin.ubuntu.com/466604/ :)
<jtv> danilos: why do you join PackageBuild twice?
<jtv> I mean, not that I'm arguing against the speedup...  :-)
<danilos> jtv, because I am smart :)
<jtv> that can't be it
<danilos> jtv, that's the same trick we use in translations: note how packagebuild-archive join takes the most time in the original query
<danilos> jtv, because it joins entire packagebuild with archive (across all rows); this forces postgres to avoid that so it's much faster :)
<jtv> It almost looks as if the original query intended something like this... two of the join conditions occurred double, just like with yours.
<lifeless> its joining to workaround the split out
<lifeless> I think the split out should not have been done at the table level: N separate tables with a common columnprefix
<lifeless> databases tables are not classes :)
<danilos> lifeless, I think both of these are much faster simply because the caches are already warm (because of your test :)
<danilos> anyway, now I go away and will be able to sleep at night :)
<danilos> cheers
 * danilos goes
<Ursinha> sinzui, hello
<Ursinha> sinzui, we had a bunch of oopses like https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1661XMLP111, MailingListAPIView
<Ursinha> sinzui, I see there's a bug for this oops, bug 531371, which is already fix released
<_mup_> Bug #531371: oops MailingListAPIView email already in use <mailing-lists> <oops> <Launchpad Registry:Fix Released by sinzui> <https://launchpad.net/bugs/531371>
<lifeless> daniloff: gnight. I think your query is very clever; It would be good for storm to do that for us
<sinzui> Urshina see my email about how I hate Launchpad developers who offer crap services for free
<Ursinha> :)
<Ursinha> I will
<sinzui> Ursinha, I am sick and cannot help, the LOSAs corrupted the DB because they thought they were being nice to monte
<Ursinha> oh, argh
<lifeless> sinzui: get well
<lifeless> I forgot you were ill
<Ursinha> sinzui, please, get well
<sinzui> Ursinha, There is a question tracking how to fix the data. Someone needs to kill the private project that should never have been mondified
<Ursinha> oh, that one with the pending mailing list? I see.
<Ursinha> thanks sinzui, sorry to ping
<lifeless> Ursinha: so, the grouping
<lifeless> Ursinha: does it just use the exception type, or the exception type + the string ?
<Ursinha> lifeless, exception type + value
<Ursinha> lifeless, bug 461269
<_mup_> Bug #461269: oops reports should be grouped by oops signature not exception type and exception value <OOPS Tools:Triaged> <https://launchpad.net/bugs/461269>
<Ursinha> lifeless, I was about to leave to have some food
<lifeless> ciao
<lifeless> I'm just opportunistic on asking stuff
<lifeless> no need to hang around for me
<Ursinha> lifeless, okay :) anything else, just ask and I'll answer when I return
<lifeless> kk
<flacoste> lifeless: your lp:~lifeless/launchpad/soyuz mp diff is screwed up
<flacoste> and lifeless, i had a test failures back from by ec2 land (feedparser branch)
<flacoste> this is the fix I applied: http://pastebin.ubuntu.com/466624/
<flacoste> do you have a better suggestion?
<lifeless> flacoste: thats fine with me, its not hugely beautiful, but its not ugly.
<flacoste> yeah, my feeling also, wondered if there was a better known idiom
<lifeless> its essentially mocking; we could use an official mock, but it wouldn't be any smaller.
<flacoste> right
<flacoste> what library would you recommend for mocking (unrelated to this branch, asking for another project)
<flacoste> do you use something in bzr?
<lifeless> we don't routinely mock
<lifeless> mocking has some risks
<lifeless> and some rewards
<lifeless> uhm
<lifeless> for your line there, even with a mocking library, I'd probably just do the lambda :)
<flacoste> ok
<lifeless> from the school of 'simplest is often clearest'
<lifeless> flacoste: speaking of reviews
<lifeless> I got the queue down to 0
<lifeless> for devel anyhow
<lifeless> ah the soyuz brnach is messed up because db-devel exists
<benji> flacoste: I've enjoyed using Gustavo Niemeyer's Mocker on another project (http://labix.org/mocker)
<benji> many other options at http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#MockTestingTools
<lifeless> I much prefer verified fakes to mocks
<lifeless> less skew
<lifeless> but this is not a late-at-night discussion I think; its been ... intense today
<flacoste> thanks benji
<benji> lifeless: I suspect Mocker can do what you want; it's quite full-featured.
<lifeless> benji: the point is to not mock.
<lifeless> benji: so no, it can't :)
<benji> what do you mean by "verified fakes"?
<lifeless> just that
<lifeless> a fake (not a mock or stub) that is verified to behave the same
<lifeless> as a full implementation
<lifeless> e.g. sqlite in-memory db's are a pretty good verified fake for disk databases.
<benji> ok, Martin Fowler's definition of "fake"
<lifeless> yes, I find the definition to be usefully precise
<mars> rockstar, ping
<lifeless> hmm
<lifeless> more count(*) taking ages
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1662EA433
<deryck> ok, I need to break for awhile.  Until later on then......
<lifeless> flacoste: may need to think about making oops' critical rather than high... teams have lots of high already :)
<flacoste> lifeless: that was the idea of zerooopspolicy
<lifeless> flacoste: I thought it said high, not critical
<lifeless> yes, it says high on the wiki
<flacoste> hmm, ok
<lifeless> flacoste: If the goal is 'in front of the queue', critical would seem appropriate to me.
<lifeless> flacoste: but I wasn't part of the discussion for the policy, I don't want to just jump in here
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1662EA447 is fun
<lifeless> anyone have ideas on what to do with *that* ?
<lifeless> I guess its getting the wadl ?
<benji> hmm; I did a rocketfuel-get and now my tests fail
<benji> is the devel branch broken?
<lifeless> shouldn't be
<lifeless> Ursinha-nom: when you return, if you could regenerate the edge oops-since-utc0 page, I think I've got bugs made for most of them now
<lifeless> rinze: please set the MP status when reviewing as well
<rockstar> mars, sorry, hi.
<lifeless> I'm -> sleep
<lifeless> gary_poster: so you know, edge is back up to 14 seconds, 12 seconds is past the knee and unsafe
<gary_poster> heh, ok, thanks for update
<lifeless> https://lpstats.canonical.com/graphs/OopsEdgeHourly/ shows it quite graphically
<lifeless> prod is still unhappy - https://lpstats.canonical.com/graphs/OopsLpnetHourly/ -  but the pending fixes should make a dramatic difference to that
<lifeless> and your pqm hack is on my kanban todo, but I've been bouncing from thing to thing all day.
<lifeless> on the bright side I seem to have gotten past the stomach ache part of this lurgy, so I can actually concentrate again.
<lifeless> and with that, I bid you all asnore.
<gary_poster> thank you and good night
<wgrant> Can someone please ec2 land https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667? danilos tried to do it last night, but apparently ended up starting two instances for the *other* branch.
<nigelb> bryceh: trying out now.  Do I get to confirm on the upstream tracker before it gets submitted?
<bryceh> nigelb, yes
<bryceh> nigelb, btw do you find that aspect important?  I've considered eliminating that as an extraneous step if it isn't considered important
<nigelb> Since I'm testing now, I'd find it important.  But when I'm using the tool, I'd find it extraneous
<nigelb> I keep getting, "Sory produce xorg in ubuntu does not exist or you're not allowed to report a bug in it" :/
<bryceh> yeah try another package.  'xorg' isn't supported, but e.g. 'xserver-xorg-video-intel' is
<bryceh> (there isn't actually an 'xorg' package upstream, it's a non-source debian package only)
<nigelb> ahh
<nigelb> bryceh: same error with xserver-xory-video-intel
<bryceh> hrm
<bryceh> nigelb, ok try now
<bryceh> weird, I was sure I'd fixed that already
<nigelb> bryceh: wow, just WOW
#launchpad-dev 2010-07-21
<bryceh> :-)
<nigelb> Its beautiful!
<nigelb> Also, there is this space "&commit= Send Upstream"
<nigelb> it should not exist I think, so the commit button would change to Send upstream?
<nigelb> ok, tested.  No.
<nigelb> Anyway its the most awesome thing I've seen.  If LP does this by default, we'd all hug you :)
<bryceh> I think that's fine, that value isn't actually used by bugzilla
<nigelb> Yeah, I just noticed
<bryceh> well, it may be a while before LP does this by default, but I'll be working on it.  meanwhile the cgi is going to be available for folks
<bryceh> I'll be interested in seeing how much use it gets
<bryceh> I'd also be interested in hearing about tangible time savings.  I calculated after spending one day forwarding bugs without this tool, vs. one day forwarding bugs with it, I was able to send 3 times as many bugs upstream in a given period of time
<bryceh> the main constraint for me in achieving an even better ratio was having to manually copy over the attachments
<bryceh> nigelb, thanks for finding that bug.  embarrassing!
<bryceh> clearly I need to write some tests
<nigelb> heh
<nigelb> another advantage when you have a tool like this, people don't go, "Oh, I'll forward it later"
<bryceh> nigelb, did you install the greasemonkey script?
<nigelb> they'll just.  "hey, it'll take only 10 minutes, why not do it"
<bryceh> yep
<nigelb> Nope, I did it manually.
<nigelb> (this is work computer, can't mess with it :D)
<bryceh> mm, if/when you can install that gm script, you'll find it REALLY makes it fun and easy to upstream bugs
<nigelb> Oh!
<nigelb> how tough would it be to make it available for gnome bugzilla too?
<bryceh> it should work with gnome bugzilla already
<bryceh> I only tested on a couple bugs though so am not 100% sure
<nigelb> gnome has more people forwarding since the apps are mostly in user space
<bryceh> but I put the support for GNOME in last week.  I'd love to get verification that it works
<bryceh> yep
<bryceh> it should work for gnome packages that have exactly the same name in launchpad and in gnome bugzilla
<bryceh> which afaik should cover most packages
<nigelb> where in the lp branch is the code?
<bryceh> it's in the scripts directory
<nigelb> its a very nifty script :)
<bryceh> thanks, yeah I've gotten a lot of good out of it, and looking forward to it being handy for others
<bryceh> nigelb, let me know if you test it on a gnome bug
<nigelb> I will as soon as I can find the time.  gotta start working :)
<bryceh> nigelb, cheers :-)
<Ursinha> lifeless, missed your request -- "partial" oops summaries, as we call, are regenerated every hour
<lifeless> Ursinha-afk: thanks
<lifeless> spm: can you time the statement in https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1662O772 ? - the 17 second one, on a slave, and on prod, please?
<spm> sure
<lifeless> if its slow, grab an explain analyze too :)
<spm> wow. that's... inconsistent.
<spm> enter query into cmd line; hit enter; runs, completes. uparrow+enter; repeat.
<lifeless> .
<lifeless> .
<lifeless> .
<spm> 1.7s, 13.1s, 0.6s, 1.0
<lifeless> we might want to fix this
<spm> wtf (where) did  13.1 come from!!!
<lifeless> was that on a slave or main ?
<spm> main
<spm> what makes that 13.1s even more interesting, i'd have thought the query was still within postgres caching. ie it should be zomg fast.
<lifeless> so
<lifeless> guesses: load, index fail, index lock contention
<spm> cache fail...
<lifeless> if we're under that much pressure, sure.
<lifeless> we're seeing the oops 180 times a week
<spm> load, unlikely - load average: 8.31, 7.91, 8.14 <== which is pretty low for wildc
 * lifeless mutters about perspective
<spm> note the 'unlikely' qualifier. :-)
<lifeless> how many cores does it have?
<spm> when you'd had dual 650Mhz CPU proxy firewalls on a 'normal' load of 50-80; still working at 450...
<spm> 16
<lifeless> ok, so thats still 8 db threads or similar not executing for a full second
<lifeless> its not actually great
<lifeless> later we will have to go into the performance knee for wildc
<lifeless> right now, we already know that backsup cause an oops spike :P
<spm> ? I thought load was a (rough) measure at the end of a 'tick' of what's executing right now; regardless of how much of the tick they've had? imbw....
<spm> ie it avergaes out to a vague approximation.
<lifeless> its blocked ready-to-run processes
<lifeless> http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages - not telling you to suck eggs, but I've quickly scanned it and its a clear explanation
<spm> heh, I suspect we're using different words to describe the same thing. :-)
<spm> ta
<lifeless> ok anyhow
<lifeless> one thread at a time
<spm> :-)
<lifeless> can you see if it does that slow pattern on a slave
<spm> huh, I was almost going to say o it doesn't, but persistence paid off. most (~ 10 runs ish) were sub 0.3s, 1 x 1.0, 1x 4.5.
<spm> s/ o / no /
<lifeless> ok
<spm> 8 cores, load average: 1.01, 1.18, 1.15
<lifeless> can you get an explain analyse of it slow in both cases ?
<spm> Oooer. it changed!
<spm> wow. I didn't notice. it changed twice. https://pastebin.canonical.com/34873/
<spm> actually... i think it was different on every run. repasting the entire test
<spm> https://pastebin.canonical.com/34874/
<lifeless> *boom*
<spm> yeah. something is *really* screwy, based on my DB understandings if the explain plan can be changed so dramatically like that. I thought these things were cached for high seconds to low minutes for precisely this sort of thing.
<spm> https://pastebin.canonical.com/34875/ to complete the set, wildcherry showed similar behaviour
<lifeless> ah
<spm> actually I quite distinctly do recall faffing around with Oracle... 10g istr and forcibly getting it to regen plans 'offline' as it were to maximise the speed of certain queries
<lifeless> its a damned view
<lifeless> omfg
<lifeless> excuse me
<spm> no perfectly understandable :-)
<lifeless> \d publishedpackage :P
<spm> It is pretty unthinkable that'd I've used oracle and actually like it. ;-)
 * spm takes a few deep breaths
<wgrant> Oh god, publishedpackage?
<wgrant> Kill it.
<spm> oh not too bad. only 12 joins.
<wgrant> The *cache table* is being slow?
<lifeless> in the fast one is its a nested loop
<lifeless> pg doesn't hve materialized views does it?
<spm> unknown to me
<lifeless> don't think so
<wgrant> You can emulate them using triggers, but it doesn't do them itself, no
<lifeless> so the fast case loops on binarypackagename_name_key on binarypackagename
<lifeless> the slow case Seq Scan on buildfarmjob  (cost=0.00..54641.45 rows=1799245 width=12) (actual time=0.019..1470.653 rows=1799572 loops=1)
<lifeless> and it gets worse from there
<wgrant> That table has only been around for a couple of months.
<wgrant> So the query probably hasn't been looked at in this form before.
<lifeless> I need to review this, I smell OOD rather than table layout optimisation
<lifeless> I saw a similar thing last night
<lifeless> spm: so I'm guessing slight statistics variation -> boom
<spm> lifeless: I guess, but... I'm kinda left wondering why plans aren't locked in stone more; if they even can be.
<lifeless> spm: do I have sql access to staging, to poke at the tables a bit ?
<spm> perhaps comparing oranges with mandarin's; in oracle land - as I recall it; you'd have a generic query plan with placeholders for the values. there's a specific name, but time and memory fades. the idea being that you just replace the bits you care about; the plan itself is indentical; so you win big on lots of repititions by not doing the analyze phase(s)
<spm> lifeless: perhaps r/o? if not, I can create now and retroactively ask francis for you to have same.
<lifeless> spm: whatever bjorn had?
<spm> I have 99.99999% no doubt he'll be fine with that.
<spm> everyone except losas and stub has r/o
<lifeless> I'm sure that will be fine
<spm> rephrase, everyone *with access* has r/o
<lifeless> if its ever not, well, efuture
<lifeless> wgrant: so why do you say to delete hte view?
<wgrant> lifeless: I was wrong. I was thinking it was one of the cache tables.
<wgrant> But it's not; it's just a view.
<lifeless> 10 million rows in bpph
<lifeless> I can has delete?
<wgrant> You can has indices.
<lifeless> meh
<lifeless> I has indices.
<lifeless> I wants delete.
<lifeless> also fti needs some serious love
<wgrant> Deleting history makes me sad.
<wgrant> What's the FTI on?
<lifeless> wgrant: keeping stuff isn't free
<lifeless> and even logn perf hurts after a while
<wgrant> lifeless: Says he who developed a DVCS which preserves all history :P
<lifeless> its all about use cases
<lifeless> actually, I want a good history edit mechanism - I wrote that up a year back or so
 * wgrant still needs someone to land a branch.
<lifeless> spm: are you able to do iostat/cpu on staging for a query for me?
<lifeless> wgrant: whats the mp
<wgrant> lifeless: One of the ones you reviewed a couple of days ago.
 * wgrant hunts.
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667
<lifeless> oh, I might need a newer tree
<lifeless> we'll see
<mwhudson> herh
<mwhudson> lp.registry.windmill.tests.test_person_picker.TesPersonPickerWidget.test_person_picker_widget is failing in my branch
<mwhudson> nice test class name!
<lifeless> wgrant: its mid kickoff
<wgrant> lifeless: Thanks.
<lifeless> losa ping
<mwhudson> how do you debug a failing windmill test?
<lifeless> WITH A SHOTGUN
<lifeless> wgrant: its in flight
<wgrant> lifeless: Even better.
<wgrant> It *is* the right branch?
<wgrant> Last night it wasn't.
<lifeless> i just cp'd your link
<mwhudson> lifeless: it seems "run it until it passes" works :(
<wgrant> mwhudson: My policy is "run until it passes. if it still doesn't pass, ignore it, because I didn't break it"
<spm> lifeless: sure, on staging DB server as in?
<lifeless> yes
<lifeless> the qyry is
<lifeless> \timing
<lifeless> SELECT COUNT(CASE WHEN Bug.fti @@ ftq('sphinx') THEN TRUE ELSE null END) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1;
<spm> lifeless: I forgot to time() it; but ~ 10ish seconds. https://pastebin.canonical.com/34879/ for the vmstat
<lifeless> \timing is your friend :)
<spm> aye... :-/
<spm> 2nd run - just shy of 9 secs
<lifeless> ok
<lifeless> a fucktonne of IO
<lifeless> spm: can you check whether lucene, solr & pylucene are available in the DC, in the event we were to want to use them.
<spm> are they debug type programs? servers?
<spm> ah the first is the search engine?
<lifeless> yes
<lifeless> one stack
<spm> lifeless: looks like, not really. some possibly defaultish libraries; but I'd suggest backporting would be needed.
<lifeless> rmadison was hopefyul
<mwhudson> i know the divmod folks became extremely angry with lucene
<mwhudson> several years ago now, mind
<lifeless> spm: it appears to be in universe, to me
<spm> lifeless: that would be my lack of experience with dealing with packages then
<lifeless> hahahha
<spm> or possibly we don't enable universe on the servers...
<lifeless> spm: whats our policy on stuff in univers
<lifeless> e
<spm> actually I can see the solr stuff, I sit corrected.
<spm> heh, lalala me. universe is fine.
<lifeless> spm: how hard am I hammering the server ?
<spm> 100% on one cpu only
<lifeless> \o/
<lifeless> select * from stat('select fti from Bug') order by ndoc desc, nentry desc,word limit 10;
<lifeless> how many cpus are there ?
<spm> 8 cores
<spm> only 32g of ram, so i'd expect more io than prod
<spm> whee that local run is evil you've made me do. also 100%
<spm> 0.0% wait state tho
<lifeless> spm: I'd like to try reindexing bug.fti on staging.
<lifeless> spm: how do we go about this?
<spm> depends on how intrusive a reindex you mean? if you just mean 'reindex table blah', that's pretty easy.
<lifeless> spm: or should I do that on dogfood
<lifeless> spm: we need to drop the bug_fti index and make a new gin one
<lifeless> something like
<spm> I'd suggest dogfood in the first instance to get an idea of timing
<mwhudson> lifeless: when did test_suite() stop being necessary in launchpad tests?
<mwhudson> (also yay!)
<lifeless> mwhudson: I don't know, but jtv tested and current zope.testing doesn't seem to need it.
<spm> lifeless: it'd also be my personal recommendation to run any reindexing past stub
<lifeless> spm: how do I get access to dogfood ?
<mwhudson> maybe zope.testing grew a new feature then
<spm> lifeless: if you don't have already? via RT
<lifeless> spm: I've spoken to stub about the fti stuff; its  basically untuned - he expressed frustration at not having had time to just sit down and zen it
<spm> heh
<lifeless> mwhudson: hell, it may be needed still - remove it and check :)
<mwhudson> lifeless: i did
 * spm can picture stub doing the OOOOOmmmmm thing before a DB layout diagram
<wgrant> spm: dogfood is *completely* useless for timing.
<wgrant> You know what sort of hardware it's on, right?
<lifeless> so the thing is
<lifeless> we have a fast-insert slow-query fti index
<spm> wgrant: I'm think of more an 'minutes vs hours vs days' idea
<lifeless> there is an alternative index
<spm> not 17.65 minutes.
<lifeless> the gin index
<wgrant> spm: dogfood cannot always give a reasonable answer over those three.
<lifeless> anyhow this isn't urgent
<spm> haha
<lifeless> I'm just starting to qualify simple-complex answers in this space
<wgrant> Recall that a DB restore took two weeks a few months ago.
<wgrant> And the primary publisher takes several hours.
<mwhudson> lifeless: this talk of gin indexes rings faint bells, i think we already use them for something...
<mwhudson> or tried
<mwhudson> still if you talked to stub you're likely much more up to date than me :)
<lifeless> mwhudson: yes :)
<wgrant> lifeless: Are Forbbiddens logged?
<wgrant>  /builders is 403ing sometimes.
<lifeless> should be
<lifeless> I don't know if they raise oopses
<mwhudson> they aren't logged as oopses when you're logged in, i think
<wgrant> mwhudson: So we can't debug them?
<wgrant> Awesome.
<lifeless> wgrant: thats not entirely true
<mwhudson> wgrant: i may have been lying there
<poolie> lifeless, jam and i looked at the memcached graphs last night
<mwhudson> wgrant: i think bigjools knows the problem though
<poolie> as i suppoe you already have before
<mwhudson> wgrant: it's private builds, unsurprisingly
<poolie> they seem clearly size-limited
<wgrant> mwhudson: He suspects private teams.
<lifeless> wgrant: however, I encourage you, in the theme of improvign things, to add trace logs for this
<poolie> and hitting only about 20% of the time
<lifeless> yeah
<wgrant> mwhudson: But I can't see any private builds, so I shouldn't see any private teams.
<lifeless> I don't really have memcache on my radar atm
<poolie> so caching more things is likely to make the cache perform worse :)
<lifeless> its a tool we're not ready for
<wgrant> It's already overused.
<wgrant> Caching things that should not be cached.
<poolie> i know
<lifeless> lpmain_staging=> SELECT COUNT(CASE WHEN Bug.fti @@ ftq('sphinx') THEN TRUE ELSE null END) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1;
<lifeless>  count
<poolie> i was thinking of letting it be refreshed from the browser by cache-control
<lifeless> -------
<lifeless>     40
<lifeless> (1 row)
<lifeless> Time: 7659.528 ms
<lifeless> lpmain_staging=> SELECT COUNT(*) from Bug, bugtask  where bug.fti @@ ftq('sphinx') and BugTask.bug = Bug.id AND BugTask.distribution = 1;
<lifeless>  count
<poolie> i now feel we should add a switch to jut turn it off, and see if the oops rate changes
<lifeless> -------
<lifeless>     40
<lifeless> (1 row)
<lifeless> Time: 526.478 ms
<lifeless> we can just turn it off
<lifeless> and no, not pastbinning that :P
<lifeless> *thats* easy
<wgrant> poolie: There is lots of stuff that should be cached.
<wgrant> poolie: But it's not the stuff that's cached at the moment.
<mwhudson> lifeless: those times are, um, different
<wgrant> lifeless: What's the difference in the plan?
<poolie> lifeless, so tbf there are a lot of cache effects on the db server
<lifeless> wgrant: give you a single guess
<poolie> running two queries one after the other and seeing the second is faster does not necessarily prove anything
<poolie> (in my limited experience)
<poolie> i agree the second looks more sensible
<lifeless> of course
<lifeless> I've run 10 of each
<spm> running 4 and having the 4th run an order of magnitude slower tho....
<lifeless> but thats a lot more noisy ;)
 * mwhudson gets some 503s and 504s from the apache frontend :(
<poolie> ah :) and it's consistent? fairy nuf
<lifeless> sometimes the slow one is worse
<spm> https://pastebin.canonical.com/34874/ <== for mindblowing enjoyment
<lifeless> ok, so edge deployment just failed to migrate smoothly ?
<lifeless> mwhudson: please file a bug
<mwhudson> it's working now
<spm> edge was just updating
<lifeless> yes
<mwhudson> ah
<lifeless> this is a bug
<lifeless> we should fix it, we're going to be doing this *much* more often
<lifeless> mwhudson: please file it ;)
<mwhudson> lifeless: ok
 * mwhudson waits to see if the bug search times out...
<spm> short of automatically adjusting the 'load balancing' away from each server and waiting for all existing connections to same to timeout, there will be glitches for some folks?
<lifeless> spm: thats exactly what we should be doing
<wgrant> Hm. Is there no way to see all of a project's branches?
<lifeless> code.lp.net/project
<wgrant> No.
<wgrant> It's batched.
<wgrant> But without a navigator.
<wgrant> So it only shows the first n.
<mwhudson> lifeless: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/608065
<_mup_> Bug #608065: get occasional 503s and 504s from apache frontend when edge is updating <rollout-reliability> <Launchpad Foundations:New> <https://launchpad.net/bugs/608065>
<lifeless> _huh_
<wgrant> Ah!
<wgrant> +branches is batched and has a navigator.
<wgrant> But isn't linked from anywhere.
<lifeless> ouch
<wgrant> mwhudson: Do you know why this is so?
 * mwhudson afks
<mwhudson> wgrant: no
<poolie> wgrant, use an api
<poolie> :/
<wgrant> poolie: There's UI. It's just got no links to it.
<lifeless> wgrant: you may find https://bugs.edge.launchpad.net/malone/+bug/607776 interesting
<_mup_> Bug #607776: +filebug-show-similar FTI query timeouts <oops> <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/607776>
<poolie> then how does it smell? :)
<wgrant> 'Seq Scan on bugtask'
<wgrant> WTF
<lifeless> yes
<lifeless> case statement fail
<poolie> lifeless, why would you even write such a query?
<lifeless> ORM
<lifeless> or confusion
<poolie> is there some innocuous stortm thing that produces it?
<lifeless> I don't know; the question is how long it will take to find and fix them all
 * poolie wonders if storm should sometimes generate Warnings
<wgrant> lifeless: Well, if we take the timeout down to 1s, they should all become fairly clear :P
<lifeless> wgrant: I want to get the timeout to 5s
<wgrant> I really love that you are actually concerned about speed, BTW.
<lifeless> 1 would be ok if we generally completed in 0.1s
<wgrant> It is something that has been missing from LP for... well... 5 years now.
<lifeless> mmm
<lifeless> All I'm offering is a detailed view
<poolie> istm storm could also warn about 1=2
<poolie> not sure if it's worth it
<lifeless> the whole team has cared and been fixing stuff
<lifeless> its important not to forget that
<poolie> yeah it's really inspiring how much the atmosphere was different last week to previously
<wgrant> True.
<lifeless> if I am lucky enough to have some experience solving performance issues and dealing with interlocking pessimisations - its because I've created my fair share!
<wgrant> Haha.
<spm> damn. that's an admission and a half. /me takes a copy for later blackmail potential ;-)
<wgrant> lifeless: Are you doing DB reviews now in BjÃ¶rn's place?
<lifeless> yes
<lifeless> two votes, stub & I
<lifeless> we've agreed that with one vote it can progress to the tree
<lifeless> two votes for deployment
<wgrant> Ah, handy.
<wgrant> rockstar: Hm, what's particularly difficult about fixing the factory? I saw it broke quite a few tests (which was why i didn't fix it then and there), but nothing that looked terribly hard to fix.
<lifeless> mwhudson: I'm not going to get to your review - sorry, I'd love to:) - I'll try to drop a couple of thoughts in it today though.
<rockstar> wgrant, most of our tests are wrong...
<wgrant> Hah.
<wgrant> Awesome.
<rockstar> wgrant, so the tests that aren't breaking are the worst.
<rockstar> wgrant, it's not "hard" per ce, just tedious, and probably not the best use of my time currently.
<wgrant> Yep.
<rockstar> wgrant, there are all sorts of hidden constraints inside soyuz that aren't really documented that I didn't know, and that we're not really exercising in tests anywhere.
<wgrant> rockstar: lalala not part of Soyuz any more lalala
<lifeless> wgrant: you're not?
<wgrant> lifeless: The code isn't.
<lifeless> soyuz.add(wgrant)
<lifeless> :P
<wgrant> lp.buildmaster's owned by everyone now :P
<rockstar> wgrant, for instance, the build farm gets DOSed if you submit a build that isn't against a distroseries that isn't part of the ubuntu distro.
<lifeless> wgrant: all code is owned by everyone anyway, in principle.
<lifeless> I'd like to see more application of that principle :)
<wgrant> rockstar: Hm, because they have no chroots?
<wgrant> Or is there something else?
<lifeless> bad error handling logic
<lifeless> AIUI
<mwhudson> lifeless: fairy nuff
<wgrant> Well, of course, it's ex-Soyuz code.
<rockstar> lifeless, probably a little bit of everything.
<lifeless> whats the easiest way to join the dots from a page id to code
<wgrant> I guess which app it's likely to reside in, then grep the app's browser/configure.zcml.
<rockstar> lifeless, this is another question I had when I first started working on Launchpad that would have been immensely helpful in my ramp up.
<wgrant> There must be a better way.
<lifeless> why does Distribution:+filebug need to know SELECT COUNT(*) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1 AND (1=1) ?
<lifeless> so next ramp up question
<lifeless> lib/lp/bugs/templates/bugtarget-filebug-search.pt calls into +filebug with id filebug-search-form
<lifeless> what maps that back to a new template / api call ?
<lifeless> grep only finds a js file
<wgrant> The dupefinder is JS.
<lifeless> ok
<lifeless> so its setting up a call into a js function
<rockstar> lifeless, yes.
<lifeless> which then calls an api to do the deed
<rockstar> lifeless, we use the term "api" very liberally, but yes.
<lifeless> it makes a call to something that isn't formatted in tal :)
<rockstar> lifeless, yes.
<rockstar> REST api, model api...
<lifeless> ok, its a self post
<lifeless> to the same template with a query
<lifeless> now... what maps *that* is my next connector to join
<wgrant> lifeless: Must be in FileBugGuidedView somewhere. I suspect it's inherited from FilebugShowSimilarBugsView.
<wgrant> (yay, consistent capitalisation...)
<wgrant> What's the query?
<lifeless> wgrant: hmm?
<lifeless> wgrant: sorry, I've clearly confused you
<lifeless> I'm working on the filebug timeouts
<lifeless> we're using fti inefficiently
<lifeless> I've spawned enough bugs to keep everyone busy for a few days, time to pitch in and help
<lifeless> also
<lifeless> volunteer needed to do a i18n patch for lp
<wgrant> lifeless: lib/lp/bugs/model/bugtask.py, findSimilar
<wgrant> (not findSimilarBugs)
<wgrant> Hm.
<wgrant> Hasn't i18n been rejected a few times?
<lifeless> wgrant: so dots do you join to get to that
<wgrant> I thought it was very much unwanted.
<lifeless> wgrant: I thought it was 'too hard'
<lifeless> wgrant: We have massive i18n in answers
<rockstar> ETOOMANYTASKSTOOLITTLEDEVELOPERS
<wgrant> lifeless: Uh, well, I looked at FileBugGuidedView, and saw that it didn't have that functionality. But there's a very suspicious base class: FilebugShowSimilarBugsView
<wgrant> That calls findSimilar.
<wgrant> lifeless: So: guesswork.
<lifeless> what took you to FileBugGuidedView ?
<lifeless> wgrant: and whats calling into findSimilar
<lifeless> there are other nuts queries involved
<rockstar> lifeless, I'm happy to show you how I connect the dots.
<wgrant> lifeless: FilebugShowSimilarBugsView calls findSimilar.
<rockstar> But also, there's ++oops++ to see some of the callins.
<wgrant> I found FileBugGuidedView by checking lib/lp/bugs/browser/configure.zcml
<lifeless> ok
<lifeless> I think thats about my new-info buffer for now
<lifeless> wgrant: did you have a db patch ?
<wgrant> lifeless: Yeah. But the MP doesn't have a description quite yet.
 * wgrant is just cleaning it up.
<rockstar> wgrant, do you know how to get the tests to tell me all the queries a test is running?
<wgrant> rockstar: I just use the postgres log.
<wgrant> lifeless: Review requested.
<lifeless> url me up
<wgrant> lifeless: https://code.edge.launchpad.net/~wgrant/launchpad/link-uploaded-ddebs/+merge/29669
<lifeless> what does binarypackagerelease represent
<wgrant> A Debian binary package.
<wgrant> ie. a .deb
<wgrant> Or a .udeb
<wgrant> Or now a .ddeb
<lifeless> man, we have terrible names
<wgrant> You just noticed?
<wgrant> Would you like a DistroArchSeriesBinaryPackageRelease, which is of course short for DistributionArchitectureSeriesBinaryPackageRelease?
<wgrant> We have one of those.
<rockstar> lifeless, those clever landscape devs already wrote a tool to do what we were talking about.
<lifeless> \o/
<lifeless> is it open sourced yet ?
<lifeless> wgrant: oh, I know. I was just compelled to comment
<lifeless> its like a class called IntegerInstance
<rockstar> lifeless, https://pastebin.canonical.com/34889/
<lifeless> jml just deleted something along those lines
<lifeless> check annotate of our tracers module; he may have been over enthusiastic
<lifeless> anyhow the thing needed is glue <-> scripts
<lifeless> meep
<lifeless> once it drops out of cache fti is -realllllllllly- slow
<wgrant> Even once stub attacks it with his magic wand?
<lifeless> a wizzards staff has a knob on the end
<wgrant> Shh.
<lifeless> lpmain_staging=> explain analyze SELECT COUNT(CASE WHEN Bug.fti @@ ftq('sphinx') THEN TRUE ELSE null END) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1;
<lifeless> (8 rows)
<lifeless> Time: 109213.606 ms
<wgrant> what.
<lifeless> lpmain_staging=> select count(*) from bug, bugtask where Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('sphinx');
<wgrant> Tried the query that has a less awful plan?
<lifeless>  Time: 54666.999 ms
<wgrant> Ow.
<lifeless> still
<lifeless> better is, well, better.
<wgrant> But still utterly awful.
<lifeless> shrug
<lifeless> one step at a time
<rockstar> Does launchpad_ftest have a different permission file than database/schema/security.cfg
<lifeless> make schema to reset it
<lifeless> ftest is built when you do make schema
<lifeless> and thus only gets permissions built at that point
<wgrant> rockstar: security.py is run only on launchpad_empty, so it should be the same.
<lifeless> or whatever :P
<rockstar> lifeless, that's what I suspected, but it doesn't seem to be working.
<rockstar> launchpad_ftest is also created when the tests run, according to the postgres logs.
<lifeless> yeah, I had the names wrong
<wgrant> launchpad_empty is copied to launchpad_ftest_template
<lifeless> its been a while
<wgrant> launchpad_ftest_template has the sampledata loaded into it.
<lifeless> I remembered the shape :P
<wgrant> Then the testrunner copies launchpad_ftest_template to launchpad_ftest freshly for every test.
<wgrant> No permission resetting is done after launchpad_empty, AFAIK.
<wgrant> (if it was, tests would take a minute or two to set up...)
<rockstar> They already take a minute or two to set up.
<wgrant> Each test takes a second or so.
<wgrant> The initial setup of the test suite around 10 seconds, here.
<wgrant> Do you use LP_PERSISTENT_TEST_SERVICES?
<lifeless> garh
<lifeless> stab stab
<lifeless> terrible idea.
<wgrant> It saves time :)
<rockstar> wgrant, I do.
<wgrant> Lots of time.
<wgrant> Almost makes TDD plausible.
<lifeless> its curing the symptom
<lifeless> but adding more root causes.
<rockstar> lifeless, +1
<rockstar> "Why am I starting up a librarian when my tests don't need one"
<wgrant> Yeah, can we please do away with layers?
<bigjools> good morning
<wgrant> Evening bigjools.
<bigjools> g'day wgrant
<rockstar> lifeless, does this look okay for that branch you reviewed already? http://pastebin.ubuntu.com/466891/
<wgrant> bigjools: When you have a moment, can you glance at https://code.edge.launchpad.net/~wgrant/launchpad/link-uploaded-ddebs/+merge/29669 and tell me if you object strongly to its approach?
<bigjools> looking
<lifeless> rockstar: yes
<bigjools> wgrant: good approach, just a few comments
<bigjools> wgrant: unit tests please, not tests in doctests
<bigjools> wgrant: also consider factoring out your new code so you can write unit tests for it
<bigjools> Running the upload processor in tests is something I want to reduce, not increase ;)
<bigjools> wgrant: finally, my alarm bells go off very strongly when I see removeSecurityProxy being used in code
<wgrant> bigjools: Fair point.
<wgrant> In my defense, it's a reasonably large test to rewrite, and I revived this branch from October last year. I'll probably replace nascentupload-ddebs.txt in a separate branch.
<bigjools> ok
<wgrant> And yes, I strongly considered factoring that out.
<bigjools> you should XXX it and file a bug plz
<wgrant> But it means a significant restructure, so... maybe for another branch.
<bigjools> that's fine
<bigjools> thanks for fixing ddebs!
<wgrant> I want Soyuz cleaned up too, don't worry.
<wgrant> How can I avoid the rSP use?
<bigjools> why is it needed?
<wgrant> BPRs are traditionally immutable.
<wgrant> But I need to set debug_package on some of them, after I have created them all.
<wgrant> I guess I could create all the ddebs first, then the debs. Hmm.
<bigjools> where does the bpr come from?
<bigjools> as in, what creates the storm obj?
<wgrant> The set of uploaded BPRs is created. I then create links between some of them.
<wgrant> Um.
<wgrant> I forget.
 * wgrant finds.
<bigjools> if it ultimately comes from getUtility then it's security wrapped
<bigjools> but this all runs zopeless, so I don't understand why rSP is needed
<lifeless> are my feet ok there?
<wgrant> It's Build.createBinaryPackageRelease, so it's wrapped.
<wgrant> bigjools: It still uses PermissiveSecurityPolicy.
<wgrant> Zopeless is not Zopeless.
<lifeless> rockstar: are my feet ok there?
<rockstar> lifeless, yup.
<bigjools> I realise - but I thought that since you're not actually logged in, it can't apply security
<wgrant> PermissiveSecurityPolicy still cares about permissions. It's just that all permissions are implicitly held.
<bigjools> other than interface checks
<bigjools> exactly
<wgrant> And nobody has permission to write to BPR.
<bigjools> so, why is it grumbling about that
<bigjools> ah
<bigjools> that would explain it
<bigjools> can you create the ddeb first then, and put it in the createBinaryPackageRelease params?
<wgrant> I remember considering that when I wrote the code late last year... let's see if I can remember why I discarded that idea.
<bigjools> ok
<lifeless> stub: hai!
<lifeless> stub: I have some info on fti for you
<lifeless> stub: are you home or in transit ?
<stub> lifeless: home
<lifeless> so, the CASE approach used in nl_search takes about 10 times longer to do a search than a count(*)
<lifeless> separately from considering whether doing an estimate + a search is worth it vs just doing a ranked search
<lifeless> stub: are there some tests for nl_search I can build on ?
<stub> I don't know - that was all bug team work. I only know it vaguely.
<lifeless> ok
<lifeless> uhm
<lifeless> I will hunt bjorns
<rockstar> Huh, I can't remember the reasoning for making people the base parent in recipe traversal.
<rockstar> Whatever it is, my mind has been changed and I think that's crackful.
<wgrant> The form should at least inform users that the recipe names are global.
<rockstar> wgrant, it does now.
<wgrant> Ah.
<rockstar> Well, it does when you try and re-use a name.
<rockstar> The problem is that I want more than one recipe named "daily" ...
<wgrant> I often try to call a recipe 'trunk'.
<wgrant> Right.
<rockstar> It should have also traversed to project or sourcepackage
<rockstar> "Paul Hummer's daily recipe" is silly
<wgrant> What happened to ~wgrant/launchpad/+recipe/daily?
<rockstar> Basically it means that I'm going to invent my own namespace.
<wgrant> I thought the initial proposal was something like that.
<rockstar> wgrant, yeah, but there was some reason for not doing it.
<rockstar> (I'm saying that now that abentley and I have implemented it, that reason is shit)
<wgrant> Haha.
<lifeless> ok
<lifeless> so bugs -> db -> stub -> bjorn -> flacoste
<lifeless> but I now have enough confidence in the issue: deleting painful code.
<rockstar> bigjools, could you maybe give me some hints here: https://bugs.edge.launchpad.net/launchpad-code/+bug/591618
<_mup_> Bug #591618: Result of SPRBuild assumes distro build <recipe> <Launchpad Bazaar Integration:Incomplete by rockstar> <https://launchpad.net/bugs/591618>
<bigjools> rockstar: gimme some time, I'm fighting canonicaladmin
<rockstar> bigjools, it will take AGES to defeat canonicaladmin though!
<bigjools> rockstar: this is just the battle, for the war, totally
<rockstar> :)
<danilos> adiroiban, jtv, woohoo, +templates works on edge (rendered https://translations.edge.launchpad.net/ubuntu/lucid/+templates in 6.52s)
<jtv> danilos: yay!
<jtv> danilos: that's with figuring out the menu links just once for the entire page?
<jtv> Hmm.... it just timed out for me.
<adiroiban> danilos: sometimes I get a time out on Ubuntu
<adiroiban> openobjects-addons is 0.6s
<danilos> adiroiban, jtv: yeah, we still have it time-out occasionally because of the rendering times, but still, it worked for me :)
<danilos> karmic consistently times out for me but it does have a lot more templates on there
<jtv> I got one loaded!
<jtv> On the 3rd attempt though.
<jtv> The mouse-over effect doesn't seem to be working... maybe one of the extra files failed to load.
<danilos> jtv, adiroiban: I guess we'll either have to fix rendering time (I talked to gary_poster and he mentioned switching to a faster TAL-rendering library chameleon as an option)
<danilos> jtv, adiroiban: or batch it :)
<bigjools> ok rockstar let's check out your branch
<rockstar> bigjools, it's just a bug, but I have a test in there that I can't get to act like production.
<rockstar> bigjools, noodles is also sitting right next to me, and so I was going to ask him.
<bigjools> rockstar: I think what should happen is that canonical_url should return None for a package built in a PPA
<bigjools> and then you can change the template code to not link it
<rockstar> bigjools, yes, I know this.
<rockstar> bigjools, the problem is that I can't reproduce this in a tests.
<bigjools> ok
<rockstar> bigjools, so I'm wondering where the formatter is so I can see why it's not linking it in my test.
<bigjools>         release = self.factory.makeSourcePackageRelease(
<bigjools>             source_package_recipe_build=None)
<bigjools>         release.source_package_recipe_build = build
<bigjools> why 2 lines?
<bigjools> transaction.commit() - urg :)
<rockstar> bigjools, good question.
<bigjools> why do you need to commit?
<bigjools> rockstar: so I think you also need to assert in that test that the build's archive is a PPA
<rockstar> bigjools, that test is the result of late afternoon hacking, and trying whatever I can to make it work.
<bigjools> ok
<bigjools> does makeRecipeBuild() default to a PPA?
<rockstar> I think it takes the default archive.
<bigjools> I'd add the assertion in the test
<rockstar> bigjools, where is the formatter?  I was thinking I would find it after I got the test working, because it wasn't clear.
<bigjools> rockstar: what formatter?
<rockstar> The SourcePackageReleaseFormatter in tales.py didn't seem to be it.
<rockstar> bigjools, the tal formatter for a source package release
<bigjools> there isn't one
<rockstar> bigjools, huh?  Then how is it getting linked?
 * rockstar cries
<bigjools> does the template do it?
<bigjools> anyway, your test is asserting the wrong thing isn't it?
<rockstar> bigjools, no, it passes it to fmt:link
<bigjools> ok
<rockstar> bigjools, no, it's asserting that it's plain text, not linked.
<bigjools> it's still wrong
<rockstar> And it SHOULD be failing, but it's not.
<bigjools> it's not a distribtion package
<bigjools> so you're asserting that the current state of affairs is correct
<bigjools> which of course it always is :)
<bigjools> fmt:link just does canonical_url on the object
<rockstar> bigjools, so are you saying we should take the whole thing out?  Don't even show a release?
<bigjools> it should say "Package X in PPA blah"
<bigjools> or something
<poolie> is there any other lib.lp.service module that sets a good example for code or test style?
<rockstar> bigjools, yes, and I'm saying "Where is the code that names out that template?"
<bigjools> is there a real example I can see?
<bigjools> I dunno, it's one of yours isn't it?  +recipe page?
<rockstar> bigjools, https://code.edge.launchpad.net/~rockstar/+recipe/daily/+build/348
<bigjools> ok let's take a looksee
<bigjools> ah I see
<bigjools> rockstar: junk the whole section
<rockstar> bigjools, the template is just using existing code.  I believe abentley even just abstracted the original build index.
<rockstar> bigjools, really?!
<bigjools> yes, it doesn't make any sense for a PPA build
<bigjools> there's no package page on PPAs
<rockstar> bigjools, it's a good thing you're in a different country, because I REALLY wanna kiss you right now.
<bigjools> lol
<bigjools> so "Result:" and the line below it - blow them away
<bigjools> or if you're in a kissing mood, just blow them
<rockstar> bigjools, could you comment on the bug with something like that?
<bigjools> sure
<rockstar> ...and I'll happily just kill it.
<bigjools> rockstar: the binary builds are plenty good enough to detail what was built
<rockstar> bigjools, I agree.
<poolie> lifeless, igc says 'congrats'
<lifeless> poolie: thanks! - tell him best wishes !
<lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/malone/+merge/30507 if anyone is interested :)
<barry> bigjools: hi.  on my ppa package details page i occasionally see (Newer version available).  is this information available in the api?
<lifeless> barry: yes
<barry> lifeless: is this the superseded stuff in the build records?
<lifeless> nah, its a reference to the suite in ubuntu
<barry> lifeless: okay, thanks. i'll try to find it ;)
<bigjools> I think it's only on the UI actually
<gary_poster> danilo: re chameleon: fwiw, in my measurements last year it saved an average of 15% from page times.  not too shabby, but not a silver bullet.  in the big picture, for common pages, network and SSL and client-side issues dwarf app-server times; and for most OOPSes we have, db time dwarfs rendering times.  but chameleon should help somewhat.  It would probably be more of a benefit for big honking pages like the on
<gary_poster> talking about.
<gary_poster> probably was truncated: "but chameleon should help somewhat.  It would probably be more of a benefit for big honking pages like the one you were talking about."
<barry> bigjools: ah.  what i'm trying to do is script auto-rebuilds for packages in the ppa that are now out of date.  basically: search to find them, apt-get source, dch -i, build -S, dput
<bigjools> barry: ummm, why are you duplicating builds in Ubuntu?
<bigjools> or are there extra local changes?
<barry> bigjools: they are builds of packages that build-dep on python-all and python-dev-all.  i build them in the ppa to pick up python 2.7
<wgrant> And to destroy the build farm for 1.5 weeks :)
<bigjools> exactly
<barry> wgrant: not any more :)
<bigjools> if you want to do rebuilds, you should use a rebuild archive
<bigjools> or your good name is going to be mud :)
<barry> bigjools: i really *don't* want to rebuild vast numbers of packages.  only about 150 of them
<barry> bigjools: your flaw is assuming i ever had a good name
<bigjools> barry: ah ok so you're rebuilding packages where its build deps changed?
<bigjools> barry: ok, s/mud/muddier/ :)
<barry> bigjools: yeah :)
<barry> bigjools: but as updates get thrown in the archive, i have to essentially re-sync them, but i think you can't do that (or can you?)
<bigjools> rockstar: err sorry I changed your bug status, it said "incomplete" on my browser, honest :)
<bigjools> we could do with collision detection there
<barry> bigjools: otherwise, apt-get update && apt-get upgrade will pick up the newer, non-py27 versions instead of the py27-built versions i want from my ppa
<deryck> Morning, all.
<bigjools> barry: you can copy from Ubuntu into your PPA very easily using the API
<barry> bigjools: syncSources?
<bigjools> barry: you can pin on your PPA too
<bigjools> and syncSources, yes
<barry> bigjools: yeah, i could do that
<bigjools> pinning would be better
<bigjools> fewer rebuilds ;)
<barry> bigjools: <cough>more (efficient) buildds</cough> :)
<bigjools> in progress ;)
<barry> bigjools: :)
<bigjools> jelmer and myself are making buildd-manager changes that will rock the world
<barry> bigjools: i'd love to hear more!  but it's lunch time so i'll come back and ask
<gary_poster> barry: I've been seeing if I could get buildout tests to pass with 2.7.  is there a known issue for Python 2.7 that output from both spawn and subprocess is...held, for some reason?
<gary_poster> For example, given hacked code like this http://pastebin.ubuntu.com/466949/ (and I've hacked to use both spawn and subprocess in tests), including that half second sleep, I get failures like this: http://pastebin.ubuntu.com/466950/ .
<gary_poster> (IOW, the "An error occurred..." should be printed below the output from the subprocess call, but is not).
<gary_poster> So, known?  I'm about to give up.
<gary_poster> (2.4, 2.5, 2.6 don't behave this way)
<wgrant> bigjools: removeSecurityProxy call gone.
<bigjools> woo
<wgrant> I shuffled things around a bit, creating the ddebs first and a couple of other bits and pieces.
<bigjools> argh, I am finding it impossible to read email quicker than it arrives today
<bigjools> death to email from machines
<bigjools> email is for people fer chrissake
<bigjools> lifeless: I've got another candidate to use for Rabbit if it's close to being implemented
<lifeless> bigjools: it needs someone to put a layer together + lazr-config schema entry for it
<lifeless> bigjools: at that point we have all the bits in place to iterate.
<bigjools> cool
<bigjools> lifeless: I want to put the new distroseries opening code (initialise-from-parent) on it
<bigjools> Steve is working on making it a Job
<wgrant> Whatever happened to CHR?
<deryck> wgrant, it's still happening, but I think some confusion in the change to the new approach and new days has slipped coverage lately.
<deryck> wgrant, it's Edwin's day today, but he's not around yet.
<bigjools> deryck: when someone updates a bug status, do you detect collisions? (with a second person updating it)
<deryck> bigjools, no.  we just have the latter one overwrite the first attempt.
<bigjools> deryck: do you have bugs about that?  If not I think I should file one :)
<lifeless> mtaylor: hi
<lifeless> bah
<lifeless> mthaddon: hi
<lifeless> mthaddon: could you do me a small favour? on staging, up the timeout to 20 seconds, I want to test a theory about whats going on here.
<deryck> bigjools, there is a "show edits in real time" bug, which is basically the same, but perhaps your issue is a bit more specific?
<bigjools> deryck: I changed a status from incomplete to triaged, but because I had not loaded the page recently I didn't see that someone else had changed it from incomplete to in-progress
<bigjools> so I override his change
<bigjools> seems like we could simply check the status we're changing from and see if the status on the bug now is still the same
<deryck> hmmm, yeah, I think that could use a new bug.  That's a simpler change than real time edits and would bring value now.
<lifeless> deryck: bigjools: landscape has a jobs system too which they say they use for ajax notifications
<bigjools> lifeless: yeah I replied to their thread
<lifeless> its slightly different in concept to ours
<lifeless> cool
<bigjools> it sounds great
<lifeless> I'm pulling on this thread
<bigjools> deryck: ok I'll file a bug for you, cheers
<deryck> yeah, it does sound nice.  That would make real time edit notifications possible, I believe.
<lifeless> yeah
<deryck> bigjools, thanks.
<lifeless> separate to mid flight detection
<deryck> right
<bigjools> deryck: BTW my mental picture of you has now shifted unassailably to one where you're playing Dominion and sledging your opponents
<deryck> heh
<deryck> I perhaps was a bit too relaxed at the epic.  I normally hide that part of myself.  :-)
<deryck> gaming brings out the worst in me.
<bigjools> deryck: lol
<bigjools> yeah me too - I hardly play any more.  I got asked by neighbours once to stop swearing so loudly.  I've since moved to somewhere where I don't have close neighbours.
<deryck> heh.  I can relate.
<bigjools> That's Counterstrike for you ...
<bigjools> food time
<lifeless> losa ping
<mthaddon> lifeless: hi
<lifeless> mthaddon: hi
<lifeless> mthaddon: I was wondering if you could raise the timeout on staging temporarily, I want to get a full trace of what this does
<lifeless> that slow query, once its not fighting disk - 300ms
<mthaddon> lifeless: done
<lifeless> \o/
<lifeless> try this
<lifeless> go to
<lifeless> https://bugs.staging.launchpad.net/ubuntu/+filebug
<lifeless> put in a search
<lifeless> click next
<lifeless> everyone^
<mthaddon> lifeless: I increased the softtimeout as well - should I have left that as is?
<wgrant> WoahJSbug.
<wgrant> Oh, no, Chromium bug.
<lifeless> mthaddon: no, thats fine
<lifeless> mthaddon: the main thing was to get enough headroom to see how it performs on this slower hardware
<lifeless> I am getting 2-3 second searches
<barry> gary_poster: no bug or change that i know of :/
<lifeless> mpt: ^ what do you think?
<lifeless> deryck: ^ you too, and I think I'll have enough confidence to say we should give this a spin
<mpt> lifeless, I tried "Internet doesn't work on Broadcom". It returned decent results (though I have no idea what they're missing), but took 25 seconds.
<deryck> lifeless, sorry, me too what?  Search at ubuntu filebug on staging?
<lifeless> mpt: ok; please try again
<lifeless> deryck: yeah
<mtaylor> lifeless: ola
<lifeless> mpt: the search index appears to be very big, so not all fitting in memory on staging, which is tiny
<mpt> "wobbly windows flicker": ~8 seconds, decent results
<rockstar> lifeless, where are you physically?
<lifeless> rockstar: in my room for the moment
<lifeless> I'll be down shortly
<rockstar> lifeless, ack.
 * rockstar will be glad to go back to his desk tomorrow
<lifeless> deryck: so this is a replacement pre-filter for bug searching
<lifeless> deryck: I haven't audited the full stack, but if we feel this is decent we might want to CP it to prod
<deryck> lifeless, yeah, pretty snappy response for me.  2-3 seconds on normal searches.  Lots of words returns in 5-6 seconds for me.
<lifeless> deryck: does it feel comparable to prod for you, or better ?
<lifeless> [ignoring hardware]
<deryck> lifeless, better actually.
<lifeless> ok
<lifeless> I shall do the do
<lifeless> mthaddon: thanks, I'm finished with staging as a test env for this.
<mthaddon> k
<mpt> "install eclipse get unmet dependencies error": 13 seconds, and doesn't return bug 603656 at all.
 * rockstar wonders how to capture mpt in a test
<lifeless> mpt: does it on production ?
<mpt> gnarrrrgh
<mpt> lifeless, I can't test on production without leaving the beta testers team. On edge, the same search gives me a timeout.
<lifeless> mpt: click the 'do not redirect me button'
<mpt> oh of course
<lifeless> mpt: if there isn't one, please (har har) file a bug because we should still show that even in a new ajax world
<mpt> I don't see that button any more
<mpt> It's supposed to be on http://launchpad.net/ right?
<lifeless> mmm
<lifeless> I thought so
<deryck> mpt, lifeless -- I believe it's a link in the footer now.
<wgrant> mpt: It's only shown on edge, and it's at the bottom.
<wgrant> And it's not on the front page.
<mpt> The footer for me is "Â© 2004-2010 Canonical Ltd.  â¢  Terms of use  â¢  Contact Launchpad Support  â¢  System status  â¢  Take our survey!"
<mpt> and on edge, "Â© 2004-2010 Canonical Ltd.  â¢  Terms of use  â¢  Contact Launchpad Support  â¢  System status  â¢  Take our survey!  â¢  r11179 beta site"
<mpt> oh, not on the front page
<wgrant> Right.
<wgrant> That would be too obvious :)
<mpt> The footer doesn't change on other pages for me
<lifeless> mpt: on the bug filing page
<mpt> oh, wait, THAT footer
<lifeless> bottom right, green
<wgrant> It's in the non-footer footer.
<wgrant> Right.
<lifeless> this is special :(
<mpt> The blue one as opposed to the white one
 * mtaylor remembers days when he did web programming...
<lifeless> times out on prod for me too
<mpt> timeout for me too
<lifeless> so
<lifeless> prod does not show it either
<gary_poster> ack, barry, thanks for looking
<lifeless> mpt: thank you very much for playing with it; unless you say 'its a disaster' I'm going to move forward to get it on edge asap
<mpt> It's not a disaster
<mpt> afaict
<lifeless> deryck: your opinion?
<barry> gary_poster: np.  i'd be interested if you track it down to a bug in python
<rockstar> lifeless, deryck's opinion may or may not involve comparisons to Windows 7, so tread carefully.
<deryck> lifeless, I say move forward on it.  If timeouts don't happen on staging, we should be better on edge/prod too.
<deryck> yes, if it's better than windows 7 bitches, I'm all in!
<gary_poster> heh
<gary_poster> barry: cool. I dunno if I'll take it that far, but we'll see
<lifeless> when do we set things 'fix committed' ?
<lifeless> ... do we set them to that ?
<rinze> lifeless: When the change lands but hasn't been deployed
<rinze> lifeless: Scripts will set them for us though
<wgrant> bigjools: Did you get around to the PPA log parser thing you were looking at a couple of weeks ago.
<bigjools> wgrant: not yet, sorry.  The code and config is all in though, we just need to turn it on
<wgrant> bigjools: Even the row-limiting config?
<bigjools> yes
<wgrant> Excellent.
<bigjools> life would be so much easier if I could block new bugs on soyuz
<lifeless> tell you what
<lifeless> ship code with less bugs, less bugs will be filed ;)
<bigjools> Or I can use Jedi mind tricks.  These aren't the bugs you're looking for.
 * bigjools waves hand
<wgrant> Do the Jedi mind tricks involve LOSAs and DELETE FROM bugtask?
<lifeless> no
<lifeless> :)
<bigjools> uh what?  Sorry, my mind had drifted to Princess Leia and 'that' outfit
<lifeless> a danish on each ear
 * mthaddon was picturing bacon on Leia's ears...
 * bigjools just spurted a mouthful of coffee
<lifeless> rotfl
<bigjools> mthaddon: I didn't know Jono was in Star Wars
<mtaylor> lifeless: really need one more bug status man
<mtaylor> lifeless: fix committed, fix merged and fix released...
<mtaylor> just saying
<lifeless> meh
<mtaylor> three different things, important to three different sets of people
<lifeless> less actually
<lifeless> overly precise is not always good
<lifeless> committed where, merged where, released where.
<mtaylor> indeed
<lifeless> really, we want 'done in context' and many contexts: more flexible, more power, nuke bugtasks too
<mtaylor> for that matter - what the hell does "committed" mean
 * lifeless handwaves furiously about stuff in deryck's domain
 * deryck welcomes domain intrusion and hand-wavery
<mtaylor> I just want to know: a) when there's a tree claiming to fix it on launchpad somewhere- when that has hit trunk, and when that has hit a "release"
<mtaylor> hrm
<mtaylor> I should have injected the b and the c there
<mtaylor> imagine them
<lifeless> sure
<lifeless> see, that fits my model better :P
<mtaylor> good
<lifeless> flacoste: btw, I'm changing how answers search works :P
<flacoste> lifeless: ok, why?
<lifeless> because its the cause of about 8 seconds overhead on searching bug dups
<lifeless> its doing work that the fti index should do
<lifeless> causing a scan of every bug, ever, in the context of the search
<lifeless> flacoste: the change shouldn't be very visible to users, based on some brief testing we did on staging.
<lifeless> flacoste: but if we do need to change the ranking we should do that at the tsearch2 layer
<flacoste> interesting
<flacoste> how a search within answers cause an 8 seconds overhead on search bug dups?
<lifeless> the nl_search code
<lifeless> common code path
<lifeless> its also causing timeouts in answers
<lifeless> flacoste: https://code.edge.launchpad.net/~lifeless/launchpad/malone/+merge/30507 - the old code is kept so we can really easily reenable if we don't like this.
<flacoste> lifeless: ok, the quality of results will decrease significantly
<flacoste> but let the user determine that
<flacoste> the slow algorithm is based on MySQL implementation of full text search
<lifeless> flacoste: I'd like to iterate and make it better
<lifeless> yeah
<flacoste> which gives very good result
<flacoste> plain tsearch2 search sucks
<lifeless> so what it does to use is due to not using their entire implementation
<lifeless> s/use/us/
<flacoste> which is fast :-)
<lifeless> we end up scanning every object in the context, across the board.
<lifeless> which is a lot of work
<lifeless> - 8 seconds on ubuntu, and growing: for a *single* term
<lifeless> when we have multiple terms, its worse.
<lifeless> because we're consulting the fti vector per-row
<lifeless> per-term
<lifeless> this is why multi term searches time out more than single term searches
<lifeless> I am proposing this as a band aid
<lifeless> get us some headroom
<flacoste> that's fine
<lifeless> and we'll either use tsearch2 better, or drop in an entirely dedicated search engine
<lifeless> great, thanks.
<lifeless> the replacement query, FWIW, is ~ 500ms on staging
<lifeless> (once disk IO is excluded)
<flacoste> and maybe tsearch2 ranking has gotten better over the year
<lifeless> stub says we haven't tuned it at all
<lifeless> you're meant to write your own rank function anyhow
<lifeless> flacoste: thanks; I'm going to focus on the landscape/server guys now for a bit
<deryck> gary_poster, ping
<gary_poster> deryck: pong
<jml> lifeless, not sure whether the 'tracer' discussion above got resolved
<jml> lifeless, there's code similar to what's in https://pastebin.canonical.com/34889/ in lp/testing/__init__.py, iirc.
<leonardr> poolie, iirc you said yesterday you were okay with the current implementation of https://code.edge.launchpad.net/~leonardr/launchpadlib/improve-workflow/+merge/29849. if that's true, can you mark as approved? otherwise, comment with the change you want me to make
<lifeless> jml: neither am I :)
<jml> lifeless, ok.
<lifeless> deryck: ping - bug 600934
<_mup_> Bug #600934: BugSubscription table need a constraint to prevent people being subscribed twice to the same bug <story-better-bug-notification> <Launchpad Bugs:Won't Fix by brian-murray> <https://launchpad.net/bugs/600934>
<deryck> lifeless, yup.  know it well.  wasn't sure if we should have marked it won't fix, but haven't got back to it today yet.
<lifeless> deryck: seems to me that merging people should merge their subs too, no ?
<lifeless> the merge code is approachable; wearing my maintainablity / data hygiene hat - this is totally something we should fix
<deryck> lifeless, agree.  But this wasn't was Brian was looking into, and sinzui seemed to suggest this was difficult to fix.
<lifeless> we wrote the code in the first place :)
<sinzui> lifeless, I have argued fixing merge for more than a year. The work is *always* considered out of scope
<lifeless> sinzui: we're agreed then that we should fix it.
<lifeless> sinzui: thats plenty for me; I know how scheduling works. If its considered really tricky, I'll happily mentor someone on it.
<sinzui> No one agrees it it is worth one cycle
<lifeless> its worth a couple of days, tops.
<deryck> sinzui, where does this code live in the tree?
<lifeless> less ~/Desktop/Downloads/signature.asc
<lifeless> bah
<sinzui> I have a script I am considering for a garbo job. to cleanup some of the data /after/ a merge. But the correct solution (and certainly assumed by brian's change) is to make merge an out of proc task that has a chance to succeed. eg, to take 27 patient tries to complete a merge of ~barry
<lifeless> sinzui: are we hitting write contention ?
<lifeless> merge should be an out of webapp thing *anyway*
<lifeless> the whole merge operation, I mean.
<sinzui> deryck, person merge...but keep in mind. We want to start new features, but we seem to be working on performance instead.
<lifeless> sinzui: in the eyes of our stakeholders, performance is as much a feature as anything else.
<sinzui> lifeless, There is too much work to do in a merge to happen in a single request.
<lifeless> sinzui: ack - agreed. Thus my comment that it shouldn't be done in the webapp transaction *anyway*
<sinzui> lifeless: I think everyone needs agree with what my team is working on, privacy, timeouts, or merge.
<lifeless> sinzui: I'm happy to consider the bug in question medium or even low priority
<lifeless> sinzui: I have no view on its urgency
<deryck> lifeless, sinzui -- so if we want the constraint bug kept around to track this, I'm fine, but I think it needs fixing up to generally represent this issue.
<lifeless> deryck: sounds good to me
<deryck> I also do think it's out of scope for our current subscription work, and would call it low priority for the bugs team, compared to everything else.
<lifeless> sinzui: we have the team leads call this evening and perhaps we can refine what we're doing in that call
<MTecknology> So... auto-fill for tags.. That's an awesome feature that never crossed my mind.
<sinzui> The bugs are reported every release, the questions are reasked. One issue is number 8 in bug heat: and then there is my old blueprint that gives a broader view of the issues: https://blueprints.edge.launchpad.net/launchpad-registry/+spec/cleansing-deactivated-accounts
<lifeless> sinzui: what does 8 in heat mean - is that high ?
<sinzui> It is the 8th item in heat
<lifeless> ok
<sinzui> karma bugs collectively dominate registry heat
<poolie> leonardr, do it!
<lifeless> flacoste: what is the mechanism for the call in 2 hours ?
<bigjools> lifeless: Francis' conf line
 * bigjools shoots and scores, new buildd-manager working on dogfood working like a champ
<bigjools> jml: don't suppose you fancy reviewing it do you?  I fear someone else would vomit at the horrendous twisted hacks.
<jml> bigjools, sure, but I won't get around to it today
<jml> bigjools, my mind feels like stale custard
<bigjools> jml: I wasn't expecting you to - thanks very much.  Maybe we can mumble over it later this week.
<bigjools> jml: there's some dodgy tests that you might be able to help me make nicer
 * jml afk
<poolie> jam, rockstar, hi, want to meet for dinner?
<rockstar> poolie, as long as I don't have to wait much longer.
 * rockstar is starvin' Marvin.
<poolie> ready now if you are
<rockstar> poolie, lemme pack up then.
<poolie> i'd like to go to the place on the plaza near here
<poolie> the google reviews are good
<jam> poolie: certainly
<rockstar> poolie, sure, as long as it's fast.
<poolie> kk :)
<poolie> COME ON! WE HAVE A 5 SECOND TIME OUT!
<jam> poolie: see you in the lobby in about 5min
<poolie> great
<danilos> anyone remembers where's the documentation for those query counting functions (or at least what they are) that thumper introduced a while back?
<thumper> danilos: assertQueryCount I think
<danilos> thumper, thanks (damn missing -i option on my grepping exercise)
<danilos> thumper, though, I still can't find it being used anywhere
<thumper> danilos: I'm pretty sure it is used in some code tests
<danilos> thumper, that's what I was grepping, I guess it's just a long day
<thumper> danilos: assertStatementCount
<thumper> sorry
<danilos> thumper, right, just as I found it inside __init__.py of lib/lp/testing :)
<danilos> thumper, np, we should probably move jtv's old query counter for +translate page to this as well
<thumper> yup
<danilos> thumper, or well, maybe not because that one counts them for the entire request being rendered
<rockstar> thumper, hi!
<m4n1sh> anyone here can help me with setting up launchpad?
<lifeless> !ask
<m4n1sh> i tried "make schema"
<lifeless> m4n1sh: just ask away, asking for a dedicated helper usually doesn't work in public channels ;)
<m4n1sh> Missing ./download-cache.
<m4n1sh> Developers: please run utilities/link-external-sourcecode.
<m4n1sh> make: *** [download-cache] Error 1
<lifeless> have you followed the wiki pages on setting it up
<m4n1sh> on running https://dev.launchpad.net/Running
<m4n1sh> this page? right?
<lifeless> thats the second one
<lifeless> Getting is the first one
<m4n1sh> i got the edgde code
<m4n1sh> not devel
<m4n1sh> *edge
<lifeless> we don't use the edge code
<lifeless> we work on devel
<lifeless> please follow the 'getting' wiki page, you've skipped a lot of steps
<m4n1sh> oh crap
<m4n1sh> i missed so much
<m4n1sh> lifeless, thanks for pointing it out
<lifeless> np
<lifeless> thumper: hi
<lifeless> thumper: uhm, salgado I think it was, was mentioning that the branch listing page is still having trouble with things with lots of bugs linked, or something
<lifeless> its a little freaky to have windmill opening browser windows in my host os when running tests in my vm
<lifeless> just saying
<thumper> lifeless: file a bug with an oops id and I can check it out
<lifeless> thumper: I told him just that :)
<lifeless> rockstar: up still ?
<rockstar> lifeless, yes.
<lifeless> #lp-reviews, if you have time
<benji> anyone ever run into a pylint warning that can't be silenced?  I added a # pylint: disable-msg=E0221
<benji> ...and it's still complaining.
<lifeless> I'm off to bed
<lifeless> gnight
<benji> night
<flacoste> poolie: i think you picked the bad branch to merge into, as your diff is huge
<poolie> i know!
<flacoste> poolie: that's on https://code.launchpad.net/~mbp/launchpad/flags/+merge/30580
<poolie> lifeless has just pointed that out
<lifeless> just think
<poolie> i know :)
<poolie> i actually realized myself
<lifeless> this *won't happen* anymore in about 6 weeks
<lifeless> or less.
 * lifeless is really gone.
<poolie> never mind the length, feel the quality!
<benji> heh
<poolie> uh is this continuing fallout from people renaming branches?
<poolie> db-devel seems to be gone
<maxb> poolie: gone?
<wgrant> Hmm.
 * wgrant wonders why ShipIt hasn't been split out yet.
<wgrant> Even if ISD can't be bothered rewriting it, it could surely just run on a separate DB with an old, static version of LP...
<benji> wgrant: it would help me if it were split out; I could fix two or three LP bugs by deleting data that LP doesn't use but ShipIt does
<wgrant> Then we could destroy Account, and safely remove stuff without having to check in history if ShipIt used it.
 * rinze waves to wgrant and benji
<wgrant> Morning rinze.
<mwhudson> wgrant: just lack of tuits i think
<wgrant> When is the Lucid upgrade happening?
<wgrant> And is there any OCR happening this week, or should I seek out another reviewer?
<rinze> wgrant: What sort of review do you need/
<rinze> I mean, what kind of branch
<rinze> ?
<wgrant> rinze: It's a nascentupload branch. https://code.edge.launchpad.net/~wgrant/launchpad/link-uploaded-ddebs/+merge/29669
<wgrant> bigjools has already approved the direction.
<rinze> wgrant: If nobody has done a review before European morning I'll do one.
<wgrant> rinze: Thanks, that would be great.
<wgrant> I have a whole stack of other branches on top of this one that I'd like to get landed this cycle.
#launchpad-dev 2010-07-22
<thumper> rinze: new nick jelmer?
<rinze> thumper: Two nicks :-) One I use for work channels and one for non-work channels in an attempt to keep the two more separate
<rinze> obviously I'm failing since it's midnight here
<thumper> rinze: :)
<thumper> rinze: and you are in a work channel
<wgrant> rockstar: What are the security concerns with 'run' in recipes?
<wgrant> We already run it in a secure environment.
<rinze> thumper: are code imports for packaging branches on the roadmap for the near future?
<thumper> rinze: not the immediate future
<wgrant> Lots of the work was done earlier in the year, though.
<rinze> wgrant: not related to code imports for packaging branches..
<rinze> (I'm not complaining, regular code imports work quite well too)
<wgrant> rinze: IIRC james_w removed lots of roadblocks relating to that.
<mwhudson> newImport is on IBranchTarget now isn't it?
<wgrant> I'm not sure if it got finished, but it was mostly done.
<rinze> ah, I didn't realize that
<rinze> James is everywhere :-)
<wgrant> He is.
<mwhudson> you might be able to create code imports for package branches over the api already
<wgrant> I wonder how happy the UI will be about that.
<mwhudson> https://edge.launchpad.net/+apidoc/devel.html#source_package-newCodeImport
<mwhudson> possibly not very
<wgrant> It's possible it will be fine.
<wgrant> But if it was working, I would have thought we'd see some of these imports around.
<mwhudson> mm, might be ok
<mwhudson> someone should try it on staging :-)
 * rinze is about to
<rinze> mwhudson: w00t, that works
<rinze> https://code.staging.launchpad.net/debian/experimental/+source/tdb
 * rinze hugs james_w
<mwhudson> cool
<wgrant> Excellent.
<rinze> mwhudson: thanks!
 * rinze gets some sleep
<mwhudson> even https://code.staging.launchpad.net/+code-imports?field.review_status=NEW&field.review_status-empty-marker=1&field.rcs_type=&field.rcs_type-empty-marker=1&submit=Submit+Query works
 * mwhudson approves the branch, lets see if the staging code import infrastructure has rotted and died again
<mwhudson> seems not
<wgrant> Yep.
<wgrant> Although codebrowse is unhappy with it.
<mwhudson> oh?
<mwhudson> boo
<wgrant> Try to browse into debian/
<wgrant> Hmm.
<mwhudson> yeah, strange
<mwhudson> http://bazaar.staging.launchpad.net/~jelmer/debian/experimental/tdb/experimental/files/head:/debian/patches is ok though
<wgrant> Odd.
<mwhudson> looks like a loggerhead bug
<wgrant> spm: Hi. There's an upload failure caused by a buildd-manager bug that requires some time-sensitive debugging. Do you have a few minutes?
<spm> wgrant: sure
<wgrant> spm: Firstly, when does librarian-gc run?
<wgrant> it destroys the evidence (which was why the debugging failed last time)
<spm> "regularly", let me verify
<spm> once a day, 6pm server
<wgrant> That was 7ish hours ago?
<spm> ath too hard; kcalc; yes.
<spm> *math
<wgrant> If so, it may not have erased everything yet. Yay.
<spm> heh
<wgrant> So, https://edge.launchpad.net/ubuntu/+source/qt4-x11/4:4.7.0~beta2-0ubuntu2/+build/1879966 somehow succeeded once, then succeeded again.
<wgrant> It shouldn't have run again after it succeeded the first time, of course.
<wgrant> Can you check how many LFAs there are with the filename 'buildlog_ubuntu-maverick-powerpc.qt4-x11_4:4.7.0~beta2-0ubuntu2_BUILDING.txt.gz', and how many with the filename 'buildlog_ubuntu-maverick-powerpc.qt4-x11_4:4.7.0~beta2-0ubuntu2_FULLYBUILT.txt.gz'?
<wgrant> (the buggy retry unlinks the log LFAs, so we can't tell what's gone on through the UI, and they get destroyed by librarian-gc as soon as it runs)
<wgrant> But they should still be around at the moment.
<wgrant> Since the event was roughly 6 hours ago.
<spm> so it succeeeded twice? that's a nice trick.
<wgrant> Yes.
<wgrant> And the second time failed to upload, since the files were already uploaded.
<mwhudson> well
<wgrant> And because the Soyuz data model is awesome, there's no record in the DB of the first attempt, except for the orphaned LFAs.
<mwhudson> if it succeeded the first time it should succeed the second time too :-)
<mwhudson> it's the retry that's mysterious
<spm> It's a Miracle, perhaps?
<wgrant> Do you want queries for my above request?
<wgrant> SELECT * FROM libraryfilealias WHERE filename IN ('buildlog_ubuntu-maverick-powerpc.qt4-x11_4:4.7.0~beta2-0ubuntu2_FULLYBUILT.txt.gz', 'buildlog_ubuntu-maverick-powerpc.qt4-x11_4:4.7.0~beta2-0ubuntu2_BUILDING.txt.gz');
<spm> if you can that'd be wonderful - just got pulled into a U1 deploy/update thingy...
<wgrant> Ah, you can do that. This isn't so critical now I know there's a while until librarian-gc runs.
<spm> so generous of you! ;-)
<wgrant> Heh, sorry. Just the last couple of times we've tried to catch this, we've been too late.
<spm> nod
<spm> http://paste.ubuntu.com/467277/ fwiw
<wgrant> Thanks.
<spm> yarp; and they're both still live too
<wgrant> Yeah. Unfortunately the it looks like the first occurence of the issue with this build was a couple of days ago, and it was just retried again this morning.
<spm> curious: The following NEW packages will be installed: libdconf0 libmpfr4 <== the latter is new
<wgrant> So we are again too late.
<wgrant> spm: Um, you're running maverick?
<spm> I'm not, no.
<wgrant> Those two are only in maverick...
 * spm raises eyebrows
<wgrant> +class ILaunchpadPublicationDirective(IRequestPublicationDirective):
<wgrant> +
<wgrant> +    priorty = Int(required=False)
<wgrant> I don't think that's how one spells 'priority'. Does this mean that interface doesn't actually do anything?
<mwhudson> wgrant: oops
<wgrant> Heh.
<mwhudson> yes, the interface is in fact not required
<mwhudson> wgrant: want to review a branch in a sec? :-)
 * wgrant isn't a reviewer yet...
<lifeless> wgrant: we should fix that
<lifeless> Ursinha: still up ?
<Ursinha> haha yes sir
<Ursinha> I wanted to talk to you
<lifeless> when should we use the oops tag? (bug 607879)
<_mup_> Bug #607879: https://bugs.edge.launchpad.net/~person/+participation timeouts <timeout> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/607879>
<Ursinha> about a timeout bug
<lifeless> oh oh, I'm in trouble ;)
<Ursinha> lifeless, timeout implies oops
<Ursinha> so it's redundant to tag timeout and oops
<lifeless> Ursinha: So, I was following https://dev.launchpad.net/PolicyAndProcess/ZeroOOPSPolicy
<Ursinha> lifeless, and I love you for that, but let me read the page
<Ursinha> lifeless, "It should be tagged with either 'oops' or 'timeout' on it."
<lifeless> Ursinha: I'll stop using both on the same tag
<Ursinha> that?
<lifeless> yeah, i think its a little subtle
<lifeless> I mean, to me the value in a oops tag is that one can look at the tag and get all oops related bugs
<lifeless> but if it causes some script/reporting issue its obviously better to have only one, which is what I'm guessing is the issue?
<Ursinha> lifeless, is it hard to get bugs tagged oops and timeout at once?
<Ursinha> I mean, the two sets
<Ursinha> lifeless, I was pondering about having only one tag to fetch all oops bugs, but I guess lp lets you query bugs with one or another tag very easily
<lifeless> Ursinha: well, theres quite a few tagged both before my filing efforts yesterday
<lifeless> it would be easy to clean the data up
<Ursinha> really? why?
<lifeless> I don't know, it was just like that
<lifeless> (I think)
<lifeless> <- 0536, not 100% awake yet
<Ursinha> :)
<lifeless> anyhow - I'll check and fix today, thats easy.
<Ursinha> lifeless, that's ok, not that critical, just trying to be consistent as I know that some scripts that fetch stats consider them separately
<lifeless> yup
<Ursinha> lifeless, about the timeout bug
<lifeless> I'm not panicking on it, I'm just rectifying something I've made a little worse.
<lifeless> easy
<lifeless> ok, about the bug - which one ?
<Ursinha> finding the number, just a min
<Ursinha> lifeless, bug  607879, that you filed for +participation timeouts
<_mup_> Bug #607879: https://bugs.edge.launchpad.net/~person/+participation timeouts <timeout> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/607879>
<lifeless> ok
<Ursinha> I see that now this bug is linked to another oops, the most offender in timeouts list, like https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1662A100
<lifeless> I think thats the oopstools confusion
<lifeless> I put the link in on a more suitable oops, I'm sure
<lifeless> yeah, I did - the ones in the bug at the top
<Ursinha> I wanted to know if they're related somehow, of if the correct bug for this oops is bug 424671
<_mup_> Bug #424671: attachments: accessing attachment's message is very slow <api> <performance> <Launchpad Foundations:Triaged by leonardr> <https://launchpad.net/bugs/424671>
<lifeless> they are only related by being 'timeout' errors with 'select' in the exception value
<lifeless> e.g. not related at all
<Ursinha> right
<Ursinha> that bug in oops-tools is really annoying
<lifeless> yes
<Ursinha> the timeouts count is ultrahigh, I guess that bug needs some more attention
<lifeless> gmb had a fix in the pipeline
<lifeless> but went on leave
<lifeless> and now deryck is too
<lifeless> is bryceh at the rally? or brian? I can ping them
<bryceh> lifeless, nope we're both in PDX
<lifeless> ok, I might see about moving it forward today
<Ursinha> cool lifeless
<lifeless> perhaps you can pick it up my afternoon and drive it? its kindof killing leanne :)
<Ursinha> thanks
<Ursinha> lifeless, 'you' me or bryceh?
<lifeless> bryceh
<lifeless> or brian
<lifeless> now where did the branch go
 * bryceh needs to learn to keep his fat mouth shut
<Ursinha> haha
<Ursinha> lifeless, the bug that should be fixed in oops tools is bug 461269
<_mup_> Bug #461269: oops reports should be grouped by oops signature not exception type and exception value <OOPS Tools:Triaged> <https://launchpad.net/bugs/461269>
<Ursinha> but I guess I already told you that
<Ursinha> this bug is kinda critical to our ZeroOopsPolicy to happen :/
<lifeless> is matsubara's branch insufficient?
<Ursinha> lifeless, no idea. he'll be back today and I'll ask him.
<lifeless> Ursinha: revno 11156 in devel
<lifeless> Ursinha: should fix that attachments thing
<Ursinha> lifeless, cool
<lifeless> so I'll test it on edge and if its better there, organise a CP
<lifeless> meep: http://www.zdnet.com/blog/security/dell-ships-motherboard-with-malicious-code/6901?tag=nl.e589
<wgrant> Awesome...
<lifeless> bryceh: http://pastebin.com/GTc6Y1LY
<lifeless> bryceh: I have /no/ idea how I seem to have triggered these failures in bugs
<lifeless> I haven't changed templates
<Ursinha> I'm going to sleep a bit
<lifeless> gnight
<Ursinha> bye lifeless
<lifeless> wgrant: or perhaps you have some idea?
<lifeless> wgrant: oh oh, excellent you're up - you wouldn't happen to be willing to qa something for me ?
<wgrant> lifeless: Sure.
<wgrant> Also, looking at that traceback now...
<lifeless> wgrant: make an api call to https://api.launchpad.net/1.0/bugs/414746/attachments - it should boom; make the same to edge - if it works, we have a CP we need to do.
<_mup_> Bug #414746: speakers cannot be muted when using headphones regression (karmic) <apport-bug> <apport-collected> <i386> <regression-release> <pulseaudio (Ubuntu):Confirmed> <https://launchpad.net/bugs/414746>
<wgrant> Most of the diff is crap -- the problem with the first and last is that the mailto: stuff is missing.
<lifeless> ok
<lifeless> well thats due to having ubuntu as a stopword now, I think
<lifeless> so I'll just nuke that
<wgrant> Ah.
<wgrant> lifeless: Timed out on edge for me.
<lifeless> darn
<wgrant> OOPS-1664ED389
<wgrant> Worked the second time, though.
<lifeless> ok, give that a few minutes and I'll get another bug up
<lifeless> well, thats an improvement
<lifeless> != fixed though
<wgrant> Still fails on production.
<lifeless> gawd yes
<lifeless> lib/lp/bugs/tests/../stories/guided-filebug/xx-sorting-by-relevance.txt is still naffed, hmm
<wgrant> Right, I'm not sure about that one.
<lifeless> turns out it was one extra row missing or so
<lifeless> I'm going to do a follow up patch to remove all filtering of terms
<lifeless> losa ping
<mthaddon> lifeless: can you give us a few - in a review meeting atm
<lifeless> of course
<lifeless> take as long as needed
<mthaddon> thx
 * wgrant is also interested on when Lucid's happening.
<wgrant> Since there's a dpkg issue that's fixed there.
<lifeless> \o/ rockstar your bugs all came through qa bot just now ;)
<lifeless> flacoste: are you still around ?
<lifeless> flacoste: if so, could you, if you agree, move rt 40477 and 40480 both to pri 88, and drop rt 40403 to pri 87, and nuke rt 31058
<mrevell> Hey
<lifeless> hi mrevell
<mrevell> Hey there lifeless
<mthaddon> lifeless: what did you need?
<lifeless> I wanted to see if I could help with the slony packages thing
<lifeless> either by doing, or perhaps talking with the server team folk here who are keen to offer assistance for backports and things like that that we need
<maxb> did my backports not work?
<mthaddon> lifeless: ah, spm (who's just EODed) is working on that one, and I believe have made some better progress on it today
<lifeless> maxb: I think they did, but its a ppa and versioning + availability of old builds trickiness
<lifeless> maxb: or something like that
<lifeless> mthaddon: ok.
<mthaddon> lifeless: although the lucid buildbot instance has been failing consistently for a few days now (had meant to chase someone about that) which would be a problem for lucid upgrades
<lifeless> mthaddon: well if you need help of any sort; please let me know :- on the packages side Jos has made it clear that he considers his team are here to support us; on other sides - well, I have the odd trick up my sleeve
<lifeless> mthaddon: I believe mars looked at that overnight; will see if there is a bug and if not make one.
<mthaddon> cool, thx
<spm> lifeless: before I go eat, and despite my EOD'd nature; yeah. it's just been a tad fiddly, and a bit of a learning experience for me. I am *hopeful* that the next submissions (early tomorrow I hope) will succeed, and then progress will occour.
<lifeless> spm: thanks!
<spm> the most recent build was a fail, becuase it built the pg 8.3, with a dependancy on pg 8.4. which is um... bad.
<lifeless> spm: the packages, or bb ?
<lifeless> stub: around ?
<stub> physically, yes.
<lifeless> whats he wiki page that the lp dependency deb updates is documented on ?
<lifeless> seems i doesn't mention doing hardy as well as lucid atm
<lifeless> brb
<stub> LaunchpadPPA
<rinze> gmb: Hi
<gmb> rinze, Morning.
<gmb> hurrah for /whois
<rinze> gmb: There's a buildbot failure for db_lp in bugtask-search.txt. It looks spurious but caused by the fact there is no sorting done before printing.
<rinze> gmb: I was wondering if you could confirm that - https://lpbuildbot.canonical.com/builders/db_lp/builds/971/steps/shell_7/logs/summary
<gmb> rinze, Yeah, that looks to be the case.
<gmb> Thought we'd fixed all of those.
<gmb> rinze, Want me to take care of it or are you already in the process.
<rinze> gmb: I was already in the process of getting a branch up.
<gmb> rinze, Okay, cool.
<rinze> gmb: Reading the doctest in more detail I'm wondering if simply sorting wouldn't defeat the test - it asserts that bug #9 is on top of the list.
<_mup_> Bug #9: Rosetta's po parser is too strict <Launchpad Translations:Fix Released by carlos> <https://launchpad.net/bugs/9>
<gmb> Gaaargh.
<rinze> gmb: Does the order not matter for the other bugs ?
<gmb> rinze, From reading the test I think not. But it's spectacularly unclear.
<gmb> *why* is it printing out bugtask ids? Hell's teeth.
<gmb> rinze, So, yes, sorting would defeat the object of the test
<gmb> Because the test is that things are sorted properly.
<gmb> So there's some bigger underlying cause here.
<rinze> gmb: Any chance you could take over? I can handle adding a trivial sorted() call somewhere, but this looks more complex.
<gmb> rinze, Sure.
<lifeless> gmb: oh hai
<lifeless> gmb: aren't you on leave?
<gmb> lifeless, Yesterday I was. Working today and tomorrow then I'm on leave for two weeks.
<lifeless> qargh
<lifeless> testfix ?
<rinze> lifeless: https://lpbuildbot.canonical.com/builders/db_lp/builds/971/steps/shell_7/logs/summary
<lifeless> hmm
<lifeless> gmb: tell me about that ^
<lifeless> I have a branch changing bug stuff, so I can put a fix in too
<gmb> lifeless, I'm trying to look at it now but it's taking an age to update my local copy of db-devel.
<gmb> lifeless, It looks like something broke with the ordering of search results. Will know more shortly, hopefully.
<lifeless> they are ordered by fti
<lifeless> aren't they ?
<lifeless> gmb: I'm about to land my malone branch - it passed ec2land
<lifeless> oh, so heat is intended too
<lifeless> why does db-devel builds break devel ?
<gmb> lifeless, The test that's failing tests that ordering by '-number_of_duplicates' works properly. Which it apparently doesn't any more. Can't say why, yet.
<gmb> lifeless, I think it's a stop-the-line measure. One builder breaks, everything stops.
<gmb> (Which is why trying to get my CP for initial_message landed was such a ballache)
<lifeless> gmb: which, btw, isn't enough.
<gmb> lifeless, Yeah, we figured as much on Tuesday IIRC.
<lifeless> also
<lifeless> holy fuck searchTasks
<lifeless> rev 11188 is a candidate
 * gmb looks while make schema runs
<lifeless> its changing duplicate bug logic
<gmb> AAAAA.
<gmb> Yep.
<lifeless> cause I'm keen to get this landed, I'd love to put a one-liner in my branch and shove it through ;)
<gmb> shame deryck made duplicateof readonly :)
<gmb> I wonder whether this is as simple as a db flush that needs to happen.
<gmb> Mind you, there's already a flush_database_updates() call in there. Hmm.
 * gmb adds a transaction.commit() for good measure
<lifeless> how dnah
<bigjools> urgh, death to commit in tests
<gmb> lifeless, yeah, didn't work anyway :)
<lifeless> doubt that thats it - you're calling into model code directly.
<gmb> True.
<wgrant> bigjools: The master/slave split means we have to, though...
<wgrant> bigjools: I wish there was a way around that.
<bigjools> there isn't
<wgrant> Damn.
<bigjools> we still have a lot of tests doing commit() unnecessarily though
<wgrant> Well.
<gmb> Agreed.
<bigjools> I'm considering the idea of having another master store for derived distros actually
<wgrant> There are two reasons I know of: 1) librarian 2) slave store
<wgrant> Are there any other reasons left?
<bigjools> not that I know about
<bigjools> we can fix (1) in tests
<wgrant> How?
<bigjools> make the file available in the librarian right away
<lifeless> you may be misunderstanding the issue there
<wgrant> I guess.
<lifeless> although, read committed may workaround it
<bigjools> the issue is that when we use slave stores to read data, we have to commit to get them to sync
<bigjools> and I think that's a good thing to still have to do in tests
<wgrant> Apart from the fact it's slow.
<bigjools> indeed
<gmb> lifeless, Ah, interesting. Bug 10 is marked as a dupe of bug 9, but bug 9's number_of_duplicates is still zero. It should be 1; that's why the sorting's off.
<_mup_> Bug #10: It says "displaying matching bugs 1 to 8 of 8", but there is 9 <Launchpad Bugs:Invalid> <https://launchpad.net/bugs/10>
<_mup_> Bug #9: Rosetta's po parser is too strict <Launchpad Translations:Fix Released by carlos> <https://launchpad.net/bugs/9>
<_mup_> Bug #9: Rosetta's po parser is too strict <Launchpad Translations:Fix Released by carlos> <https://launchpad.net/bugs/9>
<gmb> Shut up you stupid bot
<lifeless> gmb: so the aggregation is failing
<lifeless> is number_of_dupes a cached value - either db or model ?
<gmb> lifeless, Well, I think that it's a trigger that's failing to be run for some reason.
<lifeless> gmb: or a bug in the trigger :P
<gmb> I'm checking now.
<gmb> Yeah.
<lifeless> yeah ?
<gmb> lifeless, The triggered function *looks* fine (database/schema/trusted.sql:917)
<gmb> lifeless, That was a sort of "yes, that's possible" noise.
<gmb> I can't express "ugnh" properly in textual form
<lifeless> no, its wrong
<bigjools> stub: what do you think to the idea of having a new master store for derived distributions?  I am concerned about the load they'll put on the DB when doing everything a distro needs to do.
<gmb> lifeless, Really?
<lifeless> gmb: totally
<gmb> lifeless, Maybe I'm just not caffeined-up enough yet. Why?
<lifeless> bigjools: I'm keen on sharding/partitioning; I think you'll find that that isn't going to split out terribly well with our current setup; also I think the load isn't a huge concern in the short term - we are write limited yes, but we can reduce IO massively by fixing a few bugs.
<lifeless> gmb: because
<lifeless> gmb: voice?
<lifeless> I really want a whiteboard for this
<bigjools> lifeless: why do you think it won't split?
<bigjools> the write load that Soyuz generates is huge, BTW.
<lifeless> because you have links throughout the system that are doing FK's
<lifeless> and slony isn't a multi-master system
<gmb> lifeless, Sure. Though be warned, I'm in a coffee shop, so there may be bean-grinding and child noises.
<wgrant> Why is the write load so huge?
<wgrant> The read load is certainly massive.
<gmb> lifeless, Skype, mumble?
<lifeless> gmb: skype; id ?
<bigjools> publisher for starters
<gmb> lifeless, binnsgm. Starting it now.
<stub> bigjools: It might actually make things better. At the moment, we have 31 distributions but almost the entire database is linked to just 1 (Ubuntu). This can give us some wonky query plans.
<wgrant> bigjools: How's that write-heavy?
<wgrant> It just touches a couple of fields on a few publications.
<bigjools> a lot of publications
<wgrant> It's going to be doing less than the thing that sends bug notifications.
<bigjools> the scripts also mostly use the master DB out of necessity
<wgrant> True.
<bigjools> which is part of what I meant by write load I guess
<wgrant> The publisher doesn't need to be as bad as it is.
<bigjools> undoubtedly!
<bigjools> I'd like to pipeline it
<wgrant> We need to work out what to do about index generation.
<wgrant> Everything else is pretty quick.
<wgrant> (apart from source domination, but that's a trivial fix)
<bigjools> for indexes we can just append to Packages/Sources
<wgrant> But what exactly do you mean by pipelining?
<bigjools> the opposite of batching like we do now
<wgrant> I considered that... but we have to regenerate them some time.
<elmo> bigjools: gah, that's such a bad bad bad tradeoff
<bigjools> yes, we can do it occasionally
<bigjools> elmo: why?
<elmo> you're trading CPU time on one server (ours) for increased download time for millions and millions of users
<bigjools> so then we also need incremental updates - isn't there something that does that already?
<elmo> yes, but it's crack and also pessmizes things for users
<wgrant> pdiffs aren't really awesome.
<elmo> but I also don't buy the argument that the answer to 'append sucks' is incremental updates
<elmo> the answer is 'don't do append'
<bigjools> why?
<elmo> because it's not expensive to actually write a packages file in a non-append fashion
<elmo> apt-ftparchive is slow, yes, that doesn't mean doing non-append packages file generation will always be slow
<wgrant> At the moment it is. But I should work out how to effectively benchmark my caching experiments from earlier in the year.
<bigjools> someone told me that a-f is actually quick, it's generating its input files that's slow
<elmo> bigjools: they're lieing
<wgrant> Both parts are slow.
<elmo> and/or confused and/or stupid
<wgrant> The publisher logs make that clear.
<elmo> or some combination thereof
<bigjools> where that someone was Celso after he wrote the PPA publisher
<wgrant> Generating input files was much slower.
<elmo> yeah, and I've had this argument before with Celso
<bigjools> ok
<wgrant> But cprov apparently cut that time a lot just before he left.
<lifeless> gmb: http://pastebin.com/FUgp9kHc
<lifeless> gmb: http://pastebin.com/wWaziqcT
<bigjools> we could probably sit down and analyse the publisher at some point, but it's not worth it right now
<elmo> the Packages file is 26Mb; I utterly fail to believe that's something so expensive to update inline that we need to penalise our users to avoid the cost
<elmo> sorry, a lucid i386 universe Packages file
<elmo> I appreciate there's lots of them, and they add up
<bigjools> yeah, you do notice the slowness when you're not on a fast link
<lifeless> I would comment, but ETESTFIX
<wgrant> a-f file list generation appears to only take ~2 minutes now. The a-f run is something like 15 minutes. :/
<bigjools> wgrant: :o
<bigjools> I've not looked for ages
<bigjools> I do know that PPA publication is >10 minutes now :/
<bigjools> it used to be about 3
<elmo> a-f is fucking around inside a *massive* berkley DB file as part of this
<elmo> it's not fair to compare a-f with switching to an append mechanism
<elmo> a fair comparison would be a non-a-f, sane inline mechanism with an append mechanism
<wgrant> bigjools: what's taking most of that time these days?
<bigjools> by sane inline, you mean to replace entries when new versions appear?
<bigjools> and append new stuff
<bigjools> wgrant: no idea
<bigjools> I've not analysed it
<elmo> bigjools: yeah, and without the a-f DB madness overhead
<bigjools> I've just seen the cronjob blocking as it's every 5
<bigjools> elmo: ok, sounds, err, sane :)
<bigjools> if we can get the PPA publisher to do the extra overrides, a-f can go
<bigjools> IIRC
<wgrant> Well, and it has to be an order of magnitude faster.
<wgrant> But extra overrides is the only missing feature, I believe, yes.
<bigjools> yes - but that's going to be relatively trivial, we have a few people who know how to make Python code faster ;)
<elmo> wgrant: *blink* - why does it have to be?
<wgrant> elmo: Because it's currently *really* *really* slow.
<wgrant> It took roughly 7 minutes locally for a 30000-publication index, IIRC. With hot caches.
<wgrant> (that's a single component-pocket-DAS index -- not all of them)
<elmo> wgrant: oh, faster than it is now, you mean
<elmo> wgrant: I thought you were setting the benchmark as an order of magnitude faster than a-f
<elmo> which seemed harsh
<wgrant> elmo: Ah, no, sorry.
<wgrant> But it should be :)
<wgrant> I can do a full pocket of indices of that size in under 2 minutes, though.
<wgrant> Hmm. Why should main archives automatically build everywhere?
<wgrant> Isn't that what we're about to start *not* wanting, once we have derivative distros?
<lifeless> gmb: want some brain fuckage ?
<gmb> lifeless, Always.
<lifeless> I've switched to db-devel
<lifeless> done make schema
<lifeless> guess what happens when I run :!bin/test -vv -t 'duplicate_handling' -t doc/bugtask\\.txt
<gmb> lifeless, I'm going to put 50p on "bad things"
<lifeless> it works
<gmb> D^F&*QY(EUH!
<lifeless> what pg does db-devel buildbot use ?
<gmb> Good question. IDK.
<lifeless> losa ping ^
<mthaddon> lifeless: 8.3 - it's on hardy
<gmb> Interestingly, fails for me locally on... 8.3.9.
<lifeless> \o/
<lifeless> gmb: ok, try this patch and see if the unit style test passes
<lifeless> http://pastebin.com/W06ZjYMg
<lifeless> :!bin/test -vv -t 'duplicate_handling' -t bugs/doc/
<lifeless> is how I'm running the tests
<lifeless> 7 tests run for me
<lifeless> mthaddon: thanks
<bigjools> wgrant: the "main archives automatically build everywhere" is so that the restricted ARM stuff doesn't require manual SQL to be run, until we do a similar thing for derived distros as we've done for PPAs.
<bigjools> but basically it can be controlled via DAS anyway so it's not really an issue
<bigjools> wgrant: did you see the LEP for derived distros BTW?
<wgrant> bigjools: Given that there are only two main archives, it seems like a pretty bad hack to prevent people from having to check a couple of checkboxes..
<wgrant> bigjools: Is Ubuntu in scope for that LEP?
<bigjools> wgrant: it's not a hack, it;s maintaining the status quo
<bigjools> because the recent change buggered it up for main archives
 * gmb waits for tests to run.
<wgrant> Perhaps.
<gmb> lifeless, That passes.
<bigjools> wgrant: sort of, I have Debian NSS in mind
<lifeless> gmb: ><
<lifeless> gmb: so, when you said you triggered the failure locally ... ?
<wgrant> bigjools: OK. I just think it would be a really good idea to keep that in mind before designing a system that might be incompatible...
<gmb> lifeless, The failure that jelmer originally found in the build.
<lifeless> cause my patch is clearly just trying to trigger the damage
<lifeless> gmb: yes, but how
<lifeless> how do we make it go boom
<gmb> lifeless, bin/test -vv -t bugtask-search.txt
<gmb> Worked for me befoew
<gmb> Let's see if it still goes boom...
<gmb> lifeless, http://pastebin.ubuntu.com/467453/
<gmb> Still
<gmb> lifeless, And adding the number_of_duplicates count, of course, gets us http://pastebin.ubuntu.com/467457/, as expected.
<lifeless> yeah
<lifeless> you'll need a commit to get them
<gmb> Arsemonkeys.
<gmb> Question is: where?
 * gmb plays
<lifeless> gmb: after markAsDuplicate
<lifeless> we have lunch nazis happening
<lifeless> will try to assist downstairs
<gmb> lifeless, That's what I thought, but it seemed not to work.
<gmb> lifeless, Much appreciated.
<gmb> Ah, that's intriguing and frustrating. Even doing the update in the db seems not to work.
<gmb> Ahah, no, it works.
<gmb> But I managed to set number_of_duplicates to -1.
<gmb> FAIL.
<StevenK> gmb: "This bug does not exist in any other bug tracker, in the world!"
<gmb> StevenK, I like how your mind works.
<gmb> And I use the word "works" quite wrongly.
<lifeless> gmb: right, trigger failing
<lifeless> gmb: check markAsDuplicates, it *can't* read from the DB mid-transaction
<lifeless> without hilarity
<gmb> I confess, I'm starting to wonder at what point we can drag this revision out and shoot it.
<gmb> lifeless, Bingo!
<gmb> If you comment out the bug heat code after the try...except in markAsDuplicate() it works.
<gmb> So essentially by modifying the bugs' heat we're clobbering the number_of_duplicates.
<gmb> Or something.
<gmb> lifeless, http://pastebin.ubuntu.com/467465/ fixes the doctest. However, I can't comment as to the rightness of the solution.
<wgrant> bigjools: Why is the 'Publish' flag editable by users?
<wgrant> It causes great confusion, and is of dubious utility.
<bigjools> so copy archive owners can turn them on and off, mainly
<bigjools> who is confused?
<wgrant> We'll occasionally get someone in #launchpad complaining that their PPA isn't publishing.
<wgrant> There's no indication anywhere except +edit that the flag is switched off.
<wgrant> And they never know how it got turned off.
<lifeless> gmb: yes
<lifeless> let me look
<lifeless> gmb: oh, thats not safe
<lifeless> we can't commit in model code
<lifeless> stub: around ?
<bigjools> wgrant: we need a big warning on the front page to say it's turned off then
<stub> lifeless: yer
<wgrant> bigjools: Probably.
<gmb> lifeless, Yeah, I was pretty sure that was the case.
<lifeless> stub: we have a trigger interaction
<lifeless> stub: the number_of_duplicates trigger is being broken by an update which separately sets the value in the master, when we move multiple bugs to a new master in one xaction
<gmb> lifeless, The simple solution may be to have the function that updates the number of duplicates to also update the bug heat for the duplicated bug.
<lifeless> stub: any input you have would be useful
<stub> Do you have an exception?
<stub> hmm... no... i see the problem.
<lifeless> apparently there may be some storm thing which can help
<lifeless> I'm guessing storm is adding an update to that column because we read it
<lifeless> I'm going to hunt him down
<stub> So when we sent a duplicate, we need to send a message to the messaging system so the update can happen post-transaction... oh... wait...
<lifeless> yes yes
<lifeless> if you wanted to build the messaging dependencies package for hardy, I think tom would install it
<maxb> So, now devel is py2.6 capable.... should we delete the py2.5 rebuilds from the launchpad PPA for lucid? Or is there worth in retaining them? What about other series? [Mainly just gathering some initial thoughts, I'll send a mail later]
<stub> lifeless: works for me.
<wgrant> I suppose we'll need to port to 2.7 in a few months :(
<maxb> Lucid first, before we worry about Maverick
<wgrant> Maverick's not having 2.7 by default, fortunately.
<wgrant> That's for Nifty.
<maxb> woo
<stub> lifeless: https://pastebin.canonical.com/34956/
<maxb> problem. brush. carpet. :-)
<wgrant> lifeless: What's the issue with raising an AssertionError?
<wgrant> It's not a valid way to call the method.
<maxb> boo. +copy-packages timeout. I should be able to copy 21 source packages at once, right? :-)
<lifeless> stub: whats that pastebin demonstrating?
<stub> Changing the master for multiple bugs in one transaction
<lifeless> stub: oh sure, the transactions succeed
<lifeless> gmb: get a postgresql / storm trace of the calls being made
<lifeless> gmb: hunch is that we're seeing an 'update ...' call setting that column
<lifeless> gmb: also are we gaining one, or losing one, or just failing to update?
<gmb> lifeless, I'm not sure. Give me a sec; just testing a theory then I'll get the PG trace.
<lifeless> kk
<lifeless> wgrant: are the inputs invalid - ValueError
<bigjools> maxb: Yes. :/  I'm going to move package copying to the job system over the next few months
<wgrant> lifeless: That's forbidden.
<wgrant> lifeless: https://dev.launchpad.net/Hacking forbids raising TypeError, NameError or ValueError.
<maxb> bigjools: no more synchronous feedback? :-(
<lifeless> raising AssertionError is as bad or worse.
<lifeless> I'd rather you raise ValueError than assertion error, or a custom exception
<wgrant> lifeless: But it's not forbidden by the documentation :(
<wgrant> win 2
<lifeless> wgrant: best practice is historical not live ;)
<bigjools> maxb: we'll try and do it via ajax polling
<bigjools> landscape has a similar thing already
<bigjools> the problem is that the webapp request is really not the right place to do the hundreds of checks required when copying :(
<maxb> That could work. Though, it'll be irksome for people who promote PPA packages via the API
<bigjools> yeah, but... I dunno what else to do here
<wgrant> Why is copy checking so slow?
<bigjools> because it does a shedload of db queries
<bigjools> mostly unavoidable
<gmb> lifeless, stub: So, this patch http://pastebin.ubuntu.com/467471/ (includes lifeless's original patch) fixes our problem by moving the bug heat stuff into the set_bug_number_of_duplicates function. Which feels like the wrong place for it, but only because of the name of the function.
<lifeless> stub: https://dev.launchpad.net/LaunchpadPpa?action=diff
<maxb> lifeless: two versions of what?
<lifeless> the packages
<gmb> Argh. Annoyingly, fucks up the heat tests. But I think that's my fault...
<lifeless> hang a sec
<wgrant> We still support Karmic, don't we?
<maxb> lifeless: It's not clear to me what that paragraph is saying. We *already* sustain hardy/karmic/lucid in the PPA
<maxb> we haven't officially de-supported jaunty yet
<maxb> though it's likely bitrotten
<lifeless> maxb: ok, then I was confused; it seemed to me that it wasn't clear that people needed to do this.
<wgrant> I think it may have bitrotten a bit, though.
<wgrant> Right.
<lifeless> maxb: we need a hardy build of launchpad-messagequeue-dependencies
<gmb> Oh, wait, that might be because I forgot to make schema. Durr.
<maxb> lifeless: Yes, someone has ignored point 11 under the list at the top
<lifeless> maxb: ok, nuke my change then
 * maxb expands it a bit instead
<wgrant> lifeless: So, what should I do about that exception?
<lifeless> wgrant: in the review I said you could just document that it sucks
<lifeless> maxb: if you were to also refresh the builds, I'd be hugely appreciative.
<stub> lifeless: The comment sounds sane, but it is worth documenting the commands so people like me can actually do it.
<lifeless> stub: ack
<lifeless> stub: I can show you the error now
<wgrant> lifeless: Ah, true.
<lifeless> https://pastebin.canonical.com/34957/
<lifeless> stub: ^
<lifeless> wgrant: I'm usually pretty careful to not block people; please shout if you feel blocked - its probably a miscommunication
<lifeless> http://pastebin.com/sMtQapT4 <- same sql trace
<lifeless> gmb: I'm almost totally convinced this is a storm bug
<gmb> lifeless, Ah, right. Still want me to grab that storm trace?
<jkakar> Hi lifeless! :)
<lifeless> gmb: yes
<lifeless> gmb: or postgresql
<lifeless> jkakar: hi
<gmb> Right. On it.
<lifeless> jkakar: http://pastebin.com/sMtQapT4
<lifeless> jkakar: this shows the problem
<stub> lifeless: right, so the db end is fine and something else is issuing updates it shouldn't, such as storm perhaps using the version from the start of the transaction.
<lifeless> jkakar: we're getting confirmation that it actually happens
<lifeless> stub: I think so, yes.
<jkakar> So yeah, therve and I were wondering if it's a flush issue...
<stub> We can issue a foo.duplicate_bug_count = <mumble> to force it to reload that value from the database
<stub> (I forget the 'magic reload' object)
<jkakar> AutoReload.
<lifeless> stub: won't work
<lifeless> stub: because the trigger is an after trigger
<jkakar> Anyway, that might be hack fix if it worked.
 * jkakar looks at some Storm code...
<lifeless> until we commit we don't know the new value
<wgrant> lifeless: I've added a comment. Could you please land that?
<lifeless> wgrant: not until we're out of testfix
<wgrant> lifeless: Fair point.
<stub> lifeless: after triggers fire after the statement, not at the end of the transaction
<gmb> lifeless, stub: Here's the trace, from just before the markAsDuplicate() call to just after flush_database_updates() in bugtask-search.txt: http://pastebin.ubuntu.com/467476/
<gmb> lifeless, stub: The last line makes interesting reading :)
<lifeless> jkakar: so - http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/annotate/head:/lib/lp/bugs/model/bug.py
<lifeless> stub: ah, doh.
<lifeless> stub: its been a while. sorry ;)
<lifeless> ok, thats a decent, if ugly, workaround
<lifeless> does AutoReload kick in at UPDATE() or will we need a flush as well ?
<lifeless> gmb: yes, i thought so
<stub> I don't know
<lifeless> jkakar: ^
<lifeless> gmb: so try a flush + this autoreload trick
<stub> flush will hopefully happen automagically
<jkakar> lifeless: AutoReload marks the column as "flush this the next time I access it".
<jkakar> lifeless: It doesn't perform an UPDATE.
<jml> are we in some sort of testfix mode?
<lifeless> jml: yes
<lifeless> see backlog, its a fun one
<lifeless> jkakar: so, it flushes everything, then reads back that column ?
<gmb> Where fun == 'I want to stab myself in the face'
<lifeless> jkakar: and if we don't access the column at all, will storm do it itself mid commit ?
* jml changed the topic of #launchpad-dev to: testfix mode. gmb working on it | Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<gmb> jkakar, Can you help me out with AutoReload magicks? I'm scannign scrollback but I see nothing I can grok.
<jkakar> lifeless: AutoReload flushes the store, not just the column.
<lifeless> sure
<jkakar> gmb: If you set a column to AutoReload it will cause a flush on next access.
<lifeless> but you see my concern
<lifeless> we *don't access* the column
<lifeless> storm might, we don't.
<jkakar> gmb: So, Bug.duplicateof = AutoReload will cause the store to be flushed the next time Bug.duplicateof is accessed.
<gmb> jkakar, Ah, right, gotcher. Crikey that seems nasty.
<jkakar> lifeless, gmb: It's not what you want, it's useful in other contexts.
<lifeless> ok
<lifeless> where do we import AutoReload from ?
<jkakar> So I think we need to add a thing to Storm to tell it to never include the column in an UPDATE.
<lifeless> jkakar: See, lifeless isn't nuts.
<jkakar> lifeless: from storm.locals import AutoReload (it's actually in storm.store, though).
<jkakar> lifeless: :)
<gmb> lifeless, storm.locals
<jkakar> lifeless: You are referring to yourself by name which is indicative of some kind of deep seated problem, no?
<lifeless> yes
<lifeless> gmb: already have store imported in this module :)
 * gmb was cargo-culting
<jkakar> Hehe
* lifeless changed the topic of #launchpad-dev to: testfix mode. gmb, lifeless,jkakar,stub working on it | Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<jkakar> Okay, I'll take a look at what it will take to add this functionality to Storm.
<lifeless> jkakar: oh, and it replaces IntCol ?
<jkakar> lifeless: IntCol?
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/tests/test_duplicate_handling.py", line 101, in test_duplicates_are_moved
<jkakar> Oh right, SQLObject compatibility...
<lifeless>     new_bug.number_of_duplicates = AutoReload
<lifeless> ForbiddenAttribute: ('number_of_duplicates', <Bug at 0xdaa9050>)
<jkakar> Yes, it replaces IntCol on the instance.
<jkakar> I guess the object is security proxied.
<lifeless> jkakar: the above is why I was sure we were not writing to the attribute
<jkakar> Cool.
<gmb> Stabby.
<lifeless> gmb: I think we should pull this branch
<gmb> lifeless, I agree.
<lifeless> gmb: its a deep fix, the queue is stalled.
<lifeless> gmb: r=lifeless on a pull
<gmb> lifeless, Right; on it. Is it DTRT to remove it from devel and let it trickle down or should i submit merges for both?
<gmb> (That could be interesting)
<james_w> could oops generated during a test be attached as details to the result?
<lifeless> james_w: yes please
<lifeless> gmb: devel with testfix
<james_w> they are just text?
<lifeless> gmb: then queue will unblock and next buildbot run should fix it
<lifeless> james_w: serialised as yes
<lifeless> utf8 actually I think
<james_w> readable and not too large?
<gmb> lifeless, Righto. Almost done; just checking that I'm not breaking anything else in a spectacularly obvious fashion.
<lifeless> they have i18n usernames in them
<lifeless> gmb: not AFAIK.
<lifeless> wgrant: stub: jml: ^ gmbs query
<stub> eh?
<lifeless> when fixing a testfix failure in db-devel
<wgrant> If you revert it in devel, it will filter down to db-devel after the next buildbot run.
<lifeless> is landing a testfix in devel sufficient
<wgrant> Oh. Hmm.
<wgrant> I don't know if that will kick PQM.. good point.
<stub> It might need a manual merge - I think the automatic stable->db-devel merges are disabled in testfix mode
<stub> But I don't know for sure.
<wgrant> But once the commit kicks it out of testfix, devel will get tested, and merged into db-devel, which will then be tested, and not land back in testfix.
<gmb> lifeless, stub: 'tis in the queue now for devel. We'll see what happens and if necessary I'll do a manual merge.
<gmb> wgrant, ^
<wgrant> Unless db_lp decides to run again before the next stable merge, I guess...
<gmb> Yes, that would be amusing.
<wgrant> For some rather more masochistic values of amusing than usual.
<jkakar> lifeless, gmb: I think there might be a problem in your application code (and also a problem in Storm).
<lifeless> jkakar: I'm sure :)
<gmb> jkakar, lifeless, I've emailed canonical-launchpad@ about the [testfix]; can you elaborate abut the storm bug on that list please? (This should probably move to launchpad-dev@, tbh)
<lifeless> ack
<gmb> Ta
<lifeless> gmb: will you save my test improvement somewhere?
<gmb> lifeless, I'll push it up to LP, along with my patch to move stuff into the stored procedure.
<lifeless> gmb: I'm not sure thats needed/right, FWIW
<gmb> lifeless, No, I know, but I'd like it for my own reference later should we need to do it.
<lifeless> gmb: the stored procedure won't run on the master you see
<gmb> lifeless, OIC.
<gmb> Okay.
<lifeless> gmb: (if you don't update it). And if you do update it, the problem exists.
<gmb> Right
<jkakar> gmb: Yep, I'm just figuring out what you need to determine if there's a bug in Launchpad.
<jkakar> Also, therve and I are looking at what it will take to fix Storm.
<lifeless> jkakar: I grepped, I can't see anything setting that oclumn in the entire codebase
<jkakar> lifeless: Hrm.
<lifeless> jkakar: come sit by me if you like.
<jkakar> lifeless: It might be an update issue in Storm... I found this:
<gmb> lifeless, lp:~gmb/launchpad/db-devel-with-lifeless-improvements
<jkakar>         # The fromdb check makes sure that values coming from the
<jkakar>         # database don't mark the object as dirty again.
<jkakar>         # XXX The fromdb check is untested. How to test it?
<lifeless> gmb: as long as you<->deryck coordinate on it, I'm happy :)
<jkakar> It could be that a flush is getting the triggered value and causing the object to be marked as dirty. :)
<jkakar> It may be that the issue in Launchpad teaches us how to test that code path. ;)
<lifeless> jkakar: hah.hah.hah.
<jkakar> lifeless: You said you have a failing test, yes?
<lifeless> yes
<lifeless> its awful, but yes.
<lifeless> I don't have a unit test yet. I think I can construct one though.
<lifeless> its a doctest [[meep]]
<jkakar> lifeless: http://paste.ubuntu.com/467498/
<jkakar> lifeless: Can you add a validator like in the paste to the duplicateof column and run your test please?
<lifeless> jkakar: number_of_duplicates, surely ?
<lifeless> duplicate_of is set in related bugs, heat is set in the bug with the wrong number_of_duplicates value
<lifeless> jkakar: no change in behaviour
<lifeless> jkakar: lets pair. I'm in the community room
<poolie> sinzui, hi, want to talk about feature flags?
<poolie> ãããã
<sinzui> :)
<poolie> :)
<sinzui> I think I need more caffeine  first
<poolie> np
<gmb> Okay, we should be out of testfix now.
* gmb changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<jelmer> gmb: Awesomeness.
<jelmer> thanks :-)
<wgrant> gmb: Thanks.
<wgrant> lifeless: Can you send my branch off now?
<gmb> lifeless did most of the heavy brainwork.
<poolie> gmb, lifeless, woo!
<jml> woot.
<jml> I need some help with bugs / bugbranches: http://paste.ubuntu.com/467503/
<jml> there's a doctest that checks to see that editing bugbranch links changes the date_last_updated of the bug
<jml> but the test is crummy. the unit test in the paste reproduces the logic of the doctest.
<jml> I guess I want to know if editing a bugbranch object is actually possible at all. if it is not possible, I don't really see why it should be tested / supported.
<wgrant> bigjools: FWIW, I just came upon a summary of a primary publisher run I made a few months back: http://williamgrant.id.au/f/1/2010/publisher-timeline.html
<wgrant> Gives a reasonable picture of how much time goes where.
<wgrant> ie. a-f takes 16 minutes.
<jtv> salgado: test -vvc -m lp.translations.windmill -t import_queue_entry
<jtv> ...but I had it with the bug_me_too test as well, which isn't in Translations.
<lifeless> \o/
<lifeless> wgrant: sorry, still analysing
<lifeless> wgrant: hunt down, oh, hml
<lifeless> we have a fix for storm
<jelmer> gmb: It looks like we're (back?) in testfix mode
<bigjools> lifeless: I also have a fix for Storm
<jkakar> Does anyone know which (if any) Storm branch LP is using?  I want to see what's different between it and lp:storm.
<jkakar> danilos: ^^ I think the Storm code being used by LP is from you, right?
<danilos> jkakar, it's 0.15 + one or two revisions from before 0.16 came out that I needed at the time
<jkakar> danilos: Right... is there a branch somewhere?
<danilos> jkakar, r342 that is :)
<danilos> jkakar, don't think so, it's available as a tar.gz inside download-cache for LP
<danilos> jkakar, that's bzr+ssh://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk/
<Ursinha> bigjools, oi
<poolie> sinzui, hi?
<sinzui> hi poolie
<poolie> hey i think i understand your concerns now
<poolie> i might just reply on the list, or we can talk here
<sinzui> here is nice
<bigjools> Ursinha: oi oi
<poolie> i think there are a few use cases worth looking at:
<Ursinha> bigjools, e ai? :) could you tell me if bug 595957 is really fix released?
<_mup_> Bug #595957: archive uploader tries to move the changelog to the current working directory <archive-uploader> <qa-needstesting> <Soyuz:Fix Released by abentley> <https://launchpad.net/bugs/595957>
<Ursinha> bigjools, it was qa-bad and then a commit pointed to it again
<abentley> Ursinha, I have been unable to QA it.
<bigjools> abentley: I can QA it on dogfood
<poolie> 1- adding a new non-trivial feature, that we want to have off by default until we're confident in it
<bigjools> Ursinha: it should be committed
<Ursinha> abentley, but is that fix released? I don't understand why is it fix released if that was bad and now has a new branch :)
 * sinzui nods
<abentley> bigjools, I don't think you can, because I think writing to that directory isn't forbidden on dogfood.
<Ursinha> oh, that was sinzui
<bigjools> abentley: easily fixed
<bigjools> besides, I can see if it leaves a changelog behind
<abentley> bigjools, okay, please do.
<bigjools> when I get faster interwebs back ... :/
<lifeless> bigjools: hi, what is your fix ?
<lifeless> jml: how does one tell if one is in testfix mode ?
<maxb> How odd. Three scripts in the launchpad tree still have a shebang with an explicit python version
<poolie> sinzui, so in that case we should have the code declare the feature flag
<bigjools> lifeless: it's a patch that strips ordering from SQLObjectResultSet.__nonzero__()
<poolie> and then it should check it in the necessary places with getFeatureFlag() (but not too many different places)
<bigjools> which makes bool(result) run quite a lot quicker.
<poolie> sinzui, then when we're ready to test on lpnet, we turn on the flag in that database
<poolie> for your own testing you'll set the flag in your own database
<poolie> also we'll provide a context manager to let you test with that flag turned on
<poolie> still here?
<sinzui> hmm
<lifeless> bigjools: is it ready to upstream ?
<sinzui> That sounds like a great way to discover bad interactions after the branch is on edge
<sinzui> I would rather have all feature on in dev and testrunner
<bigjools> lifeless: see bug 24620
<_mup_> Bug #24620: Install from DVD+RW complains of bad MD5 <linux-source-2.6.15 (Ubuntu):Fix Released by ben-collins> <https://launchpad.net/bugs/24620>
<bigjools> lifeless: no don't look at that one
<lifeless> we could - at some cost - run the test suite twice, once on, once off
<bigjools> look at bug 246200
<_mup_> Bug #246200: SQLObjectResultSet.__nonzero__() implementation does not strip result ordering. <oops> <tech-debt> <Soyuz:Invalid> <Storm:New> <https://launchpad.net/bugs/246200>
<lifeless> however I think its likely good enough to use testscenarios to run tests of affected areas only, twice.
<lifeless> sinzui: also
<bigjools> it's pissing it down here, I'll lose my GPRS connection shortly
<lifeless> sinzui: remember that we'll have QA happening before *any* rollout.
<lifeless> bigjools: looking
<sinzui> Once more I have to decline someone's nomination to fix a bug in the trunk series
<poolie> hm, if they're all on in dev, then we are making dev further away from production
<lifeless> poolie: production is ill defined
<poolie> s//edge
<lifeless> jkakar: https://bugs.edge.launchpad.net/soyuz/+bug/246200
<_mup_> Bug #246200: SQLObjectResultSet.__nonzero__() implementation does not strip result ordering. <oops> <tech-debt> <Soyuz:Invalid> <Storm:New> <https://launchpad.net/bugs/246200>
<sinzui> I should have used my epic time to restrict bug nominations to the handful of people who need them
<jkakar> danilos: Hrm... as a suggestion, branches are a good thing. ;b
<poolie> sinzui, which bug, for curiousity?
<lifeless> bigjools: if you can get a branch up and attached to that, that would be great.
<sinzui> https://bugs.edge.launchpad.net/launchpad-web/+bug/490058
<_mup_> Bug #490058: Launchpad shouldn't present itself as a database <launchpad-web:Triaged> <https://launchpad.net/bugs/490058>
<sinzui> ^ poolie
<poolie> tx
<bigjools> lifeless: yeah I've been meaning to do so for ages :)
<lifeless> bigjools: or even a patch
<lifeless> the storm folk will take it from there I suspect ;)
<poolie> nice idea though
<danilos> jkakar, they are, and they might be one, but if it's not on code.lp.net/~danilo/storm I probably didn't create it at the time
<jkakar> danilos: Yeah, it's cool, I was just teasing... I'll get the tar.gz and see what's different from trunk.
<sinzui> poolie, of course it is a nice idea. Most ideas about changing our pathetic excuse for a front page are good.
<poolie> "real artists ship"
<poolie> no point complaining about the speed other people do stuff
<poolie> anyhow coming back to the flags thing
<poolie> do you have time to talk about it now?
<sinzui> poolie. I weep everytime I see the contradictory root pages of each app and for each pillar. We have agreed to contradict each other and confuse the user. I feel helpless to fix an issue that requires half of the lp team to fix
<poolie> :/
<poolie> we'll get there
<lifeless> sinzui: is it the agreement between them all that is the hard bit?
<poolie> if we make it easier to change things (without making it too easy to change things rashly) we can fix this
<sinzui> We said that in 2.0 and we promised we would revisit the root page problem in the release after 3.0. We lied to ourselves
<poolie> istm the concept of an 'x.0' release is part of the problem
<poolie> while True: fix_something
<sinzui> jml brought up the issue recently. He may have used that as an argument for the web ui position he is hiring for
 * sinzui knows those pages are actually used 1.0 layours
<sinzui> layouts
<poolie> so i'd like to make sure i'm not going to run into avoidable trouble with flags
<lifeless> how do we reset testfix?
<poolie> can we talk about them?
<lifeless> there is nothing in pqm to change it
<lifeless> and we're still blocked
<lifeless> sinzui: jml: ^
<sinzui> lifeless, jml sees the pages as a problem. I jumped to his side when some disagreed about the inconsistency
 * sinzui looks for email to forward
<poolie> sinzui, i'd like to see a wall with printouts of screenshots of the N top pages
<poolie> would they make any sense viewed side by side?
<poolie> then you could do wireframes of proposed alternatives
<sinzui> poolie, I think jml made that. I will look for that email too
<lifeless> mars: around ?
<poolie> of course there's not really any necessary reason why there has to be a 'bugs home page'
<poolie> for the whole of launchpad
<poolie> it's really more of a consequence of having a domain per 'app'
<mars> lifeless, yep
<poolie> not that users really see them as apps
<mars> lifeless, what's up?
<lifeless> I'm trying to get us out of testfix
<lifeless> gmb landed a revert of the problem patch, I believe
<mars> ok
<mars> lifeless, if it isn't moving, and the patch is OK, gmb suggested merging into db-devel directly and then forcing the build
<lifeless> https://lpbuildbot.canonical.com/one_box_per_builder suggests a build is in progress
<lifeless> the wiki page says that when a build starts the flag is reset
<lifeless> is that true ?
<poolie> sinzui, lifeless, i think i'll write up some use cases in that thread
<mars> I don't know.  gary_poster, ^ do you know if the testfix flag is cleared when we land on devel?  The wiki says yes, but...
<poolie> and proceed with landing https://code.edge.launchpad.net/~mbp/launchpad/flags/+merge/30581
<poolie> if i can get ec2 to cooperate
<lifeless> mars: specifically 'when you start a build'
<poolie> then if you want to talk online or in mail later, we can
<poolie> i'm going to be offline monday/tuesday next week, travelling
<mars> lifeless, which wiki page?  (Shows you how much I know about it :)
<gary_poster> mars, if the problem was devel, then eventually, yes
<gary_poster> (I'm on call)
<lifeless> mars: https://dev.launchpad.net/Trunk
<lifeless> gary_poster: it was db-devel
<gary_poster> then no, landing a testfix on devel will not help db-devl, I don't think
<gary_poster> I mean, from testfix perspective
<lifeless> after it passes on devel, will the automerge to db-devel still kick in ?
<lifeless> if so, it will right itself eventually I guess.
<mars> lifeless, that is what I would expect
<mars> rule of least surprise and all
<lifeless> mars: I might do something a little cheeky then
<lifeless> and tell bb to build db-devel
<lifeless> to unblock us
<mars> land on db-devel an kick it?
<gary_poster> land it on both branches
<gary_poster> should merge fine
<gary_poster> get us out of testfix faster
<lifeless> gmb: can you toss it at db-devel then ?
<lifeless> gary_poster: looking forward to no testfix mode ? :)
<gary_poster> yeah :-)
<mars> I am starting to understand why it would be a nice thing to get rid of.
<lifeless> mars: pipeline stalls are painful; stop-the-line should be short and fast :- this isn't
<mars> Since I technically maintain that part, and I don't even know what wiki page to look at...
<lifeless> gmb: hi
<mars> policy in the wiki text is not code - buildbot does not execute wiki documents
<Ursinha> bigjools, is noodles on leave or something?
<mars> lifeless, "If only one branch is broken, a [testfix] to the broken branch that then makes the tests pass will reopen both branches for normal use."
<bigjools> Ursinha: no, he's en route from Prague to Berlin
<Ursinha> bigjools, oh, I see
<mars> lifeless, but again, just because the wiki says so does not mean code does same
<gmb> lifeless, Hi. What's up?
<lifeless> gmb: can you land the revert patch on db-devel too please !
<lifeless> gmb: I was about to start grovelling around for it to do
<gmb> lifeless, Sure. On it now.
<lifeless> thanks!
<lifeless> and sorry for the escalating pings ;)
<gmb> lifeless, No worries :)
<gary_poster> mars, heads up: bug 608593 may move from high to critical :-)
<_mup_> Bug #608593: buildbot lucid builds dying on xvfb <Launchpad Foundations:New> <https://launchpad.net/bugs/608593>
<mars> gary_poster, will do!
<jkakar> lifeless: https://code.edge.launchpad.net/~jkakar/storm/is-empty-strips-order-by/+merge/30677 <-- This is a fix for bug #246200.  Review appreciated (it's trivial). :)
<_mup_> Bug #246200: SQLObjectResultSet.__nonzero__() implementation does not strip result ordering. <oops> <tech-debt> <Soyuz:Invalid> <Storm:In Progress by jkakar> <https://launchpad.net/bugs/246200>
<lifeless> natching
<lifeless> jkakar: done
<jkakar> lifeless: Ta. :)
<poolie> sinzui, thanks!
<poolie> does 'martin' mean me or beuno? it sounds like me, but it could also be him
<gmb> Buggeration.
<lifeless> ah, the british are here.
<gmb> Looks like we're heading for another testfix folks; I'm on it now, though.
* gmb changed the topic of #launchpad-dev to: testfix will happen soonish, sorry; gmb is working on it. | Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<leonardr> do we have a helper function that encapsulates 'stick this query string onto this URL with a & if there's already a query string and a ? otherwise"? i can't find one in urllib or lazr.uri, but i feel like there is one
<sinzui> poolie, "martin" is beuno in the design conversations
<poolie> sinzui, it's a good thread
<poolie> wbn to redesign them all as a group
<maxb> lifeless: Regarding lp-mq-deps and hardy... Given hardy has no rabbitmq at all, and a drastically older erlang, .... do we get to move to lucid before rabbit?
<poolie> even if it wasn't all implemented at least getting some wireframes for "what this kind of page should generally look like" would be nice
<poolie> I like the idea of "some features only after 10pm"
<gmb> poolie, Just the man. How would I find out which revision a  sub revision (e.g. 1.2.3) belongs to?
<lifeless> log --show-ids
<gmb> poolie, I like your ventriloquism skills.
<gmb> lifeless, Ta.
<gmb> lifeless, Can I have your r= on http://pastebin.ubuntu.com/467585/? It turns out that we needed to revert r11190 as well as 11188, otherwise the build will stay broken.
<gmb> (Abel gave me the heads-up)
<lifeless> doit
<lifeless> now I'll look at the pastebin
<lifeless> in general
<lifeless> I don't think backouts need approval, do they ?
<gmb> lifeless, IDK; better to get at least an rs, I guess.
<lifeless> yeah
<lifeless> so it looks like a patch
<lifeless> which we don't want for now
<lifeless> seems like we should remove it:)(
<gmb> lifeless, Agreed :)
 * bigjools hi-5s jkakar
<jkakar> bigjools: :)
<jkakar> bigjools: Any other Storm issues you want me to fix while I'm in the mood...? :)
<bigjools> jkakar: my issues would most certainly take you out of that mood and into a completely different one
<lifeless> mthaddon: whats the losa email address again ?
<mthaddon> losas@canonical.com?
 * mthaddon just realised it's now a spam target
<lifeless> mthaddon: oh, sorry!
<bigjools> lol
<lifeless> mthaddon: should I not copy you on a public mail via it ?
<jkakar> bigjools: It'd be good to find out what they are... even if they do turn out to be soul destroying.
<bigjools> jkakar: ok - plz to be adding comments and docstrings to the storm code :)
<lifeless> bigjools: hah!
 * bigjools just destroyed jkakar's soul
 * jkakar explodes
<bigjools> jkakar: the only other thing that hurts my brain is the prejoin stuff but I doubt you can do much about that!
<jkakar> bigjools: Actually, free has a branch that he wants us to look at that does something like prejoin but I think in a less confusing way.
<bigjools> \o/
<poolie> jkakar, i was thinking of sending you a doc-adding patch but now i understand it the moment has passed :)
<jkakar> poolie: :)
<gmb> Okay, we're out of TF mode again. There's a chance we'll hit it again due to stuff landing after the latest build started; we'll jump off that bridge when it comes.
* gmb changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<poolie> benji/garry, if i want to create a FeatureController per request, should i just do that from beforeTraversal?
<benji> poolie: sounds good to me
<benji> poolie: do you mean IBeforeTraverseEvent?
<benji> because that gets generated once for each object being traversed, not once per request
<poolie> i actually just meant patching it into LaunchpadBrowserPublication.beforeTraversal
<poolie> which is where similar things seem to be
<benji> looking at the code, that looks like the right place
<poolie> it seems like there are notifications that could be hooked but i'm not sure that would be worth it
<benji> my opinion is that events are good if there are different configurations the app runs with; in this case there is only one desired configuration, so no need to make it pluggable
<poolie> thanks benji
<benji> np
<lifeless> poolie: see the profile hooked code - it does before & after hooks
<lifeless> poolie: berforeTraversal is not what you think it is
<lifeless> what rolls onto edge - stable, right ?
<mars> lifeless, yes, stable is deployed to edge
<lifeless> ok, so I has to be patient ;)
<lifeless> hopefully I will have a pleasant surprise in the morning
<lifeless> Ursinha: hi
<Ursinha> hi lifeless
<Ursinha> shoot
<lifeless> Ursinha: in the edge errors mail sent to the list, Errors for 2010-07-21
<Ursinha> hm
<lifeless> = Top 15 Time Outs (total of 52 unique items) ==
<lifeless> 54  SELECT COUNT(*) FROM Archive ...
<lifeless> there is no bug number
<lifeless> but if I go to the first oops
<lifeless> there is a bug on the oops
<lifeless> is there a bug on oopstools about this ?
<Ursinha> lifeless, I'm filing one now
<lifeless> same for the next group
<lifeless> Ursinha: thanks a lot
<lifeless> IMO this should be high, because its workflow support for the zero oops story
<lifeless> Ursinha: I was going to look at the bug in oops tools you mentioned, but we had a massive testfix failure today, so spent lots of time digging into storm with jkakar to fix that
<lifeless> sorry :(
<Ursinha> lifeless, that's ok, I know that's not abandoned
<Ursinha> lifeless, in fact I have a discussion about that bug with gary_poster today, and the proper fix isn't trivial. I have to ask matsubara which approach he chose in the branch that's linked to the bug
<Ursinha> *I had
<lifeless> ok
<lifeless> thats a little concerning: 116741.01launchpad-main-masterINSERT INTO OAuthNonce (access_token, nonce, request_timestamp) VALUES (%s, %s, %s) RETURNING OAuthNonce.id
<lifeless> 16 seconds to login, I guess
<lifeless> bryceh: the bug attachments API call that is slow is the single largest oops cause in the system
<lifeless> bryceh: just saying :)
<lifeless> gnight y'all
<lifeless> bryceh: if you want help on it, drop me mail; I'll see what I can do tomorrow.
<lifeless> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/424671 for reference
<_mup_> Bug #424671: attachments: accessing attachment's message is very slow <api> <performance> <timeout> <Launchpad Foundations:Triaged by leonardr> <https://launchpad.net/bugs/424671>
<bryceh> lifeless, yep quite familiar with it I am
<bryceh> would be wonderful to get that sorted out
<lifeless> -> zzz
<mars> Silly question: how do I set something other than the 'trunk' series to be the focus of development?
<mars> I've created my new series and pushed the code, but no setting I can find lets me say "This is the focus"
<mwhudson> mars: isn't it on the project +edit page?
<mwhudson> yeah, it is
<mwhudson> near the bottom
<mars> mwhudson, alright, thanks.  I lost edit priveledges on the project due to group assignment, so I missed it.
#launchpad-dev 2010-07-23
<wgrant> Are we out of testfix yet?
<jelmer> we were for a bit but I missed the window
<spm> wgrant: possibly. I've kicked/forced a bunch of builds, so we remain optimistic.
<jelmer> not sure if we're out of textfix again
<spm> hey jelmer!
<jelmer> 'morning spm!
<wgrant> Morning spm, jelmer.
<wgrant> Thanks.
<spm> hey wgrant
 * thumper is getting sad writing the current email
<thumper> I'm realising how far behind we are on bzr bits in LP
<jelmer> thumper, behind in terms of the version you mean?
<thumper> jelmer: yeah
<thumper> we are very behing in bzr, dulwich, bzr-git, bzr-svn and bzr-hg
<thumper> prolly bzr-loom too
<jelmer> thumper: well, the last three have pretty much stalled for the last couple of months
<thumper> there have still been a number of commits though
<mtaylor> rockstar: hey-ho
<thumper> mtaylor: I think rockstar is travelling
<mtaylor> thumper: dammit
<mtaylor> thumper: I need him to fix things in tarmac for me
<mtaylor> thumper: it's discarding tag info, even though we thought the other day that it shouldn't
<wgrant> We're out of testfix, then?
<thumper> wgrant: I'm trying to land
<thumper> I think the force builds take us out of testfix
<wgrant> Well, mwhudson landed something a few minutes ago.
<wgrant> So it seems to work.
<mwhudson> ah cool, that landed
<wgrant> Hm.
<wgrant> When did bin/test start spitting out per-test times?
<mwhudson> it's when you have -vvv
<wgrant> Maybe I just added more -v than usual.
<wgrant> Ah.
<wgrant> Can someone please land lp:~wgrant/launchpad/rework-dominator-tests?
<thumper> mwhudson: did the email make reasonable sense?
<mwhudson> thumper: yes
<thumper> cool
<mwhudson> thumper: although i guess i had a head start on understanding :-)
<thumper> :)
 * thumper afk to pickup
<wgrant> Some of the queries in lib/lp/archivepublisher/domination.py are extended versions of some on ArchiveSet. In a normal application, I'd call the ArchiveSet method and then .find() on the ResultSet to do further filtering.
<wgrant> But IResultSet doesn't contain .find(), so I can't do that in LP.
<wgrant> Help?
<thumper> wgrant: I think perhaps an updated storm has it...
<thumper> not sure though
<thumper> could ask on #storm
<wgrant> thumper: Thanks, I've asked there.
<wgrant> thumper: Could you be convinced to ec2 https://code.edge.launchpad.net/~wgrant/launchpad/rework-dominator-tests/+merge/29668 and https://code.edge.launchpad.net/~wgrant/launchpad/link-uploaded-ddebs/+merge/29669?
<thumper> wgrant: yep
<wgrant> Thanks.
<lifeless> moin
<wgrant> Morning lifeless.
 * wgrant wishes that the 3.0 heading guidelines were complete.
<lifeless> ...?
<wgrant> lifeless: Page titles were redesigned for 3.0.
<wgrant> But they did not consider how it should work when on a page with a non-root context.
<wgrant> The root context is always shown at the top of the page.
<wgrant> But say I'm on a page relating to a PPA. It has the owner's name at the top, and no built-in indicator of which PPA it's operating on.
<wgrant> What should the h1 be?
<wgrant> 'Administer', 'Administer archive', 'Administer PPA for William Grant'?
<lifeless> oh
<lifeless> is this a widely known issue ?
<lifeless> who was doing the work, can we ask them to finish it off? Or is there a good answer ppl are just doing ?
<wgrant> I've raised it a couple of times in the last 12 months.
<wgrant> There's no consistency at the moment.
<lifeless> when you raised it, what happened; was there some sort of consensus ?
<wgrant> Nothing really came of it.
<wgrant> There's also the whole mess around how the tabs work for non-root contexts.
<wgrant> The issue was identified somewhat before 3.0, but never resolved.
<wgrant> Resulting in a confusing UI for non-projects.
<lifeless> is there something you'd like to do about it ?
<lifeless> actually
<lifeless> a better question
<lifeless> do you have a proposal that would improve consistency and be, on the whole, more useful for users.
<lifeless> and not be terrible to calculate perf wise.
<wgrant> I don't know what would be best.
<lifeless> I'm not talking best.
<lifeless> best is freaking hard man.
<lifeless> I think some consistency is good - useless consistency isn't. We need to do each page really well, but a sane default can help a great deal.
<wgrant> Even some pages on root contexts are pretty bad.
<wgrant> https://code.edge.launchpad.net/launchpad/+activereviews, for example.
<lifeless> is there some difficulty in fixing them?
<wgrant> They're trivial to fix, once we define what they should be like.
<lifeless> uh
<lifeless> why do we need a pre-fix definition here ?
<lifeless> *anything* is trivial if you can define it :P
<wgrant> If we are changing titles to suck less and be more consistent, we need to define what they are going to be consistent with.
<wgrant> 3.0 tried to do that, but failed.
<lifeless> perhaps I'm over simplifying
<lifeless> but - a) suck less, b) be more consistent : seems we could make them suck less by /doing/ and make them more consistent by picking the nicest pattern that emerges and using it as a sane default ?
<wgrant> Possibly.
<lifeless> wgrant: if the only consistent thing at the moment is that they suck, surely we should fix that :)
<lifeless> we have a ui review process, so mistakes should be caught - thats a decent safety net
<wgrant> You can't make a mistake at the moment.
<wgrant> Because there are no guidelines to break.
<wgrant> I think we just need to work out the undefined general cases, and define them.
<wgrant> This may require a change to the structure of the header.
<lifeless> wgrant: You seem to be saying folk can't improve things here without a lot of work, which surprises me,
<lifeless> is that right?
<wgrant> lifeless: It can be mostly fixed by someone just declaring how much context information should go in titles.
<wgrant> A complete fix (which fixes tabs for non-root contexts) is harder.
<lifeless> so if I say 'the right amount to look good on a page should go there', is that useful, or am I *still* missing the point
<thumper> wgrant: both your branches playing through ec2
<wgrant> lifeless: How much looks good on a page?
<wgrant> And is that amount enough?
<wgrant> thumper: Thanks.
<lifeless> wgrant: that seems to be a matter of judgment and testing
<lifeless> wgrant: You don't seem to have a hesitation doing that to lp internals; I'm unclear why you have a hesitation doing the same to page templates :)
<wgrant> lifeless: Precisely.
<wgrant> lifeless: I am no UI designer.
<wgrant> LP has a severe lack of UI design
<wgrant> And it shows.
<lifeless> are you saying we lack the knowledge to improve at all?
<wgrant> No. I'm saying that somebody with significant user experience should be consulted before we go and fix all the titles to conform to a new standard that might still suck.
<lifeless> wgrant: so - +1 on getting more import. -1 on doing /nothing at all/
<lifeless> s/import/input
<wgrant> We have no UI designer. How do we get more input?
<lifeless> we have a whole design team on call, and they are extremely happy to do small things on-request.
<lifeless> I'm not at all convinced that this is a problem we lack the knowledge for though. However, I'll grab mpt today and show him this chat and ask his opinion.
<wgrant> An attempt was made a year ago.
<wgrant> It did not work.
<lifeless> wgrant: so try something different
<wgrant> We can certainly make large improvements easily.
<wgrant> But it would be nice to get it right.
<lifeless> great!
<lifeless> the thing about right.
<lifeless> is that it lasts for, oh, 2-3 days.
<wgrant> OK, as close to right as we can easily get.
<lifeless> :)
 * bryceh goes close to left
<lifeless> bryceh: any progress on the attachments thing?
<bryceh> hmm?
<lifeless> the api call
<lifeless> in bugs
<lifeless> is there a partial branch or something like that to make it faster? If not I'll have a stab at same today.
<bryceh> nope, I'm trying to get another branch finished up before I go on vacation
<lifeless> ok
<lifeless> losa hi
<lifeless> spm: hi ?
<spm> lifeless: hola
<lifeless> how long till edge rolls out ?
 * lifeless is excited
<thumper> lifeless: excited about what?
<spm> lifeless: https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus "Edge Updates" 0800 BST. we strikethru if they're not for some reason or another
<lifeless> thumper: rev 11199 on stable
<lifeless> spm: kk
<thumper> lifeless: which has what?
<lifeless> a /massive/ search overhead reduction
<thumper> ah
 * thumper EOWs
<lifeless> have a good weekend
<lifeless> spm: so, another hour right ? :)
 * lifeless bounces
<bilalakhtar> lifeless: Are you a core-dev in ubuntu?
<spm> lifeless: pretty much exactly, yeah
<lifeless> bilalakhtar: no, why?
 * bilalakhtar is searching for core-devs to sponsor his bugfixes
<bilalakhtar> lifeless: ^^
<lifeless> put it in the sponsor queue please
<lifeless> spm: whats the url for monitoring edge rollout progress again ? I didnae bookmark it
<spm> lifeless: we don't have one
<lifeless> oh
<lifeless> thats right :)
<lifeless> we have one for staging or so or something
<spm> yup
<spm> https://staging.launchpad.net/successful-updates.txt <== tho is more "worked/not" vs progress
<lifeless> so how is edge going ? :)
<lifeless> <- not keen at all
<danilos> OOPSes while trying to log in, a known problem? https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1665ED841
<spm> from memory, it'll be around 5-10 mins before it gets beyond simply building/syncing the tree
<lifeless> danilos: ugh
<lifeless> spm: we may have to abort - benji's code may be bust
<lifeless> spm: don't stop it yet
<lifeless> danilos: does your username have utf8 in it ?
<danilos> spm, did we get a new openid rollout by ISD guys as well? (just to confirm we only have to look at our code for the problem)
<spm> lifeless: not really; if it's a hard fail; edge1 will break and the rollout will stop
<danilos> lifeless, username doesn't, but my name does
<spm> if it's more subtle than that; rolling back is trivial
<lifeless> spm: actually, thats already on edge. I was thinking a prod stop-point
<lifeless> <- EUNCAFFEINATED
 * danilos disables edge redirection and tries again
<wgrant> lifeless: Is it complaining about the XSRF token sent by SSO?
<wgrant> I can't see the OOPS, but there's already a branch to fix that.
<lifeless> wgrant: no, unicode
<wgrant> Ah.
<lifeless> danilos: file a bug on foundations
<danilos> lifeless, right, my name is the culprit, I've changed it and can log in
<danilos> lifeless, prod also works, so it's edge code
<lifeless> its a classic 'this code wasn't QA'd' scenario :-
<lifeless> which the new deployment process should address
<danilos> lifeless, I'd also lean on marking this one as critical, but will leave that to whoever gets a chance to take care of it
<danilos> anyway, bug 609029
<_mup_> Bug #609029: UnicodeDecodeError OOPS on login on edge <oops> <Launchpad Foundations:New> <https://launchpad.net/bugs/609029>
<lifeless> danilos: I agree
<lifeless> spm: hows edge, and how is the db load looking
<spm> lifeless the-impatient ;-) fwiw, your email will probably know before I do, unless something breaks hard. error reports: Subject: 	Cron <pqm@praseodymium> /home/pqm/bin/rollout_edge.sh will have the details
<lifeless> spm: yes, its live
<spm> load looks normal, as a one off snapshot ish.
<spm> looks like edge4 is/has just been updated
<spm> has
<lifeless> ok, https://bugs.edge.launchpad.net/ubuntu/+filebug does seem better on edge now
<mrevell> Howdy
<lifeless> spm: ok, still looks ok ?
<spm> lifeless: I assume so, not getting any alerts; which is the usual 1st line
<lifeless> k, thanks
<wgrant> Does PQM really take half an hour?
<jkakar> FYI, I'm getting 'Internal Server Error' here: http://bazaar.launchpad.net/~jkakar/storm/resultselect/files
<spiv> jkakar: that branch is 1.9 format, but is stacked on trunk which is 2a
<spiv> jkakar: upgrade it and it'll be fine
<jkakar> spiv: Yeah, I figured as much...
<jkakar> spiv: Regardless, I shouldn't see "Internal Server Error".
<spiv> Oh, certainly.
<spiv> I *think* there's a bug about that already, feel free to click "affects me too" on it.
<jkakar> Cool.
<wgrant> bigjools: Are there really no tests of NascentUpload that don't use a real upload?
<wgrant> I can't see any :(
<wgrant> I guess I'll work it out.
<bigjools> wgrant: probably not :(  I know there's some unit tests for dscfile
<bigjools> that's about it
<bigjools> but it's the direction we need to go so that soyuz tests don't take 35-40 minutes. :/
<wgrant> Yeah, I've started shuffling things around so it's more possible.
<wgrant> archiveuploader is really ugly.
<bigjools> ugly with boils
<lifeless> wgrant: ok, mpt review done, checklist being fixed.
<lifeless> its like the title but less so :P
<wgrant> lifeless: Ooh, thanks.
<mpt> lifeless, https://dev.launchpad.net/UI/Reviews?action=diff&rev2=34&rev1=33
<wgrant> mpt: Thanks.
<wgrant> So we're to include context in the h1, even if it's the root context?
<wgrant> (which is already displayed at the top of the page)
<lifeless> mpt: thanks!
<lifeless> wgrant: mpt said verbally that folk ignore the global nav stuff, universally
<lifeless> perhaps thats worth adding
<wgrant> In the root context case, it's in the breadcrumbs and header already.
<mpt> wgrant, remind me what "the root context" means?
<wgrant> mpt: The pillar or person containing this page.
<wgrant> The context may be a source package, but the root context is the distribution.
<wgrant> It's displayed at the top of every page.
<wgrant> (I believe the concept and term were introduced with 3.0)
<lifeless> wgrant: the point is people don't read that stuff
<lifeless> wgrant: mpt: anyhow: if we can transfer the info so that wgrant is unblocked, I will be ecstatic. I think the page improvement so far is very useful.
<wgrant> It looks like a good idea.
<mpt> Ah, I should have mentioned another thing
<mpt> More specific stuff should go earlier in the heading and the title
<mpt> less specific stuff later
<wgrant> That's ensured in the title, since it's reversed breadcrumbs.
<wgrant> Although that does result in oddities like "10.04 : Ubuntu", which I'm not really a fan of.
<mpt> yeah, reversed breadcrumbs are an unfortunate case of automation over sensibility
<wgrant> I think they are better than what came before them.
<wgrant> But not better than a fully redesigned set of custom titles.
<mpt> They may be better technically, in that pagetitles.py was hilarious
<wgrant> There is at least some amount of consistency now.
<wgrant> pagetitles.py is almost gone.
<mpt> But at least pagetitles.py let me eyeball inconsistencies.
<stub> lifeless: Have you looked at garbo.py? It is a framework suitable for many of our hourly and daily tasks - the ones that don't need external resources besides the database.
<stub> lifeless: So for instance I'll be suggesting to jtv that the cachesuggestivetranslations.py cronjob should be a task in the garbo, avoiding yet-another-cronscript-to-manage as well as bonuses such as aborts if it takes too long and fewer notifications for people to monitor.
<stub> lifeless: It needs better plugin architecture, and it isn't a replacement for a general job handler, but I think a step in the right direction.
<lifeless> stub: nice
<lifeless> stub: +1
<lifeless> stub: I do think jtv's particular case would be better served by events, but the existing event queuing things are not nice enough to draw in adopters yet
<stub> I don't know enough of the task system or the suggestions to know if that is a suitable fit.
<lifeless> stub: making garbo replace most of the existing cron jobs would help the sysadmins a lot I think
<stub> Yup. Also makes scheduling nicer - we don't end up with batch jobs running simultaneously causing load spikes.
<lifeless> anyohe that has oopstools working - could you see if https://bugs.edge.launchpad.net/oops-tools/+bug/608914 has a working patch ?
<lifeless> jelmer: wb, did you see my reply?
<_mup_> Bug #608914: Bugs not shown in the Timeouts section of oops summaries <OOPS Tools:Triaged> <https://launchpad.net/bugs/608914>
<jelmer> lifeless: yep, thanks
<poolie> james_w, istm that <https://edge.launchpad.net/ubuntu/+source/bzr> (for example) ought to link to the packaging branches
<poolie> if any
<poolie> or tell us there are none
<james_w> yes
<james_w> "Code", but yes
<rockstar> Morning folks!
<bigjools> hey rockstar
<rockstar> bigjools, what's the status of the maverick buildd stuff?
<bigjools> rockstar: still buggered as far as I know.  Are you referring to the aptitude thing?
<rockstar> bigjools, yeah,
<rockstar> bigjools, is there anything I can do to help that along?  I have some other tasks, but that's one of the blockers for jorge getting upstreams to use it.
<rockstar> (I'm going to work on the other one today)
<bigjools> rockstar: I don't know, I am not dealing with it.  Were you expecting me to be?
<bigjools> it's a problem with the recipe stuff in the slave code
<rockstar> bigjools, yeah, I thought that you were going to get abentley's patch applied to the buildd to fix it.
<bigjools> I didn't know there was a patch :)
<bigjools> we're currently having problems getting any recipe build to go through dogfood - they all hang.
<rockstar> bigjools, okay, lemme chat with abentley when he's around, and we'll sort it out.  We need to make that a priority.
<rockstar> bigjools, :(
<bigjools> rockstar: see this for example: https://code.dogfood.launchpad.net/~abentley/+recipe/bazaar/+build/263/+files/buildlog.txt.gz
<bigjools> "bzr: ERROR: No previous changelog to take the package name from, and --package not specified"
<bigjools> wtf?
<bigjools> rockstar: ok when you get know where the patch is, we can ask lamont to deploy it on the DF builders.
<rockstar> bigjools, yeah, I was just about to point that out.
<rockstar> bigjools, ack.  Thanks.
<bigjools> it takes 10 minutes to print that error out as well
<rockstar> bigjools, since you said they were hanging, I wondered if that was an artifact of killing the build.
<rockstar> james_w, ping
<bigjools> no, I didn't kill it
<rockstar> bigjools, we also have some sourcecode/ updates that need to be applied to fix some more bugs.
<rockstar> bigjools, do you have any comments on fixing https://bugs.edge.launchpad.net/launchpad-code/+bug/583403
<_mup_> Bug #583403: Sourcepackage recipe builds need much more logging <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/583403>
<bigjools> rockstar: yeah I think the logging needs to come from bzr
<rockstar> bigjools, okay.  Are those logs just capturing stdout then?
<bigjools> yes
<rockstar> bigjools, great.  I'll get a patch together and have james_w review it.
<bigjools> nyshe
<bigjools> rockstar: we're not going to get far until we figure out that long hang + the error above though :(
<rockstar> bigjools, yeah, what resources do you need in order to do that?
<bigjools> rockstar: james_w :)
<bigjools> it's running stuff I know nothing about unfortunately
<wgrant> bigjools, rockstar: That recipe is buggy.
<bigjools> well that's a good start.
<wgrant> It should be a merge, not a nest.
<rockstar> bigjools, okay, well I'll try and reproduce it here.  james_w is kind of going into maintenance mode on some of these udd tasks, and I offered to pick up maintenance for them.
<bigjools> it should fail a bit quicker though
<rockstar> (Of course, that means I need to learn them first)
<wgrant> How long does it hang?
<bigjools> 10 minutes
<wgrant> Interesting.
 * bigjools fixes the recipe
<wgrant> bigjools: Just s/nest/merge/, and drop the 'debian' from the end.
<rockstar> That might be a bzr stupidity.
<rockstar> Like bzr builds a whole working tree in memory just to get the single folder, or something.
<bigjools> dispatching the new recipe now
<bigjools> wgrant: btw I have a new buildd-manager on DF.
<bigjools> should be fun to see if it breaks production :)
<wgrant> Heh.
<wgrant> As in the new properly parallel one?
<bigjools> testing so far shows no issue
<bigjools> yes
<wgrant> Now we just need to convince it to dump uploads in a queue, and the problem is mostly fixed!
<bigjools> I do like Twisted.  I don't like the b-m code so much.
<bigjools> yes, jelmer is working on that :)
<wgrant> Excellent.
<wgrant> I've also ported lp-buildd to use a modern system version of sbuild, so we can drop our unmaintained ancient buggy fork once Soyuz does ddebs.
<wgrant> So we should have a much healthier, more maintainable build farm soon, on multiple fronts...
<lifeless> losa ping
<rockstar> wgrant, this, my friend, is most excellent news.
<lifeless> http://pastebin.com/e9D6x3hX <- I'd like timing data for that query on each DB host please.
<mthaddon> lifeless: hi
<bigjools> wgrant: \o/ \o/ \o/
<wgrant> Although I had to destroy buildrecipe in the process.
<rockstar> wgrant, now, if only we could have test coverage for the build farm.
<poolie> james_w, https://bugs.edge.launchpad.net/launchpad-registry/+bug/609116
<_mup_> Bug #609116: package page should point to package branches (or not) <udd> <Launchpad Registry:New> <https://launchpad.net/bugs/609116>
<wgrant> rockstar: Translations made a good start on testing their part of it.
<lifeless> it the literal query running on filebug queries on edge; its timing out a lot - I'm interested in figuring out if:
<lifeless>  - there is one bad host
<lifeless>  - there are bad query plans being generated
<lifeless> mthaddon: oh, and hai
<mthaddon> :)
<wgrant> bigjools: Um, where did that recipe go?
<wgrant> It has disappeared.
<wgrant> I wonder if you ran into a form bug and accidentally reassigned it yourself.
<wgrant> Heh, yes, there it is.
<bigjools> the log was there briefly
<bigjools> wtf
 * rockstar waits for the expansion of "form bug"
<wgrant> The owner <select> won't have been populated with the previous value, since it was abentley, and bigjools is not abentley.
<gmb> rockstar, Could you or a suitable one of your code team cohorts take a look at the currently-failing merge between stable and db-devel please? There's a conflict but I'm not sure whether to just accept the merge-source version or not.
<wgrant> So it will have defaulted to the first available value.
<rockstar> gmb, sure.
<gmb> rockstar, Thanks.
<rockstar> gmb, oh shit, if this is what I thinks it is...
<bigjools> wgrant: https://code.dogfood.launchpad.net/~julian-edwards/+recipe/bazaar/+build/265
<bigjools> it's hanging again
<gmb> ...
<rockstar> bigjools, it might just be that it's not logging anything properly.
<wgrant> bigjools: Ew.
<wgrant> That is possible.
<rockstar> bigjools, when I hear "hang" I think "this will not complete" but it gets back to going in a bit.
<bigjools> that is not just possible, it's highly likely
<wgrant> bigjools: Can you check the CPU usage?
<bigjools> no :(
<bigjools> the VMs don't let you ssh in
<rockstar> It generates an error message that makes no sense, this is true.
<wgrant> The error message made perfect sense.
<bigjools> we need to make this run quicker, this is way too slow
<wgrant> It probably won't occur this time, though.
<rockstar> bigjools, maybe we should find a way to keep metrics on the buildd CPU/mem usage.
<rockstar> I'm sure lifeless is all for it.
<bigjools> aaaaaaand fail
<bigjools> /usr/lib/pbuilder/pbuilder-satisfydepends: line 92: aptitude: command not found
<bigjools> as expected for maverick
<wgrant> Better fail, though.
<rockstar> Yup.
<bigjools> can we just bin aptitude
<bigjools> it causes no end of problems
<rockstar> bigjools, no idea what the ramifications are.  The people who would know are sprinting currently.
<bigjools> pbuilder could just use apt-get and this problem would disappear
<wgrant> Well.
<wgrant> aptitude resolves some things differently.
<wgrant> It used to be vastly superior to apt-get.
<wgrant> Not so much any more, though.
<rockstar> s/differently/stupidly/
<bigjools> where differently is wrongly, so I'm told :)
<wgrant> Indeed.
<wgrant> But there was rationale for using it in the past.
<jelmer> wgrant: also, it doesn't have moo powers.
<wgrant> jelmer: Heh.
<bigjools> "apt-get moo" is right up there with "bzr rocks"
<poolie> yeah i think the moment has passed
<Ursinha> bigjools, oh crap, I didn't know about that
<Ursinha> :)
<wgrant> bigjools: aptitude's moo is better...
<bigjools> heh
<bigjools> wgrant: ok the lucid build is working now
<bigjools> I wonder what it's doing that takes so long
<wgrant> It took 10 minutes to check out from here.
<bigjools> hmmm, I wonder if it needs a --lightweight
<bigjools> anyway, food time
<wgrant> A lightweight checkout might actually be a reasonable idea here.
<wgrant> Hmmmm.
<wgrant> We should test that.
<lifeless> ok
<lifeless> so fti can do the search we need in 4ms
<lifeless> but if you rank, you're fucked
<lifeless> and if you join out from bug, you're in trouble too
<wgrant> So basically, we're doubly in trouble?
<james_w> rockstar, bigjools: if you update your copy of bzr-builder you will get more logging
<lifeless> just as a noddy example:
<lifeless> SELECT bug.private FROM Bug, BugTask WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('depend|eclips|error|get|instal|unmet')    LIMIT 40;
<james_w> plus an improvement to that previous error message
<lifeless> Time: 4.552 ms
<james_w> and many other bugfixes too
<rockstar> james_w, awesome.  I need to find out how to upgrade sourcecode/ then.
<lifeless> SELECT bug.private FROM Bug, BugTask WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('depend|eclips|error|get|instal|unmet')  ORDER BY -rank(Bug.fti, ftq('depend|eclips|error|get|instal|unmet'))  LIMIT 40;
<lifeless> Time: 7467.015 ms
<rockstar> james_w, a bzr-builder and bzr-builddeb update are both in my TODO list for today.
<wgrant> rockstar: sourcecode won't help the builders.
<james_w> true
<rockstar> wgrant, I understand that, but it gives me a start.
<james_w> it will still be useful for other fixes though
<james_w> where in the log does the 10 minute hang happen?
<rockstar> To deploy the new changes, I can just put a new package in a ppa (location still unknown to me) and then have lamont update.
<rockstar> bigjools, see james_w's question. ^^
<wgrant> james_w: Immediately after the launchpad-login warnings.
<james_w> ok
<rockstar> james_w, I think it's actually in the builder part at that point, so if we have better logging, we might not have to worry about it anymore.
<james_w> rockstar: it's still in bzr-builder I think, but I believe that it may just be slow bzr access over http
<rockstar> james_w, yeah, that's what I was thinking.
<james_w> rockstar: the new bzr-builder should help with that some as well
<rockstar> james_w, yup.  Looking to update it today.
<wgrant> Is there a good reason for buildrecipe to be a shell script?
<wgrant> I need to move some of it into the Python part of the slave.
<james_w> wgrant: none that I know of
<wgrant> And it all ends up simpler (particularly the error handling) if the whole thing is integrated into the slave Python.
<lifeless> SELECT bug.private, 1 as therank FROM Bug, BugTask, to_tsquery('depend|eclips|error|get|instal|unmet') query WHERE query @@ bug.fti and Bug.id = BugTask.bug AND BugTask.distribution = 1 ORDER BY therank DESC LIMIT 40 OFFSET 0;
<lifeless> Time: 4.171 ms
<lifeless> SELECT bug.private, ts_rank(bug.fti, query) as therank FROM Bug, BugTask, to_tsquery('depend|eclips|error|get|instal|unmet') query WHERE query @@ bug.fti and Bug.id = BugTask.bug AND BugTask.distribution = 1 ORDER BY therank DESC LIMIT 40 OFFSET 0;
<lifeless> SELECT bug.private, ts_rank(bug.fti, query) as therank FROM Bug, BugTask, to_tsquery('depend|eclips|error|get|instal|unmet') query WHERE query @@ bug.fti and Bug.id = BugTask.bug AND BugTask.distribution = 1 ORDER BY therank DESC LIMIT 40 OFFSET 0;
<lifeless> bah
<lifeless> Time: 5626.790 ms
<lifeless> ok
<lifeless> plane time
<lifeless> mthaddon: thanks a lot for the stats - its useful
<mthaddon> cool
<wgrant> I guess that sort of makes sense.
<james_w> wgrant: there's a use for process separation I think, but the language doesn't have an impact on that
<wgrant> james_w: Given that it pretty much just executes a few subprocesses anyway, I don't see much value in having another layer there.
<jtv> danilos, btw: a very simple small improvement I discussed w arne & dpm is to discard blocked ubuntu PO uploads after a year.  If there's ever a new upload they'll probably just get re-blocked anyway, but it cuts down the queue by 40%
<danilos> jtv, yeah, sounds good
<james_w> wgrant: right, it's a small window of exceptions that would affect that. Making it not run builder as a subprocess and call the python functions instead probably isn't a good idea though
<jtv> also, arne came up with a great solution for some other process problems: when they approve an upload, do an instant approval run over other entries it's likely to affect.
<wgrant> james_w: Oh, certainly. Those will continue to run as subprocesses. But buildrecipe will be merged into sourcepackagerecipe.py.
<danilos> jtv, sounds so simple ;)
<james_w> wgrant: +1
<danilos> jtv, (ok, it's not too hard if "instant" is "in a few minutes" :)
<jtv> danilos: the fun is in the fine print...  this would be approvals, not blocking, and probably only needs-review uploads with the same path and for the same package/productseries
<wgrant> james_w: Thanks.
<jtv> I think that's probably not too hard.
<danilos> jtv, hum, language detection, custom language codes, different layouts? blocked is just about the problem of the same complexity so not really a big deal
<jtv> danilos: they don't mind if it doesn't work as well for different layouts.  I wouldn't think the rest would be that costly.
<jtv> Or how about just the files with the same path?
<james_w> hi gary_poster, I requested a review from you of https://code.edge.launchpad.net/~james-w/launchpad/drop-default-skin/+merge/30763 which was a change that salgado discussed with you the other day. Let me know if you would like someone else to review it.
<gary_poster> ack, james_w.  I'll look at it in a few
<flacoste> gmb: subscribing to a tag is within scope of the better subscription control story right?
<gmb> flacoste Yes.
<gmb> flacoste, Sorry, just grabbing lunch; I'll answer any questions when I get back.,
<flacoste> gmb: no other questions, bon appÃ©tit!
 * rockstar fetches breakfast
<rockstar> bigjools, ping?
<bigjools> rockstar: hi
<rockstar> bigjools, we're talking recipes in mumble right now.
<rockstar> bigjools, I suppose that was my indirect way of inviting you to talk recipes with us in mumble.
<mtaylor> rockstar: morning
<rockstar> mtaylor, hello sir
<rockstar> mtaylor, are you on the west coast?
<mtaylor> rockstar: I am.
<mtaylor> rockstar: I have some fun tarmac things to poke you about - and now I'm in the right time zone to do it!
<rockstar> mtaylor, ah, yes, I am home now, so we're only an hour difference.
<mtaylor> w00t
<mtaylor> rockstar: well... the most annoying one is that, as lifeless was hinting might be the case, tags are not propogating
<rockstar> mtaylor, hm...
<rockstar> mtaylor, is there a bug?
<mtaylor> rockstar: not yet
<mtaylor> rockstar: I made a quick list of bugs to file, which I will do this morning
<rockstar> mtaylor, well how do I fix a bug I don't know about?  :)
<mtaylor> rockstar: I know I know
<rockstar> mtaylor, also, there is #tarmac, that actually has people in it, if you get stuck and I'm not around.
<rockstar> gary_poster, is there a doc in the launchpad that tells me how I can update stuff in sourcecode/
<gary_poster> rockstar: not that I'm aware of, unfortunately, but the conf file is pretty obvious.  Lemme find it for you (somewhere in utilities...)
<gary_poster> sourcedeps.conf
<benji> is there a way I can get a full diff between the code on edge and prod?
<gary_poster> rockstar: utilities/sourcedeps.conf .  Take a look and then feel free to ask questions
<gary_poster> benji: um, yeah...you could do ``bzr diff --new=(edge) --old=(prod)`` where (edge) == lp:launchpad/stable and (prod) == lp:launchpad/production or something close to that, but that sounds to me like you're going to have an awfully unwieldy diff.  I can see why you'd ask though (the unicode login thing, I suspect)...thinking if there might be another way to approach it...
<benji> yeah; it turns out that I can't reproduce it in development and I can't imagine how my recent changes caused it
<benji> the tests for this corner of the world are pretty screwy; so much stuff is faked out that very little of the real code runs, and when it does run it often takes different code paths than in actual use
<gary_poster> benji, you've duped on edge or (mildly better because then you are not changing the real db) staging?
<benji> yep, I can dupe on edge
<benji> I just had to change my login.launchpad.net user name slightly, so no DB change worries
<gary_poster> the fact that you can't dupe locally (launchpad.dev) is concerning
<benji> gary_poster: the problem is that it's not possible to dupe in dev, not that the error doesn't exist; there are no test users with unicode in their names and the prod OpenID provider doesn't entirely work with dev
<gary_poster> benji, oh, I see.  So, to dupe, you would need to have a new user (or modified user) that has the state.  ...I *think* we have some helpers now to create users on the fly
<gary_poster> benji: utilities/make-lp-user might or might not help
<benji> k, thanks
<rockstar> gary_poster, okay, so I just submit a merge against the pqm managed branch and then update the sourcecode.conf file with the new revno?
<gary_poster> rockstar: right
<rockstar> gary_poster, okay, thanks.
<gary_poster> np
<rockstar> james_w, ping
<james_w> hi rockstar
<rockstar> james_w, abentley tells me you're making packages for bzr-builder and bzr-builddeb to be used in the buildds (so I don't have to make those).  Is this true?
<james_w> I did last time
<rockstar> james_w, I'm happy to make them and put them in the right place if you can provide me some details.
<james_w> rockstar: bzr-builddeb is a native package, and I usually upload via Debian, it's easy to tweak the version number and build a pre-release package though
<rockstar> james_w, I'm more concerned about bzr-builder's status more than anything now.
<james_w> rockstar: I'm just checking if there is anything I should fix before doing a bzr-builder release. If not then I will do that and we can get a package in to its PPA.
<james_w> I wanted to get the merge-subdirs change in, but it is blocked right now, so I will do a release now, and then one with it when it lands
<rockstar> james_w, I think the big bzr-builder bug we need the fix for is the new merge instruction.
<statik> hola launchpad hackers
<statik> i'm trying to install launchpad-dependencies on maverick, and can't find the python-profiler package that is wanted
<statik> is this something i could fix by rebuilding a package for maverick in the PPA?
<james_w> rockstar: https://edge.launchpad.net/bzr-builder/trunk/0.3
<james_w> statik: you need multiverse enabled, at least on lucid
<rockstar> james_w, did you mean to have that file start with .. ?
<james_w> rockstar: hah, no, I obviously don't have the script with that bug fixed.
<statik> james_w, ah thx, i will check that i have multiverse enabled
<rockstar> james_w, does this have the new instruction in it?
<james_w> rockstar: no
<rockstar> james_w, :(
<james_w> that branch isn't ready yet
<james_w> yeah
<rockstar> james_w, what do we need to do to get it ready?  What can I do to help it along?
<james_w> New tarball uploaded
<james_w> https://code.edge.launchpad.net/~spiv/bzr-builder/merge-subdirs-479705/+merge/14979
<james_w> rockstar: kicking it off to build into the PPA
<rockstar> james_w, awesome, thanks.
<james_w> now I'm heading out for dinner
<james_w> the packages should show up here in a bit: https://edge.launchpad.net/~dailydebs-team/+archive/bzr-builder
<rockstar> james_w, the package is basically trunk right now, yes?
<maxb> statik: Just testing, or actually going to try LP dev on Maverick? Is it time that we should be looking to properly populate the PPA for Maverick?
<mars> abentley, ping, have a moment for a quick question about testrepository and testr?  I was wondering what you have in your .testr.conf
<abentley> mars, I have the default value except in one branch where I added xvfb-run to the run command.
<abentley> mars, right now the lack of incremental output is a real downside, so I haven't been using it much.
<mars> incremental output?
<mars> like "It runs everything and spits out the buffer at the end"?
<abentley> mars, all the stuff about setting up layers, tearing down layers, and individual test runs.
<abentley> mars, pretty much, yeah.
<mars> abentley, ok, thanks.  Still sounds like it would work well for my one-off "I need to run all windmill tests for failures in a background process" type of work.
<abentley> mars, yes, it should be good for that.
<mars> many thanks
<mars> abentley, ack, next question: I did the obvious thing, "testr run -t test_me_too", and it says "no such option, -t".  Quoting doesn't seem to help.  Any way to pass options right on through?
<mars> just a sec, thought of something
<mars> maybe '-- -t foo' will work
<abentley> mars, use --
<mars> alright, "next to least surprising thing" works :)
<abentley> mars, once you've had a test run, you'll probably want "testr run --failing"
<abentley> mars, though you can also run individual tests through it to knock them off the failing list.
<mars> neat
<statik> maxb, yes i'm on maverick fulltime and trying to dip my toe back in the water with launchpad dev
<maxb> I shall copy some packages and see if they install in a chroot
<abentley> mars, (quoting doesn't work because the shell interprets quotes, so testr can't distinguish it from testr run -t\ test_me_too)
<maxb> What on earth? I'm trying to copy pocket-lint from lucid to maverick in the launchpad ppa, and getting "binaries conflicting with the existing ones"
<maxb> it's a copy with binaries, how can there be a conflict?!
<statik> maxb, thanks for looking after this
<rockstar> abentley, so, bzr experts apparently can also rescore/cancel builds.  That's why you were seeing the link.
<rockstar> (I had forgotten that thumper suggested it)
<abentley> rockstar, ah, okay.
<mtaylor> daily ppas should be able to be triggered by push to branch
<mtaylor> just saying
<lifeless> losa ping
<mbarnett> heya lifeless
<lifeless> hey
<lifeless> I want to generate some stats from staging's fti index
<lifeless> this will take hours and hours
<mbarnett> heh
<lifeless> so I'm wondering if someone can do it in a screen session on the box
<mbarnett> lifeless: sure.
<lifeless> ok, I'll prep the exact thing to run form devpad then pastebin it for you
<lifeless> actually
<lifeless> its late, I'll skip it and get it organised over the weekend, might see if someone is around tomorrow to kick it off or some such
<lifeless> have a good weekend - gotta log off the wifi here
<wgrant> benji: You know you can run canonical-identity-provider locally, right?
<benji> no!
<wgrant> benji: It's not too difficult, and lets you replicate the production setup almost exactly.
<wgrant> bzr get lp:canonical-identity-provider
<benji> I need to get that going.  Any directions?
<wgrant> Then follow the instructions in the tree.
<benji> great!
<wgrant> It's even open source now.
<benji> thanks, that'll make my life much better
<wgrant> Then you just have to tweak the LP vhost config to use that rather than login.launchpad.net, configure c-i-p to send full name and email address to launchpad.dev, and it all should work.
<benji> sounds good
#launchpad-dev 2010-07-24
<benji> XXX?
<benji> oops, wrong chan
<wgrant> Morning lifeless.
<wgrant> Your hostname confounds me.
<lifeless> I'm at denhaag
<lifeless> or in den haag
<lifeless> GNU Hackers Meeting is on today + tomorrow
<wgrant> Ahh, forgot that.
<lifeless> :P
<lifeless> so I know have a very good handle on search
<lifeless> it *may* be a tsearch2 bug
<wgrant> Excellent!
<lifeless> or it may be structural and unfixable
<wgrant> :(
<lifeless> [unfixable in tsearch2]
<wgrant> Ah.
<lifeless> short version: fti selectivity is important to speed
<lifeless> [duh]
<wgrant> Remarkable!
<lifeless> longer version: having a where clause with an index that doesn't support the order clause is slow
<lifeless> unless the selectivity is fantastic
<lifeless> more detail still
<lifeless> we're doing an foo|bar|baz query
<lifeless> so that we don't filter out bugs that only match 2 out of three terms.
<lifeless> guess what this does to selectivity.
<lifeless> wgrant: lpmain_staging=> SELECT count(*) FROM Bug, BugTask WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('depend|eclips|error|get|instal|unmet') AND (Bug.private = FALSE OR EXISTS ( SELECT BugSubscription.bug FROM BugSubscription, TeamParticipation WHERE TeamParticipation.person = 2 AND BugSubscription.person = TeamParticipation.team AND BugSubscription.bug = Bug.id)) AND (1=1) LIMIT 40 OFFSET 0;
<lifeless>  count
<lifeless> --------
<lifeless>  216995
<lifeless> Time: 4862.303 ms
<lifeless> lpmain_staging=> SELECT count(*) FROM Bug, BugTask WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('depend&eclips&error&get&instal|&unmet') AND (Bug.private = FALSE OR EXISTS ( SELECT BugSubscription.bug FROM BugSubscription, TeamParticipation WHERE TeamParticipation.person = 2 AND BugSubscription.person = TeamParticipation.team AND BugSubscription.bug = Bug.id)) AND (1=1) LIMIT 40 OFFSET 0;
<lifeless>   2040
<lifeless> Time: 403.075 ms
<wgrant> lifeless: Ow.
<wgrant> Is that mostly due to differing indices, or mostly due to having to scan through 200000 rows?
<lifeless> ordered by bug.heat: 383ms
<lifeless> bingo was a doggo
<wgrant> Pardon?
<lifeless> 200000 rows + fti overhead (it is slower) and you're expanding not reducing the workset
<lifeless> oh, and it is also why the search is near useless
<wgrant> Right.
<wgrant> lifeless: One thing: you have 'instal|&unmet' in the second query.
<wgrant> That looks like a mistake.
<lifeless> ok
<lifeless> 62ms
<wgrant> Is that missing a digit?
<lifeless> no
<lifeless> lpmain_staging=> SELECT bug.id FROM Bug, BugTask WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ ftq('depend&eclips&error&get&instal&unmet') AND (Bug.private = FALSE OR EXISTS ( SELECT BugSubscription.bug FROM BugSubscription, TeamParticipation WHERE TeamParticipation.person = 2 AND BugSubscription.person = TeamParticipation.team AND BugSubscription.bug = Bug.id)) AND (1=1) order by bug.heat LIMIT 40 OFFSET 0;
<lifeless> Time: 62.308 ms
<wgrant> Wow.
<wgrant> What if you order by rank?
<lifeless> 64 with the old query as the rank
<lifeless> 42 with the new
<lifeless> of course, this is because there are no bugs matching the query
<lifeless> what I want to do is to encode 'allow a missing term' into the fti
<wgrant> Heh.
<lifeless> as a stopgap
<wgrant> That might work.
<wgrant> How do other people do search?
<lifeless> quikcly
<lifeless> lucene is pretty popular
<wgrant> Yeah, that's the main one I'm aware of.
<lifeless> tsearch2 has the advantage of being in-db (simple) but that mean replicating it, wide rows, and less ability to isolate surges in load on other areas from it
<lifeless> lucandra is apparently nice, but cassandra is a support nightmare atm
<wgrant> Do any other teams within Canonical have search experience?
<wgrant> I can't think of any :(
<lifeless> there are some particular individuals
<lifeless> I'm speaking with them
<lifeless> and going to send out a more general mail once I marshall all my data and ideas
<wgrant> Excellent.
 * wgrant vanishes.
<lifeless> ciao
<lifeless> ok
<lifeless> skipping one term in each group
<lifeless> 679ms
<lifeless> SELECT bug.id FROM Bug, BugTask , ftq('(depend&eclips&error&get&instal&unmet)|(error&get&instal&unmet)|(depend&eclips&get&instal&unmet)|(depend&eclips&error&instal&unmet)|(depend&eclips&error&get&unmet)|(depend&eclips&error&get&instal)') as query WHERE Bug.id = BugTask.bug AND BugTask.distribution = 1 AND Bug.fti @@ query AND (Bug.private = FALSE OR EXISTS ( SELECT BugSubscription.bug FROM BugSubscription, TeamParticipation WHERE TeamPartici
<lifeless> Time: 679.074 ms
 * wgrant hates dodgy testrunners.
<wgrant> I broke a test pretty badly,
<wgrant> At the end it said:
<wgrant> Could not communicate with subprocess
<wgrant> Yet:
<wgrant> Total: 63 tests, 0 failures, 0 errors in 1 minutes 1.856 seconds.
<jelmer> wgrant: :-/
<lifeless> wgrant: thats probably a reinvocation + failure to report to subunit properly issue
<wgrant> lifeless: I believe so, yes.
<wgrant> But it still seems that the top-level runner should not handle 'Could not communicate with subprocess' as an absence of tests.
<lifeless> ack
<lifeless> please fix! [really]
 * wgrant has been nowhere near the testrunners.
<lifeless> so ? its just code :)
 * lifeless tries to provoke a bzr-hacker-ethos
<wgrant> Heh.
<lifeless> really
<lifeless> jelmer pointed this out to me just now, that he sees less partitioning in the bzr team than the lp team : yet the bzr codebase is pretty close to the lp one in size
<wgrant> Oh, I'm not scared of touching other parts of LP.
<wgrant> But the testrunner isn't.
<lifeless> what risks do you see in touching it ?
<wgrant> Well, for one thing I don't know how to change the eggs.
<lifeless> wgrant: ok, so lets recurse on that - here is what I do: I edit in place, then when happy with the result I do an upstream patch separately.
<wgrant> Ah.
<lifeless> changing the eggs after that is just edit the version config, and add the tar/egg to the download cache
<lifeless> apparently buildout.cfg can do much better, but it appears that two or three people only know how it works, which adds to the barriers.
<wgrant> Heh.
 * lifeless was a lot happier with dropping stuff in src, but - tradeoffs
<james_w> sketch: in the tree you are working on: bzr branch lp:zope.testrunner (or anything to get it's code version controlled), edit buildout.cfg to set develop to be ". zope.testrunner"
<james_w> this will tell buildout that you are hacking on both projects in parallel, and it won't use distributed eggs for them
<james_w> there are a couple of gotchas, so it can be worth editing setup.py in zope.testrunner to bump the version, and then edit versions.cfg to have that version for zope.testrunner
<james_w> that's because buildout doesn't ensure that develop targets don't take precedence over everything else
<james_w> I've never tested that in lp, but that's the process I worked out elsewhere
<lifeless> james_w: _please_ make this more visible to the team - consider adding to doc/buildout or the wiki or both!
<lifeless> I'm begging you:)
<james_w> I don't know about getting a patched egg in to launchpad, I think there's documentation on it
<lifeless> yes, that step is known
<james_w> lifeless: doc/buildout already talks about develop, but as an explanation of the key, I assume that we want instructions on doing it too?
<lifeless> its a little brief
<lifeless> written by someone that knows it, I suspect
<james_w> I can write something up, but I would need to test it, and I have to go checkout now
<lifeless> have a good trip
<james_w> lifeless: heh, thanks for pointing out my redundant code in the oops thing
<james_w> lifeless: do I have your rs to land that change via ec2?
<lifeless> yes
<lifeless> r= in fact
<james_w> thanks
<james_w> I'll get to that once I'm back somewhere with internet
<lifeless> no rush
<james_w> I was thinking that it should probably remove the intentionally triggered oopses, but if something odd happens it could get very confusing
<james_w> (in the tests for this feature)
<wgrant> lifeless: Do we have testscenarios support in LP?
<wgrant> (thanks for the review)
<lifeless> wgrant: trivial to add it
<lifeless> rmadison python-testresources
<lifeless> python-testresources |    0.1-1.2 | hardy/universe | all
<lifeless> python-testresources |    0.1-1.2 | jaunty/universe | all
<lifeless> python-testresources |      0.2-1 | karmic/universe | all
<lifeless> python-testresources |    0.2.4-1 | lucid/universe | all
<lifeless> I think its api compatible with 0.1
<wgrant> lifeless: Launchpad hates packages.
<wgrant> Despite managing them...
<lifeless> wgrant: we have dep packages for a reason
<wgrant> True.
<lifeless> we still add the dep to buildout for clarity
<lifeless> hi mars
#launchpad-dev 2010-07-25
<lifeless> moin
<mwhudson> lifeless: in transit?
<lifeless> mwhudson: den haag, GNU Hackers Meeting
<mwhudson> lifeless: o right
<mwhudson> lifeless: have you considered xapian for searching btw?
<lifeless> yes
<lifeless> gmane use it
<lifeless> but it has almost zero buzz on internet scale sites
<mwhudson> ok
<lifeless> next-rollout instructions are on the main deployments page, right ?
<lifeless> mwhudson: https://code.edge.launchpad.net/~lifeless/launchpad/librarian/+merge/30880
<lifeless> mwhudson: what happens when lazr encounters a new, unknown key?
<mwhudson> lifeless: i'd have to experiment to be sure
<lifeless> k
 * lifeless experiments
<lifeless> ah yes, it suicides
 * lifeless doesn't understand this choking-consistency approach
<lifeless> mwhudson: think I could convince you to click on 'approve' twice ?
<james_w> lifeless: hi, could I get your vote on https://code.edge.launchpad.net/~james-w/launchpad/devel/+merge/30887 I'm having trouble submitting it the manual way.
<nigelb> \o/ installing vm for playing around with LP!
<lifeless> nigelb: cool
<nigelb> I wonder if there are some low hanging bugs I could take a look at to figure out how to start.
<lifeless> james_w: we should make that easier!
<james_w> lifeless: we should
<lifeless> nigelb: well, like with most projects thare a lots of bits around
<lifeless> nigelb: I suggest, pick up something you're interested in, and follow your nose / ask for help dig in etc
<nigelb> lifeless: hm I'll wait for things to be set up so I can play around
<bryceh> nigelb, fwiw, I found the easiest things to work on were changing text in templates
<bryceh> find an error in the UI, grep source code to find where it appears, fix, fix tests, go
<nigelb> ahhh, like the bug I reported and you fixed.
<nigelb> I should hunt around around for something like that then
<bryceh> nigelb, yeppers
<lifeless> poolie: whats the MP url?
<poolie> https://code.edge.launchpad.net/~mbp/launchpad/flags/+merge/30581
<lifeless> kicked off
<poolie> thanks
<poolie> what did you type?
<poolie> ec2 test -b for me gives a weird misparsed url
<lifeless> robertc@lifeless-64:~/source/canonical/launchpad-repo/devel$ PYTHONPATH=lib python utilities/ec2 land https://code.edge.launchpad.net/~mbp/launchpad/flags/+merge/30581
<poolie> specifically two urls string joined together
<poolie> when i tested it i got some failures in codehosting, and i'm pretty sure i didn't break them
<poolie> how does that work with ec2test?
<lifeless> I think ec2test merges trunk
<poolie> you have to keep merging trunk until other people's branches are fixed?
<poolie> oh right, it does
<poolie> in this case i want it to merge db-devel, but i guess it can get that from the mp
<lifeless> https://lpbuildbot.canonical.com/one_box_per_builder will show you if we're recently broken on trunk etc
<lifeless> poolie: http://ec2-204-236-213-119.compute-1.amazonaws.com/current_test.log should let you track it in tribunal
<poolie> thanks, i'll have a poke at ec2 test
<poolie> for the snags i hit
<poolie> i just didn't want to leave thi stalled any longer
<lifeless> cool
<lifeless> I'm glad :)
<poolie> lifeless, you should paste that email onto the blog
<lifeless> which one ?
<lifeless> poolie: ^
<poolie> blog.l.n.
<lifeless> poolie: which email!
<poolie> architecture overview
<lifeless> ah
<poolie> s//architecture reviewe process/
<lifeless> hmmm, do you think users will be that interested ?
<poolie> architecture overview review process methodology definition process
<poolie> :-P
<lifeless> perhaps I'll put it on my blog
<lifeless> that should get the developer orientated types pretty effectively
<poolie> up to you
<poolie> i think it will be interesting to users on the edge of being developers
<poolie> at the moment the lp blog is a bit blah and i'd like to see it built up into something more attractive
<lifeless> its kindof there to be there
<lifeless> if you know what I mena
<lifeless> mean
<lifeless> I think the shorter iteration cycle will help a lot
<lifeless> rather than splat per month
<lifeless> splat per day
<poolie> istm you are wearing the TA hat for that post
<poolie> rather than the rbtc hat
<lifeless> sure
<lifeless> I'll think on it.
<lifeless> I don't believe in role-blogs as a concept
<lifeless> people and roles are not partitionable like that
<lifeless> I like the blog for communicating with the LP community as a thing
<lifeless> for widespread stuff, its great; its like
<lifeless> lp-dev list subset of lp-users list subset of actual lp-users subset of folk interested in lp
<lifeless> and the blog is pretty far reaching
<lifeless> (I know they aren't strict subsets, its an analogy)
<poolie> ah thanks for the buildbot link
<poolie> interesting to see the build of their bzr integration there
<lifeless> :)
<poolie> lifeless, did that vm fail? the url doesn't work
<lifeless> ec2test probably only opened my ip
<lifeless> its up to lp.archiveuploader.tests.test_uploadprocessor.TestUploadProcessor.testOrderFilenames
<poolie> jml! :)
<lifeless> jml: your diff is empty
<jml> lifeless, *your* diff is empty
<lifeless> jml: did thoust push ?
<jml> lifeless, I did!
<jml> should be good now.
<lifeless> jml: up for reviewing my librarian branch ?
<jml> lifeless, no, sorry.
<lifeless> jml: adds oops logging to it
<lifeless> ah well :(
<jml> lifeless, maybe once I've emptied my physical in-tray
<lifeless> jml: that would be nice; I would like to kick it off within 22 hours or so
<jml> lifeless, ok. I'll make a note.
<lifeless> poolie: fyi https://bugs.edge.launchpad.net/malone/+bug/607960
<_mup_> Bug #607960: timeouts on Distribution:+bugs doing searches <fti> <search> <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/607960>
<jelmer> 'morning mwhudson
<jelmer> mwhudson: do you have any experience with threads in C Python?
<mwhudson> jelmer: only a little
<thumper> morning
<thumper> I'm heading off for a dentist appt shortly
<jelmer> mwhudson: I'm trying to use pythread.h for a C worker thread without having to rely on pthreads, but I can find much documentation about the API.
<mwhudson> jelmer: i guess the headers and the source are probably the most reliable documentation anyway :/
<lifeless> wb
<lifeless> what week is it ?
<wgrant> I suspect week 2.
<jelmer> I think week 3
<wgrant> Or at least I hope so.
<jelmer> rollout was on the 6th of july
<wgrant> The Epic was week 0.
<jelmer> ah, ok - week 2 then indeed
 * wgrant consults the calendar.
<wgrant> https://dev.launchpad.net/Releases/2010Calendar says week 2.
* jelmer changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2  of 10.08 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2  of 10.08 | firefighting: buildbot slaves are down | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<mwhudson> lifeless: re https://code.edge.launchpad.net/~lifeless/launchpad/malone/+merge/30904, it seems mostly reasonably
<mwhudson> *reasonable
<mwhudson> lifeless: have you talked to the bugs developers about this?
<lifeless> no
<lifeless> why?
<lifeless> I mean - the tests and code are pretty clear.
<mwhudson> just wondering really
<lifeless> I think we're a bit too siloed at the moment, really.
<lifeless> but maybe thats just me.
<mwhudson> yeah probably
 * mwhudson attempts to articulate more clearly
<lifeless> mwhudson: while you're around
<lifeless> perhaps you could review by librarian oops branch
<mwhudson> lifeless: have you thought about ways to detect if this change will reduce the usefulness of the dup search?
<lifeless> mwhudson: yes, and - I'm sure it will, to a point.
<lifeless> mwhudson: the bugs devs that aren't on leave have all expressed their desire for and rather than or, FWIW
<lifeless> for terms and tags and so on
<mwhudson> ok cool
<lifeless> I think though, that as far as dup detection goes, we'll find that we end up with a cluster of dups that indirect to a sane master
<lifeless> so we'll get a bit more chaff up front to seed it, and then it will be better than it is today.
<lifeless> fingers crossed.
<mwhudson> lifeless: i guess what would have made me really happy to review this would be some evidence of pre-implementation discussion in the mp
<lifeless> mwhudson: I intend to bring up pre-impl as a misfeature of process
<lifeless> it was instituted to fix a bug we've now fixed other ways.
<mwhudson> lifeless: maybe
<lifeless> I think its great, and important, to socialise changes, but pre-impl doesn't do that.
<mwhudson> i don't mean pre-imp in a formal sense i guess
<lifeless> this particular change has been discussed in detail with poolie as a teddy bear
<mwhudson> i just mean 'someone other than me things this is a good idea'
<mwhudson> *thinks
<lifeless> various bugs folk on hte list
<lifeless> a prelude-to-it with francis
<lifeless> :P
<mwhudson> lifeless: ok, that's great
<mwhudson> lifeless: less back and forth if you'd said this in the mp though
<lifeless> mwhudson: whats at the heart of wanting to know that someone other than you and I think its a good idea?
<lifeless> mwhudson: I'm open to changing, but the bzr team doesn't have this friction, so the first thing I want to do is understand what it solves.
<mwhudson> lifeless: the thing here is that there's an element of a design decision here
<mwhudson> 'is this going to affect dup search quality'
<mwhudson> if it was just making tests more precise or whatever i wouldn't be quibbling
<lifeless> mwhudson: go on
<mwhudson> or if it was in an area i'm more comfortable with
<mwhudson> lifeless: so it makes me more comfortable to hear there's been discussion about it
<mwhudson> that's all
<lifeless> mwhudson: ok; I worry there is something fishy at the bottom of this : a sort of fear of fixing things outside-ones-area, or something.
<wgrant> Is that fishy at all?
<wgrant> It's not *good*, but it's the way the team is set up.
<lifeless> wgrant: yes, we all need to own the stack we work with - top to bottom
<mwhudson> lifeless: where is your librarian oops branch?
<lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/librarian/+merge/30880
<lifeless> wgrant: its only slightly setup that way
<lifeless> wgrant: it would be sad if unpaid coontributors felt safer and more able to hack on any part of the system than paid contributors. (Not saying mwhudson feels that way: we're exploring a fear *I* have after all)
<mwhudson> lifeless: approved both branches
<lifeless> thanks
<wgrant> lifeless: I'm happy hacking anywhere, but I always check with a member of the relevant team. That team controls the code, I feel, so everything should go through them.
<mwhudson> lifeless: to go over my feelings a bit, i don't think it's area-related
<lifeless> wgrant: this has kindof accreted
<mwhudson> lifeless: it's just that imho some changes require discussion
<mwhudson> and some don't
<mwhudson> discussion in the direction sense
<lifeless> wgrant: there are unmaintained areas which are kindof a side effect of partitioning. I dunno, its a theory.
<mwhudson> this one did, and i wanted to make sure such discussion had happened
<wgrant> lifeless: Are they really a side-effect of partitioning?
<wgrant> I thought they were quite deliberately completely abandoned and left to rot.
<wgrant> And by extension make Launchpad look bad.
<lifeless> wgrant: har har har
<wgrant> Well, maybe that last bit wasn't deliberate.
<lifeless> mwhudson: mmm. This is perhaps a beer talk.
<lifeless> mwhudson: In particular though, I don't want us to confuse 'high risk, needs mitigation' with 'changing an old decision'
<lifeless> mwhudson: for the search, there is a single if block to totally reinstate the old (fails with timmeout 99% of time) behaviour
<lifeless> mwhudson: so the change has the ability to be rolled back trivially
<wgrant> lifeless: Fails with timeout 99% of the time *in some contexts*.
<wgrant> It's not completely broken.
<lifeless> wgrant: thats true, its only 50% on launchpad itself.
<lifeless> and 40% or so on bzr
<mwhudson> lifeless: i probably made this into a bigger deal than necessary by explaining myself poorly
<mwhudson> lifeless: it's monday morning :-)
<wgrant> Hm. I rarely see it time out on LP projects.
<lifeless> mwhudson: :) sunday am now
<lifeless> wgrant: turn off edge redirect.
<wgrant> lifeless: Ah.
<lifeless> wgrant: you're already running the first iteration of lifeless-tuning on this.
<lifeless> mwhudson: sorry, I mean, monday am, sunday late evening.
<lifeless> mwhudson: its not a big deal, its just a background task.
<mwhudson> lifeless: cool
<lifeless> mwhudson: j'adore bzr culture, and I want to bring it into LP
<lifeless> one of those things is to unblock *even on risky things* by finding ways to reduce the impact / reduce the time to recover
<mwhudson> lifeless: well ok, but i think the launchpad culture of talking about things before doing them is actually not a bad thing
<lifeless> mwhudson: pros and cons
<lifeless> mwhudson: I adore getting solid constraints and requirements and set based design
<lifeless> mwhudson: for things of that scope
<lifeless> mwhudson: for iterating, I think a simpler model works well. Anyhow, as I say - background task.
 * lifeless twiddles waiting for spm :P
<wgrant> lifeless: When are you flying?
<lifeless> 1800
<wgrant> Ah.
#launchpad-dev 2011-07-18
<lifeless> pg upstream would probably do pg.bouncer if at all; namespace packages are a pita to package
<lifeless> hmm
<lifeless> probably want to make the rabbit fixture port allocation logic reusable
<wgrant> Bah.
<wgrant> The critical deficit is resolved already.
<StevenK> wgrant?
<wgrant> We were at 239 criticals two checks ago, 237 when I checked this morning, and now 239 :)
<thumper> hi
<thumper> what's the status of the magic diff updating?
<lifeless> magic?
<lifeless> db rebulid is good for the soul:  * 266 Time Outs
<lifeless> woo
<lifeless>  16384 | test_db1 |    2801 |    16385 | user1   | select * from pg_stat_activity; | f       | 2011-07-18 02:55:50.531462+00 | 2011-07-18 02:55:50.531462+00 | 2011-07-18 02:55:31.384875+00 |             |          -1
<lifeless> machine setup pgbouncer.
<lifeless> now we're bootstrapped, I can move onto more interesting tests.
<wgrant> thumper: Waiting on Red Squad being free to fulfill lifeless' deployment requirements.
 * thumper nods
<wgrant> Mostly needs OOPS integration and metrics now, I think.;
<michaelh1> Hey, I just had a play with bzr-git for importing the GDB 7.3 release branch into Launchpad.
<michaelh1> It works well, and can track more than one branch - you need to create a remote tracking branch first, but that's cheap
<poolie> michaelh1: rocking!
<poolie> hopefully soon lp will be able to track multiple branches
<michaelh1> yeah, so to my naive eyes it has the support that LP needs.
<poolie>  yes, it does
<huwshimi> A simple review if someone wants it: https://code.launchpad.net/~huwshimi/launchpad/email-tag-order-645962/+merge/68207
<wgrant> michaelh1: A remote tracking branch?
<wgrant> huwshimi: Ah, that has bugged me for ages!
<huwshimi> wgrant: Yeah, I finally got sick of it this morning :)
<michaelh1> wgrant: yeah, bzr-git converts any local branches to bzr branches in refs/branch-name.  It won't do remote branches, but you can set up a local branch that tracks the remote branch
<wgrant> michaelh1: Oh, doing an import like that, I see.
<wgrant> That's not how Launchpad does it.
<wgrant> huwshimi: Excellent. Approved.
<huwshimi> wgrant: Thanks mate!
<poolie> i was wondering in the shower if lp ought to show 'recently frequently duplicated bugs' as suggestions in +filebug
<poolie> and whether they would be better guesses, for some users, than matching on words in the suggestion
<poolie> or in fact hot bugs would be close to the same thing
<lifeless> well, the dup search should factor that in perhaps
<lifeless> OTOH many devs (me included) think offering dup detection is often misguided
<poolie> because the person filing the bug is not in a good position to know if it's a dupe?
<lifeless> yup
<lifeless> and superficial things like the backtrace are often not indicative of dupe-or-not
<poolie> i agree
<lifeless> sometimes they are, one-size doesn't fit all
<poolie> yep
<wgrant> That's not very Launchpad of you.
<lifeless> wgrant: ok, so you didn't like pgbouncer as a module name
<wgrant> lifeless: Correct.
<wgrant> Do I have suggestions beyond terribleness like pgbouncerfixture? Nope.
<wgrant> Hmm.
<wgrant> Let's play "make qastaging update faster".
<lifeless> wgrant: so, in the absence of suggestions, I'm going with pgbouncer... we can do a rename dance later if needed
<wgrant> Perhaps.
<poolie> could someone tell me briefly how lp's debian publication metadata importer works?
<wgrant> Perhaps we should make namespace packages not suck.
<poolie> or point me to a wiki page or code
<wgrant> poolie: grep around for gina
<wgrant> poolie: There's no wiki page that I know of.
<poolie> how does it get the data?
<wgrant> poolie: It is very unLaunchpaddy.
<poolie> is that a good thing? :)
<wgrant> poolie: debmirror runs on iron to create a local mirror. gina then walks the indices and imports everything it sees.
<wgrant> This means it fails at tracking removals etc.
<poolie> (for optional context, https://lists.ubuntu.com/archives/ubuntu-distributed-devel/2011-July/000872.html)
<wgrant> poolie: Sounds like our imports are badly timed.
<wgrant> I believe we import at the same frequency, but perhaps too offset.
<lifeless> well
<poolie> between the ubuntu imports and debian
<wgrant> 52 2,8,14,20 * * * sh /srv/debian-import.launchpad.net/scripts/mirror-update.sh >> /srv/debian-import.launchpad.net/production-logs/mirror-update.log 2>&1
<lifeless> folk have been known to upload the same version to both
<wgrant> 05 3,9,15,21 * * * LPCONFIG=gina /srv/debian-import.launchpad.net/production/launchpad/scripts/gina.py lenny squeeze sid experimental wheezy >> /srv/debian- import.launchpad.net/production-logs/gina.log 2>&1
<poolie> there seems to be a bug that codehosting oopses like https://lp-oops.canonical.com/oops.py/?oopsid=2021BZR207511 don't actually include a traceback
<poolie> or is that just a one off?
<lifeless> many others do
<lifeless> I don't know why that doesn't.
<poolie> oh well, bug 812106
<_mup_> Bug #812106: codehosting oops doesn't include traceback <codehosting> <oops-infrastructure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/812106 >
<poolie> can fix or investigate it if it ever actually gets in the way
<lifeless> poolie: please use the code not the URL for oopses - otherwise they get gc'd
<poolie> huh
<poolie> they get gc'd when there's no open bug that mentions it?
<lifeless> yes
<poolie> ok lunch
<lifeless> ok, EODing
<lifeless> till stub is around
<StevenK> wgrant: qas is WADLing
<wgrant> I sometimes wonder if the name was deliberate.
<StevenK> Haha
 * StevenK kicks vocabs more
<wgrant> StevenK: You don't happen to feel like qa/donotcareing bug #808680, do you?
<_mup_> Bug #808680: The initializeddistroseries job fails with "permission denied for relation distributionjob" <derivation> <qa-needstesting> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/808680 >
<wgrant> It's all that's blocking a 48 revision deployment.
<StevenK> qa-untestable
<wgrant> It only affects multiparent initialisation, so I'm tempted to donotcare it.
<StevenK> Same
<StevenK> Doit
 * wgrant doesit.
 * wgrant makes tea while qa-tagger updates.
<StevenK> wgrant: Do you need a hand putting the deployment request together?
<StevenK> 48? qatagger says 32
<wgrant> 15 ready to deploy, one blocker, 32 behind it.
<StevenK> The deployment report should say that
<lifeless> patch it up
<lifeless> its one of our codebases
<StevenK> Oh, sure. In the *oddles* of spare time I have :-P
<wgrant> Maybe I will decide I've had enough lazr-js for the day and try to fix it.
<StevenK> Besides, we can't. 1) We are not on maintaince. 2) Even if we were, criticals take priority.
<wgrant> lifeless: Are you in lpqateam?
<wgrant> Yes.
<wgrant> lifeless: Could you maybe change the crontab to regenerate lpqateam.canonical.com at 02,12,22,32,42,52, rather than a minute before qa-tagger?
<StevenK> Just what I wanted. A 131 line traceback
<wgrant> Oh goddammit.
<wgrant> There is a third revision involved in the mess.
<wgrant> I think it's just missing CSS, though.
<StevenK> If you say there is another rollback ...
<wgrant> With r13421 reverted, collapsibles created by activate_collapsibles (eg. "Extra options" on +filebug, "Configuration progress" on Product:+index) are no longer green.
<wgrant> Not sure we care for a day or so.
<StevenK> wgrant: File a critical/high, asign to Danilo, move on?
<lifeless> hi stub
<lifeless> stub: I have a present for you
<stub> Like your cat has a present for you?
<lifeless> better
<lifeless> https://launchpad.net/python-pgbouncer
<lifeless> stub: I wanted to simulate the 'cannot connect behaviour for the builddmaster
<lifeless> stub: and realised I had no idea what that meant pgbouncer wise, nor any way of programmatically testing it
<stub> I will have the answers to your questions today.
<lifeless> stub: so the new module bring up pgbouncer and should (as stands, or with minor tweaks - e.g. a reload() method to run pgbouncer -R) let us drive it programattically
<stub> There are two possibilities to what will happen - the connection is accepted by pgbouncer and is blocked, or the connection gets rejected. Depends on how much control pgbouncer gives me.
<lifeless> stub: FWICT dropping the user definitions (assuming we're using 'trusted') would block during connection
<lifeless> stub: that seems to be the only knob I could find that doesn't talk to the backend
<stub> lifeless: http://pgbouncer.projects.postgresql.org/doc/usage.html , in particular PROCESS CONTROLLING COMMANDS
<stub> I've tested PAUSE and it seems to do what we want, with inbound connections just blocking. I need to test SUSPEND in case we think it better to reject connections rather than queue them up for when things come back online.
<wgrant> stub: Oh, suspend rejects connections? :/
<stub> This is easy and reliable for me to control, as I just open a database connection to the internal 'pgbouncer' database and issue commands with psycopg2 or psql - no need to edit or swap config files in place.
<stub> wgrant: I don't know about suspend. if we don't want to reject, we use PAUSE.
<lifeless> stub: I think it better to reject
<lifeless> stub: unless we can guarantee < 5 second application times (because 5 seconds is the hard timeout for new pages)
<stub> wgrant: Blocking until the system is back up isn't necessarily the best choice - eg. it might be preferable for appservers to display a 'down for maintenance' message rather than queuing up requests until HAProxy displays a 'down for maintenance' message
<wgrant> Right.
<stub> lifeless: Yup. Anyway - I'm working on the script today on staging. I should also be able to reliably do 'sudo /etc/init.d/pgbouncer stop' or just kill the pgbouncer processes. That will also reject connections :-)
<lifeless> stub: indeed :P
<wgrant> I think this whole no-readonly thing is pretty hideous, but if we're going that direction then immediate notification is better than one based on a timeout.
<stub> lifeless: If you are playing now, just 'sudo pgctlcluster 8.4 main stop' and see what exception you get trying to connect to a database that is not there at all.
<lifeless> stub: anyhow, we can make the pgbouncer python module support the same thing, so we can test how the appserver etc handle it
<lifeless> stub: without pgbouncer its OperationalError: connection failure ...
<lifeless> stub: I can now play with pgbouncer of course :>
<stub> Which will be a subclass of psycopg2.Exception (Error?) - probably best to just catch that on connection attempts.
<lifeless> mmm perhaps; probably needs a code read if we want to be broad
<stub> Well, we don't really care *why* the connection failed. Unless you want to get into socket exceptions for things like no-route-to-host vs. nothing listening, but there are so many choices there we won't do full coverage.
<lifeless> indeed
<lifeless> uhm
<stub> esp. as they will be coloured by how the firewall rules are (mis)configured
<lifeless> I don't know if we do don't.
<lifeless> will all depend on the most (fast-and-reliable) way to stop pgbouncer letting stuff through
<stub> I think treating all connection failures the same, and monitoring when scripts don't run for extended periods of time, covers us.
<stub> The monitoring will catch the stuffups such as users not existing or dud firewall rules etc.
<lifeless> sure
<lifeless> I guess I have a small worry that we'll be down for 30 seconds or something with a maintenance page before anyone starts actually clicking to there being a problem when we didn't plan to go down
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 237 - 0:[######=_]:256
<lifeless> anyhow, this is minor and contingent on the actual implementation in pgbouncer anyhow
<jtv> Morning folks.  Any objects to my running process-accepted on dogfood?
<wgrant> jtv: None, but I think that bug is already QAed...
<jtv> You would know â but did you actually Q/A the new feature, or just test that it didn't affect the features we currently rely on?
<wgrant> --derived seems to behave as specified.
<jtv> That's great â thanks!
<wgrant> (I think it's probably dangerous, but we'll see)
<jtv> gulp
<jtv> Care to explain that?
<jtv> If it's dangerous, it may be good to be aware of it before we "see."
<wgrant> Given that the primary archive publisher is not meant to be bulletproof and unexploitable, automatically publishing every Ubuntu derivative on one machine seems... potentially unwise.
<wgrant> What with the whole running arbitrary shell scripts and stuff thing.
<wgrant> With paths configurable in the web UI, albeit only by ~admins at this stage.
<jtv> Arbitrary?
<wgrant> eg. things like Baltix are derivatives of Ubuntu, but probably don't want to go near bilimbi.
<poolie> lifeless: re the postmortem: why not just pop back to the last safe version, rather than landing a reversion?
<poolie> or rather than landing a reversion first
<lifeless> poolie: I'm talking about things we haven't deployed
<StevenK> poolie: If we do that then the deployment pipeline stalls for longer than 5 hours.
<poolie> ?
<poolie> ah, so it failed to deploy
<lifeless> no
<lifeless> failed qa
<wgrant> It failed QA, was not the head rev so had to be reverted.
<wgrant> Was fixed, deployed, broke production, rolled back, reverted. Another 5 hours.
<poolie> was it deployed or not?
<wgrant> The second time.
<wgrant> IIRC it failed QA the first time.
<lifeless> poolie: when it does get through to prod, we do just switch to the previously deployed version, but we still also land a rollback (async from fixing prod)
<poolie> ok i see
<lifeless> switching prod versions takes an hour, and then we're blocked on all deploys till the rollback gets through on trunk *and* we're fully qaed.
<lifeless> its a Real Big Deal, but fortunately fairly rare
<poolie> the rollback is to get trunk back to a state on which other changes can be landed
<wgrant> lifeless: Rolling production back to a previous rev is quick.
<wgrant> But still a big pain.
<lifeless> wgrant: takes a while to walk the server processes
<lifeless> wgrant: we don't have to stage code, buts its still kinda slow
<poolie> i guess splitting the test suite here could be good to
<poolie> to say that changes to bzr probably don't need all the malone pagetests run
<poolie> though, doing that manually would probably be tedious
<poolie> ok
<wgrant> poolie: Why don't they?
<wgrant> You would be unpleasantly surprised :)
<poolie> there is a good cost:benefit to running them?
<poolie> so is switching production versions "quick" or "an hour" or an hour is quick by lp deployment standards?
<lifeless> its a bit less than an hour
<wgrant> poolie: It's quicker than an hour, but an hour is quick.
<lifeless> buts it not quick
<poolie> :)
<poolie> so the timeline was:
<poolie> committed to devel; rolled out to qas; failed qa; attempted fix; fix rolled out to qas; passed qas(?); rolled out to lpnet; broke production; rolled back on production; reversion merged
<poolie> is that correct?
<lifeless> no
<lifeless> committed to devel, rolled to qa, failed qa, rolled back, recommitted with fix to devel, rolled to qa, passed qa, deployed, broke, production switched to prior version and deploys frozen, rolled back on devel, recommitted with fix to devel, ...
<lifeless> deployed to qa, deployed to prod
<poolie> ok
<poolie> so it seems like catching things on qastaging is only a little cheaper than catching them after that?
<poolie> it's still a lot of work but at least there is no production outage?
<lifeless> right
<lifeless> and we can deploy up to the bad commit
<lifeless> which (all too frequently) is useful - its a sign we're closing the queue too slowly
<poolie> what was that python docs -> web service jml used?
<poolie> rtfm something
<nigelb> readthedocs?
<spiv> Yeah
<poolie> yep thanks
<nigelb> poolie: the shortform, rtfd is more entertaining :)
<nigelb> oh, that was built in 24 hours o.O
<adeuring> good morning
<jtv> hi adeuring!
<adeuring> hi jtv!
<bigjools> morning all
<mrevell> Morning
<jtv> hi bigjools, hi mrevell!
<lifeless> allo allo ukians
<bigjools> any ever seen lp.services.job.tests.test_runner.TestTwistedJobRunner.test_memory_hog_job fail in ec2?
<bigjools> anyone*
<bigjools> I happened to have made changes to IJob in my branch but it passes locally
<jtv> I remember seeing _some_ failures in that areaâ¦ hang on, I'll search.
<StevenK> poolie: Are you still around?
<poolie> oui
<StevenK> poolie: bazaar-experts has had its celebrityness removed in prod, do you think it's worthwhile to delete the team in a few days?
<poolie> i do
<poolie> thanks for cleaning that up
<jtv> bigjools: not much luck so far, and I'm back more than a month.
<poolie> you might want to reply on the thread if it's not already clear
<bigjools> jtv: oh well
<bigjools> could be very spurious.  I'm re-landing.
<jtv> danilos: did you see bug 810309?
<_mup_> Bug #810309: Translator credits appear duplicated <Launchpad itself:Triaged> < https://launchpad.net/bugs/810309 >
<jtv> bigjools: I need to make some config settings for generate-contents-files; wgrant thankfully discovered something I had neglected to take care of.  Are dogfood and ftpmaster-publish the right, and complete, configs to edit?
<bigjools> jtv: should be
<jtv> OK, then up for review it goes.
<bigjools> what did he see?
<jtv> I didn't set the content_archive_root setting, which points the new script to where the (now legacy) ubuntu-contents dir is.
<bigjools> ah that
<wgrant> Apart from that it all looks good and a tad less awful than the shell script :)
<jtv> Once we've started using the new script, we can retire that item again; I have a separate bug filed about that.
<jtv> Thanks. :)
<wgrant> Plus it actually works.
<wgrant> jtv: Any reason not to start using the script as soon as it's on cocoplum, given that the old one doesn't work any more?
<jtv> Far as I know, only this one last point.
<jtv> (Of course there's usually _another_ one last point after that, but we can hope :)
<wgrant> heh.
<jtv> I'm trying to take it all in stages and be meticulous about filing kanban cards; seriously scared about missing something.
<lifeless> StevenK: remind me tomorrow and I'll nuke the team
<StevenK> lifeless: I think tomorrow is too early, but if you can do it, excellent
<lifeless> I own it now
<danilos> jtv, I didn't so far
<danilos> jtv, this used to be broken in the past but I believe we fixed it
<danilos> jtv, perhaps we only did so for translator-credits messages, and not for KDE messages
<jtv> I thought that difference was well-isolated though.
<jml> poolie: rtfd.org
<poolie> got it thanks
<jtv> danilos: I wonder if the sharing has anything to do with it.  It sounds like somebody changed the message's sharing status.
<danilos> jtv, well, it seems the "Shared" message is shown even if the message is read-only (there's a read-only icon), so there are a few problems there
<danilos> jtv, without looking at the code, I can't really say much else :)
<jtv> It'd be nice if we could get a maintenance squad on it soon, but I couldn't bring myself to mark it Critical.
<wgrant> I just closed like 9 criticals, so nobody will notice if you bump it :)
<jtv> That's quite a paceâ¦ the counter in the topic says 237; is that open ones or closed ones?
<wgrant> I opted not to update the topic again today; it was back up to 239 before the deploy, but should be down to 230 now.
<wgrant> We'll see soon.
<wgrant> 231, sorry.
<wgrant> Since I just filed a new one.
<jtv> Still, probably best to resist the temptation to generate Criticals at a higher rate.  :)
<wgrant> :(
<lifeless> StevenK: whats the reason for waiting ?
<StevenK> lifeless: That we can't revert to a pre-celebrity branch if we need to
<lifeless> well, could always recreate the team pretty quickly
<lifeless> but sure
<lifeless> happy to wait
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 233 - 0:[######=_]:256
<jtv> StevenK, wgrant: on mawson, I am now planning to (1) put ubuntu-contents back into /srv/launchpad.net, and (2) run generate-contents-files to see that it moves the directory back to where it is now.  Any objections?
<lifeless> is IDS in use yet ?
<jtv> Informix Dynamic Server?
<lifeless> initialise distro series
<jtv> It should be somewhere around the phase where it becomes accessible to colin only.
<jtv> (AIUI this is all behind the feature flag for the derived distros UI, which is not being enabled for the general public yet)
<nigelb> wgrant: 233/256, what is 256?
<nigelb> In progress or fix released?
<lifeless> countdown
<nigelb> countdown?
<lifeless> to 0
<nigelb> did the countdown reset from a particular date?
<nigelb> Essentially, why has the numerator changed while the denominator doesn't seem to change
<lifeless> its not a fraction
<StevenK> lifeless: Yes, IDS is in use
<lifeless> its a scalar and a meter
<nigelb> argh, right.
<nigelb> Sorry.
<StevenK> lifeless: Currently, it's only used for Ubuntu moving to the next release.
<wgrant> jtv: Sorry, was eating. Sounds good.
<jtv> I figured if nobody's responding, nobody's doing anything vital.  :)
<jtv> It's currently running.
<wgrant> Heh
<wgrant> /srv/launchpad.net/ubuntu-contents is gone, which can only be a good sign.
 * danilos tries to get expanders landed again...
<lifeless> danilos: the landing isn't the hard bit :)
<danilos> lifeless, well, sometimes it is
<lifeless> danilos: sure, I mean in this case
<StevenK> The landing is easy. Having it *stay* landed is hard :-)
<danilos> lifeless, well, the fix would have been landed sometime on Friday if there was not a test testing for the presence of the img tag
<danilos> lifeless, and no reversion would have had to happen :)
<lifeless> ah true
<danilos> of course, running lp.soyuz tests wasn't enough, so it was 5h of testing, and me being away from any computers over the weekend that made it impossible to go through :)
<danilos> anyway, this code has gone through ec2/buildbot so many times that it has been well tested by now :)
<cjwatson> lifeless: do your comments on https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema/+merge/67992 amount to an approval?
<lifeless> preferable for stub to do so
<lifeless> if he was on leave, they would.
<cjwatson> ah
<cjwatson> he did approve the version in https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations/+merge/67640 (admittedly before I renamed the column)
<cjwatson> but then I split out that change into a separate branch for the new schema changes process
<danilos> huwshimi, Hi, do you perhaps have a minute to discuss CSS sprites issue (bug 812044)?
<_mup_> Bug #812044: Expanded suggestions on +filebug leak branch icon <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/812044 >
<cjwatson> jtv-eat: if that Colin means me, I hope it'll actually be ~ubuntu-release rather than me personally
<cjwatson> otherwise opening p-series will be blocked on me, rather than in practice normally being done by me :-)
<huwshimi> danilos: Hey, sure
<danilos> huwshimi, if you've got any ideas, please add them to the bug
<huwshimi> danilos: Where can I see this bug in action?
<huwshimi> danilos: Is it in someone's branch?
<danilos> huwshimi, local LP instance (fresh devel), go to bugs.launchpad.dev/firefox/+filebug search for "a"
<huwshimi> danilos: Thanks
<jtv> cjwatson: don't worry too much â the "we'll let Colin see it" situation was always meant as temporary, for testing.
<cjwatson> ok
<LPCIBot> Project devel build #895: FAILURE in 5 hr 57 min: https://lpci.wedontsleep.org/job/devel/895/
<jtv> Can anyone help me figure out my bzr config for lp-production-configs?
<jtv> I'm getting this error from PQM: "Exception processing merge: unknown url type 'bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable'"
<jtv> No idea where either the "launchpad" (as opposed to lp-production-configs) and the "stable" come from.
<jml> jtv: use one of the debugging options on pqm-submit to figure out what email it's sending. (dry-run or verbose or something)
<jtv> thanks
<jtv> yup, it's --dry-run
<jtv> star-merge bzr+ssh://bazaar.launchpad.net/~jtv/lp-production-configs/config-810986 bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/lp-production-configs/trunk
<jtv> That's what PQM produces.
<bigjools> jtv: --submit-branch FTW
<jtv> bigjools: but then why doesn't the submit_branch setting in my locations.conf do the same?
<bigjools> jtv: is the branch in the location you think it is?
<jtv> I was wondering whether that mattered, given the incorrect path in the error message.
<jtv> I'm trying though.
<jtv> "bzr info bzr+ssh://.../config-810986"
<jtv> Yup, that works.
<jtv> Now trunk: that works as well (apart from some garbage output from bzr)
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge, benji | Critical bugs: 233 - 0:[######=_]:256
<jtv> Ahh, worked this time.
<adeuring> deryck: henninge: microphonde does not today... let try...
<deryck> abentley, ping for standup
<adeuring> can't even open the audio assistent...
<adeuring> no luck...
<deryck> adeuring, we're moving to skype, so I'll call you there for the standup/.
<adeuring> deryck: ok, just two minutes to strat another machine where skype is installed
<deryck> adeuring, ah, ok.  np.
<deryck> adeuring, we'll start and dial you in if we can.
<adeuring> deryck: I'm online on sykpe
<deryck> adeuring, you seem to drop immediately when abentley invites.
<abentley> adeuring: Skype says you're not online and it doesn't work when I invite you, either.
<adeuring> let me restart skype...
<adeuring> done
<deryck> adeuring, not working, sorry.
<abentley> adeuring: still not working.
<deryck> adeuring, we'll come back to you on IRC.
<adeuring> ok
<allenap> benji: Up for a short JavaScript review? https://code.launchpad.net/~allenap/launchpad/localpackagediffs-filter-by-package-set-bug-809786-refactor/+merge/68233
<benji> allenap: sure
<allenap> Thanks.
<benji> allenap: approved
<allenap> benji: Thanks :)
<benji> allenap: Invoking Y.use in a loop isn't something I've seen before.  (Not that it's bad, it just surprised me.)
<allenap> benji: I hope it works ;)
<abentley> henninge: I've merged from stable, but lp:~abentley/launchpad/json-serialization still has failures: http://pastebin.ubuntu.com/646546/
<henninge> hm
<henninge> let me look a little closer
<danilos> henninge, benji: anyone wants to pick up https://code.launchpad.net/~danilo/launchpad/bug-812044/+merge/68263 (60 lines of diff)
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 233 - 0:[######=_]:256
<henninge> danilos: I am not, sorry. ;)
<henninge> abentley: look at this anonymously: https://answers.qastaging.launchpad.net/
<danilos> henninge, you bastard! :)
<danilos> benji, hi, a very short branch up for review if you can find the time to look at it
<henninge> danilos, benji: nm me, I'll take it
<benji> k
<abentley> henninge: look at the mouseover text of why no "joe@example.com in question"
<henninge> abentley: I know ...
<henninge> abentley: I wonder why your test passes (or were you fixing that?)
<abentley> henninge: What test do you mean?
<henninge> abentley: the one from your paste
<abentley> henninge: That test fails.
<abentley> henninge: for me, anyhow.
<abentley> henninge: You mean why does it pass on trunk?
<henninge> abentley: oh, sorry, but other tests pass that should fail
<abentley> henninge: I have ignored email obfuscation because I knew you were working on that.
<henninge> in question-obfuscation.txt
<henninge> danilos: why did you not fix the jslint warnings?
<danilos> henninge, it's very old code, I just didn't bother, I can though
<danilos> henninge, I also wanted to avoid having to QA the file bug form extensively as well
<henninge> I think you should, unless it somehow relies on =='s casting behavior
<danilos> henninge, sure, I can do it
<henninge> danilos: thanks, otherwise it looks fine to me. r=me
<danilos> henninge, cool, thanks
<henninge> abentley: the test passes in devel. What is different in your branch?
 * henninge pulls stable
<abentley> henninge: In my branch, the anonymous user can see all cache objects, and the jsoncache rendering is done in LaunchpadView.getCacheJSON
<abentley> henninge: my bad, the anonymous user is still restricted to just the context: https://pastebin.canonical.com/49879/
<nigelb> mrevell: hi, you around?
<henninge> abentley: I am glad that's figured out
<mrevell> nigelb, I'm on the phone at the moment
<abentley> henninge: nothing is figured out.  I still don't know why the email address is not being obfuscated.
<henninge> abentley: oh
<henninge> misread you
<abentley> henninge: where did you do the obfuscation?
<henninge> abentley: in the marshaller
<henninge> abentley: maybe your code does not go through that?
<abentley> henninge: Does the marshaller not provide ResourceJSONEncoder?
<henninge> abentley: don't think so (I never heard of it)
<henninge> abentley: the marshaller is for a specific field type
<abentley> henninge: The ResourceJSonEncoder is provided by lazr.restful.tales.
<abentley> henninge: Okay, so where is the marshaller and how is it used?
<henninge> abentley: lib/lp/app/webservice/marshallers.py
<henninge> abentley: zcml/override-includes/ws-marshaller-configure.zcml
<henninge> abentley: I am sorry, I gotta run now.
<abentley> henninge: okay.
<henninge> abentley: the lazr-restful code does the apdapter lookup
<henninge> abentley: adeuring should know all about that, too
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #896: FIXED in 5 hr 35 min: https://lpci.wedontsleep.org/job/devel/896/
<jcsackett> sinzui: time to chat?
<jcsackett> benji: any time for a review? https://code.launchpad.net/~jcsackett/launchpad/form-macros-frighten-me/+merge/68277
<benji> jcsackett: sure thing
<jcsackett> thanks!
<benji> jcsackett: I take it that there's some sort of quoting issue that neccesitated using \x22 for the double-quote character on line 42 of the diff.
<jcsackett> benji: i believe so. almost all that code is as it was written before, just moved out of the template and into a js file.
<benji> we'll let sleeping escape sequences lie
<benji> jcsackett: add done
<jcsackett> thanks, benji.
<benji> my pleasure
<sinzui> jcsackett, sorry, I did not see you message. I can chat now if you like
<jcsackett> sinzui: sure. be on mumble in a moment.
<abentley> gary_poster: For anonymous access to the web site, is request.principal None or does it provide IUnauthenicatedPrincipal?
<gary_poster> abentley, should provide
<gary_poster> abentley, sorry just saw this :-/
<abentley> gary_poster: np.
<abentley> gary_poster: in doctests such as lib/lp/answers/stories/question-obfuscation.txt, it appears request.user may be None.  Is that a bug?
<gary_poster> abentley, I've seen that stuff in out doctests before.  I believe it is a bug in our test infrastructure, yes.
<gary_poster> abentley, that, or the test is representing an incomplete state.
<gary_poster> that's doubtful to be intentional though
<abentley> Should (view.user is None) ==  IUnauthenticatedPrincipal.providedBy(request.principal) ?
<gary_poster> mm...
<gary_poster> abentley, Zope itself does not have a contract for the view other than __call__.  It has a convention of 'request' and 'context' but they are not required from the perspective of the publishing or security machinery.  I'm not sure what in Launchpad sets up view.user; I expect LaunchpadView.  I'll look there for a moment
<gary_poster> abentley, user in LaunchpadView is getUtility(ILaunchBag).user
<gary_poster> I'll look there now
<gary_poster> abentley, there are multiple implementations of the ILaunchBag, including at least three test stubs, one of them apparently intended to be general purpose (lib/canonical/launchpad/tests/test_helpers.py)
<gary_poster> abentley, that one defaults to user=None
<gary_poster> abentley, so that is at least one case in which tests would have a user of None.  I'll look at the "real" implementation now...
<gary_poster> abentley, yes, it looks like the launchbag's user can be None, it the principal cannot be adapted to IPerson
<gary_poster> and I expect we do not have an IPerson implementation for anonymous
<gary_poster> abentley, I'm stopping looking now. :-)
<abentley> gary_poster: I'm actually experiencing the case where request.user is None, so it doesn't provide IUnauthenicatedUser, which is causing it to be treated as an authenticated user.
<abentley> gary_poster: thanks for having a dive into it.
<gary_poster> abentley np.  As I said, I suspect that's a test infrastructure unpleasantness.  I'm pretty sure I could find the Zope code that stuffs an unauthenticated principal on the request if you said that would be helpful.
<abentley> gary_poster: I'm not sure what the right way to proceed is.  Is IUnauthenticatedPrincipal.providedBy(request.principal) the best way to detect whether the user is authenticated?  I guess I could ask the LaunchBag instead...
<gary_poster> abentley, within the context of Launchpad, and Launchpad tests, the LaunchBag would be the path of least resistance, I think, given what you've seen.
<gary_poster> and not a horrible one--plenty or precedent
<gary_poster> of
<abentley> gary_poster: Okay, thanks very much.
<gary_poster> np
<sinzui> flacoste, ping
<lifeless> good morning :)
<jkakar> Hiya lifeless. :)
<lifeless> hi jkakar
<flacoste> hi sinzui
<sinzui> flacoste, do you have a few minutes to mumble about bug linking/dependencies?
<flacoste> sinzui: i'm on the phone already, will be a long one, i'll ping you afterward
<sinzui> thank you
<lifeless> sinzui: is that the call you wanted me on ?
<sinzui> lifeless, no. I think I know you you and OEM think my team is doing dependencies. There are TWO bug linking features planned in the next 3 months. It is very confusing and they obviously overlap
<lifeless> win
<wgrant> I guess I should revert that sendbranchmail thing.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 237 - 0:[######=_]:256
<wgrant> conflicts :(
<wgrant> allenap: I guess we should not deploy anything between r13460 and r13464?
<flacoste> lifeless: skype?
<lifeless> ringing you
<lifeless> (yes)
<flacoste> lifeless: yeah, skype crashed
<lifeless> \o/
<flacoste> lifeless: can you try again?
<flacoste> sinzui: can i skype you in with lifeless?
<lifeless> sinzui: is your skype working ?
 * sinzui thinks so
<mwhudson> wgrant: have you spent the week i've been on leaving reverting the same branch over and over again?
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[######=_]:256
<wgrant> mwhudson: No, several different ones :(
<wgrant> I've never reverted any one more than three times, though.
<mwhudson> there's a first time for everything i guess
<wgrant> And they only sometimes get ontoproduction and break it in a different way each time :)
<mwhudson> i guess that's a bit less terrible than breaking production the same way each time
<wgrant> They do that sometimes too.
<LPCIBot> Project db-devel build #730: FAILURE in 5 hr 37 min: https://lpci.wedontsleep.org/job/db-devel/730/
<LPCIBot> Project devel build #897: FAILURE in 5 hr 37 min: https://lpci.wedontsleep.org/job/devel/897/
<sinzui> StevenK, mumble?
<poolie> hullo sinzui
<sinzui> ahoy oy poolie
<poolie> wgrant: hi, following on from yesterday's question
<poolie> do you think it would be feasible to change lp so that it always exposes debian publications before it does the derived ubuntu packages?
<wgrant> poolie: No.
<wgrant> It's impossible to determine.
<wgrant> And any approximation would just delay and anger people.
<poolie> ... because it will mostly end up delaying the ubuntu package record, and that's often not needed and annoying?
<wgrant> Yes.
<wgrant> I'm not sure anybody would consider making Launchpad slower to be a feature.
<poolie> :)
<poolie> if we want to take the course of delaying the ubuntu package, we could do that only in udd where it's needed
<wgrant> poolie: We should do two things:
<wgrant>  - Get the Debian import to be more up to date.
<wgrant>  - Fix UDD.
<poolie> so, more up to date means running it more often?
<wgrant> Or perhaps merely at more appropriate times.
<wgrant> But more often is always good.
<wgrant> No reason not to.
<poolie> it's not outrageously expensive or slow?
<wgrant> debmirror should be trivial if the indices haven't changed, and if they have changed then we want to reimport.
<poolie> right
<wgrant> And iron doesn't do anything useful.
<poolie> do you know how frequent it would be now, or should i ask a losa what's in the cron job
<wgrant> I pasted the crontab lines yesterday.
 * wgrant finds.
<wgrant> 52 2,8,14,20 * * * sh /srv/debian-import.launchpad.net/scripts/mirror-update.sh >> /srv/debian-import.launchpad.net/production-logs/mirror-update.log 2>&1
<wgrant> 05 3,9,15,21 * * * LPCONFIG=gina /srv/debian-import.launchpad.net/production/launchpad/scripts/gina.py lenny squeeze sid experimental wheezy >> /srv/debian-import.launchpad.net/production-logs/gina.log 2>&1
<poolie> where did you find them?
<StevenK> lp-production-crontabs, I daresay
<wgrant> Yes.
<wgrant> iron-launchpad in lp:lp-production-crontabs
<poolie> perhaps eventually it would be good to have something that just runs continuously rather than from cron
<poolie> or, continuously with a short pause in between
<wgrant> Given what it has to do, that's no better than a */1 cron job.
<poolie> i guess so
<poolie> because they all have lock files to prevent concurrent runs?
<wgrant> No, but they would if we bumped it to */1.
<wgrant> The two scripts would run as a single frequent job, with locking, and only executing gina if the mirror changed.
<poolie> so if we set it to /1 at the moment they would get into trouble, but if we added a lock file they would cope?
<wgrant> Right.
<lifeless> poolie: also because folk directly upload sometimes
<poolie> ?
<wgrant> Occasionally.
<lifeless> poolie: ubuntu having a version before debian is a normal, if rare, situation
<poolie> right
<lifeless> changing how we import debian won't avoid it
<poolie> well
<poolie> reducing the latency will reduce the number of confusing situations
<poolie> it can't totally avoid it
<poolie> ok bug 812597
<_mup_> Bug #812597: gina imports from debian arrive too slowly <gina> <udd> <Launchpad itself:Triaged> < https://launchpad.net/bugs/812597 >
<wgrant> poolie: Are you sure that's the case?
<StevenK> poolie: The Debian archive is only published every six hours, whereas Ubuntu is published every hour.
<StevenK> *And* we do the possibly of syncing from Debian incoming if we wish.
<StevenK> s/do /do have /
<wgrant> http://webnumbr.com/launchpad-critical-bugs
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[######=_]:256
#launchpad-dev 2011-07-19
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256
<wgrant> sinzui: Did you typo your MUA's display name?
<wgrant> You've lost your capital "c"
<wgrant> sinzui: Nice presentation.
<poolie> wgrant: ?
<poolie> i filed https://bugs.launchpad.net/launchpad/+bug/812597
<_mup_> Bug #812597: gina imports from debian arrive too slowly <gina> <udd> <Launchpad itself:Triaged> < https://launchpad.net/bugs/812597 >
<wgrant> poolie: It would only be 7 hours if Debian decided to shift their publishing schedule to right after gina runs.
<wgrant> They still only publish every 6 hours, AFAIK.
<poolie> i see
<poolie> and we're pretty sure we're in sync with that?
<StevenK> It requires checking
<wgrant> The current schedule was devised to run just after they published.
<wgrant> But they like to change.
<wgrant> Without telling anybody.
<persia> Doesn't Debian have a push-mirror feature for some mirrors that runs right after publication?  Could one host a mirror, and then use post-mirroring hooks to call gina at the right times?
<StevenK> wgrant: The amount of QA to do makes me sad.
<huwshimi> hmm... is there any way to test emails on qa staging?
<lifeless> yes
<lifeless> for in you need losa to run the script by hand
<lifeless> for out you also need to run the relevant script by hand
<lifeless> but then you can access the shared mailbox to see what was sent out
<wgrant> sinzui: Anything to watch out for in oneiric atm?
<lifeless>  dev/run, ecryptfs were fragile recently, may not be totally sorted yet
<lifeless> wgrant: what presentation ?
<wgrant> lifeless: sinzui's velocity presentation from the Prague epic.
<lifeless> ah yes
<lifeless> +1
<wgrant> I use LUKS rather than ecryptfs, because it's flaky and slow.
<huwshimi> lifeless: Ok, I would like to get a notification of a tag change from a bug. What scripts do I need to run? Is there a wiki page for this?
<wgrant> StevenK: QA!
<StevenK> wgrant: I have a better idea. LUNCH!
<wgrant> I was just doing a pre-lunch QA check myself.
<StevenK> I shall update mawson now. And then break for lunch.
<wgrant> Thanks.
<wgrant> Huh.
<wgrant> Julian used one of Job's YAGNI columns.
<StevenK> Indeed
<lifeless> which one ?
<StevenK> requester
<wgrant> Hm.
<wgrant> Has someone decided to show the battery indicator at all times in oneiric? :(
 * wgrant blames thumper.
<thumper> heh
<thumper> wgrant: not me though
<thumper> but at least I have a battery indicator now
<wgrant> Bah.
<wallyworld> thumper: global menus fixed yet?
<thumper> nah mate
 * wallyworld sad
<wgrant> wallyworld: Hm, we have a picker regression.
<wallyworld> ?
<wgrant> https://qastaging.launchpad.net/launchpad/+filebug, the "Assign to" picker has quotes around the text.
<wgrant> Not there on prod.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #731: FIXED in 5 hr 31 min: https://lpci.wedontsleep.org/job/db-devel/731/
<wgrant> Normal bug assignee pickers are OK.
<wallyworld> https://code.launchpad.net/~wallyworld/launchpad/picker-displays-null-812253/+merge/68318 should fix it
<StevenK> wgrant: How do you suggest I QA r13455?
<wgrant> StevenK: Reject a delayed copy.
<wgrant> wallyworld: Ah, great.
<StevenK> That means I need to create one
<wgrant> StevenK: Easy enough to do.
<StevenK> And delayed copies make me cry.
<wgrant> Yes
<StevenK> wgrant: So copy something from the soyuz-team P3A to somewhere else?
<wgrant> StevenK: Something that's not already public, yes.
 * StevenK digs
<wgrant> Why is syndaemon enabled by default :/
<wgrant> How stupid.
<StevenK> Delayed copy of launchpad-dependencies - 0.39 (source)
<StevenK> Good enough
<StevenK> wgrant: But I can't find that delayed copy using queue?
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #898: FIXED in 5 hr 34 min: https://lpci.wedontsleep.org/job/devel/898/
<wgrant> StevenK: You'll have to get the ID manually, I guess.
<wgrant> Hopefully that will work.
<StevenK> wgrant: scripts/ftpmaster-tools/queue reject -s hardy -Q accepted 2752292 worked fine
<StevenK> Running: "info 2752292"
<StevenK> Item 2752292 is in queue REJECTED
<wgrant> Great.
<wgrant> qa-ok, I s'pose.
<StevenK> Already done
 * StevenK kicks bigjools for the fun that is r13459
<wgrant> StevenK: It is flagged.
<StevenK> Then what's left?
<wgrant> lifeless: LXC works fairly well in oneiric.
<wgrant> With most of those bugs fixed.
<wgrant> StevenK: Check that the methods don't work, I guess.
<StevenK> wgrant: For r13459?
<wgrant> Yes.
<StevenK> And lib/lp/soyuz/interfaces/archive.py contains the generic exception ForbiddenByFeatureFlag.
<StevenK> Fail.
<wgrant> Did somebody break the JS build again?
<StevenK> Oh?
<wgrant> Ah, no, I just killed buildout in a bad place.
<lifeless> wgrant: I'm hoping my sudo bug is fixed
<wgrant> lifeless: Not sure.
<wgrant> StevenK: Erm.
<StevenK> wgrant: Hm?
<wgrant> StevenK: [DOGFOOD] [PPA stevenk] [ubuntu/hardy] launchpad-dependencies 0.39 (Rejected)
<wgrant> There was a blamer.
<wgrant> Or SPR creator, possibly.
<wgrant> flacoste:
<StevenK> That is the creator
<StevenK> wgrant: Neither copyPackage{,s} appears in the API on dogfood for devel :-(
<wgrant> StevenK: You'll probably need to force it to rebuild the WADL
<wgrant> StevenK: But this is to be tested on qastaging.
<StevenK> Oh, easy fixed
<StevenK> wgrant: On qas, I get a 403 calling IArchive.copyPackage()
<wgrant> Good.
<StevenK> Ah yes, the exception is 403
<StevenK> Shall I qa-ok it?
<wgrant> StevenK: Might as well.
<StevenK> GAVIN!
<StevenK> (The 3 branches linked to bug 809786)
<_mup_> Bug #809786: Need to filter by package-set on DistroSeries:+localpackagediffs <derivation> <qa-needstesting> <ui> <Launchpad itself:Fix Committed by allenap> < https://launchpad.net/bugs/809786 >
<StevenK> 518 lines, 4720 lines, and 2878
<StevenK> Curious.
<wgrant> StevenK: Hm?
<StevenK> wgrant: I finally got +localpackagediffs to load, but I can't tell what Gavin changed/fixed
<wgrant> StevenK: It was pure refactoring that's landed so far, AFAICT.
<StevenK> All 6000 lines of it
<StevenK> wgrant: So qa-untestable?
<wgrant> Probably.
<StevenK> wgrant: So the only thing left is Danilo's revision of doom
<adeuring> good morning
<jam> morning all
<jam> I'm trying to understand an LP api timing. I could have sworn I was doing the same request last week, and it was returning in 100-200ms.
<jtv> hi jam, hi adeuring!
<jam> But this week it is more like 700-1000ms
<jam> hey jtv
<jam> is there any way to debug these sort of things?
<adeuring> hi jtv
<wgrant> jam: What sort of request?
<jtv> Can we use ++profile++ on API requests?
<wgrant> jam: 100-200ms is rather quick for LP.
<wgrant> jtv: Possibly... ++oops++ doesn't work, since it only executes when the template renders.
<jam> wgrant: getPublishedSources
<wgrant> jam: Let's see.
<wgrant> Everything should be faster this week :(
<jtv> That's on Archive?
<wgrant> Could be distroseries.
<wgrant> Or distribution.
<wgrant> I guess.
<allenap> StevenK: Thanks for looking at the changes to +localpackagediffs. As long as it loads, it's fine. The changes so far are almost all moves and shakes.
<StevenK> allenap: GAVIN! *Scary* branches! RARGH!
<StevenK> allenap: But yes, it at least loads
<jam> wgrant: https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1
<jam> Archive
<jtv> Yuck.  "clauses.append("SourcePackageName.name LIKE '%%%%' || %s || '%%%%'" % quote_like(name))"
<wgrant> But this is exact_match.
<wgrant> So that should be skipped.
<wgrant> (it also shouldn't exist at all, but such is Soyuz)
<jtv> Do we actually know we're looking at the archive code, actually?
<wgrant> Yes.
<wgrant> debian/+archive/primary
<wgrant> jam: Render time is just about identical compared to 1.5 weeks ago.
<wgrant> jam: Do you have any hard data to support the slowdown?
<jtv> The only recent direct change in that piece of code was in June, where it looks like multi-name matching was introduced.  But that should barely touch the exact-match code path.
<jtv> Unless the thing gets passed some kind of placeholder that acts like a string but isn't of type str or unicode, but I don't really know whether that's possible or whether it would work if it was.
<wgrant> AND SourcePackageName.name = E'bzr' AND
<mrevell> Good morning
<wgrant> From using that URL locally.
<bigjools> morning al
<bigjools> l
<jam> wgrant: I know I did timing tests last week, and had some perf tests.
<jam> but no, I don't have very hard data.
<jam> wgrant: take that back, I have my .bzr.log
<jam> from today: 6.778  LatestPublication.get_latest_version took 1.524s
<wgrant> Oh, this is for the checking if the branch is up to date thing?
<jam> from last week: 7.974  LatestPublication.get_latest_version took 0.214s
<jam> wgrant: yep
<wgrant> Have you moved to Australia lately?
<jam> so the ones I was testing before is "ubuntu/natty/bzr"
<wgrant> Oh.
<wgrant> That's a bit different :)
<jam> but that is still slow
<jam> 700ms orso
<wgrant> :(
<stub> jtv: Storm contains a Like operator that might spell that better if you are coding nearby.
<jam> just tested now 939ms
<jtv> stub: I'm not, but that's part of what made me nauseous.
<jtv> It's even easier than Like; IIRC you can use the regular str methods such as startswith.
<jtv> There's one for "contains" as well.
<StevenK> I recently used .contains()
<jam> wgrant: with bzr.dev tip, you can run this script: http://paste.ubuntu.com/647087/
<stub> even better
<wgrant> wgrant@carob:~$ time curl -H 'Authorization: there is none' 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null
<wgrant> real	0m0.253s
<jam> (python latest_version.py ubuntu natty bzr)
<jtv> Someone went to the trouble of coding up a hybrid storm/sqlobject method so that they could write up that horrible LIKE clause.
<StevenK> jtv: Can you point out the method so we can have it shot?
<wgrant> jam: So, it takes around 250ms from the DC.
<jtv> StevenK: Archive.getPublishedSources
<wgrant> jam: For natty it's down around 150ms, as expected.
<wgrant> Since there's a lot less data.
<bigjools> StevenK: please don't shoot that one
<wgrant> (this isn't exactly optimised)
<StevenK> It deserves it.
<jam> wgrant: I'm seeing 600ms in "_ssl.sslwrap" is that the SSL handshake ?
<wgrant> jam: I assume so.
<jam> so that is very different from last week, I believe
<wgrant> Sure your connection latency doesn't suck?
<jam> wgrant: I'm in Netherlands
<jam> ping says 36ms
<wgrant> How long does the curl I listed above take?
<jam> 321ms I believe
<wgrant> And then try a couple of times without the Authorization header.
<wgrant> (the presence of even an invalid one gets it past Squid)
<jam> wgrant: but that is because curl fails immediately with "SSL FAILED"
<jtv> morning bigjools!
<wgrant> Helpful.
<bigjools> hello jtv
<jtv> jam: want to meet up later this week?
<jam> but I can pass -s, I guess
<jam> I still don't get any output to tell if it is working
<wgrant> jam: Well, probably better to make it work rather than ignoring the error...
<jam> :)
<jam> I wasn't sure what -s did
<jam> apparently it is --quiet not "--ignore-certificate"
<jam> wgrant: with -k instead of -s, I get 892ms
<wgrant> I get no cert error locally or on carob.
<jam> wgrant: launchpad isn't in my cacerts
<jam> -k works fine for testing
<wgrant> Mine are stock Natty.
<jam> so, still the same time for a successful request, about 900ms, which could be 600ms in ssl handshaking
<jam> which is definitely slower than last week
<jam> I don't know why
<jam> is there a graph/monitoring of ssl stuff?
<wgrant> Internally it's fine.
<jam> wgrant: right, so from devpad, I see 148ms
<wgrant> 900ms from 150ms away
<wgrant> So 650ms overhead, which is about four ways...
<jam> wgrant: I'm 38ms away by "ping"
<jam> so 17 round tirps
<jam> trips
<wgrant> Odd.
<wgrant> Still, fine in and next to the DC.
<wgrant> So I am tempted to blame your machine/connection.
<wgrant> .nl hasn't imposed a Great Firewall lately? :)
<wgrant> :(
<wgrant> zope.app.testing fixed their Referer default.
<wgrant> So 350 or so tests fail with OffsiteFormPost errors.
<jam> vila: can you try the query from France?
<vila> otm
<jam> time curl -k 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -o test.txt
<rvba> jam: real	0m0.532s
<rvba> (from Paris)
<wgrant> rvba: Is that after a couple of tries?
<rvba> real	[0m0.362s - 0m0.532s]
<rvba> wgrant: that's the range I get
<rvba> After a couple of tries: real	0m0.344s
<wgrant> That's a bit better.
<jam> wgrant: of course *right* now it just dropped to 300ms, but now curl varies from 300-953ms
<jam> and it still is taking 700+ from python
<jam> which still doesn't get close to the 150ms from last week
<wgrant> Was this on the same machine in the same environment?
<jam> yes
<jam> actually, the only difference offhand is that now I'm on a wired connection to my router, while last week I was on wireless
<jam> I guess it was 200ms last week, which is close to 300.
<jam> the variability is surprising
<jam> and last week was python, not curl
<wgrant> I get 50ms from squid.
<wgrant> Which sucks, but it's always within 1ms.
<jam> what do you mean by "from squid" ?
<StevenK> We have a proxy in front of the appservers
<jam> StevenK: sure, but is he saying bypassing the squid proxy, the appservers respond in 50ms
<jam> or from the squid machine
<jam> or he gets 50ms making a request to squid without an appserver
<StevenK> No, he is saying that squid replies in 50
<wgrant> 50ms to get the copy that squid has cached.
<wgrant> ie. after requesting a few times without Authorization
<jam> wgrant: my requests also were without auth, but you're saying from where to squid?
<wgrant> carob to squid
<wgrant> Via the usual HTTPS frontends
<wgrant> wgrant@carob:~$ time curl 'https://api.launchpad.net/1.0/ubuntu/+archive/primary?distro_series=%2Fubuntu%2Fnatty&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null
<wgrant> real	0m0.050s
<wgrant> (Debian is, unsurprisingly, identical)
<jam> sure. Though that just dropped as well
<jam> it was 150ms for me when I first reported devpad numbers
<jam> even running a couple of timse
<jam> times
<jam> wgrant: so I still think there is something that is slowing queries down, that magically disappeared in the last hour
<jam> eg, we're missing some monitoring somewhere that would tell us something is going on for people actually getting a much slower launchpad
<wgrant> Even in front of squid?
<jam> wgrant: I ran the same query, and it was 150ms from devpad to api.launchpad.net
<wgrant> Hmm.
<jam> I think I managed to provoke squid to skipping just now, and I got 400ms
<jam> (that was to debian, though)
<jam> but I'm playing with setting Authorization, and I'm getting 50ms
<vila> jam: first try: 1.355s real, then 0.249s, 0.199s, 0.194s, 0.189s , 0.198s
<jam> so it sounds like squid doesn't actually pay attention to it.
<jam> vila: right, so those sound like cached and uncached responses.
<vila> further tries stabilize at 0.200s
<jam> and arguably we should be really counting the uncached values as our user-experience.
<jam> we might need to put this behind a flag
<jam> vila: I couldn't figure out what you meant by OTM, on-the-mumble?
<vila> jam: yeah :)
<wgrant> jam: Apache is configured to send anything with Authorization or Cookie straight to haproxy, not squid, AFAIK.
<jam> wgrant: tell that to my curl results :)
<jam> though I could be typing something wrong
<jam> ah, Authentication not Authorization
<jam> my bad
<jam> right, that does bypass it
<jam> that's 479ms stabilizing to 230ms on carob bypassing squid requesting debian urls
<jam> which is hopefully worst-case in DC times
<jml>  I clicked "Build now" on https://code.launchpad.net/~testing-cabal/+recipe/testtools-daily and get what looks to be a JS alert saying "Server error, please contact an administrator"
<jml> what's going on?
<jam> wgrant: ok, this is weird. curl is going to 91.189.89.224, but python is going to 91.189.89.225
<jam> and *that* seems to be the difference in timings
<jam> it is "banana" vs "nutmeg"
<StevenK> Yes, they are two frontends
<jam> StevenK: any idea why banana would be 300ms slower than nutmeg?
<wgrant> Both 50ms cached...
<wgrant> And both 200-250ms uncached.
<lifeless> stub: hi
<stub> lifeless: hi
<lifeless> stub: I thought I'd touch base on 'kick connections out and keep em out' implementation choices
<bigjools> anyone got any ideas on where to look to investigate why an api method I added is not available over the wire to dogfood?  It's in dogfood's WADL but launchpadlib  says "'Entry' object has no attribute ...."
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 238 - 0:[######=_]:256
<jam> wgrant: how do you get curl to target a server?
<jam> can you just do the IP address, or are they vhosts?
<jam> (I get ACCESS DENIED going to the IP address.)
<wgrant> jam: They're vhosts on special IP addresses.
<wgrant> time curl -H 'Authorization: blah' -H 'Host: api.launchpad.net' 'https://91.189.89.225/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -k -o /dev/null
<jam> wgrant: thanks. So bypassing SQUID going to ubuntu/natty I get a reliable 470-530ms to either server (nutmeg or banana)
<jam> and without Authorization, I get 330-360 from curl
<jam> so something about python is provoking something
<stub> lifeless: I think we are shutting down pgbouncer, which kills all the connections with 'connection terminated unexpectedly' exceptions and future connection attempts will timeout as nothing is listening. The 'suspend' and 'pause' don't terminate the active connections and ignore the 'fail if unable to login in this many seconds' parameter.
<lifeless> stub: ok
<lifeless> stub: So we'll need the test fixture to be able to shut down and bring back the one config; I'll add that into a 0.2 tomorrow if noone else gets to it
<gmb> lifeless: I don't know if you'll get chance, until your tomorrow, but I've replied to your review of https://code.launchpad.net/~gmb/launchpad/bug-491886/+merge/68174.
<stub> lifeless: We want a pgbouncer test fixture?
<lifeless> stub: we have one
<lifeless> stub: the project I put together on Monday
<lifeless> stub: so we can test this well - we don't need the whole test suite to use pgbouncer, just the tests for interrupt & reconnect detection
<stub> Right. We used to have some disconnection/reconnection tests that used a simple twisted proxy, but this moved to Storm. We don't need to use pgbouncer if we don't want - as far as PG clients are concerned, they just have a socket.
<lifeless> stub: its pretty quick to bring up, and the code is written
<lifeless> stub: I suspect we only need 3-4 tests per daemon
<bigjools> lifeless: any chance you can help me debug a webservice problem on dogfood please?
<bigjools> I know it's rather late for you
<lifeless> sure
<lifeless> sec
<lifeless> gmb: reviewed
<lifeless> gmb: generally, I'm happy for any other reviewer to do a follow up once you consider (and action or reject) my suggestions
<gmb> lifeless: Thanks, and duly noted for the future.
<lifeless> gmb: I don't think you need to consider the review pinned to me (unless I specifcally say I want that)
<gmb> Okiedoke.
<lifeless> gmb: I think the try:except: looks a lot simpler :)
<lifeless> bigjools: whats up ?
<gmb> lifeless: Agreed. I was trying to avoid the transaction.abort() but simplicity beats I-don't-want-to-have-to-deal-with-transactions.
<bigjools> lifeless: I added IArchive.copyPackage to the API. It's on DF's wadl/+apidoc but launchpadlib can't see it :/
<lifeless> bigjools: delete your lp lib cached
<StevenK> bigjools: Remove your cached WADL?
<bigjools> GAH
<bigjools> this old chestnut
<bigjools> why why why
<lifeless> if it works, file a bug
<lifeless> I would -so- like to nuke the use of WADL, but I'm old school.
<bigjools> anything with XML in it reeks of shit
<StevenK> It lends itself so well to ridicule, though. "qas is WADLing"
<lifeless> gmb: oh, and the linty stuff I don't care about really, I mentioned it for completeness
<bigjools> yeah that worked.  thanks lifeless
<lifeless> bigjools: file a bug ?
<bigjools> yeah doing so
<lifeless> \o/
<bigjools> but on where
<lifeless> launchpadlib
<lifeless> its the thing choosing to use wadllib etc; and its the thing we need a versioned dep on to ensure a newer wadllib if thats how we fix it
<cjwatson> could somebody help me land https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema?
<bigjools> lifeless: bug 812799
<_mup_> Bug #812799: Local cache not updated when it should be <launchpadlib :New> < https://launchpad.net/bugs/812799 >
<lifeless> bigjools: triage it ! :)
<lifeless> [I suggest high]
<bigjools> lifeless: jeez, gimme time to fart first
<lifeless> bigjools: well, you're old and all, I don't know I have *that* much time to wait :P
<lifeless> bigjools: [seriously, when you linked it I figured that meant you'd done all you intended to on it]
<bigjools> lifeless: my arthritic fingers at my old age don't work as quick as that :)
<stub> lifeless: pgbouncer might help test suite race conditions tearing down/creating databases
<bigjools> awesome, feature flags are case-sensitive
<stub> mmm... probably not. its more pg tripping over its own feet and waiting for its internals to catch up rather than inbound connections
<lifeless> stub: if it could force only one connection at a time to template1
<stub> lifeless: it could do that...
<lifeless> stub: anyhow, my patch for that seem to work, and we're looking at lxc for parallel in the short-medium term anyhow, now.
<lifeless> because there are so many similar glitches in our stack at the moment
<lifeless> [e.g. launchpadlib test cache dir]
<jml> cjwatson: sure.
 * wgrant curses Python
<wgrant> Don't make backwards incompatible changes in 2.7 without a non-monkeypatch way to restore the old behaviour.
<StevenK> wgrant: Hm?
<wgrant> The default continuation whitespace character in generated email headers changed from tab to space.
<wgrant> And this is set in an _method of a class.
<wgrant> So it's not really overridable.
<jml> Yeah, python-devel seems to hate backwards compatibility
<jml> Hmm.
<wgrant> Ah, mercurial indeed fix it by monkeypatching.
<jml> It turns out I'm not able to land branches any more, because I'm not allowed to edit commit messages on merge proposals for Launchpad.
<wgrant> Yeah, the emeritus stuff isn't really set up yet.
<wgrant> Most people are still in ~launchpad.
<wgrant> Want me to land it instead?
<jml> wgrant: sure. here's the line I'm using:
<jml> ./utilities/ec2 land https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema/+merge/67992 --commit-text="Add include_long_descriptions column to DistroSeries to enable future work separating long descriptions out of Packages file" --no-qa
<jml> wgrant: I would like my commit access back too please
<wgrant> :(
<cjwatson> shall I set the commit message in the MP?
<wgrant> It's done.
<wgrant> ec2 is almost running.
<cjwatson> thanks
<jml> cjwatson: even if it were set, I would need privileges to edit it, since our landing tools push some metadata onto it.
 * cjwatson nods
<wgrant> Down to 12 2.7 test failures, plus two tests that hang.
<wgrant> Nearly there.
<jml> \o/
<jml> wgrant: so the plan is to switch to 2.7 ahead of the next LTS?
<wgrant> jml: Yes.
<jml> that'd be nice
<wgrant> We have a Lucid backport already, I believe.
<wgrant> OK, down to gratuitous difflib formatting changes plus XML-RPC rewrite.
<wgrant> Yay
<wgrant> danilos: Does the zero-width space fix the slight underline you get on hover in WebKit?
<jml> lifeless: does fixtures have a way of combining two fixtures?
<jam> wgrant: to add love to this investigation, if I do 2 api requests back-to-back in the same process, the first takes 700ms, the second 200ms. something seems really wonky about ssl.sslwrap
<jam> maybe it caches host identities or something
<nigelb> oh yay, is loggerhead down?
 * nigelb moves to the correct chanel
<wgrant> jam: Hmm.
<wgrant> jam: It's possibly also doing OCSP or something slow like that.
<lifeless> jml: yes, useFixture, or the hypothetical graph stuff we've talked about vis-a-vis replacing testresources
<lifeless> jml: re emeritus - yes, its not setup yet; you didnt have to leave ~launchpad :P.... anyhow, getting it setup means moving stuff that references launchpad directly to reference role teams, and I don't know how long the piece of string is yet.
<bigjools> woo, timeouts creating an MP
<wgrant> bigjools: The branch scanner is slow, as it has to insert like 70000 branchrevisions.
<wgrant> That keeps things locked for a while.
<bigjools> what did someone just do?
<gary_poster> personally, I was typing.
<bigjools> badumtish!
<gary_poster> :-)
<bigjools> StevenK's humour has been rubbing off on gary_poster
<gary_poster> heh
<bigjools> I say humour in the loosest sense
<gary_poster> yeh yeah yeah
<bigjools> that reflects more on him than you though :)
<gary_poster> :-)
<wgrant> gary_poster: Hi.
<gary_poster> hey wgrant.
<wgrant> gary_poster: Our XMLRPCTestTransport is broken in Python 2.7. getresponse now takes a buffering kwarg, and it's not trivially obvious to me how to make our stuff work with 2.7. I noticed that there's zope.app.testing.xmlrpc.ZopeTestTransport which does a similar thing in a different, 2.7-compatible way. But compared to ours, it doesn't send a Host header or preserve the original interaction.
<wgrant> I presuuuume we probably want to rebase our stuff around ZopeTestTransport.
<wgrant> But was wondering if you had any ideas.
<wgrant> Given your Zopey foundationsness.
<gary_poster> heh
<gary_poster> wgrant, in the abstract, I no longer feel that there's a huge win to trying to coordinate with the Zope community, given its ever-vanishing existence/relevance.  Maybe that's unfair.
<gary_poster> In any case, I'm in favor of a relatively attractive path of least resistance, whatever that is.  If it would be relatively easy to make the Zope version do what we want, benji and I still have commit and release privs.
<gary_poster> That could potentially get us in one of the "update the Zope world" situations, but if there has been as little activity there as I believe, that might be a non-problem.
<gary_poster> I'm afraid I don't have anything to offer other than that ATM, though if you'd like me to dig in a bit to understand the problem you describe and have a more concrete opinion on a way forward, I'm happy to.
<wgrant> I'm happy to dig myself, was just wondering if you had any immediate insights.
<gary_poster> 'fraid not. :-/
<wgrant> Thanks anyway.
<gary_poster> np
<wgrant> Shouldn't be too hard to sort something out.
<gary_poster> agreed
<wgrant> bigjools: :( that needs to be advertised everywhere
<bigjools> wgrant: ?
<wgrant> bigjools: I'd not seen that matcher before, and it seems useful in an awful lot of tests.
<bigjools> yes, it freakin rocks
<bigjools> seriously, read the matcher docs
<danilos> wgrant, yes (regarding zero-width space)
<wgrant> danilos: Great!
<jkakar> bigjools: +1000. :)
<jkakar> testtools in general is super awesome.
<deryck> Morning, everyone.
<jtv> hi deryck
<bigjools> jkakar: yes!
<jtv> gmb: could I trouble you for review of an oversized refactoring?  https://code.launchpad.net/~jtv/launchpad/pre-788078/+merge/68374
<gary_poster> Hey deryck.  You know I'm excited, so please just forgive me :-) but where is the thunderdome JS test stuff?  Have you started actually writing tests with it, and are you reasonably pleased with it?  (I am reminded because I'm writing some JS in a module w/out tests)
<deryck> gary_poster, I have sadly not got very far with it.  I had one other remaining thunderdome bit which has taken all my attention....
<deryck> gary_poster, a branch for fixing bug heat.
<deryck> sorry :(
<gary_poster> deryck, oh :-( .  understood.  I am about to finish a bugfix branch myself, so maybe I'll race you for the last bit? :-)
<gary_poster> deryck, if you really want to do it, that's ok.  I got to play on it quite a bit
<deryck> gary_poster, ok :)  I hope to either resolve the heat work or turn to the js stuff while waiting on more info on the heat work today. But I'll see how it goes.
<gary_poster> :-) k cool
<deryck> gary_poster, but I'll keep you better informed on my progress, too. :)
<gary_poster> :-) thanks for humoring me
<deryck> np!
<henninge> How do I get the dev server to not use launchpad.js but the single js files?
<gary_poster> henninge, afaik there is no way.  when people talk about that stuff they are talking about running YUI tests.  If you discover differently, I would *love* to know too :-)
<wgrant> That ability was removed once we stopped using the bundled up YUI thing.
<wgrant> Because it ended up making 400 requests to the local appserver on each page load, which took a couple of minutes.
<gary_poster> oof
<henninge> Ah, but my memory was not wrong.
<wgrant> henninge: I know 'make jsbuild' isn't perfect, but it should be pretty fast.
<wgrant> No need for appserver restarts or anything.
<wgrant> Plus you should be doing TDD ;)
<henninge> wgrant: no, it's about debugging the js file in the browser
<henninge> lots of scrolling and searching in launchpad.js
<wgrant> henninge: The Launchpad bits should be unminified.
<wgrant> But yeah.
<gary_poster> my concern with it is that neither chrome debug tools nor firebug handles the large file well
<wgrant> henninge: You can't debug in the testrunner?
<gary_poster> trying to set breakpoints sucks in that case
<henninge> hm, could try that
<gary_poster> wgrant we have lots of legacy JS code without tests
<wgrant> gary_poster: Indeed :(
<gary_poster> henninge both firebug and chrome debug let you search for a string
<gary_poster> that's what I use
<gary_poster> I find navigation not too bad
<henninge> me, too
<gary_poster> the debugger bugs are the killer
<gary_poster> for me
<henninge> would be cool if the file had comments with the originial file name whereever that file starts
<henninge> wgrant: where is the code that creates launchpad.js?
<wgrant> lp.scripts.utilities.js.jsmin
<henninge> thanks
<wgrant> Well.
<wgrant> That minifies it.
<wgrant> But the input to that is just cat run over all the files.
<henninge> and the combining?
<wgrant> cat :)
<henninge> ah
<henninge> ;)
<wgrant> That's all in Makefile.
<wgrant> rm lib/canonical/launchpad/icing/build/launchpad.js; make jsbuild
<wgrant> Watch your screen and sob.
<gmb> jtv: Sure, I'll take a look at that branch for you.
<jtv> great, thanks
 * henninge sobs
 * StevenK sees that henninge is following wgrant's instructions
<henninge> yup, always
<cjwatson> wgrant: whee, thanks
<cjwatson> (there's still multiarch-translations proper, but need to wait for a lucid-updates landing and sort out PPA bits first)
<gmb> jtv-afk: r=me on the kilobranch. Nice work.
<nigelb> Hi, how do I update my branch? My devel branch is getting a rocketfuel-get.
<nigelb> *my feature branch
<jml> nigelb: you merge devel (or stable) into it
<nigelb> ok, doing that now.
<jml> nigelb: you need to commit after merging
<jml> nigelb: pretty important that you do that.
<nigelb> jml: oh, ok!
<nigelb> did windmill get nuked?
<jml> nigelb: yes.
<jml> nigelb: at the rally
<nigelb> that explains the conflict.
<nigelb> where is the ideal place to put a utility function (sorting)
<mhall119> quick question, can I query the launchpad api using a user SSO identity URL yet, or is it still just username?
<jml> nigelb: lp.services.utils
<jml> nigelb: tests in lp.services.tests.test_utils
<nigelb> jml: thanks again. Where is everyone today? :)
<jml> nigelb: no idea.
<bigjools> jml: "byEquality" - it just gets betterer and betterer :)
<jml> bigjools: heh
<bigjools> nigelb: I guess we must be busy hacking or something
<bigjools> jml: have you got anything to do with Murdoch taking a pie to the face?
<jml> bigjools: sadly no.
<nigelb> lol
<nigelb> I wonder if I should agree to take a pie to the face.
<nigelb> For fixing all critical bugs before uds-p
 * jml has a side bet for UDS-P. It involves drinking.
<nigelb> jml: In that case I'll hear that one out :D
<nigelb> UDS-P should of course be written as UDS :-P
<jml> :)
<nigelb> er, I'd like some help.
<nigelb> How would I generalize the sort in this http://paste.ubuntu.com/647431/
<nigelb> I want to avoid the bit where I'm doing the lower() and move it to a common function.
<nigelb> But I'm not sure how to make it generic enough to be used everywhere
<jml> nigelb: I don't think it's worth generalizing, tbh.
<jml> nigelb: hmm.
<jml> If you had to...
<nigelb> jml: That's what I've been told to do. https://code.launchpad.net/~nigelbabu/launchpad/spec-sub-sort/+merge/63315
<jml> def case_insensitive_sort(xs):
<jml>     return sorted(xs, key=lambda x: x.lower())
<nigelb> So, the only bit I'd avoid would be the lower() thingy.
<jml> well, yeah
<nigelb> I guess jtv could help throw some insight on whether that's enough. He helped with the initial review.
<nigelb> jtv: ping
<jml> nigelb: I don't have the willpower to read through that whole discussion...
<jml> nigelb: you could also try this:
<jml> def sort_for_display(things_to_display):
<nigelb> jml: Now you know why it took me this long to do this :)
<jml>   return sorted(things_to_display, key=lambda x: x.displayname.lower())
<jml> or even
<jml> def sort_for_display(things_to_display, display_attribute='displayname'):
<jml>   return sorted(things_to_display, key=lambda x: getattr(x, display_attribute).lower())
<nigelb> ok, ^ sounds like what's probably generic enough.
<jml> nigelb: yeah, you've got some fairly blocky reviews
<nigelb> It was hard to get out of that pit of demotivation I fell into after this.
<jml> nigelb: well, basically, the thing is that as long as there's a *name* for what you are trying to do, the exact level of abstraction doesn't matter.
<jml> because it'll probably change as we get it better anyway
<jml> (and the name in this case is the function)
<jml> nigelb: *nod* I know what you mean
<nigelb> jml: I was trying to use this function in different use cases and I couldn't abstract it will.
<jml> nigelb: ahh, ok.
<nigelb> *well
<nigelb> jml: I had to take some time off Lp dev before I could get myself back :)
 * nigelb is now back :D
<nigelb> jml: in that above function, should x also be a parameter to the function?
<nigelb> *shouldn't
<jml> nigelb: no. it's the argument to the key function.
<jml> nigelb: please bear in mind that I typed it out into my IRC client, and thus am missing my usual crutches and so I've probably made a NameError or SyntaxError
<nigelb> I'm testing it on python shell first
<jml> good idea
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256
<nigelb> gmb: are you EOD-ing or was the topic stale?
<nigelb> Ouch, my launchpad.dev isn't working. Is anyone around to help me figure out how to get it working?
<nigelb> I guess I'll be back tomorrow.
<deryck> sinzui, ping
<sinzui> deryck, otp
<deryck> sinzui, ok, just a question about a bug when you have a moment.
<sinzui> hi deryck
<deryck> hi sinzui.  Was only wanting to ask you about bug 812790 since you guys are working on stuff related to people, pickers, etc.
<_mup_> Bug #812790: strange "no such person" popup on user-relative branch url <Launchpad itself:Triaged> < https://launchpad.net/bugs/812790 >
<deryck> sinzui, not sure if that bug is at all related, but it seems fixed on qastaging.  So wondering if the bug should be closed?
<deryck> or I can't repro on qastaging anyway.
<sinzui> We have landed a few fixes in the last 24 hours
<sinzui> I do not see anything that specifically fixes it though
<deryck> sinzui, ok, thanks.  I'll leave as is then until we have more info.
<lifeless> morning
<jelmer> hi lifeless
<lifeless> hi jelmer
<lifeless> gary_poster: hi
<gary_poster> hey lifeless
<lifeless> gary_poster: that UNION -> OR may be performance sensitive; am just reviewing.
<gary_poster> (on call(
<lifeless> yah, no worries, just giving you a headsup
<gary_poster> lifeless, wondered about that bu seemed unlikely.  I can just revert that part.  k
<gary_poster> lifeless, that user_ids_affected_with_dupes code (UNION -> OR) has never been changed since it was originally written.  Tempted to revert anyway.  Thoughts?
<lifeless> gary_poster: reverting would be conservative and make qa easier
<lifeless> gary_poster: thats about it :)
<gary_poster> lifeless, k, agree, will do, thx.
<gary_poster> lifeless, "in the client side js, the limit of checking whether the task was
<gary_poster> new seems unnecessary - isn't it reasonable to just cross check
<gary_poster> regardless?" :
<gary_poster> Since we don't do this generically (yet) I thought it would be surprising to update other tasks in response to saying "me too" because people might confuse coincidence with consequence.  I'd be fine if we always kept statuses up-to-date with long-poll later, for instance; then we would not need this "something happened" code path at all.
<gary_poster> For now, though, I stand by my call there.
<lifeless> its up to you
<lifeless> there are a couple of bugs I can point you at :)
<lifeless> where we suffer inflight collisions [particularly on the non-ajax forms] due to stale statuses
<gary_poster> sure
<gary_poster> but that's not what this is about
<lifeless> I do see your point
<lifeless> uhm
<lifeless> perhaps huw can suggest some way [in the future] for us to show 'updated by you' and 'updated by $other'
<lifeless> gary_poster: e.g. standing by your call is fine.
<gary_poster> cool lifeless, thanks
<lifeless> gary_poster: it was only a thought in the first place; perhaps I wasn't clear enough about that.
<gary_poster> lifeless, sure.  if you didn't want us to respond to them, though, why express them? ;-)
<lifeless> well, I do like responses :)
<lifeless> I suspect I should eat before I fail-at-communication-more
<gary_poster> well, and they are good thoughts that merit a response.  so...
<gary_poster> lol
<lifeless> I don't know entirely what I meant from 'it was only' .. on.
<lifeless> -> food
<gary_poster> :-)
<lifeless> wgrant: lxc in oneiric may be better, but check out bug 802985
<_mup_> Bug #802985: [lucid] /var/lib/dpkg/tmp.ci/preinst: 399: arithmetic expression: expecting EOF: "3.0-0-generic" <lucid> <patch> <debootstrap (Ubuntu):Triaged> <eglibc (Ubuntu):Invalid> <debootstrap (Ubuntu Lucid):Invalid> <eglibc (Ubuntu Lucid):Triaged> <debootstrap (Ubuntu Maverick):Invalid> <eglibc (Ubuntu Maverick):Triaged> <debootstrap (Ubuntu Natty):Invalid> <eglibc (Ubuntu Natty):Triaged> <debootstrap (Ubuntu Hardy):Invalid> <eglibc (Ubunt
<gary_poster> lifeless, do you recommend bothering with lxc for LP right now?  I've been considering it, but the page you and William have maintained makes it look pretty...fragile.
<lifeless> gary_poster: I have a few thoughts
<lifeless> I don't think that right now the incremental learning for the team is all that great from someone just running a single test suite under it
<gary_poster> (just a vm is my other option, and what I have do noe)
<gary_poster> now
<lifeless> I think its good to be familiar with lxc though, because I think its got legs
<gary_poster> sure
<gary_poster> I hope so
<lifeless> and we're likely to want to implement parallel testing on top of it
<gary_poster> makes sense
<lifeless> the page does cover a bunch of detail today, but automation is improving, especially if using oneiric or the lxc ppa
<gary_poster> was going to try oneiric, but I've spent my OS configuration budget for the month.  August, maybe. :-)
<lifeless> heh, i hear you!
<lifeless> I've just made a KVM oneiric instance
<lifeless> to kick the tires on lxc and ensemble a bit
<gary_poster> cool, yeah, I made a vm too
<gary_poster> but lxc on a vm seemed a curiosity and a good way to explore lxc, more than a practical approach. :-)
<gary_poster> although it would still give some benefits, I suppose
<lifeless> so lxc is a moving target
<lifeless> the container rules and capabilities are changing fast enough that 6 months can easily get bug fixes relevant to us.
<gary_poster> yeah, makes sense
<lifeless> anyhow, if you want to do any of the following, lxc +1:
<lifeless>  - parallel test
<lifeless>  - test with different / new dependency stacks
<lifeless>  - run more than one lp up at a time [e.g. to play with federation]
<lifeless> or if you have any memory pressure issues on your machine (e.g. during bin/test)
<lifeless> it shouldn't be too bumpy a ride - it will either be easy, or totally impossible :>
<gary_poster> lol
<gary_poster> ok cool
<gary_poster> might give it a try soon then
<gary_poster> thanks lifeless :-)
<lifeless> np :)
<lifeless> wallyworld: wow, early up?
<lifeless> jelmer: hi
<lifeless> jelmer: why do you think bug 628323 is a subunit package bug ?
<jelmer> lifeless: see max's comment
<lifeless> I don't have that
<lifeless> what did he sya ?
<maxb> someone talking about me? :-)
<jelmer> lifeless: "This FTBFS is the result of the python-subunit package not being
<jelmer> properly built for all current Python versions"
<lifeless> I don't follow.
<lifeless> does it need a no change rebuild because the set of supported python packages has finally changed in debian ?
<lifeless> s/packages/versions/
<maxb> Is that bug number wrong, or private?
<lifeless> its in Debian
<jelmer> lifeless: it has already had the no-change rebuild, I'vconfirmed 0.0.6-2 was only built for 2.6, 0.0.6-3 was built for 2.6 and 2.7
<lifeless> I'm not sure what to conclude from that :)
<lifeless> I have to pop out to see the midwife, be back in <= 90m
<jelmer> according to the logs the bug was filed with 0.0.6-2 installed
<jelmer> so I'm going to indicate that in the bts
<jelmer> lifeless: ok. I hope all is well
<lifeless> just routine meeting
<wgrant> lifeless: Hah, nice.
<wgrant> lifeless: Oddly, lxc debootstrapped it fine for me.
<wgrant> wallyworld: Hi
<wallyworld> hi
<wallyworld> lifeless: sorry, didn't see your ping. irc notifications in unity aren't as good as i'd like so i often miss them
<wallyworld> wgrant: what can i do you for?
<wgrant> wallyworld: Was just going to tell you that the "true"/"false" thing was a real bug, but I see you've fixed it already.
<wgrant> (not sure why it can't be in the main JSON dict, though)
<wgrant> Nice simple test, too.
<wallyworld> wgrant: yes. i reverted that tales snippet. it's all a mess. today i plan to do some more yak shaving. there's 2 ways of getting a picker (via create or addpickerpatcher) and you get different results depending on which one is used :-(
<wgrant> wallyworld: Yeah.
<wgrant> Still, we are getting better.
<wgrant> Slowly.
<wallyworld> plus the config vs json_config issue
<wallyworld> at first i didn't see why there was the need to hack in those booleans in the tales
<wallyworld> all pretty ugly
<wgrant> But fixable :)
<wallyworld> i added 2 tests to the yui test case - one for the textfield plugin and one for the extra message. the textfield plugin refactoring from the tales was good because it enabled the yui test to check the proper code path
<wallyworld> whereas before the test wired it up manually
<wgrant> i only see one new test, but I guess you adapted an existing one to test the whole thing?
<wallyworld> wgrant: yes - i modified an existing test and added some extra setup guff to support it
<wgrant> The best kind of new test.
<wallyworld> the modifed test is the one i was referring to above when i said it's now a better test
<wgrant> Yup
<LPCIBot> Project devel build #900: FAILURE in 5 hr 33 min: https://lpci.wedontsleep.org/job/devel/900/
<jelmer> I'm getting exceptions starting RabbitMQ during the testsuite; is that a common issue?
<wgrant> jelmer: Shouldn't be any more. Is your hostname configured correctly?
<jelmer> 'hostname -f' returns 'localhost6', so I guess not..
<wgrant> That could be a problem.
<wgrant> It should be mostly immune to that now, but maybe IPv6 combined with dodgy hostname is doing bad things.
<jelmer> hmm, that doesn't seem to work either
<jelmer> I'll have a look tomorrow
<wgrant> sinzui: We can hear you and each other.
<wgrant> Can you hear us?
<sinzui> no.
<wgrant> That is suboptimal.
<poolie> it would be cool if launchpad could show a per-pillar view like this for the hottest bugs: http://mail.google.com/support/bin/static.py?page=suggestions.cs
<wgrant> lifeless: Hi
<lifeless> hi
<wgrant> lifeless: The malone.advanced-subscriptions.enabled, malone.advanced-structural-subscriptions.enabled, code.recipes_enabled flags are on by default and have been removed from the code. Can we remove the rules?
<lifeless> of course
<wgrant> Thanks.
<poolie> hi wgrant, lifeless
#launchpad-dev 2011-07-20
<wgrant> Huh.
<wgrant> test_initZopelessTwice fails with a recursion limit error.
<wgrant> Because it adds a warning hook.
<wgrant> Which uses failUnlessEqual.
<wgrant> Which is deprecated.
<wgrant> Which raises a deprecation warning...
<mwhudson> whee
<lifeless> win
<lifeless> wgrant: so init zope less twice sends a warning ?
<lifeless> wgrant: if so, as we don't init zopeless twice in the real world, perhaps make it raise an exception instead?
<wgrant> Are you sure we don't?
<wgrant> Trivial fix will do for now.
<wgrant> Or not.
<wgrant> 2.6 testcase doesn't have assertIs.
<wgrant> testtools time, I guess.
<lifeless> wgrant: I sure hope we don't.
<wgrant> Why?
<lifeless> because I want to nuke it entirely
<wgrant> Sure.
<lifeless> as unhygenic
<spiv> After all, if it's been raising a warning when called twice then hopefully anyone doing that would have noticed the warning.
<wgrant> spiv: We mostly just suppress warnings :)
<StevenK> Then throw an exception
<wgrant> dict ordering issue in launchpadlib, XMLRPCTestTransport borked, timeout.txt hanging in really bad ways. Apart from that all fixed.
<StevenK> wgrant: It continues to pass with 2.6, so the fixes are landable?
<wgrant> Yes.
<wgrant> They're all small test issues, apart from one real bug.
<wgrant> timeout.txt might be real too, I guess.
<wgrant> But need to get pygdb onto it at some point.
<stub> I'm getting sucky packetloss entering Los Angeles. This should affect most of Asia :-(
<stub> level3 LA
<wgrant> StevenK: Working fine for me.
<wgrant> Er, stub.
<wgrant> Maybe they've fixed it.
<wgrant> Optus' international pipe has been fairly broken for the last 24 hours, though.
<stub> Seems better now. Can't be sure it isn't my ISP messing up though - I think it is the first server on the US site, so might be entering the US/Asia cable that is screwed.
<stub> nope... still messed. just not as bad.
<poolie> lifeless: i'm seeing some udd failures due to bug 602647
<poolie> hm
<poolie> actually, no i'm not
<wgrant> It's private?
<poolie> is there another bug for internal xmlrpc flaking out and returning a 503?
<wgrant> That sounds like a timeout.
<poolie> https://bugs.launchpad.net/launchpad/+bug/294726 is pretty close
<_mup_> Bug #294726: "ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/294726 >
<poolie> probably
<wgrant> poolie: Which bug number did you mean?
<poolie> bug 602467
<_mup_> Bug #602467: retry 503 errors on the LP directory service. <launchpad> <Bazaar:Confirmed> < https://launchpad.net/bugs/602467 >
<wgrant> Ah
<nigelb> Hi, when I hit launchpad.dev, its not hitting my local instance.
<nigelb> Suggestions on where to look?
<poolie> nigelb: i guess you need to start apache
<poolie> or, connect direct to zope on port 8086
<poolie> apache acts as a front end proxy
<nigelb> poolie: There's a make run running
<poolie> you need to run 'sudo service apache start' separately
<poolie> assuming it was not started at boot time
<poolie> (in your chroot, or vm or whatever, if you're using one)
<nigelb> I just use my regular apache
<nigelb> Nope
<nigelb> no go
<poolie> what happens?
<nigelb> apache was already started
<nigelb> but I still can't hit launchpad.dev from firefox
<mwhudson> what happens when you try?
<mwhudson> you get the apache 'it works' page?
<nigelb> I get "Firefox can't establish a connection to the server at launchpad.dev."
<mwhudson> launchpad.dev is in /etc/hosts?
<poolie> are you using a chroot or something?
<nigelb> The /etc/hosts is what I checked first, its there.
<nigelb> poolie: Nope, just in my regular install
<poolie> try telnet launchpad.dev 443
<nigelb> oh grrar
<nigelb> https://launchpad.dev works
<poolie> hooray
<poolie> congrats
<nigelb> that further proves that its been a while since I've done this.
 * nigelb should contribute more.
<spiv> poolie: https://code.launchpad.net/~spiv/launchpad/bmp-inline-diffs/+merge/66634 is now ready for landing I believe
<spiv> poolie: do you still want to send it in for me? :)
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #901: FIXED in 5 hr 33 min: https://lpci.wedontsleep.org/job/devel/901/
<wgrant> Oh yay, our JS is broken in IE again.
<StevenK> Why are you using IE?
<wgrant> Was testing the expanders.
<StevenK> Ah.
<huwshimi> wgrant: Is it lint issues again?
<wgrant> It's not the usual comma at the end of a list.
<wgrant> It may be a missing closing brace, I think.
<StevenK> Just one?
<wgrant> It occurs in the middle of json-parse.js, it seems.
<wgrant> But that's YUI...
<lifeless> and bug searches are back into timeout territory
<lifeless> cute - http://elaforge.blogspot.com/2011/07/what-haskell-doesnt-have.html
<lifeless> I really must do something nontrivial in haskell one of these days
<wgrant> Indeed.
<wgrant> It's a really nice language.
<stub> I need to learn a new language one day. I think I've forgotten everything except Python and PL/SQL
<spm> perl
<lifeless> 'new language'
<nigelb> I still think spm wins ;)
<spm> hahahha
<nigelb> lifeless: what bit about the function in https://code.launchpad.net/~nigelbabu/launchpad/spec-sub-sort/+merge/63315 do you want me to abstract?
<nigelb> I got confused yesterday when I was trying.
<lifeless> nigelb: I think the point was that the sort with a sort key function is something we could spell once abstractly
<nigelb> I tried abstracting the sorted bit in there
<nigelb> but then I realized that I also do sort() there.
<nigelb> which is part of another function.
<nigelb> Does that make sense?
<lifeless> nope
<lifeless> pastebin ?
<nigelb> This is something jml wrote for me yesterday. http://paste.ubuntu.com/648007/
<nigelb> Is this the level of abstraction you were looking for?
<lifeless> nigelb: I think so, yes.
<nigelb> oh, I can't link you to lines in a merge proposal's diff :/
<nigelb> Well, line 19 of the diff in my MP is what's confusing me.
<nigelb> I can't use that function there.
<poolie> spiv, hi, i can send that for you
<spiv> poolie: thanks!
<poolie> one idea occurred to me, reading your latest update
<poolie> (thanks for completing it)
<poolie> and that was that rather than href="#" perhaps it should send people to the loggerhead view of the diff
<poolie> in case they ctrl-click it or have a non-js browser
<poolie> (i don't know what the policy is for supporting that)
<poolie> i don't want to stall this though
<spiv> If they have a non-js browser, the link won't be visible currently.
<spiv> Well, if they have a non-js but with-css browser.
<poolie> oh ok
<poolie> anyhow, it was just an idea
<lifeless> nigelb: property_cache.subscriptions = function_call()
<poolie> spiv, so, i'll just send it?
<spiv> It's a good idea, but it's a bit fiddly to implement AIUI.  At the least you'd need to use a less convenient API than expander.createByCSS I think.  Perhaps you'd enjoy making that patch? ;)
<spiv> poolie: please do!
<poolie> :P
<poolie> huwshimi: is there actually a policy on non-js support?
<nigelb> lifeless: ah, thanks. Brain seems to be turned off today :)
<huwshimi> poolie: This is the only thing I can think of off the top of my head: https://dev.launchpad.net/GradedBrowserSupport
<spiv> I'm a bit sick of staring at that code today, I spent longer than would be ideal discovering that one of the expander APIs grew a new positional parameter since I'd last merged, which of course resulted in a mysterious ânothing happens when I clickâ bug :/
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugs: 238 - 0:[######=_]:256
<spiv> poolie: oh hmm, I guess I misunderstood, and making that link go straight to loggerhead might not be so hard
<spiv> poolie: but I think it's weird UI for the browser to indicate that a link will take me somewhere and then for clicking it to do something else.
<poolie> i loved named/optional parameters when i first used python but now the bugs they lead to are one of my least favourite bits
<poolie> in the sense that the status bar will show the lh url
<spiv> named-only params would be nicer.
<poolie> for nerds who look at the status bar
<spiv> Like me :)
<poolie> it's been a long branch, i don't want to make you do any more on it
<poolie> it was just a thought when i saw your update
<spiv> I've definitely been repeated annoyed by that sort of behaviour in the past, so I'd be pretty hesistant to implement it myself :)
<poolie> in general links that point to # are i think a chance to consider whether there is somewhere better
<spiv> I agree that's the experience I'd want non-JS browsers to get in the ideal world.
<spiv> I'm not sure how best to arrange that without misleading the JS browsers.
<spiv> (It does lead to the intruguing idea that the target of the href could be used as the input to the 'generate the link for raw diff to fetch' function.
<spiv> )
<poolie> i don't think browsers really handle the case of indicating what a js link will do
<poolie> you typically either get '#' or some unreadable function lambda
<poolie> spiv, ec2 wants to know if there's a bug for this branch
<spiv> Yeah.  I might be wrong about how bad it would be, but I *have* encountered irritating behaviour with expander-like things and misleading ctrl-click behavour in LP before.
<spiv> Not that I know of, probably a good time for me to take a quick look...
<spiv> poolie: a few very vaguely related ones maybe, but no appropriate one
<spiv> poolie: but it did lead me to notice a low-importance bug of yours that's a dupe of an older critical one!
<poolie> heh
<poolie> which?
<spiv> bug 762355
<_mup_> Bug #762355: would prefer to see mp with no diff when librarian fails, rather than no mp at all <code-review> <Launchpad itself:Triaged> < https://launchpad.net/bugs/762355 >
<poolie> ok bug 813349 for the sake of qa'ing it
<_mup_> Bug #813349: hard to get from mp to per-revision diffs <Launchpad itself:In Progress by spiv> < https://launchpad.net/bugs/813349 >
<spiv> poolie: thanks!
<poolie> ok, instance is booting, we should know tomorrow
<adeuring> good morning
<mrevell> Good morning
<rvba> jtv: About your suggestion to clean my branch (removing the use of filtered_expanded): the code you propose was exactly what I wrote first but if candidates is [] then Or(*candidates) breaks the generated SQL.
<jtv> rvba: then return if candidates is empty.  :)
<jtv> Doesn't cost much to go through one more empty list comprehension.
<rvba> jtv: but make_package_condition crashes if it's called with das=None.
<jtv> But you won't be calling it.
<rvba> ah ... right.
<jtv> The extra "if" in the list comprehension prevents it from happening.
<rvba> True.
<jtv> cjwatson: you still have an approved but unlanded merge proposal for Launchpad open from last year.  What should happen with it?  https://code.launchpad.net/~cjwatson/launchpad/dpkg-xz-support-619152/+merge/32868
<jtv> and jelmer, what's supposed to happen to https://code.launchpad.net/~jelmer/launchpad/publisher-use-debian-2/+merge/45011 ?
<jtv> adeuring: you too have an ancient approved MP on the backlog: https://code.launchpad.net/~adeuring/launchpad/js-translation/+merge/55309
<jtv> lifeless: approved-but-unlanded-MP reminder: https://code.launchpad.net/~lifeless/launchpad/bug-759467/+merge/58262
<jelmer> jtv: thanks for the ping
<jelmer> wgrant: hi
<jelmer> jtv: IIRC cjwatson's branch was waiting on the installation of a newer dpkg package, IMBW
<jtv> jelmer: thanks â it's been a while so probably a good time to check up on that
<cjwatson> yes, it's due to land in lucid-updates in a couple of days
<cjwatson> when it does, I'll propose meta-lp-deps changes, and then we can try landing dpkg-xz-support again
<cjwatson> don't worry, I'm totally keeping track of this
<cjwatson> (though I wasn't for a long time, I know)
<cjwatson> it's actually newer apt and python-apt - didn't I comment on the MP or the bug about it?
<cjwatson> yes, I did comment on the MP a week ago.
<cjwatson> http://people.canonical.com/~ubuntu-archive/pending-sru.html - waiting for lucid/apt and lucid/python-apt there to hit 7 days before the next move in the game
<rvba> jtv: landing the branch now. Thanks for the review!
<lifeless> jtv: it bounced off of ec2; I haven't dug into ityet
<jtv> rvba: np
<cjwatson> thanks to whoever landed https://code.launchpad.net/~cjwatson/launchpad/germinate-ubuntustudio for me, without me having even got round to asking :)
<lifeless> <-
<poolie> cjwatson: hooray for patch piloting spirit
<poolie> (presumably rum)
<lifeless> wgrant: re namespace patches
<lifeless> [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
<lifeless> *packages*
<wgrant> lifeless: That almost looks like not a complete mess.
<henninge> wgrant: bzr log -r12957.1.6
<henninge> wgrant: wrong ;-P
<jelmer> is rabbitmq used for any core stuff yet?
<jelmer> it's being set up for the codeimports, which (AFAIK) shouldn't need it
<wgrant> jelmer: It's part of LaunchpadLayer, but only a couple of tests use it so far, and they're only to check that the layer setup is working.
<wgrant> jelmer: If you still can't get it running, I guess you could hack LaunchpadLayer to not depend on RabbitMQLayer.
<jelmer> wgrant: that's what I've just done :)
<jelmer> I'll have a closer look at it later, but for the moment I just want to get some stuff done.
<henninge> jtv1: Hi! I have a short and easy branch for you. ;-)
<henninge> https://code.launchpad.net/~henninge/launchpad/bug-792092-build-now/+merge/68521
<jtv1> henninge-lunch: reviewing.
<wgrant> danilos: Hi.
<wgrant> adeuring: Also hi.
<jtv> henninge-lunch: done
<wgrant> jtv: Are you going to be around to chase up the deployment this evening?
<jtv> No.  It's planned, it's approved, it should be announced.
<jtv> mrevell: did that work out by the way?
<wgrant> It is announced.
<jtv> I think he said so yesterday, but not sure.
<wgrant> But it's still useful to have someone around who knows what's going on.
<wgrant> You may wish to recruit a conveniently located maintenance person for this purpose.
<mrevell> jtv, yeah, it's announce
<mrevell> d
<jtv> thanks
<danilos> wgrant, hi
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv, danilo | Critical bugs: 238 - 0:[######=_]:256
<wgrant> danilos: We have another expander regression :(
<wgrant> On prod this time.
<danilos> wgrant, which one is it?
<wgrant> (see #launchpad a few minutes back)
<wgrant> When there is a conjoined bugtask, the expander hook for the conjoined slave crashes.
<wgrant> Because there is no expander.
<wgrant> This seems to break all remaining inline JS on the page.
<danilos> wgrant, indeed, I'm looking into it
<wgrant> danilos: Thanks.
<wgrant> danilos: Hopefully the fix is smallish and can skip ec2 and we can deploy it in not too many hours.'
<danilos> wgrant, yeah
<danilos> wgrant, considering jtv is not around and I am the other OCR reviewer, can you perhaps approve https://code.launchpad.net/~danilo/launchpad/fix-bugtask-expander/+merge/68536?
<danilos> wgrant, (or disprove, if you are so inclined :)
<wgrant> danilos: Approved.
<wgrant> Thanks!
<danilos> wgrant, thank you!
<danilos> wgrant, I'd say "bzr lp-land" should be fairly safe with this, what do you think?
<wgrant> Yes, definitely.
<wgrant> cjwatson: Your ubuntustudio germinate change should be deployed to cocoplum in the 1600UTC downtime window today.
<danilos> wgrant, ok, I'll try to arrange a deployment towards the end of my day (when lpbuildbot run completes), if not, I'll ask some of my fellow yellow Americans
<wgrant> danilos: I was about to ask if you could do that. Thanks!
<wgrant> danilos: We are QAed up to current stable tip, so it's just your rev left.
<danilos> wgrant, cool, thanks
<wgrant> So it will be a nice easy thing to deploy in ~5 hours.
<wgrant> Thanks for the quick fix.
 * wgrant sleeps.
<cjwatson> wgrant: cool, that's quicker than I expected, thanks
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilo | Critical bugs: 238 - 0:[######=_]:256
<gary_poster> jml, I had a pre-imp for  (private) https://bugs.launchpad.net/launchpad/+bug/812335 with flacoste yesterday and was hoping I would not need to bother you, but I just ran into some concerns.  Would you have tie for a pre-imp?
<gary_poster> or time
<bac> hello mrevell
<mrevell> hey bac, otp
<bac> mrevell: np, ping when you're free
<jml> gary_poster: sure
<jml> gary_poster: I need to eat first though.
 * jml has been putting that off on account of unusually high productivity
<gary_poster> thanks jml.  ok, I have a dentist appt I need to leave for in 15 minutes, so maybe a couple of hours later then
<jml> gary_poster: perfect.
<gary_poster> thanks again
<danilos> jml, if you put it off for long enough, productivity will go to zero but so will your (non?) living costs
<deryck> Morning, all.
<gary_poster> hiya
<henninge> Anybody noticed this missalignment before? Is there a bug for it? I don't know what to search for ...
<henninge> http://people.canonical.com/~henninge/screenshots/miss-aligned-bugs.png
<danilos> henninge, looks like my expander replacements might be the culprit
<danilos> henninge, what browser is that?
<henninge> danilos: chromium
<danilos> henninge, fwiw, the bug priority icons were not even shown in webkit-based browsers before, so maybe the issue was there but spans were zero height
<henninge> they weren't? oh
<danilos> henninge, yeah, <li>s now probably need more indentation for two sprites (expander icon and the bug icon), at least I think that's the problem; please file a "regression ui" bug and I'll look into it tomorrowish
<danilos> henninge, or, if you feel like it, fix it :)
<henninge> danilos: I have another one I want to do, I was gonna talk to you about that, too ... ;)
<danilos> henninge, shoot before I head out to lunch :)
<henninge> danilos: I want to remove TranslationBranchApprover and replace it with TranslationBuildApprover.
<danilos> henninge, hum, what's a TranslationBuildApprover? :) and why is it different at all?
<henninge> danilos: ;-)
<henninge> danilos: TranslationBuildApprover is run after templates were build from a branch.
<henninge> danilos: it is much simpler and only determines the template name from the file name.
<henninge> danilos: it does not care to check which templates are already in rosetta and which are in the branch.
<danilos> henninge, makes sense
<henninge> danilos: TBranchA stops if a template is missing in the branch.
<henninge> we need to do a lot of manual approval because of that.
<henninge> danilos: That "comparing" functionality doesn't really help.
<danilos> henninge, right, but what happens when someone renames a template?
<danilos> henninge, how do they fix the mess?
<henninge> danilos: it *could* be helpful if it could explain that to the uploader and the uploader had a chance to do something about it.
<danilos> henninge, well, I've got a window of two weeks to finish the switch of approval to maintainers
<danilos> henninge, so, they'll be able to control their templates much more
<henninge> danilos: the TBranchA gets stuck and does not approve anything, the TBuildA just creates a new template
<danilos> henninge, right, but how would one ever "join" two templates together when one renames the template in the branch then?
<henninge> good luck gary-dentist
<danilos> henninge, imagine there was a template with lots of translations (history and stuff), and maintainer renames it in a branch: it automatically gets imported as the new template with no history preserved
<henninge> danilos: not possible
<danilos> henninge, that would suck very much, wouldn't it?
<henninge> danilos: correct, we don't have a good renaming story
<danilos> henninge, I'd rather just let people approve stuff on their own than mess it up completely (I've seen people rename templates all the time, at least until they get a good setup)
<danilos> henninge, fwiw, I think your approver makes more sense for ubuntu uploads
<danilos> henninge, "your" == TBuildA
<henninge> danilos: Letting maintainers approve themselves is much better of course.
<henninge> danilos: I wrote both ...
<henninge> ;-)
<danilos> henninge, heh, right :)
<henninge> danilos: why for Ubuntu uploads?
<henninge> danilos: so I guess we need to fix bug 728876 instead
<_mup_> Bug #728876: Know when translations branch/build auto-approvers give up <Launchpad itself:Triaged> < https://launchpad.net/bugs/728876 >
<danilos> henninge, because they are for trusted packages and are basically the same thing as what we build for TBuildA to approve (i.e. they are result of intltool-update -p for GNOME packages, and whatever else others use)
<henninge> right
<danilos> henninge, perhaps, but let's first let maintainers manage the import queue themselves
<henninge> danilos: that would add valuable information to enable them to manage them correctly.
<danilos> henninge, indeed, I agree
<danilos> henninge, so, I will be very happy if it gets fixed :)
<henninge> danilos: It would be best if it could give a specific reason ...
<danilos> henninge, indeed, you can reuse error_output for that
<deryck> henninge, ping for standup
<henninge> deryck: oh, sorry
<jtv> Has anyone ever seen this failure I'm getting in EC2?  ScriptActivitySet.recordSuccess violates scriptactivity_pkey (even though that's just an id that the database assigns normally).  Sign that the database needs to be marked dirty?
<jcsackett> sinzui: available to mumble this morning?
<sinzui> yes
<stub> jtv: not sure how that would happen even if there are test isolation issues
<jtv> I can't even reproduce it locally.
<stub> jtv: You might need to dump the contents of the table before that point, which is sucky if it is only happening on ec2
<jtv> But my branch triggers it consistently in EC2, and it's probably because of something I did.
<jtv> I turned a function that gets invoked by scripts into a LaunchpadCronScript itself.  So now we have LaunchpadCronScripts instantiating a LaunchpadCronScript and invoking it.  But I'd like to understand why the id clash.
<jtv> The test in question forks off a proper script instance, so maybe that ought to mark the db as dirty.
<jtv> It does, actually.
<jtv> Oh, but in what looks like the wrong place.
<bac> jtv: thanks for the review.  i've updated it showing i took your suggestion.
<jtv> It does that in the assertion after the script has completed...  so if one of the invocations failed for a different reason, that would sabotage the "dirty" bookkeeping.
<jtv> bac: thanks!  I didn't expect you to have much time for that kind of polish, so nice surprise.
<jtv> stub: is there any circumstance under which we'd reset sequences but not the db?
<stub> Hmm... I think there might be, and your initial idea could be correct.
<jtv> I'm seeing it on LFC as well.
<stub> If the db is not dirty, we assume all the data changes roll back. But the sequences don't reset, so we reset em
<stub> (although if our tests were good, we wouldn't have to reset the sequences because we would not be depending on tests getting consistent ids)
<jtv> Then I'll just start by marking the database when the script runs, instead of after it's completed.
<jtv> Yes, I wonder how much would break if we stopped resetting them.
<jtv> Oh blast, it's one of the fun ones.
<jtv> I can reproduce the problem by running a whole lot more tests at once.
<jtv> AAAAAAAAAAAAAhhhhh!  The preceding test invokes a script as part of its test data setup!
<jtv> And neglects to mark the database dirty, of course.
<jtv> At least I see a way to make that test a lot fasterâ¦
<bigjools> jtv: we should automate that dirtying somehow
<jtv> Yes, that'd be nice as well.  If it's not too expensive.
<jml> I wonder if there's a way to ask postgres if the db has had a write since a given time
<jml> or, when the last write was
<jtv> The problem is, I suspect, that you need to find out what was the last committed write.
<jtv> (Assuming we don't need to reset sequences, which we currently do)
<henninge> danilos: bug 813540
<_mup_> Bug #813540: Bug icons in dupefinder listing get pushed to next line <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/813540 >
<henninge> danilos: I'll probably take it, looks shallow.
<jtv> jml: technically there could even be a write in progress that hasn't committed yet, though.
<jml> jtv: by the time the script has ended?
<jtv> I wonder if we know for certain that all transactions have ended at the end of the test.
<jtv> Chances are pretty good though.
<jml> jtv: if the test is buggy and leaves processes or threads running, then I guess you won't know
<jml> jtv: but tests shouldn't be buggy, and we've already got reasonably good catches for those things
<jtv> Exactly, that's the hard case.
<jtv> Although with daemons this may all get harder.
<jtv> What if the Librarian is about to commit on the test's behalf, for instance?
<jml> yeah, that'd be a problem
<jml> because those could cross boundaries.
<jml> hmm
<jtv> Well, I don't know if the Librarian ever commits to the db of course.
<jml> but that's currently a problem anyway
<jml> it just depends which side you want to err on
<jml> anyway, it might be worth playing a bit with the idea, since it would ultimately be more reliable than remembering to mark as dirty
<jtv> Very true.
<jtv> And I'd like to have a ratchet on the number of test failures we get when we don't reset sequences.
<jtv> That's probably only worth it when we actually start working on it though: we could see the ratchet close.
<jtv> (It could be a driver for getting rid of some more old-style tests)
<jtv> Anyone know where this comes from?
<jtv> <class 'zope.tales.tales.RegistrationError'>: Multiple registrations for Expression type "cache".
<jtv> Multiple zcml-for-scripts initialization maybe?
<jtv> ahh yes, must be
<jtv> Did I seriously just knock 26 seconds off test time by fixing that?
<danilos> henninge, thanks for taking care of it (was lunching)
<henninge> danilos: https://code.launchpad.net/~henninge/launchpad/bug-813540-dupefinder/+merge/68557
<gary_poster> thanks for the good luck wishes henninge :-)
<danilos> henninge, looking
<gary_poster> jml, are you available for a call now?
<danilos> henninge, r=me, thanks
<henninge> danilos: thank *you* ;-)
<danilos> henninge, btw, I agree that layout should not be table-based, and would probably need to be reformatted completely to be sane
<henninge> yup
<jml> gary_poster: hi
<jml> gary_poster: a short one, yes.
<gary_poster> hey, ok jml thanks.  I'm ready with skype (garyposter) or mumble
<jml> gary_poster: ok
<rvba> :w
<jml> gary_poster: TestGenericBranchCollectionVisibleFilter in lp.code.model.tests.test_branchcollection
<jml> gary_poster: lp.code.tests.test_branch.TestAccessBranch
<jml> did my reply to "An open invitation to do a no-downtime-deploy" go through?
<gary_poster> jml, from almost three hours ago, yes
<jml> gary_poster: cool, thank.
<jml> thanks.
<gary_poster> yw
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256
<mtaylor> wow. SSO fail
<nigelb> heh, probably should go on /topic in the other channel.
<elmo> sorry, it is being worked on
<elmo> it's back up
<mtaylor> is there any way to get a user's openid deletgate url from launchpadlib/launchpad api?
<maxb> I think that info is only revealed to admins and the user themselves by the security policy at all
<jml> mtaylor: Can't see anything obvious at https://launchpad.net/+apidoc/devel.html#person
<mtaylor> jml: yeah. me neither - I was sort of hoping there was something hidden I didn't know about
<mtaylor> maxb: it's visible on the web page
<mtaylor> maxb: view-source in the link rel  -- otherwise the calling context wouldn't know what url to visit
<maxb> mtaylor: But only for yourself, AFAIK?
<mtaylor> maxb: nope. for anybody
<mtaylor> for instance: view-source:https://launchpad.net/~jaypipes
<maxb> oh, so it is
<mtaylor> (I'm working on a script to pre-populate a user database with launchpad information for a system that will use launchpad/ubuntu openid as a point of SSO)
<mtaylor> but to match that up, I _believe_ I also need to inject the openid local_id so that the SSO code will know how to associate an sso user with pre-filled in meta info
 * mtaylor hopes he isn't about to write a web-scraping script - it's possilbe there's another way to solve this...
<maxb> If configured to allow it, SSO can tell you the lp id
<maxb> as part of the OpenID exchange
<mtaylor> yeah - I'm hoping that will be enough to make the match on the side of the SSO side ... need to chat with the other dev
<thedac> I have cocoplum OOPes alerting
<cr3> can some celebrities authenticate to the api?
<lifeless> no
<lifeless> celebrity != service account
<cr3> lifeless: that's what I thought, thanks for the confirmation!
<nigelb> I wonder what happened about the bot discussion.
<dobey> cr3: i haven't had any problems doing so ;)
<lifeless> dobey: You aren't a Launchpad celebrity
<deryck> lifeless, hey there....
<dobey> lifeless: you are no fun
<lifeless> deryck: hi :)
<deryck> lifeless, are changes to trusted.sql treated like schema changes?  Or is trusted.sql applied on every nodowntime deploy?
<lifeless> dobey: I'm plenty fun.... a little later in the day :P
<lifeless> deryck: it won't be no.
<lifeless> deryck: you need /all/ changes to be in a patch script
<lifeless> deryck: you may also need the function in trusted.sql to have 'make schema' work.
<lifeless> deryck: this is a (lot) fugly, and deserves a bug.
<deryck> lifeless, ah, gotchas.  yeah, that is fugly, but I follow.
<bac> lifeless: thanks for your comments on my merge proposal
<lifeless> sorry have to run to hospital for an appt
<lifeless> I hope lp didn't make me claim it, as I only wanted to add that headsup
<deryck> Later on, everyone.
<wgrant> danilos: Thanks for that quick fix and deploy.
<LPCIBot> Project db-devel build #738: FAILURE in 5 hr 39 min: https://lpci.wedontsleep.org/job/db-devel/738/
<poolie> spiv, bmp re-sent
<spiv> Thanks
#launchpad-dev 2011-07-21
<cjohnston> I'm trying to pull the summary from a spec, but it doesn't seem to be working. I am using elem.get just like I did for all the other info I am getting, but this one isn't working.
<cjohnston> Any ideas?
<wgrant> cjohnston: Isn't working?
<wgrant> And which library are you using? That doesn't look like launchpadlib.
<lifeless> http://blog.bitbucket.org/2011/07/07/redesigned-commits-screen/
<cjohnston> simplejson
<wgrant> It's there.
<cjohnston> wgrant: http://pad.ubuntu.com/dYgqgL4JaA
<wgrant> cjohnston: It's in the JSON that LP sends, but I don't know how you are getting the JSON.
<cjohnston> wgrant: http://bazaar.launchpad.net/~summit-hackers/summit/1.x/view/head:/summit/schedule/models/meetingmodel.py#L188
<cjohnston> Does that give you any more info wgrant ?
<wgrant> cjohnston: Not really. Launchpad is sending the correct JSON, so you'll need to check the JSON you have.
<jelmer> wgrant: hi
<wgrant> jelmer: Hi.
<jelmer> wgrant: You tried to land lp:~jelmer/launchpad/publisher-use-debian-2  earlier but it failed for some reason, do you know what it was?
<wgrant> Hmm, I recall I ran it through ec2, but cannot find an email about it..
<jelmer> wgrant: no worries, I'll have another look at it
<jelmer> wgrant: thanks
<wgrant> Sorry.
<jelmer> wgrant: Don't be :) I appreciate you trying to land it, when I didn't have time to.
 * jelmer gets some sleep
<wgrant> Night.
<lifeless> does anything still use BranchSparkView ?
<wgrant> Heh
<StevenK> lifeless: A bunch of tests?
<lifeless> nuke from orbit time
<wgrant> It was disabled during 3.0, to be fixed and revived within a month.
<StevenK> And IBranch:+spark
<StevenK> lifeless: Are you nuking it?
<lifeless> StevenK: not right now, I have managerial thingies to do
<lifeless> then some folk from .au are dropping by before they head home
<lifeless> if you are interested in nuking it, please do.
<lifeless> [red button]
 * lifeless pushes the red button
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 241 - 0:[######=_]:256
<StevenK> wgrant: Can haz review? https://code.launchpad.net/~stevenk/launchpad/destroy-branchsparkview/+merge/68627
<wgrant> StevenK: What about the JS?
<StevenK> There's JS?
<wgrant> That view provides JSON to the JS.
 * StevenK looks for the JS
<StevenK> wgrant: Three more methods found and destroyed, but the JS is proving harder to locate.
<wgrant> Already removed in r9680
<StevenK> Ah, so the JS is long-dead
<StevenK> wgrant: No review, then? :-)
<wgrant> It is reviewed.
<wgrant> Was just checking you'd got all of it.
<StevenK> Ahhh
<StevenK> I got all of it after I killed the other 3 methods?
<wgrant> Yes
<StevenK> Excellent, thanks
<lifeless> righto, time to write up docs
<lifeless> StevenK_: thanks for fixing that timeout!
<lifeless> (+spark)
<StevenK_> Haha
<StevenK_> lifeless: Is there a bug about it?
<lifeless> none filed yet
<lifeless> it was 1/0 in todays timeout report
<StevenK_> lifeless: Fair enough. My branch is currently ec2ing
<lifeless> no-qa is fine IMO
<lifeless> I'm just happy  :)
<wgrant> Aha!
<wgrant> An excuse to kill IUpstreamBugTask and co.
<StevenK> Oh?
<wgrant> Do you know of it?
<wgrant> IUpstreamBugTask, IProductSeriesBugTask, IDistroSeriesBugTask, IDistroBugTask are marker interfaces.
<StevenK> Nope
<wgrant> Which need to change when the target of a task changes.
<wgrant> Which means that retargetting bugs between target types needs to change them.
<wgrant> But that makes lazr.lifecycle sad.
<wgrant> So, I may finally kill them.
<StevenK> More code deletion? :-)
<wgrant> Since there's just about no good reason to have them.
<wgrant> There are only a couple of places that care, and they can just check the interfaces of the target directly.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #739: FIXED in 5 hr 38 min: https://lpci.wedontsleep.org/job/db-devel/739/
<StevenK> wgrant: Do you know if dogfood has query logging on?
<wgrant> StevenK: It doesn't seem to.
<StevenK> I am disappoint.
<wgrant> Ja.
<wgrant> OTOH the disk space would probably be sort of bad.
<StevenK> Meh, mawson has 150GiB free :-P
<wgrant> Now :)
 * StevenK peers at database/schema/launchpad.html
<StevenK> Hmmm. 450 results for '%ubuntu%' in PillarName
<StevenK> At least one of them is the distribution itself.
 * wgrant headdesks.
<StevenK> wgrant?
<wgrant> -        else:
<wgrant> -            assert (
<wgrant> -                "Expected IDistroSeriesBugTask or IProductSeriesBugTask. "
<wgrant> -                "Got: %r" % bugtask)
<wgrant> There should really be a warning for that.
<adeuring> good morning
<jtv> hi adeuring
<jtv> oh, and hi rvba!
<adeuring> hi jtv
<jtv> wgrant: I take it the cocoplum rollout succeeded then?
<wgrant> jtv: Soyuz only had unrelated OOPSes this morning, so yup, seems like it all went OK.
<rvba> hi jtv!
<jtv> Great.
<rvba> Good morning all.
<jtv> wgrant: for my next trick, I sort of re-did the distro-publisher code.
<wgrant> So I saw!
<wgrant> Well.
<wgrant> I saw the "diff too big" warning in the email.
<wgrant> I've not seen the change itself.
<wgrant> Similar to what you did to process-accepted?
<jtv> "Diff too big"?  I hadn't seen that.
<jtv> Should be "only" 1200 lines or so.
<jtv> I suppose.
<jtv> I didn't feel I could just go on adding to the problem.
<wgrant> Indeed.
<wgrant> So much evil to destroy :(
<jtv> The question right now is: how do I Q/A publish-distro?
<jtv> (Have a look at it â it wasn't nearly as hairy as it seemed)
<wgrant> That is a lot of tests.
<jtv> Yes, it really helps to know that the components work.
<jtv> Found one or two cases of "this couldn't have worked."
<jtv> hi bigjools
<bigjools> morning
<stub> Anyone want to review https://code.launchpad.net/~stub/launchpad/staging/+merge/68529 for me? Boring meat and potatoes staging/production rollout code.
<mrevell> Morning
<StevenK> wgrant: You remove a XXX from 2006 that references bug 55089
<_mup_> Bug #55089: IUpstreamBugTask should be renamed to IProjectBugTask <lp-bugs> <tech-debt> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/55089 >
<StevenK> Ah, which is already linked. Nice.
<stub> 1.1 billion branchrevision rows now... going up by about 50 million/month.
<mwhudson_> stub: is the disk space reduced at all by not having id as a primary key?
<stub> It is now that the nodes have all been rebuilt.
<mwhudson_> ah right
<mwhudson_> i guess it's a substantial fraction of the node rebuild time though...
<jtv> stub: I'm still amazed by the lack of alarms and sirens for that change.
<danilos> -back
<danilos> -back
<stub> mwhudson: the id column plus various indexes came to a few 10s of GB
<danilos> whoops, wrong keyboard layout and I insist on retyping it :)
<jml> really really starting to wish Launchpad has some kind of push notification
<wgrant> jml: Only once you lost your ability to direct? :)
<jml> wgrant: now I can afford to be selfish.
<wgrant> :(
<wgrant> Selfishness is what Launchpad has needed for a number of years now...
<wgrant> Stakeholder direction only gets us so far in the sensible directions.
<wgrant> We've been making it work, rather than making it not suck.
<wgrant> :)
<jml> wgrant: You tell me this only once I've lost my ability to direct? :P
<wgrant> Heh
<nigelb> heh, so hire someone who's as selfish as you  :D
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 241 - 0:[######=_]:256
<wgrant> danilos: Thanks for organising that deployment!
<nigelb> jml: communicating internally in the open is not a bad thing. I like reading the list even though I may not be able to do anything for a few items.
<danilos> wgrant, hey, you are welcome :)
<danilos> wgrant, and it should be more of a norm for the rest of us to do it
<stub> allenap: https://code.launchpad.net/~stub/launchpad/staging/+merge/68529 :-)
<allenap> stub: I'm already looking at it :)
<allenap> stub: Out of interest, why did you go for using /etc/init.d/pgbouncer instead of using PAUSE/SUSPEND/RESUME via the pgbouncer virtual database?
<stub> allenap: We need to shutdown pgbouncer because suspend/resume just blocks inbound connections, not reject them.
<stub> allenap: There is an option to tell it to reject connections that can't connect after x seconds, but it doesn't work in suspend mode (either bug or design, don't know)
<stub> allenap: in the lightning talk I gave, I sketched out a situation where connections block but after further discussions with lifeless we are just going to drop them and require the clients to cope and/or present nice errors to the users.
<allenap> stub: Okay. The init script does cause a pause then shutdown, but with only 1 second delay. Is that okay.
<allenap> ?
<stub> allenap: That is fine.
<allenap> stub: r=me
<stub> And I could shut it down by other means (psql -d pgbouncer -c shutdown, or kill(1)) if it becomes a problem.
<stub> ta
<danilos> jtv1, rvba: hi, there are a few revisions of your needing QA, it'd be nice if you can get to that today
<jtv1> danilos: I know, thanks
<danilos> yours
<rvba> danilos: will do.
<rvba> danilos: done.
<danilos> rvba, thanks
<jtv1> allenap: free for a review?  https://code.launchpad.net/~jtv/launchpad/get_derived_distros/+merge/68675
<deryck> Morning, all
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugs: 241 - 0:[######=_]:256
<jcsackett> allenap: can i request a review for https://code.launchpad.net/~jcsackett/launchpad/a-private-place-to-live/+merge/68687?
<allenap> jcsackett: Sure. I think jtv's first then you.
<jtv> allenap: I've been producing them faster than you can review them.  :)
<jcsackett> allenap: works for me. i can always do jtv's if you haven't already started. just can't review my own branch. :-P
<jtv> jcsackett: I have a few more ready now.
<jtv> So there's enough for both of you!
<jcsackett> jtv: erm, eh, fantastic? :-p
<jtv> hehheh
<allenap> jcsackett: Okay, you start with jtv, I'll do yours, then move back onto jtv's new ones.
<jcsackett> allenap: righto.
<jcsackett> jtv: r=me on the first one.
 * jcsackett likes short diffs.
<jtv> thanks!
<jcsackett> jtv: you have a preferred next branch for me to look at? i see three in the queue.
<jtv> Yes: the bug-788078 one, thanks.
<jtv> That's bigger, but I carved out pieces for a separate branch â which you just reviewed.  :)
<jcsackett> jtv: ok. looking at that one momentarily.
<jtv> Thanks.
<sinzui> allenap, jcsackett: are either of you looking at https://code.launchpad.net/~wallyworld/launchpad/picker-wiring-812255/+merge/68629
<jcsackett> sinzui: it's in the queue. :-)
<allenap> sinzui: What Jon said :)
<nigelb> oh, 2 reviewers today! perfect day to get my branch out.
<bac> allenap: could you review https://code.launchpad.net/~bac/launchpad/bug-799901/+merge/68590 at your leisure?
<allenap> bac: Sure :)
<jcsackett> jtv: correct me if this is wrong, but it looks like this adding a fair bit of new iteration to the process; is this script not one we worry about time to complete on?
<nigelb> ugh, I need some help with a branch. Does anyone have a few minutes to help figure out why I'm getting http://dpaste.com/573320/ for https://code.launchpad.net/~nigelbabu/launchpad/spec-sub-sort/+merge/63315
<jtv> jcsackett: only for Ubuntu, in which case the list it iterates over is of length 1.
<allenap> nigelb: I'll take a look.
<nigelb> allenap: Thanks a bunch!
<jtv> jcsackett: the idea is we can do the work for all _other_ distros we do it for much more quickly.
<jcsackett> jtv: so if we're doing all-derived we're well aware it could take a bit and that's just fine?
<jcsackett> jtv: ok.
<jtv> Yes.  If it's not, we can easily extend this to more fine-grained load-balancing, such as "publish a thru f on this server."
<jcsackett> jtv: ah, yes, you mentioned granularity in the cover letter.
<jtv> Thought so.  :)
<jcsackett> ok, jtv, r=me. just wanted to make sure i wasn't approving a performance bomb. :-)
<jtv> Good job.  Thanks!
<bigjools> cody-somerville, timrc: do you have time to talk about OEM's ongoing requirements with regard to our derived distros feature?
<timrc> bigjools, if I could keep natty from crashing, yes :)
<cody-somerville> bigjools, When?
<bigjools> can you do it in the next hour or so?
<timrc> next week would probably be better as cody-somerville will be back in North America
<bigjools> or now? :)
<timrc> oh..
<jcsackett> allenap: thanks for your comments, i'll address those changes. how much longer are you on review for?
<bigjools> next week is ok if that's better
<cody-somerville> bigjools, I'm at DebCamp this week. Next week would be much better for me.
<allenap> jcsackett: Another couple of hours, then I'll be back later this evening.
<jcsackett> allenap: ok. thanks.
<bigjools> cody-somerville, timrc: ok that's fine.  I want to see how attached you are to using those crazy custom suites :)
<allenap> jcsackett: I think you only *need* to address point 3.
<timrc> bigjools, we're more crazy about deriving distros from multiple sources I think :) can we do that yet?
<bigjools> timrc: yes
<jcsackett> allenap: right, but if i can i'll address the others. no reason to delay good fixes if it's not out of scope.
<bigjools> it's not quite released yet but we got it working finally, today
<timrc> bigjools, neat!
<cjwatson> what's the best way to submit a set of proposed changes for the Launchpad PPA?
<cjwatson> I guess I need to do them for all of lucid, maverick, natty :-/
<bigjools> cjwatson: if you have packages in a different PPA we can review and copy
<cjwatson> I do (well, mvo does), but only for lucid at the moment
<bigjools> well it's your fault, you keep releasing new series every 6 months :)
<cjwatson> gosh, I never knew I had a choice. :-)
<cjwatson> ok, I'll see if I can put together a staging PPA with everything
<bigjools> great
<nigelb> hrm, is subscribing to a blueprint broken on trunk?
<nigelb> s/trunk/devel
<nigelb> I just realized I can't subscribe to a blueprint on my dev env.
<allenap> nigelb: It should read user=subscribed_by not user=user.
<allenap> Line 567 of specification.py.
<nigelb> ah
<nigelb> I think I didn't notice it when "fixed" the merge conflict when I merged from devel the last time
<allenap> nigelb: On the one hand sort_for_display() is generic - it can sort by any attribute - on the other it calls .lower() on the value it finds. I think you'd be better off without it; just fix the bug in specification.py and don't worry about the general case. As jtv notes in his review, it's a much bigger problem. What we really need is an agreed way to derive a collation key, but that's out of scope for this fix.
<jtv> Isn't sort_for_display exactly what should hide that problem from us, so we can fix it once?
<nigelb> allenap: The inital bug was there was no lower() cuasing the sorting to be messed up.
<nigelb> what jtv just said.
<jtv> I am so sorry for opening this whole can of worms...  :)
<nigelb> hehe
<allenap> jtv: Not really, because it conflates sorting with getting a collation key.
<jtv> Unfortunately, yes.  But at least it will mark places where we tried to work around this!
<nigelb> ugh, bigger traceback. http://dpaste.com/573346/
<nigelb> If you could help me figure out how to start lookig at zope traceback, I'd be very grateful.
<bigjools> nigelb: line 156 is the clue
<nigelb> hm, so I look at that template?
<bigjools> nigelb: it's telling you where the traversal failed
<bigjools> in view/sorted_subscriptions
<nigelb> ah
<nigelb> so I look at browser/specificationsubscription.py where sorted_subscriptions is seems to be defined
<allenap> jtv, nigelb: There is already an agreed upon collation key function for people: lp.registry.model.person.person_sort_key
<nigelb> allenap: Oh.
 * nigelb headdesks and rewrites.
<allenap> nigelb: Hehe, don't worry, it's well hidden :)
<allenap> nigelb: Fwiw, here's a diff of the changes I would make: http://paste.ubuntu.com/649234/
<nigelb> allenap: thanks!
<allenap> jcsackett: By the way, would you have time to review a branch of mine? https://code.launchpad.net/~allenap/launchpad/localpackagediffs-filter-by-package-set-bug-809786-overlay/+merge/68647
<jcsackett> allenap: as it happens, i would. :-)
<allenap> jcsackett: Thanks :)
<nigelb> allenap: I can't import person_sort_key.
<nigelb> "ImportError: cannot import name person_sort_key"
<allenap> nigelb: Yeah, there appears to be a circular import problem. In my diff I import it near where it's used and that works.
<nigelb> allenap: ah, but I also need to use it in 225-ish
<allenap> nigelb: Yeah. See my diff :)
<nigelb> ah, right. Sorry.
<nigelb> I should learn to read :P
 * nigelb grrs.
<nigelb> No traceback. But no worky either.
<allenap> nigelb: Push your branch and I'll take another look.
<LPCIBot> Project db-devel build #741: FAILURE in 5 hr 40 min: https://lpci.wedontsleep.org/job/db-devel/741/
<nigelb> allenap: pushed :)
<jcsackett> allenap: are you planning on filing bugs for all these xxx's with "bug=???" ?
<allenap> jcsackett: I suppose so... the sneaky part of my mind had tried to forget about that :)
<jcsackett> allenap: if you'll file those bugs and update the XXX, r=me. i can go ahead and approve it now with that understanding. :-)
<allenap> jcsackett: Thanks, that's awesome.
<jcsackett> allenap: that's a nice trick with the auto re-aligning of the overlay, btw. i wonder why that's not default behavior.
<allenap> jcsackett: I'll do the same with your review.
<jcsackett> oh, thanks.
<allenap> nigelb: Right, so is it not ordering correctly in the UI?
<nigelb> allenap: no.
<allenap> nigelb: SpecificationPortletSubcribersContents.sorted_subscriptions returns the subscriptions that can be unsubscribed by the user before those that the user cannot. Could that explain what you're seeing?
<allenap> nigelb: I have to go very soon, but I'll be back later (at ~1900 UTC). I've subscribed to your branch so just update the mp with your notes and I'll take a look.
<nigelb> allenap: I'll do that :)
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 241 - 0:[######=_]:256
<nigelb> allenap: You were right. That's what I was seeing.
<bac> thanks for the review allenap -- nice suggestions
<timrc> I have a user asking what the contents of  /etc/pkgbinarymangler/striptranslations.conf looks like for one of our P3A's -- can anyone assist with that, here?
<allenap> bac: Cool, np.
<allenap> nigelb: That's cool. I reckon your branch is nearly ready to land. If you incorporate the minor changes to the test I put in the diff I reckon it will be ready.
<LPCIBot> Project devel build #908: FAILURE in 5 hr 39 min: https://lpci.wedontsleep.org/job/devel/908/
<nigelb> allenap: yeah, doign that now.
<nigelb> allenap: pushed :)
<lifeless> morning
<lifeless> cjwatson: stuff for the lp ppa? we only realy care about about lucid, natty and oneiric atm
<allenap> nigelb: I'll land it for you.
<nigelb> allenap: thanks! Also thanks for hand-holding through this :)
<allenap> nigelb: No worries, thanks for taking the time to do this. It's making Launchpad better.
<nigelb> allenap: Yeah, talking to jml at UDS helped a huge deal in getting off my mental block about patching launchpad :)
<allenap> nigelb: jml's UDS seems to have been a big success. It's hard getting started on Launchpad, but I think it's rewarding; we should be able to release your fix tomorrow or Monday.
<nigelb> yay :)
<nigelb> Now I need to find out a way to have wallyworld's working hours and my awake hours and my free hours all to fall into the same slot.
 * cody-somerville recommends a time machine.
<allenap> cody-somerville: OEM's working on one of those, right?
<cody-somerville> I'm not at liberty to say, sorry.
<lifeless> sure you, just remember to go back in time and stop him asking about it
<nigelb> haha
<allenap> nigelb: Your branch should land in a few hours once the full test suite has run. I'll check on it in the morning. I'm off now, cheerio.
<nigelb> allenap: laters! thanks again!
<cjwatson> lifeless: no maverick?  ok, that simplifies things a bit
<cjwatson> though actually a bit late since I already did those backports :-)
<cjwatson> oh well, will post things from debconf
<lifeless> cjwatson: its nice to have, but no dev uses it and we don't deploy on it
<cjwatson> ok
<lifeless> things that go on biuldds obviously need to support all the flavours buildds do
<lifeless> but IIRC this isn't one of those
<cjwatson> indeed not
<wgrant> lifeless: Yeah, this is just getting rid of the interfaces, as it was big enough.
<wgrant> lifeless: I plan to extract most of the target-specific stuff from lp.bugs.
<wgrant> Because it is pretty revolting.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #742: FIXED in 5 hr 35 min: https://lpci.wedontsleep.org/job/db-devel/742/
<wgrant> sinzui: But won't a productseries milestone vocab only show milestones within that series, which is what we want here?
<sinzui> wgrant, I think that is what I am trying to understand. a  productseries should only target to a milestone that belongs to a series. a product bug could be targeted to any milestone for any series
<sinzui> wgrant, there is not thing wrong with what you did. I just want to know WTF Lp thinks it is doing
<sinzui> It will all be moot when we simplify bugtasks to series. This vocab will be forced to have similar rules
<wgrant> sinzui: Right.
<wgrant> I feel that we should probably preserve existing behaviour for now.
<wgrant> This pipeline will already extend into 5 or 6 branches.
<sinzui> absolutely keep it the way it is. I just wanted to know if we know about this bug. Maybe it needs to be reported
<wgrant> Yeah.
<wgrant> Ah, you've approved it now. Thanks!
<wgrant> wallyworld: Should bug #806785 be closed?
<_mup_> Bug #806785: Person picker displays duplicate info when selecting recipe owner <disclosure> <picker> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/806785 >
<wallyworld> wgrant: yes,. i believe so. i'll check. perhaps i didn't link to to the branch
<wgrant> It's linked, but maybe wasn't until too late.
<wallyworld> i would have linked it before i did the mp
<wgrant> jtv: https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1171/steps/shell_6/logs/summary <- I think your ArchiveCruftChecker refactoring might have made it spew crufty to stderr?
<wgrant> Argh.
<wgrant> Series tasks.
<lifeless> they will all be at some point
<wgrant> Yeah, but the current implementation is a bit unfortunate.
<wgrant> In particular, you can presently change the sourcepackagename of a series sourcepackage task through the web UI.
<wgrant> Probably want to forbid that.
<wgrant> (I'm working on pushing all target changes through transitionToTarget, which doesn't let you change to/from a series target, because that's what nominations are for)
<wgrant> Hmm, retargetting them breaks the UI a bit.
<wgrant> If there's no corresponding DSP task, the name is not shown at all.
<wgrant> lifeless: Any opinions?
<lifeless> uhm
<sinzui> StevenK, lp:~sinzui/launchpad/dsp-picker-fields
<lifeless> so big picture
<lifeless> we want users that have sufficient privilege to just move stuff around willy nilly
<lifeless> e.g. same-series different-sourcepackagename is a no-brainer change to permit everyone
<wgrant> The current model makes that hard, but when everything is a series task it will be easier.
<wgrant> SP tasks are not meant to exist without a corresponding DSP task.
<wgrant> The UI breaks.
<lifeless> we want users that have insufficient privilege to have to go through a handshake - the nomination process - for changes they are not permitted
<lifeless> wgrant: SP being DSSP really ?
<wgrant> lifeless: Right.
<lifeless> this sound slike a conjoined masterish thing
<lifeless> anyhow
<wgrant> Similar.
<lifeless> I have no particular opinion; I'd like that we move towards the big picture
<lifeless> I'd prefer we don't let broken things happen
<lifeless> if we permit something that breaks the model today, I have no objection to restricting it
<lifeless> I'd question restricting something that isn't restricted today.
<lifeless> (and doesn't cause breakage)
<wgrant> I haven't seen any breakage around (except for really old bugs like bug #43150), so I presume it's not done much or at all.
<_mup_> Bug #43150: [SRU] maxima frontends fail to connect <wxmaxima (Ubuntu):Fix Released> <gcl (Ubuntu Dapper):Fix Released by wgrant> <maxima (Ubuntu Dapper):Fix Released> < https://launchpad.net/bugs/43150 >
<sinzui> wallyworld, http://www.jslint.com/lint.html
<wgrant> lifeless: It is tempting to forbid it, and if someone complains then chastise them and potentially hack transitionToTarget to permit changing to another SP in the same DS.
<lifeless> does it break things no matter what ?
<lifeless> or does it break things under some circumstances only ?
<wgrant> Unless you change all of them at once.
<wgrant> Meh, for now I will special-case permit it.
#launchpad-dev 2011-07-22
<spiv> Does the branchscanner run on qastaging?
<wgrant> Not regularly, AFAIK.
<spiv> That's going to make QA'ing bug 813349 inconvenient :/
<_mup_> Bug #813349: hard to get from mp to per-revision diffs <qa-needstesting> <Launchpad itself:Fix Committed by spiv> < https://launchpad.net/bugs/813349 >
<StevenK> spiv: You can request it to be run manually
<wgrant> Anyone feel like reviewing https://code.launchpad.net/~wgrant/launchpad/retarget-bugtask-model/+merge/68761?
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 240 - 0:[######=_]:256
<StevenK> wgrant: r=me
<wgrant> Thanks.
 * StevenK prods wallyworld about OCR
<wallyworld> StevenK: sorry, i was caught up finishing a branch
<wgrant> lifeless: How is HA-librarian going?
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 240 - 0:[######=_]:256
<lifeless> wgrant: rt something or other
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 240 - 0:[#######=]:256
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #909: FIXED in 5 hr 43 min: https://lpci.wedontsleep.org/job/devel/909/
<wgrant> lifeless: Hah, subtle.
<poolie> destroy beta testers?
<poolie> life imitates glados
<wgrant> Heh
<lifeless> wgrant: ?
<wgrant> lifeless: The progress bar.
<lifeless> wgrant: heh yes
<wallyworld> jtv: you around? want to mentor https://code.launchpad.net/~gary/launchpad/bug812335/+merge/68762 ?
<jtv> wallyworld: no, I'm not here.  It's almost 4 AM here.
<wallyworld> jtv: oh, sorry! you still in europe then i assume
<jtv> Yup
<wallyworld> when are you back?
<nigelb> wallyworld: It seems the intersection of my freetime + your available hours don't collide, so I wonder if I can take it to email? :)
<wallyworld> nigelb: otp, talk soon
<wallyworld> nigelb: you can email me. or ping me on irc. i'm normally lurking in my evening and will respond to any queires
<nigelb> wallyworld: I finally landed something I had pending yesterday, so I can work on linkfying + titling bugs with XHR.
<wallyworld> nigelb: excellent, well done.
<nigelb> where is it that I should start looking? :)
<nigelb> lib/lp/bugs/javascript?
<wallyworld> ok, there's 2 places - the javascript which makes the xhr call and the python code which runs serverdide
<wallyworld> serverside
<wallyworld> justa moment, i'll grab the file names
<wallyworld> the javascript file is lp/lp/app/javascript/lp-links.js - this harvests potential links when the page loads and makes a single xhr call to have those processed
<wallyworld> lp/app/browser/linkchecker.py is the python code which receives the xhr call, does the work, and returns the results as a json data structure. the js code above then processes the result and pokes the dom to update the page
<nigelb> so, I need to update the python code to return title and process that extra information in the js.
<wallyworld> the bit which wires it up is the +checklinks browser:page section in lp/app/browser/configure.zcml
<wallyworld> yes, that is correct
<wallyworld> you can add extra fields to the json dict returned by the xhr call
<wallyworld> and then update things like the title attribute of the anchor or whatever in the js
<wallyworld> make sense?
<nigelb> Yes
<nigelb> Does this already check if the bug exists?
<wallyworld> looking at the code, it only handles branch links so far, but it does check that the branch exists
<wallyworld> so you would need to extend it to support harvesting and processing bug links
<nigelb> ah
<wallyworld> you could make the xhr call send a dict with a branches and a bugs list
<nigelb> The current linkyfing seems to be server-side. Is that the right wway?
<wallyworld> yes, the serverside is the place to check the validity or permission to access info about the bug/branch
<wallyworld> what is it you want to do exactly?
<nigelb> I'd like to look for "bug 1234" type links, check if they exist, get the title, add the title attribute if it exists and remove the linkification if the bug doesn't exist.
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<wallyworld> are these would appear in text fields like comments, merge proposal descriptions and the like?
<nigelb> Yeah, the same
<wallyworld> there's a file lp/app/browser/stringformatter.py with a method linkify_bug_numbers() which parses the text and does the initial linkification
<wallyworld> this is done when the view is rendered
<nigelb> so, does this make sense.
<nigelb> add a css class for all the bugs, fetch all of the links with that class, send xhr request, modify dom.
<wallyworld> yes. the comment on the code which does the initial linkification says:
<wallyworld>         # Don't look up the bug or display anything about the bug, just
<wallyworld>         # linkify to the general bug url.
<wallyworld> so the work you are proposing would be post-processing in effect to add the extras
<nigelb> yeah, it wwas removed for performance.
<wallyworld> like for the branches, make invalid (or invisible due to privacy) links grey etc
<nigelb> yep!
<wallyworld> so using a css class to mark the bug links is good because that's how the current code for branch link processing works
<wallyworld> all the plumbing is there, you just need to extend the js and python ti support another different type of link processing
<nigelb> oh cool! I'll try tonight to start step by step. I'll update you by email since your times don't match very well.
<wallyworld> ok. but feel free to try and ping me on irc if you want. i may have a glass of red in hand but should be coherent enough to type :-)
<nigelb> haha
<nigelb> ok :)
<wallyworld> i have soccer first too, so it will be late evening before i am in front of the laptop again
<nigelb> the problem is your late is still early enough for me that I'd be at work.
<wallyworld> what time is it now where you are?
<nigelb> 9:20 AM
<wallyworld> so you ate 4.5 hours behind me
<wallyworld> are
<wallyworld> so except for fridays when i have soccer, i am normally pingable at a time which would be close to EOD for you
<nigelb> ah, I'll note that :)
<wallyworld> or even at start of your day, it's only afternoon for me :-)
<wallyworld> so we do have a fair bit of overlap
<nigelb> heh, start of the day is often problematic. I tend to sleep late and wake up fairly late :)
<wallyworld> np. i used to do that before kids :-(
<nigelb> heh, off to work. Laters!
<StevenK> wgrant, lifeless: Do either of you want to mentor wallyworld's review of https://code.launchpad.net/~stevenk/launchpad/dsp-vocab-issues/+merge/68763 ?
<wgrant> StevenK: Done.
<StevenK> wgrant: Thanks!
<wgrant> wallyworld: Around?
<wgrant> We have yet another form picker regression.
<wgrant> They don't populate the textbox.
<wgrant> Works OK on lpnet.
<wgrant> But there are no picker changes since then.
<wgrant> So WTF?
<wallyworld> wgrant: yo
<wgrant> wallyworld: See #launchpad-ops.
<wgrant> qastaging ninja-updated.
<wgrant> So there was in fact a relevant rev: r13485
<wgrant> Reverting now.
<StevenK> That makes you fairly distracted if it takes 50 minutes to update and you didn't notice :-P
<wgrant> It was mostly updated when I started looking at it, and crucially when I loaded the deployment report :)
<wgrant> I think jcsackett may have failed at conflict resolution with wallyworld's work.
<wgrant> The extra arg to create() is missing.
<wgrant> So it never gets hooked up to anything.
<wgrant> Anyway, reverting.
<jtv> Hi wallyworld :)
<wallyworld> jtv: hello
<jtv> wgrant: I don't think I touched the archivecruftchecker as such.
<wgrant> $ bzr diff -c13478 lib/lp/soyuz/scripts/tests/test_archivecruftchecker.py | diffstat test_archivecruftchecker.py |   19 ++++++++-----------
<wgrant> Oh.
<wgrant> That's tests.
<wgrant> Ahem
 * jtv stands back and admires wgrant's genius :)
<jtv> But
<wgrant> Hm.
<wgrant> But that could be it.
<jtv> I did change something in the test.
<wgrant> It is those tests that are spewing crap to stderr.
<jtv> Yes.  It's because publish-distro is now a LaunchpadCronScript.
<wgrant> It previously redirected stdout/stderr to null pipes.
<jtv> Where that test instantiates the object, it should inject a DevNullLogger.
<wgrant> But now runs PublishDistro directly.
<wgrant> Yeah.
<jtv> I can do that.
<wgrant> Thanks.
<stub> I wanna QA my db update script, which will make qastaging die for a few seconds.
<wgrant> Feel free.
<wgrant> We are bad and a while away from buildbot completion.
 * wallyworld goes out to buy food
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 240 - 0:[#######=]:256
<lifeless> stub: \o/
<lifeless> stub: we need a catchup call too
<lifeless> I couldn't last night - family thing, and am eating now; can we make a time later today?
<stub> lifeless: sure
<stub> I might grab some fud too
<lifeless> shall we say in one hours time then ?
<stub> k
<adeuring> good morning
<mrevell> Howdy
<bigjools> morning everyone
<poolie> hi bigjools, mrevell
<poolie> lifeless: still here?
<bigjools> o/
<lifeless> kindof :>
<poolie> the U3011 is amazing
<poolie> you should totally get one
<poolie> imax-mode
<lifeless> \o/
<bigjools> woo!
<lifeless> poolie: photo ?
<poolie> lifeless: https://plus.google.com/112646476239496153808/posts/ApTZ7GyWAdo
<poolie> looks like a big glowing thing
<poolie> heh, that laptop looks like a netbook
<lifeless> stub: now ok ?
<bigjools> jtv: Contents is not updated yet it seems
<jtv> oh
<bigjools> http://archive.ubuntu.com/ubuntu/dists/oneiric/ for example
<bigjools> jtv: http://launchpadlibrarian.net/75734346/2TgWVd4UTg0dWE5ieBiCMVPmNBE.txt
<jtv> grrr
<bigjools> jtv: need to set up script monitoring for this :)
<jtv> We don't have that?
<bigjools> jtv: you need to tell losas to do so
<bigjools> and this is a new LPS
<lifeless> LPS? [not LaunchpadProductionStatus I assume]
<jtv> Local Processor Status?
<jtv> Hi lifeless :)
<lifeless> hola jtv
<StevenK> LaunchpadScript
<StevenK> I guess
<jtv> Ah yes :)
<stub> lifeless: I'm going to try skype on my desktop again
<lifeless> ok
<bigjools> ARGH
<lifeless> call me  :)
<bigjools> FeatureFixture cancels all other feature flags.  I've been burned.
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: adeuring | Critical bugs: 240 - 0:[#######=]:256
<jtv> bigjools: generate-contents-files cron fix has deployed, so we'll see.
<bigjools> jtv: oh, cowboy?
<jtv> cron change â no code change.
<bigjools> ah cron, sorry
<jtv> bigjools: another thing: we should be able to scratch steps 12 & 14 off https://wiki.ubuntu.com/NewReleaseCycleProcess now, shouldn't we?
<bigjools> yup
<jtv> cjwatson: I realize we discussed this already, but I'd like to check up: if we do ^^^ then step 13 could go as well because you watch these files like a hawk anyway, right?
<stub> Anyone running Aurora ?
<henninge> adeuring: hey!
<henninge> adeuring: Moin ;)
<henninge> adeuring: Wenn du so nett wÃ¤rst ...
<henninge> bitte
<henninge> ;)
<henninge> file:///home/henning/canonical/lp-branches/devel-work/lib/lp/code/javascript/tests/test_branchdiff.html
<henninge> crap
<henninge> mist
<henninge> adeuring: https://code.launchpad.net/~henninge/launchpad/bug-813627-preview-diff-overlay/+merge/68820
<adeuring> henninge: sure, I'll look
<henninge> adeuring: thank you!
<jml> I'm writing a launchpadlib script, running it against staging for now.
<jml> am using login_with,
<jml> get this error: http://paste.ubuntu.com/649908/
<jml> what am I supposed to do with that?
<jml> why am I unauthorized? how do I become authorized? do my users need to know about access tokens?
 * jml ducking out to get some food. please don't let that stop you answering in the mean time.
<bigjools> jml: I wonder if it's anything to do with the cache bug I filed the other day
<adeuring> henninge: r=me
<henninge> adeuring: thanks again ;-)
<jml> bigjools: which one is that?
<bigjools> jml: mine was to do with cached wadl not updating but I wonder if the auth tokens sometimes get the same problem
<StevenK> I seem to recall something like that.
<StevenK> It usually happens when we blow up the session database.
<jml> bigjools: https://bugs.launchpad.net/launchpadlib/+bug/812799 ?
<_mup_> Bug #812799: Local cache not updated when it should be <launchpadlib :Triaged> < https://launchpad.net/bugs/812799 >
<bigjools> well I've always usually been directed to re-auth
<bigjools> jml: yes
<jml> bigjools: I moved .launchpadlib out of the way and still get the same problem
<jml> which is consistent with my discoveries from strace.
<bigjools> jml: it should offer to re-auth :/
<jml> bigjools: I guess it's using some kind of system keyring
<bigjools> yeah it throws the token in the password storage that your desktop uses
<jml> I can't find it in Passwords & Encryption Keys
<james_w> it's called something odd IIRC
* bac changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugs: 240 - 0:[#######=]:256
<jml> james_w: the code says it's called 'launchpadlib'
<jml> at least, that's what is passed to keyring.get_password
<james_w> hmm, ok
<jml> but that doesn't exist in Passwords & Encryption Keys
<jml> so I'm trying to figure out where the hell it's looking
 * jml busts out set_trace, since google is failing.
<bigjools> woo ajaxy mid-review diffs working
<bigjools> the link should be green though
<james_w> it's called "network password" here
<james_w> jml, ^
<LPCIBot> Project devel build #911: FAILURE in 5 hr 44 min: https://lpci.wedontsleep.org/job/devel/911/
<jml> harumph
<jml> ahh...
<jml> 'launchpadlib' is the domain. or something.
<jml> which doesn't show up in P&EK
<jml> https://bugs.launchpad.net/launchpadlib/+bug/814595
<_mup_> Bug #814595: Unrecoverable error when authorization fails <launchpadlib :New> < https://launchpad.net/bugs/814595 >
<jml> sorry if it's a little grumpy. it's a pretty serious bug & I lost a fair bit of time to it.
<jml> james_w: how did you figure out that it was called "network password"?
<james_w> jml, just by looking at everything that didn't have a clear non-launchpad purpose
<jml> james_w: heh, ok. thanks.
<benji> unfortunately the string "network password" isn't under our control; the Gnome keyring calls that class of entries "network password"
<bac> wow, adeuring, no backlog
<jml> benji: that is a shame.
<adeuring> bac: well, I did only two reviews this morning.
<bac> two more than me today!
<adeuring> seems that people are in holidays
<jml> benji: luckily, the bug can be fixed without that, since the ideal is never to be compelled to look for keys in the keyring
<benji> jml: indeed; in fact, I /thought/ that invalid auth tokens were silently thrown away
<jml> :\
<james_w> http://shnatsel.blogspot.com/2011/07/weve-just-revolutionized-alpha-testing.html <- a Launchpad success story
<jml> nice
<jml> today is obviously a 'file lots of bugs' day
<jml> almost makes me want to file bugs on Launchpad about improving my person-oriented bugs views.
<spiv> jml: in happy news my inline diffs on merge proposals patch landed on production (for ~launchpad)
 * spiv -> bed
<henninge> dpm: ping
<dpm> henninge, pong
<deryck> henninge, ping for standup
<henninge> deryck: argh, already
<jml> can I delete a PPA over the API?
<bigjools> no
<jml> bigjools: why not?
<henninge> deryck: mumble dead, try again
<bigjools> jml: because nobody write the code to do it
<deryck> henninge, ah ok
<bigjools> wrote*
<jml> bigjools: fair enough.
<bigjools> jml: are you grumpy today? :)
<jml> bigjools: not really. just while debugging the login error.
<StevenK> Should be fairly easy to add, just the decorator.
<wgrant> We should perhaps not do it until PPA deletion does something useful.
<wgrant> Which is also very easy.
<bigjools> wgrant: you criticise with no suggestion of what would be useful.  Many people already find it useful.
<StevenK> Right
<wgrant> bigjools: It doesn't free up the name or disappear.
<wgrant> It is useful for the purpose for which it was implemented.
<bigjools> there's a bug about those IIRC
<wgrant> But an API for that purpose is not particularly useful.
<wgrant> I don't think.
<bigjools> make it disappear was rather hard
<wgrant> Yeah.
<wgrant> Rename it is one line of code plus tests :)
<wgrant> And it fixes most issues that I know of.
<StevenK> Clean up job that removes the row from Archive after a day?
<bigjools> one line?
<StevenK> But that then means you need to delete the publications too
<bigjools> StevenK: that would be the naive approach :)
<wgrant> StevenK: With all the BPRs and SPRs that have been copied into other archives?
<StevenK> Tis messy
<jml> why not delete in real-time?
<wgrant> We cannot remove the DB stuff.
<wgrant> And there is stuff on disk to be removed.
<jml> ok, so that just means we use long poll, right?
<wgrant> Hah
<bigjools> we'll work on deploying long poll when Red goes to maintenance
<jml> oh, interesting.
<bigjools> wgrant, StevenK: BTW, I am very concerned about never being able to derive from distros with a) active builds b) items in the queue
<jml> well, it's not like deleting PPAs is going to be attempted before then.
<bigjools> I think relaxing b) is easy
<bigjools> I am not sure of ramifications for a)
<wgrant> It is pretty messy.
<wgrant> This goes back into the copying partial subsets of binaries thing.
<bigjools> indeed
<wgrant> Another dupe of which was filed yesterday.
<StevenK> That we should allow copying binaries only?
<wgrant> That if something built in karmic, you can only copy it from karmic, not anything later.
<jtv> Do we have a way for tests to query the log level that LaunchpadScript sets?  Apparently it gets set on a handler, not the logger, but I'm not seeing the handler.
<bac> hi sinzui
<stub> jtv: You would have to poke the internals. Why would it matter what level the script sets?
<jtv> stub: for some reason LaunchpadScript setup changes log levels.
<stub> You probably have to invoke a script and look for DEBUG, INFO etc.
<jtv> That'd be very costly.  :(
<stub> It adds log levels, and configures the various handlers to what level they emit by default.
<stub> logging.DEBUG isn't changes to a new number or anything like that.
<jtv> We have several cases where scripts instantiate and run other script objects, which seems to change the log level no matter what you do (and never change it back)
<stub> c/changes/changed
<jtv> Well no, that'd be very weird.
<stub> Yer, have no idea what happens if you try to do that.
<jtv> Point is, logger.getEffectiveLevel returns NOTSET.
<stub> LaunchpadScript seems pretty difficult to invoke except as a standalone script.
<bigjools> awesome, ec2 sent me an email saying "Test Runner FAILED"
<stub> logging.getLogger('').getEffectiveLevel() you mean?
<stub> (the root logger?)
<jtv> Would that work?
<jtv> I just tried getEffectiveLevel on the logger that LaunchpadScript created.
<stub> NOTSET just means inherit from somewhere I think
<jtv> Although it does seem to do something global, because when script A instantiates script B, that's enough to change A's logging as well.
<stub> so the logger the script just uses its parents or something like that.
<jtv> So suddenly from the moment B is instantiated, A continues without the log level you specified on the command line.
<stub> There are a number of loggers, and we only want to bother maintaining one, so we just configure that stuff on the root logger IIRC.
<jtv> That would explain.
<stub> Yer. Cause we can't force scripts to use the logger instantiated just for them, or we could, but legacy code would need to be waded through.
<stub> (and we would have to pass it around everywhere and use it whenever some code elsewhere wanted to log a message)
<stub> So yeah, it is global state and LaunchpadScript or somewhere in that mess sets it up so we see the messages per -v, -q etc. with the nice formatting and timestamps.
<stub> Maybe some flag is needed to create a LaunchpadScript without doing that setup?
<jtv> I was thinking maybe we should skip it if test_flags is given.
<stub> That sounds sane
<jtv> Except it makes for damned ugly tests.
<jtv> Because how do you test this without passing test_args?
<stub> I think you have to pass test_flags in, or it will use argv anyway, and make the test runner explode.
<jtv> Exactly.
<stub> yer.
<stub> so new flag.
<jtv> Or maybe one test could just sabotage options parsing.
<jtv> No, no, I don't think it'd help much.  :(
<stub> if you like monkey patches over explicit arguments and the probability of you forgetting to unsabotage it leak your breakage to later tests...
<jtv> Ah yes, the joy of global variables.
<bigjools> wgrant: were you talking about bug 813749 ?
<_mup_> Bug #813749: copying package that have been copied from another series doesn't work if you do not use the "original" package <Launchpad itself:Incomplete> < https://launchpad.net/bugs/813749 >
<stub> Feel free to refactor the entire LaunchpadScript API. It it horrible and difficult to extend.
<jtv> Anyway, I think it's probably reasonable to skip the logger setup if test_args is given.  The script simply own't have -v and -q.
<danilos> whoa, I just learned about https://launchpad.net/launchpad/+documentation ;)
<jtv> *won't
<bigjools> wgrant: and what is the dupe?
<bigjools> can't find it
<wgrant> bigjools: Oh, apparently I saw it and forgot to triage it.
<wgrant> Probably because I couldn't find the dupe.
<bigjools> heh
<stub> jtv: It might break some tests, but I think it unlikely enough to give it a go.
<wgrant> It was like 6am, so meh.
<jtv> stub: right ho, I'll try that.
<wgrant> But you know the general idea of the bug, right?
<bigjools> wgrant: vaguely
<bigjools> 2nd copy copies already copied binaries or something
<wgrant> karmic has lpia, while >= lucid don't.
<wgrant> So if you copy something from karmic to lucid, lucid only gets two of the three binaries.
<wgrant> So you can't copy from, say, lucid to maverick.
<wgrant> Because the copy doesn't have a superset of the archive's binaries.
<bigjools> wgrant: I think we can alter the +initseries job so that it checks the packages you're copying to see if any of them are building
<bigjools> rather than a blanket "something is building, argh"
<wgrant> Even that is probably too strict.
<bigjools> even if copying with binaries?
<wgrant> But making it more lenient than that is more than probably extremely painful and difficult.
<bigjools> wgrant: actually I'm still struggling to understand this issue
<bigjools> it's been a long week and my brain was shredded the last 2 days by derived distros problems
<bigjools> wgrant: I guess the problem is, why do we check the superset
<bigjools> or is that the bug?
<gary_poster> bac, https://code.launchpad.net/~gary/launchpad/bug812335/+merge/68762 when you have a mo
<wgrant> bigjools: See line 399 onwards in packagecopier.py
 * bigjools looks
<wgrant> bigjools: I'm not sure the rationale remains valid, but we need something like that.
<bac> gary_poster: will do it now
<gary_poster> ty
<bigjools> wgrant: I don;t undserstand the rationale
<bigjools> my typing has gone to hell
<wgrant> bigjools: The rationale doesn't explain the superset check.
<bigjools> and what is the problem it's trying to check
<bigjools> right :)
<wgrant> It's simply providing reasoning that all BPRs have unique files.
<bigjools> unique across what set?
<wgrant> All their files have different contents, that is.
<wgrant> Ah.
<wgrant> The superset check is, I suspect, to prevent different sets of builds being copied in together.
 * bigjools tries to find the original bug
<wgrant> eg. if I have an i386 binary for that source already in this archive, and I try to copy a different i386 one in.
<wgrant> Then the source will have two different i386 binaries.
<wgrant> In one archive.
<wgrant> Or something.
<bigjools> bug #387613
<_mup_> Bug #387613: CheckALL - CopyALL approach does not work well for multiple copies <api> <lp-soyuz> <ppa> <Launchpad itself:Fix Released by cprov> < https://launchpad.net/bugs/387613 >
<wgrant> Where'd you get that? Annotate?
<bigjools> "conflicting versions to be copied in the same batch"
<bigjools> annotate, then bzr log
<bigjools> it's classic Portuglish in the bug description
<bac> gary_poster: what is happening at line 134 of the main diff?
<bac> gary_poster: er, nm
<gary_poster> bac, cool
<bac> gary_poster: any reason not to used named values and do it in a single interpolation?
<gary_poster> bac, sqlvalues is what I was thinking
<bac> gary_poster: done
<gary_poster> thank you bac
<jtv> cjwatson: archive index generation should (knock on wood) work normally again on the next run.  It's a big change though, so needs watching.
<jcsackett> bac: could i add https://code.launchpad.net/~jcsackett/launchpad/form-picker-macros-also-irritate-me/+merge/68862 to your queue?
<bac> jcsackett: sure
<mtaylor> why is my launchpadlib script asking this question:
<mtaylor> Please set a password for your new keyring
<bac> benji: ^^
<mtaylor> is there a magic flag I can pass which tells somehting I'm trying to run a script which is not going to be run interactively for most of it's life?
<bac> jcsackett: is the only difference between this MP and the previously approved one the unrolling of the bad merge conflicts?
<jcsackett> bac: yup, it just undoes the rollback and fixes the one bit of bad conflict resolution.
<benji> mtaylor: there is a mechanism for non-interactive authentication; I'll have to remember what it is.  The basics are that you authenticate once interactively and then refer to that authentication token after that
<bac> jcsackett: ok
<mtaylor> benji: ok. I'll look in to it...
<mtaylor> thanks
<bac> jcsackett: did the test suite catch the bad merge issues?
<benji> mtaylor: take a look at the credentials_file argument to login_with
<sinzui> bac: no. That could have been caught by windmill if it ever worked
<sinzui> bac: yui-app tests could be written to verify that the ajax widget was activated for the field.
<sinzui> bac: The popup widget is from 2005. It never had ajax tests
<mtaylor> bac: hey there! you don't have the powers to remove unused users, do you?
<sinzui> mtaylor, we can rename unused users. Admins can merge users for other users
<mtaylor> sinzui: so, https://launchpad.net/~keystone seems to be a ghost person
<mtaylor> and it's in the way of me making a team to manage https://launchpad.net/keystone (well, someone made https://launchpad.net/~keystone-tem - but that doesn't match the rest of our naming scheme, and I'm anal)
<sinzui> No, he is real, but has not done anything substantial
<mtaylor> well poo
<sinzui> He did not choose his user name, which is what I think you care about
<mtaylor> yes. I don't mind that he's a real person - I'm sure he's lovely :)
<sinzui> I can change hi Lp-id since Lp decided it. He has never done anything in lp to related his Lp-Id to his name
<cr3> hi folks, I'd appreciate advice about handling anonymous information in a launchpad'esque way even though I realize that anonymous is mostly frowned upon in launchpad: I'm handling submissions in the launchpad hwdb and some don't have an owner, so if I'm linking the results in a testrun table for example, should the person_id be allowed to be null for anonymous submissions or should I have an anonymous placeholder person entry and its corresponding celebrity
<jcsackett> bac: sorry, didn't catch your earlier comment, but sinzui is right.
<sinzui> mtaylor, ~keystone was created by shipit. He is on the list of people to remove from Lp.
<mtaylor> whee!
<mtaylor> shipit is the thing that created a bunch of user accounts back in the day, yeah?
<sinzui> bunch is an understatement. we have 2.5 million shipit accounts. only a small percent used Lp.
<sinzui> mtaylor, ~keystone is free
<mtaylor> sinzui: thank you!
<jml> is there an API to get a launchpadlib archive object given a string like 'ppa:person/ppa_name'?
<maxb> I think lp.people['person'].getPPAByName(name='ppa_name') is the best you'll get
<maxb> The other syntax is private to "dput" AFAIK
<bigjools> what he said
<jml> ok. so I'll parse it myself
<jml> well, it's not private to dput per se, because apt-add-repository uses it
<jml> thanks maxb & bigjools
<jml> g'night all
<LPCIBot> Project devel build #912: STILL FAILING in 5 hr 59 min: https://lpci.wedontsleep.org/job/devel/912/
<cr3> are products and product serieses accessed controlled, ie is there any product for which I couldn't access /name/+series in the web interface?
#launchpad-dev 2011-07-23
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #913: FIXED in 6 hr 4 min: https://lpci.wedontsleep.org/job/devel/913/
<LPCIBot> Project db-devel build #747: FAILURE in 5 hr 43 min: https://lpci.wedontsleep.org/job/db-devel/747/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #748: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/db-devel/748/
<nigelb> GRAR. https://blueprints.qastaging.launchpad.net/ times out.
<nigelb> Oh well. Almost everything seems to timeout in qastaging.
<nigelb> except for qastaging.launchpad.net :/
<StevenK> nigelb: How many times does it time out?
<nigelb> StevenK: er, so I can keep refreshing and trying my luck?
<StevenK> nigelb: Right
<StevenK> nigelb: Two or three times might warm the DB cache up enough for it to load
<nigelb> StevenK: that fixed it, thanks :)
<nigelb> I forget, what do I change qa-needstesting to?
<nigelb> aha, qa-ok
<StevenK> Right
<StevenK> nigelb: Nice work :-)
<nigelb> StevenK: :)
<nigelb> More to come :D
 * StevenK waits for the qatagger to run, since nigelb's revs QA was blocking it
<nigelb> ouch
<nigelb> should have done this earlier then
<StevenK> Meh, we couldn't deploy when your stuff landed anyway -- it was late Friday, so it's fine.
<nigelb> ah
<StevenK> Now it's wgrant's turn. He has four revisions waiting QA.
<nigelb> hah
<nigelb> Knowing him, he'll probably respond to the ping and do it right away :P
<StevenK> Bwaha
<LPCIBot> Project db-devel build #750: FAILURE in 5 hr 52 min: https://lpci.wedontsleep.org/job/db-devel/750/
<lifeless> jelmer: I have to run
<lifeless> bah
#launchpad-dev 2011-07-24
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 245 - 0:[#######=]:256
<lifeless> bah
<lifeless> what does the apache mangling for lp dev setups
<lifeless> make install
<lifeless> and sheese.
<mwhudson> it's lovely isn't it
<lifeless> does anyone know why it doesn't use *:80 and *:443 by default ?
<mwhudson> no idea beyond "it's always been like that"
<elmo> would require root
<elmo> I'd imagine?
<lifeless> elmo: the setup script does already
<lifeless> elmo: its run as 'sudo make install'
<lifeless> if I get this: OpenID Provider Is Unavailable at This Time
<lifeless> on launchpad.dev
<lifeless> whats a likely cause ?
<lifeless> DiscoveryFailure: Error fetching XRDS document: <urlopen error [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol>
<lifeless> was logged
<mwhudson> sounds like something serving straight http on port 443 maybe?
<lifeless> ok, thats crazy
<lifeless> https://launchpad.dev/~name12/+participation has a link to +newteam
<wgrant> lifeless: Security? :)
<wgrant> lifeless: Don't really want a dev instance being accessed by the world.
<wgrant> Also, there is a separate vhost on 127.0.0.99:443 for private codebrowse.
<lifeless> yes, though that should diaf
<lifeless> (I'm now running lp on my desktop, in lxc, and accessing from my laptop
<lifeless> )
#launchpad-dev 2012-07-16
<wgrant> lifeless: Since we seem to have been fairly stubless lately, could you review https://code.launchpad.net/~wgrant/launchpad/branch-type-policy-db/+merge/114799?
<lifeless> wgrant: sure, though one day isn't all that stubless :)
<wgrant> lifeless: Thanks, replied.
<wgrant> Thanks.
<adeuring> good morning
<StevenK> Hmmm, how can I use MatchesStructure with a dict?
<bigjools> I filed a bug about that
<StevenK> bigjools: Ah, so the answer is "I can't, yet"
<bigjools> StevenK: indeed!
<StevenK> :-(
<bigjools> can't remember what I did in the maas code, I came up with a shortcut of some description
<bigjools> but I need to find time to write a patch for testtools
<StevenK> bigjools: I've put a comment and six self.assertEqual() for now. Makes me sad.
<bigjools> StevenK: don't do that
<bigjools> you can still use a MatchesStructure IIRC
 * bigjools digs
<bigjools> gah grep fails me
<bigjools> StevenK: I think there was a trick with AllMatch
<bigjools> well, you can do AllMatch and a few Equals
<StevenK> bigjools: I have my own workaround
<StevenK> bigjools: http://pastebin.ubuntu.com/1094504/
<bigjools> StevenK: haha, ship it
<cjwatson> StevenK: self.assertContentEqual(expected.items(), observed.items()) isn't dreadful either
<cjwatson> bigjools: So, following the next NDT deployment with that work to show failed PCJs, do you know of any reason why we couldn't (a) switch on soyuz.copypackageppa.enabled for everyone (b) set soyuz.derived_series.max_synchronous_syncs to 1 for launchpad-beta-testers?
<cjwatson> I've tried similar things on dogfood and haven't noticed any problems with async copies there
<dpm> hi LP developers, I'm not sure if it's just me, but tried it both on Firefox and Chromium and the 'Report a bug' link does not seem to be working. I've tried on https://bugs.launchpad.net/developer-portal/ - is this a known bug?
<cjwatson> dpm: bug 1024866
<_mup_> Bug #1024866: "report a bug" link in the involvement portlet on bug listings is broken <bug-columns> <javascript> <markup> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1024866 >
<dpm> thanks cjwatson :)
<cjwatson> dpm: https://launchpad.net/developer-portal works, or use https://bugs.launchpad.net/developer-portal/+filebug directly
<dpm> yeah, I saw that, just mentioning it here in case it wasn't known, thanks cjwatson!
<czajkowski> dpm: if it's been triaged and sinuzi has commented on it, fair chance of it being worked on
<dpm> it is
<rick_h_> dpm: I'm starting to work on it right now. Should have a fix in a bit. Thanks for hte heads up.
<dpm> rick_h_, no worries
<Bert_2> wgrant: Hi, you send me your branch for a launchpad db from cratch, it doesn't seem to come with instruction, so am I just supposed to run make schema on it, or am I supposed to copy it over my rocketfuel pull and then run make schema and make run ?
<czajkowski> rick_h_: excellent have also tweeeted it on both our accounts so people know we're working on things
<rick_h_> czajkowski: saw that, thanks!
<czajkowski> trying to put more of our communications out there
<czajkowski> and remind folks of the downtime
<czajkowski> rick_h_: if you think of anything else that should go out, shout
<rick_h_> jcsackett: ping
<rick_h_> adeuring: ping, do you know if the tal code in registry/templates/pillar-sharing-table.pt means that whole block will be within that upper defined <div> with the ns attributes?
<jcsackett> rick_h_: pong.
<rick_h_> jcsackett: avail for a 5min hangout?
<jcsackett> uhm, sure. let me grab my coffee, and i'll be on g+ in a moment.
<rick_h_> I'm poking at some sharee stuff and need a sanity check
<rick_h_> jcsackett: rgr
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 4.0*10^2
<rick_h_> benji: can you review this as soon as you get a chance so I can start the landing run? https://code.launchpad.net/~rharding/launchpad/1024866_buglink/+merge/115117
<benji> rick_h_: sure
<rick_h_> ty much, this is trying to fix the broken report a bug link
<adeuring> rick_h_: yes, all stuff should go into the main <div>...</div> sorry for the late answer, was out for lunch
<rick_h_> adeuring: awesome thanks. jcsackett thought so as well.
<adeuring> benji: could you also have please a look at this mp: https://code.launchpad.net/~adeuring/launchpad/bug-1015667-3/+merge/115110 ?
<benji> adeuring: sure
<adeuring> thanks!
<benji> rick_h_: https://code.launchpad.net/~rharding/launchpad/1024866_buglink/+merge/115117 looks good
<deryck> Morning, all.
<rick_h_> benji: ty much
<rick_h_> morning deryck
<deryck> adeuring, https://plus.google.com/hangouts/_/1dc7760a0895bb0edfcab736a3781eee1a014dd6?authuser=0&hl=en
<Bert_2> Does anyone know how wgrant's bootstrap-db-from-scratch branch works ?
<mgz> Bert_2: <http://irclogs.ubuntu.com/2011/05/06/%23launchpad.html#t01:13>
<Bert_2> mgz: thx, though do I need to call the merge from a specific directory ?
<Bert_2> (I pulled it and it seems to be for launchpad/lp-branches/devel/)
<mgz> you specifically need to merge it into a current devel branch
<mgz> so, if you have lp:launchpad at ./devel and wgrant's branch at ./bootstrap-db-from-scratch
<cr3> hi folks, can I have a recipe to build packages from two branches or two recipes for one ppa?
<mgz> `bzr merge -d devel bootstrap-db-from-scratch` will update devel with the relevent changes
<Bert_2> mgz: I did the rocketfuel thing, so I guess my lp:launchpad is in launchpad/lp-branches/devel/
<mgz> Bert_2: right
<Bert_2> so do I run the merge from inside lp-branches or from devel ?
<mgz> and lp-branches is a shared repo
<mgz> you need to merge into a branch, devel is a branch.
<Bert_2> ok, I think I understand, but I'm still not sure what I'm exactly supposed to run now,  bzr merge lp:~wgrant/launchpad/boostrap-db-from-scratch ?
<mgz> that would do, if you're in launchpad/lp-branches/devel
<Bert_2> so if I do this "ubuntu@lpdev:~/launchpad/lp-branches/devel$ bzr merge lp:~wgrant/launchpad/boostrap-db-from-scratch" I'm fine ?
<mgz> try it and see, what's the worst that could happen :)
<Bert_2> that I'd have to start all over again ? :P
<Bert_2> but I have a copy of the LXC and of the launchpad dir
<Bert_2> so that's not going to happen ;)
<Bert_2> fancy: bzr: ERROR: Invalid url supplied to transport: "bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/boostrap-db-from-scratch": no supported schemes
<Bert_2> that's odd, cause I pulled that branch yesterday and that did work :S
<mgz> do `bzr launchpad-login`
<Bert_2> I don't have an id configured ;)
<Bert_2> I didn't know that was required to pull/merge stuff locally ?
<jelmer> Bert_2: it looks like there might be a typo in that URL?
<jelmer> (boostrap vs bootstrap?)
<Bert_2> ohyeah, that's a typo XD
<Bert_2> my bad
<mgz> jelmer: oddly, it really is a branch though
<Bert_2> yeah, now it seems to be getting something
<mgz> or... info something for non-existant branches now?
<Bert_2> changes applied succesfully, awesome
<Bert_2> thx :D
<mgz> ah, climbs one level
<mgz> that's not really helpful.
<james_w> mgz, is jam off?
<jelmer> james_w: yes
<cr3> is it possible to make a branch public?
<cr3> ... that was previously private, of course :)
<mgz> should have kept jam in the fridge
<jelmer> cr3: yes, ping a webops in #launchpad-ops
<Bert_2> I have another silly question, LPCONFIG and LISTEN_ADDRESS, those are added to make install, right ? and I run that after make schema en before make run, right ?
<james_w> jelmer, thanks
<cr3> jelmer: thanks, I wasn't sure whether I was looking at the wrong part of the interface :)
<mgz> Bert_2: the Makefile has defaults for both of those
<mgz> set them in your environment if you want something different from development/127.0.0.88
<Bert_2> mgz: yeah, but I have my own config and I want things to be remotely accessible, so I need to change those, right ?
<Bert_2> oh, so I don't do "sudo make LIST_ADDRESS=* install" but I export those vars ?
<mgz> for LISTEN_ADDRESS I think only install cares so passing into make would work too
<Bert_2> but for the config I can better do export LPCONFIG=ulaunch ?
<Bert_2> (ulaunch is the name of the configdir)
<mgz> right.
<Bert_2> it's probably a good idea to add it to .bashrc as well, for future use, or am I wrong ?
<mgz> seems reasonable if you're keeping the profile around
<Bert_2> yeah, I am ;)
<Bert_2> thx a ton for your help mgz, I need to make dinner now first, then I'll go ahead and give it a good test, then try and change the name from launchpad to ulaunch
<mgz> no problem, enjoy dinner :)
<rick_h_> jcsackett: r=me with a few style comments thanks for adding tests!
<jcsackett> rick_h_: thanks. :-)
<rick_h_> jcsackett: sorry, but after going through all the tests in the 3.5 stuff I'm crazy to get tests with  better notes in them to help out :)
<cr3> jelmer: there's nobody in #launchpad-ops :(
<jcsackett> rick_h_: all good comments, i'm making those changes now.
<mgz> cr3: on the canonical irc server
<cr3> mgz: thanks!
<rick_h_> J/topic
<rick_h_> bah
<rick_h_> benji: another for your eyeballs when you get a sec please https://code.launchpad.net/~rharding/launchpad/yuiff/+merge/115170 no rush on this one
<benji> rick_h_: sure
<benji> frankban: I think it would be an improvment to use a fake .bazaar, and yeah, it seems BackupFile wouldn't be needed then (and we can always resurrect it if we need it later)
<benji> heh, wrong chan
<czajkowski> sinzui: does https://bugs.launchpad.net/ubuntu/+source/apport/+bug/764414  come under your privacy bugys?
<_mup_> Bug #764414: private master bugs are confusing and lead to more duplicate filings <pet-bug> <apport (Ubuntu):Triaged by pitti> < https://launchpad.net/bugs/764414 >
<sinzui> no. It is a bug-links problem.
<sinzui> czajkowski, when you start work, you can promote http://blog.launchpad.net/general/bug-reporting-and-search-knows-about-privacy , but you we may have a more important announcement to make.
<czajkowski> sinzui: let me do it now
<czajkowski> sinzui: done in G+/FB/ Twitter and identi.ca
<sinzui> your quick
<czajkowski> they're all open always except identi.ca
<czajkowski> people don't reply to us there they do via twitter
<sinzui> czajkowski, I think we still want to rethink the duplicate portlet. I think it leaks. We can make the UI friendly to users and in the same process, faster. Apport doesn't use the UI though.
<sinzui> I think fixing this bug will prevent people from being rude: https://bugs.launchpad.net/launchpad/+bug/943497
<_mup_> Bug #943497: warn when you are going to mark a public bug as a duplicate of a private one <bug-links> <bugs> <disclosure> <privacy-transitions> <Launchpad itself:Triaged> < https://launchpad.net/bugs/943497 >
<Bert_2> Hi, I just ran make schema on rocketfuel + wgrant's bootstrap-db-from-scratch and I'm getting the following error: http://pastebin.com/urfMfUdu what's wrong ?
<czajkowski> sinzui: nods
<czajkowski> sinzui: cheers
<czajkowski> sinzui: will leave a comment saying once that bug is fixed it will make the other issue clearer for people
<sinzui> jcsackett, how goes the landing of allowing users to hide their own comments?
<jcsackett> sinzui: free to chat?
<sinzui> yes.
<jcsackett> hangout made.
<sinzui> jcsackett, google+ on phone now lets me start a hangout
<jcsackett> sinzui: cool. had you already started one?
<jcsackett> i can close this and jump to you.
<sinzui> I was trying
<jcsackett> ah, have your invite
<jcsackett> ...which you are now out of.
<jcsackett> :-P
 * sinzui no see you in your hangout
<jcsackett> sinzui: it closed when i clicked your invite.
<sinzui> ha ha
<sinzui> me tries to start one again
<sinzui> I hear ding-a-ling sounds
<jcsackett> ok. g+ just crashed entirely for me, so i'll be there in a moment.
<sinzui> jcsackett, https://bugs.launchpad.net/launchpad/+bug/1025414
<_mup_> Bug #1025414: Bug/branch privacy badge alt text is wrong/lies <disclosure> <information-type> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1025414 >
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
<rick_h_> benji: the 3.4 is gone because I'm not aware we ever used it but to make sure things were wired up at the sprint in budapest
<Bert_2> Hi, I'm working on a production instance, zcml/override-includes/mail-configure.zcml is giving me some problems. I already switched mail:qyeyedDelivery from stub to sendmail. But I guess I also need a mail: entry for sendmail, now it's not really clear what that should look like, any suggestions ?
<wallyworld_> sinzui: want to start a fraction early to discuss my mp without boring the others?
<rick_h_> wallyworld_: ping
<wallyworld_> rick_h_: g'day, just finishing our standup, give me a minute or two
<rick_h_> wallyworld_: StevenK looks like this merged, if you see it hit buildbot would love to get a deploy overnight to help fix the broken bug report link https://code.launchpad.net/~rharding/launchpad/1024866_buglink/+merge/115117
<rick_h_> wallyworld_: k, np
<wallyworld_> rick_h_: will get a NDT done for that one if it lands today
<rick_h_> wallyworld_: ty much for helping rush it along and keeping an eye on it
<wallyworld_> rick_h_: np
<Bert_2> Hi, is there a way to find out why even though I have a custom local-launchpad-apache file in my configs and LPCONFIG is set correctly the apache config is still full of launchpad.dev ?
<Bert_2> wgrant: ping ?
<lifeless> Bert_2: wgrant is on leave
<lifeless> Bert_2: as for apache config, I believe our scripts only know how to setup launchpad.dev.
<lifeless> since they are developer scripts.
<Bert_2> so even though I made another config, make run won't actually do it ?
<Bert_2> lifeless: or what do you mean with developer scripts ?
<lifeless> make run doesn't alter apache config
<Bert_2> make schema creates the apache file then ?
<lifeless> nope
<Bert_2> then how does it start to exist ? :S
<lifeless> make schema creates a development database in postgresql for development
<lifeless> rocketfuel-setup
<Bert_2> so, what is the local-launchpad-apache for then ?
<Bert_2> (in configs)
<lifeless> its the template used by rocketfuel-setup
<Bert_2> so I was supposed to create launchpad/lp-branches/devel/configs/[LPCONFIG]/local-launchpad-apache before having rocketfuel pull the branches ?
<lifeless> no?
<lifeless> what are you trying to accomplish ?
<Bert_2> running launchpad from somewhere else than launchpad.dev
<Bert_2> I already made all the configs etc.
<Bert_2> so if I manually edit the apache virtualhost I guess I'm there
<Bert_2> lifeless: ok, got it working, but I get a very big oops page: https://ulaunch.ulyssis.org/
#launchpad-dev 2012-07-17
<lifeless> Bert_2: you may not be aware, but the theme for Launchpad isn't open source - you need to change the theme.
<lifeless> Bert_2: for that oops, its because you're missing the ubuntu celebrity
<Bert_2> I do know that
<Bert_2> I know about the theme I meant
<Bert_2> lifeless: and I don't know what the ubuntu celebrity is, that's the problem :S
<lifeless> celebrities are well known objects in the system - sysadmin teams, ops teams, various key projects (all from the perspective of launchpad.net of course)
<Bert_2> I used wgrant's bootstrap-db-from-scratch
<Bert_2> I guess that means I don't have any users
<Bert_2> (that would also explain why I can't login)
<lifeless> indeed
<Bert_2> lifeless: would you happen to know where this information is stored ?
<lifeless> there is a script in utilities I think to add a local user
<lifeless> as for the celebrities, you probably need to add them directly via sql, though the harness *may* work.
<wgrant> Bert_2: Did you run the bootstrap script after you ran make schema?
<Bert_2> no, I didn't know I was supposed to, wgrant
<Bert_2> is it the bootstrap-lp-db script in utilities I just noticed ?
<wgrant> Yeah
<wgrant> It's bitrotten a little. I'll fix it in a sec.
<Bert_2> so I shouldn't run it yet ?
<Bert_2> in the mean time I'll go and brush my teeth then ;)
<wgrant> If you run it now it'll crash pretty quickly :)
<Bert_2> k, I'll go and brush then :P
<Bert_2> clean teeth, feels awesome :D
<Bert_2> wgrant: do you have any idea whether you'll be able to clean that file out soon, cause otherwise I'm off to bed, I need to get up again in 5,5hours XD
<wgrant> Bert_2: One down, one to go
<Bert_2> k, awesome :D
<Bert_2> wgrant: when you're done I just merge and run that script ?
<wgrant> Bert_2: Right, you'll need to remerge my branch, and 'utilities/bootstrap-lp-db SOME_INITIAL_USERNAME'
<Bert_2> bzr merge lp:~wgrant/launchpad/bootstrap-db-from-scratch, right ?
<wgrant> Yep
<Bert_2> bzr: ERROR: Working tree "/home/ubuntu/launchpad/lp-branches/devel/" has uncommitted changes (See bzr status).
<wgrant> Ah, you've already merged it once, but not committed? If you've made no other changes, 'bzr revert' before remerging.
<Bert_2> I added my own config and override-includes/mail-configure.zcml
<Bert_2> but I can just copy them back afterwards, right ?
<wgrant> Yep
<wgrant> I've just pushed the latest changes to that branch, which fix bootstrap-lp-db. populate-ubuntu-from-scratch.py (which you probably don't need if you don't care about PPAs) is still broken, however.
<Bert_2> yeah, I don't need PPAs
<Bert_2> we just need bugs, questions and blueprints
<Bert_2> that's why we had to run our own instance, launchpad.net requires us to put code up there
<Bert_2> and we prefer to use our own storage for that
<wgrant> Well, we don't care where the code is, as long as it's open source or you have a commercial subscription.
<Bert_2> well, it's based on drupal, so it's open source by default :p
<wgrant> But anyway, 'utilities/bootstrap-lp-db SOME_USERNAME', then you can log in as SOME_USERNAME@example.com
<Bert_2> k
<Bert_2> example.com can be changed later ?
<wgrant> Oh, then why not just use a launchpad.net project without using Launchpad's code hosting?
<wgrant> Yes.
<Bert_2> I read it's a requirement to have code or a link to code up there ?
<wgrant> We recommend you register the location so it can be automatically imported so other users can see it, but it's by no means required.
<wgrant> If some docs say that, we should fix them :)
<Bert_2> yeah, it was somewhere on launchpad.net
<Bert_2> that's why I got into this misery :p
<wgrant> We have quite a few projects that use external code hosting but LP for everything else.
<wgrant> Some projects just use LP for bugs, or just for translations, etc.
<Bert_2> yeah, that's what we want
<Bert_2> but they have links to their code up, right ?
<wgrant> Sometimes. As long as we can see that the code is under an open source license somewhere, we really don't care.
<Bert_2> oh okey
<Bert_2> well, that's cause I now got launchpad running XD
<wgrant> But it's preferable if it's registered through https://help.launchpad.net/Code/Imports, if possible
<wgrant> Heh
<Bert_2> thx for both the info and the code
<Bert_2> I'll talk to my team about it
<Bert_2> but now I'm off to bed
<Bert_2> cause it's london tomorrow :D
<wgrant> Great, night :)
<Bert_2> thx again
<Bert_2> goodbye ;)
<wallyworld_> sinzui: if you are still around, i like your suggestions to use 'Private' and 'Confidential' for PROPRIETARY and USERDATA respectively so am happy to make those changes
<StevenK> The other way around, I'd say
<wgrant> sinzui: branch_sharing_policy with ~registry-only UI is ready to be deployed to prod, and we should probably dogfood it on our private projects as soon as it is deployed and +sharing is turned on.
<wallyworld_> StevenK: yes, you may be right there, although i was equating Private info type with Private project
<sinzui> wgrant, thank you. I agree
<sinzui> wallyworld_, I just replied to your text changes
 * sinzui looks at query
 * wallyworld_ looks at reply
 * wallyworld_ can finally do some LOC reduction \\o/
<sinzui> oh, do you want a list of methods I saw last week?
<wallyworld_> sinzui: ok, why not
<sinzui> wallyworld_, your query is very fast in staging. I think it might be doable in on run, but four is safe
<wgrant> We're probably also nearly at the stage where we can go through and rationalise all our new methods.
<wallyworld_> sinzui: yes, i'd rather err on the side of caution
<wgrant> sinzui, wallyworld_: Note that production could be twice as slow as staging.
<wgrant> Because production is still using slony
<wallyworld_> wgrant: so if it times out, we can just reduce the batch size. is that the SOP for this?
<wgrant> Precisely.
<wallyworld_> cool. so let's hope 10000 is ok
<wgrant> Just be aware that staging write timings are not particularly comparable to production atm.
<sinzui> wallyworld_,  this is my revised query. I added the statement timeout for the select. We have no fear of hitting it.
<sinzui> https://pastebin.canonical.com/70193/
 * wallyworld_ looks
<wallyworld_> sinzui: ok. so if i you your pastebin, i can mark that approved and get it actioned?
<sinzui> wallyworld_, since we are adding policies in production now, I think we can run this now, starting on staging
<wallyworld_> sinzui: yes agreed. so next step for that bug can be to remove and AAG for which there is a APG for the project?
<sinzui> yes, it is approved. have you added the request to the sql query page?
<wallyworld_> s/and/any
<wallyworld_> yes
<wallyworld_> pending approval
<sinzui> wallyworld_, I do not think these classes in lp.app are used anymore: GotchiTiedWithHeadingWidget, LinkWidget, ContextWidget
<wallyworld_> sinzui: cool. will spin up a branch. do you agree with the next step i should take for bug 1008541 ^^^^^
<_mup_> Bug #1008541: Sharing policies unconfigure existing projects <bugs> <disclosure> <sharing> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/1008541 >
<lifeless> well
<lifeless> production could be twice as slow as staging if staging had the same hardware.
<lifeless> production may need to do twice as many writes, != twice as slow as staging :)
<sinzui> wallyworld_, your sql might fix this bug. The ones for security contacts and bug supervisors have structural subscription nuances that I wanted to solve separately.
<wallyworld_> sinzui: how? my sql only adds APGs for maintainers
<lifeless> sinzui: btw I added a bug driver to obsolete-junk
<lifeless> sinzui: and nuked the resulting struct sub
<lifeless> sinzui: for less bug spam, I hope.
<wgrant> Speaking of obsolete-junk
<wgrant> sinzui: Did you nuke null over the weekend?
<wgrant> I saw you remove bugs from it
<wgrant> And now I can't find it
<sinzui> 140 projects have bug supervisors that are different from the maintainers and drivers, and they expect to get notified. They wont be when we remove the code that hard codes structural subscriptions...thus a separate problem from what I asked you to fix
<wallyworld_> yes
<sinzui> wgrant, I did. mabey i renamed is /null-and-void
<sinzui> after I deleted more bugs from it
<wgrant> Ah, great.
<sinzui> wallyworld_, and the security contact role will be deleted, so we need to setup structural subscriptions for the few projects that use the role, such as ourselved
<wallyworld_> yes
<wallyworld_> so how does my sql to add APG for maintainers help there?
<sinzui> wgrant, I was reading code to follow up on your idea of moving all the pots to one project, then doing real series deletes
<wgrant> sinzui: How does it look?
<wgrant> I know Translations not well.
<sinzui> wgrant, I think it will work. You can obsolete templates too, so they do not appear to accumulate
<sinzui> wallyworld_, when I reported the bug, I did not know how few projects used the security and bug roles. Many place the maintainer team in the roles, btu they do not need to underder sharing...you maintainer sql fixes all but 140 cases
<wallyworld_> right, understood. thanks
<sinzui> wallyworld_, so maybe we want to write the query that finds these 140 exceptions, but we also need to make sure that those exceptions are not open teams
<wallyworld_> sinzui: can do. and then we can give APG for embargoes security to those 140 bug supervisors
 * sinzui looks for query that identified the bug supervisors
<lifeless> sinzui: select count(distinct bugsupervisor) from project ?
<sinzui> lifeless, we also want to exclude maintainer. We also saw that if we excluded drivers there was about 140.
<huwshimi> Did someone break our js in trunk? Maybe combo related...
<StevenK> huwshimi: Hm?
<huwshimi> StevenK: Cannot read property 'app' of undefined, same for registry etc.
<StevenK> Can I have a bit more context?
<huwshimi> StevenK: Load any page, check js errors, that's it.
<huwshimi> (On a fresh trunk)
<huwshimi> Oh, LP_MODULES is not defined
<StevenK> LP_MODULES comes from the meta file, so it looks like it now wants the combo loader set up
<StevenK> huwshimi: Real quick: sudo mkdir -p /srv/launchpad.dev ; sudo make copy-apache-config ; make clean && make
<huwshimi> StevenK: Perfect, thank you!
<huwshimi> StevenK: Oh, except that it appears to break every time I 'make jsbuild'
<huwshimi> brb
<StevenK> huwshimi: The JS breaks or 'make jsbuild' gives an error?
<huwshimi> StevenK: jsbuild runs fine, the js just breaks again
<huwshimi> (LP_MODULES is not defined etc.)
 * huwshimi ducks out for a minute
<StevenK> 'make jsbuild combobuild'
<StevenK> jsbuild blows away the combo loader directory, so no meta file again
<wgrant> huwshimi: 'make jsbuild_watch' may be of interest
<wallyworld_> StevenK: i was thinking - what about 'Private User' for the USERDATA title. That fits in with 'Private Security' and adds the extra qualifier to 'Private' to disambiguate the context
<wallyworld_> And then the portlet can this "This report contains Private User information"
<StevenK> Hmmmmm, maybe
<wallyworld_> i'll hold off landing my branch and we can discuss tomorrow on the call
<StevenK> wallyworld_: +1
<wallyworld_> i just think 'Private' is too ambiguous and overloaded
<StevenK> wallyworld_: The bikeshed should be red. Oh, and on fire.
<wallyworld_> and it is private user data after all
<wallyworld_> and the bikeshed should be purple, not red
<adeuring> good morning
<czajkowski> adeuring: morning
<adeuring> hi czajkowski!
<czajkowski> in the office today, all the orange walls have drawings on them, looks rather cool
<mgz> jelmer: python2.7 backport not installable because:
<mgz> python2.7: Depends: python2.7-minimal (= 2.7.3-0ubuntu4~lts0) but it is not going to be installed
<maxb> mgz: Sounds like apt being typically unhelpful and reporting an intermediate dependency issue instead of the actual problem. Try 'apt-get install python2.7-minimal python2.7'
<mgz> maxb: will ssh in and try that
<mgz> python2.7-minimal: Depends: python-minimal (>= 2.6.6-3+squeeze1) but 2.6.5-0ubuntu1 is to be installed
<mgz> the fun bit is there's no python2.7-minimal in the ~pythoneers/+archive/lts ppa
<mgz> but it worked before the update aimed at fixing the ctypes import crash
<jelmer> mgz: that lists just the source packages
<jelmer> mgz: see https://code.launchpad.net/~pythoneers/+archive/lts/+packages
<mgz> that is the page I'm on
<jelmer> mgz: expand the python2.7 thingy
<mgz> ...I need to enable JS for the page to actually work?
<jelmer> mgz: yes, you'll need javascript :)
<mgz> >_<
 * jelmer drags mgz kicking and screaming out of the nineties
<bigjools> how did you get a PPA url with *code*.l.n in it?
<jelmer> bigjools: I type them by hand :)
<bigjools> lordy
<mgz> bigjools: it's either that or spend ten minutes trying to find how to get to a +recipies page from ~person by navigating links
<bigjools> can you guess what I am going to say next? :)
<czajkowski> bigjools: good night :)
<jelmer> bigjools: patches welcome?
<bigjools> jelmer: *bingo* :)
<jelmer> bigjools: I hate FOSS
<bigjools> cztab: almost!
<bigjools> lol
<czajkowski> bigjools: cheeky
<czajkowski> how are the little ones doing?
 * czajkowski hugs jelmer at least you didn't say floss! 
<bigjools> czajkowski: champion thanks!
<bigjools> they are asleep right now, so even more betterer
<czajkowski> win.
<bigjools> so much so I am heading out of the office and into the living room to watch TV and drink some whisky
<czajkowski> mate had twins 7 years ago, they are very cute to watch. they had them seperated when they were born into different cots and refused to sleep, their mum moved them into one cot against nurses wishes and they slept
<jelmer> bigjools: enjoy your evening :)
<bigjools> czajkowski: ours are in the same cot
<bigjools> jelmer: cheers, enjoy your day!
<czajkowski> bigjools: they stil at times wake up and one climbs into the others bed to sleep it's very cute.
<bigjools> ha!
<wgrant> mgz: There's a "View source package recipes" link near the PPA listing on Person:+index
<bigjools> right, night all
<jelmer> wgrant: only if they actually own any recipes
<mgz> right, the fun part was the person had a linked ppa, but the team owned the recipes
<jelmer> wgrant: which makes sense to some extent, but is confusing if you want to figure out if they own any recipes
<mgz> and from the ppa pages there seemed to be no way to get to the recipes that built them
<wgrant> mgz: The recipe is linked from the expandy section of the package
<wgrant> Since a recipe does not build a PPA, but a package.
<wgrant> Also, the expandy bits on +packages fall back to normal links for those who won't enter this century.
<jelmer> wgrant: that requires having javascript enabled though
<jelmer> wgrant: not everybody has gone with the times ;)
<wgrant> I ran NoScript for years, but even I gave in in early 200
<wgrant> 2009
<mgz> jelmer: so, I don't see why Depends: python-minimal (>= 2.6.6-3+squeeze1) is needed in the context of that ppa
<mgz> okay, so updating python2.6 in that ppa is the sensible answer.
<jelmer> mgz: yes :)
<stub> Looks like a yui tweak has broken buildbot, strangely the db branch
<rick_h_> jcsackett: ping, the linkify navigation code has hit qa staging if you get a chance to peek at the sharee side of it
<wallyworld_> rick_h_: hi
<wallyworld_> rick_h_: i just got back from soccer training and was looking to qa that bug you asked me to look at, but js on qastaging is broken
<wallyworld_> rick_h_: gotta run out and pick up my son, back in a bit
<rick_h_> wallyworld_: np, looking and starting now
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: rick_h | Firefighting: - | Critical bugs: 4.0*10^2
<wallyworld_> rick_h_: fyi, combo loader was broken on staging today, i think some scripts were looked at, not sure exactly what was done, may be related
<rick_h_> wallyworld_: yea, I'm in -ops looking into it thanks
<wallyworld_> cool, thanks
<rick_h_> wallyworld_: I was landing the branch to wire up the multi-yui feature flag at the same time so boom
<wallyworld_> good luck :-)
<mgz> anyone got ideas on what will have broken +branch redirects for bzr branches recently?
<mgz> nothing looks obvious from the log from r15627
<mgz> 1.067  hpss call:   'BzrDir.open_2.1', '+branch/qbzr/'
<mgz> 1.068               (to bzr+ssh://bazaar.launchpad.net/+branch/qbzr/)
<mgz> 5.076     result:   ('no',)
<mgz> that used to correctly get you to ~qbzr-dev/qbzr/trunk2a
<mgz> any tips for debugging what's going wrong?
<deryck> abentley, adeuring, rick_h_ -- firing up a stand-up hangout now...
<czajkowski> deryck: any of you guys blog and if so want your feeds added to planet.lp
<deryck> czajkowski, mine is already on it.
<czajkowski> yup
<deryck> adeuring, https://plus.google.com/hangouts/_/bb3726a1e785f252b50151030e19c86f5ad6b04f?authuser=0&hl=en
<jelmer> one of these days I'm going to accidentally join a standup
<mgz> you'd be welcome I'm sure
 * jelmer has a tendency to click on any link people post on IRC
<mgz> I suspect you might be saved by the google talk plugin crashing or something
<czajkowski> jelmer: likewise I almost tempted one day to join wave and say good bye!
<czajkowski> in a meeting for the next hour folks so wont be watching #launchpad.
<abentley> mgz: Could be bug #1025368
<_mup_> Bug #1025368: Launchpad not handling +branch redirects for http anymore <codebrowse> <codehosting> <regression> <Launchpad itself:Triaged by abentley> < https://launchpad.net/bugs/1025368 >
<mgz> abentley: thanks, having a look
<mgz> ah, the tarball thing, yup, could be the same issue
<abentley> mgz: Yes, I'm pretty sure it applies to all loggerhead stuff.
<abentley> mgz: I'm working on it.  Kinda confusing, but it looks like it might be a loggerhead bug.
<mgz> well, this isn't loggerhead, it's just trying to access the branch at all
<mgz> but the redirect/handypath is part of the loggerhead infrastructure?
<abentley> mgz: Oh, I misread you because the SSH stuff isn't an actual redirect.
<abentley> mgz: I don't know of any ssh issues.  In fact, I thought I'd tested SSH pretty throroughly.
<mgz> yeah, bad term, it's like an apache internal redirect
<mgz> is there a way I can get more details from inside launchpad than just "no" to is a branch here?
<abentley> mgz: e.g. bzr info bzr+ssh://bazaar.launchpad.net/+branch/bzr works just fine.
<mgz> right
<mgz> most branches are still fine, which is why I suspected privacy/stacking at first
<abentley> mgz: What? Is this SSH or not?
<mgz> but iwata reported lp:qbzr and lp:tortoisebzr are both borked
<abentley> If it's SSH, it isn't an apache thing.
<mgz> abentley: sorry, confusing things, I just don't know what we call the +branch aliases
<abentley> mgz: We call them aliases.
<mgz> okay. :)
<abentley> mgz: And the +branch-id ones are called "id aliases"
<mgz> so, some of them are borked, but I don't know why and I don't know how to find out
<abentley> mgz: Okay, I'd call this a new bug.  Might be related to 1025368, might not.
<mgz> will file.
<abentley> mgz: I don't know of a way to get more info.
<abentley> mgz: I'll check whether getByUrl works.
<abentley> mgz: getByUrl returns nothing, so it looks like a it's a problem in their common code.
<mgz> okay, so also 15609 related?
<abentley> mgz: most likely.
<abentley> mgz: The lookup function is BranchLookup.getByHostingPath
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/fix-packagediff-private-harder/+merge/115352
<cjwatson> (or any OCR come to that)
<mgz> he's meant to be on holiday!
<cjwatson> So am I :-P
<cjwatson> But he threw that critical bug in my direction this morning, seeing as I caused it ...
<mgz> :D
<rick_h_> cjwatson: will look now to try to get that fixed
<abentley> mgz: btw, I doubt stacking is related, because ~qbzr-dev/qbzr/trunk2a is not stacked on anything.
<mgz> right, seems like it's just the alias resolution.
<mgz> but I'm still not sure what's special about qbzr/tortoisebzr
<rick_h_> cjwatson: r=me
<cjwatson> Thanks, will land now
<abentley> mgz: bzr and qbzr are both public, so I don't see how privacy would affect one but not both.
<mgz> so, one thing that strikes me about qbzr and tortoisebzr is they both moved their trunk in the past
<mgz> ...but other projects that relocated aren't affected
<abentley> mgz: Didn't bzr also relocate its trunk?
<mgz> abentley: not sure, but lp:leo-editor did and that resolves.
<mgz> (if bzr did it, was before qbzr did the move for 2a, and leo-editor was since then)
<abentley> mgz: I may be wrong about bzr relocating.
<rick_h_> abentley: as a job expert, just want to sanity check the assertion in this small MP https://code.launchpad.net/~cjwatson/launchpad/simplify-packagecopyjob-tests/+merge/115159
<abentley> rick_h_: looking...
<abentley> rick_h_: All looks sane to me.
<rick_h_> abentley: ok, just wanted to sanity check the idea that it's all the same and makes sense to run the jobrunner directly
<rick_h_> vs something that could go bad in tests later
<rick_h_> abentley: ty much
<abentley> I do think it makes sense to run the jobrunner directly.
<abentley> I suppose you could argue it changes the tests slightly, in that it makes assertions about the end-state instead of the exception being raised.  I think the end-state is the most important thing, though.
<rick_h_> abentley: yea, was more worried of unexpeced side effects, possible cleanup/test pollution or timing side effects
<abentley> rick_h_: I don't see any potential for that.
<cjwatson> rick_h_,abentley: The other thing I probably should have pointed out in the MP is that all the other similar tests I could find were already using the jobrunner directly.
<cjwatson> I wondered if PCJ was an early adopter or something.
<rick_h_> ah thanks cjwatson
<sinzui> czajkowski, jcsackett: I am about to fall off the net. My computer reports power problems and I need to power off to investigate.
<jcsackett> and away he goes ...
<sinzui> rick_h_, I just occurred to me that 3 years ago YUI domready had issues so scripts were trying to mutate the dom during load, which is dangerous, so the spinners were hard-coded into the template
<rick_h_> sinzui: where is this? /me is missing the context
<sinzui> rick_h_, I was thinking out loud as I was reading code. Why is the spinner hard-coded in the template? why is the script using on load, then I remembered the age of the code and concluded that spinners are hard coded because progressive enhancement can fail if you mutate the dom as it is loaded
<rick_h_> sinzui: ah, that's an old IE issue true
<sinzui> We were using firefox 3, no webkit browser either.
<rick_h_> right, yea understandable for sure
<rick_h_> I'm trying to track down the latest IE with the issue, I recall it
<sinzui> I think I have an opportunity to clean up this code is my branch has nothing else for me to do
<sinzui> s/is/if/
<rick_h_> the other thing to realize though with spinners like that is that the combo loader call is async, so there's a delay in kicking in if you need to avoid flashing the user
 * rick_h_ isn't sure where you're looking at/etc
<rick_h_> http://blog.gidley.co.uk/2009/11/ie-7-and-domloaded.html
<sinzui> rick_h_, yes, I think that is a legitimate case to add the spinner right away. The example I see is a user initiated action
<rick_h_> ah, here is the thing I was thinking "Internet Explorer note: It isn't always safe to modify content during the available/contentready until after the domready moment. In this browser, domready will execute before the available/contentready listeners.
<rick_h_> "
<sinzui> YUI 3 ensures that available and contentready fire after domready
<rick_h_> right
<rick_h_> this was from 3.2 notes http://yuiblog.com/sandbox/yui/3.2.0pr1/examples/event/event-timing.html
<sinzui> There was some definite madness in the past. Since it was not documented in the code, I can only guess at why I keep find on load and hidden spinners in the templates
<rick_h_> well the biggest thing these days is probably the copy/paste syndrome
<sinzui> rick_h_, while i have your attention. I wanted overlays to always return focus to their EDITION when they closed so keyboard navigation does not bounce back to the top of the page.
<rick_h_> ah interesting
<sinzui> It didn't work, well, this.get('editicon').focus(); didn't work, or what every attr was the supposed trigger for the overlay
 * rick_h_ looks
<sinzui> I had some unsuccessful tests with activeElement a few weeks ago. I was thinking that when the overlay is displayed, it captures what had focus, then calls focus in destroy or hide
<rick_h_> right, was going to ask when you were trying to focus back
<rick_h_> I think it'd be a something interesting to do as when visible change or destroy as you say
<sinzui> I was playing with IE too, so I may have been dealing with some propagation differences
<rick_h_> determining what had focus is hard. I wonder if there's a way to some sort of master 'focus tracker' .delegate() event on BODY to keep tabs on focus and query it
<sinzui> We know our overlays use the editicon (or the linked value) so that might be the right answer all the time
<rick_h_> yea, I wasn't sure if they could be determined by id/explicitly
<sinzui> I agree. this certainly would be problematic if we ever convert help to an overlay
<rick_h_> I thought it was?
<rick_h_> sinzui: http://jsfiddle.net/mmcTL/11/ is a bad example because you'd want to filter the delegate to only things that are on the main page, not overlays/added content
<rick_h_> but it looks like you could delegate onto the body, watch for focus events, track them, and then that tracker could listen for some sort of restore event to refocus manually back where it was left off
 * sinzui bookmarks
<sinzui> thank you!
<rick_h_> so on the destroy/hide events of widgets, you'd fire a Y.fire('returnFocus') or something
<rick_h_> sinzui: np, interesting idea
<deryck> Ah, the joys of stable internets again.
<deryck> abentley, adeuring, rick_h_ -- hi, there. I'm connected again. :)
<abentley> deryck: yay!
<rick_h_> deryck: welcome back to the land of the wired
<deryck> I'm so kicking AT&T to the virtual curb.  Soon as I find another option.
<czajkowski> sinzui: comment on your blog post http://blog.launchpad.net/general/launchpad-does-not-have-private-projects-yet
<rick_h_> hah, greg causing trouble
<czajkowski> you have no idea the amount of trouble that line cause
<czajkowski> spend all weekend discussing it in various channels
<rick_h_> czajkowski: greg used to be local, started the MI loco, so friend of mine. Not surprised he latched onto that
<czajkowski> we're on the loco council together :)
<rick_h_> ah right cool
<czajkowski> sinzui: to be fair we're moving out stuff outta kanbahns to blueprints I though as they our tool
<sinzui> czajkowski, we are not
<sinzui> czajkowski, the launchpad team does not use blueprints
<czajkowski> no we don't currently
<czajkowski> was under the impression that was changing in the future, but it was a slow process
<deryck> czajkowski, if that line from sinzui caused a lot of trouble, I'd guess there is more to people's frustrations than any sin caused by a single sentence.
<czajkowski> deryck: people dont have much to do at weekends
<deryck> czajkowski, said people should game more then.
<deryck> open source tempers could be swayed by more gaming.  I swear it's true.
<czajkowski> deryck: or submit patches for LP :_)
<deryck> well, I was thinking something fun to do. :-P
 * deryck totally kids
<czajkowski> right finally leaving the office
<czajkowski> nn
<sinzui> jcsackett, has ec2 explained why it does not want users to hide comments?
<jcsackett> sinzui: well, i still don't know why the one run died silently, but i have resolved all the errors in the hate mail and have a run that appears to be going smoothly now.
<sinzui> will you lp-land it?
<jcsackett> once i get a completed test run back; some of the errors in the hate mail was from a few places where i had been dumb about which code needed to be altered with the feature flag, so i had to update that. i want to see a complete ec2 run.
<sinzui> understood
<sinzui> rick_h_, do you habe time to review https://code.launchpad.net/~sinzui/launchpad/add-private-member/+merge/115419
<rick_h_> sinzui: not atm, sorry. Can later tonight.
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
<sinzui> thats okay, I will ask jcsackett who I discussed the issue with
<sinzui> jcsackett,  do you have 10 minutes to review https://code.launchpad.net/~sinzui/launchpad/add-private-member/+merge/115419
<jcsackett> sinzui: I will in just a moment.
<jcsackett> sinzui: ok, looking now.
<jcsackett> sinzui: that's not even a 10 min branch to look at, given we both already know the backstory. :-)
<sinzui> I thought as much.
<jcsackett> r=me. glad it was what we had assumed.
<jcsackett> sinzui: comments branch had a successful testrun, and has landed. should be playing on buildbot soon.
<sinzui> \o/
<wallyworld_> rick_h_: yo
<rick_h_> wallyworld_: hey, howdy. in the last hour before the kid goes to bed so in/out
<wallyworld_> rick_h_: just a quick thing - i looked at qa ing the report a bug link and it's still broken
<wallyworld_> rick_h_: i can mark it as qa-ok but the bug will need more work
<rick_h_> wallyworld_: crap, broken how?
<wallyworld_> rick_h_: clicking on it does nothing
<wallyworld_> we for me anyway
<wallyworld_> well
<rick_h_> looking
<wallyworld_> i looked at https://bugs.qastaging.launchpad.net/launchpad
<rick_h_> hmm, html is fixed
<rick_h_> https://bugs.qastaging.launchpad.net/launchpad/+bugtarget-portlet-bugfilters-stats 503?
<rick_h_> hmm, not sure why it's not clicking now though, the html is cleaned up correctly so something else is wrong, no idea off the top of my head
<rick_h_> will have to look after the boy goes to bed and maybe in the morning. *sigh*
<wallyworld_> rick_h_: np. i'll mark as qa-ok and set back to in progress
<rick_h_> wallyworld_: yea, so the link is there, has an href, so someone is catching it and killing the event somewhere.
<rick_h_> the li has class js-action and inactive on it for some reason, wonder if that's some of it
<rick_h_> maybe huw can peek if he's around today as I thought his html updates where part of what changed
<wallyworld_> rick_h_: i'll see if i can get to it today, and you can pick it up for your SOD tomorrow
<rick_h_> wallyworld_: np, just trying to think out loud while looking.
<rick_h_> I could have sworn it worked locally, if you get a sec do me a fav and see if local matches qa please
<wallyworld_> ok, sec
<wallyworld_> rick_h_: works locally with dev
<wallyworld_> go figure
<rick_h_> wallyworld_: yea see...ugh so something strange is up...qastaging hates me today.
<rick_h_> ok, well that's a hint so will look more. wonder if it's combo loader related
<wallyworld_> rick_h_: yeah, i might leave it to you for tomorrow then
<rick_h_> see if you can get the combo loader feature flag turned on in qa staging (it's off by default still I believe) and then no idea wtf would be wrong there.
<wallyworld_> ok
<rick_h_> wallyworld_: rgr, sorry, thanks for the help and report, will stick on it like glue
 * rick_h_ is annoyed this 'simple' thing is still broken is
<rick_h_> is all, the 24hr to edit/land sucks with this stuff
<wallyworld_> yep
<wallyworld_> rick_h_: getting flag on qas right now
<StevenK> I know we turned it off on staging yesterday, I wasn't sure about qas
<wallyworld_> StevenK: it's off, getting it back on to see what happens
<rick_h_> StevenK: side note, I asked you to review a branch that adds a dep to launchpad-deps for unzip so we can use the upstream yui .zip files pls
<wallyworld_> rick_h_: broken with combo loaded also
<wallyworld_> loader
<rick_h_> StevenK: can you peek/help me get that updated please?
<rick_h_> wallyworld_: ok cool, that actually makes me feel better tbh
<rick_h_> ok, bath time for the boy, will check back in later .Thanks again guys
<wallyworld_> rick_h_: good luck with this...
<cjwatson> Is https://code.launchpad.net/~jml/lazr.restfulclient/multiple-instance-safe/+merge/98873 ever likely to make it into an actual release?  It'd be nice to get this into quantal and precise-updates
<cjwatson> bigjools: How would you feel about switching on soyuz.copypackageppa.enabled for everyone and setting soyuz.derived_series.max_synchronous_syncs to 1 for launchpad-beta-testers, now that failed PCJs are shown in the web UI?  I've tried similar things on dogfood and haven't noticed any problems with async copies there.
<cjwatson> It'd be pretty awesome to be able to make the async code work for everyone and remove the sync code.
<bigjools> I'd feel ecstatic but I also don't feel I'm currently in a good enough position to judge  this since I've not worked on LP since January
<bigjools> desperately trying to keep up with developments though!
<cjwatson> OK.  Should I get some other TL to have a look?
<bigjools> removing the sync code might be problematic
<bigjools> some people depend on that behaviour
<cjwatson> Hm.  Any pointers?
 * bigjools racks brain to think who objected
<cjwatson> I'm just talking about the sync mode in Archive:+copy-packages, not the Archive.syncSource API
<bigjools> ah!
<bigjools> that's probably ok then
<bigjools> it might be a good idea to blog/tweet about this change in advance
<bigjools> and as you say turn it on for beta testers first
<cjwatson> I already know the libreoffice guys will love it :-)
<bigjools> :)
<cjwatson> (Well, assuming it works)
<cjwatson> But yeah, you're right
<bigjools> I'd also email the -users list (mmm do we have a beta users list?)
<cjwatson> Removing the +copy-packages sync mode is about -200 LoC, but it's one of the last steps on the way for removing delayed copies which is -1100 or so
<bigjools> \o/
<cjwatson> ~launchpad-beta-testers just tells people to join ~launchpad-users
<bigjools> ok
<cjwatson> All right then.  I ought to go to bed, but I'll see if I can persuade jam or somebody in the morning to approve a feature flag change, and talk to czajkowski about the comms
<cjwatson> Thanks
<bigjools> cjwatson: np.  Sorry, I'd love to approve it, I would, but that would make be responsible for something that I wasn't involved in lately :)
<bigjools> me*
<cjwatson> Yeah, makes sense
<cjwatson> I can start writing a blog entry in the meantime
<bigjools> cool
<rick_h_> StevenK: thanks for the catch on the email. The reason I ping'd you specifically as I wasn't sure how to process from here on it and wanted to make sure I got it right.
<rick_h_> StevenK: so do I submit to trunk, then dput up to the ppa, and everything should catch the update there?
<StevenK> rick_h_: Just commit to trunk, a recipe does everything else.
<rick_h_> StevenK: ah, even better then
<rick_h_> StevenK: so then does something cause the ppa to pull updates on the machines? qa/etc?
<rick_h_> or do I somehow trigger production/qa servers to run an apt-get upgrade?
<StevenK> That happens seperately. You need to talk to webops about it.
<rick_h_> ok, cool. I want to make sure I do things in the right order to prevent borking things again
<StevenK> But you keep borking things anyway :-P
<rick_h_> I try not to!!! lol
<rick_h_> I'm cursed this last week I tell you
#launchpad-dev 2012-07-18
<StevenK> wgrant: I would have expected http://pastebin.ubuntu.com/1097605/ to fail on the last assert.
<wgrant> StevenK: Hm, it may just happen to work atm.
<wgrant> But we still need the proper fix so that someone can always recover access and we can remove the default sub.
<StevenK> wgrant: I was going to add an null APG for USERDATA on private team creation
<wgrant> USERDATA on what?
<wgrant> We should have (pillar, information type) and (team,) APs, I suspect
<wgrant> I don't see a need for an information type for private team APs
<StevenK> wgrant: APs don't support teams
<StevenK> That I can see
<wgrant> StevenK: That's correct.
<wgrant> But there's no pillar involved
<wgrant> So they have to be made to support teams.
<wgrant> Currently APs have a key of (pillar, information_type), but they need to also support (team,)
<wgrant> Private junk branches are probably only allowed for private teams, and they get an AP of (team,) regardless of their information_type.
<wgrant> The AP is immutable and has a single APG: the team itself.
 * StevenK glares at wallyworld_, and picks lp.code.branchmergeproposal.nominate apart
<wallyworld_> StevenK: oh no, what have i done this time?
<StevenK> wallyworld_: Written lies which weren't lies when you wrote them.
<wallyworld_> oh, is that all :-)
<StevenK> wallyworld_: Actually, why is there a confirm_reviewer and a confirm_reviewer_nomination function?
<wallyworld_> StevenK: no idea, let me read the code
<wallyworld_> StevenK: one happens client side with known information when the mp page was loaded, the other as a result of mp submission
<wallyworld_> since the user could enter info that invalidates the initial check
<wallyworld_> StevenK: confirm_reviewer_nomination is a popup when submit is clicked
<wallyworld_> the other is a picker validation plugin
<wallyworld_> like is used for assigning a new person to a bug
<wallyworld_> it happens inside the picker when a reviewer is chosen
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 4.0*10^2
<czajkowski> cjwatson: do you need me to do some sort of annoucment for you ?
<cjwatson> czajkowski: oh, right, let me find a TL to rubber-stamp^Wapprove my plan
<jelmer> czajkowski: unfortunately we can't import import nburrus/rgbdemo since it contains submodules
<czajkowski> cjwatson: heading to stand up now
<jelmer> (also, good morning :)
<cjwatson> Hm, I think I need to wait for the US to wake up
<cjwatson> czajkowski: I was thinking of a post on the Launchpad blog something like this: http://paste.ubuntu.com/1097990/.  It could use tightening up of language, moving the image to a more sensible/permanent place, and so on.
<cjwatson> (And this is as yet unapproved so don't post it yet :-) )
<czajkowski> cjwatson: hehe ok, well I can review it later with mrevell on our 1-2-1 if you want
<czajkowski> or do you need higher beings to ack it :)
<cjwatson> bigjools was cautiously optimistic about the general plan but didn't feel plugged in enough to current Launchpad to ack the feature flag changes, so I'll need to grab a team lead for that.  He suggested announcing the changes.  Reviewing the text with mrevell would be good.
<cjwatson> I went into a fair amount of detail as I thought this might be a little-known area that could benefit from a bit of light shone on it, but maybe I overdid it :)
<czajkowski> cjwatson: I just want to highlight other people are helping getting launchpad updated in areas to encourage others to do so :)
<czajkowski> cjwatson: jam is away this week so poke gary_poster or deryck
<cjwatson> Right.  Like I say, waiting for the US :)
<czajkowski> ok :)
<gary_poster> cjwatson, I'm here early because I very much need to do some more prep work for some interviews that start in 1:19.  So, I've read your proposed blog post (sounds good) and I can spend another 3 or 4 minutes thinking about it.  deryck will be around in a while.  I won't be available again for five or six hours.
<gary_poster> So, can I do anything else for you?
<rick_h_> gary_poster: tell Jim I said hi, you've got him in a bit of awe :)
<gary_poster> rick_h_, lol ok
<cjwatson> gary_poster: I may have missed you.  If not, the main thing I would like is approval to make a couple of feature flag changes: soyuz.copypackageppa.enabled on for everyone (AIUI this was disabled because of the lack of failed copy notifications, which is now fixed) and soyuz.derived_series.max_synchronous_syncs to 0 for launchpad-beta-testers so that we can see if async copies work for them
<cjwatson> soyuz.derived_series.max_synchronous_syncs is currently set to something implausibly enormous in order to effectively disable asynchronous copies.
<cjwatson> But I've been working on that area and they're working fine for me on dogfood.
<cjwatson> So I think the next stage is to see if they work for other people.  The ~libreoffice people ask me for manual copies roughly once a week at the moment so I expect they'll provide a test case soon, if nobody else does.
<cjwatson> soyuz.copypackageppa.enabled will also permit the Archive.copyPackage API method to work when the target is a PPA.
<cjwatson> Unrelatedly, what's the best Stormy SQLish way of saying "if there is a row matching condition A, return it; otherwise, if there is a row matching condition B, return it"?
<cjwatson> Coalesce looks sort of right but I'm not sure.
<wgrant> cjwatson: What are you trying to do?
<wgrant> COALESCE isn't for whole rows, usually
<cjwatson> I'm trying to do a version of the target => bug task search in Bug.setStatus that works for multiple bugs.
<cjwatson> Currently it's doing several consecutive queries in Python.
<cjwatson> Well, depending on the type of the target.
<cjwatson> (I know this is only a tiny bit of the close_bugs_for_sourcepackagerelease problem, but I'm working myself up to it.)
<wgrant> Hmmm
<cjwatson> So, say, for an IProductSeries target that method needs to find a product series task if there is one and otherwise fall back to a product task if there is one.
<wgrant> I'm not sure it's massively worth doing that, and it's going to be a bit of a challenge.
<wgrant> Not worth it because the ObjectModifiedEvent is going to cause O(tasks) write queries anyway
<wgrant> It's not really going to be practical to do all the task lookups in one query. I'd divide them up by type, doing the fallback in secondary queries as it does now
<wgrant> But batched
<wgrant> So look up tasks for all the targets. If there are productserieses left over, look up tasks for their products. If there are sourcepackages left over, look up tasks for their DSPs.
<cjwatson> Why batched (within each lookup, I assume you mean)?  I was kind of hoping it would be possible to get the query count low enough that it wouldn't be necessary
<cjwatson> I guess the number of bugs is technically unbounded
<wgrant> Right
<wgrant> Though
<wgrant> The task lookup in setStatus right now should be one query
<wgrant> And only the first the first call on a bug
<wgrant> Subsequent calls should be 0 queries
<cjwatson> Oh, because the first .target fetches all the columns?
<cjwatson> Maybe I should just bulk-load the bugs then.
<wgrant> Ah, yeah, it uses target, so it resolves the whole thing. The .target call will be up to 2 queries for every task on the bug, but still only the first time.
<wgrant> However
<wgrant> That getBugTask thing is insane.
<wgrant> We have perfectly good ways to filter tasks by target
<wgrant> Without making hundreds of queries
<cjwatson> Maybe I'm just being stupid but I couldn't find an obvious one that quite fit.
<wgrant> cjwatson: You can use a normal bug search.
<wgrant> cjwatson: See eg. Bug.userCanView
<wgrant> params = BugTaskSearchParams(bug=self.id)
<wgrant> params.setTarget(sometarget)
<wgrant> getUtility(IBugTaskSet).search(params).one()
<wgrant> or so
<wgrant> You'll probably want _noprejoins=True to avoid issuing more than one query
<cjwatson> Bit of a problem with privacy there.
<wgrant> cjwatson: It'll default to not filter by privacy at all
<wgrant> Um
<wgrant> I lie
<cjwatson> It defaults to public bugs only
<wgrant> Indeed
<wgrant> cjwatson: That's really the ideal way to do it, but the mandatory privacy filter is a bit annoying.
<cjwatson> There's the evil circumvention of finding an admin user and passing that in ...
<wgrant> Yeah, but no.
<cjwatson> Quite.
<cjwatson> disable_privacy=True I guess
<cjwatson> Well, privacy=False
<wgrant> Something like that.
<wgrant> filter_private=True maybe
<wgrant> Probably better to fix task search than use bug_target_to_key manually, anyway.
<deryck> adeuring, https://plus.google.com/hangouts/_/84943eb8075e033904f027277d6b4e951afc2038?authuser=0&hl=en
<cjwatson> frankban: Could I have a review of https://code.launchpad.net/~cjwatson/launchpad/garbo-archivepermission-duplicates/+merge/115554 ?
<frankban> sure cjwatson
<cjwatson> I've checked that SQL fragment by hand on dogfood.
<cjwatson> (Just as well since I'd never have got it right non-experimentally.)
<rick_h_> frankban: review your way if you get a sec https://code.launchpad.net/~rharding/launchpad/reportbug/+merge/115562
<rick_h_> frankban: added a link to the first branch for some context
<frankban> rick_h_: on it
<rick_h_> ty much
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: 4.0*10^2
<abentley> deryck: I'm the OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/broken-xmlrpc-lookup/+merge/115582 ?
<deryck> abentley, sure.
<abentley> deryck: Thanks.
<deryck> abentley, np
<cjwatson> deryck: Would you mind having a look at my comments to gary_poster at http://irclogs.ubuntu.com/2012/07/18/%23launchpad-dev.html#t11:53 (and, for background, my conversation with bigjools at http://irclogs.ubuntu.com/2012/07/17/%23launchpad-dev.html#t23:31)?  I'm looking for a TL to approve a couple of feature flag changes there.
<deryck> cjwatson, ah, sure.  Give me just a few minutes and I can look into all that.
<rvba> rick_h_: hey, I can't say I understand every detail of it and I see this is fixing a critical bug so I don't want to hold you too long but I'm wondering about the lines 119-120 of the diff for the branch ~rharding/launchpad/reportbug.  I'm wondering how that doesn't imply a change in the related test code...?
<rick_h_> rvba: let me load it back up sec
<rick_h_> rvba: no, the sharee test code didn't test the navigation html at all relying on the underlying code to test/do it
<rick_h_> rvba: thought about adding a test, but wasn't sure if it was worth the LoC to test what's tested already.
<rvba> rick_h_: all right, I guess that's ok if the code gets run by the tests somehow.  I was just checking.  Thanks for the explanation.
<rick_h_> np, I thought the same thing tbh so understand the concern
<rvba> Cool.
<deryck> abentley, looks good, r=me
<abentley> deryck: ty
<deryck> np
<deryck> cjwatson, in terms of the flags, you have my approval to change them.
<cjwatson> deryck: Thanks.  I'll get that onto LPS shortly.
<deryck> cjwatson, np!
 * deryck reboots, brb
<czajkowski> cjwatson: do you want it added to the blog then?
<cjwatson> czajkowski: Yeah, was just waiting for the FF change to happen before asking you.  http://paste.ubuntu.com/1098642/ is slightly updated text; is there somewhere more permanent you could put the image there?
<cjwatson> czajkowski: I've also logged into wp-admin, and mrevell says you can post in my name once I've done that
<czajkowski> he briefly explained this to me today
<czajkowski> need to find you on the list
<czajkowski> cjwatson: set to authour so lets see how this work
<czajkowski> will fix the image up once it's in the blog
<rick_h_> abentley: so care to be the 3rd person to review? https://code.launchpad.net/~rharding/launchpad/yuiv3/+merge/115592
<rick_h_> abentley: will not land until after veried everyone has the updated launchpad-deps package
<cjwatson> czajkowski: Good to go
<czajkowski> cjwatson: hmm I don't see it
<czajkowski> will have a munch of dinner and investigate
<cjwatson> czajkowski: See what?
<czajkowski> cjwatson: ah thought you'd added it to the blog and in draft
<cjwatson> Oh, do you mean you gave me permissions?
<czajkowski> will sort it out after munchables
<cjwatson> I misunderstood.  I'll write up a draft.
<abentley> rick_h_: okay
<cjwatson> czajkowski: http://blog.launchpad.net/?p=3627&preview=true
<abentley> rick_h_: 2083 is apparently the max URL length in ie8.
<rick_h_71> ah ok will update that.
<czajkowski> cjwatson: lovely
<czajkowski> posting to the places now
<czajkowski> http://blog.launchpad.net/general/beta-test-asynchronous-ppa-package-copies
<abentley> rick_h_: r=me
<rick_h_71> ty much abentley especially for the length catch.
<cjwatson> czajkowski: Thanks.  Belatedly fixed up the image a bit to not have bits of "demo" over the background
<abentley> rick_h_71: np.
<czajkowski> heh
<czajkowski> looks good
<czajkowski> thanks cjwatson
<cjwatson> czajkowski: I'll do launchpad-users@ if you like
<czajkowski> cjwatson: great
<cjwatson> https://lists.launchpad.net/launchpad-users/msg06483.html
<micahg> cjwatson: would it help if I stress tested copy-packages now?
<cjwatson> micahg: Yes please
<cjwatson> micahg: Though remember that if you copy source only then that's expensive in terms of build time
<micahg> cjwatson: no, I have a bunch of Mozilla stuff I can copy to a PPA that's a no-op for users (12 sources + ~500+ binaries)
<cjwatson> Sounds like a good plan
<cjwatson> You'll be "charged" for it in terms of quota, of course
<micahg> well, almost a no op, the PPA packages file will bloat a little, but not that much
<cjwatson> I don't think there's any hardlinking on disk
<micahg> Requested sync of 11 packages.
<micahg> Please allow some time for these to be processed.
<micahg> nice :)
<micahg> ooh, and the PPA shows what's waiting
<cjwatson> URL?
<micahg> https://launchpad.net/~mozillateam/+archive/thunderbird-stable/+packages
<micahg> ok, gotta run, will check later to see if everything worked
<cjwatson> Either I was too slow or that's shown to the owner only, I forget
<cjwatson> Would make some sense for it to be the latter
<micahg> gone now
<cjwatson> Oh, there's a bunch of Pending items there, which are probably yours
<micahg> yeah, all looking good
<micahg> (aside from not enforcing quota, but that was previous behavior as well)
<cjwatson> Mm, possibly worth fixing
<cjwatson> But otherwise sounds like flawless victory so far
<SpamapS> hm, seems diffs linked to from +queue are getting the wrong host: 'lplibrarian-private-download.internal:8000'
<lifeless> yes
<lifeless> there is a bug
<lifeless> colin put up a patch yesterday IIRC
<cjwatson_> SpamapS,lifeless: it's fixed on devel - just needs me to do a bit of QA, then a deployment
<cjwatson> I guess I should see if it's possible to delete the busted diffs and have them regenerated, afterwards
<SpamapS> excellent. :)
<cjwatson> SpamapS: And passed QA, so hopefully deploying tomorrow.
<StevenK> cjwatson: I'll be doing QA after my stand up, and I'm tempted to put up a deployment afterwards.
<cjwatson> I was considering setting status=PENDING for everything in SELECT COUNT(pd) FROM packagediff pd, sourcepackagerelease spr, archive a, libraryfilealias lfa WHERE pd.to_source = spr.id AND spr.upload_archive = a.id AND a.private IS FALSE AND pd.diff_content = lfa.id AND lfa.restricted IS TRUE;
<cjwatson> Er, with syntax, but YKWIM
<cjwatson> I think that would clear up the breakage.  There are no such rows on dogfood right now, which is a good sign that at least it probably isn't overbroad
<lifeless> cjwatson: why noy just toggle the restricted flag for them ?
<cjwatson> Wouldn't the actual diff need to be re-uploaded to the public librarian too?
<cjwatson> That's certainly how it works when copying packages from private to public archives.
<cjwatson> Seems easier to just get it to redo the diff - the bug hasn't been present for all that long so I expect it would be quick enough.
<wgrant_> cjwatson: It's all a lie
<wgrant_> cjwatson: The reupload is entirely unnecessary
<wgrant_> I forgot we even still did that
<cjwatson> Seriously?  And I spent actual effort understanding that.
<wgrant> There was perhaps an idea years ago that the two librarians would be separate
<cjwatson> And possibly even extending it.
<wgrant> But that never eventuated
<wgrant> Hm? The majority of the code you touched was for working out the files
<wgrant> You still need to work out the files
<wgrant> You just need to toggle the flag rather than reuploading.
<cjwatson> Yeah, I can't remember now.  Plausible.
<cjwatson> So OK, in that case flipping the flag is trivial enough.
<huwshimi> Can I just ignore these ec2 test failures for AttributeError: type object 'InformationType' has no attribute 'EMBARGOEDSECURITY'?
<wgrant> huwshimi: Yes, wallyworld fixed those last night after two conflicting branches landed.
<huwshimi> Ah
#launchpad-dev 2012-07-19
<micahg> cjwatson: everything is confirmed published
<wallyworld_> StevenK: did you know RemoveArtifactSubscriptionsJob doesn't handle branches where access is revoked using the sharing ui?
<StevenK> wallyworld_: Uh, no. Guess I missed a callsite. :-(
<adeuring> good morning
<adeuring> jelmer: could you please review this mp: https://code.launchpad.net/~adeuring/launchpad/bug-1020443/+merge/115721 ?
<salgado> I thought the comment box on a merge proposal would preserve the formatting, but it just ruined a review of mine :(
<salgado> https://code.launchpad.net/~diogobaeder/ubuntuone-servers/consistency-check/+merge/115441/comments/248694
<salgado> has that changed recently?
<salgado> actually, if I get the comment from that URL it looks right, but on https://code.launchpad.net/~diogobaeder/ubuntuone-servers/consistency-check/+merge/115441 it's not formatted
<jelmer> adeuring: on it
<adeuring> jelmer: thanks!
<cjwatson> Could somebody allocate me a DB patch number?  I would like to add "ALTER TABLE ArchivePermission ADD COLUMN distroseries INTEGER;" in order to support implementation of per-pocket queue admin permissions; feedback from the relevant stakeholders has convinced me that we may need to be able to grant some people access to admin e.g. quantal-proposed but not precise-proposed.
<rick_h_> cjwatson: will work on it, sec
<rick_h_> cjwatson 2209-27-1 allocated
<cjwatson> rick_h_: Thanks
<deryck> Morning, everyone.
<rick_h_> morning boss
<deryck> adeuring, https://plus.google.com/hangouts/_/8a443b035ee300fdac4f1b58feadf013d81db8e7?authuser=0&hl=en
* jcsackett changed the topic of #launchpad-dev to: http:/â/âdev.launchpad.net/â | On call reviewer: jcsackett | Firefighting: - | Critical bugs: 4.0*10^2
<mgz> so, this looks like a good example
<mgz> the current head of the qa queue is qa-bad
<mgz> so, the normal thing would be to revert that change? is that generally left to the person who landed it?
<jelmer> mgz: Yes, if they are around
<jelmer> I guess generally it's left to the person doing the QA
<mgz> so, you'd basically do `bzr lp-land --rollback=15645 lp:launchpad` then? or do you actually revert the change in a branch locally?
<mgz> rollback sounds not like "merge the inverse of this change" but I assume it doesn't actually lose subsequent things in the queue
<jelmer> mgz: yeah, merge the inverse of that change
<jelmer> mgz: You'll actually have to revert the change locally and push a branch up for merging.
<mgz> but do you need to actually do that and put up an mp?
<mgz> meh.
<mgz> so, is just tagging not fanciness.
<jelmer> mgz: well, the tagging is important for the qa bot
<jelmer> not sure if that qualifies as fanciness..
<cjwatson> You merge the inverse of that one change, not everything after it in the queue.
<cjwatson> There's something about it on dev.lp.net somewhere.
<rick_h_> right, mgz what I did the other day was to grab the branch, diff -r with the rev and the one before to generate a patch to remove that rev
<rick_h_> then create a new branch from trunk, apply the patch, submit to pqm with bzr lp-lang
<cjwatson> https://dev.launchpad.net/QA/QAForContinuousRollouts
<rick_h_> land that is
<mgz> do you just merge the branch, or put it through review?
 * mgz reads the page
<mgz> "get it reviewed, or review it yourself"
<cjwatson> I think the general implication of that is that if you are a reviewer then this is the kind of change you can self-review.
<jcsackett> mgz, cjwatson: correct--we allow self review for full reviewers on things like that.
<cjwatson> I'm not a reviewer so I think if I were doing this I would get somebody to review it.
<rick_h_> mgz: yea, as long as you're only removing the one rev it's generally a self-review.
<jcsackett> if you're not a full reviewer, well, i'm on call today. :-)
<rick_h_> put jcsackett to work!
<rick_h_> I don't have any giant JS branches today, he's going to feel lonely
<mgz> okay, I don't expect StevenK will mind me doing this as practice
<jcsackett> rick_h_: there's a somewhat easier way to do the rollback than the "create patch with diff, then apply" process, btw.
<rick_h_> jcsackett: I'm all ears
<jcsackett> bzr merge . -r [revno-to-revert]..[revno-to-revert-1]
<cjwatson> QAForContinuousRollouts recommends that.
<rick_h_> ah makes sense, one less step
 * jcsackett nods. and when you're in qa triage, less steps are good. :-)
<mgz> hm, interestingly StevenK already did the rollback last night, but it doesn't show up on lpqateam
<jcsackett> adeuring: looking at your branch in review--doesn't this need to be landed against db-devel, not devel?
<adeuring> jcsackett: no, since there is not schema change or any other "serious impact", it can be landed in devel. Seehttps://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess
<jcsackett> adeuring: ah, cool. thanks.
<rick_h_droid> mgz is it in buildbot then?
<mgz> nothing on pqm.launchpad.net
<mgz> am I looking in the right place?
<cjwatson> mgz: http://lpbuildbot.canonical.com/waterfall
<cjwatson> ETA 1h54m
<jcsackett> adeuring: the diff i see looks good, but the MP says there's a conflict in need of resolution. (the patch file appears to already exist). fix that up and i can stamp this r=me.
<adeuring> jcsackett: argh... thanks for the heads-up!
<jcsackett> adeuring: your welcome.
<cjwatson> mgz: the flow as I understand it is devel -> buildbot (whenever it next starts a run) -> buildbot completes 5/6h or so later -> qastaging's buildbot-poller runs every 15m or so IIRC and picks that up -> qastaging.lp.net updates -> the deployment report notices shortly afterwards
<adeuring> jcsackett: done
<cjwatson> this is admittedly divined from observation :)
<jcsackett> adeuring: awesome. i'm waiting on the diff to update. :-)
<mgz> and everyone lands with a full test run anyway...
<mgz> not sure what the double gating helps exactly.
<jcsackett> mgz buildbot after ec2 picks up multiple revisions. makes sure if three of us start from the same branch and do something bad with another person's work things don't get broken.
<jcsackett> (three being an arbitrary number, of course)
<cjwatson> the test run is based on whatever was current at the time you started the test run, but what lands on devel is a merge based on whatever's current when the test run ends
<cjwatson> and buildbot does sometimes fail as a result of that
<mgz> ah, another the-test-suite-takes-5-hours adaptation
<jcsackett> mgz: if you like. :-P
<jml> mgz: exactly.
<jcsackett> cjwatson: can you file and link bugs for both of your branches? neither seem like the sort of thing that should be landed with --no-qa.
<jml> cjwatson: https://dev.launchpad.net/Trunk, https://dev.launchpad.net/Trunk/Glue
<cjwatson> jcsackett: OK
<cjwatson> jml: Ah, yes, I probably read those at some point
<jml> mgz: although I think we were all complaining about the length of the PQM queue back when the test suite took 1hr. It was common to be waiting 16hrs or more for a branch to land
<jml> mgz: so arguably, mean time to land is shorter with this setup.
<rick_h_> and when the parallel test guys save us we'll be awesome
<jml> although it's very frustrating to have a change rejected because of someone else's mistake
<mgz> when we completely hosed the bzr test suite so it took going on for an hour the pqm queue became a pain
<jml> rick_h_: I would strongly recommend switching back to a more pqm-like model in that case
<mgz> so the test suite was made fast again.
<rick_h_> mgz: yea, and the parallel guys are reporting sub-50min numbers.
<jml> mgz: yeah. LP devs never did that. I got a bit sick of talking about it so I wrote <https://dev.launchpad.net/FasterTests>. It's probably a bit out of date.
<mgz> will parallel testing be done both for the buildbot step and the I'm-just-checking-my-own-branch step?
<jcsackett> adeuring: r=me, but you'll want to update the values in INSERT INTO LaunchapdDatabaseRevision to match the patch number.
<jml> mgz: the pqm model scales poorly to large numbers of developers
<adeuring> jcsackett: argh... not my best day. thanks!
<mgz> jml: it mostly scales badly for NEWS files...
<jcsackett> adeuring: it happens. :-P
<jml> mgz: arguably a work-around would be to merge them all in, run the tests, and in case of failure try again but with half of the queue, ...
<mgz> made me not want to ever write any, to avoid pointless conflicts
<jml> mgz: that's Bazaar's fault, I think.
<mgz> right, and never getting news merge working on pqm
<jml> mgz: Twisted works around that by having one file per NEWS update, and building the NEWS file as part of the release process
<mgz> yeah, that's very cute, and so twisted
<jelmer> :-)
<rick_h_> there's always a work around :)
<jml> mgz: I don't think it's a good work-around, but it's better than the pain of pointless conflicts.
<jelmer> the custom bzr news merger has worked quite well for us
<jml> I want to set up jenkins to have a pqm model for landing branches
<jcsackett> cjwatson: i'm looking at your security upload branch. i'll confess to being unfamiliar with the processes around this. i assume without this check there are stil checks to keep just anyone from uploading to the pocket?
<cjwatson> jcsackett: Much like -proposed/-updates, any upload to it would require manual approval by a queue admin
<jcsackett> cjwatson: ah, dig.
<jcsackett> thanks.
<cjwatson> jcsackett: In fact, today, anyone with upload rights to Ubuntu could attempt to copy a package into it (and it'd be similarly held for approval)
<cjwatson> Since copy permissions == upload permissions (I actually think copy permissions should be very slightly broader, but that's a different bug)
<jcsackett> cjwatson: ok. so this is completely unnecessary, as there's a manual check anyway, and if someone were operating in bad faith for some reason they would have a way around it anyway.
<cjwatson> I did ask the Ubuntu security team manager if he was OK with this, but he's on holiday this week
<cjwatson> The history of that code definitely looks like a fossil though
<cjwatson> I actually ran into it because I was trying to test something else on dogfood and it wouldn't let me upload to -security there :)
<cjwatson> In practice we probably want to stage elsewhere most of the time anyway because that makes sure everything's built before we expose it to users.  But that seems like a distro policy decision rather than something that should be hardwired into code.
<jcsackett> cjwatson: that makes sense. thanks.
<jcsackett> cjwatson: both your branches are r=me, but remember to file/link bugs so the qa can be tracked. :-)
<jcsackett> ah, i see you already have on one.
<cjwatson> yup, working on it.
<cjwatson> thanks
<mgz> meh, I should just fix this mailman thing
<stub> adeuring: Whoops. I forgot to push my dbpatches change.
<stub> ta
<adeuring> stub: no problem,. I added a note about your patch ;)
<SpamapS> Is there a minimum level of performance one should be able to expect from buildds (particularly PPA builders) ?
<SpamapS> The juju test suite wants every test to finish in 5 seconds or less..
<SpamapS> but on many of the PPA builders that just doesn't happen
<cjwatson> That sounds pretty wrong.
<cjwatson> I mean, of the juju test suite.
<SpamapS> Its just a default, but each test is meant to be very tiny
<SpamapS> on my Core2Duo 2.5Ghz machine with crap disk, they all finish in under 2s reliably..
<cjwatson> I don't think it should be allowed to assume anything in particular.  They're Xen instances, they could be timesliced off into oblivion for a while
<SpamapS> cjwatson: nannyberry, in particular, seems to be quite a bit faster than all the rest
<SpamapS> cjwatson: but, that said, I suppose we should just raise the test timeout to 20s for package builds (which is unfortunate as we will miss performance regressions)
<mtaylor> SpamapS: I would suggest not using ppa builders as a place to test performance regressions
<SpamapS> indeed, its not really a place to do that. :)
<mtaylor> SpamapS: I know of this really neat software called jenkins ...
<mtaylor> and then I'd suggest getting a bare metal machine that only runs your performance tests so that one performance test's output can be actually compared against another
<mtaylor> but that might just be me
<SpamapS> mtaylor: I heard jenkins is just cron on steroids
<mtaylor> SpamapS: you could totally do cron
<mtaylor> SpamapS: or you could use a combination of gearman and salt
<SpamapS> which means it is strong but aggressive and moody ;)
<mtaylor> to do queuing and remote command execution
<cjwatson> Right, I agree.  Performance tests in package builds are a mug's game.
<mtaylor> but whatever you do, I would not use ppa builders for performance testing
<SpamapS> well up until now they've been that just because of twisted trial's default 5s timeout
<cjwatson> You could even register DEP-8 format tests (autopkgtest) and then your tests will be automatically run in jenkins.
 * SpamapS is about to patch that away
<mtaylor> SpamapS: you could stop using twisted of course ...
<SpamapS> mtaylor: we are!
<SpamapS> -> go
 * mtaylor facepalms
<mtaylor> SpamapS: don't you love it how you come here to ask a simple question and we all start kicking you?
<SpamapS> what, a compiled language w/ static-only-linking isn't a good answer to "twisted is hard" ?
<mtaylor> SpamapS: well, "twisted is overkill for the task juju needs to perform" would be my first answer to "twisted is hard"
<SpamapS> I was feeling a bit doughy.. a good kicking will get me back into shape :)
<mtaylor> SpamapS: so, my first suggestion would have been "remove twisted and replace it with normal python calls"
<cjwatson> Gosh, that might have interfered with religion
<cjwatson> ... was that my out loud voice?
<mtaylor> oops. sorry about that
 * mtaylor tries to stop interfering with religion
 * mtaylor stops pointing out assininery around twisted and go
<cjwatson> I certainly wasn't trying to stop you :)
<jml> http://www.joelonsoftware.com/articles/fog0000000069.html
<cjwatson> Yes, I want to sellotape that URL to my head at conferences sometimes
<jml> :)
<jml> I want to find a well-written article that explains that writing new software is almost always a worse idea than you think it is.
<mgz> wouldn't bother with well written, just go for big letters
<cjwatson> "programmers always want to throw away the code and start over" IME this is the bit where Joel missed a trick; it's far from always programmers who are the driving force behind rewrites ...
<SpamapS> true, though it is generally the programmers who whine about the messy code and thus the business folk figure "well its probably time for a rewrite"
<SpamapS> "this will make us more nimble"
<rick_h_> abentley: ping, do you have time to chat on this lp.client issue?
<abentley> rick_h_: lunching.  back in 30.
<rick_h_> abentley: ack
<mgz> what's aaron's fancy script for generating lp merge proposals again?
<rick_h_> bzr lp-propose
<mgz> as in, not just lp-propose, but the one that makes a little template too
<abentley> rick_h_: back.
<rick_h_> abentley: cool, will toss a hangout your way
<czajkowski> rick_h_: do you happen to know if you need multiple projects to have bug targets
<rick_h_> czajkowski: otp sec
<czajkowski> rick_h_: cheers
<rick_h_> czajkowski: hmm, abentley can sanity check me, but to target something it could be a different distro, release, source package, etc. so not necessarily different projects
<czajkowski> YokoZar: there is your answer
<czajkowski> rick_h_: thank you
<abentley> mgz: lp-propose, with the lpreview_body plug-in.
<mgz> abentley: thanks, where's lpreview_body in lp?
<YokoZar> rick_h_: could you clarify how I would do this for a private project?
<YokoZar> *commercial project
<barry> um, i think adding comments to bug tasks may be broken.  i just got an error that says "Object:, name: u'acton'"
<sinzui> Yokozar, I don't see your question. What is it? I know a lot about projects with private bugs and branches
<czajkowski> sinzui:  need multiple projects to have bug targets
<czajkowski> 19:10 < rick_h_> czajkowski: hmm, abentley can sanity check me, but to target something it could be a different distro, release, source package,  etc. so not necessarily different projects
<sinzui> targeting rules are byzantine. In the case of a bug, a project can be retargeted to another project using ajax. Otherwise, expand the html form to select distro, package, or project
<sinzui> series and series packages can only be retargeted to other series and series packages
<YokoZar> sinzui: So, my full question is this: I currently have one commercial project registered.  In order to use Launchpad bugs instead of our own independent tracker, I'd like to be able to have different components within (eg -server, -client).  Do I need to register multiple projects for this (and pay for each one?)
<sinzui> YokoZar: yes you do
<sinzui> Lp does not understand components and shared repos
<sinzui> YokoZar: we can create a project group for your project which will let you search all the bugs in all the projects. czajkowski or myself can create one once you have two projects to add to it
<sinzui> YokoZar: also, it is okay to structure your projects so that only one of them contains the proprietary code. several projects groups mix public and private code. They commonly have a private server part, or the plugins are all proprietary
<YokoZar> sinzui: I might not code-host on launchpad at all at the moment (we have stuff in github)
<sinzui> understood
<YokoZar> sinzui: but I do need things like private PPA and private bugs by default; I suppose I can confine the PPAs to one project
<sinzui> YokoZar: if your not hosting code, consider one project and use bug tags to distinguish the components
<sinzui> ppas are owned my teams, not projects. PPAs often contain packages based on many projects.
<sinzui> YokoZar: one you add the proprietary license to your private project, you can enable private bugs, create private teams, and create private ppas, The team can be public, and you can still have a private ppas
<sinzui> YokoZar: you might be tempted to create one project, with a series that represents each component. You can target bugs to the series. you can say a bug affects many series. You cannot retarget the series task from one to another, but you can delete the task and create an alternate task.
* jcsackett changed the topic of #launchpad-dev to: http:/â/âdev.launchpad.net/â | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
<YokoZar> sinzui: Thanks, I think that will work quite well for me
<StevenK> Fail. http://sqlformat.appspot.com/
<StevenK> wgrant: Is http://pastebin.ubuntu.com/1101043/ a bong query for returning direct subscriptions?
<wgrant> StevenK: Maybe. Does it work?
<wgrant> Also, it's a bit odd to constrain both BS.bug and BTF.bug rather than joining them explicitly.
<StevenK> wgrant: It does not seem to work, no.
<wgrant> StevenK: Oh
<wgrant> StevenK: The privacy condition is inverted
<wgrant>     AND NOT (
<wgrant>         BugTaskFlat.information_type IN (1, 2)
<wgrant> [...]
#launchpad-dev 2012-07-20
<StevenK> wgrant: Right, removing the Not() now has it return the owner, subscriber and the structsub, I'd expect only the first two
<wgrant> That should be more easily debuggable.
<StevenK> You'd think, but the query is defying me
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1101068/
 * wallyworld_ looks
<wallyworld_> StevenK: yes, looks good. you may want to cast an eye over the sharing tests to ensure that all the XXXs are removed and branches are fully covered. i had a nagging feeling there was a little cleanup required, but am not sure
<wallyworld_> certainly there must be no test for the method you just fixed
<rick_h_> wallyworld_: ping, hey that report a bug fix branch should hopefully hit QA soon. It passed buildout a while ago, but had the qa-ok tag on it still so thinking it didn't get picked up for qas
<rick_h_> wallyworld_: can you keep an eye and try to get that ok'd if a NDT goes down tonight?
<wallyworld_> rick_h_: sure, np. what was the remaining issue
<rick_h_> there was some additional JS that checked the pagination state to update it as required as you went from page to page
<StevenK> I've been tracking the deployment report
<rick_h_> so when the navigation got updated, it locked the report a bug link back out again
<rick_h_> the listing_navigator stuff needs to be torn to bits tbh
<wallyworld_> yuk. well at least it's fixed now
<StevenK> rick_h_: Which revision are waiting on?
<rick_h_> yea, I might write up a -dev email on catching greedy selectors in review
<rick_h_> StevenK: 15657
<StevenK> Right, I'd expect the deployment report to show that in about ~ 15
<rick_h_> StevenK: ok cool, was trying to wait for it but I'm done I think. Appreicate it if you guys can help get that updated.
<StevenK> 15650 needs QA, as does 15654 and 15655
 * rick_h_ is sad that link's been broken for almost 3 days
<wallyworld_> rick_h_: don't feel too bad, shit happens
<wallyworld_> StevenK: if you are working on that branch subscriptions removal job card did you want to take the card and move it to Coding?
<StevenK> wallyworld_: It's only because I've gotten annoyed at this structsub branch
<wallyworld_> sure :-)
<timrc> Is there a way to give read-only access to a private ppa Launchpad pages?
<timrc> It seems that if I subscribe a team or user to the PPA, they're able to download / install packages from the PPA but are unable to view its Launchpad pages
<bigjools> that's the intention
<bigjools> there is no r/o access to the pages
<timrc> bigjools, I figured as much which turns out to be less than ideal for some of our projects
<bigjools> what is your use case?
<timrc> bigjools, We have projects that are used as baselines for other projects, so we restrict upload rights more severely than we otherwise would.  People that are associated with the project, but not entitled to upload packages, would still like to be able to view the PPA pages
<bigjools> timrc: ok it sounds like more work around disclosure. you could chat to the purple guys about it
<timrc> bigjools, I've just made a note of the deficiency for now.  I'll let smagoun push the issue if he cares too ;)
<bigjools> ok :)
<timrc> I was just confirming that it wasn't possible to do
<StevenK> I don't think archive privacy is on our roadmap
<timrc> Maybe I'll just suggest we use subscriptions and grep-dctrl or something for now
<StevenK> wallyworld_: I have an MP up for you, just waiting for the diff
<wallyworld_> ok
 * wallyworld_ taps fingers, waiting....
<wallyworld_> StevenK: i think your branch is stuck
<StevenK> Grr
<wallyworld_> StevenK: and you can't delete it. i usually have to push to a new branch and start again
<StevenK> Hm, didn't we fix branch deletion?
<wallyworld_> StevenK: not that i've noticed
<wallyworld_> it didn't work for me yesterday
<StevenK> :-(
<wallyworld_> when i had a stuck branch. i was going to look at doing something when on maintenance
<wallyworld_> i think a branchrevison trigger is at fault
<StevenK> Like deleting the branch scanner out of disgust
<wallyworld_> so branchrevision must die or at least be severely spoken to
<wgrant> It's not a trigger
<wgrant> Well
<wgrant> It's the ON DELETE CASCADE fk
<wgrant> So deleting the branch tries to delete a 120k BranchRevision rows
<wgrant> which takes more than a few seconds
<wgrant> == boom
<wallyworld_> i used the term trigger loosely :-)
<wgrant> Easy fix is to increase the timeout
<StevenK> stevenk@carob:/srv/launchpad.net-logs/production/ackee/bzrsyncd$ grep '~stevenk' scan_branches.log scan_branches.log-20120720
<StevenK> stevenk@carob:/srv/launchpad.net-logs/production/ackee/bzrsyncd$
<wgrant> StevenK: You fail at celery
<StevenK> Oh, it logs elsewhere now?
<wgrant> scan_branches.py logs to scan_branches.log
<wgrant> celery does not
<wgrant> It logs to celeryd-SOMETHING.log
<wgrant> SOMETHING is job or branch_job or something like that
<StevenK> Two scan branch jobs
<StevenK> ?
<wgrant> I'd expect two, indeed
<StevenK> Yeah, they timed out
<StevenK> I think branch deletion actually worked
<StevenK> wallyworld_: https://code.launchpad.net/~stevenk/launchpad/sharingservice-rasj-miss/+merge/115874
<wallyworld_> diff there now
<StevenK> wallyworld_: It's a different MP. Different branch, too
<StevenK> I pushed, and waited for Branch:+index to show the revisions before proposing
<wallyworld_> yeah, found it
<wallyworld_> pita having to wait to file a mp
 * StevenK puts up a branch with a DB patch that does "ALTER TABLE BranchRevision SET SCHEMA todrop;" for crimes against humanity
<wgrant> StevenK: That one'll surely fail to scan :)
<StevenK> Haha
<StevenK> Irony is lost on the branch scanner
<wallyworld_> StevenK: test_getPeopleWithoutAccess_bugs - i'd prefer to have a separate test for branches with a common _assert_getPeopleWithoutAccess method to make the pattern used for the other tests in the module
<StevenK> wallyworld_: I renamed it. So it isn't that :-P
<wallyworld_> StevenK: sure, but it's doing bugs and branches in the one tests and is different to how the other tests work
<StevenK> wallyworld_: This branch has already completly failed to scan, and you want me to push it again -- tempt fate, much? :-)
<wallyworld_> so i'd prefer a bug and branch "stub" which call the _assertXXXX bit
<wallyworld_> it's only the initiall push that is problematic
<wallyworld_> after that it scans fine each time i've found
<wallyworld_> creating a mp before the first scan borks for some reason
<wallyworld_> my request to change ensures that each test only does one thing
<wallyworld_> but uses a bit of common code
<StevenK> wallyworld_: Right, pushing that up now.
<wallyworld_> thank you
<wallyworld_> StevenK: r=me, thanks
<StevenK> wallyworld_: Yeah, I saw, thanks :-)
 * StevenK tosses at ec2 and prepares to find some lunch
<StevenK> wgrant: I'm bashing my head against this branch. http://pastebin.ubuntu.com/1101291/ is the changes I've made since, following your suggestion of doing the filtering earlier
<wgrant> StevenK: Rather than finding the subscribers and then removing the subscribers that are unauthorized, perhaps just find the subscribers that are authorized
<StevenK> wgrant: That's what I'm attempting
<wgrant> +    def forbidden_subscribers(self):
<StevenK> wgrant: That's for get_also_notified_subscribers() benefit
<wgrant> StevenK: So
<wgrant> I'd do something that works
<wgrant> And that also isn't horrible
<wgrant> If it works and isn't horrible, it's probably good to land
<StevenK> You said that about the last branch too
<wgrant> It didn't work :)
<StevenK> And look how that turned out :-P
<wgrant> If you can't work it out, I'll sort it out on Monday.
<cjwatson> Do I need [testfix] to land a rollback branch at the moment?  It's not related to the current buildbot failure, but PQM seems to be ignoring me.
<cjwatson> Ah, there we go, finally got a mail back
<cjwatson> So then the question is: is it legitimate to use [testfix] for a rollback that's not related to the buildbot failure, or is that Evil Bad and Wrong?
<wgrant> cjwatson: It's Evil and Wrong, but it's only particularly Bad if it's something that would otherwise need QA
<wgrant> cjwatson: It's still preferable to force the build first in this case.
<wgrant> Rather than using [testfix]
<cjwatson> wgrant: Because this is a transient failure?
<wgrant> cjwatson: Right
<cjwatson> OK, done
<cjwatson> buildbot failing again, same celery failure
<wgrant> Hm
<wgrant> That's the third time today
<wgrant> I wonder if there was a relevant change
<wgrant>   [r=benji][bug=1015667] run the Celery task RunMissingReady with
<wgrant>   	ignore_result=True
<wgrant> That's a bit sus
<cjwatson> My thought exactly
<cjwatson> Mother-in-law phoned or I'd have beat you to it :)
<cjwatson> *beaten
<cjwatson> In fact, that revision added the test which is now failing
<wgrant> Ah
<wgrant> It's just remarkably similar to the one that has been failing spuriously for months
<wgrant> Ah
<wgrant> Similar to the one that was failing spuriously for months until I disabled it a month ago
<sinzui> barry: wgrant found the origin of the bug you are getting. If it is true that there is a deactivated project that the bug affects, We want to try to delete the task (possibly by temporarily) reactivating it
<barry> sinzui: i don't understand what you're saying but i'm glad wgrant found the problem ;)
<sinzui> The comment is sent to Lp, lp select a bugtasks, then notices you don't have access to one of them, so sends a 404 that you see as an error
<barry> ah
<sinzui> Well I just looked in the qastaging db and there is nothing extra
<sinzui> The data is stale
 * sinzui tries staging
<sinzui> and staging is also the same
* bac changed the topic of #launchpad-dev to: http:/â/âdev.launchpad.net/â | On call reviewer: bac | Firefighting: - | Critical bugs: 4.0*10^2
* bac changed the topic of #launchpad-dev to: http://dev.launchpad.net/â¹ | On call reviewer: bac | Firefighting: - | Critical bugs: 4.0*10^2
<sinzui> barry I reactivated acton. I think you can report bugs
<sinzui> I think we need to decide if we want to delete the tasks or keep the project active
<wgrant> sinzui: I was able to reproduce it with my unprivileged account on that bug on qastaging
<sinzui> I see
<czajkowski> mpt: you have way too much time on your hands at times :)
<mpt> czajkowski, hey, it's Friday afternoon
<czajkowski> am just laughing at your email
<czajkowski> mpt: also where is my cookie!
<nigelb> I think you get cookies for fixing bugs. oh wait, that's a badge.
<nigelb> "Fixed an MPT bug" badge.
<mpt> czajkowski, sorry, I just circumnavigated the office with the cookie jar but couldn't find you
<czajkowski> mpt: bah!
<czajkowski> mpt: next one you file is going invalid just for that ;p
<mpt> nigelb, almost as good as the coveted [ MPT APPROVED ] badge
<nigelb> mpt: Ooh. I haven't gotten that one yet.
<czajkowski> nobody is ever going to get that
<jml> not even mpt
<mpt> Word.
<jml> bug 1
<_mup_> Bug #1: Microsoft has a majority market share <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <LibreOffice Productivity Suite:New> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New for brunovam> <Linux Mint:In Progress> <The Linux OS Project:In Progress> <met
<mgz> you assigning that to yourself jml?
<jml> mgz: no, I'm just abusing the channel to find out the name of your bug bot
<mgz> what, you mean you're to busy to fix that this weekend?
<jml> mgz: yeah. you know.
<jml> mgz: furniture to assemble, household budgets to do, stacks of code reviews and one or two free software projects that I intend to crush.
<cjwatson> Why is PQM still in testfix mode?  http://lpbuildbot.canonical.com/waterfall shows the last devel build was successful.
<cjwatson> Is it stuck because of db-devel?
<wgrant> cjwatson: Yes, PQM rejects everything when either builder is red, because a failure on any builder is a situation that is meant to be resolved as top priority.
<wgrant> Unfortunately, people tend to just ignore them instead, it seems :)
<wgrant> I forced it a few minutes ago
<wgrant> Failed for 16 hours, quite impressive
#launchpad-dev 2013-07-15
<StevenK> wgrant: http://pastebin.ubuntu.com/5876192/
<wgrant> StevenK: You generate different code depending on the backend?
<StevenK> wgrant: Sadly, I have to.
<StevenK>         # DISTINCT ON is not supported in sqlite3, and GROUP BY with all fields
<StevenK>         # is not supported in psql.
<wgrant> psql is a command-line client; I think you mean PostgreSQL or postgres :)
<wgrant> But sounds reasonable
<wgrant> Does it work?
<StevenK> wgrant: Doesn't that pastebin prove it does?
<wgrant> StevenK: No, the tests are inadequate, I suspect
<wgrant> eg. the SQLite implementation doesn't have an ORDER BY
<StevenK> wgrant: That's on purpose.
<wgrant> Oh
<StevenK> If I add a ORDER BY, Django adds every field to the GROUP BY and sqlite returns everything
<wgrant> How does it work, then?
<StevenK> wgrant: TBH, I was happy enough to find a query that sqlite3 actually liked and returned the right data.
<wgrant> Sure
<wgrant> And we shouldn't write queries based on what Django's ORM supports -- we should write queries that work.
<wgrant> Then work out how to make Django like them
<wgrant> The sqlite query at present only works by accident
<StevenK> GROUP BY is a terrible hack, because Django wants everyone to use their aggregation stuff
<wgrant> Sure
<wgrant> IDGAF about Django atm
<wgrant> Work out SQL that does what you want
<wgrant> Then Djangofy it
<wgrant> Possibly using lots of quotes...
<wgrant> Working > Djangoy
<StevenK> wgrant: I hit IDGAF about Django on about Tuesday of last week
<StevenK> Seriously considered Pylons
<wgrant> s/Pylons/Pyramid/
<StevenK> That
<wgrant> Would have been a much more sensible choice, but our deployment story for that is, I believe weaker.
<wgrant> (I argued for it at the time :P)
<StevenK> I remember.
<StevenK> We got lifeless'd.
<lifeless> are you passive-aggressive trolling ? :)
<StevenK> wgrant: http://pastebin.ubuntu.com/5876270/
<wgrant> StevenK: Hm?
<StevenK> lifeless: A little, this exercise has moved my opinion of Django's ORM from hate to outright loathing
<wgrant> I'm not sure why you're not just bypassing most of it here
<wgrant> You're not doing any complex object-related queries
<wgrant> SQL would probably work just as well, as we do sometimes in Storm
<StevenK> The filtering stuff I'm doing for object operation and actor works well enough
<StevenK> It's just distinct that makes me want to stab things
 * StevenK ponders being evil, grabbing the query string, appending ORDER BY and then executing it
<StevenK> Oh, blah.
<StevenK> If you ask for a raw query from a QuerySet, it quotes EVERYTHING *except* string data that you're asking the database for.
<StevenK> wgrant: I have convinced sqlite and Django to add ORDER BY
<StevenK> SELECT "auditor_auditor"."id", "auditor_auditor"."date", "auditor_auditor"."operation", "auditor_auditor"."object", "auditor_auditor"."actor", "auditor_auditor"."comment", "auditor_auditor"."details" FROM "auditor_auditor" WHERE ("auditor_auditor"."object" IN (lp-development:PackageUpload:16) AND "auditor_auditor"."operation" IN (packageupload-accepted, packageupload-rejected)) GROUP BY (object), (operation) ORDER BY object, operation, -date
<wgrant> StevenK: Does that work?
<wgrant> eg. are the operations performed in the right order?
 * wgrant gone for a while
<StevenK> wgrant: So I'm not certain why the sqlite test works without the ORDER BY, but it also works with it.
<StevenK> (And the code that actually injects ORDER BY so that Django doesn't just go ahead and add every field to the GROUP BY is truly hideous.)
<wgrant> StevenK: There could be a flaw in the test, or perhaps it works by accident due to SQLite implicitly sorting the right way
<wgrant> Does the behaviour become obviously incorrect if you invert the sort?
<wgrant> And does the auditor test suite run across both backends?
<StevenK> No
<StevenK> wgrant: Hm, if I switch from -date to date in the ORDER BY it doesn't fail
<wgrant> Right, so it's probably executing in the wrong order
<wgrant> An explain will confir
<StevenK> wgrant: EXPLAIN in sqlite is ... odd
<wgrant> It's different from PostgreSQL's, yes
<StevenK> wgrant: http://pastebin.ubuntu.com/5879198/
<wgrant> StevenK: Oh, you probably want EXPLAIN QUERY PLAN
<wgrant> Rather than the underlying VM instructions
<StevenK> Oh
<wgrant> But you can tell from that EXPLAIN that it's doing the sort at the end
<wgrant> You may need to use a subselect for sqlite
<StevenK> EXPLAIN QUERY PLAN is pretty useless too
<wgrant> What does it say?
<StevenK> wgrant: http://pastebin.ubuntu.com/5879202/
#launchpad-dev 2013-07-16
<StevenK> wgrant: http://pastebin.ubuntu.com/5879216/ without the ORDER BY. It's very strange.
<wgrant> StevenK: How's it strange?
<StevenK> I guess I'm used to postgres returning unordered data with no ORDER BY, this is always ordered
<wgrant> It's not ordered or unordered
<wgrant> It's ordered by however it happens to come out
<wgrant> So, I would use a subselect to be simple and sure
<StevenK> wgrant: So SELECT * FROM (blah..) ORDER BY ... ?
<wgrant> StevenK: Other way around
<wgrant> You need to do the sort on the inside, group on the outside
<wgrant> You don't care about the order of the groups
<wgrant> You care about the selection of the group result rows
<StevenK> wgrant: http://pastebin.ubuntu.com/5879285/
<wgrant> StevenK: That looks substantially more sensible and likely to work
<StevenK> wgrant: But they're both the same!
<wgrant> ... indeed
<wgrant> I guess it's not at all surprising :)
<wgrant> GROUP BY with extra columns isn't meant to be permitted
<wgrant> Since it doesn't have an obvious meaning
<wgrant> StevenK: Rework to use a subquery with GROUP BY and MAX(date), then join the extra columns in the outer query?
<wgrant> That has to work
<wgrant> Probably slow, but we have a proper solution for prod
<StevenK> wgrant: SELECTing what, though?
<StevenK> I need to pull out at least one extra column, 'id'
<wgrant> StevenK: Why?
<wgrant> StevenK: Select the GROUP BY cols and MAX(date)
<wgrant> Then join in the outer query, filtering by the GROUP BY cols and date
<StevenK> wgrant: http://pastebin.ubuntu.com/5879591/
<wgrant> dat caps
<wgrant> Why left join?
<wgrant> And that join is the wrong way around
<StevenK> wgrant: I was guessing
<wgrant> And it'll return duplicates... possibly best to do a second group by on the outer layer, I guess
<wgrant> Never guess :)
<wgrant> This is SQL
<wgrant> Not some other thing that you might have to guess about :)
<wgrant> LEFT JOIN has a very specific purpoise
<StevenK> Yeah
<StevenK> It looks right with JOIN
<wgrant> I'd hope so
<wgrant> Given that it's what you want!
<StevenK> wgrant: http://pastebin.ubuntu.com/5879605/
<wgrant> StevenK: I'd add a further GROUP BY on the outside to eliminate dupes with the same key and date
<StevenK> wgrant: http://pastebin.ubuntu.com/5879610/
<StevenK> wgrant: You're happy with that query and I should continue trying to get Django to actually use it?
<wgrant> StevenK: That sounds reasonable
<StevenK> Which might involve swearing, throwing screwdrivers at the wall, and if I get really desperate -- gin
<wgrant> Or ""
<StevenK> Not quite that simple, since I'd like to deal with the case where only one of object or operation were specified
#launchpad-dev 2013-07-18
<StevenK> wgrant: Finally, but the code is *HORRIBLE*
<StevenK> wgrant: http://pastebin.ubuntu.com/5886428/
<wgrant> StevenK: That looks quite reasonable.
<StevenK> wgrant: I haven't shown you the auditor code yet
<wgrant> I have read approximately two lines of auditor code ever
<StevenK> wgrant: http://pastebin.ubuntu.com/5886439/
<wgrant> Why are OBJ and OPER in ALLCAPS, and why is the SQL not formatted? :)
<wgrant> Oh
<wgrant> They're replaced, I see...
<wgrant> Are you sure you can't do something more sensible?
<StevenK> I'm not sure, no.
<StevenK> wgrant: A quick googling shows even more hideous solutions, so it works
<StevenK> wgrant: So I should release auditor{,fixture,client} and push my changes for export-pu-auditor?
<wgrant> StevenK: How is the test situation?
<StevenK> wgrant: http://pastebin.ubuntu.com/5886563/
<wgrant> === modified file 'lib/lp/services/auditor/server.py'
<wgrant> 'distinct' might be better called 'latest'
<StevenK> wgrant: Blah, that requires changes to auditor{,fixture,client}
<wgrant> Sure
<wgrant> But 'distinct' really doesn't say anything at all
<StevenK> wgrant: So it does address your concerns about a test?
<wgrant> StevenK: I'm more wondering about testing the divergent postgresql and sqlite3 implementations
<StevenK> I'm not sure how to test the postgresql implementation
<StevenK> Requires having postgres around and configured so $USER can create a db
<StevenK> wgrant: Not sure if I can reach into Django's innards and convince it to change the default db. Or if I can configure a postgres db in runtests.py with things like 'user': environ['USER']
<wgrant> I'm not sure
<wgrant> But having a way to test that the production code works might be nice...
<StevenK> I've tested it by hand, at least, but I know that isn't what you mean.
<pindonga> hi,  I'm trying to troubleshoot an issue with a user getting a different openid from launchpad than from Ubuntu SSO
<pindonga> is it possible to check if this user has merged accounts recently?
<pindonga> is this the right channel for that?
#launchpad-dev 2013-07-19
<cjohnston> Is there a way to automatically clear launchpadlib cache when there is an issue with it (outside of a cron)?
<cjohnston> I believe it is messed up due to a timeout from LP
<lifeless> rm -rf ?
<cjohnston> lifeless: I get that... We have a cron that runs for reports.qa.u.c... every once in a while it gets a timeout from LP, and then we stop getting updates due to I guess a corruption.. Without having to login to determine if this is the problem, or setup an rm cron, I was wondering if there was a fix
<lifeless> have you filed a bug about the corruption?
<cjohnston> no. against launchpadlib?
<lifeless> yes
<cjohnston> ack. will do
<cjohnston> lifeless: bug #1014621 is the same traceback
<_mup_> Bug #1014621: cache corruption on api.launchpad.net,1.0,-application,vnd.sun.wadl+xml,fc06437932a618a4b30d0d0417f9234c <api> <launchpadlib> <launchpadlib :Triaged> <https://launchpad.net/bugs/1014621>
<StevenK> wgrant: Based on my searching, Django doesn't seem to support switching databases for a test case :-(
#launchpad-dev 2013-07-21
<Bert_2> Hi, do you guys know there's some spamming going on on launchpad and what do I do when someone reopens a question by spamming?
<Bert_2> Hi, do you guys know there's some spamming going on on launchpad and what do I do when someone reopens a question by spamming?
#launchpad-dev 2014-07-14
<cjwatson> WTF sampledata
<cjwatson> wgrant: There's a sourcepackagepublishinghistory row in sampledata with a distroseries whose distribution doesn't match that of its archive.  Removing the distroseries checks (for non-PPAs) causes the careful-publishing tests to try to republish it.  Should I just make the relevant tests mark that series OBSOLETE and go lalalala, or should I maybe add a distribution condition to getPending*Publications as a guard against this?
<cjwatson> I suspect cleaning up that bit of sampledata would be an Aegean-stables exercise
<wgrant> cjwatson: I only fixed the publication factories to prevent that situation last week. Fixing a single row shouldn't be too bad.
<wgrant> There was a fair bit of test fallout from the code changes, but it wasn't intractable.
<wgrant> Let's see which pub it is.
<cjwatson> 24 was the one I found
<cjwatson> I'm more wondering whether this might be indicating something I need to guard against on prod
<wgrant> It is indeed the only one.
<wgrant> Hmmmmm.
<wgrant> Possibly.
<wgrant> I think I only ever did it on staging.
<wgrant> But I was playing around with that sort of bug before I had code access, so I forget.
<wgrant> Worth checking on staging, I suppose.
<wgrant> We should fix the data rather than the code.
<wgrant> Since the data fix is quite obvious.
<wgrant> (except for sampledata)
<cjwatson> Is it?  status = DELETED or something, I guess
<wgrant> No, obliterate the rows entirely. They would never have moved out of PENDING.
<wgrant> So they are invalid and there must be no artifacts on disk.
<wgrant> Hm, so it's mozilla-firefox in ubuntu-test primary.
<wgrant> I don't imagine there would be much direct fallout from that.
<wgrant> There are four broken SPPHs on prod.
<cjwatson> Yeah, I'd just found those on DF
<wgrant> One of them is mine, the other three aren't.
<wgrant> Oh
<wgrant> Two of them are mine, sorry.
<wgrant> And the other two are in some other PPA.
<wgrant> NCommander's.
<wgrant> And there shouldn't be any binaries, since they could have only have been copied from debian/primary, but let's check.
<cjwatson> I'm just checking that now on DF
<cjwatson> Rather slower query ...
<cjwatson> 0 rows
<wgrant> BPPH is several times larger, and there's an extra join.
<wgrant> Right
<wgrant>    id   | archive | distroseries | status | datepublished |        dateremoved
<wgrant> --------+---------+--------------+--------+---------------+----------------------------
<cjwatson> So these were Debian SPPHs that were erroneously republished in Ubuntu PPAs rather than copied properly?
<wgrant>  483272 |      71 |           50 |      4 |               | 2009-01-26 14:03:21.732962
<wgrant>  484547 |    2919 |           49 |      4 |               | 2009-06-13 12:47:14.889241
<wgrant>  484546 |    2919 |           50 |      4 |               | 2009-06-13 12:47:14.889241
<wgrant>  539825 |    7823 |           50 |      4 |               | 2009-07-31 06:13:14.751255
<wgrant> Does p-d-r really not restrict by distro?
<cjwatson> Anyway, DELETED ones aren't a problem for this
<cjwatson> We might want to clean them up anyway, but the publisher isn't going to try to republish them
<wgrant> Right, we use the source archive's series in +copy-packages, and the copier used to not validate series.
<cjwatson> Oh, that
<wgrant> The copier now validates that the series exists in the target, and also the underlying model methods will refuse such madness as of last week.
<cjwatson> OK, I guess I fix the sampledata row and run all archivepublisher and soyuz tests or something, then
<wgrant> so I think we just DELETE FROM sourcepackagepublishinghistory WHERE id IN (483272, 484547, 484546, 539825); on prod and probably correct the series in sampledata.
<wgrant> https://bugs.launchpad.net/launchpad/+bug/320398 ftr
<_mup_> Bug #320398: Copy UI uses source distribution series names rather than target distribution series names <lp-soyuz> <package-copies> <phone-rtm> <ppa> <Launchpad itself:Triaged> <https://launchpad.net/bugs/320398>
<cjwatson> Yeah, saw that from the phone-rtm list the other day
<wgrant> I never thought it would be on me to clean my evil test data up :P
<cjwatson> The more I work on Launchpad the more I understand why people hate and despise sampledata
<wgrant> Uhuh.
<wgrant> It wouldn't be so bad if the sampledata was a vaguely representative sample of anything.
<wgrant> But, at least in the Soyuz case, it's mostly invalid crap.
<cjwatson> I'm just going to try removing the two broken sampledata rows (one source, one binary).  Let's see who rusts first.
<wgrant> Yep.
<cjwatson> wgrant: Would you mind re-reviewing https://code.launchpad.net/~cjwatson/launchpad/optimise-publish-a/+merge/226312 ?  I *think* I have it right now, but it took a while to sort out the details.
#launchpad-dev 2014-07-18
<xnox> I don't think i've setup ubuntu distribution correctly in my lxc launchpad
<xnox> since it has no suites & thus no arches.
<cjwatson> xnox: https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally
<wgrant> xnox: The sampledata is... somewhat corrupt, ancient, incomplete, and otherwise undesirable.
<cjwatson> Particularly soyuz-sampledata-setup
<wgrant> xnox: There's utilities/soyuz-sampledata-setup.py to make it sane.
<wgrant> Right, that.
<cjwatson> That should be pretty much up to date - I followed that procedure when testing livefs
<cjwatson> (and updated it slightly)
<xnox> hm. so adding more debug it looks like it knows about releases/suites/pockets
<xnox> it's just i don't have anything published =)
<xnox> ./scripts/publish-distro.py -vv -C
<cjwatson> FWIW, while there certainly are cases where I've wanted to actually run a local Launchpad instance and it's good not to have to do it in a rush, for the things I've been doing I've found it's better not to have it be my first resort for testing
<cjwatson> (I couldn't have done all the livefs integration without it, but for the sorts of things I do that was the first time in about a year)
<xnox> right. let me fake it. So i'm just copying a mock archive into /var/tmp/archive/ubuntu/ (e.g. an rsync of a smallish ppa)
<wgrant> xnox: What're you trying to do?
<xnox> which should be enough to run ./cronscripts/generate-contents-files.py
<xnox> and it looks like it is =)
<cjwatson> bin/test -vvct generate_contents_files
<cjwatson> is at least slightly useful and you'll probably need to extend it anyway :)
<xnox> yeah, i have run those already as well. But e.g. wanted to be able to run stuff by hand as well.
<cjwatson> Fair enough
<xnox> cause e.g. i want it to be a larges repository to benchmark improvements, if i can make any =)
<cjwatson> Yeah, that might be worthwhile for the try-not-to-compute-everything-twice bug
<cjwatson> BTW http://people.canonical.com/~ubuntu-archive/apt-mirror.cgi/1405620000/dists/utopic/main/binary-i386/Packages.bz2 (etc.) is now a link
<cjwatson> *now a thing
<xnox> =)))) nice
<cjwatson> Hopefully it's not a complete security nightmare ... haven't hooked up any of the magic pool traversal stuff because don't have uwsgi etc. there and didn't want this to be an epic journey
<cjwatson> So I cloned-and-hacked apt-mirror-web into a trivial CGI script instead
<cjwatson> It'll 404 practically anything it doesn't like
<xnox> cause go is for kids ;)
<cjwatson> I only care about it for derive-distribution anyway :-)
<xnox> right.
<xnox> where is source for it?
<cjwatson> hm, let me push it to github or something
<cjwatson> *mutter*
<xnox> i've checked people.canonical.com:~ubuntu-archive/public_html and it's obviously not there any more =)
<cjwatson> xnox: https://github.com/cjwatson/apt-mirror/tree/ubuntu-archive
<cjwatson> real web developers please try not to laugh too loudly
 * wgrant plays pin the security hole on the CGI script.
<cjwatson> Grr, bz2.BZ2File in 2.7 doesn't accept file objects, seriously?
<cjwatson> hate Python 2
 * xnox targeted python3 only....
<cjwatson> this is for derive-distribution; don't want to rely on having your fancy new launchpadlib on my system by next week
<xnox> =/
<xnox> yeah, i think everything is in trusty to run apt-mirror, python3, python3-cherrypy, python3-uwsgi et.al.
<xnox> doesn't use launchpadlib direct, just constructs restful url to download things.
 * cjwatson considers derive-distribution's package selection
<cjwatson> It occurs to me: I've got historical Sources/Packages, and I can get the seed output at a given time as well; I think the right answer is to do a time-travelled germinate run against a nominated list of seeds + optional extra packages
<cjwatson> And then copy the closure of resulting sources
<cjwatson> Possibly germinate on all relevant architectures, in fact, but that's fine, I have the technology
<cjwatson> I can even make use of the never-used facility in germinate to plug in an alternative archive representation, which is basically perfect for this
<cjwatson> Maybe double-check against actual image contents too
<cjwatson> bzr's date revisionspec is irritating - no way to get the state of the branch *at* a particular date, you can only ask for the first revision after that date, and that breaks if the date in question is after the last commit on the branch
<cjwatson> Will have to figure it out by hand I guess
<james_w`> cjwatson: can you combine with before: to get the behaviour that you want?
<james_w`> it might get you the state at that date, but probably changes the failure mode to when the first revision is after the date
<cjwatson> james_w`: I've already done it differently, but before:date:... was where I started
<cjwatson> It evidently tries to resolve date: first, fails to do so, and then gives up on the whole revisionspec; it makes sense in a piecewise way but isn't very convenient all round
<cjwatson> I just did a crude "bzr log" parser instead and will no doubt pay the karma cost at some point.  This doesn't have to be fast or especially maintainable
#launchpad-dev 2015-07-13
<blr> wgrant: after the is merged, will it run, or is there a flag to set?
<wgrant> blr: garbo jobs are run automatically.
<wgrant> blr: Some have the ability to be enabled or disabled by feature flag, but they must implement that themselves.
<blr> wgrant: ok, just uncertain about the performance profile of this job, but I see there are others that operate over the entire product collection.
<wgrant> blr: It is not problematic. We have garbo jobs running over collections several orders of magnitude larger.
<blr> ok good to know :)
<wgrant> blr: Care to land that branch so we can deploy this afternoon?
<wgrant> (if you tried before it probably got caught in the testfix)
<blr> wgrant: sure, was just waiting for buildbot
<wgrant> blr: Thanks.
<lifeless> wgrant: webhooks; nice.
<wgrant> lifeless: That's the plan.
<wgrant> For Git pushes initially, but obviously once we have the infrastructure we'll look at expanding it.
<wgrant> Also investigating a general event system.
<wgrant> Conflict adding file lib/lp/services/webhooks.moved.moved.moved.moved.  Moved existing file to lib/lp/services/webhooks.moved.moved.moved.moved.moved.
<StevenK> Oh bzr <3
<lifeless> wgrant: you have one already - the public wabbit server
<lifeless> wgrant: (aren't webhooks just a particular client of that ?)
<wgrant> lifeless: txlongpoll would be a client of the RabbitMQ-based event system.
<wgrant> lifeless: txlongpoll would need a complete redesign to be the centre of it all.
<lifeless> sure
<lifeless> be nice to have it outside the squishy core is all ;)
<wgrant> (and the idea would be that events would also be persisted, such that activity streams can be presented, and eg. bug notifications can be responses to events)
<wgrant> Oh, certainly. I consider this webhook implementation a production-ready prototype that will hopefully be obsoleted by an event system designed with things we've learned from this.
<lifeless> wgrant: sounds like apache thingy. Uhm, linkedin's thing
<lifeless> distributed persistent log
<wgrant> Kafka?
<lifeless> yes
<rpadovani> Hey all, I'm lurking here for some months so it's time to introduce myself:
<rpadovani> I'm Riccardo, 22 years old Computer Science (2nd year) student from Italy.
<rpadovani> I contribute to Ubuntu for 3 years, beginning with my Loco and then writing code for Ubuntu Touch (core apps and webbrowser, mainly).
<rpadovani> I'm quite involved (I also joined two sprints) and I have a lot of fun. Since the start of the year I'm thinking to contribute to LP, a project I really like (definitely the best bug tracker on the internet).
<rpadovani> The only problem: I don't know Python :-P
<rpadovani> But since April I work for a startup and now they want I learn Python, so I'm studyng on Coursera and CodeAcademy and I already completed the basic tutorial on the Python website.
<rpadovani> So, what's better way to learn something new than contributing to an opensource project?
<rpadovani> (Sorry for the wall of text, now questions are coming)
<rpadovani> Do you think I could be useful if do some patches, starting from bitesize bugs?
<rpadovani> Probably at start I'll do silly patches and I'll have some silly questions.
<cjwatson> Sure, we aren't short of them
<rpadovani> So, do you think I could be useful? Let me be clear: I want to help, not to be a burden for the project.
<rpadovani> cjwatson, great :-) Do you think with a good commitment I can contribute to LP or it's better to start from something simpler?
<cjwatson> Starting with that attitude is already a pretty good sign.
<cjwatson> IMO the biggest skill (other than Python, general programming, etc.) you need to contribute effectively to LP is the ability to navigate a large codebase and not be scared by it.
<cjwatson> It's pretty regularly laid out, so once you get a feel for it it isn't nearly as hard to find things as the raw size of the tree would suggest.
<rpadovani> let's try then - I did a couple of patches for oxide, that is quite big
<cjwatson> And the test suite coverage is for the most part excellent, which helps.
<rpadovani> great to hear that
<rpadovani> I'm creating the LXC container, I'll back when it will finish to download all the internet
<cjwatson> Heh, it does that.
<cjwatson> To start with I'd suggest that you ask us about bugs you're thinking about attacking, and we can advise you if they have any hidden horribleness
<cjwatson> It's been a while since we had non-trivial contributions from outside the core team, so very happy to see somebody interested :)
<rpadovani> you're not in maintenance mode anymore, so I hope also someone more skilled will come out :-)
<cjwatson> Welcome, anyway
<cjwatson> Is there a particular bit of the site that attracts your attention most?
<rpadovani> tbh dunno yet. As I said, I really like bug trackers, so I'll take a look to bug to them - the other thing I think could be pretty improved is integration with bzr, so I'll look also to bugs about code navigations and so
<rpadovani> but, whereever I could be useful I'll try to drop a bit :-)
<cjwatson> OK, fair enogh
<rpadovani> (bit like in 1/8 of byte)
<cjwatson> *enough
<rpadovani> (my english sucks btw)
<rpadovani> cjwatson, do you think bug #194257 could be a good start? Seems very trivial, but could be a good way to take a look to lp tree and to take a look to the workflow
<mup> Bug #194257: "Recently imported branches" doesn't just list recent ones <lp-code> <trivial> <ui> <Launchpad itself:Triaged> <https://launchpad.net/bugs/194257>
<cjwatson> rpadovani: Not too hard in itself, but nearby classes in the code include "Recently registered branches" and "Recently changed branches".  What would you do with those, since the same change doesn't seem like it would make sense there?
<rpadovani> cjwatson, well, I think it should be evaluted case by case - if there is a check of some type on the datetime I leave the "Recently", otherwise I drop it
<cjwatson> rpadovani: I wonder if a better fix would be to make the code match the text, and impose some kind of cut-off on the list of branches shown
<cjwatson> rpadovani: My point is that neither "Registered branches" nor "Changed branches" makes sense as a page title.
<rpadovani> Okay, so probably is better to show branches that are newer than 1(?) month
<cjwatson> So maybe it would be better to add a filter so that all three of those really are recent.
<rpadovani> great, I'll try this approach then - again, should be easy?
<cjwatson> I'd probably go with something like 90 days, but bikeshed
<cjwatson> changed is fairly easy, but both registered and imported will take you into BranchCollection to add a filter method on date_created, which isn't trivial, and you'd have to figure out how to write tests.  I'd class this as relatively easy, but maybe not a first branch.
<cjwatson> (BranchCollection is notably obscure - I had to understand it all recently when implementing a parallel thing for Git)
<rpadovani> okay, thanks for info, so I take a look to other bugs and keep this for next weeks :-)
<cjwatson> https://bugs.launchpad.net/launchpad/+bug/892259 might not be too difficult
<mup> Bug #892259: Link to revision for merged MPs <trivial> <ui> <Launchpad itself:Triaged> <https://launchpad.net/bugs/892259>
<cjwatson> You'd have to learn about TAL, but that's going to come up pretty quickly anyway
<cjwatson> Or maybe https://bugs.launchpad.net/launchpad/+bug/530505
<mup> Bug #530505: branch_merge_proposal.createComment requires a subject, which is pointless <api> <lp-code> <trivial> <Launchpad itself:Triaged> <https://launchpad.net/bugs/530505>
<rpadovani> rocketfuel-setup is still working ZzZzZz
<cjwatson> Good news: you only have to do it once
<rpadovani> btw, I see it creates two directories, one for branches and one as shadow, so the workflow is bzr switch -b my_little_patch / bzr push lp:~rpadovani/lp-dev/my_little_patch / bzr checkout trunk ?
<cjwatson> I *cough* do things the old-fashioned way and just create a full new branch for everything
<blr> morning
<cjwatson> Though it's a bit slow since I have to do "utilities/link-external-sourcecode && make" in each new branch
<cjwatson> wgrant has a better workflow with "bzr switch" I think
<rpadovani> and you need a big hard disk :D
<cjwatson> We're pretty close to switching to git, though, so don't worry about optimising that too much
<blr> yes, git will make that irrelevant, for better or worse ;)
<cjwatson> morning blr
<blr> how was your weekend cjwatson?
<cjwatson> excellent, thanks, dinner party on Saturday
<cjwatson> much wine was had
<rpadovani> I switched from a directory for a branch to bzr switch a couple of months ago, it has some advantages
<rpadovani> blr o/
<blr> sounds pleasant!
<cjwatson> rpadovani: BTW the branch URL would be lp:~rpadovani/launchpad/my_little_patch, not .../lp-dev/...
<blr> hi rpadovani
<cjwatson> blr: hope you aren't snowed in or whatever Melbourne has been having
<blr> cjwatson: it has been bitterly cold here, but I think the snow has passed.. the frost isn't budging however.
<blr> cjwatson: did you hear wgrant was without power as well! :/
<cjwatson> I did
<cjwatson> still, on the upside, we don't live in Portland so http://www.newyorker.com/magazine/2015/07/20/the-really-big-one doesn't have to be immediately terrifying
<blr> yes, read that this morning... I was in the 1989 bay area quake, and that was terrifying enough...
<cjwatson> worst it ever gets here is basically the mild-flatulence level
<cjwatson> though my mother-in-law's house and garage are separated by a fault line, which is kind of interesting
<blr> seismologists seem to be discovering new faults in the south island fairly regularly.. not clear how we stand.
<cjwatson> today I came up with what I think is a workable plan for the gmail filterability project, although I need to try porting something non-trivial like bugs to BaseMailer to see how feasible it is
<blr> ok, a change in direction from your earlier plan, or along the same lines?
<cjwatson> same lines, just a little better fleshed out in that I found a plausible place to put the common code
<cjwatson> and confirmed that gmail actually lets you do reasonable searches/filters for that kind of thing
<cjwatson> looking back through old bugs, it looks like the former Code team developed BaseMailer, and then the tentative plan was for other teams to port stuff to it but mostly nobody ever got round to it
<cjwatson> so I think it's a decent direction
<cjwatson> sometimes I think my job title should be something more like Vinge's "programmer-archaeologist"
<blr> heh, I suppose that's not in the category of jobs that do not exist yet then :)
<rpadovani> I've all up and running \o/ I'm almost moved
<wgrant> cjwatson: I didn't realise BaseMailer was actually used that widely. That's convenient.
<rpadovani> cjwatson, so, I tried to write the patch to the bug you pointed me to. I hope it's good :-)
<rpadovani> https://code.launchpad.net/~rpadovani/launchpad/link-revision-merged-mp/+merge/264654
<blr> rpadovani: hopefully that points you in the right direction, let us know how you get on.
<blr> wgrant: potentially it does make sense to have a redirect on code/project/branch/revision/revno perhaps?
<cjwatson> I don't think a slow redirect through the webapp is going to improve people's lives, really.
<cjwatson> rpadovani: You should also figure out how to write and run a test for this.
<cjwatson> rpadovani: The process of doing that would likely have shown up your mistake here.
#launchpad-dev 2015-07-14
<mwhudson> did that garbo job for backfilling product.vcs finish yet?
<blr> mwhudson: it has yep
<mwhudson> awesome
<mwhudson> so as far as we know, we should be able to remove the special casing from the go tool?
<blr> mwhudson: I would imagine so, yes :)
<mwhudson> cool
 * mwhudson tries to find a go project on lp with >1 series
<mwhudson> oh poop
<mwhudson> go get launchpad.net/project/series/sub/directory is documented as working
<blr> mwhudson: sorry, don't follow...
<mwhudson> as is launchpad.net/~user/project/branch
<mwhudson> blr: basically you can feed a subdirectory of the branch to go get and it knows to get the whole branch and then only build the subdir
<mwhudson> i assume
<blr> mwhudson: is github special cases somehow to handle that as well?
<blr> s/cases/cased/
<mwhudson> yes, looks like it
<mwhudson> bah
<mwhudson> i guess i could cook up a re so the existing behaviour is preserved when there is trailing stuff
<mwhudson> maybe
<mwhudson> blr: can you link me to a project that uses git?
<blr> mwhudson: https://launchpad.net/~blr/+git/mushrooms-russia-history
<blr> oops
<blr> no that's no good
<mwhudson> that's like a junk branch?
<blr> https://launchpad.net/germinate ?
<blr> yes that one's git
<blr> mwhudson: right, not a project
<mwhudson> blr: i bet you haven't thought at all about what the go import path for a project hosted in git on lp should be :)
<mwhudson> blr: i just said this: https://github.com/golang/go/issues/11436#issuecomment-121124410
<blr> mwhudson: have I missed something? go-import should render for git projects too
<mwhudson> blr: for git projects too, but what about git repos that are not the repo for a project?
<mwhudson> i guess that's a rarer thing than the bzr case, given the nature of git
<blr> mwhudson: right, in this case you're only going to get the default repo for the project or series
<blr> mwhudson: how often is that used in practice? the sub-directory feature
<mwhudson> blr: i have no idea
<mwhudson> i suspect not often
<rpadovani> blr, cjwatson thannk you very much, I start to take a look at it :-) I'm not good (yet) at writing tests, so maybe it takes a bit. Thanks for pointing me to the right direction
<cjwatson> Sure.  Let us know if you need help.
<blr> mwhudson: is there anything more we can do on the launchpad side? I see russ cox replied.
<rpadovani> sorry to bother you, but I'm a bit lost writing the test. The general logic is ready and it does what I think it should do, but I have some problems forgint the href for the revision. Atm I've something like this: revision_link = '%s/revision/%s' % (target_identity, self.arbitrary_revisions[2]), where target_identity is bmp.merge_target.identity. It produces something similar to the final result: http://paste.ubuntu.com/11876524/ The
<rpadovani> problem is I need to replace the starting of the link with http://bazaar.launchpad.net and with http://git.launchpad.net. I don't understand if there is a way to have the link from some variable or I need to create it form the merge_target looking if it starts with lp:// or with ~
<rpadovani> here the code I wrote so far: http://paste.ubuntu.com/11876534/
<cjwatson> Neither of those is correct for the link.  You'll want to construct something based on self.context.getCodebrowseUrl().
<cjwatson> Something like self.context.getCodebrowseUrl("revision/%s" % quote_plus(revision)) should do the job for Bazaar.  For Git it'd be more like "%s/commit/?id=%s" % (self.context.getCodebrowseUrl(), quote_plus(revision))
<cjwatson> Oh, this is from a merge proposal, so self.context.merge_target rather than self.context there
<cjwatson> And in fact in the Git case it'll want to be self.context.target_git_repository, as GitRef.getCodebrowseUrl isn't appropriate here
<rpadovani> okay, thanks. How can I check if the test is running agains git or bzr?
<cjwatson> What class is this test in?
<rpadovani> or should I create two tests, one under class TestBranchMergeProposalMergedViewGit and the other under class TestBranchMergeProposalMergedViewBzr?
<rpadovani> TestBranchMergeProposalMergedViewMixin atm
<cjwatson> You *can* create two tests, but I would suggest instead adding a method to each of TestBranchMergeProposalMergedView{Bzr,Git} that returns the expected link text for a revision.  See how test_initial_values is implemented for comparison.
<cjwatson> Then the test in TestBranchMergeProposalMergedViewMixin can be generic.
<rpadovani> okay, thanks for the ideas, I take a look
<rpadovani> have to study now, I'll continue tonight or tomorrow, thanks for suggestions :-)
 * cjwatson glares at the swamp of bug notifications
<blr> morning
<mwhudson> blr: i don't think so, i have woken up with a cold and my brain is full of cotton wool but i'll try to follow up today at some point
<blr> mwhudson: ugh, sorry to hear that. no worries
<rpadovani> After I spent an afternoon studying about DFA and NFA and how to transform one in the other, it's time to write a test for my branch :D So, I was trying to forge the url where I should point the link to. cjwatson suggested me to the test under TestBranchMergeProposalMergedViewMixin, and then make two little helper functions under TestBranchMergeProposalMergedViewBzr and TestBranchMergeProposalMergedViewGit. Let's focus on the first
<rpadovani> one atm: so I created two versions of this helper function, but they don't work. Any hint on which I'm missing? I'm sure it's something very stupid...
<rpadovani> http://paste.ubuntu.com/11879809/
<blr> rpadovani: bmp does not have a 'context' attribute, I think you just want bmp.merge_source
<blr> rpadovani: rather than using pastebin, it might be better to make a merge proposal in the 'work in progress state'
<rpadovani> blr, well, I need to link to the target where the branch has been merged, so bmp.merge_target. But with bmp.merge_target I've only the name of the branch, not the address of the code (that is what I need)
<rpadovani> blr, ok, I'll do, then the commit log for the branch will be a mess, but if it's okay for you it is also for me :-)
<rpadovani> blr, okay, I pushed the test I wrote so far here: https://code.launchpad.net/~rpadovani/launchpad/link-revision-merged-mp/+merge/264654
<rpadovani> if you have some time to take a look thanks so much :-)
<rpadovani> meanwhile I take a look to the *real* code
<blr> thanks rpadovani, someone should be able to have a look today
<rpadovani> thanks to you guys for your time
<wgrant> blr: Could you have a look at https://bugs.launchpad.net/launchpad/+bug/1474592? Should be a quick fix in two templates or two views, plus a test for each.
<mup> Bug #1474592: golang_import_spec needs to check that the branch is accessible <branches> <privacy> <regression> <Launchpad itself:Triaged> <https://launchpad.net/bugs/1474592>
<blr> sure
<wgrant> The branch link on Product:+index and ProductSeries:+index is already guarded (possibly with a /required:launchpad.View, or possibly in the view itself)
<blr> wgrant: worth a browser test as well?
<cjwatson> Right, a view test would go in browser-land.
<cjwatson> (Unless it fits with something in stories and is too hard to put elsewhere, which sin I've committed a few times.)
<cjwatson> wgrant: Is there a way to add a person to a team in a test without triggering TestMailer.send to the added person?
<cjwatson> I have nefarious reasons.
<wgrant> cjwatson: Um, you could possibly fixture out the ITeamMembership subscribers, or create a TeamMembership manually and not fire the event.
<wgrant> TeamMembershipSet.new is illegal, but it works.
<cjwatson> Maybe a fixture is the right thing.
<cjwatson> Or maybe I should stop trying to test this in an existing doctest ...
 * beuno passes cjwatson a can of gasoline and a matchbook, winks and points at LP doctests
<cjwatson> beuno: Wouldn't be the first I've committed arson against
<cjwatson> (the first doctest, I should clarify!)
<cjwatson> (oh hi nice mr policeman)
<blr> trying not to grow the number of doc-tests at least
<beuno> I still have nightmares about 6 hour test suites, PQM and getting all levels of agree to pass at the same time
<cjwatson> It's not been that bad for a while, fortunately
<cjwatson> Thank goodness
<beuno> my last commits were around the time the test suite got pararellised, enough to have ended up with a 300 USD AWS bill because the tool didn't shut down instances properly, but not where the cloud bit was centralised
<cjwatson> When I started ec2test was still a thing but there'd been enough mistakes made for there to be prominent warnings to check that shutdown had worked
 * beuno shivers
#launchpad-dev 2015-07-15
<lifeless> beuno: parallel ftw :)
<blr> wgrant: better to add a new property for branch/repo view permission on the views, or just check in the method?
<wgrant> blr: Check in the method, or guard in the template using the "required" TALES namespace.
<blr> ah actually we already have user_branch_visible in the branch case
<wgrant> Hm, I wonder why.
<blr> but only on productseries
<blr> that seems a little odd
<blr> although maybe it is unreasonable to expect too much symmetry between product and productseries.
<wgrant> Feel free to increase symmetry where it makes sense.
<blr> wgrant: is the best way to restrict a branch by setting information_type=InformationType.USERDATA?
<blr> that appears to be what the tests in test_branchvisibility use, but it doesn't seem particularly clear.
<wgrant> blr: PRIVATESECURITY is technically the common case, but USERDATA probably works.
<blr> thanks, I think that's slightly more obvious
<blr> wgrant: buildbot is busy, but need to head out - will land that fix when I get back.
<blr> and yay for non-wonky comment mail.
<wgrant> blr: You can land it now.
<wgrant> It'll fail spuriously in a few minutes, though.
<blr> wgrant: oh I see, incoming failure :P
<cjwatson> Ah, reproduced the launchpad-buildd 131 crash.
<cjwatson> dscpath misunderstanding
<beuno> lifeless, indeed!   o/
<cjwatson> (launchpad-buildd 132 working its way through the system)
<cjwatson> wgrant: Do you know of a reason why BaseMailer specifically avoids producing a decent OOPS, and instead just does an ugly logger.warning thing?
<cjwatson> It has a comment saying "Don't want an entire stack trace, just some details", but no rationale for why that should be.
<cjwatson> I'm tempted to fix that when consolidating with bugs.
<stub> cjwatson: Quite possibly to avoid mail spam infinite loops, from when OOPSes were mailed out directly (or am I misremembering?)
<cjwatson> Could be.  At the moment, logger.warning turns into an OOPS in scripts (but in this case probably a rubbish one).
<wgrant> cjwatson: logger.warning only started doing that recently.
<cjwatson> self.logger.warning("send failed for %s, %s" % (email, e)) - right, let's substitute the entire message text of an e-mail into a clause
<wgrant> OOPSes were never mailed out directly, AFAIK.
<wgrant> But they were stored in the librarian.
<cjwatson> Ah, yes, where "recently" -> 5ya.  But more recent than the BaseMailer change.
<wgrant> Relatively recent in Launchpad history :)
<cjwatson> So I think OOPS with the message as an attachment, logger.info(request.oopsid)
<cjwatson> Not entirely sure how to get a suitable request though - we send mail from both the webapp and scripts
<cjwatson> Optional request passed to the BaseMailer constructor, maybe
<rpadovani> I was thinking... since in my branch I added a new property in the BranchMergeCandidateView class, maybe test should go in TestBranchMergeCandidateView instead of TestBranchMergeProposalMergedViewMixin?
<rpadovani> (wrt https://code.launchpad.net/~rpadovani/launchpad/link-revision-merged-mp/+merge/264654)
<cjwatson> You could justify putting a test in the former, though I'd still keep the latter test as well since you're changing the template too.
<cjwatson> Your link_to_branch_target_commit implementation is broken for git, BTW
<cjwatson> And indeed the current tests are broken too - missing getCodebrowseUrl in various places
<rpadovani> cjwatson, indeed, and how could I fix the test? I'm not able to understand how I can retrieve the git url... Also the bzr one is broken, cause it returnes the address of the repo without http://bazaar.launchpad.net at start....
<rpadovani> I tried some of the strategies you suggested me, but I'm not able to access to context, so I can't take git_repository or other usefuls properties
<cjwatson> I don't understand your comment about the context
<cjwatson> In the test, bmp is the merge proposal, which is the context for the view under test
<cjwatson> So bmp in the test is equivalent to self.context in the view
<cjwatson> Let me go write a bunch of inline comments :)
<rpadovani> thanks, I'm quite lost :-)
<rpadovani> if we will never meet, I will offer a couple of beers ;-)
<cjwatson> rpadovani: does that help now?
<rpadovani> cjwatson, definitely, I'll take care of them asap, thank you very much :-)
<cjwatson> No problem.
 * cjwatson decides he isn't likely to make much more progress on refactoring bug notifications today.  EOD time.
<blr> morning
<rpadovani> o/
#launchpad-dev 2015-07-16
<bigjools> wgrant: is it just Aus or is downloading from PPAs terribly slow for everyone?
<wgrant> bigjools: It depends on the ISP in Australia. Nothing abnormal's going on today.
<bigjools> wgrant: I'm glad I use the top ISP as recommended by Netflix then :)
<bigjools> but seriously, 70-100k/s.... eugh
<wgrant> Wow, even I usually get at least 170KB/s, and I'm on Optus residential, which is infamously bad.
<wgrant> What's your route to haetae?
<bigjools> wgrant: fun and games criss crossing the US .... http://paste.ubuntu.com/11886069/
<bigjools> I have a recollection of the PPA machine being overloaded but I've no idea if you guys improved it since I left
<wgrant> germanium was overloaded, but it was replaced in 2012 with a rather faster SAN-backed machine.
<wgrant> That quite solidly resolved the IO contention.
<bigjools> is it serving repos from the same machine or is that done properly now?
<wgrant> It's still all served from one machine, but the librarian is now all on swift so diskless archives are on the agenda.
<bigjools> heh, I remember thinking about diskless archives
<cjwatson> blr,wgrant: can I poke you two about pending QA from a day or two ago?
<blr> cjwatson: sorry, will get that qa'd this morning
<blr> wgrant: cjwatson: remind me please, how do you set a bzr alias for lp://qastaging to lpqas:
<cjwatson> blr: You'd have to hack the launchpad plugin or write your own, I think.
<blr> cjwatson: that might be too much effort to save 8 characters.
<blr> cjwatson: my qa is done incidentally
<lifeless> there's an alias plugin
<lifeless> it might be included, I don't remember
#launchpad-dev 2015-07-17
<blr> wgrant: trying to reason through something relating to a RefVocabulary - existing vocabularies with restrictions are typically providedBy() context, but in this case the target repository is set clientside.
<wgrant> blr: Yeah, the dependency is somewhat inconvenient.
<wgrant> It may be best to implement it purely client-side.
<wgrant> That is, there should be a RefVocabulary on IGitRepositoy. You may not be able to fit that into the normal form infrastructure, but your JS will be able to see the currently chosen repo and construct the URL appropriately.
<blr> ok, that makes sense. thanks
<wgrant> blr: Have you worked out a way to do it?
<wgrant> blr: I can't immediately think of any examples on forms, but the "Link a related branch" JS widget on BugTask:+index does some special stuff.
<blr> wgrant: hmm possibly, will have a look at the bugtask js, thanks
<wgrant> IIRC it manually constructs the vocab URL and even triggers a search immediately in some scenarios.
#launchpad-dev 2015-07-19
<blr> morning
#launchpad-dev 2017-07-21
<sil2100> Hello! Looking for some LP API expertise: I would like to get the list of package uploads for a selected uploader, something like we have on the LP UI in the 'Uploaded packages' section of user info
<sil2100> Is there some better way than browsing through the whole publishing history and looking for certain people there?
<sil2100> I didn't see any API call that could do it per user
<cjwatson> I'm not sure what a good approach there would be.  The exact implementation of "Uploaded packages" is difficult to export since it returns a collection of `SourcePackageRelease`s rather than `SourcePackagePublishingHistory`s
<cjwatson> Perhaps it should just be another parameter to archive.getPublishedSources
<sil2100> That would be useful, since source_package_publishing_history does have all the fields of interest
<cjwatson> There are some subtleties there that aren't obvious from the field names; but broadly, yes
<cjwatson> I also haven't checked whether we have sufficient indexes to do that query efficiently
<sil2100> cjwatson: in that case, maybe a bit unrelated to LP, but do you know of any other easy way I could access to data of which (or actually only how many) developers have been actively doing uploads in the recent x months>
<cjwatson> sil2100: Have you considered just looking at the -changes list archives?
<sil2100> hm, I guess that could be a bit better than going through all those spph
<cjwatson> there used to be a page with pie charts by uploader+release, but I've lost it
<sil2100> That one would be useful
<cjwatson> http://people.ubuntuwire.org/~stefanor/ubuntu-activity/ but it only goes up to yakkety
<sil2100> cjwatson: big thanks!
<sil2100> This will be very helpful
