#launchpad-dev 2009-11-16
<nhandler> I was looking at fixing Bug #151113 . It looks pretty straight forward, Matthew did a great job of explaining exactly what needs to be fixed
<mup> Bug #151113: Notification footer has unnecessary leading spaces <email> <trivial> <Launchpad Blueprints:In Progress by nhandler> <https://launchpad.net/bugs/151113>
<jml> nhandler, cool.
<nhandler> jml: The only thing I'm not completely sure about is what test I am meant to run.
<nhandler> I'm also not fully sure how to Demo it. AFAIK, rocketfuel does not setup a working mail server, so I would think that email notifications would not work
<wgrant> nhandler: It will use the local MTA and send emails to root.
<nhandler> Ok, thanks wgrant
<wgrant> nhandler: I just have postfix running with a local-only config, alias root to my user, and point Evolution at my mbox.
<jml> jelmer, https://bugs.edge.launchpad.net/launchpad-code/+bug/221370
<mup> Bug #221370: Cannot checkout a Launchpad hosted project <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/221370>
<nhandler> wgrant: But would a MTA be setup on whatever machine they attempt to verify the patch on? I know that I currently don't have a local MTA setup, which causes some errors when it attempts to send the message
<wgrant> nhandler: The test suite uses a separate email-sending stub, so does not care about the local MTA.
<jml> sinzui, are you around?
<sinzui> jml: I am about
<jml> sinzui, I was going to ask you about which table links packages to productseries, but I've since learned that it's 'packaging'
<jml> I thought I had another question too, but I've lost it in my task switching
<sinzui> jml: yes. A surprising name
<jml> sinzui, there was also some confusion, given that the table has changed recently
<wgrant> I think it's quite a reasonable name.
<jml> wgrant, the two are not mutually exclusive :)
<sinzui> jml: I would have moved the objects to the registry months ago it was names SourcePackageLink
<sinzui> I think I would rename SourcePackage to SourcePackageRelease now that I understand it better
<wgrant> Huh?
<wgrant> You mean DistroSeriesSourcePackageRelease?
<wgrant> s/Release//
<wgrant> SourcePackageRelease is something else.
<sinzui> The sourcepacakge is a very mutable thing. it is just a meta object for for DistroSeriesSourcePackageRelease. So when I visit the url for a DistroSeriesSourcePackageRelease, I actually get the meta object that lists the real object
<wgrant> Currently a SourcePackage is a meta-object for many SourcePackageReleases.
<wgrant> Oh, right, I see.
<wgrant> TheseNamesAreTooLongAndConfusing.
<jml> sinzui, SourcePackage should be DistroSeriesSourcePackage
<sinzui> jml: yes, that might be better
<jml> wgrant, the model is too big and confusing
<jml> wgrant, which might be because what it models is too big and confusing
<wgrant> jml: I think it's more of a problem that it has names like IDistroArchSeriesBinaryPackageRelease in it.
<sinzui> I think a SourcePackage is wrongly a QuestionTarget because it's short name lead someone to think it was a long lived object. To make the implementation work we had to use a DistributionSourcePackage. So the interface and documentation is all messed up. The names are too long an still not clear
<wgrant> There's also the horrible two versions of DistributionSourcePackage which makes some stuff pretty confusing.
 * sinzui needs a nice picture written in crayon to understand how this works
<sinzui> I only know one, and I don't know which one that is
<wgrant> (one is a DB object introduced for package-specific bug reporting guidelines. it is used for nothing else. the other is a separate object generated on the fly.)
<jml> how many packages do you think are linked to a productseries?
<jml> I'm doing some queries and trying to sanity check my results
<wgrant> jml: I would suggest a little over 3000, but the UI may well be lying.
<jml> wgrant, where are you seeing that in the UI?
<wgrant> https://edge.launchpad.net/ubuntu/karmic/+packaging
<jml> ta
<jml> wgrant, that doesn't look like 3000 to me.
<jml> it looks more like a few hundred.
<mwhudson> hi mr. jml
<wgrant> jml: Over several series!
<jml> mwhudson, hi
<jml> wgrant, cool. that matches what I'm seeing with my queries.
<jml> thanks.
<jml> mwhudson, I was sorry to hear you were unwell.
<mwhudson> jml: it's the same thing as i was under on friday
<jml> mwhudson, :(
<mwhudson> i feel a bit better now, but as i've spent all day in bed up to now it's hard to separate cause and effect
<jml> heh
<mwhudson> jml: must be late where you are?
<jml> mwhudson, it is about 11:30
<mwhudson> oh ok
<jml> mwhudson, but I had an urge to get this graph done
<jml> it turns out that we have many, many fewer upstream links registered in Launchpad than I had expected.
<wgrant> Well, the UI has always sucked.
<wgrant> Until sinzui fixed it last week.
<mwhudson> i was vaguely aware that the count was a bit embarrassing
<jml> wgrant, you are correct, and sinzui is awesome
<jml> wgrant, and I was foolish for being optimistic :)
<jml> hey, what do you think a good filter for "working branch" would be?
<sinzui> If I was awesome I would have merged the packing linking UI like beuno ask me too two months ago
<jml> mirror_failures = 0 & last_scanned_id is not null?
<wgrant> As well as the UI sucking, there is little to no benefit.
<wgrant> Besides upstream bug linking becoming slightly easier.
<wgrant> But the UI is still scary, as it screams that you should almost never link trunk to a package, when that is in fact what you normally want to do.
<jml> hmm
 * jml should actually look at the UI
<wgrant> (although that's perhaps a manifestation of another bug -- LP doesn't know much about all these placeholder projects)
<jml> I want to make sure that we have trunk imports for as many upstreams of packages as possible
<wgrant> Maybe once you have bzr-svn.
<jml> but perhaps I need to either rethink that or the way I am gathering my data about this.
<jml> wgrant, any moment now. mwhudson has promised it one hour after we have python 2.5 :)
<jml> also, jelmer says bzr-hg is finished.
<wgrant> I thought bzr-hg was very much not.
<wgrant> That's very good news.
<mwhudson> jml: oh, awesome
<wgrant> Plus 2a doesn't suck. LP might be ignored less now.
<mwhudson> (about bzr-hg)
<sinzui> I think we lost a lot of GNOME links because of the switch from main to master. We should be able to fix all those automatically
<wgrant> sinzui: How good is the p-r-f at creating appropriate series? For how many projects will it work?
<wgrant> Proper upstream linking would such less if one didn't have to find a project admin to create series first.
<jml> wgrant, all of these ideas are making me want a tag and a bunch of bugs
<wgrant> jml: Sounds like you need a BugBagâ¢.
<sinzui> wgrant: It cannot create a series, in fact, the series must exist so that the PRF can be instructed how to find releases
<wgrant> sinzui: Oh, blah.
<sinzui> wgrant: now that the PRF is very good at finding files, users see that our hack to make milestones work with project-groups is does not work with projects that do not adhere to debversion names
<wgrant> sinzui: Does anyone use ProjectMilestones besides LP?
<sinzui> bazaar might
<sinzui> They work for GNOME's core apps/libs, which I think is about 10% of all GMOME projects
<sinzui> They do not work for Launchpad, we have lots of projects like lazr or oops tools that are versioned differently
<wgrant> They do work, in the most part, for LP's ridiculous use of products.
<sinzui> yeah, they work for a pathological case \o/
<stub> prf doesn't work for pypi though, so I guess I should submit a bug on that.
<[reed]> where's kiko when you need him
<jml> Texas
<sinzui> stub: The log I have from two weeks ago shows the PRF reading pypi
<jml> I've sent some data & queries to the launchpad-dev list
<sinzui> stub: was this for pytz? Nothing was found for that
<jml> wgrant, sinzui: I'd really appreciate it if you could post your thoughts re linking so they don't languish here on IRC beyond thought and memory
<wgrant> jml: Those Lucid numbers are really encouraging.
<jml> wgrant, seriously?
<wgrant> jml: No.
<wgrant> jml: Also, your numbers for packages in main are wrong.
<sinzui> jml: I believe 1) creating a series from a parent should preserve links. 2) we can prepare a script to create the missing links to lucid
<wgrant> There are maybe 4000. Not 16000, although that will change in the next few months when the components are changed.
<jml> wgrant, what do you think I got wrong in the query?
<wgrant> jml: Ah, I didn't see the queries.
 * wgrant looks.
<wgrant> jml: You're not doin any restriction over components there.
<jml> sinzui, cool. as I said, I'd appreciate email.
<sinzui> jml: I am preparing a report of sorts myself. I am looking at each gap between Ubuntu and an upstream to get a count of how many packaging links are missing, how many series are missing, how many series are missing branches how many missing projects.
<sinzui> jml: I hope to finish tomorrow and send it to launchpad-dev.
<jml> wgrant, of course.
<jml> ta
 * jml pastes the IRC log into an easily findable file so that all the good ideas don't get lost.
<wgrant> jml: I'm not sure why you care about -updates in those queries. That will just duplicate packages unnecessarily.
<wgrant> Since except for very exceptional cases, -updates will not contain anything new, nor will it contain component changes.
<jml> wgrant, me neither. I was heavily advised :)
<jml> wgrant, I'm about to go to bed, so I won't do anything more on this tonight.
<wgrant> jml: Ah, you're at UDS?
<jml> wgrant, like the dickens.
<jml> wgrant, I looked for your name on the attendees list.
<jml> wgrant, and thus I assume you aren't here :)
<wgrant> jml: Correct.
<jml> wgrant, a shame.
<wgrant> jml: Well, I'm too frustrated with Ubuntu development to do anything useful there at the moment.
<sinzui> I'm off to bed, but before I go, I think you are mistaken jml. you are the one is Dallas Texas. There is a lot of same there.
<jml> sinzui, :D
<jml> wgrant, that sucks.
<stub> sinzui: Yes - pytz. I was assumng it doesn't find the links because of all the foo.tar.gz#md5sum=4545564564 links
<stub> sinzui: Are you aware of any projects successfully getting picked up by prf?
<sinzui> stub: I'll take a look at the issue during my lunch tomorrow
<sinzui> stub: yes I have heard from many projects that are very happy with it now.
<sinzui> stub: The last two fixes I landed were were diagnosed by users how read the code and identified why their files were missing.
<stub> sinzui: Know a product name? I might just need to copy their URLs.
<sinzui> stub: https://edge.launchpad.net/python-phynx/0.10
<sinzui> stub: and this one too https://edge.launchpad.net/python-quantities/0.5
<jml> g'nigh.
<jml> t
<wgrant> Night jml
<stub> I had http://pypi.python.org/pypi/pytz/pytz-*.tar.bz2 for my release URL. Others seem to be using http://pypi.python.org/packages/source/p/phynx/phynx-0.10.*.tar.gz. Didn't know about that second URL type, but it omits all the #md5 trailers from the URLs.
<sinzui> stub: I test the file globs in a launchpad.dev series then run the script in cronscripts
<sinzui> We should send owner emails when there are errors, and I think I know how to test the location to show the owner what files the PRF will see
<adeuring> good morning
<matgeek> elmo:  Off topic - Are you still looking for a sys admin in the Asia/Pacific region?  I have just read the posting, and I tick most of the boxes. from NZ, I am Debian dev, been using linux since 1993.
<matgeek> I'm a newbie... Just running rocketfuel-get and I am getting errors about missing branches from bzr...  is this normal?
<wgrant> matgeek: If it's just shipit and canonical-identity-provider, that's normal.
<wgrant> matgeek: Those two components remain proprietary.
<matgeek> wgrant: Thanks, that's what they are, but then I get an exception form bzr about a.git not being there when trying to remove it -- seems a bit disconcerting?
<wgrant> matgeek: What exactly is that error message?
<matgeek> wgrant:  'OSError: [Errno 2] No such file or directory: '/home/grantma/launchpad/lp-sourcedeps/sourcecode/a.git''
<matgeek> wgrant: along with a stack trace, which does not seem to be a tidy way of handling it!
<wgrant> matgeek: That's odd. I haven't seen it before.
<wgrant> matgeek: pastebin the full traceback, please.
<matgeek> wgrant: I'm dumb, where's the pastebin site please?
<wgrant> matgeek: http://paste.ubuntu.com/
<matgeek> wgrant: OK
<matgeek> wgrant: up at http://paste.ubuntu.com/319863/
<matgeek> wgrant: bazaar 2.0.2-1~bazaar1~karmic
<wgrant> matgeek: That's special. Rerun rocketfuel-get when it finishes, and see if it happens again.
<matgeek> wgrant: Already done that.  That's a pastebin of a rerun, but I will try again.
<wgrant> matgeek: Does a.git exist where it says it does?
<matgeek> wgrant: The parent of it exists, which is full of other dirs, so I expect if it did it exist it would be a dir to?
<matgeek> wgrant: ie a.git directory does not exist
<matgeek> wgrant:   would you like to ssh into my launchpad-dev virtual, and have a look first hand?
<pgquiles> matgeek: have you fixed the a.git issue?
<mrevell> How's it going Launchpad people?
<mars> good morning
<mars> flacoste, yui3 didn't land on Friday.  Apparently I am "not authorised to commit" to db-devel?
<flacoste> mars: you are
<mars> not according to PQM :)
<flacoste> mars: what --dry-run gives you?
<flacoste> mars: the commit list is shared between devel and db-devel
<mars> flacoste, --dry-run gives no errors.  Same as Friday.  It took a while, but both of my submissions came back with the "not authorized" error.
<mars> flacoste, here is the command I used: bzr pqm-submit --submit-branch=bzr+ssh://bazaar.launchpad.net/%7Elaunchpad-pqm/launchpad/db-devel/ -m "[rs=flacoste] Update launchpad to use YUI 3.0.0."
<flacoste> mars: first of all, the commit message isn't correct
<mars> oh?
<flacoste> you'll need something like [r=flacoste][ui=none]
<flacoste> rs is fine
<flacoste> but use r, since i did review it :-)
<mars> ah.  Everyone else in log uses just [rs=...]
<mars> ok, my first submission was [r=flacoste], but since it didn't stick, I tried [rs=flacoste] instead
<flacoste> mars: that's a special case for the buildbot-poller
<flacoste> rs=buildbot-poller
<mars> flacoste, gary_poster, I think merging yui3 should wait until after Python2.5, given that the JS build system broke on staging when it was running the python2.5 branch.
<gary_poster> mars: :-(
<mars> gary_poster, just a suspicion.  BjornT_ noted that only staging was broken in a specific way, and that the difference was it's running of python2.5
<gary_poster> mars: you can work that out now with the pertinent branch if that helps; and I'm happy to help
<mars> gary_poster, I'd like to take a look at staging first
<mars> to isolate the problem BjornT_ pointed out
<gary_poster> mars: ok.  let me know if I can help.
<sinzui> bac: EdwinGrubbs: stand-up in 2 minutes
<sinzui> flacoste: is the UI meeting canceled?
<flacoste> sinzui: yes, many of the attendees are at UDS
<sinzui> thanks
<flacoste> noodles, beuno, rockstar
<flacoste> barry
<flacoste> and intellectronica
<sinzui> Chex do you know why staging is not updating?
<Chex> sinzui: I am not sure, let me take a look
<sinzui> bac: I updated the the two breadcrumb bugs with my thoughts about what we need to do to fix them I think each fix is about 50 lines.
<Chex> sinzui: staging updates were disabled for gary testing there for the python 2.5 upgrade
<sinzui> Chex: staging has not been updated in more than a week. I cannot QA items that landed on the first day of week one
<gary_poster> that's pretty much done, but aiui we are also waiting for stub to fix something.  Does that jibe with the notes, Chex?
<sinzui> Chex: gary_poster: Have we had staging disabled for that long because of py2.5?
<gary_poster> Chex, no.  Mine was from this past Friday, and was added to some kind of disabling because of stub, from what Tom told me.
<gary_poster> oh, sorry, sinzui ^^^
<gary_poster> Chex, sinzui, you can remove the Python 2.5 testing from the reasons to disable staging.  I think we are good.  (Still check on the stub thing though, please.)
<sinzui> I think the 23rd is the latest I can risk with QA. I still have bitter memories of being locked out of staging and then spend release week with the entire team trying to get RCs
<Chex> sinzui: gary_poster: ok, the hold for stubs landing has completed, so were all set with staging _restore and I have re-enabled it now.
<sinzui> Chex: \o/
<mars> sinzui, ping
<sinzui> Hi mars
<maxb> So, is staging currently Python 2.4 or 2.5 ?
<gary_poster> maxb, staging probably is 2.5.  Is it important to establish confidence?
<maxb> Just curious, really. And wondering how automatic updates could be enabled if it was still 2.5, with 2.5 not on devel yet
<gary_poster> the previous build is wiped away
<bigjools> sinzui: "howdy!", did you submit wgrant's branch?
<cody-somerville> bigjools, I thought you  guys were going to waverly.
<sinzui> bigjools: I did, but it failed. The branch has db changes so had to go to db-devel. There were conflicts
<bigjools> cody-somerville: we're in the room opposite
<bigjools> sinzui: argh, ok, thanks
<bigjools> he will have to fix those then
<sinzui> I showed him the conflicts
<bac> hi sinzui
<sinzui> hi bac
<bac> sinzui: i've changed the map portlet to not be shown at all (from the <h2>Location</h2> onwards) if no data is set and the viewer is not the user.
<bac> it seemed dumb to just show the tz when it is shown above
<sinzui> bac: Yes, I think that is a very sensible decision.
<mars> rockstar, ping: during the sprint, did you switch lazr-js over to use explicit references to the needed version of YUI?
<thumper> morning
<kfogel> rockstar: ping
<sinzui> fuck, my keyboard and editor is suddenly typing in Cyrillic!
<bac> sinzui: Ð¯eally?
<sinzui> Really it was, I had to paste that message together
<intellectronica> after updating, i can't seem to run tests anymore. i get the following error:
<intellectronica> Traceback (most recent call last):
<intellectronica>   File "bin/test", line 174, in ?
<intellectronica>     from subunit import TestProtocolClient
<intellectronica>   File "/home/tom/launchpad/lp-branches/bug-481356/lib/lp/scripts/utilities/importfascist.py", line 163, in import_fascist
<intellectronica>     module = original_import(name, globals, locals, fromlist)
<intellectronica> ImportError: No module named subunit
<intellectronica> anyone knows what's going on and what i need to do to fix this?
<kfogel> intellectronica: I wonder if there's a new dep in launchpad-dependencies that you need?
<kfogel> some python testing module that would be brought in, maybe?
<mwhudson> good morning
<thumper> mwhudson: how are you feeling today?
<mwhudson> thumper: mostly better, we'll see how it goes
<thumper> flacoste: it seems we have multiple large distruptive type changes landing, how are we co-ordinating python 2.5 and YUI 3 releases?
<intellectronica> kfogel: i'm up-to-date, so if there's a new dep it must not have been added to our dependency package?
<intellectronica> gary_poster: maybe you know? ^^^
<flacoste> thumper: ask gary and mars
<thumper> :)
<gary_poster> thumper: atm, Python 2.5 wins, thanks to mars generosity.
<gary_poster> thumper Python 2.5 is landing now, hopefully
<thumper> cool
<thumper> gary_poster: will you be emailing the dev list once it is done?
<intellectronica> after installing subunit manually i'm good, it seems
<intellectronica> so i guess it is indeed a missing dependency
<gary_poster> thumper: I intend to mail everyone before then.  Once it lands, there will need to be some clean up; if I get https://bugs.edge.launchpad.net/launchpad-foundations/+bug/481512 reviewed, then once it is landed it will just mean that you ned to do a make clean and make
<mup> Bug #481512: race condition when rotating logs <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/481512>
<gary_poster> Hm, not that sorry
<gary_poster> https://code.launchpad.net/~gary/launchpad/py25makeclean/+merge/14916
<thumper> mwhudson, abentley: lets try skype
<gary_poster> intellectronica: I'm not familiar with that
<gary_poster> intellectronica: sounds like something from jml/allenap.  Let me see how you did that; maybe I can dupe
<intellectronica> gary_poster: i suspect it might be related to the parallelization work allenap and jml have been doing
<gary_poster> intellectronica: precisely
<mwhudson> i thought subunit was in sourcecode or an egg or something
<intellectronica> heh
<mwhudson> yeah, i have it in sourcecode/
<mwhudson> subunit=lp:~launchpad-pqm/subunit/trunk/
<intellectronica> gary_poster: i did that by running bin/test
<intellectronica> mwhudson: well, since installing the package works for me, why not just rely on that?
<mwhudson> intellectronica: ^^
<thumper> gary_poster: approved
<intellectronica> or is that version missing something i might need later?
<thumper> abentley: we can hear you
<mwhudson> intellectronica: um, why not install twisted or bzr from a package then?
<thumper> abentley: let me try hosting
<abentley> thumper: I'm getting that techno thing.
<mwhudson> or zope or ...
<gary_poster> thumper: thanks!
<thumper> abentley: this sucks arse
<thumper> abentley: do you have sip working?
<intellectronica> mwhudson: donnow? maybe because the packaged version is too old?
<gary_poster> intellectronica: I just tried ./bin/test -t test_all_scripts and it worked.  can try another
<abentley> thumper: No, I don't.  Haven't tried.
<mwhudson> intellectronica: in any case
<thumper> abentley: kfogel, mrjazzcat and I did a three way sip call and it was perfect
<mwhudson> intellectronica: i think the thing to worry about here is why you don't have sourcecode/subunit
<thumper> we found that twinkle worked best
<mwhudson> i don't appear to have that installed
<thumper> I didn't have it installed either
<intellectronica> gary_poster: to be precise i ran `bin/test -vv -t xx-milestone-add-and-edit.txt` but it can also be that you already have subunit installed
<abentley> thumper: Installing that.  What will I need as credentials?
<kfogel> thumper: I've since heard from elsewho that twinkle is Da Bomb too (Bradley Kuhn at SFLC uses it all the time).
<thumper> abentley: you should have them in an email form IS a long time ago...
<thumper> abentley: I asked on the #is channel to get them sent to me again
<abentley> Okay, I'll dig that up.
<intellectronica> mwhudson: yes, that oo
<mwhudson> thumper: any idea what the subject: of the email was?
<thumper> Your Canonical VOIP account details
<gary_poster> intellectronica: FWIW, I ran that test command successfully, and do not have subunit installed.
<mwhudson> hm, no hits
<gary_poster> intellectronica: no idea in specific, but would vaguely try make clean and make if it were I
<gary_poster> intellectronica: I am willing to help if you want.  Also, if we can get that working and you are not at EoD, I'd be happy to have you be the guinea pig for migrating to the Python 2.5 branch, if you were willing.
<intellectronica> gary_poster: sorry, if i'm unresponsive, i'm at UDS. anyway, it's working for me now, but my guess it's a slightly different different setup from yours since i have subunit installed from a package and not in sourcecode
<intellectronica> gary_poster: and sure, i'm happy to try py2.5 later on
<gary_poster> ...huh, weird.  ok.  Py 2.5 ok thanks.  Ping me when you are available and we'll give it a whirl
<intellectronica> gary_poster: sure thing. going into a couple of sessions in ~15m, but happy to give it a try after that
<gary_poster> ok cool
<gary_poster> thanks
<wgrant> al-maisan: Did the instructions otherwise work?
<al-maisan> wgrant: I'll be able to tell you tomorrow -- I need to finish another bit of work before I can try them.
<wgrant> al-maisan: Ah.
* Ursinha changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 2 of 3.1.11 | PQM is open  | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<flacoste> gary_poster: do you know where I can find the bug filed last week about HTML support in named operation?
<gary_poster> flacoste: I seem to recall it being in lazr.restful
<flacoste> gary_poster: nm, just found it: https://bugs.edge.launchpad.net/lazr.restful/+bug/480192
<gary_poster> :-)
<flacoste> exactly
<mwhudson> um i guess "someone" should look at landing bzr 2.1b3 into rf
<spm> mwhudson: was that a dual volunteering I note ^^ ?
<mwhudson> spm: no
 * spm breathes a not too small, not too large, sigh of relief
<Ursinha> I'm getting a "No module named boto" when trying to run utilities/ec2. What can I be doing wrong?
<wgrant> Ursinha: Have you python-boto installed?
<Ursinha> wgrant, I guess so, unless something removed that
<Ursinha> wgrant, and something did that :/
<wgrant> Ursinha: :(
<Ursinha> wgrant, I thought for a while this could be launchpad issue, because it was working after the last branch update I did
<Ursinha> crazy
<Ursinha> thanks wgrant
 * mwhudson stops writing a crazy shell script
#launchpad-dev 2009-11-17
<wgrant> Can I convince someone to ec2test and land to *db-devel* https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection-part1/+merge/14729? All except for the final two merges have been reviewed.
<jml> intellectronica, what was the subunit issue?
 * mwhudson lunches
<jml> thumper, hi
 * maxb wonders where python-2.5 is in PQM
<mwhudson> maxb: it's not in pqm :(
<spm> maxb: lp.codehosting.tests.test_acceptance.AcceptanceTests.test_push_triggers_mirror_request(sftp) is last not-buffered  test outputted
<maxb> ooh, getting there, then
<spm> maxb: what mwhudson said tho - it's a manual merge, not pqm. same box tho.
<mwhudson> ah
<spm> no failures yet either. looking good.
<jml> man
<jml> I picked the wrong week to make a USB stick w/ the launchpad repo on it.
 * jml back in a bit
<mwhudson> jml: there's a right week for that?
<spm> sweet. the branch succeeded.
<thumper> jml: hi
<thumper> spm: the python 2.5 branch?
<spm> thumper: yup. have pushed the new shiny everywhere too now
<thumper> spm: awesome
<wgrant> Awesome.
<wgrant> Curses.
 * thumper goes to remind himself what is new in 2.5 to use
<wgrant> ifs in expressions!
<spm> and pqm is re-open for srs bsns
<wgrant> srs OMG 2.5 BROKE THE WORLD bsns?
<spm> no
<spm> only for srs ZOMG 2.5 BROKE THE WORLD bsns
<thumper> wgrant: ew
<spm> the Z is critical pathness hotness shinyness
<thumper> wgrant: the syntax for that is horrible
<wgrant> thumper: Better than using four lines instead.
<thumper> wgrant: well... that is  a matter of opinion :)
<wgrant> Pfft.
<thumper> PEP 309: Partial Function Application
<thumper> oohh..
<wgrant> Ah, yes, functools?
<thumper> PEP 328: Absolute and Relative Imports
<thumper> hmm
<thumper> I see an extended coding standard being applied for imports
<wgrant> Oh, that's in 2.5 too?
<thumper> PEP 343: The 'with' statement
<thumper> \o/
 * jtv hates that one
<thumper> defaultdict
<thumper> jtv: how can you hate the with statement?
<jtv> Its design smells of over-complication for the wrong reasons.
<thumper> jtv: it gives the abiltiy to have controlled release of resources
<jtv> Plus, the main reason I would normally like it is that it saves indentation levelsâbut it doesn't
<thumper> and my favourite reason right now...
<thumper> bzr-svn
<mzz> it kinda does, somewhat (contextlib.nested)
<jtv> It doesn't give the ability to have controlled release of resources; it provides syntactic sugar for continued awkward, manual release of resources.
<mwhudson> $ cd ../bzr-svn-imports/
<mwhudson> $ bzr missing --theirs-only
<mwhudson> You are missing 1130 revision(s):
<mwhudson> ...
<wgrant> Nice.
<wgrant> You are in for a long evening.
<thumper> mwhudson: are you saying that jelmer has 1130 new revisions since we last looked?
<mwhudson> it shouldn't be that bad
<wgrant> I presume that's 1130 LP revisions.
<mwhudson> thumper: no, this is my branch that adds bzr-svn support to launchpad
<thumper> heh
<maxb> Now if only I could finish fixing bzr-rebase, I could rebase the python-migration2.6 branch on top of devel
<wgrant> maxb: Why would you rebase?
<mwhudson> wow one conflict
<mwhudson> in sourcecode/Makefile ffs
<thumper> mwhudson: \o/
<maxb> wgrant: Because most of the content of the branch is either merges from trunk, or stuff that got cherrypicked onto the 2.5 branch
<wgrant> maxb: But rebasing is bad.
<maxb> Sometimes? Yes. Universally? No.
<mwhudson> heh, i guess i can remove my hack that adds collections.defaultdict from this branch
 * mwhudson heats up his laptop running a 2.5 make build for the first time
<mwhudson> wee buildbot fall over
<wgrant> How badly?
<mwhudson> src/zope/security/_proxy.c:19:20: error: Python.h: No such file or directory
<mwhudson> so missing python2.5-dev i guess
<wgrant> Ah.
<wgrant> Easy.
<wgrant> Yep.
<maxb> huh. Shouldn't it have launchpad-developer-dependencies installed?
<wgrant> It's buildbot. That would be too normal.
<maxb> Sometimes I wish LP used bzr shared repositories instead of stacking :-/
<wgrant> Why?
<maxb> bzr is currently uploading upwards of 12MB for a simple UDD package merge
<wgrant> Is it actually stacking?
<maxb> it claims to be
<maxb> maybe I should have stacked it on the debian branch instead
 * spm is reminded how much he hates updating AMI's....
<mwhudson> spm: i wrote a script to do it ...
<mwhudson> (ec2 update-image)
<spm> mwhudson: you did? is that on prase? or local for you?
<mwhudson> sadly the buildbot and ec2 setups are more or less disjoint...
<spm> it's been a month or 3 sinece the last one I did. so am a tad rusty.
 * mwhudson has finally wrestled his inbox into some kind of submission
<wgrant> mwhudson: Sounds like you need another sprint.
<maxb> Buildbot is blaming me for something
<mwhudson> maxb: see above
<maxb> oh, that
<maxb> wonder why it's blaming me! :-)
<wgrant> It blames everyone with merged revisions since the last run.
<mwhudson> no idea
<thumper> rockstar: you around hacking?
<jml> hey
<jml> how do I query status on bugtarget.searchTasks?
<wgrant> jml: List of strings.
<wgrant> Wow, those docs suck.
<jml> wgrant, what are the strings?
<jml> wgrant, yes, they do.
<wgrant> jml: ["Triaged", "Won't Fix"] and the like.
<jml> yech
<wgrant> Hm?
<jml> I dislike this API.
<jml> also len(collection) doesn't work as advertised.
<wgrant> What does it do instead?
<wgrant> Note that collections resulting from named operations are... odd.
<jml> it raises an error.
<jml> I guess those are the kinds of collections I've got.
<jml> wgrant, do you know of a way to get the size without iterating through the list?
<MTecknology> Make sure I understand this... RAID 1+0 == [ mirror ( span drive_1 & drive_2 ) & ( span drive_3 & drive_4 ) ]   ?
<MTecknology> Sorry, wrong channel - but... can you guys answer that?
<wgrant> jml: See statik's last comment on bug #303414, but bug #274074 is the real bug.
<mup> Bug #303414: Collections returned by named operations don't act like other collections <launchpadlib :Triaged> <https://launchpad.net/bugs/303414>
<mup> Bug #274074: Missing total_size on collections returned by named operations <api> <launchpadlib :New for leonardr> <https://launchpad.net/bugs/274074>
<jml> wgrant, thanks.
<wgrant> jml: Ah, the latter has a workaround.
<mwhudson> note that total_size might be a lie in any case, i think
<jml> mwhudson, it's giving me the same numbers as len(list(collection)) in this case
<mwhudson> jml: ok
<mwhudson> jml: i'
<mwhudson> m thinking of private bugs
<jml> mwhudson, but I do think you're right.
<mwhudson> but i might be barking up the wrong tree
<jml> mwhudson, ahh, of course.
<wgrant> Some queries I know of are adjusted to exclude invisible objects from even the SQL result.
<wgrant> But I don't think that is the case here, so the numbers will indeed be wrong.
 * mwhudson grinds to a halt
<mwhudson> wgrant: the branch listings you mean?
<wgrant> mwhudson: The specific example I was thinking of was related to builds, but that too.
<MTecknology> offtopic - Has anyone ever thought about using LP to manage user accounts on a server?
<thumper> MTecknology: don't think so...
<MTecknology> thumper: I think I want to try that when I get a server to develop LP on
<MTecknology> it sounds like fun - sounds easy too
 * thumper EODs (for now)
<MTecknology> thumper: does that mean bored or sleeping
<thumper> MTecknology: it means making dinner
<MTecknology> oh
<wgrant> bigjools: Are you really here?
<intellectronica> jml: the issue with subunit was that i didn't have it. neither as a package nor in sourcecode.i fixed it by installing the package, which is presumably different from everyone else who seem to have it in sourcecode
<lifeless> intellectronica: subunit issue?
<adeuring> good morning
<stub> ImportError: /usr/lib/python2.4/site-packages/psycopg2/_psycopg.so: undefined symbol: Py_InitModule4
<wgrant> stub: You've followed the 2.5 upgrade instructions?
<stub> I'm sure those instructions are around here somewhere...
 * stub rummages
<wgrant> stub: They were sent to launchpad-dev@, prefixed with 'PLEASE READ'
<stub> Got that - just talking about make clean and friends
<ajmitch> make clean & an important 'rm'
<stub> Thats not it
<stub> For some reason buildout is deciding to put usr/lib/python2.4/site-packages in my path
<wgrant> ajmitch: That rm landed hours ago.
<wgrant> stub: Awesome.
<ajmitch> wgrant: I thought that mail said to do the rm manually if you'd updated in the meantime
<ajmitch> it didn't affect me, so I didn't worry too much about it
 * wgrant pushes SHHH off a cliff.
<wgrant> Hmmmmmm.
<wgrant> I just started getting that 'a.git' sourcecode removal error that somebody reported yesterday.
<wgrant> I wonder if it's a nested branch hidden somewhere.
<wgrant> Looks like dulwich tests.
<poolie> hi wgrant
<stub> Does anyone else see python2.4/site-packages in their bin/py ? It might not cause errors if the ordering is different.
<wgrant> stub: I'm currently 2.5ifying. Will tell you in a sec.
<wgrant> Hi poolie.
<wgrant> stub: Nothing in mine.
 * stub pouts
<wgrant> stub: Nothing crazy in ~/.buildout?
<stub> I don't have one
<wgrant> Huh.
<wgrant> stub: Tried a fresh copy of devel?
<wgrant> buildout annoys me.
<stub> develop-eggs existed with a file referencing 2.4. Nuking this directory seems to have cleaned things up. Might be because my branches tend to be old.
<wgrant> Ah.
<wgrant> stub: None of my develop-eggs/ contain 2.4.
<stub> Looks like buildout generated magic that make clean doesn't clean
<wgrant> Is this a seriously old checkout?
<stub> The file was dated July. Like I said - my branches tend to be old ;)
<wgrant> stub: Ah, I didn't consider 'old' to mean 'before the epoch'
<Ursinha> hi guys
<allenap> abentley: Do you have a few minutes to help me understand why a code import is breaking? It keeps failing with logs that look like: http://launchpadlibrarian.net/35783114/nhibernate-trunk-log.txt
<Ursinha> after having a db patch approved, what do I have to do?
<allenap> Ursinha: Hi. Land it to db-devel I think.
<Ursinha> allenap, hi :) do I have to rename it and stuff?
<rockstar> allenap, that looks like a bug in cscvs
<allenap> rockstar: Ah.
<Ursinha> allenap, that's a bug that cscvs that times out when importing large svn repositories
<Ursinha> s/that cscvs/in cscvs/
<rockstar> allenap, I think there might even be an open bug about it, but I'm about to lose internets, so I can't search.
<Ursinha> rockstar, there is
<rockstar> Ursinha, :)
<Ursinha> allenap, I think I mentioned the bug number in my handover email
<allenap> rockstar: Thanks :)
<Ursinha> rockstar, :)
<allenap> Ursinha: You probably did, I just have a sieve for memory :)
<Ursinha> hehe
<Ursinha> so :) do I have to just land it to db-devel or have to rename it as stub defined, move between directories...?
<allenap> Ursinha: Re. db patch, have you looked at database/schema/README.
<Ursinha> allenap, yes, but the post-approval I'm not clear about
<allenap> Ursinha: Did stub give you a patch number?
<Ursinha> allenap, yes
<allenap> Ursinha: So, yes you have to rename the patch with that number, then land it. (Sometimes stub offers to land these changes on your behalf, but it's been a long time since I did a db patch and I don't know if he does that any more).
<allenap> Ursinha: Is that what you were looking for, or did I get the wrong end of the stick?
<Ursinha> allenap, yes, just that! So I just rename it and keep in the pending folder?
<allenap> Ursinha: Rename it and put it in database/schema too.
<allenap> Ursinha: So it should be something like database/schema/patch-2207-05-0.sql (where 05 is whatever stub gave you).
<bac> gary_poster: i'm getting failures on ec2 related to 2.5.  have you successfully run on ec2 since the switch?
<gary_poster> bac, no.  We did run it successfully on ec2 before the switch with adding the python 2.5-devel dependency.  ec2 is supposed to (did last time I checked) update launchpad-dependencies, so I thought we would be covered even without a new image.
<bac> gary_poster: it is dying building zope.security
<bac> src/zope/security/_proxy.c:19:20: error: Python.h: No such file or directory
<gary_poster> bac: OK, yes, looks like python2.5-devel is missing.  I should have checked that again.  I'm almost done with a related task, and then will go to that.  You should be able to work around that for now; thinking.
<gary_poster> bac: the only way I can think of is a --postmortem, which is annoying.  salgado, is that how you tested ec2 before, withdoing things manually after a --postmortem?
<gary_poster> allenap, do you know where the instructions are to create a new ec2 image?  I'm not familiar with the new automation that mwhudson did.
<allenap> gary_poster: I'll have a look in the code...
<abentley> allenap: looks like the remote server is dropping the connection.
<salgado> bac, gary_poster, I used a demo instance
<gary_poster> salgado: ah ok, thank you
<gary_poster> bac, I'm on this now
<bac> gary_poster: great.  thanks.
<allenap> gary_poster: I think just ec2 update-image. You can specific an AMI name prefixed with "based-on:" if it's a blank install.
<allenap> abentley: Thanks. rockstar and Ursinha suggested it was a timeout bug, so https://bugs.edge.launchpad.net/launchpad/+bug/343207.
<mup> Bug #343207: large SVN Import fails with a timeout <Launchpad itself:Invalid> <Launchpad CSCVS:New> <https://launchpad.net/bugs/343207>
<allenap> abentley: The nhibernate import always fails on SSL negotiation though. Could that still be a timeout?
<gary_poster> allenap: right just saw that, thank you. Will dig in there.  My guess is that we do *not* want a blank install, agree?
<allenap> gary_poster: Yeah, agreed.
<gary_poster> cool, thx
<abentley> allenap: I guess it could be that the remote server doesn't support PROPFIND or something.
<allenap> gary_poster: Looks like you'll also need the ec2-* command line tools installed.
<allenap> gary_poster: It needs ec2-register specifically.
<gary_poster> allenap: oh, ok, good to know.  I don't have them locally.  so this is my revised plan:
<gary_poster> 1) install ec2-* command line tools
<gary_poster> 2) determine the appropriate next AMI name by scrounging around
<allenap> abentley: Strange. Other SF imports work okay. Is this worth spending time investigating, or shall I just attribute it to the aforementioned bug?
<allenap> gary_poster: That sounds about right :)
<gary_poster> 3) use ec2 update-image with an additional command to update dependencies, if necessary (``--extra-update-image-command='sudo aptitude upgrade'``)
<gary_poster> done
<gary_poster> full-upgrade I should have said
<abentley> allenap: It is that bug.  I'm not aware of any workarounds.  But you could bring it up with the Bazaar team.
<Ursinha> allenap, I did the renaming and the moving but the patch didn't make it
<Ursinha> allenap, do I have to do something else, like regenerating sample data?
<allenap> abentley: Okay, I'll email the team, CCing the list.
 * allenap looks
<allenap> Ursinha: What was the error?
<Ursinha> allenap, hmmm HMMM I think I figured out :)
<Ursinha> I forgot changing the line in the patch with the correct number :)
<allenap> Ursinha: If you get stuck again, I'll grab your branch and take a look.
<allenap> Ursinha: Hehe, cool :)
<mrevell> danilos, hey, how are you fixed for the first hour this morning?
<bac> gary_poster: could ec2 grow another option to run arbitrary commands at some point?  ec2 --arbitrary "sudo apt-get install python2.5-devel"
<gary_poster> bac: sure.  a hopefully minor trick would be to specify when the arbitrary command should run.  Maybe immediately prior to build is always good; not sure
<bac> gary_poster: or perhaps something less arbitrary, like a set of packages to install...
<gary_poster> yeah
<gary_poster> would prefer that myself
<flacoste> gary_poster: python2.5!!!
<gary_poster> flacoste: :-) yup
<flacoste> awesome!!!
<flacoste> gary_poster: we can drop lsprof from sourcecode that means?
<gary_poster> flacoste: yeah, should be able to do that
<flacoste> gary_poster: has it merged in db-devel yet?
<gary_poster> flacoste: should have; it merged in devel last night, and we had a successful buildbot run earlier this morning.  Looking
<flacoste> gary_poster: it seems to be part of rev 8665 which hasn't been blessed yet
<intellectronica> lifeless: yeah, subunit missing when i tried to run tests. wasn't available to me either from a package or in sourcecode (which is how most people get it)
<gary_poster> flacoste: you asked if it were in db-devel though: yes, it is, as of rev 9685
<gary_poster> 8685 I mean
<gary_poster> you probably meant 8685 too :-)
<maxb> <flacoste> gary_poster: we can drop lsprof from sourcecode that means?
<maxb> <gary_poster> flacoste: yeah, should be able to do that
<maxb> I already did that :-)
<gary_poster> maxb: go you :-)
<gary_poster> maxb, a huge thank you for all your work on the Python 2.5.  It would not have gone nearly as smoothly or quickly without you.
<maxb> And thank _you_ (and gary and salgado) for wrangling the ZTK demons into submission
<maxb> erm
<maxb> and *barry*
<gary_poster> :-) cool, very happy to work with you
<abentley> allenap: I see you've emailed the code team, but I meant the Bazaar team.
<allenap> abentley: Rats.
<allenap> abentley: Sorry for spamming. Shall I just email the bazaar mailing list?
<allenap> abentley: Scratch that.
<allenap> abentley: I'll email the Canonical Bazaar team directly. Is it worth CC'ing any mailing lists?
<abentley> allenap: You could cc the canonical-bazaar mailing list.
<jml> intellectronica, I had thought I added subunit to sourcecode
<intellectronica> jml: you prolly did. something is obviously wrong with my setup
<henninge> sinzui: ping
<sinzui> Hi henninge
<henninge> Hi sinzui
<henninge> sinzui: I just saw that you changed the view code for the featured projects display on the LP home page.
<sinzui> I did
<henninge> sinzui: it looks like the goal is to put them into one column?
 * henninge has not looked at the template yet
<sinzui> No, the view should not dictate the what the layout doe. The layout is identical to what it was yesterday
<sinzui> henninge: I replaced the custom styles with to global styles that do the same layout
<sinzui> I removed the local styles from the gloabl style sheet and inlines them into the only page that will ever use them
<henninge> sinzui: I see
<henninge> sinzui: but this looks like the list is shortened to 11 entries.
<henninge> sinzui: currently it's 21
<henninge> sinzui: and that is too short, too.
<sinzui> I did not intend to shorten it,
<henninge> sinzui: atm Ubuntu and zope are missing from the list on the home page
<sinzui> My code is not on edge
<henninge> sinzui: before the last roll-out I filed bug 467079
<mup> Bug #467079: List of featured projects on LP home page should be dynamic in length. <post-3-ui-cleanup> <Launchpad Foundations:In Progress by henninge> <https://launchpad.net/bugs/467079>
<henninge> sinzui: but I did not get r-c for it (I was too late ...)
<henninge> for the fix, I mean
<sinzui> I agree it should be dynamic. I removed the code that split the list into two lists. I did not change the length of features, may I should have double to the number
<sinzui> I will QA it when edge updates, if the list is wrong I will fix it.
<henninge> sinzui: I can just land my branch in incorporate your change.
<sinzui> Okay then. If you think the list needs to be longer just do it. If you need a reviewer, I can do it
<henninge> sinzui: cool, won't be long.
<henninge> sinzui: cool, won't be long.
<sinzui> fab.
<henninge> sinzui: can you remind me where to find the template for the home page?
 * sinzui looks at his own diff
<sinzui> lib/canonical/launchpad/templates/root-index.pt
<henninge> just found it ...
<henninge> ;)
<sinzui> ^ I should have moved it to registry, but I did not want to make the branch too long
<henninge> sinzui: I didn't know the two-colmun layout can be achieved by pure css magic. cool
<sinzui> henninge: It is used on a lot of pages. It was one of the classes in style-3.0.css that was landed with the file
<sinzui> henninge: since that CSS file is global, should only contain styles that can be used by all launchpad, do not use ids, use classes
<henninge> sinzui: I understand
<henninge> sinzui: actually, I was surprised at the time to find these styles in the global css
<henninge> putting them in the header is what I prefer, too. (If they are local)
<sinzui> The goal may have been to refactor them into classes that can be used by all the root pages.
<sinzui> That may be a goal now. If so, it is pretty secret. I have need seen any designs for the root pages.
<henninge> sinzui: using the css for the two column display transposes the matrix
<sinzui> Yes it does :(
<henninge> sinzui: I don't think many people will notice, though.
<sinzui> I was going to announce that so that if someone cared he could fix it
<henninge> sinzui: also, it adds more space between the entries, is that intended?
<sinzui> It is intended in the sense that we want all launchpad to display consistently...
<sinzui> But I think a bug was introduced into the CSS a few weeks ago. All the lists have extra space now and I am see fragments of sprites appearing.
<sinzui> There is a bug reported about the sprites, but I think we may want to fix the root cause ans revert it.
<henninge> sinzui: yes, I noticed those fragments, too
 * henninge starts firebug
<sinzui> I suspect it was EdwinGrubbs's change to wrap the action menus, but since his intent was to localise the affect to the action menu, I do not think any other list should have been affected
<henninge> sinzui: so the extra vertical space in the featured project list is the addition of padding-bottom and margin from these two styles. http://paste.ubuntu.com/320918/
<henninge> sinzui: making it 1.05em total
<henninge> that is in style-3-0.css
<sinzui> That is too must but those changes are older that a few weeks old
<sinzui> To review changes to those classes, look at the index page for products and person. The <object> information portlets use these lists a lot
<henninge> sinzui: I just think that the featured project list might be a different case and I'd rather just reset that padding-bottom locally.
<sinzui> henninge: I think we *may* want to separate `.two-column-list li` from `.two-column-list dl` I think the <dl> has too much space when I look at my profile (https://edge.launchpad.net/~sinzui), so I am in favour of find on rule for both conditions
<sinzui> henninge: https://edge.launchpad.net/gdp also shows the same issue (because it is the same markup). I think I expect to see something about 0.5em
 * henninge looks
<sinzui> Since we use 1em for  to separate blocs, lists must use less to show that they are together
<henninge> sinzui: that makes sense
<henninge> sinzui: I wonder what the general "li: padding-bottom 0.3 em" is about
<sinzui> I think it the number that is the expected separation, They are not sentences in a paragraph, so there should be separation. I think that number is the actual number that was approved, where my head thinks the number is 0.5em
<henninge> sinzui: so it is the extra 0.75em from the two-column-list definition that cause the problem.
<gary_poster> maxb: got an official +1 on ~launchpad-committers for lp-source-dependencies, meta-lp-
<gary_poster> debs, and most/all other important branches.  PPA will not be controlled with ~launchpad-committers, but going to give you individual permission to upload.
<sinzui> henninge: I think so.
<henninge> sinzui: also, I just saw that what we see on the prod/edge homepage is different to what we see on the dev homepage because on the dev homepage the images for the projects are displayed using sprites while the real projects have their own logos.
<henninge> sinzui: I see different styles used for sprites and for project logos.
<sinzui> yuck
<henninge> sinzui: the project logos seem to squeeze the lines further apart
<sinzui> henninge: That requires a tales fix. This issue is global to launchpad. I would not use the featured project list to test the fix
<henninge> sinzui: that said, removing the 0,75 em padding and leaving the 0.3 margin should give us the desired result with the real data.
<sinzui> henninge: Yes! your remark explained exactly what I have seen in some information portlets. I could not understand the extra px difference
<Ursinha> bigjools, I  haven't said hi in person to you :)
<henninge_> ah, the daily reconnect. Reminds me that it's time to go home ... ;)
<henninge_> sinzui: glad to hear I could help you
<sinzui> henninge: I am tempted to say the project sprite vs icon issue need to be fixed in a separate branch. It is certainly a separate bug. I know there are hacks in some information portlets that use clear:both to place the next item under the project with an icon
<henninge_> sinzui: Let me prepare this branch for review
<henninge_> sinzui: yeah, I am not going to to into that now. I just wanted to land a branch that was left over from last cycle ... ;-)
<sinzui> fab
<henninge> sinzui: I requested a review (the branch had been previously reviewed) but I don't think the test would pass. Actually, I don't see how it passed for you after the changes in the template.
<henninge> sinzui: I have to go home now, though. I will look into that tomorrow.
<henninge> also, bin/test segfaults for me atm.
<sinzui> henninge: make clean. make build; bin/test
<henninge> sinzui: tried it, didn't help
<henninge> sinzui: will try again tomorrow
<jelmer> bigjools, al-maisan, noodles775: Is there a new Soyuz hacking room?
<al-maisan> jelmer: we are on level 2, rooom waverly
<al-maisan> *room
<al-maisan> :P
<maxb> gary_poster: ooh, great! Let me know when it happens and I'll patch up the stacked-on urls of the reassigned meta-lp-deps branches, and reassign my ~maxb branch for the ppa python-defaults over to the team
<mwhudson> hello
<mwhudson> make is bitching at me about PasteDeploy
<mrevell> flacoste, ping
<mwhudson> what does this mean again?
<mwhudson> i know it's happened to me before
<flacoste> hi mrevell
<mrevell> hi flacoste
<mrevell> hey, how can I help flacoste?
<flacoste> hi mrevell: we'll be having 4 hours of codehosting downtime on Thu from 13:00UTC to 17:00UTC
<mwhudson> we will?
 * mwhudson hopes this is something to do with extremely large amounts of disk
 * wgrant wonders why the reasonable notification period for extremely long downtime is not longer.
<jml> wgrant, because we like to do things quickly
<wgrant> jml: This is in conflict with having a reputation as a vaguely reliable hosting provider.
<wgrant> Which is certainly not a reputation that you have now.
<jml> gary_poster, hello
<gary_poster> jml hi
<jml> gary_poster, if I set up some casual contributors with a pre-2.5 Launchpad, how hard a time will they have updating their Launchpad?
<gary_poster> jml, I don't think it will be too bad as long as they do a ``make clean`` before the ``make``
<wgrant> That's all I had to do.
<wgrant> They shouldn't run into the develop-eggs problem.
<ajmitch> even that shouldn't be needed if the first thing they do is run rf-get?
<jml> gary_poster, thanks.
<gary_poster> jml, sure
<cody-somerville> bigjools, I updated the branch with the changes bac requested.
<bigjools> cody-somerville: you're sat 10 feet away from me
<cody-somerville> bigjools, I didn't want to interrupt your conversation ;)
<jelmer> rockstar, ping
<cody-somerville> although its rather quiet at the moment
<rockstar> jelmer, hi
<wgrant> bigjools: Since the ec2 images apparently aren't 2.5-ready yet, do you have access to your monster of a home machine to test and land my branch?
<gary_poster> wgrant: they should be ready
<wgrant> sinzui: ^^
<gary_poster> wgrant: did you encounter a problem?
<bigjools> wgrant: I don't :(  I need to sort out wake-on-lan so I can turn it on remotely
<jelmer> rockstar, you approved lp:~jelmer/launchpad-cscvs/converted-from earlier. Can I land that or do I need more reviews?
<bigjools> wgrant: but I am sure someone like al-maisan will run it for you :)
<jelmer> rockstar, (the merge proposal status is still needs-review)
<rockstar> jelmer, if you can land it yourself, have at it.  I have it on my list of things to do to land it for you.
<al-maisan> wgrant: what's the branch?
<rockstar> jelmer, ah yes, I hate that status and vote are so disconnected.
<wgrant> al-maisan: https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection-part1/+merge/14729 -- contrary to what the MP says, it's now targetted at db-devel.
<wgrant> But LP doesn't appear to model that.
<ajmitch> thumper: is https://blueprints.edge.launchpad.net/launchpad-code/+spec/branch-merge-bot similar to the idea from the other day that I talked to you about?
<jelmer> rockstar: I'm not sure whether I have access, I'll give it a try
<al-maisan> wgrant: could you please provide me with a submit message?
<lifeless> intellectronica: the package is python-subunit
<rockstar> jelmer, no harm in trying.
<wgrant> al-maisan: Hm "Bits and pieces to prepare for Debian source format 3.0 support." is the only succinct way to put it.
<al-maisan> wgrant: also, is there a bug open that this branch is fixing?
<al-maisan> :)
<wgrant> al-maisan: Not directly, no.
<al-maisan> OK
<jelmer> rockstar: looks like I indeed don't have access, I'll leave it up to you
<rockstar> jelmer, okay.  I'll make sure to land it today.
<jml> is there a package of lazr.restfulclient 0.9.10 anywhere?
<wgrant> Oh, I see.
<wgrant> (that Soyuz is back up to four now)
<bigjools> \o/
<wgrant> Very good.
<thumper> ajmitch: on a call, talk to you soonish
<ajmitch> np, it can wait :)
<jml> python setup.py install on trunk lazr.restfulclient fails with error: Can't download http://bitworking.org/projects/httplib2/dist/httplib2-0.4.0.tar.gz: 404 Not Found
<rockstar> ajmitch, I think that blueprint accurately describes Tarmac.
<rockstar> :)
<rockstar> (minus the branch merge queue stuff which doesn't currently exist in the UI)
<ajmitch> rockstar: right, I was talking to thumper about the UI stuff to start with :)
<ajmitch> then went & looked at existing blueprints
<al-maisan> wgrant: your branch is running in ec2 now, if all goes well it should land in approx. 3.5 hours.
<sinzui> EdwinGrubbs: bac: Last review period we closed 201 registry bugs. This period we closed 457.
<wgrant> al-maisan: Thanks.
<al-maisan> you are welcome
<EdwinGrubbs> sinzui: how?
<sinzui> Better cadence. The trivial policy closed 60 bugs.
<jml> gah
<jml> how do I run the tests in launchpadlib?
<jml> given that https://bugs.edge.launchpad.net/launchpadlib/+bug/316694 is just adding a regex and a property, I'd like to patch it
<mup> Bug #316694: Add web_link property to resources <launchpadlib :Triaged> <https://launchpad.net/bugs/316694>
 * thumper is going make run with py2.5 for the first time
<jml> but not being able to run the tests is a pain
<thumper> jml: ask leonardr
<jml> leonardr, ^
<mwhudson> make is still breaking for me
<mwhudson> man i want subscribe to tag
<thumper> it works \o/
 * thumper goes to merge devel into current work
<mwhudson> ah, it works for me, after i make clean a bit harder
<mwhudson> phew
<leonardr> jml: you need to run the tests from within launchpad
<leonardr> 1. check out a launchpad instance
<leonardr> ln -s /path/to/dev/launchpadlib ./launchpadlib
<jml> leonardr, ok thanks.
<bigjools> jelmer: that bug you've picked is kinda hard actually
<cody-somerville> bigjools, do retried builds get the manual bool set to true or something?
<bigjools> cody-somerville: I don't remember off hand I'd need to look in the code
<jml> leonardr, like this? jml@truth:~/src/launchpad/stable$ ln -s /home/jml/src/launchpadlib/trunk/ launchpadlib
<leonardr> jml: sorry, i had to fry something
<jml> leonardr, understood.
<leonardr> yeah, whatever launchpadlib branch you've changed
<leonardr> 3. change buildout.cfg from "." to "\t.\n\tlaunchpadlib"
<leonardr> 4. run bin/buildout
<leonardr> then run bin/test -vvt launchpadlib. you should see tests in your dev launchpadlib being run
<jelmer> bigjools: so I'm finding out :-) I'll look for an easier one
<bigjools> jelmer: yeah, it needs to join on all the PackageUpload* and use the right one :)
<jml> leonardr, ok. thanks.
<bigjools> jelmer: I think I already said, but anything marked trivial is a good start, but ask first because trivial is not necessarily an accurate tag for Soyuz :)
<thumper> bigjools: I need to talk to you
<thumper> bigjools: can you go skype somewhere?
<bigjools> thumper: I can, gimme 2 mins
<thumper> bigjools: ok
<bigjools> thumper: or twinkle!
<thumper> bigjools: I could twinkle
<bigjools> I've no idea what my number is
<jml> is this known? http://pastebin.ubuntu.com/321135/
 * jml afk for a bit
<bigjools> thumper: I am 7145
<rockstar> I think I'll finish my current work before I try and merge 2.5 into it.
<bigjools> thumper: dial meh
<rockstar> Although I've taken about 2000 lines of js and have it condensed it down to about 600 lines of maintainable code.
<leonardr> jml, i recently gave my father-in-law a book written by a guy named john lange in the 19th century
<mwhudson> jelmer: hello
<mwhudson> jelmer: what versions of subvertpy/bzr-svn should we use in launchpad?
<mwhudson> (haven't researched this at all myself, so apologies if the answer is obvious)
<thumper> bigjools: it tells me you are refusing my calls ;-)
<bigjools> thumper: and it tells me you're not answering :)
<bigjools> riiiiiiiing
<cody-somerville> bigjools, I'm getting the impression that what happens is that retried builds simply don't get scored at all so are 0 by default.
<bigjools> cody-somerville: correct!
<jelmer> mwhudson: hi
<mwhudson> jelmer: hello
<jelmer> mwhudson: subvertpy 0.7.0 and the latest version of bzr-svn
<mwhudson> jelmer: as in lp:bzr-svn tip?
<jelmer> mwhudson: yeah, sorry - the latest revision
<mwhudson> jelmer: ok thanks
<jelmer> mwhudson: jszakmeister provided a fix that makes bzr-svn a bit friendlier to svn servers when retrieving logs
<jelmer> mwhudson: that fix is not in any released versions yet
<mwhudson> jelmer: ok
<mwhudson> jelmer: will this version work with both bzr 2.0.0 and 2.1b3 ?
<jelmer> yeah
<mwhudson> thats what i wanted to hear, thanks :)
<mwhudson> mwh@grond:trunk$ bzr push -r tag:subvertpy-0.7.0 ../subvertpy-0.7.0
<mwhudson> Created new branch.
<mwhudson> mwh@grond:trunk$ cd ../subvertpy-0.7.0
<mwhudson> mwh@grond:subvertpy-0.7.0$ bzr push
<mwhudson> bzr: ERROR: Working tree "/home/mwh/canonical/repos/subvertpy/subvertpy-0.7.0/" has uncommitted changes (See bzr status). Use --no-strict to force the push.
<mwhudson> wtf!!
<mwhudson> that really doesn't make any sense does it?
<rockstar> mwhudson, you're sure that dir didn't exist before?
<jelmer> mwhudson, it doesn't indeed - what changes are in that tree according to bzr st?
<mwhudson> rockstar: the "created new branch" suggests not
<mwhudson> jelmer: http://pastebin.ubuntu.com/321146/
<rockstar> mwhudson, well, maybe it pushed a branch inside that dir.
<rockstar> so you have ../subvertpy-0.7.0/trunk
<mwhudson> rockstar: not this time
<mwhudson> jelmer: it looks like the wt is the same as trunks?
 * rockstar should not assume everyone is as silly as he is
<mwhudson> yeah
<mwhudson> rockstar: well, it was a reasonable theory
<mwhudson> just not the right one this time :)
 * jelmer is sure we're all missing something obvious here
<jelmer> mwhudson: what version of bzr are you using?
<mwhudson> jelmer: 2.1.0dev3
<mwhudson> push -r 3 does even more odd things:
<mwhudson> mwh@grond:trunk$ bzr push -r 3 ../r3
<mwhudson> Path conflict: PLAN / PLAN
<mwhudson> Path conflict: __init__.py / __init__.py
<mwhudson> Path conflict: svnbranch.py / svnbranch.py
<mwhudson> 3 conflicts encountered.
<mwhudson> Created new branch.
<mwhudson> (whatever is in ~bzr-daily-ppa or whatever it's called)
<jelmer> mwhudson: that's really odd
<jelmer> mwhudson: those are the files that existed in r1
<mwhudson> let me try bzr tip i guess
 * spm waves hi to jelmer while passing thru
<mwhudson> it really seems like push -r $foo is behaving like push of tip and merge back to requested rev
<mwhudson> no not tat
<mwhudson> bzr tip does the same
<mwhudson> let's try the 2.0 branch...
<mwhudson> wtf, that's the same!!
<cody-somerville> bigjools, how often does cronscripts/buildd-queue-builder.py run?
<spm> cody-somerville: every 20 minutes; --score-only fwiw.
<cody-somerville> bigjools, so retried builds don't stay scored at 0 for very long. Is that a bug or intentional?
<bigjools> cody-somerville: intentional
<bigjools> but tbh b-q-b makes no difference
<jelmer> spm: hey
<thumper> bbq?
<bigjools> stfu mofo
<thumper> mofo?
<bigjools> bmx
<cody-somerville> bigjools, what do you mean by it makes no difference?
<thumper> wtf
<spm> nfi
<bigjools> cody-somerville: the rescoring makes no difference to the order that builds get done
<wgrant> The time value component of the build score is tiny.
<cody-somerville> bigjools, scores makes no difference to the order that builds get dispatched?
<bigjools> cody-somerville: they do, but the auto-rescoring doesn't change the situation in practice
<bigjools> because it's time-based
<cody-somerville> it makes a huge difference because after the first run of b-q-b the queue record is scored as if it was normal
<cody-somerville> so it only delays retried builds in any meaningful manner for no more than 20 minutes ever
<bigjools> that should not happen
<cody-somerville> then you haz a  bug
<bigjools> yarp
 * bigjools looks for powah
#launchpad-dev 2009-11-18
<maxb> why do we even *have* time-based rescoring?
<maxb> It seems a rather pointless concept given the queues are mostly FIFO anyway
<cody-somerville> maxb, they're not FIFO at all
<wgrant> The order seems to be based on (score, id)
<wgrant> So apart from score, they are FIFO.
<wgrant> And most builds have the same score.
<maxb> ok, that was an oversimplification. But the increments added by the time-based rescorer are smaller than most of the other score-determining factors
<wgrant> (except for a few which differ slightly on the distro builders, and private PPAs will get a 10000 point bonus)
<cody-somerville> I don't think anyone disagrees that scoring needs work
<bigjools> right, the small increments don't make enough of a difference to change the order
<cody-somerville> it also stops making a difference after 4 hours of waiting
<wgrant> The entire build dispatch algorithm needs to be redone. Trying to fix scores is unlikely to do anything.
 * mwhudson lunches (and is setting up a new modem so will be mostly offline)
<thumper> mwhudson: ping
<wgrant> Hmm, interesting.
<wgrant> I cannot access +index of some projects. I suspect this is because the development focus branch is private.
<thumper> ??
<thumper> arse
<thumper> wgrant: can you please give me an example?
<thumper> wgrant: and file a bug?
<thumper> wgrant: I'll fix asap
<wgrant> thumper: eg. https://launchpad.net/landscape. I don't have an OOPS code.
<wgrant> I will file a bug.
<wgrant> thumper: Registry or Code?
<thumper> code
<wgrant> thumper: Bug #484533
<mup> Bug #484533: Cannot access index for project with private development focus branch <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/484533>
<thumper> wgrant: thanks
<mwhudson> well that was exciting
<mwhudson> plug in new modem, doesn't work, plug in old modem, doesn't work
<thumper> mwhudson: what's that?
<mwhudson> phone isp tech support
<thumper> :(
<mwhudson> "it seems your account was suspended because of reports of spam last week"
<mwhudson> !!!!
<wgrant> Ouch.
<thumper> mwhudson: so now you have in the new modem?
<mwhudson> thumper: no, still the old
<mwhudson> and indeed, using 32 gigs of traffic in the first 18 days of the month is a bit unusual
<thumper> oh
 * thumper stabs windmill
<thumper> my windmill tests are failing to use the api!!!
<thumper> oh FFS
 * thumper frowns
 * thumper frowns more
<thumper> AppServerLayer not tearing down
<thumper> stabby stabby
 * ajmitch steps away from the knives
<mwhudson> after all that
<mwhudson> lunch, finally
<thumper> ââ¹
<wgrant> thumper: Windmill still killing you(ll)?
<thumper> yes
<thumper> works locally, but not in windmill
<bac> jml: yes, i've seen those import violations today since going to 2.5
<thumper> recipe would live at https://launchpad.net/ubuntu/karmic/+source/some-package/+recipe/recipe-name
<Ursinha> hey stub, when you return, can we chat a bit?
<stub> I'm here
<lifeless> thumper: no team/person in there ?
<thumper> lifeless: not in the url
<thumper> lifeless: but on the object, yes
<thumper> lifeless: do you think it is really needed?
<lifeless> I don't know
<mwhudson> hooray
<stub> Ursinha: ^^
<Ursinha> hi stub
<Ursinha> stub, what should I do after you approve my db-patch and give me a patch number?
<stub> If I and jml have both approved it, you need to land it on lp:~launchpad-pqm/launchpad/db-devel (with the correct patch number if you haven't changed that on your branch yet).
<Ursinha> stub, but should I change from pending folder or just rename it?
<stub> Ursinha: ^^
<Ursinha> (and  change the line inside of the patch to contain the correct number)
<stub> Oh - it needs to be in database/schema.
<stub> Most people keep them in there - pending tends to only be used by me.
<Ursinha> stub, anything else?
<stub> Nup. If the tests pass its fine.
<wgrant> stub: If I want to get some production data trivially fixed, how do I go about it?
<Ursinha> stub, I did what you said, and I'm getting this: https://pastebin.canonical.com/24833/
<wgrant> Get the query approved by the appropriate team and you?
<Ursinha> stub, what am I doing wrong?
<stub> wgrant: Sounds good. If it is really trivial I can just do it.
<stub> Ursinha: The last statement of the DB patch should be "INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 10, 0);".
<Ursinha> stub, I've changed that
<stub> Ursinha: Please pastebin the db patch
<wgrant> stub: There are some thousands of old sourcepackagereleases with a null dsc_format. None of these have been created since 2006. Until now, only the '1.0' format has been accepted, so all the old SPRs should have their dsc_format set to that.
<Ursinha> stub, it's in the other computer, a minute
<stub> wgrant: I recall that. Sounds fine to me but this one I'd want soyuz team to agree too, just in case there is some need to differentiate records made before the code that added that field landed.
<stub> Which is a stretch, but better safe than fired.
<wgrant> stub: Indeed. bigjools seemed happy with it last week, but I will get more concrete approval this evening.
<wgrant> Thanks.
<stub> We also need a bug opened to set the column NOT NULL and targetted to this cycle.
<wgrant> Shall I prepare and get landed a DB patch for that?
<ursula> stub, http://paste.ubuntu.com/321244/
<stub> wgrant: If you want, or I can just append it to my pending db stuff branch.
<wgrant> stub: That would be handy. Filing a bug now.
<wgrant> stub: bug #484602
<mup> Bug #484602: SourcePackageRelease.dsc_format should be NOT NULL <Soyuz:New> <https://launchpad.net/bugs/484602>
<wgrant> stub: I cannot target or anything.
<stub> I've got it
 * mwhudson afk for a few minutes
<mwhudson> is there a word for "tree that you could plausibly try to run debuild in" ?
<mwhudson> i.e. source tree, debian/ dir etc
<lifeless> source tree
<lifeless> no, you don't run it in the debian dir, rather in the top dir
<lifeless> you could coin a word
<lifeless> debianized source tree
<lifeless> is probably what folk would understand
<mwhudson> "debianized source tree" works
<wgrant> '[unpacked] source package' is my usual term.
<wgrant> But I like 'debianized source tree'
<mwhudson>  Results 1 - 10 of about 11,800 for "debianized source tree".
<spm> debianised perhaps?
<lifeless> spm: sadly no, I'm not making this up :)
<lifeless> spm: or I would have used an s
<mwhudson> spm: "debian" CERTAINLY did not arrive in the english language via french
<lifeless> mwhudson: but did deborah and ian ?
<spm> I knew I should have put the evil/ironic smiley in.... ;-)
<mwhudson> lifeless: deborah doesn't sound like it does it?
<mwhudson> ian i guess might have done
<mwhudson> hah, debian policy says debianised
<spm> *snort*
 * spm is reminded of the wikipedia rules on UK vs USA spelling :-D
<spiv> mwhudson: well, if it's *policy* then you have no choice...
<mwhudson> i think i shall alternate between the two in this mail and see how well that troll works
<spiv> mwhudson: pick a middle ground: "debianiced"
<mwhudson> spiv: that's a strange middle you're standing on
<spiv> mwhudson: it's the sort of middle that makes no-one happy!
<mwhudson> debianated
<mwhudson> debianorized
<spiv> debianificated
<mwhudson> endebianned
<spm> mwhudson: bigendebianned ???
<mwhudson> spm: that's clearly much too silly
<spm> sorry
<spm> will aim for a more moderate level of silliness next time
<spm> 0xdebianised ??
<mwhudson> spiv, lifeless: are you guys on launchpad-dev?
<spiv> You mean this channel?
<lifeless> mwhudson: are you human?
<mwhudson> spiv: no, the mailing list?
<lifeless> I think I am
<spiv> Probably.  I certainly get lots of mail about Launchpad...
<lifeless> is there something we should have seen and replied to?
<mwhudson> lifeless: the mail i'm about to send
<lifeless> my advice is that if you want $foo to see mail, CC them always.
<lifeless> on-a-list doesn't guarantee 'pay attention'
<mwhudson> yeah, i was thinking that
<spm> jtv: heyo! can I draw your attention to https://answers.edge.launchpad.net/rosetta/+question/89602 pls. Is this (1) a sane request and (2) If Y, how? :-)
<jtv> spm: looking...
<spm> ta
<jtv> It sounds like there was some mistake... pt and pt_BR are different enough that we treat them as separate languages.
<jtv> It's pretty exceptional for us to do that; the differences between country variants of Spanish can be pretty huge but people manage to keep those translations unified.  From that I infer that it really couldn't be done for pt and pt_BR.
<jtv> spm: I'll follow up.  I've had a fair amount of interaction with these people.
<spm> jtv: ta
<spm> jtv: fwiw, from ex-brazilians I've worked with in previous jobs (vs the wonderful Brazilians I work with now! ;-) ) They speak in quite ... derrogatory terms of pt vs pt_BR :-)
<mwhudson> my limited understanding was that they're basically different languages
<jtv> spm: to make sure I follow... they hate pt, right, not the distinction of pt vs pt_BR?
<spm> the descriptive analogy I got was pt to pt_BR is like 12th century english to modern english.
<spm> jtv: exactly.
<jtv> They may have gotten pt translations imported as pt_BR by accident, or vice versa.
<mwhudson> spiv, lifeless: mail sent fwiw
<spiv> mwhudson: ta, will take a look
<wgrant> mwhudson: Does anybody have any idea at all how private branches are going to be done?
<mwhudson> wgrant: you don't mean private branches there do you?
<mwhudson> wgrant: but i don't think so, no
<wgrant> mwhudson: I mean BFB from private branches.
<mwhudson> wgrant: ah
<mwhudson> wgrant: then no
<mwhudson> i guess it'
<wgrant> The only possibility that is obvious to me is to have them served over internal HTTPS, and have buildd-manager grant and revoke credentials.
<mwhudson> s worth thinking about a bit so we don't paint ourselves into a corner, but right now BFB at all is looking hard enough to be starting with
<mwhudson> oh at that level
<mwhudson> yes, something like that
<mwhudson> doesn't really have to be https though does it?
<wgrant> Probably not, as the buildd network seems rather secure. But it still seems like a Good Idea, and that's how it's done with P3As.
<wgrant> What other problems are there with private BFB?
<mwhudson> i guess there must be some inspiration to be found in the restricted librarian
<wgrant> mwhudson: The restricted librarian is not useful here.
<mwhudson> wgrant: well, P3Asa at least then
<wgrant> Right.
<mwhudson> wgrant: well, i'm imagining stuff like preventing people building a private branch into a public pps
<mwhudson> ppa even
<mwhudson> anyway it's late and i need to get away from the computer
<wgrant> It would be nice to design this properly from the start, so there isn't a crazy situation like there is with P3As, where file retrieval is completely different from public PPAs.
<wgrant> Indeed.
<mwhudson> wgrant: feel free to reply to my mail
<mwhudson> i don't know how P3As work, i guess i should find out
<wgrant> mwhudson: They are accessed from the archive with static buildd credentials over HTTPS. The master gives the slave a set of URLs with credentials embedded.
<mwhudson> wgrant: oh, for when a package build-deps on things in the P3A?
<wgrant> mwhudson: That's separate.
<mwhudson> wgrant: then i'm not sure what you'
<mwhudson> re talking about, but as my brain has turned to runny cheese and dribbled out of my ears, perhaps that's ok
<wgrant> mwhudson: For public archives, the slave is given a list of SHA1s and it is up to it to grab them from the librarian.
<wgrant> For private archives, the slave is given a list of HTTPS ppa.launchpad.net URLs with embedded credentials.
<wgrant> (this is for retrieving the source package, not deps)
<wgrant> But anyway, late.
<mwhudson> oh that
<mwhudson> accessing librarian files by sha1 seemed a little bonkers to me tbh
<wgrant> It's ancient Soyuz stuff. It is expected to be bonkers.
<spm> maxb: ref https://answers.edge.launchpad.net/launchpad/+question/89097 you didn't actually click on the proffered link did you? :-) And seeing as I'm about to remove that response, did you want me to tidy yours away as well?
<adeuring> good morning
<henninge> bin/test is segfaulting for me
<henninge> I did make clean ; make build but it did not help ...
<henninge> make run works fine btw
<allenap> henninge: Can you paste the full error?
<allenap> Morning adeuring :)
<adeuring> hi allenap!
<henninge> allenap: on the command line it is just "segmentation fault"
<henninge> allenap: I am currently tracing into it
<henninge> allenap: it happens on "from canonical.testing.layers import *" in testgin/__init__.py
<henninge> "from canonical.launchpad.ftests import ANONYMOUS, login, logout, is_logged_in" in layers.py
<allenap> henninge: Oh, not nice. Have you tried a "bzr clean-tree --ignored --force" for each directory in sourcecode/ ?
<henninge> tracing deeper ...
<thumper> make cleaner?
<thumper> heh
<thumper> sorry
<allenap> henninge: And then a rebuild?
<henninge> allenap: no, let me try that
<allenap> thumper: :)
<henninge> allenap: looking into sourcode I found an invalid symlink for lsprof
<allenap> henninge: Mmm, link-external-sourcecode should remove dead symlinks in sourcecode/
<wgrant> Why do dead symlinks matter?
<wgrant> update-sourcecode will remove obsolete sourcecode branches?
<wgrant> s/?/./
<henninge> wgrant: what's "update-sourcecode"? Is that run by rocketfuel-get?
<wgrant> henninge: Yes.
<wgrant> henninge: It grabs new sourcecode branches (in lp-sourcedeps), updates existing ones, and removes old ones.
<henninge> I did run it recently but I'll try and run it again
<henninge> wgrant, allenap: unchanged :(
<henninge> "make build" includes this line:
<henninge> lib/twisted/web/http.py:40: RuntimeWarning: Python C API version mismatch for module _c_urlarg: This Python has API version 1013, module _c_urlarg has version 1012.
<allenap> henninge: bac had a similar problem yesterday. IIRC, he did:
<allenap> henninge: . ~/.rocketfuel.env
<allenap> henninge: cd $LP_SOURCEDEPS_PATH/twisted
<allenap> henninge: bzr clean-tree --ignored --force
<allenap> henninge: make PYTHON=python2.5
<allenap> henninge: The Twisted Makefile has python2.4 hardcoded :(
<henninge> ah, I see
<henninge> allenap: let me try that
<gmb> I come to learn things.
<allenap> henninge: Oops, that should be: . ~/.rocketfuel-env.sh
<henninge> allenap: ;) saw that
<allenap> Cool.
<henninge> allenap: after that a "make build" again?
<allenap> henninge: I don't think it should be necessary, but it'll do no harm I guess.
<henninge> allenap: you are my hero today!
<henninge> :-D
<henninge> now the import facist is complaining ...
<henninge> oh, that is after the test
<allenap> henninge: Woohoo \o/ :)
<henninge> allenap: thank you
<allenap> henninge: You're welcome.
<allenap> gmb: Any luck?
<gmb> We'll know in a second...
<gmb> allenap: Wewt, etc. Thanks.
<allenap> gmb: Cool. I'll email the list with this too.
<maxb> spm: (Re: https://answers.edge.launchpad.net/launchpad/+question/89097) Um, proffered link? I don't remember there being a link. Yes, it would make sense to tidy away my response since the thing I was responding to isn't there any more.
<thumper> mwhudson: you shouldn't be replying to email at 40m past midnight
<thumper> mwhudson: it sets a bad precident
<thumper> :)
<sinzui> bac: does bug 483607 need updating
<mup> Bug #483607: The person profile page still shows the image and link to set somone's location <Launchpad Registry:In Progress by bac> <https://launchpad.net/bugs/483607>
<bac> sinzui: done
<sinzui> sweet. The page looks much better without a empty map section
<sinzui> bac: EdwinGrubbs: stand-up in 2 minutes
<jml> where do the doctest utilities live?
<jml> canonical/launchpad/testing/pages.py
<jml> gmb, hello
<gmb> jml: Hi
<jml> gmb, I came across a XXX comment of yours that says we won't need canonical.launchpad.testing.systemdocs.ordered_dict_as_string once we've switched to Python 2.5 because dict ordering is guaranteed when __str__() is called
<gmb> jml: Right.
<gmb> I assume that I research that before inserting the XXX...
<jml> gmb, I'm trying to find a citation before I kill it.
<gmb> Thought you might be.
<gmb> :)
<jml> gmb, heh
<jml> http://www.python.org/doc/2.5.4/lib/typesmapping.html doesn't say much about it.
<gmb> jml: I'm sure I looked it up, but I can't remember where I got the information from.
<gmb> I should have cited it in the XXX, sorry.
<jml> gmb, np.
<jml> I'll ask on #twisted
<jml> man
<jml> thing that sucks about doctests
<jml> we have a proliferation of dict-printing helpers
<jml> all of which are subtly different.
<jml> http://www.python.org/doc/2.5.4/lib/doctest-warnings.html
<mars> jml, makes you wonder if we should just define a sensible __str__() on the objects we want to print, or maybe even start using pprint inside __str__()?
<mars> unless you are talking about printing true dictionary objects
<jml> mars, actually it makes me wonder why people continue to think doctests are a good idea
<mars> I though of that too, I just didn't say it :)
<jml> mars, we should have useful __repr__() methods for more of our objects definitely
<jml> mars, in this particular context, I'm printing a thing of type dict though.
<mars> ah, so no way around that then.
<mars> just realized something.  BDD and cucumber lead to the same proliferation, but instead of dict output formatters, they lead to a proliferation of natural language interpreters.
<mars> either way, you have a bunch of helpers to translate to and from plain text
<mars> might just be necessary for the medium
<jml> meh
<jml> I still prefer building a DSL in the programming language
<jml> I am bigoted against executable natural language.
<bigjools> james_w: are package recipes tied to a distro or are they more generic?
<sinzui> bac: are you available for our weekly call?
<[reed]> gmb: huh, I had never heard of doctest before
<[reed]> neat
<[reed]> I don't get to do much python coding in my current job
<gmb> [reed]: Yeah, we use them a lot. It's a nice way of keeping strong documentation, but it's not always terribly useful. We still love good old-fashioned unit tests for a lot of things.
<[reed]> hehe
<[reed]> yeah
<gmb> So, things with lots of iterations and lots of nuanced behaviour == unittests
<[reed]> as much as it is a pain to write tests, they can be so useful :)
<gmb> [reed]: Before I can land this, you need to have sent in your contributor agreement form. Have you done that?
<[reed]> gmb: no, I didn't know there was one. If you can point me to where it is, I can probably get that done in the next hour or two, depending on what form I have to send it in.
<bac> sinzui: i am now.
<gmb> [reed]: https://dev.launchpad.net/ContributorAgreement
<gmb> [reed]: Just ping me when you're done.
<[reed]> literally, that was my first lp bzr branch ever ;)
<sinzui> bac: calling
<[reed]> ah, it's just a simple e-mail
<[reed]> ok, it's a bit scary that in less than 30 min. after I post that bzr branch, the google alert for my name goes off and informs me of it
<gmb> [reed]: If you don't contribute anything else in six months you get an email that says "we know where you live," too.
<[reed]> lol
<[reed]> gmb: ok, read the agreement and sent the e-mail... I cc'd you, as per the instructions.
<gmb> [reed]: Brilliant, thanks. I'm working on the tests now. Should be able to land it tonight, or tomorrow at the latest.
<[reed]> cool
<sinzui> bac: https://edge.launchpad.net/launchpad-registry/3.1
<al-maisan> beuno: just letting you know: jml is on level 3
<gary_poster> maxb, From following the instructions I was given, I believe you should be able to upload to the PPA.  Let me know how it works.
<beuno> al-maisan, thanks
<salgado> mars, what revno of lazr-js are you using in your yui3-final-upgrade branch?
<mars> salgado, 153
<mars> salgado, but that can be updated as needed.  Downgrading is a bit tricker.
<salgado> mars, ok, just wanted to check we'd be using tip
<salgado> mars, btw, all that I'm going to integrate is the combo loader -- flacoste's css minifier will come together as a bonus?
<mars> salgado, yes, I believe it was merged during the sprint.
<salgado> yeah, it was
<salgado> mars, do you have an idea how that's going to be integrated into LP?  if not, I guess flacoste has
<mars> salgado, I have no idea how.
<salgado> flacoste, I'm going to work on integrating the combo loader into LP, and I assume you've got an idea how that should be done, so was hoping to have a pre-impl call with you. ;)
<maxb> gary_poster: Nothing to upload right now, but the Launchpad web ui claims I can upload. :-)
<gary_poster> maxb: awesome :-)
<maxb> I guess I could start populating it for Lucid :-)
<maxb> But actually many of the packages are obsolete now we don't need python 2.4 support
<maxb> I should make a list and propose deleting them at some point
<mars> salgado, presumably the combo loader will be a stand-alone process like the librarian.  It will also need access to all of our static files, and I don't know how that part would be linked into the system.
<maxb> Is it just me or has someone broken the page layout on https://dev.launchpad.net/ ?
<maxb> oh, it's just that it wraps amazingly poorly if your browser window isn't wide enough
<maxb> hm, has someone changed something about the way drop-down select boxes get handled on edge?
<maxb> I'm seeing a major performance regression with drop-downs taking several seconds to display
<mars> maxb, which page?
<maxb> ppa pages
<salgado> mars, yeah, that sounds reasonable
<mars> maxb, you are right.  I wonder why that is.
<maxb> gary_poster: Also, I appear to have upload rights but not delete-package rights, oddly
<maxb> mars: Hmm. I see the problem on the ppa root page, and on /+copy-packages, but *not* on /+packages, oddly
<mars> maxb, it may be the JavaScript blocking the main browser thread :(
<maxb> ETOOMUCHJAVASCRIPT :-(
<mars> I can't reproduce it.  It is only on initial page load
<mars> maxb, not too much, just doing the wrong thing at the wrong time.
<maxb> oh dear. heisenbugs are not fun. It seems to be consistently affecting me
<mars> maxb, the other explanation is something not downloading fast enough.  On my page load, I see a long, long blocking on laucnhpad.js and the arrowBlank images.
<maxb> surely that shouldn't be an issue after navigating around a few launchpad pages?
<mars> if it is cached properly, then no, it should not be an issue.
<mars> maxb, since this is a good reproducible case, would you mind filing a bug about it?  And please note the exact page(s) you are visiting, and the browser you are using.
<mars> maxb, I'd like to work on front-end performance at some point, and this would be a good point to cut my teeth on :)
<maxb> Hmm, it seems to only occur on PPAs with a non-trivial number of packages
<mars> I had it happen on a PPA with two packages.  But only the first time.
<flacoste> salgado: suer, let's have a call
<salgado> flacoste, I'm ready
<al-maisan> hello salgado, do you have a minute? I'd like to ask your advice re. a branch I am currently working on.
<salgado> hi al-maisan.  what's up?
<al-maisan> I need to expose a method on the LP API and that new method would use a class that's currently in the browser layer (ProxiedLibraryFileAlias)
<al-maisan> the new method will be part of a model class
<al-maisan> so, not quite sure what the best approach is
<al-maisan> salgado: using browser-layer code in a model class is obviously less than ideal
<salgado> indeed
<salgado> al-maisan, do you really need to expose ProxiedLibraryFileAlias or do you just want people to have access to files in the restricted librarian?
<al-maisan> salgado: to be more concrete, sourceFileUrls() (http://pastebin.ubuntu.com/321744/) will need to use ProxiedLibraryFileAlias
<salgado> oh, I didn't understand it at first
<al-maisan> salgado: we can't throw librarian URLs back since these may be *restricted* librarian URLs in case of private PPAs
<salgado> I thought you wanted to expose ProxiedLibraryFileAlias, but you just want to expose a method that happens to use ProxiedLibraryFileAlias
<al-maisan> salgado: right!
<al-maisan> salgado: bigjools just pointed me to an example where we already use ProxiedLibraryFileAlias in the model layer
<salgado> oh, and ProxiedLibraryFileAlias.http_url calls get_current_browser_request()
<al-maisan> but if there's a better way of doing it I am open to it
<al-maisan> the changes_file_url() property in lp/soyuz/model/publishing.py is using it already and it does not seem to be a problem
<salgado> al-maisan, it will be if someone uses that property in a script, for example
<gary_poster> maxb, I asked bigjools about you not having delete permission and he said it was expected/desired behavior.  Only PPA owners can delete.
<maxb> quirky that I can use copy-packages but not delete-packages
<maxb> doesn't really matter, I guess
<maxb> I guess copy-packages is 'sort of' an upload. If you squint a bit
<al-maisan> salgado: can we continue later? It's lunch time here..
<salgado> al-maisan, so, basically we're making model code depend on the current browser request because we can only expose things from the model in the webservice?
<salgado> al-maisan, and I was thinking you were working late!
<al-maisan> salgado: UDS :)
<salgado> al-maisan, sure thing, go enjoy some tex-mex food. ;)
<al-maisan> salgado: "we can only expose things from the model in the webservice" .. is this a fixed rule?
<al-maisan> .. or do you know of examples where non-model code was exposed via the LP API?
<salgado> al-maisan, no, I think that's just a limitation of our existing webservice
<al-maisan> salgado: hmm .. I'll try using ProxiedLibraryFileAlias in the method that is to be exposed (although it's less than ideal) and see how it goes..
<al-maisan> definitely lunch time now
<salgado> al-maisan, it will work fine until somebody comes along and try to use that method in a script
<al-maisan> right
<al-maisan> I see
<al-maisan> salgado: thanks for taking a look!
<al-maisan> ttyl
<maxb> Woo, just got my "you have been added to launchpad-committers"
<maxb> Though with none of the ~launchpad admins around, I guess it's not going to be actually have branches for a while yet
 * maxb reassigns some branches
<mars> salgado, around?
<salgado> mars, yep
<mars> salgado, in Francis' reply to me on lp-dev, the thread about "Redesigning lazr.restful", he said that you know about common cases for update pages via AJAX?
<mars> salgado, if so, I was wondering if you could reply to the thread with one of those cases, for the benefit of myself and other readers?
<salgado> mars, the one I know about is the one he mentioned
<salgado> Adding a team member inline
<mars> ok
<mars> so how does that involve updating multiple parts of the page?
<salgado> mars, would you like to know what needs to be updated in that case?  or other cases where this would be needed?
<mars> salgado, both
<salgado> mars, if the new member is a person, we add it to the 'Recent members' section and update the count of active members
<salgado> if it's a team, we update the count of invited members
<salgado> but if the new member was a proposed/inactive member, then we need to update two lists and two counts
<mars> oh?
<salgado> moving the member from the 'pending members' section to the 'recent members' one
<mars> just thinking, that leads to a problem.  Using francis' scheme, how do you know when to ask for one, two, or many fragments in return?
<mars> salgado, anyway, please go on
<salgado> mars, all of this is in one portlet
<mars> ah
<salgado> I think it's quite unlikely that an inline action would require changes to more than one fragment/portlet
<mars> salgado, thank you, I'll reply to the thread
<salgado> mars, np
<jelmer> bigjools: yeah, it works now - thanks
<mwhudson> james_w: hi, you there?
<mwhudson> james_w: it seems like the official source package branches for indicator-applet are not in 2a format
<mwhudson> james_w: do you want to fix that or shall i?
 * thumper is fetching kickass strong coffee
<mwhudson> hm
<mars> abentley, ping
<mwhudson> did kiko make all the importd slaves fall over with those kernel imports?
<abentley> mars: on a call
<mars> k
<abentley> mars: pong
<mars> abentley, could you please explain to me how the +fragment URL scheme that you were using works?
<abentley> mars: Sure, let me refresh myself.
<abentley> mars: I manually construct the relative URL of the comment fragment, using the comment ID.
<abentley> mars: Then I use Y.io to retrieve the fragment, and invoke a callback with the response text.
<mars> abentley, so Y.io("/+bug/1234/+comments/1"), or something?
<mup> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <Launchpad Foundations:Fix Released by debonzi> <https://launchpad.net/bugs/1234>
<mars> ugh
<thumper> abentley, mars: just an FYI
<thumper> I started doing this with showing the merge proposal details on the branch page
<thumper> y.io worked fine
<abentley> mars: No, I was doing it only with code review comments.  So it was comments/1
<thumper> but when I needed to do so on the bugs page
<thumper> it blew up
<mars> abentley, ah, but same idea
<thumper> so I had to use the api to get the formatted code
<abentley> mars: And then appending +fragment on the end, so "comments/1/+fragment"
<mars> ah
<abentley> thumper: Did you try constructing an absolute URL?
<thumper> getting the fragments rendered all at once through the api is necessary
<thumper> abentley: didn't work
<thumper> abentley: there may be other reasons why
<thumper> but I couldn't find them
<abentley> thumper: I'm surprised, because I thought that limitation only applied to AJAX.
<thumper> abentley: it is ajax
<thumper> doesn't Y.io use xmlhttprequest?
<abentley> thumper: But it returns text, not xml.
 * thumper shrugs
 * mwhudson blinks
<abentley> mwhudson: You know the same-origin policy for ajax, right?
<mwhudson> well vaguely
<mwhudson> abentley: i think the part that confuses me is what you mean by "ajax"
<abentley> mwhudson: I mean making asynchronous requests for XML using javascript.
 * sinzui saw that remark coming
<mwhudson> abentley: but that's not what ajax means in practice these days, is it?
<mwhudson> as ~noone actually does the xml bit
<mars> mwhudson, that may be true, we just never talk about it :)
<abentley> mwhudson: Okay, I rephrase my comment to "I'm surprised, because I thought that limitation only applied when AJAX actually does XML".
<sinzui> AJAJ The greek hero of jam
<mwhudson> abentley: ah, i don't think that's true
<mwhudson> maybe i'm wrong!
<abentley> mwhudson: Apparently with json, it's popular to use an IFRAME hack to circumvent the same-origin policy.
<abentley> mwhudson: Though according to http://www.json.org/fatfree.html this only gives access to subdomains.
<mars> thumper, fwiw, same-orgin policy isn't a problem for the JavaScript API.  The whole webservice is exposed via every subdomain's /api URL.
 * mwhudson sticks his fingers in his hears and hides under his desk
<mwhudson> mars: that's why he had to change his code to actually use the api :-p
<mwhudson> (at least, that was my understanding)
<mars> yep :)
<sinzui> abentley that is true about the iframe, and YUI allows you to shoot yourself in the foot
<mars> sinzui, how?
<sinzui> abentley: I believe there is another reason though to use an iframe, I think it was file uploads
<thumper> mars: yes, that is how I got it working
<abentley> mars: Anyhow, I think the idea of allowing us to get the views, and get them all at once, makes a lot of sense.
 * thumper is starting to wake up 3 hours after getting up
<thumper> abentley: +1
<sinzui> mars: I think any site that circumvents the security in my browser and risks handling my data should also have its pages re-written in javascript to say security-holes are here to play, come steal user user's data today
<mars> heh
<ajmitch> how poetic
<mars> well, there was that nifty page that stole your gmail contact list...
<thumper> flacoste: ping
<flacoste> hi thumper
<flacoste> call?
<thumper> yeah
<mars> Wow, cherrypicks take a long time to run.
<mwhudson> yes
<abentley> mwhudson: call?
<mwhudson> abentley: skype ok?
<abentley> mwhudson: sure.
<mwhudson> then yes
<abentley> mwhudson: https://pastebin.canonical.com/24854/
<mwhudson> abentley: http://twistedmatrix.com/pipermail/twisted-python/2009-April/019482.html
<mwhudson> abentley: https://launchpad.net/ampoule
<mwhudson> yay skype
<abentley> mwhudson: Yeah, it's getting to bug me more and more.
<mwhudson> abentley: i'll try to get my new router going
<abentley> mwhudson: thanks.
<mwhudson> and then we can sip for great justice or something
<mars> is the layout of the subscribers portlet on edge broken for everyone, or just me?
<mars> to pick an arbitrary bug, #234567
<mup> Bug #234567: Gnome toolbars unresponsive after update and no toolbars on reboot. <toolbar> <gnome-panel (Ubuntu):Invalid by desktop-bugs> <https://launchpad.net/bugs/234567>
<wgrant> mars: Yes.
<wgrant> mars: It was fairly broken last week.
<wgrant> mars: But this week it transitioned to really broken.
<mars> wgrant, last week!
<wgrant> A bug was filed last night.
<mars> wgrant, thanks.  #484848 :)
<mup> Bug #484848: Subscriber icons in the subscribers portlet appear on a separate line from the subscribers' names <bug-page> <ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/484848>
<mwhudson> cool bug number
<wgrant> Too close to 500000 :(
<mwhudson> https://code.edge.launchpad.net/~vcs-imports/linux/trunk is running really fricking slowly
<jml> tee hee
<jml> mwhudson, hi
<mwhudson> jml: hello
 * jml waves to the rest of the launchpadders
 * jelmer waves
 * mwhudson starts working through the stack of replies to his mail of last night
<jml> so... when are we doing bzr-svn?
<mwhudson> jml: after i've replied to these mails i guess
<jml> woop woop!
<mwhudson> jml: or you or jelmer could bang on https://code.edge.launchpad.net/~mwhudson/launchpad/bzr-svn-imports a bit
<mars> flacoste, cool, thanks for the reply
<mwhudson> where do the packages that get built by the current soyuz system get signed?
 * jml looks
<wgrant> mwhudson: The sources are signed on the dev's machines.
<jml> orrible ack :)
<wgrant> s/'s/s'/
<mwhudson> wgrant: sorry
<mwhudson> wgrant: s/packages/binaries/
<wgrant> Oops, didn't read the 'built by'
<wgrant> mwhudson: They're not actually signed themselves.
<mwhudson> well they are by the time they get into the archive?
<wgrant> mwhudson: They're stuck in the repository, and the repository's Release file (containing lots of hashes) is signed.
<mwhudson> oh
<mwhudson> ok
<mwhudson> this happens during publication then?
<wgrant> Yes.
<mwhudson> are the source packages that are uploaded for a ppa published before they are built?
<wgrant> For PPAs, not necessarily.
<wgrant> For P3As, yes.
<wgrant> (because P3A builds need to get the source files from the published archive)
<mwhudson> ok
<mwhudson> i think the mist is beginning to lift
<mwhudson> (or close in, or something)
<wgrant> Now, sources don't actually have to be signed. But as bigjools says, you probably should if you can.
<mwhudson> wgrant: is the 'not necessarily' for PPAs because publication and building are essentially independent?
<mwhudson> man, i'm amazed the soyuz guys got P3As to work
<mwhudson> much respect due there
<wgrant> mwhudson: Well, they worked, but I was able to penetrate them simply three times, so it's a restricted value of work.
<wgrant> mwhudson: The source signature is only used to verify that the uploader is who they say they are.
<wgrant> mwhudson: Once archiveuploader finishes with it, it is no longer important.
<wgrant> (and archiveuploader already accepts unsigned source packages from trusted sources)
<wgrant> What could perhaps work is getting the builddmaster to sign them once it gets them from the slaves, but that's a bit ew.
<thumper> rockstar: ping
<rockstar> thumper, hi.
<spm> mars: hi; "<maxb> spm: (Re: https://answers.edge.launchpad.net/launchpad/+question/89097) Um, proffered link?" it looked like a sig; but was to.. um.... an male enlargement assistant site. To put things delicately.
<maxb> huh. I don't remember seeing a link there. Are the spammers getting clever enough to stick their links in legitimate-ish responses now !?
<mars> spm, ?
<spm> mars: gah. sorry. tab completion fail. 1001 apologies.
<mars> ok.  I didn't see the link-spam anyway?  I hate spam.
<maxb> spm removed a question comment to which I'd responded. Apparently there was a dodgy link there. I'm confused because I interpreted it as a genuine user, and wrote a response to it, without noticing.
<spm> maxb: well... one of them at least. the responses they we're making were *vaguely* on topic; just ... odd. but the link was a no-no. and some of the responses they had were pure spam.
<wgrant> sinzui: Why Jaunty->Lucid rather than Karmic->Lucid?
<james_w> mwhudson: I have a list somewhere of the packages that aren't in 2a
<james_w> I haven't done the work to upgrade them all yet, do you have a way to do it easily?
<maxb> spm: huh. there was only one response from that user at the time I wrote my response.
<sinzui>  wgrant It is a typo
<mwhudson> james_w: well there is a script now
<thumper> rockstar: it doesn't look link yui 3.0 is in devel
<spm> maxb: (mars) they hit about half a dozen different answers across the place
<thumper> mars: are you landing it?
<mwhudson> i don't know if we have an easy way of letting it rip at a bunch of branches
<sinzui> wgrant: And my script definitely knows not to do exactly that
<rockstar> thumper, it's in db-devel
<wgrant> sinzui: great.
<james_w> mwhudson: for loop?
<sinzui> in fact. I think I need to get my confidence up and ask a LOSA to run it on staging.
<mwhudson> james_w: i guess :-)
<spm> sinzui: I do recognise I'm scary to ask. just saying. ;-)
<mars> thumper, waiting for the CP on db-devel, fixing up any gross errors (I've found one so far), then merging back into devel.
<mwhudson> i don't know if we have an easy way of scheduling even one branch to be upgraded i guess
<mwhudson> rockstar: ^^ ?
<mars> thumper, maybe early next week.  But you can build on top of the branch now!
<sinzui> spm: The sample data is small, My script certainly works. as wgrant hinted. The order I fix the packing link is very important.
 * sinzui fixes bug title
<wgrant> sinzui: I don't see why you single out Feisty there.
<james_w> mwhudson: I don't think trying to do this over conference wifi would be wise
<james_w> mwhudson: I could pass the list to you?
<mwhudson> james_w: that would be great
<mwhudson> i can always do it from devpad if the new sexiness proves uncooperative
<spm> mwhudson: james_w: coming in this half way thru - I gather this may be appropriate to run on crowberry itself? or?
<sinzui> wgrant: the original bug mentioned that, and that is the point we added the initialisation.. My script runs from hoary to lucid
<wgrant> sinzui: Ah, I see.
<rockstar> mwhudson, I don't think so.
<mwhudson> spm: maybe yes, don't think it matters too much
<mwhudson> spm: you can't do it at the fs level because of the ever joyful stacking
<rockstar> mwhudson, so, what we have REALLY needs to be tested.
<spm> mwhudson: sweet....
<mwhudson> rockstar: well i guess we can test it
<mwhudson> rockstar: the way of scheduling a job now is "INSERT INTO BranchUpgradeJob ...." ?
<james_w> mwhudson: http://paste.ubuntu.com/321956/
<james_w> that's the packages
<james_w> inferring the branches from that is left as an exercise for the reader
<mwhudson> james_w: btw
<mwhudson> james_w: what should i do when i find a source package has no branches?
 * mwhudson feels a query of doom coming on
<rockstar> mwhudson, yes, but that's not even all that tested.
<mwhudson> rockstar: ok
<james_w> mwhudson: the branch URLs are predictable fwiw
<mwhudson> rockstar: so maybe i won't try to upgrade all of these branches as the first test :-)
<rockstar> mwhudson, I have UI, but didn't think it wise to land while I was sprinting/at UDS/week 3
<rockstar> mwhudson, please for the love of Thor no.
<james_w> mwhudson: for packages with no branches, file a bug against "udd" project please
<mwhudson> james_w: one per package?
<mwhudson> rockstar: >:)
<sinzui> spm: I have this script that walks the history of Ubuntu's series and copies the packaging links restore the data we have been losing with each series we create. I want to run this test on stagings database to verify it runs, and to examine the quality of data that it makes. https://pastebin.canonical.com/24860/
<mwhudson> i guess a test on staging is a good idea in any case
<rockstar> mwhudson, yeah, probably.
<james_w> mwhudson: a bunch per package
<spm> sinzui: sure. one sec.
<rockstar> mwhudson, Also, the upgrade won't actually upgrade to 2a just yet.  I didn't realize that until week 4 of last cycle.
<james_w> ~ubuntu-branches/ubuntu/<distroseries>/<sourcepackagename>/<suite>
<rockstar> We never updated the format map when we moved to 2.0
<mwhudson> rockstar: groan
<mwhudson> james_w: sorry, i'm being dense, was that aimed at me?
<rockstar> mwhudson, yes, but I think we can actually get rid of the mapping now, since a call to upgrade should upgrade properly to 2a
<james_w> mwhudson: yeah, however, I'm out of battery and so can't expand, sorry :-)
<mwhudson> james_w: the query to find all the official branches for a particular source package is not very hard
<rockstar> i.e. I shouldn't have to make some format first...
<james_w> mwhudson: that's probably the better way for you to do it then
<maxb> james_w: Do we still file bugs for native packages without branches? Or is that a general problem still?
<mwhudson> james_w: the "of doom" part was thinking about requesting upgrades of all of them :)
<james_w> maxb: file a bug please
<james_w> back in a few when I can escape from this corner of the session with no electricity
<spm> sinzui: quick Q; is this expected to be quick; or slow? do you want it run with any special options or just go for it?
<mwhudson> james_w: https://code.edge.launchpad.net/ubuntu/lucid/+source/bzr-fastimport
<sinzui> spm: I honestly do not know. I think it could complete in a few minutes because we are looking with 1000s of objects.
<spm> sinzui: oki; suck and see time.
<sinzui> spm: when it is done. We will know exactly how may objects we are playing with
<spm> sinzui: I do love Dev's who wildly overestimate how long their code takes to run. ;-) https://pastebin.canonical.com/24861/
<sinzui> spm: It better to please than to disappoint. I schedule milestones the same way
<spm> hahahaha
<sinzui> those numbers do look like the pages. I'll look at lucid on staging and see if it makes sense. I suspect I have just create 1200 links to packages that are in PPAs, not main
<wgrant> There are some links to PPA packages, but not that many.
<sinzui> ugh. It copied squeak.  That project is proprietary and should be deactivated
<sinzui> These links look correct. I see some wrong one, but they are wrong because the version changed late in Karmic, so karmic and lucid both need need fixing
<jelmer> bigjools: Hi
<jelmer> bigjools: Do you perhaps have a free moment to look at a zope issue?
<bigjools> jelmer: possibly, I am sorting out a production problem, I can try and find you later
<jelmer> bigjools: thanks - we're in presidente
#launchpad-dev 2009-11-19
 * mwhudson lunch
 * thumper is actually enjoying the javascript stuff :)
<mwhudson> man you can tell the sort of day i've had
<mwhudson> rebooted last night; hadn't started emacs until just now
<thumper> mwhudson: oh no
<thumper> mwhudson: why is that?
<mwhudson> thumper: because i've been writing email/on the phone/yakking on irc all day
<thumper> heh
<mwhudson> man i hate our vcs test fixture thingies we use in the code import testsd
<lifeless> you could use testresources
<wgrant> lifeless: OK, now I agree with you that creating SPNs on push is sane.
<wgrant> I think.
<lifeless> :)
<lifeless> wgrant: actually trying to do something can be very educational
<mwhudson> code that deletes the cwd can seem a little odd
 * mwhudson eods
* spm changed the topic of #launchpad-dev to: LP Codehosting offline for H/W update 0800-1200 UTC | This is Launchpad Development Channel | Week 2 of 3.1.11 | PQM is open  | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<mwhudson> oh stabby
<mwhudson> my ec2 test of bzr-svn-imports failed because the relevant goo isn't installed on the image yet
<thumper> mwhudson: :(
<thumper> mwhudson: hey, I have a stabby alias now :)
<thumper> ââ¹
<thumper> â
<thumper> hmm, just tried to push :(
<noodles775> thumper: you're aware that codehosting is offline right? (see topic)
<thumper> noodles775: yeah, I remembered after I tried to push
<thumper> noodles775: I have a cool branch to land though :)
<noodles775> :)
 * wgrant wonders how it could possibly take four hours, unless the disks are being entirely replaced offline.
<thumper> wgrant: the resize forces a fschk
<wgrant> thumper: Ah.
<spm> backup; resize; fsck
<thumper> spm: so what will this do for our space?
<spm> thumper: an extra 2-3 weeks breathing space :-P
<wgrant> I wonder how much space would be saved by regularly pruning unnecessary revisions from stacked branches.
<adeuring> good morning
<noodles775> Hi adeuring
<adeuring> hi noodles775!
<maxb> I'd think quite a lot, since for merged branches that amounts to deleting almost all their data
<wgrant> maxb: Exactly.
<lifeless> step 1) implement 'bzr gc'
<lifeless> step 2) implement 'bzr gc --keep-dead-heads'
<mwhudson> wgrant: i wonder that too
<mwhudson> wgrant: want a write a script that will try it?
<mwhudson> lifeless: you could go for a more inaccurate hammer: if the tip of the branch is in the repo of the stacked on branch, delete the repository from the stacked branch
<mwhudson> (i realize this isn't quite the same thing)
<lifeless> mwhudson: I wouldn't do that.
<lifeless> mwhudson: as tags would get fucked over.
<mwhudson> tags are stored in the repo?
<mwhudson> i thought it was the branch
<mwhudson> but never mind
<lifeless> tags refer to revisions
<lifeless> if you zap the repo, and there is a reference to a revision not in the stacked on repo... boom
<lifeless> you've just deleted something important
* mthaddon changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 2 of 3.1.11 | PQM is open  | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<salgado> codehosting is still down for maintenance, I guess?
<mars> thumper, the yui 3.0.
<mars> thumper, the yui 3.0.0 branch has landed on db-devel
<flacoste> morning launchpadders
<sinzui> barry: bac: stand-up in 2 minutes
<bac> morning flacoste
<matsubara> stub, Chex, gary_poster, rockstar, Ursinha, sinzui, allenap: LP production meeting in 28 min @ #launchpad-meeting
<maxb> flacoste: Hi. Would it be possible to reassign the official meta-lp-deps branches to ~launchpad-committers at this point?
<flacoste> maxb: sure thing
<Ursinha> matsubara, I'm afraid I won't be able to attend
<maxb> Thanks :-)
<matsubara> Ursinha, that's ok
<flacoste> maxb: i'm going to mark the dapper and gutsy branch obsolete, any objections?
<maxb> Makes sense
<flacoste> maxb: all done
<maxb> flacoste: Could you reassign the dapper and gutsy too? To keep them all together and so that I can patch up their stacked-on urls.
<flacoste> maxb: doing...
<flacoste> and done
<maxb> suitable surgery performed on the stacked_on_urls
<maxb> via horrid assortment of sftp and bzrlib in interactive python :-)
<matsubara> hi bigjools
<matsubara> are you going to be around for the lp production meeting?
<matsubara> in 9 min :-)
<bigjools> matsubara: no :)
<matsubara> bigjools, anyone from soyuz will be around?
<bigjools> noodles775: would you be able to fill in please?
<bigjools> and good morning/afternoon BTW :)
<noodles775> bigjools, matsubara: yep.
<matsubara> thanks noodles775
<matsubara> and bigjools
<jml> hello launchpadders
<gary_poster> <chorus> hello jml!
<jml> :)
<jml> I would like my own chorus
<gary_poster> heh
<jml> you know what one of the hidden dangers of two weeks solid non-hacking sprinting is?
<jml> rogue ec2 instances
<mars> gary_poster, it's difficult to chorus without timestamps.  I see jml, I see barry.  Could have joined 6 hours apart for all I know!
<mars> well, except that jml is probably at UDS right now, but I often forget that
 * barry sings d'oh, ray, me
<gary_poster> lol
<mars> jml, eewww
<barry> gary_poster: hi!  how screwed are my 2 week old launchpad branches now? :)
<mars> jml, 'ec2 check-for-zombies', put it in a crontab?
<gary_poster> barry: pretty screwed! :-) you have Python 2.5 and YUI 3 final landings now. ;-)
<jml> yeah, I should do something like that.
<jml> I have to put it on my server, obviously, because laptop-powered crontabs are next to useless.
<mars> we all should
 * mars runs a desktop system
<barry> gary_poster: yay!  hopefully they're all review branches and landed branches
<abentley> gary_poster: I am trying to use a new library with launchpad (ampoule).  I've added the tarball to download-cache/dist and I've edited versions.config.  But it doesn't get installed when I run make, and make doesn't error either.  What am I missing?
 * jml has "Oh What A Night!" stuck in his head
 * barry stabs jml
<jml> at last, sweet mercy
<mars> abentley, I just did that yesterday.  check doc/buildout.txt, there is a section with exactly the instructions you need.
<gary_poster> abentley: add it to setup.py.  (we need to have indirect versions in versions.cfg, so it is nice to have all the versions in one place.  direct dependencies should be declared in setup.py as usual)
<gary_poster> or see mars' reply :-)
<abentley> gary_poster, mars: thanks.
 * jml goes to look for deryck & intellectronica
<mars> jml, if you find them or beuno, tell them the bugs subscriber portlet has been broken for a week now :(
<jml> mars, do you have a bug number for that?
<mars> jml, 484848
<mars> bug 484848
<mup> Bug #484848: Subscriber icons in the subscribers portlet appear on a separate line from the subscribers' names <bug-page> <ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/484848>
<jml> (I successfully demonstrated the bug subscriber portlet last night)
<mars> jml, just a style issue, but when you see it, it makes you go "oops!"
<jml> mars, I'm sure they would gratefully accept a patch :)
<mars> jml, of course.  I did some work on it yesterday.  I'm not actually committing to fixing it though.
<rockstar> flacoste, hello sir.
<flacoste> hi rockstar, on the phone
<rockstar> flacoste, no sweat.
<rockstar> flacoste, I wanted to talk to you about fixing canonical_url in windmill.
<beuno> mars, I know of the broken prtlet, so does deryck
<rockstar> gary_poster, hi
<gary_poster> rockstar: hi.  lunching.  ping you when I return?
<rockstar> gary_poster, I got what I needed from flacoste.  Thanks though.
<gary_poster> cool
<bac-lunch> sinzui: is anyone looking at bug 485237?  you want me to?
<mup> Bug #485237: OOPS - KeyError: 'link' - in person-index.pt <oops> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/485237>
<sinzui> bac no, it was reported a few hours afo
<bac-lunch> sinzui: so is that a "no, yes"?
<sinzui> bac: no, that is a no. it is not assigned and it is not inprogress.
<bac> sinzui: righto.  but would you like to assign it to me?  i'm looking for work.
<sinzui> bac: take it
<bac> okey doke
<thumper> morning
<mwhudson_> maxb: hi
<mwhudson> or anyone else who understand how the launchpad ppa works currently...
 * wgrant knows a bit.
<jelmer> 'moin mwhudson
<jelmer> mwhudson, did you see my comment about _make_silent_logger imports in your bzr-svn-imports branch?
<mwhudson> jelmer: yes, i think i replied?
<jelmer> sorry, I must've missed that
<mwhudson> jelmer: i need to build launchpad-dependencies 0.57 for hardy and make an ami for ec2test before i can land it anyway...
<mwhudson> which is why i was asking about the ~launchpad ppa
<mwhudson> wgrant: i built launchpad-dependencies 0.57 for karmic last night
<mwhudson> wgrant: what do i need to do for hardy?
<jelmer> mwhudson: ah, ok
<jelmer> mwhudson, what about the import error?
<wgrant> mwhudson: You'll need to merge your Karmic branch into the Hardy branch.
<wgrant> mwhudson: And probably give it a version number like 0.57~hardy1, rather than the traditional 0.57hardy1 which isn't right.
<wgrant> Actually, the hardy changes might not be necessary now that 2.4 support is not required.
<mwhudson> jelmer: s/_make_silent_logger/QuietFakeLogger/
<mwhudson> i guess
<mwhudson> wgrant: okay
<jelmer> mwhudson: ah, great
<mwhudson> wgrant: oh that's true\
<mwhudson> i forget exactly why hardy is different even though i was responsible for the difference
<mwhudson> (and the bogus version number)
<mwhudson> i have this memory of being very angry though
<wgrant> Heh.
<wgrant> What's the diff?
<mwhudson> i think it's just not depending on python2.4-support on hardy
<mwhudson> trunk launchpad-dependencies still depends on all kinds of python 2.4 stuff
<mwhudson> is it too soon to drop all that?
<mwhudson> gary_poster: hi
<thumper> wgrant: can you see the landscape overview page now?
<mars> mwhudson, OT question: would bzr-explorer allow me to quickly jump between 'bzr blame', the revision number, the diff for said revision, etc.?
<mwhudson> mars: no idea!
<wgrant> thumper: It's lying, but yes.
<mars> darn :(
<mwhudson> i don't have any truck with this gui nonsense
<thumper> wgrant: :)
<mars> heh
<mwhudson> well not really
<mwhudson> but i don't know
<mwhudson> mars: bzr gannotate can do that
<mwhudson> so i expect explorer can too
<mwhudson> (or soon will)
<mars> if it builds against bzr nightlies...
<mars> there we go: sudo add-apt-repository ppa:bzr-explorer-dev/ppa
<mars> this is one of the application usage models where the command line sucks: random browsing
<gary_poster> mwhudson: hi
<gary_poster> mwhudson: I think we can drop all that stuff.  We should convert the -2.4 to the -2.5
<mwhudson> gary_poster: ok
<gary_poster> mwhudson: does that mean I should say thank you? :-)
<mwhudson> yeah, i'll do that
<mwhudson> gary_poster: will you review the branch?
<gary_poster> mwhudson: absolutely.  thank you!
<mwhudson> gary_poster: around for another hour or so?
<gary_poster> mwhudson: yes, for just under two hours
<gary_poster> ish
<mwhudson> cool
 * mwhudson prefers summer in some ways
<wgrant> bigjools: Can you please review https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection/+merge/14977 when you have time?
<bigjools> wgrant: is that the one where you already requested a review from me?
<wgrant> bigjools: It is.
<bigjools> wgrant: ok, sorry I've not had time to get to it yet, blame UDS
<wgrant> UDS does that.
<wgrant> And I'll have a hopefully small branch tonight to fix gina.
<bigjools> wgrant: I'll do my best to look at it this week, but it's most likely to be Monday
<wgrant> And I think that should be it.
<mwhudson> gary_poster: do you think sed -i -e 's/python2.4/python2.5/g' debian/control is basically speaking right?
<gary_poster> mwhudson, yeah, removing the resulting dupe python2.5-dev
<gary_poster> mwhudson we might be specifying more than we need, but that's a diff problem for a diff day, IMO
<Ursinha> bigjools, nice sticker
<mwhudson> gary_poster: ok
<bigjools> Ursinha: yes ma'am!
<Ursinha> hahahaha
 * mwhudson is confused
<mwhudson> seems the launchpad-dependencies that built yesterday are broken?
<wgrant> What's broken about them?
<wgrant> A saw a bug about something last night, and it looked like lots of deps might have been removed, but I didn't look at it in detail.
<mwhudson> The following packages have unmet dependencies:
<mwhudson>   launchpad-developer-dependencies: Depends: launchpad-dependencies (= 0.57) but 0.56 is installed
<mwhudson>                                     Depends: launchpad-database-dependencies (= 0.57) but 0.56 is installed
<wgrant> Hm. That's odd.
<mwhudson> yes
<mwhudson> the 0.58 ones i just built installed fine though...
<ajmitch> it is a little odd, since 0.57 seems to install fine on my karmic system
<ajmitch> assuming it's still karmic you're talking about :)
<mwhudson> yes
<mwhudson> i probably buggered up something locally then, good
<mwhudson> gary_poster: https://code.edge.launchpad.net/~mwhudson/meta-lp-deps/bye-2.4-hello-2.5/+merge/15058
<gary_poster> mwhudson: awesome.  on call, will lock after
<mwhudson> hooray for proposing at ??:?4:55
<thumper> sinzui: ping
<sinzui> Hi thumper
<thumper> sinzui: are we going to talk today?
<thumper> sinzui: I have 30 minutes spare now
<gary_poster> mwhudson: we can now use this on hardy because python2.5-support is available there, where python2.4-support was not?  (ISTR python2.4-support was the issue with Hardy)
<sinzui> thumper: now is good
 * gary_poster is confused, sigh.  http://packages.ubuntu.com/karmic/python2.5-support doesn't exist.  Maybe in PPA?
<wgrant> gary_poster: It's a virtual package
<gary_poster> wgrant: ah ok, thank you.  that makes sense
<wgrant> gary_poster: eg. if you apt-get install python2.5-support it will tell you that it's choosing python-support instead.
<wgrant> Because python-support Provides python2.5-support.
<gary_poster> ah, so.  only barely aware of virtual packages, thanks
<gary_poster> mwhudson: approved, with note about my question
<mwhudson> gary_poster: it's an interesting question
<gary_poster> mwhudson, we could ask losa to verify practically, or try it on a ec2 instance
<mwhudson> gary_poster: i think python2.5-support is actually part of hardy
<mwhudson> so it would be a bit exciting if it couldn't be installed
<gary_poster> heh
<mwhudson> hm, maybe not
<gary_poster> mwhudson: as I said in the review, if you are comfortable then I am.  Otherwise, verifying somehow seems like a good idea before copying over to hardy
 * mwhudson is a little confuses
<mwhudson> http://packages.ubuntu.com/hardy/python-support is part of hardy
<mwhudson> but not python2.x-support
<gary_poster> :-(
<mwhudson> so maybe we just need to depend on python-support
<gary_poster> mwhudson, maybe so,  If it were I, I must admit, I would do what had been done before, maintaining the separate Hardy branch.  Who added the split?
 * gary_poster looks at PPA
<gary_poster> I think it is gone
<gary_poster> My memory is that Francis had made the split in 0.50
<gary_poster> He would probably know
<mwhudson> i made the split :)
<mwhudson> i forget why
<gary_poster> heh
<mwhudson> it seems nothing on my system depends on python2.x-support
<mwhudson> just python-support
<mwhudson> i'll see if my package installs on hardy
<mwhudson> (on ec2)
<gary_poster> ok cool
<mwhudson> am i glutton enough to try to build a new ec2 image based on the new canonical-provided hardy ami i wonder...
<jml> ec2 test ate my test run
<jml> what should I do?
<bigjools> philosophise about how it was meant to be
<wgrant> mwhudson, gary_poster: Ah, it seems that Hardy's python-support doesn't Provide python2.x-support. Since we're no longer using some awful horribly obsolete removed Python version, it's OK to just depend on python-support.
<wgrant> (until Lucid removes python2.5, but we'll see...)
<mwhudson> jml: send rude mail to jeff bezos
<jml> :(
<gary_poster> wgrant, ok cool.  Sounds good to me.  Thank you.  Lucid: the hope is to move to 2.6 then, but as you said, we'll see :-)
<wgrant> gary_poster: You can't really move to 2.6 until after Lucid is released.
<gary_poster> wgrant, yeah I know. :-)  in some theoretical world we could move to Lucid and Py 2.6 at once.  probably a horrible idea, in addition to being potentially difficult to coordinate
<gary_poster> wgrant: well, actually...
<gary_poster> supposedly we arranged to have resources to have internally supported 2.6 in Hardy if we had gotten there
<gary_poster> Parts already exist
<gary_poster> But almost certainly not everything we need
<mwhudson> how can i install a .deb file with apt?
<mwhudson> (i think that's what i want, i want to install a .deb but meet it's dependencies from the configured archive)
<wgrant> mwhudson: You'd have to create a repository.
<wgrant> mwhudson: Otherwise, dpkg -i it.
<wgrant> Then apt-get -f install.
<wgrant> The latter will resolve unmet dependencies.
<mwhudson> ah ok
<wgrant> Although that doesn't help if it depends on other .debs that you don't have in a repo.
<mwhudson> dpkg --fsck-me-harder ?
<mwhudson> well yes, that's something i'd like to find out :)
<ajmitch> gdebi will install a single .deb & resolve dependencies, I believe
<wgrant> True.
<mwhudson> --force-depends
<mwhudson> ?
<wgrant> No. That will install it unthinkingly.
<ajmitch> --force-depends is a bit messy, you have to clean up with apt-get -f install afterwards
<wgrant> You want to dpkg -i, which will get it half-intsalled.
<wgrant> Then apt-get -f install will resolve dependencies, and finish installing the half-installed package.
<mwhudson> wgrant: aa
<mwhudson> Setting up bzr (1.3.1-1ubuntu0.1) ...
<mwhudson> hmm
<ajmitch> if you can bear a GUI, use gdebi for it I think
<mwhudson> ajmitch: this is ec2
<ajmitch> that's what I was worried about
<ajmitch> looks like it might have a cli-only mode as well, but if you have it installed now, it doesn't matter :)
<mwhudson> hm, launchpad-dependencies nearly installs without the ~launchpad ppa now
<mwhudson> (after changing to depend on python-support)
<wgrant> It should.
<wgrant> What is missing? geoip stuff?
<mwhudson> yeah
<mwhudson> specific version of psycopg2
<mwhudson> python-profiler
<ajmitch> it's a shame we can't just ram that into karmic-updates (or hardy-updates)
<mwhudson> postgresql-autodoc
<ajmitch> are there bits there that are missing from lucid?
<mwhudson> i don't know
<mwhudson> ajmitch: well python 2.5 is/will be missing from lucid >:)
<mwhudson> woo
<mwhudson> so my packages install on hardy
<ajmitch> from main, anyway, though I understand that you only want to use supported packages :)
<ajmitch> just get python2.6 support in LP done quickly :)
<gary_poster> It had more issues than we had hoped, but it certainly could be done.  We will have to do some kind of careful dance for this, with the different supports for Hardy and Lucid.  Not having overlapping Python support in concurrent LTS releases is hard.
 * mwhudson is so happy to just have 2.5...
<ajmitch> because you get new bzr-svn toys?
<mwhudson> yes
<mwhudson> and well
<mwhudson> just feeling less like i'm living in the past
<ajmitch> where you live, you'll always be in the future
<mwhudson> gary_poster: new diff on https://code.edge.launchpad.net/~mwhudson/meta-lp-deps/bye-2.4-hello-2.5/+merge/15058
<gary_poster> mwhudson: cool, +1 :-)
<mwhudson> gary_poster: but i think i'm going to apply "if it builds and installs it's ok"
<gary_poster> +1
<mwhudson> wow, something set the commit message automatically from the changelog
 * mwhudson wonders what the ppa build queue is like currently...
<mwhudson> empty
<mwhudson> awesome
<lifeless> hey
<lifeless> quick doctest help
<lifeless> say I had a string "Ran 4 tests in 0.123s"
<lifeless> and I don't care about the exact time
<lifeless> how would I write a doctest example for that ?
<jml> Ran 4 tests in ...s
<lifeless> thanks
<jml> there's an option you might need to turn on, depending on your test infrastructrue
<lifeless> jml: testtools
<lifeless> jml: I'm taking an early lunch to finish last nights itch
 * jml files https://bugs.edge.launchpad.net/soyuz/+bug/485558 
<mup> Bug #485558: 'SourcePackage' name inconsistent with rest of Soyuz objects <cleanup> <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/485558>
<jml> bigjools, bug 485558
<mup> Bug #485558: 'SourcePackage' name inconsistent with rest of Soyuz objects <cleanup> <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/485558>
<bigjools> finally, a bug :)
 * lifeless takes a bootstrap shortcut
<jml> bigjools, it's a step forward.
<bigjools> jml: from the school of JFDI
<bigjools> jml: btw it should be a bug on registry
<jml> bigjools, we shouldn't have subprojects :P
<lifeless> jml: do you need to \\ escape a . that you want to be literally matches ?
<lifeless> s/matches/matched/
<bigjools> :)
<jml> lifeless, don't know, sorry.
<mwhudson> lifeless: it's not a regexp at all
<mwhudson> lifeless: '...
<mwhudson> lifeless: '...' matches an arbitrary amount of stuff
<mwhudson> i doubt there's a way to match ... explicitly
<jml> you know
<jml> I think I'm actually going to use blueprints agian.
<bigjools> jml: don't you mean specifications?
<jml> bigjools, ha!
<wgrant> Features!
 * jml has a note that says we should consider how daily builds will interact w/ RSS feeds
<jml> but doesn't really know where to put it.
<bigjools> oo that's an interesting idea
<bigjools> jml: I was going to say to add that to https://blueprints.edge.launchpad.net/soyuz/+spec/build-from-branch
<bigjools> but I think we need another specprint for daily builds
<jml> yeah, I was thinking that we could actually benefit a lot from splitting things up
<jml> bigjools, would you mind too much if I gardened the spec a bit on Monday?
<bigjools> what do you have in mind?
<jml> bigjools, looking at it, maybe splitting thing up and making subspecs maybe.
<bigjools> yeah sure
<bigjools> it does seem to have grown
<jml> I want to start dumping thoughts on the UI, and there'll be quite a few flows.
<bigjools> ui needs a separate blueification I reckon
<jml> yeah
<jml> I'll faceprint that on Monday
<bigjools> better than facepalming it I suppose
#launchpad-dev 2009-11-20
 * mwhudson lunches
<mwhudson> i really hope bzr-hg support doesn't involve depending on any more packages
<lifeless> mwhudson: it will
<lifeless> mwhudson: mercurial
<maxb> mwhudson: We depend on python2.X-support to ensure we have our modified version of python-support in place. Granted this happens to no longer be required in hardy...karmic, but we'll need that again in lucid
<jtv> jamesh_: g'day
<jtv> oh, probably early for him... mwhudson, you're further ahead on the daylight merry-go-round; maybe you're the one to talk to
<jtv> thumper, you got any storm devs for me?
<lifeless> jml: so, a new toy for you https://code.edge.launchpad.net/~lifeless/testtools/that/+merge/15070
<lifeless> abentley: around ?
<mwhudson> maxb: fine, let's worry about that again when are targeting lucid?
<mwhudson> the amount of steps involved in depending on a new package in launchpad is fricking insane
<lifeless> mwhudson: why?
<lifeless> [as in why is it insane?]
<mwhudson> lifeless: mostly just quantity
<mwhudson> and ec2 makes some of them really annoying
<lifeless> mwhudson: are any unneeded? can we discard some waste?
<mwhudson> lifeless: having ec2 thingies that did apt-get update on start up would probably help some
<lifeless> update ++ upgrade
<mwhudson> well obv yes
<lifeless> :P
<mwhudson> partly here i'm shooting myself in the foot by trying to rush i guess
<mwhudson> let's see if ec2 update-image wants to work today
<jtv> mwhudson, hi!  I'm shopping around for a reviewer for therve's storm branch... are you available for that?
<mwhudson> jtv: a storm branch!?
<mwhudson> jtv: i can have a look
<jtv> Fixes that memory leak that we found with the help of objgraph
<jamesh> jtv: still looking for help?
<jtv> jamesh: good morning!  mwhudson is having a look, though a Storm branch review is probably not a fair thing for me to ask him for :)
<jtv> it's that memory leak where even handlers don't get cleaned up
<mwhudson> jtv: i'll only have a look when you tell me where to look
<jtv> mwhudson: ah so that did get caught in my connection outage... sorry, it's this one: https://code.edge.launchpad.net/~therve/storm/reference-changed-leak/+merge/14480
<mwhudson> jtv: i think i would rather jamesh looked at that unless he's too busy
<jtv> mwhudson: fair enough... jamesh, are you?
<jamesh> jtv: I'll have a look
<jtv> jamesh: thanks!  It's this one: https://code.edge.launchpad.net/~therve/storm/reference-changed-leak/+merge/14480
<jamesh> although more reviews isn't a bad thing
<jtv> (well, not when the reviewers aren't bikeshedders)
<mwhudson> oh
<jtv> ?
<mwhudson> jtv: something completely different from what we're talking about :)
 * jtv looks puzzled
<mwhudson> jtv: i didn't mean to say it in here, basically
<jtv> ah
<stub> thumper, mwhudson: https://code.edge.launchpad.net/~jelmer/launchpad/hg-import-schema/+merge/15061
<mwhudson> stub: what about it?
<stub> Any reason we don't want a single URL column rather than three, at least two of which will always be NULL?
<stub> If we collapsed them into a single URL column, we could even hide that from existing code using properties in the DB class.
<mwhudson> stub: no, not really i guess
<thumper> stub: perhaps we're at "third time - refactor"
<stub> I think so. I don't recall a justification when we added git_url either - maybe there was one, or maybe I was out of coffee.
<stub> Could the cvs columns be left alone for now? There might be code that likes things split into two rather than having to parse the information from a URI.
<mwhudson> stub: let's leave cvs along
<mwhudson> alone
<mwhudson> i don't think there is a url for a cvs module is there?
<mwhudson> i would be pretty happy to be wrong about that
<mwhudson> i guess we could invent one, but meeeh
<lifeless> mwhudson: config-manager invented one
<lifeless> mwhudson: parsing code is all there, import; reuse, done.
<mwhudson> lifeless: i'm going to pretend you didn't say that until at least the end of my work day :)
<mwhudson> but maybe worth considering at some point i guess
 * lifeless shrugnods
<mwhudson> stub: we couldn't hide it totally from existing code, there are some queries on those columns
<mwhudson> but that's pretty minor
<sinzui> spm: is staging going to update during my lifetime?
<stub> mwhudson, thumper: One weird edge case. At the moment, it is possible to have in the a git_url identical to a svn_url. Is there some use case for this, or is it a win enforcing uniqueness over all of them?
<spm> sinzui: I'm assuming that wasn't a challenge?? :-) at this stage the postgres restore is busticated. I understand stubs aware of and working on it.
<stub> It should be fixed. Last log I looked at (19) was using old code.
<spm> oki
<stub> (database/replication/Makefile should contain DOGFOOD_DBNAME near the top - not sure of the revno)
<sinzui> spm, stub: thanks for the update
<sinzui> If I live to see daylight I might be able to complete testing
<spm> sinzui: the lockfile was still in place from the last fail ; have removed; so hopefully!! the next run will proceed
<stub> spm: You want to get it to kick off with yesterday's dump?
<spm> no real need - it'll kick itself off in ~ 20 mins.
<spm> no it wont....
<mwhudson> stub: um, can't really think of where they'd be the same
<stub> There are no conflicts in the production data
<spm> stub: sinzui: new restore kicked off with yesterdays dump; vs waiting another 3-4 hours for 'todays' dump
<sinzui> fab
<stub> I'll check it in a few mins - it has been failing quickly.
<spm> heh. oh yee of little faith.
 * spm lunches
<lifeless> abentley: so, another datapoint on rabbit
<lifeless> there was a great retrospective mail on the u1 dev list today, about deployment issues
<lifeless> one in particular that stood out was that the server backs up if clients don't ack getting messages - mq is all about guaranteed delivery
<mwhudson> lifeless: could you forward it to canonical-launchpad maybe?
<lifeless> whereas pubsub is about broadcasting
<lifeless> mwhudson: sure
<mwhudson> thanks
 * mwhudson eods
<lifeless> mwhudson: I forwarded the mail
<mwhudson> lifeless: i read it, thanks
<lifeless> mwhudson: I don't think you heard the discussion Aaron and I had on the bus though.
<mwhudson> lifeless: this is ture
<mwhudson> true
<lifeless> which was pubsubhubbub &| rabbitmq
<spm> being ware that rabbitmq eats memory like no tomorrow and needs moderately frequent restarts?
<spm> Ahh Tim mentions that. good.
<lifeless> indeed
<lifeless> it was a very useful mail, I thought
<spm> very
<spm> stub: fyi. that staging restore seems to be proceeding nicely so far. 2.5 hours on?
<stub> looked fine when I checked
<mwhudson> oh poo
<mwhudson> having bzr-svn loaded breaks some of our other tests :(
<mwhudson> because it calls external_url and doesn't catch InProcessTransport
<lifeless> theres a bug open I think :P
<lifeless> possibly fixed even
<mwhudson> this is with bzr-svn trunk so i doubt the latter
<thumper> noodles775: hi
<noodles775> hi thumper
<thumper> noodles775: I'm pleased there is someone at the start of their day to look at the merge conflict for devel -> db-devel :)
<noodles775> Nice :)
<noodles775_> thumper: rs? http://pastebin.ubuntu.com/323108/
<noodles775_> or henninge ^^ (diff for resolving stable->db-devel merge conflict)
<henninge> noodles775_: looking
<henninge> noodles775_: sorry, distracted. r=me
<noodles775_> Thanks henninge
<adeuring> good morning
<henninge> How do I run just the page tests (stories) for my application (translations) ?
<henninge> I know "-t lp.translations" runs all translations tests, but "-t lp.translations.stories" does not work.
<stub> I'm fixing that conflict
<noodles775> stub: I thought I'd already fixed it?
 * noodles775 checks if there's another...
<stub> noodles775: Did it bounce? I don't see it in there.
<stub> My fix is with pqm now
<noodles775> stub: http://pastebin.ubuntu.com/323150/
<noodles775> stub: ok.
<stub> noodles775: You tried to land it on launchpad/devel
<noodles775> Right.
<noodles775> Forgot the submit-branch :/
<bac> hi sinzui
<sinzui> hi bac
<bac> sinzui: it looks to me like person-portlet-emails.pt is no longer used.  do you know if that is the case?
<bac> i'd like to whack it
<sinzui> ha ha
<sinzui> The only thing that uses it is notfound-traversals.txt
<sinzui> bac +1 to remove it
<bac> yep
<bac> rt
<sinzui> bac:I think I recall allenap has some method to identify unused templates
<bac> sinzui: also, i have discovered <li class="mail"> causes the sprite to be rendered twice and i don't know why
<allenap> sinzui: Do I?
<bac> my current "fix" is to remove the class and put the image in once, unless you have a better idea
<sinzui> bac: It can the the icon is designed for a specific line height. It can never be used on an element that is not a line. Since there is margine and padding on <li> elements, it is bad
<sinzui> bac: We removed a lot of thos during UI 3. I fixed a similar problem on the CVE page this week
<sinzui> bac if you can put it on the object's <a>, all will be well
<bac> ok.  i'll try that.  we do have specific li.mail rules in style.css
<sinzui> allenap: maybe I am fantasising.
<allenap> sinzui: I vaguely remember thinking about that problem. I might have written code for it, but I don't know where it is now. I would just use grep now.
<sinzui> bac: allenap: I think every lifecycle portlet should be removed. If something is using one, it should not.
 * sinzui has investigated how to remove all the dead 1.0/2.0 portlets
<flacoste> morning launchpadders!
<gary_poster> for some reason that term reminds me of "Liverpudlians"...
<flacoste> lol
<flacoste> gary_poster: did you notice that the new tags file contains all imports and not only definitions?
<flacoste> which makes it very inconvenient actually
<gary_poster> flacoste: I did not.  I expect that's a change to etags, not our tag machinery
<gary_poster> Maybe we can find a flag to do what we want
<flacoste> ah, you are right, it might be related to the Karmic upgrade
<flacoste> since I don't think we upgraded the recipe recently
<sinzui> barry: bac: stand-up in two minutes
<flacoste> gary_poster: there is an outside contributions to launchpadlib at https://code.launchpad.net/~kees/launchpadlib/safe-cred-dir/+merge/15011 waiting for a review
<gary_poster> flacoste: ack thanks
<flacoste> gary_poster: and can you arbitrate https://code.edge.launchpad.net/~jkakar/launchpadlib/testing-support/+merge/14444 at some point
<gary_poster> flacoste: yup, will do
<flacoste> thanks
<flacoste> gary_poster: and regarding the sourcecodedeps.conf parsing, you might want considering simply adding config-manager to the list of deps and using the parsing code from there
<flacoste> nothing like config-manager to parse config-manager's syntax :-)
<gary_poster> flacoste: ack
<jml> lifeless, thanks.
<jml> just a warning for those who are around
<jml> I'm going to pqm submit a branch that ec2 has been losing
<jml> so the test suite might fail. sorry.
<mars> gmb, around?
<mars> deryck, around?
<deryck> hi mars
<jml> yay buildbot failure
 * jml looks
<kfogel> rockstar: ping
<rockstar> kfogel, hi.
<rockstar> kfogel, fair warning, I'm about to go to a UDS session.
<kfogel> rockstar: hey, have you seen the internal thread "Upgrading bzr branches"?
<kfogel> rockstar: oh, nm
<kfogel> rockstar: no problem, I'll send a mail to the public list with the questions
<kfogel> rockstar: to launchpad-dev@ that is
<kfogel> rockstar: have a good session
<rockstar> kfogel, can you CC me when you send the email?  launchpad-dev@ is quite noisy.
<kfogel> rockstar: sure
<rockstar> kfogel, cheers.
<jelmer> mwhudson, hi
<mars> jelmer, just so you aren't stuck waiting, he regularly signs on at 21:00UTC, four hours from now.
<jelmer> mars: ah, thanks
<sinzui> Chex: ping
<Chex> sinzui: hello
<sinzui> Chex: I have a data fix for *production data*: https://pastebin.canonical.com/24936/
<sinzui> Chex: We ran the script on staging two days ago, and we see no reason to not run it in production now: https://pastebin.canonical.com/24861/
<sinzui> Chex: ^ The numbers will not match since we know users has had two days to add ad remove links, but they should be similar
<Chex> sinzui: ok,.. should I run that .py script as it is sitting on asuka?
<sinzui> did spm leave it there? There are the same
<Chex> sinzui: yes he did, copying it over now
<sinzui> That is best since we know that was the one for staging
<Chex> sinzui: https://pastebin.canonical.com/24939/
<sinzui> Chex: rock. that looks good.
<Chex> sinzui: no problem
<lifeless> bigjools: ?
<mwhudson> jelmer: hello
<sinzui> bac: what bug are you working on?
<bac> sinzui: i've sent off 485237 to ec2 after review but haven't grabbed another yet
<bac> sinzui: i was thinking about bug 266890 since it bothers me!
<mup> Bug #266890: Changing default bug visibility of a project to private triggers database constraint <oops> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/266890>
<sinzui> bac: I was think of doing bug 411686 myself
<mup> Bug #411686: can't merge to oblivion a private team <oops> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/411686>
<bac> sinzui: ok
<sinzui> I did not want to start a bug you were already working on
<bac> sinzui: i'm about EOD here so i may poke at the above bug some this weekend
<bac> i'll claim it now
 * mwhudson away for a bit
<jelmer> mwhudson, hi
<jelmer> mwhudson, still there?
<mwhudson> jelmer: am back now
<mwhudson> jml: is the "[r=jml]" in your recent testfix just a slip of the brain?
<jml> mwhudson, slip of my ethics, actually
<mwhudson> it seems thumper approved the merge proposal
#launchpad-dev 2009-11-21
<wgrant> bigjools: What's the best approach to testing gina support for 3.0 sources? Just add one somewhere in the test archive and fix any counts in the test that are now wrong?
<bigjools> wgrant: I'm not that familiar with the gina code + tests, but your approach sounds reasonable
<lifeless> wgrant: personally I'd make one on the fly, but there are a lot of tests that hard code content already
<wgrant> lifeless: That would be nice, but there are two problems: a) this is gina, and b) 3.0 sources can only be properly produced on >= Karmic
<lifeless> I don't see the problem
<wgrant> lifeless: The test suite has to run on Hardy. I don't really feel like backporting Karmic's dpkg. And there is no existing infrastructure for producing an archive that gina will like in a test.
<lifeless> wgrant: so, we need to be able to BFB and I epxect that has to support 3.0 sources too
<lifeless> so we need to be able to test that
<wgrant> lifeless: But the BFB code that deals with the actual source package will live inside lp-buildd, which is a completely different universe of ugly.
<wgrant> And I'm not sure 3.0 support needs to be tested there.
<wgrant> Since none of the new code has to know anything about the package's contents.
<mwhudson> thumper: the diff popups are cool
<thumper> mwhudson: I thought so :)
<mwhudson> rockstar: hello?
<mwhudson> rockstar: https://code.edge.launchpad.net/~mwhudson/launchpad/bzr-2.1b3-support/+merge/15114 :-)
<rockstar> mwhudson, hi!  I was just about to go to bed.
<mwhudson> rockstar: fair enough, do that
<rockstar> mwhudson, no, I can do this really quick.  I also need to call my wife before I'm out for the night.
<mwhudson> ok
<rockstar> mwhudson, r=me
<mwhudson> rockstar: thanks
<rockstar> mwhudson, I guess you can close the bug I just filed then...  :)
<mwhudson> yeah
<mwhudson> in fact i'll repurpose it to "upgrade to bzr-2.1b3"
<mwhudson> oh heh
 * mwhudson finds a bug i think: when you link a branch with a merge proposal, the diff link isn't properly ajaxified...
<rockstar> mwhudson, I wouldn't be surprised.  That's pretty new.
<mwhudson> yeah
<mwhudson> heh
<mwhudson> the dupe-finder is finding a heap of fixed bugs
<mwhudson> hm, maybe not quite actually
 * mwhudson goes to the pub
<wgrant> mwhudson: Are LOSAs on it?
 * wgrant cringes a bit and prepares a dpkg backport for the LP PPA.
<lifeless> wgrant: :)
<wgrant> It is not often that I complain that Debian is moving too quickly.
#launchpad-dev 2009-11-22
<wgrant> mwhudson: Will the EC2 AMIs automatically install new launchpad-dependencies deps?
<mwhudson> wgrant: no
<wgrant> Curses.
<mwhudson> they used to do that
<wgrant> Why don't they now?
<mwhudson> but somehow i lost that behaviour at some point and haven't put it back
<wgrant> Ah.
<mwhudson> i probably should
<mwhudson> building new images for ec2 test is really easy though
<thumper> morning
<ajmitch> hi thumper
<thumper> morning
<ajmitch> looks like you still need that coffee :)
<thumper> I'm drinking some was we speak (so to say)
<thumper> mwhudson: morning call?
<mwhudson> thumper: yes
<mwhudson> thumper: pick your technology
<thumper> mwhudson: lp prod is down
<mwhudson> thumper: !!
<thumper> mwhudson: skype should be fine
<mwhudson> thumper: heh, i reported that bug you just reported already :-)
<mwhudson> thumper: don't see you online
<jml> hello
<mwhudson> jml: hello
<jml> (I'm not working, but I thought I'd say hello)
<ajmitch> hello jml
<jml> ajmitch, hi
<wgrant> So, how quickly do I get mauled if I tell people that Hardy machines will soon need a dpkg backport to run the test suite, and germanium/cocoplum/iron need it too?
<thumper> mwhudson: hmm, the keyserver issue would explain why it couldn't validate my sig
<mwhudson> thumper: yes
<mwhudson> wgrant: i don't know
<wgrant> mwhudson: Who is likely to?
<mwhudson> wgrant: i guess spm would be a good start to talk to
 * thumper heads to get real coffee
<wgrant> spm: Once the disaster is sufficiently resolved, can you possibly assist with my earlier question?
 * maxb has reassigned the ~launchpad-dev branches to ~launchpad-committers
<thumper> mwhudson: call before lunch?
<mwhudson> thumper: now you mean?
<mwhudson> sure
<thumper> mwhudson: yeah
<spm> wgrant: hi; your Q ref the dpkg backport?
<wgrant> spm: That one.
<spm> wgrant: no no mauling. :-) just shoot an email to the devs list explaining why/what etc and we can run from there. that stuff will break spectacularly is a fair reason. :-)
#launchpad-dev 2010-11-22
<wgrant> lifeless: Well, he was told to try it a few hundred times, IIRC.
<lifeless> terrible advice
<wgrant> Probably, yes.
<lifeless> they're all the same, if he's in a rush to merge we could update those translations by hand.
<wgrant> I believe there's a question open.
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>     3555 /    0  LoginToken:+accountmerge
<lifeless>      298 /   55  Person:+commentedbugs
<lifeless>      231 / 6781  Archive:+index
<lifeless>       88 /  307  BugTask:+index
<lifeless>       13 /   10  Person:+bugs
<lifeless>       12 /  101  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>       11 /  260  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>       11 /   29  DistroSeries:+queue
<lifeless>        8 /    1  BinaryPackageBuild:+retry
<lifeless>        7 /   40  Milestone:+index
<StevenK> wgrant: O hai?
<wgrant> StevenK: Hi. Sort of here.
<wgrant> Exam tomorrow, but distractions are welcome :P
<StevenK> wgrant: I'm looking at modelling SourcePackageAncestry -- do you think per-distribution or per-distroseries is the way to go?
<wgrant> Hmm.
<wgrant> This needs a lot of thought.
<StevenK> I've been thinking about it over the weekend, and I'm currently on the distribution side
<wgrant> I think so.
<wgrant> But what are we modelling?
<StevenK> Just the versions present for an SPN in a distribution
<wgrant> Including relationships?
<wgrant> For what purpose?
<wgrant> Determining the shared ancestor?
<StevenK> wgrant: They'll be stored in order, with the newest first, for determing shared ancestors, yes
<lifeless> well, in-order is a lie with postgresql
<lifeless> indexed perhaps
<StevenK> It will stored as INTEGER[], which IS ordered
<StevenK> Er, TEXT[]
<StevenK> Sorry, brainfart
<lifeless> an array in a single row?
<lifeless> that seems risky
<StevenK> lifeless: It does?
<wgrant> Oh, I see.
<wgrant> That's a bit special.
<lifeless> StevenK: yes, if you have more than one mutator
<wgrant> I'd like a big brainstorm with everyone before we do anything, I think.
<StevenK> wgrant: Special bad, special good or special special?
<StevenK> I've spent about 15 minutes on this branch so far, so I'm perfectly willing to throw it away and write it off as a learning exercise
<wgrant> Arrays for this sort of thing in a relational database smell bad.
<wgrant> Particularly when we model versions already (as SPRs), and have ancestry in bzr.
<lifeless> what would be in each cell in the array?
<StevenK> lifeless: A version number
<wgrant> I think we probably need to store ancestry per (archive, distroseries, spr), but I may be wrong.
<StevenK> Per spr seems like a lot of duplicate information to me
<wgrant> Why?
<StevenK> wgrant: SPR has an version number stored with it, so the 'foo 1.0-1' spr will have ancestry of '1.0-1', '0.9-1', '0.8-1'; and 'foo 1.1-1' will have '1.1-1', '1.0-1', '0.9-1', '0.8-1'
<wgrant> StevenK: Hm? 1.0-1 will reference 0.9-1
<wgrant> 0.9-1 will reference 0.8-1
<wgrant> So we have a map like this:
<wgrant> (primary, hardy, 1.0-1) -> 0.9-1
<wgrant> (primary, hardy, 0.9-1) -> 0.8-1
<wgrant> That is the model that seems intuitive to me.
<wgrant> We don't store version strings -- we store SPR references.
<StevenK> Which is also quite hard to traverse :-)
<wgrant> Not if we drop the distroseries.
<wgrant> Which we may be able to do.
<lifeless> StevenK: why array's ?
<StevenK> lifeless: Because they can be grabbed from the db and pushed into a set, and then it's cheap to the find common ancestors
<lifeless> you should probably read all of http://www.postgresql.org/docs/8.4/static/arrays.html
<StevenK> I already have
<lifeless> StevenK: red flag alert: in postgresql whenever you say 'grab it all from the db', you're almost certainly doing it wrong.
<lifeless> StevenK: you can pull the versions from related spr's easily too.
<lifeless> that said, I'm not against denormalising here - nor am I for it as such.
<lifeless> Have you spoken with stub ?
<StevenK> Not yet
<lifeless> [by you, I mean the folk doing this design]
<StevenK> However, wgrant is starting to convince me
<wgrant> Yay.
<wgrant> Youknowimright.
<StevenK> wgrant: So, your idea is a 'ancestry' column, which links to the previous SPR? And if it's NULL, it's the first one seen?
<lifeless> (â¢)
<wgrant> StevenK: We can't do it on SPR itself.
<wgrant> StevenK: Since SPRs are shared between distroseries.
<StevenK> Perhaps I should argue more, just to deflate wgrant a bit
<wgrant> s/distroseries/distros/
<wgrant> s/distros/archives/
<lifeless> something to bear in mind
<lifeless> the data model currently blows up on duplicate uploads
<lifeless> we should fix things going boom when that happens. Would that influence this work ?
<StevenK> This is a feature
<lifeless> StevenK: its very much not.
<lifeless> StevenK: it causes two single points of failure, manual recovery, makes person merge of ppas hard.
<lifeless> StevenK: Not A Feature.
<wgrant> I would object to any person merge with PPAs.
<wgrant> There's no need for that in reasonable merge use-cases.
<lifeless> wgrant: on principle or technicalities
<wgrant> Except maybe team merges, I guess.
<wgrant> But then rename.
<wgrant> And that would screw up keys and URLs.
<wgrant> so block it.
<wgrant> block it all.
<lifeless> wgrant: hard to do does not imply not worth doing.
<StevenK> lifeless: Is this is upload to cocoplum and germanium at the same time bug?
<lifeless> StevenK: yes
<lifeless> StevenK: which stops us trivially load balancing the two uploaders.
<lifeless> StevenK: which means we can't deploy fixes to those servers but once a month.
<wgrant> yeah, it sort of fucks up your PPA if you exploit that :P
<StevenK> lifeless: Will this work impact that? No, not I can see
<wgrant> I don't know how to solve that.
<wgrant> It could be made to impact it.
<StevenK> Indeed
<wgrant> If we stored string versions.
<wgrant> That is interesting indeed.
<StevenK> lifeless: You do enjoy banging that SPOF drum :-P
<spm> good, that he does.
<StevenK> I think it can be solved, by changing the upload processor on both cocobanana and germanium to be a daemon process that scans the upload directories, and always talks to the master store
<wgrant> StevenK: Doesn't help.
<wgrant> We don't have sufficient constraints to forbid duplicate uploads.
<StevenK> That can be solved
<wgrant> If our model didn't suck, UNIQUE constraints would do it.
<StevenK> But it is a thirteen-table join, like you say
<wgrant> But I don't think we can model it sanely like that.
<wgrant> Right.
<spiv> Glancing at https://lpstats.canonical.com/graphs/CodehostingPerformance/ I wonder happened to codehosting 2 days ago?
<StevenK> wgrant: Maybe the answer currently is "Yes, we can do it, but it will blow, the database will suck, and you'll hate us" ?
<wgrant> bzr downgrade happened around then, maybe?
<StevenK> wgrant: Or perhaps we can expect a branch from you on Wednesday that fixes all of our modelling problems? *eg*
<spiv> Oh, huh, for the SFTP issue?
<wgrant> spiv: I believe so. There's an incident report.
<StevenK> wgrant: On a serious note, do you want me to look at your script on dogfood again?
<wgrant> StevenK: I don't start for three weeks :P
<lifeless> StevenK: enjoy it? no.
<lifeless> StevenK: enjoy the *benefits* of banging it?hell yes.
<wgrant> StevenK: That would be nice. But I don't know if DF will survive.
<StevenK> wgrant: There's no way to make the first query be, err, nicer?
<wgrant> I don't think so.
<wgrant> But maybe.
<wgrant> Remember, though, that the publisher takes hours on DF, but just a few minutes on prod.
<wgrant> So DF is special :P
<wgrant> Bad special.
<StevenK> wgrant: So kick the script off, and wait a few hours, and hide if IS tell me that mawson has exploded and set a few racks and Nafallo on fire?
<wgrant> Hopefully he'll be in the other DC at the time.
<lifeless> actually, the benefits of the the spofs being fixed.
<StevenK> wgrant: Remind me of the script name?
<wgrant> StevenK: populate-spr-changelog.py
<StevenK> Hah, df has moved on, and canonical.launchpad.database has died, so the script dies
 * wgrant disappears for a while.
<wgrant> Hah.
<StevenK> ARGH, everything this script wants is from canonical.launchpad
 * StevenK shakes his fist at wgrant 
<wgrant> StevenK: Heh, sorry. I am outdated.
<StevenK> wgrant: It's been running for ~ 40 minutes, since I fixed the imports
<wgrant> "its
<wgrant> my opinion that fixing our data mapping story is the highest leverage
<wgrant> item that requires TA input to achieve pervasive implementation and
<wgrant> deployment."
 * wgrant cries.
<lifeless> wgrant: why so sad?
<wgrant> lifeless: Highest leverage for achieving pervasive implementation.
<lifeless> wgrant: bingo?
<lifeless> wgrant: though you've mangled the sentence ther
<wgrant> StevenK: Has the query finished?
<StevenK> 2010-11-22 04:01:48 DEBUG3  new transaction
<wgrant> StevenK: Has London been evacuated?
<StevenK> Nope
<wgrant> lol
<StevenK> Not to my knowledge
<wgrant> It is tempting to try that query somewhere else.
<wgrant> And see if it sucks less.
<wgrant> Otherwise I guess I'll poke around once I have DF access eventually.
<lifeless> wgrant: so I'm still confused
<lifeless> wgrant: are you crying because it was jingoistic, or because it wasn't $othertask.
<spiv> Maybe because you missed one crucial square of his bingo card? ;)
<StevenK> Haha
 * spiv guesses the it was the "synergy" square.
<StevenK> wgrant: That query completed in 68 minutes
<StevenK> wgrant: http://pastebin.ubuntu.com/535084/
<StevenK> wgrant: Short answer, I killed it
<wgrant> StevenK: Yay :/
<wgrant> lifeless: The former.
<StevenK> wgrant: I've not seen an error like that, but it looks like it updated at least two SPRs :-)
<wgrant> StevenK: Indeed.
<wgrant> But I wonder if the query performs less atrociously on >mawson. :/
<StevenK> wgrant: I suspect we can get a RO counting query going, if you wish
<spm> ugh. launchpad_qastaging stop == make build. yay. not.
<wgrant> Hah. Fun.
<spm> stop should not be doing things like rm'ing directories, and moving stuff around. No. it has just one thing it needs to do and one thing only. terminate the running process. grumble grumble grumble.
<spm> orsum. and then the make fails and stop doesn't. this is gold.
<wgrant> StevenK: Can you stick your fixed script up somewhere?
<wgrant> I seem to have lost mine.
<wgrant> I can only find an old version :(
<StevenK> http://pastebin.ubuntu.com/535088/
<wgrant> Thanks.
<StevenK> Ugh, could the gina doctest be any slower
<wgrant> It's only a couple of minutes, right? :P
<spm> yes. ??
<wgrant> What are you doing to the poor creature?
<StevenK> wgrant: Having it toss changelogs at the librarian and linking them into the SPRs it creates
<wgrant> Ah, great.
<wgrant> I didn't get around to that.
<wgrant> mumble exams mumble
<spm> hah
<StevenK> $ bzr di | diffstat | tail -n 1
<StevenK>  2 files changed, 13 insertions(+), 6 deletions(-)
<StevenK> Is tiny, so far
<wgrant> Yep, it should be.
<wgrant> But the tests might be amusing.
<StevenK> Yes, I'm not looking forward to changing them
<wgrant> spm: Could you throw http://paste.ubuntu.com/535089/ at a production slave if you have time, pls?
<spm> sure
<StevenK> "Sure, but I have no time, so no" ?
<wgrant> Heh.
<spm> nah. I prefer to leave him in suspense for a few hours. the simple "sure" implies an answer that I'll do his request. but not mention of when. so it works from a bofh perspective. fwiw.
<spm> wgrant: https://pastebin.canonical.com/40005/ argh. ffs. wrong paste. one sec.
<StevenK> Haha
<spm> wgrant: http://paste.ubuntu.com/535090/
<StevenK> spm: That works. Yes, I've run the query, but you have to wait a few weeks for the results.
<wgrant> spm: Thanks.
<spm> I even knew to use the ubuntu one. but auto defaulted to the can pastebin. loltastic at self.
<wgrant> StevenK: How long does that same query take on DF?
<StevenK> wgrant: http://pastebin.ubuntu.com/535092/
<StevenK> lolwut
<StevenK> It's *quicker*
<wgrant> ... yes.
<wgrant> It must have had stuff cached?
 * wgrant stabs WADL in the mouth.
<StevenK> That's quite possible, since I re-ran it to actually get the paste. Duh.
<StevenK> Total runtime: 458.468 ms
<StevenK> And the universe makes sense again
<wgrant> That's more like it.
<wgrant> Now to get a longer one.
<wgrant> However this relies on my local appserver starting, which it seems to be reluctant to do.
<StevenK> Your machine is hinting that you have study to do
<wgrant> Silence.
<spm> mv lib/canonical/launchpad/apidoc.tmp lib/canonical/launchpad/apidoc
<spm> mv: cannot move `lib/canonical/launchpad/apidoc.tmp' to `lib/canonical/launchpad/apidoc/apidoc.tmp': Directory not empty
<spm> make: *** [lib/canonical/launchpad/apidoc/index.html] Error 1
<spm> wgrant: ^^ that error?
<lifeless> wgrant: how many exams left?
<spm> cause that's nicely broken qastaging
<StevenK> Oh, fricken apidoc
<spm> Ahh mr lifeless ^^
<wgrant> spm: ....
<wgrant> spm: I got that too.
<StevenK> spm: We have that error about once a month
<wgrant> spm: That's why I was stabbing WADL in the mouth.
<StevenK> On devel, at least
<wgrant> lifeless: Just the one.
<wgrant> 9am.
<StevenK> wgrant: For which subject?
<lifeless> spm: if I may be so bold.
<lifeless> spm: file a bug
<wgrant> StevenK: Theory of Computation
<spm> lifeless: was just about to; but was wondering if a known/quickish fix. no worries.
<lifeless> spm: note in the bug that a) we need to find the landing that broke things and mark it as bad somehow
<spm> nod
<lifeless> spm: secondly, would you be able to run 'make clean'
<wgrant> spm: Just blow away that dir.
<wgrant> Or do that.
<lifeless> spm: then make'
<StevenK> Hmmm
<lifeless> wgrant: nice stuff :)
<StevenK> Gina's changelog parser is different from the archives. Fun.
<spm> launchpad-rev-11956 would be a guess
<lifeless> wgrant: turing equivalence, NDFA etc ?
<wgrant> lifeless: Yup.
<lifeless> one of my fav subjects at uni
<spm> no. would have been before that.
<wgrant> One of the few subjects in which I've actually learnt something :/
<wgrant> StevenK: Is http://paste.ubuntu.com/535094/ nice and slow on DF?
<lifeless> wgrant: its good that you're learning stuff ;)
<lifeless> wgrant: it gets harder the older you get :)
<StevenK> wgrant: Not excessively, http://pastebin.ubuntu.com/535095/
<wgrant> lifeless: Oh, yes, the learning is a good thing. The restricted number of subjects in which it has happened isn't.
<wgrant> StevenK: :(
<wgrant> StevenK: Could you run the script with LP_DEBUG_SQL=1 and see what's slow?
<spm> wgrant: you need to do subjects like I did in CS. ie "How to be an ego ridden dork, who manages to piss off she who will one day be his wife, by having a working program in computer graphics: @3am in the labs; and doing more than was asked for in the assignment"
<wgrant> spm: Heh.
<spm> amusingly, 20 years later. she still hasn't forgiven me for that. :-D
<StevenK> wgrant: http://pastebin.ubuntu.com/535096/
<StevenK> That query will spin for ~ an hour if the last run is anything to go by
<lifeless> spm: was she competing for marks, or stood up for a date?
<wgrant> StevenK: /^troseries/ does not a valid query make.
<spm> heh, more like if you do more stuff than required, that tends to force others marks down on assignments. so she was shitty at having to do even more work. given her code wasn't running at that point in time. :-)
<StevenK> wgrant: Er, lemme fix that
<StevenK> spm: I submitted a 12 page Perl assignment for Systems Administration Programming, and got 95. The lecturer was cranky at me that she couldn't scale the marks, because if she did, the next best assignment would have gotten 60
<StevenK> wgrant: http://pastebin.ubuntu.com/535097/
<spm> StevenK: \o/
<wgrant> StevenK: Could you replace the two %s with 0 and see how long it takes?
<lifeless> StevenK: 12 pages of perl... shoulda marked you down
<wgrant> I wonder if the overly verbose Storm-generated GROUP BY is being expensive.
<wgrant> (I reduced it to a GROUP BY over SourcePackageRelease.id before)
<StevenK> lifeless: That only made me chuckle ... what got me laughing so hard was when she reached for Programming Perl and waved at me saying "Do you know how many times I had to refer to this to mark your assignment?!"
 * wgrant is glad that LP will be an escape from Perl... except for sbuild.
<lifeless> thats a rather crackful query
<StevenK> wgrant: Its still going,
<wgrant> StevenK: Aha.
<wgrant> StevenK: Excellent, thanks.
<StevenK> I suspect it will last the full 60-odd minutes
<wgrant> lifeless: Oh?
<wgrant> StevenK: Kill it pls (with fire, if you see fit)
<StevenK> wgrant: And the script?
<wgrant> StevenK: Yeah.
<StevenK> Oh, I love it when scripts don't listen to Ctrl-C, and instead just echo ^C
<spm> yay. qas is back.
<wgrant> lifeless: What is crackful about it?
<wgrant> Besides the insane groupby?
<StevenK> Pity killing the script didn't kill the query
<lifeless> whats the 'HAVING COUNT(LibraryFileAlias) = %s' for
<StevenK> If I kill a postgres child, will it get cranky?
<lifeless> yes
<StevenK> lifeless: s/%s/0/ in your head
<wgrant> lifeless: The %s is 0.
<spm> wgrant: be grateful you're not going from pascal, C and modula-2 @uni to Assembler and Cobol @ $job1. :-)
<wgrant> lifeless: It's finding SPRs without expired files.
<wgrant> spm: Ahahah.
<wgrant> And the query actually performs rather well.
<lifeless> I'm surprised it accepts it at all
<wgrant> Why?
<lifeless> count(TableName)
<StevenK> spm: Out of interest, do you have access to mawson?
<spm> StevenK: not really. I think we can login and that's about it.
<StevenK> Damn it, I want to kill that query
<StevenK> Oh well, the load is only 2
<lifeless> wgrant: so, 'LEFT JOIN LibraryFileAlias ON LibraryFileAlias.id = SourcePackageReleaseFile.libraryfile AND LibraryFileAlias.content IS NULL'
<lifeless> StevenK: there is a pg helper to kill backends cleanly
<lifeless> StevenK: or read the docs. IIRC TERM is ok
<wgrant> pg_cancel_backend, or something like that.
<StevenK> I doubt I have access to the postgres user, sadly
<lifeless> StevenK: if you were considering killing it, I assume you had some access :)
<StevenK> I wouldn't say no to access, I just like cleaning up after myself
<wgrant> StevenK: You must surely have a postgres superuser?
<lifeless> wgrant: + a HAVING constraint which implies NOT NULL
<lifeless> wgrant: so, it doesn't need to be a LEFT JOIN
<StevenK> wgrant: Well, yes, there is one, I just can't actually a shell as it
<spm> StevenK: I lie. we have more access than I recalled.
<StevenK> spm: Win. Can you switch to postgres?
<spm> I can.... :-P
<wgrant> StevenK: Don't you need a postgres superuser to run tests?
<StevenK> Dogfood doesn't run tests?
<wgrant> lifeless: How does the HAVING imply not null?
<wgrant> lifeless: HAVING COUNT(LibraryFileAlias) = 0
<StevenK> spm: kill -TERM 27507
<lifeless> wgrant: count(something that is NULL) -> NULL
<lifeless> NULL != 0
<lifeless> wgrant: no?
<spm> StevenK: that would be an unwise way of doing things; but gist of the request is understood. one sec.
<wgrant> lifeless: I don't think so...
<spm> StevenK: https://wiki.canonical.com/InformationInfrastructure/OSA/SafelyKillBackendProcess for the record
<wgrant> lifeless: http://paste.ubuntu.com/535099/
<spm> StevenK: done
<wgrant> So I guess I need to pre-select the IDs the turn them into real objects later :(
<StevenK> spm: Danke
<lifeless> wgrant: 'count(expression)anybigint number of input rows for which the value of expression is not null'
<wgrant> lifeless: eparse
<lifeless> wgrant: so an SPR without expired files
<lifeless> wgrant: is two queries
<lifeless> sprs without files
<lifeless> (exclude those)
<lifeless> and sprs with files that are not expired - that have content)
<lifeless> sorry, union them
<lifeless> (tired)
<poolie> hi lifeless
<lifeless> hi poolie
<poolie> welcome back(?)
<lifeless> heh, haven't been away ;)
<lifeless> poolie: just in orlando, still working
<poolie> has anyone tried/considered running launchpad in lxc?
<poolie> well, physically away
<poolie> or are you still there?
<lifeless> wgrant: that query wants 0 LFA's which will turn up if no LFA's are found, and we force no-files and files with content to null via the complicated join clause
<lifeless> poolie: no, nz now.
<wgrant> lifeless: Any SPR in prod without SPRFs is a bug.
<lifeless> wgrant: I think I'd write this as an EXISTS then
<lifeless> HAVING is forcing it to evaluate all results which you've helpfully inverted to make a negative.
<lifeless> but we don't care about the count
<lifeless> we care that there is one or more LFA with content.
<wgrant> True.
<lifeless> wgrant: I suspect it will perform a lot better - but you may need a DISTINCT in the EXISTS select clause
<wgrant> lifeless: The performance of the query is not problematic at the moment.
<lifeless> wgrant: an hour isn't problematic?!
<wgrant> lifeless: Except for the GROUP BY.
<wgrant> lifeless: That goes away magically when I write the query manually to group by SPR.id, rather than SPR.*.
<lifeless> aggregates are hard :P
<wgrant> You'd think pg would be smart enough to notice that there's a PK in there.
<lifeless> poolie: no idea
<poolie> it may be lighter wait than full virtualization, but still strong
<lifeless> ensemble is building around it
<lifeless> ok, afk
<poolie> jtv: woo!
<jtv> poolie: woo?
<jtv> woo whom?
<poolie> i think that's the first time i've ever had a proposal just land the first time
<poolie> well, i guess there was a fizzle in the mp
<wgrant> StevenK: Still around?
<StevenK> wgrant: Hm?
<wgrant> Could you try http://pastebin.ubuntu.com/535106/?
<StevenK> wgrant: Running
<lifeless> oh ugh
<wgrant> StevenK: Less obscenely slowly?
<lifeless> launchpad.net/linaro is a package
<lifeless> project I mean
<wgrant> lifeless: This could be amusing.
<lifeless> wgrant: what could be?
<wgrant> lifeless: Converting Linaro to a distro.
<lifeless> fortunately, NMP.
<lifeless> :P
<lifeless> 'hi duncan, please reregister. Thanks.'
<wgrant> You nasty strategists and architects...
<lifeless> we sit around all day playing buzzword bingo
<StevenK> wgrant: How long are expecting the new query to take? :-)
<wgrant> StevenK: Not long :(
<lifeless> wgrant: try an exists, you know you want to
<StevenK> lifeless: I'm not sure if non-admins can actually register a distro
<wgrant> They can't.
<StevenK> Therefore, 'hi duncan, please reregister' is completly pointless
<wgrant> lifeless: But I like avoiding subselects :(
<lifeless> they aren't always wrong
<wgrant> I know.
<StevenK> wgrant: 4 minutes, and counting
<wgrant> But I like avoiding subselects.
<wgrant> StevenK: Bah.
<lifeless> 8.4 seems to want DISTINCT a lot
<wgrant> Which query? LIMIT 1?
<lifeless> wgrant: I'm sure you'll try soon enough
<StevenK> wgrant: ENOLPSQLwhatsit
<wgrant> StevenK: :(
<wgrant> lifeless: We'll see.
<StevenK> wgrant: Kill it and run it with the DEBUG?
<wgrant> StevenK: Please.
<wgrant> Hm.
<wgrant> I guess it's probably doing the limit in the wrong place.
<wgrant> Sigh, SQL.
<stub> lifeless: I think that is a side effect of the short circuiting not happening as often (at all?) in subqueries
<lifeless> stub: yeah, I think that too
<lifeless> stub: possibly a correctness fix in some corner case. I haven't looked in the changelog to see though
<stub> (which might be trading performance for correctness, although the cases I can think of that need the short circuiting disabled should be detectable)
<stub> I actually recall removing some distincts before, as they used to slow some queries down
<StevenK> wgrant: http://pastebin.ubuntu.com/535109/
<lifeless> stub: \o/
<stub> (but others used to benefit from its addition too)
<stub> At least it is more consistent now ;)
<lifeless> wgrant: so we can use a regular JOIN on sourcepackagereleasefile right? if not playing silly buggers with having
<wgrant> lifeless: I want to find those SPRs that have no files with LFA.content IS NULL.
<wgrant> lifeless: How can I word that in a non-LEFT JOIN?
<wgrant> Unless I do an exists.
<lifeless> wgrant: Time: 6.897 ms
<lifeless> wgrant: the current group by *does not do that*
<lifeless> wgrant: the content is NULL clause matches the join, and then the hacing COUNT excludes them
<lifeless> or I may be misphrasing. Anyhow, one sec, I'll put that helpful english description into this
<wgrant> lifeless: The HAVING doesn't work without the GROUP BY...
<jtv> poolie: only thing I had to do about your MP was set a commit messageâso that all the metadata except the --no-qa option is in there and test-and-land becomes a very simple command line.  Congratulations, and thanks again for fixing this!
<wgrant> StevenK: http://pastebin.ubuntu.com/535111/
<lifeless> wgrant: so thats sprs where all their files are not null
<lifeless> 15 seconds on a first attempt
<lifeless> 11 seconds on hot cache
<StevenK> wgrant: Killed and re-running
<wgrant> lifeless: Which?
<wgrant> StevenK: Thanks.
<StevenK> Ooooh, that's MUCH quicker
<wgrant> Like actually not slow?
<lifeless> wgrant: http://pastebin.com/q6XXG2H0
<StevenK> I'm getting confused by the DEBUG
<wgrant> I think Foxtel has bought out every bit of Australian ad space.
<wgrant> I have not seen a non-Foxtel ad in weeks.
<StevenK> adblock FRW
<StevenK> *FTW
<StevenK> 2010-11-22 08:37:08 DEBUG   Iteration 0 (size 1.0): 1.043 seconds
<StevenK> 2010-11-22 08:37:08 DEBUG3  new transaction
<wgrant> I used to use it... but I go to so few adful sites that I didn't bother installing it this time.
<stub> If this is the pastbin http://pastebin.ubuntu.com/535106/ , I'm confused and the comment on the query doesn't match the query at all (comment is about finding sprf, query is finding sprs)
<wgrant> StevenK: Is it expanding the set size?
<StevenK> wgrant: http://pastebin.ubuntu.com/535113/
<wgrant> stub: The comments used to be interspersed with the relevant bits of the query. Then I rewrote it and moved them all to the top for now.
<wgrant> stub: But http://pastebin.ubuntu.com/535111/ is the latest version, which might make more sense.
<lifeless> wgrant: the comment and code are confused ;)
<wgrant> lifeless: Oh?
<lifeless> well it says 'Find any SPRFs that have expired (LFA.content IS NULL'
<lifeless> but you sau I want to find those SPRs that have no files with LFA.content IS NULL
<stub> I don't know what Count(LibraryFileAlias) actually does, but I suspect nothing useful.
<wgrant> lifeless: Have you read the version with the comments not all at the top?
<lifeless> http://pastebin.ubuntu.com/535111/
<wgrant> stub: Howso? It works fine.
<lifeless> wgrant: its unusual, sets of alarums
<StevenK> wgrant: This looks like it runs well, kill it?
<wgrant> StevenK: Yep, thanks.
<StevenK> wgrant: Bill is in the mail
<lifeless> wgrant: I'm very confused about whether you want 'expired sprfs' or 'no expired sprfs'
<lifeless> wgrant: I can get you expired ones in 6ms.
<wgrant> lifeless: I want those SPRs without expired SPRFs.
<lifeless> wgrant: why those in particular ?
<lifeless> wgrant: and why the specific case of sprs without *any* expired sprfs ?
<stub> I see. It is a confusing spelling of COUNT(LibraryFileAlias.id), which excludes NULLs unlike COUNT(*) would
<wgrant> lifeless: because I can't extract an SPR without all of its files.
<wgrant> stub: It wasn't a deliberate choice, I don't think. Does it make a difference?
<stub> COUNT(*) or Count(LibraryFileAlias.*) would match every row. Count(LibraryFileAlias.id) counts rows with a non-null id
<wgrant> stub: Hm, but COUNT(LibraryFileAlias) happily returns 0...
<stub> In SQL or Storm?
<wgrant> Both.
<StevenK> size is 2811, but 11244 were read from the file
 * StevenK grumbles at gina
<StevenK> wgrant: Oh, that script was running fast enough with 5 that I suspect it could be bumped to 20 or so easily
<wgrant> StevenK: Yay. Thanks.
<stub> http://paste.ubuntu.com/535119/ is the clearest SQL and likely fastest
<stub> But if it works
<bigjools> morning
<StevenK> 'AND SourcePackageRelease.id >= 1) AS Whatever' Haha
<adeuring> good morning
<wgrant> stub: I tend to avoid EXCEPT when it's easy. But if you say so :)
<stub> Its hard to say without benchmarking, and I haven't done that :)
<stub> In fact, that limit might have to go inside...
<wgrant> Anyway, this script works quickly now, and only has to be run once...
<StevenK> Crap, I think I made gina spin
<wgrant> stub: Is postgres not smart enough to reduce a 'GROUP BY SourcePackageRelease.id, SourcePackageRelease.foo, SourcePackageRelease.bar' to 'GROUP BY SourcePackageRelease.id'?
<StevenK> I thought GROUP BY were never twiddled with
<stub> SQL isn't
<lifeless> http://pastebin.com/bsCfW3Ya
<wgrant> stub: But surely it can see that it's just grouping across real fields, one of which is a PK.
<stub> Yes, but the SQL standard wants them listed IIRC
<stub> PG could extend the spec here I guess, but hasn't
<wgrant> stub: I know that SQL requires that they be listed. But PG also seems to execute them.
<wgrant> When it can tell that it's optimisable down to SourcePackageRelease.id.
<mrevell> Hi
<lifeless>  SELECT SourcePackageRelease.*
<lifeless>     FROM SourcePackageRelease
<lifeless>     WHERE SourcePackageRelease.id >= 1 and not
<lifeless> SourcePackageRelease.id in (SELECT SourcePackageReleaseFile.sourcepackagerelease
<lifeless>     FROM SourcePackageReleaseFile, LibraryFileAlias
<lifeless>     WHERE SourcePackageReleaseFile.sourcepackagerelease = SourcePackageRelease.id and SourcePackageReleaseFile.libraryfile = LibraryFileAlias.id
<lifeless>     AND LibraryFileAlias.content IS NULL
<lifeless> ) order by id desc limit 1;
<lifeless> stub: wgrant: ^
<lifeless> 20ms
<wgrant> lifeless: Ah, that's not bad.
<lifeless> 420.120 ms to get 55 rows
<wgrant> Would a NOT EXISTS be saner, though?
<lifeless> seemed to perform worse before
<wgrant> :(
<lifeless> but it wasn't quite identical
<lifeless> lets see
<lifeless>  SELECT SourcePackageRelease.*
<lifeless>     FROM SourcePackageRelease
<lifeless>     WHERE SourcePackageRelease.id >= 1 and not
<lifeless> EXISTS (SELECT 1
<lifeless>     FROM SourcePackageReleaseFile, LibraryFileAlias
<lifeless>     WHERE SourcePackageReleaseFile.sourcepackagerelease = SourcePackageRelease.id and SourcePackageReleaseFile.libraryfile = LibraryFileAlias.id
<lifeless>     AND LibraryFileAlias.content IS NULL
<lifeless> ) order by id desc limit 1;
<lifeless> 11 seconds
<lifeless> *yawn*
<lifeless> gnight
<wgrant> Night.
<wgrant> Thanks.
<stub> wgrant: http://paste.ubuntu.com/535123/ is 20ms to retrieve approx 55 rows, and for this use we don't care about the approx.
<wgrant> stub: Is it worth it?
<stub> I have no idea, but you guys seemed to be optimizing it :)
<wgrant> We'll be making lots more queries.
<wgrant> I don't want to optimise it, lifeless does :P
<stub> Is this a run once?
<wgrant> Yes.
<stub> Don't care then :)
<wgrant> Yay.
<stub> As long as it runs in days rather than weeks.
<wgrant> I just wanted it down from an hour on DF... which was achieved by fixing the GROUP BY.
<StevenK> 68 minutes down to about 30 seconds or so
<wgrant> bigjools: Have the recent log parser changes actually been tried?
<wgrant> I recall a couple of emails to the LOSAs but I never saw a response (although that may have just gone to canonical-launchpad and not me)
<jml> morning
<jml> this cold is finally getting to be a hassle
<bigjools> wgrant: otp, but yes and it DOSed germanium
<wgrant> bigjools: Still!?
<wgrant> :(
<bigjools> still
<wgrant> They must have pulled some nice tricks with the librarian one the first time :(
<jml> another incident?
<bigjools> wgrant: the librarian logs are nowhere near the size of the Apache ones for PPAs
<wgrant> bigjools: Ah.
<wgrant> Sad.
<wgrant> bigjools: Even with the max set?
<wgrant> And it no longer reading all the lines in first?
<bigjools> wgrant: I don't know the details, I am yet to re-create on DF.
<bigjools> mostly because I've not tried to re-create, mind ...
<wgrant> I guess I can do that in a couple of weeks.
<bigjools> I've got the production logs on it ready to go
<wgrant> Ah, great!
<wgrant> So it's not entirely hopeless.
<bigjools> but I need to fix the 2 thousand soyuz oopses per day first
<jml> mrevell: hi
<mrevell> Hell jml
<jml> That I am :)
<wgrant> bigjools: PFOE?
<mrevell> heh
<bigjools> wgrant: yes
<bigjools> wgrant: also stub's change to make log.error generate an OOPS is making me sad
<wgrant> bigjools: Yeah, it would :(
 * bigjools gets caffeine while waiting for rf-get
<stub> You don't want the behaviour, or you don't like what it is telling you?
<wgrant> stub: Soyuz is ancient and broken :P
<bigjools> stub: the code that logs an error for upload problems doesn't differentiate between what should or should not be an oops (it can be a package problem or a code problem)
<bigjools> so I need to fix the code
<jml> onwards and upwards!
<jml> wgrant: had a look at the distributionmirror_prober recently?
<wgrant> jml: We offloaded that to Registry :D
<wgrant> NFI why.
<wgrant> But it's damn convenient.
<bigjools> it's always been registry
<wgrant> jml: Why?
<jml> wgrant: when you said broken and ancient,  I was reminded of it (or at least its tests)
<wgrant> jml: Ah, yes.
<bigjools> lol
<wgrant> It's possibly even more broken and ancient than most of the rest.
<wgrant> Apart from gina.
<wgrant> Although gina was refurbished.
<wgrant> It still sucks.
<StevenK> I know :-(
<StevenK> I keep wanting to rip out all of kiko's XXXs
<wgrant> And some of the older bits of archivepublisher are fairly nauseating.
<wgrant> But one day it will be fixed.
<wgrant> Maybe.
<jml> about to try to land the branch that changes all of the twisted tests bar the prober ones to use testtools
<wgrant> Hah.
<wgrant> That bad?
<jml> they basically need to be rewritten.
<jml> it's not that hard
<wgrant> One could say that about 90% of the Soyuz tests...
<jml> but it shouldn't block getting this large branch integrated
<jml> wgrant: well, these are actually just generally incorrect
<wgrant> jml: ... oh.
<jml> (*always* return the Deferred!)
<wgrant> james_w: What happened to those massive anti-sampledata branches of yours?
<bigjools> jml: see, now I understand about always returning the Deferred :)
<jml> bigjools: yep :)
<wgrant> I really need to learn Twisted properly.
<bigjools> wgrant: I would like that
<wgrant> Since I don't really completely understand the intricacies of the new buildd-manager. And that could be problematic.
<bigjools> wgrant: it's not too hard once you get into it
<wgrant> Indeed, I probably just need to do it.
<bigjools> but there is a bit of a learning cliff
<wgrant> Can't be as bad as Zope's.
<wgrant> And I picked that up OK.
<bigjools> I tried to document all the gotchas as much as possible, I probably missed a load though
<jml> wgrant: Twisted is less abstract than Zope and has fewer grand ideas, imo.
<wgrant> Great.
<jml> wgrant: also, it has the happy advantage of having primarily XML-hating developers
<wgrant> Hah.
<wgrant> ZCML's not that bad.
<jml> the syntax isn't that bad, I guess
<wgrant> I prefer Grok's approach, yes, but it's not that bad.
<jml> the action-at-a-distance thing is a bit of a pain.
<bigjools> <wgrant> ZCML's not that bad.
<bigjools> who are you and what did you do with wgrant?
<wgrant> bigjools: There are far worse things!
<bigjools> like limb loss?
<bigjools> having heard that Twisted devs hate xml, I want to become one!
<wgrant> For one.
<wgrant> A lost limb? An XML? Or a Twisted dev?
<bigjools> wgrant: it's that time to remove jaunty's old binaries from the librarian, can you sanity check this for me, it's the same as the one for intrepid.   http://pastebin.ubuntu.com/535163/
<wgrant> bigjools: Replace the "- interval '1 week'" with "+ interval '1 week'" and you have a deal.
<bigjools> wgrant: well we don't need to add any interval really, the GC has a built in one, which is why we took it *off* in the last query as time/space was tight.
<wgrant> bigjools: At which stage is it built in? Does it set the expiry time in the future?
<bigjools> no, it adds a week to the timestamp in the query it uses
<bigjools> IIRC
<wgrant> I'm extremely uncomfortable OKing a query that I know to be buggy (probably insignificantly, but still) when it's not an emergency, and I have an exam in 10 hours.
<bigjools> heh
<bigjools> go to bed
<bigjools> what was buggy last time?
<wgrant> We didn't check dateremoved.
<bigjools> in the EXCEPT clause?
<wgrant> Rght.
<wgrant> If we don't exclude unremoved files, p-d-r will cry.
<bigjools> right
 * bigjools fixors
<wgrant> How critical are we for space?
<bigjools> 300G
<wgrant> It means nothing to me...
<bigjools> heh
<bigjools> not crtitical but in the time-to-think-about it stage
<wgrant> Yep.
<wgrant> I'll go through the query properly after my exam, then we can run it without the expiry mangling tomorrow night, then the morning after I will realise another flaw in it, and we will have no blown away the files yet :)
<bigjools> http://pastebin.ubuntu.com/535167/
<bigjools> :)
<wgrant> We've fucked it up somewhat the last three times. Let's not do it again :)
<bigjools> the p-d-r thing was the only problem last time
<wgrant> Er.
<jml> ffs
<wgrant> Yeah, looks good.
<jml> why does absolutely *every* site that wants my money demand that I create a bleeding account
<bigjools> so you can be labelled and tracked
<wgrant> jml: Everyone should accept OpenID.
<wgrant> *nudge nudge*
<jml> these guys *do* accept openid
<wgrant> Hah.
<jml> but they also need me to make up a password and so forth
<wgrant> As many do :(
<wgrant> Sounds like Launchpad!
<wgrant> Anyway, I should probably sleep.
<deryck> Morning, all.
<jml> deryck: good morning
<wgrant> bigjools: Heh, that query's still bad.
<wgrant> I will fix tomorrow.
<wgrant> Sigh.
<bigjools> sigh
<wgrant> This is hard.
<bigjools> yes
<bigjools> what's up?
<wgrant>                 AND bpph.status in (2, 5)
<wgrant>                 AND bpph.dateremoved is NULL
<wgrant> I think the first one probably wants to be removed.
<wgrant> Since the second will clearly be true in all those cases.
<bigjools> true
<bigjools> but
<bigjools> competing p-d-r processes.  Grah
<wgrant> Hm?
<jml> gror my head
<bigjools> basically we need to run this periodically
<bigjools> for all obsolete series
<bigjools> or, fix p-d-r not to care
<wgrant> That and obsolete-series.py itself, yeah.
<wgrant> Hm?
<wgrant> Why does p-d-r care?
<bigjools> the problem last time ...
<wgrant> Ah, yes.
<wgrant> Or we could forcibly delete all the PPA pubs from obsolete series.
<bigjools> no
<wgrant> :(
<bigjools> some people are using obsolete series
<bigjools> they are crackpots, I know
<wgrant> Those people are wrong.
<wgrant> Anyway, I will recheck that query tomorrow while I am still able, since that will resolve the immediate issue. But I would later like to know these wondrous use-cases for obsolete series, since I'm fairly sure they're insane.
<wgrant> Night.
<wgrant> It would make obsolete series handling much easier :(
<bigjools> nn wgrant, good luck with the exam
<jml> crappy crapzor mccrapperton
<jml> do we have an existing mechanism to use the librarian as a fixture rather than as a layer?
<jml> I guess not, since testSetUp and testTearDown both do something in the librarian layer.
<jml> which means getting the log of the librarian during tests is going to be a pain
<sinzui> jml ping
<jml> sinzui: hi. otp
 * sinzui will talk later
<henninge> jtv: is your current branch browser code branch pushed?
<jtv> henninge: yes.
<jtv> Currently going through review for the migration script with Aaron.
<henninge> jtv: Does it touch CurrentTranslationMessageView?
<jtv> Yes.
<jtv> Had to.   :(
<henninge> that's ok
<henninge> jtv: there are two branches (pre and normal). I guess the one depends on the other?
<jtv> Yes.
<jtv> The "pre" branch is waiting for review (though I'm keeping our reviewer busy; he found what looks like a serious gotcha in the script)
<jtv> If that gets through unscathed, then the regular branch is also ready for review.
<jtv> Meanwhile, I guess I'll be adding tests to the migration script and fixing the bug the review turned up.
<bigjools> deryck: is it bad that I found it hard to read anything after the line "disable windmill tests" in your last email? :)
<deryck> bigjools, I guess not ;)  but yeah, maybe so :-)
<mars> abentley, you have another bzr pipeline convert in matsubara :)
 * mars tea :) mouse battery recharge :(
<mars> wrong macro
<abentley> mars: Cool.
<jml> sinzui: ping
<sinzui> hi jml
<lifeless> moin
<deryck> morning, lifeless
<lifeless> hi deryck
<jml> sinzui: 173 oops bugs, 59 timeout bugs
<mars> good morning lifeless.  We don't normally see you around at this hour
<lifeless> mars: I'm not normally jetlagged either :)
<mars> rough.  That's twice in four weeks, isn't it?
<lifeless> bigjools: hi
<lifeless> bigjools: is https://bugs.launchpad.net/soyuz/+bug/654372 good to go ?
<_mup_> Bug #654372: Source domination is inefficient <new-release-cycle-process> <qa-needstesting> <soyuz-publisher> <tech-debt> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/654372>
<bigjools> lifeless: testing it *right now* as it happens :)
<lifeless> \o/
<bigjools> I've been doing QA all freaking day
<bigjools> unfortunately dogfood is slower than anything I can think of a funny idiom for right now
<lifeless> than a laser with a fricken shark on its handle?
<bigjools> I can think of plenty of really offensive comparisons
<lifeless> bdmurray: hi
<lifeless> https://bugs.launchpad.net/malone/+bug/594247 - is that gtg on qastaging ?
<_mup_> Bug #594247: searchTasks with structural_subscriber times out regularly <dhrb> <overlyprecise> <pg83> <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by adeuring> <https://launchpad.net/bugs/594247>
<adeuring> lifeless: gtg?
<lifeless> good to go
<adeuring> lifeless: yes, let me update the tag
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: importd system failing to import | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> lifeless: good morning
<lifeless> jml: oh hai
<bigjools> good night all
<lifeless> jml: I landed a couple of self reviewed branches since the optional review experiment started.
<lifeless> jml: could you please review those for me, per the 'robert will review everything except his stuff, jml will review that' bootstrap phase?
<jml> lifeless: sure. can you maybe email me the revnos or mps or something?
<jml> lifeless: otherwise I'll make a note to figure those out myself.
 * jml partly afk assembling whiteboard
<lifeless> make a note.
<lifeless> If I determine them I will mail you
<lifeless> jtv: when moving things to Ubuntu, if you change the projet to null, we won't get more email.
<jml> lifeless: btw, I'm getting exactly one failure w/ my testtools branch, it's a 500 response from the librarian. has your recent fixture/layer work made it possible for me to get the librarian log attached to the error?
<jml> (I can't seem to reproduce locally with a subset of the tests)
<lifeless> there is glue for that in this branch - https://code.launchpad.net/~lifeless/launchpad/librarian/+merge/39013
<lifeless> its not landed yet but you might find merging it, running the test with it, to help you
<jml> lifeless: ok, thanks.
<jml> I'll give that a try.
<lifeless> if it blows up you might find grabbing the addDetail change to be easy enough to pull out, but I think the branch is in tolerable shape.
<jml> it's also making the fixture a member of the layer, I think. not such a big deal.
<lifeless> the reason its not landed is it depends on teh databasefixture layer which is broken in 'make check' (but not bin/test) atm
<lifeless> s/layer/branch/
<jml> meh. conflicts.
<lifeless> :<
<jml> too sick/tired to deal with that now.
<jml> I guess this branch lands Wednesday at the earliest.
<lifeless> go rest
<lifeless> get well
<lifeless> nothing good comes of working sick
<lifeless> fun stuff - http://www.facebook.com/note.php?note_id=76191543919
<lifeless> deryck: hi
<deryck> hi lifeless
<lifeless> deryck: something for consideration in your js landing stuff
<deryck> lifeless, sure.  shoot.  Was just replying to your email.
<lifeless> I think its easy to end up with measurement bias in assessing the benefits of even terrible tests
<lifeless> the problem is in gathering data when there is a filter (the tests) on half the range
<deryck> lifeless, ok, I don't know that I fully follow what you mean there.  :-)
<lifeless> ok
<lifeless> so, we're mentally saying 'gee, I think that <windmill> only sometimes catches failures, so we only sometimes have defects it would catch'
<lifeless> but the actual set of failures it catches is 22 times larger than the local-test-failures-one-encounters
<lifeless> (we have ~ 22 active coding developers)
<lifeless> the tests, terrible as they are, are filtering out failures from all developers, but individually we only perceive the failures we've made ourselves.
<deryck> no, I don't think  'gee, I think that <windmill> only sometimes catches failures, so we only sometimes have defects it would catch'
<deryck> I wouldn't say it like that
<lifeless> deryck: ok
<deryck> I do think Windmill catches some failures.  I don't think that means we only sometimes have defects it would catch.
<deryck> I'm not sure what that means for your argument :-)
<lifeless> I'm not really arguing
<lifeless> throwing an issue I struggle with when evaluating the benefit of tests - any tests - into the right for discussion
<lifeless> since its part of the risk analysis.
<lifeless> s/right/ring/
<deryck> ah, right
<deryck> so how do we determine the value of these tests?  That's the heart of your analysis, right?
<lifeless> its extremely hard to reason about life without any given set of tests, because there are so many places that the filtering effect (filtering out defects) that they offer occurs at.
<lifeless> deryck: I think its an input into assessing the value of the tests, yes.
<deryck> lifeless, so you're saying that the tests filter defects in ways that we can't know, because we only the past defect the tests did catch.  But we don't know the things they catch they we were never aware of.
<deryck> that's hard to type
<deryck> it's a bit meta-physical to me. :-)
<lifeless> yeah
<lifeless> more simply
<lifeless> you never see my mistakes that tests caught.
<lifeless> *we* never see each others mistakes that tests caught.
<deryck> ah, ok
<deryck> I know mine, but not yours.
<lifeless> but when we remove the tests
<deryck> I agree with that.  But that also assumes you write js or windmill.  And well, there ain't that many of us doing it for lp. ;)
<lifeless> you will feel many more, perhaps even all, of the mistakes.
<lifeless> deryck: ack, as I say - this is a general modelling thing
<deryck> lifeless, right, and I completely agree with you in the abstract.
<lifeless> deryck: for clarity, I'm ok with what you're proposing, tentatively.
<lifeless> I don't think I'll get beyond tentative ;)
<deryck> lifeless, fair enough.  I don't think I can make you feel any better about it either :-)
<deryck> lifeless, is there anything I need to do because it's "tentative" or can I move ahead, assuming no one else objects.
<lifeless> oh, I'm nonblocking as always
<lifeless> if I wanted to stop it I would say so very clearly.
<lifeless> I'd be happiest if we just fix windmill
<lifeless> but you're doing the work - you get to decide on the right path.
<lifeless> if I was doing it, i think I'd fix windmill
<deryck> fair enough,  thanks!
<deryck> I'm replying in email on why I don't think we should do that.  or why I choose not to, rather.
<lifeless> thanks
<lifeless> flacoste: when did windmill become unmaintained ?
<lifeless> statik: we on today?
<jml> I think I overreached with this whiteboard
<statik> lifeless: sure thing, just wrapping up with IBM
<mars> lifeless, by unmaintained, do you mean, when did mikeal leave the project?
<lifeless> mars: I don't know what I mean.
<mars> lifeless, or do you mean umaintained in LP - no active work
<lifeless> mars: see flacostes mail
<mars> lifeless, oops, my bad - /me goes to read context first
<deryck> I believe unmaintained == mikeal leaving.
<deryck> windmill is not under active development anymore, as I understand it.
<lifeless> is there any reason to believe that the 512K bug wasn't inherited from Selenium?
<deryck> I don't know.  rockstar worked on that bug before and could say, I think.
<mars> deryck, well, there are developers, but the project velocity may have slowed a fair bit.  (I haven't been keeping close tabs though)
<deryck> yeah, the project web site has a new face lift.  So someone cares about it still.
<rockstar> lifeless, I wasn't aware that Windmill inherited anything from Selenium.
<lifeless> rockstar: wasn't windmill a fork of selenium, way back when ?
<rockstar> lifeless, no, not at all.
<lifeless> rockstar: I've been misinformed then. Someone should vote that comment down :)
<mars> lifeless, IIRC: Selenium was a complex piece of tech, the windmill devs thought they could have done some parts better - windmill tried a different way
<deryck> I don't see much difference myself between selenium 1 and windmill.  Just that windmill is Python.
<mars> lifeless, now Selenium 2 apparently has taken the same approach that windmill did.
<rockstar> mars, yeah, and the windmill devs hated selenium.
<deryck> or maybe it's selenium 2, I'm thinking of ;)
<mars> deryck, it was under the hood.  SSL handling, cross-domain requests, etc.  This is messy tech
<rockstar> Mikeal said himself "and, for the most part, the things we hated about Selenium are basically fixed now."
<deryck> yeah
<deryck> I actually have less issue with windmill vs. selenium than the kind of testing we chose to do with windmill.
<rockstar> deryck, everyone but the Selenium 2 folks are staying away from Selenium 2.
<rockstar> deryck, yeah, me too.
<deryck> I think using selenium is better since it's more widely used and because it forces us to think differently about how we test, but I don't think selenium is a magic bullet.
<lifeless> statik: ok, skype me when ready?
<mars> rockstar, "everyone but the Selenium 2 folks are staying away from Selenium 2" - care to elaborate?
<flacoste> deryck: they did made a release last week though
<flacoste> windmill 1.4
<statik> lifeless: you don't seem to be online in skype
<lifeless> fixed
<rockstar> flacoste, yes, but no one's really actively working on it.  It's just patches that people send in now.  That release has been on deck for a while now.
<rockstar> mars, I don't have many other details, other than Selenium 2's java server stuff is apparently really whack, I guess.
<rockstar> mars, this is all speculation, as is everything in our javascript testing story.
<rockstar> mars, one thing that is clear is that we need to be careful how we invest in any ship, because if we commit too strongly to something, we'll be sinking far too fast.
<rockstar> Er, we may find ourselves on a sinking ship.
<mars> Yep.  Always a risk (I've been down this road more than once)
<mars> Alas, there is no formula to predict which budding OSS project in a given arena will be best for your needs 2 years from now.
<flacoste> deryck: ,/bin/lp-windmill -m firebug jsdir=lib/canonical/launchpad/windmill/jstests http://launchpad.dev:8087/
<lifeless> flacoste: hi
<wgrant> I'm confused.
<wgrant> The lazr-js upgrade triples our JS size?
<wgrant> Isn't that a bug?
<flacoste> lifeless: finishing off with deryck
<lifeless> flacoste: ah, my bad.
<lifeless> flacoste: sorry ;)
<lifeless> wgrant: I think so.
<deryck> yes, it's a bug.
<wgrant> If we need more than a megabyte of JS on every page, I think we have a serious problem :/
<deryck> I agree it's serious.  hence, my effort to solve it. ;)
<wgrant> Great.
<thumper> morning
<deryck> we don't ever need 1.3 Mb of js for any page.  But we're concat'ing of all yui + lp js + lazr-js for every page.
<deryck> and yui is the real beast
<deryck> morning, thumper
<wgrant> Ahh, I see.
<deryck> yui is designed with batteries included like Python, thinking you'll dynamically load the modules you'll need.
<deryck> and we've never used it like that
<wgrant> Does U1?
<wgrant> It seems like U1 has solved this sort of thing.
<wgrant> But I can't really see..
<deryck> u1 has a smaller base yui, I believe.  but rockstar is working on this for them.  he could say for sure
<wgrant> Ah.
<salgado> hi thumper
<salgado> thumper, I think I remember reading somewhere that you had plans to work on blueprints in the near future.  have you started yet?
<thumper> salgado: not really
<thumper> salgado: I've done a little yak shaving
<thumper> salgado: but not started in earnest
<salgado> thumper, did you have any plans to expose them over the API?
<thumper> yes
<salgado> thumper, then it looks like I have good news for you.  project managers in Linaro need that ASAP, so I'm taking james_w's branch branch which started exposing blueprints over the API and try to nail down the remaining issues to get it landed
<thumper> salgado: sounds good
<salgado> lp:~james-w/launchpad/safe-blueprints-model is the branch, btw.
<rockstar> wgrant, U1 doesn't ship lazr-js, which drastically reduces its footprint.
<rockstar> wgrant, also, until recently, we were loading each file separately in U1 (bad).
<lifeless> salgado-afk: awesome
<lifeless> salgado-afk: one small request
<lifeless> salgado-afk: please don't put the blueprints in api/1.0
<lifeless> rockstar: hey
<lifeless> rockstar: remember at uds you had a features issue in a template
<lifeless> rockstar: I would love to know what the problem was
<rockstar> lifeless, uds was an eternity ago.
<rockstar> :)
<rockstar> lifeless, I remember it had something to do with template macros not having the same scope or something.  We were find that it was creating a new WSGI request for each template macro, even if it wasn't actually a net request itself.
<rockstar> I don't remember how I fixed it though.
<lifeless> ah well
<lifeless> thanks
<rockstar> lifeless, are you having an issue now?
<lifeless> no
<lifeless> I wanted to:
<lifeless>  - think about root cause fixes
<lifeless>  - have the answer handy if someone else encounters it
<LPCIBot> Project devel build (237): FAILURE in 4 hr 5 min: https://hudson.wedontsleep.org/job/devel/237/
<lifeless> meep
<lifeless> n
<lifeless> http://news.cnet.com/8301-30685_3-20023535-264.html
<lifeless> gmb: isn't it late for you ?
#launchpad-dev 2010-11-23
<wallyworld_> thumper: should i ask sk to review those 2 daily recipe build list page branches?
<poolie> bugclient now fails with
<poolie> AttributeError: 'Entry' object has no attribute 'markAsDuplicate'
<poolie> did this recently change in the api?
<lifeless> what api version are you using?
<lifeless> ?11958
<james_w> lifeless, api/1.0> why not?
<lifeless> james_w: ?
<poolie> bug 11958
<_mup_> Bug #11958: Unable to show hidden files <nautilus (Ubuntu):Fix Released by seb128> <https://launchpad.net/bugs/11958>
<lifeless> oh, that was a typo
<lifeless> pasted to the wrong window
<james_w> lifeless, <lifeless> salgado-afk: please don't put the blueprints in api/1.0
<lifeless> james_w: right. Because 1.0 is supported for many more years.
<lifeless> but jml has the issue tracker thing in train for the next year.
<james_w> why not stop the default being to add features to old versions then?
<lifeless> there's a bug open on that
<lifeless> IIRC
<james_w> ok
<lifeless> if there isn't, there should be - this has been discussed.
<lifeless> we shouldn't 'support' things we haven't finished.
<lifeless> we should do best effort etc
<lifeless> but we need to allow room for mistakes.
<poolie> hello james_w
<james_w> hi poolie
<lifeless> poolie: hi, so what api version are you using?
<poolie> i don't know
<lifeless> poolie: I ask, because its possible some things that were accidentally exposed have been shuffled, but we're meant to be very conservative about removing stuff.... trying to estimate whether to dig deeper or not.
<poolie> i seem to be just taking the default?
<poolie> <bug at https://api.edge.launchpad.net/1.0/bugs/415936>
<_mup_> Bug #415936: Merge into new branch produces strange log <Bazaar:New> <https://launchpad.net/bugs/415936>
<poolie> so 1.0, i guess
<lifeless> hmm, you're using edge too :)
<lifeless> could you switch to LPNET_SERVICE_ROOT :)
<poolie> heh, it's a hangover from that once being the only place to get it
<poolie> much better to switch things on/off on the server
<poolie> sure
<lifeless> we can't sadly
<lifeless> lp API clients start with a POST
<poolie> i mean in future
<lifeless> and blow up if thats redirected or otherwise handled by non-edge.
<poolie> sure
<lifeless> uhm, so 1.0
<lifeless> lets see
<poolie> we could change lplib to make EDGE_SERVICE_ROOT == LPNET_SERVICE_ROOT
<poolie> but that might just cause confusion
<lifeless> we're going to do something like that
<lifeless> perhaps with a deprecation warning
<poolie> hm that name doesn't exist
<poolie> ok, i have it
<lifeless> ah cool
<lifeless> what was it ?
<lifeless> ok, so markAsDuplicate is in beta
<lifeless> not in 1.0 or devel
<lifeless> I can check the changelog but my guess is that you've moved from a launchpadlib that used beta by default to one that uses 1.0 by default
<lifeless> the current idiom is to use duplicate_of = xxx
<poolie> hm
<poolie> something like that
<lifeless> you can pass beta as the api version you want
<poolie> though, this is on my maverick machine, running the packaged lplib
<lifeless> or update your code
<poolie> i doubt that changed in the last few weeks?
<lifeless> that would be unusual.
<lifeless> could you file a bug? Probably a mistake then.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (238): FIXED in 3 hr 45 min: https://hudson.wedontsleep.org/job/devel/238/
<LPCIBot> * Launchpad Patch Queue Manager: [r=abentley][ui=none][no-qa] New "images" command for bin/ec2 to
<LPCIBot> display all current test images.
<LPCIBot> * Launchpad Patch Queue Manager: [r=flacoste][ui=none][bug=677305] Downgrade bzr to 2.2.0
<poolie> hm, i see i was repeating myself and defaulting to edge in two places
<lifeless> you might find https://dev.launchpad.net/PolicyAndProcess/OptionalReviews interesting
<poolie> i've followed some of the mail about it
<poolie> ok, so even on 1.0 non-edge, it still fails
<lifeless> markAsDuplicate isn't in 1.0
<poolie> "stable interfaces in python are hard"
<lifeless> perhaps it should be ;) - thus a bug is needed.
<poolie> i'm pretty sure this was working a week or two ago
<poolie> maybe a bit more than this, but within the last couple of months
<poolie> i will file
<poolie> the shorter Optional Reviews report: "data, bitches!"
<poolie> i think that's great
<poolie> to be devil's advocate
<poolie> i think the previous experience is not so much showing everything needs review
<poolie> but rather that people will use these to route around a broken review process
<poolie> cf john's mail
<poolie> but perhaps things have now changed
<poolie> s//probably
<lifeless> well
<lifeless> I think that route around is better than stockpiling
<thumper> wallyworld_: yes, fling them StevenK's way
<thumper> wallyworld_: I'm just about to head and collect the girls from school
<wallyworld_> thumper: ok
<thumper> wallyworld_: we should have a chat after that
<wallyworld_> thumper: i'll be here
<poolie> lifeless: i think in the first version of this code i did assign to 'duplicate_of', but..
<poolie> that didn't work
<poolie> i can't remember if the change was not saved, or if it gave an error
<lifeless> wgrant: yo
<lifeless> poolie: you need to call obj.lp_save()
<poolie> i know that, and i think it wasn't enough
<poolie> imbw
<poolie> anyhow, bug 680339
<_mup_> Bug #680339: 'Entry' object has no attribute 'markAsDuplicate' <Launchpad itself:New> <https://launchpad.net/bugs/680339>
<lifeless> thank you
<wallyworld_> thumper: back in 15 mins - have to drop the kid to his McJob
<thumper> wallyworld_: ok
<poolie> thumper, wallyworld_, we're going to do a bzr 2.2.2 release at the end of this week
<poolie> that should address the problems you hit in 2.2.1
<thumper> poolie: ok
<poolie> feedback welcome, as always
<wgrant> lifeless: Hi.
<wgrant> What's broken?
<lifeless> wgrant: cesium - removing cowboys
<lifeless> wondering if you know the rev that landed the fix
<lifeless> wgrant: how was your exam?
<wgrant> lifeless: Exam was rather better than expected.
<wgrant> cesium... what was the cowboy? The builder disabling thing?
<spm> wgrant: all done? wooo!
<wgrant> Indeed.
<wgrant> lifeless: Unless there was more than just logging and not disabling builders, devel r11938 looks to be the one.
<wallyworld_> thumper: call now?
<lifeless> wgrant: there was
<lifeless> disabling the failure checking
<lifeless> http://pastebin.com/Kr44BkbD
<wgrant> lifeless: 11938 should render that pointless.
<wgrant> Though I'd, er, check with someone else.
<lifeless> wgrant: it looks complete to me
<lifeless> the only gap is that first hunk
<wgrant> Yeah, that's what I meant.
<wgrant> I think.
<lifeless> jamesh: do you remember how to tell zope to use a different thread count?
<jamesh> lifeless: there is an option in the .conf file
<jamesh> I think it might even be called thread_count
<jamesh> this is the ZConfig .conf file
<lifeless> is that 'launchpad.conf' ?
<jamesh> yes
<jamesh> lifeless: looking at one of the ancient launchpad trees on my system, one of the launchpad.conf files under configs/ has "threads 16" at the top level
<lifeless> thanks!
<lifeless> also, terrible idea :)
<jamesh> I remembered that we had bumped the count up for the demo.launchpad.net instance
<jamesh> so checked that config
<jamesh> that instance wasn't under heavy load, but it was easy for one user to block everyone else with the default thread count
<lifeless> yeah,
<lifeless> we're in the process of (with measurement) dropping down to 1 thread per appserver
<poolie> lifeless: interesting! i don't recall you posting about it
<poolie> (not that you have to, i suppose)
<lifeless> in various perf tuesday mails
<poolie> on the grounds that, if they're not wasting time, you wouldn't get more than one python thread to run anyhow?
<lifeless> we're seeing things that thread starvation is the best explanation for
<lifeless> and yes
<lifeless> that too
<poolie> ok, i do see a mention in passing
<lifeless> jamesh: yes, thats it
<lifeless> jamesh: thanks for finding it
<lifeless> poolie: we could reasonable expect total time/database time threads to be a reasonable figure
<lifeless> but its simpler to let the OS manage things at that point
<lifeless> + if we do decide to debug something only one user will be impacted
<lifeless> poolie: I did some more analysis in a ubuntu-one thread
<poolie> agree about letting the OS do it
<poolie> istm the main drawback is there are things that python knows are shareable in memory, that the OS doesn't
<poolie> like modules
<poolie> and for lp they tend to be large
<poolie> but maybe this is not a sufficiently important factor
<lifeless> its a couple hundred MB
<lifeless> 1.6GB on a fully loaded machine.
<lifeless> it could be better
<lifeless> but we should get a win regardless, or so says the theory.
<lifeless> yay
<lifeless> https://launchpad.net/~lifeless/+commentedbugs working
<poolie> timed out for me..
<lifeless> hahrugle
<poolie> in 'select count from bugtask....'
<poolie> but it's a lovely sounding url :)
<lifeless> what revno
<poolie> 11952
<lifeless> damn
<lifeless> At least 42 queries/external actions issued in 2.16 seconds
<lifeless> r11952
<lifeless> poolie: its working consistently for me
<lifeless> poolie: whats the OOPS id ?
<poolie> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1788K350
<poolie> works for me now
<poolie> is this a new feature, or just a newly-faster feature?
<poolie> worked this time
<lifeless> top OOPS since the monthly rollout
<poolie> ah i see
<poolie> maybe it was 'want an affecting bugs list' i was thinking of
<spm> wgrant: btw. I'm bemused and entertained at your dedication. Finish final exam, hop on IRC. ISTR that post my final exam at Uni, a group of us went to the Brekky Creek Hotel in Brisvegas for a Steak lunch and afternoon of entertaining the regulars in the beer garden with (badly) Monty Python songs sung.
<wgrant> spm: Well, a few of us finished early, went to pub, consumed beer, returned to the exam venue to see everyone else, then went back to my office.
<wgrant> And then IRC, yes :P
<spm> bwhahaha
<lifeless> spm: hey
<lifeless> sorry, ECHAN
<poolie> spm, ah, the brekky creek
<poolie> is it a known problem that all api calls against staging are giving 500 errors?
<poolie> and indeed the web ui too
<stub> Hayfever or cold :-( Suspect today is another sick day.
<spm> yeah, staging is borked atm. been fixing the borked prod rollout first tho
<poolie> heh, that's probably a good choice
<poolie> :) thanks spm
<adeuring> good morning
<bigjools> morning
<bigjools> how was the exam wgrant?
<lifeless> is there someone around that can qa https://bugs.launchpad.net/rosetta/+bug/669831 ?
<_mup_> Bug #669831: obsolete translations exported to the branch <code-integration> <qa-needstesting> <Launchpad Translations:Fix Committed by danilo> <https://launchpad.net/bugs/669831>
<mrevell> Morning
<bigjools> lifeless: I've landed all code to cover the cowboys on cesium, please roll it out (or I can)
<lifeless> bigjools: its done
<bigjools> lifeless: ah great, you checked or Picarded it?
<lifeless> spm deployed
<lifeless> and noone has screamed yet
<lifeless> I figured I'd chat with you briefly :)
<bigjools> heh
 * bigjools looks at log
<bigjools> lifeless: I thought it was in the nodowntime set?
<lifeless> bigjools: it was taken out when it blew up
<lifeless> when things get cowboyed we remove them from nodowntime.
<bigjools> so it's back in?  I thought I said I was going to approve that first ...
<lifeless> we may be crossing wires
<lifeless> back at uds it wasn't in, and we discussed what it would take to be in
<lifeless> that you were cc'd on a discussion and gave your blessing.
<lifeless> then the big branch blew builders away
<lifeless> so it was cowboy-fixed and removed [to stop deploys uncowboying it]
<lifeless> we cross checked that you'd landed fixes [and they were to be deployed] today, and so uncowboyed it (which implies putting it back in nodowntime]
<lifeless> bigjools: if you had wanted a further cross-check with you, I'm really sorry - I didn't realise.
<bigjools> lifeless: np, I talked with Tom not you
<bigjools> as it happens it's fine to roll, but that's because I had landed everything
<lifeless> right, we looked first ;)
<lifeless> in future, if you want a cross-check, could you note that on LPS against the cowboy, or the DeploymentException ?
<bigjools> yes but one of the cowboys is not landed, but deliberately.  How did you reconcile that? :)
<lifeless> bigjools: knowledge.
<bigjools> :)
<lifeless> I had discussed the fault with you.
<lifeless> so I knew you put it in while you diagnosed the root cause.
<bigjools> we need to figure out wtf the builders are taking >30 seconds to accept connections
<lifeless> I also included that cowboy as a ready to go patch in a mail to losas
<lifeless> so if there is a submarine present there
<lifeless> its at-hand to recowboy
<bigjools> one of the side effects of that cowboy is that jobs will "stick" on genuinely failed builders
<bigjools> I have to keep checking the log
<bigjools> lifeless: so, did we talk about figuring out the massive slave connection delays?
<lifeless> not as such
<henninge> hey jtv! ;)
<jtv> hi henninge!
<henninge> jtv: let's be chatty ;)
<jtv> henninge: I don't think this connection will support voice chat.
<henninge> jtv: oic
<henninge> you are also not on #translations
<jtv> just a mo'
<lifeless> jtv: hi
<jtv> hi
<lifeless> https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<jtv> uh-oh
<lifeless> are you able to qa  669831 on [qa]staging ?
<jtv> henninge: hang on, lifeless wants something :)
<lifeless> no panic
<jtv> lifeless: very slow connection, so will be a bit slow to respond
<lifeless> but 11960 would remove another cowboy in the datacentre
<lifeless> jtv: like I say
<lifeless> no panic
<henninge> jtv, lifeless: I can do that
<lifeless> awesome
<lifeless> ok, gnight all
<jtv> g'night
<henninge> lifeless: good night
 * jtv runs around a bit and screams for a while, just because he was told not to panic
<jtv> it's the principle of the thing
<bigjools> have you considered working on Soyuz?
<wgrant> bigjools: Better than expected.
<bigjools> wgrant: excellent, but shouldn't you be out celebrating now
<wgrant> Did that earlier :P
<wgrant> But yes, probably.
<bigjools> you party animal
<bigjools> did you get a chance to look at the expiration query?
<wgrant> Heh.
<wgrant> Will look now.
<wgrant> bigjools: Where's the latest version of that query?
<bigjools> the one I pasted :)
<wgrant> k
<bigjools> http://pastebin.ubuntu.com/535167/
<wgrant> Yup.
<wgrant> bigjools: Are the LFC, DS and DAS joins in the EXCEPT completely useless, or am I stupid and blind?
<bigjools> one sec
<wgrant> bigjools: http://pastebin.ubuntu.com/535491/
<wgrant> Should be equivalent, except with the retention condition fixed.
<wgrant> (now it will exclude if the file is unremoved or Obsolete, rather than Published or Obsolete)
<wgrant> Which should make p-d-r less sad.
<wgrant> Hmm.
<wgrant> bigjools: Did the domination thingy help dogfood at all? I guess it pales in comparison with the file list generation :/
<bigjools> wgrant: yes, file lists take ~4 hours
<bigjools> for just maverick release pocket :/
<bigjools> something has regressed a lot
<bigjools> it used to take ~30 mins
<wgrant> I will poke it in the eye in a few weeks.
<bigjools> DF is slow, but ...
<wgrant> Heh.
<bigjools> wgrant: sql looks good, I am trying it on DF
<wgrant> bigjools: So, about those people who feel like they need to use obsolete PPAs...
<bigjools> OEM
<wgrant> How did I guess :(
<bigjools> now, I am struggling to understand wtf builder sometimes take in excess of 180 seconds to accept a connection from the manager
 * bigjools -> caffeine
<wgrant> bigjools: Load graphs from non-virt builders pls.
<wgrant> It might at least give us some idea of if that's the issue.
<bigjools> jml: remember my dirty reactor failure in my b-m tests?
<bigjools> I added debug output on the Deferreds and it's caused by distribution_mirrorprober.  There's a test isolation error... :/
<wgrant> The reactor is sharedâ½
<bigjools> in tests
<wgrant> My interrobang stands.
 * jml briefly steps in from the sick room
<bigjools> man flu?
<jml> wgrant: yeah, there's exactly one reactor.
<jml> wgrant: it's arguably the biggest flaw in Twisted
<wgrant> jml: This sounds... like Zope.
<bigjools> it would be kinda hard to have more than one
<jml> bigjools: I know about those tests. my testtools-experiment branch fixes those delayed calls.
<jml> bigjools: but it's blocked on landing by some weird-ass 500 from the librarian
<bigjools> jml: yay.  So, what can I do about it in my branch?
<jml> bigjools: those tests need to be completely rewritten... let me find you my workaround
<bigjools> and why are they leaking to my test?
<jml> bigjools: because the distributionmirror_prober tests aren't using trial
<bigjools> ah ...
<jml> bigjools: so the calls are going on to the reactor, and when your tests are cleaned up ... bang
<bigjools> that's kinda bad from a test isolation PoV :/
<bigjools> wgrant:
<bigjools>  total_files | space_saved
<bigjools> -------------+--------------
<bigjools>       184899 | 307432516434
<jml> bigjools: yes. the problem is the mutable global state that is the reactor
<bigjools> indeed
<jml> bigjools: got any suggestions on how to make it better?
<wgrant> bigjools: Mm, not too implausible.
<bigjools> jml: I'd make Trial clean the reactor when starting a test?
<jml> bigjools: it can't do that. there might be in-process Twisted-using fixtures
<jml> bigjools: http://pastebin.ubuntu.com/535507/ <- should work around the problem
<bigjools> jml: at the very start of the test there should be no fixtures yet though?
<jml> bigjools: not if they are shared between tests
<bigjools> aieeee
<bigjools> jml: I'll poke your workaround in, thanks
<jml> wgrant: incidentally, it doesn't sound like Zope to me.
<wgrant> jml: Opaque global state.
<wgrant> The Zope Way.
<bigjools> jml: if it's complaining about them when the test ends, how can a fixture's Deferreds never get in the way then?
<bigjools> if it's shared between tests that is
<jml> bigjools: good point. in Trial, there's some historical crap back from when we thought setUpClass would be a good idea
<jml> (it's a terrible idea, we got rid of it, Guido added it back to Python again, and so the circle of crap continues)
<bigjools> :/
<bigjools> so if someone uses setUpClass with a Deferred, your tests will always fail
<jml> bigjools: in testtools, I guess we could clean the reactor before tests
<jml> bigjools: no, Trial postpones checking of those things until tearDownClass runs.
<jml> bigjools: *that's* the historical crap.
<bigjools> oy
<jml> as I said, clearing it out in testtools would help
<bigjools> yeah
<jml> although it wouldn't help that much.
<jml> "some test before this one was bonkers"
<jml> it's still an improvement over the current situation
<bigjools> I won't ask why the mirrorprober tests are not using Trial then.... :)
<jml> I have NFI
<jml> if I have my way, they'll be using testtools before the year is out.
<bigjools> \o/
<bigjools> hmm I need to book some holiday
<jml> anyway, all this excitement is threatening my delicate constitution
<bigjools> jml: Berocca
<jml> bigjools: it's a cold, not a hangover. :P
<bigjools> jml: :)  it still works
<bigjools> the fizzy stuff
<bigjools> get better soon anyway, go get some rest
 * jml watches The Wire instead.
<bigjools> a friend of mine swears his PS3 is medicinal
<jml> heh
<deryck> Morning, all.
<bigjools> wgrant: so, I figured out the problem with the log parser
<wgrant> bigjools: !!
<wgrant> What is it?
<bigjools> wgrant: it reads in gzip files in their entirety :/
<bigjools> see lp/services/apachelogparser/base.py
<bigjools> get_fd_and_file_size()
<bigjools> le heavy sigh
<wgrant> bigjools: Hahaha.
<wgrant> I thought gzip stored the uncompressed size in the header...
<wgrant> Yes, it's in the footer.
<wgrant> Not sure how we can access that from Python, though...
<bigjools> but does any of the python module read that
<wgrant> It's limited to 4 bytes, so it probably doesn't.
<wgrant> But let's see.
<bigjools> http://stackoverflow.com/questions/1704458/get-uncompressed-size-of-a-gz-file-in-python
<wgrant> It looks like we probably have no choice but to read in chunks.
<bigjools> did you notice the last answer on that page :)
<wgrant> Suggesting len(fd.read())?
<wgrant> Oh.
<wgrant> Hahahah.
<wgrant> I rarely look at the author...
<bigjools> I am tempted to grab the last 4 bytes
<wgrant> But 2**32...
<wgrant> I guess we can make it explode if it ever tries to write a bytes_read greater than that.
<wgrant> Since something is pretty broken if we have a 4GiB log file, I guess.
<bigjools> they are in the region of 1.2G uncompressed
<wgrant> Oh.
<wgrant> That is inconvenient...
<wgrant> Also, that's huge.
<bigjools> it's fine
<wgrant> It's far too close for my liking, but OK.
<bigjools> we can put a limit on it
<wgrant> Yes, but a limit within a couple of orders of magnitude of the current value seems like a really bad idea.
 * bigjools tries dirty hack on DF
<bigjools> score
<wgrant> It works?
<bigjools> yes
<wgrant> So I didn't break it after all :D
<bigjools> seems so :)
<bigjools> 2010-11-23 13:03:04 INFO    Parsed 5000000 lines resulting in 16085 download stats.
<wgrant> Hmm.
<bigjools> I lied, one file is 2.9G
<wgrant> Ow.
<wgrant> bigjools: Could you sum all the BPRDC.count?
<wgrant> Just to see that it actually handles most of the lines.
<wgrant> Although I guess lots of those lines will be Packages/Sources, so it might not be too similar.
<bigjools> 6880
<bigjools> it's still going though
<bigjools> it will take a while
<bigjools> and it's hammering DF
<bigjools> good time for lunch, see you later
 * wgrant sleeps.
<wgrant> Thanks for fixing that.
<bigjools> my pleasure
<jelmer> 'morning benji, abentley
<abentley> jelmer: morning.
<benji> morning jelmer, or afternoon as it were ;)
<jitu> how to add https://launchpad.net/~falk-t-j/+archive/lucid/+build/2018840   in repository?
<jitu> anybody to help?
<bigjools> jitu: follow the instructions here https://launchpad.net/~falk-t-j/+archive/lucid
<bigjools> where it says "Adding this PPA to your system"
<jitu> checking...
<jitu> bigjools, thnx
<deryck> mars, hi.  You around?
<deryck> Or maybe gary_poster could help me.  gary_poster, ping?
<gary_poster> hey deryck.  what's up
<deryck> Hi gary_poster.  Does this revno look like the right way that I would disable windmill tests to you?  http://bazaar.launchpad.net/~deryck/launchpad/rockstar-js-refresh/revision/11726
 * gary_poster is skeptical he will know, but is looking
<gary_poster> deryck: yeah, that looks like a very reasonable approach to me, especially if you have evidence it works ;-) .
<deryck> gary_poster, heh.  that's the problem, I don't. ;)  Started an ec2 test run last night that disappeared.  I assume something hung and the test was killed....
<gary_poster> :-P
<deryck> so I started another run, but was looking for some confirmation that the patch looked right. :-)
<gary_poster> deryck: I can assert that I believe that should have worked
<deryck> gary_poster, good enough.  Thanks!
<gary_poster> :-) np
<deryck> :-)
<deryck> FWIW, my current test run seems to be going well.  The tests started up much faster than earlier attempts I had.
<mars> deryck, you may need '!(MailmanLayer|WindmillLayer)' instead, but I don't know for sure.
<mars> empirical evidence needed
<deryck> mars, gary_poster, also, I opened Bug #680497 about missing test coverage.  If this didn't belong on Foundations, sorry.  I wasn't sure.
<_mup_> Bug #680497: jstests for LP JavaScript client are not running automatically <Launchpad Foundations:New> <https://launchpad.net/bugs/680497>
<deryck> mars, ok, I'll watch the run closely and see.  Thanks!
<deryck> if I see a windmill test go, I'll kill it ASAP and try again :-)
<gary_poster> deryck: Foundations: yeah, close enough :-) .  I might add the web group.
<deryck> ok, cool
<gary_poster> deryck, mars: mars makes a good point.  I'm not sure if the layer thing is an and or an or.  ./bin/test --help should say.  looking
<deryck> heh "is an and or an or" is hard to parse on two cups of coffee only
<deryck> the hung run suggests my patch might not have worked.
<gary_poster> deryck, :-) mars is right.  "--layer" is a logical or.  that is, it will run include all layers that are not Windmill ORed with all layers that are not Mailman, resulting in all layers.
<gary_poster> s/run include/include
<deryck> ah
<deryck> gary_poster, so I need the form mars suggested then, right?
<gary_poster> yes
<deryck> ok, cool. an ec2 run killin' I shall go....
<gary_poster> :-) k
<deryck> thanks mars and gary_poster!
<gary_poster> np
 * deryck dreads seeing the ec2 bill this month
<bigjools> jml: are you too ill to help a bit with the buildd-manager timeouts?  My fix hasn't worked.
<abentley> bigjools: Has lamont installed the new buildd (rev 74)?
<bigjools> abentley: you should ask him, not me ;)
<abentley> bigjools: because if he has, that's a bad sign, but if he hasn't, it could help with the timeouts.
<jml> bigjools: I can try. Wassup?
<bigjools> jml: that timeout stuff I added has had no effect :(
<bigjools> here's an example of the sequence:
<jml> bigjools: ok. so maybe we misdiagnosed the problem?
<bigjools> 2010-11-23 14:46:27+0000 [QueryProtocol,client] Resuming hassium (http://hassium..
<bigjools> 2010-11-23 14:46:32+0000 [-] Asking builder on http://hassium.ppa:8221/filecache to ensure it has file chroot-ubuntu-lucid-i386.tar.bz2
<bigjools> 2010-11-23 14:46:54+0000 [Uninitialized] Scanning hassium failed with: TCP connection timed out: 110: Connection timed out
<bigjools> the timeout is 22 seconds later
<bigjools> not even the default 30
<bigjools> so I reckon misdiagnosis is quite probable
<jml> bigjools: is your timeout stuff actually on the relevant machines?
<bigjools> jml: yes, I had Tom do a paranoia grep
<bigjools> I'm at a bit of a loss here
<jml> poking at the code now
<bigjools> we probably need some more debugging logging
<jml> yes. and the ability to switch it on and off in runtime
<jml> bigjools: do you have a traceback with that error?
<bigjools> jml: no
<jml> bigjools: isn't that unexpected?
<bigjools> it's one of the "known" errors so it doesn't run the traceback.
<jml>             BuildSlaveFailure, CannotBuild, BuildBehaviorMismatch,
<jml>             CannotResumeHost, BuildDaemonError, CannotFetchFile):
<jml> which one?
<bigjools> it'll be getting re-raised somewhere
<jml> bigjools: we should also remove some of the unnecessary layers of stack toot-sweet. We have to trawl through this code frequently enough that it's a noticeable cost.
<bigjools> wtf is a toot-sweet?
<jml> sorry, an expression from childhood. "very quickly"
<bigjools> "tout de suite"
<bigjools> :)
<bigjools> but yeah, agree
<jml> bigjools: I learnt it from folk with Middlesex accents.
<jml> anyhow
<bigjools> hah
<jml> bigjools: my reading of the code shows that it's failing in startBuild (builder.py) after the call to resumeSlaveHost succeeds
<jml> because resumeSlaveHost prepends crap to the error message, and we don't see that in the logs
<jml> likewise, the error can't be coming from any of the eb_foo in startBuild, because they also mutate the error message
<bigjools> correct
<jml> ergo, it comes from resume_done
<jml> or something after startBuild
<bigjools> it will be the first call that tries to dispatch the chroot
<bigjools> bearing in mind they are >600M I wonder if that is poking a subtle bug
<jml> there doesn't seem to be anything after startBuild that does anything particularly interesting
<jml> bigjools: do we know what type of job this is?
<bigjools> binarypackage
<jml> hmm
<bigjools> something is catching that error and re-raising it
<jml> that's, well, odd.
<bigjools> as a known exception - otherwise we'd see a traceback
<bigjools> actually - I wonder if it contains any trace info
<jml> because, afaict, it's being raised by the second line in dispatchBuildToSlave in binarypackagebuildbehavior
<jml> d = self._builder.slave.cacheFile(logger, chroot)
<jml> which has absolutely no errback added to it until _scanFailed in manager.py
<jml> (again, my reading only, not actually tested)
<jml> actually, I'm going to say some stuff with triple-x markers so the conversation can be more readily grepped for actions
<jml> XXX: change _scanFailed to have a different prefix to the log message for unexpected errors
<jml> XXX: collapse some of the unnecessary indirection in builder.py (e.g. updateStatus, updateBuilderStatus; updateBuild; _dispatchBuildCandidate)
<jml> bigjools: the other big question is why 22s
<jml> bigjools: what are the values for timeouts in production?
<bigjools> 180s
<bigjools> I simply cannot fathom where 22 comes from
<jml> perhaps we're using an unexpected config setting?
<jml> perhaps it's 20 + noise?
<bigjools> the delay on the failures in the log is not consistent either
<jml> bigjools: what's the smallest?
<bigjools> that's the smallest I've seen, but it's hard to grep
<jml> since we've got blocking code in prod, it's to be expected that there'll be variation
<bigjools> yep
<bigjools> jml: those eb_ functions are not doing much any more
<jml> bigjools: I can believe it
<jml> hmm.
<bigjools> they are used when doing the resume op :)
<jml> bigjools: what's in the logs roughly 3 minutes before that error message?
<bigjools> not a lot
<jml> bigjools: do we get the same three lines every time the error happens?
<bigjools> sorta
<bigjools> they're obviously spread around the file
<bigjools> but it's always the chroot dispatch as far as I've seen
<jml> hmm.
<jml> grepping twisted code reveals it's definitely a TCPTimedOutError
<bigjools> I wonder if the slave is disconnecting before it's replied?
 * jml looks up what ETIMEDOUT means
<bigjools> jml: heh, you know what, the timeout on the slave will still be 30 seconds, right?
<jml> bigjools: why so?
<bigjools> it's also twisted
<jml> hmm.
<jml> I don't think listening works in the same way
<bigjools> it depends on what stupidity it has
<jml> how is the buildd launched in production?
<bigjools> it's part of init.d
<jml> Timeout  while attempting connection.  The server may be too busy to accept new connections.  Note that for IP sockets the timeout may be very long when syncookies are enabled on the server.
<jml> hah
<bigjools> is that a victorious hah?
<jml> maybe.
<bigjools> show me the beans
<jml> so, if I understand correctly, the timeout passed to connectTCP does not in fact control the UNIX-level socket timeout
<bigjools> it's a callLater to cancel the Deferred I think
<jml> exactly
<jml> but the TimeoutError that generates is different to TCPTimedOutError
<bigjools> yep
<jml> which is a mapping for ETIMEDOUT
<bigjools> interesting
<jml> IIUC, the call to socket.connect_ex() in t/internet/tcp.py is timing out
<jml> unfortunately, I don't know whether the timeout is being set client-side or server-side, somewhere in Twisted, somewhere in Python or somewhere deeper
<bigjools> maybe the stack trace will tell us when I get that cowboy in
<bigjools> but I suspect not :/
<jml> bigjools: well, the timeout is being set in a different call
 * jml flicks through APUE
<jml> nope
 * bigjools otp for a bit
<lifeless> morning
<lifeless> bigjools: how did cesium go ?
<lifeless> henninge: hi; how did you go on  669831?
<lifeless> bigjools: ah, found your mail.
<bigjools> lifeless: yeah, badly
<lifeless> I've suggested a feature flag to you, to let us get rid of the cowboy but keep the code path easily off/on as needed.
<lifeless> (in mail)
<bigjools> lifeless: +1 to the flag.  But only if I don't work out the problem this week.
<lifeless> sure
<bigjools> maybe I'll do it along with an attempted fix actually
<bigjools> lifeless: the thing that we discovered earlier is that the timeouts I changed are not firing, something else is.
<bigjools> we see a TCPTimedOutError
<bigjools> if it were the timeout value firing it would be a Deferred cancelled error
<bigjools> the question is, where the heck is generating that
<lifeless> do you generate an OOPS ?
<lifeless> if so, its backtrace might help
<bigjools> lifeless: there's no trace on the exception :(
<bigjools> it should get logged, but there's nada
<lifeless> grah
<bigjools> well - there IS a traceback, it's just one line
<lifeless> is it an OOPS ?
<lifeless> or something else
<bigjools> no, I don't generate an oops
<lifeless> whats logging the error for you ?
<bigjools> because they are routine failures
<bigjools> my code!
<bigjools> see _scanFailed() in lib/lp/buildmaster/manager.py
<lifeless> so there's a more sophisticated error checker you can create
<lifeless> have you seen Release It!  - the book ?
<bigjools> it uses failure.getTraceback()
<bigjools> I have not
<lifeless> ok, and failure.getTraceback() is neutered for some reason ?
<bigjools> 2010-11-23 16:39:28+0000 [Uninitialized] Traceback (most recent call last):
<bigjools> 2010-11-23 16:39:28+0000 [Uninitialized] Failure: twisted.internet.error.TCPTime
<bigjools> dOutError: TCP connection timed out: 110: Connection timed out.
<bigjools> that's it
<lifeless> how frustrating
<bigjools> somewhat
<bigjools> it's thrown when we get a errno.ETIMEDOUT
<lifeless> righto, the pattern is Circuit Breaker
<lifeless> its kindof the ultimate variant of what you've implemented
<lifeless> its not relevant right now
<lifeless> but I'm going to explain why I was thinking of it
<lifeless> it has the idea of the thing it measures being ok, being in trouble, and being dead.
<lifeless> *if* we got the full traceback and were only logging one line
<lifeless> I was going to suggest logging the full line on the transition from in-trouble->dead
<lifeless> s/full line/full thing/
<lifeless> however, thats not the issue we're facing, so it was just a short side discussion.
<bigjools> that's what it's trying to do, effectively
<lifeless> yah
<lifeless> so
<lifeless> we're getting a short tb
<lifeless> IIRC Failure does that when the error is thrown locally and the frame above is the reactor
<bigjools> there's only 2 places the Twisted code itself can throw that error
<lifeless> so could we be dealing with non twisted code doing a regular socket call and throwing, in a callback from some twisted code.
<bigjools> twisted/internet/tcp.py
<bigjools> doConnect() calls self.failIfNotConnected(error.getConnectError ....
<bigjools> what I don't get is that none of that seems to be asynchronous
<bigjools> lifeless: bear in mind that this code works absolutely flawlessly on dogfood
<bigjools> even with more builders added
<lifeless> I think- I'd need to check some lower level code - but I think that that is:
<lifeless> 'read the last error from the socket'
<lifeless> failIfNotConnected looks like a plausible issue
<lifeless> so getConnectError generates a stackless exception object
<lifeless> and twisted.python.failure when given a regular Exception with no frame data bails
<lifeless>         elif not isinstance(self.value, Failure):
<lifeless>             # we don't do frame introspection since it's expensive,
<lifeless>             # and if we were passed a plain exception with no
<lifeless>             # traceback, it's not useful anyway
<lifeless>             f = stackOffset = None
<lifeless> thats why you're not getting anything useful.
<lifeless> *I think*
<lifeless> jml: ^ plausible?
<lifeless> the failure.Failure construction call is passing in a simple Exception object
<lifeless> that has no traceback and thus doesn't get one
<lifeless> if we passed *no* exception in, sys.exc_inf would have been called, which gets a tb object.
<bigjools> the err is an errno
<bigjools> right
<lifeless> I'm surprised that at the claim that getting a stack from scratch is more expensive than was sys.exc_info does (when the exception is triggered). But perhaps it is.
<bigjools> it might be worth a quick cowboy
<lifeless> (I mean, I know it has overhead and I avoid doing it casually myself)
<lifeless> but this function does one and not the other which is pretty surprising to me
<bigjools> what I need to work out is why we get ETIMEDOUT so quickly
<bigjools> TCP sockets take minutes to time out (hours?) by default
<lifeless> depends on the state
<lifeless> during connection its 30 seconds (IIRC)
<bigjools> I've never heard of that
<bigjools> actually, hmmn
<persia> 30 seconds of retry for a packet that won't send, but much more for just an open socket without traffic.
<lifeless> also 30 seconds or so for a SYN that isn't acked
<bigjools> right
<lifeless> but blackholes
<lifeless> bigjools: http://www.faqs.org/docs/iptables/tcpconnections.html
<lifeless> Table 4-2. Internal states
<lifeless> bah
<lifeless> thats the firewall side
<bigjools> SYN_SENT	2 minutes
<lifeless> which has timeouts set *above* what tcp needs
<lifeless> anyhow, default syn timeout is 30 seconds
<jml> lifeless: yeah, the stack analysis is correct. see #twisted for some follow-up discussion.
 * jml gone again
<lifeless> jml: I see discussion on the socket, not on the poor stack info
<lifeless> jml: thanks though
<bigjools> lifeless, did yo usee the comment " it's very easy to run out of listen queue in a python server with many short-lived connections"
<lifeless> bigjools: do you make multiple outstanding SYN attempts to a asingle build slave
<lifeless> s/you/we/
<bigjools> lifeless: the code has no knowledge of SYN attempts, it's using xmlrpc.Proxy
<lifeless> ok
<bigjools> but it will try and connect every5 seconds
<lifeless> does it make multiple outstanding xmlrpc calls ?
<bigjools> actually, 15, I  changed it
<bigjools> no
<lifeless> then that comment is irrelevant
<lifeless> if we made multiple requests at once
<lifeless> then we could exceed the default listen size (8)
<bigjools> there must be some other difference between dogfood and production
<bigjools> I have not seen this problem *a single time* on DF
<bigjools> and now I have to go to dinner
<bigjools> I'll catch up with you again later
<sinzui> flacoste, mumble
<sinzui> flacoste, https://bugs.launchpad.net/launchpad-registry/+bug/621778
<_mup_> Bug #621778: Register project from source package should include homepage URL <qa-ok> <Launchpad Registry:Triaged by jelmer> <https://launchpad.net/bugs/621778>
<lifeless> thumper: how was pyconz
<lifeless> finally
<lifeless> we're getting back into shape
<lifeless> Time Out Counts by Page ID
<lifeless> Hard	Soft	Page ID
<lifeless> 77	3783	Archive:+index
<lifeless> 54	162	BugTask:+index
<lifeless> 26	298	Distribution:+bugs
<lifeless> 25	99	ProjectGroupSet:CollectionResource:#project_groups
<lifeless> 17	46	DistroSeries:+queue
<lifeless> 15	10	Person:+commentedbugs
<lifeless> 12	256	POFile:+translate
<lifeless> 9	2	Person:+bugs
<lifeless> 6	4	DistributionSourcePackage:+changelog
<lifeless> 6	1	Person:+related-software
<lifeless> -> hospital for allergy vaccination, back in ~3 hours.
<lifeless> on mobile if needed
<flacoste> thumper: field.setTaggedValue('has_structured_doc', True)
<flacoste> widget.context.queryTaggedValue('has_structured_docstring')
<flacoste> field = Attribute('The <a>link</a>')
<flacoste> field.setTaggedValue('has_structured_doc', True)
<flacoste> field = has_structured_doc(Attribute(...)
<flacoste> field = exported(has_structured_doc(Attribute(...)))
<lifeless> bacj
<poolie> hi lifeless, flacoste
<lifeless> hiya
<lifeless> flacoste: quesetion for you
<lifeless> flacoste: if you're still around
<flacoste> lifeless: will be gone after i hit send on that email
<flacoste> but shoot
<flacoste> hi poolie
<poolie> lifeless: i think jam would like someone (maybe not you) to state that creating bin/bzr is/isn't the most tasteful way to do it
<flacoste> poolie: i say it is
<poolie> jam: ^^
<flacoste> poolie, jam: try talking to gary tomorrow for help
<lifeless> flacoste: rt 41361 - I mailed you
<flacoste> lifeless: yes, i saw that
<poolie> that's what i said too :)
<lifeless> jam: creating bin/py is the most tastesful way to do it.
<lifeless> jam: I just don't know the machinery in that stack yet.
<lifeless> flacoste: did I make sense? I didn't see a reply.
<flacoste> lifeless: i understand it, i'll have to see how it fits resource wise
<flacoste> since it needs some amount of coordination on our side
<lifeless> flacoste: very small :) - I've done the heavy lifting.
<flacoste> well
<lifeless> flacoste: anyhow, shoo.
<flacoste> it requires doing measurements
<flacoste> and saying +1 / -1
<flacoste> that's not very small in my book<
<lifeless> flacoste: that doesn't need coordination
<lifeless> flacoste: that can be done weeks or months later, if needed.
<flacoste> it needs somebody on the lp side to work with the losa
<flacoste> hmm, ok
<flacoste> but
<flacoste> actually
<flacoste> is it that important to do just one
<flacoste> if we don'T assess and then deploy it across the board?
<lifeless> its important to get some headroom
<lifeless> we can't do this on all the servers [yet] - not enough ram
<thumper> flacoste: why google docs???
<thumper> my normal gmail login can't see it
<lifeless> if this works we can probably get headroom without more hardware.
<flacoste> thumper: i'm sending a normal email
 * thumper throws hands up
<thumper> flacoste: thanks
<lifeless> thumper: a logout/login can help.
<lifeless> thumper: also turn on MultiSession
<thumper> I don't know what multi session is
<lifeless> thumper: there's a link in my facebook feed :)
<lifeless> thumper: when I whinged about logins obliterating each other on google apps/gmail
<flacoste> lifeless: but my gut feeling is that completing RFWATD takes precedence
<lifeless> flacoste: thats crucial as well.
<lifeless> :)
<lifeless> they're all critical! :>
<lifeless> flacoste: seriously, gnight.
#launchpad-dev 2010-11-24
<wallyworld_> thumper: mumble?
<lifeless> http://searchengineland.com/google-fans-throw-eggs-at-houses-blurred-in-google-street-view-56763
<wgrant> lifeless: :(
<wgrant> You made us the top timeout again :(
<lifeless> wgrant: there's an easy answer for that
<lifeless> wgrant: Fix my scaling test for binary packages.
<wgrant> I looked at that last week.
<wgrant> But it wasn't obvious what was doing it.
<lifeless> the test ?
<wgrant> I couldn't see what was causing all those queries.
<lifeless> I think the test is flawed, the page doesn't show binaries without sources
<lifeless> wgrant: fix the test, it will be trivial to see after that ;)
<wgrant> lifeless: Which test?
<lifeless> wgrant: the one I landed
<lifeless> wgrant: sorry to be vague
<wgrant> Heh, OK.
<lifeless> look at log for my most recent Archive: landing
<lifeless> late last week I think
<lifeless> it adds two tests
<wgrant> test_source_query_counts and co?
<lifeless> one for source package scaling which works well - I got the test to demonstrate terrible scaling
 * wgrant fixes.
<lifeless> the binary version *doesn't* demonstrate scaling
<lifeless> so I think its misspelt somehow
<lifeless> mwhudson: hi
<lifeless> mwhudson: test running. you were asking..
<mwhudson> lifeless: hello
<mwhudson> ah yeah
<lifeless> I replied.
<lifeless> over there ->
<mwhudson> didn't realize testtools had discovery
<lifeless> all the shiny
<lifeless> AFAIK your options are testtools/subunit .run, trial --reporter=subunit, nose with the subunit plugin.
<mwhudson> hmm
<mwhudson> AssertionError: Unable to use discovery, must use python 2.7 or greater, or install the discover package.
<lifeless> mwhudson: well, have you installed it ? :)
<mwhudson> no, but i have unittest2 installed
<mwhudson> is it packaged?
<lifeless> it may have discover as a private module
<lifeless> http://pypi.python.org/pypi/discover
<thumper> wallyworld_: hey
<wallyworld_> ho
<wallyworld_> thumper:  lib/lp/codehosting/vfs/branchfs.py
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       99 / 5038  Archive:+index
<lifeless>       75 /  219  BugTask:+index
<lifeless>       34 /  361  Distribution:+bugs
<lifeless>       32 /  128  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>       23 /  378  POFile:+translate
<lifeless>       17 /   51  DistroSeries:+queue
<lifeless>       15 /   10  Person:+commentedbugs
<lifeless>       11 /    2  Person:+bugs
<lifeless>        7 /   87  Archive:+packages
<lifeless>        7 /    5  DistributionSourcePackage:+changelog
<lifeless> wgrant: so, did you make any progress?
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/â/âdev.launchpad.net/ï¿½ï¿½ï¿½ | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> poolie: hey
<lifeless> poolie: what was the right spelling for LPNET_SERVICE_ROOT ?
<poolie> hi thar
<poolie> the trick is that they are now in launchpadlib.uris
<poolie> lifeless: ^^
<lifeless> poolie: got a code fragment ?
<poolie> bzr pull -d ~/src/hydrazine âº
<lifeless> poolie: I'm writing a blog post
<lifeless> totally different space
<lifeless> thats a very robert answer to give
<lifeless> is it
<poolie> karma :)
<lifeless> 'from launchpadlib.uris import LPNET_SERVICE_ROOT' ?
<poolie> hang on
<poolie> i wish it was fewer clicks from the project page  to the source :/
<lifeless> poolie: don't you have it local ?
<lifeless> I was assuming you could just copy paste a single line
<poolie> ok
<poolie> from launchpadlib.uris import (LPNET_SERVICE_ROOT, )
<lifeless> thanks
<poolie>         return Launchpad(credentials, service_root,
<poolie>             lplib_cachedir)
<poolie> etc
<poolie> btw is it just me or are the apis getting faster too?
<poolie> also, is there any point filing bugs about api timeouts, or should i assume the oops system will tell you about it
<lifeless> if they are not on https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=timeout
<lifeless> then feel free to file
<lifeless> please include the page id in the title to make it easy to identify dupes
<lifeless> apis will be getting faster as we fix core issues.
<lifeless> https://dev.launchpad.net/LEP/PersistenceLayer may interest you
<mwhudson> lifeless: did you mean to add control characters to the /topic?
<lifeless> mwhudson: It was a copy paste job.
<lifeless> mwhudson: they were presumably there already ;)
* mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/â/âdev.launchpad.net/ | Get the code: https:/â/âdev.launchpad.net/âGetting
<mwhudson> doesn't look like it
<mwhudson> home time, probably back later
<lifeless> bai
<thumper> lifeless: what do we do for a self review again?
<thumper> lifeless: do I just approve myself?
<lifeless> you review it youself
<lifeless> as normal
<lifeless> just like you'd review someone else
<thumper> gah
<thumper> where do I change my login token?
 * thumper goes to hunt through the email
<poolie> qastaging is still down?
 * thumper stabs mail
<lifeless> poolie: its working for me
<lifeless> poolie: https://qastaging.launchpad.net/ comes up ok, if slow.
<poolie> bug 680759
<_mup_> Bug #680759: qastaging api 503 errors <api> <Launchpad Foundations:New> <https://launchpad.net/bugs/680759>
<lifeless> poolie: I think my further updates to that bug, and your mails, crossed paths.
<poolie> yes they did
<poolie> is qastaging now the best place to nondestructively test api clients?
<wgrant> lifeless: Yeah, I got the test to fail.
<wgrant> Will look at fixing the actual bug shortly.
<lifeless> wgrant: \o/
<poolie> lifeless: just to be clear i'm not complaining, just wanted to flag this in case it wasn't known && matters
<lifeless> wgrant: can you show me what I should be doing differently ?
<lifeless> poolie: qastaging is an ok place
<poolie> you just have to be robust against timeouts
<lifeless> but its not resourced enough to be a reliable test environment (neither was staging - the DB massively outweighs the RAM on the db server)
<poolie> fair enough; that's a good practice anyhow
<lifeless> yeah
<wgrant> lifeless: Your guess was correct -- the index doesn't show binaries without sources.
<lifeless> wgrant: whats the diff look like ?
<poolie> for the sake of my education is RequestExpired also a timeout?
<lifeless> poolie: yes
<wgrant> lifeless: http://pastebin.ubuntu.com/535763/
<lifeless> poolie: exact error depends on the exact bit that decided time was over
<wgrant> It would be nice if mBPPH had an arg to do that.
<lifeless> wgrant: yeah
<lifeless> wgrant: +1 on any patch doing that :)
<wgrant> I need to look at the factory and the Soyuz tests and make them match.
<wgrant> Since at the moment each seems to try to pretend that the other didn't exist at the time of writing.
<wgrant> lifeless: Bug #676776 should have nothing to do with the buildd-manager.
<_mup_> Bug #676776: build stuck with "Uploading build" status [UI] <confusing-ui> <recipe> <Soyuz:Incomplete> <https://launchpad.net/bugs/676776>
<lifeless> wgrant: julian said 'with the workaround in place builds get stuck and I have to clean them up'
<lifeless> wgrant: IMBW of course.
<lifeless> but the symptoms sound exactly as he described
<wgrant> lifeless: They'll get stuck in PENDING/BUILDING.
<wgrant> Not UPLOADING.
<lifeless> there's no xmlrpc calls involved during uploading?
<wgrant> No. UPLOADING is set once the master has downloaded the files from the slave and thrown them into the upload queue.
<wgrant> Then it's in p-u's hands, not b-m.
<lifeless> ok
<lifeless> thanks for clarifying
<poolie> "no module gettextpo" indicates out of date sourcecode, i guess?
<lifeless> yeah
<lifeless> utilities/update-sourcecode
<lifeless> + make
<poolie> yup
<poolie> and i guess "ImportError: cannot import name LPModerate" would be similar, but update-sourcecode hasn't fixed it
<lifeless> mm
<lifeless> dunno about that
<wgrant> poolie: That's some mailman breakage. make clean usually fixes it for me.
<poolie> i think it's a test ordering dependency
<poolie> iow these tests won't pass unless something else runs first in the same process :/
<poolie> oh well
<adeuring> good morning
<bigjools> morning
<wgrant> bigjools: Did the log parser eventually finish?
<bigjools> wgrant: yes but it's got a new problem
<wgrant> :(
<wgrant> What is it?
<bigjools> hang on
<bigjools> http://librarian.dogfood.launchpad.net/57530685/1z7LhiYuLtwxWSJXAxHNvbyksXZ.txt
<wgrant> ... pardon?
<bigjools> it's not exactly a useful traceback :/
<wgrant> The real exception wasn't logged at all?
<bigjools> "Unhandled exception" is all you get
<bigjools> and then I stopped working on it as I need to debug the b-m
<jml> lunch already
 * jml is not really succeeding at work today
<jml> hey
<jml> can I persuade anyone to test Launchpad against the latest pre-release of Twisted?
<jml> c'mon, there must be at least ten people with commit access around besides me.
<bigjools> jml: I can help you
<jml> bigjools: thanks. Want to test Launchpad against the latest pre-release of Twisted? http://twistedmatrix.com/users/glyph/10.2.0pre3/
<bigjools> jml: what are you expecting me to test?
<jml> Launchpad. Update the version of Twisted that we use, run the tests, see what breaks.
<bigjools> the test suite is what I needed to hear
<nigelb> Congrats folks on getting rid of edge :)
<bigjools> jml: now, I need to learn how to update eggs :)
<jml> bigjools: doc/buildout.txt (from the root of the checkout) has a guide
<bigjools> yeah reading that already
<jml> bigjools: thanks.
<bigjools> ok that was easy
<jml> yeah. while I'm still not sold on buildout, it definitely makes upgrading dependencies easier than any other equivalent I've tried.
<bigjools> jml: ok tests running, I'll let you know in about 189 minutes
<jml> bigjools: woot. thanks.
<bigjools> it'd be nice to just select trial tests
<jml> + distributionmirror_prober tests!
<bigjools> heh
<jml> and tests that use blocking APIs to exercise Twisted servers, I guess.
<bigjools> maybe it's worth ensuring we do import as TrialTestCase everywhere
<jml> nah
<bigjools> gimme a better idea
<jml> not using trial very soon anyway
<bigjools> oh, your testtools?
<jml> yeah
<bigjools> nyshe
<jml> run_tests_with = AsynchronousDeferredRunTest
<jml> (yeah, it's a horrible name)
<bigjools> ummm
<jml> or @run_test_with(ADRT) for individual tests.
<jml> great
<jml> I have to somehow figure out which revisions lifeless self-reviewed over the last three weeks.
 * jml plays with log & grep, wishing again that Bazaar had a query language
<bigjools> jml: what about using the API?
<jml> bigjools: it only has by-status filtering, so I might as well check bzr.
<bigjools> :(
<bigjools> can you get all his MPs instead?
<jelmer> jml: bzr search?
<jml> jelmer: is that actually useful? I thought it was mostly a text search.
<jelmer> jml: I'm not sure how advanced its query language is.
<jelmer> jml: What are you trying to find?
<jml> jelmer: revisions of stable merged from branches authored by lifeless marked as being reviewed by lifeless
<jml> jelmer: I'm almost done in my search anyway, but it would be nice to know for next time
<jelmer> jml: Ah. No idea how to do that using the command line.
<jelmer> But it should be possible using a few lines of Python code and the API
<jml> jelmer: in general, I wish I didn't have to write so much Python to get stats about trunk landings to PQM-managed branches
<jelmer> jml: Do you mean the fact that it requires Python code at all? I'd think that it would only be a few lines using the graph API.
<jml> jelmer: yeah, pretty much.
<jml> bigjools: any progress on the timeout thing?
<mrevell> sinzui, Hey, around?
<sinzui> I am
<mrevell> sinzui, I was wondering if we could have a call later today. I'm revising the docs for someone who's setting up a new project (so right from registration through to getting each individual app configured) and I wanted to know if you have any views on what I should do.
<mrevell> sinzui, Also, is there a plan for resolving bug 4449?
<_mup_> Bug #4449: Project "title" is redundant with "display name" <tech-debt> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/4449>
<jml> mrevell: there's a spec by mpt that would be a good starting point.
 * mrevell looks
<jml> mrevell: https://dev.launchpad.net/RegistrySimplifications
<sinzui> I would love to talk about this. My last effort to improve th page was a disappointment. Maybe docs and inline help will help users accomplish their task
<mrevell> Yeah, I think a mix is right ... it may also be time to de-mothball the screencasting apparatus, heh.
<sinzui> mrevell, jml: as a matter of fact, I was considering  fixing all the dead attributes during the bug jam.
<mrevell> Thanks jml
<mrevell> sinzui, Ah, cool
<mrevell> sinzui, How does 17.15 UTC sound for a call?
<sinzui> mrevell, I think I am in a team lead meeting at that time
<sinzui> mrevell, 15:00 or 16:00?
<mrevell> sinzui, 15.00 works for me
<sinzui> okay
 * jml hugs addDetail
<jml> anyone seen anything like this before? http://paste.ubuntu.com/535913/ (librarian db constraint violation)
<bigjools> jml: no progress, although I've written 3 lines of code that will lock the slave manager in memory :)
<jml> bigjools: how will that help?
<bigjools> jml: I have a theory that it's getting paged out and thus can't respond quick enough to connections
<bigjools> although I'm starting to doubt that
<bigjools> but it's worth trying nonetheless
<bigjools> jml: I've also analysed the logs a bit to see if there's any pattern in the builders that are failing
<jml> a minimal repro client might be a better starting point -- at least then we can completely eliminate lp.buildmaster.
<bigjools> some are failing a lot more than others
<jml> bigjools: oh. interesting.
<bigjools> https://bugs.edge.launchpad.net/launchpad-buildd/+bug/586359/comments/5
<_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections <Launchpad Auto Build System:Triaged> <https://launchpad.net/bugs/586359>
<jml> this librarian failure is perplexing
<bigjools> jml: I also looked at logs from April and the problem is exactly the same there
<bigjools> jml: when are you getting that librarian error?
<jtv> henninge: can't get onto the other channel right now, but that export-to-branch failure you pasted was simply yet again the same general config problem.  No reason to believe that the script itself hit any real problem otherwise.
<jml> when running lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorUnit.test_finishJob_uploads_nonempty_file_to_librarian
<bigjools> I notice it's in retry_transaction_decorator
<bigjools> which is suspicious
<henninge> jtv: it ran fine mostly.
<jtv> henninge: yes, it was just logging that it was backing off from one branch to avoid a race condition.  But that caused the oopsing machinery to log an oops, and that part was broken as elsewhere.
<henninge> jtv: I filed bug 680908 about what we could do about it.
<_mup_> Bug #680908: Script translations-export-to-branch needs its own config section <Launchpad Translations:New> <https://launchpad.net/bugs/680908>
<jml> it's intermittent
<jml> but only seems to be a thing w/ my testtools branch...
 * jml does some bisecting 
<sinzui> mrevell, mumble?
<mrevell> sinzui, Sure.
<jelmer> rockstar: hi, do you know if there's a reviewer meeting today and who's chairing?
<rockstar> jelmer, no idea.  bac would be the guy to ask.
<jelmer> rockstar: thanks
<henninge> jtv: Hey
<sinzui> mrevell, http://curtis.hovey.name/2010/11/02/encouraging-contribution-on-the-project-page/
<jml> bigjools: as best as I can tell, (curse this intermittency), it's some interaction between the builder tests and the workermonitor tests.
<jml> or, to be precise, that's at least one way of reproducing the error.
<bigjools> jml: suck :/
<bigjools> if you run both?
<jml> if I run both test_builder and test_workermonitor, then it sometimes fails.
<jml> so far haven't got the error from just running test_workermonitor
<jml> got a while loop going running it over & over.
<bigjools> but only sometimes .... urgh
<jml> yeah.
<jml> speaking of only sometimes
 * jml goes downstairs to get some chocolate
<rockstar> sinzui, ping
<sinzui> hi rockstar
<rockstar> sinzui, I wonder if you might be able to help me sort out some bug notification/ownership issues, since the bugs guys are gone and you seem to always know about permissions.
<sinzui> okay
 * jcsackett hates cleaning up a bad merge state...
<rockstar> sinzui, the U1 team isn't getting notified on bugs filed on the u1 client filed through apport.  They're trying to find out why.
<dobey> hrmm, there have been some changes to privacy in lp recently haven't there?
<sinzui> rockstar, I could be via a structural subscription in a indirect team
 * sinzui looks
<dobey> so it looks like crash reports are all private until the rescanner does some magic
<dobey> and this seems to be our issue
<rockstar> dobey, but also that once the rescanner is done, it still doesn't send notifications.
<dobey> sure
<dobey> but i'm ok with not getting more e-mail ;)
 * jml sees more evidence for merging #launchpad & #launchpad-dev
<rockstar> jml, +1
<sinzui> rockstar, ~ubuntuone-hackers is the bug supervisor or ubuntuone-client. They should be getting all bug reports via email
<rockstar> sinzui, https://bugs.launchpad.net/malone/+bug/425127
<_mup_> Bug #425127: private bugs in packages people with access to private bugs are subscribed to don't generate emails <story-better-bug-notification> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/425127>
<sinzui> I was looking at the project, I will look at the package
<sinzui> If my reciprocal bug links were in production I would have seen that
<jml> hah
<sinzui> rockstar, Ubuntu One hackers has a structural subscription. A team admin can remove it
<sinzui> https://launchpad.net/ubuntu/+source/ubuntuone-client
 * jml blags
<jml> :(
<jml> If I run any subset of the buildmaster.tests.test_builder tests, I don't see the error. :(
<gary_poster> jml: May I add the Risks section (and the sub-section of experiment, goal/design/result) from https://dev.launchpad.net/Foundations/Proposals/Template to the LEP template so I can delete it ?
<jml> gary_poster: sure.
<gary_poster> thanks
<jml> gary_poster: I think I would like the LEP template to imply that it's good to think of more than one implementation
<jml> gary_poster: but I think those are definitely good things to have when thinking about an implementation.
<mars> sinzui, ping, do you have a moment to help with a CHR documentation question?
 * bigjools shakes fist at Flash plugin
<lifeless> jml: I put the things I did into the wiki page
<sinzui> mars, I am going into a meeting. I can type on irc
<lifeless> jml: but you can also query for merged mps, and then check the reviewers vs the mp branch owner
<mars> sinzui, just to clarify, so I don't do the wrong thing for reassigning Launchpad Answers (again) - "Retarget" means "Change the assigned project", correct?
<jml> lifeless: 'bzr log --line -r date:2010-10-31.. | grep r=lifeless' worked well enough
<sinzui> mars, yes, retarget does mean change the assigned project or distribution
<lifeless> jml: sure
<lifeless> thank you
<mars> sinzui, perfect, thank you
<jml> np
<gary_poster> jml, I wasn't going to include the word "Implementation" in the copy to the LEP because it is confusing.  That said, I think of "Workflows" as a bit of an implementation proposal.  If the goal is to do blah blah for the webservice, how we accomplish blah blah is described in a workflow.  That workflow is a possible approach to the goals.  The workflow is a design that has risks.
<gary_poster> So that's kind of where I think it goes
<jml> gary_poster: *nod*
<gary_poster> cool
<jml> gary_poster: I don't like having workflows in the lep, actually :)
<gary_poster> jml: gotcha
<jml> gary_poster: I only added it because I wanted to get beuno to stop shouting
<gary_poster> but then all the LEP would be about is agreeing on goals?
<jml> gary_poster: yep
<gary_poster> hm
<jml> gary_poster: goals, constraints, requirements
<jml> analysis, rather than design
<gary_poster> I see
<gary_poster> I'd be happy to have that discussion separately, jml
<jml> gary_poster: but if it gets used for design because managing two docs is too much work, I'm ok with that
<gary_poster> jml, ok, we'll try separate docs then
<beuno> oh hai jml!
<jml> beuno: hi
<gary_poster> lifeless: would you be available for a quick call nowish?
<lifeless> sure
<gary_poster> thanks
<mrevell> I'm off for the day. Night
<bigjools> night all
<jml> lifeless: the testing-cabal PPA now has testtools and testrepository building in it
<jml> lifeless: I've done enough learning for now, so I'm going to open it up for the experts to fix up my mess :)
<lifeless> jml: cool
<jcsackett> back
<jml> you know it's getting serious when you have to draw a truth table to help you debug something
<lifeless> TTF
<jml> thirty-two combinations of fail
<lifeless> TTTTF?
<jml> yeah
<jml> lifeless: in case you have any insights: http://paste.ubuntu.com/535913/ is the error I'm seeing. It's during lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorUnit.test_finishJob_uploads_nonempty_file_to_librarian but the error doesn't appear to happen when the that module's tests are run in isolation
<jml> lifeless: I can reproduce the error semi-reliably running the buildmaster.tests.test_builder tests before, but I've yet to get close to a minimal set
<jml> (debugging this one by the numbers, since my brain is too shot for creative debugging)
<jml> fuller error: http://paste.ubuntu.com/536005/
<lifeless> jml: brief call?
<jml> lifeless: very brief, sure.
<jml> lifeless: it *looks* like there was another test in test_workermonitor that was dirtying the db via the librarian without declaring as such...
<jml> lifeless: I have no idea why I could only reproduce the error with test_builder in the mix
<lifeless> weird
<lifeless> jml: I'm glad you found it
<jml> it would be relatively straightforward to put a thing in the librarian layer that checked to see if rows were added ...
<jml> lifeless: I'm not going to dig any deeper, fwiw.
<lifeless> jml: fair enough
 * jml ec2 tests & goes away to eat, drink (medicine) and be lazy
<jml> lifeless: thanks for the help.
<lifeless> anytime
<thumper> morning
<cr3> can someone explain why some methods in launchpad take a quantity argument instead of using slices?
<wgrant> cr3: Do you have an example?
<wgrant> It's possibly webservice-related, but it's hard to say.
<cr3> wgrant: grep -r quantity= returns examples such as: IBranchView.latest_revisions, IProductSet.latest
<cr3> wgrant: there are some comments relating to the webservice interface, which decorates the functions using call_with(quantity=None), whereas the default is a hard coded value like 10 for example.
<cr3> wgrant: I would've expected the other way around though, ie limiting the webservice interface to some hard coded value and allowing internal calls to slice and dice
<poolie> hi all
<lifeless> cr3: slices require lazy objects, these are method calls
<lifeless> as soon as you're not in ResultSet territory, you'll find we don't slice
<lifeless> and where we do, its generally a bug.
<cr3> lifeless: some calls like Branch.latest_revisions return a Storm ResultSet which could be sliced and diced though, haven't looked at many others, but it might just be remnants from the good ol' days
<wgrant> lifeless: But the methods should be returning (Decorated)ResultSets :/
<lifeless> wgrant: not as of yesterday.
<wgrant> lifeless: How far through the design are you?
<cr3> wgrant: do you guys normally use the launchpad DecoratedResultSet rather than the default Storm one?
<wgrant> cr3: Is there a Storm DRS?
<lifeless> cr3: when we eager load, yes.
<cr3> wgrant: no, the default ResultSet I meant
<lifeless> cr3: we eager load in many, but not enough places.
<wgrant> cr3: We use the standard ResultSet in most places.
<cr3> lifeless: wait, you use DecoratedResultSet or quantity when eager loading? (I need to finish reading that related thread on canonical-tech)
<wgrant> DRS is for eager loading.
<lifeless> DRS for eager loading. quantity when specifying up front what we need (which is a different, not necessarily better or worse) pattern.
<wgrant> quantity is for broken methods that don't return a ResultSet or derivative.
<lifeless> s/broken/other/
<wgrant> Until yesterday they were broken, now they might not be :P
<lifeless> wgrant: they weren't broken before
<lifeless> they are symptoms of our lack
<lifeless> resultsets do not cache
<cr3> wgrant: was there a fix in DRS yesterday? is it in trunk so that I can pull it and see?
<lifeless> so they aren't suitable to pass into methods
<wgrant> cr3: No, lifeless is just changing everything.
<wgrant> EVERYTHING>
<cr3> wgrant: just when I thought I understood a thing or two about launchpad :)
<lifeless> we have several overlapping problems
<lifeless> firstly, our can-read assertions are scattered all over
<lifeless> we should have them in one layer so that we don't *return from queries* data the user [whose behalf we are acting on] shouldn't see.
<lifeless> secondly, templates and logic like canonical_url is per-object
<lifeless> but for efficiency and to map well to our data storage layer we need to work per-set
<wgrant> lifeless: is the API going to be similar to the unannounced new webservice model?
<lifeless> we should have a layer that enforces set based access to the data store and makes it easy to put set/group based idioms higher in the codebase
<lifeless> wgrant: they have strong parallels and I know that the webservice folk have added to their thoughts the ideas I've put up about the group based idioms in LP's internal code.
<lifeless> wgrant: Gary expressed a desire to have them be harmonious and perhaps even trivially layerable.
<lifeless> wgrant: I think they have different constraints, but they should be similar in style even if not in exact detail - initially.
<lifeless> wgrant: the webservice stuff is on dev.l.n if you're interested
<gary_poster> wgrant, also fwiw, the new webservice model won't be announced but proposed
<wgrant> lifeless: Yeah, I've been reading it.
<wgrant> It looks really great.
<lifeless> thirdly, the failure mode we have today is death-by-queries
<lifeless> we should have a mechanism such that when we think we've prepped for a page, we turn off data access
<wgrant> gary_poster: You're silently working out how to make the webservice fantastic. I think that deserves announcement.
<lifeless> cr3: ^
<gary_poster> :-) thanks wgrant.  (where "you" includes a bunch of people.)  And I hear you.  I'll do it.
<lifeless> cr3: so I've proposed an explicit persistence layer
<cr3> lifeless: death-by-queries reminds me of an idea I had recently about unit testing where it might be desirable to decorate or provide some assert methods to make sure a certain number of queries are being performed, ie to detect when fetching a list of items results in a query per item for example
<lifeless> cr3: it will only expose things the current user can see, will return objects that have no backend-connection so cannot trigger bad behaviour by mistake
<lifeless> cr3: search for HasQueryCount in the lp code base
<cr3> lifeless: awesome! will do
<lifeless> cr3: the persistence layer will also be a clear boundary, which we can switch out for tests that are not testing the data storage story
<wgrant> gary_poster: My main remaining gripe is the lack of transactions, but that's probably impossible to do, and it's mostly mitigated by the collection fixes.
<lifeless> wgrant: batch PATCH gives you near-enough IMO
<wgrant> Exactly.
<gary_poster> wgrant, not impossible--leonardr talks about how to do it in his REST book--but not terribly attractive IMO
<lifeless> wgrant: we're -not- going to expose sql transaction objects on the net.
<gary_poster> yay
<wgrant> lifeless: Batch PATCH and the atomic collection retrieval does almost everything needed, I think.
<lifeless> yeah
<cr3> lifeless: that reminds me, I didn't notice any unit tests for the database functions themselves, ie no database/tests directory for example. I decided to have tests specifically for my database code in my latest project
<lifeless> cr3: lib/canonical/database/ftests/ ?
<wgrant> cr3: How are they different from the model tests, given that the layers are one?
<lifeless> there are some big questions about where 'is allowed to be related' logic should go in my new layer
<lifeless> but I think things like 'when a bug changes notify X Y and Z' very clearly do not belong in it.
<cr3> lifeless: well, I would expect tests to exercise each of the functions defined in trusted.sql for example, which it doesn't look like lib/canonical/database/ftests/ is for
<lifeless> equally 'who will be notified' isn't a storage problem, its built on it.
<lifeless> cr3: oh. mmm, thats kindof bootstrap code.
<lifeless> sure, you could test it. It might be nice to do so.
<cr3> wgrant: not necessarily true, some database integrity should be caught at the database layer rather than the application layer, so exercising the model layer does not validate database integrity per se
<cr3> wgrant: I also find it easier when defining a function in sql to have tests specifically exercise that function rather than going through the model layer
<lifeless> cr3: this asumes you have different layers.
<lifeless> cr3: triggers are evil though and we should get rid of ours. They are workarounds for poor ORM support.
<lifeless> perhaps. We can keep them now we have them, but we've triggered storm bugs by using them, and will in the future)
<cr3> lifeless: do you mean problems refreshing the cache in storm when triggers update objects without the orm knowing?
<lifeless> I mean bugs.
<lifeless> like updating all columns rather than changed objects
<lifeless> changed columns that is.
<cr3> lifeless: oh, has that been fixed yet? I've experience that problem myself and it's been a pain :(
<lifeless> in 0.17
<lifeless> it broke other stuff, which is fixed in 0.18
<cr3> lifeless: thanks for the info, will have a look
<lifeless> you may find https://dev.launchpad.net/LEP/PersistenceLayer my high level problem statement interesting
<cr3> lifeless: you're ambitious: # None of our tests of Launchpad functionality require a real database to execute.
<lifeless> we can't tell if its complete if we don't do that
<cr3> lifeless: I would've gone for some ratio or some rough best effort figure, I would be afraid of the 80-20 rule where you'll spend 80 percent of your time finishing off the 20 percent remaining tests hitting the database
<cr3> lifeless: whereas I'd consider it a tremendous success if 80 percent of tests didn't hit the database, testing would be blazingly fast (modulo windmill and doctests)
<lifeless> cr3: if we're going to do it, we need to -do it-
<lifeless> cr3: the last thing we need is a half (or 80%) deployed thing.
<lifeless> that would leave the remaining 20% still existing : having 20% of our pages be terribly slow is not acceptable.
<cr3> lifeless: another way of looking at it is 20% of each page being slow, which still means 5 times quicker than before. but I can appreciate your point of view though, and I suspect you already have a design in mind that actually makes it feasible. in lifeless we trust :)
<wgrant> What do Landscape and U1 do?
<wgrant> It seems that we are trying to solve all sorts of problems that they probably solved years ago.
<lifeless> landscape makes heavy use of 'collections' idioms.
<lifeless> In my discussions with jkakar he has expressed interest in what I've sketched out, and said they suffer the same things we do.
<lifeless> so, for them, they work harder at performance. Its not intrinsically easier for them.
<lifeless> U1, AIUI, are built on django
<wgrant> And U1 uses Couch and all sorts of other evil, I suppose.
<lifeless> which has a less strictly-object flavour and so is less driven to the pathology Launchpad is
<lifeless> but they've had plenty of requests taking thousands of queries.
<lifeless> I'm not aware of any Canonical web property that /has/ this solved pervasively, nor solved at the scale (of data model complexity) that Launchpad needs.
<lifeless> ... I've been asking :)
<wgrant> Has anyone else?
<lifeless> hibernates HQL, sqlalchemy's QL - these are great things to look at.
<wgrant> Right.
<lifeless> functional approaches, and things like map-reduce are very useful inspiration too
<poolie> how/when is the sampledata updated?
<lifeless> fridayish
<poolie> iow why has my merge from devel got large sampledata changes compared to devel?
<lifeless> oh
<poolie> in https://code.launchpad.net/~mbp/launchpad/dkim/+merge/41689
<lifeless> when commits are made
<lifeless> did you do 'make sampledata'
<mwhudson> large diffs tend to come from (sometimes tiny) version differences in postgres
#launchpad-dev 2010-11-25
<poolie> i don't think i did make sampledata
<poolie> let's see what bzr says
<lifeless> poolie: did you merge db-devel perhaps?
<poolie> i did, on the 19th
<lifeless> that will explain it.
<poolie> perhaps i should rebase my changes onto devel
<lifeless> yes please
<lifeless> gary_poster: /expander
<lifeless> gary_poster: should that perhaps be /+expander ?
<gary_poster> lifeless: yes, almost certainly
<gary_poster> Please feel free to change on LEP (or not)
 * gary_poster needs to run
<lifeless> ciao
<gary_poster> bye
<thumper> lifeless: gee, not even a whoop for lp:launchpad changing
<thumper> I feel deflated :P
<lifeless> thumper: whoop
<lifeless> thumper: with leave on monday, I'm feeling a little flat out today
<thumper> :-(
<thumper> http 500 on https://launchpad.net/ubuntu/+bugs?field.tag=needs-packaging
<thumper> second time lucky
 * thumper afk
<StevenK> wgrant: Do you happen to be around?
<wgrant> StevenK: Hi.
<StevenK> wgrant: Hai
<StevenK> wgrant: Can you have a squiz at https://code.edge.launchpad.net/~stevenk/launchpad/db-add-distro-parent/+merge/41430?
<wgrant> 48	+Derived Distroseries
<wgrant> 49	+--------------------
<wgrant> 50	+
<wgrant> 51	+Distroseries can be derived from other distroseries. This is completly
<wgrant> 52	+different from the derivation mentioned above.
<StevenK> Yes, I need to fix that bit
<wgrant> StevenK: What sort of review do you desire?
<StevenK> wgrant: Basically, "Do you think it's crack?"
<wgrant> StevenK: Has it been fully specced out?
<wgrant> If not, it is crack.
<wgrant> I haven't seen an analysis of all of the cases.
<StevenK> I think it's a good plan, because .parent is overloaded and this derivation work is making it be more so
<wgrant> Right, I agree.
<wgrant> But I'd like to see it fully thought through first...
<wgrant> We need to do something like this.
<StevenK> We do have a LEP :-)
<StevenK> wgrant: doctest twiddled, fwiw
<lifeless> jkakar: hi?
<jkakar> lifeless: Hi!
<lifeless> https://dev.launchpad.net/LEP/PersistenceLayer may interest you
<jkakar> lifeless: Cool, thanks, will check it out!
<jkakar> "How will we measure how well we have done? It Will Be Obvious."  <-- Cute.
<adeuring> good morning
<mrevell> Hello
<jtv> hi mrevell!
<mrevell> Hi there jtv, how are you?
<jtv> fine, thanksâyou?
<bigjools> hello
<jml> hi
<jml> I ran a bunch of tests on ec2 last night
<jml> they all passed
<jml> but the test run had a non-zero exit code
<bigjools> jml: sorry forgot to tell you about the test run yesterday, everything seemed ok.  At least I got the usual amount of local failures I get.
<bigjools> but whether they are different failures on the same tests I don't know.  I get 60 codehosting tests failing
<jml> bigjools: thanks.
<bigjools> I need to figure out my local failures at some point ...
<bigjools> jml: loads of my codehosting tests fail because they are trying to open /usr/lib/python2.6/dist-packages/bzr which does not exist, any ideas?
<bigjools> this is when it runs some bzr commands
<jml> hmm
<jml> yeah, I have an idea
<jml> but it's only half-formed
 * jml pokes a bit
<bigjools> it's doing a "bzr push -d first branch-address"
<jml> bigjools: lp.codehosting.get_bzr_path is either buggy or your environment is buggy
<jml> bigjools: try running it in ./bin/ipy to see what you get
<jml> bigjools: also, try 'ls eggs/bzr-2.2.0-py2.6*/EGG-INFO/scripts'
<bigjools> "bzr"
<bigjools> when needs running in ipy, exactly?
<bigjools> what, even
<jml> bigjools: lp.codehosting.get_bzr_path()
<bigjools> jml: it returns the non-existent path
<jml> bigjools: yay, the bug is isolated
<bigjools> :)
<jml> bigjools: I guess step through that with pdb and see what's misbehaving
<jml> bigjools: basically, it looks like you are getting system bzr rather than the bzr in eggs
<bigjools> I am going to remove the egg and re-generate it, it worked with a similar bunch of test failures
<jml> bigjools: a fine plan
<bigjools> I am tempted to blow all the eggs away, it'll clean up all the old crap
<jml> bigjools: that could also work
<bigjools> we need a development story for that
<bigjools> yay, it works now
<bigjools> thanks jml
<jml> bigjools: np
<jml> bigjools: can I persuade you to run the lp.codehosting tests with that new version of Twisted?
<bigjools> jml: am doing so as we type :)
<jml> bigjools: thanks :)
<bigjools> what's the magic command to make my download-cache get, err, downloaded again?
<jml> bigjools: as in, to get a fresh one?
<bigjools> yes
<jml> bzr co --lightweight bzr+ssh://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk/ download-cache
<bigjools> of course, it's a checkout
<jml> rather than say, a Debian archive, yes :)
<bigjools> well, we've got so many different ways of getting dependencies now, it's hard to remember what does what
<jml> definitely
<jml> I was just thinking that it would kind of make sense to distribute the eggs as a debian archive
<maxb> !
<bigjools> download-cache is a bit of a misnomer isn't it ...
<jml> bigjools: I believe the default buildout behaviour is to fetch code from pypi and run that
<jml> maxb: well, storing immutable tarballs in a bzr branch isn't exactly sensible either.
<jelmer> jml: It doesn't seem to work like that, at least not here.
<jml> jelmer: what doesn't work like what?
<jelmer> Sorry, that was a bit vague. I meant buildout doesn't seem to fetch code from pypi for me. I have to have it in the download cache.
<jml> that's because we have 'install-from-cache = true' in our buildout.cfg
<jelmer> ah, ok.
<jelmer> Means it's no longer effictively a cache though :-)
<jml> quite so.
<jml> it's a sensible name in the broader context of what buildout does, but not for our use.
<jelmer> jml: I think shipping the eggs in a deb is a good idea. it's far from ideal but better then what we have at the moment.
<jelmer> It also means it will be easier to e.g. move some of the eggs into their own proper package.
<jelmer> jml: where would you have the package install them though? In a Launchpad-specific directory that lp adds to its PYTHONPATH, or to the system python path?
<bigjools> that lp-source-dependencies is also carring loads of old versions, which is kinda annoying to download
<bigjools> carrying, even
<jelmer> it doesn't make sense that we create a heavyweight rather than a lightweight checkout
<jml> jelmer: mine's lightweight :)
<bigjools> so's mine
<jml> ahh right, I'm just reminded of what actual advantage eggs have over packages: rollback
<bigjools> you can do that easy with packages
<bigjools> and it's audited properly
<jml> jelmer: hey, did you get my email about testing packages etc?
<jelmer> jml: yep. Thanks for setting that up
<jml> my pleasure
<jml> as mentioned, the actual working daily builds are probably packaging nightmares
<jml> and most of the things aren't actually working
<jml> jelmer: so I'd appreciate it if you or others who actually know what you're doing could take a look and fixinate
<jelmer> jml: I'll see if I can contribute. Ideally we'd set these up to merge in their Debian packaging branch as well , that way we (and by we I mean lifeless, as he's the Debian maintainer) only have to do the work once.
<jml> jelmer: that'd be great.
<jml> oh
<jml> that reminds me
<jml> jelmer: the testr in that PPA has incremental output display, I think.
<jelmer> jml: I should mention, I still find daily builds a bit hard to work with in practice because it's hard to easily find out what packages are working on which are broken.
<jelmer> jml: Ah, neat.
<jml> jelmer: can you clarify?
 * jml really wants daily builds to be _easy_ to work with in practice
<jml> although tbh, probably the best thing we can do is reduce the end-to-end latency
<jelmer> jml: I have a lot of daily builds, in particular related to bzr. It's very hard to get an idea of which ones are currently broken.
<jml> ahhh
<jml> yeah, I can see why that would be a thing
<jml> jelmer: have you filed a bug about it?
<jelmer> jml: The list of recipes doesn't show which ones have last produced a source packge successfully and it also doesn't show for which resulting source packages binary packages could be built.
<jml> nor indeed which have failing binary build steps
<jelmer> jml: I filed bug 669260 about this earlier.
<_mup_> Bug #669260: last build result in recipe overview page <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/669260>
<jml> jelmer: ta
<jml> bugs folk, is there a tag for https://dev.launchpad.net/LEP/BetterBugSubscriptionsAndNotifications ?
<bigjools> jml: so I see some test errors with the new Twisted
<bigjools> I'll re-run with the old one just to make sure
<jml> bigjools: thanks. could you also paste the errors?
<bigjools> yep, once I confirm
<bigjools> jml: http://pastebin.ubuntu.com/536278/
<bigjools> w/o new Twisted they are passing
 * bigjools -> lunch
<jml> bigjools: thanks
<jml> jelmer: and 'testr failing --list' too
<jml> flacoste: http://people.canonical.com/~jml/Tests-finished-each-minute.png ; http://people.canonical.com/~jml/Tests-finished-each-ten-minutes.png ; http://people.canonical.com/~jml/Tests-finished-each-hour.png
<jml> mrevell: ^
 * mrevell l;ooks
<jml> flacoste: http://paste.ubuntu.com/536318/
<bigjools> jml: so do you remember that what seemed to a pointless xmlrpc "echo" command that the buildd-manager issued at the start of every build?
<jml> bigjools: vaguely
<jml> bigjools: does it have something to do with our mystery connect() timeout?
<bigjools> jml: I think I found why it was there
<jml> allenap: hi
<bigjools> you guessed.....wisely
<jml> bigjools: I need to know more.
<bigjools> the first packet to the Xen slaves is sometimes discarded
<bigjools> I have a test script that demonstrates this
<jml> bigjools: may I see the script?
<jml> allenap: the build just broke with that error we saw a while ago, eh?
<allenap> jml: Yep :)
<bigjools> jml: http://pastebin.ubuntu.com/536317/ which calls out to http://pastebin.ubuntu.com/536317/
<bigjools> err that';s the same
<allenap> jml: Shall we try kicking it off again, and I'll have a go at the fix you suggested in the "ec2 land submission problems" thread?
<jml> allenap: just pulling up the thread... I seem to remember saying something about it
<bigjools> jml: well anyway there's a top-level script that resets the slave and then immediately calls that one
<jml> bigjools: is the xmlrpc server running inside an instance?
<jml> bigjools: a xen instance, that is
<bigjools> jml: the reset script does not exit until it's running
<jml> allenap: I would suggest applying the fix before retrying
<allenap> jml: Okay, I'll have a bash at that now.
<jml> allenap: I mean, it's not that hard, and it might make our branches land ~3.5 hrs earlier :)
<allenap> jml: Oh yes :) I'll email the list too.
<jml> bigjools: sorry, I don't really know what the reset script does or why exiting matters
<jml> bigjools: also, in what way is that paste different from the script I put on the bug? (is it just that it actually works & reports a little better?)
<jml> allenap: thanks!
<bigjools> jml: it restarts the Xen guest and then waits until the slave manager responds with a BuilderStatus.IDLE.  At that point we know for sure it's up and working.  Afterwards we poke it with that script which sometimes fails with the first connection.
<bigjools> jml: it logs the first packet failures
<jml> bigjools: were we only seeing the error against Xen guest slaves?
<bigjools> yep
<jml> nice
<jml> bigjools: so I guess putting the echo (or even a TCP connect & disconnect) back in, with a nice big fat comment of course, would fix the issue
<bigjools> jml: lp:~julian-edwards/launchpad/timeouts-feature-bug-681426
<jml> bigjools: oh, is there a bug for the issue with xen discarding packets?
<bigjools> :)
<jml> surely someone else must have noticed this...
<bigjools> https://bugs.edge.launchpad.net/launchpad-buildd/+bug/586359
<_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections <Launchpad Auto Build System:Triaged> <https://launchpad.net/bugs/586359>
<bigjools> jml: I saw some PDF describing packets dropped from the DMA ring when system load is too high
<jml> bigjools: I meant a bug against xen -- it's not actually a canonical.buildd issue is it?
<jml> zoinks
<bigjools> we're not sure exactly where the bug is
<bigjools> we need to prove that first
<bigjools> but it looks very suspicious
<jml> so 586359 stays open even when the builder has the workaround?
<bigjools> yeah, I need to add a soyuz task
<jml> *nod*
<jml> bigjools: well, you have my sincerest preliminary congratulations :)
<bigjools> jml: team effort, I can't take all the kudos
<bigjools> but thanks :)
<allenap> jml: Could you sanity check my branch? https://code.launchpad.net/~allenap/launchpad/zope-test-in-subprocess-unicode-issue/+merge/41886
<jml> allenap: I have to look up what codecs.getreader does now :)
<jml> allenap: have you tried running a subclass of ZopeTestInSubprocess?
<lifeless> morning
<bigjools> morning lifeless
<allenap> jml: I've run all the tests that rely on it and they ran fine.
<jml> allenap: cool.
<jml> lifeless: good morning.
<allenap> jml: Thanks for that. Shall I just lp-land --testfix it now?
<jml> allenap: deffo
<bigjools> jml: care to look at https://code.launchpad.net/~julian-edwards/launchpad/timeouts-feature-bug-681426/+merge/41881 please?
<bigjools> it's short :)
<jml> bigjools: you should probably link to the bug report in the comment.
<bigjools> ok
<jml> other than that, good to go
<bigjools> cheers
<bigjools> If I finally put this one to bed I am going to be a very happy man
<lifeless> jml: could you eyeball LEP?PersistenceLayer? Thanks.
<jml> lifeless: fo shizzle
<jml> allenap: btw, do you know of anyone tackling the other failure? the list was silent.
<jml> I've got a bunch of specs to read.  Heading afk to do so & eat. Back later to catch the antipods.
<allenap> jml: Oh, the db-devel one? No, no idea. I didn't even look at it myself :-/
<bigjools> lifeless: FYI I think I figured out the timeouts
<bigjools> so I'm going to skip that feature-controlled builder fail bug
<lifeless> bigjools: It might be a good idea to have a realtime knob for number of failures regardless.
<lifeless> bigjools: but its up to you :)
<bigjools> lifeless: I don't think that's at all necessary
<bigjools> IMNSHO :)
<allenap> jml: If you mean danilo's, then I also don't know.
<lifeless> bigjools: the next time you land a branch changing or disabling that figure.... think of me :)
<bigjools> well if this branch doesn't work then I'll do it
<lifeless> 6 months, 9, I'm sure you'll remember ;)
<bigjools> lifeless: how could I not always think of you
<lifeless> awww :)
<bigjools> so, time to go.  heavy eyes from staying up late to watch the cricket
<bigjools> night al
<bigjools> l
<lifeless> gnight
<lifeless> allenap: ping
<lifeless> flacoste: hi
<salgado> lifeless, do we have only the 'devel' and '1.0' versions of the webservice or are there others?  do you know where's that configured?
<lifeless> salgado: we have beta, 1.0, devel
<lifeless> in the
<lifeless> exported() clal
<lifeless> call
<salgado> lifeless, I was asking about the config which has the list of versions
<lifeless> beta was the first devel, then when we did 1.0 we deleted stuff we didn't like in beta, and devel is like the next beta.
<lifeless> salgado: oh, the master list?
<lifeless> uhm
<salgado> lib/canonical/launchpad/rest/configuration.py
<salgado> I think
<lifeless> looks like
<salgado> yeah, that's itactive_versions = ["beta", "1.0", "devel"]
<salgado> lifeless, so, what is beta?  should the blueprints API be available there?
<lifeless> no
<lifeless> see version_descriptions
<lifeless> in that file
<salgado> oh, I see
<lifeless> new stuff should generally just go into devel.
<lifeless> if we're -totally- confident in its stability, putting it into 1.0 is fine, putting it into beta is pretty pointless given its substantial skew vs 1.0
<flacoste> hi lifeless
<lifeless> flacoste: shall we catchup/handoff ?
<flacoste> lifeless: sure
<flacoste> lifeless: i can do it later also
<flacoste> lifeless: really up to you
<lifeless> now is good
<flacoste> lifeless: skype?
<jml> allenap: no worries
<lifeless> flacoste: https://bugs.launchpad.net/launchpad-foundations/+bug/651766
<_mup_> Bug #651766: please record concurrent query count on servers <oops-infrastructure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/651766>
<jml> :(
<jml> looks like db_lp is still broken
<jml> and that something similar is now also affecting lp
<jml> lifeless: ping
<lifeless> otp
<jml> ok.
<lifeless> won't be long
<thumper> morning
<thumper> jml: I have a quick question for you
<jml> thumper: sure. I'm otp atm.
<thumper> jml: I remember in the years past you had looked at extracting the codehosting server into a separate project, did that happen?
<jml> thumper: not really. I made txsshserver, but it's kind of dead. lp.services.sshserver is much closer to the spirit of that.
<jml> thumper: my goal was always to extract the generic ssh stuff we did, rather than a separate codehosting service
<thumper> ah, ok
<jml> thumper: did you want to have a catch up call?
<allenap> lifeless: pong
<thumper> jml: sure, I'm just going through emails
<thumper> now?
<thumper> mumble?
<jml> thumper: mumble's great. one sec.
<jml> thumper: join me in the strategerie
<lifeless> allenap: I humbly suggest that the bug you qa-ok'd which timed out on staging and has additional repeated queries, isn't qa-ok :)
<allenap> lifeless: Okay, fair enough :) There's nothing built on it so I'll revert it.
<lifeless> allenap: if the additional queries *do not* happen on users outside the saved-search user group
<lifeless> allenap: then I think it is qa-ok
<lifeless> allenap: but I couldn't tell if that was the case, thus pinging you
<flacoste> wow, one of the top landing page on launchpad is https://launchpad.net/~jd-team/+archive/jdownloader
<flacoste> even above /ubuntu !
<allenap> lifeless: There are no saved searches yet, and it's not possible to create them, so this will happen to everyone. I'll revert it.
<lifeless> allenap: thanks.
<lifeless> allenap: you need to mark the bug qa-bad and add a tag bad-commit-12345 (the revno in stable thats bad)
<lifeless> allenap: when you revert, you need [rollback=12345] in your commit message (or use the --rollback option to lp-land)
<allenap> lifeless: Do I need to run the revert via ec2, or just land it?
<lifeless> I generally just land
<lifeless> if its a straight revert
<lifeless> if its days old or something ec2 can be a good sanity check ;)
<allenap> lifeless: In this case I think it'll be safe. I'll run some sanity checks locally.
<ccxCZ> I got weird behavior on api <project>?ws.op=getBranches; There is branch that's fully merged and it sometimes appear on the list and sometimes not which results in useless messages sent by my irc bot.
<flacoste> ccxCZ: for how long has it been fully merged?
<ccxCZ> ~40 minutes now
<flacoste> ccxCZ: hmm, that's weird
<ccxCZ> maybe it's gone now, let me check
<ccxCZ> but it lasted for 20 minutes at least
<ccxCZ> should the branch appear there or not?
<ccxCZ> seems it settled now, but fluctuations like this are seriously annoying
<lifeless> is your script logged in ?
<ccxCZ> no, I just do plain http request on the uri
<lifeless> then you're getting cached responses
<lifeless> from the squid cluster that anonymous requests go to.
<lifeless> If you want accurate data, you need to sign in and use the auth cookie
<wallyworld> abentley: http://paste.ubuntu.com/536445/
<lifeless> the reason its fluctuating is that we have multiple squids and one had an older version of the answer, the other a newer version
<ccxCZ> i see
<ccxCZ> is there way to stay at one proxy without having an account?
<lifeless> no
<lifeless> primarily because its a load balancing cluster; it would defeat the point :)
<jml> poolie: ping
<ccxCZ> and is there a way to tell apart older data from newer for example by some http header?
<lifeless> allenap: also
<lifeless> allenap: you might want to revert your subunit change
<ccxCZ> Date header seems usable, how often do responses stay in squid?
<jml> lifeless: why would he want to do that?
<lifeless> because subunit's parser wants bytes
<jml> no it doesn't
<jml> it *ought* to want bytes
<lifeless> it does want bytes
<lifeless> it embeds binary data literally in-line in its output
<lifeless> you'll get unicodedecode errors trying to read a subunit stream as utf8
<lifeless> if there is nonutf8 data in any of the attachments.
<jml> TypeError: _StringException expects unicode, got "lost connection during failure report of test 'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout'"
<lifeless> yes, there is a problem there.
<lifeless> passing a unicode file handle in is not a solution.
<jml> or at least, not a very good one
<lifeless> a solution would work.
<jml> you think the failures we're seeing on the bot now are related to allenap's fix?
<lifeless> no
<lifeless> I think that allenaps fix will break if/when any binary data attachments that are not utf8 get into the stream
<allenap> lifeless: Okay, I'll revert that one too :) Once we're out of testfix anyway.
<lifeless> jml: this is the current subunit code:
<lifeless>     def _lostConnectionInTest(self, state_string):
<lifeless>         error_string = u"lost connection during %stest '%s'" % (
<lifeless>             state_string, self.current_test_description)
<lifeless> error_string *is* unicode.
<lifeless> upgrading the subunit used is a solution.
<jml> ahh, *that* will be why I didn't see the bug when I looked a week ago
<jml> because I was looking at a version that already had the fix
<lifeless> yes
<allenap> lifeless: Via the Launchpad PPA
<allenap> ?
<lifeless> allenap: however we do it, we should do it.
<lifeless> (I don't know ;P)
<allenap> Okay. I'll file a bug for that.
<lifeless> thanks !
<lifeless> we probably also want one that subunit *accepted* a unicode stream at all
<lifeless> allenap: I'm sorry that you went to the effort to fix it and had this happen :(
<flacoste> allenap, lifeless: if a newer subunit isn't availabe in Ubuntu, someone needs to upload it to the ppa
<flacoste> and then upgrade the launchpad-dependencies to depends on that newer version
<flacoste> and then file a RT to have it upgraded in the data centre (PQM +buildbot images)
<flacoste> upgrading the ec2 images is also probably a good idea
<allenap> lifeless: That's alright, it was a pretty small change, and it put it where you could comment on it.
<mwhudson> "Xen guests sometimes ignore the first network packet", wow i didn't expect the builder problems to be at that level
<cody-somerville> mwhudson, where did you read that
<cody-somerville> mwhudson, ?
<lifeless> more interesting to me is why we're seeing interrupted streams; I suspect something is printing to stdout
<lifeless> *or* we're mixing stderr into stdout.
<lifeless> jml: ^
<wgrant> mwhudson: That's not it.
 * jml blinks
<wgrant> mwhudson: Non-virt builders are also affected.
<jml> lifeless: can you phrase that as a question?
<mwhudson> cody-somerville: some soyuz bug
<mwhudson> wgrant: ah
<cody-somerville> LP #586359
<_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections <Launchpad Auto Build System:Triaged> <Soyuz:In Progress by julian-edwards> <https://launchpad.net/bugs/586359>
<jml> wgrant: bigjools said otherwise when I asked him about it
<wgrant> jml: Grep logs for palmer, notice that palmer is non-virt, profit.
<wgrant> Oh.
<wgrant> It might explain the "Connection timed out" thing, but not the big problem ("User timeout caused connection failure")
<jml> right.
<lifeless> jml: does the controlling process get stdout and stderr from the child on separate fds and keep them separate everywhere that they might be fed to subunit ?
<jml> lifeless: on the buildbot? Not only do I not know, I also don't know how to find out.
 * jml checks ec2test.remote though
<jml> lifeless: they aren't kept separate in devscripts.ec2test.remote
<lifeless> they need to be
<lifeless> or this can happen:
<lifeless> failure: start_of_testWarning: something on stderr
<cody-somerville> I don't think that script is sufficient to say that Xen guests sometimes ignore the first network packet. OEM Services uses Xen guests for our image build system and we have not run into anything like this - and we would if the premise was correct.
<lifeless> as input to subunit
<jml> lifeless: sure, that makes sense.
<lifeless> jml: when that happens, we will see 'lost connection' errors.
<jml> the code is well factored and has great test coverage, so it should be easy enough for someone to fix
<jml> lifeless: what's buildbot using, I wonder.
<lifeless> the buildbot subunit support I added, and is correct.
<lifeless> we may be doing something differnet
<jml> uhhh
<lifeless> (buildbot can watch subunit streams directly)
<jml> yeah, I just don't see any subunit in the output
<jml> hang on, this is ZTIS
<poolie> jml, hi
<jml> poolie: got time for a call?
<jml> lifeless: nothing in ZTIS makes me think it has the same bug, but I'm not looking very closely.
<jml> I have to go & sleep now
<jml> (not in the fire)
<jml> g'night
<cr3> lifeless: regarding our conversation about eager loading result sets yesterday, is it preferable to return a result set with a limit, result_set.config(limit=limit), or a slice, result_set[:limit]?
<maxb> Where has the "Nominate for release" link disappeared to, in malone?!
<cr3> lifeless: I'm leaning towards the slice because, depending on the size of the limit, I'm not sure whether storm might make more than one roundtrip when iterating over the collection
<lifeless> cr3: those things are equivalent
<lifeless> maxb: its restricted to drivers
<lifeless> maxb: which was the original intent
<wgrant> No.
<cr3> lifeless: in that case, I might lean towards using config then so that I can support both defined and undefined limits (None) as keyword argument, assuming config(limit=None) behave the way I think it does
<wgrant> "Target to release" is restricted to drivers (and uploaders, in Ubuntu)
<lifeless> cr3: erm
<wgrant> "Nominate for release" is restricted to bug supervisors.
<lifeless> cr3: folk can change it after by slicing further
<lifeless> cr3: e.g. resultset = ..
<lifeless> resultset = resultset[1:2]
<lifeless> resultset = resultset[:4]
<lifeless> wgrant: i stand tweaked
<wgrant> It would be nice if Launchpad didn't impose its will over every project.
<cr3> lifeless: good point, the offset and limit aren't enforced until actually iterating over the result set, so I guess I'm confused about the appropriate way to expose this through the webservice
<cr3> wgrant: all your projects are belong to us?
<lifeless> cr3: the webservice understands slices
<lifeless> wgrant: I support having more knobs
<lifeless> wgrant: this change was done for Ubuntu as it happens
<wgrant> lifeless: !!!
<wgrant> lifeless: I know.
<wgrant> But Launchpad seems to have an Ubuntu-size-fits-all policy :/
<lifeless> wgrant: I think its more nuanced than that
<lifeless> wgrant: we get an awful lot more feedback from Ubuntu users than others though.
<wgrant> lifeless: I'd argue that that's because you have few large projects using your service, partly because it's slow, but also partly because it's so inflexible.
<lifeless> wgrant: we have a balance to strike between configurability and homogeneity
<maxb> lifeless: *sigh*. This means that non-bug-supervisors cannot correctly follow the documented Ubuntu SRU proposal process any more
<lifeless> wgrant: its important to mark that skills folk learn for working on bugs in Launchpad projects be broadly applicable
<lifeless> wgrant: I don't know that that is really achievable, because humans use tools in their own fashion, always.
<lifeless> wgrant: but unless or until jml tackle that point, we're bound to try.
<lifeless> maxb: that sounds like Ubuntu got what they wanted and then didn't follow up and adapt their own docs ;)
<lifeless> maxb: uploaders can follow the SRU process
<lifeless> maxb: the reason non-uploader non-supervisors cannot do it is because of abuse by affected users making the system far too noisy.
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.12 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/ | Get the code: https:/â/âdev.launchpad.net/âGetting
<ccxCZ> seems your http server/squid does not handle If-Modified-Since header properly
<lifeless> for anonymous requests its overriden
<lifeless> you cannot force a backend request through it
<ccxCZ> squid should still respond with 304
<lifeless> what is it responding with ?
<ccxCZ> always 200 w/ full body
<ccxCZ> http://paste.pocoo.org/show/296079/
<lifeless> thats legitimate
<lifeless> ccxCZ: please remove 'edge.' from your url
<lifeless> ccxCZ: can you show the request as well
<ccxCZ> http://paste.pocoo.org/show/296082/
<lifeless> thanks - we're getting rid of edge ;)
<ccxCZ> now my client always get 401, but wget is fine. some Useragent magic in there?
<lifeless> I don't think so
<lifeless> but I'd need to see your request to know
<wgrant> You need to send a User-Agent.
<wgrant> Doesn't matter what.
<ccxCZ> actually sending some user agent helped
<ccxCZ> one more thing, number of returned branches seems to limited to some arbitrary value (75 on api.launchpad, 50 on edge), is there way to remove this limit?
<wgrant> ccxCZ: API collections are batched.
<wgrant> So you need to request ~75 at a time.
<wgrant> I forget the exact syntax.
<lifeless> there's a new api coming
<lifeless> which will help
<ccxCZ> also what form is modified_since argument in? unix timestamp?
<ccxCZ> honestly there are lot of omissions in the 1.0 api doc
<lifeless> the docs are very very lacking
<wgrant> The primary interface to the API is launchpadlib, which handles that sort of thing for you.
<lifeless> there is a bug about this
<wgrant> But I believe it's ISO8601
<wgrant> And yes, the docs are pretty bad.
<ccxCZ> launchpadlib won't work with twisted
<lifeless> txrestful does that I believe
<ccxCZ> I know, but it seemed overly complex to me and far from complete
<ccxCZ> since I only need to request one uri, using library like this seemed like overkill
<lifeless> I agree
<ccxCZ> the thing I wrote is irc bot that monitor project's branches and posts updates to a channel. I was advised I should post to launchpad-dev about it.
<lifeless> cool
#launchpad-dev 2010-11-26
<thumper> where are the soupmatchers?
 * thumper pulls again
<lifeless> lp:
<lifeless> ?
<wgrant> lifeless: Did your parallel-supporting DB fixture ever land?
<lifeless> no
<wgrant> :(
<lifeless> feel free to hack on it
<wgrant> lifeless: Hm, that's a bit of an odd one.
<wgrant> Thanks.
<lifeless> wgrant: oh, its easy
<lifeless> wgrant: canonical.lp.dbhost
<lifeless> etc
<wgrant> Probably, yes.
<lifeless> are redudant fugly shit
<wgrant> It's just not what I would have expected.
<lifeless> the script entry point isn't resetting it quite right
<lifeless> with the result that fti.py attempts to work on launchpad_dev
<lifeless> rather than the supplied -d launchpad_test_template or whatever
<wgrant> Yep.
<thumper> gah
<thumper> I have a test that failed on ec2, but passes locally
<thumper> WTF?
<thumper> I've done a make schema
<thumper> but still weird shit ...
<lifeless> isolation error?
<lifeless> what test
<thumper> p.code.browser.tests.test_sourcepackagerecipe.TestSourcePackageRecipeView.test_request_builds_page
<thumper> wit an l at the start
 * thumper fires up make run
<poolie> thumper: hi?
<thumper> poolie: hey
<poolie> shall we have a chat if you're back?
<thumper> sure
 * thumper shoves it back through ec2 again
<ccxCZ> too tired to write the mail now, here is link if anybody wants to play with it http://webprojekty.cz/ccx/trac/wiki/lpbot
<thumper> ccxCZ: a link to what?
<ccxCZ> irc notification bot for launchpad projects
<ccxCZ> anyway gn, I have to go
<lifeless> wgrant: databasefixture and librarian branches are updated and pushed, if you're interested
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      134 / 5092  Archive:+index
<lifeless>       70 /  240  BugTask:+index
<lifeless>       24 /  249  Distribution:+bugs
<lifeless>       20 /  118  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>       11 /  265  POFile:+translate
<lifeless>       11 /   13  DistroArchSeries:+index
<lifeless>       11 /    1  ProjectGroup:+milestones
<lifeless>        7 /   35  Milestone:+index
<lifeless>        6 /  306  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        5 /    9  Person:+bugs
<wgrant> Ow :(
<wgrant> I must get back to Archive:+index.
<lifeless> wgrant: that would be the bomb
<wallyworld> thumper: ping
<jtv> poolie: a thoughtâ¦ would it save EC2 startup time (or server load) if we could use a virtual-machine image that had some fixed version of our source tree or repo pre-installed, so that it only needs to download the updates?
<lifeless> jtv: thats what we have, isn't it ?
<jtv> oh :)
<thumper> wallyworld: hey
<thumper> wallyworld: it is time for pizza and movies :)
<wallyworld> just wanted to confirm that you want me to look at bug 670452 next
<_mup_> Bug #670452: Hard to find related branches when composing recipe <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/670452>
<thumper> wallyworld: aye
<wallyworld> cool. i'll take a look once i finish seeing if i can improve that oops reporting test
<wallyworld> haveagoodweekend
 * wgrant theorises that HTML forms were designed as a cruel joke.
<jtv> wgrant: carefulâthat line of reasoning leads down a dangerous path to paranoia, misanthropy and most probably, truth.
<jtv> General observation about software stuff, not a comment about HTML forms specifically.
<wgrant> Heh.
<jtv> Think Mahabharata-style stripping of illusions.
<jtv> I could say something about pills in The Matrix if I could bear the thought of that crappy film.
<jtv> Besides, I'm regularly informed through emails from helpful strangers that given the choice, there is no good reason to spurn the blue pill.
<jtv> â¦
<jtv> You're still reading the Wikipedia entry on Mahabharata, aren't you?
<wgrant> I was debating whether or not to look it up.
<jtv> If you were debating it on your own, you probably want prozac, not viagra.  My bad.
<StevenK> jtv: Hey, the first one was awesome. It was just the other two that made me cringe
<jtv> StevenK: the first pill, the first book of the Mahabharata, or the first wgrant personality in the debate?
<StevenK> jtv: I'm sorry, the first Matrix movie
<jtv> Ah that one.  Did one thing for me: realize how precious my time was.  Only time I ever walked out of a cinema shortly after the film started.
<jtv> I've met other people who did that.
<wgrant> I only saw them a few months ago. I quite liked the first one.
<spm> jtv: I had a similar experience with my wife, sister and sisters friend at a production of Les Mis. Yay. Play is over. We can go home now. "Um Steve? no. That's just the first half, this is the intermission" AAAAAAAAAAAAAAAAAAAAA
<jtv> spm: hang on, still laughing
<spm> :-)
<jtv> Hah.  Whew.  That's better.
<spm> So I went home, ladies, enjoy the play. Ring me when you want to be picked up. :-D
<spm> Bored out of my tiny little mind. Some lady was whing about doing it tough. Some dudes with guns were whatever. Bored.
<spm> Bored against the backdrop of the French Revolution. But still bored.
<jtv> Saw one cartoon about this kind of thingâ¦ spectator #1: "man but this is boring.  I should have brought a good book!"
<jtv> Spectator #2: "that's why I always read the reviews first."  Goes back to reading Tolstoy.
<spm> bwhahahaa
<poolie> jtv, hi, it could
<jtv> poolie: lifeless noted that we already seem to do this.
<poolie> could be good to know what fraction of total time is spent pulling that source
<poolie> i suspect it's not very high
<henninge> One of the current test failures in devel is strange - it reports a "test with error" but the test does not output a traceback or anything ...
<henninge> looks like this: henning@dustpuppy:$ bin/test -vvcm lp.services.job.tests.test_runner -t TestJobCronScript.test_configures_oops_handler
<henninge> Running tests at level 1
<henninge> Running canonical.testing.layers.LaunchpadZopelessLayer tests:
<henninge>   Set up canonical.testing.layers.BaseLayer in 0.004 seconds.
<henninge>   Set up canonical.testing.layers.ZopelessLayer in 9.778 seconds.
<henninge>   Set up canonical.testing.layers.DatabaseLayer in 0.482 seconds.
<henninge>   Set up canonical.testing.layers.LibrarianLayer in 10.697 seconds.
<henninge>   Set up canonical.testing.layers.MemcachedLayer in 0.112 seconds.
<henninge>   Set up canonical.testing.layers.LaunchpadLayer in 0.000 seconds.
<henninge>   Set up canonical.testing.layers.LaunchpadScriptLayer in 0.005 seconds.
<henninge>   Set up canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds.
<henninge>   Running:
<henninge>  lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler
<henninge>   Ran 1 tests with 0 failures and 1 errors in 0.848 seconds.
<henninge> Tearing down left over layers:
<henninge>   Tear down canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds.
<henninge>   Tear down canonical.testing.layers.LaunchpadScriptLayer in 0.002 seconds.
<henninge>   Tear down canonical.testing.layers.ZopelessLayer ... not supported
<henninge>   Tear down canonical.testing.layers.LaunchpadLayer in 0.000 seconds.
<henninge>   Tear down canonical.testing.layers.LibrarianLayer in 0.003 seconds.
<henninge>   Tear down canonical.testing.layers.MemcachedLayer in 0.101 seconds.
<henninge>   Tear down canonical.testing.layers.DatabaseLayer in 0.236 seconds.
<henninge>   Tear down canonical.testing.layers.BaseLayer in 0.001 seconds.
<henninge> Tests with errors:
<StevenK> henninge: Pastebin?
<henninge>    lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler (subunit.RemotedTestCase)
<lifeless> is that test a twisted test?
<henninge> oops
<lifeless> henninge: does it have run_tests_with = ?
<henninge> sorry, meant to paste http://paste.ubuntu.com/536572/
<henninge> lifeless: I don't konw what that is but that string does not appear in the file .
<henninge> looks like this http://paste.ubuntu.com/536573/
<lifeless> henninge: what about ZopeTestInSubProcess
<henninge> actually, I just realized that that I pasted the wrong output but it's similar.
<lifeless> ah, interesting
<henninge> lifeless: no, that neither
<lifeless> this might be allenaps change to use unicode in subunit
<henninge> aha
 * henninge tries with that reverted
<henninge> nope,same result :(
<lifeless> wgrant: we're turning the publicrestrictedlibrarian back on
<lifeless> because it seems to work in tests
<wgrant> lifeless: Yay.
<wgrant> But why? :/
<wgrant> Was Apache fixed?
<lifeless> wgrant: can you give it a thorough spin
<lifeless> wgrant: To quote spm, FIIK.
<wgrant> Heh.
<wgrant> Sure.
<wgrant> Say when.
<henninge> so, r11980 seems to be the culprit for the current test failure
<henninge> at least for two of them ...
<henninge> the ones that don't produce output
<spm> wgrant: no, no changes in apache. I did a moderate amount of local faffing around to determine that the proposed changes shouldn't make a difference (apache ignores those bits), ie it should "Just Work" as is. So we switched it on, tested, off. and it worked. ie FIIK.
<lifeless> wgrant: oh, it was live at '19:21 < lifeless> wgrant: can you give it a thorough spin
<lifeless> '
<wgrant> lifeless: Still broken.
<wgrant> Can you see https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368?
<wgrant> The build log link there.
<wgrant> And lots of other similar ones.
<StevenK> The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.
<lifeless> I'm not in soyuz-team
<lifeless> wgrant: gimme a url
<wgrant> lifeless: Neither am I, but I have a sub. I thought you might be a commercial admin.
<wgrant> https://i35262589.restricted.launchpadlibrarian.net/35262589/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz?token=b8e6f2f070469191af32ce99bb92fae8
<wgrant> StevenK: Right, it's the usual uncompressed 404, served with gzip headers.
<wgrant> So there are two levels of WTFery.
<StevenK> Haha
<StevenK> HTTP request sent, awaiting response... 404 Not Found
<StevenK> Ah yes
<lifeless> spm: please turn it off again
<spm> done
<spm> so how did it work for me/us? but not here?
<spm> Oh. its the 404?
<wgrant> It works for some files.
<lifeless> spm: ok, so it *is* the 404s
<spm> not the files themselves, the 404 response?
<lifeless> wgrant: does that same build log show properly now?
<StevenK> Repeating the same wget still gives me a 404
<lifeless> of course
<lifeless> you need the source url
<lifeless> on the appservers
<lifeless> thats what redirects or proxies
<StevenK> Yes, the build log now works
<wgrant> lifeless: Proxying works fine.
<lifeless> wgrant: whats the original url ?
<StevenK> https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368/+files/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz ?
<lifeless> I suspect url mangling
<lifeless> I need to chat to spm briefly
 * spm questions the brief part
<StevenK> Bwahaha
<lifeless> so I think its the %
<lifeless> wgrant: whats the debian version string for that, unmangled
<StevenK> 1.4-2+px1
<lifeless> thanks
<lifeless> so, % urls are mangled by something in the request path
<lifeless> they are decoded
<lifeless> and then passed on decoded?!
 * lifeless bets on apache
<lifeless> wouldn't know a standard if it bit it
<StevenK> lifeless: I was thinking that
<lifeless> https://bugs.launchpad.net/launchpad-foundations/+bug/681675
<_mup_> Bug #681675: content coding for build logs is done in apache rather than the librarian <Launchpad Foundations:New> <https://launchpad.net/bugs/681675>
<lifeless> updated https://bugs.launchpad.net/launchpad-foundations/+bug/677270
<_mup_> Bug #677270: apache/squid  breaks restricted librarian on urls with percent encoded characters. <Launchpad Foundations:In Progress by canonical-losas> <https://launchpad.net/bugs/677270>
<henninge> Anybody have a clue for me? This test is failing *even* when I revert to the revision of the last successful buildbot run.
<henninge> http://paste.ubuntu.com/536585/
<henninge> It start two jobs and both seem to fail on _pythonpath_. Is that a local error for me?
<henninge> I am getting this on devel r11979
<StevenK> So does buildbot, it seems
<henninge> which is said "last known good" revision
<StevenK> henninge: If changing devel doesn't fix it, I'd blame an egg change?
<henninge> hm
<lifeless> henninge: 11980 is the twisted runner change, I was thinking it was to blame :)
<henninge> but doesn't version.cfg control which version of an egg a revision is using?
<lifeless> jml: ^
<wgrant> henninge: You need to remake.
<lifeless> henninge: you have to run bin/buildout again
<henninge> lifeless: it is to blame for the other two failing test
<henninge> wgrant: make clean ; make build? Did that ...
<wgrant> henninge: Hmm.
 * henninge tries bin/buildout
<lifeless> there's probably a missing dep on versions.cfg
<henninge> no change
<lifeless> buh
 * StevenK stares at Hudson: "took 5 min 5 sec"
<StevenK> However, r11980 is devel #250 on Hudson, and that didn't fail any tests
 * henninge tries r11980 locally again
 * henninge notices that db-devel passed revision 10000
<StevenK> org.apache.commons.jelly.JellyTagException: jar:file:/var/run/hudson/war/WEB-INF/lib/hudson-core-1.377.jar!/hudson/model/AbstractBuild/sidepanel.jelly:34:42:  java.lang.OutOfMemoryError: Java heap space
<StevenK> Wheee!
<lifeless> \//
<StevenK> Java, you suck, and I hate you
<StevenK> I think Hudson is now unhappy, since it now gives 503s
<StevenK> And even /etc/init.d/hudson stop ... doesn't
<StevenK> Heh
<StevenK> Right, it does, it just takes 2 minutes? Srsly?
<lifeless> you were in swap
<StevenK> Grah, it is 50 MiB into it
<StevenK> How is 1.5GiB not enough for Java?!
<lifeless> how much memory did you tell java it can use?
<StevenK> Whatever the default is?
<lifeless> you might have been swapped out even if you weren't exceeding the machine memory in one proc
<StevenK> It's still 48MiB into swap with apache and java dead, so it isn't that
<StevenK> This machine still has a full Launchpad development environment on it, so that can't be helping
<StevenK> And restarting Hudson means it has lost track of the build it was in the middle of
 * StevenK shoots the buildslave in the head
<LPCIBot> Project parallel-test build (15): STILL FAILING in 34 sec: https://hudson.wedontsleep.org/job/parallel-test/15/
<StevenK> lifeless: Do you want me a queue another parallel-test build?
<lifeless> StevenK: ask me on the 3rd
<lifeless> StevenK: of january
<lifeless> sorry, the 4th
<StevenK> By which it would have landed?
<lifeless> no, by which I'll be back at work
<spm> lifeless: "by which I'll be back at work"? you haven't heard? we're replacing you with a perl script. I thought you knew. ??
<lifeless> spm: never a win
<StevenK> #!/usr/bin/perl -w
<lifeless> see
<StevenK> while (1) { nag(); }
<lifeless> minus win
<spm> nah. -w is for wussies
<StevenK> Perl coder for six years at previous job. Old habits die hard
<StevenK> And there goes X in a screaming heap
<adeuring> good morning
<jml> hello
<jml> who's fixing the build?
<jml> henninge: hello
<jml> henninge: have you submitted anything to fix the build?
<wgrant> I'm confused.
<wgrant> r10005 doesn't look like a testfix to me...
<henninge> jml: I hoped to find out what to do about the third test still failing
<henninge> not much point in fixing two out of three tests
<jml> wgrant: what about it? (I haven't looked, trying to deal with the current failure)
<henninge> wgrant: no but then db-devel wasn't broken
<jml> henninge: agreed. just wanted to check
<wgrant> henninge: It's labelled as testfix.
<jml> hey look
<wgrant> But doesn't seem to have a test fix.
<jml> it's ZopeTestInSubProcess again
<jml> ffs
<henninge> wgrant: sorry, jtv is just telling me off about it, too ...
<jtv> "now now henninge, that's not nice"
<jtv> there, see?
<wgrant> I think it might confuse the QA bot a bit, but I guess we'll see.
<henninge> we'll see ...
<jml> I wonder what happened re upgrading subunit
<jml> I see we're at the "someone should do something about this" phase of discussion
<jml> ok.
<wgrant> jml: That's surely the best phase.
<jml> wgrant: I don't know, I'm a "someone else just did something about it" kind of guy.
<jml> henninge: test_timeout fails for you locally, even if you revert to r11979?
<jml> we should change these to be testtools RunTest thingies anyway
<lifeless> presumably they are in sub processes for a good reason.
<henninge> jml: yes, it does.
<jml> lifeless: I mean, there should be a testtools subprocess RunTest thingy
<jml> lifeless: rather than using a mixin with a run() method
<lifeless> +1
<jml> henninge: ok. so something is very weird.
<jml> because a 'known good' build is breaking
<jml> I'm going to try to get more info on what the child process is doing
<jml> I *really* doubt r11980 has anything to do with these failures, fwiw.
<lifeless> jml: Its plausible
<lifeless> jml: if the reactor is in a surprising state
<lifeless> it forks, it doesn't exec
<jml> how can I get at the stdout & stderr of the child process?
<bigjools> hello
 * jml cheats
<jml> lifeless: oh wait, it's not plausible at all
<lifeless> jml: ok
<jml> lifeless: because the test fails with r11979 too
<jml> also fails when not run in a subprocess
<henninge> jml: to be clear: I could verify that the other two failing tests (*not* the timeout test) pass on 11979 but fail on 11980. The timeout test failure does not seem to be connected to that revision.
<henninge> s/verify/verify locally/
<jml> henninge: do you have a plausible reason as to why?
<henninge> no, I don't ... :(
<henninge> actually ...
<jml> allenap: I don't think it's related to the unicode thing
<jml> henninge: I suspect bad local state, actually
<StevenK> jml: As well as inside the buildbot slaves?
<jml> StevenK: yes.
<henninge> jml: I don't deny that. Did you try it on your machine?
<henninge> jml: the only clue is this: http://paste.ubuntu.com/536585/
<StevenK> I find that a little hard to believe. Can haz proof?
<henninge> The test starts two jobs wich both fail on a missing _pythonpath_
<jml> StevenK: working on it.
<henninge> But I don't know how that can happen just for those two and not for everthing else.
<jml> the clue is that reverting to about 20 revisions ago still has the failure
<henninge> good point
<StevenK> jml: Hudson might actually be your proof
<StevenK> jml: Now that I think about it
<jml> also, remember, the Makefile has been changed four times recently (11937, 11957, 11958 and 11979)
<jml> StevenK: in that it works?
<StevenK> jml: Indeed
<StevenK> jml: But the Hudson slaves are different to the buildbot slaves
<jml> naturally
<jml> how do we clean out the build on buildbot?
<StevenK> I suspect we run make clean only
<jml> how do we move from suspicion to certainty?
<StevenK> We channel mars, or someone who actually does more with buildbot than try and replace it
 * jml has to go
<bigjools> rf-get and update-sourcecode doesn't want to fix the lack of a AsynchronousDeferredRunTestForBrokenTwisted module....
<allenap> jml: I did find an interesting problem. The read side of the pipe in ZTISP was being opened rU. Locally that has caused tests to error, but with no output at all. Changing it to rb means that they succeed. I don't really know why, but perhaps this is related to the unicode thing too. I'm landing a branch now to revert the getreader() thing, and to open both ends of the pipe as binary.
<jml> bigjools: that sucks. will try to look into that after the build is fixed
<jml> allenap: cool. that fixes all three failing tests?
<bigjools> jml: a make clean fixored it
<jml> bigjools: ahh good
<bigjools> not sure why I needed that though :(
<jml> bigjools: me neither. maybe I did something a bit stupid as I developing it.
<jml> it's not clear how to evolve Launchpad & an upstream in parallel
<jml> hmm
<jml> I wonder how I can make a script that downloads an openid-protected page
<allenap> jml: I think it should fix them, but then I haven't been able to replicate the exact failures. I'm hoping :)
<bigjools> jml: see utilities/pastebin
<bigjools> well, the old one
<allenap> jml: from faustian_pact import convenient_openid_wrapper
<jml> hmm. I don't think I have that one.
<bigjools> sure you do, you have bzr repo? :)
<jml> bigjools: faustian_pact is in Launchpad's history?
<bigjools> maybe not that one
<bigjools> I can't see you facing allenap on IRC when you talk
<jml> yeah, they probably haven't ported that to kde yet
<jml> db_lp broken again
<jml> any volunteers?
 * bigjools 's kde sarcasm detector app goes off
<jml> allenap: test_timeout still fails for me with your branch, but the other two are fine.
<jml> https://bugs.launchpad.net/launchpad-foundations/+bug/681770
<_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout <Launchpad Foundations:New> <https://launchpad.net/bugs/681770>
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.12 | PQM is open | firefighting: db-devel broken | https:/â/âdev.launchpad.net/ | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> jkakar_: hi
<jkakar_> Hiya jml!
<jml> jkakar: how's things?
<jkakar> jml: Going really well, thanks.  I'm sprinting in Barcelona, so it's been an exciting week.  Just got back from lunch.
<jkakar> jml: How about you?
<jml> jkakar: well. bit of a cold this week but otherwise fine. slowly mastering the art of the automobile
<jml> jkakar: you coming to London any time soon?
<jml> (also, yay sprinting)
<jkakar> jml: Driving, oooh, fun times. :)
<jkakar> jml: Will be in London in a couple weeks, yep.
<jml> all confirmed?
<bigjools> bloody windmill
<mars> bigjools, ?
<bigjools> it hangs when I run it locally, which means my test run never completes
<mars> ah, not fun.  That is new, I don't think anyone has mentioned that yet
<mars> did Deryck kill the windmill suite?
<mars> (yet?)
<bigjools> never mind Deryck, I want to kill it :)
<mars> heh
<mars> bigjools, I should ask: is this problem new?
<jml> speaking of which...
<mars> jml, your @skiptest decorator is the big win for us there
<jml> the build is still broken, failing with a windmill test. I suspect a fix like http://pastebin.ubuntu.com/533852/ might be needed
<bigjools> mars: I'm not sure :(
<bigjools> it only happens on one of my boxes
<jml> mars: wasn't mine,  I don't think
 * jml doesn't actually know anything about windmill, that paste is just what deryck did last time
<mars> jml, that fix didn't land?
<jml> mars: it landed. a different test is failing.
<mars> jml, if the needed fix is similar to what you posted, then it will have to be applied by inspecting the test for races caused by missing timeouts
<jml> mars: make sure you pass that on to whoever sets about actually fixing it.
<mars> sure, I can do that
<mars> Deryck's windmill fix has not landed
<jml> well, I guess we'll look at the bug filed for that particular breakage and find out why
<flacoste> jml: is it only you and me today on The Strategerie?
<jml> flacoste: https://bugs.launchpad.net/launchpad-foundations/+bug/681770
<_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout <Launchpad Foundations:New> <https://launchpad.net/bugs/681770>
<flacoste> jml: lost connection?
<bigjools> would it be evil to use a storm validator to auto-update a different field?
<bigjools> oh dear, it's not good when the same class name appears twice in a test file is it ...
<jml> flymake! pyflakes! happiness!
<bigjools> pyflakes didn't detect it
<bigjools> I only noticed because on of my new tests wasn't getting run
<bigjools> one*
<jml> really?
<jelmer> it gives a warning here for one of the two declarations
<bigjools> hmm
<jml> bigjools: if so, that's a bug I'm going to fix right now.
<bigjools> I wonder if I've got an old vim pyflakes
<bigjools> can someone please look at this, I've got a test failure that I cannot fathom.  It's totally unrelated to my changes yet reverting them makes the test pass.   http://pastebin.ubuntu.com/536736/
<jelmer> bigjools: It must trigger a database flush or something..
<jelmer> bigjools: my bet is that the fact you've changed builderok into a property is the culprit
<bigjools> jelmer: indeed - but how is that affecting whether something else is unicode?
<bigjools> oh, I wonder if it hits the DB and gets converted
<jelmer> bigjools: Yeah, that's what's probably happening.
<bigjools> that's nasty
<bigjools> thanks jelmer
<bigjools> jml: bin/lint.sh does not show the duplicated clas
<bigjools> s
<bigjools> see lib/lp/buildmaster/tests/test_builder.py
<jml> bigjools: I don't use or care about lint.sh
<bigjools> TestBuilder is declared twice
<bigjools> yes but vimflakes doesn't see it either - what do you care about?
<jml> pyflakes
<bigjools> that does not see it either
<bigjools> pyflakes lib/lp/buildmaster/tests/test_builder.py
<bigjools> nowt
<jml> I see.
 * jml fixes
<bigjools> jelmer: heh - we both missed the point of the test - the values should be unequal.  The unicode was a red herring.
<jelmer> argh, I see
<jelmer> bigjools: it was a database flush that was causing the issues though?
<bigjools> I don't know
<bigjools> fg
<jml> /home/jml/src/launchpad/devel/lib/lp/buildmaster/tests/test_builder.py:127: redefinition of class 'TestBuilder' from line 89
<jml> \o/
<jml> lp:~jml/pyflakes/duplicate-class-defs if anyone wants to use it.
<jml> http://divmod.org/trac/ticket/3034 if you care about upstreaming
 * bigjools claps jml
<jelmer> jml: that was quick!
<jml> jelmer: pyflakes is really easy to patch
<jml> particularly after the first patch :)
<jelmer>  jml: I tried to merge the latest trunk into the bzr lazy_import branch at one point and gave up.
<jml> oh yeah. they changed it quite significantly recently to use ast or something
<jml> still structured the same but quite different textually
<bigjools> jelmer: I found the problem
<bigjools> Builder.selectBy(builderok=True)
<jelmer> ah, a select on a non-storm property?
<bigjools> yeah
<jelmer> confusing error then..
<bigjools> the selectBy doesn't break!
<bigjools> it just returns nothing
<jml> who wants a Friday afternoon vaguely on-topic distraction?
<jml> I have pretty pictures
<jml> http://people.canonical.com/~jml/lpstats/good-and-bad-by-time-bar.png
<jml> http://people.canonical.com/~jml/lpstats/how-many-tests-are-slow.png
 * bigjools needs one of those
<bigjools> not a metric? :)
<jml> right
<jml> it *could* be a metric, but the numbers don't quite add up and I haven't figured out why
<bigjools> kidding - slightly.  What's the colours/numbers measuring?
<jml> the colours are ... bands of test run time
<jml> e.g. tests that take under a second to run, but over 0.1s are green in both graphs
<bigjools> right, so it's a distribution
<jml> http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png and http://people.canonical.com/~jml/lpstats/good-and-bad.png are different versions of the same respective data
<jml> bigjools: yes
<bigjools> is it easy to change to a line graph?
<jml> bigjools: one distribution is by number of tests within that band
<bigjools> might be easier to interpret
<jml> bigjools: the other is by time taken by tests in that band
<jml> bigjools: I think the bar graphs are close to what you mean by line graphs
<jml> unless you mean something like http://people.canonical.com/~jml/Tests-finished-each-minute.png
<bigjools> I've only ever seen distribution graphs done as a curvy line, usually looking like a bell :)
<jml> they are frequently done as bars, particularly for logarithmic distributions
<jml> (i.e. http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png and http://people.canonical.com/~jml/lpstats/good-and-bad.png)
<jml> or maybe I meant e.g.
<bigjools> but then it's Friday afternoon, I've had no sleep from watching too much cricket, and I need a beer
<jml> hah
<bigjools> apart from that I feel quite lucid
<jml> anyway, the rough conclusion is that the tests that take over a second make up a small fraction of the tests by number
<jml> (~2000 / 12000)
<bigjools> I bet we could make those 1 second or under tests faster
<jml> but make up the majority by time
<jml> (~8000 / 10000)
<jml> bigjools: we could, and that would be great, but the graphs basically say that if we _only_ did that, it wouldn't matter much.
<bigjools> maybe
<jml> if we took the test suite as it is now, and ran the fastest tests first, those taking under a second would finish in the first half hour, the remaining time would be spent in the ~15% of the tests that take over a second
<bigjools> let's just delete them all
<bigjools> btw how do doctests fit in this?
<bigjools> are they one test or lots of tests?
<jml> each doctest is counted as one test.
<bigjools> I bet those 15% are largely doctests :)
<bigjools> (including pagetests)
<bigjools> in fact the latter are slower than an Aussie outfielder
<jml> data is gathered from: ./bin/test --subunit | subunit-ls --times --no-passthrough | sed -e 's/ /,/' > test-runtime-data.csv
<jml> hah
<bigjools> nice analysis though
<bigjools> ditching pagetests gets more weight! :)
<jml> bigjools: fwiw, of the ~1500 tests that take over a second, 574 are doctests
<bigjools> oh, wow
<jml> 315 of those are pagetests (rough count)
<bigjools> I expect manty of the others are soyuz uploader tests
<jml> I did this: gzip -dc /home/jml/Downloads/testtools-experiment-r11753.subunit.gz | subunit-ls --times --no-passthrough | awk '{ print $2 " " $1 }' | sort -n > test-run-times
<jml> then used emacs to chop out all of the sub 1s tests
<jml> then wc -l test-run-times, grep -c _txt test-run-times and grep -c ' [0-9x][0-9x]' test-run-times
<jml> bigjools: slow unit tests, ordered by time: http://pastebin.ubuntu.com/536769/
<bigjools> some obvious suspects
<jml> bigjools: by name: http://pastebin.ubuntu.com/536771/
<jml> heh
<bigjools> test_all_scripts makes me weep
<jml> of course, subunit encodes layer start up as a test :)
<bigjools> it's duplicating tests done elsewhere
<jml> 82.739 gina_txt
<jml> 86.680 soyuz-upload_txt
<bigjools> the shame
<salgado> abentley, any idea what's with the list of 'properties changed' messages in the diff at https://code.launchpad.net/~salgado/launchpad/expose-blueprints/+merge/41898 ?
<jml> g'night all.
<jml> have a good weekend
<salgado> abentley, looks like a bug in the incremental diff thing?
<abentley> salgado: No, they're new to me.
<abentley> salgado: Yes, definitely that.
<salgado> abentley, can you look into that or do you want me to report a bug and/or complain on the ML?
<abentley> salgado: please file a bug.
<abentley> salgado: I will investigate, but it needs tracking.
<salgado> abentley, https://bugs.launchpad.net/launchpad-code/+bug/681885
<_mup_> Bug #681885: Spurious 'properties changed' line on incremental diffs <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/681885>
<salgado> thanks for investigating
<abentley> salgado: So I think there's a bug in bzrlib that makes the execute bit of directories and symlinks handled differently for RevisionTrees and PreviewTrees.  For RevisionTrees, it's False and for PreviewTrees, it's None.
<abentley> salgado: This bug will only be exercised if you merge from trunk or your prerequisite branch, but we certainly want you to be able to do that.
<salgado> oh, right, that makes sense.  the busted diff is the one which includes a merge from trunk
<abentley> salgado: I'm going to see if the same thing happens with merge --preview.
<abentley> flacoste: I'm confused about the recent changes to the release process, and lifeless's proposal and how they would affect the PQM closure window.
<flacoste> abentley: let's have a call about it in 30 minutes?
<abentley> flacoste: great!
<flacoste> abentley: i'm on mumble
<cr3> in launchpad, is there a preference in naming methods when getting latest items, like IPersonPublic.latestKarma compared to IPersonPublic.getLatestMaintainedPackages?
<cr3> the former could've been named getLatestKarma or the latter named latestMaintainedPackages, so I'm wondering if there's a general preference... and a motivation would also be nice :)
<salgado> cr3, the former is not an appropriate method name in Launchpad -- methods must be named after verbs
<cr3> salgado: awesome, I'm pleased so hear that methods should start with active verbs
<salgado> in general we have good coding/naming standards in Launchpad.  the only problem is that we don't always follow them
<cr3> salgado: no worries, this innevitably happens in projects when they become very large even with the best of intentions
<cr3> salgado: for example, I can imagine having to define my own oauth classes to be used by the python oauth module which has different naming conventions, and then having those same conventions leak into the rest of my code.
<cr3> in those cases, I usually duplicate method or attribute names at the end of my class definition with a comment like: # Compatibility with oauth module
 * lifeless thinks that this is noise and overspecified in launchpad
<cr3> lifeless: what's noise and overspecified exactly? naming conventions or duplicating names for compatibility purposes?
<lifeless> both
<cr3> lifeless: I'm not sure I follow, what's noisy about naming conventions?
<lifeless> the former is more to remember to meet 'ok to land' which adds little if any actual value; the latter is extra code which makes backtraces longer, will confuse folk that actually use the real module when analysing your code
<cr3> lifeless: I'm not sure I agree but I'll admit that the importance of naming conventions is mostly a matter of taste, so open to pointless debates like vim vs emacs, but I happen to find value in following conventions for consistency purposes which gives me pleasure when using interface that are predictable
<lifeless> I think there are several reasons I don't find that [very] valuable
<lifeless> firstly, its an all or nothing: either you can completely confidently guess at the right thing, or you need to do a search & read docs.
<lifeless> even in small programs there is often enough nuance needed that the obvious thing isn't necessarily the right thing.
<lifeless> secondly, unless you completely ban all external code (which is a bad idea for so many reasons), you have to work on multiple code bases when fixing and refactoring.
<lifeless> Its extremely rare to be able to consume a component and *never* find a bug or issue in it that needs fixing.
<lifeless> I do find value in contracts - e.g. in the context manager contract
<lifeless> that can be argued as a consistency thing, but I think its more reuse.
<lifeless> Its not 'same style', its interface substitutable
<cr3> lifeless: contract definately has value
<flacoste> mars: is there a reason that we didn't disable domain hashing in our google analytics tracker code?
<flacoste> mars: setAllowHash
<flacoste> mars: if i read http://code.google.com/apis/analytics/docs/tracking/gaTrackingSite.html#domainSubDomains correctly we should be using it
<flacoste> otherwise, it means that we are overestimating visitors as each visitor to a different subdomain is considered different
<mars> flacoste, I wasn't aware of that option
<flacoste> mars: ok, i'll land the fix
<mars> flacoste, thanks.  I haven't seen that option before - IIRC I read the help docs, but not the tracking setup docs
<lifeless> flacoste: did you see the apache pain ? :)
<flacoste> lifeless: yes, it was a deeper issue than i thought, but i'm glad that we have a fix through mod_rewrite
<lifeless> flacoste: we do?
<lifeless> flacoste: we left it unfixed..
<flacoste> lifeless: [B]
<flacoste> it seems that will work, no?
<lifeless> flacoste: not in our testing
<flacoste> [P,B]
<lifeless> it mangled the /
<flacoste> !
<flacoste> wow
<flacoste> that sucks big time
<lifeless> GET /57533044/foo+bar.txt?token=48
<lifeless> wrong due to +
<flacoste> would setting the header in the librarian, bypass this?
<flacoste> (as we wouldn't have to use <Location)?
<lifeless> GET /57533044%252ffoo%252bbar%252etxt?token=488
<lifeless> flacoste: the content encoding header is orthogonal
<lifeless> flacoste: it corrupts 404s, it doesn't cause them.
<lifeless> AFAICT
<flacoste> why is it working in the other setup?
<lifeless> flacoste: its not working anywhere is it ?
<lifeless> flacoste: define 'it' :)
<flacoste> well, it works when we disable the feature flag
<lifeless> the librarian isn't used then
<flacoste> we are able to retrieve these files now
<flacoste> doh, of course
<lifeless> with the flag off, the appserver serves it using the internal librarian port to obtain the content
<flacoste> it works because we are serving this content from the app server
<flacoste> right
<flacoste> missed that
<flacoste> so the only fix is either fix apache or change front-end for the restricted librarian?
<lifeless> of course, with the flag off kees has a script that gets 503 timeouts so often it takes hours to download security debs to check :(
<flacoste> lifeless: we could turn the feature flag on for kees :-)
<lifeless> flacoste: we could try to reverse engineer which things will be decoded by apache and match those rules in the librarian.
<lifeless> but that seems spectacularly fugly.
<flacoste> yeah
<lifeless> flacoste: folk are still commenting on this cluster of bugs in apache this year.
<lifeless> even though they are marked variously 'wontfix' and 'fix released'
<flacoste> how did tom feel with switching front-end?
<lifeless> there may be some dark art to make it work.
<lifeless> we didn't talk in detail about it
<lifeless> apache is a known quantity, and there is a feeling that it adds some security vs running (whatever) backend service live on the interwebs
<elmo> I'm not having whatever backend service terminal SSL :-P
<elmo> s/terminal/terminate/
<lifeless> elmo: I was thinking squid
<elmo> well
<elmo> I don't believe this is intractable
<lifeless> elmo: I wouldn't want the complexity of SSL in the librarian code - shudder.
<elmo> U1 ran into similar rewrite escaping issues and were able to work around it
<lifeless> elmo: I thought they did, but I couldn't find the resolution
<elmo> I'd like to explore fixing or working around apache and at least filing bugs about it's broken behaviour further before we seriously consider abandoning it
<lifeless> elmo: certainly
<lifeless> elmo: could you find what U1 did to address this?
<elmo> lifeless: right now?  no, but I'll have a look
<lifeless> thanks
<lifeless> elmo: in terms of fixing apache
<lifeless> I linked to one insightful comment from apache upstream
<lifeless> apparently it does a escape decode very early in its processing pipeline
<lifeless> before mod rewrite or mod proxy get close to the request
<lifeless> I guess adding a request variable at that point to save the original value and pass it down might help.
<henninge> Um ...
<henninge> I think it is getting too late for me ...
<wgrant> The DB config is far more duplicated and evil than I could ever have imagined :/
<lifeless> wgrant: looking at my branch ?
<wgrant> lifeless: Yeah, I've fixed it.
<lifeless> wgrant: I'd like to nuke the canonical.lp.dbhost etc
<lifeless> wgrant: I just didn't want to make the branch bigger
<wgrant> lifeless: Yes...
<lifeless> wgrant: what did I have wrong?
<wgrant> lifeless: The db_options option parser thing used to set canonical.lp.dbname. But you removed that when you turned it into a function.
<wgrant> Also, you missed two references to canonical.lp.dbname.
<wgrant> lifeless: I've made db_options set lp.dbname_override, which lp.get_dbname() now consults. Any less horrific suggestions?
<lifeless> seems fine
<lifeless> wgrant: want me to land it ?
<wgrant> lifeless: Please do.
<wgrant> make check runs OK now, until it hangs as it does on devel.
<wgrant> So ec2 should be happy.
<lifeless> wheres your fix? I'll pull into my branch
<wgrant> lifeless: lp:~wgrant/launchpad/databasefixture
<lifeless> how do you get other people to get emails on ec2land ?
<wgrant> ec2 land, I don't know.
<wgrant> ec2 test takes a -e option.
<lifeless> ah
<lifeless> well, I've ec2landed it
<lifeless> will forward you the epic failure, should there be one, tonight
<wgrant> Hopefully it won't be necessary!
<wgrant> Thanks.
<lifeless> also \o/ the site message is gone
<lifeless> hopefully we'll see less noddy bugs on lp itself.
<wgrant> Yep.
<lifeless> i have a reasonable query implementation in my head now
<lifeless> need to write it up
<lifeless> its away
#launchpad-dev 2010-11-27
<maxb> Can anyone disclose acceptably public bits of OOPS-1792H33, pertaining to the current conversation in #launchpad ?
<wallyworld__> maxb: what do you need to know?
<wallyworld__> maxb: looks like it was having a problem rendering the comments associated with a mp
<wgrant> jelmer: You say in bug #681974 that you flush, but you need to commit before any other process can see it.
<_mup_> Bug #681974: build processed with uploadprocessor while it is still set to BUILDING <Soyuz:Triaged by jelmer> <https://launchpad.net/bugs/681974>
<maxb> wallyworld__: Can you pastebin the traceback, unless you suspect there's private info there?
<wgrant> lifeless: Ugh, that ec2 run might have hung.
<wallyworld__> maxb: i'll have a look
<wgrant> canonical-config.txt in my local run exploded and hung, and the 600s timeout didn't operate.
<wgrant> Although it passes fine on its own... sigh.
<wallyworld__> maxb: https://pastebin.canonical.com/40201/
<lifeless> wgrant: no, it genuinely failed
<wgrant> lifeless: Huh. Around canonical-config.txt?
<wgrant> Ah, got it.
<wgrant> lifeless: It did hang.
<lifeless> wgrant:  ok
<wgrant> It just also had some proper failures.
<lifeless> wgrant: I didn't look :)
<lifeless> wgrant: I meant 'I got mail'
<wgrant> Thanks.
<wgrant> I'm rerunning it locally after fixing canonical-config.txt.
<lifeless> bah, sqlalchemy mappers don't do what I want :(
<lifeless> These class attributes exist as Python descriptors, and define instrumentation for the mapped class. The functionality of this instrumentation is very rich and includes the ability to track modifications and automatically load new data from the database when needed.
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       99 / 4871  Archive:+index
<lifeless>       75 /  256  BugTask:+index
<lifeless>       13 /  302  Distribution:+bugs
<lifeless>       10 /  347  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        8 /  122  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>        7 /  156  POFile:+translate
<lifeless>        5 /   96  Question:+index
<lifeless>        5 /   14  DistroSeriesLanguage:+index
<lifeless>        5 /    5  ProjectGroup:+milestones
<lifeless>        5 /    0  DistributionSourcePackage:+filebug-show-similar
<lifeless> wgrant: tell me when to pull :P
<lifeless> wgrant: so, how did you go?
<wgrant> lifeless: Rerunning the test suite, found a couple more failures.
<wgrant> Still going.
<wgrant> Just installed a VM in which to run the test suite, so I can develop in parallel.
<wgrant> Yay for now having lots of RAM.;
<lifeless> wgrant: and with this, you can run tests in parallel ;)
<lifeless> how many GB of memory do yiou have?
<wgrant> lifeless: Well, yes, that's why I'm now so interested in this stuff :P
<wgrant> 8GiB.
<lifeless> cool
<wgrant> lifeless: Quite a few test failures have shown up now that canonical-config.txt is fixed to not obliterate the config. :(
<wgrant> Most seem to be script-related, though.
<lifeless> thank you for pulling on tihs
<wgrant> No more testrunner explosions, so it looks like it was only canonical-config.txt that was being really bad.
<wgrant> I guess once this is fixed, the librarian stuff should mostly fall into place.
<lifeless> wgrant: well, I hope so :)
<lifeless> wgrant: I'm not aware of any actual bugs in the librarian branch
<wgrant> Well, all the scripts will be using the right config.
<lifeless> wgrant: any luck?
<wgrant> lifeless: Been doing other stuff, but looking at various isolation issues now.
<lifeless> if you want to bounce them around feel free
<afuentes> Is there a page in launchpad with stats about launchpad?
<afuentes> like more active projects?
<lifeless> jml: lp.testing.monkey_patchs looks redundant with testtools patch to me
<lifeless> jml: does it to you?
<jml> lifeless_: maybe. will look into it soon.
 * jml gotta run
<lifeless> wallyworld: http://docs.jboss.org/hibernate/core/3.3/reference/en/html/queryhql.html - you've worked with this in detail, right ?
 * wgrant stabs Windmill.
<lifeless> good morning to you too
<wgrant> Morning.
<wgrant> A few test runs later, I've fixed another couple of issues, seem to be down to fourish real failures. But Windmill hung overnight so I don't know for sure.
<wgrant> And now the testrunner won't wake up.
<lifeless> :(
<wgrant> lifeless: What was your intention with BaseLayer.appserver_config? In some places it's used as a method, in others an attribute. In some it needs to contain a ConfigFixture, in others a CanonicalConfig.
<wgrant> For now I've moved the fixture to appserver_config_fixture, and things seem to work a bit better.
<lifeless> mm
<lifeless> wgrant: its meant ot be a fixture and never a CanonicalConfig
<lifeless> wgrant: I think you're confused by LayerProcessController
<lifeless> oh.. I see
<wgrant> Yes... the method creates a CanonicalConfig.
<wgrant> And appserver_root_url assumes that it is.
<lifeless> this is a conflict with devel
<wgrant> Ahh.
<lifeless> devel added appserver_config incompatibly.
<lifeless> I would move the appserver_config method out of the way
<lifeless> or do config -> config_fixture
<lifeless> as well as appserver_config -> appserver_config_fixture
<lifeless> they are paralleled
<wgrant> Right, I failed to check the origin of the method. Probably should have.
<wgrant> I've already renamed it to appserver_config_fixture, so I'll do it for config_fixture.
<wgrant> Thanks.
<lifeless> note the helper methods that set attributes
<wgrant> I'm currently setting appserver_config_name whenever the fixture is set. Is that necessary, do you think?
<lifeless> wgrant: thats what make_config does
<wgrant> lifeless: It sets the fixture, not the name.
<lifeless> oh right
<wgrant> The new method needs the name.
<lifeless> seems reasonable offhand
<lifeless> note that the config_name may be set without a config fixture for the persistent test helper case
<wgrant> Right.
<wgrant> Apart from occasioanl Windmill hangs, all the layers seem to start now.
<lifeless> cool
<lifeless> I'd love to remove the persistent helper case
<lifeless> but I think things are still a little slow for that
<wgrant> lifeless: Why'd you remove the DB setup from _runAppServer (r11743)? That causes AppServerLayer to explode when the test before its starting committed, because the DB doesn't exist.
<wgrant> eg 'bin/test -1cvvvt blacklist_not_public -t TestBugScaling.test_messages_query_counts_constant'
<lifeless> 'before it' ?
<lifeless> the commit message makes sense to me
<wgrant> lifeless: If a test commits, testTearDown drops the DB.
<wgrant> lifeless: testSetUp will then set it up again.
<lifeless> yes
<wgrant> So if the last test in a layer commits, and AppServerLayer is started immediately afterwards, AppServerLayer will start an appserver without a DB.
<lifeless> what layer are you in
<lifeless> AppserverLayer.testSetUp should cater for that
<wgrant> AppServerLayer.setUp starts the appserver.t
<wgrant> testSetUp hasn't been called at that point.
<lifeless> so
<lifeless> to be clear
<lifeless> a test outside the layer causes the db to go
<lifeless> but the db layer is still 'live'
<wgrant> Exactly.
<lifeless> then the appserver layer is started
<lifeless> but the invariant of its higher layer being valid is false
<lifeless> sounds like a bug in the db layer
<lifeless>  / in the layer design
<wgrant> How can it know that a child is being set up?
<lifeless> this can break any other layer where the layer setup wants to use the db
<wgrant> (I've confirmed that calling testSetUp in the children's setUp fixes everything)
<lifeless> the invariant I'd look for is 'between tests, the db is available, so that child layers can build on it'
<wgrant> So perhaps testTearDown should recreate it. Hmm.
<wgrant> Or testSetUp could be called on layer changes.
<lifeless> so the reason I removed it is it was a layering violation
<wgrant> Right.
<lifeless> so
<lifeless> testSetup is slightly different
<lifeless> how about this
<lifeless> http://pastebin.com/z6wZCc9M
<lifeless> and then
<lifeless> http://pastebin.com/y1daKate
<lifeless> thats not quite right
<lifeless> it will do setUp twice per test which we don't want
<wgrant> Will it? How's it different from the case where it wasn't dropped in the first place?
<lifeless> resetting sequences is non-zero time.
<wgrant> Ah,t rue.
<lifeless> Pushing this:
<lifeless> http://pastebin.com/P2i1wbzc
<wgrant> I've pushed a bit more to mine that you might want to merge.
<lifeless> I haven't run the db layer tests yet. they may fail.
<wgrant> k, will try.
<lifeless> pushed lp:~lifeless/launchpad/databasefixture; whats your branch again ?
<wgrant> s/lifeless/wgrant/
<wgrant> Heh, you committed a syntax errorl
<wgrant> Missing colon on 697.
<lifeless> win
<lifeless> uncommitting ;)
<wgrant> pull my stuff and I'll get this working and commit.
<lifeless> pushed
<wgrant> Windmill tests are sloooooooooow.
<lifeless> :(
<wgrant> OTOH I think this run may be clean.
<lifeless> is that 'page loads are slow' or 'it polls rather than events' or '...' ?
<wgrant> Both.
<wgrant>  lp.bugs.windmill.tests.test_bug_inline_subscriber.TestInlineSubscribing.test_inline_subscriberNo handlers could be found for logger "lazr.smtptest"
<wgrant> That can't be right.
<wgrant>  (166.994 s)
<lifeless> one test ?
<wgrant> It seems so.
<lifeless> win
<wgrant> There's a disturbing number of wats.sleep(5000)s in that test.
<lifeless> frell
<lifeless> LOL https://bugs.launchpad.net/launchpad-foundations/+bug/5814/comments/2
<_mup_> Bug #5814: want to know breakdown of test run time by area of development <build-infrastructure> <Launchpad Foundations:Triaged by flacoste> <https://launchpad.net/bugs/5814>
<wgrant> Ahahaha.
<lifeless> yes, 9 minute test times being a problem.
<wgrant> Well, that was only part of the suite.
<lifeless> no
<lifeless> it was everything - just not sourcecode/*
<wgrant> Oh.
<lifeless> wgrant: still, good news about the branch
<wgrant> lifeless: 3/4 through the run, four failures which I've fixed. And your fix for the DB thing worked.
<wgrant> So we may be good.
<lifeless> wgrant: you're running everything ?
<wgrant> lifeless: Yes.
<wgrant> Hrmph.
<wgrant> test_db_naming_LP_TEST_INSTANCE_set fails.
<lifeless> bwah
#launchpad-dev 2010-11-28
<wgrant> Hmm, how many tests do we have?
<lifeless> how many would you like?
<wgrant> Since this run just finished in less than three hours.
<wgrant> 11369 tests.
<lifeless> nice
<wgrant> Not sure how long it should take on this machine, so I guess it could be right.
<lifeless> cleaner layering ?
<wgrant> Three errors left to fix, then could you throw it at EC2?
<lifeless> delighted to
<wgrant> lifeless: PgTestSetup only respects LP_TEST_INSTANCE if it matches the value in BaseLayer. But TestPgTestSetup is layerless, so BaseLayer is unconfigured.
<wgrant> I suppose I could just call BaseLayer.setUp.
<wgrant> Or mvoe TestPgTestSetup into BaseLayer...
<wgrant> Ah, it already calls BaseLayer.setUp in other places. I guess I'll do that.
<lifeless> ahm
<lifeless> I see
<lifeless> I the dynamic stuff is only part polished - that is, it works.
<wgrant> Yep.
<lifeless> but we should get rid of the support for static instances altogether (again, have to remove the persistent helper support)
<wgrant> Maybe once the librarian doesn't take 8 seconds to start.
<lifeless> zcml.
<lifeless> I keep saying I want to nuke it; I really am serious :)
<lifeless> well
<lifeless> I want to remove the global nature of it
<lifeless> we shouldn't pay to import all of lp to run 300 lines of code
<wgrant> Sure.
<wgrant> lifeless: Why is LP_TEST_INSTANCE an envvar?
<wgrant> It presumably won't be crossing process boundaries...
<lifeless> helpers
<wgrant> Hm?
<lifeless> librarian etc
<lifeless> gnerally we'll want to set a config section up
<wgrant> Don't they use LPCONFIG?
<lifeless> but it can be useful to introspect
<lifeless> also
<lifeless> the reinvoke logic
<lifeless> which reinvokes the runner
<wgrant> So, at the moment PgTestSetup respects LP_TEST_INSTANCE. But it only does that when BaseLayer is set up, and BaseLayer sets LP_TEST_INSTANCE to the PID.
<wgrant> So the two tests that PgTestSetup respects the var are a little wrong.
<lifeless> hang on
<lifeless> why do you say 'But it only does that when BaseLayer is set up,'
<lifeless> thats now how I read the code in lib/canonical/ftests/pgsql.py
<lifeless> s/now/not/
<wgrant> In PgTestSetup.__init__, the end of the PgTestSetup.dynamic branch checks that the instance name matches BaseLayer.
<lifeless> to call config.reloadConfig()
<wgrant> If it matches, it uses the calculated name. If not, it reverts to the old one.
<jml> lifeless: I have some more pictures
<lifeless> jml: cool
<jml> lifeless: http://people.canonical.com/~jml/lpstats/good-and-bad.png
<jml> lifeless: http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png
<jml> and then different versions of the same data...
<wgrant> Ah, nice.
<jml> http://people.canonical.com/~jml/lpstats/how-many-tests-are-slow.png
<jml> http://people.canonical.com/~jml/lpstats/good-and-bad-by-time-bar.png
<jml> subunit-ls + openoffice
<lifeless> jml: do you have one with the area
<lifeless> test time per test * tests in that bucket
<jml> lifeless: yes, that's what good-and-bad-by-time is
<jml> lifeless: it's the sum of the times of tests in that bracket
<lifeless> cool
<lifeless> so, start on the 100s tests?!
<jml> lifeless: yeah, exactly
<jml> I also like that we actually have six orders of magnitude of test run times
<jml> the sole 1000s test is actually ~200s
<lifeless> you could drop the first 3 off without loss of usefulness
<jml> I guess I could have been more strictly logarithmic
<wgrant> jml: Not the 120s Windmill test that I just found?
<wgrant> Or does that count as 100s?
<jml> wgrant: subunit-ls
<jml> sorry...
<lifeless> wgrant: that may be the same one, machine variation.
<wgrant> lifeless: Indeed.
<wgrant> But most of it is sleeping.
<wgrant> So...
<jml> gzip -dc some-run.subunit.gz | subunit-ls --times | awk '{ print $2 " " $1 }' | sort -n | tail
<lifeless> jml: I'd like a leverage graph
<jml> wgrant: not-found-traversal.txt used to be mostly sleeping
<lifeless> jml: http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png scaled down by tests-per-bucket
<jml> as in, the more tests the wider the bar, say?
<lifeless> jml: or less height
<wgrant> jml: Yeah, test_inline_subscriber is indeed the longest test.
<lifeless> jml: 'time spent running those tests / count(those tests)'
<jml> right
<wgrant> Followed by two Soyuz ones /o\
<jml> was just typing that
<jml> lifeless: easy enough to do
<lifeless> I think that that would show the buckets with the most return - amount of time saved per test fixed in that bucket (on average)
<jml> I might also shift to be log(run_time) rather than n s.t. 10**(n-1) < run_time <= 10**n
<jml> int(log(run_time)), of course
<lifeless> wgrant: so I'm confused
<lifeless> wgrant: this branch was spread over 2?3?months
<jml> lifeless: btw
<lifeless> and more tz changes than is desirable.
<wgrant> Heh.
<jml> lifeless: my last two blog posts link to paste bins with subunit scripts
<lifeless> jml: yes, I saw ;)
<jml> lifeless: in my ideal world, someone would make them better and upstream them.
<wgrant> lifeless: Do you see what I mean about the BaseLayer thing?
<wgrant> At the end of the PgTestSetup.dynamic branch.
<lifeless> wgrant: I'm trying to remember *why*
<wgrant> Ahh.
<lifeless> wgrant: do what makes sense to you
<wgrant> Apart from that one there are only three Twisted-ish tests failing, all of which have an 'error' but no error message.
<lifeless> if all the tests pass, we can land this devilish branch
<wgrant> Which is a little disturbing.
<lifeless> and iterate later
<lifeless> its already had plenty of skew issues with trunk.
<wgrant> Yep.
<lifeless> jml: I think that would be ideal too
<wgrant> I'll just make it set up BaseLayer and use the PID instead of 'xx'.
<wgrant> For now;
<jml> lifeless: it'll be a little while before I move on to subunit hacking though. still have a chunk of testtools stuff that I want to get done
<jml> lifeless: I'm most of the way through a draft of testtools manual for test authors
<lifeless> cool
<jml> it's making me realize there's stuff I want to be able to say but can't yet
<wgrnat> Any ideas on how to debug http://paste.ubuntu.com/537289/? Three errors left in the branch, but no error message.
<wgrnat> I've not seen this before...
<lifeless> isn't that what henninge was seeing?
<lifeless> perhaps merge devel
<wgrnat> lifeless: Hah, true, it's on devel too.
<lifeless> still?
<wgrnat> Yeah.
<lifeless> :<
<lifeless> buildbot is clean
<wgrnat> As is Hudson.
<lifeless> ><
<wgrnat> But it's reproducible on three machines here.
<wgrnat> All Maverick, though.
<lifeless> >,
<lifeless> ok
<lifeless> push
<lifeless> me pull
<lifeless> we ec2
<wgrant> Pushing...
<wgrant> Done.
<lifeless> its with ec2 now
<wgrant> Thanks.
<lifeless> the lib/lp/registry/tests/test_mlists.py was fugly
<wgrant> Did you change it?
<lifeless> we shouldn't need to use popen directly  - or at least have a helper for passing the right LPCONFIG around.
<wgrant> This one was sort of special, since it needs the appserver config.
<lifeless> still
<wgrant> I removed the other LPCONFIG stuff.
<lifeless> no i didn't
<lifeless> librarian next?
<wgrant> Just waiting for it to merge.
<wgrant> Come on LP...
<lifeless> I love the way the upstream/downstream linking is coming together
<wgrant> lifeless: --parallel is rather silent.
<lifeless> wgrant: feel free to slap any reporter you want on it
<wgrant> lifeless: No idea whatsoever how to do that!
<lifeless> wgrant: the easiest way is --subunit | subunit2pyunit
<wgrant> It seems to be working, as postgres is going crazy.
<wgrant> Aha.
<wgrant> Thanks.
<lifeless> or --subunit | tribunal -
<jml> lifeless: http://people.canonical.com/~jml/lpstats/duration-over-count.png
<lifeless> jml: interesting
<lifeless> thanks
<jml> lifeless: np
<jml> basically, there are "only" ~250 tests that are between 10s and 100s long
<jml> whereas all the other categories are between ~800 and ~5000
<jml> if I could do SQL on subunit streams, I could probably get a per-layer thing fairly easily
<lifeless> that would be nice
<lifeless> though time-in-layer isn't very intereseting to me
<jml> I guess not
<jml> I'd like to break things down by package a bit more too
<jml> but that involves still more Python that I can't be bothered writing
 * jml off
<lifeless> ciao
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      115 / 5419  Archive:+index
<lifeless>       92 /  233  BugTask:+index
<lifeless>       23 /  244  Distribution:+bugs
<lifeless>       13 /  134  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>       11 /    4  Cve:+index
<lifeless>        7 /   12  DistroSeriesLanguage:+index
<lifeless>        6 /  282  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        6 /  188  POFile:+translate
<lifeless>        6 /    4  NullBugTask:+index
<lifeless>        6 /    3  Person:+bugs
<lifeless> wgrant: zomg
<lifeless> wgrant: pqm bailed, dunno why
<lifeless> have forwarded you the mail
<lifeless> oh, I do know why
<lifeless> it thinks a schema patch is there, I think
<lifeless> sent to db-devel instead
<wgrant> lifeless: Ah, of course, I had to change fti.py :/
<lifeless> and its landed
<wgrant> Some time ago.
<wgrant> And Hudson hasn't blown up yet.
<wgrant> lifeless: librarian branch seems to be fairly happy. Just a few hardcoded URLs to sort out.
<wgrant> And an odd localhost/launchpad.dev conflict in URL generation.
<lifeless> wgrant: cool
<lifeless> wgrant: jml cherrypicked an early subsection of my work to debug some issues with librarian tests
<jml> lifeless: oh yeah, I forgot to tell you about that
<nigelb> heh, only now I saw the lp repo, love the "<kiko> we are changing the world, one commit at a time"
 * maxb chuckles at discovering /srv/importd.launchpad.net/production/launchpad-rev-11887-sigh/ in logs :-)
<lifeless> jml: :)
<jml> lifeless: I've got to head out now, but I'd appreciate it if you could look at the new & updated testtools MPs
<lifeless> shure
<lifeless> jml: did you see
<lifeless> https://code.launchpad.net/~lifeless/launchpad/subunit/+merge/42023
<jml> lifeless: I did, but not the details.
<jml> lifeless: it's good stuff :)
<lifeless> jml: https://code.launchpad.net/~jml/testtools/unexpected-success-2/+merge/42050 looks wrong
<lifeless> oh
<lifeless> its an incremental diff
<lifeless> I think that needs to be called out
<lifeless> or something
<jml> it's still a beta feature. file bugs.
 * jml really does have to go
<jml> lifeless: thanks for looking
 * thumper calmly walks away from the laptop to avoid putting his fist through the screen
<thumper> mwhudson: do you have a minute to talk?
 * thumper lunchinates
<mwhudson> thumper: in general, yes
<StevenK> wgrant: O hai
<wgrant> StevenK: Hi.
<StevenK> wgrant: What do you think of https://code.edge.launchpad.net/~stevenk/launchpad/db-spph-ancestry/+merge/41943 ?
<wgrant> StevenK: 'The previous release of this source package.'
<wgrant> It's not always.
<wgrant> It could be a previous override.
<wgrant> Also, have you checked that we can sanely populate it, in all locations that create SPPHs?
<wgrant> Also, in the case of a copy, it will be in another archive.
<StevenK> wgrant: I've been checking that we can actually populate it for existing data
<wgrant> I'd quite like to add a creator field at the same time.
<wgrant> It would mean that we had actual proper copying histories.
<StevenK> wgrant: This would work much better if I actually knew all the places SPPHs were created :-)
<wgrant> StevenK: Search for all invocations of SourcePackagePublishingHistory and newSourcePackagePublishingHistory?
<lifeless> wgrant: so, reckon on fixing Archive:+index today
<wgrant> Ah, yeah, got distracted by parallel testing.
<lifeless> a good distraction to be sure
#launchpad-dev 2011-11-21
<mwhudson> huwshimi: http://t.co/O68m8Ur7 if you haven't seen this sort of thing before
<huwshimi> mwhudson: Oh, yeah, I saw that :)
<mwhudson> good good
<StevenK> lifeless: My bug supervisor change, I'd wager
<StevenK> lifeless: So, on one hand it's just one query, but on the other, I'd prefer to reduce it.
<StevenK> However, I think this branch is large enough.
<lifeless> StevenK: if its one Full Stop, its ok. If its one per row, aieee.
<poolie> i think his talk is insufficiently smug
<poolie> i know some people who could help though
<StevenK> lifeless: One full stop. The test case adds 5 bug tasks.
<lifeless> StevenK: they all have different supervisors etc?
<lifeless> StevenK: [paranoid]
<StevenK> I doubt it.
<poolie> wgrant, ok looking at jubany now
<wgrant> poolie: Thanks.
<lifeless> StevenK: it may not matter, make an educated assessment
<poolie> is banana the front end for both edge and lpnet?
<lifeless> StevenK: I'm just noting the possibility that the test isn't able to assess a regression here
<poolie> s/the/a
<lifeless> yes
<StevenK> poolie: One of.
<lifeless> actually its primary
<lifeless> normally all traffic goes through it
<StevenK> So nutmeg usually sees next to no traffic?
<lifeless> because haproxy doesn't know how to cooperate in a cluster
<lifeless> yeah
<lifeless> any traffic nutmeg gets it sends to banana
<lifeless> if banana goes down nutmegs backup path is direct to the appservers
<StevenK> And if nutmeg drops, banana doesn't care.
<lifeless> right
<wgrant> To clarify, clients send to both.
<wgrant> But nutmeg normally forwards to banana.
<wgrant> They should really be the frontends for all LP services.
<wgrant> But codehosting/PPA are a bit special at the moment, unfortunately.
<wgrant> And mailman.
<lifeless> nutmeg apache forwards to banana
<lifeless> banana apache forwards to banana haproxy
<lifeless> haproxy then does the same dance because we can't rely on apache
<poolie> i like the fact they're actually related
<poolie> anyhow, udd certainly does try to use lpnet
<poolie> restfulclient's api is so wacky perhaps it's not having the intended effect
<poolie> hm
<poolie> that is our ip though
<huwshimi> poolie: So, with this change to the bug portlet, the ordering has changed from alphabetical, to usage count. Would you add a test for that, or should I just delete these current (now mostly useless) tests?
<poolie> i would add a test for it
<poolie> it's probably not so much likely they will come back in the wrong order as that it will blow up
<huwshimi> poolie: Ok thanks
<poolie> a python test obviously
<poolie> wgrant, it certainly thinks it has an lpnet connection
<huwshimi> poolie: Yeah :)
<wgrant> poolie: Hmm. I wonder what's using edge, then.
<wgrant> That IP address looks like jubany, doesn't it?
<wgrant> Yes
<poolie> wgrant the apache log line you quoted is to api.l.n, not edge
<wgrant> Erm, really?
<wgrant> So it is.
<wgrant> I may have synced the wrong log. That would be bad......
<wgrant> AHEM
<wgrant> Yes
<wgrant> That was an lpnet log.
<wgrant> Sorry
<wgrant> Nothing from package-import in the actual edge logs.
<poolie> that's ok :)
<lifeless> wgrant: so, I nagged brian for nought?
<wgrant> lifeless: No, I found bdmurray in my correct analyses a couple of weeks ago. He's the second-top user on real edge
<lifeless> phew. tanks
<wgrant> didrocks 28614, bdmurray 10237, ubuntuqa 5671
<wgrant> Top three from yesterday
 * StevenK looks forward to edge.launchpad.net being NXDOMAIN
<lifeless> wgrant: want to mail didrocks ?
<wgrant> Once I have more data.
<wgrant> I think didrocks' thing is new, actually.
<wgrant> It's showing up every day now.
<wgrant> But wasn't there when I last checked a couple of weeks back, IIRC.
<lifeless> bbs, grocery shopping time
<poolie> hm
<poolie> i'm trying to get its tests to run in semi isolation
<poolie> hm, launchpad's TestCase now depends on having rabbit running?
<poolie> wgrant, https://code.launchpad.net/~mbp/launchpad-buildd/lpbuildd-tests/+merge/82837(small)
<poolie> wgrant, https://code.launchpad.net/~mbp/launchpad-buildd/lpbuildd-tests/+merge/82837   (it's pretty small)
<poolie> pht
<lifeless> poolie: TestCase? shouldn't.
<lifeless> poolie: specific tests? yes, absolutely
<wgrant> poolie: Any reason to continue bumping to a new version so rapidly?
<wgrant> poolie: Also, your wraping in debian/control's Depends line is sort of screwy.
<lifeless> mwhudson: is there a generic django facility for 'I need an odd bit of metadata like a timestamp for my whole project'
<lifeless> mwhudson: ?
<huwshimi> lifeless: Can you elaborate a little more about what you want to do with it?
<huwshimi> I'm just coming in on the conversation now, so I might be being unhelpful :)
<lifeless> huwshimi: that was the entire conversation
<lifeless> huwshimi: I have a single datetime I need to store for python-oops-tools
<lifeless> huwshimi: (that is lp-oops.canonical.com)
<lifeless> huwshimi: it records the point in time which pruning has been done to
<huwshimi> lifeless: And you just need a way of storing that timestamp?
<lifeless> and reading it back, yes
<huwshimi> lifeless: There isn't a generic global config table or anything of that sort. You could do it by setting up a simple model and could probably use the unique flag. I'm just checking.
<lifeless> yeah, I can do it manually
<lifeless> was just wondering if I could avoid that overhead
<huwshimi> lifeless: Actually, you might be able to use the site model
<huwshimi> lifeless: You might be able to hook something on to that
<lifeless> looks equivalently complex - http://stackoverflow.com/questions/2821702/how-do-you-extend-the-site-model-in-django
<huwshimi> lifeless: yeah, quite possibly
<huwshimi> lifeless: It's the only unique site wide table though
<huwshimi> lifeless: It might still be simplest to add another model that is unique to a Site to store the timestamp
<lifeless> huwshimi: I very much doubt that the oops app supports multiple sites as-it is
<lifeless> huwshimi: so I'm going to do the simplest thing I can, which is a model with one row
<huwshimi> lifeless: Sure, I just thought that would be a clean way to do it. That way you can just do Site.objects.get_current().my_timestamp
<huwshimi> A default site is created when you set up the django project
<lifeless> I see; it probably is pretty clean
<lifeless> however, I've gotten all my stuff setup in the mean time
<huwshimi> lifeless: Yeah, well doing it another way would be fine too
<huwshimi> lifeless: Oh actually it probably would have been Site.objects.get_current().configs.my_timestamp
<huwshimi> lifeless: It's just how I would have done it if I was setting up project wide configs
<mwhudson> lifeless: not sure i understand
 * mwhudson catches up
<mwhudson> lifeless: model with one row sounds easiest, tbh
<poolie> lifeless, yep does fairly clearly depend on rabbit
<poolie> will file a bug i guess
<lifeless> poolie: that doesn't make sense to me, as our unit tests would die horribly wouldn't they ? (the ones with no layer)
<poolie> maybe none of them have no layer?
<poolie> istr rabbit says it can't be torn down
<poolie> so things with no layer always have it accidentally running?
<poolie> anyhow, https://bugs.launchpad.net/launchpad/+bug/892938 has a simple demonstration
<_mup_> Bug #892938: lp.testing.TestCase inherently depends on rabbit <Launchpad itself:Triaged> < https://launchpad.net/bugs/892938 >
<lifeless> argh django. datetime should *always* have tzinfo. What are you *thinking*
<wgrant> Django has a lot of similar moments.
<mwhudson> lifeless: you assume much by mentioning thought
<mwhudson> lifeless: https://code.djangoproject.com/ticket/17062 is fun
<mwhudson> oh hey look, that's fixed apparently
<lifeless> hah
<mwhudson> nice job with the bug mail, trac
<lifeless> 19 hours ago
<poolie> wgrant, i don't know, do you think i should stay on the same debian version?
<poolie> it just seemed likely to cause less confusion if i did increment it
<poolie> especially as it's sometimes rebuilt and might lose the recipe version etc
<lifeless> mwhudson: thanks for the link !
<poolie> also, do you think i should not wrap the depends?
<poolie> or just wrap them more tidily
<wgrant> poolie: Wrap them more tidily.
<wgrant> I think we should bump the version for releases.
<wgrant> Not for every revision.
<poolie> what is a release in this context?
<poolie> every time we ask it to be deployed?
<lifeless> mwhudson: does delete() return the row count deleted ?
<poolie> anyhow the basic thing is ok with me
<mwhudson> lifeless: don't know
<poolie> i'm not sure when it will actually get deployed so i suppose i'm being optimistic
<poolie> any other comments on that patch?
<lifeless> hah, win
<lifeless>   File "/home/robertc/source/launchpad/oops-tools/working/parts/django/django/db/models/query.py", line 430, in delete
<lifeless>     "Cannot use 'limit' or 'offset' with delete."
<lifeless> AssertionError: Cannot use 'limit' or 'offset' with delete.
<wgrant> Is that surprising? I didn't think postgres supported that.
<wgrant> MySQL does on UPDATE, IIRC.
<StevenK> I thought LIMIT worked with DELETE?
<lifeless> no, the surprising thing is that django generated it in the query
<lifeless> and barfs on it
<poolie> :)
<lifeless> rather than e.g. doing a subselect to identify the rows to delete
<wgrant> removed: lib/canonical/launchpad/icing/style-3-0.css
<wgrant> It cannot be!
<StevenK> lifeless: And you're suprised Django is naive?
<StevenK> wgrant: Wait. jtv will accidently resurrect it via lint changes.
<wgrant> Heh
<Dr_Who> anyone around that might be able to help with increasing the size of a pap ?
<Dr_Who> err ppa I mean
<wgrant> Dr_Who: How urgent is it?
<huwshimi> wgrant: Did that just get merged?
<wgrant> huwshimi: Yep
<Dr_Who> well I'm trying to get a large set of packages built against libjpeg-turbo so if things continue well, we'll be able to replace libjpeg with libjpeg-turbo in precise
<Dr_Who> (it's about a 2x-4x perf boost for intel and armel)
<poolie> funnier in scotland
<Dr_Who> if tomorrowish was possible wgrant that'd be awesome
<wgrant> Dr_Who: Probably best to ask at https://answers.launchpad.net/launchpad/+addquestion
<Dr_Who> ok thanks
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<huwshimi> wgrant: Oh, so it did.
<lifeless> wgrant: https://code.launchpad.net/~lifeless/python-oops-tools/prune/+merge/82840
<wgrant> Grar, I hate tests that touch DB users in bad ways.
<wgrant> Like TestPersonSetMerge
<lifeless> wgrant: https://code.launchpad.net/~lifeless/python-oops-tools/bug-892917/+merge/82841
<wgrant> lifeless: How does oops-tools' prune differ from oops-datedir-repo, apart from having an implied path and DB-backed last prune date?
<wgrant> lifeless: As for the other branch, don't you need to change the current model too>?
<lifeless> wgrant: https://code.launchpad.net/~lifeless/python-oops-tools/bug-892914/+merge/82842
<lifeless> wgrant: yes, I do, will fix that
<lifeless> wgrant: oops-tools prune only prunes the db, doesn't touch disk.
<lifeless> wgrant: for two reasons; A) if it does delete everything, I wanted a Plan B. and B) two-phase commit is hard.
<lifeless> wgrant: so I just plan on us running both pruners.
<lifeless> wgrant: other than UI cruft like help text, the duplication is ~ 4 lines long.
<lifeless> wgrant: well below my generalise-it threshold
<poolie> lp is in testfix mode or something?
<wgrant> lifeless: Ah, it should probably say that.
<wgrant> lifeless: It currently says it deletes stuff from the repository.
<wgrant> poolie: No
<poolie> Commit message [[r=mbp] delete canonical.buildd, now its moved to launchpad-buildd] does not match commit_re [(?# Strong suggestion: use ``bzr lp-land`` or ``ec2 land``.  Manual submission notes: your submission message needs at least one of [r=...] [rs=...] or [release-critical=...] AND at least one of [no-qa] [incr] [rollback=...] [bug=...].)(?is)(?=(\s*\[[^\]]+\])*\s*\[(release-critical=[^\]]+|rs?=[^\]]+)\])(?=(\s*\[[^\]]+\])*\s*\[(incr|
<poolie> no-qa|bugs?=\d+(,\s*\d+)*|rollback=\d+)\])]
<poolie> huh
<wgrant> poolie: Needs a QA tag
<poolie> it's associated with a bug
<poolie> pht
<Noldorin> poolie, hi
<Noldorin> poolie, so, i'll definitely need to get a version of LP running locally?
<poolie> yeah that's a good idea
<poolie> dev.launchpad.net has scripts for how to do it
<poolie> it is mostly scripted but it may take a while
<poolie> or, rather, it's all scripted, it will take a while, it is possible but unlikely the scripts may get snagged and then we  can help you get them going again
<Noldorin> poolie, guess  i can always have a go at designing the source-controlled wiki feature i'm interested in too then :-)
<poolie> test_poppy failed again :/
<poolie> that would be great
<Noldorin> i discussed that with mgz or somehting
<Noldorin> a few weeks ago
<wgrant> poolie: Hm, never seen that.
<poolie> i was just starting to put in markdown support on the weekend
<wgrant> poolie: Apart from your email.
<Noldorin> poolie, oh awesome. in what area? using python-markdown i guess
<poolie> yes
<Noldorin> poolie, mind linking me to the appropiate bug so i can now when to begin my efforts? :-)
<lifeless> wgrant: 'repository'. But yes, I will change it to say 'database' to avoid confusion
<poolie> Noldorin, https://code.launchpad.net/~mbp/launchpad/391780-markdown/+merge/82832
<poolie> that is the markdown thing
<poolie> i suggest you get some smaller patches in first to warm up
<poolie> and get familiar with how it works
<Noldorin> poolie, yeah, like the download types thing?
<poolie> yeah
<huwshimi> I appear to have forgotten how to merge a working branch with our main branch. Can I just do "rocketfuel-get" and then "merge ../devel" from my branch?
<wgrant> huwshimi: That will work, yets.
<Noldorin> poolie, ooh so you've already finished it, jsut ready to mergew
<wgrant> yes
<lifeless> wgrant: tweaked help pushing now
<huwshimi> wgrant: Is that not the recommended way? Or am I reading too much into the way you said that?
<lifeless> wgrant: I'm going to cook dinner; I would like to land all three and deploy tonight, for hopefully obvious reasons
<poolie> i've finished a small part of it
<wgrant> lifeless: Have you also fixed the model lengths?
<poolie> lifeless, i will look at the mail bug now
<lifeless> wgrant: yes, pushed that an age ago
<wgrant> huwshimi: Some discourage rocketfuel-get, but I think it's fine.
<wgrant> lifeless: Thanks.
<poolie> are you suggesting that i broke it, or are you just keen to accept my offer to look at it?
<lifeless> wgrant: thank you for noticing
<lifeless> poolie: bit of column A, bit of column B
<huwshimi> wgrant: Ah right, it seems to work for me, but I don't do very complicated things
<lifeless> poolie: there is some guilt by association with that area of the code, but also just that you are familiar with it, which is a Good Thin
<lifeless> g
<lifeless> wgrant: our oopses compress to about 100K - thats using a 7MB as reference
<lifeless> wgrant: I propose to add gz compression to amqp2disk to tide us over to a server with more disk
<wgrant> lifeless: If we indeed turn rabbit on by Wednesday, that sounds like a good workaround.
<Noldorin> poolie, i'm most encouraged by how active this channel is :-)
<Noldorin> bedtime now
<Noldorin> but i'll have a go at that soon
<Noldorin> and poke a dev if necessary
 * wgrant renames reconnect_stores() to reconnect_stores_DONT_YOU_DARE_USE_THIS()
<Noldorin> night folks
<wgrant> Night Noldorin.
<Noldorin> bye
<poolie> happy hacking
<lifeless> wgrant: right, really off cooking. Can you ack you're reviewing those?
<wgrant> lifeless: Yu[
<wgrant> Yup
<lifeless> thanks.
<poolie> huwshimi, would you be so kind as to read https://code.launchpad.net/~mbp/launchpad/888353-microformats/+merge/82767 for me
<huwshimi> poolie: sure
 * huwshimi reading
<poolie> and https://code.launchpad.net/~mbp/launchpad/meta-description/+merge/82769
<poolie> could someone please try running lp.poppy.tests.test_twistedsftp.TestSFTPServer.test_file_creation on oneiric?
<huwshimi> poolie: I think that looks ok, but I don't really know enough about how google handles that stuff
<poolie> it's hard to tell for sure ahead of time
<poolie> until we see how they parse the actual pages
<poolie> do you think it will be any worse?
<StevenK>     self.assertEqual(os.stat(file_name).st_mode, 0100644)
<StevenK> AssertionError: 33204 != 33188
<StevenK> poolie: ^
<poolie> i thought so
<poolie> thanks StevenK
<huwshimi> poolie: Not sure how it could be
<wgrant> poolie, StevenK: Yeah, there are a couple of tests that break due to the insane default umask change.
<wgrant> I have a list somewhere.
<poolie> oh ok
<poolie> https://bugs.launchpad.net/launchpad/+bug/892955
<_mup_> Bug #892955: lp.poppy.tests.test_twistedsftp.TestSFTPServer.test_file_creation fails on oneiric <poppy> <spurious-test-failure> <twisted> <Launchpad itself:Triaged> < https://launchpad.net/bugs/892955 >
<poolie> do we actually care?
<wgrant> We will in a few months.
<wgrant> Until then we just curse Ubuntu for doing bad things.
<StevenK> wgrant: It's now 002?
<StevenK> It is.
<StevenK> Madness.
<StevenK> I thought we set the umask inside poppy?
<lifeless> poolie: re depends on rabbit - no, it doesn't
<lifeless> poolie: your environment claims rabbit is available, but it isn't
<poolie> what environment?
<wgrant> lifeless: But you want the oldest, not the first loaded.
<wgrant> Since AFAICT you use the real age everywhere else.
<wgrant> Not the loaded age./
<poolie> lifeless, oh i see
<poolie> but having configuration data for rabbit does not imply it's running
<ajmitch> Â±
<lifeless> wgrant: for pruning we want real not claimed
<lifeless> poolie: yes it does :)
<lifeless> poolie: if its configured, and not running, thats an environmental issue
 * wgrant shops for a reviewer for https://code.launchpad.net/~wgrant/launchpad/reconnect_stores-is-a-horrible-person/+merge/82844
<wgrant> +81/-188
<lifeless> lookin
<wgrant> lifeless: Thanks.
<huwshimi> poolie: From line 59 is my test I've added, does this look ok? https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
<poolie> lifeless, so re "environmental problems" i just want to be able to run the tests within launchpad's buildout environment
<poolie> that seems like a reasonable thing to want
<lifeless> poolie: which tests?
<poolie> the buildd tests
<poolie> post splitout
<lifeless> they should be independent
<lifeless> it Must Not import from launchpad
<lifeless> and launchpad Must Not import fro it
<lifeless> *from* it.
<poolie> pht
<lifeless> I don't know what pht means here. But LP is a horrendous environment with a deep stack of assumptions in its code and structure - things like the config system are merely part of it.
<lifeless> I don't see it as *at all* interesting to let importing things from LP work for other projects: I see it as interesting to extract code from LP such that that code can be reused sensibly.
<poolie> i'm just trying to gradually unpick the horro
<poolie> anyhow
<poolie> with one trivial exception, they do no longer import from lp
<poolie> i will fix that now
<poolie> i guess it needs its own buildout config for things it does depend upon
<lifeless> I know, and I'm glad you're doing that - it doesn't mean that using an LP specific test helper in a misconfigured environment is reaosnable
<lifeless> (yes, 'development' is misconfigured - deliberately so, until make run or bin/run are executed)
<poolie> ok, whatever
<poolie> if using it outside of bin/test is not supported, i'll close the bug
<lifeless> its not
<lifeless> fixing things so that tests really can run in isolation, get an automatic config etc would be wonderful - but I think you'd need to start at the other end of the chain : the symptom you ran into is in ok code, its the rest of the stack that is flawed.
<lifeless> amongst other things, your developer db would be wiped, etc, running with the 'development' config.
<poolie> yeah
<poolie> i guess bzr has or had problems where test classes counted on setup being done by bzr selftest
<poolie> i think those are bugs
<poolie> but, perhaps this is not a good place to start
<poolie> it's closed
<lifeless> certainly, and I'd totally accept 'LP Tests don't work outside of bin/test' as a bug - and be happy if you want to start on that yak shaving exercise
<lifeless> its deep and nasty; both jml and I have spent weeks stabbing at it and making various partial improvements
<lifeless> for instance, currently you must have a test runner that can reinvoke itself because there is no guarantee for an arbitrary testA, testB pair, that testA can run in the same process as testB
<poolie> ok
<poolie> soryr for the misunderstanding
<poolie> lifeless, i'm happy to say launchpad-buildd no longer imports launchpad at all
<poolie> there are still tests in launchpad that import lpbuildd
<poolie> production code does not
<poolie> lifeless, could you read https://code.launchpad.net/~mbp/launchpad/892427-service-failure/+merge/82766 for me?
<poolie> allenap, hi?
<poolie> allenap, hi?
<rvba> Hello poolie, allenap should be here in about 1h.
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba | Critical bugtasks: 292
<poolie> ooh
<poolie> rvba, how about some reviews?
<rvba> poolie: yep, I just saw that you have a few branches up for review ;)
<poolie> :)
<poolie> i felt inspired on the weekend
<rvba> I see that!
<poolie> by how bad some search results were, etc
<poolie> rvba, thanks
<rvba> np
<poolie> i had an idea there was meant to be no space after the chevron but i can't see anything that says so
<allenap> Morning poolie, sorry I forgot to change the topic :-/
<poolie> np
<poolie> was just going to say i ended up finishing off the rabbit startup failure patch you suggested
<poolie> raphael reviewed it
<mrevell> Hello
<rvba> Morning mrevell.
<wgrant> rvba: Was it you who was working on preloading specific jobs for Builder:+history?
<poolie> mrevell, i think the only really scary bit of https://bugs.launchpad.net/launchpad/+bug/391780
<_mup_> Bug #391780: Support Markdown "stack overflow" style hyperlinks markup <feature> <lp-foundations> <markdown> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/391780 >
<poolie> is hooking it into the edit widget
<rvba> wgrant: yep
<poolie> and it may not be hard
<wgrant> rvba: Are you aware of the large OOPS regression?
<rvba> wgrant: oops, noâ¦
<poolie> and i guess see if lifeless has any real objection
<wgrant> 2285 InconsistentBuildFarmJobError: Could not find all the related specific jobs. Bug: https://launchpad.net/bugs/891600
<_mup_> Bug #891600: InconsistentBuildFarmJobError: Could not find all the related specific jobs.  <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/891600 >
<wgrant>       13 https%3A//launchpad.net/ubuntu/karmic/%2Bsource/libatasmart/%2Bbuilds (SourcePackage:+builds)
<wgrant>         OOPS-0307a9c4a7c9430eeb6a7952a3fb138c, OOPS-1e170c1660c5e4b23a4fa8db662b267c, OOPS-2ae836060fa3b0c90416132f78e4c6cc, OOPS-2d3458326e2328888803cce42c71ae2f, OOPS-4220ac65deb478aa8b47f4ddc65290d9
<rvba> wgrant: sorry, was otp.
<rvba> wgrant: I'm looking into it right now.
<rvba> wgrant: I remember wondering about this but I made the new code do what the old code did: raise an InconsistentBuildFarmJobError if no specific job can be foundâ¦ I really wonder how that's a problem now.
<poolie> how hard/easy is it to override a utility for the duration of a test?
<wgrant> rvba: Hmm
<rvba> wgrant: do you think this is so bad that we need a cowboy to stop the bleeding?
<wgrant> rvba: No, it's been around for a few days.
<wgrant> poolie: lp.testing.fixtures has ZopeAdapter and ZopeViewReplacement fixtures. ZopeUtility would be similar, I imagine.
<poolie> yeah, i think this will work
<poolie> surely this is the whole point of the complicated dependency injection thing?
<poolie> s//a
<wgrant> Hah
<poolie> it's a religious thing?
<wgrant> poolie: In its defence, raw_sendmail does have a comment saying that it's untested.
<poolie> the court finds this argument unpersuasive
<rvba> wgrant: IÂ think I'll need your help to pin point where the problem isâ¦ having this InconsistentBuildFarmJobError means that one of the specific jobs could not be found: is that possible? I mean I'm trying to understand if the bug is in the way IÂ fetch the specific jobs or if the condition to raise this error is too strict.
<wgrant> rvba: That is a little odd.
<wgrant> rvba: I note in the query log of https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-0307a9c4a7c9430eeb6a7952a3fb138c that the first few BFJ IDs in query 138 are duplicated.
<rvba> Right.
<wgrant> Which would probably do it.
<wgrant> Since it's not going to get duplicates back.
<wgrant> So the counts will differ.
<rvba> Very true. I wonder where this duplication comes from
 * wgrant scratches head
<wgrant> Ah, rendering error
 * wgrant kicks WebKit.
<wgrant> rvba: It's also already doine tonnes of BFJ/PB queries before it gets to that.
<rvba> Yes, I just saw that.
<wgrant> Which is, I must say, a little bit odd.
<wgrant> Huh.
<wgrant> Ah, nevermind.
<poolie> wgrant, ok, well, kinda fixed
<poolie> i added a test but it doesn't fail and it's not obvious how it would
<poolie> but i think you and i should stop soon
<wgrant> Heh
<wgrant> rvba: It's reproducible on DF, which is nice.
 * wgrant pokes around in the DB.
<rvba> Thanks wgrant.
<wgrant>  distro_arch_series |   id    | package_build | source_package_release
<wgrant> --------------------+---------+---------------+------------------------
<wgrant>                  95 | 1739878 |       1739878 |                 664115
<wgrant>                  95 | 1739878 |       1739878 |                 664115
<wgrant>                  94 | 1739877 |       1739877 |                 664115
<wgrant>                  94 | 1739877 |       1739877 |                 664115
<wgrant>                  93 | 1739876 |       1739876 |                 664115
<wgrant>                  93 | 1739876 |       1739876 |                 664115
<wgrant> etc
<wgrant> It is indeed duplicated
<wgrant> wtfwtf
<wgrant> Ah
<wgrant> There's an SPPH join in there.
<wgrant> That's really interesting.
<wgrant> It should be causing builds to actually show up twice when they've been copied.
<wgrant> Did you change the main query at all?
<wgrant> Query 12
<rvba> No.
<wgrant> So this may have revealed a long-standing bug.
<wgrant> Probably in SourcePackage.getBuildRecords
<wgrant> However, you should probably fix getSpecificJobs to work on distinct IDs.
<rvba> Yep.
<wgrant> Possibly making it crash if it's given a non-distinct set.
<rvba> Will do, thanks again for the help wgrant.
<wgrant> Yeah, SourcePackage.getBuildRecords needs a DISTINCT.
<poolie> wgrant, i did something towards https://bugs.launchpad.net/launchpad/+bug/885972 but i can't really understand why it's failing
<_mup_> Bug #885972: raw_sendmail creates TimedActions with invalid detail <oops-infrastructure> <regression> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/885972 >
<wgrant> rvba: Confirmed from the oops-tools DB that only SourcePackage:+builds is affected.
<wgrant> poolie: Why what's failing?
<poolie> it doesn't fail for me
<poolie> i don't know why it's failing for you
<rvba> wgrant: great, I'll finish what I'm doing right now and then work on a fix for that right away.
<wgrant> poolie: It doesn't crash.
<poolie> do you have a traceback or repro instructions?
<poolie> oh it's just lost
<wgrant> poolie: It just generates a bad OOPS.
<wgrant> Which crashes old versions of oops-tools, and lacks the detail that was intended.
<wgrant> rvba: Thanks.
<poolie> i wonder if this is a 2.4ism
<rvba> Thank *you* wgrant ;)
<wgrant> poolie: What's a 2.4ism?
<wgrant> poolie: This worked before because our RFC822 encoder stringified everything.
<wgrant> json/bson do not.
<wgrant> json crashes, bson silently omits
<poolie> afaics message['subject'] is a string
<wgrant> It's an email.header.Header
<wgrant> Its repr and str are a string
<wgrant> So it looks like a string
<wgrant> until you type() it
<poolie> nup, when i run it, i really get a string
<poolie> that's why i was wondering what path it was following to get there
<wgrant> Are you on 2.6 or 2.7?
<poolie> i am, that's why i said maybe a 2.4 thing
<wgrant> Oh.
<wgrant> Prod has been 2.6 for 18 months now.
<poolie> python 2.4 that is
<poolie> oh
<wgrant> So it could be a 2.6 vs 2.7, nothing to do with 2.4
<poolie> i will try it in ec2
<poolie> it's not documented as changed in 2.7
<poolie> wgrant, anyhow if you can give me a clue how to reproduce the failure i will have another go
<poolie> not nececssarily tonight
<rvba> wgrant: do you think I should mimic what setupCompleteBuilds did before: i.e. keep duplicates jobs?
<rvba> wgrant: I think I should not mess up with the batch size.
<wgrant> rvba: I'd preserve the dupes, I think.
<wgrant> Then fix the duping issue separately.
<rvba> Agreed.
<rick_h_> morning all
<rvba> wgrant: If you're still up for a quick review (otherwise I'll grab benji later): https://code.launchpad.net/~rvb/launchpad/relatedjobs-bug-891600/+merge/82870
<rick_h_> poolie: ping
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba, benji | Critical bugtasks: 292
<rvba> Morning benji.
<benji> morning rvba; how goes it?
<rvba> benji: Great, I hope you've had a nice weekend because you need to be ready for some reviewing action :)
<benji> heh
<rvba> benji: I'm gonna grab a late lunch then finish allenap's review when I'm back.
<benji> k, I'll take a look at the reviews in the queue
<deryck> rick_h_, https://dev.launchpad.net/JavaScriptReviewNotes
<deryck> abentley, ^^
<sinzui> My computer does not like the latest kernel update. If I know updating would be unstable, I would be on precise now
<mrevell> Hey deryck, how are we looking for taking custom bug listings into beta tomorrow?
<deryck> mrevell, It will be Wednesday.  Sorry.
<mrevell> No problem; I'd rather it were right than rushed.
<deryck> mrevell, agreed.  I'll send you an email by my EOD to let you know if we're still on track, but I feel good about hitting Wed.
<mrevell> Cool.
<abentley> sinzui: I have a request for a mailing list archive dump.  Who knows about those things?
<sinzui> losas
<sinzui> abentley, we ask a losa to send the mbox to the user if the user is a member of the team
<abentley> sinzui: Okay.
<sinzui> abentley, we restrict to team members because the mbox can contain deleted messages
<abentley> sinzui: I see.
<gary_poster> mars, hi.  Is https://bugs.launchpad.net/launchpad/+bug/663923 still active for you?  I'm probably not going to take it immediately, but it's on LP's escalated bug list, so it would be good to have a current status on it.
<_mup_> Bug #663923: Cannot view list archive of private team <escalated> <mailing-lists> <ml-archive-sucks> <regression> <Apache OpenID:New> <Launchpad itself:In Progress by mars> < https://launchpad.net/bugs/663923 >
<rick_h_> I just got that this morning gary_poster
<rick_h_> I was trying to view the archive of a list I got onto and log out/back in/etc couldn't get it to play nice
<rick_h_> heh, the same group actually
<gary_poster> rick_h_, :-) and :-/
<sinzui> jcsackett,  I think you QAed your branch that is in the kanban landing column.
<jcsackett> sinzui: you are correct; i have moved the card.
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugtasks: 292
 * deryck goes offline for lunch, back soon
<mars> gary_poster, Hi, that bug is still active but on hold for the moment.  I do not have the resources to work on it right now, but stuartm is trying to resolve that.
<lifeless> allenap: if you're still around, I would like to know more details on how scriptactivity is flawed
<rick_h_> deryck: is the inline editor in use? It looks like it's supposed to be in the merge proposal ui for the description, but not seeing it work out that way
<deryck> rick_h_, yeah, it is.  There are Zope widgets that spit out the html/css for the js widgets.
<deryck> rick_h_, let me find the file to point you at....
<rick_h_> yea, saw them in lazr stuff
<rick_h_> app/browser/lazrjs.py?
<deryck> rick_h_, yeah.  And the doctest helps understand it too:  lib/lp/app/doc/lazr-js-widgets.txt
<rick_h_> thanks, will head there next
<gary_poster> mars, ok, cool.  Thanks for update.  Could you mention status on bug, maybe, if you have not already?
<dobey> i'll try this over here instead;
<dobey> you guys know what would be awesome?
<dobey> having private attachments on public bugs :)
<rick_h_> deryck: thanks, found it
<deryck> rick_h_, cool!
<sinzui> dobey, bug 512085
<_mup_> Bug #512085: Secrecy settings for attachments <lp-bugs> <privacy> <Launchpad itself:Triaged> < https://launchpad.net/bugs/512085 >
<dobey> sinzui: ooh, thanks!
<flacoste> bac: i've marked bug #891641 as untestable
<_mup_> Bug #891641: Use of random.seed(n) in tests needs cleanup <qa-untestable> <Launchpad itself:Fix Committed by bac> < https://launchpad.net/bugs/891641 >
<flacoste> let me know if i was wrong
<lifeless> sinzui is the master of bugs
<bac> flacoste: yes, tahnks
<sinzui> I am? Do I have triage all 6000 again?
<deryck> flacoste, abentley has his qa done.  just fyi.
<sinzui> My wife and daughter are now looking for house in regions with unsafe water, earthquakes, flooding, mud slides, alligators, and no Ikea
 * sinzui stops work to do emergency presentation about when to live
<sinzui> In case you are wondering, I do have an presentation prepared in case of just such an emergency.
<dobey> about *when* to live?
<sinzui> s/when/where/
<dobey> oh. so no exploding universe at the end?
<dobey> disappointing :)
<sinzui> My universe will explode if I have to live near alligators. They are first on my list of things to make extinct.
<dobey> sinzui: well that's not a bad thing. they are tasty, and make great shoes
<sinzui> exactly. We can make the world safe, and feed the people, and look great.
<dobey> i want a pet alligator
<dobey> but they are illegal in VA :(
<nigelb> dobey: ...
<dobey> nigelb: yes?
<nigelb> dobey: I'm just o_O about the pet alligator line.
<dobey> haha
<sinzui> That did not go well. I failed to take into account that my family are geographically impaired. They do not believe that Maryland and Virginia and technically part of the South
<sinzui> BTW, the video out on the macbook air just works
<dobey> without Virginia, "The South" doesn't exist
<dobey> sweet tea, pulled pork, and fried chicken == the south
<huwshimi> I have a merge proposal that I merged with devel on, but now it's showing the diff as empty: https://code.launchpad.net/~huwshimi/launchpad/style-removal-one/+merge/82096
<sinzui> I live in Alexandria. A Northern enclave with a statue to honour the confederate dead
<huwshimi> Is this a known bug?
<dobey> now, Florida on the other hand, is not part of the south
<sinzui> dobey, EXACTLY
<rick_h_> heh, I heard more southen accents in VA than I ever did in OK
<sinzui> They do not believe me
<rick_h_> they must have courses on twang
<dobey> OK?
 * rick_h_ has family in VA we visit every year
<dobey> OK is not the south
<rick_h_> Oklahoma
<dobey> OK is the plains
<sinzui> Oklahoma in midwest, fly-over-country
<rick_h_> well I was 30min from the TX border
<rick_h_> so I got the worst of both worlds
<dobey> TX is also not the south; it's just exas
<dobey> err, just texas
<rick_h_> hmm, I think if you visit you might keep that to yourself :)
<sinzui> rick_h_, Yeah, I think suicide is a viable option in that case
<dobey> or excess; however you prefer to pronounce it
<rick_h_> moving to VA sinzui ?
<dobey> he's in va
<dobey> well, DC
<dobey> but the VA side
<rick_h_> gotcha
<rick_h_> my family's down charlottesville way
<sinzui> rick_h_, I can barely pay bills with 3 children. I think I need to move
<rick_h_> live on both sides of the mountains, always fun driving for visiting
<rick_h_> sinzui: heh, why I've stopped at 1 :)
<dobey> sinzui: heh; well, alexandria is probably the wrong place to be; though the schools are good
<rick_h_> I don't know, I think keeping away from DC is just good judgement
<sinzui> I thought my argument to live within 1 hour of an Ikea and an airport would be the point of contention.
<dobey> but that's to be expected given the average income in Alexandria
<rick_h_> what if that stuff spreads out there?
<rick_h_> shush on ikea
 * rick_h_ grumbled fake wood crappy crap
<dobey> you could just build furniture out of legos. same thing
<rick_h_> legos at least have some sturdiness to them when assembed right
<dobey> legos might be sturdier though
<sinzui> My son has already done that
<dobey> heh
<sinzui> I think a bed of nails would have been more comfortable
<rick_h_> my other hobby is woodworking, no have lots of bad things to say on ikea
<rick_h_> /no/so
<benji> lego blocks about 10X the normal size for furniture building would appeal to me
<benji> ...made from hardwood and stained
<sinzui> rick_h_, dobey, benji, http://www.youtube.com/watch?v=X-Kf-R1RKNk
<benji> now I just need an army of slaves to build my lego house for me
<sinzui> I have just solved the unity dash window ordering problem. I can fix it by plugging, then unpluging an external onito
<sinzui> monitor
<sinzui> rick_h_, You can feel smug to be in the same company are David Mitchell http://www.youtube.com/watch?v=X-Kf-R1RKNk
<rick_h_> I'll see if I can work on my smugness
<benji> sinzui: is that the dash-comes-up-below-the-active-window-but-still-has-keyboard-focus-so-confuses-the-crap-out-of-me bug?
<sinzui> yes
<lifeless> bbs, hard disk rails have arrived
<flacoste> huwshimi: i think that's because your branch was merged, so it doesn't have a diff anymore
<huwshimi> flacoste: Ah right. Was a bit confusing
<bac> hey sinzui, got a second to talk about https://bugs.launchpad.net/launchpad/+bug/480123
<_mup_> Bug #480123: Milestone names/version should be unique to series <escalated> <linaro> <lp-registry> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/480123 >
<sinzui> bac: I do not at this time I am looking for  a quite room to have the purple standup
<sinzui> bac: maybe in 40 minutes?
<bac> sinzui: ok.  ping me when you're free
<sinzui> wallyworld, jcsackett, wgrant, I am having audio troubles.I can barely hear you all
<wallyworld> sinzui: do you want to speak, not sure if we can hear you
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<lifeless> allenap: thanks for the feedback
<lifeless> poolie_: can you put a featureflag around markdown ?
<lifeless> poolie_: (if there isn't one already)
<lifeless> poolie_: rvba's performance figures are a little concerning
<poolie_> i did
<poolie_> i'd like to talk to you about it briefly after this call
<lifeless> sure
<StevenK> sinzui, wgrant, wallyworld, jcsackett: http://pastebin.ubuntu.com/745424/
#launchpad-dev 2011-11-22
<poolie_> lifeless, well, on a happier note, can i get a kind of architecture review on markdown?
<lifeless> sure
<poolie> so
<poolie> it will be behind a flag
<poolie> istm it might be good to make the rendering be a timeline action?
<poolie> or would that be redundant with just profiling it
<lifeless> if its really 200ms per text
<poolie> it could possibly go in memcached but perhaps that could be done later
<lifeless> I think we'll have issues
<lifeless> ugh no, memcached is very much not the right hammer here
<lifeless> so structurally whats going on is that you're generating html out of some content, which is exactly what the template engine does
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugtasks: 292
<lifeless> we're then embedding that result literally in the larger template
<wgrant> Hmm? memcached would be reasonable here, I think.
<poolie> the caching could be done in the template
<poolie> s//controlled by
<poolie> if it really is 200ms that's almost infeasible
<lifeless> wgrant: one word: bug comments
<wgrant> lifeless: Myes?
<poolie> ow
<lifeless> wgrant: a situation where we timed out *because* we used memcache
<lifeless> wgrant: and a place where markdown is [eventually] desired
<wgrant> Right, but it was timing out because the pages sucked, and there was absolutely minimal benefit to using memcache there.
<wgrant> It would need testing, but I think it could be useful if implemented properly.
<lifeless> so we want to use memcache on things that are both expensive to generate and able to be generated once for all viewers
<wgrant> And if the page didn't suck already.
<lifeless> but more so we want the first hit to be fast
<poolie> wgrant, tangentially, re your email bug, i will add a test that forces an email.Header into it and check its handled correctly
<lifeless> so we either want preloading (with change updates) or we want rendering to be so fast that memcache communication overhead is slower.
<lifeless> poolie: you could timeline it yes; if there are questions over its performance thats a great way to ensure we have data.
<poolie> so if it's really that slow
<poolie> that might be a problem for applying it to all bug comments
<lifeless> poolie: as there seem to be questions around it, perhaps you want to pull on that thread a little first
<poolie> there is python-markdown2 which claims to be an unspecified amount faster
<lifeless> poolie: so that we don't have markdown for home pages and then get stuck on bug comments
<poolie> exactly
<lifeless> poolie: I'm fine with landing with a restrictive feature flag to find out early adopter issues
<poolie> right
<poolie> performance to some extent we can test offline
<poolie> there is also this thing that api clients will now see markdown in the results
<poolie> i think that's ok
<poolie> it's close enough to text
<poolie> if they eventually want to start rendering it, they can
<lifeless> so will folk not behind the feature flag
<poolie> right
<poolie> so people probably don't want to go too crazy til it's widespread
<lifeless> I think timelining it for now is a good idea
<lifeless> till the issues/concerns are put to rest
<poolie> oh hang on
<poolie> rvb's measurement is actually that it's 240Âµs.
<poolie> isn't it?
<poolie> yep
<poolie> that's not so bad then
<poolie> but, worth testing
<lifeless> no, its not
<lifeless> the timeit CLI reports time per loop
<lifeless> the timeit python API reports total time
<poolie> but each iteration of the loop formats it 1000 times
<lifeless> ah
<lifeless> let me have a play
<poolie> ./bin/py -m timeit -vv -s 's="a *b*\n" ; import markdown; md = markdown.Markdown(safe_mode="escape",extensions=[ "tables",])' 'md.convert(s);'
<poolie> 1000 loops, best of 3: 231.1 usec per loop
<poolie> ./bin/py -m timeit -vv -s 's="a *b*\n" *100; import markdown; md = markdown.Markdown(safe_mode="escape",extensions=[ "tables",])' 'md.convert(s);'
<poolie> 100 loops, best of 3: 7.816 msec per loop
<lifeless> btw, choosing the packaged version is irrelevant to LP: we need things on lucid, and eggs are much more reliable than debian packages (because of version migration)
<poolie> yep
<poolie> i have it as an egg
<poolie> was just using 'packaged' as a very rough metric for 'supported/popular/worthwhile'
<poolie> also kind of handy for things like this
<lifeless> ok, yes, <1 ms is good
<lifeless> don't timeline it
<lifeless> poolie: you know about bin/py ?
<lifeless> bin/py -m timeit ...
<poolie> see the lines i pasted?
<lifeless> heh, I do know
<lifeless> *now*
 * lifeless slaps his fingers
<poolie> :) is ok
<lifeless> the scaling on longer strings is a little worrying
<lifeless> we would have trouble at 8ms per bug comment, for instance
<poolie> ok so if we say a large comment has about 100 lines of text, it's about 28.69ms to convert it
<poolie> hm, with no actual markdown in it, 100 lines, 1.798ms
<lifeless> so 100 comments is 2.8 seconds, which is 50% of the budget for an entire load
<lifeless> anyhow, no Big Red Flags
<poolie> one 100 line code block, 0.4ms
<poolie> strangely
<lifeless> it needs some love I guess
<poolie> arguably typical 10 line comment, 0.3ms
<lifeless> it might have some pathological cases that can DOS us
<poolie> yeah
<lifeless> OTOH one laptop can DOS us, so not goig to worry about this yet
<poolie> especially because the parsing is regexps
<poolie> <lifeless> wgrant: one word: bug comments
<poolie> can i buy another word?
<lifeless> timeouts :)
<poolie> oh i see
<mwhudson> yeah
<mwhudson> was going to say that, i think
<poolie> this is the "you eventually fall off a cliff" case?
<lifeless> â¸thatâ¸
<mwhudson> github run pygments but clearly limit the highlighting of any segment to say 2s
<mwhudson> would be nice to have that sort of thing
<mwhudson> (also for pygments...)
<lifeless> poolie: no, it was that we spent more time in memcache chatter than in rendering the comments; we also retrieved them from the DB no matter what
<poolie> yes, that can hook in
<poolie> huh
<lifeless> poolie: very poor interactions between late evaluation and a chatty protocol
<poolie> so, the other approach is to do it on the client side
<lifeless> poolie: nowadays memcache is just turned off, and things are much faster.
<poolie> which possibly makes rendering the page actually easier on the server, if we can reliably detect to just send json not html
<wgrant> Someone want to review https://code.launchpad.net/~wgrant/launchpad/observer-merging/+merge/82952?
<StevenK> wgrant: Line 68 of the diff -- why do you bother pulling in store?
<wgrant> StevenK: Because I forgot to remove that line when I replaced the check just now.
<StevenK> Haha
<StevenK> wgrant: That's my only niggle, r=me
<wgrant> IAPGS.get only came into existence late last night.
<wgrant> So I previously queried directly.
<wgrant> Thanks.
<lifeless> is http://appbuntu.com a legit site or a spammer ?
<StevenK> lifeless: NXDOMAIN ?
<wgrant> WFM
<lifeless> StevenK: get a better DNS resolver ?
<StevenK> Oh, *buntu*, not *ubuntu*
<StevenK> Transcribe error
<lifeless> UI fail :P
<StevenK> Well, I didn't click the link, I typed it out from memory so I could whois it
<StevenK> And my memory fails
<StevenK> Often
 * StevenK waits for lifeless to say "Frequently"
<lifeless> there is a link to http://appbuntu.com/2011/04/mengapa-ubuntu-menggunakan-launchpad/ in the blog, marked as spam but it doesn't look like spam to me
<lifeless> trying to determine if it is / isn't
<lifeless> StevenK: I did but you've clearly forgotten :P
<StevenK> I was about to suggest wgrant, but I suspect wgrant is too young for Gilbert & Sullivan
<lifeless> there was also an andi comment, but francis has address that separately, so I just deleted it
<poolie> lifeless, memcached is just off across the board?
<poolie> in a flag i think?
<lifeless> more or less
<lifeless> I need to go through and rip out the inappropriate uses of it
<poolie> hm
<poolie> the rules ui could do with some love too
<lifeless> any new use needs rather deep thought about what its actually saving
<lifeless> there is currently one reasonable use of memcache in LP - the front page blog contents
<lifeless> the rest have deep flaws (due to the interaction between views, eager loading and late evaluation), as well as various cachability rule interactions
<poolie> do you have a rule of thumb for how long it takes to query it?
<lifeless> between 1 and 5ms
<lifeless> sometimes up to 12ms
<StevenK> lifeless: TBH, since we aren't using it, I feel we should cache the blog contents differently and rip it out
<lifeless> StevenK: I'm open to that.
<lifeless> StevenK: OTOH it is a reasonable cache engine and I wouldn't want to reinvent a cache just because we haven't got much use for it
<poolie> possibly comments could do it reasonably well
<poolie> i mean, markdown could use it
<poolie> if the rendering was independent of the request (including the user)\
<poolie> and keyed off the place the text comes from, eg comment N on bug M
<poolie> anyhow, later
<mwhudson> one of the problems with the tal:cache stuff (it seems to me) is that it doesn't let you use multi_get at all
<wgrant> mwhudson: I've been assuming that's the entire performance problem we saw, yes.
<wgrant> Making a thousand round-trips is not cheap, just like with postgres.
<poolie> mwhudson, meaning one api call that gets many results?
<poolie> s/api/network
<mwhudson> poolie: yes
<mwhudson> the usual 'don't do potato programming' sort of thing
<poolie> sure
<poolie> it seems like in theory that could be hooked in to tal
<poolie> depending on what it exposes
<wgrant> It would be nice if we could do multi-stage renders :/
<mwhudson> you'd have to defer rendering parts of the template, or walk the template twice or something wouldn't you?
<wgrant> But TAL doesn't make that easy.
<wgrant> So we would probably have to do something like the current preloading.
<wgrant> Where we work out what we need beforehand, and poke it all into the caches.
<huwshimi> poolie: Not sure if you saw my note about adding a test for the tag ordering (I had to rush offf shortly afterwards). Just wanted to check if it looked ok (line 59 onwards: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689)
<poolie> hi huw
<wgrant> Rather than the probably more sensible setup of partially rendering the template, finding all the markdown that needs to be rendered, pulling stuff out of the cache, rendering what's left, filling in the template.
<huwshimi> poolie: Hey
<poolie> huwshimi, can you look at / think about the markdown branch too
<poolie> i don't know if it especially needs anything from you
<huwshimi> poolie: No problems
<poolie> is that screenshot up to date?
<mwhudson> wgrant: do you know a template engine that lets you do that sort of thing?
<wgrant> mwhudson: No
<huwshimi> poolie: That screenshot is up to date, yes
<huwshimi> poolie: This is the branch: lp:~mbp/launchpad/391780-markdown ?
<poolie> yep
<poolie> the test looks good to me
<poolie> mm
<poolie> you could add a comment specling out the meaning
<poolie> ie that the most popular tags come firstn
<poolie> perhaps it's obvious
<huwshimi> poolie: No, you're right
<huwshimi> poolie: Thanks
<rick_h_> poolie: thanks for the link and update in the bug. That's a different place than I was looking.working with.
<rick_h_> I had been looking at https://bugs.launchpad.net/launchpad/+bug/814696 and the link "Diff: 45 lines..."
<_mup_> Bug #814696: Link to show inline diffs in merge proposals should be green <qa-ok> <trivial> <ui> <Launchpad itself:Fix Committed by rharding> < https://launchpad.net/bugs/814696 >
<lifeless> mwhudson: wgrant: poolie: also that we don't want to do DB lookups for stuff we're then going to get from memcache.
<lifeless> mwhudson: wgrant: poolie: the whole template-view interaction is brittle here
<mwhudson> yeah
<wgrant> Yeah.
<wgrant> Particularly since the BugMessages are actually subviews.
<mwhudson> i've been very suspicious of passing anything other than dumb data to templates
<wgrant> Doing a pipelined render would be much nicer, but no :(
<mwhudson> s/been/become/
<poolie> ok
<poolie> we can worry about it later
<poolie> or not
<poolie> :)
<poolie> huwshimi, i'm not a very reliable reviewer for the js stuff
<huwshimi> poolie: No problems. I spent quite a bit of time chatting to wallyworld about that anyway
<poolie> aside from that it looks good
<poolie> i'm keen to see the results live
<poolie> good on you for deleting those horrible doctests
<poolie> have a badge :)
<poolie> what is the actual rule, if any, for tal indentation?
<wgrant> Nothing special for TAL. You mean XML/HTML?
<poolie> yes
<rick_h_> huwshimi: any reason not to make the taglist stuff a module with ATTR, initialize, etc?
<wgrant> huwshimi: CSS is somewhat broken.
<wgrant> huwshimi: https://bugs.qastaging.launchpad.net/launchpad/+bug/1234
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<wgrant> huwshimi: status/importance are blue
<poolie> huwshimi, can i see your current theme things some time?
<huwshimi> rick_h_: I'm not quite sure what you mean, care to elaborate?
<huwshimi> wgrant: ick!
<huwshimi> poolie: Sure, I haven't had a chance to work on it recently, but I'll try and push out a first stage to show people soon
<rick_h_> huwshimi: ok so you declared a namespace, but normally we seem to be using YUI modules, extending base or some decendent of that. For instance, see lp/app/javascript/picker/picker.js
<lifeless> mwhudson: excellent, the meme is spreading
<rick_h_> see http://yuilibrary.com/yui/docs/base/ for the YUI docs of the pattern
<rick_h_> so you'd end up with more var tagcloud = new TagCloud({any: settings});
<rick_h_> it allows for others to extend and customize later, more OOP version of things
<huwshimi> rick_h_: No reason other than ignorance
<rick_h_> huwshimi: ok, sorry, a bit new. But if you wanted review I'd definetely say to head down that route.
<rick_h_> overlay for instance is a good example
<rick_h_> there's a base, and several extentions that are used
<rick_h_> extensions that is
<wgrant> huwshimi: Ah, the stylesheets are combined in the wrong order
<huwshimi> rick_h_: Thanks I'll take a look. This was mostly me pulling the code out of a template and adding a few things/getting it working again, so I didn't really put enough thought into what I was doing :)
<rick_h_> huwshimi: totally understand, been there done that
<wgrant> typography.css is included later, and it sets link colours that override colours.css
<huwshimi> rick_h_: But, I'll take another look
<wgrant> huwshimi: Should I roll this back?
<wgrant> Or do you want to fix quickly?
<rick_h_> awesome, let me know if you need a hand with anything and I'll be happy to help in the morning
<huwshimi> wgrant: I can fix quickly
<huwshimi> rick_h_: Thankyou
 * rick_h_ is heading towards bed here
<wgrant> huwshimi: Needs to be landed in the next three hours, so not completely urgent.
<huwshimi> wgrant: That's ok, I'm on it
<wgrant> Thanks.
<huwshimi> wgrant: What should I do with these changes? Commit to the same branch and lp-land?
<wgrant> huwshimi: Ideally customising the commit message in lp-land to something sensible, but yeah, that sounds reasonable.
<huwshimi> oh breakages
<wallyworld___> huwshimi: rick_h_: with the yui module vs namespace thing, what is used depend on the heritage of the code. if it came from lazr-js, then it tends to extend the yui component framework. if it was originally lp code, then it tended to be more procedural and just use functions scoped using namespaces
<poolie> lifeless,  is there a timeline matcher?
<lifeless> poolie: not really
<huwshimi> wallyworld___, rick_h_: in this case, I guess I would need to make it a little more generic, ie, not have a hardcoded url etc. right?
<lifeless> poolie: closest is the underlying matcher of BrowsesWithQueryCount which doesn't use the timeline, it uses a custom storm filter instead
<wallyworld___> huwshimi: rick_h_: so the preference moving forward is to use more of the yui component and widget framework for new stuff
<huwshimi> wallyworld___: Is the component still appropriate for this?
<wallyworld___> but for simple refactoring, it think "whatever works" is fine
<wallyworld___> huwshimi: so, we are talking about the bug tags stuff?
<huwshimi> wallyworld___: Yeah
<wgrant> -p 5432
<wgrant> Blah
<wallyworld___> huwshimi: personally, i think the way it was done was fine - it was refactoring what was there already and was a very simple bit of code
<huwshimi> wallyworld___: If you say that I'm not going to fix it :)
<huwshimi> Mostly cause I would like to finish with this bracnh
<huwshimi> but very happy to fix it if it means doing it right
<wallyworld___> huwshimi: i'm not sure what value it would add for the effort involved. but as a learning exercise....
<huwshimi> wallyworld___: Maybe I'll consider it a lesson learned for next time :)
<wallyworld___> huwshimi: sure. fwiw, as i said, i would have done it the same way, since it was refactoring existing code and was a simple change
<rick_h_> wallyworld___: thanks for the info
<rick_h_> the stuff I've been doing and seen most of is the YUI module stuff
<rick_h_> so it just struck me as odd when I peeked at it
<wallyworld___> rick_h_: yes, i suspect you've been exposed to either the new distro pages or the former standalone lazr-js library that we sucked into the lp code base
<wallyworld___> rick_h_: then there's all the other lp javascript, which is mainly the procedural stuff
<wallyworld___> rick_h_: it's a bit of a mess, but we do want to evolve to use the yui component and widget framework for significant new stuff
<rick_h_> yea, overlay, inline editor, etc.
<rick_h_> yea, I had heard that, which I +1 :)
<rick_h_> I'm a bit of a YUI fanboi so happy to be working with this stuff
<wallyworld___> rick_h_: yep, that's all the lazr-js stuff which was written to be a reusable js lib and formerly packaged separate to lp
<wallyworld___> rick_h_: then no-one else was using lazr-js and it was too hard to use separately so we just sucked in all the source code
<wallyworld___> into the lp tree
<rick_h_> ah, yea I saw some bzr log stuff for "merge in lazr"
<rick_h_> when I tried to chase some older history items for it
<wallyworld___> yeah, we did that at the last epic, in july
<huwshimi> 'bzr lp-land' errors with this: http://paste.ubuntu.com/745525/ any thoughts?
<wallyworld___> rick_h_: i'm having "fun" today trying to upgrade to yui 3.4.1, but there's errors thrown by the yui.js code. i may perhaps ping to tomorrow about it. very frustrating
<wallyworld___> huwshimi: haven't seen that before
<huwshimi> grrr
<lifeless> your gnome session isn't
<rick_h_> wallyworld___: yuck, devel is currently on .0 then?
<wallyworld___> rick_h_: use currently are using 3.3
<lifeless> I really regret not making a fuss about the desktop integration thing
<lifeless> we've had -lots- of fallout
<wallyworld___> s/use/we
<rick_h_> oh, that's right, deryck mentioned skipped 3.4 to jump to 3.5 1st qrt
<rick_h_> qtr that is
<wallyworld___> rick_h_: oh, i didn't know that. in that case i may drop it and stick with 3.3 till then
<rick_h_> yea, there's a release schedule for 3.5 and deryck mentioned that would be the migration target
<lifeless> huwshimi: are you logged into gnome?
<huwshimi> lifeless: That's strange, I'm not running this from ssh or anything
<wallyworld___> rick_h_: there's been a fairly major re-packaging in 3.3 -> 3.4
<rick_h_> so I'd not kill yourself on it
<lifeless> huwshimi: or ssh'd into the machine ?
<huwshimi> lifeless: Yeah
<lifeless> huh
<rick_h_> wallyworld___: yea, I was checking out their MVC stuff which moved a few things around
<lifeless> that is odd
<lifeless> did unity crash on you or something ?
<wallyworld___> rick_h_: the reason for doing it is that i wanted to start looking to use the mvc stuff
<huwshimi> lifeless: Things have been a bit weird today, maybe a reboot is in order
<rick_h_> wallyworld___: yea, I'm porting backbone to the YUI MVC on my bookmark app on the side
<wallyworld___> rick_h_: since we are embarking on a major new piece of development (the managing disclosure ui) and i wanted to look at using that
<huwshimi> brb
<rick_h_> wallyworld___: ah, yea it's definitely something to plan from the start from
<wallyworld___> rick_h_: so i'm still tempted to try to get 3.4.1 working
<rick_h_> to catch the new way of doing the events and such
<wallyworld___> yep
<wallyworld___> rick_h_: so you used 3.4.1?
<rick_h_> well, let me know how it goes. I'd be curious to see it
<rick_h_> wallyworld___: yea, 3.4.1
<rick_h_> but I started out with that, so not sure what the migration points are from 3.3, I've been porting from jquery + backbone + historyjs
<wallyworld___> rick_h_: we have a "unique" way of packaging yui which may be part of the issue, not sure
<rick_h_> yea, I've not gotten my head all the way around all of that yet
<rick_h_> references and use of LEP() or something vs YUI()
<wallyworld___> rick_h_: we basically cat selected yui js files to a big launchpad.js file
<rick_h_> yea, I've gotten that part and added some things to that
<rick_h_> hopefully have my auto resizing textarea widget done tomorrow and in there soon :)
<wallyworld___> rick_h_: so 3.4.1 has split stuff out and re-done the options for seeding yui etc. a trivial change to yui-deps.py to pull in all the new files seems to generate a reasonable launchpad.js but there's an error inside yui when pages load :-(
<huwshimi> lifeless: It worked, then I discovered that I hadn't pushed my change but now can't run lp-land again.
<huwshimi> (it worked, after a restart
<huwshimi> )
<lifeless> huwshimi: same error?
<huwshimi> lifeless: Sorry, yes
<wallyworld___> rick_h_: cool about the text area widget. great to have that fixed
<huwshimi> lifeless: Can I restart my keyring or something?
<rick_h_> wallyworld___: hmm, I can't find that file
<rick_h_> but makes sense
<lifeless> huwshimi: possibly, might be listnening on dbus
<wallyworld___> rick_h_: utilities/yui-deps.py
<lifeless> huwshimi: have  alook for a .*keyring.* process
<wallyworld___> rick_h_: it's run by make
<rick_h_> bah, helps if I tell it to do it recursively
<huwshimi> lifeless: gnome-keyring-daemon?
<lifeless> sounds like it
<lifeless> if its dbus activated, killing it should be pretty transparent
<lifeless> I don't know if it is
<wallyworld___> rick_h_: it print out all the js files we want to use and these are cat'ed to launchpad.js
<huwshimi> lifeless: I'll give it a try
<rick_h_> wallyworld___: gotcha, yea not peeked at that.
<rick_h_> http://yuilibrary.com/forum/viewtopic.php?p=27897
<huwshimi> lifeless: Ah, that seems to have worked. Thanks :)
<rick_h_> that seems sucky, but maybe?
<rick_h_> lol "should be fixed in 3.5.0"
<wallyworld___> rick_h_: so you would think that being conservative and grabbing everything and yui-base and using that would work but sadly not. it fails inside YArray.hash :-(
<rick_h_> wallyworld___: well I'd hand modify the file and start small
<rick_h_> just try to get yui concat'd to launchpad.js and get it to load on some url
<rick_h_> and then start adding a module/two at a time
<huwshimi> oh failure
<wgrant> Failure?
<wallyworld_> rick_h_: yeah, i was hoping not to have to resort to doing stuff like that. i may just leave it as i've got lots of other stuff to get done :-(
<rick_h_> wallyworld_: yea, sorry. There's so much to the build of this stuff I think the only sane way is to break it down step by step
<huwshimi> wgrant: Still trying to get my keyring stuff happening
<huwshimi> so I can lp-land that change
<rick_h_> might at least try a JS test file, .html
<wgrant> huwshimi: :(
<rick_h_> and see if you can just make the few files in one of those work with yui 3.4.1
<rick_h_> just load it into the html from the cdn
<rick_h_> and see if there's anything in the modules we're writing that's harmful (like the forum post on using requires: ...)
<wallyworld_> rick_h_: i'll see what i can do in the time i have left.
<rick_h_> yea, have fun! off to bed for real this time.
<wallyworld_> rick_h_: thanks. ttyl
<wgrant> Night rick_h_.
<huwshimi> wgrant: Ok, done, that change is running through pqm now
<wgrant> huwshimi: Thanks.
<huwshimi> wgrant: No problems
<huwshimi> I don't know how but I appear to have completely broken my ssh/keyring on my computer
<StevenK> huwshimi: What's the issue?
<huwshimi> StevenK: Well, now when I try and ec2 land I get "ec2: ERROR: You must have an ssh agent running with keys installed that will allow the script to access Launchpad and get your branch."
<StevenK> What does ssh-add -l say?
<huwshimi> StevenK: "Could not open a connection to your authentication agent."
<StevenK> ssh-agent should be started on login
<huwshimi> StevenK: ssh-agent gives me output
<StevenK> It will
<huwshimi> what I assume to be normal looking output anyway
<StevenK> No matter if it is running or not
<huwshimi> StevenK: oh, right, that's helpful
<huwshimi> StevenK: How do I get it to run then?
<StevenK> huwshimi: So you can logout and back in and then see if ssh-add -l behaves or you can fix it for one terminal
<huwshimi> StevenK: I'll try the log out first
<huwshimi> brb
<huwshimi> StevenK: Ok, that fixed it this time. Thankyou!
<huwshimi> I restarted a few minutes ago, but that didn't work
<huwshimi> maybe it needed a logout rather than a restart
<StevenK> wgrant: Can haz review? https://code.launchpad.net/~stevenk/launchpad/use-userHasPriviledges/+merge/82967
<wgrant> StevenK: You can't spell privileged.
<wgrant> Not "privileges"
<wgrant> s/Not/Nor/
<wgrant> StevenK: Why isn't userHasPrivileges implemented in terms of userHasPrivilegesContext?
<StevenK> wgrant: Because I couldn't think of a clean way to do it. Suggestions welcome.
<wgrant> StevenK: Make isOneOfDrivers work on distroseries/productseries if it doesn't already.
<wgrant> Then call isOneOfDrivers(context) instead of isOneOfDrivers(pillar)
<wgrant> Mm, it would also need to work for SP and DSP, I suppose.
<wgrant> But it's doable somehow and better than duplication.
<StevenK> And make userHasPrivileges a class method?
<wgrant> No.
<wgrant> It's still an instance method.
<wgrant> It just calls the underlying class method with the instance's context.
<StevenK> You've lost me somewhere, sorry.
<wgrant> userHasPrivileges does basically the same thing as userHasPrivilegesContext
<wgrant> The logic should not be duplicated. userHasPrivileges should call into userHasPrivilegesContext
<StevenK> Oh
<StevenK> Right
<nigelb> Does launchpad do something like keyword search?
<lifeless> statik: lol, you haven't deleted all lp mail... lists.launchpad.net :P
<lifeless> nigelb: we don't treat some words differently, no
<nigelb> lifeless: Are there plans for something like that? Or can I just building with the API.
<lifeless> nigelb: no, why would there be?
<nigelb> I like how for bugzilla, I can do "bug product:Firefox" in my firefox awesome bar and see all the bugs for that product.
<nigelb> I think most modern browsers have that feature.
<lifeless> Ah, thats totally different to keyword search :)
<lifeless> thats a search grammar - and there are plans for that
<lifeless> however, I think for awesome bar stuff you should implement it in the bar
<nigelb> Ah, right. I couldn't figure out what's the right term for it :)
<nigelb> All the awesome bar stuff wants is a search box into which you can put specific words for specific things.
<poolie> lifeless, can i use fixtures as contextmanagers?
<poolie> ah nm
<lifeless> poolie: yes
<poolie> so it's not setUp/tearDown but setUp/cleanUp?
<poolie> more decoy code!
<poolie> lifeless, so is the tearDown method of ZopeViewReplacementFixture wrong?
<poolie> it looks like it will never run
<lifeless> yes, thats buggy
<lifeless> it should just use self.addCleanup
<lifeless> self.addCleanup(undefineChecker, self.replacement)
<poolie> yes
<poolie> very deceptive
<lifeless> self.addCleanup(self.gsm.adapter.register, ...
<poolie> they're run fifo?
<lifeless> same as the testtools one you are familiar with
<lifeless> they all execute, fifo, errors are collected and raised at the end
<poolie> sure
<poolie> i thought it would be tearDown to be consistent with unittest and the code already in that file reinforced that misconcetpion
<poolie> it's fine now
<lifeless> care to fix that bad example while you are thre ?
<poolie> of course
<lifeless> thanks!
<poolie> yakkety yak
<lifeless> don't test back?
<nigelb> poolie: Whats the fun if you don't shave a couple of yaks ;)
<StevenK> wgrant: Diff updated, can you have another look?
<StevenK> wgrant: And I can rename the branch if the name offends you
<poolie> wgrant, btw raw_sendmail is tested now
<poolie> down to the point it would normally call into zope
<wgrant> poolie: Nice
<wgrant> StevenK: Let me see.
<poolie> wgrant,  did you have any luck working out where an email.Header is being passed to raw_sendmail?
<poolie> paranoid_email_content_validatino seems to trap it
<poolie> oh
<wgrant> I can probably dig up a pageid if you haven't solved it yet.
<poolie> two different ways to send mail, of course
<wgrant> Yes
<wgrant> At least
<poolie> maybe i should stop worrying and just flatten the mail
<poolie> *value
<poolie> then we can see if it still fails
<lifeless> wgrant: you hjad lxc puking on resolv.conf not existing ? (due to /var/run/resolvconf being awol) ?
<wgrant> lifeless: don't think so
<wgrant> Not in months, at least.
<wgrant> But not at all that I can recall.
<jtv> wgrant: did you just move bug 887078 back from Fix Released to In Progress?
<_mup_> Bug #887078: Builder:+history timeout <qa-ok> <timeout> <ui> <Launchpad itself:In Progress by rvb> < https://launchpad.net/bugs/887078 >
<lifeless> wgrant: its not making /var/run/resolvconf automatically, or something
<wgrant> jtv: From Fix Committed to In Progress. It's not done yet, AFAICT.
<wgrant> jtv: The pages are still slow and time out.
<lifeless> erm, not the the literal packag,e whatever makes resolve.conf on lucid by default
<jtv> wgrant: I see.  Thanks.
<poolie> is there a new request, and a new timeline, for each test?
<StevenK> wgrant: Did you get distracted?
<wgrant> StevenK: builddisasters tend to do that.
<lifeless> poolie: there is no strong correlation between tests and timelines
<wgrant> Relooking now.
<lifeless> poolie: tests may make requests
<lifeless> poolie: if they do, the request gets its own timeline
<poolie> so
<poolie> i should clear the timeline if i want to observe it later?
<lifeless> the request will d that
<poolie> mm
<poolie> this is the sendmail code
<poolie> i don't know if there is a request
<poolie> do i need to make one?
<lifeless> so, for that, you want to manually setup a timeline - probably easiest to call adapter.set_request_started() and its matching thing later
<wgrant> StevenK: I am confuse.
<wgrant> StevenK: Why not pass the context into userHasPrivilegesContext?
<wgrant> StevenK: bugtask.target
<poolie> ok thanks
<StevenK> wgrant: From IBugTask.userHasPrivileges?
<wgrant> StevenK: Yes
<wgrant> You currently specialcase IBugTask in userHasPrivilegesContext
<wgrant> Why not specialcase series?
<wgrant> So you are actually passing the context in there.
<wgrant> Not a task.
<StevenK> wgrant: IE, if IProductSeries.providedby() ?
<wgrant> role.isOwner(context.pillar) or role.isOneOfDrivers(context) or role.isBugSupervisor(context.pillar)
<wgrant> probably
<wgrant> Remove the Assuming that isOneOfDrivers works on SP/DSP
<wgrant> Which it might not.
<StevenK> I can test this quickly, given wallyworld__'s bitching.
<StevenK> Hey look, he has two underscores.
<StevenK> wgrant: Seems to work fine.
<StevenK> wgrant: Anything else jump out at you?
<wgrant> Really? It shouldn't work, AFAICT.
<StevenK> wgrant: TestBugTaskUserHasPriviliges works fine
<wgrant> Even for DSP and SP targets?
<wgrant> Note that we don't want the pillar.
<StevenK> wgrant: DSP/SP don't have drivers ...
<wgrant> StevenK: No, which is why it probably won't work.
<wgrant> StevenK: They need to use the series/pillar drivers.
<wgrant> StevenK: It's probably better to make them have drivers.
<wgrant> By adding a drivers property.
<poolie> wgrant, ok https://code.launchpad.net/~mbp/launchpad/885972-sendmail-timeline/+merge/82963 should fix your timeline thing
<poolie> afaict
<wgrant> poolie: It's safely unicodable?
<StevenK> wgrant: And check distribution for DSP and both distribution and series for SP?
<poolie> one line of actual fix, ~300 lines of diff to make it testable
<wgrant> StevenK: SP should be able to just check distroseries
<StevenK> poolie: Welcome to Launchpad.
<poolie> wgrant, i think so: it provides a __unicode__ method that tries to take header encoding into account
<wgrant> StevenK: since distroseries should inherit distribution
<poolie> for fairly low values of 'safe'
<StevenK> wgrant: You'd think.
<wgrant> poolie: Ah, missed the not isinstance.
<wgrant> Was worried it could be a shittily encoded bytestring.
<wgrant> But that indeed looks reasonable.
<poolie> yeah i thought of that too
<wgrant> poolie: (un)registerUtility behaves sensibly if there's already one?
<poolie> yeah apparently it stacks them internally
<poolie> it is not very well documented
<wgrant> Aha, handy.
<poolie> :/
<poolie> to some extent i'm going to leave that for whoever uses it next
<nigelb> I may be imagining things, but did Karl Fogel work on Launchpad at some point?
<nigelb> His name and nickname seem somewhat familiar.
<lifeless> yes he did
<lifeless> community/communication mgr
<lifeless> well know for svn
<nigelb> Aha
<nigelb> Yeah, that and the book that he wrote.
<nigelb> Gerv from Mozilla keeps posting excerpts from his book to Mozilla Planet and I keep thinking "Why am I familiar with this name"
<poolie> hi nigelb
 * wgrant glares threateningly at yuixhr
 * wgrant prepares to disable.
<nigelb> Hey poolie
<nigelb> wgrant: It didn't disable itself with the glaring :)
<mrevell> Hello
<nigelb> Morning mrevell
<lifeless> rvba: hi
<lifeless> rvba: I got your query
<rvba> lifeless: hi
<lifeless> rvba: cachedproperty injection is the general answer to this sort of thing
<lifeless> rvba: see BranchCollection for some of the more complex examples that are around
<rvba> lifeless: yeah, but as soon as I materialize the object, the query gets executed so unless I remove the reference, cachedproperty won't help.
<lifeless> got a backtrace of that query being triggered ?
<rvba> Noâ¦ but I can work out an example for you if you want.
<lifeless> run your test with LP_DEBUG_SQL_EXTRA=1
<lifeless> or visit a page with ++oops++
<lifeless> the test is probably easiest
<rvba> Oh, you mean you simply want the query itself.
<lifeless> no
<rvba> Ok, hold on a sec.
<lifeless> the _EXTRA makes it spit out backtraces
<rvba> k
<rvba> lifeless: https://pastebin.canonical.com/56131/
<rvba> lifeless: btw, I wanted to say that having backtraces on oops is a real nice addition to try to find what object should be eager loaded.  They are not very easy to read but still really useful.
<lifeless> rvba: cool, thanks
<lifeless> rvba: so, that backtrace tells me a lot
<lifeless> rvba: what do you get from it ?
<rvba> The problem is the backreference from preview_diff to bmp.  When the previewdiff is created it has to look up the related bmp object.
<lifeless> nope
<lifeless> thats not what happens ;)
<rvba> ah?
<lifeless> the quest is triggered at line 166 of the pastebin
<lifeless> bah
<lifeless> query, line 165
<lifeless> diff_lines_count is accessed at line 156 of the backtrace
<rvba> Hum, security checkâ¦
<lifeless> bingo
<lifeless> the security check is triggering the backref
<lifeless> __init__ or _storm_loaded are not
<lifeless> rvba: now, if you are loading the bmp as part of the eager loading
<lifeless> you can solve this by injecting the bmp into the object
<lifeless> this involves
<lifeless> replace the storm reference with a cached property
<lifeless> and set it
<lifeless> (or you could poke under the storm hood, but so far we haven't done that)
<lifeless> for instance
<rvba> This reference is also used else were and cachedproperty cannot be 'set', right, only populated so to speak.
<lifeless> rvba: http://pastebin.com/vbiezhwF
<lifeless> rvba: it can be with gavins patch
<lifeless> rvba: or you can update the other call sites, which is arguably better
<rvba> Right, I think updating the other call sites sounds better to me.
<lifeless> rvba: does this help?
<rvba> lifeless: it does, thanks!
<lifeless> rvba: backtraces ftw :)
<rvba> Indeed!
<rvba> lifeless: btw, by gavins patch you mean ~allenap/launchpad/cached-property-bug-893074?
<lifeless> yeah
<rvba> This will just prevent one from setting a cachedproperty and thinking it's all good right?
<lifeless> actually yes :)
<allenap> There was a *lot* of test fallout from that patch :-/
<lifeless> so update the callers
<lifeless> allenap: win :)
<rvba> Okay, just checking ;)
<rvba> lifeless: I could also have the cachedreference be called _branch_merge_proposal and only change the callsite in security.py.
<lifeless> rvba: indeed
<lifeless> rvba: however that would then make semi-invisible skew
<lifeless> rvba: better I think for all callsites to use the one attribute
<rvba> lifeless: Good point.
<rvba> lifeless: on second thought, this means that all the call sites wanting to access the property will use branch_merge_proposal but those setting the property will use _branch_merge_proposal.  Or the other way around if I make _branch_merge_proposal be the cachedreference.
<rvba> I think it's cleaner if branch_merge_proposal is the reference that you can get/set without trouble.
<jtv> Anyone know what's up with staging?
<jtv> It's been saying âcode update in progressâ for a suspicious length of time.
<wgrant> jtv: staging or qastaging?
<jtv> staging
<wgrant> Hmm.
<wgrant> That shouldn't be down, AFAIK
<wgrant> qastaging is down for a DB upgrade.
 * wgrant checks.
<wgrant> 2011-11-22 09:10:51 CRITICAL lpmain_staging_slave has transaction by postgres open 0:02:01.180763
<wgrant> 2011-11-22 09:10:51 INFO    No fragile systems connected to the cluster (publish_ftpmaster, publish_distro, process_accepted, buildd_manager, process_upload)
<wgrant> Get a LOSA to retry the update.
<wgrant> Tue Nov 22 09:10:51 UTC 2011 ERROR: Failed to run full-update.py
<lifeless> rvba: mmm
<lifeless> rvba: do you agree that all the sites accessing it should use one attribute name
<lifeless> rvba:  the overhead of the (very few) sites that set it doing it differently seems inconsequential to me
<rvba> lifeless: but AFAIUI it's not possible, we want the sites getting the prop to use the cachedproperty and the sites setting the property to use the reference.
<lifeless> yes
<lifeless> one to write one to read
<rvba> So your argument is that it gets read much more often than set right?
<lifeless> one hopes
<rvba> My argument is this: setting the cachedproperty leads to silent data fuck up ;).  But Gavin's patch will fix this.
<lifeless> indeed :>
<lifeless> rvba: also, you can tell zope to not let writes to that attribute, if you want to
<wgrant> Interesting things also happen when a cachedproperty is used in a Storm query.
<lifeless> rvba: or you can set a set for it
<rvba> lifeless: right, that would be a good solution.
<lifeless> rvba: as wgrant notes, you need to use the storm reference in queries (though you shouldn't do that anyhow- should be using the _id
<wgrant> lifeless: Mm, that's a matter of opinion.
<lifeless> wgrant: perhaps, but I've yet to see a compelling argument for using the reference itself
<wgrant> It's type-safe and I've yet to see a compelling argument for using the ID.
<wgrant> Except that using the reference fails in a couple of cases, which are bugs.
<lifeless> using the reference is discouraged by storm upstream; it fails in some cases and is ambiguous at best in others.
<rvba> Too we can't explicitly define a setter for a cachedproperty Ã  la cachedproperty(_get_branch_merge_proposal, _set_branch_merge_proposal)
<rvba> s/Too/Too bad/
<jtv> wgrant: I seem to be back for the time being.  Any news on staging?
<rvba> number of (storm references in queries + places where the field is set) > number (places where the field is read)
<rvba> This is in favor of having the cachedproperty named _branch_merge_proposal.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 292
<wgrant> jtv:
<wgrant> 21:14:57  * wgrant checks.
<wgrant> 21:15:40 < wgrant> 2011-11-22 09:10:51 CRITICAL lpmain_staging_slave has transaction by postgres open 0:02:01.180763
<wgrant> 21:15:40 < wgrant> 2011-11-22 09:10:51 INFO    No fragile systems connected to the cluster (publish_ftpmaster, publish_distro,  process_accepted, buildd_manager, process_upload)
<wgrant> 21:15:41 < wgrant> Tue Nov 22 09:10:51 UTC 2011 ERROR: Failed to run full-update.py
<wgrant> 21:15:46 < wgrant> Get a LOSA to retry the update.
<jtv> Thanks.
<jtv> Not much luck with the retry though, evidently.  :(
<wgrant> I didn't ask for a retry.
<wgrant> Because gnuoy was busy with other stuff.
<gnuoy> jtv, just kicking it off again now
<jtv> great, thanks
<stub> Do we need an option to increase the long running transaction threshold? We have reasons to keep it low on production, but staging is another kettle of fish.
<lifeless> rvba: the other cases we have have the cachedproperty be branch_merge_proposal
<lifeless> rvba: I humbly suggest that the difference isn't big enough to outweight consistency
<rvba> lifeless: okay then ;) â¦ I hope the test suite will catch it if the cached property is used in a query.
<lifeless> rvba: I suggest grepping
<rvba> Of course I'll do that.
<lifeless> :)
<rvba> :)
<rick_h_> morning
<nigelb> Morning rick_h_
<rvba> Hey rick_h_.
<jml> lifeless: btw, came across this. I haven't read it yet. It's called "The Vietnam of Computer Science" and it's about ORM. Thought you might be interested: http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx
<rick_h_> uh oh, ORM debates this early?
<nigelb> heh
<nigelb> Early is all about perspective.
<rick_h_> so true
<rick_h_> woot! changes landed and are live!
 * rick_h_ does a little happy dance
<rick_h_> meh, I think before people write ORM articles they need to use SqlAlchemy for a big project first
<lifeless> rick_h_: 0100 :P
<rick_h_> too many bad ones cause people to write this stuff
<rick_h_> and lose the idea that ORM == "no need to know SQL"
<lifeless> a terrible idea to be sure
<nigelb> rick_h_: Dear god. SqlAlchemy is scar-inducing :)
<nigelb> I sometimes prefer writing pure SQL to SqlAlchemy :)
<rick_h_> lol, what? How so?
<rick_h_> oh man, when I first tinkered with Python I put SA is one of the top 5 reasons to use Python
<nigelb> Its awesome, etc.
<gary_poster> Hey StevenK.  I was going to try and increase JS timeouts yet again for my new yuixhr tests, but I was unable to build.  versions.cfg says Twisted 11.1.0, but our dist directory has 11.1.0pre1
<nigelb> Just that it takes a while to settle in.
<rick_h_> yea, that's true
<rick_h_> but all the good frameworks do
<gary_poster> StevenK Making a custom Twisted release requires changing files
<StevenK> gary_poster: Hm, lemme check.
<gary_poster> IIRC
<rick_h_> I find if it's too easy to start, you'll grow out of it before the end of the project
<StevenK> gary_poster: But 11.1.0 is a REAL release!
<gary_poster> StevenK, then maybe you did not push up the changes?
<gary_poster> to the download-cache
<StevenK> gary_poster: But it passed ec2!
<gary_poster> StevenK...
<nigelb> rick_h_: heh, agreed.
<nigelb> StevenK: Maybe you didn't test for it? :D
<gary_poster> lemme look at my download-cache again
<StevenK> nigelb: The download-cache is not seeded on ec2 instances, if buildout works and the tests pass, the upgrade is good.
<StevenK> nigelb: That is, "troll failed"
<nigelb> StevenK: Damn. I'll blame travel exhaustion for it.
<StevenK> gary_poster: It's here in my download-cache
<nigelb> StevenK: I was "closer" to you for a weekend :)
<StevenK> gary_poster: r401
<gary_poster> StevenK, sorry, bizarre system confusion.  From emacs term, bzr up in my download cache said there wasn't anything to get, and from a normal terminal in updated, getting Twisted.  I have no idea what is going on over here, but it isn't your fault :-P
<gary_poster> Thanks StevenK
<StevenK> That's what you get for using emacs to do your dirty work.
<gary_poster> :-P
<nigelb> s/to do your dirty work//g
<StevenK> nigelb: Oh?
<nigelb> :D
<nigelb> StevenK: Kuala Lumpur!
<rick_h_> lol, from ORM flames to emacs...on fire today
<nigelb> Amazing weekend with Mozilla people for Mozcamp Asia.
<StevenK> nigelb: Pfft, like Australia was that much further ...
<nigelb> StevenK: Well... it made sense to keep an Asia event *in* Asia... ;)
<nigelb> How far is Australia from KL?
 * nigelb checks
<nigelb> Interesting. Not too far.
<jml> rick_h_: have you read the article?
<nigelb> jml: It was an interesting comparison.
<nigelb> I bookmarked to finish reading later.
<rick_h_> jml, I gave it a run through skimming
<jml> nigelb: yeah, it's on my readitlater list
<rick_h_> basically seems to make the general argument people start using and ORM and refuse to lose and give it up
<rick_h_> thus enduring more pain and suffering
<jml> rick_h_: so are you saying  that if only he'd used Python & SQLAlchemy instead of whatever other technologies he's using, he'd not have a problem with ORMs at all?
<rick_h_> no, I'm saying that people get fussy over ORMs, but what ORM you use can great change your outlook
<rick_h_> having used several ORM in PHP land and Python land
<rick_h_> things like the verbosity stuff he talks about is better in python
<rick_h_> along with things like the ability of Python to add/modify objects at will make it more flexible for a lot of ORM tricks
<rick_h_> and SA improves a lot because it's two layers, the expression layer makes crazy things possible
<rick_h_> while the ORM layer makes typical things pretty and easy
<rick_h_> but most ORM solutions don't break down nicely like that
<rick_h_> but SA has its own pain points for sure
<nigelb> I've always thought it was a compromise of the lesser of the two evils.
<nigelb> And that depends on the situation.
<rick_h_> it makes you have to learn more, there's two solid layers to understand, and it expects you to actually write your own layer above it
<rick_h_> which I think is very important
<nigelb> Oh.
<nigelb> SA excepts developers to abstract SA out of their operations?
<rick_h_> well, I argue that you write an API layer that your app accesses and you wrap all your SA
<rick_h_> that way you can swap out the back end/what SA is doing without breaking your app
<rick_h_> http://python.mirocommunity.org/video/4392/pyohio-2011-sqlalchemy-tutoria
<nigelb> Hrm. Fair.
<rick_h_> for thoughts on it :)
<nigelb> I have seen projects doing exactly that.
<nigelb> https://crash-stats.mozilla.org/ for one does that.
<nigelb> Backend is something python-ish and SA.
<rick_h_> yea, I think that's the only way to go and really solves a lot of problems with DRY and such
<rick_h_> and makes the breaking down to SQL and such less icky
<nigelb> This changes my POV for a few things. Now I need to go back and think things through :)
<gary_poster> nigelb, rick_h_, I think ORMs are evil, from experience, and even pure object databases seem easier than they are.  However, a good alternative is tricky.  I am interested in the pure data + SQL composability for functions that is being explored in Clojure-land with projects like Korma. http://sqlkorma.com/ .  It allows abstractions while being much closer to what the database actually manipulates.
<gary_poster> http://nathanmarz.com/blog/clojure-or-how-i-learned-to-stop-worrying-and-love-the-paren.html has a related argument.
 * gary_poster retreats back into cave, with a madman hair wig on
<jml> gary_poster: I think Haskell has similar stuff
<jml> gary_poster: incidentally, thanks for the Clojure conference write-up. Was a good read.
 * jml has had the "simple vs easy" talk flagged for watching for a couple of weeks now. Surprisingly hard to find an uninterrupted 1hr+ of time to watch things.
<rick_h_> jml: heh, yea. I wanted to watch some YUIConf videos so scheduled it as group viewing at CHC this week
<rick_h_> make everyone else suffer my wishes! bwuhahaha
<jml> CHC?
<rick_h_> sorry, CoffeeHouseCoders, weekly meetup I run locally for devs
<rick_h_> http://coffeehousecode.appspot.com/
<rick_h_> detroit branch
<rick_h_> gary_poster: yea, that second link is a lot of stuff I'd do with SA
<rick_h_> creating almost little sql "widgets" and then using them into larger queries and such later on
<jml> rick_h_: you can do that in storm, fwiw
<rick_h_> yea, I look forward to getting into some storm stuff later on.
<rick_h_> I know I gave it a look when it first came out and really didn't follow it afterwards
<rick_h_> well, first went OSS public I guess vs "first came out"
<rick_h_> I'm still floored people suffer through the django orm though ugh
<gary_poster> jml, cool, glad you liked write-up.  Yeah, not surprised Haskell has similar stuff.  And yeah, understood about the hour.  I put it on one weekend while I was supposed to be cleaning a room.  Cleaning the room was pretty slow, but hey, I saw the video! :-)
<rick_h_> gary_poster: I like though that this clojure stuff keeps things sql'y. Keep people from getting away from that idea
<jml> gary_poster: :D
<gary_poster> right
<rick_h_> ah, imapfilter running now. This ought to help with all these lists
<jml> gary_poster: yeah, I've got to disassemble a whiteboard this weekend, might have this in the bg then :)
<gary_poster> sounds perfect jml :-)
<bac> sinzui: ping
<sinzui> hi bac
<rick_h_> do we have a sprite with a spinner not on a white background? like transparent?
<deryck> rick_h_, no, I don't think we do.  We only have spinner.gif or whatever it's called, that is over white.
<rick_h_> yea, ugh. Ok
<dobey> deryck, rick_h_: just do one in CSS
<rick_h_> dobey: http://fgnass.github.com/spin.js/ import this? :)
<rick_h_> dobey: when I tried to do a css3 spinner it had cross browser issues
<rick_h_> and a TON of vendor prefixing fun
<dobey> eh :)
<deryck> yeah, css 3 spinners are just showing off.  And painful.  We can get a new image if you need.
<rick_h_> actually, I use it on my bookie plugin, but the animation freezes while the ajax request takes place
<rick_h_> yea, I figured I'd check on the stand up the best way to move forward on it
<rick_h_> I need a spinner on a grey background and not sure if a fresh bug or something else it the right way to move it forward
<rick_h_> http://www.yuiblog.com/blog/2011/11/15/yui-3-5-0-roadmap-and-timelines/
<rick_h_> deryck: ^
<abentley> deryck: lp:~abentley/launchpad/history-model
<deryck> abentley, ok, will take a look now.
<abentley> qastaging looks unhappy.  Known issue?
<deryck> abentley, not to me.  Not sure if others know.
<rvba> abentley: deryck The update failed this morning at 09:10:51 UTC
<rvba> gnuoy has kicked it off again after that.
<abentley> rvba: Cool, thanks.
<rvba> s/has/
<rvba> Not sure were we're at nowâ¦ gnuoy?
<gnuoy> rvba, hi
* abentley changed the topic of #launchpad-dev to: qastaging is down, but being worked on.  |https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 292
<rvba> gnuoy: hey, can you tell us where we're at with qastaging's update?
<gnuoy> hmm, it hasn't logged anything for the past few hours
<deryck> abentley, a ha!
<deryck> abentley, the event name has changed since it's a new object.  It's now:  buglisting-model:fields-changed
<abentley> deryck: I don't follow.  The event is still being constructed using this.constructor.NAME + ':fields-changed' where NAME is buglisting-config-util
<deryck> abentley, no, because NAME is the buglisting model object.  the event is fired from the new object, not buglisting config util objects.
<abentley> deryck: Ah, so this is a keyword-this trap.
<deryck> abentley, right, in YUI's event handling "this" always gets bound back to the object firing the event.
<gary_poster> sinzui, the "project group report" we talked about is the +milestones page for project groups, right?
<deryck> abentley, or rather to the host object, since the event is the ATTR change event.
<abentley> deryck: Okay, adding "this" to the end of .after() fixed it.
<abentley> deryck: thanks!
<deryck> abentley, yup.  Long term, I think the addListeners stuff should move up to the model object, since it's really after that object updates that we're interested in.  And then the tests updated.
<deryck> abentley, np!
<sinzui> gary_poster, yes, you can see an example of the report that is being used: https://launchpad.net/landscape-project/+milestone/11.11.2
<gary_poster> thanks sinzui
<rick_h_> gmb: if you get a quick second: https://code.launchpad.net/~rharding/launchpad/bug_814697_missing_spinner/+merge/83034
<abentley> deryck: btw, the trick of setting a global, e.g. "debug_me=true" and doing if(debug_me){debugger;} works pretty well.
<deryck> abentley, ok, nice.  I'll have to remember that.
<deryck> gmb, I also have a branch that needs review, if you're not too overloaded.
<gmb> rick_h_, Sure
<gmb> deryck, I'm not. I'll take a look after rick_h_'s
<deryck> gmb, awesome, thanks!  https://code.launchpad.net/~deryck/launchpad/field-visibility-persistence-891780/+merge/83037
<gmb> sinzui, Any particular reason you assigned bug 867593 to me?
<_mup_> Bug #867593: Displayed number of comments hidden is sometimes +1 to the actual value <bugs> <comments> <regression> <Launchpad itself:Triaged by gmb> < https://launchpad.net/bugs/867593 >
<deryck> abentley, bug 890745 is fixed release, right?  I think it's another case of tagger not handling pre-reqs right.
<_mup_> Bug #890745: dynamic bug listings don't handle next and prev correctly on first and last batch <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/890745 >
<sinzui> gmb, lifeless assigned a bug regarding comment numbering to you, tagged it as regression, then raised it to critical. He wanted your opinion...
<abentley> deryck: that's right.
<deryck> abentley, ok, thanks.
<gmb> rick_h_, approved.
<sinzui> But that bug was a dupe of an older bug. I made you the assignee of the master, and also a related bug that I think a numbering fix will also fix
<rick_h_> gmb ty
<sinzui> gmb, so I am channeling lifeless and asking for you opinion of all bugs assigned to you tagged with "comments"
<gmb> sinzui, Uhm, okay. I'm slightly confused as I don't know which bug it was that lifeless assigned to me. But I think for the sake of sanity we should unassign those bugs. I'm up to my eyes in bug 881019 at the moment so it makes more sense for someone else to take them if they have the time.
<_mup_> Bug #881019: Lp login is broken after account merge <merge-deactivate> <openid> <regression> <users> <Launchpad itself:In Progress by gmb> < https://launchpad.net/bugs/881019 >
<sinzui> gmb, This is the bug I saw 45 minutes ago https://bugs.launchpad.net/launchpad/+bug/893375
<_mup_> Bug #893375: Comment order wrong and not all comments shown <comments> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/893375 >
 * gmb looks
<sinzui> gmb: you have my sympathies with that bug. I suspect the issues will not really be fixed until this feature is implemented with ISD: bug 770107
<_mup_> Bug #770107: launchpad relies on login.ubuntu.com but does not sync email addresses <email> <feature> <openid> <Launchpad itself:Triaged> < https://launchpad.net/bugs/770107 >
<gmb> sinzui, That may well be the case. Okay, now that I have a better idea what's going on, I'm happy enough to be assigned to those bugs.
<gmb> Thanks for the clarification.
<gmb> (I think the fencepost error is a result of the fact that the BugComment code basically consists of all the Aale of which bigjools's Luftkissenfahrzeug is always voll.
<sinzui> gmb: really do not know what is going on. the openididentifier -> account <-person <- emailaddress -> account relationship seems to permit different answers on how you join...
<gmb> sinzui, Yeah. It's a pain. At the moment I'm working on removing Person.account to see if that causes any fewer problems. Unfortunately, lots of things have now broken.
<sinzui> gmb ... So the code that might try to setup or fix user data may get an different set of linked object when logging the userin or showing a list of object
<sinzui> gmb so email  is the link between person and account?
<gmb> sinzui, That's the plan (you suggested it here https://bugs.launchpad.net/launchpad/+bug/881019/comments/1, which is where I got the idea from).
<_mup_> Bug #881019: Lp login is broken after account merge <merge-deactivate> <openid> <regression> <users> <Launchpad itself:In Progress by gmb> < https://launchpad.net/bugs/881019 >
<abentley> deryck: the only place we should be using field_visibility_defaults is when the user asks us to reset to the default.  In other places, we should be using field_visibility.
<gmb> Unfortunately, it _might_ be too inefficient. Not sure yet because I haven't got the test suite passing.
<deryck> abentley, agreed.  Did I not do this somewhere?
<abentley> deryck: Yes, in the getter and setter for BugListingConfigUtil.field_visibility
<sinzui> gmb: Yes, I was not certain it would scale.
<deryck> abentley, do you mean the valueFn?
<abentley> deryck: yes, the valueFn and the setter.
<deryck> abentley, so the valueFn only returns default values.  it will never be used in most cases, where you pass in a config.  if nothing was defined, I think we should use field_visibility_defaults there....
<abentley> deryck: If nothing was defined, I think we should use LP.cache.field_visibility.
<deryck> abentley, I guess that's fine.
<abentley> deryck: Those are the setting that were used to render the initial view.
<abentley> i.e. server-side.
<deryck> abentley, sure.  That's probably better.  I initially liked be explicit with a config, which is why I didn't sniff the cache for it....
<deryck> abentley, but that will fix the issue of the bar not being in sync with the page, and we can drop the explicit config on object creation.
<deryck> abentley, as for the setter, I'm trying to recall why I merged with the defaults there.  I had some reason that escapes me, but hopefully the tests will reveal that if you change it to merge with field_visibility.
<abentley> deryck: But you do sniff the cache, implicitly, by using get('field_visibility_defaults'), which hits the cache.
<abentley> deryck: So it's a question of which cache value you hit, and if field_visibility_defaults is defined field_visibility is also defined and better.
<deryck> abentley, right, I'm fine to change it to sniff for field_visibility.  Was just trying to explain my reasoning, i.e how the code go to where it is.
<abentley> deryck: okay.  It wasn't clear to me that the valueFn was intended to be a fallback.
<deryck> abentley, yeah, that's all I was trying to say, in that the common case, and in current template usage, it's never hit.
<deryck> abentley, but if we make that change to sniff for field_visibility, we can drop the explict config for field_visibility from the template and fix a bug.
<sinzui> Does anyone know of a way query users to know if someone was former canonical employee?
<sinzui> Ah, I had to type that to remember I can check the TeamMembership deactivated status
<abentley> deryck: I'd rather make field_visibility and field_visibility_defaults mandatory.  We can just pass the LP.cache in as the config object.
<deryck> abentley, as the config or as part of the config?
<abentley> deryck: I meant as the config, but we could pass it in as part of the config instead.
<deryck> abentley, if we do, let's make it part, so we don't have to append srcNode type configs to the cache, or our reference of it.....
<deryck> abentley, but what is the advantage of passing it in, versus sniffing for it?  We only need it in a couple places.
<abentley> deryck: I don't know what a "srcNode type config" is.
<abentley> deryck: decoupling and explicitness.
<deryck> abentley, I meant srcNode type of things that go in the config.  i.e. we have to specify things that are in ATTRS plus srcNode beyond just field_visibility.
<abentley> deryck: I don't know what srcNode is.
<deryck> abentley, srcNode is what the widget framework uses to know where on the page to stick the widget you're rendering.
<deryck> srcNode: Y.one('#mydiv-for-this-widget')
<deryck> that ^^ goes in config usually, beyond any custom attrs.
<abentley> deryck: Okay but that's needed for the BugListingConfigUtil, not the model.  Right now, the model just needs field_visibility and field_visibility_defaults.
<deryck> abentley, right.
<deryck> abentley, also, can we just pass around values in the config, rather than the whole cache.  and maybe that's what you meant.
<deryck> so field_visibility: LP.cache.field_visibility rather than config.cache = LP.cache.
<deryck> abentley, hmmm, well I guess for the model object the cache actually does make sense as the config....
<deryck> abentley, sorry, I'm stuck thinking about BLCU.
<abentley> deryck: so we certainly can do new BugListingModel({field_visibility: LP.cache.field_visibility, field_visiblity_defaults: LP.cache.field_visibility_defaults}), but it just seems verbose at this point.
<deryck> abentley, if you only needs those two bits from the cache, I prefer that.  But I'm not going to be a jerk about it. :)
<deryck> abentley, but if you need more from the cache, using it as the config is fine.
<gmb> deryck, approved, with comments.
<deryck> gmb, thanks!  Good suggestions all the way around.
* gmb changed the topic of #launchpad-dev to: qastaging is down, but being worked on.  |https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<james_w> if I've got one of the new hash-style oops that should still be findable via lp-oops shouldn't it?
<rick_h_> abentley deryck either of you guys have a sec to help me figure out what I'm missing from getting yui simulate to help me test my events?
<lifeless> yes, if it has synced to devpad
<lifeless> production doesn't have the rabbit config toggled on yet (liam is still getting the glitches out of monitoring)
<james_w> yep, just wanted to make sure I wasn't sat here hitting refresh for nothing
<deryck> rick_h_, did you include node-event-simulate in the requires line for the test module?
<rick_h_> yea
<rick_h_> and if I manually interact with the html on the page the event fires
<rick_h_> so everything seems "hooked up"
<rick_h_> but running node.simulate('keyup') or 'valueChange' isn't doing me any good for automated testing
<deryck> rick_h_, so everything in the basic example here matches what you're doing?  http://yuilibrary.com/yui/docs/event/index.html#simulate
<rick_h_> deryck: yea, loggerhead is hating me, but the relevent bits: http://paste.ubuntu.com/746203/
<rick_h_> oh, nvm
<rick_h_> I had to do it right, can't simulate valueChange and when simulating keyup, you need the value code and the keydown to go with it
<rick_h_> http://paste.ubuntu.com/746211/ changing to that works
<deryck> rick_h_, alrighty then.  there you go. :)
<rick_h_> am I missing an easy way to check assertion counts in YUI test? I see Y.assert has a count and _getCount() but nothing public I'm seeing
<deryck> rick_h_, what are you wanting to count?  how many assertions you make?
<rick_h_> yea, sanity check because I've got to do some timeout waiting
<rick_h_> so want to verify that there  were 3 assertions in this test kind of thing
<deryck> rick_h_, do you know about "waits"?  Are you using that, or actual setTimeouts?
<rick_h_> yea, I'm using a setTimeout right now. I guess if I go to wait/resume it'd "catch" the case I'm looking for
<rick_h_> nvm then, I'll try to find out later on. Seems like there's some stuff there for it, but my searches are coming up empty
<deryck> rick_h_, yeah, I think it would be better to use waits then count the assertions.
<rick_h_> gotcha, thanks
<rick_h_> deryck: for this plugin, is there a good place to document it at?
<rick_h_> the doc directories all seem really just doctests
<deryck> rick_h_, the dev wiki is where we tend to put pure documentation.
<deryck> rick_h_, and we usually do a launchpad-dev email to say "hey I did this, it's available, and docs are at $LINK" :)
<rick_h_> ah ok, was looking for things like usage examples and such for the lazr stuff in the code
<rick_h_> deryck: ok, phew...let my first big MP get a thrashing! https://code.launchpad.net/~rharding/launchpad/bugfix_891735/+merge/83068
<deryck> rick_h_, that's a nice size change. :)
<deryck> rick_h_, can you hunt around for another reviewer.  Trying to get stuff landed I'm behind on.
<rick_h_> little better than one liners so far?
<rick_h_> sure thing
<deryck> rick_h_, but if no one else can do it, I will, certainly.
<rick_h_> understand, was an FYI
<rick_h_> come out reviewers come out wherever you are please
<rick_h_> fresh meat to hack up on
<rick_h_> jcsackett abentley or should I ping anyone else in particular? no rush on things
<abentley> rick_h_: I'm happy to look at it.
<rick_h_> awesome, thanks!
<abentley> rick_h_: Could you add a screenshot, please?
<rick_h_> abentley: I guess, it just looks like a textarea
<rick_h_> I don't dress anything up at all
<rick_h_> a movie perhaps?
<rick_h_> or would that be a bit much
<abentley> rick_h_: We usually request screenshots for UI changes, but if there's nothing to see, that's okay.
<rick_h_> yea, it's invisible until you use it
<abentley> rick_h_: lifeless has a good point.  We already have auto-sizing behaviour for bug descriptions, just not on bug creation.
<rick_h_> right, I noted that in my MP. It definitely does, and it's built around multiple elements, only on edits, and the whole toolbar/save event workflow
<abentley> rick_h_: I thought you were referring to the bug title.
<rick_h_> in discussion it was thought this was light enough and different enough to be useful since the inlineedit isn't easily pulled out atm to use for this
<rick_h_> no, in the pre-impl note?
<abentley> rick_h_: It seems pretty unfortunate to have different widgest for creating text fields vs editing them, don't you think?
<rick_h_> yes, definitelty, but the original goal here was that if this gets tested and is ok, we could enable this across all textareas on the site as part of the global ui
<rick_h_> which is a bit hard to do with the inlineedit module
<rick_h_> out of this I've got a couple of bugs to push against the inline edit to help it's resizing, and then as a larger scope can look to see if somehow that can use this plugin for handling textareas
<rick_h_> but it's very tied in currently
<abentley> rick_h_: I really don't get it.  Bad enough to have an inconsistent one-off field, far worse to have many inconsistent fields.
<rick_h_> inconsistent how?
<rick_h_> I admit that just adding to this one place in the bug report isn't great, but that's where the bug/idea came out of and it needed a home to start at
<abentley> rick_h_: you'd have a better idea than me, but generally when there are two implementations of something, they are never quite consistent.
<rick_h_> I didn't want to make it global without getting it in front of more people/eyes
<rick_h_> abentley: yes, true. but this is only resizing, while the other is an implmentation that includes some resizing logic
<rick_h_> but really deals with a lot more
<rick_h_> some parts are definitely something to add to the inlineedit, support for min/max heights and such
<rick_h_> but it's really a lot more than just a resizing widget
<rick_h_> and stripping that logic out to make it global was deemed more work than it's worth at the moment
<abentley> rick_h_: By doing this, you'd be adding tech debt.  Bug #723417 doesn't seem to justify adding tech tebt-- it could just be solved by changing a CSS class or something.
<_mup_> Bug #723417: Further information input area shrunk making it difficult to input and review details <bugs> <qa-ok> <regression> <ui> <Launchpad itself:Fix Released by rharding> < https://launchpad.net/bugs/723417 >
<flacoste> rick_h_: wouldn't be possible to implement this as YUI plugin?
<flacoste> that could be used standalone like here
<abentley> flacoste: he has.
<rick_h_> yes, that's how it's implemented
<flacoste> and used on the textarea editor
<deryck> I chatted with him about the inline editor.
<flacoste> then couldn't use this plugin in the textarea inline editor?
<rick_h_> long term it's possible. Right now there isn't clear separation of that in the inline editor
<flacoste> right
<rick_h_> so it needs some updates to be able to use this plugin as a base for multiline
<flacoste> that's well possible
<deryck> abentley, flacoste -- the inline editor tightly couples the single line and multi-line editor to the same widget.
<rick_h_> right
<deryck> that's the tech debt, really.
<deryck> it's not like he can just plug the text area, which is what I also suggested.
<flacoste> right
<flacoste> the inline editor needs some love
<deryck> it's a lot of work to make the inline editor clean first.
<deryck> right
<deryck> once the inline editor is cleaned up, then yes, it would be trivial to plug the text area for resize.
<flacoste> so we are all in agreement then!
<flacoste> this approach is sane
<rick_h_> agreed, would love to hack on it some. This was kind of a chance for my first "large" work through the JS code and the process
<flacoste> once we clean the inline editor mess
<deryck> yeah.
<flacoste> we can use this new clean plugin to remove the coupled functionality in the editor
<deryck> yup
<abentley> deryck: there can be multiple pieces of tech debt.
<deryck> abentley, sure.  But I don't think this is creating debt.
<deryck> abentley, it's saying "here's the one true way to do resizing text areas."
<deryck> abentley, the debt already exists in that the inline editor was poorly factored to start with.  and can't make use of this one true way.
<abentley> deryck: To me, duplicating functionality is tech debt pretty categorically.
<deryck> abentley, we're not really duplicating.  There is no generic way to resize text areas.  The inline editor resizes as one of a million things that it does. ;)
<deryck> bear in mind, I have no issues with rick_h_ continuing work on this in a follow up branch to clean up the inline editor.
<abentley> deryck: I understand that there is no generic way to resize text areas.
<deryck> abentley, so do you think he shouldn't land this?
<abentley> deryck: I still see it as functionality duplication.
<deryck> abentley, so are you suggesting he shouldn't land it?  Or that he shouldn't land it without also fixing the inline editor?
<abentley> deryck: I am not happy either saying he shouldn't land it or that he should.
<deryck> heh
<deryck> come on, commit! :)
<abentley> deryck: This is a case where code review is covering pre-implementation material.
<rick_h_> hah, I knew my first code would get a tearing into!
<abentley> deryck: and that always sucks.
<deryck> abentley, yeah, it does.  but he and I did discuss that in a pre-imp.  it's a lot of work to rip that stuff out of the editor and add this.  and I saw value in him getting this widget first and then turning to the editor.
<deryck> abentley, would you have suggested something different?
<abentley> deryck: Yes, I would have said if we don't want to work on the in-line editor, we should delay working on a generic resizable textarea until we are ready to work on it.
<deryck> abentley, well, I never said we don't want to work on the inline editor.  But also, I do see value in this by itself.  How can someone clean up the editor if they don't have this to replace the current stuff that's bad?
<deryck> I definitely think doing both in the same branch is too much.
<abentley> deryck: They can clean up the editor by extracting its resizing functionality and making it modular.
<abentley> deryck: And then making it pluggable.
<rick_h_> so what if I were to remove the one place it's currently used, and just try to land as "added a plugin" for now based on code/tests
<rick_h_> and then work on the inline edit using this plugin as a next step before it gets used anywhere
<deryck> abentley, no, you can't extract it.  you need to first convert the single inline editor widget into two widgets, one for text inputs and one for text areas.
<abentley> deryck: I haven't looked at that code.  Maybe that's true in this case.
<deryck> abentley, it is, I promise.  I'm not trying to be a dick.  I'm just telling you it is a lot of work to fix.
<deryck> and it's really not worth pulling out; it's poorly written, using an updateSize thing that is called after so many different events.
<abentley> rick_h_: I would rather not commit unused code.  If you're actually committing to fixing the tech debt by fixing the inlineeditor, that is enough for me.
<deryck> abentley, FWIW, I'm fine with rick_h_ committing to doing the follow up work.
<deryck> i.e. I don't have anything else he has to do instead of that.
<abentley> deryck: You should also understand that I'm on my third day of de-duplicating functionality, and so I'm perhaps more sensitive to these issues than normal.
<deryck> abentley, yeah, fair enough. :)
<rick_h_> well I have to run here in a sec. What should I do from here?
<abentley> rick_h_: I see you're setting skip_animations true, so why do we need to wait "a slight sec"?
<deryck> rick_h_, if you're leaving, let's just take up the issues in the standup, if abentley doesn't mind.
<deryck> standup tomorrow, I mean.
<abentley> rick_h_: we can pick up tomorrow.  I'm the on-call-reviewer anyway, tomorrow.
<rick_h_> abentley: because in testing, chrome was too fast that if I checked right away it still grabbed the orig sizing values
<rick_h_> and the 100ms there seemed to slow it down just enough for chrome to finish the ui redraw
<rick_h_> abentley: ok, sounds good to me
<rick_h_> thanks and sorry for the trouble
<deryck> abentley, while I have you, too :)  Does mustache have a syntax for negative conditions?  I assume there's no "else" block, but is there an "if this value is None" kind of expression?
<abentley> rick_h_: np
<abentley> deryck: Yes.
<abentley> deryck: e.g. {{^repo}}{{/repo}}
<deryck> abentley, ah, thanks!  I would have never guessed the ^.  And I couldn't find it in docs.
<deryck> abentley, now searching docs for "^" I do ;)
<abentley> deryck: :-)
<deryck> it was also something hard to google for.  I got lots of "Are mustaches hot or not" hits.
<poolie> hi all
<flacoste> hi poolie
<flacoste> poolie: what's happening with https://code.launchpad.net/~xaav/loggerhead/export-tarball/+merge/66408?
<flacoste> do we have anybody on the hook to integrate this?
<poolie> no
<poolie> i think there should be
<poolie> :(
<poolie> it is a real shame it has been neglected for so long
<poolie> so
<poolie> perhaps i am well placed to shepherd it in
<flacoste> poolie: thanks!
<flacoste> i think it only requires a merge to loggerhead, updating the version in LP and then QA?
<poolie> probably a ton of qa
<poolie> i think that's the main real commitment
<huwshimi> "17045 tests run in 4:54:26.362705, 1 failures, 303 errors" hah!
<wgrant> Heh
<wgrant> Which branch was that?
<StevenK> wgrant: Opinions on http://pastebin.ubuntu.com/746479/ ?
<wgrant> I guess.
<StevenK> wgrant: I'm still working on the branch, so if you'd prefer a loop or something, say something. :-P
<huwshimi> wgrant: Oh, the branch was to do with the tag cloud, the errors were all because "ImportError: No module named lpbuildd"
<wgrant> huwshimi: Ah, merge devel.
<huwshimi> wgrant: Yeah, I figured that might fix it
<mwhudson> should make it about an hour quicker too...
#launchpad-dev 2011-11-23
* StevenK changed the topic of #launchpad-dev to: qastaging is down, but being worked on.  |https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 292
<wgrant> Erm
<wgrant> r14367 is a little odd
<StevenK> Oh?
<wgrant> Naming cookies after the LP username.
<wgrant> Very odd.
<StevenK> qas looks okay to me, is the topic stale?
<wgrant> It's been back for an hour or so.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 292
<StevenK> Good enough for me.
<wgrant> lifeless: Was there consensus on which cookie approach to take?
<lifeless> wgrant: see the discussion on the MP that just landed
<wgrant> Hmm, OK.
<wgrant> Still seems pretty hideous to include the (mutable) username in the cookie.
<lifeless> agreed
<lifeless> deryck is coming around
<poolie> wallyworld__, i guess feedback about the specifics of the disclosure ui would be beside the point?
<wgrant> Probably, but still useful.
<wallyworld__> poolie: for the email to the dev list yes. but huwshimi and mrevell etc would love to hear feedback
<wallyworld__> poolie: since we are iterating based on screenshots from 18 months ago and user testing now etc
<wallyworld__> poolie: also, that ui you saw from the email is now old. we're working on a new version, something like http://people.canonical.com/~huwshimi/policy7.png
<poolie> ok
<poolie> that looks a lot better
 * poolie cancels his draft
<poolie> maybe you can follow up with that?
<wgrant> fuuuuuuuuuuuuu
<wgrant> buildbot failed again.
 * wgrant disables yuixhr
<wgrant> gary_poster: Around?
<gary_poster> wgrant: I'm working on it
<wgrant> actual = ["Failure in lp/app/javascript/tests/test_lp_client_integration.Cache data.test_logged_in_user_has_cache_data: Unexpected error: Can't find variable: url",
<StevenK> wgrant: I thought you did that last night?
<wgrant> StevenK: I decided to give it another run to sort itself out.
<wgrant> It didn't.
<StevenK> wgrant: Or was that just a threat?
<wgrant> gary_poster: Any reason not to disable it now?
<wgrant> It's blocked all builds for more than a day now.
<wgrant> Ah, I see your email.
<wgrant> Thanks.
<wallyworld__> poolie: the ui from the screenshot is not quite ready yet. the point of the email wasn't get get feedback on the disclosure ui, but the technology used to produce the mockups
<poolie> yeah i understand
<poolie> it was just an example
<poolie> i think it's great that you did an interactive mockup
<wallyworld__> it took a bit of work (lots of javascript hacking) but it's got some great feedback into the development of the feature
<wallyworld__> if we can speed up the process by which we can deliver such mockups and end up with less wastage (throwaway code), i think that will be great
<poolie> i wonder if there is any path to having it running in an actual lp instance but with no backend
<poolie> i don't know if that would actually be harder
<wallyworld__> poolie: pyramid or grok (ie a lightweight, easily deployable app server with zope support) i think may be easier
<lifeless> wallyworld__: is there any reason you can't just edit templates in-place on lp ?
<lifeless> wallyworld__: with the same persistence throwaway story ?
<wallyworld__> lifeless: wouldn't that involve landing stuff via pqm or whatever?
<wallyworld__> or do you mean ssh into the deployment server
<wallyworld__> and edit directly via the file system?
<rick_h_> wallyworld__: is this up some place to play with? (seeing screenshot)
<lifeless> wallyworld__: ec2 demo
<rick_h_> I'm curious what the edit link does, it seems like it'd just be on the person to overlay a ui to add/edit permsisions for that user vs per permission
<wallyworld__> rick_h_: the latest screenshot isn't (yet). only the earlier version as per the email
<lifeless> wallyworld__: e.g. just edit in your own local copy to hearts content, then shove it up on the ec2 instance (or canonistack or ...)
<wallyworld__> lifeless: ah, that may work. i haven't used ec2 demo before
<wallyworld__> rick_h_: the edit icon per person uses a ChoiceSource popup to edit the permissions for that person
<wgrant> As discussed on the call yesterday, ec2 demo probably doesn't quite work any more, but would be easy enough to fix.
<rick_h_> wallyworld__: got it, missed the link when I read your email originally
<wallyworld__> the thing to remember also is that it has to be a RAD environment. hacking javascript and static html fits the bill there. but we want a bit more, like templates etc
<rick_h_> wallyworld__: are you guys using a JS tpl language at all?
<rick_h_> YUI is building in support for handlebar in their MVC stuff, but I've used jquery-tmpl and underscores stuff
<wallyworld__> rick_h_: no. but we would look to use mustache i guess since that's now in our technology stack
<rick_h_> yea, I think deryck and the guys are using it on their feature landing
<wallyworld__> (it wasn't when we started the mockup)
<rick_h_> sorry, handlebar/mustache
<rick_h_> I group the two together when I think of them
<wallyworld__> sure :-)
<rick_h_> I've always thought the couchapp stuff would be awesome for mockups, but I think it's a bit far from where you want to be
<rick_h_> wallyworld__: small ui thing, curious why the "Add an Observer" wouldn't fit over on the right like a "Report a bug" or "Register a branch" type item?
 * rick_h_ is checking out when different ui mechanisms are used like that
<wallyworld__> rick_h_: it may well go there. we were working off the supplied screenshots that were done a while ago
<rick_h_> ah ok
<wallyworld__> poolie: rick_h_: latest version http://people.canonical.com/~ianb/disclosure/
<rick_h_> sorry, it's interesting stuff so I like to pester :)
<rick_h_> I'd definitely try to go lighter on the grey for more contrast
<rick_h_> but cool
<wallyworld__> rick_h_: please continue to ask questions. fwiw, a lot of what is there in the screenshots we are working of we agree needs to be fixed
<wallyworld__> so were are using them as a starting point
<rick_h_> what would be sweet, would be if the grey boxes were buttons, and the edit link only showed on hover
<rick_h_> that would clean it up a bit readable wise imo
<wgrant> That was what I was sorting of thinking, yeah.
<wgrant> The edit links all over LP are really cluttersome.
<rick_h_> I know that hover stuff gets into the way of mobile, but I'm not sure how mobile friendly we are atm anyway
<rick_h_> wgrant: yea, I want to align them something fierce lol
<rick_h_> I've wished I had started back when the new buglist stuff had first started lol
<wgrant> Heh
<wgrant> Yeah
<rick_h_> I like my lines through my ui/design I guess, though I hate css grids...so go figure
<wallyworld__> rick_h_: lighter grey now done :-)
<rick_h_> ah, much nicer
<rick_h_> so why can I click on some and not others?
 * wgrant tries to dig up the faded edit icon.
<wallyworld__> rick_h_: atm, if you are in a team role, you get full observer rights (thats that the original screenshots called for). i'd have to check to see if that's still the case
<wallyworld__> s/team role/project role
<wgrant> wallyworld__: It's no longer the case.
<rick_h_> ah, default perms then
<wgrant> It's merely the default during project setup.
<rick_h_> gotcha
<wallyworld__> wgrant: np. i'll add the edit icons to those ones then
<wgrant> Aha, https://launchpad.net/@@/edit-grey
<wgrant> That's what I was looking for.
<wgrant> Slightly less abhorrent.
<wgrant> The initial inline bug status/importance used that until hover.
<wallyworld__> wgrant: i think anything is less abhorent than the yellow :-)
<wgrant> Surely not.
<wgrant> (I think ideally it would appear like a dropdown listbox, so it would have a down arrow instead of an edit icon. but lazr-js doesn't really do that sort of thing)
<wgrant> wallyworld__: I don't think it's useful to show "View nothing" rather than just hiding that policy. But then how do we add a new one.
<wallyworld__> wgrant: that's the dilemma
<wallyworld__> wgrant: we could have a proper popup form/widget
<wallyworld__> and a single edit icon in the table row
<poolie> wallyworld__, are the pencils on the permissions really helping anything?
<wgrant> Our JS UI toolkit is sort of limited :/
<wgrant> Yes, they're helping to lots of LP pages look crap :)
<poolie> also, having the bug/branch column have no text is a bit odd
<poolie> and the whole layout looks a bit oddly cramped
<wallyworld__> poolie: they are currently needed to allow the permissions to be edited
<poolie> technically needed?
<poolie> hm
<wallyworld__> poolie: the bug/branch column doesn't need text - it's the mechanism to navigate to the user's detailed disclosure page
<wgrant> wallyworld__: Well, might be useful to say "1234 bugs, 2 branches" or something like that.
<poolie> exactly
<poolie> and make that a link
<poolie> it just looks broken at the moment
<wallyworld__> can do that
<poolie> mm
<poolie> to make the top of the thing less cluttered
<wgrant> Also, those are really an attribute of the "restricted observer" grant for a particular policy.
<poolie> maybe move "add an observer" to the distinguished actions portlet in the top right?
<poolie> like for 'report a bug'
<poolie> "The list of observers can be large. Use Search to look for particular users.
<poolie> The search term will be matched against name, Launchpad id, email.
<poolie> "
<poolie> seems like it doesn't really need to be there
<wallyworld__> can do that too. the current layout is as per the supplied screenshots that were were given to start from
<poolie> it's pretty obvious what search will be for
<wgrant> wallyworld__: Are these mockups in a branch somewhere?
<poolie> the heavy black rule through the middle of the filters is a bit odd too
<poolie> might be better if all the filters were grouped within a box
<poolie> "Show users which can view the project because they belong to a team which has permission."
<wallyworld__> wgrant: lp:~launchpad-purple-squad/+junk/disclosure_mockups
<poolie> to start with surely you mean "who" not "which"
<poolie> but
<poolie> i don't know if that sentence is adding much
<poolie> or what it's supposed to mean
<wallyworld__> poolie: the black line is to separate the controls since the ones above the line cause a xerver call, the ones below filter in place
<poolie> :/
<poolie> ok
<wallyworld__> but rememeber, this is all exploratory, a starting point
<poolie> sure
<poolie> i'm exploring it :)
<poolie> i think, if the line was not there
<poolie> it would probably be clear that some of them act immediately and others need to be submitted
<wallyworld__> huwshimi: you may want to cut n paste the above discussion for feedback into the devel process
<wallyworld__> poolie: the line was added because users thought it wasn't clear
<huwshimi> wallyworld__: I'll have a look
<poolie> what specifically wasn't clear?
<wallyworld__> huwshimi: thanks. there's been some good feedback to add to whatever the user testers come up with
<poolie> mm
<huwshimi> wallyworld__: OK great
<poolie> i think it's a bit like the green link thing
<poolie> yes it means something
<poolie> but is it something users actually need to know about?
<wallyworld__> poolie: it wasn't clear that the bottom controls acted in place vs doing a submit
<poolie> if they need to know about it to use the thing effectively, it needs to be a distinction they'll actually understand
<poolie> hm ok
<wallyworld__> poolie: i sort of agree with you. i was just following orders :-)
<poolie> so they clicked it and
<poolie> ...
<poolie> trying to understand why that would be confusing
<wallyworld__> by that i mean i think it was indicated we need to distinguish between the two types of controls, but i could have mis remembered that
<poolie> i don't think that's a goal in itself
<poolie> the point is just that people can understand how to use it
<wallyworld__> so, so far we've gone from the original old screenshots to huwshimi's latest design which came from one round of user testing plus the ever evolving feature requirements
<poolie> nobody else specifically distinguishes them so i'm really skeptical that we need to
<wallyworld__> the end users haven't yet seen this 2nd iteration
<poolie> i think if we ask them "will foo cause a round trip to the server" they may not know but that's not a practical question
<wallyworld__> poolie: it's not so much a round trip to server thing vs a responsiveness thing. in-place obviously much quicker
<wallyworld__> but perhaps the distinction isn't required
<poolie> mm
<poolie> people will find out it's slow easily enough :)
<wallyworld__> like with the rest of lp untilthe recent performance improvements :-)
<wallyworld__> so, the idea now is to iterate to get to the point where the ui reflects the latest functional requirements. then we can bikeshed^H^H^H^H^H file tune :-)
<StevenK> wgrant: Can you have another look at https://code.launchpad.net/~stevenk/launchpad/use-userHasPriviledges/+merge/82967 ?
<wgrant> StevenK: Is DSP.driver necessary?
<wgrant> Also, I don't see any tests for SP/DSP.drivers
<wgrant> Apart from that it looks disturbingly sensible.
<StevenK> Disturbingly? :-/
<StevenK> I'm not sure about DSP.driver, I don't think so, since the codepath checks if it implements IHasDrivers and then grabs .drivers
<wgrant> Right.
<wgrant> But you seem to have added DSP.driver.
<StevenK> And .drivers. I am a muppet.
<StevenK> wgrant: Right, binned. Let me write a test for both.
<wgrant> Thanks.
<wgrant> Should be pretty quick :)
<huwshimi> wallyworld__: Which discussion exactly did you want me to look at?
<wallyworld__> huwshimi: just the feedback about the mockup which people saw after i sent the email to dev. eg search form, edit icons etc
<StevenK> wgrant: Diff updated
<wgrant> k
<wgrant> StevenK: Are you sure those new tests aren't just [] == []?
<StevenK> wgrant: Yes. I've added self.assertIsNot([], obj.drivers) where obj is the distro or the series and the tests still pass.
<wgrant> StevenK: k
<wgrant> Hah
<StevenK> wgrant: Do you want me to commit and push that change, or you don't care?
<wgrant> fhjwejiowefjio
<wgrant> StevenK: You know xfce4-smartbookmark-plugin?
<wgrant> How gina's been complaining about it for weeks?
<wgrant> StevenK: s/IsNot/NotEqual/, but yes.
<StevenK> Was that the non-free->main madness that the ftp-masters screwed up?
<wgrant> StevenK: No, it fails to unpack due to a patch referenced in series that doesn't exist.
<wgrant> It turns out that dpkg-dev is actually hideous.
<wgrant> And having reliably unpackable packages would be too easy.
<wgrant> There are vendor-specific series files.
<StevenK> Oh, right, that one.
<wgrant> So this unpacks fine on Debian, but not Ubuntu.
<StevenK> Bwaha
<wgrant> Because the bogus patch is mentioned in debian/ubuntu.series.
<StevenK> Sigh
<wgrant> Who could possibly think it was a good idea.
<wgrant> To make it ununpackable in that case.
<StevenK> So we need to patch dpkg to complain but not abort?
<wgrant> No, we probably need to slap whoever designed 3.0 (quilt) like that.
<wgrant> I'm not sure what dpkg-source can sensibly do here :/
<wgrant> I wonder if --no-preparation works.
<wgrant> It does not.
<wgrant> Ah, --skip-patches
<StevenK> We do not care about patches for gina, surely
<wgrant> Exactly.
<StevenK> wgrant: Not so fast though -- does Lucid's dpkg support --skip-patches ?
<wgrant> I'm about to find out.
<wgrant> It does.
<StevenK> wgrant: Perhaps we should get wallyworld__ to fix it. Then he can touch Soyuz and wish he didn't.
<wgrant> I wonder what happens if I transfer the unpacked tree from an Ubuntu machine to a Debian one.
<wgrant> StevenK: Do you feel like filing a bug in Debian?
 * wallyworld__ ducks for cover
<wgrant> Since I hate the BTS.
<StevenK> Against dpkg?
<StevenK> :-)
<wgrant> Hah
<wgrant> As much as I would like to consider the vendor-specific unpacking rules to be a grave bug, I don't think they'd like me much for that.
<StevenK> wgrant: Why don't you just configure reportbug?
<wgrant> STAB
<wallyworld__> poolie:  wgrant: i added the grey edit icons
<wgrant> wallyworld__: I doubt they look good, but they're probably less offensive.
 * wgrant checks.
<wgrant> Hah
<wgrant> The circle is just slightly visible.
<wallyworld__> less offensive is also what i'd say
<StevenK> wgrant: Those two extra asserts are in the diff.
<wgrant> But still looks much better.
<wallyworld__> yeah
<wgrant> With slight tweaking of the (otherwise unused, AFAIK) icon, it could look quite reasonable.
<wgrant> In fact, possible just drop the circle entirely.
<wgrant> StevenK: lol
<wgrant> StevenK: dpkg-source: warning: --skip-patches is not a valid option for Dpkg::Source::Package::V3::native
<wgrant> StevenK: So it warns if you ask for reliable extraction all the time.
<StevenK> Fucking hell.
<wgrant> It still works.
<wgrant> But spews warnings.
<StevenK> wgrant: However, here it unpacks, just exits 2
<wgrant> StevenK: Ah, is that so...
<wgrant> I didn't think about that.
<wgrant> I wonder what 2 means.
<StevenK> I wonder if I can claim serious
<StevenK> important, at the very least
 * wgrant watches for the wontfix
 * wgrant wonders why we don't have any running water.
 * wgrant goes hunting.
<StevenK> wgrant: debian/patches/series is a symlink to ubuntu.series
<wgrant> StevenK: In Ubuntu's version, or Debian's?
<StevenK> Debians
<StevenK> So Debian's xfce4-smartbookmark-plugin has a patch applied to direct people to launchpad to file bugs
<StevenK> Rather than the BTS
<wgrant> Which version?
<wgrant> In 0.4.4-1, debian/patches/series doesn't exist.
<wgrant> Because Debian's patches were included upstream.
<StevenK> I have 0.4.4-1 unpacked, and it's a symlink
<StevenK> Oh, that's dpkg-source being stupid
<wgrant> Hah
<StevenK> Nastygram filed
<StevenK> wgrant: Now that I've wrangled the BTS for you, can you wrangle my MP for me?
<wgrant> Sorry, got distracted by lack of water.
<wgrant> And dpkg-dev
<StevenK> wgrant: Thanks, tossing at ec2.
<poolie> wallyworld__, grey just looks kind of disabled
<wallyworld__> yeah, i guess so
<poolie> is there a technical thing that will break if there's no icon?
<wallyworld__> but final styling will come later
<poolie> or is it more just that we think people won't realize they can click it...?
<poolie> you get a hover underline
<poolie> that makes it pretty obviously clickable to me
<wallyworld__> i think people expect an icon
<poolie> huwshimi, remove bug tags woo
<wallyworld__> but i'll defer to the usability testing for what we need to do
<poolie> yeah, me too
<poolie> i just want the possibility of no pencils to be considered
 * StevenK stabs jtv
<poolie> the argument is basically just "we've always done that"
<jtv> What have I done now?
<jtv> Or are you just exploring different uses of pencils?
<StevenK> jtv: Inflicted that damnable buildd-manager branch on me
<poolie> haha
<jtv> SCORE!
<jtv> I was going to mention that.
<jtv> At some point.
<jtv> Almost certainly.
<jtv> Cross my heart and hope to be just outside my home when a slight drizzle starts.
<poolie> wallyworld__, so while we're waiting for the testing
<poolie> i don't know, is there any other reason than just people might expect it?
<StevenK> jtv: We've had 23mm here since 9am
<poolie> .. i don't want to be a pain
<poolie> i just really think it looks bad without adding anything
<poolie> we have to stop some day
<wallyworld__> poolie: well, i  would think we need to be consistent with the test of lp
<wallyworld__> s/test/rest
<StevenK> wgrant: Debian #649680, since the BTS sucks.
<_mup_> Bug #649680: Link titles of pages that aren't UTF8 appers as gibbrish <Hoborg IRC Bot:Fix Committed> < https://launchpad.net/bugs/649680 >
<poolie> cause the rest of lp has totally consistent ui?
<poolie> :)
<wallyworld__> mostly
<StevenK> poolie: I thought you knew!
<wallyworld__> more or less
<wallyworld__> perhaps
<wgrant> I think we need some indication that it's editable.
<poolie> consistency has value but
<wgrant> A pencil is probably not it.
<poolie> if you want to change things
<wgrant> It's really a list both.
<wgrant> List box.
<poolie> you've got to start somewhere
<wallyworld__> sure
<poolie> so one pattern people use is a little [v] icon when it pops up a control
<poolie> especially a list
<wgrant> Exactly.
<wgrant> That's exactly what this is.
<wgrant> But lazr-js doesn't really do that sort of thing.
<poolie> but really on many sites just hover underline/color/background seems to be enough
<wgrant> It prefers invoking a popup somewhere else on the page.
<poolie> it invites people to click and then they can see what happens
<wallyworld__> my thinking is that lp is being rebranded atm, so we should wait for that to establish the guidelines
<wallyworld__> we have other things to deliver
<poolie> fair enough
<wallyworld__> poolie: but i'm sure huw is open for suggestions on his rebranding work :-)
<poolie> :)
<jtv> StevenK: thank you for your sage comment on my branch.
<jtv> The bit about the vodka was particularly insightful.
<jtv> I think you're right.
<StevenK> About which bit?
<jtv> The vodka.
<lifeless> the vodka, clearly
<jtv> wgrant: do you drink vodka?  I'm asking because we need another expert opinion on https://code.launchpad.net/~jtv/launchpad/bug-717969/+merge/83022
<nigelb> A branch about vodka?
 * nigelb looks
<jtv> nigelb: nooooooo!  Don't!
<jtv> Poor man.  Went in without his Absolut goggles.  He'll never be the same.
<nigelb> Hahaha. Nice.
<nigelb> I stayed away from the code. So not scarred.
<nigelb> The only read your description and StevenK's rerview.
<nigelb> *review
 * jtv tearfully greets nigelb back
<nigelb> :)
<wgrant> Yay
<wgrant> UnicodeDecodeError in bson.loads
<StevenK> LIFELESS
<nigelb> ooh. Caps Lock.
<lifeless> StevenK
<StevenK> lifeless: I was just guessing the Unicode issue was from oops-tools or something
<poolie> hm so merging this loggerhead export patch raises the question of upgrading loggerhead as a whole
<lifeless> same as other source deps - land a change to LP to upgrade it
<poolie> ok
<poolie> tarball download works here; it should be ok to get it in tomorrow
<danhg> Morning
<rick_h_> morning
<allenap> Good morning rick_h_
<rick_h_> wallyworld__: wgrant is there a branch for the disclosure stuff I can see/find to checkout?
<cr3> sinzui: hola, I noticed your branch ~sinzui/html5-browser/trunk doesn't seem to exist anymore, which one should I be using instead?
<sinzui> !
<sinzui> I will find it and put it back
<cr3> sinzui: thanks man, it's been really nice for testing yui
<sinzui> cr3, I am helping another user. It will be a few minutes
<cr3> sinzui: no worries, take your time
<sinzui> cr3, can you see https://code.launchpad.net/~sinzui/html5-browser/trunk ?
<sinzui> I just branches lp:html5-browser
<sinzui> I just branched lp:html5-browser
<cr3> sinzui: seems to work, but it didn't work when building overnight: https://launchpadlibrarian.net/85743890/buildlog_ubuntu-precise-i386.launchpad-results_0.13-1%7Eppa0.12.04_FAILEDTOBUILD.txt.gz
<sinzui> ah. cr3, I think gary was changing lp's buildout rules. That was added recently. I created debs, but buildbot does not have them
<cr3> sinzui: the package seems to have built fine for 10.04 and 10.10 though, only later releases failed. anything I should do?
<abentley> rick_h_: I don't see assert_change being used anywhere.  Is it dead code?
<rick_h_> looking
<rick_h_> yes, changed to using the .wait() thanks
<sinzui> cr3. I will copy the package to precise. I may want to do that for a few other too
<cr3> sinzui: the build also failed for 11.04 and 11.10 though
<sinzui> I am not sure what to say, maybe gary_poster can help since he added the source deps
<gary_poster> what did I do?
<gary_poster> All I remember doing was adding html5_browser as a buildout dependency to lp
<cr3> gary_poster: can you have a look at this build failure, seems to be related to buildout: https://launchpadlibrarian.net/85743890/buildlog_ubuntu-precise-i386.launchpad-results_0.13-1%7Eppa0.12.04_FAILEDTOBUILD.txt.gz
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugtasks: 292
<gary_poster> but I didn't touch the debs
<gary_poster> sinzui, cr3, this seems to be the heart of the issue: http://pastebin.ubuntu.com/747089/
<gary_poster> I don't think I touched html5 browser's buildout either, though I could be wrong
<gary_poster> but I don't think that error has anything to do with buildout anyway
<sinzui> gary_poster, who put html5 in buildout? I only make deb packages
<cr3> gary_poster, sinzui: I'll try rebuilding in the ppa and then maybe try locally with sbuild
<cr3> sinzui: I put html5 in my buildout
<sinzui> ah, thank you for clarifying
<gary_poster> sinzui, I put html5 in LP buildout.  That has nothing to do with this error
<cr3> gary_poster: so we both have html5 in our buildouts, do you have a tarball in your sourcedeps branch or do you branch html5 directly in buildout?
<gary_poster> cr3 tarball
<cr3> gary_poster: maybe I should do the same...
<abentley> rick_h_: It's really helpful if you include my nick.  This can be a noisy channel, but using my nick makes your replies stand out.
<rick_h_> abentley: yea, sorry.
<rick_h_> I forget that not all irc has nick highlighting and such. I know some do notifications so try to avoid doing that too much
<rick_h_> thanks for the heads up abentley
<abentley> rick_h_: my IRC has nick highlighting, but your nick is the same colour as sinzui's.
<rick_h_> gotcha
<abentley> rick_h_: When you mention my nick, you get highlighted orange, and I get a bell.
<abentley> rick_h_: I try to avoid pausing in tests for two reasons: 1. such tests often fail intermittently, 2. such tests take longer than they otherwise would.
<abentley> rick_h_: what is the operation you're needing to wait for?
<rick_h_> abentley: agreed. I can see if I lessen the amount of test text I use the redraw on the clone element will be fast enough to not require it
<rick_h_> abentley: I believe it's the updating of the content from the target, to the clone, and getting that to resize before I can check it's new height value
<abentley> rick_h_: can you just change the code to be synchronous, i.e. to only return once the new value is set?
<rick_h_> it's going off off a YUI triggered event
<rick_h_> I'll see if I can find a way there and I'll trace the actual draw/etc timings to make sure where the pause is
<abentley> rick_h_: whatever is triggering the event will not return until all the event's listeners have been run.
<deryck> hmmm, wonder why we got some css changes on qastaging but not all.
<abentley> rick_h_: On line 358, you specify CHANGE_TIME as a third parameter to target.plug.  What does that mean?
<rick_h_> abentley: nothing, not sure how that got into that one call
<rick_h_> removed
<abentley> rick_h_: Cool
<abentley> rick_h_: could you change "ns" to "module" per https://dev.launchpad.net/JavaScriptReviewNotes#Javascript_Module_formatting ?
<rick_h_> abentley: sure thing
<deryck> abentley, rick_h_ -- can one of you guys check qastaging for me.  are the importance colors the same as they were?
<abentley> deryck: they look different.  "Medium" is light blue.
<deryck> abentley, ah nice.  So some odd caching on my end then.  cool.
<abentley> rick_h_: Do you need MAX_HEIGHT and MIN_HEIGHT to be separate variables, or would it work to do e.g. "min_height: {value: 10}"?
<deryck> man, getting those "columns" to line up with real data is going to be really hard to do.
<rick_h_> abentley: looking
<rick_h_> abentley: I'll need to find a different way to tell if a value has been passed in at object construction time.
<rick_h_> abentley: I'll add a test to picking up the min height from the current element (I think that's broken right now) but that needs to know if you passed a min height or not
<abentley> rick_h_: I think you could move that code into a valueFn.
<abentley> rick_h_: But why do want to pick up the min height from the current element?
<abentley> rick_h_: ISTM that the current element could be really big, and you could be reducing its size a lot.
<rick_h_> abentley: well if in the html of your textarea you say height=xxx, I want to respect that and not resize it down to the default
<abentley> rick_h_: aren't textareas usually configured in rows and columns rather than height and width?
<rick_h_> you can, but I think most move that to css these days
<rick_h_> so if the css says the height is 150, I'll use that as the new min if you didn't specify one at your .plug() call
<rick_h_> abentley: ^
<abentley> rick_h_: Okay.
<rick_h_> looking at the valueFn, if I have set a value, it seems this.get('min_height') would return value into my valueFn() call
<rick_h_> I'll check to make sure, if not, then I can remove the constant and move to a valueFn
<abentley> rick_h_: If you supply a value, the valueFn will not be used at all.
<abentley> rick_h_: If the valueFn return undefined, then the value will be used.
<rick_h_> "If both the value and valueFn properties are defined, the value returned by valueFn has precedence over the value property,"
<abentley> rick_h_: right.
<abentley> So, the precedence is supplied value > valueFN > value
<rick_h_> ok, I see. So using this.get('min_height') in the valueFn is safe because I know that no value was supplied
<abentley> rick_h_: in the valueFn, I don't see why you'd get('min_height') at all.  It would either return undefined or do infinite recursion.
<rick_h_> right, it would just check if there was a css style to use
<rick_h_> if so return it else undefined and value kicks in
<abentley> rick_h_: right.
<rick_h_> thanks, I like that a lot better
<abentley> rick_h_: someone writing YUI was thinking the same way you are.
<rick_h_> yea, I've not used valueFn and such. New toys!
<abentley> rick_h_: the condition at 186 looks very complicated.  Are you worried about the case where this._prev_scroll_height < this.get('min_height') ?
<rick_h_> abentley: no, if the prev_scroll_height < min_height it falls through to the else case and sets the height to the min
<rick_h_> I could wrap those into bool returning function calls with better names to help make it clearer perhaps
<abentley> rick_h_: how would prev_scroll_height be less than min_height>
<abentley> rick_h_: how would prev_scroll_height be less than min_height?
<rick_h_> abentley: _set_start_height should make sure that never happens
<rick_h_> but you asked if I was worried about that case, and even if it *were* to happen the else clause should cover it
<abentley> rick_h_: If it's safe to assume that this._prev_scroll_height < this.get('min_height'), then can't we rewrite the conditional as "if (scroll_height > this.get('min_height'))" ?
<abentley> rick_h_: Or maybe do new_height = Math.max(this.get('min_height'), Math.min(scroll_height, this.get('max_height')) ?
<rick_h_> abentley: yes, I think you're right. I set the default _prev due to a bug in testing and I think you're right that it greatly simplifies the conditions there now
<abentley> rick_h_: cool.
<abentley> rick_h_: You talk about preventing resize in webkit.  What happens in firefox?
<rick_h_> abentley: it doesn't allow users to drag/resize. Since it's based off a css property, I'd hope that if it's added they'd support the same standard
<abentley> rick_h_: yes, it does, in recent Firefoxes on Ubuntu.
<rick_h_> ah, you're right. And the css rule applies there
<rick_h_> abentley: so I'll generalize that comment then vs just calling out webkit
<abentley> rick_h_: okay.
<abentley> rick_h_: this._t_area is just meant as a more meaningful alias for this.get('host')?
<rvba> abentley: Can you please have a look at this MP? https://code.launchpad.net/~rvb/launchpad/revert-14355/+merge/83183
<rick_h_> abentley: yes, shorter to type, specific, and prevents calling .get() multiple times
<abentley> rvba: I can do it next.
<rvba> abentley: I'm reverting the "improvement" I've made to an expensive query because the performance is worse.
<abentley> rvba: doh!
<rvba> abentley: oh, allenap agreed to have a look at it right nowâ¦
<abentley> allenap: have at it.
<abentley> rick_h_: why does _setup_css take a parameter?
<rick_h_> abentley: no good reason.
<abentley> rick_h_: okay, please change it to use this._t_area, then.
<rick_h_> done
<rick_h_> abentley: you use the timeline tool in the chrome dev tools much?
<abentley> rick_h_: no, I don't use chrome much.
<rick_h_> ok
<abentley> rick_h_: what does it do?
<rick_h_> it shows the low level browser calls/events, layout, paint, event fired, etc
<rick_h_> I'm trying to use it to figure out why I need that pause
<rick_h_> that's ok, nvm. Just curious
<jcsackett> sinzui: would review for the mockup wallyworld and i whacked together be sending it over to dan and huw for comments, or ... ?
<sinzui> jcsackett, I think I did
<jcsackett> sinzui: ok. i'll throw the card into review then.
<jcsackett> i'll also take a quick look at that padlock icon on loggerhead; i think it's a quick fix.
<sinzui> fab
<abentley> rick_h_: Is it a case where "setStyle('foo', 'bar'); getStyle('foo')" does not return "bar"?
<rick_h_> abentley: it seems to be. I can see the event fired, but it doesn't show it hitting the event catching code
<rick_h_> and then I see it running a layout call because I called getStyle
<rick_h_> which should cause it to pick up the recalc, but it's like the event callback isn't done yet
<rick_h_> but like you say, it should be
<sinzui> I think the main reason I do not blog like it is 2005 is because the spammers comment like it is 2012
<abentley> rick_h_: What event?  this.t_area.on('valueChange' ?
<rick_h_> abentley: yes
<rick_h_> that callback has to complete before I can grab the updated height for the test to verify the change
<abentley> rick_h_: so if you set a breakpoint in resize, it never stops there?
<rick_h_> it does, but the timeline doesn't show it calling out to resize. What I'm trying to figure out is how the new value is getting pulled as the same as the original before the callback is complete
<abentley> rick_h_: so in FF, test_initial_resizable and test_max_height pass without the wait.
<abentley> rick_h_: the other two fail.
<rick_h_> abentley: it fails for me
<rick_h_> initial_resizable
<rick_h_> in FF
<rick_h_> I've commented out the rest and only working on trying to get the initial_resizable working without the wait
<rick_h_> in both browsers, without luck so far
<abentley> rick_h_: weird, it consistently works for me.
<abentley> rick_h_: FF 8
<rick_h_> abentley: ah, I'm on 7.0.1
<abentley> rick_h_: In initial_resizable, there's no event involved, is there?
<rick_h_> abentley: true
<rick_h_> it just runs resize on the first call after it determins min/max
<abentley> rick_h_: Oh, there are two waits in that one, and I missed the second.
<abentley> rick_h_: fails for me too, now.
<rick_h_> abentley: ah ok
<abentley> rick_h_: I track the event down to value-eventchange.js:157
<abentley> rick_h_: and it appears to do polling.
<rick_h_> abentley: gah, that makes a lot of sense then
<rick_h_> abentley: makes a lot of sense. If I changed the CHANGE_TIME to 50ms it would always pass, but under that it would fail, even with the wait
<rick_h_> and that's the POLL_INTERVAL
<abentley> rick_h_: In your tests, do you have a philosophical objection to firing the event yourself?  Or invoking the functions directly?
<rick_h_> abentley: the simulate code doesn't support valueChange
<rick_h_> abentley: so I had to mimic a keypress event which it does support
<deryck> abentley, hi. I realize you're on call today, but we need someone from our squad to get a deployment done today.
<rick_h_> abentley: the event code dupes the content to the clone so I could pull that out in order to make it callable for testing purposes
<deryck> abentley, and I think it's a bit too soon to ask that of rick_h_ :)
<abentley> rick_h_: I think that would be best.
<rick_h_> abentley: ok, I'll ajust for that and let you know when I get that into place
<abentley> rick_h_: Right now, it's a bit of an integration test, where YUI is part of the system under tests.
<rick_h_> sounds like deryck has plans for you :)
<rick_h_> abentley: agreed, thanks for pointint out the polling
<abentley> deryck: Okay, but it's not something I have much experience with, either, TBH.
<deryck> abentley, it's not too hard.  See the qa report here:  http://lpqateam.canonical.com/qa-reports/deployment-stable.html
<deryck> abentley, you just need to pester people to get qa done or do qa for anyone who is sleeping, if you can.
<deryck> abentley, we need to deploy up to r14369.
<deryck> abentley, and either I or flacoste can help a bit if you get stuck.
<deryck> abentley, then once the qa is caught up, you update the LPS page and ask a losa to do the deploy.
<abentley> deryck: I don't think we can deploy, because r14355 is bad, and the rollback hasn't landed yet.
<deryck> ah crap.
<deryck> flacoste, ^^
<abentley> deryck: that rolllback is pending, though.
<rvba> fwiw I've just reverted r14355 (r14374).
<deryck> so that's some better then.
<deryck> still probably 4-5 hours to the build completes then.
<deryck> abentley, that may be ok, then.  By the time you get all the qa chased up, that rev could be merged.  right?
<deryck> assuming buildnot plays along with us.
<rick_h_> can either of you load code.qastaging without a timeout?
<abentley> rick_h_: did it recently.
<rick_h_> I thought things were still 'not right' since I've not been able to load to look at my QA
<rick_h_> ok, I'll keep trying then
<abentley> rvba: Cool.
<abentley> deryck: Sure, though that means more revisions to QA.
<flacoste> deryck, abentley: so the next deployable revision will be 14374, but there is still some QA to go still
<deryck> abentley, yeah, sorry, just more to qa.  And you don't have to do it all either.  You can ask for help, for the original author to qa, etc.
<deryck> abentley, the idea is just that you'll be the one to poke people and prod the process along.
<mterry> flacoste, gary_poster, benji: barry suggested I ask you a question I have about zope.  I'm trying to fix a FTBFS for zope2.12 because the package is hardcoded to use python2.6.  It will build if I change that to 2.7, but do you have any ideas how likely that is to work run-time?  2.13 apparently explicitly added support for python2.7.  Do you know if that was a no-op or if there's a patch floating around I can use?
<abentley> rvba: what about 14366?
<rvba> abentley: I knew you would ask ;)â¦ I need the revert to be on qastaging to qa that revision.
<gary_poster> mterry, I'm afraid I have no idea--I haven't had any connection to Zope 2 in a very long time.  benji is not around today, but probably is similar.  I can give you two emails of people I'd recommend asking in a privmsg.  Would that be helpful?
<mterry> gary_poster, sure, thanks
<abentley> rvba: I don't understand.  It looks like it needs a rollback or a bugfix.  Are you doing either one?
<rvba> abentley: Well, I can qa it on anther pageâ¦ but the page I'd like to see now issued a very bad query that makes the page time out.
<rvba> s/issued/issues/
<abentley> rvba: it's marked qa-bad.
<abentley> rvba: it doesn't sound like it needs qa.
<rvba> abentley: both revisions target the same bug.
<rvba> abentley: r14355 is qa-bad (I've reverted it), the other one needs qa.
<rvba> abentley: https://bugs.launchpad.net/launchpad/+bug/867941
<_mup_> Bug #867941: person:+activereviews times out <affects-ubuntu> <code-review> <qa-bad> <timeout> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/867941 >
<rvba> abentley: not sure how to deal with that situationâ¦
<abentley> rvba: Ah.  CAN WE PLEASE HAVE REVISION BASED QA?  PRETTY-PLEASE?
<rvba> Right!
<deryck> rick_h_, this matches the new spinner you added, right?  http://people.canonical.com/~deryck/spinner-big.gif
<abentley> rvba: So once your revert is landed, you'll be able to QA the other revision, which may or may not be bad.
<rvba> abentley: right. Please take into consideration the fact that I'm almost EODed, but I can come back later this evening to do that qa.
 * deryck see doing a deploy has driven abentley to shouting :)
<rvba> hehe
<abentley> deryck: This is how I always feel when I have to interact with the deployment reports.  They don't tell me what I want to know, and they don't tell me the truth.
<rvba> abentley: waterfall says the ETA for the build is ~2h.  How long after that for the revisions to be deployed to qastaging?
<flacoste> rvba: if you leave instructions on how to do the qa, i can take care of it
<abentley> rvba: I don't know how long that takes.
<abentley> rvba: I guess we could ask a losa to help the process along.
<rvba> flacoste: thanks for the offer, I'll write the instructions on the mp, but I'll nonetheless pop up in 2h30 min to see if I can do the qa myself.
<flacoste> ok, thx
<rick_h_> deryck: yes, that matches
<rvba> flacoste: abentley I've just updated the mp https://code.launchpad.net/~rvb/launchpad/activereviews-bug-867941-eagerload2/+merge/82904. Right now the page it talks about on qastaging is not accessible because it times out (hence my previous revert of an attempt to improve one of the huge queries).
<flacoste> rvba: perfect!
<deryck> rick_h_, thanks!
<deryck> my xchat indicator seems to have died.
<rvba> I'm off now but I'll be back later tonight to see if I can do my qaâ¦
<abentley> deryck: https://code.launchpad.net/~benji/launchpad/bug-877195-code/+merge/80619 does not have enough information for me to do QA, and benji isn't around.
<abentley> rick_h_: Have you been able to do QA on lp:~rharding/launchpad/bug_814697_missing_spinner ?
<rick_h_> abentley: no, I can't get code.xxx to load other than the root page
<rick_h_> I've been trying the last hour
<abentley> rick_h_: https://code.qastaging.launchpad.net/bzrtools/+activereviews works for me.
<abentley> rick_h_: also https://code.qastaging.launchpad.net/bzrtools/+merges
<rick_h_> abentley: thanks, looking for a good sample case now
<rick_h_> gah, this is nuts
<rick_h_> abentley: so do the branches actually exist on qastaging if I wanted to "fake" enough to get something to QA?
<rick_h_> I've tried three different ones and just get "error not a branch" from bzr
<abentley> rick_h_: yes, pushing to qastaging should work.
<rick_h_> ok
<abentley> rick_h_: bzr push lp://qastaging/~abentley/bzrtools/monk works for me.
<rick_h_> ok, I was trying to pull from an existing merge request branch.
<rick_h_> I'm probably doing more wrong, working on setting up something to be able to test with
<abentley> rick_h_: no, only a few branches get copied across.
<rick_h_> abentley: ok, thanks
<abentley> rick_h_: so you need to push fresh ones, but that will work.
<rick_h_> abentley: ok, will try that
<rick_h_> abentley: any way to foce the make sync_branches on qa next?
<rick_h_> or should I just be patient
<abentley> rick_h_: yes, ask a losa to run the appropriate script.
<jcsackett> abentley: can i get a quick look at https://code.launchpad.net/~jcsackett/loggerhead/add-lock-icon/+merge/83203? it's very short.
<abentley> jcsackett: certainly.
<jcsackett> thanks.
<abentley> jcsackett: r=me
<jcsackett> thanks, abentley.
<deryck> abentley, how are we doing on prepping deploy?  Just waiting on rollback to work it's way through now?
<abentley> deryck: No, https://code.launchpad.net/~benji/launchpad/bug-877195-code/+merge/80619 does not have enough information for me to do QA, and benji isn't around.
<deryck> abentley, hmmm, perhaps one of his squad mates could help us? gary_poster or bac, any idea?
<abentley> rick_h_: You don't need to resubmit.  You can just push and tell me you've updated it.
<rick_h_> abentley: updated MP per our discussions this morning: https://code.launchpad.net/~rharding/launchpad/bugfix_891735/+merge/83217
<rick_h_> abentley: ah ok, it wasn't showing the changes after a push so I thought I had to resubmit it
<abentley> rick_h_: It should show that the diff is updating after a push, but it's easy to miss.
<bac> deryck: let me look
<deryck> bac, ok, thanks.  there are qa notes, but I'm guess abentley doesn't know the problem well enough to follow the notes.
<bac> deryck: i just saw that
<gary_poster> thanks bac
<abentley> rick_h_: bind_events doesn't do a lot.  Would it make sense to merge it into initializer?
<rick_h_> abentley: it can. I tend to move bindings out for clarification and a home to group any/all in one place
<abentley> rick_h_: It's no biggie.  When I see a short single-call-site callable, I tend to move it to the callsite.
<rick_h_> abentley: moved and pushed, no problem at all.
<abentley> rick_h_: On line 176, why set this._prev_scroll_height = this.get('min_height')?  Why not set it to the actual size, for example?
<sinzui_> flacoste, ping
<flacoste> hi sinzui_
<rick_h_> abentley: looking
<sinzui_> flacoste, I *had* included the date of 2011-11-27 in the email to project owners to say when we would stop supporting shared private bugs...
<sinzui_> flacoste, We are not technically stopping now, we are preventing users from creating new ones...
<abentley> rick_h_: Or perhaps allow it to be undefined, and let the first call to resize set it?
<flacoste> sinzui_: right
<rick_h_> abentley: so if your textarea is 40px tall, but you set a min of 100, I need it to be different so resize kicks in
<rick_h_> if I set it to undefined...I'd have to see what Math.max considers undefined
<sinzui_> but I think I should mention that there will be a bug linking feature in the future and we expect to migrate the remaining shared bugs when the feature is available
<sinzui_> ^ flacoste
<abentley> rick_h_: this._prev_scroll_height is not an input to Math.max AFAICT.
<rick_h_> abentley: ok, so tests pass with it at undefined
<rick_h_> abentley: no, you're right, I was thinking the other way around. min_height
<abentley> rick_h_: Ah.
<rick_h_> abentley: ok, removed it entirely, tests all pass with it set to default of 0 as a start point
<abentley> rick_h_: at 51, I think a second "var" would be clearer than having variable declaration spanning lines.
<flacoste> sinzui_: yes, that would be good
<sinzui_> okay
<rick_h_> abentley: ok, not a fan of the mutliple vars at a time?
<abentley> rick_h_: I'm fine with them if they don't span multiple lines.
<rick_h_> oh heh, I'd hate them on one line. Ok,
<rick_h_> updated
<abentley> rick_h_: at 53, please use Y.Lang.isUndefined or Y.Lang.isValue rather than !== undefined.
<abentley> rick_h_: Also, please indent only 4 spaces at 54.
 * rvba takes a look at qastaging to see if r14374 has been deployed.
<rvba> Not yet it seemsâ¦
<rick_h_> abentley: sounds good
<abentley> rick_h_: Your comments have more lines at the bottom than I'd expect.  I'd expect just "*/" after the last line of text.
<rick_h_> abentley: gotcha, sorry, the python docstring rules coming out in me
<abentley> rick_h_: Our docstrings have """ after the last line of text.
<rick_h_> abentley: oh, could have sworn pep is one blank line at the end for readability. I missed that so far
<abentley> rick_h_: I haven't consulted the PEP, but I think that's how we always do it.
<rick_h_> abentley: ok, thanks for the heads up. Looks like it's more a "suggestion" in the pep 257 for it "The BDFL [3] recommends inserting a blank line between the last paragraph in a multi-line docstring and its closing quotes"
<abentley> rick_h_: Interesting.
<rick_h_> abentley: little retraining...to be expected :)
<abentley> rick_h_: please remove one newline around 172.
<rick_h_> abentley: sure thing
<abentley> rick_h_: At 190, our style looks like this: http://pastebin.ubuntu.com/747478/
<abentley> rick_h_: i.e. if all the arguments don't fit on a line, we do a line break and indent.  This is not PEP8, but it's what we do.
<rick_h_> abentley: ok, since that's still long I'd wrap to a thired line at the Math ok?
<rick_h_> abentley: sure thing
<abentley> rick_h_: sure, that's fine.
<rick_h_> http://pastebin.ubuntu.com/747480/ abentley so first/second preferred?
<rick_h_> or doesn't matter
<abentley> rick_h_: first.
<rick_h_> abentley: thanks
<abentley> At 271, I've see it in other test suites, but I'm pretty dubious that saving two keystrokes is a win.
<rick_h_> abentley: ?
<abentley> if you do "var Assert = Y.Assert;"  you can save two keystrokes by doing "Assert.areEqual" rather than "Y.Assert.areEqual".
<rick_h_> abentley: ah, ok, yea I grabbed that out of other tests. Honestly, I'd prefer to just pull in the tests themselves and avoid the lookup costs of going through the object
<abentley> rick_h_: Anyhow.  No big deal either way.
<rick_h_> abentley: no problem, find and replace is my friend.
<rick_h_> updated and pushed
<abentley> rick_h_: 341, why not Assert.areSame?
<abentley> rick_h_: test_removing_content looks sensitive to browser settings, but maybe that's inevitable.
<rick_h_> abentley: no reason, just inexperience with array of assert methods
<abentley> rick_h_: Ah.  Well, since we have it, let's use it.
<rick_h_> abentley: how so? on the browser settings
<abentley> rick_h_: You're asserting that the height of test_text will be greater than 100 px, and "shrink" will be <= 100 px, but both seem to depend on font settings.
<rick_h_> abentley: ah true, I tried to pick values that would not be drastic enough to change much, but perhaps I can do better with finding smaller test values for those
<abentley> rick_h_: I don't think it matters in a practical sense.
<abentley> rick_h_: Oh, 273 is excessively long.  That should be split up into a list and joined: https://dev.launchpad.net/JavaScriptReviewNotes#Plain_Old_JavaScript
<abentley> rick_h_: have you run make lint on this?
<rick_h_> abentley: I've run jslint myself, I've not found the lint tool run during testing
<rick_h_> abentley: so I was going to ask on the long block of text, would it be better to put into the test html and pull it out vs in the JS?
<abentley> rick_h_: Sorry, don't understand "not found the lint tool run during testing"
<rick_h_> I've seen refernce to a lint tool that's run during the test/merge/etc process
<rick_h_> abentley: no, I've not run lint on it
<abentley> rick_h_: right.  It's bin/lint.sh, but usually run as "make lint".
<abentley> rick_h_: I'd prefer to keep the long block of text in the JS.
<rick_h_> abentley: ok
<abentley> rick_h_: at 379, "n" is too short as a variable name.  I suggest "node": https://dev.launchpad.net/JavaScriptReviewNotes#Plain_Old_JavaScript
<rick_h_> abentley: ok
<abentley> rick_h_: please remove the newline at 453.  And I think that's everything.
<rick_h_> abentley: can you give me some context around 453? I've been a few lines off and not sure where you mean
<abentley> rick_h_: I'm sorry.  443: http://pastebin.ubuntu.com/747500/
<abentley> rick_h_: we don't go for multiple blank lines except at module level, generally.
<rick_h_> abentley: yes, my fault completely. Just wasn't looking at the filedupe file
<abentley> rick_h_: np.
<rick_h_> abentley: thanks, changes all pushed
<abentley> rick_h_: This was far more thorough than reviews typically are, but since you'll want to get up to speed with our style, I indulged my inner nit-picker.
<rick_h_> abentley: definitely, it's completely what I was expecting
<rick_h_> though I had thought I tried to catch things a bit better, so sorry for all the issues
<rick_h_> appreciate you letting me time sink your day
<rick_h_> now I'll go get a drink and lick my wounds :)
<abentley> :-)
<abentley> rick_h_: it was the same for me.
<abentley> rick_h_: r=me.
<rick_h_> abentley: ty much sir
<poolie> hi all
<poolie> oh huw, your tags list is so soothing
<huwshimi> poolie: :D
<huwshimi> poolie: Looking forward to that being deployed :)
<poolie> me too
<poolie> huwshimi one interesting thing is on eg https://bugs.qastaging.launchpad.net/bzr/+bugs it shows some tags with 0
<poolie> i guess they are official but there are no remaining open bugs
<poolie> this is reasonable
<wgrant> poolie: I believe that's always been the case. It just wasn't obvious before because the counts weren't shown.
<wgrant> Ooh
<wgrant> Beta banner.
<jelmer> wgrant: ooh, where?
 * jelmer is still wondering how to add that for mercurial imports
<wgrant> jelmer: mmm, not sure how well it would work for imports. see eg. https://bugs.qastaging.launchpad.net/launchpad/+bugs
<jelmer> wgrant: I don't see anything there
<wgrant> You may need to be in ~custom-buglisting-demo
<poolie> yeah that is nice
<jelmer> hmm, that looks good indeed
<jelmer> takes up quite a bit more vertical space per bug than previously though
<wgrant> Yeah, and it's a lot less scannable.
<jelmer> I think the beta banner would actually work for mercurial imports - we could show it on the branch page of imports, and just add a "(beta)" bit after the choice in the import creation menu.
<wgrant> jelmer: True.
#launchpad-dev 2011-11-24
<StevenK> Is twisted.internet.error.CannotListenError: Couldn't listen on any:2121: [Errno 98] Address already in use.
<StevenK> Is ^ on ec2 normal?
<wgrant> StevenK: It started appearing a week or two ago.
<wgrant> But yes.
<wgrant> Hmm.
<poolie> StevenK, see my mail a week ago
<wgrant> We should probably present a subtle beta notification to everyone when there's something optional on a page, and allow them to opt-in then and there.
<poolie> wgrant, i agree with the ammendment that not every beta is going to be available to every person
<poolie> necessarily
<wgrant> poolie: Right, a feature flag to turn on the option for certain teams, I guess.
<poolie> perhaps plenty of them arethough
<wgrant> IIRC Gmail did something like this years ago.
<wgrant> Not sure if they still do.
<wgrant> Labs or something, but seems Google discontinued all of that.
<poolie> yes, google love it
<poolie> they do things i really envy
<poolie> such as correlating this with their logs
<poolie> to see if it makes things slower, crashier, more popular
<wgrant> I hope we can get to a point eventually where that would be useful.
<wgrant> Right now it's not.
<poolie> stevenk https://bugs.launchpad.net/launchpad/+bug/894205
<_mup_> Bug #894205: spurious test_poppy failures <poppy> <soyuz> <spurious-test-failure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894205 >
<wgrant> We need an ec2test -D
<poolie> for what?
<wgrant> So we can break into the instance once it hits that failure.
<wgrant> And see what's going on.
<poolie> oh postmortem
<poolie> can't be that hard
<poolie> wouldn't ec2 test --attached -t '-D' do the job?
<poolie> except perhaps you don't want to do that every time just in case it fails
<wgrant> Or perhaps if we just immediately emailed on failure.
<wgrant> Or indeed just ran a few instances and watched ec2 list
<poolie> yeah
<poolie> i thought ec2 list used to give a progress percentage
<wgrant> No
<poolie> but apparently not any more?
<poolie> well
<wgrant> It never did.
<wgrant> Just a time.
<poolie> maybe i'm thinkingo f bzr pqm
<poolie> i guess once you're experienced it's enough to know it takes about 4h
<wgrant> Yeah
<poolie> the other thing i would like to do here is make the flags admin ui a bit better
<poolie> oh and developer control on *staging
<poolie> so much to do
<wgrant> wallyworld__: What if we said something like "Albert Einstein | [Proprietary: everything (!)] [Security: 6 bugs, 4 branches (!)] [(+)]" (where (!) is the edit icons, (+) is the add icon)
<wgrant> wallyworld__: Gets rid of the obscure "observer" and "restricted observer" terms.
<wgrant> Removes the third column.
<wgrant> And shows more directly what's what.
<wallyworld__> wgrant: sounds reasonable at first glance
<poolie> wgrant, so
<poolie> self.factory.makeBugComment()
<poolie> fails if i don't previously log in as a user
<poolie> is this a bug? it seems inconsistent with other factory things
<wgrant> poolie: A bug, but an unsurprising one.
<poolie> this is from https://code.launchpad.net/~mbp/launchpad/888353-microformats/+merge/82767 circa line 59
<wallyworld__> StevenK: wgrant: have you used ec2 test recently? does it work for you?
<StevenK> ec2 test and ec2 land are mostly the same
<wallyworld__> yes, that's what i thought. but first ec2 test complained there was no submit branch
<wallyworld__> so i added one to branch.conf
<wallyworld__> then it gets to starting ec2 and complains there's a missing revision
<StevenK> What are you trying to do?
<StevenK> Oh, you do everything in one working directory, don't you?
<wallyworld__> just try ec2 test to confirm it still works for me
<wallyworld__> i have a lightweight checkout
<wallyworld__> i guess it needs submit_branch=... so that it knows where to grab the code from inside the ec2 instance
<wgrant> wallyworld__: I used ec2 test 12 hours ago.
<wgrant> Worked fine.
<wallyworld__> hmmm. don't know what broken for me then :-(
<wgrant> wallyworld__: rocketfuel-setup adds a submit_branch
<wgrant> To ~/.bazaar/locations.conf
<wallyworld__> wgrant: mine has a submit_branch:policy = appendpath
<wgrant> Er
<wgrant> really?
<wgrant> public_branch:policy and push_location:policy should be appendpath
<wgrant> submit_branch should not.
<wallyworld__> yeah, it's been that way forever. maybe is mis-copied from somewhere
<wgrant> http://paste.ubuntu.com/747751/ is what I use.
<wallyworld__> i'll try those settings. but i'm sure ec2 test used to work
<wgrant> ec2 land will work, since it gets it from the MP
<wgrant> ec2 test uses submit_branch.
<wallyworld__> right, makes sense
<wallyworld__> wgrant: so now the code in ec2 says public branch out of date
<wgrant> wallyworld__: The public branch is probably out of date :)
<wgrant> Or wrongly configured.
<wallyworld__> yet, i'm running ec2 test from a branch i've just bzr pulled to
<wgrant> It has the same name?
<wallyworld__> as what?
<wgrant> Your local branch and LP branch need to have the same name.
<wgrant> Or you need to configure public_location explicitly.
<wallyworld__> ah, that may be it
<poolie> i have been using ec2 test a lot recently
<wallyworld__> my public_branch is bzr+ssh://bazaar.launchpad.net/~wallyworld/launchpad/devel
<wallyworld__> my subit branch is bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
<wgrant> Ah
<wgrant> There's the problem.
<wgrant> public_branch needs to the be the public URL of the current branch.
<wgrant> That's why it uses appendpath
<wgrant> so my observer-db-2 branch maps to bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/observer-db-2
<wallyworld__> wgrant: so, how to fix? edit ~/.bazaar/locations.conf?
<wgrant> wallyworld__: Right.
<poolie> if this is misconfigured i don't understand how ec2 would be working for you at all
<poolie> or are you always giving the full url?
<wgrant> Indeed, even ec2 land should fail here.
<wgrant> If public_branch is wrong.
<wallyworld__> it's all worked so far for whatever reason
<wallyworld__> i have a single working tree and use bzr switch
<wallyworld__> if the working tree has the current branch i want to land, i just type ec2 land
<wallyworld__> if the working tree is switched to a different branch, i used ec2 land fullurl
<wallyworld__> wgrant: so my locations.conf is the same was yours, yet i get the public and submit branch mismatch
<wgrant> wallyworld__: Right, I use multiple working trees.
<wgrant> wallyworld__: Where are your branches?
<wgrant> Directly in lp-branches?
<wallyworld__> yes
<wgrant> Hmm
<wallyworld__> and the branch directories are empty
<wgrant> That should still work.
<wallyworld__> except for .bzr
<wgrant> wgrant@lucid-test-lp:~/launchpad/lp-branches$ bzr co --lightweight rip-out-accountpassword rip-out-accountpassword-co
<wgrant> wgrant@lucid-test-lp:~/launchpad/lp-branches$ bzr info rip-out-accountpassword-co | grep public public branch: bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/rip-out-accountpassword
<wgrant> WFM :/
<wgrant> You don't have it configured in branch.conf?
<wallyworld__> my branch.conf has parent_location
<wallyworld__> to point back to the parent devel chechout on disk which i pull into from lp and then branch from when i create a new branch
<poolie> [/home/mbp/launchpad/lp-branches/work/.bzr/branches/]
<poolie> submit_branch = bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
<poolie> [/home/mbp/ianb/lp-branches/] actually
<wallyworld__> that matches my submit_branch when i type bzr info
<wallyworld__> but ec2 test complains that something is out of date
<poolie> what specifically?
<wallyworld__> bzrlib.errors.PublicBranchOutOfDate: Public branch "bzr+ssh://bazaar.launchpad.net/~wallyworld/launchpad/devel" lacks revision "launchpad@pqm.canonical.com-20111123175547-p02iyjil3n13end4"
<wallyworld__> which it does if i goto ~wallyworld/launchpad/devel
<wallyworld__> but i'm running ec2 test from a branch which is fully up to date via bzr pull
<poolie> are you wanting to run the tests on devel?
<wallyworld__> i just wanted to get ec2 test working, because i ec2 demo is based off that and i want to see if ec2 demo works
<poolie> there is a special 'ec2 test --trunk' option that doesn't do a merge
<wallyworld__> sounds like my config is hosed somehow, yet ec2 land, bzr pull, bzr push etc all work
<poolie> wallyworld__, i think your setup has a confusion between a branch called devel owned by you, and the real devel owned by launchpad-pqm
<poolie> it's kind of bad this state is possible
<poolie> the short answer is to either use --trunk or test one of your own real branches
<wallyworld__> i tried one of my own real branches and got similar errors
<wallyworld__> --trunk is fine for ec2 test but i really wanted to see if ec2 demo works with a branch containing some changes
<wallyworld__> since it was stated ec2 demo is broken, and i wanted to see why/how
<poolie> really?
<StevenK> wallyworld__: ec2 test should work fine
<StevenK> Since ec2 land runs test with details from the MP
<poolie> ec2 demo also works fine for me
<poolie> wallyworld__, give me the name of one of your branches?
<wallyworld__> poolie: someone stated on the thread i started about interactive demos that ec2 demo was broken
<wallyworld__> poolie: delete-all-bugtasks-889202
<wallyworld__> StevenK: ec2 land works fine for me. but ec2 test doesn't (i haven't run it in ages, and tried a bit earlier for the first time in a while)
<wallyworld__> perhaps my workflow of having a single local checkout of devel and branching off that and using a single working tree which i switch between isn't very common
<StevenK> Ya think?
<poolie> i think it's common
<poolie> i use colos which are similar enough
<poolie> wallyworld__, can you be more specific about what's going wrong
<poolie> do you get a traceback or something?
<StevenK> steven@liquified:~/launchpad/lp-branches% ls -1 | wc -l
<StevenK> 130
<wallyworld__> i have all my branch dires in lp-branches too
<wallyworld__> but use devel-sandbox as my only dir with a working tree
<poolie> that's fine
<poolie> there is no good reason why that would affect ec2
<poolie> lifeless, what actually is bitrotten in ec2 demo?
<wallyworld__> poolie: i tried it again in my devel-sandbox with a previous branch and it worked this time
<wallyworld__> poolie: but it didn't work in my local trunk working tree that i use to branch from
<poolie> so
<poolie> that's probably best fixed by adding this configuration
<poolie> [/home/ianb/launchpad/lp-branches/devel]
<poolie> public_location = lp:launchpad
<poolie> so it knows it's just a mirror
<wallyworld__> ah, ok. that's great. i'll try it. thanks :-)
<wallyworld__> poolie: that worked. the key though is public_branch. thanks.
<poolie> wallyworld__, all good now?
<poolie> i might have some lunch
<lifeless> wallyworld__: I work as you do
<lifeless> wallyworld__: I think its pretty common
<wallyworld__> saves a lot of time checking out, and disk space too
<wallyworld__> s/checking out/branching
<mwhudson> to be pedantic, i think you do mean checking out
<mwhudson> or tree building
<mwhudson> branching is not the bit you wait for :-)
<wallyworld__> it is if you branch from lp:launchpad rather than a local ondisk copy
<wgrant> No.
<wgrant> Well, only if you don't have a shared repo.
<wgrant> In which case you are wrong.
<mwhudson> if you're using a shared repo, you're doomed :)
<mwhudson> +not
<wgrant> Exactly.
<wallyworld__> maybe i misremembered then. i could have sworn branching off lp:launchpad took a fair bit long than branching from local
<wgrant> Well, that's orthogonal.
<wgrant> You can branch locally perfectly well without colo.
<wgrant> bzr branch devel somethingelse
<wallyworld__> right
<StevenK> steven@liquified:~/launchpad/lp-branches% du -sh .
<StevenK> 21G	.
<StevenK> Crumbs
<wgrant> StevenK: How!?
<wgrant> Do you never delete branches?
<StevenK> Not usually
<wallyworld__> i don't delete branches either and mine is 1.5GB
<wallyworld__> and that includes a few non lp branches
<wallyworld__> ls -1 | wc -l -> 254
<nigelb> that's crazy.
<nigelb> I delete branches after they're deployed.
<wallyworld__> nigelb: because i only use one working tree, the branches are just directories, and i keep them around locally for easy reference
<nigelb> Nice.
<nigelb> I ran out of space on my hard-disk.
<nigelb> Too much code :)
<StevenK> /dev/mapper/sys-home   46G   39G  4.7G  90% /home
<StevenK> But I have another 390Gb that is unallocated. :-)
<nigelb> I need to buy more hard-disk space.
<nigelb> Going to work with lots of code soonish.
<nigelb> Morning poolie!
<nigelb> Err, Afternoon, rather :)
<poolie> hey there
 * StevenK deletes 10Gb of old branches
<wallyworld__> StevenK: wgrant: attempting to make a multi-pillar bug private - do you agree that it should raise ValueError or do you prefer a custom exception type?
<wgrant> A custom type.
<wallyworld__> ockey dokey
<wgrant> OMG
<wgrant> Python 3.3 will have slightly less stupid Unicode.
<wgrant> It won't just support the BMP.
<StevenK> You mean like Perl 5.10 YEARS ago?
 * nigelb sees StevenK's subtle stab.
<wgrant> Bah, no jtv.
<StevenK> (Pdb) p self.errors
<StevenK> [WidgetInputError('owner', u'Maintainer', ConstraintNotSatisfied(<Person at 0xd3471d0 foobarbaz (Foobarbaz)>))]
<StevenK> :-(
<wgrant> Hmm?
<StevenK> Product:+edit-people
<wgrant> Sure. But why is that saddening?
<StevenK> Because the error that is set is "Constraint not satisfied"
<StevenK> It's due to the validator failing, but I'm not sure how to deal with it properly.
<wallyworld__> StevenK: could be trying to assign an open team as a pillar owner
<StevenK> I am!
<StevenK> That's the point.
<StevenK> Just the error message is horrible.
<wallyworld__> that's how it bubbles up from storm i guess
<StevenK> Right
<wallyworld__> we should catch that and rethrow
<StevenK> wallyworld__: Catch it where?
<StevenK> It's already in self.errors in the validate() of the view.
<wallyworld__> hmmm. not sure then off hand
<wallyworld__> in the model code somewhere
<wallyworld__> so that by the time the view sees it, it is a nice error
<StevenK> Can you think of an example?
<wallyworld__> doesn't validate() take the form values prior to executing the form action?
<wallyworld__> so a check could be done there that the owne isn't open
<wallyworld__> and the field error set accordingly
<wallyworld__> but the vocab shouldn't allow such a team to be picked anyway
<StevenK> wallyworld__: The first thing validate() does is 'if len(self.errors) >= 1: return'
<wallyworld__> so my guess as to how validate works is wrong :-(
<wallyworld__> seems backwards
<wallyworld__> shouldn't validate happen first, then the action
<StevenK> It does happen that way
<StevenK> wallyworld__: I can explain this easier over mumble, since it seems I've confused you.
<wallyworld__> StevenK: let me look quickly at the code
<StevenK> wallyworld__: The view in question is ProductEditPeopleView
<wallyworld__> StevenK: so looks like custom_widget('owner', .....) needs to have the correct vocab passed in
<wallyworld__> so that only closed teams or people can be picked
<StevenK> wallyworld__: But I didn't use the picker to generate that error
<wallyworld__> you typed into the field directly?
<StevenK> And hit the button, yes
<wallyworld__> ok. so the picker vocab is still wrong :-)
<StevenK> Right, so I can fix that at the same time, but my question still stands
<wallyworld__> but i don't understand how validate() has errors already set
<wallyworld__> at the very start of the method
<wallyworld__> when no validation has been done yet
<StevenK> LaunchpadFormView has done it? Or storm's\?
<wallyworld__> perhaps. it seems apparent something is having a go at validation before the view nominated validate() is called
<StevenK> Agreed.
<wallyworld__> based on the form schema
<wallyworld__> and the declared fields
<wallyworld__> but that's the limit of my knowledge
<StevenK> It seems ugly to just reach into self.errors and pull out ConstraintNotSatisfied and replace it
<wallyworld__> yup. let me check something
<wallyworld__> StevenK: so the storm validator for closed teams raises a OpenTeamLinkageError
<wallyworld__> not sure how this is munged into a constraint error
<StevenK> By Zope, I guess
<StevenK> I have no idea how validators work
<wallyworld__> i do seem to recall a doc test checking for OpenTeamLinkageError
<wallyworld__> rather than ConstraintError
<wallyworld__> sadly, i have in the past set breakpoints inside LaunchpadFormView to track the flow when a form submit is done
<wallyworld__> you may need to do the same here
<wallyworld__> bbiab
<huwshimi> wallyworld__: I've just sent a somewhat lengthy reply to the list about prototyping. Once you've had a chance to read it I'm happy to have a bit more of a chat here if it helps (not saying you should read it now).
<wgrant> jtv: Evening.
<jtv> Hi wgrant
<jtv> Sorry I had to email; internet connection where I was was just unusable.
<wgrant> jtv: Did you end up testing recipe builds on DF?
<jtv> Yes.
<jtv> Ahem.
<jtv> Not on DF.
<jtv> On staging.
<wgrant> Ah, I just tested on DF anyway.
<wgrant> Looks good.
<jtv> On staging as well.
<wgrant> jtv: there is a lot of commit/DatabaseTransactionPolicy/dosomething/commit which seems like it could be done in a single context manager. You also unexplainedly abort in a few places.
<wgrant> It's all a bit of a mess, but I guess that can't really be helped...
<wgrant> Without the rewrite that can't happen.
<jtv> Right. Ideally, I'd like to abort everywhere.
<jtv> Because it's all supposed to be read-only transactions.
<jtv> But that might upset tests.
<wgrant> Yeah.
<jtv> OTOH aborts may possibly be slower, depending on implementation.
<wgrant> If it wasn't for tests I would have required you to also have the context manager assert that it was already a read-only txn.
<jtv> It's a risk that's hard to manage.  :/
<jtv> That's also why I have all these minimal read-write regions: can't afford to have anything in there yield.
<wgrant> It would be nice if we could add a hook to the reactor to confirm that the transaction is read-only before and after each callback.
<jtv> Yeah.  I asked around, but Twisted doesn't seem to have anything like that.
<jtv> Ultimately Robert is right: the twisted-based daemon and the ORM-backed logic shouldn't even be in the same process.
<jtv> Although adding more moving parts is no happier a prospect than twistification probably was.
<wgrant> Well.
<wgrant> A Twisted bridge between two XML-RPC sides is probably not that bad.
<wgrant> And the current architecture *could* work well, I suspect, except that the code architecture is still from the non-Twisted world.
<jtv> We don't have just 2 sides though: there's the slaves, the builder logic, the build manager, and Librarian.
<wgrant> Anyway, I'd really like to see you use 'with write_transaction:' or something like that, rather than the commit/DatabaseTransactionPolicy/commit
<wgrant> Like with dbuser.
<wgrant> Mm.
<wgrant> Sort of, I guess.
<jtv> Here's where I regret that it took so long.  I forget so many details.  What I remember is that write_transaction had both general problems and specific problems.
<wgrant> I don't mean the existing write_transaction.
<jtv> One of the general ones was that it doesn't enforce a fresh transaction.
<wgrant> Which is a function decorator that also retries.
<wgrant> Just something to encapsulate what you repeat.
<jtv> So you can get more of these weird âA and C have been done, but B aborted somehowâ effects.
<jtv> wgrant: Ah yes, another problem with read_transaction is that it allows you to write to the database.  I didn't want that in this case, because of the high probability that there would be accidental, untested database writes in the read-only sections.
<wgrant> jtv: Sure, the existing decorators are not useful here. I should have used a different name.
<jtv> Naming is hard.  Any specific suggestions?
<wgrant> with a_promise_that_i_am_not_evil_twisted_code
<jtv> with hand_on_heart:
<jtv> Do we provide download statistics for user PPAs anywhere?
<StevenK> In the DB
<wgrant> jtv: And the API
<jtv> Only there?
<wgrant> Yes.
<wgrant> I only added an initial API, then got busy and never designed a UI.
<jtv> wgrant: Oh well, I'll tell them it's your fault then.  Thanks.
<jtv> wgrant: meanwhile, any further thoughts on my MP?  Anything I need to change?  Do you think it's landable?
<wgrant> jtv: I don't like the duplication. It's too easy to forget a commit. Apart from that it looks pretty sane, but I need to go over it again.
<jtv> The duplication of what?
<wgrant> commit/DatabaseTransactionPolicy/commit
 * stub reads up on celery
<rick_h_> j
<lifeless> stub: how much of the last fdt was slony / fti/trusted / the actual patch ?
<lifeless> stub: e.g. how much identifiable fat do we have ?
<stub> lifeless: I don't think we have done a rollout with the new Slony yet, so I don't know the real, current answer to that.
<stub> lifeless: I don't think we can tell, because to minimize overhead we collapse everything update.py does into a single big .sql script rather than running EXECUTE SCRIPT several times. We can tell how long it takes to run security.py vs. update.py, but beyond that?
 * stub wonders if the timestamps in LaunchpadDatabaseRevision have the information, or if transaction_timestamp vs. statement_timestamp dance that has to be done means they lie.
<stub> oh... staging has been new slony for ages. duh.
<wgrant> stub: We did an fdt with the new slony on Monday
<wgrant> And probably Friday too
<stub> lifeless: http://paste.ubuntu.com/747987/ is the relevant section from staging.  51s to apply no patches (but this still resets security and applies trusted.sql)
<wgrant> Yeah, 18th and 21st there were fastdowntimes.
 * stub looks for the production logs
<wgrant> lifeless: :( postgres on carob is hungry.
<stub> http://paste.ubuntu.com/747996/ is relevant stuff from a production outage, 3:12 outage
<stub> The fat seems to be in waiting for sync and things to propagate. About 10 seconds everytime we pause and ask if everything is ok and caught up.
<stub> 13s to lock all the tables and run trusted.sql and the single db patch. The db patch thinks it took 2.3 seconds.
<stub> 47s for that update to propagate and be run on the slaves and success to be reported back to the master.
<stub> We run security.py in series, but that is only a few seconds as we bypass slony entirely for that.
<wgrant> rvba: Can you QA bug #867941, please?
<_mup_> Bug #867941: person:+activereviews times out <affects-ubuntu> <code-review> <qa-bad> <timeout> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/867941 >
<wgrant> Your rollback is on qastaging.
<rvba> wgrant: oh, great, I was just waiting for it to be deployed to qastaging.
<wgrant> rvba: buildbot finally passed :)
<rvba> Finally! :)
<wgrant> stub: The previous fastdowntime might be better to analyse.
<stub> The biggest fat I can identify is adding new stuff we have created to replication. If we create new tables, we need to call EXECUTE SCRIPT twice.
<wgrant> stub: What are our slony intervals nowadays?
<wgrant> We'll hopefully do a fastdowntime tomorrow.
<wgrant> I might do a double one.
<wgrant> The second being a no-op.
<wgrant> So we can see how that goes.
<wgrant> And we can fit both in 5 minutes, so it's fine.
<stub> default settings if you mean the slond config timings
<stub> so 2 seconds for sync-check-interval. We could lower that and see if it speeds up all the handshaking
<wgrant> Hmm.
<wgrant> What's the other one? 10s?
<wgrant> IIRC that's the deaful
<wgrant> default
<stub> wgrant: That is a heartbeat - shouldn't affect handshaking
<wgrant> You'd think not.
<wgrant> But in my testing locally it did, IIRC.
<wgrant> It's been a few months, though.
<stub> hmm
<stub> well, easy enough to tweak it and see what happens. We are not going to overload things changing these settings
<stub> But only one at a time of course.
<stub> wgrant: Any opinion on changing sync-check-interval or sync-interval-timeout first?
<mrevell> Hallo
<wgrant> stub: I am no longer sufficiently informed to have such an opinion.
<lifeless> wgrant: yeah, it is
<rvba> lifeless: Hi Rob, any way for me to access an oops reportâ¦ like manually or something? The web application seems to be still jammed ;).
<rvba> I suppose the old oops cleaning up stuff is taking more time than anticipated.
<stub> wgrant: I'll knock the sync-check-interval (how often a daemon polls to see if a sync should be sent) to half, 1s. Lets see if it affects that 10s time.
<stub> wgrant: huh. 'If the node is not an origin.... wasteful for this value to be much less than sync_interval_timeout'. So yeah, twiddle them both.
<stub> Ideally we would change this just before the rollout...
<lifeless> rvba: grep for the id in the relevant dir on carob
<lifeless> rvba: the relevant dir for staging/qasstaging is /srv/oops-amqp/<instance>/<day>
<lifeless> rvba: for production its /srv/launchpad.net/logs/production/host/<day>
<rvba> lifeless: Ok, thanksâ¦ for the tip.
<lifeless> rvba: it should be unjammed now
<rvba> lifeless: I can access the tools notâ¦ for my qastaging oops from 10 minutes ago is not there.
<rvba> s/not/now/
<rvba> s/for/but/
<lifeless> rvba: if its not there yet, its probably in the rabbit queue waiting for the consumer to catch up
<rvba> Rightâ¦
<wgrant> Do we have a graph of that, or just a nagios check?
<lifeless> for that there isn't much to do but wait ><
<lifeless> we want a graph, I think liam was doing/has done, but I can't see it on tuolumne
<rvba> I think gnuoy is working on a graph for that.
<lifeless> wwaiting for him to say 'done' before asking where it is ;)
<gnuoy> wgrant, we have graphs of oops queue lengths in staging and qastaging
<wgrant> Ooh
 * wgrant looks
<wgrant> rvba: qastaging's amqp2disk is doing lots of stuff.
<gnuoy> I've just landed a fix to stop there being missing data
<wgrant> Or at least spamming a lot of syscalls.
<rvba> +activereviews for ~ubuntu-branches issues 169 (as opposed to 1400+ previously) but I'd like to see the forced oops to make sure all the repeated statements are gone.
<rvba> 169 queries that is.
<wgrant> Yeah, it's writing out OOPSes.
<wgrant> So it should be there eventually.
<wgrant> gnuoy: Ah, nice.
<rvba> "QAStaging Rabbit Oops Queue" â¦ cool
<wgrant> 300 unacked messages is a bit odd.
<wgrant> I guess we'll see what happens once the queue reduces.
<gnuoy> There are also graphs for rabbits rss and the mnesia db size
<lifeless> wgrant: pipelining perhaps
<rvba> Very nice, thanks gnuoy!
<gnuoy> rvba, no problem
<wgrant> lifeless: Except that it's been that way for 11 hours
<wgrant> constant
<wgrant> May just be amqp2disk hanging during your terrorisation of postgres, though.
<wgrant> It's certainly not being fast at the moment, but it is working.
<wgrant> I wonder how it will cope with fastdowntime.
<wgrant> rvba: What's the OOPS ID you're looking for?
<rvba> wgrant: OOPS-9daabd974426cdd97b512b23d6906f55
<rvba> It's not in /srv/oops-amqp/qastaging/2011-11-24 yet
<wgrant> That may be because it is under a different filename.
<wgrant> However, not in the DB yet either.
<wgrant> I guess it's near the end of the queue.
<rvba> The number of queries plus the fact that it's not (always) timing out tells me it's qa ok but I'd like to be sure there is no repeated stmt leftâ¦
<wgrant> Yep
<stub> What are we using rabbit for atm? Just OOPS or is txlongpoll in there as well?
<wgrant> rvba: https://lp-oops.canonical.com/?oopsid=OOPS-9daabd974426cdd97b512b23d6906f55
<wgrant> And indeed, amqp2disk has gone silent.
<wgrant> So it was near the end of the queue.
<wgrant> And we are now caught up.
<wgrant> 123 repetitions
<wgrant> However, only one significantly repeated query.
<wgrant> So much better.
<rvba> Rarg, fetching branches infoâ¦
<lifeless> wgrant: that would fit
<rvba> I'm already fetching data for target/sources/prerequisite branchesâ¦
<lifeless> wgrant: x number pipelined, and the consumer wedged
<wgrant> lifeless: Possibly. We'll see soon.
<wgrant> When the graph updates.
<lifeless> wgrant: no, when the queue clears
<wgrant> rvba: Fortunately we have tracebacks :)
<lifeless> wgrant: if its pipelined it will be constant(ish)
<wgrant> lifeless: The queue s clear.
<rvba> wgrant: indeed! tracebacks ftw!
<lifeless> ah cool
<wgrant>     if can_access and self.stacked_on is not None:
<wgrant> You're not preloading stacked_on, I guess :)
<lifeless> thats spelt wrong
<lifeless> self.stacked_onID is not None
<wgrant> But damn, 169 queries with 123 able to be trivially eliminated... not  bad.
<lifeless> unless you're actually accessing the stacked_on branch
<wgrant> lifeless: Unless it accesses the attribute in the next line.
<wgrant> Which is probable.
<lifeless> yeah, so needs eager loading
<lifeless> rvba: nice work
<wgrant>             if self.stacked_on not in checked_branches:
<wgrant>                 can_access = self.stacked_on.visibleByUser(
<wgrant>                     user, checked_branches)
<rvba> thanks lifeless.
<wgrant> is the next bit
<rvba> Yep, you're right wgrant, stacked_on branches.
<wgrant> rvba: I guess that will only really be a significant problem for ~ubuntu-branches, because everyone else will work with only a few trunks.
<rvba> wgrant: right, but fixing up ~ubuntu-branches is my goal!
<wgrant> Yep.
<wgrant> rvba: So, qa-ok, then?
<wgrant> We should deploy.
<rvba> wgrant: yep
<wgrant> rvba: Can you tag the bug and request a deployment once the QA report is updated?
<rvba> wgrant: sure
<rvba> bug tagged
<rvba> I've got another branch coming up to reuse the same eager loading method to fix product:+activereviews.  Right now it can be as bad as issuing 1800+ queries.
<wgrant> Yeah.
<wgrant> Fortunately they're very quick queries.
<wgrant> But still really bad.
<rvba> Right, but still.
 * jml waits for diffs
<jtv> allenap: done with your review.
<allenap> jtv: Thanks!
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 292
<wgrant> Aha, green deployment report.
<rvba> wgrant: ok
<jml> 9 minutes and counting
<wgrant> jml: Is that including scan time?
<jml> wgrant: Don't think so, LP has known there are pending changes for that whole time.
<wgrant> jml: Nowadays it shows that also when a scan is pending.
<jml> wgrant: interesting.
<jml> so, yeah, probably including scan time.
<wgrant> Does the branch page also have the pending change warning?
<jml> wgrant: it's finished now
<jml> wgrant: so no :)
<wgrant> Heh
<wgrant> Excellent timing.
<wgrant> rvba: Thanks.
<rvba> wgrant: Glad to help out with that, I know you've been doing that a lot lately.
<rvba> wgrant: funny, the check performed on the stacked_on branch is recursive! The only was to load all the branches in one query is to use a recursive CTE.
<wgrant> rvba: Ah, how lovely.
<wgrant> Nearly not worth it.
<rvba> Yeah, that's what I was thinking, I'll just preload on level of stacked_on branches.
<rvba> s/on/one/
<wgrant> Good plan.
<lifeless> rvba: don't need one query
<lifeless> rvba: need 2 - one for the collection, and then the stacked set for the batch (in the pre iter hook) - and I would definitely use a CTE at this point
<lifeless> wgrant: did you say didrocks was an edge user?
<rvba> lifeless: ok, you think it's worth it? Also all these branches will have to be checked by branch._checkBranchVisibleByUser
<wgrant> lifeless: I fixed that yesterday.
<lifeless> wgrant: thanks
<wgrant> lifeless: That's why the edge graphs are back to their old levels.
<wgrant> bdmurray also migrated his stuff away
<wgrant> Will attack more tomorrow.
<lifeless> rvba: yes, they need to have visiblity checked, but the eager loading query should do that
<lifeless> rvba: as in, your eager loading query should filter non-visible stacked-on branches
<lifeless> rvba: I think its worth it yes, we have some silly deep stacking chains
<rvba> lifeless: all right, let's do this all the way then :)
<rvba> lifeless: not sure I understand "your eager loading query should filter non-visible stacked-on branches", my plan is to preload all the stacked_on branches, then let the python sort the visibility without having to hit the database.
<lifeless> rvba: as a matter of principle, you shouldn't pull stuff from the DB the user cannot see.
<lifeless> rvba: as a matter of pragmatism, filtering in the query is faster - becaues the python visibilty rules trigger late evaluation
<rvba> lifeless: Okay, so this mean I'll have code the visibility rules in sql right?
<rvba> means even
<rvba> s/code/to code/
<lifeless> rvba: they are already coded, in BranchCollection
<wgrant> It probably already is in the main query.
<lifeless> rvba: you should be able to use a (new) BranchCollection to make the query for stacked on, with the CTE as an additional filter
<wgrant> Yeah.
<wgrant> The main query for that page already does the privacy check.
<rvba> lifeless: Okay.
<wgrant> It might just not prepopulate the security adapter cache.
<wgrant> Bah, although it's not recursive.
<lifeless> right, and if it doesn't, it a) should and b) thats even more reason not to retrieve things that the user cannot see :)
<rvba> allenap: can you please have a look at https://code.launchpad.net/~rvb/launchpad/activereviews-bug-893702/+merge/83172 ?
<allenap> rvba: Certainly.
<rvba> thx
<rvba> lifeless: humâ¦ to check the visibility, BranchCollection materializes the list of visible branches (to use branch.id in (...)) â¦ and that's precisely what's making the main query really slow.  I think I'll have to be very careful and maybe I won't be able to use BranchCollection here because the remedy may be worse than the disease If I do â¦
<lifeless> rvba: or you need to teach branchcollection a couple of ways to check...
<lifeless> rvba: e.g. BranchCollection()...EagerLoadById() -> checks per branch rather than all-visible etc
<rvba> lifeless: yeah, I need to think about it. I'm sure there is proper way, but I'll need to be careful when doing the qa to make sure I'm not issuing one more giant query :)
<rvba> lifeless: If you don't mind, I'll ping you again when I have something ready so you can have a look.
<lifeless> sure
<rvba> Thanks.
<jml> hey
<andy_js> How do I get launchpad to use openid.net instead of testopenid.dev?
<jml> did you get rid of ReadyService?
<wgrant> jml: launchpad-buildd no longer uses it, but other things do.
<jml> wgrant: what does it use instead?
<wgrant> andy_js: grep for testopenid.dev in the configs and change it to openid.net, I guess.
<wgrant> jml: It tries to connect to the port.
<jml> and keeps retrying until syn/ack?
<wgrant> I believe so.
<jml> thanks.
<andy_js> wgrant: It's not that simple I'm afraid.  Changing 'enable_test_openid_provider' in schema-lazr.conf to False seems like the right solution, but after I do that the login page says 'OpenID Provider Is Unavailable at This Time'
<wgrant> andy_js: That just disables the serving of testopenid.
<wgrant> It doesn't affect its use.
<andy_js> But once you disable 'enable_test_openid_provider' launchpad no longer tries to redirect to 'testopenid.dev' (which is not available on my non-local instance)
<wgrant> No.
<wgrant> Because it can't perform discovery on it.
<wgrant> It'll complain that it's unavailable instead.
<wgrant> I don't know if openid.net supports the discovery functionality that we require.
<andy_js> So does launchpad support OpenID or not?
<wgrant> Launchpad supports OpenID, but currently only in SSO mode, to a predefined provider.
<wgrant> I only know of one instance other than launchpad.net, and it uses CommunityID as its OpenID provider.
<wgrant> launchpad.net uses Ubuntu SSO, which is powered by canonical-identity-provider.
<wgrant> However, any provider that supports Yadis or XRDS discovery should work.
<andy_js> All I'm trying to do is set up an instance of launchpad which is remotely accessible.
<andy_js> The only blocker for me is getting login to work.
<wgrant> You are aware of the branding licensing issues?
<andy_js> I thought it was free for non-commercial use.  Is there more to it than that?
<wgrant> https://dev.launchpad.net/LaunchpadLicense
<wgrant> The branding must be changed for a production instance.
<wgrant> Commercially or non-commercially.
<andy_js> Ah that's no problem.  I was going to rebrand it for my project anyway.
<wgrant> The code itself is AGPLv3, so you can use it for whatever you wish. The images and name need to be changed, but that's not an insurmountable obstacle.
<wgrant> Why are you not using launchpad.net, out of interest?
<andy_js> 1) I want complete control
<andy_js> 2) I don't think my architecture is supported (solaris-i386, solaris-amd64)
<wgrant> Ah, for PPAs, I see.
<wgrant> That's the same reason the only other instance I know of exists -- to build PPAs for Debian.
<andy_js> Yep.  The possibility to have ppa with tight integration with bug tracking and scm is really enticing
<wgrant> Getting a productionish build farm up and running is a bit of effort, but doable.
<andy_js> Is it possible to run ppa's without setting up a build-farm?
<wgrant> Not if you want binaries. There has to be somewhere to build them :)
<andy_js> So I can't just upload using dput?
<andy_js> Because I don't think our architecture is quite mature enough for doing the build-farm route.
<wgrant> There is an old option to the upload processor that should let you upload source and binaries, but it hasn't been used for a few years.
<wgrant> Why do you say that?
<andy_js> I haven't been able to create a working ISO in a few monthes so I have nothing current to install on the build servers (that, and I don't actually have any build-server :P)
<andy_js> Anway, if I can upload stuff using dput then it should be able to do what I want.
<andy_js> If it's an old crusty open that doesn't work quite right I guess I'll have to fix it
<wgrant> LP was designed from the start to have sources uploaded to it, and build binaries itself. If you just want to publish an archive with pre-built binaries, why not use something a little simpler and less terrifying, like reprepro?
<andy_js> reprepro is what I'm using now and it does the job fairly well
<andy_js> I want the other things that launchpad offers (bug tracking, mailing lists, scm management, the ability to let people create their projects)
<andy_js> And in the future I do intend to make use of the build-farm feature
<wgrant> I suspect that you underestimate the resources and effort required to maintain a complete Launchpad instance.
<andy_js> It seemed like it would be easier than writing my own software which is what I started to do.
<andy_js> Is it really that of a pain in the arse to maintain?
<wgrant> Probably, but for everything except non-Ubuntu PPAs you can just use launchpad.net...
<andy_js> Well I'm actually getting my sources from Debian so I guess that wouldn't be acceptable.
<wgrant> Hmm?
<andy_js> Hmm what? :)
<wgrant> I'm not sure of the relevance of you getting your sources from Debian.
<andy_js> I thought you just said PPA's were only for Ubuntu?
<andy_js> (on launchpad.net, anyway)
<wgrant> launchpad.net will only build for releases and architectures that Ubuntu supports.
<wgrant> My suggestion would be that you track your bugs and projects and everything except packages on launchpad.net.
<wgrant> Since you're not going to be using build farm functionality yet anyway, PPAs probably provide little to no benefit to you.
<wgrant> If you do really want to use them, you could use a local instance for PPAs, I suppose. But maintaining your own codehosting and mailinglist setups is no particularly small effort, and I don't see much benefit over using launchpad.net.
<andy_js> I want to be able to give people the ability to create their own PPA and upload package via dput.
<wgrant> Ah, including uploading binaries?
<andy_js> Yes
<wgrant> That would require some hackery of the upload policies of a local instance. The secure upload paths do not permit binaries. But that is easily changed.
<andy_js> Hackery is fine as long as what I want to do can be acheived without having to rewrite major chunks.
<andy_js> wgrant: Thank you very much for your help so far btw :)
<wgrant> It was done regularly until a couple of years ago. Security updates were built outside LP, and source+binaries uploaded at once.
<wgrant> We still have tests for that, so it shouldn't have bitrotten.
<wgrant> Anyway, I should sleep.
<abentley> allenap: could you please review https://code.launchpad.net/~abentley/launchpad/history-model/+merge/83310?
<allenap> abentley: I'm not going to be able to finish that before I have to go out. I will be working this evening, from 2000 UTC to 2200 UTC. Would it be okay if I did it then?
<abentley> allenap: sure, no big rush.
<allenap> abentley: Cool.
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<flacoste> abentley: shouldn't the Order by buttons reflects the list of fields displayed?
<abentley> flacoste: they should, but we haven't done that yet.
<flacoste> ah ok
<flacoste> gmb: you actually reported a known bug that deryck is fixing, it should have been mentioned as a known issue in the blog post
<flacoste> maybe we should do a duplicate guess contest
<flacoste> i bet we ar going to get 5 duplicates of that bug
<gmb> flacoste: Yeah, I noticed it got duped. My bug title was obviously far enough away from the original bug for the match to fail.
<lifeless> righto, vanadlism undone
<lifeless> all hail deleting of bugtasks too
<allenap> abentley: Do you have a few minutes to talk about PackagingJob?
<lifeless> allenap: just the lad
<andy_js> How do I get login to work in a production environment?
<lifeless> allenap: I want to talk to you aboot script activity ;)
<abentley> allenap: sure.
<lifeless> allenap: I'm happy to queue behind you and abentley
<allenap> lifeless: Yeah, I know :) I really wanted to reply on the wiki but I have hardly had a moment :-/
<allenap> Cool, abentley first, he's closer to EOD.
<allenap> abentley: In trying to open translations for Precise we first have to erase all existing data that has been created for it.
<allenap> abentley: We have a script to do that: http://paste.ubuntu.com/748706/
<allenap> However, there are also references to some of that data from PackagingJob records. Nearly 80k of them iirc.
<allenap> abentley: Jeroen reckoned that we could probably delete them, but that we'd need to recreate them.
<allenap> abentley: Your name and Danilo's is all over PackagingJob, hence why I'm asking you.
<abentley> allenap: My work on packaging has been SourcePackageRecipeBuilds.  Those did use a PackagingJob, IIRC.
<abentley> Oh, and I guess the translation stuff counts, too.
<abentley> allenap: So, I don't think you need to recreate those records.
<allenap> \o/
<allenap> We can just delete and be done?
<abentley> allenap: Records that are not active are strictly for historical record.
<abentley> allenap: If you're deleting the Precise translations, then you wouldn't need info about the jobs related to those Precise translations.
<abentley> allenap: So yes, I think you can just delete and be done.
<allenap> abentley: Great. So, as long as the job records are in terminal states we can trash them. If they're not in terminal states we ought to get them to terminal states before trashing them.
<abentley> allenap: I don't think you even need to get them to a terminal state, since you'll be deleting whatever they could act on.
<abentley> allenap: but of course, you want to avoid doing this for non-Precise jobs.
<allenap> abentley: Okay, tip top. Thank you, I think that's all I need to know.
<abentley> allenap: cool.
<allenap> abentley: Yes, accidentally overstepping has been my main fear when crafting this script.
<lifeless> allenap: voice?
<allenap> lifeless: Okay, let me re-read your last comment first though :)
<lifeless> sure
<allenap> lifeless: Okay, ready.
<huwshimi> lifeless: Just before I do it, I want to bump bug 894449 to critical. It creates a completely broken UI and would oops if there were such things for the UI. Is there any reason why it shouldn't be critical (in case I missed something)?
<_mup_> Bug #894449: It's possible to create an entirely useless bug listing <bug-columns> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894449 >
<wgrant> That's only High.
<lifeless> huwshimi: It doesn't prevent users doing what they want, and it only affects users explictly opted into the beta program.
<wgrant> Users can make the UI in a beta feature show nothing if they explicitly tell it to.
<lifeless> huwshimi: So I don't think it meets our guidelines for critical.
<lifeless> huwshimi: *however* there is a bug that the new UI breaks some searches, so I think we have to turn it off anyhow.
<lifeless> huwshimi: because *that* bug is critical.
<huwshimi> lifeless: OK, no problems
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugtasks: 292
#launchpad-dev 2011-11-25
<huwshimi> poolie: Bug tag list is live :D
<huwshimi> poolie: I know you've only been waiting 10 months for me to do it :)
<rick_h_> lol, congrat huwshimi
<rick_h_> where can I see huwshimi ?
<huwshimi> rick_h_: heh, thanks :)
<huwshimi> rick_h_: https://bugs.launchpad.net/launchpad/
<rick_h_> is it the tag list on the right?
<rick_h_> with the counts?
<huwshimi> rick_h_: Yeah
<poolie> huwshimi, that's awesome
<poolie> it doesn't look like a cat threw up :)
<huwshimi> rick_h_: Not sure if you noticed, but they used to look like the left version of this: https://launchpadlibrarian.net/62993362/tag_cloud.png
<rick_h_> ah, ouch
<wgrant> Hmm.
<wgrant> Interesting. The sizes were meant to scale more than that.
<wgrant> But it shows them all as identical...
<poolie> wgrant, in the 'current'?
<poolie> they did scale
<wgrant> That's what I thought.
<rick_h_> huwshimi: now what about client side filtering box? :)
<poolie> it did not work well because ...
<poolie> well for a bunch of reasons
<huwshimi> wgrant: They did scale, but the top tags were all a similar size
<rick_h_> yea, those types of tag clouds never lookd great, especially with large phrases for tags
<poolie> one of them being the words are so wide relative to the portal
<poolie> mm
<poolie> keeping the same size and varying intensity would be better
<poolie> would have been
<wgrant> Yeah.
<poolie> just showing the number is pretty good though
<poolie> so huw while you're here
<wgrant> https://code.launchpad.net/ has another odd (and woefully inconsistent with the rest of the site) cloud.
<huwshimi> rick_h_: Doing the tag filter would be nice, but this was an easier win as a first stage
<huwshimi> wgrant: Yeah, from memory all those landing pages have something similar
<rick_h_> huwshimi: oh yea, just helping you with stage 2 :)
<poolie> wgrant, i wonder if we should just redirect that to the top home page
<wgrant> Amusingly enough it also doesn't include distributions, so Ubuntu doesn't show up.
<wgrant> huwshimi: No, that's the worst bit. Only code does.
<poolie> maybe look at the logs to see if it is used
<huwshimi> wgrant: Oh really?
<poolie> i think the idea was that it would show activity
<poolie> well, that was the idea
<poolie> it was my idea
<poolie> it is not actually doing the job very well
<poolie> :/
<rick_h_> poolie: needs a color scale as well as size maybe?
<poolie> there are two mismatched shades of blue on that page too
<wgrant> Some of them used to be green.
<wgrant> IIRC those that didn't use codehosting officially.
<huwshimi> wgrant: it seems you're right
<wgrant> Can't quite recall, though.
<wgrant> https://code.launchpad.net/ also has the 1.0 buttons.
<wgrant> Ew
<wgrant> Tempting to remove the subdomains tomorrow, but that might be a bit controversial.
<rick_h_> heh, #03a and blue
<huwshimi> wgrant: yeah a few of the landing pages have those buttons
<lifeless> wgrant: on a saturday? yes, 'might' be
<rick_h_> hey, it's friday
<huwshimi> wgrant: Actually all but bugs do
<rick_h_> always good to change right before the weekend and then hide
<StevenK> rick_h_: O hai. You haz QA.
<rick_h_> StevenK: yep, am I holding up anything? Been afk for the holiday today.
<huwshimi> wgrant: I'm definitely buy a beer for anyone who removes subdomains
<StevenK> rick_h_: There's only three revisions, in the queue.
<StevenK> s/,//
<wgrant> The only sticking point is the breadcrumbs.
<StevenK> rick_h_: So I don't care much.
<rick_h_> StevenK: ok if I get it in the morning then?
<wgrant> Everything else is mostly a matter of deleting code.
<rick_h_> StevenK: ok awesome, will hit it up first thing
<StevenK> rick_h_: If you're at work on your Friday, that sounds great.
<rick_h_> StevenK: yep, will be. New guys don't have all that banked vaca :)
<StevenK> Haha
<poolie> wgrant, getting everything on one domain so that it can run from a single ip without /etc/hosts futzing would be awesome
<rick_h_> wgrant: you know, I've used LP on and off for years and only *just* noticed the breadcrumbs the other day
<wgrant> The breadcrumbs have been through a few iterations.
<rick_h_> caught me totally out of the blue
<wgrant> This last variety has been in place for nearly 2.5 years, and is extremely subtle.
<wgrant> And confusing to an extent.
<wgrant> poolie: Indeed.
<StevenK> That requires ripping out vostok
<StevenK> Which might make mwhudson sad.
<wgrant> There's a branch for that.â¢
<rick_h_> <3 I think I have a new email sig
 * huwshimi contemplates pie incentive to have subdomains removed by Budapest, or the end of Budapest
<lifeless> StevenK: vostok can be deleted
<poolie> what's a vostok?
<wgrant> vostok is Linaro's aborted attempt at making a standlone Soyuz.
<lifeless> poolie: we can't run on one domain safely, ever. But we can shrink the number :)
 * rick_h_ is grepping around too, I know I've run into that
<wgrant> lifeless: Well.
<lifeless> StevenK: I was discussing nuking vostok with mwhudson the other day.
<poolie> lifeless, you're talking about librarians, ppas, etc?
<wgrant> lifeless: The webapp can be on a single domain.
<lifeless> wgrant: launchpadlibrarian
<wgrant> And that's the main thing.
<poolie> ok
<wgrant> launchpadlibrarian is not user-facing.
<wgrant> In the important sense.
<lifeless> wgrant: internalxmlrp
<poolie> perhaps for demo purposes it would be ok to have it on the same domain
<lifeless> wgrant: and the public xmlrpc too
<lifeless> wgrant: and apis
<poolie> if there is no untrusted or important data
<poolie> huwshimi, while i've got you
<huwshimi> poolie: Yup, here
<wgrant> In dev I imagine they would just run on different ports.
<poolie> some time i'm going to land the initial markdown thing
<poolie> for people and products
<poolie> i guess i feel i want a thumbs up/down from you
<lifeless> wgrant: perhaps; I'd be keen for dev to just use ip addresses for all the services.
<wgrant> lifeless: Sure, but that requires lots of config unless you run everything on loopback.
<wgrant> Which is limiting.
<huwshimi> poolie: A massive thumbs up for the spirit of it, is there anything in particular you want validation on?
<lifeless> wgrant: why ?
<wgrant> lifeless: My machine has one non-loopback IPv4 address.
<wgrant> My VMs, too.
<poolie> just that the list of 'do later' things on the bug and mp are ok
<lifeless> wgrant: ...and ?
<wgrant> lifeless: We can't run services on separate IP addresses if there's only one IP address.
<rick_h_> StevenK: where do you go to see the pending QA and such?
<poolie> huwshimi, the other thing is
<StevenK> rick_h_: The deployment report.
<lifeless> wgrant: I didn't intend to imply separate ip addresses
<StevenK> rick_h_: http://lpqateam.canonical.com/qa-reports/deployment-stable.html
<StevenK> rick_h_: Know it, love it.
<poolie> i was contemplating, when my current lp discretionary hacks are finished, doing a really cheap timeline that just shows all the comments on bugs i'm subscribed to
<wgrant> lifeless: Ah, so separate ports, like I said. Sounds good, and is what I had planned.
<poolie> in reverse chronological order
<lifeless> wgrant: and no domains
<poolie> this seems
<wgrant> Sure.
<lifeless> wgrant: so an ip address on ec2 just works.
<poolie> possibly useful
<rick_h_> StevenK: thanks, bookmarked
<wgrant> I have 5 branches here relating to cleaning up the vhost machinery and config.
<poolie> a toe in the water; and doesn't require db or infrastructure changes
<lifeless> poolie: timelines.... thanks for reminding me. I want notification to be done as a service.
<StevenK> rick_h_: There is also db-stable.html for database stuff
<wgrant> I should probably merge and land them at some point.
<wgrant> IIRC one of them destroys vostok.
<poolie> lifeless, that sounds good but what exactly do you mean
<lifeless> poolie: I'm not 100% sure what I mean yet :).
<wgrant> blah
<poolie> lifeless, ok :)
<poolie> so like i say this seemed like a place to start
<mwhudson> lifeless, StevenK: sure, rip out vostok if it's causing any problems
<huwshimi> poolie: What happens when you dump all different types of content (headings etc) into a description, does it come out looking alright?
<abentley> wallyworld_: what particular spinner did you have in mind in your review?
<wallyworld_> abentley: there isn't any additional spinner on the page right now, but there could be one added later and it could cause problems. so the suggestion is just to make the code more robust
<wallyworld_> but if you don't think it's required, i can just +1 it
<wallyworld_> it seemed worth making the change thogu, given it's just a slightly longer select expression
<abentley> wallyworld_: There won't be a later.  Deryck is working on a different spinner that will replace this one.
<wallyworld_> ah, ok. in that case i'll just +1 it
<poolie> huwshimi, in markdown?
<poolie> it looks ok
<abentley> wallyworld_: thanks.
<wallyworld_> abentley: done, sorry for the confusion on my part
<huwshimi> poolie: Yeah, ok great
<poolie> obviously if people start putting lots of h1s it will look a bit weird but it doesn't spoil things
<poolie> ... i don't think we're going to allow the animated gif background tag :)
<huwshimi> poolie: Great, in that case I'm happy to go with what you have
<huwshimi> poolie: Awww
<poolie> rupert murdoch can buy lp for $580M
<huwshimi> poolie: With the cloud on the code page I wonder if we can turn that into an activity wall of sortts
<huwshimi> *sorts
<wgrant> That page is probably the walliest thing in LP right now.
<wgrant> It has the registered/changed branch feeds at the bottom.
<wgrant> And the cloud showing activity levels.
<poolie> huwshimi, i don't know if you have time but if you want to do a sketch of a general activity wall , version 0
<poolie> i might try to implement it
<huwshimi> poolie: I started working on this a long time ago: http://people.canonical.com/~huwshimi/profile_01.png
<huwshimi> poolie: But I'm happy to work on this a bit with you
<huwshimi> poolie: It would give us a chance to try a few thigns
<huwshimi> *things
<huwshimi> poolie: I'm just running your markdown branch, I've enabled the feature flag ("markdown.enabled default 1 true"), but I'm not getting the markdown rendered. What else might I need to do?
<huwshimi> poolie: Oh, all I did was grab a new lp branch, and merged your changes
<lifeless> poolie: so I think what I mean is a few things that all intersect
<lifeless> poolie: notification / walls / timelines all belong outside the main lp codebase: it requires a rather different view of the data than our main tables are structured for
<lifeless> poolie: it also has no requirement to be carefully modelled - in fact it has a requirement to be agile and loosely modelled, to easily be updated as other things change
<poolie> huwshimi, did you work it out with md?
<huwshimi> poolie: erm, no I got distracted by other things :)
<poolie> huwshimi, i don't know, that should be all you need to do
<poolie> nb it only works on eg person descriptions, not on all text yet
<poolie> lifeless, i agree with those as an ultimate goal
<poolie> mm, not quite sure about why it ought to be outside the main codebase though
<poolie> except in as much as perhaps it would be nice to have a less monolithic codebase generally
<huwshimi> poolie: Oh, I misunderstood (I thought it worked on bug descriptions too). Yes it's working perfectly
<poolie> just being (perhaps over) cautious
<poolie> also deferring making it work with edit widgets
<poolie> http://people.canonical.com/~huwshimi/profile_01.png looks good
<poolie> i will try for something like that
<lifeless> poolie: multiple services will need to talk to notifications; its easier to work on small codebases; nosql may well be much easier to work with; all new things should be built with SOA in mind
<poolie> k
<lifeless> notification/walls/timelines are greenfield, making them 5-10 times harder by working within the big curly DB we have would be a costly choice IMNSHO
<poolie> !
<poolie> interesting
<lifeless> [email notifications clearly aren't greenfield, the rest are]
<poolie> if they were done externally, how would the web content be visible to the user
<lifeless> in LP you would have a thin shim which makes API calls
<lifeless> same as e.g. talking to memcached, but without the potatoes.
<poolie> wall_service.get_wall_html('mbp')?
<lifeless> for instance.
<lifeless> UI and rendering is in LP
<poolie> ah, so more like
<lifeless> storage, graph traversal, summary snippets and a pointer back to the source document are in the service
<poolie> the view object makes a call to get structured machinereadable data from the wall service?
<lifeless> yup
<wgrant> Privacy is a bit of a challenge, but not insurmountable.
<lifeless> all modern sensible stuff and isolated from insane http headers
<lifeless> so json blob
<poolie> right, it will need to get it on behalf of the current interaction user
<lifeless> some implementation choices around whether the summary snippets are served as is or privacy-checked on the LP side etc
<poolie> mm
<poolie> i feel like i would like to do this in a way that for version 0 talks to the lp database
<poolie> but with a view to eventually pointing to something else
<lifeless> you're welcome to scrach that locally.
<lifeless> but
<poolie> quite isolated
<lifeless> I'm going to reject landing anything in this space in lp proper
<poolie> :( really?
<lifeless> I realise this provides a barrier to entry, but I think its better overall
<lifeless> yes, really.
<lifeless> the cost of setting up a microservice is fairly modest
<wgrant> lol
<poolie> ok
<wgrant> We are yet to have a single microservice running.
<poolie> maybe i'll test that
<wgrant> (but I agree it needs to be done as one)
<lifeless> wgrant: not true; we have 3 running so far - txlongpoll, oops-tools and arguably rabbit
<wgrant> txlongpoll and rabbit are not running.
<lifeless> wgrant: txlongpoll only needs the rabbit config switched on in LP, and thats waiting for red to put the MP forward
<wgrant> They're somewhat close.
<wgrant> But they're not running.
<poolie> so every lp action that does a wall-related activity will post it to the microservice?
<lifeless> rabbit is
<poolie> and they'll be stored redundantly there?
<wgrant> lifeless: It's not being used for anything.
<lifeless> graphs and nagios are in place for prod rabbit
<lifeless> poolie: yes
<lifeless> poolie: for whatever depth of history we want to support
<poolie> hm
<poolie> obviously there are consistency things about
<poolie> comments being deleted or hidden
<poolie> for instance
<lifeless> this is covered under the question 'some implementation choices around whether the summary snippets are served as is or privacy-checked on the LP side etc'
<lifeless> but yes
<lifeless> there are
<poolie> hm
<poolie> i think a soa would be better
<poolie> it seems a shame to block changes that give better presentation of data that is already in the db
<poolie> perhaps it is necessary if things are ever going to improve
<lifeless> so, you mentioned, for instance, listing all your own comments on bugs, in date order; this requires joining all messages from $user, across to all bugs, then checking for privact
<lifeless> a few users have > 100K messages
<lifeless> I can pretty much guarantee you'll have timeout issues in any first-draft implementation on e.g. ~janitor
<poolie> sure
<poolie> but https://bugs.launchpad.net/~pitti/ eg times out today
<lifeless> this isn't a reason not to do it, but its a demonstration of how our schema is not arranged to meet the needs of that project
<poolie> most bug list pages are timing out
<lifeless> garh
<lifeless> we fixed ~pitti
<lifeless> thats a regression
<poolie> ~mbp times out
<poolie> so
<poolie> i don't want to add more timing out pages
<lifeless> great! :)
<poolie> but i guess,
<poolie> lp does have to have a general answer to "do simple select/sort queries across the bug db"
<poolie> which is all this would need
<poolie> i'm not really confident that bringing up a separate db that will scale to that size will be easy
<wgrant> poolie: Did you really prefer a 6 digit bug to a 4 digit one?
<poolie> (even if it's allowed to throw stuff away)
<poolie> yes, becaues it has a better description
<poolie> but i don't care if you reverse it and update stuff
<poolie> lifeless, anyhow it's kind of blue sky now
<poolie> so i don't want to distract us with it
<poolie> i support soaization
<poolie> i don't want to add more timeouts
<poolie> i do want to take a small step towards timelines
<poolie> when i get to it i'll see if i can resolve those things
<wgrant> Launchpad is fixed!
<wgrant>  * 14 Exceptions
<wgrant>  * 2 Time Outs
<wgrant> Our work is done.
<poolie> ?
<wgrant> lifeless broke oops-tools, so the OOPS reports aren't as depressing as usual today.
<lifeless> well there is certainly *something* odd ;(
<poolie> hi, could i get a quick scan of https://code.launchpad.net/~mbp/loggerhead/240580-tarball/+merge/83364 from someone?
<poolie> thanks lifeless
<lifeless> jtv: you dropped off
<lifeless> poolie: got some time for bouncing ideas around ?
<poolie> lifeless, sure
<poolie> can i have some lunch first?
<lifeless> sure
<poolie> rabbit just failed again; i'm glad i fixed the feature :)
<poolie> psa
<StevenK> Grumble, storm validators suck
<StevenK> wgrant: Can haz help?
<lifeless> whats up
<StevenK> IProduct.{_owner,security_contact} both have a storm_validator of validate_person_or_closed_team.
<StevenK> Setting the owner to an open team results in an error of "Constraint not satisfied", but setting the security contact to an open team gives "You must choose a valid person or team to be the security contact for <project>"
<StevenK> And both validators are the same, so I'm not sure what is going on.
<lifeless> is there a try:except: around the other case ?
<poolie> lifeless, hi, now?
<mwhudson> ah i remember trying to figure out where constraint not satisfied came from
<lifeless> sure
<poolie> pots?
<StevenK> lifeless: They're both in the same view, neither are checked in the view's validate()
<mwhudson> StevenK: storm_validator is in the model code, i bet _owner has something funky in the zope.schema/interface side
<StevenK> mwhudson: There's a owner setter/getter, do you think that is interferring?
<mwhudson> StevenK: no, i think the error is probably coming 'before' that
<mwhudson> although owner is just an Attribute?
<StevenK> It's a PersonChoice
<mwhudson> ah yeah
<mwhudson>             vocabulary='ValidPillarOwner',
<StevenK> mwhudson: The error isn't coming from the vocab, though
<mwhudson> and i see security_contact is the same anyway
<mwhudson> StevenK: only difference i see is PublicPersonChoice vs PersonChoice ?
<StevenK> Oh, which is done via IHasSecurityContact
<mwhudson> StevenK: constraint not satisfied is raised in the form validation code in some frustrating way that blocks you from finding out why
<mwhudson> yeah
<mwhudson>         if value not in vocabulary:
<mwhudson>             raise ConstraintNotSatisfied(value)
<StevenK> Ah ha
<StevenK> Is that in zope guts?
<mwhudson> zope.schema-3.5.4-py2.6.egg/zope/schema/_field.py
<StevenK> RARGH
<StevenK> STEVE SMASH
<StevenK> mwhudson: I don't like the idea of rummaging around in self.errors in the view's validate() just for a better error.
<mwhudson> StevenK: i can't remember what i did last time i got angry about this
<mwhudson> StevenK: i'll notice that security_contact gives a nice error, maybe you can see why? :)
<mwhudson> oh
<mwhudson> well, slightly nicer
<mwhudson> one that's from the launchpad tree at least :)
<StevenK> mwhudson: That's what I'm trying to figure out! :-)
<mwhudson> StevenK: heh
<mwhudson> StevenK: i hope this pointer helped a bit at least
<StevenK> The ConstraintNotSatisfied exception is in self.errors for owner when it's an open team
<StevenK> Right, there is a bunch of machinery around security contact
<StevenK> Whereas there is none around owner
<StevenK> mwhudson: I've added a block before the OMG-Zope-gave-us-errors-must-return section and that works.
<mwhudson> StevenK: i'm not sure what that means, but i'm glad it works :)
<StevenK> mwhudson: http://pastebin.ubuntu.com/748928/
<mwhudson> StevenK: ah
 * StevenK shakes fist at NameAlreadyTaken
<huwshimi> wallyworld_: Hey mate, would you be around in about an hour to have a chat about getting the manage-disclosure mockups ready for testing?
<wallyworld_> huwshimi: i have to go to the post office quickly after school pickup so i'll be a little later than normal getting home. i'll be back around 4:45 AEDT
<huwshimi> wallyworld_: Sure, that's fine
<wallyworld_> excellent. i'll ping you
<huwshimi> wallyworld_: Cheers
<wallyworld_> huwshimi: ping :-)
<huwshimi> wallyworld_: Hey, mumble?
<wallyworld_> huwshimi: yep, assuming it starts this time without locking up :-(
<huwshimi> wallyworld_: skype, phone etc. are fine too if need be
<poolie> why is lp.dev giving me a sad face?
<poolie> oh ok now
<poolie> i'm going to send through the update to loggerhead in the lp tree
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<poolie> huwshimi, i wonder if we could usefully autosave in progress typing into local html5 storage
<poolie> perhaps if your browser crashes it will be lost anyhow
<poolie> i don't know
<poolie> maybe it's the browser's job not to crash :)
<jtv> wgrant: help, I'm stuck!
<jtv> wgrant: remember bug 876594?
<_mup_> Bug #876594: rejected builds for synced packages send mail to Debian maintainer <derivation> <qa-untestable> <regression> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/876594 >
<jtv> I'm trying to reproduce the problem in a test, but I need something like 4 layers of objects that we don't have LaunchpadObjectFactory methods for.
<jtv> Or 5.
<poolie> add em?
<nigelb> Yak shaving weekend? :)
<poolie> personally i'm doing all the bzr reviews
<poolie> is there a tasteful place to hold an object for the duration of the current request?
<wgrant> jtv: SoyuzTestPublisher
<wgrant> poolie: No, but there are some distasteful ones. What do you want to store?
<poolie> re https://code.launchpad.net/~mbp/launchpad/391780-markdown/+merge/82832 i was going to cache the markdown converter object for later reuse
<poolie> but, it turns out not to be all that expensive to just make a new one each time its wanted
<poolie> so, nm
<wgrant> Ah
<poolie> how does the 'incremental fix' thing work?
<nigelb> multiple fixes for the same bug. I think the flag is --incremental for ec2land
<poolie> yep
<nigelb> The bug isn't set Fix Committed.
<nigelb> but you get the qa tags on it.
<poolie> that sounds like just what i want then
<poolie> i'm sending up the markdown branch as a limited beta
<nigelb> ahh. Nice.
<nigelb> Also, the new Beta thing is pretty cool.
<jtv> wgrant: that's the answer I fearedâ¦ I hope SoyuzTestPublisher gives me a say over the changes file.
<mrevell> Hello
<lifeless> nigelb: poolie: actually, thats not what happens
<lifeless> the qa process wiki page covers it I think; your revision will be skipped
<poolie> is qas not updating or something?
<poolie> i have something landed a while ago that's not on http://lpqateam.canonical.com/qa-reports/deployment-stable.html
<wgrant> poolie: A while ago?
<wgrant> poolie: Has it been through buildbot yet?
<allenap> poolie: Thanks for looking at https://bugs.launchpad.net/launchpad/+bug/894045. I was on quite the wrong tack!
<_mup_> Bug #894045: sending pgp inline in plain text causes apparent crc errors <confusing-ui> <email> <gpg> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894045 >
<jtv> Where did the statuses on the translations-import-queue pages go?
<poolie> allenap, you're welcome, thanks for the tip about rabbit
<poolie> i hit the same error case today and i got a useful message
<poolie> wgrant, 3.5h ago
<poolie> where is the buildbot again?
<StevenK> lpbuildbot.c.c
<allenap> poolie: Ah, the service details stuff? You rule for sorting that out.
<poolie> :)
<jtv> dpm: do you still see the status of entries on the import-queue pages?  They no longer show up for me.
<dpm> jtv, on which distro/project?
<jtv> Any.
<dpm> jtv, oh, I don't see them either :(
<jtv> That's pretty shocking.  I'm filing a bug.
<dpm> jtv, could you please subscribe me to it when you do?
<jtv> OK
<nigelb> lifeless: Oh. Ew. Sorry for the misinformation!
<jtv> dpm: bug 894690
<_mup_> Bug #894690: Translations import queue pages no longer show statuses <critical> <regression> <rosetta-imports> <Launchpad itself:Triaged> <Ubuntu Translations:Triaged> < https://launchpad.net/bugs/894690 >
<jtv> You're subscribed.
<dpm> jtv, got the e-mail, thanks
<wgrant> poolie: buildbot takes 4-6 hours
<jtv> dpm: the information is still there, but it's hidden.  Maybe it's a simple mistake in the HTML template.  I'll see what changed.
<dpm> ok, cool, thanks jtv
<poolie> ah ok
<rvba> Just checking: If I have a branch visible by user a, I can be sure that all the branches in the chain of branches on which this branch is stacked on is also visible by user a right?
<jtv> huwshimi: I see you replaced a class="hidden" with a style="display:none" in the translations import queue macros.  Are you quite sure we don't have something in the code that removes the "hidden" class to make it visible?  Now it stays hidden, which is a bit of a problem.
<jtv> rvba: it sounds dubiousâ¦ AIUI Launchpad doesn't even really notice the stacking â it's purely internal to bzr.
<jtv> But poolie will know more.
<poolie> the access control is all on the lp side
<rvba> I'm pretty sure branch.visibleByUser goes up the stacked on chain.
<rvba> jtv: Unless I'm mistaken Huw changed a bunch of "display:none" into class="hidden" and not the other way around.
<jtv> Hmm
<rvba> https://code.launchpad.net/~huwshimi/launchpad/style-removal-one/+merge/82096
 * rvba wonders why the preview diff is empty on this pageâ¦
<jtv> Ah yes, it's "hidden" know.
<jtv> I think sometimes the diff for the latest update happens to get computed after the merge.
<jtv> Annnd the javascript does a setStyle('display', '').
<rvba> poolie: so, if I have a branch that user a can see, can I be sure that all the branches on which this one is stacked on are also visible by user a ? (I want to prefetch the chain of stacked on branches for a list a branches but I don't want to fetch non visible branches).
<rvba> jtv: a bug it is then.
<jtv> The probable cause of bug 894690.
<_mup_> Bug #894690: Translations import queue pages no longer show statuses <critical> <regression> <rosetta-imports> <Launchpad itself:Triaged> <Ubuntu Translations:Triaged> < https://launchpad.net/bugs/894690 >
<rvba> Very probable indeed.
<rvba> Also means that a test is missing.
<jtv> That's the nasty part.  This kind of change is hard to predict, so it's easy to write a decent unit test that passes.  And a full-on browser test would be overkill.
<rick_h_> morning
<rick_h_> does anyone have an IE browser handy for a quick QA thing?
<nigelb> hah.
<jml> rick_h_: I wish.
<jml> hey, what tag should I use to file bugs on the new bug listing thingummy?
<rick_h_> heh, thanks
<rick_h_> hmm, good question. sec.
<rick_h_> wjat
<rick_h_> what's the bug?
<rick_h_> probably just buglistign
<rick_h_> errr, buglisting
<jml> rick_h_: I have a few different ones, actually :)
<rick_h_> jml: heh, ok
<rick_h_> jml: nvm, looks like it's bug-columns that everyone is using
<rick_h_> https://bugs.launchpad.net/launchpad/+bugs?field.tag=bug-columns
<jml> rick_h_: yeah, thanks. Found that due to post-filing editing
<jml> really should fix that bug about tags not auto-completing on filing
<jml> would save everyone a heap of time
 * jml makes a note to dig that up
<jml> mrevell, rvba, matsubara: I think I'm done with my bug filing spree.
<nigelb> lol, spammer :P
<mrevell> jml, Thanks, much appreciated :) I'm leaving some comments
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Critical bugtasks: 292
<jml> it's a very exciting feature
<jml> and I like the direction that the new look is taking
<nigelb> When I saw the Beta "bar", it almost looked modern.
<nigelb> I'm hoping with some of the features mrevell was talking about at UDS, Launchpad will be much much much better ;)
<mrevell> nigelb, Well, it's a slow march.
<mrevell> jml, I believe the new bug listing look came out of your discussions with Huw, right?
<nigelb> :)
<jml> mrevell: did it really? I can't recall.
<matsubara> jml, some of the bugs you filed I found out during ET and told Deryck about. Developers are aware of those issues. Thanks for filing those bugs!
<mrevell> jml, That's what Huw said :)
<mrevell> jml, I hate to ask, because it's a hard one to answer, but do you have a suggestion for how we can make the "Edit your visible info" cog more prominent? Originally, we had a "Edit visible information" link but it was kinda ugly and still not that noticeable.
<rick_h_> I was just looking at that one, the problem is that you don't use it much right?
<rick_h_> you'd set your preference and not need it again
<jml> well, that's not quite right, actually
<rick_h_> so making it more prominent just gets distracting after first use?
<jml> because different projects have different listing needs
<jml> and I haven't really explored the personal listing pages yet
<jml> mrevell: tbh, I'd just make it more high-contrast
<matsubara> mrevell, maybe make it a bit darker would help?
<mrevell> Yeah and something we deemed out of scope for now was the ability to save multiple listings.
<mrevell> Two people saying the same thing; I take that as meaning it is the truth.
<matsubara> jml, personal listings don't have the sorting widget or the cog
<matsubara> don't have yet, I meant :-)
<mrevell> matsubara, It's a bug.
<matsubara> yep, I know. I filed it
<jml> mrevell: maybe make it a little bigger, and make the grey a shade darker? *shrug*
<mrevell> bug 894437
<mrevell> jml, Sounds about right
<mrevell> matsubara, heh, right
<jml> mrevell: can't we just punt that to Huw? :P
<mrevell> jml, Sure :)
<jml> also, one of your earlier comments reminded me of this quote
<matsubara> mrevell, stand up?
<mrevell> Is a cog obvious enough
<mrevell> matsubara, Yus, Skype okay?
<mrevell> Ekiga hates me atm
<matsubara> mrevell, sure. logging in
<jml> (It's about the waters of a fictional heavenly river) "When you have drunk of it you forget forever all proprietorship in your own works. You enjoy them just as if they were someone else's: without pride and without modesty."
<mrevell> Excellent :)
<jml> I think a cog icon is a fine metaphor
<jml> I'm surprised Huw hasn't replaced the edit icon in a fit of whim & fancy yet :)
<mrevell> jml, We've been talking about it; he'd like to wait until the rebrand, so he doesn't have to do two versions.
<jml> OK
<rick_h_> heh, it took me forever to figure out that was edit
<rick_h_> looked like an exclamation point, I kept thinking "hmm, this is important"
<flacoste> morning
<mrevell> Morning flacoste!
<rick_h_> morning
<rick_h_> anyone know if I'm going to break anything if I just change the dev apache config virtualhost all to *:80/443 vs the set up 127.0.0.88 and such?
<flacoste> rvba: i'd suggest you make all bugs on the bugs-columns feature 'High'
<flacoste> rvba: remember that we use High for feature work to indicate bugs we'd like to fix before release
<flacoste> rvba: when we need to reduce scope, we review the HIgh list
<flacoste> whereas we are less likely to review the Low list
<flacoste> not a big deal, but a little time-saver
<rvba> flacoste: ok, thanks for the heads up.
<flacoste> rvba: thanks for doing triaging!
<mrevell> Hey abentley, bug 887232: the sort order widget and cog don't appear on person/team pages. Do you know what's causing that bug?
<abentley> mrevell: OTP
<rick_h_> abentley: http://sorescode.com/2010/09/12/benchmarks.html
<abentley> mrevell: It looks like the ajaxification is completely broken on those pages.
<abentley> mrevell: may be related to bug #887214
<mrevell> abentley, Okay, thanks. Is it something the Orangery are working on?
<abentley> mrevell: it wasn't, but it looks like it warrants it.  I think the pages will still be usable, just in non-ajax mode.
<mrevell> abentley, Is it not fixable so that the ajax works?
<abentley> mrevell: I mean until we fix it, it should be usable in non-ajax mode.
<mrevell> abentley, I'm not sure I understand. Does that mean there's some way I can see a non-ajax version of the sort ordering and cog widgets on personal bug pages?
<abentley> mrevell: No, there's no non-ajax version of the sort ordering and cog widgets.  I was talking about the basic batch navigation.
<mrevell> Okay, thanks for clarifying.
<abentley> mrevell: FWICT, the Person bug pages are not even trying to ajaxify the listings, so it's probably an easy fix.
<mrevell> That's good to hear :)
<gmb> bac: I have a simple one for you, if you've got the time: https://code.launchpad.net/~gmb/launchpad/archive-subscriptions-bug-823473/+merge/83407
<bac> gmb: sure
<gmb> Thanks
<bac> gmb: done
<gmb> Thanks
<rvba> Hi bac, could you please have a look at https://code.launchpad.net/~rvb/launchpad/activereviews-bug-867941-eagerload3/+merge/83389 ?
<bac> rvba: will do
<rvba> Thank you.
<bac> nice branch rvba
<rvba> Thanks for the review bac!
<abentley> bac: could you please review https://code.launchpad.net/~abentley/launchpad/person-bug-listings/+merge/83418 ?
<abentley> bac: nm.  hold on.
<abentley> bac: okay,  https://code.launchpad.net/~abentley/launchpad/person-bug-listings/+merge/83418 is ready for review.
<mrevell> Night all. I shall be retiring for the weekend.
<bac> abentley: yes, i can now
<abentley> bac: excellent.  I'm just back from lunch.
<bac> abentley:  me too  :)
<bac> hope yours was as good
<abentley> It was nice.
<bac> abentley: is custom bug listing enabled in production only for some projects?
<abentley> bac: No, it's enabled for all projects, but at their "+bugs" view, not their home page.
<bac> ah, gotcha
<abentley> bac: But for project *groups*, the +bugs view *is* the home page.
<cr3> why do launchpad tests run with xvfb?
<bac> cr3: was needed when we used windmill
<rick_h_> guessing at some point we use a headless browser from the cmd line, like phantomjs type of thing that requires that to run
<rick_h_> or like windmill :)
<flacoste> bac: still needed
<cr3> bac: thanks, I know about sinzui's html5-browser but curious to know if there are still signs of windmill in launchpad. has it all been migrated?
<flacoste> for the JS tests
<flacoste> webkit requires a display
<bac> flacoste: i suspected so.  didn't mean to say it was no longer needed.
<flacoste> cr3: windmill has been gone for a couple of months
<flacoste> but the xvfb is still needed for html5-browser
<cr3> flacoste: thanks, I've been running my tests interactively but now I know what to do for ec2 :)
<flacoste> cr3: yes, we only use xvfb when running in ec2
<flacoste> or other simlar headless environment
<flacoste> it shouldn't be needed on a local workstation
<flacoste> unless you are one of those crazy developpers who have emacs as their init program
<cr3> flacoste: I don't use the emacs operating system myself :)
<flacoste> good for you :-)
<abentley> bac: could you please review https://code.launchpad.net/~abentley/launchpad/mandatory-bug-title/+merge/83446 ?
<bac> abentley: shirley
<bac> abentley: done
<abentley> bac: thanks.
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
<poolie> well now
<poolie> https://code.launchpad.net/~mbp/launchpad/798412-plusone/+merge/83449
<mtaylor> hey all - I'm having odd problems with ppa uploads
<mtaylor> https://jenkins.openstack.org/job/glance-ppa/label=build,series=lucid/273/console for instance
<mtaylor> the files are, in fact, being signed by the gpg key that is on the account of that user in launchpad
<mtaylor> so wtf?
<lifeless> bug 798957 perhaps ?
<mtaylor> lifeless: ah. so potentially clock skew... that would be reasonable given the machines... lemme see if I can fix
<lifeless> mtaylor: its a server side glitch
<lifeless> mtaylor: cosmetic,t he uploads are still processed
<lifeless> mtaylor: freaking annoying to reproduce / fix
<mtaylor> lifeless: ah. lovely
<lifeless> works for days, then boom stops.
<mtaylor> lifeless: well, fwiw, it seems to be hitting all of our ppa uploads from openstack's jenkins
<mtaylor> awesome
<lifeless> restart the service. Works.
<mtaylor> beautiful
<mtaylor> so - for now I should just "dput blah || echo true" to avoid it looking like all of my uploads are failing?
<lifeless> :9
<lifeless> :(
<lifeless> yes
#launchpad-dev 2011-11-26
<wgrant> Well, that was easy.
<wgrant> (making breadcrumbs depend on facet, not host)
<wgrant> And there we go. All working on a single webapp vhost.
<StevenK> And how large is the branch
<wgrant> Only around 3500 lines so far. 2/3 of that is just mechanical changes: help paths, removing CodeLayer and TranslationsLayer restrictions.
<wgrant> (several branches, of course)
<wgrant> 200 lines of breadcrumb changes, 1000 lines of help rework, 1800 lines of view delayering (good lord, our ZCML is terrible), 400 lines of facet menu changes.
<wgrant> Remaining things are probably just: standardising some old default view names; fixing non-rootsite canonical_url calls to do the right thing; renaming TranslationsLayer:DistroSeries:+admin to something that doesn't conflict with DistroSeries:+admin; and possibly renaming a few translations views like +export and +import to be slightly more namespaced.
<StevenK> Distroseries:+translation-admin ?
<StevenK> I'm guessing it is not really landable in it's current state.
<wgrant> Yeah, that was the plan.
<wgrant> Two of the branches are completely landable, another one probably is.
<wgrant> (DistroSeries:+admin is the only name conflict in the entire application... we've made such good use of the subdomains...
<StevenK> Hah
<StevenK> One thing I'm worried about is bookmarks.
<wgrant> I'm not sure it's worth redirecting eg. bugs.launchpad.net/launchpad to +bugs. We can probably just redirect to launchpad.net/launchpad and let people sort their bookmarks out slightly.
<wgrant> (that lets us do a plain Apache redirect from the old webapp subdomains. much cleaner)(
<StevenK> I remain unconvinced. But if removing the subdomains is a large win, let's do it.
<StevenK> And then let's kill edge entirely
#launchpad-dev 2011-11-27
<lifeless> wgrant: can you check in one of your lucid lxc's, is /var/run a tmpfs, or a naked dir ?
<nigelb> Morning lifeless
<lifeless> morning nigelb
<nigelb> :)
<lifeless> question for the floor - I need a sanity check - set_session_pkg_data() takes parameters that need obfuscating?
<poolie> lifeless, thanks for the partial review
<lifeless> no probs
<wgrant> lifeless: Obfuscating how?
<lifeless> wgrant: have a look at OOPS-3cbd11de34ca8a70afaf668853adc17e in bad-oops on carob
<lifeless> wondering if we should be preventing query param substition
<wgrant> Oh, in query logs, I see.
<wgrant> lifeless: Where is bad-oops?
<lifeless> lpoops/
<wgrant> That's not a directory.
<wgrant> Ah, there, I see.
<lifeless> wgrant: its a sidestraction from https://bugs.launchpad.net/python-oops-datedir-repo/+bug/896959
<wgrant> That probably is somewhat sensitive, yeah.
<lifeless> I happened to notice it ;)
<lifeless> wgrant: I propose to monkey patch update-db to handle unicodedecode error until the fix is deployed.
<wgrant> lifeless: That would be my suggestion.
<lifeless> done
<poolie> anyhow thanks again for the prompt review, i replied
<poolie> incidentally the button is already hidden on private bmps
<wgrant> Heh
<lifeless> poolie: how is it hidden?
<wgrant> Bug 896968, bug 896691
<lifeless> poolie: I didn't see a clause fo rit
<poolie> line 132 of the diff
<poolie> +    tal:condition="not:view/is_private"
<poolie> on the macros that insert the buttons
<poolie> i was quite pleased with that
<poolie> but maybe there's a better way
<poolie> much better than counting on every template getting it right
<poolie> lifeless, i was hoping you'd say "oh that's nice" :)
<lifeless> poolie: heh, sorry.
<lifeless> uhm
<lifeless> thats probably reasonable
<poolie> lifeless, i was hoping you'd say "oh that's nice" :)
<poolie> oops
<lifeless> deja vue
<poolie> typo, not trying to be whiny
<wgrant> I'm saying that your second-latest comment is spreading common misunderstandings.
<lifeless> feel free to note that
<lifeless> both aaron and I were spreading it
<lifeless> however we don't have -any- bug.datecreated search in bugtask.py
<lifeless> so while its true that the bug datecreated field can be different
<lifeless> its not relevant to the bug
<wgrant> bugtask.datecreated will be similarly different.
<wgrant> Or at least should be.
<lifeless> yes
#launchpad-dev 2012-11-19
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/bugzilla-greater-than-36/+merge/134825
<wgrant> StevenK: Have you tried it locally on old, new, and libav instances?
<StevenK> wgrant: I have tried with libav and abisource, but I'm having trouble working out what bugzilla version it is
 * StevenK uses the code to tell him
<StevenK> Right, abisource is 2.22.7
<wgrant> So both old
<wgrant> Perhaps try GNOME
<StevenK> libav is 3.6.2
<StevenK> So not really
<wgrant> Hm, wasn't the problem that libav didn't understand the new param?
<StevenK> libav tossed an error page with bug_id_type=include, but worked without it or with bugidtype=include
<StevenK> wgrant: I can try a gnome watch if you wish
<wgrant> StevenK: So the param name changed in 3.6.2, but the compat code was only added later?
<StevenK> wgrant: And then removed -- I can't find it in 4.2, but it may have moved
<StevenK> https://bugs.launchpad.net/bugs/bugtrackers/auto-launchpad.net
<wgrant> StevenK: Anyway, if the param was changed in 3.6.2, you'll want to spell the relation '< 3.6.2' or '>= 3.6.2', not anything about 3.6.1
<StevenK> gnome blows up
<StevenK> Which is odd
<StevenK> Fault: <Fault Client: "Content-Type must be 'text/xml,' 'multipart/*,' 'application/soap+xml,' 'or 'application/dime' instead of 'application/x-www-form-urlencoded'">
<wgrant> Are you sure you're using the right bugzilla client?
<StevenK> I grabbed a URL from prod
<wgrant> Right, but our bugzilla client behaves slightly differently depending on the plugins available on the remote instance
<wgrant> It's possible you're creating the wrong one directly.
<StevenK> http://bugzilla.gnome.org/show_bug.cgi?id=686321 is the URL I gave
<wgrant> Sure
<wgrant> But that's not really relevant
<wgrant> What's relevant is the externalbugtracker class you're using
<StevenK> Hm, indeed. That one is using XMLRPC, and didn't trigger my pdb
<StevenK> So now I need a recent bugzilla
<bigjools> it's very annoying that people can add new questions on a project after turning off Answers
<wgrant> Indeed
<StevenK> Which gives a 404 for some reason :-(
<StevenK> Bleh, icedtea has the plugin
<StevenK> As does libreoffice
<StevenK> This is maddening
<StevenK> wgrant: So I guess Bugzilla 4.2 and on include the LP plugin -- I seem to recall something like that, anyway. I guess I need to find a 3.8 running somewhere
<StevenK> wgrant: Which I can't seem o find.
<wgrant> StevenK: there's a genericised version of the LP plugin included as core XML-RPC functionality in 4.something
<StevenK> Right
<StevenK> wgrant: Which I seem to be hitting everytime I find a 4.2 bugzilla. I can't find a 4.0 or a 3.8 instance, so what do you think?
<wgrant> Hmm
<StevenK> wgrant: No real thoughts then?
<wgrant> StevenK: Well, you could possibly just make it not use XML-RPC even if it's available
<wgrant> See if that lets you test the old API on a new instance
<StevenK> Hmmm
<StevenK> Let me cowboy that in
<StevenK> wgrant: Which gives a 404, oddly
 * StevenK stabs bugzilla
<StevenK> wgrant: So, that doesn't work, since we try and probe the version by using xml.cgi?id=1 which gives a 4040
<StevenK> 404, even
 * StevenK hacks around that too
<StevenK> wgrant: Right, with some gruesome hacks and the change in the MP, updating a bug from libreoffice's 4.2.3 bugzilla works.
<StevenK> wgrant: *prod*
<wgrant> StevenK: So, it works on an old, a libav, and a new?
<StevenK> wgrant: Yup
<wgrant> StevenK: Do you have a link to the revision in the 3.6 branch?
<StevenK> I do not
<StevenK> I'm guessing, based on how libav behaves
<wgrant> How'd you determine the change is in 3.6.2, then?
<StevenK> I can't find any bugzilla 3.6.1 instance, so I'm still guessing
<wgrant> Ah
<wgrant> So you'll need to check that
<StevenK> wgrant: http://bzr.mozilla.org/bugzilla/3.6/revision/6940
<wgrant> StevenK: That's before 3.6.0.
<wgrant> So >= 3.6.0, I suppose
<StevenK> Just before 3.53.
<StevenK> 3.5.3 even
<wgrant> But it's in the 3.6 branch
<wgrant> The change referencing 3.5.3 could just be a merge, perhaps
<StevenK> 6950 	
<StevenK> 	
<StevenK> Convert .cvsignore files into a .bzrignore.
<StevenK> 	Max Kanat-Alexander 	bugzilla-3.5.3 	2010-02-01 	Diff 	Files
<StevenK> It's tagged as 3.5.3, at least
<wgrant> Hm
<wgrant> Anyway, just say 3.6.0 for now, I think
<StevenK> wgrant: http://pastebin.ubuntu.com/1369400/
<wgrant> StevenK: Right, that seems less arbitrary
<StevenK> wgrant: The MP has updated
<StevenK> wgrant: Can haz approval?
<wgrant> StevenK: Done
<StevenK> wgrant: You lose at Buildbot bingo -- xx-person-packages.txt
<StevenK> Due to sampledata, I bet
<StevenK> Since wallyworld___ didn't run the garbo job against it
<wgrant> Except there was data there
<StevenK> I
<wgrant> And the code was already removed initially, as you might recall
<StevenK> Bleh
<StevenK> I'm guessing, based on the failure
<wgrant>     >>> removeSecurityProxy(source1_mark).archive = (
<wgrant>     ...     mark_private_ppa)
<wgrant> That might do it
<StevenK> Haha
<StevenK> wgrant: Have you got a testfix yet?
<wgrant> Not yet
<wgrant> Considering rewriting the test
<StevenK> Haha
<StevenK> That good, is it?
<wgrant> It moves publications multiple times :(
<StevenK> Right, so it's a stupid test.
<StevenK> I love how the Soyuz doctests have a flagrant disregard towards consistent archives
<bigjools> StevenK: must.get.test.passing.at.all.costs
<StevenK> Without using anything but sampledata, yeah.
<StevenK> If I could, I'd delete cprov from the sampledata just to see what blows up.
<wgrant> Oh
<wgrant> wallyworld___ actually fixed this test to regen the table
<wgrant> After it molests SPPH
<StevenK> Did you remove that bit?
<wgrant> No
<wgrant> I think I just need to also clear out garbojobstate
 * wgrant tries
<wgrant> yeah
<wgrant> wallyworld___: Do you have some unpushed changes to the dbpatches branch?
<wgrant> I don't see your LPSPRC numbers there
<stub> wgrant: There is more package cleanup happening this week?
<adeuring> good morning
<wgrant> stub: We did it a few hours ago
<wgrant> stub: https://code.launchpad.net/~wgrant/launchpad/lpsprc-index-harder/+merge/134848
 * stub wonders if postgresql enums are improved enough to be useful
<stub> wgrant: That patch is fine. Should I be applying it anywhere?
<wgrant> stub: Everywhere would be lovely
<wgrant> An enum wouldn't exactly help here
<wgrant> Since we still need it ordered
<wgrant> Even if it could use the index for !=
<stub> wgrant: To replace hard coded constants like '2'
<stub> Not just indexes, but all the raw SQL.
<wgrant> Ah, I thought you meant in terms of being able to answer != from indices
<wallyworld___> wgrant: i have no unpushed changes
<wallyworld___> what is missing?
<wgrant> wallyworld___: The description for 38-0 (the creation of LPSPRC) is wrong, and 38-1 (LPSPRC person FKs) and 38-2 (LPSPRC indices) aren't there at all
<wallyworld___> hmmm. 38-1 and 38-2 were pushed to devel
<wallyworld___> from memory
<wgrant> Yeah, but they're not in allocated.txt
<wgrant> The patches exist, but the numbers aren't registered as being in use
<wallyworld___> ah, yes. i only reserved the first 38-0
<wallyworld___> i can fix if you like
<wgrant> That would be great, if you haven't thrown away all your LP-related branches yet :)
<wgrant> Otherwise I can
<wallyworld___> no, tomorrow :-P
<wallyworld___> i'll do it in a bit after my standup
<wallyworld___> which is now at 8pm
<wallyworld___> stupid timezones
<wgrant> Ew :/
<wgrant> Reminds me of the good old Soyuz days
<czajkowski> there were ever good Soyuz days ?
<bigjools> yes
<stub> wgrant: we can do prod after the backups
<wgrant> stub: Indeed, thanks
<wgrant> Still
<wgrant> They should be done in 5-10 mins
<wgrant> unless something's gone wrong
<wallyworld___> wgrant: dbpatches changes pushed
<wgrant> wallyworld___: Thanks!
<wallyworld___> np. sorry about the omission. i totally forgot
<smartboyhw> Hi guys. When I go to the new CoC page in qastaging to see the new CoC 2.0, I saw that the release date is not changed. Is it because it is still not released officially?
<smartboyhw> Or is that a bug actually?
<czajkowski> still not officaly released
<smartboyhw> czajkowski, ah OK. When is it going to be officially released then?
<czajkowski> smartboyhw: soon no eta on it
<czajkowski> we're waiting on a few things to align
<czajkowski> you;ll know as there will be an annoucement on the planet ubuntu about it
<StevenK> czajkowski: Does that mean that only 1.1 is still on prod?
<czajkowski> StevenK:  Not  sure rick_h is looking after this bug atm
<czajkowski> we've a bug tracking it
<StevenK> Should it get handed over, since they're not on maint?
<czajkowski> StevenK: he as handling the merge to ticks
<czajkowski> let me find the bug
<czajkowski> StevenK: https://bugs.launchpad.net/launchpad/+bug/1079654
<_mup_> Bug #1079654: CoC isn't version 2.0  <codeofconduct> <qa-ok> <Launchpad itself:Fix Committed by rharding> < https://launchpad.net/bugs/1079654 >
<StevenK> Ah, it hasn't been deployed yet
<czajkowski> narp
<czajkowski> and we've a few other bits to do also elsewhere
<czajkowski> must go poke website editors also
<StevenK> So we can't deploy it until those bits are done?
<czajkowski> StevenK: sent to pm
<rick_h> czajkowski: so the bug is qa'd and not looked yet if a NDT has been run. catching up this morning still
<czajkowski> nods
<czajkowski> well we're good to go on our end re other bits falling into place
<rick_h> czajkowski: we've got a bunch of QA to do this morning. I'll try to get a deploy after our stand up in 3hrs
<rick_h> czajkowski: so will try to get it deployed today on this side of the pond
<czajkowski> lovely thanks
<jtv> My attempt at merge proposal is oopsing... but it works for another branch!
<jtv> Unfortunately the oops listing seems to be oopsing as well.
<sinzui> jtv, ask a webops to unscan your branch. He will need the lp:... name
<jtv> Unscan?  Didn't know you could do that.  Thanks.
<rick_h> abentley: is there a trick to make the test runner not use a subprocess? I want to pdb and can't in a subprocess test
<abentley> rick_h: If you run a single layer, it shouldn't run a subprocess.
<abentley> rick_h: It uses subprocesses if the layer can't be torn down and there are other layers to do.
<rick_h> abentley: ah thanks.
<rick_h> yea, using --layer helps me get my pdb back yay
<abentley> rick_h: Good.
<czajkowski> sinzui: you about?
<sinzui> yes
<czajkowski> sinzui: ello, is this something you cna help with, https://answers.launchpad.net/launchpad/+question/214619
<czajkowski> we can askk web ops to change the owner, but the bit allenap has saida needs run is that something you can do?
<sinzui> only if we own then and nothing ever merged them
<sinzui> the user who merged the branches need to delete their branched
<czajkowski> :s
<sinzui> We can mark the branches as obsolete to hide them from most searched
<czajkowski> ah
<sinzui> czajkowski, we could also also change the owner so that they can restore the branch is they need to
<czajkowski> sinzui: that nmight be the easiest in the long run
<abentley> rick_h: There's no OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/transitive-confidential/+merge/134995 ?
<rick_h> abentley: sure
<abentley> rick_h: Thanks.
<rick_h> abentley: r=me
<abentley> rick_h: Thanks.
<rick_h> abentley: side question though on that. I'm trying to think of other things that might need to be checked
<rick_h> would milestones/series go into that as well?
<rick_h> e.g. "You've got a milestone out there, can't change the project"?
<abentley> rick_h: No, because the milestone and series information types are controlled by the product information type.
<rick_h> ok
<abentley> I'm certainly open to adding more checks as needed.
<czajkowski> rick_h: did you approv of this weeks desktop :)
<abentley> sinzui: Are you familiar with the query used by _getProjectsWithTheMostKarma and if so, would it be simpler to phrase it as http://pastebin.ubuntu.com/1370868/ ?
<sinzui> abentley, I think your query is sane, but I need to see the old query to remember it
 * sinzui looks
<sinzui> abentley, I like your change. I think the test for it in test_person will pass
<abentley> sinzui: Thanks.
<rick_h> czajkowski: I did notice it
#launchpad-dev 2012-11-20
<StevenK> wgrant: http://sourceforge.net/apps/trac/sourceforge/wiki/API => no bugs
<wgrant> Ah
<wgrant> There's a bug RSS feed, but that's it
<StevenK> Which doesn't help
<StevenK> wgrant: So I just created two bugwatches pointing to libav, bug 16 had a invalid bug number, and bug 17 had a valid one. After running checkwatches, bug 16 is still Unknown/Unknown but bug 17 is Fix Released/Medium
<_mup_> Bug #16: "Swedish" and "Swedish (Sweden)" should be the same language <lp-translations> <Launchpad itself:Fix Released by daf> < https://launchpad.net/bugs/16 >
<_mup_> Bug #17: System error <lp-translations> <Launchpad itself:Fix Released by carlos> < https://launchpad.net/bugs/17 >
<_mup_> Bug #16: "Swedish" and "Swedish (Sweden)" should be the same language <lp-translations> <Launchpad itself:Fix Released by daf> < https://launchpad.net/bugs/16 >
<_mup_> Bug #17: System error <lp-translations> <Launchpad itself:Fix Released by carlos> < https://launchpad.net/bugs/17 >
<StevenK> _mup_: Go away
<StevenK> wgrant: So I guess bug 634906 is fixed
<_mup_> Bug #634906: An invalid remote bug ID can cause a checkwatches run to break completely <bugwatch> <lp-bugs> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/634906 >
<wgrant> StevenK: Excellent.
<StevenK> So that's two nailed shut today
<StevenK> Three if we deploy
* wgrant changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~180
<StevenK> wgrant: Turns out urlopen that initializeRemoteBugDB calls uses the default timeout value. Perhaps we should set it to 180 and move on?
<StevenK> It might be more sinister
<wgrant> StevenK: Perhaps block the connection locally and see if you can reproduce the behaviour?
<StevenK> The urlopen call has a ensure_no_transaction decorator
<wgrant> Sure, we know it has no transaction open
<wgrant> Since it doesn't get idle-killed
<StevenK> Right
<StevenK> I was wondering if that was what was causing the spining
<StevenK> wgrant: Right, I have forbidden my computer to talk to libav's bugzilla
<wgrant> -j DROP, I hope?
<StevenK>  Well, duh
<StevenK> I don't want checkwatches getting port unreachable or wind of it
<StevenK> It's still trying
<StevenK> 36 dropped packets
<wgrant> Hmm hmm
<wgrant> Leave it for half an hour and see what happens :)
<StevenK> It hasn't logged anything about libav
<StevenK> Which is naughty
<wgrant> It usually does
<wgrant> Oh, unless it's hung on some preprobe
<StevenK> Which is likely, it's trying to get the version
<StevenK> 2012-11-20 03:59:01 INFO    Updating 1 watches for 1 bugs on http://bugzilla.libav.org
<StevenK> It seems it gave up
<StevenK> Now it's probably POSTing
<StevenK> 2012-11-20 04:01:08 INFO    Error updating http://bugzilla.libav.org/: http://bugzilla.libav.org: <urlopen error [Errno 110] Connection timed out>
<StevenK> That's disappointing
<StevenK> urlopen uses socket._GLOBAL_DEFAULT_TIMEOUT if it isn't set, which is defined as object()
<StevenK> socket.create_connection will only call socket.settimeout if the timout isn't that
<StevenK> It could be a proxy thing
<StevenK> squid just keeps a open connection
<StevenK> But I'm grasping at straws
<wgrant> Possibly
<StevenK> wgrant: It may explain why neither of us can reproduce it locally
<StevenK> And the squid I just installed gives a 504
<StevenK> But that's default config
<wgrant> IIRC I even tried it from behind squid.internal
<wgrant> And it worked
<StevenK> Strange
<wgrant> Have you checked the last couple of hangs?
<StevenK> I have not
<StevenK> But I'll stop scratching my head over this and move onto glaring at Redhat XMLRPC
<StevenK> wgrant: Ah ha. We send Bugzilla.time as a XMLRPC to redhat's tracker and get back Fault -32000
<wgrant> StevenK: So the method to get the remote tracker's current time fails?
<StevenK> Yeah
<StevenK> <?xml version='1.0'?>\n<methodCall>\n<methodName>Bugzilla.time</methodName>\n<params>\n</params>\n</methodCall>\n'
<wgrant> Oh, not even any params? Nice
<StevenK> Is what we send
<wgrant> I guess we'll need to file a bug with them :)
<StevenK> I can't find any docs about Bugzilla.time
<wgrant> What's the full traceback we get from them?
<wgrant> I assume it works on other bugzillae
<wgrant> but theirs is broken
<StevenK> wgrant: Yeah, it seems fine with others
<StevenK> Let me hack xmlrpclib
<StevenK> ({'faultCode': -32000, 'faultString': 'Cannot compare a datetime to a regular scalar at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/DateTime.pm line 1385.\n'},)
<StevenK> That's the full stack in xmlrpclib.close
<wgrant> https://bugzilla.redhat.com/enter_bug.cgi?product=Bugzilla&version=4.2 might be your friend
<wgrant> I suspect
 * StevenK stabs bugzilla
<StevenK> wgrant: http://pastebin.ubuntu.com/1371847/
<StevenK> The first run is with the ooo proxy commented out, the second with the redhat one
<wgrant> yep
<StevenK> Bleh
<StevenK> Create an account? :-(
<wgrant> Yeah
<wgrant> If only all bugtrackers supported arbitrary third-party OpenID providers :)
<StevenK> bugzilla-owner@redhat.com is the registered owner, we could just mail them ...
<wgrant> Well
<wgrant> You know we always ask for bugs to be filed
<wgrant> Let's extend the same courtesy to them!
<StevenK> wgrant: Bugzilla bug filed, LP bug updated, card created and marked as blocked
<wgrant> Did you create a bugwatch so we can check its status? :P
<StevenK> I can't!
<StevenK> And noted that in the LP bug, so bugger off
<wgrant> Hm, why can't you?
<StevenK> The watch will never get updated
<StevenK> The RedHat bugtracker is disabled
<StevenK> So there is little point
<wgrant> Ah, right, I was thinking that if they'd fix it then we'd know
<wgrant> But not if it's disabled
<StevenK> Not turning it back on, either. We can do without an extra 2500 OOPSes a day
<adeuring> good morning
<stub> Trying to bludgeon buildout into running in offline mode is a real pain in the arse
<stub> Something is seriously broken when you get download errors when running in offline mode. WTF were you looking there in the first place stupid?
<czajkowski> stub: you're being a bit harsh on yourself today
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: frankban | Firefighting: - | Critical bugs: ~180
<stub> Umm... don't we stop bugs being duplicates of each other?
<stub> oh, nm
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: frankban, rick_h | Firefighting: - | Critical bugs: ~180
<smartboyhw> Hmm yesterday I asked about the release date of CoC not correct on Launchpad QAstaging. Now CoC 2.0 is official, but then the date is still not changed....
<czajkowski> smartboyhw: date doesnt need to change
<czajkowski> it's up to date.
<smartboyhw> czajkowski, you can't say that the CoC 2.0 is released in 2005 (or can you?)
<czajkowski> I can and I did :) it needs to be done that way for reasons  like making sure anyone who has signed previous versions are still valid.
<rick_h> czajkowski: ah ok. Yea missed the date thing when I pushed
<czajkowski> rick_h: no tis fine
<czajkowski> checked with wgrant and co that it needs to be that way
<czajkowski> so tis grand
<czajkowski> thanks rick_h
<rick_h> does look strange. been many moons since I did CoC stuff (e.g. signed it)
<czajkowski> rick_h: like the blog post :)
<rick_h> ok, well I'll stop looking at what to fix then.
<czajkowski> rick_h: :)
<czajkowski> dont fix what's not broken :)
<czajkowski> I can find lots of bugs if you want to fix stuff :p
<rick_h> heh, I'll just go back to fixing the bug I've been working on
<rick_h> adeuring: ping, have time to help me out with something?
<adeuring> rick_h: sure
<rick_h> hangout ok?
<adeuring> rick_h: sure
<czajkowski> rick_h: you see errors like this before https://bugs.launchpad.net/launchpad/+bug/1081131
<_mup_> Bug #1081131: Specifications privacy: error when trying to change sharing settings <Launchpad itself:New> < https://launchpad.net/bugs/1081131 >
<rick_h> czajkowski: stand up, but will peek in a few
<czajkowski> np
<rick_h> czajkowski: I've not seen that exactly and the bug wanders a bit so not 100% sure. Most things consider owner.
<rick_h> I've not looked at the sharing ui itself which is seems he got access to bug couldn't change. Maybe deryck or abentley can speak more to that specifically
<czajkowski> nods
<czajkowski> sorry for pinging you just saw you online :) wasnt picking on you :p
<rick_h> that's ok, I'm more than happy to play flight directing guy and wave you over away :P
<czajkowski> lol
<czajkowski> deryck: how is the little one post op ?
<czajkowski> all good I hope
<abentley> I haven't worked on the sharing UI.  It does sound like it should only be shown as editable to folks with Launchpad.Edit on the appropriate members.
<deryck> czajkowski, hi.  she's good, thanks! A little sore this morning as you'd expect but she's getting better with each moment.
<czajkowski> great to hear
<czajkowski> I had mine out as an adult and was back to eating as normal 3 days later.
<czajkowski> but bf had his out there last year, took him 2 weeks to recover
<czajkowski> everyone recovers differently
<deryck> she's eating pretty good now herself, albeit soft stuff like mac-n-cheese and mashed potatoes.
<czajkowski> good going
<deryck> she was starving.  but my little would consume the doors of the kitchen if we'd let her.
<czajkowski> deryck: while you're here you might be able to shed some light on https://bugs.launchpad.net/launchpad/+bug/1081131
<_mup_> Bug #1081131: Specifications privacy: error when trying to change sharing settings <Launchpad itself:New> < https://launchpad.net/bugs/1081131 >
<deryck> czajkowski, yes, was just looking at that.  I agree with abentley's assessment above.  we need to ensure we don't show the edit icons to the wrong folks.
<deryck> czajkowski, but it does sound like we already block the action, per the bug's description/
<deryck> czajkowski, I've traiged it now.
<deryck> triaged it even
<czajkowski> grand job thanks folks
<deryck> new title to make it clearer, too.  bug 1081131
<_mup_> Bug #1081131: +sharing should not show edit icons if you don't have launchpad.Edit <private-projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1081131 >
<rick_h> deryck: or abentley either of you have a sec to review since I'm OCR? I'm particularly want to make sure I'm saying that the ProjectGroup part is correct. https://code.launchpad.net/~rharding/launchpad/filter_more_products/+merge/135164
<abentley> rick_h: I can have a look.
<rick_h> abentley: thanks
<abentley> rick_h: You are right about ProjectGroups.
<rick_h> abentley: ok cool. The use in top_projects had me thinking I was missing a connection that I would allow leakage through
<abentley> rick_h: Where possible, I've been updating APIs to accept a user parameter, rather than using ILaunchBag.user.  Have you looked at that option?
<rick_h> abentley: no, I didn't look. I'll chase it up to browser and see if I can add the user
<rick_h> cargo culting usage fml heh
<abentley> rick_h: If you could, that would be great.
<rick_h> abentley: rgr, makes sense now that you say it
<abentley> rick_h: Could you please review https://code.launchpad.net/~abentley/launchpad/proprietary-karma/+merge/135188 ?
<deryck> sinzui, I'm about to head out to lunch, but would love to catch some voice time with you after that if you have time.
<rick_h> abentley: sure thing
<abentley> rick_h: Thanks.
<rick_h> abentley: r=me, I bow before your storm-fu turning the manual sql into storm
<rick_h> though ugh at reading it
<abentley> rick_h: was a multi-step process, and I had to check with curtis about the meaning of the original.  Do you think I should try to clean up the storm version more?
<rick_h> abentley: I think it's just more I can read sql ok. So no, I think it's ok
<rick_h> as is
<abentley> rick_h: Okay.  Thanks for the review.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: rick_h | Firefighting: - | Critical bugs: ~180
<rick_h> abentley: I'm looking at trying to update the usage to include the user in the call but hitting a TAL wall: https://pastebin.canonical.com/78762/
<rick_h> any idea or example where the method in the TAL gets a parameter passed in?
<rick_h> I'm bzr grep'ing through but not coming up with an example I can see
<sinzui> rick_h, We don't want python in tal. It is difficult to test
<sinzui> rick_h, add a helper to the view to call the context with the user
<rick_h> sinzui: yea, I'm thinking of punting on the suggest for my branch tbh. It's just going to be a giant pita to make user a param to the changes
<sinzui> rick_h, or ...
 * sinzui looks for cheat code
<rick_h> sinzui: because doing that I hook the portlet to the view
<rick_h> which I guess is ok since it's the one use but that seems dirty as well
<jcsackett> rick_h: is this so you can get user in that method to do the privacy filter?
<sinzui> rick_h, we have several cases where the mode must have a user, but the callsite will not pass it. The code will get the current request to find the user. maybe
<sinzui> request = get_current_browser_request()
<sinzui> user = request.user
<jcsackett> you can also get the user from the launchbag, can't you?
<sinzui> oh, it is person = IPerson(request.principal)
<rick_h> jcsackett: yea, that's what I did but the suggestion was to avoid it
<jcsackett> rick_h: ah.
<rick_h> so trying to figure out the best way to replace the launchbag with direct calls because hte usage is in a portlet .pt
<sinzui> yeah, the launchbag is used too much
<rick_h> I think sinzui is right. The best thing is to add a helper on the view, but then it's ugly because the portlet is making the view api change
<rick_h> abentley: ok, comment added. Updated one of the 3 change points.
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~180
<jcsackett> deryck: are you around, and can you review https://code.launchpad.net/~jcsackett/launchpad/404-project-milestones-privacy/+merge/135252
<deryck> jcsackett, yes and yes :)
<jcsackett> deryck: awesome. thanks!
<deryck> jcsackett, on call now but will look shortly
<jcsackett> deryck: works for me.
<sinzui> wgrant, StevenK: https://pastebin.canonical.com/78751/
<wgrant> http://launchpadlibrarian.net/122750088/qxTB0SY4SJMb6haWptIok9Gyind.txt
<sinzui> StevenK, https://bugs.launchpad.net/launchpad/+bug/665307
<_mup_> Bug #665307: cronscripts/expire-bugtasks.py fails trying to put backtrace in librarian <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/665307 >
<StevenK> webops: The NDT r16286 deployment is done, can you move it to Past Deployments?
<sinzui> wgrant, StevenK: https://bugs.launchpad.net/launchpad/+bug/687446
<_mup_> Bug #687446: process-mail dies on malformed email addresses <email> <lp-foundations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/687446 >
<deryck> jcsackett, r=me
<jcsackett> deryck: Awesome, thanks.
<deryck> jcsackett, np.  sorry it took me so long.  had back to back calls.
<deryck> good night, everyone.
<slank> StevenK: I had trouble finding your ping again ... that NDT is moved in productionstatus
<StevenK> BAH, wrong window after all, too.
<StevenK> slank: Thanks
#launchpad-dev 2012-11-21
<StevenK> Bleh, it doesn't OOPS
<StevenK> It gives a traceback directly
<StevenK> 2012-11-20 02:54:33 INFO    Error updating http://bugzilla.libav.org/: Failed to parse XML description for http://bugzilla.libav.org bugs [u'221', u'222', u'298']: mismatched tag: line 101, column 4
<StevenK> RAGH
<StevenK> wgrant: So I'd expect a cronscript that tosses an exception to print out the OOPS number and exit, which doesn't happen. :-(
<wgrant> StevenK: Well, have you changed the code to make it do that?
<StevenK> I don't know how :-(
<StevenK> I thought I had to add the OopsHandler in scripts.logger, but LaunchpadCronScript adds it anyway
<StevenK> There is a log_unhandled_exceptions_func function, but I'm not sure if that interacts with the handler, or what goes on
<wgrant> Well, find out how the current handler works
<wgrant> That should tell you why the OOPS handler doesn't
<StevenK> wgrant: I'm not sure if OopsHandler is even used by anything that isn't a test
<StevenK> Hmm
<StevenK> logger.exception adds an ERROR, which should get tripped by the OopsHandler to give an oops
<StevenK> wgrant: I've been staring at this too long. Can you glance at http://pastebin.ubuntu.com/1373869/ and see if you can find something I'm missing?
<wgrant> What doesn't work?
<StevenK> The raise Exception in the midst of expire-bugtasks prints out a full traceback
<wgrant> StevenK: OopsHandler in its current state is designed to log an OOPS for any message greater than a warning
<wgrant> You might want a separate one for exceptions
<wgrant> Hm, though it looks like it's meant to handle it
<StevenK> Exactly
<StevenK> Like I say, been staring at it too long
<wgrant> StevenK: Hm, it works fine for me
<wgrant> I get OOPSes out of it properly
<StevenK> wgrant: Oh?
<StevenK> Can I see the output?
<wgrant> There's no special output, because you need a custom formatter for that
<wgrant> But an OOPS is logged
<StevenK> Ah
<StevenK> So I need to grow a OopsFormatter
<wgrant> Probably, yes
<cjwatson> wgrant: Hmm, if bug 817358 is specifically about lots of bugs, shouldn't it have been fixed by the introduction of ProcessAcceptedBugsJob?
<_mup_> Bug #817358: Copying packages with lots of associated bugs can cause timeout <bugs> <package-copies> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/817358 >
<StevenK> wgrant: Maybe we just make LaunchpadFormatter do it
<wgrant> cjwatson: Copies
<wgrant> cjwatson: Not accepts
<cjwatson> (I realise syncSource is always going to be somewhat terrible, but in this specific instance ...)
<cjwatson> wgrant: Yes, so?
<cjwatson> wgrant: The implementation goes through the same bits
<wgrant> Oh, I guess an in-app sync would use the same code...
<wgrant> hmm.
<cjwatson> zactly
<wgrant> Fair point
<wgrant> Fixed
<cjwatson> Ta.  Not that it affects the critical count
<wgrant> Indeed
* wgrant changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~170
<StevenK> Hmmmm, now if the handler has created an oops, how the heck do I get my hands on it
<StevenK> ... Why does the formatter get called first :-(
<wgrant> StevenK: You may have to turn it all into a formatter rather than a handler, possibly
<StevenK> Looking into a filter
<StevenK> Might help
<StevenK> I hope
<StevenK>  5102 steven    21   1 1569m 1.5g 3312 R  89.1 19.3  11:43.05 bzr
<StevenK> Fun
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-librarianformatter/+merge/135311
<wgrant> StevenK: Why do you hate freedom^Wtracebacks?
<wgrant> It would seem prudent to show both the traceback and the OOPS ID
<StevenK> I can remove that bit of LaunchpadFormatter if you wish
<wgrant> That would be nice
<wgrant> Otherwise if the OOPS goes nowhere then we have no idea what went wrong
<StevenK> -LDAP!
<wgrant> ?
<StevenK> Healing: 389 lines of code
<StevenK> wgrant: http://pastebin.ubuntu.com/1374074/
<wgrant> And corresponding test changes, presumably
<wgrant> But yeah, that sounds good
<StevenK> I've tossed it at ec2 test, so we'll see what the fallout is.
<wgrant> I mean you added two tests for the OOPS ID
<wgrant> They are presumably now broken
<StevenK> I did?
<wgrant> Oh
<wgrant> Those were just in the description
<wgrant> Right
<StevenK> Yes, that was a demo
<StevenK> wgrant: Hmm, does my change kill your milliseconds logging?
<wgrant> Remarkable timing
<wgrant> That logging is not mine, but stub's
<wgrant> But you still use LaunchpadFormatter, don't you?
<wgrant> Which should preserve it
<StevenK> wgrant: But it was created differently based on the milliseconds argument
<StevenK> LibrarianFormatter was a subclass of LaunchpadFormatter
<wgrant> Ah
<wgrant> You're right, yeah
<wgrant> You'll need to revert that bit
<stub> LibrarianFormatter should die
<StevenK> stub: I killed it
<stub> Yay. I would have but was scared of dealing with the test fallout
<StevenK> So far only one failing test
<stub> Bug #641103
<_mup_> Bug #641103: LibrarianFormatter should die <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/641103 >
<StevenK> stub: You'll note there is an MP attached ...
<stub> I could have sworn we supported multiple --log-file arguments
 * stub wonders why he didn't
<wgrant> Huh, don't we?
<wgrant> I'd always assumed that just worked
<wgrant> But I guess I've only ever used the default stdout -q/-v options plus a --log-file
<stub> no, just noticed in the MP and had to double check. It would be trivial to support it
<nigelb> bigjools: hahaha, gavinator
<bigjools> :)
<bigjools> the meme commences
<lifeless> oh no
<lifeless> you didn't
<adeuring> good morning
<czajkowski> bigjools: you're a tad evil!
<bigjools> guffaw
<czajkowski> at least this memme isn't filling up planet ubuntu ..yet!
<rick_h> morning party people
<mgz> morning rick_h
<czajkowski> howdy
<jcsackett> morning all.
<czajkowski> ello jcsackett
<adeuring> abentley: https://code.launchpad.net/~adeuring/launchpad/bug-1056881/+merge/135374
<abentley> adeuring: ack
<abentley> adeuring: If a spec is made PROPRIETARY, the AccessArtifactGrant.grantor will be the person who made it proprietary, and so they will be able to unsubscribe anyone.
<adeuring> abentley: yes
<adeuring> abentley: well, provided that they had to grant an AAG
<abentley> adeuring: Right.  They won't be able to unsubscribe anyone who had an APG.
<adeuring> abentley: is anything wrong with this logic?
<abentley> Well, it's not a great rationale for granting the ability to unsubscribe people.  Though of course I understand the limitation in the schema that makes this hard.
<adeuring> abentley: right, but there also this logic: Basically, people with lp.Edit permission can also grant access to other people, and the same permission is needed to change the information type. And revoking grnats need again the same permission
<abentley> adeuring: I'm in the check-in hangout, so I'll get back to this after.
<adeuring> abentley: gah, forgot that completely..
<jcsackett> abentley: are you reviewing today? if so, can you look at https://code.launchpad.net/~jcsackett/launchpad/filter-in-getRequestTargets/+merge/135457
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: abentley | Firefighting: - | Critical bugs: ~170
<jcsackett> abentley: well, that answers my first question. :-P
<abentley> jcsackett: Yes, I'll look at it after I'm done with abel's.
<jcsackett> abentley: awesome, thanks.
<jcsackett> deryck: i was just looking over our cards, and i think there are some comments on bug 1043902 that make a strong case for dropping it. has there already been discussion incorporating that info?
<_mup_> Bug #1043902: magic InformationType numbers in access grant related DB procedures <information-type> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1043902 >
<abentley> adeuring: This solution is an improvement, but ensure a bug is filed about the lack of a subscribed_by field in SpecificationSubscription, and reference it in the security adapter and bug comments.
<abentley> s/bug comments/test comments/
<adeuring> abentley: ok
<abentley> adeuring: You can address the too-long enum names by assigning the enum to a local variable.  That is what I do.
<adeuring> abentley: I know...
<abentley> adeuring: Other than that, r=me.
<adeuring> abentley: thanks
<deryck> jcsackett, yes, I think that can be dropped, but would love to hear for sure from adeuring. Just in case there's some bit of what he's worried about that I don't understand.
<jcsackett> deryck: fair.
<deryck> adeuring, can we drop bug 1043902?
<_mup_> Bug #1043902: magic InformationType numbers in access grant related DB procedures <information-type> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1043902 >
<adeuring> jcsackett, deryck: Its not that important for me. My main reason for writing this bug was that I like to have a kind of additional check for things secuirty related.. FOr xample, if the meaning of an integer value is changed in Python code but not in stored procedures, this can have security implications. OTOH, LP devs tend not to be insane enough to this ;)
<adeuring> so, its fine for me to drop this card
<abentley> jcsackett: in  getRequestTargets, having the user default to none means that calling code can accidentall omit it.  I think it should be a non-optional parameter, even though this means changing parameter order.
<jcsackett> abentley: there's a small number of call sites, so that works for me. same comlaint for list_product_request_filters?
<abentley> jcsackett: Same complaint for list_product_request_targets
<jcsackett> right, that's the one.
<jcsackett> small set of sites there, so no worries.
<abentley> Aside from that, this looks good.  r=me.
<jcsackett> thanks. i'll make those changes.
<deryck> adeuring, ok, thanks.  sorry was on call.
<deryck> jcsackett, do you mind dropping both the bug and the card then?
<jcsackett> deryck: on it.
<deryck> thanks!
<rick_h> abentley: review if you have the time. https://code.launchpad.net/~rharding/launchpad/translatables/+merge/135492 after talking with deryck went with the less impact approach
<abentley> rick_h: Hmm.  Perhaps we don't need to do this, since AIUI products cannot be proprietary if they have translation enabled.
<abentley> rick_h: That's not actually implemented yet, but https://bugs.launchpad.net/launchpad/+bug/1079785/comments/3
<_mup_> Bug #1079785: public artifacts are permitted on confidential projects <private-projects> <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/1079785 >
<rick_h> abentley: ah, gotcha
<rick_h> completely didn't think of that part
<rick_h> abentley: so I read that as we'd like to get to deleting things that would fall into here, but we don't currently so I'm going to go with let's make sure we don't leak for the moment?
<abentley> rick_h: I'm not thrilled with that approach, because we're not guaranteeing privacy now, and it makes more work for later when we have to disable the test and remove the code.
<rick_h> abentley: I guess if we have cards for handling the delete/clean up and the whole transitino process I'd be ok with burying this under that work.
<rick_h> but I don't want to leave a hole out there without something on the board that says it won't be a hole
<abentley> rick_h: Sure, let's do that.
<rick_h> abentley: so will yuor current branch then do all that is mentioned in that comment?
<abentley> rick_h: Probably.  I can certainly commit to doing the translation bit.
<rick_h> Ok, will put the branch in WIP and move the card back into the final with a note on it that it can be wiped once that's true
<rick_h> and I suppose the other branch I landed yesterday can probably backed as well then
<abentley> rick_h: Right.  Sorry I didn't catch it then.
<rick_h> oh well, I'll move that card back into final and alter it to a back-out work then.
<abentley> rick_h: I've added a misc card for fixing any existing data.
<rick_h> ok
<jcsackett> abentley: can you review https://code.launchpad.net/~jcsackett/launchpad/blueprints-in-ui-not-specification/+merge/135503
<abentley> jcsackett: Sure, but give me a few minutes.
<jcsackett> abentley: no rush.
<abentley> jcsackett: r=me
<jcsackett> abentley: thanks!
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~170
<sinzui> StevenK, wgrant, I am in a meeting. I may be 30 minutes late.
#launchpad-dev 2012-11-22
<bigjools> the text formatter is broken then, it linkifies bug\nNNN
<wgrant> bigjools: It deliberately does that, to cope with wordwrapped emails
<bigjools> see my text in bug 1081834
<_mup_> Bug #1081834: When setting a milestone on a bug task, the edit icon ends up as the "broken graphic" <Launchpad itself:New> < https://launchpad.net/bugs/1081834 >
<bigjools> I say tough titty if it's word wrapped
<bigjools> coz now you have a broken link
<wgrant> It's probably less bad to have that extra link than it is to miss some
<bigjools> I disagree
<wgrant> Why?
<wgrant> You can just not click on that link :)
<bigjools> tell that to oops reports
<bigjools> you can just type a bug number too
<wgrant> But I can just as easily generate a bad link malevolently by typing "bug 2" in a comment
<wgrant> Those sorts of things should be (but aren't presently) excluded from OOPS reports
<bigjools> why is it even linked if it doesn't exist?
<wgrant> Because user error can cause them
<bigjools> it's ab ug
<bigjools> should not be linked
<wgrant> It's turned grey by JS if it doesn't exist.
<bigjools> a start, I suppose
<wgrant> Users can type bad URLs
<wgrant> They can type bad bug numbers too
<bigjools> well it doesn't go anywhere, I thought it was a real link, so it's opk
<wgrant> It is a real link until the JS detects that it doesn't actually exist
<wgrant> Which isn't ideal, but I think it's a reasonable compromise
<bigjools> so robots will still click it then
<wgrant> Yes
<bigjools> I am sure there was code in the formatter to not linkify non-existent bugs
<wgrant> There was, but we removed it
<wgrant> Because it was completely terrible
<bigjools> let me guess ...
<wgrant> Making a separate query for each bug reference
<bigjools> yeah
<wgrant> And it's difficult to do otherwise without a multi-pass template system, which TAL is not
<bigjools> I remember it being a problem on changelogs
<wgrant> Anyway
<wgrant> "bug 2" is just a special type of URL
<wgrant> Users can type bad URLs
<bigjools> and when I wanted to remove it to fix performance, all of Ubuntu screamed like a thousand voices in the night
<wgrant> We still linkify them
<wgrant> Why not do the same with bugs?
<wgrant> It's not harming anyone
<bigjools> ok, you're on maintenance :)
<wgrant> 404s are a discretionary sort of OOPS, since users will always be able to generate them simply by typing a bad URL somewhere on LP
 * StevenK stabs gina.txt and twists the knife.
<wgrant> They don't unconditionally fall under ZOP
<bigjools> ok
<wgrant> And they don't degrade user experience
<wgrant> => meh
<bigjools> well at least you thought about it
<wgrant> The linkification of bullets isn't ideal, but there's not much we can do about it without breaking other cases
<wgrant> Of course, if you used punctuation properly it wouldn't be a problem :P
<bigjools> => meh
<bigjools> DWIM etc etc
<wgrant> Heh
<StevenK> wgrant: Hmmm, what about bug 837563?
<_mup_> Bug #837563: language-pack-exporter script raised a KeyError <lp-translations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/837563 >
<StevenK> I thought OOPS prefixes were long dead
<wgrant> StevenK: Sure, the REPORTIFSEEN part of that bug is no longer relevant
<wgrant> The underlying OOPS is
<bigjools> hmmm, why would ^\s not work in my sed expression
<StevenK> \s as in 'any whitespace' ?
<bigjools> yup
<bigjools> ^\s should be leading whitespace
<bigjools> yet it matcheth not
<lifeless> \\s ?
<bigjools> nope
<bigjools> removing the ^ matches but I want to be more specific
<cjwatson> [[:space:]]
<cjwatson> \s is a perl-regex-ism
<bigjools> it's in the sed man page :)
<cjwatson> not in the one I have here it's not
<cjwatson> anyway, [[:space:]] is the POSIX form
<bigjools> ^[[:space:]] didn't work
<cjwatson> full example please
<bigjools> sed -ri "s|^[[:space:]]generator:\shttp://.*:5243/|  http://bar:5243/|" test.tmp
<cjwatson> you still have a \s there
<cjwatson> and I think perhaps you mean [[:space:]]* for zero or more whitespace characters?
<bigjools> makes no difference if I change it
<cjwatson> [[:space:]] (or \s in regex engines that support that) means exactly one whitespace character
<bigjools> I want a + actually, I'll change it
<bigjools> \o/
<bigjools> thanks cjwatson
<cjwatson> yw
<bigjools> I really hate regex
<StevenK> bigjools: You now have two problems.
<bigjools> exactly
<StevenK> Or three, if you're also dealing with mod_rewrite
<bigjools> heh
<StevenK> wgrant: Can't find OOPS-8e259d8c3b45707c6a37f986d3c8127d on carob's filesystem. :-(
<StevenK> wgrant: I wonder if we can just destroy the branch puller
<wgrant> mwhudson had plans :)
<lifeless> and doesn't now ?
<mwhudson> i just don't have time now
<mwhudson> wgrant: actually i futzed a little on the flight to linaro connect about this and wanted to talk to someone (probably you!) about this
<mwhudson> wgrant: my plan went something like this:
<mwhudson> set up a launchpad user on each importd
<mwhudson> have a fkey from codeimportmachine to this user
<mwhudson> change _canWrite to something like this:
<mwhudson> +        elif branch.branch_type == BranchType.IMPORTED:
<mwhudson> +            job = branch.code_import.import_job
<mwhudson> +            if job.machine:
<mwhudson> +                return job.machine.user == requester
<mwhudson> +            else:
<mwhudson> +                return False
<lifeless> mwhudson: so did you decide to stay w/Linaro or move back into Canonical ?
<wgrant> mwhudson: That's probably the cheapest way to do it, but ideally we'd also solve stacking at the same time
<mwhudson> change things around so that a lp:// url rather than a escudero url gets passed to the worker
<mwhudson> wgrant: "solve stacking"?
<wgrant> mwhudson: Stacking imports would sometimes be nice.
<mwhudson> oh right
<wgrant> Which means access to more than just that branch
<wgrant> Which gets messy
<mwhudson> well
<mwhudson> imports would end up stacked on the project trunk in this case
<mwhudson> but they would get imported from scratch first
<mwhudson> which would not be so ideal
<wgrant> Well
<wgrant> They might end up using /+branch-id, which wouldn't stack
<mwhudson> oh true
<mwhudson> and indeed, that would be easiest
<wgrant> And safest
<mwhudson> so the fix is to make pushing to +branch-id stack?
<wgrant> Maybe
<wgrant> Or just ignore stacking
<mwhudson> i don't think attempting to solve stacking at the same time seems especially sensible
<wgrant> Indeed, perhaps not
<wgrant> But it should be kept in mind as a future consideration, perhaps
<mwhudson> once imports don't need the puller, converting mirrored to imported branches should be trivial
<mwhudson> and then one can rm -rf lib/lp/codehosting/puller
<mwhudson> which would get you lots of LOC credit :-)
<wgrant> Yep
<wgrant> That's why I haven't converted mirrors yet
<wgrant> Because of the escudero mess
<mwhudson> right
<mwhudson> someone should delete all the codeimportevent mess too :(
<mwhudson> that was mostly ddaa really but i feel bad for letting it happen
<wgrant> What's the mess there?
<wgrant> I admit I don't know those corners of the model well
<mwhudson> well, the whole concept is stupid
<mwhudson> not stupid
<mwhudson> but should have been YAGNIed before implementation
<wgrant> Oh
<wgrant> That's distinct from codeimportresult... I... seee
<mwhudson> yeah
<lifeless> thats hardly unique in LP
<wgrant> Ah
<wgrant> So CodeImportEvent is an audit log, basically
<mwhudson> yes
<mwhudson> that you mostly can't read
<lifeless> auditor being the now extant answer for that
<wgrant> Well
<wgrant> Except it isn't extant yet :)
<lifeless> ENODEPLOY ?
<mwhudson> and that means that standard methods are not used to mutate them, which means that e.g. field validators don't run
<wgrant> Yeah
<mwhudson> wgrant: https://code.launchpad.net/~mwhudson/launchpad has a few branches at the top if that's useful at all
<mwhudson> i can't really remember how far i got, i was looking at changing the worker invocation code when i got discouraged i think
<wgrant> Hmm, have you made a terrible mistake...
<wgrant> I guess we'll see
<wgrant> (pushing a series of branches which will each want to create the same new revisions, and they'll be scanned concurrently now...)
<mwhudson> oh
<mwhudson> heh
<mwhudson> they certainly don't seem to be being scanned
<wgrant> It takes a 80s at best to do a fresh scan of an LP branch nowadays
<wgrant> Often a few minutes
<wgrant> Yeah, they all OOPSed
<mwhudson> sigh
<wgrant> The branch scanner does not win any points for considering locks or using sensible transaction sizes :(
<StevenK> lifeless: We don't have the staging instance deployed yet, let alone think about production.
<StevenK> wgrant: http://pastebin.ubuntu.com/1376306/ but I'm not sure how to test it
<wgrant> Um no
<wgrant> Transaction management does not go in DB model classes :)
<StevenK> Wrap the generateIncrementalDiff in BMPJ run() ?
<StevenK> wgrant: http://pastebin.ubuntu.com/1376309/
<StevenK> That I can test, if I can patch out generateIncrementalDiff()
<wgrant> Well
<wgrant> We in general need a strategy for handling conflicts in jobs
<wgrant> eg. what mwhudson just ran into
<wgrant> Ignoring all IntegrityErrors is probably not it :)
<StevenK> wgrant: Hmmm, it looks like cronscripts/rosetta-export-queue.py already uses the slave DB and based on the logs it isn't getting reaped. bug 504821
<_mup_> Bug #504821: poimport (export)_uses a single long transaction, gets reaped <lp-translations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/504821 >
<wgrant> It's still vile
<wgrant> But indeed, I haven't seen reaping in quite some time
<StevenK> wgrant: Downgrade or close?
<wgrant> Invalid until we have reason to believe otherwise
<wgrant> It's still something we need to adjust eventually, but it's not causing any trouble today
<wgrant> And it's not actively a problem until we ratchet our paranoia up another notch
<wgrant> StevenK: In what circumstances does one not have an id?
<StevenK> I'm guessing when we get a error page or something like that
<StevenK> There are a whole stack of oopses on neem
<wgrant> Right, they certainly happen
<wgrant> But we should work out why
<wgrant> Blindly bypassing an error condition without understanding it is not the way to fix criticals, sadly
<wgrant> (unless we have no other choice)
<StevenK> I just noticed the OOPSes while looking if there was a langpack export OOPS
<StevenK> Checking if they're all the same bugtracker
<wgrant> Sure
<wgrant> It's been happening for a while
<wgrant> And it's likely to be a simple fix
<wgrant> But we need to understand why
<lifeless> using the slave db is now pointless
<lifeless> as transaction bloat is propogated back to the master
<wgrant> lifeless: Pointless for bloat
<wgrant> Not pointless for load
<lifeless> not terribly interesting either ;>
<wgrant> (not that load is an issue on any system today)
<wgrant> right
<StevenK> Oh, hah
<StevenK> mercurial has replaced their roundup instance with bugzilla and are redirecting
<wgrant> Ah
<StevenK> wgrant: Right, there's mercurial fixed, two link farm bugtrackers deleted and I yet to figure out what setuptools is doing
<wgrant> :)
<adeuring> good morning
<czajkowski> aloha
<czajkowski> cjwatson: ello how do you want this triaged, https://bugs.launchpad.net/launchpad/+bug/1081860
<_mup_> Bug #1081860: uploads to oneiric/partner are redirected to oneiric-proposed/partner <Launchpad itself:New> < https://launchpad.net/bugs/1081860 >
<cjwatson> czajkowski: I stole it for ubuntu-archive-tools
<cjwatson> So you won't have to care :)
<czajkowski> oki dokie :)
<czajkowski> thanks colin
<czajkowski> cjwatson: me again :) can you moderate a mail if you get a chance later on to devel
<czajkowski> really find it hard to type devel without going devil :/
<cjwatson> czajkowski: done
<czajkowski> cjwatson: thank you
<jml> is there a secret way of only retrieving certain keys when GETting Launchpad API resources?
<jml> some code_review_comments are really, really big.
<jml> further, the size of the download is 2x the size of the comment, because as_quoted_email is included in the dict
<jml> also, they both get stored in memory as unicode objects, which take up 4x as many bytes as the string has characters
<jml> bringing us up to 8x
<jml> then, if you are getting it as part of a collection, the naive way of using it (looping through), will give you a copy of the data in the collection _and_ a copy in the entry object
<jml> taking us up to 16x
<jml> for this use case, I don't care about the comment body at all. All I want is the dates.
<czajkowski> jml: most of the devs are off today as USA based
<jml> czajkowski: so I should try to avoid having hard problems today and tomorrow?
<czajkowski> jml: most are back tomorrow tbh, or leave a message for wgrant he'll be back online shortlyish
<czajkowski> jml: adeuring will be back jtv bigjools, or gmb
<wgrant> jml: Well, you can retrieve individual attributes directly, but you can't retrieve a subset of elements that isn't either the whole lot or a single one.
<wgrant> lazr.restful unfortunately provides no more advanced facility
<wgrant> jml: You should upgrade to Python 3.3 :)
<abentley> jml: Are these really, really big code_review_comments full of test results?
<james_w`> abentley, yes
#launchpad-dev 2012-11-23
<adeuring> good morning
<jml> wgrant: why should I upgrade to Python 3.3?
<lifeless> jml: new unicode representation
<czajkowski> lifeless: \o/
<jml> ah.
<lifeless> czajkowski: o/
<wgrant> jml: yeah, what lifeless said
<jml> hm
<jml> what would be better is being able to express my query without O(N) roundtrips.
<jml> *sigh*
<wgrant> yeah :/
<willcooke> hi czajkowski
<czajkowski> willcooke: hiya adeuring will be able to help
<willcooke> hi adeuring
<willcooke> adeuring: I'm getting this:
<willcooke> Write failed: Broken pipe
<willcooke> ConnectionReset reading response for 'BzrDir.open_2.1', retrying
<willcooke> Write failed: Broken pipe
<willcooke> bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
<willcooke> I've checked my SSL keys
<adeuring> I am not a bazzar expert... abentley,does this ^^^ ring a bell?
<abentley> adeuring, willcooke: This sounds like the connection is being dropped prematurely.
<abentley> I had a few connections dropped recently, but it's working fine for me right now.
<willcooke> abentley: so you think network problem this end?
<abentley> willcooke: Yes, that's most likely.  If we start getting this issue with more people, then it would be more likely on our end.
<abentley> willcooke: What's the command you're issuing?
<willcooke> abentley: kk - thx.  I'll try from a server somewhere and see what happens.
<willcooke> cheers all
<abentley> adeuring: Could you please review https://code.launchpad.net/~abentley/launchpad/transitive-confidential/+merge/135926 ?
<adeuring> abentley: sure
<adeuring> abentley: r=me, with some concerns about possibly bad user experience
<abentley> adeuring: That is a fair point.  I can update it so that there is an actual form validation for information type, instead of handling CannotChangeInformationType exceptions.  Can it do it in the follow-on branch?
<adeuring> abentley: sure, I did not expect that you wnated to do another round before landing the branch ;)
<abentley> adeuring: Exactly :-)
<abentley> adeuring: I'm going to try that idea for validation that I had, where you have a validation generator that yields exceptions.  The form validation will list the exceptions, but the storm validator will raise the first exception it encounters (if any).
<adeuring> abentley: nice idea!
<abentley> adeuring: Thanks.
<apw> if i have a bug which is no longer spawning new nominations (it is suspected someone has deleted on in the past), is that bug forever hosed or can they be fixed
<czajkowski> allenap: ^^^
<mgz> gavin is soo weekended already
<mgz> and the US is all off
<mgz> apw: post the problem to the launchpad-users mailing list so it doesn't get forgotten by monday
#launchpad-dev 2012-11-25
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/re-enable-workermonitor/+merge/136072
<wgrant> StevenK: Why?
<StevenK> mrs_poolie: \o/
<StevenK> wgrant: To close a critical and have less disabled tests?
<wgrant> StevenK: I mean, what's the justification for just increasing the timeout?
<wgrant> Why should these things take a minute?
<wgrant> (it's a trick question; no test can ever be allowed to run for a minute)
<mrs_poolie> hi StevenK :)
<StevenK> wgrant: Well, I'm not sold on 60, I was just seeing if increasing the timeout would stop the tests failing -- given 20 seems bad, I wasn't sure about 30.
<wgrant> Ah, they run subprocesses, I see
<StevenK> And it's a timeout, it isn't supposed to take that long, that's when we give up and fail the test, no?
<wgrant> Your __all__ is missing a comma
<wgrant> Otherwise good
<StevenK> Ah, so it is
<StevenK> wgrant: Can haz approve?
<wgrant> Done
<StevenK> Hmm, kanban actually logged me out
<bigjools> mrs_poolie: nice nick!
<mrs_poolie> hi bigjools
<bigjools> mrs_poolie: fyi, the Ghiardelli chocolate did *not* last long
<mrs_poolie> bigjools: lol. we still have lots here. going to give some away
#launchpad-dev 2013-11-18
<bigjools> wgrant or StevenK, can you decipher the error at the bottom of this please? https://launchpadlibrarian.net/156805473/buildlog.txt.gz
<StevenK> bigjools: You can't have -'s in your version with 3.0 (native)
<bigjools> StevenK: this has previously built on trusty, no changes made
<bigjools> was this a recent toolchain change?
<StevenK> Probably newer dpkg
<bigjools> StevenK: so I need to remove all the - chars?
<StevenK>       dpkg | 1.16.12ubuntu1 |         saucy | source, amd64, arm64, armhf, i386, powerpc
<StevenK>       dpkg | 1.17.1ubuntu1 |        trusty | source, amd64, arm64, armhf, i386, powerpc
<StevenK> (The error turns up in 1.17)
<StevenK> Right, and 1.17.1ubuntu1 was uploaded 7 hours ago.
<StevenK> bigjools: You have two choices. Change to 3.0 (quilt) and leave the version number alone; or leave it as 3.0 (native) and use periods as seperators
<bigjools> StevenK: ok thanks
<bigjools> StevenK: huh I thought it was already quilt
<bigjools> yeah it is
<StevenK> Based on that error, debian/source/format is 3.0 (native), no?
<bigjools> debian/source/format:3.0 (quilt)
<bigjools> in my packaging
<bigjools> also look up the log file and see it applying quilt patches ...
<StevenK> bigjools: Have you tried it in an up-to-date trusty chroot?
<bigjools> no, I only added the recipe late last week and it worked the first time I built with it
<StevenK> Yes, the new version of dpkg was only uploaded 7 hours ago.
<bigjools> this log us from an hour ago or so
<bigjools> is
<StevenK> I think we're talking past each other. It worked last week, because dpkg was still the version shipped with saucy. The new dpkg in trusty was uploaded 7 hours ago, so would certainly impact the build you did an hour ago.
<bigjools> StevenK: I understand.
<bigjools> dpkg is complaining it's a native pacakge and it definitely is not, so perhaps dpkg is buggy?
<StevenK> I'm trying to update my local trusty chroot
<bigjools> could fire up a canonistack instance
<StevenK> bigjools: Point me at your recipe?
<bigjools> yup, one sec
<bigjools> StevenK: https://code.launchpad.net/~maas-maintainers/+recipe/maas-daily-trusty
<wgrant> bigjools: bzr-builder will in some scenarios convert quilt to native. possibly if there's no pristine-tar, but i don't quite recall
<bigjools> wgrant: I don't think bzr-builder changed between today and Friday did it?
<wgrant> no, the change is purely dpkg
<wgrant> but it would explain why your quilt package has been native before and after
<bigjools> right
<bigjools> can we make it stop doing that?
<wgrant> maybe
<wgrant> on my phone atm, can't really poke right now
<StevenK> % cat -A debian/control | head -n 21 | tail -n 1
<StevenK>  (M-bM-^@M-^\nodeM-bM-^@M-^]) will be commissioned automatically on first boot.$
<StevenK> Smart quotes in debian/control? You bad people.
<wgrant> it's 2013
<wgrant> Unicode is a thing :)
<bigjools> StevenK: welcome to reality :)
<StevenK> Hah
<StevenK> bigjools: So, yes. wgrant is correct. bzr-builder will convert from 3.0 (quilt) to 3.0 (native) if you don't have pristine-tar information.
<bigjools> yay?
<bigjools> this is stretching my packaging knowledge now.
<StevenK> It isn't packaging, it's bzr innards
<wgrant> bzr-build{er,deb}, not bzr :)
<StevenK> I've tried to ignore pristine-tar, not sure if you're supposed to use it directly.
<wgrant> bigjools: So, for 3.0 (quilt) to work you need an orig.tar.*. bzr-builddeb can't generate one of those without pristine-tar information in the branch, which specifies how to turn a revision's tree into a bit-identical upstream orig tarball.
<wgrant> It sounds like your branch doesn't have a pristine-tar delta for the upstream version that you've specified.
<bigjools> hmmm
<bigjools> I thought it generated a tarball in the makefile somewhere
<bigjools> but this is all roaksoax work, so...
<bigjools> so there's a get-orig-source target in debian/rules, is this not sufficient ?
<StevenK> No, pristine-tar information is stored in the bzr history
<StevenK> If you have a tarball, pristine-tar commit <path to tarball> will add a pristine-tar delta to the branch
<bigjools> is that something we need in the get-orig-tarball target?
<bigjools> get-orig-source I mean
<StevenK> bigjools: pristine-tar wants to reproduce an exact copy of a tarball, so no, not in get-orig-source
<bigjools> StevenK: ok sorry then I am clueless, I'd better leave this for now.
<StevenK> bigjools: If you like, we could have a G+ hangout after lunch and I can go through it
<bigjools> StevenK: thanks for the offer - I'll see if I have time, busy with stuff right now
<StevenK> I'm about to disappear for lunch
<bigjools> yeah I agree
#launchpad-dev 2013-11-19
<StevenK> wgrant: So we have this thing, utilities/massage-bug-import-xml, which hardcodes python2.6. It is untested, and only mentioned by our bug import docs. Shall I just move it to lp-dev-utils, or fix it to not hardcode?
<wgrant> StevenK: I'd just fix it tno hardcode for now.
<lifeless> wgrant:  / StevenK: so the bug about python tracker imports
<lifeless> I don't understand why we'd treat a terminal state in the upstream tracker as non-terminal in Launchpad
<wgrant> lifeless: I thought we had agreement that it should be Fix Released
<lifeless> wgrant: yes, but I thought I saw barry commenting something against that
<lifeless> so cool
<StevenK> Yeah, I asked barry
#launchpad-dev 2013-11-20
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/antibfjo-1-set/+merge/195886 https://code.launchpad.net/~wgrant/launchpad/antibfjo-2-garbo/+merge/195887 https://code.launchpad.net/~wgrant/launchpad/antibfjo-2.5-no-garbo/+merge/195888
<StevenK> wgrant: I'm just about to head off for ~15, I'll cast my eyes over them when I return.
<wgrant> kk
<StevenK> wgrant: I'm not sure whether to ostracise or congratulate you for use of a local variable 'bbq'
<maxb> lol
<StevenK> wgrant: r=me * 3, -1-set has a comment
<wgrant> StevenK: Thanks
<wgrant> That comment isn't actually mine; it's copied from the Job.status equivalent. But it means "this %s is going to be replaced with self.id"
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/antibfjo-3-branch-deletion/+merge/195890 https://code.launchpad.net/~wgrant/launchpad/antibfjo-4-use/+merge/195891 https://code.launchpad.net/~wgrant/launchpad/antibfjo-5-use-more/+merge/195892
<wgrant> And https://code.launchpad.net/~wgrant/launchpad/antibfjo-7-no-set-pls/+merge/195894
<StevenK> wgrant: What about asserting that the TTB no longer exists after the destroySelf() call?
<wgrant> StevenK: The FK proves that it's gone.
<StevenK> wgrant: In -4-use, you remove SPRB.buildqueue_record, but then use it?
<wgrant> StevenK: I pushed it up to BuildFarmJobMixin
<wgrant> As it's now common
<StevenK> Ah, right.
<StevenK> wgrant: -3-branch-deletion and -4-use are approved, moving onto -5-use-more
<StevenK> I guess -6-blah is a DB patch?
<wgrant> -0, -6, -8, -10 are DB patches
<StevenK> When does BPJ die a horrible death?
<wgrant> -10 drops the tables and columns
<StevenK> Is -10 up yet?
<wgrant> No.
<StevenK> Diff against target: 817 lines (+95/-164) 22 files modified
<StevenK> You are a terrible person.
<wgrant> Ah
<wgrant> I split it, but I guess it crept back over
<StevenK> I have fear that the query changes are in one hunk, and the sqlvalues() changes are in the next hunk
<StevenK> Argh, -7-no-set is even worse
<wgrant> Mostly deletions, though
<StevenK> Right
<StevenK> job_type will die horribly, too?
<wgrant> Yes
<wgrant> But it's still needed in -7 so we know how to look up the BFJO to destroy it.
<StevenK> That's all of them approved
<StevenK> One niggle in -6
<StevenK> Er, -7
<wgrant> I will decline to address it; I feel the unwrapping under the same name at the very start of the method is justified.
<wgrant> But thanks
#launchpad-dev 2013-11-21
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/antibfjo-8-kill/+merge/196053
<StevenK> I don't think you've deleted quite enough. :-P
<StevenK> wgrant: r=me
<wgrant> thanks
<cjwatson> StevenK: Thanks for the testfix.  I ran out of time yesterday.
<StevenK> cjwatson: No worries. I couldn't work out which change broke it, but looking at what was returned with and without your rev made it obvious it was a bong test
<cjwatson> StevenK: *nod*
<cjwatson> Now I wonder why qas hasn't picked it up
<cjwatson> Oh, asuka buggered?
<wgrant> and then some
 * cjwatson runs into bug 612287
<_mup_> Bug #612287: Unable to view failed imports if there are private imports in the list <403> <branches> <code-import> <lp-code> <privacy> <Launchpad itself:Triaged> <https://launchpad.net/bugs/612287>
<cjwatson> That seems to have got sidetracked into whether specific branches should or shouldn't be private, but isn't the right resolution for that bug to show them redacted in +code-imports or skip them entirely?
<wgrant> You can usually mangle batches manually to get everything except the private one.
<wgrant> Probably.
<cjwatson> Well, yeah, but it's silly to have to
<cjwatson> Reminds me of the now-fixed /builders thing.
<cjwatson> Is there a pattern somewhere for filtering batched DB queries by visibility?
<StevenK> Most of the sharing stuff will do it
<wgrant> cjwatson: BranchCollection can do it, but I'm not sure if +code-imports uses BranchCollection.
<cjwatson> Thanks, will ponder
<wgrant> cjwatson: get_branch_privacy_filter is the guts of the query.
<cjwatson> Yeah, might need to use that directly
<wgrant> stub: Could you please review https://code.launchpad.net/~wgrant/launchpad/antibfjo-0-db/+merge/195885 and https://code.launchpad.net/~wgrant/launchpad/antibfjo-6-db-prekill/+merge/195893?
 * stub has a look
<stub> wgrant: that is all fine.
<wgrant> stub: Thanks
<wgrant> Oh amazing
<wgrant> the JS test failure on precise goes away if you show the browser window.
#launchpad-dev 2013-11-24
<lifeless> wgrant: OOPS-7f87343f612eb6d5a7b6e5b79fc5d35c
<lifeless> updating one task on a three-task bug.
<lifeless> finally stabbed it hard enough apparently
<wgrant> lifeless: Let me guess, OpenStack.
<wgrant> With 9 bazillion structsubs
<lifeless> wgrant: I guess
#launchpad-dev 2014-11-20
<cjwatson> wgrant: Can you see why staging doesn't appear to have applied patch-2209-60-0?  It's been several hours now since it landed on launchpad/db-devel.
<cjwatson> Nothing obvious in sourcherry/dbupgrade.log.
<wgrant> cjwatson: I think it's stuck.
<wgrant> cjwatson: See 2014-11-20-staging_restore.log. It started updating, but has done nothing.
<wgrant> It's the first patch since the DB restore, which may be relevant.
<wgrant> It normally sits at "Local Staging Code updated" for like 10-30 minutes, not several hours.
<cjwatson> The last one timed out trying to talk to bzr just after that
<cjwatson> sourcherry's not in LS so can't see its process list
 * cjwatson asks webops, but has to run soon
#launchpad-dev 2015-11-17
<sil2100> Hello everyone!
<sil2100> I seem to have a strange problem with launchpadlib
<sil2100> Not sure what could be the reason...
<sil2100> My script querries two archives constantly, one PPA and one ubuntu archive - everything looks fine up to a point where suddenly when doing getLatestSourcePublication() I get an error in /usr/lib/python3/dist-packages/launchpadlib/launchpad.py
<sil2100> Sorry for the paste:
<sil2100> File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 125, in _bad_oauth_token
<sil2100>     (content.startswith("Expired token")
<sil2100> TypeError: startswith first arg must be bytes or a tuple of bytes, not str
<sil2100> I do a lot of getLatestSourcePublication(), getPublishedBinaries() and such
<cjwatson> Can you pastebin your entire script?  And are you running it interactively on your normal system, or is it running in cron on a server somewhere?
<cjwatson> Oh, and also this is probably https://bugs.launchpad.net/bugs/1471894
<mup> Bug #1471894: _bad_oauth_token crashes on python3 (str vs bytes) <launchpadlib :New> <https://launchpad.net/bugs/1471894>
<sil2100> It's running on my machine, let me pastebinit
<sil2100> Oh, looks promising
<cjwatson> Either drop back to Python 2 or apply the patch in the MP attached to that bug
<sil2100> cjwatson: thanks!
<sil2100> Let me try that
<sil2100> cjwatson: could we get that SRUed to earlier releases?
<sil2100> Or maybe it did, maybe it's just not in vivid
<cjwatson> It hasn't been released upstream yet ...
<cjwatson> I can at least attempt to sort *that* bit out
<sil2100> Anyway, thanks a lot, I didn't know if it was something broken on my script or elsewhere, as I never had this error before
<sil2100> Even though I'm always using python3
<sil2100> cjwatson: ok, now I can see the real error... so when trying to do this:
<sil2100> b = archive.getPublishedBinaries(binary_name='ca-certificates', version='20141019ubuntu0.15.04.1', exact_match=True)
<sil2100> b[0].build.getLatestSourcePublication()
<sil2100> I get a 401 - unauthorized
<sil2100> I just want to get the source package of a published binary - how can I be unauthorized?
<cjwatson> sil2100: so, that's a bit awkward - the build was actually off in the security PPA, and you're unauthorized because you can't see the source package publishing history *there*
<cjwatson> sil2100: can you file a bug about this?  there are a couple of different plausible solutions and I want to consult with wgrant when he reappears
<sil2100> cjwatson: sure, will do, in the meantime I'll probably just skip this package once that happens, not much I can do right now ;)
<cjwatson> Yeah, I can't think of an easy satisfactory workaround there
<sil2100> Anyway, thanks for clearing this up
#launchpad-dev 2015-11-18
<cjwatson> wgrant: Thanks for approving https://code.launchpad.net/~cjwatson/launchpad/twisted-13.0.0-p2/+merge/277642 .  Are you happy with its dependency https://code.launchpad.net/~cjwatson/lazr.sshserver/moduli/+merge/277641 as well?
<wgrant> cjwatson: Indeed.
<cjwatson> thanks
<blr> lifeless: would it make sense to declare all deps in requirements.txt with '>=', install from our local sdist repo, and then freeze those requirements to constraints.txt?
<lifeless> blr: sure, thats roughly what generate-constraints in openstack does
<blr> lifeless: yep, that seems sensible, and will allow us to play nicely with RTD etc.
<blr> thanks
#launchpad-dev 2015-11-19
<maozhou> There is a AssertionError after running queue script, how to fix it?
<maozhou> http://i12.tietuku.com/0939c6e2d1ad6d09.png
<wgrant> maozhou: It's possible that the versions of launchpadlib and keyring that you have installed are incompatible, or your keyring file is corrupted.
<wgrant> You're using the system python-keyring, but your own version of python-launchpadlib.
<maozhou> wgrant: How to fix it?
<wgrant> maozhou: I don't know, it's not a problem I've seen before. You're running a mixed set of versions, so you'll have to debug it.
<maozhou> wgrant:  I find the keyring file is corrupted, and I have fixed it, thank you :)
<bjf> is it possible to get a version of python3-launchpadlib for trusty ?
<blr> cjwatson: who would we ask about a potential backport?
<blr> dimitri presumably?
<cjwatson> bjf: I'm sure it wouldn't be hard to produce one yourself, but there appears to be one in https://launchpad.net/~canonical-foundations/+archive/ubuntu/ci-train-backports
<cjwatson> blr: ^-
<blr> handy
#launchpad-dev 2015-11-20
<cjwatson> wgrant: Do you know if anyone's ever looked at making it easier to rename properties in Storm?  I'm considering writing a thing where you could say e.g. class Person: displayname = RenamedProperty("display_name"), with a __get__ method such that both Person.displayname and Person.display_name would DTRT in Storm queries
<cjwatson> wgrant: It ought to be possible to give it a .setter like the one on the stdlib's property(), so that we could have a renamed property with a custom setter - useful for the Person.name work
<cjwatson> Though in the latter case we kind of don't really want to rename it in an ideal world, it would be better to simply be able to set custom setters on storm properties
<cjwatson> wgrant: Actually, I suppose you can effectively do custom setters (assuming that they end up storing a value at all) with a validator.  So do we actually need to do the Person.name renaming that you were talking about?
<wgrant> cjwatson: Hm, did I suggest renaming Person.name?
<wgrant> cjwatson: It needs to be significantly reworked due to the fact that it's impossible in some cases and involves API calls in others, but it doesn't obviously need to be renamed.
<cjwatson> wgrant: Not renaming as such, but you were talking about putting a setter on it and seeing whether things still work
<cjwatson> Which I was initially taking to mean renaming to _name and adding @property name
<wgrant> Oh, no.
<wgrant> I did that weeks ago with a validator and got it down to 13 broken tests.
<cjwatson> Aha
<cjwatson> I'd still like to explore easier property renaming, but in that case just not for this particular job
<cjwatson> It's very annoying that renaming Person.displayname to Person.display_name entailed going through and hunting for Storm queries
<wgrant> It is.
#launchpad-dev 2016-11-22
<mwhudson> ha https://code.launchpad.net/~gophers/golang/+git/golang/+index# is marked as failed now
<cjwatson> yeah I need to sort out getting that backport in
<cjwatson> hopefully will get a bit of time tomorrow to push it up the hill a bit
<mwhudson> cjwatson: thanks
