#launchpad-dev 2009-12-14
* spm changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 1 of 3.1.12 | PQM is closed; RC only | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is
<spm> I lashed out on the w/e and bought some vowels. now to spend 'em.
<wgrant> dhillon-v10 has tried to most of my teams in the past 24 hours :(
<wgrant> Er, tried to join.
<spm> yay :-/
<thumper> and dhillon-v10 is?
<wgrant> spm knows him.
<ajmitch> another overly-helpful or inquisitive person?
<spm> ajmitch: exactly. described himself as "... *THE* launchpad answer person". it's nice they're so keen and all; but perhaps overdoing it.
<spm> heh. even so far as to wanting admin powers on LP so he could fix stuff.... :-)
<ajmitch> heh
<thumper> jelmer: there has probably been no change since the 4th of Dec for https://code.edge.launchpad.net/~kiko/linux/2.6.31
 * thumper heads for coffee
<thumper> bbl
 * thumper has lotsa wine now
<thumper> spm: is staging updating?
<spm> thumper: looks like it. it was hung in an earlier update; killed that; is now updating for ~ the past 40 mins.
<thumper> :-| we have some imports running
<thumper> for 4 hours so far
<thumper> bigish bzr-svn tests
<spm> ew
 * thumper stops for dinner
<mwhudson> thumper: https://code.staging.launchpad.net/~mwhudson/boost/trunk succeeded after 8 and a bit hours
<mwhudson> thumper: yay for bzr-svn i think
<thumper> mwhudson: http://staging.launchpadlibrarian.net/36556292/boost-trunk-log.txt are we missing saving a cache?
<mwhudson> thumper: bzr-svn saves its caches in ~/.bazaar somewhere -- the cache will get built once per machine
<mwhudson> thumper: which is a bit lame, but not too bad
<mwhudson> thumper: it would be better to be better, but that requires some hacking of bzr-svn
<thumper> ok
<lifeless> mwhudson: I suggest filing a moderate priority bug on that: the caches will have the following issues;
<lifeless>  - they will grow in size and not be capped
<lifeless>  - concurrent imports from the same svn repo will interfere with each other
<lifeless>  - they can take some time to build
<wgrant> Can I convince a merge proposal diff to be regenerated without pushing a new revision?
 * wgrant considers uncommit, push --overwrite, pull, push
<thumper> wgrant: not yet
<thumper> wgrant: at least I don't think so
<BjornT> gmb: what's the status of the fix for the two failing Bugs Windmill tests?
<BjornT> thumper: what's the status of the fix for the failing Code Windmill test?
<thumper> thumper: fixed awaiting release-critical stamp
<thumper> BjornT: thanks for reminding me, I need to put it on the release blockers
<BjornT> thumper: cool, thanks.
<thumper> wgrant: why is it out of date?
<wgrant> thumper: I set a prereq branch on the MP. It generated the diff with the prereq branch.
<wgrant> But then I pushed some more revs several days later (the prereq branch was still unmerged).
<wgrant> It regenerated the diff, but this time it appeared not to have taken the prereq into account.
<thumper> umm...
<thumper> it will have...
<thumper> unless you convince me otherwise
<wgrant> Argh, staging is too old :(
<wgrant> But you're right, it looks like it does do it, since now that I've repushed it has gone crazy with conflicts.
<wgrant> They are really strange conflicts.
<thumper> wgrant: staging doesn't really have the branches from production
<wgrant> thumper: I know, but it probably does have the crazy diff if a dump is being restored at the moment. But since the new diff is insane too (the pre-req has been merged), I wonder if they just don't merge well (criss-cross merges, or somesuch).
 * wgrant tries.
<thumper> wgrant: if you have a pre-req, then merge trunk into the second, then merge trunk into the first, then the pre-req into the second, you get criss-cross
<thumper> wgrant: I work with pipes when using pre-reqs
<thumper> wgrant: merge trunk into the bottom unmerged pipe
<thumper> wgrant: then 'pump'
<thumper> all done
<wgrant> thumper: Yeah, I normally do too. But not in this case.
 * thumper shrugs
<thumper> oh well
<wgrant> I love bzr-pipeline.
<thumper> me too
<wgrant> Oh.
<wgrant> I see what happened.
<wgrant> The pre-req that was merged is slightly different from the one that Julian landed -- he merged trunk into it, then landed. I didn't notice that it had landed, so I merged trunk into it and then pushed it.
<wgrant> So there is a horrible horrible devel criss-cross in that branch.
<adiroiban> jtv: hi. For branch-sync, I talked with Danilo and Henning and they told me it should also work with an empty branch. bug 494762
<mup> Bug #494762: Export to branch is not working for a fresh branch creating using LP UI <code-integration> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/494762>
<adiroiban> thanks for the email :)
<jtv> adiroiban: it's a somewhat obscure distinction... it'll work for an empty branch, but not for a nonexistent branch.
<jtv> adiroiban: if nothing has been pushed, it's nonexistent.
<jtv> but you can create an empty branch & push it.
<adiroiban> jtv: ok. so the bug is invalid?
<adiroiban> or  is for the wishlist?
<jtv> no, the bug is valid but subtleâit's not actually possible to "create" (in the strictest sense) a branch using the LP UI.
<jtv> the "register a branch" option registers a branch, but does not create it
<jtv> (although if the branch is mirrored, a cron job will then create the branch by importing data from elsewhere)
<jtv> When you register a hosted branch (as you do for translations exports), the LP UI really just "reserves a spot" for it.
<jtv> The solution that we have in mind for the bug is to leave uncreated branches out of the list of options in the branch picker.
<adiroiban> OK. Can I touch that branch now?
<wgrant> danilo[home]:
<wgrant> Argh.
<wgrant> bigjools: Are you likely to have enough time to re-review gina-3.0-support today?
<wgrant> (thanks again for getting the other one landed)
<bigjools> wgrant: yep
<wgrant> bigjools: The diff generated by LP is full of conflicts that I cannot reproduce locally -- the intermediate diff is attached, and http://paste.ubuntu.com/341024/ is the full diff.
<bigjools> ok ta
<bigjools> is the branch against db-devel?
<wgrant> bigjools: It was, but that was before 3.1.11. The MP is still against db-devel, but it's fine in devel.
<bigjools> right, that was my next Q :)
<wgrant> LP doesn't allow me to retarget it :(
<bigjools> yeah, thumper should really fix that :)
<thumper> bigjools: it is on my dashboard :)
<bigjools> \o/
<mrevell> Morning
<beuno> good morning
 * bigjools is loving thumper's change to highlight MP changes done after it was first made
 * wgrant liked that too.
<wgrant> Although no automatic diff intermediate diff :(
<bigjools> gah I need to set up a WOL on my test box
 * bigjools trudges back through the rain into the house
<bigjools> wgrant: FWIW the diff of your gina branch is fine here, I suspect there's a bug in the codehosting stuff
<bigjools> danilos: I need you to approve https://code.edge.launchpad.net/~wgrant/launchpad/gina-3.0-support/+merge/15478 for release-critical
<wgrant> bigjools: It's probably because of the prereq branch. There was something horrible going on there, but I removed the revision that didn't get merged, but the diff was still generated broken.
<danilos> bigjools, sure, looking
<wgrant> Very odd.
<bigjools> danilos: thanks - it was another branch dependent on the buildbot update that didn't get done on Friday
<wgrant> bigjools: (you merged devel into my other branch, then landed it. Later that day, I merged devel into the branch, not realising it had landed. I think that confused things.)
<danilos> bigjools, whoa, that looks huge
<bigjools> danilos: the diff is fucked
* henninge changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is
 * danilos "phews"
<bigjools> danilos: http://pastebin.ubuntu.com/341071/
<danilos> bigjools, it's still reasonably big... have you had a chance to test this already on dogfood?
<bigjools> danilos: doing so right now
<wgrant> It's also only gina.
<danilos> bigjools, ok, let me know how that goes, I'll take a look
<bigjools> danilos: if it doesn't go in, it'll end up being a CP request next week
<bigjools> and the Ubuntu mafia will knock your door down ;)
<danilos> bigjools, heh, I am not that easily intimidated :P
<bigjools> danilos: that cross-dressing guy in the bar in Barcelona did a good job :D
<danilos> bigjools, btw, that cowboy listed on LPS for expire-ppa-binaries is also landed, right?
<danilos> bigjools, heh, I hope you are not trying to do the same :)
<bigjools> danilos: I don't know, it was salgado's bug
<wgrant> bigjools: I presume the dpkg upgrades for cocoplum/germanium/iron and whatever runs debdiff are scheduled?
<danilos> bigjools, though, that'd look good on you :P
<bigjools> wgrant: argh good point
<wgrant> bigjools: buildds are pretty much sorted, fortunately.
<bigjools> yes
<bigjools> wgrant: will gina pay attention to the sourcepackageformatselection ?
<wgrant> bigjools: No.
<wgrant> gina doesn't pay much attention to anything, and the archive is trusted, so I didn't think it important.
<bigjools> I guess it doesn't matter
<salgado> adiroiban, around?
<adiroiban> salgado: hi. yes.
<salgado> adiroiban, hi there.  did you get a message telling you about the tests that were failing in your branch?
<adiroiban> salgado: nope.
<salgado> adiroiban, can you check if it was not incorrectly flagged as spam?
<danilos> bigjools, please add that to the special rollout requirements if it's not there already
<adiroiban> salgado: who is the sender?
<adiroiban> maybe it's just delayed
<danilos> bigjools, also, what about that cowboy for expire-ppa-binaries on librarian? is it landed yet?
<salgado> adiroiban, let me check who's the sender.  it should've been sent Saturday
<bigjools> danilos: do you mean the one to set the number of days?
<danilos> bigjools, there are https://pastebin.canonical.com/25653/ and http://paste.ubuntu.com/339326/
<danilos> bigjools, well, both of these, if you know anything about them
<bigjools> danilos: the first one was sorted with a "proper" fix that landed
<bigjools> 2nd one is salgado's auth store fix
<danilos> bigjools, ok, cool, thanks for the update
<salgado> jml, around?  if I use 'ec2 land' on someone else's branch, using a merge proposal, who gets email if tests fail?
<bigjools> wgrant: did you do any local testing with the gina patches?
<bigjools> dogfood doesn't have a Debian archive handy :(
<wgrant> bigjools: Not for about a month.
<wgrant> There's a small 3.0 archive around somewhere..
 * wgrant hunts.
<bigjools> that'd be very useful
<wgrant> http://people.debian.org/~brlink/v3test/
<bigjools> awesome
<jml> salgado, everyone
<wgrant> debmirror that, I guess.
<jml> salgado, you and the other person
<jml> salgado, --dry-run is your friend
<adiroiban> salgado: can you please forward the email? Thanks!
<salgado> adiroiban, I didn't get it
<adiroiban> salgado: ah.
<jml> g'night
<salgado> adiroiban, I think I detached the thing too early so it didn't send out emails. I'm submitting it again. hopefully we'll get an email this time
<adiroiban> salgado: well, life sucks :) . So you just know that it failed? I just started a new full test, but I ran one before pushing the changes
<adiroiban> salgado: ok. thanks!
<danilos> bigjools, wgrant: btw, I want to keep track of why something had to be RCed (it can be as simple as "buildbot images were not ready for this"), so what about https://code.edge.launchpad.net/~wgrant/launchpad/gina-3.0-support/+merge/15478? also, is there a bug we should add to CurrentRolloutBlockers?
<bigjools> danilos: basically I was a slacker
<danilos> bigjools, ok, good enough, thanks :)
<adiroiban> salgado: ok. although don't know if I can access those logs
<bigjools> danilos: the CP blocked everything on Friday and by the time I'd dealt with that it was very late my time
<danilos> bigjools, right, I can see several other problems CP on Friday might have caused, ta
<danilos> bigjools, btw, has that been CPed? is it worth CPing now (downtime and all) or can we wait for Wednesday release?
<danilos> bigjools, it'd be a shame to cause so many issues and not even CP it if we should :)
<bigjools> danilos: it was CPed today
<danilos> bigjools, ok, thanks
<danilos> adiroiban, btw, I probably won't have much time to help with bug 496361 until the rollout (I am doing release management this time around), but I'd be happy to help with the general direction :)
<mup> Bug #496361: Register a single view (SeriesLanguageView)  for all ISeriesLanguage-implementing objects <Launchpad Translations:New for adiroiban> <https://launchpad.net/bugs/496361>
<adiroiban> danilos: np. I still need to get the current changes landed :D
<danilos> adiroiban, yeah, most of that is not happening before rollout either unless they are RC :)
<adiroiban> danilos: yep. don't worry. I can wait and there are plenty of other unrelated bugs to nail
<adiroiban> are we going to have the final release of YUI available on edge soon ?
<danilos> adiroiban, we don't have it yet?
<danilos> adiroiban, afaik it should be on production as well
<adiroiban> danilos: hm... on devel it was only 3.0_p2
<adiroiban> let me check if it was updated
<adiroiban> ah...stupid me. it's here, but I didn't merged that branch with the latest devel
<danilos> adiroiban, look at lazr-js/build/yui to see what YUI are we actually using
<adiroiban> is there an annoucement list for such updated? I tried to read the bzr log, but I could not find any reference about updatind yui
<bigjools> wgrant: gina has the most bizarrely contrived config I've ever seen
<wgrant> bigjools: YES.
<wgrant> But I think that's as a result of bug #1234
<mup> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <Launchpad Foundations:Fix Released by debonzi> <https://launchpad.net/bugs/1234>
<bigjools> heh
<wgrant> It's now instread an unmaintainable mess of having to add each target to schema-lazr.conf WTF
<bigjools> no kidding, that's a royal pain
 * wgrant tracks down that diff, just to see if it was less foul before lazr.config.
<bigjools> wgrant: more than likely
<wgrant> Yeah, there was no need to add it to the schema with ZConfig.
<bigjools> wgrant: I am singularly failing to get an import done on DF, it's hanging, did you ever see that when you played with it?
<wgrant> No. Let me try it again locally.
<wgrant> bigjools: How did you mirror it?
<bigjools> wgrant: rsync to my box at home then copied a tarball up to DF
<bigjools> s/rsync/wget/
<wgrant> Ah.
 * wgrant tries the remember the correct gina sacrifice sequence.
<wgrant> s/the/to/
<wgrant> bigjools: It's importing OK for me.
<bigjools> ok cool
<bigjools> I think my goat is the wrong breed
<wgrant> Argh, some borkage in the later ones. I think my DB lacks some of the sections.
 * wgrant adds them and tries again.
<wgrant> I think the docs for pocketrelease and distroseries are around the wrong way.
<wgrant> pocketrelease needs to be the source (in this case 'test'), while distroseries needs to be the destination.
<wgrant> Whereas the docs in schema-lazr.conf say the opposite.
<bigjools> it doesn't help when I get a traceback from os.path.join
<wgrant> Ewww. What did you do?
<bigjools> nothing!  line 57 in gina/archive.py causes it
<wgrant> You must have left some config option blank?
<bigjools> I did, but then the prod config does too
<wgrant> bigjools: Hm, where does it hang for you?
<wgrant> Oh, that archive will fail now. Curses.
<bigjools> it's ok now
<wgrant> (it uses lzma compression parts, whereas dak recently changed to not support lzma, and I was advised to follow)
<bigjools> yeah I saw that and wondered
<wgrant> Should I hack a better archive together, or do you want to cheat and add lzma to the regexps?
<bigjools> I'd rather have a realistic archive
<wgrant> I wonder if it will import an archive produced by itself.
<bigjools> hmmm well my test run just went bang
<wgrant> Uhoh. What died?
<bigjools> lots
<bigjools> I'll capture a log this time!
<jtv> bigjools: available when you are
<bigjools> wgrant: http://pastebin.ubuntu.com/341125/
<jtv> adiroiban: just heard that the export-to-bzr bug was a new one, not the one I thought it was
<wgrant> bigjools: Oh, so not actually a test run, but a run over that dodgy archive?
<bigjools> tes
<bigjools> yes
 * wgrant uploads a tarball of his dev PPA with three v3 sources.
<adiroiban> jtv: ok. np. I will not touch that branch
<wgrant> bigjools: I'm not sure about those section errors, but the null filetype ones are because of the lzma.
<bigjools> yeah
<bigjools> jtv: ok, what's going on my man
<jtv> adiroiban: sorry, I was mentally still in the middle of our earlier conversation.  I mean it's the same problem, and with the same solution on your endâit's just that a complementary way of handling it on our end has come up.
<jtv> bigjools: hi!  was hoping we could have a callânot anything too much special, but maybe I could (1) catch up on what happened in my absence and (2) solidify my next steps
 * jtv turn s on air conditioner
<jtv> in the old days you needed friends & family to tell you you had heat on the brain and were spouting nonsense
<jtv> "too much special," indeed
<jtv> now you just see it in your own irc client
<wgrant> bigjools: http://williamgrant.id.au/f/1/2009/wgrant-v3-ppa.tar.bz2, import from lucid.
<bigjools> wgrant: ok ta
<bigjools> going OTP, will do it in a but
<bigjools> bit
<wgrant> bigjools: Do you need anything else from me?
<wgrant> I should sleep.
<bigjools> wgrant: I'll just grab your archive
<Pilky> hey, any launchpad devs available
<james_w> did someone land a change that made patches have different icons to normal attachments?
<Pilky> just wondering if anyone would be interested in seeing and commenting on some mockups I made for ideas on how to improve some parts of launchpad's UI
<maxb> It must be said that the final release-week of the year before everyone goes on holiday is possibly not the optimal time for that question :-)
<Pilky> heh
<Pilky> well I had some free time this weekend and just had a little play around
<Pilky> hoping it might be of interest, maybe for the new year
<maxb> You may be lucky, and find someone who hasn't got anything release-critical going on
<maxb> But dropping an email to the launchpad-dev list might be preferable
<Pilky> ok thanks, I'll try that
<flacoste> morning launchpadders
<adiroiban> how I can aquire devmode in a View?
<adiroiban> devmode and icingroot
<noodles775> adiroiban: what is it that you want to do? (I've normally only needed those from the template, where they are defined in the base layout template)
<adiroiban> I need to include the translations.js in a template/view
<adiroiban> i saw henning doing this for the recent translations.lp.net/+langauges page
<noodles775> Great, he'd be a good person to ask then (and should be around)... henninge ^^ :D
<adiroiban> :)
<henninge> adiroiban: hang on, I'll look it up
<adiroiban> henninge: no hurry
<adiroiban> btw. what is the policy/rule for js function? Should I add them in translations.js ? or i should create a different namespace for each view?
<henninge> adiroiban: add to translations.js unless you are doing something really big
<henninge> adiroiban: is this what you meant? http://paste.ubuntu.com/341215/
<adiroiban> henninge: yes
<henninge> adiroiban: I am not sure where icingroot is coming from
<noodles775> it's set in the base template iirc.
<henninge> adiroiban: this snippet is from lib/lp/translations/templates/translation-import-queue-macros.pt
<henninge> noodles775: I suspected something like that
<adiroiban> henninge: I saw your changes... and I have a similar setup... but I don't know why those variables are not aquired
<adiroiban> henninge: solved
<henninge> oh cool, what was it?
<adiroiban> I think they are defined as global in the main-macro
<adiroiban> so I had to add   metal:use-macro="view/macro:page/main_only"
<henninge> ah yes, you need that.
<adiroiban> as I in a +fragment, it was not previously added
<adiroiban> as I was in a +fragment, it was not previously added
<adiroiban> uh... i can not use view/macro:page/main_only in a +fragment, as it adds the header
<henninge> adiroiban: what exactly do you mean by +fragment?
<adiroiban> henninge: https://dev.launchpad.net/Web/TemplateCodeReuse
<henninge> is that a fragment of HTML returned through API?
<adiroiban> Fragment Views
<adiroiban> I'm working on distroseries-langchart.pt for filtering preferred langauges
<henninge> adiroiban: I guess you will have to do the including in the main page.
<adiroiban> henninge: I will try, but I don't think it will work, as it looks like they are done in different requrest
<henninge> adiroiban: that does not matter to the browser, though. It's the browser that will evaluate the js.
<adiroiban> henninge: just a second . what should I include ?
<adiroiban> :)
<adiroiban> the macros is already included
<henninge> http://paste.ubuntu.com/341215/ ?
<adiroiban> in the main page
<adiroiban> henninge: true. thanks!
<adiroiban> silly me
<noodles775> danilos: when is devel closing (it's not mentioned on the CRB page...)
<jtv> henninge: love what you did to the +languages page
<danilos> noodles775, it's already closed, I didn't know one had the option to not close it when we enter RC mode
<henninge> jtv: me, too ;)
<noodles775> danilos: ok, thanks.
<sinzui> Chex: ping
<Chex> sinzui: hello there
<sinzui> Chex: A user just identified https://edge.launchpad.net/~vicodin-online-35 That account must be SUSPENDED
<Chex> sinzui: ok, looking at it now.
<sinzui> mars: flacoste: anyone looking into SPAM: search for "buy" in review projects" https://edge.launchpad.net/projects/+review-licenses?field.search_text=buy&field.active=True&field.active-empty-marker=1&field.license_reviewed=False&field.license_reviewed-empty-marker=1&field.license_approved=False&field.license_approved-empty-marker=1&field.license_info_is_empty=&field.license_info_is_empty-empty-marker=1&field.licenses-empty
<sinzui> Chex: This user registered 12 project creating spam: https://edge.launchpad.net/~azartok. Please SUSPEND this account.
<Chex> sinzui: both of those accounts are now suspended.
<sinzui> Chex: you rock.
<Chex> sinzui: no problem :)
<salgado> adiroiban, we're having problems with our tools that run the test suite on amazon's ec2, but you might be able to find all test failures by running just './bin/test -vvm translations'
<adiroiban> salgado: ok. thanks. I just ran ./bin/test -vvc -m translations --layer PageTestLayer and everhing was ok. I will try with the full module
<adiroiban> salgado: or maybe I should merge again with devel
<salgado> adiroiban, that shouldn't be needed; just running all translations tests should tell you which ones are failing
<adiroiban> salgado: ok. I have started the test. it should be ready in about 1h
<deryck> sinzui, ping
<sinzui> hi deryck
<sinzui> bac: how goes the QA of mimetypes? Can you prove that I was incompetent?
<MTecknology> Could some brilliant mind please look at bug 496360 and tell me what you think?
<mup> Bug #496360: Can no longer log in with OpenID modules <Drupal OpenID module:New> <Launchpad itself:New> <https://launchpad.net/bugs/496360>
<mars> gary_poster, ^ ?
<MTecknology> The standard OpenID still works for Drupal. I drupal-teams still seems to work. I don't know how to verify drupal-launchpad. If there is an issue in drupal-launchpad it's likely the cause of a change in Launchpad. If this is the case I need to figure out what changed so I can correct this in the module.
<gary_poster> MTecknology: To my knowledge, that code has not changed for > 1 month.  I'll go verify after I send this message.  Another team is in charge of the openid code now, but we deploy it for Launchpad, so I can check for revision numbers.
<gary_poster> I don't quite understand this part of the original report: "However.. If I use the Drupal OpenID module and used the launchpad openid string. Obviously creating a different user in the process. This worked fine."  Does that imply there could be a change to the db alone (not the code)?  I don't understand what the person did in that report.
<bumba> Hello everybody! can anybody tell me why I can't post BB message type of forum, although I am rezistered with them? is it because of my java plug-in?
<gary_poster> MTecknology: mm, there were some changes for Py 2.5 support, but that should not be pertinent...
<bumba> Hello gary_poster can I ask you a question?
<gary_poster> bumba: sure, though I must admit that I don't understand what you just asked.  Is it openid related?
<danilos> noodles775, hi, I am looking at your RC
<noodles775> Thanks danilos... I'll be back and forth for a bit, but feel free to ask any questions :)
<gary_poster> MTecknology: I can verify that the code has not changed except for one test line difference (for Python 2.5) since 2009-10-18
<MTecknology> gary_poster: I'm lost right now because Drupal 5 is used for the fridge and the module code hasn't changed in long time. The code for the modules in Drupal 6 changed recently but those are trivial as well.. It affected Drupal5 and Drupal6 at the same time..
<MTecknology> gary_poster: thanks for looking at that
<mars> gary_poster, I was just doing QA on r10025, the buildonce_eggs target.  I'm not sure that the target is doing what you expect.
<gary_poster> MTecknology: ok, let me ask SAs if we have seen any DB changes recently.  That's the only other thing that could affect this AFAIK.  What's the most specific time frame you can give me for when this started breaking?
<gary_poster> mars, ok, thank you.  I tested it, but certainly may have missed something.  what are your concerns?
<mars> gary_poster, "find eggs -name '*pyc'"  and find eggs/ -name '*pyc'" return different results in trunk/.
<danilos> noodles775, btw, can you add it to CurrentRolloutBlockers as well so we can track QA and status?
<danilos> noodles775, oh, it's there, nm me :)
<mars> gary_poster, since eggs is symlink in trunk, "find eggs" won't work.  You have to explicitly tell it to enter the directory, with "find eggs/", or "find -L eggs"
<MTecknology> gary_poster: I just noticed about 2 weeks ago that something was funky. I normally stay logged in for a while so that's teh best I can do... sorry
<gary_poster> mars, for LOSA's this will not be a symlink.  you should test this by moving your egg directory aside
<mars> gary_poster, since this fix is for production, and assuming that in production eggs/ is a hardlink or directory, this may not be an issue
<gary_poster> mars: right.  If the eggs directory is missing, one is created.  that's what the LOSAs use
<gary_poster> MTecknology: eh, ok, that's pretty broad, but I'll see what the SAs can give us.
<mars> gary_poster, ok, I'll mark it as OK then
<gary_poster> thank you mars
<danilos> noodles775, I'll be leaving soon; the change itself looks good, and I can see how it's pretty important; good QA is a must though :)
<MTecknology> gary_poster: sounds like it was working for one person on the fridge on the 10th. They didn't try to log in again until today.
<gary_poster> MTecknology: ok cool, I'll add that to the description
<MTecknology> gary_poster: thanks. It could very well be a bug in Drupal or the modules. I'm just trying to follow the logical order.
<gary_poster> MTecknology: ack, makes sense
<danilos> noodles775, I'll be leaving now, and the branch looks good
<MTecknology> gary_poster: It doesn't look like Drupal changed any in this time...
<MTecknology> gary_poster: actually..... The moment where it seemed everything broke is after the last time LP went down
<MTecknology> err - the time before that maybe..
<gary_poster> MTecknology: Drupal not changed: OK.  I have my DB question floating the proper place.  That--or some certificate thing or something--is the only thing I can figure.
<gary_poster> MTecknology: everything broke after the last time LP went down: How do you mean?  I thought you said someone had success on Dec 10?
<gary_poster> MTecknology: will be afk for a bit.
<MTecknology> gary_poster: I guess too much changed for me to have any specifics on the issue. I guess I'm just hoping it's somethin with Launchpad itself because D5 and D6 have the same issue when nothing changed with either one.
<MTecknology> If there was a db change that would make plenty of sense that this exact issue come up.
<noodles775> Thanks danilos - yep, I'll run it through on dogfood first thing tomorrow.
<MTecknology> gary_poster: thanks for looking into this. I might be afk a little bit too.
<danilos> ok, approved, I assume this is not something that can even be remotely testable or usable on edge solely, so let's get it in
<danilos> noodles775, do some good QA on it, especially with the big refactorings you've had in soyuz recently :)
<noodles775> yes, I spent quite a bit of time dogfooding the IBuilder refactoring already.
<noodles775> Will do.
<bac> sinzui: i was able to confirm the incorrect mimetype downloading from staging.  i'm about to do more testing on lp.dev
<sinzui> bac: Should we revert, or try to fix it?
<bac> sinzui: i don't see any reason to revert.  we're no worse off now than before
<sinzui> bac: is the patch mime-type working?
<bac> i'd like to investigate to come up with a fix...at least understand what is happening
<bac> sinzui: http://paste.ubuntu.com/341393/  you see here that the local mimetypes modifications work, at least locally using bin/py
<bac> sinzui: this, however, is still messed up:  wget -S http://staging.launchpad.net/sachco/trunk/0.1/+download/words.tbz2
<thumper> danilos: ping
<mars> leonardr, ping, is there some way to tell launchpadlib to only use cached responses?  In essence, to work offline?
<leonardr> mars: no, because not everything is cached. for instance, collections
<leonardr> if everything were cached, then there would be no problem using it offline
<leonardr> it would work automatically
<wgrant> bigjools: Thanks.
<bigjools> np
<mars> leonardr, ah.  My use-case is running the same spam filter script over and over again.  I don't want to keep re-fetching person objects if I don't have to.
<bac> hi gary_poster
<mars> leonardr, it should be easy enough to code something myself
<leonardr> mars: well, it might be making conditional requests to see if those person objects have changed. i don't know if launchpad sends cache settings
<mars> leonardr, yeah, but getting 304s is slow, too
<mars> or HTTP HEAD requests
<wgrant> bigjools: What do you know about the dpkg upgrades?
<bigjools> wgrant: not done yet
<leonardr> mars: what i was saying was, to avoid 304s launchpad should send cache directives so the client knows how long it can wait between conditional requests
<leonardr> but i don't think we do that
<mars> ah
<mars> leonardr, alright.  Actually, in this case I think a launchpadlib caching wrapper is the right approach.  I can set it as aggressive as I want.  I'm not actually worried if the data goes stale, I just want the speed.
<leonardr> mars: got it. what kind of wrapper were you thinking of?
<mars> leonardr, so that looks like a custom script, not a launchpadlib feature
<leonardr> ok, i was just wondering if i could be helpful
<mars> leonardr, I was just going to wrap the launchpad.people object, and write out JSON to the disk (unless there is something already in Python...)
<mars> desktopcouch....
<leonardr> mars: so it sounds like you want to use launchpadlib once to get the data, save the data to disk, and then use the data locally
<leonardr> unless you need to do some complex manipulation where the full launchpadlib class api would be useful
<mars> leonardr, yep.  But I want to keep fetching more data, as I add features to the script.  Thus a cache, rather than a one-off data dump.
<mars> right
<mars> leonardr, come to think of it, a feature to load an object from JSON should be possible, since representations have a self_link.
<leonardr> mars: not sure exactly what you want. you can pass a self_link into lp_load but that will go over the network
<leonardr> do you want to create a Person from the JSON data?
<mars> leonardr, since you can call launchpad.load('some/person'), then you should be able to call launchpad.from_json('{.... self_link, 'some/person' }')
<leonardr> you might be able to do that with some hacking
<leonardr> yes, you shoudl be able to call whatever code lazr.restfulclient calls to instantiate an object from json--i don't know what that is
<wgrant> bigjools: Did my test archive import OK?
<bigjools> wgrant: yep
<wgrant> Phew.
<bigjools> I have more problems now though
<wgrant> Hm?
<bigjools> we're doing some testing on DF and when the buildd-manager collects a build, it bails out with "Scanning failed with: 'NoneType' object has no attribute 'utf_8_decode'"
<wgrant> I love the way it swallows exceptions.
<bigjools> I don't
<bigjools> I'm trying to see if there is by some miracle a traceback hiding in the error
<wgrant> IIRC you just have to remove the try/except.
<wgrant> Much like you need to hack the code to change the debug level :(
<bigjools> aha I see something that might help
<wgrant> I've never seen that exception locally, and I've gone through quite a few builds in the past week.
<mwhudson> bigjools: 'NoneType' object has no attribute 'utf_8_decode'" ??
<mwhudson> that was the error that caused so much fun in the last rollout
<bigjools> mwhudson: correct
<bigjools> really?
<bigjools> tell me mo'
<mwhudson> yeah
<mwhudson> bigjools: it's probably to do with the crazy sys.modules mangling bin/run and _pythonpath.py do
<bigjools> mwhudson: hmmm how was it fixed?
<bigjools> it's buggering up dogfood now
<mwhudson> bigjools: it was a bit random, it seems like it probably was fixed by accident by the python 2.5 upgrade
<mwhudson> (the problem was restarting the appservers)
<mwhudson> bigjools: buildd-manager will likely be a different case, slightly
<mwhudson> bigjools: can you pastebin a traceback?
<wgrant> Ha ha ha.
<wgrant> This is buildd-manager.
<wgrant> There is no traceback.
<bigjools> mwhudson: if twistd didn't swallow the traceback ...
<bigjools> well actually it's buried in the error object, Imight be able to get it
<mwhudson> oh right, yay
<mwhudson> bigjools: can you hack _pythonpath on dogfood?
<bigjools> if it doesn't require root, yeah
<mwhudson> bigjools: put 'print "deleting", k' before the del sys.modules[k] around line 238
<bigjools> mwhudson: I got a traceback anyway
<mwhudson> bigjools: ah cool
<bigjools> http://pastebin.ubuntu.com/341426/
<bigjools> mwhudson: is that pythonpath hack still useful?
<mwhudson> bigjools: yes, still worth a try
<mwhudson> if buildd-manager doesn't print anything out on startup i'd be a bit stumped...
<bigjools> it doesn't
<bigjools> well, searching for "deleting" shows nothing
<bigjools> unless it's swallowed by something
<mwhudson> hm, shouldn't be
<mwhudson> i guess you can put a print in at module level to check that
<bigjools> mwhudson: what do you mean?
<mwhudson> bigjools: if you just put "print 'bleargh'" as the first line of the module, and that doesn't show up then you know it's being swallowed somehow
<bigjools> mwhudson: which module?
<mwhudson> bigjools: _pythonpath
<bigjools> ok
<bigjools> well I;ve not done that yet
<bigjools> I just rebuilt my environment
<MTecknology> gary_poster: You hear anything back yet?
<mwhudson> bigjools: some background as to what is probably happening here
<mwhudson> bigjools: when a python module is garbage collected, all the globals in that module are set to None
<bigjools> now I get http://pastebin.ubuntu.com/341430/
<bigjools> mwhudson: different error now
<mwhudson> bigjools: !!
<bigjools> the librarian is rejecting the upload!
 * bigjools bangs head on wall
<mwhudson> bigjools: almost all of the traceback is the same though
<bigjools> yeah, it was failing uploading to the librarian before too
<wgrant> Woo corrupt librarian files on production.
<mwhudson> i don't know if that is a clue or not though
<gary_poster> MTecknology: yes, from SA: "there was an architectural change made a little while back.  [authmaster changed] i am not sure exactly when that cutover was made though".  Not sure how it could be related to the issue you describe.  I need to escalate to two other people, one in a different timezone.  Do you happen to have access to error messages on fridge?
<gary_poster> (other than "OpenID login failed" :-) )
<MTecknology> gary_poster: unfortunately no - But I can get to logs on my own system
<gary_poster> MTecknology: +1.  Could you put 'em on a pastebin?
<bigjools> gary_poster: if you have a moment can you check http://pastebin.ubuntu.com/341430/ and tell me what you think about the error?
<gary_poster> looking bigjools
<gary_poster> bigjools: I'm not a librarian guy, but it looks like we are encoding the filename string incorrectly.  Do we know what the filename was?
<MTecknology> gary_poster: I don't seem to have anything useful in apache logs..
<bigjools> gary_poster: this is the buildd-manager, it hasn't changed in many cycles
<gary_poster> MTecknology: :-(
<bigjools> gary_poster: and now I get this error on dogfood
<MTecknology> gary_poster: you have an OpenID account I assume.. You can try to log into staging.profarius.com to see what happens if you'd like.
<gary_poster> bigjools: sure, but this is an issue on launchpad calling the librarian afaict
<bigjools> gary_poster: did the librarian client change with regard to utf?
<gary_poster> MTecknology: ok.  was going to try one of the openid demo sites.
<MTecknology> sd.ubuntu-us.org if you would prefer a site with trust root verified
<gary_poster> bigjools: not to my knowledge but will look
<bigjools> gary_poster: ok - the other possibility is that it's related to the other error before I restarted the librarian: "'NoneType' object has no attribute 'utf_8_decode'"
<bigjools> which mwhudson says we had on the last rollout
<gary_poster> bigjools: ah, yes we did.  do a make clean and a make just to be careful
<bigjools> ok
<MTecknology> gary I can get you the link to the source code for that stuff if you'd like.
<gary_poster> MTecknology: no way I have time for that I'm afraid.  The sd site has timed out twice.  I'll try the other one
<MTecknology> gary_poster: I'm not sure what's up with that actually.. staging.profarius.com or profarius.com should work without issues
<gary_poster> MTecknology: yes it does but error message is as useless as one from fridge, I'm afraid.
<MTecknology> ya
<MTecknology> I'll try to grab watchdog errors if any exist
<bigjools> gary_poster: phew, that seems to have fixed it
<gary_poster> bigjools: good.  sorry for the scare.  fix for scare in progress.
<bigjools> gary_poster: :)
<MTecknology> gary_poster: I don't think this helps.... http://paste.ubuntu.com/341443/
<MTecknology> gary_poster: is 67.195.112.239 one of yours?
<gary_poster> MTecknology: agree doesn't help; and no, I don't see myself on there, oddly enough.  Going to escalate on another channel; will let you know if I discover anything.  May not get a reply till tomorrow.  If I haven't contacted you by tomorrow, please feel free to ping me.
<MTecknology> gary_poster: thanks. I think this may actually affect some people paying canonical. I'm not sure but it might be a good idea to ping stu or elachuni and ask them if they know anything.
<gary_poster> MTecknology: ok, will do (and escalating elsewhere now)
<MTecknology> :)
<bac> hi abentley
<abentley> bac: Hi.
<bac> abentley: how can i make 'bzr send' create a MP against db-devel instead of devel, without editing my locations.conf to change the target for trunk?
<abentley> bac: Specify db-devel as the first parameter to "bzr send".
<bac> abentley: simply 'bzr send db-devel'?
<bac> or the full URL i suppose
<abentley> bac: You need to specify a URL that Launchpad recognizes.  lp:launchpad will work, or lp:~launchpad-pqm/launchpad/db-devel, etc.
<abentley> You can also use the bzr+ssh or http URLs if you like.
<bac> abentley: great.  i'm creating a dbsend alias similar to the dbsubmit you suggested a while back
<abentley> bac: Cool.  I have one like that, also.
<abentley> bac: You can use a local mirror, too, if the local mirror has its public_branch set.
<bac> abentley: this page needs updating, i think.  'bzr lpsend' is deprecated, right?  https://dev.launchpad.net/WorkingWithDbDevel
<abentley> bac: It's a little mixed up.  lpsend used "send --no-bundle", which is a good idea right now, because we seem to have some bugs applying bundles.
<bac> ok
<bac> abentley: thanks for your help
<abentley> bac: No problem.
<jml> thumper, inline propose for review pls.
<thumper> jml: eh?
<jml> thumper, https://code.edge.launchpad.net/~jml/launchpad/fix-release-hot-bugs-486437 -- clicking "Propose for merge" takes me to another page. I'd love the form to be inline
<thumper> jml: one day...
<jml> thumper, :)
<thumper> beuno: ping
<beuno> thumper, pong
<thumper> beuno: hey, got a few minutes for a call?
<beuno> thumper, I'm double late, even for a south american
<beuno> tomorrow?  :)
<maxb> Whoever implemented display of PPA packages obsoleted by the primary archive, you rock! :-)
 * mwhudson lunches
 * thumper lunches
#launchpad-dev 2009-12-15
 * jml hunches
 * jml brunches
<jml> you know what
<jml> I think we should combine this channel and #launchpad
<maxb> Mhm. I'm not really convinced
<thumper> mars: I don't suppose you are still around?
<thumper> jml: what is your command to kill the mailman bit of make run_all?
<jml> thumper, it doesn't work any more :(
<thumper> :(
<jml> thumper, ps x | grep mailmanctl; kill the appropriate process
<thumper> spm: ping
<azop> Does anyone have any updates on the redmine + launchpad integration?
<azop> Bug #324387
<mup> Bug #324387: Support Redmine Bug Tracker for bug watches <bugwatch> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/324387>
 * thumper thrashses local lp
<jml> thumper, why can't I specify the review type?
<jml> oh yuck. bugs.
<thumper> jml: I don't know, why can't you?
<thumper> jml: you have to specify it before choosing the person right now
<thumper> kinda blows a bit
<jml> thumper, I'm doing a review using the new form layout that I did a review of for you recently
<thumper> and you used the keyboard?
<thumper> I found that this morning too :(
<jml> thumper, I can't hit Tab, select approve and then Tab to the review type
<thumper> yep, bug
<jml> thumper, I have to actually click inside the comment field -- that's the only thing that seems to do it
<jml> thumper, I guess that explains the comment from Michael Nelson.
<thumper> clicking on the review selector works too
<jml> about enabling etc.
<thumper> it seems to be on click not change
<thumper> if you file the bug, I'll fix it
<thumper> deal?
<jml> deal
<jml> thumper, I'm going to file the bug as critical, since I think it's a serious regression.
<thumper> jml: it isn't a regression
<thumper> jml: same behaviour as before
<thumper> yes it is a bug
<thumper> but it isn't a new bug
<jml> thumper, I'm pretty sure I've been able to hit tab before and have it work.
 * thumper tries prod
<thumper> jml: it is very weird on prod
<jml> thumper, it's broken there too, I see.
<jml> thumper, I think the new behaviour is still worse
<jml> thumper, precisely because the order of UI widgets is better.
<thumper> file the bug, I'll fix it
<jml> thumper, anyway, I'll contend that it's an rc bug.
<thumper> I'm not sure our windmill tests can test that level of messing around
<jml> thumper, https://bugs.edge.launchpad.net/launchpad-code/+bug/496820
<mup> Bug #496820: "Review type" field is never enabled if you use keyboard only <code-review> <ui> <Launchpad Bazaar Integration:Triaged by thumper> <https://launchpad.net/bugs/496820>
<thumper> but I'll add the fix
<thumper> ta
<thumper> jml: on the plus side it looks like a one liner
<jml> thumper, and the down side?
<thumper> it nees an rc
<thumper> and I don't know how to test
<jml> met
<jml> meh, rather.
<jml> release it first, clean up the bodies later.
<thumper> jml: confirmed the one line fix
<thumper> now...
<thumper> a test
<adiroiban> hi. does anyone know how to solve the "Non-english database locales are not supported with launchpad. " problem? Thanks!  http://paste.ubuntu.com/341610/
 * jml doesn't
<thumper> hmm
 * thumper doesn't
<thumper> it should be making the db with UTF-8
<thumper> stub: do you know anything about the above pastebin?
<wgrant_> thumper: Charset != language
<stub> adiroiban: The postgresql database cluster needs to be in the C locale or your ordering will be locale specific and all the tests break. You need to rebuild that db cluster in the C locale.
<adiroiban> I'm trying to install launchpad on a fresh Ubuntu 9.10 server. The strange thing is that it worked on my Ubuntu 9.10 Desktop
<stub> If you are not using PG for anything else (this will destroy your databases), pg_dropcluster --stop 8.3 main; pg_createcluster --locale=C 8.3 main
<adiroiban> stub: thanks. I'm using 8.4 and setting a machine for running full tests
<adiroiban> so I can destroy it... no problem :)
<stub> adiroiban: The Launchpad test suite triggers bugs in PostgreSQL 8.4. 8.4.2 should be out 'real soon' with the fixes.
<adiroiban> stub: thanks for the info. I'll go with 8.3 then
<adiroiban> right now launchpad-database-dependencies for Karmic depends on 8.4
<jml> how do you reload a page in a page test?
<thumper> browser.open
<thumper> or
<thumper> browser.reload()
<jml> thanks.
<jml> fwiw, I prefer IBranchCollection to the bug task search interface.
<thumper> :)
<thumper> me too
<jml> thumper, btw, love the new activity stuff on the code review page
<thumper> jml: thanks, I love it too
 * jml afk
 * thumper EODs
<thumper> will check with danilos later for rc
<mwhudson> jml: hello
<mwhudson> jml: can you have a look at this patch and see if you think it's ok as far as it goes? http://pastebin.ubuntu.com/341643/
<jml> mwhudson, ok
<mwhudson> jml: any thoughts?
<jml> mwhudson, no, not really
<jml> mwhudson, I'm not sure what the scope of the patch is
<mwhudson> jml: i guess that's good
<mwhudson> jml: i just wanted to get a single simple test passing
<jml> mwhudson, ahh ok.
<jml> mwhudson, I think that's about right.
<mwhudson> jml: and then want to check that i'm not doing anything completely tasteless
<mwhudson> jml: thanks
 * mwhudson has felt like no-one else is in this particular box with him
<jml> mwhudson, yeah, I know what you mean
<mwhudson> i guess it's time to EOD
<adiroiban> can I send merge proposals for LP using "bzr send" on the wiki there is only information about sending via web UI https://dev.launchpad.net/PatchSubmission
<jtv> adiroiban: as I recall that was supposed to work, but we've gone through tons of changes since I last used it
<jtv> adiroiban: IIRC the main thing I liked it for was the pretty-printed diffs, and nowadays the MP produces that anyway.
<adiroiban> jtv: thanks. I'll try and if fails I will use the web ui
<BjornT> adiroiban, jtv: using 'bzr send' works perfectly well. i think most people use --no-bundle, but i'm not sure whether it's necessary, or not.
<BjornT> adiroiban: don't forget that you need to gpg sign the e-mail, though
<adiroiban> BjornT: thanks for the tips. I didn't know about gpg requirement, but I can do it. that's not a problem :)
<maxb> Anyone know off hand what python-pullparser is in lp-dev-deps for?
<maxb> It's removed from Debian and Lucid
<maxb> ("ROM; Unmaintained upstream")
<stub> I would suspect parsing hardware database submissions
<stub> Possibly for parsing remote bugtracker data
<maxb> Shouldn't it be in lp-deps rather than lp-dev-deps for that sort of thing?
<maxb> grep: zero hits
<maxb> guess I'll try a testrun
<BjornT> maxb: i got some hits for pullparser. mechanize seems to use it. mechanize is used by zope.testbrowser, which is used by our page tests
<leonardr> i'm getting a segfault when i run the launchpad test suite. any ideas?
<leonardr> no output, bin/test just segfaults
<leonardr> bin/py runs fine
<wgrant> leonardr: You were gone during the 2.5 migration, weren't you?
<leonardr> wgrant: indeed
<wgrant> leonardr: make clean; make
<leonardr> are you sure that will help? this is a brand new branch
 * leonardr is trying it
<wgrant> The problem is in sourcecode/, IIRC.
<leonardr> ah
<wgrant> I think the root Makefile has been taught to clean that too, but I guess you'll find out
<wgrant> Yeah, make clean should do it all.
<maxb> BjornT: Oh, it's inside an egg, explains why I didn't see
<mat_t> Hi LP people
<mat_t> When posting a comment I often get a popup message with an error:
<mat_t> The following errors were encountered:
<mat_t>     * Object: , name: u'dead-ayatana'
<mat_t> The comment is actually getting posted, but I have to refresh the page to see it
<mat_t> This has been a case for quite a long time now... any ideas?
<intellectronica> argh, i get a segmentation fault trying to run windmill. has anyone seen something like this recently?
<intellectronica> BjornT: have you? ^^^
<bigjools> mwhudson: hey how's the build job work coming along?
<mwhudson> bigjools: slow progress really
<bigjools> is anything blocking you that I can help with?
<mwhudson> bigjools: not really, it's just boring typing at this stage :)
<bigjools> yeah it's fun making new model classes :)
<mwhudson> it's also a bit scary in some ways, you have to put so much work in before you get to something useful
<mwhudson> but well, getting through that now
<bigjools> it's also a bit of a nightmare testing with our buildfarm
<mwhudson> i look forward to having that problem, in some ways :)
<mwhudson> anyway, time for bed
<bigjools> g'night
<mwhudson> good night!
<stub> intellectronica: I got a segfault earlier today but make clean and make build cleared it up. I couldn't see a cause - the only thing I'd changed was a .py file.
<stub> (between the segfault and the previous successful invocation of that test)
<intellectronica> stub: cool, let me try that
<adiroiban> henninge jtv can you give me a quick hint about how can I import some templates in lp.dev ?
<adiroiban> the defaul import queue is empty
<henninge> adiroiban: using bzr import or upload?
<henninge> adiroiban: oh, I see what your problem is
<adiroiban> henninge: i don't know ...
<adiroiban> I only want to have them there
<adiroiban> I assume there is a script / cron-job
<adiroiban> for that
<jtv> adiroiban: just a manual upload is probably easiest
<henninge> adiroiban: if you just need something in the queue, a simple upload of a tar ball is easiest locally
<jtv> adiroiban: did you upload already?
<jtv> once it's in the queue, you need to approve the entry/entries
<henninge> adiroiban: you need to "make run_all" AFAIK. jtv?
<jtv> yes, that's right
<adiroiban> aha. I have uploaded a new POT file for a sourcepackage
<adiroiban> but I don't know where it went
<henninge> adiroiban: so, uploads should appear in the queue immediately
<jtv> check translations.launchpad.dev/+imports.  once the queue entry for the pot is approved, you run
<jtv> rosetta-poimport.py
<jtv> from the source tree;
<henninge> cronscripts/rosetta-poimport.py
<jtv> right.  I normally keep my local lp running while I do that, though don't know right now if that's really needed
<henninge> jtv: me too
<henninge> I mean, I leave it running
<henninge> (not my car)
<henninge> oh, I don't have a car atm
<henninge> ;-)
<adiroiban> henninge, jtv thanks. I should be good to go chacking that bug :)
<jtv> I would ask what happened to the one you were going to get, but workrave insists I'm on a break now :-)
<adiroiban> chasing
<jtv> adiroiban: what bug?
<adiroiban> jtv: bug 496896
<mup> Bug #496896: Permit distribution translation group owners to administer the import queue <ui> <Launchpad Translations:New> <https://launchpad.net/bugs/496896>
<jtv> ah cool
<adiroiban> jtv: with the latest switch from launchpad.Admin to launchpad.TranslationsAdmin
<adiroiban> we need to enable them in the template
<adiroiban> it should be trivial
<jtv> not sure that really should be "for distribution" and not "for Ubuntu"; we don't want just anyone to register a proprietary distro and start using Rosetta for it
<jtv> adiroiban: "trivial" is a bit of a red-flag word for me.  heard it far too many times.  :)
<adiroiban> :)
<BjornT> intellectronica: what did segfault? the test runner, or the app server? (and no, i haven't seen that before)
<jtv> lot of the circumstances have changed w bzr imports though
<jtv> adiroiban, henninge: I'm going to try & have that typing break now :)
<adiroiban> jtv: regarding new permission, danilo has review them
<jtv> adiroiban: then I'm sure it's ok
<adiroiban> i hope so. basically he wrote that code :)
<intellectronica> BjornT: i think it's test test runner, but i'm not sure. 'Segmentation fault.' was the only output i've gotten after running bin/test. make clean and make build as per stub's advice did make it work again
<adiroiban> i just pushed for review some stupid changes and he had to fix them
<stub> intellectronica: That was exactly the behaviour I saw.
<adiroiban> henninge, jtv can you please help me set up a test where a need to add translations from 2 persons to a pofile. This is what I got now http://paste.ubuntu.com/341860/
<jtv> adiroiban: we're just having a call here, hang on!
<adiroiban> jtv: np
<henninge> adiroiban: in the first code you need to call makePOTMsgSet
<henninge> adiroiban: and always use sequence=1 (or more if you have multiple)
<henninge> as a parameter
<henninge> adiroiban: ok, that is probably what you are doing in the second code, so nm.
<henninge> adiroiban: no need to call setSequence, especially not twice
<adiroiban> henninge: but in factory.makeTranslationMessage, if msgset is None, it is created...just like I did ...but that way is not working
<henninge> adiroiban: so, are you trying to create two translations for the same msgset, or do you just want to add two English strings with a translation each?
<adiroiban> I just want to have 2 translators for es
<adiroiban> for a single pofile
<henninge> adiroiban: so you don't care *what* they actually translated?
<adiroiban> nope
<adiroiban> i just want to have them in the list of contributors for that pofile
<henninge> adiroiban: so the first code should be good if you add sequence=1 and sequence=2 to the two makeTranslationMessage calls.
<henninge> and fix the "merged_translatort" typo
<adiroiban> henninge: no luck . This is what I got http://paste.ubuntu.com/341882/
<henninge> adiroiban: this fails when trying to open a url. What's the url you are trying to open?
<adiroiban> http://paste.ubuntu.com/341887/
<adiroiban> henninge: sorry. i found it.
<adiroiban> thanks.
<sinzui> salgado: gary_poster: Is it possible to make TestBrowser to accept a 410 HTTP status code so that I can test what the browser sees?
<salgado> sinzui, you can try setting raiseHttpErrors to False in your browser instance, but I'm not sure it will actually work
<sinzui> salgado: I think that is exactly what I want. thanks
<gary_poster> MTecknology: you saw stuart metcalfe's reply on bug 496360?
<mup> Bug #496360: Can no longer log in with OpenID modules <Canonical SSO provider:Fix Released> <Drupal OpenID module:Invalid> <Launchpad itself:Invalid> <https://launchpad.net/bugs/496360>
<gary_poster> leonardr: skype?
<leonardr> gary: argh
<leonardr> sorry, irc was on the other computer
<leonardr> i'm on skype now
<beuno> intellectronica, did you send out the noted from yesterdays' call?
<intellectronica> beuno: sorry, not yet. will do that now
<beuno> intellectronica, thanks, I'm trying to remember what I promised I would do  :)
<danilos> sinzui, hi, do you have a few minutes to discuss https://code.edge.launchpad.net/~sinzui/launchpad/spam-eggs-bug-495250/+merge/16174?
<sinzui> sure, since the tests fail
<sinzui> danilos: ^
<danilos> sinzui, right, so, even though I'd really like to see this in asap, I think it's something worth punting for the January release so we can make sure we get it nicely polished without introducing risks into this release
<sinzui> polished?
<sinzui> The can be CPed later this week or next
<danilos> sinzui, well, QAd properly (i.e. more eyeballs on it on edge for a while etc.)
<sinzui> danilos: actually. I am certain this branch will be CPed since we will not have a full staff watching LP at the end of the Month. I Can land this today, or at the end of the week.
 * sinzui hates this branch too much to care
<danilos> sinzui, well, can you ensure at least some QA happens? can you QA it on staging or dogfood (by cowboying it in)? otherwise, I'd rather wait for a re-roll or get it CPed later
<sinzui> Yes, wait for a reroll. Yesterday was too stressful. I am not willing to hurt myself for scope creep
<danilos> sinzui, ok, sure
<sinzui> \o/ and I have a legitimate way to test 410 in admin tests! I might be able to do the work I was supposed to do on Thursday, Friday, and Monday
<danilos> sinzui, heh, what's the trick there? can you get headers in a browser object?
<sinzui> Well there are two tricks for two conditions
<sinzui> To view pages that raise a 410:
<sinzui>     >>> admin_browser.handleErrors = True
<sinzui>     >>> admin_browser.raiseHttpErrors = False
<danilos> sinzui, ah, nice
<MTecknology> gary_poster: ya, thanks for everything :D
<gary_poster> :-)
* danilos changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only (https://dev.launchpad.net/CurrentRolloutBlockers, RM: danilos) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<henninge> sinzui: ah. lights are back on here.
<henninge> sinzui: this is the test
<henninge> http://paste.ubuntu.com/341957/
<henninge> sinzui: yes, it's pushed
<henninge> sinzui: ~henninge/launchpad/bug-495126-deactivate-users
<MTecknology> gary_poster: so - do you know when that change did happen?
<henninge> sinzui: wait, I just saw an error in the test!
<mwhudson_> morning
<MTecknology> Can I get a static link to a launchpad team image? Instead of say https://launchpadlibrarian.net/19775148/g2529.png I have https://launchpad.net/~ubuntu-southdakota/+image/large
<beuno> MTecknology, there os no such URL
<beuno> but I think it would be useful
<beuno> not sure how much work it would be
<MTecknology> beuno: I could really really use something like that about now :P
<MTecknology> beuno: for the LoCo Map
<MTecknology> I'll file a bug when I get back in ~10min - see how hard it would be
<MTecknology> THanks
<maxb> Such an URL would only be useful for pulling the images externally without consulting the API, though
<lifeless> MTecknology: short answer is no, it would allow xss attacks
<lifeless> unless its off in a different domain anyway, so you'd still not be able to just use the team url
<MTecknology> lifeless: how so?
<MTecknology> maxb: can I use the api to do that?
<maxb> XSS attacks? Even though it's presumably forced to be an image?
<MTecknology> maxb: you mean by uploading a harmful image?
<MTecknology> wouldn't that still be vulnerable in the same way?
<maxb> Can an image be harmful?
<MTecknology> ya..
<MTecknology> I was just lookin in the apidoc and I don't see an image for teams - nor do I really know how to use the api anyway
<MTecknology> nvm - mugshot_link
<MTecknology> Now I need to learn how to actually use the api to make use of this...
<lifeless> maxb: there have been many bugs found in image libraries
<lifeless> maxb: so yes
<MTecknology> Unknown consumer (None).
<MTecknology> I can't use it yet, can I?
<wgrant> lifeless: Lots of files are served from URLs like that -- they just redirect to a librarian URL immediately.
<lifeless> wgrant: gotcha
<MTecknology> Can I use the API in this way or is it not open for me to use?
<wgrant> You cannot yet easily use it anonymously.
<MTecknology> wgrant: Is there any way I can become not so anonymous?
<wgrant> MTecknology: https://help.launchpad.net/API
<MTecknology> wgrant: I was looking at that
<MTecknology> it's somewhat outside of my comprehension :P
<MTecknology> ok - I'm starting to get it.. this will need to be PHP in the end though... Thanks
<wgrant> Ew ew ew why?
<MTecknology> wgrant: I'll try to use python
<MTecknology> The only thing this is even doing is grabbing the image url :P
<wgrant> MTecknology: Retrieve a URL like https://launchpad.net/api/beta/~motuscience/mugshot_link
<wgrant> You  should be able to do that unauthenticated.
<MTecknology> wgrant: except for the error it's giving me..
<MTecknology> XML Parsing Error: not well-formed
<wgrant> MTecknology: That's because it not an XHTML document, but a raw URL.
<wgrant> I don't know why the mimetype is wrong, but you should ignore that.
<MTecknology> https://launchpad.net/api/beta/~ubuntu-southdakota/mugshot
<MTecknology> there :)
<wgrant> Ah, that redirects. Nice.
<MTecknology> that's exactly what I want :D
<MTecknology> wgrant: thanks :D
<mars> Down to running the final spam names filter.  It puts a joyful, evil grin on my face >:D
<MTecknology> mars: kill the annoying spammers?
<mars> MTecknology, that is for the losas to do.  I'm just pointing out where the bodies are buried.
<mars> maybe 1500 of them
<wgrant> Ouch.
<mars> yay for automation
<MTecknology> mars: you should automate a tool for them to destroy them too
<MTecknology> ie - box pops up; they glance and verify; destroy; tool moves on
<MTecknology> 1yr later all 1.5k are gone and they have a new batch
<mars> MTecknology, yeah, but they know the proper procedures, and I can just hand a confident list over to them.
<mars> and there are ways to do graduated deactivation in such a way that wrongfully nixed accounts have recourse
<MTecknology> mars: you lost me there - "graduate deactivation"
<mars> MTecknology, erm, "graduated deactivation": deactivate, send an email stating the reason, "Go here to reactivate if we're wrong", after X days shut it down.
<MTecknology> oh
<mars> "christmas-trees-for-sale-72"  Huh?
<MTecknology> mars: don't delete me; I'm legit
<MTecknology> regardless of what google says
<mars> edge.launchpad.net/~christmas-trees-for-sale72
<MTecknology> mars: HTTP/1.1 302 Moved Temporarily
<MTecknology>  goes to google.com
<mars> now how the heck did that get in there.  Was it part of the same operation that is generating the pharma-spam?  Or some independent job?
<jml> good morning Launchpadders
<mwhudson> jml: good morning
<jml> mwhudson, hi
<jml> *groan*
<jml> hmm
<jml> netsplits suck
<lifeless> its a DDOS
<jml> yeah
<jml> doesn't make it suck less.
#launchpad-dev 2009-12-16
<MTecknology> jml: at least it seems like it's on the decline some
<MTecknology> jml: some really bored people that have probably never been laid and never will be....
<jml> what's the test plan home page?
<jml> never mind
<jml> (I'd like to know what it is, but at least now I've done my QA item)
<jml> I'm really looking forward to landing my 'hot bugs' patch
<spm> jml: you should perhaps wear your hot pants when you do so? get in the groove of the moment.
<jml> spm, good call.
<jml> spm, I'll take photos for you.
<spm> jml: on this one, I'm prepared to believe *without* photos. a rare exception.
<jml> spm, heh.
<jml> I often want to file multiple bugs on a project at once
<jml> I wish Launchpad made that easier.
<poolie> mm
<jml> I'm filing stacks of usability bugs on 'gtg'
<jml> which means filing a bug, hitting "End", looking for the "Report another bug" thing and then clicking it.
<jml> I filed a bug about this years ago, of course.
<jml> since my previous bug tracker was trac, which makes this quite easy.
 * mwhudson noms
<MTecknology> I hate trac...
<MTecknology> jml: just do Back; change details if you know it wasn't reported...
<jml> MTecknology, I hate trac too, but it still does some stuff better than Launchpad.
<MTecknology> jml: You ever try Mantis?
<jml> MTecknology, maybe once, but never for long and not enough to remember it.
<jml> thumper, hey, I thought code review comments would get permalinks for free
 * jml lunching
<poolie> jml: what's gtg?
<poolie> btw unless it's just me (which it may be) https://bugs.edge.launchpad.net/malone/+bug/471328 seems pretty critical
<mup> Bug #471328: Also affects search error - too many matches <bug-page> <ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/471328>
<jml> poolie, "Getting Things GNOME"
<poolie> you can't choose bzr or bzr-svn from a project picker
<jml> poolie, it's a todo thing that has potential, a fairly active community and not the greatest code.
<jml> poolie, I'm living in constant tension between writing my own and helping them with theirs.
<poolie> "price: Free" :-)
<poolie> thanks ubuntu :)
<jml> poolie, I'm not sure how that could be a new bug
<jml> poolie, well, at least new to this cycle
 * poolie plays on staging
<poolie> well
<poolie> i didn't see it before
<poolie> i would have noticed because i do reassign from bzr to plugins pretty often
<poolie> maybe it's transient xmlrpc failure?
<jml> poolie, it was filed in early november, so it'd have to affect production.
 * poolie tests
<jml> poolie, anyway, it does seem pretty serious. Given that all of the bugs guys are afk atm, I'm not sure of the best way to escalate it.
<poolie> let me try to reproduce it again
<jml> I mean, other than me actually fixing the bug.
<poolie> :)
<poolie> hm on https://bugs.staging.launchpad.net/bzr/+bug/385879 i don't get that problem but just a stuck spinner
<mup> Bug #385879: EOL filter only applied to files when first checked out <content-filtering> <Bazaar:Fix Released by mbp> <Bazaar 2.0:Fix Released by ian-clatworthy> <https://launchpad.net/bugs/385879>
<poolie> oh when i tried a second time i did get "undefined"
<poolie> so it may be intermittent?
<jml> sounds plausible
<jml> mwhudson, hi
<mwhudson> jml: hi
<jml> mwhudson, I was wondering, now that we almost have a fully working bazaar.edge and we have a bzr buildbot...
<jml> mwhudson, how much of a pain would it be to automatically land working bzr updates?
<jml> mwhudson, and do you think that would be a good idea.
<jml> mwhudson, feel free to tell me to go away -- I'm only asking out of curiosity.
<mwhudson> jml: i guess it wouldn't make sense to go on getting bzr from an egg in this case
<jml> mwhudson, but eggs are teh future.
<mwhudson> jml: in fact it flies rather thoroughly in the face of the "use releases" direction buildout consciously pushes us in
<mwhudson> jml: that doesn't mean it's a bad thing though
<mwhudson> necessarily
<mwhudson> jml: having an autoupdated thingy for bazaar.edge but keeping prod using releases would be nice
<mwhudson> a bit harder to arrange though i guess
<jml> mwhudson, I was typing the same thing, more or less :)
<mwhudson> i guess there might be issues with new formats
<mwhudson> you could push a new format to baz.edge but the scanner wouldn't be able to read it
<mwhudson> well, i guess tests fail when a new format is added
<jml> mwhudson, don't we already have that problem?
<mwhudson> jml: yes
<mwhudson> well, modulo what i just said
<jml> mwhudson, yeah, but even without edge, people can still push up formats that we don't support.
<mwhudson> jml: true i guess
<mwhudson> maybe only over sftp
<jml> right.
<jml> mwhudson, anyway, it's something to keep in the back of the mind. getting it set up might save us some boring maintenance work, and give bzr better a tighter feedback loop for new server-side behaviours.
<jml> (maybe we should get poolie to do it!)
<poolie> saying the word 'egg' requires fighting robert
<mwhudson> better than fighting a bear i guess
<poolie> questionable
<poolie> more seriously, we do have nightly debs
<poolie> why not use them+
<poolie> s/+/?
<wgrant> LP has a vendetta against using packages or anything that isn't woefully insecure.
<jml> poolie, I think this is one of those cases where whoever wants to get this done will need to possess a will of iron, regardless of implementation details
<jml> poolie, and that given said iron will, they get to do it however they want.
<poolie> the good thing is i do have a will of iron
<poolie> the bad bit is it's iron filings
<jml> hahaha
<poolie> os
<poolie> so
<poolie> hm
<poolie> i may try to get it running against the system bzrlib
<poolie> not your own copy
<jml> poolie, one of the things about using nightlies to consider is that the buildbot doesn't run against them.
<jml> poolie, and we only want to update if the tests pass -- it is still a live system running against production data.
 * jml afk
<mwhudson> ok, i didn't expect this: http://pastebin.ubuntu.com/342383/
<mwhudson> (i guess i need to run it with ./bin/py or python2.5)
<wgrant> spm: buildd-manager is broken.
<jml> mwhudson, nice
 * jml does some illicit programming
<jml> don't tell anyone
<wgrant> https://edge.launchpad.net/~ubuntuone-hackers/+archive/ppa/+build/1129031 is assigned to a builder and utterly broken. I wonder if that is related.
<spm> wgrant: I assume that's not a statement of observation; rather a pls fix? :-)
<wgrant> spm: Yes please.
<spm> hrm. oddness. going for a bounce on it.
<wgrant> It might help, but there are some broken build records around...
<mwhudson> i wonder if testcasewithfactory should flush the active stores in it's tearDown
<spm> ahh. k. will chase them too. ta.
<jml> someone just commented that, CACHE_DIRECTORY = os.path.expanduser('~/.launchpadlib/cache') might be a bad idea for windows
<jml> does anyone have any good suggestions for what should be used instead?
<jml> this is for a launchpadlib cache
<spm> wgrant: out of not-idle curisoity, have you considered becomming our nagios system? :-P
 * mwhudson finds that all his tests so far violate database constraints, but none of them actually issued any queries
<mwhudson> anyway, /me eods more or less
<mwhudson> biab
<wgrant> spm: Ah, that build is fixed now. I wonder if the restart did that.
<spm> haven't been able to get the (*&^(*&^ing thing to stop yet.
<spm> I do so love issuing "stop" scripts and have them ignored
<wgrant> spm: SIGTERM kills it in not too long for me...
<wgrant> Hmm.
<wgrant> Interesting.
<spm> yeah - but there are "right" ways. I suspect the kill and kill with prejudice are in the immediate futuire here
<wgrant> I don't see why an attempt was made to build https://edge.launchpad.net/~ubuntuone-hackers/+archive/ppa/+build/1129031 at all -- it's five months old, and for a disabled archive.
<spm> 2009-12-16 04:37:04+0000 [-] Received SIGTERM, shutting down. <== LIES!!!
<wgrant> I'
<wgrant> I've not worked out why it doesn't just die quickly.
<spm> it's pretty effing annoying...
<spm> 2009-12-16 04:43:08+0000 [-] Server Shut Down.
<spm> awesome. nice and fast.
<wgrant> Is it back up yet?
<wgrant> Hum.
<wgrant> That build oopses sometimes, but not others. That is slightly concerning.
<spm> yup. all looks "normal". for values of...
<wgrant> Much better. Thanks.
<wgrant> Any idea what killed it?
<spm> the passage of time...
<spm> I tried 2 'stops', one straight up kill; it actually stopped on it's own a few secs before I was going to -9 it.
<wgrant> I mean, what broke it?
<spm> either way. wrong. bug 497282
<mup> Bug #497282: buildd-manager is reluctant on shutting down in a timely manner <Soyuz:New> <https://launchpad.net/bugs/497282>
<spm> no idea.
<spm> it looked like it was havng issues with one of the hosts. but...
<spm> wgrant: what were the symptoms you were seeing that showed it as broken?
<wgrant> bohrium was hung as usual, but that happened 10 hours before the breakage.
<wgrant> spm: No build dispatched lately.
<spm> kk
<wgrant> https://edge.launchpad.net/~ubuntuone-hackers/+archive/ppa/+build/1129031 was also sitting on /builders, assigned to a builder but with status 'Needs build'
<wgrant> That doesn't make sense.
<wgrant> And now it OOPSes intermittently like OOPS-1446ED192
<spm> weird
<wgrant> It seems like it was pretty effectively broken, since it also wasn't updating the status of builds that were in progress.
<poolie> jml, i think the answer to your 'does lplib follow a fixed release schedule' is 'no' :)
<jml> poolie, it's _really_ frustrating.
<jml> poolie, trunk has some stuff that'll make bzrlib integration much easier.
<jml> well, a little easier.
<wgrant> That problematic build just got hit again. I wonder if things have jammed up already.
<spm> wgrant: aaaah. this may be a bug we've been tracking
<wgrant> Yeah, it is.
<wgrant> For now, kill that build and restart, I guess.
<spm> https://bugs.edge.launchpad.net/soyuz/+bug/496574
<mup> Bug #496574: buildd-manager fails to deal with "Fault 8002" errors <buildd-manager> <soyuz-build> <Soyuz:Triaged> <https://launchpad.net/bugs/496574>
<spm> and there was another....
<wgrant> Oh, could be that, I guess.
<spm> meh. can't find it. the gist is that something in the soyuz part is borked. this has been *flooding* us with OOPSes. that you saw that oops, (as did I on a 2nd attempt) suggests that could be it.
<wgrant> That build has hit a builder again, but other builds seem to still be working.
<wgrant> So maybe the two horrible brokennesses are unrelated.
<jml> mwhudson, where does your lazy-import-supporting pyflakes live?
 * jml gone
<adiroiban> henninge: hi. do you have 5 minutes to answer a questions? :)
<adiroiban> here is the code http://paste.ubuntu.com/342540/
<henninge> adiroiban: Hi!
 * henninge looks
<adiroiban> i would like to get rid of the permission checking from model
<adiroiban> and only have them in security.py
<adiroiban> also to get rid of permission_helper.py
<adiroiban> as those checks are also implemented in security.py
<henninge> adiroiban: I just created permission_helper.py last cycle
<henninge> adiroiban: and I actually have a branch that extends it
<henninge> adiroiban: I discussed this at length at the Dallas sprint
<adiroiban> henninge: why not have that logic in security.py ?
<henninge> adiroiban: because it is needed in the model, too.
<henninge> adiroiban: the zope security system will only give you control over access to attributes.
<adiroiban> but we can import the class from security.py
<henninge> adiroiban: which class?
<adiroiban> AdminTranslationImportQueueEntry
<adiroiban> like the last example from the pastebin
<henninge> but that one still uses is_admin_or_rosetta_expert. Where would that come from.
<henninge> ?
<henninge> adiroiban: The challange is to control which values the parameters of a method can take in relation to the user that is setting it.
<adiroiban> henninge: true. it just replaces the self.isUbuntuAndIsUserTranslationGroupOwner(user)
<adiroiban> and similar , we can use OnlyRosettaExpertsAndAdmins
<henninge> adiroiban: To me, security.py is our interface to the zope policy. But we have more policy than that.
<adiroiban> henninge: yes. but you can do the same in lib/canonical/launchpad/security.py
<adiroiban> but with permission_helper.py, we will end up with code duplicated in security.py
<adiroiban> as we already have a check similar to is_admin_or_rosetta_expert , in security.py
<henninge> the idea was to move the code out of security.py to make it usable in model code
<henninge> adiroiban: yes, and that check should use is_admin_or_rosetta_expert.
<adiroiban> but then we still need that code for the view
<adiroiban> yep
<adiroiban> ok
<adiroiban> then reference security_helper from security.py
<adiroiban> that is also ok
<henninge> sorry, which code do we need for the view
<henninge> adiroiban: it does!
<henninge> adiroiban: security has just not been converted to use these helpers
<henninge> completely
<henninge> security.py, I mean
<adiroiban> henninge: yes. I see know.
<henninge> adiroiban: may be permission_helpers is the wrong term
<adiroiban> then I should implemenet all the translations security logic in security_helper.py
<adiroiban> and then use that code from security.py
<henninge> adiroiban: yes, and call it permission_checkers.py
<henninge> adiroiban: I am moving the file out of translations into services anyway.
<henninge> adiroiban: I will do the renaming. Can you wait until my branch is done? This should happen this morning.
<adiroiban> henninge: ok. It make more sense now :)
<adiroiban> yes. no hurry
<henninge> Great ;)
<henninge> adiroiban: actually, danilo wants me to de done with it ... ;)
<adiroiban> henninge: but why is not a good idea to use security.py classes in the model ?
<henninge> adiroiban: I have not thought much about that but my first impression is this:
<henninge> security.py is our interface to the permissions mentioned in the zcml
<danilos> I don't think it would such a bad idea, especially since they define a permission for a user to a model class
<henninge> so, if there is a <require> statement on a class that requires a certain permission, there will be a checker or it in security.py
<henninge> but, as I mentioned before, that does not give us finer control over method parameters.
<henninge> so we have to add checks in the model about parameter values.
<adiroiban> @enabled_with_permission('launchpad.Admin') can only be used in the view?
<danilos> henninge, right, but those checks can be gotten by using different privileges... ie. if has_perm(user, 'launchpad.Admin'): allow this
<henninge> If we want to use the classes from security.py for this, we'd have to add classes there that have no relation to the zcml
<danilos> adiroiban, you can define in ZML what attributed/methods are accessible for each permission, but there are cases where you want to allow only certain values to be passed in by certain users (i.e. henning has an example of setStatus on translation import queue, where only admins can set any status)
<henninge> adiroiban: on, it is used on the interface, thus on the model. But it only applies to attributes.
<adiroiban> you know better. I see security.py as the place to store all security logic
<henninge> adiroiban: yes, I think it is a question of what we define security.py to be and what is supposed to be in there.
<adiroiban> danilos: yep. I saw that code. and maybe we can have those values in a different attribute
<adiroiban> :)
<adiroiban> or just do the check in security.py
<henninge> adiroiban: I thought about that
<henninge> that would have meant to have seperate "setStatusApproved", "setStatusFailed" etc
<henninge> or "setStatusAdmin".
<henninge> I was advised against that at the sprint.
<adiroiban> henninge: ok. np.
<henninge> just saying
<adiroiban> then I will add the logic in security_helper.py
<adiroiban> and make use of it from security.py and from the model
<henninge> adiroiban: that is how I meant it to be
<henninge> adiroiban, danilos: but maybe we should take that discussion to the list as this is a general discussion about the use of security.py.
<danilos> henninge, yep, sounds like a good idea
<henninge> ok, I will do that
<adiroiban> henninge: i think that is ok to move code away from security.py ... as it can become a big file, but I think we should still use the security.py interface from the model. Instead of security_helper.is_admin_or_rosetta_expert() use security.OnlyRosettaExpertsAndAdmins.checkAuthenticated
<adiroiban> or create a new permission interface
<adiroiban> but I will wait for your branch
<adiroiban> and the discussion from the ml
<henninge> adiroiban: yes, the discussion will be of most interest as I don't want some-one else to come along next cycle or so and change it all over again. We need some consensus here across the team.
<adiroiban> henninge: ok.
<henninge> adiroiban: I just realised you cannot just use " security.OnlyRosettaExpertsAndAdmins.checkAuthenticated" from model code as checkAuthenticated is not a class method.
<henninge> adiroiban: you'd have to instantiate OnlyRosettaExpertsAndAdmins first.
<adiroiban> yes. I'm doing something like this OnlyRosettaExpertsAndAdmins( self).checkAuthenticated(user)
<henninge> I see
<adiroiban> I'm running now all tests
<adiroiban> and see if there are problems
<mrevell> morning
<maxb> Hi, I have a trivial lp-dev-deps change, if someone has a moment: https://code.edge.launchpad.net/~maxb/meta-lp-deps/no-pullparser/+merge/16232
<jtv> hi mrevell!  you have nag mail.  :-)
<mrevell> jtv, thanks, I see it :) Will get to it today
 * jtv cheers mrevell on
<mrevell> heh
<stub> henninge: A long, long time ago we specced out a system of crowds, where you would construct a crowd by combining teams and peoples. You could then query the crowd to see if a person was or was not a member of that crowd. Designed to be DB friendly, but also would make nicer things like is_admin_or_rosetta_export()
<henninge> stub: sounds cool
<jtv> al-maisan_: do you know if there are any current plans to move Build.sourcepackagerelease (and the stuff that relies on it) into BuildPackageJob?
<al-maisan_> jtv: on the phone, ttyl
<jtv> pl
<jtv> ok
<stub> henninge: The crowd was really just a list of Person (or even just their ids), and the 'is in' and 'isn't in' operations just single queries against the TeamParticipation table.
<deryck> Morning, all.
<al-maisan_> jtv: AFAIU there are no plans to relocate Build properties into BuildPackageJob
<al-maisan_> jtv: why do you ask :)
<jtv> al-maisan_: my question was misguided.  I don't need a Build for our prospective BuildFarmJob implementation, do I?
<jtv> hi deryck
<al-maisan_> jtv: you probably don't .. what data/properties do you envision being in your "TranslationFarmJob" ?
<jtv> al-maisan_: a reference to the branch it should gets its data from.
<al-maisan_> jtv: in that case it would seem there's no need to reference a `Build`
<jtv> bigjools, is this the right interpretation of you described on the phone?  http://rookery.canonical.com/~jtv/farmjob.dia
<bigjools> can you turn that into a PNG?
<jtv> bigjools: with the power of free software, yes I can!
<jtv> bigjools: http://rookery.canonical.com/~jtv/farmjob.png
<jtv> bigjools: and then I gather our BuildFarmJobBehavior would receive a BuildQueue entry to work on
<bigjools> jtv: it's a bit corrupted
<bigjools> is the << Iwhatever >> part supposed to be an "implements" ?
<jtv> bigjools: yes
<jtv> although maybe the base class does that... not too important I guess :)
<bigjools> jtv: ok it's wrong on your translations one then :)
<jtv> bigjools: ahhh, they're both wrong actually: meant to say IBuildFarmJob on both
<bigjools> jtv: that's the basic gist of things, yes.  Although you will need content classes and builder behaviours as well.
<jtv> bigjools: I knew about the behaviors but left them out here because I'm focusing on db schema right now... what are the content classes for?
<bigjools> jtv: scratch that you already have one, sorry
<jtv> ah, branch :)
<bigjools> bbiab
<danilos> adiroiban, hi, have you done the QA on https://bugs.edge.launchpad.net/rosetta/+bug/406477 as well?
<mup> Bug #406477: Need more permissions on admin template pages for ubuntu-translations-coordinators <qa-needstesting> <Launchpad Translations:Fix Committed by adiroiban> <https://launchpad.net/bugs/406477>
 * adiroiban looking
<maxb> salgado: Hi. If you're around, I'm going to be doing a lp-deps upload today, if you wanted to put the ssh removal in too
<salgado> hi maxb.  I'll try to get that in before you upload, but if it's not there by the time you get to it, don't worry
<salgado> unless you want to do it yourself and get the blame^Wcredit for it
<jtv> adiroiban: need help with that Q/A?  I have an unprivileged account for comparison
<adiroiban> jtv: yes. please test this link https://translations.launchpad.net/ubuntu/karmic/+source/kdepim/+pots/kxforms/+admin
<adiroiban> do I need to create a test account on edge or staging is also ok?
<adiroiban> I'm new to LP qa process
<jtv> adiroiban: staging should be fine, provided you're sure that your branch has been rolled out there
<adiroiban> i can not access staging right now
<GOTTMODUS> Hi everyone
<GOTTMODUS> to wich user is the svn import branch review going?
<jtv> no need to shout!  :)
<GOTTMODUS> i cant see ist from the default users site
<GOTTMODUS> ??
<GOTTMODUS> where did i shout?
<jtv> gottmodus: your nick
<GOTTMODUS> ah ok
<jtv> adiroiban: same here...  and getting a "not found" for that page on edge.  We're expecting an "access denied," no?
<adiroiban> jtv: no. I should be ok
<jtv> gottmodus: the joke does work better with all caps, but it's a bit noisy nonetheless :-)
<adiroiban> the Navigation is breacking that chain
<adiroiban> jtv: I have tested as an UTC and everything is ok
<jtv> adiroiban: it's a bit weird though that it says "there's no page with this address in Launchpad" when it should say that I'm not allowed to access the page
<adiroiban> jtv: that was the same case in the previous implementation
<jtv> adiroiban: in that case, no worry here.  :)
<adiroiban> jtv: you can try on lp.net
<adiroiban> jtv: i disabled my redirection for edge, and tested on stable and I also got a page not found error
<adiroiban> jtv: there should not be a problem, since the user does not see the admin link
<adiroiban> jtv: actually, neither do UTCs ... as the +templates pages times out
<jtv> :(
<wgrant> bigjools: Do you know if dpkg + buildds are upgraded for tomorrow's rollout?
<jtv> adiroiban: from an unprivileged account I see neither the link to that page, nor the same for another (non-deactivated) template.  So looks good to me.
<bigjools> wgrant: in progress, but not yet AFAIK
<adiroiban> danilos: I don't know what to say. I have tested on my computer, and tested on edge as UTC. Everything was ok. Also the test was written to cover an anonymous, a logged in user and an Admin
<danilos> adiroiban, right, it was mostly to do some QA, since you haven't marked it as qa-ok in the bug... also, QA is usually done on edge and/or staging, you might be surprised, but many things break only when we hit production database :)
<jtv> adiroiban: did D. find something wrong with the branch?
<adiroiban> jtv: I don't know. He approved it
<jtv> just confusion over the qa process then I guess?
<adiroiban> jtv: yes. I did not tag it with qa-ok
<adiroiban> but only qa-needtesting
<jtv> We're weird that way. :)  It's a pretty nice system actually.
<adiroiban> I was thinking that I should mark qa-needtesting, whey I need someone else to also look into it
<jtv> adiroiban: it should happen automatically (though we did have some breakage recently)
<adiroiban> jtv: well I don't know how things should be going... so as always... I am very confused
<jtv> adiroiban: need a summary?
<adiroiban> jtv: or a wikipage ? :)
<jtv> adiroiban: don't think we have one for this, since it's still only on an experimental scale...
<adiroiban> jtv: so when my branch is merged with a fixes=lp:222 the bug report is updated to fix-comminted and qa-needtesting tag is added?
<adiroiban> and then I have to test the commit on devel/staging and set it to qa-ok ?
<jtv> adiroiban: the bug status is not updated to "fix committed" iirc, but otherwise, yes
<jtv> And you try to Q/A on staging or edge, not just devel because of the unexpected differences that real-life data can make (see what D. said above)
<adiroiban> jtv: I have tested all my bugs targeted for this release and tagged them with fix-ok
<jtv> adiroiban: we usually just verb "QA" to avoid confusion with the test suite.
<adiroiban> jtv: ok. thanks for the jargon update :)
<jtv> Glad you don't mind; I know I'm pedantic but who knows what confusion it may save later :)
<adiroiban> well I still have problems with page tests / stories / functional tests ... etc
<beuno> BjornT, bug 497405
<beuno> \o/
<mup> Bug #497405: Run Windmill tests by default <story-windmill-layer> <Launchpad Foundations:In Progress by bjornt> <https://launchpad.net/bugs/497405>
<BjornT> beuno: i'm really looking forward not having to monitor the jscheck builder constantly, and chase people to fix things. better to have buildbot take care of that :)
<beuno> BjornT, yeah, and I'm happy to never have to stare at flacoste when we mention tests slowing people down to develop awesomeness
<salgado> BjornT, can we use utilities/ec2 to run the windmill tests?
<BjornT> salgado: yes. if you grab my branch, you can.
<salgado> BjornT, wow, I was expecting a "not yet" answer.  this is great news
<salgado> I might even review your branch
<beuno> joey, have you changed your default font settings?
<joey> that was fast :-)
<joey> beuno: no
<joey> should I?
<beuno> joey, that that I don't trust you
<beuno> but
<beuno> what's the values?  :)
<beuno> what are
<beuno> serif, 16, monospace, 12?
<beuno> joey, Preferences > Content > Fonts / Advanced
<joey> Loading FF. Was in chrome
<joey> can't find the option in chrome
<joey> 16 and 12 in FF
<beuno> hm
<beuno> and you see all bug pages with the side portlets at the bottom?
<joey> yes
<joey> on both FF and chromium
<joey> started about a week ago
<joey> or more
<joey> I was out for a few weeks
<joey> it was like that when I came back to work
<beuno> joey, and your fonts
<beuno> system fonts
<beuno> in appearance
<beuno> are they set at 10?
<joey> afaict, nothing has changed
<joey> it happens on two different computers
<joey> my desktop and my netbook
 * joey looks at his sys fonts
<joey> as soon as I figure out where they are. I'm not a font junkie
<joey> all listed as 10 on my desktop
<beuno> I'm running out of ideas
<beuno> WFM (and I think, in general)
<joey> try css :-)
<beuno> one last test
<beuno> can you start firefox with a clean profile, no addons?
<joey> sure
<beuno> ok
<beuno> I can reproduce now
<beuno> joey, not needed
<beuno> if you have your browser wide enough
<beuno> it happens
<joey> yeah I JUST now noticed that
<beuno> you probably have 1600+ res, right?
<joey> 1920 on the desktop, 1280 on the netbook
<joey> the clean profile I opened in a window and it was correct, and when I maximized, poof
<beuno> I can't reproduce it in 1280, but now we know there's an issue there
<beuno> sinzui, where you aware ^>
<sinzui> beuno: danilo remarked upon it. I could not reproduce it to identify if we need to add a css rule or markup to base-layout.pt
<joey> beuno: a side note... all my computers are widescreens
<joey> beuno: so that may be why I see it at other resolutions
<beuno> joey, yeah, very likely
<beuno> if the width of the sidebar is seto to 17%
<beuno> it doesn't break
<beuno> joey, bug 493518
<mup> Bug #493518: Side portlet moved again below the main content on wide-screen displays (1920x1200) <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/493518>
<Pilky> beuno: if it helps, Safari 4 doesn't have the issue, I can spread my browser window across 2 displays and it works fine
<beuno> Pilky, interesting...
<Pilky> let me just try firefox
<Pilky> oh wait, does happen in safari
<Pilky> had the wrong bug visible
<beuno> ok, that's good news then  :)
<danilos> joey, beuno, sinzui: there was a bug along those lines before, which I fixed, but it's back with a vengeance when #maincontent CSS was removed: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/435346
<mup> Bug #435346: 3.0 interface overflows sidebars and causes horizontal scrolling if you manually change browser font sizes <qa-ok> <Launchpad Foundations:Fix Released by danilo> <https://launchpad.net/bugs/435346>
<james_w> how does a branch become owned by ~vcs-imports?
<james_w> I just requested an import and it is under ~james-w, I assumed it went under the former
<cody-somerville> james_w, it shouldn't be under ~james-w unless thats changed.
<cody-somerville> james_w, whats the branch?
<james_w> ~james-w/alsa-driver/trunk
<salgado> sinzui, is there any reason why registry/templates/object-timeline-graph.pt doesn't use the new stylesheet?
<sinzui> salgado: yes it is in an iframe. It should be just canvas and thus not use any css
<sinzui> salgado: if the template now has markup displayed to users, we need to reconsider this
<salgado> sinzui, oh, right, my question was actually: "why does it use the old stylesheet instead of the new one".
<salgado> should it use none?  I'd be more than happy to change that
<sinzui> salgado: I do not think it needs css, I see every element uses a style attribute to disable the default. If I am wrong, it must use style-3.0.css.
<salgado> ok, I'll remove the launchpad-stylesheet macro from it and see what happens
<mwhudson> good morning
<thumper> morning
<thumper> mwhudson: I changed the url for https://code.edge.launchpad.net/~vcs-imports/analysesi/main, but the update still looks at the old location
<thumper> mwhudson: I'm pretty sure it is the same repository, just at a different location
<thumper> mwhudson: how do we blow away the existing checkout?
* mthaddon changed the topic of #launchpad-dev to: Launchpad will be down/in read-only from 22:00 UTC until 23:00 UTC for a code update | Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only (https://dev.launchpad.net/CurrentRolloutBlockers, RM: danilos) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<mwhudson> thumper: oh for CVS its completely terrible
<mwhudson> thumper: https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/UpdateCodeImportLocation
<mwhudson> thumper: needs a LOSA :(
<thumper> ok
<thumper> mwhudson: ew
<mwhudson> thumper: yes
 * mwhudson fights with storm
<sinzui> bac: what are your plans for Bug 447418
<mup> Bug #447418: Timeout with large milestone with lots of private bugs <timeout> <Launchpad Registry:In Progress by bac> <https://launchpad.net/bugs/447418>
<bac> sinzui: don't know yet
<bac> sinzui: just started looking.
<thekorn> intellectronica, hi, thanks to you guidance last evening I managed to fix bug 497108, I attached the branch and a screenshot of the result.
<mup> Bug #497108: show changes to the affected product field of a task in the interleaved activity log <Launchpad Bugs:New> <https://launchpad.net/bugs/497108>
<thekorn> what's the next step?
<intellectronica> thekorn: that's wonderful! thank you so much for taking on this work
<intellectronica> thekorn: go ahead and propose the branch for merging. i can review it now. the tree is closed right now (release in 1h) but we can land this as soon as it opens again
<thekorn> ok
<thekorn> intellectronica, created merge proposal at https://code.edge.launchpad.net/~thekorn/launchpad/affects_in_interleaved_activitylog/+merge/16261
<sinzui> bac: sorry, I was called away to get children from the local institution
<intellectronica> thekorn: huh, that's a very small diff :)
<thekorn> intellectronica, yes, that surprised me too
<intellectronica> thekorn: we do need to add a test, though, to demonstrate the new behaviour
<sinzui> bac: I was pondering (this weekend) of converting. the bug table to an ajax call like subscribers. I need to see all the bugs when looking at a milestone, so I am not keen to use a batch navigator.
<thekorn> intellectronica, that's what I thought, can you point me to a place where I should add them
<thekorn> should I add a doctest or a unittest
<sinzui> bac: my plan may be total rubbish because the SQL time on the page is very good as I recall. Most of the time was in python, and I think it is in the status counts, the same code we disabled on series because they caused timeout.
<intellectronica> thekorn: there should be a story test somewhere, looking for it
 * sinzui suspects something with enums, but he really does not know wtf is wrong with the code
<thekorn> intellectronica, and most importantly: is there a way to only run relevant test, and not this huge testsuite
<intellectronica> thekorn: of course :) `bin/test -vv -t $pattern` where pattern is a regex matching the paths of the tests you'd like to run
<thekorn> ok, so stories are doctests, nice
<sinzui> thekorn: that is almost nice. most stories are mediocre function tests instead of user integration tests.
<intellectronica> thekorn: see lib/lp/bugs/stories/bugs/xx-bug-activity.txt
<intellectronica> thekorn: to run it, go `bin/test -vv -t xx-bug-activity.txt`
<thekorn> ok, lets try to compile a test
<sinzui> I think I just approve the last two mailing lists that will ever need approval
<sinzui> I think I just approved the last two mailing lists that will ever need approval
<bac> sinzui: yeah, as soon as i started looking at this problem i ran into our friend get_status_counts
<sinzui> It looks so simple to loop of a fixed set of object that are cached. yet I suspect something is not cached, or worse, the access of status is very costly
<wgrant> When is 3.2.1/3.1.13/whatever it is?
<sinzui> wgrant: I think we are going to rename the series 10.05 and the milestone will be named 01 after the month
<sinzui> wgrant: We are trying to use series like we tell other projects
<al-maisan_> sinzui: the series?
<wgrant> sinzui: But that's barely useful, since LP doesn't really release, and has no use for series.
<sinzui> al-maisan_: yes rename the series to make sense, and the milestone to be the month
<sinzui> or the step
<wgrant> Anyway, I don't really care about how it's spelt -- what is the targetted release date?
<sinzui> wgrant: it does not make sense to ever create a new series since we do not release, but since a series sets the focus of development, and we mean to be very focused for this series, we would be insane to create another series
<sinzui> I wish I knew. I took a guess for the registry
<wgrant> Ah. And when is ~launchpad on vacation?
<thumper> wgrant: for me, real soon now :)
<sinzui> I think we stop getting paid to work on the 25 of December and most of us return on  January 2
<thumper> sinzui: 2nd is a Saturday
<wgrant> Ah, so just the week. Great.
<sinzui> thumper: true.
<bac> thumper: you don't get paid for saturdays?
<sinzui> I hope not, otherwise management would tell me to stop woring on what I want to work on
<sinzui> I started a branch to add karma to registry actions (with the ulterior motive to wire all the events so that I can send notifications and create sparklines). I hope to finish over the holidays
* mbarnett changed the topic of #launchpad-dev to: Launchpad will be down/in read-only from 22:00 UTC until 23:59 UTC for a code update | Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only (https://dev.launchpad.net/CurrentRolloutBlockers, RM: danilos) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
* mbarnett changed the topic of #launchpad-dev to: Launchpad will be down/in read-only from 22:00 UTC until 01:00 UTC for a code update | Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only (https://dev.launchpad.net/CurrentRolloutBlockers, RM: danilos) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
#launchpad-dev 2009-12-17
 * mwhudson nom nom nom
* mbarnett changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 3.1.12 | PQM is closed; RC only (https://dev.launchpad.net/CurrentRolloutBlockers, RM: danilos) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<wgrant> So, the source format 3.0 code is there now. But do the Soyuz machines have the new dpkg, and are the buildds running lp-buildd 54?
<bigjools-afk> wgrant: no and yes
<wgrant> bigjools-afk: Argh and yay.
<bigjools-afk> it will happen in due course then we can turn on the wotsit selection
<wgrant> Right.
<bigjools-afk> bedtime, g'night
<wgrant> Night.
<thumper> mwhudson: so... when are we going to flick the switch on the "never succeeded" svn branches?
<jml> mwhudson, where is your pyflakes branch that makes working with lazy imports not be terrible?
<jml> mwhudson, and would you like me to land it to trunk for you?
<mwhudson> thumper: i guess there's little harm in flicking the switch now
<mwhudson> thumper: we shouldn't mass retry them all
<thumper> mwhudson: limit 5 each time?
<thumper> mwhudson: how many where there?
<mwhudson> jml: https://code.edge.launchpad.net/~mwhudson/pyflakes/support-lazy-imports
<jml> mwhudson, thank you.
<mwhudson> jml: if you like; i've never proposed it as it seems very bzrlib specific
<thumper> jml: want a quick chat now? or after you have lunch?
<jml> thumper, I just had lunch.
<jml> thumper, it's all I can do to prevent my parent's from serving it before breakfast.
<mwhudson> thumper: well, by flicking the switch i meant "update codeimport set rcstype = ..."
<jml> s/'//
<thumper> mwhudson: yeah, that's what I was thinking too
<thumper> mwhudson: add a "limit 5" on the end of the query
<mwhudson> thumper: not sure off hand about how to retry them with straight sql
<mwhudson> thumper: most of the never-succeeded ones are marked FAILED, they won't be retried without intervention
<thumper> mwhudson: set the status to reviewed
<thumper> mwhudson: ah, bugger
<thumper> mwhudson: the code adds the job when it becomes reviewed
<mwhudson> thumper: yeah
<thumper> mwhudson: so direct editing of reviewed would be a 'bad idea'
<mwhudson> i think it's probably just "insert into codeimport where ..." but yeah
<mwhudson> *into codeimportjob rather
<mwhudson> "some care required"
<jtv> hi mwhudson!  you've been working on the recipe builds, right?
<spm> jtv: howdy! no long no see!
<jtv> spm: g'day!
<jtv> spm: how's life?  drop bear attacks not too bad?
<jtv> spm: I've been doing some traveling...  no mobile coverage most of the time, let alone internet.  feel like a different person
<spm> heh
<mwhudson> jtv: i have, for my sins
 * jtv resists to urge to ask about those :)
<jtv> mwhudson: I was wondering... how does one go about writing tests for that sort of thing?
<mwhudson> jtv: i don't really know, i haven't got to that point yet
<mwhudson> jtv: just bashing out the content class code and tests for same
<jtv> mwhudson: in our case we separated the "set up an execution environment" work from the "build the payload" work, so I'm facing that part a bit earlier
<mwhudson> jtv: well, i guess the interface to between the build and the master is xml-rpc
<mwhudson> jtv: so you don't need to run the build inside a vm to be able to test that...
<jtv> the last time I did anything with xmlrpc it was to protect against a virus that exploited a PHP implementation... then told the inventor of xml that it was all his fault.
<jtv> but the principle sounds sensible :)
<jtv> mwhudson: that's another part I couldn't figure out easily: where does the payload fit in?  I saw mention of a build() method on the slave, but couldn't quite match it up to an implementation
<mwhudson> jtv: what do you mean by payload?
<jtv> the actual work that we want to do on the slave
<jtv> i.e. our own equivalent of "build the package"
<mwhudson> jtv: code or data?
<jtv> oh, our "build" is a pure data thing
<jtv> very simple compared to the other stuff
<jtv> compared to the other forms of build, I mean
<mwhudson> jtv: so there are two ways you can get data onto a build machine
<mwhudson> (currently)
<jtv> I think we'll be wanting to add a third: check out a publicly available branch
<mwhudson> jtv: as an argument to an xmlrpc method, and a file from the librarian
<mwhudson> jtv: yes indeed
<jtv> I think you're talking about the "cacheFileOnSlave" stuff, right?
<mwhudson> (and ultimately, check out a non-public branch too)
<mwhudson> jtv: yeah
<jtv> for the files that aren't on the public librarian
<mwhudson> jtv: i think "cacheFileOnSlave" applies to the public librarian doesn't it?
<mwhudson> jtv: public librarian files are accessed by sha1sum, restricted librarian files by url which has credentials embedded in it
<jtv> mwhudson: I could easily be wrong, but the buildfarmjobbehavior code I looked at said it was going to the trouble to deal with private files
<mwhudson> jtv: wgrant knows far more than me about this sort of thing btw :-)
 * wgrant looks.
<jtv> I guess the Pacific side of the planet is the place to be for these things  :-)
<jtv> hi wgrant!
<mwhudson> jtv: which code are you looking at?
<jtv> mwhudson: I'm not looking at code now but IIRC there is one existing concrete implementation of IBuildFarmJobBehavior; it has a private method with a name like _cacheFilesOnSlave that does this stuff
 * wgrant tries to work out how this new split works.
<mwhudson> jtv: i haven't looked at this since the soyuz guys moved all the code around
<wgrant> jtv: YOou mean BinaryPackageBuildBehaviour._cachePrivateSourceOnSlave?
<jtv> wgrant: that's our baby
<jtv> but I don't care too much about this part anyway because what I need to do is pass the branch URL to the slave, then have the slave check out the branch
<mwhudson> jtv: in builder.py, i reckon self.slave.ensurepresent is calling some xmlrpc method
<jtv> I see now that we've been talking at cross purposes
<wgrant> mwhudson: It is.
<wgrant> jtv: lib/canonical/buildd/slave.py, def ensurepresent
<wgrant> Er, mwhudson ^^
<mwhudson> wgrant: right
<jtv> ahhhh, and all this time I was looking only in lib/lp/buildmaster and lib/lp/soyuz
<wgrant> jtv: And you should stay there until you really have to leave.
<mwhudson> jtv: oh, you hadn't had the delight of lib/canonical/buildd yet?
<wgrant> lp-buildd is *not* pretty.
 * mwhudson goes away until the screaming becomes loud enough to disturb him again
 * jtv looks worried and puzzled at the contrast between wgrant's and mwhudson's take
<mwhudson> jtv: basically, i guess we'll need to add an xmlrpc method "getbranch" or something
<wgrant> mwhudson: But you're going to be using bzr-builder to grab the branches, aren't you?
<mwhudson> wgrant: *I* am i expect, yes
<jtv> mwhudson: you've been adding slave-side stuff... have you been doing that by extending this part, or by adding new classes somewhere?
<wgrant> So it may prove better to just let the Translations code inside the builder take a branch path in extra_args, check it out itself.
<mwhudson> jtv: i haven't added slave side stuff yet
<jtv> what's bzr-builder btw?
<mwhudson> jtv: wgrant has a point here
<jtv> it rings a bell
<mwhudson> jtv: slave.py defines an abstract class BuildManager
<mwhudson> jtv: debian.py in the same directory defines a subclass DebianBuildManager
<jtv> wgrant: that's what I had in mind, yes...  it'll mean some IS changes, but it's by far the "narrowest" channel I think
<mwhudson> jtv: we'll both be wanting to define new subclasses i think
<jtv> ah, buildmanager was that thing that sounded like it lived master-side but actually managed the build client-side?
<jtv> I mean, slave-side?
<wgrant> jtv: No, no.
<mwhudson> hee hee
<wgrant> There is buildmanager, and buildd-manager.
<wgrant> And slave-scanner, and builddmaster.
<mwhudson> there is buildd-manager an
<jtv> "oh of course, how could I be so _stupid_?"  :-)
<wgrant> buildd-manager and slave-scanner are both instances of this 'builddmaster' concept.
<wgrant> They live on the master. There is only one of them.
<wgrant> buildmanager is part of lp-buildd, and lives on the slave.
<jtv> lp-buildd?  I don't think I've been introduced to that one
<wgrant> jtv: launchpad-buildd is the thing that lurks in lib/canonical/buildd
<mwhudson> jtv: the stuff in lib/canonical/buildd is turned into a debian package called launchpad-buildd (?) which is installed on the build machines
<jtv> but this _does_ sound like buildmanager is the one that sounds like it runs on the master but actually runs on the client, just like I said, no?
<wgrant> It's self-contained, except for tac-related stuff.
<mwhudson> jtv: yeah
<jtv> tactical air command?
<jtv> type approval code?
<wgrant> jtv: Nasty Twisted thing.
<mwhudson> twisted application configuration
<jtv> thrust asymmetry compensation?
<wgrant> Ahh.
<mwhudson> wgrant: better than taps!
<wgrant> mwhudson: Yeeeees.
<jtv> coffee's ready
<jtv> coming back from Laos with a few bags of the stuff was the perfect excuse to get a proper machine
<jtv> hang on
<jtv> ahhh
<jtv> with that taken care of, back to that code :-)
<wgrant> How scheduled is this work?
<wgrant> (ie. do we know what people will be doing in Wellington?)
<jtv> wgrant: still a big unknown afaik
<mwhudson> wgrant: this is exactly what we'll be working on in wellington
<wgrant> mwhudson: 'this' is a very big domain.
<mwhudson> what precisely we will do depends on what we get done between now and then i guess
<mwhudson> wgrant: well yes
<jtv> wgrant: hence our different answers :)
<mwhudson> aiui, the plan is "find something to do.  do it"
<wgrant> But I guess it is closer than I had previously thought, what with Christmas in the way.
<jtv> mwhudson: would "clean up l/c/buildd a bit" be a good point for the agenda there?
<mwhudson> one of the goals on the internal wiki page for the sprint is "# Hack on code like crazed bunnies "
<wgrant> mwhudson: Heh.
<wgrant> jtv: That code is normally hacked by infinity/lamont/whoeveritisnow
<wgrant> But I suspect a lot will need to be done in Wellington.
<jtv> what ever happened to "everyone owns all code"?  :-)
<jtv> getting fresh, freshly-frustrated eyes on code is often what drives good cleanup
<wgrant> jtv: This code primarily lives in a tree outside LP.
<wgrant> Occasionally getting copied back in.
<wgrant> Although that appears to have changed in the past few weeks :D
 * jtv hates hidden dependencies between projects
<mwhudson> jtv: part of it (i guess) is that this code has to run on architectures that launchpad probably doesn't run on
<mwhudson> like hppa
<jtv> fair enough, but does that mean that I'll have to go to this other source tree when I need to work on slave-side code?
<mwhudson> i don't think that's completely settled, but i'd lean to "yes"
<wgrant> lamont was last week doing the latest lp-buildd changes in a normal Launchpad branch, so the days of the separate tree may be over.
<mwhudson> i think that more or less has to happen
<mwhudson> wow, the hotel we're staying in in wellington doesn't get very good reviews on tripadvisor
<mwhudson> (but i think it'll be fine for us actually)
<jtv> mwhudson: the cynic in me is tempted to say "maybe there's just 1 competing hotel and they like to review each other"
<mwhudson> jtv: new zealand is a small country, but there's more than one hotel in it's capital :-)
<mwhudson> or two
<jtv> mwhudson: "oh you're from New Zealand?  Do you know Gerald?"
<wgrant> I presume the sprint itself will be somewhere in the hotel too?
<jml> you presume too much sir!
 * jml goes back to his box
<mwhudson> wgrant: yeah
<wgrant> Oh good, their website wants me to install QuickTime.
<mwhudson> thumper: did you want to talk today?  i guess it's getting late
<thumper> mwhudson: I think tomorrow is better
<thumper> mwhudson: I'm about to make hamburgers for dinner :)
<mwhudson> thumper: ok
<mwhudson> thumper: have fun :)
<jml> wuuu
<jml> adsl2
<mwhudson> jml: feel the bits!
<mwhudson> "DownStream Connection Speed 9829 kbps"
<jml> mwhudson, heh
<mwhudson> i think i'm on adsl2, but on the other hand i'm also in new zealand
<jml> mwhudson, speedtest says 17.5 Mbps
<mwhudson> jml: that's pretty fast
<jml> 0.83 up.
<jml> mwhudson, I may be the only person on this exchange with adsl2
<jml> mwhudson, given that the town has 1,700 people.
<wgrant> Which ISP was crazy enough to put an ADSL2 DSLAM down there?
<spm> internode probably
<jml> yep
<spm> despite being in canberra, I support the fact that they put dslams in remote SA towns. big +1 from me.
<jml> spm, SA?
<wgrant> TAS, wasn't it?
<spm> South Aust
<jml> spm, I'm in even _southier_ Australia
<spm> jml: I know, but they really focus on SA - in particular. for a small company, that's impressive
<jml> spm, oh, right.
<spm> heh. was funny at one event Simon was talking, one of their network ops was seeing all sorts of funky stuff from telstra. just before T went to full ADSL1. was like random testing of lines and stuff.
<jml> in some ways, I'm glad the government didn't split telstra up before privatizing
<jml> every story needs a bad guy.
<spm> :-)
<jml> abentley, with ampoule jobs, what happens if a process dies due to an unexpected error?
<bigjools> mwhudson: around?
<bigjools> or thumper?
<maxb> I think bug 316192 has regressed, should I reopen it or file a new one?
<mup> Bug #316192: bzrlib.plugin.load_plugins() loads system plugins even for non-system bzrlib <Bazaar:Fix Released by vila> <Launchpad Bazaar Integration:Fix Released by thumper> <https://launchpad.net/bugs/316192>
<maxb> I got test failures because LP's 2.1b3 bzr egg doesn't like my bzr-gtk (from .deb) or bzr-rebase (in ~/.bazaar/plugins/)
<adiroiban> hi. I'm having some problems with xpath trying to get the row (tr) element hosting a link. I'm using //table[@id='languagestats']/descendant::a[text()='French']/parent::td/parent::tr
<adiroiban> but it's not getting the right row
<BjornT> maxb: opening a new bug is usually best. it's easier to keep track of things that way. there's value in seeing that this issue was fixed once before.
<maxb> I wondered. OK.
<leonardr> james_w, i'd like to talk to you about your lazr.restfulclient branch
<leonardr> https://code.edge.launchpad.net/~james-w/lazr.restfulclient/fix-caching/+merge/15635
<james_w> hi leonardr
<leonardr> the branch itself looks fine, but i think the process is broken, because it hasn't been landed for a week
<leonardr> and i only noticed it because jml was complaining about a launchpadlib backlog
<leonardr> so: what did you expect would happen after the branch was approved?
<james_w> ah, I can't land it, so abel said that he would
<leonardr> aha
<james_w> it obviously slipped his mind
<james_w> or got rejected or something
<leonardr> it wouldn't have been rejected, it's not pqm-managed
<james_w> this was on IRC, so it's not recorded in the MP
<leonardr> all right. the simplest way to solve the problem would be for me to monitor the review list and push branches that are stuck in the process
<KLondenberg> Hi all .
<KLondenberg> Short question: I'm currently trying to get my private copy auf Launchpad working, including Codehosting and so on. Everything except codehosting seems to work fine. But when I try to push to a lp://dev/  repository, the repository gets created in the root folder (/) of the server launchpad is running on. It fails, even, if this is not done with root rights (for obvious reasons). Any clue...
<KLondenberg> ...what might be wrong, or where I should start looking into ?
<KLondenberg> Apart from that, pushes to that repository don't get picked up by the webapp, even if I "make sync_branches"
<sinzui> Chex: Can you merge lp:~sinzui/launchpad/spam-eggs-bug-495250 on staging? We want to verify this branch for a CP to hinder spammers
<Chex> sinzui: sure, hang on
<Chex> updown is down
<Chex> erf misfire
<intellectronica> andrea-bs: ok, i know why you're getting the cofiguration conflicts
<andrea-bs> intellectronica, great
<intellectronica> andrea-bs: in line 159, you allow everything in ISpecification
<intellectronica> andrea-bs: then in line 188 you try to require permissions for newMessage and setCommentVisibility
<intellectronica> andrea-bs: zope doesn't like that. you can't override attribute configuration, it has to be given only once, and a catch-all allow for an interface counts as all its attributes already configured
<intellectronica> it's a PITA!
<intellectronica> andrea-bs: the obvious (and somewhat unpleasant) solution is to change the allow declaration to use attribute names instead of the interface, and include only those attributes that you really want to allow
<intellectronica> andrea-bs: let me see if there's a better solution i can think of, but it may be that you simply have to do the above
<intellectronica> andrea-bs: as you can see, that's how we do it for IBug too
<Chex> sinzui: the staging tree already has a bunch of stuff cowboyed in, won't let me pull in your branch.. should I do a revert and try again?
<sinzui> Chex: Yes, I think staging should be reverted since we released code
<andrea-bs> intellectronica, thanks. I'm allowing all ISpecification's attributes (except newMessage & friends) by hand. I think this will require some time :)
<intellectronica> andrea-bs: i know. it's no fun. sorry :(
<leonardr> james_w: sorry, i have more questions about https://code.edge.launchpad.net/~james-w/lazr.restfulclient/fix-caching/+merge/15635
<leonardr> i'm trying to see how difficult it would be to write a test for the branch
<leonardr> and whether it will really help you
<leonardr> what named operation are you calling that's not being cached, that you think should be cached?
<james_w> I saw it with lots of things
<james_w> distro_series.getSourcePackage for instance
<leonardr> were these all named operations that returned specific objects?
<leonardr> rather than collections?
<leonardr> and when you made the change locally, did you see that the cache was actually being used?
<leonardr> i wouldn't expect it to be used because right now launchpad doesn't serve caching directives
<james_w> maybe that's the cause then
<james_w> while looking in to it I discovered that it wasn't actually the lack of caching that was the limiting factor in the script
<james_w> so I eyeballed the bug and submitted the change before moving on to fix the other issue
<james_w> it's not just caching though
<leonardr> all right
<leonardr> i'm going to land the branch because it makes sense and will help us in the future
<james_w> httplib skips all the stuff it normally does on GET
<leonardr> but i don't expect it to improve performance immediately, if it only affects named operations
<leonardr> actually there might be a way to test it regardless...
<Ursinha> Chex, gary_poster, rockstar, bigjools, danilos, sinzui, allenap: production meeting in 28 mins @ #launchpad-meeting
<bigjools> Ursinha: al-maisan is covering for me today
 * al-maisan waves
<Chex> sinzui: sorry for the delay, all set now with that lp branch on staging
<Ursinha> oi, thanks bigjools and hi al-maisan :)
<Ursinha> s/oi/ok/
<al-maisan> hellau Ursinha :)
<Ursinha> stub, Chex, gary_poster, rockstar, al-maisan, danilos, sinzui, allenap: production meeting in 5 mins @ #launchpad-meeting
<Ursinha> stub, Chex, gary_poster, rockstar, al-maisan, danilos, sinzui, allenap: production meeting now @ #launchpad-meeting
<sinzui> bac: danilo: I think the webkit CSS problem is related to: http://pastebin.ubuntu.com/343618/
<sinzui> bac: danilo: We can put some of these rules back, but we should not use ids
<sinzui> bac: danilos: The first  rule and the second rule in the removed CSS are the most likely causes.
<danilos> sinzui, the only thing that was not old CSS but what I introduced as a bug fix 2 months back was #maincontent { clear: both; }, and that's the first thing I'd add back
<danilos> sinzui, margin-right: -25% probably helped with other issues people are seeing
<bac> danilos: i'm confused about the CSS problem.  for me it is only appearing on staging not edge.  is that consistent with what others are seeing?
<danilos> bac, not that I know, it appears on everything now that we've rolled it out
<bac> danilos:  very odd.  on my webkit browser edge looks fine
<danilos> bac, what is it that you are seeing? is main content indented for the logo width?
<bac> danilos: no.  on a team page the portlets that should be top right are bottom left, below the map.
<danilos> bac, right, so that's one part of the problem... if you reduce the font size, does the main content get indented (i.e. floats around the top-left logo)
<bac> no
<danilos> bac, see bug #493518
<mup> Bug #493518: Side portlet moved again below the main content on wide-screen displays (1920x1200) <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/493518>
<danilos> bac, ok, so you are being bitten by only part of the bug :)
<danilos> bac, I am bitten by both parts, and first one is that I get https://devpad.canonical.com/~danilo/screenshots/css-issues.png
<bac> danilos: i've added a comment and two screenshots to that bug
<danilos> bac, cool, thanks
<bac> danilos: note that i'm seeing the portlets on the bottom LEFT, not the same as emmet's screenshot where they are just pushed down
<danilos> bac, right, everybody else sees them at bottom right, so it's interesting to note
<salgado> BjornT, around?
<salgado> bac, is that on staging?
<salgado> bac, staging's currently broken; we need to 'make clean' there
<salgado> mbarnett, can you revert pending merges on staging and do a 'make clean' there?
<mbarnett> salgado: sure
<sinzui> salgado: mbarnett: We need to QA my spam branch on staging. It is a CP/reroll candidate. When can we remerge my branch?
<mbarnett> sinzui: ok, it sounds like thre is some contention over staging...
<mbarnett> salgado: things have been reverted and the appserver restarted.
<salgado> mbarnett, thanks!
<mbarnett> sinzui: whenever is fine by me, it just depends on what is going on with other devs on staging
<sinzui> salgado: are you testing anything on staging?
<salgado> sinzui, not anymore
<al-maisan> Is there a need to declare on the LP API the return type of a method that returns a string?
<sinzui> mbarnett: can you merge this branch into staging and restart it: lp:~sinzui/launchpad/spam-eggs-bug-495250
<mbarnett> sinzui: i believe salgado is currently running some tests.  is that true salgado?
<salgado> mbarnett, nope, I'm finished
<mbarnett> salgado: ok, great.  thanks
<mbarnett> sinzui: ok, give me a few (on a phone call), then i will merge that in for you
<EdwinGrubbs> sinzui: ping
<sinzui> Hi EdwinGrubbs
<EdwinGrubbs> sinzui: I have a new idea for solving the addMember() status confusion. Salgado had added a True/False return value just for the REST API, but it would be better to return the actual status. Then, the UI can do the right thing no matter what the model decides to do.
<sinzui> EdwinGrubbs: That is a great idea. I agree we should do it
<bac> salgado: great, the problem i saw on staging is gone
<bac> danilos:
<bac> danilos: ^^
<danilos> bac, thanks
<mars> sinzui, ping
<mwhudson> bigjools: here now...
<bigjools> mwhudson: I am not :)
<mwhudson> bigjools: congrats
<bigjools> I just wanted to catch up about how you're doing
<bigjools> but it's late, I'm tired, and fed up with 12+ hour days
<mwhudson> bigjools: i finally got something reviewable yesterday, thudding my way along i guess
<sinzui> hi mars
<bigjools> mwhudson: \o/
<EsatYuce> Who knows about GPG key?
<thumper> EsatYuce: many people who have stopped working for the day
<EsatYuce> thumper, : but i haven't done
#launchpad-dev 2009-12-18
<mwhudson> lunchtime
<poolie> spm: so is it all back now?
<spm> poolie: yes
<poolie> could you please tell identica that?
<spm> sure. give me a few to work my way thru the howto of that... ;-)
<poolie> go to identi.ca
<poolie> click login
<spm> you're enjoying this.... ;-)
<poolie> and you're ok from there?
<poolie> login as launchpadstatus
<poolie> then type in the little bittum
<spm> right. I'm in. now for some pithy statement
<spm> gosh. that's almost easy.
<jml> hmm. thumper is no longer present.
<jml> what's the ETA for PQM opening?
 * thumper is present
<jml> thumper, oh. I was just thinking about how, with your recent changes to the comment form on the review page, it might be nice to also have an MP status field there
<thumper> jml: nah
<thumper> jml: not until I split out the code review status from the merge status
<thumper> which I still want to do
<thumper> I'm going to close irc now to focus on pdr
<jml> thumper, oh right.
<jml> thumper, ok :)
<thumper> if anyone needs me because everything has blown up, call or text
<jml> thumper, kerblammo
<jml> thumper, I mean, good bye :)
<Pilky> beuno: you around?
 * thumper signs off for the year
<jml> thumper, ciao
<jml> thumper, merry christmas & a happy new year.
 * mwhudson EOWs
<jtv> mwhudson: enjoy your we
<poolie> spm, still here? spam - https://edge.launchpad.net/buyviagra
<spm> oookay. that's weird.
<spm> gone. the weirdness was the user account was already suspended; had to reset some of the "user" fields for the projet to registry
<poolie> thanks
<spm> poolie: fwiw. I do 10am - 7pm these days. Probably till we're out of daylight savings.
<poolie> oh ok
<spm> covers the gap 'tween michael and tom, as it were.
<jml> jelmer, hi
* spm changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 3.1.12 | PQM is open; RM: danilos | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<jml> spm, wuuu
<jml> spm, in danilos still RM?
<henninge> jml: sure, it's still week 4, isn't it.
<henninge> ?
<jml> henninge, then why is PQM open?
<spm> jml: I figure it's more appropriate for him to demote himself out of the RM role ( danilo_ see topic) :-)
<henninge> jml: good point
<danilo_> jml, well, I believe the process was to stay the RM until the next release; i.e. there might be CPs and such
<henninge> jml: you are saying: No more mangement needed once pqm is open?
<jml> danilo_, ahh, ok.
<danilo_> jml, I am not sure though, it's not that clear, so let's clarify that for the next one
<jml> henninge, no more release management is needed until the next release, which is next year.
<jml> henninge, which is aaaaaages away.
<danilos> jml, years even :)
<henninge> yeah
<henninge> although not as bad as a few years back where it might have been in the next millenium!
<danilos> jml, btw, we are opening PQM because there's no need for a re-roll, at least there was none so far
<danilos> henninge, but people still dispute when was that exactly ;)
<henninge> danilos: talking about days or years?
<henninge> I know, officially it was 2000/2001
<jml> danilos, even more controversially, when was the first Easter of the new millenium? :P
<henninge> jml: .... don't .... open ..... box ..... it's Pandorra's .....
<danilos> jml, heh :)
 * spm opts for the simple answer - when he had the easter long weekend; which year is irrelevant. ;-)
<jml> spm, bah, they didn't even give us an extra day off for the millenium
<jml> OR the olympics.
<jml> shocking really
<spm> jtv, et al: one minor issue on that approve-imports - "-q" <== what's with the warning's??? ;-)
<spm> jml: truly
<jtv> spm: it's an operational problem situation that happens once every few months that requires special intervention from a human who understands what's going on with these packages.
<jtv> spm: in principle we could turn these into email notifications, but it's just one more thing that could break without us ever noticing
<spm> jtv: :-) I'm teasing. this is endemic across all of LP cronscripts. -q (ie Be QUIET!) doesn't stop warning's - which it should; we should only see ERRORs or higher.
<jtv> spm: we just shouldn't issue warnings for things that aren't worth one
<spm> on the more ..... painful scripts that refuse to be quiet we throw to logs and let scriptactivity catch 'em
<spm> right
 * jml submits a bug fix
<spm> jtv: where this particular is even worse - normally all warnings should be STDOUT; ERRORS usually to STDERR; CRITICALS certainly to STDERR. in LP's case; all scripts dump to stderr; which means if we throw to log - we will NOT know if a script is barfing (stack trace etc) unless we go looking for it. This is a Bad Thing(tm).
<jtv> spm: indeed... we've got the weirdest, most mundane stuff going out to stderr for some reason
<adiroiban> danilos: do you have time for a review regarding the distroseries access when translations are hidden? I went further than fixing the bug and trying to refactor some code... again
<danilos> adiroiban, heh, you are unstoppable :)
<danilos> adiroiban, sure
<adiroiban> lp:~adiroiban/launchpad/bug-497438
 * jtv cheers for adiroiban
<adiroiban> well, holydays are comming
<danilos> adiroiban, do you have an MP as well? (that's much easier because it keeps the patch up to date)
<adiroiban> danilos: nope.
<adiroiban> i am not sure I did the right thing
<adiroiban> and this is why did not pushed it as a MP
<adiroiban> it's about checkTranslationsViewable
<adiroiban> as it is implemented in 2 classes
<adiroiban> and each class is calling the other class method
<adiroiban> i tried to break that chain and define all the logic required by checkTranslationsViewable in a single place
<adiroiban> not it is split between registry and translations
<adiroiban> s/not/now/
<jtv> adiroiban: not sure what you mean by each class calling the method on the other...  all browser code that needs to make this check delegates to the distroseries model class, no?
<adiroiban> checkTranslationsViewable is defined in registry.model.distroseries and also in translations.browser.distroseries
<adiroiban> why not have it defined only in registry.model.distroseries ?
<adiroiban> as checkTranslationsViewable from registry.model.distroseries is calling checkTranslationsViewable from registry.model.distroseries
<danilos> adiroiban, first of all, when you inherit a permission, you probably want to use it via super() (i.e. in AdminDistroSeriesLanguage)
<danilos> adiroiban, however, since the two permissions are on different objects, you should base AdminDistroSeriesLanguage on AuthorizationBase and keep the code as it is
<danilos> uhm, that was for AdminDistroSeriesTranslations
<danilos> adiroiban, for AdminDSL, I think it makes more sense to use AdminDistroSeriesTranslations there, even though they are identical right now
<jtv> do we have any admin'ing on dsls?
<adiroiban> adiroiban: ok. I will use AuthorizationBase and base DSL to DS
<danilos> jtv, well, base both off AuthorizationBase, but call DS one in DSL implementation, if you understand what I mean :)
<jtv> danilos: ah yes that's better, thanks :)
<danilos> jtv, perhaps we don't yet, but it doesn't hurt to improve it... we also allow admins to take a look at translations for non-open distroseries
<adiroiban> danilos: should we also change this code then?
<adiroiban> class AdminDistributionTranslations(OnlyRosettaExpertsAndAdmins, EditDistributionByDistroOwnersOrAdmins)
<danilos> adiroiban, no
<danilos> adiroiban, that's exactly the point I am trying to make
<danilos> adiroiban, i.e. that one inherits OnlyRosettaExpertsAndAdmins, which doesn't depend on self.obj at all, only on the passed in user
<danilos> adiroiban, and on a permission which has a compatible object (self.obj is IDistribution in both)
<adiroiban> danilos: ok :)
<danilos> adiroiban, so, that one is fine... but eg. we can't add AdminDistroSeriesTranslations to that list though, because DistroSeries and Distribution are different objects :)
<adiroiban> that's true
<adiroiban> my idea was to make thinks simple, by always inheriting AuthorizationBase, when we are going to overwrite checkAuthenticated
<adiroiban> things simpler
<jelmer> jml: hi
<mrevell> Morning!
<wgrant> bigjools: It works! Thanks.
 * noodles775 thinks bigjools is probably stuffing as much wood into his heater as he can right now...
 * wgrant would prefer that to the weather we had here on Wednesday.
<noodles775> how hot did it get?
<wgrant> Only 39Â°C down here.
<noodles775> !
<wgrant> Hopefully we'll have no 45Â°C days this summer...
<bigjools> noodles775: is quite right
<bigjools> wgrant: you got a 3.0 package in?
<wgrant> bigjools: I tried one in my PPA, and one was synced a couple of hours ago.
<bigjools> woohoo
<wgrant> But gina doesn't seem happy.
<wgrant> Is iron not done?
<bigjools> hmmm I asked for all 4
<wgrant> The diff worked, though.
<wgrant> Not sure where that is run.
<bigjools> wgrant: what makes you think gina is not happy?
<wgrant> bigjools: It's not imported anything v3, yet you said 11 hours ago it was upgraded everywhere.
<wgrant> And I thought it was running four times a day now, although even if it's two it's still close.
<bigjools> wgrant: switch wasn't thrown until 0350 UTC
<bigjools> yrsh it's 4
<wgrant> bigjools: gina is disobedient.
<wgrant> And does not obey such switches.
<bigjools> oh of course
<bigjools> I'll get the log file checked
<wgrant> Thanks.
<adiroiban> danilos: those changes are not sound, so I will push a new revision. But I am not happy with the current way of implement the security check for distroseries translations
<danilos> adiroiban, well, there are many things I am not happy with in the code, but what works today is always much better than what might be there in the future :)
<adiroiban> I am not sure if moving the security checks in the model is the right thing to solve the problem
<danilos> adiroiban, btw, sorry, I had a call so couldn't continue looking at it
<adiroiban> danilos: np
<danilos> adiroiban, so, in general, I am not sure why do you need permissions on the model here?
<danilos> adiroiban, what we should have is launchpad.View permission always allow those with launchpad.TranslationsAdmin permission
<danilos> adiroiban, at least for all the translations objects
<adiroiban> in the model for distroseries we have checkTranslationsViewable, which raised an exception if those translations should not be displayed to the user
<adiroiban> and showing or hiding the translations depends on the user
<danilos> adiroiban, so, that approach is ugly :)
<adiroiban> and right now, before calling that fuction, the view checks if the users have the rights to see the translations and if does, that method will not be called
<adiroiban> and instead of checking for permission in a single place (like security proxy)
<adiroiban> we check in the view and the model (beside the usual check from the security proxy)
<danilos> adiroiban, well, it doesn't sound as something too hard to fix, except that we'd have to do it for all the objects we are rendering in a single distroseries (i.e. pofiles, potemplates, DSLs, templatesets, etc.)
<adiroiban> yes, and there are also some test that will fail
<adiroiban> i will try to fix it, but in another branch
<danilos> adiroiban, basically, we'd need a generic permission for all of these to allow those with check_permission(TranslationsAdmin), but for objects which can be somewhere else (i.e. pofiles and potemplates can be in a product), we'd have to have custom permissions
<danilos> adiroiban, yeah, please keep the fixes separate from refactoring, that makes it much simpler
<adiroiban> danilos: another note. I think we should stop using admin_browser and switch to an translationsadmin_brower, or better to rosettaexpertes_browser and distrotranslationsadmin_browser
<adiroiban> maybe we can find shorter names
<danilos> adiroiban, we should, true
<danilos> adiroiban, many tests might need rewriting though
<adiroiban> i think that for most we can just replace admin_browser with newadmin_browser
<adiroiban> newadmin_browser = rosettaexperts_browser
<adiroiban> and when we are dealing with ubuntu or a distribution, use utc_browser
<adiroiban> or dtc_browser (distributiontranslationscoordinators_browser)
<danilos> adiroiban, it's not that simple, I am afraid... some tests might be using admin_browser to change stuff that rosetta_browser can't (i.e. setting up a bzr branch for a productseries); I am just guessing, I don't know how many cases like these we have
<adiroiban> danilos: I agree
<danilos> adiroiban, anyway, stories are not meant to be comprehensive
<adiroiban> then where should we do those tests?
<danilos> adiroiban, well, we should do basic tests for major user stories like you suggest
<danilos> adiroiban, we should not comprehensively test all bits and pieces... i.e. if we test that UTCs have privileges, we should probably not test that rosetta experts have, and so forth... i.e. what you said yourself :)
<adiroiban> adiroiban: ok. but can we have rosetta_browser and utc_browser created by default in pagetest, so that we don't need to create them in each test?
<adiroiban> I can just start migrate the current test as I come across them
<adiroiban> with the latest change in permission system, most of them are no longer valid
<danilos> adiroiban, yeah, we can
<danilos> adiroiban, but, since those are based on sampledata, I'd rather just have a utility function which constructs them
<danilos> adiroiban, i.e. sampledata that doesn't exist
<adiroiban> danilos: True. But then we need to have something like this at the beginning of each test for distributions http://paste.ubuntu.com/343746/
<adiroiban> should we but it in factory.py ?
<danilos> adiroiban, oh, I said a utility method so you should replace that code with utc_browser = setup_distribution_translation_admin_browser()
<danilos> adiroiban, this should be a pagetest helper, not part of factory
<adiroiban> yep.
<danilos> adiroiban, it should be added to lib/canonical/launchpad/testing/pages.py and to the setUpGlobs so it's accessible widely
<adiroiban> thanks!
<adiroiban> i was just searching for that :)
<danilos> adiroiban, I just don't want utc_browser to be set-up for every pagetest, because it's complex to generate (and would slow everything down)
<danilos> adiroiban, so, don't set it up like the other browsers are set up
<adiroiban> agreed
<adiroiban> something like this:     test.globs['setup_utc_browser'] = setup_utc_browser
<adiroiban> just a reference to the function
<adiroiban> ?
<adiroiban> danilos: regarding bug 359180, it's ok to go with yui event-key and have keybindings similar to poedit?
<mup> Bug #359180: Missing keyboard shortcut for navigation <trivial> <ui> <Launchpad Translations:In Progress by adiroiban> <https://launchpad.net/bugs/359180>
<intellectronica> andrea-bs: so you've already got the ui branch?
<andrea-bs> intellectronica, I've just pushed it, but actually I haven't written code yet
<intellectronica> ah ok :)
<intellectronica> just howl if you need any help
<intellectronica> oh and pqm is open. i'll send your model branch for test and landing
<danilos> adiroiban, depends, do they conflict with any regular bindings?
<danilos> adiroiban, i.e. what is Ctrl+C for?
<adiroiban> :)
<adiroiban> my fault. it should have been Alt+C
<danilos> adiroiban, I've added a comment to the bug, fwiw :)
<danilos> adiroiban, Alt + C might also interfere with what people might be doing in their browsers (depending on the translation of their menus)
<adiroiban> but  those shortcuts will only be available on +translate page
<adiroiban> if we are going to make some complicate keybindings just to avoind any browser in any language, they will no longer be shortcuts
<adiroiban> don't know what to say
<BjornT> does 'ec2 land' work for anyone? i get this errror when trying to use it: http://paste.ubuntu.com/343765/
<Pilky> beuno: you around?
<beuno> Pilky, hey
<Pilky> beuno: hey, just wanted to discuss a crazy idea I had with you
<beuno> Pilky, I'm eating lunch, but, shoot
<beuno> I'll read on and off
<Pilky> ok, well I was thinking about launchpad UI again last night and the thought occurred to me that trying to use regular web design for it is doing it a great injustice, as it isn't a web site, but a web app on the more complex end of the scale
<Pilky> so I had the crazy idea of attempting to make a cappuccino UI for launchpad, using the API
<Pilky> and designing it as though it is an actual application rather than a powerful website
<beuno> Pilky, interesting
<Pilky> the benefit is, it would be able to be a completely separate thing, by going through the API the UI would be able to be a separate project
<beuno> would that mean making all pages ajaxy?
<Pilky> meaning a smallish team could work on it and provide consistency
<Pilky> well not as such, have you heard about cappuccino?
<beuno> I haven;t
<beuno> I expect it's a framework?
<Pilky> well, basically it is the closets to a desktop calibre API for building high end web apps
<Pilky> yeah
<Pilky> it basically has two components: cappuccino the framework and Objective-J the language
<beuno> Pilky, it's an interesting idea, I'd pitch it to the -dev mailing list
<Pilky> Cappucino is to Cocoa as Objective-J is to Objective-C
<Pilky> and it allows for applications along these lines: http://280slides.com/
<beuno> Pilky, does it scale well?
<beuno> into a gazillion requests per minute, etc?
<Pilky> well I'm not sure, I mean the application, once loaded, runs entirely in the browser as it is basically pure javascript, such things would have to be tested
<beuno> splitting the UI that much is certainly an interesting idea to explore
<Pilky> but the benefit is, rather than replacing the current LP UI it can be an alternative
<Pilky> but it would make it feel much more like an application, which is what it really is
<Pilky> the benefit of cappuccino of course is that it is LGPL so stuff can be added if needed
<beuno> yeah, would allow more experimentation
<Pilky> yeah, I've been looking for a project to allow me to really test out cappuccino and something like this would be ideal
<Pilky> I'll post something to the -dev list and see if anyone else would be interested in helping out one something like this in the new year
<beuno> sure, I'd have to look into objecive-j as well  :)
<Pilky> heh, well the website for it is: http://cappuccino.org/
<Pilky> and the IDE they're making that is currently in beta is at: http://280atlas.com
<Pilky> the benefit to me is that Objective-J/Cappuccino basically are Obj-C/Cocoa for the web so my Mac/iPhone development knowledge is quite easily transferable
<beuno> Pilky, I will poke around, i always like learning new things
<Pilky> cool
<Pilky> beuno: ok I've posted something to the list. I'm really piling up a large amount of open source projects on my plate at the moment :P
<Pilky> this will be the 4th one I'm working on, not including all the various bits of code I release
#launchpad-dev 2009-12-19
<jml> jelmer: hello
<jml> jelmer: I was going to ask you to set development focuses for a few projects.
<jml> jelmer: but I can't recall what they were.
<wgrant> bin/combine-css is dieing horribly for me.
<wgrant> Lots of errors in the CSS.
<wgrant> Ah, cleaning and makeing worked.
#launchpad-dev 2009-12-20
<mwhudson> so
<mwhudson> anyone else here today? :)
<lifeless> no
<lifeless> I is on leuvh
<jml> good morning
<jml> mwhudson, I am here
<mwhudson> jml: good morning
<mwhudson> i am about to be not here for an early lunch (and another visit to the new house)
<jml> mwhudson, ok.
<jml> mwhudson, maybe we should have a call when you get back?
<mwhudson> jml: that would be good, eys
<wgrant> mwhudson: Why did I just get a code import email with 'None' as the subject?
<poolie> hi all
<elmo> mwhudson: so, umm
<elmo> why do we think this is the problem again?
<elmo> oh, right, never mind
<elmo> in any event, the fix is to simply remove the check for x-original-to
<elmo> and use to/cc
<elmo> at least AFAICS
<elmo> i can go to the trouble of adding an X-Launchpad-To header if you like, but AFAICS the end result is the same
<elmo> with the exception, I guess of Bcc-ing, but I'm not sure we want to support that
<elmo> I guess we do
<elmo> mwhudson: in the meantime, I'm going to cowboy http://paste.ubuntu.com/344005/ so I don't have to fuck with launchpad.net's email at 11pm on a Sunday night
<wgrant> Isn't using anything other than the envelope recipient wrong?
<elmo> you can't see the envelope recipient unless the receiving MTA copies it to another header for you
<elmo> and the only disadvantage of using non-envelope is that you lose Bcc
<elmo> I agree in the long term, I should fix the MTA to copy the envelope to into another header and revert back to that
<elmo> I just don't want to do it tonight
<wgrant> Right.
<elmo> gaaaaaaaaaaaaaaah
<elmo> except of course spam's badly formed to: lines causes launchpad to lose it's TINY MIND
<wgrant> Hahaha.
<wgrant> Awesome.
<jml> heh heh
#launchpad-dev 2010-12-20
<poolie> thumper: hello!
<poolie> thumper: can you read https://code.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733 when you get back?
<poolie> thumper: hi?
<thumper> poolie: hi
<poolie> hi thumper
<poolie>  can you read https://code.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733 for me please?
<poolie> oh, you did
<thumper> yep
<poolie> what kind of a test would you like?
<poolie> that it gets set? or how killed jobs are handled?
<thumper> I think that right now, if a job isn't marked as done or failed explicitly, it will be retried
<poolie> that's good
<poolie> you said "care to add a test"
<poolie> what test?
<thumper> poolie: What error is raised when rlimit killed?
<thumper> poolie: can we catch it and fail the job?
<thumper> grr
<poolie> it's complicated
 * thumper wants to move BufferLogger into lp.testing.logger
<thumper> bug archivepublisher uses it
<poolie> you may get a MemoryError or you may get a signal
<thumper> s/bug/but
<thumper> poolie: hmm...
<wgrant> thumper: How is it different from the other five fake loggers?
<poolie> istm it would be more robust to just have the whole process terminate and have a higher level supervisor notice it
<wgrant> thumper: And why does archivepublisher stop you from moving it?
<thumper> wgrant: BufferLogger inherits from FakeLogger and writes into a StringIO
<poolie> if you're out of memory there are no real guarantees that python can keep doing nontrivial things
<poolie> this may be overly pessimistic though
<thumper> wgrant: it seems wrong to import it into a not-testing piece of code
<wgrant> thumper: oh, right.
<thumper> poolie: I'm just concerned that if we kill the job due to memory problems, it'll do it every time
<thumper> as it gets retired
<thumper> retried
<thumper> so...
<thumper> we need a way to make sure that doesn't happen
<thumper> not sure exactly how though
<thumper> unless...
<thumper> right at the start of the scan we increment the try count on the job
<thumper> and we only try to load jobs with counts less than x
<poolie> that seems like a good idea to me
<poolie> of course you should reset it when it actually completes
<poolie> i thought there would already be something like that in place
<poolie> also, what's up with
<poolie> rm: cannot remove directory `/var/tmp/bazaar.launchpad.dev': Operation not permitted
 * poolie stops recursing
 * poolie makes schema, and lunch
<thumper> poolie: I think I fixed the permission problems with that dir
<spiv> Thinking of recursion: bug 692085 is the latest stacking-related joy
<_mup_> Bug #692085: "maximum recursion depth exceeded while calling a Python object" on self-stacked branch <stacking> <Bazaar:Confirmed> <Launchpad itself:New> < https://launchpad.net/bugs/692085 >
<poolie> mm i saw your comment on that
<wgrant> Hmm.
<wgrant> It would be nice if the MP page showed if the prereq was merged.
<wgrant> By giving it a strikethrough, maybe.
<poolie> mm that would be good
<poolie> lp could use strikethrough a lot more
<poolie> haha, or smallcaps for sarcasm
<poolie> i'm adding a new test class
<poolie> and it fails with
<poolie> ComponentLookupError: (<InterfaceClass canonical.launchpad.webapp.interfaces.IPlacelessAuthUtility>, '')
<poolie> i guess because i didn't specify a layer
<poolie> what's the cheapest one that would work?
<poolie> (the test itself shouldn't need to touch the db or anything)
 * thumper unifies QuietFakeLogger and BufferLogger
<wgrant> thumper: I have something new in EC2 that uses QuietFakeLogger, so watch out if you're renaming it.
<thumper> poolie: if it is complaining about logging in, you should have the DatabaseFunctionalLayer
<thumper> wgrant: ack
<thumper> wgrant: I'm not renaming it, just deleting it :)
<wgrant> Hah.
<poolie> thanks
<thumper> wgrant: and moving it
<wgrant> thumper: Moving *and* deleting? You are truly skilled.
<thumper> wgrant: well, moving BufferLogger and FakeLogger and deleting QuietFakeLogger
<wgrant> So we are going to be down two fake loggers today.
<wgrant> Good progress.
<poolie> aka "taking it behind the shed, then shooting it"
<wgrant> It's the best we can do.
<thumper> OMG
<thumper> just deleted three other loggers
<wgrant> The one from lp.archivepublisher.tests.util?
<thumper> and two others
<wgrant> The archivepublisher one's death is currently in PQM.
<thumper> ah
<thumper> ok
<thumper> is it /dev/nul or /dev/null ?
 * thumper makes DevNullLogger to express explicit intent
<thumper> and another FakeLogger killed
<thumper> OMFG three MockLoggers?
<thumper> sorry, four
<wgrant> I didn't think there was quite that many.
<wgrant> Oh, some doctests define their own.
<wgrant> Nice.
<thumper> me either
 * thumper fetches the machete
<wgrant> WTF, a Launchpad page load is causing my disk to thrash.
<wgrant> ... oh.
<wgrant> Forgot to turn off devmode.
<wgrant> So it was creating hundreds of OOPSes.
<poolie> hooray
<wgrant> It's tempting to fix that, except, well, JS.
<lifeless> I welcome all changes to make dev and prod closer
<lifeless> in principle :P
<wgrant> Does anyone happen to know which config cesium uses?
<wgrant> It looks like it might be ftpmaster.
<wgrant> But that sounds wrong.
<lifeless> poolie: hi
<lifeless> poolie: thanks for looking at bug 314507; I'm struggling to understand how the existing code had the behaviour it has, given the test looks for 'OAuth realm'
<_mup_> Bug #314507: OAUTH server ignores ignores first element in header (rather than realm key) <api> <lp-foundations> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/314507 >
<lifeless> s/test/condiiton/
<wgrant> Hm.
<wgrant> Just had lots of devscripts tests fail in ec2:
<wgrant> _StringException: Text attachment: garbage
<wgrant> ------------
<wgrant> [<bzrlib.lockable_files._LockWarner object at 0x12e95d10>]
<wgrant> ------------
<wgrant> Anybody ever seen anything like it?
<spiv> That's a strange object to have as gc garbage.
<spiv> I mean, it does have a __del__ method, but it carefully only has a reference to a str (a repr() of a LockableFiles instance) and int, so I can't see any reason why it would be in a reference cycle.
 * wgrant pqm-submits.
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       85 /  343  BugTask:+index
<lifeless>       70 / 5123  Archive:+index
<lifeless>       22 /  193  Distribution:+bugs
<lifeless>       18 /  196  POFile:+translate
<lifeless>       12 /  122  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>       10 /  204  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        7 /   12  NullBugTask:+index
<lifeless>        6 /   36  MailingListApplication:MailingListAPIView
<lifeless>        6 /    0  Distribution:+builds
<lifeless>        5 /    4  Person:+bugs
<spiv> Archive:+index is just a big softie, really.
<wgrant> BFB is currently enabled on edge/qastaging regardless of team memberships. Do we still desire that?
<lifeless> sure
<lifeless> edge will become irrelevant when the rt to redirect to prod goes live
<lifeless> (as far as BFB is concerned)
<wgrant> lifeless: Well, what I really want to know is if I can delete them from the configs, which lets us eventually remove that code.
<wgrant> There's a lot of obsolete cruft in the configs because you can't remove things from the schema without first removing them from the prod config.
<wgrant> Some of this stuff is 4 years old.
<lifeless> wgrant: doit
<wgrant> lifeless: Thanks.
<lifeless> wgrant: the feature flag is sufficient control
<wgrant> That's what I thought.
<lifeless> wgrant: OTOH
<lifeless> wgrant: I'd really like not to have unnecessary config changes with my branch
<lifeless> which rejiggers everything
<wgrant> lifeless: Doesn't it only affect the appserver instances?
 * wgrant rereads.
<lifeless> wgrant: but perhaps you could update my branch for me ?
<lifeless> wgrant: edge are appservers
<lifeless> wgrant: I'm not sure if I would be affected or not
<wgrant> lifeless: Right, but I'm only dealing with the stuff in the root directory.
<lifeless> wgrant: I'm expressing a 'I don't want more effort' plea, is all.  :)
<wgrant> Your generated configs inherit from it.
<wgrant> Sure.
<lifeless> stub: perhaps you could look at why the query in bug 691478 is slow
<_mup_> Bug #691478: Archive:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/691478 >
<stub> Initial guess, it is horrible old SQLObject code generating horrible queries. All that _prejoin...
<wgrant> db_statement_timeout doesn't do anything outside the webapp, does it?
<wgrant> I've never seen scripts time out.
<stub> Scripts never time out
<stub> We rely on some reaper cronjobs to kill stuff behaving terribly (runtime > x hours, transactions > x minutes)
<wgrant> So why, then, does germanium's config have a special DB timeout...
<wgrant> sigh.
<stub> There is one rosetta script they hacked timeouts into I seem to recall (stop gap fix)
<wgrant> Right.
<lifeless> wgrant: db_statement_timeout is now usable in other contexts
<lifeless> wgrant: its likely not actively used, but it can be.
<wgrant> 5~/win 6
<wgrant> Grah.
<adeuring> good morning
<wgrant> stub: Can I grab a patch number now for dropping Distribution.lucilleconfig and DistroSeries.lucilleconfig, or must I get it during review?
<wgrant> Morning adeuring.
<adeuring> hi wgrant!
<stub> wgrant: patch-2208-35-0.sql
<wgrant> stub: Thanks.
<wgrant> danilos: Hi.
<danilos> wgrant, hi
<wgrant> danilos: Are you able to QA that TTBJ change?
<danilos> wgrant, I will definitely need some help, but I am not sure even dogfood is sufficient (it doesn't have codehosting set-up if I remember correctly)
<danilos> wgrant, I was QAing the easier stuff we got landed first :)
<wgrant> danilos: Right, it's probably better on staging.
<danilos> wgrant, yeah, the important code change is on staging fwiw, though I'll need help to learn how exactly was the error triggered on the build master
<wgrant> danilos: I have a regression fix a few revs behind it, so I'm somewhat interested in getting it out of the way ASAP.
<wgrant> danilos: I don't recall myself. Let me check the logs.
<danilos> wgrant, ah, makes sense
<danilos> wgrant, if I QA the part where the job is generated with appropriate build link, would you be able to QA the build master part? :)
<wgrant> danilos: Sure. I can easily hack that into the DF DB.
<danilos> wgrant, cool, let me go and generate a job on staging then (though, I need to check staging has the revision first)
<wgrant> Thanks.
<danilos> it does, cool
<wgrant> Ahh, it breaks during failure counting. I see.
<wgrant> That is approximately impossible to test on staging.
<wgrant> I will work out how to do it on DF>
<lifeless> gnight
<wgrant> Night lifeless.
<wgrant> danilos: What's a good branch to test on?
<wgrant> I haven't manually created TTBJs without local codehosting before.
<danilos> wgrant, I am using lp:upstart
<danilos> wgrant, the thing is that we want this to go through codehosting as well since that's what sets the .build property basically
<danilos> wgrant, i.e. it's a kludgy hack which makes use of the branchjob.json_data to store build_id
<wgrant> danilos: Right.
<wgrant> I saw the diff.
<wgrant> You'll verify that it was created properly on staging, and I will manually create roughly the same one on DF.
<wgrant> I will then reset the builder half-way through the build, hopefully triggering the failure counting code path.
<danilos> wgrant, right, sure
<wgrant> danilos: OK, I can reproduce the issue.
<wgrant> So we may be in a position to QA it.
<danilos> wgrant, cool
<danilos> wgrant, I am waiting for a LOSA to run scan_branches on staging, but none seem to be around
<danilos> wgrant, fwiw, if the script that jtv wrote to construct TTBJs on dogfood is still there somewhere and it uses the actual TTBJ.create() then you might try using that for QAing your part of it
<wgrant> danilos: That's what I've used.
<wgrant> Well, .create()
<wgrant> Didn't see a script.
<wgrant> Just used the harness.
<danilos> wgrant, right, that should be enough
<danilos> wgrant, if you can see a branchjob with job_type=6 you can also look at json_data and if that contains "build_id" then you should be good to QA it all
<wgrant> danilos: Right.
<wgrant> I think we have a bug, though.
<wgrant> Checking.
<wgrant> Once mawson wakes up.
<danilos> wgrant, ok
<wgrant> danilos: qa-ok from my end, but it doesn't fix the bug.
<wgrant> danilos: TTBJ.build isn't exposed on the interfaces.
<wgrant> So I still get ForbiddenAttribute.
<wgrant> After creating IBJFO['build'] it works fine.
<danilos> wgrant, ah, but it was not exposed on any other buildfarmjobold's I checked
<wgrant> But this is no more broken than it was before.
<danilos> wgrant, either I mean
<danilos> wgrant, yes, I understand, but I was under the impression that it was not needed since none of the other jobs had it on the interface: would they not hit the same problem?
<wgrant> danilos: I see IBuildPackageJob['build']...
<danilos> wgrant, I guess I missed it then
<wgrant> danilos: So, as long as it doesn't break the scanner, I think it's qa-ok. Do you want to land a branch adding it to the interface, or shall I?
<danilos> wgrant, I wouldn't mind if you do it, since I've got other things to QA as well (12101 which might also block your revision from being rolled out)
<wgrant> danilos: Sure. Thanks.
<danilos> wgrant, btw, can I ask you for some help in QAing another soyuz-affected translations bug? is it possible for you to trigger a rebuild of kde-l10n-sr package on dogfood?
<wgrant> danilos: Sure. Is this the /-stripping one?
<wgrant> Or variants?
<danilos> wgrant, nope, it's variants :)
<wgrant> Right.
<danilos> wgrant, I can QA the "/" stripping one without soyuz
<wgrant> danilos: Any particular series?
<danilos> wgrant, not really as long as it is recent, just let me know which one it is
<wgrant> danilos: Do you need some rosetta script run to process the tarball?
<danilos> wgrant, doing a cronscripts/rosetta-approve-imports.py should be enough after the custom upload has been processed (and entries added to the queue)
<wgrant> danilos: OK. It will probably take a while to build, as the DF builders are a little bad.
<wgrant> I'll let you know when it's done.
<danilos> wgrant, cool, thanks
<danilos> wgrant, it shouldn't be too bad in that it should only run msgfmt (if that) over all the PO files which is very quick
<wgrant> danilos: It took 10min on the fast i386 production builder.
<wgrant> Although that might have been before buildd-manager was fixed.
<deryck> Morning, all.
<wgrant> danilos: DF is being troublesome, but it will eventually bend to my will.
<danilos> wgrant, I can see that builders have just picked up the build :)
<wgrant> danilos: Yeah... it will hopefully even work this time.
<wgrant> There we go.
<wgrant> danilos: After a few minutes with a load of 6 and iowait of 99%, process-accepted.py is processing the tarball.
<wgrant> But it has been doing so for some minutes.
<danilos> wgrant, right, that step will take because it has to insert a few hundreds of rows
<wgrant> Oh good, now apt-get is eating the CPU and disk.
<wgrant> And I cannot kill it like I can everything else :(
<wgrant> danilos: The upload is processed.
<wgrant> I guess I need to run the approval script now.
<danilos> wgrant, woohoo, if there are many existing entries in the queue it might take ages
<wgrant> danilos: I'm about to find out and DELETE them all.
<danilos> wgrant, well, no, that would remove the new ones as well :)
<danilos> wgrant, you can delete all that are not for source package kde-l10n-sr though
<danilos> wgrant, translationimportqueueentry.sourcepackagename is the column you want to filter on
<wgrant> danilos: Thanks.
<danilos> wgrant, btw, can you pass me the generated kde-l10n-sr_4.5.85-0ubuntu1+df1_i386_translations.tar.gz please? (you can just get a libraryfilealias row from the DB for that filename and I can construct a URL)
<wgrant> danilos: 57545773
<danilos> wgrant, thanks
<wgrant> OK, I've killed all the other import queue entries.
<wgrant> rosetta-approve-imports running... hopefully mawson won't singlehandedly thaw the UK.
<wgrant> danilos: It's going through a lot of transactions, but not logging anything apart from that.
<danilos> wgrant, that's ok, it's approving files alright: https://translations.dogfood.launchpad.net/ubuntu/natty/+source/kde-l10n-sr/+imports?start=0&batch=50
<danilos> wgrant, however, the generated tarball is not what I expected
<wgrant> danilos: Is it worth letting it continue?
<danilos> wgrant, not really, it's basically qa-ok in that nothing seems to be broken, but I am just checking what the natty tarball looks like on production
<danilos> wgrant, oh, it's actually the same, KDE seems to have changed the layout yet again :(
<wgrant> danilos: It is midnight here, so I should probably sleep. But if you want anything else done to QA it properly, let me know and I'll do it tomorrow.
<wgrant> Hah, lovely.
<danilos> wgrant, we'll probably have to do it for maverick to see if the new functionality works, but then also to see what the new layout in kde 4.5.85 is and implement that :(
<wgrant> danilos: Can I reupload maverick's to maverick-updates, then?
<danilos> wgrant, sure, that wouldn't hurt, though you should get some sleep as well :)
 * wgrant uploads.
<wgrant> danilos: -proposed uploads translations too, yeah?
<danilos> wgrant, uhm, I have no idea to be honest
 * wgrant checks prod.
<danilos> wgrant, I don't think it does though
<wgrant> danilos: Do I need to rebuild the tarball, or can I just use an old maverick one from prod and hack it into a new upload?
<wgrant> That will let us bypass the builder.
<danilos> wgrant, hum, it might not help since it looks like pkgstriptranslations has changed (and not KDE layout) to strip these translations (though I'll have to check that assumption)
<wgrant> http://launchpadlibrarian.net/54801590/kde-l10n-sr_4.5.1-0ubuntu1_i386_translations.tar.gz is the one I'm looking at.
<danilos> wgrant, yeah, that should work
<danilos> wgrant, I didn't know that was an option :)
<wgrant> danilos: We have direct DB access. *Everything* is an option.
<danilos> wgrant, heh, I meant a *viable* option
<wgrant> danilos: OK, DB hacked. Hopefully process-accepted will see it.
<danilos> wgrant, cool, thanks
<wgrant> danilos: I thought it had failed. Then I looked at the right series.
<wgrant> https://translations.dogfood.launchpad.net/ubuntu/maverick/+source/kde-l10n-sr/+imports?field.filter_status=NEEDS_REVIEW&field.filter_extension=all
<wgrant> If it looks OK, I'll run the approval script.
<danilos> wgrant, yeah, looks good, we've got 4 times more files
<danilos> wgrant, and anyway, the natty is nothing to worry about since upstream didn't include the files either
<wgrant> danilos: Some stuff is approved now.
<wgrant> With variants.
<adeuring> leonardr: could you help me to figure out a test for accessing restricted librarian files via the webservice? http://paste.ubuntu.com/545953/ indicates that a webserive client can't call open() for restricted files
<wgrant> adeuring: The token restricted librarian does not work in a dev or test environment.
<wgrant> It needs wildcard DNS.
<adeuring> wgrant: I know
<adeuring> wgrant: the point is that I'd like to check at least the URL of a restricted file
<leonardr> adeuring: you want to see which url it would request, without requesting the url?
<danilos> wgrant, /me looks
<adeuring> leonardr: ;) not exactly, but...
<leonardr> launchpadlib is making the request and then following a redirect, so you would need to change its redirect handling
<danilos> wgrant, can you please kill the script? it's not working
<danilos> wgrant, I also forgot to ask if that revision is included in the dogfood codebase? :)
<adeuring> leonardr: yes, that was my idea too, but I don't see right now a nice way to tell the client to not follow the redircte
<wgrant> danilos: devel r12107, right?
<leonardr> adeuring: you can set .follow_redirects = False on the http object, which i think is Launchpad._browser
<danilos> wgrant, that's right
<adeuring> leonardr: ah, cool,I'll try that. thanks!
<wgrant> danilos: It's there.
<danilos> wgrant, ok, thanks
<danilos> wgrant, can you please copy lib/lp/translations/model/translationimportqueue.py somewhere for me?
<danilos> wgrant, and then I let you go to sleep :) promise ;)
<wgrant> danilos: http://ppa.dogfood.launchpad.net/translationimportqueue.py
<danilos> wgrant, thanks, looks good, I'll have to spend more time with it to see where is it going wrong
<danilos> wgrant, thanks for the help today
<wgrant> danilos: np
<wgrant> Thanks for QAing your stuff quickly.
<danilos> yw
 * wgrant sleeps.
<pcjc2> Once a merge request to LP is accepted, what is it waiting on at that point?
<jelmer> pcjc2: merging
<pcjc2> https://code.launchpad.net/~pcjc2/launchpad/allow-empty-comments/+merge/43449
<pcjc2> Is that automatic?
<jelmer> pcjc2: Usually a developer makes sure all tests are run and then submits it to our merge bot
<jelmer> pcjc2: at the moment, that's done manually by a Launchpad developer
<pcjc2> ok, I think gmb was going to look at it for me
<pcjc2> I'll ping him to see if there were any problems with the test-suite
<jelmer> I've just commented on the MP
<pcjc2> He was also waiting for my signed copyright assignment, which I sent - might just have got missed
<pcjc2> thanks.
<pcjc2> Once that is in, I have some bug import XMLs I'd like to test on staging - is here a good place to ask about that, or #launchpad. I understand a LOSA needs to look at it
<mthaddon> you need an lp dev before a losa for that
<pcjc2> thanks
<sinzui> gmb, deryck: I am having a WTF moment in lp/bugs/mail/newbug.py near line 68.
<sinzui> gmb, deryck: "if event_creator is not None:" really? What can assign a user a bug and not be a creator in the event? Is there a poltergeist in the system?
<deryck> sinzui: hey, two secs to wrap a test I'm on and I'll look.
<sinzui> thanks
<deryck> sinzui: so you can assign yourself, no?
<sinzui> yes, but I would then be the creator in the event
<sinzui> deryck, The clause is used to add who assigned you to the bug rational. I was rewriting the message for a bug 670064
<_mup_> Bug #670064: Mail notification refers to "bug task" database schema <email> <lp-bugs> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/670064 >
<deryck> sinzui: yeah, it does seem odd.  I would guess there is *some* reason, some corner case, that isn't obvious.  Were I trying to find out, I would delete code and run tests and see what breaks. :-)
<sinzui> There is not test for it
<deryck> sinzui: no rational email test?
<sinzui> rught
<sinzui> deryck, I think this method I am in does a lot more than I supposed. I was thinking this is about the act of assignment, but the bug could be updated by a bug watch, and you will get an email about this because you are the assignee
<deryck> sinzui: I would add a test if it were me then.  I'm surprised we don't test that though.  I thought I fixed a bug in that and had a test
<sinzui> deryck, so the creator is only for the assignment case
 * sinzui might be able to test this in a few lines
<sinzui> no, the assignee is a new recipient. I am still confused
<deryck> sinzui: yeah, it's confusing to me too.  I would write a test to see what's happening.
<sinzui> yep
<deryck> allenap: see sinzui ^^.  is there really no test for rational message?  I can look more myself here shortly, just being lazy.... if you know :-)
<sinzui> deryck, There is one for the creator, not one for the other case
 * allenap reads
<sinzui> The problem text "a bug task for" is used twice in code and tests
<deryck> sinzui: ah, gotcha.  yeah, so you could add that pretty easily.  nevermind allenap, unping. sorry for the noise
<allenap> deryck: np.
<thumper> don't we have a DocTest matcher somewhere?
<lifeless> testtools.matchers.DocTestMatcher
<lifeless> sorry
<lifeless> DocTestMatches
<thumper> ta
<bdmurray> I'm getting an oops trying to update ubuntu's bug reporting guidelines
<bdmurray> "Hard limit of 1000 exceeded."
<thumper> oops
<thumper> bdmurray: file a bug :-)
<wgrant> danilos: Hi.
<wgrant> jcsackett: Still around?
<jcsackett> wgrant: around-ish. :-)
<wgrant> jcsackett: Bug #118284 is qa-bad... what's the issue with it?
<_mup_> Bug #118284: URLs ending with a ) aren't linkified properly <bad-commit-12102> <bugjam2010> <lp-web> <qa-bad> <tales> <Launchpad itself:Fix Committed by jcsackett> <Ubuntu:Invalid> < https://launchpad.net/bugs/118284 >
<jcsackett> wgrant there's a condition that introduces an OOPS in the branch. i have a fix branch ready and it passed a full ec2 run, but pqm is bouncing testfixes. or was.
 * jcsackett looks at buildbot
<jcsackett> do we go out of blocked for testfix once a a testfix is submitted, or once that build passes?
<wgrant> jcsackett: AFAIK as soon as a testfix is submitted.
<wgrant> jcsackett: It only affects the webapp right?
<jcsackett> correct; when you view bugs with certain comments it will OOPS.
<jcsackett> wgrant ^
<wgrant> jcsackett: OK, thanks.
<jcsackett> wgrant: you're welcome. encountering a related issue?
<wgrant> jcsackett: No, just really really wanting to roll out r12112 to cesium.
<jcsackett> wgrant: i'm resubmitting my fix to pqm. if it passes buildbot before i go to bed, i'll qa it so you can get that out posthaste.
<jcsackett> sorry for the delay.
<wgrant> jcsackett: Thanks.
#launchpad-dev 2010-12-21
<wgrant> qastaging takes forever to update :(
<thumper> poo
<wgrant> added: lib/lp/archivepublisher/tests/util.py.THIS
<wgrant> thumper: ^^
<thumper> eh?
<thumper> arse
<thumper> did that slip through?
<wgrant> Yes.
<thumper> poo
<wgrant> Just landed a couple of minutes ago.
<thumper> another pp
 * thumper fixorates
<thumper> wgrant: is it breaking things?
<thumper> it passed ec2
<wgrant> thumper: It shouldn't break anything, no.
<thumper> pqm-sumitting the fix
<wgrant> Thanks,
 * wgrant lunches.
<wgrant> Has anyone looked at the db-devel failure?
<lifeless> the real question is, has the failure looked at anyone?
<wgrant> Ah, it's already running again.
<wgrant> Yay buildbot.
<wgrant> lifeless: How do we fix a stale librarian PID issue on buildbot? Get a GSA to remove the file manually?
<lifeless> yes, checking that:
<lifeless>  - the librarian is not still running
<lifeless>  - *how* it happened - was the machine rebooted? did the librarian crash? was the test killed (and if so with what signal)
<lifeless> basically gather data so we can permanently fix
<wgrant> There was a test explosion in the last run.
<wgrant> zope.testrunner does not seem to rate very highly in the not sucking department.
<lifeless> go on, be more accurate ;)
<lifeless> wgrant: please make sure there is a bug on lp, high pri explaining what you figured out
<lifeless> wgrant: test explosions should always tear down
<lifeless> etc
<wgrant> lifeless: But Twisted tests dying with KeyboardInterrupt make me cry.
<wgrant> stub: Morning.
<stub> yo
<wgrant> stub: Can I please have you db-blessing for https://code.launchpad.net/~wgrant/launchpad/die-lucilleconfig-die/+merge/44305?
<stub> In the name of the Senate and Peoples of Rome
<wgrant> stub: Do I also need a TA-blessing?
<stub> Not before landing, no. Request 2 db reviews from me and lifeless. Land after you have the number and one approval.
<wgrant> I thought thinks might be different at the moment, given the lack of TA.
<wgrant> things.
<wgrant> Thanks.
<stub> I've put in the second review request
<stub> The two reviews thing is so we can both keep on top of things, but not block devs.
<stub> Holidays etc.
<wgrant> Right.
<poolie> hi
<poolie> stub, would you be kind enough to sponsor landing of https://code.launchpad.net/~mbp/launchpad/314507-oauth/+merge/44188 for me?
<poolie> iow to send it to pqm
<stub> poolie: Has it gone through ec2 test yet?
<poolie> no
<poolie> i guess to send it to pqm via ec2
<stub> Ok. Doing that now.
<poolie> thanks
<stub> Bug #692872
<_mup_> Bug #692872: Test suite fails if previous run did not tear down Librarian fully. <Launchpad itself:New> < https://launchpad.net/bugs/692872 >
<wgrant> lifeless: Oh?
<lifeless> wgrant: did you end up landing my librarianfixture branch ?
<wgrant> lifeless: No.
<wgrant> I was told that adeuring was working on that.
<wgrant> But that was nearly two weeks ago.
<lifeless> right
<lifeless> well it needs landing
<lifeless> the cure is worse than the disease
<lifeless> we need to stop doing expendient things and actually make the foundations sane and reliable
<wgrant> Sure.
<wgrant> I'll take a look at that tomorrow.
<wgrant> Once I work out how I am going to get this regression fix deployed.
<lifeless> qa the intermediate patches
<wgrant> Difficult.
<lifeless> qa isn't restricted to the author
<wgrant> I need r12112, but r12102 is bad.
<wgrant> It is needed on cesium, which r12102 does not affect.
<wgrant> So I need manual approval.
<wgrant> But the relevant team lead and project lead are on leave.
<lifeless> whats in 12102
<wgrant> A webapp formatter change.
<lifeless> ok
<lifeless> +1
<lifeless> the losas should remove cesium from nodowntime
<wgrant> lifeless: Thanks!
<wgrant> danilos: Should the variants rev I tried to QA last night block a rollout?
<wgrant> danilos: Or does it just not fix the bug?
<wgrant> It looks qa-ok to me, but I'd like to be sure.
<stub> lifeless: Does that branch relateto Bug #692872 ?
<_mup_> Bug #692872: Test suite fails if previous run did not tear down Librarian fully. <Launchpad itself:Won't Fix> < https://launchpad.net/bugs/692872 >
<lifeless> stub: very much so
<stub> Your comments don't say if your branch fixes the issue, or if we want the issue to continue.
<lifeless> oh
<lifeless> gimme a sec to find it
<stub> I just don't want buildbot to fail if a previous run left librarian crud on well known sockets or in well known files.
<stub> (which is why we are in testfix atm)
<wgrant> (Hudson! Hudson!)
<stub> wgrant: Won't hudson have the same problems?
<wgrant> stub: It doesn't have the same catastrophic failures, plus it uses Canonicloud and recreates the instances every so often.
<lifeless> ok
<lifeless> https://code.launchpad.net/~lifeless/launchpad/librarian/+merge/39013
<stub> Recreating the instances seems nice - pita buildbot requires manual intervention.
<lifeless> stub: it won't have this problem at all
<lifeless> because of the new instance thing
<lifeless> so that MP ^
<stub> lifeless: Cool. So the bug isn't won't fix, it is pending :)
<lifeless> well
<lifeless> let me reread
<lifeless> right
<lifeless> the proposed fix is inappropriate
<stub> Argh... midair collision
<wgrant> Hmmm.
<wgrant> Something is broken.
<spiv> wgrant: Is that something software?  That always breaks.
<stub> 5963 things are broken in LP
<spiv> It's a wonder anyone ever bothers to use software, really.
<wgrant> https://code.qastaging.launchpad.net/~wgrant/launchpad
<wgrant>    - Expression: <PathExpr standard:u'menu/mergequeues/render'>
<wgrant> KeyError: 'mergequeues'
<wgrant> I don't think that's changed lately :/
<stub> lifeless: Anything stopping the librarian branch landing (apart from being in testfix mode because the librarian branch hasn't landed yet)?
<lifeless> stub: I'd love it if someone would land it
<stub> I'll stuff it through ec2 then
<wgrant> It has some test failures.
<lifeless> I suspect it will have (some) failures / may need a merge with trunk
<wgrant> Or did last time I looked.
<wgrant> I don't remember exactly which. Which may indicate that it was catastrophic enough that I was unable to get a complete list.
<stub> I'll stuff it through ec2 test to get the list.
<wgrant> Not if it explodes :)
<stub> That bad huh?
<lifeless> its altering fairly fundamental stuff
<wgrant> IIRC yes.
<wgrant> The testrunner did not end up very happy at all.
<stub> Maybe it is a post-lunch job then
<wgrant> stub: There shouldn't be anything quite as terrible as the databasefixture branch.
<wgrant> Since we now know that everything uses the right config.
<lifeless> we're pretty close to being able to truely parallelise
<wgrant> Yup.
<lifeless> we need to track down the cause of leaks
<wgrant> Although I ran into some trouble with launchpadlib tests last night.
<lifeless> something is either not calling cleanups, or the process is being killed hard
<lifeless> launchpadlib needs to be fixed
<lifeless> where is that concurrent use bug
<wgrant> It seems to enjoy always connecting to :8085
<wgrant> I don't know how that can work, since AppServerLayer is meant to be dynamic now.
<wgrant> Oh.
<wgrant> I guess we don't do custom ports yet, just custom DBs.
<wgrant> And on Natty it decides to connect to :443 instead, just for fun.
<lifeless> win
<wgrant> So I will probably beat it to death with some monkeypatches.
<lifeless> wait what
<lifeless> we maintain it
<wgrant> Hmm, maybe I can get it to use a custom base URI.
<wgrant> lifeless: True.
<wgrant> But lazr.restful has at least one 3000-line doctest.
<wgrant> Which has sort of put me off that stack a bit.
<lifeless> boom shaka
<wgrant> Maybe I should try again.
<wgrant> Also, Python decided it would be amusing to change the default email header wrapping character from \t to ' '.
<wgrant> This breaks a lot of tests :D
<lifeless> OTOH the only thing that should be testing serialised forms is emaillib
<wgrant> lifeless: That is true, but there are a lot of other 'should' and 'should not's in LP.
<lifeless> wgrant: I think I mean 'cast the serialised form to an object, use a matcher, or delete the test
<wgrant> eg. you should probably not have a single monolithic 570KLOC Python app that can be logically split into lots of pieces :)
<lifeless> wgrant: I hold a rather different view on the 'logical split' thing
<lifeless> wgrant: I grant that there are /some/ pieces we can(and should) split out
<lifeless> wgrant: but I'm not convinced that there are lots; and certainly the UI->persistence->storage story is all one logical thing
<wgrant> At least Translations, Buildmaster and most of Soyuz have no business being in with the rest of LP.
<lifeless> no, yes, no
<wgrant> The interaction points between those three and the rest are minimal.
<wgrant> And will remain so.
<lifeless> to me there are two key tests
<lifeless> a) could it be written to APIs
<lifeless> b) would it be free of lockstep changes with the core
<lifeless> if the answers are yes and yes, then I think a separate piece makes sense
<lifeless> that isn't the case with translations or soyuz
<lifeless> and soyuz is (finally) getting reuse and working in better with code
<lifeless> translations ditto
<wgrant> I'm not sure that those interactions are significant enough to prevent a split.
<lifeless> I think they are massive impediments against splitting
<wgrant> They make it harder than it would be otherwise.
<lifeless> they are fundamental problems
<lifeless> that will massively outweigh any benefits from splitting them out, unless the split edge is designed to prevent those problems occuring
<lifeless> the latter problem in particular would mean terrible deployment and testing issues
<stub> AttributeError: type object 'BaseLayer' has no attribute 'config'
<stub> Think that means I should terminate the run :)
<lifeless> thats fallout from wgrants fixes to databasefixture
<wgrant> Indeed.
<stub> I'll look at it after lunch
<lifeless> wgrant: what was teh change
<wgrant> lifeless: I was going to tell him, but then he escaped.
<wgrant> I renamed config to config_fixture.
<wgrant> Or config_name.
<wgrant> Because the other one sprung into existence.
<wgrant> don't remember which was which.
<poolie> lifeless, would you care to offer an opinion on https://code.edge.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733
<poolie> ie about setting an rlimit in cron scripts
<poolie> not an emergency, just feeling a bit stuck
<poolie> or anyone else for that matter
<lifeless> poolie: uhm
<poolie> if you're not here, you don't have to, of course
<lifeless> ideally we'd give each task an even slice of the machine memory
<lifeless> which implies knowing the machine memory
<lifeless> I'd look to features to configure that
<lifeless> as for where its set
<lifeless> I don't care whether its in-proc or in-parent
<poolie> you could do a lot more towards this
<poolie> we could have a whole lep on the idea of resource caps on jobs
<poolie> istm perhaps you want a model of "this job should never need more than x GB unless something's wrong; this machine has Y GB; run y/x of them in parallel"
<poolie> or perhaps for other jobs, you just want to cap them
<lifeless> that seems liable to fail with too many jobs for the cpus
<poolie> oh, that too of course
<poolie> anyhow, ideally it would be an operational knob, not in the code
<lifeless> there are mulitple dimensions here
<poolie> the thing for this particular mp is
<lifeless> one is not using more resources than we have
<poolie> istm it is a step improvement to at least have a cap on this job as we have on others
<lifeless> another is autoconfiguration
<lifeless> poolie: sure
<lifeless> poolie: why don't you land it?
<poolie> and no worse than having losas manually kill them, as has happened at least once (maybe only once)
<poolie> i didn't want to land it over the top of aaron and tim's concerns
<lifeless> fair enough
<lifeless> anyhow, it seems to me that you'll want to use a mock
<lifeless> because setting an rlimit in a test and then butting up against it would be unreliable at best
<poolie> but i would like to restrict the scope
<poolie> i agree
<lifeless> if you're testing with a mock, it really doesn't matter where you set it, does it ?
<poolie> a mock of what? a monkeypatched setrlimit?
<lifeless> yah
<poolie> oh, i mean restrict the scope to smaller than "make a new job system thx"
<danilos> wgrant, hi, can you please run the rosetta-approve-imports script on dogfood please? I've made all the variant languages visible to see if that case would at least work
<wgrant> danilos: Does it need a fresh upload?
<danilos> wgrant, no, we've got plenty of unapproved files left
<wgrant> danilos: Running.
<wgrant> It's doing stuff.
<danilos> wgrant, cool, thanks
<danilos> wgrant, it puts stuff in arbitrary order so I can't find it in the queue
<wgrant> danilos: Should I knock all the old ones to some other status?
<danilos> wgrant, can you please kill the run and clean up all the TIQE entries with status != 5 (needsreview)
<danilos> wgrant, or that, if it's not a big bother
<danilos> wgrant, I believe status 3 is deleted so try with that :)
<wgrant> danilos: That's what I'm doing.
<wgrant> 1 -> 3
<wgrant> That was quick.
<danilos> wgrant, cool, thanks
<wgrant> Reapproving.
<danilos> cool
<danilos> wgrant, looks good so far, but please let it finish
<wgrant> Looks good, yeah.
<wgrant> danilos: It's finished.
<danilos> wgrant, oh, is it perhaps because of a time-limit? can you run it again please?
<wgrant> Still lots unapproved.
<wgrant> Indeed, ran for a few seconds over 5 minutes.
<wgrant> And now is doing lots more.
<danilos> wgrant, I've made "sr@ijekavian" hidden to see what will happen with those (if it works properly, they would stay unapproved, if not, they might again be approved in "serbian" directly)
<wgrant> danilos: The queue is not getting shorter.
<wgrant> It's doing dozens of transactions per second, and not running for more than 30 seconds or so.
<danilos> wgrant, right, then it's surprisingly working fine with hidden languages this time around as well
<wgrant> danilos: Do we want to do another upload of the same file to another series and try it again from the start?
<danilos> wgrant, you can perhaps kill that run and I can make "sr@ijekavian" visible again so we just make sure it works fine to clean up the queue (well, mostly: sometimes not all files have matching templates)
<wgrant> Sure.
<wgrant> It's finished again.
<danilos> wgrant, not really, I am experiencing some weirdness locally as well
<wgrant> So unhide.
<danilos> wgrant, done
<wgrant> I need to learn this Translations stuff.
<danilos> wgrant, I want to get down to the local weirdness first, and then when I am certain why is it happening we can try it all over again :)
<wgrant> danilos: Right.
<wgrant> So, is it qa-ok?
<danilos> wgrant, it's all very simple, and you have a head start compared to everybody else
<wgrant> danilos: It's running much slower this time.
<wgrant> Which may be a good sign.
<danilos> wgrant, yeah, it doesn't break stuff and works fine for when languages are not hidden
<wgrant> Indeed, queue is decreasing in size.
<danilos> wgrant, I'll file a new bug if I find more problems with it
<wgrant> danilos: Can you qa-ok the bug so I can request a deployment?
<danilos> wgrant, I did that already
<wgrant> Ah, great.
<wgrant> Thanks.
<wgrant> mthaddon: Around?
<mthaddon> wgrant: yup
<wgrant> mthaddon: We need r12112 deployed to cesium (to fix bug #692114).
<_mup_> Bug #692114: Recipe builds require indices for non-main PPA components <qa-ok> <recipe> <Launchpad itself:Fix Committed by wgrant> < https://launchpad.net/bugs/692114 >
<wgrant> mthaddon: There is one qa-bad rev in front of it, but that only affects the webapp.
<wgrant> lifeless has given me his blessing.
<mthaddon> erm, maybe, but that's completely non-standard...
<wgrant> It was done for Soyuz a couple of weeks ago... r12043, IIRC.
<lifeless> wgrant: it will need a incident report, I assumed you already knew that :)
<mthaddon> that doesn't mean we're okay to do it every time...
<lifeless> mthaddon: will you be at the epic?
<mthaddon> lifeless: nope - mbarnett will be there though
<lifeless> ok, I want to go through our exception-handling workflows face to face, can you perhaps make sure he's across all your concerns?
<danilos> wgrant, fwiw, 12111 is also still "needtesting"
<wgrant> mthaddon: Ah, OK.
<wgrant> mthaddon: I was under the impression that it wouldn't be that abnormal.
<wgrant> I guess I will wait.
<mthaddon> how critical is it?
<mthaddon> lifeless: I'm not really sure I understand the scope of the conversation you're after, but we can certainly cover anything over the phone that you need to
<wgrant> It breaks recipe builds into new PPAs or new series in existing PPAs, which is probably most new recipes.
<mthaddon> and how long has it been doing this for?
<mthaddon> and how long do we expect before we can get those revisions qa-ed?
<lifeless> mthaddon: that would be great
 * mthaddon nods
<wgrant> Well, buildbot is being more agreeable now, so it's not as long as I thought. Assuming that we don't get another qa-bad which immediately pushes us back another 24 hours :/
<wgrant> But these look OK so far.
<lifeless> mthaddon: scope wise - we have cherrypicks, but we're also doing some adhoc things while I've been on leave
<wgrant> lifeless: Er, don't we not have cherrypicks any more?
<lifeless> I'd like to consolidate things a bit, and talk cost-of-execution, latency, and approvals/risk analysis
<mthaddon> yeah, we've not done a cherry pick for a good while now
<lifeless> wgrant: we certainly do have them in policy
<wgrant> I thought we'd lost that capability.
<lifeless> wgrant: we haven't had to do many - but this soyuz working-around-unqaed-things is cherrypick-like
<lifeless> wgrant: not at all
<wgrant> lifeless: We need some way to roll out urgent changes that is not blocked by irrelevant bugs in the webapp, each of which blocks us for a day.
<lifeless> wgrant: perhaps.
<lifeless> wgrant: I'm not interested in drilling into this now. At the epic I'd be delighted to.
<wgrant> Sure.
<wgrant> I will say that the process simplification and frequent rollouts are great. It just seems to border on ridiculous to block another one-line critical fix for days.
<lifeless> so we're not meant to block for days ever
<lifeless> if there is something buggered, roll it back immediately.
<wgrant> But every time we have a qa-bad we block for 24 hours.
<wgrant> Which then gives us time for another one to slip in.
<lifeless> things are *meant* to be qa'd the same day they land.
<wgrant> => infinite chain of pain
<lifeless> wgrant: how it is taking 24 hours?
<wgrant> lifeless: The change is landed during the engineer's working day.
<wgrant> 4 hours of EC2 + 1 hour of PQM + 6 hours of buildbot takes us well after the end of their day.
<lifeless> and?
<wgrant> They QA it the following morning.
<wgrant> Notice that it's bad.
<wgrant> Send the rollback through EC2.
<wgrant> Another 11 hours later, it is in stable and can be QAd.
<lifeless> so, flaw one: don't wait for the engineer to qa it.
<lifeless> we're strongly encouraged to describe how to qa things in merge proposals.
<lifeless> 2) rollbacks go straight to pqm, no ec2.
<wgrant> s/rollback/fix/, then.
<lifeless> don't fix
<lifeless> rollback
<wgrant> Since avoiding rollbacks in DVCS\{darcs} is nice.
<lifeless> no, its insane.
<lifeless> rollbacks are how we undo mistakes rapdily.
<lifeless> there is no opprobobium in having something rolled back
<lifeless> and it removes all the latency involved in analysing and fixing the issue in the bad commit
<lifeless> wgrant: if something was broken, and I was working, I'd land a rollback with no hesitation at all
<lifeless> wgrant: if we *don't* do this, the expected result is a trunk that is broken a lot.
<wgrant> lifeless: So, it sounds like everyone needs to know three things: 1) QA quickly. 2) QA other people's stuff. 3) Rollback is the first resort, not the last.
<lifeless> wgrant: makes sense to me
<lifeless> wgrant: this has been communicated before, but it bears repeating.
<lifeless> wgrant: I'll be talking about this in my TA report too
<wgrant> Nobody treats QA as priority 1.
<wgrant> When in reality it probably should be.
<lifeless> wgrant: the logic behind this is simple; once a patch is in trunk, its on the critical path to deploy, anywhere.
<lifeless> The only thing higher priority than QA is fixing a production issue directly.
<lifeless> wgrant: this is why I was making such a big deal on your first? second? day about qaing when you had a blocked thing
<danilos> wgrant, fwiw, I figured what the problem is with "sometimes doesn't work": if we have paths stored in pofiles in the DB (due to the previous buggy behavior) it can wrongly take up a wrong pofile early on
<wgrant> danilos: Ah, great.
<pcjc2> allenap, deryck: Is there anything you need me to chase RE: Contributor agreement?
<pcjc2> (https://code.launchpad.net/~pcjc2/launchpad/allow-empty-comments/+merge/43449_
<pcjc2> )
<henninge> danilos: do you know what the problem might have been with bug 487137?
<_mup_> Bug #487137: Allow Rosetta admins to create custom language codes <bugjam2010> <lp-translations> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/487137 >
<henninge> danilos: Adi had a branch ready a month before his last comment on the branch (and unassigning himself from it).
<danilos> henninge, I was just looking at that and planning to get it landed
<henninge> danilos: ;)
<danilos> henninge, yeah :)
<danilos> henninge, if you want to take it over and go through it to ensure it's all still good, go for it ;)
<henninge> danilos: but what is Adi's last comment about? Is the branch flawed?
<danilos> henninge, well, last comment is from Michael, but just because of the status we should basically re-review it now
<henninge> danilos: sorry, Adi's last comment on the *bug*
<henninge> that's a month after Michael looked at the mp
<danilos> henninge, oh, I think it's easy: we can just let project owners administer their own custom language codes
<henninge> ;-)
<danilos> henninge, especially since there won't be any per-app teams anymore, I think that's the way to go
<henninge> good point
<danilos> henninge, who ever screws it up for themselves, well, they've done it themselves
<danilos> henninge, fwiw, we should make them able to do translations approval as well
<henninge> hah!
<danilos> henninge, it's just that we won't have the people to monitor it anyway, so we should just worry about being able to clean up later
<henninge> good thing it's just the two of us here ...
<danilos> henninge, heh
<danilos> henninge, anyway, is that how Adi's branch is working? (i.e. using TranslationsAdmin privilege directly)
<danilos> henninge, (if you looked at it)
<danilos> henninge, MP suggests it is
<henninge> sorry, was afk
<lifeless> danilos: I think in general per-app responsibilities will become interrupt-team responsibilities; its not that there is noone to do it, its that its no longer such a static group of people
<danilos> lifeless, and this is one of responsibilities that it'd be better to hand of to project owners than to hand of to that team, that's all I am saying
<lifeless> danilos: +1
<lifeless> danilos: I thought that perhaps you felt there wouldn't be staff to do it at all, rather than that project owners were a better natural fit
<lifeless> danilos: so I was trying to clear that up, was all.
<danilos> lifeless, well, the staff that will remain will not really be up to the task either, it's a complex work and you have to be careful (I am talking about import queue management) because it's easy to mess stuff up
<lifeless> danilos: is that an ops thing perhaps?
<danilos> lifeless, ideally, we wanted to transfer that to project owners only when we made it very hard to make mistakes, but I think it's better to let owners make mistakes now and be able to fix it to at least some extent than to have owners depend on a team that might make mistakes as well and depend on them to fix them afterwards
<danilos> lifeless, "ops" as in operation? yes
<danilos> operational, that is
<lifeless> danilos: I love the idea of being able to fix things rather than obsessing about preventing them
<lifeless> scales better
<lifeless> feels easier to use
<danilos> lifeless, the problem is that we can't really fix them atm
<danilos> lifeless, which is why the permissions are very restricted
<lifeless> danilos: sure, we have some work to do to get there
<lifeless> its too late for me to try to understand the details
<danilos> when I say "really", I mean "make sure the DB is in the best possible state"; it's not hard to make it appear relatively decent to outside users
<lifeless> would love to do so at the epic perhaps
<danilos> lifeless, yeah, sounds good, I guess thunderdome is a good place to do that
<allenap> pcjc2: I'll follow up on the contributor agreement now.
<allenap> pcjc2: I think my colleague might have been looking in the wrong place; your name is already on the signed list, and has been since the 17th. I'll land your branch now. Thank you, and sorry for the confusion :)
<pcjc2> Thanks!
<pcjc2> danilos: +2^n (n large), for letting projects manage their own translation imports - PRETTY PLEASE ;)
<danilos> pcjc2, if that +2^n translates into someone doing to work for cleaning up the mess that people can make, I'll go all-in on that bid :)
<danilos> pcjc2, but will do it even if that work is not done to the full extent
<pcjc2> Hard to know - would  some clean-uprequire a LOSA?
<pcjc2> or are you talking about extending LP code to allow more web-access to undo screw-ups?
<allenap> pcjc2: Is there a bug related to that fix?
<pcjc2> no, sorry
<allenap> pcjc2: Okay, I'll file one, just so we can track QA.
<pcjc2> ok, thanks
<deryck> Morning, all.
<danilos> pcjc2, it's about extending LP to allow more clean-ups and do some on it's own
<danilos> pcjc2, for instance, it's totally impossible to remove templates today, and when they are left around they just cause more problems for people
<danilos> anyone around who can help QA bug 670452, bug 619555 and bug 504080? (trying to get something rolled out, so need to go all the way to 12121)
<_mup_> Bug #670452: Hard to find related branches when composing recipe <lp-code> <qa-needstesting> <recipe> <Launchpad itself:Fix Committed by wallyworld> < https://launchpad.net/bugs/670452 >
<_mup_> Bug #619555: cronscripts/request_daily_builds.py is not verbose enough on default logging <bugjam2010> <canonical-losa-lp> <lp-code> <qa-needstesting> <recipe> <Launchpad itself:Fix Committed by thumper> < https://launchpad.net/bugs/619555 >
<_mup_> Bug #504080: Please put the URL to the merge proposal in the body of the email <bugjam2010> <code-review> <email> <lp-code> <qa-needstesting> <trivial> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/504080 >
<danilos> Ursinha, matsubara-afk: hi, where's the "specification" of all the bug tags that qa-tagger handles (I am wondering how do I tag a bug to indicate that one revision fixes a previous one)
<Ursinha> danilos, let me find the link for you
<danilos> Ursinha, I did find https://dev.launchpad.net/QAForContinuousRollouts though only through the mail archive, is there anything more complete?
<Ursinha> danilos, that's a bit of a mess... here's the page I know: https://dev.launchpad.net/QAProcessContinuousRollouts
<danilos> Ursinha, right, that one is much better
<danilos> Ursinha, so, I guess the right approach to landing this should have been a rollback and then a full fixed landing
<Ursinha> danilos, yes, I think so
<danilos> Ursinha, should we perhaps add a 'fixes-REVNO' tag as well? (just wondering, because if rollback=REVNO was included in this landing I am looking at it would have been sufficient even though it doesn't really roll the change back)
<Ursinha> danilos, the tag is bad-commit-firstrevno
<Ursinha> where firstrevno is the defective one you want to rollback
<danilos> Ursinha, well, look at bug 118284
<_mup_> Bug #118284: URLs ending with a ) aren't linkified properly <bad-commit-12102> <bugjam2010> <lp-web> <qa-ok> <tales> <Launchpad itself:Fix Committed by jcsackett> <Ubuntu:Invalid> < https://launchpad.net/bugs/118284 >
<danilos> Ursinha, 12102 is a bad-commit, tagged like that, and it was later fixed by 12121 (for the same bug)
<Ursinha> danilos, fixed or rolledback
<Ursinha> ?
<danilos> Ursinha, fixed
<Ursinha> so the first revision can be rolled out to production?
<Ursinha> 12102
<danilos> Ursinha, so, 12121 is good to go out, but nothing between 12102 and 12121 isn't because 12102 shouldn't go out without 12121
<danilos> Ursinha, no
<danilos> Ursinha, basically, regarding qa-tagger, it should behave exactly like rollback=12102 imo
<danilos> Ursinha, but I am guessing developers won't use that because it doesn't make much sense
<Ursinha> right, but now it doesn't
<Ursinha> yeah
<Ursinha> I guess the approach is a rollback then a full fixed branch
<danilos> Ursinha, the difference between this and rollback is that this requires QA, where rollback doesn't
<Ursinha> I get it
<danilos> Ursinha, yeah
<Ursinha> wondering how to proceed now
<Ursinha> well, if you already qaed the fix, removing the tag is safe
<Ursinha> right now deployments are blocked due to the bad-commit tag, so when you think that can land, just remove the tag
<Ursinha> danilos, would you mind filing a bug against qa-tagger about it?
<danilos> Ursinha, not at all
<danilos> Ursinha, bug 692978
<_mup_> Bug #692978: No way to mark a revision as fixing another bad revision <qa-tagger:New> < https://launchpad.net/bugs/692978 >
<Ursinha> danilos, merci
<abentley> rockstar, https://code.qastaging.launchpad.net/~abentley/bzrtools is giving a KeyError about "mergequeues".  Do you know anything about this? http://pastebin.ubuntu.com/546278/
<rockstar> abentley, no, I don't know what that's about.
<abentley> rockstar, okay, thanks.
<rockstar> abentley, I don't know if wallyworld is working on merge queues, but I didn't but anything about mergequeues on the branch index page.
<Ursinha> any soyuz people around?
<allenap> sinzui: Did I accidentally fix bug 56038?
<_mup_> Bug #56038: BugField should define some constraints <bugjam2010> <lp-bugs> <trivial> <Launchpad itself:Fix Released by allenap> < https://launchpad.net/bugs/56038 >
<sinzui> yes
<allenap> sinzui: \o/
<thumper> morning
<thumper> abentley: s = u'Hello \N{SNOWMAN}'
<EdwinGrubbs> StevenK, jelmer_, wgrant: can I assign this question to one of you? https://answers.launchpad.net/launchpad/+question/138604
<jelmer__> EdwinGrubbs: sure
<EdwinGrubbs> jelmer: is it possible to build a package from a branch yet, or do you still need to use dput?
<jelmer> EdwinGrubbs: That's an odd problem (The question)
<jelmer> EdwinGrubbs: You can build from a branch without a dput but you'll need a recipe, which is probably more work than a dput at this point.
<abentley> jelmer, I dunno about that-- we have a reasonable default recipe.
<jelmer> abentley: "bzr bd && dput ../foo.changes" is quicker (and doesn't require a multi-minute recipe build) than clicking create recipe, adjusting the default recipe and requesting a build imho.
<thumper> EdwinGrubbs: you can point the user to the help page
<jelmer> abentley: Don't get me wrong, I'm a big fan of recipe builds and I think we're as close as we've ever been to building from branch but it's still not quite as easy as clicking a "Build this revision as a package" button in the web UI.
<thumper> EdwinGrubbs: https://help.launchpad.net/Packaging/SourceBuilds
<EdwinGrubbs> thanks
<lifeless> gmb: hi
<lifeless> gmb: you might like lp:~lifeless/launchpad/persistence
<lifeless> gmb: I couldn't stop thinking about it, so I wrote down more science fiction.
<jcsackett> wgrant, you around?
<wgrant> jcsackett: Indeed.
<jcsackett> wgrant: just wanted to let you know that the fix on the branch/bug we chatted about yesterday has gone through, but the bad-commit-tag can't be removed b/c there are revisions between that need to be qa'ed.
<wgrant> jcsackett: Yeah, I saw that... but there's now *another* bad rev before the fix.
<wgrant> Just for added fun.
<jcsackett> wgrant: yeah, i just saw that qa-bad; it's weird, i qa'ed that this morning trying to get teh queue cleared out before i hit bugs i had no idea how to qa. thought that one looked good, but apparently someone more familiar with it saw issues i did not.
<jcsackett> anyway, just wanted to keep you up to date; sorry there is yet further difficulty in getting to your revision. :-
<wgrant> Breakage happens.
<wgrant> But our process cannot deal with it :(
<poolie> hello
<poolie> could somebody please send https://code.edge.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733 to pqm for me?
<wgrant> poolie: Has it been through ec2?
<poolie> no
<wgrant> Also, thumper just nak'd it.
<poolie> ok, fine
<poolie> thumper: so shall i just reject it?
<poolie> it was just a kind of drive by
#launchpad-dev 2010-12-22
 * thumper just found another mock logger
 * thumper sighs
<thumper> holy crap, two more logger classes slaughtered
<StevenK> thumper: How many is that now? Six?
<wgrant> https://code.launchpad.net/~registry/+junk/devel-1
<thumper> about four general loggers, and 6 mock loggers
<wgrant> How did that branch end up there?
<wgrant> Maybe it was attached to a deleted team.
<wgrant> (the registrant wants it deleted, but I want to investigate a bit first)
<thumper> and add another logger class
<wgrant> Fun.
<thumper> oh and another
<StevenK> Eight?
<StevenK> How many loggers does our codebase need?
<wgrant> MORE!
<thumper> and another
<thumper> and guess how many were in soyuz?
<wgrant> 99.9%
<wgrant> With the other one being in archivepublisher.
<thumper> wgrant: one was in the job test runner module
<thumper> but it would be a curious thing to count them all
<wgrant> :(
<thumper> found the extras by grepping for 'def debug'
<wgrant> Gnrrrrgh.
<wgrant> Why has that related branches rollback not landed yet......
<thumper> is there a way to just run all the tests that you've modified?
<wgrant> bzr st | sed | xargs ...
<wgrant> So, no.
<StevenK> thumper: I'd be curious to find out after you've finished how many lines you've removed and added
<thumper> well, changed a lot
<poolie> thumper: are you here?
<thumper> yes
<poolie> can we decide about the rlimit thing? it sounds like we should just reject that patch until somebody does a larger reconsideration of the job system?
<thumper> I think so
<poolie> ok
<thumper> until we know that a job killed by rlimit x number of times will stop trying to run
<poolie> shall we file a bug or something for that, or would it be too vague (or a dupe)
<thumper> I think we should file a bug
<thumper> I don't know of one
<thumper> something along the lines of "job attempts should be incremented on acquisition"
<poolie> ok, i'll do that
<poolie> perhaps also resource limits should be in the job framework and dynamically configured
<LPCIBot> Project db-devel build (242): FAILURE in 3 hr 39 min: https://hudson.wedontsleep.org/job/db-devel/242/
<wgrant> No traceback...
<thumper> poolie: yes, I agree that rlimits should be part of the job framework
<poolie> done, and rejected
<poolie> ok
<poolie> we'll see if it gets to the top of the losa pain stack
<poolie> it's probably not there now
<wgrant> thumper: The db-devel failure is translations stuff wanting one of your ex-loggers.
<wgrant> thumper: Do you want to fix, or shall I?
<thumper> error?
<wgrant> lib/lp/translations/tests/../doc/poimport-script.txt
<wgrant>       File "<doctest poimport-script.txt[26]>", line 1, in <module>
<wgrant>         from canonical.launchpad.scripts import FakeLogger
<wgrant>     ImportError: cannot import name FakeLogger
<thumper> I can fix
<wgrant> Thanks.
 * wgrant rolls back 12111, since abentley's rollback seems to have vanished.
<wgrant> thumper: Now that we are hopefully out of testfix, can I have your rs to rollback 12111?
<wgrant> It is qa-bad.
<lifeless> wgrant: you don't need reviews to do that
<wgrant> lifeless: Even though I'm not a reviewer?
<wgrant> OK.
<wgrant> Nooooo.
<wgrant> And this has missed the next buildbot run. Woo.
<wgrant> lifeless: This process is broken. Can we please discuss at the epic how to fix it?/
<lifeless> wgrant: its a separate process
<lifeless> wgrant: that bit isn't broken AFAIK
<lifeless> lp-land --rollback as pre the wiki page
<wgrant> lifeless: I mean the exceptional changes process.
<wgrant> At the moment it is a race against the bug jam's bugs jamming up deployment for days.
<wgrant> If we have one more broken revision, recipes remain broken until the 4th of January.
<wgrant> There is clearly something very wrong here.
<lifeless> we shouldn't be landing quite so many broken revs either ;)
<wgrant> Breakage happens. The process needs to deal with it.
<lifeless> i know
<lifeless> that was part of the original analysis
<lifeless> that we need to rollback rather than blocking for 4 weeks at a time ;)
<lifeless> the other option is to start uncommitting
<lifeless> which I'm a little less keen on
<wgrant> Even rolling back doesn't help.
<wgrant> Unless you QA before anything else lands.
<wgrant> Easy improvements to this particular case would be to have obvious extensions to the qa-tagger-based production change approval policy. eg. two broken webapp revisions shouldn't block a rollout to cesium for days, because that is insane.
<wgrant> But I do not know how to fix the general case.
<poolie> wow, this guy who keeps uploading his private key/s
<poolie> i'm kind of scared what will happen once he does get them going
<wgrant> It is slightly disturbing.
<wgrant> I've only seen the one so far, though.
<wgrant> Oh, there's another one.
<poolie> he attached at least two private keys (or maybe the same key twice) plus one revocation certificate
<wgrant> Missed that.
<wgrant> The revocation certificate is at least for one of those two keys, and not a third.
<lifeless> wgrant: rollback does help, it reduces the window from detection + fix + ec2 + pqm + buildbot
<lifeless> wgrant: to detection + pqm + buildbot
<lifeless> s/window/latency
<lifeless> the *window* becomes
<lifeless> detection + pqm
<lifeless> which should be very narrow if folk are doing QA as top priority.
<lifeless> wgrant: I'll be delighted to discuss at the epic, or before (in the new year)
<lifeless> wgrant: as for deploying delta'd versions: thats a high risk thing to do; I don't like the idea of it being defacto 'oh the webapp qa has 'issues''
<lifeless> wgrant: other layers are equally liable to fritz, and its not clear that we have the support framework to deal with N-ary delta skew *and* achieve a high velocity
<wgrant> Hmmm.
<lifeless> until everyone is qaing first, coding etc second
<lifeless> we haven't really tried the process completely
<wgrant> True.
<lifeless> I'm open to radical further changes, or whatever really, as needed to fiux.
<lifeless> but I'd like to prove that it cannot work easily and effectively first
<wgrant> Test suite parallelisation may make it all work a lot better.
<wgrant> As landing and QA will be possible within a working day.
<lifeless> as will lighter tests
<lifeless> my goal is 4 hour TTL for changes
<wgrant> At the moment only Soyuz can QA within 8-10 hours of landing :(
<lifeless> huh? what do you mean
<lifeless> do you mean 'only soyuz cannot' ?
<wgrant> Only Soyuz can, because we have DF.
<wgrant> Everyone else has to wait for EC2 + PQM + buildbot + qastaging
<lifeless> everything else has qastaging
<lifeless> df has to wait for buildbot too
<lifeless> qa staging is 15 minute cycle IIRC
<wgrant> qastaging updates every half an hour, but takes >50 minutes to finish.
<lifeless> mmm
<lifeless> I'm sure its higher freq than that
<lifeless> so bb is 4 hours
<lifeless> and ec2 doesn't count as 'after landing' time
<lifeless> latency for qa staging is ~5 hours IIRC
<wgrant> EC2 has to be included in the equation.
<lifeless> Anyhow, I'm aiming, as I said, at 4 hours from 'committed to live'
<wgrant> Right.
<wgrant> That will be excellent.
<lifeless> in a developer box
<wgrant> And may fix the QA issues.
<lifeless> I don't think its necessary to fix the qa isues
<lifeless> it may allow more slack without pathologies turning up
<lifeless> the basic problem is 'done done' in kanban, and I'll be writing about that in a fortnight
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       44 /  180  BugTask:+index
<lifeless>       24 /  225  Distribution:+bugs
<lifeless>       21 / 3328  Archive:+index
<lifeless>       15 /    5  Archive:+copy-packages
<lifeless>       12 /  112  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>        9 /    3  ProjectGroup:+milestones
<lifeless>        8 /   42  MailingListApplication:MailingListAPIView
<lifeless>        8 /   28  Distribution:+addquestion
<lifeless>        8 /   21  DistroArchSeries:+index
<lifeless>        6 /    5  NullBugTask:+index
<wgrant> poolie: answers.launchpad.net/launchpadlib isn't deliberately disabled.
<wgrant> poolie: It just needs someone to toggle the answers flag on (that flag has only been respected for a few weeks).
<LPCIBot> Project devel build (328): FAILURE in 3 hr 8 min: https://hudson.wedontsleep.org/job/devel/328/
<LPCIBot> * Launchpad Patch Queue Manager: [r=sinzui][ui=none][bug=683748] Replaces launchpadlib_for_anonymous
<LPCIBot> with an extension of launchpadlib_for
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=136870][bug=674546][bug=674546][bug=202136][bug=297531]
<LPCIBot> Fixed 'nominate for a series' text. and text in links and emails
<LPCIBot> * Launchpad Patch Queue Manager: [r=deryck][ui=none][bug=672619] Expose bug subscription filters via
<LPCIBot> the web service API.
<wgrant> StevenK: Hudson has filled up again :(
<LPCIBot> Project db-devel build (243): STILL FAILING in 3 hr 16 min: https://hudson.wedontsleep.org/job/db-devel/243/
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12128,
<LPCIBot> 12129 included.
<LPCIBot> * Launchpad Patch Queue Manager: [r=stub,thumper][ui=none][bug=45270] lucilleconfig is gone.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12121,
<LPCIBot> 12122, 12123, 12124, 12125, 12126, 12127 included.
<StevenK> It has?
<StevenK> /dev/xvda              16G  4.2G   11G  29% /
<stub> lifeless: http://paste.ubuntu.com/546482/
<wgrant> StevenK: At least the devel slave has.
<stub> lifeless: look familiar? Looks like the all (?) the remaining failures are launchpad vs. launchpad.dev
<wgrant> OSError: [Errno 28] No space left on device: '/tmp/sftp-test/branches'
<StevenK> wgrant: Right
<stub> lifeless: I'm assuming the tests here need to be fixed, because the port number is dynamic
<lifeless> that (sftp-test) needs randomisation too
<wgrant> This is a bit special.
<lifeless> wgrant: care to file a bug asking for it and add it to the parallelisation page?
<wgrant> Apache is currently eating 590% CPU.
<wgrant> lifeless: Is that just a tag, or are they listed somewhere on the wiki?
<lifeless> wgrant: there is a tag
<lifeless> wgrant: defined on the wiki page ;)
<wgrant> Heh, OK.
<lifeless> stub: uhm yes
<lifeless> stub: wallyworld landed a patch which provides expected values for this sort of thing
<wgrant> lifeless: There are quite a few other things that need randomisation, at least in Soyuz.
<wgrant> But it's possible as of yesterday, now that lucilleconfig is gone.
<lifeless> stub: he posted to the list a FAQ about it, and added it to the hacking guide
<lifeless> wgrant: \o/
<stub> lifeless: Do we have a preferred spelling for the fix landed already somewhere? Or should I make it up?
<lifeless> http://xkcd.com/837/# win!
<lifeless> stub: yeah, wally's thing
<stub> Can you be any vaguer? :-)
<lifeless> stub: [Launchpad-dev] Recap: do not use hard coded url ports in tests
<wgrant> lifeless: That's for librarian URLs too, not just webapp ones?
<lifeless> thats the thread subject
<lifeless> stub: so, this will need something analagous but not identical to what Ian did
<lifeless> stub: make it up :)
<lifeless> something like self.layer.librarian_download_url() or some such
<lifeless> OTOH a doctest is terrible for this sort of thing
<wgrant> Damn, now you're guilting me into not adding to the one I'm currently editing.
<StevenK> Hah
<wgrant> Do we have any UI reviewers outside the US at the moment?
<wgrant> I am trying to think of a reasonable UI description for the Archive.publish flag.
<lifeless> what does it do
<StevenK> "Will this archive be processed by the publisher" ?
<wgrant> It controls whether packages will be published to the user-visible archive disk.
<wgrant> Right.
<wgrant> I don't want that flag exported at all.
<wgrant> But I cannot convince Julian of that.
<wgrant> Maybe if I remove it he won't notice.
<lifeless> so
<lifeless> after building
<lifeless> and inclusion in the librarian
<lifeless> just the last step
<wgrant> Right.
<lifeless> so
<StevenK> wgrant: Exported over the API or the UI, or both?
<wgrant> StevenK: UI.
<wgrant> It makes sense for copy archives.
<lifeless> 'When disabled the APT archive for this PPA is frozen and will not receive updates.' ?
<wgrant> But not PPAs.
 * StevenK tries to run 'watch euca-describe-instances'
<StevenK> Er, tries to not run ...
<lifeless> 'Disabling this checkbox will freeze the APT archive for this PPA. No changes will be made to the archive until the checkbox is reenabled.'
<lifeless> wgrant: ^ is that accurate?
<wgrant> lifeless: That's accurate.
<wgrant> But doesn't fit our usual checkbox description style.
<wgrant> Perhaps I could prefix it with 'Update the APT archive.'
<wgrant> That might work.
<lifeless> southwestern-opaque ?
<StevenK> wgrant: Periodically update the APT archive
<lifeless> wgrant: invert the sense in the UI
<StevenK> With correct spelling
<lifeless> StevenK: the cron nature is irrelevant
<lifeless> wgrant: 'Freeze the APT archive. No changes will be made to the APT archive for this PPA while the checkbox is enabled.'
<lifeless> gets rid of the double negative
<wgrant> I am retitling Archive.enabled to "Accept and build packages uploaded to this archive.", and Archive.publish to "Publish uploaded packages to the APT archive.".
<wgrant> How does that sound?
<lifeless> publishing is jargon
<wgrant> Perhaps with an extra sentence on the second one to make the lack of updates clear.
<lifeless> I was trying to avoid the term
<wgrant> It is.
<lifeless> I think that inverting the sense is appropriate here
<wgrant> Possibly.
<lifeless> possibly for both.
<lifeless> but I'm less convinced about enabled.
<wgrant> I think enabled is OK.
<wgrant> I am inverting publish.
<wgrant> Mm, except it would be nice to avoid the word 'freeze'.
<wgrant> As that would be an overload.
<lifeless> true
<lifeless> halt
<lifeless> 'Halt updates to the APT archive. No changes will be made to the APT archive for this PPA while the checkbox is enabled.'
<wgrant> I can't refer to PPAs here, unfortunately. But I will work out something along those lines.
<wgrant> Thanks.
<lifeless> why not?
<wgrant> Because it's IArchive, not IPPA.
<wgrant> Copy archives are the major user of this flag.
<lifeless> It still knows what it is
<wgrant> This is Zope.
<lifeless> yes
<lifeless> it still knows what it is
<lifeless> shove a property on the view
<wgrant> In an attribute definition on an interface?
<lifeless> uhhhh
<lifeless> ok, I hates that
<wgrant> It's an autogenerated form.
<wgrant> It is nice, except yeah.
<lifeless> <- unconvinced by autogenerated forms
<shri> Hi all
<lifeless> hi
<shri> hi lifeless
<wgrant> lifeless: Autogenerated widgets are just about necessary.
<wgrant> lifeless: If your framework doesn't have them, you reimplement them quickly...
<wgrant> But they don't need to be as inflexible as Zope's.
<wgrant> Well then.
<lifeless> ohbau
<wgrant> Pardon?
<lifeless> humour, and a typo
<StevenK> O bai, as opposed to 'O hai' since shri said hi and then quit
<wgrant> Ah, I see.
 * StevenK moves on from euca-describe-instances to euca-get-console-output
<wgrant> That broken?
<StevenK> I've uploaded a new EMI, but firing up an instance of it hasn't started
<wgrant>     halt_archive_updates = Bool(
<wgrant>         title=_("Halt archive updates"), required=False,
<wgrant>         description=_(
<wgrant>             "Halt updates to the APT archive. Uploads and builds will "
<wgrant>             "continue, but packages not be made available to APT until "
<wgrant>             "updates are enabled."))
<wgrant> lifeless: ^^ how does that look?
<wgrant> Er, missed a word there.
<wgrant> s/packages/packages will/
<StevenK> That turns into a double-negative
<StevenK> Halting is true, therefore do nothing
<wgrant> It's what lifeless suggested.
<wgrant> And I think it makes more sense.
<wgrant> Since publishing is the more natural state.
<lifeless> wgrant: great
<lifeless> except
<lifeless> tweak:
 * StevenK shakes his head at the deployment report:
<StevenK> Revision 12102 can be deployed: qa-ok
<StevenK> Revision 12102 is bad: possible blocker.
<wgrant> StevenK: It has two bugs linked.
<StevenK> Ah, that bug still isn't fixed
<wgrant> One qa-ok, one qa-i-hate-recipes.
<lifeless> "Halt updates to the APT archive. Packages will not be made available to APT while updates are halted. While halted uploads are still possible and will be built."
<lifeless> wgrant: something like that
<lifeless> wgrant: I'm not sure the build/upload stuff should be mentioned
<wgrant> lifeless: Existing packages will still be available.
<StevenK> wgrant: Er? Both bugs are 'qa-ok' ?
<wgrant> Only new stuff won't be.
<lifeless> wgrant: yes, I know; I'm riffing on your text is all
<lifeless> wgrant: I think I preferred 'no changes' as the way to communicate
<lifeless> wgrant: but its up to you
<wgrant> StevenK: Ah, true now. But one has bad-commit-blah
<wgrant> lifeless: How about s/but packages will not be made available to APT/no changes will be available to APT/?
<StevenK> Ah ha. It's fixed in r12121
<wgrant> StevenK: Yeah.
<poolie> wgrant: oh i see
<wgrant> StevenK: Which is conveniently after 12111, which I rolled back in 12138
<wgrant> So we have 12102-12137 undeployable.
 * StevenK gets a rope
<poolie> i don't feel i should shift it
<wgrant> poolie: Should we just turn it on now, I wonder?
<wgrant> Heh.
<poolie> actually why not
<wgrant> There are lots of questions there.
<poolie> it's probably no worse than having people ask questions in ubuntu
<wgrant> Right.
<wgrant> I need to file a bug about that text.
<wgrant> And that form.
<StevenK> wgrant: I note 12131 is the last rev on the report
<wgrant> StevenK: Right. 12138 won't be blessed for another 5 hours.
<wgrant> After that I have 10 revs to QA and I may be able to get a deploy tomorrow night.
<StevenK> But r12111 is qa-bad
<wgrant> Right, that's the one I reverted.
<wgrant> 12102 is fixed in 12121, 12111 is reverted in 12138
<StevenK> Right, then r12119 is needstesting
<wgrant> Is that the MP email one?
<wgrant> Yes.
<StevenK> And then r12128-r12131
<wgrant> And 12132-12137
<wgrant> Although most of that latter set are easy.
<StevenK> Are they your death-to-lucilleconfig changes?
<wgrant> No.
<wgrant> They landed days ago, except for the db-devel change.
<wgrant> They are little UI tweaks.
<wgrant> Now to fight with the Zope form machinery. :(
 * StevenK grumbles at UEC
<wgrant> What's it doing?
<StevenK> Nothing, which is the problem
<wgrant> Yay.
<StevenK> steven@hudson:~$ euca-get-console-output i-4CB608E7 | tail -n 2
<StevenK> init: plymouth-splash main process (362) terminated with status 2
<wgrant> Hm, nice.
<StevenK> A sucessful boot of a different EMI also includes that line
<wgrant> :(
<lifeless> wgrant: my brain fails at 'no changes will be available to APT'
<lifeless> wgrant: I think the thing here is that APT reads an archive.
<lifeless> you can sayd 'the archive is not changed'
<lifeless> you cannot say 'APT receives changes'
<wgrant> The issue is that there are two archives.
<lifeless> *I* would talk about *changes to the archive* - because thats what it controls.
<wgrant> The one in LP.
<wgrant> And the one that apt sees.
<lifeless> wgrant: Thats why I said 'APT archive'
<lifeless> in my proposals
<wgrant> Right.
<wgrant> But I don't reallly want to say that twice.
<wgrant> Now I remember why I don't like LP UI work.
<wgrant> We don't really have standards, so my pedantry fails.
<lifeless> well
<lifeless> there are multiple dimensions here
<lifeless> I'm being pedantic on clarity
<wgrant> Right. And we can't have guidelines for that.
<lifeless> avoiding saying APT archive twice seems silly, to me, if repeating it is clearer.
<wgrant> At the moment I'm trying to clean up all the PPA forms.
<lifeless> cool
<wgrant> But I am being foiled by the lack of definition of form UI guidelines. Are field captions title or sentence case? Should their descriptions refer to "the archive" or "this archive"? ...
<wgrant> We have both throughout the app.
<wgrant> And it looks pretty scrappy :/
<wgrant> It's not that hard to make it look fantastic. I think we just need to sit down and define guidelines, preferably with a UI designer which we don't have.
<lifeless> uhm
<lifeless> ask mpt :)
<wgrant> Hm? What's he now?
<lifeless> interaction designer
<lifeless> he's still interested in lp
<lifeless> and we don't need authoritar, we just need someone we knowledge and reasoning skills in that domain
<wgrant> Hence only "preferably"
<wgrant> If we had one, they would probably be such a person.
<wgrant> Oh, mpt is even coming to the epic. Excellent.
<lifeless> why wait? ask today
<wgrant> Then I will have a couple of issues resolved and much less inclination to actually solve everything in January.
<lifeless> sounds like progress to me
<lifeless> do you know what your squad will be allocated to yet?
<wgrant> No.
<wgrant> Has that been determined?
<lifeless> some bits I'm sure of
<lifeless> the 2 project squads will be taking the oldest live projects and running with them from the stakeholder queue ( I don't remember what they are - privacy is one, I think. IMBW)
<lifeless> blah
<lifeless> 3 project squads
<lifeless> and the 2 interrupt squad duties are fairly well defined
<lifeless> I don't know if the initial allocation to interrupt/project is done yet or not
<LPCIBot> Project devel build (329): STILL FAILING in 3 hr 11 min: https://hudson.wedontsleep.org/job/devel/329/
<LPCIBot> * Launchpad Patch Queue Manager: [rs=wgrant][ui=none][rollback=12111] Revert devel r12111. It breaks
<LPCIBot> recipe creation for source package branches.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jcsackett,
<LPCIBot> sinzui][ui=sinzui][bug=526608] Ajaxify plus sign next to ajaxified
<LPCIBot> link for targeting a bugtask to a milestone.
<LPCIBot> * Launchpad Patch Queue Manager: [r=bac][ui=none][bug=321675] Changes the 'Delete' button when
<LPCIBot> unlinking a bug and a branch to "Remove link"
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=135009, 135012, 135015,
<LPCIBot> 135018] Adds autofocus to several fields needing it throughout
<LPCIBot> Launchpad
<LPCIBot> Project db-devel build (244): STILL FAILING in 3 hr 18 min: https://hudson.wedontsleep.org/job/db-devel/244/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][rs=thumper] Fix the FakeLogger import.
<stub> What is the silly name of that test results viewer again?
<wgrant> tribunal?
<stub> thats the onbe
<stub> lifeless: of course, with the restricted URL stuff we have no choice but to invoke client.getURLForAlias, which makes a number of the tests in librarian.txt somewhat redundant...
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | BUG JAM! http://mumak.net/lp-bugjam-2010/ 100 so far! | New starter this week: wgrant  | PQM open for business | firefighting: - | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> one hundred!
<wgrant> It was 101 earlier :(
<jml> bugs get reopened, I guess.
<jml> or my script is buggy
<wgrant> jml: You made it home?
<jml> wgrant: indeed I did. I'm in my London home right now.
<jml> wgrant: also, welcome to the cabal. I trust you've received your codex and binding ring, and have been taught the seven words.
<wgrant> jml: Indeed, all those formalities are complete.
<jml> wgrant: glad to hear it :)
<wgrant> I think victory may be near.
<jml> wgrant: oh?
<wgrant> jml: There may yet be hope for deploying the fix for bug #692114 before next year.
<_mup_> Bug #692114: Recipe builds require indices for non-main PPA components <qa-ok> <recipe> <Launchpad itself:Fix Committed by wgrant> < https://launchpad.net/bugs/692114 >
<jml> wgrant: heh
<pcjc2> gmb: Thanks for merging that branch ;)
<jml> I'd been meaning to ask what our deployment options were. We're still able to do non-downtime ones, right?
<wgrant> jml: It was discovered on Saturday and the fix and test took less than two minutes... but deployment is problematic because of a chain of qa-bad revisions.
<jml> :(
<wgrant> In the window between a qa-bad revision and its reversion, another qa-bad revision can spring up :(
<pcjc2> https://bugs.launchpad.net/launchpad/+bug/692951 shows fix committed - how long before its on production machines? (e.g. staging?) so I can get some test bug imports done?
<_mup_> Bug #692951: Don't show <empty comment> placeholders for imported bug attachments <qa-needstesting> <Launchpad itself:Fix Committed by pcjc2> < https://launchpad.net/bugs/692951 >
<wgrant> pcjc2: It could be on staging now.
<wgrant> Let me check.
<pcjc2> still qa-needstesting
<wgrant> staging is not production. It updates automatically, regardless of QA.
<wgrant> It is used for QA.
<pcjc2> ok, excellent..
<pcjc2> Who should I ask nicely to do a couple of imports for me?
<wgrant> pcjc2: Staging has the fix.
<pcjc2> And it runs a clone / snapshot of the normal production database?
<pcjc2> seems our projects are present
<wgrant> We can probably go directly to a LOSA.
<wgrant> Right, it runs an old snapshot of the production DB.
<pcjc2> Have two xml files here.. 6.1M and 9.8M
<pcjc2> pre-validated on a local dev instance of course ;) - Although the gEDA one has two failures, which I'm happy with -  if preferred, I could just edit out those bugs
<wgrant> pcjc2: Do they have empty comments?
<wgrant> Oh, of course, that's why you wanted staging updated.
 * wgrant headdesks.
<wgrant> Excellent.
<pcjc2> some did
<wgrant> So it can double as QA for the bugfix.
<pcjc2> The only one which failed (and an identical duplicate) was one where someone pasted an entire build log for a program into the bug description
<pcjc2> Nice
<pcjc2> As it happens, it was filed against the wrong project, so got promptly closed. Hence - I don't care they are dropped with a validate error by the import script. (Validate error is because the description is too long)
<wgrant> Ah, right.
<pcjc2> I can find the empty comment ones. Actually.. none have a truly empty comment.. the ones in question had attachments - so the lack of an  "<empty comment>" appended is a QA of the patch having its desired effect
<pcjc2> We can probably run some dummy imports with actually empty comments if you like..
<jml> I'm afk for a bit to get coffee etc.
<wgrant> pcjc2: Where are those two files?
<pcjc2> not uploaded anywhere at the moment.. too big to email, and I don't have any web space I can get at with quota left!
<pcjc2> Is there somewhere temporary I can stick them on LP?
<wgrant> They're not *that* big.
<wgrant> Uh, good question.
<pcjc2> I found some space.. give me a minute
<wgrant> Great.
<pcjc2> also - if I gzipped them, they would probably end up tiny... but I have the space now. (Obviously too tired today)
<wgrant> Heh.
<pcjc2> http://www2.eng.cam.ac.uk/~pcjc2/geda_output_v1.xml
<pcjc2> to import against "-p geda"
<pcjc2> http://www2.eng.cam.ac.uk/~pcjc2/pcb_output_v2.xml
<pcjc2> to import against "-p pcb"
<pcjc2> My quota on that system (with the webspace) is only 512M, including all cache-files, work etc..
<allenap> 100 bugs closed so far \o/ http://mumak.net/lp-bugjam-2010/
<pcjc2> wgrant: I see the gEDA ones are importing nicely / have been imported (can't remember our total count)
<pcjc2> Same as my local tests though.. they are all marked with full bug heat.. any idea why?
<wgrant> pcjc2: Let me have a look.
<wgrant> It may be that the import script doesn't change the cached max heat value on the project.
<pcjc2> so it will auto-update at some point?
<wgrant> pcjc2: I guess they probably all have identical heat values.
<wgrant> Because they probably only have one subscriber each.
<wgrant> And none of the other heat factors apply.
<pcjc2> ah, ok
<wgrant> Do you know of a bug where we can check the empty comment behaviour?
<wgrant> pcjc2: I just subscribed to a pcb bug to see if the heat calculation was working, and it seems to be.
<wgrant> All except that bug now have just three flames.
<pcjc2> ok, great. We just need to do a little triage I guess
<wgrant> Heat is deliberately not very affected by triage.
<wgrant> You can see the metrics involved by clicking on the flames on the bug page.
<pcjc2> I meant by subscribing appropriate developers really
<pcjc2> or getting people to rate the bug with "affects me"
<pcjc2> Bug hasÂ notÂ been active* in within the past 24 hoursSubtract 1% of the bug heat score for every day of inactivity
<pcjc2> obviously this is calculated iteratively?
<pcjc2> otherwise all our historical bugs would have low bug heat I guess
<pcjc2> no.. no, it does things the sensible way... total_heat = int(total_heat * (0.99 ** days_since_last_update))
<pcjc2> although I wonder if it is looking at the date of import as the last update date
<wgrant> Yeah, it probably is.
<wgrant> That's slightly unfortunate.
<wgrant> pcjc2: Which bugs had the empty comments?
<pcjc2> almost anything with an attachment _previously_ would import with "<empty comment>"
<pcjc2> let me find an example
<wgrant> Both imports are done now.
<pcjc2> ;)
<pcjc2> net connection is a bit flaky here
<pcjc2> https://bugs.staging.launchpad.net/pcb/+bug/692470
<pcjc2> If you had access to an instance _not_ running the fix
<pcjc2> you would see it imports with <empty comment> for each of those attachments
<wgrant> Yep, I've seen that before.
<wgrant> That is a lot of attachments.
<wgrant> Looks good, thanks.
<pcjc2> I added a unit test which checks the old behaviour for a truly empty comment
<pcjc2> How much (if any) extra QA is required?
<wgrant> I marked the bug qa-ok a couple of minutes back.
<wgrant> It all looks good to me.
<pcjc2> nice, thanks!
<pcjc2> How long until it hits production?
<wgrant> I'm just working out how to QA the last couple of items.
<wgrant> Then we can hopefully deploy as soon as that's done.
<pcjc2> cool. I've sent an email to the gEDA lists requesting people take a look at things and get back to me with our own QA on the bug import
<pcjc2> I've found some breakage already.. https://sourceforge.net/tracker/?func=detail&aid=2922765&group_id=73743&atid=538811  ->   https://bugs.staging.launchpad.net/pcb/+bug/692282
<_mup_> Bug #692282: package nspluginwrapper 1.2.2-0ubuntu6.9.10.1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 135 <amd64> <apport-package> <nspluginwrapper (Ubuntu):New> < https://launchpad.net/bugs/692282 >
<pcjc2> &#226;&#8364;&#732;elementColor (color)&#226;&#8364;&#8482;
<wgrant> Ahhh XML, I love you so.
<pcjc2> probably just need to grep for failure like that and convert in my script
<pcjc2> The line I just pasted was the SF export XML. It got munged by the time it got to output.xml
<pcjc2> Ã¢â¬ËelementSelectedColor (color)Ã¢â¬â¢
<wgrant> Fun.
<pcjc2> So LP is not to blame.. just the export script I think
<pcjc2> I've got a pile of changes / extensions / fixes to return for that already
<pcjc2> @ _mup_: Naugty bot.. learn to notice which server I'm talking about ;)
<pcjc2> got to head out for a bit.. back later
<wgrant> allenap: Are you going to be able to QA that structural subscriptions API thing? Although I did the initial export of that part of the code, I don't know the new stuff, so I'm not entirely comfortable that I can QA it adequately.
<allenap> wgrant: I'm trying to QA it at the moment. It seems to work okay, but I want to check if the bug mail was sent correctly. BTW, what do you mean by "initial export of that part of the code"?
<wgrant> allenap: Oh, I thought it was structural subscriptions. It looks like I was mistaken.
<allenap> wgrant: It is structural subscriptions: bug filters for structural subscriptions. The Debian BTS stuff is unrelated.
<LPCIBot> Project db-devel build (245): STILL FAILING in 3 hr 33 min: https://hudson.wedontsleep.org/job/db-devel/245/
<deryck> Morning, all.
<jml> good morning deryck
<wgrant> Morning deryck.
<jml> deryck: how goes the per-package dupe disabling thingy?
<deryck> jml: 'tis done.  I sent email to the stakeholders on the LEP for feedback, they're happy, but I never followed up on the stakeholders list that it was done.
<jml> deryck: ahh cool.
 * jml updates the board
<deryck> jml: JFo is still concerned that they will need per package ability to turn off 'mark as duplicate' link except for bug supervisor, so I'm gathering data to see if that is really a widespread issue or not.
<jml> deryck: fair enough.
<Ursinha> hey jelmer, you're a soyuz guy, right?
<jelmer> Ursinha: Yep!
<Ursinha> cool :)
<Ursinha> jelmer, I'm seen lots and lots of oopses in the soyuz report, like this one: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1812PPA1221
<Ursinha> do you know what's this?
<Ursinha> I've seen
<wgrant> Ugh, yes, I really should fix that.
<jelmer> That's the new apache access log parser
<wgrant> They are ignorable.
<Ursinha> ah, right
<jelmer> oh, wgrant's still up :-)
<wgrant> Barely.
<henninge> danilo_, danilos: which one are you? ;)
<danilo_> henninge, make your pick (uhm, only this one)
<henninge> danilo_: Do you remeber trying to fix bug 401618?
<_mup_> Bug #401618: Remove [I]POTMsgSet.sequence <lp-translations> <message-sharing> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/401618 >
<henninge> danilo_: Did you give up on the stepthrough approach?
<danilo_> henninge, I am pretty sure we gave up on it with the idea that we'll get new +translate page and underlying structures
<henninge> ;)
<henninge> *that* is too much for a bug jam ...
<danilo_> henninge, yes, it is :)
<henninge> danilo_: how did you think to use stepthrough here? I think it would mean changing URLs for translation messages.
<danilo_> henninge, it means having a stepthrough method implementation which goes through parent (pofile) to get the right sequence number
<danilo_> henninge, we definitely shouldn't change the URLs though
<danilo_> henninge, iow, this doesn't qualify as a good candidate for the bugjam consider we are not certain what needs to be done
<henninge> danilo_: yes, but stepthrough takes a constant string (like "+tm") and we don't have that here.
<henninge> yes, I agree.
 * henninge looks for more places to remove cruft
<pcjc2> What does the "tech-debt" tag mean?
<henninge> maybe continue on the product-to-project quest?
<danilo_> henninge, if that's easy enough
<henninge> pcjc2: technical debt is stuff we coded badly for time reason and promised to fix later ...
<henninge> boy, are we in debt .... ;)
<pcjc2> ah, cool. Wondered if it was pay-back tasks for people who'd had tech-training from Canonical or something ;)
<henninge> interesting idea ... ;)
<pcjc2> Most of my code could be considered tech-debt! I'm wading through a 40+ commit patch series which re-writes a CAD packages rendering stack in OpenGL
<pcjc2> Not fun
<pcjc2> Not fun when "it works", but I can't bear to commit it as is. (This is about the 3rd refactor / re-write though!)
<danilo_> henninge, btw, you can always directly implement traverse()
<danilo_> pcjc2, the joy of free software is getting over the hump of embarrasment: code that works and is ugly is usually of bigger value than no code at all :) of course, with long-living projects, you do have to worry about maintainability as well
<pcjc2> Since I'm usually the one playing big-bad grumpy patch reviewer, I need to make clean commits
<pcjc2> Code is all published and people use it.. its just not "upstream" yet
<danilo_> pcjc2, multiple branches help with that: non-clean branch that implements this cool feature and a master that is eager to get that feature will push you to get it cleaned up while letting people get at the feature if they really want to :)
<danilo_> pcjc2, there you go, same thoughts then :)
<pcjc2> indeed
<pcjc2> I have ~8 branches here for the circuit board CAD (PCB), ~8 for the schematic CAD (gEDA)
<pcjc2> Upstream is mostly just me and a couple of other active guys. (~1 per project) Things have become a little stagnant in terms of new development
<LPCIBot> Project db-devel build (246): STILL FAILING in 3 hr 14 min: https://hudson.wedontsleep.org/job/db-devel/246/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12137,
<LPCIBot> 12138 included.
<james_w> rockstar, hey, I was going to fix bug 666979 if you could help me decide on the best solution
<_mup_> Bug #666979: soupmatchers.Tag takes a text argument that should ignore whitespace <soupmatchers:New> < https://launchpad.net/bugs/666979 >
<jml> thumper: you around today?
<thumper> jml: no
<jml> thumper: thanks. enjoy your holiday.
<jcsackett> lifeless ping
<jcsackett> nm.
<lifeless> sinzui: if you know what jc wanted, let me know? :)
#launchpad-dev 2010-12-23
<LPCIBot> Project devel build (330): STILL FAILING in 3 hr 32 min: https://hudson.wedontsleep.org/job/devel/330/
<wgrant> sinzui: Around?
<mbrigdan> So, this is probably some dumb error on my part, but when I do "make schema", it spits out a giant python traceback that ends with "ImportError: cannot import name boolean". What should I do?
<wgrant> mbrigdan: That's rather odd. Could you pastebin the full traceback?
<mbrigdan> sure thing, just give me a second
<mbrigdan> Here you go: http://pastebin.com/kpQi2Gsi
<wgrant> mbrigdan: Where did /usr/lib/python2.6/dist-packages/_xmlplus come from?
<mbrigdan> I'm pretty sure i've never seen that before today. Strange
<mbrigdan> huh, google says it may be related to python-xml, which was removed at some point
<mbrigdan> And I still have it installed. That might be a problem
<mbrigdan> Removed it, and now it looks like its working. Cool
<wgrant> Excellent.
<mbrigdan> ugh, sorry to keep bothering you, but now make run gives me a rather smaller traceback, ending with "dent authentication failed for user "launchpad_main"". Looking at google, this happened to someone else because they have two versions of postgres installed, but I'm certain I only have one.
<mbrigdan> sorry, s/dent/ident
<wgrant> mbrigdan: Have you run utilities/launchpad-database-setup?
<mbrigdan> yes, but I'm running it again, along with make schema to see if that fixes anything
<mbrigdan> huh, still not working. I just do "./utilities/launchpad-database-setup $USER", right?
<wgrant> That should do it.
<wgrant> Can you run 'psql postgres' successfully?
<mbrigdan> yeah, seems to work fine
<wgrant> You're running 'make run' as the same user?
<mbrigdan> yup
<mbrigdan> and echo $USER gives my username, just incase something fishy was going on
<wgrant> Does 'psql -U launchpad_main launchpad_dev' work?
<wgrant> mbrigdan: ^^
<mbrigdan> no, it fails with: psql: FATAL:  Ident authentication failed for user "launchpad_main"
<wgrant> mbrigdan: Run "psql postgres", and "SELECT * FROM pg_user WHERE usename='YOUR_USERNAME';"
<wgrant> Is usesuper true?
<mbrigdan> usesuper has a t underneath it, so, not knowing anything about postgres, I would assume yes.
<wgrant> Indeed.
<wgrant> What about usename='launchpad_main'?
<mbrigdan> usecreatedb, usesuper, and usecatupd are all false, whatever they are. Should I got about changing this?
<wgrant> No, that's fine. I was mostly wondering if the user actually existed.
<wgrant> Which version of Ubuntu are you using? 10.10?
<mbrigdan> err, that's maverick right?
<mbrigdan> if so then yes
<wgrant> That is.
<wgrant> Does /etc/postgresql/8.4/main/pg_hba.conf have 'local   all         all                           trust' and 'host    all         all         127.0.0.1/32      trust' lines in it?
<wgrant> If so, perhaps try restarting postgres.
<wgrant> Since I can't think what else it could be.
<mbrigdan> yes it does.
<mbrigdan> How would I restart postgres?
<wgrant> sudo service postgresql restart
<mbrigdan> When I restart postgres (or start/stop/anything else for that matter), it gives me Insecure directory in "$ENV{PATH} while running with -T switch at /usr/bin/pg_ctlcluster line 63" and then "FAIL". But it still seems to be working. Could that be the problem?
<wgrant> I've never seen that before.
<wgrant> Is there anything interesting in /var/log/postgresql/postgresql-8.4-main.log?
<mbrigdan> huh, this is in there: MST LOG:  provided username (launchpad_main) and authenticated username (matthew) don't match
<mbrigdan> followed immediately by MST FATAL:  Ident authentication failed for user "launchpad_main"
<wgrant> Hmm, so it is indeed using ident rather than trust.
<wgrant> The 'local   all         all                           trust' line in pg_hba.conf should make it work...
<wgrant> Oh, unless it's disabling it because it thinks the directory is too insecure.
<wgrant> Who owns /etc/postgresql/8.4/main and its contents?
<mbrigdan> user and group is all postgres
<mbrigdan> although, I have added other things to my personal $PATH, could that be it?
<wgrant> Ahh, that's probably it. The restart is failing, not postgres itself.
<wgrant> And I bet it failed the same way when launchpad-database-setup tried to do it.
<wgrant> Either fix PATH, fix that directory's perms, or use sudo -i.
<mbrigdan> huh, sudo -i still fails
<mbrigdan> What would I need to change the permissions to? Because I think I should already be the only one with access
<wgrant> I can't work out what generates that error message, but it sounds like there is a world-writable directory in $PATH.
<wgrant> And if it's still there after sudo -i, you've probably chmodded one of the default one.
<wgrant> +s
<mbrigdan> oh wait, I think apt started to complain about something after I updated to maverick, if only I could remember what it said
<mbrigdan> well, I found at least one problem, /usr/bin is world-writeable. Thankfully, I'm the only one to ever use this computer
<wgrant> Hah.
<mbrigdan> Oddly, I remember apt complaining about something similar after my last upgrade too, but then it just went away
<mbrigdan> and what d'ya' know, sudo -i now get postgres to restart perfectly fine
<wgrant> If you don't remember making that change yourself, I would be hesistant to trust your machine much further.
<mbrigdan> The thing is, I don't remember doing it, but my machine likes to do funny things on dist updates
<mbrigdan> like installing apache
<mbrigdan> every time
<mbrigdan> for no reason
<wgrant> Nice.
<mbrigdan> and I haven't ran any software that hasn't come from the repos, so I'm hopefully good
<mbrigdan> and, as a high school student, who backs up everything to external drives, I don't have that much that I would feel bad about losing
<mbrigdan> other than a bunch of music which I should have on CDs somewhere anyways
<wgrant> You would be one of the few such people who regularly back up, I suspect.
<mbrigdan> hah, probably. After a few VERY close calls, I set up rsync to do it in the background once a day.
<mbrigdan> I've discovered some rather interesting ways to corrupt data
<sinzui> hi wgrant
<wgrant> sinzui: Hi. Do we have form design guidelines somewhere?
<sinzui> I do not think so...
<mbrigdan> sweet, make run works now. Thanks for putting up with all of my strange problems man.
<sinzui> There were many rules, but nothing in a single place
<sinzui> wgrant, I think many of our rules are embodied in the Canonical web design guildelines that we do not *officially* use
<wgrant> sinzui: I didn't realise that we had web design guidelines.
<wgrant> Perhaps my initial investigations of the Canonical wiki were inadequate.
<sinzui> wgrant, http://design.canonical.com/the-toolkit/guides-for-websites/
<sinzui> ^ I have been trying to reconcile Canonical + Ubuntu with Lp
<wgrant> Aha
<sinzui> Colour is difficult, font-sizes impossible, but forms look like beuno's rules
<wgrant> sinzui: That mostly covers visual design.
<wgrant> I am more concerned at the moment about textual content.
<wgrant> eg. some field captions use title case, others sentence case.
<sinzui> ahh
<sinzui> yes that was once in many documents
<wgrant> Description style also varies considerably.
<wgrant> And even those with stylistic similarities differ in subtle yet jarring ways. For example, on how they refer to the object on which they operate ("the archive" vs. "this archive").
<wgrant> It is all disturbingly inconsistent.
<sinzui> wgrant, :( I think we was missing a page in the wiki
<wgrant> sinzui: dev.launchpad.net/UI has lots of links.
<wgrant> But they are all broken.
<sinzui> I recall something about the form slots
<wgrant> And I cannot find anything on wiki.canonical.com or launchpad.canonical.com.
<sinzui> I am looking at the later and do not see them
<sinzui> There was a page about the extra form slots, another about writing the title and description in the interface
<wgrant> I do not recall ever having seen those.
<wgrant> Which suggests that they were not public.
<sinzui> right
<sinzui> I recall a rule about labeling fields as optional. Then you reported a bug that we have forms where we label the field as required
<wgrant> That was quite some years ago.
<wgrant> I presumed at the time it was because of differing form libraries.
<sinzui> wgrant, I do not see anything. from the past. I think we need to focus on the future. U1, Lp, and Ubuntu should used the same rules. Maybe they have rules for us
<wgrant> sinzui: The Ubuntu and Canonical guidelines always seemed to focus on the visual side of things rather than interaction.
<wgrant> Which makes them unuseful for Launchpad, which must maintain a distinct visual identity.
<wgrant> I fear that the guidelines we seek may not in fact exist.
<sinzui> The rules of spacing and text are based on usability tests. They are thing Lp is ignoring
<sinzui> The rules of interaction for Lp, U1 and Ubuntu should be the same.
<sinzui> Maybe beuno developed something for U1
<LPCIBot> Project devel build (331): STILL FAILING in 3 hr 30 min: https://hudson.wedontsleep.org/job/devel/331/
<thumper> YAY \o/
<thumper> 2nd logger branch landed
 * thumper EOYs
<LPCIBot> Project db-devel build (247): STILL FAILING in 3 hr 36 min: https://hudson.wedontsleep.org/job/db-devel/247/
<LPCIBot> Project devel build (332): STILL FAILING in 8 min 28 sec: https://hudson.wedontsleep.org/job/devel/332/
<LPCIBot> Project devel build (333): STILL FAILING in 0.47 sec: https://hudson.wedontsleep.org/job/devel/333/
<wgrant> WTF Zope.
<StevenK> $ bzr revision-info -d /mnt/test/launchpad/workspace/devel
<StevenK> null
<StevenK> Whee
<spiv> StevenK: Hmm, not "0 null:"?
<wgrant> Wow, we've really closed 208 bugs?
<LPCIBot> Project db-devel build (248): STILL FAILING in 3 hr 15 min: https://hudson.wedontsleep.org/job/db-devel/248/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12140
<LPCIBot> included.
<danilos> wgrant, hi, just wanted to check if you have landed a fix for the interfaces for bug 685624? I didn't see any separate commits, but didn't have the time to look at it fully
<_mup_> Bug #685624: Translation template build jobs should use the new BuildFarmJob <bugjam2010> <lp-soyuz> <lp-translations> <qa-ok> <Launchpad itself:Fix Released by danilo> < https://launchpad.net/bugs/685624 >
<wgrant> danilos: Totally forgot about that, sorry.
<wgrant> Will do that now.
<danilos> wgrant, ok, thanks
<wgrant> danilos: Should I reuse that bug?
<danilos> wgrant, yeah, please
<LPCIBot> Project devel build (334): STILL FAILING in 3 hr 29 min: https://hudson.wedontsleep.org/job/devel/334/
<wgrant> danilos: Could you bless https://code.launchpad.net/~wgrant/launchpad/bug-685624-ttbj-build-interface/+merge/44558?
<danilos> wgrant, blessed may it be!
<wgrant> danilos: Thanks.
<henninge> danilos: poimport.txt has a section called "Cron Scripts" that was either deleted in db-devel or added in devel. It's causing a merge conflict now. In or Out?
<danilos> henninge, toss a coin... uhm, I don't know, we do want to have some basic testing for cron scripts so if it's not moved to a unit test, keep it :)
<henninge> okay
<wgrant> danilos: :( I don't think ec2 likes your review type very much.
<danilos> wgrant, uhm, ok, I'll try fixing it
<wgrant> I'm not sure if you can, but I guess we'll see shortly.
<danilos> wgrant, seems to have worked fine
<danilos> wgrant, sorry about the trouble :)
<wgrant> danilos: Ah, great. Thanks.
<wgrant> This more forgiving bugactivity grouping is nice.
<danilos> wgrant, btw, we were finally able to get a clean slate of stable branch for rollout yesterday, just if you haven't noticed (I know you were waiting for a fix to be rolled out)
<wgrant> danilos: I was pleasantly surprised this morning to find the deployment page empty.
<wgrant> We still had two un-QA'd items when I went to sleep last night, so I wasn't too hopeful.
<wgrant> Thanks for getting that deployed.
<danilos> wgrant, your rollback was very helpful, thanks for doing it
<wgrant> danilos: You needed that celebrity fix?
<wgrant> So many revs lately that I cannot remember them all :(
* wgrant changed the topic of #launchpad-dev to: Launchpad Development Channel | BUG JAM! http://mumak.net/lp-bugjam-2010/ 208 so far! | PQM open for business | firefighting: - | Get the code: https:/â/âdev.launchpad.net/âGetting
<danilos> wgrant, btw, before you are gone (though you should be gone :), is there a way we can do something similar on production to re-uploading kde-l10n-sr .translations.tar.gz file you did on dogfood or should we ask packagers to re-upload it?
<danilos> wgrant, or perhaps we can just do a rebuild?
<wgrant> danilos: We can't do an explicit manual upload of the tarball to the source package, like we can with a project?
<danilos> wgrant, no, unfortunately
<wgrant> danilos: We'd have to do an upload to maverick-updates.
<wgrant> The way I did it on DF was a little... unconventional.
<danilos> wgrant, I know, just checking :)
<wgrant> It involved a disturbing amount of conversation with psql.
<danilos> wgrant, so, what about rebuilding, will that work, and can we do that?
<danilos> wgrant, or will we have to have a re-upload with version change?
<wgrant> danilos: We'd have to do a new upload to maverick-proposed.
<danilos> wgrant, I am wondering because this is an ubuntu package
<wgrant> And then we'd probably have to get an SRU exemption.
<danilos> and I guess it wouldn't be very nice to change the package with the same version (though, package would stay the same, but dates would change)
<danilos> wgrant, right, then I'll just ask Kubuntu people to do it, thanks
<wgrant> Yeah, Riddell would probably be a good person to talk to.
<danilos> was just looking to see if he's around :)
<danilos> wgrant, fwiw, we don't have to release these packages, we just need to re-import translations
<wgrant> danilos: So we could probably get away with accepting an upload to -proposed and then deleting it as soon as it builds.
<wgrant> Without ever going through -updates.
<danilos> wgrant, I wouldn't know enough about what I am doing there (i.e. I am totally unfamiliar with Ubuntu policies and also about "pockets" or whatever :)
<danilos> wgrant, I'll see if Riddell has any ideas and if he'd like to do something like that
<wgrant> Right.
<wgrant> Good morning jml.
<wgrant> Your Jamometer is looking much more healthy today.
<jml> oooh
 * jml looks
<jml> zowie!
<jml> I gather we rolled out?
<wgrant> Indeed we did.
<wgrant> The deployment report was deliciously empty this morning.
<jml> wow.
<jml> "critical" doesn't mean what it used to.
<wgrant> jml: Hm?
<wgrant> You mean the critical bugs that have been sitting around for ages untouched?
<jml> wgrant: yes
<gmb> allenap: Do you know what the current EC2 image number is or how I'd find out?
<gmb> (I ask because you're the last person to touch it, IIRC)
<allenap> gmb: bin/ec2 images
<gmb> Aha.
<gmb> Ta
<allenap> gmb: 503
<wgrant> Using machine image version 503
<jml> looking at bug 272304. I now recognize that error string.
<_mup_> Bug #272304: "User timeout caused connection failure." doesn't make sense <branch-puller> <confusing-ui> <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/272304 >
<gmb> allenap: When you updated the EC2 image, did you get a lot of nonsense about installing GRUB?
<allenap> gmb: Mmm, don't remember :-/
<gmb> Hrm.
<gmb> allenap: Did you base your image off the previous launchpad-ec2test image or did you base it off the official Lucid AMI?
<allenap> gmb: Official.
<gmb> Ok.
 * gmb washes his hands of this weirdness, restarts the process.
<allenap> gmb: There are some instructions on how to do it on the wiki. I updated them last time I did it.
 * allenap really goes now
<gmb> allenap: Yeah, that's what I'm following :)
<jml> allenap, gmb: do you reckon all those story-foo tags should be "official"?
<jml> (I'm guessing official means at least "visible on the front page")
<jml> gah, you know what I mean by 'front'.
<voidspace> where do I report a bug in launchapd?
<gmb> jml: Well, they were made official out of laziness and i-can-never-type-things-right-the-first-time -ness.
<jml> voidspace: bugs.launchpad.net/launchpad/+filebug
<voidspace> ah - that's right, I bug jml about it directly until it is fixed
<voidspace> jml: no need, I can just hassle you instead
<gmb> jml: I'm thoroughly ambivalent about their actual officialness.
<voidspace> and where do I change my launchpad password?
<jml> voidspace: a veritable blessing falls from the sky and on to my lap
<voidspace> my account details page doesn't appear to provide that option...
<jml> gmb: I guess once they aren't under active development the laziness factor is less important?
<wgrant> voidspace: Check the link down the bottom of https://launchpad.net/people/+me/+edit
<wgrant> voidspace: https://launchpad.net/launchpad/+faq/51
<jml> wgrant: beat me by seconds
<voidspace> ok - so if I click on my username in the top right of the page it appears to show an account page, from which I can edit all details except my password
<voidspace> this does not compute ;-)
<voidspace> anyway - thanks
<wgrant> voidspace: There is a 'Change details' link on that page.
<wgrant> Which goes to the page that I linked you to.
<gmb> jml: Yeah. Actually, if you rephrase that as "story-* tags are considered official whilst the story is under active development" it doesn't sound like a half-bad rationale.
<voidspace> ah right, I am on that page and don't see an option to change my password
<wgrant> voidspace: It is a little awkward at the moment, since we are part-way through a migration to use OpenID for authentication.
<wgrant> voidspace: Right, there is a link down the bottom to an FAQ about that.
<jml> voidspace: yes, but if you actually read what wgrant says, there's a link at the bottom
<voidspace> FFS
<voidspace> thanks
<jml> gmb: cool.
<wgrant> Someone should throw the OpenID story at a squad next year and get this sorted out.
<voidspace> aaaand I can't change my password to something memorable because of the damn password rules
<voidspace> but I can leave it as the current weak password which also doesn't meet the rules...
<voidspace> yay
<wgrant> I've also avoided changing mine lately because I can't be stuffed respecting those rules :(
<jml> gmb: I notice that a lot of the story-* tags are phrased incrementally (e.g. "refactor-log-api"). I think tags work a little better when they are more absolute (e.g. "patch-tracking").
<jml> gmb: this is by way of an observation, I can't think of any useful thing to actually *do* as a result.
<gmb> jml: You're right, though. I think that those were "sub-stories" that grew as a result of us looking at a particular problem. As in "We can only do this part of $major_story when $minor_story is finished." so we make a tag for $minor_story to make it easier to track which bugs we need to fix first.
<jml> makes sense
<jml> wgrant: I wonder how much of the openid stuff can get cleaned up through bug fixes
<wgrant> jml: I don't think it can be properly fixed until the divorce is compelte.
<wgrant> Once the split is done, login.launchpad.net can go away,.
<wgrant> And then confusion is eliminated.
<jml> I see.
<wgrant> (it cannot go away now, because forcing everyone through login.ubuntu.com would be suicide)
<jml> hmm.
<jml> I wish there were an obvious choice for argument parsing stuff when writing a Python command-line app.
<wgrant> jml: I like the look of argparse. But since it's 2.7, I still use OptionParser mostly.
<wgrant> Is there anything better than OptionParser?
<jml> wgrant: there's twisted.python.usage and stuff based on bzrlib.options
<jml> (e.g. commandant)
<jml> wgrant: and lifeless has a new, more experimental thing in testr.
<wgrant> Oh, right, but those all have large external dependencies.
<jml> yeah, exactly
<stub> OptionParser is still pretty cool and can do a lot. Not that this is obvious from the badly layed out docs which are more a tutorial than a reference.
<jml> what does searchTasks do when it's on person?
<jml> 'related', it seems
<LPCIBot> Project devel build (335): STILL FAILING in 3 hr 30 min: https://hudson.wedontsleep.org/job/devel/335/
<jml> what do we think about this format of output? http://paste.ubuntu.com/546931/
<jml> trying to prototype a command-line display of all Launchpad inventory
<jelmer> jml: I think it makes sense to display MPs inline with the branches
<jelmer> makes more sense than displaying them separately I mean
<jml> jkakar: kind of related to that kanban board...
<jml> jelmer: yeah, I think I agree.
<jml> jelmer: you have a lot of stuff :)
<jml> jelmer: http://pastebin.ubuntu.com/546932/
<jelmer> jml: that's quite a useful overview, I forgot a lot of the stuff that's on that list :-)
<jml> jelmer: yeah, there's some on mine that I'm deleting / abandoning now.
<jml> jelmer: I guess I could try to make some HTML output so it's linked and better as a prototype for an actual LP page.
<jelmer> jml: Actually, this is something I've been wondering about..
<jelmer> Is there any reason for not importing *all* Debian bugs? It'd be very neat to be able to browse my Debian bugs in lp.
<jelmer> Just the sheer volume?
<jml> jelmer: I don't know. gmb might.
<jml> jelmer: I mean, it's the sort of thing we _should_ do.
 * gmb reads scrollback
<gmb> jelmer, jml: We should be doing it, but we (originally) didn't because of the volume. The correct way to deal with it in the first instance, I think, would be to do a batched import.
<jml> gmb: does the 'originally' imply that another reason came up?
<gmb> jml: No. The only reason we don't do it now is because we don't do it now (i.e. we forgot to ever revisit the problem)
<jelmer> should I file a bug ?
<jml> that old chestnut :)
<jml> gmb: thanks.
<gmb> jelmer: I think there's one already, but feel free to take a look and file one if not.
<LPCIBot> Project db-devel build (249): STILL FAILING in 3 hr 36 min: https://hudson.wedontsleep.org/job/db-devel/249/
<jelmer> gmb: Thanks
<jml> jelmer: also has this style of output: http://paste.ubuntu.com/546935/
<jelmer> jml: btw, it only seems to report upstream bugs/branches at the moment?
<jelmer> jml: nice
<jml> jelmer: no, it should get package branches / bugs
<jml> but maybe the way it prints them out is wrong
<jml> jelmer: http://paste.ubuntu.com/546936/
<jelmer> ah, it reports them under the upstream project
<jelmer> I think most of my ubuntu-specific branches are team-owned
<jml> jelmer: specifically, it just groups by target.name
<jelmer> jml: ah
<jelmer> jml: are you looking at this in preparation of e.g. a dashboard, or is this just random hacking?
<jml> jelmer: a bit of both
<jml> jelmer: I'd like to prototype a Launchpad dashboard, get a feel for the data etc.
<jml> lp:~jml/+junk/whip has the code. would love it if you made it better.
<jelmer> I might have a look :-)
<jkakar> jml: Yeah, I should spend some more time on that kanban board and then announce it somewhere.  I've been finding it quite useful.
<jml> jkakar: http://people.canonical.com/~jml/jkakar-wip.html
<jml> interesting
<jml> how would I debug this? http://paste.ubuntu.com/546951/
<jkakar> jml: That's really cool. :)
<jelmer> jml: perhaps it's related to the fact that bzrk is inactive?
<jml> jelmer: yeah, I think so.
<jml> jelmer: I hacked around it
<jml> jkakar: thanks. it's certainly the start of something that could be quite cool.
<jkakar> jml: Yeah, I've wanted something like that for a while.  It sounds like it, or something like it, would be nice to have in my kanban project.
<jml> jkakar: I'm not at all convinced that kanban-style presentation is the best way to go.
<jkakar> jml: I find it useful as a high-level view, but yeah, it's not a golden hammer.
<jkakar> jml: I was thinking something more like the "list of stuff" that you have, integrated into the kanban system, could be useful.
<jml> jkakar: *nod*
<jkakar> jml: One thing I noticed about your thing is that it lists branches that have been proposed for merged and 'Rejected'.
<jml> jkakar: yes. it also lists MPs for Abandoned branches.
<jml> jkakar: I'm not sure if that's a bug in Launchpad or a bug in the thing.
<jkakar> jml: In the two cases I see in my listing (lp:~jkakar/storm/resultselect and lp:~jkakar/storm/resultset-expression) they're both things that I don't really care about anymore, but I also don't want to delete the branches.
<jml> jkakar: "Abandoned" is your friend.
<jkakar> jml: Ah, I didn't even know about it. :)
<jml> jkakar: similarly, marking a branch as "Mature" or linking it to a series will get it off the list.
<jkakar> jml: Hmm, weird. :/
<jml> I don't know that it's so weird.
<jkakar> jml: I don't understand what "Mature" means or why linking to a series would get it off that list.
<jml> jkakar: because lp:foo or lp:foo/2.1 is unlikely to be work-in-prorgess
<jkakar> jml: I see.  So what does 'Mature' mean?
<jml> jkakar: no-one knows what Mature means. Here I'm using it as a convenient way to let people say "this branch isn't abandoned, it should appear on listings, but it's not really intended to ever be merged anywhere"
<jkakar> jml: Hehe, cool. :)
<jml> jkakar: so, for example, I've marked some of my unofficial VCS imports as Mature
<jml> jkakar: and my fork of pyflakes that supports lazy imports
<jkakar> jml: Right, makes sense.
<jkakar> jml: Is the script that you used to generate the wip HTML page on Launchpad somewhere?
<jml> jkakar: yes. all the details are here: http://code.mumak.net/
<jml> jkakar: I mean, lp:~jml/+junk/whip
<jml> (I have also blogged)
<sidnei> what is the reviewers channel again?
<bac> #launchpad-reviews
<bac> henninge: have you seen the failure on db-devel related to MockLogger.  Looks like some tests were not updated.
<henninge> bac: no, I have not
<bac> henninge: r12141 updated poimport.txt but your branch did not, for example.
<henninge> bac: this is poimport-script.txt, though
<danilos> bac, it might be thumper's "mock logger consolidation" branch
<bac> yes, poimport-script.txt, i meant
<bac> danilos: yes, that is the branch i think henning was trying to merge into db-devel
<henninge> bac: that is true
<henninge> but I only resolved a merge conflict in poimport.txt.
<danilos> anyone knows how are script names related to config sections?
<henninge> I am sorry, though, I have to leave right now. I cannot look into it any further.
<bac> henninge: so do we think thumper's branch is the original culprit then?
<henninge> bac: yes, in relation to db-devel differing a lot currently because of the recife branch having been merged into it.
<henninge> differing in translations
<henninge> danilos, bac: would either oof you be so kind and have a closer look, please? I really have to go.
<danilos> henninge, that's not a lot, that was barely 25k lines of diff when merged ;)
<bac> thanks henninge.  i just wanted to get your take on what was going on
<henninge> sure, you got it ;)
<henninge> bac: Have a great Christmas!
<bac> you too
<bac> danilos: poimport-script.txt is in db-devel but not trunk.  was it supposed to be removed/
<danilos> bac, I am not sure, I'll check
 * bac is unsure how to get bzr to tell about deleted files
<danilos> bac, it needs to be removed, test was replaced with a unit test lib/lp/translations/scripts/tests/test_translations_import.py
<danilos> bac, if you feel trigger happy, r=danilo for that :)
<danilos> bac, otherwise, I'll get to it a bit later after I am done shuffling a few WIP tasks
<bac> danilos: it was easier just to fix the test.  i'll submit a testfix branch in a bit
<danilos> bac, easier? it's a duplicate test, that's why it was removed in the first place :)
<danilos> bac, though thanks anyway :)
<bac> danilos: but the replacement has not landed in db yet
<danilos> bac, no? that's where I see it so it's weird
<bac> danilos: oh, you are right.  i just assumed the branch that introduced the new one had deleted the old.
<bac> danilos: i'll just zap it then
<danilos> bac, cool, thanks
<leonardr> jml, can you tell me about LaunchpadBranchLander or point me to someone who can?
<leonardr> i'm trying to figure out what to do with its call to login_with, which is about to be deprecated
<leonardr> is this a script people will run from their desktops?
<jml> leonardr: can you give me some more context? a URL? a filename?
<leonardr> jml: sure
<leonardr> lib/devscripts/autoland.py
<jml> oh
<leonardr> it could be something you run yourself or it could be something run automatically on the ec2 instance
<leonardr> i can't tell
<jml> leonardr: that's 'ec2 land'
<jml> leonardr: it's used only in cmd_land in devscripts/ec2test/builtins.py
<jml> leonardr: it's a script that developers run from their desktops.
<leonardr> jml: ok, so it's all right if it causes a browser open
<jml> leonardr: indeed.
<allenap> leonardr: I'm not going to finish your review before I go. I'll probably be back online in about 3 hours for a while. Is it urgent?
<leonardr> allenap: not at all
<allenap> leonardr: Okay, I'll either finish it this evening or have it done by the time you start tomorrow.
<leonardr> allenap: today's my last day for the rest of the year, so don't rush
<allenap> leonardr: Don't tell me that or I'll procrastinate until January ;)
<allenap> leonardr: Have a good holiday.
<leonardr> thanks, you too
<LPCIBot> Project devel build (336): STILL FAILING in 3 hr 5 min: https://hudson.wedontsleep.org/job/devel/336/
<LPCIBot> Launchpad Patch Queue Manager: [r=danilo][ui=none][bug=685624] Let
<LPCIBot> TranslationTemplatesBuildJob.build through the security proxy.
<lifeless> jml: have you looked at my persistence science-fiction?
<jml> lifeless: no. I skimmed the discussion on the list.
<jml> lifeless: I might take a look tomorrow.
<jelmer> ec2 test/land appears to be broken on natty - has anybody looked into that yet?
<jml> jelmer: perhaps that's what leonardr was just talking about.
<leonardr> jelmer: what's the error?
<jelmer> <Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>26328b8e-9dd8-4cba-af10-746d34227953</RequestID></Response>
<jelmer> I can reproduce it on two natty machines, and the lucid partition (with the same home dir as one of the natty installs) works fine.
<jelmer> leonardr: is that the same as you were seeing?
<leonardr> jelmer: i'm not testing it, i was inspecting code for problems with a branch i'm working on
<leonardr> that looks like a problem between you and aws, not related to the launchpad web service
<jelmer> That's what I was thinking, but it's strange that it works fine in my lucid install with the same credentials.
<jelmer> I'll investigate further.
<LPCIBot> Project db-devel build (250): STILL FAILING in 3 hr 14 min: https://hudson.wedontsleep.org/job/db-devel/250/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12142
<LPCIBot> included.
<lifeless> grr
<lifeless> something has broken feature flag timeouts :(
<jcsackett> can someone tell me where to look to find the javascript responsible for controlling the spinner that shows on the merge proposal status when you change status?
<lifeless> jcsackett: you pinged me yesterday
<jcsackett> ah, yes i did, lifeless.
<jcsackett> it's in relation to a bug you filed, let me dig tat up.
<jcsackett> lifeless, bug 691846. i was wondering what url facilities you were thinking of? urllib/urllib2 can't handle all of those protocols to my knowledge, but perhaps you're thinking of something i'm unaware of?
<_mup_> Bug #691846: hardcoded list of protocols for urlification is a maintenance burden and duplicate code <bugjam2010> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/691846 >
<lifeless> jcsackett: urls follow a generic ruleset
<lifeless> any url parser can parse all urls unless its totally broken - see ietf std 66
<lifeless> [the python stdlib url parser may be totally broken; I've had to monkey patch it before]. That said,
<lifeless> there are broadly two sorts of urls
<lifeless> path like urls
<lifeless> and non-path like urls
<lifeless> parsers are meant to handle both, at least as far as scheme, otherstuff
<lifeless> secondly, we have our own url facilities in Launchpad
<jcsackett> the second point was what i assumed--i am unaware of such facilities, but figured they existed.
<lifeless> for the record, urlparse supports ftp, http, gopher, nntp, impa, wais, file https shttp snew prospero rtsp rtspu rsync svn svn+ssh sftp nfs git git+ssh, and we use it directly in bzr after extending the urlparse.uses_netloc whitelist
<lifeless> thats a rather more extensive scheme list than your patch used, I think :)
<jcsackett> lifeless: i wasn't actually aware of the urlparse lib. :-)
<jcsackett> that seems quite sufficient, thanks. :-P
<lifeless> you want urlparse.urlsplit I think
<jcsackett> lifeless: that or urlparse.urlparse, it looks like.
<jcsackett> thanks.
<lifeless> throw some empty things at it
<lifeless> no, urlsplit
<lifeless> you have stuff that *isn't* a valid http url
<jcsackett> lifeless: true. ok.
<lifeless> that you want to see is just http, for instance.
<lifeless> urlsplit should handle that, as I read it
<jcsackett> cool. thanks again.
<lifeless> gary_poster: hi, around?
<gary_poster> lifeless, hi, yes
<lifeless> I just found myself filing a dup of https://bugs.launchpad.net/oops-tools/+bug/677299
<_mup_> Bug #677299: please always report at least one of each pageid that times out in the 'lpnet' daily summary <OOPS Tools:New> < https://launchpad.net/bugs/677299 >
<lifeless> I'm wondering if your team has the bandwidth to do something stopgap in that area.
<gary_poster> heh, not for the rest of the year.
<lifeless> I appreciate that grouping differently is tricky and hard
<lifeless> gary_poster: ok, I guess we'll see what the brand new year brings :)
<gary_poster> :-) ok
<lifeless> gary_poster: I'm going to triage that up to high though
<lifeless> this cost me 15 minutes today
<lifeless> (and yes, I'm still on leave :)
<lifeless> anyhow, have a great xmas or whatever you call it over there :)
<gary_poster> lifeless: high: ack.  let's circle around at start of year--several people will be pedaling hard to tie various things up before the team disperses at thunderdome but maybe we can fit things in then.
<gary_poster> have a great holiday also
<lifeless> gary_poster: hmm, my previous came across overly grumpy; sorry about that.
<lifeless> gary_poster: I agree, and understand.
<gary_poster> :-) no worries
<lifeless> I suspect in the new structure we need to move all the subprojects to canonical-engineering team maintenance
<lifeless> rather than qa-team, etc
<gary_poster> agree
<lifeless> then ask folk to save bugs.launchpad.net/launchpad-project/+bugs?field.priority=high as their go-to search
<lifeless> or something like that
<gary_poster> yeah
 * lifeless will worry about it in the new year
<gary_poster> good idea :-)
<LPCIBot> Project db-devel build (251): STILL FAILING in 3 hr 9 min: https://hudson.wedontsleep.org/job/db-devel/251/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][r=danilos][ui=none][no-qa] Remove redundant test.
<LPCIBot> Project devel build (337): STILL FAILING in 3 hr 33 min: https://hudson.wedontsleep.org/job/devel/337/
#launchpad-dev 2010-12-24
<wgrant> Now it's time to play everyone's favourite game: "work out how to run that untested Soyuz script that probably hasn't run outside cocoplum for 5 years."
<lifeless> wgrant: archive:+index is still slow :(
<lifeless> wgrant: 5K > 10 seconds
<lifeless> yesterday
<wgrant> lifeless: Yeah.
<wgrant> lifeless: I can probably get it down slightly by removing the remaining couple of duplicate queries.
<wgrant> But the main issue is that the remaining queries are slow.
<wgrant> Needs a stub.
<lifeless> wgrant: if he starts while you're still working today, you could ask him about it
<lifeless> I think I pointed him at it already, he may have some ideas
<wgrant> lifeless: I will hopefully not be around much this evening.
<wgrant> But I will poke him if our hours intersect.
<LPCIBot> Project db-devel build (252): STILL FAILING in 3 hr 32 min: https://hudson.wedontsleep.org/job/db-devel/252/
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>       33 /  189  BugTask:+index
<lifeless>       15 /  358  Distribution:+bugs
<lifeless>       14 /    2  Archive:+copy-packages
<lifeless>        9 /    0  Bug:EntryResource:markUserAffected
<lifeless>        8 / 3280  Archive:+index
<lifeless>        8 /    3  ProjectGroup:+milestones
<lifeless>        5 /  392  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        5 /    8  NullBugTask:+index
<lifeless>        4 /   92  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>        4 /   62  MailingListApplication:MailingListAPIView
<LPCIBot> Project devel build (338): STILL FAILING in 3 hr 12 min: https://hudson.wedontsleep.org/job/devel/338/
<LPCIBot> Launchpad Patch Queue Manager: [r=sinzui][ui=none][bug=579502] Do not send bug email to distro owner
<LPCIBot> if the distro does not use Lauchpad for bugs.
<LPCIBot> Project db-devel build (253): STILL FAILING in 3 hr 11 min: https://hudson.wedontsleep.org/job/db-devel/253/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12144
<LPCIBot> included.
<stub> Anyone know where the code that pulls launchpadlib doctests into the testsuite lives?
<lifeless> whats the path of one of these doctests?
<stub> eggs/launchpadlib-1.6.5-py2.6.egg/launchpadlib/docs/hosted-files.txt
<stub> nm though
<stub> Genuine problem, not an issue with the setup environment
<stub> Or maybe not... *sigh*
<stub> Can we make the current config_fixture a global?
<stub> LibrarianFixture is useless without a ConfigFixture. This will pretty much always be BaseLayer.config_fixture. I don't want to use BaseLayer.config_fixture as the default, because that is going circular.
<stub> Looking at explicitly passing it in now - not sure how many call sites there are.
<stub> Hmm... add_section modifies the generated config, but there is no way of popping those changes...
<stub> Maybe would have been better to use the standard push/pop on the config and just get it to write new config files and update LPCONFIG so push/pop also work with subprocesses.
<jml> helloall
<jml> my space bar isn't responding as well as I would like.
<jml> anyone around today?
<jelmer_> 'morning jml
<jml> jelmer_: hi
<jelmer_> it's very quiet here and on the mailing list today
<jml> I'll bet.
<jml> I'm tackling distributionmirror_prober, trying to make it more like a normal twisted script
<jml> so that it leaks resources less
<jml> has anyone got a recent subunit log of the full test suite that I could borrow?
<jml> ahh, glory. StevenK's hudson builder has the full output
<lifeless> stub: yes, that probably would be better
<lifeless> stub: given that there was no concept of keeping the necessary fragments, I felt it might be too high a bar to reach in a reasonable timeframe, so I opted for a separate, lesser interface
<lifeless> stub: the main config system being hugely complex.
<lifeless> stub: I'm interested (but not right now) in why you need to pop stuff out, but certainly you could add that.
<stub> So launchpadlib (and old-testing.txt) don't use layers. LaunchpadServerFixture.setUp() doesn't setup everything ready for use. You still need to add the new stanzas to the config, and invoke ConfigUseFixture. I've added the config_fixture as a parameter to LaunchpadServerFixture, and am about to make that also invoke ConfigUseFixture. Which pretty much means ConfigUseFixture is unnecessary, and its functionality moved to ConfigFixture.add_sectio
<stub> lifeless: push/pop is more a purity thing. At the moment, a test (or set of tests) that modify the config leave things in a different state than they started.
<lifeless> ok
<lifeless> that sounds plausible
<lifeless> by which I mean - you've clearly looked at it, and nothing there contradicts the overall arch I was working on
<lifeless> (AFAICT at this distance)
<lifeless> stub: I'd like to nuke layers
<StevenK> And I'd like to nuke buildbot, but nothing seems to be happening there
<lifeless> stub: but I need to finish the reuse-optimisation support for fixtures + glue them to testresources which has ordering-optimisations.
<stub> Sure. Tree has never suited us.
<jml> I still don't get why push/pop require names
<lifeless> jml: hysterical rasins
<jml> lifeless: yeah, but I remember asking about it when the functionality was first added and not getting a clear answer.
<lifeless> I think the custom config stuff was engineered with requirements different to our needs.
<jml> also, do you know what's awesome? threads.
<lifeless> jml: its safer in doctests because they don't [in general] have cleanups
<lifeless> and here, gnight all
<jml> lifeless: g'night.
<stub> jml: Its to make sure you pop what you think you are popping
<stub> jml: Otherwise, if code forgets to pop a config later pops succeed and your errors end up in entirely the wrong place.
<jml> stub: hmm.
<jml> looks like initializing a LaunchpadScript causes bzr functions in the same process to dump to stderr.
<stub> jml: That is supposed to be silenced in lib/lp_sitecustomize.py - not sure where the magic that includes that is (buildout.cfg mentions it)
<stub> jml: oh... parts/scripts/sitecustomize.py
<jml> stub: interesting.
 * jml manually copy-pastes those bits into a minimal test case
<jml> stub: doesn't work. c.l.scripts.logger blats all over it.
<jml> it adds an INFO-level log handler to the root logger
<jml> so everything logged at INFO level is printed to stderr
<stub> Yes, but the bzr logger should dev null events and not propagate them.
<stub> Hmm... which we don't seem to be doing
<stub> Unless NullHandler does the magic
<jml> back
<jml> but stub is gone
<jml> oh well.
<jml> I think this just leaves lazr.smtptest "No handler" complaints.
<jml> (which I have no idea how to address)
<jml> anyway, off for lunch
<jelmer> bon appetit
 * jelmer follows
<jml> back again
<LPCIBot> Project devel build (339): STILL FAILING in 3 hr 33 min: https://hudson.wedontsleep.org/job/devel/339/
<jml> anyone want to review my branch?
<jml> no behaviour changes, just moves some code around.
<jelmer> jml: Sure
<jml> jelmer: https://code.launchpad.net/~jml/launchpad/distribution-mirror-prober/+merge/44656
<jml> really starting to hate Python logging.
<jelmer> jml: At least it *has* a standard mechanism for logging
<jelmer> unlike C where every library has its own custom thing, with different log level names, callbacks, etc
<jml> yeah
<jml> I guess that's a good thing
<jml> btw
<jml> always be careful when constructing a LaunchpadCronScript in a test
<jml> if you do it wrong, it'll call sys.exit
<jelmer> Ouch
<jelmer> I bet that can be a bit of a surprise, especially if the output is redirected to something that doesn't go to the terminal.
<jml> something is changing the log level too.
<jelmer> that reminds me..
<jelmer> we added a log handler a couple of months ago that automatically logs OOPSes on Logger.error and Logger.warning.
<jml> yes.
<jelmer> doing so for warnings isn't really consistent given our policy of treating all OOPSes as high priority bugs IMHO
<jelmer> s/given/with/
<jml> yeah, quite possibly
<jml> https://bugs.launchpad.net/launchpad/+bug/694140
<_mup_> Bug #694140: canonical.launchpad.scripts.logger magic breaks bzr logging <Launchpad itself:New> < https://launchpad.net/bugs/694140 >
<jelmer> jml: You're creating an old style class in distributionmirror_prober.py
<jml> jelmer: hah. so I am.
<jelmer> jml: Actually, you're not. the __metaclass__ assignment is just in an unusual place.
<jml> easy enough to move
<jelmer> jml: It would be nice to have a couple more docstrings - for example, it isn't obvious to me that the "notify_owner" argument to mirror() means notifying the owner when their mirror is disabled
<jml> jelmer: fair enough
<jelmer> jml: Other than that I'd be happy to +1 it
<jml> jelmer: thanks.
<jml> jelmer: I've pushed up changes.
<jelmer> jml: r=me
<lifeless> happy thingy,world
<lifeless> jelmer:  on logging -> oopses
<lifeless> jelmer: if someone should read it, its worth oopsing
<lifeless> jelmer: is the logic that was followed, and I think that thats reasonable
<jelmer> lifeless: I agree, but that doesn't mean it's necessarily a high level bug.
<jelmer> s/level/priority/
<lifeless> jelmer: the question then is 'what logging level means "someone should read it".' ?
<lifeless> jelmer: actually, I think it is
<lifeless> operational problems drown us out at the moment, and that will only get better once we're on top of it.
<jelmer> lifeless: it doesn't help if we just end up with lots of high priority bugs because of this.
<lifeless> jelmer: its been active for a month or more now, and we haven't been flooded.
<jelmer> lifeless: some calls to Logger.warning have been changed to Logger.info
<lifeless> jelmer: and, if we end up fixing lots of production glitchy things because of it, then I argue it would help.
<lifeless> jelmer: yes, I saw, and thought those were appropriate choices
<lifeless> jelmer: *because* they were things which really didn't need action taken
<jelmer> lifeless: hmm
<jelmer> lifeless: This makes it harder to distinguish between informational messages and things that look odd but can occur (what I would use warning for)
<lifeless> so, ignore logging for a second
<lifeless> as a framework
<lifeless> we have something running somewhere, with no ui
<lifeless> we have a facility you can output high-volume debug stuff that will by default go nowhere
<lifeless> a facility that will be logged and you can read on disk
<lifeless> and a facility that will generate an OOPS
<lifeless> is this insufficient?
<lifeless> the first facility is for performance reasons - we can't afford to grab everything every time
<jelmer> With the way we currently handle OOPS reports and logs on disk, somewhat.
<jelmer> there is no way for something to trigger attention by a human without causing a high level bug report.
<lifeless> the second for stuff that we can afford to gather, for helping with bug analysis etc, but which doesn't require an action
<lifeless> and the third thing is for triggering human attention
<lifeless> jelmer: well, if its not *high* it will sit there and be ignored, based on current stats, for a year or more
<lifeless> jelmer: 'needs human attention sometime in 2012' isn't a very useful signal IMO
<lifeless> I gotta run. sorry!
<jelmer> lifeless: Merry christmas, ttyl!
 * jelmer is about to EOD, shops close at 7 today apparantly :-/
<jelmer> jml: Merry christmas!
<jml> https://bugs.launchpad.net/launchpad/+bug/694152
<_mup_> Bug #694152: LaunchpadScript constructor breaks logging for subsequent tests <Launchpad itself:New> < https://launchpad.net/bugs/694152 >
<jml> oh
<jml> I think https://bugs.launchpad.net/launchpad/+bug/694140 is also why lazr.smtptest keeps generating warnings about no handlers
<_mup_> Bug #694140: canonical.launchpad.scripts.logger magic breaks bzr logging <Launchpad itself:New> < https://launchpad.net/bugs/694140 >
<jml> lifeless: merry Christmas. You should sleep more :)
<jml> Good night all.
<jml> Merry Christmas.
<LPCIBot> Project db-devel build (254): STILL FAILING in 3 hr 32 min: https://hudson.wedontsleep.org/job/db-devel/254/
<LPCIBot> Project devel build (340): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/340/
<LPCIBot> Project devel build (341): STILL FAILING in 3 hr 11 min: https://hudson.wedontsleep.org/job/devel/341/
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=435316] Add tests to make sure Binary: lines
<LPCIBot> with newlines are supported in sync_source.
<LPCIBot> * Launchpad Patch Queue Manager: [r=bac, jcsackett][ui=sinzui][bug=440406,
<LPCIBot> 560701] Help with alternate language usage. List languages on a
<LPCIBot> persons translation dashboard.
#launchpad-dev 2010-12-25
<LPCIBot> Project db-devel build (255): STILL FAILING in 3 hr 36 min: https://hudson.wedontsleep.org/job/db-devel/255/
<LPCIBot> Project devel build (342): STILL FAILING in 3 hr 13 min: https://hudson.wedontsleep.org/job/devel/342/
<LPCIBot> * Launchpad Patch Queue Manager: [r=jelmer][ui=none][no-qa] Move distributionmirror prober logic into
<LPCIBot> the module
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=385711][bug=398896] Fix
<LPCIBot> ProjectGroup.products sort order and remove Author: comments.
<LPCIBot> Project devel build (343): STILL FAILING in 3 hr 10 min: https://hudson.wedontsleep.org/job/devel/343/
<LPCIBot> Launchpad Patch Queue Manager: [r=edwin-grubbs][ui=none][bug=681326][incr] New class
<LPCIBot> BugSubscriptionInfo which begins to encapsulate all
<LPCIBot> subscription information about a bug.
<lifeless> wgrant: hi
<wgrant> lifeless: Morning.
<lifeless> ...?
<wgrant> Or evening.
<wgrant> One of those two.
<lifeless> yeah'
<stas> merry x-mas everybody
<stas> I have a problem with launchpad when updating a bug status
<stas> http://is.gd/jqFSy
<stas> can you help?
#launchpad-dev 2010-12-26
<LPCIBot> Project db-devel build (256): STILL FAILING in 3 hr 34 min: https://hudson.wedontsleep.org/job/db-devel/256/
#launchpad-dev 2011-12-19
<lifeless> wallyworld_: WHATS UP?
<lifeless> bah, caps lock
<wallyworld_> lifeless: it's ok, i already sorted it. the deployment report had a rev that should have been tagged rollback=xxxx and i was wondering how to fix it
<wallyworld_> and no-one else was shown as logged into irc :-)
<StevenK> Missing rollback tags are annoying
<StevenK> Since you can't add them after the fact
<wallyworld_> yeah, so i found out
<lifeless> qatagger is fixable :)
<wgrant> lifeless: Yay for fixing bug watches.
 * StevenK doesn't get it.
<StevenK> Replicating the conditions of bug 892025, and I don't see the link. But I do on qas/prod
<_mup_> Bug #892025: Forbidden following link to configure code hosting <403> <qa-ok> <trivial> <ui> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/892025 >
<wgrant> StevenK: You probably have permission to set it, but can't see the dev focus branch.
<wgrant> Unauthorized: (<Branch u'~hexr-dev/hexr/trunk' (406169)>, 'unique_name', 'launchpad.View')<br />
<StevenK> Oh
<StevenK> So configure_codehosting should learn that too?
<wgrant> No.
<wgrant> Well, maybe.
<wgrant> It's difficult.
<wgrant> Leave it for now.
<wgrant> We'll revisit it if it's still a problem once we've fixed privacy.
<StevenK> Bah, Gavin hinted at the issue in the description too
<wgrant> I just looked at the traceback :)
<StevenK> I saw the error page and assumed it was "You can't see this page, go away"
 * StevenK stares at BugContextMenu.nominate()
<StevenK> You can Target if you're a driver, nominate if you're a bug supervisor and everyone else can bugger off?
<lifeless> wgrant: context? my email I presume?
<wgrant> StevenK: Yes
<wgrant> lifeless: Yes
<StevenK> wgrant: How is that related to bug 297528, then?
<_mup_> Bug #297528: Permissions not checked properly when deciding whether to present "Target" or "Nominate" link <lp-bugs> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/297528 >
<wgrant> StevenK: Well, drivers or uploaders can target.
<wgrant> But whatever decides to show Target/Nominate doesn't take into account uploaders.
<wgrant> So for uploaders it says it will nominate, when it will in fact target.
<StevenK> wgrant: Right, so BugContextMenu.nominate() needs a elsif I can upload, which says target.
<wgrant> Yes, but that needs to be faster than it probably currently would be.
<wgrant> StevenK: There's more MixedVisibilityErrors, if you want them.
<StevenK> Oh?
<wgrant> ProductSet:+all
<StevenK> Can I have an OOPS too?
<wgrant> Should be pretty obvious :)
<wgrant> Given what the page does
<wgrant> But OOPS-01faea627b2e0a135da5d8e3640e22ef
<StevenK> Oh, it returns every signle product?
<StevenK> wgrant: So I guess we want to hide the product if the user can't view the registering team?
<wgrant> Possibly.
<wgrant> I didn't think private teams were allowed to own projects :/
<wgrant> We may be best to leave this.
<wgrant> As the new rules should mean that the team is to be disclosed.
<StevenK> Chewie
<StevenK> Chewie exports system settings through a menu service such as dbusmenu and/or gmenu
<StevenK> Registered by <hidden> on 2011-12-02
<wgrant> That's not the one that's the main problem here.
<wgrant> But it's a similar case.
<wgrant> dpkg-coverity is the one here, though I can see the team due to some kind of subscription
<StevenK> Actually, disclosing teams in a public role should fix this, you're right
<wgrant> Right.
<wgrant> But that requires discussion in Budapest.
<StevenK> And beating lifeless with a stick
<StevenK> I think
<StevenK> So I have 1.5 days left. And Curtis has forbidden me to have any branches in progress over the break.
<wgrant> Heh
<lifeless> StevenK: hmm ?
<wgrant> We have things to discuss in Budapest.
<StevenK> wgrant: So I guess I should find a critical
<StevenK> Hm, I think bug 739070 could be fixed with two new columns, but I doubt it's completly doable in the time I have left.
<_mup_> Bug #739070: Archive:+repository-size timeout retrieving many hundreds of package sizes <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/739070 >
<StevenK> And wgrant may shoot me for wanting to add two new columns to Archive.
<wgrant> Correct.
<StevenK> :-(
<StevenK> lifeless: Due to your work with OOPSes, I guess bug 788518 can be closed?
<_mup_> Bug #788518: staging services share OOPS prefix <critical-analysis> <Launchpad itself:Triaged> < https://launchpad.net/bugs/788518 >
<StevenK> And bug 805546, I guess
<_mup_> Bug #805546: persontransferjob does not have a unique oops prefix <Launchpad itself:Triaged> < https://launchpad.net/bugs/805546 >
<wgrant> StevenK: Indeed.
<wgrant> StevenK: Are you on launchpad-error-reports?
<wgrant> StevenK: There's an issue with that OOPS work of lifeless' which somebody need to look at.
<StevenK> wgrant: I am not
<wgrant> 2011-12-18 09:52:03 ERROR   Unhandled exception -> http://launchpadlibrarian.net/87826942/iDGKk0EWB2rD9ZfBovt3JrR3nLJ.txt (unsupported operand type(s) for +: 'NoneType' and 'str')
<wgrant> From checkwatches on loganberry.
<StevenK> I guess either config[self._default_config_section].oops_prefix is None?
<wgrant> It looks that way, but I can't see how.
<StevenK> _default_config_section is error_reports
<wgrant> Indeed.
<StevenK> You were fiddling with the configs last week ... :-P
<wgrant> Yes, but lifeless fiddled with this bit :)
<StevenK> huwshimi: AH ha! You have QA to do.
<huwshimi> StevenK: I do!
<huwshimi> StevenK: Telstra have been out trying to fix my internet all morning
<StevenK> They may have succedded?
<huwshimi> StevenK: It appears that way
<huwshimi> StevenK: At least it is working, if  a little slow
<StevenK> wgrant: I don't get it either.
<StevenK> wgrant: Right, production error_reports has no oops prefix, and the code is expecting one if reporter is None.
<huwshimi> hmm.. I thought I had 4 branches to qa
<wgrant> StevenK: Lies.
<wgrant> StevenK: production/launchpad-lazr.conf's oops_prefix is PRODUCTION.
<StevenK> wgrant: Oh?
<StevenK> steven@liquified:~/lp-production-configs% grep -c oops_prefix production/launchpad-lazr.conf
<StevenK> 0
<wgrant> bzr pull
<StevenK> That's with rev 247
<wgrant> srsly?
<wgrant> r567 is the latest.
<StevenK> bzr pull gives "No revisions to pull."
<wgrant> You be pulling from the wrong place.
<StevenK> bzr+ssh://bazaar.launchpad.net/%2Bbranch/lp-production-configs/ is the wrong place?
<wgrant> That's the right place.
<wgrant> Ah
<StevenK> steven@liquified:~/lp-production-configs% bzr revno bzr+ssh://bazaar.launchpad.net/%2Bbranch/lp-production-configs/
<StevenK> 247
<wgrant> That was the crontabs revno I was looking at.
<wgrant> 247 is indeed configs
<StevenK> Now, grep yourself and take back your lies comment :-)
<wgrant> lpnet-lazr.conf is where PRODUCTION is specified.
<wgrant> And production/launchpad-lazr.conf inherits that.
<wgrant> $ LPCONFIG=production bin/py
<wgrant> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
<wgrant> [GCC 4.4.3] on linux2
<wgrant> Type "help", "copyright", "credits" or "license" for more information.
<wgrant> >>> from canonical.config import config
<wgrant> >>> config.error_reports.oops_prefix
<wgrant> 'PRODUCTION'
<StevenK> Then what crack is checkwatches on?
<wgrant> That's why I asked you to investigate :)
<StevenK> Oh, hah.
<StevenK> class CheckWatchesErrorUtility(ErrorReportingUtility):
<StevenK> _default_config_section = 'checkwatches'
<wgrant> That would do it.
<wgrant> murder
<StevenK> Kill it with a large amount of fire?
<wgrant> Ideally.
<wgrant> lib/lp/services/mailman/monkeypatches/xmlrpcrunner.py:    _default_config_section = 'mailman'
<wgrant> :D
 * StevenK feels ill.
<huwshimi> hmmm... why might I be getting this pqm failure? http://paste.ubuntu.com/774941/
<wgrant> huwshimi: Check the attachments.
<wgrant> We're in testfix.
<StevenK> db-devel went bang with 4 failures and 31 errors.
<wgrant> They are real, too.
<StevenK> ProgrammingError: permission denied for relation milestonetag looks like the root cause
<wgrant> lifeless is more than adequately opinionated on this matter.
<StevenK> lifeless said bad idea and it landed anyway?
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/force-checkwatches-production/+merge/86187
<wgrant> StevenK: disapprove
<StevenK> Awww
<StevenK> Why?
<wgrant> That will leave the reporter as just PRODUCTION.
<wgrant> Rather than PRODUCTION-checkwatches
<wgrant> I suspect.
<wgrant> You probably want to call .configure(section_name='checkwatches')
<StevenK> Yeah, I was just reading that.
<huwshimi> wgrant: The pqm failure complained about the commit message being in the wrong format. I assume that's because the branch is attached to a bug that already has a branch that has landed. Do I need to do something to get it to play nice for the second branch?
<StevenK> huwshimi: So, someone needs to fix db-devel.
<StevenK> I suspect the easiest thing to do is to rollback the milestonetag rev
<huwshimi> StevenK: I think that was intended for wgran
<huwshimi> *wgrant
<StevenK> huwshimi: The PQM failure complained about the commit message not containing [testfix] since db-devel failed on buildbot.
<StevenK> That's what "We're in testfix" means.
<StevenK> I'll fix it after lunch.
<huwshimi> StevenK: Oh right
<huwshimi> StevenK: But it also complained on Friday when other branches of mine landed
<huwshimi> StevenK: I thought it might have been because the commit message was missing [bug=894442]
<StevenK> huwshimi: PQM doesn't care about the content of the tag, it would happily deal with [bug=999999]
<StevenK> wgrant: Diff updated, if you're happy with it, I'll self-review and toss it at ec2.
<huwshimi> StevenK: I think it does. It's failed because of the commit message: http://paste.ubuntu.com/774951/
<StevenK> Note the lack of [bug=
<wgrant> huwshimi: You need either [bug=foo] or [no-qa]. That is why that one failed.
<StevenK> Or [no-qa]
<wgrant> But the current failure is because of testfix.
<huwshimi> StevenK, wgrant: Yes, that's what I was saying :)
<StevenK> wgrant: No fair approving. But thanks. :-)
<huwshimi> I was also wondering why it might not have appended that, and thought it might have had something to do with the bug already having a landed branch
<lifeless> wgrant: StevenK: It landed before I commented, which is fine; its not something I'm going to veto anyhow, because we do need to solve the users problems; I'm hopeful we can solve the deeper cause of friction.
<StevenK> lifeless: I was about to revert it anyway
<StevenK> lifeless: Did you see the two bugs I linked?
<StevenK> wgrant: Did we disable the beta bug listing on qas/prod?
<lifeless> StevenK: I'm on leave; not following much of anything. Do I need to see them ?
<wgrant> StevenK: It's disabled on prod.
<wgrant> And qas is a separate team.
<StevenK> lifeless: I'd like to close them, I was wondering if you agree.
<StevenK> wgrant: Which team on qas, I'd like to QA rick_h__'s change.
<wgrant> custom-buglisting-demo
<StevenK> wgrant: Thanks, qa-ok
<lifeless> StevenK: links ?
<StevenK> lifeless: bug 788518 and bug 805546
<_mup_> Bug #788518: staging services share OOPS prefix <critical-analysis> <Launchpad itself:Triaged> < https://launchpad.net/bugs/788518 >
<_mup_> Bug #805546: persontransferjob does not have a unique oops prefix <Launchpad itself:Triaged> < https://launchpad.net/bugs/805546 >
<lifeless> so, uhm
<lifeless> what report will ptj be using now ?
<lifeless> will someone be able to tell its ptj that an error occurs in
<lifeless> for the staging one, definitely downgrade as we are no longer losing oopses
<StevenK> Downgrade, or close?
<lifeless> if we have reasonable reporters for ~everything on staging, then its closable. I'm not sure if we do yet - because not everything uses configure(section=)
<StevenK> lifeless: I just fixed checkwatches
<wgrant> We should just use script_name and be done with it.
<lifeless> things that don't reconfigure with a specific section will be getting logged as 'STAGING' whether or not they are an appserver.
<lifeless> indeed, doing a systematic fix across the system is a great idea. Its a little more than script-Name though
<wgrant> Slightly.
<StevenK> lifeless: Happy to slam it to High?
<lifeless> StevenK: high makes sense to me, along with a comment about the status, of course
<lifeless> wgrant: jobs need consideration
<StevenK> lifeless: Which you made in October
<wgrant> Yes.
<wgrant> That's the only real issue I can see.
<lifeless> StevenK: other code changes have happened since then, so its worth noting that they *haven't* corrected everything.
<lifeless> StevenK: its almost always worth a short comment when changing things, helps keep everyone on the same page
<lifeless> wgrant: I'd like to move all lp to a single reporter, and use topic to split out scripts
<lifeless> wgrant: be a good idea to do at the same time
<wgrant> Once topics exist, sure :)
<lifeless> wgrant: they already exist
<lifeless> wgrant: its where pageids are reported
<wgrant> Ah, true.
<StevenK> huwshimi: You can reland your branch
<huwshimi> StevenK: :D
<huwshimi> thanks
<StevenK> huwshimi: Sorry, I could have told you 15 minutes ago, but I got distracted.
<huwshimi> StevenK: I can't believe you would do that to me
<StevenK> Hah
<lifeless> is https://bugs.launchpad.net/launchpad/+bug/39630 a dup of the other domination bugs ?
<_mup_> Bug #39630: Older packages are removed when newer versions FTBFS <boobytrap> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/39630 >
<lifeless> also, I'm amazed https://bugs.launchpad.net/launchpad/+bug/40096 hasn't had a look-in
<_mup_> Bug #40096: do not show architecture 'all' builds as i386 <confusing-ui> <lp-soyuz> <oem-services> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/40096 >
<nigelb> Bug listing is disabled for beta testers temporarily or something?
<wgrant> lifeless: Howso?
<wgrant> nigelb: Yes.
<nigelb> wgrant: Ah! Thanks.
<lifeless> wgrant: howso to which ?
<wgrant> lifeless: The latter.
<lifeless> wgrant: because it looks like a shallow UI tweak
<wgrant> lifeless: ROFLOL
<wgrant> How do you propose to determine that? :)
<wgrant> We'd need to set a flag on the build at upload-time.
<lifeless> wgrant: that its only arch-any ?
<wgrant> arch-all, but yes.
<lifeless> wgrant: bah, ENOPACKAGINGENOUGH
<wgrant> And then what if we now confuse people because normal i386 builds still build arch-all, so you have to wait for them anyway?
<lifeless> you mean 'we dispatch arch-all builds to the i386 queue' ?
<wgrant> In the case of an arch-any package, the i386 build builds the arch-all bits
<wgrant> So it should perhaps be presented as "i386 + all"
<wgrant> The UI is not trivial.
<wgrant> And it requires model changes too.
<wgrant> And it matters to nobody after their first 5 minutes with LP.
<lifeless> don't all the builders build -all, and we discard them on other builders? or do we actually call the specific targets..
<wgrant> So it's unsurprising it hasn't been tackled :)
<wgrant> No.
<wgrant> Only nominatedarchindep builds arch-indep.
<lifeless> k
<lifeless> do you happen to remember the bug about only i386 being allowed to build -all ?
<wgrant> Bug #158004
<_mup_> Bug #158004: Arch independent packages are only built on i386 <lp-soyuz> <ppa> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/158004 >
<wgrant> lifeless: Bug #217427 is also slightly relevant
<_mup_> Bug #217427: Please support arbitrary arch/buildd affinity for arch:all builds <feature> <lp-soyuz> <motu> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/217427 >
<wgrant> lifeless: ENOTTRIVIAL
<wgrant> Ah
<wgrant> You've already fixed it.
<wgrant> Thanks.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/40096/comments/3
<_mup_> Bug #40096: architecture 'all' builds are shown as i386 <lp-soyuz> <oem-services> <Launchpad itself:Triaged> < https://launchpad.net/bugs/40096 >
<lifeless> also overhauled the summary
<lifeless> wgrant: so what do you think of bug https://bugs.launchpad.net/launchpad/+bug/39630
<_mup_> Bug #39630: Older packages are removed when newer versions FTBFS <boobytrap> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/39630 >
<wgrant> lifeless: Half fixed.
<lifeless> wgrant: do me a favour and update its metadata?
<stub> lifeless: So since you are obviously now taking your holiday that seriously, I worked out an edge case with moving the 'add new tables to replication' code outside of the downtime window that would seize up replication.
<stub> A db patch that adds a new table, and adds a constraint to an existing table that references the new table. If data gets added to the existing table referencing the new table before the new table is replicated, we are screwed.
<stub> We could say we are fine though, provided we promise that no attempts to access the new table are made before the deploy has fully completed (so no code 'if this table exists, fill in this data')
<lifeless> stub: cannot add references outside downtime anyway
<lifeless> stub: because they require exclusive locks to update the table mgmt triggers
<stub> lifeless: We create the new table and the reference during downtime, we exit the outage, we add the new table to replication (proposed workflow)
<lifeless> stub: oh right, different case.
<lifeless> stub: uhm, yes, so we know that no code can possibly be using the new column or table
<stub> yer, I can't think of how this could bite launchpad.
<stub> Unless triggers are involved :-)
<lifeless> stub: a default value that causes trouble could be made, but that is something we can pick up in review
<stub> A default value will not be referencing a non-existing row in the new table
<stub> We could only shoot outselves in the foot with a trigger. Like BugSummary.
<stub> So if we go ahead with the change, we would need to ensure that creating tables and any code using the tables (standard or triggers) must land separately and be deployed separately
<lifeless> we could define an empty trigger so that the definition is there
<lifeless> can update the procedure in NDT, but not the existence/absence of a trigger
<stub> So it is an obscure case to catch during review, and very rare. So it is unlikely to happen, but we are likely to miss it if it does happen.
<stub> Recovery would not be amusing on the live system.
<lifeless> we can shoot ourselves in the foot many different ways
<lifeless> uhm
<lifeless> I'm ok with the risk, if we document it well and prominently
<lifeless> are you?
<stub> I'll sleep on it :)
<lifeless> I can think of other ways we can mess ourselves up and not have qa catch it :)
<lifeless> (like e.g. bad indices or queries)
<wgrant> stub: We're in huge trouble if we're touching a not-yet-replicated table anyway, aren't we?
<stub> lifeless: That doesn't involve taking the system down, manually copying data to slaves and massaging replication events
<wgrant> Because the new data won't get replicated...
<stub> wgrant: Only if it is referenced by a replicated table, which is this case
<wgrant> (unless you manually clone it across first, which seems silly given it never needs to happen)
<wgrant> stub: Does adding the table to replication also replicate the existing data?
<wgrant> I assumed it would not.
<stub> It will
<wgrant> Ah
<wgrant> That seems excessive.
<wgrant> But perhaps.
<stub> The trick is that adding a table to replication won't happen until pending events have been processed, and in this case pending events won't be processed because the slave is attempting to insert data its constraints won't let it.
<wgrant> Right.
<lifeless> stub: I'm not sure the constraints are checked on the slave
<lifeless> stub: because slony disables all the triggers on the slaves, and constraints are triggers
<stub> Hmm... easiest recovery would actually be to disable the foreign key constraints on the slave, add tables and wait for catch up, and reenable foreign key constraints.
<lifeless> stub: I suggest trying it locally and seeing if it actually borks
<wgrant> I was thinking I'd seen constraint violations on a slave locally.
<stub> foreign key constraints are not disabled IIRC.
<wgrant> But that's only from a unique index.
<wgrant> So it's possible they're disabled.
<lifeless> unique indices will stay live, thats index structure
<wgrant> Exactly.
<wgrant> I was just thinking I had evidence that they were left enabled, then realised that it wasnt.
<stub> Are continual WARNINGs that we need to fix HIGH or CRITICAL?
<stub> Bug #906193
<_mup_> Bug #906193: BugHeatUpdater never completes <Launchpad itself:New> < https://launchpad.net/bugs/906193 >
<lifeless> arguably critical - we have no way to handle/detect 'garbo job X is not getting to run' and thats an operational issue
<lifeless> being able to detect that probably involves fixing the 'blah was interrupted' warnings
<lifeless> poolie: http://pad.lv/OOPS-78b4ff3d44483f940b04d6a2ef5c321e 404's
<stub> Only emailing WARNING and above, we could easily enough tie that into alerts
<lifeless> sorry, not enough context to grok that
<lifeless> wgrant: just got an ajax oops link to work \o/
<lifeless> wgrant: OTOH https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-78b4ff3d44483f940b04d6a2ef5c321e
<stub> We (should be) running scripts logging everything to file, but only emitting WARNINGS and above to stdout.  We could tie that into nagios.
<stub> Or just grep the log files for WARNING|ERROR|CRITICAL
<poolie> hi lifeless, ok
<lifeless> we recently overhauled all the cronscripts in prod to log everything to a file
<stub> If todays log and yesterdays log contain warnings or above, flag the service as failed.
<lifeless> high logging levels generate OOPSes already
<lifeless> stub: so being interrupted isn't a problem per se; its being starved for days that matters, we need something with a longer view of things, or something.
<stub> Yes, so a day or two running with warnings should be enough to prompt someone to take a look and either file a bug or get the alert disabled for a few days until transient issues have been cleared.
<stub> We could also happily enough log each garbo job to ScriptActivity
<stub> each task I mean
<lifeless> stub: indeed, though after the rewrite please :)
<lifeless> stub: does a WARNING log item cause an OOPS or is it ERROR and above ?
<stub> Sure. Log file grepping -> nagios could be done right now if we want
<stub> It should be WARNING and above.
<stub> WARNING should generate an informative OOPS, ERROR and above a standard OOPS.
<lifeless> ok, so we're getting warnings nows. I agree with your analysis FWIW - just leave the reporting as is and fix the issues that show up.
<lifeless> stub: 'informative' doesn't exist anymore as a flag, got pruned in the redo.
<lifeless> wgrant: could you do me a favour and expand on the rather brief description in https://bugs.launchpad.net/launchpad/+bug/78466 ?
<_mup_> Bug #78466: FTPArchiveHandler doesn't support allowed_suites <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/78466 >
<wgrant> lifeless: It's not clear?
<lifeless> wgrant: its opaque as a frog sitting in ink
<lifeless> cjwatson: are you perhaps around ? could use your input on bug 87012
<_mup_> Bug #87012: Cannot start developing next ubuntu release before the prior one is released <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/87012 >
<poolie> lifeless, fixed, i think, except i'm now having network trouble
<lifeless> poolie: thanks!
<wgrant> lifeless: I think it's probably just about obsolete.
<wgrant> Not much uses allowed_suites directly any more.
<wgrant> They just check dirtiness, which a-f already does.
<lifeless> wgrant: so, invalid?
<wgrant> I think so.
<wgrant> In a disaster-recovery situation we're thoroughly screwed anyway, so it doesn't really matter.
<lifeless> wgrant: doit :)
<wgrant> Is done.
<lifeless> stub: is https://bugs.launchpad.net/launchpad/+bug/90809 in progress ?
<_mup_> Bug #90809: Launchpad should run with standard_conforming_strings=on in postgresql.conf <lp-foundations> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/90809 >
<lifeless> stub: actually, what I mean is
<lifeless> stub: did we do this, or are we in deep dodo with respect to it ?
<adeuring> good morning
<wgrant> lifeless: "This is a broken-by-design issue: its not safe to run two different
<wgrant> LPCONFIG's from the same source instance, ever."
<wgrant> We do that all the time.
<wgrant> Mostly on loganberry/ackee
<lifeless> then we're freaking lucky that its not breaking all over the shop
<wgrant> But also cocoplum/germanium/wildcherry/sourcherry
<wgrant> How?
<lifeless> and we have to stop doing that
<wgrant> What state are they going to write out?
<lifeless> the overrides zcml
<StevenK> bigjools: O hai!
<lifeless> wgrant: its presumably identical for us now, but wasn't in the past; if its not written atomically, we must be freakishly lucky
<StevenK> bigjools: So, we have two issues this weekend and today, both of them due to work your squad has done. buildd-manager broke badly on the weekend due to bugs 905853, 905855 and 906079; and germanium was reverted since the poppy-sftp breakage caused germanium to hit a load of 1700.
<wgrant> lifeless: Hm, that is remarkably, remarkably evil.
<wgrant> And has very little justification.
<lifeless> wgrant: I went blind when I found it ~ a year ago
<lifeless> wgrant: working on test parallelisation w/out lxc
<wgrant> StevenK, bigjools: Note that poppy-sftp survived until just a few hours ago.
<poolie> lifeless, confirmed that oops url does work
<wgrant> lifeless: I've actually managed never to run into that before.
<poolie> i started turning ec2test into a juju charm
<poolie> ...
<poolie> kind of ok
<lifeless> poolie: charm or charms ?
<wgrant> poolie: I considered that a couple of weeks back.
<wgrant> But decided that juju scared me.
<poolie> lifeless, one charm at first
<poolie> so it's kinda monolithic and a bit of a degenerate case
<StevenK> frankban: Hi. I reverted your milestonetag revision on db-devel since it broke buildbot.
<lifeless> bigjools: hey, so bug https://bugs.launchpad.net/launchpad/+bug/117557 - doesn't that cause an OOPS ?
<_mup_> Bug #117557: Nascent Upload code doesn't check properly for bad distroseries <lp-soyuz> <soyuz-upload> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/117557 >
<frankban> StevenK: ok, thanks
<lifeless> bigjools: you say it won't, but I don't understand how it wouldn't.
<bigjools> lifeless: the exception is caught and translated to an error returned to the user
<stub> lifeless: We are not in deep doodoo - I'll be dealing with standard conforming strings settings with the PG upgrade. IIRC the things that were quoting oldskool were psycopg2 and maybe a few items in comments.sql. Doesn't really matter as the old format is still supported.
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/118870 - you comment that its a strange view, but it looks ok to me.
<_mup_> Bug #118870: $sourcepackage/+changelog only shows one entry per suite (distroseries-pocket) <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/118870 >
<poolie> wgrant, so as an initial wrapper around starting an instance it seems ok
<poolie> using something like this to start ppa builders might be more interesting
<wgrant> lifeless: The strange mangled view I speak of was later filed as bug #144620
<_mup_> Bug #144620: Some displayed sourcepackagerelease changes files don't have attribution <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/144620 >
<lifeless> stub: what about https://bugs.launchpad.net/launchpad/+bug/120404 ? - do we still need to do this audit ?
<_mup_> Bug #120404: Database permission audit <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/120404 >
<wgrant> lifeless: It's not relevant to the bug beyond confirming it wasn't a regression, anyway.
<stub> lifeless: I'd like to, yes
<stub> lifeless: Unless we go ahead and drop permissions entirely ;)
<lifeless> wgrant: I'm not sure that the bug is actionable then, because the changelog looks ok to me
<lifeless> stub: indeed, and thats a curly discussion :)
<wgrant> lifeless: It's indeed no longer as it was described in the bug.
<wgrant> lifeless: It's not ideal, but it's not quite so bad.
<wgrant> It seems to show the changelog_entry from every publication in the series.
<wgrant> Rather than just one per pocket.
<lifeless> wgrant: can you update the bug?
<wgrant> Blah, OK.
<frankban> StevenK: can you confirm the problem is here? https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1596/steps/shell_6/logs/stdio
<StevenK> frankban: The summary is much better for that sort of thing
<StevenK> frankban: But the first error is "ProgrammingError: permission denied for relation milestonetag" which puts the smoking gun firmly in your hand.
<frankban> StevenK: ok, seen
<wgrant> lifeless: Hah
<lifeless> ?
<wgrant> I think archiveremovalredesign fixed it.
<wgrant> But annotate keeps misleading me.
<wgrant> Ah, no, was accidentally fixed in r5613
<wgrant> [r=barry] Fixing bug #179028 (+files does not work for removed
<wgrant> sources). Now SourcePackage class is able to traverse to source
<wgrant> package releases removed from disk.
<_mup_> Bug #179028: +files doesn't work for removed SPRs <lp-soyuz> <Launchpad itself:Fix Released by cprov> < https://launchpad.net/bugs/179028 >
<lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/144620 is stale too
<_mup_> Bug #144620: Some displayed sourcepackagerelease changes files don't have attribution <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/144620 >
<wgrant> lifeless: Fixed
<bigjools> wgrant: only poppy on germanium went bong?
<wgrant> bigjools: Yes.
<wgrant> Same as last time.
<wgrant> I assume it's load-based.
<wgrant> We should watch cocoplum's carefully, but I don't want to revert it unless we have to, because of the publisher optimisations.
<bigjools> yeah
<bigjools> gah it's that weird logging thing again
<wgrant> Really?
<wgrant> The top of the latest log has that.
<wgrant> But only because rotation isn't happening.
<bigjools> bWTF
<bigjools> why
<wgrant> Because logrotate isn't set up, I assume :)
<bigjools> oh it has 2 log files, one is old style one is new
<bigjools> nothing to do with that
<bigjools> there's some recursive logging going on
<bigjools> lifeless: need your halp
<bigjools> https://bugs.launchpad.net/launchpad/+bug/906211
<_mup_> Bug #906211: poppy died in a pulp of its own swappiness <Launchpad itself:Triaged> < https://launchpad.net/bugs/906211 >
<bigjools> the recent changes made no difference
<wgrant> bigjools: Huh, didn't notice that recent bit.
<wgrant> Oh.
<wgrant> bigjools: That's *after* the revert.
<wgrant> Of course your reverted changes didn't help :)
<bigjools> ah
<bigjools> what time was it reverted?
<wgrant> An hour after the >1000 load was reported.
<wgrant> For that hour it was running the new code, but restarted.
<wgrant> 07:42Z the rollback completed
<wgrant> This should be on LPS, hopefully.
<bigjools> geez, log looks perfectly normal
<wgrant> Look towards the end.
<wgrant> During the deathspiral.
<bigjools> hmmm
<bigjools> what bit?
<wgrant> There's very frequent DTPFactory logs or whatever it is.
 * wgrant relooks.
<wgrant> Oh, blah, has it rotated the logs away again.
<bigjools> it's there
<bigjools> poppy-sftp.log.1006
<wgrant> There's also one from around the outage in ~wgrant/poppy-sftp.log
<wgrant> Oops
<wgrant> just ran ls in lp_upload
<bigjools> bad idea
<lifeless> bigjools: hi
<bigjools> lifeless: hi - sorry was looking at wrong log, but still could do with your help!
<wgrant> bigjools: Look at the last couple of seconds before the world ended.
<wgrant> eg. 06:18:48
<bigjools> yeah
<wgrant> Dozens of lines of DTPFactory crap.
<bigjools> hmmm
<bigjools> I see a lot of entries from hyperair
<wgrant> That's true, but from a different IP.
<wgrant> The initial symptom was that SSH auth failed.
<wgrant> But that took a day or so to appear.
<bigjools> that has been happening on branches too
<wgrant> Hmmmmm, that's true.
<wgrant> Very occasionally.
<wgrant> But entirely unexplained.
<bigjools> well some guy filed a question about it, was consistentely failing for him
 * bigjools has cold fingers and can't type today
<wgrant> You mean the question on like Friday about it?
<wgrant> I thought that was poppy.
<lifeless> bigjools: do you think https://bugs.launchpad.net/launchpad/+bug/39630 is high still?
<bigjools> ah yes it was poppy
<bigjools> lifeless: yes
<wgrant> However, I checked when I saw that and it was working at that point.
<wgrant> So it broke then fixed itself :)
<bigjools> this looks like a DoS to me
<wgrant> It does.
<wgrant> But I'm more likely to blame poppy.
<bigjools> right - we need a connection limit
<bigjools> in the meantime, let's have a word with mr hyperair
<lifeless> doing the RT for haproxy on the uploaders will get you that for free
<lifeless> (just saying)
<wgrant> Is that 1962 a port number?
<wgrant> I'm not sure how that logging works.
<lifeless> bigjools: so, you're set, you don't need me ?
<bigjools> lifeless: yes, thanks and sorry to bother
<lifeless> hey no worries
<lifeless> I chose to be on IRC :P
<lifeless> (and to re-triage 75 high bugs...)
<bigjools> wgrant: so I think we can revert the revert
<bigjools> given that the old one has known issues
<wgrant> bigjools: There's still the SFTP issue.
<bigjools> was that happening before the new code rollout?
<wgrant> No.
<bigjools> the sftp code has not been touched
<bigjools> hard to see how it was broken
<wgrant> It's clearly broken it somehow.
<wgrant> NFI how, though :/
<lifeless> bigjools: SSHServer was touched wasn't it ?
<wgrant> It's happened twice now.
<lifeless> bigjools: which is the backing for sftp
<bigjools> lifeless: yes - but only logging
<lifeless> agreed; just being pedantic
<bigjools> lifeless: can you see how that might cause auth fails?
<bigjools> pedanticism is good here
<lifeless> bigjools: only thing I can think of is something logging triggering a barf which then oops, or something.
<wgrant> Remember that auth goes via XMLRPC
<lifeless> did we see oopses for the auth errors?
<wgrant> So it's async.
<bigjools> there's things like "unauthorized login: unable to get avatar id"
<bigjools> wgrant: ?!
<bigjools> lifeless: AHA
<bigjools> there's a timeout failure
<bigjools> right at the same time as the auth starts
<wgrant> timestamp?
 * bigjools pastes
<bigjools> https://pastebin.canonical.com/57356/
<wgrant> Ah, yes.
<wgrant> So there is.
<bigjools> seems to happen on all attempts
<wgrant> I was looking at the wrong hour.
<wgrant> Yeah
<wgrant> So it sits there for 30s.
<wgrant> And then returns a TimeoutError
<bigjools> and as usual Twisted is unhelpful with no traceback
<lifeless> firewall issue?
<wgrant> No.
<wgrant> Since a restart fixes it.
<bigjools> I think someone has got frustrated and set up a loop to keep retrying
<wgrant> Interesting.
<wgrant> bigjools: 2011-12-16 10:19:40
<lifeless> nn
<wgrant> 4 seconds before the first occurrence of the TimeoutError
<wgrant> There's lots of DTPFactory again.
<bigjools> nn lifeless
<wgrant> From that same IP.
<wgrant> Hmm
<wgrant> Interesting.
<wgrant> My tests were 3 minutes before that.
<wgrant> However, I did those same tests on cocoplum and it survived.
<bigjools> wgrant: I can see others connecting to SFTP and validating ok
<wgrant> Where?
<bigjools> 02:22:36+0000
<bigjools> INFO:poppy-sftp.Hooks:Post-processing finished
<bigjools> oh it's ftp
<bigjools> ignore me
<wgrant> heh
<cjwatson> lifeless: 87012> commented
<bigjools> wgrant: so I uploaded to cocoplum over SFTP ok
<wgrant> bigjools: Yes, cocoplum continues to work.
<wgrant> It's not had to be restarted once.
<bigjools> germanium is not happy
<wgrant> Isn't it fine now it's reverted?
<bigjools> and coupled with the other issue where it needs restarting every 6 days ...
<bigjools> ignoring the revert
<wgrant> The new code needs restarting every 6 *hours*.
<wgrant> On germanium.
<StevenK> Perhaps germanium is just getting hit so much harder than cocobanana.
<bigjools> nice,p nearly an hour to do a soyuz logsync
<wgrant> That's been my assumption for a long time.
<wgrant> bigjools: It's because germanium is in its logdeathspiral.
<wgrant> So it's redownloading 30GB of logs every time.
<StevenK> germanium is hellishly overloaded. News at 11.
<bigjools> wgrant: so, right, the first failure was just after your tests
<wgrant> It's not overloaded.
<wgrant> Our code is just shit.
<wgrant> :)
<wgrant> bigjools: Yep
<wgrant> 3 minutes later or so.
<StevenK> Sadly, that isn't news.
<wgrant> bigjools: Very odd.
<wgrant> bigjools: But I did very similar things to cocoplum several times over the weekend, AFAICR.
<bigjools> I'd prefer not to judge until we know everything
<wgrant> And it continues to live...
 * gmb wishes pushing a branch up to a local codehosting instance didn't take so flaming long.
 * gmb finds something else to do in the meantime
<wgrant> gmb: What sort of branch?
<wgrant> It's normally lightning fast for me, unless you get hit with big 2a fetch slowness.
<gmb> wgrant, A bigun. I'm working on bug 808930.
<gmb> My complaint should have read "_this_ branch"
<wgrant> gmb: If you want to cheat, you can just copy it to the right place under /var/tmp/bazaar.launchpad.dev/mirrors/
<gmb> wgrant, Wouldn't I then need to manually create the job(s) for the branch scanner?
<gmb> Or is there a cheat for that, too?
<wgrant> gmb: You can then do a no-op push to create the job.
<gmb> Ah!
<gmb> Yes, of course.
<gmb> wgrant, You wise, wise monkey you. Thanks.
<wgrant> A no-op push may not work -- but if not, just push -r-2 or so to ensure the head changes.
<gmb> Understood.
<wgrant> gmb: You may want to talk to jelmer about this.
<wgrant> Since he has a branch to rework bits of the scanner.
<wgrant> Relevant bits, I assume -- it made it several times slower.
<gmb> wgrant, We talked last week about it. This bug looks very much like death by SQL.
<wgrant> Ah
<wgrant> Didn't know that.
<gmb> Lots of SELECTS and INSERTS in RevisionSet.new()
<wgrant> Ah
<wgrant> In that case an initial scan might not trigger it.
<wgrant> The initial scan is a special-case.
<wgrant> Which is much more efficient in the general case.
<gmb> Ah.
<wgrant> But it depends on the branch.
<gmb> Hmm.
<gmb> Well, I'll get this one into codehosting .dev and see what happens. Having to re-do a bunch of stuff since my Oneiric VM decided to go wonky this morning; I had to restore from a snapshot.
<wgrant> gmb: You may be able to use the example in comment #5 to reproduce the partial scan slowness.
<wgrant> (it's even one of yours)
<wgrant> It's still broken on prod.
<gmb> Ah yes. I hadn't scanned that far down the bug :). I was working with info from benji, since I've taken this over from him.
<wgrant> There may well be two reasonably separate bugs here.
<gmb> wgrant, You mean death by SQL + the bug for which jelmer has been making optimisations?
<gmb> Or something else
<gmb> ?
<wgrant> The initial and partial scans may be separate code bugs, but both are covered in that single report.
<gmb> Ah, ISWYM.
<gmb> Good point.
<wgrant> Because we know that scanning a Launchpad branch from scratch works.
<wgrant> But bringing up to date a Launchpad branch that is a year old appears not to.
<gmb> Indeed.
<wgrant> Then there's another separate issue, where inserting a whole lot of new Revisions (not BranchRevisions) takes too long.
<wgrant> Mostly occuring with new massive svn/git imports.
<wgrant> So there's probably at least three bugs here, conflated in that one report :)
<gmb> Oh, joy.
<wgrant> I aim to please.
<gmb> :)
<gmb> I'll try and explore the first two, at least. Not sure I'll have time to tackle the third problem.
<wgrant> You may have little choice.
<gmb> Sssh
<wgrant> Scanning a massive branch into a dev instance may well fail with issue #3, because all the Revisions are new.
<gmb> Don't want to think about that yet...
<gmb> Actually, that would fit well with what Benji's observed so far, AFAICT.
<gmb> (I've got lots of how-to-follow-his-tracks info but no actual data to analyse)
<wgrant> Yeah
 * gmb lunches
<stub> We should fix that. No need to create all the revisions in one transaction.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugtasks: 3*10^2
<rick_h__> benji: have a sec to review? https://code.launchpad.net/~rharding/launchpad/batch_nav_900900/+merge/85935
<rick_h__> or sorry, I'll bug jcsackett since he just arrived and originally peeked at it
<rick_h__> so benji nvm and jcsackett can you look at https://code.launchpad.net/~rharding/launchpad/batch_nav_900900/+merge/85935 again when you get settled?
<jcsackett> rick_h__: happy to look again. :-)
<rick_h__> jcsackett: ty much
<jcsackett> same branch i looked at the other day?
<benji> k
<rick_h__> jcsackett: yep, just modified the test for the lower batch
<rick_h__> jcsackett: missed the test over in lp/app when the change was in canonical/launchpad so thanks for catching that and saving me a test run
<jcsackett> rick_h__: your welcome. with the change mod, this is r=me. :-)
<jcsackett> s/change mod/test change/
<rick_h__> jcsackett: awesome, thanks
 * jcsackett goes to drink his coffee so he typoes less.
<rick_h__> has anyone run into this ec2 land error lately? https://pastebin.canonical.com/57375/
<rick_h__> it worked over the weekend, so wondering if my package updates from last night have caused me pain now
<jml> is it possible to get the binary package urls from a BinaryPackagePublishingHistory object?
<jml> looking in publishing.py, the answer is no.
<bigjools> jml: urls to what exactly?
<jml> bigjools: the debs
<bigjools> jml: ah, no, but it's trivial to contruct a path to the archive
<jml> bigjools: what SourcePackagePublishingHistory.binaryFileUrls gets you the URLs for, except from one BPPH rather than the BPPH's associated with an SPPH
<bigjools> that's returning librarian URLs
<bigjools> I think it was done for the security guys
<jml> so it's not actually recommended to download files from the URLs that Launchpad provides?
<bigjools> either is fine
<bigjools> but I tell people to use the archive url, because it's less likely to be unavailable
<jml> bigjools: and the way that's trivially constructed is to inspect the distro, component and binary package name to make a URL to the deb in the pool area, right?
<bigjools> that sort of thing
<bigjools> but if you want to get it from the API, file a bug :)
<jml> well, I might submit a patch.
<bigjools> even betterer
<cjwatson> benji: could you have a look at https://code.launchpad.net/~cjwatson/launchpad/soyuz-test-mail-ordering/+merge/85979 ?  Just a test fix
<benji> cjwatson: sure
<bigjools> jml: actually you need to return a proxied librarian file like the other bits of code, in case it's a private archive
<jml> bigjools: yeah. I was planning on factoring out what's in SPPH, and have SPPH delegate to BPPH to figure out binary urls.
<bigjools> jml: ok, careful it doesn't end up potato programming
<jml> bigjools: will do.
<jcsackett> sinzui: do you know if wallyworld is actually working on bug 885524?
 * sinzui looks
<sinzui> jcsackett, not directly
<jcsackett> sinzui: yeah, it looks like he's working on a related bug. so i'll leave that alone to avoid stomping on toes.
<sinzui> jcsackett, wallyworld__is working on bug 904295 which is a pre-req and may actually fix the issue
<sinzui> jcsackett, lets mumble about the remaining cards. I have to pick one too
<jcsackett> sinzui: dig. be on mumble in just a moment.
<benji> cjwatson: the branch looks good
<rick_h__> adeuring: deryck  https://pastebin.canonical.com/57375/
<deryck> rick_h__, we lost you.
<adeuring> deryck: https://code.launchpad.net/~adeuring/launchpad/bug-903359/+merge/86244
<cjwatson> benji: lovely, thanks.  I have no PQM access; do you think you could send it to EC2 for me?
<benji> cjwatson: sure
<cjwatson> ta
<benji> cjwatson: EC2 has been engaged: http://ec2-184-72-167-164.compute-1.amazonaws.com/
<deryck> rick_h__, did your bazaar.conf change by any chance?  i.e. does it still exist in $HOME/.bazaar ?
<rick_h__> deryck: no, it's still there and seems normal
<rick_h__> deryck: I did have to make some changes to my locations.conf to get a manual land in pqm over the weekend
<rick_h__> but that was just adding some email settings. Removing them again doesn't help me
<deryck> rick_h__, ok, it seems to be choking over email and signing the submission some how.
<rick_h__> deryck: right
<deryck> gary_poster, are you around?
<gary_poster> deryck, hi
<deryck> gary_poster, hi.  Trying to help rick_h__ fix this error: https://pastebin.canonical.com/57375/  and I'm really not sure what's up. Have you seen that before?
<gary_poster> deryck, no.  that's a curious one.  The bzr-2.5 makes me suspicious though
<gary_poster> I wonder if 2.4 will be better
<deryck> ah yeah, missed that.
<gary_poster> though I guess that means we are all using 2.5 though?
<deryck> and it's using the egg version not his local bzr.
<gary_poster> right
<deryck> I didn't realize we did that.
<gary_poster> me either deryck :-P  or at least didn't remember, I dunno :-)
<deryck> heh
<rick_h__> my local bzr is  2.5.0~bzr6383.6136~ppa4002~oneiric1
<deryck> seems kind of wrong to me, but I'm sure someone had a good reason to.
<deryck> but I guess if we're all on dailies we'd be on that.
<gary_poster> rick_h__, I see this on my oneiric: $ bzr --version
<gary_poster> Bazaar (bzr) 2.4.2
<rick_h__> gary_poster: yea, I did an upgrade this morning and that seems to have blown me up
<gary_poster> ah ha
<rick_h__> gary_poster: I saw bzrlib and launchpadlib updates
<rick_h__> gary_poster: yea, just call me the canary in the coal mine I suppose
<gary_poster> rick_h__, heh :-)  so we have two options I guess
<deryck> yeah, I'm still on 2.4.2 as well, though I thought I ran the daily.  Guess I killed it at some point.
<gary_poster> 1) try going back to 2.4.2
<gary_poster> 2) dig in and fix it.
<gary_poster> :-P
<gary_poster> I vote for option 1 but somebody has to pay the price
<gary_poster> it would be nice if that person had some bzr experience though
<gary_poster> rick_h__, can you try option 1?
<rick_h__> gary_poster: yea looking into it. Set a pin and upgrade again?
<gary_poster> rick_h__, yeah.  (I'd do it in aptitude but that's because that's what I'm comfortable with)
<rick_h__> gary_poster: ok, looking. Been years since I've pinned a package
<gary_poster> The thing that will need to be fixed will be either pqm or bzr (or both conceivably)
<gary_poster> rick_h__, do it in aptitude then. It's easy :-)
<gary_poster> Maybe synaptic too
<rick_h__> gary_poster: installed
<gary_poster> cool
<rick_h__> gary_poster: so the 'F' command?
<gary_poster> rick_h__, this is what I do, for a temporary test change:
<gary_poster> find the package in aptitude
<gary_poster> Press enter
<gary_poster> scroll to the bottom
<gary_poster> you should see the different options to install
<gary_poster> with an i on 2.5 presumably
<rick_h__> gary_poster: yea, I see the p    2.4.2
<gary_poster> right
<gary_poster> press "+" on that
<gary_poster> that will be a temporary install I think.  If not,
<gary_poster> yeah, no that should do it.  If you want to keep it, use "=" on the package in the future
<rick_h__> ugh, looking, wants to remove 134 package
<gary_poster> (or remove the bzr daily ppa :-) )
<gary_poster> :-(
<deryck> rick_h__, also, if you're on the daily, you can remove that from sources and go back to the 2.4.2 in O.
<gary_poster> just do downgrade?
<gary_poster> to
<deryck> sorry gary_poster just said what I said :)
<rick_h__> gary_poster: well I hit + on it and it says "Suggest 134 removals"
<rick_h__> gary_poster: not sure if it's going to carry out that suggestion though I suppose
<gary_poster> deryck, I don't think that alone will downgrade (because of doing something similar recently), you have to explicitly downgrade or else the packages hang around
<rick_h__> yea, if I remove the ppa I'd still have to remove/reinstall or downgrade
<gary_poster> rick_h__, ok, if it is not working easily then this is a blindman leading...you take a list at the suggestions and see what you think, or we can find someone else to talk to :-)
<gary_poster> I mean, I am a blind man leading :-)
<rick_h__> gary_poster: going to see if I can remove the ppa and see what it wants to do to remove bzr
<rick_h__> then reinstall
<gary_poster> ok
<gary_poster> hopefully this is actually the problem :-P
<gary_poster> it seems like it though
<gary_poster> I need to head out soon
<rick_h__> gary_poster: ok thanks
<gary_poster> my wife and baby are calling me to an early lunch :-)
<deryck> adeuring, r=me on that change.  looks great!  Thanks.
<adeuring> deryck: thanks!
<deryck> rick_h__, gary_poster -- yeah, I'm sorry, I meant drop the daily from sources and uninstall/reinstall.  That's how I usually do it. Maybe that's the clunky way.
 * deryck has weak apt foo
<rick_h__> deryck: yea, that's worked for removing/adding
<rick_h__> deryck: working on updating devel/make clean/etc to test out
<gary_poster> rick_h__, I suggest shooting a note to launchpad-dev about what you encountered so that others know the problem and can investigate
<gary_poster> assuming this fixes it
<rick_h__> gary_poster: will do
<gary_poster> thx
<rick_h__> deryck: gary_poster landing is processing thanks for the notice of the 2.5 and help with that
<bigjools> deryck: apt-get package=<release name>
<deryck> bigjools, ah, thank you.
<bigjools> de nada
<deryck> adeuring, I reproduced the double count locally, saw it doing: https://bugs.launchpad.dev/ubuntu/++profile++show/+bugs
<deryck> adeuring, and this fixes it:  http://pastebin.ubuntu.com/775460/
 * adeuring is looking
<deryck> I had to build up my local data to get a couple batches as we discussed this morning.
<deryck> without the extra batch we don't even do a single count because we don't need lastBatch.
<adeuring> deryck: very nice fix!
<deryck> adeuring, thanks!  seems simple enough too.
<adeuring> yeah
<deryck> I'll fix my branch to use this and update the MP.
<deryck> adeuring, I guess we can drop all the stuff to avoid lastBatch now.  I did the pastebin patch in devel, where the older use of lastBatch still exists.
<adeuring> deryck: right, the old patch should be obsolete
<deryck> adeuring, ok, cool, that was my thinking too.
<deryck> adeuring, do you care to weigh in or review the updated https://code.launchpad.net/~deryck/launchpad/avoid-extra-buglist-count-901124/+merge/85940 ?
<adeuring> deryck: sure
<adeuring> i'll look
<deryck> adeuring, thanks
<jml> is there an API for determining a source package name given a binary package name?
<adeuring> deryck: r=me
<deryck> adeuring, thanks much!
<sinzui> jml: no. guessPublishedSourcePackageName() is closed to what you want, but I was thinking of removing it once we update all the callsites to use the new DSP vocab
 * sinzui ponders if the vocab is public in a way to used it
<lifeless> probably not
<deryck> lifeless, hi.  I believe I have finally *really* fixed the double count issue now.
<deryck> lifeless, I'd appreciate you looking again at the MP if you don't mind.
<lifeless> deryck: I have
<lifeless> deryck: :)
<deryck> ah ok :)
<lifeless> deryck: it looks good to me, the new-batch object is a common cause of this
<deryck> thanks!
<lifeless> deryck: I didn't look at the larger context; a lot of other views use a cached property rather than an explicit instance variable
<lifeless> but 6/1 1/2 dozen of the other
<deryck> yeah, I'm not picky either if you prefer the other way.
 * lifeless shrugs
<lifeless> deryck: I *had* replied, but lp hasn't seen it.
<lifeless> deryck: I suspect incoming mail processing is naffed
<deryck> ah, I wondered if that's what you meant.
<lifeless> and now I can't find it in my outbox.
<lifeless> ENOIDEA.
<lifeless> something odd is happening though, I'm trying to add a comment on https://code.launchpad.net/~deryck/launchpad/avoid-extra-buglist-count-901124/+merge/85940 and its stuck on 'Saving' for way more than the 30s ha proxy timeout :(
<deryck> hmmm, weird.
<deryck> let me try....
<deryck> lifeless, I just added a comment no problem.
<deryck> less than a second XHR call.
<lifeless> weird
<lifeless> oh
<lifeless> merge proposal
<lifeless> -> long poll
<lifeless> I wonder if I'm hitting my browser host connection limit
<lifeless> yes, I was
<lifeless> JULIAN!
<lifeless> :)
<deryck> ah ha! :)
<lifeless> bug 906482
<lifeless> bug 906482 ?
<lifeless> bah https://bugs.launchpad.net/launchpad/+bug/906482
<lifeless> deryck: this explains why I hadn't commented previously, it was stuck in a browser tab
<lifeless> deryck: for an hour or so :)
<deryck> heh, ouch
<deryck> lifeless, but I've seen your browser.  You really do have too many tabs open. ;)
<lifeless> see, thats what 16GB of ram is *for*
<deryck> heh.
<deryck> I stand corrected then. :)
<lifeless> that and skyrim
<deryck> no, that's what a PS3 is for. ;)
<lifeless> heh, Lynne plays it on the PS3
<deryck> that and dcuo.
<lifeless> it is -much- shinier on a PC
<deryck> yeah, most games are actually.
<deryck> but it's so comfortable gaming from my bean bag on the floor in front of my 47 inch TV.
<lifeless> there was a period where the consoles were better, but then moores law kicked in :)
<deryck> heh, yeah
<lifeless> bean bags++
<lifeless> anyhow, as I'm on leave, I think I'm going to go exercise some of that there gaming stuff ;)
<deryck> heh. 3 more days for me.
<lifeless> sinzui: nice curly bug there
<sinzui> ?
<lifeless> 611617
<sinzui> lifeless, yes. I really am not sure what to do to close it
<lifeless> I don't think the person-linking aspect matters, because the uploader will always be a person (they have to gpg sign the package, which they cannot do as the team)
<lifeless> the maintainer field can be a team - its free form text; there are two issues:
<lifeless>  a) only allow specifying a private team as maintainer when the uploader can see that team/ is a member of the team
<lifeless>  b) disclosure of the team because of this
<lifeless> -> I think
<sinzui> I do not think the team should be disclosed in the case of a private team's archive. Copying to a public archive makes the relationship public
<lifeless> right, I'm saying that there is a probing attack
<sinzui> ah
<lifeless> i could upload to my public ppa with the maintainer set to 'private-canonical-XXX'@lists.l.n
<lifeless> and if it links, voila, I can see the team.
<sinzui> lifeless, I was thinking of migrating the logintoken code to lp.services tonight. logintoken is not a good name. I was considering naming the module authentication, then I consider the value it offers is a verification workflow. Maybe lp.services.verification
<lifeless> thats the email verification stuff ?
<lifeless> there is a bug about that using bounces@c.c., and a suggestion to use a new envelope-sender address. Might want to fix that in passing
<lifeless> it should really be a separate service I think though, let anyone at the company trigger a 'is this a real email' probe.
<lifeless> sinzui: I've replied on stakeholders, I hope I make sense
<sinzui> lifeless, agreed. Thanks for the stakeholder reply too
<poolie> hi all
<poolie> rick_h__, gary_poster, hi, what is the bzr problem you're talking about?
<gary_poster> poolie, hi.  jelmer replied on the list about it
<gary_poster> poolie, see launchpad-dev thread "upgrade today got me bzr 2.5 and broken ec2 land"
<rick_h__> poolie: yep, did an upgrade this morning and got a bzr 2.5 with the bunch
<rick_h__> poolie: which pqm plugin wasn't ready for but it's coming it appears
<poolie> ok
<poolie> a bit out of sync
<poolie> jelmer, we might need some glue to let you check flags from inside the ssh server?
<poolie> which would be worth having anyhow
<jelmer> poolie: that would be nice
<jelmer> poolie: I was actually wondering about the forking lp service
<jelmer> poolie: it seems we have enough data to let people switch to the forking server by specifying a different bzr_remote_path
<jelmer> so that we could have the forking lp server available without immediately forcing it on all codehosting users
<poolie> that would probably work
<poolie> we could possibly do something like this in a client controlled way
<lifeless> we need, if we don't have it already, and xmlrpc flag checking api
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 3*10^2
<sinzui> wallyworld__, see https://bugs.launchpad.net/launchpad/+bug/855670
<mwhudson> lifeless: i add a xmlrpc flag checking api
<mwhudson> i keep meaning to revisit anonymous ssh connections using it
<wgrant> lifeless: "and if the package is
<wgrant> copied to a new context, inform the copier it will lead to disclosure"
<wgrant> Uh, how exactly?
<wgrant> We can't change *every single API in Launchpad* to prompt if there's a private team involved.
<wgrant> That's just stupid.
<sinzui> StevenK, this is the bug I was thinking of bug 567088
<StevenK> https://bugs.launchpad.net/launchpad/+bug/906151
<poolie> lifeless, http://www.stathat.com/
<jml> sinzui: that means there's no functional way to get binary package urls from bpph objects over the API. that's ok. will fix.
<poolie> o/ jml
<jml> poolie: hi
<poolie> hi there
<wgrant> jml: There's not always a corresponding file in the archive.
<wgrant> jml: You may be better off exposing the webapp URL of the file, on BPPH or BPB.
<jml> wgrant: and yet spph has binaryFileUrls
<wgrant> Right, those are webapp URLs.
<wgrant> Not archive ones.
<jml> wgrant: I'd be happy w/ webapp urls. and if I were to expose that'd be the natural path.
<jml> would just refactor binaryFileUrls to delegate properly
<wgrant> The source name is not relevant to those, so you can probably even calculate them now.
<jml> wgrant: I haven't even looked to see what they are
<jml> black box ftw.
<wgrant> Heh
<wgrant> /ubuntu/+archive/primary/+files/FOO_VERSION_ARCH.deb
<huwshimi> If you add a second branch to a bug that is already fix-released will that branch get deployed to qa staging?
<huwshimi> (this bug: https://bugs.launchpad.net/launchpad/+bug/894442)
<StevenK> huwshimi: Yes
<huwshimi> hmmm...
<huwshimi> StevenK: Any reason why this one hasn't then?
<StevenK> huwshimi: Is the branch linked to the bug?
<huwshimi> StevenK: Yeah
<huwshimi> StevenK: Wait, it doesn't look like it merged
<StevenK> I was about to say :-P
<huwshimi> StevenK: shush
<lifeless> poolie: yeah, basically statsd
<lifeless> poolie: we're outside the free zone anyhow, ignoring disclosure considerations
<poolie> lifeless, yeah, basically statsd+better displays, i think
<poolie> free zone?
<poolie> i'm not saying we should actually use it
#launchpad-dev 2011-12-20
<jelmer> hmm, a branch name from StevenK that doesn't make me go "WTF?"
 * jelmer is disappointed
<StevenK> jelmer: Which one?
<jelmer> StevenK: refactor-imports-redux
<StevenK> If it doesn't make you go WTF, Diff against target: 11295 lines (+1298/-1451) 531 files modified will
<wgrant> sinzui: Hm, so just launchpadstatistic, librarian, logintoken and temporaryblobstorage left.
<lifeless> poolie: we do more than 1M pages a day, we'd blow past their taster-account in no time ;)
<wgrant> sinzui: I have a branch from a couple of weeks back for temporaryblobstorage.
<StevenK> wgrant: sinzui was going to tackle logintoken
<poolie> lifeless, how can i get the raw form of an oops?
<poolie> or anyone
<lifeless> from where
<StevenK> wallyworld__: O hai. https://code.launchpad.net/~stevenk/launchpad/productset-all-lies/+merge/86314
<wallyworld__> StevenK: looking now
<wallyworld__> StevenK: any tests to amend?
<StevenK> Not that I could see
<wallyworld__> ec2 will tell us i guess
<StevenK> Tempted to just -vvm registry
<lifeless> sure there is a test to add ?
<StevenK> I'd be surprised if ProductSet:+all wasn't tested by some doctest
 * StevenK runs registry tests
<wallyworld__> StevenK: i've +1'ed it but it would be cool if there were a doc test that could be added to
<wallyworld__> or whatever
<lifeless> poolie: :/
<poolie> lifeless, ?
<poolie> i found it on disk on devpad
<poolie> why the frownie?
<lifeless> I may have misinterpreted your answer to my reply to your advert for bson-dump
<lifeless> ELONGCONTEXT
<poolie> oh
<poolie> i agree it would be good to do
<poolie> i don't know why i didn't put it elsewhere in the first place
<poolie> it was a while ago
<poolie> perhaps all the external oops stuff seemed too much in flux?
<poolie> or there were too many options for where to put it, so i took a lame default
<lifeless> I felt, apparently wrongly, that you were being a bit uhm, 'well I've done, nyar'.
<lifeless> the perils of low bandwidth comms
<poolie> ah, not really
<lifeless> poolie: I'd really like to delete utilities/*
<poolie> feeling a bit "omg so few days before holidays etc"
<poolie> if you tell me a specific place to move it to that will help
<lifeless> heh, fair enough.
<poolie> i guess, something that knows about bson encoding and will be installed for all developers
<poolie> i think splitting stuff is good but a minor consequence is that 'where do i do this' gets a bit harder
<lifeless> I'd put it either in oops-datedir-repo or oops-tools itself
<lifeless> its not urgent to move it
<lifeless> if you're busy with other stuff in the holiday lead up, just ignore it.
<poolie> i'll move it to oops-tools
<StevenK> from bzrlib.plugins.builder.recipe import RecipeParseError
<StevenK>     ImportError: No module named builder.recipe
 * StevenK peers
<jtv> StevenK, wgrant: I'm sorry to hear that I broke buildmaster again.  Never expected there'd be no missed spots at all, but didn't expect this many either.
<StevenK> Did Gavin land the fix?
<StevenK> jtv: The test coverage of buildd-master is just *horrid*.
<StevenK> Ah, reverted in r14552.
<StevenK> But marked with the bugs, and not incr. Sigh.
<jtv> Should that be incr?
<jtv> I completely forgot about that tag.
<StevenK> jtv: You're rolling back the code, so I guess the next step is fix the three bugs and land it again.
<jtv> Well I'm not rolling anything back personally; I have to go back to clearing out the house.
<jtv> But yes, I'm afraid that's the process.
<StevenK> jtv: If so, our process says the 3 bugs should be closed. Except they won't be fixed.
<jtv> Oh.
<wgrant> So, PQM's been whinging about a conflict for 6 hours now.
<StevenK> jtv: The qa-tagger will tag them needstesting, they'll get marked untestable, and rolled out.
<wgrant> Is someone going to fix that at some point?
<StevenK> I'm trying to sort out ImportError: No module named builder.recipe
<jtv> StevenK: the bit you said about qa-tagger is what will happen regardless, no?
<StevenK> jtv: Yes, but if was marked incr, the qa-tagger won't slam the bugs to Fix Committed.
<jtv> Ah, now the pieces come together.
<jtv> But I thought you said the rollback should be [incr], not the fixes themselves?
<StevenK> jtv: Right. The rollback will be marked 'as part of this bugs fix', and then when the fixes land properly, the bugs should hit Fix Committed.
<jtv> But you said the 3 bugs should be closed, withing being fixed..?
<StevenK> jtv: No, I said that what was likely to happen due to the lack of incr.
<jtv1> StevenK: you said "if so, the process says the 3 bugs should be closed."  What was the "if so" referring to?
<StevenK> jtv1: I can see we are talking past each other. I explained what would likely happen, and then shifted to talking about what should have happened instead.
<jtv1> Ah, I think I get it now.  Thanks.
<StevenK> Is checkwatches safe to run on qas?
<wgrant> StevenK: Not really, no.
<wgrant> StevenK: And it wouldn't be a very useful test anyway.
<StevenK> wgrant: Okay. Safer to qa-untestable my checkwatches branch?
<wgrant> I think so.
<StevenK> wgrant: Looking at db-devel versus stable
<poolie> lifeless, hm putting this in with the daemon seems not quite right
<StevenK> wgrant: PQM silenced. Hopefully.
<poolie> i'll put it in python-oops
<wgrant> poolie: Doesn't it belong in oops-datedir-repo?
<wgrant> I didn't think python-oops knew about BSON.
<poolie> it mentions it in the docs but it doesn't use it in the code
<wgrant> It doesn't depend on bson.
<wgrant> That's all in datedir-repo/amqp
<poolie> but it does not seem like you should need the repo code to inspect an oops file
<poolie> i could make a new package
<wgrant> Why not?
<wgrant> python-oops doesn't do serialisation.
<poolie> it seems like overkill for what is basically one line of code
<wgrant> "oops file" is a concept that's only part of datedir-repo.
<poolie> there are two potentially separate aspects
<poolie> serializing as bson
<poolie> and writing into per-date directories
<poolie> you could reasonably have the first without the second
<poolie> indeed if you just download one oops you probably will
<wgrant> Sure, but python-oops deliberately doesn't know about serialisation like that.
<wgrant> That's left to the repository implementations: datedir-repo and amqp.
<poolie> amqp has its own separate serialization?
<wgrant> It's BSON. I believe it uses datedir-repo's BSON serializer.
<poolie> foo
<wgrant> All roads lead to datedir-repo :)
<poolie> python-oops says in the readme it defines a serialization
<poolie> though i suppose it is ambiguous what 'the oops project' means
<poolie> so that's why i just put it in utilities/.
<wgrant> I think python-oops' docs are out of date.
<wgrant> datedir-repo was extracted in r9
<poolie> hm, so
<poolie> i don't know
<poolie> having the format be separate from the serialization seems good
<poolie> having no comment at all about what serialization seems dumb
<poolie> in practice multiple trees assume it is bson
<wgrant> No.
<wgrant> Multiple repository implementations use BSON.
<wgrant> datedir-repo has an option to write out rfc822 as well.
<wgrant> And it will read it perfectly happily.
<wgrant> amqp could be changed to use pickles if you were sufficiently misguided, without affecting datedir-repo.
<poolie> true
<poolie> so there's no reason this should live in one of them rather than the other
<wgrant> Well.
<wgrant> I think it makes sense in datedir-repo.
<wgrant> Since amqp's bson doesn't ever hit the disk as a file.
<wgrant> It's purely encoding as it goes into rabbit, and decoding as it comes out.
<wgrant> (it's then usually handed off to datedir-repo, where it's reencoded and written out into a file)
<poolie> yeah i see
<wgrant> So I think this script belongs in datedir-repo.
<poolie> and if python-oops-tools offered an option to download it, it would get reserialized again there
<wgrant> Possibly.
<wgrant> But maybe not.
<wgrant> I think oops-tools is pretty tied to datedir-repo.
<wgrant> Whereas amqp/datedir-repo/oops are very nicely separated.
<wgrant> They actually have sensible interfaces, and work within them!
<poolie> you know what, i'll just make it separate
<wgrant> I think datedir-repo :) But ok.
<wgrant> StevenK: Did you run that through ec2?
<StevenK> wgrant: Which? The imports branch?
<wgrant> Yes.
<StevenK> Yeah, I did
<wgrant> Hm.
<StevenK> Why?
<wgrant> A naive global format-imports should have broken stuff unless you were very lucky.
<wgrant> Due to lp.codehosting's side-effects.
<wgrant> Although I guess it is alphabetically early.
<StevenK> There were 4 failures on ec2, which I fixed before lp-landing
<wgrant> So it may be OK.
<StevenK> wgrant: Still nervous?
<wgrant> StevenK: Slightly.
<StevenK> wgrant: I can forward you the failure mail if it will ally your concerns.
<poolie> and there are at least two different python modules called 'bson'
<wgrant> poolie: Yes :/
<wgrant> And at least one of them is very buggy.
<wgrant> (the one we use)
<poolie> :)
<poolie> wgrant, lifeless, https://code.launchpad.net/~mbp/python-oops-datedir-repo/bsondump/+merge/86338
<bigjools> good morning
<AutoStatic> Good morning
<danhg> Morning all
<AutoStatic> Some colleagues have asked me if I could set up an in-house Launchpad server so they could use it for their projects. They're probably only going to use the bugtrack, blueprint and repository functionality. I'm wondering though if Launchpad isn't a bit overkill then. What's your advise? I already set up a bugtracker for them (MantisBT), a Wiki for their blueprints and setting up a repo is not much work either.
<StevenK> bigjools: Hai. Will you have a chance to do your QA today?
<bigjools> StevenK: hopefully! I got a bit blindsided yesterday
<StevenK> Yes, that's why I didn't bug you then. :-)
<bigjools> I have a theory about poppy
<StevenK> It is horribly, horribly broken and needs to die?
<bigjools> well you wrote it :)
 * bigjools just hacked on the FTP bit
<StevenK> Better than continuing to use Zope's horrible excuse for an FTP server.
<StevenK> bigjools: What is your theory?
<bigjools> StevenK: the ssh checks connect to the appservers to get the authorisation
<bigjools> when we have FDT, the XMLRPC connection fails
<bigjools> after that, it continues to fail forever until restarted
<bigjools> not sure why, but meh, Twisted
<bigjools> the swap death was caused by someone using a loop to connect
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Firefighting: - | Critical bugtasks: 3*10^2
<jml> anyone developing on precise?
<jml> AutoStatic: I'd recommend *not* running Launchpad locally.
<AutoStatic> jml: Yeah, we figured that out too: https://answers.launchpad.net/launchpad/+faq/920
<jml> AutoStatic: it's pretty huge and the operational cost is non-trivial, even at low scale.
<bigjools> allenap: in the tests in your branch, it's probably worth refactoring the bit that sets properties on objects in a r/w transaction
<jml> AutoStatic: cool.
<allenap> bigjools: Erm, which bit?
<jml> AutoStatic: so, I'm not 100% sure what your question is then :)
<allenap> bigjools: Like in test_handleStatus_OK_sets_build_log?
<bigjools> allenap: line 72/83 of the diff
<bigjools> allenap: I suspect we'll need to do that a lot more in the future
<allenap> bigjools: I don't know what a better way would be. I could instead enter read-only mode in each test individually (via a fixture) I guess.
<bigjools> allenap: I was thinking just a test helper
<bigjools> like setattr
<bigjools> but does the whole transactionny thing
<allenap> bigjools: With the removeSecurityProxy thing too I assume.
<bigjools> allenap: no, the caller can do that
<AutoStatic> jml: Well, I got an instance running locally here and my question was more or less a stepping stone to some other questions
<allenap> bigjools: Okay, I think I have a cool way to do that.
<bigjools> allenap: of course :)  cheers
<AutoStatic> jml: So I'm going to wipe out that local launchpad and convince my colleagues that they should look for something else
<jml> AutoStatic: ok.
<jml> bwahahaha
<jml> Python 2.6
<jml> sorry.
<jml> good luck with that.
<gmb> Argh. My connection drops out for ten minutes and when I get back bigjools has done the review I was doing. It's going to be one of those someone-else-does-all-the-work OCR days, is it?
<gmb> (Also, he did a better job of it)
<gmb> (Which galls)
<allenap> jml: I've had a bash at getting Launchpad built on Precise, but I lost interest (it was late). Seems like the cool kids are using a schroot (which I am) or an LXC.
<allenap> (running Lucid)
<bigjools> gmb: shurely shome mishtake :)
<nigelb> What's the firefighting section about?
<nigelb> (in the topic)
<bigjools> if we're in the middle of an incident
<nigelb> ah. It makes topic. Nice.
 * bigjools just added a million people on G+ and may live to regret it
 * nigelb just searched on G+ for "bigjools"
<nigelb> Dammit.
<allenap> bigjools, gmb: Thank you both for the reviews :)
<bigjools> nae prob
<allenap> bigjools: Fwiw, this is what I did to factor out the things you suggested: http://paste.ubuntu.com/776193/
<bigjools> allenap: not so much as a refactoring as a rewriting :)
<allenap> bigjools: Well, I'm already using it in my next branch, and will probably in the one after that :)
<bigjools> heh
<jml> allenap: I'm not suggesting you should actually make this change now, but it might be more re-usable as a Fixture.
<allenap> bigjools: How do I go about QAing the revert I did? Or do we just say it's fine because it's approximately already on cesium.
<allenap> ?
<allenap> jml: Yeah, you're right. If it causes enough friction I'll change it.
<bigjools> allenap: untestable
<allenap> bigjools: Cool.
<cjwatson> gmb: any further thoughts on my QA suggestions for https://code.launchpad.net/~mvo/launchpad/maintenance-check-precise/+merge/82125 ?
<gmb> cjwatson: No, no further thoughts (sorry, meant to reply the other day but forgot after a reboot). Could you take care of QAing if for me? I'll make sure it lands today or tomorrow.
<cjwatson> modulo holiday, yes I can
<gmb> Excellent, thanks.
<rick_h__> ./topic
<jml> hmm.
<jml> so I have a clean lucid schroot for building packages. Can I somehow leverage that to make an schroot dedicated to hacking on Launchpad?
<cjwatson> you could copy the source directory and add a new entry in /etc/schroot/chroot.d/ for it
<cjwatson> and drop the unioniness
<cjwatson> I use a 'lucid-lp' schroot
<jml> cjwatson: thanks.
<jml> (also, my next laptop will have an SSD)
<jml> hmm. I should probably do something like this for each Canonical-deployed project I work on.
<jml> bzrlib.errors.ConnectionReset: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
<jml> got this trying to fetch bzr-git w/ update-sourcecode
<jml> never mind.
<al-maisan> jml: the /etc/resolv.conf in your chroot might be out of date..?
<rick_h__> gmb: got a sec for review? https://code.launchpad.net/~rharding/launchpad/sort_labels_894744/+merge/86287
<al-maisan> jml: try "sudo cp /etc/resolv.conf <path-to-chroot>/etc/resolv.conf" and see whether that helps
<cjwatson> benji: I noticed that in the three branches of mine you reviewed yesterday, you left an Approved comment but didn't set the MP to Approved; was that deliberate?
<benji> cjwatson: generally the MP initiator sets it to approved, sometimes they might be getting a DB review too or a UI approval
<benji> I set the other one to approved because I was landing it and the machinery won't land unapproved branches.
<cjwatson> oh, I didn't know that, my reviewer's always done it for me before
<cjwatson> probably because I've always explicitly asked for landings :)
<cjwatson> benji: ah, and I can't set the MP to Approved because I'm not in ~launchpad
<cjwatson> benji: any chance of landings for always-index-d-i and sign-installer, then, if you have a chance?  It might be best to leave new-python-apt for a bit as it collides with https://code.launchpad.net/~mvo/launchpad/maintenance-check-precise/+merge/82125 and this way I do the merge rather than making somebody else do it
<benji> heh, well that would make it harder
<benji> cjwatson: sure, I'll start the landing of those in a bit
<cjwatson> great, thank you
<gmb> rick_h__: Sure thing; looking now.
<rick_h__> gmb: ty much
<gmb> rick_h__: Approved.
<rick_h__> gmb: awesome, thanks
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 3*10^2
<benji> I'd appreciate it if some kind soul would review this branch: https://code.launchpad.net/~benji/launchpad/bug-903532/+merge/86426
<benji> if that kind soul has some translations knowledge, it would be even better
<sinzui> benji, I ca take it
<benji> sinzui: cool, thanks
<sinzui> benji, r=me
<benji> sinzui: thanks
<lifeless> gary_poster: I'm around for a bit if you want to talk oopses more
<gary_poster> thanks lifeless on call
<wallyworld> sinzui: jcsackett: can we mumble now?
<sinzui> yes
<wallyworld> sinzui: fucking mumble is doing it's thing again where it consumes all my cpu. i have to reboot
<james_w> anyone want to take a look at https://code.launchpad.net/~james-w/launchpad/bpph-binary-file-urls/+merge/86470 ?
<poolie> o/ james_w
<poolie> hi all
<james_w> hi poolie
<dobey> hey poolie
<huwshimi> On the deployable revisions page it says "Revision 14556 can be deployed: orphaned". Does that mean I can't qa it?
<lifeless> either it has no bug linked, or the bug has been closed already
<lifeless> if its the latter, you can reopen the bug
<huwshimi> lifeless: Will it get picked up by the qa tagger etc. then?
<huwshimi> lifeless: Should it be Fix Committed or will any status other than  Fix Released do?
<poolie> ok now i've played with juju it is annoying me that launchpad doesn't use it
<jelmer> poolie: :)
<poolie> jelmer, i just talked to flacoste about bug 795025
<_mup_> Bug #795025: no way to gracefully disconnect clients and shut down the bzr server <canonical-losa-lp> <hpss> <launchpad> <ssh> <Bazaar:Fix Released by jameinel> <Launchpad itself:Triaged> < https://launchpad.net/bugs/795025 >
<poolie> istm there is a safer way to do it
<poolie> which is to have a signal to tell the processes to just stop listening
<poolie> then we can start a new one
<jelmer> poolie: will that work with haproxy?
<poolie> i think so?
<poolie> haproxy will detect that it's down?
<jelmer> I haven't looked at it, so not exactly sure how it's communication with services works
<jelmer> ah, so... so we shut the existing one down and then when haproxy starts another one that's using the new code?
<poolie> i think it's some combination of: seeing if the port is listening, plus pinging a separate http port that reports on the status
<poolie> more precisely:
<poolie> we tell the existing one "stop accepting connections", and it closes its listening socket
<poolie> and then haproxy notices it's down, i guess
<jelmer> that makes sense
<poolie> and then we start a new instance listening on the same port, which will be running the new code
<poolie> then the old process can either exit by itself when all the connections are done
<poolie> or the sysadmins can kill it if they want
<lifeless> poolie: uhm, thats not sufficient
<poolie> because?
<lifeless> because having the old code running for several weeks will play havoc with things like upgrading xmlrpc verbs
<poolie> ..?
<poolie> like, removing old verbs on internal xmlrpc that the old code uses?
<lifeless> yes, or rearranging things; things that you would normally do a server change, change client, cleanup old code sequence
<lifeless> this depends on being confident that the client is deployed
<poolie> mm
<lifeless> not to mention that we would like to free disk space from old deploys.
<poolie> so to have loose coupling we would want to not have those things required to happen in too short of a time window
<poolie> anyhow, after that time, we can just kill the old procesess
<poolie> the client should cope
<lifeless> right, we can allow a few hours for the old processes to gracefully go away, which is what the current plan aims at
<lifeless> we don't want to interrupt someones 6 hour epic initial push, after all.
<poolie> right
<poolie> so my plan is
<lifeless> we don't want idle heavyweight processes hanging around indefinitely either, which means a way of killing them while idle, which implies the client coping
<poolie> i think we can do this in two steps
<poolie> 1- move new connections on to the new process
<poolie> or rather, accept new connections from the new process
<poolie> 2- boot off existing clients
<poolie> 2 is a bit messy becaues
<poolie> some clients won't cope well
<poolie> and it will take unbounded time to get there
<poolie> and it's just generally more risky
<lifeless> mmm
<lifeless> remember we have some fixed paths on disk for the front-forking IPC calls, and we also have N front-end and N forking services to restart
<lifeless> doing 1 without waiting for 2 is more complex and doesn't really buy us anything
<lifeless> we're still not done-done until 2 has happened
<poolie> so doing only 1 will let us bump codehosting from every fdt deploy
<poolie> that seems highly worthwhile
<lifeless> no, it won't.
<poolie> why?
<lifeless> codehosting isn't in fdt anyhow, its a nodowntime-with-handholding deploy
<lifeless> the handholding is because of 2
<lifeless> solve the handholding problem and it can move to nodowntime
<lifeless> the constraints are that we must be safe to delete the deploy directory after the deploy.
<lifeless> well, there are probably more, but thats the key one I see.
<poolie> what specifically is the problem
<poolie> ok
<lifeless> the problem today is that the nodowntime deploy pauses for hours because we can't interrupt bzr safely, so we wait until there are only a few clients connected then manually check that they are all CI servers and whatnot
<lifeless> and then interrupt them ungracefully
<lifeless> the deploy process is 'upgrade instance 1, upgrade instance 2' - serialised - which gets us no downtime
<lifeless> during the deploy, the symlink for the active tree is updated, and after that we assume we can delete the tree at any point
<lifeless> a few trees are kept around, but when we do multiple deploys in a day, there is no fixed window for when a tree will be deleted
<poolie> sure
<lifeless> we probably need to rejigger a few things, and having a quick-stop-listening step is fine with me as long as we don't set ourselves up for messy failures that we need to ignore / whatever.
<poolie> so the handholding is that
<poolie> they want to delete the tree when the processes using it have finished
<poolie> however, that is always going to take a while, unless we're prepared to just abruptly kill connections
#launchpad-dev 2011-12-21
<poolie> so if we do 1
<poolie> we'll be able to switch to new versions at any point
<poolie> the deployment will be fast
<poolie> but there will be an open issue of garbage collection
<poolie> of two aspects
<poolie> old trees, and old processes
<poolie> with the constraint that the tree shouldn't be deleted until the processes running from it have exited
<poolie> and the possible complication that some processes may never exit
<lifeless> well, I have asked that we have a signal to send those processes telling them to exit after the next verb; other than hostile attacks, they will always exit in that situation
<poolie> right, but this may take a long time
<poolie> and it may break some clients
<poolie> so adding that thing will help with gc of processes
<poolie> but it won't make it fast enough that we want to do it within the deployment loop itself
<lifeless> I know it will break old clients, but folk do upgrade bzr fairly regularly
<poolie> ok
<poolie> well, that exists in bzr now
<poolie> i don't think calling it right away is necessarily a good idea
<lifeless> so, you want to make it easier to rev the LP bzr service
<lifeless> which is good
<lifeless> Right now, its nodowntime but manual request
<poolie> fsvo 'nodowntime'
<lifeless> what do you mean?
<poolie> it drops connections at present?
<poolie> anyhow
<poolie> yes, i want to be able to land changes
<poolie> so...
<lifeless> so you can today
<lifeless> if you want to improve the deploy process, cool. I think you have a better handle on the constraints.
<poolie> but what's running will be uncontrolled vs what's in the tree
<poolie> which is undesirable
<lifeless> I can't parse that
<poolie> the changes won't necessarily run until a restart is requested?
<poolie> so it seems like you're saying this is not actually a blocker to landing new bzr changes?
<poolie> we can do a new deployment, which will cause it to start a new version of bzr, but keep running the current frontend?
<lifeless> a ndt request for codehosting will be slow and a bit manual, but when it completes everything is running the deployed revision
<lifeless> because its slow and manual, its not included in the 'nodowntime' alias.
<lifeless> but if you have a change for the service, its a stock standard deploy request away.
<poolie> ok, so there's no reason to think this bug is a blocker to doing other changes?
<poolie> it's just something that would be nice to do?
<lifeless> its very important to do as part of getting all LP running the same codebase all the time
<lifeless> thats an operational efficiency priority
<poolie> i see it's critical
<poolie> i guess i'm wondering if it is super-critical, or if it blocks anything else
<poolie> and also, what would actually have to be done to close it
<lifeless> It adds friction to the testing of the forking daemon
<lifeless> because it takes so long to deploy a change (e.g. to stop using it)
<lifeless> to close it, we need to have deploys be time bounded and not need admin intervention; they don't have to be super fast, just admin free.
<poolie> that seems to imply the tree will get automatically deleted hours later?
<poolie> hm
<lifeless> Yes, when it reaches the front of the gc queue.
<poolie> is there any kind of technical gc queue at the moment or is that just done by losas?
<lifeless> its the last stage of the deploy scripr
<lifeless> I would be happy if codehosting deploys took 2 hours - unsupervised, totally automatic
<poolie> ok, so the deploy script could pause for a long time waiting for the process to complete
<poolie> as long as it could eg poll something to see when it was done with that tree
<lifeless> ideally shorter, but the key thing is the totally automatic aspect, which we don't have today.
<lifeless> we tried to get it with an haproxy timeout, but that broke bzr (per the not-auto-reconnecting thing), and when we set a higher timeout, we found a set of clients that never go idle.
<lifeless> those are the CI servers.
<lifeless> lots of short verbs
<poolie> ..
<lifeless> thus the two-pronged approach: tell the backend to shutdown as soon as it can, and tell bzr clients to reconnect if the channel goes away between verbs
<poolie> so perhaps this requires no more code changes, only changes to the deployment scripts?
<poolie> 1- tell haproxy to stop using that ssh server
<poolie> 2- send sighup to the bzr processes
<lifeless> has the client side reconnect been SRU'd yet ?
<lifeless> poolie: does sighup tell the bzr process to 'exit as soon as idle' ?
<poolie> 3- start a new ssh server and hook it into haproxy
<lifeless> poolie: we have a signal for the front end already, thats stops it listening
<poolie> lifeless, i can't SRU it until it has been well tested in trunk
<poolie> so there is a chicken and egg
<lifeless> poolie: so 1) is signal the frontend, and already happens. As does 3 - just adding in step 2 is probably enough, when we're happy enough with the disconnect reliability.
<poolie> yes, the bzr side of this bug means the server will drop the connection after the current request
<poolie> ok so all we need is a deployment script change to have it sighup the bzr processes?
<poolie> or, perhaps the ssh server should do that when it is itself told to shut down
<lifeless> yes, thats where I would put it in. I wouldn't want to do that until we think a majority of users will be running disconnect-ready clients though
<lifeless> we could do it on staging
<lifeless> as a way to ease into breaking the chicken-egg thing
<poolie> staging is not actually used
<poolie> so, this is why i saw a way out of letting the backend continue running for a few hours
<poolie> that will let most clients finish their business
<lifeless> so thats the current status quo
<lifeless> if we let them finish, we're not actually learning more about the safety of the code
<poolie> mm
<poolie> so, i don't know
<poolie> i don't want to just leave it stuck here
<poolie> we can do some manual tests against staging but i am not sure that will really give enough assurance
<poolie> in some ways i think the current status quo is a better design
<poolie> to just not disconnect people unless we have to
<poolie> why rely on the reconnect working when you can just avoid the issue
<lifeless> well, this is one of the reasons http has a connection:close header
<lifeless> perhaps we should add the ability to signal that to bzr, to make it more reliable.
<lifeless> That said, we *know* there are clients out there that will stay connected for weeks.
<poolie> right
<poolie> so they're going to either have to upgrade bzrlib, or cope with some disconnections
<lifeless> That is clearly unsustainable, and there is a certain elegance in having one solution.
<lifeless> yes, we intend to communicate with them once the code is in all supported bzr series, and say 'please upgrade, you can use a point release if you want'
<poolie> so, again, a chicken and egg
<lifeless> I have to go, sorry.
<poolie> np
<gary_poster> poolie, hi.  bzr has been doing something a bit weird for me.  I have a LP repo initialized with bzr init-repo --no-trees, and a lightweight checkout of local branches.  For weeks or maybe even months now, (almost?) every time I switch the sandbox to local devel pull from LP's devel I get this: http://pastebin.ubuntu.com/776983/ .  I then do break-lock and update and I'm fine, but it's kind of annoying. This is with
<gary_poster>  bzr 2.4.2. Do you have any ideas or advice?
<poolie> hi, please file a bug, with a traceback and reproduction instructions, and try 2.5
<poolie> i don't recall an existing bug about that
<gary_poster> poolie ack will do.  thanks
<gary_poster> (will do tomorrow
<gary_poster> ) :-)
<gary_poster> bye
<poolie> thanks
<lifeless> poolie: back
<poolie> lifeless, i don't want to ruin your vacation
<wallyworld_> what's the preferred style for multi-line comments? # or '''
<poolie> for a comment, definitely #
<wallyworld_> ok, thanks. makes it less readbale though
<wallyworld_> i saw that guido approves of ''' :-)
<lifeless> guido also is quoted as saying 'what was I thinking' (about other things, but shows his approval is no  guarantee...
<poolie> wallyworld_, i guess you know a ''' is not a comment?
<poolie> it's a string
<wallyworld_> poolie: yes
<poolie> so apart from the specific use of docstrings, putting comments in them feels a bit weird
<wallyworld_> but it serves the purpose :-)
<wallyworld_> stub: hi there. if you had a moment, now or later, i'd love to get a 2nd opinion on some sql. i think it will be efficient but i'd like to check
<stub> wallyworld_: sure
<wallyworld_> stub: thanks. here's something to look at. i can explain whatever is unclear. https://pastebin.canonical.com/57479/
<wallyworld_> stub: the code runs in the security adaptor
<wgrant> Oh god.
<wallyworld_> stub:  the sql is obstinately correct because the unit tests pass (the ones written so far). it's the performance aspects i'm interested in
<wallyworld_> wgrant: what's wrong?
<wgrant> This is crazy :)
<wallyworld_> what is?
<wgrant> Let's not put an expensive global bug search into person traversals..
<wallyworld_> wgrant: it's only used in the api - for views, it's pre-cached
<stub> I'm tending to agree with that, but looking at the SQL at the moment
<wgrant> wallyworld_: Or for anything under a person.
<wgrant> eg. a PPA, a branch.
<stub> What is 'user_bug_filter'?
<wgrant> Having pillar-wide visibility would render this obsolete.
<wgrant> Which means we won't be causing more timeouts.
<wallyworld_> stub: that's an existing filter implemented in BugTask used to return the bugtasks a user can see
<wgrant> Which would be a good thing.
<stub> wallyworld_: Have an example?
<wallyworld_> stub: https://pastebin.canonical.com/57480/
<wgrant> It probably won't be *that* bad most of the time, because of the lack of other constraints.
<wgrant> However.
<wgrant> It is a slippery and unnecessary slope, and will be slow in some situation.
<wgrant> +s
<wallyworld_> wgrant: that's the idea, and remember, it's only for when the API is used
<wgrant> No.
<wgrant> It's for any traversal through a person.
<wgrant> And "only for when the API is used" is particularly bad.
<wallyworld_> for the common use case, where someone opens a bugtask view, we pre-cache
<wgrant> Because this adds probably >1s per request.
<wgrant> And API makes lots of requests.
<wallyworld_> another common case is when someone navigates to a private team page
<wallyworld_> ah, but yes, any traversal through a private will trigger it
<lifeless> wallyworld_: whats this for ?
<wallyworld_> lifeless: it's in the security adaptor for limitedView permission on private teams
<lifeless> wallyworld_: ah, person visibility ?
<lifeless> wallyworld_: cool
<lifeless> you don't need the bug its, just exists
<lifeless> so you can tune this to be more simple still
<lifeless> wallyworld_: this only runs when the user doesn't have direct access to the team right ? (non-admin, non-member)
<wallyworld_> lifeless: yes, these checks occur last in the security adaptor
<wallyworld_> the least expensive ones are run first
<lifeless> cool
<lifeless> and \o/
<lifeless> wgrant: this doesn't add 1s per request, it will add (amount to be checked) per request *by someone that cannot see the team today*
<wallyworld_> lifeless: this will allow someone to click through a private team in the bug subscription portlet and see the limited details of the team for example
<wgrant> And it scales for every single reference ever.
<wgrant> It's silly and unnecessary.
<wgrant> Overcomplicated, confusing, and slow.
<lifeless> wgrant: linear probing increase on the query planner, stats-based linear search of log-time checks from there
<wgrant> lifeless: + 100 other queries for the other person foreign keys.
<stub> That clause is certainly a killer. Materializing a list of all public bug ids is a sequential scan of the bug table
<wallyworld_> it's not confusing to me, or complicated. slow perhaps, which is why i've asked for advice
<wgrant> wallyworld_: The query isn't.
<wgrant> The concept is.
<wallyworld_> not to me
<wallyworld_> it's what we are basing social private teams on
<lifeless> wgrant: not all fk's will lead to disclosure (libraryfilealias owner for instance, or message owner)
<wallyworld_> stub: you mean the bug filter clause i pasted second
<wallyworld_> ?
<wgrant> lifeless: No, but most have to, in the proposed model.
<lifeless> many will yes
<wgrant> So, we are going to be in a nice situation where you can see a private team when it's linked to something, except when you can't.
<lifeless> we may want to build a fast data structure for it in time, but this is a reasonable approximation, especially given that its an uncommon-path anyway.
<wgrant> And it's impossible to say who can see a private team.
<stub> wallyworld_: Yes. The SQL you have will certainly need optimizing for use in a security adapter. Technically, it could be called an unlimited number of times on a page (once per private team)
<wgrant> It's exactly the opposite of the disclosure project.
<wallyworld_> lifeless: wgrant: but isn't the issue here that even if a private team is on a fk, the logged in user may not be able to see that object, so the team's existent should not be shown in that case
<stub> Realistically, once or maybe twice
<wgrant> It's making disclosure of private teams entirely opaque.
<lifeless> wgrant: its making it *possible* to use them sensibly
<wallyworld_> stub: what we do now, for is pre-cache the permissions for pages which require this, so the security adaptor is not called
<wgrant> lifeless: There are simpler ways.
<lifeless> wgrant: such as?
<wgrant> lifeless: Pillar-wide scope.
<wgrant> To subscribe a private team to a bug, it has to be able to be disclosed to that project.
<wgrant> So visibility is simply based on whether you can see a pillar in which the private team is involved.
<wgrant> It makes it easy to see what is visible where.
<wgrant> And it makes it difficult to make mistkaes.
<wallyworld_> stub: for example, on the bug page, the subscribers portlet which may have many private teams listed, all the permissions are cached
<wgrant> And it is cheap to query, and very understandable.
<wgrant> (it doesn't make sense until we have private projects, however)
<lifeless> and it doesn't address things like ayatana
<wgrant> Oh?
<lifeless> private list of folk working on the project, public bugs, do public triage.
<lifeless> as a for instance.
<wgrant> The team membership is still private.
<lifeless> right
<wgrant> Visible only to members/admins/etc.
<wallyworld_> does our model even support the simple determination of which pillars a private team is involved with yet? we would need something like the TeamParticipation table, no?
<wgrant> But the existence of the team is not.
<wgrant> LimitedView is existence, not membership.
<lifeless> but hten the next team over, which is totally private, but has access to private ayatana things, what is their safeguard for disclosure ?
<wgrant> I posit that LimitedView should be granted pillar-wide.
<wgrant> Huh?
<wgrant> Confused.
<lifeless> lets talk in budapest, what wallyworld_ is doing is what we agreed made sense on the call a week or two back
<wgrant> My plan was to talk in Budapest; I didn't expect this particular case to arise so quickly.
<lifeless> or we can take this offline now and beat it to death, but I don't want my holiday to die a death of a thousand cuts
<stub> What we have is surprisingly quick in any case for the example personid (243656)
<wallyworld_> stub: is person 243656 some random person?
<stub> http://paste.ubuntu.com/777089/
<stub> wallyworld_: It was the id from your second pastebin :)
<wgrant> 243656 is a dev person
<wallyworld_> stub: ah, ok. that was from a unit test
<lifeless> stub: try with me or you
<stub> lets try it with a user that is a member of some teams then
<wallyworld_> or a user that can see a bug that a team is subscribed to but is not a member of the team
<stub> 1.2 seconds for me
<lifeless> 7.5 seconds for me
<stub> So lets see if we can optimize this.
<wallyworld_> stub: lifeless also mentioned the bug_id bits could be optimised out
<stub> yer
<lifeless> http://paste.ubuntu.com/777092/
<lifeless> oh, I fail ad adjusting
<lifeless> wallyworld_: hang on, there are two persons involved
<wallyworld_> yes, the private team we are checking the visbility of and the person who wants to see
<lifeless> there is the person being *viewed* and the person *viewing*
<lifeless> stubs test only has one id
<lifeless> the pathological case is a private team that the person doesn't know about
<wallyworld_> ah, didn;t notice that
<lifeless> the bug visiiblity code is 'which bugs can this person see'
<lifeless> but we want 'are there mutually visible bugs'
<lifeless> I think you'll need to start from ground up
<wallyworld_> yes, the team being check has it's id in the TEamParticiaption query near the top
<lifeless> yes, but thats still wrong
<lifeless> there are *two* team expansions to do
 * lifeless stops to think
<wallyworld_> lifeless: the query checks directly subscribed to and assignee
<wallyworld_> i think the query is correct
<lifeless> the private team is visible if:
<lifeless>  - it has any interaction with a truely public object
<lifeless>  - there is any interaction with a visible object
<lifeless> so the one half should be 'users visible objects' the other half 'interactions with objects'
<wallyworld_> the current implementation is for a subset of such rules, as per the comments in the python in the pastebin
<wallyworld_> just the things we can do quickly
<lifeless> and we can short circuit a bunch of that by saying 'mutual ineractions'
<lifeless> wallyworld_: yes, I saw that
<lifeless> so the blueprint check is one sided because they are only public
<wallyworld_> i included that check to hook into the team participation query
<lifeless> the select bug_id. .. would probably be better phrased as two separate things
<lifeless> a) bug subscription or assignee to public bugs
<lifeless> b) subscription or assignee to a mutually visible private bug
<wallyworld_> b) it not quite right i don't think
<stub> Better https://pastebin.canonical.com/57481/
<wallyworld_> just because a private team can see a private bug doesn't mean i can see that private team
 * wallyworld_ looks
<lifeless> wallyworld_: I know - there are two conditions: interact * and * mutual visibile
<wallyworld_> lifeless: not sure i agree with mutually visible
<lifeless> wallyworld_: so, if the private team interacts with something I cannot see, I cannot see the team, right ?
<wallyworld_> lifeless: yes
<lifeless> thats why mutually visible
<wallyworld_> i think i misunderstood you
<lifeless> if the private team does not interact with it, I can't see the team.
<stub> I believe in an exhaustive test suite and the query is right when it passes and is fast enough :)
<lifeless> And if I can't see the bug I can't see the team
<lifeless> and they cannot interact with something they cannot see
<wallyworld_> that 2nd point is irrelevant if we query for the things the team inetracts with, no?
<wallyworld_> since the result set will by definition contains the things they can see
<lifeless> depends on wgrants implementation plans for folk that are kicked out of a project
<wgrant> That's one bit I hadn't worked out yet.
<lifeless> anyhow, 'visible to user + interacted with by the team' is sufficient for now
<wallyworld_> stub: so, if i redo it based on your query, how do we want to ensure it's fast enough in practice?
<lifeless> wallyworld_: a doubly nested for loop on staging
<lifeless> for person in person for team in teams
<lifeless> stub: is that query rephrased at all, or just new constants plugged in ?
<stub> it is rephrased somewhat
<stub> https://pastebin.canonical.com/57482/ is even more reworked
<wallyworld_> lifeless: so you'd prefer this to land behind a feature flag? so we can test on staging and leave the flag off if we are not happy?
<stub> Just mild tweaks though
<lifeless> wallyworld_: always ;)
<wallyworld_> ok
<lifeless> wallyworld_: safety nets make radical change easier
<lifeless> wallyworld_: this is radical
<stub> So where we want to know 'EXISTS', don't generate more results than absolutely necessary
<wallyworld_> stub: so if i land this, and it's up on qas, can you help me test it for performance before fflag is activated?
<stub> pg will shortcut a simple query in an EXISTS, but I don't know if it is smart enough for a UNION query so I optimized those bits. Also, UNION ALL will be faster than UNION, again PG might have optimized that for us but better safe than sorry
<lifeless> stub: cool, I was just checking with me an dwanted to know whether to fiddle with c&p from your query
<stub> wallyworld_: If it is up on qas, you don't need me - you can get query times for a page request directly
<lifeless> wallyworld_: I'm pretty sure you will want to unwrap the user_bug_visibility filter and partition the checks
<lifeless> wallyworld_: IMBW but splitting on unbalanced data often makes pg happier
<wallyworld_> lifeless: not sure i understand what you mean by partition the checks
<stub> My updated query will involve changing the user_bug_visibility in any case - changing the single EXISTS to two EXISTS
<stub> oic
<wallyworld_> stub: with the testing question, i will and can do my own tests. lifeless mentioned the use of a double for loop, hence my question about additional tests
<lifeless> wallyworld_: rather than checking 'all interacted with' against 'all visible'
<stub> he means 'WHERE Bug.id IN (SELECT id FROM Bug WHERE private IS FALSE) OR Bug.id IN (SELECT id FROM Bug WHERE private IS TRUE AND (...)'
<lifeless> wallyworld_: do two checks with - 'interacted with public' || 'interacted with private-visible'
<wallyworld_> ah, ok
<lifeless> note that one half of that is dramatically cheaper
<wallyworld_> and it may short circuit the check?
<lifeless> don't need to know what the viewer did if there are any public interactions
<wallyworld_> sorry if i've stirred up a hornet's nest with this. i really appreciate your and stub's help
<lifeless> may want a CTE of the bug subscriptions, may not.
<wallyworld_> CTE?
<lifeless> common table expression - the WITH stuff
<wallyworld_> ah, ok
<huwshimi> hmmm... make run is failing with "mkdir: cannot create directory `/var/tmp/bazaar.launchpad.dev/mirrors': Permission denied"
<lifeless> anyhow, stub has a fast query now.
<stub> it still needs work
<lifeless> remember that disclosure brings in a new model for 'can view'
<stub> Worth getting it as fast as possible as there will be outliers that cause grief
<lifeless> so there are diminishing returns in optimising this today: a) its used in the may-fail-anyway case, and b) the underpinnings are going to be reworked soon
<lifeless> stub: sure, not meaning you shoul stop
<lifeless> just be aware that we'll need to do it all again in 3 weeks or so :)
<stub> yer
<stub> just looking at my last query plan on how much was never executed - short circuits are great, but they won't always happen
<wallyworld_> huwshimi: sudo remove that dir should fix it i think
<huwshimi> wallyworld_: Thanks, I'll give it a go
<huwshimi> wallyworld_: It doesn't exist
<wallyworld_> lifeless: stub: doing this now fixes the current 404 on prod when users try and click through to private teams, plus picks up the work curtis just did this week on the page templating to take account of limitedView permissions
<huwshimi> wallyworld_: Oh, the parent does, I'll remove that
<wallyworld_> huwshimi: remove the parent
<wallyworld_> beat me to it :-)
<stub> I haven't been following the frequent and lengthy discussions so am trusting a sane rationale for all this was produced :)
<huwshimi> wallyworld_: Perfect, thankyou
<wallyworld_> huwshimi: np. it's apache that writes it out as root i think
<huwshimi> silly apache
<wallyworld_> stub: lifeless and curtis agree on it all. it's a step along the way to a better solution and a fairly radical change to allow some private team details to be known. stakeholders seem pleased that we are doing it
<stub> wallyworld_: So if a team has subscribed to a public bug, or subscribed to a private bug that has been made public, that team is always visible?
<huwshimi> A simple CSS review: https://code.launchpad.net/~huwshimi/launchpad/tag-link-colour-907132/+merge/86499
<wallyworld_> stub: yes. if a user can see a bug that a team is subscribed to (it may be a private bug), that team has limitedView visibility for the user
<wgrant> Note that LimitedView doesn't provide access to the member listing.
<wallyworld_> just the name, displayname etc
<stub> Yes, and if that is a public bug then it is visible to everybody
<wallyworld_> stub: yes
<wallyworld_> since the team has placed itself in a public role
<stub> Or someone twiddled the privacy setting on a bug they are subscribed to :)
<stub> Not necessarily a member of the team
<wallyworld_> stub: and we are implementing code to warn the user who is placing the team in such a role when this happens
<wgrant> Exactly.
<wgrant> This is why it gets very messy.
<wgrant> Anyone can disclose the team.
<wallyworld_> huwshimi: with the new screenshot, it looks like "None" is lighter than the other tags?
<huwshimi> wallyworld_: Yeah, it's not a link
<wallyworld_> huwshimi: my (perhaps invalid) opinion is that the label should be different from the text content
<wallyworld_> colour
<wallyworld_> sure, links need to be different as well
<huwshimi> wallyworld_: I think that's valid, but as you say the links should be different
<huwshimi> wallyworld_: I guess this branch is just about link colours
<wallyworld_> yes. and i don;t like the grey personally
<wgrant> Should the tag links perhaps look like tags?
<wgrant> like status/importance do
<wallyworld_> grey to me means disabled
<stub> wallyworld_: While you are there, can you quickly twiddle the code that generates https://pastebin.canonical.com/57480/ to say 'IS FALSE' rather than '= FALSE' to give us maximum chance of being optimized.
<huwshimi> wgrant: I think we should do something with our tags, but if we do we need to do it everywhere
<wallyworld_> stub: sure
<stub> wallyworld_: And the UNION to a UNION ALL
<wgrant> huwshimi: I agree, but we changed status and importance in only one place...
<wallyworld_> stub: this will be in the existing bug filter code. so i'll fiddle the sql and let the existing tests validate correctness
<stub> wallyworld_: Yes. Doing my suggestion will be a win for everything.
<stub> wallyworld_: I'd like to change it to two EXISTS rather than one using UNION ALL, but that might have fallout.
<wallyworld_> stub: you mean performance regressions?
<wallyworld_> for existing bug queries
<stub> wallyworld_: The changes I suggested will not cause regressions. The change I'd like to do might, so I won't get you to make a quick hack right now :)
<stub> (just do the IS FALSE and UNION ALL)
<huwshimi> wgrant: Yeah, we need to fix that
<wgrant> huwshimi: They look all old and crap everywhere else :(
<wallyworld_> stub: ok. i'll also split the other query as suggested earlier, and add you as a reviewer for the mp to check it?
<stub> Sure
<wallyworld_> thanks :-) i may not get it done tonight, have other stuff on
<wallyworld_> gotta finish cobverting the existing doc test to unit tests
<wallyworld_> huwshimi: so were you going to dick with the colours a bit?
<huwshimi> wallyworld_: Well there are outstanding issues that will affect what other changes would be made. E.g. bug 894734
<_mup_> Bug #894734: "Assignee: None" and "Tags: None" clutter bug listings <bug-columns> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894734 >
<huwshimi> wallyworld_: And all the other bugs about content not being linked
<wallyworld_> huwshimi: and you are sure about the yucky grey?
<huwshimi> wallyworld_: Well we could look at having a saturated version of the blue
<wallyworld_> huwshimi: my opinion is that that would be better, since grey to me signifies disabled, and is used elsewhere in lp to mean that (eg bug abd branch auto linking in text content)
<stub> wallyworld_: https://pastebin.canonical.com/57483/ is what I end up with anyway. I could split up the bug check into 'subscribed to public bugs', 'subscribed to visible bugs' etc. but that would involve bypassing user_bug_filter. I thought this would be bad, as that stanza will be evolving as bug privacy evolves.
<wallyworld_> stub: yes, that's why i wanted to use that method to get the user bug filter
<wallyworld_> stub: not looking too closely, that query took 0.5, the last one took 0.3.
<huwshimi> wallyworld_: I'm about to finish up for the day (and wander over the road for the Big Bash), but I'll have a play and probably update things tomorrow.
<stub> wallyworld_: That was using incorrect parameters I think
<stub> oh no
<stub> yer - https://pastebin.canonical.com/57482/ does seem quicker, both in the cost estimate and the actual query time
<wallyworld_> huwshimi: ok. i wnt to the big bash last night. saw warnie and liz. bloody excellent. have fun
<wallyworld_> stub: ok. i'll go with the quicker one for now. i have to run up and make dinner - my wife came home from a procedure at hospital today. but i'll check irc again after dinner
<wallyworld_> thanks so much for your help
<huwshimi> wallyworld_: Yeah, should be good
<stub> wallyworld_: np. I'm done anyway - need to go play with replication
<jtv1> Any reviewers in the house?  https://code.launchpad.net/~jtv/launchpad/lint-lpserve/+merge/86498
<stub> jtv1: done
<jtv1> thanks
<gmb> Does anyone have any idea why I'd get his error http://pastebin.ubuntu.com/777225/ whilst running cronscripts/scan_branches.py against a local codehosting instance?
<gmb> Ah, hang on.
<gmb> Because I'm an ijit, that's way.
<wgrant> gmb: was /var/tmp/bazaar.launchpad.dev/mirrors 700 or so?
<wgrant> That's been happening for me recently.
<gmb> wgrant, Hmm, lemme check. I'm using dev.launchpad.net/Running/RemoteAccess and the Apache config for bazaar-internal looks a bit weird...
<gmb> No, /mirrors is rw-rw-r--
<wgrant> /var/tmp/bazaar.launchpad.dev is sensible too?
<wgrant> Apache likes to create them as root with a restricted umask :/
<gmb> Yep, that's sane.
<wgrant> Huh
<wgrant> I guess the apache error log is the next step.
<gmb> Yep.
<gmb> Haha
<gmb> [client 172.16.15.154] client denied by server configuration: /var/tmp/bazaar.launchpad.dev/mirrors/00/00/00/4d/.bzr/branch-format
<gmb> The clue's in the brackets.
<gmb> The RemoteAccess document needs a rewrite :)
<wgrant> Ah
<wgrant> I keep my container's /etc/hosts pointing at 127.0.0.88
<gmb> Wise.
<wgrant> Only the host's gets the real IP.
<gmb> As it should be.
<gmb> I'll update the page.
<wgrant> Great.
<gmb> Argh. Can anyone remember francis's non-netstat way of finding out what's hogging a port?
<StevenK> fuser?
<StevenK> fuser <port number>/tcp
<StevenK> Or lsof
<gmb> lsof!
<gmb> That's it.
<gmb> Thanks.
<gmb> Twisted: I hate you, die in a fire.
<StevenK> gmb: I tend to reach for fuser first, then netstat, then lsof. But whatever works.
<bigjools> my third successive day of doing maintenance rotation. FML
<StevenK> bigjools: I shall think of you while I play Half-Life.
<bigjools> gmb: Twisted is good, except its logging
<gmb> StevenK, Ah, cool. I went netstat->lsof, but I'll try your ordering next time.
<bigjools> StevenK: you still play Half Life?
<StevenK> bigjools: I haven't played it before. It hit 75% off on Steam today, so I grabbed all of it.
<bigjools> StevenK: it's *awesome* :)
<bigjools> I wasted my life on it 10+ years ago
<StevenK> Haha
<bigjools> if you go online with it, prepare to end yours too
<StevenK> Now for the Quake Collection to hit 75% off. Which, amusingly enough, doesn't include Quake 4.
<StevenK> bigjools: The first Portal is 50% off, so it's probably only a few quid.
<bigjools> StevenK: cool, I might get it on the PS3
<stub> $3.75 vs ???
<StevenK> stub: It's $10 normally
<stub> It was on special. $10 I think got you both 1+2
<stub> yesterday I think, maybe still is.
<stub> I'm backlogged about a year with games, so no need to pay full price :)
<StevenK> Haha
<jml> Hello, we'd appreciate a review of https://code.launchpad.net/~james-w/launchpad/bpph-binary-file-urls/+merge/86470
 * bigjools will look
<jml> bigjools: thanks.
<jml> gmb: the great thing about Twisted is that it holds your puny hatred in contempt, dwarfed as it is by its own vast loathing.
<jml> gmb: try getting that with Django!
<bigjools> jml, james_w: needs fixing, but minor stuff
 * gmb wonders if he can fix bug 808930 buy just adding another commit()....
<_mup_> Bug #808930: Timeout running branch scanner job <escalated> <linaro> <oops> <Launchpad itself:In Progress by gmb> < https://launchpad.net/bugs/808930 >
<gmb> (The answer is "probably." Whether it's the _right_ fix or not is debatable)
<gmb> "Buy" just adding another commit? Seriously Graham, _that's_ the way you're spelling it?
<wgrant> gmb: Howso?
<gmb> wgrant, Well, first, s/probably/possibly. We're suffering death by SQL (in at least one case) with a big for loop that holds the txn open whilst it adds new revisions.
<gmb> Now, adding a commit at the end of that loop may obviate the problem.
<gmb> But as a solution, it stinks.
<wgrant> Is this creating new Revisions?
<wgrant> Or BranchRevisions?
<gmb> This is new Revisions.
<wgrant> It doesn't do a massive manually constructed INSERT?
<gmb> Nope.
<wgrant> Huh.
<gmb> Lots of revision = Revision(...)
<wgrant> How did that ever work :/
<wgrant> Awesome!
<gmb> wgrant, There's a lovely comment to the effect of "this is bad, but, eh, it works."
<wgrant> So, this is the issue I thought you'd hit first.
<wgrant> Didn't think it was quite *that* bad, though.
<gmb> :)
<wgrant> There's already a mass-insertion method for branchrevision.
<wgrant> And binarypackagepublishinghistory
<gmb> Ah, cool.
<gmb> I shall crib appropriately.
<wgrant> And possibly bugnotificationrecipient and co.
<wgrant> It's ugly, but it works.
<gmb> Hey, I'm not pretty, but I do okay.
<gmb> "Ugly but works" works for me.
<wgrant> You may well need a looptuner, since inserting 200000 revisions in a single statement might not be too helpful, but we'll see...
<gmb> LoopTuner was one of the things I was thinking of.
<rick_h__> morning
<cjwatson> I'll get my pending QA done at some point over the next couple of days, BTW - currently on holiday but should be able to squeeze it in to unblock people
<cjwatson> mind you I'm not sure if anyone's planning another ftpmaster deployment this year :-)
<nigelb> ok, I have to admit, githubs "Merge" button is really really nice for code review.
<nigelb> I know that tarmac does something equaivalent for us, but I'd really really like to see something like that on LP :-)
<rick_h__> nigelb: :)
<rick_h__> nigelb: subsume reviewboard!
<nigelb> heh
<rick_h__> even though I'm not a django fan
<nigelb> rick_h__: The other day I probably complained a litle too hard about sqlalchmeny and michael bates replied to me :P
<rick_h__> beyer?
<nigelb> bah, beyer
<rick_h__> nigelb: he's really awesome at being open/communicating
<rick_h__> and a super genius
<rick_h__> but cool, he help you out then?
<nigelb> Well, eventually, it wasn't sqlalchemy that was giving me either
<nigelb> Something else threw an sqlalechemy error :)
<rick_h__> orly?
<rick_h__> now I'm curious, what was throwing an SA exception at you?
<nigelb> Yeah. Flask-Admin.
<rick_h__> ah, gotcha
<rick_h__> yea, I've not used flask other htan hello world to kind of look at it
<nigelb> I'm coding exclusively in Flask for work.
<nigelb> Its harder, but its much more flexible than a full framework.
<rick_h__> yea, I liked what I saw
<rick_h__> and if I had a small app I'd use it
<nigelb> My boss wrote something like login.launchpad.net in Flask, except its an OAuth provider and not OpenID.
<rick_h__> cool, check out https://github.com/bbangert/velruse sometime
<rick_h__> another "to check out" sometime item
 * nigelb looks
<nigelb> Interesting!
<rick_h__> yea, I really love the idea
<rick_h__> I was pitching it at my last job as a way to centralize auth through our apps
<nigelb> we have a centralized auth through our app. And have a client lib for our apps :-)
<rick_h__> cool
<rick_h__> yea, we had a ton of small little apps and used ldap internally, but for external users thought it might make sense.
<rick_h__> we'd have needed to find a way to hack an ldap provider into there somehow, but would have been cool
<nigelb> that's the one thing I like about almost all Canonical apps. Everything uses openID.
<rick_h__> right
<rick_h__> took a little bit, but thank goodness
<StevenK> nigelb: How much data?
<nigelb> StevenK: Every single person row.
<StevenK> Argh
<nigelb> :D
<StevenK> Yes, write a garbo job.
<StevenK> nigelb: lib/lp/scripts/garbo.py. Create a new class with a descriptive name, copy method *names* from the other classes.
<StevenK> nigelb: What the methods return should be self-explanatory. If you want more help from me, you'll have to wait until it's not 0030.
<nigelb> StevenK: Well, I originally, wanted to poke you tomorrow and work along asking you questions.
<StevenK> I suppose you can do that.
<nigelb> I'll make some time before I start work to finish it off.
<nigelb> I haven't worked on LP in a while :)
<james_w> bigjools, how do I know want to compare the list of urls to?
<bigjools> james_w: the stuff you created in the factory
<bigjools> make a list out of it and compare to the listified results
<james_w> bigjools, do the same url generation against them?
<bigjools> james_w: well I'd construct an expected URL
<james_w> bigjools, using ProxiedLibraryFileAlias?
<bigjools> yup
<james_w> ok
<bigjools> james_w: or even just file.htt-purl
<bigjools> err
<bigjools> file.http_url
<james_w> except that won't match as it's not proxied?
<bigjools> there's nothing to proxy
<bigjools> unless it's private
<james_w> bigjools,
<james_w> AssertionError: !=:
<james_w> reference = ['http://localhost:41286/93/filename-100190']
<james_w> actual = [u'http://launchpad.dev/~person-name-100146/+archive/unique-from-factory-py-line2685-100148/+files/filename-100190']
<james_w> : ['http://localhost:41286/93/filename-100190'] != [u'http://launchpad.dev/~person-name-100146/+archive/unique-from-factory-py-line2685-100148/+files/filename-100190']
<james_w> so it looks like they are proxied?
<james_w> archive = self.factory.makeArchive(private=False)
<bigjools> james_w: meh, my bad then.
<bigjools> the redirect happens in the appserver
<rick_h__> deryck: can you review this please? https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376/+merge/86579
<james_w> bigjools, https://code.launchpad.net/~james-w/launchpad/bpph-binary-file-urls/+merge/86470
<bigjools> checking
<bigjools> james_w: perfect, thanks. Do you need me to land it? I can't recall if you can do that
<james_w> bigjools, I had big problems doing so last time, so if you could throw it at ec2 it would be much appreciated
<bigjools> sure thing
<bigjools> james_w: can you set a commit message
<james_w> thanks
<james_w> oh yeah
<james_w> done
<rick_h__> deryck: ping, have a sec to peek at https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376/+merge/86579
<deryck> rick_h__, sure
<rick_h__> deryck: ty
<deryck> rick_h__, so it feels like the kind of thing we should test since we keep breaking because of it.
<deryck> rick_h__, but it seems hard to test, too.
<rick_h__> deryck: right, because you have to find/replace all cases whereever you're catching the history event
<rick_h__> the only *good* way would be to wrap history event access entirely with our own api
<rick_h__> deryck: I searched before, but missed this case because it was in the .pt and not in the .js
<rick_h__> deryck: I'd definitely prefer to have this stuff in a .js file and the .pt only do a single "setup()" kind of call, but didn't want to change this that much
<deryck> rick_h__, yeah.  so let's do this change as it is now.  and can you file a high bug about needing to wrap history handling to prevent this happening again?
<rick_h__> deryck: sure thing
<deryck> rick_h__, awsome, thanks!
<deryck> gary_poster, so about deployâ¦. I can hold for a bit to see if you guys can qa benji's work, if that's cool.
<deryck> gary_poster, or were you thinking this is something to wait for tomorrow on?
<gary_poster> benji, see ops:-/
<gary_poster> I mean deryck, see ops :-/
<deryck> gary_poster, gotchas, I see now.  thanks.
<cjwatson> Does anyone mind if I set up /srv/launchpad.net/ubuntu-archive/gnupg-home/ on mawson so that I can QA https://code.launchpad.net/~cjwatson/launchpad/sign-installer/+merge/85670 ?  Much like the setup on production only with a dogfood key
<cjwatson> y'know what, I'm pretty certain nobody cares so I'll JFDI and we can remove it later if it breaks somebody
 * cjwatson wonders what "The calling script may set GNUPGHOME ..." is supposed to mean in cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/10-sign-releases given that it proceeds to unconditionally override that variable
<cjwatson> maybe the right answer is ln -s $HOME/.gnupg /srv/launchpad.net/ubuntu-archive/gnupg-home
<cjwatson> any objections to upgrading mawson to >= r14564?
<cjwatson> OK, nobody else logged in, so upgrading mawson
<poolie> hi all
#launchpad-dev 2011-12-22
<wallyworld_> huwshimi: you 100% sure we can't use another colour besides grey for the links? since we use grey for disabled links elsewhere, users will be confused and assume these links are also disabled
<huwshimi> wallyworld_: Where do we use grey for disabled links?
<huwshimi> we use grey for our breadcrumb links
<wallyworld_> huwshimi: one example, if one enters text for a comment comment or elsewhere, and the text contains "bug xxx", the "bug xxx" is linked and if it is not a valid link, is rendered grey
<wallyworld_> s/comment comment/bug comment
<wallyworld_> the breadcrumbs are sort of self contained, not mixed in with general content
<wallyworld_> but if there's no good alternative.....
<huwshimi> hmmm... I can't reproduce the bug comment links in .dev
<wallyworld_> a very light blue perhaps? like the tags that are rendered in the bug details page?
<wallyworld_> ah, ok. you mention light blue looks like disabled
<wallyworld_> huwshimi: i'll find an example on qas
<huwshimi> wallyworld_: cheers :)
<wallyworld_> it is a light shade of grey btw
<wallyworld_> huwshimi: https://bugs.qastaging.launchpad.net/ubuntu/+source/aptdaemon/+bug/696954
<_mup_> Bug #696954: Maintainers cannot specify  who is in an access policy <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/696954 >
<wallyworld_> huwshimi: i thought i entered a valid bug for one example, but it is also invalid
<wallyworld_> huwshimi: the valid links are the same shade of blue as the other links
<wallyworld_> huwshimi: brb
<huwshimi> np
<huwshimi> Oh I see they only become grey links after a page refresh
<wallyworld_> huwshimi: yes, there's a background call which does the validity checking. the fact that it doesn't run when a comment is first entered is a bug
<huwshimi> wallyworld_: Ah I see
<wallyworld_> which i only just realised happens
 * wallyworld_ files a bug
<wallyworld_> huwshimi: if i recall correctly, there's a css style for disabled links which is used
<wallyworld_> and the style specifies that shade of grey
<huwshimi> wallyworld_: I'm inclined to land this with the links this colour and wait to see if there's feedback.
<wallyworld_> huwshimi: ok. sorry to be a pain rtc. i wanted to be sure we did want to go with those colours given the potential for confusion
<huwshimi> wallyworld_: No that's ok. I understand where you're coming from, but I think this might be the best option for now
<wallyworld_> huwshimi: r=me :-)
<huwshimi> wallyworld_: Cheers mate
<wallyworld_> np. how was the cricket?
<poolie> huwshimi, wallyworld_, re grey links, fwiw
<wallyworld_> hi poolie
<poolie> 1- other people use them and it's fine; i don't think there is a general style that they have to be blue or underlined anymore (though generally a hover of some kind is good)
<poolie> 2- ideally i think we would be steering towards a new general style (i guess you are) rather than doing it one by one
<huwshimi> wallyworld_: It was great fun, nice to have a second win too,
<wallyworld_> poolie: point 2 holds a bit of sway with me. i personally would like to see a consistent colour for disabled across the board, so a user never has to guess if a link is active or not depending on the context
<wallyworld_> which is sort of what they need to do now
<wgrant> huwshimi: The bug listings are now arranged pretty strangely: http://people.canonical.com/~wgrant/launchpad/misarranged.png
<huwshimi> wgrant: Yeah, there's some issues there with consistent column widths
<wgrant> huwshimi: It also looks like the vertical text alignment in the status/importance labels is off now.
<wgrant> I thought it used to be centered, but now the text is too high.
<huwshimi> wgrant: Are you using Firefox?
<wgrant> huwshimi: Yeah, this is Precise's Firefox.
<wgrant> Some 9.0 prerelease.
<huwshimi> wgrant: In Chromium the text is actually too close to the bottom :)
<wgrant> Yay :)
<micahg> 9.0 beta 6 in precise, 10.0 beta 1 will come soon
<wgrant> huwshimi: Heh, so it is.
<huwshimi> wgrant: I love web typography
<cjwatson> and Tags is about a pixel lower than the bug title text in that screenshot
<poolie> wallyworld_, i agree, a consistent style for 'disabled' would be nice
<poolie> i think they can both be grey as long as the intensity is markedly different
<wallyworld_> which they are
<wallyworld_> disabled is a lighter shade
<wallyworld_> poolie: it seems that a feature flag i'm trying to use in a test via "with FeatureFixture(xxxx)" isn't being found when I try and access it from inside a zope security adaptor. have you come across this before?
<poolie> wallyworld_, ah, i can believe you but i don't have an easy answer sorry
<poolie> i don't understand the zope magic enormously well
<wallyworld_> np
<poolie> suggest you thrash around until it works, then document that
<poolie> :}
<wallyworld_> yeah, i've been thrashing alright. more like drowning
<wallyworld_> poolie: found it. i am a dickhead. cut'n'paste error. one char difference
 * StevenK ponders quoting wallyworld_'s first two sentences out of context.
<wallyworld_> StevenK: lol
<wgrant> Hm
<wgrant> I think jtv's megalint and sinzui's apocalypse have conflicted.
<StevenK> What did sinzui apocalypse?
<wgrant> interfaces and scripts
<wgrant> jtv reordered lots of imports.
<wgrant> Some of which sinzui moved.
<wgrant> and now the test suite complains on startup.
<wgrant> Let'
<wgrant> s see how bad it is.
<wgrant> May just be the one import.
<wgrant> And fixed.
<wgrant> Hopefully buildbot hasn't finished recently.
<StevenK> wgrant: It still has 90 minutes.
<huwshimi> OK, that's EOY for me. Have a good Christmas/New Years everyone.
<StevenK> huwshimi: You too!
<Ursinha> can anyone explain why I can see a bug has 50 activities but trying to index them fails as out of range?
<Ursinha> In [215]: len(lp.bugs[736661].activity)
<Ursinha> Out[215]: 50
<Ursinha> In [216]: lp.bugs[736661].activity[0]
<Ursinha> IndexError: list index out of range
<wgrant> Ursinha: Interesting -- it works for me.
<wgrant> I wonder if you have a broken cache.
<wgrant> Which version of launchpadlib are you using?
<Ursinha> wgrant: interesting thing is that is working for me as well -- problem was reported by someone else
<wgrant> Hah
<Ursinha> I wonder if he trying to do that anonymously changes that
<Ursinha> it shouldn't, right?
<wgrant> Ah.
<wgrant> It should, but it often does.
<Ursinha> hehe
<Ursinha> why?
<wgrant> 'tis a bug.
<wgrant> Any authenticated user can see them.
<wgrant> But anonymous ones cannot.
<Ursinha> right
<wgrant> Can you file a bug?
<Ursinha> that's reported, I presume
<Ursinha> ah, sure
<Ursinha> :)
<wgrant> Not that I know of.
<wgrant> But we have had many similar situations.
<Ursinha> I'll file a bug
<Ursinha> thanks wgrant :)
<wgrant> np
<bigjools> morning all
<nigelb> Morning bigjools
<StevenK> bigjools: Did you pick up Portal? :-)
<bigjools> StevenK: no, not sure I will
<bigjools> hmmm only 50 attemps to ssh into my server yesterday
<bigjools> spammers are getting around fail2ban
<nigelb> how long is your ban time?
<nigelb> I used to have it as 24 hours :)
<bigjools> default, whatever it is
 * bigjools is fixing the db-devel conflict
<bigjools> now why does pqm say only one conflict when locally I get 132
<wgrant> bigjools: You're probably merging devel, rather than stable?
<wgrant> (132!?!?!(
<wgrant> )
<bigjools> nope
<bigjools> can someone else try?
<bigjools> down to 39 with --lca
<bigjools> ok, out of date checkout it seems
<bigjools> gah, 7 ".moved" conflicts
<wgrant> Yeah, deleted dirs do that, because of all the pycs inside :/
<wgrant> It'll work better if you have clean trees.
<bigjools> no, it's an added dir
<bigjools> I use a checkout as a sandbox
<StevenK> wgrant: I'm so tempted to patch rf-get to run 'make clean' before pulling
<wgrant> That would be very slow...
<bigjools> and I will shoot you
<rick_h__> anyone able to review https://code.launchpad.net/~rharding/launchpad/back_sort_buttons_907376/+merge/86701 for me please?
<rick_h__> backing out fix for bug 907376 so I can refix :/
<_mup_> Bug #907376: Order By Bar "buttons" do not appear on page load <bug-columns> <qa-bad> <regression> <Launchpad itself:Fix Committed by rharding> < https://launchpad.net/bugs/907376 >
<gary_poster> Is anyone working on resolving db_devel?  bigjools, you happen to know?  If not I can try to do it before my next call
<sinzui> gary_poster, I can fix the text conflict if you have not already.
<gary_poster> sinzui, I just did a merge and was about to look at it.  If you are farther along feel free
<sinzui> I have done nothing. I expect import conflicts between jtv and the four horseman of the apocalypse
<gary_poster> sinzui :-) ok I'll proceed and ask if I have questions.  thank you
<bigjools> gary_poster: I did it already
<gary_poster> bigjools, :-P
<gary_poster> ok that wasted time :-) thank you though
<bigjools> gary_poster: however, PQM hated me
<bigjools> argh
<bigjools> gary_poster: did you submit anything yet?
<bigjools> I messed up my commit line
<flacoste> gary_poster, bac, gmb, danilos, benji: lp2kanban got some publicity: http://blog.leankitkanban.com/2011/12/leankit-kanban-api-wrappers/
<benji> cool
<danilos> yeah, cool
<bigjools> allenap: I can has review pls? https://code.launchpad.net/~julian-edwards/launchpad/get-incomplete-ppa-copy-jobs/+merge/86726
<allenap> bigjools: Sure.
<bac> flacoste: yay
<bigjools> jelmer: mind if I assign you a question about java memory limits when building recipes?
<bigjools> actually I think the java bit is a red herring
<jelmer> bigjools: can you perhaps link it to a bug?
<bigjools> jelmer: sure, which one?
<jelmer> bigjools: I suspect it's this one: bug 693524
<_mup_> Bug #693524: Daily builds of Java packages fail: "Could not reserve enough space for object heap" <recipe> <Launchpad Auto Build System:Triaged> < https://launchpad.net/bugs/693524 >
<bigjools> jelmer: https://answers.launchpad.net/launchpad/+question/182645 is the question
<bigjools> thanks, I'll tell him to follow the bug
 * deryck goes offline for lunch, back soon
<rick_h__> deryck: if you get a sec can you let me know if I'm missing a step with the rollback?
<deryck> rick_h__, sure, something makes you think you've missed a step?  Or something not working?
<rick_h__> deryck: well it's just not updating and I ran the  bzr lp-land --rollback=14569 this morning
<rick_h__> so before EOD wanted to make sure I was waiting on pqm and it's not waiting on me
<deryck> rick_h__, what's not updating?  the bug?
<rick_h__> deryck: right http://lpqateam.canonical.com/qa-reports/deployment-stable.html and the bug
<rick_h__> https://dev.launchpad.net/QA/QAForContinuousRollouts says that the bug should be updated and all qa-* tags removed
<deryck> rick_h__, has it finished playing in buildnot yet?
<rick_h__> deryck: ah, problably not. I thought I had skipped that step
<rick_h__> deryck: but right, I only skipped my ec2 tests, not buildbot
<deryck> rick_h__, no, you may have skipped ec2, but you still have to pass buildnot to get merged from devel to stable....
<deryck> rick_h__, once it merges to stable the bug will be updated.
<rick_h__> deryck: ok, I'll hold tight then. Thanks.
<rick_h__> deryck: diff topic, is there a good bit of JS code dealing with ajax that's a good template for testing/doing ajax stuff in our "way"?
<deryck> rick_h__, well any of the order by bar or visible info widget stuff we just worked on is pretty typical.
<rick_h__> deryck: doh true. Was looking through app/javascript
<deryck> rick_h__, heh, yeah, np.
#launchpad-dev 2011-12-23
<rick_h__> anyone around tonight?
<rick_h__> could really use a review at https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376_fix/+merge/86710 so I can start the landing process over night
<rick_h__> it's basically the same as this with a fix https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376 but I can't self review
<nigelb> lifeless: everything okay there after the quake?
<wallyworld_> what quake? was there another one?
<wallyworld_> oh, so there was :-(
<nigelb> Yeah :(
<StevenK> Yeah ;-(
<StevenK> s/;/:/
<wallyworld_> at least it doesn't seem as bad as last time
<rick_h__> morning
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Firefighting: - | Critical bugtasks: 3*10^2
<bac> allenap: writeToDatabaseAndCommit claims to return a token but doesn't.  is the code wrong or the docstring?
<allenap> bac: I'd better fix that. I don't think call-sites care about it, but it's worth getting it right.
<bac> allenap: good
<nigelb> Wait, you guys are working today?
<bac> nigelb: of course!
<nigelb> bac: I applaud your dedication! I hope Moday is off?
<bac> nigelb: yep
<nigelb> :)
<rick_h__> yea, fun stuff
<rick_h__> anyone seen lately that bzr pqm submit ignores the email in locations.conf?
<bigjools> rick_h__: which one?  pqm has its own pqm_email
<allenap> rick_h__: I haven't seen that :-/ I have:
<allenap> pqm_email = Canonical PQM <launchpad@pqm.canonical.com>
<allenap> rick_h__: Do `bzr config` in the branch which you're trying to submit to see if it's picking up your config.
<bac> allenap: done
<allenap> bac: Thanks!
<allenap> bac: Indeed, it was mind-numbing to write :)
<bac> i'm sure of that
<rick_h__> allenap: thanks, yea that affirms my config is there
<rick_h__> but when I tried to do a bzr pqm-submit it ended up pulling up a gpg key for my personal email address and of course pqm then failed with me not auth'd to submit
<allenap> rick_h__: Ah, okay, I thought you were talking about the submission address. Let me see what I've got...
<allenap> rick_h__: Look at the diff in https://bugs.launchpad.net/bzr/+bug/257637
<_mup_> Bug #257637: bzr uses wrong key for signing <signatures> <Bazaar:Fix Released> < https://launchpad.net/bugs/257637 >
<allenap> rick_h__: I think you can set a gpg_signing_key to choose the key you want to use. (I'm still using the workaround mentioned in the comments.)
<allenap> Although it does say it should default to what bzr whoami reports :-/
<rick_h__> allenap: ah, thanks. perfect
<rick_h__> allenap: awesome thanks, that seems to have worked. Now I can kill this ec2 land run to submit this
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 3*10^2
#launchpad-dev 2012-12-17
<StevenK> Ah
<StevenK> So the two tests are terrible?
<StevenK> Effectively, they are getUtility(IPackageUploadSet).getAll(distroseries, name=u"'") with and without exact_match=True
<wgrant> Huh?
<wgrant> No
<wgrant> The tests have revealed a bug in your code
<wgrant> If API input can cause a crash, then our code is buggy
<StevenK> That's not API input, that's calling the method directly
<wgrant> Sure
<wgrant> But I can pass in a name of "'" through the API
<StevenK> wgrant: But name = name.replace("'", "").replace('"', '') probably makes you hate me ...
<wgrant> That's very PHPesque of you
<wgrant> You should escape the string according to the various syntaxes that are involved
<wgrant> Not just blindly remove characters until it works :)
<StevenK> wgrant: IE, remove everything that isn't alphanumeric?
<wgrant> That's precisely the opposite of what I suggested
<StevenK> wgrant: Hold on
<StevenK> Escaping in the ' or " case doesn't work
<wgrant> Why not?
<StevenK> Because as you said earlier, it isn't a valid tsquery syntax
<wgrant> Confused...
<wgrant> A plain ' is not valid tsquery syntax
<wgrant> Because ' is special in tsquery syntax
<wgrant> Like in SQL
<wgrant> How do we resolve this in SQL?
<wgrant> (and Python, and JavaScript, and...)
 * StevenK is clearly missing something
<wgrant> ' delimits a quoted string in Python, SQL, and a tsvector
<wgrant> How do we normal handle ', and other special characters, in a Python or SQL string?
<StevenK> By escaping them
<wgrant> Right
<wgrant> So how might we handle a ' in a tsvector string?
<StevenK> By escaping it
<wgrant> :)
<StevenK> '''' isn't already escaped, or it needs to be escaped twice?
<wgrant> How's that escaped?
<wgrant> '''' is the SQL equivalent of Python's "'"
<StevenK> Ah
<wgrant> It's escaped in SQL terms
<StevenK> wgrant: So, I have two MPs -- I'm happy to self-review the drop-garbo one, it just deletes 182 lines of code. What about the index MP?
<wgrant> That should only be landed once the query branch is ready
<wgrant> In case it requires changes
<StevenK> Mmmmm
<StevenK> I should drop my hacks on DF, add two rows to LDR and update it
<wgrant> (none of this sort of thing should ever land until the whole thing is confirmed to work, really)
<StevenK> We had confirmed it was working on Friday
<wgrant> We hadn't ever done a full test
 * StevenK executes on his plan
<StevenK> wgrant: The lib/lp/registry/model/person.py cowboy on DF is yours?
<StevenK> And lib/lp/soyuz/model/binarypackagebuild.py, I guess
<wgrant> Indeed
<wgrant> Well, person.py is me, binarypackagebuild.py is cjwatson
<StevenK> wgrant: Do you want your change preserved?
<wgrant> No
<StevenK> wgrant: No objections to landing the death of garbo branch?
<wgrant> StevenK: As long as it's done on all four DBs
<StevenK> Yeah, all five -- dev, dogfood, qas, staging and prod
 * StevenK stabs dogfood
<StevenK> Damn it, load
<StevenK> wgrant: DF's front page is unhappy, but +queue looks snappy
<StevenK> Hmmm, I think JS is still broken
<wgrant> Yeah
<wgrant> No convoy, remember
<StevenK> Then remove the flag?
<wgrant> Won't help
<wgrant> 3.5 won't work without convoy
<StevenK> Bleh
<wgrant> Though we still haven't removed 3.3, don't rely on that for much longer :)
<wgrant> Why do you want JS?
<StevenK> I can't see render times
<StevenK> And I'd like to
<wgrant> Ah
<wgrant> view source :)
<StevenK> At least 255 queries/external actions issued in 9.53 seconds
<wgrant> That doesn't sound indexed
<wgrant> What's the query?
<StevenK> However, that's 'xorg' for Quantal done
<wgrant> (or it could just be a really really cold cache)
<StevenK> Let me resubmit
<StevenK> At least 255 queries/external actions issued in 3.97 seconds
<wgrant> Right
<wgrant> Can you track down the main search query and see how long it took?
<StevenK> Cause an OOPS?
<wgrant> Or look at the query log that's inline in the page
<wgrant> (you'll probably need to poke with firebug, or view source)
<StevenK> SELECT DISTINCT PackageUpload.* FROM PackageUpload WHERE PackageUpload.distroseries = 110 AND PackageUpload.status IN (3) AND PackageUpload.archive IN (1, 534) AND ((searchable_names::tsvector @@ 'xorg:*')) ORDER BY PackageUpload.id DESC LIMIT 31 OFFSET 0 == 54 ms
<StevenK> Let me cause an OOPS, I think we're missing some preloading
<StevenK> OOPS-48338103ccc22a3accd4c8c2476a6107
<StevenK> Hmmm, and why isn't that OOPS ID on neem? :-(
<lifeless> you did a grep, not a find, right ?
<wgrant> StevenK: DF doesn't transmit to neem
<wgrant> /srv/launchpad.net/dogfood-logs
<lifeless> The OOPS id isn't used for the file name, because that would lead to trusting untrusted data.
<StevenK> lifeless: No, I did a load up the website and put the id in the field
<StevenK> I was hoping for it to be formatted nicely, not in a file on DF :-(
<wgrant> lifeless: How is trusting that data problematic, as long as you verify it's safe for use as a filename?
<wgrant> If someone wants to collide, they already can
<wgrant> It searches by ID, not filename
 * StevenK finishes writing a novel for CA
 * StevenK glares at this OOPS on mawson
<StevenK> Holy crap it's slow the first time
<StevenK> boost is still pathetic at 8.4 seconds
<wgrant> But which part is slow?
<wgrant> If boost is slow, it probably just means that the loading of the binaries is slow
<wgrant> Not that the search is
<StevenK> Mmmm
<StevenK> wgrant: http://wedontsleep.org/~steven/f.html is the query log
<lifeless> wgrant: mmm true
<StevenK> There is no one query that is terribly slow
<wgrant> C IS FOR COOKIE
<wgrant> Bad StevenK
<StevenK> wgrant: So hurry up and load it so I can rm it
<wgrant> I've loaded it and erased your cookie
<wgrant> We can see from the log that the search query is below 50ms
<wgrant> So for the purposes of this change it is fine
<wgrant> What's not fine is... the entire rest of the page
<StevenK> Indeed
<wgrant> But we already knew that :)
<StevenK> wgrant: So looks like I'll be sticking to DistroSeries:+queue after this work lands ...
<StevenK> wgrant: Anyway, index MP?
<wgrant> Several decades ago I had a branch which mostly fixed DistroSeries:+queue
<wgrant> I'm not sure if it's still around, and it's probably changed enough now that it's not useful
<wgrant> I didn't propose it at the time because cachedproperty on model objects was banned
<StevenK> I think the first step is loading everything related in one query, rather than this looping garbage it's doing
<wgrant> Well
<wgrant> You need to preload things
<wgrant> It's pretty simple
<wgrant> There's just a number of different sorts of things you'll need to preload
<StevenK> wgrant: Are you satisified about the working-ness of this change?
<lifeless> lifeless's contribution to lp: unbanning cachedproperty.
<wgrant> lifeless: Yep
<wgrant> StevenK: Well, your app code is wrong, but otherwise it seems to be good
<StevenK> wgrant: What about it?
<wgrant> StevenK: It'll probably still go horribly wrong if I include a : in my search term, for example
<wgrant> The precise parsing rules don't seem to be very well documented
<StevenK> wgrant: I need to escape : in the same way?
<StevenK> Parsing rules for tsquery seem to not be documented at all
<wgrant> http://www.postgresql.org/docs/9.1/static/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES goes some way
<wgrant> Ah, http://www.postgresql.org/docs/9.1/static/datatype-textsearch.html is what I was looking for
<StevenK> So, ' ", : * look special
<StevenK> Without that errant comma
<wgrant> Well
<stub> abel was the last person to look hard at this
<wgrant> I'd just put the whole thing in single quotes, backslashing any ' or \
<wgrant> Given we want the whole thing treated as a single token
<StevenK> wgrant: Which doesn't handle your : case
<wgrant> Doesn't it?
<wgrant> : isn't special when inside a quoted string
<StevenK> Ah
<wgrant> (and " and * aren't ever special)
<StevenK> Does that mean it turns into @@ 'foo':* for substring?
<wgrant> Right
<wgrant> Er
<wgrant> No
<wgrant> @@ '''foo'':*'
<wgrant> Or @@ E'\'foo\':*' if you're using that syntax
<wgrant> Or @@ E'\'foo\\\'\\\\\':*' if you want to search for foo'\ :)
<stub> raise StupidUserSearchError(wtf)
<StevenK> Haha
<stub> What is all this for btw? Someone working on the antique database text search?
<StevenK> stub: Searching on DistroSeries:+queue
<StevenK> Using tsvector because we like fast queries
<StevenK> And not an eight table join and four likes
<StevenK> Oh god, doesn'
<StevenK> Bleh
<StevenK> Oh god, doesn't this turn into .replace(r'\\', r'\\\\')
<stub> In that case, why have you got odd characters you need to match? For word search, you want to normalize the data in the index.
<stub> Oh... this is for the search
<stub> I don't know what the data looks like you are searching. But if odd characters are coming in, you have bigger problems with fuzzy matching like tsvector/tsquery such as case sensitivity, stop words, stemming
<wgrant> stub: Well
<wgrant> stub: stemming only happens if you use to_tsvector
<wgrant> So it's not a problem here
<wgrant> We just skip stemming and stopwords etc
<wgrant> Creating the tsvector and tsquery directly with exactly the content we desire
<wgrant> (I only opted to use ts2 because otherwise we'd need a custom GIN operator to do prefix search on an array of strings)
<StevenK> God damn I'm confused
<stub> lovely that this is all documented now. i might be able to understand that old code now.
<wgrant> StevenK: What's confusing?
<StevenK> If I paste this, I may get murdered
<wgrant> Let me sharpen my knives
<StevenK> :-(
<StevenK> Waiting for an error, anyway
<StevenK>             name = name.replace("'", "\\'").replace(r'\\', r'\\\\')
<StevenK>             name = "''%s''" % name
<StevenK> And then we enclose it in '' in the SQL condition
<wgrant> Ew
<StevenK> ProgrammingError: syntax error in tsquery: "''':*"
<StevenK> LINE 1: ...chive IN (16) AND ((searchable_names::tsvector @@ '''\''':*'...
<wgrant> One level at a time pls :)
<wgrant> Create the string as you desire ::tsquery to see it
<wgrant> Then wrap that in sqlvalues() or similar, to the do the SQL escaping
<wgrant> (also, you need to escape the backslashes first, or you'll be escaping escape characters
<StevenK> Hmm, that may not be helping
<StevenK> wgrant: The index and NOT NULL MPs are up
<StevenK> The branch I'm working on depends on the indexes, but the NOT NULL is optional
<StevenK> I think sqlvalues isn't helping me
<StevenK> Search string: ''unique-from-factory-py-line3348-101271''
<StevenK> sqlvalues escaped: '''''unique-from-factory-py-line3348-101271'''''
<wgrant> Oh right
<wgrant> sqlvalues still uses the horrid old non-standard escaping syntax
<StevenK> I'm tempted to ignore it
<StevenK> And just build the string by hand
<wgrant> Why?
<StevenK> Mainly because it's annoying me
<StevenK> However, I win
<wgrant> What is your code?
<StevenK> wgrant: http://pastebin.ubuntu.com/1444536/
<wgrant> Line 3 is odd
<wgrant> Are you sure you really want a raw string?
<wgrant> that'll replace \\ with \\\\
<wgrant> Not \ with \\
<StevenK> Ah
<StevenK> Now '\\', '\\\\'
<StevenK> I seem to not be stabbed, so that's a good thing ...
<wgrant> In a raw string, a backslash causes the following character to be interpreted literally, but the escaping backslash is retained in the final string
<wgrant> Well
<wgrant> You're still using sqlvalues
<wgrant> Which you should stop doing
<wgrant> But it's better than manually doing the SQL escaping
<StevenK> Oh?
<wgrant> SQL("whatever @@ ?", params=('blah',))
<StevenK> Then search_string turns into "''%s''" ?
<wgrant> Confused
<wgrant> Why?
<StevenK> Because you had ''' in your earlier example
<wgrant> params handles escaping for you
<wgrant> That's why it exists
<wgrant> You give it a normal string, it will inject it into the SQL safely
<StevenK> wgrant: sqlvalues gutted from name and version searching
<wgrant> Lovely
<wgrant> sqlvalues is thoroughly deprecated; you should avoid it except in legacy code
<StevenK> wgrant: http://pastebin.ubuntu.com/1444550/
<wgrant> StevenK: searchable_versions can be done with a native Storm expression
<wgrant> There are Array and ArrayContains (though the latter is @> rather than <@, I'm sure you can flip your args :)) bits in lp.services.database.stormexpr
<wgrant> And the parens in the searchable_names SQL are likely redundant
<wgrant> And "if name is not None and name != ''" sounds suspiciously close to "if name"
<StevenK> Old code
<wgrant> I'm an equal opportunity complainer
<StevenK> wgrant: http://pastebin.ubuntu.com/1444568/
<wgrant> Indentation!
<wgrant> That looks like you're appending an ArrayContains and an Array
<StevenK> Fixed
<StevenK> Any other complaints?
 * StevenK waits for the flood
<wgrant> Sadly not
<StevenK> wgrant: Should I put up an MP for this branch, then? Then you can happily nitpick more sections of the code.
<wgrant> yep
<StevenK> wgrant: That MP is now up
<StevenK> wgrant: Thanks. Shall I wait for stub about the index branch, or which?
<wgrant> Looks good, but let me just test something
<StevenK> wgrant: Ah, seeing if a composite helps?
<wgrant> Yeah
<wgrant> It still doesn't
<StevenK> :-(
<StevenK> It's still fast enough
<wgrant> Sure
<StevenK> And beats the pants off the 15 seconds it was :-)
<wgrant> (also, it's "indices", not "indicies")
<StevenK> Right, index branch landing
<StevenK> When stub reappears and has applied them both to qas and prod, I'll land the use branch
<StevenK> And when he approves the NOT NULL branch, I'll land that.
<wgrant> Right
<wgrant> Hmm
<StevenK> Hah
<StevenK> DRS is already used for DistroSeries:+queue
<wgrant> Yes
<wgrant> IIRC it only preloads PU*
<wgrant> Not everything else
<StevenK> BPF, SPRF, SPR
<StevenK> It may be trying to add in BPB
<StevenK> But it at least needs BPB and BP
<StevenK> *PB
<StevenK> stub: O HAI. Can you have a look at 40-2 in devel and apply the indicies to prod and qas?
 * stub looks
<StevenK> stub: https://code.launchpad.net/~stevenk/launchpad/index-searchables-for-pu/+merge/140120 is the MP
<stub> StevenK: Backups still running, so it needs to wait a bit longer
<stub> I'll do staging and qas
<stub> StevenK: It is on qas, staging already had it from nightly updates
<cjwatson> StevenK: you can drop my binarypackagebuild.py cowboy if you like
<StevenK> cjwatson: It was two lines, and didn't hurt to restore.
<StevenK> I certainly committed more evils to DF
<StevenK> stub: Surprised the backups are still going
<stub> yeah
<stub> 9 hours so far, only got as far as translationmessage
<stub> less than 30 mins into that one
<stub> oh, only one minute into translationmessage
<StevenK> I thought they only took six?
<wgrant> wildcherry's backup would normally have finished about 20 minutes ago
<stub> yeah, suspect high io. Running at just over 10% iowait atm.
<wgrant> It's possible that allocate-revision-karma is causing excessive load as it catches up with 15 million revisions of backlog
<stub> revisionkarma catchup
<stub> yer
<wgrant> It's about half done
<stub> and our io utilization has been creeping up anyway as things get optimized and we become less cpu bound.
<wgrant> Yeah
<wgrant> I've been noticing more and more IO issues in explain analyzes lately.
<wgrant> Blocks just don't read as quickly as they used ot
<stub> wgrant: production or dogfood?
<wgrant> Prod
<wgrant> dogfood's always been dreadful :)
<wgrant> Reading in cold blocks on prod is just not as fast as it probably should be
<stub> I'll blame contention on production, but we have a few knobs to twiddle if it looks like it needs it
<stub> At the moment, we are tuned so pg thinks random access is pretty darn fast due to almost always having our data in disk cache. That might be a lie now.
<wgrant> Yeah
<wgrant> With the DB this big...
<stub> its always been big, it is the size of the hot set of data that is important.
<wgrant> It wasn't all that long ago that the DB was 250GB
<wgrant> Now it's more than 350GB
<wgrant> We've gone from 2x RAM to 3x RAM
<stub> yeah, but ram is 128GB
<StevenK> Hmm, that's a bit horrible
 * StevenK glares at buildbot
<StevenK> Anyway, this is wgrant's fault.
<StevenK> He told me to Storm-ify it.
<nigelb> StevenK: When in doubt, blame wgrant?
<StevenK> It works with other tests
<StevenK> So maybe the doctest is just bong
<wgrant> StevenK: The doctest is probably calling it with a str, not a unicode
<StevenK> Yes
<StevenK> wgrant: The previous call has unicode, but the one that errors doesn't.
<wgrant> Right
<wgrant> Easy fix
<StevenK> Yeah, landing it in a few
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: benji | Firefighting: - | Critical bugs: <150
<gary_poster> sinzui, hey.  Am I right that https://code.launchpad.net/~chrisjohnston/launchpad/fdl/+merge/140216 will not need a db change, because 330 is all we have in the DB?  On a related note, can you or benji get that landed soon?  I don't have an LP dev environment ATM
<sinzui> correct
<cjohnston> mornin sinzui
<gary_poster> thanks sinzui
<sinzui> np, hi cjohnston
<gary_poster> benji do you have an LP dev environment to land cjohnston's branch, above, sometime today?
<rick_h_> gary_poster: I'll get it going. cc/ benji
<gary_poster> thank you much rick_h_
<cjohnston> thanks rick_h_
<rick_h_> cjohnston: went ahead and sent it off to ec2 just in case it hits some stray tests. See you in the afternoon.
<cjohnston> thanks
<cjohnston> rick_h_: I spoke with gary_poster more about my issue with lpsetup, I guess it's because I'm on quantal
<gary_poster> that's my hope anyway :-/ was working for us on precise
<rick_h_> cjohnston: ah the lxc issue? I have lxc working on here, but I did things more the hard way to customize the setup so not sure
<rick_h_> cjohnston: oh right, benji started to help me with some issues on quantal and I ended up just manually creating an lxc the way I wanted and doing more manual setup in there
<rick_h_> back in budapest
<cjohnston> Gotcha.. It would be really nice if the barrier to help for the community as low as possible
<cjohnston> rick_h_: ping
<rick_h_> cjohnston: pong
<rick_h_> cjohnston: what's up? around just a couple min
<rick_h_> cjohnston: reforcing the build on buildbot, spurious failure
<rick_h_> hopefully that's what you were looking for
<rick_h_> will check back in later tonight
<cjohnston> yup
<cjohnston> so its all good then
<cjohnston> thanks
<rick_h_> cjohnston: yea, ec2 came back clean so just need to smack build-not around a bit
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: <150
<cjohnston> cool
#launchpad-dev 2012-12-18
<cjohnston> sinzui: would you say that bug #295872 can be closed since it has been reworded and not had a reply from the op?
<_mup_> Bug #295872: more info required about sprints and meetings on subscribe pages <confusing-ui> <lp-blueprints> <sprints> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/295872 >
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: frankban | Firefighting: - | Critical bugs: <150
<czajkowski> frankban: morning
<frankban> czajkowski: morning
<czajkowski> gmb: morning any idea on https://answers.launchpad.net/bugzilla-launchpad/+question/216825 not sure why you're sub'd to it
<gmb> czajkowski: I'm not sure what he's getting atâ¦ it looks like maybe he wants to have a CSV export of bugs; I don't think we offer that (allenap, do we?). I'm subbed to it through the Bugzilla Launchpad plugin, which is what he filed it against.
<czajkowski> ah I see
<czajkowski> morning btw
<czajkowski> sinzui: aloha
<allenap> czajkowski, gmb: The only vaguely relevant thing that comes to mind is +text for individual bugs.
<sinzui> czajkowski, allenap, gmb. I am still a sleep. I replied to the bug suggesting an API script and provided a sample that will make a CSV for all the bugs in a project
<czajkowski> sinzui: morning lotta bugs in there atm and not sure how to triage them
<czajkowski> when you're more awake aka had some coffee fancy going through them so I can learn
<sinzui> hi czajkowski. I am awake.
<czajkowski> sinzui: not sure what https://bugs.launchpad.net/lpsetup/+bug/1090934  should be triaged
<_mup_> Bug #1090934: Error: unable to find the ip address of the LXC container: Command '['/usr/bin/env', 'lp-lxc-ip', '-i', 'eth0', '-n', 'lptests']' returned non-zero exit status 1 <lpsetup:New> < https://launchpad.net/bugs/1090934 >
<sinzui> czajkowski, high. We don't need to fix it until the next LTS or if it blocks canonical developers from doing their job
<czajkowski> nods
<sinzui> czajkowski, the key here is blocking us or buildbot
<sinzui> So the scheduled fix is more than a year away :(
<czajkowski> sinzui: what tags do I need to add
<sinzui> non
<sinzui> I was going to say none, but I think we want to add LTS and add a comment that the script has to work in LTSs and we believe this bug means it will not work in LTS T
<sinzui> The loggerhead bug is low, No comment, no tags
<czajkowski> sinzui: thanks
<jcsackett> sinzui: can you refresh my memory on something? for +sharing, is the intent that drivers should be able to modify data on that page, or just see it?
<sinzui> Only see it
<jcsackett> sinzui: ok, thanks.
<czajkowski> sinzui: do you mean will work in non LTS ?
<sinzui> czajkowski, it does not have to work for a non-LTS like quantal We use it on LTSs to test Lp.
<czajkowski> ah I see
<czajkowski> thanks for the calrification
<sinzui> czajkowski, lpsetup is a "nice to have" for developers who do not use non-lts
<czajkowski> nods
<timrc> sinzui, why isn't it possible to make a public team, private?
<sinzui> because something the team has done has become a public
<timrc> sinzui, I thought that private teams could operate in public contexts now
<timrc> and only identifying details would be shared
<sinzui> I cannot say exactly what. If the team owns something that cannot be owned by a private team or if its identity is know to a bug, mailing list, or public ppa, we know google knows all about it
<sinzui> timrc, no, the mailing list is public forever
<timrc> sinzui, sure but what if the list of people in the team before it's made private is irrelevant... e.g. the reason we want to make the team private is to hide the identity of new members?
<sinzui> The team membership detail could be cached...
<timrc> sinzui, the pre-privatization detail...
<sinzui> and its involvement in bugs and answers was not sanitised. lots of data leaked though those interactions
<sinzui> timrc, honestly Lp does not support it. No one wants to pay us to sort out the details of changing team visibility
<timrc> sinzui, that's the answer I wanted :)
<sinzui> timrc, I advise everyone in this case to create a replacement team that is setup right, and merge the problem team into it
<sinzui> You have to setup memberships again, and you loose PPAs and mailinglists, but it gets around the team restrictions
<timrc> sinzui, cool... so would adding the public team to the private team be a satisfactory way of creating a new private team with the members of the old public team?
<sinzui> adding/
<sinzui> ?
<timrc> sinzui, yeah, I got "Add member", specify the team, then as an admin of that team, accept the invite?
<timrc> er goto
<sinzui> timrc, You can a private team to a public team. The public team members will be permitted to know about the private team even if the members do not intersect because the super team has the right to vet its members
<sinzui> I think that is a safe solution
<sinzui> s/You can a/You can add/
<timrc> It appears that I can add a public team to a private team as well, but you're saying that's not safe?
<timrc> It seems safe in the sense that people can't arbitrarily add themselves to the public team to gain access to the private team unlawfully
<sinzui> timrc, sorry, I had to deal with a deliver
<sinzui> y
<sinzui> timrc, let's not confuse visibility public/private with exclusive/inclusive membership policies
<sinzui> timrc, we do not trust inclusive team (Open and Delegated) because they do not vet membership. All exclusive (Restricted and Moderated) teams can own trusted things because they check their members. You can add exclusive teams as members and you don't need to worry about if their visibility is public or private
<sinzui> timrc, Since you care most about hiding some member, I think you suggestion of a private subteam is sensible
<jcsackett> anyone free to review https://code.launchpad.net/~jcsackett/launchpad/edit-icons-permissions/+merge/140556
<czajkowski> jcsackett: evening
<timrc> sinzui, so why is "embargoed" for private projects (re: http://blog.launchpad.net/general/private-projects-and-private-blueprints-leave-beta) not simply "Proprietary, can be public" to keep it consistent with other sharing policy information type meanings
<timrc> the meaning of embargoed in the context of branch sharing, for example, seems different than in the context of project type
<sinzui> timrc, yes, it is
<sinzui> timrc, different teams built the features that the meanings were not reconciled
<wgrant> Hm
<wgrant> How is the meaning different?
<wgrant> Stuff in an Embargoed project can't be public
<timrc> wgrant, there is another information type for sharing, called Proprietary, can be public which is what is described at that  blog post
<sinzui> nor is the use of embargoed symmetric. you cannot have embargoed bugs, but you can have branches and blueprints
<timrc> but as embargoed
<wgrant> timrc: That's not an information type. It's a rule for what information types the project can contain, and it's not the same as Embargoed.
<timrc> ok, so there's Embargoed and then there's Embargoed, got it ;)
<wgrant> Nope
<wgrant> Where is the second meaning
<wgrant> ?
<sinzui> I think the real issue is that a project can be embargoed, but no policy recognises embargoed bugs
<timrc> The meaning of Embargoed in terms of a sharing policy: http://pastebin.ubuntu.com/1448655/
<timrc> wgrant, ^
<timrc> The blog describes an Embargoed project as "Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary."
<wgrant> Oh
<wgrant> They actually still went with that?
<sinzui> Embargoed can also become public if the project changes.
<wgrant> But in terms of what they actually mean in terms of Launchpad restrictions, they are the same
<wgrant> But the described uses are different
<sinzui> timrc, from your perspective, nothing goes public
<timrc> sinzui, right, we are just contemplating the use of private projects in the new year, so I'm trying to really understand all this stuff now :)
<timrc> So within my team we have the unique ability of adding external sources for PPAs... let's say that the external dependency is a snapshot of -updates for some Ubuntu series, will the PPA still build with the latest -updates and supercede any dependencies in my snapshot with whatever's latest?
<timrc> or will adding an external dependency cause the PPA to not build with the latest Ubuntu -updates
<StevenK> You can edit the external dependencies to not pull from -updates
<timrc> StevenK, Ah, yes that's right
<timrc> StevenK, Thanks for the reminder
#launchpad-dev 2012-12-19
<StevenK> wgrant: Raring +queue for NEW on prod: 91 queries/external actions issued in 0.94 seconds
<wgrant> Excellent
<StevenK> Done is still dreadful, at 3.3 seconds
<wgrant> But the search query is only about 300ms
<wgrant> So that bit's fine
<wgrant> Well, "fine"
<StevenK> It was 70ms on qas :-(
<wgrant> It'll probably be quicker now it's hot
<wgrant> Let's see
<wgrant> Yeah, 75ms now it's hot everywhere
<StevenK> cjwatson: I'm going to close bug 33700, and you should able to change the queue API script now too
<_mup_> Bug #33700: could queue filters match source as well as binaries? <lp-soyuz> <qa-ok> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/33700 >
<cjwatson> StevenK: ok, great, thanks - I'll sort that out tomorrow morning, with the bug closure mail serving as a reminder
<cjwatson> I just need to flip over the exact-match default, right?
<StevenK> Right
<StevenK> versions will have exact_match no matter the setting of it now, too
<cjwatson> yeah, I don't think we'll notice that bit in the script :)
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: <150
<cjwatson> StevenK: queue client changes done now
<StevenK> cjwatson: \o/
<cjohnston> anyone have an idea as to why https://code.launchpad.net/~chrisjohnston/launchpad/181725 didn't pick up the code changes?
<StevenK> It probably failed to scan
<StevenK> cjohnston: I'll get it re-done, hold a mo
<cjohnston> ty
 * StevenK grumbles at the branch scanner.
<cjohnston> hehe
<StevenK> cjohnston: To unblock you, you can rename the branch and re-push, or I can look again after I wake.
<cjohnston> StevenK: ack
<cjohnston> StevenK: it seems to have refreshed
<cjohnston> StevenK: looks like one of your branches may have had the same issue
<czajkowski> cjohnston: you trying to buil up brownie points :)
<cjohnston> depends.. what can I get with them
<sinzui> cjohnston, I expect both your branches to land in about 6 hours
<cjohnston> sinzui: cool.. I'm hoping to have a couple more over the holidays
<sinzui> I will be about to review
<cjohnston> is there a direct link to the 500 error page?
<czajkowski> sinzui: you're meant to be on hols mister :)
<cjohnston> deryck: ping
<sinzui> I could be if we turned Lp off for the holidays
<cjohnston> can we turn everything off?
<czajkowski> sinzui: I'll keep an eye on Rts next week so don't worry
<czajkowski> and I've already sent a reminder mail to canonical list re commercial projects
<czajkowski> and had 7 mails :)
<cjohnston> https://bugs.launchpad.net/launchpad/+bug/483945 needs to become super most highest absolutely critical ;-)
<_mup_> Bug #483945: No way to ask Launchpad to refresh a stale diff <code-review> <lp-code> <mp-preview-diff> <openstack> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/483945 >
<czajkowski> cjohnston: no it doesnt :)
<sinzui> bug reports cannot be ignored. Lp was attacked twice during holidays by spammers
<czajkowski> our criticals are triaged as oops or timeouts or stuff like that
<cjohnston> its bothering me and keeping my code from being able to land in launchpad
<cjohnston> it was sarcastic
<czajkowski> cjohnston: bothering is a a bit flippiant also :)
<deryck> hi cjohnston
<cjohnston> What's the chance that there are ton's of bugs that are open that arent accurate anymore
<cjohnston> deryck: bug #413597
<_mup_> Bug #413597: "clearfix" class should be removed from multiline editor widget <css> <javascript> <lp-web> <opera> <trivial> <Launchpad itself:Triaged> <LAZR Javascript Library:Triaged> < https://launchpad.net/bugs/413597 >
<cjohnston> I'm wanting to take a stab, but seems things have changed in the code base, so I just want to make sure I do it right
<rick_h_> cjohnston: I'll help you out with that if you need. I hacked on it a bunch when I added the auto expanding and such when I first started
<cjohnston> rick_h_: sure thing
<deryck> cjohnston, yeah, that was about to me my suggestion, to talk to rick_h_ :)  He's deeper in all that now than me.
<rick_h_> oh hmm, opera is blacklisted?
<sinzui> cjohnston, we now have unscan branch that fixes the stale diff. Someone could add an action to the UI to run it. the root issue is that Lp fails to scan branches. No one would want that bug fixed if Lp's branch-scanner could be trusted
<deryck> rick_h_, in the day it was one of those, if Y.UA == Opera return kind of things.
<rick_h_> deryck: yea, I think I cleaned some of that up. Will have to refresh.
<deryck> rick_h_, not a black list per se.
<deryck> rught
<sinzui> rick_h_, I test with opera. It works
<rick_h_> cjohnston: but yea, I'm probably the guy you want on that stuff
<deryck> right, even
<cjohnston> sinzui: ya, I want that button ;-)
<cjohnston> rick_h_: I think it may be in lib/lp/app/doc/lazr-js-widgets.txt now
<deryck> rick_h_, cjohnston we should probably close any tasks on lazr js too, since we don't use that anymore.
<cjohnston> It doesn't look like anything was ever decided on bug #666379, and its been two years since it was last commented on.. Suggestons?
<_mup_> Bug #666379: "Community" term is misused <lp-code> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/666379 >
<cjohnston> and sinzui could you please take a look at bug #666738.. I commented on it, though may also want lifeless' input as well as to how to best fix
<_mup_> Bug #666738: cannot tell who is 'essential' and who is not for a spec <confusing-ui> <lp-blueprints> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/666738 >
<cjohnston> deryck: does that mean that the bug isn't a bug anymore?
<deryck> cjohnston, the js bug?  it's still relevant for launchpad, just not lazr-js.  we moved all the lazr-js code in tree, and don't maintain it separately now.
<cjohnston> gotcha.. just the effects lazr-js stuff
<sinzui> cjohnston, I replied with an idea
<cjohnston> ty
<rick_h_> cjohnston: so on the js bug, looks like the clearfix is in the template used, but watch out as there's matching css rules in forms.css for .lazr-multiline-edit .clearfix h3
<rick_h_> but like sinzui says, might just work if the conditional was removed for opera so we'd want to check if the issue still exists first
<cjohnston> conditional?
<sinzui> rick_h_, I think we have one in the code because it was unhappy with the *last* version of YUI
<rick_h_> if (Y.UA.XXX
<sinzui> We absolutely should remove the guard and test with the current version
<rick_h_> sinzui: ah
<rick_h_> cjohnston: so yea, there's no opera conditionals in inlineedit/editor.js any more
<sinzui> most the other opera guard were fixed by yui 3.4
<cjohnston> rick_h_: so that means it just needs to be tested in opera?
<rick_h_> cjohnston: yea, and looks like just tested to work and then closed off as fixed released long ago
 * cjohnston goes to install opera
<sinzui> Is this about the icon placement and field is not outlined bug for opera?
<rick_h_> ah, yea it refernces the icon placement
<rick_h_> #412579
<_mup_> Bug #412579: Icons are not aligned properly for multi-line editor in Opera <LAZR Javascript Library:Won't Fix> < https://launchpad.net/bugs/412579 >
<rick_h_> so guess check that, if it's still not right we can go after the clearfix in the template
<sinzui> rick_h_, cjohnston. it is mostly fixed
<sinzui> there is a 2 pixel misalignment
<cjohnston> does that need to be fixed?
<sinzui> no
<cjohnston> so just mark fix released?
<sinzui> the issue was worse when the tab-treatment obscured text.
<sinzui> The editor works very well now that we can wee what is being edited
<cjohnston> I'm guessing that #483945 isn't the most easiest bug out there.. even though it says trivial?
<_mup_> Bug #483945: No way to ask Launchpad to refresh a stale diff <code-review> <lp-code> <mp-preview-diff> <openstack> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/483945 >
<cjohnston> too had I'm not fixing any critical bugs to make the critical bug chart go down.. lol :-(
<czajkowski> cjohnston: tis ok we have purple doing a shopping job on that list
<cjohnston> czajkowski: ya, but it would be cool for me :-P
<cjohnston> czajkowski: did you see the pictures of the book I posted on FB?
<czajkowski> narp
<cjohnston> http://goo.gl/hOuIt
<czajkowski> congrats
<cjohnston> deryck: so does your comment from earlier mean that all of https://bugs.launchpad.net/lazr-js needs to be marked effecting lp and closed for lazr-js?
<deryck> cjohnston, basically, yes.  assuming the bug still exists, and assuming the bug wasn't filed on behalf of another project using lazr-js.
<cjohnston> deryck: that maybe sounds like something someone with more experience should do
<deryck> cjohnston, yeah, unfortunately.  rick_h_ could advise if you take a stab at it.
<cjohnston> I feel like unless its blatantly obvious I may ask more questions that it would just be best for him to sort
<rick_h_> ugh, yea. That'd take some processing to see if they still apply. Many of the 'lazy should haveXX' would just go away. if LP doesn't need it then it's never going to happen
<sinzui> I am template to close all those "lp should" bugs reported by Lp staff over 5 years ago. If no user want the feature, I don't want to see the bug in listings
<sinzui> s/template/tempted/
<czajkowski> sinzui: want a hand?
<sinzui> sure, sort the listing by age, look for bugs reported by former staff, close the bug if it is about an idea that with no dupes or affects other users
<czajkowski> ok
<czajkowski> sinzui: what about the rosetta bugs? https://bugs.launchpad.net/launchpad/+bug/1420
<_mup_> Bug #1420: IRosettaStats current interface doesn't suit database/pofile.py very well <lp-translations> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1420 >
<sinzui> POTs are still misusing the code
<czajkowski> pots are misunderstood :)
<lifeless> cjohnston: oh hai :)
<cjohnston> hey lifeless
<cjohnston> lifeless: https://code.launchpad.net/~chrisjohnston/launchpad/666738/+merge/140758 work for you?
<cjohnston> lifeless: updated
<cjohnston> lifeless: do I have to do anything special when wrapping the text into two lines?
<lifeless> cjohnston: do you need to wrap it?
<cjohnston> depends on how long you guys want it
<lifeless> Dunno :). sinzui ? ^
<cjohnston> makes the line 124 char
<sinzui> wrap what? in what
<cjohnston>           src="/@@/subscriber-inessential"
<cjohnston>           title="This person has indicated interest in the blueprint but is not required to be present at every discussion."
<cjohnston>         />
<sinzui> CSS rules in page do the wrapping for us.
<cjohnston> in the file
<cjohnston> template file I guess
<sinzui> pt can be exempted from wrapping because of cases like this
<cjohnston> ok. so a 124 char line is fine?
<cjohnston> lifeless: just pushed, should be ready in a momeny
<cjohnston> moment
<sinzui> flacoste, I thought I deleted mailing-list-experts from production
<flacoste> sinzui: feel free to merge it into oblivion
<sinzui> The team is not used. It is only mentioned in Lp's comments
<sinzui> oh, and those are disabled tests too
<StevenK> wgrant: http://pastebin.ubuntu.com/1451041/
#launchpad-dev 2012-12-20
<StevenK> wgrant: Can you poke at http://pastebin.ubuntu.com/1451410/ and see if you can make sense of why my preloading changes don't seem to help at all?
<wgrant> StevenK: What's not working?
<StevenK> wgrant: It seems to me like the preloading PackageDiff call isn't working at all, since the code still loops loading PackageDiffs one by one
<wgrant> Well, that's not really surprising
<wgrant> How would that work?
<wgrant> The PK is ID, not (from, to)
<wgrant> So simply loading them into the Storm cache isn't going to do much good
<wgrant> As the view is going to be querying by (from, to)
<StevenK> wgrant: Eh? I'm using getDiffsToReleases() which should okay, no?
<wgrant> Sure, that'll load the right ones
<wgrant> But how will subsequent calls know that they're already loaded?
<wgrant> How will they find them?
<StevenK> wgrant: What's more concerning is I can't even see the preloading query in the OOPS
<wgrant> StevenK: It's there for me
<wgrant> I see the initial diff grabbing the actual packagediff rows
<wgrant> s/diff/query/
<wgrant> Then a subsequent query looking for the existence of a packagediff for the single source
<wgrant> So it all seems fine
<cjohnston> Question guys.. the bugs I fixed are starting to have qa-needstesting, am I allowed to test them? if so, what's the process for testing them?
<StevenK> cjohnston: Try it on qastaging
<wgrant> Just check that they don't obviously break anything on qastaging.launchpad.net
<wgrant> If the fix is good, change qa-needstesting to qa-ok
<cjohnston> ok.. cool.. wasn't sure if as the code fixer I was allowed to do the reviews
<wgrant> We normally QA our own stuff if we're around
<cjohnston> ok.. anyone know what the URL is to create a new distro?
<StevenK> I think you need to be an admin for that.
<wgrant> Ah, yeah, normal users can't create them
<cjohnston> do you two have admin access?
<wgrant> No
<cjohnston> I can't test line 46 then https://code.launchpad.net/~chrisjohnston/launchpad/181725/+merge/140653
<wgrant> I'm not sure that such a trivial change warrants the effort of obtaining admin privileges
<wgrant> The test suite is a pretty good indication that it works fine
<cjohnston> ok.. the public views look fine.. so I can mark it qa-ok?
<wgrant> Right
<cjohnston> its just a text change anyway
<cjohnston> ok..thanks wgrant and StevenK
<StevenK> cjohnston: You're welcome
<StevenK> wgrant: Hmm, yeah. Now I see the preload
<cjohnston> 4 coomits in one week.. thats a record for me.. by about 3
<StevenK> Heh, I don't track how many commits I do
<cjohnston> StevenK: your kind-of on the LP team... I'm community :-P
<StevenK> So it appears my attempts at preloading to drop the query count don't really help.
<StevenK> I take that back. Preloading changesfile drops from 237 to 213.
 * StevenK stabs zope.tal
<StevenK> It looks like the tal has a condition on not: packagediff or so, and that's forcing storm to grab each packagediff, LFA and then LFC
<wgrant> StevenK: I'd expect a packagediff existence check, then a packagediff retrieval query, then LFA and LFC queries for each diff
<StevenK> Yeah, I'm trying to staple packagediff into CPU so I can stop those
<StevenK> Why it chains through to LFA and LFC, though ...
<StevenK> package_diffs is a SQLMultipleJoin, and the TAL seems to only want a fmt:link
<wgrant> Why wouldn't that want LFA and LFC?
<wgrant> It needs LFA for the link
<wgrant> And LFC for the size
<StevenK> wgrant: 237 down to 186.
<wgrant> Getting there
<StevenK> Now 167
<StevenK> (214-186: Preloading PUBs, BPBs and binary SPRs, 186-167: Preloading all SPNs for source and binary SPRs)
<StevenK> 153, preloading all DASes.
<StevenK> Hmmm, looks like preloading PCJs is causing duplicate queries against PackageUploadBuild
<StevenK> Not sure how, given uploads should be a list of objects, and builds is a cachedproperty
<wgrant> Check what is generating the queries/
<StevenK>     keys.update(map(partial(getattr, owning_object), foreign_keys)) in load_releated
<StevenK> load_related, sigh
<StevenK> Oh, wait a minute, why is that calling checkPermission?
<wgrant> It may be trying to access a secured attribute to find the related IDs
<StevenK> Bleh, rewritting the PCJ query just pushes the calls to my LFA preloading
<StevenK> And changing the LFA preloading makes the query count jump to 178
<StevenK> Back down to 154, but this is terrible :-(
<StevenK> Bleh, and that just pushes it down to 'for item in uploads' :-(
<wgrant> What are you doing?
<wgrant> StevenK: How's it pushing it down? It should be cached
<StevenK> wgrant: I'm trying to avoid the stacks of queries that the permission check is causing
<StevenK> What I mean by pushing down is further along in the method -- it was inside loadPackageCopyJobs, and then in the bulk loading for LFAs for changesfile, and now it's at the end
<wgrant> StevenK: For the same objects?
<StevenK> Yeah
<wgrant> How?
<StevenK> uploads = list(self.batchnav.currentBatch())
<wgrant> They must be cached
<StevenK> upload_ids = [upload.id for upload in uploads]
<wgrant> If checkPermission succeeds once, it will be cached
<wgrant> So I don't see how you're pushing them back
<StevenK> wgrant: I can upload an OOPS if you wish
<wgrant> How did you do the permission preloading?
<StevenK> I haven't
<wgrant> How'd you move it, then?
<StevenK> By avoiding load_related() and using upload_ids
<wgrant> Why?
<wgrant> Don't you want to check permissions?
<StevenK> Have you read the PU permission check? It's hideous
<wgrant> Sure
<wgrant> But that means you might want to prepopulate the permissions another way
<wgrant> Not just somehow attempt to work around ever accessing an attribute of the protected object
<wgrant> In the end you are going to have to access a PackageUpload
<wgrant> So there's no point introducing ugly hacks to attempt to avoid checking permissions
<StevenK> wgrant: I'm curious why the upload_ids = [upload.id for upload in uploads] doesn't cause them
<wgrant> StevenK: id is presumably zope.Public
<StevenK> Ah
<StevenK> Right, since we have the SPRs for each PU preloaded
<StevenK> wgrant: load_related() avoidance hacks removed
<wgrant> StevenK: You're fixing bug #835645, aren't you?
<_mup_> Bug #835645: DistroSeries:+queue timeout paginating <lp-soyuz> <queue-page> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/835645 >
<StevenK> wgrant: Hopefully
<czajkowski> aloha
<lifeless> gary_poster: btw
<lifeless> gary_poster: I know you guys are not focused on LP any more, but I've added some more stuff to testr that may be of interest :) -
<gary_poster> lifeless, hiya. :-)
<lifeless> gary_poster: hiya :)
<lifeless> https://testrepository.readthedocs.org/en/latest/MANUAL.html#remote-or-isolated-test-environments and
<lifeless> https://testrepository.readthedocs.org/en/latest/MANUAL.html#automated-test-isolation-bisection
<gary_poster> lifeless, so this could be used to build test runs across machines the way statik talked about way back when, yeah?  among many other things.  cool!
<gary_poster> nice lifeless.  I'm guessing you are using testr in anger, then :-)
<lifeless> gary_poster: well I have been for ages; I got some employer sponsored time to step it up a level though.
<lifeless> gary_poster: openstack's nova project now uses testr as its official test runner
<gary_poster> lifeless, awesome!  great.  thanks for highlighting the changes--I'll keep 'em in my back pocket
<lifeless> gary_poster: and yeah, multiple machines would be easier with this I think. Aaron's wrapper should become simpler too.
<gary_poster> cool
<lifeless> gary_poster: the isolation analyzer is my favourite bling right now though. I think it would have made your analysis work a lot easier.
<gary_poster> ah yes, this is what I was going to do in Go and lost interest in :-)
<lifeless> Heh :).
<sinzui> jcsackett, you about?
<czajkowski> sinzui: would you mind looking at https://bugs.launchpad.net/launchpad/+bug/1092643  if you have time please.
<_mup_> Bug #1092643: Release Management and Downstream Tracking <feature> <Launchpad itself:New> < https://launchpad.net/bugs/1092643 >
<sinzui> done
<czajkowski> sinzui: cheers
<czajkowski> jcsackett: if you like mashups http://bootiemashup.com/blog/
<jcsackett> czajkowski: cool!
<sinzui> wgrant, I revised the branch to convert the 400 to a 404 https://code.launchpad.net/~sinzui/launchpad/path-info-bad-request/+merge/140698
#launchpad-dev 2012-12-21
<StevenK> wgrant: With Component and Section preloaded, down to 70.
<wgrant> Excellent
<wgrant> What are the rest?
<StevenK> wgrant: PackageDiff and related LFA and LFC; SPPH; ArchivePermission
<wgrant> Right
<StevenK> wgrant: So I think it's reasonable
<StevenK> We've dropped 160 queries
<StevenK> wgrant: http://pastebin.ubuntu.com/1453695/
<StevenK> wgrant: No thoughts at all, I see.
<wgrant> StevenK: Hmm
<wgrant> Looks reasonable so far
<StevenK> wgrant: Hmmm, given IPackageUploadSet.getAll() is the meat inside IDistroSeries.getPackageUploads(), and getAll() does the propertycache dance, I guess the other bug is sorted too?
<wgrant> StevenK: It probably is, but you need to check that the delegations to SPR's checker work
<wgrant> (by "work" I mean "don't issue tonnes of queries")
<StevenK> wgrant: Hmmm, something has broken the upload webservice tests
<wgrant> Oh?
<StevenK> wgrant: TestPackageUploadWebservice calls factory.makeBuildPackageUpload() (and then creates a BPR, but the factory call does that ...), trying to access upload.builds[0].build leads to an IndexError
<wgrant> StevenK: Sounds like something's creating a PUB without using the method that knows to invalidate stuff
<StevenK> But it uses addBuild, which does the invalidation!
<StevenK>     def addBuild(self, build):
<StevenK>         """See `IPackageUpload`."""
<StevenK>         del get_property_cache(self).builds
<wgrant> It's probably clearing the cache too early
<StevenK> We should delay clearing the property cache?
<wgrant> Clear the cache once the relevant change has been made
<wgrant> Otherwise the cache may be recalculated with the old data
<StevenK> Then all three of the methods are wrong
<wgrant> Lots of code is wrong
<wgrant> Most code is wrong
<StevenK> Oh, they used to del get_property_cache, return, the searchable_{name,version}s stuff is now in the middle
<StevenK> wgrant: I moved the del to just before the return, but it still dies. Guess I need to create the PUB, delete the property cache and then return
<wgrant> StevenK: Right, that's what I meant.
<wgrant> It's not necessarily safe to clear the cache before you've made the change
<wgrant> Because some operation (eg. accessing a protected attribute) may cause the cache to be recalculated
<wgrant> At any time
<StevenK>     print upload.builds[0].build
<StevenK> IndexError: list index out of range
<StevenK> :-(
<StevenK> Do not get it. We create the upload in the factory, create a BPB and a BPR, add it to the PU, where we destroy the cached property, and then return it, and the test says upload.builds is []
<wgrant> Give me a branch :)
<wgrant> Or diff
<StevenK> wgrant: http://pastebin.ubuntu.com/1453948/
<wgrant> StevenK: Looking
<wgrant> Hm
<wgrant> StevenK: I wonder if it's not flushing properly
<StevenK> wgrant: I was thinking about that -- something that is more aggressive about caching for tests
<wgrant> An explicit flush fixes it
<wgrant> Odd
<StevenK> wgrant: As in a store flush?
<wgrant> Yes
<wgrant> It looks like the SQLMultipleJoin isn't causing a flush
<StevenK> Shall I rewrite them into properties that do a IStore.find() ?
<wgrant> No
<StevenK> Since SQLMultipleJoin annoys me
<wgrant> We should work out why it's not working
<wgrant> Oh
<wgrant> of course
<wgrant> checkPermission is wrapped in @block_implicit_flushes
<wgrant> I'd explicitly flush before invalidating the cache in each case
<StevenK> Right
<StevenK> Ah ha, excellent
<StevenK> wgrant: Shall I change that print to a assert just to check that this doesn't bite again?
<wgrant> StevenK: No
<wgrant> Because it will crash if it fails
<wgrant> Already
<wgrant> Won't it?
<StevenK> No, because I stopped creating a BPR
<StevenK> And then the method just iterates over builds, which won't care if it's empty
<wgrant> Ah
<StevenK> So I can revert the change to create another BPR, which I think is silly, but does test that builds is usable, or an assert
<StevenK> Haha, one test depends on the second BPR
<StevenK> Eh, back it goes
<StevenK> wgrant: So you'd like another test to check query counts?
<wgrant> Please
<StevenK> wgrant: For IDistroSeries.getPackageUploads() via the Webservice or what?
<wgrant> That's what I'd do
<wgrant> Given that it's going to be lazr.restful triggering the launchpad.LimitedView checks
<StevenK> wgrant: http://pastebin.ubuntu.com/1453995/
<wgrant> StevenK: What are those two adjacent SPPH queries?
<StevenK> Identical. Odd.
 * StevenK tries to work out the backtrace
<StevenK> wgrant: Do not get it
<StevenK> Also not helped by the backtrace being on one line
<StevenK> Gah
<StevenK> Can't even convert it to something readable
<stub> When does patch-2209-40-3.sql get applied? I merged db-devel when I shouldn't have
<mpt> Whoa, I'd forgotten Launchpad used to have "mentorships" ... That's a blast from the past
#launchpad-dev 2013-12-16
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-twistedjobsource/+merge/199070
<wgrant> StevenK: burn
<StevenK> wgrant: Thanks, burned.
#launchpad-dev 2013-12-17
<stub> wgrant: The large file branch is ready for review. Code changes were minimal to improve the paranoia checks.
<wgrant> stub: Thanks. Will look shortly.
#launchpad-dev 2013-12-19
<xnox> Hm, i seemed to have lost filters on the bug page?
<xnox> e.g. sort by bug #, sort by heat, etc.?
<xnox> oh, reload helped.
<xnox> back to normal.
#launchpad-dev 2013-12-20
<stub> Where does dkimpy come from?
<stub> Got it. My update scripts need to update harder.
#launchpad-dev 2014-12-16
<stub> https://code.launchpad.net/~stub/launchpad/trivial/+merge/244833
<wgrant> stub: That librariangc fix is sufficient to keep it from whining?
<stub> I think so, yes.
<stub> It is the same area we already silence wining about the .pid file which ends up in there
<wgrant> Yep
<wgrant> stub: Couple of comments, but pretty much fine. Thanks.
<stub> Gah, I need a new VM for LP tests
<stub> The problem with writing tests is you find bugs.
<wgrant> Heh
<wgrant> What now?
<stub> Oh, just stupid syntax errors etc.
<stub> quality engineering, thats me
<wgrant> ah yes
<wgrant> The old module that doesn't parse problem.
#launchpad-dev 2014-12-17
<adoniscik> how do you set the name of the package with stdeb if I want to create separate packages for python2 and python3?
<adoniscik> I want X to be renamed python-X and python3-X
<wgrant> adoniscik: I'd examine the manual and/or code of stdeb, and then ask in #ubuntu-packaging if you still can't work it out.
<wgrant> That's not really anything to do with Launchpad.
<adoniscik> okay, I will ask there.
<xnox> adoniscik: export PYBUILD_NAME=X \n %: \n \t dh $@ --with python2,python3 --buildsystem=pybuild
<adoniscik> cool, xnox That looks like it's intended for pybuild. How does it relate to stdeb?
<adoniscik> dput refuses to upload a deleted package that it believes "has already been uploaded"
<wgrant> adoniscik: You need to change the version if you change the package.
<adoniscik> wgrant, I deleted the package instead since it never built properly in hte first place. I don't see the package but dput seems to think it is there. Should I still increment the version?
<wgrant> adoniscik: That dput check is local and can be overridden with -f, but Launchpad will reject a reuse of the same version even if you've deleted it.
<wgrant> You must change the version if you change the package.
<adoniscik> where is that set, Standards-Version?
<wgrant> adoniscik: debian/changelog
<wgrant> And you should always use dch to manipulate that.
<wgrant> It sounds like you should read the Ubuntu packaging guide.
<adoniscik> I edited the changelog and ran debuild again, but launchpad rejected the package because it said the file already exists. I added a letter to the version, which the rejection email indicated in the title, but ignored in the body. Weird...
<wgrant> adoniscik: What exactly did you change it to, and what was the exact text of the error message?
<adoniscik> from "foo (version) dist; urgency=low" to "foo (versiona) dist; urgency=low"
<adoniscik> then I dput the new changes file, which launchpad successfully received. The email noted the new version in the title, but the old file name in the body
<wgrant> Exact text of the error message :)
<wgrant> Was it the orig.tar.gz?
<adoniscik> foo_0.1.2.orig.tar.gz
<wgrant> Also ensure you've read and understood https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version
<wgrant> Right.
<wgrant> " The version number of a package. The format is: [epoch:]upstream_version[-debian_revision] "
<wgrant> The orig.tar.gz is the upstream tarball, and that has the upstream version.
<wgrant> That is anything before the final -.
<wgrant> The orig.tar.gz is meant to be as the name suggests: the original tarball as released by upstream, bit-identical.
<wgrant> How did upstream's release tarball change without changing the version number?
<adoniscik> sorry, I am not sure I follow. I merely tagged an "a" to 0.1.2-1, making it 0.1.2-1a. If I follow you, this does not really constitute a version change, so it rejected it?
<wgrant> It is a version change, but it's not an upstream version change.
<wgrant> And it sounds like you changed the upstream tarball, so you need to change the upstream version.
<wgrant> But how did the upstream tarball change? That normally indicates that something else has gone wrong.
<wgrant> In your new version, upstream_version=0.1.2, debian_revision=1a.
<adoniscik> and the old?
<adoniscik> because I reran debuild after changing the changelog?
<adoniscik> I thought that was how I would reflect the updated changelog
<wgrant> The old version was upstream_version=0.1.2, debian_revision=1
<wgrant> Running debuild won't have changed your orig.tar.gz.
<wgrant> Debian tools don't touch it, normally, since it's meant to be identical to the one you downloaded from the upstream author.
<adoniscik> I have 1 orig.tar.gz, and two debian.tar.gz
<adoniscik> two source.builds
<wgrant> Yes.
<adoniscik> that's good, right?
<wgrant> But if Launchpad rejected it because the orig.tar.gz conflicted, you must have changed the contents of the orig.tar.gz without changing the orig.tar.gz's version.
<adoniscik> well it did say the file exists and "uploaded version has different contents"
<wgrant> The upstream md5sum is 0a9c0986d9b6c1d3f88135afa3da2230
<wgrant> That doesn't match the one you uploaded to Launchpad before.
<wgrant> Does it match the one you have on disk now?
<adoniscik> of the tgz?
<adoniscik> the orig.tgz is b7532f1be6f7e034a9f04a6e47759bbb
<adoniscik> the modified debian.tgz is 99d9aa3fc63361a73d517f1a63e61b05
<wgrant> adoniscik: That doesn't match either the one you uploaded to Launchpad or the one on PyPI. How did you construct the two orig.tar.gzs?
<adoniscik> i downloaded it from pypi, extracted it, ran "python setup.py --command-packages=stdeb.command sdist_dsc", edited the rules file to use pybuild, etc.
<adoniscik> sorry for the trouble, I am trying to upload my first package :)
<wgrant> I've not used stdeb.
<wgrant> But if it generates an orig.tar.gz itself then that's pretty weird.
<wgrant> In future, use the upstream-provided tar.gz instead.
<adoniscik> I also used debuild
<adoniscik> you mean from pypi?
<wgrant> Yes.
<wgrant> In this case you'll still have to change the version, since the real upstream version's filename had a bad tarball behind it.
<wgrant> So, I would grab the orig.tar.gz from pypi, and name it something like FOO_1.2.3+upstream1.orig.tar.gz
<wgrant> Then add a new changelog entry for version 1.2.3+upstream1-1
<wgrant> (the "+upstream1" is just an arbitrary string to make the version larger than the old one, while also suggesting what happened)
<adoniscik> I didn't know I could directly upload those archives. so I just need to put a copy of the tgz from pypi and rename it to foo.orig.tar.gz (i.e., add the orig)
<wgrant> When 1.2.4 is released, ensure that you use the orig.tar.gz from PyPI instead of anything generated by stdeb.
<wgrant> A normal Debian source package is composed of an orig.tar.gz, and debian.tar.gz, and a dsc.
<wgrant> The orig.tar.gz is traditionally exactly the upstream release tarball.
<wgrant> THe debian.tar.gz just contains the debian/ directory of your source package. debuild (well, dpkg-buildpackage, called by debuild) generates it by tarring up the debian/ directory.
<wgrant> The dsc just provides metadata and the hashes of the orig.tar.gz and debian.tar.gz.
<wgrant> You generate none of those files yourself, and debuild generates all except the orig.tar.gz.
<wgrant> debuild works out which orig.tar.gz to use based on the latest version in debian/changelog.
<wgrant> So if your changelog starts with "foo (1.0) trusty; urgency=low", it'll use foo_1.0.orig.tar.gz in the parent directory.
<adoniscik> cheers, I'll try again. most informative.
<wgrant> (note that some older or native Debian source packages have a different set of files, but none you're likely to run into)
<adoniscik> so the package built, but only for python3, presumably since that's what I ran stdeb with. I suppose I need to run it again with python2. If I wanted to create separate packages, should I just edit the changelog to rename foo to python-foo or python3-foo?
<adoniscik> thanks for your patience, wgrant
<wgrant> adoniscik: AFAICR stdeb just generates debian/* from metadata in setup.py. You would normally do that once, trim any unnecessary bits, then edit the autogenerated files to meet your needs.
<wgrant> One of those needs might be support for multiple Python versions.
<wgrant> There's no reason to have a separate source package; it's normal and common for a single source to generate python-foo and python3-foo.
<wgrant> See the example debian/rules that xnox gave earlier, but you'll also need to add a new Package stanza to debian/control.
<adoniscik> oh good. where does debian/control get the contents of ${python:Depends}? I created a new stanza renaming the package and ${python3:Depends}
<wgrant> adoniscik: https://wiki.debian.org/Python/Packaging is probably of interest.
<wgrant> But ${python:Depends} is a "substvar". Check the dh_python2 manpage for details of the substvars it provides.
#launchpad-dev 2015-12-14
<xnox> cjwatson, wgrant - could you check what's up with https://launchpad.net/ubuntu/+source/elixir-lang/1.1.0~0.20150708-1/+build/8373458 ? i presume it's hung, it should build in like 4h, not 4 days...
<xnox> run away process or some such?
<wgrant> xnox: IIRC it hangs on lots of architectures, but let's see.
<wgrant> buildd   59940  0.0  0.0   4972  2056 ?        S    Dec11   0:00 epmd
 * xnox ponders if at the end of binary it should try to killall epmd ;-)
<wgrant> xnox: The build has now finished.
<xnox> cool, thanks
<xnox> https://bugs.launchpad.net/ubuntu/+source/elixir-lang/+bug/1525734
<mup> Bug #1525734: builds hangs sometimes <elixir-lang (Ubuntu):New> <https://launchpad.net/bugs/1525734>
#launchpad-dev 2015-12-16
<wgrant> cjwatson: I guess those PostgresConnection leaks aren't a coincidence :/
<cjwatson> wgrant: Yeah, although if I'm not mistaken I was seeing them before my psycopg2 upgrade as well
<cjwatson> wgrant: Indeed, builds 897 and 898 both have a bunch
<cjwatson> r17870 and r17871 respectively
<wgrant> Yeah, they happened occasionally.
<wgrant> But two in a row straight after an upgrade is slightly worrying.
<wgrant> Hopefully just a coincidence.
 * cjwatson tries forcing once more for luck
<cjwatson> LayerIsolationError: Test left uncollectable garbage
<cjwatson> [<storm.databases.postgres.PostgresConnection object at 0x15d376ec>] (referenced from [(<storm.databases.postgres.PostgresConnection object at 0x15d376ec>,), [<storm.databases.postgres.PostgresConnection object at 0x15d376ec>], {'_connection': <storm.databases.postgres.PostgresConnection object at 0x15d376ec>, '_order': {}, '_dirty': {}, '_implicit_flush_block_count': 0, '_sequence': 0, '_event': <storm.variables.EventSystem object at ...
<cjwatson> isn't that saying that it's referenced from itself?
<cjwatson> ... 0x15d3790c>, '__provides__': <zope.interface.Provides object at 0x1132a30c>, '_lp_store_initialized': True, '_register_for_txn': False, '_cache': <storm.cache.GenerationalCache object at 0xe000d6c>, '_database': <lp.services.webapp.adapter.LaunchpadDatabase object at 0x1132e74c>, '_alive': <WeakValueDictionary at 391910156>}])
<wgrant> cjwatson: Referenced from a tuple, a list, and a dict.
<cjwatson> oh right
<cjwatson> wgrant: no luck reproducing this locally so far
<wgrant> cjwatson: Hm, you've run a big chunk of the tests?
<cjwatson> I tried running one test that happened to fail a couple of hundred times, and tried running lp.blueprints.browser.  I'll try a bigger chunk
<cjwatson> gc.get_referents(*gc.garbage) is returning [{'_closed': True, '_database': <lp.services.webapp.adapter.LaunchpadDatabase object at 0x1107174c>, '_event': <storm.variables.EventSystem object at 0x172c95ac>, '_raw_co
<cjwatson> nnection': None}, <class 'storm.databases.postgres.PostgresConnection'>]
<cjwatson> so the cycle could either be via an event hook, or something in LaunchpadDatabase (but neither that nor any of its superclasses hold any non-trivial references that I can see)
<cjwatson> ... ok, that did reproduce locally, once in the entire test suite.  now to try the whole thing again with -D :-/
<cjwatson> (and hopefully that won't perturb it away)
<wgrant> cjwatson: Ah, I apparently actually dug into a reproducible version of this in my Product.access_policies changes a few months ago. A MutableValueVariable held a reference through _event_system, but I changed the approach used in the code rather than debugging all the way.
<cjwatson> Curious, since MVV's hooks don't seem to do anything that should produce circularity
<cjwatson> I mean, it only adds methods with zero args as hooks
<cjwatson> At some point I may manage to get lucky and catch this in pdb.
<wgrant> cjwatson: If you want to bet that the two PostgresConnection leaks aren't unrelated, it may be worth reinstating my broken garbo job and poking at that test.
<wgrant> "â product-aps-set broke buildbot spectacularly.
<wgrant> â£ garbage: {{{[<storm.databases.postgres.PostgresConnection object at 0x15d673ec>]}}}
<wgrant> "
<wgrant> cjwatson: http://paste.ubuntu.com/14057795/ reproduced it consistently.
<cjwatson> OK.  I'm running --layer=DatabaseFunctionalLayer at the moment with pdb.set_trace() inserted just before the exception; if that doesn't catch it (or maybe even if it does) then I'll try that.
<wgrant> Specifically setting the access_policies list.
<cjwatson> Though it would certainly be nice to have a shorter and more reliable reproducer, so yes ...
<cjwatson> Won't get to it before tomorrow morning though
<wgrant> Well, the problem is that we don't know if they're actually related.
<cjwatson> Indeed.
<wgrant> And we won't know until the garbo one is fixed and buildbot passes :/
<wgrant> Feel free to hand over any notes you have and I'll continue poking.
<cjwatson> I don't yet have meaningful notes other than what I've mentioned here, since my reproducer was so long and unreliable.
<cjwatson> Just running with http://paste.ubuntu.com/14057865/
<cjwatson> (But I'm sure you could have figured that much out)
<cjwatson> I've been multitasking with working on getting my git-build-recipe tests passing
<cjwatson> Currently at 119/135 ...
<wgrant> cjwatson: 119/135 tests complete, or 119/135 runs succeeded/
<cjwatson> wgrant: 119/135 tests succeeded
<cjwatson>  lp.registry.tests.test_distro_webservice.TestGetBranchTips.test_same_results> /home/cjwatson/src/canonical/launchpad/lp-branches/more-layer-isolation-info/lib/lp/testing/layers.py(473)testTearDown()
<cjwatson> -> BaseLayer.flagTestIsolationFailure(
<cjwatson> oho
<wgrant> really
<wgrant> It's an interesting test to break on. It's a slightly weird method.
<cjwatson> Hesitant to ascribe too much significance to the particular test.
<wgrant> Indeed.
<cjwatson> (Pdb) p obj['_event']._hooks
<cjwatson> {'register-transaction': set([]), 'flush': set([(<bound method JSONVariable._detect_changes of <storm.variables.JSONVariable object at 0x182407ec>>, ())])}
<wgrant> Oho
<wgrant> Very relevant, then.
<cjwatson> So *could* be, but how's that circular
<cjwatson> I guess that bound method refers to that variable
<wgrant> Yep.
<wgrant> Bound methods hold a reference to their object, funnily enough.
<wgrant> But I'm not sure why this wouldn't happen all the time, unless it sometimes gets unregistered.
<cjwatson> (Pdb) gc.get_referents(meth.im_self)
<cjwatson> [<storm.variables.EventSystem object at 0x1b102f0c>, <class 'storm.variables.JSONVariable'>, {u'include_long_descriptions': True, u'backports_not_automatic': False}, Undef, (Undef, u'{"include_long_descriptions": true, "backports_not_automatic": false}'), <storm.properties.PropertyColumn object at 0xdcb19cc>, <storm.variables.EventSystem object at 0x1e15914c>]
<cjwatson> So I guess my publisher_options change could have tickled it?
<cjwatson> Maybe made it more probable, at least, since DistroSeries is everywhere
<wgrant> It is not impossible.
<wgrant> I did think about that when reviewing it, but buildbot seemed happy...
<cjwatson> The only things with __del__ are Result and Connection, unless there's something inside psycopg2
<wgrant> Right, but it's probably something like PostgresConnection holding a reference to the event_system so it can fire events on commit or similar.
<cjwatson> There'd have to be a reference back to the connection somewhere for it to be relevantly cyclic, unless I misunderstand the gc
<wgrant> Indeed, but the variable or event_system probably knows the store which knows the connection, or something like that.
<cjwatson> ah
<cjwatson> (Pdb) gc.get_referents(meth.im_self)[6]._hooks
<cjwatson> {'stop-tracking-changes': set([(<bound method JSONVariable._stop_tracking of <storm.variables.JSONVariable object at 0x182407ec>>, ())]), 'changed': set([(<bound method Store._variable_changed of <storm.store.Store object at 0x1b10294c>>, ()), (<bound method Relation._break_on_local_diverged of <storm.references.Relation object at 0x15e3b66c>>, ({<storm.references.Relation object at 0x126da46c>: {}, 'sequence': 1, 'primary_vars': ...
<cjwatson> ... (<storm.variables.IntVariable object at 0x19a5487c>,), <storm.references.Relation object at 0x1237eacc>: {}, 'invalidated': True, <storm.references.Relation object at 0x126da74c>: {}, 'store': <storm.store.Store object at 0x1b10294c>, <storm.references.Relation object at 0x121ecb8c>: {}},))]), 'start-tracking-changes': set([(<bound method JSONVariable._start_tracking of <storm.variables.JSONVariable object at 0x182407ec>>, ())]), ...
<cjwatson> ... 'object-deleted': set([(<bound method JSONVariable._detect_changes_and_stop of <storm.variables.JSONVariable object at 0x182407ec>>, ())]), 'resolve-lazy-value': set([(<bound method Store._resolve_lazy_value of <storm.store.Store object at 0x1b10294c>>, ())]), 'flushed': set([])}
<wgrant> Right.
<cjwatson> there's the store
<cjwatson> so these must normally be unhooked or nothing would work
<wgrant> Yeah, maybe _stop_tracking isn't being called somewhere along the line.
 * wgrant tries to revive the reproducible case.
<wgrant> Oh I have vague memories of this.
<wgrant> I could only reproduce it with a reset in some circumstances, I think.
<cjwatson> I'm a bit suspicious of the object-deleted implementation in the C extensions.
<cjwatson> That's at least sometimes how _stop_tracking is called.
<wgrant> Confirmed that r17587 plus that test I pasted reliably generates garbage
<cjwatson> This JSONVariable is only referred to by its bound methods
<wgrant> Yep
<cjwatson> I wonder if it's possible to end up with the same hook attached multiple times?
<cjwatson> Via multiple MutableValueVariable.get calls
<wgrant> It indeed seems to hook twice, hmmm.
<wgrant> But then my unhook log isn't triggered at all.
<wgrant> Oh, probably unhooked once in set, which I didn't log.
<wgrant> cjwatson: Oh, it's a set anyway.
<wgrant> Unhook is never called AFAICS.
<cjwatson> It's not called via object-deleted -> _detect_changes_and_stop -> _stop_tracking?
<wgrant> Nope.
<wgrant> Logging more and tweaking the test to not leave garbage, to see how it's called normally.
<wgrant> It is possible, of course, that it only leaks on reset.
<wgrant> And it depends on test ordering.
<cjwatson> reset (or invalidate) could cause multiple gets by clearing the cache
<wgrant> Interesting.
<wgrant> If I remove the reset at the end of the test, it actually gets unhooked by a new set() call.
<wgrant> I guess it gets set to AutoReload when the transaction is aborted at the end of the test.
#launchpad-dev 2015-12-17
<wgrant> That is the case.
<wgrant> So that makes complete sense.
<wgrant> The question is whether your case is the same.
<wgrant> It's the same cycle, but is it also caused by the object getting lost due to the reset.
<cjwatson> Does http://paste.ubuntu.com/14059967/ help?
<wgrant> It's a set.
<wgrant> It certainly shouldn't.
<cjwatson> Oh, so it is
<wgrant> In the garbage runs there is no case in which unhook is called from within MutableValueVariable.
<wgrant> Even if it were only added once, it would still leak.
 * wgrant looks for any reset hooks.
<wgrant> Ah
<wgrant> There are none.
<cjwatson> invalidate probably won't cause this, since it does _mark_autoreload
<wgrant> Right, exactly.
<wgrant> abort does invalid, which does AutoReload.
<wgrant> Which is why this normally works.
<wgrant> We reset between tests, but normally abort or commit first.
<wgrant> Look at Store.reset -- it just clears the dicts.
<cjwatson> Yeah
<cjwatson> With a "beware" comment at the top
<wgrant> Hm, but the object itself is presumably destroyed.
<wgrant> Its destructor could tell its variables to GTFO, presumably.
<wgrant> But ew __del__
<wgrant> I wonder how slow it would be to fire hooks on them all as they are reset...
<cjwatson> I was wondering whether the tp_traverse implementation might be incomplete
<wgrant> Hmm.
<wgrant> Isn't the cycle legitimate?
<wgrant> Or does PostgresConnection not refer back to the event_system?
<wgrant> What would the tp_traverse bug be?
<cjwatson> Missing MutableVariableValue._event_system, possibly.  Just trying without C extensions to narrow it down
<wgrant> Ah, possibly.
<cjwatson> OK, still garbage from that, so probably a red herring.
<wgrant> (Pdb) p gc.get_referents(gc.garbage[0])[0]['_event'] is gc.get_referrers(gc.garbage[0])[1]['_event']
<wgrant> True
<wgrant> gc.garbage[0] is the PostgresConnection
<wgrant> So looks legit to me.
<wgrant> Unless we can remove a reference or use a weakref, it's uncollectable.
<wgrant> Hm, can we weakref the hook, I wonder.
<wgrant> Presumably if the object is still known by the store, its variable will still be alive.
<wgrant> We only need to detect changes if we're possibly going to flush them to the store, so if we're no longer in the store it's irrelevant.
<wgrant> Right?
<cjwatson> Yeah, and quite a few other hooks are weakreffed
<wgrant> Oh are they
<wgrant> So they are
<cjwatson> Well
<cjwatson> Their args are
<cjwatson> Maybe not a perfect example
<wgrant> Hmmm actually.
<wgrant> This may not be valid.
<wgrant> What if I have the object, query 10 million other objects so it's evicted from Storm's cache, then change the mutable value, delete my reference, then flush.
<cjwatson> It's a little hard to see how to get away without a reset hook.
<wgrant> Heh and even the reset hook isn't sufficient here, I don't think.
<wgrant> Oh, unless it keeps weakrefs to even evicted objects.
<wgrant> In which case it would.
<wgrant> s/unless it/unless the Store/
<cjwatson> Cache eviction could call a hook too.
<cjwatson> But starts to sound rather slow.
<wgrant> It'd also be a weird hook
<wgrant> Basically a "leak pls" hook.
<cjwatson> Or a "make changes not work" hook
<wgrant> Because we can't unhook _detect_changes just because the object is evicted; it may be alive elsewhere.
<cjwatson> Yeah
<wgrant> Ah
<wgrant> _alive is a WeakValueDictionary.
<wgrant> _cache is separate.
<wgrant> So a reset hook may work.
<wgrant> It also can't be any slower than an abort.
<cjwatson> I have to go to sleep, so will pick this up in the morning if you haven't figured it out by then ...
<wgrant> Oh of course it needs _alive. Even if it's evicted from the cache it has to use the same object if it's reretrieved by key.
<wgrant> I might end up wrapping our stores, will see.
<wgrant> Or just replacing all our reset calls, short-term.
<cjwatson> The possible slowness would be in having to go and look for hooks on ten million objects.
<wgrant> Right, but abort and commit already do that.
<wgrant> And we already do that for every test.
<cjwatson> Outlawing reset would probably be a reasonable stopgap.
<wgrant> It's terrible, but already done.
<cjwatson> Well, outlawing it without an immediately previous abort/commit
<cjwatson> There aren't too many of those
<cjwatson> Anyway, sleeeeep
<wgrant> Yup,.
<wgrant> Night, thanks for investigating.
<wgrant> Trying to work out why I need that Milestone construction in there.
<wgrant> The test doesn't leave garbage unless I've created something else referring to that object since the last commit.
<wgrant> http://pastebin.ubuntu.com/14060983/ on top of trunk is minimal so far.
<wgrant> cjwatson: I suspect that http://paste.ubuntu.com/14071248/ will be sufficient.
<wgrant> It turns out that object-deleted often doesn't get called at all, because it's only invoked by a weakref callback.
<wgrant> And the objects can sometimes outlive the weakref callbacks.
<wgrant> s/weakref callbacks/weakrefs/
<wgrant> (obviously the callback would never be invoked anyway in the uncollectable circular case, but removing the hooks also seems to eliminate at least one case of uncollectability. so hopefully it's enough)
<cjwatson> wgrant: I'm trying a full test run with that patch now.  Shall I land it if it checks out?
<wgrant> cjwatson: Or even before it checks out, if you're so inclined.
<wgrant> The Storm test suite passes with it.
<wgrant> And it's not obviously terribly wrong.
<cjwatson> Can you commit it to our Storm branch and I'll sort out an sdist build and an LP landing?
<cjwatson> I agree it seems at least plausible.
<wgrant> We may need to detach a couple of extra hooks in the deletion hook, but this fixes at least one case.
<wgrant> If it goes well I'll also merge storm trunk tomorrow, as it fixes that restricted librarian "Processing failed" thing that we run into occasionally.
<cjwatson> Oh?  What was the change there?
<cjwatson> r466?
<wgrant> If that's the thing about making "cannot send data" ish a DisconnectionError, yes.
<cjwatson> Yeah, that
<wgrant> Pushed up to revision 408.
<wgrant> 3 tags updated.
<wgrant> oh no bzr what did you do
<cjwatson> wgrant: http://paste.ubuntu.com/14072599/ if that helps
<wgrant> I was hoping you'd say that.
<wgrant> 0.20                 free.ekanayaka@canonical.com-20130628093022-ztq82k68kepzznxx
<cjwatson> Be glad I didn't reflexively "bzr up" first.
<wgrant> And two lds tags
<wgrant> Hm.
<wgrant> I wonder if I tried a merge into it at some point and it remembered the tags.
<wgrant> Harmless, anyway.
<cjwatson> Presumably those are ghosts with respect to our branch, but in the same repository.
<cjwatson> Total: 19377 tests, 0 failures, 0 errors in 195 minutes 26.711 seconds.
<cjwatson> good sign
<cjwatson> Yay, buildbot happy again.
<cjwatson> wgrant: thanks!
<wgrant> Oh good, buildbot works.
<cjwatson> Something has to (probably).
<wgrant> Indeed...
#launchpad-dev 2015-12-18
 * cjwatson finishes the year by pushing an initial version of lp:git-build-recipe (in git, of course) that almost almost works
<cjwatson> The main bit that's still broken is nest-part
<cjwatson> But most other things are at least minimally functional and it passes all non-skipped tests
<jblakeman> Hi, new to Launchpad dev and looking to submit a patch for the API. Specifically, I'd like to make the methods `getSummarizedMirroredArchSeries` and `getSummarizedMirroredSourceSeries` public, but am having trouble finding where public methods are defined in the source. Relevant question: https://answers.launchpad.net/launchpad/+question/274646.
<cjwatson> jblakeman: methods with webservice exports have @export_read_operation etc. decorators on their declarations in lib/lp/*/interfaces/
<cjwatson> there are quite a few examples, so once you know that pattern you can at least have a go at cargo-culting; the build system is pretty strict about making sure you have the right type declarations in place
<jblakeman> cjwatson: Great, thanks for the tip! I'll give it a shot.
<cjwatson> I haven't looked at the relevant methods to see if there would be any gotchas here.  Write unit tests, it's the easiest way to make sure you have it working :)
<beuno> ...and getting the merge proposal approved!
<cjwatson> well yes, but it saves time up front too :)
<beuno> yes yes!
<cjwatson> I only actually run the webapp for something like 1/10 of the branches I submit (the hard ones)
<cjwatson> so it's yak-shaving you can often skip
 * beuno nods and steps away from the hornets nest
<jblakeman> Well will this will be a good exercise for me as I'm pretty new to writing software (hobby turning to career) and have never contributed to a major open source project before.  I'll try to avoid asking a bunch of noob questions.  Wish me luck ;)
<cjwatson> :)
<cjwatson> I'm on vacation now, but will see questions eventually
#launchpad-dev 2016-12-21
<sil2100> Hey guys! How can I, using LP API, check for packages in the UAPPROVED queue for a series?
<sil2100> ...nevermind, found it!
#launchpad-dev 2017-12-19
 * cjwatson crosses fingers and sends virtualenv-pip to PQM
<cjwatson> Also have now diverged lp-source-dependencies git, so make sure to have switched to that
<cjwatson> first buildbot run with pip predictably taking forever ...
<cjwatson> (it's populating its wheel cache)
<cjwatson> ... well, it built, at least ...
 * wgrant finds the fireworks
<wgrant> oh no fireworks yet, sad
<cjwatson> I don't think I've seen that lp.codehosting.codeimport.tests.test_worker.TestBzrSvnImport.test_stacked error before
<wgrant> Indeed
<wgrant> It's a bit odd.
<cjwatson> not obviously anything to do with pip, but ...
<wgrant> But I built the tree in a couple of containers and ran the full test suite a few times locally
<wgrant> And I assume you did too
<cjwatson> Yeah
<cjwatson> Could be something wrong with subvertpy, I suppose, but not sure why it'd only show on buildbot
<cjwatson> hmm, what, this test fails in isolation locally too
<cjwatson> maybe it has an implicit dependency on something else having run first?
<cjwatson> yeah, if I run all of TestBzrSvnImport, it passes; if I run TestBzrSvnImport.test_stacked alone, it fails
<wgrant> Aha
<cjwatson> at least that's the less hideously painful way round for test isolation bugs to manifest
<cjwatson> reminds me slightly of the bzrlib.plugins.git.cache thing
<cjwatson> wgrant: This is actually much more interesting than expected
<wgrant> cjwatson: oh ho?
<cjwatson> Tracing it through, lp.codehosting is being imported from sitecustomize, which is apparently early enough that sys.getfilesystemencoding returns None, which causes bzrlib.i18n.load_plugin_translations to fail
<cjwatson> So the svn plugin doesn't necessarily get correctly installed
<wgrant> And the test requires it be in Swahili?
<wgrant> Or does the load_plugin_translations failure actually break more interesting things silently?
<cjwatson> It causes bzrplugins/svn/__init__.py to silently fail part-way through
<wgrant> Ah, helpful.
<cjwatson> Specifically, before it gets round to installing its prober
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/avoid-importing-bzr-plugins-from-site/+merge/335379 ?
<cjwatson> passes that individual test, that test suite, and all of lp.code.model.tests
<cjwatson> I haven't done a full run but it should be OK
<cjwatson> Hmm, idle.  I think I'll self-review and go ahead with that; the test suite should catch anything overly interesting here
<cjwatson> subsequent pip runs on buildbot indeed quite a bit faster
<cjwatson> Still not quite right - need to adjust how bzr subprocesses are run, I think.
<cjwatson> OK, I think it just needed a subvertpy wheel built against precise (ugh, but the proper fix isn't in a released version of subvertpy yet)
<cjwatson> yay, it passed
<cjwatson> ... and qastaging has updated successfully
<wgrant> Nice
#launchpad-dev 2017-12-20
<cjwatson> I've tracked the mysterious occasional test failures with Twisted 16.5.0 down as far as a 1/256-probability crypto implementation bug in paramiko < 2.0.0.  I think I can work around it in lazr.sshserver.
#launchpad-dev 2017-12-21
<cjwatson> wgrant: If you're still around, could you look at https://code.launchpad.net/~cjwatson/lazr.sshserver/work-around-short-paramiko-signatures/+merge/335515 ?
<wgrant> cjwatson: Upgrading paramiko isn't easy, I guess?
<cjwatson> wgrant: xenial has an older version
<cjwatson> wgrant: while we could probably SRU clients, it'd take a while, and I'm sure there are some non-Ubuntu ones
<wgrant> cjwatson: Oh, a client bug, I see.
<wgrant> Er, of course, lazr.sshserver doesn't use paramiko
<cjwatson> We should probably upgrade our in-tree version for LP tests as well, but I considered that to just be papering over the problem
<wgrant> cjwatson: Sounds reasonable, then. r=me
<cjwatson> Thanks
<cjwatson> Also what an absolute pig to find - the tests eat server-side logs, so I had to resort to stracing test runs to pick out inserted log messages
<wgrant> Yeah, nice find
<cjwatson> (Arguably should fix in Twisted; not sure)
<cjwatson> oh good grief, carob is failing to build the current tree because it's picking testtools from /usr/lib/python2.7/dist-packages/ when building extras rather than the one in the tree
<cjwatson> system-site-packages, not even once
<cjwatson> (this is ultimately why bazaar.staging is broken at the moment)
