#launchpad-dev 2010-03-01
<jelmer> mwhudson, awesome
<jelmer> mwhudson, yes, the fix for the symlink bit has landed
<jelmer> mwhudson: so the newer dulwich and bzr-git are confirmed working for the kernelk?
<mwhudson> jelmer: no, i haven't tried the kernel yet
<mwhudson> i was hoping spm was going to be working today
<jelmer> mwhudson: have you tried gedit?
<mwhudson> jelmer: yes, locally, and it worked
<jelmer> mwhudson: did you pull from a local git repo or a remote one?
<mwhudson> so i'm reasonably confident :)
* thumper changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | Channel logs: http://irclogs.ubuntu
<mwhudson> jelmer: remote
<jelmer> mwhudson: cool
<mwhudson> i'll try the kernel overnight perhaps, if i don't get it set up on staging
<jelmer> I'm also trying the kernel here locally
<jelmer> it'd be great to have staging trying it, I think it should take 3 days or so to do the actual import
<thumper> jelmer: are you looking into reusing the pack file?
<mwhudson> jelmer: trying from a local repo or remote?
<jelmer> thumper: I started in Wellington but haven't looked at it since.
<jelmer> mwhudson: remote
 * thumper afk for a bit again
<mwhudson> jelmer: ah, ok
<mwhudson> jelmer: how many revs through now?
<jelmer> 3300
<jelmer> but I had 5k earlier today, restarted after I made that encoding fix
<mwhudson> jelmer: how many revs are you pulling in each go?
<jelmer> mwhudson, 1k
<mwhudson> jelmer: cool
<jelmer> mwhudson, once the new version is rolled out, it would be nice if all existing failing git imports could be retried because of the symlink issue
<mwhudson> jelmer: git?  or bzr-svn?
<jelmer> mwhudson, git particularly, but bzr-svn for an older issue with symlinks
<mwhudson> or both?
<mwhudson> ah ok
<mwhudson> well, if there was an api for code imports...
<mwhudson> i guess we really need to look at fixing that
<jelmer> if they were more like mirrors you could just call requestMirror() I guess >-)
<mwhudson> well boringly that does something rather different for imports...
 * thumper pokes jelmer in the eye
<mwhudson> (and the difference is now important, to avoid exposing users to partially imported incremental imports)
<thumper> mwhudson: we don't really need to QA the removal of the old puller code do we?
<thumper> mwhudson: as it isn't being used
<thumper> mwhudson: I maked it ?? on the wiki page
<thumper> mwhudson: so we should just move it to done-done on the kanban board, right?
<mwhudson> thumper: no, not really, i guess it's just "make sure codehosting basically works"
<thumper> mwhudson: I've pushed branches
<thumper> mwhudson: they have been pulled and scanned
<mwhudson> which it doesn't at the moment, but hopefully that's just apache needing to be gracefulled
<mwhudson> thumper: ok, that sounds good enough for this item
<thumper> I'll move it if you like
<mwhudson> i just did
<thumper> have you tried the reduced concurrancy?
<mwhudson> yes, but it also includes a command line arg to specify the limit from the slave
<mwhudson> that hasn't been tried on staging
<thumper> ok
<thumper> so need a losa then?
<mwhudson> similar for the "pull mirrored branches in a separate process" item
<mwhudson> thumper: yeah
<mwhudson> ha
<mwhudson> ha
<mwhudson> assertSqlAttributeEqualsDate is broken
<mwhudson> never fails
<thumper> haha
<thumper> oops
<mwhudson> will make ec2testing this branch a little more interesting
<thumper> mwhudson: I'm off for the afternoon, back later tonight to talk to europeans
<mwhudson> thumper: ok
<deryck> thumper, you around?
<mwhudson> deryck: he'll be back in a couple of hours i think
<deryck> mwhudson, ah, ok.  thanks.  too late for me then.
<mwhudson> deryck: send email, i guess
<mwhudson> (is this about the max_bug_heat thing?)
<deryck> mwhudson, yeah.  wondering if it was still unresolved.
<mwhudson> deryck: it is still unresolved
<deryck> mwhudson, ok, crap.
<mwhudson> deryck: noodles said "this needs a bug person to sort out" and i believed him
<mwhudson> deryck: if you provide instructions i'm happy to execute them :-)
<deryck> mwhudson, yeah, no worries.  I was just out all day with family and just now seeing email
<mwhudson> deryck: fair enough!
<deryck> mwhudson, let me look at the failure and see.  It may be simple enough.
<deryck> mwhudson, how do you feel about this?  http://pastebin.ubuntu.com/386061/
<deryck> mwhudson, I ask because I'm tired.  IPerson will never need or use max_bug_heat/
<deryck> and the failing test will pass with this.
<mwhudson> deryck: well, it looks like it will fix the test failures
<mwhudson> deryck: seems a bit of a crappy solution though
<deryck> mwhudson, yeah, that's my feeling, too. :(
<mwhudson> is max_bug_heat perhaps on the wrong interface?
<deryck> mwhudson, I don't think so.  It really needs to be on everything that is an IHasBugs except IPerson.
<deryck> mwhudson, really, there's no harm in adding to IPerson.  But it's a DB patch required.
<mwhudson> i guess given the timing of the situation, a quick fix is probably a good idea
<mwhudson> so um yeah
<mwhudson> seems ok to me
<mwhudson> bung it in a merge proposal and get thumper to stamp it?
<deryck> mwhudson, ok, I will.  And I'll do an XXX that we need a proper DB patch later.  Seem fair?
<mwhudson> deryck: sounds right to me
<deryck> mwhudson, cool.  thanks for the chat about it.
<mwhudson> deryck: np
<lifeless> deryck: I'm curious why not IPeople
<deryck> lifeless, is there such a thing as IPeople?
<lifeless> sorry, IPerson
<deryck> lifeless, oh, you mean, why not have max_bug_heat for it at all?  max_bug_heat is for a pillar specific view of hot bugs.  I guess I'm not sure what a person hot bugs list looks like.
<deryck> max_bug_heat controls the distribution of the flames.
<lifeless> well, what does a projects hot list look like?
<lifeless> I assume its how many 'active' indicators the bugs have, out of the 'set of bugs for this project'
<lifeless> Seems to me that the same metric applied to a persons bugs would give a non-rubbish result
<deryck> lifeless, yeah, you may be right.  We just have done any work yet on using hot bugs for people. Maybe we should.
<deryck> person bugs just seems like a hazy grab-bag term of different kinds of bugs related to me, whereas project bugs makes sense.  So where to hang heat for a person?
<deryck> related bugs?  Assigned bugs?  commented?  Everywhere? :-)
<lifeless> yeah, I can see the impact
<deryck> mwhudson, I need to bail now, due to short time for sleep before work. :)  Can you or thumper land that branch for me if he signs off on it?
<mwhudson> deryck: sure
<deryck> mwhudson, many thanks.  Sorry the failure blocked work for so many hours today.
<mwhudson> deryck: not having any sysadmins around has been more obstructive, don't worry ;-)
<deryck> heh, fair enough
<deryck> cheers.  have a nice one!
<wgrant> mwhudson: They're not back in sane timezones yet?
<mwhudson> wgrant: travelling today i think
<wgrant> Ah.
<mwhudson> or maybe unconscious with jetlag
<wgrant> Heh.
 * mwhudson winds down for the day
<poolie1> hi
<poolie1> can someone remind me where the tuolumne config is?
<poolie1> nm, i found it in https://code.edge.launchpad.net/~canonical-losas/tuolumne-lp-configs/trunk   i think
<thumper> gah
<thumper> trying to land deryck's branch
<thumper> but ec2 land isn't working
 * thumper makes a clean build
<thumper> can anyone else get ec2 land working?
<thumper> even with a --dry-run?
<wgrant> thumper: It fails with a launchpadlib NotImplementedError for me.
<wgrant> wadllib, even.
<thumper> wgrant: thanks, me too
<wgrant> thumper: Possibly related to the hard-coded /beta in devscripts.autoland, when the default version on edge is now 'devel'?
<thumper> wgrant: ah, perhaps
<wgrant> It should be fine, but it's the first thing that comes to mind.
<wgrant> Oh, hmm, it doesn't hardcode it for edge, only dev and lpnet.
<thumper> any other ideas?
<wgrant> thumper: Which version of launchpadlib are you using?
<wgrant> Yeah, so, it's requesting a /devel URL for me.
<wgrant> ANd then complains when it gets a /beta resource type.
<thumper> I'm using the one in the tree
<wgrant> thumper: It looks like the served WADL is on crack.
<wgrant> wget --header="Accept: application/vnd.sun.wadl+xml" https://edge.launchpad.net/api/devel/
<wgrant> Has lots of /beta references in it.
<thumper> :(
 * wgrant works out how to invoke the launchpadlib version override.
<wgrant> thumper: add version="beta" lib/devscripts/autoland.py's Launchpad.login_with call, clear your cache, and try again.
<wgrant> And pester leonardr to fix lazr.restful, too!
<thumper> wgrant: where is the cache?
<wgrant> thumper: ~/.launchpadlib/cache, because you guys seem to hate XDG.
<adeuring> good morning
<thumper> gah
<thumper> commit re fail
<thumper> although I can't see how
<thumper> Commit message [[testfix][release-critical=thumper][r=thumper] Return None for\n\tmax_bug_heat on IPerson until max_bug_heat can be fully added.] does not match commit_re [(?is)^\\s*\\[testfix\\]\\s*\\[(?:release-critical|rs?=[^\\]]+)\\]]
<thumper> can anyone else see it?
<bigjools> thumper: r-c is first IIRC?
<wgrant> Not in that RE.
 * bigjools hates regex
<wgrant> ETOOMANYBACKSLASHES
<thumper> it has testfix first
<bigjools> try it
<thumper> hmm...
<thumper> trying, although it should only succeed if it isn't testing what it says it is testing
<thumper> lifeless: regex help needed
<bigjools> lol
<al-maisan> is that what "special circumstances" is for ;)
<lifeless> thumper: what ?
<lifeless> al-maisan: its for digging out of pits :P
<thumper> lifeless: the commit message fail above
<lifeless> thumper: you have a \n
<lifeless> \t
<lifeless> in the message. Why ?
<thumper> email subject wrapping I believe
<thumper> lifeless: it's been happening for ages
<thumper> lifeless: it is the start of the string that gets me though
<thumper> lifeless: to me it looks like it matches
<thumper> oh ho
<thumper> it is different to the devel one
<lifeless> thumper: no =thumper
<lifeless> [release-critical][rc...
<thumper> lifeless: yeah, I just saw that too
<lifeless> perhaps. ICBW
<lifeless> are you submitting with pqm-submit?
<thumper> ICBW?
<thumper> yes
<noodles775> thumper: bzr log --line --limit 1000 |grep release-critical seems to show that ui=none is required? (or at least, has been used for successful landings in the past)
<thumper> noodles775: not for testfix
<lifeless> thumper: Icould be wrong
<lifeless> I.C.B.W.
<al-maisan> intercontinental ballistic wiki?
<thumper> lifeless: yep, no =thumper
<thumper> lifeless: it is in the queue
<jml> good morning Lanuchpadders
<wgrant> Morning jml.
<al-maisan> hello jml
<bigjools> hey jml
<bigjools> you're not going to be staying in my local pub any more
<bigjools> wgrant: you probably saw that I landed your chroot_url export branch, I don't supose you tested it on edge?
<wgrant> bigjools: I did, and altered the local Soyuz docs to use it.
<wgrant> bigjools: Thanks.
<wgrant> So yes, QA OK.
<bigjools> awesome, thanks
<jml> bigjools, ok
<jml> bigjools, where am I going to be staying
<jml> bigjools, and how do I get there
<jml> bigjools, and what are the dates again?
<bigjools> jml: http://www.sarova.com/abbey/hotel_introduction.asp
<bigjools> the Cheltenham Gold Cup is on at the same time - this was the last hotel with rooms
<lifeless> jml: back in London ?
<bigjools> jml: 17, 18,19th March
<bigjools> jml: room booked 16th, 17th, 18th
<bigjools> jml: you get there on a train from Paddington, which I guess is quite convenient for you :)
<wgrant> bigjools: Any idea why there is a Pending publication (717553) in primary jaunty-release from 2009-09-07?
<bigjools> no idea, but if it's in -release you know why it's not being published
<wgrant> Right, but I have to wonder how it got there.
<bigjools> me too
<bigjools> maybe a copy
<jml> bigjools, thanks.
 * wgrant checks the queue.
<wgrant> Possibly.
<bigjools> jml: I'll email more details this week
<wgrant> Hm, no.
<wgrant> bigjools: Looks like it was an override.
<bigjools> yay :/
<wgrant> It was published in universe 6 months earlier.
<wgrant> And the newer publication is multiverse.
<bigjools> !
<wgrant> Yeah, changeOverride should probably die in that case.
<wgrant> change-override.py doesn't check.
<bigjools> rock and roll
<wgrant> bigjools: Also, seen bug #529710?
<bigjools> maybe when mup decides to show me
<wgrant> It won't.
<bigjools> private
<wgrant> Yes.
<bigjools> wgrant: :(
<wgrant> Oh good, I only rebuilt the test database, not the dev one.
<wgrant> bigjools: Commented.
<bigjools> wgrant: thanks
<wgrant> win 32
<wgrant> Argh.
<wgrant> gmb, bigjools: ~launchpad-security emails would be getting to you through ~launchpad, which has a contact address set.
<bigjools> you'd think :)
<wgrant> (so they'll be on the launchpad-bugs list)
<jml> bigjools, btw, I have no idea how to QA my item on https://dev.launchpad.net/StrategyTeamTestPlans/10.02
<jml> (yes, I know it's too late)
<gmb> Hahaha.
<gmb> bigjools: Did you know about the launchpad-bugs list
<gmb> ?
<bigjools> gmb: maybe
<bigjools> what does it get?
<gmb> bigjools: Well, apparently, all the bugs ever. I'd forgotten it existed.
<wgrant> Everything. Including the security bugs I filed yesterday which somebody might want to notify people about.
<gmb> bigjools: But wgrant is right; your security notification might well be there.
<bigjools> jml: stick it in the ?? section
<bigjools> we're not QA-ing any of those recipe things et
<bigjools> yet
<jml> bigjools, ok, thanks.
<gmb> Thing is, subscribing to it is an exercise in sipping from the firehose.
<bigjools> that's prob why I don't do it
<bigjools> we need a totally separate security-related list
<bigjools> I get too much fucking email already
<wgrant> I think we actually need implicit subscriptions to be fixed.
<wgrant> In that they should apply if the user can otherwise see the bug.
<bigjools> gmb: so, the bugs go to a private list then?
<gmb> bigjools: Looks that way; probably a holdover from the days before a) Open sourcing and b) structural subscriptions
<bigjools> gmb: so I get things straight - what is the notification "path" now?
<gmb> wgrant, bigjools: I believe fixing subscriptions is high on the bugs team list for the next 6-monthly cycle, but I could be wrong about that. You'd have to ask deryck.
<bigjools> ie. teams etc
<deryck> Morning.
<gmb> bigjools: So, it looks like security bugs go to LP Security, of which ~launchpad is a member. ~launchpad has a contact address (to stop us from getting spammed to death), which is launchpad-bugs@lists.ubuntu.com.
<bigjools> ok
<bigjools> so lp-security should have its own contact address that I can subscribe to
<wgrant> Since they've likely not been received by anybody, can somebody please notify relevant people about bug #529348 and bug #529370?
<gmb> bigjools: Yes, it should. That would be the sane solution.
<bigjools> gmb: I'm taking this to the internal list
<gmb> bigjools: Righto.
<bigjools> gmb: thanks for your help
<gmb> np
<thumper> night all
 * thumper -> bed
<bigjools> nn thumper
<bigjools> wgrant: any thoughts on https://bugs.edge.launchpad.net/soyuz/+bug/529950
<mup> Bug #529950: gina fails to update "clxclient" in the LP Debian mirror <Soyuz:New> <https://launchpad.net/bugs/529950>
<wgrant> bigjools: It's valid.
<wgrant> Hm, at least in Packages it can happen now.
<wgrant> Let me check.
<bigjools> wgrant: so is there a definition of which one is the right onw?
 * bigjools 's caffeine stream has too much blood in it, brb
<wgrant> It's probably because dak now has aromic arch-indep domination.
<wgrant> But the opposite of our version.
<wgrant> s/aromic/atomic/
<wgrant> bigjools: Shouldn't you just import both?
<wgrant> (if that's not easy, the latest is most appropriate)
<wgrant> bigjools: It looks like it will be easy enough to fix. Just add an extra level of dicts.
<bigjools> wgrant: yeah
<bigjools> wgrant: so does the spec say the most recent version or does it say the first in the file?
<wgrant> bigjools: For apt, the latest version will win.
<wgrant> But there's no reason we shouldn't import all of them.
<wgrant> (Ignoring pinning, apt will pick the latest version. If there is more than one instance of the latest version, the first entry in sources.list will win.)
<wgrant> bigjools: Native binary indices are unordered -- this appears to be because getBinaryPackagePublishing doesn't have a default order clause, whereas getSourcePackagePublishing does.
<wgrant> This isn't deliberate, I guess?
<bigjools> I highly doubt it
<wgrant> I guessed not, but it seems a bit odd that only one is ordered.
<bigjools> probably either for a test or UI
<wgrant> I guess.
<wgrant> bigjools: Isn't bug #528459 just the NBS case?
<mup> Bug #528459: PPA does not delete old packages after new build <Soyuz:Incomplete> <https://launchpad.net/bugs/528459>
<bigjools> I am asking for an example to double check
<bigjools> it also becomes a test case if there's a problem
<wgrant> True.
<deryck> gmb, there are two bugs in the QA ready lane for filebug -- add an api, and poll for status.
<deryck> gmb, Are these done, or can they be done to free some space on the board?
<gmb> deryck: I am testing both as we speak (well, as soon as a LOSA becomes available for the second one)
<deryck> gmb, excellent, thanks.
* kfogel changed the topic of #launchpad-dev to: Launchpad Development Channel | Launchpad read-only 22.00-23.30 UTC Wed 3 March for 10.02 rollout (http://is.gd/9rUhO) | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | Channel logs: http
<deryck> morning, kfogel :-)
<kfogel> deryck: hey, morning.
<gmb> deryck: Offline blob processing FTW! https://bugs.staging.launchpad.net/malone/+bug/528792
<deryck> gmb, very nice!
<deryck> oh happy fat data day!
<gmb> deryck: And that's all the +filebug stuff QA'd, too. Epic win.
<deryck> gmb, very epic.  A good start to our Monday.
<gmb> deryck: Now I'll finish that blog post.
<gmb> Would've finished it Friday but had issues with my dev env. Happily, that's now fixed too.
<mup> Bug #528792: No right-click menu on Notification Area panel item <amd64> <apport-bug> <lucid> <gnome-panel (Ubuntu):New> <https://launchpad.net/bugs/528792>
<kfogel> deryck: I've done everything I can at Gravatar to get my image showing for my canonical email, but the kanban board still doesn't show it (for bug 471195 for example)
<mup> Bug #471195: Private bugs don't set the body class to "private" <css> <privacy> <Launchpad Bugs:Triaged by kfogel> <https://launchpad.net/bugs/471195>
<deryck> kfogel, do you show up on any other gravatar site for your canonical.com addr?  (You can post a test comment on the latest post on my personal site and see.)
<kfogel> deryck: ok, ready for you to mod the comment through
<deryck> kfogel, done.  and avatar'ed.
<kfogel> deryck: avatar is there. how come there's a strikethrough on my name, though?
<kfogel> (just curious -- doesn't matter for kanban purposes)
<deryck> kfogel, strikethough is my visited link convention.  not a great one, I'll admit.
<kfogel> deryck: it makes me feel... struck through :-).
<deryck> kfogel, heh.
<deryck> yeah, I should change it.  But I don't love on my site like I should.
<deryck> kfogel, so I hard reloaded the kanban board and you show up there for me now.
 * deryck stares at adeuring and thinks "gravatar," making an I-am-telepathic face
<adeuring> argghhh
<kfogel> deryck: hunh.  I was doing hard reloads too, and yet it wasn't showing up.  Now I'd test again, but my Firefox has again jammed up, as it does approximately every 12-18 hours.
<adeuring> yeah, i suck with this stuff
<deryck> kfogel, no worries.  Maybe some delay on gravatar's cache.  You want me to delete your comment from my site, or let it stand?
<kfogel> deryck: rebooting, just to see if that shakes the dust out.  enough is enough.  bbiab.
<kfogel> deryck: oh, delete it I guess.  My parents will do a google search and then worry about my weight :-).
<deryck> heh
<deryck> kfogel, I've lost 3 pounds since that post. :-)
<kfogel> deryck: man, writing it must have been hard work!
<kfogel> :-)
<deryck> heh
<kfogel> deryck: gravatar showing up for me in Chrome.  Go figure.
<deryck> kfogel, weird.
<deryck> adeuring, allenap, gmb, intellectronica, kfogel -- standup in two?
<adeuring> yes
<gmb> Yarp
<intellectronica> yes
<kfogel> deryck: ready
<leonardr> when i try to buildout launchpad i'm getting a ClientCookie error. i've gotten this error before but i don't remember what to do about it
<leonardr> the first error i get is that launchpad tries to download a setuptools egg and is ***BLOCKED***, which is also familiar
<leonardr> my download-cache and eggs symlinks point to the directories i'd expect
<leonardr> flacoste, maybe you can end my torment? -^
<flacoste> leonardr: do you have pastebin of the whole buildout run?
<leonardr> flacoste: i'll paste it now
<leonardr> flacoste: http://paste.ubuntu.com/386348/
<leonardr> there's bootstrap.py and buildout
<leonardr> never mind the ***BLOCKED** thing, it's irrelevant
<leonardr> i ran bootstrap with the wrong arguments
<leonardr> gary: you might know better than flacoste what my problem is -^
<gary_poster> leonardr: does not look familiar.  did you simply try make clean and make?
<leonardr> gary: this is a branch new checkout
<leonardr> http://paste.ubuntu.com/386347/ shows what happens when i checked out and ran make
<gary_poster> leonardr: sorry had dr call.  Try merging my branch...getting
<gary_poster> leonardr: lp:~gary/launchpad/bug491705
<gary_poster> gmb: ping?
<gary_poster> gmb: want to know if you are available to diagnose your buildout problem from Friday.  Let me know if you can, or I should try you later in week.  (Current request is to start with clean branch (make clean is fine), merge my branch, apply http://pastebin.ubuntu.com/384463/ , and give me full output of ``make SHHH=``
<gmb> gary_poster: I've actually found and fixed the problem this morning. There were stray zope.testing packages-3.8.3 on my system, even after apt-get removing it. Removing those packages made the problem go away
<gary_poster> gmb: great...ish. ;-) I can't try making it more robust for everyone else, then.  Do you now know a way I can dupe the original situation, maybe?
<gmb> gary_poster: If you install zope.testing 3.8.3 for python 2.5 and then leave the package lying around in the site-packages directory that should do the trick, I think.
<gary_poster> gmb ok awesome will give it a try thanks.  install via apt I assume you mean?
<gmb> Yes
<gary_poster> cool thanks
<leonardr> gary: same problem. bootstrap warns about ClientCookie, buildout chokes on it
<gary_poster> leonardr: this sounds like an old problem we had when people had badly-removed ClientCookie packages from apt.  Lemme see if the traceback suggests someplace to look.
<leonardr> gary: it definitely seems familiar to me
<leonardr> let me update my packages
<gary_poster> leonardr: I would start Python2.5, and see if ClientCookie is importable.  If so, get the __file__.  Then see if apt thinks that the package is installed.  If apt doesn't think that ClientCookie, or that version of ClientCookie, is installed, then it sounds like the old problem.  Clean up the bad ClientCookie and try again.
<leonardr> gary: it's the latter. i see broken symlinks. 'clean up' by removing or by fixing manually?
<gary_poster> leonardr: ...it depends on what the symlinks are.  If you think the symlinks are supposed to still be there, then fix 'em.  If not, remove 'em.  If you are not sure, I guess I'd want to know exactly what the symlinks were and what they used to point to and what apt thinks is installed for me to make my own guess.
<leonardr> gary: i don't have any 'cookie' package installed, so i removed them
<gary_poster> sounds right, leonardr
<GPenguin> how easy is it to extract the "karma" related code from the entire base?
<GPenguin> or lets put it different - could a python beginner expand the karma related code easily or would it be smarter to "hire" somebody to do it?
<leonardr> gary: getting a new problem now. i got a similar problem, reverted my merge of your branch, and then got this
<leonardr> gary: http://paste.ubuntu.com/386369/
<leonardr> gary: 3.7.3 is the version installed in site-packages
<gary_poster> GPenguin: not ignoring you, but don't know that part of code.  I strongly suspect the only way a Python beginner could do it is with more than a little assistance (to know where to look in the code, and to write automated tests sufficient to pass review).
<leonardr> so once again, the site-packages version is taking precedence over the version installed by ubuntu
<gary_poster> leonardr: did you have that problem with my branch?
<leonardr> er
<leonardr> i had a similar problem... let me find it
<james_w> leonardr: when you have a minute I would appreciate your input on bug 529998, as you will be able to point to the faulty component.
<mup> Bug #529998: Error caused by binding to anonymous collection resource <wadllib:New> <https://launchpad.net/bugs/529998>
<leonardr> gary: with your branch i got this error message
<leonardr> Error: There is a version conflict.
<leonardr> We already have: zope.testing 3.8.2
<leonardr> i believe that was the version you specified in your change to versions.cfg
<gary_poster> leonardr: could you give a pastebin of the full output, *or* let me know if that is happening clearly in bin/buildout, rather than bootstrap?
<gary_poster> leonardr: if it is happening in bootstrap, then this sounds like Graham's report, and I was hypothesizing that http://pastebin.ubuntu.com/384463/ (in addition to my branch) might help
<leonardr> gary: it happened in bootstrap, but looking at the scrollback revealed that i didn't run bootstrap with the right options
<leonardr> i'm trying again now
<gary_poster> leonardr: ok, in bootstrap makes me mildly happier (I was optimistic that I had squelched problems in bin/buildout with my branch). I doubt the options will help, but .  The patch I gave might.
<leonardr> gary: no, it happens in buildout for me. bootstrap runs without warnings or errors
<gary_poster> lovely.
<leonardr> i'll try the patch
<gary_poster> that won't help bin/buildout
<leonardr> right
<leonardr> gary: now that i know it might not be my problem, i'm going to try to do work on a branch i already have checked out
<gary_poster> leonardr: I really have to dupe.  I've solved everything I know of that can cause this in my branch, except for in bootstrap, which is not where you are encountering the problem.  Just to doublecheck, would you please pastebin your bin/buildout?
<leonardr> gary, sure
<gary_poster> The bin/buildout that is causing the problems, of course.
<leonardr> if you need me for this that's a different story
<gary_poster> thanks leonardr.  I certainly don't want to block you any more that you already have been.  Maybe you can make progress on the other branch while I occasionally ask questions of you for this one.  I'll try to be out of your hair as quickly as possible.
<c_schmitz_> hi everyone
<c_schmitz_> A really nice thing about Launchpad is the translations part
<c_schmitz_> would it be possible to use if without the other modules?
<c_schmitz_> *use it
<c_schmitz_> if I install Launchpad on my own site?
<c_schmitz_> Or are the modules all tightly connected to each other?
<c_schmitz_> I would be very glad for any hints/experiences on that
<gary_poster> c_schmitz_: within the hosted service, sure.  on your own site, the code is installed in a bundle, and we don't support that  (we want to encourage people to use the hosted service).
<leonardr> gary: http://paste.ubuntu.com/386381/
<leonardr> brand new checkout of your branch
<c_schmitz_> gary_poster: I see
<danilos> c
<gary_poster> leonardr: could you pastebin the contents of the bin/buildout script?
<leonardr> sure
<leonardr> gary: http://paste.ubuntu.com/386384/
<james_w> thekorn: thanks, that would have saved me quite some time had I found those :-)
<thekorn> james_w, oh :( I wish this bug could somehow be fixed
<james_w> thekorn: I have it fixed locally
<james_w> I'd just like Leonard's input to know whether it is the correct fix
<thekorn> james_w, you mean the .total_size bug or the more general "does not behave like a collection"-bug ?
<james_w> well, the bug that means you get a crash every time you access an attribute of an anonymous collectio
<gary_poster> leonardr: ok thanks.  Have an idea.  Could you give me the content of the addsitepackages function in /home/leonardr/canonical/lp-branches/from-gary/parts/buildout/site.py?
<leonardr> sure
<leonardr> gary: i'll tell you right now it's mostly site-packages repeated a lot
<gary_poster> leonardr: ok that's enough
<leonardr> gary: and then there's the zc.buildout-1.5.0devgary at the end
<gary_poster> leonardr: ok, will give you a patch in a minute or two for you to try.
<leonardr> grat
<gary_poster> leonardr: http://pastebin.ubuntu.com/386394/ for eggs/zc.buildout-1.5.0dev_gary_r109427-py2.5.egg/zc/buildout/buildout.py .  Lines 19-21 are the important bits.
<leonardr> all right
<gary_poster> leonardr: oh forgot to say, you need to rerun bootstrrap
<gary_poster> then bin/buildout should be ok
<gary_poster> (because site-packages should not be in in parts/buildout/site.py any more)
<leonardr> gary: still not working, doing a make clean
<gary_poster> k
<leonardr> gary: site-packages is no longer in site.py but i get the same problem
<gary_poster> !!
<gary_poster> leonardr: *exactly* the same problem (during recipe installation)?
<gary_poster> While: Installing recipe z3c.recipe.i18n.
 * gary_poster hates not being able to dupe problem :-(
<leonardr> gary: yes, exactly the same
<leonardr> we already have: zope.testing 3.8.2
 * gary_poster curses
<leonardr> gary: what if i put debug in whatever outputs "we already have"
<leonardr> to see what the conflict is?
<leonardr> do you know where that code is?
<gary_poster> leonardr: yeah, it is in zc/buildout/easy_install.py, Installer.install, but the conflict is with zope.testing 3.8.2, like it says, which you and I both have installed in apt, so I don't have a lot of optimism on what information we can get extra there.  I think I'll get out of your hair awhile so you can focus more fully on the critical bug fix.  When that is done, let me know.  I'll look at it a bit more.  Thank you!
<leonardr> ok
<leonardr> gary: i no longer have any working branches, so i can't make any more progress on wgrant's problem
<gary_poster> leonardr: ah, lovely.
<leonardr> the conflict is with zope.testing 3.8.2 and what? the version mentioned in versions.cfg?
<gary_poster> leonardr: yes 3.8.1
<gary_poster> leonardr: possible workarounds: (1) uninstall zope.testing (2) try setting 3.8.2 in versions.cfg and hope it works.
<deryck> adeuring, could you do a review for me?  Should be easy.
<leonardr> ok...
<leonardr> Getting distribution for 'zope.testing==3.8.2'.
<leonardr> make: *** [/home/leonardr/canonical/lp-branches/untouched-trunk/bin/py] Error 1
<leonardr> gary: that would imply that it can't find the download-cache at all
<leonardr> ('couldn't find a distribution')
<leonardr> well, that would imply it's not looking in site-packages
<leonardr> so when it's time to look for the version number, it looks in site-packages
<leonardr> and when it's time to look for the package, it looks in download-cache
<gary_poster> leonardr: is this after you tried option 2?
<leonardr> gary: yes
<gary_poster> leonardr: option 1 didn't work/was too annoying because something important depended on it?
<leonardr> gary: #2 seemed easier
<leonardr> i'll try #1 now
<gary_poster> ok
<leonardr> gary: #1 says it'll break python-zodb
<leonardr> but i might not need that since launchpad has its own
<gary_poster> leonardr: yeah, what package wants zodb?  leonardr I'm trying a restart to see if that somehow makes me able to dupe
<gary_poster> ...nope, still works fine...
<gary_poster> leonardr: will try installing python-zodb...if you see what depends on it, tell me, and I'll add that
<adeuring> deryck: yure
<leonardr> gary: so, i removed all the zope stuff
<adeuring> (and sorry ffor the late answer)
<leonardr> now i get
<leonardr> Error: There is a version conflict.
<leonardr> We already have: zope.testing 3.7.3
<gary_poster> leonardr: which aptitude doesn't know about?  If so, find it and kill it.
<leonardr> gary: given a path, can aptitude tell me which package owns that path?
<gary_poster> leonardr: I would see if zope.testing is installed.  If not, kill it.
<deryck> adeuring, thanks.  See:  https://code.edge.launchpad.net/~deryck/launchpad/fix-affects-me-widget-529049/+merge/20380
<leonardr> zope.testing is not installed
<gary_poster> leonardr: (IOW, I don't know, but I don't think it matters)
 * adeuring is looking
<gary_poster> ..aaaand it works just fine for me when I have python-zodb installed.
<gary_poster> leonardr: this is an up-to-date karmic, yeah?
<leonardr> gary: yeah
<gary_poster> k
<leonardr> buildout is taking a while now...
<gary_poster> that's a good sign
<gary_poster> leonardr: do you not share eggs, or have you not made a build in a while?  Usually goes about 6 seconds for me when all eggs are built
<leonardr> gary: i made a build last week. it's possible that earlier code was working with my strangely installed zope packages
<gary_poster> huh
<gary_poster> leonardr: buildout still going??
<adeuring> deryck: r=me
<deryck> adeuring, awesome.  thanks.
<leonardr> gary: seems to be ok, i'll let you know
<gary_poster> ok leonardr thanks
<deryck> adeuring, there's something else I needed to mention to you.... what was it? .... oh, yeah, what's that word?....  gravatar.
<deryck> :-)
<adeuring> yeah...
<deryck> heh
<deryck> adeuring, I'll leave you alone about it now.  Promise. :-)
<gary_poster> leonardr: I think I got it.  I think lines 29-31 of http://pastebin.ubuntu.com/386428/ would have fixed it.  I'll see if I can write a test for it when I get back from lunch.  I still don't understand why it doesn't bite me.
<leonardr> gary, can you spare a few for some Makefile brainstorming?
<gary_poster> sure leonardr
<leonardr> gary: current behavior is to create the wadl file like so
<leonardr>  LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl.py > $@.tmp
<leonardr> however, now create-lp-wadl.py creates an arbitrary number of wadl files, not just one
<leonardr> i changed the script to take a path template as a command-line argument
<leonardr> so now LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl.py $@.tmp
<leonardr> where WADL_FILE is lib/canonical/launchpad/apidoc/wadl-$(LPCONFIG)-%(version)s.xml
<leonardr> unfortunately the old $@.tmp pattern doesn't work anymore
<leonardr> it used to be we were writing to .tmp in a directory that already existed, then renaming
<leonardr> i'm thinking of making an apidoc.tmp directory, generating the files in there, and moving the _contents_ to apidoc
<leonardr> does that sound good?
<gary_poster> leonardr: the concern I have right now is how to make it so that we don't rebuild those for every make invocation
<gary_poster> do you have a plan for that?
<leonardr> gary: right now we explicitly clear out the cache so that the wadl is generated on every make invocation. you want to change that?
<leonardr> if we don't want that, we can just not clear the cache
<gary_poster> we do?!  looking at Makefile
<leonardr> gary: it's in the create-lp-wadl.py file
<gary_poster> leonardr: because it is controlled in the Makefile with $(WADL_FILE): $(BZR_VERSION_INFO) then it will only be regenerated if the wadl file is missing or the bzr version info file changes
<gary_poster> leonardr: that kind of constraint needs to stay.  devs get crankier and crankier the longer make run takes unnecessarily
<leonardr> gary: ok, i can change that so it makes sure the 'devel' wadl is present. we'll always have that
<gary_poster> leonardr: yeah, I was going to suggest something like that
<gary_poster> leonardr: as far as apidoc.tmp, that sounds reasonable as a pattern.  I don't have a string opinion as to names (and may be missing a concern), but apidoc.tmp specifically is ok with me
<mwhudson> yeah, unnecessary make run time brings out the pitchfork wielding mobs
<gary_poster> leonardr: another possibility is to use Python's temp directories
<gary_poster> :-)
<leonardr> gary: from a makefile?
<gary_poster> leonardr: I guess I was thinking of moving the logix to ./utilities/create-lp-wadl.py
<gary_poster> logic
<leonardr> that might be simpler
<mwhudson> you have to be a bit careful using mkdtemp if you want to do os.rename later
<gary_poster> mwhudson: we'll just me moving the contents
<gary_poster> be
<gary_poster> so that's ok, yeah?
<mwhudson> gary_poster: but if /tmp and your launchpad tree are on different filesystems that'll fail, i think?
<gary_poster> mwhudson: ah, so.
<gary_poster> shutil copying would still be fine
<gary_poster> or whatever that package name is...
<gary_poster> yah that's it
<gary_poster> so leonardr, either do everything in create-lp-wadl, but use shutil to do the final move-the-contents thing, or do the directory dance you originally proposed.  I'm fine with either, though slightly prefer the all-Python approach.
<leonardr> gary: where does $@ come from? is that the file that exists in the conditional?
<gary_poster> $@ is the make target I believe, leonardr
<gary_poster> so, $(WADL_FILE)
<gary_poster> in this case
<leonardr> ok, so that's what i need to change to DEVEL_WADL_FILE
<gary_poster> yeah
<leonardr> ok
<thumper> morning
<kfogel> deryck: re bug #471195: when you were talking about javascript the other day, was that for javascript to reload the page (with the new body class tag) after someone sets the bug to "private"?
<mup> Bug #471195: Private bugs don't set the body class to "private" <css> <privacy> <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/471195>
<kfogel> deryck: ...or for something else?
<thumper> deryck: ping
<thumper> deryck: https://lpbuildbot.canonical.com/builders/db_lp/builds/560/steps/shell_7/logs/summary
<thumper> deryck: db builder running right now has that failure
<deryck> thumper, looking now.
 * deryck primal screams
<mwhudson> weee
<mwhudson> blowing up in error reporting, for that extra pain
<deryck> not sure I follow at all what's happening there.  but updating db-devel locally and running locally to see if makes more sense looking at the fille.
<mwhudson> deryck: i know that error
<mwhudson> deryck: the script is trying to report an oops
<mwhudson> deryck: but no oops directory is configured in the config
<mwhudson> --> bang!
<deryck> ah
<deryck> two levels of fail then.  my lucky day. :-)
<deryck> I'm running ./bin/test with no shirt on.
<deryck> never go change to running clothes and check IRC in the middle
<deryck> intellectronica, any chance you're around?
<wgrant> sinzui (or other Registry people): Have you seen bug #529370?
 * sinzui looks
<sinzui> wgrant: No I had not
<sinzui> wgrant: apparently I do not get notified when something is really important
<sinzui> wgrant: Thank you very much for following this up
<wgrant> sinzui: No, we discovered that last night. The email goes to launchpad-bugs@l.u.c.
<wgrant> I believe the issue was taken to the list.
<sinzui> I do not seem to get that. Bug supervisor and security are so fucked when it comes to ACLs
<wgrant> Yes...
<wgrant> sinzui: Interesting that this one is critical for 10.02, when the other two (both of which are more critical, and one of which involves only deleting code) are 10.03.
<sinzui> wgrant: my team completed QA last week, We are available to fix issues now
<wgrant> Ah, good.
<sinzui> wgrant: if It is rejects for RC, I will peruse a CP
<sinzui> s/rejects/rejected/
<wgrant> sinzui: Thanks.
<leonardr> gmb, are you around?
<deryck> thumper, I don't think I can get this fixed before I need to be away.  (Kids little league soccer/footbal practice.)
<thumper> deryck: the db_devel failure?
<deryck> thumper, yeah
<deryck> thumper, there's a problem with intellectronica's method to updates max_bug_heat, though it's not clear to me what's going wrong.
<thumper> deryck: abentley tells me that you are missing an error dir in the config
<thumper> deryck: and probably that is all
<thumper> deryck: maybe that is all
<deryck> thumper, maybe that is another issue.  But I think there's a real error.  If I comment out setMaxBugHeat, the test passes.
<thumper> deryck: hmm, ok
<thumper> deryck: so... what's the plan?
<thumper> deryck: it is only 9pm in London
<thumper> heh
<deryck> thumper, just had one more idea....  let me try.
<deryck> heh
<abentley> deryck, agreed, you have two issues: your test is erroring, and your error handling isn't working because you don't have an error directory configured.
<deryck> abentley, right
<deryck> thumper, I can be back later if intellectronica doesn't come back tonight.  which we shouldn't expect. ;)
<deryck> I know that line 220 of lib/lp/bugs/model/bugtarget.py is the issue.  setMaxBugHeat is, I think, getting a value it doesn't like.
<deryck> but I can't get pdb to play well with this test, so can't work it out.
<deryck> thumper, ^^
<thumper> deryck: hmm, shouldn't then max_bug_heat be on IBugTarget? (I was thinking this last night)
<deryck> thumper, probably.  but last night is unrelated to this. :-)
<thumper> deryck: yeah
<deryck> thumper, so I'll just have to have a look later.  I don't expect anyone on my team to be back tonight.  Very sorry.... but I have to do a practice with my daughter.
<thumper> deryck: ok, I might call Tom :)
<deryck> thumper, I think that's fair.  We shouldn't have landed this without running through ec2.
<deryck> thumper, in fact, I'll call him.
<thumper> deryck: thanks
<intellectronica> thumper: i hear that there's a failure related to bug heat, but looking at the builders they look fine and i didn't get any failure mail. got any info on the problem?
<thumper> intellectronica: https://lpbuildbot.canonical.com/builders/db_lp/builds/560/steps/shell_7/logs/summary
<thumper> intellectronica: it is still running
<intellectronica> thumper: i'll have a look, thanks
 * mwhudson afk for a few minutes
<intellectronica> thumper: i have a fix
<intellectronica> thumper: https://code.edge.launchpad.net/~intellectronica/launchpad/bug-heat-testfix/+merge/20404
 * thumper looks
<thumper> intellectronica: confirmed to fix it?
<thumper> intellectronica: store.flush needed somewhere in a unittest?
<intellectronica> thumper: yes. it's the last error of this kind. i fixed a bunch of them earlier and simply left this user out
<intellectronica> thumper: no, i don't think a store flush is needed to test this
<intellectronica> but i don't know much about this particular test, and it worked before, so i'm satisfied with just fixing the permissions
<james_w> https://code.edge.launchpad.net/~james-w/launchpad/fix-getRequestedReviews/+merge/20378
<intellectronica> the problem was introduced because i made that job, indirectly, do more work that touches more tables, and as a result had to give the permissions to all users who call it
<james_w> ^ a get violates a unique key constraint? It only checks on the get? or doctests are masking the issue?
<thumper> intellectronica: ok
<james_w> the first get is actually triggering it to the actual things that were set up at the start of the test?
<leonardr> gmb, i'm going to throw some stuff out here--send me an email if i'm not around when you read it
<intellectronica> thumper: do i need to land it with [testfix] ?
<thumper> intellectronica: I don't think so
<thumper> intellectronica: as it hasn't finished yet
<intellectronica> ok
<leonardr> i'd like to remove wadl-testrunner-devel.xml from launchpad. bzr annotate seems to indicate that you originally introduced that file. with the multi-version code i've been working on, i don't think that file does anything useful except add another thing that needs to be tested. i'd like to hear if you agree, and if not, what good the wadl-testrunner-devel.xml file does from a performance standpoint or whatever standpoint leads
<leonardr>  you to disagree
<wgrant> james_w: Storm won't flush the changes to the DB until it needs to.
<wgrant> james_w: A GET will normally do a store.find somewhere, which will cause a flush.
<james_w> wgrant: right, so this is likely something in the diff that I've added, rather than the auth changes meaning that it is trying to create a new person when the request comes in?
<james_w> because I'm having trouble seeing how my changes add to or alter to the Person table to violate that constraint
<wgrant> james_w: makeBranchMergeProposal probably created a new Person.
<wgrant> james_w: Do a flush after that, and see if it fails.
<thumper> james_w: it is probably the renaming of a branch or person in the test
<wgrant> + >>> proposal.source_branch.owner.name = 'target'
<james_w> ah
<thumper> james_w: pass in the owner or project to the makeXXX factory function
<james_w> trying to actually get a dev environment back up so that I can actually run the tests
<james_w> so much for coding purely using the force
<wgrant> gary_poster: The change should be easily RCable, right?
<wgrant> It is just a deletion and probably a test inversion.
<gary_poster> wgrant: you are talking about the security bug you made?  If so, yes.  This should be a trivial branch, unles a test run proves otherwise.
<wgrant> gary_poster: Right. Great!
<wgrant> Thanks for getting onto it.
<gary_poster> Thank you, wgrant.
<wgrant> it'll break a few apps, but they'll find out soon enough.
<gary_poster> :-) true
<wgrant> (that's actually how I discovered it)
<wgrant> sinzui: Thanks for fixing that.
<sinzui> wgrant: thanks again for following up the issue
<wgrant> np
<thumper> wgrant: which one?
<wgrant> thumper: Which what?
<thumper> wgrant: which security bug
<thumper> mwhudson: https://code.staging.launchpad.net/~mwhudson/gedit/trunk looks good
<wgrant> https://bugs.launchpad.net/bugs/529370, https://bugs.launchpad.net/bugs/529348 or https://bugs.launchpad.net/bugs/529710
<wgrant> thumper: It would be nice if it gave a more useful progress indicator, but I guess that's more difficult.
<thumper> wgrant: "it gave a more useful progress indicator", what it?
<thumper> wgrant: and that first bug number didn't exist for me
<thumper> wgrant: nm, I think it was the trailing comma
<thumper> wgrant: and the first one is fix committed
<wgrant> thumper: 'it' being incremental imports. They just say n/1000 every time.
<wgrant> It's going to be unobvious to everybody that it's actually working.
<mwhudson> should be easy to change
<thumper> wgrant: file a bug :)
<wgrant> thumper: Against bzr-git?
<thumper> mwhudson: do we echo it exactly?
<mwhudson> yeah
<mwhudson> (to both of you, i guess!)
<thumper> mwhudson: could we hook in our own ui factory?
<mwhudson> thumper: well, so "echo it exactly" is definitely not right
<mwhudson> we do have our own ui factory that only updates once a minute
<thumper> mwhudson: ah
<mwhudson> and doesn't try at all to do fancy terminal things
<mwhudson> we don't fiddle with the text though
<thumper> mwhudson: so we could update our own ui factory to show that it is '345/1000 (partial import of 9876)'
<thumper> something like that?
<mwhudson> thumper: much easier to do it in bzr-git
<thumper> should be trivial right?
<thumper> hmm..
<mwhudson> thumper: well no, our code wouldn't know that the revision count is 9876 (easily, anyway)
<mwhudson> hmm
<mwhudson> not totally flexible actually
<mwhudson> would "fetching 1000 revisions 2/2345" "fetching 1000 revisions 5/2345" and then stopping at 1000/2345 make sense
<mwhudson> or "fetching 1000 out of 2345 revisions 4/1000" ?
<mwhudson> i guess the latter
* kfogel changed the topic of #launchpad-dev to: Launchpad Development Channel | Launchpad read-only 23.00 UTC Wed 3 March - 00:30 UTC Thu 4 March, for 10.02 rollout (http://is.gd/9tVfU) | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
 * thumper afk
<james_w> wgrant: did you have to anything special with python-debian to get it working under 2.5?
<wgrant> james_w: I don't believe so.
<wgrant> james_w: Hm. You may have to reinstall/reconfigure it after installing the 2.5-capable python-support and python-central.
<james_w> oh
<wgrant> Although I thought the PPA's python-support was modified to automatically compile modules when adding a new supported version.
<james_w> I wasn't aware that I needed python-support to change as well as python-defaults for it to work for 2.5
<wgrant> james_w: You're not fully upgraded from the PPA?
<james_w> no
<james_w> I have it pinned at 50
<wgrant> Heh. Good luck with that.
<james_w> well, I don't mind the troubles when it helps me understand what's being added
<mwhudson> thumper: does "fetching 1000 out of 5354 revisions 880/1000" look better to you?
<wgrant> mwhudson: that doesn't give any indication of overall progress. "fetching 1000 revisions from revision 3000: 3521/9876", maybe?
<mwhudson> not sure that that information is terribly accessible
<lifeless> nested pb
<lifeless> outer one you know - you control it
<lifeless> inner is bzr driven
<lifeless> write a custom UI to log this to wherever your 'show to the user' stuff goes
<mwhudson> er
<mwhudson> we already do most of that
 * wgrant should really learn this bzr thing one day.
<lifeless> wgrant: anytime you want, tutorials on demand
#launchpad-dev 2010-03-02
<maxb> james_w: ooi, why pin the PPA?
<james_w> maxb: I don't want to give free reign to LP developers when I am supposed to be testing the development release of Ubuntu
<james_w> maxb: the closer I can stay to pure lucid the better, and the more I know about the changes made the better
<maxb> The reason for needing python-support is because python-support disregards python-defaults and embeds its own version list :-/
<james_w> wow
<maxb> fair enough, but the PPA's delta is small. It's mostly no-change rebuilds
<james_w> yeah
<james_w> it's not that it's an LP ppa specifically, but that it is an extra archive at all
<thumper> mwhudson: any reason why "Modify worker/monitor/UI to handle partial success" task is still blocked for QA (on kanban)?
<mwhudson> Ursinha: sorry for breaking the scripts!
 * mwhudson hates wikis, by the by
<mwhudson> thumper: no reason, fixed
<thumper> mwhudson: awesome
<Ursinha> mwhudson: lol
<Ursinha> thumper: hi, you're the RM this cycle right?
<thumper> Ursinha: aye
<Ursinha> thumper: mwhudson, are you aware of this oops: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1518S10
<james_w> aha!
<james_w> ... in doctests is greedy!
 * thumper frowns
<Ursinha> thumper: I saw about 400 of them on staging, in 02/26
<thumper> Ursinha: likely to be when codehosting was updated before the xmlrpc server or vice versa
<mwhudson> Ursinha: that'll be the codehosting system not in sync with the appserver
<thumper> Ursinha: as long as they are not happening now we should be good
<mwhudson> Ursinha: what thumper said
<Ursinha> thumper: mwhudson, that's fine, I' ll let you know if they start happening again
<Ursinha> thanks :)
<mwhudson> thanks
<Ursinha> thanks :)
<Ursinha> oops
<Ursinha> :)
<mwhudson> wow, the staging code import slave is _much_ slower than my laptop
<mwhudson> we gots some old hardware here...
<thumper> mwhudson: how do I debug a particular function?
<thumper> with pdb
<thumper> I'm in the harness
<thumper> and I want to step through a function
<mwhudson> thumper: put "import pdb; pdb.set_trace()" in it?
<mwhudson> oh
<mwhudson> i think pbb.run("f()") ?
 * thumper grunts
<thumper> it seems I need locals somewhere
<mwhudson> or maybe pdb.runcall(f, a1, a2)
<mars> mwhudson, neat trick.
<thumper> yup
 * thumper steps through
<mars> thumper, if something blows up in the harness, you may also be able to use pdb.pm() to dissect the traceback.  Haven't tried it though.
<thumper> gah
 * thumper files first bug
<mars> oh, cool
<mars> in ipython, "setting %pdb on within ipython will make it such that the pdb debugger will automatically be started at the point of an exception"
<mars> err, that should read '%pdb on', as the command
<mars> so that would work in the harness too
<mars> iharness, specifically
<thumper> oh clucking bell
<thumper> mwhudson: got a minute?
<mwhudson> thumper: sure
<mwhudson> (sorry for the delay)
 * thumper finishes editing the wiki
<thumper> mwhudson: I was just thinking about the bugs I've just filed
<thumper> mwhudson: and how urgent they really are
 * mwhudson looks at email
<thumper> they have to do with the qa of the email stuff
<thumper> mwhudson: also I've had an idea about oops 1521SMPSSH1
<mwhudson> thumper: 530455 looks like inspiring head-wall interaction
<thumper> mwhudson: skype?
<mwhudson> thumper: sure
<mwhudson> thumper: is that the "None has no attribute close" one?
<thumper> yeah
<thumper> or no attribute write
<mwhudson> er ok
 * thumper -> dinner
 * mwhudson eods
<kfogel> thumper: g'night
<thumper> kfogel: night
 * thumper EODs
<noodles775> wgrant: hey, have you come across this when trying to send a key to your local keyserver? http://pastebin.ubuntu.com/386801/
 * noodles775 checks if it's lucid-related...
<wgrant> noodles775: Yeah, Lucid-related.
<wgrant> There is a patch to pygpgme.
 * wgrant finds.
<noodles775> Thanks!
<wgrant> I think it's attached to a bug somewhere, but I've just been passing it around for now.
<noodles775> OK, I'll update the wiki page.
<wgrant> http://pastebin.ubuntu.com/386803/
<noodles775> Ta!
<wgrant> Bug #452194
<mup> Bug #452194: Call to gpgme_check_version is required when using gpgme >= 1.2.0 <PyGPGME:New> <https://launchpad.net/bugs/452194>
<spm> wgrant: any objections if we add an API to talk to you for all bug searches? no particular reason for asking... ;-)
<wgrant> Heh.
<adeuring> good morning
<jtv> wgrant: my branch for the soyuz dev startup work is up for review.  I *think* I found a way to attach a real GPG key automatically, but I'm not sure it's complete because the UI still says the key is missing.
<wgrant> jtv: Link?
<jtv> wgrant: yes
<jtv> https://code.launchpad.net/~jtv/launchpad/bug-527170/+merge/20433
<jtv> wgrant: gets your fingerprint by running gpg --fingerprint
<wgrant> jtv: I'd prefer it if it took it as an argument.
<wgrant> But let me read.
 * wgrant wonders why GPGKeySet.new() takes a person *ID*.
<jtv> wgrant: it seems to be a pattern in soyuz
<jtv> I've seen it before
<wgrant> La la la GPG isn't really Soyuz.
<jtv> (Okay, I remember seeing it *once* before but by the bluffer's guide that makes it a pattern)
<wgrant> The occasional part of Soyuz does it for performance reasons, though.
<jtv> rv42
<jtv> ahem
<wgrant> It's odd that that doesn't work.
<wgrant> Does it actually get added to the table?
<jtv> wgrant: yes, it shows up as added to the test account (I called it ppa-user), but when I click through to it, I get a page saying that the key was not found
<jtv> This may be because I'm not originally getting the key from zeca, I suppose.
<wgrant> Ah, yes.
<wgrant> It's a page on keyserver.launchpad.dev, right?
<wgrant> Hm, but GPGHandler is meant to use zeca, isn't it?
<jtv> wgrant: I believe so, yes
<jtv> wgrant: at least I got around the CoC signing trouble by inserting a CoC signed with a fake key.
<wgrant> Right.
<jtv> that should save a bit of aggro all by itself, though it's not really germane to the rest of the gpg goings-on
<jml> good morning Launchpadders.
<wgrant> jtv: Yeah, although it's easy with the admin console.
<jtv> hi jml!
<wgrant> By the way, the CoC infrastructure sucks.
<jtv> heh
<jtv> one battle at a time  :)
<lifeless> wgrant: har har
<wgrant> lifeless: Hm?
<lifeless> CoC..sucks
<wgrant> Heh.
<wgrant> (unintended)
<jtv> *sigh*
<wgrant> Uhoh.
<wgrant> What's happened now?
<jtv> wgrant: puerile humour has happened
<wgrant> Ah.
<lifeless> jtv: oh noes.
<jtv> I _could_ add here a reference to the gay & lesbian rights group of the same name, but
<jtv> okay, I admit it: I couldn't think of a good one
 * wgrant must try Landscape at some point.
<bigjools> morning dudes
<bigjools> jtv, just the man I need to speak to
<jtv> uh-oh
<jtv> bigjools: I do have a standup coming up I believe, so we may have to work around that, but otherwise might as well try to strike while the iron is hot
<jtv> lifeless, wgrant: for the record, I'm a big fan of puerile humour.  I really was sighing only over my failure to come up with a good one.
<bigjools> jtv: really quick - is there a way to make PG return a zero as a default value to a query?
<wgrant> COALESCE(expr, 0)?
<jtv> bigjools: what wgrant said
<jtv> That means "expr, or if it's null, 0."
<bigjools> parfait
<jtv> Works for any number of args.
<bigjools> I suck at sql
<bigjools> thanks
<jtv> SELECT COALESCE(min(id), 0) FROM MyTable
<jtv> wgrant: to test, run ./utilities/start-dev-soyuz.sh *first* (I'll be updating the docs) and then ./utilities/soyuz-sampledata-cleanup.py using the --email option to provide your real email address.
<jtv> I think I'll rename that latter script to reflect its expanded scope.
<wgrant> jtv: The first seems better as a Makefile target.
 * wgrant backs up his DB and tests.
<jtv> wgrant: cool idea...  I'm thinking one step at a time though, since I'm also learning in the process.
<jtv> otp now
<wgrant> k
<wgrant> security.py is too slooooow.
<mrevell> Good morning
<jtv> wgrant: while you're waiting for that, I'm off the phone
<jtv> hi mrevell!
<mrevell> Hi jtv :)
<jtv> wgrant: as for making that script a make target, I think it's a good idea.  One problem is setting up the right email address.
<wgrant> jtv: I mean the soyuz-starting one.
<jtv> wgrant: oww
<wgrant> I don't think the other one makes sense.
<jtv> wgrant: before turning that into a make target I'd like to see some more robustness... have separate targets for "make sure service foo is running," that sort of thing.
<jtv> Right now it just fails somewhere along the line if something's already running.
<jtv> Not good for make targets.
<noodles775> Hey jtv, have you already included the removal of the sample build (id:8) that is currently building on bob in the sample data?
<wgrant> jtv: See for example 'make run_codehosting'
<wgrant> noodles775: Good idea.
<jtv> noodles775: if that one wasn't in w's original, I don't think I did
<wgrant> It wasn't.
<noodles775> jtv: ok, it'd be great to do... if you don't have time, I'll do it as a followup.
<jtv> Then I didn't.
<jtv> noodles775: I've already submitted my branch for review; I'm sure it needs more work, but I want to keep the chunks manageable.
<wgrant> Probably because I classically ran it before starting a buildd, so buildd-manager detected that the builder was offline and unassigned the build.
<noodles775> jtv: no worries, I'd be happy to add it later :), thanks for simplifying the whole process!
<jtv> wgrant: that make target looks useful... is there any documentation that's easier to read than the source and specifies how it deals with already-running daemons?
<jtv> noodles775: glad to see you jump in!
<wgrant> jtv: bin/run is buildout evil. Kill it with fire.
<jtv> wgrant: did I mention I caught a whiff of certain patterns sometimes?  :-)
<wgrant> Excellent, it added my correct test key.
<jtv> wgrant: if you click through to it though, you'll see that knowledge of it is not complete and I have no idea whether that's likely to be an immediate problem.
<wgrant> Where are directories normally created?
<wgrant> /var/tmp/poppy and /var/tmp/zeca should probably be added there.
<jtv> wgrant: I added that in the start-soyuz script
<wgrant> jtv: Ah, right. We should probably work out how to do all this properly.
<jtv> wgrant: ideally, yes, though the effort should be commensurate with use.
<wgrant> jtv: Ah, that 404 is just because zeca is pretty braindead.
<wgrant> It doesn't do indices; if you replace 'index' with 'get' it will work.
<jtv> wgrant: whoo!
<jtv> any easy way we can fix it up?
<wgrant> Not really.
<jtv> Oh well.  I'll admit to being selfish here: if it works, I have bigger fish to fry.
<jtv> Or should I admit to being shellfish?
<wgrant> zeca didn't even take uploads until a little over a year ago. It was a really trivial thing that just served files.
<wgrant> Heh.
<wgrant> It works, just the link won't.
<wgrant> I think it'll do for now.
<jtv> \o/
<wgrant> Actually, I remember now that zeca didn't originally actually work as a full dev server, but I patched it so it would.
<wgrant> I must have just decided that that wasn't worth fixing.
<jtv> so it never really works in dev?
<wgrant> The link on the person index doesn't, no.
<wgrant> But the internals will all work fine.
<jtv> btw it turned out that GPGHOME gets set somewhere to a tmp directory, and that's why it was hard retrieving real-world keys.
<wgrant> Why does that make it hard?
<jtv> I couldn't figure out why I wasn't finding any keys using pygpgme
<wgrant> Don't you retrieve them from the server?
<jtv> No, ultimately I forked off a gpg _with the variable removed_.
<jtv> I tried the --home-dir option, but that didn't do the trick for some reason.  Maybe I used it wrong.
<wgrant> Oh, so you're not sending the key to the server?
<wgrant> In that case it won't work.
<wgrant> I guess mine just already had it.
<jtv> gah
<jtv> wgrant: I'll see if I can delete my key from zeca and re-test
<wgrant> jtv: rm /var/tmp/zeca/*
<jtv> wgrant: what about zeca.test?  That's where most of the stuff is.  Or is that _only_ the links to the keys that come in ftests?
<wgrant> jtv: I believe that's only used in the test config.
<jtv> wgrant: you were right :(
<jtv> key not found
<jtv> wgrant: so... how _do_ I get that key into zeca?
<wgrant> jtv: gpg --keyserver keyserver.launchpad.dev --send-key DEADBEEF
<wgrant> I thought that was in the docs.
<jtv> wgrant: ah yes, so it is
<jtv> I just couldn't place it in the larger picture
<wgrant> Ah, right.
<jtv> (from my perspective, all that's missing is the ritual goat sacrifice)
<wgrant> Heh.
<jtv> Anyway, not too hard I'd say!
<wgrant> It all makes perfect sense to me now!
<jtv> a good ritual goat sacrifice can really work wonders can't it?
<wgrant> Always.
<noodles775> wgrant, jtv: what have I missed? http://pastebin.ubuntu.com/386876/
<wgrant> noodles775: ./scripts/publish-distro.py -C
<noodles775> ta.
<wgrant> I appear to have not added that to the docs.
<noodles775> wgrant: It's there, I just missed it as I had to go back and forth a bit (initially had a corrupt chroot).
<noodles775> Ah, I see, it's only htere after the first build, where as it needs to be run before.
<jtv> wgrant: is it safe to assume that the key id is always the last 4 octets of the fingerprint?
<wgrant> noodles775: Yeah.
<wgrant> jtv: That's what it is, so yes.
<jtv> cool
<wgrant> jtv: But you can also --send-key the fingerprint.
<jtv> even better :)
<wgrant> I just don't, because I'm lazy.
<wgrant> But code has a convenient attribute of not being lazy.
<jtv> should we have told noodles775 that he also missed the goat sacrifice?
<jtv> Just a thought.
<wgrant> Possibly.
<wgrant> noodles775: Have you done this before?
<noodles775> oh, please do...
<noodles775> Nope, just going through for the first time (so I can test some stuff locally).
<wgrant> Aha.
<wgrant> Only a couple of people have used it, AFAIK.
<wgrant> Although it's a lot more streamlined now.
<noodles775> Building happily now, thanks wgrant/jtv.
 * jtv points at wgrant
 * wgrant points back.
<noodles775> arg, still failed (chroot problem) http://pastebin.ubuntu.com/386885/ ...but that's just because the package I'm testing with is targeted at security right?
<wgrant> noodles775: Ah, yeah, you can't target to security.
<noodles775> OK.
<wgrant> Unless there's a full copy of the archive somewhere that I don't know about apart from drescher.
<wgrant> Er, cocoplum.
 * wgrant is stuck in the past.
<jml> mrevell, hi
<mrevell> hi jml
<jml> mrevell, sorry I missed our call. for some reason my calendar didn't buzz at me
<mrevell> jml, No worries. I'd heard a rumour you were sprinting at Millbank.
<jml> mrevell, no, I'm just at Millbank
<jml> mrevell, there's a decorator doing my room at home, so I can't work there.
<mrevell> jml, Ah right. I can do a call now, if you're available. If you don't mind, I'll just grab a tea first, though.
<jml> mrevell, yeah. me too.
<jtv> wgrant: looks like security.py does a lot of needless name-dropping.  I mean, it seems to drop a lot more users than it needs to.
<jtv> stub, is that right?  reset_permissions in security.py seems to try and drop every user from every group.
<jtv> well, from only two groups actually
<jtv> wgrant: sending the key seems to be working
<wgrant> jtv: Great.
<jml> mrevell, whenever you're ready
<mrevell> jml, Cool, just sorting Skype.
<jml> mrevell, when I said "skype was ok"
<mrevell> :)
<stub> jtv: Yes. It is a dumb implementation that needs improving.
<stub> jtv: That along with revoking all permissions and regranting them means live updates are pretty much impossible and the script is a low slower than it needs to be.
<jtv> stub: for now I'm really just curious if that'd cost significant time, or whether it's a harmless good-enough implementation.
<stub> Its not harmess as it means we have to do security modifications manually. If it was smart enough to only apply the changes, we could run security.py against the live database without causing all the active sessions to bomb. It is also slow, as when you scale that up to 150 tables that is an awful lot of queries to issue (slower still when you need to quadruple that to run on all replicas)
<jml> mrevell, you've dropped out. should I call you on a landline?
<mrevell> If that's okay jml, will message you a number
<stub> (although the first bit we could probably fix by doing things in a single transaction now I think of it)
<jtv> stub: sounds like an interesting challenge.  I know you mentioned it months ago as something I could tackle.
<jtv> IIRC there's only one commit in that script, though maybe it gets called from a loop somewhere
<deryck> Morning, all.
<jelmer> hey Deryck
<wgrant> bigjools: are you still busy, or do you have time to talk about PPA download stats?
<bigjools> wgrant: busy as in looking at this security vuln
<bigjools> but we can talk
<wgrant> bigjools: Ah, no, you have better things to do :)
<bigjools> heh :)
<bigjools> wgrant: any suggestions on how to tackle it are welcome ;)
<bigjools> I've only just literally opened up the file in question to look
<bigjools> oh and don't talk about it on here of course
<wgrant> Of course.
<jtv> lifeless: help!  When I do  branch, subdir = Branch.open_containing(url) ; export(branch.basis_tree(), '/tmp/foo')  from my LP branch, all I ever seem to get is the LP source tree.  Shouldn't that give me the branch at "url"?
<jml> noodles775, hello
<noodles775> Hi jml
<jml> noodles775, I'm now finally looking at the fourteen emails I've got about build from branch since I went to PyCon
<jml> noodles775, and I thought, maybe it'd be easier just to talk with you :)
<noodles775> jml: sure, call in 10mins?
<jml> noodles775, perfect.
<deryck> jtv, ping
<jtv> deryck: good evening
<deryck> jtv, there's a bug from you on the malone 10.02 milestone... still in progress.  But I think you finished this.  Can you look at it and update the status or move it off?
<jtv> deryck: will do
<deryck> jtv, thanks
<jtv> deryck: that one disappears with a lazr-js upgrade, so I'll have to go after that.
<jtv> I think.  :/
<deryck> jtv, I just did an LP branch updating to latest lazr-js.  So once it lands that should be fixed then.
<jtv> deryck: on closer inspection, this looks like it was one that still had a mystery problem.  :(
<deryck> jtv, ah, ok.  Do you think you'll look at it again?  If so, let's move to 10.03.  if not, we can un-milestone and take it out of progress.
<jtv> deryck: I ended up being stumped IIRC.  Better to let the next volunteer try.
<deryck> jtv, that works.  Thanks for looking into it initially, though.
<jtv> deryck: good luck with the little bastard :)
<deryck> heh
<deryck> thanks
<noodles775> james_w: Hi, are you free for a call regarding the mockups?
<noodles775> (at some point).
<james_w> sure
<james_w> either now or after lunch?
<noodles775> james_w: if you're going to lunch now, then after is fine, otherwise now is good for me.
<james_w> I would be going for lunch in 30 minutes or so
<james_w> depends how much you have on your mind :-)
<noodles775> Perfect, I only expect a 10min call :)
<james_w> nice
<jtv> noodles775: I successfully set external_dependencies, but aren't they supposed to show up in the UI?
<james_w> I'll hold you to that ;-)
<noodles775> :-D
<noodles775> jtv: yes, I would have expected them to.
<jtv> noodles775: I checked in "make harness" as well as the db, and I really have a string in there.  Need a newline at the end maybe?
<james_w> noodles775: I should make your skype make silly noises?
<noodles775> jtv: I'd check what's being passed to the form for the UI (in browser/archive.py). It might have something to do with the expanded_dependencies field, I haven't checked.
<noodles775> james_w: calling now.
<deryck> gmb, there is a launchpad-users email question about timeouts with +filebug, saying this is on edge....
<jtv> noodles775: thx
<deryck> gmb, I'm a little suspect of it, though, because it looks like the old timeout message, and we should be async now, right?
<gmb> deryck: I've replied
<deryck> gmb, ah, ok.  go, you! :-)
<gmb> deryck: If the user has JS disabled then they'll still hit the timeout with long bug titles.
<deryck> ah, right.  that makes sense.
<intellectronica> gmb: so, i'm trying to create bug heat calculation jobs. i haven't figured out a way to do this with sql only, so i'm writing a script. but it occurred to me that maybe you do know of a way of doing that with just sql. any hints?
<gmb> intellectronica: Er... Lemme have a think for a sec.
<gmb> intellectronica: So, no, I can't think of a SQL-only way to do it I'm afraid.
<gmb> Because the job table is quite complex.
<gmb> (Alhtough the BugJob table is stupidly simple)
<intellectronica> gmb: oright, so a script creating jobs using the model object is the sensible way to go
<gmb> intellectronica: Yes.
<gmb> intellectronica: Though... how many are you planning to update?
<intellectronica> gmb: cool, thanks
<persia> Erm, why can't it be done with SQL?  It's just math, isn't it?
<intellectronica> gmb: i was thinking of updating all bugs for malone
<intellectronica> is that unrealistic?
<gmb> intellectronica: Okay, sure, that sounds pretty sane.
<gmb> intellectronica: To clarify, do you want SQL for creating jobs or just for updating heat?
<gmb> Cos updating heat is easy peasy.
<gmb> (As persia just pointed out)
<intellectronica> gmb: no, i want to create jobs, because i want it to go through the webapp code and trigger recalculateMaxBugHeat too
<gmb> intellectronica: Right, fair enough. Just checking that I understood properly
<deryck> intellectronica, will this trigger max_bug_heat across all projects, or only for malone?
<deryck> by "trigger" I really mean "update" :-)
<intellectronica> deryck: only for malone
<deryck> intellectronica, so maybe generate one job for any ubuntu bug, too, so we can see max_bug_heat updated for ubuntu bugs and check display of that, too.
<intellectronica> deryck: ok, makes sense
<deryck> intellectronica, thanks
<sinzui> jpds: take a look at bug 181298 that I do not think you have seen. It may be invalid now
<mup> Bug #181298: Get rid of DistributionMirror.official_candidate <mirror> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/181298>
<james_w> using the API to add a tag on staging isn't working, any chance that this has broken server-side?
<james_w> ah, it's ok, lists are odd in the LP API and I'm doing it wrong
<thekorn> james_w, hehe, yes, .append is not working, you need to overwrite the list
<james_w> hey thekorn
<james_w> thekorn: are you able to easily reproduce bug 336866?
<mup> Bug #336866: When adding tag or updating description, lp_save() gives "HTTP Error 412: Precondition Failed" <tech-debt> <lazr.restful:Confirmed for leonardr> <lazr.restfulclient (Ubuntu):Triaged> <https://launchpad.net/bugs/336866>
<james_w> I'm having difficulty
<james_w> ok, I can do it with a newMessage call and manipulating tags, but not just tag manipulation
<thekorn> james_w, yes I can reproduce this bug sometimes, but it is not clear to me under which condition this is reproducible
<james_w> bug.newMessage seems to alter the bug, but it doesn't cause a refresh of the bug object
<thekorn> james_w, one of my ideas (as descibed in https://bugs.edge.launchpad.net/launchpadlib/+bug/341950/comments/2) was that the http_etag attribute is not updated properly
<mup> Bug #341950: lp_save gives HTTP Error 412: Precondition Failed when adding a comment to a bug, but adds the comment anyway <api> <launchpadlib :New for intellectronica> <https://launchpad.net/bugs/341950>
<james_w> but I don't see why that should affect just changing tags
<thekorn> argh, someone killed staging ;)
<jml> hmm
<adiroiban> sinzui: hi, for bug 527728, the component name should be exported as âcomponentâ , âcomponent_nameâ or âlatest_published_component_nameâ ?
<mup> Bug #527728: Export source package component in API <api> <Launchpad Registry:In Progress by adiroiban> <https://launchpad.net/bugs/527728>
<sinzui> adiroiban: I think component_name is correct
 * sinzui looks
<jml> mrevell, where's that terminology wiki page?
<adiroiban> jml: https://dev.launchpad.net/PlainEnglish ?
<sinzui> adiroiban: are you adding support to publish the component object, or flattening the data to return the string?
<jml> adiroiban, thanks.
<adiroiban> sinzui: flattening to return a string... since IComponent has only one attribute
<adiroiban> the name
<mrevell> jml, Took me a while to find as I gave it a stupid name. Thanks for getting it adiroiban :)
<sinzui> adiroiban: SP.latest_published_component_name
<adiroiban> sinzui: thanks!
<adeuring> a reference to branches
<noodles775> james_w: can you pls go over the following use-case text and edit as necessary? https://dev.launchpad.net/BuildBranchToArchiveUI/UseCasePackagingBranch
<james_w> sure
<james_w> thanks
<leonardr> flacoste, gary: hopefully i've fixed it
<gary_poster> leonardr: great news!
<EdwinGrubbs> gary_poster: do you know if there is a way to pass a variable to a view included with <span tal:content="context/@@+someview" />
<gary_poster> thinking, EdwinGrubbs
<gary_poster> EdwinGrubbs: Eh, only hacks (maybe you could stash something in request.annotations with a Python expression?).  Alternatively, you could maybe make the view class make the call instead, and have your template call a method on your view class?  The view class could be a lot more flexible.
<leonardr> gary, still seems to be another problem...
<gary_poster> leonardr: ack.  lemme know if  can help, as before.  You still have an hour ;-)
<leonardr> gary: it's a problem with launchpadlib, not with the web service but with the token authorization process
<gary_poster> ack.
<gary_poster> I guess it still needs to be fixed before we can deploy the other fixes though, so that the tests pass.
<leonardr> the strange thing is, it didn't happen last time, and i've undone the only change i made to launchpad last time
<deryck> leonardr, ping
<leonardr> deryck: hey, i'm very busy right now but if you tell me what's up i can get back to you
<deryck> leonardr, sorry.  just wondering if read_only attributes affect etag calculation if they change?
<leonardr> deryck: they shouldn't
<leonardr> deryck: sorry, i actually don't know
<deryck> leonardr, ok, no worries.  we can discuss more later.
<kfogel> adeuring: if I've modified lib/canonical/launchpad/javascript/bugs/bugtask-index.js (see http://paste.ubuntu.com/387117/), is there some command I have to run to make it take effect, like to rebuild a compressed javascript file or something?  The change doesn't seem to have had any effect.  On the other hand, the HTML for https://bugs.launchpad.dev/firefox/+bug/1 in my local instance refers to https://launchpad.dev/+icing/rev10420/bu
<kfogel> ild/bugs/bugtask-index.js, and fetching that JS file shows my change is present.  Hmmm.
<mup> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Invalid> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <OpenOffice:Invalid by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invalid by jahyire2006> <Ubuntu Jaunty:In Progress> <ubuntu-expre
<kfogel> shut up mup
<adeuring> kfogel: I have no idea... Might also be a caching problem within your browser
<deryck> kfogel, when you run the dev server, you're in devmode, which uses the individually linked files.
<deryck> kfogel, so if your change isn't working, there's something wrong.  No make step is required.
<kfogel> deryck: thanks
<deryck> kfogel, you can do poor man's debugging with Y.log to see if the area of the code is called, if you don't want to breakpoint with firebug.
<kfogel> deryck: thanks -- actually, I think learning how to breakpoint with firebug (which I did not know I could do until you said it just now) might be a good thing.
<deryck> kfogel, in code, add a `debugger;` statement.  Or else open the script in firebug and toggle break points next to lines.
<deryck> kfogel, the debugger statement is the easiest to me.
<kfogel> deryck: thank you
<deryck> kfogel, np
<kfogel> deryck: whoa!  So I tried going to the .js line in firebug, and my changes are not there in firebug!  (Even though they are when I fetch the script using wget.)
<deryck> kfogel, hmm, that's odd.
<deryck> kfogel, need to do a call, but I can help debug further shortly.
<kfogel> deryck: np.  I'll let you know if I solve it first.
<deryck[lunch]> kfogel, sorry, off call, but reallllly need to eat now. :-)
<kfogel> deryck[lunch]: np
<kfogel> deryck[lunch]: I'm going to grab food too, so we're sort of in sync.  I think I have a clue as to what the problem is; will explain when we're both fed.
<jml> I'm getting ValueError: ("Missing 'Version:' header and/or PKG-INFO file", ClientCookie [unknown version] (/usr/lib/python2.5/site-packages))
<jml>  
<jml> actually, nm.
<jml> wait, definitely still getting it
<jml> even with current stable
<jml> ValueError: ("Missing 'Version:' header and/or PKG-INFO file", ClientCookie [unknown version] (/usr/lib/python2.5/site-packages))
<jml> leonardr, is there a document on exposing things over the API?
<leonardr> jml: you mean for developers?
<leonardr> launcpad developers
<jml> leonardr, for Launchpad contributors, yes.
<jml> leonardr, I mostly know how to do it, but I'm helping someone else right now and want to save myself writing a poorer version line-by-line on IRC :)
<leonardr> jml: it's not too great, but https://dev.launchpad.net/API
<jml> leonardr, thanks.
<jml> kfogel, when you've got a moment, can you link to https://dev.launchpad.net/API from https://help.launchpad.net/API?
<jml> kfogel, I would but I don't have permission.
<maxb> And if you're poking those pages, make them stop lying about the API still being in closed beta
<maxb> :-)
<jml> leonardr, there's still no way to have a read operation return a dict or a list that reliably preserves ordering is there?
<leonardr> jml: no, i haven't had time to work on that
<jml> leonardr, ok, thanks.
<kfogel> jml: sure.  but grrr, what's with this not having perms to edit help.l.n?  I've filed an RT on this, but heard nothing yet AFAIK.
<jml> what does this error mean? AssertionError: No IEntry adapter found for IGPGKeySet (web service version: devel).
<jml> kfogel, yeah, that sucks.
<jml> kfogel, send flacoste the RT number -- he can negotiate a priority bump
<kfogel> jml: I will.   Heh, yes, it's RT 37638 and it's about *exactly* this API page.
<james_w> jml: missing export_as_webservice_entry() perhaps?
<jml> james_w: I don't think so
<jml> james_w, class IGPGKey(IHasOwner):
<jml>     """OpenPGP support"""
<jml>     export_as_webservice_entry()
<jml> class IGPGKeySet(Interface):
<jml>     """The set of GPGKeys."""
<jml>     export_as_webservice_collection(IGPGKey)
<james_w> jml: IGPGKeySet will be a top-level collection?
<james_w> lp.gpgkeys?
<jml> james_w, I don't want it to be
<james_w> I thought export_as_webservice_collection was for top-level only, but perhaps I mis-read
<jml> james_w, if I remove that export, I get the same error, actually.
 * jml tries
<jml> yeah, if I remove the export then AssertionError: No IEntry adapter found for IGPGKeySet (web service version: devel).
<james_w> is there an existing attribute/method you want something parallel to, or is this different to everything else?
<jml> IPerson.gpgkeys
<jml> that's what I want
<jml> it's a list of objects
<jml> there are many like it, but this one is mine
<james_w> like IPerson.confirmed_email_addresses
<jml> yes
<jml> ahhh, I see the issue.
<james_w> that uses exported(CollectionField()) rather than a collection class it seems
<jml> the code here had value_type=Reference(IGPGKeySet)
<jml> in IPerson
<jml> the error made me think that I should be looking in IGPGKeySet
<jml> abentley, I'm not compus mentus enough to do a review of your scanner events patch
<jml> abentley, but I think that Fixtures([ServerFixture(...)]) can be replaced with ServerFixture(...)
<abentley> jml, Ah, cool.
<jml> abentley, and I think that leaving the @adapter decorators in the code is still a good idea. Moving the registration into zcml seems like a good idea though
<abentley> jml, what's the value in retaining the adapter decorators?
<jml> abentley, I guess it's a taste thing.
<jml> abentley, it means less configuration, and I think appropriately less.
<jml> since, say, got_new_revision is never going to handle events other than INewRevision
<abentley> I don't understand how it means less configuration.
<jml> abentley, means less configuration in the zcml, I guess.
<jml> afaict, it's only a taste thing.
<jml> as I said though, I'm not particularly compus right now
<jml> (wuu jetlag)
<abentley> jml, are you suggesting that if the adapters were retained, we could omit the for= stanzas values from the subscriber elements?
<jml> abentley, oh yes.
<jml> abentley, you could definitely do that.
<abentley> jml, it sounds like that's not actually what you meant, though.
<jml> abentley, sorry, I was thinking that really hard. I must be in a Faraday cage or something
<jml> abentley, yes. what I meant was, retain @adapter(TipChanged), remove the for= line from zcml, jml prefers this but can't explain why.
<abentley> jml, thanks for explaining that.  For my taste, it's nicer to have both the "for=" and "handler=" in one place, though.
<jml> abentley, ok
<jml> abentley, I guess the new job-driven scanner is already loading zcml
<abentley> jml, it is.  We has a test that an email job is created by the scanner.
<jml> abentley, cool.
<jml> abentley, one of the reasons for the old system was avoiding the 2+ second zcml load time
<abentley> jml, UpdateBranches is a LaunchpadCronScript, so it also loads the zcml.
<jml> *nod*
<abentley> jml, so even with the old script, we were loading the zcml.
<jml> abentley, I'm surprised and even sceptical
<jml> abentley, but will defer to your more current knowledge
<jml> oh wait
 * jml does some logic
<jml> yeah, you're right.
<abentley> jml, see LaunchpadScript.run
<jml> abentley, well, more relevantly to my addled brain, we getUtility(IBranchScanner) at least once :)
<abentley> jml, that too.
<mwhudson> uh yeah, anything that talks to the db pretty much has to load the zcml
<thumper> morning
<kfogel> deryck: so, what I've learned is a) firebug swears those new statements in my patch are being executed, and b) they don't seem to actually affect the body tag in the page, and c) the statements already there, which did the same class mods for the privacy_div tag, also have no effect.  IOW, this code has never been working, even before.  There must be some additional step necessary to get the class change to take effect?  http://paste
<kfogel> .ubuntu.com/387117/
<kfogel> leonardr: know enough about javascript to know the answer to my question to deryck above?
<kfogel> leonardr: the paste is for bug 471195
<mup> Bug #471195: Private bugs don't set the body class to "private" <css> <privacy> <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/471195>
<leonardr> kfogel, give me a minute
<kfogel> leonardr: np
<leonardr> kfogel: i suspect i will not know enough about javascript; maybe you should try to contact mars in case that's true
<kfogel> mars: ayt?
<kfogel> leonardr: pinging him
<leonardr> kfogel: confirmed, i have no idea what that paste means
<mwhudson> kfogel: i'm pretty sure that calling addClass, replaceClass etc takes effect immediatly
<kfogel> leonardr: heh, np.  Basically I'm adding/removing one of the class attribute values from a document's <body> tag, but the change isn't taking effect.  Then I realized old code to do effectively the same thing also was having no effect, and that made me wonder what's missing here.
<kfogel> mwhudson: there's a replaceClass?  I might try using that; it would simplify the code.
<mwhudson> um
<mwhudson> maybe?
<mwhudson> i think so
<kfogel> mwhudson: but it's definitely not taking effect, unless there's some weird browser caching issue going on.
<kfogel> I have to manually reload the main page to get it to take effect.
<mwhudson> my knowledge of all this is pretty ancient, and not from the launchpad context
<kfogel> mwhudson: *nod*
<mwhudson> kfogel: i guess i'd try to come up with a tiny example, i.e. write some html by hand
<mwhudson> (there is a replaceClass btw)
 * mars reads the backchat
<kfogel> mwhudson: yep
<mars> kfogel, yep, start with http://developer.yahoo.com/yui/3/api/Node.html#method_replaceClass, add a debugger statement on the line before
<mars> 'debugger;'
<kfogel> mars: thanks!
<mars> replaceClass is too convenient to pass up :)
<mars> Node also grew a setContent method, but set('innerHTML'...) works too
<mars> http://developer.yahoo.com/yui/3/api/Node.html#method_setContent
<kfogel> mars: btw, I'm using http://paste.ubuntu.com/387235/ to look for stuff in Launchpad js files, with commands like 'find . -name "*.js" | xargs grep "replaceClass" | ~/bin/no-longer-than'
<kfogel> mars: otherwise you get those massive js lines that block out everything else
<mars> kfogel, ?  Are you running in devmode?  The JS should be plain, uncompressed
<kfogel> mars: i'm in devmode, but my working tree still has the compressed js files
<mars> and they are being used?
<mars> in Firebug, in the Script tab
<mars> and in the document head?
<kfogel> mars: no, AFAICT they are not.  I'm debugging with firebug and it's stepping across exactly the lines I expect it to be stepping across, in the files I expect it to be in.
<mars> ok
<kfogel> mars: I'm talking about when I'm trawling the code for examples to guide me -- not run time, coding tim.
<kfogel> time
<mars> ah
<mars> I had to create a custom search for that :/
<mars> +lib/lp, +lib/canonical/lazr, +lib/canonical/launchpad/javascript
<mars> Paul is about to fix that though
<jelmer> mwhudson: btw, bzr+bzr-git should now support enough to enable colocated branch fetching in code import
<mwhudson> jelmer: oo
<mwhudson> jelmer: something to try to get working for the next rollout?
<jelmer> mwhudson: yeah, that'd be nice
<mars> kfogel, let me know when you hit that debugger statement, then we'll use the console to test some nodes
<jelmer> mwhudson: would require an extra field in the UI for the branch name at the moment
<jelmer> mwhudson: is that what you'd want in the end too, or would you want to have that in the URL?
<kfogel> mars: sure, one minute.  and thanks.
<mwhudson> jelmer: we couldn't say git://git.gnome.org/gedit.git,feature-x ?
<jelmer> mwhudson, not yet
<mwhudson> jelmer: it would be easier if we could put it in the url
<jelmer> mwhudson, ok, in that case we should wait for a bit more
<mwhudson> jelmer: or even better, import all the branches into a colocated branch
<mwhudson> but i guess that's a little way off :)
<mwhudson> ffs
<jelmer> mwhudson: welcome back (-:
<mwhudson> jelmer: sorry, did i miss anything there?
<jelmer> mwhudson: nope
<mwhudson> k
<jelmer> mwhudson: I guess colocated branch support in one of Bazaar's formats is further away
<jelmer> mwhudson: it would require a new format as well as support in loggerhead/launchpad/qbzr/bzr-gtk
<mwhudson> yeah
<leonardr> gary, thumper, all tests pass successfully
<thumper> leonardr: awesome
<gary_poster> leonardr: fantastic
<thumper> leonardr: devel has closed
<thumper> leonardr: so land on db-devel
<leonardr> thumper, can you walk me through it? are there instructions anywhere?
<thumper> leonardr: bzr pqm-submit --submit bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel
<thumper> -m "..."
<leonardr> thumper: anything special i need to put in the -m? [rc=thumper] or something like that?
<kfogel> mars: okay, I'm on https://bugs.launchpad.dev/firefox/+bug/1 with my firebug console open, having just restarted my dev instance.  The current patch in my working tree is: http://paste.ubuntu.com/387241/ .  I am about to go to the public/private portlet and toggle from public to private.
<mup> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Invalid> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <OpenOffice:Invalid by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invalid by jahyire2006> <Ubuntu Jaunty:In Progress> <ubuntu-expre
<kfogel> mars: mup is wrong, and some day I want to fix this bug in mup :-).
<thumper> leonardr: yeah
<thumper> leonardr: [release-critical=thumper][r=gary][ui=none]
<leonardr> thumper: submitted, let me know if you see problems
<mars> kfogel, ok.  That should hit the debugger automatically then.  BTW, you do not have to restart between template and JavaScript file changes in devmode - convenient.
<thumper> leonardr: ok
<kfogel> mars: cool.  I was being paranoid.
<kfogel> mars: okay, I'm breakpointed (brokepoint?) on line 525 of that js file ("lp_bug_entry = updated_entry;")
<kfogel> about to enter the if(private) condition
<mars> kfogel, ok.  First see if it enters the conditional branch you expect
<mars> kfogel, then we will try a node test or two
<kfogel> mars: it does
<mars> ok
<kfogel> I'm now on line 528.  (So far, same behavior as I have seen before.)
<mars> kfogel, in the console, try 'Y.one('body')'
<kfogel> oh, I forgot to mention that when I started, when the bug page was "public", the body tag had the class public.
<mars> see what it returns
<mars> ok
<kfogel> mars: just type that at the >>> prompt in the console?
<mars> yep
<kfogel> mars: OH
<kfogel> d'oh
<kfogel> syntax
<kfogel> error
<kfogel> forgot my semis
<kfogel> one sec
<mars> well, JS doesn't /technically/ require them... but lets not go there.
<kfogel> mars: BODY#document.tab-bugs . main_side public yui-skin-sam yui_3_0_0-4-1267566838367281 { _yuid="yui_3_0_0-4-1267566838367281",  more...}
<kfogel> mars: looks good to my untrained eye... ?
<mars> kfogel, looks ok.  You can also try Y.one('body').getAttribute('class') to see the class list
<mars> more easily
<kfogel> oooooh
<kfogel> thanks
<kfogel> let me try it
<kfogel> "tab-bugs main_side public yui-skin-sam"
<kfogel> as I expected
<mars> or >>> d = Y.Node.getDOMNode;
<mars> >>>  d(Y.one('body'));
<mars> ok
<mars> kfogel, now try the replaceClass call, then print the node again
<kfogel> mars: I'm going to just step it, right?  (not write it out)
<mars> I wonder if Firebug supports lazy typing?  >>> Y.one 'body'
<mars> kfogel, yep
<kfogel> mars: oh frig... ffox crashing again
<kfogel> this happens like once a day now
<kfogel> freezes up my whole system too
<mars> Yep! :)
<mars> Welcome to Front End Engineering!
<kfogel> sigh
<kfogel> ok
<kfogel> mars I may disappear for a bit, back as soon as I can
<kfogel> if restarting ffox doesn't work, I will have to reboot
<mars> I still think part of the reason Chrome is popular among web devs is because it *isn't* loaded down with a dozen extensions
<mars> kfogel, k
<mars> no extensions makes for a much faster browser
<kfogel> mars:  I return.
<mars> kfogel, ok, let me know when you get the debugger context back
<kfogel> mars: you know, I wonder if it's running launchpad that kills my system.
<kfogel> mars:  I just rebooted, and I'm already back to sluggishness levels like the ones that caused me to reboot.
<kfogel> mars: s'okay, can work with this for now.  but ffox will be slow.
<mars> kfogel, odd, should be fine on a fresh boot.  How much RAM do you have?
<mars> I have 4GB.  Usually works well.
<kfogel> mars: anyway: bug is marked as public right now, body class confirmed as public, and am about to toggle to private with breakpoint set in firebug.
<kfogel> mars: I have 2G
<mars> Once every few months it hits swap
<mars> ok
<mwhudson> kfogel: going from 2 to 3 gigs really made my life better
<mars> I run between 2.1 to 3GB pretty consistently.  So 2GB may well not be enough.
<mwhudson> sad, but...
<mars> Dual core or more helps too
<kfogel> mwhudson: wow.  Only wish I had known this when I bought it.
<kfogel> mars: it's a dell xps M1330
<kfogel> dual core
<mars> should be upgradeable.  IIRC beuno runs 3GB on his
<mwhudson> i got so fed up i just walked into a computer store and paid $70 for a 2 gig stick
<kfogel> mwhudson: I think that's in my future.
<beuno> I haz 4gb
<mwhudson> speaking of which, if anyone wants a 1 gig SO-DIMM.....
<kfogel> mars: just hit breakpoint again, on the if(private)
<mars> k
<kfogel> mars: whups
<kfogel> nope
<mars> ?
<kfogel> when I toggled, the spinner spun, and then the box (in which I checked "private") got a red error that I have seen once before:
<kfogel> "The following errors were encountered: No operation name given"
 * mars shrugs
<mars> Haven't seen that one
<mars> Did it do that last time?
<kfogel> mars: not sinc once yesterday
<kfogel> mars:  sorry, delay in responding is the machine, not me
<kfogel> takes that long to switch back to this window!
<kfogel> "A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue.
<kfogel> Script: https://launchpad.dev/+icing/rev10420/yui/attribute/attribute.js:104"
<mars> kfogel, what were you doing when it raised that error?
<kfogel> mars: well, it's hard to say, because it's hard to know what events ffox heard and which ones it didn't.
<kfogel> I was trying to get the bug back to pristine state (i.e., public, with body class saying "public"), so I could repeat the toggle.
<mars> kfogel, ok, maybe comment out the debugger statement, get everything running without.  Then uncomment it, F5
<kfogel> mars: note there was never a debugger statement -- I was just using firebug
<kfogel> mars: I just had to kill the browser, though.  It was unusable.
<mars> kfogel, then why did it stop on the if (private) call?  (Or did it?)
<kfogel> mars: sorry, I havne't been able to explain the chaos over here
<mars> ok :)
<kfogel> mars: after reboot, the machine quickly became unusuable again
<kfogel> mars:  I tried to debug, but thne spent much effort just trying to get ffox to respond
<kfogel> I've killid it now
<kfogel> mars: I've shut down ffox and launchpad dev instance server.
<kfogel> mars: yet box is still crawling.  frigg.
<mars> bitten swap?
<kfogel> top shows nothing
<kfogel> mars: what does "bitten" mean?
<mars> as in, "you took a bite out of it".  Swap space is bitter stuff.
<mars> slows the whole system down, even after you have freed up memory elsewhere.
<mars> for a little while at least.
<kfogel> mars: hunh.  I don't know if I ever touched it.
<kfogel> I'm going to see if I can find a computer store.  I'll be literally unable to work until this is fixed.
<kfogel> bbiab
<mars> hehe, ok
<mars> kfogel, I may be offline by then
<mars> almost EOD here
<mars> kfogel, ping me tomorrow, I can work with you some more on this
<mars> kfogel, I'm in EST, same as you
<kfogel> mars: thanks
<kfogel> mars: still there?  I got RAMz.  rebooting again :-).
#launchpad-dev 2010-03-03
 * mwhudson lunches
<leonardr> thumper: there was a conflict between my branch and db-devel
<leonardr> http://pastebin.ubuntu.com/387304/
 * thumper looks
<leonardr> gary and i verified that the test is present in devel and not present in db-devel, but we didn't find any record that it had been _removed_ from db-devel
<leonardr> and are not sure why it would have caused a conflict
<leonardr> <gary_poster> leonardr: just to be sure, maybe bring it up to thumper?  It looks like it is in code, so he might know.  I don't see any removals of this code in February (http://bazaar.launchpad.net/%7Elaunchpad-pqm/launchpad/db-devel/changes?filter_file_id=xxbranchmergeproposa-20081124161423-sto2nyh0ew6f80p4-1)
<thumper> leonardr: if you edited something at the end of the file... ??
<thumper> leonardr: sometimes it happens
<thumper> merge in db-devel
<thumper> keep the test
<thumper> it is new
<leonardr> thumper: problem is, the test doesn't pass in db-devel, maybe due to sample data differences?
<thumper> I don't know who fixed it, but I'm happy it is :)
<thumper> um..
<thumper> what's the error?
<leonardr> thumper: all_comments['entries'] is empty
<thumper> leonardr: where is your branch?
<thumper> leonardr: I'll update db-devel
<thumper> merge in your branch
<thumper> and take a look
<leonardr> thumper: lp~leonardr/launchpad/multiversion-wadl
 * thumper postpones lunch a little
<thumper> leonardr: I don't get a conflict merging your branch into db-devel
<thumper> leonardr: have you merged db-devel into that branch and resolved the conflict?
<leonardr> thumper: yes, actually, sorry. try merging the penultimate version
<thumper> leonardr: that's fine, I'll just look at the test
<thumper> leonardr: your branch cleanly merged into db-devel for me
<thumper> leonardr: and the test passed
<thumper> leonardr: with a make schema before I started
<thumper> leonardr: not sure what to say...
<leonardr> thumper: that fits with what happened to me
<thumper> leonardr: so what is the problem?
<leonardr> thumper: gary and i thought it was weird and wanted to alert you to it, is all
<thumper> oh
<thumper> ok
<thumper> just a weird bzr merge bug I think
<gary_poster> thumper, leonardr, sorry for the ruckus, just didn't want to make a mistake.
<thumper> gary_poster: that's ok
<gary_poster> leonardr: I thought the test didn't pass for you?
<gary_poster> leonardr, thumper: so we are merging the file without the test?
<gary_poster> oh the make schema
<gary_poster> I wondered if that were it
<thumper> gary_poster: I don't think so.  I started with an up to date db-devel, merged in leonard's branch, and the test is there
<thumper> so it seems the conflict resolution is fine
<gary_poster> but I still didn't see the test in db-devel when I pulled.  but if you are happy, I am happy.  I'll run along now. ;-)
<thumper> and the test passes
<gary_poster> cool
<thumper> so I think we are good
<gary_poster> ok, thank you thumper and leonardr
<thumper> gary_poster: did you have an up to date db-devel?
<leonardr> hold on, i had the wrong view of an ambiguous statement
<leonardr> "the test"
<leonardr> let me restate what happened
<thumper> lib/lp/code/tests/../stories/webservice/xx-branchmergeproposal.txt
<thumper>   Ran 23 tests with 0 failures and 0 errors in 5.421 seconds.
<gary_poster> thumper: yes
<leonardr> i merged db-devel into my branch and saw the conflict in xx-branchmergeproposal
<leonardr> the conflict was a set of assertions about merge proposal comments
<leonardr> i removed it and then 'the test' (xx-branchmergeproposal) ran
<thumper> leonardr: you removed this? http://pastebin.ubuntu.com/387304/
<leonardr> yes
<leonardr> if i did not remove it, xx-branchmergeproposal would not run
<leonardr> i'm fairly sure this was after a make schema (becuase the test wouldn't run at all without a make schema)
<leonardr> but i'll try again
<thumper> hmm...
<gary_poster> thumper, I do not see that snippet in my copy of db-devel, which is fresh
<thumper> leonardr: so to confirm, your branch as I merged it doesn't have this test?
<gary_poster> I also did not see it in loggerhead
<leonardr> thumper: yes
<gary_poster> getting loggerhead link
<leonardr> thumper: ok, wait
 * thumper looks at devel
<leonardr> the branch at lp:~leonardr/launchpad/multiversion-wadl does not have the test, because i had to remove it to merge in db-devel
<gary_poster> http://bazaar.launchpad.net/%7Elaunchpad-pqm/launchpad/db-devel/annotate/head%3A/lib/lp/code/stories/webservice/xx-branchmergeproposal.txt
<gary_poster> thumper: I do not see that snippet in loggerhead link above, nor in my fresh copy of db-devel
<thumper> ok, this IS an issue
 * thumper adds the test and tries again
<thumper> ah
<thumper> dumb arse
<thumper> it was my change
 * thumper smacks forhead
<thumper> forehead
<thumper> yes it is correct to have it remoevd
<thumper> I removed it!
<thumper> d'uh
<thumper> the db-devel branch as descriptions for merge proposals
<thumper> the initial comment doesn't exist any more
<thumper> we're golden
<thumper> sorry for my confusion
 * leonardr considers quitting irc before another reversal
<thumper> the reason you got the conflict is that you changed some text in that block
<thumper> for the new api url
<leonardr> yeah, i changed beta to devel
<thumper> and I deleted it on db-devel
<thumper> right
<thumper> that is why there was a conflict
<thumper> it is correct for it to not be there
<thumper> on db-devel
<leonardr> ok, i just got email that it's been merged with pqm
<thumper> sweet
<gary_poster> leonardr, thumper thanks :-) running away also.
<thumper> ok
<adiroiban> hi. any idea what should I do to fix this error: AssertionError: No IEntry adapter found for IPOTemplate.
<adiroiban> i'm trying to export IPOTemplate via API
<james_w> heh
<james_w> adiroiban: jml was just asking a very similar question a few hours ago
<james_w> adiroiban: what code changes have you made?
<adiroiban> export_as_webservice_entry() in IPOTemplate
<adiroiban> end marked a simple TextLine as exported
<james_w> hmm
<james_w> if you just make the first change does it give the same error?
<adiroiban> and this one http://paste.ubuntu.com/387327/
<adiroiban> without the changes from http://paste.ubuntu.com/387327/, there is no error, but the langauge is not exported
<adiroiban> s/language/template/
<adiroiban> it looks like the template API needs to be referred by some other property in order to be exported
<james_w> yeah, I just wondered if it was the export_as_webservice_entry() was enough to cause it
<wgrant> adiroiban: Can you pastebin your diff and the error?
<wgrant> lazr.restful is very good at confusing people.
<adiroiban> error http://paste.ubuntu.com/387331/
<adiroiban> diff http://paste.ubuntu.com/387332/
<james_w> adiroiban: try dropping the export_as_webservice_collection() line
<james_w> and the @collection_default_content() one too
<adiroiban> james_w: I will try... but those are for IPOTemplateSet ... not for IPOTemplate
<james_w> yes
<wgrant> But the first references IPOTemplate.
<adiroiban> same error
 * wgrant pokes around.
 * wgrant waits for buildout to do whatever it does.
<adiroiban> lazr.restful is resposible for creating the IEntry adapter?
<wgrant> create_as_webservice_entry() should do everything.
<wgrant> Er, export_as*
<adiroiban> for me lazr.restful is still a voodoo afair
<adiroiban> if I'm not using all_potemplates from distroseries, there are no errors
<wgrant> Yeah.
<adiroiban> so I guess the IPoTemplate is not read
<adiroiban> since it is not listed in apidoc
<adiroiban> wgrant: do I need to include IPOTemplate in lib/lp/translations/interfaces/webservices.py ?
 * wgrant blinks.
<wgrant> Aaaaaaaa.
<wgrant> Yes.
<wgrant> Everything I've ever touched has been in canonical.launchpad.interfaces, so I didn't know about that registration.
<adiroiban> what is that? :) I have exported the API for ILanguage, which is in lib/lp/services/worlddate
<wgrant> But it looks like Translations has removed their stuff from there.
<adiroiban> and I did not have to added it
<wgrant> Right, it's imported in lib/canonical/launchpad/interfaces/__init__.py, where all of LP's interfaces used to be available.
<wgrant> canonical.launchpad.interfaces is webservice-registered.
<wgrant> But Translations stuff appears to have been largely stripped out of c.l.i (as everything eventually should be).
<adiroiban> ok
<adiroiban> I added IPotemplate
<adiroiban> and it is not listed in apidoc
<adiroiban> and wadl file
<wgrant> You've removed and regenerated the WADL and apidoc?
<adiroiban> yes
 * wgrant tries.
<wgrant> It's there for me.
<adiroiban> wgrant: sorry
<wgrant> (but as p_o_template, specify export_as_webservice_entry('po_template') for it to be more sane)
<adiroiban> s/not/now/
<wgrant> Ahh.
<adiroiban> same here... p_o_templates
<adiroiban> I will go to sleep now
<adiroiban> many thanks for your help
<wgrant> Remember to fix that name at some point.
<wgrant> np
<adiroiban> james_w: has jml fixed his no IEntry addapter problem?
<james_w> I believe so
<james_w> his was actually a different cause I think
<adiroiban> aha
<james_w> putting this on the wiki somewhere would be a good idea
<adiroiban> james_w: yes. I'm adding it right now
<adiroiban> at the Prerequisites
<wgrant> Yeah, I've never seen that documented anywhere.
<james_w> \o/ adiroiban
<wgrant> Only Code and Translations have their own registrations
<adiroiban> wgrant: yep.
<adiroiban> maybe we need to ask on ML
<wgrant> Why?
<adiroiban> why the other modules are not using webservice.py
<wgrant> Their exports predate c.l.i's deprecation.
<wgrant> they should probably use it.
<wgrant> They shoul dprobably create and use their own modules, that is.
<wgrant> So we can kill off c.l.i eventually.
<adiroiban> ok
 * thumper afk for a bit
<thumper> back for a bit
<adeuring> GOOD MORNING
 * adeuring should remove the caps lock key
<spm> :-)
<mwhudson> adeuring: or rebind it to control, as god intended
 * wgrant wonders why there is still a schoolbell directory in the tree.
<lifeless> delete i t.
<lifeless> see what happens
<wgrant> It's used by c.l.components.cal, which itself doesn't seem to be used.
<wgrant> In fact that module shouldn't even import.
<lifeless> dead code.
<lifeless> delete.
<lifeless> have vcs, can undelete.
<wgrant> Yeah, the only reference is in a module that doesn't work any more. How odd.
<lifeless> it was imported for the ill fated calendar support that jamesh put together
<lifeless> which would have been really nice, but wasn't given enough dev time
<lifeless> [heard that before? :P]
<wgrant> Yeah, I know why it *was* there. I remember that poor old feature.
<lifeless> seriously.
<lifeless> delete. klesscode.
<jamesh> how much of the calendar code is left in the tree?  I remember it was just hidden for a while (along with the bounty system)
<lifeless> jamesh: dunno, possibly lots.
<lifeless> hey, I'll be able to run lp on my new laptop ;)
<jamesh> couldn't you run it on your old one?
<lifeless> nup
<lifeless> only got 2GB of disk, and only 2GB of ram
<wgrant> jamesh: It's gone.
<lifeless> with all its overhead I'd be silly to try
<wgrant> Except for this one odd file with two classes.
<wgrant> lifeless: As long as you're 32-bit and not doing stupid Soyuz stuff, 1GiB is doable, 2GiB is fine.
<wgrant> (RAM, that is)
<mrevell> Morning
<jamesh> lifeless I used to do LP development in 2GB of ram on a 64 bit system
 * bigjools struggles with 2GB on a 64-bit box
<lifeless> wgrant: 64bit
<wgrant> I think bounties are gone as well.
<wgrant> bigjools: Yeah, I had to upgrade.
<bigjools> I'm getting a new Stinkpad soon
<bigjools> with 4GB and an SSD :D
<wgrant> Oooh SSD.
<wgrant> I wonder how much happier that makes the test suite.
 * bigjools does a little dance
<lifeless> jamesh: yeah, its got heavier AFAICT; and my evo + browser chew up tonnes
<bigjools> ask thumper
<lifeless> bigjools: me too :)
<wgrant> lifeless: Evo eats LOTS.
<lifeless> bigjools: x201s, 8GB, 128SSD
<bigjools> lifeless: you already have one?
<bigjools> I'm getting a T410
<lifeless> bigjools: ordered today.
<lifeless> bigjools: i7 640LM
 * bigjools ordered a week ago
<bigjools> i5 though :)
<thumper> bigjools: a little faster
<thumper> not run everything though
<bigjools> I couldn't find a way of avoiding the Windows tax though, it makes me sick
<jamesh> lifeless: that does sound like a nice machine.  Looks pretty expensive though...
<wgrant> bigjools: How are Windows refunds with UK Lenovo?
<lifeless> jamesh: massive discount
<bigjools> wgrant: I dunno, I will try
<bigjools> it's not the refund as such, it's the bloody sales figures that sticks in the throat
<lifeless> bigjools: i5 ~= i7, need to check specific models and chipsets to compare in more detail.
<bigjools> i5-540M
<wgrant> bigjools: Hopefully refund figures count for something.
<jamesh> lifeless: more than the discounts that they always seem to claim are in effect?
<bigjools> I doubt it
<wgrant> Are UK models as ridiculously overpriced as Australian ones?
<bigjools> probably, we pay 17.5% VAT here
<bigjools> however I got an awesome deal, ended up saving ~GBP300
<lifeless> jamesh: yeah
<lifeless> jamesh: ends 5th march, 25% off
<jamesh> lifeless: I guess I'll have to watch for specials when I get round to looking for a new laptop
<lifeless> jamesh: would have been really quite cheap if I didn't get the 8GB ram and the SSD.
<lifeless> jamesh: if your 6 year anniversary happens in < 3 months, you could do it now :)
<bigjools> I paid GBP1200, would have been ~1000 without the SSD
<jamesh> lifeless: it's not quite that close
<lifeless> bigjools: I think we'll both be happy
<lifeless> bigjools: I wanted the small form factor
<bigjools> yeah I like my 14 inches
<bigjools> </quote>
<lifeless> :>
<lifeless> I'm happy with 12.1"
<lifeless> I used to want a lot more.
<jamesh> lifeless: I wish they went for a thinner bezel and smaller size for the newer thinkpads
<wgrant> My 14" isn't too bad, but I would have preferred a 12" or 13" I think.
<bigjools> it's not what you have it's what you do with it
 * bigjools likes WXGA+
<lifeless> jamesh: I'll have to show you the 201s, its 0.8" high with screen closed
<wgrant> jamesh: The T400 screen bezel is asymmetric and HUGE.
<jamesh> lifeless: that sounds like an improvement over my X60s
<deryck> Morning, all.
<adiroiban> danilos: hi. can we schedule a pre-implementation chat for bug 525371 ?
<mup> Bug #525371: API for reading POTemplates details <api> <Launchpad Translations:Triaged by adiroiban> <https://launchpad.net/bugs/525371>
<danilos> adiroiban, sure, let me review the languages API first :)
<adiroiban> ok
<gmb> james_w: Did you get success and failure emails about your branches that I ran through EC2 yesterday?
<james_w> gmb: I did not :-(
<gmb> james_w: I did, so I'll forward them to you. I guess ec2 test doesn't look up the email addresses to which to send the messages like ec2 land does.
<james_w> thanks
<james_w> gmb: cool, so is the SUCCESS one landed?
<gmb> james_w: No, because we're in PQM freeze for release. I'll land it for you as soon as PQM reopens (ping me on Friday if you haven't heard from me about it).
<james_w> aha!
<james_w> I'll fix up the other one
<james_w> who knew there were actually tests for sync-source.py
<jelmer> james_w: there was one
<james_w> oh, I managed to break that one
<jelmer> james_w: and there were more that tested a bit of code that was only used by the tests, not actually by sync-source.py (-:
<lifeless> jelmer: hey
<lifeless> jelmer: #tr - please answer, I'm waiting on you before crashing.
<jpds> Anyone know why I would get ImportError: No module named tickcount - on lucid running make run?
<maxb> jpds: dpkg -L python-tickcount ?
<maxb> Does it mention a /usr/lib/pyshared/python2.5/tickcount.so ?
<jpds> 0.1-0ubuntu10launchpad1
<jpds> Yep.
<maxb> Can you python2.5 -c 'import tickcount' ?
<jpds> No.
<maxb> hrm
<maxb> oddness
<maxb> ls -la /usr/lib/pymodules/python2.5/tickcount.so
<maxb>  ?
<jpds> No such file.
<maxb> Are you fully updated from the launchpad PPA?
<noodles775> Ah, I had this issue after upgrading to lucid...
<jpds> maxb: Just ran rocketfuel-setup.
<noodles775> You just need to re-run `sudo apt-get update/upgrade` after rocketfuel-setup.
<noodles775> (so it pulls in python-support from the lp ppa)
 * noodles775 runs to lunch
<maxb> Do that. If it doesn't work, 'sudo update-python-modules -v python-tickcount.public' should manually poke python-support into reinstalling the symlinks for that package
 * bigjools stabs the branch scanner
<james_w> how do I rollback a transaction in bin/harness?
<salgado> james_w, transaction.rollback() or abort()?
<james_w> ah, abort works, thanks
<james_w> odd that it's named that way
<salgado> yeah, I never remember which one is the correct
<bigjools> james_w: fwiw, bin/iharness is far nicer to use
<james_w> why doesn't make harness run that then?
<salgado> james_w, make iharness does
<james_w> gmb: tests fixed for the second branch and devel merged, so should be good to go when PQM re-opens as well
<gmb> james_w: Cool, thanks.
<danilos> adiroiban, btw, I won't have time to do a pre-imp for potemplate API today; if you want we can do it tomorrow or you can try getting hold of either jtv or henninge
 * beuno hugs bac 
<bac> beuno: and i didn't even have to make stuff up!
<beuno> bac, nobody else has to know that  :)
<adiroiban> danilos: ok. no problem.
<danilos> adiroiban, I'm going to go through the languages API branch now, sorry for being late and lazy
<jml> hey guys
<jml> didrocks & I are trying to expose GPG keys over the API
<jml> that means we need to give them a URL
<intellectronica> jml: gmb just worked on a similar thing. maybe you can find his branch for reference
<intellectronica> but basically you need to add a url in zcml, and traversal code
<jml> any thoughts on https://launchpad.net/+gpgkey/KEYID
<jml> intellectronica, that's good to know -- I'll look for that
<jml> intellectronica, any thoughts on the proposed URL?
<intellectronica> jml: i'm not sure i understand the question
<intellectronica> well, i'm sure i don't
<jml> intellectronica, heh ok
<jml> intellectronica, we can give GPG keys any URL we want. we have to make a decision about what the URL will be
<intellectronica> jml: is that the GPG keys for a person?
<jml> intellectronica, yes.
<jml> intellectronica, although all GPG keys are tied to a person, afaict
<gmb> jml: See http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/revision/9053 for reference to how I exposed TemporaryBlobStorage via the API.
<intellectronica> if so, then how about lp.net/~person/+gpg-keys ?
<jml> intellectronica, we need a URL for the individual objects
<intellectronica> jml: maybe use the key's fingerprint?
<jml> you mean like "https://launchpad.net/+gpgkey/KEYID"?
<intellectronica> so ~person/+gpg-keys/AB12045
<intellectronica> yeah, key id, terminology...
<jml> why put it under ~person?
<BjornT> jml: what's the url of an EmailAddress?
<kfogel> deryck: so chrome hangs forever trying to visit https://bugs.launchpad.dev/firefox/+bug/1  (and shut up, mup!), but ffox can visit it.  Go figure.
<mup> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Invalid> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <OpenOffice:Invalid by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invalid by jahyire2006> <Ubuntu Jaunty:In Progress> <ubuntu-expre
<jml> BjornT, good question!
<kfogel> deryck: oh, I think it's b/c the bug is in private mode right now
<kfogel> and chrome didn't have the cached creds
<BjornT> jml: another question: does it make sense to access the collection of all gpg keys?
<jml> BjornT, I don't think so, but I wasn't planning on exposing IGPGKeySet
<jml> BjornT, actually...
<intellectronica> jml: because it was uploaded by a person and uniquely belongs to that person, no?
<intellectronica> do we allow more than one person to upload the same key?
<jml> BjornT, being able to fetch a key by id sounds like a useful thing, but it's yagni for me now.
<jml> intellectronica, we don't. I checked.
<BjornT> jml: also, consider how a new key is added, or an old one removed.
<intellectronica> jml: in that case i think it makes sense for it to dangle from a person
<intellectronica> jml: i would also make it possible to iterate through all the keys for a person
<jml> intellectronica, yeah, that's what we've implemented right now
<intellectronica> by exposing a collection
<jml> intellectronica, we just need need to give the bloody things a URL :)
<intellectronica> so it makes sense that if you iterate over ~person/+gpg-keys every item will be of the form ~person/+gpg-keys/KEYID
<jml> there's also a pendinggpgkeys collection
<jml> for each person
<jml> BjornT, how is that relevant?
<jml> BjornT, I mean, it's definitely an interesting thing to consider :)
<jml> intellectronica, do you think it makes sense to iterate over ~person/+pending-gpg-keys in the form of ~person/+gpg-keys/KEYID?
<jml>     "gpgkey_fingerprint_key" UNIQUE, btree (fingerprint)
<jml>     "gpgkey_owner_key" UNIQUE, btree (owner, id)
<jml> those are the constraints
<intellectronica> jml: yes, i think it does. keys are globally unique, but in launchpad they belong to a person and it never makes much sense to refer to them without the context of a person
<didrocks> jml: I'm wondering, how do you "ack" a key so that it goes from pending to accepted key? isn't the "send an email thing and decrypt it?" (it's been years I've done that, I don't remember)
<jml> didrocks, I don't know.
<jml> didrocks, my brain is big enough to fit one problem at a time :)
<didrocks> jml: it's big enough to understand LP's mind, mine apparently not ;)
<jml> didrocks, I use my hind-brain for that.
<didrocks> heh
<intellectronica> jml: i think you need to encrypt your key and paste the result somewhere. but also been a while since i've done that
<intellectronica> i mean encrypt _with_ your key
<kfogel> adeuring: how do you run a windmill test?  bin/test -v -v -t TEST_BASENAME doesn't seem to do it.
<intellectronica> kfogel: https://dev.launchpad.net/Windmill
<adeuring> kfogel: as intellectronica says ;)
<kfogel> intellectronica, adeuring: d'oh.  THank you.
<kfogel> intellectronica, adeuring: just out of curiosity: no one would object if "bin/test -t NAME_OF_WINDMILL_TEST_FILE" did the right thing, would they?
<adeuring> kfogel: no, not really ;)
<intellectronica> kfogel: no, but you'd need to record somewhere which domain the test should run on
<kfogel> intellectronica: well, or that could be deduced from the path to the file
<adeuring> kfogel: but the test setup for windmill is quite complex, so it might be better to have another script that decides if that is needed for a given test and either starts the windmill stuff of invokes the oridinary test
<kfogel> adeuring: not sure I understand.  Basically, right now we have to specify "--layer=FooWindmillLayer", but that information could be deduced from the test every (?) time.
<kfogel> That is, the Foo could be deduced, and the need for the --layer at all could be deduced.
<jml> intellectronica, BjornT: thanks.
<adeuring> probably.
<kfogel> adeuring: hmm, I keep getting cert error popups, and no way to say "it's okay"
<adeuring> kfogel: congratulations, that's why I like windmill tests ;) wait a second, I can send you my recipe to fix this by mail
<kfogel> adeuring: ...and I'll add it to the Troubleshooting section :-)
<adeuring> (but that's not the reocmmended way)
<kfogel> adeuring: and I can't get my prompt back -- it traps C-c, even though the test windows are gone.
<adeuring> kfogel: really? that's new for me...
<kfogel> adeuring: killed the python process, now I've got it back.
<adeuring> kfogel: mail sent
<kfogel> thanks adeuring
<kfogel> adeuring: hunh, this time when I ran it I got no cert errors, even though the previous time I'd never accepted the cert (I accidentally hit "Get me out of here" instead of "I understand the risks").  How bizarre.
<adeuring> kfogel: indeed...
<kfogel> intellectronica: I'm in lib/lp/bugs/windmill/tests/test_bug_privacy_settings.py, hoping to add a line that tests that the body tag's class lost "public" and gained "private" after the bug is marked private (I would do this around line 81).  Not sure how to ask for the value of the class attribute of a particular tag, though.  I'm reading at getwindmill.com, but if you know of any examples, please say.
<intellectronica> kfogel: windmill is a bit of a black art. i'm not sure there's a way. if there isn't what you'll have to do is use the javascript validator, which allows you to run some javascript code and return a bool. unfortunately the js code in the validator doesn't have access to yui, so to determine whether you have the class you'll probably have to grab the attribute, parse it and check that it contains (or doesn't) the class you're interested in
<kfogel> intellectronica: *nod*  I've seen some code in YUI that does that itself, can use that as a guide (YUI's implementation of removeClass, I think).
<intellectronica> kfogel: something like `my_element.getAttribute('class').split(' ').indexOf('my-class') < 0`
<kfogel> intellectronica: when you say "use the javascript validator", where's an example of that?  I see all these validator=foo in the code, but they mostly seem to be in some weird hybrid language I don't recognize.  Is that some form of javascript?
<intellectronica> kfogel: no, most validators use some very weird syntax that's windmill specific, but there's one kind of validator that allows you to pass js code. let me find an example
<intellectronica> kfogel: it's client.asserts.assertElemJS
<kfogel> intellectronica: thanks
<intellectronica> kfogel: you can see a usage example in test_official_bug_tags_management.py
<Ursinha> barry: ping
<BjornT> kfogel: fwiw, testing that the body tag classes sounds like something that is best done in a JS unit test, not in Windmill.
<intellectronica> BjornT: how can you do that without encapsulating the entire body in a js widget?
<kfogel> BjornT, intellectronica: watching your conversation
<intellectronica> well, i suppose you could just test the privacy widget. there's only one <body> above it, so it's safe
<BjornT> intellectronica: well, i said it sounds like it would be better to use a JS test, i didn't say it would be easier :) i wouldn't think that you need to encapsulate the entire body in a js widget, but it might require some re-design of how the privacy control is hooked up. i haven't looked at the code, and have no intention of doing it right now :)
<intellectronica> kfogel: but though i agree with BjornT that it would be nicer, i think it would be _a lot_ less work to just add that one test to the existing windmill test, so i'd recommend doing that for now
<intellectronica> BjornT: well, as i added later, it's actually not that hard to do, with body being singular. but unless you insist i'll recommend to kfogel that he use windmill for now, and we'll think on how to split testing between windmill and unit tests better in the future
<BjornT> intellectronica: no, i don't insist, that's why i said "fwiw"
<Ursinha> hi sinzui, is bug 531371 a rollout blocker?
<mup> Bug #531371: oops MailingListAPIView email already in use <mailing-lists> <oops> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/531371>
<kfogel> intellectronica, BjornT: heard, understood :-).
<sinzui> Ursinha: no, it is just an oops, and I fixed it with the help of chex
<Ursinha> sinzui: right, thanks
<adiroiban> is there a special setup for running windmill tests? I get TranslationsWindmillLayer Error: no display specified
<intellectronica> adiroiban: are you trying to run the test headless?
<intellectronica> it sounds like the test runner is complaining that it doesn't have an x server to use. usually running the tests on your desktop your desktop's x server will be in use
<intellectronica> i never had to setup anything especially for that
<adiroiban> intellectronica: i'm running the test on a remove machine
<adiroiban> via ssh
<intellectronica> adiroiban: well, that would explain the problem you're having
<intellectronica> there's some way to run the tests headless, but i don't remember off the top of my head how
<adiroiban> ok
<adiroiban> well, I can start X on that machine and set the DISPLAY variable
<intellectronica> yes, that should work, but since you're not going to see anything anyway why not just run headless
<adiroiban> but I was thinkint that bin/test is smart and can deal with this situation
<intellectronica> smart it's not :)
<adiroiban> i can not find any headless option in bin/test --help
<kfogel> intellectronica: how does this look to you?  (it passes)  http://paste.ubuntu.com/387755/
<sinzui> salgado:  do we get to close bug 2164 when the release is complete, do we retarget the bug to canonical-identity-provider
<mup> Bug #2164: Login form should ask for e-mail address only once <ui> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/2164>
<salgado> sinzui, I guess we can mark it as invalid or fix released at that point, but it certainly should be closed as the new form doesn't ask for the e-mail twice
<sinzui> salgado: so you can assign it to yourself, mark it fix released and get the glory
<sinzui> and the karma
<salgado> yay!
<kfogel> adeuring: (see above msg to intellectronica, if you have time to review a small windmill patch)
<salgado> sinzui, are you going through the login/registration/reset-password bugs and checking which ones we can close after the roll out?  if not maybe I should be doing that
<sinzui> salgado: I am going through the UI bugs that gary says he does not want. I think 1/3 should be closed or moved to a different project
<sinzui> salgado: I certainly will search for bugs that we can close because of the login change
<gary_poster> thank you sinzui! jml ^^^
<jml> sinzui, thanks.
<EdwinGrubbs> sinzui: do you know where the view.page_title gets set to the view.label if the view.page_title does not exist? The page works on launchpad.dev, but create_initialized_view().render() fails on view/fmt:pagetitle
<sinzui> EdwinGrubbs: that means someone must add a page_title to the view. barry did not create a fallback and I reported a bug about that. we have lots of views that does this
<sinzui> label = 'me'
<sinzui> page_title = label
<EdwinGrubbs> sinzui: I don't understand why this doesn't also cause an error with "make run".
<sinzui> EdwinGrubbs: because the error is in tales which is a runtime issue
<EdwinGrubbs> sinzui: I mean after I use "make run", I load the page, and it displays the page_title as the reverse breadcrumbs.
<sinzui> right
<sinzui> tales is compiled and executed at request time
<sinzui> EdwinGrubbs: I do not understand the problem. When I get an error with fmt:pagetitle, I add a page_title to the view and move on
<sinzui> EdwinGrubbs: remember that launchpad/pagetitles.py should not exit. It still does because we have not really complete the 3.0 update
<sinzui> s/exit/exist/
<EdwinGrubbs> sinzui: ok, I was assuming that launchpad.dev should have the same errors as the doctests, but I guess the doctests are more anal so that you will provide a more interesting page_title that the one it defaults to in launchpad.dev.
<sinzui> right
<intellectronica> kfogel: i have time now if you still need a review
<kfogel> intellectronica: hey, yeah, actually review and/or questions
<kfogel> intellectronica: basically, see http://paste.ubuntu.com/387755/
<kfogel> intellectronica: since then, I've tried to consolidate the code a bit:
<kfogel> intellectronica: I'll make a new paste, hold on
<intellectronica> kfogel: my guess is in HAVE_PRIVATE_CLASS you want to use >= rather than >, no?
<kfogel> intellectronica: so the above one succeeds, yet this fails:
<kfogel> http://paste.ubuntu.com/387786/
<kfogel> intellectronica: oh, good idea (though coincidentally it doesn't matter right now because that class val is never first, but yes
<kfogel> )
<intellectronica> kfogel: also, for readability, i would use """ instead of concatenating strings.
<kfogel> intellectronica: just being consistent with the other vars in thehre...
<kfogel> intellectronica: as soon as windmill finishes here, I'll show you the puzzling failure that results from the second paste
<intellectronica> kfogel: i'm surprised the id you pass works. does the body tag actually have an id of "document"?
<kfogel> intellectronica: I also wanted to consolidate the element.getAttribute('class').split(' ') into a variable and just use that variable, but stopped on this failure.
 * kfogel waits for windmill
<intellectronica> kfogel: the fact that you pass an elements to the validator doesn't mean that you have to use it. i'm sure document.body.getAttribute() will work just fine
<kfogel> intellectronica: oh, interesting, ok
<kfogel> intellectronica: ok, so this did NOT fail just now: http://paste.ubuntu.com/387790/
<kfogel> intellectronica: but, I'd still like to abstract out the repeated bit of code in that js string
<kfogel> I tried "var classes = element.getAttribute('class').split(' '); "
<kfogel> and I tried "var classes = element.getAttribute('class').split(' '), "
<kfogel> neither worked
<kfogel> (then I used "classes.indexOf" below, of course)
<mwhudson> good morning
<kfogel> mwhudson: morning
<intellectronica> kfogel: i guess the problem is what you pass is an expression, so if you want to have statements in it you need to wrap them all in a function and call it immediately
<kfogel> intellectronica: *nod*
<kfogel> intellectronica: let me try that
<intellectronica> kfogel: something like `(function() {var classes = "blah blah"; return classes.split(' ')[0]})()`
<kfogel> intellectronica: ah, anon function called immediately, got it
<kfogel> intellectronica: feeling all lisp-nostalgiac now
<intellectronica> :)
<intellectronica> js really is scheme with c syntax
<intellectronica> is there some magical way to get the context given a request (in code that doesn't have direct access to the context)?
<rockstar> Windmill should really be renamed to Sandman, because it eats my Memory, slows everything down, and will likely kill me.
<jtv> salgado, can you give me some feedback about l.c.webapp.tales._get_custom_icon_url?  ISTM it's unnecessarily fetching context.icon from the db just to figure out whether one has been set.  I think that's what we're seeing in some very sporadic timeouts.
<jtv> (Yes, that does imply there are worse problems on the page, but I've already addressed those and I think this would make a nice extra :)
<jtv> rockstar: I don't know what the sandman eats, but otherwise you've pretty much described a millstone.
<rockstar> jtv, if you haven't read the graphic novel "Sandman" I highly recommend it.
<jtv> rockstar: another one for the list of "things I'd love to follow up on in principle but will sadly never get around to," I fear.  But I do appreciate it.  :-)
<jtv> intellectronica: I don't suppose you could dig up the view?
<intellectronica> jtv: no, that's precisely my problem
<salgado> jtv, it returns the URL of the context's icon, so I guess it needs to fetch the icon's details (but not the actual icon, which should live in the librarian) from the DB
<intellectronica> it's in deeply nested code that doesn't have access to the view
<jtv> intellectronica: then that sounds like one of those "don't want to do whatever you're trying to do" cases  :/
<salgado> intellectronica, IPerson(self.request.principal)
<jtv> salgado: it returns the icon's URL if there is one, but plenty of projects don't have one
<intellectronica> oh how i miss the good ol' days of the launchbag
<salgado> intellectronica, ignore that; I misread context as user.  (somehow)
<intellectronica> salgado: oh, but what do i do with the person? the context i'm looking for isn't a person
 * jtv backspaces what he was saying to salgado
<intellectronica> heh
<jtv> salgado: ah, now I see...  the fetch is only not needed when it's None
<salgado> when the context is None or doesn't provide IHasIcon
<jtv> salgado: thanks, it seems discussing things really helps.  :)
<kfogel> intellectronica: https://code.edge.launchpad.net/~kfogel/launchpad/471195-private-bug-body-class-private/+merge/20590
<kfogel> intellectronica: (btw, I tried document.body.getAttribute("class") and it didn't work -- returned null.  But grabbing it by id is okay too, I think, and it allows us to use the same code to test the body and the privacy div.)
<intellectronica> kfogel: cool. if it works it works. the code looks fine. r=me
<kfogel> intellectronica: thanks
<kfogel> intellectronica: now I must debate with myself whether to put it through ec2 land on the theory that by the time the tests are done, PQM will be un-gated :-).
<kfogel> intellectronica: oh, I think for ec2 land to DTRT, you have to actually mark the approval on the MP, right?
<intellectronica> donnow
<intellectronica> but i have approved the MP anyway
<intellectronica> kfogel: anyway i don't think by the time ec2 will finish pqm will have already opened. that probably only happens tomorrow, no?
<kfogel> intellectronica: Oh, I thought it happened once the rollout was done, but could be wrong.  I'll just do ec2 test.
<deryck> Later on everyone...  tea and my comfy chair are calling me! :-)
<kfogel> intellectronica: ever seen this? http://paste.ubuntu.com/387817/
<kfogel> intellectronica: (I'm just testing some ec2 land stuff with --dry-run; I intend to run ec2 test later)
<intellectronica> kfogel: sorry, was on the phone with my mum
<intellectronica> anyway, no i never saw that error
<kfogel> intellectronica: np.  ignoring it, since I'm doing ec2 test anyway
* Chex changed the topic of #launchpad-dev to: Launchpad will be down/in read-only from 23:00 UTC until 00:30 UTC for a code update | Launchpad Development Channel | Launchpad read-only 23.00 UTC Wed 3 March - 00:30 UTC Thu 4 March, for 10.02 rollout (http://is.gd/9tVfU) | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review i
<lifeless> kfogel: around?
<lifeless> bug 252954
<mup> Bug #252954: renaming non-versioned directory in Eclipse causes error <Bazaar Plugin for Eclipse:Fix Released by verterok> <https://launchpad.net/bugs/252954>
<lifeless> bug 252945
<mup> Bug #252945: "bzr push" uploads 1-2 MB just to send a one-line change revision <hpss> <push> <Bazaar:Fix Released> <https://launchpad.net/bugs/252945>
<kfogel> lifeless: looking
<lifeless> that one
<kfogel> 252945
<lifeless> how can we get savannah to move on this
<lifeless> yes
<kfogel> lifeless: they need more hands on deck, basically.  It could be any of us, though of course someone experienced at bzr would be best.
<kfogel> lifeless: I just added a comment to that bug giving details.
<lifeless> I don't think any of us know what their issue is;
<lifeless> they have ssh, just enable bzr, is what our reaction is.
<beuno> "Launchpad will be going offline for maintenance in a minute."
<beuno> sounds very generic  :)
 * mwhudson afk to get lunch, will be hanging around near irc though if things go belly up
<thumper> mwhudson: if it goes horribly wrong, you'll get a text if it is you, otherwise prolly not
<thumper> :)
<mwhudson> thumper: fairy nuff
<thumper> mwhudson: we should test lp with twisted 10
<mwhudson> thumper: funnily enough i have a branch that does just that
<mwhudson> thumper: i ran the tests with 10.0.0pre1, they all passed
<thumper> mwhudson: awesome
#launchpad-dev 2010-03-04
* Chex changed the topic of #launchpad-dev to: Launchpad will be down/in read-only from 23:00 UTC up to 02:00 UTC for a code update | Launchpad Development Channel | Launchpad read-only 23.00 UTC Wed 3 March - 00:30 UTC Thu 4 March, for 10.02 rollout (http://is.gd/9tVfU) | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review i
<jml> thumper, mwhudson: you should upgrade :)
<mwhudson> jml: week 4!
<jml> mwhudson, oh yeah :(
<jml> we're going to fix that some day soon
<jml> actually, I should ask gary about that.
<jml> he was going to do it as a part of an "only roll out QAd changes" mechanical solution
<jml> they're orthogonal, but he wanted to use the former as a carrot for the latter
<jml> turns out the latter is really really hard
<mwhudson> jml: is this related to MergeWorkflowDraft ?
<jml> mwhudson, that's Bjorn's one right?
<mwhudson> jml: yeah
<jml> mwhudson, I think it's different.
<jml> anyway, really bed time
<jml> (why do all you kiwis work such insane hours!)
<mwhudson> launchpad is behaving very strangely during this rollout
<wgrant> Timing out, OOPSing, saying it will be going down really really soon, then read-only, then timing out again...
<mwhudson> wgrant: and despite saying "You cannot make any changes.", i can see all the editing options
<wgrant> mwhudson: YOu're not logged in, are you?
<wgrant> It's kicked me out again.
<wgrant> So it shows all the actions.
<wgrant> Even though launchpad.Edit and launchpad.Admin have been revoked from every object.
<lifeless> jml: because we're committe?
<lifeless> d
<lifeless> d
<lifeless> d
<mwhudson> i'm logged in
<wgrant> Hm.
 * mwhudson off for a bit
<wgrant> How is the internal community commit/review privileges discussion going?
* Chex changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.02 | PQM is in release-critical mode (thumper is RM) | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
<thumper> mwhudson: https://code.edge.launchpad.net/~kiko/linux/2.6.31
<thumper> mwhudson: first pass succeeded
<mwhudson> thumper: woo
<mwhudson> thumper: it's still going to take aaaaagees
<thumper> mwhudson: yep, but at least it won't die part way through
<thumper> just take ages
<mwhudson> yeah
<jtv> stub: I cheated
<stub> I'm dobbing
<jtv> stub: I *did* have a look at your branch
<jtv> dobbing?
<stub> strine
<jtv> oh, sorry mate, I don't speak that
<stub> 'informing the authorities', normally parents
<jtv> apart from "beware drop bears near billabong"
<jtv> ah
<jtv> what in plain English we would call "ratting you out"
<jtv> *drat
<jtv> forgot to bring "dropbeertjes" from Holland
<jtv> (beertjes == little bears)
<jtv> stub: _anyway_...  the new feature looks cooler than I could have imagined.  I suspect we'll have to learn about or build a whole new set of tools to figure out where the savings are at this level of the stack.
<jtv> (Although the hottest-bug listings come to mind :-)
<stub> We have tools to measure page generation times by analizing the zserver trace logs. Run the report, land your tweak, run the report a day later. If things look better, its a win.
<stub> I'm thinking off adding cache_stores, cache_hits & cache_misses to opstats today. If the hits are low we are choosing badly. But we will have to find out what 'low' is by trial an error.
<jtv> But "land your tweak" also involves convincing a reviewer that you're doing something useful, which you'll only really find out in the next stage.
<stub> No, convincing the reviewer it is a good experiement.
<spm> losa cowboys on staging?
<stub> It really needs edge - not enough genuine traffic on staging to generate genuine reports.
<spm> point
<jtv> spm: good point, but a bit unreliable in terms of performance and staging updates
<spm> jtv: unreliable staging updates would work IN your favour here :-)
 * jtv suspects spm of having his IRC client beep on the word "strine"
<jtv> spm: unless they suddenly _do_ happen, is my point :)
<spm> was the word 'beer' actually... in '"dropbeertjes"'
<stub> What we have to avoid is using the cache to optimize particular pages. We add caching to a page template, it actually makes the initial rendering slower. So this is purely about increasing overall performance - reducing the average time of a particular class of pages or perhaps freeing up resources for other stuff.
<jtv> spm: funny because it's quite probably true
<jtv> stub: that was one question I had: how sensitive is the caching to exact path & query parameters?
<stub> So make an educated guess, then measure the results over a day.
<stub> It is very sensitive - the getKey() method encapsulates all this. query parameters or url changes, cache is not reused.
<jtv> good to know
<spm> the only danger i'd see in this process approach, is late in a cycle. we really wouldnt want experimental code accidently getting rolled to prod
<wgrant> Sounds like MergeWorkflowDraft needs to be undraftified.
<jtv> Would this also be a useful hook for profiling the fragments?
<stub> I suspect it will be hard to make pages perform worse. It might be possible to put lots of pressure on the cache slowing things down, but we can fix that by increasing our cache size. But we shall see once I can land and do some real world tests.
<jtv> wgrant: thou speakest in riddles, verily as Greek
<spm> stub: please, for my sanity, do now throw down challenges like that. folks have a way of trying to achieve them. (....hard to make pages perform worse)
<jtv> stub: is there a don't-cache-at-all mode to give us more flexibility there?  Especially handy with a hypothetical pofiling option
<stub> jtv: Sure. You shutdown memcached.
<lifeless> fragment caching++
<lifeless> ESI yaaaay
<jtv> stub: I was thinking of a slightly finer granularityâas an addition to the list of caching modes for a given fragment
<stub> Its similar to ESI, except we don't have to work out htf to get page templates to generate non-wellformed XML fragments and have access control so 'fred' doesn't see 'bobs' cache when we want that.
<jtv> End System Identifier?
<jtv> ElectroStatic Interference?
<stub> edge side includes
<stub> Squid tech
<jtv> ...and there goes a new GTF entry.
<stub> (and devs don't need to run squid locally)
<wgrant> Just memcached :P
<jtv> lifeless, stub: gtf entry #23517, with thanks
<mup> Bug #23517: adept is not translated in french <kde-i18n-fr (Ubuntu):Fix Released by carlos> <https://launchpad.net/bugs/23517>
<stub> Chewing up a whole 1MB of ram for its cache!
<jtv> no mup, stay out of it
<stub> jtv: esi is at least 6-7 years old...
<lifeless> stub: well, on the squid side, the main pt is ~= an ESI processor anyway
<jtv> stub: well, I didn't have it yet
<wgrant> Is the TALES extension going to land soon?
<lifeless> Not sure about the xml angle; dunno why you would care about welformedness per se;
<stub> As soon as pqm opens
<lifeless> fred vs bob we should solve anyway
<wgrant> Excellent.
<lifeless> Vary is generally sufficient
<jtv> stub: status on the mp is still Needs Review IIRC
<stub> lifeless: PageTemplates is cool provided you want to generate well formed XML. If you want to generate anything else, your shit out of luck.
<lifeless> stub: I get that; I don't get why that is an issue for ESI
<wgrant> Why are there multiple Linux imports going?
<wgrant> Wouldn't it make approximately several hundred times more sense to import the latest branch, stack the rest on it, and have all of them imported in only slightly longer than the last branch?
<lifeless> wgrant: ask kiko :P
<stub> lifeless: You are right. I had a different use case when I first looked at it, and for it to be useful to me then I would have needed to cache non-wellformed XML fragments due to page layout issues (cause we still had to use tables back then for any non-trivial page layout).
<lifeless> stub: ahha! mystery explained :)
<stub> You can't cache non-wellformed fragments with the TALES syntax either
<stub> (cause they are generated with ZPT)
<jtv> wgrant: that does sound like it makes several hundred times more sense, but maybe (and I really mean I don't actually know) the resulting number is considered less than the sense index for other things that could be done with the engineering time.
<jtv> wgrant: we get similar questions about translations imports sometimes
<wgrant> jtv: It doesn't take any engineering time, though.
<wgrant> It's not an engineering issue.
<wgrant> Somebody just clicked Retry on too many imports.
<jtv> wgrant: you're saying somebody expended effort to produce needless work for the server?
<wgrant> jtv: It looks that way. The imports should have been suspended for months/years.
<jtv> wgrant: they're that old?  Did somebody revive an entire category of old imports to apply a bug fix or something?
<wgrant> jtv: One has been suspended for four months. I don't know why they're active again.
<jtv> wgrant: al-maisan landed a branch this cycle that IIRC suspends imports when disabling an... archive I believe.  Maybe that's related.
<wgrant> jtv: There was a 10.01 branch that suspended binary builds when disabling archives.
<wgrant> Is that what you're thinking of?
<jtv> oh, last cycle, OK
<jtv> Probably it.  I reviewed the branch, but can't say all the details stuck.
<jtv> stub: I'm trying to simulate read-only mode in a view unit test.  Do you know how I do that?  I tried setting test_request.annotations['launchpad.read_only_mode'] to True but that doesn't do the trick.
<jtv> And I don't know whether zope will give is_read_only the same request that the test set up for the view.
<jtv> lifeless: can squid reuse a single connection to a server between requests from different clients?
<lifeless> yes
<lifeless> be a pretty freaking terrible proxy that can't do that
<jtv> So if the connections live long enough, I guess a local proxy (with the necessary setup to deal with https) could save me some ssl connections on multi-browser LP testing.
<jtv> Oh, but it'd all be one session to LP, wouldn't it
<jtv> ...would it?
<lifeless> one ssl session
<lifeless> != one lp session
<lifeless> lp doesn't see ssl
<jtv> lifeless: I was wondering if LP sessions were designed on the assumption that nobody's proxying the SSL sessions.
<lifeless> we proxy them
<jtv> If they weren't, then great
<lifeless> apache does SSL unwrapping
<lifeless> backends speak http
<jtv> So it all ends up in the same stomach after you swallow it anyway.  Gotcha.
<jtv> (And the stomach can deal with that fact and fails to go antiperistaltic when it sees chocolate ice cream juxtaposed to haddock)
<lifeless> only if the haddock is deep fried
 * jtv checks
<jtv> blast.  no haddock.
<stub> jtv: I don't know sorry. When I left it, it was a config variable. Now we switch using a .txt file, but I'm not sure how it works in tests.
<jtv> stub: I don't see a lot of code using is_read_only; we do in a few places to smooth over some bits that can oops out in read-only mode, and maybe that's just weird of us.
<jtv> stub: I think I'll try reproducing the request-checking logic from is_read_only first... maybe it returns nothing at all and then I'll have to get into the function's head in a different way.
<jtv> morning al-maisan!
<al-maisan> Good morning jtv, how are things :)
<stub> I would think the process would be construct your request, set the read only annotation on it, invoke the publication machinery with that request.
<jtv> al-maisan: things are a lot warmer here than in Amsterdam, thank you :)
<stub> (because the publication machinery is what needs to detect read only mode to install the read only database policy that triggers your bug)
<al-maisan> jtv: we had the first lunch on our terrace yesterday :)
<stub> Which might be a saner way of constructing your test - just install the read only database policy and see if the code works.
<al-maisan> jtv: It was a bit chilly but nevertheless we ate outside :)
<jtv> al-maisan: I had a hard disk temperature warning the other day where the temperature was lower than the one on my Gnome weather applet.  :)
<al-maisan> jtv: oh :)
<jtv> stub: it's based on a text file afaics... though there's also a global variable.
<jtv> al-maisan: to be fair, it was a "temperature has been too high in the past" warning
<stub> jtv: Yes, but what that text file does is tell the publication machinery to install the read only database policy
<jtv> bon dia dpm
<jtv> stub: ah ok
<al-maisan> jtv: pheww ;)
<dpm> hey jtv, bon dia :)
<jtv> "based on periodic measurements taken from my server's on-board sensors, global warming is going to kill us all within the coming week"
<stub> This is LaunchpadDatabasePolicyFactory in canonical.launchpad.webapp.dbpolicy, which is what is called when the publication machinery does IDatabasePolicy(request)
<jtv> stub: thanks
<al-maisan> jtv: Oh well, we have nothing to worry then ;)
<jtv> al-maisan: not until about halfway through the week, and for different reasons, not after the coming week :)
<al-maisan> jtv: maybe you should pursue career options in astrology? ;)
<jtv> stub: the app code checks is_read_only() itself though, which wouldn't be fooled that way... is that wrong?
<stub> jtv: You would need to do both then. Maybe that can be simplified.
<jtv> al-maisan: "Pluto is in the house of...  Pluto is not in anybody's house, because nobody lives at the inner edge of the Kuyper belt.  Sorry."
<al-maisan> :-)
<jtv> stub: actually there's no reason to have the db policy at all, as long as I can convince is_read_only to return True for a moment.
<stub> jtv: Quick hack would be to make is_read_only() return true if the currently installed database policy is LaunchpadReadOnlyDatabasePolicy
<stub> jtv: Ok. I would have thought you would need the database policy installed to trigger the error, which you would then fix possibly using explicit checks to is_read_only() and choosing a different code path.
<stub> Ideally, we would never need that sort of check but I suspect there are a few valid use cases buried in these 200k lines of code somewhere...
<jtv> stub: in this case, checks to is_read_only deeper down in the call tree upset the logic that tries to describe to users what they can and cannot do with given translations.
<jtv> stub: "can't do X, can't do Y, the usual explanation fails, BOOM."
<jtv> I don't just want to replace the BOOM with "oh, we must be in read-only mode" and risk getting it wrong.  If worse comes to worst, I'll either create the read-only file (won't be cleaned up if you trip over your power lead during that split second though) or add a way to make is_read_only lie to you.
<stub> Make it lie if tests need it, and LaunchpadFunctionalLayer can provide the API to twiddle it.
<stub> The 'you can't do that because we are in read only mode' exception should already generate a 'you can't do that now' error page. Perhaps some code is swallowing exceptions it shouldn't be?
<stub> Oh,... you are looking before you leap, so yeah.
<stub> Nobody else worries about that - read only mode is rare.
<jtv> stub: looks like is_read_only is merely getting a different request object than I'm creating, so I can work with that
<jtv> BTW I am seeing some repeat requests in these oopses, so somebody is clearly coming away with the impression that Launchpad doesn't work for them.  So might as well fix them.
<stub> I think the error page is 'you can't do that now, maybe later' so it is reasonable for users to hit reload a few times (more reasonable to have a cup of tea between reloads though).
<jtv> stub: that's not the case here... it's an assertion failure on a non-form page
<jtv> So somebody visits a page and just gets an oops.
<stub> Sounds like the problem is they get an oops instead of a ReadOnlyModeViolation exception getting raised and rendered by the error handler.
<stub> You might be papering over the real bug
<jtv> stub: no, because it's a read-only, non-form page.
<jtv> Just visiting the page should not be a mode violation.
<jtv> Somebody visits a page and just gets an oops.
<stub> I understand that, but it still shouldn't be generating an OOPS.
<stub> In fact, I can't see how it would oops due to being in read only mode if the page is genuinely read only. But I haven't seen the OOPS report.
<stub> Oh... because lower level code manually checks is_read_only() and returns values that other code can't handle?
<jtv> 'zacly
<stub> So you shot yourself in the foot ;)
<jtv> stub: we work as a team.  So we shoot each other in the foot, which is a lot easier to do.
<stub> synnergy
<jtv> and occasionally, gangrene
<jtv> al-maisan can praise himself lucky he missed that bit of TMI
 * al-maisan resists the urge to ask what he missed .. :P
<jtv> al-maisan: good.  Just be grateful your connection blinked.
<al-maisan> :)
<jtv> mwhudson: can tests access branches by their http URLs in any way?
<adeuring> good morning
<jtv> good morning adeuring
<adeuring> hi jtv!
<jtv> thumper: can you tell me of a way in which a test can access a branch by its http URL?
<thumper> jtv: http://bazaar.launchpad.net/~user/project/name
<jtv> thumper: in a test?
<thumper> jtv: not at this time of the night
<jtv> thumper: I'll take that as "too hard to test" then.  Thanks for responding though!
<thumper> jtv: take it as "I'm tired"
<jtv> thumper: ok, I'll go bother North-Americans when they get here.  Thanks.
<wgrant> jtv: What're you trying to do?
<jtv> wgrant: trying to test the bit where a buildfarm slave retrieves the contents of a branch through http
<wgrant> jtv: I think you are violating the spirit of our poor slave code by adding tests to it.
<jtv> wgrant: good point, thanks bye!
<jtv> wgrant: then again, it wouldn't be a slave if its life was meant to be easy
<wgrant> jtv: Heh.
<jtv> Right now I'm testing it with a whatever-URL-LP-wants-to-give-for-it, but that little bit of extra realism would be nice
<wgrant> HTTP branch access normally goes through Apache, doesn't it? I don't think it works for tests.
<jtv> wgrant: that's what I think too, but I feel I would be remiss if I didn't at least look for a way.
<wgrant> jtv: Indeed.
<mrevell> Morning
<mwhudson> jtv: no :/
<mwhudson> jtv: in the testrunner config, branch.warehouse_url is a file:/// url that can be accessed
<mwhudson> (with luck and a following wind, this area of the testing is pretty horrible)
<jtv> mwhudson: I figured.  Accepted risk of this part of the code, I suppose.  But had to try.  Thanks!
<mwhudson> jtv: at some point we'll maybe come up with a testing http server that we can use in tests
<mwhudson> but it doesn't exist yet
<jtv> mwhudson: given what the weather was like in NZ this "summer," I can see that you'd want the extra dissipation from your CPU
<wgrant> I think I've almost dried out from that lovely Saturday.
<mwhudson> jtv: summer waited until the day after lca
<mwhudson> it was a gorgeous day today
<mwhudson> but gosh yeah, the weekend between the sprint and the conference was pretty special
<jtv> mwhudson: I didn't.  Which is a shame because someone I missed at both LCA _and_ FOSDEM (note opposite sides of globe) had business calling me recently
<jtv> To be fair, FOSDEM was colder than LCA.
<jtv> Barely.
<jtv> Mostly because I wasn't wearing my coat when I went outside.
<mwhudson> i took this yesterday: http://www.flickr.com/photos/mwhudson/4402652072/
<wgrant> I thought it would be reasonable to walk between the hotels on Saturday. But no, no, it had to start ABSOLUTELY POURING when I was a few minutes out.
<mwhudson> well, it doesn't actually ever get hot in wellington
<mwhudson> (max temp ever recorded: 29 deg C)
<wgrant> Nice.
<jtv> mwhudson: remind me to drop by sometime...  Shame my next round-the-world flight goes the other way around
<jtv> mwhudson: wgrant and I met in the street that day because we were blown along different roads towards the same intersection
<wgrant> Ah, yes, that's right.
<jtv> wgrant: good thing you managed to retract that elbow
<jtv> If I'd flinched I wouldn't have been able to grab the lamppost.
<wgrant> Heh.
<jtv> mwhudson: I can't begin to explain how exotic that view is to someone from a country where a subliminal slope in the landscape is counted as the national Mountain.
<mwhudson> jtv: wellington is VERY like that
<mwhudson> jtv: thailand has lumpy bits though, doesn't it?
<jtv> mwhudson: to me wellington looks a lot more like your picture than like where I come from
<mwhudson> jtv: i mean, very not flat
<mwhudson> sorry, bit unclear
<jtv> quite :)
<mwhudson> emma's workplace is 1000m across, 200m down
<jtv> the most densely populated parts (such as "here") are pretty flat
<jtv> What parts get flooded has more to do with the state of the sewage system than elevation.
<wgrant> That seems to be the case in most places
<wgrant> But Wellington's founders were clearly crazy.
<deryck> Morning, everyone.
<mrevell> Guten morgen deryck
<bigjools> eyup deryck
<adiroiban> BjornT: hi! do you know how can I run windminll test in âheadlessâ mode ?
<deryck> intellectronica, what happened to your branch for better queries when calculating max_bug_heat?  Did it make the rollout?
<intellectronica> deryck: it landed, but didn't make the rollout
<intellectronica> iiuc it's designated for a reroll or cp?
<deryck> intellectronica, that's what I thought, too.  Just didn't see the card on the kanban board.
<intellectronica> deryck: it seems cards disappear from the rightmost columns? i definitely moved the card there last night
<intellectronica> are you or someone else moving the cards, or is there some automated process?
<deryck> intellectronica, ah, you had it in done-done?  I moved all of them to archive late yesterday, with the release out of the way.  no worries.
<deryck> intellectronica, or maybe someone came behind me as part of the release and moved your card, if it was the lone card there.
<BjornT> adiroiban: the easiest is to use xvfb-run. e.g. "xvfb-run -s '-screen 0 1024x768x24' bin/test -t test-to-run"
<adiroiban> BjornT: well, I'm trying to produce an testing environment similar to ec2 on one of my remote computers
<adiroiban> on which I have ssh and root account
<adiroiban> ok. so I have to manualy start the X framebuffer
<wgrant> 'make check' will use xvfb-run, won't it?
<bigjools> DISPLAY= make check
<bigjools> much easier :)
<didrocks> jml: ok :)
<jml> hey folks
<didrocks> hey guys :)
<jml> didrocks & I are working on exposing some API stuff to make it much, much easier to opportunistically write Ubuntu apps
<leonardr> salgado, jml, do you know if there is a reason why the launchpad web service does not publish public ssh keys? is it for security reasons?
<leonardr> it came up twice yesterday
<jml> leonardr, no, there isn't. I'm actively helping didrocks expose them :)
<leonardr> ok, awesome
<jml> leonardr, it's just https://launchpad.net/+people/me/+sshkeys after all
<didrocks> there should be a branch hanging somewhere :)
<jml> but!
<jml> what is perhaps more interesting is exposing adding new SSH keys via the API
<jml> and adding new GPG keys
<BjornT> adiroiban: ec2 uses xvfb-run. you can look in test_on_merge.py, which is what ec2 runs.
<leonardr> jml: yeah, i meant the functionality to add the keys as well as look at them
<didrocks> (for the note, that's the hardest part for Quickly user)
<didrocks> not uploading the right gpg key, and so onâ¦
<jml> leonardr, well, I guess the question is how much easier OAuth is to hack than whatever the browser normally does
<jml> leonardr, a question I'm almost wholly ignorant of.
<adiroiban>  thanks!
<jml> so where were we
<jml> ahh yes.
<jml> we want to add SSH keys so that people can upload branches
<jml> and we want to create the user's first PPA, and guide them through everything up to that point
<jml> so we've exposed GPG keys and SSH keys...
<persia> Users still just get/push public GPG keys and LP checks the keyserver, right?
<jml> I don't know
<james_w> no
<jml> At least we need to link a GPG key to an account and to sign the CoC
<persia> Please confirm.  Lots of folk set expiration dates in keys and then republish updates if they don't wish them to expire, which is supported now, and would break if that wasn't still true.
<james_w> you decrypt a piece of text LP sends to you to get a string out of it which you give back to LP
<persia> james_w: No?  Was that way in November (last time I updated my key)
<jml> persia, if that's a genuine concern then send an email please, I've got too much on write now to get distracted on another thing
<jml> so, my questions...
<jml> what's the right internal api for adding GPG keys?
 * persia will just test and file a bug if it breaks, expecting that it would only break from a structural change
<james_w> https://help.launchpad.net/YourAccount/ImportingYourPGPKey
<james_w> jml: you mean internal API, or exposed API?
<didrocks> james_w: yeah, I remember about that. I think we can pause and tell user "check your email, past what's in it there" (and Quickly does the decrypt and send it)
<jml> james_w, internal
<james_w> jml: that I do not know
<james_w> didrocks: yeah, that's probably about the best you can do absent events/callbacks in the API
<jml> didrocks, I guess for this we should just expose an API that takes a fingerprint and does whatever https://launchpad.net/people/+me/+editpgpkeys does
<didrocks> james_w: jml: agreed.
<jml> james_w, statik has started some work on events/callbacks
<didrocks> (so, quickly should upload the public key in the ubuntu keyserver first, not sure about the delay before refreshing, but I'll take that by experience :))
<jml> didrocks, yeah, that sounds right.
<jml> didrocks, so, I guess that means we should add a 'registerGPGKey' method to IPerson
<didrocks> jml: oki
<jml> didrocks, I don't know what to do with the codeofconduct stuff
<jml> didrocks, nor with "creating a PPA"
<jml> but we can figure those out, or maybe bigjools will hepl
<didrocks> jml: right, I think first target is gpg and ssh key
<bigjools> jml: are you talking about adding a PPA through the API?
<didrocks> (in any case, for the CoC, Quickly should still present this with a "do you agree?" before upload the signature)
<didrocks> bigjools: yeah
<bigjools> I have always avoided doing that because it's at risk from rogue scripts creating loads of junk PPAs
<didrocks> bigjools: the issue is that a lot of Quickly users don't get how to create one and also some others app are doing screen scrapping for that IIRC
<didrocks> but I still understand your concern, and that's why I still think project creation should be manual and don't intend to automatize it in Quickly :)
<bigjools> yeah - would we allow api creation of people etc?  same problem. and opens things up nicely for spammers.
<jml> bigjools, aren't they already at risk from screenscrapers?
<bigjools> yes but that involves more effort
<jml> what's the cost of a PPA with nothing in it?
<bigjools> risk of spam descriptions cluttering the search
<bigjools> also
<bigjools> once you create one you can't change your name
<bigjools> so I want people to see the big fat warning on the web ui
 * bigjools -> food
<jml> bigjools, we could work around those problems first by deleting inactive empty PPAs and secondly by encouraging clients to warn their users.
<bigjools> ppa deletion is a whole 'nother kettle of fish :)
<jml> didrocks, ok let's keep going :)
<didrocks> jml: oki
<jml> didrocks, we need to find the code behind the editgpgkeys page
<jml> didrocks, and then push it down into the model and then expose it.
<didrocks> jml: make sense
<jml> didrocks, ValidateGPGKeyView in canonical.launchpad.browser.logintoken
<jml> the name "logintoken" scares me a little, I don't know how it fits in...
<jml> _getGPGKey gets the key from the server
<didrocks> jml: sorry, still grepping (my VM is very slowâ¦)
<jml> didrocks, fix your install :)
<didrocks> jml: if only I would have been able to install on my lucid box at Portland :)
<jml> didrocks, heh
<didrocks> jml: ok, got it :)
 * jml is getting his zope migraine again
<jml> didrocks, afaict, 'validate' is called first
 * james_w hands jml an IAsprin
<jml> didrocks, and then 'continue_action_gpg' is called
<jml> hahaha
<didrocks> james_w: oh, you have a collection of IAsprin_set so? ;)
<jml> didrocks, the interesting methods are _getGPGKey and _activateGPGKey
<didrocks> jml: ok
<jml> didrocks, they've got way too much logic for browser code imho.
<jml> didrocks, so I guess when I get back from the half-hour meeting I'm about to go to, I'm going to move them mostly into model code... maybe as methods on IGPGKeySet or something
<didrocks> a lot of intercall hidden functions in that file :)
<jml> yeah
<jml> and too much state stored on objects as a substitute for passing parameters :(
<jml> be back soon
<didrocks> jml: have a good meeting! Thanks again :)
<adiroiban> using lazr.restful, can I export as a webservice operation, a method decored with @property ?
<gmb> Argh. Is there something that I have to do to get store.flush() to actually work in a test? For some reason the objects that I create before doing store.flush() don't get stored unless I do a transaction.commit().
<Ursinha> hey gmb, I see this oops happened during the rollout, do you know what could have caused that? https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1523EC1637
<gmb> Ursinha: Er... I think you're asking the wrong guy. That looks like a Soyuz oops.
 * noodles775 looks
<bigjools> ARGH
<Ursinha> gmb: I don't actually know if that's Bug's, Soyuz' or foundations
<bigjools> soyuz
<gmb> \0/
<gmb> I mean, too bad.
<bigjools> :p
<Ursinha> lol
<bigjools> another release-critical for me
<Ursinha> bigjools: there are 23 recorded oopses of that kind
<bigjools> not surprised
<bigjools> I can fix it
<Ursinha> it seems all while the rollout was on
<bigjools> easy fix - dunno how it slipped through
<bigjools> hmmm
<Ursinha> bigjools: let me check again
<Ursinha> found 40 more
<bigjools> Ursinha: the same url does not oops any more
<Ursinha> bigjools: yes, I saw that too
<bigjools> I think I know why it happened
<bigjools> the database changed and edge was still running old code
<bigjools> so it's a rollout issue
<Ursinha> bigjools: that's why I thought that could be a foundations issue
<bigjools> Ursinha: could be, I don't know the exact details of how rollouts are done
<jml> didrocks, I'm a loser.
<didrocks> jml: no, just very busy :)
<jml> salgado, hi. I'm looking at the GPG key stuff in browser/logintoken
<jml> salgado, what is a logintoken?
<salgado> jml, a temporary place where we store things that need validation (e.g. email addresses and gpg keys) before they can be added to their tables
<jml> salgado, ahh, I see.
<jml> salgado, so "context.consume()" means "remove it from the table"
<salgado> the logintoken is also what's used to handle the validation
<salgado> almost. it means mark the token as consumed so that it can't be used again
<jml> salgado, that makes sense
<doctormo> jml: are you working with didrocks to add the ssh key management?
<jml> salgado, there seems to be a fair chunk of gpg key validation code in browser/logintoken
<jml> salgado, any objections to me moving some of the logic to the model class?
<didrocks> doctormo: right, jml is making the gpg part and I try to adapt it to ssh :)
<jml> doctormo, yep. working on GPG right now.
<jml> doctormo, didrocks has a branch that does the beginning of the SSH key stuff.
<jml> doctormo, lp:~jml/launchpad/expose-gpgkeys-bug-389872 for the GPG work
<doctormo> didrocks: In my code I download existing ssh public keys to compare them with local keys.
<jml> man, I still need to file an FFe for Twisted 10.0
<salgado> jml, nope, that should be ok, but you might want to check with registry as I think that's their code, no?
<jml> salgado, I'm checking with you because your name is all over the 'bzr blame' output
 * didrocks was sure jml would choose the "blame" alias :)
<salgado> jml, not for the gpg-specific bits, I'd expect.  I've always managed to stay away from that code
<jml> heh heh
<jml> didrocks, I don't have time to type out anno...
<jml> didrocks, the equivalent SSH code is in...
<jml> didrocks, lib/lp/registry/browser/person.py, class PersonEditSSHKeysView
<jml> didrocks, I think the add_ssh and remove_ssh methods could be split and put into model code too.
<didrocks> jml: so, I should pull first, right? You changed that?
<jml> didrocks, I'm going to go head-down to do the GPG stuff, if you were here in person I'd talk it through ...
<didrocks> oh no, ssh
<jml> didrocks, no, unchanged.
<didrocks> ok
<jml> didrocks, ask me any questions, I'll ping here as I make progress.
<didrocks> jml: ok, I've to push some packages into ubuntu right now, but I stick around (will do the ssh part later)
<Ursinha> d/3
<Ursinha> #fail
<deryck> intellectronica, I can't find that card or any record of it -- the one you just landed to db-devel yesterday.
<deryck> intellectronica, I'll just add a new card to "Landed."  You didn't QA this yet, right?
<intellectronica> deryck: I QAd this by cowboying it onto staging
<intellectronica> maybe i'm wrong about the card, but i was pretty sure i did something with the ban when it landed
<intellectronica> deryck: should i create a card to put in the archive?
<deryck> intellectronica, yeah, create a new card, but put it in Done-Done, please.
<Ursinha> intellectronica: hi :) today I'm choosing random bugs' people to ask about oopses :) I'm not sure if this oops is a known issue: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1524C2016
<asabil> hi all
<intellectronica> Ursinha: i'm not sure either, and can't look immediately (in the middle of a review and i have a call in 2m). promise to get back to you later
<Ursinha> intellectronica: ah, ok :) I can ask in the prod. meeting, don't worry
<Ursinha> :)
<Ursinha> intellectronica: thanks anyway
<Ursinha> allenap: (or intellectronica :), could you help me with the oops I mentioned above, some time today?
<allenap> Ursinha: Do you mean the one you asked about in the prod meeting?
<Ursinha> allenap: yeah - I've asked intellectronica a bit earlier today too
<jml> everything is far too complex.
<intellectronica> Ursinha: otp, will have a look soon
<jml> there's a comment here that implies that it's possible to active a GPG key without being logged in.
<jml> that seems odd to me.
<intellectronica> jml: it makes sense in theory, since gpg in itself can provide the necessary validation
<intellectronica> maybe it can be done via email?
<jml> intellectronica, it's in browser code
<jml> intellectronica, which doesn't _necessarily_ mean it's not done by email
<intellectronica> jml: well, cherche le pagetest
<jml> intellectronica, my French is weak. Does that mean "sucks to be you"?
<intellectronica> :D
<intellectronica> jml: just in case this was a series question, the phonetic similarity should have been a giveaway. it means 'look for the ___'.
<jml> intellectronica, it's not a serious question :)
<jml> I might even bust out the word "cognate" to prove it, if called upon to do so.
<intellectronica> Ursinha: i don't know what's the deal with that oops, and i don't want to push it back, but it looks like an authentication bug, rather than a bugs bug
<intellectronica> Ursinha: is there a bug for it?
<Ursinha> intellectronica: not yet, I wasn't sure if that was a known bugs problem or a new foundations one
<intellectronica> Ursinha: my guess is foundations, but feel free to subscribe me to the bug just so that i can follow up on it later
<Ursinha> sure, thanks intellectronica
<intellectronica> sorry i couldn't help more
<jml> phew
<deryck> intellectronica, I moved your confirmation dialog bug into WIP but blocked, citing other work had to be done.
<deryck> intellectronica, but I think we should consider it like this, to force us back to it sooner rather than later.
<deryck> ok?
<intellectronica> deryck: cool, makes sense
<deryck> intellectronica, excellent.
<jml> didrocks, I have pushed up some code
<jml> still no exposed API
<jml> but maybe the diff means something.
<didrocks> jml: ok, I'll have a look, thanks  :) /me really busy untilâ¦ I don't know :)
<jml> didrocks, ok. cool. I'll keep plugging away at this.
<didrocks> jml: thanks :)
<intellectronica> flacoste: just a quick reminder that you've got an r-c request from me. another bug heat branch for the re-roll.
<flacoste> intellectronica: ok, i'll look at it soon
<intellectronica> cheers
<leonardr> gary, bug #532025
<mup> Bug #532025: Hard-coded version names in old scripts are duplicated in new versions of lazr.restfulclient <lazr.restfulclient:New> <https://launchpad.net/bugs/532025>
<gary_poster> ah, I see leonardr
<leonardr> gary: i don't think it's terribly urgent, but i could fix it in a couple hours
<gary_poster> leonardr: I'm in favor of doing it now.  Please put it on the kanban board.  Force it in (we will be out of the limits)
<leonardr> ok
<leonardr> gary: ok, i've got another bug w/r/t the fake launchpadlib browser which i'm filing now
<gary_poster> ok cool
<allenap> Ursinha: Just read the scrollback. I'm at the same conclusion as intellectronica re. the OOPS from earlier. If you have filed a bug for it, or will, please can you subscribe me too?
<Ursinha> allenap: sure
<leonardr> gary, the unpleasant truth is in bug 532055
<mup> Bug #532055: Trusted credential-management apps are broken and may be doomed <launchpadlib :New> <https://launchpad.net/bugs/532055>
<Ursinha> allenap: have you seen this issue? bug 532078
<mup> Bug #532078: Count of bugs with patches is wrong, +patches shows different information <Launchpad Bugs:New> <https://launchpad.net/bugs/532078>
<mwhudson> good morning
<Ursinha> morning mwhudson
<gary_poster> thanks leonardr.  flacoste, when you have a moment, you might be interested in looking at https://bugs.edge.launchpad.net/launchpadlib/+bug/532055 also (if you haven't already ;-) )
<mup> Bug #532055: Trusted credential-management apps are broken and may be doomed <launchpadlib :New> <https://launchpad.net/bugs/532055>
<mars> deryck, ping
<deryck> mars, on call
<mars> ok, np
<deryck> mars, off now, what's up?
<mars> hi deryck, just wondering where the +patches view template is.  Trying to track down the revision popup positioning bug.
<deryck> mars, I believe it's bugtarget-patches.pt in lip/lp/bugs/templates
<mars> my copy of trunk may be messed up, files missing
<deryck> kfogel could confirm this.  ^^
<deryck> heh
<deryck> kfogel wasn't here, I'd guess, mars. :-)
<mars> kfogel / kfogel_, what was the name of the new +patches view template?
<kfogel> deryck, mars: hi
 * kfogel reads
<kfogel> mars: uh, one sec
 * deryck spins with all the kfogel*
<kfogel> bugs-patches-view.pt I think
<kfogel> mars: ^^
<deryck> kfogel, bugtarget-patches.pt maybe?
<kfogel> deryck: yeah, kfogel_ is from my under-surgery ubuntu box, whereas this one (talking now) is from my Debian lifeboat box.
<kfogel> deryck: oh, that might be right
<mars> lol
<kfogel> mars: ^^
<kfogel> mars: is it missing for you??
<deryck> mars, it's only in db-devel for me, but I haven't updated all day.
<mars> kfogel, yes, but I may have a serious branching issue at play instead
<kfogel> mars: ok
<mars> deryck, oh?  db-devel only would explain it :)
<mars> I'm looking in devel
<kfogel> mars: but it should have been merged over, no?
<mars> kfogel, did you land your changes on devel, or on db-devel?
<mars> but yes, with the rollout, devel and db-devel should be in sync
<deryck> mars, kfogel -- perhaps we're waiting on a re-roll to merge db-stable back into devel?  Not sure how we normally do this honestly.
<mars> bzr pull db-devel spat out 48 text conflicts.  Didn't know pull could do that...
<mars> deryck, kfogel, I would like to rule out branch insanity on my end first, before delving too deep.  I'm using the +patches code as a lifeline - it's on edge, so it should be in devel
<deryck> mars, I just updated and it's not in devel for me either.
<kfogel> mars: landed on db-devel
<kfogel> mars: all that work was on db-devel, because it involved db changes.
<mars> kfogel, ok.  My understanding was that a db-devel landing shows up on staging, but not on edge.  *If* I'm remembering correctly, we did that for the YUI3 upgrade.
<kfogel> mars: I believe that is correct.
<mars> ok, so first I will rule out db-devel insanity on my end...
<kfogel> mars: dev.launchpad.net/Trunk, but I think you're right -- db-devel changes show up on edge and main launchpad when we do a new release.
<kfogel> which we either have done, or are in the middle of doing :-).
<mars> right
<kfogel> mars: devel always goes to edge; db-devel goes to staging.
<kfogel> mars: just to be pedantically repetitive and prolixly verbose about it.
<mars> so +patches is visible on edge, therefore...
<kfogel> mars: therefore I think it must be in devel?
<kfogel> mars: is it visible in non-edge?
<mars> I am not insane, this exists: https://bugs.edge.launchpad.net/gwibber/+patches
<mars> I am not insane, this also exists: https://bugs.launchpad.net/gwibber/+patches
<thumper> guys, db-devel hasn't been merged back into devel yet
<mars> thumper, so we're in limbo!
<thumper> mars: go write some docs or something :)
<mars> well, this means something was chewing up my local launchpad branches.
<kfogel> thumper: I take it PQM is still gated then?
<thumper> kfogel: yep
<kfogel> CUZ I'M IN UR PQM WAITIN TO LANDZ
<kfogel> ok
<kfogel> I guess that branch will wait :-)
<sinzui> adiroiban: I replied to bug 531261. There are some places in the code that may show you how to make ISeriesMixin complete
<mup> Bug #531261: Move ISeriesMixin to lp.registry.interfaces.series <cleanup> <tech-debt> <Launchpad Registry:In Progress by adiroiban> <https://launchpad.net/bugs/531261>
<adiroiban> sinzui: ok. I will try to move them.
<adiroiban> I was also checking parent attribute, but distroseries already have the âparentâ attribute and it returns the parent distribution
<sinzui> adiroiban: so we need to add a parent to productseries?
<adiroiban> sinzui: productseries already has a âparentâ and it return self.product
<sinzui> so we can move attrs like bug_supervisor to the mixin right?
<adiroiban> sinzui: ah.. sorry. It looks like both âparentâ implementations are ok
<adiroiban> sinzui: yes. we can move them in the mixin
<adiroiban> I will try to move as much as posible and will come back with questions/issues. Thanks for now :)
<adiroiban> sinzui: do we still need âidâ attribute in IProductSeriesPublic and IDistroSeriesPublic ?
<sinzui> I think so.
<adiroiban> where are they implemented?
<adiroiban> I can not find them in model.productseries or model.distroseries
<mwhudson> garara
<mwhudson> why is the code in canonical.launchpad.webapp, some of the most delicate and central in the whole system, so poorly tested?
<mwhudson> or tested in ways i can't find, anyway
<mwhudson> flacoste: you there?
<flacoste> mwhudson: i am
<mwhudson> flacoste: do you know where the tests for adding the notification messages about read only mode are?
<flacoste> hmmm
<flacoste> i would bet in the pagetest
<mwhudson> ah
<flacoste> mwhudson: xx-read-only-=mode.txt
<flacoste> without the =
<flacoste> in lib/canonical/launchpad/pagestests
<mwhudson> ah, you are right
<mwhudson> flacoste: do you know why api requests don't explode from having notifications added in read-only mode?
<mwhudson> flacoste: i was looking at fixing https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403281
<mup> Bug #403281: public xmlrpc requests broken during read only period <oops> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/403281>
<mwhudson> but it seems like testing is going a bit awkward
<flacoste> mwhudson: API requests don't use notifications
<flacoste> mwhudson: and tbh, i don't understand how XML-RPC adds notification
<mwhudson> flacoste: well, it blows up trying
<mwhudson> INotificationResponse(request) raises CannotAdapt
<flacoste> do you know where notifications are added?
<mwhudson> flacoste: yeah, publication.py:212 ish
<flacoste> mwhudson: api probably don't blow up because they can probably adapt
<flacoste> they are a subclass of LaunchpadBrowserRequest
<mwhudson> right
<flacoste> that's probably not the case for the XMLRPC ones
<mwhudson> but PublicXMLRPCRequest isn't
<flacoste> you could override maybeReadOnlyRequest in the XMLRPC publiation
<flacoste> to not add the notification
<flacoste> or make PublicXMLR_PCRequest provide INotification...
<mwhudson> what i was thinking of doing was INotificationResponse(request, None) and coping with the None returned
<flacoste> that's also valid
<mwhudson> flacoste: can you see what the try:/except ComponentLookupError: is for in maybeNotifyReadOnlyMode ?
<flacoste> and probably the best option
<flacoste> actually, i think it was meant to catch your use case
<flacoste> but it's possible that a zope ugprade change the exception hierarchy
<mwhudson> :/
<mwhudson> ah
<flacoste> or that code was bad in the first place
<flacoste> never tested....
<flacoste> i'd say just change the exception
 * mwhudson deletes it
<mwhudson> oh, not do the , None thing?
<mwhudson> oh, i made up the CannotAdapt exception, it raises TypeError :(
<flacoste> so handling None is probably better
<flacoste> since TypeError can easily mask a programming error
<mwhudson> right
<mwhudson> flacoste: thanks for the hints
<flacoste> my pleasure
<wgrant> jml: Hmm, is it really a good idea to make authentication tokens writable by normal OAuth tokens?
<flacoste> mwhudson: is lp-serve something Launchpad specific or is it part of bazaar itself?
<mwhudson> flacoste: it's launchpad specific, but fairly small
<mwhudson> flacoste: it's like bzr serve, but backed onto the launchpad vfs, not the file system
 * mwhudson stabs TestReadOnlyModeSwitches
<flacoste> mwhudson: ok, how impossible is it to have that command show the branch it's serving in the command line?
<mwhudson> i guess i should just write a new class
<mwhudson> flacoste: you mean using setprocargs or whatever that's called?
<flacoste> mwhudson: you mean that the branch served is only known after the program is started, right?
<mwhudson> flacoste: right
<flacoste> mwhudson: is there a way now when a bzr lp-serve process starts eating all memory to know what was happening in it?
<flacoste> what branch was being served
<mwhudson> flacoste: not really :/
<flacoste> is there a bug for that already?
<mwhudson> flacoste: i guess lsof
<flacoste> you mean, seeing which file it has open
<mwhudson> yeah
<flacoste> yeah, that's heavyweight
<flacoste> i'll file a bug for the losa's sake
<mwhudson> there's a bug about recording access statistics for branches
<mwhudson> which is related
<mwhudson> but nothing for this losa use case that i know about
<mwhudson> flacoste: i hear crowberry was running out of memory even now it's got 32 gigs?
<flacoste> yep
<mwhudson> owwwwwwwwwww
<flacoste> a bzr lp-serve process was chewing 21G of memory!
<flacoste> that's a bug in bzr for sure
<mwhudson> !!!
<mwhudson> !!!
<flacoste> for knowing the branch in cause would help
<mwhudson> yeah, i can see that
<sinzui> That is a bug a big a Mothera.
<wgrant> That is impressive.
<mwhudson> i think setting an rlimit of 4 gigs or so for lp-serve processes would be uncontroversial...
<flacoste> mwhudson: can you file a RT about this? or is this something we should do in code?
<mwhudson> that's something we can do in code
<wgrant> mwhudson: Do you know why there are three Linux imports running concurrently? That seems insane, given how long even a single one will take.
<mwhudson> wgrant: all but one got suspended
<wgrant> mwhudson: Ah, so they are. Great.
<mwhudson> flacoste: https://bugs.edge.launchpad.net/launchpad-code/+bug/532213
<mup> Bug #532213: limit memory bzr-serve processes can use <trivial> <Launchpad Bazaar Integration:Triaged by mwhudson> <https://launchpad.net/bugs/532213>
<flacoste> thanks
<flacoste> mwhudson: i also filed https://bugs.edge.launchpad.net/launchpad-code/+bug/532210
<mwhudson> flacoste: ok
<donatas_s> Hello
<donatas_s> There are problem with launchpad server:  Sorry, there was a problem connecting to the Launchpad server.
<mwhudson> donatas_s: which url?
<donatas_s> It just start to work, but there was problem with all launchpad url
<gary_poster> one appserver has been acting up
<mwhudson> flacoste: able to review this publication fix, seeing as we talked about it already?
<mwhudson> it's pretty tiny
<flacoste> mwhudson: sure
<mwhudson> flacoste: https://code.edge.launchpad.net/~mwhudson/launchpad/read-only-xmlrpc-bug-403281/+merge/20707
<mwhudson> oh, damnit i forgot to add comments to the tests
<jml> flacoste, re lp-serve
<jml> flacoste, I got *this* close to writing decent branch access logs for it
<jml> wgrant, I don't know, honestly.
<wgrant> jml: The point of OAuth is to allow users to revoke application privileges.
<wgrant> If applications can add authentication tokens, that becomes difficult.
#launchpad-dev 2010-03-05
<wgrant> This might tie in interestingly with the ideas presented in bug #532055
<mup> Bug #532055: Trusted credential-management apps are broken and may be doomed <launchpadlib :New> <https://launchpad.net/bugs/532055>
<jml> heh heh
<wgrant> Does anybody know what is happening with https://bugs.launchpad.net/bugs/529348? It's a trivial fix..
<adiroiban> is there any documentation for @cachedproperty ? I don't know why, if I use it, I get raise ClosedError("Connection is closed") in the tests
 * mwhudson lunches
<jtv> Good morning antipodeans and others
<thumper> I was going to say hi to jtv
<thumper> but he has no staying power :)
<jtv> I think I used to be able to do this: push a branch to launchpad.dev... what am I missing
<jtv> ah, it's documented!
<thumper> jtv: yeah
<mwhudson> bzr-svn is at least better than cscvs: after some 10 hour long failures, bzr-svn succeeds in 46 minutes: https://code.edge.launchpad.net/~vcs-imports/maxosx/trunk
<thumper> w00t
<jtv> hi stub
 * stub waves groggily and moans something unintelligable
<jtv> stub: I could murder a lumberjack...
<jtv> cuppa?
<jtv> (I've been at it for a few hours already... don't ask what got into me)
<stub> Maybe in a couple of hours...
<stub> I'm still stuffed full of roast beef and belgian fries
<jtv> John's cooking?
<stub> need consciousness...
<stub> yer
<jtv> Sorry I missed it...  I grabbed some delicious "hello sir" from 3/1 to eat while watching a film at home with the missusâher ideaâwho then decided to skip the film so as to get up early for the temple this morning.
<jtv> TIT
<jtv> stub: meanwhile, I'm just moving some stuff over to the slave store.
<stub> jtv: I could probably be arsed dragging myself down for a coffee now
<jtv> stub: good, because what I just cooked up looks too horrible to swallow
<jtv> stub: I can be on a motoecy in a few minutes
 * stub crawls off looking for his pants
<wgrant> I just love seeing XXX comments with text like "I have no idea what the code below is meant to do."
<mwhudson> wgrant: soyuz code i presume?
<wgrant> mwhudson: But of course.
<mwhudson> thumper: around to review a branch?
<mwhudson> 1 whole line of changes!!
<thumper> mwhudson: yeah
<thumper> mwhudson: I have one for you too
<mwhudson> thumper: https://code.edge.launchpad.net/~mwhudson/launchpad/git-http-imports-bug-438929/+merge/20716
<mwhudson> thumper: ok
<thumper> mwhudson: actually, you know what?
<thumper> mwhudson: it can wait
<thumper> mwhudson: otherwise I'll be proposing against the wrong branch
<thumper> mwhudson: just to not show you the db-devel changes
<thumper> mwhudson: when db-devel is merged into devel it'll make more sense
<mwhudson> thumper: heh, ok
<mwhudson> thumper: my rc branch landed fine btw
<thumper> mwhudson: yours is done
<thumper> cool
<thumper> mwhudson: which one was that?
<mwhudson> thumper: the one fixing xmlrpc requests in read only mode
<thumper> mwhudson: ah, was that what was screwing us over?
<mwhudson> thumper: which screwage do you mean?
<mwhudson> but no, probably not
<thumper> mwhudson: about connections hanging around
<mwhudson> thumper: no
<thumper> oh
<mwhudson> at least, that seems very unlikely
<thumper> so what was the change?
<mwhudson> thumper: it just fixes loads of spurious oopses
<thumper> mwhudson: fair enough
<thumper> mwhudson: how's your afternoon going?
<mwhudson> thumper: well, i was being fairly unproductive, so i decided to fix a couple of trivial code import bugs
 * thumper nods
<thumper> I've been watching the code imports fairly closely
<mwhudson> that git one and https://bugs.edge.launchpad.net/launchpad-code/+bug/513182
<mup> Bug #513182: +code-imports should show VCS type <code-import> <trivial> <ui> <Launchpad Bazaar Integration:In Progress by mwhudson> <https://launchpad.net/bugs/513182>
<thumper> we seem much better now
<thumper> cool
<thumper> I've been making the description changes get emailed :)
<mwhudson> cool
<thumper> mwhudson: what if we renamed +junk to +branch?
<thumper> confusing?
<mwhudson> seems a bit random, somehow
<mwhudson> it doesn't really distinguish it from the other, um, branches
<persia> If you change +junk (and I've seen some chatter about that bug), could you leave an invisible redirect for the various immutable docs out there (mostly email archives)
<mwhudson> surely
<persia> Wonderful :)
<thumper> I've been trying to think of a meaningful anem
<thumper> name
<thumper> +own
<thumper> +my
<thumper> +personal
<thumper> all a bit weird
<persia> Weren't there some suggestions in the bug?  If not, there were certainly some in -motu and I can dig up logs.
<thumper> I want to make sure it still makes sense with a team branch
<thumper> persia: there were
<thumper> persia: discussed in -motu?
<persia> That bug was.
<persia> (someone found +junk offensive, and ranted for a bit, and a few names were thrown out)
<persia> I'll see if I can find the log.
<thumper> bug 147407
<mup> Bug #147407: Junk sounds too harsh <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/147407>
<mwhudson> thumper: thanks for making rockstar do make run_codehosting btw :)
<thumper> +user
<persia> Which doesn't work for teams.  And the conversation I remember ended being a tutorial on how to stick a branch on people.ubuntu.com
<thumper> +misc
<thumper> +code
<thumper> I think I'd rather have +branch over +coe
<thumper> +code that is
<thumper> the branch to change this will be relatively simple, and I'd like to get it done
<thumper> I'm going to go with +branch
<poolie> hi thumper
<thumper> and see what happens
<thumper> hi poolie
<thumper> I'm just trying to walk out the door
 * thumper afk
<wgrant> Odd. Somebody added LFC.sha256 a year ago, but didn't add the three extra lines to actually get it populated for new files.
<wgrant> al-maisan, noodles775: Since LP again won't have emailed anybody, have a poke about https://bugs.edge.launchpad.net/bugs/532445 and https://bugs.edge.launchpad.net/bugs/532454
<al-maisan> wgrant: thanks, looking.
<henninge> Hi wgrant! ;)
<wgrant> Morning henninge.
<henninge> wgrant: thanks for the help the other day, I got the buildd to work and command it via rpc.
<wgrant> henninge: Excellent.
<wgrant> henninge: I see I forgot the ensurepresent call. I'm glad you worked it out in the end.
<henninge> wgrant: yes, I got that from the code ... ;)
<wgrant> Being able to see the code is handy.
<henninge> wgrant: that chroot you pointed me to, is that the same that is used on the production builders
<henninge> or will be
<henninge> ?
<wgrant> henninge: It's the same.
<henninge> wgrant: Can I adapt it to my purposes?
<wgrant> henninge: You can now directly get the current URL from https://launchpad.net/api/devel/ubuntu/lucid/i386/chroot_url
<wgrant> henninge: Please don't.
<wgrant> Your script should be able to run on the chroot you get from LP.
<wgrant> Unless we want to start having separate chroots for each job type, which we have so far avoided.
<wgrant> and would like to continue to avoid.
<wgrant> You'll see that the recipe build job installs the packages that it needs itself.
<henninge> yes, I was trying that, too.
<henninge> I get  "permission denied" when trying to run apt-get.
<henninge> wgrant: what do you mean by "recipe build job" ?
<wgrant> Sounds like you are running that from inside a script that runs as the user.
<henninge> so, what user does the buildd run as? root?
<wgrant> henninge: See lib/canonical/buildd/buildrecipe
<wgrant> It runs as buildd, but can sudo.
<henninge> That's what I am doing.
<henninge> wgrant: there is no sudo in the chroot, though
<wgrant> henninge: No. You have to sudo from outside.
<henninge> wgrant: so should the build job inside the chroot do everything as root?
<wgrant> henninge: Probably not.
<wgrant> You need a script running outside the chroot to:
<wgrant>  1) sudo into the jail and install your dependencies
<wgrant>  2) jump into the jail as the normal user, and run pottery or whatever it is you want to do.
<henninge> wgrant: oh, I think I was not aware that I don't need to be root to call chroot.
<wgrant> henninge: You do need to be.
<wgrant> henninge: see the 'su -c' calls in buildrecipe.
<wgrant> That is run inside the chroot to switch back to the right user.
<henninge> looking at that now
<wgrant> So you end up doing 'sudo chroot su -c "whatever command" buildd'
<henninge> oh, forgot about su with all the sudoing ...
<wgrant> er, of course there's the chroot path in there somewhere.
<wgrant> Yeah, it's horrifyingly messy.
<henninge> yes, shure
<henninge> s/h//
<wgrant> 'shure' makes more sense!
<henninge> ;-)
<henninge> wgrant: thanks, I feel a lot more enlightened
<wgrant> henninge: Excellent.
<wgrant> You may still beat recipes to production..
<henninge> wgrant: I see that buildrecipe installs sudo in the chroot. I uses both sudo -u and su -c
<wgrant> henninge: Heh, I wonder why.
 * wgrant looks.
<wgrant> Ah, right, I remember from the sprint now.
<wgrant> The environment variables.
<henninge> wgrant: sudo is nicer to handle with call
<wgrant> Yeah.
<wgrant> It was going to be risky and difficult to escape them and stick them in a string for su -c.
<henninge> wgrant: also, it is a bit hard to try out the installing because the sources.list in the chroot points to ftpmaster.internal
<henninge> I'll just have to overwrite that
<wgrant> henninge: You can pass a list of sources.list entries in the build call.
<wgrant> Assuming that you're using the normal chroot infrastructure code (unpack, override sources.list, upgrade).
<henninge> hm, I have not yet seen the "overide sources.list" part
<wgrant> Add to the extra_args of your build call: "'archives': ['deb http://blah/ubuntu blah blah blah', '..']"
<henninge> I see
<henninge> ok, found it
<henninge> wgrant: I am not using the DebianBuildManager btw. Maybe I should.
<noodles775> Where are you looking? (/me wants to follow along as much as I can)
<wgrant> henninge: I think you should.
<wgrant> Let me look.
<wgrant> I created it, but that was almost two months ago now so blaaaah.
<wgrant> Yeah, you should probably use it.
<henninge> noodles775: DebianBuildManager.doSourcesList
<noodles775> henninge: ta.
<henninge> noodles775: it calls slavebin/override-sources-list
<henninge> wgrant: ok, let me look into that
<wgrant> henninge: If you want to see how normal builds work, look at the build logs on production.
<henninge> wgrant: where do I see them
<henninge> ?
<wgrant> https://edge.launchpad.net/ubuntu/+builds?build_text=&build_state=built
<henninge> wgrant: cheers
<henninge> oh yeah, very interesting
<adeuring> good monring
<wgrant> henninge: What's still to do with the slave and master template code?
<henninge> wgrant: what do you mean?
<henninge> adeuring: moin
<wgrant> henninge: Well, what are you doing to it?
<adeuring> moin henninge
<wgrant> I thought it was mostly working.
<henninge> wgrant: well, obviously it's not ...
<wgrant> Heh, yeah, I've just looked at the stuff in the tree and seen that it doesn't do much.
<henninge> It's obviously never been run like I am trying to run it now.
<wgrant> Right.
<henninge> I have added the call to the actual template generation code but getting it to be run is my challange now.
<henninge> wgrant: but I think I am getting there
<wgrant> The call to that code seems to already be there.
<wgrant> Although the filename is wrong.
<henninge> you mean "generate...py"?
<wgrant> And you can save an awful lot of code by using DebianBuildManager.
<wgrant> Yeah, that.
<henninge> that's just a dummy, too.
<henninge> wgrant: oh no, sorry
<henninge> I already landed that part ... ;-)
<wgrant> It doesn't look like it will actually check out the branch, though.
<wgrant> And it imports from lp.*, so it won't work in the slave.
<henninge> wgrant: I added stuff to debian/rules
<henninge> so the importing works
<wgrant> henninge: Oh.
<wgrant> That seems a bit ugly. But I guess the whole slave thing is pretty ugly.
<henninge> My thinking ...
<henninge> ;)
<wgrant> henninge: Is there a reason for that file to live in lp.translations at all?
<henninge> wgrant: I was trying to mimic the way tachandler was included.
<henninge> it seemed to replicate the structure from the canonical tree, so I did the same for the lp tree
<henninge> wgrant: the code is only needed in the buildd, so it could live in canonical/buildd .
<wgrant> henninge: tachandler is copied in because it's used on both sides.
<wgrant> So I would just put your file in lib/canonical/buildd and be done with it.
<henninge> I guess that's true.
<henninge> wgrant: but it seems nicer for all translation code to live lp.translations ...
 * wgrant is very tempted to split up lib/canonical/buildd's contents into subdirectories.
<noodles775> Into what for example?
<henninge> good idea
<noodles775> per app?
<wgrant> translations, recipe, binary
<noodles775> Right.
<wgrant> Per build type, yeah.
<wgrant> And a general one.
<wgrant> Since the dir is getting pretty big.
<wgrant> And there is obvious separation.
<noodles775> +1
<henninge> +1
 * henninge has to relocate, back in 20 min max.
<mrevell> Morning
 * henninge is back
<jml> mrevell, good morning
<mrevell> Good morning jml
<jtv1> I'm having some trouble with codehosting on my dev machine...  I "make run" and "make run_codehosting."  I push a branch to lp://dev/...  I make changes to the branch, and when I push again, I'm told the branches have diverged.
<jtv1> Anyone know what I'm doing wrong?
<henninge> jtv: is make run_codehosting new? I just do make_run
<henninge> used to, I mean
<jtv> henninge: I believe it's new, yes
<jtv> killall -9 firefox
<jtv> ahhhh
<jtv> I should put that in cron.hourly
<jtv> stub: I believe you said tests were going back to a single store... do we have a bug for that?
<stub> launchpad is, not just tests
<stub> single replication set
<wgrant> How are lp_* going to be replicated to c-i-p?
<stub> I'm not counting that as part of launchpad ;)
<jtv> stub: oh, so still replicated...  Using the slave store will require some extra commits in tests then.
<stub> Yup.
<jml> didrocks, hi
<didrocks> hey jml
<didrocks> jml: sorry, but yesterday, it was like hell :)
<jml> didrocks, understood
<didrocks> jml: I pulled your branch and get scared by the amount of change :)
<jml> didrocks, wgrant makes a very interesting objection -- if we have an API for adding, say, SSH keys, then a malicious application that gained your approval could add a key, then use that key to get your branch data even after you revoked privileges from the application
<wgrant> jml: I think modification of that belongs in the trusted client. Not the cleanest solution, but just about anything else is unsafe.
<didrocks> jml: GC already does that manually (upload a ssh key for you). I'm not sure not exposing that will prevent this usage
<wgrant> didrocks: People should soon be educated to not give their password to such applications.
<wgrant> And that will stop working soon.
<wgrant> I hope.
<didrocks> wgrant: right :)
<didrocks> that's just too much hope on the user side I'm afraid against a promise for a "wonderful application making your life better (and spamming you, destroying your data and so onâ¦)"
<didrocks> but that's just my opinion :)
<jml> didrocks, anyway, I'm not sure how to proceed. The work we've done already is good & useful and should be merged.
<didrocks> jml: right, but unfortunately, that won't save the "I don't achieve to upload my gpg/ssh key or uploading wrong one, and so onâ¦"
<jml> didrocks, hmm.
<jml> adeuring, I notice that the bugs with patches view includes fix released bugs
<adeuring> jml: yes
<jml> adeuring, is this deliberate?
<adeuring> yes
<adeuring> ...but I can't remember the exact reason. Probably because "fix released" is intersting for upstream
<adeuring> kfogel might remember better ;)
<jml> ok. I'll ask him.
<wgrant> Doesn't that just mean it's going to get longer and longer and become less and less useful?
<henninge> wgrant: I think DebianBuildManager has too much specific stuff that I don't need.  Any reason doSourcesList, doUpdateChroot, doReapProcesses should not be in BuildManager instead (and their confgis moved to allmanagers) ?
<wgrant> henninge: What specific stuff don't you want?
<henninge> wgrant: what's ogre model btw ;-)
<wgrant> henninge: It's not used any more. The master used to pass in a component rather than a full sources.list
<wgrant> ogre model refers to the layering of the components.
<wgrant> main includes just main. restricted includes restricted and main.
<wgrant> universe includes universe and main.
<wgrant> multiverse includes multiverse, universe, restricted and main.
<henninge> ah, I see.
<henninge> wgrant: changes file handling for example I don't need
<henninge> gatherResults will work differently for me, too.
<wgrant> Yeah, so just gatherResults, _parseChangesFile and getChangesFilename?
<jtv> wgrant, henninge: the FSM for DebianBuildManager is also a lot more extensive than we need, isn't it?
<henninge> jtv: yes, that's another thing
<wgrant> jtv: Why?
<wgrant> I think you need everything there.
<jtv> wgrant: our job is simpler... no need to install build dependencies, for one
<wgrant> DebianBuildManager doesn't handle build-dependencies AFAIK.
<jtv> Then who installs them before building a package?
<wgrant> jtv: sbuild in the binary case, or the recipe build script in the recipe case.
<jtv> oh, it's further down in the hierarchy
<wgrant> Yes.
<jtv> So is DebianBuildManager not limited to building Debian packages then?
<jtv> The branches we work with don't necessarily have a debian/ dir etc.
<henninge> jtv: no, it doesn't look that way
<henninge> wgrant: Yes, you are right with that list (gatherResults...)
<jtv> Looks like
<wgrant> henninge: Why not just override gatherResults?
<jtv> Yes, that does seem to make sense
<henninge> wgrant: yes, the obvious next step
<wgrant> You already need to do similar things on the master side, where we've made assumptions in superclasses that you have a build.
<wgrant> So you might as wlel make it easy for yourself and just override the two methods.
<henninge> correct
<wgrant> (the one to do the actual building, and gatherResults)
<henninge> wgrant, jtv: yes, I will do that.
<henninge> thanks
<jtv> It does mean that DebianBuildManager is a total misnomer
<jtv> (what's this Ogre thing btw?  Do we need that at all?)
<wgrant> jtv: It doesn't. You're building in a Debian chroot.
<wgrant> You don't care about that. We should remove ogre support.
<jtv> ahh, Debian only in _that_ sense
<wgrant> It hasn't been used in nearly three years.
<jtv> So... what build managers do we have that are not debian build managers in that sense?
 * jml rebooting
<wgrant> None at this point.
<wgrant> But there could quite conceivably be some later.
<wgrant> stub: Do you know why there is a LFC.sha256 column, but it's not actually being populated?
<stub> because soyuz team decided they needed it, added the column, but didn't need it enough to make the librarian actually store it when files are uploaded.
<bigjools> lol
<wgrant> Odd, since it was about 3 lines extra to do it.
<stub> Yup. Also about 3 lines to drop the column again...
<wgrant> But we'll need the column soon.
<wgrant> We should be using more than MD5 in the indices.
<stub> I've heard that for maybe a year now ;)
<wgrant> This is Soyuz. Time is different here.
<bigjools> my clock says it's 1984
<stub> So three line fix to the librarian to store the sha256. A quick and ugly data migration script to fill in the values for the existing files. A DB patch to set the column to NOT NULL.
<stub> Party like its 1999
<wgrant> Quick, ugly, and several-week-taking?
<jml> didrocks, so what are we going to do?
<stub> wgrant: Its only a few terrabytes - shouldn't take too long.
<wgrant> Was SHA1 there from the start?
<jml> bigjools, "It was a bright cold day in April, and the clocks were striking thirteen."
<stub> I think it was, yes.
<wgrant> So it's not been done before :(
<bigjools> jml: we need to talk!
<didrocks> jml: that's a good questionâ¦  :/
<stub> Its pretty trivial. I can hack up the migration if nobody else wants to.
<jml> bigjools, we do.
<jml> bigjools, lemme book a time
<jml> bigjools, 12:30 ok? or we can talk now until 11
<bigjools> jml: no 12:30 is bad, I have a hospital appt at 1
<bigjools> so now is good
<jml> ok. I'll skype you.
<jtv> henninge: you may have to pretend that the dir with your packages is an apt cache... something like -o Dir::Cache=/tmp/extra-packages
<henninge> jtv: let me try that out
<jtv> was I stopping you?
<wgrant> henninge, jtv: Alternatively, upload them to a local or real PPA.
<wgrant> What custom packages are there?
<henninge> wgrant: no, it's launchpad-buildd
<henninge> wgrant: I am using pbuilder to try it out and want to install it in there
<henninge> actually, pbuilder might be able to do it, too
<wgrant> henninge: Ahh.
<wgrant> pbuilder login
<henninge> exactly
<jtv> henninge: does that solve it?
<deryck> Morning, all.
<henninge> jtv: havn't gotten that far, got distracted. Read it on the wiki page next week ... ;-)
<jtv> henninge: I'm also looking at ways to do this
<noodles775> henninge: ah, you are capturing all this on a wiki page? Great!
<henninge> noodles775: not yet but we just decided to do it
<jtv> henninge: haven't found anything better than adding a "file:" URL into the slave's /etc/apt/sources.lists.d
<jtv> hi joey!
<joey> hi jtv
<wgrant> jtv: henninge: Why can't you just dpkg -i it?
<jtv> wgrant: need to satisfy the deps as well
<henninge> wgrant: I want dependencies to be pulled automatically
<wgrant> jtv: dpkg -i blah.deb; then 'apt-get -f install'
<jtv> eh
<jtv> I guess that'd work too  :)
<wgrant> I thought I described this on the wiki page.
 * jtv facepalms
<jtv> wgrant: which wiki page is that?
<jtv> There's so much text out there, sometimes it's like not having any
<henninge> wgrant: which wiki page
<wgrant> jtv: HowToUseSoyuzLovally
<henninge> uh-oh
<wgrant> s/v/c/
<henninge> Lovely
<henninge> ;)
<wgrant> Or Run or whatever it is.
<wgrant> Heh.
<jtv> Use, yes
<noodles775> wgrant: I thought henninge was going to document something like, extending the build-system for translations.
<henninge> noodles775: that's next
<noodles775> Great.
<jtv> But first, "running a local build slave"
<jtv> at all
<wgrant> I think the wiki page probably describes it OK.
<noodles775> Yeah, so do I? I followed the current wiki page and binary builds went through fine. BUt I think you guys are already talking about custom stuff for translations right? (ie. talking directly with the buildd via xml-rpc)
<jtv> maybe... it's a bit hard to get the general drift when you're not "into" soyuz.
<henninge> noodles775: yes
<noodles775> OK.
<henninge> I'll just describe what I did and we can converge afterwards.
<noodles775> Great!
<wgrant> noodles775: The talking directly to the buildd is just for testing.
<wgrant> The real setup will go through buildd-manager.
<jtv> wgrant: whee, is that an easier way of getting the latest chroot tarball that you found?
<wgrant> jtv: I didn't find it; I added it last week.
<jtv> wgrant: ah :)
<jtv> very cool.  One gotcha: the URL on the wiki page doesn't quite work for some reason, but if I go up one "directory" I get the information
<wgrant> jtv: It does work if you wget it.
<wgrant> Your browser is probably trying to interpret it as XHTML.
<jtv> niiice!
<jtv> wgrant: adding a command line to download the tarball in one go
<wgrant> jtv: So I can say 'grab-chroot-from-launchpad ubuntu lucid i386' and it will download it and upload it to my dev instance?
<jtv> wgrant: nothing so fancy, but that would be a good idea wouldn't it?
<jtv> for now it's just a wget command line
<jtv> There must be some way to make the architecture default to whatever you're currently running...  all my attempts so far give me i686 when I want to see i386
<jtv> (Which is nice for those pre-Pentium users out there of course :)
<wgrant> jtv: Just use i386.
<wgrant> It'll work everywhere Launchpad runs.
<jtv> not going to break on native amd64?
<jtv> (wow, lsb_release takes a while to collect lots of data, but no architecture there)
<wgrant> Hm, it's possible that the buildd package will default to amd64 there. It's one line in a config file to switch it to i386, but that's a good point.
<wgrant> 'dpkg --print-architecture' will work.
<jtv> yay
<jtv> But this is going to be relatively hard work for replacing a wget command line.  Laziness is beginning to appeal...
<wgrant> Yeah.
<jtv> wgrant: of more urgency is a fix needed to the setup script I wrote: it fails to register your ssh key(s)
<jtv> easily fixed, but saves more trouble right now
<wgrant> jtv: Refactor make-lp-user, I guess?
<jtv> either way I think my script should use make-lp-user; the question is should the GPG key registration be moved in there.
<wgrant> Probably. It and the CoC signing are both useful.
<jtv> Ah yes, now I see why I hesitated about that: it means adding options parsing.  :)  make-lp-user <user> [<team> [<team> [...]]]
<wgrant> Ahh.
<jtv> not exactly hard though :)
<jtv> I could lift the -e option across wholesale
<henninge> noodles775, wgrant: First iteration but not final yet (have to try it all out once more) https://dev.launchpad.net/BuildFarm/TryOutBuildSlave
 * henninge lunches
<jtv> wgrant: new & improved make-lp-user is underway.  :)
<jtv> wgrant: bug 532354
<mup> Bug #532354: Set up ssh keys for ppa-user <Soyuz:Triaged> <https://launchpad.net/bugs/532354>
<wgrant> jtv: Excellent.
<jml> jtv, why does make-lp-user need to change to address that bug?
<adiroiban> hi. I have this interface inherited by IProducSeriesPublic and IDistriSeriesPublic http://paste.ubuntu.com/388916/, does anyone know why active and summary is visible in API, while drivers_collection_link is not ?
<jtv> jml: it already registers your ssh keys; it seems madness to register your gpg keys in another script.
<jtv> jml: and I suspect automated gpg key registration can be useful for other uses as well.
<jtv> (it's optional, of course)
<jml> jtv, ahh yes. that makes sense. the bug itself says "ssh", so I was confused.
<jtv> jml: that is my fault.  From my perspective, I found that I duplicated much of the make-lp-user functionality, but not this partâand it was needed for full usage.
<jml> heh
<jtv> So for me the immediate problem was, "my ssh key isn't registered when I set things up this way."
<jml> adiroiban, I don't know, sorry.
<wgrant> jtv: Pure Soyuz people need no SSH keys.
<wgrant> adiroiban: Can you pastebin the entire diff?
<wgrant> adiroiban: There's nothing obviously wrong there.
<jtv> wgrant: I can't quite tell if that's a "real men don't need [...]" joke or reality.  ;)  But it'll come in handy in any case.
<wgrant> jtv: A combination. But in reality they're only useful for branches, which I don't often use.
<jtv> wgrant: let's just hope the bridges will continue to sprout though :-)
<wgrant> Indeed.
<jtv> It may also be a good idea to rename the script...
<jtv> But now the problem is, what do we call the build farm _plus_ soyuz?  :-)
<bigjools> wgrant: urgh, you've forced me to remember how utterly shite dscfile.py is
 * bigjools does some refactoring to make testing easier
<jtv> wgrant: ^^^ well done, you made him improve the code :-)
<wgrant> bigjools: Yeah, it's not very nice.
<adiroiban> wgrant: http://paste.ubuntu.com/388921/ , but it's rather huge ... don't worry, I will continue to dig
<wgrant> Ah, right, it does the format and file combo checking as well.
<wgrant> Now I remember it.
<wgrant> It fits a lot of evil into 800 lines.
<wgrant> adiroiban: (not related to the problem, but DistroSeries should probably be an ISeries, not an IHasSeries)
<wgrant> adiroiban: I think IHasDrivers is probably clobbering IHasSeriesMixin.drivers.
 * wgrant checks.
<wgrant> At least I presume IHasDrivers has a drivers attribute.
<adiroiban> wgrant: true :) thanks!
<wgrant> Yeah, that's it.
<deryck> intellectronica, is the branch for bug 531433 going to land before the re-roll?
<mup> Bug #531433: migration-assistant crashes ubiquity <apport-bug> <i386> <ubiquity (Ubuntu):New> <https://launchpad.net/bugs/531433>
<deryck> hmmmm
<deryck> bug 531443
<mup> Bug #531443: Bug heat flames should be calculated based on the context, not the bugtask's target <story-bug-heat> <Launchpad Bugs:In Progress by intellectronica> <https://launchpad.net/bugs/531443>
<lifeless> jml: you might enjoy https://code.edge.launchpad.net/~jelmer/launchpad/testr/+merge/20511
<jml> lifeless, I do.
<didrocks> jml: ok, got some time now. Should we first push the "read only mode"?
<wgrant> jml, didrocks: Do you have a launchpad.View adapter for them so they'll appear anonymously?
<jml> didrocks, yes.
<jml> wgrant, no, not yet. what's involved there?
<wgrant> jml: Just a launchpad.View adapter inheriting from AnonymousAuthorization.
<wgrant> Otherwise lazr.restful will return empty collections for anonymous users.
<wgrant> (it checks for launchpad.View, and there is no default anonymous launchpad.View adapter. The global (ie. Interface) authenticated one is deprecated, so it was forbidden to extend it to cover anonymous too.)
<wgrant> So each exported object that will appear in a collection needs a launchpad.View adapter, even if it is actually zope.Public.
<lifeless> jml: not sure why it would have been big... we did the heavy lifting already.
<lifeless> gnight all
<jml> lifeless, I didn't know the config lifting was done
<allenap> abentley: Do you have time to help me with an issue I'm having with TestTwistedJobRunner? I'm trying to run it in a sub-process using subunit.IsolatedTestCase, but the assertions are failing. My branch is lp:~allenap/launchpad/isolate-tests.
<lifeless> jml: I was referring to zope testerunner with subunit and load-list
<jml> lifeless, oh ok.
 * lifeless switches off his monitor. *yawn*
<deryck> adeuring, what about bug 511240 for the re-roll?  Did the branch make it in?
<mup> Bug #511240: bug heat calculation should use Bug.users_affected_count_with_dupes instead of Bug.users_affected_count <story-bug-heat> <Launchpad Bugs:In Progress by adeuring> <https://launchpad.net/bugs/511240>
<deryck> ah, I found the branch.
<intellectronica> i'm getting test failures in doc/launchpadlib.txt in EC2, which i can't reproduce locally and are unlikely to be related to my changes. anyone seen anything like that lately?
<intellectronica> deryck: also, my branch still hasn't landed, because of ^^^^^ (and some real errors i found in an earlier run) :(
<deryck> intellectronica, ok.  at least we caught the errors.
<wgrant> Hm, daily CHR again?
<danilos> wgrant, yeah
<adeuring> deryck: yes, it landed (sorry for the delay, was out for lunch)
<deryck> adeuring, no worries.  Thanks for the update.
<jml> didrocks, I've proposed my branch for merging w/ read-only
<jml> didrocks, in terms of quickly's goals, I think your stuck for lucid -- sorry.
<didrocks> jml: well, do you think we can do something for +1, discuss at UDS, maybe?
<jml> didrocks, yeah.
<didrocks> jml: I just have to propose my branch as well?
<jml> didrocks, please propose your branch
<didrocks> jml: so, still some room for adding announcement?
<jml> didrocks, it'd make an interesting UDS conversation... I don't know who the right Launchpad folk would be
<jml> didrocks, oh right, announcements.
<jml> lemme look
<didrocks> jml: sweet, thanks :)
<jml> :(
<didrocks> jml: ?
<abentley> allenap, hi.
<abentley> allenap, what assertions are failing?
<allenap> abentley: self.assertEqual(1, len(runner.completed_jobs)) is the first one. It looks like it's not actually running the jobs, even though iterReady() is called.
<abentley> allenap, or perhaps the jobs are failing?
<allenap> abentley: incomplete_jobs is also empty.
<abentley> allenap, does it run runJobInSubprocess?
<allenap> abentley: I don't think so...
<allenap> abentley: TwistedJobRunner.getTaskSource() is called once only; the generator it forms is never pulled from.
<jml> didrocks, it's hard.
<abentley> allenap, does the reactor start?
<allenap> abentley: I'll check that and get back to you.
<didrocks> jml: really? seems I don't have any luckâ¦ :(
<jml> didrocks, well, maybe someone here will know
<jml> hey guys, if I want to expose the ability to create announcements on projects over the API, what should I do?
<james_w> is there a method on IProject that takes some text and other things and creates the annoncement?
<allenap> abentley: It does start. (Fwiw, I've instrumented my branch with http://paste.ubuntu.com/388968/)
<intellectronica> james_w: i think it's project.announce()
<intellectronica> james_w: yeah, see lib/lp/registry/doc/announcement.txt
 * didrocks opens as well
<james_w> yes
<james_w> so, lib/lp/registry/interfaces/announcement.py
<james_w> IMakesAnnouncements.announce should be exported
<james_w> and probably IHasAnnouncements too
<intellectronica> james_w: i'm happy to review a branch :)
<james_w> and IAnnouncement too I guess
<james_w> intellectronica: I'm trying to answer Jono's question ;-)
<intellectronica> oic
<didrocks> james_w: jml's, no? :)
<didrocks> (and mine as well in fact :))
<didrocks> thanks a lot james_w, intellectronica
<intellectronica> james_w:  i thought by 'should' you meant 'the gnomes should export it' :)
<intellectronica> didrocks: likewise i'm happy to help if you need. though i think by now you're an api export guru already, no?
<james_w> @export_write_operation() is the basic decorator for announce()
<james_w> you can pass some options to control the exported name and things
<didrocks> intellectronica: far far far from it :)
<james_w> there are other decorators for tweaking some other things, a grep for export_write_operation will find you some things to consider
<didrocks> I'll give it a try beginning at the next week. Sounds easier and more streamline than gpg and ssh stuff :)
 * didrocks logs the conversation
<abentley> allenap, see http://paste.ubuntu.com/388976/
<didrocks> intellectronica: james_w: thanks for the help, I hope to be able to do something useful next Monday with this
<james_w> sweet
<intellectronica> didrocks: np. shout if you need help. it will be great to have this stuff exported.
<didrocks> intellectronica: be sure I'll shout for help :)
<didrocks> first, just pushing the ssh branch
<allenap> abentley: Ah, that's odd. Any ideas why it might be doing that? Also, what incantation did you use to get the log output? Somehow I was logger-blind when reading that file.
<abentley> There's a debug logger named "gloop" defined near the ParallelLimitedTaskConsumer.  I just fed it in.
<abentley> I can't think why launchpad_ftest wouldn't exist, but I'd guess it was due to missing environment variables.
<allenap> abentley: Ah, that's a good thought.
<abentley> allenap, interestingly, when turn TestJobRunner into an IsolatedTestCase, it tries to create duplicate keys.
<jml> if you'll pardon a wild-arsed guess, is that because the factory has all-new internal state when it's in a subprocess?
<allenap> abentley: Gah.
<abentley> jml, the duplicate keys?
<jml> abentley, yes.
<allenap> jml: Ah, that sounds like a problem. Perhaps the factory should use a uuid or a counter driven from a file instead of a per-process counter?
<abentley> jml, this is a constraint on the primary key not being duplicated.  That's a serial, and I don't think the factory would have anything to do with that.
<jml> abentley, ok. that makes sense. sorry for the noise.
<abentley> jml, no worries.
<sinzui> danilos: Can you trade chr with me? I am sprint on the 8. (monday)
<danilos> sinzui, sure, please mark the change in the calendar as well (or remind me: I love when I am reminded :)
<sinzui> danilos: okay
<MTecknology> I'm curious, you guys ever try to use nginx and launchpad?
<danilos> sinzui, thanks
<intellectronica> MTecknology: no, we rely on apache quite heavily. also, since the front-end web server does little of the work in launchpad, there probably won't be much benefit in replacing it even if nginx is slightly more efficient
<MTecknology> intellectronica: oh, I was just curious because for my web servers it made a massive change. I was going to have to upgrade hardware because I was running out of resources and then I changed and magic happened. But not for the speed of script processing
<leonardr> jml: by some freakish chance are you around? otherwise i'll email you
<jml> leonardr, it's not even 3pm on a working day :)
<jml> leonardr, it's not so much freakish chance as par for the course
<leonardr> jml: can we skype?
<jml> leonardr, sure.
<gary_poster> jml, what's your skype id?
<gary_poster> I'm garyposter
<leonardr> jml: it's going to be me and gary
<jml> blackjml
<intellectronica> jml: you know what's freakish? i'm getting errors, on ec2, in doc/launchpadlib.txt, which i can't reproduce locally. does it ring a bell?
<intellectronica> argh, i meant to ask leonardr, not you jml
<leonardr> intellectronica: show me the errors
<jml> intellectronica, good good :)
<intellectronica> leonardr: http://pastebin.ubuntu.com/389001/
<allenap> abentley: I think the problem is that, by the time <testcase>.run() is called, Zope has already done some set-up (based on the layer perhaps/probably), and the forking breaks things.
<allenap> abentley: Wrapping each test case with an IsolatedTestSuite makes things work, but is not ideal or correct; it does not isolate each test method in a test case from one another.
<abentley> allenap, that sounds plausible.
<abentley> allenap, is TestCaseWithFactory.setUp / tearDown being called?
<allenap> abentley: I don't know, but I didn't suspect that it wasn't...? Ah, is this to do with the ids problem in TestJobRunner from earlier?
<abentley> allenap, I still can't imagine what would cause the TestJobRunner problem.
<abentley> allenap, it sounds like it might not be valid to derive from both TestCaseWithFactory and IsolatedTestCase.
<allenap> abentley: No, I can't either. It's odd. It works if wrapped in IsolatedTestSuite, so I think it's a bad interaction between Zope and everything else.
<allenap> abentley: Right now, I suspect it's not safe to derive from IsolatedTestCase when using the Zope test runner, at least with our layer set-up.
<abentley> allenap, quite possible, considering that layers span multiple tests.
<abentley> allenap, I think layers have their own setup/teardown, so you might be able to invoke those manually.
<abentley> allenap, I'm not sure what IsolatedTestCase is meant to imply, but if it means test cases don't share resources like the librarian, it doesn't sound like a good idea to me.
<allenap> abentley: It forks before running each test method (by overriding TestCase.run) and sends the results back to the calling process using subunit. It's meant to be basically transparent, but there's something nasty going on there.
<abentley> allenap, it only forks, or it forks and execs?
<allenap> abentley: Just forks.
<allenap> See run_isolated() in lib/subunit/__init__.py. There's not a lot of code (IsolatedTestCase and IsolatedTestSuite are tiny.)
<abentley> allenap, seems to work okay if I do the fork myself: http://pastebin.ubuntu.com/389042/
<allenap> abentley: Oh, now that's really confusing :)
<abentley> allenap, one approach would be to gradually turn my run into run_isolated, until it breaks.
<Ursinha> hi rockstar, is bug 531687 really triaged? there's not importance set on it..
<mup> Bug #531687: Accessing a merge proposal during the rollout (ie R/O mode) oopsed <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/531687>
<rockstar> Ursinha, huh.  Apparently something went wrong in the setting of it.
<Ursinha> rockstar: thank you very much :)
<allenap> abentley: This diff against my branch - http://paste.ubuntu.com/389095/ - works fine... until you uncomment line 48, and I have no idea why.
<allenap> abentley: Anyway, thank you for your help today. I have to go now, but I'll check back here later in case you think of something. Bye :)
<abentley> allenap, you're welcome.
<abentley> allenap since this seems subunit-specific, lifeless or jml might be able to help.
<jml> have you tried with the latest subunit code?
<jml> g'night all
#launchpad-dev 2010-03-06
<wgrant> 'make lint' is now giving warnings that it can't import lp.whatever for just about every import.
* thumper changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.03 | PQM is in open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
<adiroiban> hi. can someone put this into a public pastebin  https://lpbuildbot.canonical.com/builders/lp/builds/648 ?
<wgrant> It looks to me like lib/lp/buildmaster/tests/queuebuilder.txt won't be run
<wgrant> Since it's not in doc/.
<wgrant> Can anybody confirm?
<adiroiban> wgrant: do you want me to run bin/test -t queuebuilder.txt on my system ?
<wgrant> adiroiban: I know that doesn't catch it, but I'm wondering if something else does.
<wgrant> Since it still passes.
<wgrant> So it has probably been run recently.
<adiroiban> wgrant: http://paste.ubuntu.com/389926/
<adiroiban> it looks like queuebuilder.txt is not included in the search
<wgrant> adiroiban: That's what I thought, but it might somehow be included under another name.
<wgrant> I've moved it into doc/ for now anyway, since that's where it belongs and works.
#launchpad-dev 2010-03-07
<lifeless> jml: I'm curious how https://code.edge.launchpad.net/~jml/zope.testing/subunit-output-formatter/+merge/19825 is progressing
<jml> lifeless, it's paused for the moment, due to me releasing Twisted
<jml> lifeless, I'm going to get back to it soon.
<asabil> hi all
<asabil> anyones know how to turn a project into a meta project ?
<asabil> is there any script in the launchpad source tree ?
<lifeless> moin
<lifeless> asabil: rename the project; make a meta project.
<asabil> lifeless, how ?
<lifeless> you'll need LOSA support - file a question on answers.launchpad.net/
<lifeless> bah
<lifeless> answers.launchpad.net/launchpad
<asabil> lifeless, I am asking in launchpad-dev
<lifeless> yes, sometimes folk do this ;)
<lifeless> ok, so do you want apis, or admin interface hints?
<asabil> admin interface hints
<asabil> is there an admin interface for launchpad ?
<lifeless> when you're logged in as an admin, some pages show an admin link, yes
<asabil> oh I found that
<asabil> but no mention of creating a super project
<lifeless> not sure where it is in the ui - check the zcml pages
<lifeless> a meta project is a 'project' in the lp codebase.
<asabil> ok thanks
<asabil> and a side question
<asabil> I am getting a ForbiddenAttribute error when listing the branches on a specific project
<asabil> any idea ?
<asabil> Module lp.code.browser.branchlisting, line 1277, in initial_branches
<asabil> branch for branch in branch_query[:max_branches_from_query]
<asabil> ForbiddenAttribute: ('__len__', <storm.store.ResultSet object at 0x7ad9bd0>)
<lifeless> __len__ isn't on that interface apparently
<lifeless> or something like that
<asabil> lifeless, it happens only on a specific project
<lifeless> is it private?
<asabil> nope
<lifeless> dunno then
<asabil> ok
<lifeless> oh possibly lots of branches
<asabil> found the bug
<asabil> max_branches_from_query == -1
<asabil> or not
<mwhudson> jelmer: did you write some greasemonkey scripts for all the import herding you've been doing recently?
<lifeless> anyone that works on oops tools around: https://lp-oops.canonical.com/oops.py/?oopsid=1527L2262
<lifeless> oopses
<lifeless> which is ironic
<thumper> morning
<jelmer> mwhudson: no, what I do basically is open all 50 links on the table that lists the imports and then when I have a spare moment I process a tab and close it
<mwhudson> jelmer: :)
<jelmer> I think I've processed most of the failing bzr-svn imports that could be restarted
<jelmer> there are still a few that are fixable but require the branch to be nuked and created again from scatch.
<jelmer> Unfortunately it's not possible to see what sort of subscriptions people have that are subscribed to the branch.
<mwhudson> jelmer: yeah
<mwhudson> that can be done server side, when we can find a sysadmin
<wgrant> And then there are those like mine (multidistrotools) that you retried yesterday that can probably never succeed because somebody imported a bzr branch into svn, versioning even the .bzr...
<mwhudson> ugh
<mwhudson> well
<mwhudson> https://bugs.edge.launchpad.net/launchpad-code/+bug/533225 will help with that?
<mup> Bug #533225: shouldn't create working tree for foreign branch imports <code-import> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/533225>
<mwhudson> won
<mwhudson> 't help you do anything useful with the resulting branch i guess
<wgrant> Right, I saw that yesterday.
<jelmer> well, you could push it back into svn or git and then check it out using either of those tools ? ;-)
<wgrant> But then realised that it wasn't going to help when somebody actually tried to use it.
<wgrant> Heh.
<mwhudson> if you can convince whoever's running the svn repo to delete .bzr from tip though, then that should be enough
<wgrant> True.
<wgrant> I wonder if the repo is still alive.
<wgrant> Doesn't look like it.
<jelmer> mwhudson: btw, is there any reason for launchpad refusing usernames and passwords in import URLs?
<mwhudson> jelmer: er, pass?
<jelmer> mwhudson: it seems like if the importer is allowed to use a particular combination it's not so secret?
<mwhudson> jelmer: i plead "before my time"
<jelmer> mwhudson: (e.g. tigris requires or required test/test as username+pw)
<jelmer> mwhudson: ah :-)
<mwhudson> if there's a reason to change it, i guess we should do that
<jelmer> (bazaar transports support usernames/passwords in URLs so using that would avoid involving the LOSAs where public credentials are involved)
<mwhudson> that sounds like a reason to change it
 * jelmer files teh bug 
 * thumper goes back to reducing unread count in his inbox
<wgrant> Do tree builds really need to take this long?
<wgrant> combinecss and create-lp-wadl take approximately far too long.
<wgrant> And I'm sure that most rarely use either.
<thumper> mwhudson: you looking at bug 497428 ?
<mup> Bug #497428: Import on a large git repository is failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/497428>
<mwhudson> thumper: i'
<mwhudson> thumper: it was reported against launchpad-cscvs
<thumper> ah
<mwhudson> the fix is to scrap the branch and re-import it i think
<mwhudson> which means waiting for spm to materialize
<wgrant> Public holiday.
<mwhudson> yes
<mwhudson> so, tomorrow
 * thumper runs to get Maia and food
<mwhudson> mmm food
#launchpad-dev 2011-02-28
<wgrant> lifeless: One could say the same for any other sourcecode change.
<wgrant> We could probably have make check that.
<wgrant> Do a no-op update-sourcecode run
<wallyworld> lifeless: how do you get the query count to show in the top right corner of the page?
<wgrant> 'visible_render_time default 0 on'
<wallyworld> wgrant: thanks
<wgrant> StevenK: Is this for the base version thing?
<lifeless> wgrant: that might be nice
<lifeless> wgrant: shipit is particularly weird though because it breaks on zcml, not on $specific tests.
<wgrant> lifeless: bzr-builder is also very good at breaking model imports.
<lifeless> ah
<lifeless> wgrant: anyhow, yes, these things all need communication :)
<lifeless> or we could stop using source code
<wgrant> Heh, good luck with that.
<wallyworld> why do we use source code and not eggs for that stuff?
<StevenK> wgrant: Yes
<wgrant> wallyworld: eggs came along much later.
<wallyworld> and we don't want to go in and eggify the source code stuff...?
<wgrant> Well, I don't really like eggs.
<StevenK> And everytime shipit is moved into an egg, it breaks and leaves this rotten smell ...
<wallyworld> wgrant: what is your preference to eggs?
<lifeless> grah
<wgrant> wallyworld: Real packages are nice, because they aren't programming languages feeling that they need to reinvent the package manager :)
<StevenK> We have 3 ways to inject dependencies into our tree: eggs, sourcecode and binary packages from the PPA.
<lifeless> sample branches are not scanned
<lifeless> no wonder its approximately useless.
<wgrant> Also, real packages don't involve downloading unsigned code over HTTP.
<StevenK> wgrant: You mean that doesn't give you a warm fuzzy feeling? :-P
<wallyworld> wgrant: yes, but python has eggs, java has jars etc - it a common paradigm. i can see the point of keeping such things separate from the overhead of a full blown package that is not universal across os's rpm vs apt etc
<wgrant> It is a common paradigm, but it also fucking stupid.
<wgrant> :(
<wallyworld> wgrant: you forgot IMHO :-)
<StevenK> It isn't just his opinion.
<StevenK> *Anyone* who has done a bunch of distribution work shares it.
<StevenK> Except lifeless, because he's special.
<thumper> :-( :-( make: *** [check] Error 1
<thumper> I tried running ec2 test --attached
<thumper> everything seems to pass
<thumper> then I get this ^^^
<thumper> I had several detached ec2 test runs,
<wallyworld> i haven't done any distribution work so i don't yet appreciate the drawbacks of eggs etc. all i see are packaging overheads and competing implementations
<wgrant> thumper: Full output pls.
<thumper> and they all fail
<StevenK> wallyworld: So, we have a wheel. It's round, it mostly turns without issues, but it has been reinvented by PyPI, CPAN, Gems, ...
<spiv> wallyworld: you can't win, really
<wgrant> And then the reinventors of the wheels decide that they don't want anything to do with distributions.
<wgrant> eg. Ruby
<thumper> wgrant: I've emailed it to you
<wgrant> "lol, why would people want distribution packages? end users should use gems"
<StevenK> It seems to be a bunch of NIH to me
<spiv> wallyworld: random python upstreams of course would rather not learn new packaging tools and conventions for Debian, OS X, Windows, etc...
<wallyworld> i'm hit, bleeding badly, can't hang on much longer, copping it from all directions.....
<StevenK> wallyworld: Well, you did just jump up and wave your shirt in the air and yelled "I'm over here, come get some!"
<spiv> wallyworld: but at the same time Debian, Fedora, OS X, etc don't want dozens of non-cooperating packaging systems where one would do
<wgrant> thumper: That is a little bit special.
<wallyworld> spiv: i guess that's one of my points - each os has it's own packaging mechanism, vs eggs for python which is universal
<wgrant> wallyworld: Universal... for Python.
<spiv> wallyworld: it's the "for Python" bit that's the killer
<lifeless> StevenK: uhm, don't malign me please.
<spiv> wallyworld: what happens when a Python package depends on a C library
<StevenK> And then never mind rpm vs apt vs emerge ...
<spiv> wallyworld: etc :)
<lifeless> StevenK: I *loath* gems jars eggs etc
<StevenK> lifeless: I was poking fun, not trying to do so
<lifeless> StevenK: and I've been pretty vocal about that
<lifeless> I would love to just use debs for lp dependencies.
<wgrant> thumper: Ah, there it is.
<StevenK> lifeless: Hm. I had you in my mind as the other way -- so noted, sorry
<wallyworld> spiv: fair point. if only each os would stop adopting it's own packaging implementation
<wgrant> thumper: There are failures in the stream.
<lifeless> but the deb python maintainers want *upstream* to decide on how to handle conflicting APIs for multiple versions of the same package on disk
<lifeless> StevenK: which is never ever ever ever ever going to happen
<wgrant> thumper: But the summary doesn't show them because of a bug I'm currently writing a test for.
<spiv> wallyworld: if only each os was a derivative of Ubuntu :)
<thumper> wgrant: why aren't they in the email?
<lifeless> because python folk wave the virtualenv magic flag
<lifeless> the result is that every python transition is lockstep
<wgrant> thumper: jml broke it when he added ec2 list.
<lifeless> (and I don't mean python VM, I mean individual packages
<wgrant> The fix is easy, but a test is slightly harder.
<StevenK> Package management is hard
<lifeless> and that results in us not being able to shift dependencies at all, which is why we're not using debs.
<lifeless> see for instance my threads on debian-python (both direct and through folk I've been poking about this)
<lifeless> StevenK: its easy if we accept that upstream have different drivers than us
<StevenK> Well, we're using a mix of debs, eggs and source trees
<wgrant> And random things that people thought it was a good idea to import into the tree.
<StevenK> wallyworld: Do you see why we don't discuss this often?
<wgrant> Heh.
<StevenK> wgrant: :-(
<wallyworld> StevenK: yeah, i had no idea what a feeding frenzy it would create. wow :-)
<lifeless> StevenK: and the debs give us routine production fail, the source treess toolchain isn't being fully used and the eggs we have to cripple the toolchain because we like verified code
<StevenK> Baptism of Fire, part #58585
<_mup_> Bug #58585: Installer is crashing (all disk activity stops) at scanning mirror message <ubiquity (Ubuntu):Invalid> < https://launchpad.net/bugs/58585 >
<lifeless> StevenK: but you forgot, we *also* use:
<lifeless>  - bundled code in hidden bits of the tree
 * thumper sighs
<wgrant>  - more bundled code in other better hidden bits of the tree.
<StevenK> lifeless: The debs only fail because there seems to be this magical step where packages jump from the PPA into CAT ....
<wgrant> StevenK: The main problem is the multiple versions issue.
<lifeless> StevenK: thats not the fail
<lifeless> StevenK: the fail is when version A and version B are incompatible with each other as far as LP is concerned.
<lifeless> StevenK: we then have to simultaneously upgrade the LP tree at the same time, or we have a fail
<StevenK> :-/
<lifeless> which means a more complex rollout - take the entire appserver machine offline, (== 20% drop in capacity - when we are already overloaded), upgrade debs, deploy new LP, bring the servers back online
<lifeless> StevenK: what we need is what C/mono etc libraries have - sonames.
<lifeless> then we can install the new debs, lp ignores them, and then we upgrade lp to one that uses the new debs, bingo it works.
<lifeless> StevenK: in a java world things like e.g. osgi go a long way toward this
<StevenK> ... or write code that can work with both version of the code. But I understand that isn't always possible.
<lifeless> but in python 'import' is global, there is no built in concept of 'library version' - you cant say 'import bzrlib 2.2 as bzrlib'
<lifeless> and the plumbing for this goes deep
<lifeless> StevenK: its both not always possible and its /harder/
<wallyworld> osgi rules!
<lifeless> given the choice of two ways to achieve something, only particularly opinionated or previously burnt folk will choose the harder way.
<lifeless> wallyworld: for all its mind bending stuff, yes, yes it does.
<lifeless> so its a long term goal for me to beat enough sense into python to permit this
<lifeless> or JFDI a workaround in the distro using pkg_resources.require()
<wgrant> That could work.
<lifeless> but I suspect we'd end up auto rebuilding the entire python stack
<wgrant> What is Fedora doing about this sort of thing?
<lifeless> because the debian python group /don't get it/
<lifeless> because they are thinking desktops and 'upgrade every 5 years' style server environments.
<lifeless> wgrant: I've not seen anything - reality is most folk deploy python stacks using eggs / the django thingy / virtualenv etc - for the previously discussed reasons.
<lifeless> *and* most really agile things I've seen are php or java, not python, so there isn't that much web world pressure on distros to fix this
<lifeless> (because python is 10 times slower than either php or java)
<StevenK> To write, run, deploy, or all three?
<lifeless> all three
<lifeless> deploy is about equal actually
<lifeless> php apps tend to be very lean
<lifeless> java apps aren't, but they have -wonderful- tooling around development
<lifeless> and for execution time, php and java eat python for breakfast and lunch
<lifeless> the only /serious/ python thing at *scale* I know of is youtube : and they don't run CPython, rather they run bleeding edge experimental binaries
<StevenK> We aren't serious?
<lifeless> StevenK: we're tiny
<lifeless> StevenK: all our problems are self inflicted
<lifeless> we've < 1/2T of data
<james_w> wgrant, I think Fedora still haven't got so supporting multiple versions of the python runtime, so I don't think they've tackled this
<wallyworld> lifeless: python has good development tooling too. you don't *have* to use emacs :-P
<wgrant> james_w: Hah.
<lifeless> a single (very high end) commodity PC server can run our entire backend load
<wgrant> And I'm not going to ask about other distros like Arch... given that they've switch /usr/bin/python to 3.x.
<lifeless> StevenK: we have the potential to be serious if we get our shit sorted and /fast/
<lifeless> I will be delighted when we start getting user adoption scale problems
<wgrant> A year ago I would have said we were doomed to be slow forever.
<lifeless> rather than 'oh and we don't delete obsolete data' scaling issues.
<wgrant> But I think that may not be the case.
<lifeless> wgrant: I wouldn't have taken the job if I thought it was insoluable
<lifeless> bah
<lifeless> unsolvable
<wgrant> lifeless: Yes, but you are crazy :P
<StevenK> It doesn't dissolve in water either
<wgrant> Yet very effective.
 * StevenK peers at subunit
<StevenK> subunit-filter's default operation is to strip sucessful tests. Except one turns up ...
<StevenK> I just love how the Australian Tax Office can give a recorded message saying that they are too busy to take the call ... for 3 weeks solid
<lifeless> StevenK: if the stream gets corrupted by things printing to stdout
<StevenK> lifeless: Such as test failures?
<lifeless> StevenK: you can see what looks like something being not-filtered, whenthe parser actually thinks 'this is junk'
<lifeless> StevenK: test failures shouldn't print to stdout, the raise exceptions internally
<lifeless> StevenK: you can see if this is the case with --no-passthrough
<lifeless> that will squelch what subunit thinks is chatter on stdout
<StevenK> I'm already using --no-passthrough
<lifeless> mm
<lifeless> then if you're seeing a success,
<lifeless> ah perhaps a skip ?
<lifeless> some things fold skip to success
<lifeless> and older subunit had a bug in that area
<lifeless> try --no-skpi
<StevenK> Ahh, no, it's these tests
<StevenK> There are two are cunningly named
<lifeless> woo, another stab at timeouts complete
<lifeless> lp:~lifeless/launchpad/bug-722956
<lifeless> I need a review https://code.launchpad.net/~lifeless/launchpad/bug-722956/+merge/51481
<lifeless> diff is generating still
<SpamapS> lifeless: so.. curious.. how hard is it to setup a project as a distro w/ source packages and such?
<wgrant> lol
<lifeless> SpamapS: impossible
<lifeless> SpamapS: starting this week julians team is working on making it possible
<lifeless> SpamapS: a lot of ground work is in place
<lifeless> they were interrupted by the reorg
<lifeless> StevenK can tell you how much work remains
<StevenK> Lots
<wgrant> Tonnes.
<lifeless> o/~ I need a review o/~ to the sound of 'I need a hero'
<wgrant> Getting it working isn't that hard. Getting it effectively usable is another thing.
<StevenK> I'm working on low-level plumbing
<SpamapS> lifeless: ahh.. thats good.. I think the principia ensemble project I started is going to need package tracking.
<lifeless> are they actually packages ?
<lifeless> or just vaguely similar
<SpamapS> they're called formulas..
<lifeless> should be called instruments
<lifeless> or scores
<lifeless> anyhow ;)
<wgrant> Heh
<SpamapS> lifeless: agreed.. would have been much cooler. :)
<SpamapS> though if it hadn't been formula.. I wouldn't have been able to come up with the name Principia Ensemble
<lifeless> SpamapS: if they are similar but not the same, then I wouldn't shove them in as packages - packages are tied up in the whole package building machinery
<SpamapS> lifeless: yeah I don't need that.. I just need sub trackers
<wgrant> Or do you just want the bugtracking aspects of source packages?
<lifeless> SpamapS: think crisp and clean; we can bring up a dedicated interface pretty directly.
<wgrant> Right.
<lifeless> SpamapS: project groups do that, but the components are 'project' which is pretty big.
<lifeless> if we were to refine 'bug.target' we could do subtrackers on projects quite easily.
<SpamapS> Yeah I'd expect there to be 300 - 500  formulas
<lifeless> thats the sound of 20 engineers screaming at me
<StevenK> Only 20?
<lifeless> StevenK: 5*5 in theory but we're short atm
<lifeless> plus I know at least 2 that agree
<wgrant> lifeless: We're not short any more :(
<lifeless> wgrant: we are, its just going to stay like that for a while
<lifeless> wgrant: are you actually working on https://bugs.launchpad.net/launchpad/+bug/575450
<_mup_> Bug #575450: Archive:+copy-packages nearly unusable due to timeouts <lp-soyuz> <ppa> <timeout> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/575450 >
<wgrant> lifeless: I haven't touched it for a couple of weeks, and don't have branches with any significant improvements. You may steal it if you wish.
<wgrant> It's not really difficult.
<wgrant> I was just trying to make some infrastructure improvements to make testing easier.
 * lifeless gifts wgrant with a laserlike obsession with production performance
<wgrant> eg. ++oops++ doesn't work on form submissions or the webservice (as yet unsolved), query counts are difficult and non-deterministic (both solved)
<lifeless> wgrant: those are great things to do
<lifeless> wgrant: which query counts ?
<lifeless> wgrant: oh, you mean the session thing?
<wgrant> That, and your new matcher.
<wgrant> Although there is still that Storm thing where it will do a SELECT 1 to check if an object exists the second time. But I wonder if store.reset() will fix that.
<wgrant> lifeless: The most difficult bit of those timeouts it the DAS queries.
<thumper> wallyworld, StevenK: mumble?
<wallyworld> thumper: ok
<wgrant> lifeless: They're produced by createMissingBuilds' calls to getBuildByArch... and that is all a really big mess.
<lifeless> zomg
<lifeless> DistributionSourcePackage - 778 repeated lookups
<wgrant> Where? BugTask:+index?
<lifeless> bug 705713
<_mup_> Bug #705713: Bug:EntryResource Timeout trying to set bug private - death by sql / late evalation <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/705713 >
<wgrant> Heat?
<lifeless> not obviously
<lifeless> just 1.4K sql queries
<wgrant> It was a good thing, though.
<lifeless> bbiab, doctor visit.
<thumper> wallyworld: https://dev.launchpad.net/PolicyAndProcess/MaintenanceRotation
<SpamapS> ok so the consensus right now is that sub-trackers will just have to be implemented using tags at the moment.
<SpamapS> ?
<wgrant> SpamapS: Unless you want to use a project group, which you probably don't.
<SpamapS> wgrant: right I don't want to register 600 projects.. some of which will  have 8 - 20 lines of code.
<lifeless> I think tags should work well
<lifeless> we probably have a few small bugs to sort out in making them scale and be usable enough
<lifeless> but that woul dbe significantly less than actual sub trackers.
<lifeless> less work I mean
<lifeless> wgrant: so what are you stabbing today?
<wgrant> lifeless: The last of checkwatches.
<lifeless> wgrant: would you like to get the loggerhead thing too?
<wgrant> lifeless: You mean jam's branch that I reviewed last week?
<lifeless> I think it just needs a merge to trunk + a merge to the lp deploy branch + a sourcecode rev
<lifeless> + a bug filed for capturing timeouts
<lifeless> yeah
<wgrant> launchpad_loggerhead is in the tree now.
<lifeless> 15K in one hit. It would be a record of some sort
<wgrant> It just needs to land.
<lifeless> oh true
<wgrant> Which relies on us being out of testfix.
<StevenK> Which relies on a LOSA
<wgrant> Right.
 * StevenK writes a test, getting UploadFailed, and sighs at this layer crap
<wgrant> Is it unacceptable to cheat by forcing the build then landing stuff?
<StevenK> Haha
<StevenK> It will fail in the same way and never make it to stable
<lifeless> if your thing passed ec2 its fine
<wgrant> Right, it will just fail again, but at least stuff will get landed.
 * wgrant does it.
<lifeless> wait
<lifeless> you could rollback the apt thing
<wgrant> Could.
<lifeless> file a bug with the checklist of missed systems
<wgrant> Ah, it's building again anyway.
<StevenK> Who landed that, anyway?
<lifeless> link it to a branch to reinstate it
<lifeless> jtv I think
<wgrant> StevenK: jtv. The RT says it's rolled out everywhere.
<wgrant> But apparently not buildbot.
<wgrant> (it did ask for it to be rolled out to buildbot)
<lifeless> lp.soyuz.scripts.tests.test_publishdistro.TestPublishDistro.testPublishDistroRun
<lifeless>  soyuz-set-of-uploads_txt
<lifeless> ^ is that the apt thing ?
<lifeless>  soyuz-upload_txt
<wgrant> Yes.
<lifeless> from ec2. sigh, I need a new ami don't I
<wgrant> Erm. AMI 508 should be enough.
<wgrant> Was this a really old branch?
<lifeless> wgrant: I have a different tree for submitting from
<lifeless> so ec2 vaguaries don't bit me - but when the upgrade is mandatory, it bites
<lifeless> thumper: wallyworld: now you're on maintenance - 5th top timeout : 26 /   18  RootObject:+daily-builds :)
<wallyworld> lifeless: i'm working on that one right now
<wallyworld> actually, just context switched to land some branches from alst week
<lifeless> we can't land anthing today
<wallyworld> lifeless: i probably missed the discussion as to what's wrong with landing stuff?
<lifeless> buildbot's vm was not updated with the new apt, so when jtv landded the code to depend on it we went into testfix and cannot get out without a losa
<lifeless> or reverting his branch
<wgrant> And reverting his branch when the fix is so easy and it's not going to cause a QA chain of pain seems bad.
<wallyworld> ok. thanks
<SpamapS> lifeless: is it possible the compaction error is happening because of the low fd limit?
<lifeless> SpamapS: no
<lifeless> SpamapS: its a classdefnotfound error
<lifeless> SpamapS: got a lucid vm ?
<lifeless> SpamapS: I've pushed my test code to lp:~lifeless/oops-repository/trunk
<lifeless> SpamapS: just needs the cassandra ppa, the lp ppa
<wgrant> Hmm.
<wgrant> Codebrowse is down?
<SpamapS> lifeless: ah ok.. right
<lifeless> SpamapS: I think its a missing jar that isn't wrapped into the war for whatever reason
<lifeless> SpamapS: but, branch my code, run make check.
<lifeless> (oh, read README and install deps)
<SpamapS> 3 hours later.. ;)
<lifeless> wgrant: faaaark looks like it
<lifeless> and yet, noone has complained
<lifeless> spm: are you lurking? cause if you are, and aren't on leave, I'd rather bug you than wake elmo.
<StevenK> 3am London time :-(
<wgrant> pjdc: ?
<wgrant> lifeless: Is codebrowse downness critical now? I thought it had some ridiculously high threshold before it could be treated as such.
<wgrant> Hm.
<wgrant> There is no guava.
<wgrant> Everything else is pingable, so I presume the
<wgrant> machine itself is down.
<StevenK> guava replies to pings here?
<wgrant> In..deed.
<wgrant> But not from the DC.
<wgrant> Looks like it is up, and carob can rsync from it, but not ping it.
<wgrant> All the other prod machines are pingable :/
<mwhudson> wgrant: guava is in the dmz
<mwhudson> for stupid reasons
<wgrant> Hmm.
<wgrant> guava CPU graphs are a bit sad.
<wgrant> https://lpstats.canonical.com/graphs/GuavaCPU/
<wgrant> mwhudson: I suppose this isn't documented anywhere that non-IS can see?
<mwhudson> wgrant: not that i know of
<wgrant> Heh.
<mwhudson> i guess you can probably see the rt i filed for "please move guava out of the dmz" about two years ago
<wgrant> Anyway, looks like both codebrowses wandered off to chew on a CPU each at 01:20
<wgrant> Ah, Francis rejected it.
<wgrant> Sad.
<wgrant> StevenK: OOPS-1884A1684
<lifeless> wgrant: its gritical
<lifeless> wgrant: we have two redundant codehosting instances
<lifeless> why was the move rejected ?
<lifeless> wgrant: so yes, if its down, it needs to be fixed. This is bread and butter facilities for our users.
<wgrant> lifeless: See the bottom of https://rt.admin.canonical.com/Ticket/Display.html?id=33173
<wgrant> lifeless: I recall that previously it has been said that it was not as critical as everything. I disagree, but that is how it was.
<lifeless> if we've killed the arch mirror
<lifeless> we can move within the main network
<lifeless> wgrant: the old definition of critical is what you're thinking of, I think
<lifeless> wgrant: that got overhauled late last year
<wgrant> I recall it also had the Soyuz threshold at 24 hours, or something stupid like that.
<lifeless> any launchpad service or feature becoming unusable
<lifeless> immediately
<lifeless> ...
<lifeless> We aim to have none of these, ever.
<lifeless> ...This includes beta features
<StevenK> wgrant: :-(
<wgrant> StevenK: ?
<StevenK> wgrant: The OOPS you linked me
<wgrant> Ah, yes.
<StevenK> I bet I just generated another oops
<StevenK> Hm. Only the no signer packages do that.
<StevenK> Why do I have this bad feeling there's a deleted recipe involved.
<wgrant> There is.
<StevenK> :-(
 * StevenK tries to swap in the branch details.
<StevenK> wgrant: So a test for this would be create a recipe, link it to an SPR, call recipe.destroySelf() and then look at the view?
<wgrant> StevenK: I think so.
<wgrant> You may need to make sure that the SPRelease has no signer.
<StevenK> That's going to be fun to do in TAL.
<wgrant> Why?
<wgrant> (I suspect it's more your new thing than the signer, but best to be thorough)
<StevenK> I certainly am blaming my addition
<StevenK> wgrant: I have a test written, let's see ...
<StevenK> wgrant: Confirmed, the test fails in the same way as the OOPS.
 * thumper is attacking a sub-100k bug number
<thumper> made more fun by the blueprint and email aspect :)
<StevenK> Whee, 350 lines of traceback
<StevenK> wgrant: So, I can ignore printing the built by if the recipe is deleted, or I can say 'deleted recipe' -- what are your thoughts?
<wgrant> I'd say "deleted recipe"
<StevenK> ... or can I? How do you do if else in TAL?
<StevenK> Or can haz a link to TAL documentation that doesn't make my eyes bleed?
<wgrant> You'll have to use a tal:condition.
<wgrant> There's nothing like genshi's py:when.
<lifeless> tal:iban
<StevenK> lifeless: Two drums and a cymbal fall off a cliff.
<lifeless> boom boom tish ?
<StevenK> PTRuntimeError: ['Compilation failed', 'HTMLParser.HTMLParseError: EOF in middle of construct, at line 34, column 9']
<StevenK> :-(
<thumper> pretty simple review: https://code.launchpad.net/~thumper/launchpad/blueprint-email-handler-bug-83414/+merge/51488
<thumper> and un-contentious
<wgrant> Isn't it illegal to have a Blueprint branch which adds more lines than it deletes?
<lifeless> no
<thumper> lifeless: thanks
<lifeless> do the simplest thing possible
<lifeless> velocity with solid code
<thumper> wgrant: not when the addition is tests
<lifeless> doesn't matter what it is
<lifeless> its part of the same code base
<lifeless> same rules as for anywhere else apply
<lifeless> *if* IssueTracker goes ahead, it will get folded in
<lifeless> if it doesn't, we need it maintained, and that means having a level playing field for patches to it.
<lifeless> we *must not* make any part of the code less pleasant to work on than another, culturally, technically or otherwise.
<StevenK> :-([A[APTRuntimeError: ['Compilation failed', 'zope.tal.taldefs.TALError: Invalid variable name "recipe" in expression u\'not sprb/recipe\', at line 37, column 9']
<StevenK> Can I not do that in TAL?
<lifeless> StevenK: what are you trying to do
<wgrant> StevenK: Not like that.
<lifeless> its also a bad idea even if you could
<lifeless> you need to use the ID if you don't want to trigger lazy loading
<wgrant> 'not: sprb/recipe'
<LPCIBot> Project devel build #482: STILL FAILING in 5 hr 29 min: https://hudson.wedontsleep.org/job/devel/482/
<StevenK> Hm. I think that's with the new apt
<lifeless> wgrant: probably a bad idea unless the rest of the page is guaranteed to show the recipe
<lifeless> wgrant: not: sprb/recipeID
<wgrant> lifeless: The rest of the page is guaranteed to show the recipe.
<lifeless> wgrant: kk. I will keep pushing this meme :)
<wgrant> Does anybody have a Lucid LP machine?
<wgrant> I guess I'll fire up an ec2 instance.
<lifeless> I develop in a lucid vm
<lifeless> whats up
<wgrant> Ah.
<wgrant> Can you run test_generateConfig?
<wgrant> In devel, with the new apt.
<wgrant> That is apt *ubuntu9.3+ppa1, or *ubuntu9.4
<lifeless> 0.7.25.3ubuntu9.3+ppa1
<wgrant> Does test_generateConfig pass?
<lifeless>  lp.archivepublisher.tests.test_ftparchive.TestFTPArchive.test_generateConfig
<lifeless>  lp.archivepublisher.tests.test_ftparchive.TestFTPArchive.test_generateConfig_empty_and_careful
<lifeless>   Ran 2 tests with 0 failures and 0 errors in 7.168 seconds.
<wgrant> TestFtparchiveIndices
<wgrant> StevenK: Is it easy enough to fire up something identical to the Hudson slave and run a test on it?
<wgrant> s/Hudson/Jenkins/
<lifeless>   Running:
<lifeless>  lp.archivepublisher.tests.test_publisher.TestFtparchiveIndices.testAllIndicesArePublished
<lifeless>  lp.archivepublisher.tests.test_publisher.TestFtparchiveIndices.testNoIndicesForDisabledArchitectures
<lifeless>  lp.archivepublisher.tests.test_publisher.TestFtparchiveIndices.testWorldAndGroupReadablePackagesAndSources
<lifeless>   Ran 3 tests with 0 failures and 0 errors in 19.608 seconds.
<wgrant> :(
<wgrant> Thanks.
<StevenK> wgrant: Yes
<wgrant> StevenK: Could you fire one up, check the apt version, try test_generateConfig and TestFtparchiveIndices, and then stab a-f in the face?
<wgrant> Ah, it's actually apt-utils that matters, but they are presumably the same.
<wgrant> Although I wonder...
<StevenK> Let's find out
<wallyworld> lifeless: are you able to run a query against qastaging or prod and gather stats? or do i need a dba?
<StevenK> wallyworld: All squad leads can run ro queries against {qa,}staging
<StevenK> wgrant: Waiting for the builder to install
<wallyworld> StevenK: cool thanks. i'll ask thumper :-) i also need an explain done
<wgrant> StevenK: Thanks.
<StevenK> wgrant: I'm doing it by hand, btw
<wgrant> :(
<lifeless> wallyworld: I can and thumper can
<lifeless> wallyworld: pastebin away
<StevenK> The following polling activities are currently in progress: db-devel 7 hours 24 minutes
<StevenK> Eeeeep
<StevenK> lifeless: How can I clean that up?
<wallyworld> lifeless: it's a first cut at improving the dialy builds query. still has the group by but cuts out 3 tables from the join and appears perhaps significantly faster
<lifeless> StevenK: the poll ?
<wallyworld> lifeless: https://pastebin.canonical.com/44001/
<StevenK> lifeless: Right
<lifeless> StevenK: have a look - see if there is a wedged bzr process (and if so file a bug :))
<wallyworld> lifeless: i just am interested in a comparison with what it does currently to see how much improvement there is
<StevenK> There is. What can I do to it to get some debugging?
<wallyworld> just tried to navigate to +daily-builds on qastaging and that timed out so qastaging's db should be sufficient to test against
<lifeless> wallyworld: Time: 11231.630 ms
<wallyworld> lifeless: :-( that's no good. i was expecting much better
<lifeless> https://pastebin.canonical.com/44002/
<lifeless> wallyworld: If I were working on this, I'd be executing on that thread we had about how the current page can be redesigned.
<lifeless> wallyworld: I don't see any way of making it fast by sql tricks under the current schema
<lifeless> your query is Time: 616.543 ms hot
<lifeless> but IIRC the existing query is about that hot too
<wallyworld> lifeless: yeah, but i just wanted to experiment a bit to see how much removing 3 tables from the big fat join would affect the results
<lifeless> the 11 seconds is cold
<lifeless> what tables did you drop, and what is the new definition of the results
<StevenK> lifeless: So this bzr has an open socket, and is blocked in recv() -- doesn't seem like much of a bug to me
<wallyworld> lifeless: i dropped the joins to person, sourcepackagename, archive tables and got that data using a single query for each using the pre_iter hook
<lifeless> StevenK: hmm, network should have killed it by now though
<lifeless> StevenK: 7 hours ...
<StevenK> You'd think
<wallyworld> lifeless: so the results are the same but without the ordering on archive , spname
<StevenK> lifeless: I'm happy enough to wield kill and move on
<lifeless> StevenK: yeah
<lifeless> wallyworld: ah, you'd post process to do the archive ordering ?
<wallyworld> lifeless: well, considering it if the query results were any good but alas. i really thought cutting out 3 tables would help
<lifeless> wallyworld: it may well have
<lifeless> wallyworld: staging cold is pretty cold
<lifeless> the query plan cost of 5320.82 is pretty reasonable
<wallyworld> lifeless: i did it to just to see the numbers plus to teach myself how to use the pre_iter hook and do eager loading and insert SQL() expressions etc into storm queries
<StevenK> https://code.launchpad.net/~stevenk/launchpad/link-deleted-recipe-ppa-page/+merge/51490
<lifeless> wallyworld: its almost certainly an improvement:)
<lifeless> wallyworld: I encourage you to get it reviewed and land it, even if you do more subsequent work on the page.
<wgrant> wallyworld: Did you use SQL() just because you could, or is Storm missing something?
<wallyworld> lifeless: no, i had to use the SQL since i wasn;t selecting a whole table (class) anymore and storm complained if i used that expression in the group by or select clauses
<wgrant> Hmm....
<wgrant> You should be able to do that in Storm.
<wallyworld> lifeless: my thoughts with this were i would try it as a quick win and if it helped, deploy it and iterate again to doa better solution, so you're thinking matches mine
<lifeless> wallyworld: whats the class you were not selecting anymore ?
<lifeless> (not selecting the whole table, I mean)
<StevenK> wgrant: Can haz review? ^
<wallyworld> lifeless: SourcePackageRelease - the only thing i need from that table is the sourcepackagename column to that i can load the sourcepackagename objects
<wgrant> StevenK: Sure.
<wallyworld> lifeless: anything in the select() has to go in the group by so i wanted to remove as much as possible
<lifeless> wallyworld: SourcePackageRelease.sourcepackagenameID is what you should use
<lifeless> wallyworld: see my mail about fooID vs foo - last tuesdays performance tuesday mail
<StevenK> wgrant: Ignore the comment for test, changing pushing now
<lifeless> wallyworld: SourcePackageRelease.sourcepackagename is a Reference and storm utterly fails to compile those as you might want it to
<wallyworld> lifeless: let me try that.  i thought that sourcepackagename was the id column in this case but i'm wrong. on a related note, i got a bit confused between xxx_id and xxxID
<lifeless> wallyworld: so xxx_id is how folk writing native storm often spell xxxID.
<lifeless> wallyworld: xxxID is magically created by the sqlobject metaclass
<lifeless> woo - another perf fix - lp:~lifeless/launchpad/bug-711064
<wallyworld> lifeless: ah ok. i'm not fluent in storm yet. *much* better at Hibernate :-)
<wallyworld> lifeless: off to pick up the kid from school. will clean it up when i get back and put up a merge proposal as a first step
<lifeless> coolio
<StevenK> If Hibernate is what you do at night, I'm pretty good at that too
<wallyworld> StevenK: it's what i did before i joined Canonical :-)
<wgrant> StevenK: Any progress on Jenkins?
<StevenK> wgrant: "update-sourcecode"
<wgrant> Yay..
<StevenK> However, the node is about 2 hops from crowberry, so bzr is nice and fast
<wgrant> Launchpad looks so nice in the Ubuntu font's Light variant...
<wgrant> Gnargh.
<StevenK> wgrant: "make schema"
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-711064/+merge/51491
<lifeless> another I can has review plox
<lifeless> hmm
<lifeless> linking to a bug report has broken
<lifeless> if you put an invalid thing in, it just goes to lala land
<wgrant> lifeless: https://code.launchpad.net/~stevenk/launchpad/link-deleted-recipe-ppa-page/+merge/51490
<lifeless> bah
<lifeless> back to bugtask:+index or distribution:+bugs
<StevenK> Dear buildout, MOVE IT
<lifeless> wgrant: StevenK: btw, thats awfully close to 'too much code in the template'
<wgrant> lifeless: BugTask:+index
<wgrant> lifeless: It is.
<StevenK> lifeless: Absolutely
<lifeless> I'd consider a view property to give you want you want instead
<StevenK> So I'm happy to refactor slightly
<lifeless> I value the fix more than the pretty
<lifeless> Its just a thought for future work.
<lifeless> no point reworking what you have
<lifeless> dinner
<StevenK> lifeless: Well, I already thought it was ugly when I submitted it ...
<StevenK> I'm unsure if I can just add a property onto SourcePackageRecipeBuild and have it return a linkifed string
<wgrant> StevenK: You'd do it on the view.
<StevenK> wgrant: make schema done. Remind me the tests to run, sorry?
<wgrant> StevenK: test_generateConfig should do.
<wgrant> StevenK: But apt and apt-utils versions first pls
<StevenK> ii  apt                             0.7.25.3ubuntu9.3+ppa1          Advanced front-end for dpkg
<StevenK> ii  apt-utils                       0.7.25.3ubuntu9.3               APT utility programs
<wgrant> Ahaaaa
<wgrant> There is our problem.
<wgrant> does apt-get upgrade want to fix it?
<StevenK> Yes
<StevenK> I don't run apt-get upgrade on bring-up
<wgrant> You explicitly upgraded apt? Or does it just install launchpad-dependencies each time?
<StevenK> It explicity installs launchpad-developer-dependencies
<wgrant> OK, I'll fix that.
<wgrant> And update the RT.
<wgrant> Thanks.
<StevenK> wgrant: You think the buildbot slaves have the same issue?
<wgrant> StevenK: It would not surprise me.
<StevenK> Neither
<LPCIBot> Project devel build #483: ABORTED in 58 min: https://hudson.wedontsleep.org/job/devel/483/
 * StevenK tries to figure out how to get a URL in a view
<wgrant> canonical_url
<StevenK> wgrant: http://pastebin.ubuntu.com/573322/
<wgrant> StevenK: Use a structuredstring or I will
<wgrant> stab you with an XSS.
 * StevenK smirks
<StevenK> wgrant: Let me know when you've fixed launchpad-dependencies and friends and I'll tell Jenkins it can build.
<wgrant> StevenK: Uploaded.
<wgrant> We should really run process-upload more frequently.
<StevenK> We should wait for the publisher anyay
<StevenK> *anyway
<wgrant> I will coerce bigjools into bumping it to */1 tonight.
<StevenK> Really?
<wgrant> Well, I will at least try.
<StevenK> Personally, I liked his idea that process-upload is always running
<wgrant> We already run it that often on cesium.
<wgrant> Sure.
<wgrant> That would be nice.
<wgrant> But it's much harder than changing one bit in a crontab.
<StevenK> wgrant: So, you mean "from canonical.launchpad.webapp.menu import structured" or do I misunderstand?
<wgrant> StevenK: That seems to be where it lives :(
<wgrant> Karma, I hate you.
<StevenK> wgrant: Happy with the method name?
<wgrant> StevenK: You could possibly drop source_package_, but apart from that it's good.
<wgrant> And return None instead of "", I think.
<wgrant> jtv: Hi.
<StevenK> Yeah, already fixed that
<jtv> hi wgrant
<jtv> hi StevenKâI'm afraid I only gained a few hours on my previous timezone!
<wgrant> jtv: Did you talk to IS about the apt upgrades?
<wgrant> It seems like buildbot only has apt upgraded, not apt-utils.
<wgrant> (a-f lives in the latter, so buildbot is now broken)
<jtv> Damn.
<wgrant> I'm wondering if you have any details beyond what's in the ticket.
<wgrant> Since we are ISless.
<jtv> Are we!?
<wgrant> Until London wakes up, yes.
<jtv> Where's the ticket?
<StevenK> jtv: spm would have only arrived home a few hours ago
<wgrant> https://rt.admin.canonical.com/Ticket/Display.html?id=43891
<jtv> oh, poor manâwhere were they sprinting?
<wgrant> London.
<StevenK> LHR-SYD-CBR is like 34-38 hours
<wgrant> Not quite *that* bad.
<StevenK> Door-to-door, it can be
<wgrant> True.
<StevenK> Right, I think that's enough time -- re-enabling Jenkins
<wgrant> StevenK: Not published in Lucid yet.
<wgrant> Will be in 4 minutes, hopefully.
<jtv> I wonder whether it's worth rolling back the change.  Probably not.
<wgrant> Not at this stage.
<StevenK> That's okay, I think I broke Jenkins view of the build anyway
<LPCIBot> Project devel build #484: STILL FAILING in 58 sec: https://hudson.wedontsleep.org/job/devel/484/
<jtv> It's probably not worth rolling back the change, but it's damned annoying.
<StevenK> LPCIBot: Shhh
<wgrant> Hudson should be fixed in a few minutes, and buildbot once we have an mthaddon.
<wgrant> And that should be in just a few hours.
<wgrant> So we'll live :)
<jtv> Still, buildnot build 666 couldn't very well have gone without problems, could it?  Just not appropriate.
<wgrant> Heh
<StevenK> wgrant: tal doesn't like structured()
<wgrant> StevenK: What are you doing?
<StevenK>     <li tal:content="view/recipe_build_details" tal:condition="context/sourcepackagerelease/source_package_recipe_build" />
<wgrant> StevenK: s/context/structure context/
<wgrant> Er.
<wgrant> Except on the other one.
<jtv> er :)
<wgrant> s/view/structure view/
<StevenK> wgrant: http://pastebin.ubuntu.com/573328/
<wgrant> StevenK: The tal:condition is redundant now.
<StevenK> wgrant: Really?
<wgrant> StevenK: recipe_build_details has an 'if sprb is not None'
<StevenK> wgrant: Oh, that diff is against the approved MP rather than submit:, but I guess you figured that out
<wgrant> StevenK: ppa:launchpad is now all published.
 * StevenK fiddles with Jenkins
<wgrant> Jedson may be happy now.
<StevenK> Jedson. *facepalm*
<StevenK> wgrant: So you're happy with my refactor and I should commit and push?
<wgrant> This whole one-RT-account-to-rule-them-all thing is pretty confusing.
<wgrant> StevenK: No, because you're bypassing the entire popint of structured()'
<wgrant> Spot the XSS.
<wgrant> Also:
<wgrant> Single quotes around XML attribute values: Australia says no.
<StevenK> Double quotes? :-)
<StevenK> Orsum. structured has no help
<wgrant> Do you know what it does?
<StevenK> Spits out an instance of IStructuredString
<wgrant> Well, yes...
<StevenK> And if I'm reading this doctest right, it escapes its second argument
<wgrant> Right. It takes a format string as its first argument, and interpolates it after escaping the following arguments
<StevenK> Ah ha. Fixed by dropping % (
<wgrant> Right.
<StevenK> wgrant: Also dropped the structured() from 'deleted recipe' -- what's the point? :-)
<wgrant> Indeed.
<StevenK> wgrant: Would you like a new diff, or a commit and push?
<wgrant> StevenK: A new diff would be nice.
<StevenK> wgrant: Oh, from the new builder bring-up:
<StevenK> Get:1 http://ppa.launchpad.net/launchpad/ppa/ubuntu/ lucid/main apt-utils 0.7.25.3ubuntu9.3+ppa1 [240kB]
<wgrant> Excellent news.
<StevenK> wgrant: http://pastebin.ubuntu.com/573331/
<StevenK> Now to create an EBS volume with devel and db-devel checked out
<wgrant> StevenK: Not sourcecode too?
<StevenK> wgrant: Well, yes
<wgrant> shipit has the LP history, so it takes ages.
<StevenK> Jenkins can't check out shipit
<wgrant> StevenK:
<wgrant> You are missing quotes now...
<StevenK> "recipe <a href=\"%s\">%s</a>",
<StevenK> Rargh!
<wgrant> Use single quotes on the string.
<wgrant> So you can use unescaped double quotes inside it.
<StevenK> Now you're just being difficult. Fixed.
<StevenK>     <li><structured-string '<a href="%s">Built</a> by %s for <a href="%s">%s</a>'></li>
<StevenK> :-(
<wgrant> StevenK: Ah, add /escapedtext to the TALES
<StevenK>     <li tal:content="structure view/recipe_build_details/escapedtext" />
<StevenK> wgrant: ^
<wgrant> Yeah.
<wgrant> Our templates should really do this automatically.
<wgrant> A couple of people have made that same mistake recently.
<StevenK> Oh, damn it
<StevenK> Now it escapes the recipe link
<wgrant> Hah, I wondered if it might do that.
<wgrant> I presumed it would be smart enough to find notice that it too was a structuredstring...
<StevenK> *And* confuses TALES enough to get: LocationError: (None, 'escapedtext')
<StevenK> OH
<StevenK> Damn it
<StevenK> That's TALES complaining when recipe_build_details() returns None
<wgrant> Hah.
<StevenK> F. A. I. L.
<wgrant> You could make recipe_build_details return the .escapedtext
<StevenK> It's a property or a method?
<wgrant> Property.
<StevenK> Now to figure out how to stop it escaping the recipe link
<lifeless> noddy queries R us: WHERE Person.id = $INT LIMIT $INT:
<StevenK> Hah
<wgrant> That sounds like a .find(Person, id=1).any()
<wgrant> I don't think Storm is stupid enough to do that itself.
<StevenK> wgrant: And no, structured() just does it, there's no special casing for an instance of IStructuredString
<wgrant> I may have to declare war on our shitty string handling infrastructure at some point.
<StevenK> wgrant: Before or after your current war of "Death to lib/canonical" ?
<wgrant> StevenK: I only deleted lib/canonical/widgets last week :(
<wgrant> The active subwar is "death to Zopeless", which is going OK.
 * lifeless whispers death to timeouts
<wgrant> More like death to checkwatches.
<lifeless> tim is looking at checkwatches mantis handling
<lifeless> I'm awed
<wgrant> Also, tomorrow it is death to long PQM and cronspam.
<StevenK> Heh
<lifeless> wtf
<lifeless> SQL time: 2724 ms
<lifeless> Non-sql time: 11740 ms
<wgrant> Which view?
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1884H2231
<wgrant> Huh...
<wgrant> Ah, anonymous.
<lifeless> I wonder how many of these timeouts are due to e.g. storm overreleasing the gil (when its about to grab it again 2us later) and getting terrible scheduler performance as a result
<lifeless> also memcache really isn't pulling its weight
<lifeless> 232.1366124msmemcache
<lifeless> bah
<lifeless> 24ms anyhow
<lifeless> on a get
<lifeless> time to turn it off
<StevenK> And remove it?
<lifeless> no
<lifeless> its useful for the front page
<lifeless> we should eager load the blog posts thoughj
<lifeless> and depend on them being in memcache
<StevenK> Would s/\(its\) \(useful\)/\1 only \2/ be more accurate?
<lifeless> not entirely
<lifeless> need to analyse case by case
<lifeless> stub: hi
<lifeless> stub: we didn't get the staging restore script quite right, I've filed a bug and assigned to you; it didn't set the value for the flag correctly (should be 'on' not NULL).
<jtv> StevenK: I'm puzzled by something in the derived-distro work.  Hoping you can help.
<StevenK> jtv: Hm?
<jtv> Well, AIUI there was supposed to be a static maximum of 1 layer of derivation.
<jtv> I got the impression that this was a completely separate relationship from those within a distro.
<wgrant> I hope not.
<jtv> Ah
<wgrant> We can have Debian Sid -> Ubuntu 10.04 -> Linaro 10.05
<jtv> I'm asking because DistroSeriesDifference.new rather suggests that this is all based on parent_series.
<wgrant> But the relationship is separate from the distro.
<StevenK> Direct parent, we don't really want diffs from anything else
<wgrant> Right. At the moment the derivation relationship is not modelled.
<wgrant> So everything uses parent_series.
<jtv> So as things stand, we can't have Debian Sid â Ubuntu 10.04 â Linaro 10.05.
<wgrant> At the moment we can have no derivation at all.
<jtv> So how does parent_series figure in all this?
<StevenK> It was the best place to put it, but it's the wrong place.
<wgrant> It does not affect derivation. Derivation needs a new attribute.
<jtv> Okay, so that check for "this is a derived distroseries" will use the new attribute?
<StevenK> Right
<jtv> Thanks.
<lifeless> uhm
<lifeless> why a new attribute?
<jtv> ?
<jtv> Because it's not the same thing as the "parenthood" relationship we have between ubuntu releases.
<wgrant> There are two types of derivation.
<wgrant> Right.
<lifeless> why do we have that relationship?
<jtv> Arguably I suppose we _could_ model this as "parent_series is not of the same distro," but then you still can't model the Sid â 10.04 â 10.05 chain.
<jtv> Why do we have which one?
<lifeless> jtv: sure you could; linaro isn't ubuntu, its derived from. new distro
<lifeless> jtv: we do we have a series->series relation within ubuntu, we don't have that for productseries.
<wgrant> We use parent_series to do some fairly revolting build-related stuff right now.
<jtv> Not to mention translations.
<lifeless> details
<lifeless> oh crud
<jtv> There have been times I've wished for a parent_series on productseries, or a "trunk" series for Ubuntu, but I guess they're just different ways of workign.
<jtv> *working.
<lifeless> we loaded the bug subscription filters on qastaging didn't we
<StevenK> Ubuntu does have trunk. It's called .currentseries
<wgrant> jtv: The divergences are bugs.
<jtv> wgrant: huh wha?
<jtv> StevenK: sid is a trunk, natty is a current series.  Not the same to me.
<jtv> Or if I'm wrong, I'll look mighty stupid for all the times I've told people "I'll do that when sid gets released."
<StevenK> jtv: Huh? How is sid a trunk?
<wgrant> Ah, I see your point.
<StevenK> Sid will never get released
<StevenK> But, right
<lifeless> so take bzr; it has a trunk and a series that will be released
<jtv> StevenK: which is exactly why I like to tell people I'll do their tedious work for them once it gets released. :)
<StevenK> Yes, I get it
<lifeless> our *data* model permits both
<lifeless> why does our *data model* differ between productseries and distroseries in this fashion.
<jtv> That's a good question.
<StevenK> I have some people you could ask. They don't work for us any more, though.
<lifeless> let me ask it differently.
<jtv> If we take the view that There Has To Be a parent series, then either it's always trunk or things get a bit weird.
<lifeless> what stops us removing 'parent_series'
<StevenK> lifeless: The fact that Ubuntu O might want to be initialised
<lifeless> StevenK: I don't understand the relevance
<wgrant> i-f-p doesn't have to use parent_series.
<jtv> But we're not actually making use of the tree structure that the current data model supports through parent_series, are we?
<StevenK> It does currently, though
<jtv> AFAIK it's all neatly linear.
<jtv> Maybe it's really a holdover from when distroseries was distrorelease.
<jtv> When you don't separate series from releases, then parent_series (or rather, parent_release) makes more sense.
<jtv> Or am I getting my history screwed up and was it ProductSeries that used to be ProductRelease?  I _think_ I got it right the first time.
<lifeless> grrr
<lifeless> I *wish* the ++show decorator didn't trim the report.
<lifeless> I think I'll fiddle that next
 * jtv checks the vestigial test factory
<wgrant> jtv: It was DistroRelease
<jtv> Yup
<wgrant> That's why we still have dr and dar everywhere.
<jtv> But you see my point?  parent_release makes a lot more sense when you don't have series.
<wgrant> A little.
<jtv> AFAICS the series replace a generic tree structure of derived releases with a fixed two-level one: series are created in progressive order and within each series, releases are created in progressive order.
<wgrant> Except that we don't model distroreleases yet, but yes.
<jtv> Parenthood probably isn't a very good relationship between releasesâyou get issues like "what if nobody tagged the most recent common ancestor?"
<jtv> And "if I backpatch a bugfix to a maintenance series, isn't it arbitrary to say that the resulting maintenance release is a child of the series' preceding release but not from the fix in the development series?"
<jtv> So maybe lifeless has a point, and we want "predecessor" (which we could probably derive) rather than "parent" (which is more generic than AFAIK we actually use)
<jtv> More importantly, of course,
<jtv> I'm out of coffee.
<wgrant> And out of Internet?
<lifeless> we're still testfuxed, right ?
<wgrant> Yes.
<wgrant> Hudson should be testfixfixed, but buildbot needs an upgrade.
<lifeless> oh my
<lifeless> -fail-
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/bugtask.py", line 512, in _get_task_for_context
<lifeless>     for bugtask in list(bug.bugtasks):
<stub> lifeless: Is NULL ever a valid feature flag value? I suspect not, since there is no way to add that via the web ui.
<wgrant> lifeless: That looks like the NullBugTask infestation.
<wgrant> Or is that somewhere else?
<lifeless> stub: no, its not
<lifeless> wgrant: no, its the traversal
<lifeless> wgrant: the traversal lazy loads *every bugtask*
<lifeless> wgrant: sorry, late evaluates every bugtask target
<wgrant> lifeless: That shouldn't be lazy...
<wgrant> Right.
<lifeless> wgrant: its late, my phrasing is fubared
<lifeless> stub: for clarity, NULL isn't valid - thats why staging is down
<lifeless> stub: but it wasn't meant to be null, or '', it was meant to be 'on'
<lifeless> stub: if you could fix staging that would rock
<stub> Just clarifying Bug #726128
<_mup_> Bug #726128: staging restore script creates invalid feature rule <oops> <Launchpad itself:Triaged> <Launchpad Staging Scripts:In Progress by stub> < https://launchpad.net/bugs/726128 >
<stub> lifeless: staging db updated
<lifeless> stub: brilliant, thanks, thats made it happy
<lifeless> stub: , all - have a great day
<lifeless> I'm going to call it a night for now
<lifeless> I hope to be mass landing branches in the morning :)
<lifeless> stub: you're on call reviewer now aren't you? if so - https://code.launchpad.net/~lifeless/launchpad/bug-722956/+merge/51481 and https://code.launchpad.net/~lifeless/launchpad/bug-711064/+merge/51491 are product from today.
<stub> lifeless: k
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: henninge | https://code.launchpad.net/launchpad-project/+activereviews
<jtv> StevenK: did you get my questions in this channel around one hour ago?  Or perhaps answer them already?  My connection broke not long after I asked.
<StevenK> jtv: If the other context is "I am out of coffee" none of them looked like questions?
<jtv> "This connection has been quiet for minutes now.  That's not normal for Internet (http, torrents, and games).  Let's drop it."
<jtv> StevenK: then they didn't make it out.  Mind if I paste?  It's pretty basic but I like to have my assumptions confirmed sometimes.  :)
<StevenK> jtv: Sure, go ahead
<jtv> OK, here goes:
<jtv> StevenK, to make sure that I still have the right picture: we're trying to get to a state where every new sourcepackagepublishinghistory record for a distroseries that's in a derivation relationship also triggers creation of zero or more distroseriesdifferencejobs for distroseries that it's derived from and ones that are derived from itâ¦ correct?
<jtv> and if so, the processing of a distroseriesdifferencejob may result in a distroseriesdifference being created or updated, right?
<jtv> Or deleted, if the difference goes away.  I guess that's how synchronizing a package would go through the system.
<StevenK> jtv: "distroseries that it's
<StevenK>                derived from"
<StevenK> Only. Since the child didn't change, and we only do one level of indirection
<jtv> StevenK: OhâI thought reviews "upstream" from the derived series were to be reviewed (and optionally synchronized) through the same UI.
<StevenK> jtv: Right, but not all the way upstram
<StevenK> *upstream
<jtv> OK, so all I need to worry about is "this new SPPH may constitute a new, or removed, difference from the parent DS."
<StevenK> jtv: *Or* the child. Or both
<StevenK> In the Debian Sid -> Ubuntu 10.04 -> Linaro 10.05 case, if a new SPPH is thrown to Ubuntu, we want new DSDs for the Debian <-> Ubuntu case and then Ubuntu <-> Linaro case
<StevenK> s/then/the/
<StevenK> wgrant: Tossing the ugly TAL branch at ec2
<jtv> StevenK: okay, but then I don't understand the correction you made earlierâ"distroseries that it's derived from, only."
<StevenK> jtv: Following on from my earlier example, if there is new SPPH in Debian, I thought you meant Debian <-> Ubuntu and Debian <-> Linaro
<jtv> Ah OK, no I meant exactly what you said.
<jtv> So violent agreement there.
<StevenK> Agreed
<jtv> Cool.  Then I can start looking at how to approach implementation.
<mrevell> Hello
<wgrant> Morning mrevell.
<jtv> hi mrevell!
<jtv> StevenK: I suppose a derived series can have a SPPH with the same SPR as a SPPH in the parent release, in which case there's no need for a DSD?
<StevenK> jtv: Exactly
<jtv> s/parent release/parent DS/
<jtv> (in case I'm not making my acronym quota)
<jtv> I think there was mention of pre-existing derived seriesâ¦ how are those modeled?
<StevenK> There was? I can't recall
<wgrant> jtv: Just wait until you start dealing with recipes.
<wgrant> Then you have SPRs producing SPRs.
<wgrant> And then they end up creating DASBPRs!
<jtv> cool
<jtv> The last bit of course makes perfect senseâbut I wouldn't think it was new.
<jtv> Ahh, the days when an inches-thick A4-sized operating system manual would boast "this is the only acronym in the UI module"â¦
<jtv> YKYBRUTLWâ¦
<wgrant> I think I got that one.
<jtv> That is one of the many forms of YKYBRUTLW: YKYBRUTLWYUALT
<jtv> You Know You've Been Reading Usenet Too Long When You Understand Acronyms Like These.
<wgrant> Heh.
<StevenK> Ooh, ITYJMTUOTS
<StevenK> I Think You Just Made That Up On The Spot
 * wgrant is beginning to suspect that checkwatches is in fact an elaborate trap set for those who want to write tests for it.
<wgrant> It's like Soyuz, except with 10 extra levels on indirection, some of which are XML-RPC.
<jml> heh
<jtv> wgrant: yesâ¦ there are some helpers for it though; allenap knows
 * jtv hates xmlrpc
<StevenK> Vallium helps with checkwatches
 * jtv once told one of the makers of XML it was all his fault
<allenap> Strychnine is the checkwatches drug of choice.
<allenap> wgrant: You have my unending sympathy. What are you trying to test?
<allenap> The question "what are you trying to test" can be hard to answer with checkwatches.
<wgrant> allenap: Exception handling in _updateBugTracker.
<wgrant> I think I will just entirely patch out _getExternalBugTrackersAndWatches, rather than trying to get a custom ExternalBugTracker in.
<allenap> wgrant: Yeah, good idea.
<jtv> StevenK: when comparing SPPHs for two DSes, can I just compare the two SPPHs with the most recent base_versions?  Or do I need to do this separately for each base_version I find?  Or is there even some guarantee that all I need is the most recent SPPHs?
<StevenK> jtv: You know, I did finish work about three hours ago, right? :-)
<jtv> StevenK: oh yeah?  Then why are you here responding to my question?  :-P
<StevenK> Because I have this OCD about red in my IRC client
<jtv> The correct answer was a timeout.  :)
<StevenK> jtv: If you feed in the two latest SPRs into DSD, it should set the base correctly, but I suspect that isn't your question
<jtv> No, it's more how I find the right SPPHs with the right SPRs in them.
<StevenK> An SPPH has one SPR. You already have the SPPH that caused the change, so grab the latest SPR for that SPN from the other DS
<allenap> wgrant: Do you know if anyone figured out why ec2 runs were not reporting any results?
<jtv> So that's the latest for that SPN, not the latest for that SPN plus base version?  I'm asking since I have to pre-populate based on existing data, so we'll have older SPPHs.
<jtv> I guess a DS picks one base_version for an SPN and sticks with it?
 * jtv should leave StevenK in peace now
<StevenK> Heh
 * StevenK goes to see how dinner is progressing
<jtv> good night!
<wgrant> allenap: I landed that fix this morning.
<wgrant> allenap: jml's excellent new ec2 list had the not so excellent side-effect of breaking that.
<wgrant> But it was an easy fix.
<wgrant> StevenK: Confirmed does not exist :(
<wgrant> jtv: buildbot seems happy now.
<adeuring> good morning
<allenap> wgrant: Fantastic! :)
<gmb> henninge: I have a 69-line JS branch if you've got time to review it.
<henninge> gmb: throw it at me!
 * gmb lobs https://code.launchpad.net/~gmb/launchpad/fix-subscribe-form-preloading-bug-722450/+merge/51510 at henninge 
 * henninge sometimes is a good catcher ... like just now.
<gmb> :)
<henninge> gmb: Sorry for being a bit slow. I got distracted but I also don't have much practice in JS reviews.
<henninge> gmb: I don't see the last sentence of your cover letter reflected in the code.
<gmb> henninge: That's okay. The diff looks a little funky anyway (I guess it's because of where I put the new function). If you'd rather wait for someone who's got JS experience to take a look, that's fine; it's not urgent.
<henninge> no, I want the JS experience ... ;)
<henninge> gmb: my other question is: why can you do away with the spinner?
<gmb> henninge: Just a sec; someone at the door
<henninge> as long as they are not trying to sell windows ...
<LPCIBot> Project devel build #485: STILL FAILING in 5 hr 43 min: https://hudson.wedontsleep.org/job/devel/485/
<henninge> gmb: I am off for lunch, we can continue later.
<gmb> henninge: Okay, sure.
 * gmb takes the opportunity to lunch, too.
<jelmer> jml: are there any plans to remove the "Recipe builds are experimental" box on recipe pages?
<jml> jelmer: there'd better be!
<maxb> Are there plans to expose performDailyBuild() in the web UI ?
<jelmer> maxb: There is a button for that actually
<jelmer> maxb: but it only appears when there is a new version that can be built
<maxb> hm
<maxb> I have not seen this button
<maxb> And I have usefully called it over the API
<gmb> henninge: I'm back from lunch, so just ping me when you want to resume our discussion.
<leonardr> maxb: my squad is working on that now
<maxb> oh *there's* that link
<maxb> Now that's shiny
<jelmer> it is
<tumbleweed> leonardr: care to give me a hand with getting the new launchpadlib behaving over ssh? I proposed for ubuntu-dev-tools, but it seems a bit of a nasty regression that I can't use it over ssh https://code.launchpad.net/~stefanor/ubuntu-dev-tools/launchpadlib-1.9/+merge/51475
<leonardr> tumbleweed: try putting "export `gnome-keyring-daemon`" into your bashrc. that will make the gnome keyring run in your x session when you ssh in
<tumbleweed> leonardr: I tried doing that (not in bashrc, and didn't seem to do anything
<tumbleweed> do I really want one gnome-keyring-daemon per shell?
<leonardr> tumbleweed: there's another workaround, let me try to find it
<leonardr> i don't actually know which workaround is better
<tumbleweed> me knows very little about gnome-keyring, which probably doesn't help
<leonardr> i now know about gnome-keyring but not about x
<leonardr> tumbleweed, what happens when you try this with gnome-keyring-daemon exported?
<tumbleweed> leonardr: gnomekeyring.IOError
<jelmer> we've seen that with bzr-gtk as well
<tumbleweed> jelmer: yeah I have too
<jelmer> we're working around it by ignoring gnomekeyring if it raises IOError
<tumbleweed> well, we have to get the launchpad credentials from somewhere
<tumbleweed> so if we ignore that error, we need a fallback
<tumbleweed> is there a standard location for an on-disk credential store, or must that be provided to login_with?
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jelmer> 'morning cr3
<cr3> jelmer: hola senor!
<leonardr> tumbleweed: python-keyring handles the on-disk credential store
<leonardr> so, yes, there is a standard location
<leonardr> but i would much rather figure out how to make gnome-keyring work for everyone
<leonardr> tumbleweed, try this out:
<leonardr> export `dbus-launch`
<leonardr> it does look like you're supposed to have one daemon per session. maybe that's something the gnome-keyring people could optimize
<tumbleweed> it would be nice if launchpadlib printed the auth URL, rather than diving straight into elinks
<tumbleweed> i.e. make elinks optional. I've always quit it and pasted theURL into my desktop's open browser
<tumbleweed> anyway, I can't seem to login via elinks
<tumbleweed> leonardr: I grabbed the token-auth URL from the process command line, authed it, quit elinks, and now I get "keyring.backend.PasswordSetError
<leonardr> tumbleweed: launchpadlib does both. it prints the auth url and opens a web browser. i guess you don't see it because elinks takes over the console?
<tumbleweed> yeah
<tumbleweed> IIRC it used to wait for one to press enter after quitting elinks?
<leonardr> tumbleweed: right. i took that out because it was unnecessary. launchpadlib can see whether you've authorized the token on its own
<tumbleweed> leonardr: yeah, but I used to be able to quit elinks, do the auth, then press enter
<tumbleweed> unless it's polling launchpad, in which case ignore me :)
<leonardr> tumbleweed: it should work the same now, except without pressing enter
<leonardr> yeah, it's polling launchpad
<leonardr> tumbleweed: this PasswordSetError you get
<leonardr> is that when you export dbus-launch?
<tumbleweed> yes
<tumbleweed> I have a desktop session open on the machine I'm ssh-ing into
<leonardr> ok, is it a gnome desktop?
<tumbleweed> yeah
<leonardr> can you paste me the traceback of the error?
<tumbleweed> leonardr: http://paste.ubuntu.com/573478/
<leonardr> tx
<leonardr> argh, it's re-raised
<leonardr> let me try dbus-launch and see what happens for me
<leonardr> tumbleweed: for some random reason i seem to be using an unencrypted file instead of a keyring... solving that problem first and will then see how my setup compares to yours
<tumbleweed> heh, cool
<tumbleweed> I only have one machine running natty (my laptop), so I haven't played around too much
<leonardr> tumbleweed: i would like to see the original context of that exception. if you wouldn't mind going into credentials.py and commenting out the try/except clause from CredentialStore.save()
<leonardr> we could make some progress
<tumbleweed> sure
<tumbleweed> leonardr: http://paste.ubuntu.com/573490/
<tumbleweed> sorry, haven't dived into this wholehartedly, busy writing an ubuntu-dev-week talk for this evening
<leonardr> np
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: benji | https://code.launchpad.net/launchpad-project/+activereviews
<leonardr> benji, welcome to ocr
<benji> leonardr: I'm a real boy now.
<leonardr> did someone recently change the oauth 'invalid access token' message?
<leonardr> i think it changed recently and broke launchpadlib
<cr3> is there any reason why bugs can't be searched by date range?
<henninge> gmb: let's finish this review ;-)
<gmb> henninge: Ah, I asked benji to take over since you'd gone off-call.
<gmb> Sorry, didn't realise you were still around for some reason.
<henninge> gmb: that's ok, I wasn't feeling too great earlier and took a longer break.
<gmb> henninge: Okay. Hope you feel better.
<henninge> that's why I took myself off so I won't get more requests.
<henninge> yes, pain killers kicked in ...
<henninge> thanks
<leonardr> salgado: i'm still asking you damn questions about oauth
<leonardr> is there a mechanism (manual or otherwise) for clearing expired access tokens out of the database?
<salgado> leonardr, heh, that's fine
<leonardr> because it looks like they never used to be cleared out of the database, but now they are
<leonardr> iow i encountered a problem that i would have encountered a long time ago if they were cleared from the database regularly
<salgado> leonardr, I don't remember seeing anything that would delete access tokens
<leonardr> ah, i know what it is. this is on staging
<salgado> but I think we've always had working code to reject them when they were expired
<leonardr> and on staging, the whole database is cleared
<salgado> oh, right
<leonardr> salgado: yeah, i had it working for expired tokens
<leonardr> benji: an easy branch for you: https://code.launchpad.net/~leonardr/launchpadlib/handle-unknown-token/+merge/51563
<benji> leonardr: thanks
<jcsackett> gmb: have a moment?
<gmb> jcsackett: Sure
<jcsackett> gmb: i was told you landed some work at one point to allow admins to mark bug comments as not visible, but i can't find what that might be. could you point me in the right direction?
<gmb> jcsackett: You were misinformed, sadly. I have no recollection of doing that.
<gmb> It *might* have been intellectronica that did it, back in the day, but I honestly have no idea.
<jcsackett> gmb: that's disheartening. :-p
<gmb> Yeah.
<jcsackett> gmb: thanks for clearing that up, though.
 * jcsackett goes back to the hunt.
<LPCIBot> Project devel build #486: STILL FAILING in 5 hr 26 min: https://hudson.wedontsleep.org/job/devel/486/
<lifeless> moin
<sinzui> jcsackett: ping
<jcsackett> hi sinzui.
<sinzui> jcsackett: do you have time to mumble about bug/726628
<sinzui> bug 726628
<_mup_> Bug #726628: flag-expired-memberships can trip an assertion check <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726628 >
<jcsackett> sure.
<manish> how does launchpadlib open the browser for authentication?
<manish> use gnome-open?
<sinzui> I think xdg-open
<manish> thanks, trying
<manish> sinzui: works. Thanks
<jcsackett> sinzui: no need to chat later; found a fix.
<sinzui> \o/
<LPCIBot> Project db-devel build #402: STILL FAILING in 5 hr 54 min: https://hudson.wedontsleep.org/job/db-devel/402/
<jcsackett> clear
<thumper> leonardr: mumble?
<leonardr> thumper: wow, it's 4 already. sure
<thumper> wallyworld, leonardr: https://dev.launchpad.net/MaintenanceRotationSchedule
<lifeless> leonardr: hi
<leonardr> hi lifeless
<lifeless> leonardr: I have a question about how to glue some stuff up in the webservice
<leonardr> sure
<lifeless> if you look at bug.bugtasks
<lifeless> its exported
<lifeless> but the code doesn't know the current user
<lifeless> so we end up querying for all bugtasks
<lifeless> then doing late evaluation access checks on each task
<lifeless> how would I change it to use a method rather than a property, so that the user can be passed in
<lifeless> (but still look like a collection to the web service)
<leonardr> good question. you can use @call_with(user=CURRENT_USER) to pass in the current user
<leonardr> i need to look at the code to see if you can do that and still make it look like a collection
<leonardr> it may not be possible now
<lifeless> ok
<lifeless> well I'm currently hacking on BugTask:+index which shares the issue
<lifeless> I'll add a new method for now
<lifeless> and if we can figure out how to glue it up, that will be cool.
<maxb> Hmm. It would appear that SourceForge runs anonymous mercurial access on port 8000. Should I file a question and assign to the LOSAs asking for accomodations in the firewalling of the importds?
<leonardr> bug 271010
<_mup_> Bug #271010: OOPS accessing a non-existent oauth token page. <api> <lp-foundations> <oauth> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/271010 >
<leonardr> lifeless: you might want to file a bug about it. 'accessor method' would be a useful analogue to 'mutator method'
<lifeless> leonardr: on lazr.restful ?
<leonardr> lifeless: yes
<leonardr> and when i put it like that, i know we don't have this feature
<lifeless> we do it for /branches
<lifeless> kindof
<lifeless> leonardr: https://bugs.launchpad.net/lazr.restful/+bug/726803
<_mup_> Bug #726803: not possible to export a method as a collection with bound parameters <lazr.restful:Triaged> < https://launchpad.net/bugs/726803 >
<lifeless> leonardr: I hope I captured it clearly
<leonardr> lifeless: looks great
<huwshimi> jml: I'm available whenever you are.
<thumper> maxb: yes
 * StevenK glares at buildbot
<leonardr> thumper: the python-keyring bug i filed: https://bitbucket.org/kang/python-keyring-lib/issue/40/failures-happen-at-random-points-in-the
 * benji is startled to see buildbot glare back at StevenK.
<leonardr> benji, you might be interested in that bug as well -^
<StevenK> Haha
<benji> leonardr: hmm, taking a look
<StevenK> Windmill hang in the last devel run, and buildbot has completly ignored that devel has changed for about five hours
<wgrant> Oh come on, that failed like 5 hours ago :/
<wgrant> Nobody forced it :(
<StevenK> Just did.
<benji> leonardr: I concur very much with the intent of your bug report; in my opinion the proximate cause is the unnecessarily eager choosing of the backend
<maxb> thumper: filed, thanks
<lifeless> oh wow
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: benji | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> we've been disclosing the existence of private bugtasks all over the place forever.
<lifeless> hmm
<lifeless> current user isn't clearly visible to traversal code :<
<jcsackett> benji: i know it's getting late, but is there any chance you have time to do a review on 150 line MP?
<benji> jcsackett: if it's https://code.launchpad.net/~jcsackett/launchpad/bug-comment-visibility/+merge/51637, I just did. :)
* benji changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jcsackett> benji: it is indeed. thanks a lot. :-)
<benji> np
<wgrant> :( QA
<jcsackett> sinzui: standup?
<huwshimi> sinzui: What did you want to investigate with light/regular?
<LPCIBot> Project devel build #487: STILL FAILING in 5 hr 32 min: https://hudson.wedontsleep.org/job/devel/487/
 * thumper sighs
<thumper> qa-bad makes me sad
<thumper> how to I say a new landing fixes a qa-bad?
<wgrant> thumper: --rollback=1234
<wgrant> And tag the bug bad-commit-1234
<thumper> I don't want to roll it back
<thumper> I want to fix it
<wgrant> I know.
<wgrant> But you have to --rollback=1234
<thumper> that kinda blows
<wgrant> Yup.
<lifeless> so fix it
<wgrant> What's bad?
<lifeless> qatagger is the project
<thumper> wgrant: the updating of the debversion and base branch
<thumper> the context is updated
<thumper> but there is a JS error on the page
<thumper> I think it came from a weird merge
 * thumper is fixing
<wgrant> Is it trivial to fix?
<thumper> yes
<thumper> delete one line
<wgrant> Great.
<wgrant> Bypass ec2.
<thumper> wgrant: what is the pqm commit message I need for the rollback syntax?
<thumper> wgrant: I don't have a merge proposal for the tweak
<thumper> wgrant: nm
<wgrant> thumper: [rollback=$BADREVNO]
<wgrant> No QA tags other than that.
<sinzui> huwshimi: sabdfl replied the the regular/light discussion on warthogs. Light ishould look good with content. as 12px, I thought it was too light. Maybe my screen is too small to judge.
<wgrant> sinzui: It looks too light next to the monospace font that we use.
<wgrant> But apart from that i think it looks great.
<huwshimi> sinzui: Yes I saw that comment.
<sinzui> wgrant: are you looking at my branch that changes most of the font rules?
<wgrant> sinzui: I was going to look at that at some point.
<wgrant> sinzui: Is that your -1 branch?
<lifeless> *sob*
<lifeless> Distribution._init
<lifeless> *stab stab stab*
<huwshimi> sinzui: Not sure what to think really.
<sinzui> if you think it is too light now, ~sinzui/launchpad/fixed-font-sizes-1 will make you cry with light
<wgrant> lifeless: Yes, I pointed that out to you last week :)
<lifeless> similarly Person._init
<sinzui> I favour landing it. We can change the font easily if users think it is too dark
<wgrant> That's fine.
<wgrant> sinzui: Do you have screenshots?
<huwshimi> sinzui: I'm happy with that
<sinzui> one moment
<wgrant> Hmm, all the heading styles have changed.
<sinzui> wgrant: this is the proposal http://people.canonical.com/~curtis/new-font-rules.png
<huwshimi> sinzui: I suspect there is a reason behind Mark saying that light is best for content. There must be some some data or research behind it and if it goes into the guidlines we can think about it more then.
<wgrant> sinzui: The new monospace style is much nicer.
<wgrant> But some of the headings are too big.
<sinzui> huwshimi: I think the same. Since ubuntu.com is using 13px, I would not use it when judging light font
<wgrant> eg. https://launchpad.dev/bugs/1, the CVE References is huge.
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<sinzui> wgrant: h2's are meant to be huge. I made ours smaller than suggested
<huwshimi> sinzui: It's possible that CVE heading should not be a h2
<sinzui> wgrant: I noted in the css and in discussion that many headings should probably be h3.
<wgrant> I think we probably want to fix that first.
<wgrant> I don't read the MP diff before looking at this sort of change, because it can affect how I explore.
<sinzui> A h2 for simple list seems to be an exageration
<maxb> From https://answers.launchpad.net/launchpad/+question/147254, would anyone fancy telling me about OOPS-1885CMP2 and OOPS-1885CMP4 ?
<wgrant> maxb: Sure.
<huwshimi> sinzui: What this is helping us do is fix hour html as well
<wgrant> maxb: I am trying to locate them, since they are not in the usual places.
<sinzui> huwshimi: I agree. I want to stab the "see more" in the portlet first
<maxb> fun
<lifeless> CMP
<lifeless> code imports?
<wgrant> CreateMergeProposals.
<maxb> Create Merge Proposal (from email)
<lifeless> ah
<wgrant> Possibly loganberry-bzrsyncd...
<lifeless> wgrant: they should be in lp-oops
<wgrant> They're not.
<lifeless> wgrant: are they not? if not please file a bug on oopstools
<wgrant> I plan to, once I work out where they are.
<lifeless> (or better yet fix directly, but theres still a lot of black magic in there)
<lifeless> wgrant: production-configs will tell you
<wgrant> lifeless: It tells me the path.
<wgrant> Not the machine.
<wgrant> It appears to be crowberry, oddly.
<lifeless> it also tells you the service
<lifeless> and cronscripts tells you the machine from the service
<lifeless> StevenK: please tell me you are about to kill Distribution._init as part of the derived distros work.
<wgrant> maxb: They are not on devpad, which may just mean they are not part of the incremental sync. Just triggered a full sync, which will take a few minutes.
<wgrant> sinzui: Interestingly, your branch almost fixes sprite alignment in Firefox 4.
<wgrant> It's also less broken in Chromium now.
<sinzui> wgrant: yes
<lifeless> speaking of sprites
<maxb> ok, thanks
<lifeless> the sprite code generates ValidPersonChecks
<lifeless> wtf
<sinzui> wgrant: I still have not mastered all the variables in positioning
<wgrant> lifeless: Yeah, since it shows a desaturated icon for invalid people.
<wgrant> lifeless: lp-oops can see them, but they don't seem to sync regularly.
<wgrant> maxb: ValueError: tag/value separator not found in line '=20\\n'
<wgrant> In bzrlib.
<wgrant> Let me check the content.
<james_w> codebrowse, are you there?
<wgrant> james_w: Is it down again?
<wgrant> Yes.
<wgrant> Not again...
<james_w> 503 for me
<wgrant> LOSA ping: ^^
<wgrant> It started dying on the 27th.
<wgrant> No obviously relevant changes :/
<lifeless> wgrant: see anything wrong with r -1 of lp:~lifeless/launchpad/bug-724033
<mbarnett> wgrant: seeing codebrowse issues again?
<wgrant> mbarnett: Yes.
<lifeless> mbarnett: hallelujah
<wgrant> Restart one, please.
<wgrant> Not the other.
<mbarnett> interesting, not alerting yet
 * mbarnett logs in 
<lifeless> mbarnett: was anything changed on guava?
<lifeless> last weekend
<wgrant> lifeless: I will tell you once one of the codebrowses is awake again.
<mbarnett> lifeless: not that i am aware of .
<wgrant> mbarnett: It's died three or four times since last week.
<wgrant> And not before that.
<mbarnett> 2011-02-25 17:15:20 status installed libc-bin 2.11.1-0ubuntu7.8
<mbarnett> there were a handful of changes when puppet was installed on the box
<wgrant> No sudo upgrades? :P
<mbarnett> heh
<lifeless> just libc-bin; guess we need to check its changelog
<mbarnett> hmm,
<mbarnett> 2011-02-25 17:15:20 status half-configured libc-bin 2.11.1-0ubuntu7.8
<wgrant> Nothing interesting in 7.8
<wgrant> But who knows what the old version was.
<lifeless> the apt log should say
<lifeless> mbarnett: what was the replaced version ?
<wgrant> The dpkg log should too.
<mbarnett> there are a ton of changes with the puppet install
 * mbarnett stops looking for a moment to restart codebrowse
<wgrant> mbarnett: Leave one process hung, pls.
<wgrant> And a list of packages changed for puppet would be nice.
<mbarnett> codebrowse2 is left.
<wgrant> Thanks.
<wgrant> Still borked.
<wgrant> And of course now it works.
<mbarnett> https://pastebin.canonical.com/44061/
<wgrant> Meh, all ruby.
<mbarnett> yeah
#launchpad-dev 2011-03-01
<mbarnett> rubylicious
<wgrant> mbarnett: Which was the old version of libc-bin?
<wgrant> That's the only vaguely interesting one, and it looks boring :(
<mbarnett> https://pastebin.canonical.com/44062/
<wgrant> lifeless: That's what I would have done.
<wgrant> mbarnett: Hmmm, that's a bit more interesting.
<mbarnett> less rubylicious
<wgrant> Yet still very boring.
<wgrant> mbarnett: Do you know how to apply pygdb to codebrowse?
<mbarnett> i do not
<wgrant> mwhudson: Could you assist with interrogating codebrowse2?
<wgrant> mbarnett: We tried to do it yesterday, but the process accidentally got killed.
<mbarnett> loggerhead history doesn't seem to have naything interesting
<mwhudson> wgrant: yeah, but let me grab some lunch so i can concentrate a bit
<mbarnett> unfortunately, we are about to go losaless.
<wgrant> mbarnett: No spm? :(
<mbarnett> spm: is pretending that flying half way around the world is tiring.
<wgrant> Heh.
<mwhudson> ah
<mbarnett> well, i do actually need to go assault my face with some dinner.  Do you want me to leave codebrowse 2 in its degenerate state and try and hop back later?   Or shall i restart it to avoid the potential (at least a bit) of another outage?
<wgrant> Personally I'd leave it.
<wgrant> They seem to die in tandem now.
<mbarnett> yeah
<mbarnett> okay
<wgrant> lifeless: Your closing-bugs-from-changelogs.txt failure is really confusing.
<wgrant> Ah, I see now.
<lifeless> wgrant: yes, thus asking for help
<wgrant> lifeless: I have a quick fix, but I am looking at ways to actually fix the test properly.
<wgrant> Meh, quick fix it is.
<wgrant> lifeless: http://pastebin.ubuntu.com/573729/
<wgrant> lifeless: It looks for a packageupload with a changesfile in the upload_distroseries.
<lifeless> cool
<wgrant> So if we set the upload_distroseries to something else, the test can continue to illegally create multiple packageuploads.
<lifeless> I'll just wrap this arc and then apply and test
 * wgrant foods.
<maxb> wgrant: Thanks, I believe I have successfully comprehended the user's corrupted merge directive email now
<wgrant> maxb: I can give you more traceback or the email text if you need them.
<maxb> The user pasted the email text in the question, and on close inspection it is indeed malformed, so it's ok
<wgrant> Great, thanks for dealing with that.
<maxb> Its an interesting feature, which I keep thinking I might use
<maxb> Apart from gpg-signing mails without involving thunderbird is something I've not figured out yet
<wgrant> Has anybody had an ec2 run Windmill-fail this week?
<wgrant> maxb: Doesn't bzr do that for you?
<StevenK> maxb: gpg | mail -s ... ?
<StevenK> wgrant: My evil plan to fix my branch from yesterday using + didn't work. :-(
<wgrant> StevenK: :(
<wgrant> My evil plan to disable WindmillLayer is getting stronger.
<StevenK> wgrant: So I suspect the only option is to stop using structured and escape the arguments before interpolating
<wgrant> StevenK: Or interpolate the recipe manually, before you go into structured.
<StevenK> Not a bad idea, just struggling how to implement that.
<StevenK> And trying not to context switch
<StevenK> Switching between branches of devel and db-devel really sucks
<maxb> oh, I meant gpg-signing in a way integrated with bzr send, that doesn't involve saving the merge directive off to a file, manually signing, and sending
<lifeless> maxb: ask in #bzr, or read the docs ;)
<maxb> I read the source, and ended up trying to hack a plugin :-)
<lifeless> maxb: -why-?!
<lifeless> wgrant: was there a sqlite change perhaps?
<wgrant> lifeless: That was my guess.
<wgrant> But it doesn't look like it.
<wgrant> But the list I obtained may well not be exhaustive.
<maxb> Why? Well, I have to confess, because it felt like something that ought to be possible :-)
<lifeless> maxb: I mean, it is possible for most mailers I know of, withiut plugins
<lifeless> maxb: what mailer do you use?
<LPCIBot> Project db-devel build #403: STILL FAILING in 5 hr 35 min: https://hudson.wedontsleep.org/job/db-devel/403/
<lifeless> do we still use personlocation for anything?
<wgrant> No.
<lifeless> sinzui: ^
<wgrant> NFI where those queries come from, either.
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/registry/model/person.py", line 708, in time_zone
<lifeless>     if self.location is None:
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/propertycache.py", line 116, in __get__
<lifeless>     value = self.populate(instance)
<lifeless> time localisation
<wgrant> Oh, that's on PersonLocation? :/
<wgrant> I always assumed that was just the lat/lon.
<lifeless> apparently not
<lifeless> guess how many queries all_distro_archive_ids creates
<wgrant> That could be a bug.
<wgrant> Four? Five?
<wgrant> primary, partner, debug.. maybe only three.
<lifeless> actually two
<wgrant> :(
<lifeless> from this call to getCurrentSourceReleases
<lifeless> hmm
<lifeless> time ot land tihs and get the branch changing that landed
<wgrant> Ahh
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-724033/+merge/51676
<wgrant> Has anybody looked at the Windmill SNAFU in depth?
<wgrant> I am wondering if the xvfb-run issue is in fact a symptom of the killing, not the hang.
<lifeless> xvfb-run issue?
<wgrant> lifeless: See the end of https://lpbuildbot.canonical.com/builders/lucid_lp/builds/669/steps/shell_6/logs/stdio
<wgrant> We get an X error.
<wgrant> I've been presuming that was seen at the start of the hang.
<wgrant> But maybe it's because of the mass murder after the hang is detected.
<lifeless> I don't see an X error
<lifeless> oh
<wgrant> There is the occasional thread leak on ec2.
<lifeless> yes, I think the X process went before the tes
<lifeless> t
<wgrant> And locally it just completely fails.
<thumper> StevenK: mumble?
<StevenK> thumper: I was about to have some lunch, but sure.
<thumper> StevenK: it'll be quickish
<lifeless> wgrant: thanks, thats off to ec2 now
<wgrant> lifeless: Great.
<lifeless> who is meant to be OCR today
<StevenK> WILLIAM!
<lifeless> wgrant: I can haz review please
<wgrant> Oh, so I am.
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: wgrant | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> lifeless: Ah, I see you are realising the precaching resultset this time :P
<lifeless> ftw
<lifeless> on the rusty scale, that interface is problematic
<wgrant> Yes
<thumper> https://code.launchpad.net/~thumper/launchpad/fix-mantis-warnings-bug-218384/+merge/51679 anyone?
<wgrant> https://code.launchpad.net/~wgrant/launchpad/squash-cw-protocolerrors/+merge/51680, also anyone?
<wgrant> thumper: Looking.
<thumper> wgrant: thansk
<thumper> wgrant: I'll look at yours
<wgrant> Thanks.
<thumper> wgrant: do we need the transaction.commit(), or is a store flush enough?
<wgrant> thumper: checkwatches checks that no transaction is active.
<wgrant> So it has to be a commit.
<thumper> ok
<lifeless> grr
<lifeless> 0 tests run in 4:11:34.318468, 0 failures, 0 errors
<lifeless> wgrant: has your fix landed yet ?
<wgrant> lifeless: merge devel.
<thumper> wgrant: shouldn't the ec2 image start with devel?
<thumper> wgrant: or is it client side?
<wgrant> thumper: Yes, but ec2test-remote.py is copied from the local branch.
<thumper> wgrant: ok
<lifeless> wgrant: I had a windmill fail in ec2
<wgrant> lifeless: Could you mentor https://code.launchpad.net/~thumper/launchpad/fix-mantis-warnings-bug-218384/+merge/51679?
<wgrant> lifeless: When?
<lifeless> today
<wgrant> Forward pls.
<wgrant> I want all the data I can get.
<lifeless> sent
<huwshimi> How do I actually get a lazr-js branched landed to trunk (after the mp step)?
<lifeless> grr
<lifeless> wtf:
<lifeless>     print test_tales("branch/fmt:bzr-link", branch=branch)
<lifeless> Differences (ndiff with -expected +actual):
<lifeless>     - <a href=".../~eric/fooix/bar" class="sprite branch">lp://dev/fooix</a>
<lifeless>     ?            ^
<lifeless>     + <a href="http://code.launchpad.dev/~eric/fooix/bar" class="sprite branch">lp://dev/~eric/fooix/bar</a>
<lifeless>     ?          +++++++++++ +++++++++ ^^^                                                 ++++++     ++++
<wgrant> lifeless: Dev focus setting fail?
 * thumper agrees with wgrant
<thumper> huwshimi: commit to trunk
<thumper> huwshimi: I don't think it is PQM controleld
<huwshimi> thumper: Thanks.
<wgrant> thumper: Thanks.
<lifeless> thumper: could you eyeball bug 702524?
<_mup_> Bug #702524: Merged branches keep 'Development' status when their merge proposal is marked as 'Merged' <Launchpad itself:Confirmed> < https://launchpad.net/bugs/702524 >
<thumper> lifeless: yep
<thumper> lifeless: although I think I know the answer
<lifeless> thumper: can you mentor wgrants review of https://code.launchpad.net/~lifeless/launchpad/bug-724033/+merge/51676 ?
<StevenK> wgrant: Can haz review? https://code.launchpad.net/~stevenk/launchpad/db-recipe-description-really-this-time/+merge/51682 ?
<wgrant> lifeless: Can you mentor my review of StevenK's?
<StevenK> wgrant: :-(
<StevenK> wgrant: I know where you live ...
<wgrant> :(
<lifeless> wgrant: yes
<lifeless> just chewing through 3 hours mail ;)
<wgrant> Thanks.
<StevenK> wgrant: What was your plan with adding recipe first?
<thumper> lifeless: yes, I'll look at wgrant's review
<wgrant> StevenK: You could put a %%s in the template, format the other args into it using structured(), then format the escaped recipe in using %.
<wgrant> *or* you could fix structuredstring to work nicely with other structuredstrings.
<wgrant> Does anybody have a valid objection to disabling WindmillLayer until someone works out what is wrong?
 * StevenK tries to remember how to do type checking in Python
<wgrant> I have never managed to get it to run reliably on any local machine, it is entirely broken on Natty, it leaks threads sometimes in random tests, and sometimes hangs entirely.
<wgrant> StevenK: isinstance?
<lifeless> wgrant: Revision 12469 can not be deployed: needstesting
<lifeless> looks like its half-qaed - were you looking at that ?
<wgrant> lifeless: One was a test change.
<wgrant> The other needs a LOSA or somebody to decide that we don't care.
<james_w> is there a library of matchers for use with storm anywhere?
<lifeless> there are some in the lp tree
<lifeless> like StormRecorder + HasQueryCount
<wgrant> lifeless: Do you have any hints for debugging thread garbage?
<james_w> I'm more interesting in query/result set stuff currently
<james_w> https://code.launchpad.net/~james-w/launchpad-work-items-tracker/storm/+merge/51684
<james_w> (includes unit testing of lplib-using code)
<lifeless> I trhink the storm folk have started using matchers
<lifeless> so storm specific stuff perhaps land there?
<james_w> nothing jumps out from the filenames
<thumper> GRRRR
<thumper> stupid dist utils
 * thumper stabs it in the face
<lifeless> oops
<lifeless> oh well, hopefully it won't break the build
<wgrant> lifeless: Hm?
<lifeless> wgrant: I lp-landed rather than ec2 landed
<wgrant> Heh.
<thumper> lifeless: do you know anything about setup.py, tests and the additional_tests method?
<lifeless> thumper: a -little-
<lifeless> thumper: lazr-js ?
<thumper> lazr.enum
<lifeless> ah
<thumper> trying to run: ./setup.py test
<thumper> should that cause the additional_tests method to be checked?
<thumper> because it isn't
<thumper> I have a pdb break point in there
<thumper> and it isn't breaking
<lifeless> http://mail.python.org/pipermail/python-checkins/2006-March/050452.html
<thumper> I'm trying to get it to run the actual tests
<lifeless> so additional_tests is for non unittest tests
<lifeless> what non unittest tests do you have?
<thumper> it is to load doctests
<lifeless> ah, what it means is 'tests that loadTestsFromName does not find'
<lifeless> so, I personally /hate/ setup.py's test support
<thumper> I'm starting to too
<lifeless> I would always just use a test_suite method
<StevenK> lifeless: O hai. Can haz review of wgrant's review of my MP?
<lifeless> I did?
<StevenK> lifeless: I can't see your scribble on https://code.launchpad.net/~stevenk/launchpad/db-recipe-description-really-this-time/+merge/51682 ?
<lifeless> done
<lifeless> must have glitched
<StevenK> lifeless: Thanks, tossing it at ec2
<wgrant> lifeless: Do you know how to get details on threads that are leaked by tests?
<lifeless> wgrant: yes
<wgrant> I've stuck a pdb in there, but I can't get much useful from the Thread.
<lifeless> ok
<lifeless> so depending on how it was constructed
<lifeless> subclasses have a replaced run() (IIRC), typical uses supplies a callable to the thread which its run() will call
<StevenK> wgrant: Can you see any clues on http://pastebin.ubuntu.com/573778/ ?
<lifeless> wgrant: you can also attach gdb
<lifeless> wgrant: and get pretty good info from thread apply all pybt
<wgrant> lifeless: Thanks.
<lifeless> I thought we'd changed thread leaks to be non fail though
<wgrant> They are a non-fail.
<wgrant> But Windmill sometimes leaks them, and Windmill sometimes fails.
<wgrant> I am hoping that this is not a coincidence.
<lifeless> wtf- librarian.txt fail in my /branches optimisation branch
<wgrant> lifeless: I saw one of those a week ago.
<wgrant> What is the error?
<wgrant> A 200 UploadFailed?
<lifeless> yeah
<lifeless> passes locally
<wgrant> Yeah.
<wgrant> And on ec2 >9/10.
<StevenK> wgrant: Can haz clue?
<wgrant> StevenK: recipe_build_details is probably raising an AttributeError or similar.
<StevenK> Hrm. A better traceback would be awesome.
<StevenK> TypeError: not all arguments converted during string formatting
<StevenK> :-(
<StevenK> Ah ha, I get it.
<lifeless> \o/ another one bites the dust
<lifeless> wgrant: so, bug 688130 - what needed losa answering?
<_mup_> Bug #688130: Statistics clean-ups and extra tests <lp-translations> <qa-needstesting> <upstream-translations-sharing> <Launchpad itself:Fix Committed by adeuring> < https://launchpad.net/bugs/688130 >
<wgrant> lifeless: It needs the rosetta pofilestats script to be run.
<lifeless> I think my branch:+index patch is reasonably successful :)
<lifeless> https://code.qastaging.launchpad.net/~software-store-developers/software-center/trunk/+index rendered first hit on qastaging. Which is awesome, if I do say so myself.
<LPCIBot> Project devel build #488: STILL FAILING in 5 hr 29 min: https://hudson.wedontsleep.org/job/devel/488/
<huwshimi> thumper: Actually it appears that we do use PQM for lazr-js
<StevenK> Jenkins, you make me sad.
<StevenK> checkwatches.txt passed, but a Windmill test failed? Baaaaah
<huwshimi> thumper: Does that mean landing with lp-land or something?
<wgrant> huwshimi: pqm-submit
<huwshimi> wgrant: Thanks
<wgrant> bzr pqm-submit -m "[r=foo] bar" --public-location=bzr+ssh://bazaar.launchpad.net/~path/to/branch --submit-branch=bzr+ssh://bazaar.launchpad.net/~path/to/trunk
<StevenK> No fair actually landing with '[r=foo] bar'
<huwshimi> wgrant: Thanks, I was just trying to figure all that out :)
<huwshimi> StevenK: Oh, cmon
<wgrant> lifeless: bug-724033 seems to have an empty commit message.
<StevenK> wgrant: Sucess!
<wgrant> StevenK: Exccellent.
<lifeless> wgrant: see https://pqm.launchpad.net/
<wgrant> lifeless: Ah, great.
<jtv> StevenK, wgrant: DistroSeries.parent_series seems to have shifted in meaning a bitâ¦ should we keep the old "previous series in same distro" meaning or get rid of it?
<wgrant> jtv: It hasn't shifted.
<wgrant> Any derivation stuff that is using it now is not altering its meaning; it's just using it until we introduce something to replace it.
<jtv> Do we expect the replacement to work in the same way?
<StevenK> Awwwww. The user on prod with deleted recipes doesn't exist on qastaging
<wgrant> That is not known.
<jtv> wgrant: okay, do we expect to have DistroSeriesDifferences between e.g. maverick and natty?
<wgrant> jtv: I don't believe so.
<wgrant> Not in the initial implementation, at least.
<StevenK> That isn't the point
<StevenK> Publishing details
<StevenK>     * Built by deleted recipe for Jelmer Vernooij.
<StevenK> \o/
<wgrant> Yay.
 * wgrant is QAing the greatest OOPS fix of all time.
<StevenK> wgrant: Oh, the checkwatches stuff?
<wgrant> StevenK: No, the loggerhead thing.
<wgrant> Unfortunately it is slightly non-trivial, because both qastaging and staging are borked.
<StevenK> Sigh. Can't hear myself think.
<StevenK> Some ungrateful person with a leaf blower is outside. Doesn't he know that the BoM forecast high winds for tonight, so it's POINTLESS
<wgrant> Heh.
<wgrant> Hmm, this is slightly upsetting.
<jtv> wgrant: I fervently hope that remark is not connected to SteveK's.
<wgrant> No, codebrowse.
<jtv> Ah well, talk of upsetting.
<jtv> On the bright side, "parallel a-f" has landed on qastaging.
<wgrant> And mawson.
<wgrant> And it works.
<jtv> You tested it?
<wgrant> Yes.
<jtv> Great!  Thanks.  I'll mark it qa-ok then.
<wgrant> I already did.
<wgrant> Didn't I?
<jtv> Oh, looks like you did.  :)  Pages don't refresh very well here.
<StevenK> GAH! He's back!
<StevenK> I have somewhere that guy can store the leaf blower when he is finished using it ...
<wgrant> D:
<wgrant> Codebrowse, you suck.
<wgrant> lifeless: So, you know how that loggerhead fix was meant to stop loggerhead from OOPSing when haproxy checked it?
<jtv> StevenK: do SPRs ever get deleted?  I'm asking because SPPH.supersededby isn't indexed.
<jtv> StevenK: do SPRs ever get deleted?  I'm asking because SPPH.supersededby isn't indexed.
<wgrant> jtv: Not at the moment.
<wgrant> :(
<wgrant> No IS for a week and a half makes me very sad.
<lifeless> wgrant: yes?
<wgrant> lifeless: Well, my testing showed that it in fact introduced a new OOPS. It does, but in a much rarer case than the one it fixes.
<wgrant> Bug #726985
<_mup_> Bug #726985: codebrowse OOPSes with GeneratorExit when connection closed early <Launchpad itself:Triaged> < https://launchpad.net/bugs/726985 >
<wgrant> I think we should still deploy.
<wgrant> Also, feel like waking elmo?
<lifeless> wgrant: is that if you abort from a client ?
<wgrant> guava has had a heart attack again.
<lifeless> *sigh*
<lifeless> is s gsa around ?
<wgrant> I pinged pjdc/bradm a few minutes back.
<wgrant> No response.
<lifeless> give them 10
<lifeless> then dig up the number and call
<lifeless> I'm mid-dinner, sorry
<wgrant> k
<wgrant> lifeless: I believe the client has to be in the DC, bypassing haproxy.
<elmo> bouncing
<wgrant> elmo: tiaz might already be.
<wgrant> (see #is)
<jtv> StevenK: do you have a clear idea of what DSDJ will look like, database-wise?
<StevenK> jtv: One row
<jtv> StevenK: one row per object?  Ground-breaking.
<jtv> Maybe I'm just not aware of the actual design questions you're facing.  :)
<StevenK> jtv: A DSD already has an interface and a schema ...
<StevenK> Oh, the job itself. Er. Not sure. :-)
<jtv> Yes, the part I'm quite interested in right now is whether, for the initial population of DSD, I can create DSDJ rows directly in SQL.
<jtv> As tends to happen, a join structure has formed itself in my mind's eye quite early on and it takes ages to figure out any reasons why it might be wrong.  :)/
<jtv> I was also wondering about supersededbyâ¦  When is a SPPH "superseded" exactly?
<jtv> I imagine I can ignore ones that have that column set, but might that also be a sufficient condition?
<jtv> In other words, if I selected from SPPH on sourcepackagename and distroseries where superseded is null, would that give me (more or less) just the SPPHs I wanted or would there be a lot of dead wood in there?
<wgrant> Selecting on supersededby is not going to give you anything useful.
<jtv> Ah
<jtv> Conversely, will I be able to ignore ones where supersededby is not null?
<wgrant> You should filter on status, if anything.
<jtv> (And also, why is supersededby not indexed?  Do SPRs never get deleted?)
<wgrant> SPRs do not get deleted at the moment.
<wgrant> What are you trying to do, exactly?
<wgrant> And why?
<jtv> Trying to figure out how to get the "latest" SPPH for a given source package.
<lifeless> jtv: due to the bug about not having a sourcepackage object in the db
<wgrant> Source package, source package name, or distribution source package?
<lifeless> jtv: this is actually hugely expensive
<lifeless> getCurrentSourceReleases does it
<jtv> wgrant: source package
<wgrant> jtv: Are you sure?
<jtv> lifeless: ah, thanks
<lifeless> well
<lifeless> it gets what most people will think of as a source package :)
<lifeless> which is not a SourcePackage or  DistroSeriesSourcePackage
<wgrant> If you have a SourcePackage, you should be able to call .currentrelease.
<jtv> wgrant: I was until you asked that.  :)  A given sourcepackagename for a given distroseriesâ¦  I *think* that's what I want!
<wgrant> Right, that's an actual SourcePackage.
<lifeless> except its versioness
<lifeless> which is mondo confusing modelling
<jtv> lifeless: I imagine what most people think of as a source package is what we call a distributionsourcepackageâ¦ I've never heard of DistroSeriesSourcePackage.
<wgrant> Most people would think of an SPR as a source package.
<lifeless> jtv: no, most people think of source packages as 'source package release'
<jtv> Ah
<wgrant> Indeed.
<lifeless> jtv: there is a 'most recent source package release'
<lifeless> I'd like to really overhaul the soyuz model
<lifeless> but not while we're in the hole
<wgrant> lifeless: How?
<jtv> So should I be looking for the most recent SPR instead of the most recent SPPH when looking for distroseries differences?
<jtv> lifeless: for someone who's postponing stuff until we're not in the hole, you sound suspiciously eager to get back into it.  :)
<jtv> More power to you, butâ¦ sounds like a tall order.
<wgrant> SELECT id FROM spph WHERE spph.distroseries = yourfavouritedistroseries AND spph.pocket = 0 AND status IN (1, 2) ORDER BY datecreated DESC LIMIT 1
<wgrant> Oh, and spph.archive = yourfavouritearchive
<lifeless> wgrant: I don't have a specific schema handy.
<jtv> wgrant: would the pocket and archive matter?
<StevenK> What are your vague thoughts, then?
<lifeless> wgrant: I would start by gathering all the actual use cases the distro *want* - partly what we do today, partly their wishlist.
<lifeless> and design to directly match that as a fast schema, with efficient solutions across the board.
<lifeless> and finally look at how to migrate.
<lifeless> StevenK: I can list some problems
<wgrant> jtv: The archive is essential. The pocket is less essential, but you probably still want to restrict it.
<lifeless> StevenK: we infer data on every operation;  we have 1:M relationships that our business rules want to be 1:1;  our schema supports invalid states
<wgrant> jtv: Since we don't know what it means to have a DSD to a non-Release pocket.
<jtv> wgrant: oh do I get all the PPAs if I don't restrict on archive?  Also, another thing that I think matters is the package.  :)
<wgrant> jtv: Um, yeah, package would be nice too.
<jtv> So that's a join through SPR I guess.
<wgrant> Right.
<lifeless> StevenK: our schema should naturally constraint the states to be valid; data learnt should not be reinvestigated on every operation
<lifeless> StevenK: we should be able to do things like '100 most recent daily builds' in 3-4ms.
<lifeless> StevenK: without 'denormalisation'
<jtv> wgrant: if I neglected to restrict by archive, would that mean that I would get all the PPAs as well?
<wgrant> jtv: And copy archives and partner.
<lifeless> StevenK: basically the schema wasn't built to support fast queries, it was built as an object model for the archive/build space: which as a way to design a database leads invariably to performance and maintenance problems.
<lifeless> jtv: and other distributions in future ;)
<wgrant> lifeless: No, because we constrain by distroseries already.
<jtv> lifeless: wouldn't the distroseries restriction take care of unwanted distros?
<lifeless> yes, mea culpa
<lifeless> as long as we don't try to share distroseries across distributions ;P
<jtv> Hmm
<jtv> That'd certainly simplify the work I'm doing now.
<lifeless> jtv: unless you also make productseries sharable across related project, -no-.
 * jtv is not currently well enough to grok that sentence
<jtv> anyway
<wgrant> Aha. All four hangs today have been the same branch, but some were partly drowned in other requests.
<lifeless> jtv: I'm saying we don't want the 'projectseries' and 'distroseries' concepts to drift further from each other.
<wgrant> Two different branches were involved yesterday.
<wgrant> But it was the same branch at the same time on each instance.
<jtv> lock?
<wgrant> The requests were 15s apart, though.
<wgrant> And they do not match up in all cases.
<jtv> oh, lifeless, a quick note: the check for read-only mode goes through the FS.  Might it make sense to log that in the external-action timeline?
<lifeless> yes
<jtv> I have this vague worry that we don't know how much it's costing us.
<lifeless> so, two things
<lifeless> a) if its checked outside the request context, we can't put it in the action timeline
<lifeless> but b) if its checkout outside the request context, it may not cost anything (e.g. if the main thread does it)
<lifeless> and c) we should do that in a thread and maintain it as global state anyhow
<jtv> Checks outside the request context are likely to be few (per request).  I'm more worried about calls inside requestsâand therefore frequency more than latency.
<wgrant> Wiiindmiiiill!!!
<jtv> Here, have some cheese, ja?
<lifeless> jtv: anyhow, at the moment, its local FS, if its being checked per request it will be hot and approximately free.
<jtv> lifeless: I also know that the +translate page used to check it oh, at least dozens of times.
<lifeless> meep
<jtv> meep indeed.  I fixed that, and then you did your timelines, and now I think it'd be nice to stop ourselves from making the same mistake again.  :)
<lifeless> wgrant has made some excellent progress towards us being able to fix the plumbing too
<wgrant> ... have I?
<lifeless> wgrant: checkwatches
<lifeless> wgrant: was the major source of omgwtfness
<wgrant> Ah, right.
<wgrant> Just pqm-submitted the next branch of that.
<jtv> Dear infrastructure.  A connection _can_ have a useful lifetime beyond 5 minutes, even if it does go quiet sometimes.
<lifeless> woo 711064 passed ec2
<jtv> lifeless: meep indeed.  I fixed that, and then you did your timelines, and now I think it'd be nice to stop ourselves from making the same mistake again.  :)
<lifeless> I think I have just 2 outstanding branches now
<lifeless> jtv: got that :)
<lifeless> 20:16 < jtv> meep indeed.  I fixed that, and then you did your timelines, and now I think it'd be nice to stop ourselves from making the same mistake again.  :)
<jtv> Just making sure :)
<lifeless> 20:17 < lifeless> wgrant has made some excellent progress towards us being able to fix the plumbing too
<wgrant> lifeless: What sort of increase are we looking for there?
<lifeless> 20:17 -!- jtv [~jtv@27.130.65.129] has quit [Read error: Connection reset by peer]
<wgrant> s/for/at/
<lifeless> wgrant: there are 12 queries averaging 1 second each; I've dropped the count to 2/3rds - they were doing a pair for a language, then one for the alt language.
<wgrant> Ah.
<jtv> lifeless: anyway, I'm not sure we ever consciously made a decision of the sort of timescale a switch to read-only mode should operate at: instantaneous, when-the-GIL-is-passed, per-requestâ¦
<lifeless> wgrant: so 8 vs 12, so 4 seconds perhaps ?
<stub> jtv: Sounds like you are back in TH. TOT doesn't want to hold an SSL connection open for more than a few minutes for me, even if it is busy.
<lifeless> jtv: end of request is implied by our contract
<jtv> stub: I am, though it's a non-governmental ISP
<jtv> lifeless: makes senseâ¦ so then we should keep RO-mode in the request.  We've had a small number of oopses from the switchover happening in mid-requestâthough maybe that was just an inappropriate check.
<lifeless> jtv: I'd be happy to have it in a background thread that polls every (say) 1000ms; and new requests read the current state from a global
<stub> IIRC, RO mode shouldn't kick in until the next request - there is a thread local that is checked at the start and end of the request.
<lifeless> jtv: that would - prepare for us signalling via e.g. rabbit instantly, or using a single centralised ini file. Would eliminate per-requeset overhead.
<lifeless> interesting
<lifeless> nearly no increase in OOPS with memcache turned off for a day on bugtask+index
<lifeless> \o/
<lifeless> in fact, a *decrease*
<lifeless> 132 /  272  BugTask:+index
<lifeless> 187 /  476  BugTask:+index
<wgrant> Huh.
<wgrant> How expensive were the sets?
<lifeless> I saw individual misses at 24ms
<wgrant> I'm more interested in the sets.
<lifeless> sure
<lifeless> go look in the 27th oops report
<lifeless> I don't remember the figures offhand
<jtv> wgrant: when I look for distroseries differences, I guess the archive restriction should be that the purpose be "primary."  Debug archives don't sound like things that should be shared with derived distrosâ¦ is that correct?
<wgrant> jtv: Debug archives only have binaries, anyway.
<wgrant> So yes, primary archive only.
<wgrant> If you go near partner, I will delete you.
<jtv> Just such a shame to have to join through Archive just to get the purpose.
<jtv> wgrant: don't worry, I don't believe in partner swapping.
<wgrant> jtv: You could just distribution.main_archive
<jtv> Or as I like to say: "you should have kept the receipt then"
<wgrant> May be cheaper than joining.
<lifeless> wgrant: can a distribution have a currentseries of None ?
<jtv> wgrant: ahhh thanks, that's going to be cheaper yes.
<wgrant> lifeless: Yes :D
<lifeless> wgrant: ><
<lifeless> wgrant: I'm looking at the bugtask-target-link-titles_txt failure I forwarded you
<wgrant> lifeless: If there is no current series, there is no current version.
<jtv> wgrant: Distribution.main_archive happens to be defined as a query on Archive, you unhelpful twit
<LPCIBot> Project db-devel build #404: STILL FAILING in 5 hr 35 min: https://hudson.wedontsleep.org/job/db-devel/404/
<wgrant> jtv: It is, yes.
<wgrant> jtv: May still be faster.
<lifeless> wgrant: obvious change made then
<wgrant> lifeless: The only case where there isn't a currentseries is when there are no series at all.
<jtv> wgrant: could beâmore of a "join or fetch separately" tradeoff
<wgrant> Right.
<lifeless> win
<lifeless>         return hash(self.distroseries.id) ^ hash(self.sourcepackagename.id)
<lifeless>     AttributeError: 'NoneType' object has no attribute 'id'
<jtv> Aigh!  SourcePackage.currentrelease is not cached, and SourcePackage.format evaluates it twice.
<wgrant> lifeless: WTF?
<wgrant> jtv: What is SourcePackage.format?
<jtv> "it's in the model class"
<wgrant> Huh.
<wgrant> I didn't know that existed.
<lifeless> wgrant: dict.get(SourcePackage(packagename, None))
<wgrant> jtv: Why are you using that?
<jtv> wgrant: I'm not, just noticed it by accident
<lifeless> wgrant: SourcePackage hashes them together, and hte None comes from the same currentseries
<wgrant> Ah.
<wgrant> lifeless: :(
<wgrant> jtv: SourcePackageRelease.format was introduced a long long time ago and never used.
<lifeless>     - evolution (Published Distro): 2.0
<lifeless>     ?                               ^^^
<lifeless>     + evolution (Published Distro): No releases
<lifeless>     ?                               ^^^^^^^^^^^
<lifeless> presumably, again, because the current series *isn't actually published in*
<jtv> wgrant: I suppose I should be relieved.  :)
<wgrant> jtv: No, you should delete it :P
<jtv> wgrant: with all sorts of sadistic pleasure
<lifeless> yay dead waiting on librarian ><
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/canonical/librarian/client.py", line 188, in addFile
<lifeless>     response = self.state.f.readline().strip()
<lifeless>   File "/usr/lib/python2.6/socket.py", line 397, in readline
<lifeless>     data = recv(1)
<lifeless> KeyboardInterrupt
<lifeless> wgrant: of course, our friend Archive:+packages is back
<lifeless> ok this is annoying me
<wgrant> lifeless: What is?
<lifeless> repeated librarian hang in addFile
<wgrant> Hmm.
<wgrant> What does your librarian say?
<spiv> wgrant: "sssh!"
<wgrant> Damn.
<lifeless> if we get a deploy done tomorrow morning
<lifeless> we can lower the timeout again
<lifeless> I think
<lifeless> omg
<lifeless> cve:+index you suck
<lifeless> 100 lookups for distribution
<lifeless> 'FROM Distribution
<lifeless> WHERE id IN ($INT)
<lifeless> '
<wgrant> Hah.
<lifeless> how to bypass storms cache layer in one easy hit
<lifeless> wgrant: bug 727023
<_mup_> Bug #727023: Cve:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727023 >
<lifeless> wgrant: data for you on librarian shutdown issues
<lifeless> 2011-03-01 13:59:53+0530 [-] Failed to unlink PID file
<lifeless>         Traceback (most recent call last):
<lifeless>         Failure: exceptions.OSError: [Errno 2] No such file or directory: '/tmp/tmplhlibr/librarian.pid'
<lifeless> 2011-03-01 13:59:53+0530 [-] Server Shut Down.
<lifeless> wgrant: it shutdown ok, but its a little concerning
<wgrant> Hmm.
<wgrant> Never seen it hang before :/
<rvb> Hi everyone, first day on the job for me today! I hope you lp guys remember the "guy from the future" from Dallas ;-).
<rvb> lifeless: Hi
<rvb> wgrant: Hi
<wgrant> Welcome rvb!
<rvb> wgrant: thx, I'm quite happy to be here and ready for action! I'm waiting for bigjools to come online but I still wanted to say hello.
<lifeless> rvb: hi, welcome
<wgrant> Julian should be around in 20 minutes or so.
<rvb> lifeless: thx
<StevenK> rvb: Welcome!
<rvb> StevenK: Hi
<lifeless> wgrant: can you eyeball lib/lp/bugs/browser/tests/bugtask-target-link-titles.txt and tell me why the evo publication in 'Published Distro' is not in the currentseries ?
<wgrant> lifeless: Is "because someone thought it was a good idea in 2005, and it was so" a valid answer?
<lifeless> wgrant: I think its oversight because getCurrentSourceReleases was bong
<lifeless> wgrant: The answer I need is 'change X to make it be in the current series.
<lifeless> wgrant: which may simply be 'do X to *set* a current series'.
<wgrant> lifeless: Have you tried overriding the series that getPubSource uses?
<lifeless> didn't know you could
<wgrant> It wouldn't be very useful if you couldn't.
<wgrant> ... so I guess that's a reasonable assumption.
<lifeless> wgrant: I make no assumptions about usability and soyuz guts.
<adeuring> good morning
<wgrant> Morning adeuring.
<adeuring> hi wgrant!
<wgrant> adeuring: Bug 688130 needs QA, and I'm not really clear on how to do it.
<_mup_> Bug #688130: Statistics clean-ups and extra tests <lp-translations> <qa-needstesting> <upstream-translations-sharing> <Launchpad itself:Fix Committed by adeuring> < https://launchpad.net/bugs/688130 >
<adeuring> wgrant: I'll look at it
<wgrant> Thanks.
<lifeless> wgrant: this is a disaster
<wgrant> lifeless: What is?
<wgrant> lifeless: That test?
<lifeless> yes
<wgrant> Heh.
<lifeless> wgrant: tip of lp:~lifeless/launchpad/bug-279513 if you're feeling interested
<lifeless> it has a currentseries version of 1.0, not published
<lifeless> oh... frell
<lifeless> the current series changes when they make a new series midflight
<lifeless> so it invalidates out the current series
<wgrant> Hah, handy.
<rvb> bigjools: Hi
<bigjools> rvb: Morning!
<lifeless> wgrant: whats the official way to freeze a distro series ?
<lifeless> wgrant: just assign the status ?
<wgrant> lifeless: Yes.
 * bigjools brb
<bigjools> rvb: how are you doing?
<rvb> bigjools: all right!
<rvb> bigjools: ready for action!
<bigjools> rvb: I tried to send you a private message, did you get it?
<rvb> bigjools: no
<mrevell> Hi
<rvb> bigjools: all right, I got it now
<StevenK> gmb: O hai! Are you up for a review?
<gmb> StevenK: Not just yet. If it's in the queue I'll take a look in the next 45 minutes or so.
<StevenK> gmb: At some point today is perfectly fine.
<LPCIBot> Project devel build #489: STILL FAILING in 5 hr 25 min: https://hudson.wedontsleep.org/job/devel/489/
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: wgrant, gmb | https://code.launchpad.net/launchpad-project/+activereviews
* wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: gmb | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> gmb: Hi.
<gmb> wgrant: (Mor|Eve)ning
<wgrant> gmb: Could you please review https://code.launchpad.net/~wgrant/launchpad/pygments-1.4/+merge/51725?
<wgrant> Fixes the codebrowse hangs that have plagued us since the weekend.
<gmb> Sure
<gmb> Ooh, complex.
<gmb> wgrant: Done.
<wgrant> gmb: Thanks.
<wgrant> Very complex, yes.
<bigjools> wgrant: I love the way the decprecation warnings were overridden
<wgrant> bigjools: That's the way to do it.
<wgrant> adeuring: Any progress on that QA? I hope to be able to request a deploy in the morning to unbreak codebrowse.
<wgrant> And I really don't know how to do it myself :/
<adeuring> wgrant: sorry, got distracted, but I'm looking now,. I still  need  some time to figure out how to test the branch properly
<wgrant> adeuring: Thanks.
<bigjools> wgrant: I meant that it's ignoring a valid problem
<wgrant> bigjools: Oh, right.
<StevenK> gmb: Thank you for the review
<gmb> np
<Ursinha> morning launchpadders
<wgrant> Rolling back thumper's branch.
<bigjools> morning Ursinha
<wallyworld_> WTF? ec2 land -> 0 tests run in 4:13:40.345985, 0 failures, 0 errors yet log shows no test failures.
<wgrant> wallyworld_: You've not merged devel lately?
<wgrant> I fixed that yesterday morning.
<wallyworld_> wgrant: i did merge a few hours ago
<wgrant> wallyworld_: Into the branch that you ran ec2 from?
<wgrant> That's the one that matters, not the one that ran it *on*.
<wgrant> Ah, crap, this testfix means my codebrowse branch is going to fail while I'm asleep.
<wgrant> I might just lp-land it, then.
<wallyworld_> wgrant: i'm confused. yes i did merge devel into the branch i submited to ec2 land, but ec2 land just take one's branch and merges into the latest trunk anyway, no?
<wgrant> wallyworld_: Sort of.
<wgrant> wallyworld_: The bug was in a file that is copied from the local branch.
<jml> wallyworld_: may I see the log?
<jml> wallyworld_: actually, nm.
<wallyworld_> ah, didn't realise that was the case
<wallyworld_> so the last 2 lines of the log should say "All tests passed" but instead say Tests failed (exit code 1)
<wallyworld_> make: *** [check] Error 1
<wgrant> Can you forward it to me or jml?
<wallyworld_> i guess ill try merging devel again now and resubmit
<wgrant> There's probably a failure in there somewhere.
<wgrant> devel is broken at the moment.
<wgrant> Fix in PQM.
<wallyworld_> wgrant: ok. i'm fairly sure there's no failure but i can forward to log
<wgrant> wallyworld_: There is an error.
<wgrant> Search for 'error:' in the output.
<wgrant> I would paste you details, but unity has decided to curse me.
<wallyworld_> wgrant: !@^!^!@%^#! i'll take a look. i would have sworn on my mother's grave i had a proper look
<wgrant> lp.code.browser.tests.test_sourcepackagerecipe.TestSourcePackageRecipeView.test_request_daily_builds_action
<wallyworld_> wgrant: shit. thanks. i stupidly searched for "eror:" thanks for looking
<wgrant> Haha.
 * wallyworld_ hands head in shame
<wallyworld_> hangs!
<wallyworld_> i can't type
<wgrant> At least you aren't Unity.
<wgrant> Can someone watch buildbot tonight and make sure it stays happy?
 * deryck finds it sad wgrant has to ask someone to watch buildbot
<wgrant> deryck: I often get up and find that it has been broken for 6 hours.
<wgrant> (often because of Windmill *cough*)
<deryck> yeah
<wgrant> Do you have any ideas on that?
<wgrant> it leaks a thread or two in most runs, and occasionally fails. Although I got ten runs of just WindmillLayer through ec2 without a failure :(
<deryck> wgrant, yes.  Two things... I *think* this is always related to not using timeouts in asserts.  And doing a layer in a for loop will cause a hang and we could attach a debug then.
<deryck> wgrant, I hope to look at it soon, but gladly welcome anyone else checking into it.
<wgrant> deryck: I was hoping to be able to reproduce it in a minimal ec2 run, since I have never been able to run it reliably locally.
<deryck> by for loop, I mean -- for i in `seq 5`; do test --layer=CodeWindmillLayer; kind of thing.
<deryck> wgrant, you can't run windmill locally at all?
<wgrant> deryck: Almost every test fails.
<deryck> can you increase the timeout constants and get it passing?
<wgrant> They also hang occasionally on open()
<wgrant> Possibly.
<wgrant> I need to investigate.
<wgrant> But production keeps exploding.
<deryck> wgrant, right.  not saying you used look into it.  Just meant that anyone really should look into this.  But I will as soon as I can.
<deryck> s/used/should/
<wgrant> Well, nobody looks into test failures :)
<wgrant> Or production issues.
<wgrant> I am hoping it is as trivial as the librarian one.
<deryck> this is true.
<wgrant> But somehow I doubt it :(
<deryck> sad but true
<lifeless> night all
<wgrant> Night lifeless.
<wgrant> Night all.
<LPCIBot> Project db-devel build #405: STILL FAILING in 5 hr 31 min: https://hudson.wedontsleep.org/job/db-devel/405/
<sinzui> allenap: how goes you merge person job branch. Do you want me to land it it?
<allenap> sinzui: I am trying to land it, but it refuses to go. Database permission errors in ec2 that I cannot replicate locally.
<allenap> sinzui: I am looking at it now in fact.
<sinzui> okay. I too spent a couple hours with that yesterday
<sinzui> allenap: someone will need to add featureflagchangelog. I added that table in db-devel, and that was the one I forgot to give update permission on for merge
<allenap> sinzui: Okay, I'll add it now.
<sinzui> I do not think you can. It does not exist in devel
<sinzui> allenap: or are you planing to land in db-devel instead
<allenap> sinzui: I'll land in devel if I can, but if it rejects me I'll go to db-devel.
<sinzui> oh duh, security.cfg changed, you do want to land in db-devel
<allenap> sinzui: The problem I have had is that I grant only UPDATE on several tables to a new user, person-merge-job. Now this seems to work locally, but in ec2 it fails because an UPDATE with a WHERE clause needs SELECT granted too.
<sinzui> interesting
<sinzui> losa ping
<viciousprimate> sinzui: heya, got dropped by my irc proxy.  i was lead to believe you have an issue worth discussing.
<sinzui> The criminalisation of owning Chihuahuas?
<benji> Indeed, Chihuahuas can not be "owned" they must run free through the streets like rabbid overgown rats.
<viciousprimate> heh
<sinzui> viciousprimate: what would you like to discuss?
<viciousprimate> sinzui: oops, i didn't realize i didn't re-register as myself (lost my irc proxy this morning), one sec.
<sinzui> ooh. just like Scooby Doo when Velma pulls of the villan's mask.
<mbarnett> sinzui: sorry, i was trying to respond to a losa ping there, just failing with my technology this morning.
<mbarnett> i would have gotten away with it, if it wasn't for those vicious primate.
<sinzui> mbarnett: I am testing the the feature flag change log https://staging.launchpad.net/+feature-rules can you change the priority on a few of the items for me to review. maybe change all the 1s to 2s
<sinzui> mbarnett: you will need to enter a comment too
<mbarnett> so, for instance, change the 1 to 2 in "memcache	pageid:BugTask:+index	1" and "visible_render_time	team:launchpad	1	on" ? 	
<sinzui> yep
<mbarnett> kk, i'll change and log.
<mbarnett> sinzui: done
<sinzui> mbarnett: This looks okay to me https://staging.launchpad.net/+feature-changelog
<sinzui> thanks mbarnett
<mbarnett> welcome
<sinzui> jcsackett_: ping
<jcsackett_> hi sinzui.
* leonardr changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: gmb, leonardr | https://code.launchpad.net/launchpad-project/+activereviews
<leonardr> gmb, care to review https://code.launchpad.net/~leonardr/launchpad/bug-271010/+merge/51760?
<LPCIBot> Project devel build #490: STILL FAILING in 5 hr 26 min: https://hudson.wedontsleep.org/job/devel/490/
<sinzui> jcsackett: 1. I realised that giving the flag-expired-memberships a grace period would not work. The message contains a link to a url you cannot access, and it says your membership will expire 2 hours ago.
<jcsackett> sinzui: that does sound like a problem.
<sinzui> jcsackett: since the user has gotten 6 other emails about this, I updated the docstring to explain such cases will be silently dropped
<jcsackett> sinzui: that sounds completely reasonable. if the user has already been warned, there's really no need to email them again in this edge-case.
<gmb> leonardr: Sure
<sinzui> jcsackett: 2 I am looking at bug 106338. It might interest you if you think you can get to it in a few days.
<_mup_> Bug #106338: Editing a bug targeted to a release crashes if you directly edit the untargeted task  <api> <lp-bugs> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/106338 >
<jcsackett> sinzui: oh goody, conjoined bugs.
<jcsackett> i think i can get to that tomorrow or the day after, depending on how current message work goes.
<jcsackett> sinzui: happy to take a look then, if that's a reasonable time table.
<sinzui> jcsackett: okay. I will set it aside. I just looked at the oopses and see that the issue has changed, but I think it still fixable in 400 lines
<jcsackett> sinzui: cool. preimp talk about it tomorrow during our usual 1-1?
<sinzui> sure
<leonardr> danilo, gmb, i will take https://code.launchpad.net/~danilo/launchpad/bug-720826-emails/+merge/51747
<gmb> leonardr: I'm already looking at that
<gmb> Sorry
<gmb> Forgot to claim it.
<leonardr> oh, nm
 * leonardr assigns it back
<gmb> Thanks
<gmb> leonardr: Wow, it OOPSed when I tried to claim it right after you had.  That seems a little over the top. I'll file a bug.
<danilos> gmb, leonardr: yay, fights over my branches again, I love that :)
<danilos> it gets claimed back and forth, even oops fly around
<bigjools> sinzui: question for you
<bigjools> https://launchpad.dev/ubuntutest versus https://launchpad.dev/gentoo - the latter does not have the "Add series" link and I can't work out why
 * sinzui looks
<bigjools> the whole <div> has tal:condition=context/series
<bigjools> which is repeated for the <ul>, heh
<bigjools> so it seems like the addseries link should never be there unless you have existing series
<sinzui> bigjools: IDerivativeDistribution will show it if you are a driver. IDistribution does not.
<sinzui> oh, I think you are correct.
<bigjools> however, ubuntutest does not have any series
<bigjools> but it shows the link :)
<sinzui> bigjools: I think this is vestigial from when we registered usless distros then realised we had to cripple the ui rather than fix the schema
<bigjools> yeah I was guessing it was shitty sample data
<bigjools> I need to fix this for derived distros
<sinzui> Distros do not automatically get series when created I think
<sinzui> bigjools: I think you need to rename the current IDerivativeDistribution to IRemixDistribution. baltix will not be built by soyuz or get translations
<bigjools> yeah, we'll have to look at that
<sinzui> bigjools: The template is just stupid https://launchpad.dev/gentoo/+series will let you add one
<bigjools> sinzui: indeed
<danilos> leonardr, gmb: hi, one small branch up for review: https://code.launchpad.net/~danilo/launchpad/devel-bug-720826-clear-level-on-delete/+merge/51768 (diff will show up in a bit)
<leonardr> i'll take it
<danilos> leonardr, thanks
<stub> I've been getting PQM failures on a particular branch: http://paste.ubuntu.com/573990/
<stub> Anyone seen similar?
<bigjools> never seen that
<bigjools> sinzui: is it my imagination or is there no link presented for /distros/+add anywhere?
<leonardr> danilos, r=me
<danilos> leonardr, thanks
<stub> bigjools: Just me attempting to land on the wrong branch + logs of noise warnings
<bigjools> stub: pebkac is also my middle name
<danilos> leonardr, btw, I don't know of any better way to indicate a base-level default myself; let me check the interface
<danilos> leonardr, the interface and model actually use a different value, the model one is correct (not that I'd know how to extract it nicely from the model or the interface)
<leonardr> danilos, you mean they use different constants?
<danilos> leonardr, yeah, I'll fix that bit
<danilos> and I see I can get the default with IBugSubscriptionFilter.getDescriptionFor('bug_notification_level').default
<danilos> leonardr, do you think I should use that in the code? (I'd leave the test as-is)
<leonardr> yeah, use that in the code
<danilos> leonardr, cool, thanks
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: leonardr | https://code.launchpad.net/launchpad-project/+activereviews
<danilos> gmb, thanks for the review
<gmb> np
<bigjools> sinzui: so the +addseries was appearing when there was an experimental series that's not actually showing in that portlet
<sinzui> bigjools: yes, that is what +series showed. Sorry for not stating that
<bigjools> sinzui: no problem - I intend to try and make it all work much nicer
<sinzui> bigjools: experimental, obsolete are not manages by the core developers, so we hide them
<bigjools> this stuff needs to be easy to use for non-admins when we do the derived distros
<lifeless> moin
<sinzui> Subclasses enums are not equal to the superclass enum
<sinzui> :)
<leonardr> gmb, are you still reviewing my branch or should i try to get someone else to do it?
<LPCIBot> Project db-devel build #406: STILL FAILING in 5 hr 30 min: https://hudson.wedontsleep.org/job/db-devel/406/
<lifeless> bbiab - allergy vaccination time, and the city is still a mess from the quake, so may be longer than desirable :(
<sinzui> can someone who cares about storm and enum give me an opinion about bug 727331. I think I want to implement enum equality rather than create a vocab factory
<_mup_> Bug #727331: Subclassed DBEnum item is not equal to the super class item <infrastruture> <storm> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727331 >
 * sinzui looks for another bug
<jam> lifeless: I poked at https://bugs.launchpad.net/launchpad/+bug/727148
<_mup_> Bug #727148: Bzr timeouts on SSH connections <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727148 >
<gmb> leonardr: Ark, sorry, I thought I'd voted on it.
<jam> And I can confirm routing failure from people.canonical.com
<jam> but success from home
<gmb> Let me just take another look.
<jam> which hints that it is a datacenter firewall/routing issue
<gmb> leonardr: Done. Apologies; I'd obviously looked and gone "yeah, that's fine" but the brain->LP interface was down.
<leonardr> cool
<leonardr> did something happen to lib/canonical/widgets recently? it seems to be gone from my copy of devel, but there's still code that tries to import it
<leonardr> that importing code looks bad to me. has no anyone else noticed this?
<leonardr> looks like the widgets were moved to lp.apps.widgets.itemswidgets and the shipit code wasn't changed
<leonardr> weird... anyway
<james_w> leonardr, run update-sourcecode I believe
 * thumper is here, coffee in hand
<sinzui> leonardr: I moved many of the widgets to lp.app. Other parts of the tree got widgets too. wgrant destroyed the dir after stabbing shipit is a spoon until it yielded.
<leonardr> sinzui: thanks, i've figured it out. i didn't know shipit was in sourcecode
<sinzui> leonardr: it is symlinked. it is not obvious and you may not know it until ec2 sends you hate mail.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #491: FIXED in 5 hr 26 min: https://hudson.wedontsleep.org/job/devel/491/
<thumper> hmm...
<thumper> wgrant: ping
<maxb> Would someone be able to send me OOPS-1886CMP2 including the full traceback and the text of the triggering email? (re https://answers.launchpad.net/launchpad/+question/147254)
<maxb> CMP oopses are apparently stuck locally on crowberry and not being propagated centrally unless something was fixed since yesterday
<sinzui> thumper: may I have your opion of bug 727331.
<_mup_> Bug #727331: Subclassed DBEnum item is not equal to the super class item <infrastruture> <storm> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727331 >
 * thumper looks
<wallyworld> thumper: leonardr: standup?
<thumper> sinzui: hmm... this was an interesting design decision
<thumper> sinzui: if you are around, we could talk after my standup
<thumper> wallyworld: yeah
<sinzui> thumper: wont fix is an legitimate reply. I could make a vocab factory and test in 50 lines, I just do not want to
<lifeless> jam: thanks
 * lifeless is back
<thumper> sinzui: can you mumble?
<wgrant> thumper: Hi.
<thumper> wgrant: hi
<thumper> wgrant: sorry about my branch yesterday, I was sure I did ec2 land
<thumper> bug my bash history tells me different
<wgrant> I noticed it landed very quickly, but decided not to query you.
 * sinzui starts mumble
<wgrant> fuuuu
<wgrant> buildbot has failed.
<maxb> wgrant: Hi, would you be able (now or later) be able to dig out OOPS-1886CMP2 for me, including traceback & email this time? (Same issue as yesterday)
<wgrant> maxb: Looking.
<wgrant> maxb: I've requested a log sync.
<maxb> thanks
<lifeless> bah
<lifeless> https://api.qastaging.launchpad.net/1.0/branches
<lifeless> I may have  broken that api
<lifeless> yeah
<lifeless> ForbiddenAttribute: ('branchID', <lp.code.model.seriessourcepackagebranch.SeriesSourcePackageBranch object at 0x2b58d42a97d0>)
<wgrant> And we were doing so well for the last couple of weeks.
<wgrant> Now we have the chain of pain and suffering again :(
<lifeless> well
<lifeless> its the result of not deploying daily
<lifeless> we can still deploy some stuff
<wgrant> We need a qa-reallydontcare
<wgrant> For things that are hard to test, largely inconsequential, and remain un-QA'd through two European days.
<wgrant> thumper: I think we may have to untestable your Blueprint thing.
<wgrant> Or convince a friendly GSA to setup Blueprint mail.
<thumper> wgrant: yeah
<lifeless> wgrant: qa-untestable it
<lifeless> wgrant: untestable covers a multitude of cases
<wgrant> lifeless: The worst 12469 can do is break pofilestats, and it was already broken until last week.
<wgrant> So yeah, doing so.
<wgrant> maxb: BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:bc9ad731c2af5f4d5b71cbad086b2a0758b8936a',)]
<wgrant> Still want more details?
<wgrant> lifeless: Do you have a fix in progress for your breakage?
<lifeless> wgrant: was just about to start the branch
<lifeless> wgrant: if you want to do it that would be awesome
<wgrant> lifeless: 'fraid I have enough other time-critical stuff to do.
<lifeless> wgrant: its just add the ID (and check for others i may have used that are neither covered by tests nor in the existing interfaces)
<lifeless> wgrant: I'm happy to do it
<wgrant> lifeless: Sadly you are going to miss the next buildbot run by a few minutes.
<wgrant> We might want to hold that or something.
<lifeless> wgrant: I'm assessing https://bugs.launchpad.net/launchpad/+bug/724033 first - checking I haven't broken bugs too
<_mup_> Bug #724033: BugTask:+index timeouts - distributions and milestones being late evaluation loaded - repeatedly - on bug 230350 <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/724033 >
<wgrant> OK, I am disabling Windmill until someone (maybe me) sorts it out.
<wgrant> Unless someone complains quickly.
<lifeless> SQL time: 5017 ms
<lifeless> Non-sql time: 10126 ms
<lifeless> grr
<lifeless> https://bugs.qastaging.launchpad.net/ubuntu/+source/afflib/+bug/230350/+index
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<wgrant> lifeless: Hey, you are making progress on single-threaded appservers.
<wgrant> There is hope.
<lifeless> this is on qastaging
<wgrant> Ah.
<wgrant> Point.
<lifeless> so I think this particular case is genuine
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1886QS95
<lifeless> things like query 4 make me wonder about pathological stuff
<lifeless> (4 by time)
<lifeless> holy cow
<wgrant> Hm?
<lifeless> query 20 in the timeline
<lifeless> we have -lots- of tasks on this bug
<wgrant> Ohh, one of these, right.
<lifeless> after query 36 we waste .7 of a second doing  ???
<wgrant> There are only a few.
<lifeless> bug one down to 352 queries
<wgrant> Unauthenticated?
<lifeless> authenticated
<wgrant> I don't believe you.
<wgrant> Oh.
<lifeless> hit it on qa staging
<wgrant> No memcache, I guess.
<wgrant> That's cheating.
<lifeless> more than that
<lifeless> bug 724033
<_mup_> Bug #724033: BugTask:+index timeouts - distributions and milestones being late evaluation loaded - repeatedly - on bug 230350 <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/724033 >
<wgrant> Huh, confirmed.
<lifeless> I wouldn't be so mean as to get your hopes up needlessly
<wgrant> buildbot does, so why not you too?
<lifeless> because I am a real machine
<wallyworld> thumper: if you have time to look, here's the WrongStoreError i've been getting https://pastebin.canonical.com/44117/
<lifeless> wgrant: whats wrong with 218384
<wgrant> Bug #218384
<_mup_> Bug #218384: OOPS processing Mantis bugwatches <bad-commit-12487> <bugwatch> <lp-bugs> <oops> <qa-bad> <story-reliable-bug-syncing> <Launchpad itself:Fix Committed by thumper> < https://launchpad.net/bugs/218384 >
<wgrant> lifeless: It exploded the test suite.
<lifeless> ass
<wgrant> See what I mean?
<lifeless> thumper: did you ec2 land it, or direct?
<wgrant> Direct.
<lifeless> wgrant: you're rolling it back ?
<wgrant> lifeless: I rolled it back last night.
<lifeless> ok, cool
<wgrant> The rollback is through buildbot.
<wgrant> lifeless: Still 77 Person and 48 VPC queries :(
<wgrant> 37 TP, 21 Milestone.
<lifeless> wgrant: yes, step at a time.
<wgrant> StevenK: Around?
<wgrant> I wonder if we can set up Jenkins to run WindmillLayer in a separate job.
<wgrant> So it doesn't bitrot too much and we are still pushed towards fixing.
<lifeless> wgrant: we can
<lifeless> wgrant: you'll need an entry point to run it is all
<wgrant> Great.
<wgrant> Thanks.
<jcsackett> ah, mumble, how i love your crashing...
<lifeless> ok, sigh
<lifeless> that branches collection is barely tested :(
<lifeless> one liner to fix.
<wgrant> lifeless: No other fallout?
<lifeless> it gets to the last clause of the eager load method
<lifeless> wgrant: check 12505 in lp:~lifeless/launchpad/bug-722956
<lifeless> wgrant: are we in testfix?
<wgrant> lifeless: No.
<maxb> wgrant: thanks for the oops digging. This is sounding like it could be bug 718723, though if so I'm confused why it didn't bite me when I tried it
<_mup_> Bug #718723: fetch from merge directive to stacked branch unable to fill in chk pages <Bazaar:Confirmed> < https://launchpad.net/bugs/718723 >
<wgrant> lifeless: Aha.
<lifeless> wgrant: this is a storm class, the branchID Int column was already defined - I didn't add it in my original patch.
<lifeless> wgrant: and I didn't think to check that it was on the interface, given it already existed.
<lifeless> its in PQM now.
<wgrant> Thanks.
<wgrant> I am giving Windmill one last chance to hang on my laptop before I kill it.
<lifeless> we are in testfix
<wgrant> Then we should have a reliable test suite again.
<wgrant> lifeless: Oh, I thought a new build had started.
<wgrant> Fail.
<lifeless> resubmitted
<wgrant> Thanks.
<wgrant> lifeless: No [rollback=]?
<lifeless> bah
<lifeless> ETOOLATE
<wgrant> OK, we'll just have to work it out manually.
<lifeless> mbarnett: is spm around today?
<mbarnett> lifeless: yup,
<mbarnett> lifeless: or i should say, that is the plan.
<lifeless> wgrant: shall we get 12486 deployed before mbarnett vanishes?
<lifeless> s/ed/ing/
<mbarnett> need a nodowntime fired off?
<lifeless> mbarnett: I'm thinking so
<wgrant> I was considering that.
<lifeless> wgrant: you're still fighting fires?
<wgrant> lifeless: Clubbing Windmill to death.
<lifeless> wgrant: I'll prep it
<wgrant> Thanks.
<lifeless> Ursinha: unless you want to, for more practice? Its a rather bigger one this time
<wgrant> Our biggest in a while :(
<wgrant> Even the next one is not going to be exactly small.
<lifeless> mbarnett: when do you turn into a puff of smoke ?
<mbarnett> lifeless: in a little under an hour most likely.
<lifeless> mbarnett: ok, I'll have this prepped in 15 - need to shove some food in my gob first
<mbarnett> kk
 * mbarnett understands the need for feasting. 
<lifeless> Ursinha: I'm prepping it
<lifeless> mbarnett: its up
<lifeless> 26 bugs
<mbarnett> lifeless: it shall be done.
<lifeless> 2 and a half days
<wallyworld> lifeless: the linked_bugs property on BranchMergeProposalView doesn't filter out private bug. do you agree this is incorrect and i should  use check_permission() to stop private bugs being exposed?
<lifeless> yes and no
<lifeless> better to load them correctly in the first place
<lifeless> that way the collection size will be correct, etc et
<wallyworld> lifeless: the code to load them is in the model - the linked_bugs attribute on a branch, which is defined an an SQLRelatedJoin
<lifeless> yes
<lifeless> its a bug
<lifeless> on several levels :)
<wallyworld> so there's no concept there of what user it trying to do the accessing
<lifeless> right
<lifeless> fix that
<lifeless> give me a sec to do gc on my branches
<lifeless> I may have some half done thing in this area
<wallyworld> lifeless: ok. i'm working on another issue and came across this. if i were to do a quick fix to the linked_bugs property on the view, that works fine for now and there's no issue with collection sizes etc
<wallyworld> that i can see
<lifeless> wallyworld: uhm, I think thats stale
<lifeless> check_permission will trigger late evaluation anyhow
<wallyworld> late evaluation of what? the object having its permission checked?
<lifeless> yes
<lifeless> wallyworld: ok, I've paged this in
<lifeless> I got half way through refactoring
<lifeless> see Branch.getLinkedBugTasks
<lifeless> most of the templates now use linked_bugtasks rather than linked_bugs (because in LP we work with bugtasks not bugs)
<wallyworld> ok. but sometimes we want bugs
<lifeless> wallyworld: no, we don't.
<lifeless> wallyworld: we think we do, but its a mistake.
<wallyworld> like showing the linked bugs for merge proposals? so we want to show linked bug tasks instead?
<lifeless> we already show bug tasks.
<lifeless> wallyworld: bugs have no importance or severity
<wallyworld> ok
<lifeless> wallyworld: see DecoratedBug.bugtask *or* the implementation of Branch.getLinkedBugTasks
<lifeless> wallyworld: either of those show the selection for the bugtask to show
<lifeless> wallyworld: so, yes, the data model links to *bug*, but the UI then *picks a task* to show.
<wallyworld> BranchMergeProposalView has both linked_bugs AND linked_bugstasks properties, so it seems some work is needed there
<lifeless> I added linked_bugtasks to land my performance for BugTask:+index
<wallyworld> ah ok
<lifeless> there was some stuff still referenving linked_bugs on the BMP template so I left it in the short term.
<lifeless> wallyworld: we were doing 1000 queries dealing with linked_bugs
<wallyworld> so getLinkedBugTasks does the right thing with respect to permissions/private etc?
<lifeless> wallyworld: due entirely to late evaluation
<maxb> Oh dear.... if I was to suggest that emailed in bundles to create merge proposals was completely broken, does that sound possible?
<lifeless> wallyworld: yes, what it returns is usable, visible and [mostly] eager loaded already.
<lifeless> maxb: yes
<wallyworld> if so i'll remove linked_bugs from bmp view
<wallyworld> and write some tests - there were none at all for this
<lifeless> there are view tests
<lifeless> but yeah, none dealing with permissions
<lifeless> OTOH
<lifeless> the *only* reason there are permission issues
<wallyworld> not that check that private linked bug[tasks] are not visible
<lifeless> is because code wasn't using the bugs interface for getting bugs.
<wallyworld> because it was navigating the object model starting at a branch
<lifeless> no
<lifeless> because it used direct queries to grab bugs
<lifeless> rather than getUtility(IBugTaskSet).search
<lifeless> which is the -only- defined way to get bug tasks.
<lifeless> everything else either:
<lifeless> a) layers
<lifeless> b) is buggy
<wallyworld> it wasn't using direct queries that i saw - it was calling branch.linked_bugs
<lifeless> linked_bugs = SQLRelatedJoin(..)
<wallyworld> which may be using a direct query under the covers
<lifeless> ^ - thats a direct query
<wallyworld> but conceptually is an object model navigation thing
<lifeless> but its broken
<lifeless> we can't use object model navigation for most of our work
<wallyworld> the direct query is an implementation detail :-)
<lifeless> not IMO
<wallyworld> agreed +100
<lifeless> visibility depends on the current interaction
<lifeless> we have some tensions
<lifeless> we have half a container
<wallyworld> yes, and using object model navigation sucks for that, i agree.
<lifeless> we don't properly support (e.g. fast, efficient, clear) accessing current user from 'anywhere'
<wallyworld> everything should be service based imho
<lifeless> nor do we pass current user around pervasively
<wallyworld> yep
<lifeless> the big thing I want is thinness
<lifeless> I want us to drop about 50% of our classes
<lifeless> 60-70% of our interfaces
<wallyworld> no argument from me for dropping code
<lifeless> probably a 2012 thing
<lifeless> we have many areas that are self-inflicted complexity.
<wallyworld> so for now, i'll use your new getBugTasks stuff and refactor out the use of linked_bugs from bmp view
<lifeless> (e.g. the conjoined master thing: I *still* haven't had an explanation of /why/ that exists)
<lifeless> wallyworld: cool
<wallyworld> that will solve the immediate permisisons issue
<wallyworld> and then i can use that as the basis for the current issue i am working on, wheich is that the branch index page blows up if there's a private bug linked to a mp
<lifeless> wallyworld: cool. Thats the last use of DecoratedBug btw
<lifeless> you can delete it after you're done.
<wallyworld> resulting from the previous stuff i did to show the mp and linked bugs for revisions
<wallyworld> ok will do
<wallyworld> lifeless: thanks for the input
<lifeless> my pleasure
* StevenK changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: leonardr, StevenK | https://code.launchpad.net/launchpad-project/+activereviews
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: leonardr, StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> StevenK: So, how much work is an extra Jenkins job with --layer=WindmillLayer?
<StevenK> For just devel or both {,db-}devel?
<wgrant> Just db-devel.
<wgrant> Little point running it for both.
<wgrant> And db-devel will catch everything.
<StevenK> It's pretty easy
<wgrant> It could even just run in the downtime when one of the other slaves is idle, if you don't have enough.
<StevenK> I don't think that is required
<StevenK> Wow. checkwatches.txt didn't break
<wgrant> I introduced some new checkwatches tests yesterday, which may have caused it to reorder.
<lifeless> wallyworld: this is one of the sample pages that was dying
<lifeless> https://code.qastaging.launchpad.net/~software-store-developers/software-center/trunk/+index
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/die-windmill-die/+merge/51836
<StevenK> wgrant: Perhaps checkwatches.txt needs to die :-)
<wallyworld> lifeless: yeah, i think from memory that one is listed in the bug report
<lifeless> wallyworld: to give you an idea of data sizes with bug-branch links: 375 links
<lifeless> wallyworld: which is also why I filed the bug about merge-proposal bug links - if that branch *ever* merges into another, the branch index will be showing 375 bugs against that revision.
<lifeless> wallyworld: which is, frankly, insane.
<lifeless> (a preexisting data model problem to your work: your using what we have in the model just made it much clearer)
<wallyworld> lifeless: when i worked on the bug asking for this functionality, i never envisiaged a branch would have so many linked bugs
<wallyworld> actually, a mp i mean
<lifeless> right. But the mp 'related bugs' is defined as 'those of the source on in the linked bugs of the target'
<wallyworld> because for each recent revision, we show the mp which resulted in that revision and the linked bugs are against the mp
<lifeless> wallyworld: which is a definitional problem: really it should be 'the open bugs of the source from when the MP was started till it landed'
<lifeless> wallyworld: a merge proposal represents a temporal range; but the bugs we show have no limits. There are other bugs about fallout from this
<wgrant> StevenK: It probably needs to die, yes, but it's not breaking in buildbot so I care slightly less.
<wallyworld> lifeless: the way we work in lp, the life of the source branch is closely tied to the life of the mp
<wallyworld> so its a moot point for us
<wallyworld> but you are saying other projects work differently?
<lifeless> wallyworld: *we* don't work that way.
<lifeless> wallyworld: you may. I don't. Stub doesn't. Others don't.
<wallyworld> ah ok
<lifeless> wallyworld: the lp project definitely doesn't: devel merges to stable multiple times a day.
<wgrant> StevenK: So, I can has review?
<wallyworld> i create a new feature branch for each new bug etc
<lifeless> wallyworld: stable to db-devel. db-devel to db-stable.
<wallyworld> yes, you are right
<wallyworld> my thinking was far too narrow
<lifeless> :)
#launchpad-dev 2011-03-02
<wallyworld> no, not :)
<wallyworld> :(
<lifeless> wallyworld: one common idiom is an 'integration' branch holding things merged from other people that is merged to trunk
<lifeless> another is a 'bits' branch where you do little trivial things
<lifeless> see e.g. stubs pending-db-patches branch
<wallyworld> yes, i totally ignored those
<wallyworld> in my thnking
<lifeless> wallyworld: its ok, really. LP has -massive- use cases
<wallyworld> bad mistake. mad at myself for making it
<lifeless> and our object model being so heavy warps our thinking.
<lifeless> I do it all the time, and have to back up and do another run at the problem.
<wallyworld> that's true actually, about the object model
<wallyworld> especially for newer people like me
<lifeless> yes
<lifeless> its a map, with lots of here-b-dragons, and not the territory.
<lifeless> wallyworld: here's another case - one I'm looking at at the moment
<wallyworld> it's also "decay" in the codebase
<lifeless> wallyworld: https://bugs.qastaging.launchpad.net/ubuntu/+source/afflib/+bug/230350/+index
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<lifeless> that has 160 bug tasks.
<lifeless> *160*
 * wallyworld looks
<lifeless> wallyworld: this is just an example of where users go waaaaay outside the expected bounds us mere developers created.
 * wallyworld tried to look but got a timeout
<lifeless> right
<lifeless> thus I'm looking at it :>
<wallyworld> :-)
 * lifeless goes back to bug-279513
<StevenK> Hm
<StevenK> Jenkins, I'm disappointed.
<wgrant> lifeless: Thanks.
<StevenK> I was expecting a "trigger another build when this one is started."
<lifeless> StevenK: I think the advanced triggers plugin can do that
<StevenK> lifeless: I can't see that on the plugins page
<lifeless> StevenK: #jenkins I think
<lifeless> a quick google doesn't show me anything
<lifeless> it would be easy to do a plugin for it
<lifeless> but I'm reasonably sure there is one
<StevenK> I can't see one either, but I can work around it
<LPCIBot> Project db-devel build #407: STILL FAILING in 5 hr 36 min: https://hudson.wedontsleep.org/job/db-devel/407/
<StevenK> wgrant: So, what do I do to this to make it only run windmill tests?
<wgrant> StevenK: How do you call the test suite at the moment?
<StevenK> make TESTOPTS="--subunit" check | subunit2junitxml -f -o test-results.xml
<wgrant> Add '--layer=WindmillLayer' to TESTOPTS
<StevenK> make TESTOPTS="--subunit --layer=WindmillLayer" check | subunit2junitxml -f -o test-results.xml
<wgrant> Yup.
<StevenK> Added
<wgrant> Thanks.
<StevenK> Scheduled a new build of it
<wgrant> They only take ~35 minutes on ec2.
<wgrant> StevenK: Is there a bug for the checkwatches.txt thing?
<StevenK> Not that I can recall
<wgrant> OK.
<StevenK> It's only 500 lines, it should be shot.
<wgrant> I need to rewrite most of it to get rid of more OOPSes.
<wgrant> So I might.
<wgrant> But checkwatches :(
<StevenK> Haha
<wgrant> Ah, good.
<thumper> oops
<thumper> forgot to add a particular tag to that landing
<wgrant> thumper: Which?
<thumper> wgrant: the mantis reland
<thumper> I did an ec2 land
<wgrant> It seems to have been rejected anyway.
<wgrant> Ahh.
<wgrant> What was missing? It doesn't want to be --rollback.
<wgrant> It looks fine to me.
<thumper> oh, ok
<wgrant> I already rolled it back last night.
<wgrant> So we are clear QA-wise.
<thumper> ok
<thumper> good
 * thumper needs lunch
<wgrant> Finished: SUCCESS
<StevenK> wgrant?
<wgrant> StevenK: windmill worked.
<StevenK> \o/
<wgrant> Thanks!
<StevenK> It took an hour, though. :-(
<wgrant> Yeah.
<StevenK> I should be able to cut about 30 minutes off Jenkins build times by working on EBS
<StevenK> But I have no idea where to start
<lifeless> is bug 181368 fixed?
<wgrant> Maybe.
<wgrant> Certainly not released.
<wgrant> But maybe fixed.
<lifeless> wgrant: I thought the publisher was in nodwontime
<wgrant> lifeless: No, because of poppy.
<wgrant> No separate tree yet.
<thumper> https://code.launchpad.net/~thumper/lazr.enum/case-insensitive-getTokenByTerm/+merge/51843 anyone?
<thumper> why does jenkins now show recent success for db-devel?
<StevenK> It doesn't?
<thumper> last success 6 days ago
<thumper> that isn't recent to me
<StevenK> Oh. s/now/not/ ?
<thumper> oh
<thumper> yeah
 * thumper is on school run
<StevenK> Because checkwatches.txt has been failing pretty much every test run
<StevenK> But not on ec2 or buildbot :-(
<thumper> hmm....
<thumper> why?
<StevenK> Neither wgrant or I have any clue
<wgrant> thumper: I adjusted a couple of things relating to that test.
<wgrant> thumper: So I'm going to look at it this afternoon.
<wgrant> And either fix or delete it.
<thumper> ok
<wallyworld> lifeless: you mentioned before that the lp templates mostly show bug tasks and not bugs..... however i think we want to stick with just displaying bugs for revisions listing on the branch index page
<lifeless> why?
<wallyworld> a bug can have many bug tasks and we really just want the one bug item which the user can click on don't we? there's already a lot of info displayed
<lifeless> wallyworld: bugtask:+index already picks a single task
<lifeless> wallyworld: see the method I pointed you at ;)
<wallyworld> the bug summarises the issue the mp is solving
<wallyworld> if you mean the getLinkedBugTasks(),  i think we need an anlogous getLinkedBugs() method
<wallyworld> we don't always want the entire list of bug tasks, sometimes just the bugs
<wgrant> We have too many h2s.
<wgrant> But it is tempting to keep them because they just look so nice.
<wallyworld> lifeless:  i think the current extended revision listing looks ok as is, but we just want to be able to hide private bugs properly
<lifeless> wallyworld: you're not making any sense.
<lifeless> wallyworld: get linked bug tasks already chooses *precisely one task*
<lifeless> wallyworld: the difference you are arguing for is precisely empty.
<wallyworld> but a bug can have many associated bug tasks, no?
<lifeless> yes, but thats unrelated to the discussion
<lifeless> because getLinkedBugtasks never returns more than one task per found bug
<wallyworld> just saw that now
<lifeless> wallyworld: you *must* have a task to show importance, status, tags, context. Bugs on there own are approximately never interesting.
<wallyworld> why is that?
<wallyworld> why only return just the one task?
<lifeless> wallyworld: because thats what bugtask:+index spent 500 queries figuring out.
<lifeless> wallyworld: I was merely fixing the performance not the definition.
<wallyworld> ok. fair enough. where in lp then can the user see all bug tasks associated with a branch?
<wallyworld> and why is the method called getLinkedBugTasks if it only returns one?
<lifeless> because it returns one task per bug - it returns many tasks
<lifeless> just not many tasks per linked bug
<wallyworld> of course, reading comprehension problem :-(
<lifeless> there is nowhere in LP that you can see all the bug tasks associated with a branch. The model links to bug, but linking to bug isn't useful for the rest of the system.
<lifeless> s/model/schema/
<wallyworld> ok
<lifeless> wallyworld: series branches only show bugtask for bugs which have an open bugtask; regular branches show a bugtask for all their bugs.
<lifeless> wallyworld: whether this is right or wrong is a separate discussion IMO : I want to just highlight a few key things:
<lifeless>  - everything about a bug except visibility, description, comments,attachments - is on bugtask
<lifeless>  - our /convention/ throughout the system is to not show bugs, only bugtasks.
<wallyworld> and also on bug are the associated branches from memory
<lifeless> wallyworld: bugs, questions, specs and other htings are indeed related by bu.
<lifeless> *bug*
<lifeless>  - we show bugtasks because we show status and importance. And we badge things with 'has brnach', 'has spec' etc etc.
<wallyworld> ok. there's currently functionality which constructs mp emails using bugs not bugtasks. i guess this needs changing too?
<lifeless> wallyworld: case by case: we're talking web UI here.
<lifeless> the web UI is different to email because:
<lifeless>  - its live: folk expect data to be presented rather than clicking through
<lifeless>  - email they expect it to be stale - to be given deltas rather than seeing it as it is right now
<wallyworld> i ask because i think (from memory) the email stuff uses branch.linked_bugs and that needs to be removed
<wallyworld> because that also exposes private bugs to email recipients
<wallyworld> so it possibly needs to replaced with something using getLinkedBugTasks
<wgrant> The codebrowse OOPSes have stopped!
<StevenK> Surely not!
<wgrant> Maybe it's just hung again :P
<lifeless> wallyworld: right, I'd use getLinkedBugTasks
<lifeless> wallyworld: you could use task.bug
<lifeless> wallyworld: or you could show bug status as well in the email
<wallyworld> lifeless: yes, i'm doing just that elsewhere atm
<wallyworld> using task.bug i mean
<wallyworld> have to look at performance though. hopefully any alternatives are no worse that using branch.linked_bugs
<wallyworld> wgrant: saw the email about windmill. me sad. a windmill test of mine actually caught a legitimate bug so we do need those tests
<StevenK> wallyworld: Which are being run on Jenkins
<wgrant> wallyworld: Sure, we need to turn them back on at some point.
<wgrant> But for now they are out of the critical path, but Jenkins will tell us if things go bad.
<wallyworld> StevenK: yes but not as part of ec2 land hence we can now land bad code
<wgrant> wallyworld: It was turned off for most of last year anyway.
<wgrant> The number of issues that it catches is miniscule.
<wallyworld> sure, we just got lucky
<wgrant> We would be better served by finding them later and being able to fix them more quickly.
<wgrant> Which is what this gives us.
<wgrant> Because branches will no longer be delayed by 10 hours due to Windmill being shit.
<wallyworld> well if we use more and more XHR then miniscule won't be necessarily true
<wgrant> This is a short-term thing.
<lifeless> huh
<lifeless> qatagger is lying
<wgrant> Eventually we will convince deryck to fix it.
<wallyworld> wgrant: just wanted to hear that it is short term :-)
<wgrant> lifeless: Oh?
<lifeless> look at it now
<lifeless> then file a bug :)
<wgrant> You mean how it shows it as deployable?
<wgrant> It has always done that.
<lifeless> no
<lifeless> I mean the 2nd line
<lifeless> earlier today it was showing 12486
<lifeless> which was correct
 * wallyworld needs food
<lifeless> now its showing 12488
<lifeless> rather than 'nothing can be deployed'
<wgrant> lifeless: That was probably before the rollback was in place.
<lifeless> wgrant: no
<lifeless> wgrant: it wasn't
<lifeless> the /rev/ can be ok without it being show deployable at the top
<wgrant> Ah.
<lifeless> wgrant: this is a previously unobserved corner case
<lifeless> the rule fo rdeployable is 'all the revs before deployable and any rollbacks for broken revs included in the deploy
<lifeless> wgrant: https://bugs.launchpad.net/qa-tagger/+bug/727552
<_mup_> Bug #727552: when first rev is bad, with a later rollback, qatagger incorrectly reports it deployable atthe top <qa-tagger:Triaged> < https://launchpad.net/bugs/727552 >
<wgrant> lifeless: Thanks.
<lifeless> its probably an off by one
 * thumper hangs head
<thumper> WTF...
<LPCIBot> Project devel build #492: FAILURE in 6 hr 8 min: https://hudson.wedontsleep.org/job/devel/492/
<wgrant> WTF @ Hudson
<wgrant> But checkwatches.txt passed again...
<wallyworld> what's the sop for deprecating something in the web services interface?
<lifeless> we don't
<thumper> getUtility(ILaunchBag).user.preferredemail.email inside the email handler
<lifeless> if its in beta, leave it there.
<wallyworld> even if it's a security/privacy flaw?
<lifeless> if its in 1.0 leave it there.
<lifeless> if its devel, just delete it from devel.
<wgrant> wallyworld: What is?
<thumper> there isn't a user in the launchbag for process mail
<lifeless> wallyworld: change its behaviour
<wallyworld> because we export branch.linked_bugs
<wgrant> wallyworld: lazr.restful checks launchpad.View on each object in a collection before it returns it.
<lifeless> wallyworld: just change it to only return visible bug. But I fixed this already I thought.
<thumper> it is handled by lazr.restful
<thumper> we don't need to mess with it
<wallyworld> currently it's still defined as an SQLRelatedJoin
<lifeless> wallyworld: do you *think* its broken, or have you *shown* it
<wallyworld> ok. didn't realise lazr.restful did some filtering
<lifeless> wallyworld: it will be doing sucky late evaluation at the moment
<lifeless> wallyworld: and the collection size will be wrong.
<poolie> lifeless, i guess "7% headroom" comes from 13/14
<lifeless> wallyworld: so its improvable, but it won't actually be disclosing
<lifeless> poolie: 1/14, yes.
<lifeless> poolie: I realised the flaw right after I hit send.
<poolie> but i don't really see how that corresponds to requests/day
<poolie> oh ok
<poolie> nice mail otherwise though
<lifeless> poolie: we'll free up 1 second for every request that is > 13 seconds today.
<lifeless> poolie: which is a smaller number. A thousand or so
 * wallyworld still thinks we should be able to mark stuff as deprecated if required whether it be a security hole or implementation issue
<lifeless> wallyworld: we can in extremis, but it needs to be last-resort
<wallyworld> python, java etc all do it :-)
<lifeless> wallyworld: this is why I don't want *any* new verbs / attributes or types in the 1.0 web service.
<poolie> so you'll save ~701s of cpu time per day?
<poolie> oh, a bit more than that if you also kill those in the >13s range
<poolie> right
<lifeless> wallyworld: we made a promise to not break 1.0 for the folk using it in shipped Ubuntu versions.
<poolie> over 32 cores?
<lifeless> poolie: the 701 is the >14 requests.
<poolie> right, got that
<poolie> so say 1000 requests, and you'll cut off one second each
<lifeless> poolie: currently over 18 python processes.
<wallyworld> so we could mark something as deprecated and then support it until the lts runs out
<lifeless> wallyworld: there is no deprecation marker.
<lifeless> wallyworld: we'd just remove it from devel.
<poolie> so it's less than .001% more headroom
<wallyworld> yes, and i'm suggesting we need such a marker :-)
<lifeless> wallyworld: I think it would be counterproductive.
<poolie> still worth doing
<lifeless> wallyworld: 1.0 users cannot use the replacement.
<lifeless> wallyworld: devel users don't need the old method anymore.
<wallyworld> but in the general case, marking things as deprecated is a fairly common paradigm
<lifeless> sure
<lifeless> this isn't the general case
<lifeless> its a webservice with fairly specific constraints
<lifeless> one of which is that things work smoothly and seamlessly with the 1.0 version of the service.
<lifeless> Separately, I don't want use carrying the debt of things we want to remove *in devel* until we delete every prior edition.
<lifeless> we have enough debt.
<wallyworld> sure, but we should still be able to work towards a 1.1 version, no? ie we need a story around how we evolve the interface
<lifeless> wallyworld: we have one
<lifeless> wallyworld: whats the problem you are trying to solve?
<wallyworld> talking generalities. the problem of being able to marked certain things as "do not use, will disappear in a future version". give people time to migrate away
<lifeless> wallyworld: but it doesn't for us.
<wallyworld> this is more a pub discussion :-)
<lifeless> wallyworld: the timescale impedance is immense.
<lifeless> wallyworld: 5 years vs daily change.
<lifeless> wallyworld: and new things *are not accessible* until you step over the version, at which point - unless we carry that *5 year* burden for *10 years* - the old stuff is flat gone.
<wallyworld> yes, the timeframes are large
<wallyworld> it would be interesting to know which apis were being actively used. i don't suppose we gather those sorts of metrics
<lifeless> we have logs that can tell us reasonably well
<wallyworld> moot point for now. i was more just thinking out loud. i have to stop doing that
<lifeless> wallyworld: its fine - I do that too
<lifeless> wallyworld: but expect to be engaged on such utterances ;)
<wallyworld> :-)
<lifeless> wallyworld: happens to me all the time - surely you've seen me go 'I have no real answer to hand' :)
<wallyworld> yes :-)
<lifeless> \o/ I now can start coding today.
<lifeless> this is why I'm so unproductive, no cycles to write code
<wallyworld> doesn't help you get dragged into various discussions all the time
<jtv> You know your jet lag is bad when you catch yourself thinking that you want database sharting.
<thumper> it seems to me that there should never be a user in the ILaunchBag during process-mail
<lifeless> why not ?
<lifeless> when the sender of the mail is identified we should setup a participation and security policy for them.
<lifeless> I filed a bug about this.
<StevenK> Rargh, and then devel #492 fails with _lock_actions garbage
<wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/unbreak-incoming-mail/+merge/51850
<wgrant> Slow scan and diff generation are slow.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> is lib/lp/soyuz/tests/../stories/soyuz/xx-distributionsourcepackagerelease-pages.txt of any use
<lifeless> or is it redundant with unit tests?
<StevenK> lifeless: That doctest makes me sad
<wgrant> StevenK: It's a doctest.
<StevenK> Haha
<lifeless> can I just delete soyuuz ?
<wgrant> Please.
<wgrant> But delete checkwatches first.
<StevenK> lifeless: Patches welcome
<lifeless> sob
<lifeless> why is default distroseries different to current
<wgrant> Default distroseries?
<lifeless>     - http://launchpad.dev/ubuntutest/+source/testing-dspr/1.0
<lifeless>     + http://launchpad.dev/ubuntutest/hoary-test/+source/testing-dspr/1.0
<lifeless> in that test
<lifeless> after I tell getPubSource where to publish - the ubuntutest currentseries
<wgrant> That's a DSPR wersus a DSSPR...
<wgrant> Wow. versus.
<lifeless>     >>> anon_browser.open(
<lifeless>     ...     'http://launchpad.dev/ubuntutest/+source/testing-dspr')
<lifeless>     >>> anon_browser.getLink('1.0').click()
<lifeless> are the lead in lines
<lifeless>     >>> print anon_browser.url
<wgrant> getLink('1.0') is a little ambiguous.
<lifeless> wgrant: I'd really like to just delete the test :)
<lifeless> wgrant: so
<lifeless> you think there are multiple 1.0 links ?
<wgrant> lifeless: Quite possibly.
<wgrant> Can't check details right now though, sorry.
<lifeless> wgrant: argh
<wgrant> Oh?
<lifeless> wgrant: I *think* I may have found a reason for distro.getCurrentSourceReleases.
<lifeless> wgrant: perhaps not a good reason
<lifeless> wgrant: /ubuntu/+source/packagedeletedinseries-1
<wgrant> lifeless: Latest upload?
<lifeless> yeah
<lifeless> what about dropping 'latest upload' as uninteresting
<wgrant> Component, Maintainer and Architectures are potentially useful.
<wgrant> Urgency and Latest upload are not.
<lifeless> sob
<lifeless> maintainer etc all come from 'current'
<lifeless> so, does this need to handle packages never published in 'currentseries' ?
 * lifeless thinks
<wgrant> It would be nice. But it's not really critical.
<lifeless> I'm going to shelve the branch
<lifeless> its taking up too much time
<wgrant> I say use the current series' publications, otherwise say it's not published.
<lifeless> makes sense to me too
<lifeless> but the perf win for this is no where near big enough to justify the time investment
<lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-279513/+merge/51063 - abandoned
<lifeless> anyone know - does storm sniff tables in SQL(...) ?
<lifeless> whats a good test class for Archive:getPublishedSources?
<lifeless> wgrant: ^
<lifeless> wgrant: also, soyuz permissions - are they lists of attributes, or just-add-to-interface-and-its-done ?
<wgrant> lifeless: You may need to create a new class in test_archive.
<wgrant> lifeless: Most of Archive is tested in archive.txt.
<wgrant> lifeless: As for permissions, it depends on the class.
<wgrant> You'll have to check configure.zcml.
<lifeless> wgrant: thanks
<stub> lifeless: No, SQL is essentially a string. Storm doesn't know how to parse SQL - just generate it.
<lifeless> stub: thanks
<lifeless> I'll use a using clause
<lifeless> whats the easiest way to find what a Archive:EntryResource:getPublishedSources?ws.op=getPublishedSources looks like given that its timing out
<wgrant> You want to know the data from a specific URL that is timing out?
<wgrant> (also, QA party)
<lifeless> wgrant: yes
<lifeless> wgrant: we're qa'd ?
<wgrant> No, it's time for a QA party to QA all the stuff that is now on qas.
<lifeless> wgrant: ah
<lifeless> I'll do mine straight up
<lifeless> !!! frell
<wgrant> ?
<wgrant> Still broken?
<lifeless> TypeError
<lifeless> just getting a traceback
<lifeless> ('Could not adapt', <lp.code.model.seriessourcepackagebranch.SeriesSourcePackageBranch object at 0xaefff50>, <InterfaceClass lp.code.interfaces.linkedbranch.ICanHasLinkedBranch>)
<wgrant> Hah.
<wgrant> I suggest rolling both back.
<wgrant> ... and adding tests.
<lifeless> doit
<wgrant> Me, or you?
<lifeless> you, if you would; its 7pm here and I'm about to get snatched for dinner
<wgrant> k
<lifeless> I landed this by accident if you recall :(
<wgrant> But it passed buildbot
<lifeless> yes
<lifeless> so ec2 would not have helper.
<lifeless> *helped*
<lifeless> I'm just saying its cursed.
<wgrant> Heh.
<wgrant> In PQM.
<lifeless> thanks
<wgrant> OK, how do I delete launchpadlib's credentials?
<wgrant> I am trying to log in as one of my alternate SSO accounts on qas.
<wgrant> But I can't work out how to remove my system authorization.
<wgrant> Hm, maybe my system launchpadlib still uses GNOME Keyring.
<wgrant> Ah, there.
<lifeless> so, we get 800 hits a day on /branches
<lifeless> > 800 - 800 soft timeouts
<lifeless> so probably 0.02% of traffic or so, if we wanted to let it die for a day.
<lifeless> just a thought
<wgrant> It should be through buildbot in 7 hoursish.
<wgrant> And there is no QA required after what is there now.
<wgrant> So, in the likely event that nobody requests one overnight, we can do it in the morning.
<wgrant> wallyworld: I like your fix for the AJAX build request.
<wallyworld> wgrant: great. it seems to work better now
<wgrant> Assuming that this revert works, we are QA'd up.
<LPCIBot> Project windmill build #2: FAILURE in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/2/
<wgrant> My point exactly.
<wgrant> wallyworld: Could you check that test? It passes locally, but this probably indicates that it has a race.
<lifeless> grr
<lifeless> TacException: Error running ['/usr/bin/python2.6', '-Wignore::DeprecationWarning', '/home/robertc/launchpad/lp-branches/working/bin/twistd', '-o', '-y', '/home/robertc/launchpad/lp-branches/working/daemons/librarian.tac', '--pidfile', '/tmp/tmpZckiy9/librarian.pid', '--logfile', '/tmp/tmpZckiy9/librarian.log']: unclean stdout/err: No handlers could be found for logger "QueueProcessorThread"
<wgrant> Oh, it's setting daily_build, not requesting a build.
<wgrant> So not your fault :P
<LPCIBot> Project db-devel build #408: STILL FAILING in 2 min 18 sec: https://hudson.wedontsleep.org/job/db-devel/408/
<wgrant> bzr: ERROR: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/.bzr/repository/packs/09fb66d3dfb9df78e2bb1ca7b6c7aac9.pack is redirected to https://launchpad.net
<wgrant> Not again :/
<lifeless> and poolie is gone
<wallyworld> wgrant:  i don't think that test is one of mine. i would have to look to be sure
<wgrant> lifeless: It doesn't appear to have been during a push.
<lifeless> ok
<lifeless> something solidly else then
<lifeless> the redirect is berk
<wgrant> PQM has been checking my branch for 25 minutes.
<lifeless> speak of the man
<wgrant> Shouldn't push for another couple.
<wgrant> It always used to redirect on 404, if branch-rewrite failed.
<wgrant> Not sure if it still does.
<wgrant> It doesn't.
<wgrant> poolie: https://hudson.wedontsleep.org/job/db-devel/408/
<wgrant> Another pack redirect.
<wgrant> But PQM wasn't pushing at the time.
<lifeless> well
<stub> lifeless: Just a guess, but shouldn't that command line be bin/py or python2.6 -S for all the buildout deps and environment config to be pulled in?
<lifeless> stub: bin/twistd is a similar alias
<poolie> hi wgrant
<lifeless> stub: that wraps /usr/bin/twistd with the local python etc etc
<stub> yup. ic.
<lifeless> the fail is the 'no handlers' message
<stub> lifeless: We silence some loggers in lib/lp_sitecustomize.py if that is appropriate here.
<poolie> wgrant, well, the big thing is to just work out what the redirect message is, and who is sending it
<poolie> but i'll reopen the bug
<lifeless> stub: I haven't looked
<wgrant> poolie: I don't think devpad has codehosting HTTP logs :(
<lifeless> stub: it may be a message we want - won't know until we dig
<stub> wtf do I feel like I have a hangover? Maybe I should have gone out last night and at least had a reason?
<lifeless> wgrant: for crowberry ?
<lifeless> stub: cause you slept in :)
<wgrant> lifeless: Yes.
<stub> lifeless: Getting better. Awake at noon today. Might even make our call tomorrow :-P
<LPCIBot> Project db-devel build #409: STILL FAILING in 0.47 sec: https://hudson.wedontsleep.org/job/db-devel/409/
<poolie> are there logs at all for this?
<wgrant> Er, still?
<wgrant> Ah, looks like the slave needs a good killing.
<wgrant> poolie: I presume there are logs from Apache...
<poolie> i discovered that's not a reliable kind of presumption
<wgrant> I know we don't keep haproxy logs.
<jam> Hi poolie, funny to see you online when I'm waking up :)
<wgrant> Morning jam.
<wgrant> jam: You'll be pleased to know that loggerhead has stopped OOPSing.
<jam> \o/
<jam> wgrant: what was up ?
<wgrant> Thanks for fixing that.
<jam> Or that is the 16k oops / day thing?
<wgrant> jam: The 16000 daily socket errors.
<poolie> :)
<poolie> congratulations on waking up
<jam> there are more fixes in loggerhead trunk, to handle various bits of that correctly all the way through the stack
<poolie> at what seems like a reasonable time in that tz
<poolie> as it happens i'm just answering your mail
<jam> poolie: Still getting my head around the area, and setting up K with the new schools, etc. So I'll probably only be around a half-day today
<wgrant> jam: Although when QAing it I found bug #726985
<_mup_> Bug #726985: codebrowse OOPSes with GeneratorExit when connection closed early <Launchpad itself:Triaged> < https://launchpad.net/bugs/726985 >
<jam> but yeah, jet lag doesn't seem to be too bad for me this time
<jam> my son isn't quite sure about the new TZ, though.
<wgrant> Heh.
<stub> lifeless: So we only gain using pgbouncer for appserver connections. At the moment, at any point in time we have 160 connections to the master db. With pgbouncer, that will usually be sitting at 40-50 (about half the main connections are active at any point in time, only < 10 of the session connections are active at any point in time - they run in autocommit and normally are touched only at the start of an appserver request).
<jam> wgrant: seems pretty easy to fix up. I'm curious who is killing the generator, or how it is falling out of the stack early, etc.
<jam> I think there is already a StopIteration check, probably fine to add GeneratorExit
<wgrant> jam: Yeah, I think so.
<wgrant> It's not OOPSing on prod, at least.
<jam> right
<stub> hmm... can use it too for librarian, codehosting and friends too. I think everything but the cronjobs.
<jam> I think the issue is just that *any* exception is getting logged
<jam> even the ones that don't mean much.
<jam> like GeneratorExit means we exited early
<jam> but is likely to not actually matter
<jam> btw, poolie, welcome back from your bike ride, sounds like you had a good time
<lifeless> stub: we only have 18*4 = 72 appserver threads
<lifeless> stub: so 1/2 active would be a reduction of 32 on that 160 ?
<stub> lpnet1-8,lpnet11-14 each using 4 threads. lpnet15 using 32.
<deryck> Morning, all.
<stub> I'm not even counting edge actually - guess I should be here.
<wgrant> Morning deryck.
<lifeless> stub: 15 is getting reconfigured
<stub> Looks like lpnet15 is a bogus config
<lifeless> stub: yes, along with 9 and 10 that are not currently running
<wgrant> deryck: I have some questions about writing Windmill tests properly, if you have a moment.
<lifeless> we have 13 lpnet servers, 5 edge
<lifeless> for 18*4 once lpnet15 is fixed
<deryck> wgrant, I think the sprint I'm in will start shortly.  But ask away and I'll respond as I can.
<wgrant> deryck: Ah, didn't realise you were at one.
<lifeless> stub: will be going up to 80 with singlre threaded (but we'll actually have two threads per appserver to permit opstats queries
<deryck> wgrant, yes, in South Africa.  Ensemble sprint.
<stub> lifeless: So If the 50% active connections remains, it could be a win. If you are correct about GIL contention and stuff the activity will grow. I need to decide if it is enough of a win anyway to bother.
<deryck> wgrant, this is why I haven't looked at Windmill this week yet. ;)
<lifeless> stub: I'd be inclined to move ahead with it, if only because we can more easily kick everyone off
<lifeless> stub: though you were saying we had cross transaction temp tables?
<wgrant> deryck: Looking at lib/lp/code/windmill/tests/test_recipe_index.py, test_inline_recipe_daily_build just failed on our Windmill-specific Jenkins job.
<wgrant> deryck: It is a race of some sort.
<wgrant> deryck: The lack of a wait between the getUserClient and click looks very suspicious to me.
<wgrant> Should there be one there?
<LPCIBot> Project db-devel build #410: STILL FAILING in 0.49 sec: https://hudson.wedontsleep.org/job/db-devel/410/
<wgrant> I don't really know how much waiting a page load does.
 * deryck looks
<wgrant> StevenK: Could you please kill/clean the db-devel Jenkins slave?
<deryck> wgrant, what line in the test?  I don't see a getUserClient call.  do you mean the getClientFor stuff?
<wgrant> deryck: https://hudson.wedontsleep.org/job/windmill/lastCompletedBuild/testReport/ has the failure.
<stub> lifeless: Yes. A pgbouncer instance runs in a single pooling mode. Session or Transaction interest us. Transaction would save us idle database connections, but doesn't support cross transaction resources. Session doesn't change our utilization, so we would be trading some extra connection control for crappier authentication and extra complexity.
<wgrant> deryck: Er, getClientFor, yes.
<deryck> ok
<StevenK> wgrant: i-3EDE07B4
<StevenK> ?
<wgrant> deryck: It loads the page them immediately tries to click on a JS widget.
<wgrant> deryck: Which sounds prone to races.
<lifeless> stub: So I think what I care about is perf and reliability
<lifeless> stub: if we won't gain measurable perf, and will code reliability, lets defer it.
<wgrant> StevenK: Yes.
<lifeless> stub: we can work on getting rid of those tables and cursors later.
<deryck> wgrant, there's a waits.forPageLoad in the getClientFor method before it returns.  So I think it's fine to click immediately after the method.
<stub> I'm looking at pgpool-ii to see if it gives us better options.
<wgrant> deryck: :(
<lifeless> stub: cool
<lifeless> stub: go with what makes the most sense to you
<wgrant> deryck: So that waits for all the JS to finish?
<stub> I don't think we want to drop temporary tables and cursors entirely - they are the only sane way of doing some bulk operations. The rosetta scripts make use of them extensively.
<deryck> wgrant, sorry, sprint starting.  will look again when I can.
<wgrant> deryck: Thanks, no rush.
<wgrant> It's not blocking anything any more.
<deryck> wgrant, but yes, the js should be loaded and done by that method return.
<lifeless> stub: they aren't crash proof though
<lifeless> stub: I think if we make a crash proof implementation for those scripts, we wouldn't need cursors / cross trnasaction temp tables.
<StevenK> wgrant: Done. Why?
<poolie> lifeless, re bug heat, perhaps it would be useful to try to gauge how many users actually find it useful?
<wgrant> StevenK: It failed to check out the branch due to an odd pack redirect.
<StevenK> AGAIN?!
<wgrant> Yes.
<poolie> i think it could be potentially useful but it never actually seems useful
<lifeless> poolie: the distro do use it
<poolie> and like it?
<poolie> that's good to know
<lifeless> poolie: I think we should finish and polish it before assessing whether its good or not.
<lifeless> like many things its been done just-enough, rather than made excellent
<poolie> hm
<poolie> that doesn't seem like a very general algorithm
<poolie> but, i think it's worth working out whether the problems are essential or just bugs
<stub> lifeless: The pattern is to prepare the data into a holding area and then perform bulk operations in multiple transactions with small batches.
<lifeless> stub: yup, thats what I thought it was.
<lifeless> stub: the crash proofness I refer to is the assumption [or fact] that identifying the work is expensive.
<stub>  I think you are proposing the holding area be a real table rather than a temporary one, which is doable but a little more effort (db patch to create the table, extra code to clean out the holding area when necessary)
<lifeless> stub: for instance, yes. alternatively make identifying the remaining work cheap.
<stub> lifeless: Even when identifying the remaining work is cheap, it is still more expensive than doing it upfront. Garbo and the librarian-gc have made use of pretty much every possible technique.
<lifeless> do storm StringColumns support a .like() call for queries?
<lifeless> e.g. foo.like('%bnar%')
<stub> I think they do, or there is a LIKE function.
<lifeless> stub: I think it depends on the operation whether upfront checking is needed, is all I'm saying
<lifeless> stub: I recognise the need for variety.
<stub> yes. For our current use cases, I think we could minimize their use if we wanted to, but I don't think it would be sane to eliminate them altogether.
<lifeless> stub: if we *were* to eliminate them, we could:
<lifeless>  - parallelise more easily
<lifeless>  - kill the tasks during db deploys more easily
<lifeless>  - use pgbouncer more easily
<lifeless> its just a thought
<stub> storm.expr has a Like binary operator. Don't know about foo.like() syntax.
<stub> yup
<LPCIBot> Project devel build #493: STILL FAILING in 4 hr 43 min: https://hudson.wedontsleep.org/job/devel/493/
<wgrant> That's not good.
<lifeless> does IntColumn.__eq__(Enum) work ?
<wgrant> I don't know, but it would certainly make you a bad person.
<wgrant> Why?
<lifeless> wgrant: migrating Archive.getPublishedSources to storm so I can DRS it
<poolie> who has permission to mark a branch as an official source package branch?
<lifeless> the tb
<poolie> because they're the owner of the distro series it's going into?
<wgrant> lifeless: I think you just need ~ubuntu-branches membership.
<lifeless> Because they are in the special group created by jmls initial work
<wgrant> Celebrities, yay.
<lifeless> and that group is meant to only have the TB in it now
<lifeless> because it effectively controls upload rights
<lifeless> we should indeed move that to a right granted by owning the distro
<poolie> ok
<poolie> so allowing them to only push the button if they also own the branch would fix this?
<wgrant> We also need to delete bazaar-experts.
<poolie> it would avoid them shooting themselves in the foot by accidentally trusting someone else's branch
<lifeless> poolie: uh
<lifeless> poolie: I am missing a lot of context here I presume
<lifeless> what button, what problem, what accidental trust ?
<poolie> nm
<poolie> i sent mail
<poolie> bug 516709
<_mup_> Bug #516709: revisit official package branch permissions <launchpad> <lp-code> <Launchpad itself:Triaged> <Ubuntu Distributed Development:Invalid> < https://launchpad.net/bugs/516709 >
<wgrant> poolie: Owner == Maintainer
<poolie> not urgent
<wgrant> And that is shown on Distribution:+index.
<poolie> ok
<poolie> there's still a bug as to how you're supposed to know owner==maintainer
<wgrant> Pillars don't have the "owner" term exposed in the UI. It's called "Maintainer" instead (and "Registered by" is the owner for both of those objects, but that is a bug)
<wgrant> Where have you seen a reference to a distribution owner?
<wgrant> Or a project owner?
<poolie> robert's comment in the thread about that bug, and the api
<lifeless> poolie: its not clear what you are trying to optimise for in that discussion
<poolie> > Lets aim for 'excellent' not 'least work' (while still aiming for 'done soon').
<poolie> that's kind of vague
<poolie> who could disagree?L
<lifeless> well
<poolie> but it's not very helpful in working out what is actually simplest
<lifeless> simplest to implement? use? describe?
<poolie> any
<lifeless> I'm essentially getting confused
<lifeless> I'm not sure if you're trying to redesign the distro branches feature [which isn't fully implemented, and the design calls for the permissions to override - thats why there is a bug open for that]
<lifeless> or something else
<poolie> i'm trying to clarify what if anything needs to be done for https://bugs.launchpad.net/udd/+bug/516709
<_mup_> Bug #516709: revisit official package branch permissions <launchpad> <lp-code> <Launchpad itself:Triaged> <Ubuntu Distributed Development:Invalid> < https://launchpad.net/bugs/516709 >
<lifeless> poolie: we need to ensure the identity statement that only uploaders to the distro can upload to the distro
<poolie> sure
<poolie> there are various possibilities to get there
<lifeless> what are your constraints
<lifeless> what are your requirements, and the requirements of your stakeholders?
<poolie> i think the only hard requirement is, just as you said, that only package uploaders should be able to write to these branches
<poolie> afaict that is in fact already met
<lifeless> well, its not.
<poolie> then there are two soft requirements
<poolie> 1- it should not be too easy to make mistakes that open up access-
<lifeless> because the permissions | together rather than excluding.
<poolie> 2- it should not complicate lp's access control user model any more than it is at present
<poolie> which permissions?
<lifeless> poolie: I'm going to put on my paranoid hat and say that 1 is not a soft requirement.
<lifeless> poolie: its a *hard* requirement that it be effectively impossible to open up access.
<lifeless> poolie: this has been well established over the years of UDS discussions.
<poolie> what i mean is it is not a black and white easy
<poolie> how is 'too easy'?
<lifeless> if its possible to have something look like its distro uploader only, it must be distro uploader only.
<poolie> well, that's really my second point, that the user model should be understandable
<lifeless> I'd rather for 2) we say 'we want it to be easy to understand and use'
<poolie> s//black and white issue
<poolie> sure, i agree with that too, i'm just trying to be more specific
<poolie> ok
<lifeless> poolie: so we have a trivial to code tolerable solution mooted in the bug
<lifeless> poolie: I assert that something needs to be done because we don't satisfy the identity requirement today.
<poolie> it has the drawback that we will then have branches owned by say ~mbp that can't be written by ~mbp
<lifeless> right
<poolie> i think this does poorly on transparency and understandability
<lifeless> if you want to incrementally improve things, and aim for shortest path to tolerable source uploads from bzr
<lifeless> then I think you should accept that we already have 'branches own by ~mbp that ~ubuntu-devel can write to'
<lifeless> this neither much worse, nor much better than the status quo.
<poolie> i didn't know we could already have such branches
<poolie> where?
<poolie> but this is on the path to bzr source uploads
<poolie> because having them be adequately secure is important
<lifeless> poolie: if we bless, for instance, ~mbp/bzr/packaging as lp:ubuntu/bzr
<lifeless> then ubuntu-devel get write access to ~mbp/bzr/packaging
<poolie> right, and i keep it
<poolie> that's what i ithkn 516709 is about
<lifeless> yes
<poolie> is blessing existing branches actually useful?
<lifeless> I can't tell if we're violently agreeing or what
<lifeless> poolie: unless you rework the feature from groundup, its necessary.
<lifeless> I'd love to see it reworked.
<poolie> it seems to me, from this thread, that we should get away from blessing existing branches
<poolie> why?
<poolie> i mean, why would it require redoing the feature?
<lifeless> because there is *no* lp:ubuntu/bzr branch - at all, and no space for it.
<lifeless> lp:ubuntu/bzr is *defined* as being an alias.
<poolie> so, given the thread so far, it seemed like it would be cleanest to make it an alias to a branch owned by, say, ~techboard or something similar
<lifeless> that will still require lp changes
<lifeless> and will need care to accomodate linaro etc
<lifeless> who have a different governance structure
<lifeless> [but the distro permissions wouldn't need reworking, if we go with 'distro permissions are all that matter for blessed branches']
<poolie> hm
<poolie> good point about linaro
<poolie> however, i don't think that making the branches owned by the series owner should be a problem
<wgrant> (I'm pretty sure that source package branches need a complete rethink for when we want to support multiple distributions. More complete than you suggest.)
<lifeless> wgrant: Possibly.
<poolie> that just effectively formalizes that the only  way to them is through the distro permissions
<lifeless> poolie: this sets off all sorts of alarm bells
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #3: FIXED in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/3/
<lifeless> poolie: and I really don't want to spend a lot of time painting the bikeshed
<poolie> sure
<poolie> and it's late
<lifeless> poolie: the requirement we were given was 'match the distro permissions'.
<lifeless> I can see two routes to that:
<lifeless>  - genuine *distro* branches that are not aliases.
<lifeless>  - *completely* overriding the permissions when blessed.
<poolie> those both have some problems
<lifeless> I've seen no proposals that meet the requirement other than those two. The former is a moderate amount of work. The latter is going to be confusing if the feature is misused but easy to do and secure.
<poolie> the first is apparently a fair amount of work; the second complicates the already hellacious security model
<lifeless> the incremental complication is near-zero.
<lifeless> Because the elephant is inside the room already.
<poolie> i'm confused because i think i'm proposing exactly what you proposed on 18/2
<lifeless> its fine to argue we shouldn't arbitarily bless branches. But thats orthogonal to meeting the requirement.
<poolie> > I'd make *that* owner for these branches the owner
<poolie> of the distro series the branch is for, not a celebrity.
<lifeless> poolie: you've elided the context
<lifeless> 'We probably want an owner in the same sense that a team has an owner:
<lifeless> someone that has administrative privilege over the thing but no direct
<lifeless> access to the content of the thing'
<poolie> sure
<poolie> anyhow, it's late
<lifeless> that is, people that are members of distro.maintainer can say 'this is the branch' or 'thats the branch' but cannot write to the content of the branch unless they separately have upload permissions
<lifeless> this is part of doing genuine distro branches
<poolie> i'm trying to work out a path to unblock the problems identified in <https://bugs.launchpad.net/udd/+bug/516709/comments/8> with the existing branch
<_mup_> Bug #516709: revisit official package branch permissions <launchpad> <lp-code> <Launchpad itself:Triaged> <Ubuntu Distributed Development:Invalid> < https://launchpad.net/bugs/516709 >
<poolie> i don't want to bake into the user model the idea that branches owned by random users can be blessed
<lifeless> poolie: its already there
<lifeless> I appreciate the motivation, but *its already there*
<lifeless> poolie: conflating 'fix this previous design shortcut' and 'meet the stakeholder requirements' will push your critical path way back.
<poolie> hm
<poolie> ok, good night
<lifeless> heh
<poolie> i'll think about it
<lifeless> poolie: I'm in town in the weekend
<lifeless> which reminds me, I need to mail folk and say 'hi, meetup?'
<wgrant> jam: Around?
<poolie> lifeless, so essentially you're disagreeing with jml's assertion in #8 that the usability consequences need to be fixed at the same time?
<poolie> you may be write that will be faster
<poolie> s//right
<lifeless> yes
<lifeless> its a bit like your thing on reviews
<lifeless> that asking for more work at the same time often just stalls things.
<poolie> true
<lifeless> designing a better solution than we collectively came up with the first time around is possible.
<lifeless> Buts its going to be more work than finishing our initial design.
<poolie> yeah
<poolie> i wonder what people do own official branches at the moment and how they use them
<poolie> does anyone rely on having separate access through that, or does anyone have it but not need it
<poolie> it may not actually cause a problem
<poolie> lifeless, this feels like the kind of thing where lp puts in a poor user model, and then spends more time explaining and supporting it than it would have taken to just finish it properly in the first place
<lifeless> poolie: quite possibly. its a 4 year old design
<lifeless> poolie: the design was optimised for delivery not explainability or usability.
<lifeless> like many things at the time.
<poolie> ok, perhaps you're right we should split them, then separately get away from blessing existing branches
<poolie> is that in fact what you're saying?
<lifeless> yes
<lifeless> I'm not trying to defend blessing at all
<lifeless> its fugly
<adeuring> good morning
<poolie> ok, i see
<poolie> well, really good night then
<bigjools> morning all
<wgrant> Morning bigjools.
<wgrant> P-a-s makes me cry.
<maxb> it does that to most people :-)
<mrevell> Hi
<bigjools> wgrant: I'm sure a boy of you calibre can fix it
<bigjools> your*
<wgrant> Hah.
<wgrant> lifeless: How do I push a config overlay onto the FS?
<lifeless> wgrant: see layers.py
<wgrant> Thanks.
<bigjools> wgrant: so, sparc builders
<wgrant> bigjools: We worked out the problem.
<bigjools> wgrant: yeah, I'm interested to hear details, steve said it was a corrupt bq record?
<wgrant> bigjools: https://wiki.canonical.com/IncidentReports/2011-03-02-LP-build-with-wrong-status
<wgrant> No BQ.
<wgrant> But the BFJ was BUILDING, when it should have been FAILEDTOBUILD.
<wgrant> Very similar to Bug #717969
<_mup_> Bug #717969: storeBuildInfo is sometimes ineffective <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/717969 >
<wgrant> Except the opposite.
<StevenK> Oh, yes, sorry, a BFJ record
<bigjools> wgrant: the report doesn't say *why* the build candidates were not being considered
<wgrant> bigjools: The 80% rule.
<wgrant> Due to the 80% PPA rule, this phantom build prevented further sparc builds in that PPA from being dispatched.
<bigjools> which PPA?
<wgrant> It's in the description!
<wgrant> https://launchpad.net/~ubuntu-mozilla-security/+archive/ppa
<bigjools> urgh
<bigjools> ok I need to teach you how to write incident reports
<bigjools> :)
<wgrant> Yes.
<wgrant> this was all written well afterwards, so it was sort of not very well done.
<bigjools> the other bug is that nonvirtual PPAs should not be doing the 80% check
<wgrant> Indeed.
<bigjools> so the sBI bug is serious
<wgrant> I have seen it only three times. But yes, it is rather serious.
<bigjools> I concur that there's a missing commit
<wgrant> NFI where, though.
<bigjools> but what's worse is that we have a problem with transaction boundaries
<wgrant> Oh?
<bigjools> in that the boundary is wrong, if we can get in an unrecoverable situation
<bigjools> the b-m should have seen the build and fixed it
<wgrant> Yeah.
<bigjools> the first step is a minimal test cast to re-create it
<wgrant> But the first step of that is working out what has gone wrong.
<bigjools> no, it's not
<bigjools> it's working out a test case that re-creates it :)
<wgrant> How do you propose to write a test case when we don't know how to reproduce it?
<wgrant> Well, yes.
<bigjools> "what has gone wrong" is not the same as "how to reproduce it
<bigjools> "
<bigjools> but we're on the same wavelength, so no worries
<bigjools> wgrant: need to talk to you at some point about folding the SoyuzTestPublisher into the LaunchpadObjectFactory
<bigjools> the stuff the LOF does is completely wrong in some cases
<wgrant> bigjools: james_w had a branch to do that at one point.
<wgrant> And I've wanted to get that landed for a long time.
<wgrant> I occasionally improve STP when writing tests... but some of it is just so wrong that it needs a big rethink.
<bigjools> yes
<bigjools> I want to deprecate STP
<bigjools> but the LOF needs fixing, it creates bad data
<wgrant> Yup.
<wgrant> Still, not data that is as bad as the sampledata :)
<bigjools> <shudder>
<danilos> stub, hi, I'd like to create a (constraint) trigger that stops a row from being removed if it's the last one for a foreign key
<danilos> stub, I have https://pastebin.canonical.com/44140/ so far: I am using a subselect to be able to add a LIMIT 1, and COUNT() to ensure I get a value in the variable ("IF FOUND" didn't work for me)
<wgrant> henninge: Hi.
<henninge> Hi wgrant!
<stub> a foreign key references a single row...
<danilos> stub, also, one thing I am worried about is race conditions
<wgrant> henninge: Did you land that sharing information thing just now?
<danilos> stub, right, but I might have multiple rows that reference that same single row in the other table
<henninge> wgrant: yes
<danilos> stub, basically 1 StructuralSubscription -> n BugSubscriptionFilters
<henninge> wgrant: PQM was *very* quick.
<wgrant> henninge: How long did it take from when you submitted it?
<wgrant> henninge: It should have been less than a minute.
<danilos> stub, BugSubscriptionFilter.structuralsubscription is what references StructuralSubscription
<henninge> wgrant: seemed like it
<wgrant> henninge: Great, thanks.
<stub> danilos: So normally, we would just allow the removal and have a garbo job clear out the lost rows.
<henninge> wgrant: did you do that? Great work!
<henninge> thanks a lot
<danilos> stub, oh, I want to stop removal of the final row, to avoid race conditions
<danilos> stub, iow, I stop the removal in python, but it's potentially easy to cause a race condition and still remove it
<stub> And you want to trigger an OOPS instead?
<wgrant> henninge: Yeah, previously we ran buildout and then bin/test to run a test that ran one line of shell... now we just do that directly in PQM.
<wgrant> It's a little bit faster.
<henninge> just a tiny bit ... ;)
<danilos> stub, will that happen if delete removes 0 rows?
<danilos> stub, (and even so, I'd prefer that)
<danilos> stub, or, I could allow removal and then insert another row, would that be better?
<wgrant> I don't understand why you want to prevent removal.
<danilos> wgrant, so python code can assume that there is always at least one BugSubscriptionFilter for every StructuralSubscription (that's what we have already agreed on as the model)
<stub> danilos: The DELETE trigger can allow or block the deletion. I don't think it can make it fail silently or remove a different row. That would be lying.
<wgrant> danilos: Right, but that doesn't really explain why :)
<danilos> wgrant, well, to clean up the semantics, allow easier linking etc (i.e. use just bugsubscriptionfilter for linking instead of having to have complex logic for choosing structuralsubscription of bugsubscriptionfilter)
<danilos> wgrant, today, you can have "no filtering" in two different ways: SS with no BSFs, or SS with BSF with no filters
<danilos> wgrant, we don't like the duality
<wgrant> Ah, right, forgot that you wanted to link them to notifications.
<stub> I don't see why a structuralsubscription must have >= 1 bugsubscriptionfilter's myself.
<danilos> stub, right, the trigger I have so far seems to give me the right behaviour, I am just wondering if that's the optimal or good way to do it?
<danilos> stub, see above, though we can argue if they are strong enough reasons
<stub> danilos: It doesn't work if two transactions delete the last two referencing rows in separate transactions at the same time.
<stub> Both sessions think there is another reference, both succeed, both rows are removed.
<danilos> stub, that's what I was wondering; I see that one can have a CONSTRAINT TRIGGER to happen at the end of transaction, which would work better, but that one can only be an AFTER trigger, meaning it can't stop deletion (naturally, since everything else in the transaction might be messed up)
<danilos> stub, also, would it be possible to explicitely lock rows that I care about?
<stub> Your just shuffling the race condition around
<stub> Yes, SELECT FOR UPDATE will do that.
<stub> But this is a complex way of proceeding. Ideally we find a non-complex way of solving the problem.
<danilos> stub, agreed re shuffling race condition around, but having more safeguards in place reduces the chances of it happening, and ideally, I'd be avoiding it completely
<danilos> stub, any suggestions on a non-complex way? (not that I see this as overly complex)
<stub> We just keep outerjoining with the table and not fighting the model SQL gives us.
<danilos> stub, that's something I'd have to (re)discuss with my team, since we are generally relying on this already in a few places; if it's not too many places, perhaps we can go back
<danilos> stub, one of the reasons we are going this way is to allow easier merging of structuralsubscription and bugsubscriptionfilter tables in the future
<wgrant> bigjools: Do you want to review https://code.launchpad.net/~wgrant/launchpad/generalise-add-missing-builds/+merge/51873, or should I grab someone else?
<stub> fwiw, there is nothing in the data model preventing a bug from having no linked bugtasks, but that hasn't been a problem.
<wgrant> bigjools: Also, is there an existing wiki page on adding new archs, or should I create one?
<bigjools> wgrant: I can do it in a bit
<wgrant> I searched around earlier, but found nothing :/
<wgrant> Which I guess is unsurprising, given the questionable results of the last two new ones :)
<bigjools> wgrant: feel free to start one
<danilos> stub, right, so your suggestion is to just ignore the potential race condition? (in our particular case, it is very unlikely, but if we don't put any safeguards in, a malicious person will be able to do it, especially if we allow API access)
<stub> danilos: So to do this. In Python, you do 'SELECT * FROM BugSubscriptionFilter FOR UPDATE', then check there is at least one other link. If there is, proceed with the deletion.
<stub> If not, render an error message
<stub> If two transactions try that at the same time, one will block until the other commits, then a serialization exception is raised and Zope retries the request
<stub> danilos: Doing it in the Python level means you catch the case with a nice error
<danilos> stub, right, the reason I wanted this as the trigger is because I ran into a place where someone was not using the appropriate Python API, but instead removed a row directly using store.find(...).remove()
<danilos> stub, true, agreed
<stub> danilos: The trigger would need to duplicate this logic, but it will cause an OOPS.
<danilos> stub, right, but if we guard the proper API in Python, this would indicate a bug, which is exactly what we want :) though, I can't make it work in the trigger
<stub> danilos: I'd ignore the race condition - do you really care if there is a structuralsubscription with no bugsubscription? Malicious isn't an issue - you can only shoot yourself in the food.
<danilos> stub, do you know how to "discard results of a SELECT using PERFORM"?
<danilos> stub, right, but I hate to see food flying around :)
<danilos> stub, and you are definitely right, we can cause problems for ourselves in many more different ways, so I'll just guard against it in the python code
<danilos> stub, thanks
<stub> danilos: Yes, but the fallout from all the extra complexity you are adding is likely worse than the problem.
<stub> PERFORM is a pl/pgSQL statement so only useful in a stored procedure. Syntax similar to SELECT but it throws the results away.
<danilos> stub, I assume that last comment is re trigger implementation?
<stub> It depends on how contrived your race condition is.
<stub> If you have a form """( ) bugsubscription a (x) bugsubscription b""", it is reasonable to assume someone clicks 'delete', thinks 'oh shit', changes the check box and clicks submit again. So catching this in Python is reasonable.
<stub> If you need to contrive an example with two browser windows opened to the correct page and the user submitting both at once, you might not want to bother.
<danilos> stub, atm it's the latter, but the UI is not settled yet
<stub> I certainly wouldn't bother catching the case of the form submission being processed at the same time as some background script running that does the same removal (esp. as I doubt we would have a background script performing that removal)
<danilos> stub, I can imagine the UI becoming what you described first, and it seems it's not hard to guard against it with SELECT FOR UPDATE
<danilos> stub, ok, and just for fun, the trigger works with PERFORM FOR UPDATE, so something I learned today :)
<stub> Right. Should be able to simplify it too. Once you have locked all the rows referencing the structural subscription, you only care if there is a result from SELECT TRUE FROM BugSubscriptionFilter WHERE structuralsubscription=OLD.structuralsubscription AND id <> OLD.id'
<stub> I think you can lie and silently not do the deletion, but I think it is better to raise an exception here.
<danilos> stub, ah, SELECT TRUE trick is a very nice one
<danilos> stub, is it worth doing a LIMIT on that query as well? (just for my education, it seems it is :)
<stub> danilos: That wouldn't lock all the rows then
<stub> Oh... the check if the result exists. Yes.
<danilos> stub, ah, so I would put the FOR UPDATE in this query as well?
<stub> Thought you meant on the FOR UPDATE, which should not have a LIMIT :)
<danilos> stub, right :)
<bigjools> wgrant: DistroSeries:+queue just hove into view on the oops reports
<bigjools> I hate that page
<wgrant> bigjools: Yes. :(
<bigjools> I remember when it used to have 3000+ queries on it.  I was quite chuffed when I got it to ~600
<bigjools> that looks big now :/
<wgrant> A lot of it is the same problem as +copy-packages: createMissingBuilds is shit.
<bigjools> there's also a massive problem when there are many packages with custom files
<wgrant> Yeah.
<bigjools> I only optimised binary and source uploads, custom packages are still bad
<wgrant> It's fairly easy to optimise now that we're allowed to do caching on the model.
<wgrant> GETs, that is.
<wgrant> POSTs are still hard.
<bigjools> we can redirect
<bigjools> I think it does that already
<wgrant> We already do. Most of the work is skipped on POST.
<wgrant> *Most*.
<bigjools> :)
<Daviey> bigjools, I /just/ noticed bug #610491 is fixed... but I can't get it to work... getPublishedSources(package_signer=IPerson)... i would have thought would return a packageset?
<_mup_> Bug #610491: [API] Please expose getPublishedSources(package_creator,package_signer) <api> <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/610491 >
<bigjools> Daviey: why do you think it was fixed?
<bigjools> getPublishedSources returns publications, not packagesets BTW
<Daviey> bigjools, ah yeah... that is what i want
<bigjools> we can add the filtering as per the bug, but it's low priority
<danilos> StevenK, hi, are you still awake? :)
<bigjools> I'm sure everyone wants a fast, non-oopsing Launchpad first :)
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos | https://code.launchpad.net/launchpad-project/+activereviews
<Daviey> bigjools, i thought it was fixed because i confused it with bug #372704
<_mup_> Bug #372704: expose Signed-by and Changed-by via API <api> <lp-soyuz> <Launchpad itself:Fix Released by julian-edwards> < https://launchpad.net/bugs/372704 >
<bigjools> heh
<Daviey> ETOOMANYSIMILARBUGS! :)
<jml> My laptop is busted.
<StevenK> danilos: Hai?
<danilos> StevenK, you were listed as OCR, I've removed you thinking you might be done with that
<StevenK> danilos: It's 1130pm, I so am :-)
<danilos> StevenK, ok, cool, enjoy the night then :)
<bigjools> jml: that sucks
<wgrant> jam: Hi.
<jam> hi wgrant
<wgrant> jam: We have another chance to roll out the forking service next week. Do you think we should try?
<jam> wgrant: I would like to land one more patch, but I think I can make it by next week
<jam> https://code.launchpad.net/~jameinel/launchpad/lp-serve-child-hangup/+merge/50055
<jam> but poolie approved it, pending switching to signal.alarm()
<wgrant> jam: Right, I've seen that sitting around.
<wgrant> I think I would prefer an explicit handler.
<jam> well, it seems approved from the conversation, I'll certainly put it up again for review
<jam> wgrant: something to handle SIGALRM?
<wgrant> Yeah.
<wgrant> But I guess it doesn't really matter.
<jam> wgrant: it makes the testing easier
<jam> since otherwise I have to worry about it when testing that specific code
<wgrant> True.
<jam> I'll think about it
<jam> I can certainly just install a handler for the test itself
<wgrant> Thanks.
<jam> wgrant: any idea why ^C in the middle of the test cancels the current test with OK, rather than generating a traceback and failure where it is blocked?
<jam> I think this changed with testtools
<jam> maybe jml ^^ is better to ask (though prob not in this timezone :)
<jml> jam: it's complex. depends on the test.
<jam> jml: I've seen it on just about all test runs I've tried
<jam> wgrant: do we have a way to tell that a process, which is not a direct child, has exited?
<wgrant> jam: "We" meaning victims of POSIX? I don't think so.
<jml> poll ps :P
<jam> jml: yeah, that's what I thought was required. which... I'm not doing. But thanks for confirming
<jml> jam: I guess you could do something like upstart
 * danilos -> food
<jam> wgrant: https://code.launchpad.net/~jameinel/launchpad/lp-serve-child-hangup/+merge/51893
<jam> if you want to give it a look ovre
<jam> the only other thing I can think of, is wanting to have the full TIMESTAMP and PID in log files, rather than mutter's default of time-since-started
<jam> but that doesn't seem like a blocker for rollout
<wgrant> jam: Thanks. I'll have a look tomorrow.
<leonardr> hey, benji, i'd like to talk about the way in which exceptions in launchpad are turned into http responses in lazr.restful
<leonardr> the more i look at this, the more it looks like two systems that do the same thing
<leonardr> for instance, you filed bug 631711
<_mup_> Bug #631711: Ability to specify HTTP result code when using lazr.restful.error.expose <lazr.restful:Triaged> < https://launchpad.net/bugs/631711 >
<leonardr> but we have that--it's the webservice_error() decorator
<leonardr> but the webservice_error() decorator on its own doesn't work
<leonardr> you have to call expose() on the exception object as well
<leonardr> and i can't change webservice_error() to call expose() on the exception class, because expose() is for objects
<leonardr> so you have to change it in two places--call webservice_error() on the class and expose() on the object
<leonardr> this has been bothering people for a while and i think i'll do something about it
<leonardr> but, i'd like to check in with you since it looks like you are responsible for the other system
<benji> leonardr: I'm on a call at the moment.  I'll be off in about 5 minutes.
<leonardr> sure
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #494: FIXED in 5 hr 11 min: https://hudson.wedontsleep.org/job/devel/494/
<benji> gary_poster: thanks, we'll need it; I'm almost certain we're upside down, but there's always the option of a short sale
<LPCIBot> Project db-devel build #411: STILL FAILING in 6 hr 4 min: https://hudson.wedontsleep.org/job/db-devel/411/
<benji> gary_poster: thanks for the pointer, it's hard to find a good agent
<gary_poster> benji, ack, and following you around on the channels is exciting ;-)
<benji> LOL
<benji> ok, I really need to do something about that
 * benji starts a local chapter of Channel Hoppers Anonymous
<benji> leonardr: the root difference is that one is about exposing exception classes and the other is about exposing exception instances (especially instances of built-in exceptions)
<leonardr> right
<leonardr> the problem is that on its own, exposing an exception class does absolutely nothing
<leonardr> you also have to expose the instance
<benji> oh, you do?  I thought that if the framework caught an instance of an exposed exception class then it would handle it specially
<leonardr> no, it doesn't
<leonardr> i don't know if it used to and doesn't anymore, or what
<benji> hmm
<leonardr> ideally there would be one method that you could call within a class definition, or on a class, or on an instance
<benji> two courses of action suggest themselves: make exposing an exceptoin class actuall do something or do away with the class-based mechanism alltogether
<leonardr> the class-based mechanism is very useful because you can take care of the whole thing in one place
<leonardr> the reason you filed that bug is because expose() only works for exception classes that give a 400 error code
<leonardr> so... let me try to come up with a solution
<abentley> henninge: standup?
<henninge> abentley, adeuring: Can we do skype?
<abentley> henninge: sure.
<adeuring> surwe
 * adeuring needs to start a machine where skype is installed. 2 minutes, please
<leonardr> benji: argh, we actually have _three_ systems, only one of which works
<benji> leonardr: yow
<leonardr> but two of the systems are alternate spellings
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos, abentley | https://code.launchpad.net/launchpad-project/+activereviews
<leonardr> benji: do you know of any reason why it shouldn't be allowable to call expose() on an exception __class__ (or do the equivalent--basically, force a class to implement a "error handled specially by the web service" interface
<benji> leonardr: that seems reasonable; I'd make it a class decorator (as well as a normal function) so you could use it either way
<bigjools> sinzui_: will your connection stay up long enough to answer a question? :)
<sinzui_> doubtful. unity cannot decide to show menu in the menu bar or the window. My windows are shaking with anger
<bigjools> sinzui_: well let me try. rvba is having problems using create_initialized_view in a test with a principal of None
<sinzui_> he is rendering content?
<bigjools> https://pastebin.canonical.com/44154/ causes https://pastebin.canonical.com/44155/
<bigjools> yes
<rvba> sinzui_: calling render on the view triggers a LocationError: 'name_link'
<rvba> sinzui_: (Hi by the way ;-))
<sinzui_> He needs to pas the principal because menus links and login often needs to get the user form the interaction
<bigjools> what about an anonymous user?
<sinzui_> Most Lp pages will not render with an anonymous user. The interaction is not setup right with anon
<bigjools> hah
<rvba> sinzui_: ok
<bigjools> didn't know that
<sinzui_> he can pass the arg like
<bigjools> I guess I was lucky to find a page that does when I've done this
<sinzui_> principal=distro.owner to hack the case
<rvba> sinzui_: login(someuser) won't do?
<sinzui_> the login() will still work for permission
<sinzui_> no
<bigjools> rvba: login() is for the zope interaction
<sinzui_> rvba, We prefer login_person() so that we have the person to pass as the principal
<rvba> sinzui_: got it
<rvba> bigjools: I though every security check was "zope interaction"
<bigjools> rvba: yes, logging in sets up the user for the interaction
<rvba> bigjools: what is the part that is *not* zope interaction?
<leonardr> benji: see if this makes sense to you: http://pastebin.ubuntu.com/574501/
 * benji looks
<bigjools> rvba: anything that's not using database objects, views and utilities generally
<bigjools> so not much :)
<rvba> bigjools: ok, good to know
<rvba> sinzui_: thanks for the clarification
<benji> leonardr: that sounds like it accurately describes the status quo
<leonardr> ok, now to figure out a better system
<leonardr> i'm kind of thinking of a system in which we do a named operation based on __lazr_webservice_error
<leonardr> er, a named adapter lookup
<leonardr> since the way we describe this class of exception is by saying what response code we want to send instead of 500
<leonardr> so if you could do @error_status(400) on an exception class, or expose(exception, 400) on an instance
<leonardr> lazr.restful would do an named lookup adapting to IWebServiceErrorView, '400', and you would get a ClientErrorView
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
<danilos> abentley, hi, do you mind taking a look at https://code.launchpad.net/~danilo/launchpad/devel-bug-720826-delete-race/+merge/51890?
<abentley> danilos: sure.  OTP.
<danilos> abentley, great, thanks, I'll be around for discussion as well
<leonardr> benji: what do you think of this? http://pastebin.ubuntu.com/574528/
<leonardr> (skip to "a better system")
<abentley> danilos: back.
<danilos> abentley, generally, the branch is quite simple; few things I have on the top of my head: perhaps I should split the delete_final test into a test-per-attribute, though I am not sure about that; second, I am not testing the race condition handling
<danilos> "I have on the top of my head" is an interesting way to put it, though :)
<abentley> danilos: I haven't seen "FOR UPDATE" before.  Is that why you didn't use Storm expressions?
<danilos> abentley, yeah
<danilos> abentley, that's how one locks rows in Postgres, I just learned that today from stub :)
<abentley> danilos: cool way of verifying that an object is deleted.  I've been grabbing the ID and then querying for it.
<danilos> abentley, this should handle race conditions where someone tries to delete it from a different place, the check succeeds in both, and then you try the delete; this way, one of them gets locked out until the check and delete are done
<abentley> danilos: I meant using self.assertIs(None, Store.of(bug_subscription_filter))
<danilos> abentley, ah, right :)
<abentley> danilos: what is getDescriptionFor?
<danilos> abentley, oh, it returns you things like Choice(blahblah) object from the interface; I've looked it up using help(Interface)
<danilos> abentley, perhaps I could say it returns an attribute "descriptor"
<abentley> danilos: Okay, cool.
<abentley> danilos: if I were writing it, I'd probably look up the default on a separate line, just to avoid clutter.  But that's a matter of personal taste.
<abentley> danilos: r=me.
<danilos> abentley, a thought did occur to me as well, so since it's not just me, I'll do it :)
<danilos> abentley, and thanks!
<abentley> danilos: np.
<lifeless> Ursinha: matsubara-lunch: whats the landing process for qatagger ?
<Ursinha> lifeless, it's all explained on the README file (I believe)
<Ursinha> lifeless, wait, let me find the wiki page
<lifeless> Ursinha: its not :)
<lifeless> Ursinha: I have this branch diogo approved, I would like to land it
<Ursinha> lifeless, first what I do is to merge it on trunk
<Ursinha> after that, a make dist
<Ursinha> I pick the package and copy over to devpad, untar it, make install
<Ursinha> and create a qa link for that folder
<Ursinha> change the cron entry and watch
<Ursinha> ah, I copy the files last-revno-* from current to qa folder
<leonardr> benji: got some time for more brainstorming?
<Ursinha> and create a link for logs and current-milestone
<Ursinha> I'm sure that's explained somewhere, I guess a wiki page I wrote
<Ursinha> just a moment
<Ursinha> lifeless, did you manage to get access to lpqateam?
<Ursinha> on devpad, that is
<lifeless> Ursinha: haven't moved that forward
<lifeless> Ursinha: thats *deploy*
<lifeless> Ursinha: how do I get my change into trunk ?
<Ursinha> ah
<Ursinha> for me that's land :)
<lifeless> Ursinha: I like it :)
<Ursinha> lifeless, if that's approved, merge into devel branch and push
<Ursinha> then you have to deploy
<Ursinha> that's what I described
<lifeless> Ursinha: ah, so I can't run the test suite at the moment
<lifeless> some buildout error
<lifeless> perhaps you could check I haven't broken a test and push it to devel if its ok ?
<Ursinha> ah, I know
<Ursinha> I guess that's trying to find a launchpadlib version that isn't there anymore, just a moment
<lifeless> yes
<lifeless> Error: Couldn't find a distribution for 'launchpadlib==1.6.3'.
<Ursinha> lifeless, it seems mars removed his pypi folder that contains the dependencies, I'll add one to ~ursula
<Ursinha> you can fix the find-links to point to ~ursula, I'll fix that
<lifeless> Ursinha: can you merge this one into trunk - I need to go chase two in-progress timeout bugs
<Ursinha> lifeless, of course :)
<lifeless> Ursinha: I appreciate you fixing the process blockers immediately - thats great stuff.
<benji> leonardr: sure, give me about 10 minutes
<leonardr> k
<lifeless> Ursinha: thanks
<Ursinha> lifeless, np
<lifeless> bigjools: are you around for much longer ?
<bigjools> lifeless: 10 mins
<lifeless> I'm having trouble converting archive.getPublishedSources to storm
<bigjools> ho ho :)
<lifeless> I want to do that to change the prejoins to set based eager loaders
<bigjools> why are you doing that?
<bigjools> ok
<lifeless> https://bugs.launchpad.net/launchpad/+bug/727560
<_mup_> Bug #727560: Archive:EntryResource:getPublishedSources <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727560 >
<lifeless> 15 second query atm
<bigjools> meep
<lifeless> you could say that
<lifeless> bigjools: are there unit tests for this, or just 'archive.txt' ?
<bigjools> lifeless: just the doctest :/
<lifeless> thanks
<bigjools> feel free to rip it out
<lifeless> migrate to unit tests ?
<bigjools> however, that method is used *everywhere* - loads of other code and tests use it
<lifeless> that would make it a lot easier to debug
<bigjools> yes, migrate if you want
<bigjools> archive.txt is way too big
<lifeless> I'll have a look see
<lifeless> thanks
<bigjools> np
<bigjools> lifeless: did you want any more help?
<sinzui_> abentley, do you have time to review https://code.launchpad.net/~sinzui/launchpad/sane-definition-status-0/+merge/51932
<bigjools> too late if you do, I'm off :)
<bigjools> good night all
<leonardr> benji: i think i've reduced my remaining problems do "why doesn't this component lookup succeed?" want to take a look?
<benji> leonardr: sure
<leonardr> ok, pushing
<leonardr> lp:~leonardr/lazr.restful/consistent-error-exposing
<leonardr> if you run bin/test there's a breakpoint right at the bit where everything goes south
<leonardr> look at the SystemErrorView registration in configure.zcml. that's the one that's failing
<leonardr> and look in _resource.py, the except clause in ReadWriteResource.__call__
<leonardr> that's where the lookup happens
<abentley> sinzui_: happy to.
<leonardr> i've manually verified IException.providedBy(e), IWebServiceLayer.providedBy(self.request), and Interface.providedBy(IWebServiceErrorView)
<leonardr> hm, it succeeds if i copy simple.py and remove the thing i'm trying to adapt to
<leonardr> but i get the wrong status code
<leonardr> i get SystemErrorView
<leonardr> rather than WebServiceExceptionView
<benji> looking at it now
<leonardr> i think i got it, actually. i'm just not sure why i had to start adapting to Interface (presumably the default) instead of IWebServiceErrorView
<leonardr> let me get it working and then you can critique the working version
<leonardr> benji: pushed working version
<leonardr> basically, WebServiceExceptionView is now the default view for all exceptions unless another view is registered
<benji> looking
<leonardr> and if the exception happens to have been annotated (by any mechanism), WebServiceExceptionView will defer to the annotation when it comes to the status of the response
<leonardr> and the exception handler does a lookup for any exception, not just for a client error
<abentley> sinzui_: Why are you addressing this issue by providing a new enum?  Why not just check that the value is in a set of acceptable values?
<benji> leonardr: looks good very good to me
<leonardr> benji: ok, maybe you'll like this other change, which takes it one step further...
<leonardr> benji: check the new version. it *always* turns an exception into a view, and gets rid of IWebServiceError altogether
<benji> k
<benji> leonardr: wouldn't that mean that an "accidental" exception would result in a 400?  It seems a 500 would be more appropriate.
<leonardr> benji: an accidental exception would turn into an un-annotated WebServiceExceptionView
<leonardr> and its status would be the default, 500
<benji> leonardr: ok, sounds like a plan; simpler code and it behaves the way we want
<leonardr> all right, i'll fix the test failures and submit a formal mp
<leonardr> benji: this does mean that the publisher will not be handling any errors. is that ok?
<lifeless> leonardr: will we still get OOPSes on unannotated exceptions?
<benji> leonardr: hmm, does our custom publisher do retries on DB availability errors (like conflicts)?  If so, it won't get the error to know that the error happened.
<leonardr> lifeless: i don't know. lazr.restful will set X-Lazr-OopsId if request.oopsid is present, but i imagine there's more to "getting an oops" than that
<sinzui_> maybe I should install unity-2d. The constant shaking windows is making me ill.
<lifeless> leonardr: I mean the server side mechanism where raising is called()
<benji> leonardr: lifeless makes a good point; we may want to only catch the errors that were marked as "not exceptional exceptions" instead of intercepting potentially hair-on-fire exceptions (MemoryError comes to mind)
<lifeless> benji: you credit me with too much... that is also an excellent point
<leonardr> benji: what if we raised exceptions if their error code was 500, for whatever reason? either because no other code was specified or because the error was explicitly marked as a server error?
<lifeless> benji: my concern was more prosaic - I want our feedback loop for fixing crasher bugs to remain intack.
<lifeless> leonardr: benji: may I ask what problem you're trying to solve ?
<lifeless> s/trying to solve/solving/
<benji> leonardr: I'm not sure I understood that part ("what if we raised exceptions if their error code was 500...")
<leonardr> lifeless: we have 2.5 systems for telling lazr.restful to treat a given exception/exception class as a certain http response code, and only one of them works
<leonardr> that's done; this is ancillary cleanup work, in which the question is "which exceptions can lazr.restful handle on its own, and which should it delegate to the publisher?"
<leonardr> benji: so, earlier we effectively re-raised exceptions if their status code was not 400
<lifeless> oh. Have you considered 'if lazr.restful does not explicitly know, its not its business' ?
<benji> leonardr: I favor reverting that last bit (r181)
<leonardr> lifeless: yes. the question is what it means to 'explicitly know' this
<lifeless> leonardr: that would seem to be dependent on the system you've chosen for telling lazr.restful how to determine result codes for exceptions
<leonardr> benji: if i revert the last bit, then exceptions will be re-raised unless they have been annotated with a return value. i don't think that's necessary or sufficient
<benji> leonardr: that's not the way I read the code.  It seems to me that we reraised if the exception wasn't marked as "an exception that the client caused".
<leonardr> an exception that the client caused -> 4xx response code
<leonardr> we can make that explicit
<leonardr> let me do some code and tests and show you waht i mean
<benji> I think we're talking about this bit, right? http://pastebin.ubuntu.com/574616/
<leonardr> benji: yeah. the comment is wrong
<benji> leonardr: the comment before or after the change is wrong?
<leonardr> before
<leonardr> if an exception is annotated as 301 or 503 (or 200 for some reason)
<leonardr> lazr.restful would have handled it
<leonardr> even though its decoration does not indicate that it's the client's fault
<benji> ah, ok
<leonardr> i think it makes sense to only handle exceptions where the response code is 4xx
<leonardr> and let the publisher handle the rest
<leonardr> that way, if you explicitly decorate an exception as 5xx, it will become an oops
<benji> the exceptions-as-control-flow thing was throwing me
<lifeless> leonardr: +1
<benji> the issue there would be that (if I'm understaning all the mechanisms correctly) that if you decorate an exception as a 50X but the code in question re-raises the exception, the publisher's error view may well retorn something other than 50X (say it may have 500 hard-coded).  Does that jive with your understanding?
<leonardr> benji: yes, that might happen
<leonardr> i think that failure mode is minor compared the the failure modes that might happen if lazr.restful handled all the errors
<benji> It seems to me that since we're not crying out for more controll over 500s, we should designate this decoration mechanism as for status codes from 400 and down, that way we don't have to get our fingers in the "a *real* error happened reporting story" while also greatly improving the "client did something wrong" story.
<leonardr> i think just handing 4xx is good enough--that's effectively what we were doing before (except we only handled 400)
<leonardr> and the publisher may have some way of handling exceptions that are marked as redirects, or something weird like that
<benji> +1
<leonardr> ok, i think we have a winner
<LPCIBot> Project db-devel build #412: STILL FAILING in 5 hr 17 min: https://hudson.wedontsleep.org/job/db-devel/412/
<lifeless> oh yay, tz boobytraps in archive.getPublishedSources
<leonardr> benji: https://code.launchpad.net/~leonardr/lazr.restful/consistent-error-exposing/+merge/51946
<leonardr> punt to abentley if you don't want to do this right now
<benji> leonardr: I'd love to do the review, but I have some pressing work I really need to put some time into
<leonardr> abentley, can you take a look?
<LPCIBot> Project windmill build #5: FAILURE in 1 hr 13 min: https://hudson.wedontsleep.org/job/windmill/5/
<thumper> matsubara: ping
<matsubara> hi thumper
<thumper> matsubara: sorry for not getting back to you earlier
<thumper> matsubara: you were asking about some recipe testing
<thumper> matsubara: what do you need from me?
<matsubara> thumper, can we have a quick call?
<thumper> matsubara: sure... skype or mumble?
<matsubara> thumper, let's try mumble
<thumper> matsubara: ok
<matsubara> I'm on the product channel
<matsubara> thumper, https://dev.launchpad.net/QA/ExploratoryTesting
<leonardr> benji, i'm not sure why we have both webservice_error and @error_status--they do the same thing with slightly different syntax. do you have any ideas? should we standardize?
<benji> leonardr: nope, no idea; I'm pretty sure both existed when I started looking at them
 * leonardr will leave it alone for now
<thumper> leonardr: mumble?
<leonardr> thumper, yes!
<leonardr> thumper: https://code.launchpad.net/~leonardr/lazr.restful/consistent-error-exposing/+merge/51946
<thumper> leonardr: ok, I'm left a little confused with your merge proposal
<thumper> leonardr: I think I was fine up until the last test
<thumper> which left me questioning
<leonardr> thumper: an alternative is to prohibit the caller from marking an exception with a response code not in the 4xx or 5xx series
<thumper> leonardr: in test_decorated_exception_instance_becomes_error_view when the resource is called, you get a result
<thumper> leonardr: but in the last test, when you call the resource it raises
<thumper> why?
<leonardr> thumper: because of line 27 in the diff
<leonardr> if the response code is 4xx we handle the response (since the error was the client's fault)
<leonardr> in all other cases we let the publisher handle it
<thumper> ahh
<thumper> ok
<thumper> although the 200 case seems a bit of an edge case
<thumper> I guess it is there for completeness
<leonardr> thumper: yeah, we could also prohibit it, like i said
<leonardr> but if the user wants to do that, we don't really know what they're trying to do, so better let the publisher handle it
<thumper> approved
<leonardr> great
<leonardr> i'll land but not do a release, as integrating into launchpad may require other changes
<thumper> ok
<lifeless> matsubara: please triage new bugs :) - 728059
<matsubara> lifeless, will do
<matsubara> I just filed them
<poolie> i just got Error 324 (net::ERR_EMPTY_RESPONSE): Unknown error. from https://bugs.launchpad.net/~canonical-bazaar/+assignedbugs
<poolie> good morning btw
<lifeless> wallyworld: BrowsesWithQueryLimits is your best option for that test
<wallyworld> lifeless: yeah. i'll need to modify it to suit.
<wallyworld> currently it takes a context object and i need to give it a url
<wallyworld> i'm keen for the branch to land to improve +daily-builds performance even more
<lifeless> wallyworld: I would give it an object and a view selector
<lifeless> defaulting to +index
<lifeless> wallyworld: then you can use the Root object and +daily-builds
<wallyworld> ok. haven't really thought too much about it yet. have to context switch back. sounds good what you're suggesting
<lifeless> wallyworld: or you could make a parallel one (perhaps sharing code) that takes a url
<lifeless> wallyworld: we need to talk separately about your bug task search patch
<wallyworld> ok. i'll take another look and see if i can decorate the rs outside the search implementation
<lifeless> wallyworld: can you tell me what you need to do ?
<wallyworld> basically the current search functionality only returns bugtask and i need to also access tables from the query eg BugBranch
<lifeless> [this is why I don't like itsy bitsy landings without context - they are harder to review]
<lifeless> wallyworld: /why/
<wallyworld> so i can, for each bugtask, create a data structure that allows be to lookup a bugtask based on branch id, so i can find all info i need with one query
<wallyworld> s/be/me
<wallyworld> otherwise i would have to do one query per branch
<lifeless> wallyworld: this is the privacy thing you're fixing ?
<lifeless> wallyworld: I can give you a one liner to do what you need a different way
<wallyworld> and when using the search method and passing in linked_branches=any(), the BugBranch table is in the query but i don't get to access it currently
<wallyworld> yes
<lifeless> visible_bugs = getUtility(IBugTaskSet).searchBugIds(BugTaskSearchParams(user=self.user, bug=any(*(bug.id for bug in bugs_you_have_today))
<lifeless> this is one extra query, but should be fairly lean, and it will be constant.
<lifeless> the problem with passing a result decorator into the query is that you actually want a wider tuple out, which such a decorator won't give you
<wallyworld> actually mine does. it works :-)
<wallyworld> and doesn't cause any extra sql
<wallyworld> but i'll take a look at your suggestion
<lifeless> pastebin it ?
<wallyworld> lifeless: https://pastebin.canonical.com/44193/
<wallyworld> lifeless: with your suggestion, at first look, it still seems like i won't have access to the info i need with one query, which is bugtasks for each branch
<wallyworld> i'll look in more detail in case i'm missing something
<lifeless> ok
<lifeless> uhm, I'm worried about the interface basically
<lifeless> there is a compiler in there that is fiendish
<lifeless> what you have will actually cause more queries
<lifeless> if you have *visible* private bugs
<lifeless> because their visibilty won't be cached
<wallyworld> hmm. ok
<lifeless> if you tested with *invisible* private bugs, it will seem fine.
<lifeless> because you're fixing half the problem
<wallyworld> the main problem is that the search() assumes you only want bugtasks and not other stuff from the query
<wgrant> sinzui: No critical issues with your CSS changes?
<lifeless> wallyworld: thats what its designed for
<sinzui> no critical
<wgrant> OK, we shall deploy then.
<wgrant> Thanks.
<wallyworld> hmmm. don't like the design then :-)
<lifeless> wallyworld: there are some options - eager load (e.g. into a cache property) the branches; ask for related things in the result
<lifeless> wallyworld: Here is another problem with your fix
<lifeless> wallyworld: you'll get multiple rows back for the same bugtask.
<lifeless> wallyworld: (if branch A and B are both linked to bug C, then you'll get a row for (C, A) and for (C, B))
<wallyworld> so long as the related branch in each row is correct that won't matter
<wallyworld> i building a dict of branch->related bug tasks
<lifeless> wallyworld: why won't it matter? It will still cost cpu: cache refreshing of duplicate rows in storm.
<wallyworld> but the rows for (C,A) and (C,B) are valid and required
<lifeless> wallyworld: I would do a separate single query on bugbranch for (BugBranch.bugID.is_in(bug_ids), BugBranch.branchID.is_in(source_branch_ids))
<wallyworld> sounds reasonable. get the bug ids from the search() and then another query to load the required info
<wallyworld> if visible private bugs were cached, my way would require less queries :-P
<wallyworld> thanks for the input. i'll get back to it after food and after i resubmit the +daily-builds perf improvement stuff
<wgrant> Looking at the stable merge failure.
<wgrant> sinzui: Oh, yesterday I also killed Windmill and made PQM fast.
<huwshimi> wgrant: oh just that, I'm surprised you remembered at all!
<sinzui> wgrant: you made 5 minute PQM (a merge in less than 5 minutes) actually work as promised!
<sinzui> I think we gave up on that 18 months ago
<wgrant> Sadly prod config landings are still slow.
<wgrant> But they are rare enough.
<poolie> hi all
<wgrant> Oh no :(
<wgrant> Now PQM is going to spam us more often with stable->db-devel conflicts :(
<jcsackett> wgrant: the 5 min for main launchpad stuff is amazing. i think we can all bare the configs being longer. :-)
<wgrant> Huh.
<wgrant> A test went missing when r12399 was automatically merged into db-devel.
<lifeless> orly?
<wgrant> lifeless: Compare the stable r12399 and db-devel r10214 diffs, lib/lp/registry/tests/test_person.py
<lifeless> wgrant: if it was already in db-devel
<lifeless> then it wouldn't show in the diff
<wgrant> It's not in db-devel now, and doesn't seem to have been removed since.
<lifeless> ok, thats more worrying then
<wgrant> But let's see the tree at that rev...
<wgrant> No, not there in 10214 either.
<lifeless> ok, thats whack
<lifeless> if you merge the two revs locally, does bzr do that?
<wgrant> Waiting for that to happen.
<wgrant> Warning: criss-cross merge encountered.  See bzr help criss-cross.
<wgrant> Heh.
<wgrant> And it's missing.
<wgrant> So the criss-cross kills it without a conflict.
<wgrant> This is sort of really bad.
<lifeless> we can teach pqm to abort on criss cross
<wgrant> I'll revive the test now and look through the stable->db-devel diff to check that nothing else is missing.
<spiv> lifeless: not just whack, wiggida wiggida wiggida wack!
<lifeless> hi daddy :)
<spiv> lifeless: well it's a criss-cross problem :P
<jcsackett> ...gonna make you jump, jump.
<lifeless> spiv: oh, I missed the ref.
<lifeless> spiv: 5:45 starts on thursday ;)
<wgrant> I think I may be too young for this.
<jcsackett> wgrant: http://en.wikipedia.org/wiki/Kris_Kross
<wgrant> Ah, yes, 1992.
<spiv> wgrant: http://www.youtube.com/watch?v=SmO4xdnf6Gk, especially around the 40 second mark
<huwshimi> wgrant: That was the year you were born, right?
<wgrant> huwshimi: '91.
<huwshimi> wgrant: oh, my apologies :D
<wgrant> I am at least a few months older than this... abomination.
<lifeless> no excuses then
<huwshimi> wgrant: That would only make you a little younger than they where when the song came out
<lifeless> wow
<lifeless> I'm not an apple afficiondo
<lifeless> but this is sweet hardware: http://www.apple.com/ipad/features/
<wgrant> Yes, even I have to admit that :(
 * huwshimi is wondering if Ubuntu will work on his released-last-week MacBook Pro
 * huwshimi is about to find out
<lifeless> bah
<lifeless> how is %foo% meant to be escaped for storm
<lifeless>   File "/home/robertc/launchpad/lp-sourcedeps/eggs/storm-0.18-py2.6-linux-x86_64.egg/storm/database.py", line 366, in _check_disconnect
<lifeless>     return function(*args, **kwargs)
<lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/testing/pgsql.py", line 115, in execute
<lifeless>     return self.real_cursor.execute(*args, **kwargs)
<lifeless> ("SELECT COUNT(*) FROM SourcePackageName, SourcePackagePublishingHistory, SourcePackageRelease WHERE SourcePackagePublishingHistory.archive = %s AND SourcePackagePublishingHistory.sourcepackagerelease = SourcePackageRelease.id AND SourcePackageRelease.sourcepackagename = SourcePackageName.id AND (SourcePackageName.name LIKE '%' || 'cd' || '%')", (9,))
<lifeless> IndexError: tuple index out of range
<lifeless> argh, I'm blind
<lifeless> sqlobject vs storm difference on % escaping. Baby saviour wept.
<wgrant> Are you blind for not seeing it, or blind because you saw it?
<lifeless> blind having fixed it
<lifeless> " '%%%%' || %s '%%%%'" % quote_like(name)
<lifeless> damned if I know why the change was needed
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project windmill build #6: FIXED in 49 min: https://hudson.wedontsleep.org/job/windmill/6/
<lifeless> oh fail
<lifeless> julian used bool(resultset) all over with getPublishedSources
<lifeless> wgrant: pop quiz
<lifeless> binary_copies)                                                                                       |            clauses.append(
<lifeless> lib/lp/soyuz/scripts/packagecopier.py line 564
<lifeless> would .one() or .first() be appropriate?
<wgrant> lifeless: Which line?
<wgrant>         source_copy = source_in_destination[0]
<wgrant> ?
<lifeless> yes
<wgrant> first
<wgrant> one will cras
<wgrant> +h
<lifeless> is a failover server?
<lifeless> \o/ test passes
<wgrant> Yay, appservers updated.
<lifeless> hah, and a trivial query reduction in distroseriesdifference.py
<wgrant> Oh?
<lifeless> *sql is not python*
<lifeless> second hunk
<lifeless> http://pastebin.com/0ijzdQjD
<wgrant> Ah
<lifeless> also in findSourcePackageRelease
<lifeless> same bug
<lifeless> and getSourceAncestry
<lifeless> also know as LBYL is slow
<lifeless> actually,
<lifeless> -        if pubs:
<lifeless> +        try:
<lifeless>              return pubs[0]
<lifeless> -        else:
<lifeless> +        except IndexError:
<lifeless> is cleaner
<wallyworld> abentley: you still there?
#launchpad-dev 2011-03-03
<thumper> https://code.launchpad.net/~thumper/launchpad/epic-email-bug-description/+merge/51977
<thumper> wallyworld: abentley won't still be here
* thumper changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<abentley> wallyworld: not really :-)
<wallyworld> abentley: i was hoping for a quick +1 on the recipe build perf mp after i fixed the test :-)
<lifeless> wgrant: have a look at the same urls in https://bugs.launchpad.net/launchpad/+bug/711064
<_mup_> Bug #711064: POFile:+translate timeouts <qa-ok> <timeout> <Launchpad itself:Fix Released by lifeless> < https://launchpad.net/bugs/711064 >
<lifeless> wgrant: (click through I mean) - quite a bit snappier :)
<wallyworld> abentley:  i take your point about deleted recipes but the page was only designed to show builds with recipes and this mp is a perf tweak. extending what the page is designed to do can be done separately :-)
<abentley> wallyworld: It doesn't matter what the page is designed to do, because this is model code.
<wallyworld> the page will never show builds without recipes due to how the query is structured
<lifeless> abentley: I thought it was view vode
<abentley> wallyworld: anyone can see it and reasonably assume that it is safe to call.
<wallyworld> ie select stuff from recipe join build .....
<abentley> lifeless: no, model.
<wallyworld> the recipe will never be None on the page
<abentley> wallyworld: It does not matter whether the recipe will ever be None on the page.
<wallyworld> so it is always safe to call
<wallyworld> sorry, i'm missing something i think
<abentley> wallyworld: it is not safe to call if the recipe is None, and there's no reason to believe that it will be non-None for an arbitrary caller.
<wallyworld> but it isn't a view off of a recipe
<wallyworld> it is a view off ILanchpadRoot which runs a query and displays some data
<wgrant> lifeless: Still 4s :(
<lifeless> wgrant: down from ... 12
<wallyworld> and that data will only ever contain valid recipes and associated build info
<abentley> wallyworld: It looks very much like you've added two properties to RecipeBuild.
<wallyworld> yes, and the recipe there will never be None
<abentley> wallyworld: is that not the case?
<abentley> wallyworld: Yes, it will sometimes be none.
<wallyworld> how?
<wallyworld> the query is guaranteed to return records with recipes
<abentley> wallyworld: What query?  The query is not connected to RecipeBuild.
<wgrant> lifeless: :(
<abentley> wallyworld: It is fully possible for an arbitrary caller to have a RecipeBuild with a None recipe, and to access recipe_name.
<wallyworld> abentley: the query which is used to implement findCompletedDailyBuilds()
<wallyworld> but i guess if they are paging through the results, the recipe could be deleted after the page was first rendered
<abentley> wallyworld: You have no guarantee that the RecipeBuild was returned by that methode.
<abentley> wallyworld: Arbitrary code in Launchpad can access RecipeBuilds and then access recipe_name.
<wallyworld> abentley: the RecipeBuildRecord used in the collection backing the view is constrcuted directly from the result set
<wallyworld> the RecipeBuildRecord is a named tuple
<wallyworld> with data needed by the view shoved into it
<abentley> wallyworld: so you're saying that even though this code lives in model, it's actually view code?
<wallyworld> well, its code which queries the model and returns some data
<wallyworld> the data is currently used by a view but it could be used elsewhere too
<wallyworld> the main class is RecipeBuildRecordSet
<wallyworld> there's also a BugTaskSet for example that is model code
<wallyworld> the rendering of the data is done in the view layer
<abentley> wallyworld: so my main source of confusion is that I thought we were talking about SourcePackageRecipeBuild, not RecipeBuildRecord, which I don't think I've seen before.
<wallyworld> ah ok. the number of times i've be confused between SourcePackageRecipeXXXX, SourcePackageReleaseXXX is too many to mention. so many similar names
<wallyworld> etc
<abentley> wallyworld: well, it's complicated by the fact that SourcePackageRecipeBuild and RecipeBuildRecord seem to refer to exactly the same thing.
<wallyworld> not exactly :-)
<abentley> wallyworld: How not?
<wallyworld> the later is akin to a denormalised chunk of data, the other is a straight model object
<abentley> wallyworld: but they both refer to the same thing, and you can query either to get the same data.
<wallyworld> model object in this case = persistent database object perhaps
<wallyworld> i would have to take a closer look, but can SourcePackageRecipeBuild directly provide the SourcePackageName, the Archive, the lastest build time etc?
<abentley> wallyworld: You can get those from SPRB.source_package_release.
<wallyworld> but not without navigating the object model
<lifeless> onely one page of getPublishedSources references to update
<lifeless> ><
<wallyworld> SourcePackageRecipeBuild is designed to package all required data into a readonly single object, a snapshot of the system state if you like, for easy access
<wallyworld> to fit the use case
<wallyworld> i meant RecipeBuiildRecord
<wallyworld> not SourcePackageRecipeBuild
<abentley> wallyworld: I get that you have easier, more efficient access to that data.
<abentley> wallyworld: I still think the RecipeBuildRecord refers primarily to a SourcePackageRecipeBuild.
<abentley> wallyworld: And the rest is just precaching.
<wallyworld> and it comes straight off the query so it can be got with fewer object instantiations, so more efficient
<lifeless> wgrant: another in _collectLatestPublishedSources
<abentley> wallyworld: I wasn't talking about whether it was efficient or direct.
<abentley> wallyworld: I was talking about what it referred to.
<wallyworld> ah ok
<wallyworld> so given the scope of the mp, ie a perf tweak, you're happy to +1? :-)
<abentley> wallyworld: I will reluctantly +1 it.  I don't see why you're leaving booby-traps for people who extend it later.
<abentley> wallyworld: When the fix is a two-line conditional.
<wallyworld> i don't get what the booby traps are. it's not that uncommon not to pass around entire persistent objects / model objects and instead construct objects more suited to a given use case?
<wallyworld> but the conditional is not required
<wallyworld> i can add it though
<lifeless> wallyworld: AIUI you're saying that this memo object is never constructed without a recipe
<LPCIBot> Project db-devel build #413: STILL FAILING in 4 hr 50 min: https://hudson.wedontsleep.org/job/db-devel/413/
<wallyworld> yes
<abentley> wallyworld: the booby-trap is that if someone ever constructs a RecipeBuildRecord whose SourcePackageRecipeBuild.recipe is None, then recipe_name will raise an AttributeError.
<lifeless> wallyworld: and that someone constructing it without a recipe would be changing its contract, so there will be other changes they have to make.
<wallyworld> since it is constructed in the resultset decorator
<wgrant> StevenK: If you see any bug comment spam, you can now hide it yourself.
<lifeless> wallyworld: I have no particular opinion here, just trying to make sure I understand
<wallyworld> the RecipeBuildRecords are not meant to be constructed with null recipes
<abentley> wallyworld: If that is the contract, then you could put it in the constructor.
<wallyworld> yes, i could an an assert
<wallyworld> add
<StevenK> wgrant: I didn't think that rolled out yet
<wallyworld> hopefully namedtuples let me do that
<abentley> wallyworld: that would make sense to me.  From looking at that object, I see nothing to suggest such a contract.
<wallyworld> fair call
<wgrant> StevenK: Just in the last hour or so.
<wallyworld> i think doing that will remove a lot of the confusion
<wallyworld> i guess from my perspective, i never expected one would want or need to construct the object without a recipe
<wallyworld> lifeless: you had a question?
<abentley> wallyworld: I expect we'll want to construct it with arbitrary SourcePackageRecipeBuilds before long.
<abentley> wallyworld: and that would include ones without a recipe.
<StevenK> wallyworld: Looking at production data, there are many SourcePackageRecipeBuilds where recipe is None, because the user has deleted the recipe.
<wallyworld> abentley: at the moment, it's only constrcuted from the resultset decorator, but that could change
<lifeless> wallyworld: no, I asked questions to establish a clear picture of the debate
<abentley> wallyworld: right.
<wallyworld> StevenK: but those records will never be returned due to how the query is constructed
<wallyworld> lifeless: oh, i didn't see your questions
<lifeless> wallyworld: you answered them :)
<StevenK> wallyworld: I am +0 on adding *another* class for Recipe builds
<StevenK> This strikes me as needless duplication
<abentley> StevenK: I don't think anyone is suggesting that.
<wallyworld> lifeless: ok. hopefully you were satisfied with the answers
<StevenK> abentley: Then I apologise, since I've obviously read what wallyworld is doing wrong.
<abentley> StevenK: wallyworld is just extending the existing RecipeBuildRecord class.
<StevenK> wgrant: Can haz howto?
<wallyworld> StevenK: it's not duplication since it's not a model object - it's akin to a DTO to package some data needed for a given use case
<wgrant> StevenK: lp.bugs[1234].setCommentVisiblity(comment_number=2, visible=False)
<wgrant> With Visibility spelt properly.
<StevenK> wgrant: I can't do it via the web UI?
<wgrant> StevenK: No UI.
<wgrant> But it's a trivial API script.
<StevenK> Bah, fail.
<StevenK> You'd think the API script would be on the wiki somewhere ...
<wallyworld> abentley: thanks for the input. i think we managed to reach an acceptable solution once i add the asserts, or maybe i'll just cater for null recipe if we think that's a future possibility, but that will mean changing the page template too perhaps
<wgrant> StevenK: It is two lines...
<wgrant> StevenK: I'm not sure that anybody has bothered to publish it.
<lifeless> I can has OCR? https://code.launchpad.net/~lifeless/launchpad/bug-727560/+merge/51989
<wallyworld> lifeless: seems there's no OCR but i can have a go if you like
<wallyworld> or you may prefer a proper review first up
<lifeless> wallyworld: feel free; we can tap thumper to mentor ;)
<thumper> ?
<thumper> lifeless: https://code.launchpad.net/~thumper/launchpad/epic-email-bug-description/+merge/51977
<lifeless> thumper: I have a moderate size, 90% mechanical because of stormification branch needing review
<lifeless> thumper: on yours now
<lifeless> typo
<lifeless> lots *of* text to add
<wallyworld> thumper: i was going to have a go since no one was around but if you want it....
<lifeless> wgrant: want to review thumpers, then I'll mentor?
<lifeless> wgrant: I see 2 things.
<thumper> wallyworld: no, you take it
<wallyworld> thumper: great. i'd like the practice
<wgrant> lifeless: Sure, looking.
 * wallyworld quickly finishes daily recipe build perf fix first so ec2 can chew on it
<thumper> :-(((
<StevenK> wgrant: Right, all done. Six comments hiden
<StevenK> *hidden
<wgrant> StevenK: Thanks.
<thumper> time out proposing a review
<StevenK> Whee
<wgrant> thumper: Was it scanning?
<thumper> possibly
<thumper> https://code.launchpad.net/~thumper/launchpad/process-email-oops-on-person-adaptation/+merge/51992
<wgrant> thumper, lifeless: Reviewed the email size thing.
<thumper> wgrant: ta
<lifeless> wgrant: thumper: mentored
<lifeless> wgrant: want to mentor https://code.launchpad.net/~thumper/launchpad/process-email-oops-on-person-adaptation/+merge/51992 ?
<thumper> lifeless: ok
<thumper> d'uh
<thumper> me didn't read
<wgrant> lifeless: Looking.
<wgrant> lifeless: Done.
<lifeless> thumper: second one reviewed.
<wgrant> Thanks.
 * thumper relocates
<wallyworld> lifeless: wondering why candidates[0] and not candidates.first()
<wallyworld> or are they the same thing in terms of efficiency
<lifeless> wallyworld: .first() is clearer, they should be equivalent for efficiency. Without reading the code I am not /sure/ that .first() raises IndexError, so I left some [0] around for that case.
<lifeless> wallyworld: also some tests were going to be less clear with .first() so I left those alone too
<wallyworld> lifeless: ok. i assume that because only Archive.getPublishedSources() was change and not the others, the the 2 resultset types are interchangable?
<wallyworld> forgive my storm ignorance
<lifeless> wallyworld: which 2 resultset types?
<lifeless> wallyworld: (there are, uhm, 4 or so around)
<lifeless> wallyworld: and interchangable /where/
<wallyworld> SQLObjectResultSet and DecorateResultSet
<lifeless> they are not interchangable and not compatible
<lifeless> thats why I stormified getPublishedSources so that I could use DRS
<wallyworld> ok. i saw a few getPublishedSources() methods and initially thought they were off the one interface but not so
<wallyworld> so my question doesn't make sense anymore
<lifeless> its entirely possible I have some things in this that ec2 will highlight as faults
<lifeless> but I think I identified the types correctly
<wallyworld> yeah, i suspect i was seeing an issue that's not there. hard to tell sometimes from looking at diffs
<wallyworld> storm sure seems better than what it replaced, looking at the diffs - less string literals and more object based approach
<lifeless> how do you make a scanned branch in a test ?
<lifeless> \o/
<wgrant>  * 3472 Exceptions
<lifeless> flat cunt for timeouts
<lifeless> *count*
<wallyworld> lifeless: keep it clean :-)
<lifeless> nice drop in bugtask:+index timeouts
<wgrant> We should be below 2000 excepts per day tomorrow.
<wallyworld> wgrant: what was the peak value before the tuning work started in earnest?
<wgrant> wallyworld: I don't really know. But a few days ago it was 23000.
<wallyworld> !!!!
<wgrant> 16000 of those were that loggerhead thing, though.
<wgrant> So it's sort of cheating.
<wallyworld> still 7000 (or anything with that many zeros at the end) is too many
<wallyworld> except if its your bank balance - then too many zeros are never enough :-)
 * wallyworld needs lunch
<lifeless> thumper: hey
<StevenK> wgrant: Test failures with my TAL branch, do agree with http://pastebin.ubuntu.com/574779/
<wgrant> StevenK: Looks reasonable.
<StevenK> And the tests pass, so I'll toss it at ec2 again
<wgrant> ec2 is only 3.5 hours now.
<wgrant> It's nice.
<StevenK> Heh, yes
<thumper> lifeless: yus?
<lifeless> thumper: I have a problem
<thumper> ok
<lifeless> thumper: I https://bugs.launchpad.net/launchpad/+bug/722956
<_mup_> Bug #722956: BranchSet:CollectionResource#branches timeouts <bad-commit-12489> <bad-commit-12505> <qa-bad> <timeout> <Launchpad itself:In Progress by lifeless> < https://launchpad.net/bugs/722956 >
<lifeless> thumper: I had a branch that eager loads that
<lifeless> thumper: but it failed on qastaging
<lifeless> thumper: i fixed a typo, but then it failed again with a TypeError doing some adaption
<lifeless> thumper: IIRC the error properly, it looked like a source package series branch
<lifeless> thumper: but I don't know how to set one up in the test suite
<lifeless>     
<lifeless>     def test_renders_with_source_package_branch(self):
<lifeless>         source_package = self.factory.makeSourcePackage()
<lifeless>         branch = self.factory.makeBranch(sourcepackage=source_package)
<lifeless>         branch.updateScannedDetails(None, 0)
<lifeless>         logout()
<lifeless>         lp = launchpadlib_for("test")
<lifeless>         list(lp.branches)
<lifeless> is what I have
<lifeless> but I suspect its not a 'source package series branch'
<lifeless> because the url is a long one with ~user in it
<wgrant>    3 TypeError: ('Could not adapt', <lp.code.model.seriessourcepackagebranch.SeriesSourcePackageBranch object at INSTANCE-ID>, <InterfaceClass lp.code.interfaces.linkedbranch.ICanHasLinkedBranch>)
<thumper> lifeless: a package branch is one whose target is a distro series source package
<lifeless> thumper: yup, I've got that far
<lifeless> thumper: then there is the *SeriesSourcePackageBranch* which is the blessed ones
<thumper> lifeless: also, use factory.makePackageBranch
<thumper> right
<lifeless> how do I get from what I have to the blessed one
<thumper> off the top of my head...
<thumper> something like
<thumper> ICanHasLinkedBranch(branch.target).branch = branch
<thumper> ICanHasLinkedBranch(naked_product).setBranch(branch)
<thumper> copied from a test
<thumper> lifeless: lib/lp/code/model/tests/test_branch.py:803:
<thumper> lifeless: is one way
<thumper> lib/lp/code/model/tests/test_linkedbranch.py
<lifeless> looking, thanks
<lifeless> thumper: thanks!
<lifeless> thumper: have reproduced the boom
<lifeless> oh, no, I haven't
<lifeless> I made the test do it instead ><
<lifeless> thumper: thanks, *now* I have the failure reproduced.
<thumper> cook
<thumper> cool
<lifeless> its a bit messy - I copied all of lib/lp/code/model/tests/test_branch.py:803: basically, but thats tolerable for now
<lifeless> ok, fixed
<lifeless> lp-land time.
<thumper> wgrant: you were right, I was checking the entire email message
<thumper> wgrant: I've fixed it, and pushing now
<wgrant> thumper: Great, will look in a sec.
<lifeless> thumper: can you mentor https://code.launchpad.net/~lifeless/launchpad/bug-727560/+merge/51989 please?
<wgrant> thumper: Why do you need ZopelessAppServerLayer?
<wgrant> That actually starts up an appserver process, IIRC.
<thumper> wgrant: as opposed to?
<thumper> the content gets stored in the librarian
<wgrant> Plain LaunchpadZopelessLayer.
<thumper> the email bits that is
<thumper> the script runs in a zopeless environment
<thumper> so the test should mirror that IMO
<lifeless> LaunchpadZopelessLayer mirrors that
<thumper> to be honest, I copied the layer from the TestCodeHandler test
<lifeless> the appserver layers are IMNSHO hideous
<thumper> I'm not entirely sure
<wgrant> AppServerLayer is a bug.
<thumper> I could try it with the LaunchpadZopelessLayer
<lifeless> thumper: I suggest you try LaunchpadZopelssLayer
<lifeless> jinx
<wgrant> LaunchpadZopelessLayer provides a librarian.
<thumper> so... what does the ZopelessAppServerLayer provide?
<wgrant> An appserver in a separate process.
<lifeless> a running zope instance in a subprocess
<lifeless> it was added for mailman
<lifeless> and should be nuked from orbit along with the way we intgrate mailman
<wgrant> Most stuff using it is probably a bug.
<thumper> hmm...
<thumper> needed for the code handler :(
<wgrant> Oh?
<thumper> TestCodeHandlerProcessMergeDirective
<thumper> don't ask me why
<thumper> I'm not sure
<wgrant> It doesn't need it.
<wgrant> It may use it.
<wgrant> But it doesn't need it.
<thumper> if fails without it
<thumper> and passes with it
<wgrant> That may just mean that the test is broken.
<thumper> :)
<thumper> changed the layer
<thumper> and pushed
<wgrant> Thanks.
<StevenK> lifeless: I spoke to Barry about Mailman 3 at the distro sprint. He said he's close, but there's two things that need fixing.
<lifeless> StevenK: I know
<lifeless> StevenK: its one of those things that will happen in 2013
 * lifeless sadly isn't kidding
<StevenK> That makes me sad. :-(
<lifeless> its not on our todo list for -ages-
<lifeless> and barry has not got time to do a major overhaul in lp space
<lifeless> good intentions aside
<wallyworld> lifeless: in your mp, why the need for 'distroseriesID = Attribute("DB ID for distroseries.")'. i thought storm did that automagically?
<lifeless> wallyworld: thats on the interface.
<lifeless> wallyworld: if its not on the interface security proxies kick you out
<wallyworld> ah, yes. thanks
<StevenK> lifeless: Well, he also said the two things are broken in core, which means they might get fixed soon. :-)
<lifeless> thumper: you need to click 'approve' as well
<thumper> lifeless: it isn't approved
<thumper> lifeless: wallyworld said needs info
<lifeless> thumper: ian approved it
 * thumper looks again
<thumper> I bet he added comments while I had it open
<thumper> where's our long-poll
<wallyworld> and optimistic locking :-)
<lifeless> thumper: thanks!
<huwshimi> Is it just me or are a bunch of Canonical's websites down?
<StevenK> Such as?
<huwshimi> StevenK: Well Everything. I can't access the irc server, ubuntu.com, canonical.com etc.
<wgrant> All good for me.
<wgrant> What does traceroute say?
<StevenK> And me.
<huwshimi> Traceroute errored after one of internode's servers, but I seem to be connecting again now
<StevenK> I suspect a route failed over or so
<huwshimi> oh well as long as it was just me
<huwshimi> still flaky though
<wgrant> huwshimi: I've just started getting ~50% packet loss in the US.
<StevenK> To which part of the US?
<StevenK> My route to Fremont is fine.
<huwshimi> wgant: which isp are you with?
<wgrant> StevenK: Within L3 in San Jose.
<wgrant> huwshimi: Optus.
<wgrant> StevenK: My route to Fremont is fine too.
<wgrant> Doesn't go through L3.
<wgrant> Oddly.
<StevenK> I hit Exetel, Alter, and then my server
<wgrant> I have just one (anonymous) hop between Optus and HE.
<StevenK> I have one anonymous hop, but it's Telstra's DSL bridges that they like hiding
<lifeless> I'm well and truely sick of hakcing on BugTask:+index
<lifeless> won't another page please start timing out massively. I need a change.
<lifeless> wallyworld: I thought you were nuking '155 BugWatchUpdateWarning'
<lifeless> ExternalBugtracker for BugTrackerType 'GOOGLE_CODE' is not known
<wallyworld> lifeless: i've no idea what you are referring to :-)
<lifeless> bach
<lifeless> wgrant: ^
<wgrant> lifeless: That was me.
<wgrant> But it requires checkwatches.txt to be redone.
<wgrant> It is the hardest OOPS to get rid of.
<wgrant> So I did the others first.
<lifeless> wgrant: the doctest ?
<wallyworld> i can see how you keep getting us confused, but for future reference, i'm the good looking one
<wgrant> lifeless: Yes.
<lifeless> wallyworld: its all about the tabs
<lifeless> wallyworld: if I fumble the g, it tab completes to you
<lifeless> wgrant: my sympathies
<wallyworld> yeah i know, i was just messing with you
<lifeless> wgrant: if you need a tactical nuke, let me know
<wgrant> lifeless: We can probably drop that bit of the doctest, but it may be the only test that running the script logs OOPSes properly.
<wgrant> I will hopefully get back to that next week.
<wgrant> There are what.. 300-400 of them a day?
<lifeless> yeah 10% of remaining daily exceptions
<wgrant> Note that 1500 of yesterday's exceptions are already fixed.
<wgrant> So we are down to ~2000.
<lifeless> wgrant: -lovely-
<wgrant> So it's more!
<lifeless> hmm
<lifeless> I find the new style strains my eyes :(
<wgrant> I think body and comment text is slightly too small.
<lifeless> they are to faint
<wgrant> I've had a couple of friends say the same thing.
<lifeless> too
<lifeless> wgrant: hmm
<wgrant> Hmm?
<lifeless> wgrant: reckon you could run dogfood with sql backtraces enabled
<lifeless> and find where the milestone lookups in https://lp-oops.canonical.com/oops.py/?oopsid=1888C48#repeatedstatements are being triggered ?
<wgrant> lifeless: Sure.
<lifeless> ditto the distribution one
<lifeless> thats another second if we fix both of those.
<lifeless> https://bugs.launchpad.net/launchpad/+bug/724033 to record what you find; I'll work up fixes.
<_mup_> Bug #724033: BugTask:+index timeouts - distributions and milestones being late evaluation loaded - repeatedly - on bug 230350 <qa-ok> <timeout> <Launchpad itself:In Progress by lifeless> < https://launchpad.net/bugs/724033 >
<wgrant> lifeless: Could you run http://paste.ubuntu.com/574807/ on staging?
<wgrant> I'm merging the tracebacks branch.
<lifeless> qas or s?
<lifeless> wgrant: ^
<wgrant> lifeless: Whichever has the newer DB.
<lifeless> $meh
<wgrant> Doesn't really matter.
<lifeless> its running
<wgrant> Thanks.
<wgrant> Shouldn't take too long.
<lifeless> you want the output, or timing ?
<wgrant> Output please.
<wgrant> Shouldn't be much.
<wgrant> I hope.
<lifeless> will tell you when it finishes
<wgrant> It only takes a couple of minutes on DF...
<lifeless> https://pastebin.canonical.com/44198/
<wgrant> Thanks.
<wgrant> lifeless: DF running with backtraced OOPSes.
<wgrant> lifeless: It may time out too early, but we can just bump the limit I guess.
<lifeless> hit up https://bugs.launchpad.net/ubuntu/+source/afflib/+bug/230350/+index on it
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<wgrant> Oh, that one, right.
<lifeless> (you're logged in)
<lifeless> any my show-entire-profile is landed so I can get useful data from bug 1
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<wgrant> lifeless: It timed out too early the first time. Trying some stuff.
<wgrant> Got it.
<lifeless> cool
<wgrant> The OOPS is large, and bzip2 is slow.
<lifeless> wgrant: thanks
<lifeless>          738            0      0.6724      0.0146   canonical.launchpad.utilities.celebrities:75(__get__)
<lifeless>         +738            0      0.2368      0.0034   +lp.registry.model.person:3369(get)
<lifeless> :/
<wallyworld> if an attribute is defined to be a CollectionField but there no exported() wrapping the declaration, does that mean it isn't exported to the webservice?
<wallyworld> and hence i can delete it?
<wgrant> That's right.
<wallyworld> \o/
<wgrant> Which?
<wallyworld> related_bugs on branchmergeproposal
<wallyworld> it's being replaced by getRelatedBugTasks
<wallyworld> need to pass in a user so that only the correct visible bug tasks are returned
<wgrant> Indeed, not exported.
<wallyworld> cool. excellent.
<lifeless> \o/
<lifeless> I am going to hold a freaking party when we have a 0 oops day
 * wallyworld has already removed DecoratedBug and linked_bugs and other stuff. need to start new pipe to do the last bit
<wgrant> lifeless: Did you find anything else useful in that OOPS?
<lifeless> wgrant: I
<wallyworld> lifeless: how many oopses today then?
<lifeless> 've just got back
<lifeless> wallyworld: read the summary :>
<lifeless> ~ 15K
<wallyworld> oh, didn't see it, wndow too small
<wgrant> lifeless: A zero OOPS day is actually not too implausible.
<wgrant> We have lots of OOPSes remaining.
<wgrant> But most of them are seen very rarely.
<wgrant> Zero exceptions could be doable in the next two months.
<lifeless> fixing bugs has the impact of fixing bugs.
<wgrant> The exception report will be fairly sensible tomorrow.
<wgrant> With maybe 10 distinct checkwatches exceptions.
<lifeless> curtis is on a rampage
<wgrant> Yes.
<lifeless> http://webnumbr.com/launchpad-oops-bugs.all
<wgrant> Ja, saw that this morning.
<wgrant> It's encouraging.
<spiv> lifeless: here, let me fix that Y axis for you: http://webnumbr.com/.join%28bzr-pqm-queue-length,launchpad-oops-bugs%29 ;)
<poolie> :)
<lifeless> spiv: lol
<StevenK> Bwahaha
<lifeless> spiv: still looks nice.
<spiv> lifeless: it does!
<lifeless> wgrant: distros are due to the vocabulary too
<wgrant> lifeless: Aha.
<wgrant> So that's the two big repetitions fixed.
<lifeless> its eager loading (something I added)
<lifeless> wgrant: identified.
<wgrant> Shhhh.
<lifeless> Fixing takes a little longer
<lifeless> because a new object is invoked per row.
<wgrant> Ah.
<wgrant> Sad.
<lifeless> so I need to:
<lifeless>  - eager load from the outer view class
<lifeless>  - inject into the bugtasks
<lifeless>  - use the resulting attribute
<lifeless> wgrant: I'm going to wag that user_can_edit_milestone is the cause of the person lookups
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #414: FIXED in 5 hr 19 min: https://hudson.wedontsleep.org/job/db-devel/414/
<wgrant> That probably means that devel is about to fail.
<StevenK> :-(
<lifeless> its not
<wgrant> lifeless: For authenticated requests it is mostly permission lookups in there, yes.
<StevenK> devel just started a build, though
<wgrant> Oh.
<wgrant> The bad builder must have been culled at some point.
<wgrant> I didn't think there was enough idle time.
<StevenK> wgrant: Did you see Jenkins last night, building devel, db-devel and windmill at the same time?
<wgrant> StevenK: Yes.
<wgrant> This morning it did that too.
<StevenK> Now if only parellel-test passed
<wgrant> Heh.
<StevenK> Executing star-merge bzr+ssh://bazaar.launchpad.net/~thumper/launchpad/inline-recipe-name-edit at Thu Mar  3 03:46:22 2011
<StevenK> bzr: ERROR: no such option: -0
<StevenK> wgrant: ^
<wgrant> Ugh. But it's in the check that lifeless wanted to delete anyway.
<wgrant> old bzr is old.
<StevenK> I'm not used to PQM being this quick!
<StevenK> 2 minutes from EC2 shutting down until PQM said sucess
<wgrant> Yes.
<wgrant> And with ec2 an hour quicker....
<StevenK> Remove the doctest layer too
<wgrant> And also ec2 and buildbot reliable.
<wgrant> StevenK: Why are Hudson's test durations all 0? Does the subunit -> JUnit XML converter not bring that across?
<wgrant> s/Hudson/Jenkins, your domain is confusing/
<lifeless> probably old subunit
<lifeless> it had a bug
<StevenK> wgrant: I've been meaning to rename it
<StevenK> lpci.w.o or so
<StevenK> In fact, my plan for that RT is we grow ci.l.n
<StevenK> But I've not told lifeless of that plan, so I may get veto'd
<wgrant> Under c.c, maybe.
<lifeless> we shouldn't put things under the lp domain
<lifeless> unless they are part of the lp offering
<wgrant> But anything, even QA environments, under l.n is a bug.
<StevenK> So lpci.c.c works?
<lifeless> fbm
<StevenK> But pqm is under l.n :-P
<wgrant> that is a bug.
<lifeless> or reuse either the pqm or buildbot sites so we don't have to buy another ssl cert
<wgrant> lifeless: We have a *.canonical.com
<lifeless> by reuse I mean proxypass a subpart, of course
<wgrant> Most stuff uses that.
<lifeless> wgrant: ah, I wasn't aware it was wildcarded
<StevenK> I also dislike the buildbot name
<wgrant> eg lpbuildbot uses it.
<StevenK> I'd also like it to be open, but that's a whole other argument
<wgrant> If we can tell production-devel to permanently die, it can remain public.
<StevenK> production-devel hasn't been used in what, 4 months? 5?
<lifeless> thats not a good indicator of need
<lifeless> [given the use case for it]
<StevenK> I don't think the branch is even updating anymore
<lifeless> its not
<StevenK> So we can't use it anyway?
<lifeless> explictly updated as needed
<StevenK> I'd rather it just die
<wgrant> Everyone would.
<wgrant> But there are still use cases.
<StevenK> Which are security and what else?
<wgrant> Nothing.
<wgrant> Or a possibly super-urgent non-security fix, I guess.
<StevenK> So, we cowboy a fix and then request a deploy over the top?
<StevenK> (When it's landed and QA'd)
<wgrant> That is what would have to replace it.
<lifeless> ok, tracked down the person lookups
<wgrant> lifeless: In the OOPS?
<StevenK> wgrant: Oh, so it doesn't sound like crack? :-)
<wgrant> Because the set on dev may be different.
<lifeless> wgrant: its clear as daylight
<wgrant> lifeless: Oh?
<lifeless> and easy to fix too
<lifeless> https://bugs.launchpad.net/launchpad/+bug/726370/comments/3
<_mup_> Bug #726370: BugTask:+index timeout - late evaluation of Person/ValidPersonCache <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726370 >
<wgrant> Ahh
<wgrant> That's not all of them, though.
<wgrant> It will be most from that particular bug, I guess.
<lifeless> it will be a big chunk
<lifeless> up to 2 per row
<wgrant> Right.
<wgrant> Won't help bug #1 much at all, though.
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
 * StevenK wishes for bugtask deletion/hiding more
<StevenK> RARGH, stop rebuilding the WADL!
<wgrant> Why?
<StevenK> wgrant: Because Bug 1 is drowning in them
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
<StevenK> Damn it
<wgrant> Heh.
<StevenK> WTB better Makefile, PST
<lifeless> right, I think thats fixed
<lifeless> quick smoke test
<StevenK> In [12]: c.get_versions()
<StevenK> Out[12]: []
<wgrant> OK, looks like I have killed 1/3 of the copying queries.
 * StevenK pouts at debian.changelog
<wgrant> Oh?
<lifeless> wgrant: have you used the DistroSeriesSet.getCurrentSourceReleases optimisation yet ?
<wgrant> lifeless: No.
<wgrant> lifeless: Not relevant here, is it?
<wgrant> (do you want me to test your fix on DF?)
<wgrant> actually, it's probably doing enough at the moment.
<wgrant> Half way there.
<lifeless> lp:~lifeless/launchpad/bug-726370 rev 12518
<lifeless> wgrant: if you wanted to see if that helps,t hat would be great
<lifeless> wgrant: the other thing to do is to break into pdb in https://bugs.launchpad.net/launchpad/+bug/726370/comments/4 and get the template details for that event
<_mup_> Bug #726370: BugTask:+index timeout - late evaluation of Person/ValidPersonCache <timeout> <Launchpad itself:In Progress by lifeless> < https://launchpad.net/bugs/726370 >
<wgrant> lifeless: Can't right now.
<wgrant> The librarian is getting a bit of a workout.
<lifeless> yeah
<lifeless> queuing it up
<wgrant> lifeless: That's probably an email address in a message.
<wgrant> Yeah, there are a few of them.
<wgrant> https://bugs.launchpad.net/ubuntu/+source/afflib/+bug/230350/+text
<_mup_> Bug #230350: Missing Debian Maintainer field <Ubuntu:Invalid> <afflib (Ubuntu):Fix Released by saivann> <alac-decoder (Ubuntu):Fix Released by warp10> <axiom (Ubuntu):Invalid> <beneath-a-steel-sky (Ubuntu):Fix Released> <bibletime-i18n (Ubuntu):Fix Released by txwikinger> <binkd (Ubuntu):Fix Released> <bzr-builddeb (Ubuntu):Won't Fix> <capisuite (Ubuntu):Invalid> <chinput (Ubuntu):Fix Released by warp10> <chmsee (Ubuntu):Fix Released> <ciso (U
<lifeless> shhh mup, sssh
<wgrant> At least it got truncated!
 * lifeless gambles that this cannot possibly break tests.
<wgrant> lifeless: ec2 is only 3.5 hours now, but go ahead.
<lifeless> do as I say, not as I do.
<wgrant> We may also need to preload BugActivity actors, but we might already do that.
<lifeless> don't think so
<lifeless> theres something triggering TP lookups too
<wgrant> That's probably permission checking.
<lifeless> I suspect another task row thing
<lifeless> wgrant: [duh] :P
<wgrant> Also most of that here should be archivepermissions.
<wgrant> Yet I don't see many of them.
<wgrant> Ah, of course.
<wgrant> Because I am a component uploader for most of them.
<wgrant> Hm, no, no archivepermission queries at all.
<wgrant> Huh.
<wgrant> Maybe because I'm an admin.
<stub> This must be the first good storm of the year. My office roof is leaks :-P
<lifeless> oh right, its thursday
<StevenK> stub: Your office roof is one big leak?
<lifeless> StevenK: he's hosting a wiki
<StevenK> Haha
<wgrant> lifeless: How aggressively do you tend to test your preloading?
<StevenK>  lp.registry.tests.test_distroseriesdifference.DistroSeriesDifferenceTestCase.test_base_version_multiple
<StevenK>   Ran 1 tests with 0 failures and 0 errors in 1.268 seconds.
<StevenK> SUCCESS!
<wgrant> eg. I've adjusted getBuiltBinaries to optionally grab BPFs and LFAs too.
<StevenK> wgrant: ^ With changelog parsing goodness
<wgrant> StevenK: Excellent!
<stub> Great. My roof leaks and I'm talking tinglish.
<wgrant> StevenK: Do you want to coerce a LOSA into getting the populator running on prod at some point?
<StevenK> Didn't we try that?
<wgrant> Never on prod.
<wgrant> I discussed it with spm, but nothing ever happened.
<StevenK> wgrant: I suspect the best answer is 'file an RT'
<wgrant> Possibly, but it's your squad now :P
<StevenK> I suspect the script has bitrotted some
<wgrant> Shouldn't be much.
<wgrant> I'd be surprised if it was more than import locations.
<lifeless> wgrant: I tend to not want to overlap unit tests
<StevenK> wgrant: And uh, DB user
<lifeless> wgrant: e.g. a single smoke test pushing all the scaling buttons and make sure its flat
 * StevenK prods at it
<wgrant> StevenK: True.
<lifeless> wgrant: depends on the layer
<wgrant> lifeless: They are all doctests at the moment, which I might replace.
<StevenK> lifeless: Thoughts on getting an out-of-tree script run against prod?
<lifeless> what sort of script ?
<wgrant> One-off to populate SPR.changelog for old rows.
<lifeless> lplib? model?
<StevenK> Model
<lifeless> how long will it take to run ?
<lifeless> minutes? hours? days?
<wgrant> That's not entirely clear. Possibly a few days.
<lifeless> in tree, garbo job.
<StevenK> That's a good idea, actually
<wgrant> Easy enough to port the existing one to that.
<StevenK> wgrant: What are you doing to mawson?
<StevenK> Er. Maybe that was me
<wgrant> It's me.
<StevenK> Fess up
<stub> to populate SPR.changelog, do we need to download packages and unpack them? Or pull things from the Librarian?
<wgrant> StevenK: Download and unpack packages, then upload a file to the librarian.
<StevenK> The former
<wgrant> stub: ^^
<wgrant> stub: Download packages from the librarian, that is.
<stub> So garbo can do that, but need to confirm with the losas that it a) has access to the librarian b) there is suitable disk space in $TMP (if suitable might be large).
<lifeless> stub: it runs on loganberry, right ?
<lifeless> stub: main processor runs there IIRC - and it stashes raw mails in the librarian
<wgrant> loganberry was having disk space issues, but I believe they're resolved now.
<wgrant> And the largest package is only a few hundred megabytes.
<stub> Not an ideal fit though since this will almost certainly be running with maxchunksize=1
<wgrant> stub: What is the target transaction length?
<stub> Normally about 4 seconds
<StevenK> I'm not happy with the query wgrant wrote for it, either
<wgrant> StevenK: Oh?
<StevenK> The first query took 65 *minutes*
<wgrant> The new one is fine!
<stub> But in this case that is likely irrelevant - just set the maximum chunk size to 1 and do one at a time.
<lifeless> yup
<lifeless> also set a limit on the query of 5
<lifeless> or 50 or whatever
<StevenK> And make sure it cleans up
<StevenK> Or the LOSAs will be unhappy
<lifeless> naturally :)
<stub> Query to determine work to do takes 65 minutes? better hope that is just a minute or two with a LIMIT or garbo timeouts will kick in and no iterations will ever run.
<StevenK> stub: If you can't see any issues with it, I'll start ripping the script apart
<StevenK> stub: The query has been fixed since then :-)
<stub> ok :)
<stub> Garbo hourly jobs are currently stopped if they run more than 10 minutes. Daily I think can run for 1 hour.
<stub> A stand alone script may well make more sense in this case.
<wgrant> It's already a looptuner.
<lifeless> stub: 15 minutes, not 10.
<wgrant> aaaa PQM Is fast.
<stub> wgrant: It is a few lines to convert that to a stand alone script, which can then run 24x7 until it completes. More work for the losas though.
<lifeless> stub: I suggested garbo
<wgrant> stub: It is *already* a standalone script.
<wgrant> lifeless said garbo.
<lifeless> stub: because it will need less losa work, and thats a massive bottleneck for us right now.
<stub> Right. If the caveats I mentioned don't raise alarm bells, no problem.
<StevenK> I'm happy enough to follow IBug.reindex's lead and do it as a garbo job
<StevenK> steven@liquified:~/launchpad/lp-branches/populate-spr-changelogs% pyflakes lib/lp/scripts/garbo.py | wc -l
<StevenK> 34
 * StevenK sobs
<adeuring> good morning
<maxb> gosh, it's a novelty to see time-based greetings on IRC which actually match your local timezone :-)
 * maxb is on a bus to work
<bigjools> morning all
<wgrant> Morning bigjools.
<lifeless> stub: shall we have a brief call ?
<stub> lifeless: Gimme 2 mins
<lifeless> thumper: ping - care to see if you've fixed bug 604654 in passing ?
<_mup_> Bug #604654: save a roundtrip on POSTs <lp-foundations> <Launchpad itself:Incomplete> < https://launchpad.net/bugs/604654 >
<mrevell> Hello
<lifeless> stub: sure
<stub> lifeless: back
<lifeless> stub: skype?
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> allenap: Morning.
<allenap> wgrant: Morning :)
<wgrant> allenap: Could you please review https://code.launchpad.net/~wgrant/launchpad/bug-728246-copy-expiry-queries/+merge/52020?
<allenap> wgrant: With pleasure.
<wgrant> allenap: Thanks.
<lifeless> stub: https://bugs.launchpad.net/launchpad/+bug/727560 - as an example of a nonbugs query
<_mup_> Bug #727560: Archive:EntryResource:getPublishedSources <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727560 >
<lifeless> stub: 5 seconds to grab the first batch *after* removing unnecessary joins
<stub> lifeless: You are still broken up
<stub> ha
<lifeless> https://launchpad.net/oops-repository
<wgrant> Are recipes for +junk supported?
<wgrant> They broke last week.
<gmb> allenap: Morning. I have a branch up for review if you've got the time: https://code.launchpad.net/~gmb/launchpad/prevent-bnl-nothing-oopses-bug-721400/+merge/52036
<allenap> gmb: Sure :)
<gmb> Ta
<allenap> I'm still on wgrant's right now, but I'll try and get a move on.
<gmb> No massive rush.
<gmb> But puppies die for every half hour that this review takes.
<gmb> And I have a finite supply
<wgrant> adeuring: Hi.
<adeuring> hi wgrant
<wgrant> adeuring: Bug #705176 is blocking rollouts at the moment. It's marked [incr], but qa-tagger wants QA tags and I wonder if there is any QA that can be done?
<_mup_> Bug #705176: Improve sharing information on translation pages <upstream-translations-sharing> <Launchpad itself:In Progress by henninge> < https://launchpad.net/bugs/705176 >
<adeuring> let me check...
<wgrant> adeuring: Thanks.
<allenap> gmb: Line 43, you start a feature_flags() context, but flags are not changed. Is it needed?
<wgrant> allenap: Also thanks.
<gmb> allenap: Yes. If you don't do it in the feature flags context it will pay no attention to the subscriptions bug_notification_level (the field just doesn't exist if malone.advanced-subscriptions.enabled isn't True.
<allenap> wgrant: Sorry I took so long. I have been very distracted this morning talking unittest with one of the server team :)
<wgrant> allenap: No worries.
<allenap> gmb: How does the feature_flags() context know to configure malone.advanced-subscriptions.enabled? Honest question, feature flags still baffle me.
<gmb> allenap: It's added in the test setUp().
<gmb> allenap: The feature_flags() wossname just (re)populates the feature flags stuff; if you don't have that context in place then all feature flags are off.
<allenap> gmb: Right, that's bonkers, but okay :)
<gmb> allenap: A bit, yes. I think it's because they're tied to the request somehow, but frankly it's all a bit voodoo to me.
<gmb> allenap: Ta for the review.
 * gmb lunches
<jam> I just got a bug post spam message. I thought I saw something about making it easier to hide them.
<jam> Can anyone help with: https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/540280/comments/17
<_mup_> Bug #540280: openssh (or gvfs) asks for a password everytime I change directory in sftp <lucid> <regression-release> <bzr-gtk (Ubuntu):Triaged> <gvfs (Ubuntu):Triaged> < https://launchpad.net/bugs/540280 >
<stub> Should database model objects for the session database tables go in lib/lp/services/session/model.py or lib/lp/services/session/model/session.py ?
<stub> or lib/lp/services/model/session.py?
<jam> quick process question, if I want to update a sourcecode dependency, do I just update utilities/sourcecode.conf and run utilities/update-sourcecode? Is there anything else that should be done other than submit that 1-line change for review?
<stub> jam: Think that is it. Maybe let the mailing list know so people think of it when they see the inevitable weird error messages.
<jam> thanks, stub
<danilos> mrevell, hi
<mrevell> hey danilos
<danilos> mrevell, hey-hey
<danilos> mrevell, I want you to do stuff for me â will you? :)
<mrevell> danilos, I will, will. What do you want?
<danilos> mrevell, (I hope you are not going to ask what it is now :)
<danilos> I knew it!
<mrevell> hah
<mrevell> :)
<danilos> mrevell, heh, so, I was going through the mockups on https://dev.launchpad.net/LEP/BetterBugSubscriptionsAndNotifications/Testing/EditingRound2 and gary_poster forwarded me your commentary as well
<danilos> mrevell, however, I found a few other cases which are not covered in the mock-ups that I want to implement, and I am going to be missing the text for them
<danilos> mrevell, I wrote it all down on https://dev.launchpad.net/LEP/BetterBugSubscriptionsAndNotifications/DirectSubscriptionsOnBug
 * mrevell looks
<mrevell> Okay Daniel, how can I help-?
<danilos> mrevell, now, I'd be happy if you can come up with descriptions for every of the "potential actions" in there
<danilos> mrevell, well, both ACTION and DESCRIPTIONs
<danilos> mrevell, no need to do it for those which are already covered in user testing and existing mockups
<danilos> mrevell, (just haven't extracted that yet)
<mrevell> danilos, Where ACTION is the link text and DESCRIPTION is some kind of tool-tip?
<danilos> mrevell, exactly
<mrevell> No problem. When do you need it?
<danilos> mrevell, I think stuff that is missing is mostly team-related and supervisor related
<danilos> mrevell, by yesterday, but tomorrow morning is fine as well :)
<danilos> mrevell, of course, if you are busy, I'll probably come up with stuff myself since I am away next week, and let others fix it :)
<mrevell> Heh. Hmm .... okay, I have a couple of other things to do this afternoon but I can fit this in too. I'll have something with you in the next few hours.
<cr3> hi folks, congrats on taking a whole second off total time of all launchpad pages!
<danilos> cr3, wasn't me, I promise!
<cr3> danilos: I hear it was IS this time around, do you happen to know if it would be possible to know more about what they did to the load balancer?
<danilos> cr3, I believe lifeless and LOSAs know all about it, and that's where the thanks should go
<cr3> danilos: the LOSAs are an elusive bunch though, so I thought of thanking the whole launchpad-dev folks in the hope it would include LOSAs too
<danilos> cr3, heh, I think they don't want to be considered part of LP team because I am part of the LP team: who'd want to be grouped together with me?
<maxb> Speaking thereof, are the LOSAs back from their sprint yet? I've had a fairly trivial LOSA-needs-to-run-stuff question open for ages :-/
<cr3> "I don't care to belong to any club that will have me as a member" -- Groucho Marx
<gary_poster> maxb, it doesn't happen to be a SQL-against-staging thing, does it?
<maxb> no, it's a branch stacking fixup thingy
<maxb> they have a script, they just need to execute it
<gary_poster> ack maxb, can't help then
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap, jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
<adeuring> allenap, jcsackett: could one of you please review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-681434/+merge/52071 ?
<jcsackett> adeuring: i'm looking one MP right now, but it's short. happy to look at that in a moment.
<adeuring> jcsackett: thanks!
<sinzui> allenap: jcsackett: will either you have time to review https://code.launchpad.net/~sinzui/launchpad/lookup-bug-redirect-0/+merge/51978 today
<jcsackett> sinzui: just did it.
<jcsackett> it was the short one previously mentioned. :-)
<sinzui> I see
<sinzui> never mind
<sinzui> thanks
<jcsackett> your welcome. :-)
<deryck> sinzui, hey, I haven't had time to think long about your email due to sprint, but I think your assessment of those rules seems right to me.
<deryck> sinzui, maybe gmb or allenap could comment more fully if you need.
<sinzui> deryck: thanks for reminding me to follow that up
<allenap> deryck, sinzui: Actually, I meant to comment on that already. I'll do that now.
<deryck> awesome.  Thanks, allenap!
<adeuring> jcsackett: thanks for your review!
<jcsackett> adeuring: welcome. sorry i forgot to ping you i was done. :-)
<adeuring> jcsackett: no problem :)
<m4n1sh> need some clarification related to PPA terms of service
<m4n1sh> if I have a library which is not packaged
<m4n1sh> but still I want to use it in my application
<m4n1sh> I am including it in the source tree
<m4n1sh> the source is available
<m4n1sh> but the source contains the blob
<m4n1sh> if i package it and put it in PPA, with a link to the actual source, would it be against ToS?
<leonardr> allenap or jcsackett, i request a review of my awesome branch: https://code.launchpad.net/~leonardr/launchpad/bug-106338/+merge/52085
<allenap> leonardr: Sure, I'll look at it in a bit :)
<m4n1sh> leonardr: can you answer this ToS question? I am not very sure
<leonardr> m4n1sh: sorry, i don't know
<mrevell> m4n1sh, Hi, I may be able to help
<m4n1sh> anyone can?
<m4n1sh> mrevell: yeah, Please
<bigjools> what sort of blob?
<mrevell> m4n1sh, What's the licence of the blob?
<mrevell> And why can't you include it as source?
<m4n1sh> I can
<bigjools> add it as a depenency in another package
<m4n1sh> but again it might be replicated in other appls too
<m4n1sh> bigjools: the project newtonsoft-json isn't package
<m4n1sh> *packaged in debian
<m4n1sh> or anywhere
<m4n1sh> and I wanted to use the library quickly without adding the whole gigantic source code in my project
<m4n1sh> the license of newtonsoft-json is MIT
<bigjools> you must include sources
<mrevell> m4n1sh, If you don't want to include the library's source in your package, which is probably a good idea, then you'll need to package it separately.
<mrevell> and, as bigjools says, add it as a dependency.
<m4n1sh> yeah, which means that increases the work
<bigjools> if you provide a binary and no source, it's violating the licence isn't it?  I don't know what MIT says.
<m4n1sh> MIT is permissive
<m4n1sh> can be closed
<bigjools> ok
<m4n1sh> but the thing is that including a blob in open source code is looked down upon
<bigjools> I'm not entirely sure of the PPA ToS here
<m4n1sh> and might be against ToS
<m4n1sh> I just wanted to be sure, why do all these stupidity if it can be avoided :)
<allenap> leonardr: The mp says "Text conflict in lib/lp/bugs/tests/test_bugs_webservice.py".
<leonardr> allenap: ok, i also got some test failures, so i will ping you when i have everything resolved
<allenap> leonardr: Cool.
<leonardr> allenap, try now
<allenap> leonardr: Will do.
<leonardr> allenap: now pushing a small additional revision that fixes bug 728507
<_mup_> Bug #728507: Expose classes of exceptions with webservice_error() instead of calling export() on each instance. <api> <tech-debt> <Launchpad itself:New> < https://launchpad.net/bugs/728507 >
<allenap> leonardr: All looks good.
<leonardr> allenap: just to make sure, did you get r12504?
<allenap> leonardr: Yep.
<leonardr> great
<jcsackett> allenap: can you take a look at https://code.launchpad.net/~jcsackett/launchpad/spam-messages-visibility/+merge/52097?
<jcsackett> or are you swamped?
<allenap> jcsackett: I'm not going to be able to get to it until I return after 2000 UTC.
<jcsackett> allenap: that's alright; it needs a db review too, so it won't move out of review till tomorrow.
<jcsackett> would you rather i hunt down someone else?
<allenap> jcsackett: Okay, I'll look later.
<jcsackett> allenap: fantastic. thanks. :-)
<rvba> allenap: can you take a look at https://code.launchpad.net/~rvb/launchpad/fix-missing-add-distro-link/+merge/52104?
<allenap> rvba: I don't think I'll have time today. jcsackett, could you take that one?
<rvba> allenap: nothing urgent really, and I'm off for today too
<jcsackett> allenap, rvba: sure, i've got it.
<rvba> jcsackett: all right then, I'll see your comments tomorrow morning then ;-)
<jcsackett> rvba: indeed. have a good evening. :-)
<mtaylor> whatever happenend to merge queues
<allenap> rvba: Cool. Fwiw, when asking for a review from the on-call reviewer there is usually an expectation that you'll be around for a while to answer questions. If you can't be around, mention it; most reviewers will take the proposal anyway, but it's worth checking.
<rvba> allenap: good to know
<lifeless> sinzui: ping
<lifeless> bug 728609 - can you guys take that on? its an interrupt from oem
<_mup_> Bug #728609: Elements are not closed on bug page source <escalated> <oem-services> <Launchpad itself:Triaged> < https://launchpad.net/bugs/728609 >
<sinzui> lifeless:  crack markup?
<lifeless> link elements are not closed, xml fail for greasemonkey
<sinzui> I will fix this now
<lifeless> jamesf in -ops reported it
<lifeless> sinzui: how do we hide messages? https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/540280/comments/17
<_mup_> Bug #540280: openssh (or gvfs) asks for a password everytime I change directory in sftp <lucid> <regression-release> <bzr-gtk (Ubuntu):Triaged> <gvfs (Ubuntu):Triaged> < https://launchpad.net/bugs/540280 >
<jcsackett> leonardr: ping.
<leonardr> jcsackett, hi
<jcsackett> i just saw you linked a branch to 106338; this is a full fix to the bug?
<jcsackett> sinzui asked me to look at that other day, so i assigned myself. i've done a little work but yours looks like it covers the issue.
<jcsackett> leonardr ^
<leonardr> jcsackett: yes, it's a full fix
<jcsackett> leonardr: cool. i'll unassign myself and assign you. thanks.
<leonardr> i'm now working on similar bugs like 345904
<jcsackett> leonardr: dig.
<lifeless> gmb: yo
<lifeless> gmb: your subscription form should be loading more snappily now
<leonardr> lifeless: i'm considering having lazr.restful treat all IntegrityError exceptions as 400 ("Bad Request"). do you think there are situations in which an OOPS would be appropriate, instead?
<lifeless> leonardr: IntegriterError's get retried by the publisher
<lifeless> messing with them is almost certainly a mistake
<lifeless> they are a routine part of working with our DB
<leonardr> lifeless: i'm not proposing messing with them in any way that inteferes with the normal usage of them
<leonardr> i'm trying to fix bug 345904
<_mup_> Bug #345904: API: IntegrityError when trying to create a newTask to a bug that already has one for that project <api> <lp-bugs> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/345904 >
<leonardr> which is that, once the integrity error has been retried and retried, it's raised to the client
<leonardr> and becomes an OOPS
<leonardr> in the case of 345904, the integrityerror was caused by the end-user, so an oops is not appropriate
<lifeless> in the normal web ui
<lifeless> the user would also expect a nice error
<jkakar> FYI, I'm getting repeated OOPS emails related to branch that doesn't seem to be rebuilding its diff.  The latest one is: OOPS-1888MPJ140
<jkakar> I've been received one every 10 minutes for the last 70 minutes.
<lifeless> so my immediate reaction is to ask 'why does the web service not do the same thing'
<lifeless> jkakar: thanks
<jkakar> lifeless: np
<lifeless> those oopses aren't syncing all that often, or something
<leonardr> lifeless: perhaps some essential validation code is in the view when it shouldn't be
<lifeless> leonardr: yes, thats what I suspect is the case.
<lifeless> leonardr: it would be easy for this to be created by a user at the moment. But also
<lifeless> leonardr: if two users add a task at the same time, we would get a mid flight collision, and it wouldn't be either users fault.
<lifeless> leonardr: for *this specific* integrity error, it seems to me we should be handling this in e.g. model code, raising an appropriate error if the task is already there.
<leonardr> ok
<lifeless> the idea of a systemic fix is appealing
<lifeless> but sadly I don't think the causes for integrity error are homogeneous enough
<lifeless> leonardr: curtis has a lot of experience with validators
<lifeless> leonardr: I suggest asking him about how to move that call from view to model
<leonardr> k
<leonardr> sinzui, let me know when you have a minute
<abentley> leonardr: is there a less hacky way of getting Launchpad._root_uri ?
<leonardr> abentley: what are you trying to do?
<abentley> leonardr: I am trying to create a WebServiceTestRequest with the correct root URI: WebServiceTestRequest(SERVER_URL=str(launchpad._root_uri))
<leonardr> to answer your immediate question, i think ._root_uri is the simplest way to get the root uri
<abentley> leonardr: so that I can get web service url with canonical_url.
<leonardr> abentley: you might like the code i'm about to land
<leonardr> what do you want to do with the web service url?
<abentley> leonardr: so that I can do Launchpad.load() on that URL to get the restfulclient version of that object.
<leonardr> yes, you will like this code
<leonardr> in the meantime, you can pass force_local_path=True into canonical_url
<leonardr> that will give you a relative url, which you can pass into load()
<leonardr> my branch adds api_url to lib.lp.testing
<abentley> leonardr: Cool.  We may hit conflicts if you've updated ws_object.
<abentley> I was updating it, too.
<leonardr> abentley: what is ws_object? sounds like something i should be aware of
<abentley> leonardr: it accepts a Launchpad model object and returns the corresponding lazr.resfulclient object.
<leonardr> abentley: i didn't know about that--we should change it to use api_url
<abentley> leonardr: my thoughts exactly.
<_thumper_> sinzui: ping
<thumper> bac: ping?
<bac> thumper: on the phone
<thumper> bac: ack
 * thumper smirks
<thumper> anyone familiar with our release file upload limit of 200 meg?
<thumper> my understanding is that it is per file limit only
<thumper> and we don't have a per-project limit on uploaded files, is that right?
<allenap> jcsackett: I'm not going to get to your review this evening. Would you like me to do it in the morning (i.e. before you come online)?
<jcsackett> allenap: that would be fine. thanks for offering. :-)
<allenap> jcsackett: Cool :)
<sinzui> hi thumper, leonardr
<leonardr> sinzui, take a look at https://bugs.launchpad.net/launchpad/+bug/345904
<thumper> sinzui: hi, do you know the answer to my question above?
<_mup_> Bug #345904: API: IntegrityError when trying to create a newTask to a bug that already has one for that project <api> <lp-bugs> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/345904 >
<bac> thumper: i agree with what you said
<sinzui> thumper: released uploaded are not actually limited we observer lp/network/user fails at about 200 meg. Some users have uploaded 640meg, other cannot get 20 megs uploaded
<thumper> bac: ok, thanks
<thumper> sinzui: ok, thanks too
<bac> sinzui: is that true?  i thought we did have a hard limit, even though a lot of time there are time outs before people can push a file of that size
<bac> there was a hard limit the last time i worked on it
<sinzui> leonardr: the issue here is that the method/factory that tried to create a duplicate bugtask for a product without first checking that it already exists
<leonardr> sinzui: right. the validation code that should be in the model is in the view
<leonardr> and i don't know enough to move it
 * bac reboots. bbiab
<sinzui> leonardr: I think this is a simple check to look for the same target among the existing bugtasks. the view is doing this. I think we need to push the check into the model and have the view call it in validate() and addTask() call it.
<sinzui> leonardr: I need to get my children from school. I can help after I find the code you are looking at
<leonardr> sinzui: the method that does the validation is valid_upstreamtask. so i should explicitly call that in addTask, and raise an exception if it's false?
<sinzui> thumper: lifeless: the invalid markup on bug pages comes from the LP.cache js in the page. We cannot place xhtml into a jsblock without <![CDATA[xxx]]>. Recall that the TALES engine goes belly up when we place CDATA instructions in the page
<sinzui> leonardr: yes, raise an exception in that method is right.
<thumper> sinzui: we aren't placing xhtml into a jsblock are we?
<thumper> sinzui: we aren't there...
<thumper> sinzui: ah... you mean the bit at the bottom
<thumper> sinzui: where we added the xhtml representation
<thumper> sinzui: do you recall why TALES barfs?
<lifeless> sinzui: thumper: did that change in the last couple of days?
<lifeless> this is apparently a regression
<thumper> lifeless: it changed at the end of last week I think
<lifeless> then yeah, that fits - we had a rough couple of deploys
<leonardr> sinzui: ok, i thought it had to go into some special validation code
<lifeless> why do we include xhtml in the initial json on the page? Won't it be redundant with the initial page state?
<thumper> sinzui: don't we do something like // <![CDATA[
<thumper> sinzui: ... // ]>
<allenap> sinzui, thumper: simplejson.encoder.JSONEncoderForHTML.
<thumper> allenap: which does what?
<sinzui> thumper: we do not do that because the tal process died when we tried
<allenap> thumper: It also escapes <>& in json.
<thumper> allenap: we'd need to then unescape it when we want to use it
<thumper> but that may be a sane approach
<allenap> thumper: No. It escapes into javascript escapes, i.e. < becomes \\u003c
<allenap> So it works out of the box.
<thumper> allenap: ah...
<thumper> sinzui: since you've claimed the bug, want to try it?
<sinzui> allenap: I investigated that lat last year and discovered Lp has an old version
<sinzui> maybe I remember incorrectly
<allenap> sinzui: I think you're probably right. The code is very simple though, so even if we don't upgrade simplejson, we can use that class.
<allenap> Bug 684430.
<_mup_> Bug #684430: Use JSONEncoderForHTML when preparing JSON fragments for pages <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/684430 >
<sinzui> yes
 * jcsackett has very sad internet connection
<thumper> I got a bunch of TestPoppy errors from ec2
<thumper> wisted.internet.error.CannotListenError: Couldn't listen on any:2121: [Errno 98] Address already in use.
<thumper> anyone else getting these?
<lifeless> thumper: why do we put xhtml in the initial page render ?
<lifeless> thumper: I understand why we optionally send it in api responses
<thumper> lifeless: because it is part of the entry code
<lifeless> thumper: but does it serve any value in the initial render ?
<thumper> lifeless: I'm not entirely sure...
<thumper> lifeless: not as far as I'm aware
<lifeless> thumper: I thought it needed to be explicity requested ?
<thumper> the initial one has it
<leonardr> thumper: i don't think it's useful now, but the same code that updates the page when that html data changes can be used to create the _initial_ state of the page
<leonardr> if oyu're talking about what i think you are
<lifeless> leonardr: but we ship a valid initial state of the page for non-javascript clients anyway
<lifeless> (or am I wrong?)
<leonardr> lifeless: you're not wrong
<leonardr> sinzui: does this make any sense? http://pastebin.ubuntu.com/575180/
<wallyworld> leonardr: thumper: standup?
<thumper> aye
<leonardr> wallyworld: @operation_for_version('devel')
<leonardr> at the bottom of the annotations
<gary_poster> allenap, jcsackett, I have a large branch up for review if you are willing.  https://code.launchpad.net/~gary/launchpad/bug723999/+merge/52133
<jcsackett> gary_poster: i believe allenap is out for the day, so that pretty well leaves me. :-)
<gary_poster> ok jcsackett, pretty please then? :-)
 * jcsackett looks
 * jcsackett whistles
<jcsackett> over 1000 loc, eh? you weren't kidding. :-p
<gary_poster> :-D
<jcsackett> nearly half deletes tho. i like that.
<gary_poster> and lots of comments!
<sinzui> leonardr: that looks right. I expected the assert to be self.assertWithContent()
<leonardr> sinzui: does that require an exact match? the string includes the random product name
<sinzui> it does require an exact match. You can interpolate the random name, or just create the product with name='stoat'
<sinzui> We do not have enough weasel family members in our tests
<sinzui> otter, mink, stoat, badger, skunk, dotson.
<LPCIBot> Project devel build #500: FAILURE in 5 hr 11 min: https://hudson.wedontsleep.org/job/devel/500/
<lifeless> thumper: sinzui: leonardr: so can we fix this by removing the xhtml in the initial load? smaller initial page, less render overhead
<leonardr> lifeless: yes, i think so, since i doubt we have any code depending on that xhtml right now
<lifeless> how do we make that happen ?
<leonardr> lifeless: i think we undo a small change i made to lazr.restful
<lifeless> leonardr: cool
<leonardr> lifeless: yes, we change WebLayerAPI.json to ask for JSON_TYPE, not JSON_PLUS_XHTML_TYPE
<lifeless> sinzui: ^
<leonardr> sinzui: it looks like assertRaisesWithContent doesn't take keyword arguments, the way assertRaises does. that cna't be right, can it?
<sinzui> leonardr: I am shocked if it does not
<leonardr> sinzui: TypeError: assertRaisesWithContent() got an unexpected keyword argument 'target'
<lifeless> you might like the testtools MatchesException matcher
<lifeless> also
<lifeless> assertRaises returns the exception
<lifeless> so you can write
<lifeless> self.assertEqual('foo bar', self.assertRaises(...).args[0])
<leonardr> that's useful
<lifeless> I generally use the matcher based approach now though
<lifeless> its pretty cool
<thumper> how do you use a matcher with raises?
<lifeless> the Raises matcher
<lifeless> you the thing being matched needs to be no-arg callable
<lifeless> e.g.
<sinzui> leonardr: I cannot find WebLayerAPI in the lp tree. can you give me a clue
<leonardr> sinzui: it's in lazr.restful
<lifeless> self.assertThat(partial(callable, target='bar'), Raises(MatchesException(AssertionError('the target must be foo')))
<leonardr> src/lazr/restful/tales.py
<lifeless> or
<lifeless> self.assertThat(partial(callable, target='bar'), Raises(MatchesException(AssertionError))
<lifeless> if you only care about the type
<lifeless> or
<sinzui> ah out of my depth
<lifeless> self.assertThat(partial(callable, target='bar'), Raises())
<lifeless> if you only care about it raising /something/
<lifeless> there is a helper 'raises' which makes this a little easier to spell - raises(foo ) -> Raises(MatchesException(foo))
<leonardr> sinzui: i will make a branch for you to try
<lifeless> so the first example is idiomatically spelt as
<sinzui> I am pull it now
<lifeless> self.assertThat(partial(callable, target='bar'), raises(AssertionError('the target must be foo')))
<leonardr> sinzui: ok, just make that change, bump the minor version number, and mention it in the changelog
<leonardr> lifeless: thumper and i realized that we meant to talk to you about web service versioning on tuesday and never did
<leonardr> here's our proposal
<leonardr> instead of numbering versions (which implies that a later version is a certain amount better than any given earlier version)
<leonardr> we will name versions, similar to how ubuntu releases are named (which implies only that a later version is a certain number of releases later than an earlier version)
<leonardr> so, instead of calling the next release 2.0, we might give it a 'c' name like 'cauliflower'
<leonardr> we would require all new web service annotations to reference cauliflower
<leonardr> devel would refer to cauliflower, but once we cut a new version ('dandelion'), devel would start pointing to that version
<leonardr> this removes the need to think about which version you are really releasing in when you say 'devel'
<leonardr> the need to do anything special to the annotations upon release
<leonardr> and it lets us pick release names in advance without saying anything about the magnitude of the change in that release
<leonardr> i'm going to work on something like this system on monday--we can go back and forth on the details without affecting the basic shape of the system
<jcsackett> gary_poster: methods like getSubscriptionForBugTask on the StucturalSubscription mixin fall under the category of "things you left in for tests you think need to be cleaned out later", yes?
<gary_poster> jcsackett pretty sure yes; will double check after I get off a call
<jcsackett> gary_poster: gotcha.
<wgrant> Has anybody looked at the buildbot failure?
<lifeless> leonardr: would cauliflower be exposed at all outside our python source tree?
<lifeless> leonardr: I don't really get the 'implies that a later version is a certain amount better than any given earlier version' bracket
<wgrant> Has anybody looked at the failing script?
<leonardr> lifeless: earlier you objected to the idea of saying '2.0' right at the start instead of saying 'devel' up to the moment of release, and then renaming everything to '2.0'
<leonardr> i thought you were saying that we might regret calling it '2.0' due to the magnitude (or lack thereof) of the changes present in that version
<lifeless> no
<lifeless> uhm
<lifeless> we don't want everything in devel to be in (1.0)+1
<poolie> hi lifeless, wgrant, leonardr, everyone
<lifeless> I object to trying to decide that at the start.
<wgrant> Morning poolie.
<leonardr> lifeless: so, we have two changes in devel, neither of which are in 1.0. we release 1.1, which contains change A, but change B is still only available in devel. is that what you mean?
<lifeless> leonardr: yes
<lifeless> leonardr: I think what goes in 1.1 needs to be an explicit decision. It's our point for saying 'A is stable, we are willing to commit to 5 years support of that'.
<lifeless> leonardr: consider that B might have been in devel for 2 days when 1.1 is created.
<leonardr> lifeless: in that case, we should not have a 'devel' at all. those two changes should be in different versions from the start
<lifeless> leonardr: why? the question is - at what point does something we have exposed become something we are willing to support long term.
<lifeless> leonardr: is it simple duration? is it whether we've signed off the work with jml ? is it whether its highly valued by the distro ?
<lifeless> leonardr: I don't think there is a single, simple answer.
<leonardr> lifeless: this may be too complex for the existing version system to handle
<lifeless> leonardr: I think it handles it fine: we make a 1.1. We move the things we want to support to it. We announce it, and we shift the launchpadlib default version for the next ubuntu to 1.1.
<leonardr> lifeless: so, at the point we release 1.1 we go through every change in devel and make the decision to backport or not?
<lifeless> yes
<lifeless> I think we'd want to do that even if we had a way of flagging them in advance.
<lifeless> 5 year commitments are serious business :)
<sinzui> leonardr: I am waiting for `setup.py test` to complete before I commit. I see you just landed a change. Do you want to review it. Do I just commit to trunk when the branch is approved
<leonardr> sinzui: i recommend running bin/test instead of setup.py test
<leonardr> lifeless: what does this mean for our current process? we require that all new annotations be explicitly flagged to devel?
<sinzui> leonardr: I do not have a bin/test, but I see 4 failures
 * sinzui reverts and runs the tests again
<leonardr> sinzui: here's the process i use
<leonardr> python bootstrap.py; bin/buildout; bin/test
<sinzui> thanks
<lifeless> leonardr: yes, I think thats all it means. Making the default devel would make this easier: unflagged would mean unreleased, flagged would mean released.
<lifeless> leonardr: what we have is pretty good
<leonardr> thumper: are you ok with this?
<thumper> leonardr: yeah. If we want to make sure we explicitly choose what to release
<thumper> then this seems ok
<wgrant> What happened to build failures being a stop-the-line event? :(
<jcsackett> wgrant: they're not? or do you mean in terms of people, not pqm/buildbot?
<wgrant> jcsackett: In terms of people.
<wgrant> lucid_db_lp failed more than 8 hours ago.
<wgrant> With an obviously spurious failure.
<jcsackett> y'know, i don't get many buildbot emails anymore.
 * jcsackett makes not to see if a filter is accidentally grabbing them.
<gary_poster> jcsackett, yes lib/lp/bugs/model/structuralsubscription.py getSubscriptionsForBugTask is a prime example of a now-useless function that only exists in that branch to keep test-munging out of the diff.
<jcsackett> gary_poster: fantastic.
<sinzui> ouch. look like I will be compiling for a while
<jcsackett> incidentally, sorry this is taking so long. i'm having to route around a lot of network issues on my end.
<gary_poster> np, it was giganto branch anyway jcsackett.  You'll only get thanks from me.
<wgrant> lifeless: Is https://hudson.wedontsleep.org/job/devel/500/testReport/junit/lp.registry.browser.tests.test_milestone/TestDistributionMilestoneIndexQueryCount/test_bugtasks_queries/ one of yours?
<wgrant> lifeless: The third query (Person, between the TeamParticipations) is intermittent.
<leonardr> sinzui: you need a ~/.buildout/default.cfg
<leonardr> mine says:
<leonardr> [buildout]
<leonardr> eggs-directory=/home/leonardr/.buildout/eggs
<leonardr> download-cache=/home/leonardr/.buildout/download-cache
<leonardr> with that in place you will only go through the huge compile step once
<lifeless> wgrant: I think bac wrote it
<lifeless> but it interacts with bugtask:+index work obvious
<lifeless> ly
<wgrant> I'll increase the query count by 1 for now.
<sinzui> leonardr: yes I have one of those. I did not have everything that lazr.restful wanted though
<wgrant> Hmm, unless I can easily get tracebacks into there.
<leonardr> ah
<lifeless> wgrant: if you can run it with LP_DEBUG_SQL_EXTRA=1
<wgrant> lifeless: What does _EXTRA do?
<lifeless> backtraces
<wgrant> Ah.
<wgrant> Thanks.
<lifeless> =====backtrace, ...., sql,======
<wgrant> Thatisquitealotofoutput
<bac> wgrant: i think abel wrote that test and i have modified it
<sinzui> :/ me thinks compiling mozilla might be faster
<wgrant> bac: There are no cache invalidations before the query checks :(
<lifeless> ok
 * lifeless -> sydney
<wgrant> bac: So it's sort of not really valid, and very unreliable.
<lifeless> if you need me in the next 2 hours, call my mobile
<bac> wgrant: sorry, i was thinking of a different test.  the one you're talking about was written by edwin.
<sinzui> leonardr: what should I rename test_web_layer_includes_html_with_json_representation => disabled_web_layer_includes_html_with_json_representation
<sinzui> leonardr: also, I think lazr.restful may hate python 2.7
<leonardr> sinzui: web_layer_includes_json_representation
<sinzui> okay
<leonardr> ie. test the other thing or get rid of the test
<leonardr> sinzui: it's possible. are you getting errors?
<sinzui> I see two failures in a pristine tree. I am on natty
<sinzui> leonardr: https://code.launchpad.net/~sinzui/lazr.restful/json-not-xhtml-0/+merge/52141 I think you should pull the branch and verify the tests are as you expect
<leonardr> ok
<wgrant> sinzui: Hi, could you run a query on staging for me please?
<sinzui> wgrant: yes
<wgrant> sinzui: query 36 (the last one) from https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1888QS70... could you grab an EXPLAIN ANALYSE ?
<sinzui> wgrant: http://pastebin.ubuntu.com/575213/
<wgrant> sinzui: Erm. Could you try that on qastaging, please? I wonder if there is some difference -- it seems to take 13 seconds there, not 60ms.
<sinzui> well this taking time
<wgrant> Intriguing.
<sinzui> wgrant: http://pastebin.ubuntu.com/575215/
<sinzui> wgrant: maybe the difference is hot and cold caching
 * sinzui runs again
<wgrant> sinzui: I tried that page 20 times yesterday.
<wgrant> Got no better.
<wgrant> the plan is almost identical.
<wgrant> Yet the times are off by two orders of magnitude.
<sinzui> wgrant: qastaging run a second time: http://pastebin.ubuntu.com/575218/
<wgrant> Huh.
<wgrant> Thanks.
<wallyworld> anyone up for a review? https://code.launchpad.net/~wallyworld/launchpad/remove-decoratedbug/+merge/52003
#launchpad-dev 2011-03-04
 * wallyworld hates "An updated diff will be available in a few minutes. Reload to see the changes. " My reload button is all worn out
<StevenK> You still have F5
<wallyworld> StevenK: oh, i was wondering what that keyboard button was. The "F5" has all worn off  from overuse :-P
<wallyworld> and another review for someone https://code.launchpad.net/~wallyworld/launchpad/remove-mp-relatedbugs/+merge/52069
<thumper> :(
<thumper> dumb windows email clients
<wgrant> Oh?
<wgrant> Dumber than Gmail?
<thumper> what does the mail RFC say about encoded email headers?
<thumper> we are getting some with windows-cp1252
 * wgrant checks 2047.
<wgrant> Ah, it permits anything IANA-registered. :(
<wgrant> But windows-cp1252 doesn't appear on IANA's charset registry...
<thumper> wgrant: meaning?
<wgrant> Its proper name is apparently windows-1252, not windows-cp1252
<thumper> yeah, I'm getting that one
 * thumper sighs 
<thumper> my problem, not the code
<wgrant> Oh?
<thumper> the email['from'] field is encoded with windows-1252
<wgrant>  * 2513 Exceptions
<wgrant> :(
<wgrant> ~500 more than I expected.
<wgrant> Ah, MP job breakage.
<lifeless> timeouts are down!
<thumper> Content-Type: text/html; charset=windows-1252
<lifeless> -woo-
<thumper> that is what the email says
<lifeless> ScopedCollection:CollectionResource:#person-page-resource is new
<wgrant> thumper: That's valid.
<thumper> but the headers are encoded with that charset too
<lifeless> as is Product:+filebug-show-similar
<wgrant> thumper: Hm, do you have the librarian URL for that email?
<thumper> it's spam
<wgrant> 88/176 for BugTask:+index. That's nice.
<thumper> http://launchpadlibrarian.net/65554538/9a948026-45ee-11e0-86fc-001e0bc3957e.txt
<lifeless> freaking awesome
<wgrant> thumper: Ugh.
<StevenK> lifeless: Aren't you on a plane?
<wgrant> I would have thought so.
<lifeless> boards at 10 past
<lifeless> in wellington
<wgrant> Oh, right.
<wgrant> thumper: So, catch the lack of encoding and reject, I suppose :/
<thumper> wgrant: well, at the moment it gets rejected through the oops :)
<wgrant> Unless we have a convenient way to apply UnicodeDammit to MIME.
<wgrant> Damn.
<wgrant> The exception summary doesn't quite get to single-ocurrence OOPSes yet.
<wgrant> Tomorrow!
<lifeless> RootObject:+recently-registered-branches is a little slow
<lifeless> wgrant: 131 OperationalError: FATAL: Ident authentication failed for user "person-merge-job" FATAL: Ident authentication failed for user "person-merge-job"
<lifeless> OOPS-1888REPORTIFSEEN109, OOPS-1888REPORTIFSEEN110, OOPS
<lifeless> should be easy
<poolie> wallyworld, in https://code.launchpad.net/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/52004
<poolie> would that also work, and be cleaner, as just a collection link?
<lifeless> wgrant: you might like to remind folk setting up jobs that they *must* make a new config and *must* have the losas set it up : we *cannot* [because of oops ids] just use the default config
<wgrant> lifeless: Sure, if we had a LOSA :)
<poolie> or is that already what it does?
<lifeless> wgrant: I say you, because I'm on a plane :>
<lifeless> poolie: we should talk about collections vs queries.
<lifeless> poolie: It may be my failing, but I rather dislike the billion-collections approach.
<poolie> we you and me?
<wallyworld> poolie: can collection links take parameters?
<lifeless> we you and me
<wallyworld> i don't think so
<poolie> no, they can't
<poolie> but this doesn't seem to take parameters
<wallyworld> it takes the user
<poolie> oh?
<lifeless> bug 726803
<_mup_> Bug #726803: not possible to export a method as a collection with bound parameters <lazr.restful:Triaged> < https://launchpad.net/bugs/726803 >
<wgrant> thumper: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1888BZR102541 needs looking at.
<poolie> wallyworld, i thought that meant it depends on the implicit request user?
<wallyworld> poolie: we need the user since getRelatedBugTasks() needs it so that it can see what bug tasks are allowed to be exposed
<wgrant> thumper: It's a single user running a script against a single branch, AFAICT always looking for the next revno.
<poolie> in that case, yes, of course collections can look at that
<lifeless> >< http://news.cnet.com/8301-17938_105-20028475-1.html
<wgrant> thumper: Happens a few hundred times a day, and gets logged as an OOPS.
<thumper> do we know who?
<wallyworld> poolie: we set a default user to be the request user but i think we still need to allow other users to be specified?
<wgrant> thumper: It was easy enough to track down, but I don't remember.
<wgrant> thumper: Regardless, it should not OOPS.
<thumper> hmm...
<thumper> agreed
<thumper> I wonder why it does
<wgrant> thumper: Oh, right:
<wgrant> user_id: 1689259
<thumper> wgrant: is there a bug
<wgrant> thumper: I don't believe os.
<poolie> wallyworld, oh, i see, it makes an optional parameter
<poolie> http://trpdsaya.tumblr.com/post/3463810356/omg-this-app-is-so-restful
<poolie> also http://trpdsaya.tumblr.com/post/3600386585/the-url-structure-on-this-site-is-so-pretty for that matter
<wallyworld> poolie: yes. i think i have done it the right way
<wallyworld> but i'm happy to be told i need to do it another way
<lifeless> I think its fine for now
<lifeless> its in devel
<lifeless> if we want to impove it we can :)
<lifeless> later y'all.
<poolie> much better to have it there at all
<poolie> i was just wondering
<wallyworld> np. always good to ask questions
<wgrant> StevenK: I wonder if mawson is going to survive publishing all of natty, seriously this time.
<StevenK> Haha
<wgrant> StevenK: Will parallel a-f with all of the files present OOM it, I wonder?
<StevenK> You tell funny jokes
<poolie> it would be kind of nicer if it was not a ws.op thing, but rather
<wgrant> StevenK: It has finished downloading all the files, so is now calculating lists for a-f. I guess that will take another couple of horus.
<poolie> https://api.launchpad.net/devel/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/52004/+related-bug-tasks
<poolie> and then you could filter it etc
<poolie> but that probably needs more infrastructure elsewhere
<wallyworld> poolie: you suggesting that that api be added?
<StevenK> wgrant: cocobanana and mawson have roughly the same amount of RAM
<StevenK> wgrant: *and* cocobanana runs rsyncd, which eats babies.
<wgrant> StevenK: I thought cocoplum would have more.
<wgrant> Plus cocoplum doesn't have 223GB of DB.
<StevenK> No, it has a 400GB archive instead :-)
<wgrant> But it doesn't need to keep a big chunk of that in RAM.
<wgrant> Also, mawson now has 120GB of that archive :P
<StevenK> Haha
 * StevenK looks at changing the db user for a garbo class
<michaelh1> Hi there.  When a merge request comes in I'd like to do a bzr export, fire it into my build farm, then update the merge request with the automatic build results.  Are there existing tools past launchpadlib I can reuse?
<StevenK> Rargh, TunableLoop can't do that?
<thumper> michaelh1: not as far as I'm aware
<michaelh1> thumper: ta.
<StevenK> wgrant: I don't know if the populate job can run under garbo, just due to DB perms
<jtv> jcsackett, really still here?  If so, got a review for you!  https://code.launchpad.net/~jtv/launchpad/bug-623391/+merge/52114
<jcsackett> jtv: ah no, actually. can't seem to change channel info though. could you change it to take me off?
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<jtv> thanks StevenK :)
<wgrant> StevenK: You know that you can change DB perms, right?
<StevenK> Assuming allenap is sleeping
<jtv> He should be!
<wgrant> Looks like we need to get memcached killed on the lucid_db_lp slave :(
<wgrant> And no LOSAs :(
<StevenK> wgrant: Sure, but we don't change db perms on prod. Or am I out of date?
<wgrant> StevenK: If a script needs more permissions, give it more DB permissions.
<wgrant> We grant new permissions during nodowntime rollouts.
<StevenK> wgrant: I can do that. I'm still wondering how the heck to test it
<wallyworld> thumper: you able to review any of my branches?
<wallyworld> https://code.launchpad.net/~wallyworld/launchpad/remove-decoratedbug/+merge/52003
<thumper> wallyworld: aye
 * thumper looks
<wallyworld> and  https://code.launchpad.net/~wallyworld/launchpad/remove-mp-relatedbugs/+merge/52069
<wallyworld> thanks :-)
<wallyworld> this one should also be good to go since i fixed the issue raised https://code.launchpad.net/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/52004
<thumper> wallyworld: 710 lines?
<wallyworld> which one? isn't the limit 800?
<wallyworld> thumper: much/most of the line count is removing stuff
<wallyworld> there's lots of red there
<poolie> hi thumper
<poolie> are you going to fix 728827 by downgrading the message from warning to info or something?
 * wallyworld has to take kid to train station
<thumper> poolie: yeah
<poolie> great
<thumper> done even
<poolie> sorry, i didn't realize just having a warning message caused an oops
<poolie> i guess it makes sense
<thumper> neither did I
<thumper> I had to hunt to work out where it was happening
<wgrant> It only started a few months ago.
<poolie> only started giving a warning then?
<wgrant> Warnings and errors only became OOPSes then.
<poolie> ok
<poolie> i wonder how much it is worth documenting that vs just having people gradually know it
 * wgrant kicks buildbot-poller
<StevenK> wgrant: Hm?
<wgrant> StevenK: It pulled devel->stable but did not merge stable->db-devel.
<wgrant> Possibly because db-devel was testfixed.
<StevenK> Mmmm, the deployment report makes me happy.
<wgrant> Yes.
<wgrant> But there are several more revs that would be nice.
<wgrant> eg. 12522
<wgrant> Which is in stable, and cuts a couple of seconds off binary copies.
<StevenK> We're just waiting for qastaging, then?
<thumper> https://code.launchpad.net/~thumper/launchpad/mail-header-oops/+merge/52156 anyone?
<thumper> I'm just pushing the conflict resolution merge
<poolie> i will do that
<poolie> do the review
<poolie> thumper, how does the x-launchpad-original-to header get into the mail?
<poolie> by an incoming gateway?
<wgrant> Yes, an MTA adds it somewhere.
<thumper> poolie: yeah
<poolie> i can't retrieve any of the sample urls like http://launchpadlibrarian.net/62571148/916041d6-2558-11e0-b038-001e0bc3957e.txt
<poolie> did they expire or something?
<thumper> I think they were garbage collected
<thumper> I'm not sure exactly how that works
<poolie> anyhow i commented
<thumper> poolie: I replied
<poolie> oh, blah
<poolie> i just got a "Your message was rejected" from launchpad-bugs-owner@lists.canonical.com
<thumper> :)
<thumper> https://code.launchpad.net/~thumper/launchpad/strip-email-attachment-path/+merge/52159 anyone?
<wallyworld> thumper: i'll take a look
<wallyworld> thumper: with my 3 mp's,  they all follow on from each other. do i have to give them to ec2 one at a time, or can i just land the last one?
<thumper> wallyworld: you can just do the last one
<wallyworld> thumper: can i approve the middle one even though it says needs fixing - i've addressed the requested change (to export to the "devel" ws api not 1.0) but lifeless i don't think is online atm
<thumper> wallyworld: socially, no
<wallyworld> yeah thought so :-)
<wallyworld> i'll wait
<thumper> wallyworld: programatically, yes
<wallyworld> jtv: got a question for you if you are around
<jtv> wallyworld: I'm around, as long as you're not planning to bother me with questions
<wallyworld> oh :-(
<wallyworld> jtv: bug 407260
<_mup_> Bug #407260: Translations export branch can't be team-owned <lp-translations> <Launchpad itself:Triaged by wallyworld> < https://launchpad.net/bugs/407260 >
<jtv> ah that
<jtv> (clever of you not to use question marks)
<wallyworld> i tried to write a test case to reproduce it
<wallyworld> but my test worked
<wallyworld> so clearly i'm missing something
<jtv> Yes, IIRC the problem is with the branch picker
<StevenK> wgrant: It seems I can't do self.getCandidateSPRIDs(0).count() == 0
<wallyworld> i tried to adapt test_translations_export_to_branch
<wallyworld> and made the branch owner a team
<StevenK> wgrant: FeatureError: Single aggregates aren't supported after a  GROUP BY clause
<wgrant> StevenK: No, since some of them will fail.
<wgrant> That too.
<wgrant> But the garbo job will probably never end, since some will fail to unpack.
<wgrant> This may put an end to that idea.
<StevenK> Until we remove it
<jtv> wallyworld: perhaps more precisely, my recollection is that we have no vocabulary for "a branch that I'm *an* owner of"
<wallyworld> jtv: oh, so it's a gui problem
<wgrant> StevenK: But there may be enough that every run only processes the broken ones, without reaching the end.
<wallyworld> jtv: the back end code works regardless, but the gui doesn;t allow the desired branch to be selected
<jtv> wallyworld: pretty muchâit only occurs when setting the branch
<wallyworld> jtv: thanks. i thought it was a back end problem
<wallyworld> but the test worked, so clearly not :-)
<StevenK> wgrant: I'll reimplement the finish_at bit then
<jtv> wallyworld: Not that I can think of, noâ¦ we decided not to check for ownership during actual commit, IIRC; as an owner you get to write out a permanent license for LP to commit to it.
<wallyworld> jtv: i'm doing this one because a user brought it up on #launchpad and it seems a good one to fix, since it affects a few people and has been around for a while
<wgrant> StevenK: How do you plan to preserve this across garbo sessions?
<wallyworld> thanks for the clarification
<StevenK> wgrant: I don't
<wgrant> StevenK: The problem will still exist.
<jtv> wallyworld: yes, I've been wanting to do that for ages.
<wallyworld> jtv: cool. see, me asking questions wasn't so bad after all :-)
<jtv> Only because I wasn't around.
<StevenK> wgrant: How many do you expect won't unpack?
<wgrant> StevenK: Potentially thousands.
<StevenK> That makes me sad.
 * thumper is getting the muchies
<thumper> or munchies even
<StevenK> But it's only 5:30pm?
<thumper> yup
<thumper> after beer o'clock on a Friday
<wallyworld> thumper: have a beer for me too
<thumper> wallyworld: ack
<wallyworld> or a nice red even
<wallyworld> i can't tonight - have to go out in the pouring rain and coach soccer :-(
<jtv> StevenK: something I'm wonderingâ¦
<jtv> â¦if you're still here.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #501: FIXED in 5 hr 9 min: https://hudson.wedontsleep.org/job/devel/501/
<StevenK> jtv?
<Ursinha> this bot is so happy
<StevenK> Haha
<jtv> StevenK: as you know I'm generating DistroSeriesDifferences directly based on SPPH/SPR comparison between two DS.
<jtv> I create the DSDs with source_version set to SPR.version from the derived series, and parent_source_version set to SPR.version from the parent series.
<jtv> Up to one of those can be null, if the difference type is not different_versions.
<jtv> (And on a sidenote, I suspect that unique_to_derived_series could be very subtly misleading and that specific_to_derived_series might be better)
<jtv> I just thought I'd ask if there is still substantial work to do for the case where a package is only in the parent or only in the derived series.
<StevenK> jtv: If it's only in the parent or only in the child, we don't want a DSD
<jtv> Err
<jtv> Then why are there DSDTypes for those cases?
<StevenK> jtv: I have no idea -- but there *no* *point* generating diffs for them
<thumper> two or three critical bugs closed today as they code paths no longer exist and oops hasn't happened in ages :)
 * thumper EODs
<jtv> StevenK: then maybe all you want is to block the things?
<jtv> bye thumper
<thumper> have a good weekend y'all
<jtv> same to you
<StevenK> thumper: Have a good weekend
<jtv> TGIF & ODIM
<jtv> The Yin & Yang of the business world.
<jtv> StevenK: I meant, maybe the only thing we really want from packages that only exist on one of the two sides is the ability for the owner to either block or synchronize?
<StevenK> jtv: Exactly
<jtv> Because if that is the case, and it can be done entirely in SQL, then we have no need to bother the DSD script with those cases.
<jtv> The way the script works now, figuring out the differences between a DS and its parent and creating all the DSDs all happens in SQLânone of the data ever gets pulled into the python code.
<wgrant> jtv: Does http://paste.ubuntu.com/575313/ mean anything to you? We didn't see anything like that in the old code, but everything looks fine anyway.
<jtv> wgrant: are we sure that the old code would have reported output like that in the first place?
<wgrant> jtv: I am not entirely sure.
<jtv> ISTR the old code being a bit iffy in that department.
<wgrant> Anyway, once I've run the publisher twice with this to get timings, I will rerun with the old one to compare the time and diff the trees.
<wgrant> Hopefully the next run will not take 24 hours :)
<jtv> Well it'll be weekend for you either way won't it?  :)
<wgrant> A good point.
<wgrant> With luck this weekend will have fewer production incidents than the last.
<wgrant> jtv: OutputLineHandler has a bug.
<jtv> ?
<wgrant> >>> s = OutputLineHandler(sys.stdout.write)
<wgrant> >>> s('foo')
<wgrant> >>> s('bar\n')
<wgrant> foobar>>> s('bar\n')
<wgrant> foobar>>> s('bar\n')
<wgrant> It never empties the buffer if the input ends with \n.
<wgrant> Not critical, but explains some of the ridiculous output I've been getting.
<jtv> wgrant: damn, I thought I'd covered that specifically.
<jtv> IIRC the \n was stripped out somewhere _before_ it got to the OutputLineHandler
<poolie> hi
<wgrant> Yay, netsplits.
<poolie> wgrant, do you agree with me on https://code.launchpad.net/~thumper/launchpad/mail-header-oops/+merge/52156
<wallyworld> jtv: just to check i understand, "you" still handle translation approvals? eg blue squad now on maintenance, and there's a queue entry waiting approval. do i need to do anything?
<jtv> wallyworld: do you have a translations veteran on the team?
<wallyworld> jtv: our team is me, tim, leonard, and steve but steve is on loan to the red team at the moment
<wallyworld> so no :-(
<jtv> (By the way, I can't tell you how much it means to me for another developer to tell me there is "a" queue entry awaiting approval)
<StevenK> wallyworld: jtv is on red, so he is aware of what I'm working on :-)
<wallyworld> jtv: i filtered on the suggested "ANy Project" and "Pot only". was that not correct?
<jtv> Yes, it isâand presumably on the also-correct Needs Review as well.
<jtv> wallyworld: want to walk through it with me?
<jtv> I don't think it'll be very hard.
<wallyworld> jtv: yes please, so i can communicate back to the blue team
<jtv> Just click on the edit icon next to "no import target selected yet."
 * wallyworld clicks
<jtv> Well I say "just," but I mean "for starters."  :)
<wgrant> poolie: I've wondered that myself. For the field to be inserted after a blank line, our MTA would presumably have to be buggy.
<wallyworld> jtv: there is no edit icon
<wgrant> poolie: I agree that investigation is required.
<wallyworld> for me
<jtv> wallyworld: then we haven't added you to the rosetta-admins team yet.  Let me fix that.
<wallyworld> ok
<poolie> it seems kind of unlikely to me that that bug would be triggered iff the message is spam
<poolie> indeed if it's reliably doing that, we should sell it ;-)
<poolie> it's possible it's some interaction between a spam scanner and something else
<wgrant> It is a reasonable assumption that something with header fields in the body is spam.
<wgrant> But I cannot see why *this* field would be there.
<poolie> note that we're not checking for a body that looks like it contains headers
<poolie> just for that string, anywhere in the body
<poolie> i should say so on the mp i suppose
<wgrant> Yes.
<jtv> wallyworld: I've invited the launchpad-chr team to rosetta-admins, as per my email to the list.  But now we need that invitation accepted.
<jtv> I'll email Diogo about it.
<wallyworld> jtv: thanks. can we walk through an approval another time?
<jtv> Absolutely.  I'll just do this one for now.
<wallyworld> i'd still like to learn how to do it
<wallyworld> thanks
<wallyworld> jtv: i think that's the key thing to know about while on maintenance? anything else critical to be aware of?
<jtv> wallyworld: plenty of recurring questions, but for most things IMHO it's best to find out as you go along.
<wallyworld> jtv: ok. thanks again
<jtv> We have lots of CHR documentation on the internal wiki, so you could try reading through that and following some of the links.  But I wouldn't recommend trying to keep up with all the FAQs.  On-demand caching is the ticket.
<wallyworld> jtv: yeah, there was so much material i'd thought i'd try and cheat a bit by asking about the "viewing highlights"
<jtv> Sensible.
<wallyworld> :-)
<huwshimi> Have a good weekend people
<poolie> bye huwshimi
<jtv-eat> Still no reviewers available?  Got two dependent branches blockedâ¦ starting with https://code.launchpad.net/~jtv/launchpad/bug-623391/+merge/52114
<adeuring> good morning
<jtv> Still no reviewers in the house?  Need a critical pair of eyes for https://code.launchpad.net/~jtv/launchpad/bug-623391/+merge/52114
<jtv> (And then I'll try to get someone else for the follow-up branch :)
<henninge> adeuring: ^ Aren't you reviewing today?
<henninge> Hi jtv! ;-)
<jtv> hi!
<adeuring> jtv, henninge, yes. I'll look at your branch. jtv
<jtv> Wunderbar!
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring | https://code.launchpad.net/launchpad-project/+activereviews
<maxwell69> hello
<maxwell69> I have a problem to build launchpad
<maxwell69> when I do maake schema I get an error
<maxwell69> with zc.buildout
<maxwell69> "couldn't find index page for zc.buildout
<maxwell69> I think the program is trying to download zc.buildout but I get ***BLOCKED*** by --allow-hosts
<jtv> adeuring: I've been having some system & connection problemsâ¦ did I miss any questions?
<adeuring> jtv: no, and, btw, no complaints so far. nice to read the diff
<adeuring> ... but i need some more time ;)
<jtv> sure
<maxwell69> Hello jtv, I have a problem with the "make schema" command
<stub> Give me a name for a marker instance for something that might be a Storm subclass, or an instance of a Storm subclass. Besides IFooStormObjectOrClass.
<jtv> maxwell69: I may be able to help, but it's Friday night here and I'm putting out some fires.  :)  What's the problem?
<stub> nm. got one.
<jtv> stub: a marker interface you meant?
<stub> yes
<stub> IUseSessionStore is what I'm actually after.
<maxwell69> I'm trying to buils launchpad but I have an error:
<maxwell69> "couldn't find index page for zc.buildout"
<maxwell69> the program try to download zc.buildout but I have this error:  ***BLOCKED*** by --allow-hosts
<adeuring> jtv: r=me
<jtv> thanks adeuring!  I ought to find someone else to bother with the follow-up branch I guessâ¦
<maxwell69> but my connction wors well and the website is aviable
<adeuring> jtv: nah, i'll take that too :)
<jtv> adeuring: thanks again!
<wgrant> maxwell69: How did you check out the source?
<jtv> maxwell69: it sounds horribly like a zope + system setup problemâ¦  which I wouldn't know much about.
<wgrant> maxwell69: Did you use rocketfuel-setup?
<wgrant> I suspect not.
<maxwell69> wgrant: yes, I followed instructions on https://dev.launchpad.net/getting
<jtv> Now, --allow-hosts sounds vaguely familiar
<jtv> Maybe I'm just thinking hosts.allow.
<wgrant> It should be getting buildout from download-cache.
<wgrant> maxwell69: What is in the download-cache directory inside your LP branch?
<maxwell69> wgrant: I have "dist" directory
<wgrant> maxwell69: Does that contain a few hundred files?
<maxwell69> wgrant: and I manually put zc.buildout-1.5.1 dir
<maxwell69> wgrant: in dist I just have setuptool***.egg
<wgrant> maxwell69: There should be several hundred eggs in there.
<wgrant> Is download-cache a symlink?
<maxwell69> wgrant: I'm affraid not
<wgrant> adeuring: Hi.
<adeuring> hi wgrant
<wgrant> adeuring: Could you please review https://code.launchpad.net/~wgrant/launchpad/turn-off-parallel-a-f/+merge/52185?
<adeuring> wgrant: sure; let me just finish another review for Jeroen
<wgrant> adeuring: Sure.
<wgrant> Thanks.
<maxwell69> wgrant: should I re-execute rocketfuel-setup?
<wgrant> maxwell69: What do you have in ~/launchpad/lp-sourcedeps?
<maxwell69> There is no lp-sourcedeps...
<wgrant> maxwell69: Hmm. Did rocketfuel-setup give any errors?
<adeuring> jtv: r=me again
<jtv> adeuring: and thanks again
<maxwell69> wgrant: there is warnings: DocumentRoot [/var/tmp/archive] does not exist
<wgrant> maxwell69: That's fine.
<maxwell69> and ./rocketfuel-setup line 362: cd: /usr/local/bin: no such file or directory
<wgrant> That would be it.
<wgrant> maxwell69: That should exist on fresh systems... could you create it and run rocketfuel-setup again?
<maxwell69> ok, I have a call,I"ll do it and tell you what hapend :)
<maxwell69> thanks
<adeuring> wgrant: r=me
<wgrant> adeuring: Thanks.
<wallyworld> wgrant: i have a branch against devel which it turns out requires lines added to security.cfg. what's the best way to land it?
<wallyworld> do i need to do anything extra?
<wgrant> wallyworld: Is it just extra permissions, without new users?
<wallyworld> wgrant: yes. just 2 extra select permissions
<wallyworld> for an existing user
<wallyworld> or section
<wallyworld> whatever it's called
<wgrant> So, mthaddon says you should talk to LOSAs, but that sort of change is no problem at all.
<wallyworld> [merge-proposal-jobs]
<wgrant> Since we now grant new permissions on every rollout.
<wallyworld> cool. so just land as normal?
<wgrant> Yup.
<wallyworld> excellent. thanks.
<mthaddon> if it's not adding new users, and is just changing existing permissions for users, that's fine
<wallyworld> mthaddon: yes, adding public.bugsubscription = SELECT to [merge-proposal-jobs]
<wallyworld> so just a new permission
 * mthaddon nods
<matsubara> jtv, ping
<jtv> hi
<matsubara> jtv, so, I think we can actually disable the launchpad-chr team, can't we?
<jtv> matsubara: can we?  I don't know
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: adeuring, bac | https://code.launchpad.net/launchpad-project/+activereviews
<matsubara> I think it was used as some form of access control/receive email
<matsubara> let me dig that
<matsubara> lifeless, hi
<bac> hello adeuring.  busy today?
<lifeless> hi
<matsubara> lifeless, prefix LPNETPREFIX has been loaded. is there any other?
<lifeless> hmm
<lifeless> that may mean the config tom deployed was slightly incorrect
<lifeless> matsubara: could you add servers through to Z?
<matsubara> lifeless, yeah
<matsubara> lifeless, and those are already handled by oops-tools. A to Z
<lifeless> ok
<lifeless> we'll need move to AA AB AC etc soon
<matsubara> lifeless, really? do we have that many instances up?
<matsubara> it'd be nice to get rid of the prefixes by that time as you proposed in the LEP
<matsubara> jtv, sorry, I'll dig in and let you know here or by email
<wgrant> lifeless has plans.
<jtv> matsubara: well, I'm not particularly involved with the launchpad-chr team so all that it would take for me is to add rosetta-admins to another team.
<adeuring> bac: hi bac, well, I did a few reviews.
<lifeless> matsubara: I think we'll need the monday.
<matsubara> lifeless, add the right prefixes to lp-production-config and oops-tools will read from there
<matsubara> lifeless, instructions are here https://dev.launchpad.net/Foundations/QA/OopsToolsSetup  to make oops-tools recognize new prefixes (I can do it today, Monday I'll be off)
<lifeless> matsubara: can you do me a favour? I'm roaming at the moment - drop me a quick mail with that link
<matsubara> lifeless, sure
<lifeless> matsubara: and enough info for me to step a losa through it
<lifeless> (if its not on the page ;P)
<maxwell69> wgrant: Hi again, I created the directory, launched the rocketfuel-setup again, files have been downloaded but when I do "make schema" I have the same error
<maxwell69> Link to http://pypi.python.org/simple/ ***BLOCKED*** by --allow-hosts
<wgrant> maxwell69: Is download-cache a symlink now?
<maxwell69> wgrant: it is a dir
<wgrant> maxwell69: rm -r download-cache; rm -r eggs; rocketfuel-get
<wgrant> Oh, also rm -r sourcecode
<maxwell69> wgrant: inlp-branches/devel ? just to be sure
<wgrant> maxwell69: Yeah.
<maxwell69> OK, downloading...
<maxwell69> wgrant: Ok, I have now: "make: Entering directory `/home/maxwell/launchpad/lp-branches/devel' Missing ./download-cache. Developers: please run utilities/link-external-sourcecode. make: *** [download-cache] Error 1 make: Leaving directory `/home/maxwell/launchpad/lp-branches/devel'"
<wgrant> maxwell69: in ~/launchpad/lp-branches/devel, run 'utilities/link-external-sourcecode ~/launchpad/lp-sourcedeps'
<maxwell69> Ok now it is a symlink to "lp-sourcedeps/download-cache"
<wgrant> Great.
<wgrant> make might work now.
<maxwell69> wgrant: looks good.
<maxwell69> takes a lot of CPU :)
<maxwell69> wgrant: sorry, I m a newbie, but i m not dev, juste a sysadmin :)
<wgrant> maxwell69: Yeah, the first run of make will take a while do unpack all the eggs.
<maxwell69> wgrant: the tail: utilities/shhh.py make -C sourcecode build PYTHON=python2.6 \ 	    LPCONFIG=development make: *** sourcecode: No such file or directory.  Stop. make: *** [compile] Error 2
<wgrant> maxwell69: :(
<wgrant> maxwell69: mkdir sourcecode, and run link-external-sourcecode again.
<wgrant> The missing /usr/local got your tree into a bit of an odd state.
<maxwell69> I ran make again:  make[1]: *** No rule to make target `build'.  Stop. make[1]: Entering directory `/home/maxwell/launchpad/lp-branches/devel/sourcecode' make[1]: Leaving directory `/home/maxwell/launchpad/lp-branches/devel/sourcecode' make: *** [compile] Error 2
<maxwell69> I have symlinks in the sourcecode dir
<bigjools> wgrant: do you have a feel for how much of the publisher (and associated cron stuff) is specific to Ubuntu?
<benji> gary_poster: I had it like that but it looked really odd to me (this looks more like the edit and unsubscribe menu options on the bug page), that being said, I don't feel strongly about it; here it is all on one line: http://i.imgur.com/xRhom.png
<wgrant> bigjools: The Python? Not at all. cron.publish-*? lol
<gary_poster> benji, whee, let's go to #clojure next! ;-)
<bigjools> wgrant: :D
<wgrant> bigjools: publish-distro/process-accepted work fine on multiple distros.
<bigjools> wgrant: you published debian didn;t you?
<benji> oh I swear!
<wgrant> bigjools: Yes.
<bigjools> cool
<gary_poster> :-)
<bigjools> wgrant: I want to try and set something up so we can spawn off separate runs for publishers for different distros
<henninge> abentley: http://paste.ubuntu.com/575490/
<wgrant> bigjools: Definitely.
<bigjools> wgrant: or DDs will be somewhat interesting
<wgrant> bigjools: This may mean you finally get to rewrite cron.publish-ftpmaster in Python.
<bigjools> wgrant: YES
<wgrant> And cull crap like dsync.
<bigjools> there's a blueprint about adding user hooks to it
<bigjools> I want to make it generic enough to be re-used
<bigjools> and then have a job runner fire off the scripts based on a schedule managed in LP
<wgrant> Right.
<bigjools> wgrant: can you remember how much of the publisher config is ubuntu-specific?
<bigjools> or more to the point did you change much to get debian working
<bigjools> I'm trying to work out if any of it "clashes" if you have >1 distro published
<wgrant> bigjools: Nothing clashes.
<bigjools> sweet
<wgrant> But there are restrictions on where they can be.
<bigjools> yeah I am mostly thinking of paths
<wgrant> eg. archive dependency code assumes that all distros live under http://ftpmaster.internal/$DISTRO
<wgrant> And the publisher assumes that everything lives under /some/common/prefix/$DISTRO
<wgrant> But assuming you live within or lift those constraints (paths can be easily changed using separate configs, URLs are harder), there are no conflicts.
<bigjools> can't solve this with config
<bigjools> needs to come from the DB
<wgrant> Right.
<wgrant> It will need to be sort of like lucilleconfig, except with one path instead of 7 plus 2 other values that never change.
<bigjools> I want to remove as much config as possible and have a management form
<bigjools> "sort of like lucilleconfig
<bigjools> "
<bigjools> was your mouth out
<bigjools> wash*
<bigjools> jeez, type man
<wgrant> Also possibly without the ini data.
<wgrant> I guess all you need now is a single path, and possibly a machine identifier.
<wgrant> And some way to define the base URL for buildds to use.
<wgrant> bigjools: Anything else?
<bigjools> wgrant: I can't think off-hand, more will come to light when we work through it
<wgrant> Yeah, it will be interesting to see what falls out.
<bigjools> migrating to the DB will be easy enough
<wgrant> The publisher config is still a little bit insane, but it's much better than it was.
<wgrant> Anyway, night.
<maxwell69> wgrant: I paste it to have a clear view : http://paste.ubuntu.com/575507/
<wgrant> maxwell69: 'bzr revert sourcecode/Makefile'
<wgrant> And it nears 2am, so I hope somebody else can help you if you have further issue :)
<wgrant> issues
<henninge> abentley: got a bit distristracted but could try it now
<henninge> abentley: that solved it, thanks.
<bigjools> wgrant: Lucille is totally gone now right?
<bigjools> LucilleConfig I mean
<bigjools> meh you're probably asleep
<jcsackett> rvba: don't know if you saw, but per your answers and bigjools i've marked your MP r=me.
<rvba> jcsackett: yes I saw thanks, I'm waiting to be setup with PQM and I will ec2 land
<jcsackett> rvba: cool stuff. :-)
<jcsackett> where does lp.dev put oops files?
<bigjools> in the log directory I think
<jcsackett> leonardr: ping.
<maxb> Is there a schedule for when staging restores happen?
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: bac | https://code.launchpad.net/launchpad-project/+activereviews
 * sinzui reconciles stable with db-devel
<dobey> bah netsplits
<LPCIBot> Project devel build #504: FAILURE in 4 hr 44 min: https://hudson.wedontsleep.org/job/devel/504/
<sinzui> pqm hates me. It wont except my branch to fix db-devel and stable.
<sinzui> What is wrong with my message: "[testfix][rs=sinzui] Resolve conflicts with stable."
<sinzui> I have removed [testfix] and [rs=sinzui] and I still get a failure message with a 0 byte rejection log :(
<dobey> hmm, LP recipe build request dialog seems to be broken. it immediately disables the series i want to build for with a "(Pending build)" comment, even though there are no pending builds. :-/
<jelmer> dobey: it seems like a regression that was introduced in the last day or two
<jelmer> bug 728789
<_mup_> Bug #728789: "Request build" dialog indicates builds are pending when they are not and doesn't allow new builds <recipe> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/728789 >
<dobey> jelmer: ok thanks
<sinzui> I haet pqm
<sinzui> hate, dislike, despise.
<jelmer> dobey: fwiw the API still works
<dobey> jelmer: right, except for the part where it has a quota. and that i'd have to write a script to request specific builds :)
<jelmer> dobey: http://pastebin.ubuntu.com/575664
<jelmer> dobey: the web UI and the API have the same quota afaik
<dobey> jelmer: maybe, but my bot runs as a bot user, and on the web i'm me, so it's not so bad. can we get rid of the quota, or raise it significantly or something?
<dobey> and yeah, i've written the code before for tarmac. i guess i should put a script in lptools for requesting builds
<jelmer> dobey: I wouldn't object to that :) the current quota mechanism is too rudimentary
<benji___> wgrant: am I reading https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus#Cowboys%20%28requires%20IncidentReport%29 correctly in that both listed there have been landed and do won't have to be re-cowboyed on the next release?
<jcsackett> benji: there's a good chance wgrant is actually here. it's early saturday morning for him.
<jcsackett> s/is/isn't/
<benji> jcsackett: I forgot to check my time zone decoder ring
<jcsackett> if the usual lp deployment process is being followed, yes, the cowboys are already part of the main codebase, so you should be fine.
<gary_poster> matsubara, hi.  Does the devpad sync from the losas help the oops tool respond any faster?  It doesn't seem to, but maybe I am being impatient. :-)
<matsubara> gary_poster, a little bit. you can request the sync from the prod servers to devpad but then have to wait the cronjob that loads the oops from the filesystem into the db
<matsubara> that cronjob runs every 7 min
<gary_poster> ack, thanks matsubara
<matsubara> np
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
<wgrant> benji: Yes, they landed weeks ago. No need to worry about them this time.
<lifeless> wgrant: so to measure lpnet latency
<lifeless> wgrant: use the icing
<wgrant> lifeless: That is a very good idea.
<lifeless> wgrant: grab that a few times, gives you a baseline
<benji> wgrant: thanks for the info
<wgrant> lifeless: Seems to be 320ms, as expected. So the DC latency is often <50ms, it seems.
<lifeless> (now)
<wgrant> Right.
<lifeless> opstats would let us graph that
<lifeless> except ssl
<lifeless> bwah
<lifeless> I wonder, whats our fastest db page
<lifeless> also
<jelmer> lifeless: launchpad.net/404 ? :-P
<lifeless> whats the open id sso service times
<lifeless> jelmer: *db* page
<jelmer> lifeless: What do you mean with database page then ? a 404 accesses the database
<lifeless> it doesn't show anything useful though
<lifeless> its not a successful request
<lifeless> wow, page id fail... :createPPA
<lifeless> (https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-monthly-pageids.html)
<wgrant> Firefox is not a fan of that page.
<lifeless> its the svg
<wgrant> Hah. It takes 7 minutes to write out all 142000 records in the natty binary file lists at once, or 10 minutes per batch of 10000.
<wgrant> The limit and offset must be fairly expensive...
<lifeless> temp table time
<lifeless> except don't do that cross xaction
<wgrant> It's due to a compound cross-table sort key.
<wgrant> I think I might reduce it to a single one, add a buffer, and then do the rest in Python.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews | PQM: Release Critical RM: benji
<wgrant> Real parallel a-f reduces what is a 16 minute process on cocoplum to a 5 minute process on mawson. Excellent.
<maxb> I've just clicked in a join request for ~loggerhead-team, with the desire to be able to re-open merge proposals for branches de-merged by the shuffling off of recent trunk to the experimental branch
#launchpad-dev 2011-03-05
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #505: FIXED in 5 hr 17 min: https://hudson.wedontsleep.org/job/devel/505/
<lifeless> wow
<lifeless>  Hard / Soft  Page ID
<lifeless>      111 /    0  LanguageSet:CollectionResource:#languages
<lifeless>      104 /  201  BugTask:+index
<wgrant> Good to see +copy-packages up there, too.
<wgrant> Because I have a fairly major improvement.
<lifeless> I'm just happy bug bloody task is not #1
<wgrant> Heh.
<StevenK_> lifeless: I think about this on the way home last night -- do you still feel memcache has something to give us?
<lifeless> yes
<lifeless> in particular the front page blog posts are (somewhat) sensibly storedin memcache
<lifeless> its not hthe olnyway of doing that
 * StevenK_ works on an EBS volume for Jenkins
 * StevenK fails to see how to convince Jenkins to attach the EBS store
<wgrant> :(
<StevenK> I have created it and populated it, though
<wgrant> Hmm.
<jam1> wgrant: are you still around?
<jelmer> wgrant: 35k tests in ~15m, how many is launchpad at at the moment ? :)
<wgrant> jelmer: :(
<wgrant> jam: Not often at 3am, sorry.
<jam> wgrant: well, you had been around at about midnight, then :)
<jam> of course, it is midnight here now
<wgrant> jam: Shh.
<jam> wgrant: anyway, I mostly wanted to know if you could ec2land the lp-child-hangup patch, since benji has okayed it.
<wgrant> Just saw that.
<wgrant> That's great.
<wgrant> He hasn't actually r-c'd the MP yet, but I will ec2test it so we can land as soon as he does.
<wgrant> jam: Could you merge devel into that branch?
#launchpad-dev 2011-03-06
<jam> wgrant: just merged devel 12533, and pushed. No conflicts, fwiw.
<jam> now, time for me to go sleep
<jam> hopefully benji will comment on the proposal soon, so we can get it landed before downtime.
<jam> but, probably not today :)
<benji> jam: is this the email I replied to about 12 hours ago?
<jam> benji: right. I guess you need to put your Release Manager stamp on the merge proposal itself
<jam> at least, that is what I gleaned from wgrant's comment
<benji> I can do t
<benji> hat
<benji> (If someone can tell me how.)
<jam> don't look at me :)
<benji> heh
<benji> jam: which MP is it?
<jam> https://code.launchpad.net/~jameinel/launchpad/lp-serve-child-hangup/+merge/51893
<jam> is the lp proposal
<jam> and https://code.launchpad.net/~jameinel/lp-production-configs/enable-forking/+merge/52297
<jam> is the config change
<benji> ok, let me see if there's something obvious I can do to them
<wgrant> benji: Review with type 'rc', or 'release-critical'. I forget which.
 * wgrant checks.
<benji> cool, that's along the lines of what I was thinking
<wgrant> jam: I'd already done that locally, so it's already in ec2.
<wgrant> I'll lp-land it once it succeeds.
<wgrant> lib/devscripts/autoland.py:        if 'release-critical' in review_type:
<benji> cool, release-critical it is
<wgrant> Thanks.
<benji> done
<benji> I'm going away now, but much like he-who-must-not-be-named, if you mention my name I'll find you whereever you are.
<wgrant> Heh.
<wgrant> jam: I won't land the prod configs change until we have confirmed LOSA availability for the rest of the day of the deployment.
<wgrant> Just in case.
<wgrant> Since we had no spm last week.
<jam> wgrant: sounds reasonable
<LPCIBot> Project devel build #508: FAILURE in 15 min: https://hudson.wedontsleep.org/job/devel/508/
<wgrant> bzrlib.errors.InvalidHttpResponse: Invalid http response for https://xmlrpc.launchpad.net/bazaar/: Unable to handle http code 502: Bad Gateway
<wgrant> That's not good.
<StevenK> Whee
<StevenK> Slave deleted
<wgrant> It wasn't a slave fault.
<wgrant> But OK.
<StevenK> I know, just want to certain it doesn't affect a later build
<wgrant> Ah, true.
<StevenK> I can check out lp:difftacular locally
<StevenK> Ah, it was a change. Let me schedule a build.
<lifeless> ola
<wgrant> Hi lifeless.
<lifeless> epic lounge time
<wgrant> SYD?
<lifeless> yeah
<lifeless> hmmm
<lifeless> I think tomorrow I'll add the new appserver instances, patch oops-tools to not truncate the lists of incidents and iterate on my bug 727560
<_mup_> Bug #727560: Archive:EntryResource:getPublishedSources <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727560 >
<wgrant> List of incidents? You mean list of exceptions?
<lifeless> wgrant: hows our in dc latency going ?
<lifeless> wgrant: yeah, the 'X of Y' in the headings.
<lifeless> oh, and with the report not truncating, drop the soyuz specific report.
<wgrant> lifeless: Up to 100ms.
<wgrant> lifeless: Yeah, I think we can probably do that now.
<lifeless> mmm, still tolerable.
<wgrant> The main report will become a lot shorter this week.
<lifeless> not not brilliant
<lifeless> wgrant: it doesn't really matter how long it is.
<wgrant> lifeless: Bug #730011
<_mup_> Bug #730011: ~registry should be able to unsuspend accounts <chr> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730011 >
<wgrant> Can you see any reason not to allow ~registry to traverse to suspended accounts?
<lifeless> possibly
<lifeless> it would depend on the style of suspension
<lifeless> resurrect a 'banned cause a bot was spamming via your account' -> fine
<lifeless> suspended cause they were a dirty dirty hacker -> fine IFF we make the cause /realllllly clear/
<wgrant> Right.
<lifeless> suspended if they used legal action to get all details removed and DPA stuff ... no
<lifeless> or at least, I'd want legal to comment on the third
<lifeless> I think if you get charlie and elmo to agree, that its pragmatically fine.
<lifeless> remembering that non-canonical are in ~regstry too
<lifeless> we may have 0 of the third case and not handle them in the normal way anyhow
<lifeless> which would make the concern irrelevant
 * lifeless just fixed 7 oops a day
<wgrant> Oh?
<lifeless> https://wiki.ubuntu.com/UbuntuMediaCenter?action=diff&rev2=50&rev1=49
<wgrant> Hah.
<wgrant> They're not real OOPSes.
<lifeless> well
<lifeless> they are within the set of 'canonical' web properties that its useful to report on bad links
<lifeless> so I don't feel comfortable deleting them
<wgrant> True
<lifeless> and another 9
<lifeless>  10 NotFound: Object: <Product at INSTANCE-ID>, name: u'latest-bugs.atom '
<lifeless>    GET: 10 Robots: 0  Local: 10 Most Common Referrer: http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%20
<lifeless>      10 http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%20 (Unknown)
<lifeless> I wonder if thats real, or bad data *within* lp
<wgrant> There are a few other odd ones like that.
<lifeless> which will be a pita to correct because its a users data
<lifeless> erm
<lifeless> wtf
<lifeless> http://answers.launchpad.net/index.php
<lifeless> GET: 12 Robots: 0  Local: 12 Most Common Referrer: http://answers.launchpad.net/index.php
<wgrant> eg. '%20source' instead of '+source' sometimes, as a referer.
<wgrant> Which is clearly fake.
<wgrant> But why.
<lifeless> hack attempts ?
<wgrant> Presumably.
<wgrant> But why.
<lifeless> because we're running php ...
<wgrant> Oh, the index.php one is clearly a hack attempt, yes.
<wgrant> But the other s/+/%20/ probably aren';t.
<lifeless> you're talking https://launchpad.net/ubuntu-russian-guide/lucid-guide/lucid-guide-1.0/%20download/ ?
<lifeless> ok, flight time
<lifeless> ciao
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #509: FIXED in 5 hr 7 min: https://hudson.wedontsleep.org/job/devel/509/
<lifeless> bah, frozen
<lifeless> we'll see how this db deploy goes, I think we're probably ready to move freeze to monday morning
<lifeless> wgrant: lp:~lifeless/launchpad/bug-727560 has had all the tests that failed in my ec2 run fixed
<wgrant> lifeless: We almost ended up having to delay it to then anyway.
<lifeless> because of the conflict?
<wgrant> Yes.
<wgrant> And we were running out of mbarnett.
<lifeless> say its not so
<wgrant> I think we should freeze on Monday morning and possibly aim to unfreeze by the next morning.
<lifeless> 5 hours should be all we nned
<lifeless> direct landing to devel
<lifeless> one buildbot
<lifeless> deploy to qas.
<lifeless> test
<lifeless> wgrant: I think we should freeze closer, too - but have been taking incremental steps
<lifeless> less risk
<wgrant> Sure.
<wgrant> I think the future will become more clear after a couple more normal releases.
<wgrant> 11.02 was rather anomalous.
<lifeless> we need to be able to handle that too though
<wgrant> Do we?
<lifeless> there will always be exceptions and surprises
<lifeless> we need to have a way to cope
<wgrant> There will.
<wgrant> But they can be handled by reversion, if we make the window too narrow.
<lifeless> be that 'don't merge that patch' or 'don't do a deploy' or whatever.
<lifeless> well, there's no guarantee that reverting some patch P out of a sequence N >= P items long will be itself regression free
<lifeless> I'm not trying to say /how/ we should handle it
<lifeless> but we need to think about how to handle it
<wgrant> Right.
<lifeless> avoiding the issue is one way, but tends to be rigid and expensive most of the time
<lifeless> having a fallback plan is a different way
<lifeless> like a second deploy date booked 2 days later
<lifeless> a third is much more regular db deploys
<wgrant> Lots of patches are short and suitable for regular DB deploys.
<wgrant> 11.02 was not.
<lifeless> the problem isn't (usually) the patch size
<lifeless> its doing the deploy itself.
<lifeless> readonly mode is /expensive/ to invoke
<lifeless> we can't do readonly mode more than once every 3 days
<lifeless> not reliably
<wgrant> Why not?
<wgrant> The rebuild?
<lifeless> because after doing readonly mode the readonly replica has to be rebuilt
<wgrant> True.
<lifeless> 11.02 was fine patch size
<wgrant> Although that only take 24hourish.
<lifeless> the problem there was stale locks on the sso replica
<wgrant> Estimates said that the patches would almost exhaust the window.
<lifeless> estimate was on crack
<wgrant> They were in fact much shorter, but that's not what we thought before hand.
<lifeless> no, they weren't.
<lifeless> the estimate was inflated by calculating slony slaves as linear sums
<wgrant> Hence they were shorter than the estimates.
<lifeless> but ddl propogates to the slaves same as normal commits - in paralle.
<lifeless> well, depends on your definition.
<lifeless> if you run the right formula on the original patch time, the predicted time was right no
<lifeless> *on8
<wgrant> Sure.
<wgrant> But the original patch time is not the interesting one.
<lifeless> what is?
<wgrant> The final value that tells us how long the window needs to be.
<lifeless> I don't understand why you prefaced your prior assertion with 'but'.
<lifeless> given I too was talking about the predicted time.
<lifeless> I think we're both pretty clear on the constraints
<lifeless> http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ looks nice
<lifeless> gnight
<wgrant> Night.
<wallyworld> lifeless: are you able to +1 on this mp now that i've exported to the correct version as requested: https://code.launchpad.net/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/52004
<lifeless> wallyworld: actually I'm thinking we probably shouldn't export it at all
<lifeless> wallyworld: its entirely redundant with a bug search for branch=X
<lifeless> wallyworld: I dunno
<wallyworld> lifeless: i have no real opinion on it. it just seemed like something that people would want to do.
<StevenK> wgrant: Have you seen the OOPS report today?
<lifeless> StevenK: do you mean sundays? tordays is 3 hours away
<StevenK> Likely
<wgrant> StevenK: You mean the QISEs that started on Friday?
<StevenK> wgrant: Indeed.
<wgrant> Basically, delayed copies suck.
<wgrant> And are completely unnecessary.
<wgrant> Particularly with the branch I am about to propose.
<StevenK> wgrant: Oh, do you have a nice suggestion of how I can take a .dsc in the tree and import it into the librarian and create publishing history for it?
<wgrant> If we have an spm today we can reject those two PUs.
<wgrant> StevenK: Why?
<wgrant> Testing the changelog populator?
<StevenK> Right
<wgrant> StevenK: I'd to it manually.
<wgrant> makeSourcePackagePublishingHistory probably doesn't create any SPRfs.
<wgrant> So you can add them yourse.f
<wgrant> And I cannot type.
<StevenK> Haha
<StevenK> More caffeine? Less caffeine?
<lifeless> wgrant: fyi (cause I know you're interested) - https://code.launchpad.net/~lifeless/lp-production-configs/pending/+merge/52345
<lifeless> how does one get a bug notification level of nothing ?
<wgrant> lifeless: I think you need the advanced subscriptions UI :(
<wgrant> That's why I haven't already done that
<lifeless> ditto tims patch I guess ?
<lifeless> wgrant: you could give yourself that on dogfood, for the former
<wgrant> Ah, true, Julian's not around.
<lifeless> wgrant: him being around would interfere?
<wgrant> He would go on a rant about how it's only for Soyuz testing :P
<lifeless> its a team qa resource
<lifeless> wgrant: so, want to file a new bug via email with 32K of text or whatever? [I still haven't setup dkim on my domain yet]
<wgrant> lifeless: My DKIM isn't trusted, but I can gpg-sign it...
<lifeless> wgrant: yeah
<lifeless> then we can get spm to run process-mail
<lifeless> and we should get a reject in the staging mailbox
<wgrant> I think it runs regularly, actually.
<wgrant> Judging by the number of OOPSes it generates.
<wgrant> On staging, at least.
<wgrant> I will spam both.
<StevenK> Hm
<StevenK> You'd think we'd have a helper in the test suite for "Here, punt this file into the Librarian"
<wgrant> StevenK: Why do we need a helper?
<wgrant> The API is not that terrible :)
<StevenK> Clearly, I'm failing to read Python this morning, and require tea.
<wgrant> lifeless: I'm currently making Bugs usable on mawson again with the same hack we used on qas.
<wgrant> But it's taking a while.
<thumper> lifeless: can I get you to mentor two reviews by wallyworld?
<lifeless> wgrant: what was that ?
<lifeless> thumper: sure
<thumper> lifeless: https://code.edge.launchpad.net/~thumper/launchpad/mail-header-oops/+merge/52156 and https://code.edge.launchpad.net/~thumper/launchpad/strip-email-attachment-path/+merge/52159
<thumper> damn edge
<thumper> I just won't leave my browser memory
<thumper> s/I/It/
<wgrant> UPDATE bugmessage SET index=id WHERE index IS NULL;
<lifeless> wgrant: we ran the indexer on qastaging :)
<wgrant> lifeless: There were 3000 unindexed messages there when we tried to upgrade it on Saturday.
<wgrant> Production is still fine.
<wgrant> So I suspect the indexer stopped too early.
<lifeless> ><
<lifeless> ok
<wgrant> Error message:
<wgrant> The description is too long. If you have lots of text to add, use an
<wgrant> attachment instead.
<wgrant> Excellent.
<lifeless> woo
<lifeless> qa-ok ?
<wgrant> Just waiting for a successful one to go through.
<lifeless> http://webnumbr.com/launchpad-oops-bugs.all
<StevenK> Where's spiv with his perspective graph? I liked that one better.
<wgrant> lifeless: qa-ok
<StevenK> wgrant: And you also did r12527?
<wgrant> StevenK: i've just given up waiting for mawson.
<wgrant> Going to live with the breakage.
<thumper> lifeless: thanks for the review mentors.
<thumper> lifeless: got any quick pointers to fake librarian usage?
<thumper> also...
<thumper> anyone remember the magic commands to add a loopback delay (and to clear it again) ?
<StevenK> Rargh, LP's icon is broken still
<lifeless> StevenK: where?
<StevenK> lifeless: On any page -- last time, wgrant and I nailed it down to the librarian returning an incorrect content-type
<lifeless> thumper: uhm, just grep for FakeLibrarian, its pretty self explanatory
<lifeless> StevenK: ah, yes
<StevenK> https://bugs.launchpad.net/launchpad/+bug/643224 is where I noticed it
<_mup_> Bug #643224: dkim whitelist should be in configuration <dkim> <feature-flags> <lp-foundations> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/643224 >
<thumper> ok
<lifeless> we should probably critical this
<StevenK> And does the font used have to change every day?
<StevenK> It's getting annoying.
<wgrant> At the moment, yes.
<StevenK> lifeless: Do eet
<lifeless> oh man
<lifeless> this is going to be fugly.
<wgrant> Bug heat, please die.
<wgrant> lifeless: We are deployable and releasable, yet disappointingly LOSA- and RM-less.
<StevenK> Well, it is Sunday for BjornT
<StevenK> DOH
 * StevenK glares at irssi
<lifeless> I'm thinking its time to ditch rm in a month or two
<wgrant> I'd say so.
<StevenK> And do what instead?
<lifeless> StevenK: there's approximately zero to do.
<StevenK> I disagree -- given the fun wallyworld had last rollout, for example
<lifeless> StevenK: the only nontrivial thing remaining is blessing additional patches during freeze, and thats /only/ needed for fixes to DB patches which were incorrectly qa-ok in db-stable, or which break when combined with the tip of devel.
<StevenK> Last rollout is not what I meant, but you see what I mean
<lifeless> StevenK: that had no need to be centralised
<wallyworld> there needs to be a point of contact that everyone knows about - as someone said to me, it's like herding cats
<lifeless> wallyworld: there does?
<wallyworld> i think so
<lifeless> why?
<wallyworld> if/when something goes wrong, there needs to be someone (the rm) accountable for liasing with whoever is needed to get it sorted out
<wallyworld> also to chase up people for qa etc
<wallyworld> to nominate what db-devel rev no will be used
<lifeless> point by point
<wallyworld> etc
<lifeless> losas are doing the deploy, they are the only sane contact point to initiate any fubar
<lifeless> qa chaseup is not needed - if its not qaed, its not in the deploy.
<lifeless> the nomination of revisions is trivial, the report plus a quick eyeball over it can be done by anyone.
 * thumper scratches his head
<wallyworld> re qa: but what if there's a rev holding stuff up and the person just needs reminding or even the qa could just be done by someone else
<StevenK> wallyworld: And yet you complain when wgrant does your QA? :-)
<wallyworld> yes but "done by anyone" usually means "done by no-one" unless "someone" is accountable
<wallyworld> StevenK: only because there's certain things i want to test :-)
<lifeless> wallyworld: no, this is a false meme
<lifeless> its possible to have a culture where team accountability matters
<wallyworld> StevenK:  but close to a release, when stuff is being held up by a qa delay, you just want to get it done
<StevenK> And I'm sure everyone is aware of the deployment report.
<lifeless> in this specific case -
<lifeless>  : if its not qaed, the pipeline stalls but the earlier stuff still gets released.
<lifeless>  : if its not qaed and there is a revision after it that someone else wants included in the deploy, they have motivation to qa the earlier revision.
<StevenK> lifeless: Er, but that doesn't work?
<StevenK> Ah
<lifeless> StevenK: by doesn't, you mean 'does'.
<lifeless>  : if its not qaed, and there is nothing after it and the person that landed it doesn't care and noone else cares - then clearly the change isn't important enough to worry about us not having it in the deploy.
<StevenK> lifeless: I was going to point out that we by definition want to deploy tip of stable since we merge db-devel into it
<wgrant> StevenK: That was a matter of some contention last time.
<wallyworld> but i maintain that qa is often best done by the person who wrote the code since they know the specific in's and out's and corner cases etc that need to be tested
<lifeless> StevenK: we only merge teh qa-ok'd revisions of db-stable, we *do not* merge db-devel.
<lifeless> wallyworld: then again, all the other devs have motivation to talk to them.
<wgrant> wallyworld: I agree, but sometimes the pipeline is blocked and minimal QA is better than blockage.
<lifeless> wallyworld: and if the change looks hard to qa, others can roll it out.
<lifeless> *back it out*
<lifeless> there are lots of options that don't depend on a single person nagging
#launchpad-dev 2012-02-27
<wgrant> Hm
<wgrant> BugSubscription.date_created has tzinfo=UTC
<wgrant> I thought we mostly used naive UTC datetimes?
<wgrant> Hm, it seems to vary.
<nigelb> Morning
<StevenK> O hai
<StevenK> wgrant: What do you think of https://code.launchpad.net/~stevenk/launchpad/db-add-audit-spph and https://code.launchpad.net/~stevenk/launchpad/spph-approved_by-model as two thirds of bug 885739?
<_mup_> Bug #885739: queue and override manipulations should have an audit trail <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/885739 >
<wgrant> StevenK: Ah, I was going to talk to you about those.
<wgrant> StevenK: I think this is another patchwork thing.
<StevenK> wgrant: Oh?
<wgrant> StevenK: This is the sort of thing that should probably be on PU
<StevenK> Right. But it's a bit nasty to get at PU from SPPH, isn't it?
<wgrant> But it's difficult because none of this has been designed since 2005, and there's now 5x more of it.
<StevenK> I agree PU is probably the better place for approved_by
<StevenK> overrride_changed_by belongs on SPPH, and I figured out that the date was utterly pointless since a new SPPH is created.
<wgrant> We probably want something like ArchiveAudit which BPPH and SPPH reference.
<wgrant> It records all the sponsorship etc. crap, along with the PU if present.
<wgrant> It also records the details of the operation, for example the fact that the overrides were changed.
<StevenK> Heh, you mean like SPPH.spondor ? :-)
<wgrant> Rather than making SPPH even more obese for no good reason.
<wgrant> Yes.
<wgrant> That totally doesn't belong on SPPH.
<StevenK> Agreed
<StevenK> wgrant: So, can has mumble about ^ ?
<wgrant> I thought Launchpad had a policy against doing Soyuz correctly.
<nigelb> lol
<wgrant> The idea is that $randomperson just adds more columns to random places in the schema until things sort of work.
<wgrant> StevenK: Anyway, can mumble in a few.
<wgrant> Just finalising some triggers now.
<wgrant> Delicious, delicious PL/pgSQL
<StevenK> Haha
<nigelb> So, that's what wgrant eats for breakfast. Code.
<wgrant> My secret is revealed.
<StevenK> wgrant: A few what?
<StevenK> :-)
<wgrant> Hmm, my window decorations have gone missing.
<huwshimi> wgrant: I'm starting to feel like I'm going to need a hardware "unity --replace" button
<StevenK> lifeless: O hai
<wgrant> huwshimi: About half the time my kernel hangs, but the other half I just Ctrl+Alt+F1 and do it there.
<huwshimi> wgrant: Yeah, that seems about the same as me
<StevenK> wallyworld_: Would you mind +1ing https://code.launchpad.net/~stevenk/launchpad/kill-spph-ancestor/+merge/94716 ?
<wallyworld_> StevenK: np, looking now
<wallyworld_> StevenK: so ancestor is never used anywhere? tests or anything?
<StevenK> Hmmm, it is referenced in the factory
<StevenK> Still digging
<wallyworld_> ok, i assume you'll update the branch and then i can +1
<StevenK> wallyworld_: Hah, turns out both wgrant and I were wrong.
 * StevenK deletes the MP, since it's dangerous to land and would cause us to lose data
<wallyworld_> first time for everything
<wallyworld_> wgrant: how close is your multipolicy-3 branch to being merged?
<wgrant> wallyworld_: -50 minutes
<wallyworld_> wgrant: cool. i have a branch that i  need to merge in your changes before i finish
<wgrant> wallyworld_: That is, it landed 50 minutes ago.
<wallyworld_> ah, i read '-' as '~'
<wallyworld_> thanks
<wgrant> So did StevenK.
<wgrant> Silly northerners.
<wallyworld_> we old farts have poor eyesight
<wgrant> Hmmmmmm
<wgrant> Declarative factory next week, I think.
<StevenK> wgrant: Indicies on ArchiveAudit are going to be ... fun
<wgrant> Easy enough.
<StevenK> Oh?
<wgrant> There's only a single date, and everything else is easy.
<StevenK> I want an index for every column?
<StevenK> Aside from date, that is
<wgrant> So, individual indices aren't immensely useful.
<wgrant> Indices for individual columns, that is.
<wgrant> Postgres can sometimes do a reasonable job by ANDing them together. But generally you'd create indices that satisfy the queries that you want.
<wgrant> Which is, in this case, none.
<StevenK> I was attempting to think ahead
<wgrant> Until we want to run queries over that table, single-column indices on id and the person FKs are what you want.
<wgrant> (just enough to satisfy the foreign key constraints in sensible time, since we don't know what sort of queries we want otherwise)
<lifeless> StevenK: hi, I'm on leave today; if its urgent please give me a ring, otherwise ping tomorrow
<StevenK> lifeless: Tomorrow is fine
<StevenK> wgrant: Can I not create a table and add comments to comment.sql in the same patch?
<wgrant> StevenK: You can.
<StevenK> psycopg2.ProgrammingError: schema "archiveaudit" does not exist
<wgrant> You're doing it wrong.
<StevenK> Indeed
<wgrant> Did you forget to s/TABLE/COLUMN/?
<StevenK> Perhaps :-)
<StevenK> wgrant: And I suspect we can ignore adding perms onto archiveaudit until we start to use it
<wgrant> No
<wgrant> The person foreign keys mean that there has to be SELECT and UPDATE on archiveaudit to the person merger.
<lifeless> fk's
<StevenK> Which runs as launchpad_main?
<wgrant> Hell no.
<StevenK> I see it
<StevenK> lifeless: Go away, you're on leave.
<stub> Reviewing PL/SQL code doesn't look any more fun after second coffee.
<StevenK> Haha
<stub> StevenK, wgrant: For person merge, the tests unfortunately still run as launchpad_main I think, and there is a 'person-merge-job' user or similar that is actually used on prod.
<StevenK> That is disappointing.
<StevenK> A quick bzr grep agrees, though.
<StevenK> Hmmm, is bug searching horrible again?
<wgrant> StevenK: No, should be better than usual, for a couple of reasons.
<StevenK> wgrant: Keeps timing out for me
<wgrant> OOPS ID?
<wgrant> Large listings are certainly not snappy, but then the dynamic stuff never has been.
<StevenK> wgrant: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-732cfe7bfe2e69be0875af0907fee4e1
<wgrant> 2.6s here
<wgrant> StevenK: Still happening?
<wgrant> I've been idle long enough that I'm on a slave.
<wgrant> You're still on the master.
<wgrant> Which may be relevant.
<StevenK> It just worked.
<wgrant> The karma DoS is in progress.
<wgrant> May be relevant.
<StevenK> Heh
<wgrant> (it's something like 2000 writes per second at times)
<wgrant> StevenK: Shouldn't archive, distroseries, pocket be NOT NULL?
<wgrant> Also date.
<StevenK> Good point.
<wgrant> And this shouldn't land until we've got code for everything to populate it.
<wgrant> So we can actually be sure it fits.
<StevenK> Right
<adeuring> good morning
<mrevell> G'mornin'
<wgrant> stub: Thanks.
<wgrant> The DISTINCT ON shouldn't be necessary at all IIRC, except that there's no UNIQUE bugsubscription(bug, person) :/
<stub> If there are multiple people with multiple results in there, you might get multiple (bug,person) rows intermixed I think.
<stub> oic.
<wgrant> The join should be unique already, except that the origin table is invalid because of the lack of unique constraint.
<wgrant> (and there are violations)
<stub> There might have been a time we did allow multiple subscriptions from a person. I wonder if person merge can still create them?
<wgrant> I believe that's how they all came about.
<wgrant> I suspect it's an oversight rather than a feature.
<stub> I'd go with the ORDER BY just in case as this is a temporary trigger
<wgrant> stub: Indeed, wasn't suggesting otherwise :)
<wgrant> I just added that as a quick hack to workaround the bad data and forgot to make it deterministic.
<jelmer> 'morning launchpadders
<danhg> Morning jelmer :)
<nigelb> HI jelmer!
<wgrant> czajkowski: Those bugs are on some project that someone randomly added to launchpad-project.
<wgrant> czajkowski: We should get a webops to remove it from the group.
<czajkowski> wgrant: context ,ooking at a lotta bugs this morning and trying to work them out
<wgrant> Ah, it was nearly two hours ago now.
<wgrant> You triaged three bugs on gazungaos
<czajkowski> wgrant: nods the kubuntu folks
<czajkowski> wgrant: but yes removing it from the group would be good also
<czajkowski> thanks sorry being thick, lotta traiging and back and forthh going on this mornign
<wgrant> Hm? Not Kubuntu.
<wgrant> It's some random project unrelated to Ubuntu.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugtasks: 4*10
<czajkowski> where does the ubuntu bug pull information from for the twitter feed?
<deryck> adeuring, https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<danilos> flacoste, lifeless: hi, I wonder if it would be possible to export a Sprint API function that allows registering people by email address and calls ensurePerson() on that email? we'd want this to integrate better with our summit registration form which already collects a bunch of data about people
<abentley> adeuring: lp:~abentley/launchpad/lazr.jobrunner
<flacoste> danilos: well, we don't want to add the ability to register people in Launchpad without their confirmation
<flacoste> danilos: so that would either need to be limited to a very specific user
<flacoste> danilos: or that API should email the user to ask for confirmation
<danilos> flacoste, right, understood, thanks (that'd be a a simple solution for our integration of registration form/launchpad/summit), but I guess we'll have to fix summit as well :)
<danilos> flacoste, basically, we have a requirement that whoever registers for our conference has a Launchpad account, but we could only get to the point of them having SSO account
<danilos> flacoste, we'll have to point them to log into Launchpad before we allow them to register
<flacoste> danilos: that's interesting
<jml> I'm waiting on Launchpad to acknowledge my recent 'push'
<flacoste> danilos: maybe you could add a ensurePersonFromSSO that would create a LP profile for someone with a SSO account
<jml> I've been waiting for some time now
<flacoste> jml: interesting, i think abentley landed some change to the branch scanner
<flacoste> might be related, although i'm not sure if they were deployed
 * flacoste checks
<flacoste> yep, was rolled out
<flacoste> yesterday
<abentley> jml: what branch?
<jml> abentley: lp:~jml/pkgme-service/directory-already-exists
<abentley> jml: So I see pushes from 30 minutes ago, but still "Updating branch".
<jml> abentley: yeah, I'm waiting for r73
<abentley> jml: looking...
<abentley> jml: It appears we were just slow.
<jml> abentley: thanks.
<abentley> jml: There's been pretty heavy traffic on the branch scanner today.  There was a block of 8 jobs that started at 15:05:10 and finished at 15:10:25.  I'm guessing you pushed in that timeframe.
<abentley> jml: The next block was a block of 30 jobs that ran from 15:11:09-15:13:43.
<abentley> jml: then a futher block of 13 from 15:14:09-15:17:54.
<jml> abentley: ah, ok.
<danilos> flacoste, that'd work for us, should we go that route perhaps?
<flacoste> danilos: yes, i think so, the complex bit is in finding out if the email address has a SSO account
<danilos> flacoste, right, we can't rely on account table anymore
<danilos> flacoste, do you have any idea on how we could do that? would we need something on the SSO side as well?
<flacoste> danilos: yes, there might be an existing api for this on canonical-identiy-provider
<danilos> flacoste, ok, I'll check if there's something along the lines
 * czajkowski glares at launchpad and wonders why it has to misbehave! 
<nigelb> czajkowski: Remember to ask StevenK about how launchpad is made. he has that long line about everything being held together by strings and jaffa tape.
<flacoste> gary_poster: any reason why buildout doesn't use the versions specified in setup.py (vs the one in versions.cfg)?
<czajkowski> flacoste: evening, good weekend?
<flacoste> hi czajkowski, yes, it was great, went to party at a friend's new appartment on Saturday, how about you?
<czajkowski> good thanks, nice weekend went to see a rubgy game, and now working on community council work after dinner
<flacoste> nice
<czajkowski> yup the weather is actually improving :D
<gary_poster> flacoste, not sure, but I think versions.cfg is applied afterwards, so it wins.
<flacoste> gary_poster: ah, right, and you usually don't want to pin in setup.py, so that makes sense
<wallyworld_> StevenK: wgrant: jcsackett: the trains are not running today and i have to help out my neighbour by driving her kids to school so i'll miss the standup
<cr3> hi folks, how do you assign the on call reviewer?
<lifeless> flacoste: hi
<flacoste> lifeless: hang out?
<lifeless> sending you an invite
<lifeless> flacoste: it should be in your stream
<flacoste> lifeless: you invited my canonical.com account?
<lifeless> possibly not :)
<flacoste> lifeless: got it
<cr3> benji: ^^^ since you're the on call reviewer, ping :)
<benji> cr3: we have a schedule, hold on a sec and I'll find it
<cr3> benji: thanks, I'd like to adopt the same process in our team instead of inventing our own :)
<benji> cr3: the schedule is at https://dev.launchpad.net/ReviewerSchedule; you may also be interested in http://bradcrittenden.net/post/358363191/getting-your-code-into-launchpad
<benji> cr3: and this too https://help.launchpad.net/Code/Review
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<cr3> benji: thanks for all the info, much appreciated!
<benji> cr3: my pleasure
<wgrant> lifeless: Why do bugtask inserts take 290ms on prod?
<wgrant> Is bugsummaryjournal very slow?
<flacoste> lifeless: i think your ISP crashed
<lifeless> flacoste: bah my adsl
<flacoste> lifeless: yep, i need to go, i suggest we resume that conversation tomorrow?
<lifeless> wgrant: shouldn't be, is it ?
<wgrant> lifeless: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-6e6b4b29ef635939095468528e92c7ec
<lifeless> wonder if the rollup is honkeydory
<lifeless> nup 0
<wgrant> lifeless: Huh
<wgrant> Can you try an EXPLAIN ANALYZE in a txn?
<wgrant> See which trigger takes the time.
<wgrant> I was worried about my trigger adding 0.4ms sometimes
<wgrant> I see that won't be a problem :)
<lifeless> if you prep a test I will run it for you
<lifeless> today is not a good day
<lifeless> wgrant: is our heat logic uncontended yet ?
<wgrant> lifeless: maxheat is no longer updated or even calculated, and garbo-hourly is doing nothing.
<wgrant> So indeed it should be uncontentious.
<wgrant> bugtask only has three triggers, and bugsummary is the only one that seems like it could really do much.
<lifeless> is the bug private with a lot of subscribers
<wgrant> It's public security.
<wgrant> 11 subscribers
<wgrant> 9 trillion contexts
<wgrant> https://bugs.launchpad.net/ecryptfs/+bug/732628
<_mup_> Bug #732628: TOCTOU in mount.ecryptfs_private <kernel-cve-tracking-bug> <patch> <eCryptfs:Fix Released by kirkland> <ecryptfs-utils (Ubuntu):Fix Released by kirkland> <linux (Ubuntu):Fix Released> <linux-ec2 (Ubuntu):Invalid> <linux-fsl-imx51 (Ubuntu):Invalid> <linux-linaro (Ubuntu):New> <linux-lts-backport-maverick (Ubuntu):Invalid> <linux-lts-backport-natty (Ubuntu):Invalid> <linux-lts-backport-oneiric (Ubuntu):Invalid> <linux-mvl-dove (Ubun
<wgrant> Like 14 source packages
<wgrant> Each with 5 series
<wgrant> sigh, kernel team abusing packages
<kirkland> holy smokes
<lifeless> wgrant: possibly inefficient distro rollup check with each row being in the same distro or something
<lifeless> wgrant: should be able to reproduce locally. I suspect a bugsummary query inefficiency
<wgrant> lifeless: Also the kernel team abusing Launchpad
<wgrant> That bug has 73 tasks :/
<wgrant> Maybe I should introduce a limit of 3 package tasks per distro.
<wgrant> Would only break the kernel team.
<wgrant> But would also fix Launchpad.
<elmo> wgrant: how is that "abuse"?
<wgrant> elmo: It's an order of magnitude more tasks than Launchpad is designed for, because the kernel is duplicated in 13 source packages.
<wgrant> 14, sorry
<micahg> wgrant: would break transition bugs as well
<StevenK> poolie: O hai. A user reported yesterday that http://pad.lv/nnnnn => works fine; https://pad.lv/nnnnn => 404 , is that known?
<poolie> yes
<poolie> i guess making it work but with an invalid cert would be a start
<wgrant> poolie: Or just get a free StartSSL cert?
<wgrant> Not sure if IE likes them, but all non-Windows browsers I've run into too.
<wgrant> s/too/do/
<poolie> oh, can you?
<wgrant> startssl.com
<poolie> i guessed that :)
<wgrant> Their web UI sucks, but it works.
<poolie> is it worth anything?
<poolie> i guess the browser vendors have decided there is a modicum of security
<wgrant> It's probably more secure than most other CAs...
<wgrant> given they all just do hopeless email validation anyway.
<wgrant> With the added bonus of being corrupt.
<wgrant> Ah, they have an IE logo, so looks like even IE likes them now.
<poolie> ok i'll see what i can do
<poolie> feel free to file a bug against pad.lv if you wish
<timrc> are people able to create distributed distributions? I'd like to create a full mirror of beta1 our developers can use to bootstrap their live-build images
#launchpad-dev 2012-02-28
<wgrant> baaaaaaah
<wgrant> No CREATE TABLE IF NOT EXISTS in 8.4
<wgrant> Added in 9.1 :(
<elmo> WGRANT, Y U NO CREATE TABLE IF NOT EXISTS
<wgrant> Heh
<wgrant> lifeless_: the temp journal flush is very slow because it calls a function for each row. Fixing it to do a bulk insert is still fairly slow, because of the bugsummaryjournal FKs and TRUNCATE being slower than DELETE for small tables like this.
<StevenK> wgrant: I'm questioning if my IBug leak fix is really QAable :-/
<wgrant> StevenK: Did the test suite pass?
<wgrant> (yes, it did => qa-ok)
<wgrant> Or untestable
<StevenK> It's on qas, so both ec2 and buildbot did
<wgrant> Indeed.
<lifeless_> wgrant: the flush shouldn't block journal inserts though
<wgrant> lifeless_: The flush is called as part of the trigger.
<wgrant> temp journal flushes to journal within the trigger
<wgrant> Then garbo rolls up the journal
<lifeless_> ah, skimmed over the word temp
<lifeless_> thanks for debugging
<lifeless> not bad - 386 Exceptions
<wgrant> lifeless: 100 more than normal :(
<wgrant> hwdb crap, probably
<lifeless> also 50 from oops-tools thyself
<lifeless> touch 500.html -> -50
<wgrant> Heh
<wgrant> Yeah
<wallyworld_> wgrant: what's the timeframe for the landing of the code to mirror data into the new access policy schema?
<wgrant> wallyworld_: I expect the trigger will land in a few hours (just waiting for a branch to be merged from stable to db-devel)
<wgrant> wallyworld_: But that doesn't include the sampledata updates.
<wgrant> (and we need to create AccessPolicies on prod and dev before the triggers will do much)
<wallyworld_> ok. i have a couple of options to do next. i'll wait for this stuff
<lifeless> just don't use sampledata
<lifeless> its deprecated anyhow
<wallyworld_> i wasn't going to use sample data
<lifeless> ah cool, I was misled :P
<wgrant> lifeless: We have to update the sampledata
<wgrant> Or none of the private bugs will be valid.
<lifeless> wgrant: how many tests use private bug sampledata?
<wgrant> Probably quite enough :(
<lifeless> might be worth a check
<wgrant> sampledata is also still critical for test performance, unfortunately.
<wallyworld_> lifeless: above, i was talking about waiting for the triggers to land, not sample data
<wgrant> wallyworld_: https://code.launchpad.net/~wgrant/launchpad/bug-mirror-access-policies/+merge/94722 is the MP, and it has tests which use normal factory methods.
<wallyworld_> thanks
<lifeless> wallyworld_: yes, I was misled by wgrant bringing in sampledata to the convo
<wallyworld_> np :-)
<wgrant> lifeless: Well, for most purposes this isn't useful without the sampledata updates.
<wgrant> Perhaps for wallyworld it is :)
<wallyworld_> perhaps
<lifeless> wgrant: I would love it if you check and see what the effort to delete the private bugs would be from the sampledata
<wgrant> lifeless: I have a sampledata vs. factory-ng battle experiment planned for next week.
<lifeless> you had me at battle
<wgrant> lifeless: Well, the current factory is reasonably awkward and extremely slow, so I'm going to attempt to improve it.
<wallyworld_> lifeless: there needs to be some tal for the basic page chrome. the other tal that's there was from an earlier branch where john just put in sample data as a placeholder. it's totally replaced at runtime. i still see some value in retaining that
<lifeless> wallyworld_: so, I'd love us to get rid of the duplication
<lifeless> wallyworld_: how is up to you :)
<StevenK> lifeless: Your DSL is still misbehaving
<StevenK> ?
<lifeless> StevenK: I sispect so
<StevenK> wgrant: O hai, Mr OCR.
<wgrant> Lies.
<wgrant> wut
<wgrant> how is it bad
<wgrant> I ran it over THE WHOLE TREE two months ago
<StevenK> Tell me about it
<StevenK> I only thought to because Aaron added a whole of bunch of 'from x import (\nsingle module,\n)'
<wgrant> Bah
<wgrant> One of them is mine :(
<StevenK> I also didn't notice the problematic codehosting import
<StevenK> In the diff
<wgrant> Most of those are now handled.
<wgrant> by a # FIRST comment.
<StevenK> Right.
 * StevenK notes one is wrong and reverts it
<wgrant> Oh?
<StevenK> lib/lp/services/librarianserver/testing/server.py
<wgrant> What's wrong with it, beyond the superfluous canonical import?
<StevenK> It's been moved to the wrong place.
<wgrant> It shouldn't exist in the first place.
<wgrant> There's a few others around too.
<wgrant> I've listed in them in my review that I'm about to finish.
<StevenK> Right
<wgrant> But in general anything importing anything from canonical is wrong, unless it's using __file__ to find icing.
<StevenK> lib/lp/services/librarianserver/testing/server.py makes use of canonical.__file__
<wgrant> Ah, it's possible they all do.
<wgrant> But it seems unlikely.
<wgrant> If they do still need them, fix the formatter to know that canonical is local.
<wgrant> Reviewed, anyway.
<StevenK> I've already pushed the change that does
<wgrant> Hah
<wgrant> Look at how it uses canonical.__file__
<wgrant> 'tis stupid.
<wgrant> Pls fix.
<wgrant> :q
<StevenK> To do what instead?
<wgrant> It uses canonical.__file__ so it can get the root of the tree.
<wgrant> Possibly use lp.__file__ instead.
<wgrant> Most other cases are probably similar.
<wgrant> Might as well wipe them all out now.
<wgrant> canonical.launchpad.__file__ is usually valid, though, as it's used to find icing.
<StevenK> wgrant: What do you think about http://pastebin.ubuntu.com/860097/ ?
<wgrant> StevenK: Looks reasonable.
<StevenK> wgrant: Recommendation for the use of canonical.__file__ in lib/lp/services/config/doc/canonical-config.txt?
<wgrant> StevenK: s/canonical/lp/g
<StevenK> steven@liquified:~/launchpad/lp-branches/format-imports-cleanup% bzr grep canonical.__file__ | wc -l
<StevenK> 0
<stub> wgrant, StevenK: I think config.root gets you the root better
<wgrant> stub: It's testing config.root.
<stub> oh :)
<wgrant> I was going to say the same thing until I actually read the test.
<wgrant> zomg
<wgrant> it works
<StevenK> Which bit?
<wgrant> Bug searches without any joins at all.
<wgrant> (including for security checks, which were the big remaining pain)
<lifeless> wgrant: interesting
<wgrant> Still waiting for DF to finish regenerating the data, but the plan looks good.
<lifeless> whats the schema for that ?
<lifeless> wgrant: ^
<wgrant> Include AccessPolicyArtifact.policy and AccessArtifactGrant.grantee in arrays on BugTaskFlat. Not feasible to index, but a whole lot faster in a scan than joining against BugSubscription.
<wgrant> Because we can calculate the policies and grantees to match against once at the start of the query.
<lifeless> interesting
<lifeless> I'm very glad you're spending time trying different htings
<wgrant> Hmm
<wgrant> Well, that's better than expected.
<wgrant>  Seq Scan on bugtaskflat  (cost=0.00..958352572.67 rows=193926 width=4) (actual time=0.020..970.982 rows=91164 loops=1)
<wgrant>  Total runtime: 1026.971 ms
<wgrant> So even completely unindexed, as long as it's hot it only takes a second to do a full scan.
<wgrant> I think that is a reasonable next victory.
<wgrant> 'cause this table should be pretty damn hot.
<stub> GIN and GIST indexes support array operations. Landscape are using them.
<wgrant> stub: Yeah, but it's not too useful here.
<stub> Not sure if they are the operators you need though (IS IN etc.)
<stub> k
<wgrant> It's possible we can get something out of BitmapAnding
<wgrant> But in general this shouldn't be too bad.
<wgrant> Because the arrays are all small
<wgrant> Normally only one policy, and <5 grantees.
<wgrant> The index may provide significant benefit if you're unable to see most of the bugs, eg. in a private project.
<wgrant> But we'll see.
<stub> Still pulling up 91k rows, which is a lot even if half the estimate
<wgrant> (the above is a COUNT(*) of a default Ubuntu bug search, with bugs visible to me)
<wgrant> So it's meant to be 91k rows
<stub> ok
<stub> And better than what we already have for sure :)
<wgrant> Oh yes.
<wgrant> Oh
<wgrant> And it's still treating the CTEs as correlated.
<wgrant> Huh
<wgrant> It's not meant to do that...
<wgrant> Ah, no it's not, just the top-level.
<wgrant> need more rams
<wgrant> Bah, forgot to include tags in this rebuild.
<wgrant> Hmm
<wgrant> But ideally need stats for tags
<wgrant> Which means forcing them into a tsvector AFAIK :(
<wgrant> Anyway, this will hopefully be fast enough for now, and we can easily garbo up a new fact table online later.
<wgrant> Hmm, under-selective FTI is slow, even with GIN. Possibly just that DF can't really fit the full indexes and tables in cache.
 * wgrant declares the experiment a great success and shall propose it tomorrow.
<lifeless> awesome
<lifeless> wgrant: if you have a test script I can run it on asuka
<wgrant> lifeless: Are you around tomorrow?
<lifeless> a
<lifeless> yes
<wgrant> lifeless: Great, will throw stuff at you then.
<nigelb> 24
<nigelb> ugh
<lifeless> StevenK: perhaps 'make' should format imports
<wgrant> That's a bad, bad precedent I think.
<adeuring> good morning
<wgrant> Lint could, though.
<wgrant> Morning adeuring.
<lifeless> wgrant: also, please to try doubling our data set with the flatbugtask perf tests
<lifeless> wgrant: [estimate behaviour if we sit on it for a while after deploying]
<wgrant> fuuuuu
<wgrant> but OK
<wgrant> still need to set up proper indexes and such tomorrow.
<wgrant> But even with no indices it's largely better than we have now.
<wgrant> Even on dogfood.
<wgrant> Which can't fit the full table and indices in RAM
<wgrant> (well, it could if there was nothing else running, but there is)
<lifeless> yeah
<lifeless> If you can honestly state you're not worried about behaviour with our dataset doubled, I will take your word on it.
<lifeless> My point is that we need to make /some/ thought and assessment about it.
<wgrant> A lot of sort and tag/fti filter cases still degrade to an in-memory sort, so yeah, I'm a little worried.
<wgrant> But I've looked at it a bit.
<wgrant> fti often turns into a BitmapAnd, which drops order, so when the result is large it gets bad.
<wgrant> But it's still not more than 1.3s hot, even when it's sorting the whole set.
<wgrant> I'm going to look tomorrow morning at throwing tags into the first iteration.
<wgrant> But I don't have much hope because array stats aren't implemented.
<wgrant> And stats matter for Ubuntu's tags.
<mrevell> Mornin'
<wgrant> Although it may not be so bad to always do a full scan for tag searches if they're inline.
<nigelb> Mornin' mrevell
<czajkowski> aloha
<wgrant> Still, it no longer feels like bug search performance is entirely hopeless.
<danhg> aloha vera
<rick_h_> morning
<salgado> StevenK, hey, do you know when you might have some time to finish the review of https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883?  I'm happy to bug somebody else if you won't have time for it soon
<mabac> I've proposed  the Work items UI if anyone is up for reviewing that: https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790 :)
<deryck> adeuring, abentley, rick_h_ -- I'm stuck on the phone with my bank. Can we hold standup until this completes please?
<adeuring> deryck: np
<rick_h_> deryck: gotcha
<abentley> deryck: sure thing.
<deryck> thanks, guys.  sorry.
<czajkowski> lifeless: when you're about can you perhaps have a look over https://bugs.launchpad.net/launchpad/+bug/942523  please ?
<_mup_> Bug #942523: [Wish] Add a section Trophees in launchpad profile with the Trophee linked to Launchpad & Ubuntu Community <accomplishments> <launchpad> <ubuntu> <Launchpad itself:New> < https://launchpad.net/bugs/942523 >
<deryck> adeuring, abentley, rick_h_ -- sorry, almost done. let's hangout in 5.
<abentley> deryck: ack
<deryck> adeuring, abentley, rick_h_  -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<sinzui> jcsackett, do you have time to mumble
<czajkowski> jtv: are you about?
<jtv> czajkowski: I am
<czajkowski> jtv: I dont fully understand the bug here. https://bugs.launchpad.net/launchpad/+bug/942645
<_mup_> Bug #942645: Credit strings show up untranslated for kcm-gtk <Launchpad itself:New> < https://launchpad.net/bugs/942645 >
<jtv> czajkowski: you know what "translation credit strings" are?
<czajkowski> jtv: yes
<jtv> Launchpad pretends that it has translations for those, but actually those are generated on the fly.
<czajkowski> ah ok
<jtv> For some reason, in this case, the Launchpad UI lapses in its pretense that it has translations for these strings.
<jtv> If âlapsesâ is the right word.
<jtv> Argh.  Look at the computed percentages here!  That ought to be fixed: https://translations.launchpad.net/ubuntu/precise/+source/kcm-gtk/+pots/kcm-gtk/de/+details
<jtv> czajkowski: if memory serves, we had some mechanism for preventing this â in which case this needs investigation for which unfortunately I do not have time.  But it's also possible that this was never solved, in which case there's probably a duplicate bug.
<czajkowski> jtv: nods no worries shall investigate further
<czajkowski> thank you for the help
<jtv> Thanks.
<abentley> adeuring: I just noticed your lazr.jobrunner branch is lp:~adeuring/launchpad/lazr.jobrunner, not lp:~adeuring/lazr.jobrunner/foo.  Intentional?
<adeuring> abentley: usual sloppyness...
<abentley> adeuring: Oh, okay.
<jcsackett> sinzui: changes to enum branch pushed.
<sinzui> fab
<sinzui> jcsackett, r=me
<jcsackett> sinzui: thanks.
<abentley> jcsackett: do you have a sec?
<jcsackett> abentley: sure.
<jcsackett> what's up?
<abentley> You're a contributor to ec2 land, and I'm trying to harmonize it with lp-land.
<abentley> jcsackett: A difference is that ec2 land updates the commit message.  Do you find that useful?
<jcsackett> abentley: you mean the bit about setting the [r=] tags?
<jcsackett> or the commit message option you can pass it?
<abentley> jcsackett: I mean that it sets the [r=] tags etc on the merge proposal.
<jcsackett> abentley: i find that useful, yes, in that i would never remember to set those tags myself.
<jcsackett> abentley: i thought lp-land does the same thing, though ...
<abentley> jcsackett: You wouldn't have to set them yourself.
<abentley> jcsackett: It would already use them for submitting the proposal.
<abentley> jcsackett: It's just whether it rewrites them onto the merge proposal.
<jcsackett> abentley: oh! no, i do not find that useful. i actually find it kind of odd that the MP gets tagged with that stuff outside the actual landing.
<jcsackett> and if you have an error with an ec2 land and run it again, i have noticed you can end up with two sets of the [r=] tags. which is vexing.
<abentley> jcsackett: Really?  It was supposed to avoid duplication, but I guess it's buggy.
<abentley> jcsackett: Okay, I'll leave that functionality out.
<jcsackett> abentley: ok.
<abentley> jcsackett: thanks.
<jcsackett> abentley: you're welcome.
<lifeless> czajkowski: hi; crazy bug yes.
<czajkowski> lifeless: one for you then :)
<lifeless> czajkowski: I'd just mark it low triaged
<czajkowski> lifeless: ah so it may happen ?
<lifeless> czajkowski: if someone were to add support, with appropriate security, and maintain it, we'd (probably) have no problem doing it, though we wouldn't do it ourselves
<lifeless> probability 0 that someone will
<lifeless> mrevell as product manager can say 'no way never' if he likes :)
<czajkowski> heh
<lifeless> I get to set the bar we need to reach when we do something, but for /what/ we do, thats up to gentle suasion!
<czajkowski> grand job
<lifeless> czajkowski: you should never be concerned about triaging wrongly: the worst that can happen is someone can use the [formal] escalation process to request the thing be prioritised
<lifeless> czajkowski: if its wrong in a simpler manner (e.g. is a regression / isn't or whatever, one of your peers will probably just fix it up)
<czajkowski> lifeless: true, but with that I jst wondered would there be further discussion on the topic given the amount of blogging about the topic and other mails
<lifeless> oh there probably will be
 * lifeless sadfaces
<james_w> would someone help us out and do a release of oops_datedir_repo please?
<james_w> I don't want to have to depend on the branch in another oops module
<lifeless> sinzui: hey
<sinzui> hi lifeless
<lifeless> sinzui: I am sure you will know this
<lifeless> sinzui: why do I get messages 'foo has deactivated registry from bar' every day
<lifeless> james_w: sure
<sinzui> :)
<lifeless> sinzui: is it 'merge' or 'delete' ?
<sinzui> I was thinking about that this morning
<sinzui> It is delete
<james_w> lifeless, thanks
<lifeless> I need to write code; this week has had no code writing and I'm getting so very very deprived
<lifeless> sinzui: so for delete, I'm guessing that that person wasn't involved at all, but the notification happens automagically
<sinzui> There is some step I want to avoid in the merge process when merging with ~registry. I do not know what case is.
<sinzui> lifeless, they do, but something is out of order...
<lifeless> sinzui: separately, there was a bug we found where an admin approved a membership but the notification claimed someone else did
<lifeless> sinzui: I wonder if it could be the same thing
<sinzui> We remove super and subteams when deleting. by the point of the merge...there should be no memberships that could trigger that email
<lifeless> well, a component of the same confusion
<lifeless> sinzui: there is an owner
<lifeless> sinzui: and presumably a creator. I wonder if we have fallback code
<lifeless> crazy cracy fallback code
<sinzui> lifeless, the person we see in the email is the one that initiated the merge job that performs the delete
<lifeless> sinzui: ok, so 'foo deleted team bar' is what the mail should say (and it should perhaps go to the old members only)
<sinzui> lifeless, No
 * lifeless doth not understand
<sinzui> There is a step missing. Only the owner user doing the action should get an email...but that is not the email we are seeing
<sinzui> I think the act of merge/delete queue *2* jobs
<sinzui> 1st job queues the membership changes which is the email we are seeing.
<lifeless> sinzui: wouldn't it be confusing for the members to have the team disappear? That they were part of ?
<sinzui> 2nd job queue the merge. The merge happens first, so that the email go out and report ~registry was removed when the team was never a member
<lifeless> sinzui: ah, I see, you're saying that yes the members get a notification (perhaps 'removed from team' which is a little misleading); and separately we purge it
<sinzui> lifeless, I looked at the staging memberships of the people reported in the email. They are the team owners and the team is not related to ~registry. So the email really is about ~fnord was deactivated from ~pting
<lifeless> sinzui: unless registry gets put in fpting first
<sinzui> lifeless, this is the *very odd* part merge will not take place is any memberships exist. I assume team participation are indeed complete, so the emails should be impossible to send
<lifeless> james_w: done
<james_w> thanks lifeless
<lifeless> james_w: can you do me a favour and close off any fix-committed bugs?
<james_w> zero open bugs
<james_w> done!
<james_w> plus moving LP to pymongo's bson and getting rid of the compatibility code is probably a good idea, given that the tests in datedir and amqp aren't checking against both implementations
<james_w> one inconsistency I've found is that pymongo refuses to serialize things it doesn't know about
<james_w> so you can't just put arbitrary objects in the oops
<james_w> I don't know what the expected behaviour is there
<lifeless> that would be nice
<lifeless> we had a nasty bug where we tossed something bson didn't understand in
<lifeless> and it just skipped it
<lifeless> then oops-tools barfed
<james_w> ah
<james_w> skipped is clearly the wrong thing to do
<james_w> I assumed it did repr or something
<lifeless> sinzui: so, I think we should move all of _merge into the job
<lifeless> sinzui: there could be a lot of work there, may need multiple transactions to avoid concurrency issues, and there is no UI except for 'yay I did it' or 'boo cannot be done'
<lifeless> sinzui: (and, I think we should notify about deletes, but thats an orthogonal discussion)
<james_w> the options seem to be to repr every value, or try and bsonify every value individually and repr if it fails
<james_w> (we're trying to add task *args and **kwargs to the oops, and we don't know the types up front)
<lifeless> james_w: I thought you said pymongo errors ?
<james_w> it does
<james_w> and I don't want it to
<lifeless> james_w: I think that is great
<james_w> I want to put the data in without caring about the type in this case
<lifeless> safe_unicode them all ?
<lifeless> I suspect that (with the exception of collections) is probably going to DWIM for you
<james_w> where does safe_unicode live?
<lifeless> createhooks
<james_w> good idea, /me switches
<rick_h_> lifeless: ping
<lifeless> hi
<rick_h_> howdy, I'm trying to figure out your comment in https://code.launchpad.net/~rharding/launchpad/watch_jsbuild/+merge/94379
<rick_h_> about both the make target and buidlout template?
<rick_h_> can you elaborate for the buildout impaired what I've done wrong/correct way to fix?
<deryck> lifeless, hi.  I'll get in line behind rick_h_, but would love to get your review of a preload attrs for buglistings branch.
<rick_h_> hah, my faster typing pays off
<deryck> I let you win. ;)
<rick_h_> I feel a monty python skit involving a black knight coming on
<lifeless> deryck: url ?
<lifeless> (OTP :P)
<deryck> lifeless, https://code.launchpad.net/~deryck/launchpad/preload-tags-for-buglistings-901122/+merge/95023 (and thanks!)
<lifeless> rick_h_: I made that comment on the initial diff
<rick_h_> lifeless: ok, cool
<lifeless> rick_h_: it looks sane now, but perhaps s/foo.py.in/foo.in/
<rick_h_> lifeless: ok
<lifeless> (e.g. I don't think the language is relevant for an executable interface)
<rick_h_> right, ok nop
<rick_h_> np that is
<lifeless> deryck: reviewed
<lifeless> with 2 comments (second coming in a sec)
<deryck> lifeless, ok, thanks, man.
<jcsackett> sinzui: i will be a few minutes late to standup.
<sinzui> okay
<thumper> hi people
<thumper> is launchpad still using windmill?
<lifeless> nup
<lifeless> died in a fire
<james_w> someone should put the smackdown on windmill on -tech
<deryck> lifeless, so I've got the last branch up for review, but you don't have to review it now.  I'll add the scaling test in that final branch.
<deryck> lifeless, but if you're curious: https://code.launchpad.net/~deryck/launchpad/adapt-badges-listing-item-901122/+merge/95063
<deryck> thumper, we still have windmill tests lying around, with the hope they'll get ported to our new yuixhr framework.
<deryck> thumper, but they haven't been running for some time.
<deryck> if I ever need LOC credit for something I'm working on, I plan to delete them then. ;)
<thumper> do what is the new magic?
<lifeless> html5-browser
<deryck> yeah, just plain ol' yui run headless with html5-browser.
<deryck> I'm out guys.  catch you all later.
<sinzui> wgrant, https://code.launchpad.net/~sinzui/launchpad/mailman-archive-0
#launchpad-dev 2012-02-29
<StevenK> cody-somerville: Are you able to QA r14876, which is on qas now?
<StevenK> wgrant: The notfound template already says: "This page does not exist, or you may not have permission to see it."
<wgrant> StevenK: IN about 10 tonnes of other text.
<StevenK> It's the first text after the heading
<cody-somerville> StevenK, How do I QA it?
<StevenK> cody-somerville: You added an attribute to the webservice. Can you get it and set it in the way you expect?
<cody-somerville> StevenK, Cool. Okay. I can do that. Do I tag the bug or something if successful?
<StevenK> cody-somerville: Yes. There is currently a 'qa-needstesting' tag on the bug. If it works fine, remove that tag and add 'qa-ok'
 * StevenK smacks wallyworld 
<wallyworld> ?
<StevenK> wallyworld: 116+from lp.app.interfaces.services import IService right at the top of lib/lp/registry/browser/tests/test_product.py ?
<wallyworld> auto import fail
<wallyworld> will fix in current branch
<StevenK> wallyworld: I spent hours fixing imports yesterday and the next feature branch gets it wrong. :-(
<wallyworld> it's only one of many i added, easily fixed :-)
<StevenK> wgrant: Tempted to do distribution via SQL. There are less than 50.
<wgrant> StevenK: Possibly.
<wgrant> lifeless: staging should have my first patch in about 30 minutes.
<wgrant> http://people.canonical.com/~wgrant/launchpad/flatbugtask.sql is the second one which I described yesterday.
<wgrant> It's missing triggers on the access policy bits, and there's no handling of toggling of Product.active, but it should otherwise be reasonably consistent.
<lifeless> wgrant: drop that in a file on devpad
<wgrant> lifeless: carob:~wgrant/flatbugtask.sql
<lifeless> is it dependent on the first patch ?
<wgrant> It'll apply and partly work without it, but none of the access stuff will be there.
<wgrant> Once the first patch is applied we need to build its data (SELECT bug_mirror_legacy_access(id) FROM bug), then build the second lot of data (SELECT bugtask_flatten(id) FROM bugtask).
<lifeless> ok, ping me
<wgrant> Yup, just prewarning you.
<wgrant> lifeless: I also need a way to do mass async updates when Product.active changes. Jobrunner job? Dirty flag with garbo job? The latter is a bit evil but somewhat lighter than the current jobrunner situation.
<lifeless> jobrunner
<wgrant> :(]
<lifeless> should be light enough
<wgrant> 2KLOC later...
<lifeless> I'm sure you'll make it lighter as you go
<wgrant> Heh
<wgrant> I guess we need a pattern for this sort of thing anyway.
<wgrant> StevenK: Bah, I said I had created a card already :)
<wgrant> 2012-02-29 01:42:39 INFO    2209-11-3 applied just now in 0.4 seconds
<wgrant> lifeless: https://pastebin.canonical.com/61200/ are the incantations.
<lifeless> wgrant: do you want them run now ?
<wgrant> lifeless: Please. \timing would be handy too.
<wgrant> lifeless: Will be interested to see how long the two mass function calls take in a single batch.
<lifeless> one transaction?
<wgrant> Might as well run it in three, just in case one fails.
<lifeless> heh, started it a begin
<lifeless> will do over if it fails
<wgrant> lifeless: You applied flatbugtask.sql, right? :)
<wgrant> It defines the last function.
<lifeless> no, not yet
<lifeless> wgrant:
<lifeless> INSERT INTO accesspolicy (product, type) SELECT id, 3 FROM product;
<lifeless> INSERT 0 34729
<lifeless> Time: 3833.312 ms
<lifeless> INSERT INTO accesspolicy (product, type) SELECT id, 4 FROM product;
<lifeless> INSERT 0 34729
<lifeless> Time: 4182.285 ms
<lifeless> INSERT INTO accesspolicy (distribution, type) SELECT id, 3 FROM distribution;
<lifeless> INSERT 0 36
<lifeless> Time: 9.049 ms
<lifeless> INSERT INTO accesspolicy (distribution, type) SELECT id, 4 FROM distribution;
<lifeless> INSERT 0 36
<lifeless> Time: 9.061 ms
<wgrant> lifeless: Looks reasonable.
<lifeless> SELECT bug_mirror_legacy_access(id) FROM bug;
<lifeless> Time: 228686.012 ms
<lifeless> SELECT bugtask_flatten(id) FROM bugtask;
<wgrant> Not too bad, considering a lot of it was probably cold.
<lifeless> (1097473 rows)
<lifeless> Time: 994126.901 ms
<lifeless> bugtask_flatten(id) returns a pile of null :P
<wgrant> Hmmm, slower than it should have been.
<wgrant> But thanks.
<wgrant> So.
<wgrant> You should now have a marvellous bugtaskflat with 20 or so indices
<lifeless> select count(*) from bugtaskflat;
<lifeless>   count
<lifeless> ---------
<lifeless>  1097473
<wgrant> Excellent.
<wgrant> Try https://pastebin.canonical.com/61202/
<lifeless> 102984, 1238ms
<wgrant> :(
<lifeless> explain analyze 625ms
<wgrant> So 500ms planning?
<wgrant> Or 1238 cold?
<lifeless> 9ms
<wgrant> coldish, that is
<lifeless> 1238 coldish
<wgrant> Ah
<wgrant> So it's now 600ms?
<wgrant> ish?
<lifeless> 420, 500, etc
<wgrant> That's more like it.
<wgrant> Care to throw in a bit of FTI?
<wgrant> Like BugTaskFlat.fti @@ ftq('kernel hang') or so?
<wgrant> (yes, the privacy constraint is hideously structured, but it works it out -- I need to rework the Python to spell it better)
<wgrant> The SQL I provided uses gist for fti, but it might be interesting to compare it to gin on sourcherry.
<lifeless> pastebin what you want :)
<wgrant> lifeless: https://pastebin.canonical.com/61204/
<lifeless>  Aggregate  (cost=82175.45..82175.46 rows=1 width=0) (actual time=637.165..637.166 rows=1 loops=1)
<wgrant> Well that sucks.
<wgrant> Can I have the whole thing?
<lifeless>  Limit  (cost=82182.00..82182.19 rows=75 width=8) (actual time=527.338..527.366 rows=75 loops=1)
<lifeless> https://pastebin.canonical.com/61205/
<wgrant> Thanks.
<wgrant> A
<wgrant> Ah
 * wgrant blames gist.
<wgrant> A bit of planner stupidity too, but mostly gist.
<wgrant> lifeless: DROP INDEX bugtaskflat__fti__idx; CREATE INDEX bugtaskflat__fti__idx ON bugtaskflat USING gin (fti); and retry?
<wgrant> Will probably take a while
<wgrant> GIN generation is slooooow.
<StevenK> wgrant, lifeless: O hai, can we have a mumble about ArchiveAudit?
<lifeless> sure
<lifeless> skype?
<lifeless> or fanout ?
<StevenK> You can't do mumble?
<lifeless> it hates me
<wgrant> Hangouts still don't work for me, and are even less tasteful than Skype.
<lifeless> it needs http://www.rowetel.com/blog/?page_id=454
<StevenK> It worked last time :-P
<wgrant> So I suggest we Skype :)
<lifeless> StevenK: it was traumatic
<lifeless> StevenK: and I've got Cynthia beside me at this time of day
<StevenK> I can make it more trauatic and turn on the webcam
<StevenK> *traumatic
<lifeless> I know what you look like already
<lifeless> I can hear you
<lifeless> lies
<lifeless> precise
<lifeless> diaf
<lifeless> sound panel sees my input
<lifeless> what command ?
<wgrant> pavucontrol
<StevenK> lifeless: https://code.launchpad.net/~stevenk/launchpad/db-add-archiveaudit/+merge/94726 is the WIP MP
<wgrant> lifeless: Any chance you could recreate the FTI index as GIN as I described above?
 * StevenK peers at his garbo test
<lifeless> CREATE INDEX
<lifeless> Time: 159917.895 ms
<lifeless>  Limit  (cost=82270.96..82271.14 rows=75 width=8) (actual time=1031.584..1031.633 rows=75 loops=1)
<lifeless>  Limit  (cost=82270.96..82271.14 rows=75 width=8) (actual time=183.615..183.666 rows=75 loops=1)
<wgrant> lifeless: blink
<wgrant> So it's 3x faster?
<lifeless> looks like it
<wgrant> Might try with GIN straight away, then
<lifeless> what about table scans
<wgrant> Given that we're going to be restrictedly testing this for a while anyway.
<lifeless> ok
<wgrant> They're unlikely to happen given the other indices.
<wgrant> But yes, it is a concern.
<stub> We can drop in GIN already if we don't care about occasional searches breaking, but in the past we have :-)
<wgrant> stub: The new table has strategic indices.
<StevenK> Haha
<wgrant> Rather than random ones that are mostly useless, like bug and bugtask.
<stub> The 'all stopwords' query is the usual one that breaks things.
<lifeless> we could take a calculated risk
<lifeless> easy enough to change index style
<lifeless> and hell, 1 second is still >> 8 seconds
<lifeless> StevenK: I had a quick look for a audit service, couldn't see one
<lifeless> wgrant: let webops know if/when you want that stuff dropped from staging
<wgrant> lifeless: Easy enough to change index style, and easy enough to turn the feature flag off :)
<stub> lifeless, wgrant: If we are rebuilding the search, I think we should a) use GIN indexes wherever possible and b) catch the exception we see when it attempts a full scan and return a 'too many results' type page.
<lifeless> wfm
<wgrant> Probably, yeah.
<stub> Might as well stick a LIMIT in the query too, generating the 'too many results' thing even if the index hit did work. Nobody wants page 1 of 600.
<wgrant> Well
<wgrant> Hopefully we'll be using StormRangeFactory soon
<wgrant> Which will remove the COUNT(*) entirely.
<stub> bug_fti is off the 'most bloating indexes' list, yay!
<wgrant> stub: yup :)
<wgrant> buildbot failure is mine, I'm on it.
<adeuring> good morning
<czajkowski> aloha
<[reed]> wgrant: throw in IE's X-XSS-Protection as well
<[reed]> Chrome supports it, too
<[reed]> wgrant: depending on how that gets presented, you may wish to add "; includeSubDomains" to the HSTS header, too
<wgrant> [reed]: Can't use includeSubDomains, due to some questionable historical decisions.
<wgrant> Will look into X-XSS-Protection, though -- didn't realise Chrome did it too. Thanks.
<nigelb> oh wow. totally did not expect [reed] in here ;)
<[reed]> I'm everywhere.
<nigelb> Yes, TIL.
<[reed]> also, I've been complaining about some of these things for a while, at least in LDS
<nigelb> Wait, you come to UDSes and LDSes?
<[reed]> LDS == Landscape Dedicated Server
<nigelb> Ahhhhhh.
<[reed]> I have come to two UDSes, though
<wgrant> X-XSS-Protection looks like a bit of a misfeature.
<[reed]> I'll probably stop by the Oakland one
<wgrant> Potential for security issues, even.
<[reed]> wgrant: I think most of those issues have been worked out... it's been around for a while now
<[reed]> "Registration for the next UDS opens on 24th February 2012."
<[reed]> it's a bit past that
<[reed]> lol
<[reed]> wgrant: if you want to be super-awesome, send an appropriate CSP header ;)
<wgrant> [reed]: Isn't that thoroughly unstandard and still in flux?
<wgrant> Ahhh
<wgrant> I see X-XSS-Protection has evolved a bit since I dismissed it as completely insane a couple of years ago
<wgrant> Notably with the addition of mode=block
<[reed]> indeed
<wgrant> Anyway, yeah, was looking at doing CSP, but it's going to be a bit less trivial to ensure everything still works, and it seems like there's work on making it more sensible and standard.
<[reed]> sure
<wgrant> But since you're suggesting it, I might have to look again :)
<[reed]> hah
<[reed]> I know a bunch of Mozilla sites implement it
<wgrant> Because you added it? :)
<[reed]> but I also know it took them quite a while to do the right thing
<nigelb> haha
<[reed]> hah, not for CSP, but I did add other headers ;)
<wgrant> Heh
<[reed]> pretty much every security "feature" Bugzilla has gained in the past several years was because I wrote a patch for it :p
<wgrant> I've wanted to do this to LP for 18 months, but there were various bits of stupidity that needed to be fixed first.
<wgrant> Like certain URLs which were forced to HTTP for no good reason.
<[reed]> wanna fix Landscape as well? :P
<wgrant> And crufty bits of Zope that needed upgrading.
<wgrant> sigh
<wgrant> Heh
<[reed]> don't get me started on how terrible Landscape is from a security perspective ;)
<wgrant> Yeeeeeah
<wgrant> It's the one major Canonical webapp that I haven't tried to poke holes in.
<wgrant> I'm not entirely sure I want to try :)
<[reed]> I'd appreciate it if you would! ;)
<[reed]> I've poked a bunch of holes in it already
<[reed]> but I know there are plenty more
<[reed]> it's never had a formal security review, as far as I can tell
<wgrant> juju is the next target of my wrath.
<czajkowski> wgrant: you still up ?
<wgrant> czajkowski: Vaguely.
<czajkowski> can you help me in launchpad with a question:/ re  merge proposals
<czajkowski> please
<wgrant> czajkowski: mgz is one of us :)
<wgrant> Well, bzr.
<wgrant> But close enough.
<czajkowski> wgrant: aye thats what I thought too
<rick_h_> morning
<czajkowski> rick_h_: morning
<mabac> rick_h_, good morning. :) since there's no ocr can I pester you with another review? or perhaps could you point me to someone I could bug? ;) https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790
<rick_h_> mabac: sure thing, will peek at it
<mabac> rick_h_, cool, thank you very much!
<salgado>   File "/home/salgado/launchpad/lp-branches/workitems-migration-script/lib/lp/services/database/bulk.py", line 244, in create
<salgado>     IStore(cls).execute(Insert(db_cols, expr=db_values))
<salgado> TypeError: __init__() got an unexpected keyword argument 'expr'
<salgado> has anybody seen this before?
<salgado> hmm, looks like I'm using an old version of storm (storm-0.19.0.99_lpwithnodatetime_r404-py2.6-linux-i686.egg)
<salgado> that's the version I should be using, in fact, but my branch is using storm-0.19.0.99_lpwithnodatetime_r402-py2.6-linux-i686.egg
<salgado> I thought rocketfuel-get would update the generated scripts (bin/test, etc) to use the latest source but I guess that wouldn't make sense -- I should probably do that manually when I merge devel
<salgado> is make clean; make the correct way to deal with that?
<rick_h_> salgado: yea, think you'd need rocketfuel get, but then to merge with devel to lget the latest versions.cfg or what not. Not sure if storm is a packaged thing or a source dep
<salgado> rick_h_, yeah it's in lp-sourcedeps, not deb-packaged.  I guess there's nothing faster than to "make clean" and rebuild from scratch?
<rick_h_> salgado: might get away with a bin/buildout
<rick_h_> but not tried it so not sure
<salgado> oh, right, I'll try that next time
<rick_h_> reading and working on makefiles a lot lately, but still some voodoo on what fits where to me :)
<rick_h_> mabac: ok, notes posted, I ping'd jcsackett for a follow up
<mabac> rick_h_, great thanks! will have a look asap!
<rick_h_> mabac: most of my feedback just surrounds how I like to test things that parse like this: https://pastebin.canonical.com/61235/
<rick_h_> mabac: so that as bugs/issues are found it's very trivial to add a new test case and see at once what works/doesn't kind of thing
<rick_h_> if that makes sense
<rick_h_> but it's a preference thing on my end
<mabac> rick_h_, thanks. I don't have access to the canonical pastebin though
<rick_h_> ah, sorry sec
<rick_h_> http://paste.mitechie.com/show/552/ mabac
<rick_h_> but anyway, for that to work our parsing/raising exceptions need to be a bit split/etc
<mabac> I didn't really know if I should always raise LaunchpadValidationErrors or perhaps let parse errors pass through as well
<rick_h_> yea, not sure on the standard since I've not done much with it
<mabac> I'll see if I can use a regexp for the assignee as well. and move the work items header re to a module level constant
<rick_h_> mabac: awesome
<rick_h_> rvba: ping, just wanted to toss out a heads up that on the js builder stuff. Not sure if you guys can use that over in maas land.
<rvba> rick_h_: pong
<rick_h_> rvba: I've got to update the wiki with some notes, but give it a shot with what you guys are doing and let me nkow how I can make it more 'general' for other projecs
<rick_h_> I want to find out what's too LP specific so I can hit up the other JS people out there and figure I can bug your team first :)
<deryck> adeuring, https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<salgado> how do I tell the test runner to drop into pdb on the first failure?
<benji> salgado: -D
<rvba> rick_h_: I'm in the middle of something right now, can you please paste the link to the page you want me to have a look at and I'll get back to you shortly.
<salgado> benji, and that's an argument to bin/test I guess?
<benji> salgado: yep
<rick_h_> rvba: https://lists.launchpad.net/launchpad-dev/msg09077.html no hurry, just more fyi
<salgado> benji, great, thanks!
<salgado> benji, it doesn't seem to work if I use it with testr (raises BdbQuit immediately), but I guess that's known/expected?
<adeuring1> deryck, abentley, rick_h_sorry, again a crash...
<deryck> adeuring, np, we'll wait on you.
<rvba> rick_h_: I see.  The problem with maas is that it's a distributed package.  To add a dependency: either it's packaged in precise or we have to include the source code in our tree.
<rick_h_> rvba: ok, gotcha
<benji> salgado: it's not known by me, and expectations lie at the root of suffering.
<rvba> rick_h_: But I keep track on what you guys do on lp :).
<rvba> rick_h_: We have the same dilemma about using/not using convoy.
<adeuring> deryck: can you send me the link to the hangout again?
<rick_h_> adeuring: https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<adeuring1> deryck abentley rick_h_ seems that i should this machine for the hangout; haven't installed the plugin yet...
<rick_h_> rvba: sorry, standup. Ok cool, I wasn't sure how you guys were doing the JS stuff but know you asked about the combo loader bits at the epic
<rick_h_> rvba: so yea, the convoy stuff in the jsautobuild probably doesn't help you guys at all then if you're not going combo loader/convoy
<rvba> rick_h_: I really would like to use the combo loader/convoy but I'm not sure it's going to happen for 12.04 for the reasons mentioned above.
<rick_h_> rvba: right
<adeuring1> abentley: let's use mumble...
<abentley> adeuring1: sure
<rvba> rick_h_: I'll benchmark what it costs us (in terms of page load time) to not use it.
<rvba> rick_h_: if this is really significant, I'll try to get someone to package it or we will include the code in our tree.
<rick_h_> yea, you guys are on disk/local right so I doubt it's horrible.
<rvba> We will see.
<rick_h_> but the pyinotify stuff is cool if you guys do hav a make jsbuild type step that you don't want to have to run on ever save
<rick_h_> but ok cool. thanks for thinking/taking a peek if it fits into your stuff at all.
<rvba> Looks very useful indeed.
<mpt> Hi, I'd like to test a bug that occurs when reporting bugs, but staging.launchpad.net times out when doing something as simple as showing a package page
<mpt> Is it likely that staging will work again soon?
<rvba> rick_h_: I'll keep you in the loop when I'll do the benchmark.
<czajkowski> mpt: I'll ask and find out
<mpt> czajkowski, actually, I found a package that has hardly any bug reports (which I guess was causing the timeouts), so it's not urgent for me now. But staging seems rather underpowered for meaningful testing.
<czajkowski> mpt: no harm in asking and finding out for sure
<mpt> (The bug I was testing I have now reported as bug 943321)
<_mup_> Bug #943321: Private bug report wrongly says "Subscribed by <reporter>" for automatic subscribers <Launchpad itself:New> < https://launchpad.net/bugs/943321 >
<cr3> hi folks, launchpad uses a special syntax in brackets to identify the person approving in commit messages for example. where might this be documented?
<rick_h_> cr3: somewhat here: https://dev.launchpad.net/PQMCommitMessages
<rick_h_> cr3: basically it's [r={lpuser}]
<cr3> rick_h_: thanks!
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugtasks: 4*10
<abentley> cr3: There are tools that will generate this for you: "bzr lp-land" from the bzr-pqm plugin, and ec2 land in the LP tree.
<cr3> abentley: why use things like [bug=90791] when you can use bzr commit --fixes lp:90791 instead?
<abentley> cr3: I don't believe that PQM respects that revision property.  It goes strictly on the basis of the message.
<cr3> abentley: ok, so in practice, nobody really types any of those bracketed messages, right?
<abentley> cr3: right.
<cr3> abentley: that makes sense then, thanks!
<abentley> cr3: And if you use commit --fixes, then ec2 land or lp-land will include [bug=] for you.
<cr3> abentley: sweet, so the developers are encouraged to use bzr properly and get lots of benefits for free
<danilos> mrevell, flacoste: so, we're having some problems with answers.launchpad.net: 1. there are packages linked to some of our projects, so we get weird pages like https://answers.launchpad.net/linaro â I don't seem to have permissions to remove those Packaging records anymore; 2. there are requests to either make "external" setting for answers app do something useful, or get rid of the tab â I know that Launchpad is not designed as a p
<danilos> roject hosting service directly, but we want to be able to clearly point people at support resources
<danilos> hi, btw :)
<danilos> mrevell, flacoste: fwiw, for 1, I'm asking czajkowski in #launchpad if she can help remove them, but I am mentioning that as a more general problem for things we don't have control of
<mrevell> thanks danilos, I'm otp, so slow to respond
<danilos> mrevell, sure thing, just raising this as it's becoming an issue for us and I'd like your general opinion on it when you find the time
 * deryck goes offline for lunch, back soon
<stokachu> Hi im messing around with Oauth and launchpad and was curious if we could add ability to return some user data when grabbing the access_token?
<stokachu> Right now there is no way to dynamically pull an id/display_name from an initial access token request
<stokachu> so people will need to know their display name ahead of time
<stokachu> hmm i guess i could do a /people/+me
<lifeless_> stokachu: so once you have an OAuth token you can request details from /~ exactly.
<stokachu> lifeless_: ok so when calling api.staging.launchpad.net/people/+me it'll response with json from /~username automatically?
<stokachu> i think thats how im reading it in docs
<lifeless> stokachu: yes, likewise /people/+me
<stokachu> ok sweet
<stokachu> let me hack it some more -- ive almost got a drupal plugin going
<stokachu> ahhh i think im adding a json format by accident 'data' => 'Object: <lp.registry.model.person.PersonSet object at 0xa2310d0>, name: u\'+me.json\'',
<stokachu> it already does json without me telling it
<sinzui> stokachu, correct
<sinzui> stokachu, out oauth/launchpad lib code is  think of most things as simple string and numbers. dates have to be written as strings I think
<stokachu> ah gotcha
<stokachu> if i authorize against launchpad.net and attempt to call api.staging.launchpad.net that will fail b/c of auth correct?
<sinzui> correct
<sinzui> Lplib scripts will prompt you to authorise for the new host
<stokachu> ok
<stokachu> crap still getting unauthorized
<stokachu> so here is my header information http://pastebin.com/uB4axged
<sinzui> stokachu, pastebin the script or the part that does the login/auth
<stokachu> ok
<stokachu> http://pastebin.com/8RdQruEi - so this is my library
<stokachu> line 137-149 is where im having the issue
<stokachu> im wondering if im not providing enough header information
<sinzui> Ah I see you are implementing a PHP lib that duplicates part of LaunchapdLib. You will need to manage the JSON serialisation yourself
<sinzui> stokachu, I think you should look at http://bazaar.launchpad.net/~lazr-developers/launchpadlib/trunk/files/head:/src/launchpadlib/ which shows how credentials and serialisation is done
<stokachu> ok ill check that
<stokachu> i dont think im getting to the point of parsing the response yet
<stokachu> when i do 'request' => 'GET /1.0/people/+me HTTP/1.0 it gives me unauthorized
<stokachu> even though +request-token and +access-token succeeds
<sinzui> stokachu, I think the request is missing the Lp cookie that contains who you are. +me looks for the user in the interaction of the request, and that will come from a cookie
<stokachu> ok, i may be using the wrong scheme here.. i see launchpad uses http authorization scheme
<stokachu> and i may be pushing it through just post
<lifeless> our oauth module is homebrewed; its possible we only implemented the Authenticate header approach for OAuth
<stokachu> ok i see, lifeless, i think youre right i dont see anything related to the authorization in the headers
<stokachu> hmmmmm
<sinzui> lifeless, stokachu. I think partial implementation is the case here. I recall something came up a a few years ago where where made a change to stay compatible, but did not elect to do a full implementation
<sinzui> abentley, rick_h_, jcsackett, wallyworld (Ian) has three important in the review queue. I am mentally challenged at the moment. I can review the disclosure rules that are in the branches, but most of the implementation is better reviewed by a rocket scientists
<stokachu> :P
<abentley> sinzui: I'll have a look.
 * sinzui has taken a second dose of cough medicine and now has that disconcerting feeling of contentment.
<stokachu> haha
 * sinzui fears that when he yawns, a hole will open up behind him and will fall out of the universe
<abentley> sinzui: rofl.
<stokachu> what are the request bugs? i'd like to see all 3 consumer requests methods
<stokachu> or at least the one where you send as HTTP POST
<sinzui> abentley, I am not kidding. me + cough syrup = mescaline induced hallucinations. I never want that to happen again
<abentley> sinzui: I'm sorry.  "disconcerting feeling of contentment" was too much for me.
 * sinzui does want to re-enact the Odyssey again
<sinzui> abentley, it is okay to laugh. my wife does
<sinzui> I am serious that I am mentally challenged
<stokachu> ok im going write a helper routine to implode my parameters into an Authorization request
<stokachu> see if i can get authed then
<StevenK> james_w: O hai
<james_w> hi StevenK
<StevenK> james_w: lifeless said you guys have a standard way of bringing Django apps, is that right?
<james_w> StevenK, ish
<james_w> we have what we currently think to be the best way
<james_w> but none of our apps probably comply fully with that :-)
<StevenK> james_w: Are you able to point me at some details?
<james_w> StevenK, lp:pkgme-service is the newest one which you may want to use as an example
<james_w> https://wiki.canonical.com/JamesWestby/DjangoDeploymentBestPractices has some notes
<StevenK> james_w: Thanks, I'll look over both
#launchpad-dev 2012-03-01
<wgrant> cody-somerville: Hi, can you QA your authorized_size thing?
<StevenK> And jelmer's thing
<wgrant> jelmer's QAed half of his, but bug #296153 is still pending
<_mup_> Bug #296153: does not mirror bzr branches over ftp <lp-code> <qa-needstesting> <Launchpad itself:Fix Committed by jelmer> < https://launchpad.net/bugs/296153 >
<StevenK> wgrant: Your testfix switched to rSP() in the factory?
<wgrant> No, rSP doesn't work there.
<wgrant> I logged in as an admin instead.
<wallyworld> abentley: hi, with the mustache/handlebars template in a file, i notice there's no reference to it in the buglisting yui tests
<wallyworld> so it doesn't appear the rendering aspects are tested in that case, whereas in the pillar sharing stuff, i have yui tests for the rendering and therefore need the load the template in the tests
<wallyworld> that was one of the reasons for having the template in the javascript
<wallyworld> i guess i could pull in the template in the test html
<jelmer> StevenK, wgrant: bug 296153 is qa-ok
<_mup_> Bug #296153: does not mirror bzr branches over ftp <lp-code> <qa-ok> <Launchpad itself:Fix Committed by jelmer> < https://launchpad.net/bugs/296153 >
<rick_h_> wallyworld: yea, that's a big issue we've got to figure out for JS testing as we move more client side rendering
<wallyworld> rick_h_: i'll seeing how it flies pulling it out in a file, as per the code review request. i was happy enough to leave it in the js for now but there's been a couple of objections
<wallyworld> rick_h_: in this case, there's NO server side rendering, it's all client
<rick_h_> wallyworld: right, but you want the tests and the live site using the same template if possible so that they match and test correctly
<rick_h_> but it'sa bit tough/pia
<rick_h_> pita that is
<wallyworld> rick_h_: of course i want to use the same stuff :-) that's why it was in the js
<wallyworld> i think i can put in a file and load the file in the test html
<wallyworld> trying that now to see if it has legs
<wgrant> jelmer: Thanks.
<rick_h_> wallyworld: yea, that's what I've done in my app is create a jstemplate file that I include in both places
<wallyworld> rick_h_: if i have a script element with a src attribute, what's the magic to get yui to be able to get at the contents of the referenced file?
<rick_h_> Y.one('#id').getContent() I think
<rick_h_> oh, it's got a src attribute? hmmm, not sure if that works
<wallyworld> yeah, getContent() fails
<wallyworld> well, i used src since i wanted to reference the single copy of the mustache template file
<wallyworld> that same copy that the view passes to the js in prod
<wallyworld> maybe ScriptNodeDataSource
<wgrant> wallyworld: I would hope that you couldn't get at the content.
<wgrant> That would be a security issue unless very carefully restricted.
<wallyworld> wgrant: it's in the test html
<wallyworld> to load the mustache template data from a file to pass to the widget
<wallyworld> on prod, the pythov view does this
<wallyworld> but i want to use the same template for testing
<wgrant> Ah, interesting.
<wallyworld> i currently have the template in the js but people don't like this
<wallyworld> so i'm trying to find another way
<wallyworld> buglistings does it this way but there' no yui rendering tests using the template
<wallyworld> so they didn't need to solve this
<wallyworld> so it appears html5 has a File api which may be useful
<wgrant> rofl
<wgrant> glwt
<wallyworld> well spec draft is from oct 2011 so it's "ancient" :-)
<wgrant> Hmm
<wgrant> I can't talk to bits of the DC
<wgrant> Ah, there we go.
<lifeless> wallyworld: hi
<wallyworld> lifeless: hello
<lifeless> wallyworld: you should deliver the template via the API/view involved - combo loader eventually
<lifeless> wallyworld: is there a reason that that won't work ?
<wallyworld> lifeless: that's all well and good but i need the same content for yui tests
<lifeless> I don't see the problem
<wallyworld> that's the issue here
<wallyworld> tests have a html file and a js file
<wgrant> There's no API or view in a YUI test.
<lifeless> the yui tests are loaded and run by the lp test driver which is part of our regular python stack
<lifeless> how will the yui tests work in a non-launchpad.js world ?
<wallyworld> yui tests don't have our python stack, just html and javascript
<rick_h_> the yui tests are just calling .html files iwth JS that runs on them
<rick_h_> our test stack is just a vessel to load .html files and parse response
<wallyworld> eg you can load the test html file in a browser
<lifeless> rick_h_: and the html5-browser wrapped around that
<wallyworld> and the tests will run
<wallyworld> with no lp running at all
<wallyworld> just pure js
<rick_h_> we'd need to find some way to write templates to a .html file and include it into the test.html file (iframe or something?) and pull that html file into the zope template as required for the JS on page to use
<lifeless> no yui ?
<wallyworld> yui, yes
<lifeless> rick_h_: guh what
<wallyworld> but reasd off disk locally
<rick_h_> I've not had a chance to think it through 100% yet. I just know I hit it when updating the tests and it's on my list.
<wallyworld> lifeless: so you see the problem now?
<lifeless> rick_h_: so lets skip mustache
<lifeless> wallyworld: I see the corner we've painted ourselves into
<rick_h_> so for instance, in my outside application, tests are run via the application so that I can import the same template file in the tests that the application uses
<lifeless> wallyworld: we've violated abstractions and are now suffering
<wallyworld> lifeless: well, it's how yui tests have been written since day 1
<wgrant> lifeless: You might be thinking of YUIXHR
<wgrant> Which uses an appserver.
<lifeless> I might be
<lifeless> so, lets break this into bits
<wallyworld> lifeless: there's no server side rendering here
<lifeless> unit tests should not be testing the template outputs; thats orthogonal
<wallyworld> so involing the view is bogus
<lifeless> testing that a template is rendered and executed is sensible, but the code under test is supplied the template by the server
<wallyworld> the view unit tests do need to test the rendering of data
<wallyworld> sure, but we need a way to test without the server where the yui unit tests can get at the template
<wallyworld> single sourced
<wallyworld> and that's the issue
<lifeless> so, I think I'm going to get my nose out of this discussion, because I clearly have a different model for whats going on
<wgrant> lifeless: What about the unit tests for the template rendering?
<wgrant> Surely they need to be testing the template output.
<lifeless> I will throw a few key constraints in though to save us later
<lifeless> firstly, the templates -must not- be inline in our js or tal files
<wallyworld> impossible to write yui unit tests without code copying then
<lifeless> we will be compiling them and shipping the compiled result as js to use, accessed probably via the combo loader, or in a pinch via the json cache
<lifeless> if the template is inline in our js or tal, that won't work, and we'd be screwed. This is just around the corner when the combo loader comes live && YUI 3.5
<wallyworld> nope, the combo loader can handle that ust fine
<lifeless> orly
<wallyworld> the template is built up inside our javascript
<wallyworld> and accessed via a method on the view widget
<lifeless> it will extract the template from another file, compile it with nodejs, write that to a temp file and then we'll serve that?
<lifeless> I feel very confused.
<wallyworld> no, and we use this in several places, we use Node.create(template) where template is a concatenated string of html
<lifeless> I'm talking about handlebars templates.
<wallyworld> on in the case of mustache, to_html(template)
<wgrant> wallyworld: The template is rendered
<wallyworld> same thing
<wgrant> The rendered template is given to Node.create
<wallyworld> yes, i was using an example
<lifeless> wallyworld: its not the same thing, the template gets *compiled*
<lifeless> wallyworld: it comes down as javascript, in its own source file.
<wallyworld> i was trying to issultrate that the template does not have to come from server side
<wallyworld> it can be constucted the the js
<wallyworld> we use that right now in the code
<wgrant> The template has to.
<wallyworld> no
<wgrant> The JS can't calculate it client-side.
<wallyworld> yes, it does
<wgrant> No.
<wallyworld> we use it now
<lifeless> wgrant: handlebars compiler can run client-side, but it is overhead
<wgrant> It can't, because we have to specify it somehjow.
<wgrant> I mean a (potentially compiled) template.
<wgrant> Has to be transmitted from the server
<wgrant> at some point.
<wallyworld> no, let me find the example
<lifeless> we wouldn't want a compiled template from a foreign site, but from LP itself, we want compiled templates I think.
<lifeless> this 'no' business doesn't really elucidate the issues
<wgrant> wallyworld: You are *rendering* the template client-side.
<wallyworld> we concatentate strings to form a mustache template and then call to_html
<wgrant> Oh god
<wgrant>  /wrists
<lifeless> wallyworld: thats a creative use of mustache
<lifeless> wallyworld: its not how it is intended to be used though, so I'm curious why that is done
<wallyworld> lifeless: it's an extenstion of a prior "pattern" in our js
<lifeless> wallyworld: Possibly, but humour me
<lifeless> wallyworld: assume I've no prior knowledge and need to be taught why we do this to a mustache template
<wallyworld> lifeless: going way back, people used Y.Node.create(html) where the html was ['stuff', 'stuff'].join(' ')
<lifeless> wallyworld: ok; but what joins that into how we use mustache? why is it relevant?
<wallyworld> then, in some previous code review, it was suggested that a mustache template be used to get the html
<wallyworld> so, html = Y.lp.mustache.to_html(template) where template = [xxx].join(' ')
<lifeless> wallyworld: ok, I'm clear on what is being done, I'm not clear on why.
<wallyworld> it was used for a local popup dialog or something
<lifeless> wallyworld: you may not know, which is fine.
<lifeless> wallyworld: if you don't, we can skip trying to understand as a group, and move on; if you do know, I'd really like to grok this.
<wallyworld> lifeless: so, it wasn't for main view content, but for a confirmation dialog
<wallyworld> where the data to ask for confirmation for was best rendered using mustache
<wallyworld> since it contained a list of things
<wallyworld> easier than building the html by hand
<lifeless> wallyworld: that would normally be to_html(template, data)
<wallyworld> yes
<lifeless> where data has that list in it
<wallyworld> yes
<lifeless> so why was this done differently ?
<wallyworld> define "this"?
<lifeless> 14:49 < wallyworld> so, html = Y.lp.mustache.to_html(template) where template = [xxx].join(' ')
<wallyworld> i left out the data param
<wallyworld> pseudo code and all that
<lifeless> why isn't template static though ?
<wallyworld> it is
<lifeless> Perhaps I misunderstand
<rick_h_> it is, but it has to live somewhere. Usually, you write out a <script type="text/template"> block that contains the static html.
<lifeless> template = [xxx].join(' ') looks like it is string joining to create the template
<lifeless> that is not static
<rick_h_> that ends up getting made available on the page via your template
<lifeless> its dynamic
<lifeless> rick_h_: not helping
<wallyworld> lifeless: our coding standard dictate that approach
<wallyworld> lifeless: no static, but coding standards force us to do it that way
<wallyworld> break up the string into bits
<rick_h_> lifeless: ok, can we do a hangout or something? I can hop on the phone. wallyworld is correct, this is a known issue I've seen coming.
<wallyworld> line length and all that
<lifeless> rick_h_: sure, we can do that
<rick_h_> and I can walk through it
<wallyworld> mumble ?
<wgrant> Oh
<rick_h_> I can't, I'm out at a coffee shop without headset/etc
<rick_h_> I can use skype or hangout via my cell
<wgrant> You mean just as a replacement for """?
<lifeless> skype or hangout - they have echo cancellation
<wallyworld> wgrant: js dorsn't have """
<rick_h_> yea, but in a noisy coffee shop with 10 other techies talking/hacking
<wgrant> Right, but that's a faaaaaaaaaaaaaaaaaaaaaaar more restricted case than what you said :)
<wallyworld> let's try skpe but mine is flakey
<rick_h_> ok, my skype is mitechie
<wgrant> It made it sound like you were generating the template dynamically, rather than just concatenating strings in a static way because JS is hideous.
<wallyworld> wgrant: oh, i didn't mean to imply that
 * wallyworld installs skpye
<lifeless> wallyworld: you said it had to be in the js file because it was built up like this
<lifeless> wallyworld: but if it is only built up like this because it is in the js file, then that is circular logic
<rick_h_> lifeless: it *needs* to be in the JS file for the unit test .html file to have access to it
<wgrant> Exactly.
<wallyworld> lifeless: i meant to convey it was in the js to avoid code cut and paste for tests
<wgrant> @lifeless
<wallyworld> i believe that's what i said, but it has been misinterpreted :-)
<lifeless> rick_h_: I get that, but see my point about compilation above.
<lifeless> who wants in on this kall ?
<wallyworld> me as soon as i install skype
<timrc> I have a general yui testing question (that I also asked in #yui) and since you guys have a bit of experience, perhaps you would know... I'm trying to test DOM events on a form, e.g. when I select an item from this dropdown it updates a textarea based on the choice, but do not know how to do this short of embedding the tests in the page
<lifeless> timrc: sounds like you are testing across abstractions
<timrc> probably ;) I accomplished this with jasmine using fixtures, but now I must yui
<rick_h_> timrc: check out node-event-simulate to trigger events and then check callbacks are run/processed
<timrc> rick_h_, but can I actually load the form into the dom in a seperate test page?
<rick_h_> timrc: so no, you'd use the setUp of the suite to generate the html required
<rick_h_> the .html page the tests run on is a giant fixture
<rick_h_> you can preload with html, use setup/teardown to setup html/clear it between tests
<rick_h_> timrc: see lib/lp/app/javascript/indicator/tests/test_indicator.js
<rick_h_> if you can access the LP source for an example
<timrc> rick_h_, will do. damn typing with a 7 mo old in your lap is challenging
<rick_h_> heh, understand completely :)
<wgrant> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/app/javascript/indicator/tests/test_indicator.js
<rick_h_> thanks wgrant, wifi here is awesome and didn't want to try to pull it up on my phone :)
<lifeless> short urls R us
<rick_h_> by awesome I mean fail
<wgrant> rick_h_: What sort of phone do you have?
<rick_h_> nexus
<wgrant> rick_h_: I need someone with an iPhone to try that page
<lifeless> handheld
<wgrant> Ah
<rick_h_> sorry, galaxy nexus
<lifeless> wgrant: grab anyone from design
<wgrant> iOS Safari doesn't like my CSS
<wgrant> Making the line numbers crap.
<wgrant> lifeless: Hah
<rick_h_> guy here has one, sec let me get him the url
<rick_h_> wgrant: have a guy tring it out
<wgrant> rick_h_: Ah, thanks. We've had one report that it sucks, mostly want confirmation before I work out how to dig further.
<wgrant> But I suspect that I have to throw it at someone else, since I don't have any way of running iOS Safari
<wgrant> I love Apple so much.
<rick_h_> wgrant: well it *looks* ok, the only issue is that the line number font is < code font so they don't line up
<timrc> rick_h_, no way to say "load this url into the DOM" and then test the result? :)
<wgrant> rick_h_: Yeah, that's the issue.
<rick_h_> wgrant: so you run out of line numbers, but it reads ok otherwise
<wgrant> Thanks.
<rick_h_> timrc: so what'd you'd have to do is iframe into the .html page the content you want loaded, and then copy it out via JS in your setUp method
<wgrant> The CSS needs redoing, but it works on desktop Trident/WebKit/Gecko/Presto :(
<rick_h_> timrc: off the top of my head, the issue is that it's raw plain .html so you can't *include*
<timrc> rick_h_, ah clever, but I can sort of see where lifeless is comin from when he says "testing across abstractions"
<rick_h_> wgrant: yea, I mean the colors/etc look ok
<rick_h_> timrc: definitely
<timrc> rick_h_, maybe jasmine encouraged bad behavior... typically you would test form interactions with selenium, no?
<rick_h_> timrc: so if your code is responsible for the generation of the html it's less an issue
<rick_h_> timrc: well you're hitting BDD vs TDD
<rick_h_> timrc: it's a root thing that behaviour is much higher up the stack of a functional test
 * wallyworld stabs apt-get update... sooooo sloooow
<rick_h_> timrc: so the stuff you're doing with jasmine are really where the sst type stuff would come into play, or in YUI would be if you're creating full Y.Widgets that the html/interactions are tied into a single unit to test with
<timrc> rick_h_, I think what I'll do for at least now is take my jasmine fixture and copy-paste it into my yui test html file that should achieve what you're talking about
<rick_h_> timrc: rgr
<rick_h_> timrc: it's something we've got to work on as we start to get more complicated and need higher level behavior tests than we currently run
<lifeless> mmm
<lifeless> so BDD doesn't want to test the whole stack
<lifeless> thats a false meme
<timrc> apparently people want to build armhf images yesterday, so I'll take the path of least resistance, here
<rick_h_> timrc: yea, like that example wgrant posted, we generate bootstrap html in the test to run against, it's a common 'fixture' practice
<lifeless> BDD wants to test the interactions between components - e.g. when A is the SUT test that asking A to do X results in A asking B for help
<lifeless> TDD might also look at the state of A before and after, and BDD frowns more on that than pure TDD does
<lifeless> but even within TDD there are huge differences of opinion there
<rick_h_> right
<lifeless> the key thing that *both* TDD and BDD say here
<lifeless> is that if you are using a known good component
<lifeless> DO NOT TEST IT
<lifeless> test /how/ you use it.
<rick_h_> right, so he's starting with a BDD "fixture" and wanting to copy/pull that into his TDD somehow as a "fixture"
<lifeless> not what it does
<rick_h_> in jasmine, it's including an external html file so he wants to "import" that into the YUI .html test file, but not a good way to do that since it's raw .html without server side includes/imports of any sort
<lifeless> yeah
<timrc> jasmine structures tests as it("should do this when this condition is met", function())
<rick_h_> timrc: yea, and you can create a test in YUI that's "should do this ..." that get run after setUp()
<rick_h_> we use the test_something_does_something: function format
<rick_h_> but the text string "this should that" also work
<rick_h_> it just doesn't nest like a jasmine tests would
<lifeless> so I think I'm saying that testing here is really checking that the event is raised, and that the widget is subscribed to it
<lifeless> testing between them is a bit odd
<timrc> so maybe this is actually a job for something higher up the stack, when the user selects item A, the textarea is populated with X, Y, Z ?
<rick_h_> yea, normally we'd split the test. One that the event is fired by mocking the callback
<rick_h_> and secondly, that given a method with the right/wrong input data that the callback does what we think it should do
<rick_h_> timrc: right, that starts to sound like a live functional test which might be run via sst
<rick_h_> or you could test that when the user selects an item, an event is fired with A's data
<rick_h_> and then, firing a custom fake event with data XXX, the textarea will populate with other stuff
<rick_h_> so two different sets of tests, one against the select/drop down, and one against the textarea
<rick_h_> that'd be breaking it more into unit tests, using two suites, you could have different setUp() and tearDown() and those would bootstrap a select element to test against
<rick_h_> and the second suite would setup a textarea that you test against
<wgrant> StevenK: How goes the AccessPolicy creation?
<rick_h_> oh StevenK so I heard convoy RT is right after the librarian san stuff fyi
<timrc> rick_h_, I'm quickly learning why QA is its own discipline, but I'm happy to learn.  I fear that the complexity of testing is going to prevent me from getting very flashy in Offspring :)
<rick_h_> timrc: yea, I mean that's why it's a giant can of compromise
<wgrant> rick_h_: librarisan is hopefully only a couple of days away :)
<rick_h_> the idea is that you get somehting that tests, you can add things you break to, but it's always going to have gaps/wholes
<rick_h_> wgrant: ooh, that makes me so happy
<rick_h_> it sounded like it was something rather large/far out
<wgrant> Well
<wgrant> It's actually running partly on the SAN atm
<wgrant> Just on the old machine via NFS.
 * timrc hisses at the mention of NFS
<wgrant> The firewall was opened this morning, so it's just a couple of deployment and monitoring things left before we can switch.
<wallyworld___> rick_h_: lifeless: finally got skype installed
<rick_h_> wallyworld___: cool
<lifeless> wallyworld___: rick_h_: ok, gimme a couple minutes to put a load of washing on
<wallyworld___> ok
<rick_h_> lifeless: rgr
<rick_h_> I've got 30min before close
<wallyworld___> rick_h_: i tried calling you but my request was rejected
<rick_h_> wallyworld___: sorry, missed your request, added you to my contact list now
<lifeless> wallyworld___: ah, if you're hosting, call me in
<timrc> rick_h_, anyway, thanks for the help, I think I have enough to go forward.  Really appreciate it
<StevenK> wgrant: I still have test failures. :-(
<wgrant> StevenK: Oh?
<StevenK> wgrant: Let me pastebin a diff and the failures
<wgrant> That would be ideal.
<StevenK> wgrant: http://pastebin.ubuntu.com/862865/
<wgrant> StevenK: What happens if you try to run it manually?
<wgrant> It's also not inconceivable that you might have to commit/abort the transaction before you can see the result.
<StevenK> wgrant: http://pastebin.ubuntu.com/862870/
<StevenK> But __call__ in the job calls commit :-(
<wgrant> That's in the job.
<wgrant> Which is hopefully a separate process, and therefore a separate transaction.
<StevenK> No other test calls commit after self.runHourly()
<wgrant> Ah
<StevenK> I'm of the opinion that I've missed something utterly stupid and I've been chasing it for hours.
<wgrant> You've run make schema to ensure that the security change made it to launchpad_ftest_template?
<StevenK> Yup.
<wgrant> Ensure that statement logging is enabled and watch your postgres log during the test run.
<wgrant> also, most of __call__  can be replaced with a list comprehension.
<StevenK> Oh?
<StevenK> I'm already using one in what I thought was a clever way.
<wgrant> You construct an empty list, then iterate over a tuple, on each iteration adding some tuples to the list.
<wgrant> policies = [(distro, policy) for distro in self.findDistributions()[:chunk_size] for policy in (USERDATA, PROPRIETARY)]
 * StevenK sprinkles in AccessPolicyType
<wgrant> Although this is unfiltered, so you can also just use itertools.product
<lifeless> ok, skype call done
<lifeless> tl;dr for folk here:
<lifeless>  - handlebars really wants to be treated more like C - server side compile, separate source, mustache is a middle ground
<lifeless>  - can possible just use an iframe to suck in templates we need, ian is trying that, should work on mustache too [for the tests]
 * StevenK laughs at bigjools new profile picture
<wgrant> Heh, yes.
<StevenK> wgrant: LP_DEBUG_SQL=1 isn't good enough?
<wgrant> StevenK: Not sure if that works for garbo subprocesses in tests.
<StevenK> Right
<wgrant> We often run subprocesses in a sanitised env.
 * StevenK reaches for postgres' conf
<lifeless> can whitelist that parameter probably
<StevenK> I leave it off so /var doesn't fill up after a days test runs
<lifeless> sure
<lifeless> I meant for subprocs
<lifeless> so that when you set it, it wrks
<wgrant> I leave it on because my hard disk isn't from 1990.
<StevenK> /dev/mapper/sys-var   4.6G  1.6G  2.8G  37% /var
 * StevenK sighs at the hidden scrollbars crap
<StevenK> Makes dealing with long output bloody tedious
<wgrant> Does gnome-terminal have scrollbars at all any more?
<wgrant> Oh, it does.
<wgrant> But the thumb is invisible afaict.
<StevenK> Right, 100,000 lines of output later
<StevenK> (I wish I was joking)
<lifeless> rick_h_: oh, and the site I was pointed at demoing yui_handlebars has the precompiled templates referenced by its combo loader
<lifeless> rick_h_: so I think that is actually more common than you may think :)
<StevenK> wgrant: http://pastebin.ubuntu.com/862893/
<wgrant> StevenK: So, garbo seems to be behaving properly. What about the test?
 * StevenK is just about to dump everything containing accesspolicy into a paste
<StevenK> wgrant: http://pastebin.ubuntu.com/862902/
<wgrant> StevenK: So, I would suggest breaking with pdb and then poking around in psql
<wgrant> Or potentially from within pdb if you are brave.
<StevenK> Perhaps my find* and isDone methods are to blame
<wgrant> I doubt it.
<wgrant> We can see they're being inserted.
<wgrant> Ah, you may be right.
<wgrant> Distro 3 is not new.
<StevenK> Oh?
<wgrant> Also, calling AccessPolicy.create like that is illegal
<wgrant> getUtility(IAccessPolicySource).create
<StevenK> What the? Now it's working?
<rick_h_> lifeless: linky?
<rick_h_> lifeless: on the precompiles template stuff?
<StevenK> rick_h_: I think's awesome you're using using convoy with bookie
<StevenK> s/'s/ it's/
<rick_h_> StevenK: yea, getting some good tests out of it and swapping code back/forth
<rick_h_> StevenK: though it's nginx vs apache
<rick_h_> well, gunicorn in dev, but in prod nginx
<StevenK> The nginx fanbois turn me off trying it
<rick_h_> heh, I'm a fan. I read through an apress book (I think) and since I"m running ec2, lower memery usage ftw
<StevenK> There are a bunch of them in #linode and anytime someone turns up with an Apache problem, regardless of the problem, they spend hours trying to convert the guy to use nginx
<rick_h_> heh
<rick_h_> yea, well I know apache has a lot more uses
<rick_h_> but for lower memory it's a win
<StevenK> wgrant: So here's an interesting data point. If I run test_AccessPolicyDistributionAddition in isolation it works, but running test_garbo causes it to fail
<lifeless> rick_h_: grah, uhm, lets see
<rick_h_> lifeless: np, just curious
<lifeless> rick_h_: beta.tdp.me
<lifeless> rick_h_: http://tdp.me/combo?template/template-goal-table/template-goal-table-min.js
<rick_h_> lifeless: ah cool
<lifeless> irssi-2012-02-09:09:00 #launchpad-dev: < rick_h> lifeless: so this guys in #yui gave me this for some handlebar exapmles: http://beta.tdp.me/static/scripts/bundle.js http://beta.tdp.me
<lifeless> :)
<rick_h_> yea, I remember seeing it now
<lifeless> dunno who, jshirley I think
<rick_h_> yep
<lifeless> the nginx casual approach to http turns me off :)
<StevenK> wgrant: Running under garbo I never see the created distribution
<rick_h_> wallyworld___: http://jsfiddle.net/83xZY/ seems to work
<rick_h_> wallyworld___: note that it's running off the on load event
<wallyworld___> rick_h_: i get an internal yui error
<wallyworld___> will try of the onload
<wallyworld___> i also tried via the dom element directly but got a permission error
<rick_h_> wallyworld___: bah, that's what I was afraid of, it has to be same domain and prefix
<rick_h_> file://
 * wallyworld___ tries that
<wgrant> sameorigin on the file scheme is a bit awkward :/
<rick_h_> yea, but it's meant to prevent http/https crossing
<rick_h_> I'm guessing more https making http calls
<wgrant> I mean I don't think any two URLs from the file scheme are considered same-origin.
<rick_h_> ah, yea I'm not sure of that
<rick_h_> I was hoping maybe the issue he hit was that the browser was file:// while his iframe src="../../" without an explicit file:// :/
<wgrant> (the scheme distinction is most often useful to prevent HTTP from doing HTTPS XHR)
<wgrant> Ah
<wgrant> "Starting in Gecko 1.9, files are allowed to read only certain other files.  Specifically, a file can read another file only if the parent directory of the originating file is an ancestor directory of the target file. Directories cannot be loaded this way, however."
<rick_h_> but I'm guessing at this point
<wallyworld___> rick_h_: nah, still permission denied
<rick_h_> heh, because it's not hard enough...
<rick_h_> well then ajax requests won't fly either...
<StevenK> Why the heck isn't this new distro turning up in the store? :-(
<wallyworld___> rick_h_: i'm about to give up and say that i need to land the mp with the template in the js
<rick_h_> wallyworld___: I'll +1 that with a note that it's something I'll investigate as I work on fixing hte combo loader going forward
<wallyworld___> ok, excellent. thanks
<rick_h_> it'll be a large todo item so it doesn't fall away, but as part of the combo loader completion story
<rick_h_> it's going to require more testing than I can do past my bedtime :)
<wallyworld___> yeah :-)
<rick_h_> sorry again that it fell into your lap like this
<wallyworld___> i just want to get myself unblocked for now
<rick_h_> wallyworld___: definitely
<wallyworld___> rick_h_: no need to say sorry, i was dreading this issue as well
<StevenK> What's the issue?
<rick_h_> StevenK: long story :)
<rick_h_> wheee, and mailman day approacheth!
<wallyworld___> StevenK: we want to have the mustache template in one place
<wallyworld___> availabole for yui tests
<wallyworld___> and for prod
<StevenK> rick_h_: It's been Mailman day here for 15 hours
<wallyworld___> and it's hard to do
<rick_h_> StevenK: heh, just got my first one of the day
<StevenK> Oh yeah, mail. I guess I should read that.
<rick_h_> wallyworld___: you try out the auto js builder bits I emailed the list about?
<rick_h_> I need to find people that can try/test that if you get some future JS work to tinker with
<stub> w00t. pg9.1.3 packages out, containing the plpythonu fix we want for inplace upgrades
<rick_h_> wallyworld___: or someone else on your team
<wallyworld___> rick_h_: not yet, but it looks nice
<rick_h_> wallyworld___: k, let me know if you guys can give it a spin and such
<rick_h_> eventually want to see if we can mail the -tech about it if it works out ok
<wallyworld___> rick_h_: will do. it's on my todo list to try out for sure
<rick_h_> ok, time to go pass out, night
<wallyworld___> rick_h_: see ya. thanx for your time
<wgrant> oops
<wgrant> Was wondering why one of my triggers was taking 25ms to reflatten a bug and all its tasks.
<wgrant> Forgot that I hadn't removed the 250 tasks I'd created on the bug yesterday.
<stokachu> http://pastebin.com/ah8aLQdL - ive managed to setup the authorization header which then seems to redirect me properly from /people/+me to /~adam-stokes
<stokachu> but then i get a 303 see other error
<stokachu> any ideas
<wgrant> stokachu: 303 is the redirect
<wgrant> It's not an error.
<wgrant> Also, *really* not a good idea to give your OAuth tokens to the world.
<wgrant> You probably want to revoke that one.
<stokachu> i will
<stokachu> is staging still down?
<wgrant> It should be up.
<wgrant> Has only been down for a couple of hours over the last few weeks.
<stokachu> ah , there was a code update an hour ago
<wgrant> Hmmm
<wgrant> That should only take it down for a couple of minutes, I'd expect.
<wgrant> Perhaps something is going wrong with the updates.
<wgrant> It's up now.
<stokachu> ah ok... so im in this redirect from the last response
<stokachu> but should i be calling ~adam-stokes directly?
<wgrant> Depends on your HTTP library.
<wgrant> Some will follow the redirect automatically, others won't.
<stokachu> hmm let me try to hardcode my name in and see what i get
<wgrant> /people/+me does an HTTP redirect to the current logged in user.
<StevenK> Or /~
<stokachu> ah yea so this oauth library isn't following redirect
<stokachu> i dont guess anyone is familiar with drupal 7 and its oauth stuff
<StevenK> There's a Drupal 7 now?
<StevenK> :-)
<stokachu> lol
<wgrant> You know
<wgrant> It would be really nice if we had a sensible data access layer.
<wgrant> So we could, like, not be terrible.
<stokachu> its only a little complicated
<stokachu> not to bad
<wgrant> Not talking about the API, but that's not ideal either :)
<stokachu> haha
<StevenK> Hmmmmm, what database user does mail handling use?
<wgrant> StevenK: Should be processmial
<wgrant> processmail
<stokachu> so if anyone is curious i have a 90% working drupal module for launchpad
<stokachu> it just doesn't do anything yet but auth
<stokachu> :X
<wgrant> stokachu: What are you planning to use it for?
<StevenK> wgrant, wallyworld___: Either of you feel like reviewing https://code.launchpad.net/~stevenk/launchpad/change-bug-privacy-via-mail/+merge/95312 ?
<stokachu> mainly a dashboard between lp private and salesforce
<wallyworld___> StevenK: i'm deep in some code changes, can do soon
<wgrant> StevenK: OMG SOMEONE FIXED IT
 * wgrant +1s
<StevenK> Hahah
<StevenK> I tagged the bug disclosure, I figured it was tiny enough we could claim it as a feature branch
<stokachu> and possibly some helper stuff for submitting SRU's
<StevenK> wgrant: So, looking at the not found page, you'd like some vertical whitespace between the heading and the first paragraph?
<wgrant> StevenK: I don't know.
<wgrant> Users currently don't get it.
<wgrant> So the page needs *some* change.
 * StevenK sprinkles in <strong> along with changing the heading to "Not Found" and puts it up for review
<StevenK> IE, let's make a token effort and see what Dan comes back with
<czajkowski> aloha
<stub> Didn't I add a 'previous OOPS' feature?
<stub> OOPS caused due to thread state, so I need to know the previous screwup. Could have sworn I added that into the OOPS already...
<stub> Did... so why isn't it in my OOPS?
<lifeless> stub: you did
<stub> Yer, found other OOPSes with the key. But the one I'm interested in doesn't have it. Might mean it was the first OOPS generated by that thread, but I somehow doubt it.
<stub> But it is generated from beforeTraverse, so things might get weird.
<adeuring> good morning
<mrevell> hi
<StevenK> danhg: Hai, have you seen https://code.launchpad.net/~stevenk/launchpad/not-found-page/+merge/95313 ?
<danhg> Hmmm.... thanks StevenK
<mabac> StevenK, hi there. did you get a chance to look at salgado's changes on https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883 ? just wondering if there's anything I can handle it before he wakes up. :)
<StevenK> mabac: Hai. I have commented.
<mabac> StevenK, cool, thank you
<mabac> StevenK, let's see if I understand that. :) you don't want us to revert the last change, it's only about setting that feature flag, right?
<StevenK> mabac: So, I'd like to see the loop move from the experimental loops to the usual one, and a feature flag a guard when it runs
<mabac> StevenK, ok got it. thanks
<StevenK> mabac: Of course, along with adding two tests, one to see the job doesn't run with no feature flag, and the inverse.
<mabac> StevenK, of course
<rick_h_> morning
<czajkowski> rick_h_: ello
<mabac> rick_h_, good morning. thanks for reviewing my branch! I have separated the parsing and validating parts of parseLine but don't know if there is a better error to raise when parsing than LaunchpadValidationError?
<rick_h_> mabac: I'm not sure on that end. I'll see what jcsackett brings up on that front
<mabac> rick_h_, ok thanks
* rick_h_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett, rick_h* | Firefighting: - | Critical bugtasks: 4*10
<jcsackett> rick_h_, mabac: almost done with that MP. i see no problem with LaunchapdValidationError, since ulitmately the method is only used as part of the set of calls from validate.
<jcsackett> if you didn't raise it as such there, you would probably want to reraise from within validate anyway.
<rick_h_> jcsackett: ok cool
<mabac> jcsackett, cool thanks.
<jcsackett> mabac: questions/comments posted on your MP.
<mabac> jcsackett, thank you. I'm on it
<jcsackett> mabac: cool. there's nothing major, just a few little things.
<mabac> jcsackett, doesn't look to bad. ;) I'll fix it up today.
<jcsackett> mabac: awesome. thanks. :-)
<mabac> jcsackett, I can remove that assert entirely btw. it can just catch programming errors as you noted
<jcsackett> mabac: that's what i thought.
<jcsackett> glad i wasn't off base.
<salgado> mrevell, did you see my email about announcing the work items changes we're doing in LP?
<mrevell> salgado, Sorry yes. I'll be on the phone for another 20-30 minutes then I'll replu.
<mrevell> s/replu/reply
<salgado> mrevell, that's cool, thanks!
<salgado> jcsackett, StevenK reviewed https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883 and said he was happy with it but wanted a new feature flag to prevent it from running on production until we're ready.  I was wondering if you'd be willing to review the feature flag changes and land it for me so that we can start QAing it in production?
<salgado> s/production?/staging?/
<sinzui> danhg, do you want try a screen share where we build a mockup together?
<danhg> Hey sinzui, That sounds great. I'm also working with Matthew on a broader process for testing -> mock-ups -> implementation.
<sinzui> danhg, great We can try this now and when you are ready
 * sinzui has never tied hangouts screen sharing
<salgado> matsubara, if you'd like to try and break the new work-items UI, it's at lp:~linaro-infrastructure/launchpad/workitems-widget
<matsubara> salgado, thanks
<matsubara> salgado, is it being reviewed?
<salgado> matsubara, yes, although most of the changes requested were in tests
<matsubara> ok
<matsubara> salgado, by tomorrow it'll be on qastaging, do you think?
<salgado> matsubara, that depends on QAing the migration script, I'd say.  do you need to wait until it's there to QA it?
<matsubara> yeah, it's better to do it on staging or qastaging with the full db
<jcsackett> salgado: sure, i'll take a look at it.
<salgado> matsubara, in this case the UI is just a text entry, similar to the whiteboard.  the only way to break it is by entering unparsable stuff and hoping for the worse, so having it on staging won't make any difference
<matsubara> ok
<salgado> mrevell, I guess you also didn't have time for the help text (https://bugs.launchpad.net/launchpad/+bug/944119)?
<_mup_> Bug #944119: Add help popup to the new work-item editor <Launchpad itself:New> < https://launchpad.net/bugs/944119 >
<mrevell> salgado, Hey, I can still do this. Is it okay if I just give you the text?
<salgado> mrevell, definitely!
<salgado> mrevell, as for the announcement, do you guys take care of that as well or should we write it ourselves?
<mrevell> salgado, We can take care of that. It'd be great to have an idea of anything in particular you want us to say, though. When do you expect to be ready to make the announcement?
<salgado> mrevell, ok, I'll write down the important points.  hopefully early next week
<mrevell> Thanks. I'll get the help text to you tomorrow.
<salgado> mrevell, great, thanks!
<mabac> jcsackett, there I believe I've fixed your suggestions for lp:~linaro-infrastructure/launchpad/workitems-widget
<mabac> jcsackett, I'll clock off now but if there's anything that's not clear salgado-lunch will be able to answer and I'm back online in ~14 hrs :)
<czajkowski> salgado-lunch: give me a shout when you;'re back
<czajkowski> we should talk :)
<abentley> adeuring: I've got tests that prove a job runs under Celery.  Any advice on how to clean them up? ~abentley/lazr.jobrunner/run-via-celery
 * adeuring is looking
<adeuring> abentley: "make check" just gives me "TypeError: 'NoneType' object is not callable" ...
<abentley> adeuring: Yeah, I talked to barry about that and he couldn't help me, so for now I'm using nosetests.
<adeuring> abentley: how do I start one?
<abentley> bin/python `which nosetests` should do it.
<adeuring> ah, got it
<abentley> adeuring: I particularly dislike the way I'm finding the celeryd binary.  Seems there should be a better way.
<abentley> adeuring: Also, the fact that config files are python modules prevents me from generating a config in a temp directory as I normally would.
<adeuring> abentley: what is your problem with the cleanup?
<abentley> adeuring: I don't know what you mean.  The 'NoneType' is not callable thing?
<adeuring> abentley: I menat your quetion "Any advice on how to clean them up?"
<abentley> adeuring: I think the implementation is dirty, particularly the way I find the celeryd binary and the fact that the testing config is a file in the tree.
<abentley> adeuring: Do you have any suggestions for improving the test implementation?
<adeuring> abentley: "cmdname = os.path.join(get_root(), 'bin/celeryd')" ? I don't see any better option
<adeuring> abentley: regarding the configuration of celeryd: the config module could read the parameters from a temp file, in order to set things like BROKER_HOST etc
<abentley> adeuring: How would it find the temp file?  Environment variables?
<adeuring> abentley: rigt. Or use some hard wired path: /tmp/celery_test.cfg or whatever
<adeuring> abentley: slightly crazy idea: pipe the config data: you use popen()...
<abentley> adeuring: And then config.py reads stdin?
<adeuring> abentley: yes, but may be soimply crazy...
<adeuring> I don't know if celeryd itself checks stdin...
<salgado> hi czajkowski
<czajkowski> salgado: howdy
<czajkowski> salgado: are you free tomorrow to talk re getting information out about blurprints?
<jelmer> pun intentional ? :)
<salgado> czajkowski, sure. tomorrow around 11:00 UTC ok with you?
<czajkowski> salgado:
<czajkowski> salgado: you on the canonical calender?
<salgado> czajkowski, yep, salgado should be my username there
<salgado> I see you've found me already :)
<wilx> Hi.
<jelmer> hi wilx
<czajkowski> salgado: lovely talk tomorrow so
<salgado> deryck, hi there. do you have any idea how hard it'd be to fix bug 415365?
<_mup_> Bug #415365: Errors are not handled properly in inline text editing <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/415365 >
<deryck> salgado, hey, let me look....
<deryck> salgado, by fix, do you mean prevent the error, or do you mean cope with the error better?
<salgado> deryck, just display the body of the http response when it's not a 200, I'd say
<deryck> salgado, right.  so to do that, it should be fairly straightforward, making showError do the right thing, rather than inserting into the inline widget as it's trying to do now.
<salgado> deryck, some of those widgets are used on fields that have validators (e.g. bug tags and the new blueprint work-item) so there will always be errors; we just need to display them
<salgado> deryck, oh, so showError already has the error message from the response? I thought that's what we were missing and I'd have to get that
<deryck> salgado, I think it has the error, and just not dealing with it well.
<deryck> salgado, but it's been awhile since I touched that :)
<salgado> deryck, yeah, and I'm completely clueless when it comes to YUI.  can I stick a "debug" call in there to inspect it in the browser?  I seem to recall doing something like that
<deryck> salgado, yup.  add "debugger;" on the line you want to break on and firebug or chrome console with break there.
<salgado> ah, debugger
<salgado> cool, will try that
<deryck> salgado, you can also open the file in those tools and manually click lines to add break points without editing code.
<salgado> deryck, sounds more promising as it looks like I'd need to regenerate launchpad.js after changing showError in the code itself, no?
<deryck> salgado, yes, you'd need to run make jsbuild again.
<salgado> deryck, what's weird is that the validate() method uses showError (when the widget doesn't accept empty input) and in that case it works fine, correctly displaying the message passed to it
<deryck> salgado, so maybe the message is getting dropped somehow?
<salgado> deryck, nope, showError is being passed the body of the response
<salgado> deryck, showError inserts the error on the correct place but since this is a multi-line field the text area appears on top of that
<deryck> salgado, I'd have showError open an overlay, rather than try to insert into the page.
<deryck> there's tons of error_overlay examples lying around
<salgado> yeah, that makes sense; /me tries that
<lifeless> james_w: you should show pkgme to gary_poster
<james_w> lifeless, for use in something
 * gary_poster googles pkgme
<james_w> lifeless, or just to get some stuff packaged?
<lifeless> james_w: I believe he is, or has, someone packaging up a new thing
<lifeless> james_w: which is in python
<lifeless> james_w: and so a prefect candidate
<gary_poster> james_w, yes.  Very simple package, and bac is writing up descriptions on how to do it
<james_w> gary_poster, a standard setup.py app?
<gary_poster> yes james_w
<james_w> it should work then
<gary_poster> and bac found the existing packaging instructions worthy of simplifying and rewriting
<james_w> that's a good thing to do anyway
<gary_poster> cool
<james_w> but pkgme may take the pain out of simple things entirely
<gary_poster> thanks lifeless, james_w .  bac, http://pkgme.net/ fwiw
<james_w> lp:pkgme
<gary_poster> saw that too, but the instructions seemed nicer to link to :-)
<lifeless> right, thats my good deed for the day, off to create havoc
<gary_poster> heh
<rick_h_> jcsackett: ping, can you peek at: https://code.launchpad.net/~rharding/launchpad/911857_extract_js/+merge/95438
<rick_h_> sorry for the big add/remove due to tabbing in
<jcsackett> sure, rick_h_.
<rick_h_> jcsackett: ty
<bac> thanks gary_poster and james_w
<bac> gary_poster: it may be useful for the py part of charm-tools
<gary_poster> bac, makes sense
<bac> james_w: should pkgme work for precise?  there is no ppa and the description says only natty and oneiric
<james_w> bac, yes it should
<bac> great
<james_w> bac, where do you see natty and oneiric?
<james_w> I'll update it
<bac> james_w: https://launchpad.net/~pkgme-committers/+archive/dev/+packages ... but now that i look closely that seems to be generated text
 * james_w copies forward
<james_w> done
<james_w> yeah, the text is autogenerated, it has updated now
<salgado> jcsackett, if you have a few more minutes, I've just proposed a new one. this time it's trivial: https://code.launchpad.net/~salgado/launchpad/bug-415365/+merge/95447
<salgado> how do I run a test under lib/lp/app/javascript/tests/?
 * salgado tries lading the html files on his browser
<rick_h_> salgado: yep, that works for dev
<rick_h_> and then you can load it with the test runner based on the filename
<rick_h_> which test file are you running?
<jcsackett> rick_h_: r=me.
<rick_h_> jcsackett: ty
<jcsackett> salgado: can you include a screenshot, since this is changing some of what's displayed on error?
<salgado> jcsackett, sure, https://bugs.launchpad.net/launchpad/+bug/415365/+attachment/2798570/+files/overlay-error.png
<_mup_> Bug #415365: Errors are not handled properly in inline multi-line text editing <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/415365 >
<salgado> rick_h_, inlineedit/tests/test_inline_edit.js (adding a test for the bug above)
<rick_h_> salgado: yea, so ./bin/test -x -cvvt inline_edit would run those tests
<salgado> rick_h_, nice, I'd never have expected that to work
<rick_h_> salgado: but yea, I'd just run from the browser usually in dev and the cmd line just to make sure things are cool from the html5 browser
<rick_h_> autoreload makes the browser version nice
<rick_h_> salgado: replied to your MP
<rick_h_> let me know if it doesn't make any sense
<salgado> rick_h_, heh, just found that out myself; one of the tests was failing :)
<salgado> I added lp.app.errors but that wasn't enough
<rick_h_> salgado: yep, figured you were onto that when you asked about running tests
<rick_h_> right, errors requires someoverlay modules
<rick_h_> make sure you link to them out of the build directory like the others pls
<bac> hi james_w, i've grabbed the trunk of pkgme and installed it with setup.py.  attempting to run errors due to non-execute bits on all of the 'wants'.  i manually chmod all of them but get a different error.  did i do something fundamentally wrong?
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10
* rick_h_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<rick_h_> salgado: running away, but ping me if you hit any issues and I'll be checking in later
<salgado> rick_h_, link to what?
<james_w> bac, nope
<rick_h_> salgado: ok, just that the html files links should be like the current ones, through build/...
<james_w> bac, the executability thing is a known problem with distutils
<james_w> bac, what's the other error you got?
<rick_h_> salgado: such as <script type="text/javascript" src="../../../../../../build/js/lp/app/formoverlay/formoverlay.js"></script>
<salgado> oh, I see
<bac> james_w: vala backend returning non-integer
<james_w> bac, hmm, odd
<salgado> rick_h_, let me double check, but I guess it'd fail if I had it wrong (I copied from another test.html file)
<james_w> bac, did it say what it did return?
<rick_h_> salgado: you should be good then if you copied since I updated them all recently
<salgado> rick_h_, yep, it looks just like the others in that template :)
<rick_h_> salgado: coolio
<bac> james_w, wait that seems to be another permission problem with with auto_conf_config
<bac> james_w, i'll go forward fixing the execute privileges
<james_w> bac, ah, helpers/* will need to be +x too
<bac> ok
<rick_h_> deryck: jcsackett had to run, can you second my ok on this please? https://code.launchpad.net/~salgado/launchpad/bug-415365/+merge/95447
<deryck> rick_h_, done.
<rick_h_> ty
<salgado> I'm trying to run js tests inside an lxc and am getting "RuntimeError: Gtk couldn't be initialized".  any ideas what I'm missing?
<rick_h_> salgado: http://bazaar.launchpad.net/~sinzui/html5-browser/trunk/view/head:/README installed?
<salgado> rick_h_, I think I do; I get that when import html5browser
<rick_h_> ok
<rick_h_> so most of these things need  xvfb  but not sure why you'd get a gtk issue
<rick_h_> sinzui: any ideas? ^
<rick_h_> salgado: have a traceback to paste perhaps?
<salgado> this is what actually happens when the test imports html5browser: http://paste.ubuntu.com/864114/
<lifeless> folk who would like a nicer deploy report should check out lp:~james-w/+junk/deploy-tracker
<salgado> rick_h_, seems to work with xvfb-run
<lifeless> http://ec2-23-20-113-165.compute-1.amazonaws.com/ - turn on 'team info' during SSO handshaking
<sinzui> salgado, that looks like the wrong screen.
<sinzui> salgado, does html5browser work when run from a console on out owne machine?
 * sinzui should have added the show_browser option so we could see it run
<salgado> sinzui, I tried 'bin/test -t inline_edit' inside a container and it failed with that traceback when html5browser tried to import Gtk.  running with xvfb-run worked
<salgado> I don't have html5browser installed on my machine; only on that container where I run LP
<sinzui> salgado, okay. This is a bit out of my realm of confidence
<sinzui> salgado, Is this lucid, oneiric or precise in the container
<salgado> sinzui, oneiric
<sinzui> okay, that is the version I wrote first and actually expect to work
<sinzui> salgado, the official answer is that Gtk cannot find the display. If you always run in a container you may need to pass the display in the env. Maybe we need to smarten up our test script to warn if there is no display set
<salgado> rick_h_, would you mind landing https://code.launchpad.net/~salgado/launchpad/bug-415365/+merge/95447 for me as well? :)
<sinzui> I think html5browser should catch the check error and explain the display it thinks it has.
<salgado> sinzui, indeed, with the DISPLAY set it works
<sinzui> salgado, I reported Bug #944348 about this
<_mup_> Bug #944348: warn when display is not set. <HTML5-browser:Triaged> < https://launchpad.net/bugs/944348 >
<salgado> sinzui, nice, thanks!
<rick_h_> salgado: ok, working on getting landing going
<salgado> rick_h_, great, thanks again :)
<rick_h_> salgado: np
<abentley> deryck: There's no OCR.  Would you be able to review https://code.launchpad.net/~abentley/lp-dev-utils/ec2/+merge/95468 ?
<deryck> abentley, sure.
<abentley> deryck: thanks.
<deryck> abentley, that's alot of python. :)  Is it all straight cut over from lp tree, except for the change about commit tags in MPs?
<abentley> deryck: Here are the changes I made: http://pastebin.ubuntu.com/864191/
<deryck> abentley, ah, that's very helpful thanks.
<abentley> deryck: Mostly it's import name changes, but there are some minor tweaks.
<deryck> abentley, looks fine.  And we know it works. :)  r=me
<abentley> deryck: thanks!
<lifeless> abentley: btw the location of trunk has changed
<lifeless> abentley: its now lp:~canonical-launchpad-branches/lp-dev-utils/trunk (if you aren't using hte lp:lp-dev-utils alias)
<abentley> lifeless: I used lp:lp-dev-utils
<mtaylor> lifeless: hey! is it worth filing a ticket about slowness of mailing list messages?
<lifeless> no
<lifeless> we like it slow
<mtaylor> awesome
<lifeless> mtaylor: actually, there is one I think, and its a well known issue
<lifeless> mtaylor: I can point you at the discussion on the -dev list if you like
<lifeless> we're || close to a solution
<mtaylor> I'll just take your word for it that you're aware and have it almost fixed
<mtaylor> as you rarely lead me astray
<lifeless> quadratic archive page writes
<lifeless> sinzui has a grackle.
<mtaylor> jeez
<bigjools> has he seen a doctor?
<mtaylor> that doesn't sound good
<sinzui> I have not, but having spent about 9 hours out of my skull I think I might
<sinzui> I still fear that If I yawn, the a hole in the universe will open and I will fall though it again
<deryck> Later on, everyone.
<sinzui> mtaylor, There is a specific critical bug that is the causes of slowness. I speculated about how to address the immediate concern
<mtaylor> sinzui: awesome
<sinzui> mtaylor, lifeless, wgrant: I think this bug needs fixing: bug #942305 right away
<_mup_> Bug #942305: forster is load spiking due to LP mailman integration <canonical-losa-lp> <mailing-lists> <Launchpad itself:Triaged> < https://launchpad.net/bugs/942305 >
<sinzui> I have an idea in it that need validation
<lifeless> looking
<mtaylor> sinzui: looking as well - as is jeblair
<lifeless> sinzui: which lists are key?
<sinzui> ubuntu-x-swat is the leading cause
<wgrant> It was a few months ago, at least.
<wgrant> By a significant margin.
<wgrant> Similar to the way we don't generate ubuntu-bugs@lists.u.c HTML archives.
<sinzui> we can look at the messages in the arch queue to see which are the pathological lists
<wgrant> lifeless: Heh, slony takes 80%+? Try 98-99.5%
<lifeless> wgrant: understate overdeliver
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Firefighting: - | Critical bugtasks: 4*10
#launchpad-dev 2012-03-02
<wallyworld> sinzui: you available to answer a question?
 * wallyworld goes to have lunch with bigjools
 * StevenK stabs mail handling
<wgrant> StevenK: Oh?
<StevenK> Two reasons. 1. Why does LP insist on sending me 5 mails about an MP that I just created and approved, and 2. I can't find what sends mail when a bug you created is marked as a dupe
<wgrant> StevenK: See BugDuplicateChange in bugchange.py
<StevenK> Oh fuck me, it's an *adapter*
<StevenK> I've been looking in model code
<StevenK> Right, this code is utterly disgusting.
<lifeless> StevenK: 'notification service'
<StevenK> lifeless: This is the same argument about bug changes. You don't get mailed about changes *you yourself* performed
<lifeless> StevenK: actually you do
<wgrant> Unless you are crazy and ask for them to not be sent.
<lifeless> StevenK: unless you have a) a new account or b) have turned the option off
<StevenK> Perhaps we want to drop the number from this notification.
<wgrant> It's difficult.
<StevenK> "** This bug has been marked a duplicate of private bug %d"
<lifeless> StevenK: the bug number they cannot see ?
<wgrant> Because the notifications are global.
<wgrant> For branch links it was solved by not notifying about private linkages.
<wgrant> But that can't be done hter.e
<lifeless> StevenK: so basically, a) apport is broken, and b) until its fixed we're committed to supporting something insane.
<lifeless> StevenK: there is an agreed upon fix for apport that pitti will, sometime, get to.
<wgrant> That's what I said on the call this morning :)
<wgrant> Anda  few months ago]
<lifeless> StevenK: I think tackling this before that fix is just churn, and hard. But we could do the fix ourselves, its retracer side not on-cd-side
<StevenK> I'd like to fix this "properly" and not give people links in mail that they can't actually follow.
<wgrant> Impossible.
<wgrant> Without rewriting the bug notification system.
<StevenK> wgrant: Then why didn't you say so on the call this morning? :-)
<wgrant> Huh?
<wgrant> I said to kick apport.
<wgrant> Maybe you discussed this during my fglrx crashes.
<wgrant> I never heard discussion about changing notifications.
<StevenK> Possibly
<StevenK> I said that I'd like to do that on the call and Curtis said it sounded great to him.
<StevenK> But that may have been the cold and flu meds talking. :-)
 * StevenK ponders calling this new service 'audit system service' ... changing that into an acroymn as an exercise for the reader
<timrc> sinzui, Can I be allowed here? https://launchpad.net/launchpad/+sharing :) pleease
<StevenK> No, you can't be.
<StevenK> Not yet.
<StevenK> lifeless: O hai. Any thoughts on a witty name for this audit service? :-)
<wgrant> StevenK: Any progress on the AP garbo job tests?
<StevenK> wgrant: I'd like some help. You mentioned yesterday it might be transaction boundaries
<wgrant> Break into it after garbo runs with pdb and see what's in the DB.
<StevenK> I've done printf debugging pulling things from the store
<lifeless> StevenK: http://pypi.python.org/pypi/auditor
<StevenK> Not found
<StevenK> :-P
<wgrant> We do like stomping on the global namespace, don't we.
<StevenK> Haha
<StevenK> Well, like lifeless said, we would like others to use it
<wgrant> Sure, but we're not the One True Auditor.
<StevenK> my-first-auditor
<StevenK> lp-audit-service?
<wgrant> It's not LP
<StevenK> I don't want to proclaim canonical- either
<lifeless> auditord
<lifeless> auditd
<lifeless> also unused there
<wgrant> First fix Python namespacing, then call it lazr.auditd
<lifeless> there is an auditd in ubuntu
<lifeless> wgrant: deprecated
<wgrant> lifeless: Only because Python namespacing is the most awful thing in the world.
<StevenK> And it doesn't touch zope
<lifeless> wgrant: it is, and also it is awful
<lifeless> mcollective-plugins-centralrpclog might be interesting to look at
<wgrant> It's the most awful thing in the world, and it's also awful?
<lifeless> yes
<wgrant> Good good
<lifeless> excatly
<lifeless> you could call it scientologyd
<StevenK> Right. Flights to Christchurch booked.
<StevenK> Now lifeless will pay.
<lifeless> when do you want the spare bed put up ?
<StevenK> Plan is to kick you in the face for that last comment and then fly home. :-P
<lifeless> well, it audits
<lifeless> seriously, auditord IMNSHO is fine
<StevenK> You said auditor first
<lifeless> I did
<lifeless> thats fine too
<lifeless> but the namespace thing is true
<lifeless> so auditord for the server, probably auditor-client for the client if you decide it needs one
<lifeless> (I don't think it does)
<StevenK> I was going to write a module under lp.services that translates a given object into a string represantation of its EID, but I don't think it needs a client, given we can just scream HTTP at it
<lifeless> yup yup
<mwhudson> <wgrant> lifeless: Only because Python namespacing is the most awful thing in the world.
<mwhudson> wgrant: have you written anything in common lisp?
<lifeless> mwhudson: nothing common about it
<StevenK> Perl namespacing is pretty good
<wgrant> Python namespacing is fine, but packaging of it is a nightmare.
<mwhudson> ah yes, no arguments there
<StevenK> To be fair, it also omits one of SystemD's greatest strengths: Roman numeral-compliant project naming. System D is obviously way better than System V.
<StevenK> -- Steve Langasek
 * StevenK cackles
<wgrant> StevenK: Distro quotes section of LWN?
<StevenK> Yup
<StevenK> wgrant: http://pastebin.ubuntu.com/864582/
<wgrant> StevenK: Diff?
<StevenK> wgrant: http://pastebin.ubuntu.com/864586/
<StevenK> wgrant: Confused yet?
 * wgrant tries.
<StevenK> wgrant: I can push a branch if you wish
<wgrant> StevenK: Sorry, actally looking now.
<wgrant> Let's see.
<StevenK> Hah
<StevenK> wgrant: Did you get distracted again?
<wgrant> The tests take about 10 years to run, so yeah.
<wgrant> Easy to get distracted
<StevenK> Haha
<nigelb> Heh
<wgrant> StevenK: http://pastebin.ubuntu.com/864662/
<StevenK> wgrant: It's just the where clause?
<StevenK> Hm, it's a different query entirely
<wgrant> StevenK: And I used itertools.product instead of the nested list comprehension.
<wgrant> I'm not quite sure why the old one wasn't working, and the debug cycle is too slow, so I just rewrote it from scratch instead.
<StevenK> Hmm, now I get SilentLaunchpadScriptFailure instead
<StevenK> I HATE debugging garbo failures
<wgrant> StevenK: make schema to check the security is right?
<mabac> huwshimi, thanks for reviewing the workitems-widget branch! I'll switch so we have whiteboard first and then the work items text area. and I'll have a go at adding a edit mode message to the whiteboard
<huwshimi> mabac: OK great!
<mabac> huwshimi, do you (or anyone else :) have an example of something similar in lp? that is a message that looks or behaves like what you're after?
<mabac> rats
<StevenK> wgrant: Interdiff between yours and mine: http://pastebin.ubuntu.com/864684/
<StevenK> wgrant: But DistributionAddition gives SilentLaunchpadScriptFailure :-(
* wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10
<adeuring> good morning
<mabac> I'm trying to find a way to show a message on blueprints when the whiteboard is being edited. I think I'm on to something using tal:condition in specification-index.pt. Does anyone know how I can detect the state of a TextAreaWidget?
<mabac> If there's a standard way to do something like this, I'd be happy to learn about that instead. :)
<czajkowski> salgado: ping
<salgado> hi czajkowski
<czajkowski> salgado: ready for G+
<salgado> czajkowski, yep, will be there in a sec. do you have a URL for the hangout?
<czajkowski> salgado: setting up now
<mabac> salgado, quick question for you. :) how would you go about displaying a message when the whiteboard is being edited? it's for alerting the user to not put work items in it.
<mabac> salgado, I've run into a dead end with tal conditionals in the template.
<salgado> mabac, maybe that text widget thing has a way to specify some help text to be shown below it?
<czajkowski> salgado: nice to meet you
<salgado> czajkowski, nice to meet you too, and thanks for the help! :)
<czajkowski> np
<czajkowski> once mrevell has the post done I';ll post it to the loco contacts list, add it to my blog so it ends up on the loco radar and then onto jono
<rick_h_> morning
<mabac> salgado, didn't find anything like that. so I'm going for something simple and dynamic. I got a pointer to the js from danilos so I'll try that
<czajkowski> rick_h_: g'day
<danilos> mabac, btw, if the JS widget is capable of supporting something, do not hesitate to expose it in the lazrjs.py TextAreaEditorWidget either :)
<mabac> danilos, ok let's see what I can find :)
<salgado> mabac, ok, good look with that; notice that after you change anything in a .js file you'll need to run 'make jsbuild' to see them on the browser
<mabac> salgado, ok thanks. I would have missed that
<salgado> that's about everything I know about javascript in LP:  cover your eyes, make changes, run 'make jsbuild' and hope it will work
<rick_h_> :) that'll be easier soon
<mabac> :)
<rick_h_> I've been hacking on tools for that, but a bit complicated to setup atm
<mabac> salgado, I did this but hiding it when the whiteboard is not being edited got tricky: http://dl.dropbox.com/u/400334/whiteboardnotice.png
<salgado> mabac, yeah, we'd need to have hooks with the widget to do that, I think.  I'm checking the docs of YUI's widget class to see if there's anything there that could be useful
<rick_h_> what widget salgado ?
<salgado> rick_h_, the base for InlineEdit: http://yuilibrary.com/yui/docs/widget/
<rick_h_> ok, so widgets have a ton of events to watch for
<rick_h_> looking you *might* be able to watch for the visibleChange attribute?
<rick_h_> you're looking to fire something when you click edit right?
<mabac> rick_h_, yup that'd be good enough
<rick_h_> mabac: I'm just not 100% sure that the visible is following the editor part of the widget or not
<mrevell> salgado, Is launchpad.dev the only place I can play with the work items box?
<rick_h_> mrevell: yea, it's through testing/pqm, but don't think it's on qastaging yet
<rick_h_> so salgado tests passed and such this morning it looks like
<mrevell> cheers
<salgado> mrevell, rick_h_, yes, only launchpad.dev for now.  mabac is trying to add a warning to let people know they should not enter work items on the whiteboard
<mrevell> salgado, Is it not behind a feature flag?
<salgado> mrevell, nope. do you think it's worth doing that?
<mrevell> salgado, If it's going to end up on production before it's ready for use, then yes.
<rick_h_> mrevell: sorry, that's my confusion
<rick_h_> I landed a different branch for salgado yesterday
<rick_h_> the work items is still in reivew/changes
<mrevell> Sorry, I'm confused. So what's landing now?
<salgado> mrevell, it will not land before it's ready
<salgado> mrevell, the new UI is still under review
<mrevell> Gret :)
<StevenK> Can someone who isn't me log into carob and look at the pqm logs to see what the heck the buildbot-poller is doing?
<mrevell> Or rather, great
<rick_h_> StevenK: where do I look to see "what the heck the buildbot-poller is doing?
<StevenK> rick_h_: /srv/pqm-logs or some such, there's a log for it
<StevenK> You may need to sync logs
<rick_h_> StevenK: how often does it poll?
<rick_h_> StevenK: https://pastebin.canonical.com/61452/ is last bit from the buildbot log
<StevenK> I was trying to *not* think about how crap buildbot and its poller is at 2320 on a Friday night. :-(
<StevenK> rick_h_: Sync pqm logs
<rick_h_> sorry, someone an expert on buildbot? /me's never used it before until LP so don't know what I'm looking for
<StevenK> rick_h_: buildbot and buildbot-poller are distinct systems.
<StevenK> And buildbot-poller was written by us.
<rick_h_> so am I looking to sync the buildbot system or a different one?
<rick_h_> StevenK: and am I supposed to be looking at something other than /srv/lp-pqm-logs/pqm_logs/buildout.log then for the poller logs?
<StevenK> rick_h_: That log you pasted is the right one.
<rick_h_> ok, requested buildbot sync, not seeing anything new yet
<StevenK> It will use 'write' to tell you when the sync is completed
<StevenK> IE, you'll notice.
<rick_h_> ah ok
<salgado> StevenK, does this have anything to do with the fact that devel has not been merged into stable yet?
<StevenK> That's one of things the poller does, yes.
<rick_h_> StevenK: so sync completed, nothing new in the log
<salgado> StevenK, hmm, my changes were actually committed to ~launchpad-pqm/launchpad/stable before they were blessed by buildbot, it seems.  I see them committed ~8h ago
<StevenK> No, that's because it merged your revision
<salgado> oh, that's because we pull?
<StevenK> I'm not sure what the poller does, but it's magical.
<salgado> heh, ok
<StevenK> It sends a message to PQM, and it does it.
<salgado> what's the difference between staging and qastaging?
<StevenK> staging runs off db-stable, and qastaging runs off stable
<StevenK> Typical staging is used to test DB changes nowdays
<StevenK> salgado: production-{devel,stable} from your time are long dead, we deploy to production from stable
<rick_h_> Something seems to have kicked, qa bot updated tag/status for my change now
<rick_h_> but log still seems same
<StevenK> rick_h_: Can you kick that manager of yours to do his two items of QA?
<rick_h_> StevenK: will do
<salgado> StevenK, right, makes sense :)
<deryck> adeuring, abentley, rick_h_ -- hey. firing up hangout.
<adeuring> ok
<abentley> deryck: I don't see you.
<deryck> abentley, hey, are you trying?  https://plus.google.com/hangouts/extras/talk.google.com/orange-standup
<czajkowski> sinzui: have you been spring cleaning the licience reviews?
<sinzui> I review the all other/* licenses every Friday. I close projects, give canonical project licenses, send out license renewals
<czajkowski> sinzui: ah ok, as I was waiting on one to change its status and wondered where it had gotten to .
<sinzui> Today I gave out a license and mark a project approved since it changed their license to gpg affero
<sinzui> oops
<sinzui> sorry
<czajkowski> nods that was the one I was wondering about
<czajkowski> as you had a note on the whiteboard
<sinzui> That note was from 2009 I think
<sinzui> czajkowski, You reported a bug about the batching for all /projects...
<czajkowski> sinzui: aye as it came via a RT and it does seem rather strange we'd display a negative figure tbh.
<sinzui> ...only the pathological user with a historical interest will ever use it...
<czajkowski> ok
<sinzui> eg me because I read the license info of 5,000 projects when we decided all projects had to have a license
<czajkowski> heh ok
<abentley> adeuring: chat?
<adeuring> abentley: sure. mumble?
<abentley> adeuring: sure.
<adeuring> abentley: still there?
<mabac> rick_h_, salgado I changed the workitems-widget branch a little so it displays a message when editing the whiteboard. I suspect it might be considered somewhat hackish so if you guys would like to have another look, it'd be great. :) https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790
<salgado> mabac, on a call now, will check in a minute
<mabac> salgado, thanks. going afk now. might be able to check in again later. maybe ping huw if he comes online to have a look too?
<rick_h_> mabac: peeking
<mabac> rick_h_, thanks. if it's to hideous, comment on the mp and I'll deal with it as soon as I can.
<sinzui> amI online?
<rick_h_> yes
<sinzui> thanks
<nigelb> sinzui: No :D
<abentley> adeuring: Internet is back.  Shall we try again?
<adeuring> abentley: sure
<sinzui> nigelb, :)
<nigelb> :D
<mabac> rick_h_, salgado just wanted to check back in. is there a verdict? :)
<rick_h_> mabac: not a huge fan of the hook at the moment. I'm going to propose an alternative. I need to get a checkout and play with it a little bit.
<jelmer> abentley: btw, nice to see ec2 moved
<jelmer> I was wondering if we should put lp-dev-utils in the launchpad PPA
<salgado> mabac, I'll defer to rick_h_ on this one as I know close to nothing about YUI and LP's widgets
<rick_h_> mabac: personally the editor needs to provide an event you can just listen for. It seems it doesn't right now, so I want to add it
<mabac> rick_h_, thanks, that'd be great. any help is very welcome on this.
<rick_h_> mabac: thanks for your patience
<mabac> salgado, it's obvious from my patch that I have no clue. :)
<mabac> rick_h_, no problem. I spent too much time to get that hack working so I'm happy you want to assist me. I'll pop out for the weekend then.
<abentley> jelmer: I guess that gives it a better install story, but I think there's a bunch of packaging needed.
<salgado> mrevell, you didn't forget that text for the help popup, did you? ;)
<mrevell> I did not salgado :)
<salgado> mrevell, if we could get it by Monday it'd be great; no need to hurry too much
<mrevell> I started later on it that I'd have liked but it's under way.
<mrevell> No prob
<salgado> mrevell, great, thanks!
<salgado> hmm. I've got a job on the garbo hourly script and it doesn't seem to be running enough iterations of it for the whole work to be done
<salgado> my job runs for only a couple minutes
<salgado> is there a limit on the time or number of iterations that I'm not seeing?
<salgado> lifeless, around?
<salgado> anybody up for a quick review? https://code.launchpad.net/~salgado/launchpad/workitem-migration-allow-inactive-milestones/+merge/95624
<sinzui> salgado, what happens the work items when I make their active milestone deactivated (untargetable)
<salgado> sinzui, there should be no milestones to which a WI cannot be targeted
<salgado> that's what I'm changing, so that they can be targeted to inactive milestones
 * sinzui looks again
<sinzui> ah
<salgado> sinzui, the UI/DB allows that (I confirmed a few minutes ago); it's just the migrator script that was broken
<sinzui> Right. Lp UI often does not permit it, but the API will let you do it
<sinzui> salgado, r=me
<salgado> sinzui, also, I'm thinking of moving the migrator script from garbo-hourly to garbo-frequent as it takes a couple minutes on the first 2 or 3 runs and then will run much faster than that.  does that sound ok to you?
<sinzui> yes
<salgado> sinzui, do you mind if I sneak that change in this same branch?
<sinzui> no
<salgado> sinzui, great, if you could have another look there  ;)
<salgado> come on, LP, update my diff
<sinzui> still looks okay
<sinzui> I'll just add a comment to the review
<salgado> sinzui, great! can you land it for me as well?
<sinzui> yes
<salgado> sinzui, superb, thanks a bunch!
<rick_h_> salgado-lunch: ping when you get back
<lifeless> salgado-lunch: hi
<salgado> rick_h_, hi
<rick_h_> salgado: howdy, so I replied to mabac's MP
<rick_h_> salgado: I just wanted ot make sure it made sense, and since it seems we're off on TZ, wasn't sure if I should ping/update you on it to help with him
<salgado> rick_h_, I agree with you that it's probably better to do it in specification-index.pt instead of trying to generalize it at this point
<rick_h_> salgado: ok cool. Sorry to make trouble/more work
<rick_h_> salgado: totally not against adding a generate notification to the editor, but want ot be more thorough on it as a longer term thing
<salgado> rick_h_, I completely understand that :)
<rick_h_> cool, so hopefully that diff helps makes the fix simple, that's working and tested from his branch
<rick_h_> just have to undo the other stuff
<salgado> rick_h_, you're giving us code that's simple and documented; it's a huge help!
<salgado> I'm sure mabac will be thankful, and so am I :)
<jcsackett> abentley (or anyone): you know that thing where lp-propose stops using the template from lpreview body? how does one go about fixing that? :-P
<abentley> jcsackett: first, run "bzr plugins -v" to ensure lpreview body is installed, and running from where you expect.
<jcsackett> abentley: confirmed; it's installed and running from my .bazaar/plugins
<abentley> jcsackett: now try running lp-propose and see if the target branch matches "lp:(~launchpad-pqm/)?launchpad(/(db-)?devel)?"
<jcsackett> target is lp:launchpad
<jcsackett> hrm. that's the same branch, but not a mach for that format.
<jcsackett> oh, wait, i may have misread that regex. i think that is a match.
<abentley> jcsackett: Hrm.  I think the displayed target branch is not what it checks against.
<jcsackett> abentley: ah.
<abentley> jcsackett: So the submit branch's public URL should match 'bzr\+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/(db-)?devel'
<jcsackett> abentley: ok.
<jcsackett> abentley: as in the setting in locations.conf?
<abentley> jcsackett: Yes, though bzr info will tell you what bzr thinks the setting is.
<jcsackett> abentley: ok, the submit branch had gotten borked when i mucked with my locations the other day. stuff works now.
<jcsackett> thanks!
<abentley> jcsackett: np
<sinzui> jcsackett, r=me
#launchpad-dev 2012-03-03
<lifeless> wgrant: btw http://wiki.postgresql.org/wiki/Serializable is likely to fuck us over. Excuse the direct language.
<lifeless> wgrant: if we don't already drop our serialization level :)
<wgrant> lifeless: Yeah, I'd seen that. IIRC most of our stuff is READ COMMITTED.
<lifeless> wgrant: btw
<lifeless> wgrant: arrays and indexing, what did you learn.
<lifeless> bigjools: so tell me about these subscription services
<bigjools> lifeless: in a bit, busy
<wgrant> lifeless: For bug access checks, array indexes work but aren't really useful (since most bugs are public it's faster to just read them all and filter inline). For tag searches I haven't really looked yet, but I suspect they won't work well unless the planner can use stats on the columns, which AIUI is only implemented for tsvector so far.
<lifeless> wgrant: thanks
<lifeless> wgrant: I'm thinking subscriptions you see
<lifeless> wgrant: +tag and -tag arrays, f'instance
<wgrant> lifeless: GIN-indexed string or int array columns should work very well for that. http://www.postgresql.org/docs/9.1/static/intarray.html provides some additional support for integer arrays.
<wgrant> But most stuff is doable with GIN natively now.
<lifeless> wgrant: is that in trunk or we'd need the addon ?
<lifeless> wgrant: in core I mean
<wgrant> lifeless: Still contrib, even in 9.1, AFAIK
<lifeless> so, we could map tags to ints via a data table
<lifeless> and ditto queries
<lifeless> and use that
<wgrant> That would probably be optimal, yes.
<lifeless> we've run contrib before
<lifeless> this seems safer than the online defrag contrib module
<lifeless> bigjools: would maas be a reasonable project to cargo cult buildout for django w/system packaged django use ?
<bigjools> lifeless: I *think* so, flacoste just fixed it to do that properly
<lifeless> lp:mass right?
<lifeless> bah
<lifeless> maas. thingy.
<bigjools> I have not tried it myself yet having been mostly on a plane or in a jetlagged blubbery mess
<bigjools> maas, yes :)
<lifeless> indeed
<lifeless> moving + tzs -> nightmare
<wgrant> Jetlagged blubbery messes are the best situations to rework critical infrastructure :)
<bigjools> wheeee
<lifeless> actually there is some research supporting that
<wgrant> Forces you to accidentally think outside the box :)
<lifeless> drunk works too
<wgrant> Because you're delirious.
<bigjools> currently trying to book inspections for rentals next week. Lazy Aussie agents don't work saturday afternoons it seems
<lifeless> http://www.wired.com/wiredscience/2012/02/why-being-sleepy-and-drunk-are-great-for-creativity/
<lifeless> bigjools: ^
<bigjools> wow, I should be one of the most creative people in the world then
<lifeless> right now, yeah, you probably would be
<bigjools> or does it require simultaneous sleepy and drunk?
<lifeless> nah
<bigjools> :)
<lifeless> basically, get the supervisor module turned off -> more options considered when thinking about something
<lifeless> http://tools.ietf.org/id/draft-briscoe-tsvwg-re-ecn-tcp-09.txt is brilliant
<lifeless> last paragraph of section 5.1
<bigjools> !
<lifeless> indeedy
<nigelb> woah, bigjools, you moved?
<nigelb> I like how LP is now having more people in the East.
#launchpad-dev 2012-03-04
<lifeless> ahahha
<lifeless> wgrant: I found 250-odd lines of bugheat you missed :)
 * lifeless claims the LoC win
<lifeless> oh, and a table too. WIN.
<nigelb> lol
 * lifeless strokes his precioouuuus
<nigelb> uhh, wha?
<lifeless> nigelb: spare LoC to write new stuff in
<lifeless> is a precious resource (though really quite easy to make)
<nigelb> heh
<lifeless> speaking of which
<lifeless> you still keen to do that js oops glue ?
<nigelb> yep
<nigelb> my ubuntu laptop is temporarily out of action
<nigelb> the adapter caught fire spontaneously
<nigelb> I've been frustratingly trying to work on a macbook.
<lifeless> meep
<lifeless> Install Ubuntu ? :)
<nigelb> Nah, getting a new adapter on Monday :)
<nigelb> all my open source coding was on the Ubuntu machine. Every single project is on hold :(
<lifeless> bug https://bugs.launchpad.net/launchpad/+bug/946164 FWIW
<_mup_> Bug #946164: bugjob is orphaned code <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/946164 >
<wgrant> lifeless: In my defense, I thought that was dropped 18 months ago.
<lifeless> wgrant: no opprobobium attached
<lifeless> wgrant: it just shows how sticky fat is
<lifeless> wgrant: are there docs on the deletion dance /
<wgrant> lifeless: No, it's more fun this way.
<lifeless> ok
<lifeless> so you get to talk me through it :)
<wgrant> ALTER TABLE bugjob SET SCHEMA todrop;
<wgrant> EOF
<lifeless> FK's ?
<wgrant> They only matter when they're to another table that you're dropping, or to branch, or to person and involved in a unique constraint or otherwise have special person merging code.
<wgrant> None of those seem to apply here.
<lifeless> so the drop happens during the downtime ?
<lifeless> wgrant: ^
<wgrant> lifeless: Yes
<wgrant> upgrade.py drops anything that's in todrop at the end.
<wgrant> It could be done after the downtime if we removed foreign keys first.
<lifeless> wgrant: could you add someting to the DB schema patches policy page or something, noting this ?
<wgrant> I've expanded the dropping bits of https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess
<wgrant> lifeless: ^^
<lifeless> thanks
<lifeless> anyone know what we keep ./database/schema/archive for ?
<lifeless> ww
<lifeless> wow
<lifeless> arora is a terrible browser
<nigelb> lifeless: Aurora? Why? :(
<lifeless> no, arora
<nigelb> ah webkit
<lifeless> apt-cache show arora
<lifeless> grah
<lifeless> whats the fix for
<lifeless>     key = self._config_stack.get('gpg_signing_key')
<lifeless> AttributeError: 'BranchConfig' object has no attribute 'get'
<lifeless> wgrant: can you ec2 land https://code.launchpad.net/~lifeless/launchpad/bugjob/+merge/95773 ? my landing story is still naffed ..
<wgrant> Hmmm, now I wonder if we have someone on the team who was a bzr developer for years ;)
<lifeless> and who is going to bed :P
<lifeless> [and yes, you do - Aaron :)]
<wgrant> Curses.
<lifeless> if it wasn't 23:17 I would probably dig, I'm sure Aaron has fixed this upstream, but it is 23:18 and ESLEEPNEEDED
<wgrant> Give my gettys back, you damn LXC
<lifeless> nn
<wgrant> Night.
<nigelb> heh
<lifeless> StevenK: is MVE meant to be solved at this point ?
<wgrant> I need an SSD.
<wgrant> lifeless: No complaints about your Intel one?
<StevenK> wgrant: Right, clearly I'm missing something, since making your changes in the garbo job the error is "duplicate key value violates unique constraint "accesspolicy_pkey""
<wgrant> StevenK: Check the query log.
#launchpad-dev 2013-02-25
<wgrant> StevenK: Hmm
<wgrant> StevenK: That's not really sufficient
<wgrant> I don't think
<StevenK> Why not?
<StevenK> The access checks will check only branch rows
<wgrant> Doesn't the existing preload thing handle stacking etc.?
<StevenK> It does do stacking as well, yes
<StevenK> But almost all of preloadDataForBMPs isn't needed
<wgrant> You probably need prereqs as well
<StevenK> wgrant: Okay, I agree in terms of stacking, since that will impact the access checks.
<StevenK> wgrant: Why prereqs?
<StevenK> Those will already been pulled in via target_branch == self?
<wgrant> StevenK: Howso?
<wgrant> StevenK: prereq != target
<StevenK> Oh, it may be checked for access
<StevenK> Right
<wgrant> you need access to all three branches, and their stacked-on branches
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/ppa-pub-skip/+merge/150264
<StevenK> wgrant: r=me, with one niggle.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/preload-landing_candidates/+merge/150255 again?
<StevenK> wgrant: And we'll have to go through and delete all of the empty dirs when that fix is deployed?
<wgrant> StevenK: Yes
<StevenK> It would have create pool and dists and everything?
<StevenK> I'm just wondering how much of the tree it did create off the bat
<wgrant> Yes, it created pool/ and dists/
<wgrant> Anyway
<wgrant> You've checked that this change actually works?
<wgrant> accessing source_branchID doesn't call into the security proxy?
<StevenK> It does not seem to
<wgrant> Great.
<wgrant> Oh
<wgrant> Because you added them to zope.Public?
<wgrant> Yes
<wgrant> That's one bad bad security declaration
<StevenK> source_branch and prerequisite_branch were already there
<StevenK> But yes, IBranchMergeProposal needs fixing
<wgrant> Yes
<wgrant> Your changes aren't worse
<wgrant> BUt what's there today is awful.
<wgrant> r=me, then
<StevenK> Shall I rip that crap out now in a seperate branch?
<wgrant> Fixing it may be challenging, but is necessary
<wgrant> So yeah, a followup would be good
<StevenK> IBranchMergeProposalPublic etc etc
<StevenK> Like the fun I had for IBug
<wgrant> Not necessarily
<wgrant> But you can if you want
<wgrant> The problem is not that the attributes are listed in the ZCML
<wgrant> The problem is that they are all public.
<StevenK> Oh, no permission means zope.Public, not launchpad.View?
<wgrant> <allow> means zope.Public
<StevenK> wgrant: You lose at buildbot bingo
<wgrant> Yeah, landing now
<StevenK> I do like one line testfixes
<adeuring> good morning
<czajkowski> jml: you about ?
<cjwatson> wgrant: Any thoughts on my last set of fixes for https://code.launchpad.net/~cjwatson/launchpad/bpph-phase/+merge/144154 ?
<StevenK> wgrant: My QA looks good -- aside from the extra 50 odd queries by something failing to preload people
<wgrant> StevenK: I assume that's the subscriber list
<wgrant> So should be trivial to fix
<wgrant> Care to look into it? :)
<StevenK> I'm reading the traceback from one of the queries
<StevenK> Which looks to hit TAL, then view and then TAL again
<StevenK> Some indication what line is implicated in the template would be awesome
<wgrant> StevenK: Any luck on Branch:+index?
#launchpad-dev 2013-02-26
<StevenK> wgrant: Sorry, been dealing with medication and breakfast
<StevenK> wgrant: I thought branch subscriptions was done via AJAX?
<StevenK> Whereas these queries seem to not be
<wgrant> StevenK: I don't think branch subscriptions are
<wgrant> Just bug subs
<StevenK> The code seems to think they are.
<StevenK> I think it might be reviewers and such for the MPs themselves
<wgrant> StevenK: What suggests that the subscriber list is populated by AJAX?
<wgrant> I can see it in the Branch:+index HTML
<StevenK> Ah
<StevenK> I'm still trying to figure out how that list is calculated so I can preload
<wgrant> StevenK: Let's see
<wgrant> StevenK: BranchPortletSubscribersContent.subscriptions
<StevenK> MismatchError: queries do not match: 0 != 105
<StevenK> But I'm being mean with 50 subscribers
<wgrant> StevenK: r=me
<wgrant> Do consider testing on DF to see if there are more things you can fix
<StevenK> The query log looked pretty good on qas
<wgrant> StevenK: Indeed
<wgrant> I have to wonder what those productseries queries are for, though
<StevenK> But I can spend an eon updating DF if you wish
<wgrant> Maybe for rendering a branch link
<wgrant> Well, four branch links
<wgrant> Seems like you can just land it :)
<StevenK> wgrant, wallyworld_: https://www.facebook.com/photo.php?v=236170989852903
<StevenK> Will probably require flash
<wallyworld_> what is the friendface of which you speak
<StevenK> wallyworld_: Hello, IT ...
<wallyworld_> StevenK: you copping a bucketing down there too?
<StevenK> Nope
<wallyworld_> everything here is so soggy
<wallyworld_> rain, rain fuck off
<StevenK> It hasn't rained since it bucketed down on Saturday
<wgrant> It hasn't rained here since 10 minutes ago!
<wallyworld_> and it hasn't snowed since 5
<StevenK> wgrant: It's Melbourne. Duh.
<wgrant> It actually hadn't rained in maybe a week or two until last night
<StevenK> Looks like it's going to start raining tomorrow and not stop for a while.
<StevenK> If the BOM is anything to go by
<wallyworld_> that would be our weather moving south i guess
<StevenK> You can keep it for a few days
<StevenK> wgrant: Hm, there's some more repeated person lookups :-(
<wgrant> StevenK: :(
<wgrant> What's the TB?
<StevenK> wgrant: http://pastebin.ubuntu.com/5566852/
<wgrant> Ah, helpful
<wgrant> StevenK: Might be best to find the name of the person with that ID, and then see how they're shown on the page.
<StevenK> That's a good idea
<StevenK> First one is the registrant
 * StevenK picks someone else
<StevenK> And they don't appear on the page
<StevenK> They're subscribers, but they should cached from my preloading
 * cjwatson loses buildbot bingo
<czajkowski> jtv: you're up lat
<czajkowski> e
<jtv> czajkowski: not that late, really
<czajkowski> no ?
<jtv> czajkowski: late for work, not for being up
<czajkowski> ah yes that is true
<cjwatson> wgrant: Damn, bpph-phase failed QA.  https://code.launchpad.net/~cjwatson/launchpad/bpph-phase-2/+merge/150690 should repair matters ...
<cjwatson> The web UI bits look fine though, e.g. https://dogfood.launchpad.net/ubuntu/precise/i386/base-files
<wgrant> cjwatson: Ah. Is there a reasonable test?
#launchpad-dev 2013-02-27
<cjwatson> wgrant: Well, I flipped the order round in the test as well to match; not sure I can really do better automatically since they're all just passed through as strings
<wgrant> cjwatson: We don't have some kind of test for the output?
<cjwatson> Let me see if I can spot why the tests weren't spotting the "base-files/admin" garbage
<wgrant> Yeah
<wgrant> I would have expected an integration test of some kind to catch that.
<cjwatson> wgrant: I've pushed an integration test now.  Thanks for the suggestion.
<wgrant> cjwatson: That looks safer, thanks.
<wgrant> r=me
<cjwatson> Thanks.  I'll re-QA in the morning.
<cjwatson> (Also ugh test_ftparchive.py is horrible.)
<wgrant> Yes, yes it is
<wgrant> StevenK: There's still an awful lot of person queries on qas
<StevenK> :-(
<StevenK> Yes, I was trying to work it out last night, and didn't get very far
<StevenK> And I've been distracted by IBranchMergeProposal being obstreperous today.
<StevenK>     self.assertEqual('flibble', bmp.superseded_by.description)
<StevenK> AttributeError: 'thread._local' object has no attribute 'interaction'
<StevenK> And I'm using with person_logged_in() :-(
<wgrant> It's apparently not logged in
<StevenK> Why would person_logged_in uh, not?
<StevenK> Without asserting or something
<wgrant> You're not doing anything viewish?
 * StevenK tries to figure why superseded_by now tosses a KeyError in zope.interface.
<StevenK> Oh. Because I can't read.
<StevenK> wgrant: So while I have tests running, whyfor does my preloading not love me on qas? :-(
<StevenK> The preloading did love me locally
<wgrant> StevenK: Have you examined the tracebacks?
<wgrant> You may want to be in a padded room first
<StevenK> I thought I did, have you generated an OOPS?
<wgrant> https://oops.canonical.com/oops.py/?oopsid=OOPS-1c124234d939329af74ae127df8224c7
<StevenK> wgrant: I don't get it. It loops over view/subscriptions which has the preloading?
<wgrant> StevenK: I'm tnot seeing that..
<StevenK> wgrant: lib/lp/code/templates/branch-portlet-subscribers-content.pt and lib/lp/code/browser/branchsubscription.py
<wgrant> StevenK: How'd you get those from the OOPS?
<StevenK> Guessing, since the TAL isn't exactly forthcoming
<wgrant> Which query are you looking at?
<StevenK> WHERE Person.id = 3807340 LIMIT 1 near the bottom
<wgrant> I think you misread the traceback
<StevenK> Oh? I can see TAL is implicated, and IBranch.hasSubscription
<wgrant> Yes
<wgrant> There's no lp.code.browser.branchsubscription in there
<wgrant> Nor view/subscriptions
<StevenK> Clearly it was, since the query count did drop; but I'm having trouble seeing which bit of TAL is causing the query
<wgrant>   File "/srv/qastaging.launchpad.net/qastaging/launchpad-rev-16512/lib/lp/services/webapp/menu.py", line 493, in enable_if_allowed
<wgrant>     link = func(self)
<wgrant>   File "/srv/qastaging.launchpad.net/qastaging/launchpad-rev-16512/lib/lp/code/browser/branch.py", line 333, in subscription
<wgrant>     if self.context.hasSubscription(self.user):
<wgrant>   File "/srv/qastaging.launchpad.net/qastaging/launchpad-rev-16512/lib/lp/code/browser/decorations.py", line 97, in hasSubscription
<wgrant>     if sub.person == user:
<StevenK> Ah
<StevenK> That's terrible
<StevenK> wgrant:  4 files changed, 126 insertions(+), 159 deletions(-)
<StevenK> That moves as much as I could into lp.View for BMP
<wgrant> What's left?
<StevenK> id, registrant, source_branchID, prerequisite_branchID, next_preview_diff_job, private, source_branch, target_branch, prerequisite_branch, votes, getUsersVoteReference
<wgrant> next_preview_diff_job, votes, getUsersVoteReference sound very excessive
<wgrant> Which failures did they cause?
<StevenK> Let me move them back to View
<wgrant> Also, registrant is less bad but I can't see why it's necessary
<StevenK> wgrant: http://pastebin.ubuntu.com/5569759/
<wgrant> StevenK: That's not a hugely useful traceback :)
<wgrant> StevenK: I could tell that code was calling it
<wgrant> The question is what code, and why?
<wgrant> In the case where it fails.
<StevenK> wgrant: http://pastebin.ubuntu.com/5569770/
<wgrant> StevenK: Look at that test
<wgrant> Think about what it's trying to do :)
<StevenK> wgrant: I'm concerned about viewing BranchMergeProposal:+index as anoymous
<wgrant> StevenK: Why?
<wgrant> StevenK: If I can't see the BMP, I can't see BMP:+index
<wgrant> So it's not an issue that not being able to see the BMP prevents me from seeing something that BMP:+index needs to see
<StevenK> Oh, hah
<StevenK> It's a silly test
<wgrant> Yes
<wgrant> The test attempts to assert that a user who can't see the merge proposal can't see reviews requested from private teams
<wgrant> So it's asserting a security hole
<wgrant> It shouldn't be possible to tell whether a team is a reviewer for a proposal that you can't see.
<StevenK> Indeed
<StevenK> wgrant: Right, that's all tests passing with next_preview_diff_job, votes and getUsersVoteReference back in View
<wgrant> StevenK: Lovely. What was the issue with next_preview_diff_job?
<StevenK> A doctest using it directly
<wgrant> Ah
<StevenK> So I sprayed it with rSP
<wgrant> ez
<StevenK> wgrant: I can move registrant, or I can push it up
<wgrant> StevenK: I don't see why registrant would be required (it's not relevant for security), so please move it to launchpad.View and fix the minimal fallout
<wgrant> In general, security adapters should be using rSP so basically nothing but id is public, but the pattern today seems to be to keep security-relevant bits public
<StevenK> source_branch and friends seem to cause oddness if they're not public
<wgrant> Sure, lots of things will want those
<wgrant> They're not completely terrible to have exposed, and there's going to be lots of fallout => probably not worth changing
<wgrant> Registrant isn't completely terrible to have exposed either, but there won't be much fallout => change
<StevenK> Right, and if you can't see either branch, you can't see the MP anyway
<wgrant> If you can get the BMP object somehow you might be able to
<wgrant> That's why we're moving these to launchpad.View in the first place
<StevenK> I was considering that, if you can get the BMP object somehow won't the branch security adapters come into play anyway?
<wgrant> You won't be able to see details about the branches
<wgrant> But that doesn't mean the attributes should be public
<wgrant> It just means it's not worth the fallout to make the View afterwards
<StevenK> Mmmm
<StevenK> Haha
<StevenK> Fallout is one test that wants with person_logged_in(mp.registrant):
<wgrant> Handy
<wgrant> StevenK: Looking
<StevenK> I wasn't fussed, TBH
<wgrant> StevenK: Why are generateIncrementalDiff/revision_end_date/getIncrementalDiffs still named explicitly and in Public?
<StevenK> None of them exist in the interface
<wgrant> StevenK: Still, no reason for them to be public?
<wgrant> Also, that AnyPerson wants to be AnyAllowedPerson
<wgrant> And you'll want to replace the existing duplicatastic AnyAllowedPerson adapters with a single generic one
<wgrant> Where checkAuthenticated delegates to View, and checkUnauthenticated returns False
<StevenK> I think you should put all this on the MP :-)
<wgrant> Shh
<StevenK> wgrant: I thought AnyPerson was okay because we wanted randoms to be able to comment on MPs?
<wgrant> StevenK: Not randoms who can't see the MP in the first place
<wgrant> AnyAllowedPerson is usually View+authenticated
<StevenK> Ah
<StevenK> Right
<wgrant> Whereas AnyPerson is just authenticated
<StevenK> So it used to be make sense in the public world, but does not in the post-public world
<wgrant> Right
<wgrant> AnyPerson doesn't usually make sense for objects that have privacy
<wgrant> (but I'm pretty sure bugs still use it)
<StevenK> wgrant: Haha, cheater.
<wgrant> I try.
 * StevenK goes to shoot things in the face in BioShock 2
<StevenK> Oh, sorry. Set them on fire, and then shoot them in the face.
<czajkowski> aloha
#launchpad-dev 2013-02-28
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/less-public-bmp/+merge/150731
<wgrant> StevenK: Sounds good
<wgrant> Thanks for fixing those security adapters
<wgrant> By deleting them
<StevenK> Heh
<StevenK> wgrant: Trying to render create_initialized_view(branch, '+index', rootsite='code')
<StevenK> results in a large and angry traceback
<StevenK> Due to breadcrumbs, from what I can see
<StevenK> wgrant: http://pastebin.ubuntu.com/5572431/
<wgrant> StevenK: Some views won't render like that
<wgrant> I don't quite recall why
<wgrant> webops: As lp_publish@pepo, could you run `LPCONFIG=ftpmaster-publish scripts/ftpmaster-tools/obsolete-distroseries.py -d ubuntu -s jaunty`?
<StevenK> Hah
<StevenK> wgrant: The pastebin is my success
<wgrant> Oh, wrong chan
<wgrant> StevenK: Right
<wgrant> That's the usual workaround
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/moar-preload-people-branch-index/+merge/150953
<wgrant> StevenK: Why not just make hasSubscription not completely insane?
<wgrant> Hm
<wgrant> Also, is that the right method?
<wgrant> Ah, it is
<wgrant> But I think your change is still insane.
<StevenK> wgrant: http://pastebin.ubuntu.com/5572472/
<wgrant> StevenK: see DecoratedBranch.hasSubscription
<StevenK> Argh
<StevenK> Argh
<StevenK> Hm
<StevenK> I can probably delete that, then
<wgrant> Delete what?
<StevenK> DecoratedBranch.hasSubscription
<wgrant> Not really.
<wgrant> For one, your change to Branch.hasSubscription is wrong
<wgrant> For another, it's not clear that it's valuable.
<wgrant> I'd change your branch to just make DecoratedBranch.hasSubscription not crap
<StevenK> wgrant: http://pastebin.ubuntu.com/5572479/
<wgrant> StevenK: How does that change anything?
<StevenK> self.subscriptions is a cached list, apparently
<wgrant> Sure
<wgrant> But it's exactly the same as what the method was doing before, except slower because there's no short-circuit
<wgrant> So it'll just execute *more* queries.
<StevenK> So I should do the personID check, then?
<wgrant> Right
<wgrant> That's how you would avoid making person queries.
<StevenK> wgrant: http://pastebin.ubuntu.com/5572483/
<wgrant> That looks more sensible.
<StevenK> Tempted to remove the last three lines of the docstring for being wrong and causing more queries
<StevenK> Well, the lines were wrong, they are not any more ...
<StevenK> wgrant: MP updated.
<wgrant> StevenK: r=me
<StevenK> wgrant: And BMPs are very very unhappy on qas
<StevenK> But I suspect that was known
<wgrant> What do you mean?
<StevenK> I tried to look at a few MPs on qas, and they OOPS due to librarian content
<wgrant> Yes
<wgrant> That's always been the case
<wgrant> You have to create new ones
<StevenK> wgrant: What do you think the plan is for bug 736014 ?
<_mup_> Bug #736014: BugTask:+huge-vocabulary timeout <target-picker> <timeout> <vocabulary> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736014 >
<StevenK> We've discussed it a few times, but then Branch and BMP ate my brain.
<wgrant> StevenK: DistributionSourcePackageCache/DistroSeriesPackageCache
<wgrant> (I believe those two parallel class names were designed to have the same acronym just to make me angry, since there's no other reason for one to have Source and one not)
<StevenK> Those two classes make me sad
<wgrant> z,cb 3
<wgrant> bah
<StevenK> Cat?
<wgrant> Accidental Dvorak
<lifeless> so thats what, 'f*ck u' ?
<wgrant>  /win 3, but close enough
<noodles775> Hi! Is anyone up for a small-ish review (190l): https://code.launchpad.net/~michael.nelson/launchpad/include-binary-size/+merge/150831
<czajkowski> wgrant: StevenK what is the story with on call reviewer nowadays ?
<lifeless> on what :P
<lifeless> noodles775: you know buildbot is 30m or something now :)
<czajkowski> lifeless: mornin'
<noodles775> lifeless: no? wow. fwiw, the tests didn't seem to run correctly on my instance: http://paste.ubuntu.com/5572841/ (that's using `time ./bin/test`)
<lifeless> noodles775: http://blog.launchpad.net/performance/parallel-testing-is-live
<lifeless> noodles775: what size instance (machine RAM) ?
<cjwatson> About 40m I think
<noodles775> lifeless: it's an m1.small (2G), nothing else running on it (I just set it up for LP development yesterday)
<cjwatson> I know wgrant said that adding hashes would be "probably overcomplicated", but cf. bug 1088527
<noodles775> Wow - 40mins from 6hrs.
<noodles775> Looking
<_mup_> Bug #1088527: Needlessly asks for exact package name, file size, uploaded version <ca-escalated> <Developer registration portal:New for michael.nelson> <Launchpad itself:In Progress by michael.nelson> < https://launchpad.net/bugs/1088527 >
<cjwatson> Err, sorry
<cjwatson> cf. bug 1007195
<cjwatson> is what I meant to say
<_mup_> Bug #1007195: Librarian-backed HostedFile objects do not expose SHA-1 hash <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1007195 >
<cjwatson> (Doesn't specifically have to be the SHA-1 of course; I think that was just what the librarian keyed off at the time or something
<cjwatson> )
<cjwatson> Anyway, don't let me get in the way by expanding the scope of your work, it was just in case it was trivial to allow for it
<noodles775> cjwatson: sure - it should be trivial. So s/include_sizes/include_meta/ and add the the sha1 to the dict?
<cjwatson> I guess something like that
<lifeless> noodles775: sounds like there is a leak in there somewere, I'd use a bigger instance
<lifeless> noodles775: lp's cloud test regime uses x1.xlarge IIRC
<noodles775> lifeless: will do next time - I just fired this one up for this bug and normaly only use small for dev work. Thanks.
<noodles775> lifeless, cjwatson: updated with your suggestions (except for HasLength). Thanks!
<lifeless> noodles775: why not HasLength?
<noodles775> lifeless: I commented on the MP, but http://paste.ubuntu.com/5572967/ (and I haven't updated lp's deps) (or am I missing something obvious?)
<lifeless> 0.9.14 is quite old
<lifeless> 0.9.29.is current, and 30 is about to come out
<noodles775> Nice
<lifeless> so should be as simple as dropping a dep in launchpad-dependencies and updating versions.txt
<lifeless> gnight!
<noodles775> G'night :)
<lifeless> I can't see any incompat warnings in those releases
<lifeless> there's a bunch of shiny improvements though, new matchers and more
<lifeless> right, -> sleep.
<noodles775> Would someone be able to send this branch to land for me? I've done updates after comments from RobC and ColinW: https://code.launchpad.net/~michael.nelson/launchpad/include-binary-size/+merge/150831
<czajkowski> gmb: any idea how we take care of stuff like this now awadays
<noodles775> Np, I can wait for one of the LP engineers - the branch isn't that urgent :)
#launchpad-dev 2013-03-01
<StevenK> wgrant: Hmmmm, the OOPSes for bug 1135518 make me sad.
<_mup_> Bug #1135518: +merges times out <branches> <fallout> <privacy> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1135518 >
<wgrant> StevenK: Why?
<wgrant> There's nothing of particular note.
<lifeless> presumably that is why
<StevenK> wgrant: Because I think I caused the breakage, and I can't work out what bit would benefit from preloading from the tracebacks.
<wgrant> The top three repeated queries in 161287 are visibility/series/stacking preloading
<wgrant> I don't think it's a regression
<wgrant> It's just an awful page with a fairly easy fix.
<wgrant> And the second OOPS is the same
<wgrant> visibility, series, then stacking
<wgrant> I don't see any reason to believe that it's a regression
<wgrant> Person:+merges had a request with >600 queries last week
<wgrant> So not a regression.
<StevenK> Excellent
<StevenK> I'll fix it anyway
<StevenK> wgrant: I have a failing test for IHasMergeProposals:+merges, and it includes the SELECT 1 FROM () query madness
<wgrant> StevenK: Right.
<StevenK> My first attempt didn't
<wgrant> Oh, why no?
<wgrant> t
<StevenK> Probably not enough privacy sprinkled in
<noodles775> wgrant: Thanks for the review - I've pushed 16527 with your test changes for query counts.
<wgrant> noodles775: Great. Do you still have PQM privs, or do you want someone to land it for you?
<noodles775> wgrant: that would be great (I don't think so).
<wgrant> noodles775: Done
<wgrant> Hopefully buildbot won't hate it
<noodles775> heh, let's see :) Thanks!
<wgrant> np
<cjohnston> could someone maybe explain what https://code.launchpad.net/~vorlon/launchpad/lp.994110/+merge/105078/comments/303500 means please? I don't understand how the MP doesn't fix that problem.
#launchpad-dev 2014-02-26
<jtv> Hi wgrant â translations job I see.  :)
<wgrant> jtv: Ended up on a tangent that wound up in that region, yeah
<jtv> A scion of the commercial l18n world was recently asked "do you know translations.launchpad.net?"  To my surprise, the answer was "of course!"
<jtv> I always thought those worlds were entirely separate.
<wgrant> Interesting.
<jtv> Met some people from a serious l18n conference that's going on right now, and it turns out to make a nice introductory label.
#launchpad-dev 2014-02-27
<wgrant> jtv: Do you happen to recall the purpose of the weird URL generation around POTemplate and POTemplateSubset? https://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/translations/browser/potemplate.py#L745 is the relevant code. POTemplateURL links back to POTemplateSubsetURL, but that then links straight back up to the DistroSeries, constructing the SourcePackage URL manually, rather than linking back up to a ...
<wgrant> ... SourcePackage. I can't see why it doesn't just return the SourcePackage in the inside property.
<wgrant> I suspect there's no good reason for that, and I intend to fix it, but maybe there's something lurking in the depths of your memory.
<wgrant> (there are custom IFacetMenu adapters in place for IPOTemplate and IPOFile to make things look correct despite the weird URL generation, and I discovered this madness when I removed them and tests failed)
<wgrant> OK, that was added almost 9 years ago by SteveA, and I guess someone decided to add hundreds of lines of workarounds in the ensuing years rather than just fixing it..
 * wgrant fixes.
<jtv> wgrant: let me look...
<jtv> That would have been before I got in on the game, and yes there were plenty of times when we didn't dare change things that were done for Reasons Unknown.
<wgrant> Heh
<wgrant> Yeah, it's moved around a bit, so it took a while to annotate it past the apocalypse etc. and work out just how ancient it was.
<wgrant> Thanks for looking
<wgrant> I think I'll just kill it and hope nothing breaks.
<jtv> Yes, be bold.  Far too much better-keep-it in there.
<jtv> I do recall we had some problems with the weird breadcrumb chains for POTemplates and POFiles.
<wgrant> Yeah, I saw the weird overridey bits in my travels yesterday and deleted them, because the defaults should have worked fine
<jtv> Because a POTemplate nests in either a ProductSeries, or a combination of a SourcePackageName and a DistroSeries (there was no direct representation of that in the DB at the time)
<wgrant> But then that broke buildbot because it went POTemplate -> POTemplateSubset -> DistroSeries rather than the expected POTemplate -> SourcePackage -> DistroSeries
<wgrant> Right
<jtv> And then a POFile happens at the crossroads of a POTemplate and a Language, so again there is no good natural breadcrumbs chain.
<wgrant> Yep
#launchpad-dev 2014-02-28
<bigjools> is it my imagination or does http://archive.ubuntu.com/ubuntu/dists/saucy/main/uefi/ not have any checksum files...
<wgrant> bigjools: uefi custom uploads are only there for secure boot signing
<bigjools> I know ...
<wgrant> So for security purposes the checksums aren't necessary
<wgrant> Does MAAS need to reliably sync them?
<bigjools> yes
<wgrant> We could add an index if someone actually has a valid use for it, I suppose.
<bigjools> we verify other boot-related files that are downloaded from there
<bigjools> seems odd that they are not checksummed too
<wgrant> We wanted the initial implementation to be as small as possible while still being functional, in case the format needed changes. If we'd added an index at the start, things would have relied on it and it would have been impossible to fix a mistake later. But now we know that what we have works, so it may be more reasonable to add it.
<bigjools> ok
<bigjools> maas needs to netboot uefi systems so there's a patch going into maas trusty RSN
<wgrant> Can you file a bug? We'll want to discuss it with Foundations.
<bigjools> sure
<bigjools> https://bugs.launchpad.net/launchpad/+bug/1285919
<_mup_> Bug #1285919: uefi archive files don't have signed checksums <Launchpad itself:New> <https://launchpad.net/bugs/1285919>
<wgrant> cjwatson: Did you put much thought into this when you did the initial implementation?
<cjwatson> bigjools: Why does MAAS need to look in that directory at all?  It's mainly there so that we can fetch from it in the grub2-signed source package's build process, which bundles the result into a .deb.
<cjwatson> bigjools: Any code outside of those implementation details should be using the .debs.
<cjwatson> (Which isn't to say we shouldn't establish an index; but you should not be using it in MAAS anyway ...)
<cjwatson> wgrant: ^-
#launchpad-dev 2015-02-23
<thomi> wgrant: when trying to fix the test for the group security issue: I can't add the private_team to the main_team since (I think) the main team's owner isn't a member of the private team. Do I have to make the main_team_owner own the private_team, and then switch the owner to private_team_owner afterwards, or is there a better way?
<thomi> (does that even make sense?)
<wgrant> thomi: You could do it in "with admin_logged_in", or make the super-team owner a member of the sub-team temporarily.
<thomi> wgrant: ahh - I forgot about admin... that's useful
<wgrant> thomi: Can you fix the commit message on that branch?
<thomi> wgrant: done
<thomi> wgrant: I was thinking: it'd be handy to have an automated script that did the re-scan dance - perhaps you already have something like that?
<wgrant> No. Automating it reduces the motivation to fix it properly :)
<wgrant> And it is fixable properly now.
<thomi> wgrant: with the new db hardware?
<wgrant> Yeah
<thomi> I understand we're testing it in the slave pool, before switching it to db master?
<wgrant> Still some teething issues, but everything is performing much better now.
<wgrant> The old DB servers are dead to us.
<wgrant> One new server is now the master, the other is the slave.
<thomi> nice
<thomi> so... why does this still fail? Is it just a matter to tweaking the db performance?
<wgrant> Some queries aren't performing quite as well on the new servers, and need analysis. And the two queries in question could use rewrites anyway.
<wgrant> (the schema also needs fixing in the post-git world, but that's a fair bit more work and we should be able to make the existing one mostly work again)
<thomi> interesting
<thomi> I'm surprised that certain queries perform _worse_ on newer hardware
<wgrant> It's very different hardware.
<wgrant> Completely different IO performance.
<thomi> curious
<wgrant> wildcherry (the old master) had like 24 disks. aurora has 4 disks, but SSD caching with bcache.
<thomi> ahhh
<wgrant> Currently we're running in writethrough mode, which means that big random writes can easily saturate.
<thomi> what's the db query that makes diff generation slow / flaky?
<wgrant> (same with random reads, though most tables are cached on the SSDs... except branchrevision, the 500GB table that this query touches)
<wgrant> It's not the diff generation itself.
<wgrant> It's the step before: the branch scanner.
<wgrant> It reads the branch's full history from bzrlib, then creates a row in the BranchRevision table for each revision.
<wgrant> That's a lot of rows.
<thomi> O.0
<thomi> for what purpose? there can't be that many places where we need that information, surely?
<wgrant> It's used in just a few places: to show the recent revisions on Branch:+index, to show the unmerged revisions on BranchMergeProposal:+index, and to detect when a branch has been merged into the other.
<wgrant> The last use case is the one that's difficult to implement without something roughly like this table.
<thomi> ahhh.. yeah
<thomi> hmmmm
<wgrant> I have a PoC which stores it more sensibly.
<wgrant> Because, funnily enough, thousands of slight variants of an append-only DAG have some redundancy
<thomi> heh
<wgrant> There's about 4 billion rows in that table now.
<wgrant> thomi: https://lpstats.canonical.com/graphs/AppServer5XXsLpnet/20150125/20150224/
<wgrant> That's the webapp's OOPS/timeout rate. You can probably see where we tested the new servers as slaves for a week or so, then when we promoted one to the master, then when I fixed one remaining timeout.
<thomi> nice
<thomi> what happened late on the 16th?
<wgrant> That's the teething issue.
<thomi> ahh
<wgrant> We still don't know what caused it, but something was causing 200Mbps of replication traffic and maxing out the write IO on both.
<wgrant> Oh, no, that was the spike on the 20th.
<wgrant> The 16th/17th was just issues with wildcherry being crap. It probably realised we were about to shoot it in the head, so it panicked.
<thomi> I realise now it was actually late on the 17th,
<blr> Dasiy daisy...
<thomi> haha
<thomi> there's all sorts of graphs in here :D
<wgrant> Hmm
<wgrant> My chances of having power all afternoon are looking slim
<wgrant> Nasty storm 20km away and power already flickering..
<thomi> either your storms are worse than ours, or your infrastructure is worse (or both)
<thomi> wgrant: does this look correct to you? (minus the dumb spelling mistake): http://bazaar.launchpad.net/~thomir/launchpad/devel-fix-group-security/view/head:/lib/lp/app/tests/test_security.py#L223
<wgrant> thomi: That looks correct to me. Does it correctly fail?
<wgrant> (and does it stop failing if you remove the retraction?)
<thomi> wgrant: no - it fails in the first check - something about 'if self.obj.is_team and user.inTeam(self.obj.teamowner)'
<thomi> just trying to get my head around what 'self.obj' and 'user' are, in that context
<thomi> we do "self.forwardCheckAuthenticated(user, self.obj, 'launchpad.View')" which understandably fails, since main_team_owner isn't a member of the subsidiary private team
<wgrant> thomi: Which line fails?
<wgrant> Ah, you could well be giving the Authorization adapter a security-proxied object.
<thomi> wgrant: line 949 in security.py raises an exception when trying to access self.obj.teamowner
<wgrant> The adapters in lp.security normally get naked objects, as they're the things that check whether non-naked objects can be accessed.
<thomi> ugh - I have to use that... ummm... thing
<thomi> removeSecurityProxy
<wgrant> Try importing zope.security.proxy.removeSecurityProxy and doing 'checker = PublicOrPrivateTeamsExistence(removeSecurityProxy(private_team))' instead
<wgrant> Ah, in fact, the other tests already do that.
<thomi> wgrant: so, with that in place, I now can't get it to fail, I think because IPerson.super_teams doesn't take into consideration the TeamParticipation status - so security.py ln 1038 succeeds.
<thomi> that seems like a bug though, surely calling person.super_teams should only return approved teams?
<wgrant> thomi: One point of clarification: TeamParticipation is the transitive closure of APPROVED and ADMIN TeamMemberships.
<thomi> ahh, ok
<thomi> in that case, something else is happening :D
<wgrant> And super_teams isn't even cached.
<thomi> hmm, ok, so my memberhip retraction isn't working it seems
<wgrant> That's not helpful.
<thomi> :D
<wgrant> Oh, you're doing it in the wrong direction.
<wgrant> member.retractTeamMembership(team), oddly enough.
<thomi> O.0
<thomi> I even read the docstring
<wgrant> I thought there was a more sensible method elsewhere, but I can't see it.
<thomi> lol
<wgrant> I guess it's that way to make the permissions easier.
<wgrant> Since a person should be able to remove themselves from any team.
<wgrant> So you just protect person.retractTeamMembership with launchpad.Edit on the Person, and pretend you didn't just give it the most confusing name ever.
<thomi> haha
<thomi> ok, that now passes without the retraction, and fails with it, so that's good :D
<wgrant> Excellent.
<wgrant> The test is half the battle.
<wgrant> Well, probably more than half in this case.
<thomi> yeah
<thomi> now I hope that the solution we discussed earlier actually works :D
<thomi> it seems to :D
<thomi> wgrant: should I also fix this for TeamMembershipStatus.EXPIRED ? Seems like it's going to cause the same problem
<wgrant> thomi: Yep.
<wgrant> thomi: A team admin should probably be able to see it if there's any TeamMembership at all.
<thomi> wgrant: was Person.setMembershipData the convenience method you were thinking of earlier?
<wgrant> No.
<wgrant> I think it may have been a figment of my imagination.
<wgrant> Though that looks like it does the right thing.
<thomi> wgrant: thinking about the manual QA for https://bugs.launchpad.net/launchpad/+bug/1423428   - after reading https://dev.launchpad.net/Soyuz/QA it seems like I can't dput a src package that I know will fail to build... any advice?
<mup> Bug #1423428: bad english when retrying a failed build <confusing-ui> <qa-needstesting> <trivial> <ui> <Launchpad itself:Fix Committed by thomir> <https://launchpad.net/bugs/1423428>
<wgrant> thomi: Ubuntu has lots of failed builds, though you probably don't have privs to retry them. If you don't have any failures in your PPA, I can test one.
<wgrant> thomi: Can you link the bug to the branch?
<wgrant> For devel-fix-group-security, I mean.
<thomi> hmmm, I didn't already? sure
<thomi> wgrant: done.
<thomi> I'll do the manual QA for both those tomorrow first thing
<wgrant> thomi: Yep, we have nothing urgent to deploy, so tomorrow's fine. Thanks.
 * wgrant offline for a bit for router recabling.
<sil2100> Hey! I wanted to use the launchpad-api to get the source-package name for a selected binary package
<sil2100> I tried using binary_package_publishing_history, but it doesn't have any information about its parent source package
<sil2100> Is there any way of doing that? Earlier I had to be querying packages.ubuntu.com instead, but that's not an entirely clean solution
<sil2100> I know it's relatively easy the other way around, since source_package_publishing_history has getPublishedBinaries...
<cjwatson> I don't think there's a sensible way currently, unless you've gone via source_package_publishing_history.getPublishedBinaries() to start with.  We could perhaps expose BPPH.binarypackagerelease.build._getLatestPublication() or something like that as a new attribute ...
<thomi> hey wgrant
<wgrant> thomi: Morning. Just saw the Fix Committed -> In Progress... does the bugfix not work?
<thomi> wgrant: crap - that was unintentional. stale page on my end I guess
<thomi> wgrant: but I'm having difficulty doing the QA - how can I get my user on qastaging to get in a few more groups? is there an admin account, or..?
<thomi> I think I'm missing persmissions, so I don't see a 'retry build' link on https://qastaging.launchpad.net/ubuntu/+source/cupsys/1.2.2-0ubuntu0.6.06.4/+build/435478 for exampl
<thomi> e
<wgrant> thomi: Do you have a failed build in a PPA that you have access to, perhaps?
<thomi> wgrant: I just checked, it seems I don't
<thomi> at least, none of my personal ones
<thomi> I can start going though all the teams I'm in I guess
<wgrant> Damn you and your former QA team apparently living up to its name.
<thomi> wgrant: I found one :D
<thomi> ...and it works
<thomi> nice
<wgrant> :)
<wgrant> I would be pretty worried if it didn't :P
<thomi> yeah
<thomi> now... the other bug will be harder to setup
<wgrant> Quite. You'll may want to create a second account, or get someone else to assist you.
<thomi> yeah
<thomi> ..and figure out how to manually expire membership I guess
<thomi> "The name 'private-master' has been blocked by the Launchpad administrators." - damn those launchpad administrators!
<thomi> wgrant: I created a second account, but it seems the second account can't create private teams? I guess I need to be in ~canonical or something to do that?
<wgrant> thomi: You need to own a project with a commercial subscription, yeah. What you can do is create the team with your real account, then change the owner to the new account.
<thomi> ahhh, cunning
<wgrant> Or I can add your new account to some team.
<thomi> probably useful for the future - account name is thomi-r (or... is the db reset regularly? if so, then it won't be useful for the future I guess)
<wgrant> Yep, qastaging and staging have their DBs replaced regularly.
<wgrant> thomi: Try now.
<thomi> ok, I'll do it the other way then
<thomi> nice, thanks
<thomi> wgrant: ok, that seems to work. Only thing I'm worried about is that I can't set this up without the main_team_owner being a deactivated member of the private sub-team (since you need to be a member of the private sub-team in order to be able to add it to the main team).
<thomi> I wonder if that's invalidated the test
<wgrant> thomi: What if you create both teams with one account and link them together, then revoke the subteam's membership, add the other user as an admin of the superteam, then see how it works as the other user?
<thomi> ahhh, yeah, that might work
 * thomi trashes the qa database some more
<thomi> that works too - setting to qa-ok
<wgrant> Excellent, thanks.
#launchpad-dev 2015-02-24
<bigjools> wgrant: I am having to use Gerrit. It has made me *really* appreciate LP's code review pages, believe it or not.
<StevenK> Gerrit isn't that bad.
<StevenK> But yes, LP's BMPs are better, when the branchscanner isn't being terrible.
<bigjools> StevenK: Gerrit seems to be doing what Git ought to be doing itself
<wgrant> bigjools: I've not properly used Gerrit myself, but its featureset was inspiration for some of the particular recent enhancements.
<bigjools> wgrant: yeah I can see the parallels
<lifeless> bigjools: what gerrit are you using? They're all slightly different
<bigjools> lifeless: 2.9.3
<lifeless> bigjools: I meant which site :)
<bigjools> lifeless: it's a private instance
<lifeless> bigjools: ah :)
#launchpad-dev 2015-02-25
<cjwatson> wgrant: "We only need to keep domain-specific methods on the Registry objects for API purposes" - do we even need that if we have a git_service or something like that that is itself exposed on the API, like sharing_service?
<cjwatson> Creating all the infrastructure for a new service to avoid domain-specific registry methods only to add domain-specific shims to registry doesn't seem to gain an awful lot, but it would be more productive if we could avoid the shims altogether.
<wgrant> cjwatson: Right, I mean more for historical reasons.
<cjwatson> Oh, you mean the existing methods?  Right.
<wgrant> If you don't hate the sharing_service style of exposing a set of related operations elsewhere, by all means do so.
<wgrant> I don't think it's pretty, but I think it's a lot better than having every method for everything ever on Person and Product.
<cjwatson> I'm not wild about it, I think it's potentially hard to find.
<cjwatson> But as you say.
<cjwatson> It might be good if we had a "see also" on apidoc.
<wgrant> Indeed.
<wgrant> Though for git it may be more sensible.
<wgrant> Because the operations are more obviously interrelated.
<wgrant> Git repositories have operations on them, but anything above the level of a git repository is on the single sharing service.
<cjwatson> Yeah.  It'll take me a day or so to reorganise.
<cjwatson> (But that's fine, I'd rather get it into roughly the right shape now.)
<wgrant> Yeah, I think this should end up OK, and shouldn't be too huge a change at this stage.
<wgrant> And if we can get an API that looks nice without being inside Registry, I'll be pretty pleased.
<wgrant> I did fix lazr.restful last year to make Person a bit less unmanageable (adapters can add their own methods), but the apidoc still lists them all together, even if the interfaces and implementation are in a separate class.
<wgrant> s/are/can be/
<wgrant> That was more for splitting existing stuff out without breaking compat, though.
<cjwatson> wgrant: Hm, so I think http://lpbuildbot.canonical.com/builders/lp-devel-precise/builds/346/steps/shell_9/logs/summary is all due to decorating Branch.recipes with preloading.  Would you prefer to allow branchscanner to see codeimport (which I can do in devel rather than db-devel, IIRC) or to pull the decoration up to BranchRecipeListingView?
<cjwatson> Or something else I haven't thought of.
<wgrant> cjwatson: I knew it'd break some DB perms, but that's an impressive set of failures.
<wgrant> I'd just update security.cfg in devel
<cjwatson> Maybe Branch could gain an undecorated equivalent of recipes just for use in bzrsync.
<cjwatson> Since all it needs to do is set recipe.is_stale
<wgrant> That's not a terrible idea, yeah.
<cjwatson> Or even markRecipesStale or similar
<cjwatson> Let's see
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/recipe-timeouts/+merge/250913, if you're still alive?
<cjwatson> All the failing/erroring tests now pass locally.
<wgrant> Hmm.
<wgrant> Not a huge fan, but it'll work and isn't objectively uglier than the other options.
<cjwatson> Because of the extra method?
<wgrant> Right.
<cjwatson> I figured it was slightly better than leaving a potential performance bomb around for other users.
<cjwatson> wgrant: Thinking about it, rather than adding a new service, why don't I just put these operations directly on GitRepositorySet?  I'm already planning to expose that as /git_repositories on the API, so that would be more economical.
<cjwatson> And I already have a getDefaultRepository method there.
<wgrant> True
<wgrant> That works.
<cjwatson> Indeed Product.getDefaultGitRepository calls it!  Not sure what I was thinking in not at least centralising the common code.
<cjwatson> (in set)
<cjwatson> It'll have to do its own permission checking, I suppose.
<wgrant> SharingService has some precedent there.
<wgrant> But some of those would have needed custom privilege checks anyway.
 * wgrant sleeps.
<cjwatson> night
<cjwatson> thanks for the help
<blr> wgrant: cjwatson: starting to think it might be useful to have a pygit2 fixture library of some sort - could provide a dict of the desired repo state, number of refs, branches, remotes, commits etc to build a pygit2.Repository object
<blr> will push all of that into an api test helper, but might be useful to abstract it out further.
<wgrant> Yeah, we'll definitely want something like that.
<thomi> wgrant: first pass SS is https://launchpadlibrarian.net/198750892/Screenshot%20from%202015-02-26%2012%3A09%3A37.png
<thomi> wgrant: I guess the 'add a new address' is why we need new authentication for +editemails?
<wgrant> thomi: right, you need to reauthenticate to change your authentication information.
<thomi> makes sense I guess :D
<wgrant> Oh.
<wgrant> So, I wouldn't break that convention there.
<thomi> ugh - I just noticed I have it in two places in the SS
<thomi> I meant to undo the one in the top line :D
<wgrant> Every other item there is edited by the edit icon next to the title. I was thinking that there could be a "Subscribe to mailing lists" link or something like that.
<wgrant> But making the edit link for email addresses inconsistent with the rest probably isn't a great idea.
<thomi> OK. Do you think it's worth moving mailing list subs to a separate page?
<thomi> then it could be linked form the right-hand box thingy... or beneath the email header
<wgrant> I would split it into a separate page, yeah. You'll need to review internal links to +editemails (eg. subscribe/unsubscribe buttons on team pages), and add a link in the Emails section and on the existing +editemails page.
<thomi> wgrant: ok... I'll take a look
<thomi> I have no clue how to add another page, but... how hard can it be? :D
<wgrant> thomi: Let me know when you have questions.
<wgrant> Yep :)
<thomi> wgrant: should clicking 'create mailing list' on a team when running launchpad.dev actually do something? or maybe the mailman backend is missing?
<wgrant> thomi: initscript-start doesn't start it, but 'make run' should.
<wgrant> IIRC
<thomi> hmmm, that's what I'm using.
<thomi> maybe I'm being impatient
<thomi> I get "This team's mailing list will be available within a few minutes."
<thomi> but I haven't actually waited a few minutes
<wgrant> thomi: Oh, run doesn't include mailman any more, that's right.
<wgrant> Try run_all
<wgrant> You can see what runs mailman in Makefile
<thomi> ahh
<thomi> much better - now I get an oops instead of a blank widget :D
<wgrant> Heh
<thomi> sweet - got a separate page rendered correctly. time for lunch
<wgrant> :)
#launchpad-dev 2015-02-26
<thomi> wgrant: lots of actions on these pages send emails, which cause tracebacks like  http://pastebin.ubuntu.com/10419041/ - is there an easy way to get those emails to be delivered to a local account... or maybe just not sent at all ?
<wgrant> thomi: install a postfix or similar in your container, configured for local deluvery only
<wgrant> the emails are all sent to root@lovalhost in dev mode.
<thomi> wgrant: ahhh, ok
<thomi> wgrant: you recommend postfix? I on;y have experience of exim, and that was *years* ago
<wgrant> well, postfix is whwt i know best, and debconf provides a one-click local-only option
<wgrant> but exim will work fine too
<thomi> nice
<thomi> I remember exim being a PITA
<StevenK> thomi: 3 or 4?
<thomi> StevenK: hmm... it's been long enough that I can't remember. It would have been  ~ 15 years ago, so....
<thomi> maybe 3?
<thomi> wgrant: comments on placement of 'edit mailing list subscription' link? https://bugs.launchpad.net/launchpad/+bug/1425646/+attachment/4327910/+files/Screenshot%20from%202015-02-26%2015%3A11%3A55.png
<mup> Bug #1425646: It's too hard to find the +editemails page. <trivial> <ui> <Launchpad itself:New> <https://launchpad.net/bugs/1425646>
<thomi> not sure if it should go there, or in the global navigation panel on the RHS
<blr> thomi: seems okay, although isn't strictly speaking 'user information'
<thomi> blr: yeah.. OTOH, if it's over on the RHS then it's separated from the other email-related settings
<thomi> so I'm not sure what's more usable... maybe both? no, that seems like a bad idea...
<blr> thomi: would almost be nice to have something analgous to Latest Memberships for lists
<blr> but that's a bit out of scope
<thomi> yeah
<thomi> also, you tend to end up subscribed to a lot of lists.. or at least have a lot of mailing lists in that list
<blr> better there than on the RHS I think
<blr> the wording of the options on the menu could use improvement, it really isn't clear what the difference between "administer" and "administer account" is without loading the views
<blr> doesn't help that the view headings are different from the links!
<blr> adminster => review person, adminster account => review person's account
<thomi> blr: agreed, but ... also not in scope for what I'm looking at right now :D
<blr> just sayin' :)
<thomi> yeah
<thomi> I think at least one of those aren't seen by mere mortals though
<blr> the UI stuff is a big rabbithole
<wgrant> Administer options aren't shown to mortals, yeah
<wgrant> we need tor rrdesign the settings things entirely
<wgrant> there should be a Settings link which has a submenu yo all of the settings
<thomi> updating doctests :(
<wgrant> 1:(
<thomi> hmm, I think we need a link at the top of each of +editemails and +editmailinglists that points to each other
<wgrant> sonewhere on each, yeah
<thomi> wgrant: ever thought of upgrading ./bin/test to produce subunit v2 output? (with an appropriate option, of course)
<thomi> combined with trv, it'd make these results a lot easier to read IMO
<wgrant> thomi: It can do subunit v1, but I think our subunit's too old for v2.
<thomi> wgrant: what's the barrier to upgrading it?
<wgrant> thomi: Mostly not wanting to break things that might depend on its output, eg. buildbot.
<thomi> wgrant: we can upgrade it and have it spit out v1 by default
<wgrant> thomi: That link looks good. Does the page work?
<thomi> wgrant: of course!
<thomi> I'm just adding the links between the two page snow
<thomi> *pages now
<thomi> then will add tests for those links
<wgrant> :)
<wgrant> And the new page doesn't require reauthentication?"
<thomi> nope
<wgrant> Marvellous.
<thomi> yeah, it's pretty sweet :D
<thomi> I'm sure I've done something stupid though
<thomi> hmmm, I'm really not sure where to put these additional links.
<thomi> I'll leave it for today, and come back to it tomorrow.
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/git-defaults/+merge/250474 should be reviewable again, in the unlikely event you're still vertical.
<cjwatson> wgrant: And https://code.launchpad.net/~cjwatson/launchpad/git-lookup/+merge/250628 as well now, phew.  Fairly substantial redesign, ended up being able to throw away most of IGitNamespaceSet.
<thomi> cjwatson: still around perchance?
<cjwatson> thomi: little bit
<thomi> cjwatson: I read your comment on bug 1425646 - I added a screenshot with both links as text links... what do you think?
<thomi> I'd love to get this finished today (as I have to go work on CI next week) - I also added to that bug screenshots of the mailing list & email settings pages
<thomi> so, feedback desired - blr and wgrant too ^^
<mup> Bug #1425646: It's too hard to find the +editemails page. <trivial> <ui> <Launchpad itself:New> <https://launchpad.net/bugs/1425646>
<cjwatson> sure - commented
<cjwatson> (with the caveat that I'm relatively new to doing serious LP UI work)
<thomi> cjwatson: thanks. My eye for resign is... lacking I'll try that out
<cjwatson> yeah me too :)
<cjwatson> what's the equivalent of a tin ear?
<thomi> heh
<thomi> cjwatson: something like https://launchpadlibrarian.net/198833330/colin_suggestion2.png ?
<cjwatson> I think I prefer that personally.  But if others feel that uses too much vertical space then maybe the links can just be spaced out a bit more horizontally.  My main problem was that they were bunched up right against each other, I think.
<cjwatson> Oh, I would put "Change e-mail settings" first since it's directly associated with the list of addresses just above.
<thomi> good catch
<thomi> I won't upload a new SS - I'm already spamming the bug
<thomi> blr: any thoughts on the above?
<blr> thomi: that looks better - I think we're kind of stretching the semantics of <dl> and <dt> there as well, they're not really definitions (arguably should be <Hn>s with the colons removed)
<blr> you'd have to fix the css up if you changed that though
<thomi> blr: and if we fix it in one place, we have to fix it everywhere
<thomi> what is <hn>?
<blr> <h1> <h2> etc
<thomi> oh, right
<blr> I wouldn't worry about it, from the user perspective they're jus tall block elements, not important.
<blr> if we get some design resource it might be worth going back and addressing that kind of thing
<blr> cjwatson: the create-api branch is ready for re-review if you have a moment to look over it.
<cjwatson> Right, K+J just got back from karate so I'm about to be distracted so let's see
<blr> cjwatson: no worries, will have the ref list bit for you soon too.
<cjwatson> oh cool
<cjwatson> I spent all of today rearranging my defaults/lookup branches
<cjwatson> but they feel more sensibly-structured now
<cjwatson> blr: r=me with comments
<blr> cjwatson: thanks colin :)
<blr> cjwatson: the print statements are placeholders until I sort out some proper logging really
<cjwatson> Yeah, but they might as well be good placeholders. :-)
<blr> that is true!
<wgrant> cjwatson: Excellent, a good redesign was what I thought might work well.
<wgrant> Will rereview today.
<blr> wgrant: did you get any feedback from the juju devs yesterday?
<blr> I missed the tail end of it
<wgrant> blr: Juju should apparently be detecting that the instance failed to start and trying the other AZ, but it doesn't wait quite long enough.
<wgrant> https://bugs.launchpad.net/juju-core/+bug/1425808
<mup> Bug #1425808: OpenStack provider doesn't try another AZ if the scheduler fails to find a valid host <juju-core:Triaged> <https://launchpad.net/bugs/1425808>
<blr> wgrant: great
<thomi> wgrant: can you please remind me what to do to get a branch scan completed? I thought it was a script in lp-dev-utils?
<cjwatson> thomi: http://paste.ubuntu.com/10436489/
<cjwatson> it's not in lp-dev-utils
<thomi> ahh
<thomi> can we add that to lp:launchpad ?
<thomi> or ... I guess if the problem is going to go away soon
<thomi> maybe it's not worth it
<wgrant> I just Ctrl+R lp-shell, Ctrl+R resc, then fix the branch name :)
<wgrant> And no, that sort of code doesn't belong in lp:launchpad. lp:lp-dev-utils maybe.
<wgrant> cjwatson: Hm, is git-defaults meant to be half the length it was?
<cjwatson> wgrant: Sounds about right.  Removed a fair bit of cruft, and I applied some mixin refactoring to the tests to let them be a lot less wordy.
<wgrant> Yeah, looks good, just surprising.
<cjwatson> I reckoned it was a good sign.
<cjwatson> git-lookup becomes a bit longer, IIRC, but not too much.
<cjwatson> Oh.  It becomes slightly more longer than git-defaults became shorter.  Oh well.
<wgrant> Heh
<cjwatson> Not directly related, it's just that I tore a bunch of stuff out of git-namespace.
<wgrant> Yeah
<wgrant> git-defaults is muuuuch cleaner now, thanks for reworking it.
<cjwatson> I almost wonder whether I needed to do PersonDSP at all.  But it'll probably still be handy for browser stuff.
<wgrant> I think it may become sensible to make it exist only for browser purposes, yes.
<wgrant> Handing around an owner and a target isn't terribly onerous most of the time.
<wgrant> It is probably easier to excise once this series is landed, though.
<wgrant> Which is why I didn't request its death earlier.
<cjwatson> Yeah.  Ditto the anomalous argument order of get_git_namespace.
<cjwatson> Anyway, family time, night.
<wgrant> Night.
<cjwatson> Thanks again for the review.
<wgrant> Thanks for not telling me to get stuffed when I ask you to basically rewrite it :P
<cjwatson> Heh
<cjwatson> Rough direction helps.  I think my cognitive error had been thinking that all traverse-style methods had to have roughly the same signature, and in particular a single return value.
<wgrant> Which would usually be a good policy, but it makes things so much uglier here that breaking with convention probably works well.
<wgrant> It's a similar problem to series, really. We're trying to handle two dimensions in a single object for no clear reason.
<thomi> I wonder if I could get a review on https://code.launchpad.net/~thomir/launchpad/devel-fix-editemails-link/+merge/251176 please?
<thomi> I'm sure I've done something stupid.
<wgrant> thomi: Sure. Just rereviewing a couple of Colin's big ones, then I'll look at yours.
<wgrant> I doubt it :)
<thomi> lol - we'll see
#launchpad-dev 2015-02-27
<thomi> wgrant: I fixed all your concerns in that branch - care to re-review?
<blr> thomi: more a case of cornice not having any idea how to deal with that case for a resource
<blr> probably need to look at the source
<wgrant> blr: Could you use cornice with object traversal rather than URL dispatch?
<thomi> blr: ahhh, I've never used cornice, so I may not be much help
<blr> wgrant: hmm possibly
<wgrant> You could have a traverser that consumes segments until it hits something that matches a ref or cannot possibly lead to a ref.
<wgrant> We do that in LP in several places.
<blr> sounds like what we need here, will keep digging.
<wgrant> And for variable-length path pieces you have little other choice.
<wgrant> They occur rarely in most types of applications, of course.
<blr> right, this isn't something normal people should need :P
<wgrant> thomi: Looks good, thanks. Can you revise the commit message to describe more accurately that you're splitting it out of +editemails?
<thomi> yup
<thomi> wgrant: how's that?
<wgrant> thomi: It's in buildbot. Thanks.
<thomi> cool - my first visible fix to lp :D
<wgrant> It will be good to be able to change mailing list subs without having to reauthenticate.
 * wgrant lunches.
<thomi> wgrant: crap - I just realised I forgot to link the bugs. Is it too late to do that now?
<wgrant> thomi: yes, but i linked the two you filed earlier :)
<thomi> wgrant: ugh, thanks
<thomi> wgrant: just want to make sure that it's OK to use purge-inbox.py from lp-dev-utils on the staging inbox? (I'll keep 1 weeks worth)
<thomi> I'm taking your silence as a tacit agreement :D
<wgrant> thomi: Yep, I normally prune to two or three days.
<wgrant> If someone expects mail to stay in the staging mailbox, that's their own problem :)
<thomi> yeah - even a week's worth is 34k mails :(
<wgrant> Hm, shouldn't normally be that much. Odd.
<thomi> hmm, no, something's not right
<thomi> I'm purging mails, but offlineimap is still downloading them
<wgrant> purge-inbox.py probably doesn't expunge until the end.
<wgrant> offlineimap likely downloads anything that hasn't been expunged.
<thomi> I'm waiting till purge-inbox has finished
<cjwatson> wgrant: git-lookup, path/segments> this was because it's convenient that way for the callers; the ones that pass an iterable of segments do so because they're expecting to have more to traverse afterwards.  (For example my browser Person traversal code uses this.)  Also with a path it's inconvenient to indicate what's left, whereas with an iterable of segments we can just consume the used ones.
<wgrant> cjwatson: Right, sounds fine.
<wgrant> Right, just need to sort out the cache maintenance garbo jobs on Monday, and translations should all render fast, except for ddtp-ubuntu which should render in about two seconds.
<cjwatson> wgrant: I'm having some difficulty working out what you mean in your git-finish-sharing review.  The method you're commenting on doesn't handle specs, and gitrepositories_by_id is needed to compute visible_gitrepository_ids.  What are you trying to solve there?
<cjwatson> wgrant: Do you mean something like http://paste.ubuntu.com/10446516/, and relying on the Storm cache to make it efficient that we're walking over the objects twice as often?
<cjwatson> wgrant: And likewise for getVisibleArtifacts?
<wgrant> mrh
<wgrant> does that really not do specs?
<wgrant> sigh
<cjwatson> Apparently not
<wgrant> Though it isn't used except for branches, IIRC.
<wgrant> Indeed.
<cjwatson> Yeah, and git won't need it even with subscriptions because we have no stacking.
<wgrant> s/specs/branches/ in my comment, the rest stands.
<wgrant> Yeah.
<cjwatson> But anyway, presumably the same kind of thing applies to getVisibleArtifacts.
<wgrant> The only "problem" I was trying to solve was that it was hideous and convoluted.
<wgrant> We construct a map because reasons.
<cjwatson> It seems more efficient to construct a map, but maybe that's a meaningless microoptimisation.
<wgrant> uHeh
<wgrant> At the scale this method is used at, the overhead of constructing the dict is probably greater than the penalty of avoiding it.
<cjwatson> OK, I'll see how it all looks without.
<wgrant> (though the old implementation walks through the original list twice anyway, due to the two keys() calls. so it's really very little difference)
<wgrant> The only reasonable optimisation I might consider is making visible_foo_ids a set.
<cjwatson> That's not going to actually try to look up attributes of database objects.  But hopefully the Storm cache will be enough.
<wgrant> But I'd not do that in this case because the benefit is minimal and it would cause a linebreak :)
<wgrant> Oh, well, if the input parameters are Storm resultsets then it could duplicate work, that's true.
<wgrant> But the objects won't be invalidated unless they somehow get evicted from the cache, and even then accessing the PK won't cause a re-query.
<wgrant> Anyway, I'm not fussed either way. My suggestion was just a lot more readable, IMO.
<cjwatson> Looks like they're always real lists, and in the getVisibleArtifacts case they're always one-element lists.
<cjwatson> At least in model code.
<cjwatson> (And browser.)
<cjwatson> I have some time while buildbot churns :-)
<cjwatson> getVisibleArtifacts doesn't even use the values except for bugs, meh.
<cjwatson> OK, simplification pushed now.
<cjwatson> srsly buildbot
<cjwatson> wgrant: I had a stab at your other two code branches, but I don't think I'm remotely qualified to review https://code.launchpad.net/~wgrant/launchpad/bug-736005-trivialise/+merge/250447
<cjwatson> OK, so I think I have the turnip charm deployable locally again and actually running the API service.  I'll work on porting my little lp.code.githosting module over to the new code on Monday, and then it'll be possible to finish off xmlrpc tests and push that next branch.
<wgrant> cjwatson: Thanks. The person most qualified to review that is either a statistician or voodoo priest.
<cjwatson> stub: https://code.launchpad.net/~cjwatson/launchpad/upgrade-requests/+merge/251337 just in case there was some reason you didn't complete that upgrade
<cjwatson> but with that I just managed to push a new repository to a juju deployment of turnip, so back in business there \o/
<cjwatson> just one more small patch to push to the charm for that when I get a moment, and then back to the remaining 2000-ish lines of git-megabranch
#launchpad-dev 2015-02-28
<wgrant> cjwatson: Aha, excellent.
<cjwatson> There we go, https://code.launchpad.net/~canonical-launchpad-branches/charms/trusty/turnip/devel/+activereviews does the job.
#launchpad-dev 2015-03-01
<blr> cjwatson: thanks for those MPs, hope the charm wasn't too painful to work with.
<cjwatson> Tolerable :)
<cjwatson> More juju unfamiliarity than anything else.
<cjwatson> There are one or two other things I'd like to fix up when I get a moment, but it's looking pretty close.  Did you get a stagingstack tenant?
<thomi> hey cjwatson, I don't suppose you know where I can find buildbot logs for launchpad?
<cjwatson> thomi: like http://lpbuildbot.canonical.com/waterfall ?
<thomi> cjwatson: perfect - thanks!
<cjwatson> n
<cjwatson> p
<cjwatson> maximal irc inefficiency, whee
<thomi> heh
<blr> cjwatson: see ops
<thomi> cjwatson: hmmm, I'm really looking for the importfascist logs. I expected them to appear in the test stdout (as they do when running ./bin/test), but I don't see them there
<cjwatson> that I don't know
<cjwatson> the seven (I think) failures there have been there as long as I can remember though
<thomi> yeah, I wont to see if I can fix at least one of them
<thomi> but without a log of them.... and running the tests is slow :D
<cjwatson> They show up even when running just a single test IIRC.
<thomi> plus, I don't know how many tests to run before the message will be triggered
<cjwatson> So just pick something and use -t.
<thomi> ...oh
<thomi> well, that'd be faster
<thomi> hah - they get printed when you Ctrl+c a test run as well :D
<cjwatson> Something in the unit test layer is quick enough, so say bin/test -vvct TestDpkgArchitectureCache.test_any
<wgrant_> thomi: The importfascist isn't a test, as such.
<wgrant> It's just an import wrapper which reports warnings, so any non-subunit bin/test invocation that imports the entire codebase (ie. pretty much anything without -m) will give you them all.
<thomi> wgrant: ahh ok.
<thomi> wgrant: if you upgraded to subunit v2 you could report that even on test results :D
<thomi> actually, that's not strictly true
<wgrant> blr: How'd you fix the routing issue?
<blr> wgrant: oh, it was stupidly easy, a pyramid regex matcher works.
<blr> e.g. {ref:.*}
<wgrant> blr: I guess that's not a problem if there are no segments afterwards, true.
<blr> yep, in that case the ref is always the last var to match
<wgrant> :)
<blr> always nice when things are easier than you anticipate.
<blr> wgrant: thomi was showing me testtool's matchers this morning, it might be nice to have some custom matchers for git objects or ref collections
<wgrant> blr: Yeah, that will make a lot of sense once we start introducing lots of functionality that writes to repos.
<thomi> woooo! there's now only *6* import policy violations :D
<wgrant> thomi: Yay!
<wgrant> You worked out the ZCML etc?
<thomi> wgrant: I think so. Just proposing a merge so you can tell me what I did wrong :D
<wgrant> blr: What's left of the initial API server work?
<thomi> then I'll feel more confident about handling the others
<blr> wgrant: the hairy ones, diff and merge
<blr> err diff and clone
<blr> rather
<wgrant> Yeah, they're not quite obvious. I suspect we'll need to redesign them a few times as their uses become clearer.
<blr> was thinking I would start in on diff
<wgrant> Yeah, diffing two specified commits directly is probably a good place to start. We'll work from there.
<blr> that sounds sensible
<wgrant> cjwatson: python-pygit2 doesn't seem to actually work for me without libgit2-dev and python-dev, and doesn't depend on them. Does that affect Debian too?
<wgrant> oh derp
<wgrant> I was trying to import it while inside a clone of it.
<thomi> wgrant: any chance I could get some feedback on https://code.launchpad.net/~thomir/launchpad/devel-fix-import-violations-specificationworkitem/+merge/251395 please?
<wgrant> thomi: Ah, looked before but it had no diff.
<thomi> wgrant: yeah, I had to poke the branch scanner a few times :(
<wgrant> 'tis Monday :(
<thomi> I thought that'd make it better though
<thomi> Monday == few users?
<wgrant> Monday == 48 hours of no LP branch pushes, so caches get cold.
<wgrant> thomi: Any chance of a quick test of that new method?
<thomi> wgrant: on the *Set class?
<thomi> good idea..
<wgrant> Yep
<blr> wgrant: thanks for the review, will tidy that up.
<wgrant> blr: I hunted around for a built-in pygit2 map of object types to their normal identifiers, but I couldn't find one.
<wgrant> Other than, like, using obj.__class__.__name__.lower()
<blr> wgrant: I know, weird eh, I did the same
<blr> wgrant: was thinking I should send them a pull request for that.
<wgrant> We're gooing to need to do a bit of that, particularly around MPs.
<wgrant> There are features in libgit2 that they've wrapped too deeply.
<blr> yeah, some of the library doesn't strike me as very pythonic.
<wgrant> It's better than it was 18 months ago.
<wgrant> thomi: I'd create say three workitems, two with one milestone and one with another, then unlinkMilestone(milestone_with_two_items) and assert that all three now have the correct state.
<wgrant> 'cause that test is passed by a trivial "UPDATE specificationworkitem SET milestone = NULL" :)
<wgrant> Also, we normally use assertIs with None, though it doesn't matter on CPython.
<thomi> wgrant: good point - changing now...
<wgrant> thomi: Fortunately the factory makes that quite painless.
<thomi> yeah :D
<wgrant> You'll see a lot of older tests do awkward stuff or use sampledata when they could easily use the factory. It's always appropriate to fix those in a driveby if you feel the need.
<thomi> wgrant: do you mind the asserts that ensure that the factory created things correctly? I add them to assure myself that it's doing what I think it is, and I'm never sure if I should remove them afterwards or not...
<thomi> I like the kind of "asserts to demonstrate prior state > ACTION > asserts to demonstrate resulting state" pattern
<wgrant> thomi: No, I usually do that myself.
#launchpad-dev 2016-03-06
<jtv> Could my import branch be failing because of encoding problems in the revision history?  Launchpad oopses every time I try to restart it, and the last log from 2013 says: http://launchpadlibrarian.net/128992530/vcs-imports-libpqxx-trunk.log
<cjwatson> yes; there's a good chance it's irreparably stuffed I'm afraid.
<cjwatson> IIRC bzr-git can't round-trip non-UTF-8 commit messages through bzr and therefore refuses
<cjwatson> https://bugs.launchpad.net/bzr-git/+bug/1489872
<mup> Bug #1489872: Doesn't handle non-utf8 characters <Bazaar Git Plugin:Confirmed> <https://launchpad.net/bugs/1489872>
<jelmer> yes
<jelmer> although that error looks like an old-style svn import
<cjwatson> yeah, I was just getting there myself
<cjwatson> jtv: you could try creating a fresh import of the same branch; I'm not sure that it's possible to convert an existing one
<jelmer> you would indeed have to create a new one
<jtv> Hi cjwatson!  How's life?
<jtv> And hi jelmer  :)
#launchpad-dev 2018-02-26
<cjwatson> wgrant: Could you look over https://code.launchpad.net/~cjwatson/turnip/+git/turnip/+merge/339505 ?  I'd like to upgrade LP and turnip more or less in sync.
<wgrant> cjwatson: Seems reasonable apart from the dep commit message
<cjwatson> Er yes, thanks.
<cjwatson> Also I plan to land https://code.launchpad.net/~cjwatson/lazr.sshserver/fix-bad-key-crash/+merge/339497 shortly, because seriously I should have taken half an hour to debug that like three years ago.
<wgrant> Huh really, is it that simple
<cjwatson> Yeah, I was surprised
<cjwatson> wgrant: Reminded by dealing with spam - https://code.launchpad.net/~cjwatson/launchpad/optimise-spec-search/+merge/338425 would be nice
<cjwatson> ooooops, lazr.sshserver 0.1.[56] are busted
 * cjwatson applies obvious fix (misplaced signature padding)
 * cjwatson lands the various Twisted 16.5.0 upgrades and crosses fingers
#launchpad-dev 2018-02-27
<mwhudson> welp, the snap build webhooks have no indication of architecture?
<wgrant> mwhudson: You'll need to query the API to get that
<mwhudson> ok
<mwhudson> wgrant: is the hook supposed to fire when the store upload is completed?
<wgrant> mwhudson: AFAIK
<wgrant> I'm not hugely familiar with the snapbuild webhooks, but the code looks right and I'm pretty sure BSI depends on that working.
<mwhudson> https://launchpad.net/~canonical-foundations/+snap/subiquity is reporting to https://requestb.in/1mgrebj1?inspect and i'm not seeing store_upload_status get past pending
<wgrant> I can't investigate right now, I'm afraid.
<wgrant> Maybe later this afternoon
<mwhudson> it's not urgent at all, i was just confused
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad-buildd/lxd-hostname/+merge/337127 https://code.launchpad.net/~cjwatson/launchpad-buildd/snapception/+merge/337126 https://code.launchpad.net/~cjwatson/launchpad-buildd/twisted-17.1.0-compat/+merge/340019 would be a good set to get into a buildd release soonish, if possible
<cjwatson> (and https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545, but that's harder ...)
#launchpad-dev 2020-02-24
<tomwardill> sigh, ssl
<ilasc> sigh, Turnip unit tests :)
<tomwardill> at that stage of 'how did it ever work'
<ilasc> :( same here, except I know how.... I commented out unit tests and now: "Ran 246 tests in 27.055s OK", that's how it worked :)
<ilasc> *my* unit tests
<tomwardill> I'm trying to work out how the dev certificate ever worked
<ilasc> hmmm tough one to debug ..
<cjwatson> tomwardill: I thought there was some commentary about how that was put together
<tomwardill> cjwatson: there's a script for it
<cjwatson> Right
<tomwardill> I've successfully altered and used the script
<tomwardill> but this morning, another cert generated using the script doesn't work
<cjwatson> Peculiar
<tomwardill> yeah
<tomwardill> I'm sure I've done something silly somwhere
<SpecialK|Canon> I guarantee I've done something silly somewhere
<SpecialK|Canon> (What were we talking about?)
<cjwatson> Incidentally, I've got the test suite to start up on Python 3.  15000 lines of diff or thereabouts, so I'll be trickling in various MPs over time
<cjwatson> Whole hundreds of tests pass!
<cjwatson> Let's quietly ignore the thousands that don't for now
<ilasc> :)
<ilasc> +1
<SpecialK|Canon> cjwatson: niiiiice
<cjwatson> (And I'm also going to be firmly putting this aside for a bit even though it's a fun toy, because there are a few other things I really need to get done)
<SpecialK|Canon> can you dig^Wpush it in any useful form?
<cjwatson> Not really yet, but certainly an aim
<cjwatson> Well I mean I suppose I can throw the branch up on the understanding that it's all a pile of poo
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+ref/py3
<SpecialK|Canon> git push origin here-be-dragons ;)
<SpecialK|Canon> (fab, thanks :))
<cjwatson> I will not be supporting this in any way shape or form for the moment :)
<cjwatson> And it'll rebase freely
<cjwatson> Oh also it relies on lazr.restful work that isn't pushed anywhere yet
<cjwatson> That's more annoying to casually push because bzr
<SpecialK|Canon> is there any expectation that branches are in any way supported?
<SpecialK|Canon> er
<SpecialK|Canon> that non-"mainline" branches etc. etc.
<cjwatson> Well maybe not supported, but I mean "don't ask me questions about how to get it running"
<SpecialK|Canon> right sure
<cjwatson> My lazr.restful branch is https://code.launchpad.net/~cjwatson/lazr.restful/py3-declarations/+merge/378548 plus https://paste.ubuntu.com/p/nX9zfPPg4q/
<cjwatson> The biggest remaining problem with lazr.restful is that it runs into cgi.FieldStorage bugs on Python 3 and I haven't worked out how to sort those out, especially since cgi.FieldStorage is going away in 3.9 or so
<cjwatson> Makes it hard to get the test suite going properly
<cjwatson> It's no doubt workaroundable somehow, just haven't spent the relevant time on it
<cjwatson> Oh and also https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+ref/py3 for dependencies
<SpecialK|Canon> tomwardill: iirc you asked something about Buildbot versions a while back?
<SpecialK|Canon> tomwardill: I know ~0 about Buildbot but I did just clock that we're running 0.8.5 and latest stable is 2.6.0
<tomwardill> well, that's fun :)
<SpecialK|Canon> (I know long-term we're looking at a variety of things we can/should do there, but part of me wonders about the ROI on even a small version bump?)
<cjwatson> SpecialK|Canon: This is more a function of praseodymium still running precise than anything else
<cjwatson> (We run distro buildbot packages, or backports of them, I forget which)
<cjwatson> We should at least get that upgraded to xenial; praseodymium is a weird and critical system that I have roughly 0 visibility into, which is more or less why we haven't
<SpecialK|Canon> Makes sense, thanks
<cjwatson> SpecialK|Canon: https://portal.admin.canonical.com/C110859 for history of the last major buildbot care-and-feeding work we did
<cjwatson> And I suppose the superseded https://portal.admin.canonical.com/C106558 too
<cjwatson> I think precise â xenial is likely rather less rough than lucid â precise was but I've never tested it ...
<SpecialK|Canon> I can't find sluagh and radande on https://wiki.canonical.com/Launchpad/Instances - are they pining for the fjords?
<cjwatson> That page doesn't really list build/test infrastructure
<SpecialK|Canon> right ok
<cjwatson> And indeed, there's probably more upgrade complication on the worker side
<cjwatson> Due to the pre-release implementation of ephemeral containers that they're currently using
<cjwatson> But hopefully 0.8.12 master and 0.8.5 workers can interoperate so we could decouple that upgrade?  Not sure
<cjwatson> I'm unlikely to be able to look at it for ... a while, but would be happy to braindump what I know on somebody else if need be
<cjwatson> Fortunately buildbot is such that we don't need continuous availability.  If the answer ends up being hold breath, upgrade, fix up on the other side, that may be workable as long as we're reasonably confident we'll be able to put the pieces back together
<SpecialK|Canon> Is there an enumeration of "what buildbot does for us" anywhere?
<SpecialK|Canon> I'm wondering to what extent we can avoid "a careful upgrade from a 2012 version" and just go for "drop it and go with something different, whether that's latest-stable buildbot or a Jenkins or similar"
<SpecialK|Canon> I'm not suggesting we do the latter lightly, or that it's without risk/cost etc.
<cjwatson> We're not wedded to buildbot if something else works.  The main thing is making sure that test runs don't get substantially slower, since they already take a long time; our current setup does careful IIRC 24-way parallelisation spreading tests across multiple containers on beefy-for-the-time machines
<cjwatson> I have no idea how long it would take if we just did a naive implementation on current Jenkins/scalingstack, and those would be useful numbers to have
<SpecialK|Canon> Ah, yes, I'd forgotten that nuance, thanks
<cjwatson> We do have some automation around buildbot runs which deal with merging {master,db-devel} into {stable,db-stable} when their corresponding test runs pass, and merging master into db-devel when tests on master pass
<cjwatson> That would be easy enough to reinvent on top of something else if we needed to
<cjwatson> I don't think there's really anything else worth mentioning
<cjwatson> I'd also say that I don't know of any problems particularly attributable to the old version of buildbot at the moment (we have buildbot unreliability in various ways, but AFAIK those are all due to bugs in the LP test suite, often related to parallel testing).  Do you?
<cjwatson> Of course keeping tools current is a worthy goal in and of itself
<cjwatson> And there are possibly interesting things we could do with less basic tooling.  But just trying to get a sense of why you're asking
<SpecialK|Canon> Oh sorry yes
<SpecialK|Canon> Standard "can we make things easier/better by upgrading/removing this and what's the ROI like?@
<SpecialK|Canon> *"
<SpecialK|Canon> And I was reminded of buildbot now I'm on canonical-launchpad and get emails about failures
<cjwatson> I think I fixed one of the main sources of buildbot unreliability in https://github.com/testing-cabal/txfixtures/pull/13 + https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379456
<SpecialK|Canon> Ah yes I saw that - nice
<SpecialK|Canon> Just to be clear I'm not saying I think buildbot is unreliable, just that the emails prompted me to go looking at it
<cjwatson> The remaining ones seem to be lp.services.job.tests.test_celeryjob.TestRunMissingJobs.test_run_missing_ready_does_not_return_results (e.g. http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/865/steps/shell_9/logs/summary), lp.buildmaster.tests.test_interactor.TestSlave failing in various ways (e.g. ...
<cjwatson> ... http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/864/steps/shell_9/logs/summary, http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/860/steps/shell_9/logs/summary), lp.testing.layers.RabbitMQLayer:setUp timing out (e.g. http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1052/steps/shell_9/logs/summary)
<cjwatson> And to be fair there is one thing that is an issue with buildbot, although it was triggered by upgrading zope.testrunner; buildbot has an old version of subunit that doesn't understand the much more robust v2 protocol, so it sometimes raises exceptions when trying to parse the subunit output from builds, which is v annoying
<cjwatson> All the remaining issues are hard to reproduce locally or I'd have fixed them already ;-)
<cjwatson> But if any of us ever happen to run across one of them locally, and we aren't dealing with some kind of seriously urgent fire at the time, then I'd urge dropping everything else to take the opportunity to figure it out - it will pay off
<SpecialK|Canon> Neat, cheers
<cjwatson> I think upgrading praseodymium to xenial would let us fix the subunit thing.  Probably
<tomwardill> gah, sigh
<tomwardill> still had a py3 virtalenv in my turnip instance
<tomwardill> this did not end well
<tomwardill> or infact, start well
<cjwatson> Would anyone be up for reviewing this small series?  https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/379682 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379683 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379684
<cjwatson> I believe it should significantly improve our deployment speeds
<cjwatson> This is sort of as discussed in the Tuesday weekly meeting a week or two ago: instead of worrying about making sure every deployment target can build wheels, arrange for a full set of wheels to be part of the deployment artifact
<cjwatson> And entirely sidesteps the fact that pip's automatic wheel cache isn't relocatable
<SpecialK|Canon> nice
<cjwatson> Anyone object to me self-approving https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379509 (six-xmlrpclib)?  It's big but basically mechanical
<SpecialK|Canon> cjwatson: no objections - I skim-reviewed just for lightweight value
<cjwatson> Thanks, I'll take that as sufficient and the test suite can find anything I might have missed
<tomwardill> cjwatson: is it possible to login as someone when using 'make iharness'?
<tomwardill> I'm getting a bunch of ForbiddenAttribute errors with pretty much everyting (including DistributionSet.getByName)
<cjwatson> tomwardill: iharness uses execute_zcml_for_scripts, so logging in isn't usually necessary
<tomwardill> hmm
<cjwatson> Can I see your current code?
<tomwardill> cjwatson: https://pastebin.canonical.com/p/7R5HG32NKK/
<cjwatson> Oh, that's a known annoyance of iharness
<cjwatson> If you just let the default REPL try to print things for itself then it wants to get at __dict__ of objects and the proxy tends to forbid that
<cjwatson> But you can assign to variables, or you can do print(foo)
<tomwardill> aha!
<tomwardill> yep, print(ds_set.getByName('ubuntu')) works
<tomwardill> thanks
<cjwatson> Would be nice to work out a way to make that less irritating, but it's easy to work around once you know about it
<cjwatson> Also https://code.launchpad.net/~cjwatson/lpjsmin/tox/+merge/379695 to start bringing another dependency into shape
<cjwatson> And https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379654 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379657 are simple (and much shorter) import rearrangements
<tomwardill> ooo, I've made a recipe
<tomwardill> now to work out how to dispatch that
<tomwardill> cjwatson: +1 to all
<cjwatson> I haven't done the views for that yet IIRC, so that might be mildly tricky
<tomwardill> the death of buildout makes me sad though
<cjwatson> I suppose you can .requestBuild in harness
<tomwardill> yeah, that's why I'm in iharness poking things
<cjwatson> Yeah, buildout was nice in its day
<cjwatson> Even in 2014 I was still going "OK, how the hell do I get this thing to build", though
<cjwatson> Not sure how much that's about buildout and how much about the way our projects use(d) it
<tomwardill> https://pypi.org/project/isotoma.recipe.django/1.2.2/
<tomwardill> I think that was everyone's response to buildout tbh
<cjwatson> Then https://code.launchpad.net/~cjwatson/lpjsmin/py3/+merge/379696 makes it actually work on py3
<cjwatson> Was short enough I didn't bother trying to split it up further
<cjwatson> Yeah, recipes were quite nice when you found/wrote one tht worked
<tomwardill> cjwatson: if I've done requestBuild() do I need to do anythign to process that and dispatch it?
<tomwardill> (I've got LP running via 'make run' and have buildd-manager running)
<cjwatson> tomwardill: transaction.commit()
<cjwatson> tomwardill: But not otherwise; check buildd-manager logs to see if it's complaining about anything, e.g. a missing base image
<tomwardill> ah, that is feasible
 * tomwardill locates logs
<SpecialK|Canon> Was playing with tomwardill's lp-btw (sorry I forgot the name already)
<tomwardill> hmm, nothing in the buildd logs, the build appears to be in the state 'NEEDSBUILD'
<cjwatson> tomwardill: the other trick here is to make sure you have at least one virtualised buildd
<cjwatson> or it won't ever dispatch
<tomwardill> aha!
<tomwardill> that might be it
<cjwatson> you can make it be fake-virtualised via a trick I forget
<SpecialK|Canon> what's with all the relinking stuff we do with the rocketfuel* scripts?
<cjwatson> (i.e. so it doesn't actually try to call ppa-reset)
<tomwardill> 2020-02-24 16:56:09+0000 [QueryProtocol,client] exceptions.TypeError: ('Could not adapt', <OCIRecipeBuild ~name16/ubuntu/+oci/test-oci-project/+recipe/test-oci-recipe/+build/1>, <InterfaceClass lp.buildmaster.interfaces.buildfarmjobbehaviour.IBuildFarmJobBehaviour>)
<tomwardill> looks like I have a bug!
<SpecialK|Canon> yay!
<cjwatson> Ah yes, configure builddmaster.vm_resume_command to /bin/true or something
<cjwatson> SpecialK|Canon: relinking?
<cjwatson> oh that
<cjwatson> SpecialK|Canon: it was particularly good for bzr to avoid having a bazillion copies of dependency checkouts for every branch you ever had, but it's still handy for worktrees
<SpecialK|Canon> naively it looks like it's reimplementing virtualenvs
<cjwatson> no
<cjwatson> it's for sourcecode/, which is not virtualenv
<cjwatson> it's sort of more like codetree
<cjwatson> (in fact I have an unreviewed MP somewhere to make it use codetree ...)
<SpecialK|Canon> I recognise that the README says not to ask about sourcecode/
<SpecialK|Canon> but uh
<cjwatson> but the symlink stuff is to avoid having to have lots of checkouts
<cjwatson> https://paste.ubuntu.com/p/kKftSPNTXM/
<cjwatson> (let's leave aside that my primary worktree links into an old bzr directory ... I should fix that)
<SpecialK|Canon> Don't we have lots of checkouts anyway? Isn't that what rocketfuel-get does just beforehand?
<SpecialK|Canon> right, ok
<cjwatson> To be fair there may well be cruft in rocketfuel-get; I don't use it much
<cjwatson> But I think the idea is that after it pulls it goes around and updates all your worktrees (nÃ©e branches)
<cjwatson> Was more necessary with bzr
<SpecialK|Canon> Actually sorry when you say "more like codetree"
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/373737
<SpecialK|Canon> I thought the primary value of codetree was for cross-language / cross-visibility stuff?
<cjwatson> the relinking stuff isn't like codetree, but sourcecode/ in general is
<SpecialK|Canon> vs. e.g. `pip install --editable`
<SpecialK|Canon> ah ok
<cjwatson> Not particularly; codetree is for cheap-and-cheerful vendoring-by-checkout
<SpecialK|Canon> I very much enjoy the number of deleted lines in that MP!
<cjwatson> With pinned versions
<cjwatson> See https://git.launchpad.net/launchpad/tree/utilities/sourcedeps.conf, which you'll probably notice looks awfully like a codetree input file
<cjwatson> (I think it was originally config-manager, back in the day)
<SpecialK|Canon> https://bazaar.launchpad.net/~codetree-maintainers/codetree/trunk/view/head:/README.md#L14 I think you're right!
<SpecialK|Canon> Oh wait you mean sourcedeps was
<cjwatson> Well, both
<cjwatson> It's possible the pile of stuff in lib/devscripts/ predates config-manager being a separate thing.  I haven't bothered to go back and dig through the history
<cjwatson> But in any case, no point maintaining all of that in-tree IMO
<cjwatson> It is separately true that everything in sourcecode/ (now that mailman is gone) could in principle be moved into the virtualenv, and that this would be an improvement
<cjwatson> There are some details of bzr/brz plugins that I'm not sure about
<cjwatson> So in that sense, yes, this is reimplementing virtualenv and should die
<cjwatson> It's just a matter of working out all the details at this point
<SpecialK|Canon> Nodnod - I hope it's clear I'm not suggesting it's easy/straightforward! Just trying to get a clearer understanding in terms of things I'm more familiar with
<cjwatson> Yeah
<cjwatson> loggerhead is https://bugs.launchpad.net/loggerhead/+bug/383360 / https://bugs.launchpad.net/loggerhead-breezy/+bug/1831661
<mup> Bug #383360: Loggerhead doesn't declare its dependencies in setup.py <easy> <loggerhead:Triaged> <loggerhead-breezy:Fix Released> <https://launchpad.net/bugs/383360>
<mup> Bug #1831661: loggerhead can't reasonably be installed using pip due to self-import in setup.py <loggerhead-breezy:New> <https://launchpad.net/bugs/1831661>
<cjwatson> I'm hoping old_xmlplus and pygettextpo can go away; I have a semi-tested patch to convert the latter to Babel
<cjwatson> For the former, I think I was complaining the other day about how painful it was to parse XML DTDs in Python; that's why
<cjwatson> Some of the others relate to work jelmer has been doing to build more previously-in-plugins functionality directly into breezy
<cjwatson> (I'm explaining in detail here since I hope somebody will be enthusiastic about picking up some individual part of the puzzle ...)
<SpecialK|Canon> Could you explain in detail in a LP bug / Trello card please? I'm enthusiastic but also timeslices...
<cjwatson> https://bugs.launchpad.net/launchpad/+bugs?field.searchtext=sourcecode :)
<SpecialK|Canon> touche ;)
 * SpecialK|Canon needs to sort out his compose key
<tomwardill> ooh, it tried to dispatch a build
<tomwardill> but failed because the builder is missing some configuration
<tomwardill> cjwatson: my buildd manager / buildd appears to have got itself stuck in 'cleaning', despite me restarting everything (I think)
<tomwardill> oh wait
<tomwardill> literally as I type that, it does a thing
<tomwardill> "Scanning 10.104.9.248 failed with: builder_proxy_auth_api_admin_secret is not configured."
<cjwatson> Right, probably in a slightly inappropriate timeout loop or similar
<tomwardill> whereabouts should I put that configuration?
<cjwatson> configs/$LPCONFIG/launchpad-lazr.conf
<tomwardill> oh, it's LP side, not buildd side?
<cjwatson> tomwardill: Yep
<cjwatson> buildd-manager contacts the snap proxy to get a token to dispatch to the builder
<tomwardill> aaah, right
 * tomwardill configures, restarts everything, waits for it to wake up
<cjwatson> It's complex to set up at the moment; if you don't have it there already then you could just unset builder_proxy_host?
<cjwatson> But maybe you have it
<tomwardill> aah, I don't have it, so yeah, that makes most snese
 * tomwardill waits more
#launchpad-dev 2020-02-25
<tomwardill> https://pastebin.canonical.com/p/3pH2jJRN7h/ it dispatched a thing!
<ilasc> yay! nice!!
<tomwardill> build didn't work due to archive domain things, but closer!
<ilasc> :) +1
<wgrant> Ooh, fancy.
<tomwardill> looks like my build is being dispatched with 'archive.launchpad.test', which obviously isn't a thing. The build gets a 403 forbidden. Is there anyway I can override/make that work?
<tomwardill> I tried hacking /etc/hosts on the buildd machine, but that didn't appear to work
<wgrant> tomwardill: That is a thing, but the default apache config restricts the IP addresses that can access it
<tomwardill> aha!
<tomwardill> that would explain the 403
<tomwardill> wgrant: don't suppose you have an example of an apache config that works tehre? I am failing to create one. I always get 403
<wgrant> tomwardill: "Require ip 10.4.0.0/255.255.255.0" (for my LAN subnet), or "Require all granted"
<tomwardill> aha, now I've upgraded to a 404
<tomwardill> do I need to create a mirror or something in there?
<cjwatson> tomwardill: You can also override the publisher config if you like.  /ubuntu/+pubconf IIRC
<cjwatson> Or you can hack override-sources-list on the builder to search/replace stuff, which I sometimes do when I can't be bothered to do anything neater :)
<tomwardill> pubconf did it, thanks
<tomwardill> oooh, it got to the point of trying a git checkout
<tomwardill> I'll call that progress
<wgrant> Yeah, I usually use +pubconf, or if I need to test primary archive publication then I include the real primary archive in Archive:+admin's manual sources.list entries
<wgrant> I don't think I've hacked override-sources-list in a long time
<tomwardill> now to make turnip be on port 443
 * tomwardill stacks the yaks high this morning
<tomwardill> I've changed http://git.launchpad.test:9419/ in launchpad-lazr.conf (in configs/development), but still appear to be getting the old setting. Have I missed another config file?
<tomwardill> hmm, it's changed it in the LP UI, but build is still getting dispatched with the old one
<cjwatson> tomwardill: Did you restart buildd-manager after changing launchpad-lazr.conf?
<tomwardill> ah, maybe not
<cjwatson> tomwardill: But also, what's wrong with 9419?
<cjwatson> Oh, never mind, ignore that
<cjwatson> Confused 9419 with 19417 (the API port) somehow
<tomwardill> 2020-02-25 11:08:55+0000 [-] Iterating with success flag 0 against stage BUILD_OCI
<tomwardill> 2020-02-25 11:08:55+0000 [-] Returning build status: OK
<cjwatson> nice
<ilasc> where do we normally download wheels from ?
<ilasc> I went with piwheels.org but is that recommended practice ?
<wgrant> I'd tend to build them myselr
<wgrant> No particular need to trust random binaries from the Internet
<wgrant> But I don't know what others do.
<tomwardill> piwheels is specifically for the raspberry pi, so probably not a huge help for anything that needs compilation
<cjwatson> It's extremely rare to need to download Python stuff from anything other than pypi.org
<cjwatson> lp-signing uses wheel dependencies at the moment because I was following the snap store approach, but now that I've worked out the mechanisms for building wheels as part of the deployment artifact in LP I'm quite tempted to go back to sdists for that
<cjwatson> I would definitely not touch piwheels.org
<cjwatson> tomwardill: Is there any useful QA you can do on your recipe branches?  https://deployable.ols.canonical.com/project/launchpad
<tomwardill> I don't think easily, without having the UI/API for project
<cjwatson> Well, either do whatever you can or mark it green :)
<tomwardill> fair :)
<tomwardill> hmm, builddmanager is giving me: exceptions.IOError: [Errno 2] No such file or directory: u'/var/tmp/builddmaster/grabbing/20200225-141618-OCIRECIPEBUILD-12/1/ubuntu/manifest.json', but the file does exist on disk. Is there any likely cause, before I start debugging my way through it?
<cjwatson> tomwardill: I can't think of a likely cause.  I'd probably capture an strace
<cjwatson> right, time for a deployment I think
<tomwardill> hmmm, I think I have a callback race type thing
<cjwatson> Ah, that was something that did come to mind in passing, but I figured you'd get it from an strace
<tomwardill> yeah
<tomwardill> just trying to work out the appropriate defer incantations for it
<tomwardill> cjwatson: https://pastebin.canonical.com/p/yCYv9T8xZh/ that's my strace result, which to me looks like it's attempting to read the file before it's finished downloading it (the rename line is after the read line), does that seem reasonable?
<tomwardill> If so, it's a twisted defer thing
<tomwardill> (yay)
<cjwatson> tomwardill: Indeed, probably just a missing yield
 * tomwardill yields all the things
<cjwatson> +        self._slave.getFile(file_hash, file_path)
<cjwatson> But getFile returns a Deferred
<cjwatson> So _fetchIntermediaryFile has to be @defer.inlineCallbacks and has to yield that, and _downloadFiles has to yield calls to it
<cjwatson> Otherwise what happens is that it runs up to the point of creating the Deferred but nothing ever arranges to do the rest of the work only once the Deferred has called back
<cjwatson> In some brave new world we might eventually be able to detect this with type-checking
<cjwatson> (i.e. ignoring the result of something known to return a Deferred should normally be a type error)
<tomwardill> aha, nice
<cjwatson> Could I have reviews of https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379648 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379650, please?  Both broadly the same kind of py3-related change
<tomwardill> cjwatson: +1 to both
 * tomwardill -> EOD -> gym
<cjwatson> Thanks
<cjwatson> wgrant: Hmm.  On staging, 22 POTemplate rows with source_file_format = 3 (XPI), most recent change in 2011; 1 TranslationImportQueueEntry row with either format = XPI or path like '%.xpi', imported in 2011.  Nothing relevant visible in loganberry's rosetta logs (zgrep -i xpi *).  Anywhere else you can think of where I should be looking for evidence of the XPI import code being used?
<wgrant> That's about what I expected.
<wgrant> I forget exactly how TIQEs are pruned
<wgrant> But the POTemplate situation is pretty damning.
<cjwatson> I'll hunt a bit more, but at the moment it does look quite dead
<cjwatson> Ah yes, TranslationImportQueue.cleanUpQueue exists, so lack of TIQEs indeed doesn't necessarily say much
<cjwatson> https://git.launchpad.net/launchpad/tree/lib/lp/translations/interfaces/translationimportqueue.py#n95
<cjwatson> There are some much more recent POFiles attached to those POTemplates though.
<cjwatson> launchpad_staging=> select pofile.id, pofile.path, pofile.datecreated, pofile.date_changed from pofile left join potemplate on pofile.potemplate = potemplate.id where potemplate.source_file_format = 3 order by date_changed desc limit 3;
<cjwatson>    id    |       path        |        datecreated         |        date_changed
<cjwatson> ---------+-------------------+----------------------------+----------------------------
<cjwatson>  3181454 | midbrowser-id.po  | 2020-01-03 03:03:18.547327 | 2020-01-03 08:47:12.812078
<cjwatson>  1296031 | midbrowser-hu.po  | 2010-01-28 03:13:06.945838 | 2017-12-18 12:28:58.183975
<cjwatson>  1670525 | firefox-3.6-ak.po | 2011-01-11 01:03:09.596781 | 2017-05-04 18:50:15.359629
<cjwatson> (3 rows)
<cjwatson> 3066 rows overall
<wgrant> That just means someone's submitted a new translation to them
<cjwatson> Some that actually end in .xpi too
<wgrant> Note what the po names are...
<wgrant> Neither of those packages have existed in a Long Time.
<cjwatson> launchpad_staging=> select pofile.id, pofile.path, pofile.datecreated, pofile.date_changed from pofile left join potemplate on pofile.potemplate = potemplate.id where potemplate.source_file_format = 3 and pofile.path like '%.xpi' order by date_changed desc limit 3;
<cjwatson>    id    |  path  |        datecreated         |        date_changed
<cjwatson> ---------+--------+----------------------------+----------------------------
<cjwatson>  1296371 | ia.xpi | 2010-01-28 20:35:57.113959 | 2017-03-03 17:30:22.902555
<cjwatson>   597359 | oc.xpi | 2008-04-15 19:33:07.436437 | 2016-10-10 07:48:45.302841
<cjwatson>  1295804 | oc.xpi | 2010-01-27 08:31:49.812962 | 2016-10-10 07:48:03.928331
<cjwatson> (3 rows)
<cjwatson> Not obviously all that active, though surprisingly recent
<cjwatson> /ubuntu/+source/firefox last seems to have had translations in natty
<cjwatson> So I'm tempted to add a feature flag to make that importer raise some clearly-named exception, deploy, wait a month or so, and if we don't see that exception anywhere in OOPS reports or poimport logs, nuke it from orbit
<wgrant> Wouldn't anything in the last month either have its TIQE still exist or have updated date_changed one one of those two tables?
<cjwatson> Right, but just in case I missed something
<wgrant> Let's delete it now, and add it back when the ghost of asac gets its revenge.
<wgrant> I really don't see any reason to be cautious here
<cjwatson> *shrug* OK, fair enough
<cjwatson> I guess it's easy enough to revert if needed
<wgrant> Uhuh.
<wgrant> I also said last night it had probably been a decade since it was used
<wgrant> And your data says nine years
<cjwatson> Did you have a specific memory of something being deleted there or was it a good educated guess?
<wgrant> No specific memory, but that was around the time Chromium came into the picture and so Firefox switched to its rapid release schedule.
#launchpad-dev 2020-02-26
<tomwardill> 2020-02-26 10:22:22+0000 [HTTP11ClientProtocol,client] Gathered OCIRECIPEBUILD-19 completely. Moving 20200226-102222-OCIRECIPEBUILD-19 to uploader queue.
<tomwardill> well, there's a thing \o/
<ilasc> yay! nice!!!!
<SpecialK|Canon> niiiiiiiice
<SpecialK|Canon> :D
<cjwatson> Python 3 review requests for today: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379791 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379799 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379861
<cjwatson> tomwardill: Nice.  Let me know when you have everything pushed and I'll review
<tomwardill> cjwatson: ocibuildbehaviour MP updated, and the buildd branch also updated
<cjwatson> Righto
<tomwardill> fairly minor changes to both though
<pappacena> Hello again, eveyrone! :-)
<pappacena> Quite a lot of messages here while I was out!
<ilasc> hey buddy, welcome back :)
<ilasc> yes, we've been very chatty and I actually forgot to reply to the guys yesterday, thanks for all the "source of our wheels" comments yesterday1
<cjwatson> Could I please have a review of https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379901 (Consider metadata_override in LiveFS.requestBuild)?
<cjwatson> It was much simpler once I realised that jsonb = jsonb does the right thing in modern PostgreSQL
<SpecialK|Canon> Nice
<tomwardill> Attempt 1 at a default branch population: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379902
<cjwatson> Looking
<tomwardill> mostly borrowed the code from the target repository widget
<cjwatson> tomwardill: OK, some ideas for improvement in there
<cjwatson> (but a good start)
<cjwatson> pappacena: ugh, you're going to have to revert https-mirrors.  It can't land until the DB patch is on production
<cjwatson> Sorry, I should have said that explicitly
<pappacena> Oh, gosh... I forgot about that!
<cjwatson> Just propose "git revert" of it and you can self-approve that
<cjwatson> Then you can put it back once the DB patch has landed
<cjwatson> I didn't quite notice the email about top-approving in time
 * tomwardill looks
<pappacena> Is our production turnip working fine? It's taking a while to calculate the diff on this git revert...
<cjwatson> It is indeed sad
<cjwatson> I think.  Lots of worker timeouts
<cjwatson> Not much obvious useful debugging messages
<pappacena> uhm... strange...
<cjwatson> But if you look at https://grafana.admin.canonical.com/d/000000044/telegraf-host?orgId=1&from=now-30m&to=now&var-juju_controller=prodstack-45-bootstack-ps45-prodstack-is&var-juju_model=prod-launchpad-git&var-service=All&var-juju_unit=nfs-ganesha%2F3&var-juju_unit=turnip-pack-backend%2F2&var-juju_unit=turnip-pack-backend%2F3&var-host=All&var-mountpoint=All the load average is gradually rising
<cjwatson> Possibly a sad NFS mount
<pappacena> swap usage seems a bit high on one of turnip machines to
<pappacena> *too
<cjwatson> taken to #is internal
<pappacena> Thanks. Anyway, the diff was calculated. I'll take a look and top-approve the revert
<cjwatson> Cheers
<pappacena> "Voting criteria not met"?
<cjwatson> tomwardill: thanks
<cjwatson> pappacena: you need to cast an approve vote as well as top-approving
<cjwatson> i.e. Review -> Approve and Save Comment
<cjwatson> the landing bot looks through the set of votes on the MP and wants them all to be positive
<cjwatson> if you cast an approve vote then that'll claim the review request for launchpad-reviewers
<pappacena> ah, right!
<pappacena> Makes sense. Thanks!
<tomwardill> cjwatson: updated MP
<pappacena> Ok, I'll open a new MP reverting this git revert now. I'm sorry about this mess, folks. I really forgot about the db patch!
<SpecialK|Canon> Just for my understanding - how come the tests passed if the db patch hadn't landed?
<SpecialK|Canon> ...because we run them asynchronously, sorry, yep
<SpecialK|Canon> Was thinking of the wrong system, sorry, carry on
<cjwatson> SpecialK|Canon: they didn't pass :)
<SpecialK|Canon> Oh
<SpecialK|Canon> I mean, yes
<cjwatson> but as you say, post-hoc tests
<pappacena> Yep :-)
<SpecialK|Canon> That's just not a blocker to merge here, right
<cjwatson> as it happens they haven't run yet for reasons
<cjwatson> pappacena: I made the same mistake at least once when I was starting out in LP
<cjwatson> tomwardill: thanks, re-reviewed
<cjwatson> pappacena: I believe you can go ahead and top-approve the DB MP
<pappacena> yes... too many days away from this task, I totally forgot it adds a new column. BTW, I'll top-approve the db-patch MP, ok? (https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/379504)
<pappacena> ah, cool... I was writting it... hehe
<cjwatson> snap, yes
<pappacena> Done! Thanks!
<tomwardill> cjwatson, pappacena: I've updated the default branch MP, is landing it okay, or would it get in the way of the revert/landing, etc going on? Happy to just leave it till tomorrow if it's safer.
<cjwatson> tomwardill: It's fine, go ahead
<tomwardill> landing :)
<pappacena> +1 :)
<tomwardill> next on that list: trying to work out how to make it have the short form of the ref name ('master' vs 'refs/heads/master')
<tomwardill> but that can wait for another day
<cjwatson> Oh, I didn't notice that bit
<cjwatson> tomwardill: I think it would be fine to just manually remove any leading refs/heads/ if it exists.  Or you could extract the logic in GitRefMixin.name into a helper if you wanted; I don't think there's an existing helper
<tomwardill> yeah, I think a helper would be nice there
<tomwardill> I'll have another look at it in the next OCI gap :)
#launchpad-dev 2020-02-27
<wgrant> NOTICE:  identifier "binarypackagereleasedownloadcount__archive__bpr__day__count__idx" will be truncated to "binarypackagereleasedownloadcount__archive__bpr__day__count__id"
<wgrant> so. close.
<wgrant> cjwatson: Index addition to optimise BPRDC queries, if you have a moment: https://code.launchpad.net/~wgrant/launchpad/+git/launchpad/+merge/379942
<tomwardill> wgrant: if you're about, I have a column in the OCI stuff that needs to be NOT NULL, is it safe to add a migration to do that (given we should have 0 rows atm)
<wgrant> (prevents HOT updates, but almost certainly worth it given that updates are async)
<wgrant> tomwardill: Yep
<tomwardill> righto
 * tomwardill does the thing
<cjwatson> wgrant: Is it worth including country as well?  Archive.getPackageDownloadCount uses that
<wgrant> cjwatson: It'd make the index a bit wider, and nobody really cares about the country breakdown, but it's true it wouldn't cost much more at all.
<wgrant> I suppose we might as well.
<wgrant> It's how I'd model it in Cassandra were I not doing it in 2009.
<wgrant> commit 1c040ea5ee863a784fb85b1a6455668f94e0d0be
<wgrant> Author: William Grant <me@williamgrant.id.au>
<wgrant> Date:   Sat Feb 20 15:36:29 2010 +1100
<wgrant>     Add and test Archive.updatePackageDownloadCount.
<wgrant> Just made it ten years
<wgrant> nice
<wgrant> Ten years and close to a billion rows before it needed significant revision.
<SpecialK|Canon> Nice
<wgrant> When we upgrade to PostgreSQL >=11 we can replace this index and binarypackagereleasedownloadcount__archive__binary_package_rele (the UNIQUE constraint) with a new UNIQUE constraint with INCLUDE (count), but for now we can afford the extra 30GB.
<wgrant> Right, that additional column is worth it just because it makes the commit message wrapping less ugly.
<wgrant> cjwatson: Updated.
<wgrant> cjwatson: btw I got some bug update timeouts earlier, but they seemed shorter lived -- and now much easier to debug, as there's a lot less that can be going wrong in the new triggers.
<wgrant> I wonder if we can capture interesting bits of pg_locks regularly
<cjwatson> wgrant: Should it be "count, country" rather than "country, count" so that the query in getDailyDownloadTotals can use the partial index more effectively?
<cjwatson> (Feel free to tell me that this doesn't matter in practice)
<cjwatson> wgrant: Anything interesting in the bug update oopses?
<wgrant> cjwatson: It's important that it be country, count, or the queries that filter on country can't use it effectively. You can only efficiently filter on a prefix -- if you filter on something other than a prefix, you end up having to scan through the intermediate columns.
<wgrant> But if you're not filtering on country then you are scanning through all the tuples anyway, so the fact you can't filter by country isn't problematic.
<cjwatson> wgrant: Right, but doesn't getDailyDownloadTotals filter on only archive, bpr, day, count?
<wgrant> It doesn't filter on count. It selects it.
<cjwatson> Oh no, I see, it's getting count from the index but ... that
<wgrant> https://oops.canonical.com/oops/?oopsid=OOPS-ae3af2c30556a12f4fe8f8acf345a419
<cjwatson> Got it
<wgrant> https://oops.canonical.com/oops/?oopsid=OOPS-f0f9a7577246c3195e93f200cbb32ca1
<cjwatson> wgrant: r=me
<wgrant> cjwatson: And I assume db=you?
<wgrant> Ah yes
<cjwatson> Indeed
<wgrant> Will get it applied hot.
<wgrant> And hopefully we weren't relying too much on HOT
<wgrant> But given the nature of PPAALP, probably not
<cjwatson> So those timeouts rule out the theory (which may have been only spitballing to start with) that it was a lock that prevented the creation of temporary tables
<wgrant> It was only spitballing, though I'm pretty confident we did narrow it down to the bugsummary triggers at one point.
<wgrant> But there are very limited locks that can have caused that timeout. Mostly a row lock on bugtaskflat.
<cjwatson> Presumably the bugsummary triggers, though it's annoying that the pseudo-traceback doesn't say that
<cjwatson> I wondered whether it might be a deadlock with the BugSummaryJournalRollup garbo job somehow, but logs show that that job wasn't running at the time of https://oops.canonical.com/oops/?oopsid=OOPS-ae3af2c30556a12f4fe8f8acf345a419
<wgrant> But unless we're SERIALIZABLE they shouldn't be able to conflict, since the trigger only inserts journal rows.
<wgrant> I think reporting on long locks on bugtaskflat might be interesting.
<cjwatson> Agreed.  How do we organise that?
<cjwatson> Perhaps stub can help
<tomwardill> stand back, twom made a DB patch! https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379960
<wgrant> tomwardill: Wasn't the intent that git_repository_url was to be an alternative?
<wgrant> Like snap.git_repository_url
<cjwatson> I recall deciding YAGNI on that for the moment
<tomwardill> wgrant: I think eventually, yes. But for now, git_repository_url doesn't exist.
<tomwardill> so atm it's possible to create OCIRecipes that make no sense as there's no source
<wgrant> Fair enough, db=me
<tomwardill> so I should be able to Top Approve that, then provide the equivalent code patch?
<wgrant> I don't think there's a particular sequencing constraint between the DB and code patches here, but yes.
<cjwatson> Right, code can land concurrently with that as long as the code does the corresponding checks (which IIRC it does)
<ilasc> Signing Service question: looking at the code / data model I think we have a 1:1 mapping for: Client (one) <-> (one) ClientPublicKey
<ilasc> can I please get a confirmation on that being the case in the real world too, sqlAlchemy requires explicit mapping definition
<cjwatson> ilasc: No.  A client may have multiple public keys.
<cjwatson> It's explicitly not one-to-one, because that would make it impossible to do rotation.
<cjwatson> ilasc: Note the existence of Client.public_keys
<ilasc> cjwatson: excellent, indeed saw that now that u mentioned it, thank you!
<tomwardill> ah, cool. All the code appears to assume that git_repository is required
<tomwardill> although I found a typo in an error message
<cjwatson> tomwardill: Will take a while anyway - Thiago has a patch ahead of you in the DB queue
<tomwardill> no problems, I think there's just the DB patch for that, so it can wait a bit :)
<cjwatson> Let me get a ticket in for that one at least
<wgrant> cjwatson: I think we can consider relaxing the one patch per deployment constraint for obviously fine cases.
<wgrant> We've done it once or twice in the past.
<wgrant> And it's been a long time since we've actually run into deployment trouble.
<cjwatson> This is true
<cjwatson> The documented rule is "not without manual approval" anyway
<cjwatson> So we can just manually approve obviously fine cases :)
<tomwardill> tiny typo fix while I don't forget: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379961
<cjwatson> I updated LPS for Thiago's patch, but I'll just wait until Tom's passes buildbot and gets onto staging, and then update the commit ID before filing the ticket
<wgrant> Like I wouldn't deploy the bugsummary triggers and the BPRDC index at the same time, because they both have unpredictable performance implications.
<cjwatson> tomwardill: r=me
<cjwatson> Yeah
 * tomwardill tries to remember where he left the OCIProject UI patch
<cjwatson> wgrant: Could you comment on the indexing problem in https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/379218 ?
<cjwatson> I'm not sure we've split up {,Archive}SigningKey in the right way.
<cjwatson> Maybe key_type needs to be denormalised onto ArchiveSigningKey?
<wgrant> cjwatson: Hm, are type and purpose somewhat orthogonal?
<wgrant> The restriction is on the purpose, not necessarily the type.
<wgrant> But yes, if we want to say type == purpose then the way to express the constraint would be to denorm SigningKey.type onto ArchiveSigningKey and add a composite FK
<cjwatson> wgrant: Not really usefully orthogonal, because lp-signing needs to know more than just the bare key type in order to implement the right signing logic
<wgrant> True.
<wgrant> I'm thinking of cases like existing OpenPGP archive signing keys, which are used to sign not just indexes but also directly signable customs.
<cjwatson> Composite FK as in ADD CONSTRAINT archivesigningkey__key_type__signing_key__fk FOREIGN KEY (key_type, signing_key) REFERENCES signingkey (key_type, signing_key) right?
<wgrant> Right.
<cjwatson> I'm trying to think if even the OpenPGP case would benefit from type != purpose.
<cjwatson> I can't quite see how it would.
<cjwatson> It would only benefit if we had a use case for multiple different OpenPGP keys per (archive, [series]) that were used for different things.
<wgrant> Right
<pappacena> Can't we create a trigger to check if we are not creating more than one ArchiveSigningKey with the same key type, and prevent the duplication of key_type in ArchivesSigningKey table?
<cjwatson> pappacena: composite FKs are simpler than triggers
<wgrant> We have that for UEFI today, though it happens that the alternate key is only held by Microsoft.
<cjwatson> And denormalisation is permitted
<wgrant> Making the trigger non-racy would also be somewhere between difficult and impossible.
<wgrant> We're likely to need more granularity on things like FIT keys later
<wgrant> But I guess we can do that later.
<cjwatson> I was thinking that that granularity would be more likely to be expressed by extra columns on ArchiveSigningKey (e.g. sourcepackagename, not that I'm enthusiastic about that)
<cjwatson> But perhaps I don't understand what the likely changes are
<cjwatson> At any rate, the composite FK approach doesn't preclude type != purpose later if we need to; we'd add an extra purpose column, relax the index, and maybe add a constraint for the permitted combinations of type and purpose
<cjwatson> s/index/unique constraint/
<wgrant> Yep, fair enough.
<wgrant> sourcepackagename for FIT, arch for UEFI, that sort of thing
<cjwatson> Yeah
<wgrant> The signing service could conceivably have a key type of X.509 with purpose restriction UEFI, then LP ties that to series and arches and packages
<wgrant> Also a key type of OpenPGP with purpose restriction APT_INDEX and DIST_UPGRADER
<pappacena> Just to make sure I understood it correctly, this granularity doesn't exist today, right?
<wgrant> Since currently those are the same key, but there's no actual reason for that.
<wgrant> Right, not needed today. It's only by upload type and series so far.
<cjwatson> For X.509, I prefer to have the signing service have different key types if the signing process requires using a different binary (sbsign vs. kmodsign), even if they're technically the same type of key material.  But otherwise fair enough.
<cjwatson> To me that's slightly more than a restriction of purpose (which is about which keys may validly be selected for use).
<cjwatson> Combination of key type and mechanism, perhaps, but collapsed into key_type
<wgrant> Right, but I'm wondering if in some cases they might be the same key material.
<wgrant> Which we could just duplicate, of course.
<pappacena> I think we could just have several ArchiveSigningKey with different "filters" pointing to the same SigningKey object (on LP's database and lp-signing service)
<pappacena> And having different key_types (meaning "purpose", and not really the key file format), there shouldn't be a problem with the unique constraint.
<cjwatson> Yep, we already have that situation for keys that have currently been manually copied between archives
 * SpecialK|Canon watches Chromium spin
<cjwatson> It's true that we might end up with SigningKeys that have the same material in some cases.  Duplication probably isn't the worst thing there
<wgrant> OK
<cjwatson> tomwardill: http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/874/steps/shell_9/logs/summary   oh dear, that DB patch is going to have trouble until we get some code changes landed on production
<tomwardill> that's... weird
<tomwardill> I ran those tests
<cjwatson> Did you definitely apply the DB patch to the launchpad_ftest_template DB?
<cjwatson> tomwardill: I'm going to go ahead and request that Thiago's DB patch be deployed, since yours seems likely to be blocked for a bit
<tomwardill> hmm, maybe no. I ran ./database/schema/update.py
<tomwardill> cjwatson: sure, go ahead
<tomwardill> I'll replicate and fix
<cjwatson> database/schema/upgrade.py defaults to, err, I'm not sure which but probably launchpad_dev
<cjwatson> It takes a -d option
<tomwardill> yeah. The one time I don't just run `make schema` and try to be all fancy, it bites me :)
<cjwatson> Well, you don't need to do that necessarily
<cjwatson> You can run upgrade.py on all five (launchpad_empty, launchpad_dev_template, launchpad_dev, launchpad_ftest_template, launchpad_ftest_playground) to make sure
<pappacena> I think it defaults to launchpad_dev... I ran it few days ago without the '-d'...
<cjwatson> And usually best to do likewise with security.py
 * tomwardill improves notes
<cjwatson> Having a "make -C database/schema migrate" or similar to do them all might not be the worst thing in the world
<cjwatson> Though with a comment that it's only for development
<tomwardill> aaah, there we go, that's all the errors I was kind of expecting
<tomwardill> Total: 100 tests, 0 failures, 52 errors, 0 skipped in 52.836 seconds.
 * tomwardill coughs
<tomwardill> hmm, why would `self.git_repository = value.repository` trigger the NOT NULL constraint? I'm setting the value, not trying to read it
<tomwardill>  cjwatson: I can't work out why this line: https://git.launchpad.net/launchpad/tree/lib/lp/oci/model/ocirecipe.py#n156 when called via the makeOCIRecipe method in factory.py is causing a storm flush. Any hints?
<cjwatson> tomwardill: Have you tried it with LP_DEBUG_SQL=1 ?
<tomwardill> cjwatson: yeah, it seems to do a select, then an insert: https://pastebin.canonical.com/p/dfSbgR9NmN/
<tomwardill> the insert is missing the git_repository value
<tomwardill> but I've no idea why
<tomwardill> (well, it's doing the insert on the line that the git_repository value is being set, so it's more 'why is it doing this insert' rather than 'why is git_repository missing')
<cjwatson> This sort of thing usually happens when it needs to flush something in order to get an ID
<cjwatson> IIRC in Snap I had to order some things very carefully to avoid this
<cjwatson> Which of the failing tests is this?
<tomwardill> lp.oci.tests.test_ocirecipebuild.TestOCIRecipeBuild.test_queueBuild
<tomwardill> although I think it's the same symptom in most of them
<cjwatson> OK, give me a few minutes to upgrade
<cjwatson> Ah, of course.  Snap doesn't run into this because it's intentionally possible to have a Snap without any source at all, in order to avoid blocking repository deletion on deleting snaps built on it.  Instead, we just "detach" the snap when the source is deleted, leaving it with all of branch, git_repository, and git_repository_url being NULL.
<cjwatson> And there's a check in SnapBuildBehaviour.extraBuildArgs to refuse to dispatch a build of a snap that's in this state.
<tomwardill> aha
<tomwardill> that explains the checks in there (like the git_ref setter allowing it to be None)
<cjwatson> So while we could fix this by more careful ordering in some way (because you need to get hold of stuff from the GitRef row before you populate anything in the new OCIRecipe row), I think it would be better to just revert this DB patch for the same reasons.
<cjwatson> And implement the same detaching logic for OCIRecipe.
 * cjwatson slowly loads stuff back into their brain
<tomwardill> sure, that makes sense
<cjwatson> Sorry for misleading you about this
<cjwatson> You'll need to add a bit of stuff to the machinery around GitRepository.destroySelf as well
<tomwardill> no worries :)
<cjwatson> Look for detachFromGitRepository
<cjwatson> We can just revert the DB patch directly rather than doing anything more clever; because it hasn't passed buildbot, it isn't on staging or anything like that
<cjwatson> Anyone who's applied it locally will have to manually unapply it (DROP NOT NULL and delete the LaunchpadDatabaseRevision row), but that's probably just the two of us
<tomwardill> incoming MP :)
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/379978
<cjwatson> db=me
<tomwardill> landing
<cjwatson> for db in launchpad_empty launchpad_dev_template launchpad_dev launchpad_ftest_template launchpad_ftest_playground; do psql -c 'ALTER TABLE OCIRecipe ALTER COLUMN git_repository DROP NOT NULL;' -c 'DELETE FROM LaunchpadDatabaseRevision WHERE major = 2210 AND minor = 8 AND patch = 4;' "$db"; done
<cjwatson> ^- manual revert
<tomwardill> ace :)
 * tomwardill blindly copy pastes random things from irc...
<cjwatson> Robert'); DROP TABLE Students;--
<tomwardill> there's at least one company named similar in companies house records
<cjwatson> Anyone up for reviewing https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/379682 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379683 to make it possible to build a wheel for unittest2?
<tomwardill> how much work would it be to drop unittest2 entirely? Or is a dep for something more fundamental?
<cjwatson> testtools mainly
<tomwardill> ah, that would do it, yes
<cjwatson> In fact testtools exclusively
<cjwatson> https://github.com/testing-cabal/testtools/pull/277
<tomwardill> +1 to both
<cjwatson> Thanks
#launchpad-dev 2020-02-28
<cjwatson> wgrant: Fancy some pip fun?  https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379684 arranges to build wheels as part of our deployment artifact, which is something on the order of four or five times as fast on my laptop as building them from scratch (as happens when we rsync out from one path on carob to a different path on target machines).
<cjwatson> Should get deployment times back to in the area of what they were before the conversion to pip.
<wgrant> cjwatson: what's the [0-9]* in the LP wheel name?
<wgrant> But sounds like a good idea
<cjwatson> Version
<cjwatson> $ grep ^__version__ setup.py
<cjwatson> __version__ = '2.2.3'
<cjwatson> I just wanted to avoid future amusement with some dependency's name happening to start with "lp-" or something
<wgrant> Ah, fair enough, indeed
<wgrant> No underscores for us
<wgrant> r=me
<cjwatson> Excellent, thanks
<cjwatson> atemoya:
<cjwatson> Thu 27 Feb 12:44:33 UTC 2020 Building code
<cjwatson> Thu 27 Feb 13:09:14 UTC 2020 Updating code
<cjwatson> Thu 27 Feb 18:13:05 UTC 2020 Building code
<cjwatson> Thu 27 Feb 18:32:48 UTC 2020 Updating code
<cjwatson> (both builds before the change above)
<cjwatson> Fri 28 Feb 01:32:28 UTC 2020 Building code
<cjwatson> Fri 28 Feb 01:36:38 UTC 2020 Updating code
<cjwatson> So that phase is indeed about five times as fast there.
<wgrant> Very nice.
<lifeless> noice
<tomwardill> ooh, a bug fix from 2017, that's exciting :)
<pappacena> Well, I think we still have some bugs from 2011 open... ð
<tomwardill> little steps, little steps ;)
<pappacena> One at a time. We will get there. :-)
<SpecialK|Canon> We have some bugs from before that ;)
<SpecialK|Canon> I put a card on the list for Frankfurt to do a little triage of our Critical bugs
<cjwatson> We have bugs from 2005 open ...
<tomwardill> huh, you can't sort bugs by age
<SpecialK|Canon> I can believe they're still bugs, but we should maybe talk about the definition of "Critical"
<tomwardill> oh, I guess Number is basically the same
<cjwatson> tomwardill: You can, use the cog at the left of the list of sort columns
<tomwardill> https://bugs.launchpad.net/launchpad/+bug/25
<mup> Bug #25: Allow discussion/commenting on translations <feature> <lp-translations> <ui> <Launchpad itself:Triaged> <https://launchpad.net/bugs/25>
<pappacena> wow... #25!
<cjwatson> Launchpad was of course the first project to use Launchpad for bug tracking (leaving aside the special case of bug 1)
<mup> Bug #1: Microsoft has a majority market share <canonical> <eoan> <iso-testing> <microsoft> <package-qa-testing> <Clubdistro:Fix Committed> <Computer Science Ubuntu:Fix Committed by compscibuntu-bugs> <LibreOffice:Fix Committed> <dylan.NET.Reflection:Fix Committed> <dylan.NET:Fix Committed>
<mup> <EasyPeasy Overview:Fix Committed by ramvi> <Ichthux:Fix Committed by raphink> <JAK LINUX:Fix Committed by jp-charras> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:Fix Released> <The Linux OS Project:In Progress> <Neobot:New> <Novabot:New> <OpenOffice:In Progress by
<mup> lh-maviya> <ReactOS Core Operating System:Incomplete> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:Invalid> <Ubuntu Malaysia LoCo Team:In Progress by apogee> <Wine:Confirmed> <Ubuntu:Fix Released> <Arch Linux:New> <Baltix:Confirmed> <Debian:In Progress>
<mup> <Fedora:Confirmed> <Fluxbuntu:Confirmed> <openSUSE:In Progress> <Tilix:New> <https://launchpad.net/bugs/1>
<cjwatson> I'm not sure what happened to bug 2; it's not just that it's private or something, it literally doesn't exist
<SpecialK|Canon> thanks mup
<cjwatson> OK, I think the wheelhouse work has shaved about 40 minutes off our deploy-to-production time.  Not bad
<pappacena> Great!
<SpecialK|Canon> "not bad" he says
<SpecialK|Canon> sounds pretty excellent to me, nicely done
<ilasc> :)
<cjwatson> (Comparing timestamp differences from the first couple of sections of https://deploy-logs.admin.canonical.com/index/legacy-lp/ndt/2020-02-25-ndt.log and https://deploy-logs.admin.canonical.com/index/legacy-lp/ndt/2020-02-28-ndt.log)
<ilasc> I second that!
<SpecialK|Canon> Am I right in my reading of that first one being 2h end-to-end?
<cjwatson> SpecialK|Canon: Sort of, but bear in mind that it's four steps run in sequence so if the operator isn't paying much attention then that will extend the time
<cjwatson> SpecialK|Canon: A fairer comparison is to add the four timedeltas between "Starting <foo> step" and "Ended <foo> step"
<SpecialK|Canon> cjwatson: ah thank you
<cjwatson> The work I just did takes about 20 minutes off each of the build step (within nodowntime-security-update.sh) and the deploy step (within ./parallel.py --config=lp --options='--push-new-code' --targets='nodowntime')
<SpecialK|Canon> I make that about ~45m of gaps, so around 1h15 in the first instance
<SpecialK|Canon> Heck even of 2h, 40m is quite a chunk!
<SpecialK|Canon> It's only getting more impressive ;)
<tomwardill> cjwatson: looks like we'll need to drop the NOT NULL constraint on OCIRecipe.git_path, as well
<tomwardill> Snap does the same
<cjwatson> tomwardill: Hm, yes, that makes sense
<tomwardill> I'll do a DB patch for it
<cjwatson> You could get away with a hack to avoid it but it would be gross
<cjwatson> tomwardill: Maybe add something like the consistent_git_ref constraint that Snap has while you're there
<cjwatson> tomwardill: While it was NOT NULL it wasn't needed, but now it is
<tomwardill> ah yeah, makes sense
<tomwardill> cjwatson: should/can I reuse the DB patch number from the reverted one? Or best to assign a new one.
<tomwardill> Not sure what happens if there's a gap in the run
<cjwatson> tomwardill: Reusing is fine in this case since it never went anywhere interesting
<tomwardill> cool :)
<cjwatson> Just update the description in dbpatches/allocated.txt
<tomwardill> cjwatson: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380038
<tomwardill> DB patch, code patch incoming soonish
<cjwatson> tomwardill: db=me
<tomwardill> gah, pushed code to wrong branch
 * tomwardill undoes
<tomwardill> matching code, lifting the detachFromGitRepository code into OCIRecipe from Snap https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380039
<tomwardill> what I have mostly learnt is that I cannot spell detach reliably.
<cjwatson> tomwardill: LGTM, but blocked on the DB patch being rolled out
<tomwardill> aye, it's not blocking any of the other OCI work though, so no worries :)
 * tomwardill unticks 'git_repository should not be null' on todo list, then ticks it again.
<cjwatson> tomwardill: Do you have time for a quick review of https://code.launchpad.net/~cjwatson/lpjsmin/py3/+merge/379696 and/or https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379996 before you finish up?
<tomwardill> sure
<tomwardill> cjwatson: +1 to both
<cjwatson> Great, thanks
<tomwardill> huh, test failures on my DB patch. I definitely ran them, and I definitely had the change applied this time!
<cjwatson> Hm, you swapped those two lines in the git_ref setter in your latest code patch
<cjwatson> Which of course isn't on db-devel yet
 * tomwardill arghs
 * tomwardill headdesks
<cjwatson> My guess is that you need to land a patch on master that just swaps those lines and nothing else
<tomwardill> yeah
<cjwatson> Then we'll need to make sure to deploy that to production before rolling out the DB patch
<cjwatson> (If we were a larger team this would require backing out the DB patch; as it is I think we can probably cope)
<tomwardill> patch incoming
<cjwatson> Cool.  I need to finish up soon but I can at least cope with occasionally thwacking buildbot
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380046
<tomwardill> just running tests
<cjwatson> r=me
<cjwatson> But yeah, will be quickest to confirm that this does fix the db-devel failures before landing
<tomwardill> hmm, I should check the tests against the old DB too
<cjwatson> I'm not worried about that in this case
<cjwatson> But well, maybe worth a try in fact
<tomwardill> running now
<tomwardill> oh boo, that fails
<tomwardill> storm.database.IntegrityError: null value in column "git_path" violates not-null constraint
<cjwatson> Ugh
<tomwardill> I think backout out the DB patch and doing a two-stage deploy might be safest?
<cjwatson> What does the first stage look like, though?
<tomwardill> drop the NOT NULL on the git_path
<tomwardill> then deploy the code change
<tomwardill> then add the constraint
<cjwatson> Ah, I see.  A bit elaborate, but at 6:30pm on a Friday I don't have a better plan.
<cjwatson> So not really back out - you can just land a change that edits that DB patch in place.
<cjwatson> Given that there's nothing on production there's probably some alternative strategy, but let's go with this
<tomwardill> how about a revert for the DB patch, given 18:30 friday, etc. Then fix it up next week?
<cjwatson> I think it would be fine to back out just the new constraint
<tomwardill> so just delete that line from the patch?
<cjwatson> DROP NOT NULL on its own is pretty safe.
<cjwatson> Yeah
<tomwardill> running tests
<tomwardill> okay, they passed with both old and new code: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/380048
<cjwatson> tomwardill: db=me, thanks
<tomwardill> cjwatson: landing, apologies for the late evening mess!
<cjwatson> tomwardill: np.  you'll need to force buildbot
<cjwatson> wait until you see the change at the left of http://lpbuildbot.canonical.com/waterfall first
<tomwardill> on it
<cjwatson> Right, mostly gone.  See you in FRA
<tomwardill> o/
