#launchpad-dev 2009-11-09
<adeuring> good morning
<wgrant> Morning adeuring.
<adeuring> hi wgrant!
 * wgrant prepares a few post-exam branches.
<gmb> Right. Big pot of Java, time to resolve some conflicts.
<wgrant> gmb: Is that the branch from... before the open-sourcing?
<gmb> wgrant: You know, I can't remember when I started this fricking thing. Possibly. Async dupefinder.
<gmb> It's one of those "everyone thought it was less important than I did" branches.
<wgrant> gmb: Yeah, that's the one that has been on +activereviews since the start.
<gmb> Hahaha.
<wgrant> But if you fix it, Ubuntu will drown in bugs even more :(
<gmb> wgrant: Well, I actually think I'm going to just junk it and start over. The conflicts are pretty huge and they're mostly to do with the template redesign. I can cherrypick the javascript out of the original branch
<gmb> wgrant: Nah, we'll just disable it for Ubuntu so that the interface keeps timing out all the time.
<gmb> Fuck, the branch doesn't even *build*.
<gmb> Man alive.
<wgrant> gmb: Awesome.
<gmb> Right, I'm gonna delete me some branches.
<wgrant> adeuring: Can you please ec2test & land those two branches of mine?
<adeuring> wgrant: thanks for the reminder! will do it right now
<wgrant> adeuring: Thanks.
<gmb> Oh for AJAX filtering in bug searches.
 * maxb has a small change for the pygettextpo makefile: https://code.edge.launchpad.net/~maxb/pygettextpo/makefile-fix-version/+merge/14256
<wgrant> bigjools: Do you want to do a quick review of https://code.edge.launchpad.net/~wgrant/launchpad/sourcepackageformat-to-sourcepackagetype/+merge/14581?
 * wgrant is splitting the super-branch.
<bigjools> sure
<gmb> Can we take TAL out and shoot it, please?
<gmb> It's harshing my mellow.
<bigjools> twisting your melons
<bigjools> wgrant: if only all reviews were as easy
<gmb> bigjools: This melon is for display purpose only.
<bigjools> gmb: slightly scratched?
<gmb> 'Chafed' is the word.
<mrevell> Morning troops.
<bigjools> eyup mrevell
* mrevell changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 1 of 3.1.11 | PQM is open - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<bigjools> wgrant: I'll land that for you
<wgrant> bigjools: Thanks.
<wgrant> gmb: How many branches did you get through before son-of-bride-of- or whatever it is?
<gmb> wgrant: 3. I should have made more of an effort to split them up, really, but there were certain very large changes that were pretty much atomic.
<bigjools> wgrant: did you see https://bugs.edge.launchpad.net/soyuz/+bug/478691 ... jeez
<mup> Bug #478691: empty ppa builders should allow more builds from a single ppa <Soyuz:New> <https://launchpad.net/bugs/478691>
<wgrant> bigjools: I did say that would happen.
<wgrant> bigjools: The current algorithm sucks. It just sucks less than the old one for non-daily PPAs.
<wgrant> I do not know how to make it better :(
<bigjools> me neither
<bigjools> other than adding more builders
<wgrant> Conserving some fraction of builders for expected short builds seems like it could work OK. But it needs a lot of careful thought.
<wgrant> We need to start discriminating against either daily PPAs or long builds. I cannot see a way around it.
<bigjools> we could auto-lower the priority for anything with a high build ETA
<wgrant> That doesn't help much.
<wgrant> It just means that the rest of the queue has to clear for 5 seconds, and the daily builds will immediately monopolise every builder for three hours.
<bigjools> who said anything about removing the current fix?
<bigjools> it seems we have 2 situations
<wgrant> The current fix is annoying.
<bigjools> either the long-build PPA owners complain, or everyone else complains
<bigjools> either way there's a lot of complaining
<bigjools> the current fix annoys the fewest people
<wgrant> I will also complain when I upload a few several-minute builds and it takes a couple of hours to build.
<wgrant> While 45 builders sit idle.
<gmb> Wow, bugspam from adeuring1!
<adeuring1> gmb: yeah, time tp update some bug statuses ;)
<gmb> adeuring1: Oh, sod, you're right. I'd forgotten.
<bigjools> wgrant: have you seen that happen?
<wgrant> bigjools: I haven't watched that closely, but I'm 99% sure it has.
 * gmb does the same thing
<wgrant> bigjools: Why wouldn't it have? It's a very common case.
<wgrant> Upload a few new versions to each series.
<wgrant> In most cases (at least in my TZ) that will not put any significant load on the build farm for very long at all.
<bigjools> wgrant: ok - so the only thing I can think of to make this better is to allow the extra build to start if there's an empty queue
<wgrant> bigjools: But then that will cause daily builds to monopolise again.
<wgrant> We need to actively discriminate against either daily build PPAs or long builds simply on the attribute of being daily build PPAs or long builds.
<bigjools> wgrant: the daily builds need more than one builder to complete within 24 hours it seems
<wgrant> bigjools: They do.
<bigjools> rock -> me <- hard place
<wgrant> bigjools: The current restriction is too restrictive for even daily builds.
<wgrant> But there needs to be *some* restriction.
<wgrant> Just a slightly smarter one.
<bigjools> so bump to 2 builders from 1?
<wgrant> ETOOSTATIC
<bigjools> or allow all builders as before unless we know it's a long build
<wgrant> Not all, I don't think.
<wgrant> But some large fraction.
<wgrant> And a much smaller fraction if they are going to be long.
<wgrant> But that still has a problem if there are lots of daily PPAs.
<bigjools> hmmm
<bigjools> we need more builders
<wgrant> So some fraction needs to be reserved for un-long builds.
<wgrant> al-maisan_'s quit message is very relevant.
<bigjools> so is my away message :/
 * wgrant doesn't know that one.
<wgrant> Ah.
<wgrant> I see.
<bigjools>  /away
<bigjools> fail
<wgrant> Describes this situation rather well.
<wgrant> Somebody needs to think and draw up lots of cases.
<al-maisan> wgrant: I was reading up on this topic a bit and the only paper that seemed even remotely pertinent was: http://www.springerlink.com/content/t4t2643046427702/
<al-maisan> wgrant: you may have free access to it since you're still a university student
<wgrant> al-maisan: Let me see if I can get it through the uni library proxy.
 * al-maisan was appalled about the lack of any grammar proof-reading in this article
<wgrant> Doesn't look like I have access to that, sadly.
<wgrant> bigjools: Bug #476036 probably wants to be worked on with some priority.
<mup> Bug #476036: sbuild needs to run dpkg-source inside the chroot <Launchpad Auto Build System:New> <https://launchpad.net/bugs/476036>
<bigjools> wgrant: yeah
<bigjools> I'll poke lamont
<bigjools> wgrant: http://pastebin.ubuntu.com/313996/
<bigjools> brb, need caffeine
<wgrant> bigjools: That looks like it should only fix the opposite situation :(
<wgrant> Although.. hm.
<bigjools> I've not looked at it in detail - it was to fix the upload problem
<leonardr> i upgraded to karmic and now i'm getting the infamous "no module named psycopg2" error when running launchpad tests. does anyone know the solution (and is there a troubleshooting page on the wiki for this kind of thing)?
<leonardr> i can import psycopg2 from a normal python shell but not from the bin/py shell
<mars> leonardr, the upgrade kills the launchpad-developer-dependencies package, and the LP ppas - did you re-enable them?
<leonardr> mars: thanks, argh
<leonardr> that's my least favorite feature of ubuntu--it gets me every time
<mars> heh, same here.  I wish we could fix the Software Sources dialog so it doesn't just say 'disabled by upgrade': 'and exactly which PPA was disabled, anyway?'
<bigjools> it got me too, that one
<mars> hehe
<leonardr> mars: i'm not getting "ImportError: libgettextpo.so.0: cannot open shared object file: No such file or directory"
<leonardr> s/not/now/
<leonardr> looks like i might be missing a zope package...
<leonardr> but perhaps that package ought to be put in sourcecode/
<sinzui> leonardr: I think that is pygttext in sourcecode.
<leonardr> hmm
<leonardr> ok, i guess the problem is that it's looking for .so.0 and only .so is in that directory?
<sinzui> leonardr: I think that is sourcecode/pygettextpo
<sinzui> We don't hack on pygettextpo or old_xmlplus (dtdparser) I wonder what needs to happen to make them eggs
<leonardr> adding a symlink doesn't help
<leonardr> i wonder who's looking for .so.0?
<leonardr> gary, maybe you have a clue
<leonardr> i'm trying to get launchpad working again after karmic upgrade
<gary_poster> ok
<leonardr> when i run 'import gettext' from bin/py or try to run the tests, i get this error:
<leonardr> ImportError: libgettextpo.so.0: cannot open shared object file: No such file or directory
<leonardr> sinzui said it might be looking in sourcecode/pygettextpo
<sinzui> leonardr: I did, but I see 'import gettext' as something else :(
<leonardr> sinzui: i gave the symlink the wrong name. giving it the right name makes it work.
<stub> leonardr: Shot in the dark, but try removing all the 'build' directories under sourcecode. make clean and friends doesn't seem to clean sourcecode properly which I found out this weekend (trying to use a rsynced tree build on 64bit on a 32bit machine)
<sinzui> leonardr: gettext somes with python
<mars> stub, yeah, there was a mailing list thread about that IIRC.  "make clean doesn't follow the GNU Makefile conventions"
<gary_poster> leonardr: I don't know.  What happens when you try ``$ locate libgettextpo.so``
<gary_poster> ?
<mars> stub, ideally we would have 'make distclean', which would do as you say: kill everything, as if it were a fresh checkout of trunk/.
<stub> Do you mean like realclean should be doing?
<mars> I think so.  In GNU terms, distclean == realclean
<mars> we just grab a new copy of the branch, but the tradition is to run 'make distclean' before 'make dist'
 * maxb finds "bzr clean-tree --unknown --ignored --detritus --force" to be rather useful
<maxb> gary_poster: Hi. ftr, ztk-2.5 passed everything (barring the broken c-i-p needing test) after a merge of 3.1.10 final
<gary_poster> maxb: awesome!  (and I regard the broken CIP as a burning issue for this week, so hopefully that will be squashed)
<gary_poster> broken CIP-needing-test I should say
<gmb> Hell, tracking down these 'o is null' errors in YUI is a pain.
<bigjools> wgrant: still up?
<gary_poster> maxb: also ftr, per your question this weekend about wanting to see a discussion I had had about packaging, foundations is moving their more focussed channel to freenode: #lp-foundations .  I documented it on https://dev.launchpad.net/Help .
<maxb> gary_poster: I have a feeling freenode would prefer you kept to a single project identifier - i.e. don't be both #launchpad and #lp
<gary_poster> maxb: hm, ok, so #launchpad-foundations you mean?
<maxb> yes
<gary_poster> alright
<Ursinha> hey danilos
<danilos> Ursinha, hey hey
<Ursinha> I see one of those TraversalErrors on Saturday, in translations, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1407F2349
<Ursinha> danilos, is this supposed to happen?
<danilos> Ursinha, well, yes :)
<danilos> Ursinha, look at https://bugs.launchpad.net/rosetta/+bug/462891/comments/11, this is now in translations-export.pt, and there are a number of other cases where it can happen (either on translations pages or on the other pages listed there)
<mup> Bug #462891: user.preferredemail shouldn't be None <oops> <qa-ok> <Launchpad Foundations:New> <Launchpad Translations:Fix Released by danilo> <https://launchpad.net/bugs/462891>
<danilos> Ursinha, it can also happen somewhere else in code, so it's probably best to fix this in preferredemail property, but I'd like foundations team to take care of that
<danilos> Ursinha, we've just worked around the most common one
<Ursinha> danilos, I understand
<danilos> Ursinha, my other branch attached to the bug fixes it in the property, but I don't know how to test it
<sinzui> danilos: ping
<danilos> sinzui, hi
<sinzui> danilos: I am looking at https://edge.launchpad.net/ubuntu/lucid/+packaging. I want to know what has to happen to make these packages linked. When I look at one, It appears to be linked: https://edge.launchpad.net/ubuntu/lucid/+source/gnome-termina
<sinzui> danilos: Is there something else that must happen to satisfy translations?
<danilos> sinzui, it'd be nice for those links to automatically propagate over series, or even better, to be shared among them
<danilos> sinzui, for example, https://edge.launchpad.net/ubuntu/lucid/+source/gnome-terminal/+packaging shows links but there's no karmic or lucid in there
<sinzui> danilos: I see the auto-linking bug. I scheduled it for this release. I am not sure what is missing since the example is linked to a  series (though I know it should be master, not main)
<danilos> sinzui, and it'd be PITA to ask people to manually enter them with every ubuntu release
<danilos> sinzui, right, I'll let you worry about solving that problem then
<danilos> sinzui, the other thing that translations need is a nice way to guide people from an ubuntu source package to register a project and automatically link it to that source package in all relevant ubuntu releases
<sinzui> danilos: So you thinking is that while the linked series is carried forward, the packaging link is not created
<danilos> sinzui, yeah
<sinzui> danilos: bac and I discussed this briefly. I think that is good for this series, but preventing regressions (like the missing links) is more important. If I can fix these this release, we know the remaining work is to lower the barrier to setup a mirrored project
<danilos> sinzui, ideally though, we'd have an option of just linking a product and (distribution, sourcepackagename) apart from the existing links ( (distroseries, sourcepackagename) <-> productseries)
<danilos> sinzui, right, that's about it then
<sinzui> danilos: I have seen that bug about just product.
<danilos> sinzui, as I said, the above are just my thoughts on what'd be nice, but I don't want to try to impose my solution to the problem :) you are free to fix existing system if you think that's better and/or easier
<sinzui> I'll collection my notes this week and send the to you and the others to comment on.
<danilos> sinzui, excellent, sounds good! thanks for looking into it
<leonardr> gary, stub: reopening the pygettextpo drama...
<leonardr> i followed stub's suggestion and removed the build directories, then did make clean and make
<leonardr> now there's no longer any gettextpo.so in sourcecode/pygettextpo, and i don't know how to build one
<gary_poster> leonardr: looking
<gary_poster> leonardr: rocketfuel-get might do the right thing
<gary_poster> but looking
<gary_poster> I don't have a .so there either
<gary_poster> oh yes I do
<gary_poster> leonardr: I'm trying to dupe
<leonardr> gary: ok, rocketfuel-getting
<gary_poster> leonardr: skeptical of myself.  feel free, but probably won't help is my new new new opinion
<leonardr> i agree
<gary_poster> leonardr: not helpful news, but I removed build from pygettextpo, did a make clean and then a make and got gettextpo.so back.
<gary_poster> leonardr: maybe do ``make clean`` and then ``make SHHH=`` and pastebin the output for me?
<gary_poster> kfogel: hi.  I forget, are you off today or Fri?  Would like to ask you for permission to proceed with trying to get ~launchpad-committers to happen, per email from maxb this weekend.  Maybe need permission from flacoste; he has not weighed in yet.
<leonardr_> gary: i don't know how much of my rambling you saw earlier, but here's the result of make (with an error)
<leonardr_> https://pastebin.canonical.com/24523/
<gary_poster> leonardr_: I didn't see anything.  Looking
 * leonardr_ hopes internet is back for good
 * maxb mumbles about private pastebins
<kfogel> gary_poster: (on phone, or about to be, but will look as soon as off.  it's just a standup call)
<kfogel> maxb: :-)
 * gary_poster helps leonardr_ and maxb : http://paste.ubuntu.com/314235/ :-)
<maxb> leonardr_: Can you try running 'make PYTHON=python2.4 PYTHON_VERSION=2.4' in sourcecode/pygettextpo/ ?
<leonardr_> sure
<maxb> leonardr_: And also, do you somehow not have python2.4-dev installed? Nor launchpad-developer-dependencies ?
<leonardr_> argh, it looks like 2.4-dev was removed in the karmic upgrade
<maxb> Well, yeah, that'll break things :-)
<gary_poster> heh
<gary_poster> thanks maxb
<gary_poster> leonardr_: check launchpad-dependencues too
<gary_poster> that should have required python2.4-dev
<gary_poster> IIRC
<maxb> I think lp-dev-deps depends on the other two
<gary_poster> maxb: ah ok
<gary_poster> maxb: do you have an opinion about the python2.5-dev thing you raised on the weekend, and to which I replied briefly?
<gary_poster> That is, do you think I should include python2.5 versions of all of the python2.4 things?
<gary_poster> I'll go to flacoste if you are not sure.  He is the keeper of this key.
<maxb> Theoretically, if we want a working python 2.4 and python 2.5 launchpad environment, we should include them,
<gary_poster> That's what I figured
<maxb> Practically, however, I don't think any packages that support python2.4 but don't support python2.5 will exist
<maxb> So really it's only a matter of theoretical correctness, not actual breakage
<maxb> Oh, and on the subject of policy w.r.t the bzr branches - policy? what policy? :-)
<maxb> Basically "bzr push what you dput"
<gary_poster> OK.  Maybe I'll stick with the current change, and then when we go to Py 2.5 with the codebase, switch all the 2.4 bits to 2.5, at which point it sounds like we will be theoretically correct also.
<maxb> Agreed.
<gary_poster> Cool.
<gary_poster> policy: heh, fair enough.  Perhaps I was thinking more of instructions
<gary_poster> That is, do I check out the branch, and then that's the same as if I had gotten a source deb?
<maxb> You are right, it would be good to lay down some advisory instructions, rather than assuming that everyone's "common sense" is aligned
<gary_poster> heh
<maxb> gary_poster: That's the intent.
<gary_poster> maxb: ok, cool.  I'll make a stab at an initial wiki doc for more knowledgable people to correct.  After lunch. :-) Thanks much.
<maxb> We probably ought to specify the appropriate cmdline to invoke dpkg-buildpackage too - it's a bit pointless uploading a copy of the branch in the source package (which 0.55 has)
<gary_poster> ah, yes, that sounds like something I would never have thought of
<gary_poster> ok running away for a bit
<Ursinha> rockstar, hi
<Ursinha> rockstar, I'm chr this week and checking all the pending vcr-imports
<Ursinha> rockstar, I see one that the repo url is valid, but it has no files in it, yet, maybe
<Ursinha> rockstar, should I approve it or is it going to bork somehow?
<leonardr_> gary: i'm having another, even stranger problem
<leonardr_> i'm trying to set up a launchpadlib branch, and it's using lazr.restfulclient 0.9.3 even though setup.py specifies >=0.9.9
<leonardr_> do you know what could cause that to happen?
<leonardr_> i'm going to remove my old lazr.restfulclient eggs but it's annoying
<gary_poster> leonardr_: so, this is not in launchpad, but in a branch.  Yes, I know what would cause that.
<gary_poster> 1 sec
<leonardr_> np
<leonardr_> this is rich, i don't even have a 0.9.3 egg
<gary_poster> leonardr_: the explanation of what is going on is long and annoying.  I'll describe at some point, but for now summarize that you will need to do one of these: (1) remove *all* lazr debs; (2) use the zc.buildout version that launchpad uses; or (3) use a non-system Python.  I recommend 3, and that is what I do for the non LP work.
<leonardr_> ah, the deb is to blame
<leonardr_> i see. now that apport uses launchpadlib, my previous strategy (1) is no longer tenable
<leonardr_> gary, can you run me through (3)?
<gary_poster> leonardr_: download a Python from Python.org.  Install with a --prefix that puts it somewhere in your home dir.  I then add it to a ~/bin (using names like 24python 25python and 25python) and set my PATH to include ~/bin.  use that Python for bootstrap. Done
<gary_poster> (there are many variants; that's just what I do)
<leonardr_> all right
<mrevell> Night colleagues
 * maxb wishes for a time machine to go back and stop the concept of a lazr.* namespace being invented
<leonardr_> gary: --prefix is a configure argument, right? i'm getting through configure and make but python setup.py is failing me
<gary_poster> leonardr_: yes, configure.  don't use python's setup.py.  do ``./configure --prefix=...`` then ``make`` then ``make install``
<leonardr_> aha, make install, thanks
<leonardr_> ok, that's working, but i need to install some dev packages like lib-devreadline to make this environment nicer
<leonardr_> gary: i'm investigating this now, but maybe you know. what dev package do i need to install to compile a version of python that speaks https?
 * leonardr_ trying libssl-dev
<gary_poster> leonardr: that's what I was going to recommend, yes
<gary_poster> I have it installed
<kfogel> gary_poster: back
<gary_poster> kfogel: cool
<kfogel> gary_poster: sorry, remind me of context.  the ~launchpad-committers team will be for [?]
<gary_poster> kfogel: everyone who can commit to LP.  Therefore, LP Canonical employees plus people like maxb, wgrant, and so on.  Use cases are primarily for making a place for branches that everyone can access who should be able to access.
<gary_poster> there was some discussion of other uses
<maxb> kfogel: e.g. I'd quite like write access to the meta-lp-deps branches, and I'd like a place to move the branches currently owned by the launchpad-dev open team to.
<kfogel> gary_poster: hunh.  I remember being in favor of this, but I can't remember why now.  Everyone has r/o access to all branches, except security branches.  Would this new list have access to sec branches too?
<kfogel> gary_poster: or is this about write access?
<maxb> This is about community write access
<gary_poster> kfogel: write access, for branches not managed by pqm
<kfogel> gary_poster: if about write access, is the idea that any branch to which both a canonical dev and a community dev can read, they can also both write?
<gary_poster> kfogel: not necessarily because of PQM
<gary_poster> but in some cases
<kfogel> gary_poster: so, I thought our goal was for community devs to be able to merge via PQM the way we do.  Is that not the case?
<gary_poster> kfogel: that is the case but we do not have all branches managed by PQM
<gary_poster> kfogel: cases include lazr.*, launchpad-dependencies branches, and the Python 2.5 sprint work
<kfogel> gary_poster: but I mean, that's the main use case right?  That is, community devs being able to submit to PQM for stable, db-stable, devel, db-devel ?
<maxb> It is a shame that ~launchpad-dev is already taken as a name - ~launchpad-committers is a bit of a long name to type in branch urls
<kfogel> maxb: yeah
<maxb> kfogel: PQM submission is less of an issue in the short term, I think, because there's always a Canonical reviewer that we can nag to land branches for us
<gary_poster> kfogel: that would be a great eventual usage, and yes, that would be more important once it happened, and maybe having the group around would facilitate that change.  But that's not why I'm talking about this right now
<kfogel> gary_poster: so this is for other, non-PQM-managed branches right now...
<kfogel> gary_poster:  I'm getting kind of an edgy feeling about this.  Let me explain:
<leonardr> gary: ok, hopefully the last one. in my launchpadlib branch built with my custom python, i can now import all of the modules mentioned in setup.py -- except, for some reason, lazr.uri. any ideas?
<kfogel> We have a bunch of branches.  Some (previously mentioned) are managed by PQM.  Others are: lazr.*, launchpad-dependencies branches, and the Python sprint branches, and probably others to come.
<gary_poster> maxb, kfogel: yeah, agree about name, though I don't care a heck of a lot TBH.  The name could be changed mechanically, I suspect.  Not sure what the ramifications would be, and how easy it would be to work around them.
<gary_poster> leonardr: probably not a dependency?
<kfogel> gary_poster: will it always be the case that if we want developer jrandom to have write access to one of those branches, we'd want him to have write access to all of them?
<maxb> gary_poster: I was more thinking about mere finger-work typing "bzr foo lp:~launchpad-committers/...."
<kfogel> gary_poster: i.e., is membership in the team a statement of "we trust this person's intentions and competence"?
<leonardr> gary, aren't the dependencies what's listed in setup.py? i don't have a separate versions.cfg for this
<kfogel> gary_poster: I'm not saying that's bad, just trying to make sure we agree what we're doing.
<gary_poster> leonardr: yes
<gary_poster> kfogel: your questions are harder :-)
<leonardr> so yeah, lazr.uri is mentioned in setup.py as a dependency but is not importable
<gary_poster> leonardr: ok give me a branch and I'll look at it
<kfogel> gary_poster: I think now is the time to ask them, since we're apparently creating One Team To Rule Them All :-).
<kfogel> gary_poster: but yeah, answer leonard first, it's no problem
<leonardr> gary: it's a branch off of lp:launchpadlib. it's got to be some problem with my setup
<gary_poster> kfogel: my inclination is JFDI.  Here's my argument.
<leonardr> i can import the system lazr.uri (from the deb) using the system python
<maxb> kfogel: Well, there's a bunch of branches currently owned by ~launchpad, for which some means of arranging for community write access would be nice :-)  These branches, afaik, are (1) meta-lp-deps, (2) sourcecode deps, (3) adhoc collaborative launchpad branches (e.g. python 2.5).  If there's a sensible granularity of access, maybe those should be split up.
<gary_poster> kfogel: right now, we only accept someone as a contributor if they have shown competence in some way (do we have some kind of champion process or something?), *and* are willing to sign a contributors agreement.  In all cases, including Canonical employees, that's pretty much what we have.  If we need something fine grained, maybe we can add it later.  But I contend that social pressures, and the fact that someone can always 
<gary_poster> a group, will likely be sufficient here, as they have been in other open-source cases (the zope.org repos come to mind: you get a blanket permission there; same with plone; probably same with other projects
<gary_poster> may have been truncated.  tail:
<gary_poster> But I contend that social pressures, and the fact that someone can always be removed from a group, will likely be sufficient here, as they have been in other open-source cases (the zope.org repos come to mind: you get a blanket permission there; same with plone; probably same with other projects
<gary_poster> and, may I add:
<gary_poster> )
<gary_poster> and "." even
<gary_poster> ok leonardr trying
<kfogel> gary_poster: I completely agree with you so far.  Some of your text got elided.
<kfogel> <gary_poster> and, may I add:
<kfogel> <gary_poster> )
<kfogel> <gary_poster> and "." even
<gary_poster> sorry, that was me trying to be funny :-/
<kfogel> gary_poster: oh, heh
<leonardr> gary: if i set an impossible version for lazr.uri in setup.py i do get an error
<gary_poster> so, done
<kfogel> gary_poster: so, +1 here, I'd just say make sure to spell this out when proposing to Francis that this new team have write access in whatever places we're initially proposing it will.
<gary_poster> kfogel: ok, will do.  thank you
<kfogel> gary_poster: thank _you_!
<kfogel> high time
<maxb> Does the tip of sourcecode branches get sucked into rollouts automatically? If so, I guess I see why they might need to not be opened to community committers.
<kfogel> maxb: I don't know.
<gary_poster> maxb: not for production.  for edge and such, yes
<gary_poster> leonardr: what version of py did you use?
<maxb> Another access question is the ~launchpad PPA. It's working quite well with me emailing sync requests, but I wouldn't say no to direct upload access if there was a way, and people were happy to give it to me
<leonardr> gary: 2.6.4
<gary_poster> leonardr: ok
<gary_poster> maxb: eh...I'll add that to the email I guess, but I'm in favor of one step at a time :-)
<leonardr> gary: i see references to .buildout/eggs/lazr.restfulclient (etc) in sys.path, but not lazr.uri
<maxb> Sure... just trying to assemble all possible uses for a community write-access team
<gary_poster> cool
<gary_poster> leonardr: bah, I wonder if this is a setuptools c11 problem }:-(
<leonardr> gary, what's a c11?
<leonardr> at this point i'm tempted to remove apport packages and all other things that use launchpadlib to restore solution (1)
<gary_poster> the newest version that phil eby released after much annoyance in the distutils list that is not worth recounting.
<leonardr> ah
<gary_poster> leonardr: sorry, as usual I'm trying to multitask--while waiting for one process to finish I look at another.  My build is in the middle of bin/buildout
<leonardr> gary: np
<leonardr> gary: fyi, rick spencer is not going to work on the gtk launchpad credential client until after uds
<gary_poster> leonardr: urg. :-( ok
<lfaraone> gary_poster: ping.
<gary_poster> leonardr: I was able to import lazr.uri just fine.
<gary_poster> lfaraone: hello
<gary_poster> leonardr: uh.  are there any helpful error messages you can share?
<lfaraone> gary_poster: What additional information would you need, re branded domains.
<lfaraone> gary_poster: I talked to kfogel and kiko about it about a week ago in #launchpad.
<leonardr> gary: unfortunately not. it just doesn't show up in sys.path
<gary_poster> lfaraone: Hi.  The information I need is internal.  The point of the email I sent was largely to let you know that we were not ignoring you.
<leonardr> i'll look through the output of buildout
<lfaraone> gary_poster: ah, okay. (I was wondering... :)
<gary_poster> :-)
<lfaraone> gary_poster: let us know if you need anything.
<gary_poster> lfaraone: will do, thank you, and sorry for not being clear :-)
<lfaraone> gary_poster: on an entirely unrelated note, our codehosting system is git via gitorious. if some of us were to submit the patches, could we enable launchpad to work with gitorious as it does with the native bzr codehosting?
<leonardr> gary: i get an "Unbuilt egg for ClientCookie [unknown version]" warning, but that's probably not relevant
<lfaraone> (like linking branches, etc)
<gary_poster> lfaraone: that's entirely out of my area, sorry.  I know we are interested in improving our git support, but the approach is something that the strategy team is working out with the bzr team and codehosting team.  I'm just a foundations guy, helping out because your initial question touched on my area. :-)  I will be happy to raise the question on launchpad-dev on your behalf if you'd like (though you can do the same).
<lfaraone> gary_poster: well, on the other hand, I don't want to ask for too many things, at the risk of becoming a nusiance.
<gary_poster> leonardr: hm...are you sure you used the home-installed Python?  That ClientCookie thing is something that typically comes from a messed up site-packages
<gary_poster> lfaraone: I dunno, if you don't ask, you don't get.  :-)  you can at least maybe get an idea of the plans, and see if they match your goals
<lfaraone> gary_poster: understood.
<leonardr> gary: do i need to use home-installed python for everything, or just bootstrap? i've been using it for bootstrap
<leonardr> yeah, bootstrap writes the right #! lines, so it shuold be fine
<leonardr> gary: however, you're right that it is looking in /usr/lib/python2.6/dist-packages for the ClientCookie
<gary_poster> leonardr: just bootstrap, and then bin/buildout without a Python invocation
<leonardr> yeah, that's what i did
<gary_poster> leonardr: grr.  never seen that.  something is wrong in set up, and no idea what it is.  leonardr, what happens if you start up that self-installed version of Python and look at sys.path?
<gary_poster> does it include site-packages?
<leonardr> gary:
<leonardr> >>> [x for x in sys.path if 'site' in x]
<leonardr> ['/home/leonardr/bin/python/lib/python2.6/site-packages']
<leonardr> so, yes
<gary_poster> leonardr: yeah, sorry wasn't clear
<gary_poster> that's your local site-packages
<gary_poster> which is presumably empty
<leonardr> ok. is there a site packages in my personal install?
<leonardr> did i configure incorrectly maybe?
<gary_poster> leonardr: is this using a launchpadlib that had previously used a system python?
<gary_poster> IOW, you are reusing a checkout
<gary_poster> ?
<gary_poster> trying to duplicate that way
<gary_poster> heh that didn't get very far
<gary_poster> but leonardr, if you think that's possible, try a fresh checkout.
<gary_poster> leonardr: if I am right that this is the bug, then you probably could just remove bin and .installed.cfg and rerun bin/buildout.
<leonardr> that's a promising line of inquiry
<gary_poster> but I'd try the fresh checkout to be sure
<leonardr> yeah, trying that
<gary_poster> leonardr: that means that you have zope.proxy build with the system python
<gary_poster> you have to delete the egg and rerun buildout
<leonardr> ok, should i maybe delete all my cached eggs?
<gary_poster> leonardr: sure.  The ones with C extensions are the only ones affected, but that certainly covers the bases.
<leonardr> gary: oops mystery solved: https://bugs.edge.launchpad.net/lazr.restful/+bug/474522
<mup> Bug #474522: TypeError when lazr.restful tries to unmarshall URI <api> <oops> <Launchpad Foundations:Triaged by leonardr> <lazr.restful:New> <https://launchpad.net/bugs/474522>
<leonardr> gary: i hope this is not a portent of things to come: after deleting the eggs, bin/buildout is now reaching for the deb version of wadllib
<gary_poster> leonardr: OOPS error: good catch.  deb version: that's insane.  this is in the clean checkout version, you are sure?
<leonardr> gary: yeah. i specified version 1.1.4 of wadllib and then it downloaded the egg
<gary_poster> leonardr: it downloaded the egg, and then somehow started using site-packages version?
<leonardr> gary: no, here's what happened
<leonardr> originally i ran with an unmodified setup.py
<leonardr> it complained: "lazr.restfulclient wants 1.1.4 of wadllib, and i have 1.1.2"
<leonardr> 1.1.2 being the deb packaged version
<leonardr> i changed setup.py to require 1.1.4 explicitly, and it downloaded 1.1.4
<leonardr> gary: and now, once again, i can't import lazr.uri from my shell
<leonardr> /usr/lib/python2.6/dist-packages is in my sys.path
<leonardr> as before, there are many ~/.buildout/eggs/lazr.* references in sys.path but lazr.uri is not among them
<gary_poster> I'm really very suspicious that you are using a system Python sometime.  Maybe check your history?  I know the pertinent code, and that's the only way I can think of this happening.
<leonardr> yeah, i'm inclined to agree
<leonardr> it's possible that i ruined this branch by slipping up and using a system python after the initial setup
<leonardr> in that case i should probably alias 'python' to my python
<gary_poster> for excitement, try rm-ing the bin directory and .installed.cfg
<gary_poster> and then rerun bootstrap and bin/buildout
<gary_poster> and see if that makes things better
<gary_poster> without having to do a fresh checkout
<leonardr> ok
<leonardr> gary: no effect. and i checked .installed.cfg before deleting it and 'executable' was correct
<leonardr> but, now i have a checkout that works and one that doesn't
<leonardr> i'll diff them and see what's different
<gary_poster> leonardr: ok
<gary_poster> probably messy :-/
<leonardr> gary: the only difference is that my non-working checkout seems to have a garbled setuptools name:
<leonardr> __buildout_signature__ = z3c.recipe.tag-0.3.0-py2.6.egg zc.recipe.egg-1.2.2-py2.6.egg zc.buildout-1.4.2-py2.6.egg setuptools-h8seYOqeLdvVn0YIl04xMQ== zc.buildout-1.4.2-py2.6.egg
<leonardr> and the non-working version doesn't define installed_develop_eggs
<gary_poster> weird
<leonardr> gary: do you need to use your system python when testing launchpad too?
<gary_poster> leonardr: we go to great lengths, for better or worse, to use the system python when testing/developing launchpad
<leonardr> ok, i'm about to find out if it works
* flacoste changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 1 of 3.1.11 | PQM is open  | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<kfogel> lfaraone: ping
<kfogel> Is database/schema/launchpad-2207-00-0.sql the "most recent master schema" for Launchpad?
 * gary_poster doesn't know :-/
<kfogel> (Background: I'm trying to get stats on number/percentage of failing code imports for CVS, same for SVN, same for git.)
<kfogel> gary_poster: s'okay.
<kfogel> flacoste: ^^
<kfogel> flacoste: (my db question above)
<kfogel> wgrant: ping
<flacoste> kfogel: whatever is in the tree is the most recent
<kfogel> flacoste: I'm asking a more basic question -- I don't know which filename I want.
<flacoste> kfogel: what do you want?
<kfogel> flacoste: I'm looking for a master description of our schema.  Though it's not incredibly urgent, as I am using psql cmds to find it out anyway.
<kfogel> flacoste: overall background (my goal) is in this mail from jml: https://lists.canonical.com/mailman/private/canonical-bazaar/2009-October/004732.html
<flacoste> kfogel: there is no master schema file, you get the schema in use by starting with that file and then applying all the patches
<kfogel> flacoste: ah, okay.  thanks.
<dobey> leonardr: ola :)
<flacoste> kfogel: i always use psql when building that sort of query, or you can use another GUI tool (no particular recommendation here)
<kfogel> flacoste: psql is exactly what I'm doing, yeah.
<leonardr> dobey: ok
<dobey> leonardr: so i take it i should pull some bzr branch to look at some code?
<leonardr> to start with, check out lp:launchpadlib
<leonardr> i've created a class that encapsulates the process of asking the end-user for their username, password, and desired access level, then sending that information to launchpad to authorize a request token
<leonardr> independant of any particular user interface
<dobey> ok
<leonardr> and i've written two user interfaces: a scriptable one that i use for testing, and a console-based one that reads data from sys.stdin
<dobey> so it's pretty much designed around how the oauth stuff in launchpad currently works?
<leonardr> yes, it's very dependant on the specifics of launchpad, and i filed a number of bugs to make launchpad's behavior more predictable
<leonardr> basically we are pretending to be the user's web browser and the user's brain looking at the launchpad web page
<dobey> right
<leonardr> the most important code is in src/launchpadlib/credentials.py, the RequestTokenAuthorizationEngine class
<leonardr> the code that tests it is in trusted-client.txt
<leonardr> and the test for the full workflow (where you pass in a RTAE for a specific UI type into Launchpad.login_with()) is in introduction.txt
<leonardr> here's how the RTAE works
<leonardr> there are a number of hook methods called at various points in the process
<leonardr> startup(), input_username(), user_refused_to_authorize(), etc
<leonardr> at each step the UI is supposed to display something to the user and/or accept input
<leonardr> most of these methods take a 'suggested message' as an argument
<leonardr> oh, i actually wrote three of these classes--i took the existing behavior that opens the user's web browser and asks them to hit enter, and implemented it as an RTAE
<dobey> right. hrmm
<dobey> i think for ubuntu one we're wanting to registering/subscribing as well, without opening a browser
<leonardr> aaaargh
<dobey> could this be extended to handle that, or does it sort of handle that all in the background and just open the browser, and we'd need to write something else new to deal with the subscribe/register part?
<leonardr> well, what i have now doesn't open the browser (unless you choose the "browser" ui)
<dobey> oh
<leonardr> but the workflow only knows about one launchpad page: +authorize-token
<dobey> i was reading the YOU_NEED_A_LAUNCHPAD_ACCOUNT string
<dobey> and it says "i'm opening the browser"
<leonardr> that's a failure condition - "come back later"
<leonardr> ah
<leonardr> yes, if you use the command-line client it will open your browser to the login page as a courtesy
<dobey> is that a bug then? :)
<leonardr> but it's still a failure condition, you can't continue
<leonardr> i'd say it's a minor bug that the generic error string mentions command-line-specific behavior
<dobey> ok, so it doesn't actually open the browser?
<leonardr> RTAE never opens the browser or does anything on the client side
<leonardr> it calls hook methods and it makes http requests to launchpad and screen-scrapes the html
<leonardr> there's a "browser ui" hook method that will always open the browser and make the user log into launchpad through the browser
<dobey> ok
<leonardr> s/hook method/subclass/
<dobey> there's no way to register from the class though?
<leonardr> there's a "command-line" subclass that will ask for the username and password if the user has a launchpad account, and if they don't have one, tells them to get a launchpad account and opens their browser
<leonardr> no
<leonardr> that would mean more screen-scraping
<dobey> ok
<dobey> yeah i don't really like screen-scraping
<wgrant> kfogel: Hi.
<dobey> i was actually thinking of writing an "OAuth Desktop" extension spec, to respond with JSON for the client to build some UI, and do stuff
<leonardr> if you're serious about this we should probably create a second web service that lets you register a user account, request a request token, authorize the request token
<wgrant> bigjools: Sorry, no 2am during exams :(
<kfogel> wgrant: Can I mine you for some information (if you're familiar with our DB schema -- if not, then don't worry about it)?
<dobey> and perhaps a similar thing on top of openid (though i know less about how openid works than oauth)
<wgrant> kfogel: Sure.
<leonardr> your oauth desktop extension spec sounds like my second web service
<dobey> heh
<dobey> well if i did the oauth extension spec, i could implement it in my python oauth lib, and would be trivial for anyone to use in python
<dobey> and it wouldn't really take a lot of time to do i don't think
<leonardr> you'd have to get the launchpad team to sign off on it and implement it
<dobey> right
<dobey> would you be the one doing that work? :)
<kfogel> wgrant: thanks.  Let me give you quick background first, then tell you where I am.  I'm trying to graph stats on failing code imports in Launchpad, by type of import (e.g., CVS, SVN, SVN via bzr-svn, and GIT).  I've got access to poke around in a read-only fashion using psql, and I've just now found lib/lp/code/enums.py and the RevisionControlSystems enum therein.  I see also that we have these four tables:
<kfogel> er, six:
<kfogel>  codeimport codeimportevent codeimporteventdata codeimportjob codeimportmachine codeimportresult
<kfogel> wgrant: so I'm trying to figure out the connections between the tables, naturally.
<leonardr> dobey: probably not, i'm booked on other stuff
<kfogel> wgrant: is any of this sounding familiar to you, or are you also a beginning in this area?
<kfogel> s/beginning/beginner/
<leonardr> dobey: i wouldn't expect a whole lot of outside enthusiasm for your spec, because what we're doing is a security problem
<dobey> leonardr: who should i discuss that with?
<leonardr> the only secure client for your launchpad password is your web browser
<leonardr> i'm talking about the general oauth-using public
<dobey> leonardr: afaict, oauth is a security problem
<leonardr> the only reason i'm doing this is that it's even worse to have everyone writing their own authentication client
<leonardr> dobey, elaborate?
<dobey> leonardr: if it were up to me, it would just be username/password and be done with it
<dobey> well oauth was designed for the web, not the desktop
<dobey> the "general oauth-using community" is "the web"
<kfogel> wgrant: ah, ignore enum 3, svn via bzr-svn.  psql says there are none of those -- it must be a placeholder for something we haven't turned on yet.
<dobey> typing one's password in a browser isn't really any more secure than typing it in a gtk+ dialog
<wgrant> kfogel: SELECT COUNT(*) FROM codeimport WHERE review_status=40; should get the count that are regularly failing.
<wgrant> Now, the various types...
<leonardr> dobey: it sounds like you've got the desktop pov that if you installed it on your computer, you trust it
<dobey> leonardr: no. my pov is that i don't trust anything
<kfogel> wgrant: *nod*  trying it now.  Though when I did 'select distinct status from codeimportresult;', I got these three codes: 200, 300, 310
<dobey> leonardr: if i haven't installed it on my computer, i certainly shouldn't be trusting it *more*
<wgrant> kfogel: codeimport, not codeimportresult.
<leonardr> dobey: not sure what's the hypothetical thing not installed on your computer
<wgrant> kfogel: 40 is FAILING, and is set on the codeimport when it has failed repeatedly some number of times.
<dobey> leonardr: "the web"
<kfogel> wgrant: thanks.  (I know you said codeimport -- I was just pointing out there's this *other* table named codeimport result whose numbers apparently mean something entirely different).
<dobey> leonardr: i'm not sure what makes people think the browser is a more trusted place to type a password, than a dialog box
<leonardr> dobey: because you already type your launchpad password into a browser all the time
<kfogel> wgrant: so, I'd expect the count to return something other than zero (from codeimport where review_status=40), yet it returns 0.
<wgrant> kfogel: Hmmmm. That's unfortunate.
 * wgrant pokes further.
<dobey> leonardr: yes, and people type their paypal passwords in a browser all the time. yet they still get taken by phishing mails
<dobey> they open their browser to a web page that looks like paypal, type their password, and then their account is empty
<kfogel> wgrant: but yes, I see in enums.py the same thing you see, about review_status.  40 looks like the thing we want.
<leonardr> dobey: the solution is not to increase the number of places where we can tell people it's ok to type their password
<dobey> leonardr: i'm not saying it is. in fact, i want a single place for it on the desktop
<dobey> leonardr: but that single place, is certainly not a web vm
<kfogel> wgrant: I am continuing to poke too.
<leonardr> people already type their launchpad passwords into a desktop application called 'mozilla'
<leonardr> there's no web vm
<dobey> mozilla is a web vm
<dobey> as is opera, or epiphany, or midori
<leonardr> and any other application you create will have the same essential properties
<dobey> no
<leonardr> plus, it will not look like the place people are used to already typing in their launchpad passwords
<kfogel> wgrant: asking internally about that zero row count too.
<dobey> leonardr: which is already the case with this new class, no?
<wgrant> kfogel: The sample data isn't much good for code imports :(
<leonardr> dobey: yes. that's why i don't like this new class and only did it because the alternative is even worse
<kfogel> wgrant: ooooooh, good point, I forgot it's not real data.
<kfogel> wgrant: thanks.
<leonardr> this way, any non-browser ui mechanism you want to use will use the same underlying code
<wgrant> kfogel: Oh. I presumed you were getting that zero row count from production. This makes more sense now.
<leonardr> all the mechanisms will be in the same package and can be audited together
<kfogel> wgrant: it does make sense, but it sure isn't going to make it easy for me to test what I want to test either :-).
<dobey> what i really want is a central framework for accessing/using web services/APIs
<dobey> but i haven't written it yet
<kfogel> wgrant: let me see, maybe I can get access to prod db or a copy thereof.
<wgrant> kfogel: Has stub produced his sanitised DB yet?
<wgrant> That would be convenient for this.
<kfogel> wgrant: that's exactly what I'm going to check on.
<dobey> although it's high on the list of needing to get written
<kfogel> wgrant: he doesn't come online for a while, I will ping him tonight (his morning)
<leonardr> dobey, what other web services are you using?
<leonardr> gary, maybe you want to review my trivial branch https://code.edge.launchpad.net/~leonardr/launchpadlib/restfulclient-0.9.10/+merge/14666 ?
<gary_poster> leonardr: sure looking
<dobey> leonardr: facebook, various bugzillas, ubuntu one, launchpad, twitter, identi.ca, the palm pre thing, google mail/calendar/docs/etc...
<dobey> leonardr: practically everything i do on the comptuer every day is 'use some network service of some sort'
<leonardr> right
<dobey> but there are a million different ways to access them all
<dobey> and it's overly painful, and firefox eats up all my resources
<dobey> and it doesn't integrate with my workflow/system
<gary_poster> leonardr: approved
<leonardr> there are different ways to access them because there's no consistent design
<leonardr> although: i don't know how deep you've looked into it, but all the google web services are basically the same
<rockstar> abentley, have you seen that the bmp comments javascript is erroring?
<dobey> well, they all extend the same basic API for the most part
<rockstar> abentley, Y.all("a.menu-link-reply") is null
<dobey> but accessing my mail on google, and messages people send me on facebook, are not the same
<dobey> (though they should be)
<leonardr> dobey: agreed. they are not the same because they were designed by different people
<dobey> leonardr: yes, but they both provide APIs
<dobey> leonardr: so you don't *HAVE* to use their UI
<dobey> it's just that there are no good alternative UIs
<dobey> and having to type the same password in 20 different places to access different parts of the service from different apps is also really lame
<dobey> so to start off i want to centralize authentication to web services in the desktop
<rockstar> thumper, around?
<dobey> there are definitely still some interesting remaining issues with doing so, but i think their solutions will become apparent as the project grows and code gets written
<dobey> leonardr: i need to call it a day now, and go do other stuff, but we can continue the discussion later/tomorrow.
<dobey> leonardr: thanks for discussing it :)
<leonardr> sure, np
<dobey> later :)
<kfogel> wgrant: I'm just going to proceed using the staging data and pretending that review_status=30 ("suspended") is what we want (even though in real life 40 is what I'll use).
<leonardr> dobey: probably not relevant to you, but one point about how the google services "all extend the same basic API"
<leonardr> that's by design. it didn't have to be that way. amazon put out a bunch of web services that were totally different. google decided to publish things that are superficially different (calendars and spreadsheets) in the same way
<wgrant> kfogel: By 'staging data' you mean sample data? Or do you actually have access to staging?
<kfogel> wgrant: I'm querying against the db staging is using.  it has some code import records in state 30, enough to work with.
<kfogel> wgrant: I think I'm okay now, btw -- I mean, from here the query is obvious.
<wgrant> kfogel: But that should have some in 40 as well.
<kfogel> wgrant: the trick was learning to ignoer the codeimportresult table :-).
<kfogel> wgrant: it doesn't have any 40 at all.
<maxb> Does anyone fancy reviewing this for me? https://code.edge.launchpad.net/~maxb/pygettextpo/makefile-fix-version/+merge/14256
<lfaraone> kfogel: pong
<rockstar> thumper, hi
<rockstar> maxb, r=me
<maxb> rockstar: Thanks. How do I get it landed?
<rockstar> maxb, dunno.  I may have to harass barry, but luckily, he's sitting on the other end of the table from me.
<maxb> heh
<maxb> gary_poster: did you get around to thinking about that meta-lp-deps wiki doc? I was thinking about writing some, but my biggest blocker is where to put it in the wiki! :-)
#launchpad-dev 2009-11-10
<henninge> jml: ping
 * wgrant sighs at misinterpreted comments.
<sinzui> spm: ping
<gary_poster> maxb: the meta-lp-debs information would probably go now in a new section on https://dev.launchpad.net/Hacking based on the current arrangement.  I don't love that, but coming up with a different arrangement is not something I want to be a blocker for this.
<maxb> that's.... one huge wiki page
<gary_poster> uh-huh
<gary_poster> well, that's what a wiki is all about, right?  Putting all your information in?  On one page?  With no hyperlinks off of it?  That's entirely in the wiki spirit.
<gary_poster> :-D
<spm> sinzui: yo
<sinzui> spm: I am looking for evidence of duplicate packaging links in production. Can you run https://pastebin.canonical.com/24356/ and paste me the results?
<spm> sinzui: 1st is 0 rows
<spm> so is 2nd
<sinzui> \o/
<sinzui> thanks spm
<spm> np. sorry for the dealy was out to lunch with the other canberran canonicalers
<sinzui> np. I am just spending my hours listening to music and triaging bugs
<thumper> mwhudson: how goes the sprint?
<mwhudson> thumper: hi
<mwhudson> thumper: quite interestingly for the most part
<mwhudson> thumper: quite a lot of talking
<mwhudson> thumper: but there's some hacking happening now
<thumper> mwhudson: do you think that there'll be time to do the bzr-svn stuff?
<mwhudson> thumper: very likely
<mwhudson> thumper: has the 2.5 support landed yet?
<thumper> I've not looked
<wgrant> It hasn't.
<wgrant> But bzr-svn works with 2.4...
<mwhudson> wgrant: the tests segfault
<mwhudson> which doesn't lead to warm fuzzy thoughts
<wgrant> Ahh.
<mwhudson> thumper: about the BFB thing
<thumper> mwhudson: yes...
<wgrant> launchpad-dependencies is updated with 2.5 stuff, so I don't imagine there are many more blockers.
<mwhudson> thumper: is there an awareness that its not really build from *branch* but more likely build from *recipe* ?
<thumper> mwhudson: have you tried the bzr-svn tests since 1.0 was released?
<thumper> mwhudson: yes
<thumper> mwhudson: I think so
<mwhudson> thumper: ok good
<mwhudson> thumper: i worry about the db schema that was mentioned
<mwhudson> thumper: but didn't look super closely
<thumper> mwhudson: I was going to send an email about it at the end of last week, but I didn't get around to it
<spm> thumper: here ya go! http://www.stosyth.gov.uk/images/editor/Sep03a-round-tuit.jpeg
<thumper> spm: ta
 * spm expects thumper to live up to his nic after that ;-)
<adeuring> good morning
<wgrant> adeuring: Morning. Thanks for landing those two branches.
<adeuring> Hi wgrant. welcome :) And thanks for all your contributions!
 * wgrant has another 2000 lines of that branch to split up.
<maxb> wgrant makes a good point: I should ask gary_poster for an update on losa signoff on the big 2.5 move
<maxb> the branch is ready and waiting
<wgrant> bigjools: Morning.
<bigjools> morning wgrant
<mwhudson> hello bigjools
<wgrant> bigjools: You poked me at 2amish?
<mwhudson> (and everyone else of course!)
<bigjools> hi there mwhudson
<bigjools> wgrant: I did, but I can't remember why!
<wgrant> bigjools: Ah. Anyway, thanks for landing that branch.
<bigjools> wgrant: no prob
<mwhudson> firefox: giving you a reason for multiple cpus
<wgrant> And 128GiB of RAM.
<wgrant> bigjools: What approach should I take for the rest of the 3.0 branch? There are 1800 lines of diff remaining, but there are a couple of ways I can split it.
<bigjools> wgrant: how many lines after you discount sample data?
<wgrant> bigjools: A little over 1500.
<bigjools> wgrant: ok we need to split that into two
<wgrant> bigjools: Let's see what happens if I strip out the lp.archiveuploader changes.
 * thumper is frustrated that bigjools, wgrant and mwhudson all hash to the same colour for nick colouring
<bigjools> wgrant: I can't remember what your branch looks like now - but ideally you'd have one branch that you can land independently first
<bigjools> black and white FTW
 * wgrant has no colours either.
<mwhudson> thumper: colours are for the bourgeois!
<thumper> it is effectively black and white right now
<thumper> bigjools: what did you think of my email?
<bigjools> thumper: not got to it yet
 * thumper cracks the whip over bigjools
<bigjools> my wife already does that
<bigjools> thumper: +1
<thumper> bigjools: cool
 * thumper waits for kiko and flacoste to answer
<wgrant> Ah, yes, new CoC version.
<wgrant> Somebody will have to touch the CoC bits for the first time in four years...
<mrevell> Morning!
<thumper> mrevell: still want to know what we did?
<mrevell> thumper, Yeah, please.
 * thumper looks at the milestone
<matgeek> I am here!
<matgeek> Just got the home server going.  Running rockfuel-setup as we speak
<thumper> mrevell: mostly oops fixes, and some ajaxy stuff
<thumper> mrevell: code review comments without extra page loads
<thumper> mrevell: setting the commit message
<thumper> mrevell: setting the branch status
<thumper> mrevell: sped up the branch listing page for ubuntu package branches
<mrevell> thanks thumper
<wgrant> sinzui: Did popey ever communicate with you about CoC 1.1?
<wgrant> bigjools: Before the next bit of this is merged, some data repair is needed on production.
<bigjools> wgrant: oh?
<wgrant> (UPDATE sourcepackagerelease SET dsc_format='1.0' WHERE dsc_format IS NULL, in particular)
<bigjools> ok
<bigjools> why is that needed?
<wgrant> There has been stuff in nascentupload to prevent NULL dsc_formats for some years, but a LOSA confirmed that there are some in the production DB.
<wgrant> I presume from before this change.
<bigjools> 2 questions:
<bigjools> 1. what happens if the data is not repaired?
<bigjools> 2. does your branch prevent it from being null again?
<wgrant> It has been prevented by nascentupload for years.
<stub> A NOT NULL constraint should be added (on another branch, since that needs to land on launchpad/db-devel)
<wgrant> If the data isn't repaired, some copies will crash when they try to verify that the format is supported by the destination series.
<wgrant> stub: Oh yes, certainly.
<bigjools> ok
<matgeek> Umm, I am a bit of a newby round here, but I can't find an obvious  way to subscribe to the launchpad mailing lists...
<wgrant> matgeek: You need to join the team.
 * wgrant wishes that bzr merge -i allowed rewriting of hunks.
<matgeek> wgrant: How do you do that may I humbly ask please before I forget my pie?
<wgrant> matgeek: Wow, that is awkward to find.
<wgrant> matgeek: "Join the team" at the top of the right-hand column of the team
<matgeek> wgrant: Which page should   I be on?  I am looking at https://edge.launchpad.net/launchpad
<wgrant> matgeek: That's the project, not a team. Which mailing list do you want?
<wgrant> launchpad-dev or launchpad-users, I presume.
<matgeek> launchpad-dev and launchpad-users
<wgrant> Which are https://launchpad.net/~launchpad-dev and https://launchpad.net/~launchpad-users respectively.
<matgeek> wgrant: thanks! &-)
<wgrant> bigjools: I have (816 + sampledata) lines of diff in the first branch. I could reasonably split it again, however.
<wgrant> Anyway, sleep now.
<sinzui> wgrant: yes.I outlined what some one would have to do there was a brief discussion ending with Mark simplifying what has to be done. The Launchpad team is not working on this, but in the course of the discussion I concluded I could add the new CoC and increment the version number in the model in with little effort to meet the base requirements
<sinzui> wgrant: If you want to know more detail or hack on the issue, ping me
<bigjools> wgrant: 816 + sampledata is fine
<maxb> gary_poster: Hi. Do you have info from LOSAs / IS on when they might give the go ahead for converting devel to Py2.5 ?
<gary_poster> maxb: Hi.  it is primarily up to me/us AFAIK.  I need to get the Hardy launchpad-dependencies set up and tested, then I can ask them to update staging to those dependencies, and then we can test on staging, and then we can ask them to update edge and land.  If you were able to help on the Hardy version that would be fabulous, but I was planning on doing it.
<gary_poster> I have a national holiday tomorrow so I'll be away then
<maxb> bzr merge, fix changelog, dput, bzr push :-)  Should take 5 mins
<gary_poster> maxb: heh, agree, especially for you (unless you put the step-by-steps up?).  Other things that take precedence for me immediately, but I should be able to do that and try to arrange the staging today at least.
<maxb> It would be easier for me to do if I could actually push to the branches :-/
<gary_poster> ack.  Well, working on that too.  Well, I sent the email. Sprint activity is slowing down the response.
<maxb> If we're handing this off for consumption other than developers trying out 2.5, perhaps we should fix up the dependencies properly after all
 * maxb will have a think about it later
<gary_poster> ok
<mrevell> Night my friends
<henninge> danilos: hey!
<danilos> henninge, hey hey
<danilos> henninge, how's it going? fun JSing?
<henninge> danilos: yes, only problem is that we are missing an API ... :(
<henninge> danilos: to keep up the fun, I wanted to ask you if you can look at an oops, I just stumbed upon:
<henninge> danilos: https://translations.launchpad.net/gvb/trunk
<henninge> danilos: It oopses on the link formatter for a branch on that page.
<henninge> same for +translations but I think that is the default view, isn't it?
<danilos> henninge, there's already a critical bug I am working on (re translation credits, setTranslationCreditsToTranslated fails with a traceback so that's not done yet
<danilos> henninge, so, I'll come to it when I come to it :)
<henninge> danilos: oh, did I know about that?
<henninge> if so, I forgot ...
<henninge> danilos: cool, thanks
<danilos> henninge, no, I've seen it only when running the script over all the messages, it failed at 10% but we can definitely see this with the live code as well
<Ursinha> danilos, do you know what can be possibly happening here: http://paste.ubuntu.com/315276
<danilos> Ursinha, I have no idea, it seems weird (perhaps launchpadlib version mismatch?)
<danilos> Ursinha, have you retried and do you get prompted to re-authorize the same app again?
<Ursinha> danilos, yes, twice weird
<danilos> Ursinha, well, I've heard there are weird things happening with our auth stuff :)
<Ursinha> danilos, :)
<Ursinha> danilos, well, I need to land that branch :) what do you suggest?
<danilos> Ursinha, if you've independently ec2 tested it, you can use bzr pqm-submit
<danilos> Ursinha, you can also try clearing up all the oauth tokens and getting a new one, but it needs to persist
<Ursinha> danilos, the problem is that's not persisting
<Ursinha> danilos, I run the script and it's not creating the credentials
<Ursinha> guess because of the error in lplib.credentials
<jtv> leonardr: thanks again.  I just filed bug 480192.
<jtv> mup?
<jtv> https://bugs.edge.launchpad.net/lazr.restful/+bug/480192
<Ursinha> jtv, I think mup didn't like the dot at the end
<Ursinha> bug 480192
<Ursinha> or it's dead :)
<danilos> Ursinha, token should be created and registered when you press 'enter'
<Ursinha> danilos, yes, and I think Launchpad.credentials does this job of creating the file
<jtv> mup: heel.
<danilos> Ursinha, if that's the case, then you'll have to find lplib expert to debug that, since I am about to head home :)
<Ursinha> danilos, launchpad.credentials.save_to_path
<Ursinha> ok L)
<Ursinha> :)
<jtv> danilo-afk: see ya :)
<Ursinha> leonardr, hi
<danilo-afk> jtv, cheers
<danilo-afk> henninge, btw, on that page I see import of translations set but no branch linked, that's weird
<danilo-afk> henninge, https://translations.edge.launchpad.net/gvb/trunk/+translations-settings
<danilo-afk> anyway, really afk :)
<henninge> danilo-afk: right! then that's the problem.
<leonardr> hi ursinha
<Ursinha> leonardr, hi, do you know what I"m possibly doing wrong here: http://paste.ubuntu.com/315276
<leonardr> ursinha: yes, you are using an old version of lazr.restfulclient. upgrade to 0.9.10 and it should work
<henninge> danilo-afk: I'll file a bug about it
<Ursinha> leonardr, but what updates lazr.restfulclient? I thought that was updated with the lp sources
<leonardr> ursinha: i'm about to update the lp sources to include 0.9.10 right now
<Ursinha> leonardr, oh, so I'm not doing anything (much) stupid :)
<leonardr> no
<jtv> gary_poster, here's the lazr.restful bug we just filed for returning html from named operations: https://bugs.edge.launchpad.net/lazr.restful/+bug/480192
<cody-somerville> why is lucid at the bottom of a source package page?
<cody-somerville> https://edge.launchpad.net/ubuntu/+source/gst-plugins-base0.10
<cody-somerville> totally missed that theres a newer version of that package in lucid
<sinzui> cody-somerville: that is a crappy sorting bug
<sinzui> We will have it fixed on edge in a few days
<sinzui> I'll take that bug now. I think I can get tis fixed today
<gary_poster> maxb (and indeed everyone who knows): improvements warmly invited--nay, beseeched--for https://dev.launchpad.net/Hacking#Changing%20the%20launchpad%20dependency%20debs .
<thumper> morning
<thumper> gary_poster: what's the state of the python 2.5 landing?
<wgrant> mbarnett: Can you please run 'SELECT MAX(dateuploaded) FROM sourcepackagerelease WHERE dsc_format IS NULL;' on production? I want to check that nothing has created such broken data in recent years.
<gary_poster> thumper: still hoping to do it this week, though I have tomorrow off.  The only thing holding it back is getting the meta debs done properly, testing on staging, and having the LOSAs ready for it on edge (which I'm hoping will be easy).
 * thumper nods
<thumper> gary_poster: it is just we have work blocking on 2.5 landing
<gary_poster> thumper: ack
<mbarnett> wgrant: https://pastebin.canonical.com/24582/
<wgrant> mbarnett: That's not something I have access to.
<mbarnett> wgrant: doh!  sorry... knee jerk reaction there..
<mbarnett> wgrant: http://pastebin.ubuntu.com/315398/
<wgrant> mbarnett: Excellent, thanks.
<mbarnett> wgrant: welcome
 * mwhudson is not sure he needs mail telling him that no branch upgrade jobs ran every 10 minutes
<gary_poster> heh
<rockstar> thumper, hi
<thumper> rockstar: hi
<thumper> rockstar: how's the sprint going
<rockstar> thumper, it's going great for code.  I'm getting shiteloads done.
<rockstar> thumper, so I saw your comment on my mp, and I also knew that we didn't want to use the whole vocabulary.
<rockstar> thumper, I was reminded of it while getting the UI review.
<thumper> ok
<rockstar> thumper, so I'm removing it.
<thumper> removing what?
<rockstar> thumper, just fyi.
<rockstar> Er, changing it to the correct vocabulary.
<thumper> rockstar: cool
<thumper> rockstar: I want to keep the functionality
<rockstar> thumper, yes.  It's a regression otherwise.
<thumper> rockstar: the only reason I hadn't done it is that I was considering splitting the statuses
<rockstar> thumper, also, branch picker, where are you with that?
<thumper> rockstar: done nothing
<rockstar> thumper, mind if I do it.  I got accosted by aquarius here.
<thumper> rockstar: I got blocked on the functionality I was after
<thumper> rockstar: go for it
<rockstar> Er, "mind if I do it?" with a question mark.
<rockstar> thumper, cool, thanks.  I have the benefit of having aquarius and beuno together long enough to know how to do it.
<thumper> excellent
<thumper> go for it
<thumper> JFDI
<rockstar> :)
<rockstar> thumper, also, there's a bug where the scanner isn't updating the branch and repository formats when they change.  That bug blocks me from landing the upgrade UI.
<thumper> it isn't that it isn't updating it
<thumper> it is that the branch isn't getting scanned
 * thumper thinks
<thumper> I haven't looked into it
<thumper> I suggest talking with mwhudson about it
<rockstar> thumper, ah, okay.  Well, either way, I need to be able to depend on what the database says the formats are.  abentley and I had a long conversation about it.
<mwhudson> i suggest crying as a tactic
<rockstar> However, yesterday when I pulled the trigger on upgrading the lazr-js branch to 2a, I noticed the UI wasn't reporting the change.
<rockstar> mwhudson, I will do whatever is necessary; bawl if I hate to.
<mwhudson> rockstar: a sane way would probably be to invent a BranchScanJob and have the xmlrpc method mirrorComplete create one
<rockstar> mwhudson, okay.  Would you like me to do it, or would you like to? (I'm gone for two weeks, hoping to use these holiday swap days I keep accumulating when I return)
<mwhudson> rockstar: good question
<thumper> mwhudson: this sound dangerously like having only one way to do something...
<mwhudson> thumper: which bit?
<thumper> mwhudson: making a scanner job :)
<mwhudson> thumper: heh
<mwhudson> of course, the real fix is to do the puller in the scanner, but well
<mwhudson> er
<mwhudson> other way around
<mwhudson> rockstar: it should be pretty easy if you just do it in a minimalist way
<mwhudson> rockstar: i can try to do it after hours tonight i guess
 * thumper goes to get Maia
<rockstar> mwhudson, I wouldn't exert yourself.  It's not a huge deal.
<mwhudson> ok
<MTecknology> quick question.. Is LP developed in a that you can turn on/off part of it? Like if I get a junky system to devote to testing it and have a hard time running it, can I turn off translations and branch uploads?
<rockstar> mwhudson, I was just hoping to incite a riot with politically driven verbage like "blocker"
<mwhudson> rockstar: i see
<mwhudson> MTecknology: to some extent, yes
<rockstar> MTecknology, well, when you do "make run" you technically have most of code hosting disabled.
<mwhudson> MTecknology: for example you can just not run the scripts involved in code hosting
<rockstar> But you can't just turn off the web parts of code hosting.
<mwhudson> MTecknology: you can't easily turn off the branch tab though
<MTecknology> I'm just concerned about system usage. I'll probably ask more when I install it if it isn't entirely apparent. I'm also goign to try to practice developing with the LP API
<MTecknology> Thanks :D
<MTecknology> I didn't destroy conversation, did I?
<rockstar> MTecknology, many of us are currently sprinting.
<MTecknology> rockstar: I don't like running since I had my back surgery
<rockstar> MTecknology, http://en.wikipedia.org/wiki/Sprint_%28scrum%29
<MTecknology> rockstar: thanks, I never knew what that actually meant. I'll be quiet and let the smart guys run around their code.
<spm> mwhudson: "...no branch upgrade jobs ran every 10 minutes" agreed. throwing to logfile...
#launchpad-dev 2009-11-11
<wgrant> spm: Can you please run 'SELECT dsc_format, COUNT(*) FROM sourcepackagerelease GROUP BY dsc_format;' on prod for me?
<spm>  dsc_format | count
<spm> ------------+--------
<spm>             |  70050
<spm>  1.0        | 388445
<spm> wgrant: ^^
<wgrant> Good lord.
<cody-somerville> ugh
<wgrant> Hmmm.
<wgrant> Thanks spm.
<spm> np. I'm delighted with the response from yourself and cody-somerville tho :-D
<wgrant> Doesn't *really* matter, since we'll just clobber all the null ones. But it's a bit strange that there are so many.
<maxb> I have https://code.edge.launchpad.net/~maxb/pygettextpo/makefile-fix-version/+merge/14256 pending, who do I nudge about making it happen?
 * thumper stabs launchpad.dev
<wgrant> thumper: What is it doing to you today?
<thumper> wgrant: stabbing launchpad
<thumper> wgrant: some small tweaks, some bigger ones
<thumper> wgrant: mostly ui stuff
<wgrant> Ah.
<thumper> sinzui: still around?
<thumper> wgrant: perhaps you know...
<thumper> if someone is starting a new project
<thumper> what is the minimum they need to do to upload files?
<cody-somerville> thumper, can you provide more context to your question?
<thumper> someone cornered me in the weekend after moving a project to launchpad
<thumper> and asked "how do I add download files"
<thumper> I couldn't find the link
<wgrant> thumper: Add a release.
<wgrant> Files are added to releases.
<thumper> how do you add a release?
<thumper> is there a friendly button?
<thumper> ah
<wgrant> (releases are created on top of milestones within a series, but I think there is a button which adds a milestone and immediately turns it into a release)
<thumper> create release
<thumper> that should at least be mentioned somewhere on the main page
<thumper> it isn't intuitive to a new user
<wgrant> It's Registry. Of course it's not intuitive.
<wgrant> Although it's much better than it used to be, the whole series thing is very obscure.
<wgrant> thumper: Can the codehosting-related stuff be dropped from #launchpad's topic yet?
<thumper> I'd say so
<thumper> I'm not aware of any issues right now
<spm> thumper: actually related to that - that emergency codehost sync logs we setup back when - I assume we can remove that as well now?
<thumper> spm: what was set up exactly?
<spm> thumper: huh. nm. it's been removed. la la. from memory, it was sync the codehost *logs every 3 mins
<thumper> spm: yes we can stop that :)
 * spm does nothing and pretends he does - all gone!
<thumper> spm: gee, thanks
<spm> haahahahaha
<spm> I recall seeing it the other day, and assumed it was still there; but alas, one of the other losas was probably on a clean up phase :-)
 * thumper smacks head on the table
<thumper> refactoring the wrong test
<thumper> d'oh
 * wgrant tries to work out how to add breadcrumbs.
<thumper> wgrant: automagically
<wgrant> thumper: I find that unlikely.
<thumper> wgrant: it kinda is
<thumper> wgrant: by default it uses the canonical_url_iterator
<thumper> wgrant: you can override the heirarchy it uses
<wgrant> thumper: I know how that stuff works.
<thumper> wgrant: take a look at lp.code.browser.branch.BranchHeirarchy
<wgrant> But some objects don't have a breadcrumb.
<wgrant> eg. builds.
 * thumper tries to remember how branches did it
<thumper> BranchMergeProposalBreadcrumb
<thumper> You need an IBreadcrumb adapter
<wgrant> Ah, looks like r9842 does the sort of stuff I want.
<wgrant> Thanks.
<thumper> np
<wgrant> We should destroy custom titles whereever possible, right?
<thumper> maybe
<thumper> not sure on the edict
 * wgrant trusts the old 3.0 heading rules for now, since they are at least better than what is implemented now.
<thumper> wgrant: what is your time split between real study and LP right now?
<wgrant> thumper: It depends how you calculate it. Last week was pretty much fully studying, due to the first batch of exams. There are no non-trivial exams remaining, so not much study is happening now.
 * wgrant glances at pagetitles.py and screams.
<thumper> wgrant: it is deprecated thank god
<wgrant> It still has calendar stuff, I see.
<wgrant> How useful and modern.
<thumper> heh
<ajmitch> I take it that's a bad thing? :)
<wgrant> Well, the calendar system was closed aaages ago, and actually removed more than two years ago.
<ajmitch> a great opportunity to clean up old code
<spm> heh. get quadruple code karma for removing old code
<ajmitch> does it have to be old code?
<spm> that was a pun, not serious. :-)
 * ajmitch can think of a few things that could be removed
<spm> soyuz... malone... :-P
<wgrant> There are lots of <browser:pages> with completely different views using the same view class. If I want to specify a title without using pagetitles.py, does that mean I have to create a subclass and rip it out into its own <browser:page>, or is there some less inconvenient method that I'm missing?
<thumper> wgrant: make the page_title a property that is smart?
<thumper> wgrant: or yes, subclass
<wgrant> Subclassing seems less revolting.
<wgrant> Just makes the ZCML suck a bit more.
<wgrant> Neeeed Grooook.
<wgrant> thumper: Has Martian been considered at all for LP?
<thumper> yes
<thumper> but that is about as much as I know
<thumper> flacoste knows more
 * ajmitch even attended a talk in the weekend that mentioned that
<wgrant> How did kiwipycon go?
<thumper> pretty good actually
<thumper> I enjoyed myself
<thumper> which is always a good measurement
<wgrant> How many were there?
<ajmitch> it was full, so about 150
<wgrant> Not bad.
<ajmitch> everything ran smoothly, talks were good
 * wgrant notices the team leads mailing list, and discussion on it about it being private -- it isn't.
<poolie> hee hee
<poolie> srsly?
<wgrant> Sadly.
<ajmitch> let's hope that there's not too much interesting gossip on there
<citrus> We know the list is public, and what we put in it is public. We will state if we removed something that was private
<wgrant> I used LP to make the private list, but I do have one concern.  It appears
<wgrant> that some people I added through LP just have an email address outside of
<wgrant> Canonical.  I was hoping to only use our Canonical addresses for this list.
<wgrant> That indicates that at least Brian believes it's private.
<citrus> Brian is mistaken
<citrus> karls emails are the best. His notes might include a Dalek invasion during an interlude in the meeting
<wgrant> I see.
<wgrant> Anyway, thought it best to mention it, since you guys don't have an awfully good track record of keeping private mailing lists private.
<citrus> No we dont. The owner who demand privacy seem to want public lists.
<citrus> sÃ© la viÃ©
<wgrant> https://launchpad.net/builders/rothera/+history has a back link at the bottom. Were it not for that link, that template could go away. Now that we have good breadcrumbs, is there any reason to keep it?
<citrus> I agree that the link is a distraction. we are also removing the links that use >> and <<
<wgrant> citrus: What is the label policy for pages with broken context?
<wgrant> (ie. anything under a distro)
<wgrant> Should they include the object, since the main heading is not correct?
<wgrant> s/broken context/broken root or watermark thingy/
<citrus> wgrant: I do not understand. label is a loaded and very confusing term in launchpad now that it is used for forms, more headings, and surprisingly to fill the pages primary heading slot
 * citrus starts throwing things when he has to think about what the consequences are setting a view's label and page_title
<wgrant> citrus: I mean the label on the view, which becomes an h1.
 * citrus starts throwing things
<citrus> The label should be the title of the context object, or the heading of the context's subordinate information, or the action the user is performing on the context
<wgrant> citrus: That's fine for projects.
<citrus> I wish I had good examples for this
<wgrant> But not for anything else.
<wgrant> Well, projects and distros.
<wgrant> But not project series, distro series, distro arch series, distro source packages, or disto series source packages.
<wgrant> Or builders.
<wgrant> Or most of the rest of the LP objects that could conceivably have actions.
<citrus> it works for https://edge.launchpad.net/builders/rothera/+history
<wgrant> citrus: That label isn't one of the three possibilities.
<citrus> "the heading of the context's subordinate information"
<citrus> The context is a builder, the information is that it is history
<wgrant> citrus: But this includes the context's identity, whereas most other views do not.
<citrus> indeed
<wgrant> I am wondering whether it should or not.
<wgrant> Since it is neither specified nor consistent.
<citrus> We are not consistent. I got angry and stopped working on heading for 4 weeks
<citrus> what colour is your bike shed?
<citrus> The history example includes the smarty quotes which I think are vestigial to 1.0
<wgrant> I'm mainly trying to correct the braindead titles produced for queue and build pages, but labels got dragged along too.
<citrus> There are many headings that can be improved, many well thought heading were made bad the day all the page titles changed
<wgrant> Yes.
<wgrant> The actual titles are easy to fix.
<citrus> wgrant, I have learned that I have to argue for each heading. I lost a lot of arguments
<wgrant> Because they always include context.
<wgrant> But labels have no such luxury.
<citrus> The page_title is documented as not being required. Yet I have had to add that to more than dozen views last month to fix breadcrumbs. I am sure it is required.
<citrus> I think a good label focuses your attention and task.
<wgrant> Won't breadcrumbs include '+somethingstupid' unless you specify page_title?
<citrus> Oh.now I see how that is screwed up
<wgrant> ?
<citrus> https://edge.launchpad.net/builders/rothera/+history
<citrus> ^ I think the <h1> is in the tempalate's content. The view does not provide a label
<wgrant> I know.
<wgrant> My new version of the build listing template fixes that.
<wgrant> The page_title is obviously 'Build history'
<wgrant> The correct label is not at all obvious.
<citrus> The correct page_title is "build history" because rothera (i386) is already in the breadcrumbs/title
<wgrant> Right.
<citrus> The page_title we are seeing could be the label if we extend beuno's rule that the heading for a page that is alternate view of an object should extend the object's title...
<wgrant> I suspect that the label should be "Build history for ârothera (i386)â", but I really do not know.
<citrus> eg: Lucid: Edit Lucid
<citrus> eg: My team: My team members
<citrus> wgrant: +1
<wgrant> citrus: Inconveniently, all of the rest of the build pages live within a distroseries, and nobody seems to know how to refer to those.
<citrus> I would not be surprised if each page was updated by a different developer
<citrus> adding a post-3-ui-tag to a bug improves the chances of getting it scheduled. We are still allocating 50% of our capacity to oopses and UI bugs
<citrus> wgrant: I just came across several of your bugs for foundations. They should have been triaged and tagged with post-3-ui-cleanup. I updated the bug so that they should up in jml and flacoste's reports
<wgrant> citrus: Thanks.
 * wgrant is tempted to email the list about the label issue.
<citrus> go ahead. A few weeks have passed and I think the rules are a little clearer.
<citrus> We might get a codified rules out of this.
<citrus> oh, but a lot of engineers are distracted by the lazr-js sprint and UDS
 * wgrant shall compose the email later after work stuff.
<wgrant> Thanks for your help.
<thumper> citrus: what's with the name change?
<thumper> citrus: and what are you doing up?
<citrus> I learned that I have today off
<citrus> So I am working according to my own agenda
<citrus> Except I am too tired to fix the answers pages, remove the style sheet, or kill off main-template
<thumper> citrus: plz add wiki, kthxbye
 * citrus install wikimedia
<thumper> citrus: I'm about to use locals().extend(setup_func()) to a pagetest :)
<thumper> locals().update that is, not extend
<mwhudson> citrus: fix blueprints!!1
 * mwhudson hides
<citrus> mwhudson: please give me a UI review so that I can fix the worst parts
<citrus> https://edge.launchpad.net/blueprint/+milestone/3.1.11
<citrus> The problem with fixing a menu or removing the 1.0 button means I need to find ui reviewers who are all sprinting
<mwhudson> citrus: argh, can't really help you there :(
<citrus> I really need an alternate personality so that I could review my self
<adeuring> good morning
<al-maisan> moin adeuring
<adeuring> hi al-maisan!
<maxb> Who do I talk to about getting stuff landed to sourcecode/ branches?
<mrevell> Morning
<pgquiles> what source should I get for setting up a launchpad in production? branch 'stable'? branch 'db-stable'? branch 'devel'? branch 'db-devel'? is there a tarball anywhere? what are those 'db-something' branches?
<maxb> pgquiles: You are aware of the image licensing constraint, right?
<maxb> a.k.a. "Canonical does not grant you a licence to use the existing images in a production instance"
<pgquiles> maxb: yes
<pgquiles> I read that, in the FAQ IIRC
<pgquiles> atm I'm trying to set it up, we'll replace the images, etc when/if I get it to work
<pgquiles> we are only going to use launchad internally, though, not public site
<maxb> I'd love to do the same, but I don't have the time to replace all the images :-/
<maxb> Anyway, https://dev.launchpad.net/Trunk explains how the four branches relate
<pgquiles> thanks
<pgquiles> how can I download launchpad 3.1.10? I can't find a branch/tag/tarball/whatever by that name :-?
<wgrant> bigjools: Want to review my branch at some point, or shall I use OCR?
<wgrant> bigjools: I'd also like to get the production data unbroken.
<bigjools> wgrant: I can, I'm just not sure *when* I can :/
<bigjools> go for OCR and I'll try and tag along
<bigjools> can someone do me a favour please: du -sh /var/lib/postgresql
<wgrant> bigjools: Only 400MiB at the moment, but I've seen it get to a couple of gigabytes during tests.
<bigjools> wgrant: ok ta - that's annoying, it won't fit in a ramdisk any more :(
<maxb> pgquiles: You're going to find that the open-sourced bits of Launchpad are deliberately only the bits that are developer oriented. Anything that only pertains to production ... like the VCS branches pertaining to what it is exactly ... isn't public.
<pgquiles> maxb: huh? so launchpad has been opensourced in a pretty useless way, hasn't it? :-/
<maxb> It's been opensourced in a way which is perfect for allowing community contributors to enhance launchpad.net
<maxb> That clarifying clause.... was perhaps conveniently omitted from the initial publicity
<maxb> Don't get me wrong, I'm really happy that Canonical went as far as they did, when they didn't have to opensource it at all, but there are still strings attached
<maxb> AFAIK, the best us community people can do to see the sourcecode of a particular production version is to look for the last revision in db-stable before stable->db-devel auto-merging re-commences
<pgquiles> that's what I've done, I've modified the rocketfuel-setup script to install the db-stable branch
<pgquiles> but I was also interested in the buildd configuration branch, which is proprietary :-/
<wgrant> buildd configuration branch?
<maxb> buildd configuration?
<stub> The production branches are private so security fixes can land without exposure. Nothing lands on the production branches that doesn't also land on launchpad/devel or launchpad/db-devel. I don't know about the buildd config branch (including if it exists) - it probably contains keys that can't be exposed but I'm guessing.
<maxb> Ultimately this boils down to : Canonical doesn't want people to run their own internal Launchpad instances, but people see "Launchpad is Open Source", and want to, and are then disappointed when they find out they can't.
<maxb> s/can't/can't without lots of rebranding work and devising their own production-esque configs/
<wgrant> One doesn't have to be too creative to get hold of productionish configs.
<wgrant> But the configs aren't that interesting.
<maxb> I've not really thought much about the second, because the first is a big blocker.
<wgrant> pgquiles: What did you mean by "buildd configuration"?
<pgquiles> wgrant: I want to install PPAs here. There is this project ( http://launchpad.net/lpbuildbot ) which seems to contain interesting stuff, but it's closed source
<wgrant> pgquiles: That's not related.
<wgrant> pgquiles: That's the configuration for the buildbot which runs the Launchpad test suite.
<pgquiles> oh, I see
<pgquiles> thanks
<pgquiles> wgrant: there are no tarballs for releases, are there?
<wgrant> pgquiles: No. It's all deployed through bzr internally.
<pgquiles> so how do you upgrade to the next version (say, 3.1.11) ? bzr pull ?
<wgrant> I don't know.
<stub> You want 3.1.11 at the start of the month or at the end of the month? You generally want the latest lp:~launchpad-pqm/launchpad/db-stable
<maxb> Public bug 368889 references private bug 280578 - could someone have a look at it and determine if it needs to be private, or if it's just a leftover from pre-opensourcing? Thanks.
<mup> Bug #368889: https://code.edge.launchpad.net/~twisted-dev/twisted/trunk reports wrong branch format <branch-puller> <branch-scanner> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/368889>
<pgquiles> ok, I got it working using db-stable (devel seems to have some problem with loom.branch not being found. But db-stable (3.1.10 r8660) works like a charm!
<pgquiles> now reading about soyuz, let's see if I can set up the autobuilders
<Ursinha> allenap, hi
<allenap> Ursinha: hi
<allenap> Ursinha: Whassup?
<Ursinha> allenap, I'm sorting out a problem of bug mail spam a user reported in feedback@
<Ursinha> allenap, he says he's getting email of a bug that's not related to him in launchpad
<pgquiles> it seems that rocketfuel-setup does not install the keyserver ( keyserver.launchpad.dev goes to "page not found" and gpg --send-keys also fails). Do I need to install/configure anything else? (db-stable r8661 here)
<maxb> pgquiles: "make run" only starts a variety of more essential dev services. Investigate "make run_all"
<maxb> NB that the dev keyserver is a minimal shim to make the tests work, not the same software that runs the actual keyserver
<pgquiles> maxb: make run_all doesn't start the keyserver either
<pgquiles> I only need the keyserver for the PPAs
<maxb> pgquiles: Have you seen https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally ?
<pgquiles> maxb: that's exactly what I'm trying to follow
<maxb> pgquiles: Have you actually done start-soyuz.sh?
<maxb> The keyserver is called zeca
<pgquiles> maxb: but it does not work for me. Even make-ubuntu-sane.py is broken, it looks like ubuntu.seriases is an empty list :-? odd
<maxb> Have you run "make schema" ?
<pgquiles> maxb: this is the output of make-ubuntu-sane.py: http://pastie.org/693926
<pgquiles> maxb: yes, after rocketfuel-setup
<pgquiles> the only thing I did different from the instructions at http://dev.launchpad.net is I'm using db-stable instead of devel, and it's because devel fails due to some error with bzr.loom
<maxb> pgquiles: I believe that attribute got renamed recently
<pgquiles> maxb: you don't know the new name, do you?
<maxb> s/serieses/series/, I *think*
<pgquiles> maxb: you think *right* :-)
<pgquiles> thanks!
<pgquiles> I've updated make-ubuntu-sane.py in the wiki
<Ursinha> allenap, hi :) is it possible to unsubscribe and stop receiving bug mail from a bug I've filed?
<allenap> Ursinha: Yes, I think/hope so. You're just added as a direct subscriber I think.
<Ursinha> thanks allenap
<pgquiles> ugh, keyserver still not working :-(
<mrevell> night all
<pgquiles> ah, I found the problem: /var/tmp/zeca did not exist. I created it with proper permissions and now the keyserver works fine.
<pgquiles> I followed https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally to add builders but now I want to remove them. I've tried with "remove" instead of "add" ( scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 remove -f chroot-ubuntu-lucid-i386.tar.bz2 ) but it does nothing. What am I doing wrong?
<thumper> morning
<jkakar> jml: In case you hadn't noticed, I've started some work to make it possible/pleasant to write unit tests for launchpadlib-using applications: https://code.edge.launchpad.net/~jkakar/launchpadlib/testing-support/+merge/14444
<thumper> jkakar: awesome
<jkakar> thumper: It's not quite ready to land, but I thnk it's the right way forward.  Would love to get some reviews/feedback and push it forward (hint, hint). :)
<thumper> :)
<wgrant> Hm. So has it really been decided that assertEqual(expected, observed) is the Right Way?
<thumper> wgrant: that is how the code team does it
<wgrant> How odd, but OK.
<thumper> wgrant: but I don't think there is a "LP approved" way
#launchpad-dev 2009-11-12
<cody-somerville> Running make results in the following error:
<cody-somerville> Error: Couldn't find a distribution for 'lazr.restful==0.9.17'.
<cody-somerville> any clues?
<mwhudson> cody-somerville: bzr update download-cache maybe?
<cody-somerville> I was just going to try that
<cody-somerville> mwhudson, thanks. That seems to have fixed it.
<mwhudson> rocketfuel-get will do that for you
<mwhudson> (but i don't use that)
<cody-somerville> I don't use it either
<mwhudson> i'm sure some people do :)
<maxb> Has anyone floated the idea of making "Policy:  You must be a team member to subscribe to the team mailing list. " configurable in the past?
<thumper> cody-somerville: I have a simple bash script I use for updating stuff
<thumper> cody-somerville: involves cd and bzr pull :)
<maxb> ditto :-)
<maxb> rocketfuel-* is mostly too magic for my liking
<thumper> maxb: my thoughts exactly
<thumper> I prefer a simpler magic
<cody-somerville> I want to interact with an object in launchpad via python shell. Whats the easiest way to go about that?
<mwhudson> cody-somerville: launchpadlib isn't too bad for interactive use
 * thumper agrees with mwhudson
<cody-somerville> this function isn't exposed via the API
<mwhudson> cody-somerville: submit a patch :-)
<cody-somerville> nor should it be exposed <g>
<cody-somerville> or maybe it could be
<maxb> getting launchpadlib going in a python shell is harder than it needs to be
<maxb> unless there are helpers that I'm unaware of
<maxb> bug 425856. I solicit comments :-)
<mup> Bug #425856: non-team members cannot subscribe to launchpad/team mailing list <Launchpad itself:New> <https://launchpad.net/bugs/425856>
<mwhudson> maxb: getting going is a bit of a pita yes
<cody-somerville> mwhudson, thumper: basically, I want to cal an SQL helper method to see what the result is
<mwhudson> cody-somerville: the correct interface for that is "a losa" i think?
<mwhudson> if you can run an irc client in you python terminal, that might work :)
<mwhudson> cody-somerville: or do you mean locally?
<spm> SELECT CAN HAZ CHEEZEBURGER FROM GREASYS
<cody-somerville> :)
<cody-somerville> mwhudson, locally
<wgrant> cody-somerville: 'make harness'
<mwhudson> yes, make harness
<cody-somerville> can I do that when launchpad is running?
<mwhudson> yep
<cody-somerville> thanks
<cody-somerville> Now, whats the easiest way to get the object for a particular bug filed against a source package in a distribution?
<thumper> cody-somerville: using its number
<cody-somerville> Is there some zope magic I invoke against the interface or some helper function?
 * thumper looks
<thumper> getUtility(IBugSet).get(number)
<cody-somerville> thanks
<cody-somerville> I suppose I should refer to the doctests
<cody-somerville> lots of good examples there
<mwhudson> well
<mwhudson> lots of examples, at least
 * ajmitch saw some lovely doctests in thumper's talk :)
<thumper> I didn't read the doctests for that
<thumper> I went to lib/lp/bugs/interfaces/bug.py
<ajmitch> oh it was in the testing discussion, wasn't it?
<thumper> heh
<ajmitch> you showed off branch.txt
<jml> :(
<thumper> jml: it was an example of a bad doctest
<jml> good good :)
<ajmitch> jml: don't worry, he didn't say it was all crap
<mwhudson> branch.txt is very largely crap
<ajmitch> thumper: where can I read about the factory stuff you mentioned earlier?
<thumper> ajmitch: look at any tests in the lp.code area
<thumper> ajmitch: or look in lp.testing.factory
<thumper> ajmitch: we tend to use TestCaseWithFactory from lp.testing
<ajmitch> ok, thanks
<thumper> ajmitch: if you are using a doctest, there is a factory glob object
<jml> factories are cool
<jml> but
<jml> ours is really slow
<jml> and it ought to have tests of its own
<ajmitch> lp.testing.factory does look rather large to wade through
<jml> thumper, you know what would be great
<thumper> jml: unlimited cosmic power?
<jml> thumper, being able to see all the trunk branches I have commit access to
<jml> thumper, what do you think?
<thumper> jml: useful for some people
<thumper> jml: what is your main use case?
<jml> thumper, I want to find my projects.
<jml> thumper, I have quite a few of them now.
<thumper> jml: I'd much rather have a way to personally associate myself with a project
<thumper> jml: that is of more interest to more people
<jml> thumper, I don't get it. You want to enter even more data?
<thumper> yes
<thumper> if
<thumper> I can have a page that just shows me about projects I care about
<thumper> kinda like a "follow" this project
<jml> hmm
<jml> I think I'd still want to be able to see the projects I have commit rights to
<cody-somerville> wgrant, where is the configuration for lucile kept at?
<wgrant> cody-somerville: Which bit?
<cody-somerville> publisher
<wgrant> Some in the DB, some in configs/development/launchpad-lazr.conf, some in configs/schema-lazr.conf
<wgrant> And the occasional bit hardcoded.
<wgrant> https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally is my documentation on setting the whole thing up.
<thumper> jml: where would you put the link?
<jml> thumper, why not make it part of the personal code page
<thumper> could do...
<thumper> patches accepted
<ajmitch> jml: I suppose that you could have write access to quite a few projects indirectly through teams?
<jml> ajmitch, yes.
<jml> thumper, I'm actually still waiting on a chance to get my other patches landed.
<jml> thumper, can you please review upload-permission-joy?
<jml> how do I expose a function over the API that takes a list of URLs as a parameter?
<thumper> rockstar: how do I assert that an xpath doesn't exist in a windmill test?
<Ursinha> hey wgrant :)
<wgrant> Hi Ursinha.
<Ursinha> my lp setup still wants to use the 0.9.9 version of lazr.restfulclient when I have the 0.9.10 inside the download-cache folder
<Ursinha> what do I have to do to lp use the newest?
<wgrant> Merge from devel.
<wgrant> (or alter versions.cfg manually)
<Ursinha> thanks wgrant :) I'd like to know which file defined that
<wgrant> You might also have to run bin/buildout again, I guess.
<jml> yay
<MTecknology> Where's the bestest best best place to get help on LP API; right here?
<thumper> MTecknology: yep
<MTecknology> thumper: How do I get my hands on it so I can use it? I feel like I'm about to dive into a sea of sharks with this but my company wants two of us to learn all we can about it
<thumper> MTecknology: to what end?
<MTecknology> One reason is because we want to modify some core tools we use to utilize the openid+login process of Launchpad so the use of LP is entirely transparent to the user. We're also looking into our own server to host the LP server
<MTecknology> If we can get that, then we're looking into using it for a support ticket system and other things
<thumper> MTecknology: the project is launchpadlib, and I believe it has good docs
<jml> next question
<jml> how can I expose a method that returns a dict?
<jml> mapping URLs to IBranch
<wgrant> jml: Give up, I'm afraid :(
<jml> wgrant, that sucks.
<wgrant> It does.
<jml> wgrant, I guess a list will work ok in this context though.
<wgrant> You can return a dict, but launchpadlib will just see it as a dict of URLs to URLs.
<MTecknology> thumper: thanks. I was trying to figure out how we could do everything that's being planned and it hit me that Launchpad can handle every piece of that
<wgrant> MTecknology: I don't see what the LP API has to do with this.
<thumper> jml: I think leonard is working on exposing a dict over ReST
<jml> thumper, that's good.
 * thumper throws hands in the air
<thumper> how can XPath have "and" and "or" but not "not"?
<MTecknology> wgrant: I don't really understand what the api is; but I was told to get it so we can see if it will help us
<jml> MTecknology, https://help.launchpad.net/API
<jml> kfogel, hi
<kfogel> jml: hey
<jml> kfogel, how's things?
<kfogel> jml: well, I discovered halfway through the day that today was a U.S. holiday, that was interesting :-).  How's the sprint going?
<jml> kfogel, good.
<jml> kfogel, less coding than I had hoped, but probably the better for it.
<kfogel> jml: ah, too bad, but that's what happens when you know too much
<poolie> hello
<poolie> nice reply there kfogel
<poolie> it's thanksgiving isn't it?
<kfogel> poolie: thanks.  I was wondering if it was too harsh; glad you didn't find it so.  It's not Thanksgiving yet, btw, but it is Veterans Day in the US.
<thumper> poolie: veterans day
<poolie> oh right
<thumper> thanksgiving is in two weeks
<poolie> armistice day is observed here but not a holiday
 * kfogel is glad the whole world knows US holidays, and wonders if it has something to do with all our missiles
<Ursinha> lol
<Ursinha> jml, do you know which commit message ec2 script considers when using the land command?
<jml> Ursinha, the one on the merge proposal page
<thumper> kfogel: no, just approved rockstar's holidays, that's all
<jml> Ursinha, you can provide one as an option to ec2 land
<kfogel> thumper: heh
<thumper> Ursinha: but don't add all the [xx] rubbish to that commit message :)
<Ursinha> thumper, oops, why?
<thumper> Ursinha: because the land commands adds them
<thumper> Ursinha: based on who reviews/approves, bugs et al.
<Ursinha> thumper, makes sense
<jml> Ursinha, --dry-run ftw
<Ursinha> jml, :)
<maxb> Speaking of the [xx] rubbish, is there a reference to what it all means anywhere?
<jml> maxb, if it's not on the dev wiki, then probably not.
<jml> maxb, the source code for ec2 land might actually have pointers to docs :)
<maxb> not that I've been able to find
<wgrant> The only one that was unobvious to me was rs.
<jml> maxb, in that case, I won't be able to help you find docs, since I looked for docs when I wrote land :)
<maxb> Well, yeah, I think I've figured most of them out by example by now. But [!log] still mystifies me (seen in some sourcecode branch I forget which)
<jml> maxb, don't include it in the news announcement for the release.
<jml> maxb, we used to use it to make preparing announcements easier. We don't any more.
<kfogel> poolie: nice response -- I didn't mean to guilt-trip you, though! :-)
<poolie> not at all
<jml> I am feeling unhappy
<jml> and it is because of software
<jml> again
<jml> If I return a list of branches over the API, but some of those branches are None, then the 'None' objects don't appear in the list.
<jml> AFAICT, this means that helper methods like IBranchSet.getByUrls are impossible
<thumper> jml: and there isn't a map?
<jml> thumper, there isn't.
<jml> I'm going to have to think of a different example for my UDS session.
<kfogel> jml: is there no way to match up the returned branches to the URL(s) originally requested?
<jml> kfogel, not easily.
<jml> kfogel, there are multiple attributes on branch that the URL might be matched on.
<kfogel> jml: oh
<jml> and some that aren't even attributes
<jml> although matching against the returned bzr_identity, then matching against the returned branch.url and then falling back to matching the returned unique_name against the requested URL path would _probably_ get you most of the way there
<jml> https://bugs.edge.launchpad.net/lazr.restful/+bug/481090
<mup> Bug #481090: Cannot define a method that returns a dict <lazr.restful:New> <https://launchpad.net/bugs/481090>
 * thumper ndos
<PingJocky> good evening
<thumper> afternoon :)
<MTecknology> Can the LP API allow users to manage teams/users if the person using the API has that permission?
<MTecknology> I know the API is used to pull info, not if it can pushing or altering
<MTecknology> nevermind - I guess if I pay attention the first page kinda says I can do that
<maxb> and the api documentation says exactly what can be done
<MTecknology> maxb: I'm still looking aroudn for that. I don't refer to documentation much so reading through it is hard work for mew
<maxb> https://launchpad.net/+apidoc/
<MTecknology> maxb: :D
<MTecknology> thanks
<jml> hey, people here know stuff
<jml> I'm trying to prepare a USB stick for UDS that I can use to accelerate people having a Launchpad development environment
<jml> my plan is to set up a minimal chroot, run rocketfuel-setup, make sure everything works, and then whack it all in a tarball
<jml> does that sound sane?
<wgrant> Awesome idea.
<wgrant> As long as you rerun rocketfuel-setup at the other end, that does sound sane.
<jml> well, I'm also not 100% clear on what people on the other end will need to do to get the chroot.
<wgrant> Oh, you're going to copy the entire chroot? Hmm.
<jml> that was my original plan
<wgrant> That'll run into postgres port conflicts and other mess if they already have postgres installed.
<jml> oh yeah
<wgrant> Better to make rocketfuel-setup make less of a mess, so you don't really need a chroot.
<jml> wgrant, I'm not going to have time to do that before UDS :)
<wgrant> Plus even if you do have a chroot, you still need to modify /etc/hosts outside the chroot.
<jml> so maybe just having the repo on a USB stick
<wgrant> + sourcecode
<jml> right.
<jml> and make sure that rf-setup can be run in such a way as to respect the existing file
<jml> s
<wgrant> Get a clean ~/launchpad, remove devel from it, but keep a copy of rocketfuel-setup in there.
<wgrant> It will already.
<wgrant> If you extract the tarball into ~/launchpad as it expects, it will Just Work Very Quickly.
<thumper> jml: what about a vm on a stick?
<jml> thumper, the problem is then people need to know how to get a vm set up
<jml> thumper, it's all about taking the smallest amount of time as possible so I can get on with the rest of the session.
<thumper> jml: people need to know how to get a chroot set up too
<thumper> jml: just a thought
<jml> thumper, well yeah, that's why I'm not doing it :)
<wgrant> If you're expecting quite a few people to set it up, it might be a good idea to remove some of the crap from rf-setup this week.
<wgrant> eg. the SSH config stuff is no longer necessary.
<jml> wgrant, maybe
<jml> wgrant, are there bugs filed about all of this?
<wgrant> jml: Bug #419028
<mup> Bug #419028: Don't edit SSH config in rocketfuel-setup <build-infrastructure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/419028>
<jml> wgrant, know of anything else that ought to be removed / cleaned up that doesn't require much thinking?
<wgrant> jml: I really feel that it's unnecessary to install rocketfuel-* in $PATH, but that's a slightly more destructive change.
<jml> I agree on both counts.
<wgrant> OTOH, it won't affect any existing installations.
<wgrant> Apart from that, rf-setup isn't too bad.
<wgrant> launchpad-database-setup, OTOH...
<jml> wgrant, http://bazaar.launchpad.net/~jml/launchpad/rf-setup-cleanup/revision/9863
<wgrant> jml: Great.
<wgrant> That probably makes it noninteractive, too.
<wgrant> Hm, not quite.
<wgrant> But close enough.
 * mwhudson_ reviewatrons
<wgrant> Is there no link back from Loggerhead to the branch in LP?
<wgrant> mwhudson_: Nice review comment, but your diff highlighter sucks.
<mwhudson_> wgrant: there is a link in the top left
<mwhudson_> wgrant: "watches pelcome" or something
<wgrant> Ah, so there is.
<wgrant> Oh, right, it assumes it's a header.
<adeuring> good morning
<mrevell> Morning
<wgrant> bigjools: I don't see what your SourcePackageFormatSet methods do.
<bigjools> wgrant: ok I'll explain in a moment
 * thumper has many branches waiting for review
 * thumper has fixed two oopses today
<thumper> my work here is done
 * thumper goes to wash disher
<thumper> s/disher/dishes/
<thumper> it's late
<bigjools> wgrant: ok let's chat
<bigjools> so, SourcePackageFormatSet would be a utility
<bigjools> which is security wrapped
<bigjools> we don't generally use __init__ methods in model classes
<bigjools> the utility has the job of creating new instances
<bigjools> so that it returns security-wrapped objects
<pgquiles> I followed https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally to add builders but now I want to remove them. I've tried with "remove" instead of "add" ( scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 remove -f chroot-ubuntu-lucid-i386.tar.bz2 ) but it does nothing. What am I doing wrong?
<pgquiles> (sorry for repeating the question, noone answered yesterday and I haven't found a solution yet)
<bigjools> pgquiles: that command deals with chroots, not builders
<wgrant> bigjools: Right, yes. So it would be add(distroseries, format), and then there would be something like get(distroseries, format)?
<wgrant> pgquiles: What did you expect it to do?
<bigjools> wgrant: yep, pretty much, although not get(), getBySeriesAndFormat()
<bigjools> wgrant: get() is normally used for getting by ID
<wgrant> bigjools: Right. Thanks.
<wgrant> bigjools: Now, what did you mean about the XXX?
<wgrant> bigjools: There were some words omitted.
<bigjools> wgrant: I read it back and it made sense to me, so maybe it's just my twisted English :)  Basically, I would say don't worry about it and remove the XXX
<pgquiles> wgrant: I want to remove a builder from launchpad.dev/builders, and I want to remove a chroot because I did manage-chroot.py -s karmic -a i386 add -f chroot-ubuntu-lucid-i386.tar.bz2 (I told LP to add a Lucid chroot for a Karmic series, my bad)
<bigjools> pgquiles: mark it as manual and it disappears
<wgrant> bigjools: "I'm trying to think if it would be bad to reject the upload if we detect orphan files, and I can't think of one." <- one what? reason to reject, I presume.
<bigjools> wgrant: yes
<bigjools> wgrant: my twisted English, sorry
<wgrant> pgquiles: You actually want to mark the build as inactive, or something like that.
<wgrant> Not manual -- that will still show up.
<wgrant> pgquiles: You're sure that manage-chroot.py remove does nothing?
<wgrant> bigjools: Thanks. Fixing now.
<wgrant> I've never had cause to remove a chroot myself.
<pgquiles> wgrant: I don't know if it does anything but builders are still there
<pgquiles> even worse: when I try to add a new chroot, it adds it but it does not show in LP :-/
<wgrant> pgquiles: Chroots are not shown in LP.
<pgquiles> wgrant: the launchpad.dev/builders/bob/+edit I mean
<wgrant> pgquiles: That has nothing to do with any chroot.
<pgquiles> Â¿?
<wgrant> pgquiles: A builder is it a builder. Its builds will use chroots, but it isn't directly related to them.
<pgquiles> according to https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally you have to add a chroot, then mark Bob the Builder as OK
<wgrant> Those steps are not related beyond both being prerequisites for a working build system.
<bigjools> ah yes mark it as inactive to remove it from /builders
<pgquiles> I don't understand anything :-( After I added a chroot, it was shown in the builders list and I could mark it OK and it worked fine
<wgrant> The existing builder is shown in the builders list.
<wgrant> Not the chroot.
<pgquiles> what I want to do is remove a chroot (tried with 'remove' but it does nothing) and add a new chroot for jaunty (but after the 'remove' for lucid, it no longer works)
<pgquiles> bigjools: it will not show the builder in /builders but it's still in LP, you can re-enable it again
<pgquiles> wgrant: do you mean the builder is there but with no chroot?
<bigjools> pgquiles: that's correct
<pgquiles> ok, so, how do I completely remove a builder?
<pgquiles> as in "this builder no longer has a chroot associated, I don't want it to be shown, even as inactive"
<wgrant> Builders are not associated with chroots.
<wgrant> You cannot completely remove a builder without removing lots of other sample data.
<pgquiles> mmm so I add chroots to launchpad
<pgquiles> and then I add builders
<pgquiles> and builders will know what chroot to pick when a package needs to be built
<pgquiles> is that it?
<wgrant> Yes.
<pgquiles> great :-) so builders are smarter than what I thought (and definitely smarter than I :-) )
<pgquiles> then, to fix my bad "manage-chroot.py -s karmic -a i386 remove -f chroot-ubuntu-lucid-i386.tar.bz2", can I just run "manage-chroot.py -s lucid -a i386 remove -f chroot-ubuntu-lucid-i386.tar.bz2" and that's it?
<wgrant> bigjools: SourcePackageFormatSet, or should it be SourcePackageFormatSelectionSet?
<bigjools> wgrant: the latter
<bigjools> bit of a mouthful :/
<wgrant> There's worse.
<pgquiles> one more question: virtualized builders. Are there any docs on how to set up one? (I can't find anything)
<bigjools> no, there are no docs
<pgquiles> and I guess it's not exactly easy, is it?
<bigjools> depends on how much you know about virtualisation
<pgquiles> I know virtualization but I've never set up Xen (which is what LP uses IIRC). Can it use something else (VMware, VBox, qemu, KVM) ?
<bigjools> you can use what you like, it's nothing to do with LP
<pgquiles> great
<bigjools> exactly what are you setting up LP for?
<pgquiles> for internal use at work
<pgquiles> we are developing our own distribution for military use, based on kubuntu + packages for our software and backports from lucid, debian unstable and some stuff I've packaged and it's not available from debian or ubuntu
<pgquiles> I think I'm the first one who requested to buy Launchpad for on site use but it was not available at the time (2+ years ago)
<pgquiles> given that it's military use and our software is closed source, hosted LP commercial is not an option
<maxb> I wonder how long it will be before enough people start wanting to rebrand launchpad that we actually end up with launchpad hosting projects/branches for a rebranding effort of itself
<bigjools> I find someone internally using an open-source LP to host closed-source stuff extremely distasteful
<pgquiles> bigjools: does Canonical sell a LP appliance, training, whatever? I've tried to BUY that but I was told (by Mark himself) Canonical does not sell that still
<pgquiles> and I cannot upload military stuff to somewhere else, no matter how much security you promise
<maxb> bigjools: why?
<maxb> The way I see it, using Launchpad in that way takes nothing away from the Launchpad project, and might be the source of bug reports, fixes, and an interested contributor
<pgquiles> I've fixed a missing quote in HowToUseSoyuzLocally and added the explanation on the non-association between builders and chroots, which was not obvious from reading the page
<wgrant> It was a hasty conversion last week from a Tomboy note; I'm not surprised I missed a quote.
<wgrant> Was the attachment change this morning just s/serieses/series/?
<pgquiles> wgrant: yes, that was it -  maxb told me what was failing
<pgquiles> HowToUseSoyuzLocally says I should add an external dependency on "deb http://archive.ubuntu.com/ubuntu %(series) main restricted universe multiverse" to the PPA but when I try to do that from "add dependencies" in the PPA, LP fails. Is that the right place? Is that the line I should enter in the "Add PPA dependency" text entry?
<maxb> "# We use Hoary as the root, as Breezy and Grumpy are broken." ....... Grumpy? :-)
<wgrant> Grumpy Groundhog
<wgrant> The long-fabled bleeding edge daily builds Ubuntu.
<wgrant> Which somehow made it into the sample data.
<pgquiles> mmm one little trick in /etc/hosts and now archive.launchpad.dev goes to archive.ubuntu.com... and it's building!
<wgrant> pgquiles: If you followed my instructions, you shouldn't need to do that.
<pgquiles> wgrant: I followed your instructions but I don't know how to add the external dependency on archive.ubuntu.com, adding it in the "Add PPA dependencies" page in LP does not work (Zope error) :-?
<wgrant> pgquiles: See the field at the bottom of the PPA's +admin page.
<bigjools> wgrant: heh, so that's why you said it was handy
<pgquiles> wgrant: ah! that was it
<pgquiles> wgrant: I thought I should add it to the "Edit PPA dependencies" page
<pgquiles> thanks!
<wgrant> bigjools: Yeah, it's particularly convenient for testing primary archive stuff.
<wgrant> bigjools: If I'm just dealing with PPAs, I just change archivepublisher.base_url to a local mirror.
<bigjools> yeah
<pgquiles> added to the wiki
<wgrant> pgquiles: Note that that same thing is mentioned two steps earlier.
<pgquiles> oops, right
<wgrant> pgquiles: That looks better.
<pgquiles> wgrant: sorry if this is a stupid question but, the section about "dealing with the primary archive", does it refer to a local mirror of archive.ubuntu.com (i. e. archive.launchpad.dev) ? the steps listed there, are they for adding PPA-built packages to archive.launchpad.dev?
<wgrant> pgquiles: The primary archive is archive.launchpad.dev. But it's nothing about copying in PPA packages.
<pgquiles> so those steps are for building the packages and adding them to archive.launchpad.dev *instead* of a PPA, is that it?
<wgrant> Yes.
<pgquiles> perfect, thank you
<pgquiles> now looking into setting up virtualized builders!
<wgrant> bigjools: Want an intermediate diff?
<bigjools> wgrant: yes please
<pgquiles> wgrant: btw, /var/tmp/zeca is not created in the by rocketfuel_setup/start_soyuz.sh/whatever
<wgrant> pgquiles: Ah, yes, forgot about that.
<wgrant> bigjools: What do you think about running "UPDATE sourcepackagerelease SET dsc_format='1.0' WHERE dsc_format IS NULL"? No new broken SPRs have been created since 2006, so it should be a safe one-off fix.
<bigjools> wgrant: yes, sounds sane
<matsubara> Chex, gary_poster, rockstar, bigjools, danilos, citrus, allenap: LP production meeting in 15 min @ #launchpad-meeting
<gary_poster> danilos: do we have team lead call in 15, as Google lists?
<danilos> gary_poster, no, as per email yesterday
<danilos> matsubara, thanks
<gary_poster> danilos: oh, must have missed it in the email swamp.  Looking...
<gary_poster> found, and thanks
<sinzui> salgado:  ping
<salgado> hi sinzui
<sinzui> salgado:  I am working on bug 471770. There is a conflict between AdminMergeBaseView.doMerge() which sets the dup email address to NEW and PersonSert._mergeMailingListSubscriptions which moves the subscription (with a NEW email) to the new acount
<mup> Bug #471770: mailing list subscription email address are in NEW status <oops> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/471770>
<sinzui> salgado: I think I should change PersonSert._mergeMailingListSubscriptions to remove all the subscriptions and send an email to the user that he has to re-validate his address *and* resubscribe...
<sinzui> salgado: but maybe I should change the subscriptions to the preferred email address, or ensure that the the affect email addresses are VALIDATED
<pgquiles> wgrant: ping
<salgado> sinzui, I think I'd rather remove the subscriptions and send out emails for the user to re-validate the address and subscribe
<sinzui> salgado: thank you. That is what I will do.
<mrevell> night all
<Ursinha> gmb, ping
<thumper> any one around who can review?
 * thumper needs some reviewers
<wgrant> pgquiles: Hi.
<pgquiles> wgrant: hey. I was wondering why the instructions to use Soyuz locally tell you to download a pre-built bootstrap (with all the steps and guessings that involves) instead of telling how to build a bootstrap of your own with debootstrap
<pgquiles> in the meanwhile, I've added how to use debootstrap to the wiki
<pgquiles> is there anything special in the bootstraps from launchpadlibrarian.net ?
<wgrant> pgquiles: The production chroots have various modifications (see /root/bootstrap), and it's so much quicker and easier.
<wgrant> Plus it is better at convincing me that my full stack changes will work in production.
<pgquiles> wgrant: mmm I see. So, how are the chroots from launchpadlibrarian.net generated? debootstrap + manual steps?
<wgrant> pgquiles: I presume debootstrap + copy in that bootstrap script + run that bootstrap script.
<wgrant> But I don't exactly know.
<pgquiles> wgrant: should I ask in the mailing list?
<wgrant> pgquiles: Launchpad developers will not know. It's not really a Launchpad question.
<pgquiles> oh
<pgquiles> who can I ask?
<maxb> pgquiles: lamont. I'd be interested in the answer.
<pgquiles> maxb: is he/she on IRC usually (meaning: tomorrow) ? or should I ask by e-mail?
<maxb> He's certainly sometimes on IRC. LP says he's America/Denver TZ. But for a complex question, email might be best
<pgquiles> perfect, thanks
<rockstar> maxb, I think he's EOD'd already.
<pgquiles> maxb: he's on #debian-devel at OFTC
<pgquiles> EOD?
<rockstar> End of day.
<pgquiles> oh
<maxb> pgquiles: and also on a variety of #ubuntu-* on freenode
<pgquiles> I was thinking Esoteric Order of Dragon
<pgquiles> :-)
<pgquiles> off to bed, it's late here
<pgquiles> thank you all, see you tomorrow!
#launchpad-dev 2009-11-13
<pgquiles> maxb: I was still around. debootstrap --variant=buildd, then running the root/bootstrap script almost gets it. I needed to modify the script to add 'universe' to etc/apt/sources.list and to create user and group buildd. I'm still missing 3-4 .conf files but I'll continue tomorrow. I've e-mailed lamont my findings and waiting for his answer.
<wgrant> pgquiles: LP will override sources.list; you shouldn't need to add anything to it.
<maxb> pgquiles: thanks. looking forward to what you hear
<pgquiles> wgrant: I only needed to add that to the bootstrap script for installing policyrd-script-zg2, which is in universe. The bootstrap script removes 'universe' after that (that was already in the script, I'm guessing there is some other script I'm missing)
<wgrant> pgquiles: Ah, I see.
<pgquiles> wgrant: my modified bootstrap: http://www.elpauer.org/tmp/bootstrap
<wgrant> pgquiles: You can probably give deboostrap an argument to do that for you.
<pgquiles> mmm maybe --components=universe, I'll try tomorrow
<pgquiles> now really off to bed :-)
 * thumper needs JS help
<thumper> rockstar: ping?
<thumper> mars: ping?
<thumper> ProtocolErrorException: Protocol error: 405
<thumper> WTF?
<thumper> anyone got any clues?
<thumper> it is an XMLHTTPRequest
<cody-somerville> Method not allowed
<thumper> but why?
<thumper> it is going from bugs.launchpad.dev -> code.launchpad.dev
<thumper> is that allowed?
<thumper> if not, I'm going to have to reconsider an approach
<cody-somerville> did you use put or post?
<thumper> GET
<thumper> ah
<thumper> no
<thumper> WTF?
<thumper> OPTIONS
<thumper> why is it asking for options?
<thumper> dumb yui
<thumper> tracing NET with firebug causes firefox to hang for ages on each refresh for me
<thumper> :(
<ajmitch> bugs.launchpad.dev -> code.launchpad.dev might be considered cross-domain & forbidden, I'm not sure on the rules for how that is set
<cody-somerville> yui should provide a way around that
 * thumper sighs
<wgrant> That counts as cross-domain.
<wgrant> You cannot work around it.
<wgrant> But why are you wanting to do that? The API is the same on all vhosts.
<thumper> wgrant: I was just fetching an URL
<thumper> wgrant: obviously I need to change that
<thumper> :(
<thumper> clucking bell
<wgrant> What are you trying to do?
<thumper> wgrant: show more details about the merge proposal on the bugs page
<thumper> which includes a link to the diff
<thumper> which will popup on the bugs page
<thumper> got it working on branches
<wgrant> Ah, I see.
<thumper> it just uses Y.io
<thumper> but doesn't work cross-domain
<thumper> also
<thumper> I really wanted some custom tags
<thumper> but that is HTML5
<thumper> well, custom attributes that is
<thumper> beuno: I can haz HTML5 plz
<cody-somerville> thumper, http://developer.yahoo.com/javascript/howto-proxy.html
<cody-somerville> cross-domain is easy to work around using a server side proxy
<wgrant> But that is crazy.
<cody-somerville> no its not
<cody-somerville> its the accepted best practice
<thumper> cody-somerville: I think the API is the way to go in this case
<cody-somerville> almost all javascript libraries come with support for it
<wgrant> In this case the target is controlled by thumper too.
<wgrant> So proxying it is just foul and ugly.
 * cody-somerville agrees.
<cody-somerville> wgrant, are you using ec2 btw?
 * cody-somerville is in the process of signing up.
<wgrant> cody-somerville: I've only used it a few times.
<wgrant> Stuffing around + a couple of ec2tests.
<ajmitch> how much does it cost to run through the test suite?
<wgrant> A couple of dollars, I'd say.
<cody-somerville> really?
<cody-somerville> how long does it take to run?
<wgrant> 3 or 4 hours on a high-CPU instance.
<wgrant> IIRC.
<wgrant> Or was it 2.5 hours.
<wgrant> I don't remember.
<thumper> can anyone recommend a good recursive diff?
<thumper> for comparing trees
<thumper> nm
<wgrant> diff -r?
<cody-somerville> hmm... I get ImportError: No module named _pythonpath when I try to run ec2 test
<thumper> damn
<thumper> must be friday
<ajmitch> friday & nearly beer o'clock
<cody-somerville> Its only Thursday here :(
 * thumper wants to put his fist through the laptop right now
<thumper> how do you get an attribute of an object using the xhtml representation?
<wgrant> I doubt that you can; the XHTML representation is meant to be customised.
<thumper> wgrant: yeah, I'm emailing the dev list
<thumper> I think it is possible, I just don't know how
<thumper> email sent
<thumper> beer o'clock
<ajmitch> the magic words have been spoken
<jml> hey
<jml> my patch got rejected because we are in testfix mode
<jml> wtf
<adeuring> good morning
<mwhudson> bigjools: gee thanks :)
<bigjools> mwhudson: don't mention it :)
 * bigjools notices it's Friday 13th
<poolie> hello jml!
<poolie> i've already run rocketfuel-setup on this machine
<jml> poolie, oh
<poolie> so i don't know if this is a good test
<jml> poolie, hmm.
<jml> poolie, thanks.
 * jml has to re-do the whole thing anyway
<wgrant> jml: Which technique did you end up going with?
<jml> wgrant, a tarball with the repo, sourcecode and download-cache
<jml> wgrant, the downside is that it requires 250MB-300MB of package data.
<wgrant> jml: Either tar up /var/cache/apt/archives, or call apt-get with '-o Apt::Install-Recommends=no'
<wgrant> jml: By excluding Recommends, you omit TeX and lots of other big stuff.
<jml> wgrant, if I did archives, people would just copy stuff off the USB stick to their archives/ dir?
<wgrant> jml: Correct. apt will see it there and use it.
<jml> wgrant, will the relevant packages still be on my machine?
<jml> I installed LP an awfully long time ago
<wgrant> jml: Probably not. Best to do it in a chroot, anyway.
<jml> or maybe I'll just rely on there being a mirror at UDS
<jml> hmm.
<wgrant> Possibly.
<wgrant> But who knows how bad it will be this time.
<jml> I admire your cynicism.
<jml> the thing is I'll need 64 & 32 bit packages anyway.
<mwhudson> start at http://archive.ubuntu.com and get clicking
<jml> :(
<wgrant> jml: Without recommends, it needs only ~100MB.
<mwhudson> wgrant: how likely do you think it is we actually depend on a recommend ?
<jml> wgrant, ok. I'll patch rocketfuel-setup to do that
<wgrant> mwhudson: Not at all. I run without them.
<mwhudson> wgrant: oh good
<wgrant> http://paste.ubuntu.com/317645/ is the list of skipped packages. Only questionable things there are pyflakes and python-docutils, and the former is definitely optional.
<wgrant> And I'm not sure the latter is used at all.
<mwhudson> pyflakes is used by 'make lint' i guess
<mwhudson> docutils, hm
 * wgrant removes it and tries.
<mwhudson> it's in versions.cfg
<mwhudson> though heavens knows why
<wgrant> Ah, yes, it's an egg.
<mwhudson> and scripts/_ginalog.py but well
<wgrant> So we don't need the package.
<mwhudson> oh right yes
<mwhudson> we also don't need the egg i would guess
<wgrant> WTF is ginalog!?
 * wgrant greps.
<mwhudson> oh right docutils is used in generating the apidocs
<jml> which is obviously a critical thing to build in every single branch
<mwhudson> clearly
<pgquiles> I am supposed to create my own production configuration, disabling devmode, etc, right? or is there a make run_all_production I'm not seeing?
<jml> hello
<jml> I have fixed the test failure.
<stub> jml: It hasn't got to buildbot yet
<wgrant> Does a testfix immediately revert the testfix state, or must buildbot run the test suite first?
 * jml doesn't know
<mwhudson> something in between
<stub> I believe it reverts immediately
<mwhudson> it reverts when buildbot-poll sees the change
<mwhudson> and it runs out of a */5 cron i think
<mwhudson> this is explained https://dev.launchpad.net/Trunk/Glue
<stub> Ahh... its still with pqm
<mrevell> hi
<sinzui> danilos: ping
<danilos> sinzui, hi
<sinzui> danilos: I am looking at https://lpbuildbot.canonical.com/builders/db_lp/builds/244/steps/shell_7/logs/summary I cannot reproduce this failure on my system or in ec2.
<sinzui> danilos: The failing line was
<sinzui>         self.assertEqual(
<sinzui>             self._getReviewables(no_older_than=self.base_time), [self.pofile]
<danilos> sinzui, let me take a look in the test
<sinzui> danilos: ^ What do you think, Could this be a time issue in the test suite?
<danilos> sinzui, it could be, but I believe the times used in the test are in days, so highly unlikely, let me check
<sinzui> danilos: I am inclined to force a build. I think the failure is spurious. I am not sure in this case though if I should disable the test. I wonder if the issue was really somewhere else in the suite.
<danilos> sinzui, actually, it doesn't, it uses one millisecond, but I don't see how/why this should fail anyway... it writes the dates directly as two values where one is base_time + timedelta(0,0,1)
<sinzui> Can I change it to an hour?
<danilos> sinzui, sure
<sinzui> danilos: I changed the test to an hour
<sinzui>     later_time = self.base_time + timedelta(0, 3600)
<sinzui> and it still passes. I want to land this change
<danilos> sinzui, I suspect this is more likely related to commits than time resolution, but if you see further problems with it, please file a bug and I'll look into it shortly
<sinzui> I doubt there will be further problems.
<mwhudson> allenap: i will reply to your your reply to my mail
<mwhudson> allenap: but not for a little while :/
<allenap> mwhudson: Thanks, I look forward to it :)
<mars> mrevell, ping
<mrevell> hi mars
<mars> hi mrevell, just looking at the styling issue
<mrevell> ah yes
<mars> here, I'll send you a screenshot from my system
<mars> BjornT_, ping, looking at mrevell's popup help styling issue.  Looks like the YUI folder is not being linked in devel?
<mars> BjornT_, visit https://edge.launchpad.net/+help/openpgp-keys.html, check the CSS stylesheets - they are returning 404s
<mars> so, what landed on devel to cause that...
<mars> BjornT_, I suspect it broke recently: my yui-3final-upgrade branch is styled properly, so either a) it broke last night, since I pulled from trunk, or b) it broke in devel a while ago, and my branch fixes it
<pgquiles> maxb: wrt chroots, more or less answered in #debian-devel
<mars> BjornT_, looks like the CSS URLs were broken before the sprint.  Your build system changes maybe?
<BjornT_> mars: maybe. could be that the 'current' symlink didn't get included with my changes
<BjornT_> mars: a quick fix would be to change the css link to point to 3.0.0pr2 instead of current, which probably makes sense anyway
<BjornT_> mars: this url works: https://edge.launchpad.net/+icing/yui/3.0.0pr2/build/cssreset/reset.css
<mars> BjornT_, ok.  I'll see if I can identify where the CSS URL is generated
<mars> and change it
<BjornT_> mars: hmm, might be some other problem. i merged in the latest yui-3final-upgrade and lazr-js trunk into my branches, and i get similar errors for js files
<mars> BjornT_, oh?  I tried my branch here, and the style links were fine. Could be an earlier build artifact though
<BjornT_> mars: could be a lazr-js change then. are you using latest trunk?
<mars> BjornT_, I tried it with the latest trunk, and it was broken.  I also tried it with trunk/ from last Friday, still broken.
<BjornT_> mars: oh wait, looks like i'm using an old egg, and not my dev egg...
<BjornT_> mars: that's a bug in the build system. bin/jsbuild doesn't update symlinks. it only checks whether there is a symlink, not that it's pointing to the right location
<mars> BjornT_, should 'make clean' or 'make clean_js' handle that then?
<BjornT_> mars: wel, make clean should remove the build directory, but bin/jsbuild should be able to update an existing build dir
<mars> BjornT_, check the upgrade diff, line 40: http://pastebin.ubuntu.com/317110/
<mars> BjornT_, that could be extended.  'make clean' calls 'make clean_js' as well.
<mars> I actually wish that target was called 'jsclean', not 'clean_js', but we already have a 'clean_code' target in there.
<BjornT_> mars: right, the Makefile changes look good
<BjornT_> mars: bin/jsbuild should still be fixed, though, since people usually doesn't do 'make clean'
<mars> BjornT_, good point
<mars> you know, if we had a 'make update' target that did run 'make clean' for you... :D
<BjornT_> mars: i only run 'make' and 'make run', and i wouldn't want any of them to automatically run 'make clean' :)
<mars> BjornT_, why not?  If 'make clean' does the right thing, then it should only clean up generated build artifacts.  Nothing you hand-edited should be affected.
<BjornT_> mars: run. but i see you don't build lp often :) it takes a long time building after a 'make clean'...
<mars> that is true
<Ursinha> hi sinzui, I'm trying to go to https://code.edge.launchpad.net/startup-manager/trunk/+addbranch and am getting a similar oops as the one when trying to subscribe to bugmail
<Ursinha> sinzui, is the problem related? https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1410G2138
<sinzui> I am not certain
<sinzui> Ursinha: I am logged in so I should not see the message
<sinzui> Ursinha: how did you get to the link? I do not see it
<Ursinha> sinzui, https://edge.launchpad.net/startup-manager/trunk, click on Submit code
<sinzui> Launchpad is crack for showing overview pages on the code hsot
<sinzui> ah, from the involvement menu
<sinzui> That is very difference
<sinzui> The error is true. you submit code to a project, not a series.
<sinzui> That is why we are seeing a 404
<Ursinha> sinzui, bug 482256
<mup> Bug #482256: 'Submit code' OOPSes in the project series overview page <oops> <Launchpad Registry:New> <https://launchpad.net/bugs/482256>
<mrevell> Okay, night all. See you in Dallas if you're gonna be there.
<Ursinha> hey rockstar, are you there?
<rockstar> Ursinha, hi
<rockstar> Ursinha, what's up?
<Ursinha> rockstar, I'm trying to remember :)
<rockstar> Ursinha, okay.  Well, we're back from lunch, but deryck and I are going to pair for a bit, so I may have to ignore you.
<EdwinGrubbs2> mars: ping
<mars> hi EdwinGrubbs2
<thumper> rockstar: you around?
<rockstar> thumper, I am.
<thumper> rockstar: I emailed my javascript frustrations to launchpad-dev
<rockstar> thumper, yes, I read it.
 * mars has not gotten to it yet :(
<thumper> rockstar: intellectronica answered some, but didn't really tell me much new (except the hidden <a> which I'll probably use)
<thumper> rockstar: 2 remaining items
<intellectronica> thumper: what else is still unclear?
<EdwinGrubbs> mars: I've been working on getting some lazr-js tests pass in IE. hasAttribute('tabIndex') always returns true in IE, but getAttribute('tabIndex') returns 32768, so I'm thinking of changing the test to pass if one of those is correct. Do you have any better ideas?
<thumper> rockstar: *how* to I use LP lib to get the preview_diff attribute of the MP as xhtml?
<thumper> rockstar, intellectronica: and why does the diff look different in the popup vs in-page
<rockstar> thumper, intellectronica can probably help there.  I don't think that interface is flexible enough for what I need, so I haven't used it.
<salgado> thumper, accept: LP.client.XHTML in the config
<intellectronica> thumper: when you make a request using the lp client, pass `accept: LP.client.XHTML` as part of the configuration object
<rockstar> thumper, flacoste has drafted a new proposal that's less restful but more practical, which would then help.
<thumper> intellectronica: is there an example of attribute access using xhtml in the bugs JS somewhere?
<rockstar> thumper, wait, why do you want to get the preview_diff HTML through the API?
<intellectronica> thumper: yes, look at bugtask-index.js, for example
<thumper> rockstar: because you can't it the code.lp.net from the bugs.lp.net page
<thumper> rockstar: it won't let you
<thumper> I don't know why
<thumper> it is not posting, just getting
<mars> EdwinGrubbs, nope, sounds like a fun IE bug.  Have you considered using .get('tabIndex')?  They may have normalized the return value in the YUI code
<rockstar> thumper, bugger.
<thumper> rockstar: yes
<thumper> ok,
<thumper> that just leaves the last point
<thumper> I have the same code generating the diff for the in-page view and the popup
<thumper> but when I set the content of the pretty overlay
<thumper> it renders the table outside the div it should be in for some freaking weird reason
<EdwinGrubbs> mars: get('tabIndex') is way worse. It returns 0. The difference may be due to one using the IE extension to getAttribute and the other not. http://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx
<mars> EdwinGrubbs, isn't '0' what you want?
<mars> since that is the default?
<rockstar> thumper, that might have to do with markup and css.
<intellectronica> thumper: actually, an example more relevant to your case is get_comment_HTML in lib/canonical/launchpad/javascript/lp/comment.js
<thumper> :(
<thumper> intellectronica: thanks
<EdwinGrubbs> mars: well, on firefox get('tabIndex') returns -1, so it's not consistent.
<rockstar> thumper, so you might have to mess with the markup it creates.
<thumper> intellectronica: I have written an adapter already, needed for the commit message
<thumper> rockstar: any ideas what would cause it to render outside?
<rockstar> thumper, honestly, I expected that when you were talking about it.  It may need a separate view entirely.
<thumper> rockstar: it isn't just rendering, the firebug dom shows it outside
<rockstar> thumper, not without looking at the exact markup.  It might be using floats or tables or any number of bad things.
<thumper> rockstar: there are tables
<mars> EdwinGrubbs, some raw data: http://www.weba11y.com/blog/2007/11/
<thumper> but the table is in a div, so should be ok
<rockstar> thumper, yea, tables are amazingly complex.
<rockstar> thumper, I probably can't look now, but if you point me to a branch, I'll do my best to get to it.
<thumper> the fmt:diff renders a table
<thumper> rockstar: my popup-diff branch
<thumper> rockstar: and thanks
<rockstar> thumper, ah, okay.  It was in my queue of things to do today, but deryck and I had yaks to shave.
<thumper> I'll be working on this later, as I'm using it as part of my assignment due Monday:)
<mars> EdwinGrubbs, try: node.get('tabindex');  // note the lowercase 'i'
<thumper> rockstar: I did change the behaviour so it gets a spinner after the link, nice fallback to the librarian file, and popup only once loaded though
<thumper> rockstar: it was pretty easy to do
<EdwinGrubbs> mars: from that blog post, it seems like you would want NoTabIndex to set the tabIndex to -1 instead of using removeAttribute('tabIndex')
<thumper> rockstar: took maybe 10 minutes
<rockstar> thumper, great.
<thumper> I'm off to buy meat for the BBQ :)
<EdwinGrubbs> mars: .get('tabindex') is a separate variable than getAttribute('tabindex') since get() is case-sensitive.
<mars> EdwinGrubbs, yes, it looks like you are right
<rockstar> EdwinGrubbs, we might want to document that somewhere.
<mars> EdwinGrubbs, there is also this: http://developer.yahoo.com/yui/3/node-focusmanager/index.html
<EdwinGrubbs> rockstar: there will definitely be a very verbose comment about this.
<mars> EdwinGrubbs, trying to remember if setting tabIndex to -1 on the overlay <div> will remove all of its child controls from the tab order as well...
<mars> EdwinGrubbs, according to http://www.w3.org/TR/html401/interact/forms.html#adef-tabindex, the value needs to be between 0 and 32767.  So IE is setting it to the last possible tab element on the page.
<EdwinGrubbs> mars: it's actually setting it to 32768, which is an illegal value.
<mars> ah
<mars> EdwinGrubbs, eewww: On IE it is not possible to determine whether a div has no tabindex or
<mars> tabindex="0" using elem.getAttribute("tabindex") or elem.tabIndex
<mars> because a default value of 0 is returned. Microsoft provides an
<mars> extension to getAttribute() that can be used to determine if the
<mars> tabindex is unset. If a second parameter of value "2" is passed to
<mars> getAttribute(), a value of 32768 is returned whenever tabindex is not
<mars> set. See Microsoft's documentation on getAttribute:
<mars> double-eewww
<EdwinGrubbs> mars: yes, that's the link I sent you
<mars> EdwinGrubbs, yep, but I didn't understand why it was necessary.  That is not fun.
<EdwinGrubbs> mars: so my main question is whether I should just fix the unit test, or if I should also have NoTabIndex set it to -1.
<mars> EdwinGrubbs, set it to -1.  That is a modern browser extension.  Don't worry about IE.  Please ensure that child elements in the Overlay are still tab-navigable.
<mars> EdwinGrubbs, if IE is being a pain, then put an exception into the test just for it, with the relevant links.  I don't see a better way to handle this at this time.
<EdwinGrubbs> mars: IE seems to handle -1 exactly like the other browsers. It was just the removeAttribute() that was really difficult to verify, and I didn't know if there was a reason NoTabIndex should use that. I will test the overlay, and I think -1 is much better solution for NoTabIndex.
<mars> EdwinGrubbs, should the extension be renamed to RemoveTabIndex then?
<mars> or RemoveFromTabOrder, or something
<EdwinGrubbs> mars: hmmm, maybe DisabledTabIndex.
<mars> EdwinGrubbs, DisableTabIndex sounds good.  Y.augment(Foo, DisableTabIndex);
<EdwinGrubbs> ok
<EdwinGrubbs> mars: can I send you the reviw later today, so you can look at it Monday? I have some other IE fixe in the branch, but it's not that big.
<mars> EdwinGrubbs, sure
<wgrant> pgquiles: Why are you now installing debhelper and python-twisted?
<wgrant> Neither of those are necessary.
<wgrant> pgquiles: And make run_all doesn't do anything extra that Soyuz needs.
<pgquiles> wgrant: launchpad-buildd requires debootstrap and python-twisted, IIRC
<pgquiles> wait, debhelper?
<wgrant> pgquiles: launchpad-buildd does need python-twisted, but it is included in the package's dependencies. It does not require debootstrap.
<wgrant> And you only need debhelper if you are building packages yourself.
<pgquiles> wgrant: yeah, that was it. But the next step in the howto is building the launchpad-buildd package, so I need debhelper.
<wgrant> pgquiles: Ohh, true.
<wgrant> But debootstrap/python-twisted are not required.
<pgquiles> dpkg -i launchpad-buildd fails, debootstrap and python-twisted are dependencies in debian/control
<pgquiles> (IIRC, I don't have a checkout here)
<wgrant> pgquiles: Ah, then perhaps you meant 'apt-get -f install'
<wgrant> No point installing them explicitly, since those deps could change.
<pgquiles> wgrant: mmm when I install launchpad-buildd with dpkg -i as said in the howto, dpkg returns an error
<pgquiles> I'm trying to browse code.launchpad.net to find the control file and check the dependencies for launchpad-buildd
<wgrant> pgquiles: If you then apt-get -f install, it should finish installing the package.
<pgquiles> ah, yes
<wgrant> Depends: python-twisted, debootstrap, dpkg-dev, linux32, file, bzip2, sudo, ntpdate, adduser
<pgquiles> yeah
<wgrant> It should not depend on debootstrap, but that's another matter.
<pgquiles> after my research yesterday and a bit of IRC talking with lamont this today, I think I know how to build a chroot like the ones you can download form launchpadlibrarian.net
<pgquiles> I need to find out about some config files but I'll probably write an article about that this weekend
<wgrant> pgquiles: What additional stuff was necessary?
<pgquiles> wgrant: it's esentially debootstrap --variant=buildd --components=main,universe, then chroot to that folder, create buildd group and user and run root/bootstrap
<pgquiles> dump kernel config and two or three more config files I can't remember now
#launchpad-dev 2009-11-14
<MTecknology> I'm reading about scrum and I think just using names like pigs and chickens is making me want to use it for a big project I'm about to start
<MTecknology> scrum isn't for initial development of a project, is it?
<thumper> I don't suppose there are any LP JS experts are around?
<thumper> flacoste: are you really around?
<thumper> I'm thinking not
 * thumper has a webservice that isn't being rendered properly
 * thumper thinks he has it working
<thumper> gah
<thumper> obj/fmt:api_url doesn't work
<wgrant> It does.
<wgrant> It's just not entirely obvious what it does.
<wgrant> I remember I fought with it will arguing with the Bugs JS.
 * wgrant tries to remember what it does.
<wgrant> Ah, yes, that strangeness.
<wgrant> thumper: What is wrong with it?
<thumper> wgrant: it is broken
<thumper> wgrant: it is broken for cross app work
<thumper> fine if you are dealing with objects that live on the same root site
<thumper> the code is obviously broken
<wgrant> Well, you have to construct the root site yourself.
<thumper> no
<thumper> there should be no root site
<thumper> it should be a relative path
<wgrant> It is, isn't it?
<thumper> the api_url function does this by passing None in for the rootsite
<wgrant> I wish I could remember what I discovered about it a few months back.
<thumper> but this just means use the default
<thumper> which for a merge proposal is "use code.launchpad.X when on bugs.launchpad.X"
<wgrant> Ahh.
<wgrant> (that code is actually mine)
<thumper> really?
<wgrant> Yes. It was previously just self.url(), which was even worse.
<thumper> I'm going to add a force_local path only to canonical_url
<thumper> which api_url can use.
 * thumper makes another pipe
<wgrant> How do merge proposals override their rootsite?
<thumper> they don't
<thumper> x00t
<thumper> w00t even
<thumper> my code works
<thumper> done for now
<thumper> will fix tomorrow
<wgrant> bigjools: Thanks.
<bigjools> np
<bigjools> was just checking in for my flight tomorrow and remembered I hadn't done your review yet
<wgrant> bigjools: Will you have time during UDS to review the (much simpler) second half?
<bigjools> wgrant: hopefully, it depends on the schedule
<bigjools> wgrant: you branch failed in deathrow.txt
<bigjools> your
<wgrant> bigjools: Argh, obviously hadn't checked archivepublisher lately.
<wgrant> I hate you, broken sample data.
#launchpad-dev 2009-11-15
<wgrant> sinzui: Ah! Thankyou for fixing the location stupidity.
<sinzui> wgrant: thanks. I knew someone would notice before I could submit the branch for review
<intellectronica> sinzui: need a review?
<sinzui> intellectronica: I often do. This one branch will make a lot of people happy: https://code.edge.launchpad.net/~sinzui/launchpad/location-bug-262193/+merge/14878
<sinzui> intellectronica: I have a queue of branches long enough to keep Mondays and Tuesdays reviewers busy
<intellectronica> sinzui: it's your lucky day. i'm stuck in the most boring town in the world
<wgrant> Since I imagine bigjools won't be around for a little while, can I convince somebody to quickly review the last two test-fix revisions of https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection-part1/+merge/14729 and ec2test-land it?
<sinzui> intellectronica: Indeed. I refused to go.  Dallas, is still Texas, and that is still nowhere
<wgrant> bigjools reviewed 9857, but it failed a couple of tests.
<sinzui> yuck sample data
<wgrant> sinzui: Yes. Ignore that diff.
<sinzui> wgrant: I certainly will. I can review it and land it
<wgrant> sinzui: Thanks.
<intellectronica> sinzui: r=me. great to see an unused feature go
 * intellectronica sheds a tear of joy as he sets a branch's status inline for the first time
<sinzui> intellectronica: thanks.
<sinzui> wgrant: is there a commit message for this?
<wgrant> sinzui: Sorry, forgot about that -- I've just set it on the MP: "Refactor bits and pieces to prepare for Debian source format 3.0 support."
<thumper> sinzui: hey
<sinzui> hi thumper
<thumper> sinzui: I have a branch I'd love to get the quick treatment
<thumper> https://code.edge.launchpad.net/~thumper/launchpad/fix-canonical-url-for-api-2/+merge/14876
<thumper> sinzui: found during my api work for popup diffs
<sinzui> wgrant: your branch has conflicts:
<sinzui> Text conflict in database/sampledata/current-dev.sql
<sinzui> Text conflict in database/sampledata/current.sql
<sinzui> Text conflict in lib/lp/archiveuploader/dscfile.py
<sinzui> thumper: I'll review it
<thumper> sinzui: ta
<sinzui> wgrant: when I get sampledata conflicts, I regenerate both current files
<wgrant> Yay, bitrot :(
 * wgrant merges.
<wgrant> sinzui: Where were you landing that? I see no conflicts with devel.
<sinzui> db-devel. sql changes have to land in db devel
<wgrant> sinzui: We were sneaky and landed the DB changes in 3.1.10.
<wgrant> sinzui: But the sample data didn't get updated.
<sinzui> wgrant: bugger
<wgrant> Can you not land sampledata changes to devel?
<sinzui> Do you need the sampledata
<sinzui> I cannot land the sampledata in devel
<wgrant> Ah, damn. I will see if I can avoid it.
<sinzui> hmm
<sinzui> well
<wgrant> I'm not exactly sure when the patches are applied.
<sinzui> Something in PQM is supposed to verify that nothing in database/ changes when merging into devel. So ec2 will pass it, then pqm will fail
<wgrant> If they are applied *after* the sampledata is loaded, it should be OK.
<sinzui> Is the sample data used in testing?
<wgrant> It is.
<wgrant> Actually, I guess the security.cfg changes are in this branch too, so merging this now to devel might break one particular circumstance on edge. I will go over this with bigjools and stub next week, I guess.
<sinzui> database/schema/security.cfg changes cannot be landed in devel because those changes will not be applied to edge
<sinzui> wgrant: are you thinking of breaking the branch in devel and db-devel parts?
<wgrant> sinzui: Very little point.
<sinzui> thumper: r=me, not changes needed
<thumper> sinzui: no changes?
<sinzui> it is good to land as it is
<thumper> sinzui: thanks
<thumper> beuno: ping
 * thumper wonders where beuno is
<beuno> thumper, hi. I'm in Dallas.
<nhandler> sinzui: I saw you landed one of my branches. Just a heads up that although that branch appears to fix the bug, after some discussion in here, it was determined that it was not *properly* fixing the bug (which is why I didn't propose it for merging)
<sinzui> nhandler: what do you think is the right fix?
<thumper> beuno: around now?
<thumper> I have the unfortunate problem of having the diff render propertly in Konquerer but not Firefox
<maxb> jml: perhaps you could compare pg_hba.conf from zbir's machine with one that works?
<jml> maxb, not a bad idea.
<maxb> launchpad-database-setup sets everything up to use 'trust' auth (or lack of auth), so the fact that it's attempting 'ident' sounds broken
<jml> maxb, it looks like it might be something to do with having 8.4 and 8.3 on the same machine
<maxb> ah.... and the wrong one on 5432, I take it?
<jelmer> maxb, ping
<maxb> pong
<jelmer> maxb: I'm trying to install the required launchpad dependencies and I see a dependency on a specific version of python-support
<wgrant> jml: Yeah, the 8.3 cluster will have installed on 5433 instead.
<maxb> jelmer: You mean on the package python2.4-support ?
<jml> Launchpad's 'make run' is connecting to the 8.3 cluster
<jelmer> maxb, yeah
<jml> but launchpad-database-setup modified the 8.4 setup
<thumper> anyone know firefox and css really well?
<wgrant> jml: Really? The other way makes sense, but not that...
<jelmer> maxb: Is there any reason it needs to depend on that specific version? I have 1.0.4 here, should that work?
<maxb> jelmer: We do this to ensure the python-support which actually includes python 2.4 support is installed (and doesn't get removed on upgrades)
<jml> wgrant, that's what science is telling us.
<jelmer> maxb: I mean the dependency on python-support=1.0.3ubuntu1launchpad~maxb2
<maxb> huh, where's that?
<wgrant> jml: launchpad-database-setup appears to verify that 8.3 is running on 5432...
<wgrant> jml: Oh, but launchpad-database-setup supports 8.4 now.
<wgrant> So it will indeed configure 8.4.
<wgrant> jml: How do you know LP is connecting to 8.3?
<jml> wgrant, when we shut down 8.3, we get connection errors.
<jml> on running 'make run'.
<wgrant> jml: Huh. Anyway, quick fix is to drop 8.4 from the list of supported versions in launchpad-database-setup and rerun it.
<wgrant> That might work.
<wgrant> Although then it might complain about ports.
<jml> wgrant, yeah, that's what we're trying now.
<wgrant> jml: Apart from this mess, how long did it take to get running?
<jml> wgrant, not very long at all.
<jml> wgrant, 'make build' takes a while, and package downloads are pretty fast.
<maxb> jelmer: where is this dependency? I am not familiar with any equals-single-version deps like that.
<jelmer> maxb: Trying to figure that out at the moment
<maxb> well, what fails~
<maxb> ?
<wgrant> Is staging still running the 2.5 branch?
<al-maisan> http://pastebin.ubuntu.com/319585/
<wgrant> al-maisan: Nice. But what does launchpadlib think of it?
<al-maisan> wgrant: sorry, this was just a test fragment I was showing to jml
<al-maisan> wgrant: the branch in question is here: lp:~al-maisan/launchpad/352094
<al-maisan> bug #352094
<mup> Bug #352094: Expose ISourcePackage.branches <api> <package-branches> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/352094>
<thumper> \o/
<thumper> fixed my rendering issues
<al-maisan> nice :)
#launchpad-dev 2010-11-15
<wgrant> spm: Code imports are back online, right?
<lifeless> ola!
<wgrant> lifeless: You have successfully reached $destination?
<lifeless> wgrant: yes
<lifeless> wgrant: no thanks to air nz who rebooked a flight a day later
<wgrant> lifeless: ... nice.
<lifeless> *that* I have to sort out still - meant to have an email from the helpdesk supervisor... but no sign.
<lifeless> time to up fire skype.
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM is in release-critical mode | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<henninge> Hi jtv! ;)
<jtv> hi henninge
<bigjools> hello
<wgrant> Morning bigjools.
<bigjools> g'day wgrant
<bigjools> how can so much email be sent over one weekend  :'(
<wgrant> :(
<wgrant> bigjools: I ran into an issue with the new buildd-manager this morning.
<wgrant> bigjools: The builder chrootwaited, and then it started dying like this:
<wgrant> 2010-11-15 10:26:29+1100 [QueryProtocol,client]   File "/home/wgrant/launchpad/lp-branches/devel/lib/lp/buildmaster/model/buildfarmjobbehavior.py", line 193, in updateBuild_WAITING
<wgrant> 2010-11-15 10:26:29+1100 [QueryProtocol,client]     build = queueItem.specific_job.build
<wgrant> 2010-11-15 10:26:29+1100 [QueryProtocol,client] exceptions.AttributeError: 'NoneType' object has no attribute 'build'
<wgrant> It's possible something was screwy with my system, but I don't think so.
<bigjools> new as in the one I just fixed but is not rolled out yet?
<wgrant> bigjools: Yes.
<wgrant> bigjools: r11926, post-invalidation-fix.
<bigjools> what were you doing with it to make it do that?
<wgrant> The slave was under considerable strain, but chrootwaited legitimately, and this was picked up by buildd-manager. It's possible that there was a timeout, but there's nothing in the logs.
<wgrant> http://paste.ubuntu.com/532214/
<bigjools> hmm
<wgrant> It looks very relevant to that fix :/
<bigjools> can you force a chrootwait and see if it happens again?
<wgrant> I was half-studying at the time, so didn't. But I'll try now.
<wgrant> Also, buildd-manager seems to put more detailed error messages in the failure notes now, including the URL for failed file retrieval. Does that handle privacy etc. correctly?
<bigjools> no that's a leak :/
<bigjools> and not an easy thing to fix
<wgrant> No.
<mrevell> Morning
<wgrant> Morning.
<wgrant> bigjools: Plain chrootwait worked fine.
<wgrant> There must have been a timeout somewhere.
<bigjools> wgrant: so it might be a loading issue
<wgrant> The slave was having some issues responding earlier (it's a rather slow ARM device).
<wgrant> bigjools: Er, it came up on qastaging?
<wgrant> That's a little concerning.
<bigjools> wgrant: yes, lifeless is trying to get a publisher working on it
<wgrant> That still shouldn't happen unless the DB is screwed.
<bigjools> quite
<bigjools> nobody asked me so I don't know the details
<wgrant> Speaking of qastaging, could you time a query on there or staging for me?
<wgrant> (it's the source domination one)
<bigjools> sure
<bigjools> paste me something
<wgrant> Do you know natty's ID?
<bigjools> 106
<wgrant> bigjools: http://paste.ubuntu.com/532246/
<wgrant> Will probably return nothing.
<wgrant> But I only really care about speed.
<bigjools> with cold cache, about 10 seconds
<bigjools> with warm cache, 500ms
<wgrant> Excellent, thanks.
<bigjools> you want the query plan?
<wgrant> I guess that's a good idea.
<wgrant> But it seems like it should beat the current 3-minuter!
<bigjools> wgrant: http://pastebin.ubuntu.com/532247/
<bigjools> yes :)
<wgrant> And no temp table required, yay.
<bigjools> I love performance fixes
<wgrant> Even if it is only 10% of the publisher :/
<bigjools> it will be attritional
<bigjools> think of the PPA gains though
<wgrant> It shouldn't be too big there. I think removing the dozens of cache purges per archive might help more.
<wgrant> But it's hard to say.
<wgrant> There is so much crap on DistroSeries.
<wgrant> lazr.restful makes me sad.
<lifeless> moin
<jml> lifeless: hi
<jml> lifeless: where are you?
<lifeless> orlando
<lifeless> caribe royale
<lifeless> by room
<lifeless> *my room*
<lifeless> I can get more specific than that but its probably a little redundant at that point
<jml> lifeless: heh, yeah.
<jml> lifeless: I mostly meant what tz
<lifeless> flacostes
<lifeless> AIUI
<jml> lifeless: *nod*
<lifeless> 6 hours out from my normal
<wgrant> Uh.
<wgrant> Where did devel r11926 go?
<wgrant> Hmmm. Maybe it was caught in the branchChanged timeouts.
<wgrant> LP doesn't seem to know about it, but it's in the branch, and LP sees it in stable.
<bigjools> jml: did you get around to updating the derived distros LEP with your thoughts on opening new series early?
<deryck> Morning, all.
<jml> bigjools: no, I haven't.
<wgrant> bigjools: aaaaaaaaaaaaaaaaaa
<jml> bigjools: it's on my list though.
<bigjools> jml: ok no rush, I'm busy with the b-m changes for now anyway
<lifeless> wgrant: it was caught in that I suspect
<lifeless> wgrant: we haven't actiond that follwup - someone needs to write ze script.
<deryck> I think I'm at windmill + big honkin' js file fail again, but I have no way to really know this.
<deryck> Maybe I need more coffee.
<wgrant> lifeless: How was the escalation a policy violation?
<wgrant> Does it not permit escalation of issues that make us all look awful?
<lifeless> wgrant: https://wiki.canonical.com/Launchpad/PolicyandProcess/DefinitionofCriticalPolicy which you can't see
<wgrant> Hah.
<wgrant> Soon...
<lifeless> there is a table
<lifeless> I'll paste it, hang on
<wgrant> I think that if we want to be a credible project hosting service, we might want to actually be able to keep it properly functional outside the working week :)
<lifeless> wiki paste fail
<lifeless> but I'm sure you'll figure it out
<lifeless> yes
<lifeless> thus I'm going to drive a discussion about this with the whole team.
<lifeless> I think most of us would agree that this was critical.
<lifeless> So the mismatch isn't cultural its technical in the sense that we've written down something that doesn't match our own expectations.
<wgrant> Right.
<lifeless> -> food
<jml> well
<wgrant> But then there are other things like the person picker being broken for more than a week.
<jml> also, we only have 6x24 coverage from ops.
<lifeless> jml: the discussion has to be broader than the canonical lp dev team.
<jml> but that's kind of orthogonal to treating probs like these as critical
<lifeless> jml: but it needs to happen. In this instance - and *every other instance its mattered* we've had the ops support, via IS escalation.
<wgrant> jml: That may be, but it has to be easy enough to get assistance during that other 24 hours.
<lifeless> jml: What I'd like is to help get charlieS appropriate resources to not need special escalation.
<jml> wgrant: to me, that seems a separate discussion to "what is critical"
<jml> lifeless: yeah, that'd be awesome
<wgrant> jml: It's not entirely orthogonal, I don't think.
<lifeless> jml: at the same time I'd like us to keep reducing incidences of zomg moments
<jml> lifeless: I would very much like that too.
<bigjools> so it's agreed - let's fix everything
<jml> lifeless: although "keep reducing" implies something about the current situation that I'm not sure I agree with :)
<jml> wgrant: insofar as if we had infinite resources, we wouldn't have to care about whether a thing is critical or not, I guess.
<wgrant> jml: Right, exactly. You're probably not going to invoke a sparse emergency resource for something that isn't critical.
<lifeless> jml: I pushed some new stuff to fixtures, testresources, testrepository and mailed you a brain dump for something meeting layers functional behaviours
<jml> lifeless: I saw the email
<jml> lifeless: but not the recent changes
<jml> lifeless: every time I think about testr I get annoyed with myself that my recent changes to it aren't in the binary that my system gives me
<lifeless> jml: stay away from the dark side
<lifeless> think positive
<jml> lifeless: if I did that, I doubt I'd ever write any code at all :)
<lifeless> http://ftp-master.debian.org/new.html - python-fixtures
<lifeless> jpds: hi
<jml> lifeless: what do you mean by "edge node"?
<lifeless> jml: layer = Foo
<lifeless> jml: Foo will be on the edge of the DAG, at least while the tests using it are running.
<jml> lifeless: you mean a leaf node?
<lifeless> close enough; sleep deprived ravings :)
<jml> lifeless: np. :)
<lifeless> jml: root node strictly speaking - the pointers go away from it
<jml> lifeless: ahh yes, I see what you mean
<jml> lifeless: "there is nothing in the graph that (depends on|points to) Foo yet"
<lifeless> yeah
<jml> lifeless: "leaf node" is probably the best name for that, despite the directions of the arrows. e.g. a package that nothing depends on is sometimes called a leaf package.
<lifeless> sure
<lifeless> jml: I ran out of puff somewhere around texas, or I think I would have just got it done
<jml> :)
<jml> lifeless: I know the feeling.
<jml> yay.
<jml> email, in tray & IRC back log dealt with. I hardly know what to do next.
<jml> I know, lunch and the thirty or so things on my 'launchpad' list.
<lifeless> jml: reset - did you read the prose in fixtures/NEWS about it ?
<lifeless> jml: under shared dependencies
<jml> lifeless: no, haven't. can do.
<deryck> mars, ping
<rockstar> deryck, yo
<deryck> hi rockstar
<rockstar> deryck, how goes the lazr-js?
 * deryck wants to drive rockstar's lazr-js branch into the ground like nick fairly on a UGA quarterback.
<rockstar> deryck, also, war eagle.
<rockstar> deryck, :)
<deryck> war eagle!
<rockstar> deryck, are the tests still failing?
<deryck> rockstar, yup.  I'm too the point of creating my own lazr-js branch pre yui 3.2 and back porting the two fixes I need....
<deryck> but that's non-trivial too
<deryck> rockstar, my guess is that's it's the size of launchpad.js again that's playing poorly with windmill.  1.3 Mb file now.
<rockstar> deryck, yeah, that's what I suspected too, but I couldn't get any evidence.
<deryck> rockstar, there's lots of thread contention and mars said earlier that this is likely because we're trying to go off-site for resources, but I can't find that happening.
<deryck> rockstar, trying another ec2 run now with any yui config option that relates to css or combo loading set to false to see if that helps.
<deryck> rockstar, beyond that, I really have no idea what to try next.
<rockstar> deryck, I think our configs already block going to the net.
<deryck> rockstar, right.  but if yui is trying to go to the net and being blocked a test could hang, right?  Which is some of what seems to be happening.... i.e. windmill test hangs and the test runner kills the run.
<rockstar> deryck, it could fail the test, because it wouldn't be able to load js or css.  I can't imagine it would hang a browser indefinitely.
<rockstar> Did I tell you that mikael said windmill is dead?  No one is working on it anymore.
<deryck> rockstar, well not indefinitely, but at least 600 seconds, which is enough to kill the run.
<deryck> rockstar, and yeah, I think you mentioned that.
<rockstar> deryck, okay.  I'm rooting for you.
<deryck> rockstar, :)  thanks.
<rockstar> deryck, also, fwiw, I'm deleting most of lazr-js, and I think it's sane to require that you have a working use case in an app before moving the feature "upstream" into lazr-js.
<rockstar> This "trickle down" things causes too many headaches.  The wizard widget would have done well to be Launchpad for a bit first...
<deryck> rockstar, really?  Your deleting stuff that is unused?  Is there really that much unused?
<rockstar> deryck, unused or unneeded, yes.
<deryck> wow, ok.
<deryck> rockstar, is landscape using a combo loader?
<rockstar> deryck, I replaced the PrettyOverlay with about 10 lines of javascript and some CSS.
<deryck> cool
<rockstar> deryck, I don't think so.  I'm going to try and convince beuno that we need the combo loader for U1.
<deryck> rockstar, we have to figure something out.  1.3 Mb is ridiculous for a single js file, if we keep this 3.2 upgrade.
<rockstar> deryck, yeah, and U1 doesn't combine any of its js.
<deryck> rockstar, yeah, in devmode FF goes gray with this 3.2 upgrade because of all the files loaded.  ./utilities/yui-deps.py|wc -l == 446 :(
<jml> hmm.
<jml> sinzui: strongly tempted to fix blacklisting.
<beuno> I am convinced.
<beuno> he just has to do it
<sinzui> jml, The card is in the registry backlog. EdwinGrubbs and I agree to add a column for the team that can override the constraint
<jml> sinzui: oh cool.
<sinzui> jml: the editing ui landed on Friday. I used it
<sinzui> jml: https://launchpad.net/+nameblacklist
<lifeless> jpds: sorry, lost your paste
<rockstar> deryck, have you brought this specific issue up on the list (in its own thread, not as a mention on another thread).
<deryck> rockstar, which issue (of the numerous I ranted about)? :-)  The file size?
<rockstar> deryck, at this point, I think we seriously need to think about killing windmill.
<rockstar> It's either windmill goes or we stop adding javascript.  That's all there is to it.
<rockstar> deryck, because the yui 3.2 may not break it (and we can stay on YUI 3.0 forever...) but eventually we're going to hit this again.
<rockstar> And by "we" I mean "you launchpadders" since beuno owns me for the next six months.
 * beuno saved his reciept
<deryck> rockstar, yeah.  but I'm not hosting that discussion.  I've said my peace about this mess, and no one else seems concerned.
<bigjools> hanging windmill tests.... :'(
<jml> sinzui: hello
<sinzui> hi jml
<abentley> jkakar: Remember how I solved my getBuilds problem on Friday by using a subselect?  Now I've got a resultset where count() == 1 but list() produces a 25-element list.
<jkakar> abentley: On a call, can't help you just now, will ping in a while.
<abentley> jkakar: okay.
<bigjools> jml: async uploading for b-m works on DF.  We now have a fully async b-m.  \o/
<jml> bigjools: sweet :)
<abentley> bigjools: Yay!
<jml> bigjools: may I see the branch?
<bigjools> jml: it's 1k lines of diff - but please be my guest. https://code.launchpad.net/~julian-edwards/launchpad/async-file-uploads-bug-662631
<bigjools> abentley: indeed.  I got you a login on dogfood BTW.  when you're ready we can have a chat.
<jml> bigjools: ta, I won't do a review but I'd like to glance at least.
<bigjools> I just made a WIP MP
<abentley> bigjools: Anytime today is good.
<bigjools> jml: I added a new method to fetch all the files from the builder in one go
<jml> sinzui: http://mumak.net/lp-bugjam-2010/ btw
<lifeless> sinzui: bigjools: we still have a [smaller] issue with Archive:+packages https://bugs.launchpad.net/soyuz/+bug/675621
<_mup_> Bug #675621: Archive:+packages timeouts <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/675621>
<bigjools> lifeless: grar, that page is a PITA
<bigjools> I wonder if we can make our model speed-friendly sometimes
<lifeless> bigjools: I made the source only version happy
<lifeless> bigjools: we just need to fix the binary package version
<bigjools> it's a start
<bigjools> we have the exact same problem on the queue page
<bigjools> I optimised the hell out of it a couple of years ago but the custom uploads need sorting out still
<lifeless> the high python time is a red flag to me
<lifeless> but I want to get the sql time sane first
<bigjools> yarp
<lifeless> flacoste: fyi rt 42472
<jkakar> abentley: Here now, sorry for the delay.
<abentley> jkakar: No worries.
<sinzui> deryck[lunch], I would like your +1 for my data fix for https://bugs.launchpad.net/launchpad-registry/+bug/660475. The last three comment show the sql, the run time, and examples of what is fixed between production and staging.
<_mup_> Bug #660475: Merged profile listed in the Top contributors <404> <trivial> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/660475>
<abentley> jkakar: I've tracked down the bug to specifying order_by on a column of a table that's not in the resultset.
<jkakar> abentley: Cool.
<abentley> jkakar: http://pastebin.ubuntu.com/532464/
<abentley> jkakar: The ResultSet used to refer to BuildFarmJob before I introduced the subselect.
<jkakar> abentley: Ah, interesting.
<abentley> jkakar: I guess this is valid, if crazy, SQL?
<jkakar> abentley: It sounds suspicious... do you know exactly what query is being generated?  It'd help to see it.
<abentley> jkakar: I don't.  If I drop to a debugger, can I see it?
<jkakar> abentley: If you do a from storm.tracer import debug; debug(True) it'll get printed to the screen when the code path is executed.
<abentley> jkakar: http://pastebin.ubuntu.com/532473/
<jkakar> abentley: Yeah, so the table is being included, which makes the query valid, but there are no BuildFarmJob columns so the order by is effectively useless.
<abentley> jkakar: Hmm, why is BuildFarmJob in the FROM?
<jkakar> abentley: Because it was specified in the order by.
<jkakar> abentley: When you don't explicitly define tables (with store.using) Storm does some automatic inspection to make sure all the tables you need in your query are included in the FROM clause.
<abentley> jkakar: I see.  And in this case, it found the BuildFarmJob in the order_by clause.
<abentley> jkakar: But I don't know if that is actually desirable behaviour.  Because if a table is only mentioned in the order_by clause, the order_by clause won't be effective.
<jkakar> abentley: Yeah, in this particular case it is undesirable, but in others, like an implicit join, you want the behaviour.
<jkakar> abentley: Perhaps we could special-case the logic to check for order by's that don't match any columns and blow up, though.
<abentley> jkakar: But if there's an implicit join, the table will be mentioned in the WHERE clause, no?
<jkakar> abentley: Yes.
<abentley> jkakar: So getting the table from the order_by seems like it doesn't help implicit joins.
<jkakar> abentley: True.
<jkakar> abentley: It would be good to file a bug about it.
 * jml gone
<sinzui> jcsackett, I cannot locate the bug that lead me to question why we were querying official_*. I am certain though that we should question the intent of any SQL operation that wants to know the answer. Talk to me if you see something that is not 100% needed.
<jcsackett> sinzui: dig.
<abentley> jkakar: https://bugs.edge.launchpad.net/storm/+bug/675667
<jkakar> abentley: Thanks.
<_mup_> Bug #675667: Storm accepts order_by clauses that mention columns not present in the output <Storm:New> <https://launchpad.net/bugs/675667>
<deryck> sinzui, you have my +1.  That's well put together and easy to follow in the comments.  I appreciate the before and after shots from lpnet and staging, too.
<sinzui> thanks
<deryck> np
<joey> bac: around?
<jcsackett> when working with the sqlbase cursor, what's the easiest way to get a look at the actual sql being thrown at the database?
<lifeless> a storm tracer
<lifeless> what are you trying to determine/debug/fix/?
<lifeless> we have some precanned ones
<jcsackett> lifeless: sorry, didn't see the reply. i actually just finished checking it out (with a storm tracer). i just needed to confirm some values being passed in were what i thought they were.
<lifeless> jcsackett: alternatives
<lifeless> LP_SQL_DEBUG=1 bin/test or whatever
<lifeless> OOPS's gather whats being sent [but elide the parameters at the moment]
<lifeless> QueryCollector (grep for it) gathers it
<jcsackett> lifeless: thanks. the LP_SQL_DEBUG one in particular sounds useful for situations i've hit. the tracer in this instance was perfect since i could flip it on around the one spot i cared about and then turn it back off.
<leonardr> joey: calling project.lp_refresh() after "project = lp.projects[project_name]" will solve the problem
<leonardr> in your version of launchpadlib, lp.projects[project_name] is a shim object. it doesn't go over the network until you access one of its properties
<joey> leonardr: ok will try now
<leonardr> so you won't get a KeyError if it doesn't exist
<leonardr> that also explains why httplib2 debug wasn't printing anything
<leonardr> there was no network call
<joey> leonardr: same with team?
<leonardr> joey: yes, wherever you do lp.object[name]
<joey> lazr.restfulclient.errors.HTTPError: HTTP Error 404: Not Found
<joey> leonardr: so it works for instantiated projects now
<joey> leonardr: but gives me that 404 for non-existent objects that I try to refresh
<joey> leonardr: and that's IN a try
<leonardr> joey: can you give me an example command line?
<leonardr> oh, i see what you mean
<joey> leonardr: https://pastebin.canonical.com/39779/
<leonardr> try catching NotFound instead of KeyError. KeyError is dictionary semantics
<leonardr> and in this version of launchpadlib, lp.object[] doesn't have dictionary semantics
<joey> leonardr: NameError: global name 'NotFound' is not defined
<joey> sorry my python skills are showing
 * joey reaches for his python book.
<leonardr> joey, it's lazr.restfulclient.errors.HTTPError
<leonardr> the one you had in that paste
<leonardr> in later versions there is a specific NotFound error, but not in this version
<leonardr> so try catching HTTPError and checking its .status
<joey> leonardr: yeap that did it
<joey> leonardr: strangly team is now a keyerror
<joey> leonardr: thanks a heap. It's working on staging now
<joey> leonardr: I didn't want to bypass it just in case there were other issues. Thankfully it looks ok
<leonardr> ok, good
<cr3> leonardr: flacoste just informed me about contributes_to in lazr.restful and I just finished reading the documentation, expect a hug for you and salgado next time I see one of you :)
<leonardr> cr3: i think you can give most of that hug to salgado
<cr3> leonardr: ok, I'll only give you half a hug then and 1.5 hugs to salgado
<leonardr> sounds like a good allotment
<cr3> leonardr: by the way, do you happen to know where contributes_to is being used, perhaps ubuntuone or something? I just made sure I had a fresh copy of lp-branches/devel and couldn't find any usage of it
<leonardr> cr3: i'm fairly sure salgado was using it in launchpad, but it might have been something to do with the openid provider or ubuntu one. you should ask him
<james_w> I don't think he actually got to using it, as unfortunately that project got paused due to higher priority work coming in
<leonardr> ah
<flacoste> leonardr: i think the KeyError and NotFound error is probably due to a server side change, which means that we broke backward compatibility
<leonardr> flacoste: i'm pretty sure it's due to a client-side change that we reverted because it caused a lo
<leonardr> a lot of confusion
<flacoste> ah ok
<wallyworld_> thumper: i've pushed the new code so you can look at the new diff in the mp before we talk
<thumper> wallyworld_: ok, ta
<jelmer> lifeless: hi
<lifeless> hi
<mars> lifeless, thank you for catching the landing numbers error.  That changes things a bit :)
<jelmer> wgrant: hi
<lifeless> mars: I'm curious where you got 1/53 minutes :)
<mars> lifeless, https://spreadsheets.google.com/a/canonical.com/ccc?key=0Aq6EjGubW4qydF90TElqaV9CRFR0QndmX0twV3ZDN3c&hl=en
<mars> lifeless, Google docs date calculation insanity.  What works in OO Calc does not work when ported.
<mars> lifeless, it came out to 27 branches/day, which is one every 53 minutes
<mars> the total of 127 above that did not click
<mars> fixed
<mars> 27 days, not 27 branches/day
<lifeless> heh
<lifeless> hey, adding 99% to that would be nice.
<lifeless> 100% is 'max', which isn't terribly interesting.
<mars> ?
<lifeless> your quartile breakdown
<lifeless> 25th percentile
<lifeless> etc
<mars> the quartiles - I already had to truncate those
<lifeless> could you add 99th percentile?
<mars> yeah, I looked for a way to do that, nothing popped out - one of those moments when you say "I must be missing something obvious"
<lifeless> 3*stdev + avg
<lifeless> is close enough
<mars> hmm, ok
<wallyworld_> thumper: talk now?
<thumper> wallyworld_: talk talk, or irc talk?
<wallyworld_> maybe talk talk
<thumper> ok, let me get set up
<mars> lifeless, not much difference - 343 hours at the 99th percentile vs. 356 for the 100th
<wallyworld_> thumper: https://code.launchpad.net/~wallyworld/launchpad/built-packages-listing/+merge/40634
<lifeless> mars: thats good to know - thanks for adding it
<flacoste> sinzui: we can't still add a private team as a member of another private team?
<sinzui> No
<sinzui> We certainly cannot do that until we prevent open teams from being members of restricted and moderated teams
<sinzui> see the team backlog
<sinzui> We also need to fix lots of TP calls
<wgrant> jelmer: Hi.
<lifeless> hi flacoste
<flacoste> hi lifeless
<flacoste> sinzui: how is that related? a private team is a private team, no? it can only be restricted
<flacoste> since you have to be a member to have access
<flacoste> lifeless: how's Orlando?
<lifeless> flacoste: good, its been a very interesting day.
<lifeless> we haven't done a lot of modelling yet, nor tackled things like coexistence.
<flacoste> will it be useful for us?
<thumper> IOError: [Errno 2] No such file or directory: 'logs/thread-140648257480448.request' ??!?
<sinzui> They relate because that is the path we believe that must be taken to achieve private in private...
<thumper> when running a test
<wgrant> thumper: Looks like nothing creates logs/ yet (except maybe dev appserver startup?)
<thumper> wallyworld_: I think that was because your branch removes the logs directory
<thumper> wallyworld_: but doesn't add it in make build
<wgrant> Hm, no, it's in bzr.
<thumper> wgrant: wallyworld_ removed it in one of his branches
<wallyworld_> ????
<sinzui> flacoste ... recall the argument of private in private was muddled by for a year. the real issue is proper constraints, private teams should be members of, or comprised of team that can be trusted. That is what must be fixed to allow use to build a hierarchy. Once we have established trust, we can fix Lp to not leak your trust
<wallyworld_> i thought i added it in the branch the consolidated the log files
<wgrant> It is there in devel.
<flacoste> sinzui: i think this is the public in a private team issue
<wallyworld_> wgrant: thumper: i did have it so that the logs dir was explicitly created but was told in the code review just to add it to bzr
<thumper> wallyworld_: but your current branch I'm messing with removes it
<flacoste> sinzui: which could be solved independantly of private in private
<flacoste> or private in a public
<wallyworld_> thumper: not on my system it doesn't....
<thumper> wallyworld_: line 611 (the last) of https://code.launchpad.net/~wallyworld/launchpad/built-packages-listing/+merge/40634
<sinzui> flacoste it is not.  We have discussed this many times and I cannot summarise it over IRC. consider this. We all want. It has been tried many times, and all were failures. We have a path to success and I can see this fix happening soon
<sinzui> flacoste You are hoping that a strict rule of private in private is easier to implement. It is not
<flacoste> sinzui: your team has put more thoughts than I in this, so I trust your judgement
<wallyworld_> thumper: hmmm. can't explain that. the logs dir is still in lp:devel at least
<thumper> wallyworld_: yes it is.
<thumper> wallyworld_: it is removed in your built-packages-listing branch
<thumper> wallyworld_: you may want to revert that removal :-)
<lifeless> flacoste: perhaps
<lifeless> flacoste: the modelling is the crucial bit
<wallyworld_> thumper: yep. i checked the logs of the branch. i merged from devel trunk after creating the branch and it was removed then for some reason
#launchpad-dev 2010-11-16
<wallyworld_> rockstar: ping?
<rockstar> wallyworld_, hi.
<wallyworld_> hey, with merge queues, i had started to do personproduct listings as a next step for person listings. is that something you still need me to do?
<wallyworld_> and i guess product listings
<wallyworld_> btw, i missed getting the person listings landed before the cut off. the branch passed ec2 ages ago but i was never lucky enough to find a non-testfix window to get the thing landed
<rockstar> wallyworld_, I'm not sure why we'd need personproduct listings.
<rockstar> Oh, product listings.
<rockstar> Um, I'm not sure we do, but thumper might have an opinion.
<rockstar> We have this idea that we need listings of everything, and I don't necessarily agree with that.
<wallyworld_> ok. wrt not getting the branch landed - it was done in devel to pick up the db changes but after the rollout, thise db changes will be in prod
<rockstar> wallyworld_, so, one thing that would be helpful with merge queues would be to take the JSON constraint out of the config.  You were right, we shouldn't force JSON on users.
<wallyworld_> should i merge the stuff i did in db-devel into devel and land that after rollout? to get the person listing stuff in?
<wallyworld_> ok. you want me to remove the json constraint?
<rockstar> wallyworld_, yeah, if you could.
<wallyworld_> ack
<rockstar> wallyworld_, it's still (currently) only editable through the API anyway.
<rockstar> wallyworld_, I think landing on devel once the devel rev has been tagged is okay, no need to merge (provided there are no conflicts)
<wallyworld_> ok. i'll do that, so that you can pick up what i've done
<wallyworld_> rockstar: is there a bug filed already for the json issue?
<rockstar> wallyworld_, no, I don't think so.
<wallyworld_> ok
<rockstar> wallyworld_, the duplication of bug and kanban task prevented me from ever being organized enough.
<wallyworld_> :-)
<rockstar> wallyworld_, the only other backend thing that really needs to get done immediately is queuing a merge proposal, which I'm working on right now.
<wallyworld_> ok
<rockstar> Launchpad still has to be in a chroot though, because you can't develop ubuntuone and launchpad together on the same system.
<wallyworld_> that's too bad
<wgrant> buildout mess?
<wgrant> Not fixable with virtualenv?
<rockstar> wgrant, well, first, Launchpad eats postgres. All of it.
<wgrant> It only does that once.
<wgrant> After that it leaves it alone.
<abentley> lifeless: Can you recommend a commandline testrunner compatible with testrepository?
<spiv> abentley: I'd guess trial --reporter=subunit
<abentley> spiv: I don't have a dependency on Twisted.  Doesn't trial depend on it?
<spiv> yes
<spiv> (but your tests don't have to depend on Twisted or Trial, if that makes a difference)
<spiv> There are probably subunit plugins for other tools like nose, but that seems no different to me than using trial.
<lifeless> abentley: python -m subunit.run
<lifeless> abentley: or as spiv says trial --reporter=subunit
<spiv> lifeless: oh, subunit has a runner?  Cute!
<abentley> lifeless: cool.
<lifeless> there is a nose subunit plugin floating around but its not really all that polished AIUI
<elmo> rockstar: u1 have a very good system of spinning up a non-system postgres for their test suite, you should be able to mix and match
<rockstar> elmo, yeah, and at some point, I'll sort that out.
<rockstar> (Probably not on my first day with the u1 team)
<lifeless> abentley: oh, and bzr selftest --subunit
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      180 /   53  Person:+commentedbugs
<lifeless>       78 /  268  BugTask:+index
<lifeless>       46 / 2908  Archive:+index
<lifeless>       19 /  331  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>       14 /  312  Distribution:+bugs
<lifeless>       10 /   12  Person:+bugs
<lifeless>        8 /  100  MaloneApplication:+bugs
<lifeless>        8 /    5  DistroSeries:+queue
<lifeless>        5 /   15  DistroSeriesLanguage:+index
<lifeless>        5 /    6  ProjectGroup:+milestones
<abentley> lifeless: sure.  And the Launchpad test runner too.  But this is a separate project.
<lifeless> sinzui: ^ Person:+commentedbugs has a fix in lp from stub
<wgrant> Hmm... Archive:+index is still broken?
<lifeless> abentley: righto, was just enumerating as things occured to me ;)
<lifeless> wgrant: broken - not. Not all it can be - yes.
<wgrant> lifeless: Hard timeouts == broken.
<lifeless> wgrant: sure
<lifeless> 46 is tolerable while we're recovering from 8.4
<lifeless> wgrant: compared to hundreds per day
<lifeless> wgrant: also note - no XMLRPC ones ;)
<wgrant> Indeed.
<wgrant> A bit disturbing that that was all just an overloaded appserver.
<james_w> I'm a bit surprised that +branches pages don't show up on that list, it suggests to me that branch listings are rarely used, which is concerning in itself
<wgrant> Is +branches linked from anywhere?
<james_w> sorry, I was being overly precise
<wgrant> +code-index?
<james_w> whatever you get when you go to the "Code" tab
<james_w> particularly person and person+pillar listings give me timeouts
 * james_w causes a few OOPS for those pages to try and get lifeless to fix them :-)
<lifeless> james_w: https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-pageids.html
<lifeless> thumper: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1781G129
<abentley> lifeless: what would test_id_option be for subunit.run ?
<lifeless> abentley: subunit.run doesn't support filtering at the moment
<lifeless> cat .testr.conf
<lifeless> [DEFAULT]
<lifeless> test_command=PYTHONPATH=. python -m subunit.run foo.tests.test_suite
<lifeless> abentley: ^ something like that is all you'd need
<abentley> lifeless: but I wouldn't be able to use testr run --failing?
<lifeless> it will work, but it would't filter
<lifeless> I really should add filtering to subunit.run
<abentley> lifeless: Yes, I think that would be very helpful.
<spiv> Trial can filter :P
<abentley> spiv: indeed, and I guess that makes it the best tool for the job, at the moment.
<spiv> *nod*
<abentley> spiv: err, how do I make it filter?
<spiv> abentley: you can pass it test names
<spiv> (or test class names, or test module names)
<abentley> spiv: So what do I specify for test_id_option ?
<spiv> e.g. if there's a test called package.tests.module.Class.method, trial will run it with any of "trial package", "trial package.tests", "trial.tests.module", "trial.tests.module.Class", "trial.tests.module.Class.method" (although obviously all but the latter are likely to find more tests than just that one)
<spiv> heh, I typoed some of those badly
<spiv> Basically "trial $python_dotted_name" will run all the tests contained in that dotted name
<spiv> It'll probably fail to work nicely if you have test ids that don't correspond to python dotted names (e.g. if you have test multiplication via testscenarios or something)
<abentley> spiv: I can't use something like that.  I need something that allows me to specify a file with test ids.
<spiv> trial `cat $file` ?
<abentley> spiv: testrepository does not appear to support that.
<spiv> sh -c "trial `cat $file`" ?  ;)
<spiv> Hmm, what a shame.
<spiv> So close and yet so far!
<abentley> spiv: testrepository expects that there is an option you can supply to the elsewhere-specified command that will make it read a file listing the test ids to run.
<abentley> spiv: So I can't change the command, only append an argument.
<spiv> abentley: sounds like a job for a shell script until one of these tools gets better :(
<abentley> spiv: Yeah, seems like.
<wgrant> spm: Around?
<wgrant> I think germanium is pretty upset.
<wgrant> No uploads accepted or packages published for nearly half an hour.
<wgrant> losa ping: ^^
<poolie> jml: up yet?
<wgrant> Could someone prod ISD about the issue in #launchpad?
<nigelb> SSO isn't LP per se is it?
<wgrant> Exactly.
<wgrant> Needs ISD.
<wgrant> And they like to hide.
<nigelb> wgrant: :/
<nigelb> I remember seeing a sort of rant mail about it in launchpad-dev
<wgrant> It's not an awfully wonderful situation.
<poolie> i will try
<wgrant> Thanks poolie.
<poolie> what is the issue, in one sentence?
<poolie> and how critical?
<wgrant> Another email address that SSO won't send mail to.
<wgrant> Two support cases have been opened about it.
<wgrant> But both have been ignored.
<poolie> :/
<wgrant> ixokai in #launchpad
<poolie> and the thing with germanium is separate?
<wgrant> Yes.
<wgrant> That doesn't need ISD, fortunately.
<wgrant> It just needs a LOSA, hopefully.
<poolie> ah, spm is away today
<wgrant> Ahh.
<mthaddon> o/
<poolie> mthaddon, meet wgrant :-)
<mthaddon> yeah, just catching up on backscroll in #launchpad
<wgrant> mthaddon: The main issue is that germanium's publisher is somehow unhappy.
<mthaddon> wgrant: any clues as to where to look?
<wgrant> mthaddon: The logs for cron.publish-ppa might have something interesting.
<wgrant> I don't yet know where they live.
<mthaddon> yeah, looking at that now
<mthaddon> can't see anything untoward in there
<wgrant> Is there anything not untoward there?
<mthaddon> looks like there was issues with claiming the lockfile for a while, but seemed to clear itself up a few mins ago
<mthaddon> definitely seeing stuff processing now - you sure it's still an issue?
<wgrant> Ah, so it is.
<wgrant> It's happening now.
<wgrant> Not quite done, but getting there.
<wgrant> I wonder why it was broken.
<mthaddon> k
<wgrant> Thanks.
<adeuring> good morning
<wgrant> Morning bigjools.
<bigjools> morning
<wgrant> The PPA publisher was MIA for ~2 hours earlier, but mysteriously recovered once mthaddon started poking around.
<bigjools> :/
<bigjools> that happened the other day too
<bigjools>  the librarian went awol
<wgrant> Ah, interesting.
<bigjools> no librarian = sad soyuz
<stub> Are we running Librarian with tokens now?
<bigjools> no idea
<stub> Not sure what has changed with the Librarian apart from that the last few years. Or Apache stuff could be wonky I guess.
<wgrant> We're not.
<stub> Or Squid
<wgrant> It's still webapp proxying.
<mrevell> Bonjour
<jml> poolie: am now
<jml> File "/srv/launchpad.net/production/launchpad-rev-11926/lib/lp/registry/interfaces/person.py", line 172, in validate_person_common obj, getattr(obj, 'name', None))) PrivatePersonLinkageError: Cannot link person (name=canonical, visibility=PRIVATE) to (name=None)
<jml> I get this when I try to make ~canonical a member of ~canonical-data-science.
<jml> What's the deal?
<stub> http://docs.fabfile.org/0.9.3/tutorial.html
<bigjools> jml: did you have any comments about the buildd-manager changes?
<jml> bigjools: none from my skim. glad that BuilderSlave has a multi-file fetch method.
 * jml skims again
<bigjools> jml: and shockingly easy to implement
<jml> bigjools: yeah
<jml> bigjools: I found that once I got the hang of Twisted, a lot of stuff like that just falls into place.
<bigjools> jml: it's starting to click for me now
<bigjools> I need to read up on all the protocol stuff though
<jml> *nod*
<jml> bigjools: I don't think you need files_downloaded  (in packagebuild.py)
<jml> you could also do:
<jml>   d = slave.getFiles(filenames_to_download)
<jml>   d.addCallback(lambda x: self.storeBuildInfo(self, librarian, slave_status))
<jml>   d.addCallback(build_info_stored)
<jml>   return d
<bigjools> possibly, I'll take a look.  I went through many iterations writing that code and I expect there's some cruft
<jml> spot the lie: http://paste.ubuntu.com/532951/
<bigjools> no lie :/
<bigjools> well, maybe a half-truth
<jml> when was the "next Paris summit"?
<jml> :)
<bigjools> yeah :)
 * jml is looking forward to finishing the damn testtools-experiment branch.
<bigjools> thank you again for your tutelage jml
<jml> np.
<jtv> danilos, henninge: I can't find any remaining call sites for migrate_kde_potemplates.pyâ¦  am I missing one?  Or can I just drop it?
<henninge> jtv: no idea but that sounds dropable ;)
<danilos> jtv, I think we had it killed already
<jtv> danilos: yes, I think we didâand this is just a remaining piece.
<jtv> Then there's message-sharing-populate.py.
<danilos> jtv, do away with it then... that too
<danilos> jtv, there was another one, remove-obsolete-translations I think
<jtv> Oh!  And here I just updated that one.  :)
<jtv> That's in something called KNOWN_BROKEN.
<jml> I'm offline to get some more writing, diagramming and strategizing done. (so leet). Available on mobile, back in time for standup.
<lifeless> gmb: hey
<lifeless> gmb: so I wanted to chat with you & deryck about bug subscriptions
<lifeless> gmb: we seem to be a little confused about them. Perhaps a LEP - not about the 'subscribe-to-search' bit - but about subscriptions /per se/ would clarify things?
<lifeless> gmb: I mean, the 'we email dup subscribers' is deeply embedded in the code - we've unpleasant queries that exist just to answer that question.
<gmb> lifeless: I think a LEP to clarify what we actually want to do with subscriptions themselves is a very good idea.
<lifeless> danilos: hi
<lifeless> danilos: I'm replying now, but would love more data about the use case!
<danilos> lifeless, which one? (I am kinda mentioning both :)
<lifeless> danilos: these guys recommend  model by 'what queries do you want to do'
<lifeless> gimme 2 minutes, I'll get this reply to you.
<danilos> lifeless, sure thing
<danilos> lifeless, I have a vague feeling that it wouldn't really work for us at all, but it's worth asking the questions while you are there :)
<lifeless> danilos: hi
<lifeless> danilos: sorry about that, had to move to the training room :)
<danilos> lifeless: training is good for your health :)
<lifeless> :)
<lifeless> danilos: anyhow, does my reply help? Should we discuss it some more?
<danilos> lifeless: sorry, got distracted myself
<bigjools> sinzui: hi - from your bug comment are you volunteering to fix bug 654476?
<danilos> lifeless: paraphrasing for the second use case seems about right
<sinzui> bigjools, I thought I had volunteered a few weeks ago
<bigjools> sinzui: really?  I didn't know we'd narrowed it down to the issue I last mentioned on the bug
<sinzui> bigjools, well, we scheduled the team membership policy fix for this release. I think add a check for PPA as well as other teams takes but a few minutes of work
<bigjools> sinzui: indeed, ok thanks.  I'll re-assign the bug to your team.
<sinzui> okay
<lifeless> bigjools: I'm really happy that the builddmanager is coming along so nicely
<bigjools> lifeless: you are by no means the only one!
<bigjools> lifeless: I have one more scalability issue to talk to you about though
<bigjools> when you've got some free time it would be good to chat in more deptch
<lifeless> bigjools: shoot
<bigjools> oh, free now?
<lifeless> We're doing some simple stuff right now
<lifeless> I can track IRC well enough to get bootstrapped on your issue
<bigjools> lifeless: ok, so, the main issue is that every time we add a new builder, it makes the b-m issue "the" query is issued at regular intervals.  That does not scale.
<bigjools> partly because the b-m is using the master store
<lifeless> what does this query do (in english)
<bigjools> figures out what to dispatch next
<bigjools> *cough* a bit like a queueing system one might say *cough*
<lifeless> do we model the queue directly, or do we model imputs and assemble the queue every time?
<bigjools> there is no queue as such, it issues the query every 15 seconds (was 5 but I slowed it) to see if there's a candidate
<bigjools> so more like the latter
<lifeless> so long term I think we need to overhaul the entire queueing model for buildds
<bigjools> the buildqueue used to model a "queue" but things got quite complicated in the last year ;)
<lifeless> as I think I've said before
<lifeless> to reduce starvation and so forth
<bigjools> it's what everyone has said before I think :)
<lifeless> in the short term
<bigjools> it's not a short-term problem
<bigjools> but when we get those 200 extra builders ...
<lifeless> what about leaving the builder idle until the next 15 second poll
<bigjools> I don't understand what you mean by that
<lifeless> ok
<lifeless> so lets say we had the 200 builders all attached
<lifeless> and nothing to build
<lifeless> they are presumably in some state where we can dispatch a build whenever we want
<bigjools> yes
<lifeless> and then if I upload a build, after processsing etc, your 15 second frequency queue sampling operation will find a build and dispatch it.
<lifeless> bigjools: I'm assuming your query is something like this - psuedo code:
<lifeless> needed_build_requests = len(builder for builder in builders if builder.is_idle)
<lifeless> build_requests = issue_THE_query(needed_build_requests)
<bigjools> no
<lifeless> so that if there are no idle builders you don't issue the query at all
<bigjools> there is a separate query for each builder
<lifeless> bigjools: is it grouped by equivalent builders?
<bigjools> no
<lifeless> like, surely all virtualised i386 builders could be queried at aonce
<bigjools> yes that would be eminently sensible
<lifeless> anyhow, what I meant was
<bigjools> but since forever it has always issued one query per builder
<lifeless> put the builder in whatever state a builder waiting for a build is
<jml> sinzui: hi
<sinzui> hi jml
<lifeless> and let the next lookup after 15 seconds sort it out
<jml> sinzui: thanks for deduping my bug
<jml> sinzui: I was wondering if you know of a work around?
<lifeless> bigjools: but if you're going to do one-per-builder anyway...
<sinzui> jml: maybe make canonical the owner of the team?
<jml> sinzui: even though I can't add canonical to the team?
<lifeless> bigjools: then I think making the modelling be able to answer the query instantly is probably the key thing.
<bigjools> lifeless: so we could mitigate things a lot by remodelling to issue one query per (arch,virt)
<bigjools> the query is already fairly quick to be fair
<bigjools> it's just issued *a lot* :)
<lifeless> bigjools: and add a limit for the number of available builders ?
<jml> sinzui: "The person/team named 'canonical' is not a valid owner for Canonical Data Science CoP."
<lifeless> bigjools: (such that you can short circuit when limit=0)
<bigjools> yes, we'd need to track state carefully - but this is a major, major change to the b-m operation
<lifeless> bigjools: the idea being that that would let you drop it down to checking every 1-2 seconds, for instance.
<bigjools> lifeless: I think that this is a good direction to head.  It's an almost total rewrite of the b-m to do it though :(
<sinzui> jml: yes that is correct, it will not be valid because it will cause a membership change. I think OEM/ISD has employed scripts to add members from one team to another
<jml> sinzui: so, effectively there is no practical work-around to the problem beyond manually adding every member of ~canonical to the new team?
<sinzui> jml: I really do not think there is an acceptable work around. This bug has been a super pain for 18 months
<jml> or scriptedly
<jml> sinzui: ok.
<jml> sinzui: we are going to fix this with the new privacy model, right?
<sinzui> Our pleas for help were largely ignored last year. So while I want to fix this. I cannot say it will make everyone happy.
<sinzui> restating the problem as a membership policy issue, not visibility issue was a break-through for us two months ago
<jml> sinzui: making everyone happy is not a requirement. covering this use case (make a team that's visible to members of another team) is.
<jml> multi-tasking is melting my brain.
<sinzui> The need for team hierarchy will be lessened when users and teams can be permitted to observe without being members. And explaining that members need to guard their messages on mailing lists and contact-this-team emails may suffice
<sinzui> jml: This is a super hard problem. Lots of people have worked on this and it has befuddled everyone.
<lifeless> bigjools: the long term one would be 'be able to answer a single nodes needs very very efficient'y
<lifeless> bigjools: though I do think querying for a set of buuilders needs at once is sensible regardless.
<sinzui> bac checked the TP leaks a few months ago and reported that they were quite bad.
<bigjools> lifeless: totally.  In fact when we start using a real queueing solution we just have one queue per (arch,virt) and be done with it.
<bigjools> lifeless: one of the complications right now is that stuff can be in the queue but we have rules that stop it getting dispatched
<lifeless> oh?
<lifeless> is there lots of that? or just a few jobs at a time?
<bigjools> lifeless: there's two: 1., private sources need to be published, 2. we don't allow PPAs to issue too many concurrent builds
<lifeless> jam: cassandra is basically bzr's store ;)
<deryck> gmb, hi.  In the continuing saga of lazr-js upgrade, I have a clean test run!  hurrahs heard round the world!
<gmb> deryck: Let joy be unconfined.
<lifeless> gmb: so you'll do a LEP then?
<jam> lifeless: well, pack files, but certainly different indexing at the top level, and it isn't multi-writer safe :)
<gmb> lifeless: Yes. I'll start drafting one on Thursday (I'm away tomorrow)
<lifeless> gmb: awesome, thanks!
<lifeless> jam: the indexing is a little different, but not all that much ;)
<jam> lifeless: the in-memory-only hash map is pretty different
<lifeless> jam: its (rowkey, columnname) tuples for the indices AFAICT.
* You're now known as ubuntulog
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
* You're now known as ubuntulog_
* You're now known as ubuntulog
<jcsackett> so, back from vacation and i'm trying to get the branch-lander working again per matsubara's email, but all i get when trying to authorize is a redirect circle between the login page and "confirm sharing these details" page. anyone run into this when the change landed last week?
<matsubara> jcsackett, what's the URL for that page?
<jcsackett> matsubara: login.launchpad.net
<jcsackett> auto-taken there when i run land.
<matsubara> jcsackett, sounds like you're having problem with sso authentication. do other LP pages work for you?
<jcsackett> matsubura: i had thought they were, but you're right, i can't login period right now on this machine (oddly on another one i'm having no problems). i'll fidget with it a bit more.
<jcsackett> thanks.
<jcsackett> ah, botched my ssl configuration.
<bigjools> jml: was there some sort of bug in Trial to do with leaving a dirty reactor sometimes?
<jml> bigjools: it's not a bug in trial, it's a bug in the tests that cause it.
<jml> (or the code that the tests call)
<bigjools> jml: I know that, but I was wondering if there was something in Trial too.  I have a test that's randomly leaving a dirty reactor :/
<bigjools> so far only happens on a full test run
<jml> bigjools: it's almost certainly not a bug in trial.
<bigjools> ok, good to confirm if nothing else
<jml> bigjools: a very common class of test that leaves junk in the reactor does so intermittently
<jml> bigjools: point me at the test and the error. I might be able to spot the problem.
<bigjools> jml: test_binarypackagebuild/test_date_finished_set
<jml> bigjools: in which branch?
<bigjools> jml: the one you reviewed
<bigjools> jml: lp:~julian-edwards/launchpad/async-file-uploads-bug-662631
<jml> bigjools: ta
 * bigjools wishes the dev wiki would redirect back to the page you were on after logging in
<deryck> mars, ping.
<mars> Hi deryck, what's up?
<deryck> Hi mars.  I got my lazr-js upgrade branch passing ec2 now, but I still have horribly long run times....
<deryck> mars, can I send you a log and some notes this morning and see if you have any idea what's going on?
<mars> deryck, do you still see thread trash lying around?
<jml> bigjools: which subclass? (specifically, what is self.build set to?)
<deryck> mars, yes
<bigjools> jml: TestHandleStatusForBinaryPackageBuild
<bigjools> it seems to work for the recipe one
<mars> deryck, I can look, certainly.  Fire the mail over.  I can reply with a HTTP on-the-wire technique I successfully piloted the last time we did this, it should sniff out the off-site accesses, if any are around.
<jml> bigjools: ta
<deryck> mars, ah, ok.  That would be useful to try.
<jml> bigjools: what's the left-over?
<bigjools> jml: http://pastebin.ubuntu.com/533122/
<jml> wow. that's a lot.
<bigjools> I would guess at a timeout
<deryck> mars, thanks for the help!  Two emails fwd'ed to you.
<bigjools> jml: it's probably the getFiles() - it sets up a lot of callbacks
<jml> bigjools: self.buildqueue_record.builder.cleanSlave() returns a Deferred but the code doesn't handle it.
 * bigjools looks
<jml> bigjools: maybe not the bug here, but it's a start.
<jml> bigjools: in _handleStatus_OK
<bigjools> I didn't change any of that
<bigjools> so it was there before
<jml> bigjools: but you've moved more tests to run under trial with the stricter checking
<bigjools> jml: and that same calls is repeated in a few places
<bigjools> and that code is in production right now
<jml> bigjools: perhaps it doesn't matter when the slave gets cleaned, as long as it's eventually
<bigjools> I think it's luck that it's not breaking
<bigjools> the polling loop will wait until it's clean
<bigjools> this is probably the test problem though
<jml> bigjools: I'm inclined to support the luck theory
<bigjools> but why is it intermittent
<jml> bigjools: it's the most obvious part.
<jml> bigjools: because sometimes it finishes before the test does?
<bigjools> ah
<jml> and sometimes it finishes after
<bigjools> mebbe
<bigjools> fun
<jml> yeah
<jml> can't guarantee order of execution without deferreds.
<deryck> mars, I *just* found more combo loading attempts going on.
<jml> ugh.
<mars> deryck, where?
<mars> deryck, outside your search-and-replace sweep?
<deryck> mars, vocab picker uses YUI().use
<jml> 10 personal emails to answer. My friends, family and contributors to my free software projects must hate me :(
<deryck> mars, yeah, missed it somehow.
<deryck> no one hates jml :-)
<jml> :)
<deryck> mars, found a bunch with a more liberal grep now.  Oh, well.  Easy sed fix and maybe this will be finished.
<jml> I think I might declare tomorrow to be LEP day
<jml> in any case, g'night all
<mars> good night jml
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (225): FIXED in 4 hr 3 min: https://hudson.wedontsleep.org/job/devel/225/
<poolie> hello all
<wallyworld> thumper: standup now?
<thumper> wallyworld: hi
 * thumper plugs in
<jcsackett> sinzui: did you ever find that branch regarding lpstats you talked about this morning?
<maxb> matsubara-afk: Were you specifically wanting another lucid build of bzr-pqm, or did you just misclick whilst requesting the maverick one?
<bryceh> Anyone recognize this error?  "Error: Couldn't find a distribution for 'zope.pagetemplate==3.5.0-p1'." http://pastebin.ubuntu.com/533225/
<bryceh> just started seeing it yesterday after merging current db-devel.  not seeing it with the same branch on ec2 though
<mwhudson> bryceh: is your download-cache up to date?
<jcsackett> EdwinGrubbs: do you know anything about the lpstats thing sinzui mentioned this morning? bug 675596
<_mup_> Bug #675596: lpstats queries need to be updated to use usage enums <Launchpad Registry:Triaged> <https://launchpad.net/bugs/675596>
<bryceh> mwhudson, perhaps not (been away for past week)
<sinzui> jcsackett, I am looking from the trunk. I do not know the project name :(
<sinzui> I do not even know who owns it
<jcsackett> sinzui: dig. so it's not just launchpadusagestatistics.py or something, is it?
<bryceh> mwhudson, do I simply delete download-cache/* or is there a command to refresh it?
<EdwinGrubbs> jcsackett: https://lpstats.canonical.com/graphs/
<sinzui> jcsackett, You have never seen the source code
<jcsackett> sinzui: that was what i had assumed (but now what i was hoping. :-P)
<mwhudson> bryceh: bzr up download-cache
<sinzui> jcsackett, https://code.launchpad.net/pgmetrics
<sinzui> jcsackett, I think I am wrong
<jcsackett> sinzui: yeah, this code doesn't look like it could produce that site.
<wgrant> jcsackett, sinzui: Isn't it tuolumne?
<sinzui> wgrant, yes, which is a brilliant name for code that can run on many machines
<jcsackett> sinzui: what's the name mean?
<lifeless> wgrant: pgmegrics is probably the glue for tuolumne for postgresql that we use.
<lifeless> jcsackett: the source code is tuolumne
<lifeless> but I don't remember offhand which  branch we use. flacoste will know.
<jcsackett> lifeless: right, given sinzui's comment i was just curious if the name itself had some meaning. google only reveals a county with the same name.
<mwhudson> jcsackett: that's where the name comes from
<jcsackett> ah.
<mwhudson> the project was created by mthaddon, who was living in california and visiting yosemite at the time
<flacoste> yeah, it's a climber ref
<jcsackett> dig.
<flacoste> jcsackett: you want to read https://wiki.canonical.com/Launchpad/PolicyandProcess/AddLPStatsGraph
<jcsackett> flacoste: already reading. :-P
<flacoste> all you need sohuld be there
<flacoste> if anything is missing or unclear, please help by asking questions and feeding it back into that page
<jcsackett> not quite all--i'm making sure graphs don't break as we migrate the official_* booleans to the *_usage enums.
<jcsackett> rather than adding a graph.
<flacoste> jcsackett: right, but it's the same process
 * jcsackett nods.
<flacoste> and you won't need to add a graph
<flacoste> only update the metrics
<jcsackett> flacoste: yup. sinzui has shown me a branch dealing with similar reqs.
<jcsackett> or a revision, rather.
<wgrant> gmb: You're not still around, are you?
<gary_poster> Hey thumper .  crowberry doesn't need access to memcache, right?
<thumper> gary_poster: what are we putting in memcache?
<gary_poster> thumper: AFAIK it's just used for template snippets right now.
<gary_poster> but I'm asking to make sure :-)
<gary_poster> memcache is blocked from crowberry now, thumper, so this is a "is the status quo OK" question
<gary_poster> (not asking about a change)
<thumper> I think that's fine
<gary_poster> ok thanks thumper
<lifeless> whats on crowberry that is *trying* ?
<wgrant> Could someone EC2 https://code.launchpad.net/~wgrant/launchpad/bug-654372-optimise-domination/+merge/40854?
<lifeless> flacoste: hi
<flacoste> hi lifeless
<lifeless> flacoste: rt 42199 and 40480 should have reversed priorities
<lifeless> flacoste: so that we have more insurance and can switch around without downtime.
<lifeless> oh, bah mea culpa.
<lifeless> flacoste: ignore me, tis fine.
<flacoste> ok
<flacoste> i will
<lifeless> I misread as production ;)
<lifeless> night
<thumper> wallyworld: can you check the code team oopses for yesterday, the TypeError ones for +check-links
<thumper> wallyworld: has this been fixed?
<wallyworld> thumper: will check
<wallyworld> thumper: there's a few from 15th but trying those urls right now results in no oops
<wallyworld> thumper: the stack trace in the oops shows the branch which fixes it wasn't deployed yet
<wallyworld> on the 15th
<thumper> wallyworld: I'd like you to take a look at this: https://bugs.edge.launchpad.net/launchpad-code/+bug/675517 once you've finished the current page you're looking at
<_mup_> Bug #675517: Certain error tracebacks can leak through codehosting <oops> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/675517>
<wallyworld> thumper: ok
#launchpad-dev 2010-11-17
<cody-somerville> wgrant, https://edge.launchpad.net/builders
<thumper> StevenK: ^^
<wgrant> fuuuuuuuuuu
<wgrant> That was meant to be fixed.
<wgrant> And we can't rollback after tomorrow. :/
<wgrant> spm: Around?
<elmo> we can cherrypick rollback
<elmo> he doesn't appear to be
<elmo> what do you need?
<wgrant> We have no StevenK this week, AIUI.
<wgrant> The new buildd-manager is still horribly broken. Can you see if there's anything interesting in its log?
<elmo> yes, all sorts of crfap
<wgrant> Well, that's relatively more opaque than I would have hoped.
<thumper> are we running the new buildmanager then?
<wgrant> We are.
<pjdc> wgrant: are we looking for anything in particular in the buildd manager log?
<wgrant> pjdc: I have a section of it. It's not very helpful.
<thumper> any idea where the log is on devpad?
<thumper> wgrant: do you know which machine the buildd manager runs on?
<wgrant> thumper: cesium
<wgrant> I have bits of the log already.
<pjdc> thumper: looks like they land in devpad:/x/launchpad.net-logs/production/cesium
<thumper> pjdc: yep, there
<wgrant> Twisted errors make me sad.
<wgrant> I wonder if the fermium connection error at 2010-11-17 00:08:13 is the root.
<wgrant> Oh, no.
<wgrant> It was a few minutes before that.
<wgrant> Sad.
<wgrant> Can you see where it started?
<wgrant> pjdc: There have been no known network glitches this morning?
<pjdc> wgrant: not to my knowledge.
<wgrant> :(
<thumper> 010-11-16 00:01:03+0000 [-] <titanium:http://titanium.ppa:8221/> communication failed (User timeout caused connection failure.)
<thumper> 2010-11-16 00:01:03+0000 [-] <titanium:http://titanium.ppa:8221/> failure (None)
<thumper> 2010-11-16 00:01:03+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> communication failed (User timeout caused connection failure.)
<thumper> 2010-11-16 00:01:03+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> failure (None)
<thumper> 2010-11-16 00:01:03+0000 [-] <charichuelo:http://charichuelo.ppa:8221/> communication failed (User timeout caused connection failure.)
<pjdc> hmm, i could check if fermium was airlocked around the
<thumper> 2010-11-16 00:01:03+0000 [-] <charichuelo:http://charichuelo.ppa:8221/> failure (None)
<thumper> 2010-11-16 00:01:03+0000 [-] <thorium:http://thorium.ppa:8221/> communication failed (User timeout caused connection failure.)
<thumper> 2010-11-16 00:01:03+0000 [-] <thorium:http://thorium.ppa:8221/> failure (None)
<thumper> seeing things like that
<thumper> hmm...
<thumper> perhaps a pastebin would be better
<wgrant> thumper: Isn't that >24 hours ago?
<thumper> wgrant: yep, I'm looking for failures
<thumper> hmm...
<thumper> buildd-manager.log-20101116 has up to 2010-11-16 00:01:05+0000
<wgrant> Yeah, they're named sort of wrongly.
<thumper> buildd-manager.log starts at 2010-11-17 00:00:45+0000
<wgrant> They're named after the day they're rotated.
<thumper> and no log file in the middle
<wgrant> There's no -20101117?
<thumper> nope
<wgrant> Um.
<thumper> pjdc: can you see a -20101117 file on cesium?
<pjdc> thumper: looking
<pjdc> thumper: yes, there's a buildd-manager.log-20101117
<thumper> pjdc: can you get that to devpad plz?
<thumper> according to the graph, we started loosing builders approx 12 hours ago
<wgrant> Does LPS say when cesium was updated?
<pjdc> thumper: landed in /tmp
<thumper> pjdc: ta
<thumper> Scanning failed with: <Fault 8002: 'error'> <-- look suspect
<wgrant> That's fairly normal.
<thumper> Unhandled error in Deferred:
<wgrant> That's not.
<thumper> that's probably not
<thumper> 2010-11-16 23:51:34+0000 [-] builder promethium failure count: 5, job 'amd64 build of widelands 1:15bzr5723-ppa1~natty1 in ubuntu natty RELEASE' failure count: 1
<thumper> 2010-11-16 23:51:34+0000 [-] Scanning failed with: User timeout caused connection failure.
<wgrant> There are hundreds of those.
<wgrant> And that's not the first.
<thumper> Failure: twisted.internet.defer.CancelledError:
<thumper> ??
<thumper> no
<thumper> it isn't the first
<thumper> I'm looking for suspect lines
<thumper> is there a log entry for disabling a buider?
<wgrant> There used to be.
<wgrant> I never got around to reviewing the full 6000 line diff of this branch, though :/
 * wgrant looks.
<thumper> Miserable failure when trying to examine failure counts: :-)
<wgrant> .... wow.
<wgrant>         except:
<wgrant> GRGWOGIFJEWF
<wgrant> Anyway, what was the miserable failure?
<wgrant> thumper: You'll be pleased to know that not all failBuilder callsites log.
<thumper> wgrant: I'm grepping
<thumper> wgrant: I can kinda tell
<thumper> I suppose we should do a launchpad status thingy
<wgrant> But we should be able to tell from the logs.
<wgrant> Since only two of the callsites are unobvious, and one has enough logging around it that we should be able to work it out.
<thumper> wgrant: can you see devpad?
<wgrant> thumper: Not for another few weeks :(
<thumper> just wondering, I'll copy the logfile to people.c.c
<wgrant> Thanks.
<thumper> hmm...
<thumper> I'm getting refused ssh
<wgrant> That's excellent.
<thumper> hmm... something screwy is going on...
<thumper> ok, it's on its way up
<thumper> wgrant: I'm not entirely sure what to look for
<wgrant> thumper: Thanks.
 * wgrant examines.
<wgrant> It's tempting to add more explicit logging, restart it, and hope it breaks again.
<thumper> test rollout of 11926 on 2010-11-16
<thumper> that's from LPS
<wgrant> I was hoping for higher granularity. But I guess the log should help with that.
<thumper> bug https://bugs.launchpad.net/soyuz/+bug/671242
<_mup_> Bug #671242: New buildd-manager disabling everything in sight <qa-ok> <Soyuz:Fix Committed by julian-edwards> <https://launchpad.net/bugs/671242>
<wgrant> 10:03, by the look of things.
<wgrant> thumper: Right, 11888 was deployed, but that broke with that bug.
<wgrant> 11926 fixed that.
<wgrant> But we now have another one.
<thumper> this is a different problem?
<wgrant> Yes.
<wgrant> I believe.
* thumper changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: buildd-manager is disabling things again | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<wgrant> I think this problem may be partially described in that bug, but it's not the one that was identified and fixed.
<wgrant> So, we have major problems at 14:41:45, 15:03:20 and 16:31:22, 23:39:11, and possibly 18:11:22, 19:52:18,
<wgrant> 23:39:11 is the big one which took out everything.
<wgrant> Each major failure starts with a single scan failure, than a huge number 9 seconds later.
<thumper> elmo: ping?
<wgrant> He left a while ago.
<thumper> :(
<thumper> I think we are losaless
<wgrant> pjdc should be around, though?
<thumper> pjdc: what do you know about LP deployment?
<wgrant> If you want to revert, we don't need a full deployment. It's just a symlink change and restart of buildd-manager.
<pjdc> thumper: not much. i assisted elmo with an emergency cowboy about a year ago, but that's about it.
<thumper> :(
<thumper> lifeless: are you really there?
<thumper> pjdc: on cesium there was a rollout yesterday
<thumper> pjdc: I'm hoping that they kept the old code around
<wgrant> thumper: Last time that was the case.
<thumper> pjdc: as the new code is disabling all the builders
<wgrant> thumper: A LOSA just flipped the symlink back to the 10.10 rollout.
<wgrant> soyuz-production-rev-9886
<thumper> wgrant: if the buildd-manager is restarted, will it recheck the disabled buiders?
<pjdc> thumper: if i'm looking in the right place, there are two tree, 11926 and 9886
<lifeless> thumper: yes
<pjdc> thumper: 9886 looks pretty old
<thumper> pjdc: 11926 is the broken one
<thumper> pjdc: 9886 will be from db-devel
<thumper> so... probably the last rollout
<thumper> Wed 13th of Oct
<thumper> that's the date on rev 9886 of db-devel
<wgrant> thumper: No, we'll have to flip a flag on each to get them back.
<wgrant> 9886 is what we reverted to after 11888 failed. It's the last rollout.
<thumper> pjdc: can you do that?
<thumper> wgrant: how do we re-enable the builders?
<pjdc> thumper: so that'd be change the symlink, restart the service?
<thumper> pjdc: AFAIK
<pjdc> thumper: i take it things can't get worse at this point?
<lifeless> thumper: again? nooo
<lifeless> this bodes badly for the rollout tomorrow.
<lifeless> like really.
<lifeless> thumper: 9886 is fine, its the last db-stable  deploy
<lifeless> wgrant: you got log files etc - whats up ?
<lifeless> it was stable for hours - did this happen recently? could it be librarian changes? [hope not]
<lifeless> .
<wgrant> Yeah, cesium is as broken as it can be.
<wgrant> lifeless: I have the log change.
<wgrant> s/change//
<lifeless> whats causing this
<lifeless> before we change stuff
<wgrant> It was mostly OK for 4 hours.
<wgrant> After 13 hours it just completely melted down.
<wgrant> Still looking to see if I can get anything useful from the logs.
<lifeless> can we just restart it in the meantime and toggle the builds back on ?
<wgrant> thumper: The 'Builder OK' flag on Builder:+edit does it. Otherwise there might be a script around.
<lifeless> or will it kill them immediately ?
<wgrant> lifeless: I guess we could try that.
<lifeless> we need to figure out what happens tomorrow @ rollout time
<lifeless> which is what, 8 hours away
<thumper> lifeless: right now it is a release blocker IMO
<thumper> I'm starting an incident report
<wgrant> lifeless: I think we probably cherrypick 11808, 11815 and 11926 off cesium.
<lifeless> wgrant: 11926 itself is a problem ?
<wgrant> lifeless: No, but 11808 probably won't revert unless we revert 11926 first.
<wgrant> Er.
<wgrant> Not 11926.
<wgrant> That other one.
<lifeless> sorry, I left my mind reader in asia
<wgrant> The fix for the issue that caused us to roll cesium back from 11888.
<wgrant> 11898
<wgrant> So 11808, 11815 and 11898.
<wgrant> Do we know if enablement has pulled any buildds today?
<lifeless> cody-somerville: ^
<pjdc> wgrant: it's been quiet since the 12th, as far as i can tell
<wgrant> It doesn't explain everything (since non-virt buildds had the same error), but it might be something.
<lifeless> wgrant: whats the error
<wgrant> lifeless: 2010-11-16 16:31:13+0000 [-] Scanning failed with: User timeout caused connection failure.
<wgrant> 2010-11-16 16:31:13+0000 [-] Traceback (most recent call last):
<wgrant> 2010-11-16 16:31:13+0000 [-] Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.
<wgrant> lifeless: In most of the major failures in the log, there is one of those followed by dozens 9 seconds later.
<thumper> lifeless: do you object to rolling back the buildd-manager code on cesium?
<wgrant> Perhaps we should try enabling things and see if they stay alive for long.
<mwhudson> does the buildd-manager still do blocking things?
<lifeless> thumper: I want to be sure we understand it
<wgrant> Only when downloading files from slaves, I believe.
<thumper> mwhudson: I believe it is twisted now
<wgrant> mwhudson: ^^
<lifeless> pjdc: can you please:
<thumper> wgrant: I thought that jelmer fixed that
<mwhudson> thumper: fully?  it's been somewhat twisted for a long time
 * thumper needs food
<lifeless>  - restart the builddmanager
<wgrant> thumper: jelmer fixed it so it uploads the downloaded files asynchronously.
<lifeless>  - reenable a couple of fast buildds
<lifeless>  - see what happens over a few minutes
<wgrant> thumper: A branch is coming to download them async, too, but it's not done yet.
<lifeless> thumper: go eat, nothing will change radically while you eat
<pjdc> i'm not too familiar with the buildd pool. can someone suggest candidates?
 * pjdc picks 3 amd64 official builders
<wgrant> pjdc: A fairly random pick of the various categories: roseapple, allspice, doubah, samarium
<wgrant> Couple of new non-virt, and an old and new virt.
<pjdc> works for me
<pjdc> restarting buildd-manager
<pjdc> started, doing the buildds now
<wgrant> Hm.
<wgrant> Maybe we should turn the logging up.
<wgrant> (lib/lp/buildmaster/manager.py, s/logging.INFO/logging.DEBUG/)
<pjdc> re-enabled those four, plus yellow and crested since i had the tabs all ready
<wgrant> Great.
<wgrant> There are some odd five minute gaps in the log, and it would be nice to know if it actually does anything in them.
<thumper> pjdc: the queue for amd64 is empty though
<thumper> not sure if that'll show much
<pjdc> thumper: looks like doubah's done the business though, showing as disabled again
<wgrant> The builders were failed regardless of whether there was anything to build or not.
<wgrant> Oh, already?
<thumper> oh ok
<lifeless> wgrant: does the buildd manager *read* from the librarian ?
<wgrant> lifeless: I don't think so.
<wgrant> I can't think why it would.
<lifeless> do builders ?
<wgrant> Yes.
<lifeless> what code path do they use to get their urls ?
<wgrant> Ahh. cesium provides them, I believe.
<wgrant> But it doesn't use the restricted librarian.
<lifeless> even for security builds etc?
<wgrant> No -- private build files are retrieved from the archive.
<wgrant> Since the builders can't have restricted librarian access.
<wgrant> (well, I guess they could now)
<lifeless> just to be sure
<lifeless> pjdc: are we seeing access denied for 91.189.89.189 or 91.189.89.188 from the builders that fail (or from cesium for that matter)
<lifeless> wgrant: what time was the first builder disabled ?
<wgrant> lifeless: Can't tell. But the first major incident was probably 14:41:45. 23:39:11 was the really big one.
<pjdc> lifeless: cesium can't connect to those IPs on 80 and 443
<elmo> oh
<elmo> is this the restricted librarian
<lifeless> pjdc: ah, but is it *trying*
<elmo> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
<elmo> god damn it
<pjdc> lifeless: i'll check
<lifeless> elmo: there are two code paths for it in lp. internal stuff like the merge proposal diff code will use lp.internal still
<wgrant> cesium will be wanting to upload to it. Is that also going to be broken?
<elmo> 188     IN      PTR     wildcard-restricted-launchpadlibrarian-net.banana.canonical.com.
<lifeless> elmo: I really really wouldn't expect this to be connected, but Just In Case.
<lifeless> wgrant: no, uploads have not altered *at all*
<wgrant> That's what I thought.
<lifeless> wgrant: are those times UTC?
<wgrant> lifeless: Yes.
<lifeless> ok, so 9 hours apart.
<wgrant> There were some in between.
<lifeless> (range of) unless we're seeing two different things
<lifeless> then its not the publicrestricted librarian work
<wgrant> When did that happen?
<pjdc> lifeless: a few rejects (5 total) for cesium going to both IPs on 80 and 443
<lifeless> wgrant: and really? sed to change log levels ?
<wgrant> lifeless: Yup :D
<lifeless> pjdc: thats -very- interesting
<lifeless> wgrant: I can has bug, and fix.
<lifeless> elmo: did you just remove the publicrestricted feature flag?
<pjdc> lifeless: sorry, false alarm. those were my tests attempts.
<lifeless> elmo: from https://launchpad.net/+feature-rules
<elmo> lifeless: I haven't touched anything
<lifeless> interesting
<lifeless> cause the setting is gone ;)
<wgrant> Hmm.
<wgrant> It's possible that translations jobs might read from the librarian.
<wgrant> I don't know them well.
 * wgrant looks.
<wgrant> But they wouldn't be private.
<lifeless> pjdc: did you?
<lifeless> nb we really need that audit log
<pjdc> lifeless: sorry, did i what?
<lifeless> sounds like a no to me
<lifeless> ok, one thing at a time
<lifeless> uhm
<wgrant> We should probably have an 'assert not libraryfilealias.restricted' in BuilderSlave.cacheFile.
<lifeless> pjdc: please reenable doubah again.
<wgrant> But I doubt that's the problem here.
<pjdc> lifeless: ok
<lifeless> wgrant: or enable it
 * thumper has to go get kids
<wgrant> lifeless: Hm>
<lifeless> pjdc: if this fails, its not the publicrestricted librarian.
<pjdc> lifeless: doubah re-enabled
<elmo> https://launchpad.net/ubuntu/+source/fglrx-installer/2:8.780-0ubuntu3/+build/2049941 <-- roseapple worked (for one build) - did we know that?
<wgrant> Yeah, most things work for a while.
<elmo> ok
<wgrant> It's not related to what the builder is doing at the time.
<wgrant> It may be related to what others are doing, but who knows.
<lifeless> wgrant: how do we know that?
<wgrant> lifeless: Because it affects dozens of builders at a time, whether they're idle or building recipes or building binaries.
<wgrant> doubah's gone.
<lifeless> wgrant: what if the timeout or some such is pseudo global, and one hung builder breaks all the ones open for the time window involved
<wgrant> lifeless: Exactly.
<wgrant> 12:59:04 < wgrant> It may be related to what others are doing, but who knows.
<wgrant> So, doubah is dead with a TCP timeout.
<lifeless> wgrant: in which case its related to what *one* does
<wgrant> I wish we had a traceback.
<wgrant> It would be sorta helpful to know what timed out.
<wgrant> lifeless: buildd-manager won't have cached the old FF?
<lifeless> wgrant: it wasn't restarted when the problem happened
<wgrant> True.
<wgrant> Can we disable everything, enable doubah, and see what happens?
<lifeless> wgrant: it was 3 hours ago now that the ff was turned on (and apparently off again)
<wgrant> Oh, so ages after the world exploded. I see.
<lifeless> wgrant: yeah, I'm convinced we're clear
<mwhudson> lifeless: Chex turned the feature flag off after i complained that private codebrowse wasn't working
<lifeless> mwhudson: did that make it work?
<mwhudson> (which seems entirely unrelated to me, but after he did it, private codebrowse started working again)
<lifeless> wtf
<mwhudson> yes
<lifeless> codebrowse uses the librarian?
<mwhudson> i'm betting some kind of coincidence
<lifeless> pjdc: can you please turn the flag on again - its listed under sql queries etc on LPS.
<pjdc> lifeless: that's a long page. what am i looking for exactly?
<lifeless> publicrestrictedlibrarian default 0 on
<pjdc> that doesn't mean much to me. is that a command?
<lifeless> its a line you put in https://launchpad.net/+feature-rules
<lifeless> https://dev.launchpad.net/LEP/FeatureFlags has plenty of docs - see the bottom o the page in partiuclar
<pjdc> ah, ok. so bung it in at the end, hit "Change"?
<lifeless> yes
<pjdc> done
<lifeless> mwhudson: still working ?
<mwhudson> lifeless: will check
<lifeless> 'Server denied check_authentication' is what you saw?
<mwhudson> lifeless: yes
<lifeless> zomg
<lifeless> pjdc: and remove it?
<mwhudson> lifeless: it works now
<lifeless> oh
<lifeless> pjdc: don't remove it
<mwhudson> lifeless: oh, it failed for you?
<pjdc> lifeless: ok, doing nothing :)
<lifeless> mwhudson: failed once
<lifeless> mwhudson: worked on second url
<lifeless> I think its stubs openid change
<mwhudson> lifeless: random
<lifeless> mwhudson: so coincidence that that was the first private branch url you tried since 11926 was deploy.
<lifeless> pjdc: thanks
<lifeless> ok
<lifeless> so back to the buildd
<lifeless> pjdc: did doubah do day carooba?
<wgrant> So, I'd like to see this happen:
<wgrant>  - Disable all builders.
<wgrant>  - Shut down buildd-manager.
<pjdc> lifeless: you lost me at "do
<wgrant>  - Change log leve.
<pjdc> "
<wgrant>  - Enable doubah.
<wgrant>  - Start buildd-manager
<lifeless> pjdc: you reenabled doubah
<lifeless> pjdc: did it die again?
<wgrant> It did.
<lifeless> ok
<lifeless> pjdc: could you do what wgrant just described
<pjdc> disable all builders incluing those currently building? or just the idle ones?
<wgrant> All, ideally.
<lifeless> all
<lifeless> we're wondering if doubah is broken
<pjdc> the alls have it
<lifeless> and a bug is making all the others get nuked when it goes *if* they happen to be lined up with it in the polling period
<wgrant> Well, I'm mostly hoping we can get a minimal case to fail.
<wgrant> I doubt there's anything wrong with doubah.
<lifeless> ok. I'm thinking that.
<lifeless> what is doubah - virt i386?
<wgrant> Since I picked four semi-randomly from a pool of 60.
<wgrant> Yeah.
<wgrant> A fairly beefy one, too.
<pjdc> all disabled, stopping the buildd-manager
<pjdc> wgrant: how is the log level changed?
<wgrant> pjdc: Aherm.
<wgrant> pjdc: s/logging.INFO/logging.DEBUG/ in lib/lp/buildmaster/manager.py.
<lifeless> wgrant: you *are* going to fix that.
<wgrant> lifeless: It is Twisted evil.
<wgrant> Which I don't know awfully well.
<lifeless> hasn't stopped you in the past
<wgrant> True.
<pjdc> wgrant: like so? http://paste.ubuntu.com/533321/
<wgrant> pjdc: Yup.
<lifeless> wgrant: and you have help
<wgrant> Flip doubah back on, and start b-m up.
<wgrant> And let's hope it fails.
<pjdc> doubah enabled, b-m starting
<lifeless> wgrant: what time do recipe builds auto create ?
<wgrant> lifeless: Probably a couple of hours ago.
<pjdc> b-m started
<wgrant> lifeless: They were happening around the time it was noticed.
<wgrant> Er.
<wgrant> doubah's dead already?
<wgrant> Hm, no.
<wgrant> Must have been cached.
<pjdc> shows as building here
<wgrant> Yeah, it is now.
<wgrant> OK, it's started.
<wgrant> Only died once.
<wgrant> :( it seems to be happy.
<lifeless> should we bring up another virt i386 ?
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      230 /   59  Person:+commentedbugs
<lifeless>      111 / 5615  Archive:+index
<lifeless>       76 /  295  BugTask:+index
<lifeless>       12 /  398  Distribution:+bugtarget-portlet-bugfilters-stats
<wgrant> Worth a try.
<lifeless>       12 /  341  Distribution:+bugs
<lifeless>       10 /    5  Person:+bugs
<lifeless>        9 /    7  ProjectGroup:+milestones
<wgrant> (virt i386 is good because it gives us all job types)
<lifeless>        8 /    2  BugTask:+create-question
<lifeless>        5 /   47  Distribution:+archivemirrors
<lifeless>        5 /   17  DistributionSourcePackage:+publishinghistory
<pjdc> wgrant: shall i enable actinium then?
<wgrant> pjdc: Sure.
<pjdc> wgrant: ok, enabled
<wgrant> Really?
<wgrant> Maybe I'm on a slave, but actinium looks dead.
<wgrant> If it has just died, this is great news indeed.
<pjdc> looking dead here too
<wgrant> !!!
<wgrant> We may have some hope of untangling the logs this time.
<wgrant> Could you throw the log since the restart somewhere I can see it?
<pjdc> will do
<pjdc> see query
<wgrant> Thanks.
<wgrant> ...
<wgrant> 2010-11-17 02:26:50+0000 [Uninitialized] ForbiddenAttribute: ('build', <TRANSLATION_TEMPLATES_BUILD branch job (2146072) for ~gwibber-committers/gwibber/trunk>)
<thumper> ???
<wgrant> This logging is a lot more descriptive :)
<wgrant> Hm, so actinium was aborted.
<wgrant> It was resumed, then just a few seconds later a dispatch was attempted... that's far too quick.
<wgrant> So, actinium probably wasn't hit by the root issue. :(
<wgrant> We don't wait long enough for the resume to complete.
<wgrant> But that doesn't explain the 'User timeout caused connection failure' thing, or why non-virt builders were broken too.
<wgrant> OK. I think we should try to get it to break horribly again. So we should reset the failure counts and reenable everything, I suppose.
<wgrant> Sigh.
<lifeless> if we bring everything up
<lifeless> will we log useful data?
<wgrant> I hope so.
<wgrant> Maybe we should make failBuilder log before we do that, though.
<wgrant> So we can see when things are disabled.
<lifeless> wgrant: can you prep a cowboy
<wgrant> Doing so.
 * thumper has to head afk
<wgrant> pjdc, lifeless: http://pastebin.ubuntu.com/533329/ should do it.
 * wgrant pelts buildd-manager with rocks and sets it on fire.
<pjdc> wgrant: so, shut down, apply patch, enable all (this might take a while), start up?
<wgrant> pjdc: Yup.
<wgrant> pjdc: Do you know if there's a script to enable them all?
<wgrant> Otherwise there's SQL...
<pjdc> wgrant: no idea, i've only ever done them manually
<wgrant> We may need SQL to reset the failure counts anyway. We'll see shortly.
<pjdc> b-m stopped, patch applied
<lifeless> wgrant: api script :)
<wgrant> lifeless: Yeah, yeah, on my todo list.
<wgrant> It's reasonably unfortunate that all this has happened when we have no available LOSAs in this TZ, no available Soyuz developers in this TZ, and both of the buildd admins in this TZ also unavailable.
<lifeless> s/unfortunate/normal/
<wgrant> No... We'd normally have a LOSA, a Soyuz developer, and two buildd admins.
<pjdc> okay, that's all of them enabled
<pjdc> anything else before b-m is started?
<wgrant> So we are now running with the log level change and the additional failBuilder logging?
<pjdc> yep, left the loglevel change in place, and applied your cowboy
<wgrant> Start it up!
<wgrant> I expect most of the them will disable themselves again in about 30 seconds :(
<pjdc> started
<wgrant> :(
<wgrant> So, everything seems to be happy now.
<wgrant> I guess we just leave it until it explodes in a few hours, and hope the new logging tells us something useful.
<pjdc> that shouldn't be far off when the UK wakes up, so that might work out
<wgrant> Given that we've failed to reproduce it elsewhere, it is tempting to let the rollout go ahead and just automatically undisable builders until we work out what's going on :/
<wgrant> The 14 builders that are disabled now probably just need their failure count reset (it's already over the threshold, so the initial failure to connect because the builder is still resuming causes them to be disabled).
<wgrant> Something like this:
<wgrant> UPDATE builder SET failure_count=0, builderok=true WHERE name IN ('hawthorn', 'actinium', 'hassium', 'lansones', 'muntries', 'radium', 'rosehip', 'sandpaperfig', 'terranova', 'fermium', 'lawrencium', 'nobelium', 'papaya', 'plutonium');
<pjdc> if it's not critical, that's probably best left for a losa
<wgrant> Probably, yeah.
<wgrant> Not critical. Just makes it harder to see if it's broken without watching logs.
<wgrant> Thanks for your help.
<pjdc> you're welcome!
<wgrant> adare and ross are now broken in other ways :(
<wgrant> But that can wait.
<lifeless> mwhudson: https://bugs.launchpad.net/launchpad-foundations/+bug/676372
<_mup_> Bug #676372: "Server denied check_authentication" from bazaar.launchpad.net private branch since 11926 deployed <regression> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/676372>
<poolie> hi all
<poolie> i am running './bin/test' in a vm, and it has been stuck for hours, with the last output being
<poolie> Started ['/tmp/tmpecWY0y.mozrunner/mozrunner-firefox', '-profile', '/tmp/tmpecWY0y.mozrunner', 'http://bugs.launchpad.dev:8085/windmill-serv/start.html']
<poolie> in 1.109 seconds.
<poolie>  
<poolie> halp?
<wgrant> Is there a firefox window lurking around?
<poolie> not that i can see
<poolie> i'm ssh'd in to the vm without -X
<poolie> i will see if there's a firefox process
<poolie> there is not, though there is a zombie
<wgrant> mthaddon: Around yet?
<henninge> Hi wgrant!
<henninge> ;)
<wgrant> Morning henninge.
<henninge> wgrant: heard you got engaged
<wgrant> Oh?
<henninge> That Kate really is a nice girl
<henninge> oh sorry, wrong W... ;-)
<henninge> wgrant: What's that about the buildmanager?
<wgrant> henninge: Well, it may or may not be a release blocker.
<wgrant> henninge: We have a not utterly terrible workaround, so it's probably OK.
<henninge> wgrant: what does the workaround include?
<wgrant> henninge: Uhh, frequently reenabling all the builders manually.
<henninge> ;-)
<henninge> How frequently?
<wgrant> Unsure. It was OK for 4 hours yesterday. And it's been OK for 4 hours so far today.
<henninge> hours, wow ...
<henninge> The affected code is on cesium, right?
<wgrant> Yeah.
<wgrant> Hopefully jml and bigjools will save the world tonight.
<henninge> So that is (again) part of the nodowntime hosts
<henninge> so a fix can be deployed any time.
<wgrant> Yeah.
<henninge> wgrant: I am sure they will! ;-)
<wgrant> So, it's a pretty terrible bug, but we can work around it easily enough with a script.
<henninge> The only reason I can imagine this being a blocker for the roll-out would be if any fix would include db changes.
<henninge> which is not that far fetched, I guess.
<wgrant> It won't.
<poolie> wgrant: congrats!
<wgrant> poolie: Hm?
<poolie> or is he just totally confused?
<wgrant> I hope he's just totally confused.
<poolie> ah, me clicks
<wgrant> Or there's some news about me that I don't know.
<poolie> William Soxe-Gotha-Coburg-Windsor
<wgrant> Ahhhhhhh, of course.
<bac> hi henninge
<henninge> hi bac!
<adeuring> good morning
<henninge> poolie, wgrant: Yeah, I messed up the joke. I meant to say "sorry, wrong prince" ... ;)
<henninge> Moin adeuring!
<adeuring> hi henninge
<wgrant> Haha.
<wgrant> bigjools: Morning...
<bigjools> morning
<wgrant> bigjools: Have you heard the wonderful news?
<bigjools> which?
<wgrant> bigjools: We're about to release with a pretty screwed buildd-manager :)
<bigjools> fuck sake
<wgrant> It disabled 60 or so this morning.
* mthaddon changed the topic of #launchpad-dev to: Launchpad down/read-only from 10:00-12:00 UTC for DB update | Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: buildd-manager is disabling things again | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<wgrant> It seems to be reasonably happy now, since we restarted everything 7 hours ago.
<wgrant> But it was OK for a few hours yesterday too :/
<bigjools> it was disabling builders because they were unresponsive
<bigjools> it's supposed to do that
<wgrant> TCP timeouts and no route to host errors are different.
<bigjools> how?
<wgrant> This was "User timeout caused connection failure" or something like that.
<bigjools> that's because they don't respond within the timeout
<wgrant> Dozens of them in one second?
<bigjools> what sort of time did this happen?
<wgrant> 14:41:45, 15:03:20 and 16:31:22, 23:39:11 are some that I saw.
<wgrant> 23:39:11 was the big one.
<bigjools> that's when the daily recipes kick off
<wgrant> But the last two incidents there start with a single error, then 9 seconds later dozens.
<mrevell> Hello
<wgrant> There were also a few other odd errors in the logs.
<wgrant> And it's not waiting long enough for builders to resume.
<wgrant> But apart from that it's happy now.
<bigjools> that's a problem because there's nothing we can do to fix that
<wgrant> Hmm?
<bigjools> the connection timeout is hard-coded in the python libs :/
<wgrant> Odd...
<bigjools> the reset script waits until some event in the builder, which is supposed to be when it's ready to accept a connection
<bigjools> then that connection often times out
<wgrant> 2010-11-17 02:35:58+0000 [QueryProtocol,client] Resuming actinium (http://actinium.ppa:8221/)
<wgrant> 2010-11-17 02:36:04+0000 [-] Asking builder on http://actinium.ppa:8221/filecache to ensure it has file chroot-ubuntu-lucid-i386.tar.bz2 (http://launchpadlibrarian.net/51974282/chroot-ubuntu-lucid-i386.tar.bz2, d267a7b39544795f0e98d00c3cf7862045311464)
<bigjools> we're seeing the fruits of that now because I am actually disabling stuff
<wgrant> 2010-11-17 02:36:25+0000 [Uninitialized] Scanning failed with: TCP connection timed out: 110: Connection timed out.
<bigjools> whereas the old one never disabled anything
<wgrant> It waited 6 seconds from firing the resume trigger.
<wgrant> Maybe the script is buggy.
<bigjools> no
<bigjools> 6 seconds is about right
<bigjools> they reset very quickly
<wgrant> The VM is created and boots in 6 seconds!?
<bigjools> yes
<wgrant> Nice.
<bigjools> the first connection is to send the chroot, and that's why you see it timing out
<bigjools> we can get around this for now by removing the code that fails builders
<bigjools> which is essentially what the old b-m was not doing
<wgrant> I think we need to disable failure counting.
<wgrant> It took out lots of builds as well.
<wgrant> (and fourteen or so builders need their failure counts manually reset)
<bigjools> digh
<wgrant> I still find it unlikely that dozens of builders failed to respond all in the same second, several times, unless there were network glitches that nobody knows about.
<wgrant> The 9 second delay betwen the first failure and subsequent stream on at least two occasions is also rather suspicious.
<bigjools> if it's a network glitch then it's more likely that they all go at once
<wgrant> Anyway, cesium is currently running the new code with two cowboys: one setting loglevel to DEBUG, and another to log whenever a builder is failed.
<wgrant> We also need to fix the failure counts of those builders, and probably do a mass-giveback :/
<bigjools> failure counts are reset on a successful dispatch
<wgrant> They are.
<wgrant> Hmm.
<bigjools> for a builder to get failed it has to go wrong on 5 consecutive occasions
<wgrant> But the issue is that the first failure will immediately knock them out again.
<bigjools> no, that's not true
<wgrant> It will, since the count is currently 5.
<wgrant> We reenable, they time out, and are immediately disabled.
<wgrant> No five strikes rule for them.
<bigjools> ok, re-enabling should reset the count
<bigjools> that's a bug
<wgrant> It should.
<wgrant> But it doesn't.
<wgrant> And we were LOSAless today, so we couldn't do it manually.
<bigjools> I think the recipe builds are thoroughly screwing the builders
<wgrant> So "User timeout caused connection failure" occurs when the TCP connection is accepted, but there's no HTTP response?
<bigjools> everything works fine until they come along
<bigjools> that happens when the connect() fails
<wgrant> We're still running the old lp-buildd with in-chroot bzr-builder, aren't we?
<bigjools> yes, we rolled them back
<wgrant> If that happens when connect() fails, then why this:
<wgrant> "TCP connection timed out: 110: Connection timed out."
<wgrant> That's a separate error.
<bigjools> I think I'm going to just remove the failure counting stuff for now
<wgrant> Sounds like a good idea.
<bigjools> wgrant: did you ask someone to restart it at 0126 UTC?
<wgrant> I think the first one was lifeless, but yeah, it was around then.
<bigjools> there were no problems with it at that time
<wgrant> It had taken out all but a few buildds an hour earlier. We wanted to see if we could reproduce it fresh with just a couple of active builders, to see if we needed to roll back and work out what to do about the release.
<bigjools> I think the problem is recipe builds for sure, I just need to reproduce on DF
<bigjools> the builder is doing something that makes it unresponsive
<wgrant> That's not the whole thing.
<wgrant> palmer was disabled. It is non-virt and had been idle for 30 minutes.
<bigjools> hmmm
<bigjools> oh jeez the log is massive with debug on
<wgrant> So we knew it was either several undetected network glitches throughout the day manifesting without any TCP timeouts, or something with one builder was glitching everything else out.
<wgrant> So we turned up logging and hoped it would reappear, since the INFO logging is sort of completely sparse.
<wgrant> We can't tell when the problematic scans were triggered, and there are five minute gaps in the log :/
<wgrant> And I can't reproduce it locally however much I try :(
<bigjools> it's a nightmare
<wgrant> Yeah, just a bit.
<bigjools> from the log, it starts going wrong at the exact same time the faily (sic) recipe builds get kicked off
<bigjools> around 23:35Z
<wgrant> That's the big incident, yeah.
<wgrant> But there are several smaller ones in the preceding 9 hours.
<bigjools> other indicents are almost certainly another batch
<wgrant> Possibly.
<bigjools> there are some Fault 8002:
<wgrant> Yeah, but they're everywhere...
<bigjools> that's a protocol fault
<bigjools> hmmm /me sees something
<wgrant> What has been seen?
<bigjools> this might have something to do with the huge blocking file fetch
<wgrant> I considered that.
<wgrant> But the 23:39 incident suggests not.
<wgrant> The nearest fetch before that was about 6 minutes earlier.
<bigjools> I think it's a number of different things that cause blocks
* henninge changed the topic of #launchpad-dev to: Launchpad down/read-only from 10:00-12:00 UTC for DB update | Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 (but closed during the roll-out)| firefighting: buildd-manager is disabling things again | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<wgrant> bigjools: So, just going to cowboy out failure counting after the rollout and hope that we can work it out?
<bigjools> yes
<wgrant> :/
<bigjools> one of the things that the failure counting did was to remove in-progress jobs from builders if they failed a poll
<bigjools> I might have to rethink how that work
<bigjools> s
<bigjools> damn, this stuff is hard
<wgrant> It should all be fine.
<bigjools> "should"
<wgrant> Except for those unexplained User blah blah blah errors, and the reset script lying.
<wgrant> Apart from that and the occasional other translations exception, it seems to be OK.
<wgrant> 2010-11-17 02:26:50+0000 [Uninitialized] ForbiddenAttribute: ('build', <TRANSLATION_TEMPLATES_BUILD branch job (2146072) for ~gwibber-committers/gwibber/trunk>)
<wgrant> That's the translations exception.
<bigjools> sigh
<wgrant> Yes.
<wgrant> Does the reset script wait until the slave responds to HTTP?
<wgrant> How hard is readonly bazaar.launchpad.net?
<wgrant> Surely not that bad?
<lifeless> wgrant: we tested it on qastaging yesterday. it works with one small bug
<lifeless> wgrant: however, we're doing machine maintenance.
<wgrant> lifeless: What's the bug? It's not read-only?
<lifeless> https://bugs.launchpad.net/launchpad-code/+bug/676124
<lifeless> if we weren't doing maintenance on that machine, we'd have tried keeping it up this time.
<wgrant> Ah.
<wgrant> Great.
<lifeless> adeuring: ping
<wgrant> Huh, codebrowse works?
<jml> LP seems to be r/w for me now
<wgrant> Ah, so it is.
<lifeless> morning jml
<jml> lifeless: hello
<wgrant> Indeed, morning jml.
<wgrant> Could someone please ec2 https://code.launchpad.net/~wgrant/launchpad/bug-654372-optimise-domination/+merge/40854?
<jml> wgrant: on it
<wgrant> jml: Thanks.
<wgrant> bigjools: re. bug #676262, I suspect they were both ABORTING (since abort() doesn't actually end up killing sbuild). That's a situation we ran into a few hours ago.
<_mup_> Bug #676262: launchpad lost track of a build <Soyuz:Incomplete> <https://launchpad.net/bugs/676262>
<wgrant> (with those same two builders)
<wgrant> Damn ppc :(
<jml> wow
<jml> I got a crazy error when doing ec2 land
<jml> http://paste.ubuntu.com/533423/
* mthaddon changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 (but closed during the roll-out)| firefighting: buildd-manager is disabling things again | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<danilos> henninge, https://pastebin.canonical.com/39840/
<adeuring> lifeless: pong (sorry, did not look at the IRC windows after returning from the kitchen...)
<lifeless> adeuring: hey
<lifeless> adeuring: remember how in APIs and restricted files we hard coded handing out the internal url ?
<adeuring> lifeless: not exactly... let me check again
<lifeless> adeuring: the token based librarian is deployed now
<jml> lifeless: https://bugs.launchpad.net/launchpad-code/+bug/554206 might be relevant to some stuff you are doing
<_mup_> Bug #554206: Need a read-only version of bazaar.launchpad.net for codehosting and codebrowse <canonical-losa-lp> <codebrowse> <codehosting-ssh> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/554206>
<adeuring> lifeless: I remember that firewall settings in the DC needed some teaking
<adeuring> ...tweaking...
<lifeless> right
<wgrant> Why is [ui=none] in every commit message? Can't it just be omitted?
<adeuring> lifeless: mizuho needed access to private Librarian files, and that machine "saw" a librarian URL having a host name with an "internal" domain part
<jml> wgrant: the [ui=foo] field was added as a way of strongly encouraging UI reviews for any UI change
<jml> wgrant: a huge number of changes do not affect the UI
<jml> wgrant: and I suspect that many people skip UI reviews
<lifeless> adeuring: yes
<wgrant> jml: Is it more than 1% of commits that have ui=somethingelse?
<lifeless> adeuring: right, so you did a patch for the API to show the internal url
<jml> wgrant: you can run log & grep as easily as I
<lifeless> adeuring: but its not needed now
<adeuring> lifeless: did I? seems that I need a memory refresh.... looking now
<wgrant> jml: True.
<lifeless> adeuring: you did :)
<lifeless> adeuring: rev 11506
<jml> henninge: now that the rollout is done, can we fix canonical/launchpad/interfaces/__init__?
<henninge> jml: oh. ...
<adeuring> lifeless: thanks! so, time to fix bug 629804
<_mup_> Bug #629804: implement access to private Librarian files for launchpadlib clients <Launchpad Foundations:New> <https://launchpad.net/bugs/629804>
<henninge> jml: well, it's still on the list to do post-rollout but you can prepare a branch. By the time it gets deployed from stable, that should all be done ;-)
<henninge> jml: "it" is "fixing +inbound-email-config.zcml"
<henninge> ;-)
<jml> henninge: ok. will do.
<matsubara> maxb, misclicked
<henninge> jml: just check again before marking the revision as deployable.
<jml> henninge: *nod*. do you recall the bug number?
<henninge> I am not sure it had a bug.
<henninge> jml: nm, it's fixed. ;-)
<henninge> so I guess you can just submit it [no-qa]
<jml> henninge: will do. ta.
<henninge> which is true because we already know it works on qa/staging ... ;-)
<jml> aeoueoia
<jml> lp-land has a bad token, but I don't know where to find it
<lifeless> adeuring: I've unduplicated it
<jml> how do I work around this problem? http://paste.ubuntu.com/533431/
<adeuring> ok
<adeuring> lifeless: I'll do it once I've finished my current work
<adeuring> ...i mean; I'll fix the bug...
<lifeless> adeuring: do you have an estimate for when that will be?
<lifeless> adeuring: if its going to be not-immediate, I might just do it
<lifeless> s/not-immediate/not-today
<adeuring> lifeless: i think I can probably start tomorrow
<adeuring> lifeless: you beat me ;)
<adeuring> problem is that I am quite slow with context swtiches...
<lifeless> adeuring: I'll drop you a mail to let you know if I get to it or not.
<adeuring> lifeless: coool
<jml> wgrant: your branch is being tested in ec2: http://ec2-50-16-92-112.compute-1.amazonaws.com/
<wgrant> jml: I can't see that, but thanks!
<jml> it'd be kind of neat to add a phone-home thing to the ec2 script so we could have a page showing what's being built (as well as test results)
<deryck> Morning, all.
<adeuring> morning deryck
<jml> bigjools: I added something to the derived distributions LEP about opening vs initialization; do you need anything more?
<bigjools> jml: inspiration
<bigjools> thanks :)
<jml> bigjools: np.
<jml> bigjools: also, I notice that https://launchpad.net/launchpad-project/+bugs?field.tag=buildd-scalability has no bugs.
<bigjools> it should do
<bigjools> I tagged loads
<bigjools> jml: ah it's because they've all been released
<jml> bigjools: nice.
<bigjools> jml: https://bugs.launchpad.net/soyuz/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=&field.
<bigjools> bug_supervisor=&field.bug_commenter=&field.subscriber=&field.tag=buildd-scalability&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on
<bigjools> aiieee sorry
<jml> bigjools: looking at the LEP and based on random IRC sampling, I'm guessing we're still missing "When a builder becomes free, we must dispatch a queued build to it within a maximum of 30 seconds.", "Design for a system with 200 builders" and "Not starve low-scored builds when there are higher-scored builds in the queue"
<stub> Having trouble following https://dev.launchpad.net/LaunchpadPpa. debsign -S fails with 'debsign: Can't find or can't read changes file !'
<bigjools> jml: missing from where?
<jml> bigjools: what I mean is, have we met those requirements?
<bigjools> jml: I need to have a call with you about that
<jml> bigjools: ah, ok :)
<bigjools> :)
<bigjools> but later
<bigjools> I am up to my neck in buildd-manager issues
<bigjools> right after a dispatch of 10 or more recipes, there's nothing in the log for 4 minutes
<bigjools> which is somewhat suspicious
<jml> yeah, later is good
<wgrant> The queue isn't just empty?
<bigjools> no, it's the gap between "startBuild" and the "RESULT" stuff
<wgrant> This is why I wanted better logging :(
<bigjools> in fact the latter never appears
<bigjools> yes we all want better logging
<wgrant> Ah.
<bigjools> but one thing at a time
<wgrant> That's very interesting indeed.
<stub> Shouldn't bzr builddeb actually create a .deb?
<jelmer> stub: You have to go back to the parent directory or ../result where the changes file was added.
<jelmer> stub: By default it creates binary packages (.deb's), with -S it creates a source package.
<stub> But where?
<bigjools> wgrant: something is blocking too long when it's dispatching a recipe build
<jelmer> stub: In the parent directory or ../result
<stub> jelmer: I don't have a ../result and nothing new in the parent directory
<wgrant> bigjools: After the "Initiating build foo on bar"?
<jelmer> stub: you can specify a directory manuall with --result-dir
<bigjools> wgrant: in Builder.startBuild() it logs the build start (behavior.logStartBuild)
<bigjools> then there's nothing logged until it fails
<bigjools> at that point, there's a few things that could have gone wrong but the lack of logging means it's hard to tell
<stub> jelmer: Garh. They were in my branch, not my checkout of the branch
<stub> jelmer: Guess that would be a bug...
<jelmer> stub: yeah, that seems a bit strange
<wgrant> bigjools: So we don't even know if it made it into resume_done?
<bigjools> I suspect it has, that's the most reliable part of the process
<wgrant> True.
<bigjools> my suspicions lie in the file disaptching and initiation
<wgrant> But it never made it to got_cache_file... hmm.
<bigjools> we don't know
<bigjools> there's no info level logging
<wgrant> got_cache_file logs fairly obviously.
<wgrant> Ohh, crap.
<wgrant> True.
<jml> deryck: there are a couple of LEPs about bug duplication...
 * bigjools is changing some debug to info
<jml> deryck: one's in drafting (https://dev.launchpad.net/LEP/DisableFilebugDuplicateSearchOption) and the other (https://dev.launchpad.net/LEP/ACLMarkAsDuplicate) isn't on the LEP page
<lifeless> wgrant: We Can Haz Runtime Log Changing Please
<wgrant> lifeless: debug 4 eva
<_mup_> Bug #4: Importing finished po doesn't change progressbar <Launchpad Translations:Fix Released by carlos> <Ubuntu:Invalid> <https://launchpad.net/bugs/4>
<wgrant> Ahem.
<lifeless> rotfl
<lifeless> ok foods
<jml> lifeless: I guess there's https://bugs.edge.launchpad.net/soyuz/+bug/667958
<_mup_> Bug #667958: Web diagnostic tool for build manager <buildd-manager> <Soyuz:Triaged> <https://launchpad.net/bugs/667958>
<jml> but that's not quite the same thing
<bigjools> dynamically changeable log levels is totally essential for decent production debugging
<wgrant> bigjools: Is there anything in the current debug level that isn't interesting, except for the hundreds of "Scanning foo" messages?
<wgrant> Given the frequency and obscurity of issues, it'd be nice to keep as much data as possible...
<bigjools> the problem is that I don't want the log swamped
<bigjools> it makes it harder to notice issues
<bigjools> so I am trying to carefully select important messages for the info logging
<bigjools> but hindsight is awesome
<wgrant> Heh.
<deryck> Hi jml.  Yeah, the first should be done.  And the second was meant to sketch out the idea and go back to marjo et al and get feedback....
<deryck> jml, remember, we talked about this and said, let's do what everyone agrees on and is easy first, and get consensus on if the second is even required.
<deryck> unfortunately, I didn't ping anyone about the second yet.  I'll do that today.
<jml> deryck: ahh right. I forgot to refactor that new knowledge into the LEP page :)
<jml> deryck: so I'll bump the first LEP to the Deployed section?
<deryck> jml, in progress.  I think I assumed approval and moved ahead.
<deryck> jml, sorry to assume ;)
<jml> deryck: no, that's all good :)
<deryck> thanks!
<lifeless> jml: gary has a variant of the LEP template with stuff specific to his team; I've suggested you might be amenable to folding those into the main template
<jml> lifeless: sure, I'll have a look
<jml> lifeless: if someone points me at a thing :)
<lifeless> sure
 * jml is also thinking (again!) about tracking LEPs at blueprints.launchpad.net/launchpad
<lifeless> dunno when he'll do that
<lifeless> jml: lets fix it first.
<lifeless> jml: -please-
<jml> lifeless: I reckon I could do a useful muck-around experiment that wouldn't affect anyone other than me.
<lifeless> would it be a good use of your time?
<lifeless> also, can we chat about reset (voice) ?
<jml> lifeless: sure. gimme a couple of minutes to put my phones back together
<maxb> Is the "builders are being disabled" topic comment in #launchpad still valid after the rollout?
<lifeless> yes
<jml> lifeless: and yes, it would be a good use of my time.
<lifeless> hmm,  didn't mean that to be snarky. Sorry
<jml> lifeless: it wasn't at all snarky. I was going to elaborate but got distracted by yet another networking problem.
<bigjools> jml: you remember how we added timeouts to the async xmlrpc by cancelling the Deferred?
<jml> bigjools: yes
<bigjools> jml: in those cases we get a CancelledError, but I am seeing hundreds of " User timeout caused connection failure."
<bigjools> what causes those?
<bigjools> it's a TimeoutError, sorry.  I can't fathom how that would happen before the cancel
<bigjools> huh actually - that's the 30 second connection issue
<bigjools> which is much lower than our configured value for everything else
<bigjools> jml: I'm tempted to inherit from Proxy and override stuff
<jml> bigjools: yeah. I can't think of anything better right now. You ought to file a ticket and submit a patch too.
<bigjools> jml: there's already a ticket, but the fix needs to go in quite a few places I think
<bigjools> I'll file another anyway
<bigjools> right - I need vittles
<jml> bigjools: yeah, a specific ticket for xmlrpc.py would be great. thanks.
<bigjools> nod
<lifeless> maxb: hey
<maxb> hi
<lifeless> maxb: what do you think of us having a custom python build - with http://bugs.python.org/issue10440 applied
<maxb> If it really is just an integer constant, why do we need to modify python for that?
<maxb> Instead of just defining the value locally
<lifeless> it can be different in different libcs, by definition.
<lifeless> we can hardcode '1' as the constant, but its less portable and thus a bit ugly.
<maxb> Well, it's a tiny patch, so it's hardly much effort to roll a modified package. The question then is the ongoing maintenance effort and how long it would be needed for
<lifeless> yeah
<maxb> I'd be tempted to consider putting the constant in a tiny module of its own, to avoid needing to rebuild every time there's an Ubuntu update out
<maxb> Also, given Launchpad only targets Ubuntu, and a fairly narrow range of distroseries, even the non-portable solution is probably viable
<lifeless> true on both counts
<lifeless> will mull on it
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: buildd-manager is disabling things again | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<bigjools> lifeless: can you think of a way of creating a tcp endpoint that doesn't reply in a twisted test?  I need to test a timeout and winding the reactor forwards is no good if the tcp connects or refuses to connect immediately
<lifeless> sure
<lifeless> bind, listen, but don't accept
<bigjools> in real life I'd suspend a process but that's not ideal in a test
<lifeless> Actually, that might not work. But its worth a go
<bigjools> I suspect it would get connection refused wouldn't it?
<bigjools> hmmm
<lifeless> no
<lifeless> accept is what takes a queued connection and gives you the new fd for it
<elmo> alternatively iptables  + -j DROP
<bigjools> ah right
<elmo> (although that requires root)
<bigjools> not ideal for LP's test suite
<elmo> sure, was just giving it as an option as a one off
<bigjools> elmo: how evil is it to try and connect to something like 10.255.255.1 ?
<lifeless> bigjools: evil; some machines it will error immediately ;)
<bigjools> lifeless: grar
<lifeless> bigjools: because someone, somewhere has that ip
<lifeless> bigjools: or routers that will see it and REJECT
<bigjools> it doesn't get past my own router
<bigjools> oh well it'll do as a stub for now
<bac> Reviewers Meeting starting at top of the hour: abentley, adeuring, allenap , bac, danilo, sinzui, deryck, EdwinGrubbs, flacoste, gary, gmb, henninge, jelmer, jtv, bigjools, leonardr, mars, salgado, jcsackett, benji
<deryck> thanks bac
<flacoste> bac: apologies from me
<bac> np flacoste
<henninge> what's this?
<henninge> http://paste.ubuntu.com/533506/
<henninge> No handlers could be found for logger "librarian"
<bigjools> henninge: you already have a librarian running
<henninge> seriously? didn't know that ...
<bigjools> kill it and the pid file and /var/tmp/fatsam.test
<henninge> what's the process called?
<bigjools> it's a twistd
<henninge> ps ax | grep libra returns nothing
<henninge> ps ax | grep twist - nada
<henninge> :-(
<bigjools> ummm then I dunno, I've only ever seen that when there's another librarian hanging around
<henninge> thanks
<henninge> why is the librarian logging in a +0530 time zone anyway???
<henninge> India?
<jml> henninge: there are no Canonical LP developers in that tz
<henninge> outsourcing?
<jml> henninge: we set the TZ there to avoid accidental TZ assumptions
<jml> henninge: or something
<henninge> ;-)
<henninge> but do you have an idea why the librarian layer might be failing?
<lifeless> henninge: rm /var/tmp/fatsam.test/librarian.pid
<henninge> already done. twice ;)
<lifeless> ps fux | grep twistd
<lifeless> ?
<lifeless> oh
<lifeless> netstat -n | grep 58085
<henninge> nothing
<lifeless> or something like that
<lifeless> is the second upload port thats barfingk
<henninge> maybe I should mention that this is not devel ? It's the recife branch
<henninge> but the test worked yesterday
<lifeless> sinzui: btw your script to close bugs is closing bugs that shouldn't be closed - because of RFWTAD
<henninge> a second run always gives me "TacException: Could not kill stale process /var/tmp/fatsam.test/librarian.pid.
<henninge> so I remove that dir and try again.
<lifeless> nothing changed overnight
<lifeless> I think you have another process using the port
<lifeless> thus the netstat - check lazr-schema / the test schema to see what port it will be using
<sinzui> lifeless, they were fix committed in 10.11, but were not intended to be released?
<thumper> bigjools: did you get to the bottom of the problem?
<lifeless> sinzui: no, our process assigns bugs to milestones *before* they are fixed, not *after*
<sinzui> lifeless, are these really 10.12 bugs
<lifeless> sinzui: they are 'some work done, but not finished'
<lifeless> sinzui: things like:
<bigjools> thumper: I * think* so - I think it's slow builders that don't respond to connection requests within Twisted's 30 second default timeout.  The recipe builds hammer the builders.
<lifeless>  - landed code but it didn't fix it
<lifeless>  - needs a cronscript enabled via an RT ticket
<thumper> bigjools: so why does it take down all types of builders then?
<bigjools> thumper: thanks for doing the incident report
<bigjools> thumper: I don't know, it might be a coincidence.
<lifeless> who is looking at the 'report a bug' feature not working ?
 * thumper doesn't believe in coincidence
<bigjools> I am putting in a fix that increases the connection timeout - copy & paste from Twisted FTW :/
<sinzui> lifeless, I think that is a bug. The engineer should know when he intends to release Auto-assigning is convenient, but it does not exempt the person from correcting the milestone when he knows it will not be release with the milestone. eg we knew this when PQM was frozen
<bigjools> thumper: I've seen slow builders doing exactly that for a while now - it's just that we never disabled them before this release.
<lifeless> sinzui: sure, I'm not blaming the script or you :) - getting info on how to address - what policies we need to change
<sinzui> lifeless, I can add a sanity check (qa-ok in tags)
<lifeless> sinzui: I think thats an excellent idea
<lifeless> sinzui: also I'm closing most bugs - those that are linked from revs - when we do incremental deploys
<lifeless> I have to go eat or miss out, bbiab
<sinzui> lifeless, i will have script for you by the end of my lunch
<bigjools> jml: I guess you're not near your PC then
<dobey> leonardr: around?
<leonardr> dobey: yes
<dobey> leonardr: http://pastebin.ubuntu.com/533530/ <- am getting this as a result of a getMembersByStatus() on a team with status=u'Administrator'
<dobey> leonardr: any idea why that would be?
<leonardr> dobey, what is the code in allowedcontributors.py?
<lifeless> deryck: ping
<deryck> hi lifeless.  on tl call
<dobey> leonardr: http://bazaar.launchpad.net/~rockstar/tarmac/main/annotate/head%3A/tarmac/plugins/allowedcontributors.py#L62
<lifeless> deryck: are you aware that bug filing is reportedly broken ?
<deryck> lifeless, no.  how so?
<lifeless> deryck: two independent reports
<lifeless> 1) apport user filed a bug in launhcpad
<lifeless> 2) james hunt mailed tom who forwarded it in the lp rollout thread
<leonardr> dobey: so the 'approved' one succeeds but the 'administrator' one fails?
<dobey> leonardr: that appears to be the case, yes
<deryck> lifeless, I believe allenap is looking into that.
<deryck> I'll follow up after tl call to make sure, and cover if not
<lifeless> cool
<dobey> leonardr: and unfortunately i have to call it twice, because i can't do status=[u'Approved', u'Administrator']; like i can do with other similar get APIs, but i guess that wouldn't fix this specific problem either :)
<leonardr> dobey: i have no clue why it should work once and then fail. just for fun, you might try assigning launchpad.people[team] to a variable
<leonardr> so you're not using it twice
<leonardr> and if that doesn't work, try assigning to a variable and then printing out its name before invoking those named operations
<leonardr> i'm just seeing if various known problems are in play here (in which case upgrading would help)
<dobey> leonardr: what would i upgrade to exactly?
<leonardr> dobey: a later launchpadlib/lazr.restfulclient
<dobey> leonardr: is there one newer than what is in 11.04 already?
<leonardr> dobey: there is, but the one in 11.04 should have the fix i'm thinking about already
<dobey> ok
<leonardr> dobey: my only suggestion is to put a breakpoint in get_representation_definition and see what it does differently the first time vs. the second
<dobey> leonardr: ok; i've changed it to assign the team to a variable and print the team twice as suggested; will see what happens next time that code gets hit
<rockstar> launchpad is being very slow today. :(
<rockstar> abentley, are there any issues with the new lp-serve happening right now?
<thumper> rockstar: the new forking lp-serve isn't enabled yet
<rockstar> thumper, oh, the bug was marked as Fix Released.  :(
<mars> sinzui, ping
<thumper> rockstar: yes, I know.  jam commented on it too saying as much
<rockstar> Ah, I hadn't seen the comment, just the status change.
<jam> rockstar: right, still trying to work through getting everything qa'd, etc. It isn't considered a qa blocker because it is disabled in production
<jam> I'm noticing that my download-cache has grown to about 500MB, anyone know what files I can nuke?
<jam> I'd like to think that I don't need 12 versions of "zope.testing-*"
<rockstar> jam, basically, you can nuke any files that aren't in versions.cfg
<jam> rockstar: which is in the lp root?
<rockstar> jam, yes
<jam> well, that isn't particularly fun to cross-reference...
<rockstar> thumper, urbanape just pointed out to me that when diff is too big, it says "Truncated for viewing."  That's wrong, because if it was really for viewing, it wouldn't be truncated...
<jam> rockstar: so why is download-cache a bzr branch that is versioning all of these tarballs? seems odd to me
<jam> especially given that it is storing all old versions together in the same working tree
<rockstar> jam, I am not the one to ask about that, but I *think* it was supposed to be a temporary solution we concocted two years ago.
<jam> (for example, it contains 20 bzr tarballs)
<jam> the .bzr/repository is actually bigger than the launchpad repo at this point
<abentley> jam: you do not need to convince us.  We know it's wonky.
<jam> another quick question. Anyone know how lp-production-configs are placed at runtime so I can simulate a runtime environment locally?
<jam> (how does the launchpad codebase find the values in lp-production-configs)
<lifeless> its put at the configs directory in the root I think
<lifeless> and then LPCONFIG=configname
<thumper> rockstar: in an email you mean/
<thumper> ?
<thumper> rockstar: I thought it just said that on the page itself
<thumper> rockstar: and in that case you are viewing it and it is truncated
<rockstar> thumper, in the view, you are viewing it, and it is truncated, but it's not truncated FOR viewing.  It's truncated FROM viewing.  :)
<lifeless> maxb: so, python 3.2 will have my patch :)
<thumper> rockstar: it is truncated to allow you to view it otherwise it times out :-)
<maxb> lifeless: And when are we migrating LP to Python 3? :-)
<thumper> I'd not approve a textual change to "truncated from viewing" as it doesn't make grammatical sense
<rockstar> thumper, yeah, it was pedantry from the start.
 * thumper closes laptop to go and buy a 3g stick
<thumper> rockstar: well we do work for pedantical :)
<rockstar> thumper, although the fact that it's truncated drastically reduces its usefulness.
<thumper> rockstar: the download link still works
<thumper> rockstar: the fact that it is over 5000 lines drastically reduces its usefulness :)
<sinzui> hi mars
<rockstar> thumper, this is true as well.
<jam> lifeless: I know about LPCONFIG=xxxx, but how is the "qastaging.conf" file found?
<jam> it is just copied into the launchpad source tree?
<jam> or is schema-lazr.conf (the symlink) pointed to something else, or?
<rockstar> jam, it's symlinked.
<jam> rockstar: to what file?
<rockstar> jam, it's a file from lp-production-configs.
<jam> rockstar: so they explicitly point schema-lazr.conf to schema-qastaging.conf for example?
<jam> If so, why do you also need LPCONFIG=qastaging?
<mwhudson> morning
<jam> morning mwhudson
<lifeless> jam: qastaging says 'the qastaging' dir which has a launchpad-lazr.conf file
<jam> lifeless: sure, but there are 4 schema-XXX.conf files
<jam> and no "schema-lazr.conf" or "schema-launchpad.conf", etc in the top of the dir
<jam> anyway, I'm getting my problem solved without using it yet
<jam> but still, I don't know yet how to set up something that resembles production
<lifeless> jam: schema-xxx is irrelevant
<jam> lifeless: so you still haven't answered how launchpad finds lp-production-configs/*.conf then
<lifeless> I thnk its
<lifeless> rm configs
<lifeless> mv lp-production-configs configs
<lifeless> IMBW
<lifeless> losa can tell you though - ask chex
<jam> k
<jam> lifeless: any idea of a 'clean' way to invoke the bzr that is packaged with the launchpad tree? or should we just be invoking /usr/bin/bzr ?
<jam> (IOW, how are the dependencies found in production)
<jam> `pwd`/eggs/bzr-2.2.0-py2.6-linux-i686.egg/EGG-INFO/scripts/bzr is obviously not a long-term solution
<jam> or Chex ^^
<mwhudson> um
<mwhudson> i think launchpad looks for lp-production-configs/$LPCONFIG/launchpad-lazr.conf then for configs/$LPCONFIG/launchpad-lazr.conf
<mwhudson> the other config files get brought in by extends: ../foo.conf in those config files
<jam> mwhudson: so it is just 'lp-production-configs' in a generic sibling dir?
<mwhudson> jam: pretty sure, let me look at some code
<jam> mwhudson: doing that, I get "Can't find qastaging in ..."
<jam> in a traceback
<mwhudson> jam: "production-configs", not lp-production-configs
<mwhudson> my mistake
<jam> mwhudson: confirmed that it works
<jam> (via symlink at least)
<mwhudson> cool
<jam> mwhudson: and ./production-configs is also in .bzrignore
<mwhudson> heh heh
<jam> losa ping. I don't know if you have time, but mthaddon was looking at rt#42199 last night, and I think I've responded to what he needed. I don't know whether that means there is a hand-off or whether it is just going to wait for him to get back.
<_mup_> Bug #42199: evolution causes gpg stale locks <Evolution:Fix Released> <evolution (Ubuntu):Fix Released by desktop-bugs> <https://launchpad.net/bugs/42199>
<lifeless> jam: not a sibling dir, child dir
<jam> lifeless: nope, at the root "launchpad/configs launchpad/production-configs"
<jam> at least, that worked for me
<jam> and that is what is in .bzrignore
<lifeless> kk
<wgrant> Were we in testfix overnight?
<weather15> Hello Everyone
<wallyworld> abentley: thumper: now?
<gary_poster> lifeless: https://bazaar.launchpad.net/~launchpad-pqm/launchpad/production-stable/revision/9000
<abentley> wallyworld: sure.
<thumper> wallyworld: just here
<wallyworld> abentley: %@$!!#$ audio died again.
<wallyworld> brb
<weather15> I have a wuick question about the Launchpad source
<weather15> When running make schema is this part of a normal output? Unknown entry URL:                     ScalarValue Unknown entry URL:                     archive_dependency Unknown entry URL:                     archive_subscriber Unknown entry URL:                     binary_package_release_download_count Unknown entry URL:                     branch_merge_queue Unknown entry URL:                     branch_subscription Unknown e
<wgrant> That's normal.
<weather15> Okay Thank's wgrant
<weather15> Wgrant:  is this a typical end output: make[1]: Leaving directory `/home/weather15/launchpad/lp-branches/devel/database/schema' rm -f -r /var/tmp/fatsam
<wgrant> weather15: Yes.
<weather15> wgrant: Thanks
<weather15> wgrant: I'm running Ubuntu Server
<weather15> In this case how can I access Launchpad.dev?
<weather15> SSH Tunnel?
<weather15> or is there Apache settings to change?
<wgrant> weather15: Have a look at https://dev.launchpad.net/Running/RemoteAccess
<weather15> Also should I follow these instructions? 2010-11-17T16:11:40 WARNING root Developer mode is enabled: this is a security risk and should NOT be enabled on production servers. Developer mode can be turned off in etc/zope.conf
<weather15> I plan on going into production
<wgrant> Running a production Launchpad instance is not a simple task.
<weather15> wgrant: Do I need to have more then 1 IP?
<wgrant> weather15: Only if you want to be able to browse private branches.
<weather15> Okay I do
<weather15> two IP's on my local net or on the Internet?
<wgrant> Wherever you want it to be accessible from.
<weather15> okay
<weather15> I guess if I run it on my local net then I will have all public repos
<weather15> Then I only need 1 IP
<maxb> weather15: OOI, which LP applications do you intend to use in production?
<weather15> Pretty much all
<maxb> Interesting, I'd only imagined people using bugs & code in a local setting
<weather15> That's most likely what will happen but I'm not sure yet
<weather15> I'm focused on getting it working now
<maxb> You know about the whole image licence pain, right?
<weather15> no
<maxb> https://dev.launchpad.net/LaunchpadLicense
<maxb> especially the 4th paragraph
<weather15> "The image and icon files in Launchpad are copyright Canonical, but unlike the source code they are not licensed under the AGPLv3. Canonical grants you the right to use them for testing and development purposes only, but not to use them in production (commercially or non-commercially). "
<weather15> That Part
<wgrant> That part.
<weather15> I know about that
<weather15> I was wondering how to change those images
<weather15> rather 1 IP i guess Ubuntu is not getting IP's on my second interface
<weather15> what do you do when launchpad.dev will not resolve on the network?
<weather15> I guess because I have only 1 IP bazaar will not work
<weather15> is this true?
<weather15> It seems I have 2 IP's now
<weather15> do you replace a.b.c.d here <VirtualHost a.b.c.d:80> with your ip?
<leonardr> james_w, who's the best person to talk to about getting new versions of launchpadlib and friends included in natty?
<james_w> leonardr, Luca probably
<leonardr> james_w: ok, makes sense, thanks
<bigjools> wgrant: so, in case I am asleep at 2330 (highly likely), I've put another cowboy on cesium to fix the buildd manager.
<weather15> Any one know the answer to my previous question?
<weather15> It says "Or, if you did allocate a suitable second IP address:      *        Change the <VirtualHost 127.0.0.99:80> line to <VirtualHost a.b.c.d:80>     *        Change the <VirtualHost 127.0.0.99:443> line to <VirtualHost a.b.c.d:443>"
<wgrant> bigjools: Removing failure counting?
<weather15> is this what I should use or replace a.b.c.d with the IP on my second NIC
<wgrant> weather15: The latter.
<wgrant> bigjools: Do we also have more logging now?
<bigjools> wgrant: some more yes
<weather15> wgrant: with my IP correct?
<wgrant> weather15: Yes.
<weather15> Thanks
<wgrant> bigjools: Well, I guess we'll see how it goes!
<wgrant> bigjools: Did you and jml work anything out?
<jml> wgrant: I didn't!
<bigjools> wgrant: default connection timeout on twisted xmlrpc is 30 seconds, I've made it use socket_timeout instead
<bigjools> I am seeing some builders *still* failing with that though
<wgrant> bigjools: Hmm. I don't think that really explains everything, but it might fix the resume thing.
<bigjools> not just resume, all xmlrpc requests
<wgrant> jml: Also, why did PQM eat my branch?
<jml> wgrant: I don't know. I didn't see that it got eaten
<bigjools> and no it does not explain everything
<bigjools> but it's a start
<wgrant> jml: It said it submitted, but then nothing :/
<wgrant> bigjools: Yeah, I guess.
<jml> wgrant: I don't know. I won't be able to get around to looking into it tonight â sorry.
<jml> wgrant: maybe you can convince someone else to land it. the tests all pass. if not, I'll do it first thing tomorrow
<mwhudson> a builder taking 30 seconds to accept a connection seems pretty crazy too
<wgrant> jml: Sure, no rush.
<mwhudson> is the listen queue overflowing on the slave side or something?
<mwhudson> i guess that's pretty hard to tell
<wgrant> mwhudson: The builder is an archaic Twisted mess gluing together shoddy shell scripts.
<wgrant> It's allowed to be crazy, I think.
<jml> meh
<jml> we allow it to be crazy
<mwhudson> wgrant: even so
<mwhudson> wgrant: is the builder one of these half twisted things that does blocking operations in the reactor thread?
<wgrant> mwhudson: Sometimes.
<jml> the build manager is
<bigjools> but not for long
<bigjools> mwhudson: I think >30 seconds happens when the slave manager was swapped out under load or something
<mwhudson> oh right
<bigjools> that's my guess....
<jml> bigjools: db queries are blocking calls
<wgrant> bigjools: Doesn't explain all the non-virt failures :(
<bigjools> wgrant: it might, actually
<bigjools> jml: true, very true.
<wgrant> bigjools: How? Unless buildd-manager leaks exceptions across multiple builders, I don't see how...
<weather15> for the allow for
<bigjools> wgrant: if the previous build went into swap ...
<weather15> would this work for 10.0.0.1? 10.0.0. or 10.0.0?
<bigjools> on the same builder, I mean
<mwhudson> ... but you'd still need to fill up the listen queue, right?  connecting to a listening socket doesn't involve the userspace process doing the listening iiuc
<weather15> For the Allow from
<wgrant> weather15: That's just normal Apache configuration.
<wgrant> mwhudson: Hmmm? It needs to call accept(), right?
<weather15> Yes but I need to set the sllow from
<weather15> *allow
<bigjools> mwhudson: I don't know
<bigjools> some people have said that it needs to accept()
<weather15> would 10.0.0 work or would I have to use 10.0.0. to allow my local network?
<weather15> on 10.0.0.x
<bigjools> it's been a while since I I did socket stuff
<mwhudson> i can't remember either
<bigjools> weather15: I suggest you ask Apache questions in the right channel
<bigjools> you will almost certainly get a more knowledgeable answer
<wgrant> bigjools: Hmmm. I see that palmer had been aborted 10 minutes before the failure. So it was probably still building. So that's plausible.
<weather15> looks to me like Allow from 10.0.0.0/255.255.255.0 will work
<mwhudson> science suggests that i am right about accept
<bigjools> science rocks
<wgrant> Although the fact that it timed out at the same time as the rest is a bit suspicious, perhaps buildd-manager was blocking for the preceding couple of minutes. Insufficient logging :/
<bigjools> yeah, impossible to tell
<bigjools> although if it was slow with the DB ...
<wgrant> siiigh
<mwhudson> i guess you can turn on statement tracing in buildd-manager
<bigjools> log armageddon!
<mwhudson> yeah
<wgrant> bigjools: Ah, this is why we needed to clean out accepted... so we can have hundreds of gigabytes of logs!
<mwhudson> more realistically, you can probably have a tracer log any statement that takes longer than say 5 s
<bigjools> not sure that will help if there's a cumulative effect of 10*1s for example
<mwhudson> ... or collect aggregate stats, min, max, mean, stddev kind of thing
<wgrant> If it happens again today, I think we should run with full logging tomorrow.
<bigjools> I am too tired to think straight now
<mwhudson> fair enough :-)
<bigjools> we are full logging now, except the madness of statement tracing
<wgrant> Even the 'Scanning foo' messages?
<wgrant> And the extra logging in failBuilder that was cowboyed in earlier?
<bigjools> everything
<wgrant> Great.
<bigjools> not that one - because we're not currently failing builders
<wgrant> Ah, heh.
<bigjools> assessFailureCounts is commented out
<bigjools> so it will report on the counts but never do anything about it
<wgrant> Perfect.
<bigjools> I need to split the failure count stuff in two though
<bigjools> 1 set for dispatch attempts and 1 set for contact attempts
<weather15> OKay My Launchpad install can be accessed with one problem
<weather15> What do you do about this error? Error code: ssl_error_rx_record_too_long
<weather15> SSL received a record that exceeded the maximum permissible length.
<wgrant> Your Apache configuration is broken. It's probably serving normal HTTP on 443.
<weather15> Okay I'll check it again
<weather15> Any idea as to where to look?
<weather15> I don't see anything wrong with it
<weather15> is there something wrong with the keys>
<wgrant> No.
<wgrant> Have you tried restarting Apache?
<thumper> wallyworld: I've pulled you branch and am looking at it...
<weather15> I have a problem
<jml> phwoar
<weather15> my Apache config no-longer exists
<jml> food helps
<weather15> what do you do in this case?
<thumper> wallyworld: found it
<thumper> I wish we had different root objects for each virtual domain
<jml> bigjools: did you file a patch upstream for the xmlrpc timeout thingy?
<wallyworld> thumper: just finished breakfast. what was it?
<thumper> wallyworld: I told you wrong, the canonical_url of IBazaarApplication is http://code.launchpad.dev/+code
<thumper> wallyworld: so... we should hang off ILaunchpadRoot
<thumper> or whatever it is
<wallyworld> thumper: ah ok. i saw some other stuff hanging off that and was wondering.....
<wallyworld> i'll fix it
<wallyworld> thanks
<thumper> wallyworld: also, the location of the link on the code homepage needs to be fixed
<wallyworld> thumper: where would you like me to stick it? :-)
<weather15> is this good or bad?  WARNING Bad object name 'public.todrop_branchmergerobot' 2010-11-17 22:48:01 WARNING No permissions specified for [u'public.lp_openididentifier'] * Disabling autovacuum
<thumper> wallyworld: I think we should have some nice text below the import text mentioning recipies
<thumper> wallyworld: they are going to be one of our prime features
<thumper> wallyworld: lets mock something up and get it to mrevell to check
<wallyworld> thumper: also, i added a 30 day window to the query. not sure if we want that or not or make it user selectable
<thumper> wallyworld: that may be fine for now
<thumper> we may want to give the users an option
<thumper> later
<wallyworld> thumper: ack the mockup. the initial intent was just to get something working :-)
<bigjools> jml: no, only a bug so far
<thumper> wallyworld: yeah, understand that
<wallyworld> thumper: +1 on the option. i was going to have a selection on the listing page itself, like we do for branch listings
<jml> bigjools: ok.
<bigjools> jml: I'll do one tomorrow
<weather15> any ide a what to do about this? Traceback (most recent call last):      * Module zope.publisher.publish, line 134, in publish       result = publication.callObject(request, obj)     * Module canonical.launchpad.webapp.publication, line 483, in callObject       return mapply(ob, request.getPositionalArguments(), request)     * Module zope.publisher.publish, line 109, in mapply       return debug_call(obj, args)       __trace
<jml> bigjools: neat.
<weather15> ??
<weather15> More Ouput: File "/home/weather15/launchpad/lp-sourcedeps/eggs/zope.publisher-3.12.0-py2.6.egg/zope/publisher/publish.py", line 134, in publish     result = publication.callObject(request, obj)   File "/home/weather15/launchpad/lp-branches/devel/lib/canonical/launchpad/webapp/publication.py", line 483, in callObject     return mapply(ob, request.getPositionalArguments(), request)   File "/home/weather15/launchpad/lp-sourcede
<bigjools> jml: maybe someone will write a test if I attach a patch :)
<weather15> ????
<bigjools> good night
<jml> bigjools: g'night.
<wgrant> Night bigjools.
<weather15> What's this mean? No such file or directory: '/var/tmp/mailman/data/master-qrunner.pid' Is qrunner even running? rm -f logs/thread*.request bin/run -r librarian,google-webservice,memcached -i development
<weather15> mailman not running
<weather15> causin gthis problem?
<weather15> Traceback (most recent call last):      * Module zope.publisher.publish, line 134, in publish       result = publication.callObject(request, obj)     * Module canonical.launchpad.webapp.publication, line 483, in callObject       return mapply(ob, request.getPositionalArguments(), request)     * Module zope.publisher.publish, line 109, in mapply       return debug_call(obj, args)       __traceback_info__: <bound method OpenID
<mars> weather15, what command did you run to get that output?
<weather15> mars:I went to the login page: https://launchpad.dev/+login
<mars> weather15, are you using 'make run' in the launchpad source tree?
<weather15> yes
<mars> and it did not produce obvious errors about starting mailman?
<weather15> no it did
<weather15> heres the full ouput: make run utilities/shhh.py PYTHONPATH= python bootstrap.py\                 --setup-source=ez_setup.py \                 --download-base=download-cache/dist --eggs=eggs \                 --version=1.5.1 mkdir -p /var/tmp/vostok-archive utilities/shhh.py make -C sourcecode build PYTHON=python \             LPCONFIG=development utilities/shhh.py LPCONFIG=development /home/weather15/launchpad/lp-branches/d
<weather15> Aparently I can't paste it all
<mars> weather15, pastebin.ubuntu.com
<weather15> http://pastebin.ubuntu.com/533640/
<wallyworld> thumper: just wondering aloud, to me it's bad that the tests passed (2 different page/view creation steps too) but the app failed to run in practice. agree? something to fix?
<mars> weather15, on line 28, that looks like an error when the server is first run - it tried to clean up a PID file that doesn't exist.  I wouldn't worry about it.
<thumper> wallyworld:  the problem is that you weren't loading the page, and clicking on the link
<thumper> wallyworld: we had page tests for things like that
<mars> weather15, what did you see when you tried launchpad.dev/+login ?
<thumper> wallyworld: the unit tests were going directly to the page
<thumper> wallyworld: so you never saw the actual url
<weather15> mars: http://pastebin.ubuntu.com/533641/
<thumper> wallyworld: you could add a test that gets the browser for the page
<thumper> wallyworld: and tests the browser.url
<thumper> wallyworld: that would have caught it
<wallyworld> thumper: ok. i assumed that calls like create_initialized_view(root, "+daily-builds", rootsite='code') would use the same zope infrastructure as is used to load a page etc
<thumper> wallyworld: it does
<thumper> wallyworld: but the code root page was using a relative url hard coded
<mars> weather15, that is new.  https://launchpad.dev works?
<thumper> wallyworld: it wasn't generating the url in the same way that the tests were
<wallyworld> ok
<weather15> for me using the source and by setting it in my /etc/hosts file
<wgrant> weather15: Your Apache config for testopenid.dev is still broken.
<weather15> thr documentation never mentioned that
<weather15> what do I have to do?
<wgrant> You must have broken it when you were changing the config.
<wgrant> It's in with the rest.
<mars> weather15, read the rocketfuel-setup script, it has a bash Here Document inside that sets up the /etc/hosts file.  You can compare with that.
<weather15> there's no mention of openid in the apache config
<weather15> this is what the LaunchPad part looks like of .etc/hosts: 10.0.0.3      launchpad.dev answers.launchpad.dev archive.launchpad.dev api.launchpad.dev bazaar-internal.launchpad.dev beta.launchpad.dev blueprints.launchpad.dev bugs.launchpad.dev code.launchpad.dev feeds.launchpad.dev id.launchpad.dev keyserver.launchpad.dev lists.launchpad.dev openid.launchpad.dev ubuntu-openid.launchpad.dev ppa.launchpad.dev private-ppa.launchpa
<wgrant> It will probably go to the first matching vhost, then.
<wgrant> weather15: Try adding 'ServerAlias testopenid.dev' to the bottom two sections in the Apache config.
<wgrant> Alongside launchpad.dev and *.launchpad.dev
<weather15> okay done
<mars> wgrant, weather15, on my system, the only location of testopenid.dev is in the /etc/hosts file
<wgrant> mars: Right.
<wgrant> mars: So it uses the default vhost.
<weather15> I have launchpad starting now lets see what happens
<wgrant> flacoste: http://paste.ubuntu.com/533638/ fixes the .htpasswd thing.
<wgrant> flacoste: Not sure why.
<wgrant> (it reverts part of the problematic rev)
 * jml off
<flacoste> wgrant: weird
<wgrant> flacoste: Just a little.
<flacoste> i thought that umask played only when creating a file
<weather15> That doesn't explain why this is not working
<wgrant> In both cases this creates a file.
<wgrant> But somehow O_TRUNC changes things.
<wgrant> Or Python is doing something stupid.
<weather15> still: Traceback (most recent call last):      * Module zope.publisher.publish, line 134, in publish       result = publication.callObject(request, obj)     * Module canonical.launchpad.webapp.publication, line 483, in callObject       return mapply(ob, request.getPositionalArguments(), request)     * Module zope.publisher.publish, line 109, in mapply       return debug_call(obj, args)       __traceback_info__: <bound method
<wgrant> weather15: Does accessing testopenid.dev in a browser work?
<flacoste> wgrant: 'w' would use O_TRUNC?
<wgrant> flacoste: Yes.
<flacoste> wgrant: ok
<weather15> server side yes
<weather15> client side no
<flacoste> wgrant: wallyworld is going to coordinate deploying that as a cow-boy
<wgrant> flacoste: Great.
 * flacoste updates incident report
<weather15> is that the problem
* flacoste changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: buildd-manager is disabling things again & https://wiki.canonical.com/IncidentReports/2010-11-17-LP-Private-PPA-500-errors  | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<wgrant> Yay Soyuz.
<weather15> Problem still exists
<weather15> Oops!  Sorry, something just went wrong in Launchpad.  Weâve recorded what happened, and weâll fix it as soon as possible. Apologies for the inconvenience.  (Error ID: OOPS-1782X11)  Traceback (most recent call last):      * Module zope.publisher.publish, line 134, in publish       result = publication.callObject(request, obj)     * Module canonical.launchpad.webapp.publication, line 483, in callObject       return mappl
<flacoste> wgrant: shouldn't we set the umask explicitely there? instead of relying on the env
<weather15> this URL works: http://testopenid.dev/
<weather15> it returns:Test OpenID provider for launchpad.dev
<weather15> I wonder if this has something to do with it: https://code.launchpad.net/~bac/launchpad/bug-524302/+merge/22180
<weather15> ????
<weather15> output on server side is different:
<weather15> does this make more sense? http://pastebin.ubuntu.com/533653/
<weather15> mars wgrant?
<mars> weather15, try stopping the service, then running 'make clean && make' in the source tree.
<weather15> okay will do
<mars> weather15, https://launchpad.dev/+icing/rev5/build/lp/lp.js should be a real file when the server is running
<weather15> okay it's executing now
<mars> the build system should create that JavaScript file for you.  You may want to check the source tree to see that it was created
<mars> (when make finishes)
<weather15> when I run make run I get this http://pastebin.ubuntu.com/533655/
<weather15> mars wgrant?
<wallyworld> abentley: ping
<wallyworld> mars: ping - i need a cowboy eyeballed before asking a losa to deploy it
<weather15> ??
<wallyworld> StevenK: ping?
<wgrant> wallyworld: How many eyeballs does it need?
<wgrant> Is yours insufficient?
<wallyworld> wgrant: just one. the change as per your pastebin just reverts it to as it was before 11982 landed
<wallyworld> https://code.edge.launchpad.net/~wallyworld/launchpad/htpasswd-access-permissions/+merge/41115
<wallyworld> wgrant: i wasn't sure if i needed to ask a reviewer to eyeball it or not
<wgrant> Oh, right, forgot you weren't a reviewer yet.
<wallyworld> :-)
#launchpad-dev 2010-11-18
<wgrant> Oh.
<wgrant> I see why this happens now.
<wgrant> There's a proper fix, but this reversion is probably better.
<wgrant> (for now)
<weather15> why?
<wgrant> wallyworld: ^^
<wallyworld> wgrant: ?
<wgrant> wallyworld: I (and everyone else, it seems) was misunderstanding the code. write_htpasswd is given the name of a temporary file which was just created, with mode 600.
<wgrant> Not the old .htaccess file.
<wgrant> So this makes sense.
<wgrant> I now understand why that patch works.
<weather15> no solution to my problem?
<wallyworld> thumper: i need that cowboy eyeballed asap. see mp ^^^^^^^^^
<wallyworld> thumper: only one losa left and he is waiting
<wgrant> weather15: What happens if you try to retrieve testopenid.dev from the server?
<wgrant> wallyworld: The MP looks empty to me :/
<thumper> wallyworld: looking
<wgrant> Ah, there.
<wallyworld> wgrant: fixed. i was being dumb
<weather15> Service Temporarily Unavailable
<wgrant> weather15: You killed 'make run'?
<thumper> wallyworld: you aren't really reverting the fix are you?
<weather15> wgrant see this: http://pastebin.ubuntu.com/533655/
<wallyworld> thumper: no. but that effectively what it does
<wallyworld> it a new branch
<wallyworld> bad comments i guess
<thumper> ah...
<wgrant> Preferably don't merge that fix, but please cowboy it.
<thumper> what is really changing here?
<wallyworld> it will be cleaned up. i justed wanted to get the cowboy done
<thumper> wgrant: doesn't open(..., 'w') overwrite?
<wgrant> thumper: Yeah, but it doesn't set the mode.
<wallyworld> thumper: see pastebin http://paste.ubuntu.com/533638/
<thumper> wgrant: so really what is the difference for the cowboy?
<wgrant> thumper: What do you mean?
<wgrant> thumper: The problem is that write_htpasswd is being called on a tempfile.
<wgrant> The tempfile is 600.
<abentley> wallyworld: pong
<wgrant> thumper: open(..., 'w') truncates, but does not set the mode to the 644 that we require.
<wgrant> thumper: This is unobvious, and apparently a drive-by cleanup a week or so ago.
<wallyworld> abentley: hi, just was looking for some eyeballs to review a cowboy. see mp a few lines above ^^^^^^ thumper is onto it
<mbarnett> if the file doesn't exist at all (in the case of a newly privatized PPA) is the file going to be initially created with the correct permissions?
<weather15> wgrant what can I do?
<wgrant> mbarnett: That's irrelevant. It creates a new tempfile, writes to that, then renames it over the top of the old one, if any.
<wgrant> (to attempt to be atomic)
<thumper> wgrant: I just don't see why this is any different
<thumper> wgrant: how confident are you that this works?
<wgrant> thumper: It deletes the file and creates it again.
<wgrant> Umask is 0022
<thumper> and the existing file is 600 why?
<wgrant> I've tested it locally, and it's just the same as the old code which worked.
<wgrant> thumper: Because that's what mkstemp does.
<thumper> mbarnett: do the cowboy plz
<wgrant> Yay.
<wallyworld> thumper: thanks
<thumper> wgrant: why is the existing filename mkstemp?
<thumper> wgrant: I'm just trying to understand
<wallyworld> thumper: did we want the cowboy to become the accepted fix
<thumper> wallyworld: probably not
<abentley> wallyworld: in fact, it's something that looked wrong when I reviewed the code, and I said so at the time.
<thumper> wallyworld: I think it should be looked at as it isn't obviously wrong
<wgrant> thumper: generate-ppa-htaccess creates a new temp file (using mkstemp), calls write_htpasswd on it, then renames it to .htpasswd.
<mbarnett> thumper: thanks.
<wallyworld> thumper: abentley: np. who should i assign the bug to?
<wgrant> I can't see the original MP (it's a private branch), but it looks like something which could have been fixed as part of a review.
<thumper> wallyworld: don't assign it to anyone
<mbarnett> patched
<thumper> wallyworld: but pass it on to soyuz
<wallyworld> mbarnett: thanks!
<mbarnett> now we need to update a subscirption and make sure it worked!
<wgrant> mbarnett: Have you fixed existing file modes?
<wallyworld> thumper: ack
<thumper> wgrant: would it be better to look at the mode of the file rather than depend on the umask?
<wgrant> thumper: Of course.
<wgrant> And it should be tested.
<mbarnett> wgrant: yeah, all were set properly
<wgrant> But I lack time to do that right now.
<mbarnett> wgrant: re-running my find to make sure no more snuck in
<weather15> Traceback (most recent call last):   File "bin/killservice", line 23, in <module>     import lp.scripts.utilities.killservice   File "/home/weather15/launchpad/lp-branches/devel/lib/lp/scripts/utilities/killservice.py", line 25, in <module>     from canonical.lazr.pidfile import ( ImportError: No module named pidfile make: *** [initscript-stop] Error 1
<weather15> ???
<abentley> wgrant: Yeah, it was considered a security vulnerability that we were publishing private PPAs before setting up their auth.
<wgrant> abentley: I know the bug and fix, but I can't see the comments in the MP.
<wgrant> I was wondering if the problematic change was a review fix.
<mbarnett> no errors in the logs on the subsequent ppa generation runs.  gonna change a subscription now and make sure it is happy
<abentley> wgrant: it was.
<wgrant> Hah.
<abentley> wgrant: rev 11842.1.12 of stable.
<wgrant> ... why did I not think to check the revision history.
<wgrant> and the build farm seems to still be alive.
<wgrant> Is PQM untestfixed at the moment?
<lifeless> who knows?
<poolie> seiflotfy: hey
<seiflotfy> hi poolie
<seiflotfy> hi all
<poolie> seif is interested in feeding launchpad actions into zeitgeist
<seiflotfy> so
<poolie> which i think would give an incredible experience
<poolie> "that bug i touched the other day"
<seiflotfy> Using Zeitgeist as a backend in Launchpad will allow users to have a personal timeline of their activities regarding themselves or specific project. This will allow easier tracker of when things were done commited and view a chronicle of the project in terms of all (blueprints/bugs/etc..) mashed together.
<seiflotfy> To do so Zeitgeist will just need 1 minor change which is use a different SQL DB to allow multiple read and write.
<poolie> seif i just saw this general topic of zeitgeist-like-things was mentioned in the Economist last week
<seiflotfy> really
<seiflotfy> where?
<seiflotfy> iiiiiiiiiiiiiiiiithe magazine
<seiflotfy> sorr
<seiflotfy> sorry typo
<seiflotfy> sor
<seiflotfy> ok
<seiflotfy> fixed the keyboard
<seiflotfy> sorry
<seiflotfy> ok
<seiflotfy> ...........
<poolie> :)
<james_w> seiflotfy, what's the db backend of zeitgeist? Can it be replicated?
<seiflotfy> sqlite
<seiflotfy> james_w, we have an extension that forwards zeitgeist stuff to couchdb
<poolie> seiflotfy: http://www.economist.com/node/17388382?story_id=17388382
<seiflotfy> james_w, its easy to exchange the backend
<seiflotfy> the trick is in what shoudl be stored
<james_w> seiflotfy, so LP could write a different backend to use a more appropriate technology for it?
<seiflotfy> yes
<seiflotfy> i was thinking of postgre sql
<james_w> ok, what would zeitgeist provide in that situation, over some hacked up timeline view?
<seiflotfy> james_w, i dont understand you question
<james_w> seiflotfy, just trying to understand what benefit using zeitgeist itself would have
<mortenmj> hello
<seiflotfy> yeah
<james_w> I agree that the functionality would be good for LP, but I'm unsure whether the code itself makes sense
<seiflotfy> james_w, i dont understand the question though
<poolie> if you're talking about having zeitgeist directly access the launchpad pgsq, that's probably not feasible
<seiflotfy> no no no
<poolie> are you talking about having zeitgeist run on the client, or also having it run in the datacentre?
<seiflotfy> poolie, it shoudl run on the datacenter with its own DB
<seiflotfy> james_w, the code is stable
<james_w> seiflotfy, my concern would be that zeitgeist was designed for activity tracking on desktops. Using it in a webapp is quite different. I want to separate the concept from the implementation.
<seiflotfy> and being used by canonical already for unity
<seiflotfy> james_w, not really
<poolie> how about having lp emit data into couch, in the dc, and then having zeitgeist pull some per-user parts of that?
<seiflotfy> we developed it for using it with webapps too
<seiflotfy> james_w, i can track twitter stuff easily
<seiflotfy> james_w, poolie , zeitgeist supports exchangable backends per default :)
<seiflotfy> so the DB where the data is stored is not a problem
<seiflotfy> its about how it is stored
<seiflotfy> james_w, I already started mapping out how user activities on launchpad will be logged
<seiflotfy> and tbh its pretty easy
<james_w> I'd suggest that you send it to the mailing list, you'll reach a larger fraction of LP developers that way
<seiflotfy> james_w, sure thing
<seiflotfy> http://pastebin.ca/1995138
<seiflotfy> poolie, this is how a bug report would look like
<seiflotfy> i can save this as it is in zeitgeist
<seiflotfy> this will allow me to query a timeline for user/project/team/bug
<poolie> ok..
<poolie> spiv and i might get some lunch, biab
<poolie> agree with james, sending it to the list too could be good
<james_w> seiflotfy, what about privacy?
<seiflotfy> james_w, in which sense
<seiflotfy> james_w, its easy to set the privacy issues of who gets to see what as a layer between zeitgeist and launchpad
<seiflotfy> its a simple logic
<james_w> right, but you need to account for that
<james_w> bugs can be private
<seiflotfy> depends on how the team is set up
<wallyworld> thumper: could you plz stamp this so i can progress it https://code.edge.launchpad.net/~wallyworld/launchpad/person-mergequeue-listview/+merge/41122
<seiflotfy> james_w, the logic part for privacy is an issue that should be taken seriously and I agree on that, but its easier than oyu think
<seiflotfy> james_w, can u give me the launchpad list
<james_w> seiflotfy, I'm not saying it's hard, I'm saying it's a requirement for LP, so if your proposal doesn't mention it then it will be the first objection that people make
<seiflotfy> james_w, taken into consideration
<seiflotfy> james_w, the problem i dont have it all thought out yet
<seiflotfy> i would like to do it with the launchpad team
<seiflotfy> think it through with u guys
<james_w> https://launchpad.net/~launchpad-dev
<seiflotfy> james_w, join the team or write to launchpad-dev@lists.launchpad.net ?
<james_w> I think you have to do the former to do the latter
<seiflotfy> ok
<seiflotfy> i am applying to join then
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      198 /   62  Person:+commentedbugs
<lifeless>      138 / 5176  Archive:+index
<lifeless>       78 /  267  BugTask:+index
<lifeless>       44 /  308  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>       36 /    1  Bug:EntryResource:newMessage
<lifeless>       28 /  267  Distribution:+bugs
<lifeless>       26 /   53  Archive:+copy-packages
<lifeless>       21 /  135  POFile:+translate
<lifeless>       17 /   24  DistroArchSeries:+index
<lifeless>       15 /   10  ProjectGroup:+milestones
<wgrant> Yay, we aren't winning any more :D
<lifeless> sum the two figures ;)
<wgrant> :(
<poolie> seiflotfy: yeah, privacy seems to be the hard thing
<StevenK> wallyworld: Pong
<seiflotfy> poolie, not at all
<seiflotfy> i think it can be done as a logic between launchpad and the server-zeitgeist
<poolie> excellent
<seiflotfy> poolie, its pretty easy
<wallyworld> StevenK: hi, no matter now. there was an cherry pick i wanted to get eyeballed. all sorted, thanks
<seiflotfy> poolie, i am sending out an email to the launchpad and zeitgeist lists
<StevenK> wallyworld: I'm not working this week in any case :-)
<seiflotfy> this would get us into good discussion :)
<wallyworld> StevenK: lucky bastard. you on leave?
<StevenK> wallyworld: Yup, until Monday
<wallyworld> StevenK: well get off irc then and enjoy your holiday :-)
<seiflotfy> poolie, i sent out an email to the dev list :)
<seiflotfy> poolie, i am off to bed again
<seiflotfy> its 3:40 am here
<poolie> ok
<poolie> sleep! :)
<seiflotfy> n8
<poolie> wallyworld: can you have a look at bug 674581
<_mup_> Bug #674581: Sourceforge Mercurial Import Fails <hg> <mercurial> <souorceforge> <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/674581>
<poolie> and say whether it's a dupe or a known limitation?
<wallyworld> poolie: ok, i'll look
<wallyworld> poolie: not sure - i'm a bit uneducated as yet on this stuff. perhaps thumper could comment? ^^^^
<thumper> ah... whut?
<wallyworld> thumper: could you take a quick look at poolie's bug above. i can dig further. i thought you may just look at it and know what was happening based on past experience
<thumper> wallyworld: most hg imports fail
<thumper> around 80%
<thumper> bzr-hg is not really going anywhere
<wallyworld> ok. so it's a limitation in our importer
<thumper> I'm looking at the imports
<thumper> wallyworld: the only thing I could suggest is to try the import locally
<wallyworld> thumper: thanks. poolie see thumper's comment ^^^^^^
<poolie> it's failing to connect to the server
<thumper> poolie: I see that, have you tried it locally?
<poolie> i can connect tothe url
<poolie> i didn't actually try bzr-hg
 * thumper tries
<thumper> poolie: bzr-hg fails miserably with that import
<thumper> I've commented on the bug
<thumper> I think that the server is looking at the client type
<thumper> hg client works fine
<thumper> bzr gets the http redirect
<thumper> and so fails on the unknown bzrdir format
<thumper> poolie: I added a bzr-hg task
<poolie> i see
<lifeless> anyone seen 'Module canonical.launchpad.webapp.login, line 440, in expireSessionCookie
<lifeless> value = request.response.getCookie(session_cookiename)['value']
<lifeless> on launchpad.dev logging in ?
<lifeless> TypeError: 'NoneType' object is unsubscriptable<br />'
<wgrant> No.
<wgrant> I blame Cassandra!
<lifeless> hah
<lifeless> so, did you gain much from my notes?
<wgrant> Mm. It looks good for some stuff (like the session bits you mentioned). But I need to look into it more deeply to see how it could feasibly replace much more than that.
 * thumper EODs
<lifeless> sinzui: ping
<lifeless> wgrant: its what, 4pm for you ?
<wgrant> lifeless: 4pm, yeah.
<lifeless> hmm, actually, I can just look at my mail in the morning.
<lifeless> I have a fix for Person:+commentedbugs
<lifeless> Its a one-word fix.
<wgrant> Hah.
<lifeless> https://code.launchpad.net/~lifeless/launchpad/malone/+merge/41127
<adeuring> good morning
<henninge> danilos: Hi!
<jelmer> wallyworld_, wgrant: Thanks for dealing with the permissions issue crisis! I'm looking at a proper fix now.
<wallyworld_> jelmer: no problem at all. i updated the incident report with a few extra details which i also included in the email
<jelmer> wallyworld_: Thanks.
<bigjools> hello
<wgrant> jelmer: Great.
<wgrant> jelmer: An unobvious bit of code, that one :/
<wgrant> jml: Thanks for landing that.
<bigjools> wgrant: my fix seems to have worked
<wgrant> bigjools: Yup, looks OK today.
<bigjools> no, to work out wtf they take so long to respond
<wgrant> bigjools: The "fix" being the removal of failure count assessments?
<bigjools> now*
<wgrant> Or increasing timeouts?
<bigjools> yes
<bigjools> nothing would have got failed if the assessment method had been left
<wgrant> Huh.
<wgrant> Interesting.
 * bigjools will land the fix properly today
<wgrant> So you increased the TCP timeouts across the board?
<bigjools> all xmlrpc connections
<wgrant> Hmm.
<bigjools> that's the *connection* timeout
<wgrant> Can we grab logs from palmer and co. and see what was happening around the time of the timeouts?
<wgrant> Also, do we have resource graphs from the non-virt builders? I'm guessing not.
<bigjools> yeah I'll sort that out
<wgrant> :(
<bigjools> we do
<wgrant> !
<wgrant> Yay.
<wgrant> There may be hope yet.
<bigjools> as in like the one I showed you for the virtual ?
<wgrant> No, I mean load and memory usage and stuff.
<bigjools> oh that
<bigjools> ummm
<bigjools> I'll check
<bigjools> do you know which ones (apart from palmer) to save me looking them up? :)
<wgrant> All of them, I believe it was.
<wgrant> One or two may have survived. But I don't recall which.
<jml> wgrant: it did it automatically
<wgrant> jml: Huh?
<jml> wgrant: I think there was some kind of mail queue delay
<wgrant> Ah.
<jml> wgrant: because another branch that I had forgotten about also landed.
<wgrant> Heh.
<jtv> stub: any idea why our makefile has "schema" depending on "clean_codehosting"?
<stub> No idea
<jtv> Having it depend on build definitely looks like overkillâwe don't _really_ need new WADL to build a new DB, do we?
<stub> Or should I say 'because that Makefile has become utterly unmaintainable'
<jtv> Too lateâI'm hacking & slashing at it.
<stub> I don't know when we need WADL.
<wgrant> jtv: make schema will be deleting branches, so it probably makes sense to clean up their directories too...
<jtv> wgrant: make schema deletes branches?  That's surprising.
<wgrant> jtv: It resets your DB.
<wgrant> So...
<jtv> wgrant: I'm waiting with bated breath for you to finish that sentence!
<wgrant> jtv: Is Codehosting going to be wonderfully happy if it tries to create a branch and there's already something there?
<jtv> wgrant: I still fail to see the connection between that and the DB schema.
<wgrant> jtv: 'make schema' will remove all your extra branches. If you create a new one, it's going to use an ID that's already on disk.
<jtv> wgrant: nope, I'm still not getting it.  What are my "extra branches"?
<wgrant> jtv: Any branches you create on top of the sampledata.
<jtv> Oh, you're saying it needs to reset the local codehosting state in conjunction with the schema.
<wgrant> Which are probably the same as those that are in the directories that clean_codehosting deletes.
<jtv> (Why not the local librarian state though, I have to wonder)
<jtv> Yes, schema also depends on clean_codehosting.
<wgrant> Hmm. clean seems to remove just about everything except the dev librarian.
<wgrant> Oh.
<wgrant> schema removes the dev librarian.
<jtv> ISTM schema may only need to depend on $(PY) and clean_codehosting.
<wgrant> Not even compile?
<wgrant> Looks like you may be right.
<jtv> I can't see compile doing anything more fundamental than UI-level work.
<jtv> Or wait, no, compile also did mailman.
<jtv> (Which it could do perhaps 10 seconds faster with a judicious -j option, but the change I have in PQM right now should make that sort of pointless)
<jtv> But I imagine mailman is not necessary for what we want from "make schema."
<wgrant> I would hope not.
<wgrant> Unless something imports it.
<wgrant> Which ew.
<jtv> We'll see, when my current branch clears PQM.
<wgrant> What does it do?
<jtv> Not break when you "make -j2" :-)
<wgrant> Hah.
<jtv> It's a pretty even break between "generate WADL" and "do everything else," so gratifyingly close to a 2Ã speedup.
<jtv> benji & I got worked up about build speeds last night.  He may have a way to skip the half a million optparse "add_option" calls from the WADL generation.
<jtv> Unexpected?  It was to me.
<jtv> In a 4-minute profiling run of wadl generation, add_option accounted for about one minute.
<bigjools> wow
<jtv> (Profiling added less than 100% runtime overhead for me, so estimate the gains at 30s)
<wgrant> jtv: Oh, last time I tried to profile the WADL generation it didn't finish...
<jtv> See?  Things aren't so bad these days!  :-)
<wgrant> Heh.
<jtv> It could be worth doing separate wadl generation runs for separate API versions, though single-cores shouldn't sniff at the repeated zcml overhead either.
<jtv> Ironically, "make" would be just the tool for limiting that job to only the files that need updating.  But the finer details are hidden away in a python script.
<wgrant> Yes :(
<wgrant> That was quite a mail backlog.
<bigjools> jml: did you change something recently in testtools?  I am getting errors about "can't import StartsWith" and I've dist-upgraded and updated-sourcecode...
<wgrant> And updated download-cache and remade?
<bigjools> I've never had to do that before
<wgrant> And the dozens of other dependency mechanisms? :D
<bigjools> that's the nice thing about standards, there's so many to choose from
<wgrant> But really, testtools is an egg. Update download-cache and run make.
<bigjools> utilities/update-sourcecode does all that
<wgrant> Doesn't that just update sourcecode?
<wgrant> rocketfuel-get is the one that does it all.
<bigjools> it just calls the latter
<wgrant> I think you have it the wrong way around.
<bigjools> last time I looked, anyway
<wgrant> rocketfuel-get calls update-sourcecode.
<bigjools> that's what I said
<wgrant> update-sourcecode pulls everything in sourcecode/
<bigjools> make clean seems to have fixed it
<bigjools> the Windows solution
<wgrant> Oh.
<deryck> Morning, all.
<bigjools> wotcha deryck
<wgrant> bigjools: There was an email about that a week ago.
<wgrant> An egg was changed.
<bigjools> ahhhhhhh I remember now
<bigjools> I fixed it on one box but not my other
 * bigjools goes back to tracking down PoolFileOverwriteErrors
<wgrant> :(
<wgrant> Copies?
<bigjools> trying to work that out but I 99% suspect so
<wgrant> :(
<bigjools> wgrant: https://launchpad.net/~rhpot1991/+archive/ppa/+packages
<bigjools> see mythbuntu-lirc-generator
<bigjools> I wonder if it's down to .debs moving between sources
<wgrant> bigjools: Let's see..
<wgrant> That doesn't sound likely.
<deryck> *sigh* why do we put up with this internet thing?
<wgrant> But this is Soyuz.
<bigjools> wgrant: the files were put in the librarian a few hours apart
<bigjools> currently hunting down where they are published
<wgrant> bigjools: I am confuse.
<wgrant> bigjools: That file isn't in any of the other sources in the PPA.
<bigjools> 2 files with the same name
<bigjools> yep
<wgrant> bigjools: Something is really screwed.
<wgrant> There was never a binary by that name published.
<wgrant> Of any version.
<wgrant> On any arch.
<wgrant> So WTF is it doing on disk?
<wgrant> It's not a half-written file?
<bigjools> no idea
<bigjools> hmmmm
<bigjools> it might be a merged account
<wgrant> Ahh.
<bigjools> the same file is in rhpot1991-merged
<wgrant> That would do it.
<wgrant> That is good news.
<bigjools> sigh
<bigjools> so I can either block merges of persons with ppas
<bigjools> or work out htf to merge them
<wgrant> I say you should block merges if there are undeleted PPAs on the source.
<wgrant> We have no hope of merging them.
<wgrant> bigjools: Can we convince the publisher to rename PPAs once it's deleted them, like we do with account deactivation?
<bigjools> we'd need to track the old and new names to do that
<wgrant> Why?
<bigjools> oh - once they are deleted it's fine
<bigjools> missed that but
<wgrant> Exactly.
<bigjools> bit
 * bigjools prepares sql to fix data
<bigjools> then, I need to solve the double-copy thing
<wgrant> Just the one PPA?
<bigjools> my life is fun-filled
<bigjools> wgrant: argh, there's a PFOE in ubuntu too
<wgrant> bigjools: Which file?
<lifeless> bigjools: hey
<lifeless> can I ask a favour
<bigjools> a few
<lifeless> https://pqm.launchpad.net/ - see the second job there from me - '[r=lifeless][ui=none][bug=668138] Add a DISTINCT to the commentedbugs subselect for performance.', bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/malone
<bigjools> gfortran-arm-linux-gnueabi_4.4.4-9_amd64.deb, gcc-arm-linux-gnueabi_4.4.4-9_amd64.deb, gobjc-arm-linux-gnueabi_4.4.4-9_i386.deb ...
<bigjools> well basically all the debs for gcc-defaults-armel-cross
<bigjools> lifeless: yo
<wgrant> :(
<wgrant> That package gave me nightmares.
<lifeless> bigjools: that should, in theory, be merged in about 1/2 an hour to an hour
<lifeless> bigjools: then buildbot, then qastaging etc
<lifeless> bigjools: it should fix our current highest frequency oops - but I'm going to be in transit ...
<bigjools> lifeless: you want me to watch it, or what?
<lifeless> yeah
<lifeless> if it disappears out for some reason, can you resurrect it - it passed ec2 overnight
<bigjools> bug 668138]
<_mup_> Bug #668138: Person:+commentedbugs timeouts <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/668138>
<bigjools> lifeless: sure
<bigjools> lifeless: why is that not "in progress" ...
<lifeless> when you EOD, if you can tag flacoste or someone
<bigjools> yarp
<lifeless> great
<lifeless> I'll try to check in from airports etc
<lifeless> thank you
<bigjools> I've subscribed the bug so I will expect a qatagger email
<bigjools> np
<bigjools> now, back to my thousands of oopses
<wgrant> bigjools: So, the package is fairly evil, and epochs are involved.
<wgrant> But filename conflicts should still have been caught :/
<wgrant> I hate epochs.
<bigjools> wgrant: there's a bug somewhere, should be easy to re-create
<jml> bigjools: we added StartsWith to testtools recently and removed it from lp
<wgrant> bigjools: I don't think we ever check for binary file conflicts on upload...
<wgrant> bigjools: We do for copies. And we check that the version isn't older than one already in the archive.
<wgrant> But I cannot see a binary file check anywhere :/
<bigjools> hmmm
<bigjools> I thought there was one
<wgrant> So did I.
<wgrant> There's one in dscfile
<wgrant> And PackageUploadSource does it as well.
<wgrant> But not PackageUploadBuild.
<bigjools> doesn't the source upload check inspect the debs mentioned in the control file?
<wgrant> There is no source.
<wgrant> This is a binary upload.
<bigjools> in the original source I mean
<wgrant> Hmm? You can't get binary details directly from the source.
<bigjools> I guess not file names
<bigjools> crap
<wgrant> You can't get anything at all.
<wgrant> Normally they will sort of match.
<bigjools> you can get the package name
<wgrant> Not necessarily :/
<wgrant> It's conventional, but I don't believe policy requires it.
<wgrant> And lots of packages violate it.
<wgrant> We probably just need to do the binary file check.
<wgrant> And thwack people who write packages like that :)
<bigjools> yay
<lifeless> policy can't require it, because folk have transitional packages and so forth
<wgrant> Right.
<wgrant> I mean more to thwack people who write packages which duplicate versions.
<bigjools> well the lack of a check in the uploader is thwackable
<wgrant> Certainly.
<bigjools> especially since it's done in the package copier!
<wgrant> It's just distressing that it has been missing for nearly five years.
<bigjools> sounds like some code refactoring is in order
<wgrant> I'd like to push all copies and uploads through PackageUpload, really.
<bigjools> mmmm maybe, I dunno
<bigjools> need to think about that, because I want to put copies through a job at some point
<wgrant> bigjools: I think we probably need to attempt to discuss and think about redesigning all the upload and copy stuff at some point... the current model seems to be creaking.
<wgrant> Badly.
<bigjools> if you can define exactly where and why it's creaking, yes
<wgrant> A few things immediately spring to mind... We have no copy tracking. We unnecessarily reupload files on privacy changes. We have incorrectly duplicated inefficient checking logic. archiveuploader is madness.
<wgrant> We ideally want NSS to not suck. And backports would be nice. And we have issues with copying between series and archives with different architectures.
<wgrant> And any solutions to these are going to be hacked onto a system that was designed before archives existed, and even before pocket copies were envisioned.
<bigjools> reuploading actually helps, it stops the request from timing out when the security team unembagoes something!
<bigjools> but yeah, it's unnecessary.  Wish we'd known that at the time.
<wgrant> Yeah :/
<bigjools> I need fud
<lifeless> lp.translations.windmill.tests.test_languages.LanguagesFilterTest.test_filter_languages
<lifeless> bb test failure
<lifeless> https://bugs.launchpad.net/rosetta/+bug/676980
<_mup_> Bug #676980: lp.translations.windmill.tests.test_languages.LanguagesFilterTest.test_filter_languages failing <Launchpad Translations:New> <https://launchpad.net/bugs/676980>
<jml> hi
<jml> hi
<jml> (oops)
<gmb> Arrgh. Why does LaunchpadFormHarness use '127.0.0.1' rather than 'launchpad.dev'? That's odd.
<jml> I was just about to raise the test failure issue
<jml> lifeless: what should I do about it?
<lifeless> jml: you could analyse the test to see if it has race conditions like those fixed in other tests recently.
<lifeless> jml: you could run it locally and see if it passes.
<lifeless> jml: both of those would be useful to do, I think.
<jml> lifeless: you mean, actually address the root cause of the problem?
<lifeless> <awe>
<jml> ok.
<jml> I really did have better things to do today.
<lifeless> jml: amen
<deryck> or we could just disable the windmill tests entirely ;)
<jml> deryck: the sky would fall in
<deryck> jml, clearly, some are of this opinion.
 * deryck isn't afraid of falling sky
<lifeless> deryck: oh good morning
<deryck> hi lifeless
<lifeless> deryck: can you take on the +commentedbug thing I handed off to Julian before - you'll be awake longer, and its right up your alley :)
<danilos> lifeless, I don't remember seeing that test fail ever in the past, and I am sure we haven't changed anything remotely related :(
<lifeless> danilos: ouch
<deryck> lifeless, sure, I don't mind.
<lifeless> danilos: I'm sure we'll figure it out soon enough
 * deryck looks at scrollback
<lifeless> deryck: its in devel now, needs to get to stable, be qa on qastaging and then (assuming all other bits are qa'd) deployed : will fix the highest oops we have :)
<danilos> lifeless, yeah, let's see if it's spurious first, and if it is, then really ouch... I prefer reliably failing tests :)
<deryck> lifeless, sure, I'll keep an eye on it.
<lifeless> deryck: cool
<deryck> jml, lifeless, danilos -- there is no timeout value on the assert in the windmill tests.  passing in the past has likely been timing luck.
<jml> deryck: I know next to nothing about windmill tests. Got something I can look at to figure out what you mean?
<danilos> deryck, quite some luck for roughly 6 months or so since the test was introduced, but I won't dispute it
<deryck> indeed, this is browser testing for you ;)
<lifeless> ok, must run
<lifeless> good luck!
<deryck> jml, danilos -- http://pastebin.ubuntu.com/533852/
<deryck> jml, danilos -- so what happens is this: there is a page load wait and then the assert, but if the page load is complete but the page not entirely initialized, these sort of asserts can fail.  the timeout buys you some time but can potentially make the test slower.
<deryck> however, the assert will pass as soon as the element is available.  so doesn't always wait the full timeout.
<jml> deryck: I see.
<danilos> deryck, yeah, makes sense, thanks
<danilos> deryck, since you've already gone through the trouble, do you mind landing a fix as well? :P
 * danilos whistles innocently
<deryck> heh
<deryck> sure, I can do it.  rs=danilos? :-)
<danilos> deryck, yep
<danilos> deryck, even r=danilos â I did look at it and see no problems :)
<deryck> excellent :-)
 * deryck branches and runs test for paranoia
<danilos> deryck, assigned bug 676980 to you, thanks again :)
<_mup_> Bug #676980: lp.translations.windmill.tests.test_languages.LanguagesFilterTest.test_filter_languages failing <Launchpad Translations:Triaged by deryck> <https://launchpad.net/bugs/676980>
<deryck> np at all!
<deryck> this is all I've been doing for 3 weeks anyway... trying to get windmill tests to pass. *sigh* ;)
<jml> deryck: thanks.
<jml> anyone working on bug 636193?
<_mup_> Bug #636193: feature flags need to self document <feature-flags> <Launchpad Foundations:Triaged by mbp> <https://launchpad.net/bugs/636193>
<bigjools> deryck: disabling windmill.... what a great idea.. :)
<deryck> bigjools, you know, no one ever says differently, until you ask on a mailing list about that option. ;)
<bigjools> deryck: we're a bunch of 2-faced....
<deryck> although, I think we need a new process doc.... if you're in favor of keeping windmill, you are required to do the next lazr-js upgrade branch.
<bigjools> lol
<lifeless> all hail the great free internet gods
<bigjools> wgrant: https://bugs.launchpad.net/soyuz/+bug/66974
<_mup_> Bug #66974: Binary versions not checked correctly <boobytrap> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/66974>
<bigjools> wgrant: that is an *old* bug :/
<deryck> mars, hi.  you learn anything from that test run I sent you yesterday?
<mars> deryck, no, unfortunately.
<mars> I looked through, and looked at the windmill source code a bit.  Nothing jumped out at me.
<mars> But I did find that the socket code was doing something odd
<mars> it only logs socket connections after they are connected.  So the connections causing problems may not be logged, or they could be left open
<deryck> mars, ok, so what do I do here?  I'm 3 weeks in on this.  rockstar has 3 weeks prior to that.  it's majorly blocking work, but I need to get the new lazr-js in.  How do I get it landed?
<mars> deryck, I am thinking about it.  There should be some way around this.
 * rockstar votes "dump windmill" and runs away.
<mars> deryck, what test did you run that had the thread trash?
<mars> deryck, locally, the exact command please
<deryck> mars, pick a test, pick any test, ;)  but I did:  xvfb-run ./bin/test -cvvt test_bug_commenting
<deryck> mars, the bug commenting test runs pretty quickly, which is why I picked it.
<mars> deryck, I am going to trying running it through wireshark, maybe that will catch something
<deryck> mars, ok.  I'll just hang out and wait on you.  Thanks for the help!
<mars> jml, around?  danilos just sent a list mail about a testtools failure in a twisted test case.  It sounds like something you may know about?
<jml> mars: it's not a testtools failure
<lifeless> jml: whats the tag to say 'this is a feature request'
<danilos> lifeless, "feature" :)
<jml> lifeless: feature, iirc. there's a wiki page w/ bug tags
<lifeless> jml: I feel a strong desire to mark all 'feature' bugs as wishlist.
<lifeless> jml: but I'm restraining myself.
<jml> lifeless: thank you
<mars> jml, ah, by 'testtools failure' I meant 'it may be something to do with testtools', not within testtools itself
<jml> mars: it doesn't have anything to do with testtools
<danilos> lifeless, well, we've basically stopped using wishlist priority and usually mark feature bugs as 'low' instead
<mars> my mistake
<lifeless> jml: thank you for restraining myself :) ?
<jml> lifeless: yes.
<lifeless> jml: you're welcome
<lifeless> danilos: why don't we use wishlist?
 * jml so does not have headspace for this discussion right now
<danilos> lifeless, because it's orthogonal to priorities, I guess... but it was an ancient discussion and I don't remember the specifics anymore; I am sure sinzui does, though :)
<lifeless> jml: for reference - https://bugs.launchpad.net/launchpad-project/+bugs?field.importance=High&field.tag=-feature
<sinzui> lifeless, https://dev.launchpad.net/BugTriage
<sinzui> lifeless, ^ since there are three states, and Lp defines the use of critical, we have two importance states left, high, because we only work on high things, and low, because we might do fix the issue in the future
<lifeless> sinzui: that diesn't discuss wishlist at all
<lifeless> sinzui: I'm fine with not using medium - there is a similar experiment running in testtools and related projects.
<lifeless> however, I'm not sure bugs /are/ a work queue. Nevertheless - thanks for the link
<sinzui> I use medium in my own projects. I rejected critical
<sinzui> I want real words. do now, do soon, do when at an opportunity
<lifeless> sinzui: I think thats a different concept to bugs per se, but related.
<lifeless> anyhow, I have to go through security, I hope to be back online again in a bit
<jml> mars: oh, I'm sorry, I didn't see the *actual* error that's in the full log but not the email
<mars> jml, no problem, I should not have punted to you without understanding it first myself
<mars> deryck, argh, I keep running into the 'text=bytea' problem.  I can't even run launchpad.dev :(
<mars> this after a database drop and rebuild
<mars> back to code inspection, maybe that will spark something
<deryck> mars, ok.  I didn't have that issue.
<deryck> mars, are you running against my branch?
<mars> deryck, yes
<deryck> mars, ok, cool
<jml> danilos: answered. hopefully enough for you to go on.
<danilos> jml, thanks
<danilos> jml, I've ran the tests locally already, and just submitted directly to pqm
<jml> danilos: are you going to try to address the problem?
<mars> deryck, I am looking at the windmill.server.https code now, maybe there is some way to figure out which threads are still around, and why
<mars> deryck, taking an early lunch, back in a little while
<danilos> jml, most likely not, since I'm off after tomorrow and I got a few pressing issues for today and tomorrow
<jml> :(
<jml> well, maybe someone else will pick it up
<deryck> mars, ok, on call.  talk to you soon.
<EdwinGrubbs> leonardr: ping
<jml> james_w: do you know if https://bugs.launchpad.net/launchpad-code/+bug/365098 has been fixed?
<_mup_> Bug #365098: Anyone who can write to a sourcepackage should be able to set the official package branch <package-branches> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/365098>
<james_w> I don't think so
<jml> ok.
<jml> thanks
<leonardr> edwingrubbs, hi
<EdwinGrubbs> leonardr: I was wondering if I should just add some parameters to a REST method to slice it, or if can re-use the REST API's ability to slice collections. I would just convert the method to a collection, but it has other parameters.
<leonardr> EdwinGrubbs: if your method returns a resultset, it should already be sliceable
<EdwinGrubbs> leonardr: right now, it returns a list so that only one call is necessary to get all the data that the timeline graph needs. Would I be able to use a DecoratedResultSet, or does each item in the result need to be an exported entry?
<leonardr> EdwinGrubbs: a list should also be sliceable, but at that point what's the point? are each of these items huge?
<EdwinGrubbs> leonardr: each item is not huge. If it just needs to be sliceable, then the DecoratedResultSet should work.
<leonardr> EdwinGrubbs: yes, or just returning a list should also work
<leonardr> try it and see
<bigjools> leonardr: can I poke you about a webservice problem for a moment?
<leonardr> bigjools, sure
<bigjools> leonardr: thanks - you reviewed a branch of mine a while ago and I am just getting around to land it
<bigjools> https://code.edge.launchpad.net/~julian-edwards/launchpad/api-expose-builders/+merge/39379
<bigjools> I've not changed it and it's now failing tests: http://pastebin.ubuntu.com/533928/
<bigjools> I can't figure out why
<leonardr> ok
<bigjools> I suspect some wadl problem but I've done "make clean;make" before running the test
<leonardr> bigjools: yes, you'd get that problem if nopriv_launchpad was using an old wadl. do you have any caches from launchpad.dev in your ~/.launchpadlib?
<leonardr> it would be a bug for the testrunner launchpadlib to look there, but it could be happening
<bigjools> I am getting deja vu
<bigjools> yes there's a cache there
<leonardr> remove that cache and see what happens
<bigjools> ok, running ...
<leonardr> if it works, then file a bug against launchpad saying that launchpadlib_for is still looking in .launchpadlib instead of a temp directory. it might be something that can be fixed by upgrading launchpad's version of launchpadlib, or it might have been _caused_ by upgrading launchpad's version of launchpadlib so the order of arguments are different
<bigjools> ok
<bigjools> still failing :/
<bigjools> I removed ~/.launchpadlib/api.launchpad.dev/
<leonardr> bigjools: ok, step into launchpadlib_for and see what cache directory it comes up with
<leonardr> then, keep an eye on that cache directory
<leonardr> look at the wadl and see if 'builders' is mentioned
<bigjools> leonardr: the code has "cache = tempfile.mkdtemp ...."
<dobey> hey guys. does the source package recipe builder stuff not use the same buildds as PPA builds to build the source packages?
<bigjools> dobey: why do you say that?
<dobey> bigjools: https://code.edge.launchpad.net/~ubuntuone-hackers/+recipe/protocol-dailies/+build/7779
<bigjools> dobey: natty is not ready for recipe builds
<dobey> bigjools: what is prevening it from being "ready" exactly? it makes it hard to build dailies for the ubuntu development release :-/
<bigjools> dobey: talk to abentley
<dobey> abentley: hi. :) ^^ what's blocking source recipes from working on natty?
<abentley> dobey: What's preventing it from being ready is that bigjools thinks maybe it caused the build farm failures of a couple weeks ago.
<abentley> bigjools: Success with memory limit on recipe build: http://librarian.dogfood.launchpad.net/57529959/buildlog.txt.gz
<abentley> https://code.dogfood.launchpad.net/~abentley/+recipe/test/+build/4803
<dobey> hrmm
<bigjools> abentley: nice
<bigjools> abentley: the new manager is better about timeouts now so we should try again tomorrow
<bigjools> well, when I get it rolled out tomorrow
<abentley> bigjools: I thought the new manager was already rolled out?
<bigjools> cowboyed
<bigjools> well - the timeout fix to it is cowboyed
<bigjools> the fix landed today on devel
<bigjools> I'll roll it tomorrow
<abentley> bigjools: cool.  I'll get this rlimit branch reviewed, and we can throw that in the mix, too.
<bigjools> leonardr: "builders" is not in the wadl that's in the cache dir :/
<leonardr> bigjools: in that case your top-level collection is not being properly published
<bigjools> leonardr: I wonder if the recent interfaces apocalypse broke it - I bet that's it actually
<bigjools> I need to register it in the zcml
<bigjools> abentley: excellent - we can coordinate tomorrow
<bigjools> thanks for the help leonardr
<leonardr> sure
 * bigjools -> fud
<Ursinha> bigjools, after fud... could you please give your ideas on bug 673985?
<_mup_> Bug #673985: Provide a way to mark something as QA done before it was landed <qa-tagger:New> <https://launchpad.net/bugs/673985>
<dobey> leonardr: btw, i added the print of the team object yesterday, and it still failed, but the object was valid and pointed at the right thing. trying a 3 second sleep now to see if that helps at all.
<leonardr> dobey: remind me what the exact problem was?
<dobey> leonardr: the two calls to getMembersByStatus
<jml> the planet is down :(
<abentley> jml: And the sky is up.  Unless you're in Australia :-)
<jml> abentley: :D
<jml> I'm off for the day. Tomorrow, email email email and maybe some naughty programming.
<leonardr> dobey: i don't have any advice beyond stepping through the code and seeing what's different about the data structure or the code execution path in the first time vs. the second time
<dobey> leonardr: right. i was just letting you know where i was at now :)
<leonardr> sure
<deryck> bdmurray, ping
<bdmurray> deryck: hey
<deryck> bdmurray, hey.  So we're expiring Ubuntu bugs now each day.  Doing 200 a day as we discussed....
<deryck> bdmurray, and I'll let this run for about month until we clear the ~6000 bug backlog.
<bdmurray> deryck: right
<deryck> bdmurray, and then we'll open it up to expire for every project.
<deryck> bdmurray, so if you see any issues, let me know.
<deryck> mars, hi.  Any luck on my branch?  (he says too anxiously) :-)
<mars> hehe
<mars> deryck, I had some time to think about it.  The problem is figuring out which threads are still around, and why
<deryck> mars, right.  That's where I got stuck, too.  I pdb'ed into the middle of windmill.server.https and just couldn't figure out how to work that out.
<mars> deryck, I was thinking, as a first step, could we debug the https server to print a threadID whenever a new thread is started, and again when it is told to shut down?
<mars> deryck, and a threadid when it makes a socket connection
<mars> that way you could see "tID 2: started; tID 2: http://blah; tID2: stopping"
<deryck> sure, that makes sense to me.
<mars> any thread that does not print "stopping" is a problem
<deryck> right
<deryck> mars, are you trying this now?
<mars> I dislike this approach, it's dirty, inelegant, but it might work
<mars> deryck, I thought about it.  Do you think you could try patching for it?
<deryck> mars, I don't think I understand where windmill is starting and stopping threads completely, but I can try to work it out.  Also, I had issues with print statements being swallowed earlier and had to pdb to get at objects.
<mars> deryck, try printing to sys.stderr, it should work
<deryck> ah, ok
<mars> deryck, perr = functools.partial(print, file=sys.stderr)
<deryck> ok
<deryck> mars, thanks.  That gets me going again.  Will try again here in a second.
<mars> deryck, no problem.  I think windmill.server.https is the right place to start.  Each server instance is a ThreadingMixin, so you will have a bunch of them.
<deryck> right, that's what I thought, too.
<mars> thankfully that part is close to the standard library,
<mars> deryck, ah, just saw WindmillHTTPServer.process_request() (line 433).  It calls thread.start().  That is one good option to begin at.
<deryck> mars, gotcha.  Thanks.
<mars> deryck, let me know how it goes.  I am going to start hacking on my other deadline-driven work.
<deryck> will do
<EdwinGrubbs> mars: do you happen to remember where we configure certain object types not to be security proxied?
<mars> EdwinGrubbs, sorry, I have no idea
<EdwinGrubbs> sinzui: ^^^
<deryck> thumper, ping
<mars> EdwinGrubbs, pick an object you know is not proxied, then grep the ZCML for it?
<sinzui> Edwin Utility verses SecuredUtility in ZCML
<jamesh_> sinzui: that just determines whether the returned utility is security proxied.  It doesn't set a policy for a class
<EdwinGrubbs> sinzui: yes, but a dict, str, and I believe some other classes we have specified do not get security proxied when they are returned from a secured utility.
<jamesh_> EdwinGrubbs: there is some Python code that does this for a Storm type in storm/zope/__init__.py
<EdwinGrubbs> jamesh_: thanks
<dobey> leonardr: ah-hah. i've found the problem
<leonardr> dobey: ?
<dobey> leonardr: foo = team.getMembersByStatus(); foo.extend(team.getMembersByStatus())
<dobey> leonardr: apparently the .extend() there causes launchpadlib to attempt to alter the list via the API
<dobey> leonardr: which of course fails (though the traceback is whacky)
<leonardr> dobey: so it thinks that 'extend' is a webservice method?
<dobey> leonardr: i guess. if i do foo = [x for x in team.getMembersByStatus()] to assign it, and then do foo.extend(team.getMembersByStatus()) after, it seems to work
<leonardr> ok, good to know
<dobey> leonardr: is that a bug i should file against lplib/wadllib/restfulclient? or is it just me trying to be too clever and i shouldn't do that?
<leonardr> dobey: i think you're trying to be too clever. objects you get from the web service play by different rules than normal python objects, and part of our next project is making this more clear
<dobey> ok
<deryck> booyah!  I have some knowledge now, mars!
<mars> deryck, great!  What did you find out?
<deryck> mars, so all the threads that are left behind are the launchpad.js file.
<deryck> mars, http://pastebin.ubuntu.com/533988/
<mars> I thought we used lp.js now?
<deryck> mars, it's called launchpad.js, I thought.
<deryck> mars, lp.js is a file in lp.app.javascript
<mars> ah
<mars> deryck, please please /please/ send me patch that made this possible :)
<deryck> mars, and launchpad.js on disk is 1.3 Mb now.
<deryck> about 450k compressed.
<mars> ok, that's a bit better
<deryck> mars, presumably windmill is trying the compressed form, right?  but I guess the browser and/or server is hanging making sense of it?
<mars> hmm, could be.  I wonder, it must be working, right?  Otherwise the page would not function.
<mars> deryck, that is the only javascript file?
<mars> no, Mochikit.js as well
<deryck> mars, but it is the only js file when not in devmode that we build.  and playing locally in devmode or not, everything works fine.  Thought in devmode the browser can slow in firebug because of all the individually linked files.
<deryck> mars, meant only file besides mockikit
<deryck> mars, and here's my windmill patch to make all this noise.  :-)  http://pastebin.ubuntu.com/533990/
<deryck> mars, does this suggest our only way forward is making the file smaller or getting dynamic loading via a combo loader working?
<mars> deryck, I am not sure
<mars> if that suggests the only way forward
<deryck> ok
<mars> deryck, did the file size change much across versions?
<deryck> mars, oh way bigger.
<mars> ?
<mars> why?
<mars> did the YUI core grow?
<deryck> yes, many more files now.
<deryck> they assume combo loading not concating. :-)
<mars> so we may need to look at unloading some stuff from our own bundle then
<mars> new functionality that we do not need
<deryck> yeah, that's one way.
<deryck> I can consider the options now. at least I know what's going on.
<LPCIBot> Project devel build (228): FAILURE in 3 hr 42 min: https://hudson.wedontsleep.org/job/devel/228/
<LPCIBot> * Launchpad Patch Queue Manager: [r=danilos][ui=none][bug=676980] Fix up a translation windmill test
<LPCIBot> to be more stable by adding a timeout value to an assertProperty.
<LPCIBot> * Launchpad Patch Queue Manager: [r=EdwinGrubbs][r=benji][ui=none][bug=318842][bug=383615][bug=488491]
<LPCIBot> Enable tests.
<LPCIBot> * Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][no-qa] Changes to
<LPCIBot> cron.publish-copy-archives copied from the actual changes made
<LPCIBot> in production to get it working
<LPCIBot> * Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][no-qa] launchpad-buildd updates for
<LPCIBot> SIGILL to not automatically retry the build
<LPCIBot> * Launchpad Patch Queue Manager: [r=jelmer][ui=none][bug=676479] Make XMLRPC connections to builder
<LPCIBot> slaves respect the config's timeout value. Also add more logging.
<LPCIBot> * Launchpad Patch Queue Manager: [r=benji,edwin-grubbs][ui=none][no-qa] Support "make -j2"
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=668138] Add a DISTINCT to the commentedbugs
<LPCIBot> subselect for performance.
<LPCIBot> * Launchpad Patch Queue Manager: [r=benji,
<LPCIBot> edwin-grubbs][ui=none][bug=676489] fix bug 676489 by adding code in
<LPCIBot> the apache log parsing to handle a space in the path.
<_mup_> Bug #676489: librarian apache log parser failing to parse bogus line in access log <librarian> <oops> <Launchpad Foundations:Triaged by gary> <https://launchpad.net/bugs/676489>
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa] Add person merge queue listing page
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap,gmb][ui=none][bug=594247] use JOIN and LEFT JOIN to link
<LPCIBot> tables needed to search for bugtask releated to a structural
<LPCIBot> subscriber;
<LPCIBot> refactoring of BugTaskSet.search() and BugTaskSet.serachBugIds();
<LPCIBot> more tests for BugTaskSet.search()
<LPCIBot> * Launchpad Patch Queue Manager: [r=jtv][ui=none][bug=675562] Fail requests for master stores if we
<LPCIBot> are in read-only mode,
<LPCIBot> no matter what database policy is currently installed.
<LPCIBot> * Launchpad Patch Queue Manager: [r=gmb][ui=none][bug=654372] Optimise source domination down from a
<LPCIBot> few minutes to less than a second.
<rockstar> Thanks LPCIBot.  I was trying to read mars and deryck's conversation...
<mars> lol
<mars> I wonder why IRC bots aren't aware when a conversation is going on - they could just wait 10 seconds for an idle moment in the channel
<benji> good point, mars
<rockstar> deryck, mars, so, YUI's *-min files concatenated together, the files are about 850K
<EdwinGrubbs> leonardr: I'm not sure if you would consider this a bug, since I can get around the issue by returning a DecoratedResultSet of IEntry objects instead of dicts. ResourceOperator.should_batch() decides not to batch list objects, which is why a list of dictionaries is encoded fine, but a DecoratedResultSet of dicts does not work.  BatchingResourceMixin.batch() pukes on dicts in result sets because it tries to call checkPermission()
<EdwinGrubbs>  and EntryResource() on the dict. For some reason, checkPermission() blows up because it tries to pass the dict to weakref.ref().
<mars> rockstar, IIRC we rolled our own bundle before
<deryck> rockstar, plus all our files.
<rockstar> deryck, mars, yeah, this is WITHOUT lazr-js (I've just submitted the lazr-js abandonment branch to PQM)
<rockstar> mars, deryck, roll your own bundle is actually rather complicated now, since the dependency graph is a little frustrating to follow in any scale.
<mars> :(
<deryck> rockstar, what do you mean send to pqm?  You've another yui upgrade branch?
<leonardr> edwingrubbs: in general our support for named operations that return lists of non-entries is poor
<EdwinGrubbs> leonardr: I guess the security proxy knows not to bother calling checkPermission() on dicts, so maybe the REST API should copy its logic.
<rockstar> deryck, sorry, I'm working in the U1 context now...
<deryck> ah
<mars> rockstar, deryck, not fun.  We are fighting the grain of the library then.
<deryck> we need to get a proper loader going.  tis the one true way
<leonardr> EdwinGrubbs: go ahead and file a bug, i don't think we'll get to it for a good while though
<rockstar> mars, deryck, this is not a bug that's ever going to get fixed, I suspect.
<EdwinGrubbs> leonardr: ok, thanks
<mars> deryck, that would be nice, but we need to find one possibly lame way if we want to land your patch right now :)
<mars> rockstar, what but isn't going to get fixed?
<mars> bug
<poolie> hi all
<mars> Hi poolie!
<rockstar> mars, the 512K bug.
<mars> right
<rockstar> mars, did I tell you that mikeal said "windmill is dead?"
<mars> rockstar, I disagree.  I think it will be fixed, but it will be 5 years from now :)
<mars> rockstar, yes, we discussed that
<rockstar> mars, no one is working on windmill anymore.
<mars> rockstar, wait, was the 512K bug in windmill, or in Firefox?
<rockstar> mars, I know I keep banging this drum, but windmill is not the tool of the future.
<rockstar> mars, it was in windmill, because we could run in the non-devmode and it worked fine for us.
<mars> rockstar, that may be true, and I am all for investigating other alternatives, but I don't think tossing out windmill will help us land the YUI 3.2 patch right now.
<rockstar> mars, I don't see why it wouldn't.  Windmill is what's blocking the landing.
<mars> that should read, I don't think we can take tossing out windmill as an option to get this patch landed
<rockstar> mars, so then we wait until the next time someone needs to update the javascript
<mars> rockstar, you mean, we will have to wait until then to evaluate and replace the tool?
<deryck> I think we need to sort out the tool asap, and I also don't think the tool gains us that much right now. ;)
<rockstar> mars, well, if we don't do something drastic, we'll continue to roll our own.
<rockstar> mars, deryck, it's on my list of things to do for the next six months to set up our own combo loader.  This might help in the future.
<deryck> so I think in the short term, I'll see what files we can exclude.  But this is non-trivial, too.  Requires build changes, even if minor build changes.
<rockstar> mars, deryck, just today, beuno and I weighed the cost of rolling our own yui v. just having a larger file, and we just decided on the larger file, because rolling your own becomes a maintenance nightmare.
<mars> deryck, I think you just have to change the file list in base-template-macros, no?
<deryck> I thought it was just a for file in dir: concat script.
<rockstar> deryck, you know how I feel about this.  As we workaround it this time, we forget about it until we hit it again, and then we try and work around it again.
<deryck> but maybe it gets the list from the base macro template.
<deryck> rockstar, you know I *totally* agree :-)
<deryck> these tests aren't even really very good.
<deryck> I don't think they gain us anything.
<deryck> but I've said all this before ;)
<deryck> well, gain us much. ;)
<lifeless> afternoon :)
<rockstar> deryck, so you can roll your own, but be prepared for some really obscure errors.
<deryck> rockstar, yeah, I'm not found of this option either.  But I don't see I have any other option.  It's disable tests or roll my own smaller file (and hope for the best).
<rockstar> deryck, also, make sure you run with filter: 'debug' in your LAZR_CONFIG, otherwise, you won't see breakages.
<rockstar> deryck, I'm not convinced that we can get much smaller.  We have A LOT of javascript.
<mars> deryck, fwiw, if we had some insight on browser-level testing from U1, Landscape, ubuntu.com/ISD, or current industry best practice, then cool.  But it will take that level of investigation to convince everyone to toss the tool out.
<mars> rockstar, deryck was saying the YUI core grew between versions - that is features, isn't it?
<mars> we may need to switch to -base.js everywhere
<mars> and then build up
<deryck> mars, I disagree. No one understands the tool or any other option.  There is just fear and paranoia.  I'm not willing to do such research for a pointless lost argument...
<deryck> mars, and it's too much to change on my own.
<deryck> i.e. a weekend hacking to proof of concept.
<mars> right, I understand
<rockstar> mars, well, not entirely, and some of those features are for new core things.
<rockstar> mars, remember, we're going from 3.0 to 3.2.  That's a pretty big jump.
<mars> deryck, fwiw, I picked windmill in the first place (2 years ago now).  If you want, I can explain the reasoning some time.
<rockstar> Anim, for instance, now wants to use Transition in some browsers.
<deryck> mars, no, that's ok. :-)  I'm not against your original choice.  There are things I quite like about windmill even....
<mars> rockstar, yeah, and unfortunately I have not been tracking the library - I don't know much about the changes
<deryck> mars, I'm against the blind loyality to a system that at this point doesn't provide us much.
<poolie> jam: aren't there two north american losas?
<jam> As I understand it
<poolie> jam, here, unless it's private
<mars> poolie, Chex and mbarnett
<jam> I'm not sure why mthaddon is the primary one working on my request
<jam> I assume it is a comfort level thing?
<jam> He's the only one who has posted to the rt
<deryck> anyway, I need to EOD mars and rockstar.  I'll see what I can work out tomorrow.
<rockstar> deryck, okay, keep me posted.
<poolie> bye deryck
<mars> deryck, I agree, and I am for evaluating alternatives - all software has a lifecycle.  (Even zombies like Emacs and vi!)
<deryck> lifeless, I think your branch is still playing in buildnot.
<deryck> heh, didn't mean that typo.
<mars> hehe
<rockstar> mars, mikeal said that basically everything they hated about Selenium when they started Windmill is basically fixed now.
<lifeless> deryck: kk
<deryck> but one last thing on this....  it's not *just* the tool.... it's that we have no idea why to even use the tool.  browser testing can't be approached like unit testing.
<mars> deryck, good point
<deryck> we will end up with the same crappy tests run by another tool
<mars> :D
<mars> funny, and true
<deryck> which is why I advocate disabling the tests. :-)  Kill the pain.  Get on with work, and figure out to how to do this right before trying again.
<rockstar> mars, U1 is probably going to go with YETI.
<mars> rockstar, no idea what that is
<mars> something else that needs shaving
<mars> ?
<deryck> anyway, bye all.
<rockstar> mars, YETI = Yahoo Easy Testing Interface. It's how the YUI tests get run.
<jam> poolie: as an aside, I haven't seen any real l-osa interaction on any channel other than -ops, and even then I've gotten misses (though probably if I haven't heard a response in 30min, I should just send it again)
<mars> rockstar, got it: http://yuilibrary.com/projects/yeti/
<rockstar> mars, yeah.  It's more for pre-commit tests than it is for CI.
<mars> "Yeti is designed to help you run tests before you commit. It compliments existing CI tools like Selenium and Hudson which run tests post-commit. Yeti is not a replacement for those tools."
<rockstar> But I don't see why it can't also be used for CI, but reid advised against it.
 * lifeless looks for another handoff
<mars> rockstar, windmill and stuff test the full house of cards - that is why we started using them, and why they fall over all the time
<poolie> jam, like francis said, they're just swamped
* flacoste changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: -  | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<poolie> perhaps it's worth just shelving it entirely until they can give you some dedicated time?
<rockstar> mars, yeah, but windmill was actually developed for CI type tests.
<poolie> hi flacoste
<flacoste> hi poolie
<rockstar> And the tests we wrote for it are very much CI type tests.
<jam>  /wave
<flacoste> hi jam
<mars> rockstar, yep.  Deryck's argument partly hinges on our understanding of, and deriving value from, that fact.
<jam> poolie: I certainly think that the best use of time would be to just pre-arrange a couple of hours, and have a live conversation and testing of it.
<jam> I'm not all that great at sysadmin stuff, the production config is nothing like local config, etc.
<mars> rockstar, we also have the complimentary xUnit suite with YUITest
<jam> I don't have access to production to see what it is actually like
<poolie> it seems like you could (are?) waste a lot of mental energy waiting around to see if they've responded
<mars> rockstar, so, two tools, two roles.  What do other projects do in the industry, and at this company?
<rockstar> mars, I think that Launchpad is kinda the leader in js testing, actually.  This means we're kinda feeling around in the dark.
<jam> poolie: well, today I mostly switched off, and was doing code review, etc, so not a lot of waste
<jam> but certainly I spent some time making sure the init script worked on my machine
<jam> but it doesn't match production
<lifeless> rockstar: within Canonical?
<rockstar> lifeless, yes.
<lifeless> qa folk - https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html seems to be missing 10 new commits, am I just eager, or is something wrong ?
<flacoste> rockstar, mars: yeah, nobody else is really paying that much attention to automated JS testing
<mars> lifeless, you can check the logs on devpad, if you want.
<lifeless> mars: https://devpad.canonical.com/~lpqateam/qa_reports/logs/output-launchpad-stable.log does not have a timestamp in it
<rockstar> flacoste, landscape is, and they're having problems too.  jstestdriver randomly decides not to run tests.
<mars> lifeless, really?  That's an oversight
<flacoste> rockstar, mars: Landscape started with Selenium and dropped it
<mars> lifeless, we should fix that
<flacoste> rockstar, mars: now Landscape is ysing yuitest with jstestdriver as a harness
<lifeless> mars: have a look :)
<rockstar> flacoste, yeah.  I assume, at some point, we'll swing for the fences and actually get a home run.
<flacoste> rockstar, mars: ISD started using Selenium, than Windmill now looking into Selenium2 (never really systematized on anything)
<rockstar> ...Or 6 runs, if you're in that part of the world.
<mars> flacoste, ok, so is the whole integration testing idea a mountain not worth climbing, or are we just not committed enough?
<poolie> i'm going to try to pilot more today, depending what else is in my inbox
<flacoste> mars: deryck would say not worth climbing, i say not commited enough
<flacoste> mars: industry standard is selenium-grid
<rockstar> flacoste, yeah, we're in a black hole.  U1's fix seems to be "don't go to long before we can fix it" and lifeless' recent changes might be helpful there.
<mars> flacoste, rockstar, or possibly, no one save BigCos like Yahoo have made it to the top.  The OSS world is still building convenient, stable tools to work with?
<flacoste> or any custom solution built on top of selenium, (like google does)
<rockstar> mars, Yahoo still struggles with it.
<rockstar> mars, only small groups test at Yahoo.
<lifeless> hudson has selenium-grid glue I think
<lifeless> FWIW
<flacoste> mars, rockstar: go read Bjorn's report from last year Google Testing conference
<mars> rockstar, I remember the Google "If you try it, don't go here, or there" slides :)
<rockstar> lifeless, yeah, YUITest has hudson glue as well.
<mars> flacoste, ah, I remember now
<flacoste> poolie: i'm free for the next 1h45 if you want to talk, but i'm fine with resuming next week if you rather not
<mars> rockstar, only small groups test at Yahoo??
<flacoste> mars, rockstar: iirc, google was working on webc or something like that would should be more reliable than selenium for piloting the browser, but not sure where this is at now
<mars> rockstar, so the industry standard is still manual testing?
<poolie> flacoste: now's good for me; we haven't spoken for a while
<lifeless> mars: industry standard is probably user complaints.
<lifeless> mars: depending on how you define industry, and standard.
<mars> lifeless, haha
<rockstar> flacoste, are you talking about jstestdriver?  That's what Google uses to test gmail.
<mars> lifeless, we are working at the bleeding, gory edge of the whole thing
<flacoste> rockstar: i don't think so, it was also in Bjorn's report
<rockstar> mars, yeah, I was surprised too.  I had lunch with the Yahoo Sports team, and they just farm it out to their QA team.
<mars> flacoste, are you talking about that project that became Selenium2?
<mars> rockstar, could just be institutional momentum
<mars> Remember what a shock it was to start on LP and find how far ahead the practice here is
<mars> rockstar, could also be ease of implementation: perhaps everyone in the industry knows how to do manual JS+QA.  Few know automated testing.
<flacoste> mars: it might be
<mars> So, do we have some options here?
<rockstar> mars, yeah.  U1 makes sure that the barrier for rolling out a fix in cases like that is very small.
<mars> rockstar, right, and that helps a lot, I'm sure.
<jderose> rockstar: MooTools seems to be one of the more test-driven JS libs, and they're using Windmill: http://www.getwindmill.com/  <= you guys ever try it (i haven't)
<mars> "cap deploy"
<rockstar> jderose, you cant' see the backchat where I say terrible things about the mothers of the windmill developers.
<jderose> hehe, okay, that answers my question
<mars> jderose, the Launchpad project has a decent sized windmill test suite, all Python, integrated as python unittests.
<rockstar> jderose, launchpad uses windmill, and no one believes me when I say we shouldn't.  :)
<jderose> rockstar: so what should you use?
 * mars notices they redid the windmill homepage
<lifeless> sinzui: do we support safari?
<lifeless> sinzui: - https://bugs.launchpad.net/launchpad-web/+bug/97266
<_mup_> Bug #97266: Suggestions menu next to "Choose..." doesn't do anything <launchpad-web:Triaged> <https://launchpad.net/bugs/97266>
<sinzui> yes, in the form of webkit engine that underlies safari, konqueror, and chromium
<lifeless> sinzui: do you think that that bug is high ? I'm torn.
<sinzui> it is not high because the work around is to type what you see in the list for item 1
<mars> lifeless, "Safari 1.3"?
<sinzui> that is broken in chromium now
<lifeless> sinzui: it is ?
<sinzui> yes
<lifeless> I think I'll leave it high and add confusing-ui
<rockstar> jderose, that's still up for discussion.
<lifeless> we're making chromium default in Ubuntu I think, aren't we?
<lifeless> mars: EOldBug
<sinzui> yep, still broken
<mars> lifeless, if Curtis didn't say it affects current Chromium, I would say it is not a supported browser.  We only support Safari 3+ now.
<lifeless> hmm
<lifeless> is https://bugs.launchpad.net/malone/+bug/86352 still relevant
<_mup_> Bug #86352: SchoolTool imported bugs have invalid reporters <oops> <tech-debt> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/86352>
<lifeless> we've changed our email handling, haven't we?
<mars> lifeless, old, but here was the plan, back when I played webmaster: https://dev.launchpad.net/GradedBrowserSupport
<sinzui> mars, please listen to what I said. This is a webkit issue. I just tested the bug (again) and it is still broken in chromium Maverick
<sinzui> It is broken in Kubuntu maverick
 * sinzui tags the bug so it is with its friends
<mars> sinzui, I understand.  Unfortunately a GBS table like the one I wrote doesn't take root-level engine bugs into account well.
<mars> It assumes you can't easily discern a bug in the engine vs. a bug in the browser's integration of said engine
<sinzui> I tag most of the bugs in launchpad-web  with the browser engine when it is known
<sinzui> opera is another class of victim
<mars> sinzui, +1, that is a big step forward then
<sinzui> I was pondering how to tad IE issues last night. I think the versions with undocumented changes make it difficult to say if a  issue is ie, 6, 7, or 8. I believe we have one bug that is definitely IEs problem, but it is only fixed in version 8
<lifeless> sinzui: 'usersthathatefreedom'?
<sinzui> The inline help iframe sitting invisible over the login button is a nightmare. I think the only fix is to re-implement the the feature with ajax instead of an iframe
<sinzui> does anyone who uses opera have a moment to check two presentation issues
<lifeless> ok
<lifeless> boarding
<lifeless> ciao
<weather15> Hello Everyone
<weather15> After configuring Apache to access Launchpad from a remote machine I get this error: Secure Connection Failed                                                                      An error occurred during a connection to launchpad.dev.  SSL received a record that exceeded the maximum permissible length.  (Error code: ssl_error_rx_record_too_long)
<weather15> My Apache config looks right any idea as to what's wrong?
<mwhudson> that usually means that you're not talking ssl when something is expecting you to i think
<mwhudson> not really a launchpad issue
<weather15> is this something know that the instructions in the documentation do not work?
<weather15> *known
<weather15> Everything here is as said in the documentation
<weather15> I would think this would work! <VirtualHost 10.0.0.7:443>
<weather15> ??
<weather15> wgrant are you outthere?
<weather15> *out there
<weather15> It seems that Mod-ssl is not loading
<weather15> any idea as to why this does not work? It seems to that it has to do with Launchpad's Apache Config
<weather15> Looks to like Apache is not even running on port 443:tcp6       0      0 :::443                  :::*                    LISTEN
<EdwinGrubbs> leonardr: is there a specific IField that works for exporting a dict that can just be converted to json?
<leonardr> EdwinGrubbs: i don't know. i have no experience with named operations that return dicts. you might ask bdmurray
<rockstar> wallyworld_, hi
<wallyworld_> rockstar: otp
<wgrant> EdwinGrubbs: I don't think you declare it as anything.
<wgrant> EdwinGrubbs: It just works.
<rockstar> wallyworld_, okay. I was just pinging you to see how the war was going.
<maxb> It seems that leaving the code import machines deactivated for 2 days has left the import queue seriously backlogged
<wgrant> Hm, really?
<wgrant> Aren't most imports tried every 6 hours?
<maxb> Hmm, actually no, but something's screwed up
<maxb> The importds are full of imports, but new imports registered on Monday haven't had their initial import run yet
<wgrant> Hmm.
<wgrant> Something is up, yeah.
<wgrant> Lots of branches haven't imported since the 15th.
<rockstar> Wasn't there an incident where we disabled the importds?
<wgrant> Over the weekend, yeah.
<wgrant> But they were reenabled.
<maxb> https://code.launchpad.net/~vcs-imports/sope/debian-tfh is the earliest new import that has not been processed
<wgrant> A suspicious number are stuck like this:
<maxb> To put a time box on when the problem started
<wgrant> 2010-11-18 23:19:40 INFO    Getting exising bzr branch from central store.
<wgrant> 2010-11-18 23:19:40 INFO    [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
<wgrant> 2010-11-18 23:19:40 INFO    35 bytes transferred
<wgrant> Even some that started an hour ago.
<wgrant> eg https://code.launchpad.net/~rpm5/rpmlint/trunk has been going for 51 minutes, but normally takes 20 seconds.
<maxb> In fact, I wonder if there's a single successful import since the 15th
<maxb> The never used to log "[chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
<maxb> ", afaik, before
 * mwhudson looks at a graph
<wgrant> They seem to time out after an hour.
<wgrant> So we will be backlogged.
<wgrant> Yeah, they time out after almost exactly 60 minutes.
<wgrant> Oh, killed with KeyboardInterrupt.
<wallyworld_> rockstar: off the phone now. how's life in greener pastures?
<rockstar> wallyworld_, it's a nice evening in Buenos Aires.
<rockstar> wallyworld_, how goes merge queues?
<wallyworld_> rockstar: wow i didn't realise you were there. lucky you :-)
<wallyworld_> rockstar: i have a branch almost ready to land (for the product queues etc). but it's been on hold most of the week due to other tasks
<wallyworld_> rockstar: i landed the person listing branch yestersay into devel
<rockstar> wallyworld_, awesome.
<rockstar> wallyworld_, so I guess that means you haven't pulled out the json constraint on the configs?
<wallyworld_> rockstar: no :-( i have a few items left on my todo list. should get to it monday or tuesday. is that too late. i can do something over the w/e if you need it
<rockstar> wallyworld_, okay.  I'll see if I can get to it this weekend, in any case.  I'm also going to work on getting the merge proposals queueable.
<mwhudson> uh yes, the codeimport graphs is terrible
<wallyworld_> rockstar: i would think that make the mp queueable would take priority?
<wallyworld_> rockstar: the json cleanup can happen after that?
<rockstar> wallyworld_, yeah, but it's small.
<wallyworld_> rockstar: that's a sad problem for you to have but what does it have to do with json? :-)
<rockstar> :)
<wallyworld_> rockstar: i'll ping you then before i start on anything and we'll take a view from there
<rockstar> wallyworld_, cool.
<rockstar> wallyworld_, jml went over the LEP, might be good for you to read it.
<mwhudson> hang on
<wallyworld_> rockstar: will do
<mwhudson> why is bzr on the importds using the paramiko  transport
<maxb> IIUC, the implication from the various logs on https://code.edge.launchpad.net/~vcs-imports/transaction/trunk suggests it was back when it was working too
#launchpad-dev 2010-11-19
<maxb> Is someone visual on the need to initialize natty-cat-lpbuildd?
<wgrant> It probably won't ever happen.
<wgrant> Since we're moving to run bzr-builder outside the chroot.
<wgrant> But we had to roll production back from that.
<wgrant> Probably because bzr likes to eat RAM.
<maxb> Hrm. Someone really should just copy maverick-cat-lpbuildd for now, so that natty recipe builds aren't unilaterally doomed until then
<wgrant> I suggested that.
<maxb> mwhudson: Are you on the case with the importd breakage, or should I send a heads-up email to launchpad-dev@ to increase visibility of the problem?
<mwhudson> maxb: a heads up email is probably a good idea
<mwhudson> i'm completely clueless and there's no losa around now
<mwhudson> hm, seems like there's a problem on staging too: https://code.staging.launchpad.net/~vcs-imports/pydoctor/trunk
<lifeless> ola
<wgrant> Afternoon.
<lifeless> anyone else seeing ': lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave.test_dispatchBuildToSlave ' fail in devel ?
<lifeless> ugh yes, we're in testfix
<wgrant> Yeah, broken here.
<lifeless> care to do a binary chop to find the rev?
<lifeless> Its after 11936
<wgrant> I presume 11938
<wgrant> Yeah.
<wgrant> It can't have been tested.
<wgrant> Is that the only failure?
<lifeless> lets not jump to conclusions
<lifeless> no
<lifeless> forwarding you the mail
<wgrant> lifeless: Those are some interesting failures.
<wgrant> (the webservice ones)
<lifeless> yes
<wgrant> But http://pastebin.ubuntu.com/534111/ fixes the real failure.
<wgrant> 11938 changed the log level and added a new message.
<lifeless> thumper: https://bugs.launchpad.net/launchpad-code/+bug/677290
<_mup_> Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>
<wgrant> lifeless: Ahh, the XML-RPC change. Of course.
<wgrant> lifeless: Also, bug #677270 raises a few issues:
<_mup_> Bug #677270: restricted librarian broken, content decoding error <Launchpad itself:New> <https://launchpad.net/bugs/677270>
<wgrant> 1) Some restricted files 404.
<lifeless> wgrant: my WAG about xmlrpc firewalling appears wrong.
<wgrant> 2) gzip encoding is set on the 404 page, when it's not gzipped
<wgrant> 3) It's not obvious to people that those URLs are meant to be private.
<lifeless> wgrant: well its up to them if they want to share it
<wgrant> lifeless: Everything else in LP requires a cookie.
<lifeless> wgrant: it was a design feature; we could make it one-time-use if we want
<wgrant> It's not obvious that the librarian URL is sufficient to grant access.
<wgrant> I don't think making them one-time tokens is a fantastic idea, but we should probably think about something.
<lifeless> wgrant: you have a valid issue, I don't know what to do right now, and a) importds and b) testfix and c) plane flights
<wgrant> lifeless: Of course.
<wgrant> (also, 11938 can't have been tested -- it was in the queue too soon after the review)
<lifeless> wgrant: 404's - no idea why, file bugs.
<lifeless> wgrant: gzip? zomg. file a patch.
<wgrant> lifeless: I suspect they're related. I can't see why it 404s in that case, and it only gzips in that case.
<lifeless> thumper: ping
<lifeless> bac: ping
<lifeless> abentley: perhaps you're still here?
<mwhudson> hmm, bzr upgraded to 2.2.1 on the -11 in devel
<lifeless> [work wise]
<mwhudson> what are the chances that got rolled out on the 15th?
<lifeless> mwhudson: have a look between 11887 and 11926
<abentley> lifeless: FSVO "here".
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: importd system failing to import | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> abentley: we've got a firefight on our hands, and I'm about to be offline for an extended period
<lifeless> abentley: I'm looking for a volunteer to get the incident report rolling, and handoff to someone in asia as soon as such a person can be located
<mwhudson> lifeless: if it turns out to be the bzr upgrade, i'll try to get a downgrade organized
<mwhudson> lifeless: i can do that
<lifeless> mwhudson: thanks
<lifeless> mwhudson: I haven't tweeted or anything
<abentley> mwhudson: thanks.
<mwhudson> i can do that too
<lifeless> mwhudson: you'll need to cowboy
<mwhudson> lifeless: if it's the bzr upgrade, that's easy
<lifeless> mwhudson: because https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html is out of date
<lifeless> mwhudson: we can't tell whats up qa wise.
<mwhudson> lifeless: fantastic
<lifeless> I think we need to do a manual check and nodowntime deploy of the revision that merged db-stablew
<lifeless> then it will work again.
<lifeless> mwhudson: also we're in test fix
<lifeless> mwhudson: per https://bugs.launchpad.net/launchpad-code/+bug/677290
<_mup_> Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>
<lifeless> mwhudson: so, I don't see any sane option except cowboy
<mwhudson> yeah
<mwhudson> at least the code import systems are isolated
<mwhudson> in that touching them isn't going to **** up anything else
<lifeless>  https://devpad.canonical.com/~lpqateam/oops-summaries/code-import-2010-11-18.html claims very few oopses
<mwhudson> well, very few imports are running
<mwhudson> because the average import time has increased by like 300
<lifeless> mwhudson: it only had 16
<lifeless> if every import is failing
<mwhudson> that is low
<lifeless> I'd still expect more than 16 attempts a day
<mwhudson> yeah, should be 4*2*24 or so
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1783CIW11
<lifeless> has room to improve
<lifeless> mwhudson: so, definitely a bug on why didn't see it
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      282 /   56  Person:+commentedbugs
<lifeless>      101 / 5112  Archive:+index
<lifeless>       83 /  256  BugTask:+index
<lifeless>       31 /  317  Distribution:+bugs
<lifeless>       28 /    2  LoginToken:+accountmerge
<lifeless>       24 /  241  POFile:+translate
<lifeless>       21 /   28  Person:+bugs
<lifeless>       16 /   14  DistroSeries:+queue
<lifeless>       11 /    0  BinaryPackageBuild:+retry
<lifeless>       10 /  403  Distribution:+bugtarget-portlet-bugfilters-stats
<wgrant> 13:27:02 < lifeless>       11 /    0  BinaryPackageBuild:+retry
<wgrant> That is disturbing.
<lifeless> I don't have the oops code sorry
<lifeless> wgrant: so are you seeing folk copying and pasting the restricted urls into public channels or some such ?
<wgrant> lifeless: Just the one bug so far.
<lifeless> experienced user? new user?
<wgrant> Not sure. Bug #677270
<_mup_> Bug #677270: restricted librarian broken, content decoding error <Launchpad itself:New> <https://launchpad.net/bugs/677270>
<wgrant> OEM.
<mwhudson> well i can reproduce locally
<wgrant> The importd thing?
<mwhudson> yesd
<mwhudson> yes, it's the 2.2.1 upgrade
<mwhudson> wth!
<mwhudson> it's hanging somewhere inside of SFTPTransport.__del__
<bac> hi lifeless, mwhudson: i just got the ping for this issue.  i can help later if you need to hand off
<mwhudson> bac: thanks, i will need to fairly shortly
<bac> mwhudson: fwiw, the links in the bug report download ok in webkit-browsers
<mwhudson> bac: that's not the issue i'm working on :-)
<bac> mwhudson: i'm in a meeting right now but will be around later
<mwhudson> hm, seems present in 2.3b1 too...
<bac> mwhudson: do you know who is working on the issue lifeless raised?  wgrant?
<mwhudson> bac: no
<lifeless> which issue?
<lifeless> bac: I've raised enough I don't know to which you refer
<bac> lifeless: the one you pinged me about.  the one requiring an incident report and a handoff to someone in asia.  sorry if i didn't comprehend the full scrollback.  just trying to get up to speed.
<lifeless> bac: mwhudson is currently handling it
<lifeless> bac: he will want to handoff to you within the hour
<mwhudson> oh sorry
<bac> ok
<wgrant> Also, someone needs to land that testfix.
<lifeless> bac: perhaps you could to that ^ too ?
<lifeless> if you have time
<bac> lifeless: i can in a bit when i get out of this meeting
<bac> wgrant: where is the testfix branch?
<maxb> Well that was bizarre. LP just temporarily decided I didn't have permission to view https://code.launchpad.net/+code-imports/+machines
<mwhudson> i bet a code import branch that was private could do that
<mwhudson> pretty sure we don't filter for visibility anywhere
<wgrant> bac: I don't have a branch. But there's a patch on https://bugs.launchpad.net/launchpad-code/+bug/677290 which I could turn into a branch if you don't want to.
<_mup_> Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>
<mwhudson> bac: i'm running away
<poolie> well done mwhudson !
<LPCIBot> Project devel build (229): STILL FAILING in 3 hr 41 min: https://hudson.wedontsleep.org/job/devel/229/
<LPCIBot> Launchpad Patch Queue Manager: [r=danilo][ui=none][bug=676657] Memory-limit recipe builds.
<lifeless> sinzui: https://bugs.launchpad.net/malone/+bug/244998 - looks like a candidate for bridging the gap to me
<_mup_> Bug #244998: "Also affects project" is inconsistent and obscure when in package context <confusing-ui> <motu> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/244998>
<lifeless> wgrant: is https://bugs.launchpad.net/soyuz/+bug/251685 happening still?
<_mup_> Bug #251685: PPA upload hangs with 1K to go <poppy> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/251685>
<wgrant> lifeless: I think so.
<wgrant> lifeless: We've certainly not knowingly fixed it (although it doesn't affect the SFTP version of poppy).
<poolie> is there a new styleguide-type requirement introduced by the 'no more globs' launchpad changes?
<poolie> just trying to work out how to resolve conflicts in import statements in the right way
<lifeless> whatever works, try not to re-export things where possible.
<wgrant> poolie: utilities/format-imports forces imports to match our style. But that's not changed directly by the anti-glob policy.
<lifeless> not that its evil in principle, but the export * from foo idiom is pretty unmaintainable.,
<poolie> sure, i just saw it now imports many things from different places
<poolie> i'll just tweak it if they fail
<poolie> has anyone seen
<poolie> KeyError: 'STORM_CEXTENSIONS'
<lifeless> run 'make
<lifeless> '
<poolie> thanks, that was my guess
<lifeless> new storm eggs seem to be missing the C extensions or some such
<lifeless> it happens to me every storm upgrade we do
<bac> thanks wgrant.  i have to relocate and will look at that patch shortly
<wgrant> I think we need to do something about the footer.
<poolie> about its appearance?
<wgrant> The fact that it tells people to file bugs against Launchpad.
<lifeless> we're getting about a bug a day for Ubuntu
<wgrant> Sometimes several.
<lifeless> but OTOH we were getting that before anyway
<lifeless> its a little higher frequency but really not a Big Deal
<wgrant> The frequency increased immensely once the footer made it onto production.
<lifeless> wgrant: mmmm, I'd like stats for that actually
<poolie> "Please report any problems with Launchpad." has a highly ambiguous "with" :-)
<lifeless> yes
<lifeless> improvements appreciated
<lifeless> its in lp-production-configs
<poolie> "I have a persistent self of ennui"
<poolie> seems like that could merge with "contact support" too
<wgrant> Ah hm.
<wgrant> That link bypasses all the warnings.
<poolie> although for that matter if someone lands on an ubuntu bug page, "contact support" is an obvious way to ask about problems with ubuntu
<poolie> which one?
<poolie> oh wow, so much fail
<poolie> Please describe the bug in a few words, for example, "weather applet crashes on logout":
<poolie> it's _really_ inviting people to file ubuntu bugs, isn't it?
<wgrant> https://launchpad.net/launchpad-project has a bit of a warning, but it's not terribly obvious.
<wgrant> There is also a warning at the bottom of the page once you've entered the summary and description.
<wgrant> But that's it;.
<poolie> i thought there was a way to put text onto the filebug page?
<poolie> conveniently below the fold on my browser
<wgrant> There is.
<wgrant> Yeah.
<wgrant> For this case it probably needs to be in a yellow alert above the initial dupesearch.
<poolie> and anyhow, once people have entered their text, they're fairly invested in continuing
<wgrant> Although mpt will probably throttle me for suggesting that.
<wgrant> Yeah.
<wgrant> And you have to read a lot of text to see the warning.
<wgrant> If only we had a designer.
<poolie> a little bird told me 'soon'
<wgrant> lifeless: The problem with AJAX file uploads is that they're impossible.
<wgrant> You have no choice but to post a form in an iframe.
<lifeless> is that what gmail does?
<wgrant> I don't know... it's been quite a few years since I used it.
<wgrant> But probably.
<poolie> i think they prefer to use a flash component
 * wgrant vomits.
<poolie> otherwise i think it's blocking but imbw
<wgrant> I wish bug listings could be convinced to show a set of tags.
<lifeless> ciao
<poolie> sayonara
<poolie> jml, dkim tests are running, we'll see what happens
<poolie> wgrant: shall we file a bug for the footer problems?
<poolie> there might be an mpt bug already
<wgrant> poolie: It was restricted to edge until its demise.
<wgrant> So there's probably not an existing bug.
<poolie> really? the whole footer?
<poolie> or just the invitation to file bugs
<wgrant> Just the invitation.
<poolie> i'll file one then
<wgrant> Thanks.
<poolie> ok bug 677342 and bug 377339
<wallyworld_> poolie: you still there?
<poolie> hi, i am
<poolie> wallyworld_: i am
<wallyworld_> poolie: i had to go out for a few hours - belinda's car broke down in the middle of a car park :-(, had to get tow truck etc, but i think i saw that the lp bzr 2.2.1 upgrade has broken something?
<poolie> that sucks about the car
<wallyworld_> yeah :-( will be expensive
<poolie> was that the old one or the new one?
<wallyworld_> the new(er) one
<poolie> anyhow, apparently gc of an sftp socket causes a hang
<poolie> i tihnk it's now resolvede
<wallyworld_> ok. and that behaviour came about due to the upgrade?
<wallyworld_> they rolled back to 2.2.0?
<wgrant> Right, the importds are back on 2.2.0.
<wgrant> And it's probably already fixed in the 2.2 branch.
<wallyworld_> just the importds?
<poolie> so it should stay fixed when they go to 2.2.2
<wallyworld_> ok. so 2.2.2 should be released around nowish?
<wallyworld_> so i should upgrade lp to that?
<wallyworld_> are there any tests i should have run apart from the test_bzr* ones when i did the lp upgrade to 2.2.1?
<poolie> indeed, it should be pretty soon
<poolie> not that i know of
<poolie> could be worth finding the particular bug
<wallyworld_> yep. i'm not entirely across the importds stuff. i just want to not see the same thing happen next time
<poolie> iwbn if we could upgrade just one importd first and see how it behaves
<wallyworld_> in the code review for the 2.1.1 upgrade, i listed the bzr tests i ran
<poolie> but, perhaps it's not so bad, it's not immediately user critical if they have trouble
<poolie> don't all the tests get run?
<wgrant> LP doesn't run the bzr tests, but presumably the bzr tests ran before 2.2.1 was released.
<wallyworld_> yes, when it lands via ec2 which it did. but i also specifically ran a few by hand first
<wallyworld_> wgrant: yes, i ran the ones i found, something like test -vvt test_bzr*
<wallyworld_> wgrant: there were the ones i mentioned in the mp comments. i was hopeing that if there were any others i would be told to run them
<wallyworld_> s/there/these/
<poolie> jtv: hi?
<jtv> hi
<poolie> hi, i'm just looking at your parallel build stuff
<poolie> it's great it's done
<poolie> some of the use of phony targets is a bit strange though
<poolie> i think we can do even better
<poolie> in particular i think we shouldn't depend on the phony target buildout_bin, but rather on $(BUILDOUT_BIN), the actual files
<poolie> or perhaps on specific files, like $(PY)
<poolie> were the $(PY) dependencies not catching everything we needed?
<jtv> Hehâ¦ what you're saying is pretty much the way it was.  :)
<poolie> ok, and it didn't work?
<jtv> The old way broke with parallel builds.
<jtv> There were two problems:
<poolie> yes, i know
<poolie> i was trying to fix some of them myself
<jtv> It'd be fine AFAIC to depend on $(PY) instead of on buildout_bin where appropriate.
<jtv> (I really just got annoyed with the different ${PY}/$(PY) spellings)
<poolie> :)
<jtv> So if you want to make that change, go ahead.  I don't think it'll change anything in the effects, since depending on $(PY) will still require the buildout_bin target to be built.
<jtv> (Which is the way it needs to be in order to avoid breakage)
<jtv> Yes, definitely a lot of room for improvement still and nice to hear you're interested.
<poolie> the problem with depending on a phony target is
<poolie> everything that depends on it must always be rebuilt
<jtv> True.
<poolie> since it does not exist on disk, make can't know it's up to date
<jtv> I missed that.
<poolie> secondly, when you put that on the left hand side, and you do "rm $@", make tries to delete buildout_bin
<poolie> and doesn't delete the files it previously was deleting
<jtv> Whoopsie!
<poolie> :) teamwork :)
<jtv> Actually I took out the $(RM) at one point.
<poolie> i just worked out why that's there (i think)
<poolie> which is that buildout tries to be smart and not touch the file if the contents would be the same
<poolie> unfortunately then make always rebuilds it
<jtv> Gah.
<jtv> Wouldn't a "touch" have sufficed then!?
<poolie> i think so
<poolie> i'm kind of inferring it
<jtv> I have a feeling that this would solve another problem.
<jtv> I did run into an unnecessary rebuild at one point when I tried "make -j3 default schema" (with the don't-build-everything-for-schema patch).  Made one of the targets fail.
<poolie> it might
<jtv> That makefile needs some love.
<poolie> getting more precise makefile rules is generally good
<wgrant> buildout!!!
<jtv> poolie: â¦and the stupid thing bounced again.  Did I miss anything?
<poolie> not sure
<poolie> https://code.launchpad.net/~mbp/launchpad/mbp-trivial/+merge/41268
<poolie> are my suggestions
<poolie> let me know what you think?
<wgrant> "It would be great if we could also avoid the fairly slow js build
<wgrant> happening every time..."
<wgrant> Why should the JS build take more than a second?
<poolie> i don't know
<poolie> for me it does, doesn't it for you?
<wgrant> It does, yeah.
<poolie> it's well over 10s on my laptop; i really notice it
<wgrant> I just think we should make it less insane rather than reducing the frequency at which we call it.
<wgrant> There's no reason for it to take a non-trivial amount of time!
<poolie> is it just catting the files, or is it doing complicated optimization?
<wgrant> It's some lazr.js voodoo.
<wgrant> I don't know what lazr.js is meant to do.
<poolie> thus my shallow inclination to call it less
<wgrant> Or just delete it like U1 did.
<poolie> could be good
<poolie> ok, good night
<wgrant> Night.
<adeuring> good morning
<henninge> Moin adeuring ;)
<adeuring> hi henninge!
<bac> hi adeuring
<adeuring> hi bac
<mrevell> Guten morgen
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (230): FIXED in 4 hr 2 min: https://hudson.wedontsleep.org/job/devel/230/
<bigjools> $ time make -j2
<bigjools> real    1m38.038s
<bigjools> \o/  thanks jtv
<jtv> bigjools: my pleasure... poolie got in on the game too, so expect more improvements soon
<bigjools> jtv: I can't work out why -j4 takes longer
<jtv> bigjools: otp, will explain in a bit
<bigjools> no prob
<bigjools> I'm trying to make the webservice pick up my interface in the new brave world of post-jml interfaces apocalypse, and failing miserably.
<bigjools> does this look ok? http://pastebin.ubuntu.com/534239/
<jtv> bigjools: back.  About -j4: of course 2 CPUs don't necessarily give you a 2Ã speedup, because the real bottleneck is often memory.  That much is obvious.
<bigjools> jtv: right, I suspect it's disk-bound
<bigjools> will try on my SSD in a sec
 * jtv won't be any help to bigjools about that pastebin, and wait for someone else
<jtv> bigjools: I don't think it's I/Oâthen -j4 would help you cover the latency and so be faster, assuming you have plenty of memory.
<jtv> But because of that, running multiple threads/processes at the same time will slow down each of them individually.
<bigjools> jtv: if it's thrashing the disk head it'll be slower
<jtv> Of course.
<jtv> But I don't think the footprint will be large enough to do that.
<bigjools> I've seen the same problems when running parallel test suites by virtue of VMs
<jtv> I haven't kept up with I/O schedulers; sure there may be a bit of extra seeking.
<bigjools> make -j2 on a fresh branch bails out with bin/py missing
<jtv> Dependencies still aren't entirely complete, I guess.  Martin found a missing one earlier today.
<jtv> In any case, as per my email, there's currently not much theoretical gain left beyond -j2 because the wadl script is the bottleneck.
<jtv> This is one of those nasty things in parallelization: if you distribute n+1 tasks across n CPUs, what's your ideal speedup?
<jtv> The answer is n/2.
<jtv> Which is very disappointing.
<bigjools> :(
<jtv> In the theoretical ideal case, n jobs finish nÃ faster than with 1 CPU, and then n-1 CPUs sit around twiddling thumbs while the lucky one does the one remaining job, still at the same speed.
<bigjools> since we write three wadls, won't -j3 help?  or is something else at play
<jtv> We'd have to break up the script.  One nasty thing there is we'd pay 3Ã the ZCML overhead.
<bigjools> urgh
<jtv> In parallel, ideally, but that's no good to whoever isn't joining the SMP party.
<jtv> I think ZCML is really costing us quite a lot.
<bigjools> both my boxes have 4 cores.  It makes me weep to see them idle :)
<jtv> You'll be interested to know that, because of the slowdown effect I mentioned earlier, AMD is actually dropping SMT.
<jtv> Because multiple threads will still slow down a single thread, and by and large people simply have enough cores now.
<bigjools> reading *all* the zcml is costing us a lot
<jtv> Do we ever do less?
<bigjools> we read it all in every startup.  It's crazy.  We talked about moving it to registering via code on the ML some time ago
<jtv> I haven't looked much further but I get the impression that, with my next makefile tweak, "make schema" is basically 45 seconds of "spend several seconds processing zcmlâ¦ do something very quickly, done.  Start new script: spend several seconds processing zcml... work flash done.  Start new script:" etc.
<jtv> I got the last question at James Clark's XML presentation last month.  I asked him to swear that neither he, nor any of the other team members to his knowledge were taking money from CPU or memory vendors. :)
<jtv> bigjools: I'm still not getting any problems with "make -j2" on a fresh branch.  Are you sure you were using the updated Makefile and that you'd run the link-sourcecode script first?
<bigjools> jtv: I * thought* I was... but it seems ok now.  Ho hum.
<jtv> Well that's the problem with debugging parallel builds.  Everything goes Heisenberg.
<bigjools> :)
<bigjools> now I am surprised.  My i5 with SSD is slower than the AMD Phenom
<jtv> And the second time you run it?
<bigjools> doing so now
<jtv> I mean, again on a clean branch
<bigjools> it failed
<jtv> but with the good stuff in fscache
<jtv> failed!?
<bigjools> error about import gpgme ... !
<bigjools> ran again and it worked
<bigjools> yes, it's Heisenberg
<jtv> That's a missing dependency.
<jtv> I get the impression that some of the targets' dependencies were written from a mental model of "build this, then this, then that."
<jtv> And some of them are definitely missing.
<bigjools> quicker 2nd time, still 3 seconds slower than the old Phenom
<jtv> Is it a dual-core?
<bigjools> 4 core
<bigjools> well, the i5 is a fake 4 core
<jtv> Part of the fun of optimizing CPUs for this sort of thing is that the applicability of your benchmarks matters hugely.
<bigjools> it's got the old hyperthreading guff
<jtv> Vendors like AMD and Intel have their internal benchmarks.
<bigjools> random gpgme import fail again
<jtv> From the wadl script?
<jtv> Did you run a "make clean" between builds?
<bigjools> yes
 * jtv tries frantically to reproduce the problem
<bigjools> it's failing quite often
<bigjools> if I run again without a clean it works and continue
<bigjools> s
<jtv> Though I went through that "make clean ; make -j2" routine so often that I would have seen it by now.
<bigjools> I'll paste in a bit
<jtv> I've seen a failure like that, but with "make -j3 default schema" and the lightweight "make schema."
<jtv> I guess it's just too system-specific or something.
<jtv> bigjools: just for my piece of mind, could you grep ^buildout_bin Makefile ?
<bigjools> jtv: it's on my other machine so not easy to paste, were you expecting output?
<jtv> Yes.
<bigjools> it's a fresh devel pulled down 20 mins ago
<jtv> Then it should have the updated makefile, definitely.
<jtv> It may just be the problem Martin spotted, where the virtual buildout_bin target gets re-built unnecessarily.
<bigjools> jtv: http://pastebin.ubuntu.com/534245/
<jtv> bigjools: where did you get that verbose output?
<bigjools> jtv: the command I used is at the top of the paste
<jtv> I don't see that subvertpy build output though.
<deryck> Morning, all.
<deryck> adeuring, hey, did you see lifeless' email on the librarian?  Do you have time today to get that landed for him?
<jml> bigjools: looks right to me
<bigjools> jml: that's what I thought too - but the stuff is not in the wadl :/
<jml> bigjools: hmm.
<jml> bigjools: debugging this stuff is a pain too.
<bigjools> yes
<bigjools> I've put deliberate typos in the zcml and the interface to see if they get read, and they do
<bigjools> webservice.py I mean
<jml> bigjools: do you actually have the webservice interface decorators in builder.py?
<bigjools> jml: yes, my branch worked fine until your changes landed
<jml> bigjools: yay inventory :)
<jml> bigjools: umm... hmm... thinking.
<bigjools> I've moved off to something else until leonard shows up :)
<jml> bigjools: not a bad idea.  I'll background it while I go off and pay for the drinks I forgot to pay for last night.
<bigjools> lol
<bigjools> left the CC behind the bar?
<jml> not even that. they had table service and when we were done we just waltzed out not even thinking that we had to pay.
<jml> anyway, off. back soon enough.
<bigjools> and, dine and dash
<bigjools> jml: ahhh make -j3 is fastest yet on my i5
<bigjools> jtv, sorry
<jtv> bigjools: that's unexpectedâ¦ I have no idea how CPU governors will play into this btw.
<jtv> Say you have 4 cores and 2 build processes, and maybe postgres and some other daemon that both wake up to do work from time to time.
<jtv> Say each tends to stick to one core.
<bigjools> 1m42s fwiw
<jtv> Not bad.
<jtv> Though slower than -j2, no?
<bigjools> better than the 5m it used to take!
<jtv> I get something like 1:32 on -j2
<jtv> Yes, that's what I did it for.  :-)
<bigjools> hmmm I was getting 1m53s on -j2
<jtv> Poor boy.
<jtv> Then I guess this is good, yes.  :)
 * bigjools will try and beat 1:32
<jtv> But it also means that the performance characteristics are very different in your setups.
<jtv> Mine are pretty consistent, and that may explain how a missing dependency could be "hidden" on my machine fairly consistently.
<jtv> Surprising how you don't seem to bottleneck on the wadl build.
<jtv> Are you sure the wadl build is completing?  Maybe benji's optimization landed already?
<adeuring> deryck: yes, I've seen his mail. but I haven't found yet much time to look at the test failures
<wgrant> bigjools: Can you access it if you ignore the WADL?
<bigjools> wgrant: que?
<wgrant> bigjools: What if you try to retrieve a builder through the API manually, not using the WADL at all. Does that work?
<bigjools> wgrant: I don't know how to do that
<wgrant> bigjools: Browse to https://launchpad.dev/api/devel/builders/bob
<bigjools> well, I probably do but I can't remember
<bigjools> jtv: -j2 is 2:12 !
<jtv> bigjools: why not the 1:53 you got before?
<bigjools> who knows
<jtv> bigjools: what does your load graph look like with -j2 or -j3?  Is there a lot of time where you've got only 1 CPU busy?
<bigjools> I'll check later - busy running tests now
<deryck> adeuring, ok.  Do you think you'll have some time today to look at it?
<bigjools> ah windmil, I hate you
<adeuring> deryck: well, hard to say. Right now, I'm looking a bit more, but if anybody wants a review... ;)
<deryck> adeuring, right.  I understand.  If you have time, great.  If not, np.
<bigjools> wgrant: yeah it works without the wadl
<wgrant> bigjools: Have you tried deleting the WADL to force its regeneration?
<wgrant> That's sometimes needed.
<bigjools> yes
<wgrant> :(
<bigjools> make clean until my fingers bleed
<wgrant> I'm not sure that's enough.
<bigjools> it is
<bigjools> it removes the whole apidoc dir
<wgrant> Hm, should be, yeah.
<bigjools> and I see it getting re-generated on the next make
<wgrant> :(
<wgrant> I've run into this before. But I cannot remember how I resolved it.
<bigjools> ISTR there needs to be a magic string in the interface somewhere
<bigjools> otherwise the wadl parser ignores it
<james_w> bigjools, what are you doing?
<james_w> trying to export a new interface?
<bigjools> james_w: I had something that worked until jml got rid of canonical.launchpad.interfaces, now I've added some zcml + a webservice.py in lp.buildmaster but it dunt werk
<bigjools> but yeah it's a new export of IBuilder
<wgrant> You're sure that ZCML file is actually included?
<bigjools> yes
<wgrant> buildmaster is a little incomplete.
<james_w> you added a webservice:register?
<bigjools> yep
<james_w> should work then :-)
<bigjools> yep :)
<bigjools> https://code.launchpad.net/~julian-edwards/launchpad/api-expose-builders/+merge/39379
<james_w> bigjools, what's the collection you export?
<bigjools>  /builders
<james_w> bigjools, and no builder stuff ends up in the wadl?
<bigjools> nada
<james_w> I can't see anything obvious, sorry
<bigjools> thanks for looking
<bigjools> I'm going to eat and hopefully Leonard will be around when I get back
<deryck> reboot.  brb.
<LPCIBot> Project devel build (231): FAILURE in 3 hr 40 min: https://hudson.wedontsleep.org/job/devel/231/
<LPCIBot> Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=677020] The advanced subscription features
<LPCIBot> are now flagged correctly on the StructuralSubscriptionView.
<abentley> jelmer, bigjools: ping
<jelmer> abentley: hey
<abentley> jelmer: can we chat about async uploads and source pagackge recipe builds?
<jml> bigjools: it occurs to me that leonardr won't be around today.
<jelmer> abentley: yes, sure. mumble?
<abentley> jelmer: sure.
<abentley> jelmer: https://bugs.launchpad.net/launchpad-code/+bug/676776
<_mup_> Bug #676776: Recipe build stuck with "Uploading build" status [UI] <confusing-ui> <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/676776>
<abentley> jelmer: http://pastebin.ubuntu.com/534287/
<bigjools> jml: bugger
<bigjools> in that case, hey benji!
<benji> hey bigjools, what's up?
<bigjools> benji: I've got an API problem, do you have a moment to help please?
<benji> sure
<bigjools> great, thanks.  Can you look at this branch https://code.launchpad.net/~julian-edwards/launchpad/api-expose-builders/+merge/39379
<bigjools> it exposes /builders on the API.  It used to work before the canonical.launchpad.interfaces removal
<bigjools> I added the extra zcml declaration and webservice.py in lp.buildmaster
<bigjools> but it still doesn't generate the wadl
<bigjools> I'm at a bit of a loss
<benji> looking
<benji> bigjools: I couldn't find a problem by inspecting the diff so I built your branch and it seems the WADL and HTML docs were built correctly; there's an entry for "builder" that includes the description "Builder instance represents a single builder slave machine..."
<bigjools> benji: what about /builders ?
<bigjools> that's the first line in the test and it fails
<bigjools> more to the point does xx-builders.txt work there :)
<benji> I'll run it...
<benji> in the mean time, try rebuilding the WADL like so: LPCONFIG=development bin/py ./utilities/create-lp-wadl-and-apidoc.py --force "lib/canonical/launchpad/apidoc/wadl-development-%(version)s.xml"
<bigjools> benji: trying that now - what are all those "Unknown entry URL:" things about?
<bigjools> I vaguely remember us having to do something about those in the past but then everyone stopped bothering
<bigjools> benji: and re-generating the wadl didn't help :(
<benji> I don't know the full details, but it's XSLT not being told to be quiet about a particular element or value.
<benji> darn
<bigjools> I have a feeling something was needed in lazr.restful
<bigjools> benji: another thing comes to mind, I have another vague recollection of having to put some magic string in the interface somewhere to make the wadl generator see a top-level collection
<benji> I'm still trying to run the xx-builders.txt test. (make schema, librarian kill error message)
<benji> bigjools: I got this error "AttributeError: 'Launchpad' object has no attribute 'builders'"
<bigjools> benji: that's it
<bigjools> "builders" is not in the wadl
<benji> ok, cool; digging deeper
<bigjools> thanks
<bigjools> abentley: loads of codehosting tests always fail locally for me (mercurial module missing for example), do we have missing dependencies in sourcecode or the dep packages?
<abentley> bigjools: I haven't heard of that problem before.
<bigjools> problems seem to follow me
<abentley> bigjools: mercurial is provided as an egg.
<bigjools> I see it in eggs
<abentley> bigjools: It's not bzr-hg?  That's provided in sourcecode.
<abentley> bigjools: Try "bin/py -c 'import mercurial'"
<bigjools> abentley: it's failing when doing "from mercurial import error as hg_errors" with "No module named mercurial"
<abentley> bigjools: And when you try it from bin/py, does it puke?
<bigjools> yes
<abentley> bigjools: Something is wrong with your mercurial egg, then.
<bigjools> it's the bzr-hg stuff in sourcecode that's importing it
<bigjools> seems so
<abentley> bigjools: You could torch your eggs directory and run "make".
<abentley> jelmer: It appears that upload failures cause two emails now: A "build-style" "failed to upload" message and an upload-style "rejected" message.
<bigjools> abentley: yeah :/
<jelmer> abentley: this is sprb failure emails specifically?
<abentley> jelmer: Yep.  Why would anyone care about binary builds :-D
<jelmer> abentley: with our analysis from earlier this afternoon in mind, that would make sense.
<benji> bigjools: here's the fix: http://pastebin.ubuntu.com/534311/
<bigjools> benji: argh!
<abentley> jelmer: perhaps processChangesFile is the wrong place for notification?
<bigjools> thanks... I feel stupid now
<jelmer> abentley: Hmm
<abentley> jelmer: It could raise exceptions, and then the caller could handle them appropriately according to the upload type, perhaps.
<bigjools> abentley: yeah trashing the eggs worked
<bigjools> thanks
<jelmer> abentley: The special casing in PackageUpload.notify() is what's biting us there, too.
<deryck> rockstar, ping a ring a ling
<abentley> bigjools: I'm glad.
<rockstar> deryck, what's going gangster?
<deryck> rockstar, you know, livin' large, kickin' js ass and takin' names.
<rockstar> deryck, straight.
<deryck> rockstar, so question about your work on this branch initially--
<rockstar> deryck, shoot
<deryck> rockstar, did you link all yui modules in base-layout-marcro, or make some determination about which we actually used?
<rockstar> deryck, I only added the ones that were complaining on the pages that I looked at.
<deryck> ah, crap.  I worried about that.
<deryck> rockstar, so this could be the smallest number we can get away with?
<rockstar> deryck, yeah, although it might be bringing in things it may not care about anymore as well.
<rockstar> deryck, that's a long shot though.
<deryck> rockstar, since we don't do dynamic loading, how can it bring in something?  Requirements should be a straight chain, no?
<bigjools> benji: I think I'm going to file a bug about that.  Silently failing is wrong.
<benji> it's probably a good idea to file a bug, but I suspect the silence is by design; you can enable different parts of a web service using ZCML, so how can it know when you intentionally don't want to expose a service and when you do?
<jml> also, export in __all__ so it's obvious that it is being imported to be re-exported
<jml> rather than for some kind of side effect or by accident
<rockstar> deryck, well, I meant "maybe we're manually bringing in a file for a dependency that doesn't exist anymore in YUI 3.2"
<deryck> ah, right
<deryck> ok, that makes sense.  I'll just poke at it to see.
<bigjools> benji: right - at least I think it should warn about things declarated but not exported.
<bigjools> declarated?  I mean declared.
<benji> that might work
<rockstar> deryck, so we're basically concluding that it's the size of the js file that's causing all these headaches?
<deryck> rockstar, it is indeed the size of the file.  I'm 100% certain of that.
<rockstar> deryck, that's unfortunate.
<deryck> rockstar, yeah.  and yui without any lp js code is 1.1 of that 1.3 mb.  and getting it smaller is hard.  Just cut 250 files (around datatype) and got it to 900Mb.
<rockstar> deryck, that's odd.  U1 has yui AND the U1 code down to ~870K.
<rockstar> deryck, are you including lazr-js in the "yui" you speak of?
<deryck> rockstar, perhaps we're including too much then.
<deryck> rockstar, I thought lazr-js was compiled separately to lazr.js
<rockstar> deryck, no, not in launchpad it's not.
<rockstar> deryck, at least, I'm pretty sure it's not.
<deryck> ah, I see now.  It is but included in launchpad. js.  if I'm reading this right.  let me play some more....
<deryck> rockstar, yeah, so with our yui deps, I get 1020k.  That's no lazr.js and no code from the lp tree.  Using the files you linked in base-layout-macros.pt
<deryck> rockstar, perhaps we can trim some, but that's still a pretty ginormous file.
<mars> rockstar, do you remember what the exact source of the 512K JS file bug was?
<mars> socket code?  Gzip hard-coded limit?
<rockstar> mars, it was spinning on a socket call, yes.
<rockstar> mars, but we found that it was indeed a 512K limit, and I put code in there to halt when it's larger.
<rockstar> Unfortunately, I disabled that in the branch that is now deryck's because I thought "Oh, at this point, either windmill will have fixed the bug, or we'll have to disable windmill."
<mars> rockstar, do you have any pointers to where the discussion may have been?  #tarmac, a LP bug, #launchpad-dev?
<rockstar> mars, I believe the discussion was on the mailing list.
<mars> rockstar, ok, thanks, that is somewhere to start
<abentley> jelmer: in that bug, the user said he received an email, not two.  I wonder if the failure happened sending the second email?
<jelmer> abentley: that would certainly explain a lot.
<jelmer> abentley: since it's the second email that was attempting to access the 'builder' table.
<abentley> Launchpad Development Channel: Pulling (mirrors, imports) now fixed. | Week 4 of 10.11 | PQM open for 10.12 | firefighting: importd system failing to import | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<abentley> Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: pulling (mirrors, imports) now fixed. | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<sinzui> deryck, ping
<deryck> hi sinzui
<sinzui> deryck, I am looking at a css issue with this page: https://bugs.launchpad.net/launchpad
<sinzui> The rounded border around the form is only used on this page and only when a search was not performed
<sinzui> deryck, It is messing with font color. I need to fix it to put a branch in review, but I do not think this style should exist. Why does this form need an exceptional style?
 * sinzui should see if .portlet looks right
<mwhudson> has anyone tried to see if maxb's bzr fix helps the code import situation yet?
<deryck> sinzui, it doesn't need it's own style.  Was just a design element that was preserved.  We can drop the border if it gets you going.
<sinzui> deryck, using the portlet classes, the page might look like this: http://people.canonical.com/~curtis/upstream-bugs.png
<deryck> sinzui, +1.  Looks better to me actually.
<sinzui> deryck, my branch is near done, but I am dissatisfied with how I am leaving it
<deryck> sinzui, how so?
<sinzui> deryck, I expect the link the evolution in ubuntu to retry my search in ubuntu. It does not because or a common bug with this form. The search param are not added to the Advanced|Simple search links
<sinzui> I see there is a method that will encode the params for a URL. Deryck, to you know of a reason why we do not want to ensure the params are appended to these alternate search links
<deryck> sinzui, I see no reason not to append.  I assume it's just oversight that it doesn't currently.
<sinzui> I will start a second branch to see if I can fix this. this will also mean that changing the sort order will preserve the query
<sinzui> ah
<sinzui> I see now. We need to fix the refine this search/search again problem because we do not know if the user is starting a new search
<deryck> right, that makes sense
<sinzui> If this works, I will fix 3 or more bugs
<jml> g'night all.
<deryck> thanks for taking this on, sinzui.
<sinzui> np. I am sure I will benefit from the fix if I can land it
<LPCIBot> Project devel build (232): STILL FAILING in 3 hr 43 min: https://hudson.wedontsleep.org/job/devel/232/
<LPCIBot> * Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=666838] Export /builders and IBuilder on
<LPCIBot> the webservice.
<LPCIBot> * Launchpad Patch Queue Manager: [r=adeuring][ui=none][no-qa] Fix spurious test failure in
<LPCIBot> test_message_sharing_migration.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jcsackett,
<LPCIBot> sinzui][ui=sinzui][bug=667900] Add form to DistributionSourcePackage
<LPCIBot> +index page to set upstream project link more easily.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jcsackett,
<LPCIBot> sinzui][ui=none][bug=677077] Find merge proposal by revno
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=salgado,
<LPCIBot> sinzui][bug=670431] Suggest branch owner as recipe owner
<LPCIBot> * Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][bug=673590] Change the log level of
<LPCIBot> QueueInconsistentStateError during upload processing so that
<LPCIBot> the logger handler for cron scripts doesn't generate OOPSes.
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa] Factor out gathering of POFiles for
<LPCIBot> translations export to bzr branches.
<lifeless> morning
<lifeless> flacoste: hi
<jcsackett> i'm not sure how i've gotten this far without already encountering this issue, but if how does one set up a store on an object such that Store.of returns something other than None?
<lifeless> any object returned from a store
<lifeless> new objects need to be added to a store first
<jcsackett> lifeless: okay, so i must just have a logic error. i thought classes needed some sort of setup.
<jcsackett> lifeless: thanks!
<lifeless> Store.of(Class) ? IIRC that will return the default store
<lifeless> which can be a problem when something gets forced rather than defaulting
<wgrant> lifeless, jcsackett: Store.of(object) will only work if object has been retrieved is a model object from a store.
<wgrant> The IStore, IMasterStore, ISlaveStore objects are Launchpad-specific and work on classes.
<wgrant> s/objects/adapters/
 * lifeless wonders if his branches landed.
<wgrant> Can someone please harrass ISD about ixokai's issue (in #launchpad)?
#launchpad-dev 2010-11-20
<lifeless> wgrant: harassment rarely helps
<lifeless> asking nicely sometimes does ;)
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      472 /    0  LoginToken:+accountmerge
<lifeless>      165 /   58  Person:+commentedbugs
<lifeless>      150 / 5663  Archive:+index
<lifeless>       77 /  280  BugTask:+index
<lifeless>       22 /  333  Distribution:+bugs
<lifeless>       18 /   33  Archive:+copy-packages
<lifeless>       11 /   84  ProjectGroupSet:CollectionResource:#project_groups
<lifeless>        9 /    2  Person:+bugs
<lifeless>        8 /   22  DistroSeries:+queue
<wgrant> lifeless: Well, these tickets were filed weeks ago, ISD doesn't maintain a public presence, and this is becoming a pattern.
<lifeless>        6 /  195  POFile:+translate
<wgrant> Wow, someone tried to merge a bit.
<lifeless> wgrant: perhaps you'd like to raise this on lp-dev
<wgrant> It was raised a few weeks ago.
<wgrant> Perhaps I should raise it again.,
<wgrant> But that won't get to ISD, because ISD seem to like to pretend that they don't exist.
<lifeless> all 472 on one url
<wgrant> Ah, good.
<wgrant> So just one *very* persistent user.
<lifeless> wgrant: thats really not a good mindset to approach the discussion from
<wgrant> Probably not.
<wgrant> But then I'd argue that offloading a Launchpad-branded critical service to a fairly unresponsive team without a public presence isn't a good approach :)
<lifeless> the ops side of that team *is* the losas
<lifeless> they are pretty publicly available when needed
<wgrant> Hmm, so LOSAs handle the tickets?
<lifeless> I don't know the specific responsibilities - but ISD aren't ops; they might do user support analagous to our chr
<wgrant> Ah, I see.
<lifeless> wow
<lifeless> 1 17070.0 1 SQL-launchpad-main-master UPDATE TranslationMessage SET submitter=455815 WHERE submitter=3368841
<wgrant> Hmm.
<wgrant> Awful cache, or just rather prolific mergee?
<lifeless> https://bugs.launchpad.net/rosetta/+bug/677713 if you're interested
<_mup_> Bug #677713: LoginToken:+accountmerge with many translations <timeout> <Launchpad Translations:New> <https://launchpad.net/bugs/677713>
<lifeless> bizarre
<wgrant> Is this sense-ubuntu into sense?
<lifeless> yeah
<lifeless> 3000 rows in total
<lifeless> shouldn't have been an issue
<thumper> hi lifeless
<thumper> it's a shame you aren't here
<lifeless> hi thumper
<lifeless> how is it
<thumper> lifeless: I think it's pretty good
<thumper> lifeless: even though there aren't as many people as last year
<thumper> the seats are more comfortable
<thumper> although no couches
<nigelb> lifeless: Interesting notes about Cassandra :)
<LPCIBot> Project devel build (233): STILL FAILING in 3 hr 41 min: https://hudson.wedontsleep.org/job/devel/233/
<LPCIBot> * Launchpad Patch Queue Manager: [testfix][rs=deryck][ui=none][no-qa] Disable test_languages windmill
<LPCIBot> test, which fails spuriously, until we can work out what is happening.
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=669831] Export POFiles to bzr branch when
<LPCIBot> only a template is changed.
<lifeless> nigelb: thanks
<lifeless> thumper: have a good time there; say hi when appropriate for me ;)
<lifeless> gah. 677750
<lifeless> bug 677750 that is
<_mup_> Bug #677750: lib/canonical/launchpad/pagetests/webservice/xx-service.txt failing <Soyuz:New> <https://launchpad.net/bugs/677750>
<wgrant> .... sigh.
<LPCIBot> Project parallel-test build (13): STILL FAILING in 1 hr 46 min: https://hudson.wedontsleep.org/job/parallel-test/13/
<jml> what the hell is PQM actually doing?
<jml> "cleaning working directory" cannot take ten minutes, surely
<jml> twenty
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (234): FIXED in 4 hr 2 min: https://hudson.wedontsleep.org/job/devel/234/
<lifeless> jml: branching lp
<lifeless> jml: as I've said before
<lifeless> jml: thank you for fixing julians booboo
#launchpad-dev 2010-11-21
<lifeless> moin
<jml> lifeless: hi
<lifeless> hiya jml
<jml> lifeless: just polishing off the last of the testtools/twisted-related failures
<lifeless> \o/
<jml> lifeless: looks like iterating the reactor is necessary for many tests because of bugs in Twisted (either core or APIs)
<jml> lifeless: I'm thinking of giving AsynchronousDeferredRunTest some kind of option that controls whether we do that iterating
<lifeless> jml: don't measure, don't get. [e.g. thats kindof predictable]
<lifeless> jml: I noticed you deleted the mention of iteration from the docstring
<jml> lifeless: I actually had deleted iteration at one point, because I thought we could do without it. (I fixed the motivating use case at the root)
<lifeless> jml: of a helper function; I assume you had good reason but I wanted to mention that it feels kindof lacking - that its not very clear whats going on.
<jml> lifeless: but now I've restored it along with the docstring
<lifeless> jml: an option feels nice; then we can report on the failing code periodically
<jml> lifeless: anyway, I'm thinking maybe a 'strict' optional parameter to ADRT that defaults to True.
<jml> lifeless: more hassle now, fewer backwards compatibility problems later
<lifeless> I think thats swings and roundabouts
<lifeless> folk will learn to always put strict=False in
<lifeless> it will never be able to go away.
<jml> lifeless: good point.
<lifeless> perhaps a parallel class
<lifeless> or something like that
<jml> lifeless: suggested names?
<lifeless> ADRTIterating
<lifeless> ADRTBrokenTwisted
<lifeless> uhm
<jml> lifeless: what advantage does that have over an option?
<lifeless> jml: the non iterating codepath doesn't need conditionals
<jml> lifeless: hmm
 * jml gives it a shot
<jml> lifeless: btw, would appreciate your thoughts on https://bugs.launchpad.net/testtools/+bug/654474
<_mup_> Bug #654474: An unexpectedSuccess is like a failure not a success <regression> <selftest> <testtools> <Bazaar:Confirmed> <subunit:New> <testtools:In Progress by jml> <https://launchpad.net/bugs/654474>
<lifeless> they are shallow, but added
<lifeless> jml: I'd like us to do a release this week.
<lifeless> And get it up to debian unstable
<lifeless> is that reasonable?
<jml> lifeless: yes, but as mentioned before, I would like to get this Launchpad branch that actually uses the deferred stuff landed before we do a release.
<lifeless> jml: yes, I'm asking 'all things considered'
<jml> lifeless: however, that seems likely to happen early this week.
<jml> lifeless: and after that, a release would be very welcome.
<lifeless> ok, please let me know if that is likely to slip, so that I can put 0.9.5 up - maxb is keen to help bzr in debian build sanely
<lifeless> I've been thinking about the datamapping layer vs test parallelisation
<lifeless> I'm still of the opinion that test parallelisation is past the really hard bit
<lifeless> if I did nothing but it until the epic, it might be working by then
<lifeless> but I don't think so - latency is going to be a killer on getting it all figured out and sorted.
<lifeless> and I only have 3 work weeks between now and then
<jml> lifeless: *nod*
<jml> lifeless: I think parallelization needs someone to own it, coordinate, report on progress etc.
<jml> lifeless: that is if we actually want to make progress on it
<lifeless> So, balanced consideration - a data mapping layer, with test double - thats our most urgent infrastructural need with the largest payoff both short and long term.
<lifeless> jml: I've found a small bug in the canonical.lp.host glue in my databasefixture branch, else that would have landed.
<lifeless> and immediately after it my librarian branch.
<jml> lifeless: what's your plan w/ data mapping layer then?
<lifeless> jml: spend 9-10 days sketching and bootstrapping it
<lifeless> jml: talk a lot
<lifeless> jml: and at the epic recruit a task force for it
<jml> lifeless: ok.
<jml> lifeless: makes sense.
<lifeless> oh, and do plenty of literature searches
<lifeless> we *may* find something precanned that we can 'fix up'.
<jml> lifeless: sounds a little big & risky, but a risk worth taking.
<lifeless> jml: indeed
<jml> lifeless: any feelings on diverging from stdlib unittest behaviour re unexpected success?
<lifeless> I think we need to remove the 'please report bugs in launchpad' link from the footer.
<lifeless> jml: I don't think the stdlib unittest behaviour has had a lot of user feedback and polish
<lifeless> jml: remember that it was done in situ, not by merging an established implementation.
<jml> *nod*
<jml> lifeless: we need to have some way for users who find problems w/ Launchpad to report those problems.
<lifeless> jml: agreed
<lifeless> jml: we're getting primarily ubuntu bugs at the moment
<jml> lifeless: fsvo "primarily"
<lifeless> poolie filed a bug about the page we're directing them to and how it could help with the confusion
<lifeless> jml: its about equal to genuine Launchpad bugs.
<lifeless> jml: we're not dying, but none of those bug reports will get good treatment in Ubuntu because they aren't apport filed
<lifeless> and many should be questions in the first place.
<lifeless> s/q/Q/
<lifeless> wtf with bug 606296
<_mup_> Bug #606296: Fix codehosting acceptance tests to not use sample data <tech-debt> <openwns-app:Fix Released by www-martinez187> <https://launchpad.net/bugs/606296>
<jml> shrug
<lifeless> jml: new user confused by UI I guess
<lifeless> jml: the link in the footer used to be edge only
<lifeless> jml: I made them the same during the convergence part of unifying edge
<jml> lifeless: ahh right
<lifeless> jml: so I think rolling it back to what we had on prod before (nothing) will stop confusing users who are new to the environment.
<jml> lifeless: indeed.
<lifeless> jml: until we have a better UI for /launchpad/+filebug
<lifeless>  - at which point I'd like to bring the link back, because I really value user feedback
<lifeless> jml: you agree?
<jml> lifeless: yeah
<jml> lifeless: I also think we need to think more broadly about user feedback / beta stuff anyway
<lifeless> agreed
<jml> ok. that's all the tests running locally
<jml> not sure how the ForBrokenTwisted experiment will pan out with actual use, but worth a shot
<lifeless> well, how does the code look ?
<lifeless> and how will we/users know when they can use the unbroken one?
<jml> lifeless: they'll use the unbroken one by default. the trick is how do you know that the unclean warning you get is from your buggy tests or twisted's buggy APIs
<lifeless> would it be feasible to whitelist?
<jml> lifeless: I don't think so.
<lifeless> jml: by twisted release?
<lifeless> jml: I mean, release a new testtools in a few weeks when twisted is fixed
<jml> lifeless: sorry, I don't understand what would actually be on the white list.
<lifeless> jml: versions of twisted that are not broken.
<jml> lifeless: oh right. but you can use it with current Twisted for many things
<jml> lifeless: anyway, I'm doing a full test run now. mildly optimistic.
 * jml off.
<lifeless> gl
<jml> lifeless: failures. looks like I'm going to have to practically rewrite the distributionmirror_prober tests.
<lifeless> jml: can we land what you do have working ?
<jml> lifeless: yeah, we could actually. Wouldn't be able to delete the old layers, of course
<jml> lifeless: funny thing is, the failures in the distributionmirror tests only showed up in completely different tests. I wouldn't have known it was distributionmirror unless I had accidentally left DelayedCall debugging enabled
<jml> lifeless: I'd like to poke a little to see if I can get the errors better lined up with the tests that caused them.
<lifeless> that would be nice
<lifeless> Time Out Counts by Page ID
<lifeless> Hard	Soft	Page ID
<lifeless> 3555	0	LoginToken:+accountmerge
<lifeless> 266	54	Person:+commentedbugs
<lifeless> 203	6295	Archive:+index
<lifeless> 77	280	BugTask:+index
<lifeless> 12	93	ProjectGroupSet:CollectionResource:#project_groups
<lifeless> 10	245	Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless> 10	26	DistroSeries:+queue
<lifeless> 10	9	Person:+bugs
<lifeless> 8	1	BinaryPackageBuild:+retry
<lifeless> 6	38	Milestone:+index
<lifeless> 3553 https://launchpad.net/token/.../+accountmerge
<lifeless> one url
<jml> will get to both of those later, if I can.
<lifeless> wgrant: do you know where sense Hofstede hangs out on IRC?
<jml> bed for now
<lifeless> jml: gnight
<mwhudson> lifeless: he's 'sense' i think
<lifeless> I think he has merge on autopost with wget ;)
<mwhudson> hm?
<mwhudson> oh
<lifeless> mwhudson: https://devpad.canonical.com/~lpqateam/lpnet-oops.html - 3555 oopses, one url ;)
<mwhudson> ah
<mwhudson> that's a bit rude
#launchpad-dev 2011-11-14
<StevenK> lifeless: I guess we can look forward to drowning in scriptactivity warnings about oops-prune when your branch gets deployed, then. :-(
<lifeless> why? we will delete its entreis
<StevenK> mwhudson: My change only means bin/ec2 won't use AMIs created by you -- that's all.
<mwhudson> StevenK: oh right
<mwhudson> StevenK: sorry for the noise
<lifeless> this is perhaps a little to eager, as poolie just added himself
<lifeless> I certainly think non-staff need to be removed
<StevenK> mwhudson: It's perfectly fine. :-)
<StevenK> lifeless: Well, to be fair, I JFDI'd, rather than communicated about it.
<lifeless> I think thats fine
<lifeless> just suggesting perhaps it needs a tweak
<poolie> StevenK, i was only suggesting to remove non staff
<poolie> anyhow
<poolie> ConfigurationConflictError: Conflicting configuration actions
<poolie>   For: ('protectName', <class 'lp.services.salesforce.proxy.Voucher'>, u'__str__')
<poolie>     File "/home/mbp/launchpad/lp-branches/work/configs/development/salesforce-configure-normal.zcml", line 16.4-18.4
<poolie> is this my fault?
<wgrant> What did you change?
<poolie> https://code.launchpad.net/~mbp/launchpad/678090-affected-count/+merge/81108
<poolie> i exposed a new method on bug
<poolie> nothing to do with sf
<wgrant> Your indentation is buggy, but agreed.
<poolie> where?
<wgrant> 295 of the diff
<wgrant> You have a two-space indent.
<poolie> so does the code i copied :)
<poolie> but i will fix it
<wgrant> So it does :(
<poolie> am i allowed to put blank lines around those declarations?
<poolie> i think it would be more readable
<wgrant> VWS is discouraged but not forbidden, unless lifeless sees you doing it.
<poolie> i thought it was required between class members?
<poolie> between _methods_
<poolie> according to pep8
<wgrant> Between methods, sure.
<wgrant> That's not discretionary VWS.
<wgrant> Discretionary VWS is strongly discouraged.
<poolie> i'll leave it then
<poolie> indents are fixed
<poolie> could you read the diff for me?
<wgrant> What produces that error?
<wgrant> Starting a dev appserver? Starting the test suite?
<poolie> running the tests
<poolie> um
<poolie> http://pastebin.ubuntu.com/737824/
<wgrant> Is that the right branch?
<wgrant> It's an odd name.
<poolie> https://code.launchpad.net/~mbp/launchpad/678090-affected-count/+merge/81108
<poolie> is the branch in which that error occurred
<wgrant> Right, but it's in a directory named 'work', which suggests it's either old or wrong or polluted with bad ZCML or config symlinks or something like that.
<poolie> it's a colo workspace
<poolie> i have just one tree
<wgrant> Ah
<lifeless> huwshimi: I'm glad to see you writing up UI patterns; I'm a bit surprised that we don't have such things already
<huwshimi> lifeless: Yeah, it's something I've been meaning to do for ages :)
<lifeless> heh, I know that feeling :)
<rick_h> huwshimi: link?
<huwshimi> rick_h: https://dev.launchpad.net/UI/Patterns
<huwshimi> rick_h: It's not much more than a stub at the moment
<rick_h> cool, I was looking for something like that the other day so very cool
<rick_h> I think it was http://www.amazon.com/Designing-Web-Interfaces-Principles-Interactions/dp/0596516258/ref=sr_1_10?ie=UTF8&qid=1321231747&sr=8-10 that had some good things about creating documented sets of ui "modules" and describing interactions and such
<huwshimi> I certainly would like to come up with a pattern for how to write that kind of documentation too.
<rick_h> sec, checking if I got it on kindle, pdf, etc
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<lifeless> who is OCR today ?
<lifeless> I want the bot to know this
<StevenK> stub
<lifeless> stub
<lifeless> yeah
<StevenK> lifeless: And how does the bot deal with sick days/days off, etc? :-)
<poolie> hi huw, i'm happy to see that too
<poolie> huwshimi, i do wonder a bit how much of that is describing the current state vs endorsing it
<poolie> eg green links
<poolie> wallyworld_, could you reread https://code.launchpad.net/~mbp/launchpad/678090-affected-count/+merge/81108 for me some time?
 * wallyworld_ looks
<wallyworld_> poolie: looks like a nice change. are feature flags reset after each test? i normally use "with FeatureFixture ". perhaps also the flag name could be declared as a constant
<wallyworld_> poolie: the tests all seem to set the fflag. are there any tests without the fflag?
<poolie> there are thousands of tests without the flag :)
<poolie> some of them covering this
<poolie> i don't have any tests that assert what the default flag value is
<poolie> not sure that would be worthwhile
<mwhudson> "are tests reset after each test" -- yes
<poolie> flags too
<mwhudson> if not, i'm coming after you with a rusty pickaxe :-)
<mwhudson> ah yes
<poolie> that would be insane if not
<mwhudson> indeed
<poolie> wallyworld_, re making it a constant
<poolie> making it a constant pointing to a string would feel kinda half baked i think
<poolie> perhaps the features should change to be really registered as objects not tuples
<poolie> then the code can mention the object
<poolie> which would make it harder for them to be orphaned
<poolie> i guess for out of process lookups and maybe tal we would still use just a symbolic name
<poolie> could do that separately though i think
<wallyworld_> poolie: i was just talking about the tests
<wallyworld_> poolie: i think i got confused. the flag seems to be to used to revert to the old behaviour? ie by default, without the flag, dupes are counted, which is the new behaviour?
<poolie> yes
<wallyworld_> hence wouldn't all the 1000's of  existing tests need to be changed?
<wallyworld_> to count differently
<poolie> this is a 'get out of jail' that restores the old less useful behaviour if it turns out this is too expensive
<poolie> wallyworld_, there were no previous tests about counting affected users across dupes
<wallyworld_> yes,  but my point is that i would have expected to see the flag used in the "1000's of existing tests" to make them work properly
<poolie> none of the affectsmetoo tests create dupes
<wallyworld_> ah ok
<poolie> aside from the ones i adedd
<wallyworld_> so, back to my original question :-) shouldn't some of the new tests added be with and withou tthe fflag?
<poolie> but the other tests reassure me that my change hasn't broken the bugs page when you file dupes, look at dupes anonymously, etc
<poolie> oh, like i should copy eg test_other_users_affected_count and do it with and without the flag?
<poolie> maybe i should
<wallyworld_> yes, otherwise you are not testing the new functionality
<poolie> sorry no eg test_counts_affected_by_master
<poolie> mm
<poolie> most of the tests do try the new behaviour
<poolie> they explicitly set the flag to '' ie false, ie 'not disabled'
<wallyworld_> ah, i misread that. because normally setting the fflag to *anything* equates to true, my eyes misread the tests
<wallyworld_> i missed the ''
<wallyworld_> whenever i see a fflag set, my mind assumes it is being set to a value which evaluates to true
<poolie> :/
<wallyworld_> poolie: so do you really need to set the flag to ''?
<poolie> i want to also do a followon which makes them properly typed
<poolie> ie if its declared bool, you get a real bool
<lifeless> mmm
<lifeless> I am skeptical
<poolie> wallyworld_, i just thought it was more explicit to say "this is the behaviour i expect when this is enabled"
<lifeless> I think that will make them more fragile.
<wallyworld_> poolie: what i mean is, i the default is false, why have effectively a noop at the start of the test?
<poolie> explicitude
<wallyworld_> had the opposite effect on me :-)
<poolie> lifeless, one concrete probelm is that they suffer from bool('false') == True
<wallyworld_> but it was my misreading
<lifeless> poolie: OTOH they can't raise TypeError
<wallyworld_> poolie: fwiw, most of the other tests i see and also write myself do not set the fflag at the start of the test if the default is required
<lifeless> poolie: I'd be happy with the editor page being taught that a bool entry has a 0 length string to indicate False
<poolie> you're saying you don't want bad input data to cause a typeerror at the point the flag is used?
<poolie> it will be less fragile if it's just given some arbitrary interpretation, as it is at the moment?
<lifeless> amongst other concerns yes; I think there is a good leanness is having it be None-or-a-string
<lifeless> its a clear common substrate
<lifeless> No matter what crap gets into the table, that substrate won't fail
<wallyworld_> poolie: i've already had this argument with lifeless :-)
<lifeless> if you have a type system, you, by definition need to handle date entered with one type being read out with a different type
<lifeless> bah, my commas are in terrible places there
<poolie> lifeless, i'm pretty sure there's already some code that does cast-to-bool and some that does =='on'
<poolie> this seems poor
<lifeless> I'm happy - delighted even - for the editing UI - to have such complexity in it, and I'm happy for helpers to work with what the substrate returns.
<poolie> i agree with being robust about invalid stuff in the table
<mwhudson> is there some magic to installing launchpad-dependencies on oneiric?
<mwhudson> i get launchpad-dependencies : Depends: libsvn-dev but it is not going to be installed
<poolie> you need ppa:bzr too
<poolie> and maybe universe?
<mwhudson> ah, i'll try with the bzr ppa
<lifeless> poolie: My constraints are:
<poolie> no it's in main
<mwhudson> i have universe enabled
<lifeless>  - the substrate Must Not Fail
<lifeless>  - Ever
<lifeless>  - single pages can die, thats tolerable
<mwhudson> E: Unable to correct problems, you have held broken packages.
<lifeless>  - the editing page -should- never fail, but we could tolerate a failure there
<mwhudson> thanks apt, how about telling me which
<wgrant> mwhudson: sudoa pt-get install libsvn-dev?
<poolie> sure
<wgrant> You have to drill down manually.
<wgrant> But it'll eventually tell you.;
<poolie> thanks for being clear about the constraint
<poolie> i agree with that
<mwhudson> wgrant: a seemingly endless excercise in dependency chasing
<wgrant> mwhudson: Yes
<mwhudson> isn't sorting that out what apt is for? :)
<lifeless>  - the type system has to be clearly a hint - the underlying data -is- untyped and I don't want folk to forget that
<lifeless>  - fin
<poolie> i guess you can try aptitude
<poolie> sure
<lifeless> If you want to design something up that works within those constraints, I'll be happy to be a sounding board and/or review the design for you.
<poolie> not today
<poolie> i will take you up on it later
<lifeless> Personally, I wouldn't do it. I'd write helpers that work on the substrate rather than changing the substrate.
<poolie> hm
<mwhudson> wgrant: ah
<mwhudson>  krb5-multidev : Depends: libkrb5-3 (= 1.9.1+dfsg-1ubuntu1.1) but 1.9.1+dfsg-1ubuntu2.1 is to be installed
<poolie> i was just going to add a function get_flag_whatever(name)
<mwhudson> which is a bit strange
<poolie> does the casting to the type declared for that flag
 * mwhudson disables -proposed
<poolie> this is what most callers want
<poolie> it can squelch errors if we want
<StevenK> mwhudson: arch-indep has been published before arch-dep
<poolie> i'm not planning to change the db or anything
<poolie> separately, in the editor, things that are bools, which seems to be the majority, can be shown as such
<mwhudson> StevenK: ah, so the archive/my mirror is broken?
<StevenK> I'd check your mirror first
<StevenK> But the pulse may have happened between publications as it were
<wgrant> StevenK, mwhudson: Except that that's not meant to happen any more.
<StevenK> There is that, too
<wgrant> That's why the publisher takes 90 minutes.
<StevenK> Perhaps it doesn't work over pockets
<wgrant> That shouldn't matter, since apt should see the right version in some other pocket.
<wgrant> And pick that.
<wgrant> Hopefully.
<mwhudson> nz.archive.ubuntu.com does not have a record of working 100% of the time
<StevenK> Ah, citylink
<StevenK> Heh heh
<lifeless> mwhudson: the US mirror works pefectly :P
<poolie> if you add the real archive at the end of sources.list
<poolie> it will read from your nearby mirror if possible and otherwise fall back
<ajmitch> StevenK: you say that like it's a bad thing :P
<StevenK> ajmitch: But it is? :-P
<mwhudson> bah
<mwhudson> disabled -proposed and switching to archive.ubuntu.com and still the same nonsense
<lifeless> mwhudson: 'lxc-create'
<wgrant> mwhudson: You haven't held krb5-multidev or something?
<wgrant> mwhudson: apt-get install krb5-multidev
<wgrant> What does it whine about?
<mwhudson>  krb5-multidev : Depends: libkrb5-3 (= 1.9.1+dfsg-1ubuntu1.1) but 1.9.1+dfsg-1ubuntu2.1 is to be installed
<mwhudson> wgrant: as before
<wgrant> apt-cache policy krb5-multidev
<mwhudson> i do have some pinning, but no holding afaict
<mwhudson> wgrant: http://paste.ubuntu.com/737890/
<wgrant> Well then.
<wgrant> Ah.
<wgrant> Pinned.
<wgrant> Why is security pinned over updates?
<wgrant> They're naturally 500.
<mwhudson> ah
<mwhudson> because i followed some stupid instructions i suspect
<mwhudson> to whit: https://wiki.ubuntu.com/Testing/EnableProposed
<wgrant> The first sign that something was up should have been the pre-Lucid screenshot.
<wgrant> That's never a good sign.
<StevenK> mwhudson: 'narsil' ?
<mwhudson> StevenK: lotr/tolkien references in machine names seemed like a good idea a decade ago
<wgrant> mwhudson: So, pin proposed below 500 if you want, but I'd recommend not pinning up updates/security.
<mwhudson> wgrant: yeah, i deleted the !proposed sections
<wgrant> Although I just run with -proposed unpinned. It's hardly worse than running a dev series...
<wgrant> Great.
<poolie> likewise
<wgrant> That reminds me, I must upgrade to precise soon.
<lifeless> right, tomorrow I write the untestable launchpadlib glue, and then we have fast oops pruning done.
<lifeless> one more tick in the box.
<huwshimi> ok, tal hackers, in this paste: http://paste.ubuntu.com/737935/ the function "view/style_for_add_milestone" returns a style, instead I want it to return a class. I have modified the function, but I'm not sure how to modify the tal to return the class when there are already classes on the <a>.
<poolie> huwshimi, i would guess you need to define it as a python expression that returns the whole string
<poolie> eg putting the other classes into style_for_add_milestone
<poolie> wallyworld_, so what was your overall verdict?
<mwhudson> huwshimi: you can say things like tal:attributes="class string:bish bash bosh ${view/other_classes}"
<mwhudson> (if that's what you're asking for, not really sure)
<poolie> oh right
<wallyworld_> poolie: looks ok to me, my misreading of the fflag aside. my personal preference is not to set the flag if the test is expecting the default behaviour
<poolie> thanks
<poolie> i have mixed feelings about the test
<poolie> i think i will leave it for now
<wallyworld_> ok. the actual business logic looks ok i think
<huwshimi> mwhudson: Thank, that worked perfectly
<huwshimi> *thanks
<mwhudson> huwshimi: np
<poolie> StevenK, if nothing falls over after that ec2 landing i will update the image later in the week
<huwshimi> A pretty trivial CSS review if someone's keen: https://code.launchpad.net/~huwshimi/launchpad/style-removal-one/+merge/82096
<wgrant> lol bigpond
<wgrant> Afternoon bigjools.
<bigjools> eyup
<bigjools> crappiest internets evar
<StevenK> Welcome to Australia
<bigjools> except when I was in Brazil and the connection went every time it rained
<StevenK> And it's bigpong, so it's even worse.
<wgrant> s/Australia/Telstra/
<bigjools> hey, it's free with the apartment
<StevenK> There is a reason for that.
 * bigjools looks for slab of meat to throw on bbq
<StevenK> You could always visit wallyworld_ and suffer through using TPG.
<bigjools> I was at his place last night
<StevenK> Brisbane is only 31, disappointing
<StevenK> It hit 38 here today
<wgrant> In November? I doubt that.
<wgrant> Except you're right.
<wgrant> Wow
<wgrant> WTF
<bigjools> not that I'd know inside the airconned car and airconned shopping mall
<StevenK> Don't doubt me, I'm bloody melting
<bigjools> is this all you guys talk about before EU wakes up? :)
<StevenK> Not usually
<bigjools> 38 is not pleasant
<StevenK> And no air-con here
<wgrant> O_o
 * wgrant submits a new definition of "unwise" to the dictionary.
<bigjools> doesn't your equipment go bleargh?
<StevenK> Nope
<bigjools> wgrant: s/unwise/tight fisted/  :)
<StevenK> bigjools: Rented apartment, I have no choice
<bigjools> bugger
<wallyworld_> StevenK: tpg is good :-P
<StevenK> s/\(good\)/\1 for nothing/
<wallyworld_> have you ever used it?
<StevenK> I have used both their dial-up and broadband service, yes
<StevenK> But I have not been a customer of theirs
<wallyworld_> for me it's fast and cheap
<wallyworld_> never have drop outs
<StevenK> Wait until you call support
<StevenK> Heh heh
<wallyworld_> why would i wan tto do that?
<wallyworld_> never had a need to in over 4 years
<StevenK> Just because you never have had to doesn't mean you won't need to.
<wallyworld_> use them for mobile too. heaps of data and calls for not much cash
<wgrant> $ grep wallyworld.*tpg.*Ping\ timeout irclogs/Freenode/#launchpad-dev.log | wc -l
<wgrant> 272
<wgrant> QED
<StevenK> Hahahaha
<wallyworld_> why should i care about that latency? i don't play games
<StevenK> wgrant: Pointless use of wc -l :-P
 * StevenK waits for "Silence!"
<wgrant> Latency?
<wgrant> I didn't see anyone talk about latency...
<StevenK> A ping timeout is not latancy
<StevenK> *latency
<wallyworld_> sorry, i misread
<wgrant> (well, unless >5min latency still counts as latency)
<StevenK> Haha
<wgrant> Which it probably doesn't, unless you're on Mars.
<wallyworld_> but won't it still timeout if i suspend the laptop to move rooms etc
<wallyworld_> so those figures are bogus
<wgrant> Ah, if you run your IRC client on a laptop, indeed.
<wallyworld_> i only use my laptop
<wallyworld_> i honestly have no issue with tpg
<wallyworld_> bigjools: found a slab of meat yet?
<wgrant> people I know who used to use them got terrible speeds and dropouts.
<bigjools> wallyworld_: oh yes
<StevenK> And some beer?
<bigjools> Boags
<wallyworld_> wgrant: my speed is around 10Mbps and I'm a fair way from the exchange
<bigjools> and some sparking rose
<StevenK> Is it room temperature beer? :-P
<wallyworld_> hahah
<bigjools> StevenK: who tf drinks that!
<bigjools> cellar temperature is more ideal
<StevenK> The British, if you believe the jokes
<wgrant> wallyworld_: Not bad.
<wallyworld_> i guess i'm one of the lucky ones
<bigjools> at least we have beer, rather than watery piss
<wallyworld_> you mean Fosters? you lot own that now
<poolie> 38C beer would certainly be unpleasant
<bigjools> I am proud to say I've never drank Fosters
<poolie> StevenK, i bought a Vornado fan last year when my ac was broken
<bigjools> lager.... <shudder>
<poolie> it is very quiet and compact
<bigjools> http://lifehacker.com/181510/make-your-own-air-conditioner
<poolie> mm
<poolie> switching to using only my laptop helped tooo
<StevenK> bigjools: Except the link in that article is a 404
<bigjools> StevenK: ha!
<poolie> also "How to Work from Home Without Going Insane"
 * StevenK stabs lg.com
<bigjools> wallyworld_: I was up at Chapel Hills today
<bigjools> s/s$//
<wallyworld_> how'd it go?
<bigjools> great school
<bigjools> nice area
<bigjools> some lovely houses!
<StevenK> bigjools: s/s$// doesn't match anything :-P
<bigjools> StevenK: silence
<wallyworld_> bigjools: yes, it is a great area. Mt Cootha being so close is a bonus for sure too
<bigjools> wallyworld_: yeah, some great trails
 * wallyworld_ has to take the poor dog to the vet now to get his tail looked at
<bigjools> wallyworld_: oops
<bigjools> so, I buy a $19 recharge for a prepay SIM and end up with $50 credit. This is odd.
<wallyworld_> bigjools: we can't stand the blood being cast everywhere when he wags it. it looks like a scene from CSI
<StevenK> bigjools: Sounds right.
<wallyworld_> bigjools: that $50 is an inflated value
<bigjools> on the plus side, 1GB of data
<wallyworld_> you really need to look at how many minutes talk time it buys you
<wallyworld_> it's very confusing here in Oz
<bigjools> or how much data it gives me so I can use Skype :)
<wallyworld_> yep :-)
<StevenK> bigjools: Which provider?
 * wallyworld_ goes to round up the dog
<bigjools> StevenK: Virgin Mobile
<StevenK> Ah, neat.
<StevenK> There is a "Virgin Usage Meter" app on the market too
<bigjools> oh cool
<bigjools> well, the phone has built-in meters
<StevenK> That one screen-scrapes from Virgin's site just in case the two don't agree
<bigjools> StevenK: I just realised the humour in "Virgin Usage Meter"
<StevenK> Haha
<poolie> :)
<StevenK> Are you going to rate it down for not doing what the app title suggested? :-P
<wgrant> Hmm, now, the ultimate question.
<wgrant> Did he suspend his laptop, or was it Telstra...
<StevenK> Haha
<bigjools> Bigpong is up and down like a whore's knickers
<wgrant> Like I said.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<stub> Must suck when your cow-orkers in the third world have better 'Net than you do.
<stub> bigjools: You on holiday or is this a more permanent move?
<bigjools> stub: yes
<bigjools> stub: holiday for now...
<jtv> stub: don't forgetâcow-orkers _in disaster zones_ in the third world.
<nigelb> bigjools: Where are you? :)
<nigelb> i.e. where are you vacationing
<nigelb> jtv: cow-workers?
<bigjools> nigelb: Brisbane
<nigelb> ;)
<nigelb> bigjools: Neat. :)
<jtv> nigelb: yes, people who ork cows.
<nigelb> heh
<bigjools> I ate a cow last night
<nigelb> o.O
<nigelb> is that a figure of speech?
<bigjools> not at all - you should have seen the size of the steak I had
<jtv> nigelb: how so?  Julian ate a cow last night, and presumably used a nife and an ork.
<jtv> It all makes so much sense.
<nigelb> jtv: You are milking this for all its worth aren't ya? :D
<bigjools> jtv: BA-DUM-TISH!
<nigelb> bigjools: Gah, Now I want steak.
<jtv> nigelb: Ooh I see what you did there, with the cow theme!
<nigelb> This is turning into a cow-themed punland.
<stub> jtv: Well, I was thinking on New Zealand. But I guess it holds true here too ;)
<jtv> :)
<nigelb> I remember a joke sometime back to the likes of "Next time you take a vacation, go to an island with poor internet conectivity. But you already live in New Zealand"
<nigelb> jtv: We'll be in closer time-zomes next weekend!
<jtv> Oh, right!
<nigelb> I'm sad I can't make it to Bangkok yet :|
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba | Critical bugtasks: 277
<wgrant> stub: Thanks.
<wgrant> lifeless: Want to recheck https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81104?
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
<jtv> StevenK, wgrant: I'm trying to get back into bug 876594.  That unwanted rejection is generated by UploadHandler, right?
<_mup_> Bug #876594: rejected builds for synced packages send mail to Debian maintainer <derivation> <qa-untestable> <regression> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/876594 >
<mrevell> Morning
<jtv> hi mrevell
<nigelb> Morning mrevell
<mrevell> Hey there
<stub1> rvba: https://code.launchpad.net/~stub/launchpad/oops/+merge/82119
<rvba> stub: k
<mvo> https://dev.launchpad.net/PatchSubmission <- this page says that step #1 is to go to #launchpad-reviews, but that channel is apparently no longer in use - could someone update the wiki please (it looks like I don't have permissions to do so)
<rvba> mvo: sure, I'll do it.
<mvo> thanks!
<mvo> so I worked on lp:~mvo/launchpad/maintenance-check-precise to make sure that the "apt-cache show pkgname" "Supported: 18m" info is accurate for precise, this required some tweaks to the code. I wonder if someone could run the testsuite for me on this branch (or trigger it in ec2 or somesuch). I have a local LP VM, but its really outdated. my QA for the code was simply to ensure that for lucid it produces the same output as it did before
<rvba> mvo: Let me have a look.
<rvba> mvo: could you please write a MP explaining the rationale for this?
<mvo> rvba: sure, I will note in the MP that I have not run the testsuite locally yet
<rvba> mvo: cool.
<mvo> I put it here: https://code.launchpad.net/~mvo/launchpad/maintenance-check-precise/+merge/82125
<mvo> and have some lunch now
 * mvo waves
<mvo> I set it to WIP as long as no testsuite was run against it
<rvba> mvo: you can set the status as "Needs Review". We usually run the test suite (in ec2) after we've reviewing the mp.
<rvba> s/reviewing/reviewed/
<jelmer> lifeless: hi
<rick_h> morning
<jelmer> hi rick_h
<jtv> Buildbot just failed: hung twisted thread or something along those lines, in the branch puller.
<jelmer> fun :(
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba, benji | Critical bugtasks: 277
<rvba> Morning benji.
<benji> good morning rvba
<mvo> rvba: aha, thanks! I set it back to needs-review now
<deryck> Morning, all.
<rick_h_> howdy
<deryck> Hi, rick_h_.  Welcome to Launchpad!
<deryck> officially and all ;)
<rick_h_> thanks!
<deryck> rick_h_, do you have access to the Canonical wiki now?
<mrevell> Oh, hey rick_h_, is it your first day? Welcome!
<rick_h_> yes, been working through it this morning
<rick_h_> couple of glitches, but progressing
<deryck> rick_h_, can you point me at the doc you're working through?
<rick_h_> sure
<abentley> rick_h_: Good morning, and welcome to the squad.
<rick_h_> abentley: thanks
<jcsackett> morning, folks.
<deryck> Morning, jcsackett.
<rvba> benji: I'll review jtv's mega lint branchâ¦
<benji> rvba: sorry, I was AFK visiting the alter of coffee in my kitchen; yeah that sounds good
<jtv> rvba: thanks for picking it up
<nigelb> benji: wow, you have an altar for that? Neat.
<benji> :)
<flacoste> rick_h_: welcome to Launchpad!
<deryck> Does anyone know what's up with db-devel failed build?  buildnot is getting on my last nerve.
<jtv> allenap: looking at test_stalePidFile.  You've got a comment in there saying it's safe as long as pids don't wrap aroundâ¦ did you mean it's safe as long as pids don't get reused?
<rick_h_> thanks flacoste
<allenap> jtv: I'll take a look...
<allenap> jtv: Yes, but from what I could discover, Linux doesn't reuse pids until the counter wraps.
<jtv> allenap: upon consideration I think that's what's meant.  TBH I think the world would fall apart quickly if pids got re-assigned quickly.  :)
<allenap> jtv: Indeed! :)
<jtv> Call it our dirty little secret.
<jtv> Strange thing is, I'm suddenly getting a DeprecationWarning there instead of a UserWarning about deprecation.
<deryck> jelmer, hi.  Have you seen the db test failures?  Are these related to your work?
<jcsackett> sinzui: are you free to mumble?
<sinzui> yes
<jelmer> deryck: no, which failures?
<jelmer> deryck: I haven't landed anything in quite some time
<deryck> jelmer, flacoste thought it might have been your bzr-builder updates.  but now he suspects his own work. ;)
<deryck> jelmer, so un-ping.  Sorry. :)
<flacoste> deryck: i don't suspect my change at all
<flacoste> it is something bzr related
<deryck> flacoste, ah, sorry.  I misunderstood.
<flacoste> TestPullerMasterIntegration' object has no attribute '_lock_actions'
<flacoste> look likes an API change
<flacoste> could be poolie change also
<flacoste> that run upgrade before tests
<jelmer> hmm, I think I've seen that failure before as part of a spurious test failure
<jelmer> I'll have a look
<jelmer> deryck: do you know for how long this has been failing?
<jelmer> or is it just a one-off thing?
<deryck> jelmer, we've only had this one failure like this in the builder.
<flacoste> https://lpbuildbot.canonical.com/builders/lucid_db_lp
<flacoste> deryck: but the these tests always fail for you locally, no?
<sinzui> jcsackett, https://code.qastaging.launchpad.net/private-project-test/
<deryck> flacoste, I'm trying against current db-devel now to confirm.  I was on an old rev, before the last 4-6 revs, to see a clean state....
<deryck> flacoste, and was failing then.  but trying on current tip now.
<jelmer> flacoste: thanks
<jelmer> I'm pretty sure I've seen this as part of a spurious test failure before
<jelmer> not sure what the background of it is though, it seems pretty odd for a simple operation like that to take more than 20 seconds
<mrevell> Hey sinzui! On a project-group overview page, should all member projects be listed? If yes, is there a delay between adding a project to a group and it showing up?
<nigelb> the launchpad planet could use some love :)
<nigelb> Its a pain to look at now :)
<sinzui> mrevell, I think there is a 15-60 minute delay. The list is cached since it is prone to timeout
<mrevell> alo ^^^ does that help?
<mrevell> thanks sinzui
<flacoste> i still have this OperationalError: fe_sendauth: no password supplied
<flacoste> whenever i attempt to run DatabaseLayer tests
<deryck> flacoste, ok, so these tests do pass for me locally running latest db-devel.
<deryck> flacoste, should we force the build?
<flacoste> deryck: might as well
<flacoste> or disable the tests
<deryck> flacoste, ok.  thanks.  if it fails again, we can disable.  cool?
<flacoste> sure
<deryck> ok, thanks
<deryck> alright, build force.  land 'em if you got 'em.
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugtasks: 277
<flacoste> where are the deployment reports nowadays?
<flacoste> http://lpqateam.canonical.com/qa-reports/deployment-stable.html
<flacoste> rvba: how is the qa of rev 14289?
<rvba> flacoste: I'm on it.
<flacoste> rvba: great, thanks
<rvba> flacoste: qa-ok
<lifeless> good morning
<jtv> morning lifeless
 * jtv double-takes
<jtv> Hmmâ¦ morning?
<jtv> Stupid timezones.
<lifeless> :)
<Ronnie> what version of python-launchpadlib is advised. the 1.6.0 from the 10.04.3 repo or the 1.9.9 from pip ?
<lifeless> newest you have available is the general recommendation
<flacoste> lifeless: would you have any idea where to start debugging http://pastebin.ubuntu.com/738453/
<flacoste> lifeless: i suspect i missed a step in the pgbouncer transition
<lifeless> flacoste: local?
<flacoste> yes
<flacoste> i'm basically unable to run any tests invovling the DB
<lifeless> well, the fixture sets up phbouncer itself
<lifeless> but
<lifeless> thats not using pgbouncer, I don't think.
<lifeless> I suspect your pg install isn't setup correctly
<flacoste> or it might be the oneiric upgrade
<lifeless> indeed
<flacoste> i don't have anything else than LP in pg
<lifeless> what pg version are you runniung with ?
<lifeless> or yeah try the initial setup script again
<lifeless> launchpad-database-zergage
<flacoste> 8.4.8
<lifeless> huh
<lifeless> does psql launchpad_ftest_template work ?
<flacoste> launchpad-database-setup seems to have done the trick
<flacoste> the tests are now running
<flacoste> thanks
<benji> lifeless: I'm wondering why my DB branch (https://code.launchpad.net/~benji/launchpad/bug-877195-db) isn't showing up on http://lpqateam.canonical.com/qa-reports/deployment-db-stable.html and I suspect it's because I forgot to target the original MP to db-stable and resubmitted the MP after you and stub had approved the other and I haven't sought you to approve the new one.  Can you confirm that theory?
<lifeless> benji: mp's aren't needed for the deployment report
<lifeless> benji: if they are present they are used as an additional source of bug links, is all
<lifeless> benji: the deployment report depends on:
<lifeless>  - the instance running
<lifeless>  - the code being deployed to said instance
<lifeless> has your code reached db-stable, and is the running staging revno >= the revno your branch landed in?
<benji> lifeless: yes to having reached db-stable, and no to the second (https://staging.launchpad.net reports the revision before mine)
<benji> so, I assume that means I need to ask/wait for staging to be updated
<lifeless> yes
<lifeless> as your rev may be the reason its not deployed (if the patch fails)
<lifeless> the logs are on devpad to let you check this
<abentley> deryck: has adeuring's feature-flag-in-javascript stuff landed?
<adeuring> abentley: I received a positive PQM message at 1900 UTC
<adeuring> erm, 1800 UTC
<abentley> adeuring: Ah, well if it was today, I'll need to merge devel.
<adeuring> abentley: right, it landed just recently -- testfix hassles...
<deryck> abentley, what adeuring said ;)  Sorry was otp.
<abentley> adeuring: That adds a list of "beta_features" to the IJSONRequestCache?
<adeuring> abentley: this list was already been added, but the list is empty by default
<abentley> adeuring: I see.  I was hoping for a list of active feature flags, like that at the bottom of the page: Features: {'baselayout.careers_link.disabled': None, 'hard_timeout': None,
<adeuring> abentley: that's what you get
<abentley> adeuring: where do I get it?
<adeuring> but a list, not a dict
<adeuring> abentley: enable the feature flag and view any bug listing
<abentley> adeuring: So this feature of including feature flags itself is enabled by a feature flag?
<adeuring> abentley: well, to acutally see a non-empty list, you need my recent branch
<adeuring> abentley: no. Take my branch that landed two hours ago, and enable the LEP feature
<abentley> adeuring: And this shows all feature flags, unlike the empty list already present?
<adeuring> abentley: it is a bit more complicated:
<adeuring> 1. the view needs a non-empty property beta_features, as BugTaskSearchListingView
<adeuring> 2. you must enable this feature
<abentley> adeuring: So if beta_features is non-empty, it lists all active feature flags?
<adeuring> abentley: let's mjumble, that's easier
<benji> lifeless: since I can find no reference to my revision (11143) in the log (sourcherry/2011-11-11-staging_restore.log), I assume no attempt has been made to deploy it to staging; what is the best course of action to get it deployed?
<lifeless> benji: how long ago did it pass buildbot ?
<lifeless> benji: if < 1 hour, wait.
<lifeless> if > - check with los a
<benji> lifeless: it got buildbot love on Friday, so I'll talk to a losa about it
<benji> ooh, I found something lifeless: the log for deploying the revision (sourcherry/2011-11-12-staging_restore.log) is very unhappy but in a way that is meaningless to me.  The first (real) error seems to be this bit: https://pastebin.canonical.com/55752/
<deryck> abentley, adeuring -- I'm going to change work locations here in about 20 -30 minutes and will be offline for 30 minutes while I do.
<adeuring> ok
<abentley> deryck: ack
<rick_h_> benji: can you do me the fav of checking https://code.launchpad.net/~rharding/launchpad/bugfix_814696/+merge/82214
<rick_h_> and hand holding and all that new guy stuff :)
<benji> rick_h_: gladly
<abentley> deryck: http://people.canonical.com/~abentley/visible-information.png
<lifeless> benji: ahha
<lifeless> benji: slony 1 vs slony 2 restore, I bet
<deryck> abentley, oh wow.
<deryck> abentley, is that chrome or firefox?
<abentley> deryck: FF
<lifeless> benji: this is the first staging restore post slony 2 AFAIK
<abentley> deryck: It actually looks kinda cool, but...
<deryck> abentley, ok, ack.  I'll take a look here shortly.
<benji> lifeless: interesting
<lifeless> abentley: if that was deliberate, wow :)
<benji> rick_h_: looks good; we're not friendly to the tab characters though, so you should replace it with spaces (you can clean up the other tabs in that general area while you're at it if you want)
<rick_h_> ok, will do
<benji> rick_h_: I can't decide if that usb-only monitor on your blog is awesome or insane; possibly both
<rick_h_> benji: I'll bring it in Jan
<rick_h_> the insane part is now I'm thinking about getting another :)
<benji> heh
<benji> rick_h_: unfortunately I won't be there in January, fortunately I'll be distracted by having a new baby
<rick_h_> oh, congrats
<rick_h_> know how that goes
<rick_h_> ok, so I removed all the tabs, found one in another file and fixed that for the why not factor
<rick_h_> where would I go from here?
<rick_h_> while the diff is regen'ing
<benji> sounds good, just push the tweaks and you can mark the MP as approved
<benji> if there were more substantial changes you could ask for another review
<rick_h_> k
<rick_h_> you marked it as "Approve" so what am I marking as approved?
<abentley> adeuring: So it turns out that list is actually a list of feature flags that have been used, rather than all feature flags.
<abentley> adeuring: the idea of re-purposing your data is now more appealing.
<adeuring> abentley: even better :)
<benji> rick_h_: at the top
<rick_h_> ah, got it thanks
<adeuring> deryck: could you review this mp: https://code.launchpad.net/~adeuring/launchpad/bug-sorting/+merge/82217 ?
<deryck> adeuring, indeed, I can.
<adeuring> deryck: greatm thanks!
<deryck> adeuring, there's no diff on the mp.
<adeuring> oh...
<adeuring> deryck: gahh... did a "bzr push" before a "bzr commit". The current version of the branhc is not pushed to LP
<deryck> adeuring, ah, now I see a diff.
<deryck> adeuring, r=me
<adeuring> deryck: great, thanks!
<deryck> adeuring, I'm not crazy about that isinstance stuff, but I see that's the pattern in that bit.
<adeuring> deryck: do you mean the "isinstance(col, tuple)"? I know that this is duplicated, but to together iwth the "if orderby_col.startswith("-")" we need to handle for cases. Hard to not have some sort of duplication
<huwshimi> Morning
<adeuring> morning huwshimi
<deryck> morning huwshimi
<deryck> adeuring, it's not the duplication.  just doing type checks in general also bother me.
<deryck> s/also/always/
<adeuring> deryck: I understand ;) I could change the other entries in the mapping, making the isinstance() calls obsolete, but I'll postpone this to the next branch related to bug search sorting
<deryck> adeuring, ok, cool.  It wasn't a big deal to me since the surrounding code was like this; just noting it.
<deryck> adeuring, But I welcome later updates to clean this up if time allows.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<sinzui> StevenK, lp/soyuz/model/archive.py addArchiveDependency() looks promising and lp/bugs/model/bug.py addNomination() is helpful too. But about this last example, maybe an losa cannot fix a nomination
<wgrant> >>> urlparse.urljoin('https://api.qastaging.launchpad.net/', 'devel', 'foo/bar')
<wgrant> 'https://api.qastaging.launchpad.net/devel'
<wgrant> jcsackett: ^^ there's your problem
<flacoste> lifeless: do you have any reservation on turning on incremental diff on for everyon?
<flacoste> (see email thread)
<wgrant> flacoste: Incremental diffs, or the inline MP diffs?
<wgrant> They are different things for very similar purposes.
<flacoste> inlin MP diffs
<flacoste> which is displaying an incremental diff
<lifeless> flacoste: so, I know that loggerhead times out
<wgrant> Right, but there's the separate incremental MP diff feature.
<lifeless> its an inconsistent UI
<lifeless> s/UI/UX/
<lifeless> flacoste: its probably ok to turn on
<wgrant> It also doesn't have a spinner or error handling at all.
<lifeless> flacoste: fortunately I should be able to do timeout oopses for loggerhead in a few hours work \o/
<wgrant> And it's blue instead of green.;
<deryck> rick_h_ is fixing the green link bug as his first bug fix.
<deryck> hi wgrant and lifeless, btw :)
<thumper> hi deryck
<deryck> hi thumper
<lifeless> deryck: hi :)
<wgrant> Evening deryck.
<wgrant> lifeless: Could you have a final look over https://code.launchpad.net/~wgrant/launchpad/observer-db/+merge/81104 before I land it?
<lifeless> k
#launchpad-dev 2011-11-15
<huwshimi> wgrant, wallyworld: Did either of you follow the email thread from curtis about represting the security/privacy policies as something like tags? Are we essentially back to something like: http://people.canonical.com/~huwshimi/policy2.png ?
<wallyworld> huwshimi: not sure. that ui has issues also eg tags should be separately editable
<wallyworld> and it doesn't scale so well i don't think
<huwshimi> wallyworld: It won't scale because that list could become quite long?
<huwshimi> wallyworld: The list of policies, I mean
<wallyworld> huwshimi: yes
<huwshimi> hmmm
<huwshimi> wallyworld: Any suggestions about what might work?
<wallyworld> not really off hand. it's hard to concisely represent all the info isn't it
<wgrant> At present there is probably only going to be 1-4 policies.
<wgrant> I don't really have any suggestions :/
<wgrant> 'tis difficult.
<wallyworld> that's still going to be a lot of text
<wallyworld> perhaps the table cells could be a list of policies, one under the other
<huwshimi> wallyworld: Yeah, just what I was typing
<wallyworld> more vertical space per row, but nicer looking?
<wallyworld> and each could then have an edit icon next to it
<huwshimi> wallyworld: OK I'll mock that upo
<huwshimi> *up
<wallyworld> oko
<wallyworld> :-P
<rick_h_> if it's treated tag like, why not something like a tag control that builds a little ui around each so that it helps break them up/make them distinct?
<wallyworld> it's the "excessive" use or horizontal space i think that's one issue
<wallyworld> arranging vertically should help perhaps
<huwshimi> I guess the difficulty with the tag idea is that we have to display both the policy and the permission, and this makes it a more complicated construct
<rick_h_> is it possible to icon-ify one part?
<rick_h_> if there's only a couple permissions Restricted/Not restricted and such?
<huwshimi> rick_h_: That's a possibilty, one concern is that the permissions are fairly complicated to understand as they are...
<rick_h_> gotcha
<rick_h_> I know when I've got too much info I tend to try to color/icon part and text the other, or somehow break it apart
<rick_h_> especially if they can combine in different ways
<wallyworld> and i think we have a usability policy of avoiding icons where the user needs to guess or know their meaning?
<rick_h_> but I've not seen the permissions stuff yet
<rick_h_> yea, that's true, and doesn't look like there's a solid tooltip/helper function in place in the first couple of things I check out
<wallyworld> reliance on tooltips is also best avoided for accessibility too afaik
<wgrant> Not just accessibility, but anything that doesn't have a mouse.
<wgrant> eg. phones/tablets
<rick_h_> right
<huwshimi> wallyworld, wgrant: Hmmm... http://people.canonical.com/~huwshimi/policy6.png or http://people.canonical.com/~huwshimi/policy7.png or http://people.canonical.com/~huwshimi/policy8.png are all options, but I don't feel like I'm getting anywhere good.
<wgrant> I really don't know :/
<wallyworld> 6 is the best of those i think
<wallyworld> how often will people have more than one?
<timrc> Has there ever been discussion around the concept of upload/build hooks for PPAs?  I would like some way of telling a service downstream of Launchpad to take an action, rather than having it poll Launchpad
<timrc> I have no idea how this could be implemented in a safe manner, though
<lifeless> timrc: pubsubhubbub
<wgrant> It depends. For Ubuntu, most people will have at least two. For other projects, most people will probably have one.
<wallyworld> so i think vertical stacking will work ok
<timrc> lifeless, neat, reading about it now
<huwshimi> wallyworld wgrant: I can't really think of any other options right now, so I might propose these options for now unless there's anything else either of you can think of now?
<wallyworld> huwshimi: i can't, but i'm sure others will have strong opinions
<huwshimi> wallyworld: haha, I'm sure they will
<huwshimi> wallyworld: I'm sure there will be lots of vague "just putting it out there" opinions too.
<rick_h_> huwshimi: from that ui, does edit have controls to change or can you just remove the permission?
<timrc> lifeless, so if I understand this correctly, LP would need to provide the feed, is this something that you guys are working on or wanting to work on?  I imagine this gets a little more complicated in my case where I'm dealing with P3A's
<huwshimi> rick_h_: I believe the edit screen has an option remove in the same way the "assigned to" picker on a bug does
<wallyworld> lifeless: https://pastebin.canonical.com/55760/  do you have any clue why the evaluation of an attribute (person.name) would end up creating a different Request object when calling checkPermission and not use self.request from the view, causing the permission caching to fail?
<rick_h_> huwshimi: what do you use for mockups?
<huwshimi> rick_h_: Inkscape
<rick_h_> ah cool, was trying out a bunch of web/etc ones and didn't find one I was 100% happy with
<wgrant> wallyworld: IWebServiceClientRequest may well be killing some thread-local.
<wallyworld> wgrant: ok, i'll see if taking that away helps
<wgrant> Worth a try.
<huwshimi> rick_h_: Yeah, there are plenty of decent options out there, just need to find something you're comfortable with
<wallyworld> i then would need to figure out how to replace it
<wallyworld> wgrant: still broken sadly
<wgrant> wallyworld: That's upsetting.
<wallyworld> yep. nothing obvious that i can see
<wgrant> wallyworld: Oh
<wgrant> IndirectSubscribers?
<wgrant> ARe you sure that's what you want?
<wgrant> That's structural subscriptions, mostly.
<wallyworld> you mean do we want to print the names of private teams which are indirect subscribers?
<wgrant> Well, what are you testing?
<wallyworld> that the subscription portal showing direct and indirect subscribers does not render <hidden> for private teams
<wallyworld> whether direct or indirect
<wgrant> What suggests that the caching is failing?
<wgrant> Indirect subscriptions are shown <hidden>?
<wallyworld> i've traced the code execution
<wallyworld> the caching call add the permission to self.request
<wallyworld> the person.name eval ends up in the permission checking code and a totally different request is checked to see if the permission exists on that request
<wgrant> Ah]
<wallyworld> when i say request, i mean request instance
<wgrant> How are you calling the view?
<wgrant> In a browser?
<wallyworld> using LaunchpadFormHarness
<wallyworld> i copied an existing test
<wgrant> That may well not set the thread-local request. Do other permission caching tests do it that way?
<wallyworld> well it seems to be a wrapper around create_view and friends and tests using create_view directly work i think
<wallyworld> i'll have to double check
<wallyworld> the tests for precache_permission_for_objects() simply create a LaunchpadTestRequest and do a login
<wallyworld> i've got some other tests which call create_initialized_view
<wallyworld> so long as a pass the same request used when calling login() to the create view, it works
<wgrant> Right, that makes sense.
<wgrant> Semi-thread-locals ftl.
<wallyworld> makes sense. but doesn't explain why eval of person.name wants to create its own, different request instance
<wgrant> It should be using the thread-local.
<wgrant> Whereas precache_permission_for_objects is using the one given to the view.
<wgrant> They could well be different when the view is called.
<wgrant> Hmm.
<wgrant> bitbucket and github's privacy controls are somewhat more limited than I expected.
<wgrant> wallyworld, huwshimi: Have you looked at them at all?
<wallyworld> no
<wgrant> Issue privacy is on a project basis.
<wgrant> So either the project's issues are public, or they are private.
<wgrant> There's no equivalent of our restricted observer concept.
<wgrant> Everyone can see everything, or only nominated people can see everything. Nobody else can see anything at all.
<nigelb> wgrant: Github's privacy is a little bit clunky as well.
<wgrant> Groups are also namespaced within an account.
<nigelb> Its per repo. and you need to keep creating teams
<wgrant> Yeah.
<wgrant> I haven't explored GitHub yet, since they don't have free private repos.
<nigelb> Most people don't use it, so there are less complaints.
<nigelb> I hear about this often from Mozilla folks who do use it.
<nigelb> The Private and public repos have the same repo organization. Except the private one has that flag ticked to private :)]
<wgrant> bitbucket has a very simple, easy, fast, but extremely limited privacy system.
<wgrant> nigelb: Are GitHub groups per-repo, or per-account/organization?
<wgrant> bitbucket's are per-account, not per-repo.
<rick_h_> github is organization->repo
<nigelb> Per organization
<rick_h_> https://github.com/pylons is an example, organization, list of repos, and then users part of the org
<huwshimi> wgrant: No I haven't seem them
<wgrant> I know roughly how their organizations work, but I can't see groups.
<wgrant> Are groups only accessible to the org owners?
<nigelb> Yeah
<nigelb> and admins
<wallyworld> wgrant: got it to work by ditching LaunchpadFormHarness and using create_view directly
<wallyworld> and i see that canonical_url(naked_subscriber, rootsite='mainsite') gives a different result to canonical_url(naked_subscriber)
<wgrant> Right, by default it will use the current rootsite.
<wallyworld> first one gives http://launchpad.dev/xxx, the second, http://127.0.0.1/xxx
<wgrant> fmt:url used to do the same, but now it defaults to mainsite for projects/distributions/people.
<wallyworld> perhaps everything should do tyhat
<wallyworld> makes more sense
<wgrant> Well, it will be moot soon.
<wgrant> Because !mainsite will cease to exist.
<wallyworld> yeah
<wallyworld> \o/
<nigelb> Define 'soon'? :)
<wgrant> Undefined as yet.
<nigelb> Also, are you working on changing Launchpad's private branches structure?
<wgrant> We're working on changing just about everything relating to privacy.
<wgrant> Do you have anything in particular in mind?
<nigelb> Nah. I remember you telling its very bad.
<nigelb> If you're fixing, that's very good news :)
<wgrant> lifeless: Have you had a chance to look at my MP yet?
<nigelb> rick_h_: I take it you had a good first day, after we scared you in #launchpad ;)
<wgrant> Hah
<wgrant> bitbucket's email notification broker lets you specify any email address. No verification.
<nigelb> #win
<rick_h_> nigelb: good start. Head swimming a bit
<nigelb> heh
<nigelb> That sounds like my first few days of trying to patch LP.
<rick_h_> yea, I mean the toolchain is about opposite of everything I've used the last few years.
<rick_h_> so when everything is foreign, lots of docs to read/reference
<wgrant> Ah, Pylons, I see.
<rick_h_> yea, pylons, pyramid, sqlalchemy, mako
<rick_h_> jquery, though I have tinkered with YUI and like it a lot
<wgrant> There's been a bit of debate around the company about whether YUI is still the right choice. The consensus seems to be that they're pretty similar, except that YUI requires a bit more structure/setup to start off, but that's probably beneficial in the long run.
<wgrant> I don't have too strong an opinion, but I tend to agree.
<rick_h_> YUI vs ?
<wgrant> jQuery, of course :)
<rick_h_> oh, I'd not say they simliar.
<wgrant> Aren't really any other contenders at the moment...
<wgrant> Similar in terms of ease of use once you're going.
<rick_h_> I just got through ranting on jquery at a conference only to find out one of the guys in the group was a core contrib of jquery
<rick_h_> oops
<wgrant> Heh
<rick_h_> the tooling, organization, and built in structure is a big win in YUI
<wgrant> Yes.
<rick_h_> to compete you'd have to get jquery + requirejs + backbonejs + historyjs
<rick_h_> and start stacking together like a mad men
<rick_h_> I've started the move myself with my OSS bookmark app
<rick_h_> the big win for jquery is that low inertia to get started and just size of community
<wgrant> Exactly.
<wgrant> But that seems to be about all it has going for it.
<rick_h_> but then again, I hear the same things about PHP, and I don't care to use that any more either :)
<rick_h_> I'll have to catch the next time the debate starts up, I'm good at those heh
<nigelb> heh
<nigelb> I hated YUI when I had to use it for the first time.
<rick_h_> well I took the long path there. jquery -> dojo (ugh) -> YUI and kind of 3 bears situation
<rick_h_> one was too light, one was too "java" feeling, YUI seemed just right
<wgrant> It's very much like PHP/Java/Python, I think.
<rick_h_> but 90% of the places I've worked has been all jquery, so definitely know how it goes
<rick_h_> I like to use that comparison, the core jquery dev thought it was a bit unfair though
<wgrant> PHP and jQuery make it very easy to hack stuff together quickly, but you're setting yourself up for pain.
<rick_h_> but I'm picky I guess, I don't care for django either
<wgrant> I hate Django.
<wgrant> Pyramid looks pretty nice, from what I've seen.
<rick_h_> yet it's the big popular kid on the block
<rick_h_> yea, I really liked pylons because there wasn't a ton to it. You could really read/understand/trace the code nicely
<rick_h_> pyramid puts a bit more on top, but it's some good bits
<rick_h_> Chris over there is really good about code quality, testing, and all that
<wgrant> And it reuses the good bits of Zope.
<rick_h_> right
<rick_h_> nice thing is it makes stuff like zcml optional
<wgrant> You know what they say, "those who don't understand Zope are doomed to reinvent it, poorly".
<rick_h_> routes vs traversal, all that
<wgrant> *cough* Django
<lifeless> wgrant: *hah*
<wgrant> Right. But IIRC it's flexible?
<rick_h_> very
<lifeless> wgrant: you won't like the conclusion I've been coming to vis-a-vis django
<wgrant> repoze.bfg certainly did both when I last looked at it.
<wgrant> lifeless: Oh?
<rick_h_> which is the downfall for many. They want to be told "You will put that code there! and only use that tool!"
<lifeless> wgrant: the leanness makes it easy to write efficient code
<lifeless> wgrant: something I am very pro
<rick_h_> makes the docs harder to get into since you have to choose your own adventure
<wgrant> lifeless: Have you looked at Pyramid (or repoze.bfg, its predecessor)?
<rick_h_> I can never get past django orm and templates.
<wgrant> They're lightweight.
<rick_h_> it's like making me use eclipse because it's got all the prebuilt buttons for things
<wgrant> They don't impose nasty stuff on you, like Django.
<wgrant> Yes.
<lifeless> wgrant: I need to have a re-look, last I looked a repoze.bfg it was still built around objects-are-cheap
<lifeless> wgrant: Which is an inappropriate structure for the real world
<rick_h_> lifeless: pyramid has travelled a bit from repoze.bfg days
<wgrant> lifeless: A bit, as anything Zope based is sort of going to lean toward. But it's rpetty flexible.
<wgrant> Unlike classical Zope 3, it doesn't pretend that ZODB is the solution to all problems.
<lifeless> OTOH settings.py must DIAF.
<rick_h_> but again, with that it's a lot about the tools you couple it with
<wgrant> Right.
<wgrant> But in Django you have no choice, really.
<rick_h_> <3 mako and SA
<wgrant> It all comes along.
<lifeless> wgrant: its not zodb so much as the culture of code structure that evolved *because* of zodb
<rick_h_> bah, 90% of pyramid users aren't zodb
<wgrant> Right, but stuff like Zope security is very object-oriented.
<lifeless> rick_h_: I'm not sure the % matters TBH
<wgrant> Which causes as huge trouble.
<wgrant> s/as/us/
<rick_h_> yea, that's put together kind of interesting
<rick_h_> there's the zope stuff with full __acl__ and traversal support in there
<rick_h_> but there are places it gets kind of hooked into for the rest of us
<rick_h_> you end up writing more of your own auth I guess
<rick_h_> but at least they're not entirely reinventing it
<wgrant> Django works OK if you can work inside the framework and don't need to do anything very special.
<rick_h_> yea, for the small one off sites I see the use case
<wgrant> But if you want a non-hopeless ORM, or a different template system, or non-trivial security...
<wgrant> You're stuffed.
<rick_h_> at least they've finally bought into wsgi ;)
<wgrant> You have to throw out half the rest of the framework, or implement it yourself, but the unused bits of the framework sit around haunting you and breaking and obstructing things.
<lifeless> wgrant: the djagno orm appears to be easier to drive that storm, for bulk loading. Just saying.
<wgrant> lifeless: Hm, not when I last looked.
<wgrant> Admittedly that was a while ago.
<rick_h_> multiple fk models...the end
<lifeless> it has some graph traversal primitives
<wgrant> Also, Storm is not exactly the model of perfection any more.
<rick_h_> I gave a SA tutorial at PyOhio, one of the top 5 reasons to use Python is to get access to SA imo
<wgrant> Storm was written back when SA wasn't so good.
<wgrant> And the only real alternative was SQLObject, which was even worse.
<rick_h_> yea, I remember when storm came out doing the side v side and such
<rick_h_> lol, then turbogears went from SO to SA was a good day :)
<rick_h_> /then/when
<wgrant> LP still uses Storm's SO compat layer in lots of places, unfortunately.
<wgrant> There's a *lot* of code to migrate.
<rick_h_> yea, that's what I heard today
<rick_h_> well, should head out if I'm going to get back in the morning
<rick_h_> thanks for the tool talk :)
<wgrant> See ya.
<mwhudson> <lifeless> wgrant: the djagno orm appears to be easier to drive that storm, for bulk loading. Just saying.
<mwhudson> really?
<mwhudson> oh i think you mean something other than what i thought by bulk loading
<lifeless> mwhudson: getting a subset of the objectgraph where depth > 1
<mwhudson> yeah, i thought you meant "getting a lot of data into the db"
<wgrant> select_related and prefetch_related indeed look quite handy.
<mwhudson> because django is pretty awful at that
<wgrant> huwshimi: Yes! Kill the default bug listing!
<wgrant> It is pointless.
<wgrant> mwhudson: Hm, have you seen GitHub's Deploy Keys?
<mwhudson> wgrant: no
<mwhudson> wgrant: hey, that looks a little familiar
<wgrant> mwhudson: Indeedily.
<nigelb> heh
<huwshimi> mwhudson: :)
<nigelb> Is that mwhudson did sometime back?
<mwhudson> well, i talked about something like that
<nigelb> *that what.
<huwshimi> hmm... how might I figure out all the situations where Launchpad sends emails? Is there a function I can grep for?
<StevenK> sendmail
<StevenK> I think
<StevenK> It's quite a disgusting function
<wgrant> There are a number of abstractions on top of that, though.
<wgrant> Just to make matters difficult.
<huwshimi> that's ok, that should be enough for me to get started
<nigelb> wgrant: s/difficult/moar fun/g
<nigelb> :)
<StevenK> jtv: Oh! Did anything happen with your SPN speed-up branch?
<jtv> StevenK: good point.  Want it?  I'll assign it to you.
<jtv> Happen to have a bug for this?
<jtv> If not, I'll just file one.
<StevenK> I do not have a bug for it.
<StevenK> jtv: wgrant is OCR today *cough*
<StevenK> jtv: I'd just like to read the diff
<jtv> OK, coming up.
<jtv> I'll ping you when it's ready for reading.
<wgrant> lifeless: Still lurking?
<StevenK> jtv: Danke
<lifeless> yes
<lifeless> your branch is pending
<lifeless> tell me, have we tested bug / branch search overhead with it ?
<wgrant> I haven't tested bug/branch searches on production data with this iteration of the schema.
<wgrant> But it should be underhead, not overhead.
<lifeless> !data
<lifeless> bbs
<wgrant> Shh
<jtv> StevenK: still waiting for the diff to update, but it's at https://code.launchpad.net/~jtv/launchpad/bug-890542/+merge/82241
<jtv> And there it is.
<jtv> wgrant, care to review it?
<StevenK> jtv: One thing I wanted for the garbo job is it logging how many {B,S}PPHs it has updated.
<jtv> okokok
<wgrant> jtv: The SPPH __call__ checks for completedness twice.
<wgrant> The check at the end was removed from BPPH, but for SPPH it was replaced with something else.
<jtv> Ah, thanks.
<jtv> That's a redundant line now.
<wgrant> I also don't think the SPPH/BPPH aliases are worth it, particularly when you can use find()'s short form.
<wgrant> jtv: Are you using DF at the moment?
<jtv> Only running process-upload on it.  I can stop that.
<jtv> Once the UK comes online I'm hoping to get the buildmaster test going again.  I'll need my local code changes for that.
<wgrant> I don't want to touch the code, just to do touch the DB in bad ways.
<wgrant> Which will involve taking out locks on most bug/branch rows for a while, etc.
<jtv> No skin off my nose.
<jtv> As long as I can get back to testing the buildd stuff in a few hours.
<StevenK> Ah, bah. drivers is a list
<jtv> stub: do we have access to the affected-rows count after a Store.execute?  If so, where?
<StevenK> jtv: Er? len(spphs) ?
<jtv> Hmm yeah, you're right.
<StevenK> jtv: That's what I suggested it -- it's cheap.
<StevenK> s/\(wh\)at/\1y/
<jtv> Can I just logging.getLogger('populate-bspph-bspn')?
<StevenK> self.log is set, I think
<StevenK> self.log.info() ...
<jtv> On the TunableLoop?
<jtv> Ah yes.
<jtv> Mind you, I wrote this stuff.
<jtv> My genius is overshadowed only by my incompetence.
<jtv> If I have climbed higher, it was by standing on the faces of giants.
<StevenK> Because the face is higher than shoulders?
<jtv> And it's more demeaning.  No point being a bigger fish if the pond has room enough for the others.
<stub> jtv: I think so, yes. I think I made it the return value of Store.execute() if the no-result-set option is passed?
 * stub has a look
<jtv> stub: turns out we don't need it just now, but may be useful to be aware of anyway.
<stub> jtv: Search for rowcount in lp/scripts/garbo.py - first hit is an example.
<jtv> Argh rowcount.  I was thinking affected_rows.
<stub> jtv: it isn't the return value. It is an attribute on the return value.
<jtv> Yesâ¦ just saying I did grep but for the wrong identifier.  :/
<jtv> StevenK, wgrant: diff has updated
<wgrant> jtv: Approved.
<jtv> thanks!
<jtv> I do resent the discriminatory comment though.  You will be hearing from HR.
<StevenK> Haha
<jtv> Note the spacing between those sentencesâtake that!
<jtv> Oh, the persecution around here.  This is just like the time when we were ridiculed for knowing SQL's inequality comparison operator.
<jtv> lifeless, stub: some nice news on foreign-key locks.  There's postgres work going on that should effectively eliminate the problem.  The complexity of the change worried me (will it really be completed?) so I proposed a simpler solution, which so far has been received well.
<stub> \o/
<jtv> stub, for background if you're interested: the ongoing work was to add a special lock type, which I think is worrisome because of the many lock interactions and O(nÂ²) correctness problems etc.
<stub> jtv: Just looked at the thread.
<stub> jtv: Just need it implemented and backported to 9.1 :-)
<jtv> Sounds easy.  :)
<jtv> And then, of course, we'd need to set immutable constraints on our key columns.
<stub> jtv: thats a no brainer for all our serial primary keys. a little thought might be required for tables without id, but I suspect all of them would be good to go too.
<jtv> stub: That's what I figured â easiest way to avoid key update conflicts is to disallow updating them, and how often do you want to update keys of anything in normal operation?  Schema documentation alone could make an immutable index nice-to-have.
<stub> jtv: Yes. Especially if you are using a key for external systems (eg. a component in a URL you don't want to change)
<jtv> Hmm yes
<jtv> I wonder what other opportunities might open up if an entire table were covered by immutable constraints.
<stub> jtv: Might make partitioning safer too
<jtv> Yeah.  Replication might be a bit easier, though the code's already there so it might not actually improve things.
<jtv> stub: you're not in the kitchen, btw
<stub> eh?
<jtv> #launchpad-kitchen
<stub> Never heard of it
<lifeless> jtv: thats a nice design, and cool news
<jtv> Thank you.
<lifeless> wgrant: reviewed. Sorry for the slow turnaround
<nigelb> jtv: there's a kitchen channel on freenode?
<jtv> nigelb: no, sorry.  Internal.
<nigelb> Ah, guessed as well :)
<wgrant> lifeless: Thanks.
<wgrant> lifeless: I considered removing APAs for public artifacts, but I'm not quite sure how we want to handle policy changes wrt. restricted observers.
<wgrant> lifeless: eg. if I toggle a bug from private to public and back, does it lose its restricted observers?
<wgrant> I don't really know yet.
<rvba> lifeless: I'd like your opinion about a pattern I've used recently to pre fetch data related to objects in bulk and save quite a few queries.  The easy case is when the other data is looked up by primary key, in this case, eager loading stuff and relying on Storm's cache works great.  But when the related data is not looked up using a primary key, the only solution I've found is to have a
<rvba> cachedproperty and populate it manually.  Is this a pattern that you would recommend? I'm asking because I remember Julian saying it was kinda dangerous to have a cachedproperty on a model objectâ¦ I wonder if all he meant was that you should be careful and call clear_property_cache where it is appropriate or if there was more to it.
<rvba> lifeless: here is the branch I'm talking about https://code.launchpad.net/~rvb/launchpad/builders-timeout-bug-887078-eager-load-cachedproperty/+merge/82198
<lifeless> wgrant: pick something understandable
<lifeless> wgrant: e.g. is it less surprising to have unexpected restricted observers, or to lose them when one person does both actions
<lifeless> rvba: clear_property_cache is done automatically on sqlbase objects
<lifeless> rvba: see the base class
<lifeless> rvba: the caches dict comprehension isn't needed, because you only look the caches up just one
<lifeless> once
<lifeless> rvba: also on line 56 of the diff you wrap the line, but its < 80 characters all combined
<rvba> lifeless: k
<lifeless> the second eager loading method does use the caches variable
<lifeless> so I suspect you built one and copied the structure ;)
<rvba> true :)
<lifeless> this looks fine to me
<rvba> On line 141 I *had* to call clear_property_cache.
<lifeless> there used to be a 'do not use cachedproperty on model objects' concept; I removed that edict in my first month as TA - cached property is extremely useful
<lifeless> rvba: yes, I see that, and its fine
<lifeless> rvba: (or you could clear just that one cached property, you know :)
<rvba> lifeless: right :)
<lifeless> cachedproperty has the nice properties of being local to the data - not lookaside - and being very visible
<lifeless> anyhow, +1 on the branch
<rvba> thanks, I'll continue working on timeouts!
<lifeless> it is exactly the sort of move-to-set-based work we need
<rvba> There is still room for improvement on https://code.launchpad.net/~user
<lifeless> induitibly
<rvba> https://code.launchpad.net/~lifeless â 113 queries/external actions issued in 4.47 seconds
<rvba> I'll get back to that (and to +activereviews) when I'm done with the builder:+history pageâ¦
<adeuring> good morning
<rvba> Morning adeuring.
<adeuring> hi rvba!
<danhg> Morning
<mrevell> Hallo
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 277
<rvba> gmb: mvo has put together a branch to be merged in lp yesterdayâ¦ I asked him to write a MP first, looks like he did it, maybe you can have a look at the branch now if you have some time on your hands? https://code.launchpad.net/~mvo/launchpad/maintenance-check-precise/+merge/82125
<gmb> rvba: Sure, I'll take a look once I'm finished with the current review.
<rvba> gmb: thanks.
<rick_h_> morning part people, man you guys don't mess around overnight
<StevenK> We don't?
<rick_h_> not used to waking to that much email
<nigelb> heh
<rick_h_> guess it's mainly merge notes
<nigelb> Yeah, the team should get sub'd to everything.
<jml> I'm suddenly unable to connect to bazaar.launchpad.net
<jml> $ ssh bazaar.launchpad.net
<jml> ssh_exchange_identification: Connection closed by remote host
<benji> jml: I get something different (and more expected?): No shells on this server.\nConnection to bazaar.launchpad.net closed.
<jml> crap
<jml> hmm. I get that now too.
<benji> Internet wind.
<rvba> same here.
<jml> ok. just 'wind', as benji says.
 * benji reboots into Windows to edit some video before work.
<rick_h_> ouch
<benji> heh
<nigelb> Aww, huw isn't here.
 * nigelb will give him a hug tomorrow.
<nigelb> I just caught up on lp-dev emails.
<rick_h_> oh man, all the good stuff is on that list. Thanks for the heads up nigelb
<rick_h_> jml and benji looks like some peeps in #lp are getting the same issues
<jelmer> win 40
<deryck> Morning, everyone.
<rvba> Hey deryck.
<rick_h_> morning
<nigelb> rick_h_: :)
<abentley> deryck: I've noticed that if you disable importance and status, their column still takes up space.  Had you noticed that?
<deryck> abentley, I had yeah.
<abentley> deryck: cool.  I'll leave it with you.
<deryck> abentley, that will likely be the last branch I do this week... a tune up of the page as fields are shown and hidden.
<deryck> abentley, or rather, the last branch in the mental list I have of polishing branches :)
<abentley> deryck: when the user changes the sorting, but loading the new batch fails, should the sort icon return to its previous value?
<abentley> deryck: when the user changes the sorting, but loading the new batch fails, should the sort icon return to its previous value?  Should we accomplish this by going "back" in history?
<deryck> abentley, hmmmm
<abentley> deryck: scratch the second bit.  History is updated when we get the new batch.
<deryck> abentley, generally, yes, I think the page should not change if the new batch isn't loaded.
<deryck> abentley, I don't have a pref about the mechanism we achieve this with.
<abentley> deryck: I guess we need a mechanism to sync the batch navigator with the sorting widget in general, so we should use that.
<deryck> abentley, right.
<deryck> adeuring, I don't think the colors are opacity are quite right on the beta page banner, but I can tune this up this week in a css polish branch.
<deryck> s/are/or/
<adeuring> deryck: that would be great
<deryck> abentley, and FWIW, the form overlay lines up well in FF for me.  I'll have to dig deeper what's going wrong for you.
<deryck> abentley, maybe the large display you use and overlay scaling?  as a guess.
<abentley> deryck: width is definitely the issue.
<abentley> deryck: i.e. total browser width.
<deryck> yeah
<abentley> deryck: it starts to happen around 987 pixels wide, and the effect is amplified as the browser gets narrower.
<deryck> abentley, ah, ok.  Great, thanks. I can repro now.
<abentley> deryck: np
<rick_h_> http://code.google.com/p/closure-stylesheets/ I see decorators everywhere
<rick_h_> rvba: ping
<nigelb> heh, I love that two channels in two networks are discussing the same thing
<rvba> rick_h_: pong
<rick_h_> rvba: have a sec to chat/mubmle on proper design implementation to fix https://bugs.launchpad.net/launchpad/+bug/741463 with me?
<_mup_> Bug #741463: popup diff close button misplaced <Launchpad itself:Triaged by rharding> < https://launchpad.net/bugs/741463 >
<rvba> rick_h_: sure, let me have a look at the bug.
<rick_h_> thanks
<rick_h_> nigelb: oh yea? where's the other room?
<nigelb> rick_h_: #webdev on irc.mozilla.org :)
<nigelb> rick_h_: (the closure stuff)
<rick_h_> nigelb: ah, gotcha
<rvba> rick_h_: is this a chromium specific issue?
<rick_h_> no, I don't believe so
<rick_h_> it's more to do with the content of the popup
<rvba> rick_h_: do you have a link to a page with this popup?
<rick_h_> see my comment at the end of the bug
<rick_h_> sure
<rick_h_> https://bugs.launchpad.net/launchpad/+bug/814696
<_mup_> Bug #814696: Link to show inline diffs in merge proposals should be green <code-review> <trivial> <ui> <Launchpad itself:Triaged by rharding> < https://launchpad.net/bugs/814696 >
<rick_h_> click the diff link in my related branch
<rvba> ah ok, it the inline diff linkâ¦
<rick_h_> right
<rick_h_> that popup has the grey box that most other popups don't have
<rick_h_> so wanted to discuss what's the UI guideline consistant way, add heading text, just pad down, etc
<rvba> on a bug, the popup "Subscribe someone else" has the same close button.
<rick_h_> right, but it doesn't have a grey header at the top to cloud up the close button
<rvba> true
<sinzui> jcsackett, do you approve of my running this in the production: https://pastebin.canonical.com/55796/
 * rvba fires up firebug.
<rick_h_> so we could margin the diff UI to make sure that there's space for the close icon, padd down the diff, or to add some sort of ui component around that close button that keeps it clear
<jcsackett> sinzui: it looks fine to me.
<rvba> rick_h_: right now it's positioned using float right and height/margin-top.
<sinzui> thanks
<rick_h_> rvba: right, but just moving that position won't do it much good
<rick_h_> so for example, jquery ui dialog adds a grab bar/close button combo to prevent the dialog content from interfering: http://jqueryui.com/demos/dialog/
<rick_h_> for some popups, the large heading does that for us
<rvba> I would be in favor of adding a proper heading like we do elsewere.
<rick_h_> ok, I'll start with that path then.
<rick_h_> thanks
<rvba> rick_h_: welcome. This is a trivial case because you're just getting started but for more complex case, Huw (huwshimi) is the one to talk to.
<rvba> cases even
<rick_h_> ok, deryck gave me a few names and I figured I'd pick randomly to get to introduce myself to everyone
<rick_h_> you got lucky this time :)
<rvba> rick_h_: btw, this popup looks completely different in chrome and in firefox.
<rvba> rick_h_: that's a good idea to get to "know" the teamâ¦
<rick_h_> rvba: yes, FF seems to limit the width unlike chrome
<rvba> rick_h_: exactly
<rick_h_> I'll note that as part of this as well
<rvba> good.
<rvba> this could be a drive-by fix.
<rvba> rick_h_: feel free to ping me again or assign me as a reviewer for this once you come up with a MP.
<rick_h_> will do
<rvba> rick_h_: but note that I'm Europe so I'll EOD pretty soon ;)
<rvba> "*in* Europe" even
<rick_h_> thanks for the heads up
<rvba> np
 * jcsacket1 is getting very tired of "run_all" evidently causing kernel panics ...
<rick_h_> c
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<adeuring> deryck: could you please run a few SQL queries for me?  https://pastebin.canonical.com/55802/
<deryck> adeuring, sure.
<adeuring> thanks!
<deryck> adeuring, you want each one twice?  to see if warm cache matters?
<adeuring> deryck: yes, that would be great
<deryck> adeuring, https://pastebin.canonical.com/55803/
<adeuring> deryck: thanks!
<deryck> adeuring, np!
<adeuring> deryck: one more SQL query, please: https://pastebin.canonical.com/55805/ (I'd like to know how the new slow distribution related queries compare with "established sorting)
<deryck> adeuring, ok
<deryck> adeuring, https://pastebin.canonical.com/55806/
<adeuring> deryck: great, thanks!
<adeuring> deryck: so, a similary slow query... I think I'll just ignore the fact that the new queries look a bit quetsionable when run with the ubuntu as the target...
<deryck> adeuring, yeah, just get them working, and we can drop or tweak in beta.
<adeuring> deryck: could you review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-sorting-2/+merge/82298 (< 150 lines)?
<deryck> adeuring, sure
<adeuring> deryck: cool, thanks!
<deryck> adeuring, looks great.  Thanks for removing the need for isinstance!  Otherwise, pretty straightforward and r=me. :)
<adeuring> deryck: thanks :)
<gary_poster> I tried to submit the change to increase the per-test yuixhr timeout to avoid occasional spurious failures, but we are already in testfix.  Anyone recognize this?  https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1557/steps/shell_6/logs/summary
<gary_poster> It looks like a bunch of bzr errors
<deryck> gary_poster, saw the exact same errors yesterday.  Couldn't repro locally so forced build.
<deryck> gary_poster, flacoste suggested disabling the tests if we saw it again.
<flacoste> deryck, gary_poster: yes, please disable
<gary_poster> deryck, ugh, so yuixhr is not the only culprit. :-(  OK, I'll force this build to get my fix in, and then...mm, ok.  I wonder if I should disable just the first one as an experiment
<flacoste> i think we disable codehosting integration tests in the past already
<flacoste> maybe not those, maybe others
<gary_poster> flacoste, I'm tempted to try disabling only the first one, with the hypothesis that it might be messing up the rest.  You fine with that as an experiment?  I'll report on it to the list
<flacoste> gary_poster: sure!
<flacoste> and welcome back!
<gary_poster> So if it happens again, people can know the experiment fails
<gary_poster> cool
<gary_poster> thanks flacoste :-)
<deryck[lunch]> and with that, I'll ear :)
<deryck[lunch]> eat, rather.
<flacoste> gary_poster: chriscoulson is reporting he can't push to bzr on #launchpad
<flacoste> ssh_exchange_identification: Connection closed by remote host
<flacoste> gary_poster: can someone on your squad investigate?
<flacoste> pretty please
<bac> flacoste: i got that a few times trying to bzr branch
<gary_poster> flacoste :-) sure
<flacoste> he says it started about 30 minutes ago
<gary_poster> bac, are you up for digging into for just a bit?
<gary_poster> I can join you in a few minutes
<bac> flacoste: then after i rechecked the project name it went away.  nothing different, just worked later
<gary_poster> if you'd like
<bac> gary_poster: sure
<gary_poster> thank you
<rick_h_> is that the same that was going on this morning?
<gary_poster> we don't know, rick_h.  What was going on this morning? :-)
<rick_h_> benji: was looking this morning I think
<rick_h_> bzr issues, then retry would work
<rick_h_> not sure what came of it
<flacoste> gary_poster, bac, rick_h_: you might want to discuss the details with chriscoulson in #launchpad
<flacoste> he might confirm it's the same or different problem
<benji> flacoste: actually, gmb and gnuoy were looking at it; restarting the LB made the symtom go away
<gary_poster> yeah, was hoping bac would do that :-)
<benji> I can confirm that I'm seeing the symptom again.
<flacoste> benji: LB = ? (librarian?)
<gary_poster> load balancer?
<benji> flacoste: load balancer
<flacoste> load balancer!
<flacoste> of course
<benji> everyone yell it together: "Load Balancer!"
<flacoste> lol
<benji> :
<gary_poster> LOAD BALANCER!
<benji> :p
* bac changed the topic of #launchpad-dev to: Code hosting is intermittently down.  Being investigated.  https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
* gary_poster changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<abentley> deryck: there's no OCR.  Could you review https://code.launchpad.net/~abentley/launchpad/next-prev/+merge/82308 ?
<deryck> abentley, sure.  it will be just a minute, though.  Finishing something with rick_h_ before he EODs.
<abentley> deryck: okay.
<sinzui> jcsackett, may I have your r for https://pastebin.canonical.com/55809/
<flacoste> benji: there is some evidence that bug #808930 is still not fixed
<_mup_> Bug #808930: Timeout running branch scanner job <escalated> <linaro> <oops> <Launchpad itself:In Progress by benji> < https://launchpad.net/bugs/808930 >
<flacoste> unless we pushed a new bzr version in the last two days
<flacoste> which i don't remember seeing
<benji> flacoste: good; I was hoping for some good evidence... or that I would never hear of it again ;)
<jcsackett> sinzui: Sure, looks fine to me. Very similar to the last one.
<benji> flacoste: nope, no bzr in the last two days AFAIK; what is the new evidence?
<flacoste> benji: fair enough :-) i reopened the bug
<sinzui> Yes, it is. we will not run this until Ian's changes are in production though
<flacoste> benji: a recent merge proposal that failed
<jcsackett> Ok.
<benji> flacoste: ok; I assume you're putting that in the bug
<flacoste> benji: i did
<benji> thanks
<deryck> abentley, looking at your review... we normally spell those conditionals like:  if (Y.Lang.isValue(current_batch.prev))
<deryck> abentley, perhaps this would play better and not force you to specify nulls everywhere in the test?
<abentley> deryck: I prefer to be more precise.  If current_batch.prev is undefined, that's ill formed data.
<abentley> deryck: to me, it's the difference between "I don't know what the next batch is" and "there is no next batch."
<deryck> abentley, ah, right.  I looked at the check wrong; rather I missed it was a leading to a return.
<deryck> abentley, so for consistency I think we should do Y.Lang.isNull(VALUE).  but otherwise, it looks good.
<deryck> I realize the === is standard language stuff, but we've been driving people to use the isXXX wrappers for awhile now.
<abentley> deryck: what's the advantage of the isXXX wrappers?
<deryck> abentley, for this sort of thing, not much.  the isValue is simpler to right than 3 or 4 checks against non-valuey values....
<deryck> abentley, I think the argument is just to be consistent using Y.Lang.
<abentley> deryck: Yes, I agree about isValue.  Very handy.
<deryck> above I meant write, not right.  sorry.
<deryck> abentley, I think isNull is nice too that you don't accidentally write ==, but lint should catch that.
<abentley> deryck: that's true.
<deryck> abentley, plus if we're consistent about it, it makes it easier on devs looking at js code. no misreading or mis-cut-n-pasting. :)
<deryck> abentley, also, this is in our style guide under "Plain Old JavaScript."  Not that we follow that wonderfully, but it was thought through at some point.
<abentley> deryck: Okay.  I certainly missed that.
<allenap> sinzui: I'm thinking about getting rid of the DistributionSourcePackage/DistributionSourcePackageInDatabase split. Have you ever given this thought before? I'm trying to see if I can steal someone else's plan.
<deryck> abentley, r=me with that minor change.
<abentley> deryck: thanks.
<deryck> np!
<deryck> I think gmb has the new all-time best term for our meetings:  Thunderepic!
<deryck> heh.  found === checks in my own code.  hard to get in that habit.
<deryck> abentley, care to return the favor?  I need a review too.
<abentley> deryck: sure.
<deryck> abentley, thanks! https://code.launchpad.net/~deryck/launchpad/field-visibility-dry/+merge/82316
<sinzui> allenap: purple think this can happen. We were going to do it, but we ran out of time. Our remaining concern are cases where Lp works with a virtual DSP before creating the real thing
<allenap> sinzui: Okay, that sounds doable. I don't really have any more questions yet, but at least I know it's not a fool's errand. Thanks.
<abentley> deryck: I'm not sure it's a good idea for us to maintain both LP.cache.field_visibility and ListingNavigator.field_visibility as models of field visibility.
<deryck> abentley, ok.  do you mean BugListingConfigUtil.field_visibility rather than ListingNavigator?
<abentley> I mean ListingNavigator.  I don't know about BugListingConfigUtil.field_visibility.
<abentley> But if you mean that there are actually three models of field visibility, then I think we definitely need to reconsider.
<deryck> abentley, well BugListingConfigUtil keeps a reference internally, but uses the cache as the model or definitive source.
<deryck> abentley, but I was tossed up about keeping the reference around even, so don't mind dropping it if you think so.
<abentley> deryck: I see.
<deryck> abentley, is your earlier question about something I've done in this code?  Or a general question about field_visibility in ListingNavigator?
<abentley> deryck: Yes, it's about you wanting to update LP.cache in case other folks are using it.
<abentley> deryck: that means it's a model of some of the page state.
<deryck> abentley, ah ok.
<deryck> abentley, I see what you mean now.  My understanding was that was what we were using.  if it's only effort for default page load values, then I can drop that bit.
<abentley> deryck: I'm using the wrap_resource version of the cache, so it should be a separate copy.
<abentley> deryck: if it isn't, then you're changing LN's internal state, and LN
<abentley> deryck: is changing BLCU's internal state.
<deryck> abentley, gotcha.  yeah, I'll drop that then and only pass off to LN the way we currently do.
<lifeless> StevenK: oh hi mr OCR
<lifeless> https://code.launchpad.net/~lifeless/python-oops-wsgi/bug-888866/+merge/82088
<abentley> deryck: I tend to think the BLCU and LN should explicitly share a model.
<deryck> abentley, perhaps.  I don't mind doing the event-based hand off right now, until we see it becomes an issue.
<abentley> deryck: Actually, the status widget needs to share that model, too.
<abentley> deryck: since only visible fields should be sortable.
<lifeless> StevenK: and https://code.launchpad.net/~lifeless/python-oops-datedir-repo/oops-prune/+merge/82324
<deryck> abentley, ah, right.
<lifeless> StevenK: please :)
<deryck> abentley, I was going to do an event thing again.
<deryck> abentley, but it would be nice if the events could fire and each piece look into the model for state.  where the model is independent of our widgets.
<abentley> deryck: Yes, I think that would be nice, too.
<deryck> abentley, do you mind this branch as it, and let me press ahead with some polish and we can discuss this issue early tomorrow?
<abentley> deryck: sure.
<deryck> abentley, awesome, thanks!
<abentley> deryck: why does the valueFn for field_visibility_defaults check LP.cache.field_visibility?
<poolie> hi all
<deryck> abentley, I was routing everything through field_visibility in the cache.  This is the part I'll change I'll change per this discussion.
<deryck> hi poolie
<abentley> deryck: but why doesn't it check  LP.cache.field_visibility_defaults?
<deryck> abentley, ah, sorry.  That's a bug.  I misread your question.
<deryck> abentley, I can't believe the test didn't catch it.  That's scary.
<abentley> deryck: hehe.
<deryck> abentley, ah, I see it's just the existence check, not the value.  and since both are found in the test setup.
<deryck> abentley, I'll do a little wrapper method for "does the cache exist and have these values" to prevent this goof again.
<abentley> deryck: okie.
<abentley> deryck: r=me with that bugfix.
<deryck> abentley, awesome sauce.  Thanks!
<rick_h_> so what does r=me stand for? I've not been able to figure that shortcut out
<deryck> rick_h_, in the commite message the tools we use to land add r=ircnick for who approved the branch.
<deryck> rick_h_, so it's shorthand for r=deryck or "you have an approval vote from me"
<rick_h_> ah, ok so less a shortcut and more a reference
<deryck> rick_h_, exactly.
<deryck> ok, until tomorrow then.  later all!
<jcsackett> sinzui: in an ongoing saga of useless technology, i cannot seem to connect to mumble.
<sinzui> jcsackett, :( Do you have a message for us on IRC about loggerhead and or team branches
<jcsackett> sinzui: loggerhead branch should be out in the wild soon. i've been battling a kernel panic whenever i try to run codehosting stuff, but think i have that resolved. i intend to work longer today to catch up, because i'm irritated.
<jcsackett> sinzui: finally found bug 813146, which includes stuff to deal with it, so trying that.
<_mup_> Bug #813146: kernel panic when running Python test suite on ecryptfs <3.0.0-11.18> <apport-collected> <kernel-request-3.0.0-11.17> <kernel-request-3.0.0-12.20> <oneiric> <patch> <regression-release> <rls-p-tracking> <unity-2d> <eCryptfs:In Progress by tyhicks> <ecryptfs-utils (Ubuntu):Invalid> <linux (Ubuntu):In Progress by tyhicks> <ecryptfs-utils (Ubuntu Oneiric):Invalid> <linux (Ubuntu Oneiric):Confirmed> <ecryptfs-utils (Ubuntu Precise):In
<jcsackett> sinzui: it's not been a good day. :-P
<jcsackett> (thus saga of useless technology).
<StevenK> sinzui, wgrant, wallyworld: http://pastebin.ubuntu.com/739703/
<lifeless> StevenK: hi
<StevenK> lifeless: Distracted by e-mail and breakfast
<StevenK> lifeless: Approverated with a small comment.
<lifeless> both ?
<lifeless> StevenK: ^
<wgrant> lifeless: person merge and fastdowntime
<wgrant> Any suggestions?
<lifeless> fire and brimstone ?
<wgrant> Since it checks for foreign keys.
<wgrant> So if I add a new column referencing person, person merge breaks.
<lifeless> yeah
<lifeless> don't add the reference
<wgrant> Two-step schema patch, or a whitelist of empty tables?
<wgrant> Mmm.
<wgrant> That seems bad.
<wgrant> But maybe,.
<lifeless> aka two step schema patch
<lifeless> add the column
<lifeless> update person merge
<wgrant> I know
<lifeless> make it a reference
<lifeless> start using it
<wgrant> But two-step schema patches for everything seems bad.
<lifeless> less bad than 1 hour downtimes, and currently I think its an either-or choice
<wgrant> Well, I'd prefer two-step code patches.
<lifeless> if that works, I'm fine with it
<StevenK> lifeless: There was a second one?
<lifeless> 10:45 < lifeless> StevenK: oh hi mr OCR
<lifeless> 10:45 < lifeless> https://code.launchpad.net/~lifeless/python-oops-wsgi/bug-888866/+merge/82088
<lifeless> if you'd be so kind
<StevenK> lifeless: Aren't you, you know, supposed to not be working today?
<lifeless> StevenK: yesterday was a write-off
<lifeless> so its a bit backwards.
<StevenK> wallyworld: launchpad.See? Really?
<wallyworld> StevenK: sinzui said it had to be a verb, and i thought something analogous to View. any better suggestions welcome
<wallyworld> i originally had launchpad.Exists
<lifeless> Exists is a verb
<wallyworld> i guess so. i'll have to check the email to see why that was not preferred
<wallyworld> i prefer Exists
<lifeless> benefits of learning french :)
<lifeless> etre
<StevenK> wallyworld: And why can't we use lp.View?
<wallyworld> lifeless: Exists is a verb but not a transitive one like View or Moderate
<wallyworld> StevenK: because we want something that is less than View
<wallyworld> ie they can only see the basic details
<wallyworld> of a private artifact by virtue of the fact they can see an associated object
<wgrant> They're not all verbs.
<wgrant> eg. launchpad.BugSupervisor, launchpad.Driver, zope.Public...
<StevenK> Personally, I remain unconvinced.
<wallyworld> sure, but the preference for verbs was indicated
<wallyworld> i'm happy to change it to Exists or something better
<wallyworld> whatever the consensus is
<StevenK> I think my issue is adding a new permission.
<wgrant> We need a new permission.
<wgrant> (or to abolish permission, but that's a story for another day)
<lifeless> so  I think the idea behind verb is to be able to say 'user can <permission> <object>'
<lifeless> which clearly zope.Public doesn't fit.
<lifeless> and the others wgrant mentioned.
<StevenK> In which lp.See makes more sense
<lifeless> I think exists is ambiguous and confusing on that basis.
<StevenK> Observe ?
<lifeless> TellExists
<lifeless> the key thing to hint at is that they only get shallow metadata
<StevenK> You can observe the object, but you can't view it in the 'user can <perm> <object>'
<lifeless> you could, for instance, grant zope.Public on private objects when the user can see their existence.
<wallyworld> i like Observe
<lifeless> This would be appropriate.
<wgrant> Although Observe already has an alternate meaning in Disclosure.
<wgrant> It means "see absolutely everything"
<lifeless> right, using observe would be horribly confusing.
<StevenK> lp.C
<lifeless> tell me, why is this a new permission vs refined granting of existing permission
<wallyworld> we could redefine observer in disclosure :-P
<StevenK> What could possibly go wrong
<wallyworld> we didn't want to change what View meant
<lifeless> is it because the existing permissions grant access to the whole object, so we've got no middle ground to work with
<wallyworld> yes
<lifeless> yeah, fair enough
<lifeless> so
<lifeless> RestrictedView
<lifeless> is my suggestion
<wallyworld> that works
<lifeless> or LimitedView
<wallyworld> i guess Exists was used because it more closer defines what semantics were are trying to implement
<wallyworld> ie can the user know of the existance of this private artifact
<lifeless> KnowExists
<wallyworld> that works too
<wallyworld> i think any choice will not be "perfect"
<lifeless> If I was doing this, i think I would use KnowExists, or LimitedView
<wallyworld> LimitedView is nice because it modifies an existing permission which people understand. well that appeals to me anyway
#launchpad-dev 2011-11-16
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 277
 * wallyworld types s/launchpad.See/launchpad.LimitedView/g
<StevenK>  /g is probably not necessary
<wallyworld> but harmless none the less
<wallyworld> it was figurative anyway - my ide is doing it :-P
<StevenK> wallyworld: :%s/\(launchpad.\)See/\1LimitedView/
<wallyworld> now you're just showing off
<lifeless> don't you use an IDE ?
<lifeless> ;)
<wallyworld> yes, but i don't do regex search and replace very often
<mwhudson> bzr ls -0VRk file | xargs -o sed -i -e 's/
<mwhudson> the fact that i can type that without thinking much is a little scary
<mwhudson> (should be -0)
<wgrant> bzr sed!
<mwhudson> yes... i think
<wallyworld> i like my IDE because for non-trivial search and replace you can preview the changes and veto any matches which should not be included
<wgrant> sed, diff...
<wgrant> done
<wallyworld> sure, but text based (harder to read the output) and not nearly as user friendly
<wallyworld> ie can you click through from a match to the source code
<poolie> lifeless, fwiw i think you should just capture all traceback
<wgrant> lifeless: 20 frames isn't many
<wgrant> (thanks to TAL)
<lifeless> whats a better number ?
<poolie> 60?
<poolie> 50?
<StevenK> lifeless: So what about ++profile++sqltrace ?
<lifeless> what about it ?
<StevenK> lifeless: Oh, you want to do it all the time?
<lifeless> in the query log in OOPSes, yes
<StevenK> Oh, and do the work in the OOPS handler?
<lifeless> the timeline, yes.
<lifeless> oops handler is too late
<lifeless> I found a methodology error too
<lifeless> timeit python API
<lifeless> != timeit CLI behaviour
<lifeless> StevenK: this doesn't stop the sqltrace profiler beind useful, it just means we have equivalent data whenever we get an OOPS
<lifeless> it does point the way at possibly simplifying the profiler implementation
<huwshimi> I assume I can ignore these errors and do a lp-land? http://paste.ubuntu.com/739827/
<StevenK> huwshimi: That's odd. That's local, or via ec2?
<huwshimi> StevenK: ec2. from yesterday afternoon
<StevenK> Wow.
<huwshimi> StevenK: Did I break the world?
<StevenK> huwshimi: I just find it surprising that something else is listening on 2121 on an ec2 instance while the test is running
<wgrant> StevenK: Some things listen on random ports.
<wgrant> eg. the librarian
<huwshimi> StevenK: So I can ignore it then?
<hloeung> poolie, jelmer: ok, packages are there now. Where would you like me to install / update?
<wgrant> hloeung: clementine
<wgrant> (the guest on concordia)
<hloeung> hmm, it seems I don't have SSH access into clementine
<hloeung> nevermind, found another way around that
<wgrant> Correct.
<wgrant> It's a guest that is torn down every build.
<wgrant> You need to update its image.
<hloeung> ah right
<lifeless> StevenK: oh hai ;) https://code.launchpad.net/~lifeless/python-oops-tools/bug-890001/+merge/82334
<lifeless> StevenK: (diff coming)
<benji> hi lifeless; if I put a fastdowntime deployment on the deployment page would you (or stub) have time to do it in the next few hours?
<lifeless> benji: no; we only have one window a day for fdt
<lifeless> benji: (because they are downtime)
<StevenK> lifeless: r=me
<benji> lifeless: darn, well that's understandable, I'll put it up during my day tomorrow to get in line for the next one; thanks
<lifeless> benji: they are also done by losa FWIW. Perhaps you mean a live schema patch ?
<lifeless> benji: next one is in ~ 6 hours
<StevenK> Which patches are unapplied?
<benji> lifeless: oh right, I think stub said that this (adding a table) was doable live
<lifeless> benji: does the table have FK references ?
 * benji tries to remember
<lifeless> if it does, it requires downtime.
<wallyworld> sinzui: if you are still around - do you have any objection to me adding isBugSupervisor(obj) and isSecurityContact(obj) to IPersonRoles? to compliment isOwner(obj) and isDriver(obj)
<benji> lifeless: yep it does; fastdowntime it is
<wallyworld> sinzui:  or do we want to avoid referencing bug model artifacts from registry
<wallyworld> i can see why we would not want to do that
<wgrant> benji: So, request it for today's window.
<wgrant> benji: Since we can't do it tomorrow.
<sinzui> wallyworld, go ahead. They are consistent. <- StevenK wallyworld may be doing some of the role checking you are working on
<wgrant> (there is an extended outage for a slony upgrade)
<StevenK> I wonder if my db-devel patch has hit prod
<wgrant> StevenK: No
<wgrant> StevenK: is that before benji's?
<wallyworld> sinzui: thanks. i thought they were consistent but wasn't sure if we considered registry a lower layer than bugs, hence the question about referencing
<StevenK> wgrant: Let me check
<wgrant> benji: I also added your patch number to the allocation listing this morning, since it seems you missed that.
 * wgrant lunches.
<StevenK> wgrant: db-stable r11140
<sinzui> wallyworld, registry is considered the lowest and only required app.
<wgrant> StevenK: That's unfortunate.
<StevenK> wgrant: Oh? Why?
<wgrant> Because we can't do benji's tonight.
<wgrant> And I have two landing now.
<wgrant> And tomorrow's window is used.
<wgrant> Well, only one of my two is landing now.
<StevenK> Lean on lifeless for more than one patch a day?
<wgrant> But the other is ready, blocked on the first being deployed.
<wgrant> Which probably can't happen until Wed.
<wallyworld> sinzui: yes. so the fact that registry (PersonRoles) would now reference interfaces declared in the bugs layer may be considered "dirty"
<StevenK> FDT is not ... fast
<wgrant> wallyworld: Dirty but sometimes unavoidable.
<wgrant> wallyworld: And bugsupervisor is registry anyway.
<StevenK> wallyworld: That's a layering violation
<sinzui> wallyworld, yes it is, but it will not be. forever. at least not for security...it will go to registry because many artefacts could be labeled security
<wallyworld> wgrant: IHasBugSupervisor is bugs
<StevenK> Move it down, I'd suggest
<StevenK> In a seperate branch
<StevenK> sinzui: When were you thinking of chatting? I'll be lunching soon
<wallyworld> StevenK: yes, that was the basis of my question. i didn't like the layering violation
<wallyworld> i think the consensus is just do it now and fix the layering later
<StevenK> Fixing the layering should be very quick
<StevenK> And I don't like leaving tech debt around if it can be helped.
<StevenK> We already have too much!
<wallyworld> yes, possibly touch many files though
<wallyworld> i won't leave it hanging around
<StevenK> It can be done via regex ... :-P
<lifeless> its questionable to say its bugs, given the upcoming issuetracker concept :)
<StevenK> ForbiddenAttribute: ('bug_supervisor', <lp.registry.model.distributionsourcepackage.DistributionSourcePackage object at 0xcbf0290>)
<StevenK> :-(
<sinzui> StevenK, wallyworld. we want the DSP to have a bug supervisor. It is not in scope (yet) for disclosure
<sinzui> see bug 191809 and bug 619218
<_mup_> Bug #191809: A DistributionSourcePackage needs a bug supervisor role to control permissions <lp-bugs> <search> <ubuntu-qa> <Launchpad itself:Triaged> < https://launchpad.net/bugs/191809 >
<_mup_> Bug #619218: It isn't possible to set the bug supervisor for a source package <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/619218 >
<huwshimi> lifeless: If you get the chance at some stage I would like to get another review of this: https://code.launchpad.net/~huwshimi/launchpad/avatars-everywhere-712894/+merge/81430
<sinzui> well I say the later is a dupe of the former bug
<wallyworld> not a recent problem then
<sinzui> no. dsp could have a field that returns the distro bug supervisor so that that the interface is met
<lifeless> wgrant: remember that oops that was breaking staging amqp oops loading
<wallyworld> yes it could
<lifeless> wgrant: anyhow, its hilarious
<lifeless> StevenK: hi. Diff coming soon - https://code.launchpad.net/~lifeless/python-oops-tools/bug-889982/+merge/82337
<lifeless> wgrant: this was the cause - https://bugs.launchpad.net/launchpad/+bug/890951
<_mup_> Bug #890951: bad oops prefix in staging translations-export-to-branch <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/890951 >
<lifeless> sinzui: what was the subject of the mail that kicked off the thread about arbitrarty cocs for teams ?
<lifeless> sinzui: I had some thoughts but have lost the thread
<StevenK> sinzui: I'm just confused that IHasBugSupervisor.providedBy(self.pillar) is returning True for DSPs
<sinzui> lifeless, Proposed team agreements feature
<lifeless> cool
<lifeless> I'm aware this is a community driven thing
<lifeless> I thing there is a good inflection point in LP to push most of it out of LP entirely - so they can do whatever they want as a django app or similar.
<lifeless> s/thing/think/
<StevenK> Now I'm being mocked.
<lifeless> I will mail my thoughts in in a bit.
<StevenK> In [3]: IHasBugSupervisor.providedBy(dsp)
<StevenK> Out[3]: False
<lifeless> StevenK: by me asking for reviews?
<StevenK> lifeless: Certainly not!
<lifeless> :)
<StevenK> lifeless: I've approved your last one
<StevenK> OH, it's the bloody view change
<lifeless> poolie: bug 885972 might be up your alley
<_mup_> Bug #885972: raw_sendmail creates TimedActions with invalid detail <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/885972 >
<sinzui> trying to see all of the disclosure feature is like looking into the face to Cthulhu; madness
<wgrant> lifeless: Haha, handy.
<wgrant> sinzui: Rather.
<wgrant> Intertwined and partially invisible madness :(
<StevenK> % uptime
<StevenK>  14:19:32 up 14:07,  3 users,  load average: 31.39, 24.86, 12.65
<StevenK> Dear RabbitMQLayer, DIE
<lifeless> wgrant: oh, you just read scrollback ?
<wgrant> lifeless: Yes.
 * StevenK is slowly getting happier with this branch
<StevenK> 4 files changed, 53 insertions(+), 54 deletions(-)
<jtv1> StevenK, wgrant: no joy from dogfood buildmaster.  :(
<wgrant> Well, it's not running at the moment... that could be relevant.
<wgrant> I wonder why not.
 * wgrant starts.
<wgrant> It is building stuff now.
<lifeless> hmm
<lifeless> what, if any, js library do we use in oops-tools, I wonder
<lifeless> (I want expandable sections on the timeline)
<lifeless> also, I want a tool to report on out of date deps
<lifeless> does setuptools/buildout perhaps know already ?
<lifeless> StevenK: oh hai :) https://code.launchpad.net/~lifeless/python-oops-datedir-repo/timeline/+merge/82344
<StevenK> sinzui: Can haz chat, if you're still around?
<StevenK> lifeless: r=me
<jtv1> StevenK, wgrant: builds are failing on dogfood, but I don't see anything explaining the failures.  Am I missing something, or did I simply break the build farm?
<wgrant> Where are the failures?
<jtv> https://dogfood.launchpad.net/~jtv/+archive/ppa/+packages?batch=150
<poolie> https://dev.launchpad.net/EC2Test/Image we use an EBS instance too, right?
<StevenK> We do not.
<poolie> no, sorry, instance storage?
<StevenK> Right
<StevenK> It is not EBS-based.
<StevenK> Er, backed
<poolie> and 'large'
<StevenK> I think it's xlarge
<StevenK> It's in the code anyway
<mwhudson> ec2 test predates ebs backed images
<mwhudson> i think instance store makes more sense for ec2 test-ish ephemeral vms anyway
<poolie> yeah
<poolie> i may try actually changing it to put more on a ramdisk
<mwhudson> ah yeah, that may be a win
<StevenK> I'd like EBS-backed instances for Jenkins, only for the reason that we can start testing quicker if we have a seeded download-cache and such
<poolie> wee
<mwhudson> there are much more powerful instances now too, may be worth trying them out
<poolie> 23MB/s apt update
<mwhudson> (partly the power is in multi-cores, which doesn't help so much of course)
<mwhudson> "High-Memory Quadruple Extra Large" instances are really quite something
<poolie> so apparently the local store is kinda slow
<poolie> don't have data
<poolie> may try it now while this is running
<mwhudson> ebs is even slower :-)
<mwhudson> but yes ramdisks do help afaik
<mwhudson> "High-Memory Extra Large Instance" would almost certainly be better for us than basic extra large (fewer cores, but more 'compute units' per core)
<mwhudson> cheaper too
<mwhudson> maybe "High-Memory Double Extra Large Instance" with everything stuffed onto a ramdisk would be fast enough that the extra cost is mediated?
<StevenK> That's c1.2xlarge?
<StevenK> We currently use c1.xlarge instances
<mwhudson> no, m2.xlarge or m2.2xlarge
<mwhudson> don't think c1.2xlarge is actually a thing?
<StevenK> I was guessing
<mwhudson> heh :)
<mwhudson> the names are hard to remember, there's almost a pattern
<jtv> StevenK, wgrant: no logs of my build failures whatsoever afaict.  I think it's all broken.
<StevenK> Yes, almost.
<wgrant> jtv: Indeed, it looks that way.
<jtv> :-(
<wgrant> Did you try it locally?
<jtv> No.
<StevenK> storeBuildInfo is now utterly fucked?
<StevenK> Or is failure counting screwing us over?
<jtv> Or log retrievalâ¦ who knows.
<wgrant> The builds dispatch fine.
<wgrant> I saw logs.
<jtv> Failure accounting will have seen some changes, I think.
<StevenK> I saw logtails
<jtv> Yes, there were logs previously.
<StevenK> *COUNTING*
<jtv> Ah.
<jtv> Failure counting will have seen some changes, I think.
<jtv> With retries in particular, IIRC.
<jtv> Some information or other from a previous failure now survives into a next dispatch, that previously didn't.
<poolie> mwhudson, trying a tmpfs hack now
<mwhudson> poolie: woo
<lifeless> poolie: will you look at that sendmail issue? [I d/c'd before apologies if you replied already]
<mwhudson> poolie: i think i'll email launchpad-dev about trying m2.xlarge, do you want to follow up about tmpfss?
<StevenK> lifeless: You need better internets
<lifeless> StevenK: I know :(
<lifeless> StevenK: https://code.launchpad.net/~lifeless/python-timeline/backtrace/+merge/82348
<poolie> cat is sleeping on my mouse cord
<StevenK> No diff, no review.
<poolie> makes it hard
<StevenK> :-P
<StevenK> poolie: Your mouse has a cord? :-P
<poolie> that may be my problem
<poolie> lifeless, are you talking about bug 885972
<_mup_> Bug #885972: raw_sendmail creates TimedActions with invalid detail <oops-infrastructure> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/885972 >
<StevenK> My keyboard and mouse are cordless. It's quite nice.
<lifeless> poolie: yes
<lifeless> StevenK: there is a diff
<lifeless> :P
<jtv> StevenK: don't rub it inâmy wrists have been aching ever since bluetooth stopped working
<StevenK> Haha
<jtv> (*mostly* stopped workingâ¦ it still works for some things)
<StevenK> Mine are abusing 2.4GHz
<lifeless> StevenK: I'm sorry for muddling the LGPL change in there, I should have self-reviewed it separately; I claim tiredness.
<lifeless> StevenK: it is a separate commit, you may be better clicking on the commits
<poolie> lifeless, i'll see what I can do
 * jtv goes for a little cry or food or something
<StevenK> lifeless: I'm already trying to keep track of my own branch, there is six other branches you've tossed me over the course of the day, and sinzui has sent me a novel that I need to reply to. My brain, it is leaking out my ears.
 * StevenK stabs lifeless for the 1,400 lines.
<StevenK> # The fifth element defaults to None:
<StevenK> Bruce Willis would not approve.
<lifeless> poolie: thanks
<lifeless> StevenK: yeah, I know. Gotta love small reusable components!
<lifeless> StevenK: don't worry, my next branch is on LP itself.
<StevenK> lifeless: You don't specify 'v3' for LGPL in setup.py
<lifeless> StevenK: you can't
<StevenK> You did for AGPL
<lifeless> yes
<lifeless> http://mail.python.org/pipermail/catalog-sig/2011-November/004028.html
<StevenK> And you seem to repeat LGPL once too many in README
<lifeless> hmm, let me see
<lifeless> thats a nuisance, cause I've got that copied all over the place now.
<lifeless> Nice catch
<lifeless> its in every darn little header too
<lifeless> fuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
<lifeless> I'll just delete that last line.
<poolie> mwhudson, really a one line patch? that's cool
<mwhudson> poolie: it's two, it turns out
<mwhudson> but yes
<mwhudson> very easy
<poolie> i get the chance to add better handling of instance creation errors
<poolie> by printing the reason code
<poolie> apparently they have sold out of computers
<mwhudson> heh
<lifeless> yes we have no banaanas ?
<StevenK> No bananas today!
<poolie> http://support.rightscale.com/06-FAQs/FAQ_0112_-_How_come_I_get_an_insufficient_capacity_error_when_launching_an_EC2_instance%3F
<mwhudson> i saw a fun fact a while ago: amazon adds more capacity to ec2 every day than they had running the entire amazon organization for the first five years of its existence
<lifeless> yeah
<lifeless> scary shit
<StevenK> mwhudson: *!!!*
<poolie> yep, i heard that too
 * mwhudson fails to google up the source for that
<lifeless> it was in their newsletter
<poolie> the other one is that google is the largest pc manufacturer in the world
<poolie> perhaps fsvo pc
<mwhudson> "launch an instance in the -Any- availability zone" -- _surely_ we do that already
<StevenK> Step 5 of that answer makes me eyeroll.
<poolie> and fsvo manufacturer
<lifeless> mwhudson: no, you get to choose a zone
<lifeless> mwhudson: this is 'are you feeling lucky punk' zoning
<mwhudson> i know you can specify a zone, but you can also say "i don't care which zone" can't you?
<mwhudson> i can't see anything in devscripts/ec2test that would specify a zone
<mwhudson> (or a region, come to that)
<mwhudson> anyway, time to go
<poolie> hm maybe i can also fix this '400 bad request' in shutdown
<lifeless> the endpoint you connect to
<lifeless> isn't it
<mwhudson> lifeless: no, zone is a parameter to runinstances
<mwhudson> (he says confidently, after reading the api docs)
<mwhudson> region, i think you're right
<lifeless> meh, I'll need to page in then
<mwhudson> Placement.AvailabilityZone | The Availability Zone you want to launch the instance into Type: xsd:string Default: EC2 chooses a zone for you
<mwhudson> anyway, time to wander around in the sun for a bit
 * mwhudson eods
<lifeless> now for the joy of updating LP
<poolie> that error  might have meant there were none anywhere in east1"
<poolie> that error  might have meant there were none anywhere in east1"
 * StevenK seriously ponders a patch to bin/test that makes --subunit the default if output isn't a TTY.
<poolie> ok so this does seem a _lot_ faster
<poolie> because the job i started later has finished ahead of the original version of the code i started half an hour earlier
<poolie> it may be measurement error, i can't really believe it
<wgrant> Which job?
<poolie> update-image with fs performance tweaks vs without
<wgrant> Hmm, what does update-image do that can be temporary?
<poolie> branching and setting up lp seems massively faster
<wgrant> Assuming you mean a tmpfs.
<poolie> this is with data=writeback etc
<poolie> also
<wgrant> Ahh
<wgrant> poolie: https://launchpad.net/builders/actinium
<wgrant> Building now
<poolie> ec2instance discards stdout when it runs commands on the slave
<poolie> no
<poolie> ok, stdout is out of order with stderr
<wgrant> poolie: Huh
<poolie> is possibly hard to avoid
<wgrant> poolie: maverick and oneiric built successfully 5 hours ago
<poolie> there are two pipes; they have an ordering
<poolie> we are not as smart about reading from them as a regular ssh would be
<poolie> conceptual pipes
<poolie> for some reason we have our own select/recv loop
<StevenK> lifeless: https://code.launchpad.net/~stevenk/launchpad/subunit-default-not-a-tty/+merge/82352
<lifeless> does that work?
<lifeless> I thought optparse options didn't like being written to
<lifeless> also where are you needing this?
<StevenK> Nowhere, I'm scratching an itch
<stub> overwriting optparse options works
<lifeless> StevenK: I'm just worried it may mess something up for other people
<lifeless> StevenK: is there a --non-subunit option if they want non-subunit there ?
<stub> Although you are usually better off using parser.set_defaults() *before* parsing the command line
<StevenK> I don't think so.
<StevenK> lifeless: It has seriously taken me 5 minutes, I don't mind tossing it if need be.
<lifeless> StevenK: it would be nice to have an opt-out, I think.
<lifeless> StevenK: i've been known to run non-subunit through less, for some nitty debugging situations
<stub> (branch_nick, revno) doesn't really identify the revision of a tree that was used because branch_nick is not unique. Is there a better key available? I need to record what branch+revno was used for the latest database update
<poolie> wgrant, mwhudson, something like https://code.launchpad.net/~mbp/launchpad/ec2-tmpfs/+merge/82355
<poolie> still testing it
 * wallyworld off to the vet again. poor dog :-(
<lifeless> stub: revid is a globally unique identifier
<lifeless> stub: can look that up in the history of a given branch to identify the local revno for it
<stub> lifeless: For a revision, or for the branch at that revision?
<lifeless> stub: revision
<stub> So I could, say, keep a table with every revid in the branch that has been applied to production.
<stub> Or just the lastest revid, hopefully 'good enough' (although someone cowboying the wrong way could send automation crazy)
<lifeless> so the branch has many more revids than db schema patches
<lifeless> whats the big picture here
<stub> there are more to db updates than schema patches (trusted.sql, update.py, its dependencies...)
<stub> https://bugs.launchpad.net/launchpad/+bug/845464
<_mup_> Bug #845464: deployment report should fail revisions in stable containing undeployed db changes <fastdowntime-later> <Launchpad itself:Triaged> <qa-tagger:Triaged> < https://launchpad.net/bugs/845464 >
<lifeless> stub: can we make trusted.sql into patches, and ignore update.py ?
<stub> We need to know what has been deployed. That is what is safe to merge from db-stable -> devel.
<lifeless> yes, I agree
<lifeless> uhm, yes, if you record a single 'latest rev deployed' in the prod DB, and we only deploy forwards, then you can just do a merge graph check to see what revs are safe to merge from db-stable to devel
<stub> Knowing the db patch numbers that have been applied isn't perfect, as patch numbers could conflict if people mess up, and we want the version-applied rather than a potentially modified one.
<stub> Knowing the revision number isn't perfect, as there is a chance we didn't deploy from db-stable
<lifeless> well, lets not overthink it - because the things that can go wrong are infinite :)
<stub> But we probably won't get perfect, so looking for a balance of correct and good enough
<lifeless> and we'll get cascading failures in both of those cases.
<lifeless> in the former case we'll have developers going zomg wtf, not to mention failures on staging, failures on pqm
<lifeless> (More reliably too if we drop the redundant patch info from the file - less ways for devs to mess up)
<lifeless> for the latter case, hmmm, no guarantee of cascade failures, OTOH can we automate it so we can't messs up ?
<stub> At the moment I'm thinking of lastest revid and assuming that the branch is lp:launchpad/db-stable
<lifeless> I think that would work fine
<lifeless> s/stest/test/
<poolie> hm i guess we could fold together stdout and stderr on the server and that would avoid the issue
<nigelb> mwhudson: Oooh. nice idea with using the ec2 with higher memory!
<nigelb> The shaping launchpad's next 18 months thread is full of win :)
<poolie> it sure is
<lifeless> wallyworld: wgrant: Picking two random js fluent folk; what js should we (or do we?) use in python-oops-tools? Whats the easiest way to tell? If none is in use, whats the easiest way to get up and running with YUI, (given that unlike LP the user base is captive, and having it greased-lightning fast isn't critical)
<wgrant> lifeless: It doesn't have any JS at the moment, except for the jQuery that's built into Django's admin UI.
<lifeless> so, what I want to do is take the text traceback
<lifeless> StevenK: https://code.launchpad.net/~lifeless/launchpad/oops-polish/+merge/82359 if you are still reviewing
<StevenK> I'm not, sorry.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
 * micahg hugs LP devs for the cancel build option
<wgrant> micahg: Watch out, it's permanent and non-retriable.
<micahg> understood (does it mention that on the cancel page)?
<wgrant> I don't know. I didn't realise it was publicly available until a couple of days ago.
<micahg> I've usually only wanted it in cases where I know it's going to fail and will be uploading a new version
<lifeless> wgrant: anyhow, that MP above
<lifeless> wgrant: will get us backtraces from actions in the LP timeline
<lifeless> wgrant: I want to put them in the linear query section of the oops viewing template in python-oops
<lifeless> wgrant: hidden by default
<lifeless> wgrant: with a clicky clicky thing to show it
<lifeless> wgrant: I'm not bothered about automated correlation or anything at this point
<lifeless> wgrant: I'm seeking opinionated 'do it this way its easiest' answers.
<wgrant> lifeless: I've never started a fresh YUI project before.
<wgrant> jQuery is trivial.
<micahg> ah, but the cancel button only seems to be on pending builds, but that's progress :)
<wgrant> micahg: It works on building virtual builds.
<micahg> ah, ok, I just checked a native one
<wgrant> It doesn't work on non-virtual builds, because we can't really cancel them.
<lifeless> wgrant: so, opinionated recommendation ?
<wgrant> lifeless: Work out how to use YUI, I guess.
<wgrant> I don't have a good solution.
<lifeless> hmm, but jquery is bundled
<wgrant> With Django, yes.
<lifeless> wouldn't it make sense to use one library within the one site ?
<wgrant> Because Django is hideous.
<nigelb> wgrant: So much hate for Django :(
<poolie> the ec2 ubuntu archive seems to be unreachable
<poolie> :/
<poolie> fwiw the default az is that aws chooses an az for you
<nigelb> That's probably for the best.
<nigelb> I've seen cases where aws says some zone doesn't have the machine you want.
<nigelb> *doesn't have any more of
<nigelb> That's when I realized there exists such a limit per AZ.
<poolie> i wonder if we should mirror that image in to singapore
<wgrant> Isn't Singapore significantly more expensive?
<poolie> only slightly more expensive
<poolie> i think 57c/hr vs 50
<wgrant> Ah, not so bad.
<poolie> also it may not be on fire at the moment
<wallyworld> lifeless: i've not used jQuery, but if it's bundled with Django and python-oops-tools uses that, it makes sens to me to just use jQuery
<nigelb> poolie: wait, what's on fire?
 * wallyworld now has 2 launchers running after unity crashed :-/
<poolie> i'm having a lot of trouble launching anything in us-east-1, and lots of things are failing with network glitches after they are launched
<StevenK> Heh, so effectively AWS is.
<nigelb> poolie: Now you're scaring me.
 * nigelb has workplace depending on us-east-1.
<StevenK> There is now us-east-2
<StevenK> But migrating AZs is horrid
<nigelb> poolie: AWS status says there was some DNS issues in us-east-1
<nigelb> Ugh, but that was at least 4 hours ago. and that's marked as resolved already.
<jtv-afk> StevenK, wgrant: successful package build on dogfood!  Now it says "uploading build."
<jtv> It's been "uploading build" for a long time thoughâ6 minutes!
<wgrant> jtv: Is the build upload cronjob enabled?
<jtv> Oh, that's a separate cron job?
<wgrant> Yes
<jtv> I looked at the âcrontab -lâ output for the launchpad user, but the only thing mentioning âuploadâ there is process-upload.
<wgrant> #*/5 * * * * LPCONFIG=dogfood /srv/launchpad.net/codelines/current/scripts/process-upload.py -C buildd --builds -v /srv/launchpad.net/builddmaster -a jelmer@canonical.com >> /srv/launchpad.net/dogfood-logs/upload-builds.log 2>&1
<wgrant> It's disabled.
<poolie> nigelb, ok, image building in singapore!
<nigelb> poolie: *whee*
<poolie> maybe
<nigelb> heh
<lifeless> pop quiz
<poolie> for me that is ~165ms away
<lifeless> how bad would it be to just add the tb as another table column
<nigelb> how far is us-east-1?
<poolie> 275ms
<nigelb> that's significantly faster!
<poolie> yeah
<poolie> at least if you're going to do anything interactive
<poolie> us still seems to be having network problems though
<nigelb> ah, right. I can't ping our machines from outside.
<poolie> us still seems to be having network problems though
<poolie> and so am i :)
<nigelb> Did you raise a ticket?
<poolie> i thought it was me
<lifeless> jtv: I believe you are OCR today?
<jtv> lifeless: I believe you are right.
<lifeless> I have a midsize branch https://code.launchpad.net/~lifeless/launchpad/oops-polish/+merge/82359 if you have the time
<lifeless> its just updating to newer dependencies and dealing with the fallout
<jtv> lifeless: I'll take it
<poolie> wow actually despite the smaller mean ping time that is noticeably more responsive
<lifeless> jtv: thank you
<lifeless> poolie: despite or because of ?
<jtv> wgrant: would it be okay to reproduce the command line from that cron job?
<lifeless> jtv: if there are no passwords or production paths in it, it should be fine.
<lifeless> jtv: if you're in doubt, you can paste it to me and I can venture an opinion
<jtv> That's dogfood, and I'm copying it off public IRC.  :)
<lifeless> hah :P
<wgrant> jtv: Huh?
<wgrant> jtv: Oh, you mean run it manually?
<jtv> Yes
<wgrant> Sure, or just reenable the cron job...
<jtv> Note the bit about being smacked on the head.
<jtv> \o/ successfully built
<wgrant> He's away for ages, he won't notice.
<wgrant> And there's no auditing.
<wgrant> So we just have to delete IRC logs and you're safe :)
<jtv> The secret to a happy life: impunity!
<nigelb> poolie: I just did an apt-get update on a machine and it seems to hit the mirror fine.
<poolie> lifeless, "despite the measured difference being small the subjective difference is large"
<lifeless> ah :)
<lifeless> log senses :)
<poolie> yeah
<poolie> or perhaps there is less jitter or variability or something
<poolie> nup, there's very little
<poolie> i don't know then
<poolie> error: lp.code.model.tests.test_codeimportjob.TestCodeImportJobWorkflowNewJob.test_dateDueOldPreviousResult [ multipart
<poolie> is this spurious? i did'nt break it i think
<poolie> s//i don't think i broke it
<wgrant> /date/i -> spurious
<jtv> lifeless: question about ll.132â133 of the diffâ¦ was it meant as a feature for "name1" to be able to occur twice with different values?
<jtv> (req_vars0
<jtv> *(req_vars)
<jtv> It doesn't make much sense to me, but I wonder why the test seems to want to exercise that.
<lifeless> jtv: it was a bogus concept
<lifeless> jtv: all our producers and consumers dictified it.
<jtv> Amusing.
<lifeless> jtv: the change here eliminates it
<jtv> Well, this does look a lot better.
<poolie> you have got to be ... kidding me
<lifeless> you'll see a lot of s/dict(// in the diff :)
<poolie>             counter = count(randint(0, 1000000))
<lifeless> poolie: thats in your TCP stack ?
<poolie> that's in the test factory
<poolie> eventually the same number will turn up twice
<jtv> Yowzaâthe tests even depended on ordering of req_vars.
<poolie> http://irclogs.ubuntu.com/2011/04/13/%23launchpad-yellow.html#t18:36
<nigelb> Interesting.
<lifeless> jtv: this is a long running source of low grade pain
<lifeless> jtv: I'm very very happy to finally have hit the point in this oops yak shaving exercise that addressing it was reasonable.
<wgrant> poolie: Was that your failure?
<poolie> that is giving me a very wgrant moment
<wgrant> Oh?
<poolie> who on earth counts on random integers being unique in a test
<StevenK> We do.
<wgrant> It's probably not random.
<lifeless> jtv: after this, there is https://code.launchpad.net/~lifeless/python-oops-tools/polish/+merge/82365 which shows backtraces on each SQL query in lp-oops, something that I think will help folk tracking down sources of thousand-query-death.
<wgrant> Oh, it uses getUniqueString
<wgrant> I see
<lifeless> jtv: its also pretty small and self contained.
<nigelb> "a wgrant moment" heh.
<poolie> wgrant, there may be some other explanation
<lifeless> jtv: I will be back soon - getting dinner for the family
<wgrant> We see similar sorts of things when a test runs a script but doesn't mark the DB dirty. Sequences don't get reset, so extremely confusing errors crop up a few tests later.
<poolie> but that randint call looks like a pretty big red flag
<wgrant> But in this case you're right, I think.
<poolie> i wonder why it is trying to make them unique per-thread?
<poolie> why would you want that
<wgrant> poolie: No idea.
<poolie> 'wgrant moment' meaning swearing at people
<poolie> anyhow, https://bugs.launchpad.net/launchpad/+bug/891028
<_mup_> Bug #891028: TestFactory getUniqueInteger isn't <spurious-test-failure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/891028 >
<lifeless> jtv: I am back, if that matters :)
<jtv> lifeless: I am sure you are happy to be back.  And that is a good thing, right?
<lifeless> :P
<jtv> Nice error page we have nowâ¦
<jtv> (Which is what I get instead of your other MP)
<lifeless> hmm, works fir me
<lifeless> I think you hit fdt
<jtv> Yes.  It was a nice frowney.
<mrevell> Hello
<adeuring> good monring
<jtv> morning mrevell, morning adeuring
<mrevell> Howdy howdy howdy
<poolie> hi mrevell
<lifeless> jtv: so did you have any thoughts on https://code.launchpad.net/~lifeless/python-oops-tools/polish/+merge/82365 ?
<jtv> Not yet.
<lifeless> k
<jtv> I think I'll have my standup first.
<lifeless> fair enough
<lifeless> I'm going to halt() - tl meeting in the early am
<lifeless> jtv: so if you don't get to it, its not a problem, I can nab first-up OCR tomorrow
<jtv> I should get to it anyway.  Good night!
<lifeless> nn
<poolie> mrevell, i liked curtis's karma rant
<poolie> i was going to speculate but i won't
<mrevell> poolie, Let's speculate; it's fun :) I have a reply in mind, but I want to stick to Curtis' request of "doing something" rather than just bikeshedding. Dashboards and walls are on the roadmap for next year. Karma ... let's kill it and, if we replace it with anything, let's replace it with human-awarded recognitions. "poolie thanked mrevell for xxxxx", or similar.
<poolie> i would possibly, if it gets to the top of my queue
<poolie> put up a patch that hides all mention of karma as a number
<poolie> so you just see Person has been active in Project
<poolie> or maybe "N most active people"
<mrevell> Right, so long as the people in that list are there because of the number of useful actions rather than their karma score.
<poolie> well
<poolie> i think this is complementary to, but decoupled from, having a better sense of what is useful action
<poolie> maybe if we had a really well calibrated measure of useful we'd be ok to show it as a number, but maybe not even then
<poolie> mrevell, oh the other thing as far as 'thankyou'
<poolie> would be to add some social network buttons
<poolie> i realize this will be controversial and might be a bad idea
<poolie> also might not actually be useful
<poolie> might be interesting
<wgrant> "wgrant liked this bug"?
<wgrant> That doesn't sound very useful...
<poolie> yeah one drawback is that a lot of lp is bugs
<poolie> mm
<poolie> i can imagine voting up your merge proposals, or thanking you for fixing things
<mrevell> Karma, as anything other than a list of actions, is a false representation of the value of someone's contributions. I think we can make useful judgements about what is a useful action but when we start weighting actions and so on, it leads to discontent. THere are some people who like karma and I know that turning other types of software into pseudo-games is a thing right now, but I'd rather see human recognition o
<mrevell> f good work done than a calculation that is basically broken.
<poolie> yeah
<mrevell> poolie, As for +1s and Likes, it fits in with what some people have asked for. Like you say, you'd limit it to certain actions.
<mrevell> So
<poolie> i think if i saw a person's timeline, or a project's timeline, that would give me a much better calibrated sense of their contribution
<poolie> amhnews's timeline :)
<mrevell> poolie, Right yeah
<mrevell> So, you wouldn't like a bug, you'd like that someone had reported a bug.
<mrevell> Or rather, thank them
<mrevell> in some way
<poolie> maybe thank them for fixing it
<poolie> maybe +1 a project
<mrevell> poolie, or reporting a bug, I think that's fine.
<mrevell> Yeah, +1/Like a project
<mrevell> or a patch
<mrevell> or a branch
<poolie> so
<mrevell> Hmm, interesting.
<mrevell> I can feel some LEPping coming on.
<poolie> so it seems at least interesting if not totally justifiable
<mrevell> :)
<poolie> mm
<poolie> downsides:
<poolie>  - may break things
<poolie> - people will bitch about it
<poolie> - gives personal data to $company
<poolie> - need to choose which ones to do
<poolie> - opportunity cost
<mrevell> 1. Everyone we do breaks things
<mrevell> 2. Depends who and whether they have a point.
<mrevell> 3. We should have very clear and effective opt-outs or even make it opt-in.
<mrevell> 5. Dashboards and walls are important, IMO; this is just a bit of icing that we might be able to get on top.
<mrevell> s/Everyone/Everything
<mrevell> poolie, Do you think this will break things more than usual?
<poolie> probably less
<poolie> less than changing the buildds :)
<wgrant> How will it work with our ever-expanding volume of confidential data?
<poolie> it's a really good question
<poolie> one answer is that i think we run urchin.js on all pages at the moment
<poolie> (?)
<mrevell> wgrant, I think that's the part where I get to say, "Ha, mere implementation detail" :)
<poolie> so google already know all the urls and get to run js on them
<wgrant> poolie: I believe so, yes, which is pretty awful.
<wgrant> But being open to Google Analytics compromises is one thing.
<wgrant> Most of the web is open to that.
<poolie> :)
<mrevell> I have a call shortly. I'm spending to day preparing the LP roadmap for the stakeholder meeting tomorrow. We have a skeleton Activity Walls LEP: contributions welcome: https://dev.launchpad.net/LEP/ActivityWalls
<wgrant> What do the social network buttons do?
<wgrant> I've never used one.
<poolie> so this would potentially encourage people to click "+1 woo hoo" on the Ubuntu Electric Car project
<poolie> despite it being private
<adeuring> stub: could you have a look at this MP: https://code.launchpad.net/~adeuring/launchpad/bug-sorting-by-attachment-age/+merge/82315 ?
<wgrant> poolie: That's sort of what I'm worried about.
<stub> adeuring: yup. saw that earlier - will go over it now
<poolie> i was wondering about that too
<adeuring> stub: thanks!
<poolie> perhaps a Must is that they only appear on public pages
<wgrant> I don't think it's really useful for something like LP.
<wgrant> But maybe.
<poolie> if that can be easily cheaply computed
<poolie> wgrant, they typically act as a counter of how many people clicked it, they may post it to a feed you create on the web
<wgrant> But then we're further restricting functionality by privacy, which seems sort of bad.
<poolie> they can skew google's search results
<poolie> which is i think one of the more interesting outcomes
<poolie> https://plus.google.com/105614526446850346960/posts/VZYQR5rsKUA
<poolie> the first step would be to give less stupid search snippets
<poolie> i think i'm tired
<nigelb> poolie: any luck with singapore?
<stub> adeuring: done.
<adeuring> stub: thanks!
<stub> adeuring: I can throw together the populate script after the patch is deployed if you want (for i in range(1,1000000): cur.execute(...); con.commit())
<adeuring> stub: that would be great!
<stub> Can I get the revision id of the last committed revision in my branch with the bzr command line?
 * stub finds log --show-ids
<wgrant> stub: bzr version-info --custom --template="{revision_id}"
<stub> ta - that is nicer
<poolie> nigelb, it's still building there
<poolie> i think it will be ok unless there's another glitch or bug
<poolie> mrevell, https://www.google.com/support/webmasters/bin/topic.py?hl=en&topic=1140191
<mrevell> Ah thanks poolie
<mrevell> Which Google account? martin.pool@canonical.com?
<mrevell> arg
<poolie> :)
<stub> slow but :-/
<stub> 10 seconds more to fastdowntime. Hopefully faster on real disks.
<stub> oh... don't test on a remote checkout idiot
<poolie> woo, ok, a spicy singaporean AMI!
<rick_h_> morning party people
<wgrant> Morning rick_h_
<nigelb> poolie: Yay! :)
<nigelb> (sorry, meetings suck. I was stuck in one for an hnour)
<nigelb> ~>
<wgrant> poolie: HMMMM
<wgrant> https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1559/steps/shell_6/logs/summary
<wgrant> There are a few potential suspicious revisions :(
<rick_h_> hey, I've been fighting that for the last 2 days
<rick_h_> the random generated "person-name" is always the same id number
<wgrant> Have you ever managed to reproduce it locally?
<rick_h_> no, not once
<rick_h_> always works locally
<rick_h_> tried 4 times, my bugfix branches 3 times and ran devel last night to see if maybe it was me
<rick_h_> always get the same 923594
<rick_h_> but the getUniqueInteger method should be 'random' starting point
<poolie> rick_h_, i filed a bug
<poolie> rick_h_, where are you btw?
<rick_h_> Michigan USA
<poolie> this is like the things i was fighting a few weeks ago where
<poolie> amazon seemed to have great aptitude for finding timing dependent failures
<rick_h_> I thought maybe as well, but I've done 4 runs over 2 days (3 runs yesterday) and it's always the same
<poolie> always the same test?
<wgrant> rick_h_: What's the earliest devel revno that showed the problem?
<rick_h_> I just want to land my pretty "add a css class" bugfix :)
<poolie> i wonder if it is not coincidental
<rick_h_> poolie: yes, same test, same invalid person name
<poolie> whoa
<wgrant> It may be the DB dirtiness issue that I alluded to earlier.
<rick_h_> sec, let me see if i can find my original failed test branch
<wgrant> Someone should try running the previous 100 tests or so
<wgrant> See if it's reproducible.
<wgrant> Locally, I mean.
<rick_h_> https://code.launchpad.net/~rharding/launchpad/bugfix_814696 is my branch I first got it
<rick_h_> so 14294 was where I started out at?
<wgrant> The ec2 email should say which version of devel was used.
<rick_h_> sorry, still getting back into bzr
<rick_h_> yea, I'm still searching for that original email
<poolie> add a -r option to ec2 test
<wgrant> It automatically merges devel before it starts running tests.
<rick_h_> ah, nice
<poolie> there isa  --trunk otpion
<poolie> add a -r and see if trunk sucked in the past
<wgrant> It's easy enough to bisect, but I'm hoping to find out how far back we might have to look.
<wgrant> Without running too many ec2 instances...
<poolie> rvba, does your patch seriously cut that page from 1000+ to 7 queries?
<poolie> wgrant, there's only say 200 per month
<poolie> less than 8 runs should be enough
<wgrant> That's true.
<wgrant> But anyway, hopefully gary or someone can sort this out before I get a chance to.
<rick_h_> ok, found that first test email. Source: bzr+ssh://bazaar.launchpad.net/~rharding/launchpad/bugfix_814696 r14296
<rick_h_> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r14301
<wgrant> Hmmm
<wgrant> So it wasn't the testfix.
<poolie> +sed -ie 's![ \t]\/[ \t]*ext3[ \t]*defaults[ \t]!/ ext3 data=writeback,commit=3600,async,relatime!' /etc/fstab
<poolie> isn't that just such a good idea
<poolie> don't tell elmo
<rick_h_> poolie: ooh pretty
<nigelb> poolie: elmo will probably shoot you or shoot himself.
<poolie> i hope it's actually faster after all these round trips
<nigelb> heh
<rvba> poolie: I was lunchingâ¦ what patch?
<rvba> Gosh, I've been bitten by this too: "NameAlreadyTaken: The name 'person-name-923594' is already taken."
<rick_h_> do you always get it? or is it temp?
<rvba> I just got it in an ec2 run.
<rick_h_> ah ok
<rick_h_> rvba: ping
<rvba> rick_h_: pong
<rick_h_> I'm finishing up getting that bugfix for the close button we talked about yesterday
<rvba> Great
<rick_h_> I'm wondering if you know the what for on the green "steps" bar that shows up on the edit sub link?
<rick_h_> I thought it was mostly decorative/header showing kind of thing
<rick_h_> but the docs seem to imply some sort of multi-step process or something in overlay.js?
<rvba> let me have a look.
<rick_h_> Hmm, I guess some links do have it only half across, some all the way,
<rvba> hum, this step 'thing' is new to meâ¦ let's see where it's used.
<rick_h_> https://bugs.launchpad.net/launchpad/+bug/741463
<_mup_> Bug #741463: popup diff close button misplaced <trivial> <ui> <Launchpad itself:In Progress by rharding> < https://launchpad.net/bugs/741463 >
<rick_h_> most of the ajax links on the right have it in some fashion
<rick_h_> "link to related branch" is half across
<rick_h_> subscribe someone else as well
<rick_h_> https://bugs.launchpad.net/launchpad "subscribe to bug mail" is all the way across
<rvba> weird
<rvba> rick_h_: oh I see
<rvba> rick_h_: it's a kind of weird looking progress bar for multi step stuff.
<rick_h_> right, that's what the code seemed to say, but things like that "subscribe to bug mail" didn't seem to describe 100% done of some process
<rvba> rick_h_: like when you "subscribe someone else", first you search for a name (step1) and then you pick up someone from the list you get (step2).
<rick_h_> ok, so that overlay itself has another step after submitting the form then
<rick_h_> so if I'm loading this diff, should I include it as just 100% done then? Or leave it out since there's not really a process to go through
<rick_h_> you're just loading the content
<rvba> rick_h_: I suppose it should be 100% because because there is no next step ;)
<rick_h_> ok, will go that route then
<rick_h_> thanks
<rvba> rick_h_: thank you, you made me discover something I simply never spotted before ;)
<rvba> rick_h_: did you see other pickups where the green bar is simply absent?
<rick_h_> rvba: yea, like the "this bug affects me"
<rick_h_> but that's a slightly different overlya
<rick_h_> overlay
<rvba> I suppose it's still based on the very same js code.
<rvba> rick_h_: in fact, maybe it's best to leave it out then.
<rick_h_> rvba: yea, there's a flag in the JS to show progress bar
<rick_h_> in the pretty overlay ATTRS
<rvba> rick_h_: the presence or not of the green bar does not seem to be very consistent across lp.  But it seems to be associated with "performing an action" within an overlay. Â In your instance, you don't *do* anything with the overlay, it's simply showing information.
<rvba> So yeah, on second though, I think you can leave it out.
<rick_h_> yep, I kind of wanted it since i thought it looked good, but seems to have a reason/madness if a bit inconsistant
<rick_h_> ok, I can agree with that then
<rvba> Cool.
<jcsackett> morning, all.
<rvba> Hey jcsackett.
<jcsackett> hi, rvba. :-)
<rick_h_> morning
<rick_h_> rvba: ok, branch up, put you in as requested reviewer
<rvba> rick_h_: sure, I'm in the middle of something right now but I'll get to it later.
<rick_h_> np, thanks for the help getting things figured out
<rvba> welcome.
<rick_h_> adeuring: we were talking about that ec2 test failures this morning. That's the error I was dealing with yesterday on the call
<adeuring> rick_h_: ah, thanks for the hint!
<nigelb> hah, adeuring mailed about the same failure that everyone's talking about.
<deryck> gary_poster, we have a persistent build failure affecting ec2 and buildbot runs, which seems like a timing issue in factory generated stuff....
<deryck> gary_poster, I'd ping bigjools, too, but he isn't here. :)  it seems like the kind of thing a maintenance squad should look at...
<deryck> gary_poster, given no one can pin down a change that introduced it.
<deryck> gary_poster, what do you think? :)
<gary_poster> deryck, sorry, my IRC ping failed me somehow, sorry.
<deryck> np
<deryck> I see you are aware from mail.
<gary_poster> yeah
<gary_poster> yeah, deryck, I'll dig into it or get someone else to
<gary_poster> deryck, do you know when this started?
<gary_poster> the most recent buildbot failure was the first one I've seen
<deryck> gary_poster, I know that rick_h_ was having the issue in ec2 early yesterday.
<gary_poster> deryck, ah ok, so another intermittent issue. :-/  ok, thanks
<deryck> gary_poster, well, it's not intermittent, I don't think.  Not entirely.  rick_h_ gets it consistently in every run as does adeuring.
<rick_h_> yes, I've completed 4 runs over the last 48 hours, each with the same failure and everyone has the same person id as the same failure id
<gary_poster> ah ok
<nigelb> gary_poster: If it helps, poolie ran into it all day as well I think.
<abentley> rick_h_, adeuring, deryck: Here are my current aliases, with comments about what they do: https://pastebin.canonical.com/55863/
<rick_h_> thanks abentley
<adeuring> abentley: cool, thanks
<deryck> abentley, yes, thanks!
<deryck> rick_h_, let's mumble top of the hour.  cool?
<rick_h_> sounds good
<abentley> rick_h_: Here's a snippit that can't be expressed as an alias:  bzr lp-find-proposal -r mainline:annotate:lib/lp/bugs/javascript/buglisting.js:20
<abentley> rick_h_: It means "find me the merge proposal in which landing this line was discussed".
<rick_h_> cool, going through the app auth process
<abentley> rick_h_: it needs to run in the branch the proposal was targeted against.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugtasks: 277
<bac> rick_h_: so are you seeing all fail with person-name-923594 ?
<rvba> bac: I've had the problem in ec2 a few hours ago too.
<bac> rvba: can you confirm it was with exactly the person name i listed?
<rvba> bac: yep (NameAlreadyTaken: The name 'person-name-923594' is already taken.)
<jelmer> hi bac, rvba
<jelmer> I've just had an ec2 failure with that error as well
<bac> thanks jelmer
<rvba> hi jelmer.
<jelmer> I had a quick look, but I didn't see any recent changes in that area. Has either of you looked into it further?
<rvba> not me, bac is on the case ;)
<rick_h_> bac: yes
<bac> thanks rick_h_
<rvba> rick_h_: btw I've reviewed your mp (you should have received an email about that).  I think you need to add a test for the change and then you're good to go!
<rick_h_> rvba: yep, thanks. Great feedback, appreciate it
<rick_h_> working on it now
<rvba> Great.
<allenap> abentley: Could you sanity check a script for me? https://code.launchpad.net/~allenap/launchpad/wipe-precise-translations/+merge/82422
<abentley> allenap: looking.
<allenap> Thanks.
<abentley> allenap: It looks okay, but I'm no expert on this domain.
<allenap> abentley: Thanks, awesome.
<mtaylor> it's posssible to do a bug import into a project that already has bugs, yeah?
<gary_poster> adeuring, when did you first see the person-name-923594 problem?  rick_h, are you the only one who saw the problem 48 hours ago to your knowledge?  Is there reason to suspect that it might be in a diff that you made?
<gary_poster> It's odd that everyone can reproduce it now
<gary_poster> and buildbot only just failed with it now
<gary_poster> but people report it earlier
<rick_h_> gary_poster: I was the first to see it because I was trying to run my first ec2 test Tues morning.
<rick_h_> but no change of mine has landed
<gary_poster> rick_h_ ok cool thanks
<rick_h_> and that first run was just to add a css class to an item
<gary_poster> rick_h_, that was it! ;-)
<adeuring> gary_poster: recieved a failure messsage  yesterday, 22:39UTC
<rick_h_> I just think I got lucky to start running ec2 tests at the right moment
<gary_poster> ok thanks adeuring
<gary_poster> rick_h_ you said before that you had seen the error 48 hours earlier, which would have been Monday morning.  Was it Monday or Tuesday that you first saw it?
<rick_h_> sorry, guess I started trying to run tests monday afternoon
<rick_h_> and we had the ec2 issues that you helped me through
<rick_h_> so the actual run didn't go until Tues
<adeuring> gary_poster: first line from the log of the test run: Tests started at approximately Tue, 15 Nov 2011 18:04:11 UTC
<adeuring> bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/, revision 14301
<rick_h_> I got the failure email about 2pm EST on tues
<gary_poster> rick_h_, adeuring, ok thanks, that's a big help.
<adeuring> rick_h_: the revision number from your failure could help Gray too
<adeuring> s/Gray/Gary/
<rick_h_> adeuring: yea, it was the same, 14301
<gary_poster> ok thanks
<rick_h_> Source: bzr+ssh://bazaar.launchpad.net/~rharding/launchpad/bugfix_814696 r14296
<rick_h_> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r14301
<gary_poster> yeah, 14301 then
<deryck> abentley, I filed bug 891239 after doing qa on my integration work.  Is that what your branch fix from yesterday addressed?
<_mup_> Bug #891239: CustomBugListings js error: Cannot read property 'mustache_model' <bug-columns> <Launchpad itself:Triaged> < https://launchpad.net/bugs/891239 >
<mtaylor> sinzui: how do I add a file to a question so that a losa can get at my bug import xml file again?
<sinzui> mtaylor, launchpad does not support attachments to questions
<sinzui> see bug 77123
<_mup_> Bug #77123: Attachments in answer tracker <feature> <lp-answers> <Launchpad itself:Triaged> < https://launchpad.net/bugs/77123 >
<mtaylor> sinzui: so how do I send the bug xml file?
<mtaylor> sinzui: upload and add a link?
<sinzui> mtaylor, yes
<mtaylor> sinzui: ok. cool
<mtaylor> sinzui: sweet. https://answers.launchpad.net/launchpad/+question/179006 if you get bored
<abentley> deryck: no.
<deryck> abentley, ok, thanks.  And we timed out the same.  I just confirmed running your branch it's not fixed by it.
<deryck> abentley, I'm happy to keep poking and fix it.
<abentley> It was https://bugs.launchpad.net/launchpad/+bug/890745
<_mup_> Bug #890745: dynamic bug listings don't handle next and prev correctly on first and last batch <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/890745 >
<abentley> deryck: My nets went down again.
<deryck> abentley, np.  wb.
<deryck> abentley, I've got some serious storms brewing here, so could lose power or nets, too.
<deryck> rick_h_, adeuring ^^
<abentley> deryck: Surfing on 3G currently.
<deryck> bummer
<rick_h_> ouch
<bac> hi allenap, you still here?
<bac> wallyworld: ping
<deryck> abentley, interestingly chromium fires a history change event on load that ff does not.
<abentley> deryck: Oh well, I guess we want that to support HistoryHash stuffs, too.
<rick_h_> deryck: yea, that's a pita
<rick_h_> you basically have to catch the event and check if you've got anything on the stack
<rick_h_> and edge case it
<rick_h_> there's a big discussion on trying to decide one way or another, but different browsers choose to fire that on load or not to
<deryck> rick_h_, yeah, such is the pain of browsers.
<deryck> rick_h_, I think I like FF's approach on this one.  Load really isn't a history change.
<rick_h_> yea, same here
<rick_h_> but I think the idea is that you might be able to (on a pure client side app) determine a page when loaded
<rick_h_> where normally the server takes care of handling the initial url params
<rick_h_> so you need to trigger "I'm here, and here's my first page of data load info"
<abentley> rick_h_: YUI indicates what caused the event, so it should be pretty clean.
<deryck> yeah, it just seems "load" represents that already. ;)  and "history" represents everything there after.
<rick_h_> cool, I've not used that yet, only history.js
<adeuring> deryck: my daily request to run SQL queries: https://pastebin.canonical.com/55909/
<deryck> adeuring, ok, np
<adeuring> deryck: thanks!
<deryck> np!
<adeuring> deryck: sorry, I forgot a LIMIT clause...
<adeuring> deryck: these queires please: https://pastebin.canonical.com/55910/
<adeuring> deryck: my first paste mised a LIMIT clause. please use this one:  https://pastebin.canonical.com/55910/
<deryck> adeuring, ok.  sorry I got my system locked up somehow.
<adeuring> np
<abentley> deryck: we were going to chat?
<deryck> abentley, ah, forgot.
<deryck> abentley, let's do it first thing tomorrow.  still got plenty going here.  and need to switch locations for work here shortly.
<abentley> deryck: might as well make it part of our regular Thursday call, eh?
<deryck> abentley, yeah, that works.
<deryck> adeuring, the first query is taking forever.... still going.
<deryck> oh just finished
<adeuring> deryck: ouch...
<deryck> adeuring, they're done now.
<deryck> adeuring, https://pastebin.canonical.com/55913/
<adeuring> deryck: thanks!
<deryck> adeuring, I need to switch locations and drop girls off at gym.  if you need anything else, email.  I'll hit it when I return.
<adeuring> deryck: ok
<deryck> later on, all.
<poolie> hi
<poolie> so, this time my singapore ami build completed
<poolie> let's see how it goes
<deryck> adeuring, did you need anymore queries?
<deryck> the gym has wifi :)
<adeuring> deryck: no, thanks. I'll ignore the quite slow distibution related query fow now; the others wer fast enough, I think.
<deryck> adeuring, awesome
<poolie> StevenK, thanks for name-to-service
<poolie> o/ deryck
<poolie> adeuring, thanks for the beta banner, it looks really nice
<adeuring> poolie: thanks :)
<deryck> hi poolie
<poolie> hi, how are you?
<deryck> poolie, I'm good, thanks.  How's things with you?
<deryck> abentley, here's how I decided to fix this:  http://pastebin.ubuntu.com/740645/
<deryck> oops, time to go.  later on everyone.
<flacoste> when a branch scanning job fails
<flacoste> how do we get the scanner to scan the branch again
<flacoste> for example: https://code.launchpad.net/~stub/launchpad/replication
<lifeless> push to the branch
<lifeless> even a no-op will create a new job
<abentley> flacoste: We used to lock the branch and unlock it, but now that Bazaar Experts has been disbanded, you'll need to get a losa to do that.
<flacoste> abentley: what command would they run?
<flacoste> lifeless: i can't push to the branch
<abentley> flacoste: push is probably the simplest.
<flacoste> anything that can be done without downloading the branch first?
<lifeless> flacoste: any sftp session that writes, or any bzr session that takes a write lock
<flacoste> lifeless: sorry, i'm thick, do you have any specific command suggestion :-)
<flacoste> bzr break-lock?
<lifeless> branch.lock_write(); branch.unlock()
<abentley> flacoste: Sure, 'bzr init; bzr push ~stub/launchpad/replication".  It doesn't have to work, it just has to lock.
<flacoste> abentley: ah thanks
<lifeless> abentley: still on OCR? if so - https://code.launchpad.net/~lifeless/python-oops-tools/hostname/+merge/82466
<abentley> lifeless: sorry.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<lifeless> abentley: no worries
<lifeless> uhm, who can I nab, my tz... wgrant ^
<lifeless> wgrant: 1 liner
<wgrant> lifeless: Looking
<wgrant> You ask too much.
<wgrant> lifeless: Want to add reporter while you're there?
<lifeless> wgrant: sure
<lifeless> wll
<lifeless> well
<lifeless> appinstance.name I think is the model here
<lifeless> wgrant: pushed
<wgrant> lifeless: Done
<wgrant> Thanks
<poolie> hi
<poolie> re https://bugs.launchpad.net/launchpad/+bug/891373 where is the bzr-builder people are supposed to use to test against launchpad?
<_mup_> Bug #891373: recipe help page has the wrong ppa name for bzr-builder <Launchpad itself:Triaged> < https://launchpad.net/bugs/891373 >
<poolie> in ppa:launchpda?
<poolie> not a good idea for people to add that though, since it has so much stuff
<wgrant> poolie: We've never had anything to do with that.
<wgrant> poolie: james_w/bzr handed a bzr-builder to us/IS.
#launchpad-dev 2011-11-17
<lifeless> wgrant: https://code.launchpad.net/~lifeless/python-oops-datedir-repo/bug-891251/+merge/82469
<wgrant> lifeless: Why is the umask 077 in the first place?
<lifeless> I don't know the backstory
<lifeless> we have multiple services like this though, have for ages
<wallyworld_> StevenK: FYI, in my hide bug comments mp, i've added 2 methods to Bug which i believe may be similar to what you need - userCanSetCommentVisibility and userInProjectRole
<StevenK> wallyworld_: I fear we may step on each others toes, then.
<StevenK> wallyworld_: Let me finish what I'm in the middle of, and I'll show you what I have.
<wallyworld_> ok
<wallyworld_> we should try and consolidate
<StevenK> Exactly
<StevenK> wallyworld_: http://pastebin.ubuntu.com/740741/
<wallyworld_> StevenK: so it looks like code under # Otherwise, if you're a member of the pillar owner, drivers..... is similar to my userInProjectRole(), but my method iterates over all affected pillars and also checks for security contact
 * StevenK grumbles at Unity
<StevenK> If I run indicator-weather, I want you to stay in the notification area!
<wallyworld_> StevenK: i've added new methods to IPersonRole: isBugSupervisor(obj) and isSecurityContact(obj). you could use these, plus the existing isDriber() API instead of your code
<StevenK> isDriber()? Sounds interesting.
 * StevenK ducks.
<wallyworld_> you know i can't type :-P
<StevenK> Why all affected pillars?
<wallyworld_> because it's to control who can hide a bug comment
<StevenK> Ah
<lifeless> win
<StevenK> Right
<lifeless> win win win win win
<wallyworld_> ie the user is validated against the bug, not an individual bugtask
<lifeless> 2011-11-17 05:45:37+0530 [-]     self._oops_datedir_repo = DateDirRepo(config[section_name].error_dir)
<lifeless> 2011-11-17 05:45:37+0530 [-]   File "/var/launchpad/tmp/eggs/oops_datedir_repo-0.0.12-py2.6.egg/oops_datedir_repo/repository.py", line 93, in __init__
<StevenK> wallyworld_: Right.
<lifeless> 2011-11-17 05:45:37+0530 [-]     self.metadatadir = os.path.join(self.root, 'metadata')
<lifeless> 2011-11-17 05:45:37+0530 [-]   File "/usr/lib/python2.6/posixpath.py", line 67, in join
 * StevenK kicks lifeless 
<lifeless> 2011-11-17 05:45:37+0530 [-]     elif path == '' or path.endswith('/'):
<lifeless> 2011-11-17 05:45:37+0530 [-] AttributeError: 'NoneType' object has no attribute 'endswith'
<lifeless> we have an error_dir setting of None somewhere.
<lifeless> StevenK: its 6 lines, deal
<StevenK> lifeless: Except with your internets, it takes 20 seconds per line
<lifeless> TacException: Unable to start /var/launchpad/test/daemons/poppy-sftp.tac. Content of /tmp/tmp_Nitmg/poppy-sftp.log:
<lifeless> this might explain a lot, actually.
<wallyworld_> StevenK: not sure if we want to introduce a isUserInProjectRole(pillar). your stuff doesn't require isSecurityContact(). maybe it's sufficient for you to just use the IPersonRoles stuff
<wallyworld_> StevenK: https://pastebin.canonical.com/55928/
<wallyworld_> plus there's the other existing IPersonRoles helpers
<wgrant> lifeless: We have lots of missing errordirs.
<wgrant> lifeless: You've not seen that before?
<lifeless> wgrant: it wasn't fatal before
<lifeless> wgrant: (for some bizarre reason)
<wgrant> That traceback was always there.
<lifeless> wgrant: poppy is failing to start in ec2
<lifeless> wgrant: this is the error
<wgrant> Hmmm
<wgrant> I see
<lifeless> wgrant: I dispute 'always there'
<lifeless> wgrant: what may well look similar is raising an oops barfing if errordir is None
<lifeless> wgrant: this evaluates it for sanity a little earlier, because it was simpler.
<wgrant> Right.;
<lifeless> as such I think its a net win, but it puts back my landing my oops polish branch
<lifeless> poolie: which reminds me, any word on that email oops raising critical?
<sinzui> StevenK, do you have time to mumble about goals
<StevenK> sinzui: Certainly.
 * StevenK fights with Unity on his laptop
<lifeless> how does this even work on prod
<lifeless> we've no error dir or oops prefix set for poppy.
<wgrant> poppy isn't meant to OOPS, is it?
<lifeless> hah
<wgrant> I didn't know it had anything installed to raise OOPSes.
<lifeless> SSHService
<lifeless> takes a parameter for the oops config section to use, and installs twisted top level oops glue
<lifeless> so yeah, it does
<lifeless> but it has never ever worked
<wgrant> Hah
<wgrant> Well, poppy-sftp is not yet 18 months old.
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: Me! | Critical bugtasks: 277
<lifeless> wgrant: can you please do an incremental review on https://code.launchpad.net/~lifeless/launchpad/oops-polish/+merge/82359 for me ?
<lifeless> wgrant: if you do, I can lp-land this bad boy (all the tests that failed are fixed by these commits)
 * StevenK kicks subunit for being stupid
 * subunit kicks StevenK for using me badly
<lifeless> StevenK: seriously, whats the issue?
<StevenK> lifeless: I have a stream, I want a list of failing tests only.
<wgrant> huh
<wgrant> MP diff in less than 30 seconds
<StevenK> Something must be wrong, then. :-P
<StevenK> lifeless: OCR: Me! is about as useful as [rs=me] in a commit log.
<lifeless> StevenK: if you can't figure it out you don't need a review
<huwshimi> lifeless: I have a review for you :P
<huwshimi> ok, ok, enough with the hassling
<lifeless> StevenK: | subunit-filter --no-skips | subunit-ls
<lifeless> StevenK: I seem to be giving you this recipe weekly, or something
 * wallyworld_ goes to have lunch with bigjools who is melting in the heat
<lifeless> wallyworld_: he's in.au? say hi
<lifeless> wgrant: so, incremental on https://code.launchpad.net/~lifeless/launchpad/oops-polish/+merge/82359 ?
<wallyworld_> lifeless: here's in brisbane for 2 weeks to check out schools and houses
<huwshimi> wallyworld_: Tell him to move somewhere a little further south
<lifeless> somewhere nice :P
<poolie> lifeless, yeah i might try changing the counters
<poolie> ec2 on tmpfses is doing approximately zero io
<poolie> that's nice
<wgrant> lifeless: Sure
<poolie> i haven't measured the 'before' though
<wallyworld_> huwshimi: it still gets hot down there too. adelaide for example is often hotter than brisbane
<wgrant> Yes, but it's *Adelaide*.
<wallyworld_> could be worse - could be sydney or melbourne
 * wallyworld_ runs away
<wgrant> lifeless: Why are you UTF-8ing the URL in errorlog.py?
<lifeless> wgrant: to eliminate the double escaping
<wgrant> Mmm, OK
<wgrant> Anyway, approved.
<lifeless> wgrant: the u1/isd wsgi stack gets de-%coded, de-utf8 coded urls
<mwhudson> this whole visiting countries before moving to them is very cautious!
<lifeless> wgrant: so oops tools treats a unicode url as probably garbage and immediately utf8 and % encodes it
<lifeless> wgrant: thats why our oopses look bong url wise atm
<lifeless> wgrant: I figure the only semi sane way out is to treat bytestring urls as 'from the wire' and unicode ones as 'from noone knows where'
<StevenK> lifeless: That might be hint that subunit needs better docs
<poolie> deleting 'import threading' and 'import random' from test code can only be good
<StevenK> wallyworld_: How is he melting in the heat? It's like 29 there?
<wgrant> Eep
<wgrant> regression
<wgrant> 1716 InconsistentBuildFarmJobError: Could not find all the related specific jobs.
<StevenK> Crap
<poolie> review for build failure please? https://code.launchpad.net/~mbp/launchpad/891028-notunique
<wgrant> poolie: The critical bit is probably fixed by the rollback. We'll see in half an hour or so.
<poolie> oh you rolled back everything?
<poolie> all 5 revisions or whatever
<wgrant> [Branch ~launchpad-pqm/launchpad/devel] Rev 14304: [testfix][r=bac][rollback=14301]
<poolie> it's still a bloody stupid bug, if it is what it seems to be
<rick_h_> heh, that's what I was planning on doing last night
<rick_h_> poolie: so when I brought that up in our stand up, it got brought up that there's some part of the tests that use the number for a "number of iterations" or something?
<rick_h_> hmm, well you're still count() though nvm
<lifeless> StevenK: well, TBH you're the only person that has this trouble
<poolie> if they are counting on getting particular versions they need to be shot
<lifeless> StevenK: perhaps the other users are unix diehards
<lifeless> StevenK: have you considered using testr ?
<lifeless> okies, time to vote, and due to early start, EOD too.
<lifeless> enjoy y'all
<StevenK> lifeless: Clearly, I am not smart enough to use subunit.
<thumper> can anyone give me a one liner to search for critical and high bugs using launchpadlib?
<thumper> and I don't suppose I can search in one go over both packages and projects can I?
<thumper> wgrant, poolie, StevenK: ^^?
<StevenK> I don't think searchTasks() will bend to that particular case.
<wgrant> somecontext.searchTasks(importance=['Critical', 'High']), I expect
<wgrant> It will only search over one context, or all contextys.
<StevenK> So you probably want to fetch your DSPs and then sprinkle in your products and call searchTasks over each of them
<wgrant> jtv: Can we have our builders back yet?
<thumper> wgrant: thanks
<jtv> wgrant: actually I was hoping to get you to try it for ubuntu uploads first.
<jtv> Only my mouse pointer went off somewhere and using my system's a bit hard.
<wgrant> lifeless: Are the tracebacks not going to ENOSPC us reasonably quickly?
<thumper> wgrant: how do I get a source package from a distro (ubuntu) in launchpadlib to use as a bug target?
<jtv> wgrant: any chance you could try out a bunch of concurrent ubuntu builds on dogfood?
<wgrant> thumper: lp.distributions['ubuntu'].getSourcePackage(name='some-broken-compiz-plugin')
<lifeless> wgrant: the new pruner is ready and usable
<wgrant> lifeless: Ah!
<wgrant> lifeless: That is good news.
<lifeless> wgrant: its in oops-datedir-repo 0.0.12
<poolie> lifeless, i'm having a bit of trouble understanding why you would want bugs in the help text to just not be tracked at all
<thumper> ta
<lifeless> wgrant: and I've ping in a few rt's about it :P
<wgrant> lifeless: Right, but there are like 100 RTs :)
<wgrant> And not much IS this week.
<lifeless> poolie: uhm, I may well have knee jerked.
<wgrant> jtv: Maybe.
<lifeless> poolie: the combination of not-code and critical got to me
<wgrant> jtv: Ubuntu in particular, or just non-virtual?
<lifeless> poolie: perhaps a better reaction would have been to retarget it to launchpad-moin-help-theme and priority low
<poolie> imbw, i thought broken links were critical
<poolie> it's not a bug in the theme either
<poolie> i will as you suggest justnot mention the ppa
<poolie> people can use the one on oneiric
<lifeless> poolie: oopses are critical; broken links generated from within LP oops (by design) and those are thus critical
<lifeless> \o/ mirror finally got onto multiverse
<lifeless> now I can install locally, in principle.
<wgrant> mirror?
<lifeless> I've been bootstrapping an ubuntu archive mirror for a while now
<thumper> gah
<lifeless> slow nets
<lifeless> long pipe
 * thumper foiled by unicode again
<wgrant> Oh.
<lifeless> 150kB peak rate/s
<wgrant> I was about to say, the mirror package has been around for as long as I can remember.
<poolie> jesus
<poolie> the use of randomness in tests just gets worse
<lifeless> allenap: https://bugs.launchpad.net/txlongpoll/+bug/891251 may interest you
<_mup_> Bug #891251: txlongpoll oopses are recorded with the wrong permission, causing oops loader script to fail with a permission denied error <Python OOPS Date-dir repository:Fix Released by lifeless> <txlongpoll:Triaged> < https://launchpad.net/bugs/891251 >
<jtv> wgrant: in principle, we want to exercise all kinds of job we have.  I just don't feel we can realistically do it.  I think I'll try TTBJs though.
<poolie> lifeless, thanks for the review of the randomness patch
<poolie> it turns out the testtools one will not quite fit
<lifeless> ah well
<poolie> it's good to know it's there though
<jtv> wgrant: I think it would make sense to do some ubuntu uploads on dogfood, in case it exercises other code paths that write to the database without having been permitted to do so.  Once my branch lands, there's a good chance that we'll find more of these.  We should minimize that risk though.
<wgrant> jtv: You can't throw your 10 zillion hellos at Ubuntu?
<jtv> Can I?
<wgrant> Why not?
<jtv> Indeed, why not?
<jtv> wgrant: what location should I feed to dput?  My âincomingâ setting is â%(dogfood)sâ.
<wgrant> jtv: dogfood:ubuntu
<jtv> Thanks.
<wgrant> Or dogfood:ubuntu/oneiric to override the series
<poolie> it's kind of interesting how the lp test suite doesn't use a full cpu even when it's not doing io
<poolie> i don't know what it is doing
<jtv> poolie: sledgehammer trick â ctrl-C it and look at the backtrace.
<poolie> mm i know
<jtv> My apologies.
<jtv> I guess we're not doing enough context switches (yet) to make that the cause.
<poolie> it's ok
<jtv> I mean, to make context switches the cause of such symptoms.
<poolie> we might be
<poolie> there is a lot of interprocess io
<poolie> mwhudson, still here? could you read https://code.launchpad.net/~mbp/launchpad/ec2-region/+merge/82487
<mwhudson> poolie: looking
 * mwhudson goes cross-eyed
<mwhudson> time to try meld
<poolie> :/
<jtv> poolie: I'd be quite interested to learn how you see the interprocess I/O going on, incidentally!
<poolie> strace
<poolie> lots of reading and writing on sockets
<jtv> Ah   :)
<poolie> i guess talking to the db
<jtv> Does strace show the difference between a socket and a file?
<mwhudson> there is also iotop i guess?
 * jtv runs that
<jtv> Fun, thanks!
<poolie> not directly
<poolie> though if you see it previously polling on the fd, or things like that, it's a good clue
<jtv> True.
<jtv> There's also the "wa" column in "top," but that's only a global number.
<StevenK> poolie: Multiverse is required
<poolie> StevenK, it's added by the sed line a bit higher up
<poolie> in a way that does not hardcode the region
<StevenK> Right
<jtv> wgrant: where do I observe my uploads to ubuntu?  /ubuntu/+builds?  Or is there more?
<StevenK> /ubuntu/+source/<spn>
<jtv> Thanks.
<poolie> StevenK, is that what you were asking about?
<StevenK> poolie: Yes, but your branch still makes me ... uneasy
<poolie> cause it's changing something dangerous?
<StevenK> Because it's changing something that isn't well tested.
<poolie> yeah i know
<StevenK> And ec2 issues are hard to debug
<poolie> much like testing lplib clients, it's hard to test it without a whole server cluster on the other end
<poolie> fwiw this makes several failure modes easier to understand
<StevenK> Which isn't helped by the ec2 test runner going "Hrm, can't work out what is going on. sudo halt"
<poolie> yeah
<poolie> we could easily have an option to not do that
<mwhudson> poolie: surely you can delete _convert_instance_type now?
<poolie> fwiw i've run it several times here with no unfixed errors
<poolie> done
 * mwhudson is reminded that, now we don't do the private dependency thing, ec2 test could detach plenty quicker
<poolie> that would be nice
<poolie> especially when travelling
<StevenK> mwhudson: Fix eet?
<poolie> StevenK, i'm kind of glad i broke my personal ice of finding this too scary to change
<mwhudson> StevenK: not today
<mwhudson> poolie: \o/
<wgrant> Well, there are several tests or
<wgrant> functions reached from tests that reset the random seed, for instance
<wgrant> test_token_creation just sets it flat out to zero
<wgrant> What
<mwhudson> it's not that bad code
<wgrant> seriously?
<poolie> "wgrant moment"
<wgrant> Heh
<mwhudson> !!!!!!
<poolie> someone from ubuntu will tut me if i write down here what i actually said
<poolie> "i want something kinda random but not actually random"
<mwhudson> the reason for the randomness iirc was to stop people putting 'person-name15' in doctests
<mwhudson> :(
<poolie> haha
<wgrant> Yeah :(
<poolie> it now starts at 100,000 and counts up across all tests
<StevenK> Which didn't actually work
<poolie> people will realize pretty soon it's not predictable
<poolie> :/
<StevenK> Since people started doing factory.makePerson(name='fred')
<poolie> i hope they don't count on it being 6 digits but we can only do so much
<StevenK> TBH, I write my tests to not care. I just want an IPerson.
<mwhudson> StevenK: that's ok
<jtv> wgrant: process-upload.py doesn't like âdogfood:ubuntuâ as my dput upload path.  It says: Could not find distribution 'dogfood:ubuntu'.
<mwhudson> if you care about the name, you can care about the name
<mwhudson> you just have to be explicit about it
<wgrant> jtv: dogfood:ubuntu, not dogfood:dogfood:ubuntu
<poolie> so can i get an upvote?
<poolie> https://code.launchpad.net/~mbp/launchpad/891028-notunique/+merge/82480
<nigelb> DId you guys fix the random thing not being actually random? :)
<jtv> wgrant: whoops, you're right.  Sorry.
<poolie> i think doing random.seed(0) at the start of the tests is arguably a good idea
<mwhudson> poolie: approved your branch btw
<poolie> let's not pretend
<poolie> thanks
<mwhudson> poolie: yeah, seed dependent failures are bad
<poolie> nigelb, see above
<nigelb> \o/
<nigelb> Nice.
 * nigelb reads the mail thread as well
<nigelb> How did these tests work until this week.
<poolie> there used to be say 2500 getUnique calls before you reached this test
<poolie> now there are 2501, and that causes the counters to collide
<poolie> or 2512312 and 2512313
<rick_h_> lol, damn +1
<poolie> not necessarily exactly one more
<nigelb> oh. wow.
<poolie> but the patterns aligned
<poolie> i'm not totally sure but it looks a lot like that
<mwhudson> poolie: approved your other branch too
<poolie> i have to confess i have not run its own tests locally yet
<poolie> for ec2
<wallyworld_> lifeless: oops raising question for you. attempting to delete the last bug task raises a CannotDeleteBugtask exception and oops in browser. this doesn't normally happen except if two users are involved and step on each others' toes. i am going to fix the web ui to display a nice message. if i catch the exception in the view code, will that still generate an oops on the server side? do we care?
<wgrant> wallyworld_: If you catch and don't reraise, it won't log an oops.
<lifeless> the view code is server side
<wgrant> Only unhandled exceptiongs log oopses.
<lifeless> if you catch it, its handled, fin.
<wallyworld_> lifeless: cool. i was hoping that would be the case, just wanted to double check. thanks
<wallyworld_> wgrant: thanks to you too
<jtv> wgrant: found out why my builds aren't getting through â I have no upload rights to Ubuntu Main.
<jtv> Anything simple I can poke in the database to change that?
<wgrant> Add yourself to ubuntu-core-dev
 * jtv fiddles
<huwshimi> wallyworld_: Following up on yesterday, I'm just now taking a look, and not quite sure how I override the io request for the test (in https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689)
 * wallyworld_ looks
<huwshimi> wallyworld_: It's trying to get a URL for the request from the HTML
<huwshimi> wallyworld_: I can pass in the config, but what I really need to do is pass it the response
<huwshimi> wallyworld_: I might be missing something here
<wallyworld_> huwshimi: so you want to force io_provider to be an instance of MockIO so that your test can check the correct url is being used in the get and also to allow you to poke in a return value
<wallyworld_> so in the call to setup_taglist(config), you make config = {io_provider: your_mockio_instance}
<wallyworld_> your test will call setup_taglist
<wallyworld_> once setup_taglist is called by your test, then you poke in the return value
<wallyworld_> by calling a method on the mockio instance
<poolie> are there any objections to me landing the deletion of lib/canonical/buildd?
<poolie> it will then only be an external dependency
<wgrant> If the tests pass, go ahead/
<poolie> we have already deployed from the external tree to all the buildds
<poolie> thanks
<wallyworld_> huwshimi: see for an example get_intensity_listing in test_buglisting.js (there are many others also)
<wallyworld_> huwshimi: ListingNavigator is constructed with a mockio instance
<wallyworld_> huwshimi: thwn later, mock_io.last_request.successJSON(xxxx) is called to simulate a value sent back from the xhr call
<wallyworld_> clear as mud?
<huwshimi> wallyworld_: hmmm
<wallyworld_> is that a good hmmm or a bad hmmm
<huwshimi> wallyworld_: Sorry I'm dealing with too many things at the moment, it's a slightly unsure hmmm, I'll have a proper look in a sec
<wallyworld_> np. take your time
<jtv> wgrant: I've got some uploads, which process-upload.py said it set to UNAPPROVED.  What comes next?
<huwshimi> wallyworld_: Oh, so I need to call the function that would be called on io:success manually?
<wgrant> jtv: Approve them :)
<jtv> How?
<wgrant> Add yourself to ubuntu-archive
 * jtv fiddles
<wgrant> And go to https://launchpad.net/ubuntu/someseries/+queue
<wgrant> +dogfood
<wallyworld_> huwshimi: no, that is done when you poke a successful response back into the mock io provider using for example successJSON(xxx)
<wallyworld_> huwshimi: you can also pass in data simulating an error response
<huwshimi> wallyworld_: Ah I see
<wallyworld_> huwshimi: so, if you factor out the success handler to it's own method, you can do stuff like test that method separately without and io involved at all
<wallyworld_> just call it from your test with data and see that the DOM is manipulated as expected
<wallyworld_> there's a few ways to skin the proverbial cat
<wallyworld_> huwshimi: more examples in test_bugtask_delete.js
<huwshimi> wallyworld_: Yeah, was looking at that
<huwshimi> wallyworld_: I think I'm understanding now
<wallyworld_> :-)
<huwshimi> wallyworld_: I didn't realise that setting the mockio.success() would call "io:success"
<wallyworld_> ah, right. it's handy like that.
<wallyworld_> it allows you to simulate a server request -> response without getting into all the io glue
<wgrant> poolie: Did you break lp-buildd?
<wgrant> make: dh_lintian: Command not found
<wgrant> Trying to build 100 in hardy
<huwshimi> wallyworld_: Is there a way to make the log console more verbose? It's telling me about an error, but doesn't tell me where
<wallyworld_> huwshimi: you looking at the log in the page or the browser console (F12 in firefox)
<wallyworld_> ?
<poolie> wgrant, i hope i didn't break it
<poolie> where did that happen?
<wgrant> https://launchpad.net/~launchpad/+archive/ppa/+build/2932600
<wallyworld_> huwshimi: i thing firebug's browser log may be more verbose
<wgrant> Maybe it was jelmer(?)'s cleanups.
<wallyworld_> huwshimi: other than that, it's debug js time or add Y.log()s
<huwshimi> wallyworld_: Hmm.. that just shows me an error in yui.js
<wallyworld_> huwshimi: ah, you haven't included all dependencies in the html file
<wallyworld_> you need to include each lp js file required, one by one
<wallyworld_> that'
<wallyworld_> s most likely what's wrong
<poolie> wgrant, i don't think i've landed any changes to it since the last deployment
<wallyworld_> if you look at an existing html file, you'll see a block for including all the yui stuff, then each lp file included under that
<huwshimi> wallyworld_: Ah I see
<poolie> i have one outstanding change up for review
<huwshimi> wallyworld_: Thanks, I was thinking, but I've added it to use(), but we don't have a combo loader :)
<wallyworld_> yep :-)
<huwshimi> wallyworld_: :(
<poolie> wgrant, yes, jelmer changed launchpad-buildd to run dh_lintian during its own build
<wallyworld_> not yet anyway. it keeps getting talked about from time to time
<poolie> i guess he assumed it was in build essential or something
<StevenK> It's in debhelper, just the version in hardy is too old
<StevenK> Check if dh_lintian is in $PATH in debian/rules, or backport debhelper
<poolie> k
<poolie> i have to go out for a bit
<poolie> please file a bug or fix it or both
 * StevenK is down to 7 test failures, from 132.
<poolie> running on tmpfs is 10 minutes, 5% faster
 * wallyworld_ goes to get the kid from school
<StevenK> At 5pm?
<StevenK> Oh, bloody daylight savings.
<wgrant> lol qld
<nigelb> I thought australia was sane about DST?
<wgrant> No DST is sane.
<StevenK> No
<wgrant> But we are a bit special.
<huwshimi> it's getting better
<wgrant> The eastern states are all in the same timezone, except that Queensland doesn't do DST.
<wgrant> So they stay an hour behind.
<StevenK> WA no longer does DST
<wgrant> And occasionally one of the other states will decide to change their DST rules one year because why not and what could go wrong.
<StevenK> I'm not sure about SA and NT
<wgrant> And then the WA government will announce with two weeks' notice that they're doing DST.
<wgrant> And then abolish it again the next year.
<huwshimi> nigelb: So no, not really :)
<StevenK> Currently, New South Wales, Victoria, Tasmania, Australian Capital Territory and South Australia apply DST each year, from the first Sunday in October to the first Sunday in April. The Northern Territory, Queensland and Western Australia do not observe DST.
<wgrant> With notable exceptions.
<StevenK> In December 2008, the Daylight Saving for South East Queensland (DS4SEQ) political party was officially registered, advocating the implementation of a dual-time zone arrangement for Daylight Saving in South East Queensland while the rest of the state maintains standard time.
<StevenK> Oh, sure. That won't be confusing at *all*
<wgrant> "On 12 April 2007, New South Wales, Victoria, Tasmania and the Australian Capital Territory agreed to common starting and finishing dates for DST."
<wgrant> IIRC there were changes due to the Commonwealth Games in '06
<wgrant> And probably the '00 Olympics as well.
<StevenK> That sounds right
<wgrant> But they were well-announced.
<wgrant> Unlike WA's stupidity.
<StevenK> There certainly were for the '00 Olympics
<StevenK> We had another two weeks of DST
<wgrant> I don't recall.
<StevenK> I think it was only NSW.
<StevenK> Because lolympics
<wgrant> Nah, probably applied everywhere, but I don't remember.
<StevenK> Well, you were 8
<wgrant> 9
<StevenK> Damn, I was close
<wgrant> Yeah, all the AEDT-observing states and territories were affected.
<StevenK> Ah
<StevenK> Heck, I was out of high school in '00
<wgrant> Yes, but you're prehistoric.
<StevenK> Back in my day, people respected their elders.
<huwshimi> wgrant: I'm not sure about prehistoric, maybe more like carefully not remembered
<nigelb> Fun!
<huwshimi> So, I have a js module that works fine except when I load it to be tested it can't access Y.lp.client. Any suggestions for what obvious bit of info I'm overlooking?
<huwshimi> I may have to provide some more info....
<huwshimi> ugh, nevermind
<huwshimi> How is it helpful for NodeList.hasClass('foo') to return a list of booleans?
<huwshimi> Frustration.
<wgrant> What else would it do?
<huwshimi> wgrant: Well it would be nice to provide a way to actually get the nodes
<huwshimi> instead of "[false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, true, true]"
<huwshimi> wgrant: I'm just not sure what situation getting an array like that would be useful for
<wgrant> huwshimi: .all(".foo") doesn't work?
<huwshimi> wgrant: Well I can just do .filter('.foo')
<huwshimi> wgrant: I'm just confused by the usefulness of .hasClass on a nodelist
<wgrant> Or that.
<huwshimi> or lack of
<wgrant> Well, you could conceivably want to check.
<wgrant> It's the logical thing for hasClass to do
<jtv> wgrant: question about findBuildsByArchitectureâ¦ it only returns cases where BPPH.architecturetag == BPB.architecturetag.  Doesn't that exclude arch-all for all but the nominated architecture?  If so, is that intentional?
<wgrant> jtv: Where's that?
<wgrant> I haven't seen that API before...
<jtv> Sorry.  SPR.findBuildsByArchitecture
<wgrant> Huh, didn't know that existed.
<wgrant> ... and indeed it doesn't.
<jtv> Argh.  I meant, getBuildByArch.
<wgrant> Ah.
<wgrant> I know that well enough to realise that I probably don't want to dig into it just before dinner.
<wgrant> But let's see...
<wgrant> jtv: It's meant to return the build for that arch.
<wgrant> As the name suggests.
<wgrant> Arch-indep binaries from the wrong architecture aren't useful in the search for such a build.
<jtv> Yesâ¦ just wondering if the implication for arch-indep builds is intentional.
<wgrant> Because they are published on all arches, from a single build.
<jtv> Right.
<wgrant> There aren't arch-indep builds.
<wgrant> There are i386 builds that only build arch-indep binaries.
<wgrant> But they're still i386 builds.
<jtv> Right, that's the ones I mean.
<jtv> If that's intentional, that answers my question.  Thanks.
<jtv> There's an annoying timeout on that query when approving uploads.  Turns out to be ridiculously faster to query all architectures at once.
<jtv> Even without re-using the result across architectures.
<wgrant> Oh yes, createMissingBuilds is a horrible person.
<jtv> Goes from 0.8s to 0.3ms.
<jtv> Repeated for every architecture in the series.  The 0.3ms version would make even that repetition easy to eliminateâbut why bother?
<wgrant> The whole thing needs to be rewritten.
<wgrant> It's crap.
<wgrant> I never got around to it when I was on maintenance.
<wgrant> But it's a clear problem for both copies and queue operations.
<wgrant> It got a bit simpler with DDs.
<jtv> I'm beginning to understand the appeal of Gentoo.  :-)
<wgrant> Heh
<jtv> âYeah, whatever, we'll just exaggerate the speed advantage of a native compile and let the users take care of the building.â
<nigelb> Oh dear.
<wgrant> Would make things *so* much easier.
<jtv> For the distribution side.  Not necessarily for the users.  :-)
<wgrant> The archive management side is all I care about :)
<nigelb> wgrant: traitor :P
<rvba> Hi lifeless, are you around?
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
<jtv> wgrant: I have my approved uploads, but how do I set the build farm to work on them?
<allenap> lifeless: I'll put that txlongpoll bug on the board. I have too much WIP right now as it is, but I'll try and work on it next.
<lifeless> allenap: thanks; it should be trivial, I didn't do the txlongpoll part because I figure its in active fiddling from you guys atm
<lifeless> rvba: no, but whats up ?
<wgrant> jtv: They'll build automatically.
<jtv> Even on dogfood?
<rvba> lifeless: ;) â¦ In the mean time, jtv helped me and I got my problem sorted. We 'discovered' that having a cache size of 100 is a little bit small.
<wgrant> jtv: Yes.
<lifeless> rvba: the dev cache? I thought wgrant was landing a bump of that to 10L
<jtv> wgrant: curses.
<lifeless> rvba: we should make it match prod IMO
<wgrant> jtv: Oh?
<rvba> lifeless: yes, the dev cache, and no
<wgrant> lifeless: Mmm, I decided I didn't have enough reason to do it.
<wgrant> lifeless: It is somewhat helpful possibly.
<jtv> No need to make it match production.  Plenty of objects will be disposable anyway.
<wgrant> jtv: Ah, you don't have any non-virt builders...
<lifeless> wgrant: future request; please let folk you've discussed things with know of plan changes :)
<jtv> wgrant: the company won't give me more hardware!
<wgrant> lifeless: I didn't think it was a plan.
<wgrant> More of a thought.
<lifeless> wgrant: ah, I thought it was a definite
<lifeless> we see about one headscratch a week for scaling tests where the dev/testrunner cache size is too small
<rvba> true ;)
<lifeless> I need to dig around and see how to switch the cache to non-evict mode for appservers.
<lifeless> it would save some CPU cycles and avoid this sort of problem
<lifeless> but in the mean time - +1 on anyone changing it to match, or approximate, production.
<jtv> lifeless: non-evict mode is StupidCache, which is what we used to have.  They required regular restarts.
<lifeless> jtv: given we discard the entire cache between requests, I'm at a loss for why that would correlate with restarts being needed
<lifeless> jtv: how did we decide it was the storm cache ?
<jtv> lifeless: oh, you mean the appserver cache _in tests_?
<lifeless> jtv: no, in prod
<jtv> lifeless: rvba tried it.
 * jtv â otp
<lifeless> in prod, we throw away the storm cache between requests
<lifeless> What I'm actually worried about is performance tanking when we start evicting and lazy reloading objects.
<lifeless> so a cache that does not evict, but still has an upper cap, would address my concerns.
<lifeless> AIUI thats not the same as stupidcache
<allenap> lifeless: That /could/ have similar problems, just with a different subset of retrieved objects (i.e. the most recently fetched ones). But at least it's not spending time evicting stuff that's already cached.
<jtv> lifeless: completely forgot that we discard the cache between requestsâ¦
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 277
<jtv> I don't see how you could usefully get a limit on cache size without eviction.  The StupidCache does neither; GenerationalCache does both but at a very low cost.
<rvba> allenap: can you have a "second look" at my MP please ;) https://code.launchpad.net/~rvb/launchpad/activereviews-bug-867941/+merge/82375
<allenap> rvba: Sure.
<jtv> lifeless: GenerationalCache detects that it gets too large, and drops a single large bucket of up to half the cached objects.
<jtv> (The most recently used half-or-more objects stay cached)
<lifeless> allenap: I am suggesting that it error when the cache would be full
<lifeless> allenap: that would make it fail hard, which we can then review why it exceeded 10K live objects (or whatever)
<lifeless> jtv: ^
<allenap> lifeless: Okay, that's interesting. I like that.
<jtv> lifeless: IIRC GenerationalCache has a separate âI'm full, do something about itâ method that you could override.
<lifeless> jtv: that might be an easy route to implementation - thanks
<jtv> Obviously I'd like to see us continue to use the generational cache.  :)
<lifeless> why?
<lifeless> [not trolling] - its an implementation detail, right? Is there something particularly good about it other than it choosing to discard things rather than thrash ?
<jtv> lifeless: simply because I designed it specifically for our production use-case (as well as the tiny-cache use-case Storm was written for at the time) and I hate to see that work no longer be useful to us.  :)
<jtv> At the time we kept all objects in cache forever.
<jtv> Of course there's also scripts, where it's still useful.
<lifeless> ah :)
<lifeless> so, scripts are different, I haven't got a view on their needs yet - not one I could articulate or argue around
<jtv> I'm not sure if we cleaned out all the nasty manual cache-clearing code in our scripts, although we did clean out some.
<jtv> Some scripts are definitely best off with the generational cache: access too much data for StupidCache, working sets too large for the old LRUCache (not sure Storm even kept that oneâI saw one script spend only infinitesimal time outside of the LRU accounting code), and lots of data that can be forgotten soon after use.
<jtv> The enormous LRU overhead only happened in an experiment on a test server, of course; the default LRU cache was just tiny.
<stub> Should we hard fail if the cache 'fills up', or generate a soft OOPS?
<stub> 'fills up' in quotes given the current 10k limit is a number I pulled out of my arse a few years ago
<jtv> allenap: rvba: more succinct version of what I said earlier.  :)   ^^^
<rvba> jtv: indeed ;)
<jtv> wgrant: maybe I do need to mark some builders as nonvirtual for this..?
<lifeless> stub: soft oops is another option; something at least, rather than degrading silently and (probably) heavily
<wgrant> jtv: Yes.
<jtv> Will I kill anyone?
<jtv> lifeless: there may even be separate actions you can override for âdemote current generationâ (i.e. âconfigured cache size exceededâ) and âretire old generationâ (i.e. âthis is not the first time I had to revolve generationsâ).  But maybe that's a bit arbitrary, since there can be any amount of overlap between the generations.
<jtv> Think of them as equally-sized L1 and L2 caches, give or take.
<jtv> wgrant: and in case you're interestedâ¦ https://code.launchpad.net/~jtv/launchpad/bug-891493/+merge/82514
<wgrant> jtv: Will review after the downtime. Sounds really good, though.
<jtv> Thanks.
<jtv> wgrant: still no joy on the buildfarm.  :(
<jtv> Oh don't tell me it's simply not running againâ¦
<jtv> Nope, nope, it's running.
<wgrant> I just checked that.
<wgrant> Nothing in the logs for nearly 24 hours.
<wgrant> Restart it?
<jtv> Yeah, I guess
<wgrant> That worked.
<jtv> Exciting.
<rvba> lifeless: maybe you will want to give your opinion about the possible fix suggested on https://bugs.launchpad.net/launchpad/+bug/867941
<wgrant> jtv: Approved with a comment.
<jtv> Thank you!
<wgrant> rvba: I wouldn't optimise this too hard now -- the privacy schema is in the middle of changing entirely.
<rvba> wgrant: but the main problem does not come from the private branches (even if I know that it might look like it does)
<rvba> The huge number of public branches is the problem here.
<rvba> Seq scanning the public branches 4 times in one query without any optimisation is the problem. (see the original query)
<wgrant> Hmm.
<stub> rvba: You should be able to create temporary tables on the slave dbs just fine.
<rvba> stub: oh really?
<rvba> Thanks for looking into that stub.
<rvba> jtv: â
<stub> rvba: (at least on production. You might get an error with the test suite in case we are being overly paranoid?)
<jtv> stub: I tried creating one in a read-only transaction and that wasn't allowed.
<jtv> That is the only basis for my conclusion that it won't work on a slave, so disprove that and I stand corrected.
<stub> jtv: Ok. So that is just in our test suite. This might be a use case for turning that behaviour off or adjusting it.
<stub> jtv: I didn't think of temporary tables when I implemented it.
<jtv> I tried it in psql, actually.
<stub> Right. We don't use read only transactions when talking to the slaves. There are triggers on the relevant tables that block updates.
<stub> And if we do, we can fix that.
<stub> I am surprised that building a temporary table and an index is faster than embedding it as a CTE.
<stub> 1.2s is still high, but if it is an improvement its a step in the right direction.
<rvba> jtv's proposition is a serious improvement over the CTE version.
<jtv> It's a massive improvement.
<jtv> Unfortunately.
<stub> yer. I was assuming pg would be smart enough to build any necessary indexes in RAM to make queries faster.
<jtv> I didn't check whether the speedup was a consequence of the index or of the plan.
<rvba> I /think/ the speedup is explained line 20 21 (https://pastebin.canonical.com/55836/) but I'm not sure what it means exactly.
<jtv> rvba: do you happen to have the CTE version handy?
<stub> I'm braindead now and not looking at query plans :-)
<rvba> jtv: https://pastebin.canonical.com/55835/
<jtv> rvba: I meant the *plan* for the CTE version, sorry
<rvba> jtv: hang on
 * jtv hangs on
<rvba> jtv: https://pastebin.canonical.com/55780/
<rvba> jtv: 2nd result
<stub> https://pastebin.canonical.com/55945/
<jtv> Arghâwhere does that formatting come from?
<rvba> stub: this is just another run of the CTE version?
<stub> https://pastebin.canonical.com/55946/ (had \x on sorry)
<stub> rvba: yer, from prod
<jtv> I wonder what a CTE Scan node really does.
<rvba> Total runtime: 6029.359 ms ("my" run) != Total runtime: 1817.746 ms (your run)
<rvba> Crazy.
<jtv> That could be those buffer sizes I mentioned.
<rvba> stub: can data hotness or something like this explain such a difference?
<stub> rvba: Where are you running the query?
<stub> rvba: first time was 2s
<rvba> I asked Tom H to run it on the prod db.
<rick_h_> rvba: thanks for the MP review
<rvba> FTR the original query (without a CTE) is ~4.5 s (this is from oops reports)
<rvba> rick_h_: you're welcome.
<rvba> stub: so the CTE is a serious improvement after all.
<stub> rvba: If it is random, could be conflicts with something like the branch scanner making updates
<stub> rvba: I expect the CTE will perform the same as a temporary table for a single query, and a temporary table will win if you can reuse if for subsequent queries.
<stub> rvba: But I haven't investigated much - this is just my assumption
<rvba> stub: Right, but I see very different things in the 2 query plans:
<rvba> CTE Scan on visible_branches source_visible_branches  (cost=0.00..9412.66 rows=470633 width=4) (actual time=0.001..971.882 rows=465499 loops=1)
<rvba> That's from the CTE version (obviously)
<rvba> Index Scan using visible_branches_ids on visible_branches  (cost=0.00..4.00 rows=1 width=4) (actual time=0.017..0.017 rows=1 loops=86)
<rvba> And that's from the other version (with a temporary table)
<rvba> The difference in the number of returned rows puzzles meâ¦ maybe there is an optimization that the index allows somewhereâ¦
<stub> rvba: yer. Adding the index is a wild card. I would expect that what you gain from being able to use the index is outweighed by the cost of building the index, but again this is just my assumption.
<jtv> I think that's the number of rows it inspects, rather than the number of rows it returns.
<rvba> Ah, that might explain the differenceâ¦ and the speedup.
<jtv> I think the CTE Scan is effectively a HOT table scan.
<rvba> stub: jtv said the creating the index was instantaneous.
<rvba> s/the/that/
<jtv> (HOT standing for Heap-Only Tuples, tuple allocations that never need to be written to diskâI /think/ those would be used here but am not sure)
<rvba> It would be nice if we could create the index in the CTE.
<jtv> Quite.
<jtv> I suppose nobody figured a single query would re-use the CTE's contents quite this much.
<stub> You could move the join with the CTE outside of the UNION, joining twice instead of four times. I think. Brainfried - I can look at it tomorrow if someone emails me the queries etc.
<stub> That might win for temp table too
<rvba> I suppose we want to avoid using temp tables if we can use CTE instead.
<stub> much of a muchness
<jtv> Yes.
<jtv> In terms of code maintenance, I think CTEs are much to be preferred.
<rvba> stub: I'll see if that possibleâ¦ but the code that builds this query if pretty crazy to say the least ;)
<stub> It would be nice to rewrite the query so it doesn't need to materialize a big list of all public and private-but-visable branches
<stub> ie. select all branchmergeproposals where the branch is not transitively private, UNION ALL branchmergeproopsals where the branch is transitively private but visible
<stub> Just because slinging around 400k resultsets is always going to be slow
<rvba> Right.
<stub> Anyway... done for now. Email me with details if you want me to look at it tomorrow, or have fun :-)
<rvba> Again, I'll have to see if that's feasible without a massive refactoring but it's worth having a look.
<rvba> stub: jtv thanks a lot for looking into that.
<jtv> np
<rvba> I was misguided by my unlucky run :(
<stub> Also, I suspect we see some timing issues due to branchscanner sometimes locking things for longer than we would like. There are caches on the Branch table it maintains.
<rvba> I've added our recent findings to the bug as comments.
<timrc> is there an agenda for the Launchpad Thunderdome? This page looks a little neglected, https://wiki.canonical.com/Launchpad/Sprints/ThunderdomeJan2012/Agenda
<allenap> matsubara: Hey there. Fancy reviewing a python-oops-tools branch? https://code.launchpad.net/~allenap/python-oops-tools/use-psycopg2-2.4.1/+merge/82537
<matsubara> allenap, sure, waiting for the diff to update
<allenap> It's taking an age :-/
<matsubara> allenap, r=me
<matsubara> thanks for the fix!
<allenap> matsubara: Thanks :)
<sinzui> timrc, you found the correct page. We have not decided what the theme of work will be
<timrc> sinzui, there was some off discussion about  moving some of Ubuntu's image building into LP... that topic is of particular interest to us.  If that makes it on the agenda, we'd like to know
<timrc> s/off//
<sinzui> timrc, no. Our thunderdomes usually focus of tech-debt. fixing common issues that requires a lot of contribution.
<timrc> sinzui, ok, good to know
<sinzui> timrc, We have a poor history of completing feature work that starts in an epic. Well we do, but it takes months because the work was not owned by a team that had a date to deliver it.
<deryck> Morning, all.
<rick_h_> morning
<deryck> The week of buildnot has made what's left of my hair turn even more gray.
<rick_h_> hah, it's called "distinguished"
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugtasks: 277
<sinzui> I was just thinking I should strip my hair, then when colour grows in I might not look so old
<rick_h_> I just shave it off, saves on the hair cuts
 * deryck is going for nearly head shaved next round, too
<jcsackett> allenap: you waiting for someone specific to look at your precise wiping script, or just haven't found a a reviewer yet?
<sinzui> I do not like the thought of having more hair on my body than on top of my head.
<allenap> jcsackett: No, Aaron had a look yesterday. I'm never going to land it, so he didn't vote. I'll move it out of the review queue.
<allenap> Thanks though Jon :)
<jcsackett> allenap: no need to thank me for *not* having to review something, but you're welcome. :-P
<deryck> So is anyone looking into buildnot?  I think the last email blames flacoste and lifeless.
<timrc> sinzui, as a user, I would vote for fixing stability issues and doing tactical things to the UI to make it more enjoyable to use (and if that means fixing tech debt, then we have a win-win)
<deryck> rick_h_, you have access to your work email now, right?
<rick_h_> deryck: yes
<sinzui> timrc, indeed that is was we have done over the last 3 epics
<deryck> rick_h_, just making sure, since I CC'ed your work email this morning.
<rick_h_> yep, do you have time to chat on that pre stand up?
<rick_h_> I wanted to chat on it if you get a sec
<deryck> rick_h_, sure.  I'll jump in mumble now.
<bac> allenap, jcsackett: could one of you review https://code.launchpad.net/~bac/launchpad/bug-891641/+merge/82544
<jcsackett> bac: sure.
<jcsackett> bac: looks fine to me. r=me.
<bac> thanks
<cr3> hi folks, any plans to upgrade YUI to 3.4 in Launchpad?
<deryck> flacoste, hi.  The latest build failure has you and lifeless listed as blamelist.  Did you take a peak yet, or do you need someone else to look?
<deryck> cr3, hi.
<flacoste> deryck: i cannot be responsible for the latest failure, i swear
<flacoste> deryck: it's a XSL change that isn't tested
<deryck> cr3, yes and no.  I want to skip 3.4 and jump to 3.5 in January.
<rick_h_> deryck: I'm getting an issue starting my local env because of the timeline egg needing to update
<flacoste> deryck: unless it failed at the build step
<rick_h_> where am I looking to pull that? make egg, build_egg and download-cache only seem to check the dir and all report "up to date"
<cr3> deryck: hola, good to know. no rush, thanks!
<deryck> rick_h_, if you use the rocketfuel script to update devel, it should work.  I think rocketfuel-get.
<rick_h_> ah, thanks
<deryck> flacoste, ok, I'll look and see what rev introduced an XSL change.
<flacoste> deryck: i mean, my change is to the WADL XSL file, and that cannot fail any tests
<deryck> rick_h_, but in short, you have a download-cache that needs to have bzr up run in it.
<deryck> flacoste, ah, sorry, I follow now.
<flacoste> deryck:
<flacoste> reference = u'a moment ago by Person-name-359157'
<flacoste> actual = '14 seconds ago by Person-name-359157'
<flacoste> that looks like a timing issue
<rvba> deryck: http://paste.ubuntu.com/741266/
<rvba> oops, too late ;)
<deryck> flacoste, ah, same one then.  I missed that at first glance.
<rick_h_> deryck: yea, I manually did a bzr pull and missed on getting the egg
<deryck> flacoste, can we kick the build then?  Since changes should be in to fix that now.
<gary_poster> I can't build the tree
<deryck> rvba, thanks anyway! :)
<rvba> :)
<deryck> flacoste, sorry, ignore me again, I see it's not the same.
<deryck> too much red lines to me.  I got lost. ;)
<gary_poster> I keep on getting complaints that builder.py can't import lpbuildd.slave
<deryck> gary_poster, so perhaps you and rick_h_ are seeing someone forgot to commit to the download-cache then?
<gary_poster> deryck, no, I don't think so
<gary_poster> possible
<rvba> gary_poster: ah ! I had this one in ec2 this morning.
<gary_poster> rvba, oh!  did you resolve it?  I've tried make clean && make to no avail, and have been trying other random things
<rvba> gary_poster: http://paste.ubuntu.com/741269/ ?
<deryck> rick_h_, could be sourcecode needs updating too then.  rocketfuel-get should do the right thing for you.  if not, make clean && make can correct these things.
<rick_h_> yea, rocketfuel-get got me running
<gary_poster> precisely rvba
<rick_h_> thanks
<rvba> gary_poster: I solved it by merging devel and fixing the stupid missing directories conflicts.
<rick_h_> oh that, yea same here
<gary_poster> rvba, maybe I fixed the conflicts incorrectly...but when I diff my trunk against the main tree in doesn't report a diff
<rick_h_> ooh, wrong one. I had left over directories from lp/buildd dirs
<gary_poster> (and this was a problem in my pristine devel branch)
<rvba> Right, left over directories masquerading as conflicts.
<jml> there's a thing you can do to avoid that
<gary_poster> what is it jml?
<jml> Add bzr.transform.orphan_policy=move to bazaar.conf
<jml> not sure where the documentation for that lives, got the tip directly from vila
<rvba> jml: good to know! thanks!
<gary_poster> huh, cool
<gary_poster> thanks jml
<jml> np.
<rvba> that's an insider's tip then :)
<gary_poster> I guess I'll wipe away my tree and do it all from scratch and hope that does the trick
<rvba> That would be the next random thing I would try too ;)
<gary_poster> :-)
<gary_poster> thanks
<rick_h_> gmb: ping
<gmb> rick_h_: Hi
<rick_h_> gmb: howdy, I wanted to see if you had time to do a pre-implementation chat with me on https://bugs.launchpad.net/launchpad/+bug/723417
 * gmb looks
<rick_h_> sometime in the not too distant future
<gmb> rick_h_: I could do without context switching for the next while... How does ~16:30UTC sound?
<rick_h_> sure thing
<gmb> Cool.
<gmb> rick_h_: I'll ping you when I'm ready.
<rick_h_> thanks, appreciate it
<gmb> np
<deryck> abentley, for our call discussion:  http://people.canonical.com/~deryck/field-state.png
<abentley> deryck: okay.
<deryck> abentley, this represents my understand, which may well be wrong.  but I thought it was easier to explain my thinking with a pic.
<jml> When I change a milestone on a bug, I get a broken image link
<jml> e.g. https://bugs.launchpad.net/testrepository/+bug/613181/undefined
<jml> instead of the edit button that I've come to know.
<deryck> jml, that was fixed once before already.  seems that gets broken a lot.
<jml> deryck: interesting, thanks.
<gary_poster> OK this is kinda not cool.  I'm still unable to build LP ("ImportError: No module named lpbuildd.slave"), despite fresh checkouts and lots of random jiggling of things around. :-(
<jelmer> gary_poster: python-lpbuildd is a debian package
<gary_poster> jelmer, ah!
<jelmer> gary_poster: if you update launchpad-developer-dependencies it should be pulled in
<gary_poster> right, I should have remebered that thing to jiggle
<gary_poster> thanks jelmer
<jelmer> gary_poster: it's just one of the gazillion ways in which launchpad pulls in dependencies
<gary_poster> yes, I was thinking something similar jelmer. :-(
<gmb> rick_h_: I'm free for a chat now if you are.
<rick_h_> gmb: sure thing
<gmb> Cool.
<gmb> rick_h_: Skype or Mumble?
<rick_h_> what works for you? mumble, skype?
<rick_h_> lol, either here
<rick_h_> mumble and jump in orange room with me for a sec then?
<gmb> rick_h_: Sure. Bear with me a sec...
<timrc> getting timeouts on staging when visiting, https://bugs.staging.launchpad.net/
<timrc>  OOPS-20c822b69c6c8fa38258f4a19efebcf6
<allenap> timrc: If you keep hitting refresh does it eventually work?
<allenap> Although the first query in the page is hideous.
<timrc> allenap, it says, "Initiating launch sequence"... JK... I'll try
<allenap> :)
<timrc> allenap, hitting reload once did work
<allenap> timrc: Okay, it's probably a cold disk cache. That happens a lot on staging and qastaging unfortunately.
<timrc> allenap, ah, okay
<timrc> allenap, is it actually calculating those statistics in real-time?
<allenap> timrc: Yes, I believe so.
<timrc> yikes
<adeuring> allenap, jcsackett: could you please review this mp: https://code.launchpad.net/~adeuring/launchpad/bug-sorting-3/+merge/82562 ?
<allenap> timrc: Private bugs make things a little complicated.
<jcsackett> adeuring: i'm in the middle of reviewing one right now, if allenap can't get to yours after that i'll take a look.
<allenap> adeuring, jcsackett: I'll take it.
<adeuring> allenap: thanks!
<timrc> seems like a waste of CPU cycles
<deryck> allenap, jcsackett -- I have a branch to get us out of testfix if one of you could kindly review it.
<deryck> it's short.
<jcsackett> jcsackett:
<jcsackett> deryck: just looked.
<jcsackett> r=me.
<deryck> jcsackett, rockin' thanks!
<deryck> alright, playing in pqm.  we should be out of testfix shortly.
<allenap> adeuring: Have you tried some of those subqueries on production? They look like big red bad performance markers to me.
<adeuring> allenap: right, there are problems for some bug targets, but we want to get a first version running (protected by a beta flag)
<adeuring> allenap: but for other tagretsm the queries are quite fast
<allenap> adeuring: Okay.
<abentley> allenap or jcsackett: could you please review https://code.launchpad.net/~abentley/launchpad/view-flags/+merge/82570 ?
<allenap> abentley: If it'll take me less than 20 minutes, sure!
<abentley> allenap: I think it should.
<allenap> Okay, deal.
<allenap> abentley: Why "is_beta" rather than, say, "is_default", and/or including the default value?
<abentley> allenap: The use case of is_beta is to indicate whether the feature is in beta.  We could change the way we determine whether a feature is in beta, but we would still care about whether it was in beta, not whether it was the default.
<GRiD> hey guys, fyi: https://code.launchpad.net/~launchpad-pqm/launchpad/devel/+activereviews seems to break
<allenap> abentley: Okay, fair.
<allenap> abentley: If related_features is empty the code still loops through flag_info. I guess that's not a big overhead, but it feels wasteful. It would be nice if there was a pre-prepared flag_info_map or similar, then you could loop through related_features instead.
<abentley> allenap: It's really not expensive to loop through flag_info, because it's a list of tuples of strings.  I don't know why poolie chose to implement it that way.  If there was a flag_info_map, I don't think we'd need the list.
<allenap> abentley: Okay.
<allenap> abentley: I have to go now, but I'll complete the review asap when I'm back.
<abentley> allenap: okay, thanks.
<rick_h_> gmb: ping, heading your way: https://code.launchpad.net/~rharding/launchpad/bugfix_723417/+merge/82577
<rick_h_> anyone know how I can work around this gpg issue when trying to use ec2 land? https://pastebin.canonical.com/55983/
<rick_h_> my GPGKEY env var is set right, but the email on that is different since I added my GPG pre-starting
<rick_h_> ok, that was part file permissions, here's a trace with not finding a gpg key for the work email addr https://pastebin.canonical.com/55984/
<rick_h_> ok, nvm, I just changed it up for now and I'll add a new gpg key on the @canonical email address since I can't run different ones in the bazaar.conf it appears
<flacoste> we should revert mbp buildd removal
<deryck> jcsackett, I have an itsy-bitsy teanie-weanie css change branch.  what say ye?
<lifeless> morning yeall
<flacoste> morning lifeless
<flacoste> you've got stakeholders email!
<lifeless> I do?
<flacoste> you do!
<lifeless> I see a bunch
<lifeless> which in particular is for me ?
<flacoste> the one you are CC-ed on
<flacoste> escalation request from skaet
<lifeless> hah. lets see now ;)
<flacoste> but i'm sure you'll have interesting opinions on others as well!
<jcsackett> deryck: I am just finishing up a very late lunch, and then I am happy to take a look.
<deryck> jcsackett, thanks! it's my latest on the review queue.
<deryck> rick_h_, hi.  I see you're moving ahead with that expanding idea.... might I suggest you look at YUI plugins?
<rick_h_> deryck: sure thing
<jcsackett> deryck: R=me.
<deryck> rick_h_, it's a nice way to add functionality to something to existing classes....
<deryck> rick_h_, so you could plug anything that is a Y.Node instance, for example.  And then that node would know how to expand as you type.
<rick_h_> ok, is there a place I should look at where we'd add the plugin?
<deryck> jcsackett, awesome sauce, thanks!
<rick_h_> I was starting to look at the stuff in formwidgets and seeing if it should be a type there
<deryck> rick_h_, well, you'd create the plugin in one branch, independent of any use in lp.
<rick_h_> in talking with gmb and looking at things, wasn't sure how to "apply" it since in a nice way across things since there's not a good hook currently
<rick_h_> ic
<deryck> rick_h_, honestly, I'm not sure how you'd apply it globally.  I'd have to have a think about that myself.
<rick_h_> s/branch/repo  ?
<deryck> rick_h_, I think I mean branch.  But not sure if we're having bzr-->git break down. :)
<deryck> rick_h_, the thing you create and push to lp. :)  You need one for the widget, then a follow up branch to wire it up in lp....
<rick_h_> right right now I started out thinking I'd add it to lp/app/javascript somewhere and add the object and the tests to the existing ones
<deryck> rick_h_, you can, of course, do this in one branch, but I think you'll make a better plugin or widget if you do that on its own.
<deryck> rick_h_, exactly.
<rick_h_> right, ok. That's basically what I'm doing ok
<rick_h_> I wasn't sure if you were thinking of making it a more generic YUI item that could be pushed to their gallery or something outside of LP
<deryck> ah
<deryck> no, not at this point.
<rick_h_> ok, sounds good
<deryck> rick_h_, might be worth seeing if the gallery has something like this already.
<deryck> rick_h_, has to be a common pattern.
<rick_h_> yea, I didn't see anything but the gallery is a pain to find things in imo
<rick_h_> hah, never mind. just don't use their search ui.
<rick_h_> http://yuilibrary.com/gallery/show/text-expander
<deryck> rick_h_, that's exactly what you want.
<deryck> rick_h_, we're putting stuff from the gallery in:  lib/lp/contrib/javascript/yui3-gallery/
<rick_h_> loading it up to test now and will see how it does
<rick_h_> ok
<deryck> rick_h_, you could probably add that and wire it up in one branch now.  since it's just an import from the gallery.
<lifeless> flacoste: done
<flacoste> lifeless: "deep diving into tsearch2 plumbing" -> that put it outside the scope of escalated bug?
<lifeless> flacoste: depends - who do we have that knows postgresql internals well enough to dig deep there - probably only jtv, but IMBW. Anyone else will need a longer run up, though I have confidence there are many folk that *can* do it, it would be less like maintenance scope, I suspect.
<flacoste> lifeless: and stub?
<lifeless> flacoste: AFAIK he hasn't been down into the implementation API. I haven't asked though.
<lifeless> flacoste: its not a matter of knowing data model - its C/C++ coding
<lifeless> flacoste: *I think*
<flacoste> ok
<flacoste> i'll leave it out for now
<flacoste> not like the escalated queue was empty anyway
<lifeless> hah, no.
<lifeless> mwhudson: hi; why does bug 881144 have linaro tasks ?
<_mup_> Bug #881144: field.tags_combinator=ALL gives same results as with ANY when searching all bug reports <bugs> <qa-ok> <regression> <search> <Launchpad itself:Fix Released by rvb> <Linaro Android:Fix Released> <Linaro-Ubuntu:Fix Released> < https://launchpad.net/bugs/881144 >
<mwhudson> lifeless: no idea off the top of my head
<mwhudson> lifeless: i guess because of asac's comment
<lifeless> I'm extremely tempted to delete them :)
<mwhudson> i presume the "Removed milestone. Wait for new TI LT Kernel." comment was misdirected
<lifeless> plus toggling on and off the status of the two tasks
<deryck> abentley, I *just* used Y.each on an object literal.  hurray for new knowledge.
<abentley> deryck: :-)
* jcsackett|afk changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 277
<deryck> lifeless, I replied briefly in the bug about client/server rendering, but for the sake of higher bandwidth, can we chat here about sessions?
<lifeless> sure
<lifeless> or voice if you want
<wgrant> deryck: Could you QA https://bugs.launchpad.net/launchpad/+bug/887646?
<_mup_> Bug #887646: Advanced search showing new bug sorting widget <bug-columns> <qa-needstesting> <ui> <Launchpad itself:Fix Committed by deryck> < https://launchpad.net/bugs/887646 >
<deryck> wgrant, oh, I thought I did way back.  sorry.
<deryck> lifeless, so is the sesssion cookie hashed, or plain strings?
<StevenK> deryck: There are two bugs linked to the branch. Both need to be qa-ok
<lifeless> deryck: oh, you want me to know whats in the code ?:P
<deryck> wgrant, StevenK -- yeah, I'm so sorry.  That's been qa-ok for awhile.
<deryck> lifeless, right. when I say "read" I mean the js code needs to see plain english, not a string it needs to decode.
<lifeless> deryck: lib/lp/services/session/model.py
<wgrant> Huh, what are you doing with sessions?
<wgrant> The JS should not know about them.
<StevenK> wgrant: I'll sort out a deployment request
<deryck> wgrant, I'm not doing anything with sessions.  lifeless wants me too. :)
<lifeless> wgrant: a 'because' would help. Also see the bug
<deryck> he means js should be able to decipher the hash.  which is what I mean.
<wgrant> O_o
<wgrant> no
<deryck> s/should/should not/
<deryck> wgrant, does the replace make that correct now?  do we agree? :)
<wgrant> Yes
<deryck> ok :)
<lifeless> deryck: so server side its a pickle
<wgrant> The content is a pickle.
<lifeless> deryck: we have state in there, and we can add more state, and so forth
<wgrant> The cookie is a hashed reference to the pickle.
<deryck> right
<lifeless> deryck: I haven't suggested the raw content be sent over the wire at any point
<deryck> no, you haven't, sorry, didn't mean to give that impression.
<lifeless> deryck: AIUI you need to store a new variable in a clients browser
<deryck> yes, it's like a url query string basically.
<lifeless> deryck: you also don't care if this is lost when the user clears local state, switches machines etc.
<deryck> right
<wgrant> We used to just store them in additional cookies.
<wgrant> For storing portlet expanded state.
<lifeless> so one complication here is that at least for a sort widget, the view has to render differently
<lifeless> (in fact it has to render differently for non-js browsers with columns as well)
<lifeless> s/columns/selected fields/
<lifeless> I don't really care how this is done, the thing that has little alarms going off is the assertion the view doesn't care :)
<deryck> lifeless, it doesn't care because we do it all on the client.  we do render a basic view with the same template, but it doesn't get any of the js widgets to change the display of fields....
<deryck> so if no js, no widgets, no need to read cookie.
<deryck> the cookie is only useful to the js widgets that redraw the list.
<lifeless> deryck: so you'll deliver it with content A, and immdiately redraw with content B ?
<lifeless> deryck: naively, I'd expect that to look a little jerky
<lifeless> deryck: I probably don't understand the issue
<deryck> lifeless, no, we render based on the model in js.  so thing of it like a bunch of widget state being updated.  one widget use cookies to maintain it's state across page loads.
<deryck> then after looking into all that state, we render the template on the client.
<deryck> it's actually quite sexy ;)
<lifeless> deryck: yes, I get that. Whats the *initial* html delivered to the client though ?
<lifeless> deryck: doesn't that initial content have a batch, rendered on the server (and mirrored into the client cache) ?
<lifeless> deryck: (I agree its sexy, its a very nice thing you've been building!)
<deryck> lifeless, not rendered.  there is an initial server rendered batch, but AIUI, we're not showing that on page load.  only if you have js disabled....
<lifeless> ah
<deryck> lifeless, if js is enabled, we render on the client completely.
<lifeless> and I presume the js is lightning fast ;)
<lifeless> so, something you could /consider/ is to use this state to render the initial batch on the server exactly as the client needs it, saving a few client side cycles (and not costing server side ones as we render anyway)
<deryck> lifeless, it is pretty fast, yes, and we've been mindful of that.  we think we may have a couple ways to speed it up even more.
<lifeless> then the client side render would only need to kick in when someone changes something
<lifeless> anyhow, I understand a bit better now.
<deryck> ok, cool.
<lifeless> francis and I both reaching for the session state is possibly just knee-jerk reactions
<lifeless> I wouldn't want to see a massive proliferation of cookies though
<lifeless> because cookies are sent on *every* request, they can become quite big.
<deryck> no, it makes sense for a lot of cases.  and I agree, we don't need a lot of cookies.
<deryck> and if we need more client state as our js use grows, and we don't want to get into js apis to poke at the session, we could share a single cookie for all client-side state.
<lifeless> so my inclination would still be to use the session state, emit the knob to js explicitly (e.g. via the cache) only on pages that need it, and use an API call or whatever to tell the server its being set
<deryck> yeah, I agree we might need to get to that.  I think that's a bit much for my small cookie use now.
<lifeless> IIRC facebook do something similar; they have just enough cookies to id the user, the machine, and then the rest of it is js
<lifeless> deryck: the reason for my concern is you only have a few hundred bytes before requests will be getting split into multiple frames
<lifeless> deryck: so a 15 byte unique name + value in a cookie is actually a Big Deal.
<deryck> I see what you mean.
<lifeless> deryck: multiple frames for a request is a huge performance hit, as I'm sure you know
<deryck> indeed
<wgrant> How do cookies interact with having multiple searches open?
<lifeless> if you want to use a cookie, I suggest doing a bit of defensive digging into our current request sizes - we may be already terrible, in which case I won't hold you to a higher standard.
<lifeless> or we may be very safe, in which case cool.
<lifeless> or we may be right on the border, particularly with POST requests, and I would really push back hard on you at that point
<deryck> lifeless, ok, how about this?  The branch I have for this is quite small.  Let me get it working.  Then in the week of polish we do, we can explore this further and make sure we're good.  fair enough?
<lifeless> deryck: the risk is that you'll cause a site wide one round trip additional latency for every user.
<lifeless> deryck: I think thats a pretty big risk.
<lifeless> deryck: and I have not good sense about the probability.
<lifeless> deryck: mmm, scratch that, my logic is bong.
<lifeless> *users that cause the cookie to be set*...
<deryck> right
<lifeless> so, sure, go ahead - but lets make sure the t's are crossed and the i's are dotted *before* it is enabled beyond ~launchpad.
<deryck> not everyone, I'd guess *most* won't have this cookie.
<lifeless> deryck: as soon as they fiddle with the feature they will get it
<deryck> correct.  well, fiddle with the display of fields.
<deryck> you can sort all day without changing fields.
<lifeless> so how does that sound? land now, research and fix if needed before widespread (> ~launchpad) use ?
<deryck> lifeless, are you ok with opening it for beta users without running this issue to ground?
<lifeless> I'm worried about opening it to beta users
<lifeless> deleting the cookie, if we change our mind and want to remove the overhead, will be a nuisance.
<deryck> lifeless, ok, fair enough.  I'm fine to land now, and dig into it completely before opening further than ~launchpad.
<lifeless> If you're up for making that automatic (e.g. delete-cookie in a reply if we see it in a reqest), then I don't mind ~beta-users getting it
<deryck> lifeless, let's just settle the issue before opening to beta.  should be easy enough to do tomorrow, to work out how we're doing on page size.
<deryck> lifeless, that's the main issue, right?  the total page size.
<lifeless> deryck: request size
<deryck> lifeless, right, sorry.
<lifeless> deryck: set-cookie in the odd response is nothing, cookie: in everyrequest, is :)
<lifeless> s/is :/may be :/
<deryck> right, I follow.
<deryck> I'll do some follow up tomorrow then.  fair enough?
<lifeless> deryck: particular things to look at are the sizes of advanced search requests and comment containing posts I guess.
<lifeless> deryck: yes, thats great. thanks for grabbing me to discuss this!
<deryck> lifeless, np!  Thank you for the discussion!
<deryck> Later on, everyone.
<poolie> huwshimi, can't wait for your bug tags to land
<poolie> do you want to have that talk about markup today maybe?
<huwshimi> poolie: :)
<huwshimi> poolie: Sure, maybe do you want to have a chat after lunch, I just need to finish off a few things this morning
<poolie> wfm, ping me then
<huwshimi> poolie: Thanks :)
 * StevenK can't find versions.cfg in lp:txlongpoll
<StevenK> How does this even work?
<lifeless> I think allenap has a branhc proposed fo rmerging that will add that
<lifeless> and/or check setup.py - it may have version constraints there
<StevenK> No constraints in setup.py for Twisted
<lifeless> then its probably grabbing latest-always
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 287
<rick_h_> is there a webui to PQM? I had to change my .bzr emai addy and so not sure I'm going to get anything from it
<rick_h_> see my ec2 instance is terminated and wonder if things went ok or fail
<lifeless> https://pqm.launchpad.net
<wgrant> But it's not very useful :)
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugtasks: 287
<rick_h_> heh, ok
<wallyworld_> poolie: just curious about the change to restore the prng state as opposed to the original proposed change. is yours more correct?
<poolie> wallyworld_, i think it's preferable to have tests not mutate the global state any more than is necessary
<wallyworld_> sure. agree with that
<StevenK> 287. When did that happen. :-(
<wgrant> StevenK: Lots of escalations overnight.
#launchpad-dev 2011-11-18
<wallyworld_> poolie: as you say in your comment, your change will make tests less random. isn't that a bad thing? don't we want tests to pass with unpredictable input?
<poolie> this is a bit philosophical
<StevenK> iamfuzz: O hai!
<iamfuzz> StevenK, o/
<poolie> i think you should test stuff intentionally, not count on randomness
<poolie> tests that happen to run before this one will get one type of input and tests that happen to run after it will get a different prng
<iamfuzz> StevenK, thanks for the help btw.  Trying to get this up to evaluate as the tool for our build farm here
<poolie> it's hard to see how that's helpful
<iamfuzz> StevenK, easy peasy getting it running but can't import keys.  It looks like I may have screwed something up irt the bazaar rewrite rule or something perhaps
 * wgrant is scared now
<iamfuzz> StevenK, when attempting to import a key, it looks like it tries to hit this URL based on the logs:  https://bazaar.launchpad.dev/pks/lookup?search=0xD69FC46438D3A33619E5577D39BE3335F26A3797&op=get
<iamfuzz> which if I hit directly, gives more the "Lost something?" 404 page with broken images
<wgrant> iamfuzz: Hmm, it should be keyserver.launchpad.dev, not bazaar.launchpad.dev.
<wgrant> Have you changed the configs at all?
<iamfuzz> however, hitting bazaar.launchpad.dev directly does properly redirect to launchpad.dev
<wallyworld_> poolie: isn't the whole point of using a rng to provide different input each time?
<iamfuzz> wgrant, the apache config, yes
<wgrant> iamfuzz: The Apache config shouldn't matter here.
<StevenK> wallyworld_: Are you still reviewing poolie's branch?
<wgrant> iamfuzz: The test keyserver doesn't go through apache
<wgrant> it's keyserver.launchpad.net:11371
<wgrant> s/net/dev/
<poolie> wallyworld_, so, generally, my experience is that using a rng in tests is a bad idea
<poolie> if there are some random values that catch a bug, which is possible
<poolie> there are two problems
<poolie> one is, the particular values are likely to come up only very rarely, so the bug may be latent for a long time
<poolie> secondly, it will be hard to reproduce it, so perhaps hard to understand why it's failing
<poolie> oh, and also, it may fail at inconvenient times
<poolie> for example on buildbot but not ec2 etc
<iamfuzz> wgrant, https://keyserver.launchpad.dev:11371/pks/lookup?search=0xD69FC46438D3A33619E5577D39BE3335F26A3797&op=get  just hangs on me
<iamfuzz> yep, timed out
<wgrant> iamfuzz: http, not https
<iamfuzz> ah there we go
<iamfuzz> much better
<iamfuzz> Error handling request: No keys found
<wallyworld_> StevenK: yes
<poolie> wallyworld_, so this particular test is testing code that itself uses the rng, so i think it has a valid reason to touch it
<poolie> but it should isoltae itself
<iamfuzz> wgrant, what is it attempting to parse for its pgp key info to not be finding any keys? or is there a utilities/ script I need to run?
<wgrant> iamfuzz: gpg --keyserver keyserver.launchpad.dev --send-keys SOMEKEYID
<wallyworld_> poolie: that all makes sense. thanks for explaining
<wgrant> iamfuzz: The dev keyserver doesn't know about any keys by default.
<iamfuzz> wgrant, ah doh, sorry bout that.  I had assumed it would sync with the ubuntu keyserver
<wgrant> iamfuzz: No, that would take a looooooooooong time :)
<iamfuzz> point taken :-)
<wgrant> You can point your local LP at the Ubuntu keyserver, but that gets problematic when you want signing keys for your PPAs, as it will upload them to the real keyserver where they can never be deleted.
<iamfuzz> wgrant, worked like a charm, many thanks.  Now I just have to get email working on here and should be golden
<wgrant> iamfuzz: Email from the webapp is generally redirected to root@localhost.
<iamfuzz> even better
<wgrant> Or you can grab the token with "SELECT * FROM logintoken"
<wgrant> Then https://launchpad.dev/token/SOMETOKENVALUE
<wgrant> If you are lazy and don't want to decrypt the email.
<iamfuzz> wgrant, I'm not *that* lazy ;-)
<iamfuzz> you guys have made it far easier than I expected
<wallyworld_> StevenK: r=me. you've put the egg in download-cache i'm sure? :-)
 * iamfuzz crosses fingers soyuz works out as well :-)
<StevenK> wallyworld_: I have, and there are two branches
<wallyworld_> StevenK: already reviewed the other one :-)
<wgrant> iamfuzz: I spent the two days after the code was released working out and documenting how to make Soyuz work. https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally should be adequate.
<wgrant> Although I guess you've already seen that, if you have the keyserver running.
<poolie> wallyworld_,  should i put that on the mp maybe
<wallyworld_> poolie: i think that would be great
<wallyworld_> poolie: also so that the original branch author etc can see why the change was made
<poolie> wallyworld_,  thanks for asking
<wallyworld_> poolie: np. i was being a bit cautious just to make sure i didn't approve something that would fuck things up because i didn't appreciate the reason for the change
<poolie> sure
<poolie> i would like to make people appreciate randomness in tests is not a feature
<mwhudson> poolie: of course, you don't always get to choose -- i remember a test failure that turned out to be assuming the email packages way of generating message ids was collision resistant
<wallyworld_> poolie: i don't think i've ever used a rng in any tests but made the mistake here of assuming that because it was used, a requirement was to have as much randomness as possible
<mwhudson> but sadly it just uses second level timestamp + randint(100000)
<StevenK> wallyworld_: You do, by side effect
<wallyworld_> i guess if are a writing a comms protocol test or something like that you need one
 * StevenK sighs at our Makefile
 * wallyworld_ off to get some medicine from the doctor. back soon hopefully
 * StevenK fixes rm to be $(RM)
<jelmer> StevenK: is there a particular reason for that?
<StevenK> jelmer: For which, sorry?
<jelmer> StevenK: $(RM), it surprised me too
<jelmer> the last project I encountered that substituted RM did it because of broken rm implementations
<StevenK> jelmer: It's cross-platform, which isn't really a concern for us, but I'm a pedant like that.
<jelmer> ah, ok
<jelmer> launchpad on windows ? :-)
<StevenK> Bwahaha
<StevenK> We need POSIX
<StevenK> Launchpad on OS X could be possible, which is a scary prospect
<mwhudson> i've contemplated it, but funnily enough always found better things to do
<wgrant> The most difficult bit would be porting python-apt, I expect
<wgrant> The most adventurous thing I've done is running it on kfreebsd.
<rick_h_> python-macports
<rick_h_> I can see it now
<StevenK> mwhudson: Haha
<mwhudson> i think jamesh said he did it once in the early days, or thought about it?
<mwhudson> anyway, talking of better things to do --> lunch
<poolie> wallyworld_, https://bugs.launchpad.net/launchpad/+bug/1734 is arguably fixed by hiding bug comments
<_mup_> Bug #1734: Bug discussion is append-only and cannot be gardened <feature> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1734 >
<poolie> pht
<poolie> wgrant, so what's next?
<poolie> i guess i will get rid of the runtime master-client dependency
<wgrant> poolie: RIght, just remove that one import.
<wgrant> poolie: Replace it with the string, like the rest of the file uses already.
<wgrant> Then check that the appserver runs without python-lpbuildd installed, and reland.
<poolie> k on it
<wgrant> Thanks.
<wallyworld_> poolie: that does look like it could be regarded as a dup of 885672 but when the branch lands, it's going to be quite limited to who we expose it to (at least initially)
<poolie> there are bunch of not-quite dupes
<poolie> it may be worth tearing them apart
<wgrant> I think it's pretty different.
<wgrant> This isn't for gardening.
<wgrant> This is for spam and emergencies.
<wgrant> Gardening is very different.
<wgrant> Ah, jelmer is still awake!
<wallyworld_> wgrant: sure, but the solution is the same even though the problem descriptions are different
<wgrant> wallyworld_: Is it?
<wallyworld_> wgrant: or StevenK: could you +1 this. it looks ok but i'm not confident that i understand the implications enough https://code.launchpad.net/~jelmer/launchpad-buildd/dpkg-buildpackage-sa/+merge/82623
<wgrant> wallyworld_: I'm doing that right now.
<wallyworld_> ok thanks
<wallyworld_> wgrant: allowing the user to hide their own comments is a form or gardening i think
<wgrant> Ermmmmm
<wgrant> poolie: Why does ~launchpad-dev own lp:launchpad-buildd.
<wgrant> ?
 * wgrant fixes.
<nigelb> wgrant: wow.
<nigelb> I wwas about to ask that.
<nigelb> (because of the MP requests coming on the m/l)
<poolie> wgrant, oh was that the list?
<poolie> the short answer is that trusted pickers aren't used everywhere
<wgrant> It's the list, yes.
<wgrant> Which is an open team.
<wgrant> https://dev.launchpad.net/CreatingNewProjects
<poolie> sorry
<wgrant> We should really abolish open teams.
<dobey> emancipation is key
<poolie> i do think teams need some kind of superclass split
<poolie> PersonGroup, Team(PersonGroup), MailingList(PersonGroup)
<poolie> something ilke that
<poolie> they are too overused
<lifeless> mmm
<lifeless> maybe
<poolie> and being able to use the same team for many different purposes is not necessarily beneficial
<poolie> maybe not
<poolie> anyhow
<poolie> it's kind of stupid that it's possible to make this mistake
<poolie> even if i am stupid for making it
<lifeless> well, the instructions could do with automation
<wgrant> Sadly some people do like having open branches.
<lifeless> there are lots of groups which could be chosen
<lifeless> there is only one answer per that page
<wgrant> But we should probably add <blink>YOU ARE PROBABLY STUPID</blink> to branches owned by open teams.
<nigelb> heh
<nigelb> wgrant: <marquee>
<poolie> ...
<poolie> don't people have to sign a coc or something to use ppas?
<poolie> seems like this should be required for auto built branches
<poolie> but
<lifeless> I think jml removed that
<wgrant> poolie: I think that requirement was waived.
<lifeless> or planned to
<poolie> yeah, that's probably better
<poolie> amhnews :)
<lifeless> wow buildbot is in bad shape
<wgrant> Oh?
<wgrant> Hmm, even more.
<wgrant> Excellent.
<wgrant> Ah.
<wgrant> It was restarted but not cleaned.
<wgrant> LOSA ping: please clean and restart pigeonpea
<lifeless> 6 fails in a row
<lifeless> 7 when the current one falls over entirely
<hloeung> wgrant: ok
<hloeung> wgrant: done
<wgrant> hloeung: Thanks.
<poolie> wgrant, like this: https://code.launchpad.net/~mbp/launchpad/800295-protocol/+merge/82629
<StevenK> Dear buildbot, please DIALCF. No love, all of us.
<StevenK> Dear PQM, you're next. No love, me.
<nigelb> heh.
<StevenK> Right, I think that's all of the 127 test failures fixed.
<StevenK> lifeless: So, do you have eight MPs for me to review? I'm feeling dangerously productive. :-P
<lifeless> StevenK: TGIF
<StevenK> +1,000
<huwshimi> ugh, there appears to be quite a bit of our CSS that just isn't used anymore
 * mwhudson fails to be surprised
<huwshimi> mwhudson: Yeah, but it makes me nervous
<mwhudson> huwshimi: you get to delete code! bonus
<huwshimi> mwhudson: Yeah
<huwshimi> mwhudson: But I worry that I've missed something :)
<huwshimi> mwhudson: Even when my search of the whole codebase comes back with nothing
<mwhudson> yeah, i guess it's hard to write automated tests for css
<mwhudson> we certainly construct class names from enums, which doesn't help grep...
<jtv> wgrant: the dogfood builds yesterday worked, but I do see some errors in the log: CannotFetchFile, seemingly with empty URL & info.  Any idea what that might mean?
<mwhudson> StevenK: (from #-ops) ah, i see  https://bugs.launchpad.net/launchpad/+bug/704080 now
<_mup_> Bug #704080: If there is only one recipe, it should redirect right to the recipe <confusing-ui> <qa-ok> <recipe> <Launchpad itself:Fix Released by stevenk> < https://launchpad.net/bugs/704080 >
<StevenK> Indeed
<mwhudson> that bug was about the list on a branch page, which says _1 recipe uses this branch_
<wgrant> jtv: Well, possibly that it couldn't fetch the file. I've not seen them before, AFAICR.
<mwhudson> when the link says "View source package recipes" it's more confusing
<jtv> wgrant: but then there ought to be a URL in the exception message, no?
<jtv> It looks as if it's getting no URL at all, and hence the error.
<wgrant> jtv: No idea.
<wgrant> Hmm
<wgrant> That's not helpful.
 * mwhudson files https://bugs.launchpad.net/launchpad/+bug/891918
<_mup_> Bug #891918: redirect from /~/+recipes to recipe when only one recipe can be confusing <confusing-ui> <Launchpad itself:New> < https://launchpad.net/bugs/891918 >
<StevenK> mwhudson: So perhaps we should just hijack that link on Person:+index when they only have 1 recipe.
<mwhudson> StevenK: that would certainly work
<mwhudson> it's not a very important bug i guess
 * StevenK glares at mwhudson for not triaging :-P
<mwhudson> ah yeah
<StevenK> mwhudson: I'ved fixed it anyway
<mwhudson> ta
<huwshimi> If I want to make modifications the CSS files that are loaded I assume I modify "buildout-templates/bin/combine-css.in"
<huwshimi> And then how to I get that rebuilt into /bin so that it gets run with "make css_combine"?
<StevenK> huwshimi: Change the template and then 'make clean && make'
<huwshimi> StevenK: ah ha
<wgrant> You can possibly also just bin/buildout
<StevenK> Personally, I like to make sure, but that requires sitting through buildmailman
<huwshimi> awesome, bin/buildout worked
<StevenK> RARGH, conflicts.
 * StevenK smash.
<lifeless> StevenK: question
<lifeless> StevenK: did you consider just telling owners etc to be members of the bugsupervisor team ?
<StevenK> Why should they? They *own* the pillar.
<lifeless> delegation
<StevenK> Sure, but they should also have the permission.
<lifeless> why?
<StevenK> Can I say a cop out and reply 'Because Curtis says so' ?
<lifeless> if you must
<lifeless> the UI and help will need changing too
<StevenK> admins can do everything. owners delegate planning and bug responsibilities to drivers and bug supervisors. Drivers work with release planing, which encompasses bugs and specs, Bug supervisors can work bugs.
<lifeless> given there are now 4 teams that can grant the permissions
<lifeless> StevenK: yes, sure, so owners, drivers can be members of bug supervisors, and get those bits directly.
<lifeless> StevenK: which will drop to 25% the number of teams that need examining by the DB when querying
<lifeless> since supervisor no long implies notification, there shouldn't be an issue with mail receipt etc
<StevenK> lifeless: So, it's a Friday afternoon. I wanted to get this reviewed and landed before EOW. Not argue about it. :-(
<wgrant> It does imply it.
<wgrant> Unless you really know what you're doing.
<lifeless> what you're doing is inconsistent with the observer design, which will be an absolute list
<lifeless> its more than a little surprising
<lifeless> StevenK: I don't want to argue about it either.
<lifeless> StevenK: I've mailed the bug.
<StevenK> http://pastebin.ubuntu.com/741893/ Uhhhhh?
<lifeless> StevenK: I think the change to check permissions rather than checking teams is good and important; I thin the changes to the security model are questionable.
<wgrant> make
<StevenK> Ah
<lifeless> strictly bin/buildout
<lifeless> need newer timeline
<wgrant> Uhoh
<wgrant> Something is wrong
<wgrant> buildbot hasn't failed again yet...
<lifeless> ?
<lifeless> heh
<StevenK> Hah
<wgrant> One of the last eight builds has passed...
<lifeless> spwaking of ROW
<lifeless> bah
<lifeless> EOW
<lifeless> night all
<wallyworld_> lifeless: good night. but i bet you come back on later :-)
<wallyworld_> StevenK: my first look at the diff - i think you can replace self.context.userHasPrivileges with self.userHasPrivileges in the view code
<StevenK> wallyworld_: I hadn't pinged you yet because I was fixing the conflict.
<StevenK> wallyworld_: Which line of the diff?
<wallyworld_> StevenK: there's a few eg 85/86
<wallyworld_> there's a mixin but the method it provides isn't used it appears
<StevenK> Ah, right
<wallyworld_> since you are fixing something else anyway..... :-)
<StevenK> wallyworld_: I was pushing the conflict change as I replied. Use of the property in the view fixed.
<wallyworld_> thanks
<StevenK> wallyworld_: Anything else, or you're still reading?
<wallyworld_> StevenK: still reading. just groking the various permission consolidations
<wallyworld_> StevenK: just saw that you could use roles.isBugSupervisor(self.pillar) instead of doing it yourself
<StevenK> wallyworld_: Oh, has that landed on devel?
<StevenK> I wasn't sure, so I left it.
<wallyworld_> ah, stupid me, no sorry, not landed yet. but close i think. let me check
<wallyworld_> StevenK: my ec2 run failed with 7 TestPoppy errors
<wallyworld_> othereise it would have landed
<wallyworld_> i think these are spurious?
<StevenK> lp-land it, and I'll re-merge
<wallyworld_> thanks, doing it now
<wallyworld_> StevenK: in now
<StevenK> wallyworld_: Thanks, merging.
<wallyworld_> StevenK: also, i think you need a few unit tests to explicitly check that a person of each role type has privileges
<StevenK> lifeless: Oh, and you know most of the code paths I'm changing were doing these checks anyway? I've just pulled them all together into one method.
<StevenK> wallyworld_: I think the bug code is well enough tested, TBH.
<StevenK> wallyworld_: But I can if you insist.
<lifeless> StevenK: just carry on, I'll discuss with sinzui
<StevenK> lifeless: So land it?
<wallyworld_> it is well tested but this branch introduces behaviour which allows additional users/roles to do things and we are not explicitly testing for that
<wallyworld_> the existing tests are great for regression testing but we need new tests for progression tests
<lifeless> StevenK: Up to you - I'm not blocking it
<StevenK> wallyworld_: I think the only change is admins, TBH.
<wallyworld_> and people in project roles, no?
<StevenK> wallyworld_: Most of that code was already there.
<StevenK> I trusted the existing test suite while developing this.
<wallyworld_> if it were me, i'd do a test case called TestUserHasPriviliges
<wallyworld_> to explicitly test the new method in isolation
<StevenK> wallyworld_: Like I say, happy to do so, if you insist.
<StevenK> I'm happy with this branch since it removes more code than it adds.
<wallyworld_> StevenK: that would be great if you could. it's a compromise because i'm not asking for additional tests on the view methods :-)
<wallyworld_> and it won't take long just for one new test case :-)
<StevenK> wallyworld_: One?
<wallyworld_> one test case but several tests
<StevenK> There are 4 celebrities to test and 6 pillar tests
<wallyworld_> yes, but these will be 10 tests foe the one test case
<wallyworld_> StevenK: off to get kid from school. will check your mp again when  i get back
<poolie> huwshimi, hi?
<huwshimi> poolie: Hi
<poolie> it's after lunch and not yet eod :)
<huwshimi> poolie: Ah right, sorry about that. I'm just trying to finish quite a large refactor
<poolie> np i sympathize
<poolie> perhaps i will start drafting something and then we can talk
<huwshimi> poolie: yeah that sounds great
<huwshimi> poolie: Feel free to shoot questions at me
<sinzui> wgrant, do you have time to mumble?
<wgrant> sinzui: Sure
<StevenK> sinzui: lifeless has concerns with my bug. But he's cooking.
<wgrant> Although I make it 5 past midnight, so am tempted to object so you go to sleep.
<wgrant> But let me find my headset.
<sinzui> I replied. We want fix the root problem with launchpad, not make project contributors more angry with convoluted nested teams that require them to get spam
<StevenK> sinzui: Thank you.
<huwshimi> Refactoring anything (in this case CSS) is nerve-wracking without tests of any kind.
<StevenK> wallyworld_: Are you back?
<huwshimi> A review for someone if they're kind enough: https://code.launchpad.net/~huwshimi/launchpad/css-file-split-891896/+merge/82636
<wallyworld_> StevenK: back now. bloody traffic really bad today :-(
<wallyworld_> huwshimi: looking in a sec
<StevenK> wallyworld_: I've pushed the test
<huwshimi> wallyworld_: Cheers :)
<huwshimi> wallyworld_: Just to warn you: "4855 lines (+2306/-2391) 20 files modified"
<StevenK> Crumbs
<huwshimi> StevenK: Yeah, I was deleting Soyuz
<nigelb> huwshimi: Good job!
<StevenK> Not large enough.
<huwshimi> StevenK: I wasn't fond of the name, so thought, "Why not!"
<nigelb> You have too many pluses for deletion.
<huwshimi> nigelb: Shhh :P
<nigelb> heh.
<nigelb> Soyuz is sweat and blood, not code.
<StevenK> wallyworld_ is a bad person. My branch removed more than it added, and then he wanted tests.
<nigelb> Oh and probably curses.
<StevenK> nigelb: Bailing wire and gaffa tape, too
<StevenK> Probably some pitch too
<nigelb> To hold it all together? Nice idea.
<wallyworld_> i am not bad, i'm just drawn that way (says me channel Roger Rabbit)
<wallyworld_> channeling
<wallyworld_> huwshimi: so the files which include import.css won't pick up the stuff in components but the two cases of that it doesn't seem to matter
<huwshimi> wallyworld_: Yeah, that was my intention. That should just import the base layer of our styles
<huwshimi> wallyworld_: Sorry I should have explained on the MP
<wallyworld_> np. and you don't see a case where somthing will need to import the base layer and additional compoent styles?
<huwshimi> wallyworld_: There may be, but I suspect for testing you'd want to include them specifically. In the future we may want to provide a case for that, but for this initial commit I let it be
<wallyworld_> sure, just checking :-)
<huwshimi> wallyworld_: There may be a base set of components that are generic enough for that case
<huwshimi> wallyworld_: np
<wallyworld_> huwshimi: and apart from that, all the red/green is just cut'n'paste essentially?
<wallyworld_> you've also run combine-css manually and checked the output?
<huwshimi> wallyworld_: Yes, just splitting it between the files. I deleted 3 styles I think when I couldn't find any instances of them in the entire codebase (references to old yui 2 styles)
<huwshimi> wallyworld_: Yeah the combine works and everything renders fine on the pages I checked
<wallyworld_> and you checked pages that use something from components?
<huwshimi> wallyworld_: Yes, for example there are components on the sidebar, and tables and stuff which all render fine
<wallyworld_> cool. sounds good, thanks
<huwshimi> wallyworld_: NO, thank you
<huwshimi> erm that no was a little forceful :)
<wallyworld_> huwshimi: np. r=me
<huwshimi> wallyworld_: Thankyou!
<huwshimi> wallyworld_: unfortunately for you it's all smoke and mirrors :P
<wallyworld_> huwshimi: ?
<huwshimi> wallyworld_: I was trying to making a joke about smoke testing. I'll be about my business.
<wallyworld_> huwshimi: it's been a long week, too long for bad puns :-)
<huwshimi> wallyworld_: haha, I know you feel. Too long to be attempting puns too it appears
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 287
 * wallyworld_ off to have dinner with bigjools :-)
<wgrant> Dammit, it's been hotter down here today
<wgrant> Than it was in Brisbane.
<wgrant> 33 vs 26
<wgrant> You're not doing a very good job of discouraging bigjools, .au :(
<jtv> I thought it was supposed to be summer over there.
<jtv> Why is it colder than over here?
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 289
<jtv> wgrant: the errors in the buildmaster are all on rhenium â seemingly lost connections.
<wgrant> jtv: rhenium is not infamous. But if it's only that one, perhaps it is special.
<jtv> What really pains me at the moment is that the biggest remaining worry is code paths that violate read-only database policy (and thus need explicit exemptions).  Which means we ought to try each kind of upload.
<jtv> The most likely failure resulting from my branch is âread-only transactionâ errors.
<jtv> What kinds of build-farm jobs do we have?  I tried package builds, and I can try TTBJs, but everything else is a mystery to me.
<wgrant> Recipe builds.
<wgrant> Which are easily QA'd on staging
<wgrant> You may wish to cowboy it there.
<jtv> How is staging's buildfarm set up?
<wgrant> What do you mean?
<wgrant> wallyworld_: Aw
<wgrant> wallyworld_: I just noticed that the whole squad except for you is arriving in Budapest on the same flight.
<jtv> wgrant: where does staging get its builders?
<wgrant> jtv: clementine/concordia belong to it
<jtv> So it's got its own?
<wgrant> But it may also have access to the donated production builders
<wgrant> Right.
<wgrant> Like dogfood's ferraz/rubidium
<wgrant> They both have access to all four.
<adeuring> good morning
<jtv> morning adeuring!
<wgrant> Morning adeuring.
<adeuring> hi jtv!
<adeuring> hi wgrant!
<jtv> wgrant: if my feeling about the most likely failures is correct, we don't really need 4 builders any more.  One or two should be enough to trigger the errors I'm looking for, and half a dozen probably isn't enough to trigger the ones we're ultimately after.
<wgrant> jtv: I except so
<jtv> *expect?
<wgrant> That's the one.
<jtv> restecp
<jtv> And staging has its own codehosting too, right?
<wgrant> Right.
<wgrant> Which dogfood lacks.
<wgrant> You *can* test recipes on dogfood.
<wgrant> Or at least could a year ago
<wgrant> Because it sort of uses production's codehosting.
<jtv> I think it's time to hand back those builders I borrowed, and continue with staging.
<wgrant> Agreed.
<wgrant> They'll need to be upgraded before they're sent back.
<jtv> Would've been good to stress-test against hordes of builders, but that doesn't seem to be in the cards.  And at least we'll have eliminated a good portion of these bugs.
<jtv> wgrant: what kind of upgrading do we need to do?
<wgrant> jtv: The production builders have all been upgraded a few times since yours were donated.
<wgrant> If we just send them back to production, they'll be running different code to the rest.
<wgrant> Which would be bad.
<jtv> How do we go about upgrading them?
<wgrant> Poke a LOSA or lamont.
<wgrant> We'll probably do a new upgrade on Monday.
<wgrant> Or maybe tonight.
<wgrant> Because we like to live dangerously.
<jtv> Friday night?
<wgrant> Right, dangerously.
<hloeung> jtv: I could upgrade the builders so that they're running the same as the others if you like
<wgrant> (and the changes are tiny and limited to a regression fix and a tiny bit of extra debug info)
<jtv> hloeung: that'd be greatâand then I'd like to get them back into the production buildfarm as soon as possible.
<hloeung> jtv: which ones are we talking about btw?
<jtv> (Make sure they don't fall behind again etc.)
 * jtv digs up the RT
<jtv> See RT 48981
<jtv> wgrant: looks like your aalib build broke rhenium.
<wgrant> Oh?
<jtv> That one's so old, I wouldn't be surprised if some essential file had flushed out of the librarian or something.
<hloeung> jtv: thanks
<wgrant> That's possible.
<jtv> Although apparently the build did suceed.
<jtv> https://librarian.dogfood.launchpad.net/80686189/buildlog_ubuntu-oneiric-amd64.aalib_1.4p5-38build1_BUILDING.txt.gz
<adeuring> lifeless: could you have a look this MP: https://code.launchpad.net/~adeuring/launchpad/bug-sorting-by-attachment-age/+merge/82315 ?
<poolie> vila, what do you mean "there's no feedback" about rejected mail
<poolie> if lp doesn't like it, it will say so
<vila> poolie: not in my case, never
<poolie> you may be acclimatised to ignore it
<vila> IIRC it happens when I reply to merge proposals and forget to sign
<poolie> it does send mail
<poolie> it is either a bug or perhaps it's getting lost somewhere
<vila> ha, lost somewhere rings a bell :-} My ISP have been dropping pqm failure emails until recently
<vila> spammers defeating the store-and-forward robustness of mail transport is the main reason I hate them :-/
<burgerga> Hello, I have a small question: on the page of launchpad itself on launchpad it says that feature requests are handled in the wiki, however in the wiki I cannot immediately see where I should do this, can anyone point me in the right direction
<poolie> what's the request?
<poolie> i suspect actually contrary to that text we really want people floating requests by irc, mail or bugs-
<burgerga> ah oke
<burgerga> I would like to save a bug report as draft
<burgerga> since often I have to start all over when I'm writing a lenghty one and have to go ;)
<poolie> i suspect there is a bug for it
<poolie> i will look
<burgerga> think i found it
<burgerga> https://bugs.launchpad.net/launchpad/+bug/327728
<_mup_> Bug #327728: Create tasks <feature> <lp-registry> <Launchpad itself:Triaged> < https://launchpad.net/bugs/327728 >
<burgerga> thanks
<burgerga> :)
<poolie> hm
<poolie> i think that's a bit different
<burgerga> yes it covers more
<poolie> smaller bugs are more likely to be fixed :)
<poolie> as long as they're not too small
<burgerga> yes, I think they changed the bug report to "create tasks" and then marked it as duplicate of another bug
<burgerga> so I think I should change it back to "save drafts"
<poolie> and undupe it
<burgerga> ok will do, thanks for the help
<poolie> i just got a failure in  lp.translations.tests.test_rosetta_branches_script.TestRosettaBranchesScript.test_rosetta_branches_script_oops for the second time running
<poolie> jtv, halp?
<jtv> Hi poolie
<poolie> actually the smart thing here is for me to just leave these alone and someone else will solve it over the weekend
<poolie> so i will do that
<jtv> poolie: did you check out the imports thing?
<jtv> poolie: I emailed about it the other dayâ¦ âimport lp.codehostingâ has side effects.  Removing it (because it looks redundant) or putting it in its regular alphabetical place may break things.
<jtv> And we have a maintenance script that may do either.
<poolie> :/
<poolie> this is bug 890816
<poolie> ?
<_mup_> Bug #890816: Intermittent test failures: TestPullerMasterIntegration.test_mirror <Launchpad itself:Triaged> < https://launchpad.net/bugs/890816 >
* adeuring changed the topic of #launchpad-dev to: #launchpad-dev ist: https://dev.launchpad.net/ | On call reviewer: adeuring | Critical bugtasks: 289
<poolie> jtv i have only one revision and it doesn't mention that
<poolie> doesn't mention lp.codehosting
<poolie> could it really be related?
<jtv> Let me just try again to load up that bug page
<jtv> poolie: ah noâ¦  this is twisted trouble.  :(
<poolie> my branch is https://code.launchpad.net/~mbp/launchpad/800295-protocol/+merge/82629
<poolie> o/ mrevell
<mrevell> Hello
<danhg> Morning all
<mrevell> hey danhg
<poolie> mrevell, i sent a mail about markdown
<mrevell> poolie, Oh cool /me looks
<poolie> we could talk if you want
<mrevell> poolie, Sure. Skype took a dislike to my mic yesterday, let me see if that's still a problem.
<poolie> maybe just here
<poolie> am still fighting the buildd gorilla
<mrevell> poolie, Oh yeah. This came up in the stakeholder meeting yesterday
<poolie> buildds, or markup?
<mrevell> poolie, Bryce put forward a request for some form of mark-up in descriptions etc
<poolie> yeah i saw, so i thought i would finally write it down
<poolie> kind of sketchy but its a start
<mrevell> poolie, Do you have any idea how long it'd take to get to your proposed first step? (markdown for project descriptions, behind a ff)
<poolie> in the best case a day?
<poolie> in the worst case, forever
<poolie> i think hooking it up will be cheap but deploying stuff is a pain in the butt
<poolie> but after the last couple of weeks my butt is a bit tougher
<poolie> other people could probably estimate it better
<poolie> if i took those steps would it be accepted?
<mrevell> poolie, From a product PoV, absolutely.
<poolie> nice
<poolie> obviously robert and the more experienced developers can have their say too
<nigelb> omg unicorns and ponnies if you can pull it off though :)
<mrevell> heh
<nigelb> This inline commenting blog post worries me. We should have it in LP :|
<poolie> we should
<poolie> that needs someone with more ajax than me
<poolie> nigelb, i suspect that too, like this, may not be infeasible if we just decide to do it
<mrevell> nigelb, poolie: It's something that's been on my roadmap from day one and we discussed it in the stakeholder meeting yesterday. Basically, I want it in Launchpad; lots of people want it in Launchpad. Today, I get to take all the requests from the stakeholder meeting, combine them with our own desires for LP and then see what we can fit into roughly 18 months.
<poolie> mrevell, i guess the reason i ask "will it be accepted"
<nigelb> We can also bend those "things that can fit in 18 months" with outside help :)
<poolie> is because it's been blocked by a kind of perfectionism a bit before
<poolie> so i'm proposing a pretty grungy stepwise approach
<poolie> but i think we have mostly put that perfectionism behind us
<mrevell> poolie, Yeah, perfectionism has left us with a remarkably imperfect product.
<nigelb> heh
<mrevell> poolie, fwiw, what you propose to do looks like just the sort of small, realistic, incremental change that I want us to do more of in LP.
<poolie> \heart
<nigelb> :)
<poolie> let's see
<wgrant> This seems like a pretty unobjectionable incremental change.
<nigelb> Which one? Markdown or inline commenting?
<wgrant> Markdown.
<wgrant> Inline commenting is not small.
<wgrant> At all.
<nigelb> Yeah.
<wgrant> Because we have dynamic diffs.
<nigelb> Ah. Right.
<nigelb> #win.
<nigelb> We should be able to do some sort of metadata on that diff somewhere, shouldn't we?
<mrevell> Yeah, in-line commenting has a budget of 8 weeks in my proposed roadmap (the revised version coming soon to a -dev list near you) and I'm probably being optimistic.
<wgrant> Ooh, we have a roadmap?
<wgrant> It's really hard to get any kind of estimate.
<wgrant> It would probably require a major rework of the MP system.
<wgrant> It's not at all clear how it would work.
<poolie> so
<poolie> if i was going to look for a small thing
<poolie> i would do some kind of js that just quotes lines from the diff in to the edit window
<poolie> maybe with text next to them
<poolie> something like that
<poolie> it may be a bit of a detour but it is not scary in size
<poolie> it would improve it
<poolie> but all the comments are still just text blocks; you don't need a comment of attachments that float around as the diff changes
<poolie> maybe that's too small of a win though
<lifeless> StevenK: would this -  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-4a5b9cd9bd803fdc1960fbd685d6871d#statementlog have been easier to work on , for that bug nomination thing you needed backtraces of ?
<lifeless> StevenK: its pretty crudely rendered ATM; I think making the sql format not do those crazy long lines would be another good incremental improvement
<rvba> adeuring: Hi, could you please have a look at https://code.launchpad.net/~rvb/launchpad/activereviews-bug-867941-hugequery/+merge/82650 ?
<adeuring> rvba: sure
<rvba> Thank you.
<jtv> cjwatson: do you ever have the problem that approving uploads on DistroSeries:+queue pages is too slow?
<adeuring> rvba: vrey nice work, r=me
<rvba> adeuring: thanks for the review!  We will see how this improves the performance of the page in question.
<adeuring> rvba: right, it's hard to predict. Might be worth to try the query on staging
<rvba> adeuring: I've had the query run on production a few times and it improved things quite a lot.
<adeuring> rvba: even better
<rvba> 4.5s â 2s
<rvba> This is still not ideal but that's a start.
<adeuring> rvba: right, but 2 seconds is uite good for such a complex query. There might be other parts of the code that could be improved
<jtv> It's quite a lot, from the sound of it.
<rvba> adeuring: one the runs: https://pastebin.canonical.com/55946/
<cjwatson> jtv: frankly, most of the time I don't use +queue because every time I do it times out so much that I regret it.  (Even though I know I should use +queue in preference to privileged shell commands if possible.)
<cjwatson> The last time I tried to use +queue it timed out even displaying the queue.
<cjwatson> It's working at the moment - for now
<jtv> cjwatson: I found a 0.8s query in there that was repeated several times, that I got down to 0.2â0.3ms.  I just Q/A'ed it, so it should be on production soon.  I'd like to ask you to try it again, and please please keep a very critical eye on it both in terms of performance and correctness.
<jtv> That's when submitting the form; I don't think it'll help page display.
<jtv> I do know that query is used in a few other places though, so hopefully you'll feel improvements in several places.
<cjwatson> jtv: OK, that's good, I'll go back to trying to use it from time to time
<cjwatson> (+queue still has several other awkwardnesses that block me from using it for everything, though.)
<jtv> cjwatson: thanks.  Given how important and complex the code is, the sooner you can give it the third degree the better!
<cjwatson> Notably https://bugs.launchpad.net/launchpad/+bug/828649.
<_mup_> Bug #828649: queue UI: apparently no way to override individual binaries <derivation> <queue-page> <Launchpad itself:Triaged> < https://launchpad.net/bugs/828649 >
<jtv> Well, one step at a time!  I'm particularly concerned that this should be correct, and it seems the same query pops up in other places as well.
<cjwatson> I've also passed your request on to others; for example I think pitti uses +queue more than I do, for stable release managemnt
<cjwatson> +e
<jtv> Thanks!
<rick_h_> morning
<rvba> morning rick_h_
<deryck> Morning, folks.
<rick_h_> morning
<rick_h_> http://blog.labix.org/2011/11/17/launchpad-rietveld-happycodereviews is crazy. I was complaining to jcastro that lp needed to integrate reviewboard for years and then there's this hah
* bac changed the topic of #launchpad-dev to: ist: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugtasks: 289
<mrevell> rick_h_, dev.launchpad.net/Projects/CustomBugListings
<gary_poster> adeuring or bac, when/if you get a chance could you please review https://code.launchpad.net/~gary/launchpad/bug724609/+merge/82689?
<bac> gary_poster: i will
<gary_poster> thanks bac
<gary_poster> thanks for tackling buildbot allenap. I was just about to start, and am still willing to, but if you are happy with fixing it, that's great by me
<gary_poster> I'll go do some more CHR :-P
<allenap> gary_poster: I'm happy to do it.
<gary_poster> cool thanks again allenap
<allenap> No worries :)
<jcsackett> adeuring or bac: can i add https://code.launchpad.net/~jcsackett/launchpad/private-bugs-open-teams-and-delegate-teams-oh-my/+merge/82703 to your review work today?
<bac> jcsackett: i will take it now
<jcsackett> thanks, bac.
<sinzui> jcsackett,  I want to run this in production to clean up the data to match the rules in production: http://pastebin.ubuntu.com/742379/
<jcsackett> sinzui: taking a look now.
<jcsackett> sinzui: this looks fine to me.
<jcsackett> do our new rules require maintainer roles to be exclusive? that must have drifted pass me in the email masses.
<sinzui> jcsackett, yes. they do. The maintainer automatically get access to all private and security issues in a project.
<jcsackett> sinzui: oh right. that makes sense.
 * jcsackett is not completely with it yet, today.
<timrc> is there anyway updates to staging can be done in the background and then brought to the foreground as soon as they are done to cause minimal disruption of service? or was I completely crazy to use staging for testing purposes?
* adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Critical bugtasks: 289
<jcsackett> bac: thanks for the review.
<allenap> bac: Up for a short review? https://code.launchpad.net/~allenap/launchpad/a-moment-ago-1/+merge/82706
<timrc> and why are we doing updates on a Friday? that seems dubious at best
<sinzui> timrc, staging's DB has not been updated on 7 weeks
<timrc> so maybe it's just my luck :)?
<sinzui> It is so stale that its data does not match the rules in production code
<timrc> it seems that whenever I need to run these unit tests (which work against staging), there's a disruption of service
<sinzui> timrc, staging should update on our saturdays. It has a long history of irregularity. I think it updated only 26 times last year
<sinzui> timrc It does get code updates several times a day. It has been unavailable more than usual because of the failed DB restores I think
<timrc> sinzui, is there any reason why we can't have two copies of staging? while one is getting updated, the other is serving requests?
<sinzui> timrc, we do. qastaging
<timrc> ah okay, so I should write my tests to try one and then default to the other?
<sinzui> its data is really ancient
<timrc> sinzui, the data matters not to me, as long as the api is consistent
<sinzui> timrc, it is. I often switch by scripts between staging and qastaging based on availability.
<timrc> sinzui, awesome
<timrc> I had it in my head that qastaging was something that was for internal use by the LP team
<jcsackett> timrc: we will do horrible things to qastaging's data in the pursuit of qa. if you don't care about that, meh, no worries. :-)
<jcsackett> sinzui: you want/have time to mumble?
<timrc> jcsackett, nah, I just need to be able to file bugs against a certain project and then verify things are set correctly...
<jcsackett> sounds like an excellent use of qastaging, then. :-)
<timrc> jcsackett, sinzui: thanks for your assistance / guidance
<bac> allenap: sure
<allenap> Thanks.
<allenap> Thanks Brad.
<sinzui> jcsackett, yes
<jml> I find myself missing buildout.
<jml> What's wrong with me?
 * deryck must have been too loud in the coffee shop call with rick_h_ 
<deryck> people are staring at me oddly
<deryck> jml, I like buildout.
<deryck> maybe something's wrong with me though ;)
<jml> deryck: I find it compares favourably to pip+virtualenv
 * jml has to go
<jml> crap. am late
<deryck> later!
<rick_h_> deryck: so the two bugs that have pqm success messages I can move the cards to "deployment ready"?
<deryck> rick_h_, ah, no, not yet... :)
<rick_h_> ok cool
<deryck> rick_h_, they need to play in buildnot for anywhere from 4-72 hours depending on how much buildnot hates you....
<rick_h_> ok, makes sense. wasn't sure what level was "ready" and "done"
<deryck> rick_h_, whenever it does merge to "stable" from "devel" which happens after the rarely succesful buildnot run....
<deryck> rick_h_, you'll get an email that the bug was updated with "qa-needstesting" tag...
<deryck> rick_h_, and then you'll need to manually qa your change on our qastaging server.
 * deryck looks for a wiki page
<deryck> rick_h_, https://dev.launchpad.net/QA/QAForContinuousRollouts
<rick_h_> ty
<deryck> rick_h_, that should explain how to qa and how to tag the bugs.
<deryck> rick_h_, if you have questions when you get to it, just grab one of us on irc.
<rick_h_> will do
<deryck> rick_h_, and of course the "rarely successful build" stuff was me joking.  ... mostly. ;)
<timrc> sinzui, does LP reap bugs marked as 'invalid' on staging and qastaging or do they persist?
<abentley> timrc: I'd be surprised if they have special handling.  Everything gets wiped when we do a full restore.
<abentley> bac: could you please review https://code.launchpad.net/~abentley/launchpad/pre-cache-batches/+merge/82716 ?
<deryck> ok, I'm out.  Later on, everyone!
<bac> abentley: at line 214 of your MP are the args in the right order?
<abentley> bac: looks right to me.
<abentley> Y.bind takes function, this, and then optional function parameters.
<abentley> bac: So the output of load_model will be the third parameter to update_from_new_model
<sinzui> timrc, abentley is correct. We lets the odd data accumulate because we expect a db restore to happen on Saturdays. And I do expect one tomorrow since the slony update was successful in production
<bac> abentley: the signature is:  update_from_new_model: function(query, fetch_only, model){
<bac> abentley: and the bind is:
<bac> +                success: Y.bind(
<bac> 214	+                    this.update_from_new_model, this, query, fetch_only),
<bac> abentley: ok, nm, i see where i was misreading it
<abentley> bac: I hate non-bound instance methods with a fiery passion, if that makes you feel any better.
<bac> abentley: it is confusing.  but in my head i saw query and fetch_only reversed
<abentley> bac: ah.
<abentley> bac: thanks for the reivew.
<rick_h_> http://www.youtube.com/watch?v=wCexiX_eUJA&feature=youtu.be&hd=1 YUI MVC yay
<sinzui> jcsackett, Can you look at the revised SQL to update production? https://pastebin.canonical.com/56052/ We are certain that the script does not need to check for open teams in open teams.
<sinzui> ^ bac can you look at the SQL I want to run in production?
 * bac looks
<benji> the Fedora Board is asking us what should be done with https://launchpad.net/fedora (at https://launchpad.net/fedora).  If anyone has things I should tell them (or would like to say yourself), let me know.
<sinzui> benji, We cannot delete it or deactivate it without a code change. The data is not valuable. I have long wanted to be able to delete or deactivate distros that mis inform users that they can contribute to fedora using Lp
<jcsackett> sinzui: looks fine to me, given your assurances.
<bac> sinzui: the SQL looks good to me.
<sinzui> jcsackett, thedac confirms the expensive part of my failed query never return exceptions to avoid
<jcsackett> sinzui: that's very good to know.
<benji> sinzui: in that case I'll tell them the first bit ("We cannot delete it or deactivate it without a code change.") and leave it at that.  They'll probably loose interest after that.
<sinzui> benji, I image two strategies if this becomes important:
<sinzui> pistol: Update db to obfuscate "fedora"
<sinzui> whip: add active column to schema, update sets/searches to only list active distros; send a 404 when users visit deactivated distros
<benji> sounds reasonable, thanks sinzui
<benji> bac: heh; I'm glad my CHR is over for today.
<rick_h_> CHR?
<benji> rick_h_: two things 1) I am aflicted with cross-channel-itis; which means that I sometimes reply to someone in a different channel than the one in which they were speaking to me, and 2) the maint. teams do things like triage bugs and answer user questions and it's called CHR
<rick_h_> ok, I see a lot of nicknames/codes flying and try to make sense of them. Couldn't figure out what CHR would stand for
<bac> rick_h_: Community Help Rotataion -- come on, man, that should be obvious
<flacoste> gary_poster: do you remember the distribute mess?
<flacoste> specifically, i'm trying to run lazr.restful tests
<flacoste> but running bootstrap.py it's trying to grab distribute
<flacoste> and then all hell breaks loose
<flacoste> because distribute is worst than freebase
<gary_poster> flacoste, heh
<gary_poster> I vaguely remember this.
<gary_poster> flacoste, I think if you use a clean python, it all works
<flacoste> that's not the answer i'm looking for :-)
<flacoste> it's bootstrap.py seems antique compared to lazr.restfulclient
<flacoste> which worked fine
<flacoste> so i'll try to update this first
<gary_poster> flacoste, I think I tried this and it did not work
<flacoste> gary_poster: seems to have done the trick
<flacoste> oh, and i added include-site-packages = false in buildout.cfg
<flacoste> as that's what we want for lazr components
<flacoste> usually
<flacoste> closer to what virtualenv gives you
<flacoste> that's a geat change you made there
<flacoste> sucky that upstream isn't keeping this
<gary_poster> flacoste, I'm glad it worked for you.  I do think I found a problem with the new stuff when I tried it for lazr.restful.  Maybe it had to do with a dependency?  I'd be happy to review it/try to break it on Monday if you wanted.  If it works, that would be great
<flacoste> hmm, it's now building lxml2...
<flacoste> do we really need that dependencies
<flacoste> sigh
<gary_poster> :-)
<gary_poster> I have to run go pick up kids
<gary_poster> bye, have a great weekend
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 289
<rick_h_> bac: doh!
<sinzui> I am seeing about 10,000 private branches that the project owners cannot access.
<sinzui> I think I need to discount merged branches...but I believe that team/person merges, and team/project restructuring easily looses branches
<timrc> 289 critical bugtasks... what's the prize for reaching 300?
<sinzui> seppuku
<sinzui> I was tempted last night to delete everything in my local lp branch and propose it for merging with devel because it would 80% of all Lp bugs
<wgrant> sinzui: +1
<iamfuzz> wgrant, so glad to see you :-)
<wgrant> Uhoh!
<wgrant> iamfuzz: What broke?
<iamfuzz> wgrant, Actually went smoother than expected, but I can't get the buildd to kickoff a build
<iamfuzz> wgrant, I uploaded two packages, and they're both still just sitting there waiting
<iamfuzz> wgrant, I enabled bob the builder (nice touch)
<iamfuzz> wgrant, but he seems lazy today
<iamfuzz> wgrant, and the queue shows nothing
<iamfuzz> wgrant, I go to my ppa, and there they sit waiting to build
<wgrant> iamfuzz: bob is a non-virtual builder, PPAs default to virtual.
<iamfuzz> wgrant, can i lie and just check that box?
<wgrant> Head over to https://launchpad.dev/builders/bob/+edit, check "Virtual", enter anything at all into the VM host box, and see if that solves it.
<wgrant> Yeah, the command the dev config uses to reset the VM is "echo $vmhost"
<wgrant> So it will always succeed.
<iamfuzz> wgrant, looks like that did the trick!
<iamfuzz> wgrant, so is it really as simple as installing launchpad-buildd on another box, then entering its details into LP and voila, another builder?
<iamfuzz> wgrant, I assume remote builders will upload files back to the soyuz host properly?
<wgrant> iamfuzz: You'll probably need to play around with /etc/hosts and/or DNS to get it to resolve ppa.launchpad.dev/archive.launchpad.dev/launchpad.dev, but yeah, pretty much.
 * nigelb waves from a closer timezone!
<iamfuzz> wgrant, how does one create apt sources for the buildds to pull from?
<iamfuzz> wgrant, I ran this guy: scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 add -f chroot-ubuntu-lucid-i386.tar.bz2
<iamfuzz> wgrant, but get a dependency wait on everything still
#launchpad-dev 2011-11-19
<iamfuzz> nm, I see
<iamfuzz> guess I need to basically setup a mirror at archive.launchpad.dev
<wgrant> iamfuzz: If you look at HowToUseSoyuzLocally, it should say to add external dependencies for the PPA
<wgrant> Which'll let you get away without populating archive.launchpad.dev.
<iamfuzz> wgrant, I just changed /etc/hosts to have archive.launchpad.dev -> archive ubuntu.com
<wgrant> iamfuzz: Or that.
<iamfuzz> wgrant, many thanks for all your help the past cpl days.  I'd still be scratching my head otherwise
<iamfuzz> working like a charm end-to-end now
<poolie> hi, is bugs.l.n unreachable?
<poolie> back now
<wgrant> It's been OK for me.
<poolie> just a glitch, maybe something on my isp
<poolie> nigelb, https://code.launchpad.net/~mbp/launchpad/meta-description/+merge/82769 woo :)
<poolie> now i think i'll do something off the computre
<nigelb> poolie: \o/ EPIC.
<poolie> the 'before' is pretty bad
<poolie> which is arguably mostly their fault
<nigelb> Yeah
<nigelb> G+ doesn't scrape content properly.
<nigelb> Facebook does it really well.
<poolie> presumably they will hook up something better at some point
<nigelb> Even following G+ instructions may not reveal best results (especially title)
<poolie> this is still better for fb though
<poolie> one bug it summarized as
<poolie> well you can see
<poolie> https://www.facebook.com/martin.pool/posts/107832539332513
<nigelb> Ouch
#launchpad-dev 2011-11-20
<poolie> awesome
<poolie> markup
<rick_h_> sweet poolie
<poolie> hey
<poolie> thanks
<poolie> jfdi ftw :)
<rick_h_> I like the markdown since there was the JS renderer for it
<rick_h_> client side instant preview ftw
<rick_h_> lol, often the only way to go jfdi
<poolie> i think i will stop for tonight soon
<poolie> but it does render some stuff
<poolie> recognizing our existing patterns will be a little bit of work
<poolie> but not too hard
<poolie> just a 40-line regexp
<lifeless> erm, argh:
<lifeless> The following packages have unmet dependencies:
<lifeless>   launchpad-dependencies: Depends: python-apt (>= 0.7.94.2ubuntu6.4) but 0.7.94.2ubuntu6 is to be installed
<lifeless> E: Version '0.7.94.2ubuntu6.4' for 'python-apt' was not found
<mtaylor> lifeless: fail
<lifeless> mtaylor: not exactly success, no
<lifeless> mwhudson: thanks :P
<mtaylor> lifeless: I try to be helpful
<mwhudson> lifeless: ?
<lifeless> mwhudson: recordSuccess
<mwhudson> lifeless: ah right
<mwhudson> lifeless: it occurred to me that in some ways codehosting has some of the properties of a microservice already
<lifeless> mwhudson: it does
<lifeless> mwhudson: its ripe for split out - see for instance the thing thomi and thumper did
<lifeless> which includes a split out, but not a migration to the split out code
<thumper> also a simplification as we didn't care about sftp
<mwhudson> lifeless: well, in as much as "it does not talk to the db directly"
<mwhudson> lifeless: yeah, split should be relatively straightforward
<lifeless> mwhudson: yes, though that is merely one component of the microservice criteria :)
<wgrant> Wasn't it originally more of a microservice?
<wgrant> Talking to XML-RPC rather than the DB?
<lifeless> wgrant: codehosting still talks xmlrpc, and yes it started that way
<lifeless> wgrant: you may be thinking f the codebrowsing url mapper
<lifeless> which was an API but was moved to DB
<wgrant> I thought codehosting talked to the DB for some things now.
<wgrant> I forget.
<wgrant> lifeless: rmadison is your friend
<lifeless> codehosting itself? no
<lifeless> we have db access on the machine for oops prune
<wgrant> lifeless: Speaking of oops-prune...
<lifeless> ggrrr
<lifeless> dpkg: error processing /var/cache/apt/archives/mountall_2.15.3_i386.deb (--unpack):
<lifeless>  unable to make backup link of `./lib/init/fstab' before installing new version: Invalid cross-device link
 * lifeless embugginates on lxc
<wgrant> Yes :/
<wgrant> I can't remember how I fixed that.
<wgrant> Google might know.
<wgrant> LXC is extremely fragile.
<wgrant> And buggy as hell.
<wgrant> It takes over my TTYs every couple of weeks :(
<StevenK> And we want to run our testsuite, which is extremly fragile, and buggy as hell on top of it ...
<StevenK> Sounds like win.
<lifeless> fortunately the bugs don't intersect much
<wgrant> Yeah
<wgrant> This is probably the sort of thing that LXC is suitable for.
<wgrant> ie. no security requirements
<lifeless> ubuntu-cloud folk want to run openstack test runs with openstack deployed on lxc
<lifeless> poolie: on markdown, yes, we do need the nofollow rule.
<poolie> k
<poolie> that should be easy enough to add
<poolie> do you need person descriptions treated differently to all other text?
<lifeless> per curtis' rules?
<lifeless> uhm
<mwhudson> wgrant: 'codehosting' in general talks to the db a bit, eg for jobs
<lifeless> Arguably no, but perhaps we should treat -all- comments by low karma folk as not-capable-of-making-links.
<mwhudson> wgrant: but the ssh server daemons & bzr lp-serve processes do not
<wgrant> mwhudson: Ah, right.
<lifeless> put another way, I htink the no-links-for-newbies is a good thing, and the limitation of only doing that on the persons homepage is what should change
<lifeless> however, I'd suggest getting a definite from curtis - I don't have a strong enough opinion any which way to override his preference, which appeared to be 'do not link low karma person homepages'
<poolie> it would be interesting to run them all through eg http://www.surbl.org/
<poolie> i think 'not enough reputation to make links' generically could be good
<poolie> mm
<poolie> more so if you only earn reputation by doing useful things
<huwshimi> Again, I'm guessing this test failure is nothing to do with me? http://paste.ubuntu.com/744818/
<mwhudson> huwshimi: allenap fixed a bunch of those bugs recently i think?
<mwhudson> as in, over the weekend
<huwshimi> Hmm.. guess I should just land it then
<wgrant> huwshimi: I believe that was fixed on Friday.
<wgrant> So yes, land away.
<huwshimi> mwhudson, wgrant: thans
<huwshimi> *thanks
<poolie> huwshimi, hi
<huwshimi> poolie: Morning
<poolie> so, markdown! :)
<huwshimi> poolie: Markdown, indeed
<wgrant> + Alternative fix from #postgresql is to change the sort to 'datecreated +
<wgrant> + interval '0'' which runs this query otherwise unmodified in 300ms on
<wgrant> lifeless: wtf?
<wgrant> + qastaging.
<lifeless> wgrant: there are 12K bugs found by the bug supervisor rules
<lifeless> wgrant: grabbing those and sorting separate -> fast
<lifeless> wgrant: walking 150K tasks one by one consulting to see if the task is in that 12K -> slow, as there are only 4 bugs matching the full filters
<wgrant> Ah
<poolie> can i send https://code.launchpad.net/~mbp/launchpad/ec2-region/+merge/82487
<lifeless> +interval '0' makes the sort computed and not satisfiable by index. Today anyhow.
<poolie> i have used it locally for several tests and it seems pretty solid
<poolie> or actually i suppose https://code.launchpad.net/~mbp/launchpad/ec2-kill/+merge/82631 which incorporates it
<mwhudson> poolie: i can't see why you shouldn't land the ec2 changes
<poolie> yup
<mwhudson> poolie: monday am is probably a good as time as any to break it, and i can't think of anything more you can do that would make landing safer really
<poolie> exactly
<mwhudson> now someone finish off parallelization and let's have a high cpu quadruple extra large instance or whatever it's called at the problem :-)
<poolie> yeah
<poolie> you're right the times are extremely variable so it's hard to say if tmpfs is faster
<poolie> just a funny coincidence i got plausibly stable times
<wgrant> I haven't tested LP on tmpfs on the cloud before.
<poolie> perhaps 3h30m is the "if nothing goes wrong" time
<wgrant> Only locally.
<poolie> maybe if there's no contention
<mwhudson> poolie: yeah, i guess it depends how much disk io other vms on the physical box are doing blah blah blah
<wgrant> It's critical when doing it parallel on a physical machine.
<poolie> yes
<mwhudson> i wonder if there is any way we could have ec2 test report run time to some central location, trends and averages should still be meaningful
<poolie> i have heard ec2 io can bog down
<poolie> mm
<poolie> would be good
<poolie> even just mail to a mailbox
<poolie> since it already knows how to send mail
<mwhudson> poolie: apparently EBS is pretty slow
<mwhudson> (if you have a database on EBS SOP is to raid 0 stripe it, apparently)
<mwhudson> but can believe local io is slow too
<wgrant> A DB on EBS?
<wgrant> Sounds somewhat insane.
<mwhudson> wgrant: it's what simpledb probably does, i think?
<wgrant> No clue.
<mwhudson> my ec2 neurons are fading
<lifeless> well
<lifeless> simpledb is a distributed hash right ?
<mwhudson> maybe i mean the other thign :)
<lifeless> I am pretty sure simpledb is dynamo
<mwhudson> rds?
<mwhudson> yes, rds
<lifeless> yes, rds is sql dbs
<poolie> wgrant, uh, we run a db on ebs :)
<poolie> or, we did prior to my branch landing
<wgrant> poolie: Do we?
<mwhudson> poolie: no we didn't
<wgrant> I thought we ran a test DB on instance storage.
<wgrant> Not a real DB at all.
<poolie> oh, true, sorry
<wgrant> And certainly not a real DB on EBS.
<poolie> well, my point is, what is "real"
<poolie> you could have large but fairly disposable dbs
<huwshimi> I have changed the way tags are ordered in the portlet. This breaks this test: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/bugs/stories/bug-tags/xx-official-bug-tags.txt in this way: http://paste.ubuntu.com/744836/
<huwshimi> Any suggestions about how to fix that doctest in a way which isn't just re-ordering those tags
<poolie> i'm really keen to see that land
<huwshimi> I don't want the output in the doctest to just be a jumbled mess, or at least that seems like a bad idea to me
<poolie> "then why are you using doctests?" :)
<mwhudson> delete the test?
<huwshimi> poolie: Thanks, very helpful :P
<poolie> rewrite it as python
<poolie> is the test meant to be asserting the official tags come first or something?
<huwshimi> poolie: I don't know!
<mwhudson> huwshimi: semi seriously, the question you have to answer here is "what is the intent of this test"
<mwhudson> well, entirely seriously now
<mwhudson> i was being semi serious before
<mwhudson> it seems to be testing that all the official tags are displayed, but not all of the unofficial ones
<StevenK> -        self.assertThat(recorder, HasQueryCount(LessThan(100)))
<StevenK> +        self.assertThat(recorder, HasQueryCount(LessThan(101)))
<StevenK> :-(
<huwshimi> mwhudson: I can't even make sense of the original comments/output
<mwhudson> if that's still true, then maybe printing out the number of tags that .startswith('official') and .startswith('unofficial') is a good fix?
<poolie> if the test is not adding value i think you should bias towards deleting it
<mwhudson> huwshimi: it doesn't seem /that/ cryptic to me :)
<mwhudson> but +1 poolie
<huwshimi> mwhudson: "The 10 unofficial tags will be the 10 most popular tags"
<mwhudson> huwshimi: ah yeah, that bit is harder i guess
<poolie> write a python test that clearly says what you want the behaviour to be
<huwshimi> mwhudson: Oh I see, they are more popular, so 10 and 11 are pushed out
<lifeless> StevenK: what is increasing the query count?
<mwhudson> huwshimi: right
<mwhudson> huwshimi: it's a pretty terrible test :-)
#launchpad-dev 2012-11-12
<wallyworld_> wgrant: since upgrading to postgres 9.1, my /var/lib/postgresql/9.1/main/base directory has gobbled up all my diskspace in / and now i'm really low. do you know how to fix?
<wgrant> wallyworld_: Do you have lots of old testrunner databases, perhaps?
<wallyworld_> maybe. how to i check?
<wgrant> \l in psql
<wallyworld_> wgrant: yes, so it seems i have 100s of the little fuckers
<wallyworld_> thanks, is there a quick way to delete them all?
<wgrant> Um, easiest way might be to just rebuild your cluster from scratch
<wgrant> ie. rerun utilities/launchpad-database-setup
<wgrant> As long as you have nothing important in your pg9.1 installation
<wallyworld_> yeah, was thinking i would need to do that.
<wallyworld_> no, just lp
<wallyworld_> thanks
<lifeless> wgrant: nice bug - 1071562
<lifeless> stub: o/
<stub> lifeless: o/
<stub> awsome, our ppas scale that well!
<wgrant> stub: Well, not exactly...
<wgrant> lifeless: Yeah
<wgrant> I spent today starting to clean that up
<wgrant> There's only about a million extra pubs
<lifeless> wheee
<lifeless> much fallout ?
<stub> Ideally, this sort of thing should clean itself up automatically - heuristics to remove unloved publications. I guess heuristics is a scary word to use when discussing deleting stuff though.
<adeuring> good morning
<lifeless> adeuring: hi
<adeuring> hi lifeless!
<czajkowski> morning folks
<czajkowski> stub: just the person
<czajkowski> stub: https://support.one.ubuntu.com/Ticket/Display.html?id=20963  I need a DB person, sinuzi has left a comment on the last bit
<wgrant> czajkowski: sinzui's last comment is not correct
<wgrant> OpenID identifiers can be transferred if the SSO account's primary email address shows up on a different LP account.
<wgrant> That's likely to be what happened here.
<wgrant> czajkowski: It sounds like the user might want to merge the old LP account into the new, perhaps
<wgrant> They can use https://launchpad.net/people/+requestmerge to do that
<czajkowski> wgrant: they've tried that and failed also
<czajkowski> it's gone back and forth from us to isd for weeks now, be nice to give them a resolution one way or another
<wgrant> czajkowski: Failed how?
<wgrant> Oh, is this the one the bug was filed about a couple of weeks back?
<wgrant> And fixed a day later...
<wgrant> No, different account
<wgrant> So, what failure?
<wgrant> Simply saying it failed is not entirely useful :)
<czajkowski> wgrant: gimmie a tick trying to find log
<czajkowski> and multitasking before I've had a cuppa tea is not helping
<wgrant> Heh
<czajkowski> wgrant: seems te user has been dealing with 4 people and is possibly a little confused and said it faied as it didnt work
<czajkowski> perhaps due to the bug wallyworld_ has just fixed
<czajkowski> https://answers.launchpad.net/launchpad/+question/212913 similar issue. but bug now fixed.
<stub> .me has a look
 * czajkowski hands stub a bucket of tea also 
<stub> The RT sounds like a request to merge andri-ch into andreichiver, which hasn't been done
<wgrant> That is indeed what I would suggest
<stub> Then perhaps clean out openid identifiers *if* they can't log in.
<stub> Its confused by all the openid noise at the start, which is irrelevant to the actual request
<stub> Where is our merge form?
<wgrant> 20:06:08 < wgrant> They can use https://launchpad.net/people/+requestmerge to do that
<wgrant> :)
<wgrant> But czajkowski says that they have said it fails
<stub> The admin one, I'll assume that the self help one is failing
<wgrant> https://launchpad.net/people/+adminpersonmerge
<wgrant> Or https://launchpad.net/people/+adminpeoplemerge
<wgrant> Yeah, the latter
<stub> webops: Per RT 20963, can you please merge andreichiver and andri-ch ? andreichiver is the account to keep.
<mthaddon> checking
<stub> I dropped my superpowers, first time I've missed it :)
<mthaddon> stub: this is Ubuntu RT? I don't see that ticket
<stub> (16:00:35) Laura (czajkowski): stub: https://support.one.ubuntu.com/Ticket/Display.html?id=20963  I need a DB person, sinuzi has left a comment on the last bit
<stub> Ubuntu One
<mthaddon> ah
<stub> ECHAN
<czajkowski> so many rt systems :)
<wallyworld_> jml: hello
<wallyworld_> sorry, wrong nic
<wallyworld_> jam: hello
<jam> wallyworld_: hey
<wallyworld_> i just got back from soccer
<wallyworld_> so, which medium do you want to use?
<jam> wallyworld_: we've got a google hangout for the group, it is attached to the invite: https://plus.google.com/hangouts/_/188d69cd3499f77266a3d184bc2a9433292f8979#
<wallyworld_> jam: i'm in the "you need to install plugin" loop from hell, let me see if i can escape the roundabout
<jam> wallyworld_: k, we'll just chat behind your back :). But if you need us to switch to mumble, just let us know
<czajkowski> wallyworld_: I found changing browsers helped that issue. used to happen me a lot using chromium, but not FF
<wallyworld_> czajkowski: i got it working, thanks! i installed the deb by hand
<czajkowski> ahh
<czajkowski> yes I have had to do that in the past also
<czajkowski> even when it was already there
<rick_h> morning
<czajkowski> rick_h: ello
<czajkowski> rick_h: you not off today ?
<rick_h> czajkowski: no, swapped for friday after thanksgiving
<czajkowski> rick_h: https://bugs.launchpad.net/launchpad/+bug/1077941  one for orange
<_mup_> Bug #1077941: changing maintainer of proprietary project gives incorrect message <privacy> <private-projects> <projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1077941 >
<czajkowski> can you mention it to deryck please.
<rick_h> czajkowski: will do
 * cjwatson goes back to trying to sort out the xx-copy-packages.txt failures I induced last week by trying to remove the sync copy path
<cjwatson> Oh God the pain
<rick_h> cjwatson: /me sends much coffee
<czajkowski> cjwatson: I'll swap you for hiccups for over 90 mins shall I for pain!
<cjwatson> Hmm.  No ta
<rick_h> hah, for the other side of the world crowd wgrant wallyworld_ http://aws.typepad.com/aws/2012/11/asia-pacific-sydney-region-open.html
<wallyworld_> \o/
<jcsackett>  morning all.
<czajkowski> jcsackett: ello
<rick_h> morning jcsackett
<czajkowski> grabing some lunch bbiab
<jcsackett> abentley: i know you've got a call in a few moments. any chance you can fill me in on the mismatch card before then?
<abentley> jcsackett: Sure.  Actually, the call's in an hour.
<jcsackett> abentley: ah, ok.
<abentley> jcsackett: Chat in the standup hangout?
<jcsackett> sure.
<czajkowski> rick_h: can you pop into lp please
<rick_h> czajkowski: sure thing
<czajkowski> thanks
<rick_h> yay IE tests passed on first try woot!
<czajkowski> whoo
<czajkowski> good start to the week rick_h
<rick_h> jcsackett: you fueled up for another review?
<rick_h> https://code.launchpad.net/~rharding/launchpad/wire_new_banner/+merge/133927 or sinzui/deryck
<jcsackett> rick_h: in the middle of a test case; i can look at it in a few though.
<jcsackett> (just don't want to lose my train of thought)
<rick_h> jcsackett: not a problem, thanks
<abentley> bac: I've made some tweaks to the branch you reviewed on Friday.  Do you have a chance to take a look?  It's r16240: https://code.launchpad.net/~abentley/launchpad/optimize-spec-query/+merge/133716
<jcsackett> rick_h: ok, taking a look now.
<abentley> bac: BTW, the merge proposal page is completely lying about the contents of r16240.  It's actually http://bazaar.launchpad.net/~abentley/launchpad/optimize-spec-query/revision/16240
<abentley> rick_h: When you saw these spurious changes that made it look like you were applying abel's code, was it on the merge proposal web page?
<jcsackett> rick_h: this branch is a continuation of the previous one, right? any chance of resubmitting it with that as a prereq to cut down on diff noise? i feel like i'm rereading a lot here.
<rick_h> abentley: yes
<rick_h> jcsackett: it should be cut down on noise as the first branch is landed
<rick_h> jcsackett: the noise in this one is the deletion of the old code, shouldn't be any big adds
<abentley> rick_h: Was it one of the fold-out diffs, or the "Preview diff" at the bottom?
<jcsackett> rick_h: yes, you're right. i'm mixing up when i looked at your branch last week and when i looked at the massive branch.
<rick_h> abentley: preview at the bottom
<rick_h> I was reviewing my diff before doing the MP comments
<abentley> rick_h: Okay, never mind then.
<abentley> deryck: Could you please have a look at my follow-on changes as we discussed on the call?  http://bazaar.launchpad.net/~abentley/launchpad/optimize-spec-query/revision/16240
<deryck> abentley, sure
<abentley> deryck: Thanks.
<jcsackett> rick_h: r=me, with some comments.
<rick_h> jcsackett: ty lookimg
<rick_h> err ing
<deryck> abentley, looks good to me.
<abentley> deryck: Thanks.
<deryck> np!
<rick_h> jcsackett: good calls, thanks for the catches
<jcsackett> rick_h: you're welcome.
<rick_h> deryck: ping
<deryck> rick_h, otp
<rick_h> snap-l: deryck np
#launchpad-dev 2012-11-13
<wgrant> wallyworld__: I've reviewed your branch
<StevenK> "And it's all rubbish"
<StevenK> :-P
<wgrant> Not quite :)
<wallyworld__> wgrant: thanks, i'll look after i get back from school pickup
<wallyworld__> wgrant: i can't use rs.find().set() because that doesn't return the rowcount
<wgrant> Ah
<wallyworld__> also, i can't include lpsprcdateuploaded < dateuploaded in the where condition, i have to use greatest() so rowcount correctly says if that row exists or not
<wgrant> wallyworld__: You may be better off doing an initial SELECT to work out whether you need to UPDATE or INSERT
<wallyworld__> that's an extra select i'd rather avoid
<wallyworld__> unless you think it won't affect things
<wgrant> But that turns each batch into 4 queries
<wgrant> Currently you're doing 2+n queries
<wallyworld__> hmmm. i do i update for creator != null, 1 for maintainer != null, and a possible insert
<wgrant> (aha, my first test run on DF crashed with the INSERT conflict that I mentioned)
<wgrant> Right
<wallyworld__> as well as the initial select
<wgrant> You do the initial select, two updates for each row, and an insert
<wallyworld__> no, a possible insert
<wgrant> I'd suggest instead the initial select, some computation, a batch select, a batch update, and a batch insert
<wgrant> True
<wallyworld__> hmmm. you really think that would be better?
<wgrant> It's going to reduce the query count by a factor of 1000 or more
<wallyworld__> most times the insert wil not be required once the table is populated
<wgrant> Sure, but the insert is negligible
<wgrant> Because it's batched
<wgrant> Its elimination does not help performance to any measurable degree
<wallyworld__> so you want to initially select from lpsprc where spr_id in (all the spr ids from the slice)?
<wallyworld__> and see what archive/spn/distroseries records exist
<wgrant> Right
<wgrant> Otherwise you can't identify which rows need inserting unless you do a thousand separate UPDATEs for the batch
<wgrant> Which is obviously substantially more expensive than a single SELECT
<wallyworld__> my thinking was most times all the updates would be required anyway
<wallyworld__> once the table is populated
<wgrant> But then you have 1000 queries instead of 1
<wgrant> Still
<wallyworld__> i mean, if the updates are required, you still have 1000 queries
<wallyworld__> plus the initial computation as an overhead
<wgrant> Not with my technique
<wgrant> You can do a batch update using either a few CASE expressions or a literal subselect
<wallyworld__> i'd have to think a bit about that
<wallyworld__> i could insert the records to be updated into a temp table. or you suggesting  a sub select from the original spph table?
<wgrant> UPDATE lpsprc SET dateuploaded = sub.dateuploaded FROM (VALUES (some, cache, row, key, that_dateuploaded), (some, other, row, key, other_dateuploaded)) AS sub WHERE lpsprc.some = sub.some AND [...] AND lpsprc.dateuploaded < sub.dateuploaded;
<wallyworld__> ok, thanks, i'll see how that works
<wgrant> (it's not a pattern we use anywhere else yet, because we rarely have code adequately architected to be able to benefit from a bulk update)
<wallyworld__> yeah
<wgrant> This is a garbo job we should try to get Rightâ¢, as it's likely to be used as a model for many more in the future.
<wallyworld__> i haven't used that pattern before elsewhere either
<wgrant> And I don't think it's too much effort to get it Rightâ¢, apart from getting Storm to cope with that UPDATE syntax
<wgrant> But it should be relatively easy to do with some of the helpers in lp.services.database.bulk
<wallyworld__> yeah, no sure if thre would be native storm support for it, may need a bit of SQL("..")
<wgrant> There isn't native support AFAIK. You'll probably just need to define a VALUES stormexpr near your helper and then do a bit of SQL() with column definitions maybe. But the hideousness can be encapsulated in lp.services.database.bulk, as the insert/load horrors are
<wallyworld__> i'll see what falls out
<wallyworld__> fun fun fun
<wgrant> Storm has the ROW constructor nowadays, but the only use of VALUES expressions that I know of is the INSERT special case
<wgrant> wallyworld__: btw, I've confirmed on DF that my suggested optimisations for the getPendingUpdates query are indeed of great import.
<wallyworld__> ok, ta. i'll let you know when it's ready to look at again, will be after dinner and a few errands
<wallyworld__> may well need another iteration
<wgrant> Yeah
<wgrant> But this is looking very good, and I think the correctness concerns I had have been ironed out.
<wgrant> and with the bulk update this will be extremely fast, which is good because I was banking on the "consider more rows, but calculate and process them far more quickly" strategy.
<wallyworld__> hopefully this pattern could be applicable to other similar reporting type scenarios where we do live calculations off a fairly normalised model
<wgrant> wallyworld__: Exactly
<wgrant> wallyworld__: This effort probably isn't completely worth it just to fix those pages, but it is the first of a new style of caching table that has a huge number of potential uses.
<wgrant> This is our first table with eventual consistency and just a few minutes of lag.
<wgrant> Everything else we have is either updated by triggers or a huge daily cronjob :)
<StevenK> We should destroy nightly.sh for being a terrible idea and making a lot of people very angry
<stub> In favour of?
<stub> smaller celery tasks on demand might be nicer
<wallyworld__> afternoonly.sh
<stub> even if it would mean more overhead
<stub> manyana.sh
<adeuring> good morning
<cjwatson> wgrant: Can you think of any reason why Archive._copySources (for syncSource/syncSources) couldn't just pass unembargo=True as a direct replacement for allow_delayed_copies=True?
<cjwatson> It seems too eay
<cjwatson> *easy
<wgrant> cjwatson: As long as it's been fixed to not reupload then it should be fine
<cjwatson> Yeah, I destroyed the reuploading a while back
<wgrant> Yeah, I thought I recalled that
<cjwatson> I guess I have to destroy delayed copies in two pieces, to avoid a situation where we created a delayed copy just before a rollout and then have no way to publish it
<wgrant> Yeah
<cjwatson> And we might have to keep a tiny amount of code around for the sake of old entries in DONE queues and the like
<wgrant> cjwatson: Yeah, but most of the code can die
<adeuring> frankban: could you please review this mp: https://code.launchpad.net/~adeuring/launchpad/product-lp-limitedview-3-1/+merge/134075 ?
<frankban> adeuring: sure
<adeuring> frankban: thanks!
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: frankban | Firefighting: - | Critical bugs: ~200
<wgrant> adeuring: LimitedView delegates to View by default, so you probably don't need to check for View at all
<adeuring> wgrant: ah, right, let me try this
<wgrant> The ifs will still be a little ugly, but it's at least slightly less bad
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: frankban, rick_h | Firefighting: - | Critical bugs: ~200
<czajkowski> rick_h: morning
<rick_h> morning czajkowski
<czajkowski> rick_h: I come with a gift :) https://bugs.launchpad.net/launchpad/+bug/1078239
<_mup_> Bug #1078239: loading team blueprints leads to timeout Person:+index <fallout> <lp-blueprints> <privacy> <private-projects> <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1078239 >
<rick_h> czajkowski: ok, https://bugs.launchpad.net/launchpad/+bug/1075365 is in qa mode so we should hopefully have that ready for a deploy in a couple of hours
<_mup_> Bug #1075365: Timeout when trying to visit previous sprint pages <fallout> <lp-blueprints> <private-projects> <qa-needstesting> <regression> <timeout> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/1075365 >
<rick_h> and if that doesn't fix it...well then back to the code
<czajkowski> rick_h: great
<rick_h> czajkowski: so card is added to our board and we'll make sure to follow up.
<jcsackett> morning all.
<rick_h> party
<abentley> morning.
<jml> hi
<czajkowski> sinzui: what does the tag specification mean?
<sinzui> The error involves the specification object
<rick_h> deryck: here's the test case I'm working on with the matchers https://pastebin.canonical.com/78291/ and the html content appears in line 151 of the test failure output
<rick_h> which is the browser.contents
<rick_h> I moved that <noscript> outside of the target in case the contents were what was messing up the match (#150)
<deryck> rick_h, ok, just one second and I'll look....
<rick_h> deryck: np
<deryck> rick_h, looking now.....
<rick_h> hah https://pastebin.canonical.com/78298/
<deryck> rick_h, so it does get a match but then the test fails?  Is that what your "hah" paste there suggests?
<rick_h> deryck: no, it fails to find a match when the only thing I send (via pdb in there) is the exact string that the Tag() is setup for
<deryck> rick_h, ah, I see now.
<rick_h> I would expect that second paste to find a match since it's nothing but the matching tag?
<deryck> rick_h, I wonder if it's tripping over underbars, since we normally use hypens.  and we've just never hit this before because of convention.
<rick_h> deryck: ok, will change the Tag and try it that way
<deryck> rick_h, I don't really see anything odd, though. Like you said on stand-up, it's pretty straight forward.
<rick_h> yea, wasn't expecting to lose my morning when I saw the test failure, but I must be doing something wrong :/
<rick_h> deryck: ah crap, you know what. There's a second test taht failed
<rick_h> it only had one error in the email, but I 'fixed' that first test and a second one came out. That error message is from a later test
 * rick_h does a giant *DOH*
<deryck> rick_h, ah, ok.  np.  so you need me to look at the second test too?  Or you're okay now?
<rick_h> no, we're ok. I just missed that the failure moved on me
<rick_h> thanks for looking
<deryck> np!
<abentley> deryck: Since access policies are calculated based on sharing policies, letting artifacts have information types forbidden by their policies means they can become inaccessible.
<abentley> deryck[lunch]: e.g. if the sharing policies change to forbid Proprietary, the access policies for Proprietary will be removed, and any Proprietary artifacts will become inaccessible.
<sinzui> frankban, rick_h, do either of you have time to review https://code.launchpad.net/~sinzui/launchpad/unicode-project-group-bug/+merge/134154
<rick_h> sinzui: sure thing
<rick_h> sinzui: actually I've got a sprite question for you as well
<sinzui> okay
<rick_h> sinzui: I'm working on adding private images to other tables of data starting with the +upcomingwork http://uploads.mitechie.com/lp/work_items.png
<rick_h> I started to add another column much like is done in the branch listing
<rick_h> but ugh and ugh
<sinzui> yes
<rick_h> and I noticed that in some places it's done with the sprite private
<rick_h> but then that would replace the current sprite of the blueprint/project in the upcoming work table
<rick_h> are there any rules for this?
<rick_h> I was also trying to work out a way to build the list of classes to have both sprites, but it seems to be done adding a new method on the view to build the css based on the current context and double ugh
<rick_h> and not sure if we can stack sprites, haven't look that far yet
<sinzui> We cannot stack sprites if you mean show more than one for an item
<rick_h> sinzui: right ok. So then should I be working on adding another column to hold the private indicator or swap out the default sprite for a private sprite?
<sinzui> The badge and column approach is "canonical", and the "canonical" way does not scale and the layout is more by accident
<sinzui> I am happy with any solution that does not require tables
<rick_h> heh, well the tables are already there so hacking into there.
<rick_h> I suppose I can append the image to the end of the actual string of text that links to the blueprint/product
<rick_h> so they won't line up, but will actually look associated with the right bit of text
<sinzui> I like the suggestion of having an alternate sprite for private items and the "private blueprint" class can do this if we place it in the right place in our sprite css
<rick_h> where the screenshot shows that it's going to look joined with the wrong content
<sinzui> I have a zany idea
<rick_h> heh, I'm up for zany. Especially if it looks uncrappy :)
<sinzui> I think we have a large inline sprite file. We can create an image with two icons per row and adjust the left padding to ensure both show for the css selector
<sinzui> oh, I am wrong
<rick_h> we'd need a class/icon for every combo of private:item
<sinzui> We must have factored out the large inline sprites
<rick_h> ah
<sinzui> we could do this, and I think we can place them in block-sprites-1.css.in
<rick_h> ok, so the idea then is to add new sprites that combine branch, product, blueprint, bugs with private and add a new css class for those that's .sprite.XXX.private with the new double wide sprite image?
<rick_h> and add those css rules into block-sprites-1.css.in?
<sinzui> rick_h, yes
<sinzui> rick_h, I suppose you realise that this issue is really cause by the single-background image limitation in CSS. If we used a sprite font and css-content, we could stack all we want
<rick_h> sinzui: yea, which is why I was trying to see how the private was indicated in other places like the branch badge setup
<rick_h> sinzui: but yea, sprite would be preferred so would be good to add them and make updating the UI eaiser in the long run/lower maint.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: rick_h | Firefighting: - | Critical bugs: ~200
<sinzui> we never get proper alignments with floats and/or wraps. only a font could provide what we want
<rick_h> heh, font is nice but ugh on the work to do that
<deryck> abentley, can we change things such that if you change your sharing policy but have any remaining artifacts of a different info type, then we don't remove the access?
<deryck> abentley, or should we forbid the change until you reconcile the issues/
<deryck> ?
<abentley> deryck: Certainly we can consider existing artifacts when changing access policies.
<abentley> deryck: I'm still chewing over whether we want to permit artifacts to vary from the policies.
<abentley> deryck: With my branch, we'll be essentially refusing to change the policy if there are any existing artifacts.
<abentley> deryck: But only in the limited circumstances of changing the project information type from public to proprietary.
<deryck> abentley, right. so we need to think on it and settle on what to do in the other cases.
<abentley> deryck: So I wonder whether we should make that a more general rule.
<deryck> abentley, right, that was my initial thought.
<deryck> abentley, but I don't mind if we take a day or two to think/talk through the implications of that change.
<abentley> deryck: Cool.
<rick_h> sinzui: do you know if there's somewhere we have the rest of the original sprite images? blueprint, private, produt/project aren't in the images/src dir.
<sinzui> yuck
<rick_h> I can just stack the .png would prefer to do the .svg if we have them
<sinzui> I don't know where those sources would be. rick_h early Lp art was contracted and we didn't always get sources
<rick_h> sinzui: ok cool, not a problem. Just want to make sure I'm not blind :)
<deryck> rick_h, hi there.  can you review a branch for me?
<rick_h> deryck: sure thing
<deryck> rick_h, thanks! https://code.launchpad.net/~deryck/launchpad/top-level-bug-search-and-privacy-1069895/+merge/134164
<rick_h> heh, never would have thought that tales.py would do link construction for the images/badges/sprites
<rick_h> deryck: r=me with a couple of questions and one comment
<deryck> rick_h, thanks!
<rick_h> deryck: are you up for a quick small review please since I'm OCR? https://code.launchpad.net/~rharding/launchpad/security_banner_107842/+merge/134187
<deryck> rick_h, sure
<deryck> rick_h, r=me
<rick_h> deryck: ty much
<deryck> np!
<wgrant> wallyworld__:             ss_clauses.append(In(
<wgrant>                 Row(BugTaskFlat.distribution_id,
<wgrant>                     BugTaskFlat.sourcepackagename_id),
<wgrant>                 Select((SS.distributionID, SS.sourcepackagenameID),
<wgrant>                        tables=[SS])))
<wallyworld__> wgrant: https://pastebin.canonical.com/78343/
<wgrant> wallyworld__: launchpad_dev=# SELECT * FROM (VALUES (1, 2, 3), (4, 5, 6)) AS foo(bar, baz, quux);
<wgrant>  bar | baz | quux
<wgrant> -----+-----+------
<wgrant>    1 |   2 |    3
<wgrant>    4 |   5 |    6
#launchpad-dev 2012-11-14
<wallyworld__> wgrant: i've pushed the latest changes, seems to work ok
<wgrant> wallyworld__: /me looks
<wgrant> Sorry, was at lunch
<wallyworld__> ok
<wallyworld__> np
<wallyworld__> wgrant: you had a look at the garbo job?
<wgrant> wallyworld__: Sorry, fighting with a combination of Unity, fglrx, and Storm
<wgrant> wallyworld__: It looks good, although lp:~wgrant/launchpad/another-reporting-cache-garbojob-1071581 has a significantly shorter version.
<stub> wgrant: Your security.py changes landed? -  2012-11-13 01:19:45,892 INFO Â  Â Outage complete. 0:00:00.755889
<wgrant> stub: Yeah
<wgrant> stub: It sped up prod by not as much as expected :(
<wgrant> Only a bit more than it did staging
<stub> But little more can be done with the scripts, which is good. Any other improvements need to happen in pgbouncer or our PG setup if we can identify the issues
<wgrant> Yeah
<stub> unlikely to be worth it I think
<wgrant> We have a -v run from prod with my changes, somewhere
<stub> still haven't got the basic pgbouncer patch accepted upstream :-/ No responses when I brought it up on the mailing list.
<wgrant> Ah, you brought it up on the ML?
<stub> hmm... might have posted from the wrong address and got silently spamfiled
<wallyworld__> wgrant: just got back from school pick up, looking now
<stub> oh, no. it is there. http://permalink.gmane.org/gmane.comp.db.postgresql.pgbouncer.general/945
<wgrant> wallyworld__: Just pushing some further cleanups (and a fix to my changes)
<wallyworld__> ok
<wgrant> I think I've beaten Storm sufficiently near death that it's not completely hideous.
<wgrant> The only thing that needs improving before it's genericised is typecasting in the VALUES expression, but it's easy enough to work around here
<wgrant> Now I can actually read it, I will recheck correctness :)
<stub> Is this in Storm or Launchpad?
<wgrant> LP for now
<wgrant> Once it's stable it'll go back to Storm
<wgrant> stub: This is a bulk UPDATE helper, which in postgres can get a liiiittle messy
<stub> We should railroad through our other patch too. Given the speed of new Storm feature work, the argument that our mods could cause problems is a bit flaccid.
<wgrant> Heh
<wgrant> wallyworld__: Won't this think that a key already exists in the cache table if there's any row for that (archive, distroseries, spn), without regard for the creator or maintainer?
<wallyworld__> wgrant: it loads those and then constructs map keys based on creator and maintainer for in memory processing
<wgrant> wallyworld__: Oh right, I'm stupid, don't mind me :)
<wallyworld__> maybe i could tighten the filter a bit
<wgrant> That's going to be slightly excessive in terms of loaded objects, but it shouldn't matter too much
<wgrant> Though it's also not difficult to fix
<wallyworld__> i couldn't be bothered doing a Values() class first up, but it's there now
<wgrant> It means we can reuse _dbify_value, which simplifies the compilation of the values a lot
<wgrant> Otherwise I wouldn't have bothered either
<wgrant> But I basically just lifted the code from my bulk INSERT work
<wallyworld__> ah, ok. i lifted the code from Update() to get the core BulkUpdate SET compiling
<wallyworld__> so you changed that to allow for the dbify stuff
<wgrant> I changed bulkupdate not because of the dbify stuff, but because I needed it to include the table name in the RHS of each SET expression
<wallyworld__> looks good to me. i like the ClassAlias stuff
<wgrant> Previously it always used the COLUMN_NAME compilation mode, which meant it said "SET sourcepackagerelease=sourcepackagerelease", which has an ambiguous RHS
<wgrant> That means Storm's compile_update is buggy, but people don't tend to do complex UPDATEs in Storm
<wallyworld__> yeah
<wallyworld__> i exported Values in stormexpr, that is my only change i think
<wgrant> The ClassAlias stuff is a fairly gross but very effective and relatively attractive hack
<wgrant> Ah yeah, that too
<wgrant> Also, the stormexpr import in garbo is in the wrong place
<wallyworld__> except the test fails
<wgrant> Forgot to fix that
<wallyworld__> i fixed that import
<wgrant> I assumed it was just failing due to my test DB being out of date
<wgrant> What's the failure?
<wallyworld__> test_garbo hides the failure sadly
<wallyworld__> i'll debug it
<wgrant> Ah
<wgrant> So yeah, I learnt some interface-violating Storm tricks during my bulk-insert, bugtaskflat, bugsummary, and targetnamecache arcs which hopefully make this a bit more pleasant
<wallyworld__> we really should commit out stormexpr stuff to storm and get to using trunk
<wallyworld__> s/out/our
<wgrant> The main delta between Storm trunk and our branch is the With stuff
<wgrant> The stormexpr stuff can all be easily upstreamed independently of that
<wallyworld__> there's a datettime related diff too?
<wgrant> Oh yeah
<wgrant> Storm trunk Date columns now return a datetime.date, rather than a datetime.datetime
<wgrant> There's a bit of test fallout that we opted to avoid sorting out immediately
<wallyworld__> right
<wgrant> It seems like our BulkUpdate could actually be merged straight into upstream's Update
<wgrant> By just making Update take a tables=
<wgrant> But later :)
<wallyworld__> Update does take a tables already I think?
<wallyworld__> unless i misremember
<wgrant> It takes a table and a default_table
<wgrant> I do not know what the difference is
<wallyworld__> me either, haven't dug that far down
 * wallyworld__ afk, bbiab
<wgrant> It doesn't have an awful lot of callsites
<wgrant> In Storm itself there are 0
<wgrant> wallyworld__: Ah, the job works better when I haven't completely broken inserts
<wgrant> As it turns out
<wgrant> wallyworld__: You'll need a new index if you continue with the (archive, distroseries, spn) filter, but other than that it works fine on DF
<wgrant> 2012-11-14 06:28:20 DEBUG2  [PopulateLatestPersonSourcePackageReleaseCache] Done. 60852 items in 140 iterations, 283.422512 seconds, average size 434.659565 (214.705383185/s)
<wgrant> Which is not bad for DF
<wgrant> Ah, damn, we need more casts
<wallyworld__> wgrant: just merged your latest stuff, got a conflict, resolving
<wgrant> wallyworld__: Yeah, and even your version of the BulkUpdate breaks if it ends up UPDATEing a single row, as it infers the NULL creator or maintainer to be text, not integer
<wgrant> Testing a fix now
<wallyworld__> ah ok, thanks
<wgrant> I guess it can't really infer all the way up in an UPDATE, unlike an INSERT
<wgrant> wallyworld__: OK, fix with explicit types pushed
<wallyworld__> ok, i was still getting test failures with the latest prior to that
<wgrant> Now restarting unity again...
<wallyworld__> :-(
<wgrant> <3 fglrx
<wgrant> gnome-terminal no longer feels like I am SSHed to the moon
<wallyworld__> wgrant: the update is broken
<wallyworld__> update sql
<wallyworld__> i don't think it likes the ::integer syntax
<wallyworld__> if i run it in an sql editor it asks for a value "for parameter :integer"
<wgrant> What's the generated SQL?
<wgrant> The test passes here
<wallyworld__> https://pastebin.canonical.com/78351/
<stub> wallyworld__: ::type syntax is PG specific, it might be confusing your editor
<wallyworld__> bin/test fails though
<stub> It is missing quotes around that date string
<wallyworld__> and with sql logging it stops just after trying to execute the update
<wgrant> wallyworld__: That's not valid SQL
<wgrant> Did you get that from LP_DEBUG_SQL=1?
<wallyworld__> yes
<wgrant> I have a patch for that
<wgrant> sec
<wgrant> The existing one just uses python string formatting to do it
<wgrant> Rather than psycopg2's method
<wgrant> so you end up with None instead of NULL
<wallyworld__> yes
<wgrant> u'FOO' instead of E'FOO', etc
<wgrant> http://pastebin.ubuntu.com/1357406/
<wgrant> I might land that soon
<wallyworld__> i didn't notice the None first up
<StevenK> Hmm, BulkUpdate. That might be awesome for IBug's method to change information type
<wallyworld__> but the garbo test still fails for me sadly
<wallyworld__> so there's something else not quite right
<wgrant> wallyworld__: Odd, it passes here. Possibly a mismerge?
<wallyworld__> perhaps. there were several conflicts, so i copied your entire garbo.py
<wallyworld__> i'll try again
<wgrant> (the problems with the SQL you pasted are just the None and the unquoted timestamps, AFAICT)
<wgrant> Hm
<wgrant> Your stormexpr.py is odd
<wgrant> UPDATE LatestPersonSourcePackageReleaseCache SET sourcepackagerelease=sourcepackagerelease,
<wgrant> publication=publication,
<wgrant> My version doesn't do that
<wgrant> date_uploaded=date_uploaded
<wgrant> The RHS of those SET expressions is meant to be qualified with a table name
<wallyworld__> yeah
<wallyworld__> the None vs NULL is a logging issue as you say
<wgrant> Oh, also, while you're looking at compile_bulkupdate, the 'state.push("context", EXPR)' should just be 'state.context = EXPR'
<wgrant> Won't make a difference since it's the root expression, but it's slightly wrong
<wallyworld__> my stormexpr was slightly out of date i think
<wallyworld__> test passes
<wallyworld__> wgrant: so i've just pushed changes to the mp, incl that state.context change. test passes too
<wgrant> Great, will approve
<wallyworld__> thanks for all the help making it even better
<wallyworld__> we can add the index tomorrow i guess if required
<wgrant> We'll certainly need it
<wgrant> As otherwise it has to seqscan
<wallyworld__> but should be ok to land now so that we get data into the table
<wgrant> Not really. I can get the index added tonight if you want
<wgrant> But we can't sensibly deploy this without the index
<wgrant> But r=me, and thanks for obliging my experimentation
<wallyworld__> np.
<wallyworld__> let's do the index tonight
<wallyworld__> i'm keen to see this land
<wallyworld__> and data to be there
<wgrant> Yep
<wgrant> Do you have time to do the branch, or should I?
<wallyworld__> i am going out for dinner soon but can do it when i get back. we can do it in devel right?
<wgrant> Yep
<wgrant> Then get someone to apply it live
<wallyworld__> s/someone/some poor webops
<wallyworld__> so just an index on (archive, distroseries, spn)
<wgrant> Right
<wallyworld__> ffs, db-devel broken
<adeuring> good morning
<wallyworld__> wgrant: https://code.launchpad.net/~wallyworld/launchpad/report-cache-index-1071581/+merge/134274
<wgrant> wallyworld__: Index names are in the public schema scope, not a particular table, so we usually prefix them with the table name
<wgrant> sometable__some_col__some_other_col__idx
<wallyworld__> ok
<wallyworld__> wgrant: name os too long - says it will be truncated, is that ok?
<wgrant> wallyworld__: It's better to have the end cut off than the table name, but sometimes we'll use abbreviations in the column namesa
<wallyworld__> ok, will abbreviate the col names
<cjwatson> wgrant: It's a minor point, but do you have any feeling on the logging question I asked at the end of the comments in https://code.launchpad.net/~cjwatson/launchpad/always-copy-packages-asynchronously-2/+merge/131928 ?
<wgrant> cjwatson: We historically ran scripts on production at INFO for not very good reasons
<wgrant> I tend to tweak them up to DEBUG when I have cause to touch their crontab lines
<wgrant> I'd suggest that might be a good idea here
<cjwatson> Righto
<wallyworld__> wgrant: diff updated, index still about 6 chars too long, but i guess it will do
<wgrant> wallyworld__: Right, r=me
<wallyworld__> thanks
<wgrant> wallyworld__: This should be the last branch apart from the one to remove the old inline calculation, right?
<wallyworld__> bloody hope so
<wgrant> Heh
<wallyworld__> but there will be one to remove feature flag
<wallyworld__> stub: could you possibly apply this index to qastaging? https://pastebin.canonical.com/78357/
<wgrant> That will presumably be the one that rips the old stuff out
<wallyworld__> yeah
<stub> wallyworld__: Got a patch number to go with that? Or are we going to remove it again in a tick?
<wallyworld__> 2009-38-2
<stub> 2009?
<wallyworld__> 2209 sorry
<stub> all done on qastaging
<wallyworld__> thanks
<wgrant> wallyworld__: You might want to increase the garbo-frequently log verbosity on qas (and potentially prod; I forget what it's set to there)
<wallyworld__> stub: rev 16267 is landing now (has the patch). when do we apply to prod?
<wallyworld__> wgrant: ok, can a web ops do that
<stub> wallyworld__: whenever you like
<wallyworld__> stub: now would be great
<wgrant> wallyworld__: Yeah
<wgrant> wallyworld__: It's in the crontabs, the --log-file option
<wgrant> You probably want DEBUG2
<wgrant> to get the looptuner logging
<wgrant> With row counts etc.
<wallyworld__> ok
<stub> wallyworld__: it is running
<stub> (will take a bit longer on prod depending on other transactions)
<deryck> rick_h, I moved our call by 1/2 hour later, but the TL call wraps then, too.  So I may be a couple minutes late.
<deryck> rick_h, if that's cool, I'll ping when I get free.
<rick_h> deryck: np, whenever works for me. Thanks for the heads up
<rick_h> deryck: just ping when you're free
<abentley> deryck: I just created an embargoed project, and it has the correct policies: https://qastaging.launchpad.net/asdff/+sharing
<deryck> abentley, gah!  I wonder what happened differently for me.
<abentley> I don't know.  I doubt it's a difference between doing it locally and doing it on qastaging.
<abentley> deryck: You created it with Embargoed, you didn't change it to Embargoed afterwards, right?
<deryck> abentley, right.
<deryck> abentley, trying again now locally to see what happens.
<deryck> abentley, hmmm, yeah, no blueprints.  I wonder if it's because I don't have the blueprints flag enabled.
<abentley> deryck: That seems possible.
<deryck> abentley, yeah, that's it.  sorry man.
<deryck> abentley, I'll update that bug to be just about bugs sharing policy then.
<abentley> deryck: +1
<abentley> sinzui: Could we talk about sharing policies?  Doesn't have to be right away.
<sinzui> abentley, sure.
<sinzui> I can talk now if you like
<abentley> sinzui: Cool.  I've invited you to a hangout.
<abentley> sinzui: https://plus.google.com/hangouts/_/31454706a4303baaad1308627a542146c5b83ded?authuser=0&hl=en-GB#
<sinzui> abentley, one moment as I fight an authorisation failed at google
<abentley> deryck: Outcome is: we will support inconsistency between old artifacts and current sharing policies.  This was requested by stakeholders.
<deryck> abentley, ok, sounds fine to me.  we need to do some work to support this, or existing behavior is okay?
<abentley> deryck: We need to do some work.  We need to consider artifact information types when doing AccessPolicy pruning.
<abentley> deryck: It's possible we should also not complain that a project has public artifacts when switching information type to proprietary.
<deryck> abentley, ok.  we need to add a bug/card for the first item.  personally, I think it's good to complain about the mismatch, though. We don't have to block it, if we need to support this.
<deryck> abentley, but a warning is useful.
<abentley> deryck: I think a warning is most consistent with what we promised stakeholders they would be able to do.
<deryck> abentley, sounds good.
<abentley> I will add a bug/card for the first item.
<deryck> abentley, thanks for doing that!
<adeuring> rick_h: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/product-lp-limitedview-branch-view/+merge/134315 ?
<rick_h> adeuring: sure thing
<adeuring> ricthanks!
<adeuring> rick_h: tanks
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: - | Firefighting: - | Critical bugs: ~200
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: abentley | Firefighting: - | Critical bugs: ~200
<rick_h> adeuring: r=me with one typo
<adeuring> rick_h: thanks!
<mgz> rick_h: he has to add a tyop for you to approve?
<rick_h> mgz: requirement, leaves something for me to fix later as a nice simple bug  when I feel like I'm not getting much done :P
<rick_h> mgz: move those cards across the board!
<mgz> :)
<deryck> rick_h, how about meet in 5 minutes?
<rick_h> deryck: rgr
<sinzui> jcsackett, I marked your branch qa-ok. This is my test: https://bazaar.qastaging.launchpad.net/+branch/launchpad/revision/14654
<sinzui> expand all works
<sinzui> wgrant, do you have any insight into https://bugs.launchpad.net/launchpad/+bug/1078796
<czajkowski> sinzui: I thought that was a new bug I created hw did you find the other duplicate?
<sinzui> czajkowski, wgrant agreed with my speculation that the root cause is a naughty process locking the archive...nothing to change it not the web, not the api, not even another process running on our servers
<czajkowski> ok
<sinzui> we have a number of bugs that have this root cause. We think we need to change our scripting architecture to honor timeouts like the web and API does so everyone can share the data
<czajkowski> strange how it only happens some ppas and not all but just noticed it a lot this week given I'm increasing a lot of ppas
<sinzui> czajkowski if the person notices that they are out of space, Lp is accepting and building those packages...so those are the most likely have a lock :(
<czajkowski> ahh
<StevenK> I thought we accepted binary uploads from buildds if they're over quota, but not source uploads
<wgrant> Yes
#launchpad-dev 2012-11-15
<cjwatson> wgrant: dogfood seems to have builders attached at the moment - is there a particular reason I shouldn't start buildd-manager to drive them, or are they busted?
<wgrant> cjwatson: They're not meant to actually be up
<cjwatson> OK, I will pretend I didn't see them
<wgrant> cjwatson: buildd-manager probably hasn't been started since I restored the DB
<wgrant> Since I removed it from upgrade-dogfood-launchpad to avoid having to kill it every time I wanted a page to not timeout
<cjwatson> Commented out in upgrade-dogfood-launchpad, and indeed I haven't seen it running
<adeuring> good morning
<stub> How are we going with LP on Precise? Wondering if I need to back port something
<wgrant> stub: We have no immediate plans to upgrade prod
<wgrant> It'll be next year some time
<czajkowski> adeuring: morning any advice on https://answers.launchpad.net/launchpad/+question/214223
 * adeuring is looking
<czajkowski> adeuring: thank you
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: jcsackett | Firefighting: - | Critical bugs: ~200
<jcsackett> wgrant: my understanding is that we shouldn't be setting a proprietary branch as the productseries branch for a public productseries--you can have branches that are prop/priv whatever, but shouldn't set them as the series branch.
<jcsackett> (from the MP you looked at)
<wgrant> jcsackett: That is breaking several years of existing practice, and puts every internal Canonical project into an illegal state.
<wgrant> It also doesn't make sense
<wgrant> Things can be more private than their container -- that's no problem
<wgrant> The problems arise when something is *less* private than its parent
<jcsackett> wgrant: we're not forbidding proprietary branches for a product (the actual container); just the linked productseries. but your point is interesting. canonical is using proprietary productseriesbranches on public projects? those aren't projects that will be migrated to proprietary?
<wgrant> jcsackett: lp:foo is a productseries branch
<jcsackett> wgrant: granted. that's not directly answering my question. do we have proprietary productseries branches on products that shouldn't be migrated to proprietary themselves?
<jcsackett> and the intent is absolutely to stay that way.
<wgrant> jcsackett: Yes. The U1 and LS servers, for example, have proprietary code but are otherwise public
<wgrant> Launchpad until 2009/07/21
<wgrant> etc.
<jcsackett> wgrant: ah yes. of course. i am not thinking clearly.
<jcsackett> wgrant: ok, i'll bring this up on our standup. this is meant largely to handle transitional issues, but i think there are edges that need more sorting.
<wgrant> Indeed
<deryck> jcsackett, ping for standup
<jcsackett> deryck: having plus problems--restarting the plugin. be there in moments.
<deryck> jcsackett, no worries
<deryck> abentley, adeuring, rick_h, jcsackett let's try http://tinyurl.com/orange-standup
<deryck> heh
<deryck> hey! bye!
<abentley> deryck: Is it dying for everyone else, too?
<deryck> abentley, no, just you and jcsackett
<jcsackett> we're cursed. :-P
<sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/missing-potmsgset/+merge/134483
<rick_h> adeuring: ping, do you know if the card you added I'm working on came from a bug somewhere? "filter private products in ProjectView.untranslatables"
<adeuring> rick_h: no, there is no bug or oops. I simply looked for methods that return products
<rick_h> adeuring: ok cool
<adeuring> rick_h: might be even worth to check if this method is used at all ;)
<rick_h> adeuring: yea, it's used but it's just projectgroup.products
<rick_h> which is already filtered and there's tests for that property with non-public projects
<rick_h> so just wanted to make sure I didn't need to add a note anywhere else
<rick_h> e.g. bug
<adeuring> rick_h: ah, ok, then just drop the card :)
<rick_h> so I'll note in the card the findings and move to done so we keep track that we looked there
 * sinzui thinks project groups listing project work fine now
<rick_h> yep, should be good awesome card! :)
<deryck> a lot of the cards remaining are like that.... i.e. just check to make sure we're safe in that area of the code/site.
<maxb> ooi, what is the status of Barry's Lucid Python 2.7 PPA? Is that something LP is still looking at using?
<jcsackett> abentley: you around, or off at lunch? questions from your MP.
<jcsackett> abentley: comments left on MP.
<sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/publishbinary-without-publications/+merge/134527
<jcsackett> sinzui: you caught me just in time. :-) yeah, it's short enough i can take a look.
<jcsackett> sinzui: r=me.
<sinzui> jcsackett, this is also short  https://code.launchpad.net/~sinzui/launchpad/missing-potmsgset/+merge/134483
<jcsackett> sinzui: already did that one. :-)
<sinzui> thank you
<jcsackett> you're welcome. :-)
 * jcsackett lunches
<rick_h> deryck: so build-not is having fun today
<rick_h> out of the last 4 runs, 3 times LP failed to come up and the next to last one was something out of the actual parallel testing layer
<deryck> rick_h, yeah, I wondered just now based on your tweet if we have a serious problem. doesn't seem random now.
<rick_h> http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/294/steps/shell_9/logs/err.html
<deryck> I just forced now, FWIW
<deryck> to try to get my stuff in.
<rick_h> yea, but the lp not coming up is typical
<deryck> grrr, it fails faster than I can force and land my branch.  something seems borked now.
<deryck> I see a reference to bug 504291 in the summary
<_mup_> Bug #504291: DisconnectionErrors (already disconnected) happening again <fastdowntime> <lp-foundations> <oops> <Launchpad itself:Incomplete by stub> <Storm:Invalid> < https://launchpad.net/bugs/504291 >
<rick_h> deryck: well looks like the normal beanch is in an ok state but the db one failed now
<deryck> rick_h, yeah, I'm asking in our ops channel to see if there's something we can do to kick buildbot.
<abentley> jcsackett: Yes, I was on lunch.  Thanks for the review.
<jcsackett> abentley: thanks for responding on the MP.
<jcsackett> rick_h: can i bother you to look at https://code.launchpad.net/~jcsackett/launchpad/filter-private-products-vocabulary/+merge/134542 ?
<abentley> deryck: bug #1078410 can be marked invalid and its card deleted, right?
<_mup_> Bug #1078410: Embargoed Products are not setup correctly when created <private-projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1078410 >
<abentley> deryck: Or were you going to change it to just be about adding embargoed for bugs?
<deryck> abentley, well, we need a bug for BugSharingPolicy.EMBARGOED_OR_PROPRIETARY....
<deryck> abentley, right
<deryck> abentley, or else mark it invalid and add another bug.  either way is fine.
<jcsackett> i think rick_h is not here. deryck or abentley, can one of you review https://code.launchpad.net/~jcsackett/launchpad/filter-private-products-vocabulary/+merge/134542 for me?
<abentley> jcsackett: On it.
<jcsackett> abentley: thanks. :-)
<deryck> yeah, thanks, abentley :)
<abentley> jcsackett: Is test_search_respects_privacy_no_user actually testing the case of no logged-in-user, or an arbitrary one?  I could have sworn setUp gave us the latter.
<jcsackett> the former.
<jcsackett> abentley: it actually looks like the setUp for this testcase doesn't cover an arbitrary user either, but no user vs no user with rights shouldn't be different given the filter clause.
<abentley> jcsackett: Please add a descriptive comment to your tests.  Other than that, r=me.
<jcsackett> abentley: you got it. thanks.
<jcsackett> deryck: can you think of anyway that you would be searching for a milestone where you wouldn't have access to the product? i think all of the milestone access points require you to already access the product.
<jcsackett> been reading through the milestone tests and callsites, and i don't see anything that violates that idea...but a second opinion is welcome.
<jcsackett> actually, i'll just open the above question up to anyone.
<deryck> yeah, sinzui would absolutely know that. :)  But there is the project group milestone case.
<deryck> jcsackett, so you could have access on a group and the group's milestone but an individual project could be private
<deryck> I think this is handled already though
<jcsackett> by group you mean projectgroup, right?
<jcsackett> deryck: yeah, i think that's handled, and i don't believe it's using the MilestoneVocabulary.
<sinzui> I think abentley addressed the cases where milestones from a private project are filtered when instantiating ProjectGroupMilestones
 * sinzui looks on qastaging for early tests
<deryck> right, he did, IIRC
<deryck> jcsackett, and yeah I meant projectgroup.  I was lazy typing. :)
<deryck> jcsackett, and I agree, I can't think of any case where you'd be using the vocab and not have project access.
<jcsackett> deryck: ok, i think i'll move this card out of in progress then, as i think there's no further work to be done.
<deryck> nice!
<deryck> cards are dropping like that all over today. :)
<jcsackett> i'm not sure if rick_h knows it, but we're having a race.
<jcsackett> because he doesn't know it, i think i have a competitive advantage. which is just as well, as it's the only way i'll win. :-P
<deryck> jcsackett, I always go for the win, whether the opponent knows it or not. :)
<jcsackett> deryck: :-P
<wgrant> deryck: Hi, have you verified the future scaling performance of your bug search changes?
<deryck> wgrant, no, I have not.  Are you concerned?
<wgrant> deryck: Very
<deryck> wgrant, and hi, btw.
<wgrant> The work we did earlier in the year was to *eliminate* joins, but this adds three or four more
<wgrant> And it's to handle a circumstance that probably shouldn't ever exist
<wgrant> So it seems like a lot of risk for minimal gain
<deryck> wgrant, yeah, I did consider the extra join could cause problems.  I was going to be careful in qa and if it looked risky back it out.
<wgrant> Well, it *is* risky and it *will* cause some problems eventually, but it's difficult to say on qastaging as bug search timeouts there are often dismissed as cold cache
<wgrant> But I would be extremely wary about making this change
<deryck> wgrant, ok, fair points.  I wouldn't dismiss timeouts as cold cache, FWIW.  I usually look at an OOPS and see what it tells me.  And compare to staging oops even.
<wgrant> The bug search queries were redesigned and tweaked for performance, so making unnecessary changes which quadruple the number of considered tables needs to be considered and tested very carefully
<deryck> wgrant, I've been a bit nervous about this change anyway. for that same reason.  and stuff I saw in tests had me nervous too.
<deryck> wgrant, so I'm fine to just revert for paranoia's sake.
<sinzui> Why are we search for impossible data? privacy is transitive, you cannot ever have public data in a confidential container.
<deryck> well it will be impossible.  it's not currently.
<wgrant> Right, so it sounds like it should be reverted
<sinzui> I think we can fix the data faster than writing code...I bet it can be run in 2.5 seconds in sql by webops
<deryck> yeah, agreed.
<wgrant> (though we really should have been enforcing the privacy invariants before rolling out the beta, it's pretty easy to fix up later)
 * sinzui would do the same with his current bug if he could workout what the right state is :(
<wgrant> sinzui: Which bug is this?
<sinzui> wgrant, https://bugs.launchpad.net/launchpad/+bug/1077351
<_mup_> Bug #1077351: SourcePackageRecipeBuild:+index LocationError build <oops> <recipe> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1077351 >
<wgrant> Aha
<sinzui> I wanted to make the three items be build failed/success, but I need to look for evidence of success
<deryck> wgrant, I'm about to go offline for some family stuff.  but I'll revert when I get back tonight.
<deryck> wgrant, if you get anxious about it, I don't care if you revert it.
<wgrant> Well, there's some interesting stuff I'd like to deploy this morning, so I might revert it now
<deryck> wgrant, sure, feel free.  sorry about that.  just don't have the time to hang around now.
<wgrant> np
<deryck> later, everyone.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~200
<wgrant>  whaawmp        | 0.2.10.1-py24 | 32517 |   899
<wgrant> StevenK: ^^
<wgrant> SELECT product.name, milestone.name, productrelease.id, COUNT(*) FROM product JOIN milestone ON milestone.product = product.id JOIN productrelease ON productrelease.milestone = milestone.id JOIN productreleasefile ON productreleasefile.productrelease = productrelease.id GROUP BY product.id, milestone.id, productrelease.id ORDER BY COUNT(*) DESC LIMIT 200;
<bigjools> good morning
<sinzui> wgrant, are my qa instructions sane: https://code.launchpad.net/~sinzui/launchpad/publishbinary-without-publications/+merge/134527
#launchpad-dev 2012-11-16
<StevenK> wgrant: Apparently, you broke buildbot
<StevenK> I suspect the libav thing is the Host: header, too
<StevenK> But still checking
 * StevenK looks how to register a bugtracker
<wgrant> StevenK: Just let it autocreate
<wgrant> Add a product task on a bug, specifying the URL
<StevenK> It didn't work that well for me, but I think I got there
<StevenK> 2012-11-16 00:31:01 DEBUG   No watches to update on http://bugzilla.libav.org/
<StevenK> :-(
<StevenK> wgrant: I created a project, registered the bugtracker, set the project to report in LP, filed a bug, changed the project details to point to the registered bugtracker and set the watch on the bug
<StevenK> Hmmm, look at that
<wgrant> You might need to manually set the next update time
<StevenK> Yeah, I reset the watch and checkwatches has picked it up
<StevenK> 2012-11-16 00:33:23 INFO    Error updating http://bugzilla.libav.org/: Failed to parse XML description for http://bugzilla.libav.org bugs [u'298']: mismatched tag: line 101, column 4
<wgrant> Ah yes
<wgrant> Bugzilla's XML generation is somewhat interesting
<StevenK> It isn't XML
<wgrant> How far off XML is it?
<StevenK> It's returning an HTML error page
<wgrant> Ahh
<StevenK> Bugzilla has suffered an internal error. Please save this page and send it to Reinhard Tartler with details of what you were doing at the time this message appeared.
<StevenK> Cannot seem to handle bug_id and include together.
<wgrant> Is there anough info in the page to see what's wrong, or do you need to try poking siretart tonight?
<StevenK> The cannot seem to handle along with the code has shown me 'bug_id_type': 'include',
<StevenK> So I'm checking the version parsed out
<StevenK> Right, I got it to work
<StevenK> wgrant: http://pastebin.ubuntu.com/1361616/
<wgrant> StevenK: What does bug_id_type: include do?
<StevenK> But I don't know if that will break every other bugwatch ever
<StevenK> wgrant: I have no idea, I took a guess based on the error message
<StevenK> And Google has failed me
<StevenK> wgrant: http://pastebin.ubuntu.com/1361625/ also works, and it seems Google actually knows about bugidtype a lot more
 * StevenK stabs Bugzilla. Hard.
<StevenK> http://www.bugzilla.org/status/changes.html is not helpful. In the slightest
<wgrant> Wow
<wgrant> That's an impressive page
<StevenK> Impressive, but unhelpful
<StevenK> http://bzr.mozilla.org/bugzilla/4.0/revision/6940/Bugzilla/CGI.pm seems to show that bug_id_type is the new style, but only for version 4.0 and higher
<wgrant> Hm
<wgrant> So bugidtype maps to bug_id_type?
<wgrant> Maybe talk to siretart about upgrading...
<StevenK> We still support bugzilla 2.16 in the code
<wgrant> Hmm
<wgrant> What does annotate say about that arg in our tree?
<StevenK> That's an interesting question.
<StevenK> It was touched by jtv in r12227 as part of fixing a redirect on submit issue, and then it was changed in r5896 by gmb when he refactored the world of external bug trackers
<StevenK> And r5896 was when lib/lp/bugs/externalbugtracker/bugzilla.py starts existing, so I'm not sure where the code was before.
<StevenK> Ah, lib/canonical/launchpad/components/externalbugtracker.py
<StevenK> And that was touched in what turns out to be r3745 by kiko when he added issuezilla support
<StevenK> So it's been bug_id_type in our code since at least 2006
<mwhudson> if it was when kiko was still writing code...
<StevenK> mwhudson: In the deep, dark ages, when dinosaurs still ruled the earth?
<mwhudson> something like that
 * StevenK grumps at bug 1049366
<_mup_> Bug #1049366: Incorrect bug status imported from freedesktop-bugs <bugwatch> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1049366 >
<StevenK> It works locally
<StevenK> wgrant: I've commented on the libav bug, since siretart is subscribed
<StevenK> But I'm not sure about that freedesktop bug
<wgrant> StevenK: Have you checked the prod logs?
<StevenK> They're syncing
<StevenK> wgrant: I can't see that remote id in the logs, but according to the UI it was checked
<wgrant> StevenK: Does the UI give a timestamp?
<StevenK> 3 hours ago
<StevenK> wgrant: Haha. Resetting it has given " Update failed with error 'Unable to set link remote bug to Launchpad' 5 minutes ago"
<wgrant> Aha
<StevenK> *But* it set the status correctly
<StevenK> wgrant: local OOPSes disappear into the ether?
<wgrant> StevenK: Yes. If you kill rabbitmq they'll go into /var/tmp/lperr, or you can run a local amqp->stdout thing
<wgrant> I think there's one around
<StevenK> wgrant: The rabbit that's owned by me, or rabbitmq?
<wgrant> StevenK: Yours
<StevenK> .... WTF does that mean
<StevenK> Fault: <Fault -32000: 'Cannot compare a datetime to a regular scalar at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/DateTime.pm line 1385.\n'>
<StevenK> Oh, is that from the remote tracker ...
<wgrant> Yes
<wgrant> Unless you ported checkwatches to perl
<wgrant> In which case you would be declared to be an enemy combatant
<StevenK> But it might behave better
<StevenK> So, I have no idea about that error, and we haven't had people shooting us
<czajkowski> jtv: ello you about?
<jtv> hi czajkowski
<czajkowski> jtv: have you ever come across a complaint like this in translations, https://support.one.ubuntu.com//Ticket/Display.html?id=25271
<jtv> Grrr auth dance not linking to actual ticket
<czajkowski> no that is a known bug in RT
<czajkowski> pita
<jtv> Reported by "Genghis Khan"...  I thought his son Aga had taken over the business?
<jtv> czajkowski: yes we get this sometimes.  Have a look at the work to get a feel for how real the complaint is.  Assuming it is, contact the accused and tell them to report, clean up their act, or yield the post.
<jtv> If there's no response in a few weeks, change the team's ownership (whoever complains is usually volunteering).
<jtv> If the response is "I'll do my best from now on," set a deadline of a few weeks and check back with the complainer's help.
<jtv> A translation team's job is to manage translations, not to get a monopoly on entering them.
<jtv> And of course, bear in mind that the faceless nature of the interaction between the two parties you're dealing with naturally leads to strife.  Be ready to build bridges if intentions were good.
<adeuring> good morning
<jtv> Hi there adeuring
<czajkowski> jtv: ah ok so kinda like dealing with lcoo teams
<adeuring> hi jtv!
<czajkowski> what do you mean by yield the post?
<jtv> Ownership of the translation team.
<czajkowski> ahhh
<czajkowski> jtv: thanks for the help
<jtv> The owner's first job is to vet applicants for the team _and approve some_.
<jtv> After that, manage the translation process.
<jtv> "Do some translations" doesn't even enter into it â anyone with an account can do that.
<jtv> If anyone volunteers, make sure they understand this responsibility.  Don't mistake "I want to translate" for "I want to manage translation."
<jtv> I think sometimes people start out with "I want to translate," acquire a translation team along the way, and then don't hear about the management side of the job so they ignore it.
<jtv> Having provided czajkowski with bedside reading material for the near future, jtv goes about his way :)
<czajkowski> jtv: cheers
<jtv> :)
<StevenK> wgrant: I guess the p-d-r complaints are due to OMG backlog?
<czajkowski> StevenK: what complaints?
<StevenK> czajkowski: The scriptactivity mails
<czajkowski> ahh
<czajkowski> thought you were referring to an OMG article
<StevenK> p-d-r was turned on after being off for a long while, so a OMG-normous backlog is to be expected
<czajkowski> nods
<czajkowski> so have we resolved the ppa issue as we're turning this on or are they seperate issues?
<StevenK> czajkowski: Turning on p-d-r will resolve the issue. Once it has caught up
<czajkowski> lovely that will make my inbox and irc happy :)
<wgrant> StevenK: Yeah, it's working, just taking a while to catch up
<wgrant> Been going nearly 6 hours now
* wgrant changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~190
<jcsackett> adeuring: MP for review is https://code.launchpad.net/~jcsackett/launchpad/filter-milestone-vocabulary-for-projectgroups/+merge/134667
<adeuring> jcsackett: I'll look
<jcsackett> adeuring: thanks. :-)
<adeuring> jcsackett: r=me, with a remark about getProductPrivacyFilter()
<jcsackett> adeuring: i don't know that we need that--we're only using the function when we're expressly filtering out private products. artifact listings are handled separately when they exist, right?
<jcsackett> probably worth bringing up on monday though, you're right.
<adeuring> jcsackett: yes, the intent is to filter private projects a user can't see -- but people with artifacts grant for a private project would see a private project in the vocab but would not be able to access it.
<jcsackett> adeuring: i may not be understanding you right; the getPrivacyfilter method only checks against the product itself--if you have access to a bug, you still won't get the product. or are you suggesting they should be able to see the product in that instance?
<adeuring> jcsackett: getProductprivacyFilter() uses Select(Product.id, granted_products), where the granted_products clause does not distingiush between records where the column artifact is null and not null, meaning that people will be have products listed that they cannot access
<adeuring> this may or may not be desirable -- i am simply not sure ;)
<jcsackett> adeuring: ah, dig. yeah, there's definitely room for discussion. good catch. :-)
<deryck> sinzui, hi.  so we had some confusion on our stand-up call this morning, I'm hoping you can resolve....
<sinzui> okay
<deryck> sinzui, wgrant reverted my work and I agreed to it, thinking that we couldn't have public bugs on private projects....
<sinzui> you cannot..transitive privacy
<deryck> sinzui, but abentley was under the impression that we wanted to warn but not prevent users from having these mixed policy situations.
<deryck> sinzui, some PES use case for mixed artificats he mentioned after discussing with you
<abentley> sinzui: You have also said that we don't want to have transitive privacy where bugs are private.
<sinzui> We warn when the user is not in the policy and that someone instead needs to agree to grant access for a subscription
<abentley> i.e. userdate
<abentley> s/userdate/userdata
<sinzui> abentley, thank you for calling me out. I mean confidential, not private userdata
<abentley> sinzui: Could you expand?
<sinzui> abentley, non-confidential is not transitive. bugs, branches, and specs can be private with a public context as they has always been
<sinzui> confidential is transitive. you cannot have a public comment or attachment on a confidential bug
<abentley> sinzui: Can you please define confidential?
<sinzui> you cannot have a public branch stacked on a confidential branch...there is a long history of Lp implying it can be done. This is a data/state error. the user see a 403 because Lp enforces that traversal/stacking access rules
<deryck> sinzui, when you say confidential do you mean either proprietary or embargoed projects?
<deryck> sinzui, or do you mean something more inclusive by confidential?
<sinzui> Confidential is Private, Private security, Proprietary, and Embargoed. They are all information types that have sharing rules to restrict access
<abentley> sinzui: Good, wanted to make sure I understood what you meant by confidential.
<sinzui> We designed sharing to take advantage of Zope and bzr stake already does -- make a make a quick check of the user can proceed to the next thing.
<deryck> sinzui, so to make it concrete --  a proprietary project could have proprietary bugs or emabargoed or private but not public....
<deryck> sinzui, and a public project can have any kind of bug.  did I get that correct?
<sinzui> limitedview + subscriptions + artefact grants are an escape clause that stakeholders required so that they could continue to use contractors to work on a small amount of NDA data
<sinzui> deryck, yes
<deryck> sinzui, excellent.  so abentley's work should in fact block you from changing a public project to proprietary if you have any public bugs, rather than just warn.  right?
<sinzui> yes!
<sinzui> We abentley has learned about the nasty migration problem with hwe
<deryck> but we should warn when you have any other confidential information type for bugs if setting a project to proprietary?  trying to understand where the warning idea came in.
<abentley> Sigh.  I wrote it that way originally, but after talking with you, I thought we could not forbid that, since we do not forbid other sharing policy violations.
<sinzui> We wanted to enforce changing all non-compliant artefacts to the declared type, but HWE's has bugs that do not conform...they need to dismantle some projects to separate the data
<sinzui> deryck, We never intend to allow a public project become proprietary. It must be created properly
<sinzui> Teams have the same issue. They really need to be made private. we allow users to try to make a team private, but it often fails because information was leaked when it was setup :(
<sinzui> ...the most common cause of leaks was setting up thing public, so both teams changed registration to ensure things could be setup right during registration
<deryck> sinzui, I don't see why we would prevent that.  we would allow embargoed, right?  why not proprietary?  Any project wanting to use proprietary that already exists would be ruled out.
<sinzui> Because you cannot remove the data from Google or the way back machine
<sinzui> Changing the information type does not undo the leak of a NDA with a OEM/OED
<deryck> true, so it's not truly super secret.  but why would I want an embargoed project just because it's the only private info type I can use?
<sinzui> Proprietary and Embargoed are intend to truly super secret
<sinzui> Public with some proprietary artefacts is the the intermediate project, like ubuntuone-server
<sinzui> Remember we we commission to build this to stop NDA leaks.
<sinzui> There is not NDA that is sort of private
<deryck> sinzui, I understand the use case your wanting to prevent -- i.e. leaks.  What I don't understand is why it matters if someone wants to turn a public project private.  yes, they have no assurance that the name of the thing wasn't already discovered, but they could otherwise work in private going forward, i.e. with future series and milestones.
<deryck> no one will understand that we're blocking private projects because they didn't start out that way.  that makes literally no sense to a user.
<sinzui> deryck, sharing already detects contradictions in policy with the current state of artefacts. If you choose Proprietary only, but there are Private, the sharing overlays list the non-supported state. A garbo job looks at the inconsistencies. When all the artefacts transition to the allowed  creation/transition states, the unused policies are removed from the sharees.
<sinzui> deryck, because Lp/Canonical cannot do what the user thinks will happen.
<deryck> I can assure you that the user has no idea what will happen. :)
<deryck> this notion that proprietary == never ever discoverable is our own invention.  it's only for PES.  other people will want to use private projects and it makes no sense why they can't.
<deryck> and I don't see how allowing others to use the feature undermines PES's concerns.  they can create wholly new proprietary projects and never worry if they desire that level of secrecy.
<sinzui> deryck, If we want to entertain public -> Confidential-type, then you need to write a job that purges all the blocking conditions: remove linked packages, shared bugs, etc...
<deryck> no, I don't think so.  I think we can block until they clean up their data themselves and warn in that condition.  I don't have issue with that kind of soft block....
<sinzui> NO
<sinzui> NO
<sinzui> NO
<deryck> why shout at me?  clearly we're not understanding each other.  but there's no need to freak out about it.
<sinzui> We already had to audit leaks from inclusive teams in trusted roles
<sinzui> Lp must enforce a transition to a state is valid
<deryck> look, I'll make sure we're careful about it.  that's why I'm having this discussion now. please give me a little bit of the benefit of the doubt.
<sinzui> Confidential projects cannot share bugs, have answers, link to packages. Trying to switch such a project MUST fail because we cannot easily unleak
<deryck> that's part of our work... that's what we're doing.  we've already made it where you can turn to private if you have answers.  we block until you fix it.  we're doing work to make sure you unlink branches, or else allow us to do it for you.
<deryck> sinzui, is it your argument that there's just too many of these type of relationships to ever allow it?
<sinzui> You cannot make a private team public if it had a mailing list because the conversations were provided knowing that the information was confidential
<deryck> sure, that makese sense.  that's different, IMHO, to making something public private.
<sinzui> conversely you cannot make a public team private is it has a public ppa...people are subscribed to archive
<deryck> can you not unsubscribe them from the ppa?
<deryck> I'm asking only to follow the trail here.  not saying we should do that.
<sinzui> deryck, as I said you can entertain public => confidential projects, but you need to enforce the transition. You need to scope out a process that cleans up the inconsistencies. I think that is more than a month of work
<sinzui> since you rejected allowing teams to transition because of scope...and there is more need for that then surely you will reject projects. teams
<sinzui> Teams have two tricky transitions...projects have dozens
<deryck> I didn't reject teams, you did. :)  But that decision was made outside the private projects work we inherited.
<sinzui> deryck, Canonical needs to be certain its projects meet our NDA agreements. If project creation is not good enough, we need to fix it
<deryck> those are two different concerns.  project creation is fine.  it works.  it's secure.  how does transitioning from public to private undermine project creation?
<sinzui> We promised teams would be done once private projects were done. They were the last part of discusure
<deryck> I thought team were already done.  I don't follow what
<deryck> what's left to do.
<abentley> sinzui: We already guard against transitioning a product to proprietary if it has linked packages.
<sinzui> Canonical needs to make teams public when their projects become public. It took 2 admins, 2 commercial admins, and 2 PM admins 2 days to do an incomplete unembargoing of webapps after the announcement. Private teams own the projects and the code, so it was impossible for user to get the code
<sinzui> abentley, right, so if Lp says it cannot make the project Proprietary then we have a choice of saying sucks to be you, or building something that can help make the transition possible
<deryck> I think we're all over the place here, sinzui.  just to be issues and concerns from too many ankles.  that's paralyzing.
<deryck> sinzui, so how about this....
<abentley> sinzui: It's not a big "sucks to be you", though.  Just unlink the packages...
<sinzui> abentley, deryck if I create a project, then immediately notice that It was supposed to be Proprietary, I have less than 5 minutes to fix that before Google has indexed every word and team name on that page
<abentley> sinzui: I'm okay with safe-but-annoying, and reducing annoyance where we can.
<deryck> sinzui, to get abentley going again, we do agree that we need to forbid transitions to proprietary if there are public artifact's....
<deryck> sinzui, what we disagree on is whether or not we should forever block that transition.  right?
<sinzui> I am certain the transition would happen...only the bug supervisor could conflict with rules during registration, and I think we only show it if the project is declared to be Proprietary/Embargoed
<deryck> sinzui, likewise, we are uncertain if there is any work we need to do on teams, or how teams relate to these transitions, which you are concerned about it.
<sinzui> deryck, I don't out-right disagree with the transition, I am concerned that the transition wont be safe
<deryck> ok, cool.  I'm also concerned definitely.  common ground! :)
<deryck> but it's just code.  We own it.  We wrote it.  We can change it.  it might be hard, but I don't like the code owning me.  I own the code.
<deryck> sinzui, so I think we can get back to work on it now.  maybe Monday or Tuesday you and I should talk more about teams to make sure I'm clear on your concerns there.  cool?
<sinzui> deryck, we have 3 people on maintenance with 170 criticals. Since Canonical does not want any one on maintenance, I prefer to take less risky steps that ensure 3 people can keep it running, possibly prepare Lp for no developers
<deryck> sinzui, sure, that's a fair argument. But you're assuming that we'll fail.  I'm not so pessimistic.
<sinzui> I am not assuming failure. Since the intent was to remove all maintenance teams this week, I think there our plans to complete work are undermined by business concerns
<sinzui> 3 maintenance staff are a compromise
<jcsackett> can someone point to a product series picker on LP that uses the search for product and series functions? i can't seem to find one.
<sinzui> jcsackett, I don't think there is a series picker.
<czajkowski> sinzui: how do we feel about - SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
<sinzui> jcsackett, milestones let you retarget the series, and the packaging link also retargets. Neither though uses a picker
 * sinzui tries to remember
<sinzui> czajkowski, I search for SIL in reviewed and approved projects and see I have approved it
<sinzui> it is okay
<sinzui> czajkowski, There was a font license that was rejected, but that clearly is not SIL
<czajkowski> nods
<czajkowski> sinzui: thanks
<czajkowski> just doing project reviews now before my EOD
<jcsackett> sinzui: yeah, i think there's a good chance the code in questions isn't actually used properly anywhere.
<sinzui> I suspect is is old.
 * jcsackett nods
<jcsackett> i verfied that nothing that should work as is stopped working and set the bug to qa-ok.
<sinzui> jcsackett, We changed packaging link to a multistep form and that probably factored out the last case
<jcsackett> sinzui: that makes sense.
<abentley> jcsackett: There's no OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/fix-bp-timeouts/+merge/134744 ?
<jcsackett> abentley: sure.
<abentley> jcsackett: Thanks.
<deryck> abentley, I need a review, too, if you don't mind.  This one should be pretty easy.
<abentley> deryck: sure.
<deryck> abentley, thanks! https://code.launchpad.net/~deryck/launchpad/bugs-embargoed-or-proprietary-1079352/+merge/134746
<abentley> deryck: r=me.
<deryck> abentley, awesome, thanks!
<jcsackett> abentley: looks good. r=me.
<abentley> jcsackett: Thanks.
<jcsackett> sinzui: can you add me to the aaron-curtis-test-2 project on qastaging? i need to poke more at the data--i can't replicate in .dev or tests according to our expectations.
<jcsackett> or abentley ^
<abentley> jcsackett: I can't access it.
<jcsackett> abentley: ok. sinzui?
<sinzui> jcsackett, it is shared with you now
#launchpad-dev 2013-11-12
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/slimmer-bfjo/+merge/194767 https://code.launchpad.net/~wgrant/launchpad/even-slimmer-bfjo/+merge/194774
<StevenK> Haha
<StevenK> wgrant: r=me * 2, I look forward to the branch that destroys BFJO
<wgrant> Thanks
<wgrant> There's 7 branches so far
<StevenK> wgrant: slimmer has a question on the approval, but eh
<jtv> Hi chaps
<StevenK> wgrant: So that return ('') is supposed to be a string?
<StevenK> jtv: O HAI
<jtv> All well?
<wgrant> StevenK: Yes. It's a straight move and scheduled for more refactoring soon.
<StevenK> Right
<StevenK> wgrant: You fail at buildbot bing
<StevenK> *bingo
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bfjb-from-bfj/+merge/194805
#launchpad-dev 2013-11-13
<StevenK> wgrant: So SPRBJ and TTBJ will die too?
<StevenK> wgrant: r=me
<wgrant> StevenK: They are the BFJO implementations.
#launchpad-dev 2013-11-14
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/delete-branch-title/+merge/195170
<wgrant> StevenK: Can't you just not override the slot?
<StevenK> wgrant: If I remove the <h1> completly, then the page has no title at all
<wgrant> StevenK: Perhaps page_title vs. label?
<StevenK> I'm just checking that
<wgrant>               <h1
<wgrant>                 tal:condition="view/label|nothing"
<wgrant>                 tal:content="view/label"
<wgrant>                 metal:define-slot="heading"
<StevenK> Yup
<StevenK> wgrant: MP updated
<wgrant> r=me
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/no-mSPRBJ/+merge/195180
<StevenK> wgrant: r=me
<wgrant> thx
<dodiesis> Hi all!
<dodiesis> I need some help in suing the launchpad http API...
<dodiesis> is this the right place?
<dodiesis> s/suing/using/
<wgrant> dodiesis: #launchpad is usually better, but either works.
<dodiesis> wgrant, tnx.. I'll go there..
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/no-bfjo-pls/+merge/195192 https://code.launchpad.net/~wgrant/launchpad/ttb-not-ttbj/+merge/195195
<wgrant> Second one is oversized due to test moves
<StevenK> wgrant: r=me for the first, with a comment about more death. The second will have to wait.
<wgrant> All of the remaining _makeBuildQueue methods are still used
<wgrant> Ah, one is not.
<StevenK> Yes, and they're all seperate WithFactory
<StevenK> So it can die
#launchpad-dev 2013-11-15
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/no-bfjo-ui/+merge/195347
#launchpad-dev 2014-11-10
<kb9vqf> wgrant: Quick question: what script/cronjob is responsible for kicking off the actual package copy jobs?  I tried to use the copy packages feature today but all of the copy jobs are stuck "waiting".
 * kb9vqf has also been picking through the Launchpad source trying to find the package copy job sources without much success
<kb9vqf> Found it: ./cronscripts/process-job-source-groups.py FREQUENT
<kb9vqf> The cron jobs should really be documented somewhere :-)
#launchpad-dev 2015-11-09
<cjwatson> lifeless: hi, would it be possible for there to be a new testtools release soonish?  I'd like to disentangle some more of the bits of yak-shaving related to new Twisted
<lifeless> cjwatson: I thought I had; certainly I can
<cjwatson> That would be lovely, thanks
<blr> morning
<blr> cjwatson: will that put us within reach of moving turnip to py3?
<cjwatson> blr: Maybe, I'm not sure.  Not my current itch :-)
<cjwatson> twisted.web.xmlrpc was ported in Twisted 15.3.0, which IIRC was one of the major blockers, and that is blocked by testtools at the moment.
<cjwatson> So it would certainly bring us closer.  But my direct motivation is the SSH protocol work.
<wgrant> Conch still remains, doesn't it?
<wgrant> I haven't checked in a while.
<wgrant> Also, morning.
<cjwatson> True, I think twisted.conch probably requires porting.
<cjwatson> https://twistedmatrix.com/trac/search?q=conch+python+3&noquickjump=1&ticket=on has a few.
#launchpad-dev 2015-11-10
<maozhou> If the release have not been updated , Can I rebuild the package on launchpad?
<wgrant> maozhou: Can you explain your question in more detail?
<maozhou> wgrant: Can I rebuild the package if  I just change the  chroot  file?
<wgrant> maozhou: A successful build cannot be retried. The only way to rerun the build is to upload a new version of the package.
<maozhou> wgrant
<maozhou> wgrant: ok, thanks :)
<blr> morning
<wgrant> Morning.
<lifeless> o/
#launchpad-dev 2015-11-12
<timrc> So this is interesting.. https://api.launchpad.net/1.0/functools32 returns 404 and some data :)
<cjwatson> that's a little exotic
<cjwatson> oh, I see, that's actually the stringified exception
<cjwatson> so it's perhaps best considered as a not-found page, just ugly because it's api.
<cjwatson> it would be a little clearer if it included the exception name as well as the value
<cjwatson> then it would be "NotFound: Object: ..."
<timrc> cjwatson: Ah that makes sense.
<cjwatson> timrc: I can't immediately find exactly where that's handled, so feel free to file a bug if you want
<timrc> cjwatson: Sure.
<cjwatson> (not, I think, hugely important - returning an explanatory message body with 404 is perfectly OK, it's just that in this case it could stand to be just a little more explanatory)
<timrc> cjwatson: I don't think it's hugely important either.
<cjwatson> though I'm quite sure figuring out how to fix it would be educational
<timrc> Haven't logged into explicitly to file a LP bug in a long while :)
<timrc> in*
<timrc> https://bugs.launchpad.net/launchpad/+bug/607754
<mup> Bug #607754: api 404 has weird content <api> <lp-foundations> <Launchpad itself:Triaged> <https://launchpad.net/bugs/607754>
<timrc> That looks to be the problem, so no need to re-file.
<cjwatson> indeed, thanks
<blr> morning
#launchpad-dev 2016-11-16
<mwhudson> cjwatson, wgrant: i don't know if this sort of nitpicking is useful but https://code.launchpad.net/~gophers/golang/+git/golang (a freshly created import, yay!!) says "Members of The Go Language Gophers can push to this repository."
<mwhudson> oh, uh, i wonder if pointing it at a gerrit is going to be a bit strange
<mwhudson> 2016-11-16 18:46:16 INFO     * [new branch]      refs/changes/50/12050/3 -> refs/changes/50/12050/3
<mwhudson> hope this isn't going to result in 30k branches for launchpad to try to render
<mwhudson> ah no, good
<mwhudson> ah uh, hahaha
<mwhudson> http://launchpadlibrarian.net/293699135/gophers-golang-+git-golang.log -> 2016-11-16 18:48:35 INFO    Unable to set default branch: The remote server unexpectedly closed the connection.
<mwhudson> 2016-11-16 18:49:07 INFO    fatal: cannot exec 'send-pack': Argument list too long
<mwhudson> however master appears to be there so...
#launchpad-dev 2016-11-17
<cjwatson> mwhudson: certainly useful, but would be more useful in a bug report :)
<cjwatson> mwhudson: the "Unable to set default branch" thing is harmless and fixed in devel
<cjwatson> mwhudson: the send-pack error is new to me though, so please file a bug about that as well as about the "can push to" notice
<cjwatson> that's reasonably hilarious, it must actually be git push --mirror that's failing there
<cjwatson> ok, that bit is fixed by git d9dd4cebec3107545b4354a198ecbc5a141f3f46
<cjwatson> so I know how to fix it but would still like a bug for tracking
<cjwatson> for my own reference I think we also need to backport 77cab8af4ad6c25995e40a5ababc7a17dc369fe0 for the fetch side
<cjwatson> wgrant: Could you review https://launchpad.net/~cjwatson/+archive/ubuntu/launchpad/+sourcepub/7140781/+listing-archive-extra ?  Will need to go into precise-cat-lp this time.  Also we'll need to remember that either we upgrade the codeimport hosts all the way to xenial or we need to get the 2.7.3 backport from the Launchpad PPA into trusty-cat-lp.
<mwhudson> cjwatson: yes, fair point
<mwhudson> https://bugs.launchpad.net/launchpad/+bug/1642694
<mup> Bug #1642694: git-to-git imports say they can be pushed to by owner <code-import> <Launchpad itself:New> <https://launchpad.net/bugs/1642694>
<mwhudson> https://bugs.launchpad.net/launchpad/+bug/1642699
<mup> Bug #1642699: git-to-git import from gerrit with many changes "fails" <Launchpad itself:New> <https://launchpad.net/bugs/1642699>
<wgrant> cjwatson: I think that looks reasonable.
<wgrant> Not sensible, but reasonable :)
#launchpad-dev 2017-11-13
<xnox> what is the tool name and it's location which merged bzr branch with the reviewr [r=id] syntax, etc?
<cjwatson> xnox: No single tool.
<cjwatson> xnox: You might be thinking of tarmac, possibly?
<cjwatson> xnox: Or since you're asking here, LP itself uses PQM
<cjwatson> (which is lp:pqm AFAIK)
<xnox> possibly.... something i can run locally that merges an lp branch and generates the commit message of form "merge foo into bla, description, [a=foo] [r=bla]" style message.
<xnox> ack. will look around.
<cjwatson> tarmac is probably the closest to being productised
#launchpad-dev 2017-11-16
<Pegasus_RPG> Hello. I'm thinking about hacking on Launchpad to add the ability to edit comments. Does Launchpad.net run on the trunk version or on some stable milestone?
<cjwatson> very close to tip
<cjwatson> if we've just done a deployment, then it runs exactly on the tip of lp:launchpad
<Pegasus_RPG> Cool. The wiki says to talk about it in here before starting to hack. Are there any pitfalls regarding this issue? https://bugs.launchpad.net/launchpad/+bug/80895 and probably https://bugs.launchpad.net/launchpad/+bug/668267 as well
<mup> Bug #80895: comments on bugs/answers/merge proposals/etc cannot be edited <canonical-losa-lp> <feature> <lp-answers> <lp-bugs> <Launchpad itself:Triaged> <https://launchpad.net/bugs/80895>
<mup> Bug #668267: No way to remove nuclear launch codes from comments on bugs, answers, merge proposals. <canonical-losa-lp> <dhrb> <lp-bugs> <Launchpad itself:Triaged> <https://launchpad.net/bugs/668267>
<cjwatson> have you thought about the social issues?  e.g. what's to stop somebody engaged in an argument on a bug from editing a comment further up and gaslighting people by claiming they never said something?
<Pegasus_RPG> I was thinking just to allow the author to edit their own comments
<Pegasus_RPG> (There are many times I wish I could do that...)
<cjwatson> OK, that's too broad and won't fly on its own.  Matthew's suggestions in #80895 are one possibility, but it does need care and thought
<mup> Bug #80895: comments on bugs/answers/merge proposals/etc cannot be edited <canonical-losa-lp> <feature> <lp-answers> <lp-bugs> <Launchpad itself:Triaged> <https://launchpad.net/bugs/80895>
<cjwatson> (i.e. time-limited)
<cjwatson> Consider also the approach exemplified in https://www.dreamwidth.org/support/faqbrowse?faqid=47 : "You can edit your own comments on any journal, as long as the comments have not yet been replied to. You can't edit other people's comments posted on your entries, or edit comments after they've been replied to"
<Pegasus_RPG> Good point. That sounds fair.
<Pegasus_RPG> (That's how phpBB works too)
<cjwatson> There's also the possibility of leaving the original comment around for inspection, but off to the side, like what happens if you edit a bug description
<cjwatson> That doesn't address the "accidentally posted your house address in a comment" issue, but does address the typo issue
<Pegasus_RPG> The admin hide ability that's already there addresses that though, right?
<cjwatson> Yes, and people can in fact hide their own comments entirely too
<cjwatson> Though I think possibly only on bugs (I can't recall about questions)
<cjwatson> All of bugs, questions, and merge proposals are distressingly separate here
<Pegasus_RPG> So that case (sensitive info) is covered. I just want to have the ability to edit my own comments because I tend to have additional thoughts or notice typos after I post.
<cjwatson> Partially covered, because only bugs (or possibly only bugs and questions0
<Pegasus_RPG> yeah that's a problem too
<Pegasus_RPG> (Why are the three mechanisms separate code-wise?(
<cjwatson> This is quite a big project.  I'd suggest trying to identify smaller pieces that you can tackle in smaller units
<Pegasus_RPG> Sure. I would start just on the bugs since that's what we use the most.
<cjwatson> Aim for each change to come in at 800 lines max
<cjwatson> Partly because somewhat different requirements; partly because they were initially built by different subteams
<Pegasus_RPG> ...and nobody is interested in a giant refactor I'm sure. :)
<cjwatson> They're not totally separate, but certainly not as well-factored as they could be
<Pegasus_RPG> Funnily enough, I haven't yet found out what language LP is written in. I'm assuming Python?
<cjwatson> Yes
<cjwatson> three quarters of a million lines of it
<cjwatson> Not to scare you. :-)
<Pegasus_RPG> Nah, Mixxx is pretty big as well, though it's C++. (We use Python for the Scons build system though.)
<cjwatson> (Some dependencies have bits of C and such for performance, but it's almost entirely Python, built on the Zope stack.)
<Pegasus_RPG> Is LP still actively developed? I noticed last-changed dates on some wiki pages like the roadmap are years ago.
<cjwatson> I apologise in advance for the extremely slow bootstrap process.  It'll get better once we've finished switching to git, but I'm waiting for our sysadmins to find time to get to my buildbot upgrade ticket before we can make further progress on that ...
<cjwatson> You can look at http://bazaar.launchpad.net/+branch/launchpad/changes to see commit dates, which are somewhat more accurate
<Pegasus_RPG> Yeah... bazaar's slowness on LP is the reason we no longer host our code there
<Pegasus_RPG> (That and git's merge wizardry)
<cjwatson> LP had its staffing heavily cut a few years back, and it's certainly a much smaller team than it was, but there are still a couple of us working on it.
<cjwatson> (And several major features that have been developed since that cut)
<Pegasus_RPG> Okay that's good to know. It has its strong points over other code-hosting/collaboration sites which is why we still use it
<Pegasus_RPG> But we do indeed miss the integration of bugs and code/branches
<cjwatson> Right, that was what first sold me on LP as an Ubuntu developer ten years ago or so :)
 * Pegasus_RPG wonders if other sites expose enough info to be able to have LP yank the needed info over their APIs...
<cjwatson> We do import a bunch of stuff.
<cjwatson> Not anything like merge proposals, though.
<Pegasus_RPG> Yeah... the biggest reason we use the other site we do is the ease of doing merge proposals, discussions, comments, etc.
<Pegasus_RPG> It would at least be great if LP could monitor the APIs and auto-link a bug to someone's merge proposal on the other site.
<Pegasus_RPG> But one thing at a time. :)
<Pegasus_RPG> I assume there are plans to have LP work as seamless with Git as it does with Bazaar?
<cjwatson> Pegasus_RPG: It almost does today.  Just a few remaining bits of parity and some tidying up of the merge proposal UI.
<Pegasus_RPG> oh sweet
<cjwatson> I think the only missing features are translations integration, useful subscriptions, and RSS feeds.
<Pegasus_RPG> On that train of thought, I told LP to import our trunk from the other site, and it did, along with the branches, but there's no facility to specify in a series that the code for that series is in one of those already-imported branches. Should I file a bug on this?
<cjwatson> For Bazaar or Git?
<Pegasus_RPG> Git
<Pegasus_RPG> "If the code is already in a Bazaar branch registered with Launchpad link the branch to this series. "   I need that but for Git
<cjwatson> The association with Git branches is somewhat deliberately weaker than with Bazaar.
<cjwatson> Mainly because Git branches are more ephemeral in database terms, and often in practice too.
<cjwatson> Why specifically do you need a hard database-level association (as opposed to e.g. writing something in the series description)?
<Pegasus_RPG> How so? We treat our git branches pretty much like we did Bazaar ones
<Pegasus_RPG> we make them for stable release series as well as bug fixes
<cjwatson> LP doesn't get to intervene when git branches are deleted in the same way that it does when bzr branches are deleted
<cjwatson> So having a hard association with a series is problematic
<Pegasus_RPG> Oh I guess I don't. It would just make it more convenient to see the link on the series page in the same place.
<cjwatson> https://bugs.launchpad.net/launchpad/+bug/1580167
<mup> Bug #1580167: Hide the "Code for this series" UI in projects that are configured to use git <confusing-ui> <git> <lp-code> <ui> <Launchpad itself:Triaged> <https://launchpad.net/bugs/1580167>
<cjwatson> (That said, we might need to tackle this if we do translations integration)
<Pegasus_RPG> (BTW, mup is including the > in the URL, making it not directly clickable.)
<cjwatson> I'm not sure I have any idea who runs mup; I /ignore it :-)
<Pegasus_RPG> (or it's my IRC client...)
<Pegasus_RPG> Testing: <https://bugs.launchpad.net/launchpad/+bug/1580167>
<mup> Bug #1580167: Hide the "Code for this series" UI in projects that are configured to use git <confusing-ui> <git> <lp-code> <ui> <Launchpad itself:Triaged> <https://launchpad.net/bugs/1580167>
<Pegasus_RPG> yep, my client, sorry
<cjwatson> The <URL> convention is reasonably old, although it's always been disputed
<cjwatson> If we did a series link for git, I think we'd have to do the same kind of thing we do for MPs, where you can have a link but the link is allowed to go away.
<cjwatson> i.e. stored as repository + a text column for the branch; the link changes style when it stops referring to anything; and deleting the entire repository checks and either prevents deletion or breaks the link
<Pegasus_RPG> Sound good to me
<Pegasus_RPG> When one imports a Git tree so it can be reached at lp:project, is it made to look like a bazaar tree to LP or does it fully understand Git at this point? (e.g. if I did bzr branch/checkout lp:project, would that work?)
<cjwatson> It's native.
<cjwatson> No bzr/git translation in either direction is involved.
<Pegasus_RPG> ok good
<Pegasus_RPG> so that command would fail
<cjwatson> Yes
<cjwatson> lp:turnip (historical reasons ...) is the slightly specialised git server; it uses git send-pack/receive-pack for most of the hard work, but has some translation layers on top.
<cjwatson> e.g. so that we don't have to move storage around when its logical location in the LP taxonomy changes
<Pegasus_RPG> okay checkout of lp:launchpad done
<Pegasus_RPG> err branch
<cjwatson> You're going from https://dev.launchpad.net/Running/LXD ?
<cjwatson> (If not, I strongly recommend doing so)
<Pegasus_RPG> uh no, https://dev.launchpad.net/Getting
<Pegasus_RPG> oh crap, thanks for the warning
 * cjwatson edits that a bit
<cjwatson> You can do the Running/LXD bit and then run utilities/rocketfuel-setup inside the container in the branch you now have checked out (possibly moving it into the right place first; you can have it be somewhere other than that page says, but you need to adjust ~/.rocketfuel-env.sh if so, so depends how strongly you feel about your homedir layout)
<Pegasus_RPG> I first need to upgrade my kernel, heh
<Pegasus_RPG> Does anyone know if there is an OAuth API for LP/Ubuntu One so that we could have people use their LP credentials to access our wiki for example? Searching the Web only refers to the now-defunct one.ubuntu.com service.
<Pegasus_RPG> Similarly, can people use other OAuth ID providers (e.g. Facebook) to log in to Launchpad?
<cjwatson> Other services can authenticate against login.ubuntu.com, which is the same as login.launchpad.net (just a different frontend hostname).
<cjwatson> However, it is not possible to use other providers to sign in to Launchpad.
<cjwatson> login.ubuntu.com is an OpenID provider.
<cjwatson> https://help.launchpad.net/YourAccount/OpenID
<Pegasus_RPG> OpenID 2.0 or OpenID Connect?
<Pegasus_RPG> looks like 2.0
<Pegasus_RPG> Unfortunately that is falling out of favor and OpenID Connect (based on OAuth) is taking its place
#launchpad-dev 2017-11-18
<tsimonq2> cjwatson: So I've been working on bug 439470 some more, I'm struggling about how I should actually implement the API function (CveSet.new)
<mup> Bug #439470: Cannot attach currently-unknown CVEs via linkCVE() <api> <lp-bugs> <platform-want> <Launchpad itself:In Progress by tsimonq2> <https://launchpad.net/bugs/439470>
<tsimonq2> cjwatson: I think I have the function that should be put in there, but I don't know where to put it.
<tsimonq2> cjwatson: At the bottom of lib/lp/bugs/interfaces/cve.py *seems* right, but it's not working well/
<tsimonq2> cjwatson: Here's my diff, maybe you can see something that I'm not:
<tsimonq2> $ bzr diff | pastebinit
<tsimonq2> http://paste.ubuntu.com/25991905/
#launchpad-dev 2017-11-19
<cjwatson> tsimonq2: The method implementation in lp.bugs.model.cve needs to have the same name as its corresponding interface declaration in lp.bugs.interfaces.cve.
<cjwatson> tsimonq2: Also have generally the same signature.  Actually I'm a bit confused about what method you're trying to add.
<cjwatson> tsimonq2: If you've added an interface declaration to lp.bugs.interfaces.cve, then the implementation of that method would go in the corresponding class in lp.bugs.model.cve.  Its signature should normally be the same as that in interfaces with the addition of an initial "self" argument (it's also permissible for it to have extra private keyword arguments that aren't mentioned in the interface).
<cjwatson> tsimonq2: Have a think about whether you should just be exporting the existing CveSet.new instead (possibly with different default behaviour for the API) rather than adding a confusingly-similar CveSet.add.
<cjwatson> tsimonq2: Or perhaps "newPlaceholder" or something like that?
<tsimonq2> cjwatson: Sure, I'll think more about it, thanks.
