#launchpad-dev 2010-06-21
<lifeless> Chex: I'm hoping you are still around
<lifeless> https://code.edge.launchpad.net/~mbp/bzr-email/242262-mail-deadlock
<lifeless> ^ needs an upgrade
<[1]davide> Hi all
<[1]davide> I installed launchpad on a vm (https://dev.launchpad.net/Getting) and it's running fine
<[1]davide> but when I try to register a user I just get the login form
<[1]davide> and no way to register ... any hints?
<StevenK> [1]davide: utilites/make-lp-user will add a user for you to a development instance
<[1]davide> Thanks Steven, I'll check that out immediately
<[1]davide> Steven I get a "gpg: keyserver internal error" while adding the user with this command: /home/vmlaunchpad/launchpad/lp-branches/devel/utilities/make-lp-user davide -e davide@davide.galletti.name
<adeuring> good morning
<mrevell> Morning
<lifeless> hai
<deryck> Morning, all.
<james_w> why did I get a reject from launchpad-bugs when I proposed a merge to the launchpad project?
<jml> james_w, I don't know.
#launchpad-dev 2010-06-22
<davide> hi all
<davide> I have a problem with utilities/make-lp-user
<davide> ./make-lp-user  -e davide@davide.galletti.name davide
<davide> gpg: sending key E0206B3F to hkp server keyserver.launchpad.dev
<davide> if I issue the command:
<davide> ...
<davide> gpgkeys: HTTP post error 7: couldn't connect to host
<davide> ...
<davide> any hints?
<StevenK> davide: Are the Launchpad services running on the machine?
<davide> yes I guess they are running;  https://launchpad.dev works
<wgrant> jtv: Morning.
<jtv> wgrant: I'd answer the same thing but let's be honest, we're both well past morning.  :-)
<jtv> Hi!
<wgrant> Shh.
<jtv> Oh... sorry.  :)
<wgrant> So, I've had a few people complain to me that Rosetta spams them too much for package uploads.
<wgrant> They are really not interested in success notifications.
<wgrant> And really wish they would go away.
<wgrant> Some normalish uploads result in hundreds of emails.
<wgrant> This is considered suboptimal.
<henninge> wgrant: Hi! Thanks for relaying that.
<henninge> wgrant: what would be considered optimal, then?
<dpm> ah, that's bug 353648
<_mup_> Bug #353648: Template import success notifications shouldn't be sent to package uploaders <Launchpad Translations:Triaged by danilo> <https://launchpad.net/bugs/353648>
<wgrant> henninge: Failures only.
<jtv> Not surprising.
<jtv> In fact I think it was anticipated.
<henninge> jtv: It also is not a new problem, is it?
<jtv> No, not a new problem.
<jtv> Success notifications were reasonable when the normal thing to do was upload a file or tarball, manually.
<adeuring> good morning
<henninge> jtv: but we can tell success by simply looking at the queue, can't we?
<henninge> do we know of anybody that depends or works with these success emails?
<henninge> adeuring: moin
<adeuring> hi henninge1
<jtv> hi adeuring
<adeuring> hi jtv!
<jtv> henninge: we used to get some Ubuntu import statistics that way, but that's a long time ago.
<wgrant> It's not a new problem, no. It's been a point of concern for a long time now.
<jtv> I think the reasonable assumption is that unless you hear otherwise, a change goes inâand soon.  That too has improved a lot.
<wgrant> That's what we do for the builds themselves.
<wgrant> It works well.
<jtv> The only case I can think of where they could really be nice to have is for uploads that get held up.  So a "you have entries stuck in needs-review" email could well be enough to replace success messages.
<jtv> Maybe it's also helpful to have a "the overall queue has 213,561 enrtries and the backlog is 3 days" message, like I did for the export queue.
<jtv> (You won't usually see that because it only shows up when there's something noteworthy to say)
<henninge> jtv: Also, our idea about mailing queue review comments could fix some of that.
<jtv> Only very indirectly, I think, in that they speed up the interaction.  But we'd only be sending those when we were indeed reviewing the entries.
<henninge> jtv: recife branch has a new revision! ;)
<henninge> jtv: do we tag bugs on the integration branch as qa-needstesting?
<jtv> henninge: not automatically, no...
<henninge> I noticed that ...
<henninge> I mean, manually.
<jtv> henninge: as long as we keep the full list then we can leave that until we start integrating.
<henninge> jtv: well, we have the full list in form of the commits to the integration branch.
<jtv> Not ideal.
<jtv> We should make sure the bugs are attached to the blueprint, and maybe tagged.
<jtv> The kanban board can be a distraction from that sort of housekeeping.
<henninge> We have not been using a tag and I am not sure we reliably attached them to the blueprint.
<jtv> henninge: I'll just check my own ones...
<henninge> jtv: I wonder what will happen when we merge the integration branch into .... db-devel(?) Won't the qa-tagging script pick them up anyway?
<jtv> henninge: probably, that's true...
<henninge> jtv: so we need to make sure to have bugs on branches - which we usually do.
<jtv> There are a few that we can't Q/A individually anyway, such as "rename the flags"
<henninge> yes, I think I remember we already agreed that QA will have to wait until then. We will have to land that early in a cycle.
<james_w> bigjools: could you give me a pointer to a test using a "script hook"? I haven't heard of them before.
<bigjools> james_w: sure, it's pretty simple, hag on
<bigjools> hang, even
<bigjools> james_w: see lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py
<james_w> thanks
<bigjools> james_w: it has getScript and a runScript helpers
<bigjools> one of them calls Popen, the other grabs the script class and calls its main()
<bigjools> james_w: see cronscripts/generate-ppa-htaccess.py for the top-level script
<bigjools> it doesn't do much other than instantiate the script class and run it
<james_w> yay, tests that aren't run in lp.soyuz.adapters
<bigjools> :/
<james_w> how does the runner find tests?
<bigjools> I thought it looked for anything called test_*
<bigjools> seems not
<james_w> it does
<james_w> missing __init__.py
<bigjools> gargh
<bigjools> that is such a stupid Python feature
<lifeless> discovert can work around it a bit
<lifeless> namespace packags can too
<lifeless> also there is some magic/aweful coming up in newer pythons too
<deryck> Morning, all.
<lifeless> deryck: 'lo
<deryck> hi lifeless
<ajmitch> hi
* mrevell changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.06 | PQM is open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
<ajmitch> mrevell: out of interest, when's the cutoff for landing code for 10.06?
<mrevell> ajmitch, Friday this week, AFAIK
 * ajmitch might try & get the blueprint api stuff reviewed at least
<ajmitch> ok
<james_w> "Confirm this transaction? [yes, no]"
<james_w> just want you want from a webservice, an command line prompt
<wgrant> That's a webservice?
<james_w> why isn't packagecopyrequest just part of the job system?
<cakofony> Is there any way to get the "uses launchpad for" information from the api?
<bigjools> james_w: the job system wasn't mature enough when copy archives were written
<james_w> bigjools: so you would have no problem with making it a job?
<bigjools> it was done in a way to make it possible though
<bigjools> nope
<james_w> ok
<bigjools> that's the intention
<cakofony> Using the api, is it possible to iterate through all the developers? or count them?  It only returns up to 50 for me
<james_w> cakofony: it's set to return the 50 top contributors to LP, and not iterate through all people, presumably to prevent abuse
<cakofony> james_w: is there any way to get the full list of developers?  I'm working on an open source project that gathers data from forge sites (with sleeps, we dont want to ddos anybody)
<james_w> no, there isn't
<cakofony> can I search for all the people who have contributed to each project?
<james_w> cakofony: as for "uses launchpad for", I don't think that's exposed right now, you can file a bug requesting it
<james_w> no, you can't do that either as far as I know
<cakofony> Alright, thanks for your help! ^.^
<sinzui> jml, talk>
<sinzui> talk?
<jml> sinzui, hi
<sinzui> hi jml
<jml> sinzui, sorry, meeting ran over time
<jml> sinzui, still up for a quick mumble?
<sinzui> yes
<sinzui> jml: http://changelogs.ubuntu.com/changelogs/pool/universe/w/wsjt/wsjt_5.9.7.r383-1.4/copyright
<james_w> are there docs on creating a db patch?
<james_w> https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess
<james_w> is changing security.cfg an entirely manual thing? How do I know what is appropriate to add?
<lifeless> yes
<lifeless> uhm
<lifeless> generally you add 'just enough' permissions
<lifeless> and if they seem broad reconsider the design
<james_w> luckily most of the role names actually make sense
<lifeless> \o/
<james_w> anyone around who knows the job system and can provide some advice?
<james_w> I would like to know the preferred way to implement a new job
<james_w> Whether a fairly generic job with json, implying serialising some things to strings to lookup as keys when the job is run, or a more-specific job type that can have References to the other models, saving that serialisation step, would be preferred.
#launchpad-dev 2010-06-23
<poolie> stub, lifeless, is there any facility in launchpad at the moment for configuration that can be changed by sysadmins?
<poolie> in particular if i want a list of trusted dkim domains
<poolie> or a switch to turn dkim on/off
<lifeless> yes
<lifeless> the config files
<poolie> ok
<lifeless> they have settings for dev, staging, edge, production
<lifeless> hm
<lifeless> oops
<james_w> anyone around who knows the job system and can provide some advice?
<james_w> I would like to know the preferred way to implement a new job
<james_w> Whether a fairly generic job with json, implying serialising some things to strings to lookup as keys when the job is run, or a more-specific job type that can have References to the other models, saving that serialisation step, would be preferred.
<poolie> hi james_w
<poolie> unfortunately i don't know about that
<james_w> hi poolie
<mrevell> Morning
<james_w> BjornT: hi, do you have an opinion on whether a new job type should use as many References as possible, or serialise to JSON and look up the objects again based on keys in the JSON?
<lifeless> james_w: hi
<lifeless> james_w: I'm not sure there is a general answer to that question
<james_w> hi lifeless
<lifeless> james_w: I'd consider things like:
<lifeless>  - are there security implications on any referenced tables
<james_w> I'm happy to give specifics
<lifeless>  - are there performance implications ...
<lifeless>  - what would be simplest for your code
<lifeless>  - are there race conditions - do you need to freeze parameters so they don't change under you
<lifeless> james_w: sure, though I'm signing off - 4am rise tomorrow - shortly
<lifeless> james_w: oh and obviously the inverse of the previous one, do you have parameters that *have* to be fresh
<james_w> g'night
<thumper> james_w: hi
<thumper> james_w: what are you looking to do?
<james_w> hi thumper
<james_w> I want to create a job type for the creation of copy archives
<james_w> these have a source archive, target archive, source distroseries, source packagesets, source component, etc.
<thumper> james_w: what we did for BranchJob was to just have the branch that it was linked to, and everything else in a json blob
<thumper> the general idea is:
<james_w> an ArchiveJob to parallel BugJub and BranchJob would be useful elsewhere, but there is far more linked data here than in any of the other jobs
<thumper> if you need to query against the value, it needs to be a column
<thumper> otherwise a blob is fine
<thumper> james_w: the question relates to what you end up querying against
<thumper> and what the job really links to
<thumper> and how it needs to be selected...
<thumper> what invalidates the job...
<thumper> how it will be deleted...
<james_w> just target archive needs to be queried against I think
<thumper> what other jobs will there be for archives?
<james_w> don't know yet
<james_w> there are a bunch of things that could be possible to have there, bulk copying packages, even publication
<thumper> but it is likely that there will be more?
<thumper> I'd suggest make a mirror of the branch job table for now
<thumper> although I hate myself for saying it...
<thumper> the duplication rubs me the wrong way
<thumper> but is the best we have for now
<thumper> but instead of a branch reference, have the archive
<james_w> there's a hell of a lot of boilerplate to create a new job type
<thumper> I wouldn't say there is that much...
<thumper> a lot has been moved into base classes
<thumper> but yes,
<thumper> there will be duplication and boilerplate
<james_w> https://dev.launchpad.net/Foundations/JobSystem
 * thumper isn't working :) just helping
<thumper> I'm in Perth having a holiday
<james_w> well, thanks :-)
<thumper> but as you can see, I have my laptop :)
<thumper> can't go on holiday without it, well, not for more than a few days
<thumper> I need my fix
<deryck> Morning, all.
<maxb> In relation to the just-gone discussion in #launchpad, maybe we should have uses_soyuz and uses_lp_distromirrors attributes on distributions?
<wgrant> maxb: We already have IDistribution.full_functionality
<wgrant> Currently a celebrity-based hack, but it works.
<jml> maxb, or even IDistribution.build_system :)
<bigjools> ICanHazBuilds
<wgrant> But even just hiding series that don't have any PPA-capable DASes would be a good start.
<bigjools> wgrant: BTW did you notice that we got the first successful recipe build done in prod yesterday?
<wgrant> bigjools: I saw some bugs fly past that suggested it.
<wgrant> Is this after chmoding the source dir on cesium?
<bigjools> no, fixing the init script for the b-m
<wgrant> Ah.
<bigjools> which had an interesting "cd" in it
<bigjools> so, part of this success is down to you, thanks a lot man :)
<wgrant> I'm glad it's finally running.
<wgrant> A few months later than I expected, but.....
<wgrant> mrevell: The PPA terms of use state that anything OK for Ubuntu is OK.
<wgrant> And just about anything redistributable is OK for Ubuntu.
<mrevell> wgrant, Thanks for that. I'm not saying you're wrong but that's not how we've enforced it before. For example, we nearly closed a PPA a year or two back that had the flash installer in it but the owner reworked it.
<wgrant> Right, it seems like it's wrong, but it seems to be what the terms of use say.
<wgrant> Hmm.
<wgrant> bigjools: The bzr-builder PPA doesn't also have a modern bzr in it?
<bigjools> is that a question or a statement?
<wgrant> I just tried a Hardy build, and it seems to be using real Hardy bzr.
<wgrant> I didn't even know bzr-builder would work with something so old.
<bigjools> the production system doesn't use a PPA for the bzr-builder dep
<wgrant> Oh, bzr-builder is in -updates?
<bigjools> no, there's an internal archive
<wgrant> Ah, right.
<mrevell> wgrant, Thanks for raising that. As ever, you're the guy who keeps me on the right track :) I'll review the terms and how we enforce them.
<wgrant> s/bzr-builder PPA/internal archive/
<bigjools> heh - no idea
<bigjools> I'll let abentley chase that one up
<james_w> wgrant: bzr-builder doesn't do anything particularly interesting with bzrlib, just equivalents of branch and merge, which have been around for ever.
<wgrant> Ah.
<wgrant> james_w: http://launchpadlibrarian.net/50794525/buildlog.txt.gz <- do I need to backport python-debian too?
<james_w> wgrant: quite possibly
<james_w> we could perhaps fix it in bzr-builder
<wgrant> Is the PPA publisher working? I copied a package about 12 minutes ago -- it's still Pending.
<wgrant> Ah, there.
<wgrant> Looks like it missed two runs.
<james_w> how do I get the key for a enum item?
<james_w> I want to store something that I can use to get that enum value back
<james_w> .value?
<jml> james_w, have you figured it out yet?
<james_w> no
<james_w> yes
<jml> james_w, cool. :)
<jml> sorry for not trying to answer earlier.
<james_w> PackagePublishingPocket.items[PackagePublishingPocket.RELEASE.value] I think
<james_w> just waiting for my tests to confirm
<james_w> yes
<james_w> which layer should I use for a test class that will just be using the db (and possibly librarian), but doesn't care about the web ui and stuff?
<james_w> is LaunchpadZopelessLayer just for script tests?
<james_w> bigjools: do you have a problem with PackageCloner creating build records for the packages it copies too? Currently it just copies sources and the builds are created elsewhere. I would like to consolidate for ease of testing.
<bigjools> james_w: remind me where they're created at the moment?
<james_w> scripts/populate_archive.py
<bigjools> urgh
<bigjools> yeah change it
<jml> I'm getting an error trying to build gettextpo
<jml> it's the first time I've tried to build lp in a few months, so I might have missed some obvious / crucial step
<jml> http://paste.ubuntu.com/454033/
<jml> help?
<dobey> anyone know why the merge proposal diffs aren't updating in launchpad, and it's not sending the mails for proposal status changes?
<jml> dobey, no, I don't, but I saw abentley say something vaguely operational about it on another channel
<abentley> dobey, no, at this point it's just under investigation.
<james_w> jml: that's odd, if pydoc was failing it would normally say "Error" before "Leaving directory" I think
<dobey> abentley: ok, thanks
<james_w> that's probably the last step either way, so that doesn't tell you whether it failed
<james_w> if you are using parallel make then the actual error can be further back
<jml> james_w, how would I know if I'm using parallel make?
<james_w> I don't know, apart from it doing more than one thing at once
<james_w> if there's no error before it starts printing pygettextpo stuff then you probably aren't
<james_w> I don't think it's default for LP, so unless you ran "make -j"...
<jml> hm.
<jml> oh, I missed a tb
<jml> http://paste.ubuntu.com/454037/
<jml> oh huh
<jml> I think I have a local copy of Python
<jml> If I want to add some tests for bugtask, should I add them to test_bugtask.py, test_bugtask_0.py or test_bugtask_1.py?+
<james_w> heh
<james_w> probably make test_bugtask_jml.py?
<jml> james_w, maybe I should use the database id for my row in Launchpad to avoid confusion
<jml> deryck, If I want to add some tests for bugtask, should I add them to test_bugtask.py, test_bugtask_0.py or test_bugtask_1.py?
<deryck> jml, I don't know.  Let me look more after the call.
<jml> deryck, thanks.
<james_w> gah, IPackagesetSet has a get method that works different to all the other get methods I have come across
<jml> yeah. we need some consistency.
<james_w> are there any scripts in LP that would be run by an admin and just create a job for later processing? Would we want such a script to create the job but synchronously process it? Or just do the work and leave the job for webservice requests?
<jml> james_w, I'm not sure what you mean.
<james_w> populate_archive.py currently creates a copy archive and sits there while everything is copied
<james_w> I'm working on making copy archive creation a job such that we can do it over the API and the like if we wish
<james_w> I'm wondering if I should:
<james_w> a) have the populate_archive.py script create the job and exit, leaving it up to a cronscript to do the work and the person running the script to poll the web ui to find when it is done
<james_w> b) have the populate_archive.py create the job, but pre-claimed, and then kick off the running of the job synchronously such that it can report the results to the admin running the script
<james_w> c) have populate_archive.py ignore the job system and just run the code in pretty much the way it is now, and just have the job system as an alternative entry point
<james_w> d) other
<james_w> e) none of the above
<james_w> 3) Uzbekistan in the 1920s
<jml> james_w, sorry for the delay, I'm on a call that got interesting :)
<james_w> np
<james_w> it's an idle question as I hack the code to actually make any of this possible
<jml> james_w, I don't know enough about what populate_archive.py does, I think. a) seems the most consistent
<jml> and we should have a widget on the ui for tracking in-progress jobs
<james_w> so populate_archive.py creates something like launchpad.net/~jml/+copy-archive/some-new-archive
<james_w> then it spends a few minutes copying all the sources from Ubuntu in to it and creating build records so that you can testbuild a snapshot of Ubuntu
<james_w> this is currently something that is usually done by an admin for doko
<james_w> so I guess it's a question for the losas? they are the only ones that will use the populate_archive.py interface.
<jml> james_w, yeah, maybe.
<jml> james_w, now that you explain it though, I'm more inclined toward a)
<james_w> Perhaps it will be changed to do an API call instead, in which case we can't really do synchronous, as we don't even have a way to poll for completion of this job
<jml> james_w, although it's something of a regression in the UI of the script, we should just fix the webservice api so that the script can behave the way it did.
<jml> james_w, after all, this isn't the only case where someone wants to run scripts that kick off long-running jobs.
<james_w> yeah
<jml> deryck, thanks for looking at the test_bugtask thing btw
<james_w> jml: do you have any notes from your earlier call with Julian?
<deryck> jml, yes, just back from coffee.  Looking now....
<abentley> dobey, the script got stuck and we've restarted it.  Please let us know if you're still experiencing problems.
<dobey> abentley: ok thanks
 * dobey waits for a flood of e-mail
<abentley> james_w, we do already have scripts that create jobs and let a cronjob handle running them.  e.g. the branch scanner.  Daily builds is similar, though it hands off to the build farm, not a cron script.
<deryck> jml, so I think just use test_bugtask.
<deryck> jml, the others seem like someone was thinking something that is not now clear :-)
<jml> deryck, that had occurred to me :)
<jml> deryck, thanks.
<james_w> abentley: they are scripts run by admins?
<abentley> james_w, they are normally run from cron scripts themselves.
<abentley> Or as cron jobs, I should say.
<james_w> abentley: right, this will only ever be run by hand on request
 * deryck will make a note to confer and refactor test_bugtask* as appropriate
<jml> 5 seconds to start the functional layer
<jml> gary_poster, do you know by what mechanism the API documentation lists the possible choices for "hardware_bus" on https://edge.launchpad.net/+apidoc/devel.html#bug_target for searchTasks?
<gary_poster> jml: no, but my guess would be by introspecting a vocabulary specified in an interface.  I could probably find out quickly.  Should I?
<jml> gary_poster, yes please. I'm trying to figure out how it knows to introspect as well.
<gary_poster> k
<gary_poster> jml: it almost certainly introspects the Choice field on line 95 of lp/bugs/interfaces/bugtarget.py .  This in turn points to a vocabulary, which is apparently, per the comment, set to HWBus in ./canonical/launchpad/interfaces/_schema_circular_imports.py:165
<james_w> lazr.restful.marshallers.SimpleVocabularyLookupFieldMarshaller knows how to iterate items on an enum to get the valid choices
<jml> does that get used to generate the wadl
<gary_poster> yes
<jml> ok. thank you gary_poster & james_w
<gary_poster> np
<jml> actually, it looks like it's used to make the json, not the wadl
<james_w> jml: http://paste.ubuntu.com/454134/
<jml> james_w, thanks.
<james_w> jml: in return any hints you could give me of how I would get a SourcePackagePublishingHistory given it's id would be greatly appreciated
<james_w> there's no ISourcePackagePublishingHistorySet I can see
<james_w> hmm, there's an IPublishingSet
<jml> james_w, you can cheat & use storm. getUtility(ISomethingOrOther).find(SourcePackagePublishingHistory, SPPH.id=your_id).
<james_w> jml: thanks, I'll resort to that if this doesn't work
<james_w> getUtility(IPublishingSet).getByIdAndArchive() apparently. Though for some reason that returns you a ResultSet and you have to call .one() on it yourself.
<jml> :(
<jml> james_w, there's only one callsite for that, and no unit tests.
<jml> (or doctests)
<james_w> jml: are you suggesting I fix it up?
<jml> james_w, I am. But I can completely understand if you'd rather not.
<james_w> I'm already in the middle of a monster branch
<james_w> so adding one more thing to it isn't going to hurt that much
<james_w> hmm, my ec2 instances seem to be dying without mailing me
<jml> james_w, good news and bad, even while I momentarily gaze away
<jml> gary_poster, james_w, fwiw, I was working on https://bugs.edge.launchpad.net/lazr.restful/+bug/590561. Now I've stopped.
<_mup_> Bug #590561: searchTasks API documentation doesn't document acceptable status arguments <lazr.restful:New> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/590561>
<gary_poster> on call
<jml> np. it's not urgent.
<james_w> more fun: http://paste.ubuntu.com/454140/
<james_w> oh, that one's my mistake
<jml> james_w, nyah-ah-ah
<jml> james_w, if they are dying without mailing, I'd recommend running them attached.
<jml> james_w, there's no other way of guaranteeing you'll get output.
<james_w> jml: just kicking that off now
<jml> james_w, cool.
<jml> james_w, I'm off now to find something to eat.
<james_w> jml: enjoy
<jml> james_w, and thence to not work.
<jml> james_w, enjoy your evening
#launchpad-dev 2010-06-24
<james_w> jml: https://code.edge.launchpad.net/~james-w/launchpad/fix-publishing-getbyid/+merge/28359
<adeuring> ood moring
<lifeless> I wonder
<lifeless> how much work would be involved to make 'mark x as dup of y' move all of x's dups to be dups of Y
<lifeless> its awefully manual at the moment
<poolie> wbn
<poolie> even just when i try to make x a dupe of y and y is already a dupe of z
<poolie> it could just do it
<poolie> rather than giving me an ugly error
<poolie> lifeless: oh also i was contemplating a command to attach a local file to a bug
<poolie> do you know of anything at the moment?
<wgrant> lifeless: I think deryck implemented that recently.
<lifeless> no, that would be nice. Bughugger probably has a gui one.
 * wgrant hunts.
<lifeless> wgrant: wabbits?
<wgrant> https://code.edge.launchpad.net/~deryck/launchpad/do-the-right-thing-dupe-move-78596/+merge/27144
<wgrant> Not merged yet.
<StevenK> poolie: apport hooks probably do that
<lifeless> StevenK: no, they do something quite different.
<lifeless> StevenK: it uploads to a staging area, gets a magic prefix, etc
<StevenK> Ah
<poolie> StevenK I want: lsusb | lp-attach 123123 --name lsusb.txt
<lifeless> oh
<lifeless> there is an apport thing to *add* to a bug
<lifeless> under the ubuntu-bug banner
<wgrant> apport-collect?
<lifeless> ubuntu-bug -u bug
<poolie> mm i want something with less policy
<poolie> i may try one sometime
<lifeless> bug.addAttachment
<lifeless> its in the API
<poolie> i know
<poolie> but thanks
<poolie> i anticipate it should be a pretty small bit of code
<lifeless> hmm
<lifeless> I don't like the question spam
<lifeless> when there is a linked bug
<poolie> not my fault :)
<poolie> or is it?
<lifeless> no
<lifeless> well maybe you linked two things
<lifeless> but it appears to be that questions mails me directly
<lifeless> when a bug status attached to a Question is altered.
<poolie> ah yes
<lifeless> and I'm subscribed indirectly to the bug.
<poolie> that bugs me too
<poolie> also it's not obvious why the mail is being sent
<lifeless> I'm enbugginating
<lifeless> and.... bug dup suggestions are timing out again
<lifeless> https://bugs.edge.launchpad.net/launchpad/+bug/597981
<_mup_> Bug #597981: launchpad tells me twice about changes to bugs that are linked to questions <Launchpad itself:New> <https://launchpad.net/bugs/597981>
<lifeless> poolie: also https://bugs.edge.launchpad.net/launchpad/+bug/597982
<_mup_> Bug #597982: emails about 'questions' have many different subject lines <Launchpad itself:New> <https://launchpad.net/bugs/597982>
<poolie> cute
<poolie> if  you create an attachment and don't specify the mime type
<poolie> launchpad assumes it is chemical/x-mopac-input
<poolie> i think that's pretty reasonable, don't you? :)
<lifeless> rotfl
<lifeless> thats awesome
<poolie> i guess it's alphabetically first or something
<poolie> might be behind bug 204560
<_mup_> Bug #204560: Incorrect MIME type when uploading an audio/x-musepack (*.mpc) file <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/204560>
<mrevell> Guten morgen
<jml> lifeless, wrt to your recent comment on the factory... I wonder how much an LP factory could be broken up.
<lifeless> jml: I'm doing some experiments in the space
<lifeless> ask me at the epic, I hope to have better answers
<lifeless> and I may have had opportunity to look at the LP factory by then.
<lifeless> jml: trivially it could be made into a Facade
<lifeless> or given .foo.bar subfactories
<deryck> Morning, all.
<deryck> jml, I'm reviewing your branch to add search tasks modified_since parameter... I'm curious why you wrapped the factory makeProduct call in IBugTarget.  We usually just pass the made product straight to searchTasks.
<jml> deryck, because the test doesn't care about it being a product, I guess.
<jml> deryck, it just wants a BugTarget and doesn't really care about the implementation
<jml> deryck, wrapping means that the test reads the way I think about it.
<deryck> jml, ok, fair enough.
<deryck> jml, and I'm curious too about your addition of the login method, rather than passing something like user='test@canonical.com' to the setUp of the test.  Is this to completely avoid sample data?
<jml> deryck, basically, yes.
<jml> deryck, I hate login with a particular fixed email string. I can never remember what privileges it's supposed to have.
<jml> deryck, again, the idea is that the code communicates "login in as someone, anyone, it doesn't matter"
<jml> deryck, because that's actually what we want, right?
<lifeless> \o/
<lifeless> clarity, its a shocker
<deryck> jml, yup, that's fine.  I think the branch fine as is, then.  r=me.  I'll add some comments to the MP for posterity's sake.
<jml> deryck, thanks.
<didrocks> jml: FYI, I fw you the email sent to the ML about LP and Quickly (hope you will receive it this time :))
<jml> didrocks, I just forwarded it to launchpad-dev
<jml> didrocks, that's definitely the first time I've seen that email.
<didrocks> jml: ok, what's wrong with the adress I used? I got no email about rejection or anything else
<didrocks> jml: and my @ubuntu.com adress is registered on LP, I thought that you can send to a LP ML even not being member of the team
<jml> didrocks, I don't know.
<didrocks> jml: weirdâ¦ if you receive any answer, can you check that I'm in CC :)
<jml> didrocks, ahh, fwiw, your original emails were in the moderation queue
<didrocks> jml: oh ok, so make sense now :-)
<didrocks> thanks jml
<jml> I don't know who gets those emails.
<jml> as in, who is told about pending moderation.
<didrocks> maybe nobody, hence the pending issue :-)
<Mez> trying to setup a local LP install on lucid - cant start up postgrsql again
<Mez> "Invalid data directory"
<maxb> Mez: That's really not enough info to enable anyone to help you
<Mez> maxb: No worries, I'm cleaning up, getting rid of postgres, and reinstalling ;)
<Mez> trying a manual DB setup, rather than the automated one
<Mez> seems postgresql8.4 had it's /etc stuff wiped somehow
<Mez> ah, does LP support postgres 8.4 ?
<maxb> Yes
<Mez> it's not in the docs ;)
<Mez> (which just mentions 8.3, leafing me to ask the Q)
<maxb> Although, launchpad-database-dependencies won't install all the supplementary packages for 8.4, so you'll have to do it yourself
<Mez> like?
<maxb> postgresql-plpython-8.4 postgresql-contrib-8.4
<Mez> that may have been where it went wrong then
<Mez> ok, now I've got my DB running, however - make schema doesnt work
<Mez> http://pastebin.com/wNmNYmti
<maxb> Mez: 'make SHHH= schema' to turn off the silly output hiding thing so the output is more informative
<Mez> maxb: pretty much the same
<Mez> http://pastebin.com/q3JSfNWR
<Mez> I removed the download-cache and reran rocketfuel-get
<henninge> Hi adiroiban ! ;)
<Mez> maxb: any siggestions ?
<Mez> ah, get a copy from the python site, that works
<maxb> oh, is a file missing from download-cache?
<maxb> Mez: I have a dist/setuptools-0.6c9-py2.6.egg in my download-cache
<Mez> maxb: no, the file was there, but the file wasnt matching it's MD5 sum
<maxb> huh. weird
<Mez> I re-downloaded it directly from the site, and it seems to have fixed it.
<Mez> *sigh*
<Mez> now even more issues
<Mez> no build-essential (gcc)
<james_w> so "ec2 land" is running all the tests, and they all pass, but it either isn't doing anything, or the mails aren't getting out
<jml> james_w, hmm.
<jml> james_w, you've double checked spam filters I suppose
<james_w> The code seems to be hiding the request to send to pqm if successful, but seeing as I don't get a success email I'm suspecting the latter
<jml> hmm.
<jml> james_w, actually, I don't understand that last sentence of yours.
<james_w> I'm reading the code to try and work out what it is trying to do, but I can't see where "cmd_land" sets things up such that a request will be made to pqm
<jml> james_w, it generates a command line for 'ec2 test' and then runs that.
<james_w> yes
<jml> james_w, the "-s" option in _get_landing_command
<jml> which is submit_pqm_message
<james_w> ah
<james_w> thanks
<james_w> it was hiding :-)
<jml> np
<james_w> ah
<jml> ultimately it all gets passed as options to remote.py
<james_w> I changed my bzr whoami to have my linaro address, I wonder if that's it
<jml> which is what actually does the interesting work
<jml> james_w, I bet you that is it.
<rockstar> sinzui, do you have some time for mumble really quick?  I need a UI sanity check.
<sinzui> I do
<rockstar> sinzui, what channel are you in?
<sinzui> i do not know?
 * rockstar thinks pre-imp calls with sinzui are awesome because they are so short.
<sinzui> rockstar, that is odd. I talk to much
<sinzui> I have no social life
<sinzui> bac and EdwinGrubbs can attest that I do not know when to shut up
<rockstar> sinzui, that may be teamlead-itis.  thumper has the same disease
<sinzui> :)
<james_w> what was the name of the effort to restructure launchpad in to the lp.app layout we have now? Was it called the apocalypse?
<jml> james_w, yes.
<james_w> thanks
<jml> james_w, progress is charted here: https://lpstats.canonical.com/graphs/CodeBaseFileCount/20090101/20100625/
<james_w> jml: can you confirm I have the right idea on: https://dev.launchpad.net/APPocalypse
<jml> james_w, punned!
<james_w> salgado's idea
<james_w> I was thinking branchpocalypse
<jml> james_w, that seems the right direction
<jml> james_w, the way I'd do it is split things by service/process, rather than by app.
<jml> at least to start with.
<jml> james
<james_w> jml: example?
<jml> james_w, buildmaster, codehosting-ssh, etc.
<james_w> ok
<james_w> I think starting with lp.whotsits would be good
<jml> james_w, yeah, but the ones of those I'd start with would be the ones that correspond to near-standalone services
<james_w> I don't want us to go too far beyond the already established lines, as we only need to split a certain amount for our aims.
<jml> james_w, also, I have a .dot file for you.
<james_w> right
<james_w> jml: is that the one that will break my eyes, if not my poor little laptop?
<jml> james_w, yes. http://people.canonical.com/~jml/lp-clustered.dot
<james_w> merci
<jml> james_w, no mercy.
<lifeless> jml: poolie said something, and I may be misremembering the details /completely/ about a resolution at a recent previous epic
<lifeless> jml: to squash the timeout oops vigorously
<jml> lifeless, there was, but it died.
<lifeless> oh
<jml> lifeless, and it was subtly different.
<lifeless> is there a mail trace about that? I would like to read it
<jml> lifeless, depends on which initiative you mean
<jml> lifeless, there was one to reduce / eliminate timeouts
<jml> lifeless, there was another to reduce / eliminate all oopses.
<lifeless> ok
<lifeless> do you know what killed it ?
<lifeless> tools/size-of-work+time-allocated/motivation/playing-mallet-on-the-metric ?
<jml> it dissipated, basically.
<jml> lifeless, I'll grep my email to see if there was a closure email, but probably your best bet is to talk w/ flacoste.
<lifeless> whose awl today
<lifeless> ok
<jml> lifeless, nope, no email
<jml> gotta go. dinner time.
<lifeless> thanks, ciao
<lifeless> hi bac
<bac> hi lifeless
<lifeless> bac: did my subunit/testr notes make sense to you?
<lifeless> bac: and if they didn't can I debug that; if they did, did they work for you? [and if not...]
<bac> lifeless: yes it did, thanks.
<bac> lifeless: i haven't actually tried it yet, though
<bac> lifeless: trying now...
<lifeless> brb
<lifeless> mmm, in the court of the crimson king
<lifeless> rhythmbox plays this to me first, every time :P
<lifeless> bac: we could add an import command
<lifeless> testr import attachment.gz
<lifeless> which would:
<lifeless>  - sniff the file type
<lifeless>  - 'do the right thing'
<lifeless> but I'd like to keep the core building blocks really dumb
<bac> lifeless: that would be nice...but the reason i raised the issue was really an appeal for human readable summary of failures in the email message.  re-running them is secondary.
<lifeless> bac: right, I addressed that too
<bac> lifeless: and i think the second part of your email recipe does that
<bac> lifeless: yep
<bac> so many thanks on both counts
<lifeless> for instance, if I grab a pqm failure from the bzr project
<lifeless> we have the make output
<lifeless> and the tests
<lifeless> its all quite customisable
<lifeless> https://pastebin.canonical.com/33967/
<lifeless> is the end of a recent failure mail
<lifeless> it uses a very similar bit of code to what I supplied
<sinzui> lifeless, ping
<lifeless> hi
<lifeless> sinzui: ^
<sinzui> lifeless, the milestone has been a real problem for me for a year because of timeouts. There are many enhancements I cannot consider until I can make that page reliably load...
<lifeless> sinzui: yes, I'm just curious is all
<sinzui> I can see in a timeout that 4.5 seconds was sql time, everything else was python maybe
<lifeless> sinzui: I meant absolutely no impediment to your landing it
<sinzui> We have had several developers look into it.
<sinzui> No, I am hoping your brains can help me really solve the problem.
<lifeless> sinzui: I'm really sorry if I wasn't clear about that, I was asking totally from curiosity :)
<lifeless> oh, ok cool
<lifeless> uhm
<lifeless> did you happen to get an lsprof trace of a typical render from a laptop ?
<lifeless> if its mainly python that sort of thing can be really enlightening
<sinzui> I discovered recently the the diversity of information seem to be a greater factor than number of bugs/blueprints. lots of private object or private secondary objects lost the page down, to does having 100+ assignees and a diversity of statues
<sinzui> lifeless, I think edwin or bac looked into lsproof. but we could not produce the issue in dev until I wrote a script that makes the diversity I saw in ubuntu timeouts
<lifeless> so that raises a couple of background tasks - get more data on oops to better reproduce (but gary/stub are working on that already I believe, which is great)
<lifeless> I think I'd approach it thusly:
<lifeless> which ever is greater, sql/python; focus on that (you've said python)
<lifeless> for the thing being focused on, get a profile trace of a page that would have timed out.
<lifeless> that is - on staging or something
<lifeless> * turn off timeouts
<lifeless> * install a profile middleware
<lifeless> * run the page
<lifeless> turning off the timeouts is useful, because that way we can see the total places tha time goes
<lifeless> otherwise we'd be optimising on only (say) 20 seconds out of 40.
<lifeless> then, locally, as you have done already, setup something to trigger badnesses
<lifeless> optimise that locally until (at minimum) you've reduced the local time by the same fraction needed to make the total time the profile took be acceptable
<lifeless> e.g.
<sinzui> staging was setup with lsprof. I need to see if that is true since we changed packaging and python versions
<lifeless> if its 16 seconds in python now to timeout, and 32 seconds in python to profile, and we want to make the python time give a total page render < 20 seconds, we need to take 1/2 the time it does
<lifeless> iterate on that locally
<lifeless> sinzui: does this add anything to what you where doing?
<lifeless> sinzui: I don't have any immediate stuff beyond 'gather some more data' though, I'm sorry :(
<sinzui> yes. It is a sound outline to find the root cause(s)
<lifeless> you know the ++oops++ magic decorator thingy
<lifeless> ^ technical term
<sinzui> I do
<lifeless> what might be really nice
<lifeless> is a magical ++profile++ decorator enabled only on staging (and for non-devs it turns itself off once the user is determined)
<mars> ooo
<lifeless> it could write the profile to a directory that gets rsynced to devpad
<mars> +1
<lifeless> and put the filename in the timing block at the end of the page
<lifeless> so you could just do this
<lifeless> wait a couple of mintes
<lifeless> and have a kcachegrind ready analysis
<sinzui> thanks
<lifeless> (I can show you the exact functions in bzrlib to call to use its lsprof glue; though we may want a tweaked version or a lock around the profiling to kick other threads off, or something)
<lifeless> mars: launchpad-foundations would be a good place to record that idea, yes ?
<mars> lifeless, yep!
<lifeless> I shall file a bug
<mars> lifeless, did you get kcachegrind/lsprof to work for bzrlib?
<lifeless> mars: years ago :)
<mars> My kcachegrind bin/test patch causes a cachegrind memory explosion
<lifeless> mars: run this: bzr --lsprof-file foo.callgrind st
<lifeless> mars: kcachegrind foo.callgrind
<mars> I wonder if you run something different than I do
<lifeless> oh
<lifeless> for tests we do it per-test
<mars> I just grabed lsprof from... somewhere
<lifeless> not for the whole test run
<mars> ah, ok
<lifeless> the whole test run is *freaking huge*
<mars> I tried the whole run.
<lifeless> GB's of arcs
<mars> ... well that would explain a few things...
<lifeless> and thats in bzr which is considerable leaner than lp test-time wise
<lifeless> mars: have a look in bzrlib.lsprof
<lifeless> mars: it has shinies
<mars> So my patch is sane then
<mars> where'd it go...
<lifeless> mars: (use it directly, we do fix it)
<lifeless> I quite like the chromium speed-tracer
<mars> lifeless, the zope testrunner takes a --profiler= argument.  I came up with a monkey-patch for zope.testing that enables one to cachegrind
<lifeless> mars: how does it compare to the performance tools you've been working with and posting about
<mars> speed-tracer - is that the one with the zooming timeline across the top?
<lifeless> mars: yeah, built into the browser
<lifeless> per-item timelines etc
<lifeless> things like
<lifeless> Request Timing	@21093ms for 18897ms
<lifeless> Response Timing	@39991ms for 661ms
<lifeless> Total Timing	@21093ms for 19559ms
<lifeless> (which is for the html in https://edge.launchpad.net/bzr btw)
<mars> From the tutorial it looks like they may have improved it
<mars> they used to not distinguish between DOM events
<mars> I really wanted to know "How much blocking was JS execution, and how much was waiting for the load event"
<mars> And it was near impossible to answer that
<mars> So I could not (before) answer a question like "the page has a one second gap between visible and the UJS being active - why?"
<mars> What I really want to see is this: http://developer.android.com/guide/developing/tools/traceview.html
<mars> with the whole zooming timelines and such
<mars> hey, they've improved it!
<mars> they actually documented the file format - that means you could write a Python traceview profile formatter
<lifeless> ok, bug https://bugs.edge.launchpad.net/launchpad-foundations/+bug/598289 filed
<_mup_> Bug #598289: support a profile decorator for use in staging and development environments <Launchpad Foundations:New> <https://launchpad.net/bugs/598289>
<lifeless> I've captured what we discussed in brief, I think.
<mars> Now I just have to find some sleep I can sacrifice to get 100 points of time.  Where to look...
<mars> lifeless, looks good
<mars> lifeless, would be nice for the epic.  Heck, any profiling would be nice for the epic.
<lifeless> we could do this before the epic
<lifeless> so that its available
<lifeless> mars: what do you think?
<mars> lifeless, I have no idea if that is possible
<mars> sorry
<mars> and I'm already working on another performance tool for the epic
#launchpad-dev 2010-06-25
<lifeless> mars: no worries
<lifeless> mars: one doesn't find out if one doesn't ask
<adeuring> good morning
<mrevell> Yo
<deryck> Morning, all.
<jml> morning
<jml> deryck, I'm doing some stuff that uses my recent bugs changes and probably counts as QA
<deryck> jml, ok, thanks.  I'll update the bugs.
<jml> deryck, perhaps best to wait until it's all ok on my end :)
<deryck> jml, ah, ok.  I assumed you meant it was ok there.  No problem for me to wait.
<lifeless> night y'all
<jml> deryck, woot. they both work. (see http://people.canonical.com/~jml/convergence/ which now includes incomplete bugs)
<deryck> jml, nice!
<jml> (and also, it no longer takes me the better part of an hour to synchronize bugs, because of modified_since)
<deryck> adeuring, hi.  You might be interested in bug 598484.
<_mup_> Bug #598484: +bugs-text disappeared for meta projects <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/598484>
<deryck> adeuring, I think we should fix this before the rollout next week, and I assume we'll need to get mars RC for this unless you are super super fast today.
<adeuring> deryck: argh... yeah, you have a point.
<deryck> adeuring, I'm adding a card to the bugs backlog on the kanban board now.
<adeuring> deryck: OK
<deryck> adeuring, I went ahead and assigned to you, so feel free to drag to WIP when you can work on it today.
<deryck> adeuring, sorry to just drop one on you :-)  But thanks for taking it on. :-)
<adeuring> deryck: sure
 * sinzui needs to find a project owned by ~registry that needs a bug tracker registered. Probably a project mirrored from google or a track-based project
#launchpad-dev 2010-06-26
<lifeless> mars: don't suppose you're around ?
<lifeless> anyone know where the code for zope.app.publication is (upstream)
<User> Hey
<User> I have a few questions..
<lifeless> evenin'
<User> Mornin'
<lifeless> hmm, I need a zope.interfaces quick-answer
<user___> Hey, I have some questions for you guys.. Anyone care to help me out?
<user___> I need some help with getting it, building it, etc.
<lifeless> morning
<user___> Ok, I'm trying to download the latest stable revision of Launchpad, and I plan on integrating this into a production environment, and I need help with the config files to download it. And also the Apache setup. I'm using webmin and virtualmin if that makes any difference.
<user___> I also read that the Launchpad images should be changed as they're copyrighted (makes sense), but does this include the ENTIRE image theme too?
<lifeless> yes, all the images are custom
<user___> What do you mean?
<user___> Oh.
<user___> So I have to replace the entire image theme then?
<lifeless> I'm not sure what you mean by 'entire image theme'. So I can't answer that question
<user___> Ok, let me see if I can explain this clearly.
<user___> "The images/icons are still copyrighted traditionally, to protect Launchpad's visual identity."
<user___> That states that the images are copyrighted, and I know that that statement includes Launchpads logo, which makes sense.
<user___> But the images/icons also covers the little things too, like the plus buttons, user icons, etc.
<lifeless> yes
<user___> So since it
<lifeless> they were all custom designed to generate the look and feel
<user___> So since it's copyrighted, that means I'll have to use different icons, and thus replace all of them.
<user___> Oh.
<user___> Ugh, that's frustrating, but understandable.
<user___> I'll have to make a replacement then..
<user___> Anywho, now that question is cleared. Can anyone help me with getting the latest stable source and building it?
<lifeless> have you folloed the wiki pages for running launchpad on dev.launchpad.net?
<user___> Yes I have. Multiple times, and they keep resulting in problems and errors for me.
<lifeless> well
<lifeless> lets work through the problems
<user___> I also noticed that the rocketfuel-setup script downloads the latest development, and I don't want it, because it's not the stable source. Is there a way to get the stable source (I'm sure it could work through getting a certain revision)?
<user___> Or am I stuck with the development one?
<lifeless> I suggest getting going with devel before worrying about particular revs
<user___> Alright.
<lifeless> simply because if you're having trouble, adding variation to the happy path will make it worse, not better.
<user___> It'll take awhile to download and build.
<user___> Ok, I'll be back with a few errors and whatnot.
<lifeless> sure
<lifeless> if I'm not here - at a conf this weekend - I'm sure someone will be able to give you a hand
<user___> I've seen that Bazaar is closely linked with Launchpad, as you can download branches with commands like lp:
<user___> How would one go about making it possbile to work with another similar site (that uses Launchpad code)
<user___> Just curious.
<user___> Ok, the problems I get are these. It's everytime I try to make schema. What I'm trying to do is put the installation under /home/website/, have all the subdomains work with Apache (with a main page at dev.website.com), then have Launchpad startup everytime the server starts.
<user___> I figure it's better to get the stable release because of this anyways, it should be more stable, therefore more likely to build right, less bugs, etc.
<user___> Maybe it's best I give up and try to use Trac?
<user___> I wanted to use this because it's clearly superior.
<wgrant> user___: Why do you want to run your own instance?
<user___> I have my own server and website, and I dislike using other software on other sites. I like it when it's on my own.
<user___> It's 'prettier'.
<wgrant> Launchpad is not simple to run.
<user___> Besides, it gives me more knowledge.
<user___> Haha, so I've figured.
<user___> But worthit is also a good word too choose.
<lifeless> moin, agian.
#launchpad-dev 2010-06-27
<lifeless> wgrant: ping
<wgrant> lifeless: Hi.
<lifeless> hai
<lifeless> I was going to ask a bin/test question
<lifeless> but I figured it out.
<lifeless> also
<lifeless> if you want to cry
<lifeless> read
<lifeless> wgrant: are you at pyconau ?
<wgrant> I'm not.
<wgrant> That's all the way up in Sydney, isn't it?
<lifeless> :P
<lifeless> yes
<lifeless> evening
<jml> wgrant, sydney is perhaps closer to half the way up
<wgrant> jml: Shh.
<lifeless> who is on call reviewer atm ?
#launchpad-dev 2011-06-20
 * wallyworld_ off to a funeral. back later today
<lifeless> :(
<wallyworld_> yeah. sad.
<lifeless> \o/ deleting code
<wgrant> lifeless: Oh?
<lifeless> getBugCounts
<lifeless> unused
<lifeless> and getBugContextWhereSQLClause too, or however it was spelt
<wgrant> Ah, yeah, I noticed that looked a bit duplicated when I did the initial query rewrite.
<lifeless> _getBugTaskContextClause
<lifeless> actually
<lifeless> I could use a teddy bear
<lifeless> got 5?
<lifeless> wgrant: ^
<wgrant> Sure.
<lifeless> skype>
<wgrant> If it stops segfaulting...
<lifeless> \o/
<lifeless> I can ring your landline if thats easier?
<wgrant> Nah, just give me a sec to boot my laptop.
<wgrant> Hmm. Nobody is online... not even Skype Test Call.
<wgrant> Ah, there we go.
<StevenK> Cockatoos need to die.
<StevenK> Washing out on the balcony, and they attacked the pegs, so now the clean clothes need re-washing :-(
<StevenK> Stewart Smith
<StevenK> have noticed over $timeperiod improvements in #launchpad responsiveness and stability. #awesome
<StevenK> lifeless: ^
<lifeless> StevenK: hanging out in #drizzle ? :)
<StevenK> lifeless: No, that was from Failbook
<lifeless> oh cool
<lifeless> he was going to tweet it
<lifeless> also expecting a blog post too
<wgrant> :/ big deploy queue
<wgrant> And nothing we can do about it :(
<lifeless> wgrant: because of the subscribers thing?
<wgrant> Yeah.
<lifeless> how many revs apart was the initial landing and the fix?
<wgrant> Landed 13243, reverted 13247, relanded 13248, first fix 13258, still not OK.
<lifeless> oh? I thought danilo landed a fix?
<wgrant> It's OK for anonymous users now, sure.
<wgrant> Doesn't mean it's OK :)
<lifeless> how isn't it ok?
<wgrant> It's a 12000 line diff, so we can't be even slightly sure it's OK.
<wgrant> And IMO it's a regression.
<wgrant> I can't see who is subscribed to a bug any more :/
<wgrant> Oh, actually, I can, but it's not at all obvious.
<wgrant> ("Notified of all changes" is direct subscriptions, "Maybe notified" isn't valid English and is structural subscriptions, and the other categories are something else that I forget)
<lifeless> so, why can't we deploy ?
<wgrant> Last I heard they were still QAing it.
<lifeless> ugh
<lifeless> how can we help them
<wgrant> 22:33 < danilos> matsubara, right, so we still haven't marked it as qa-ok because I'd prefer to get a little bit more QA done first
<wgrant> That was Friday night.
<lifeless> what defines when a view does/doesn't have a self.user ?
<lifeless> or more precisely
<lifeless> why would a view not inherit from LaunchpadView ?
<lifeless> BugTargetBugListingView is what I'm staring at
<lifeless> ok, random q #2
<lifeless> is there a way to say 'any class that implements this Interface should allow the interface to be used'
<lifeless> rather than explicit <allow interface=xxx> statements for each implementing class?
<mwhudson> lifeless: i think "being incredibly ancient" is the only reason to not inherit from LaunchpadView?
<mwhudson> -?
<lifeless> well, who knows
<wgrant> mwhudson's understanding is the same as mine.
<lifeless> and on the second point ?
<wgrant> No. Why?
<lifeless> reducing duplication
<lifeless> grrrrrrrrrrrrrah
<lifeless> BugSummary.tag in [BugSummary.milestone_id]
<lifeless> True
<wgrant> Yeah...
<mwhudson> just by fluke?  or because types are being strange?
<lifeless> I'm guessing some of the query builder magic either has a corner case or a bug
<wgrant> It's the same Storm misfeature that wiped out part of the primary archive last year when I used in instead of is_in.
<wgrant> __contains__ returns an expression.
<wgrant> Which evaluates to True.
<wgrant> Wait, no.
<lifeless> but thats not applicable here
<wgrant> __eq__ returns the expression.
<wgrant> Which evaluates to true.
<lifeless> bug 799602
<wgrant> s/expression/Expr/
<_mup_> Bug #799602: something funky in comparisons makes introspecting things tricky <Storm:New> < https://launchpad.net/bugs/799602 >
<lifeless> and, I have bugsummary2 enabled stats portlet
<wgrant> These version numbers are a bit Chromy.
<lifeless> ?
<wgrant> Version 2 after like a month :P
<StevenK> Haha
<lifeless> wgrant: well, it won't be versioned live
<lifeless> its just a handle
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 205 - 0:[######=_]:256
<lifeless> I can has review ? https://code.launchpad.net/~lifeless/launchpad/bug-793848/+merge/65158
<StevenK> 1,110 lines at 5:20pm? Uh, no.
<lifeless> its mainly deletes
<lifeless>  21 files changed, 215 insertions(+), 337 deletions(-)
<lifeless> the way we assess patch size is pretty crazy
<StevenK> You only complain about that when you go over
<lifeless> of course
<lifeless> its crazy all the time
<lifeless> but doesn't come up in conversation otherwise
<StevenK> I shall pass, sorry. I'm developing a headache from DSP/DSPR/DSBP.
<lifeless> no worries
<lifeless> I'm getting one just reading those abbreviations
<StevenK> Welcome to my day.
<StevenK> henninge: O hai, Mr OCR.
<henninge> Hi StevenK! ;)
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 205 - 0:[######=_]:256
<henninge> StevenK: what can I do for you?
<lifeless> \o/
<lifeless> henninge: https://code.launchpad.net/~lifeless/launchpad/bug-793848/+merge/65158 when you have a chance ;)
<StevenK> henninge: https://code.launchpad.net/~stevenk/launchpad/db-add-bprc/+merge/64783 . I don't mind you if review lifeless' first, or mine, but my branch is smaller. :-)
<henninge> wow, 1110 lines of diff ...
<lifeless> yes yes
<lifeless> deletes do that :)
<henninge> oh, deteles are cool ;)
<lifeless> Diff against target: 1110 lines (+215/-337) 21 files modified
<lifeless> is a more useful metric I think
<lifeless> given the vast chunk of the line count is context
<henninge> yes yes :-P
<lifeless> :)
<lifeless> I think StevenK's was up first, you should save mine for later :>
<lifeless> (especially as I just noticed a spelling error to fix)
<StevenK> My DB review was on Friday, I spent most of the weekend doing the interfaces and model
<wgrant> StevenK: Reviewed.
<wgrant> Bah, you just roped henning into it, I see.
<henninge> wgrant: I was about to start ...
<henninge> wgrant: that's what the "claim review" button is for :-P
<wgrant> I don't trust buttons.
<adeuring> good morning
<henninge> Hi adeuring!
<adeuring> moin henninge!
<StevenK> wgrant: Thanks.
<wgrant> lifeless: How can column.__eq__(column) evaluate immediately?
<lifeless> wgrant: perhaps it can't
<lifeless> wgrant: I have a headcold. I'm excused from thinking.
<wgrant> True, true.
<StevenK> wgrant: Sadly, it is not a full path name, it is 'bin/true', for example.
<wgrant> StevenK: So it is. How stupid.
<lifeless> on the unicode thing
<lifeless> unicode(foo) is always unsafe
<wgrant> Unless it's always an ASCII string, yes.
<lifeless> yes
<StevenK> It's a path name
<wgrant> So using it on a potentially non-ASCII string is wrong.
<wgrant> StevenK: Paths aren't ASCII.
<lifeless> there are some source packages with non-utf8 paths around; I don't know about binary packages offhand
<StevenK> Query the conflict checker?
<lifeless> just check a current Contents.gz
<lifeless> easier
<StevenK> I don't know how to check for UTF-8
<lifeless> for line in thing: line.decode('utf8')
<lifeless> if it goes boom it wasn't utf8
<rvba> wgrant: Hi William, if you have time to take a look at my MP before you EOD it would make me happy ;) (https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676)
<rvba> I've worked with jtv on Friday on the perforance and I plan to do that again with Julian (testing real world queries) today. But if perf needs improvement I'd like to do that in another branch.
<StevenK> It goes boom
<lifeless> well then :)
<lifeless> so we need to change the db patch
<StevenK> Which is fine, it hasn't landed yet
<wgrant> Or say that if people have non-UTF-8 paths then they should die.
<lifeless> -or- we need to say that we'll exclude such things from the contents.gz
<wgrant> rvba: Looking, thanks for the reminder.
<lifeless> I'd favour talking to the distro now and then excluding non-utf8
<lifeless> it will make it a lot easier to work with.
<StevenK> usr/lib/OpenVerse/language/Espanoles                        universe/net/openverse
<StevenK> usr/lib/OpenVerse/language/Espa<F1>oles                     universe/net/openverse
 * StevenK facepalm
<lifeless> btw I'm pretty sure you can read the files straight out of the ar
<lifeless> lemme have a quick look
<wgrant> ar then tar, yeah.
<wgrant> It's easy enough to do.
<wgrant> But apt_pkg can do it for us..
<wgrant> (but the tarball may be compressed in several different ways)
<StevenK> It does?
<StevenK> wgrant and I already had this discussion
<StevenK> apt.debfile.DebPackage.filelist
<StevenK> lifeless: Your MP is out of date?
<wgrant> rvba: bigjools accepts the orphaned binary issue?
<wgrant> rvba: And the binary and source file conflict issue?
<rvba> wgrant: Hum ... since I now apply the same copying method to binary and source those two problems are not really there any more ...
<wgrant> rvba: they're still very much there.
<lifeless> StevenK: apt_inst.debExtract(open(local_file.name), capture_filepaths, "data.tar.gz")
<wgrant> Or apt.debfile.DebPackage.filelist...
<rvba> IIRC Julian was not really concerned by the orphaned binary issue right now since it's ugly but not an imminent danger to the integrity of the data.
<lifeless> StevenK: perhaps
<StevenK> lifeless: And I say again, what's wrong with apt.debfile.DebPackage.filelist ?
<lifeless> refreshing the page :)
<wgrant> I already complained at him for using subprocess, and he fixed it.
<StevenK> wgrant had this argument 5 hours ago.
<rvba> wgrant: can you elaborate on the "binary and source file conflict issue" ?
<wgrant> rvba: Orphaned binaries are ungood but do not cause irreparable damage. Binary and source file conflicts do.
<StevenK> wgrant: 13 is the number of files in pmount.deb, excluded directories. Do you want a comment or me to calculate it in the test?
<wgrant> rvba: Say I have hello 1.2-3 in some old series.
<lifeless> StevenK: wasn't meaning to cause extra work, was just looking at the need for a temp dir
<wgrant> StevenK: I'd like some verification that the filenames are right.
<StevenK> wgrant: Right, fair enough.
<StevenK> lifeless: I'm switching to tempfile.TemporaryFile()
<wgrant> rvba: So, from that I'll probably have a hello_1.2-3_i386.deb. That gets superseded in these series.
<StevenK> Or, attempting to.
<wgrant> rvba: What happens if I now initialise a new series in my distribution from another distro with a different hello_1.2-3_i386.deb?
<lifeless> TempDir is fine if we don't expect to be kill -9ing the process
<lifeless> but we may end up doing that :>
<rvba> wgrant: then only the package present in the *first* parent will be copied.
<rvba> s/package/packages/
<lifeless> StevenK: NamedTemporaryFile is as safe (or unsafe) as TempDir, FWIW
<wgrant> rvba: But that's enough.
<wgrant> rvba: Because the package present in the first parent is different from the one already in the archive.
<rvba> wgrant: each time we copy a parent into the (new) archive, we exclude from the copy any package with a name (sourcepackagename or binarypackagename) already present in the destination archive.
<wgrant> rvba: In the destination archive *and suite*.
<danilos> rvba, hi, do you think you'd be able to QA 795396 fix soon? allenap, how about you and 798263? :)
<rvba> danilos: hi! one sec.
<danilos> rvba, cheers
<StevenK> Damn it, does NamedTemporaryFile() delete the file on close?
<allenap> danilos: Okay, I'll do that now.
<lifeless> StevenK: yes
<lifeless> StevenK: it can't unlink it before then
<StevenK> Bah
<danilos> allenap, thanks a bunch!
<lifeless> StevenK: if a regular temporary file onw't work, then use your current code.
<wgrant> lifeless: Why use TempDir when a NamedTemporaryFile is fine?
<rvba> danilos: I'll be able to QA 795396 right now (if DF is up to date)
 * rvba checks DF
<lifeless> wgrant: shrug, was saying 'do the simplest thing'
<StevenK> NamedTemporaryFile() currently isn't, since I'm using copy_and_close() to grab the contents of the filename from the librarian.
<lifeless> wgrant: they are functionally equivalent
<lifeless> wgrant: in terms of diskspace and leaks
<wgrant> lifeless: They are. But creating a file is probably simpler than creating a directory and then creating a file.
<StevenK> And so copy_and_close() close()'s the temporary file and it gets deleted.
<wgrant> That's rather antisocial of it.
<lifeless> wgrant: I think StevenK has found a difference :>
<rvba> StevenK: Hi Steven, can you update DF?
<danilos> rvba, just let me know when it's done because I have a mixture of revisions landed that should only be rolled out together, and I can't mark the earlier one qa-ok until everything in between is qa-ok :)
<wgrant> rvba: I'm there already. Will do it now.
<rvba> danilos: sure, I will.
<rvba> wgrant: thanks.
<danilos> thanks all
<StevenK> Unless I can use something other than copy_and_close() to copy from the librarian?
<lifeless> you can, theres a pump_file in bzrlib.osutils
<wgrant> rvba: Should be coming back now.
<rvba> wgrant: so your point is that since I check that the package is already there in the destination archive and suite, it's still possible to have the exact same *file* (in two different archives) ... right?
<rvba> wgrant: thanks.
<StevenK> % grep -c pump_file eggs/bzr-2.3.3-py2.6-linux-x86_64.egg/bzrlib/osutils.py
<StevenK> 0
<rvba> s/two different suites/two different suites/
<wgrant> rvba: Right.
<rvba> s/two different archives/two different suites/ even
<wgrant> It may not be in the current suite.
<rvba> wgrant: right, I get it ... that's the point ou already made last week but I guess I did not understand it properly ...
<wgrant> It's an edge case, but this is the sort of place that an edge case will crop up and nomnomnom goes the primary archive. And we want primary archives to not be eaten alive.
<lifeless> +1
<rvba> wgrant: right.
<rvba> danilos: QA ok for 795396.
<danilos> rvba, great, thanks!
<rvba> np
<bigjools> morning all
<StevenK> Looks like copy_and_close() is it. :-(
<lifeless> StevenK:     pumpfile(from_file, to_file, read_length=-1, buff_size=32768, report_activity=None, direction='read')
<lifeless>         Copy contents of one file to another.
<lifeless>         
<lifeless>         The read_length can either be -1 to read to end-of-file (EOF) or
<lifeless>         it can specify the maximum number of bytes to read.
<lifeless>         
<lifeless>         The buff_size represents the maximum size for each read operation
<lifeless>         performed on from_file.
<lifeless> StevenK: (bin/py -m pydoc bzrlib.osutils)
<StevenK> lifeless: It seems NamedTemporaryFile() has a delete parameter
<rvba> wgrant: Thanks for the clarification, I'll work on this today.
<lifeless> StevenK: at which point its no longer a temporary file
<wgrant> rvba: Thanks. This isn't an easy thing, unfortunately :(
<rvba> wgrant: right, but with your help I'll see it through and land the 2500+ SLOC which depend on this ;)
<lifeless> StevenK: check the implementation - both the __del__ handler and the context manager will leave the file behind if you pass delete=False
<lifeless> StevenK: which is very much not what you want
<lifeless> myself, I'd use the tempdir and get on with my life ;). If you want to do something different, pumpfile will let you copy things around relatively sanely.
<StevenK>     self._debfile = apt_inst.DebFile(open(self.filename))
<StevenK> SystemError: E:Archive is too short
<StevenK> That's right after calling pumpfile()
<wgrant> seek(0)
<lifeless> you have a file already
<lifeless> seek(0) and pass the file in rather than a new file object.
<lifeless> henninge: I hope my review isn't frying your brain
<henninge> lifeless: no, it won't ...
<lifeless> do we have a list of supported browsers ?
<jml> good morning launchpadders
<lifeless> (bug 799491)
<_mup_> Bug #799491: Posting a comment in rekonq does not insert it into the page <Launchpad itself:New> < https://launchpad.net/bugs/799491 >
<jml> huwshimi & I are in Millbank
<lifeless> hola jml
<jml> lifeless: umm... I *thought* we did. huwshimi, do you know about such a list?
<huwshimi> lifeless: Yeah I think there is something like that around. Let me dig it out
<huwshimi> lifeless: This is what I was thinking of: https://dev.launchpad.net/GradedBrowserSupport
<lifeless> KLMN - its 2 years old?
<lifeless> anyhow
<lifeless> what I really care about is
<lifeless> do we consider bugs w/js on rekonq important, or do we say 'thanks for noticing, now please give us a patch' ?
<lifeless> [paraphrasing obviously]
<bigjools> rekonq uses webkit FWIW
<lifeless> yeah
<lifeless> still manages to fail - multiple different bugs open for it
<bigjools> not sure why the KDE people thought we needed another browser
<StevenK> wgrant: So, unicode() is wrong, what you prefer I use? filename.encode('utf8') and catch the encoding exception?
<lifeless> StevenK: if you're willing to not put non-utf8 stuff in the contents file, then yes.
<lifeless> StevenK: the alternative is to change the column from text to a byte array
<StevenK> And then use RawStr() ?
<jml> lifeless: if it's just a rekonq bug & not a webkit bug, it's the latter.
<lifeless> definitely not webkit; chrom* is fine
<jml> huwshimi: btw, I'm done w/ email stuff.
<huwshimi> jml: I'm almost there. 5-10 mins if that's ok?
<jml> huwshimi: sure.
<StevenK> wgrant: What would you prefer?
<huwshimi> jml: OK, done for now'
<jml> huwshimi: cool. want to come over here?
<huwshimi> jml: OK, coming
<lifeless> StevenK: excluding some files will need an ack from the distro I think
<lifeless> StevenK: that might be easy to get
<lifeless> I would prefer to drag stuff into the unicode-only world
<lifeless> so encourage you to seek that ack
<StevenK> lifeless: Given nothing uses this, and isn't likely to until I write a population script are you personally happy with me leaving it as is and landing it while I wait for the ack.
<lifeless> StevenK: I'd be happy with you doing .decode('utf8'), catching errors and skipping those files, and seeking the ack async
<lifeless> StevenK: I wouldn't be happy with the cast-as-is landing
<StevenK> Right
<jml> jelmer: do you need anything from me wrt build-from-branch-into-(archive|primary)?
<danilos> lifeless, whatever fails on webkit browsers is likely to fail on safari as well (I've hit a few problems like that with epiphany as well)
<danilos> for instance, use of "class" keyword for a variable fails in webkit browsers
<jelmer> jml: No, though thanks for checking. (I'm waiting to double check with poolie, then I'll send an announcement of the LEP to the list)
<jml> jelmer: ok cool.
<lifeless> bigjools: have you put any thought into query schemas for soyuz ?
<bigjools> none at all
<lifeless> k
<bigjools> sorry - too busy with DDs
<bigjools> when we start maintenance I will
<lifeless> no need to apologise
<lifeless> I ask so that if I get some time to look at it, I'm not reinventing already thought out stuff
<bigjools> sure
<danilos> bigjools, hi, do you know if jtv will show up today or if there's someone who can help QA bugs 394645, 537335?
<bigjools> danilos: he's having a day off
<danilos> bigjools, ah, ok, thanks; seems a pretty big change so I don't feel comfortable doing the QA, and if we can't find anyone else to do it, I suppose we won't have a rollout today
<bigjools> danilos: are they his changes ?
<danilos> bigjools, yeah
<bigjools> let me checkl
<danilos> bigjools, spread over 3-4 revisions in the deployment report
<danilos> bigjools, https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<danilos> bigjools, (the first blocking revision is QAd but I need 13258 to be included in the deployment before I can mark it as qa-ok)
<lifeless> danilos: I think its your patch blocking atm :)
<danilos> lifeless, see above, it's not
<lifeless> danilos: ah I see
<danilos> lifeless, I mean, I am keeping it like that on purpose
<danilos> (haven't used proper metadata in the commit message)
<lifeless> its fine, just a tad confusing
<lifeless> gnight everyone
<huwshimi> allenap: Hi, are you around?
<allenap> huwshimi: Yes, hi.
<allenap> huwshimi: Want to have a chat?
<huwshimi> allenap: Do you have a minute and want to talk about the initialization errors UI?
<allenap> huwshimi: What's your preference? Skype, Mumble, phone?
<jml> lifeless: g'night.
<huwshimi> allenap: Skype or Mumble is fine. I might try and trigger the error first though.
<huwshimi> allenap: Unless you have a screenshot?
<allenap> huwshimi: The error does not display anywhere in the page; it's an error in an job that runs outside of the app server.
<huwshimi> allenap: Ah I see
<allenap> huwshimi: I'm online in Mumble, in Launchpad/Red.
<huwshimi> allenap: OK, one minute
<huwshimi> allenap: Let me know if you have any more questions about that.
<huwshimi> allenap: I would also like to have a chat after you've implemented it to see how it went and how easy it was to do the UI side of things
<allenap> huwshimi: Cool, okay, thanks.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge, benji | Critical bugs: 205 - 0:[######=_]:256
 * jml afk
<henninge> adeuring: I won't make it to the call be fore :15
<henninge> Hi deryck, I wont make it until :15
<deryck> hi henninge.  see my #launchpad-ops message. :)
<henninge> not on there atm
<adeuring> henninge: everything postponed for >30 minutes :)
<henninge> cool
<deryck> henninge, yeah, sorry.  I need to wait to bottom of hour for standup.
<deryck> abentley, adeuring -- ping for standup
<Ursinha-brb> this is taking forever, argh
<maxb> Hi all - is r13249 deployable, and when is the next nodowntime deploy likely to occur?
<danilos> Ursinha-brb, hi, is there a way for me to mark a revision as fixed by one of the subsequent commits? (i.e. 13243/13248 is fixed by 13258)
<maxb> Ah, that half-answers my question :-)
<Ursinha-brb> danilos, if the revision referencesthe same bug, it'll be marked as needstesting and the first revision should be fine
<Ursinha-brb> what links the revisions are the bugs they refer to
<Ursinha-brb> maxb, we generally do the nodowntime deployments when we reach more than 5 revisions ready to deploy
<Ursinha-brb> when we reach five, I mean
<maxb> Ursinha-brb: amount of ready revisions isn't really the problem right now :-)
<Ursinha-brb> hehe
<maxb> It seems there's some deployability firefighting going on
<Ursinha-brb> let me see
<Ursinha-brb> hm
<danilos> Ursinha-brb, right, but they are linked to different bugs, and I don't want the first revision to be deployable before the latter one is :) I was hoping there'd be a qa tag like "qa-fixed-in-XXX" :)
<Ursinha-brb> there's not, unfortunately
<danilos> maxb, fwiw, I am intentionally blocking on r13243 before everything becomes deployable up to and including 13258, but there are some complex revisions that I can't QA myself
<danilos> Ursinha-brb, ok, then I'll just have to keep an eye on the deployment report, thanks
<maxb> 13260 is a rollback og 13521, so presumably we have to go all the way from 13242 -> 13260 at least, in one hop :-/
<maxb> Perhaps I should check back tomorrow :-)
<danilos> maxb, yeah
<wgrant> danilos: Please keep lifeless and myself up to date... we will attempt to deploy tomorrow.
<wgrant> Since you can't keep the tagger up to date :(
<danilos> wgrant, well, there's not much I can do: jtv's revisions need QA and it seems complex enough a change that I couldn't QA it this morning; if we decide to ignore the anon user problem, we can just mark qa-ok bug 772754, but that will only bring us two revisions forward
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
<wgrant> Argh
<wgrant> And no bigjools.
<danilos> wgrant, I did bring it up this morning with Julian, fwiw
<henninge> Hi benji!
<benji> hi henninge
<henninge> benji: Are you already working on a review?
<henninge> I just did Bryce.
<benji> henninge: not at the moment, I'm helping on something else
<henninge> ok
<adeuring> henninge, benji: could one of please review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-796867/+merge/65209 ? (~100 lines diff)
<henninge> adeuring: I can do it in a minute
<adeuring> henninge: great, thanks!
<jml> sinzui: hi
<sinzui> hi jml
<jml> sinzui: huwshimi & I are going to go into a room with a conference phone
<jml> sinzui: oh yeah, I've invited huwshimi to join us :)
<sinzui> I saw
<sinzui> read
<jml> sinzui: umm... can we dial a landline or voip number for you?
<sinzui> sip 7642
<jml> sinzui: thanks.
<jml> sinzui: "the number you have dialled is currently unavailable"
<sinzui> jml: call again
* henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 205 - 0:[######=_]:256
<jml> sinzui, huwshimi: âThe perfect search engine would understand exactly what you mean and give back exactly what you want.â
<jcsackett> benji: can you take another look at my MP? i believe the diff had been out of date, which confused the issue of your comment. https://code.launchpad.net/~jcsackett/launchpad/package-pickers-navigation-jumps/+merge/65222
<benji> jcsackett: sure, it'll be a little while though
<jcsackett> benji: no worries. thanks. :-)
 * jml is off for the day
<henninge> adeuring: approved. Sorry, I thought I had already done that.
<cjwatson> wgrant: any progress on deploying my germinate-lubuntu branch?
<adeuring> henninge: thanks!
<jelmer> abentley: hi
<jelmer> abentley: can bug 586944 be closed? AFAICT it's fixed
<_mup_> Bug #586944: Launchpad should provide a ui for daily builds <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/586944 >
<abentley> jelmer: hi.
<abentley> jelmer: I have no idea.
<jelmer> abentley: You filed it, and there's a merged branch associated with it.
<abentley> jelmer: okay, that's a different issue than I thought.  Yes, that bug could have been closed a year ago.
<jelmer> abentley: ok, I'll close it. Thanks
<benji> jcsackett: I finally got to look at your branch, it looks fine.  Please file a bug for the bad JS building.
<jcsackett> benji: will do, thanks!
<sinzui> Hail jcsackett. Do you have time to mumble?
<jcsackett> sinzui: sure. mumble or sip?
<sinzui> mumble?
<gary_poster> abentley, could you either handle https://answers.launchpad.net/launchpad/+question/160629 or tell me what I can do about it?
<gary_poster> deryck, someone wants to delete a hwdb submission.  Any ideas on what to tell them?
<abentley> gary_poster: Now that 13241 has been deployed, the next scan of the branch should succeed.
<gary_poster> Great news abentley, thanks.  I'll pas that on
<gary_poster> s
<abentley> gary_poster: in fact, it looks like that has already happened, as the last revision is from 3 hours ago.
<gary_poster> abentley, yeah, I had just come to the same conclusion.  Thanks again
<abentley> gary_poster: no problem.
<deryck> gary_poster, I thought they could delete it from their user profile on lp, but would need to poke at code or urls to confirm.
<gary_poster> deryck, ok.  I'll pass that on to the next CHR person :-)  Thanks
<deryck> np :)
<bac> hi sinzui, would you have a couple of minutes to chat with me for a pre-imp for bug 776437?
<_mup_> Bug #776437: Enable ARM builders for PPA via API <api> <escalated> <not-pie-critical> <oem-services> <ppa> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/776437 >
<sinzui> okay
<sinzui> bac: I am on mumble now
<bac> sinzui: ok, let me mumble up
<abentley> benji: could you please review https://code.launchpad.net/~abentley/launchpad/memlimit-jobs/+merge/65256 ?
<benji> abentley: sure
<abentley> benji: thanks.
<LPCIBot> Project windmill-devel build #252: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/252/
<bac> sinzui: is bug 760849 the root cause of this problem?  if i hadn't exported the collection (IProcessorFamilySet) would the error not have been seen?
<_mup_> Bug #760849: No way to restrict export_as_webservice_collection to a given API version <api> <lazr.restful:Triaged> < https://launchpad.net/bugs/760849 >
<bac> jcsackett: ^^
<abentley> sinzui: is it expected that bin/test --layer=YUITestLayer may segfault?
<jcsackett> bac: not sure of the question. there is the problem that once you export a collection it is exported on all versions of the API. which is a pain.
<bac> jcsackett: so is the solution to claim each entry first appeared in 'beta', as you did for IQuestion?
<jcsackett> bac: that is *a* solution. i don't know that it's *the* solution.
<bac> jcsackett: by "solution" i mean unholy work-around
<jcsackett> bac: ah, yes. it's definitely that. :-P
<jcsackett> bac: as it was explained to me (though i can't recall by who), the main thing is to make certain that the smallest possible set of stuff gets exported as of 'beta', and to make sure you don't export anything to it that's going to change.
<bac> jcsackett: rt
<sinzui> abentley, no. I was hoping deryck would follow up on the new package I build
<deryck> sinzui, I wasn't clear we had a new package, sorry.  I tired from devel last we talked and still seg faulting....
<deryck> sinzui, shall I update and try again?
<benji> Ursinha: do you know where I can find the disable_project.py script mentioned in https://dev.launchpad.net/Registry/ProjectReview?
<sinzui> abentley, The segfault is not seen on lucid or the natty/oneiric computers I have. I beleieve the issue a is a combination of libs
<jcsackett> benji: lp-dev-utils, i think.
<sinzui> benji, lp-dev-utils
<benji> cool, thanks guys
<abentley> sinzui: I'm running oneiric.
 * benji edits said wiki page.
<abentley> sinzui: sorry, natty.
<sinzui> deryck, please do, but maybe it is pointless since abentley is seeing it now
<sinzui> abentley, were you seeing this last Thursday when we talked about import gtk?
<abentley> sinzui: No, I wasn't running YUITestLayer then.
<abentley> sinzui: I was running twisted tests, and they were interacting badly with the import of html5browser.
<lifeless> moin
<sinzui> abentley, When I looked at deryck's report, the segfault happens in the first test of the second app (bugs)
<sinzui> abentley, the tests were never played in the same order, all app/ tests completed, bugs dies on the first test
<abentley> sinzui: http://pastebin.ubuntu.com/629993/
<sinzui> abentley, do you have the version of python-html5-browser that landed Saturday?
<abentley> sinzui: I do see an upgrade available.  I'll install it.
<sinzui> 0.0.4-0~18-9
<sinzui> abentley, the new packaging tries to avoid mixing gtk2/3 libs that can be implied by gir
<sinzui> I suspect that is the origin of the segfults
<abentley> sinzui: same result after upgrade.
<sinzui> :(
<sinzui> abentley, I am uncertain as to how to proceed. Have you tried ./bin/test -vvc --layer=YUI -t lp/bugs
<sinzui> to see if those tests complete
<deryck> sinzui, no luck still.  on latest package.  http://pastebin.ubuntu.com/629997/
<abentley> sinzui: I get segfault from your suggestion.
<benji> abentley: I'm done with https://code.launchpad.net/~abentley/launchpad/memlimit-jobs/+merge/65256, it looks good
<abentley> benji: cool, thanks.
<sinzui> Well you gentleman agree that is does not work. I have no clue. ec2 and jenkins run it, though I can see that there webkit lib behaves differently from what we use
<sinzui> abentley, deryck: I still suspect that there is an interaction with gtk2
<abentley> benji: I see your suggestion to use utilities/format-new-and-modified-imports, but it has no effect.
<sinzui> abentley, deryck: Mixing the two libs will always cause problem. My packaging efforts center
<sinzui> around be very precise about what must be installed
<benji> abentley: darn, I thought it worked like bin/lint.sh in that it used the branch info to figure out which files to examine
<abentley> benji: "utilities/format-new-and-modified-imports -r submit:" does the trick.
<benji> cool
<sinzui> deryck, abentley: I have another project that also needs to ensure libs do not mix. I am waiting for the builds to  so that I can test them. Maybbe that will help be find the right mix of gobject/pygtk/webkit
<deryck> ok, cool
<sinzui> deryck, !
<sinzui> I just saw compared your paste to http://pastebin.ubuntu.com/629993/
<sinzui> ^ abentley's
<sinzui> I think then that changes to tests/libs have mutated when/where we get a segfault
<deryck> I don't follow, sorry
<sinzui> deryck, I see three tests cases in that test_lp_names. maybe there is something naughty in that file
<sinzui> Maybe you could comment out some of the tests to identify an issue with how we write the test
<deryck> sinzui, ah, since we had the same test to fail.
<deryck> sinzui, FWIW, the failure is still dependent on order for me....still always the 7th regardless of which test.
<sinzui> deryck, I believe the test changed thursday/friday
<sinzui> ah
<deryck> ok, I'll comment out some stuff, though.  I don't mind trying.
<sinzui> bad things come in sevens
<deryck> sinzui, a couple more just now to demonstrate:  http://pastebin.ubuntu.com/630006/
<sinzui> deryck, there are seven app tests, what happens when you add -t lp/bugs?
<deryck> sinzui, bugs passes.
<deryck> trying app by itself
<deryck> bugs has 6 tests ;)
<sinzui> ho theres a shock
<deryck> app fails the same way at test 7.
 * deryck is trying all the app names....
<cr3> is there a function to blacklist pillar names like 'projects' and 'people'?
<abentley> sinzui: In my Natty vm, I get a different failure mode: http://pastebin.ubuntu.com/630014/
 * sinzui looks
<sinzui> abentley, and like deryck. The failure is in the seventh test.
<sinzui> I could sort the tests to take the confusion out of future analysis
<deryck> sinzui, so I've tested each module.... and only app fails.  and it's the only module with more than 7 tests.
<sinzui> deryck, it has exactly seven tests
<deryck> hmmm, exactly 7
<deryck> right
<sinzui> I see that bugs and app setup the tests identically
<sinzui> only the name App/Bugs names change
<deryck> sinzui, so I removed one test from app and the 7th still seq faults.  this time being the first bugs test.
<deryck> 7 is the magic number
<sinzui> deryck, I was thinking of changing YUIUnitTestCase.checkResults() return immediately to see if the tests return
<deryck> sinzui, ok.  you want me to try that as well?
<sinzui> deryck, please
<deryck> ok
<deryck> sinzui, just return at the top of the method?  Do nothing in the method?
<sinzui> Return immediately. I wonder if the json parsing is a mess
<deryck> sinzui, same seq fault.  same test 7.  even with the return.
<sinzui> ah, but we do json parsing in setup()
<sinzui> I am not thinking clearly
<deryck> sinzui, hmm, it's not as consistent now, though.  12th test fails now sometimes.  so getting further in some runs.
<sinzui> Deryck. I was thinking of make setup() return early by to see if browser/json is the issue
<sinzui> self._yui_results = dict(testing=dict(result='pass'))
<sinzui> return
<deryck> ok, trying that now....
<sinzui> deryck, abentleyI just brought another natty machine online to test withc
<sinzui> This has a rushed setup of deps.
<sinzui> While all the tests pass, I see something surprising in stderr before the tests start
<sinzui> Failed to load gnomesegvhandler
<deryck> sinzui, so playing in setup a bit.  and `page = client.load_page(html_uri, timeout=self.js_timeout)` seems to cause the seq fault...
<deryck> sinzui, at least I can return before that line, no issue.  return after it, boom.
<sinzui> deryck, okay, that is my-lib > webkit+gtk > gtk2/3 | libwebkit
<sinzui> deryck, are you running another webkit app at the moment? Empathy, Gwibber perhaps
<deryck> sinzui, both actually ;)
<deryck> sinzui, shall I kill them all and see... ?
<sinzui> :( so am I
<sinzui> I was hoping they were setuping up something right in my env
<sinzui> I am not running them in the new machine that reported gnomesegvhandler was not found
<deryck> ah, I see.
<sinzui> deryck, I am distracted now. My son was just stung by a bee
<deryck> sinzui, no problem.  I'm at EOD anyway.  We can chat again tomorrow about it.
<sinzui> abentley can you run
<sinzui> GTK_DEBUG=ALL ./bin/test -vv --layer=YUI -t lp/app 2>1 >  _test.log
<sinzui> and send/pastebin me the log
<abentley> sinzui: sure.
<abentley> sinzui: haven't dcced in ages.  Is it working?
<sinzui> I see the window after I accepted it, but I see no progress
<abentley> sinzui: mailed.  Must go.
<sinzui> Thank you very much
<mwhudson> jelmer: 203	+ except Exception, e:
<mwhudson> 204	+ if e.__class__ in self.unsupported_feature_exceptions:
<mwhudson> you know you can write this except  self.unsupported_feature_exceptions: ?
<mwhudson> although that gets subclasses i guess
<jelmer> mwhudson: no, I didn't realize that - thanks :) Subclasses would be nice to handle in this case too
<mwhudson> jelmer: or even isinstance(e, self.unsupported_feature_exceptions)
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 205 - 0:[######=_]:256
* benji_ changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 205 - 0:[######=_]:256
#launchpad-dev 2011-06-21
<sinzui> wgrant, stevenK mumble>
<wgrant> Still no builders :/
<LPCIBot> Project windmill-devel build #253: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/253/
<jcsackett> sinzui, wgrant, stevenK, wallyworld: i have to run. have a good one!
<lifeless> is getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR) totally equivalent to IMasterStore(class) ?
<wgrant> lifeless: No.
<wgrant> MASTER_FLAVOR
<lifeless> so when should we use IMasterStore?
<StevenK> I'd like to kill getUtility(IStoreSelector)
<lifeless> I'd like to kill all of it
<wgrant> lifeless: IStoreSelector and I(Master|Slave)?Store are alternatives.
<lifeless> we have had performance bugs talking to both stores
<lifeless> wgrant: sure; but when should we use which one?
<LPCIBot> Project devel build #820: STILL FAILING in 5 hr 33 min: https://lpci.wedontsleep.org/job/devel/820/
<wgrant> lifeless: There is no difference.
<wgrant> lifeless: Assuming you request the same flavour.
<lifeless> Is thre an IDefaultStore then ?
<StevenK> No, IStore
<LPCIBot> Project parallel-test build #52: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/52/
<wgrant> Right, DEFAULT_FLAVOR is IStore
<lifeless> right
<lifeless> so put another way, we probably need to do IStore everywhere and nuke the slave/master use outside of the policy setup
<wgrant> lifeless: Possibly.
<lifeless> why wouldn't we?
<wgrant> lifeless: Because in the past we have sometimes explicitly executed expensive queries on the slave.
<lifeless> in terms of db bloat, long transactions (> a few seconds) are bad on any replica
<wgrant> Load, not length.
<lifeless> yes, I know thats what you wrote
<lifeless> I'm picking it apart
<lifeless> in terms of memory footprint, queries that are large will spill to disk
<lifeless> in terms of cpu overhead, we've got more headroom on the master than on our slaves.
<lifeless> if its a script, I want them talking to appservers anyway.
<lifeless> I could see a case for parallel execution of complex queries, but our current stack isn't the right place for that anyhow
<wgrant> StevenK: http://paste.ubuntu.com/630105/
<LPCIBot> Project windmill-db-devel build #407: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/407/
<StevenK> wgrant: Mind having another look at https://code.launchpad.net/~stevenk/launchpad/db-add-bprc/+merge/64783 ?
<wallyworld> sinzui: ping
<wgrant> StevenK: Reviewed.
<StevenK> wgrant: You mean the long description of IBPP?
<wgrant> Its docstring.
<StevenK> Yes, as in the 3 line bit of the docstring?
<wgrant> Yes.
 * StevenK fixes that by removing it.
<StevenK> wgrant: with dest_file as tempfile.NamedTemporaryFile(): ?
<StevenK> Sigh, my knowledge of with sucks
<wgrant> StevenK: Other way around, but yeah.
<lifeless> -> buy a webcam for dublin
<jtv> morning friends
<wgrant> lifeless: qas is more than four months old. Shall we restore?
<wallyworld> wgrant: have you tried to ec2 anything lately? i'm getting loads of what appear to be unrelated (to my changes) errors
<wgrant> wallyworld: Not for a while. What's up?
<wallyworld> wgrant: lots of stuff, from ImportError: cannot import name PackageBuild in soyez/model/archive.py to bug subscription errors, and much more
<wgrant> wallyworld: This isn't your superconflictfun branch, is it?
<wallyworld> wgrant: no, i shelved the latter pipes and landed something based on earlier work
<wallyworld> the diff in the mp looks as i expect
<wgrant> wallyworld: What if you merge devel? No crisscrosses?
<wallyworld> not last time i tried which was yerterday. i'll try again. i just wanted to check first to see if it was "just me"
<wgrant> buildbot is not crying. So probably just you.
<wallyworld> :-(
<wgrant> The subscription stuff crisscrossed and had to be reverted when it was landed.
<wgrant> I'd merge devel into your branch and then check the rancestor diff.
<wallyworld> doesn't really explain the PackageBuild import error and some others
<wallyworld> since none of my changes went near there
<wgrant> Exactly.
<wgrant> Crisscross.
<wgrant> Particularly if you merged any of the Bugs branches at any stage.
<wgrant> Or anything even slightly related to them.
<wallyworld> i've only merged devel
<wgrant> Ever?
<wallyworld> so far as i can recall
<wallyworld> just merged in devel, no aparent problems
<wallyworld> but then again, there weren't that many new changes from yesterday
<wgrant> wallyworld, StevenK: Do you see any reason to not restrict the email address search to (email = ? OR email LIKE ? || '@%%')?
<wgrant> I doubt searching for partial email addresses is common, except for just specifying a local part.
<wgrant> (and we get heaps of false prefix matches :/)
<wallyworld> wgrant: i think that's a goof idea to implement
<wallyworld> wgrant: bug 798847
<_mup_> Bug #798847: Person picker search for partial IRC nickname doesn't return intended result <disclosure> <exploratory-testing> <person-picker> <Launchpad itself:Triaged> < https://launchpad.net/bugs/798847 >
<wallyworld> wgrant: do we really want to search on partial irc nicks
<wgrant> wallyworld: I saw that and considered won'tfixing it.
<wgrant> It's a barrel of worms.
<wallyworld> i agree
<wgrant> Can of worms.
<wgrant> Whatever the idiom is.
<wallyworld> barrel of monkeys too
<wallyworld> especially since the example was 'jools' and the user was searching for 'bigjools'
<wallyworld> normally one would enter a prefix, not the last characters in a name
<wgrant> That really requires tokenising IRC nicks further.
<wgrant> And uh, no.
<wallyworld> you mean you search for partial suffixes?
<lifeless> wgrant: sure
<wgrant> wallyworld: No. But to implement it would either mean matching any substring, or we'd have to guess at tokenising it without whitespace.
<wgrant> So... no.
<wgrant> Kill it.
<wallyworld> wgrant: yes, that's why i said i agree we shouldn't do it :-)
 * wgrant won'tfixes.
<wgrant> Or perhaps I should wait for sinzui.
<wallyworld> wgrant: i was going to ask him about that and also the picker batch widget bug but he had already EOD
<wallyworld> i should say the pagination control bug 798772
<_mup_> Bug #798772: Pagination in person picker should appear after the results, not before <disclosure> <exploratory-testing> <person-picker> <Launchpad itself:Triaged> < https://launchpad.net/bugs/798772 >
<wgrant> wallyworld: We probably want to completely rework that.
<wgrant> wallyworld: Remove the page numbers and stuff.
<wallyworld> that is done in lazr-js so are we saying we want to change the paginator location for *all* pickers or just personpicker
<wgrant> wallyworld: Since it makes lifeless war on batchnav very hard.
<wgrant> So if we're going to move it to the bottom and break from lazr-js, we should really change it.
<wallyworld> well, one step at a time perhaps
<wallyworld> don't want too much scope creep
<wallyworld> i'm wondering how that bug got tagged disclosure anyways
<wgrant> Which?
<wallyworld> the paginator one
<wgrant> It was discovered during exploratory testing of the new shiny.
<wallyworld> above ^^^
<wgrant> So it's our problem unless we decide it isn't.
<wallyworld> sure, but it's been there forever and is not related to our work
<wallyworld> at all
<wgrant> Our work was to make the pickers not suck.
<wgrant> Search was part of that.
<wgrant> Displaying more relevant info was another.
<wallyworld> hmmm. not sure how having the paginator at the top makes the pickers suck
<wallyworld> i like it there
<wallyworld> less mouse movement
<wgrant> Indeed.
<wgrant> But some disagree.
<wallyworld> well then they are wrong :-P
<wgrant> Indeed!
<lifeless> FWIW, paginator can be above or below IMO; page #'s however... poor idea
<lifeless> which number will your result be on?
<wgrant> lifeless: Poor and pointless.
<lifeless> its like a lottery
<wgrant> If you
<wgrant> If you have to click past the first page for a given person more than once, we have failed.
<wallyworld> the only thing i see them adding any value for is to indicate how many results there are, but that can be done other ways
<StevenK> WOW.
<StevenK> When did we hit bug 800000?
<wgrant> wallyworld: But it's limited now, so it's only useful for very small numbers.
 * StevenK prods _mup_ 
<wgrant> Huh, so we did.
<wgrant> Just noticed I filed bug #800043
<_mup_> Bug #800043: IRC nickname search in person picker is case-sensitive <disclosure> <person-picker> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/800043 >
<wgrant> 800000 must be private.
<wgrant> Yeah.
<lifeless> wallyworld: showing the result set size is generally uninteresting
<wallyworld> should we have a 1000000 party next month?
<wgrant> Not even I can see it.
<lifeless> wallyworld: there are a very few specific cases when its anything other than geek chic
<StevenK> bug 799999
<wallyworld> lifeless: i think it adds value but that's just me
<lifeless> wallyworld: consider the phone book; do you care how many booths there are in it ?
<wallyworld> yes!
<lifeless> wallyworld: and how many freds ?
<wgrant> It gives a hint as to the selectivity of your search.
<wgrant> Whether you're likely to find the person in the next few pages.
<lifeless> yes, the magnitude matters
<wallyworld> when searching, if there are too many results, it gives you a cue to be more specific, especially if you want to browse the results
<lifeless> few/10s/100s/many
<wallyworld> wgrant: +1. what i was trying to say
<lifeless> the precise count doesn't matter most of the time.
<lifeless> and magnitude estimation isn't one of those times.
<wallyworld> agree with that too
<wallyworld> we should do it as a slider :-)
<wallyworld> as you move  the slider, the results pan past
<wallyworld> that would be way cool imnsho
<lifeless> vertical slider would be nice
<wallyworld> yes
<lifeless> even just vertical next/prev markers
<lifeless> rather than horizontal for things that are laid out vertically.
<lifeless> that always gets me.
<wallyworld> and also doesn't really suit right-left locales either
<lifeless> question
<lifeless> are anonymous users None, or something else?
<lifeless> in terms of LaunchpadView's self.user
<wallyworld> i guess None but am not 100% sure
<wallyworld> i recall some test code doing a self.user == None check
<wgrant> None
<wgrant> Bah.
<wgrant> The email address prefix is useful :(
<wgrant> For eg. "jon" won't catch jcsackett without it.
<wgrant> Because his displayname is crazy.
<rvba> wgrant: Hi! I'm sorry to bother you again with this but could you have a look at the multi parent branch? (https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676)
<wgrant> rvba: And what about sources?
<wgrant> And are you regretting trying to do this yet? :P
<rvba> Well, not yet :)
<wgrant> Ah, good, Soyuz has got your soul already.
<rvba> wgrant: So I suppose the same trick should be used for the sources then ...
<wgrant> I guess so :/
<wgrant> Also this is probably going to be very slow.
<wgrant> But maybe not.
<rvba> I'll estimate the overhead of this.
<lifeless> wgrant: jcsackett working is a bit of a special case
<wgrant> lifeless: It is, but it helps for some other cases too.
<wgrant> Because name stemming is hard.
<lifeless> wgrant: folk that don't make their metadata match the identity they tell their peers are going to be pretty hard to find
<wgrant> lifeless: Thanks.
<adeuring> good morning
<rvba> wgrant: Fix pushed (tests added: test_binaryfile_already_in_archive/test_sourcefile_already_in_archive).
<lifeless> grar
<lifeless> serial conflicts
<lifeless> *hates* on 4 hour test runs
<StevenK> Then fix it, dear Eliza?
<lifeless> StevenK: its in the pipeline
 * StevenK is only trolling.
<lifeless> I know
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 205 - 0:[######=_]:256
<maxb> Morning all. How is the deployment report looking today?
<jml> good morning Launchpadders
<jml> I was getting productive email stuff done and thought, "why am I so productive"
<jml> and then I realized my IRC client was off
<wgrant> maxb: Somewhat dire.
<wgrant> Still blocking on lots of reasonably nasty jtv QA in the middle of the 12000 line branch.
<wgrant> And Red squad seems to not be here today.
<wgrant> (your rev is after the 12000 line branch, its reversion, and its relanding, but there is another QA fix after that, and before that is jtv's stuff)
<rvba> wgrant: Julian is ill. Jtv has trouble with irc. The rest of us is here ;)
<wgrant> rvba: Do you know if he's working on his QA?
<rvba> wgrant: jtv?
<wgrant> rvba: Yes.
<rvba> wgrant: I don't know ... He can be reached by email.
<rvba> wgrant: I've fixed the problem we talked about earlier this morning (https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676)
<wgrant> rvba: I think we need to chat with Julian about this.
<rvba> wgrant: ok.
<wgrant> Hmmm.
<wgrant> All domestic flights in south-eastern .au cancelled tomorrow. And all internationals delayed probably until Friday.
<wgrant> This does not bode well.
<StevenK> Indeed
<jml> wgrant: Chile volcano?
<huwshimi> jml: yeah
<jml> huwshimi: you managed to escape though.
<huwshimi> jml: There have been delays for the last week or so
<nigelb> Wait, a volcano in Chile is affecting .au? o.O
<huwshimi> jml: Yeah, just
<wgrant> nigelb: The ash is everywhere.
<wgrant> Well.
<wgrant> Mostly above us now.
<lifeless> nigelb: ash going west
<wgrant> But still everywhere.
<nigelb> Ahh. I forgot about the earth being round temporarirly.
<jml> nigelb: yeah. all that hippy crap about us being interconnected beings on one planet turns out to be true
<nigelb> jml: lol
<nigelb> wgrant: Are you going to miss the sprint or rally or whatever its calle dnow.
<nigelb> *called now
<wgrant> nigelb: Hopefully not!
<nigelb> :)
<wgrant> I think it's still a Thunderdome.
<wgrant> But I'm not sure.
<wgrant> Maybe just an Epic.
<wgrant> Which is colocated with the Platform Rally.
<nigelb> Fun!
<nigelb> jml: On reading the blog post about your depature, I first feared you were leaving Canonical :)
<jml> nigelb: nope
<jml> nigelb: very glad to be staying
<nigelb> jml: :)
<cjwatson> LP's loss is our gain :-)
<nigelb> heh
<lifeless> jelmer: hi
<lifeless> jelmer: did you redefine the meaning of persisted enums in your code import patch?
<jelmer> lifeless: hi
<jelmer> lifeless: These were prematurely added constants according to mwhudson, never used in real life.
<lifeless> k
<lifeless> argh, the deploy queue just got riskier still
<lifeless> jelmer: how would you feel about rolling back that code import autoupgrade thing?
<jml> just came across https://dev.launchpad.net/LaunchpadHackingFAQ
<jml> there's a blast from the past.
<lifeless> jelmer: we've a few high risk things already in the queue, and with yours added we may be unable to deploy for days
<lifeless> jelmer: if you're -really- confident in it, its fine.
<lifeless> jelmer: I'm also curious why you didn't just fix the scanner to handle the new backup directory scheme bzr uses ( I think there is a code hosting bug on that)
<lifeless> s/just//
<wgrant> lifeless: s/days/even more days/
<wgrant> But I think jelmer's fix looks good, FWIW.
<jml> *sigh*
<jml> https://dev.launchpad.net/Hacking is a fork of LaunchpadHackingFAQ
 * jml sets about merging the two and deleting the latter
<wgrant> cjwatson: Nothing of note this week that will prevent a couple of minutes' downtime for cocoplum services, while we update it for your germinate change?
<lifeless> allenap: how is rabbit fixture looking ?
<allenap> lifeless: I landed it on Saturday :) No failure so far (that I'm aware of) \o/
<lifeless> \o/
<lifeless> allenap: did you file a bug upstream?
<cjwatson> wgrant: nope, please go ahead
<allenap> allenap: No. I wonder if I should file it on Erlang or RabbitMQ.
<lifeless> allenap: rabbit for starters
<lifeless> they offer programs after all :)
<allenap> lifeless: Okay.
<lifeless> allenap: also, do you want to land my layer patch now the substrate is solid ?
<allenap> lifeless: Yeah, good idea.
<jml> do we have a standard wiki macro for "obsolete page"?
<nigelb> jml: remember jorge's lightning talk? ;)
<nigelb> Delete! Delete! Delete!
<jml> nigelb: yeah, that was the Ubuntu wiki
<jml> and I have deleted some stuff already
<nigelb> I didn't know there was help on tal stuff till now.
<nigelb> I went and read zope documentation for that and didn't understand much.
<StevenK> Tal is quite ... opaque.
<nigelb> StevenK: Interesting choice of words.
<jelmer> lifeless, sorry, I missed that
<jelmer> lifeless, bzr's scheme hasn't changed, but it has one method to backup an existing and one for renaming a directory out of the way
<jelmer> they have different naming schemees. the second is only used by "bzr join" at the moment.
<lifeless> jelmer: so how did we end up with the latter scheme named directories during the incident?
<jelmer> lifeless: I called that method manually
<jelmer> I don't think bzr should have two name schemes for backed up .bzr directories
<lifeless> you're going to change join ?
<jelmer> lifeless: I filed a bug about it
<deryck> Hi, abentley and adeuring
<abentley> deryck: hi.
<deryck> abentley, adeuring -- so it's very noisy in this place.  Can we do an IRC standup?
<abentley> deryck: sure.
<deryck> let's wait on adeuring to ack before we start.
<abentley> deryck: we could make a temporary channel, or use #launchpad-meeting, if you like.
<adeuring> deryck: abentley: I'm here
<deryck> abentley, sure, let's move to #launchpad-meeting to avoid too much noise here.
<deryck> adeuring, hi.  changing channels ^^
<jml> jelmer: you'll be in Dublin, right?
<jelmer> jml:Yep
<adeuring> deryck: could you please run this SQL statement on staging: http://paste.ubuntu.com/630399/ ?
<deryck> adeuring, sure.... grabbing it now
<abentley> adeuring: why "select 1"?  Doesn't that just return "1" always?
<danilos> matsubara, hi, is there a way to easily search for any OOPSes beginning with "IntegrityError: duplicate key value violates unique constraint \"tm__potmsgset" across all of our systems?
<adeuring> abentley: I have in idea yet. I guess that the idea is to see if the is any record matching the WEHER clause
<matsubara> danilos, I can query the oops db using SQL. there's no UI to do such search in oops-tools
<adeuring> s/in idea/no idea/
<adeuring> abentley:  I even don't know yet where the query comes from in Python ;)
<abentley> adeuring: I think SELECT 1 will always return 1.
<danilos> matsubara, can you please do that for the above exception text? after "tm__potmsgset" there could be different things
<adeuring> abentley: except if there are no result rows
<danilos> matsubara, if LIKE "blah%" is too slow, you can look for actual constraints from https://bugs.launchpad.net/launchpad/+bug/603530/comments/5
<_mup_> Bug #603530: Still violating tm__potmsgset__potemplate__language__no_variant__diverged__impo <lp-translations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/603530 >
<matsubara> danilos, thanks. LIKE syntax is not returning me any results
<abentley> adeuring: Ah, I see.
<deryck> adeuring, here you are.  with two runs, to warm the cache.  https://pastebin.canonical.com/48802/
<adeuring> deryck: thanks!
<deryck> np
<adeuring> abentley: ^^^ So, now I'll stare for a while at the result (practically, I'll print it out so I can make notes with a pencil)
<danilos> matsubara, cool, I think we can call that bug closed, so we just solved one critical bug :) yaay ;)
<matsubara> danilos, not so fast
<matsubara> danilos, https://devpad.canonical.com/~matsubara/danilo-integrity-error.txt
<danilos> matsubara, oh, :(
<danilos> matsubara, these are all still from last year, before this code was rolled out
<danilos> ok, now there's a few new ones
<matsubara> danilos, reload. the file was incomplete. it should show you 972 rows
<danilos> matsubara, yeah, the final ones seem to be from March 1st
<matsubara> danilos, hmm perhaps I should order that by date to make things easier
<danilos> matsubara, yeah
<danilos> matsubara, that'd be helpful :)
<matsubara> danilos, ok, refresh and it'll be ordered
<danilos> matsubara, thanks; considering they were happening every day multiple times until March 1st, I'll still call the bug fix released if it hasn't ever happened since
<matsubara> danilos, sounds good
<danilos> matsubara, also, I can't get to any of those OOPSes myself
<danilos> matsubara, i.e. does https://lp-oops.canonical.com/oops.py/?oopsid=1886M2307 work for you?
<matsubara> danilos, yeah, those are not referenced in the LP db, so the oops purge deleted them
<danilos> matsubara, ah, ok, thanks
<sinzui> deryck, can you run GTK_DEBUG=ALL ./bin/test -vv --layer=YUI 2>1 >  _test.log and send/pastebin the log for me
<deryck> sinzui, definitely.  Thanks for continuing to work on this!
<deryck> was going to ping you too :)
<adeuring> abentley: There is a line "sort method: external merge" in the "explain analyze" output: That for the "UNION" statement, I assume. One of the two subqueries needs 13 seconds, and the other a few milliseconds. So, the first thing that could be changed is: Run the queries separately. that would save 7 seconds. Not enough, but it could be a start. Next thing I'll do is to figure out where the query is issued in Python.
<deryck> sinzui, unfortunately, not much to see:  http://pastebin.ubuntu.com/630410/
<sinzui> oh, I get huge amounts for data
<sinzui> maybe my stderr redirect is mangled
<abentley> adeuring: if it's just running an existence check, I wonder whether adding LIMIT 1 to the subselects would help.
<adeuring> abentley: right, that could help. Lets' try it!
<deryck> sinzui, I did 2>&1 > file, which I thought was the correct form.  but same output.
<deryck> sinzui, maybe I need debug version of a package?
<adeuring> deryck: could you please try another query: http://paste.ubuntu.com/630412/ ?
<deryck> adeuring, sure.
<sinzui> deryck, I fear that I can see the needed data because I am using development vrsions of gtk libvs
<sinzui> deryck, I think your libs are compiled with debug off
<deryck> sinzui, yeah.
<deryck> sinzui, I don't mind compiling if you want, but I fear that might also fix the issue :)
<sinzui> I get 2000 lines of output
<huwshimi> deryck: Do you want to reschedule our call today? Sounds like things are a bit crazy where you are.
<deryck> huwshimi, it's still top of next hour, right?
<huwshimi> deryck: Yeah
<deryck> huwshimi, I'll be fine for that.
<huwshimi> deryck: OK sure, as long as it's easy enough for you
<deryck> huwshimi, oh yeah, np.  I'll tell everyone else to go away in 40 minutes. :)
<sinzui> deryck, you would need libgtk2*-dbg and libwebkitgtk-3*-dbg and I do not know if they replace your current libs
<huwshimi> deryck: haha
<deryck> sinzui, ok, let me apt-get install
<deryck> adeuring, https://pastebin.canonical.com/48809/
<adeuring> deryck: thanks!
<deryck> np!
<deryck> sinzui, I need to switch back to my main office.  be back online in 10.  and I'll finish this install and try again then.
<adeuring> abentley: ^^^ so, some progress, but the sorting for the first subquery needs 4 seconds, even in the second run. I'll look now where the query is issued and will try to understand why it is executed.
<deryck> sinzui, I'm back working on this.  dbg package for libwebkitgtk is taking forever to download.
<sinzui> :(
<sinzui> I wish one of my computers would show the segfault
<sinzui> though my daughters are happy I stopped taking their computers to test this
<deryck> sinzui, so nothing new in the log even after the dbg packages
<sinzui> :(
<nigelb> jcastro: woah, good one!
<nigelb> argh
<deryck> sinzui, can you paste the command you're using to get the log once more?
<sinzui> New yucky attempt: strace -Ff -tt ./bin/test -vv --layer=YIU 2>&1 | tee _strace.log
<deryck> now there's some noise :)
<sinzui> deryck, old with noise: GTK_DEBUG=ALL ./bin/test -vv --layer=YUI 2>&1 | tee  _test.log
<adeuring> abentley: The "bad query" is executed during Snapshot(ubuntu) in lp.app.browser.launchpadform.LaunchpadEditFormView.updateContextFromData(). So, a "regular property", don't know yet which one it is.
<sinzui> I am familiar with the gtk output. strace will challenge my archeology skills.
<huwshimi> deryck: Ready when you are. How do you want to do this?
<deryck> huwshimi, 3 minutes and I'll meet you in mumble, cool?
<sinzui> okay
<huwshimi> deryck: Yeah that's fine
<deryck> sinzui, heh, the paste is straining my browser to return control to me. :)
 * deryck tries chromium
<deryck> sinzui, I can't get a pastebin to load it, so I'm pushing it to my home on devpad as yui-strace.log.  15% now. :)
<sinzui> okay
<huwshimi> deryck: Do you want to just let me know when you're done. There's no hurry
<deryck> huwshimi, joining mumble now
<huwshimi> deryck: Sure
<deryck> huwshimi, meet me in Orange 1 o 1.
<huwshimi> deryck: no problems
<adeuring> abentley: tracing through Snapshot.__init__() with pdb and watching the postgersql statement log, it seems that Distribution.has_published_sources issues the bad query.
<gmb> adeuring: Ping for when you've got a second.
<adeuring> gmb: yes?
<gmb> adeuring: There's a question on answers.lp.n about deleting HWDB submissions; I have no idea about it. Could you take a look if you ahve the time: https://answers.launchpad.net/launchpad/+question/161972
<gmb> ?
<adeuring> gmb: sure
<gmb> Thanks
<flacoste> danilos: what's happening with the qa of bug 772754?
<_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
<flacoste> rvba, allenap: can one of you assess if we can deploy revision 13259, please?
<danilos> flacoste, same as yesterday, it's qa-ok if r13258 is included in the deployment, but revisions in between are still in qa-needstesting; I've emailed jtv asking him to QA his 3 non-trivial revisions
<gary_poster> flacoste, danilos, jtv replies saying that bigjools needed to do the qa
<flacoste> danilos: the only needstesting that i see before 13258 are yours actually
<gary_poster> but he is out
<rvba> bigjools is not here today :(
<danilos> flacoste, maybe they've done it in the last 30 mins or so
<flacoste> rvba: i know, that's why i'm asking you :-)
<flacoste> or allenap
 * rvba looking
<flacoste> that's related to what you guys have been working on
<flacoste> danilos: what do we need to do to make the report happy, for you revision, do you know?
<danilos> flacoste, they are all for 772754, and I've marked it as qa-ok now that it's all clear up to 13258
<danilos> flacoste, wait for it to update now :)
<flacoste> danilos: awesome!
<flacoste> thanks
<danilos> yw
<timrc> jcsackett, hello, you there?
<rvba> flacoste: just to make sure, you're talking about http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13259 ?
<flacoste> rvba: yes
<rvba> Looks pretty safe to me, it's a simple change in a message displayed on the difference page.
<flacoste> jcsackett: i think you made a mistake in your commit message
<rvba> If DF is up to date I can even qa it.
 * rvba checks.
<flacoste> jcsackett: revision 13251 looks like it's still blocked, but I think your 13260 commit actually reverts it
<flacoste> jcsackett: but it was tagged rollback-13521
<flacoste> jcsackett: if that's the case, maybe changing the tag will make the qatagger script happy
<flacoste> danilos: i think deployment is still blocked on jcsackett and rvba as we cannot deploy without the rollback of revision 13251
<rvba> flacoste: I marked bug 798222 (fixed by 13259) as qa-ok because the fix is a one-liner fixing a simple 'display' bug.
<_mup_> Bug #798222: +localpackagediffs should explicitly say "no diff available" <derivation> <qa-ok> <Launchpad itself:Fix Committed by julian-edwards> < https://launchpad.net/bugs/798222 >
<rvba> flacoste: is that ok with you?
<flacoste> rvba: fine withe me, thanks
<poolie> gmb: can you pilot barry's patch to bug 797281?
<_mup_> Bug #797281: LP API broken in oneiric with python-httplib2 0.7.0-1 <oneiric> <lazr.restfulclient:In Progress> <python-httplib2 (Ubuntu):Fix Released by barry> <python-httplib2 (Ubuntu Oneiric):Fix Released by barry> < https://launchpad.net/bugs/797281 >
<gmb> poolie: Sure thing.
<poolie> ta
<LPCIBot> Project windmill-devel build #258: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/258/
<jcsackett> flacoste: sorry, didn't get msg updated when you pinged me.
<flacoste> poolie, gmb: i'm actually on it
<jcsackett> 13260 is indeed a rollback of 13251.
<adeuring> deryck: could you try another query: http://paste.ubuntu.com/630443/ (probably not very much progress, but anyway...)
<deryck> adeuring, sure.
<deryck> sinzui, my strace is up on devpad now, btw.
<gmb> flacoste: Oh. Bugger, I've just merged that and pushed it since it was r=.
<flacoste> gmb: fine by me :-)
<flacoste> gmb: technically, we should run the tests manually
<flacoste> since there is no automated PQM test run there
<flacoste> just to make sure
<gmb> flacoste: yes, that thought occurred to me after I'd pushed it. I'll do that now.
<flacoste> but i'm sure barry did it
<flacoste> :-)
<sinzui> deryck, thank. I pulled it a while ago
<deryck> sinzui, ok, cool, thanks for the help!
<deryck> adeuring, looks fast to me.  :)  https://pastebin.canonical.com/48812/
<jcsackett> flacoste: i'm not sure what the tag issue is; looking at our qaprocess stuff in the wiki it's tagged appropriately. a bad commit tag for the bad revision, and all qa tags removed. thoughts on what it should be?
<flacoste> jcsackett: the message was [rollback=13521]
<adeuring> deryck: uh, yes, thanks! I assumed that we would have an ORDER BY which would slow down the query, but this is good eoungh, I think
<deryck> adeuring, excellent.
<adeuring> deryck: now I just need to duble-check that it was the right query :)
<jcsackett> flacoste: that was the bad revno. qaprocess says rollback=badrevno, unless i'm being dense?
<flacoste> Ursinha: how can we unblock the qatagger?
<deryck> adeuring, so you need me to run it again?
<adeuring> deryck: no, I just want to double-check if I did not make a copy_paste error or some other nonsense
<flacoste> jcsackett: deployment-stable.html shows "[r=jcsackett][bug=707234][rollback=13521] " has the commit message
<flacoste> which is not the correct revision
<flacoste> but the tags are correct
<jcsackett> flacoste: aaah, crap, yes i see.
<deryck> adeuring, ah, ok.
<flacoste> anyway, we should be good to deploy up to 13265
<flacoste> so i'm going to ask for that
<flacoste> after lunch
<jcsackett> sounds good.
<timrc> jcsackett, do you think you'll be able to attempt to ec2-land my changes again :)?
<abentley> sinzui: I also get the segfault on my natty vm, now that I've properly updated it.
<sinzui> abentley, does the vm install anything special? Is it natty + rocketfuel?
<abentley> sinzui: Not special.  An X86-64 Ubuntu desktop.  I can provide a package list if useful.
<abentley> sinzui: it does have libgtk2.0 on it.  Shall I pull that?
<sinzui> oh
<sinzui> deryck, are you 64bit?
<sinzui> abentley, a package list would rock.
<deryck> sinzui, indeed I am.
<deryck> sinzui, I should have mentioned that.  it's important.  sorry.
<jcsackett> timrc: sure, paste me the link, and i'll see to it in a moment. :-)
<timrc> jcsackett, I pushed a new branch 4 days ago that fixed those test failures.. should I merge in trunk before attempting this again.
<timrc> ?
<timrc> (it's been 4 days)
<timrc> https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-4/+merge/64840
<timrc> jcsackett^
 * jml pulls stable, first time in a while
<adeuring> gmb: fancy a review of a small MP: https://code.launchpad.net/~adeuring/launchpad/bug-799785/+merge/65373 ?
<gmb> adeuring: Sure
<LPCIBot> Project windmill-devel build #259: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/259/
* gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 205 - 0:[######=_]:256
<gmb> adeuring: Approved
<adeuring> gmb: thanks!
<Ursinha> flacoste: sorry, I was lunching and forgot to change my nick
<Ursinha> let me read the backlog
<jcsackett> timrc: you should basically always update from trunk before trying to land.
<Ursinha> flacoste: if the problem is that the commit msg has the wrong number, quick workaround is remove the tag from the bug; proper solution is to fix bug 659629
<_mup_> Bug #659629: qa metadata for commits is write-once and cannot be updated <qa-tagger:Triaged> < https://launchpad.net/bugs/659629 >
<LPCIBot> Project db-devel build #652: STILL FAILING in 6 hr 10 min: https://lpci.wedontsleep.org/job/db-devel/652/
<flacoste> Ursinha: you mean remove the bad-commit-13521 tag?
<flacoste> err, bad-commit-13251
<Ursinha> yes
<Ursinha> if we know that is ok to land, I believe adding qa-ok would do the trick
<LPCIBot> Project parallel-test build #56: STILL FAILING in 46 min: https://lpci.wedontsleep.org/job/parallel-test/56/
<LPCIBot> Launchpad Patch Queue Manager: [r=lifeless][bug=797820, 800043] In ValidPersonOrTeamVocabulary,
<LPCIBot> move non-exact non-FTI matches back down into the FTI range. Also
<LPCIBot> match IRC nicknames case-insensitively.
<jml> g'night all
<Ursinha> night jml
<m4n1sh> Isn't a wait for 18 hours  a bit too much for PPA build
<m4n1sh> https://launchpad.net/~zeitgeist/+archive/ppa/+build/2582837
<LPCIBot> Project windmill-db-devel build #412: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/412/
<maxb> Undesirable, sure. The available resources just aren't there
<LPCIBot> Project devel build #823: STILL FAILING in 6 hr 5 min: https://lpci.wedontsleep.org/job/devel/823/
<m4n1sh> sad
<abentley> m4n1sh: It appears we lost the borrowed builders on June 17 and the queue's been a little nuts since then.
<m4n1sh> abentley: borrowed builders?
<abentley> m4n1sh: A bunch of our builders are actually borrowed from another department.  Hardware enablement?  Can't recall.
<m4n1sh> hmm
<m4n1sh> never mind, can wait
<abentley> sinzui: So much stuff depends on libgtk2.0-0...
<sinzui> abentley, yeah
<sinzui> abentley, I really suspect 64bit is the issue. Maybe deryck can confirm he is using 64bit
<deryck> sinzui, I am
 * deryck thought he confirmed that this morning
<sinzui> deryck, sorry, I was being an idiot
<sinzui> Now I need to get a 634bit install
<deryck> sinzui, no, no worries.  Just couldn't completely recall myself. :)
<flacoste> m4n1sh, abentley: yes, about half of the builders are taken for hardware enablement at critical points in the Ubuntu release cycle, it usually lasts a few days (though sometimes they forget to put them back in the main pool)
<timrc> jcsackett, okay, merged trunk 39 minutes ago... (sorry I've been pulled 50 different directions)
<timrc> jcsackett, https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-4/+merge/64840
<jcsackett> timrc: i saw when you did, i've sent it out.
<timrc> jcsackett, great, thanks :) I will begin praying now
<jcsackett> sinzui: have some time to chat?
<sinzui> jcsackett, you request is timely. yes
<bac> flacoste: i'm trying to do a collection assignment in the API/lplib but am getting NoBoundRepresentationError.  am trying archive.enabled_restricted_families = [arm] -- any hints?
<bac> where arm is a proper IProcessorFamily object
<flacoste> bac: we don't support that
<flacoste> bac: that's why i suggested exporting the mutator as an operation
<flacoste> setEnabledRestrictedFamilies(families)
<bac> flacoste: i must've missed that suggestion
<flacoste> or add/removeEnabledRestrictedFamily()
<flacoste> i probably wasn't clear
<flacoste> so get -> through the collection
<bac> flacoste: or it flew over with all the other stuff we discussed
<flacoste> set -> through operations
<flacoste> one or many
<flacoste> add/Remove/set
<flacoste> i don't have a strong opinion
<flacoste> set is probably fine
<bac> flacoste: thanks!
<flacoste> since they can get the collection, turn that into an array
<flacoste> list
<flacoste> and then call set with the changes
<lifeless> moin
<jcsackett> sinzui: have a moment?
<lifeless> pop quiz
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 205 - 0:[######=_]:256
<lifeless> whats a great micro-framework for python webapps, present (and usable) in lucid ?
<lifeless> gary_poster: hi; are there docs on setting up a new buildout-as-we-use-it for a new project?
<cody-somerville> lifeless, django? :P
<lifeless> cody-somerville: *micro*
<cody-somerville> lifeless, python-flask advertises its self as a micro framework but isn't available in lucid. However, in addition to python-django I did find the following in lucid: python-quixote, python-pylons, python-cherrypy3, python-turbogears2, and python-webpy
<LPCIBot> Project parallel-test build #57: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/57/
<lifeless> cody-somerville: yeah, none of those are microframeworks
<lifeless> except webpy kindof
<lifeless> I think I'll give web.py a shot
<lifeless> but first, lxc
<LPCIBot> Project windmill-devel build #260: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/260/
<lifeless> *hate* sample data.
<lifeless> corrupt conjoined master row in the sample data.
<wallyworld_> which table?
<lifeless> bugtask
<lifeless>  bug 3
<_mup_> Bug #3: Custom information for each translation team <feature> <lp-translations> <Launchpad itself:Fix Released> <Ubuntu:Invalid> <mono (Ubuntu):Invalid> < https://launchpad.net/bugs/3 >
<wallyworld_> very hard to get rid of sample data given how many tests use it
<lifeless> shows 'status tracked in sarge'
<lifeless> which means that the sarge row is a conjoined master
<lifeless> but the sarge row is assigned to a milestone, and the master row is not.
<wallyworld_> i assume that would never be allowed to happen in practice since our business logic would enforce the required ata integrity rules
<lifeless> this makes bugsummary fail the distro portlet test because the sample data isn't consistent - the old code /may/ have queried series tasks
<lifeless> I'm looking into the old code now :)
<lifeless> right
<lifeless> the prior code was counting milestones on either distro or distroseries
#launchpad-dev 2011-06-22
<lifeless> ah, so the problem is that milestones are not unique
<sinzui> StevenK, wgrant, mumble?
<lifeless> sinzui: hi
<lifeless> sinzui: milestones
<lifeless> sinzui: are they meant to be per-series ?
<sinzui> life no, their name space is per project, their parent is a series
<sinzui> lifeless, ^
<lifeless> so its expected folk may have one bug, with two tasks on different series, and the same milestone  on both tasks?
<lifeless> or would that be an unusual use ?
<sinzui> It is possible. the milestone is the release version. Project release versions are uniqu
<lifeless> sinzui: so, when you're done with that call
<lifeless> I'd like to talk through this with you
<lifeless> to help me solve a quandry
<sinzui> lifeless, I will be available to talk in ~30 minutes
<lifeless> excellent
<lifeless> I will fiddle with lxc containers until you ping me
<LPCIBot> Project windmill-devel build #261: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/261/
<wgrant> sinzui: http://paste.ubuntu.com/630585/
<LPCIBot> Project db-devel build #653: STILL FAILING in 6 hr 9 min: https://lpci.wedontsleep.org/job/db-devel/653/
<wgrant> sinzui: http://paste.ubuntu.com/630589/
<sinzui> wgrant, thanks
<LPCIBot> Project windmill-db-devel build #413: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/413/
<wallyworld_> sinzui: i get this when i try the yui tests - https://pastebin.canonical.com/48843/
<wgrant> wallyworld_: Upgrade.
<lifeless> sinzui: ping me whenever you are ready
<timrc> jcsackett, woo
<sinzui> lifeless, okay
<lifeless> sinzui: is skype working for you atm ?
<sinzui> lifeless, I suspect so
<sinzui> I have sip too
<lifeless> can we use skype?
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugs: 205 - 0:[######=_]:256
<lifeless> sinzui: I'd prefer skype unless you have a different preference - its echo cancellation is very good
<timrc> lifeless, this is not OEM-blessed work, but would it be valuable to remove buildd_secret from the web view? I'd be willing to look into it
<lifeless> timrc: I think its only visible to admins atm, and useful for debugging
<lifeless> timrc: IMBW
<timrc> lifeless, that's a good point
<lifeless> sinzui: if another time would be better, thats fine with me; I'm a little blocked till I resolve this however
<sinzui> lifeless, We love to talk
<lifeless> sinzui: I cxan't see you on skype
<sinzui> lifeless, coming online  now.
<wallyworld_> wgrant: sorry, i exited prematurely. you were still talking
<wgrant> wallyworld_: Just complaining that libwebkitgtk-3.0-0-dbgsym is 250MB.
<wallyworld_> that's quite large for sure
<cody-somerville> timrc, furthermore, its the only mechanism to change the buildd_secret if it were to get leaked without having to use SQL.
<cody-somerville> timrc, but more importantly, congratz on getting your first change landed! :)
<LPCIBot> Project devel build #824: STILL FAILING in 6 hr 32 min: https://lpci.wedontsleep.org/job/devel/824/
<lifeless> wgrant: I'm available to talk whenever you want about ubuntu disclosure / security bugs [sinzui says you know what this is about but he'll be email you regardless]
<wgrant> lifeless: I shall await the email.
<wgrant> sinzui: I have a somewhat symbolised backtrace now.
<wgrant> sinzui: http://paste.ubuntu.com/630607/
<lifeless> hah
<lifeless> sinzui: http://pastebin.com/85p07rfv makes my test pass; there is no sample data with series milestones outside of debian. ok with this?
<sinzui> lifeless, I am
<sinzui> wgrant, thank you very much
<lifeless> woohoo, its lp-land time
<wgrant> lifeless: It seems we have a regression.
<wgrant> lifeless: Probably in flacoste's branch.
<wgrant> 00134-09052@SQL-launchpad-main-master SELECT SourcePackagePublishingHistory.ancestor, SourcePackagePublishingHistory.archive, SourcePackagePublishingHistory.component, SourcePackagePublishingHistory.datecreated, SourcePackagePublishingHistory.datemadepending, SourcePackagePublishingHistory.datepublished, SourcePackagePublishingHistory.dateremoved, SourcePackagePublishingHistory.datesuperseded, SourcePackagePublishingHistory.distroseries, ...
<wgrant> ... SourcePackagePublishingHistory.id, SourcePackagePublishingHistory.pocket, SourcePackagePublishingHistory.removal_comment, SourcePackagePublishingHistory.removed_by, SourcePackagePublishingHistory.scheduleddeletiondate, SourcePackagePublishingHistory.section, SourcePackagePublishingHistory.sourcepackagerelease, SourcePackagePublishingHistory.status, SourcePackagePublishingHistory.supersededby FROM Archive, ...
<wgrant> ... SourcePackagePublishingHistory, SourcePackageRelease WHERE SourcePackagePublishingHistory.archive = Archive.id AND Archive.distribution = 1 AND Archive.purpose IN (1, 4, 7) AND SourcePackagePublishingHistory.sourcepackagerelease = SourcePackageRelease.id AND SourcePackageRelease.sourcepackagename = 31856 AND SourcePackagePublishingHistory.status IN (2, 1) ORDER BY SourcePackagePublishingHistory.id DESC LIMIT 1
<lifeless> wgrant: doing what?
<lifeless> wgrant: it looks like it could be bug 721643
<_mup_> Bug #721643: BugTask:+editstatus-page timeout changing source package <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/721643 >
<wgrant> lifeless: It was editstatus, yeah.
<lifeless> wgrant: changing the source package?
<wgrant> lifeless: Indeed.
<wgrant> But the query is different.
<wgrant> I wonder if the archive join is killing it.
<wgrant> Previously it named the IDs explicitly.
<lifeless> 15K archives
<lifeless> plenty of room for wtfs
<wgrant> Could you perhaps explainalyze it on staging?
<lifeless> pastebin the q somewhere for me ?
<lifeless> or gimve me the oops id ?
<wgrant> http://paste.ubuntu.com/630614/
<lifeless> wgrant: its going to be -terrible- cold
<lifeless> wgrant: its been running for 4 minutes now
<wgrant> lifeless: What if you drop the archive join and replace it with archive IN (1, 534)?
<lifeless> http://paste.ubuntu.com/630616/
<lifeless> wgrant: 1.8 seconds hot
<lifeless> without changing the q
<wgrant> lifeless: That's still pretty slow.
<wgrant> lifeless: What if you do change it?
<lifeless> its not brillirant
<lifeless> 74ms
<lifeless> http://paste.ubuntu.com/630617/
<StevenK> 74ms isn't brilliant?
<lifeless> 1800ms isn't brilliant
<sinzui> wgrant, lifeless would the editstatus timeouts be less of an issue if we completed this bug: https://bugs.launchpad.net/launchpad/+bug/399756
<_mup_> Bug #399756: Remove the inline bugtask edit form <javascript> <lp-bugs> <tech-debt> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/399756 >
<wgrant> sinzui: Only marginally.
<wgrant> sinzui: This query is enough to make a single AJAX request timeout anyway.
<lifeless> sinzui: I don't think they are related
<lifeless> sinzui: the inline form doesn't do that much more work
<lifeless> wgrant: rollback time ?
<lifeless> wgrant: or hotfix?
<lifeless> wgrant: (you've looked at flacostes patch already I presume; I haven't)
<wgrant> lifeless: It's >1000 lines, IIRC.
<wgrant> So... Hm
<wgrant> Possibly it's even the 2300 line branch.
<wgrant> Yay
 * wgrant goes digging.
<wgrant> Ah, mostly lint fixes.
<wgrant> But so many :(
<wgrant> lifeless: So, hotfix is tiny, if we want to.
<wgrant> I'm not sure if we really want to roll back.
<wgrant> Given the fun.
<lifeless> no, its been traumatic
<lifeless> lets hotfix
<wgrant> k
<wgrant> Preparing branch.
<lifeless> preparing incident report
<wgrant> Thanks.
<lifeless> bug # ?
<wgrant> None as yet.
<wgrant> Will file if you want.
<wgrant> But you seem to be handling the paperwork fine :P
<lifeless> thanks :(
<lifeless> and today I was finally hoping to do microservices.
<wgrant> Well, what did you expect after 6 days :(
<lifeless> epic fail?
<wgrant> Yes.
<lifeless> what rev broke stuff?
<lifeless> and whats the OOPS id you have?
<wgrant> 13263, OOPS-1999DR15
<wgrant> Probably not synced yet.
<wgrant> But carob:~wgrant/logs/soybean/2011-06-22/05353.DR15, if you want it.
<lifeless> nah
<lifeless> just for the paperwork
<lifeless> bug 800485
<_mup_> Bug #800485: timeout changing sourcepackage names in bugs <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/800485 >
<wgrant> Thanks.
<wgrant> Tests pass with the fix, unsurprisingly.
<wgrant> http://paste.ubuntu.com/630623/
<lifeless> diff ?
<lifeless> thanks
<lifeless> doit
<lifeless> this may help that other timeout? or is it undoing a spurious change flacoste made?
<wgrant> The latter.
<wgrant> He Stormified the query, and changed it to use a nice join instead of nasty all_distro_archive_ids.
<wgrant> But again, it has caused a performance issue :(
<wgrant> Probably because SPPH is so skewed.
<lifeless> ah
<wgrant> sinzui: And what about apport bugs? :)
<sinzui> wgrant, I do not know. Some or public some are private. I do not know its rules,, but I do not need to know why some of my bugs are private if we make apport behave like the current web UI
<StevenK> The bugs are created private since the backtrace may contain private information.
<sinzui> I wish apport was smart enough to know I do not have personal information in the bug an I really do want someone to see it
<wgrant> The bugs start with only the robot subscribed. It processes the coredump into a retracted backtrace.
<wgrant> It then makes a decision as to whether the backtrace should be kept private.
<wgrant> If so, it subscribes the crash bug triagers.
<sinzui> So apports  visibility rules are not directly affected since they are not flagged as secuirty
<wgrant> if not, it makes the bug public.
<wgrant> sinzui: Right, but this would give Ubuntu owners direct access to sensitive passwords.
<wgrant> So it is a change of behaviour.
<sinzui> all six people?
<LPCIBot> Project parallel-test build #58: STILL FAILING in 1 hr 1 min: https://lpci.wedontsleep.org/job/parallel-test/58/
<sinzui> wgrant, how can they not see them now? because they are not bug supervisors? They can change that at any moment
<wgrant> sinzui: Hm?
<wgrant> sinzui: Apport reports bugs with only the robot and the reporter subscribed.
<wgrant> No bug supervisor is involved.
<sinzui> ah, sorry. I misunderstood that.
<sinzui> This somewhat relates to the IRobot discussion. I think the case here is that bug report is not finalised so is not in a full state.
<wgrant> It's more similar to the security case.
<wgrant> Which suggests that the security case is not really that special.
<wgrant> Which suggests that it should not be much of a special case.
<cody-somerville> "Launchpad requires that the project maintain(er) be a member of bug supervisor team." <-- I don't believe this is currently true. Are you proposing it become true?
<sinzui> I think privacy mean personal data, which is the cited reason for hiding the bugs.
<sinzui> cody-somerville, it is true, but I know how to break the rule. We have lots of orphaned bugs because the maintain changed *after* the bug supervisor was set
<wgrant> It is not true.
<wgrant> The fields are unrelated.
<wgrant> Except that the maintainer can change the bug supervisor.
<sinzui> I can set the supervisor to myself or a team I am in. I cannot set it to another team
<lifeless> back
<sinzui> Nope still cannot make ~zeitgeist mange the bugs in py projects
<lifeless> sinzui: or a team you own ?
<sinzui> lifeless, I must be a team admin
<lifeless> sinzui: ownership grants Launchpad.Admin
<sinzui> Yes, that also mean I can make myself a member when ever I chose. I did that two weeks ago and bac could not stop me
<lifeless> sinzui: yup, thats what we agreed on wasn't it ?
<sinzui> yes it was.
<lifeless> I do wonder, perhaps project owners should be able to set any team as supervisor
<sinzui> I have all the access I want
<lifeless> but thats a different discussion
<lifeless> testblahblah might be a junk project :(
<sinzui> I think users need to say they need to do that. Ubuntu and canonical projects use bugs and security differently than everyone else
<LPCIBot> Project windmill-db-devel build #414: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/windmill-db-devel/414/
<lifeless> sinzui: I have replied
<lifeless> sinzui: I have some bits I don't understand yet
<lifeless> sinzui: and perhaps some confusion
<lifeless> sinzui: I think apport is quite easily fixable in the very short term, with a crash db being the long term fix.
<wgrant> People don't read :(
<wgrant> /projects/+new says "You do not need to register a project to: [...] Activate a PPA"
<jtv> StevenK, wgrant: mind if I restart the appserver on dogfood?
<wgrant> jtv: Please do.
<jtv> thanks
<wgrant> lifeless: The primary archive does use a-f.
<lifeless> wgrant: the bug is about ppas
<wgrant> lifeless: But the slowness that Julian speaks of is a-f.
<lifeless> yes
<lifeless> I read the cache comment as 'can you not jus tuse a-ft with a cache db to get ppa contents.gz working
<lifeless> '
<lifeless> to which the answer is no, we don't use a-f for ppa contents.gz
<lifeless> wgrant: do I have something wrong in that logic?
<LPCIBot> Project windmill-devel build #262: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/262/
<StevenK> lifeless: Right, we do not use a-f for PPAs
<StevenK> lifeless: Who is asking about Contents for PPAs?
<lifeless> there was a comment in the bug about it
<StevenK> https://bugs.launchpad.net/launchpad/+bug/796997 is tracking my work on it, but it *will* be slow going.
<_mup_> Bug #796997: Contents generation should be done from the database <qa-untestable> <soyuz-publish> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/796997 >
<lifeless> StevenK: yeah, its fine
<lifeless> he was suggesting using a-f with the cache switch, which is just irrelevant for contents.gz w/ ppas
<StevenK> I doubt I can populate BRPC for a bit. 6.6 million BPRs with unexpired .debs
<lifeless> yeah
<lifeless> its pending the disk space increase as much as anything
<StevenK> Right
<StevenK> I figure I can take it slow, wait for db-stable to get merged into devel, and work from that.
<lifeless> sure
<lifeless> though if you are not modifying existing tables, you can add the new ones live
<lifeless> no downtime
 * StevenK shrugs
<StevenK> My brain is hardwired for "any changes to the db == db-devel"
<lifeless> hopefully that will be history in a few weeks.
<StevenK> Excellent. Then I can progres to "any change to the db == " *SEGV*
<StevenK> I have a play branch, I'll be hitting up mawson probably at the sprint
<wgrant> Hahaha
<StevenK> Hmm?
<wgrant> http://people.canonical.com/~wgrant/launchpad/big-plans.png
<wgrant> Should I be scared?
<StevenK> Bwahaa
<wgrant> Grar.
<wgrant> RT
<wgrant> DIE
<wgrant> I can't Cc myself to stuff :(
<wgrant> So, let's see this query plan.
<wgrant> It is obscene.
<wgrant> Perhaps we just want to bite the bullet and create a summary table.
<lifeless> 404
<lifeless> wgrant: what situation ?
<wgrant> lifeless: Efficient querying of binary and source publication presence and overrides.
<wgrant> lifeless: By name.
<wgrant> At the moment we have to join through BPR to BPPH.
<lifeless> yes
<wgrant> Since there's no name on BPPH.
<lifeless> Can denorm that onto it
<wgrant> Probably.
<lifeless> if that will fix the queries, its a perfectly cromulent solution
<wgrant> Guess it wouldn't be that big.
<wgrant> Let's see what happens...
<lifeless> wgrant: what was the png ?
<wgrant> lifeless: A half-fullscreen terminal.
<StevenK> Amusing, is what it was.
<wgrant> The top half has the word QUERY in it.
<wgrant> The bottom half was the seperator at the top of the table.
<lifeless> heh
<wgrant> None of the actual plan fit on the screen.
<wgrant> Because it is so hideous.
<wgrant> HIDEOUS
<wgrant> Let's see if mawson melts...
<LPCIBot> Project db-devel build #654: STILL FAILING in 5 hr 58 min: https://lpci.wedontsleep.org/job/db-devel/654/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #825: FIXED in 5 hr 37 min: https://lpci.wedontsleep.org/job/devel/825/
<wgrant> Hm.
<wgrant> So those two persistent codeimport failures are gone?
<StevenK> I think they've moved from persistent to spurious
<StevenK> Wait, wrong word :-/
<wgrant> s/spurious/intermittent/?
<StevenK> Right
<jtv> StevenK, wgrant: either of you willing to offer a shoulder to cry on?  Got a bit of a soyuz question.
<wgrant> jtv: Fire away
<jtv> Thanks.  It's about how the +queue view batch-loads the various items associated with a batch of PackageUploads.
<wgrant> Oh dear.
<wgrant> "Slowly"
<wgrant> "Awkwardly"
<jtv> AFAICT it retrieves the PackageUploadBuilds in self.builds_dict.
<jtv> That starts with BinaryPackageReleases.
<nigelb> I guess working on LP eventually does eventually need a shoulder to cry on.
<jtv> From there, it collects the builds.
 * jtv looks up what those are
<wgrant> nigelb: Particularly Soyuz, yeah.
<wgrant> jtv: Odd, it shouldn't be able to start with BinaryPackageReleases.
<jtv> BinaryPackageBuilds.
<nigelb> wgrant: And here I thought Blueprints were bad.
<wgrant> jtv: Since it goes PU -> PUB -> BPB -> BPB
<wgrant> BPB -> BPR, sorry.
<jtv> Don't worry, I hadn't gotten that far yet.  :)  What's the correct line then?
<wgrant> ?
<jtv> PU â PUB â BPB â BPR?
<wgrant> Yes.
 * jtv decodes
<jtv> Ah yes
<jtv> In my case, I'm more interested in the SPR than the BPR but it's much the same story otherwise.
<wgrant> PUS â PUS â SPR
<wgrant> Grar
<wgrant> PU â PUS â SPR
<jtv> I thought the FKs go PU â BUB â BPB â SPR
<StevenK> One of your arrows is backward
<wgrant> No, it's right.
<jtv> Don't worry about the PUS; I think those aren't related to my current problem.
<wgrant> PU â PUB â BPB â BPR
<jtv> I'm interested in the chain from PUB to SPR.
<wgrant> Why?
<wgrant> That's not normally something you want to do.
 * jtv images the rest of the LP team looking on in bafflement
<jtv> In this particular case, I need to find the PS.
<jtv> Packageset, sorry.
<wgrant> Normally it'd be PU â PUS â SPR â SPR
<wgrant> Ah.
<StevenK> SPR -> SPR? WTF?
<jtv> StevenK: ISWYM
<wgrant> StevenK: SourcePackageRelease â SourcePackageRecipe
<wgrant> Except I am wrong.
<jtv> OIC â WTF indeed
<wgrant> PU â PUS â SPR â SPRB â SPR
<StevenK> Right
<wgrant> jtv: So, for a source upload you wouldn't go near PUB.
<wgrant> Why are you trying to?
<jtv> I'm not saying it has to be a source upload.
<StevenK> AssertionError: [] is not []
<jtv> Why would there even be a PUB for a source upload?
<StevenK> Sorry?!
<wgrant> jtv: There wouldn't be. But I'm not sure we care about Packagesets for binary uploads now.
<jtv> StevenK: you're creating empty list objects with different identities.
<wgrant> jtv: If you want to, it's PU â PUB â BPB â SPR
<StevenK> Oh, right
<StevenK> Sigh
<StevenK> Stupid Python
<jtv> wgrant: trueâmaybe I should go back to what my real problem is.
<jtv> Which is that I'm missing the packageset "core" for a PU for netapplet 0.99.6-1 in the sample data.
<jtv> (Fun with doctests)
<wgrant> jtv: What have you changed?
<wgrant> Is it a mixed upload?
<jtv> I'm not sure.  It's listed as (source) but that may be sort of a straw to clutch at.
<jtv> The packagesets do work for alsa-utils, which also says it's (source).
<jtv> And the packgesets look pretty simple to me, when it comes to source uploads.
<jtv> Ahhh, maybe they're not actually.
<jtv> Because again, the view doesn't go straight through the PUS to get the SPR.  It goes through SPRF.
<wgrant> Hm? SPRF is at the end.
<wgrant> After SPR
<jtv> Should be, yes.
<jtv> That's what you get when documentation simply says things like "a list of SPRs" without being precise about which ones they are.
<wgrant> Hm?
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
<jtv> wgrant: I can see now that the view probably needs a bit of tightening up of these access routes.  It should bulk-load classes in a more sensible step-by-step manner, following the primary access paths.
<wgrant> jtv: Indeed. It was one of the earlier places to do preloading.
<jtv> I think I walked into a trap because I wasn't familiar with the "natural" access paths.  That's going to be a growing risk, or source of friction, with squads.
<wgrant> Mm.
<wgrant> Everyone will be familiar with the model eventually.
<jtv> But how long will it take and how hard will it be?
<wgrant> A while and not terribly.
<wgrant> I even know translations somewhat now.
<jtv> I suspect the rest of LP is relatively easy when you started with soyuz!
<wgrant> Fair point.
<jtv> Also, I'm pretty sure that a lot of the Soyuz knowledge will suffer from degrading memory _and_ code changes after a few rotations on other stuff.
<wgrant> I don't know about others, but it's pretty irrevocably scarred into my mind.
<jtv> Good, that's one innocence ruined; who's next?
<jtv> I think an investment in clarity here would pay off handsomely, both in terms of performance and in terms of developer effectiveness.
<jtv> \o/ got that test passing!
<jtv> Wonder how many others I broke.
<wgrant> What criminal act have you performed on the view?
<jtv> Bulk-loaded PUS, SPR, Packageset.
<jtv> Also, we have putâ¦  a bag over its head.
<wgrant> Ah, excellent.
<jtv> (Because the Blackadder angle was just too good to resist)
<wgrant> I had a branch that trialled that long, long ago. Before it was accepted that we could bulk-load collections.
<wgrant> Because it issues so many queries :(
<jtv> Quite.
<jtv> I suspect some of the complexity would evaporate from that view if we would just take the bulk-loading one step at a time.
<wgrant> It needs to be ripped out and replaced.
<jtv> Right now there's a fair amount of "I've got objects at step 1 in the reference chain; get me the ones at step 3."
<wgrant> The bulk-loading is rather messy.
<wgrant> Needs to be redone with helpers.
<jtv> Well yes, ripping out would feel good too.
<wgrant> And sense.
<jtv> One thing that bothers me about the whole thing is that we _could_ fetch the PUS/PUB/PUC and so on in Ajax as you fold open one queue item in the UI, _but_ we still need to know that they're there so we can decorate each with the right icons.
<jtv> Which reminds meâ¦
<wgrant> Yes.
<wgrant> They're very cheap to get, though.
<wgrant> Just have to remove the scaling.
<jtv> â¦do we have any icon that we might use to represent a package sync?
<wgrant> We do not.
<wgrant> I think Julian was going to talk to Huw about that.
<wgrant> Or something.
<wgrant> but it's been a while.
<wgrant> You see we're sort of scratching for icons already.
<wgrant> Which is why we use the Ubuntu logo for translations...
<wgrant> Maybe not translations.
<wgrant> Maybe it was dist-upgraders and debian-installerse.
<jtv> (The page could also do with some sprites)
<wgrant> But something that shouldn't have the Ubuntu logo, anyway.
<jtv> Yes, saw some of that.
<jtv> Actually it's quite easy to see in my new code:
<jtv> I've got a method that says "here's a list of icons you may need; for each you get condition, alt, title, image name.  Now spit out HTML for the ones that apply."
<wgrant> Excellent.
<jtv> I'm hoping that a future step could also notice that "hang on, I have a sprite for this somewhere."
<jtv> Even further into the future though, maybe we could just render PUs though and have the icons filled in asynchronously.
<jtv> wgrant: you wouldn't be interested in reviewing my view change by any chance?  It's about a thousand lines, but it might give you some satisfaction.
<wgrant> jtv: It would be quite satisfying to review cleanup of that view.
<jtv> It's not a full cleanup by any stretch of the imagination, but I hope a step in the right direction.  I'm hoping that CompletePackageUpload will evolve into a PackageUploadView.
<jtv> wgrant: might it make sense to use copy.png for a PCJ upload?
<wgrant> jtv: Not sure.
<wgrant> It probably makes more sense in the translations UI.
<jtv> wgrant: I guess copy.png is for the action, rather than as a description.  "Sync" icons are a dime a dozen though; I suppose the world has a fairly well-defined idea what they look like.
<lifeless> yeah
<lifeless> so we should do something different P
<jtv> In this case, I doubt that would help.
<danilos> hum, anyone knows if we have trouble accessing SF SVN branches? (re question https://answers.launchpad.net/launchpad/+question/162211, "svn ls" works for me)
<jtv> danilos: they used to block us, though I thought that was resolved.
<jtv> And hi.  :)
<jtv> wgrant: would you be willing to review my branch?
<danilos> jtv, hi :)
<jtv> which reminds me
<jtv> current topic is: https://dev.launchpad.net/ | On call reviewer: StevenK,jtv | Critical bugs: 205 - 0:[######=_]:256
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK,jtv | Critical bugs: 205 - 0:[######=_]:256
* lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK,jtv | Critical bugs: 211 - 0:[######=_]:256
<wgrant> :(
<danilos> jtv, which reminds me...
<jtv> lots of embarrassed coughing all around âº
* danilos changed the topic of #launchpad-dev to: lifeless Ð¿ÑÐ¾Ð¼ÐµÐ½Ð¸ ÑÐµÐ¼Ñ Ñ https://dev.launchpad.net/ | On call reviewer: StevenK, jtv, danilo | Critical bugs: 211 - 0:[######=_]:256
<jtv> promeny temy?
<danilos> haha
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv, danilo | Critical bugs: 211 - 0:[######=_]:256
<jtv> ah, same word as "theme," right?
<jtv> Greek "thema"
<danilos> jtv, yeah, "promeni temu" :)
<jtv> subject
<lifeless> I changed the topic? yes I did
<danilos> lifeless, I didn't have to put that fact in the topic, though :)
<jtv> regardless of language :)
<lifeless> indeed :)
<jtv> "Extra!  Extra!  lifeless has changed the topic!  Read all about it."
<danilos> jtv, btw, other than the +languages/xx page, do you think dropping tm__language__submitter__idx could harm anything else? (if you can remember anything off the top of your big, bald head)
<jtv> danilos: err.. POFile filter?
<jtv> Which I know you've been looking atâ¦
<danilos> jtv, it actually seems to help that :)
<jtv> wow
<jtv> That may mean that the index actually needed more columns.
<danilos> jtv, not surprising, considering the index is for a (language, submitter), and for those active translators, it's going to be more
<danilos> jtv, yeah, I am trying to add potmsgset in
<lifeless> allenap: if you used a temp table, did you analyze it first ?
<danilos> jtv, unfortunately, if I just add another index, postgres still uses tm__language__submitter__idx even after analyze
<jtv> danilos: I believe you've been looking at putting TM in a different type of DBâ¦  some less radical things that I think might help include: using arrays for msgstr1â¦msgstr5 and moving the is_current_u* flags out of the table.
<lifeless> danilos: whats the new index you added ?
<jtv> danilos: maybe the fields are in an inconvenient order?
<jtv> lifeless: any chance of us getting hypothetical indexes?
<lifeless> danilos: (perhaps the index is being selected due to sort order; making sure the columns match the sort order both left->right and asc/desc)
<lifeless> jtv: is it in 9.0 ? :)
<jtv> It's a 3rd-party patch.
<jtv> I think on 8.something & up.
<jtv> http://pqxx.org/development/libpqxx/wiki/HypotheticalIndexes
<lifeless> jtv: http://sourceforge.net/projects/hypotheticalind/ ?
<jtv> I think that's the one, yes
<jtv> I got them to put up some documentation a while ago.  :-)
<danilos> lifeless, hum, interesting, never tried that out
<lifeless> ah, I see
<jtv> postgres does know reverse index scans though.
<danilos> jtv, let me read through that as well :) I love all things hypothetical
<lifeless> danilos: when pg can satisfy the requested sort from index, it will take that index even if it makes things worse (AFAICT) - perhaps our query cost estimators are whack.
<jtv> lifeless: that would indicate a problem with cost estimation, yes.
<lifeless> danilos: so if you have a better index and its competing on a sort-matching index, you need to make the better one also sort-match.
<jtv> We may even have tried too hard to _make_ it use indexes.
<danilos> lifeless, in this particular case, it's sorting by things outside this index, so doesn't seem related
<lifeless> ok
<danilos> jtv, lifeless: I get roughly an order of magnitude improvement just by dropping the index for the query from bug 534203
<_mup_> Bug #534203: Timeouts on POFile:+filter (filter by person) <lp-translations> <timeout> <Launchpad itself:Triaged by danilo> < https://launchpad.net/bugs/534203 >
<jtv> danilos: it may be interesting to learn what index the optimizer uses instead on the faster query, and comparing that to the one you dropped.
<lifeless> danilos: yeah, I saw the comments.
<lifeless> danilos: I'm fine with dropping the index FWIW, if we fairly sure nothing else is thrown out.
<danilos> jtv, well, I did look into it, which is why I thought adding potmsgset in would help :)
<jtv> danilos: but you didn't enlighten us.  :)
<lifeless> [unless the index is needed for FK cascade deletes]
<jtv> I think we have separate indexes for each FK.
<danilos> jtv, heh, right, I did that in the bug report
<LPCIBot> Project parallel-test build #59: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/59/
<danilos> jtv, so, I am just creating tm__potmsgset__language__submitter__idx and dropping the tm__language_submitter__idx and I'll see how that works
<lifeless> jtv: hypotheticals : +1 for dev, staging, qastaging.
<lifeless> jtv: -0 for prod
<jtv> Of course.
<danilos> jtv, this hypothetical indexes stuff sounds very cool!
<jtv> Yes, it means you can just shower your tables with indexes just to see what the ideal ones would be.
<lifeless> o/~ its raining indices
<jtv> ?
<lifeless> jtv: its raining men, shower, - I have a weird associational memory
<jtv> But what is "o/~"?
<jtv> a horizontal shower?
<lifeless> oh, a note symbol from sheet music
<lifeless> e.q. quaver
<jtv> argh
<LPCIBot> Project windmill-db-devel build #415: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/415/
<jtv> I would've thought something like d^
<lifeless> indeed, I didn't invent it :)
<allenap> lifeless: I did. I tried a temp table and, iirc, a couple of variations on WITH. The temp table improved thing marginally, but there was still that 300000+ rows loop in the plan. The WITH clause reduced execution time from ~6s to <2s, but the UNION approach got it down to ~0.7s.
<lifeless> allenap: did you compare the plans ?
<lifeless> allenap: I want to know whats going wrong
 * jtv thought allenap claimed to invent the ascii quaver
<allenap> lifeless: I still have them. I'll organize them and attach them to the bug report.
<lifeless> allenap: its thinking that the selectivity is all whack
<lifeless> allenap: the union thing uhm, freaks me.
<allenap> lifeless: Yeah! It's a like a weird hack.
<lifeless> I can't look into it tonight, but I will happily have a fiddle around tomorrow
<lifeless> sinzui: bug 800544
<_mup_> Bug #800544: series milestone tasks not shown on project|distro scope bug portlets <bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/800544 >
<henninge> Hey danilos! ;)
<danilos> henninge, hey-hey, how's it going?
<henninge> danilos: good, I have the next two days off ... ;-)
<lifeless> henninge: nice
<danilos> henninge, yeah, sounds nice, so I better get hold of you for a chat today :)
<henninge> danilos: this morning, I'll already be travelling in the afternoon.
<jml> yo
<danilos> henninge, how about in 35 mins then? I just want to finish some performance testing first
<jml> 'sup dawgs?
<jml> (also, would someone kindly glance at <https://code.launchpad.net/~jml/launchpad/buildd-cleanups/+merge/65378>)?
<lifeless> stuff
<henninge> danilos: sounds good
<danilos> henninge, cool, talk to you then
<henninge> ok
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv, danilo | Critical bugs: 211 - 0:[######=_]:256
<allenap> lifeless: https://bugs.launchpad.net/launchpad/+bug/798301/+attachment/2177909/+files/plans.txt
<_mup_> Bug #798301: Time-out on +localpackagediffs <derivation> <timeout> <Launchpad itself:In Progress by allenap> < https://launchpad.net/bugs/798301 >
<danilos> jml, looks good, r=me
<jml> danilos: thank you.
<StevenK> bigjools: Are you feeling better?
<bigjools> StevenK: quite a lot better thanks
<jtv> huwshimi: I think we need a new icon, to mark package uploads in the UI that are "package synchronizations."
<huwshimi> jtv: OK. I'm not familiar with that part of LP. What's the difference?
<lifeless> huwshimi: have you heard of soyuz ?
<huwshimi> lifeless: Of course :)
<jtv> Hi.  This is soyuz, and only a small group of people really need to deal with it.  A package upload can be a source upload, or contain builds, or contain translation tarballs, and so on.
 * lifeless gets out before the dogpile is too large :P
<jtv> A new item they can have attached is a job to sync a package.
<huwshimi> jtv: And is that a transitory status?
<bigjools> wgrant: hello
<jtv> Only to the extent that the package upload object itself is.  This copy job will be its raison d'Ãªtre.
<jtv> So it's more a characterization of the object you're looking at.
<bigjools> jtv: I don' think you need to block on this BTW
<jtv> bigjools: I'm not, but might as well get it going.  :)
<bigjools> ok :)
<jtv> allenap: the performance problem isn't simply a matter of running out of work_mem?
<allenap> jtv: Absolutely no idea!
<allenap> jtv: Can I tweak that before running the query?
<jtv> allenap: you could, but it's just a tiny bit nasty to do that as a matter of course.
<lifeless> jtv: its selecting a 300K row scan
<jtv> But it's something you can try in-session while experimenting on any of the test dbs.
<jtv> lifeless: so it's a quantum leap to a different plan then, not a gradual increase in run time?
<huwshimi> jtv: Maybe you can provide a bug for this with a screenshot of what we currently have.
<jtv> (for some value of "gradual" of course)
<huwshimi> jtv: Is this part of a feature you're working on?
<jtv> huwshimi: we have nothing for it yet.  It's a new item that a package upload can have.
<huwshimi> jtv: Oh right
<jtv> Yes, this is the feature we're working on.
<lifeless> jtv: I can't drill into it tonight, but the plan when its slow looks poor
<jtv> lifeless: I gather that's it pretty much good enough for now, so maybe we should just go with what we have and worry more if it becomes a problem again?
<lifeless> jtv: (I have a 0530 am wake up tomorrow which is why fiddling tonight would be bad :)
<jtv> bird in the hand etc.
<jtv> I know how you feel â I'm on 05:00 at the moment.  Followed by a few more hours of sleep, thankfully.
<lifeless> jtv: the use of a union to work around a poor plan is a bit unnerving
<lifeless> jtv: allenap: I think identifying the cause is worthwhile; that or just the expedient thing of clamping the batch size (which is a 1-line fix)
<henninge> danilos: did you want to do voice?
<jtv> lifeless: I'll still need to get to that part, so no idea how unnerving it is yet.  Guess I'll look at that first.
<henninge> s/did/do/
<huwshimi> jtv: OK, so what icon do we currently use for package uploads that are not package syncs?
<jtv> huwshimi: each has a collection of icons depending on what types of items are attached.
<jtv> We need to add one more icon to that.
<danilos> henninge, yes please, though mumble is kind of dead for me, so it'd have to be skype
<jtv> We do suffer a paucity of icons there.
<allenap> lifeless: Clamping the batch size is going to displease the users, from what I've heard. If we can work around it one way or the other I think it's worth it.
<lifeless> allenap: what they want and what they need are totally different things
<danilos> henninge, if that doesn't work for you, irc is fine as well
<lifeless> allenap: seriously - they are talking about paginating through thousands of entries; thats a terrible UI
<jtv> allenap, lifeless: Sometimes the answer may just be to have separate pages for separate purposes, and optimize them to extremes for their specific purposes.
<huwshimi> jtv: Ok, I'm just want to look at something so I can see what to base this new icon off (or at least make it fit with what we have for the other types).
<jtv> huwshimi: try /ubuntu/oneiric/+queue
<jtv> (Try selecting a different status than New to see more)
<huwshimi> jtv: Ah thanks
<lifeless> allenap: bigjools initial reaction to the bug was to say '300 is url hacking, low pri' - which says to me that supporting large batches isn't important to the feature at this point.
<danilos> henninge, heh, call when ready
<bigjools> lifeless: more complicated than that
<allenap> lifeless: Yes, but then you put it back up to Critical!
<allenap> lifeless: But fair. I think we can leave this alone for now.
<huwshimi> jtv: OK, can you file a bug for it anyway, just so I have something in the system and explain what you want and I'll get onto it.
<jtv> huwshimi: certainly â hang on
<huwshimi> jtv: Thanks
<allenap> jtv: I've mostly heard rumblings that better search for those pages is what's needed. I think that's the way to ensure that batches stay small.
<jtv> huwshimi: bug 800573
<_mup_> Bug #800573: Need icon for sync package uploads <derivation> <Launchpad itself:Triaged by huwshimi> < https://launchpad.net/bugs/800573 >
<huwshimi> jtv: Thanks
<jtv> thanks for looking into it â I assigned it to you
<huwshimi> jtv: No problems
<wgrant> bigjools: Hi.
<bigjools> wgrant: hey, I just want to help rvba finish his branch
<bigjools> he's done the stuff you asked but I have one more question
<wgrant> bigjools: Oh?
<wgrant> (I have further concerns, but not sure how to resolve them)
<bigjools> if he blocks binaries with conflicting files, we'll end up with builds with incomplete binaries and wondered about the effects of that
<wgrant> Yes
<bigjools> and wondered if we should block the source for the binary too
<wgrant> We really need to copy the sources, I think, not the publications.
<wgrant> Or we're going to end up with a patchwork mess of series.
<wgrant> Which are not consistent.
<wgrant> Not legal.
<wgrant> Not recoverable.
<bigjools> yes it is confusing
<bigjools> so you think making a new spr and its files is better?
<wgrant> No, no.
<bigjools> what do you mean by sources then?
<wgrant> I think rephrasing initialisation as a copy of all of the sources is better.
<wgrant> Rather than a copy of all the source and binary publications.
<wgrant> But I don't know how that will work.
<bigjools> well yes I think copying binaries is crack anyway
<bigjools> but it's in the spec, so ...
<wgrant> I mean using the usual copy logic, and just syncing all sources with binaries.
<bigjools> using the packagecopier?
<wgrant> That's the behaviour we want, ideally.
<bigjools> I think we tried that originally
<bigjools> it was so slow as to be unusable
<jtv> allenap: 35?
<wgrant> Of course.
<bigjools> hence the lovely packagecloner
<wgrant> But it's the behaviour we want.
<bigjools> true
<wgrant> And the package copier used to suck.
<wgrant> It still sucks.
<wgrant> But it's not quite as bad.
<wgrant> I don't know if we can use it.
<rvba> The package copier will end up really slow if I continue to add new checks to it.
<wgrant> But the current approach of just excluding the specific bits that conflict is.... messy and problem-prone.
<bigjools> so I've said to rvba that if we block conflicting sources then he needs to block their binaries too
<rvba> and slow.
<wgrant> bigjools: Certainly.
<bigjools> apart from that I want to land this puppy
<wgrant> bigjools: And I think probably the other way too.
<bigjools> wgrant: yes, I mentioned that too
<wgrant> Which means we basically want the full consistency checks that the copier provides.
<wgrant> We need to optimise that for mass-syncs anyway.
<wgrant> Eventually.
<wgrant> And a large mass-sync could be easily 10% of the archive.
<wgrant> So...
<wgrant> We're approaching the same order-of-magnitude.
<bigjools> right
<wgrant> Given what it is, duplicating this logic does not sit well with me at all.
<bigjools> rvba: ok so add the "don't copy corresponding binary/source if you block a source/binary" change and then let's land it
<wgrant> But, similarly, using the package copier for initialisation is also a little bit scary.
<bigjools> wgrant: very
<bigjools> and no I hate this duplication
<rvba> bigjools: ok
<bigjools> it's only there as a special case to make intialisation quick
<wgrant> Yeah.
<wgrant> Eventually the copier will be just as fast :)
<bigjools> but the multi-parent stuff fecked that all up
<wgrant> Yup
 * bigjools is getting the irish lingo already
<lifeless> allenap: I put it to critical because we want to be able to jump on every oops that happens
<lifeless> allenap: when means no spurious oopses
<bigjools> lifeless: in this case it's indicative of a lack of good search options
<lifeless> allenap: if something is a users fault, it needs special reason to be an OOPS
<lifeless> bigjools: yeah
<wgrant> bigjools: Hmm.
<bigjools> wgrant: ?
<lifeless> bigjools: but still, we need to either (make it not timeout) | (make it not oops)
<wgrant> bigjools: I'm wondering if we can do a known-good initialisation (from a series in the same distro, or into a fresh distribution) the quick way, and then do the others afterwards using the copier to ensure consistency.
<bigjools> lifeless: well I think it's already fine with the default batch size, it only oopses because someone hacked that
<lifeless> bigjools: agreed
<bigjools> so it's a rather special case
<lifeless> bigjools: my recommended fix is to clamp the batch size
<wgrant> bigjools: This would keep the Ubuntu case fast.
<bigjools> lifeless: +1
<lifeless> bigjools: batchnav has a parameter for this
<lifeless> bigjools: which will make url hacking throw a non-oops error (UFD I think)
<bigjools> lifeless: +1
<lifeless> or if it does oops today, its one we can blanket ignore
<bigjools> -1 to ignoring oopses - need to stop them happening in the first place
<lifeless> bigjools: oh I agree
<bigjools> by whatever means
<bigjools> wgrant: so
<bigjools> wgrant: interesting
<lifeless> bigjools: by blanket ignore I was meaning ignore the exception in the oops raising() method
<wgrant> bigjools: Since in most cases the big copy will be known-good.
<wgrant> bigjools: natty->oneiric is within an archive, so it's fine.
<wgrant> natty->newderivativedistro is fine, because it's a new archive.
<bigjools> wgrant: yes, indeed.  and I think in retrospect this is what should have been done
<bigjools> I have a first class degree in hindsight from the university of life
<jml> huwshimi: btw, I'm just starting to work through my hand-off tasks. if you'd like to talk, now would be a good time.
<wgrant> Retrospect is handy; keeping domain experts together can be too :)
<bigjools> wgrant: yes :/
<huwshimi> jml: Do you mean about the designs?
<jml> huwshimi: about whatever :)
<huwshimi> jml: Sure, we can talk about whatever, but at least with the designs I'll need to wait until I've had a bit more of a chance to look at them to day... maybe this afternoon for that
<jtv> allenap: 35..?
<jml> huwshimi: ok. let's chat then.
<huwshimi> jml: Sure
<jml> ambiguity resolved!
<huwshimi> jml: ambiguity is never resolved
<jml> it expires?
<huwshimi> jml: Could be
<allenap> jtv: I plucked that one out of the air. It means that there will be two subselects in the UNION for the default batch size of 75.
<lifeless> allenap: fwiw see bigjools +1ing clamping the batch size rather than doing the union *terrifying* patch
<lifeless> and with that, I'm running for sleep
<bigjools> a combo of both?
<allenap> lifeless: Okay, cheerio, and thanks.
<wgrant> bigjools: We are considering denormalising BPR.binarypackagename onto BPPH.
<bigjools> BPPH not BPR?
<wgrant> Yes. We need to be able to index BPPH on either BPR.binarypackagename or BPN.name.
<wgrant> I guess we could just go all the way and stick BPN.name into BPPH
<lifeless> wgrant: that is what I was suggesting
<wgrant> (overrides during copies are too slow without that :/)
<wgrant> lifeless: Well, BPN.name is quite a bit wider than BPN.id.
<lifeless> wgrant: lets do some numbers, but not now.
<lifeless> zzz
<wgrant> Night.
* jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilo | Critical bugs: 211 - 0:[######=_]:256
<LPCIBot> Project db-devel build #655: STILL FAILING in 5 hr 36 min: https://lpci.wedontsleep.org/job/db-devel/655/
<jtv> say danilos, perhaps you can review my thousand-line branch?  https://code.launchpad.net/~jtv/launchpad/bug-798521/+merge/65297
<danilos> jtv, I was considering it, but just couldn't get around to doing it :) any reason why it's not two smaller branches?
 * jtv stops himself from mentioning a disgusting joke
<jtv> It was all sort of stuck together.
<jtv> There are a few parts I could isolate, in retrospect, but this is soyuz: if I'd done it up front I would have had to fix the first branch while doing the second.  :/
<jtv> Things working as specified but not as desired, that sort of thing.
<danilos> jtv, yeah, you should try pipelines sometime if you haven't already :)
<jtv> I have.  Gave me enough trouble that I no longer bother.
<danilos> jtv, anyway, looking through your branch now
<jtv> Thanks!
 * henninge is outta here
<jml> huwshimi: let me know when you want to head out for lunch. (note that the standup is in ~1hr 15m)
<huwshimi> jml: We can go now. I probably just need to stop work on this now anyway
<jml> huwshimi: OK. Let's do that.
<LPCIBot> Project parallel-test build #60: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/60/
<LPCIBot> Project windmill-db-devel build #416: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-db-devel/416/
<jtv> thanks danilos!
<jtv> Or hvala
<danilos> jtv, it wasn't me!
<jtv> suuuurre
<deryck> Morning, all.
<deryck> abentley, hey, I'm around now.
<deryck> abentley, adeuring -- https://wiki.canonical.com/Launchpad/Sprints/Thunderdome2011/Agenda
<huwshimi> jml: I'm available if you want to talk.
<jml> huwshimi: oh, right. good idea. I'll just finish off this email.
<huwshimi> jml: No problems
<huwshimi> deryck: Morning
<huwshimi> deryck: I noticed this bug which is only a year old: #561586. Does this change anything about what we talked about yesterday?
<_mup_> Bug #561586: Move javascript code to lp.app directories <javascript> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/561586 >
<wgrant> jml: What's broken about the contributions page?
<wgrant> Hmm, I possibly didn't land my fixes...
<wgrant> Still seems to work OK, although I should probably remove my non-Canonical persona from there somehow.
<jml> wgrant: it hasn't been updated since May. Maybe that's not a problem. Also, you're listed as non-Canonical.
<jml> wgrant: anyway, in a meeting. please reply via email.
<deryck> sinzui, hi.  Don't suppose you had any luck on 64 bit errors with YUI layer yesterday?
<sinzui> deryck, I am reading wgrants gdb. The fault in the JS JIT
<deryck> ah, cool.
<sinzui> I am contemplating downgrading to libwebkitgtk-3*.so
<deryck> adeuring, shall we chat now?
<adeuring> deryck: sure
<sinzui> deryck, abentley, wgrant: The YUI segfault is a webkit's JIT. There is a history of jit failures for amd64 in webkit : https://launchpad.net/ubuntu/+source/webkit/1.3.13-0ubuntu1 I am exploring using the older version of the lib
<wgrant> sinzui: That patch is for the ARM JIT.
<wgrant> But OK.
<sinzui> wgrant, sorry, I pasted the wrong one.
<abentley> sinzui: Ah.
<deryck> interesting.
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 211 - 0:[######=_]:256
<huwshimi> deryck: I'm back now
<jml> bigjools: do you understand my XXX comment in test_resumeHost_success?
<bigjools> jml: OTP, gimme a few mins
<jml> bigjools: ok.
<jml> bigjools: http://paste.ubuntu.com/630835/ (for when you're ready)
<deryck> Hi, huwshimi.  So I don't think that bug has any impact on us....
<deryck> huwshimi, that relates to the plan to get everything, js or otherwise, out of lib/canonical/launchpad and into lib/lp
<deryck> huwshimi, and we can always spin up a thread on launchpad-dev list and see what others think.
<huwshimi> deryck: Aren't we talking about reversing that decision?
<jml> I haven't heard anything about that.
<deryck> huwshimi, I don't think so.  We're saying "have a single place of all js files"
<deryck> huwshimi, I assumed that would be something like lib/lp/javascript
<deryck> jml, it's an idea huwshimi and I are tossing around.
<jml> I saw sinzui post something about moving stuff to lp/app
<jml> deryck: I meant, I haven't heard anything about reversing the decision to move stuff out of l/c/launchpad
<sinzui> jml, deryckI stumbled on a bug yesterday about removing canonical/launchpad/js. There is about 1h of work left to remove it
<deryck> jml, yeah, there hasn't been.  huwshimi was worried our desire to move to a single js dir would reverse that bug.
<deryck> sinzui, huwshimi and I have been discussing moving all the app-specific js files to a single js directory.  you have any concerns about that?
<jml> oh ok
<jml> sorry for the noise :)
<deryck> jml, no worries :)
<abentley> deryck: so the js for translation-sharing would be moved away from its html file?
<deryck> abentley, yes
<deryck> that's by design actually.... to get people to stop doing in-page scripts.  decouple the page and the js a bit.
<sinzui> Only that putting the files in a deprecated directory is wrong. some where in lp/ is fine. I would like to think that lp/services/javascript would be the right place. I do not know if our js code can be considered a standalone, reusable library yet
<abentley> deryck: it seems to me we'd need to set up a parallel hierarchy in that js directory.
<deryck> sinzui, yeah, I was thinking somewhere in lib/lp.  I'm not picky about location.  and yeah...
<jml> fwiw, lifeless disagrees with the current 'lp' structure
<deryck> sinzui, it's not a stand alone library.  nor intended to be, I don't think.
<LPCIBot> Project windmill-devel build #263: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/263/
<deryck> sinzui, the idea is just to make it easier to discover all the js we have, and prevent people writing the same code over and over again.
<jml> he has some valid points, but I can't help but feel slightly discouraged at the thought of another whole tree refactoring
<deryck> yeah, I don't want to get into that either.  I hope this is a much simpler endeavor.
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugs: 211 - 0:[######=_]:256
<deryck> abentley, so yeah, we would need some parallel structure or grouping of files, but huwshimi and I hope it will be functional, not app-specific....
<deryck> abentley, so grouping all the pickers, all the dom-editing stuff, or something like that.
<abentley> deryck: I'm totally in favour of extracting everything we can from the page-specific files.  I'm not enthusiastic about moving the page-specific files.
<deryck> abentley, so +1 to no in page scripts, but -1 on moving the js files themselves.  Is that a good reading of that?
<abentley> deryck: yes, and also +1 on grouping all the pickers, alll the dom-editing stuff, or something like that.
<deryck> abentley, which would cover most of our js, I think.  so you're only -1 in the case where the js is tied heavily to a page, e.g. the translation sharing js.
<deryck> right?
<abentley> deryck: Not really.  I assume that most of the per-page JS files have a little bit of JS that's tied heavily to that page.
<deryck> abentley, yeah, and I see that as a bad pattern.  unavoidable completely, I agree, but we abuse the tie between page and js files.
<sinzui> deryck, abentley: could either of you untar this in a working tree and run all the YUI tests: http://people.canonical.com/~curtis/html5browser-fix.tar
<deryck> sinzui, I can in just a sec.
<sinzui> untar in the root
<deryck> sinzui, root of the lp tree?
<abentley> deryck: So you're thinking the content of the page should completely control how the JS attaches to it?
<deryck> abentley, I don't know if that's what I mean.  I don't think so, but not sure I understand what you mean....
<deryck> abentley, I mean just that we should decouple the page from the script as much as possible, make re-usable widgets primarly, and have page-specific scripts be as small and decoupled as possible.
<deryck> I also see us moving to an event-driven system, where page-specific stuff, if event reacting, rather than the long procedural-like files we have now.
<deryck> as I mentioned on the stand-up this morning. ;)  Though I know you don't care for that.
<abentley> deryck: I said "most pages have a little bit of js that's tied heavily to that page", and you said "that's a bad pattern".  I think you mean "It's bad to have even a little bit of JS that's tied heavily to a page".  Is that what you mean?
<deryck> abentley, I think it's bad to have tight coupling.  Where there is a close link between js and page, it's bad.  I think there will always be some link, of course.  so if it is truly "a little bit" then no, I don't think that's bad....
<deryck> abentley, I meant more that the "little bit of js linked to page" is a common pattern and abuses.  I question that it's really just a little.  so that's why I said it was a bad pattern.
<abentley> deryck: I think that all the javascript that doesn't need to be linked to the page should be extracted, leaving behind the little bit that does need to be linked.
<abentley> deryck: For example, I think most of the translation-sharing-details code could be extracted.
<deryck> abentley, I think we agree more than we disagree then.
<abentley> deryck: Cool.
<cjohnston> jml: is there a version of the community contrib page that I can see?
<jml> cjohnston: you can't *see* the contrib page?!
<cjohnston> nm.. I clicked the cronjob link. :-/
<jml> bigjools: re-ping re http://paste.ubuntu.com/630835/
<bigjools> jml: ok ok :)  just got off TP
<deryck> sinzui, I untarred into the lp tree.  ran make.  ran tests. still seq faulting.
<cjohnston> I would make a suggestion/request to whoever takes it over.. maybe show the number of bugs fixed as well as top level landings.. I have had 4 landings, but fixed 6 bugs. so there is the potential for a discrepancy making it look like less work has been done than what has actually been done
<sinzui> deryck, to confirm you have lib/html5browser/__init__.py ?
<jml> cjohnston: that's a good idea
<deryck> sinzui, yup.  and I see a .pyc file in there, so it's being used.
<sinzui> deryck, :( this means that natty amd64 has broken webkit 1.0, 3.0
<deryck> sinzui, that sucks :(
<bigjools> jml: it's to do with the fake slave, I think it just echo-ed its own vm_host so we can test the stdout
<deryck> sinzui, I can spin up a 32bit vm for myself.  but that's not ideal for everyone.
<sinzui> The change I made was to run everyone on the same python, pykgtk, and webkit as lucid amd64
<bigjools> jml: although I can't remember how "SlaveHelper is being moved around" affected that
<jml> bigjools: what do you think I meant by "pass the expected vm_host into the client slave"?
<jml> bigjools: the moved around thing was probably, "Oh, we should change this but it'll be mega conflict central if I do it now"
<bigjools> jml: I suspect it means to pass it to getClientSlave() and hence the __init__ for BuilderSlave
<bigjools> it's hard-coded to "vmhost" right now
<bigjools> not sure why we didn't file a bug and add it to th eXXX
<jml> bigjools: probably because we were in a rush. I'm realizing increasingly that being able to just put them down quickly is part of the glory of XXX comments.
<bigjools> jml: I tend to do TODO instead of XXX if I intend to fix it in the same branch
<bigjools> and your todo plugin helps :)
<sinzui> deryck,  do you have the python-webkit package installed?
<jml> where a bug is most useful is for XXX comments that are like, "when $FOO is done, we can fix this awful mess"
<jml> so there's a bug for $FOO
<bigjools> welllll
<bigjools> also, "I want to do $FOO but not right now"
<deryck> sinzui, yes.  1.1.8-1ubuntu2
<sinzui> :(
<sinzui> I believe you have the same packaging rules are lucid amd64
<bigjools> jml: anyway did I get anywhere near close to answering your question?
<jml> bigjools: yeah, I think so, thanks.
<bigjools> cool
<abentley> jml: I know, let's write a tool that files a bug for every XXX!
<jml> :(
<LPCIBot> Project windmill-devel build #264: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/264/
<nigelb> jml: Nice docs.
<nigelb> " Angels fear to tread."
<gary_poster> flacoste, I'm trying to come up with a proposal on how to have wadl served by apache (for bug 607961).  To your knowledge, am I right that api.launchpad.net/beta serves both wadl and json depending on the request's Accept, and that the json service root is not served anywhere else other than api.launchpad.net?
<_mup_> Bug #607961: wadl generation timeout? <lp-foundations> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/607961 >
 * gary_poster was looking at Apache for that info.  I'll look in zcml when I return
<flacoste> gary_poster: your assumptions sound correct to me
<flacoste> gary_poster: there is also a "feature" that i think you can ask the wadl of any resource, but i don't think anything uses that
<flacoste> gary_poster: so in practice, it's only the main wadl resource we care about
<gary_poster> cool thanks flacoste
 * deryck is not doing lunch, yet.
 * gary_poster thought he was doing lunch, but was wrong
<sinzui> deryck, can you run this http://people.canonical.com/~curtis/test_yui.py
<sinzui> deryck, python test_yui.py <my-built-tree>/lib/lp/app/javascript/tests
<deryck> sinzui, sure.
<sinzui> ^ that will use python2.7 without the zope testrunner
<sinzui> I doubt this will work
<deryck> sinzui, yeah, seq fault again.  sorry :(
<sinzui> deryck, okay. thank you
<deryck> np
 * deryck really lunches now
<jml> I'm off for the day
<jml> only two more working days until Dublin
<jml> I'm excited
<LPCIBot> Project db-devel build #656: STILL FAILING in 5 hr 54 min: https://lpci.wedontsleep.org/job/db-devel/656/
<LPCIBot> Project windmill-devel build #265: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/265/
<LPCIBot> Project parallel-test build #61: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/61/
<lifeless> bigjools: hi
<bigjools>  /o\
<lifeless> bigjools: I'm worried I might cause you grief if I suddenly enforce the (agreed) 1s/5s thing for the dsd new pages (e.g. +initseries)
<lifeless> bigjools: do you have any thoughts / concerns around this ?
<bigjools> lifeless: 1s?
<lifeless> 1s soft timeout
<bigjools> ah
<bigjools> 1s is pushing it
<bigjools> initseries is totally ajax btw
<lifeless> so its phrased as 1s 99th percentile
<lifeless> the soft timeout just grants access to info on the things over the 1st that don't timeout
<bigjools> the diff pages are 2-3s at the moment but when gavin lands his performance fix it might get better
<lifeless> so its an aid
<bigjools> yeah
<bigjools> did you see what he did BTW?
<lifeless> +0
<bigjools> yeah... wtf :)
<lifeless> so its avoiding an index
<lifeless> that same index is what broke sourcepackagename checking on bugs yesterday, I think.
<lifeless> so we have a root cause we need to diagnose
<bigjools> wouldn't it be great to say "WITH index SELECT ..."
<bigjools> can we analyse what indexes all the queries use?  would be interesting to see what needs that slow one
<lifeless> stub is working on that with the goal of dropping unused indices
<lifeless> same data should help us answer what-uses it AIUI
<lifeless> anyhow, on the timeout thing
<bigjools> awesome
<lifeless> I could look at the PPR for each page and rather than jump to 5s
<lifeless> perhaps jump to max(5s, current 99th percentile)
<bigjools> can do
<lifeless> what do you think?
<bigjools> we're well under 5s except for when someone hacks the batch size
<lifeless> ok
<bigjools> like I said, 2-3s
<lifeless> so I'll just set them to 5s
<lifeless> I don't have a proper list yet
<bigjools> how are you doing that, rules for specific pages?
<lifeless> the tooling for that is in our to-do list
<lifeless> yeah
<bigjools> that's going to get to be a big list
<lifeless> moderate size yes
<lifeless> but we could have a few hundred new things before its too big
<lifeless> and by then I hope we'll have dragged the default down more
<bigjools> I hope by then the sitewide is 5s ..
<lifeless> ELOCAL
<bigjools> anyway, way past EoD for me, ttyl
<lifeless> back
<lifeless> ciao
<bigjools> o/
<deryck> heh, several of us decided to catch up on sinzui's email at the same time.
<deryck> I could say something about great minds and all... :)
<flacoste> lifeless: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1998QASTAGING103
<flacoste> that's the OOPS i got yesterday on qastaging
<flacoste> the problematic query is fast (63ms) in there
<lifeless> flacoste: \o/
<lifeless> flacoste: ok, so the moral is 'have nothing wrong as a baseline'
<flacoste> yeah
<flacoste> but that still weird
<flacoste> because even then
<flacoste> wouldn't have prevented the regression from going to production
<flacoste> since the query seems fine
<flacoste> could it be bloat on production?
<gary_poster> world-at-large, do we still prefer that imported branches be for trunks?  If so, how strongly?
<flacoste> gary_poster: no
<gary_poster> cool, I'll update pertinent CHR page, thanks flacoste
<flacoste> gary_poster: but we don't support importing from non git
<flacoste> gary_poster: there reason for that was related to subversion
<flacoste> if i recall
<gary_poster> flacoste--oh--only one of hg as well?
<flacoste> as supported by cscvs
<flacoste> but not sure if the restriction applied
<lifeless> flacoste: bloat, extra data, different pg parameters related to memory sizing
<flacoste> support i mean: do not support importing non-HEAD git branches
<gary_poster> o I see
<lifeless> flacoste: thanks for digging in
<gary_poster> Ah got it: I had just missed the necessary detail already on the page
<gary_poster> "Now that we use bzr-svn, non-mainline svn branches can be imported in such that they will be compatible with mainline svn imports. However, they will not be compatible with old svn imports based on CSCVS.
<gary_poster> For Git and Hg, there's no way to specify anything other than the mainline at the moment (Bug #380871)."
<_mup_> Bug #380871: support for colocated branches <code-import> <lp-code> <Bazaar:In Progress by jelmer> <Bazaar Git Plugin:Fix Released by jelmer> <Bazaar Hg Plugin:Fix Committed by jelmer> <Launchpad itself:Triaged> < https://launchpad.net/bugs/380871 >
<gary_poster> oh
<gary_poster> oh ok, fixed everywhere except LP
<jelmer> gary_poster, and bzr
<gary_poster> heh ok
<gary_poster> thanks
<gary_poster> flacoste, we have a question about whether we block LP from Cuba (https://support.one.ubuntu.com/Ticket/Display.html?id=2772, "The access to Launchpad it's negated for the Cuban developers or do you have technical problems that causes that we can access you services from Cuba?").  I'm reasonably confident we don't, but am I missing anything?
 * gary_poster wishes he remembered Google analytics access ids
<LPCIBot> Project windmill-db-devel build #417: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/417/
<flacoste> gary_poster: we don't
<gary_poster> k thanks flacoste
<flacoste> gary_poster: we are UK hosted
<flacoste> so no US export restrictions
<gary_poster> heh
<gary_poster> flacoste, who has Google analytics access for LP, and how could I get it?
<lifeless> its more nuanced that that AIUI
<lifeless> we have a US arm
<lifeless> we could have restrictions imposed on us
<lifeless> so far we don't
<gary_poster> oh
<gary_poster> ok
<lifeless> but we can not promise that we won't.
<gary_poster> sure
<gary_poster> promises about the future are generally risky :-)
<lifeless> yeah
<flacoste> lifeless: but OOPS-1998QASTAGING102 showed the issue
<flacoste> so cold cache problem
<lifeless> combination
<flacoste> 12 or 63ms hot
<lifeless> poor cold cache, + crap plan on prod
<flacoste> but 9000 when not cached
<flacoste> probably
<flacoste> lifeless: anything we could do cheaply to have "instant" OOPS access?
<flacoste> i didn't look at that first OOPS
<flacoste> because it wasn't available immediately
<flacoste> so i reloaded
<flacoste> and retried
<lifeless> flacoste: you can rsync them directly in the dc immediately.
<flacoste> and eventually, only looked at the 'last' oops
<flacoste> ah ok
<lifeless> its fugly but works
<flacoste> lifeless: do you think we could make that automatic on 404 when visiting lp-oops?
<flacoste> oops not found -> let's try a rsync and try again
<flacoste> without the dev having to do anything
<lifeless> that would be fun
<lifeless> I see no reason why not
<flacoste> cool, i'll file a bug and eventually scratch the itch
<lifeless> hmm
<lifeless> in ff5 the filebug form is stuffed
<LPCIBot> Project windmill-devel build #266: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/266/
<deryck> lifeless, what do you mean by "stuffed"?
<lifeless> you know the expander
<lifeless> for extra options?
<lifeless> its missing
<lifeless> the include-an-attachment border is showing greyed out
<lifeless> and no widgets at all from 'extra options' down to 'submit bug report'
<lifeless> for launchpad/+filebug, where I do have perms to do that
<deryck> lifeless, weird, works for me.
<deryck> lifeless, everything looks normal to me in FF5.
<lifeless> deryck: funnky
<deryck> indeed
<deryck> lifeless, you running ff5 anyway special, or just the normal ubuntu upgrade?
<lifeless> the -proposed one
<lifeless> that micahg asked for testers on
<deryck> ah
<deryck> I just regular ol' updated today and am running that one.
<sinzui> jcsackett, do you have time to mumble?
<jcsackett> sinzui: sure.
<flacoste> deryck: https://dev.launchpad.net/Code/BazaarUpgrades
<flacoste> that's the code QA page i was talking about
<flacoste> we should have a central QA page
<deryck> flacoste, ah, ok.  Thanks.  I presume lifeless was referring to the qa tagging instructions as a central qa page?
<lifeless> yes I was
<lifeless> or at least as an entry point to same
<lifeless> ok, -> monthly allergy injection time
<gary_poster> lifeless, zero rush (I'm EoD soon), but if sometime today you could review my most recent comment on https://bugs.launchpad.net/launchpad/+bug/607961 for broad concerns, advice, or agreement, I'd appreciate it.
<_mup_> Bug #607961: wadl generation timeout? <lp-foundations> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/607961 >
<flacoste> lifeless, deryck: /QA is also a central QA page
<lifeless> gary_poster: tab is open, will look when I return
<gary_poster> :-) thanks lifeless
<flacoste> i looked at the qa tagging one and wouldn't find a place where to put the information in
<flacoste> which wouldn't be out of place
<flacoste> i emailed diogo to see if he could work on a clean-up of the wiki
<flacoste> let's see what he says
<deryck> ok, cool.
<deryck> Thanks for chasing that up flacoste!
<LPCIBot> Project windmill-devel build #267: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/267/
<deryck> Can we please kill those windmill notices?  Windmill is dead to us.
<deryck> wgrant, maybe you're the contact for that? ^^
<sinzui> windmill is dead to me
<sinzui> Windmill was never alove for me
<abentley> deryck: We can stop tilting at windmills?
<flacoste> deryck: StevenK is I think
<sinzui> deryck, StevenK own the jenkins setup. It is lying now, it was not lying yesterday about the broken YUI test
<sinzui> I fixed the YUI test
<deryck> abentley, indeed.  we'll move to selenium2/webdriver at the Thunderdome.
<abentley> Yay!
<deryck> flacoste, sinzui -- ok, thanks.  StevenK can we kill the Windmill runner?  Not needed now.
<poolie> gary, hi?
<deryck> until tomorrow then, everyone....
<poolie> cheerio der
<poolie> deryck
<micahg> lifeless: firefox 5 is out for everyone
<micahg> at least in naty
<micahg> natty
<wgrant> Yay, no more Windmill.
<sinzui> wgrant, StevenK, wallyworld_, I will be late to our meeting. Can we postping it about an hour?
<wallyworld_> sinzui: ok
<wgrant> sinzui: Sure.
#launchpad-dev 2011-06-23
<LPCIBot> Project windmill-devel build #268: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/268/
<wgrant> lifeless: Your LXC guide is vastly superior to the U1 one.
<wgrant> LXC looks really nice.
<LPCIBot> Project parallel-test build #62: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/62/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #657: FIXED in 5 hr 53 min: https://lpci.wedontsleep.org/job/db-devel/657/
<wgrant> jelmer: You seem to have removed the custom KDE layout from bzr-svn on the basis that they've moved to git, but I see new svn revs yesterday...
<sinzui> wallyworld_, do you have time to mumble
<wallyworld_> sinzui: yes, starting mumble now
<wgrant> lifeless: Hm, did you run into trouble with needing to run buildout under linux32 when using an i386 container on an amd64 kernel?
<LPCIBot> Project windmill-db-devel build #418: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/418/
<lifeless> wgrant: I'm just fiddling with that now
<lifeless> wgrant: yeah the u1 seemed to ignore issues ;)
 * StevenK sighs at the topic. Wasn't the critical bug count only 205 yesterday?
<lifeless> wgrant: feel free to tweak the page!
<lifeless> wgrant: hallyn has a magic lxc build coming to his ppa that will automate the bind mount
<wgrant> lifeless: I'm not sharing my ~, but it all seems to work OK as long as make always runs under linux32.
<wgrant> Otherwise buildout uses x86_64 meliae :(
<lifeless> interesting
<wgrant> Surely it doesn't try to parse uname...
<wgrant> Still, it is buildout.
<lifeless> that could be because arch reports 64bit (which is uname based I wager)
<lifeless> wgrant: file a bug ?
<wgrant> Maybe/
<lifeless> arch claims equivalent to uname -m
<wgrant> Now, automated COW-on-tmpfs LXC parallel testing...
<lifeless> LVM
<wgrant> Slow!
<wgrant> But maybe.
<lifeless> 'meh'
<wgrant> Well, I need a faster disk.
<wgrant> And/or SSD.
<lifeless> only deltas get written
<wgrant> Sure.
<lifeless> you may need more bandwidth available
<lifeless> how many cores?
<wgrant> But I don't care about persistence.
<wgrant> 6
<lifeless> doit
<lifeless> testr --paralell
<lifeless> I've been thinking about adding a profile option
<lifeless> testr --parallel --profile=lxc
<LPCIBot> Project devel build #828: FAILURE in 5 hr 23 min: https://lpci.wedontsleep.org/job/devel/828/
<wgrant> lifeless: How does --parallel divide the tests?
<lifeless> by time
<wgrant> Huh?
<lifeless> testr records the runtime of tests
<wgrant> How does it know about new tests?
<lifeless> they get round robined
<wgrant> Right, but how does it know to dispatch them?
<lifeless> it runs --list-tests
<wgrant> Ahh
<wgrant> I see.
<lifeless> if you supply a list it could skp that
<lifeless> wgrant: linux32 on lxc-start
<wgrant> Ahh, that could work.
<lifeless> s/could/does
<lifeless> its in the wiki page now
<lifeless> sudo lxc-start -n lucid-test-lp -d usr/bin/linux32 sbin/init
<lifeless> robertc@lifeless-64:~$ ssh 192.168.122.44
<lifeless> $ arch
<lifeless> i686
<lifeless> wgrant: so you're not bind mounting; nfs?
<wgrant> lifeless: Nothing at the moment.
<wgrant> Since I don't want to break my external environment.
<wgrant> Which rebuilding it for i386 will.
<lifeless> another reason to use packages for deps
<lifeless> wgrant: the eggs are arch specific
<lifeless> wgrant: so nothing should break
<wgrant> lifeless: It's not just eggs.
<wgrant> lifeless: Some of sourcecode/ has some binaries.
<wgrant> Oops
<lifeless> ?
<wgrant> lxc-destroy does not just shut down the container.
<lifeless> -stop is what you wanted
<wgrant> Yeah, I'm too used to Xen.
<lifeless> but its glitching with lucid + arch diff for some reason
<wgrant> Also, I ended up running two aufs clones of a container while still running the original container. That doesn't work very well.
<lifeless> heh
<lifeless> wgrant: the binariies in subverypy etc are also arch guarded paths.
<wgrant> https://lpstats.canonical.com/graphs/LPProjectCriticalBurndown/20100624/20110624/ is upsetting.
<wgrant> lifeless: IIRC it's pygettextpo or pygpgme
<lifeless> ah, yes
<lifeless> bad gettextpo
<lifeless> urgle, we symlink gettextpo.so ?!? into lib
<wgrant> Of course!
<lifeless> we should import gettextpo.gettextpo, then we could indirect sensibly
<wgrant> Yes, but that wouldn't be ancient and crufty enough.
<lifeless> and I think I'm finally setup
<lifeless> with a tonne more memory available.
<lifeless> and a 32 bit build for leaner tests
<lifeless> ugh, something is generating awful spew and breaking testr -- -t codehosting :(
<wgrant> :(
<lifeless>   File "/home/robertc/source/launchpad/lp-branches/working/lib/canonical/launchpad/scripts/__init__.py", line 62, in execute_zcml_for_scripts
<lifeless>     'Setting up Zopeless CA when Zopefull CA is already running'
<lifeless> AssertionError: Setting up Zopeless CA when Zopefull CA is already running
<wgrant> Yay
<lifeless_> that was unexpected
<wgrant> Oh?
<lifeless> I'm guessing OOM kill got gdm
<wgrant> Nice!
<StevenK> Haha
<StevenK> Certainly fixed the memory pressure, though
<lifeless> given I was watching a railsconf video I can't really be sure
<jtv> StevenK, wgrant: does either of you have time to show me how to create a sync upload on dogfood?
<lifeless> wgrant: 15:11 < hallyn> lifeless: do'h, that was easier than i thought, lxc already does the right thing, you just need 'lxc.arch = i686' in the config file.
<wgrant> lifeless: Ah!
<wgrant> lifeless: Which channel is that?
<wgrant> #ubuntu-virt?
<wgrant> Why is my pipe to the DC so terrible :(
<lifeless> wgrant: #ubuntu-server
<lifeless> also, https://bugs.staging.launchpad.net/ubuntu/+bugtarget-portlet-bugfilters-stats
<lifeless> what part of our code would be doing fork in twisted ?
<lifeless> Upon execvpe sleep ['sleep', '2'] in environment id 280573852
<lifeless> :Traceback (most recent call last):
<lifeless>   File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/process.py", line 412, in _fork
<lifeless>     self._setupChild(**kwargs)
<lifeless> ...
<lifeless>   File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/posixbase.py", line 171, in wakeUp
<lifeless>     util.untilConcludes(os.write, self.o, 'x')
<lifeless>   File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/python/util.py", line 783, in untilConcludes
<lifeless>     return f(*a, **kw)
<lifeless> OSError: [Errno 9] Bad file descriptor
<LPCIBot> Project parallel-test build #63: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/parallel-test/63/
<lifeless> anyone around that knows our mm integration ?
<wgrant> I know a liiiittle.
<wgrant> Enough to debug it mostly.
<lifeless> so
<lifeless> lib/lp/services/mailman/tests/test_mm_cfg.py
<lifeless> runs in FunctionalLayer
<lifeless> does FunctionalLayer *start* mailman ? I thought it didn't.
<lifeless> thus, I am confused.
<lifeless> it wants a /var/tmp/mailman directory
<lifeless> this is unsafe for parallel testing
<lifeless> I want to make it dynamic
<wgrant> Doesn't buildmailman create /var/tmp/mailman?
<wgrant> (vomit)
<StevenK> Barry wants to kill buildmailman
<wgrant> Yeah, I think it does.
<StevenK> I fully support this
<wgrant> buildmailman configures mailman into /var/tmp/mailman
<wgrant> lifeless: So, I think you probably want to run them in a layer that is non-parallelisable.
<wgrant> Or just delete mailman.
<lifeless> why?
<wgrant> Why what?
<lifeless> why non-parallelisable (btw that does not exist) or delete (don't we use it?)
<wgrant> We use it, but it's crap.
<LPCIBot> Project windmill-db-devel build #419: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/419/
<wgrant> And it's hard to efficiently parallelise because you'd probably need to rebuild mailman to change the path.
<lifeless> breaking lists.launchpad.net would probably be considered a regression
<wgrant> Quite possibly.
<lifeless> I can't quite tell if you're trolling or not
<lifeless> We should be able to setup a transient mailman per test runner fairly directly
<wgrant> Directly but slowly.
<lifeless> I have a fixture around that creates a python module
<lifeless> lib/mailman isn't even importable
<lifeless> it shouldn't be under lib
<lifeless> 23 seconds
<lifeless> anyone got attachments to /var/tmp/mailman ?
<lifeless> I'd like to shove it under the source tree
<lifeless> e.g. ./var/mailman
<StevenK> I'd like to kill it from the tree entirely and move to MM3
<lifeless> we'll want a network fake that we can use to make sure we interact with it correctly
<lifeless> I'd love to see that done and it move out of tree
 * wgrant throws rocks at aufs.
<lifeless> I think I need to tweak fixtures a little first
<jtv> hey there wgrant!
<wgrant> Hi jtv. Did you work out what you needed?
<jtv> Nope
<wgrant> StevenK possibly knows, or I can work it out.
<jtv> Big J said it was easy though
<jtv> Haven't heard from him either.
<wgrant> But it was a feature introduced by your squad, so you should know :P
<lifeless> did he say 'a small matter of programming' ?
<StevenK> Unlikely
<StevenK> Since he and I wrote the support
<jtv> I suppose I can just run "make harness" on dogfood, but it'd be nice if I had a UI way to do it.
<wgrant> jtv: the DSD sync thing will create jobs over an FF-configurable threshold, won't it?
<wgrant> When the job runs, it will check if it's allowed. If not, it will put itself in the queue.
<jtv> No, that's not the case anymore.  It has to do that in all cases now.
<jtv> Do we cron DSDJobs now?
<jtv> On dogfood, I mean?
<wgrant> I'd check the crontab.
<jtv> ah â distroseriesdifference_job.py, every 2 minutes
<jtv> StevenK, wgrant: mind if I sync some packages on df for a while?
<wgrant> jtv: Go ahead.
<StevenK> It won't love you, but go ahead
<jtv> StevenK: I can't live without love, but I probably can without yours.  :)
<wgrant> Well, this seems to be working OK.
<jtv> (Hmm the error reporting on +localpackagediffs actually works pretty well for me nowâ¦ needs a bit more tweaking but it's clear there's an error and it takes up no new space)
<wgrant> I have three tmpfs aufs COW LXCs running the test suite.
<jtv> want want want!
<wgrant> Now I just need to automatically provision them (mkdir, mount, mount, sed, lxc-start does it), and tie it into testr.
<jtv> well that's the interesting part, isn't it?
<wgrant> Well, aufs is infamous for its flakiness.
<jtv> I don't think I know it.
<wgrant> We used it at uni for a while and had to give up.
<wgrant> It's like unionfs, except less flaky.
<wgrant> But still flaky.
<wgrant> I believe it's still used for livecds.
<lifeless> wgrant: nice
<lifeless> wgrant: you'll want to bring one up for --list-tests
<jtv> So you're giving each container its own union mount overlayed on the branch setup?
<wgrant> jtv: Right, I have the base container with postgres and LP installed.
<wgrant> Then I create a tmpfs for each instance.
<wgrant> Mount that as an RW overlay over the RO base container for each instance.
<lifeless> wgrant: hows the memory pressure ?
<wgrant> lifeless: It's i386, so it's nothing.
 * jtv underlines "per-directory journaling QoS" on his filesystems wishlist
<wgrant> I only have 8GiB of RAM and killed Chromium recently. But I have 3GB free and a couple of GB cached.
<lifeless> wgrant: well, when you get to 6 instances, it may be less -nothing- :>
<wgrant> True.
<wgrant> Let's see what happens if I spin up a few more...
<jtv> Wouldn't it be great if you could tell a regular filesystem, "feel free to forget the entire contents of this directory on next mount, if it's more convenient than recovery"?
<StevenK> tmpfs?
<wgrant> jtv: Indeed.
<jtv> StevenK: no, that ties the policy to the mechanism.  And you have to manage it, e.g. unmount it when you no longer need it.  And it causes worries about how it interacts with swap.
<StevenK> Meh, they're just pages
<jtv> What I want is a /tmp that's still a regular directory on my regular disk, but doesn't waste my time journalling.
<StevenK> Then create it as ext2
<jtv> *cough*
<lifeless> seriously
<StevenK> Or use tune2fs to remove the journal
<jtv> *cough* *cough*
<lifeless> which will btw make it ext2
<wgrant> lifeless: 6 running now, hovering at 64% RAM used, all 6 cores >95%
<jtv> I do still want journaling _and_ per-file system on my FS by default, thank you.
<lifeless> wgrant: schweet
<StevenK> lifeless: Sort of. It can be ext3 with other ext3 features and no journal
<wgrant> And it's all tmpfs, so no iowait.
<jtv> wgrant: and I don't suppose there's any CPU overhead to lxc?
<wgrant> Despite my system drive currently being a shitty WD Green.
<lifeless> jtv: bugger all
<wgrant> jtv: It's just namespacing, basically.
<wallyworld_> should i expect that we could add a new (null) column to an existing table "live"? and populate it with a script? and then mark the column as not null
<wgrant> jtv: All runs on the system kernel.
<jtv> Sweet.
<lifeless> jtv: uses built in namespace facilities in the kernel, needs a new userspace library instance etc
<lifeless> and you have a bit of networking thunks going on
<wgrant> lifeless: The first two runs were on their own. DatabaseLayer took 1:21 to run. The sixth just passed DatabaseLayer, took 1:22
<jtv> Is it the trick where the kernel basically just keeps separate pointers for its data structures, so there are no cross-references?
<wgrant> lifeless: So it seems to scale rather well.
 * jtv is so looking forward to this
<lifeless> jtv: https://dev.launchpad.net/Running/LXC
<jtv> Thanks
<wgrant> It's just a bit lighter-weight than KVM.
<lifeless> jtv: thats now debugged to the point that it works; wgrants aufs and sed stuff will sit on top of that basic config
<lifeless> its a *tonne* lighter than kvm
<jtv> Should be, yes.  No qemu.
<jtv> â¦Right?
<wallyworld_> i tried to get lxc working on my box but got errors with the network devices trying to start it up :-(
<lifeless> wallyworld_: what errors?
 * wallyworld_ looks
<lifeless> wallyworld_: were you following my notes or the u1 notes or something else again ?
<wallyworld_> lifeless: your notes
<wallyworld_> ian@wallyworld:~/projects/lp-branches/devel-sandbox$ sudo lxc-start -n natty-lp
<wallyworld_> lxc-start: failed to attach 'vethgHMwnY' to the bridge 'virbr0' : No such device
<wallyworld_> lxc-start: failed to create netdev
<wallyworld_> lxc-start: failed to create the network
<wallyworld_> lxc-start: failed to spawn 'natty-lp'
<wallyworld_> lxc-start: No such file or directory - failed to remove cgroup '/sys/fs/cgroup/cpu/natty-lp'
<lifeless> wallyworld_: do you have libvirt installed ?
 * wallyworld_ checks
<lifeless> wallyworld_: if you've used virt-manager you will
<lifeless> but if you've not, even if you have used virtualbox, you probably don't.
<jtv> By the way, exactly when does test_on_merge.py run?  (I'm assuming this shifted a bit since the script was named)
<wallyworld_> lifeless: no libvert installed
<lifeless> jtv: it doesn't
 * wallyworld_ fires up apt-get
<lifeless> wallyworld_: you want libvirt-bin
<jtv> lifeless: then we can get rid of it?
<lifeless> jtv: no, we may well want it when we do the parallel testing work
<jtv> ISTRM getting the near-useless "one huge chunk twice an hour" output.
<jtv> The reason I ask is, we have CommandSpawner now which is probably better at processing output from sub-processes.
<wallyworld_> lifeless: \o/ thanks!
<jtv> I think test_on_merge just waits for a huge fixed-size chunk of output and then dumps the whole thing (even if the chunk ends in the middle of a line, hence the huge chunks)
<jtv> Using MF would fix both.
<lifeless> jtv: IIRC test on merge did more than just that
<jtv> It does do more.
<lifeless> jtv: I would be happy to see test on merge ported
<wallyworld_> lifeless: so on a 64 bit system (which i have), do i understand correctly that using a 32 bit lxc will use less memory running lp? or did i make that bit up?
<lifeless> I wouldn't be happy to have it deleted yet
<lifeless> wallyworld_: significantly less
<wallyworld_> so i would install postgres etc etc into the vm?
<lifeless> wallyworld_: we'll want to run the production tests on 64 bit to catch C bugs, but for devs it should be totally fine to run 32 bit
<lifeless> wallyworld_: yes
<jtv> The MF would need some extensions, I'm sure, but it's basically a ready-made simple API to managing parallel sub-processes.
<wallyworld_> cool. i have 4GB and am constantly up at ~90% usage
<lifeless> wallyworld_: I suspect this will help
<wallyworld_> awesome. will try it. thanks
<lifeless> jtv: MF ?
<wallyworld_> may have to set up my ide to do "remote" debugging :-)
<jtv> wgrant: I wonder if the unionfs overlay would also make a faster way to restore the test playground db between tests.  :-)
<jtv> lifeless: it stands for CommandSpawner.
<lifeless> jtv: no, because you'd need to bounce postgresql
<jtv> bugger :)
<wgrant> Hmm.
<wgrant> aufs seems to sometimes interact badly with the hardlink security checks.
<wgrant> In particularly when rm -r'ing directories on the underlay.
<jtv> lifeless: CommandSpawner was originally conceived as a Mother class that produces and manages child processes, i.e. a Forker.
<wgrant> Still, with /var/tmp/bazaar.launchpad.dev gone from the underlay it is happy again.
<wallyworld_> jtv: does that make it a Mother Forker? since it manages child processes?
<jtv> wallyworld_: shhhh
<wgrant> Hmm.
<wgrant> I guess the 6 runs don't really compete for cache.
<wgrant> Since they are using the same underlying files.
<lifeless> yes
<lifeless> the writes are the only IO load you should have
<wgrant> At least this isn't shared aufs on top of Solaris NFS.
<wgrant> That is *fun*.
<wgrant> And kernel OOPSish.
 * jtv runs in terror
<wgrant> Ah, bug #729338
<_mup_> Bug #729338: yama hardlink restriction misbehaves under aufs <linux (Ubuntu):Triaged by apw> < https://launchpad.net/bugs/729338 >
<wallyworld_> anyone want to do 2 small reviews?
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/picker-textfield-focus/+merge/65608
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/admins-can-unsubscribe-bugs/+merge/65615
<jtv> I'll take the first.
<wallyworld_> thanks
<wallyworld_> the 2nd one is only 44 lines
<jtv> Damn, picked the wrong one.
<wallyworld_> why? javascript?
<jtv> No, not 44 lines.
<jtv> Oh, wait, this one's 40 lines.  I win!
<wallyworld_> :-)
<jtv> â¦as soon as the page will load.
<wallyworld_> i did say small
<wgrant> Hmm.
<wgrant> Some tests are pretty RAM-hungry.
<wgrant> Almost OOMed there, because they were in sync.
<lifeless> yeah
<lifeless> you need the full run to guage the footprint
<lifeless> wgrant: you have what, 6G ?
<wgrant> test_process_job_sources_cronjob creates heaps of processes.
<wgrant> 8G
<lifeless> wgrant: on a hex cpu ?
<wgrant> Full LP processes :(
<wgrant> Yes.
<lifeless> wgrant: thats... unusual
<wgrant> Oh?
<lifeless> AIUI the general thing is to fill 3 banks with identical memory and you get twice the memory bandwidth
<lifeless> wgrant: http://www.intel.com/support/motherboards/desktop/sb/CS-011965.htm#triple
<wgrant> lifeless: This is an AMD chip.
<lifeless> ah
<lifeless> ignore me then :)
<wgrant> Does Intel have any hex-core chips out yet?
<wgrant> (I was just going to go with a cheap AMD quad-core, but the hex-core was only slightly more expensive so I thought why not...)
<jtv> wallyworld_: first one's done.
<wallyworld_> jtv: thanks!
<lifeless> wgrant: http://ark.intel.com/Product.aspx?id=53568&processor=E7-2803&spec-codes=SLC3M
<wgrant> lifeless: OK, but Xeons released a month ago don't count.
<lifeless> wgrant: is yours 6 cores + ?threadpercor ?
<StevenK> Haha
<StevenK> Why not?
<wallyworld_> jtv: i had the same question about not using 'unseen'. i should have done it as a driveby perhaps
<lifeless> wgrant: they've had 6 core xeons for a while, though the 10 core ones look *shiny*
<jtv> wallyworld_: I was wondering whether it might be deliberate, e.g. so as not to hinder inclusion in lazr-js.
<wallyworld_> maybe. i was going to ask tomorrow
<jtv> wgrant: I actually had reason to go with a 2-core not so long ago because it had better single-thread performanceâ¦  that wasn't for me though: what you're doing will probably make lots of cores more attractive.  :)
<LPCIBot> Project db-devel build #658: FAILURE in 5 hr 57 min: https://lpci.wedontsleep.org/job/db-devel/658/
<lifeless> jtv-eat: 2-core 4-threads?
<jtv-eat> lifeless: probably, though AMD has realized that SMT is no longer a very good idea.
 * jtv-eat logs in halfway across the world to check
<lifeless> wgrant: so I was aksing, is yours 6(12) or 6(6)
<wgrant> lifeless: Oops, missed that. 6(6).
<jtv-eat> Bulldozer?
<wgrant> Is Bulldozer out yet?
<jtv-eat> dunno
<wgrant> Q3, IIRC.
 * jtv-eat decides to live up to his name
<LPCIBot> Project windmill-devel build #269: STILL FAILING in 3 min 26 sec: https://lpci.wedontsleep.org/job/windmill-devel/269/
<wgrant> StevenK: Could you please perma-kill the Windmill builders?
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #829: FIXED in 5 hr 54 min: https://lpci.wedontsleep.org/job/devel/829/
<StevenK> The jobs themselves or the slaves?
<wgrant> 06:59:26 < deryck> Can we please kill those windmill notices?  Windmill is dead to us.
<wgrant> 07:08:09 < deryck> abentley, indeed.  we'll move to selenium2/webdriver at the Thunderdome.
<wgrant> 07:08:33 < deryck> flacoste, sinzui -- ok, thanks.  StevenK can we kill the Windmill runner?  Not needed now.
<StevenK> I missed that
<StevenK> wgrant: Delete or disable?
<wgrant> StevenK: Disable pls.
<wgrant> I don't trust that we'll have anything good by next week.
<StevenK> Both done
<wgrant> Thankyou sir.
<nigelb> Ooooh, selinium instead of windmill. Nice.
 * StevenK kills a slave while looking at Jenkins
<StevenK> I need to work out why Jenkins is now taking nearly six hours
<lifeless> http://arstechnica.com/business/news/2011/06/amd-launches-second-fusion-cpu-gives-glimpse-at-future-of-cpugpu.ars
<jtv> StevenK: how could it possibly perform well without Oracle?
<jtv> lifeless: el Reg had a nice article about HP's new APU-based laptops IIRC.
<StevenK> jtv: Your implant is acting up.
<jtv> errrhuh?
<jtv> be more Enterprise, Steve!
<jtv> (i.e. massively slower at three times the price)
<lifeless> allenap: actually I'm redoing your implementation for those bugs - see https://bugs.launchpad.net/testtools/+bug/801031
<_mup_> Bug #801031: gather_details evaluates details <testtools:Triaged> < https://launchpad.net/bugs/801031 >
<bigjools> good morning
<danilos> bigjools, good morning
<allenap> lifeless: Cool. I eagerly await the outcome :)
<lifeless> allenap: its pushed
<lifeless> allenap: along with a new stock fixture and gathering details from child fixtures
<allenap> lifeless: Ah, I've just seen it. Neat :)
<lifeless> yah, its nearly the inner loop from gather_details
<lifeless> and 0.3.6 pushed
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 211 - 0:[######=_]:256
<huwshimi> mpt: Hi. Did you want to talk about that UI lightning talk today?
<mpt> huwshimi, yep, in an hour or so if that's good for you
<huwshimi> mpt: Yeah that's fine with me
<jelmer> wgrant: still there?
<wgrant> jelmer: Indeed.
<wgrant> you're well today?
<wgrant> Or at least moreso?
<jelmer> wgrant: Yeah, feeling much better. Thanks :)
<jelmer> wgrant: I was wondering if perhaps the qastaging machine didn't have an existing copy of the imports, causing every import to be a full from-scratch import.
<jelmer> (and thus not being able to test the upgrade, no matter what the database says about the current format)
<wgrant> jelmer: I thought I used an import that was there. But I suspect I was wrong. When I QA'd it yesterday I made sure a LOSA copied an un-upgraded copy into place first, and it worked fine.
<wgrant> jelmer: After copying the staging area across, I called requestMirror() on the branch and confirmed that the externally visible copy was pack-0.92.
<wgrant> Then forced an import, and it became 2a.
<jelmer> wgrant: ah, cool
<jelmer> wgrant: thanks for qa'ing that, btw. I'm looking at QA'ing my other two branches atm
<wgrant> jelmer: Thanks. Did you see my KDE question?
<wgrant> lifeless: Slightly under 3.5 hours.
<wgrant> lifeless: For 6 concurrent runs of the full suite.
<lifeless> wgrant: thats awesome
<wgrant> About 15 minutes slower than when I ran the test suite locally, but it's probably longer now.
<wgrant> And that's with some accidental swapdeath.
<wgrant> Hi Thunderbird.
<jelmer> wgrant: isn't their plan to migrate everything to Git eventually, or are some branches going to stay in svn?
<wgrant> jelmer: I don't know. But this will break SVN now, won't it?
<wgrant> Or are existing branches OK?
<jelmer> wgrant: the layouts aren't used for anything on the Launchpad side of things, they're used to discover branches if you import an entire repository
<jelmer> wgrant: there was also a bug in bzr-svn that prevented it from opening something as a branch if the hard-coded layout (like we had for Apache or KDE) said it wasn't a branch
<wgrant> Oh, I thought it was still somehow used to work out the root for consistent fileids and stuff.
<wgrant> OK.
<wgrant> Tests with failures: lp.services.job.tests.test_runner.TestTwistedJobRunner.test_memory_hog_job (subunit.RemotedTestCase)
<wgrant> So the test suite is *deliberately* memory-hungry.
<wgrant> Awesome!
<jelmer> wgrant: it's actually allocating a lot of memory to test that?
<wgrant> Not too much, actually.
<wgrant>     def run(self):
<wgrant>         self.x = '*' * (10 ** 6)
<wgrant> Probably just be a coincidence that it blew up while I was running out of ram.
<rvba> wgrant: FYI, bigjools and I decided to refactor the multi parent like this: if the destination archive is empty: use the packagecloner, if not: use the (much safer but slower) packagecopier.
<wgrant> rvba: You'll need to optimise intra-archive copies with the cloner too.
<wgrant> Or initialising oneiric+1 will take foreverÂ²
<bigjools> yes
<bigjools> did we discuss that case rvba?
<bigjools> I *think* we did but my memory sucks todday
<bigjools> as does my typing
<rvba> I don't think so.
<rvba> This case will use the packagecopier.
<rvba> wgrant: what kind of optimisation do you have in mind?
<wgrant> rvba: Copying within an archive (eg. natty->oneiric) should use the packagecloner.
<wgrant> And it should be done first.
<wgrant> I think.
<wgrant> Or it's going to be slow.
<bigjools> rvba: this is the special case of initialising a new distroseries
 * wgrant eats.
<bigjools> where there's already a series
<rvba> bigjools: Yes, the previous_series case.
<bigjools> rvba: the packagecopier only needs to be used if there's >1 parent
<rvba> bigjools: my understanding was that the packagecopier should be used if the destination archive is not empty.
<rvba> To avoid conflicts.
<lifeless> wgrant: did rabbitmq start ok in your lxc ?
<lifeless> its whinging in mine
<rvba> bigjools: ?
<bigjools> rvba: there won't be any conflicts in a new distroseries
<bigjools> s/new/empty/
<bigjools> rvba: so you can refine your case to empty distroseries
<bigjools> not empty archive
<rvba> bigjools: Understood.
<rvba> bigjools: I'll do that in the following branch then (the one that adds support for this init from [previous_series]
<rvba> bigjools: wgrant first branch ready to be reviewed: https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676
<rvba> This branch adds *only* multi parent backend support.
<bigjools> 1394 lines /o\
<rvba> Sorry about that.
<bigjools> :)
<bigjools> I'll look at the last changes
<rvba> YEs
<rvba> Thanks.
 * rvba fixes the conflicts introduced by merging this in the 4 dependent branches.
<bigjools> rvba: I don't think you need that duplicate source check any more
<bigjools> the packagecopier will take care of that
<rvba> bigjools: I think I have removed any duplicate source check ...
<bigjools> rvba: ah ok, it was in revno 13190
<bigjools> rvba: perhaps you can attach a partial diff that you want reviewed on the MP?
<bigjools> just email it with an attachment
<bigjools> or is it all in revno 13191?
<rvba> bigjools yes: http://bazaar.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/revision/13191
<bigjools> ok I'll look at that
 * bigjools gets some caffeine in preparation
<rvba> Thanks.
<wgrant> lifeless: No, that was one of the 5 tests with errors.
<wgrant> lifeless: Because my hostname isn't configured.
<wgrant> lifeless: I presume.
<wgrant> lifeless: (the other tests were all mmcfg stuff relating to aufs)
<bigjools> rvba: first comment:
<bigjools> def getPublishedSources(name=None, names=() ....
<bigjools> rvba: can you make the code accept name as a single item or a list
<bigjools> two params is ugly
<rvba> bigjools: ok.
<bigjools> rvba: the external API should be ok still, you'd just do a type check in the method
<rvba> bigjools: Right, I'll do that.
<bigjools> Python FTW
<rvba> I'm afraid wgrant concerns about duplicated files is still not addressed :(
<bigjools> why's that?
<bigjools> if you use the packagecopier for subsequent syncs it should be
<rvba> but how about the first copy ... the one that will use packagecloner?
<bigjools> that can never conflict
<bigjools> the series is empty
<rvba> oh ... that's right ... sorry, my mind is a mess I must say.
<bigjools> understandable :)
<bigjools> rvba: well to be more specific, intra-archive to an empty series can't conflict
<rvba> Right.
<bigjools> inter-archive to an empty series *can* conflict but we prevent that elsewhere
<bigjools> brb
<wgrant> Right. Intra-archive copies to a new series are always fine. Inter-archive copies to a new archive are always fine.
<lifeless> allenap: btw the +0 hack suggests prod has something wrong with it
<lifeless> allenap: a root cause affecting other pages
<allenap> lifeless: Oh no, another rabbit hole :)
<lifeless> :)
<lifeless> I don't think you need to dig into it
<bigjools> just fall down it
<lifeless> but I think you should note in the patch that avoiding the index is a workaround for an undiagnosed issue
<allenap> lifeless: Okay, I'll add that to the follow-up branch.
<lifeless> it would be nice to diagnose without impacting prod
<lifeless> but servers with 128GB of ram and TB's of capacity are not cheap
<lifeless> allenap: (and that said, the +0 thing is -much- cleaner than the union)
<allenap> lifeless: Agreed, emphatically.
<lifeless> allenap: another example of this was the sourcepackagename issue the other day
<lifeless> allenap: the query joined with archive is fast (when hot) on qastaging, terrible on prod
<allenap> lifeless: Is there a bug about this anywhere? That (archive, status) index probably offers fairly low selectivity on SPPH. I wonder if it exists in staging/qastaging.
<lifeless>     "securesourcepackagepublishinghistory__archive__status__idx" btree (archive, status)
<lifeless> thats staging
<lifeless>     "securesourcepackagepublishinghistory__archive__status__idx" btree (archive, status)
<lifeless> allenap: we possible want (status, archive)
<lifeless> allenap: -or-
<lifeless> (archive, status where status in 1,7,534) [or whatever it is]
<bigjools> heh, those look like IDs :)
<lifeless> if its a stats problem a partial index will help more than a reversed index I think
<allenap> lifeless: Okay, I'll throw this discussion in a bug report.
<lifeless> thanks!
<bigjools> rvba: why are you doing:
<bigjools> for pocket in archive.getPockets():
<bigjools> and running do_copy in a loop?
<rvba> bigjools: I thought this was the only way to use do_copy because I had to specify the destination pocket in do_copy.
<bigjools> hmph
<bigjools> ok
<rvba> bigjools: It's stupid ... so if you have any idea around that I'm glad to hear about it :).
<bigjools> rvba: I was hoping that do_copy would use the same pocket as the source
<bigjools> but it appears not
<rvba> Me too.
<bigjools> rvba: so this is a bit simplistic anyway
<bigjools> the pocket should always be RELEASE
<bigjools> I think the cloner does that?
 * rvba checks
<rvba> Indeed.
<rvba> Wait ...
<rvba> pubrec.sourcepackagerelease.createBuild uses PackagePublishingPocket.RELEASE
<rvba> bigjools: no, you're right, initialize_distroseries has the destination pocket hardcoded to RELEASE
<bigjools> rvba: that's the only available pocket in an unreleased series
<bigjools> so just hard-code it and remove the loop
<rvba> bigjools: ok.
<huwshimi> mpt: Available anytime. I can come find you if you like or you can come find me... I've moved further around the office
<lifeless> hmm, oops
<lifeless> I lp-landed rather than ec2.
<lifeless> ah well, its only a dep update, and should be sane.
<lifeless> buildbot will hate on me if its not.
<bigjools> rvba: on line 670 of packagecopier.py, why did you need to add the extra "if len(source.getBuiltBinaries()) != 0:" ?
<bigjools> does copyBinariesTo blow up without it?
<rvba> bigjools: Yes
<bigjools> rvba: I think it would be better to make copyBinariesTo not blow up
<rvba> That's a side effect of adding 'strict_binaries'
<bigjools> just return Non if binaries is None or empty
<bigjools> right
<rvba> bigjools: ok.
<bigjools> rvba: you need to be wary of multiple calls to getBuiltBinaries() there, it's a crazy query
<bigjools> making copyBinariesTo will fix that double call
<bigjools> making copyBinariesTo work will fix that double call
<rvba> Understood.
 * rvba looks at getBuiltBinaries
<bigjools> issuing the same query more than once is crack regardless :)
<cjwatson> wgrant: any news on that cocoplum downtime for germinate-lubuntu?
<cjwatson> wgrant: I'm more or less unblocked on the disk space side now ...
<wgrant> cjwatson: I've not seen mrevell this week, and he normally organises downtime. Perhaps I should just arrange it myself...
<cjwatson> is there much to organise in this case?  nobody except IS normally notices if I put cocoplum onto manual for a bit :-)
<huwshimi> wgrant: He's on leave this week
<wgrant> cjwatson: Well, we should restart poppy or it might get a bit angry.
<wgrant> cjwatson: And we consider that to be downtime worthy of announcement, historically.
<bigjools> you need to restart poppy for a germinate change?
<wgrant> We could just cowboy it, but that upsets people.
<wgrant> And we've already had one this week :(
<wgrant> Plus keeping cocoplum up to date is nice.
<bigjools> why does poppy need a restart for a germinate change?  Remember it's a separate tree now.
<jml> huwshimi: can I persuade you to join me for an early lunch?
<wgrant> bigjools: Not on cocoplum it's not.
<wgrant> Unless that changed without anybody telling me.
<wgrant> And I'm pretty sure it didn't.
<huwshimi> jml: Probably, I was going to have a chat with mpt sometime this morning but I might be able to delay that. Did you want to lunch now?
<jml> huwshimi: now would be ideal for me
<huwshimi> jml: ok
<huwshimi> jml: One sec
<huwshimi> mpt: Hey, can we delay until this afternoon or tomorrow sometime?
<gmb> allenap: If you've got time, a branch for you to review (diff hasn't generated yet): https://code.launchpad.net/~gmb/launchpad/private-branches-bug-657004/+merge/65639
<jml> huwshimi: if you've already got something scheduled then don't break it just because I'm hungry and I want to talk to you :)
<jml> huwshimi: we aren't barbarians
<gmb> We aren't?
<gmb> Damn.
<jml> gmb: I can't speak for anyone north of Islington.
<huwshimi> mpt: I'm heading out to lunch, so let's figure out a time later
<allenap> gmb: Certainly.
<mpt> huwshimi, ok, sorry I was held up with this USC stuff
<huwshimi> mpt: No totally fine, we didn't schedule a proper time anyway
<huwshimi> mpt: I just didn't want you to be sitting around waiting for me
<bigjools> wgrant: still there?
<wgrant> bigjools: A bit.
<bigjools> wgrant: I am trying to work out wtf is going on in the _create_missing_builds method in the cloner
<wgrant> bigjools: Bad things.
<bigjools> wgrant: can you work out why it needs to call sourcepackagerelease.createBuild after createMissingBuilds doesn't make any builds?
<bigjools> the only place that passes always_create is lib/lp/soyuz/scripts/initialize_distroseries.py:
<bigjools> and I would have thought that forcing a build is wrong wrong wrong if there is already a build
<wgrant> bigjools: I would tend to agree with your assessment.
<wgrant> bigjools: What does annotate say?
<bigjools> it smells like StevenK
<StevenK> It was me, yes
<StevenK> For the case of copying only sources
<wgrant> cMB should create builds in that case...
<bigjools> but createMissingBuilds ...
<bigjools> cMB will always DTRT
 * bigjools is tempted to rename SPR.createBuild to SPR._createBuild
<StevenK> So, the package cloner only calls cMB in specific circumstances ...
<bigjools> it looks like it always calls it
<lifeless> night all
<wgrant> Night lifeless.
<bigjools> nn lifeless
<bigjools> StevenK: so why does it call it?
<StevenK> I'm struggling to remember
<LPCIBot> Project db-devel build #659: STILL FAILING in 5 hr 37 min: https://lpci.wedontsleep.org/job/db-devel/659/
<bigjools> rvba: given that none of us know why on earth the cloner is doing that extra call, I would just rely on createMissingBuilds in your code.
<rvba> bigjools: Got it.
<wallyworld_> wgrant: you still there?
<wgrant> wallyworld_: Mostly.
<wallyworld_> wgrant: just wondering - lxc doesn't support fuse but rocketfuel-setup tries to install it and it fails
<wallyworld_> so i've told rocketfuel not to install launchpad-developer-dependencies and am seeing what breaks
<wallyworld_> curious if you saw the same problem
<wgrant> wallyworld_: I always change rocketfuel-setup to use 'apt-get install -o Apt::Install-Recommends=no'
<wgrant> wallyworld_: Which doesn't install fuse.
<wallyworld_> ah. will try that. thanks
<wallyworld_> if it works, i can update the wiki
<LPCIBot> Project parallel-test build #64: STILL FAILING in 1 hr 18 min: https://lpci.wedontsleep.org/job/parallel-test/64/
<wgrant> jelmer: How goes the QA?
<jelmer> wgrant: confirmed that the classify-exceptions branch is working with a few imports
<jml> hey
<jml> what version of Python do we run on?
<wgrant> jml: 2.6
<jml> sweet
<jml> if I have a list of bug ids, is there an efficient way to ask Launchpad which of them are closed?
<jml> or am I going to do N roundtrips?
<bac> hi bigjools, are you familiar with using WebServiceTestCase?  i'm trying to figure out how to test an @collection_default_content
<jml> jelmer: wow
<jml> jelmer: I just saw your patch
<jml> jelmer: out of curiosity, have you had a chat w/ mwhudson about it?
<jelmer> jml: not recently, but we chatted about doing this earlier
<jelmer> jml: I'm hoping I can get him to do one of the reviews of the branch if it is deemed a good idea
<jml> jelmer: ok. it's kind of a big deal. are you planning on going all the way with this?
<jml> (otherwise it's a lot of dead code to be adding)
<jelmer> jml: yes, I would be happy to take this all the way
<jelmer> wgrant: when is the next deployment happening?
<wgrant> jelmer: Whenever your QA is done.
<jelmer> wgrant: oh :)
<jelmer> I'm still waiting for staging to come back
<wgrant> Well, someone (probably me) should probably QA timrc's thing first, to get another couple of revs. But once your stuff is done we can at least deploy *something*.
<jelmer> (waiting to see if https://code.staging.launchpad.net/~maxb/cvs2svn/svntest succeeded)
<bigjools> bac: not massively familiar with that.  It just gives you a "browser" object doesn't it?
<wgrant> jelmer: You know we have an importd on qastaging now, right?
<jelmer> wgrant: I do now :)
<wgrant> rvba, jtv: When you get a chance, could please take care of your outstanding QA items?
<wgrant> jelmer: We got it working earlier this week.
<wgrant> Because waiting for staging sucks.
<jtv> wgrant: I guess that means my branches have deployed somewhere.  :)
<bac> bigjools: you get a web service object you can make method calls on.  that's ok if you're not familiar
<rvba> wgrant: Sure.
<bac> wgrant: you familiar with WebServiceTestCase?
<bigjools> bac: right, I hope it's the client side of things so that lplib is exercised
<wgrant> bac: You should be able to just use it like launchpadlib.
<bac> wgrant: i
<rvba> wgrant: The crazy multi parent branch is fixed if you want to have a look at it one more time ;) (https://code.launchpad.net/~rvb/launchpad/init-series-bug-789091-devel/+merge/63676)
<wgrant> bac: eg. the collection_default_content of IBuilderSet can be checked using something like list(lp.builders)
<bac> wgrant: i'm trying to exercise a method marked with @collection_default_content.  in lplib the top level collection object is callable and invokes that method
<bac> wgrant: right
<jelmer> wgrant: \o/
<bac> wgrant: but the ws_object version of it is not callable
<wgrant> bac: The collections should not be callable, AFAIK.
<bigjools> they are generators aren't they?
<wgrant> Something like that.
<wgrant> But they're still not callable.
<bac> wgrant: perhaps callable is the wrong term
<bac> wgrant: but with an ws_object version of builders i cannot do what you showed earlier from lplib
<wgrant> bac: I suspect that wsObject only works on Entries :(
<wgrant> But self.service.builders looks like it should work.
<wgrant> Maybe.
<bac> hmm
<wgrant> Yes, that should work.
<bigjools> I see that people are adding sourcecode eggs but not deleting old ones :(
<bac> wgrant: \0/
<wgrant> jelmer: Still going...
<bac> wgrant: thanks!
<wgrant> bac: Does it work?
<bac> indeed
<wgrant> I'm pretty sure it will, but you never know.
<bac> yes, i just demonstrated it
<wgrant> Great.
<jml> bigjools: well, you have to be careful about deleting old ones.
<bigjools> jml: ECONTEXT
<jml> <bigjools> I see that people are adding sourcecode eggs but not deleting old ones :(
<wgrant> If you delete them before the next release, I will be very sad.
<wgrant> Because we will be unable to deploy.
<bigjools> jml: sure, but do people actually check?
<jml> bigjools: I honestly don't know how. And if I did, and cared, I'd probably make a post-commit hook or something that did it for me.
<bigjools> the reason I care is because a) I hate having my disk filled with crap, and b) I spent ages cleaning it up not so long ago
<bigjools> matsubara: how did the exploratory testing go yesterday?
<matsubara> bigjools, I'll put the results in the wiki and ping you
<bigjools> great thanks
<rvba> wgrant: QA done.
<wgrant> rvba: Thanks!
<rvba> np
<matsubara> flacoste, you mentioned an email about some wiki clean up, but I didn't get any. Did you send it? What's the subject?
<bigjools> rvba: presumably it worked then? :)
<rvba> bigjools: yes, I'm not an admin on DF anymore :)
<flacoste> matsubara: QA wiki clean-up
<flacoste> jml was cc-ed
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugs: 211 - 0:[######=_]:256
<flacoste> jml: did you receive your opy?
<jml> flacoste: yes, I did. I haven't had the privilege of talking w/ matsubara yet today
<matsubara> jml, flacoste: I didn't receive it. could you resend?
<flacoste> matsubara: my bad, my addressbook completed to your old async address
<matsubara> flacoste, hmm it should have arrived anyway unless the asyncers disabled my old address
<flacoste> matsubara: resent
<matsubara> thanks flacoste
<flacoste> matsubara: they did:     host smtp.async.com.br [189.19.234.109]: 550 5.1.1 <matsubara@async.com.br>... User unknown
<matsubara> :-(
<matsubara> flacoste, got the new copy you just resent. I'll take care of that clean up.
<jml> the trick is to get all the QA info grouped together, I reckon
<jml> flacoste: Millbank 2439
<flacoste> jml: ok, will dial in 2 minutes
<jml> :)
<jelmer> wgrant: ah, staging and qastaging share their import slave?
<jelmer> just got a database lock on staging because of my import run on qastaging
<jml> reviewer sought: https://code.launchpad.net/~jml/launchpad/xxx-cleanup/+merge/65663
<wgrant> jelmer: They do. I would have hoped they wouldn't share locks, though :(
<jcsackett> jml: looking now.
<jml> jcsackett: thanks!
<timrc> wgrant, did you run into any trouble with my commit?
<wgrant> timrc: It is just after midnight. I was planning to QA it tomorrow if nobody had done it by then.
<timrc> wgrant, gotcha
<jcsackett> jml: r=me.
<jml> jcsackett: thanks
<jml> lib/canonical/launchpad/pagetests/webservice/apidoc.txt fails for me in stable
<jml> anyone else have that?
<jml> is there a make target I need to build first?
<bac> jml: if you actually change API stuff the make file does not rebuild the wadl
<bac> jml: you can either
<bac> rm -rf lib/canonical/launchpad/apidoc/
<bac> or 'make clean build'
<jml> bac: thank you
<jml> bac: lib/canonical/launchpad/apidoc doesn't exist :\
<jml> ahh ok. I think I know what's going on.
<jml> bac: thanks for the pointer.
<bac> jml: working?
<jml> bac: *maybe*. tests take a while :)
<timrc> am I suppose to be able to authenticate with staging.launchpad.net via lynx?  I'm able to login in successfully but then it takes me to a page with nothing but the word 'Continue'.  When I submit again, it eventually takes me to the same page :/
<timrc> I'm never presented with a page asking me the level of access I want to grant
<jml> timrc: I'm not sure. I don't think lynx is one of our supported browsers.
<timrc> jml, poop.  okay... off hand do you know which console browser is supported?
<jml> timrc: maybe none of them?
 * timrc 's frown deepens
<jml> timrc: I don't know where our supported browsers list lives.
<jml> timrc: but surely we have to have *some* way of authorizing console apps.
<timrc> maybe google knows, brb
<timrc> jml, ok according to https://help.launchpad.net/API/launchpadlib lynx is supported for > 1.5.5 and I'm using 1.6.1 hm
<jml> timrc: ok. quite possibly it's a bug. can you authorize against production?
<timrc> jml, I'll attempt to do so
<jml> benji: random thought: would it make sense for exported webservice references to allow 'schema' to be a string containing the fully-qualified Python name for an interface?
<timrc> jml, same thing using LPNET_SERVICE_ROOT
<jml> :(
<jml> timrc: looks like a regression then :\
<nigelb> er
<nigelb> lynx works
<jml> timrc: how can I trick Python into opening lynx to do the auth? (so I can try to reproduce locally)
<nigelb> I've used it before
<jml> nigelb: well, we know it *used* to work :)
<nigelb> jml: yes, as of 2 weeks back when I set up tarmac
<timrc> jml, I do not know, honestly... it defaults to lynx for Ubuntu server (probably uses sensible-browser to make that determination)
<timrc> nigelb, did you have to setup lynx or did you use default params?
<jml> hmm. that'll slow down fixing.
<timrc> er config
<nigelb> timrc: default
<nigelb> jml: sudo update-alternatives --config x-www-browser
<timrc> nigelb, okay
<jml> nigelb: I'm not changing my default browser just to repro a bug
<timrc> jml, hehe
<nigelb> hehe
<nigelb> You can change it and then change it back :)
<timrc> jml, I use vm's to prevent having to do that
<jml> !
<jml> when I was a lad, we had environment variables
<jml> and we used them
<jml> and were grateful for them!
<nigelb> haha
 * timrc did not exist in the 1970's
 * timrc ducks
<nigelb> neither did I
<nigelb> :P
<nigelb> if you have a vps, try it on that.
<nigelb> it should use lynx by default I think.
<benji> jml: that might help with the circular reference problems; zope.dottedname would be an obvious implementation (http://pypi.python.org/pypi/zope.dottedname)
 * benji wonders how sarcasm could ever not be good.
 * benji also wonders if he'll ever figure out how to use multiple irc chans
<timrc> nigelb, you must have the magic touch...
<nigelb> timrc: did it work? :)
<jml> benji: thanks. I, of course, would have reached for the equivalent Twisted code :)
<timrc> nigelb, not in my vm... I'm going to attempt natively... maybe bridged networking has something to do with it (completely unsubstantiated theory)
<jcsackett> timrc: i've had problems with lynx before. used elinks and had no problems, though.
<flacoste> gary_poster: lp2kanban should sort the tags i think, it's flip-flopping
<flacoste> Tags: from lp-translations, timeout to timeout, lp-translations
<flacoste> Tags: from timeout, lp-translations to lp-translations, timeout
<gary_poster> flacoste, heh, ok, I'll put a card up
<jcsackett> timrc: IIRC lynx has an issue with a cookie or something that made it not work for me. but obviously, nigelb and our docs run counter to that.
<gary_poster> thanks
<timrc> jcsackett, interesting
<timrc> jcsackett, I'll give elinks a try
<jcsackett> timrc: i ended up really liking elinks and set it up as my default for console browsers.
<timrc> jml, okay so you can specify BROWSER="/usr/bin/elinks" python foo.py to select the browser launchpadlib will use
<jml> timrc: ahh thanks.
<timrc> jcsackett, elinks did the trick... thank you very much
<nigelb> jml: okay, so environment variables still work lda ;)
<nigelb> *lad
 * jml has to go
<poolie> cheerio jml
<huwshimi> poolie: Who did you send that javascript logging email to yesterday? I never got it.
<poolie> hrm
<poolie> canonical-tech
<poolie> are you on it?
<poolie> you should join
<cr3> poolie: nice reports on velocity, thanks!
<poolie> thanks!
<cr3> where are all the yui files listed from lazr-js to generate launchpad.js?
<poolie> gary_poster: hello?
<gary_poster> hey poolie
<poolie> hi there
<poolie> i ran one of the performance tools mentioned at velocity on some lp pages
<poolie> and it points out that we have some etags on files that don't seem to actually need them
<poolie> since they hav ethe revision in the name
<poolie> this is really separate from the wadl
<huwshimi> poolie: I'm not, I'll join
<poolie> is this probably a (Low) bug, or is there a reason for it?
<gary_poster> poolie, do you mean "beta" and "1.0" and so on?
<gary_poster> oh
<gary_poster> you said separate from wadl :-P
<poolie> yeah
<gary_poster> so which pages do you mean, poolie?
<huwshimi> poolie: the link you emailed errors on Launchpad
<poolie> sorry to mix it in with this bug
<poolie> silly launchpad
<gary_poster> :-)
<poolie> which link, huw?
<poolie> gary_poster: let me see if i can find the tool again
<gary_poster> k
<huwshimi> poolie: https://launchpad.net/~canonical-tech
<huwshimi> poolie: I think it's  404, but Launchpad doesn't really tell me
<poolie> works for me
<poolie> ah
<poolie> it's privtae
<poolie> how on earth do you join then?
<huwshimi> poolie: heh, no idea
<allenap> poolie, huwshimi: One of the list admins has to add you.
<huwshimi> allenap: Ah
<allenap> huwshimi: I'm trying to find one to ask....
<huwshimi> allenap: Thank you
 * huwshimi wonders why we 404 when we mean permission denied
<allenap> huwshimi: Done (by dragnob).
<huwshimi> allenap: Thanks
<allenap> huwshimi: It's probably to do with disclosure. Permission denied indicates that the resource exists. Of course, it could just be a bug ;)
<gary_poster> huwshimi, generally, some "private" things are really private.  We don't want people to know they exist.  The name's existence is a security leak.  Not everything needs that much paranoia, but starting with paranoia and relaxing is easier than the other way around.
<gary_poster> yeah :-)
<poolie> so dragnob's a good person to ask?
<huwshimi> gary_poster: Ah I see
<poolie> it might be nice if there was a "contact the admins if this thing exists"
<poolie> but perhaps just mail will do
<poolie> gary_poster: i can't reproduce the warning in the tool
<poolie> i think it was in gogle page speed
<poolie> but, "curl -Ov https://code.launchpad.net/+icing/rev13265/lazr/build/lazr/assets/skins/sam/negative.png
<poolie> does show the response has an etag
<poolie> and that's apparently a bad idea for resources that ought to be immutable
<poolie> since it could be set to just never expire
<poolie> i don't know if that actually has any performance impact or if it's just a warning
<poolie> but since they mentioned it i was curious
<cr3> rockstar: what's this about lazr-js being deprecated? has the superseding package made it to launchpad in sourcedeps perhaps?
<gary_poster> poolie, ah!  yeah, agree.  It would have an effect if Apache were set to pay attention to inode for ETag generation, but would be innocuous otherwise.
<poolie> btw i'm pretty sure that lazr.restful does try to use the etags
<poolie> hrm
<gary_poster> poolie, I think it is worth a bug for lazr.js icing
<poolie> if i had time i would rip out its persistent object cache
<poolie> (leaving the wadl cache)
<gary_poster> we are not sure if there is a problem
<poolie> it seems to cause bugs without ever helping things
* allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 211 - 0:[######=_]:256
<poolie> ok, i'll file
<gary_poster> but it's worth a bit of a dig into it anyway
<gary_poster> thank you
<LPCIBot> Project parallel-test build #65: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/parallel-test/65/
<poolie> bug 801241
<_mup_> Bug #801241: sends etag on immutable icing resources <Launchpad itself:Triaged> < https://launchpad.net/bugs/801241 >
<sinzui> abentley, do you have time to mumble about solution to the yui-segfault issue?
<abentley> sinzui: Sure, how about in 15 minutes?
<bac> jcsackett: can you do a review at your convenience?  https://code.launchpad.net/~bac/launchpad/bug-776437/+merge/65692
<jcsackett> bac: i was just looking at that actually. :-)
<sinzui> abentley, that is good thanks
<rockstar> cr3, you might want to talk to deryck in the context of launchpad itself, but I don't think the long term plan is for anyone to use lazr-js anymore.
<jcsackett> i need to finish up one thing, then i'll take a look at it.
<bac> jcsackett: wow, you're a machine
<jcsackett> or rather, take a longer look at it.
<jcsackett> bac: i just keep on eye on the queue on my OCR days. helps me figure out how i'm breaking up my day. :-)
<rockstar> cr3, why do you think you need to use lazr-js?
<cr3> rockstar: I'm working on a project that aims to use similar technologies, methodologies, etc. as launchpad: http://ec2-50-18-91-92.us-west-1.compute.amazonaws.com/
<cr3> rockstar: so, if launchpad jumps off a bridge then I'll jump too
<rockstar> cr3, lp:phazr is an re-implemtation of most of the useful things in lazr-js
<cr3> rockstar: this is probably a question for deryck but I'd be curious to know when launchpad intends to migrate from lazr-js to that
<abentley> sinzui: ready
<LPCIBot> Project db-devel build #660: STILL FAILING in 5 hr 30 min: https://lpci.wedontsleep.org/job/db-devel/660/
<rockstar> cr3, deryck said he's in the process now, last time I talked to him.
<cr3> rockstar: awesome, I'll keep a close eye on that then
<cr3> rockstar: I had another question for you but it turned out to be a bug in yui upstream already reported by sidnei, thanks for the info!
<jcsackett> bac: r=me.
<bac> jcsackett: thanks
<abentley> sinzui: Oh, that third option was: start the browser in a subprocess, but then use that browser process to visit all the pages, e.g. using IPC.
<abentley> sinzui: So it's a combination of the other two options.
<sinzui> abentley, ah right
<bac> jcsackett|lunch: i'll look into that idea
<LPCIBot> Project devel build #831: FAILURE in 6 hr 0 min: https://lpci.wedontsleep.org/job/devel/831/
<lifeless> moin
<lifeless> wallyworld_: the lxc setup on that page supports fuse
<lifeless> wallyworld_: what didn't work for you?
<sinzui> jcsackett, do you have time to mumble?
<jcsackett> sinzui: give me five, and sure. :-)
<jcsackett> sinzui: http://people.canonical.com/~jc/images/Details%20Mockups/
<bac> hi cr3 -- do you need a hand landing https://code.edge.launchpad.net/~cr3/launchpad/hwsubmissionset_search/+merge/63768
<cr3> bac: absolutely, thanks for following up on that branch. is there anything I can do?
<bac> cr3: i haven't looked at it in detail yet.  do you think it is ready for ec2 and landing?
<cr3> bac: yep, it's been reviewed by lifeless and stub, all changes they suggested have been applied so I'm very comfortable... as if I were wearing a belt and suspenders
<bac> cr3: ok.  could i get you to write a 'commit message' on the MP?
<cr3> bac: not sure I follow, this is my first contribution to LP
<bac> cr3: np.  click on the above link to go to the merge proposal.  towards the top is a JS control "set commit message".  that is the brief description of the branch that will be used as the PQM commit message
<bac> so just type a sentence that describes the reason for the branch
<cr3> bac: done, let me know if you might word anything differently or if I might be missing special notation
<bac> cr3: no special PQM notation any more.  the tools take care of it!
<bac> cr3: looks good
<lifeless> abentley: hi
<abentley> lifeless: hi.
<lifeless> bug 690021 - will we get an OOPS when a job runs into the memory limit and fails ?
<_mup_> Bug #690021: scan_branches terminated for excessive memory abuse <canonical-losa-lp> <lp-code> <Launchpad itself:Incomplete> < https://launchpad.net/bugs/690021 >
<abentley> lifeless: yes.  Python will raise MemoryError.
<lifeless> thanks
<abentley> lifeless: np
<LPCIBot> Project parallel-test build #66: STILL FAILING in 1 hr 12 min: https://lpci.wedontsleep.org/job/parallel-test/66/
<abentley> lifeless: Also, I've done the same for sendbranchmail: https://code.launchpad.net/~abentley/launchpad/memlimit-sendbranchmail
<lifeless> abentley: cool
<lifeless> this is good stuff
<abentley> lifeless: yeah, it's good to reduce the consequences of memory-munching, and AIUI, the bzr team have reduced the memory-munching significantly.
<lifeless> I'd kindof like to memlimit the appservers too
<lifeless> with python hitting swap is never the right thing (gc languages :()
<abentley> lifeless: if you do, make sure that only the process the OOMed is affected, and it's probably a good idea to treat that process as dirty.
<abentley> lifeless: i.e. the environment is suspect, and it shouldn't be reused for another request.
<lifeless> abentley: pythons MemoryError raising should be fairly reliable, no ?
<abentley> lifeless: yes, I haven't seen it fail.  Though even with an RLIMIT_AS of 0, I had to allocate 10 ** 6 bytes to trigger it.
<lifeless> I'd certain make the current request bail
<lifeless> I don't think we'd need to restart the appserver
<lifeless> perhaps I'm wrong and it is more fragile?
<abentley> lifeless: The issue with the branch scanner was that a previous OOM was causing the following scan to OOM.
<lifeless> abentley: I guess because of slab allocators?
<abentley> lifeless: I thought it might be actual memory leaks.  I don't know enough about slab allocators.
<lifeless> abentley: so python doesn't call malloc() per object
<lifeless> it allocates slabs of various sizes which objects can go into; I'd need to go check the source to be more precise than that
<abentley> Oh, the 10**6 because of the slab allocators?  Probably.
<lifeless> no, the OOM on the next job
<lifeless> so a slab can only be released to the OS when all the objects in it have been freed
<lifeless> there may also be an implementation issue with the memlimit - if it just looks at setbrk for instance
<lifeless> anyhow - thats a good gotcha to know about
<abentley> lifeless: the bug for this was 786804
<wallyworld_> lifeless: got lp running in a 32 bit lxc container on my 64 bit os. one pita though - the .so files need recompiling so lp can run but then this stuffs up things outside the container requiring the .so files (like ec2 land). what's the cmd to recompile just the c source?
<lifeless> wallyworld_: the eggs are fine, but the things doing build_ext i will give grief
<lifeless> wallyworld_: make clean && make build should do it
<wallyworld_> lifeless: i've done that but it takes too long. i was hoping for a cmd to just redo the so files
<lifeless> wallyworld_: I have a separate tree just for ec2 land
<lifeless> wallyworld_: make clean & make build in the sourcecode subdir might work
<wallyworld_> thanks. i suspect it will. pita though :-(
<michaelh1> Hey, what's the best way of getting a faster launchpadlib at the moment?
<lifeless> michaelh1: move to London
<lifeless> wallyworld_: thats why I have a separate tree :>
<lifeless> wallyworld_: I'd like to move ec2 land etc out of the main tree; they have no business being there
<wallyworld_> yes agreed :-)
<jelmer> mwhudson: hi
<mwhudson> jelmer: hello
<mwhudson> jelmer: yes i saw your merge proposals :)
<jelmer> mwhudson: heh, ok.. wasn't sure if you'd see them through the floods of Launchpad-related emails :)
<lifeless> wallyworld_: so, fuse
<lifeless> wallyworld_: why do you think lxc doesn't support it ?
<wallyworld_> because of the errors and a google search where one of the devs said so :-)
<lifeless> what errors?
<lifeless> the config I provided supports fuse
<LPCIBot> Project db-devel build #661: STILL FAILING in 5 hr 32 min: https://lpci.wedontsleep.org/job/db-devel/661/
<lifeless> wallyworld_: can you msg me your /var/lib/lxc/$containername/config file ?
<wallyworld_> trying to install launchpad-dependencies (from rocketfuel-setup) - the package post processing said things like "/dev/fuse cannot be accessed" etc. don't have the specifics handy
<lifeless> that suggests you didn't have the right devices whitelisted
<lifeless> the config I'm asking for will help diagnose this
<wallyworld_> lifeless: https://pastebin.canonical.com/48968/
<lifeless> yeah
<lifeless> you must have made your container before I added fuse support to the wiki page
<wallyworld_> ah right
<lifeless> #fuse (workaround for Bug:800886)
<lifeless> lxc.cgroup.devices.allow = c 10:229 rwm
<lifeless> # part of the Bug:798476 workaround -
<lifeless> # remove if you are running a 64 bit lxc or
<lifeless> # 32 bit on 32-bit base os
<lifeless> lxc.arch = i686
<lifeless> add that to your config file
<wallyworld_> lp still runs up ok for simple smoke testing. i guess codehosting stuff may be broken without fuse
 * wallyworld_ fires up vi
<wgrant> wallyworld_: Codehosting works fine without fuse.
<wallyworld_> so what part of lp needs it?
<lifeless> AFAIK nothing
<wgrant> wallyworld_: The test suite passed for me except for a rabbit startup failure (probably because I changed the hostname without fixing /etc/hosts) and a few mailman tests that are an aufs bug.
<lifeless> however the stack of things for e.g. windmill and so on may well
<wgrant> wallyworld_: Nothing. It's just Recommended by one of the LP dependencies.
<lifeless> wgrant: you ran w/out fuse?
<wgrant> lifeless: Yes.
<wallyworld_> as am i since i couldn't instal lit :-)
<wgrant> I couldn't either.
<lifeless> right
<lifeless> we may want to disable recommends in rocketfuel-setup regardless
<wallyworld_> would be a lot quicker to run - fae less downloads
<wgrant> Indeed.
<wgrant> I've wanted to do so for months, but the review overhead always made me think twice.
<wallyworld_> i also had to install aptitude by hand since rocketfuel used that by default
<wallyworld_> not sure why
<wallyworld_> whereas apt-get works just fine
<wgrant> It shouldn't any more...
<wallyworld_> since when?
<wallyworld_> ah, i was likely using an old copy
<wgrant> Possibly, if you copied across your original version.
<wgrant> It certainly uses apt-get now.
<wallyworld_> yeah, my fault
<wgrant> lifeless: So, what kind of hardware are pilinut and pigeonpea?
<lifeless> FIIK
#launchpad-dev 2011-06-24
<sinzui> wgrant, mumble?
<nigelb> Is everyone free to use the LP logo to denote link to Lp profile?
<lifeless> wgrant: try this please:
<lifeless> sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
<lifeless> then start rabbit
<nigelb> ah, found the license. Ignore me.
<lifeless> wgrant: hi
<lifeless> wgrant: have you tried that rabbit fix?
<wgrant> lifeless: On a call right now. Will test soon.
<lifeless> thanks
<michaelh1> lifeless: can't move to London: ash cloud.  Plus, not enough earthquakes...
<LPCIBot> Project devel build #832: STILL FAILING in 5 hr 38 min: https://lpci.wedontsleep.org/job/devel/832/
<wgrant> :q
<wgrant> lifeless: I'm using avahi for now, but any idea how to get the IP address of a guest?
<lifeless> cat /var/lib/lxc/$host/var/lib/dhcp3/dhchp3.eth0.leases
<lifeless> or check the libvirt dhcp server log/status
<wgrant> I was hoping I could inspect the interfaces somehow.
<lifeless> ask hallyn in #ubuntu-server?
<wgrant> Hmm.
<wgrant> Why does avahi get stuck registering on first startup :(
<mwhudson> is there a tag for new person picker bugs?
<wgrant> person-picker
<StevenK> mwhudson: But if you file bugs, wgrant will frown at you.
<mwhudson> ah i think it was just https://bugs.launchpad.net/launchpad/+bug/800654
<_mup_> Bug #800654: Assign Me link on person picker visible when it should be hidden <disclosure> <picker> <qa-ok> <Launchpad itself:Fix Committed by wallyworld> < https://launchpad.net/bugs/800654 >
<mwhudson> ... maybe
<StevenK> RARGH, why does apt-mirror feel it needs to download most of the archive?
<mwhudson> i guess i should get devel running and see if it happens there
<wgrant> mwhudson: Or qastaging.
<mwhudson> wgrant: good point
<mwhudson> ah bug still present there
<mwhudson> the person picker on $team/+addmember has the "assign me" stuff
<mwhudson> wgrant: does this sound like a bug you know about?
<wgrant> Is that a problem?
<wgrant> Apart from the wording?
<mwhudson> wgrant: yes, it's the "add member" picker
<mwhudson> wgrant: neither "remove assignee" or "assign me" make sense
<mwhudson> at all
<wgrant> "Assign me" does, except it should be "Pick me" or so.
<wgrant> "remove assignee" does not.
<mwhudson> well maybe
<mwhudson> it's not a /common/ use case for a team
<wgrant> wallyworld_: ^^
<mwhudson> i guess if you're an owner you might still want to add yourself
<wallyworld_> the assign me link has been on the picker for a while. the above bug was to hide it when not needed
<wallyworld_> what's the issue with the links? i'm not quite across what is being said above
<wgrant> "A while" is about a week, right?
<wallyworld_> no, longer
<wallyworld_> months
<wgrant> No...
<wallyworld_> the last epic iirc
<wgrant> Only since jcsackett refactored it a couple of weeks ago?
<wgrant> Before then it was only on the bugtask assignee picker.
<wgrant> AFAIK
<wallyworld_> no, the refactoring was to make the code which provided the links better and consistent
<wallyworld_> across the picker refactoring
<wallyworld_> but the functionality was added last epic
<wallyworld_> oh, only the bugtask picker
<wallyworld_> perhaps that's correct
<wallyworld_> yes, i think so
<wallyworld_> so perhaps we should plug the "leak"
<wallyworld_> only the bugtask picker needs "Assign Me"
<wallyworld_> others may want "Pick Me" as mentioned above, but do we really need that?
<mwhudson> http://people.linaro.org/~mwh/silly.png
<wallyworld_> my fix for the above bug was for the bugtask case - do not show "Assign Me" if "me" was already assigned to the bug
<mwhudson> wallyworld_: should i file a bug?
<mwhudson> wallyworld_: yeah, i see that now (about the bug i linked to)
<wgrant> https://bugs.launchpad.net/launchpad/+bug/787595
<_mup_> Bug #787595: person picker could have a link to choose myself when I am a valid choice <disclosure> <person-picker> <qa-ok> <Launchpad itself:Fix Released by jcsackett> < https://launchpad.net/bugs/787595 >
<wgrant> It's new.
<wallyworld_> mwhudson: yes please, i think it's bad that way it is atm
<wallyworld_> mwhudson: perhaps it's just the wording that is bad
<mwhudson> wallyworld_: i beg to differ in this case
<mwhudson> wallyworld_: particular when i'm already in the team
<wallyworld_> fair point
<wallyworld_> could you please include the screenshot in the bug
<mwhudson> yay double filing
<wallyworld_> that will help reinforce the need for a fix in case whoever reads the bug report can't immediately see the issue
<mwhudson> wallyworld_: https://bugs.launchpad.net/launchpad/+bug/801388
<_mup_> Bug #801388: some person pickers show "assign me"/"remove assignee" when that makes no sense <Launchpad itself:New> < https://launchpad.net/bugs/801388 >
<wallyworld_> thanks.
<wallyworld_> i think perhaps the quickest fix is to have those links off by default and only enabled when needed. initially just for bugtask assignee
<wallyworld_> we can add other cases as and when required
<wgrant> wallyworld_: But jcsackett only made it global three weeks ago, as requested.
<wallyworld_> hmmm. but there's been some collateral damage :-)
<wallyworld_> certainly "Assign Me" only makes sense in certain contexts
<wallyworld_> "Choose Me" is better in other contexts
<lifeless> this is polish, and very visible polish at that
<wgrant> Indeed
<lifeless> wgrant: rabbit test ?
<wallyworld_> so, i think: default wording should be "Choose Me". bugtask assignee picker can override wording. select team member picker can disable links
<wallyworld_> but the links will remain on by default
<wallyworld_> and we can deal with other cases as the arise
<wallyworld_> they
<wgrant> lifeless: Doesn't help. I think something might be wrong with epmd.
<wgrant> rabbit says it starts fine, but the LP side of things can't see it.
<lifeless> wgrant: the outer rabbit ?
<wgrant> lifeless: By disabling the hardlink restrictions on the host the system rabbit now starts. But the LP rabbit does not.
<wgrant> Ah, and it's still broken without aufs.
<lifeless> wgrant: so I have similar symptoms
<lifeless> the host rabbit I have running and responding to rabbitmqctl
<lifeless> Status of node 'tmpMn00ML@lucid-test-lp' ...
<lifeless> Error: unable to connect to node 'tmpMn00ML@lucid-test-lp': nodedown
<lifeless> is what the test suite sees
<wgrant> Yes.
<wgrant> Status of node 'tmpp9GAiQ@lucid-lp-temp-uxeO' ...
<wgrant> Error: unable to connect to node 'tmpp9GAiQ@lucid-lp-temp-uxeO': nodedown
<wgrant> diagnostics:
<wgrant> - nodes and their ports on lucid-lp-temp-uxeO: [{rabbit,57084}, {tmpp9GAiQ,41334}, {rabbitmqctl1385,39815}]
<wgrant> So, only two aufs-specific problems now. And one of them is fixed by disabling the overzealous hardlink protection.
<wgrant> The other is that containers do strange things on reboot unless you remount.
<wgrant> Possibly they unmount too aggressively.
<lifeless> I had my pg corrupt itself
<lifeless> there is a known bug in upstart in lucid
<wgrant> I guess I should test the overhead of not using aufs.
<wgrant> See how it affects the cache.
<wgrant> Are LVM and the block cache friendly enough that snapshots will share cache?
<lifeless> AIUI -
<lifeless> new blocks written will be distinct
<lifeless> reads will be common
<wgrant> Let's use btrfs and see what happens.
<lifeless> I've found btrfs a little slow
<lifeless> anyhow
<lifeless> gl
<wgrant> Intriguing.
<wgrant> Same reboot issue with btrfs.
<lifeless> AIUI its a race condition, not fs
<lifeless> kill -9 too early
<wgrant> lifeless: Ah.
<wgrant> lifeless: Shutting down the guest remounts the root FS ro.
<wgrant> lifeless: Normally that fails, since there are files open rw on the host.
<wgrant> But with aufs/btrfs, the host doesn't usually have rw handles, so the remount succeeds.
<lifeless> wgrant: the guest fs ?
<wgrant> lifeless: Yes. But that is a tree in a host mount, so it takes the whole mount ro, it seems.
<lifeless> thats special; I haven't seen that happen to me
<wgrant> Are you running it on an FS that isn't your root fs?
<lifeless> no
<wgrant> Then you're always going to have files open, so the ro remount will fail.
<lifeless> its mount table has nothing but proc etc + the home bind mount
<lifeless> bug filing time ;)
<wgrant> Possibly.
<wgrant> dbus won't start on a btrfs snapshot :(
<lifeless> big loss :P
<wgrant> Well, means no avahi.
<wgrant> Which means I have to go hunting in dhclient's status files for the IP address.
<wgrant> Ah, stale pid got caught in the snapshot.
<jtv> Anyone mind if the dogfood appserver gets a solid kick?
<wgrant> Kill it!
<wgrant> Even better if you're QAing your stuff :)
<jtv> BTW I finally put LC_ALL in my mawson .profileâ¦ what a relief!
<wgrant> Yes!
<jtv> LC_ALL=C, that is
<wgrant> It is a bit spammy otherwise.
<jtv> I didn't dare do it for the launchpad user, butâ¦
<wgrant> It doesn't matter.
<wgrant> It should persist through the sudo.
<wgrant> Shouldn't it?
<jtv> It does.  Just such a nuisance for everyone to have to do this.
<wgrant> Bah.
<jtv> Bah indeed.  That branch I needed to Q/A hasn't hit db-devel yet.
<jtv> Maybe I'm pulling the wrong df branch.
<jtv> I'm going to restart the df appserver again.
<wgrant> Well, this bah was because I accidentally deleted the root subvolume and not just the snapshots.
<jtv> You accidentally the root volume?
<wgrant> Indeed.
<jtv> 8 out of 10 doctors recommend not deleting your root volume.
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 211 - 0:[######=_]:256
<wgrant> Hmm, Sunday is going to be warmish in Dublin, and the rest of the week is only slightly warmer than it is here. What summer!
<jtv> hi wallyworld_!  I won't be around much today because of travel stuff.
<jtv> wgrant: and "here" means winter, right!
<wgrant> jtv: Right.
<wallyworld_> jtv: hi, me too. i've been in and out this morning getting stuff sorted
<jtv> At the Wellington sprint I kept saying "winter" not because the hemisphere had me confused, but because the weather was so bad.
<jtv> It was cold everywhere I went that winter, pretty much all around the world.
<jtv> It went onto the books as the hottest winter ever.
<mwhudson> the weather in wellington that week was really very horrible for the time of year
<mwhudson> heck, it was pretty horrible for any time of year
<wgrant> Excuses, excuses.
<wgrant> Still.
<wgrant> Was better than the 46 we had back home during that week.
<wgrant> lifeless: Hmm, it works.
<lifeless> econtext
<lifeless> wgrant: ^
<wgrant> Bah.
<wgrant> Thought testr run --parallel was working.
<wgrant> Just blew up.
<lifeless> wgrant: well, progress
<lifeless> we need to fix the mangling of stdout
<jtv> This is getting far too annoying.  Rebooting.
<lifeless> -> shoopermarket
<StevenK> "Shupper Shulloper"
<StevenK> (Not sure how prevalent 12th Man is)
<LPCIBot> Project db-devel build #662: STILL FAILING in 5 hr 33 min: https://lpci.wedontsleep.org/job/db-devel/662/
<jtv> Well there's a reboot I came to regret.  Looks like today's update renders my MacBook unbootable!
<wallyworld_> lifeless: have you tried connecting to postgres using an sql client from outside the lxc container?
<StevenK> wallyworld_: O hai. https://code.launchpad.net/~stevenk/launchpad/dsp-vocab/+merge/65762
 * wallyworld_ looks
<lifeless> wallyworld_: I have not; you'll need to reconfgure postgresql to permit that
<wallyworld_> lifeless: any ftfm page you can point me to? if you don't have it handy, i'll search for it
<wallyworld_> rtfm
<lifeless> http://wiki.postgresql.org/wiki/Client_Authentication
<LPCIBot> Project devel build #833: STILL FAILING in 5 hr 42 min: https://lpci.wedontsleep.org/job/devel/833/
<jtv> How do we test that zope won't allow a user to access something?  I'm trying assertRaises(ForbiddenAttribute, <operation>) but that, er, fails with a ForbiddenAttribute exception.  :-)
<jtv> In doctests it's easy, but what about TestCases?
<jtv> attrgetter?
<jtv> yup, getattr does the trick.
<StevenK> wallyworld_: I think I'm calling SimpleTerm() correctly. And secondly, toTerm() isn't supposed to be called directly, in which case the query in searchForTerms would have warmed up the cache.
<wgrant> Yay, all containers 6 connected to testr...
<wgrant> lifeless: Much slower with btrfs, it seems :(
<wgrant> lifeless: But I've got testr integration mostly working now.
<wgrant> Seems to be running OK.
<wgrant> hopefully will take less than an hour :)
<StevenK> wallyworld_: Prod?
<lifeless> wgrant: \o/ \o/ \o/
<wgrant> lifeless: (back to aufs, seems to be running quickly)
<wgrant> And a lot less memory pressure now it's not 6 instances of the one memory hungry test running at once.
<LPCIBot> Project parallel-test build #67: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/parallel-test/67/
<jtv> hi rvba
<wgrant> Hmm. 12000 tests done, CPU idle... but it's just sitting there.
<wgrant> Upsetting.
<wallyworld_> StevenK: sorry, i was out buying dinner before. i'll take another look at it
<StevenK> wallyworld_: Did you have another look? I hate to be pushy, but I'd love to land this before my flight.
<wallyworld_> StevenK: started to but now i have to head off to soccer. i was going to finish when i got back but that doesn't help you. all the other cases of initialising the simple term i know of pass the object as the first param, hence my confusion
<bigjools> morning all
<wallyworld_> StevenK: i guess it depends on what you expect or want the vocab to represent
<wallyworld_> and how it's values will be used - that's the context i am missing
<StevenK> wallyworld_: So, since this isn't used any where, I don't think it matters much. It can be fixed when it gets hooked up the UI?
<wallyworld_> if you want. if i were doing it i would prefer to get the terms correct so that the next branch was cleaner. you need to get a +1 from jtv anyway, so maybe he can provide a view :-)
<jtv> Was I invoked?
<jtv> hi bigjools
<wallyworld_> jtv: i need to head off to soccer. there's a review that needs +1. StevenKwants to land it before his flight. i had some questions as to whether the SimpleTerm was being correctly constrcuted
<jtv> StevenK: url?
<wallyworld_> jtv: https://code.launchpad.net/~stevenk/launchpad/dsp-vocab/+merge/65762
<jtv> I'm on it.
<wallyworld_> jtv: thanks. i'll have a look when i get back to see what transpired
<jtv> bigjools: I just remembered â differences in how overrides were set were what led me to the fixes I made.  It's just possible that (a) my branches fix the queue-tool overrides problem as well, (b) trying the overrides on sync jobs from the script now fails more spectacularly because of the security changes, or (c) the script doesn't set overrides at all somehow.  I forget the details.
<bigjools> jtv: well let's see!
<bigjools> I did write you a lovely addOverride method
<jtv> for which I thank you
<jtv> (not saying how, but I do)
<jtv> StevenK, wallyworld_: to cut things short I apprved, with remarks.
<StevenK> jtv: For your reference, there is a self.assertProvides()
<jtv> the great thing about standards...
<jtv> GAAAAAAAAAAAAAAAAAH
<jtv> All that effort to enter objectives and *one* *lousy* *bad* *page* *load* undoes it all for me.
<jtv> Which happen all the time.
<jtv> And the page is constructed such that even if you have it all ready for copy & paste it's still a chore and a risk.
<bigjools> my external monitor has a bug
<bigjools> ok, I wish I had discovered bzr qdiff a long time ago, it makes reviewing much easier
<bigjools> wgrant: do you want to double check rvb's multi-parent branch?  I am happy for it to land.
<wgrant> bigjools: I'm not here, but doesn't that need to use the copier if the archive is distinct and not empty?
<wgrant> Now it uses the cloner whenever the series is empty.
<bigjools> what did I miss ...
<bigjools> wgrant: what do you mean by distinct?
<bigjools> ah never mind
<wgrant> bigjools: Intra-archive copies can't produce file conflicts, inter-archive copies can. Regardless of series content.
<bigjools> yeah
<jml> allenap: thanks again for fixing up the doctest id to not have the 'tests/../doc' bit in it
<jml> allenap: makes running tests so much easier and eliminates an annoying bit of friction
<allenap> jml: Hehe, I'm glad :)
<LPCIBot> Project db-devel build #663: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/663/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #834: FIXED in 5 hr 52 min: https://lpci.wedontsleep.org/job/devel/834/
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Critical bugs: 211 - 0:[######=_]:256
<deryck> Morning, all.
<deryck> Morning, abentley
<deryck> abentley, shall we do a standup?  Just you and I today.
<abentley> deryck: morning.
<abentley> deryck: sure.
<deryck> cool
<bac> morning deryck
<jml> deryck, huwshimi: fwiw, it looks like landscape use lazr.anim and the lazr-js loader.
<jml> wiki page updated accordingly
<huwshimi> jml: From memory we use lazr.anim but don't use the loader
<deryck> jml, yeah, I think they use the green flash and spinner stuff.
<jml> deryck: ok. cool. I raised this because the wiki said, "they don't" and lifeless said, "they do", and I'd hate for us to get bogged down in a discussion about a verifiable fact.
<deryck> jml, right.  understood.
<deryck> what is this wiki page we speak of?
<huwshimi> deryck: The agenda for the Thunderdome
<deryck> ah
<StevenK> deryck: O hai -- I have silenced Windmill on Jenkins.
<deryck> StevenK, thank you sir!
<deryck> jml, huwshimi -- looked at the page.  I'm very confident that no one else cares if we abandon lazr-js. No one is supporting it, even if they use bits.
<wgrant> Burn!
<wgrant> Burn it.
<wgrant> U1 has abandoned it entirely, yeah?
<huwshimi> wgrant: I believe so
<jml> wgrant: I just spent 10 mins finding out for landscape. I think it's someone else's turn to verify facts.
<huwshimi> deryck: You have spoken to rockstar about lazr-js. Are they using it?
<wgrant> The only lazr-js references in the ubuntuone-servers tree are in test/lint configs.
<rockstar> huwshimi, we are not.
<huwshimi> rockstar: OK, just checking. Do you have plans to use any of it in the future?
<rockstar> huwshimi, no.  The first thing I did when rotating to U1 is to get rid of it.
<huwshimi> rockstar: OK thanks :)
<rockstar> huwshimi, after I finish this (very large) project, I'll probably start focusing on bringing phazr in.
<huwshimi> rockstar: Ah right. I thought you might have been using that already
<rockstar> huwshimi, well, in a way, we are.  We're just not using it as a dependency.
<deryck> but to be clear, the goals for phazr are much different... generic, extensible widgets, that each project can build in....
<deryck> the goal is for each project to do their own js development in their own tree, though.
<deryck> rockstar, right ^^ ?
<rockstar> deryck, indeed.
<rockstar> huwshimi, to be clear, much of what is in phazr now is stuff that we have in U1, but are pushing to a place where LP et al. can use it.
<huwshimi> rockstar: Right, got it.
<LPCIBot> Project parallel-test build #68: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/68/
<rockstar> huwshimi, ideally, I'd like to be able to push phazr stuff upstream into the yui3-gallery or yui3 as well.
<huwshimi> rockstar: That sounds great. yui could do with some more contributors
<bac> hi cr3, your branch passed all of the ec2 tests last night but got kicked out of PQM due to testfix mode.  it is in PQM right now.
<cr3> bac: thanks for the update, when should it finally land in production?
<deryck> rockstar, huwshimi -- I agree... we really should be thinking about yui as the dependency more.
<deryck> and less about lazr or phazr.
<bac> cr3: it'll go onto qastaging.launchpad.net later today.  when that happens the bug will be tagged with 'qa-needstesting'.  at that point, you should verify that everything works as expected and the tag changed to qa-ok or qa-bad.
<bac> cr3: er, s/the tag changed/change the tag/
<rockstar> deryck, indeed.  lifeless' concept of owning the whole stack is important in this case.
<jml> hey
<jml> what's bugtask.priority for?
<wgrant> 2005
<maxb> lol
<wgrant> severity was renamed to importance, priority was apparently never deleted.
<deryck> heh
<deryck> what wgrant said ;)
<wgrant> Maybe it was 2006.
<jml> I'm no DBA, but wouldn't deleting it mean faster queries?
<wgrant> a while ago, anyway
<jml> also, there are bounty tables in the db
<deryck> this stuff should be removed, yes.
<maxb> I can imaging narrowing a huge table by a single int not necessarily being worth the downtime
<maxb> *imagine
<wgrant> Well, it depends on the table :)
<wgrant> But BugTask.priority isn't BranchRevision.id.
<cr3> bac: I should do that? will I get a notification or will the notification appear in the bug report?
<bac> cr3: you'll get bug mail when the tag is added
<cr3> bac: awesome, thanks for the heads up, I get so much bug mail it's possible that I could miss it but I'll pay particular attention to that one
<nigelb> o/
<nigelb> mhall119 and I need some help with LP API.
<nigelb> Or rather the optimal way to use the LP API.
<jml> shoot.
<nigelb> loco.ubuntu.com needs to know the members of all the teams under loco-teams
<mhall119> we need to get user data from everybody who's a member of https://launchpad.net/~locoteams
<mhall119> most will be indirect members
<nigelb> Does doing a +participants call make sense?
<nigelb> and does it overload LP too much.
<nigelb> especially for a team with 18370 members.
<mhall119> oh, and we'll need to keep our data updated, so we'll need to do this regularly
<jml> huwshimi: now officially at https://dev.launchpad.net/Projects/Rebranding
<jml> mhall119, nigelb: good question
 * jml flees to a meeting
<huwshimi> jml: Thanks
<mhall119> lol
<nigelb> haha
<nigelb> he's already fleeing from LP team ;)
<sinzui> jcsackett, do you have time to mumble for the minutes my computer is operating?
<jcsackett> sinzui: sure.
<jml> my script tells me that we've closed 500 critical bugs since the start of the year and had 150 opened. that can't be right, can it?
<nigelb> Oh, its the friday before the sprint. jml will not get a pie on his face :(
<jml> ahh, I see. I'm not counting bugs that have been raised to Critical that were already opened.
<jml> hmm.
<jml> I guess I'd need to look at BugActivity to get that info
<bac> mhall119, nigelb: you may want to investigate using the team.membership_details method
<bac> mhall119, nigelb: it'll give you back a list that only includes a link to each member, not a fully populated person object
<bac> but, sadly, it doesn't account for subteams.  you'd have to dive into each separately
<nigelb> ouch
<nigelb> doesn't team.participants do that diving for me automatically?
<bac> but you can get a list of those with team.sub_teams
<bac> nigelb: it does.  but it returns fully populated person objects...each of which is pretty big
<nigelb> in our use case there is a team with only sub teams.
<mhall119> bac: that's going to be a lot of hits to LP
<bac> nigelb: if your structure is that flat, one team with one layer of subteams underneath, it would be pretty easy
<nigelb> so its either one big request to LP or a whole bunch of small requests (18000).
<bac> nigelb: no, it would be one request per subteam
<nigelb> ah, get a participants call in each subteam?
<mhall119> and the sub-team request would give us a list of populated person objects?
<mhall119> it's not going ot be quite that flat
<mhall119> as lots of loco teams have subteams
<bac> mhall119: no, it would give you a populated team.  you'd then ask for membership_details on each of those teams
<bac> mhall119: and you might get a lot of duplicate persons who are members of multiple locos
<nigelb> I should probably try doing this with launchpadlib and see how it goes
<mhall119> bac: which is fine, we can ignore them
<bac> nigelb: if you do this:  'import httplib2; httplib2.debuglevel=1' before instantiating your lplib object, you can see the traffic being generated
<nigelb> oh, w00t, that would help us understand the calls better :)
<bac> mhall119, nigelb: stating the obvious, you could then keep a minimal reference to each person, find the changes, and only have to ask lplib for the full updates for the new people or those that leave.  first time will be rough, though.
<nigelb> bac: the other thing we have to figure out make sure we don't count people who changed LP names and include old/new as different people.
<bac> nigelb: urgh
<mhall119> nigelb: we're getting the new django-openid-auth package ready that'll let us follow renames
<nigelb> mhall119: not in the gap when we do this.
<mhall119> worst case we'll have both old and new usernames in LD for a short time
<nigelb> yeah.
<sinzui> deryck, abentley I believe you can now get html5-browser 0.0.6-0 which works with 64bit natty
<deryck> sinzui, yay!  Will try here shortly.
<abentley> sinzui: I get a bunch of Permission Denied: http://pastebin.ubuntu.com/631839/
<sinzui> oh dear
<sinzui> abentley, I know that is the out-of-proc calling the pyc
<sinzui> I need to see if I lost code, or setup.py did something
<sinzui> abentley, was this on the first of second run?
<abentley> sinzui: second run, I believe.
<sinzui> That is a little more comforting
 * sinzui reads code
<abentley> sinzui: the first run also failed.
<sinzui> oh, okay then
<sinzui> abentley, I see python support and my change are not compatible. The file is not executable when it is copyied/installed.
<sinzui> I think I can have a fix deployed in a couple of hours
<huwshimi> jcsackett: Hi
<jcsackett> huwshimi: hello.
<huwshimi> jcsackett: Just having a look at those mockups. Thanks for that.
<huwshimi> jcsackett: Just wondering if you looked at wording changes
<huwshimi> jcsackett: Or is that something that'll be looked a separately?
<jcsackett> huwshimi: no, this was just about the icon/signage.
<jcsackett> i think "Details..." vs "More info..." etc is something we can look at separately.
<huwshimi> jcsackett: Ah good. OK just making sure that's still going to be looked at.
<jcsackett> while the mockups center on the "Details..." link, the question of new windows is something that can be answered more generally, so looking at that first. :-)
<sinzui> abentley, thank you for your assistance. I have a fix. I am preparing a release
<abentley> sinzui: Cool.
<huwshimi> jcsackett: Yeah OK, understand.
<timrc> abentley, ping, question about addArchiveDependency... is this not a valid use case? https://pastebin.canonical.com/49006/ (being able to add a dependency with more than one component?)
<timrc> oh, oops, let me post to pastebin.ubuntu.com (for the record)
<abentley> timrc: I don't know.  I just exported the existing functionality.
<timrc> http://pastebin.ubuntu.com/631848/
<timrc> abentley, okay, fair enough...
<abentley> timrc: You could ask bigjools.
<timrc> bigjools, should one be able to specify multiple components when adding an archive dependency?
<bigjools> timrc: no
<bigjools> it works dependent ones out automatically
<bigjools> so if you say "main" it only uses main.  If you say "universe" it'll pull in universe and main
<timrc> bigjools, interesting... that seems rather nonobvious, but I'm a noob, so that could be it
<bigjools> timrc: it's the OGRE model :)
<bigjools> it has to be like this because many universe packages depend on ones in main, but not vice-versa
<timrc> bigjools, if I specified universe in my sources.list but left out main, it would not make that assumption though, right?
<bigjools> timrc: no, you'd end up with uninstallable packages if you did that
<timrc> personally I'd expect one to have the same intuition about setting up archive dependencies as they would for their own systems
<timrc> at any rate, I'll note the subtlety
<bigjools> timrc: well the problem is that there are many stupid people out there and LP tries to save their feet from getting holes in
<bigjools> timrc: the UI page goes to some extent to make this more obvious, FWIW.
<gary_poster> abentley, hi.  question from CHR.  https://code.launchpad.net/~lyx-outline-devel/lyx/lyx_2.0.x (new import of svn://svn.lyx.org/lyx/lyx-devel/branches/BRANCH_2_0_X) is not importing correctly (http://launchpadlibrarian.net/74011932/lyx-outline-devel-lyx-lyx_2.0.x.log).
<gary_poster> Is this because there is another svn import (https://code.launchpad.net/~vcs-imports/lyx/trunk of svn://svn.lyx.org/lyx/lyx-devel/trunk) created on 2009-02-03 and so therefore maybe a CSCVS import?
<gary_poster> More generally, do you have a suggestion on what I can do for this person?
<timrc> bigjools, aye, fair enough
<abentley> gary_poster: I don't think imports can interfere with each other.  Lemme look.
<gary_poster> thanks
<abentley> gary_poster: This looks like a timeout.
<abentley> gary_poster: The import system kills timed-out workers with SIGINT.
<gary_poster> abentley, oh, ok.  so...is the timeout on his side somehow, on the svn server?  svn ls svn://svn.lyx.org/lyx/lyx-devel/branches/BRANCH_2_0_X seems fast enough
<abentley> gary_poster: I think what's happening is that repacking is taking too long and not giving enough progress updates, so the import system concludes that it's hung.
<gary_poster> oh
<abentley> gary_poster: this could probably be fixed by having a LOSA manually issue "bzr pack" on the branch.
<abentley> gary_poster: Though it looks like the following failures were different.
<sinzui> jcsackett, I took the survey. I could not complete the form twice because the question requires a different option for each picture. I wanted to mark 2 as clear, but I cannot
<abentley> gary_poster: e.g. bzrlib.errors.PathError: Generic path error: '4d28ec951c2cd6f54f2f6567909f2054.rix': Failure: unable to rename to '../indices/4d28ec951c2cd6f54f2f6567909f2054.rix'
<jcsackett> sinzui: that was the point--ranking them in preference so we get more unambiguous results. i can redo it if you think that's a bad idea. it's a quick edit.
<abentley> gary_poster: You'd have to get a losa to examine the branch, but I suspect the repository is broken.  Probably has an ../indices/4d28ec951c2cd6f54f2f6567909f2054.rix without the rest of the related files.
<sinzui> jcsackett, it was not clear to me that I had to rank each picture, sorry
<gary_poster> abentley, ok, thank you.  (1) If I point a LOSA to the LP page, will they know where to look?  I don't know
<gary_poster> where it would be
<jcsackett> sinzui: as i'm thinking about it, i think forced ranking isn't necessary.
<gary_poster> (2) if the repo is broken, then a LOSA should blow away the repo and then the import can try again?
<jcsackett> sinzui: i've removed force ranking.
<gary_poster> (3) Should I file a bug about the scenario you described?
<gary_poster> (done)
<abentley> gary_poster: I don't know.  I'm not familiar with where the import branches are stored, etc.
<gary_poster> abentley, ok fair enough :-)
<gary_poster> abentley, I didn't highlight you for my questions 2 and 3 above.  did your "I don't know" apply to them, or should I wait for a reply?
<gary_poster> (I figured it applied to question one)
<abentley> 2.  I think so.  3. There are at least two bugs; the repack timeout, and the inability to continue after the repack timeout.
<sinzui> jcsackett, http://blog.launchpad.net/?p=2646&preview=true
<gary_poster> ack abentley, thanks
<jcsackett> sinzui: i cannot see previews.
<sinzui> jcsackett, even after you login to the blog?
<jcsackett> logged-in, i can only see the dashboard (not do or open anything in it).
<sinzui> jcsackett, used the drop-down menu to see the drafts.
<sinzui> The menu says New Posts by default
<jcsackett> sinzui: i don't see a dropdown in my view.
<sinzui> :(
<sinzui> jcsackett, will the survey link change?
<jcsackett> sinzui: it has not changed.
<sinzui> jcsackett, try it now
<sinzui> I think I made you an editor
<jcsackett> sinzui: \o/ i see it.
<jcsackett> sinzui: last line. s/Please, take the link opens a new window survey/Please take "link opens a new window" survey/.
<jcsackett> er, s/Please, take the link opens a new window survey/Please, take the "link opens a new window" survey/
<jcsackett> worth keeping "the" in there, i think. :-P
<sinzui> jcsackett, do you not have permission to edit?
<jcsackett> sinzui: ah, i do. i just didn't see the link.
<jcsackett> sinzui: okay, i think it's good. you have any other thoughts before we throw the publish switch?
<sinzui> no more thoughts. publish!
<jml> :)
<gary_poster> sinzui, on https://dev.launchpad.net/Registry/RegistryReview , the section titled "Remove owned teams" seemed to imply that I should remove ~launchpad and ~canonical-bazaar.  Am I correct in assuming that I should update the wiki page to say that those two teams may be members, or do I misunderstand?
<sinzui> gary_poster, I do not see any participations: https://launchpad.net/~registry/+participation
<gary_poster> sinzui, I guess I just woefully misunderstood the text.  I thought https://launchpad.net/~registry/+members#active was pertinent
<gary_poster> sinzui, actually, I thought from the text that both directions were pertinent
<sinzui> gary_poster, We better revise the text then, removing those teams would be catastrophic
<gary_poster> I had just deleted a team from the participations
<sinzui> thanks
<gary_poster> sinzui, so I want to change "Remove owned teams
<gary_poster> ~registry does not delegate responsibilities to other teams, it does not use other teams to manage the project's it stewards. Launchpad does not provide a to know what teams a person or team owns, You can examine your admined teams and investigate those you do not understand." to
<gary_poster> "Remove owned teams
<sinzui> abentley, deryck: I believe html5-browser - 0.0.7-0~25~natty1 is now published
<gary_poster> ~registry only delegates responsibilities to ~launchpad and ~canonical-bazaar.  Any other teams should be removed (https://launchpad.net/~registry/+members#active)"
<gary_poster> Agree?
<sinzui> gary_poster, more than that. ~registry should not own a team or be a member of a team. We do not participate in a community
<gary_poster> sinzui, we already have this:
<gary_poster> Remove superteams
<gary_poster> These have appeared because of merge-delete team defects.
<gary_poster> I was going to leave that alone
<sinzui> gary_poster, The team does not participate in a community.
<gary_poster> it has a nice lplib snippet that helps
<gary_poster> OH!
<gary_poster> owned teams
<abentley> sinzui: I got it.  Now everything is failing with AssertionError: js timed out.
<gary_poster> The snippet I gave still confuses me a lot
<gary_poster> the titles says owned teams
<gary_poster> but we *are* delegating responsibilty
<sinzui> abentley, did you build the tree?
<gary_poster> I'm just going to leave this.  I don't feel I understand it well enough
 * sinzui only sees that when js is not built
<abentley> sinzui: I ran make, and it still happened afterward.  running make clean.
 * sinzui hacks installed lib
<abentley> sinzui: make clean; make does not help.
<sinzui> abentley, is do not see  `make` creating the js
<sinzui> abentley, does `make jsbuild` make the js
<abentley> sinzui: it does nothing.
<deryck> sinzui, sometimes you need a make clean_js && make jsbuild to get a rebuild of js.
<sinzui> deryck, This is my lib again :(
<deryck> ah, sorry :(
<sinzui> deryck, I added this to see exactly what you are seeing:
<sinzui> html5browser.requires_external_process(True)
<sinzui> There is a module level config so that we can disable the hack when natty or oneiric are fixed
<deryck> sinzui, gotcha.  I'm seeing the same as abentley.  So should I change something this config to see what happens?
<sinzui> deryck, no.
<sinzui> This is another manifestation of pyshared built the lib differently that what setup assumed
<sinzui> deryck, I can see what you are seeing using the switch in the module and my current tree. You will need to wait another 90 minutes :(
<deryck> sinzui, ah, gotcha now.  Ok, no worries.  Thanks for continuing to work on this.
<sinzui> yep bad magic
<sinzui> I just ran a success. I will need another install to ensure my fix is carried all the way to /usr/lib/pymodules/python2.6/html5browser
<sinzui> at least this shows the cost of the fork is not scarry
<sinzui> ;;;;p89999999999999999'
<jml> yes.
<sinzui> deryck, abentley-lunch: I just verified my newest package fixes the permission/timeout issue seen when running from the Lp testrunner. I am copying the package to Lp's ppa now
<deryck> nice!
 * bigjools heads off
<bigjools> see you all in Dublin Town
<LPCIBot> Project devel build #835: FAILURE in 5 hr 31 min: https://lpci.wedontsleep.org/job/devel/835/
<sinzui> deryck, abentley-lunch: I think html5-browser - 0.0.8-0~26~natty1 is now available
<deryck> ok, trying now....
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build #664: FIXED in 5 hr 38 min: https://lpci.wedontsleep.org/job/db-devel/664/
<abentley> sinzui: It works!
<sinzui> abentley, \o/
<sinzui> abentley, I saw a 40% increase in time to run the tests (65s). How log does it take for you to run them
<abentley> sinzui: 1 minutes 1.359 seconds.
<sinzui> thanks
<abentley> sinzui: did you use any of the solutions we talked about?
<sinzui> abentley, I have a branch trying the single instance. I hope to complete to today
<abentley> sinzui: neat.
<deryck> sinzui, works for me too!  Hurrah! :)
<sinzui> fab
<sinzui> I am just waiting for a buildbot iteration to verify this phase is complete
<deryck> sinzui, so now that I can run the entire suite nicely, I see my branch shaves about 20 seconds off the run... from 1:12 to :53 between branches.
<deryck> sinzui, just by cleaning up some of the waits stuff.
<sinzui> You are my hero!
<timrc> When I go to "permanently" delete a PPA, when I attempt to create a new PPA with the same name, Launchpad complains that it already exists... that seems wrong to me, but I'm guessing it's be design :/ ?
<timrc> Is there a way to ever reclaim that PPA name (for testing purposes)?
<timrc> I see the PPA names of PPA's I deleted weeks ago still grayed out on my lp page, so I'm guessing not
<LPCIBot> Project parallel-test build #69: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/69/
<abentley> bac: could you please review https://code.launchpad.net/~abentley/launchpad/resubmit-inactive/+merge/65820 ?
<bac> abentley: sure
<deryck> sinzui, if you're interested, I have my yui speed up branch up for review.
<bac> deryck: i just reviewed it.  nice!
<lifeless> timrc: yea
<lifeless> timrc: I think you can undelete them though
<timrc> timrc, maybe delete is the wrong word, then, maybe 'disable'?
<deryck> bac, thanks!
<deryck> bac, and yeah, I feel like it needs pulling out as well, since it's repeated in several spots, but wasn't sure how.....
<deryck> bac, unless we patched effects duration regardless of whether we use it or not.
<deryck> but I thought this had value on its own and wanted to land it as is.
<cjohnston> I would like to hear some feedback on bug 801514.. Would making a team contact a required field have a negative effect as I think it might? I can see users and teams not wanting to be forced to have a team contact
<_mup_> Bug #801514: making team contact a mandatory field when creating a team <Launchpad itself:Triaged> <LoCo Team Directory:New> < https://launchpad.net/bugs/801514 >
<mhall119> cjohnston: who are you asking?
<cjohnston> Any of the devs of LP
<cjohnston> or users
<maxb> cjohnston: That is not feasible at all. It would immediately forbid any use-case of using teams as a handle on a group of related launchpad subscriptions
<cjohnston> maxb: that's kinda what I was thinking.
<maxb> For example, we have a ~bzr-codereview-subscribers team to make it easier to subscribe to the set of all main bazaar series branches
<cjohnston> I just didn't want to make a decision for LP on my own.
<cjohnston> maxb: would you mind marking that in the bug please?
<maxb> So noted
<cjohnston> Thanks maxb
<cjohnston> maxb: just confirming that this would make the bug "Won't Fix" for LoCo Directory?
<maxb> I don't really understand how this impacts loco directory, I'm just making a statement that removing the ability to have launchpad teams without contact addresses would be an irksome loss of functionality
<cjohnston> PM maxb please?
<maxb> Well, if you like, but why not continue on the channel?
<maxb> Are we even talking about the same thing?
<maxb> I'm talking about the email field in Launchpad teams
<bac> deryck: yeah, those were my thoughts.  very landable now. thanks.
<lifeless> cjohnston: yes, its wontfix.
<cjohnston> thanks lifeless
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 211 - 0:[######=_]:256
<Ursinha> I hate whoever installed sl on devpad
<LPCIBot> Project devel build #836: STILL FAILING in 5 hr 25 min: https://lpci.wedontsleep.org/job/devel/836/
#launchpad-dev 2011-06-25
<maxb> hah
<maxb> sl is amusing.... the first time.
<lifeless> sl?
<lifeless> oh
<LPCIBot> Project parallel-test build #70: STILL FAILING in 1 hr 34 min: https://lpci.wedontsleep.org/job/parallel-test/70/
<LPCIBot> Project parallel-test build #71: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/parallel-test/71/
<LPCIBot> Project devel build #837: STILL FAILING in 5 hr 32 min: https://lpci.wedontsleep.org/job/devel/837/
<LPCIBot> Project parallel-test build #72: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/parallel-test/72/
<LPCIBot> Project db-devel build #667: FAILURE in 5 hr 59 min: https://lpci.wedontsleep.org/job/db-devel/667/
<Ursinha> good bye launchpad people, see you all in dublin :)
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #838: FIXED in 5 hr 55 min: https://lpci.wedontsleep.org/job/devel/838/
<LPCIBot> Project parallel-test build #73: STILL FAILING in 1 hr 14 min: https://lpci.wedontsleep.org/job/parallel-test/73/
<LPCIBot> Project db-devel build #668: STILL FAILING in 5 hr 33 min: https://lpci.wedontsleep.org/job/db-devel/668/
<LPCIBot> Project db-devel build #669: STILL FAILING in 5 hr 32 min: https://lpci.wedontsleep.org/job/db-devel/669/
#launchpad-dev 2011-06-26
<LPCIBot> Project parallel-test build #74: STILL FAILING in 1 hr 20 min: https://lpci.wedontsleep.org/job/parallel-test/74/
<LPCIBot> Project db-devel build #670: STILL FAILING in 5 hr 50 min: https://lpci.wedontsleep.org/job/db-devel/670/
<lifeless>  et voila, une microservice - lp:~lifeless/+junk/gpgverify
<wgrant> lifeless: That was easy.
<lifeless> wgrant: its a little incomplete
<lifeless> wgrant: its missing: - a test fake for other services; oops integration
<lifeless> wgrant: and it could -really- use some internal tests ;)
<lifeless> wgrant: oh, and a timeout facility ;)
<wgrant> lifeless: Indeed, but it is more of a microservice than we've ever had before, so it is a good start.
<lifeless> wsgi oops could stand to be -massively- refactored ;)
<wgrant> Perhaps rethought.
#launchpad-dev 2012-06-18
<wallyworld_> wgrant: i ran into some issues with by branch but finally got them sorted. it would be great if you could review it
<wallyworld_> https://code.launchpad.net/~wallyworld/launchpad/create-aag-privacy-transitions2-1009364/+merge/110706
<wgrant> wallyworld_: Looking.
<wallyworld_> thanks
<wgrant> wallyworld_: That bugtasksearch change is worrisome.
<wgrant> wallyworld_: eg. it breaks TextualBugTaskSearchListingView
<wallyworld_> i didn't run all the tests
<wallyworld_> we can leave it as it was but it will continue to return duplicates
<wgrant> Normal bug searches return duplicates too.
<wallyworld_> which is unfortunate
<lifeless> how so ?
<wgrant> It just happens that this method only returns the bug ID, so they're more obvious.
<wallyworld_> lifeless: since when would it be expected a search result should be ok to return dups?
<wgrant> They're not strictly duplicates.
<wallyworld_> they are if they are ids
<wgrant> In that particular format they are.
<wgrant> I'm not sure it's useful to suppress them.
<wallyworld_> i can use a set instead of a list when i consume them
<wgrant> And the way you've done it there breaks textual searches and kills performance.
<wallyworld_> so no problem, i'll revert and work around it
<wgrant> I agree the method's current result is less than optimal.
<wgrant> But it's all-round easier and not too much uglier to work around it using set()
<wallyworld_> apart from that, ok?
<wgrant> + if information_type in PRIVATE_INFORMATION_TYPES:
<wgrant> shouldn't that check that it's coming from something public?
<wallyworld_> probably
<wallyworld_> but maybe not
<wallyworld_> since i person may have had access before but not now
<wallyworld_> due to the new info type
<wgrant> Ah, true.
<wallyworld_> eg they had an APG
<wgrant> Yep, you are completely correct.
<wgrant> But a comment to that effect might be handy.
<wallyworld_> ok, will d. this stuff can hurt one's brain
<wgrant> Yup
<StevenK> wgrant: Can? Does.
<StevenK> Er, wallyworld_, even.
<wallyworld_> yep
<wgrant> wallyworld_: findPeopleWithoutAccess doesn't seem to have a good reason to be on APGF, and it doesn't seem to check policy grants.
<wgrant> APGF is only flattened in one direction, not the other.
<wgrant> A grant for an artifact shows up in APGF linked to all the artifact's policies.
<wgrant> But a grant for a policy doesn't show up linked to all of the artifacts, because that can be hundreds of thousands of rows.
<wgrant> And that set is much quicker to query the other way.
<wallyworld_> wgrant: it's there because it queries the aggf table, hence is with the other methods which do such queries
<wgrant> wallyworld_: It queries the subset of APGF that is AAG.
<wgrant> APGF is good for getting all the grants related to a policy.
<wgrant> It *doesn't* work for getting all the grants related to an artifact.
<wallyworld_> hmm. i will look at reworking it
<wgrant> I'm pretty sure that CTE also just overcomplicates things.
<wallyworld_> it makes the core query a bit simpler to grok
<wgrant> It should be a pretty simple query through TP, AAG, APA and APG
<wallyworld_> which you are saying you would prefer in the service?
<lifeless> wallyworld_: so its not that duplicates in that method are desired, its that; a) forcing distinct in the DB forces a sort and merge, and sometimes we don't want that; secondly bug search is a horrid engine today, and keeping it sane is tricky
<wgrant> I think it's probably best in the service.
<wgrant> Given it's sort of over-arching and doesn't fit in any of the others.'
<wallyworld_> lifeless: yeah, understood, i was just responding to my assumption that you were questioning my view on whether duplicates in search results were good or bad
<lifeless> ah
<lifeless> so, on general searches, we have a discontinuity in the UI
<lifeless> we show tasks
<lifeless> but claim to show bugs
<wgrant> Well
<wallyworld_> yeah
<wgrant> Tasks don't exist.
<lifeless> this confuses folk :) and is where the 'duplicates' come from
<wgrant> The UI deliberately never mentions tasks.
<wallyworld_> lifeless: sure, but in this case, i invoke the api that gives ids, and those are indeed duplicated
<lifeless> wgrant: abstractions leak; this leaks.
 * StevenK stabs Django
<wgrant> lifeless: Of course.
<wgrant> wallyworld_: Apart from that method being wrong, and the bug search thing, this looks good, thanks.
<wallyworld_> wgrant: cool, already fixed the search method, will move the other code, thanks for looking
<wgrant> We need a couple of nice big cleanup pass through all this once the legacy stuff is removed.
<wgrant> Great.
<StevenK> wgrant: http://pastebin.ubuntu.com/1046567/
<StevenK> lifeless: I can now import auditor.manage and auditor.settings in bin/auditor-manage, but it now can't find the module auditlog :-(
<lifeless> thats auditor.auditlog, right ?
<wgrant> StevenK: Hm, you didn't check security.py-cleanliness before you landed?
<StevenK> lifeless: Yeah, but loading that in bin/auditor-manage doesn't help.
<StevenK> wgrant: Obviously not. :-(
<wgrant> StevenK: Anyway, should be fine. Land a fix now, but the default EXECUTE should be sufficient to make it all work for tonight.
<StevenK> wgrant: Where should the three of them go? launchpad_main ?
<wgrant> (in postgres, functions default to granting EXECUTE to PUBLIC)
<wgrant> StevenK: public, with the ret.
<wgrant> st
<wgrant> rest
<wgrant> Pretty much every function should be in public.
<StevenK> wgrant: Right, land to db-devel?
<wgrant> StevenK: Indeed.
<wgrant> As long as nothing further has landed there after your patch.
<wgrant> (other than merges from stable)
 * StevenK re-checks the deployment report.
<StevenK> Looking good on that front.
<StevenK> wgrant: Some of the functions are listed with no rights. How do I tell if the new functions should be no rights?
<wgrant> StevenK: The ones that don't need any rights should have no rights :)
<wgrant> Things that are only called as helpers from SECURITY DEFINER functions don't need to be executable by normal users.
<StevenK> wgrant: http://pastebin.ubuntu.com/1046591/
<wgrant> It's not a problem for all of these methods to be publicly executable.
<wgrant> But they don't all need to be.
<wgrant> StevenK: That should break.
<wgrant> accessartifact_denorm_to_artifacts is called by accessartifact_maintain_denorm_to_artifacts_trig
<wgrant> Which is not SECURITY DEFINER.
 * StevenK adds EXECUTE rights to it.
<wgrant> branch_denorm_access too
<wgrant> build_access_cache can be EXECUTE if you want, but I don't believe it's necessary.
<StevenK> wgrant: Hah, maintain_transitively_private() isn't in security.cfg.
<wgrant> StevenK: It's a trigger function.
<StevenK> wgrant: Those security.cfg changes don't cause any tests in test_branch_access_policy_triggers, test_bugtaskflat_triggers and test_bug_mirror_access_triggers to fail.
<wgrant> That's a start.
<StevenK> wgrant: Shall I push it up?
<wgrant> Might as well.
<wgrant> Throw it off to ec2.
<StevenK> Oh, stub needs to sticky beak it?
<lifeless> nup
<wgrant> Not for security.cfg, no
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/db-branch-nap-security/+merge/110709 and it's currently in the middle of setting up ec2 test
<wgrant> StevenK: Thanks.
<StevenK> Except it's testing against devel. Fail.
<StevenK> ec2 kill needs to die early if you don't specify an instance id rather than failing with an obtuse traceback.
<wgrant> StevenK: Probably doesn't matter, if your branch is from db-devel in the first place.
<StevenK> Mmmmmm
<StevenK> Too late, was the cry
<wallyworld_> wgrant: i've pushed the changes to my branch
<wgrant> wallyworld_: Looking
<wallyworld_> thanks
<wgrant> wallyworld_: The branch got cursed again?
<wallyworld_> yep
<wgrant> Excellent.
<wgrant> Maybe we can work out what it's hanging on.
<wgrant> webops: Hi
<spm> yo
<wgrant> spm: The branch scanner hates some branches sometimes, hanging for more than 5 minutes before timing out. I'd like to trigger a hang and then get pg_stat_activity checked to see what it's doing.
<spm> can we do that in staging?
<wgrant> We've never reproduced it there.
<wgrant> Or on dev.
<spm> have we ever tried? :-)
<wgrant> Yup
<spm> just checking, your wording could have been taken as very cautiously spaken
<wgrant> Heh
<wgrant> True.
<spm> so we know which branches do this on a regular basis? it's not the on disk bzr side perhaps?
<wgrant> Nothing does it on a regular basis, AFAICT. It's normally a new branch that gets permanently cursed.
<wgrant> Pushing it up under another name works.
<spm> huh, weird.
<wgrant> Precisely.
<spm> how do you propose to trigger? and do we know which DB server(s) we need stats from? is it worth doing a * * * * * pg_stat_dump > logfile?
<wgrant> spm: branchscanner user on wildcherry
<wgrant> We trigger by wallyworld forcing a scan on that branch.
<wgrant> With a push
<spm> is wallyworld_ planning on promising cake to go with this debug? cause, you know, cake.
<wallyworld_> wgrant: you want me to do that now?
<wallyworld_> spm: i have no cake - the dog ate it
<spm> so I hear
<spm> wgrant: select * from pg_stat_activity where usename='branchscanner'; or do you want the full thing, jic?
<wgrant> spm: That'll do
<wgrant> wallyworld_: Please.
<wgrant> I have my rsynctail running to observe the hang.
<wallyworld_> wgrant: just pushed -r-2 and then -r-1
<wgrant> It be scanning.
<wgrant> spm: Can you check pg_stat_activity now?
<spm> been grabbing heaps the past min or so
<wgrant> Great.
<wgrant> Been going for a minute now, which is itself a good sign that it's hung.
<wgrant> Should normally take about 50s.
<spm> probably should have while looped, but anyway
<wgrant> yay, it has hung.
<spm> appears stuck in a SELECT
<wgrant> First time we've caught it in the act.
<wgrant> What is the SELECT?
<spm> it's a secret, that will be exposed on production of cake
<wgrant> Sneaky.
<spm> indeed. a cunning plan even
<spm> wgrant: https://pastebin.canonical.com/68304/
<spm> still at the same place nearly a minute later too
<wgrant> Aha
<wgrant> Thanks.
<wgrant> Should time out in 20s or so
<lifeless> so, code review + http://okfnlabs.org/annotator/demo/
<spm> still holding atm
<wgrant> spm: It's done, btw.
<wgrant> timed out, as expected.
<spm> and yes, cleared here as well
<wgrant> spm: any more interesting queries?
<spm> not for branchsscanner
<wgrant> I mean from before the SELECT of Doom.
<spm> I can paste you a rather "long" session history, if you're interested?
<wgrant> That might be handy, thanks.
<spm> <IDLE> in transaction <== a fair bit
<wgrant> Better than having to do this again.
<wgrant> Yeah
<spm> wgrant: https://pastebin.canonical.com/68305/
<wgrant> spm: Thanks!
<wgrant> It has traditionally been suspected that bzr slowness was the issue here, but this proves otherwise :)
<wgrant> Now, it doesn't actually tell us how it's being so slow, but at least it's something.
<spm> wgrant: I'd like to point out that your email address is in that query; QED, it's all your fault. <== advanced sysadmin debug.
<wgrant> That's good solid logic you have there :)
<spm> I'm glad you approve
<wgrant> Hmmm
<wgrant> So the branch has a full set of BranchRevisions, but no last_scanned_id
<wgrant> This is causing subsequent runs to try to clear the entire history.
<wgrant> But it doesn't explain what happened during the initial failure.
<wgrant> wallyworld_: How soon after pushing the branch did you create the MP?
<wallyworld_> wgrant: almost straight away
<wgrant> A saw a WIP MP pretty soon after the push
<wgrant> Looks like that is how to repro it.
<wgrant> I wonder if we're deadlocked with the MPJ runner.
<wallyworld_> plausible
<wgrant> 2 minutes so far...
<wgrant> That means we have two bugs.
<wgrant> Fail
<wgrant> Took 2.5 minutes, but it succeeded
<wallyworld_> branch revision has a lot to answer for
 * wallyworld_ -> school and back
<wgrant> wallyworld_: Why not do the grants after _reconcileAccess, so you can use APAs rather than having getPeopleWithoutAccess duplicate the pillar determination logic?
<StevenK> wallyworld_: So, QA? :-)
<StevenK> wgrant: Hmm, so you think MPJ and the scanner lock?
<StevenK> Plausible, indeed.
<wgrant> Discounted in this case, it seems.
<wgrant> The first logged MPJ was 12 minutes after the scan timed out.
<wgrant> wallyworld_: https://bugs.launchpad.net/launchpad/+bug/808930/comments/20 is the result of the debugging, btw.
<_mup_> Bug #808930: Timeout running branch scanner job <escalated> <linaro> <oops> <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/808930 >
<wallyworld_> wgrant: so we really want to delete BranchRevision :-)
<wgrant> wallyworld_: But it's my friend :(
<wallyworld_> wgrant: i'm not sure i follow your review suggestion above
<wallyworld_> i do call ensureAccessGrants after reconcileAccess
<wgrant> wallyworld_: Right, then you don't need to manually determine the pillars for bugs and branches.
<wgrant> You can just use the APAs
<wallyworld_> wgrant: so, i use IAccessPolicyArtifactSource.find() to fill out the policies_to_check array?
<wgrant> wallyworld_: AccessPolicyArtifact.artifact = abstract_artifact.id AND AccessPolicyGrant.policy = AccessPolicyArtifact.policy AND AccessPolicyGrant.grantee = TeamParticipation.team
<wgrant> Python never needs to see the policies.
<wgrant> They're purely used by the join from AccessArtifact to AccessPolicyGrant.
<wallyworld_> ah ok.
<lifeless> gary_poster: btw, trunk testrepository has a --concurrency option for you
<wgrant> wallyworld_: Oh, also, that whole thing is pretty strange. Your top-level query has "Person.id IN person_ids AND Person.id NOT IN policy_grantees AND Person.id NOT IN artifact_grantees", but then policy_grantees and artifact_grantees also filter on person_ids.
<wallyworld_> wgrant: i was attempting to limit the size of the in lists
<wallyworld_> the top level In(Person.id, person_ids) is because we only want people withotu access from those that are specified
<wgrant> wallyworld_: They're not lists, they're subquery expressions :)
<wgrant> It optimises through them.
<wallyworld_> sure, but the subsquery still returns results
<wgrant> Not really, no.
<wallyworld_> internally
<wgrant> Again, not really :)
<wgrant> Theoretically perhaps.
<wallyworld_> so if i limit the size of the subquery, it's more efficient
<wallyworld_> it is with oracle anyway
<wgrant> Not necessarily.
<wgrant> Also not necessarily.
<wgrant> It depends on how you're limiting the size.
<wgrant> So
<wgrant> I'd pull the TP join out.
<wgrant> There's no reason to have it all the way down there.
<wgrant> If I were postgres, I'd plan that as a nested loop, and the person_filter inside the subqueries would have no effect whatsoever.
<wallyworld_> ok, i'll take a look. maybe a run on dogfood to check ite performance is needed
<wgrant> Any reasonably query planner can see that.
<stub> The planner is smarter than you, except when it isn't.
<wgrant> In this case it should be :)
<wallyworld_> wgrant: this should be ok i think https://pastebin.canonical.com/68312/
<wallyworld_> except for flat maybe
<wallyworld_> yes, flat is gone
<wallyworld_> https://pastebin.canonical.com/68313/
<wgrant> wallyworld_: If I were writing it, I'd phrase it as the set difference between the requested people and the result of https://pastebin.canonical.com/68315/
<wgrant> But your new version looks correct.
<wgrant> (although the distincts on the subqueries are pointless)
<wallyworld_> wgrant: looks like your version returns the oppoiste of what is wanted?
<wgrant> wallyworld_: Right, positive queries are much easier to write directly.
<wgrant> Anyway, yours is OK if you drop the distinct=Trues
<wgrant> Mine is much faster, but we shouldn't have performance issues here anyway.
<wgrant> Since the dataset is small.
<lifeless> wgrant: distinct on subquery is a planner hint
<lifeless> wgrant: as is offset 0;
<lifeless> s/hint/sledgehammer/
<wallyworld_> wgrant:  i've removed the distincts, i can rework to match your logic
<wgrant> lifeless: Maybe in postgres 6.1
<wgrant> lifeless: But IN implies distinct on the subquery nowadays
<wgrant> Plus we have no reason to use planner deception techniques here.
<wgrant> You should use them once it's shown that you need them; not before :)
<wgrant> wallyworld_: The rework's not critical, but I wonder if it makes more sense as a getPeopleWithAccess anyway
<wgrant> wallyworld_: eg. for determining who gets notified.
<wgrant> (for later structural subscription enablement)
<wallyworld_> wgrant: we can add a method for that, but i'd rather use the db for now to find people without access
<wgrant> You're probably better off having a single method and doing integral set difference in Python.
<wallyworld_> i used distinct onthe sub queries from prior experience, but whio know what postgres dowesa
<wallyworld_> wgrant: i was trying to avoid anyting in python
<wgrant> It's almost always prettier to do set difference in Python than it is to describe non-trivial negative assertions in SQL :)
<wgrant> Perhaps.
<wgrant> StevenK: Going to qa-ok your db-stable rev?
<StevenK> I got distracted by auditor
<StevenK> wgrant: db-branch-nap-security => db-devel        [OK]       (up for 3:29:40) i-b2f88acb
<wgrant> That's encouraging.
<wallyworld_> wgrant: are you ok to +1 the mp and i'll land after i've had a look at rejigging the query to you your logic? i'd still like to keep everything in the db for now
<wgrant> Just +1ing now, yep.
<wallyworld_> and we can pull out common query expr for notofications later if needed
<wallyworld_> thanks
<wgrant> If loggerhead wakes up
<wgrant> There we go
<wgrant> Thanks.
<wallyworld_> wgrant: thanks. btw, the bloody bug subscription removal  job still oopes on qas :-(
<wgrant> wallyworld_: No idea why?
<wallyworld_> error reading 'infoformation types' from job metadata
<wgrant> wut
<wgrant> examining log
<wallyworld_> yeah, haven't looked too much nto it
<wallyworld_> the should should only be able to be constructed such that metadata is written with info type key
<wgrant> Oh
<wgrant> What if it's old data?
<wgrant> From before the jobs were merged.
<wallyworld_> ah, that is it i think
<wgrant> It's certainly what it looks like.
<wallyworld_> so, i'll do a branch to make the code robnust
<wgrant> Mmm
<wgrant> No
<wgrant> It shouldn't be robust against information_types being missing.
<wgrant> Oh, if it's in getOperationDescription, hm, maybe.
<wallyworld_> it's trivial to do though
<wgrant> It is, yeah
<wallyworld_> missing = []
<wallyworld_> so close to being able to turn all this shit on
<wgrant> Heh
<wgrant> Yeah
<wallyworld_> finally
<wgrant> StevenK: fdt requested
<StevenK> \o/
<StevenK> wgrant: What shall we do with this security.cfg branch, then?
<wgrant> StevenK: It'll be landable on db-devel before this is deployed.
<wgrant> Land it on db-devel before this is deployed.
<wgrant> We can then merge db-devel instead of the relevant rev of db-stable.
<wgrant> Since nothing further of consequence has landed on db-devel to stop us from doing that.
<wgrant> Keeps the MP happy.
<StevenK> wgrant: Sounds like a good plan.
<StevenK> wgrant: db-devel r11688
<adeuring> good morning
<wgrant> StevenK: Thanks.
<lifeless> wgrant: in 8.4 distinct made a different in subqueries, FWIW. Haven't tested in 9.1 yet.
<czajkowski> jml: have to ask, what headset do you use and do you recommend it ?
<jml> czajkowski: I use my Bose QC3 headphones with a mic cable. I've successfully used Logitech USB headsets in the past.
<jml> czajkowski: ultimately, what matters is that it's well-tested and that it doesn't leak noise into the mic.
<jml> perhaps it's inevitable when you have > N people, but it always seems as if someone is on a combination of hardware/software that they are only using for the first time that day.
<jam> wgrant: I'm planning on re starting the fix and then copy translations for Q today. I was hoping to get more insight into how it failed on friday. In case it is more of a systemic issue.
<jam> What process was it that you thought was adding data? (User filling out translations for P and that somehow ending up in Q?)
<wgrant> jam: Translations are shared between series.
<wgrant> So a translation in Precise can show up in Quantal.
<wgrant> Or even a translation in an upstream project.
<jam> wgrant: so I understand that they can be shared, and that is my understanding of why we are running a 'copy' script in the first place. But I'm trying to understand how/what decides to put a Q entry in when submitting a P entry.
<wgrant> jam: The normal Translations model code in the webapp.
<jam> wgrant: I'm poking through the code base, and I'm noticing a garbo script that touches the PO tables. Is that something we should be stopping along with the cron scripts?
<jam> The script is ScrubPOFileTranslator
<jam> anyone else able to answer how and when garbo runs and how likely it is to be inserting data into the db for a table we're trying to clean up?
<wgrant> jam: That's in garbo-daily.
<wgrant> It was introduced by jtv recently as part of this work.
<wgrant> It maintains POFileTranslator, both adding and deleting
<jam> wgrant: right, it was something he wanted to move into a refresh-the-db stuff so that it doesn't block the copy stuff
<jam> but I don't know if we need to deactivate it, etc.
<wgrant> 5 23 * * * $LP_PY /srv/launchpad.net/production/launchpad/cronscripts/garbo-daily.py --abort-script=72000 -q --log-file=INFO:/srv/launchpad.net/production-logs/garbo-daily.log
<wgrant> It died with fastdowntime, so it is not a concern.
<wgrant> And it would have been dead on Friday too.
<jam> wgrant: died with fastdowntime?
<jam> as in, garbo-daily isn't working correctly?
<jam> right now
<wgrant> jam: fastdowntime killed its DB connection, as it was meant to.
<wgrant> It's working fine, but not running atm since we killed it 72 minutes ago
<jam> ah, and so won't be running again until tomorrow/tonight at 23:05
<jam> UTC?
<wgrant> Right.
<czajkowski> jelmer: vila mgz could one of you look at https://answers.launchpad.net/launchpad/+question/200576  please.
<jelmer> czajkowski: on it
<czajkowski> jelmer: lovely thanks
<czajkowski> am looking forward to reading that fo my reference
<jam> wgrant: another small thing (hopefully), we were getting email spam while the cron jobs were stopped. Is that something we can supress since we know that we manually stopped them?
<wgrant> jam: You'll need to ask ops to disable the relevant script-monitor cronjobs.
<czajkowski> cjwatson: on the installer does it ask you which drive to install Ubuntu on ?
<jamestunnicliffe> Hi, since I don't have commit access but I have an approved branch, could someone kick off the test + check in cycle on this please: https://code.launchpad.net/~dooferlad/launchpad/workitems-bugfix/+merge/110563
<jam> wgrant: so the table that was getting new data was POFileTranslator (IIRC). However, I see this in the db:
<jam> https://pastebin.canonical.com/68246/
<jam> sorry bad paste
<jam> http://paste.ubuntu.com/1047080/
<jam> so why is new data being inserted there if it is considered 5-year old legac
<jam> legacy.
<czajkowski> jamestunnicliffe: not sure who does that is it a reviewer
<cjwatson> czajkowski: You should get to choose in the event that you have multiple drives, last I checked, yes
<jam> maybe pofiletranslator is still new, but the others are not?
<czajkowski> cjwatson: that's what I thought, but do kinda feel bad for https://answers.launchpad.net/ubuntu/+question/200739
<czajkowski> cjwatson: last time I just installed ubuntu and wiped the machine when I bought it.
<jam> yeah, that looks right. As it adds a function that does pretty much the same thing, to update POFileTranslator, but doing so from different source tables.
<wgrant> jam: Huh?
<wgrant> jam: That's dropping two functions to work with old tables.
<jam> wgrant: I was misreading a comment. It was saying a table was legacy.
<wgrant> Right.
<jam> I thought it was saying POFileTranslator
<jam> was the legacy table
<jam> but it was saying the other ones were
<wgrant> Right, no :)
<wgrant> jam: Anyway, I'd just try again
<wgrant> See what happens.
<jam> yeah, I'm still waiting on ops, so I figured I'd try to at least understand how things are working together.
<wgrant> To be clear, Translations is by far my weakest point.
<wgrant> I don't really know how it works.
<wgrant> Only very vaguely.
<jam> If a db table is getting updates while doing COPY that seems a good way for copy to break
<jam> It may not be the specific problem we've been running into.
<jam> But it was the problem we ran into on Friday.
<jam> for the delete step
<wgrant> Not really.
<wgrant> That was just deleting some stuff which violated FK references.
<wgrant> Unrelated to the COPY.
<jam> wgrant: we got an insert into POFileTranslator *after* we deleted everything.
<wgrant> Sure.
<jam> Which isn't related to copy
<jam> but it means something was happening concurrently
<wgrant> 21:33:40 < jam> But it was the problem we ran into on Friday.
<wgrant> Right, people still translate through the web UI
<jam> wgrant: right, we didn't get into copy on Friday.
<wgrant> That's the concurrency.
<wgrant> Ah
<jam> my concern is that copy has been failing in the last couple of attempts,
<jam> And I was trying to understand where the moving parts are
<jam> in case there was an obvious trip up point.
<jam> I could see copy failing if it expected something to be empty
<wgrant> Has the copy been reached since the big changes that jtv did?
<jam> no
<wgrant> I thought that was in the second script.
<wgrant> Right.
<wgrant> So, ignore that, it should be quick enough now hopefully.
<wgrant> First step is to get the deletion working.
<jam> well, he thought delete should be small minutes, and at 40min it failed because of concurrent updates.
<wgrant> And only debug it if it's broken :)
<jam> and he isn't here for me to chat with today.
<jam> that was my original goal
<wgrant> This run should be a little quicker.
<wgrant> Since it's already done most of the work.
<jam> it deleted only about 40k rows, from POFIle, and I don't have a feeling for how many there should be.
<jam> is it possibel to get that info from staging?
<jam> or are those tables not copied over
<wgrant> Didn't it get all the way through the first three phases, then fail on the fourth?
<wgrant> The entire database is copied over.
<jam> wgrant: there are 7 'statements' it runs via batches.
<jam> it got to 3
<jam> 'delete_pofile'
<jam> and got through 40k rows of that one
<jam> I personally don't have a feel for how much data is there.
<wgrant> Let me see.
<jam> wgrant: its my understanding that some devs have direct SQL access on staging. It would appear that you might be one of them :)
<wgrant> No
<wgrant> I have access to dogfood
<jam> ah
<wgrant> Which is less up to date
<lifeless> jam: as a teamlead you should get ro access to stagng and qastaging
<wgrant> and about a billion times slower
<lifeless> jam: you may need to request that via RT
<jam> lifeless: and somehow figure out how to use it :)
<lifeless> jam: thats the easy part
<lifeless> night all
<wgrant> lifeless: Why can't mortals have it too? :(
<jam> lifeless: night
<jam> isn't there a specific rt address you are supposed to file for launchpad-ish issues?
<jam> rather than just rt@
<wgrant> jam: There's about 100k
<wgrant> jam: launchpad@rt.canonical.com
<jam> wgrant: so about halfway through that one. and then however many for the other tables
<wgrant> jam: Right. But that should be the only widely referenced one.
<jam> translationtemplateitem, packagingjob, translationimportqueueentry_potemplate, potemplate
<jam> sure
<wgrant> As long as the POFiles go, the POTemplates should have no trouble leaving.
<wgrant> And the others are fine.
<jam> wgrant: do you think the 40 min was a db load thing? (Anything to indicate db load was particularly high/low on Fri afternoon?)
<wgrant> jam: I thought it was expected to be 30Â±20min
<wgrant> The copy afterwards should be <10min AIUI
<jam> well, I believe jtv said "minutes" but to me that is <30min
<jam> you could argue it is just <1hr
<wgrant> Oh, I had them the wrong way around.
<wgrant> The fix script has never been used before AIUI
<wgrant> So i's unsurprising that we don't know how long it takes.
<jam> I'm a bit surprised that DBLoopTuner says it can only delete 20rows in 2s.
<jam> https://pastebin.canonical.com/68241/
<jam> (goal_seconds was 2.0)
<jam> speaking of DB stuff, hey stub
<wgrant> Well
<stub> ...and his amazing bouncing 'net connection
<wgrant> The queries probably aren't exactly well-optimised.
<wgrant> There are some multi-table joins in the where clauses of those deletes.
<wgrant> SOme of the FKs are also unindexed.
<wgrant> eg. poexportrequest.pofil
<wgrant> But that table should be small.
<wgrant> However, the join is slow.
<wgrant> I suspect that's why it takes so long.
<wgrant> On DF it chooses a merge join.
<wgrant> Which is entirely ridiculous.
<wgrant> But it can't be doing that on prod; it would be even slower then.
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2
<gary_poster> I saw on the bugreport that you had added it.  Thank you lifeless!
<jml> gary_poster: hi
<gary_poster> hey jml!  I was just considering procrastinating from my other tasks by replying to your email. :-)
<jml> gary_poster: out of curiosity, is there a hit-list of bugs that have to be fixed before we can all enter into 40 min test time nirvana?
 * jml almost remembers when it was 40 minutes even without parallelization. I think it was 50m when I started. 
<jml> gary_poster: it just strikes me that if ever I were to do some spontaneous Launchpad hacking, that would be the single most selfish thing I could do.
<jml> (or lazy or whatever vice pundits are parading as a virtue these days)
 * jml is also procrastinating
<gary_poster> jml, There is a paralleltest tag, but I use the kanban board for my daily view so I'm not sure how accurate it is.  We are at a relatively small number of actual bugs.  Of the ones on my board, bug 1013921 is a nice to have (should increase speed by about 5 minutes) and is up your alley; investigating the fact that bug 994752 didn't really give us the stability we needed is not yet cooked enough.  The main thing we
<gary_poster> are waiting on now is getting the machines ready
<_mup_> Bug #1013921: our zope.testing fork needs to emit subunit time immediately before test start and immediately before test completion <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1013921 >
<_mup_> Bug #994752: lxc-start-ephemeral's use of dhcp lease table is fragile <patch> <verification-done> <lxc (Ubuntu):Fix Released> <lxc (Ubuntu Precise):Fix Released by stgraber> <lxc (Ubuntu Quantal):Fix Released> < https://launchpad.net/bugs/994752 >
<gary_poster> that is, purchasing the new two 24 core machines for the data center
<jml> wow.
<jml> it's such a pity about the fork. :(
<jml> I'm sorry for my part in that.
<gary_poster> I'm not sure you had much to do with it, if any.  The big problem is that we never want to pay the cost for a full zope package upgrade
<gary_poster> At least, that's my guess in retrospect...
<jml> every time I think about it I think, "We should stop using zope.testing. That means we must stop using layers. OK, let's start with the librarian layer. Ugh, that has some problem that lifeless tells me that I always forget about."
<jml> and then give up
<gary_poster> heh
<gary_poster> and then I think "oh well, if we ever get to SOA it will SOLVE THE WORLD!"
<jml> well, I have an opportunistic todo for incrementally approaching that
<jml> some of the ugly monkey patches to zope.testing are mine, that makes upgrade harder
<jml> also I think I might have done one of the first local patches. once you start down that path it's very hard to get back to the main road.
<gary_poster> it's true that it is hard.  And yeah, once we got to p4, it seemed less like a temporary situation and more like a road.  We're on...p13 now, thanks to us. :-/
<gary_poster> Some of which includes our hilarity such as
<gary_poster> # p11 reverts p9.
<gary_poster> # p12 reverts p11, restoring p9.
<gary_poster> whee! :-)
<gary_poster> I was directly involved in that, so I think I am allowed to giggle
<jml> heh :)
<jml> There's also a sense in which zope forked away from us too.
<gary_poster> yeah
<gary_poster> it dying doesn't help anything either
<jml> I can just imagine someone rubbing their hands and thinking, "At last! Our plot to alienate the last of our users is complete!"
<jml> And then doing a wicked opera laugh.
<gary_poster> lol
<jml> fwiw, I did this as a braindump of things to have in a new project: https://docs.google.com/a/canonical.com/document/d/1A8qTOYZd7p9dhjmlnnEKW46h38ydt9yNbmfgX3yhPGw/edit â it's not a tested checklist, and it's depressingly long & in need of automation. I just started lp:libdep-service, so I'll probably use that to test this
<gary_poster> jml, I like that list--"one other core contributor" in particular as a good insight
 * gary_poster looks up libdep-service
<gary_poster> ah cool
<jml> it's nothing special, just another microservice.
<czajkowski> cjwatson: can I assign you https://answers.launchpad.net/launchpad/+question/200735  or can you let me knowthe info that needs to go to that page please.
<gary_poster> matsubara, hey.  I'm going to reboot to see if my camera works again then, and then I'll be ready
<matsubara> gary_poster, sure, I don't have a camera, so if you want to use phone only it's fine by me
<gary_poster> matsubara, oh ok, cool
<rick_h> ivory: https://plus.google.com/hangouts/_/f2e9cf4efd2664584bc55d299ce1ff8075141388?authuser=0&hl=en-US
<gary_poster> we can be blue heads together then
<matsubara> heh
<matsubara> ok, let me create the hang out
<matsubara> gary_poster, https://plus.google.com/hangouts/_/bbe5d4266ba176d0f201f8eed73db4ac6b50640a?authuser=0&hl=en
<dobey> from second life to google hangoutsâ¦
<cjwatson> czajkowski: Yeah, you can assign that to me
<czajkowski> cjwatson: cheers
<gary_poster> matsubara, http://ec2-23-20-214-51.compute-1.amazonaws.com:8010/waterfall
<nigelb> wtf. gmail just marked all of lifeless' emails as spam. :/
<jml> heh heh.
<jam> dpm: the copy translations scripts has run, need to verify the results now
<dpm> jam, cool :)
<nigelb> jml: I've been wondering why lifeless hasn't been sending emails lately, heh.
<cjwatson> czajkowski: done
<czajkowski> cjwatson: oh if they were all only that simple today
<cjwatson> czajkowski: Simple for you, anyway ;-)
<czajkowski> cjwatson: I'm having a feel sorry for the users day today
<czajkowski> issues with Ubuntu is getting logged in lp
<czajkowski> I am intrigued how they get as far as creating a LP ac, filing a bug or a question, ticking Ubutu -> LP and not just leaving it as Ubuntu
<jam> dpm: I see https://translations.launchpad.net/ubuntu/quantal is available, I won't have the time today to verify anything, but I'll look into that tomorrow.
<dpm> jam, thanks. I'm a bit busy today too, but I'll look at it tomorrow too.
<jml> cjwatson: https://code.launchpad.net/~jml/lazr.restfulclient/disable-ssl-cert-verification/+merge/110503 ; https://code.launchpad.net/~jml/launchpadlib/ssl-creds/+merge/110848
<cjwatson> jml: I *think* those are all the sites I found myself having to modify, yes.  LGTM
<jml> cjwatson: I managed a successful API call to dev with those.
<jml> cjwatson: thanks.
<cjwatson> (But I'm not an LP reviewer)
<jml> benji: https://code.launchpad.net/~jml/lazr.restfulclient/disable-ssl-cert-verification/+merge/110503 ; https://code.launchpad.net/~jml/launchpadlib/ssl-creds/+merge/110848
<cjwatson> IIRC it's worth checking that it also works the second time round, with cached credentials.
<jml> cjwatson: ah, thanks.
<jml> all good.
<jml> benji: ta
<benji> jml: np
<cjwatson> wgrant: For PCJ unembargo support, I'm taking the approach of pushing the update_files_privacy work down to _do_direct_copy "if not archive.private and has_restricted_files(source)", and adding explicit unembargo flags to a couple of places (Archive.copyPackage and the PackageCopier script) to avoid people unembargoing private files by accident; then for the time being making UnembargoSecurityPackage a trivial shell ...
<cjwatson> ... around PackageCopier.  Does that sound about right as an initial approach?
<cjwatson> So PlainPCJ will unembargo files happily, but only if it has an appropriate flag set in its metadata.
<cjwatson> This seems to be passing the initial set of plausible tests I can think of, although I'm waiting for 'bin/test -vvct soyuz' to run.
<cjwatson> I suspect that delayed jobs need to be ripped out in a separate landing, just in case there are pending delayed jobs when a rollout happens.
<cjwatson> Er, delayed copies.
<rick_h_> sinzui: <3 the sprite stuff.
<sinzui> rick_h_, thank you. I noticed you playing with awesomefont. Its CSS rules for icons are very similar to the rules I wrote for sprites...inline-block + text-indent to control what is seen
<rick_h_> sinzui: right, works out pretty well so far
<rick_h_> having a FF issue with awesomefont atm, but hopefully once I work it out it'll be a nice way to go
<rick_h_> talked with some dev friends also kind of neat to think that in the high-res world coming they'll work on smoothly hopefully
<sinzui> yes. I have no experience with that issue.
<rick_h_> yea, I don't think most of us have yet, only really been in the mobile space
<sinzui> I last worked with embedded on in 1999.
<sinzui> rick_h_, oh, are you certain that firefox really understands the type? Does the server have to know the mime-type?
<rick_h_> sinzui: yea, I'm not sure what it is. The error is generic that firefox's font-sanitizer rejected the font
<rick_h_> so it might be a header from the server, not sure
<rick_h_> was hoping someone from FF land would see/point me out
<rick_h_> tried finding the code/source behing the sanitizer to see what kind of rules it enforces
<rick_h_> but yea, with the 'retina' macbook, high-res assets in sites is going to get more and more important I think
<sinzui> I believe dobey have some deep knowledge of browsers
<dobey> ?
<rick_h_> already a ton of work around sizing of images/etc around responsive and <3 the ability to size icons via css font-size for that
<rick_h_> dobey: oh, we were just discussin I was having some FF issues with an icon-font due to their font-sanitizer rejecting my font
<rick_h_> discussing, ugh mondays
<dobey> oh
<dobey> html VMs
<rick_h_> looks like someone's starting to help http://stackoverflow.com/questions/11072655/firefox-font-face-fail-with-fontawesome/11085126
<rick_h_> but anyway, might be something interesting for our type of sites as things go forward. sinzui has been doing some awesome work cleaning up our sprite usages and it resembles using an icon font
<sinzui> huwshimi has been suspiciously quiet for weeks. He was planed to make the sprites greyscale and colour on hover/activate. A font would be much easier to maintain.
<rick_h_> well I know we were keeping him busy some on our side for a bit
<czajkowski> gary_poster: loving the retrospective mins of yellow team meetings
<gary_poster> thank you czajkowski!  It helps me write the darn things to get positive feedback :-)
<czajkowski> gary_poster: I can almost get the tone and accent in my head when reading it
<gary_poster> czajkowski, lol
<czajkowski> it's nice to view meetings postively and in a fun interactive manner
<czajkowski> gary_poster: want me to put them over on the LP FB and G+ pages to show others what we're doing
<czajkowski> was in the new office today adn all I got was wouldnt it be nice if LP did this, and my answer is yes it would be but the squads are doing X,Y,z
<gary_poster> czajkowski, uh.  The secret is that I'm really kind of shy, so my initial internal reaction is "augh!".  Even though I was an opera singer in a former life.  But feel free if you think it would help anything. :-)
<lifeless> gary_poster: hiya
<lifeless> gary_poster: dunno if you saw in scrollback, I've added a knob for concurrency for test
<gary_poster> hey lifeless.  thanks for the two parachutes you sent
<lifeless> r
<lifeless> gary_poster: ah you did, cool.
<gary_poster> yeah, definitely, and also the hint for the zope.testing thing
<gary_poster> (the subunit times)
<lifeless> gary_poster: oh right
<lifeless> gary_poster: yeah, useful tool - testr uses it itself to timestamp incoming streams in case they are not timestamped.
<lifeless> gary_poster: so you get socket latency effects, but better than round robin.
<gary_poster> cool, lifeless.  we may stick the times directly in the stream but if that's problematic at all we'll investigate throwing the adapter in.  (I don't expect it to be problematic at all, but who knows.)  What you said though...that sounds cool, but so wouldn't testr be applying that to the parallelized streams?  Or maybe it's not in there yet for that use case?
<gary_poster> you don't have to dig in now, just was curious
<czajkowski> gary_poster: done thanks :)
<gary_poster> czajkowski, :-) cool
<lifeless> gary_poster: timing at source and sink is fine
<lifeless> gary_poster: the decorator shuts up if a stream is timestamped already
<gary_poster> ah!
<gary_poster> so it sees any time and says "I'm outta here"
<gary_poster> makes sense
<lifeless> close :)
<gary_poster> :-)
<lifeless> it has the idea of a clock, and if its unset, anytime time is needed, it asks the system
<lifeless> if its set, it uses what it was last set to
<lifeless> and a stream with times in it has the effect of setting its clock.
<gary_poster> ah!
<gary_poster> yes
<gary_poster> I saw those entered into the stream by testr
<gary_poster> but since the clock is only advanced at not particularly helpful times..
<gary_poster> the adapter cannot put particularly helpful times back in
<gary_poster> btw, lifeless, the clock can go backwards, right?
<lifeless> testrepository/repository/__init__.py line 84 for reference.
<lifeless> yes, the clock can go backwards (a necessity with multiplexing)
<gary_poster> right
<gary_poster> cool
<gary_poster> ok I need to restart to see if my webcam will work then :-P
<gary_poster> thanks and biab
<lifeless> uhm, so get_inserter is whats doing the timing
<lifeless> gary_poster: found something interesting, replied to the bug
<lifeless> gary_poster: the testr auto-timing location is deep in the pipeline (after the mux stage), so is going to be producing garbage atm.
* gary_poster changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<sinzui> jcsackett, mumble?
<jcsackett> sinzui: omw.
<wgrant> cjwatson: That sounds good.
<wgrant> cjwatson: Delayed copies should be ripped out in a separate landing anyway.
<wgrant> cjwatson: Let's not conflate changes that shouldn't be conflated.
#launchpad-dev 2012-06-19
<wallyworld_> wgrant: StevenK: it seems IBugSet.createBugWithoutTarget() is only used internally by createBug(), it doesn't seem to be exported, there is a doctest for it. i'd like to remove the method from IBugSet. thoughts?
<cjwatson> wgrant: Right.  Now if only I can get the last couple of tests passing ...
<wgrant> cjwatson: What's broken?
<wgrant> wallyworld_: I believe that used to be used by the email interface, but it's not any more. Destroy.
<wgrant> wallyworld_: I deliberated avoided removing it last month because the email interface uses it.
<wallyworld_> wgrant: yay, will allow me to simplify the AAG branch and fix failing tests
<cjwatson> doctest hatred of some kind in xx-archive and xx-ppa-files.  (Not in front of the laptop right now - I'll have another crack at it tomorrow)
<wgrant> But now you've made me check and it doesn't actually use it any more.
<wgrant> cjwatson: Ah, right.
<cjwatson> Maybe it was too ambitious to change the default for allow_delayed_copies.
<wgrant> cjwatson: I imagine the diff should be fairly small?
<cjwatson> Not horrible for what it does.  Several layers of tentacles.
<cjwatson> Since the new unembargo option needs to go all the way up to the webservice layer.
<wgrant> Yeah
<cjwatson> wgrant_: https://code.launchpad.net/~cjwatson/launchpad/change-override-forbid-release-stable/+merge/110942 is that changeOverride bug-fix you asked for
<lifeless> cjwatson: thanks for the 10GB reply
<lifeless> cjwatson: the alternative to debdelta is !debsatall, which isn't something we've had much discussion on.
<cjwatson> I confess I've never heard of it.
<cjwatson> No relevant Google hits.
<cjwatson> I think the short answer on debdelta is that it would be done by now if only mvo had any spare time.
<lifeless> cjwatson: I meant, not using debs, not a specific thing :) Unless of course you got that and ran with it :>
<cjwatson> So sticking somebody reasonably clued on it for a couple of months ought to be enough to get it thrashed out.
<cjwatson> Oh I see.
<cjwatson> debdelta will probably be quicker, and will have widespread benefits if we get it done.
<wgrant_> cjwatson: canUploadToPocket and checkLegalPocket seem pretty similar to cannotModifySuite
<wgrant_> More specifically, they all duplicate the Release-is-immutable-in-stable-series logic.
<lifeless> cjwatson: sure. I'm a little worried about failure modes like 'user edited a file'
<lifeless> cjwatson: which something a little more dynamic in nature could deal with rather more gracefully than we can today.
<cjwatson> wgrant_: Yes, I wasn't sure how far I should go in refactoring that swamp
<wgrant_> cjwatson: That's putting it nicely.
<cjwatson> wgrant_: But I take it you want to go a bit further?
<wgrant_> cjwatson: I'd extend canUploadToPocket to do this. But you don't have to if you don't want.
<cjwatson> In particular I wasn't sure whether it would be a good idea to change the behaviour of the publisher to avoid touching anything that canUploadToPocket refuses.
<wgrant_> It's less obvious that it's correct for the publisher to do that.
<wgrant_> I think it probably is correct for changeOverride to use canUploadToPocket, however.
<cjwatson> checkLegalPocket is semantically closer, but has a less near to being helpful interface.
<wgrant_> canUploadToPocket's name is just crap
<wgrant_> checkLegalPocket is a publisher-specific method which should be somewhere else and implemented in terms of canUploadToPocket
<cjwatson> OK.  I'm happy to rearrange it in terms of canUploadToPocket.
<wgrant_> But it looks fine apart from that, so if you don't care enough to fix it I will +1
<cjwatson> Nah, I'd rather get it as right as possible first time
<lifeless> worth a read - http://shirky.com/writings/group_enemy.html
<mwhudson> lifeless: the middle section makes me think of time
<mwhudson> * daker has quit (Ping timeout: 246 seconds)
<mwhudson> oh ffs, twice today!
<mwhudson> lifeless: http://xkcd.com/1043/
<lifeless> mwhudson: LOL
<mwhudson> there is something that's changed in precise (i think inside ff) that really screws up my clipboard experience
<lifeless> mwhudson: do you have a three button trackpad ?
<lifeless> or hmm, three button nipple ?
<lifeless> bigjools: wallyworld_: HUSH.
<mwhudson> lifeless: i do, but i use an external keyboard & mouse almost all of the time
<lifeless> ok, cause I find the middle button of the three connected to the nipple extremely frustrating to use
<wallyworld_> huh? me has said nothing
<mwhudson> (i certainly can't blame either of today's stuffups on the thinkpad)
<lifeless> wallyworld_: no, but you usually do :P
<bigjools> nipple is the correct term
<wallyworld_> ah, guilty by assumption :-)
<bigjools> just don't flick your tongue over it
<wallyworld_> lol, knew i could proxy to bigjools :-)
<lifeless> see
<lifeless> exactly my point
<wgrant> wallyworld_: Are you sure that new query is good?
<wgrant> It looks broken to me.
<wallyworld_> in what way? the tests pass
<wgrant> It seem like it'll find people with *any* team participation that doesn't have permission.
<wgrant> Rather than finding people where *all* team participations don't have permission.
<mwhudson> lifeless: it's an interesting article, i've read it at least once before but that was before "facebook" and "twitter" were words that appeared on the 6pm news...
<wallyworld_> wgrant: let me look
<cjwatson> wgrant: Happier with this try?  It's rather more verbose, but gets rid of duplicate logic in several places now.
<cjwatson> (E.g. it rephrases checkLegalPocket mostly in terms of canUploadToPocket.)
<wallyworld_> wgrant: at first read it looks ok - not(a or b) = not(a) and not(b)
<wgrant> wallyworld_: Sure, that bit of the logic is fine.
<wgrant> wallyworld_: But you're joining against TP
<wgrant> You want to find people where all the teamparticipations match the OTs.
<wgrant> s/OTs/NOTs/
<wgrant> But that query will find people where any of them do.
<wallyworld_> yes, that seem true
<wgrant> eg. if foo is a member of bar and baz, and bar has access but baz does not, the query will say foo doesn't have access.
<wallyworld_> so i need a extra constraint
<wgrant> a GROUP BY with HAVING(COUNT(teamparticipation) = 0) is probably clearest
<wgrant> But it's horrid
<wgrant> Which is why I suggested inverting the method and doing the difference in Python
<wgrant> It's more useful and far simpler.
<wallyworld_> but it's python and the results could be large, they may not be, but we can't guarantee that
<wgrant> wallyworld_: Not really.
<wgrant> We're passing in a set of people
<wgrant> The set we get back out will be at most that large.
<wgrant> And it's probably only a set of ints.
<wallyworld_> ok, i can rework it
<wgrant> cjwatson: Looking, sorry
<wgrant> wallyworld_: Unless you want to do a set of horrid left joins with group by and having, that's probably best.
<wgrant> But you can do it all in SQL if you really wanr.
<wgrant> cjwatson: Should cannotModifySuite be replaced too?
<wgrant> If you're feeling adventurous, canUploadToPocket would be even better as Archive.canModifySuite
<wgrant> cjwatson: But that's an excellent improvement, thanks.
<cjwatson> wgrant: cannotModifySuite> Doesn't that go back to the conversation above?
<cjwatson> 02:49 <cjwatson> In particular I wasn't sure whether it would be a good idea to change the behaviour of the publisher to avoid touching anything that canUploadToPocket refuses.
<cjwatson> 02:49 <wgrant_> It's less obvious that it's correct for the publisher to do that.
<cjwatson> I'll look at refactoring it into Archive.canModifySuite tomorrow
<wgrant> cjwatson: That's a good point, ignore me.
 * wallyworld_ sighs. i may triggered the branch scanning bug again :-(
<wgrant> cjwatson: But moving the archive-related stuff onto Archive is a good thing; should have been done during ArchiveRework in 2006 :)
<wgrant> So please move that if you have a moment.
<wgrant> wallyworld_: Was an MP involved again?
<wallyworld_> yep
<wgrant> :(
<wgrant> Missed the first run
<wgrant> Was hoping we could actually see what was going on there.
<wgrant> spm: Not at lunch, are you?
<spm> am back now, but about to be exclusively doing stuff
<wgrant> Ah, k
<wgrant> Nevermind, then.
 * wallyworld_ taps fingers. branch scanning seems to be taking forever
<wgrant> wallyworld_: It just retimed out a few seconds ago
<wgrant> Your new push should process in a sec
<wallyworld_> hope so
<wgrant> It's scanning
<wgrant> Done
<wallyworld_> branch pages doesn't seem to think so
<wallyworld_> ah now it does
<wgrant> Hm
<wgrant> It's rescanning the original one now, for the third time.
<wgrant> Did you repush it?
<wallyworld_> nope
<wallyworld_> wgrant: if you could peek at this, much appreciated https://code.launchpad.net/~wallyworld/launchpad/create-aag-privacy-transitions-followup2/+merge/110945
<wgrant> wallyworld_: Looking
<wallyworld_> thanks
<wgrant> wallyworld_: r=me, thanks.
<wgrant> Although neither the MP nor commit message nor any of the other commit messages actually say what the issue is :)
<wallyworld_> wgrant: thanks for spotting the issue
<cjwatson> wgrant: All right, since sleep continues to elude, I've done that move now - and landing, since I see you approved it already (thanks)
<wgrant> cjwatson: Marvellous.
<wgrant> Apart from the not sleeping thing.
 * StevenK headdesks.
<wgrant> ?
<StevenK> wgrant: I ran make schema and was then going to run the garbo tests again and managed to fat finger make clean && make schema somehow.
<wgrant> Heh
<wgrant> TDD on LP really sucks
<wgrant> Just saying
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/populate-branch-ap/+merge/110950
<lifeless> EWTF 7878.23s  OOPS-ebfdc5eaa4642ee738b09283e0964a82  BranchSet:CollectionResource:#branches
<lifeless> wgrant: have you looked into https://oops.canonical.com/?oopsid=OOPS-f08635d8a0548f5b1bafb1bb0209c980#repeatedstatements ?
<wgrant> lifeless: Heh
<wgrant> I didn't think it would ever get that bad.
<czajkowski> morning
<adeuring> good morning
<lifeless> wgrant: and yet...
<stub> Add one query per request, and 5 tests fail. We have some sensitive query counts in there...
<stub> Bump 'em by one, and they will be less sensitive when 1 request ends up being removed from every request in a future landing.
<cjwatson> Make them equality tests rather than maxima?
<jam> dpm, jtv: I'm looking over the Q translations, I'm running into a couple things. The #contributors doesn't match between Q and P, but maybe that is the Garbo job, or maybe just newer stuff?
<jam> I'm getting timeouts trying to load the actual page, but it does eventually load.
<jam> for example this: https://translations.launchpad.net/ubuntu/precise/+source/gfxboot-theme-ubuntu/+pots/bootloader/es/+translate vs https://translations.launchpad.net/ubuntu/quantal/+source/gfxboot-theme-ubuntu/+pots/bootloader/es/+translate
<jam> The checkboxes are also a bit different.
<jam> On P it defaults to 'New translation', but on Q it is marked on "Current Spanish".
<jam> Though aside from the first, the pages seem identical.
<jtv> jam: otp
<jam> jtv: np
<jam> so aside from the Contributors stuff, all the pages I've looked at for specific translations have been identical between P and Q
<jtv> jam: back.  The contributors are updated by the garbo job, right.
<jam> jtv: so on this page: https://translations.launchpad.net/ubuntu/quantal it seems to say there are 237045 untranslated strings for Bosnian, but when I go: https://translations.launchpad.net/ubuntu/quantal/+lang/bs it only lists 37371. Maybe I'm not looking at the right page?
<jtv> jam: the checkboxes sound strangeâ¦ it may be an unanticipated side effect of Quantal translations still being closed.  The important thing is that you get identical items between P and Q, both labelled Current Spanish, with the same text.
<jtv> The statistics are also updated asynchronously, so don't rely on those.\
<jam> jtv: yeah, the checkbox seems to be if Firefox puts the cursor inside the text entry box
 * jtv looks at pages
<jam> jtv: how asyncrhonously? Given the copy happened last night
<jtv> !
<jam> jtv: we were able to get the copy to succeed yesterday around 14:00 utc
<jtv> Actually I guessed from what you said earlier, but it's nice to have that part be such a non-event.
<jtv> How long did it take?
<jam> jtv: well, we had some old 'temp_' tables that required st-ub to fix
<jtv> Ah yes.
<jam> but then the copy happenid in about 5 min
<jam> vs the 1.5hrs to get the delete script to run
<jtv> There'll be less of that holding-table business.
<jam> (about 3 retries)
<jtv> Five minutes!  It's hard to express how happy that makes me.
<jam> jtv: I don't have the background to fully appreciate it, but yes, very quick vs what it looks like in the past.
<jtv> Do note that please on the wiki page.
<jam> Already been noted
<jam> but it will be highlighted later.
* gmb changed the topic of #launchpad-dev to: grahambinns
<gmb> Arse
<jam> I'm personally satisfied enough that the copy completed correctly. As you expected, the statistics are all wrong, but the strings all look correct.
<jtv> Wellâ¦ 2007: take Launchpad offline for the copy.  2008: strain the app to its limits for I don't remember how many days as we copy dozens of millions of rows.
<czajkowski> gmb: LOL!
* jam changed the topic of #launchpad-dev to: The topic for #launchpad-dev is: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
* gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: gmb | Firefighting: - | Critical bugs: 3.47*10^2
 * gmb needs to remember that he sucks at IRC.
<czajkowski> meh could in fact be worse
<czajkowski> you in the Fin?
<gmb> Also, I need to write a bot for taking ReviewerSchedule and updating the topic (because I do this crap on a regular basis)
<gmb> czajkowski, Yep. It's massive!
<gmb> I feel tiny
<jtv> gmb: And a bot that says âgood morning, volunteer!â as you log in on your review day.
<gmb> Quite
<jam> jtv: definitely well done
<jtv> jam: those Spanish translations look fine to me.  I won't say there's no problem anywhere, but it passes the basic sanity test that really matters right now.
<czajkowski> gmb: aye it is nice alright
<jam> jtv: I haven't found a page with translations strings that aren't identical.
<jam> Just "Contributors" for a given entry, etc.
<jam> and the summary of "Untranslated" doesn't quite match.
<jam> but all those are summary statistics stuff.
<jtv> The credits should start to fill in, but one of the breakthroughs of this year is that that is now a separate problem.
<jam> jtv: so its says the next step is activating the translation imports. Does it seem reasonable to do so?
<jtv> The only numbers you can trust, really, are on the batch navigator for a page â e.g. 1234 items in the âtranslated itemsâ view.
<jtv> jam: I would say so, and I'm sure dpm would be much relieved.  Ask him to tell you some stories as well.  :)
 * dpm reads scrollback...
<jtv> jam: it's useful to note when the imports are started, because that way we can more or less track in the UI whether a full generation of translations has been imported yet.
<jtv> Bon dia dpm :)
<jtv> jam: and now for a short intermission between self-congratulation sessionsâ¦ I see Approved items on the import queue for Quantal, but they're not attached to any files to import to.  That looks like an omission in the cleanup script.
<jtv> If we enable imports, that will break.
<jam> jtv: I don't quite follow your thread, links?
<jtv> https://translations.launchpad.net/ubuntu/quantal/+imports
<jtv> These are the files awaiting import into Quantal.
<jtv> Approval, in translations-import-queue terminology, is a bit of a misnomer.
<jtv> It also involves attaching an upload on the queue to a POTemplate or POFile in the database that it's supposed to go into.
<jtv> (There must be one; these objects are created as part of the approval process).
<jtv> For most files, it's the import queue gardener (rosetta-approve-imports.py) that does this.
<jtv> And the importer assumes that any file it's going to import will already have been approved, and so is attached to a POTemplate or POFile.
<jtv> So we need something like:
<jtv> UPDATE TranslationImportQueueEntry SET status=5 WHERE status=1 AND distroseries = (SELECT id FROM DistroSeries WHERE distribution=1 AND name='quantal');
<jtv> Let me just ask about that on -ops.
<wgrant> jtv: That's not doable through the API nowadays?
<jtv> Hmmâ¦ it might be, but I imagine not on a batch basis.
<wgrant> Certainly not on a batch basis, no
<dpm> jtv, jam, having a quick look at P and Q, the translations look ok, but I'm also concerned about the mismatch in the number of contributors. Apparently we got +100 contributors on Q according to LP, which is certainly not the case
<cjwatson> gmb: Oh hello Mr. OCR.  There's a question for you in https://code.launchpad.net/~cjwatson/launchpad/export-change-override/+merge/109549
<dpm> sorry, I mean for Catalan
<dpm> it might be different for other languages
<jam> dpm: most of the ones I've seen have almost 2x the contributors in P than Q
<gmb> cjwatson, Looking now.
<jam> dpm: I'm seeing P having 145 more contributors, how are you getting 100 more in Q?
<jtv> dpm, jam: it'll take garbo some time to derive all that contributor information.  Try looking at templates whose names start with numbers, or 'a'.
<jam> jtv: the first one is "Abkhazian" which matches (with 0 contributors), but Acehnese is 55 in P vs 18 in Q
<jtv> Template names, not languages.
<jam> ah
<daker> mwhudson, do you need me ?
<jtv> jam: also, if the daily garbo job was interrupted for thisâ¦
<dpm> jam, sorry, you're right, I was looking it the other way round -> https://translations.launchpad.net/ubuntu/quantal - Catalan: 216 contributors, https://translations.launchpad.net/ubuntu/precise - Catalan: 318 contributors
<dpm> so Q is 102 less contributors
<jam> dpm: well that number just jumped from 20 min ago
<dpm> than P
<jam> for Q
<jam> It was 155 just a bit ago
<jam> jtv: it was interupted before the copy, but should be running today.
<jam> dpm: so the number seems to be going in the right direction
<dpm> gotcha
<jtv> Well, there we go then.  :)
<gmb> cjwatson, Okay, it looks like copy_field isn't doing what it should. Revert to your earlier solution and add a comment explaining why you're not using copy_field. r=me on everything else.
<jam> jtv: currently OTP, but I want to sort out the import queue things with you in a bit if that's ok. Or are you at EOD
<jtv> jam: I'll be here.
<cjwatson> gmb: Fair enough, thanks.  (It's blocked on another landing at the moment, but hopefully after that.)
<gmb> cjwatson, Okay.
<wgrant> jtv, jam Are you aware of the poimport errors?
<wgrant> 2012-06-19 09:46:16 ERROR   Entry is approved but has no place to import to. -- 'po/skanlite.pot' (id 6579415) in ubuntu Quantal package skanlite
<wgrant> Lots of that
<jam> wgrant: I just unchecked the import queue, and we're chatting about it. Still OTP, though.
<jtv> That'd be the thing I said would happen.  :)
<jam> jtv: back.
<jtv> OK
<jml> benji: I've annotated the SSL exception in those two branches you reviewed yesterday. I'd welcome a re-review.
<benji> jml: sure, I'd be glad to
<jml> benji: thanks.
<benji> jml: looks great
<jml> benji: thanks. Do you know what the landing process is for those two projects?
<benji> since the value of "e" is known to be an exception the "% (e,)" bit is superfluous, but it won't hurt anything
<jml> benji: a little paranoia is healthy.
<benji> Just merge to trunk, update NEWS, and make a release... I think there is a document for that, hold on a sec and I'll find it
<jml> benji: you never know when SSLHandshakeError will decide to subclass tuple!
<benji> heh
<benji> that'll be the day Python jumps the shark
<benji> jml: https://dev.launchpad.net/HackingLazrLibraries#landing-your-branches and https://dev.launchpad.net/HackingLazrLibraries#releases look like what you need
<jml> versions :(
<jml> benji: thanks.
<benji> versions.cfg loves you
<jml> benji: SSLHandshakeError isn't in the buildout-depended version. Should I try to match httplib2 behaviour in currently depended version or should I bump the version dependency?
<benji> you can bump it; versions is for development and if development doesn't match reality it can be changed
<jml> benji: I don't have commit access.
<benji> jml: I'll land it for you if you want
<jml> benji: thanks.
<jml>   [r=benji] Add environment variable, LP_DISABLE_SSL_CERTIFICATE_VALIDATION, to
<jml>   disable SSL certificate checks.  Most useful when testing against development
<jml>   servers.
<jml> that's my commit message.
<benji> k
<jml> benji: no need for the bump, I just hadn't run buildout in a while.
<benji> cool
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: gmb, rick_h | Firefighting: - | Critical bugs: 3.47*10^2
<rick_h_> bah, sorry jcsackett wrong channel, reset irssi and my channel numbers get moved on me
<maxb> rick_h_: /layout save is your friend :-)
<jam> dpm: according to https://wiki.canonical.com/Launchpad/Translations/UbuntuOpenings the next steps are up to you. (steps 8+)
<maxb> (Remember to also /save afterward)
<cjwatson> rick_h_: Thanks for r= on https://code.launchpad.net/~cjwatson/lazr.restfulclient/cgi-parse-qs-deprecated/+merge/111001.  I don't have commit access; could you land it for me
<cjwatson> ?
<dpm> jam, excellent, thanks for your work in getting the copy done. I can now open the translations and set up language packs before the end of the week.
<rick_h_> cjwatson: sorry, my LP env isn't setup yet on the new machine. Can we rope someone else in atm? adeuring can you help land please? ^
<adeuring> cjwatson, rick_h_: sure.
<rick_h_> ty adeuring
<jam> jelmer: did you get a chance to re-submit my MP?
<jelmer> jam: ah, sorry, I misunderstood.. thought you were going to get somebody to run the fancy parallel test script for it
<jelmer> resubmitting now
<jam> jelmer: I was meaning for the sprint week on that.
<bac> mrevell: "crave the metal"  -- ha!
<mrevell> bac, :)
<bac> mrevell: i think the narrator for The Big Lebowski would add a certain western flair that might be appealing.
<jml> benji: hi. I don't mean to hassle but I haven't seen my MPs get merged. Anything I can do?
<deryck> Morning, all.
<benji> jml: I've been on calls.  Working on it now.
<jml> benji: np, thanks.
<czajkowski> sinzui: morning
<sinzui> hello
<mrevell> bac, I like that idea :) I wonder what he charges
<benji> jml: launchpadlib 1.10.0 released; releasing lazr.restfulclient now
<jml> benji: woot, thanks.
<benji> jml: ok, lazr.restfulclient 0.13.0 is released
<jml> benji: wonderful :)
<jml> this will make local testing of API scripts much easier.
<benji> yep, it's a good addition
<deryck> adeuring, ivory -- https://plus.google.com/hangouts/_/4da3cee079179cf73f401a0a94ca9813bedf7ca8?authuser=0&hl=en
<jam> Is someone around that can help me understand: http://people.canonical.com/~wgrant/librarian-pruning.html
<jam> It mentions a few SQL statements to run, but then seems to run them under different circumstances.
<jam> (the very first query says "All SPRs" vs "Expired SPRs", but I don't see how the query is restricted to only Expired SPRs
<jcsackett> rick_h_: ack on the comments, looking through making those changes now.
<rick_h_> jcsackett: cool, hopefully removes some more LoC though the attr docs might bring it back, but at least it's not more code
<jcsackett> rick_h_: yeah. i usually think SLoC rather than just LoC. whitespace and comments really aren't the problem we're trying to curb.
<jcsackett> ...well, the suckless version of SLoC. Significant, not Source.
<rick_h_> :)
<rick_h_> jcsackett: let me know if I'm wrong on the event stuff. I kind of saw maybe that method of calling was required due to current scope in event passing
<cjwatson> jtv: I gather from https://code.launchpad.net/~jtv/launchpad/katie-and-gina-are-bad-bad-girls/+merge/75472 that you have some experience running gina into a scratch table on dogfood.  Do you know if this is documented anywhere?  I have a branch I'd like to land, but I'm not sure how to QA it.
<jtv> That's a long time ago.  Let me refresh my memory.
<jtv> cjwatson: ah, I see what you mean now.  I just ran it normally, but also selected some data into a scratch table for before/after comparison.
<jtv> But I don't really remember any of it.
<cjwatson> Ah, OK.  Hmm.
<czajkowski> jtv: what time is it over there ?
<jtv> Late:54
<timrc> has the LP team developed a generic YUI3 Model/ModelList sync layer akin to Backbone.Sync?
<cjwatson> $ ls -ld /srv/launchpad.net/gina-mirror/dists/sid/
<cjwatson> drwxr-xr-x 5 launchpad launchpad 4096 2010-11-25 01:45 /srv/launchpad.net/gina-mirror/dists/sid/
<cjwatson> Eep, that's old.  Probably doesn't have any of the packages I care about in it.
<cjwatson> Maybe I could download one of them into the mirror by hand?
<cjwatson> Oh, main/source/Sources.gz is 2011-09-12.  Still.
<jtv> ISTR this stuff depends heavily on what happens to be in the local archive on dogfood.  Maybe there's ways of injecting individual packages directlyâ¦ I don't know.
<cjwatson> I bet it'll get confused by the database (hence /debian/+source/*) being substantially newer than the mirror.
<cjwatson> I do need to do some approximation of a full run.  I have unit tests for my changes to lib/lp/soyuz/scripts/gina/*, but scripts/gina.py doesn't seem to be meaningfully tested.
 * deryck goes offline for lunch, back in an hour
<jtv> cjwatson: I can only recommend waiting for Australia to wake upâ¦
<cjwatson> :-)
<czajkowski> it's an odd timezone to wait for, but given the folks there don't sleep it's not too long a wait either
<flacoste> gary_poster: you should get your blog on the LP planet, the retrospectives would show up on it
 * cjwatson upgrades mawson while he waits, given that when StevenK last tried that it took three hours
<czajkowski> flacoste: +1
<gary_poster> flacoste, cool will do.  I don't see how to do that on planet.launchpad.net, but looking around more.  If you know (or czajkowski?) hints welcome
<flacoste> gary_poster: you might have to ask mrevell, i don't know the specifics tbh
<gary_poster> ok cool
<mrevell> gary_poster, I can add you to the planet. It's not especially intuitive.
 * cjwatson notices there's a doctest for gina.  Phew, that's slightly less bad than it might have been.
<mrevell> gary_poster, It's so long since we added one that I can't remember if anyone in the LP team can do it or just a few people.
<gary_poster> mrevell, oh cool thanks.  Lemme get you a tagged link...
<gary_poster> mrevell, http://codesinger.blogspot.com/feeds/posts/default/-/launchpad or http://codesinger.blogspot.com/feeds/posts/default/-/canonical .  they are the same atm :-)
<mrevell> thanks gary_poster.
<gary_poster> thank you
<czajkowski> gary_poster: see your blogging of team meetings is going to catch on :)
<gary_poster> czajkowski, heh, we'll see :-)
<jcsackett> rick_h_: took a bit to cycle back, but replies on my MP and code updates.
<rick_h_> jcsackett: loading up now
<james_w> gmb, rick_h_: does one of you have time for https://code.launchpad.net/~james-w/python-oops/refactor-publishing/+merge/109428 ?
<rick_h_> james_w: will look in just a sec
<rick_h_> sorry, two snuck into the queue on me while I wasn't looking
<james_w> thanks rick_h_
<rick_h_> jcsackett: r=me, thanks again for taking the time
<jcsackett> rick_h_: thanks for the review and the suggestions. :-)
<rick_h_> jcsackett: fav, when you assign me a review make sure to ping me. I didn't notice it until I loaded the list for OCR today. I totally would have missed it otherwise
<jcsackett> rick_h_: i didn't ping you b/c i knew you were OCR today, and i put it up last night. :-)
<rick_h_> ah, well nvm then. Glad I didn't hold you up too much then
<jcsackett> didn't hold me up in the least. thanks again. :-)
<rick_h_> james_w: so is there a back story here I assume?
<james_w> rick_h_, there is, yes
<james_w> I guess I should have updated the description, sorry
<james_w> rick_h_, the oops stack has a hack where it decides if a report has already been published by seeing if it has an 'id' key
<james_w> but it's valid for an app to allocate an id when it generates an oops, e.g. for django to show the id to the user
<rick_h_> ok
<james_w> there is a wrapper function in python-oops that publishes to a publisher only if the id isn't set
<rick_h_> ok, so this is trying to build that in with the flag
<rick_h_> ?
<james_w> which you want to use for reliable oops delivery (try and publish to amqp, if that fails drop it in a directory and have something else try again when amqp is back)
<james_w> so if the app allocates the id then you never get that fallback, and oopses are lost
<james_w> so you remove the wrapper and instead duplicate the work as you will always publish to the dir on disk, even if amqp succeeeded
<james_w> talking to Rob about it we decided that whether an oops has been published should be remembered by the config, rather than stored in the oops
<rick_h_> ok
<james_w> and then publishers would be added to the config declaring if they should only get oopses that are unpublished
<james_w> that would be an API break from the current API of appending publishers to a list
<rick_h_> james_w: ok, r=me with a couple of nitpicks, one is just something that maybe is me coming unaware to the conversation
<james_w> Rob suggested that instead the "publish to a list of publishers" thing be extracted out and taught all of that stuff
<james_w> so I did that, and left the old code in for compatibility
<rick_h_> ok, kind of see it. Now an oops pro, but nothing here looks like it'll go boom! :)
<rick_h_> /now/not
<james_w> thanks rick_h_
<james_w> rick_h_, good comments, thanks
<james_w> I'll fix all of that up
<james_w> I'm not sure about deprecation warnings
<james_w> I don't know if there is a policy we can follow without thinking
<rick_h_> yea, not sure either. I've not really seen us do that, but if we're clearly deprecating things in an API seems like we really want to get the word out
<rick_h_> since I know python-oops stuff is used in a few places I believe
<rick_h_> sinzui: any feedback on officially deprecating something like this in a library? python-oops
<sinzui> rick_h_, I think we want to raise the warning to tell developers we are going to remove the API. We do not know this much since we have do not have a lot of code that is reused. We do rely on the deprecated warning from our libs to tell us to fix code before it breaks
<james_w> rick_h_, yeah, a few
<james_w> I'm not too keen on warnings immediately when the change is made as that tends to annoy developers
<james_w> I'll be trying to shepherd internal users along
<james_w> I don't know how many external users we have
<rick_h_> sinzui: ok, thanks. So james_w I'd think we'd just use http://docs.python.org/library/warnings.html
<rick_h_> james_w: well, it'll take time for them to get the new version I'm sure, so don't think it'll annoy current items right off the bat
<james_w> no, but IME, "YOU'RE USING SOMETHING YOU SHOULDN'T BE AND I'VE ONLY JUST TOLD YOU BUT I'M GOING TO ANNOY YOU ABOUT IT UNTIL YOU FIX IT EVEN THOUGH YOU CAN'T WRITE CODE THAT WILL WORK ON BOTH VERSIONS" tends to be what developers see :-)
<sinzui> james_w, that other developer can choose to suppress warnings, as we do. We are informed, and we chose to delay the fix.
<rick_h_> james_w: yea, but not seeing how we make sure we get back to this and eventually remove deprecated code
<james_w> that's true
<james_w> rick_h_, I absolutely agree on the XXX
<james_w> then someone will clean it up as part of the LoC policy before too long :-)
<rick_h_> hah
 * sinzui recalls that Lp needs to switch to hashlib to get to python 2.7
<rick_h_> james_w: well, personally I'd prefer to see the warning. I ok'd it, and as long as the XXX and bug are there, it's a trail, but not an obvious one for users of the module
<james_w> rick_h_, yeah, I'll make the change to emit a deprecation warning too
<rick_h_> james_w: thanks, appreciate it
<james_w> thanks for the review
<gary_poster> To Whom It May Concern: the most recent devel commit from stub might have caused some test failures.  http://ec2-174-129-111-89.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/3/steps/shell_9/logs/summary  I'll investigate more later
<cjwatson> sinzui: AFAICS it already has
<cjwatson> sinzui: Unless there's stuff in sourcedeps I'm missing
<stub> gary_poster: I landed something?
 * sinzui looks
<gary_poster> stub, http://ec2-174-129-111-89.compute-1.amazonaws.com:8010/changes/7 ?
<stub> oh, damn. I lp-landed instead of ec2 land
<stub> sorry
<sinzui> cjwatson, if so we can delete code from lp_sitecustomize
<sinzui> cjwatson, and script_isolation.py
<gary_poster> stub, cool, are you reverting or otherwise fixing or should I do it or try to corral someone to do it?
<stub> I can revert it unless someone has something ready. Just syncing trunk down.
<sinzui> jcsackett, I would like your review of this SQL that clean up historical data. This relates to the branch you reviewed to teach the commercial project job to delete commercial subscriptions for non-proprietary projects: http://pastebin.ubuntu.com/1049490/
<cjwatson> sinzui: From a quick glance through, I think ZODB3, windmill, pycrypto, and ipython still have unguarded md5/sha imports
<jcsackett> sinzui: ok.
<cjwatson> sinzui: I'm not sure why 2.7 is relevant, though; md5 and sha were deprecated in 2.6, and seem to be no less available / no more deprecated in 2.7 than in 2.6
<jcsackett> sinzui: r=me, looks good.
<stub> What is the magic rollback syntax?
<stub> for the commit line?
 * stub finds the guide
<stub> hmm... doesn't say
<sinzui> cjwatson, windmill...We don't use it. I thought all the code was removed. Someone gets to delete it
<sinzui> I thought the sha module was removed in 2.7
<cjwatson> Nope, still there
<rick_h_> yea, have it here
<rick_h_> just deprecated still
<cjwatson> Gone in 3.0 of course
<abentley> stub: [rollback=revno]
<abentley> rick_h_: Could you please review https://bugs.launchpad.net/launchpad/+bug/891715 ?
<_mup_> Bug #891715: API doesn't recognize standard parent branch URLs <api> <Launchpad itself:Triaged> < https://launchpad.net/bugs/891715 >
<abentley> rick_h_: I mean https://code.launchpad.net/~abentley/launchpad/real-branch-name/+merge/111082
<rick_h_> abentley: looking
<rick_h_> abentley: r=me
<abentley> rick_h_: ty
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: -  | Firefighting: - | Critical bugs: 3.47*10^2
<lifeless> gary_poster: what was the lxc workaround?
<jcsackett> sinzui: got a few minutes to talk?
<sinzui> yes
<jcsackett> cool. i'll send a hangout invite momentarily then.
<gary_poster> lifeless, change /var/lib/lxc/lptests/rootfs/etc/init/networking.conf to not wait for udev (that is, use "start on (local-filesystems)"); and remove the container's /etc/init/udevtrigger.conf (rm /var/lib/lxc/lptests/rootfs/etc/init/udevtrigger.conf)
<lifeless> hah!
<lifeless> so some udev concurrency limit ?
<jcsackett> sinzui: laptop with webcam won't start google+, so i'm in messenger via the phone app.
<gary_poster> lifeless, yeah something like that.  hallyn didn't get deep into it because he didn't want to focus on it if what he gave us was good enough.  I said it seemed good enough to me.  When we switch to Precise, we can remove the hack.
<lifeless> bac: hi
<lifeless> bac: p:~bac/zope.testing/1012171 - I notice that you've had some trouble with the Content stuff there; what was in the file already appears crufty to me - there are solid helpers in testtools that would make life a lot easier.
<lifeless> bac: I've made a couple of notes in the MP, but looking in testtools.content and testtools.content_type may give you even more ideas.
<lifeless> bac: there is one specific defect you should fix though, which is that iterating 'asd' vs ['asd'] will be considerably slower, if the length of the string is nontrivial.
<benji> lifeless: thanks for pointing out the helper on the MP; bac is away right now but I'm sure he'll implement the suggestion in the comment
<lifeless> benji: anytime
<benji> the bit about the API really being an iterable of strings reminds me of why I hate strings being iterable in Python
<lifeless> indeed
<lifeless> I hate that too
<lifeless> but breaking that to make the API clearer sucks more :(
<lifeless> actually, what I hate is type(char) and type(str) being equivalent, same same though
<benji> absolutely; the fault here clearly lies with Guido, circa 1990 :)
<benji> I don't mind that so much; it would be possible to make strings not iterable, but instead have, say, a .chars attribute that is, for when you really do want to iterate over the chars
<lifeless> cat, skin, # of ways ;)
<benji> :)
<lifeless> by which I mean that that would work too
<czajkowski> poor cat
<czajkowski> :)
<timrc> has the LP team developed a generic YUI3 Model/ModelList sync layer akin to Backbone.Sync?
<timrc> Hm, it would help if I mentioned that I'm specifically referring to a sync layer for Yui.App
<bac> lifeless: thanks for the note
<lifeless> bac: np
<StevenK> Conflict adding file logs.  Moved existing file to logs.moved.
<StevenK> Conflict adding file logs.moved.  Moved existing file to logs.moved.moved.
 * StevenK facepalms
<StevenK> Oh. Sigh. PPAVocabulary has no tests.
<StevenK> Looks like none of the Soyuz vocabs do.
<sinzui> https://qastaging.launchpad.net/launchpad/trunk/+addrelease
<rick_h_> timrc: no, we're not up to 3.5 yet, but the combo loader integration is in beta so I'd love to hopefully get it some point this year
<rick_h_> timrc: rockstar over on the U1 team has been doing some work on the YUI3 Y.App stuff so you might ping him, but don't think they're to that point
<rick_h_> timrc: if you're interested I can chat about it, I've used the model layer on some outside stuff, but nothing LP specific.
<timrc> rick_h_, we are re-implementing some stuff in hexr and are going to use Y.App on the front-end and django-tastypie on the backend... I believe bigkevmcd wants to implement a generic sync layer akin to Backbone.Sync... so I was wondering if anyone in the company has done it yet
<timrc> seems like people are building model-specific sync layers
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/gina-deb-vendor-debian/+merge/111120 - dogfood gina QA advice wanted
<wgrant> cjwatson: "meh"
<cjwatson> That good?
<wgrant> I'd suggest a combination of not caring and constructing a fake archive that contains the relevant package.
<cjwatson> Yeah, I just wasn't sure how that would interact with the existing dogfood.lp.net/debian.
<rick_h_> timrc: it seems it'd have to be tied to the api end
<cjwatson> Unless you mean import it into an entirely fresh distribution or something.  (Sounds hard.)
<rick_h_> since tastie-pie would have requests formulated a certain way, I'm not sure anyone would have anything yet
<cjwatson> Kind of a shame gina doesn't seem to have a dry-run mode.  If it did, it'd be sufficient to run it with that and make sure it didn't explode.
<cjwatson> Maybe I should add one?
<rick_h_> timrc: but yea, I could easily see a Y.Model.sync -> tastiepie module
<wgrant> cjwatson: Could just comment out the transaction.commit()s...
<wgrant> cjwatson: It's easy enough to import into a new distro, but then you can't use the actual problematic package.
<cjwatson> Oh yeah, it's dogfood, I guess that's an option
<wgrant> Because DEB_VENDOR will be different
<cjwatson> Right
<cjwatson> I could go with comment-out-commits as a plan
<cjwatson> And just inject the broken qjackctl or something with a spoon
<timrc> rick_h_, yeah with specific regards to tastypie, I believe there is a project called backbone-tastypie which overrides the default sync() call to  build models and collections using its JSON format
<timrc> the same goes for saving models and collections back to the server
<timrc> so something similar for Y.Model.sync as you suggest
<rick_h_> timrc: right, so basically in the YUI.model you'd monkey patch the sync method and generate the generic request with the model's ATTRS
<rick_h_> timrc: so makes total sense, especially with the current state of the tooling standards, but nothing existing I know of for it as the Y.Model stuff is really knew to us as well as the tastiepie default
<rick_h_> timrc: I'd be happy to help/look over anything that comes up, but since LP is so far from tastiepie and 3.5 I don't see us directly using/getting to anything
<timrc> rick_h_, awesome, thanks
<timrc> wgrant, do the api servers throttle "heavy" users?  I've written a script to generate a graph of teams and their relationships using a sample of users across our organization (in the hopes I can discover as many business-relevant teams as possible and how they're related).  I
<timrc> argh
<timrc> I've tried to design the script so that it doesn't hit the API if it's already seen a team
<lifeless> timrc: we firewall bad users.
<lifeless> timrc: we don't current have dynamic throttling.
<lifeless> Don't be a bad user :P
<timrc> :)
<lifeless> timrc: also, you can't see most of the business relevant teams, because they will be private.
<timrc> lifeless, I'm super special
<timrc> or something
<timrc> I have fairly good insight into most of the important PES teams
<lifeless> sure, but the organisation is rather larger than PES :P. Just saying ;)
<timrc> lifeless, oh well allow me to qualify the organization as "PES" in this case
<lifeless> ah, sure ;)
<StevenK> wgrant, wallyworld_
<StevenK> Bah
<StevenK> https://code.launchpad.net/~stevenk/launchpad/archive-empty-description-vocab/+merge/111118
<StevenK> wgrant, wallyworld_: https://code.launchpad.net/~stevenk/launchpad/archive-empty-description-vocab/+merge/111118
<wallyworld_> StevenK: sorry, i will need a bit of time to finish my qa
#launchpad-dev 2012-06-20
<jcsackett> wallyworld_: Don't know if you want to chat, but I'm back if you want to. Hackerspace meeting ran long.
<wallyworld_> jcsackett: sure, just give me a minute
<jcsackett> Anytime. Going to have to be irc tho. Too loud here for anything else. PM me when you want.
<wallyworld_> jcsackett: tl;dr; we will be adding a (+)Team link to the person picker, will transition picker content to simplified add team form like we do for confirmation stuff in the picker
<wallyworld_> this outcome is different to what you discussed with sinzui yesterday
<jcsackett> Yeah. Y'all convinced him?
<wallyworld_> jcsackett: we also need to look at simplifying the open/closed team choices from 4 radio buttons down to 2 choices (open/close) with an option checkbox for each (thus providing the 4 scenarios)
<wallyworld_> we talked it through
<wallyworld_> was a good discussion
<wallyworld_> jcsackett: i believe you will be asked to look at the open/closed policy choices as per the above
<jcsackett> Cool. I'll bug sinzui about it tomorrow then. :-)
<wallyworld_> and then we can combine that work with the new picker/overlay form
<wallyworld_> sounds good
<wallyworld_> StevenK: were there no other soyuz vocab tests you could add to? ie did you have to create a new test module just for that one test?
<wallyworld_> wgrant: we have a small workflow issue. share all es with aaa. subscribe aaa to es bug. no aag created because apg exists so bug is visible. change sharing permission for es for aaa to some. bug not visible and no aag created
<wgrant> wallyworld_: Yeah, that's where conflating subscription and visibility continues to be awkward..
<wgrant> wallyworld_: I suspect we should just create the usual revocation job and remove the subscription.
<wallyworld_> wgrant: not sure that's the best thing to do. i mean the user intent is to be subscribed to that bug still, but only see that and not all other private things
<wgrant> wallyworld_: Doesn't that same argument apply to the team's members?
<wallyworld_> in which scenario?
<wgrant> eg. foo a member of bar, foo is subscribed to an artifact on a policy on which bar has an APG
<wgrant> bar's APG is revoked.
<wgrant> Possibly changing to Some.
<wgrant> Or to None.
<wallyworld_> yes, the case for apg revocation converting existing subscriptions to aag should be transitive
<wallyworld_> maybr
<wallyworld_> no, i mean the team gets the aag
<wallyworld_> and then the members can see the bug while they remain in the team
<wgrant> That's illegal
<wgrant> Since the team doesn't have a subscription.
<wgrant> It can't have an AAG.
<StevenK> wallyworld_: There are no other Soyuz vocab tests that I could find.
<wallyworld_> really? that's not good
<wallyworld_> not even in doc tests?
<StevenK> wallyworld_: I grepped the entire tree for the vocab names
<wallyworld_> wgrant: my assumption was the team did have a subscription, but if not, then i think what you say sounds ok
<wallyworld_> StevenK: r=me
<wallyworld_> wgrant: so everything looks ok so far on qas with the triggers removed, except for an oops in the job created when a team member is removed.
<StevenK> wallyworld_: Thanks, I've tossed it at ec2.
<wgrant> wallyworld_: What's that OOPS?
<wallyworld_> OOPS-96532e891268827ddb456c0488c160ef
<wallyworld_> i haven't looked at it yet, created a card
 * wallyworld_ goes to get key cut, bbiab
<StevenK> wgrant: So, query?
<wgrant> StevenK: The query itself is probably fine.
<wgrant> But we likely want an index on access_policy
<StevenK> There is one?
<StevenK> The EXPLAIN ANALYZE mentions an index scan
<wgrant> It was probably an index scan on information_type.
<wgrant> However
<wgrant> There's few enough private branches that that might not be a problem.
<StevenK> wgrant: It's worse with an index on access_policy
<StevenK> wgrant: http://pastebin.ubuntu.com/1050184/ both hot, top is without the index on access_policy.
<wgrant> StevenK: Try now
<StevenK> wgrant: Same as the top one
<wgrant> StevenK: Also, what's the actual query you're using?
<StevenK> wgrant: Oh, sorry. http://pastebin.ubuntu.com/1050188/
<wgrant> So, that's not quite the actual query.
<wgrant> The real thing will have a LIMIT foo at the end
<StevenK> wgrant: http://pastebin.ubuntu.com/1050199/
<wgrant> StevenK: It seems to be fine without the extra index.
<wgrant> Not marvellously fast.
<wgrant> But the public branches aren't clustered badly enough that it will be a problem.
<cjwatson> Is utilities/prioritygrid.py in use any more?  It looks obsolete since I don't believe those wiki prioritisation tables are being used any more (?).  Maybe 146 easy LoC for somebody.
<lifeless> nuke from orbit
<StevenK> wgrant: Right, so do you want to approve the MP, then?
<cjwatson> lifeless: righto, will do
<wgrant> StevenK: Done
<lifeless> cjwatson: classic NIH stuff that
<lifeless> cjwatson: https://dev.launchpad.net/VersionThreeDotO/Soyuz/Inputs - last edit 2009
<wgrant>     def __getitem__(self, name):
<wgrant>         for series in self.series:
<wgrant>             if series.name == name:
<wgrant>                 return series
<wgrant>         raise NotFoundError(name)
<wgrant> wtf
<wgrant> wallyworld_, StevenK: Extremely challenging review for one of you: https://code.launchpad.net/~wgrant/launchpad/bug-1015289/+merge/111139
<wallyworld_> wgrant: looks ok to me, i assume we have an approximatedate tal formatter
<wallyworld_> there are no tests needing updating?
<lifeless> wallyworld_: wgrant: if you guys have FDT pending stuff, I'm willing to grant exceptions to the one-patch-per rule; I'll be updating docs shortly to reflect that.
<lifeless> needs to route through me or flacoste on a case by case basis.
<wgrant> wallyworld_: All the tests use sampledata, so they're a bit awkward to test this with. It's the formatter we use for dates basically everywhere, so I don't think tests are very useful.
<wgrant> lifeless: We deployed the last of the two week-long batches on Monday.
<wallyworld_> wgrant: ok, r=me
<wgrant> wallyworld_: Thanks.
<wallyworld_> np
<StevenK> lifeless: Oh, because I don't do DB patches? :-P
<lifeless> StevenK: because you haven't been whinging about it :)
<lifeless> StevenK: (actually, I forgot to ping you as well, mea culpa)
<StevenK> lifeless: wgrant is very good at whinging, so I've been letting him play to his strengths ...
<lifeless> LOL
<wgrant> Heh
<wgrant> Ah, 8-way UNION ALLs, my favourite.
<lifeless> wgrant: the openid thing thats terrible, or something else ?
<wgrant> lifeless: My bugsummary recalculator.
<wgrant> lifeless: The OpenID thing being the name-generation timeout you pointed out yesterdayish?
<lifeless> funky chicken
<lifeless> wgrant: is that what it was?
<wgrant> lifeless: Yeah
<wgrant> email address started with launchpad@
<lifeless> wgrant: win.
<wgrant> So it tried to generate a nick from that
<wgrant> Conflict, so add number
<wgrant> Conflict, so increment number
<wgrant> Conflict, so increment number
<wgrant> Conflict, so increment number
<lifeless> haha
<wgrant> Although we may have another bug, since there aren't *that* many, I don't think.
<lifeless> perhaps it should select count(*) from person where name like 'prefix-%' instead.
<lifeless> and docs updated
<lifeless> -> to buy a PC powersupply for his reprap
<wgrant> My god
<wgrant> The algorithm isn't actually that stupid
<wgrant> It's even worse
<wgrant> I don't even...
<wgrant> what the heck is it trying to do
<mwhudson> does the name blacklist get in on the fun too?
<wgrant> (for those playing at home, it gets from 'launchpad' to '78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgs-launchpad-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze')
<mwhudson> actual lol
<mwhudson> (must be tired)
<StevenK> Oh my ...
<wgrant> So it tries to add a random suffix.
<wgrant> Then a random prefix
<wgrant> Then mutate a character
<wgrant> Then a prefix and a suffix
<wgrant> Then all three
<wgrant> And somehow it gets 78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgs-launchpad-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze
<wgrant> Still
<wgrant> Not as bad as it could have been
<wgrant> It does up to 1000 attempts
<wgrant> Each of which adds a character to the prefix and suffix, and mutates the name further
<lifeless> wgrant: how did it get to the number it got to ?
<wgrant> So if we didn't ahve the timeout...
<wgrant> I suspect it's because it contains launchpad
<wgrant> So is blacklisted
<lifeless> hah
<wgrant> Hm
<lifeless> so, just plain fail.
<wgrant> Except then the mutation should pass
<wgrant> So maybe not
<wgrant> Hmmmmmmmmmmmmmmmm
<wgrant> mmmmmmm
<wgrant> mmmmmmm
<wgrant> mmmmmm
<wgrant> The mutation it ends up with is cdpq7ln4t
<wgrant> Which is already the name of a person
<wgrant> Surely we don't actually start with a predictable seed.
<wgrant> ROFL
<wgrant> ROFL
<lifeless> whatever could go wron with that ?
<lifeless> back soon, to hear the end of the story
<wgrant>     # We seed the random number generator so we get consistent results,
<wgrant>     # making the algorithm repeatable and thus testable.
<wgrant>     random_state = random.getstate()
<wgrant>     random.seed(sum(ord(letter) for letter in generated_nick))
<wgrant> Hahahahahahah
<mwhudson> ah uh what
<wgrant> Because that's a good idea.
<mwhudson> i presume this means there are some truly comedy nicks already
<wgrant> So because so many people have a local user part of 'launchpad'...
<wgrant> yes
<wgrant> that
<wgrant> Checking for just what we have...
<wgrant> That must mean that 78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgs-cdpq7ln4t-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze already exists
<wgrant> Or it's too long
<wgrant> http://launchpad.net/~78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgs-cdpq7ln4t-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze :)
<mwhudson> https://launchpad.net/~78luphr-lannyuwgd-a811i2i
<mwhudson> hahahahahahahahahahahahaha
<mwhudson> ahah
 * mwhudson runs out of breath
<wgrant> I'd seen crazy stuff like this before
<wgrant> But assumed it was just users being strange
<mwhudson> given
<mwhudson>             "This should be impossible to trigger unless some twonk has "
<mwhudson>             "registered a match everything regexp in the black list.")
<mwhudson> i'd be inclined to blame SteveA
<wgrant> Nah
<wgrant> Not really
<mwhudson> (probably not for the seeding though)
<wgrant> Back in his day it included the domain
<mwhudson> ah
<wgrant> But we stopped including the first segment of the domain because people complained it gave away their email address.
<mwhudson> heh
<mwhudson> sha256 the address instead? :)
<wgrant> launchpad_dogfood=# SELECT COUNT(*) FROM person WHERE name LIKE '78lup%';
<wgrant>  count
<wgrant> -------
<wgrant>    306
<wgrant> And that's less than half of them.
<mwhudson> length(name) > 100 ?
<mwhudson> or whatever the sql is for that
<wgrant> 275
<mwhudson> oof
<wgrant> Lots of those are deactivatedaccount, though
<wgrant> Lots is actually only 14
<StevenK> Lots, in troll.
<wgrant> Most of the rest are the 78luphr things
<wgrant> Except for the ones that are generated from ubuntu@ instead of launchpad@
<wgrant> eg. mrzx4l98d4tp89jab6giohdrjqysbyjs4npz2ccq25kvjmf5h8u4cmidcko7s4tfr6ur1teuv4ju1af4k-bp84z2-hwbqs6tox1bv6csee9psn5309v7488f3dugifm692db2xfq8n1fsz7l87835tr0q36m2p3ftwpoqoy6v6
<StevenK> Memorable.
<wgrant> admin@ and info@ are also not uncommon
<mwhudson> https://launchpad.net/~mrzx4l98d4tp89jab6giohdrjqysbyjs4npz2ccq25kvjmf5h8u4cmidcko7s4tfr6ur1teuv4ju1af4k-bp84z2-hwbqs6tox1bv6csee9psn5309v7488f3dugifm692db2xfq8n1fsz7l87835tr0q36m2p3ftwpoqoy6v6 -> "Are you Ubuntu?"
<wgrant> Heh
<mwhudson> "we are all ubuntu!"
<StevenK> Ubuntu does not use Launchpad. This page was created on 2012-05-03 when the landscape-api_12.04+bzr4143-0ubuntu1.8.04+jenkins547 package was uploaded to hardy/RELEASE.
 * StevenK cackles
<wgrant> So
<wgrant> Things that shouldn't be deterministic:
<wgrant>  - Collision resolution algorithms
<lifeless> indeed.
<StevenK> steven@undermined:~/launchpad/lp-branches/destroy-old-privacy-ui% bzr di | diffstat -s
<StevenK>  20 files changed, 72 insertions(+), 425 deletions(-)
<StevenK> I am disappoint.
<wgrant> StevenK: Must have missed some stuff.
<StevenK> wgrant: But where?
<wgrant> Yes.
<StevenK> steven@undermined:~/launchpad/lp-branches/destroy-old-privacy-ui% bzr grep show_information_type_in_ui
<StevenK> steven@undermined:~/launchpad/lp-branches/destroy-old-privacy-ui% bzr grep show_information_type_in_branch_ui
<StevenK> steven@undermined:~/launchpad/lp-branches/destroy-old-privacy-ui%
<StevenK> I'm going to toss it at ec2 and see how much carnage there is.
<lifeless> wgrant: did you file a bug about th eappserver src ip visibility ?
<wgrant> lifeless: No.
<lifeless> wgrant: will you?
<wgrant> lifeless: Not unless there's a good argument for it :)
<lifeless> the law isn't good enough?
<wgrant> The DPA doesn't, AFAIK, say "Launchpad engineers aren't allowed to see source IP addresses in request logs to diagnose operational issues"
<wgrant> That may be implied somehow, but I haven't seen an argument to that effect.
<wgrant> Perhaps there is a wiki page on this.
<stub> Huh. So I accidentally landed my branch by using bzr lp-land instead of ec2 land yesterday, and reverted when this was noticed due to buildbot failure. I then ran ec2 land as intended. And it landed, and buildbot is green.
<stub> The errors buildbot yesterday looked like they should consistently fail too...
<wgrant> stub: What was the buildbot failure like?
<wgrant> stub: It was only on the non-real buildbot.
<wgrant> So it might have been using 8.4, for example
<stub> ohh.... that would explain it
<stub> xx-dbpolicy thinking the slave was the master etc.
<wgrant> I don't know what gary's setup is these days.
<wgrant> But it's reasonable plausible that it's still on 8.4, since we haven't used 9.1-specific stuff yet.
<wgrant> Hm
<wgrant> Except I think StevenK did last week
<stub> Real buildbot is green so the landing is good.
<wgrant> 2209-16-6 uses ORDER BY in an aggregate, so that should have weeded out any 8.4 installations
<wgrant> Do you recall the traceback?
<stub> It must have been some other buildbot - can't see read on the board
<wgrant> It was an ephemeral parallel buildbot test from gary.
<wgrant> It's gone now.
<wgrant> Anyway, let's migrate staging to streaming replication now :)
<stub> yer, looks like an ec2 instance in my history
<wgrant> Right? :)
<stub> yes, that is what this landing is. when it is on staging and qastaging, I can convert the staging dbs.
<lifeless> wgrant: its the conclusion of IS, after discussion with legal etc.
<stub> Unfortunate side effect - at the moment, we have a single cluster so all the RAM on sourcherry can be dedicated to it. With the new setup, we have to run two clusters so ram usage will be less optimal because we will have two copies of the hot set in shared_buffers, and probably should reduce shared_buffers too.
<stub> no, that isn't quite right.
<stub> but it will still be less optimal, as load isn't split 50/50.
<wgrant> lifeless: Is this insanity documented anywhere? :/
<lifeless> wgrant: rt 52417
<lifeless> wgrant: I wanted a solid ruling, so I sought clarity.
<wgrant> Doesn't this also mean that non-EU sysadmins shouldn't be able to see it?
<wgrant> Since that would involve transferring the data to somewhere outside the EU
<lifeless> wgrant: I'm not going to go seeking trouble; elmo has presumably walked through all of that
<lifeless> wgrant: the issue for our purposes is sysadmin/non-syadmin/reason-for-access
<wgrant> brb, starting new planet
<wgrant> I guess sysadmins will have to do all further analysis of load issues.
<bigjools> did you remember the Genesis device?
<lifeless> yes
<lifeless> wgrant: I think thats an extreme proposition to take
<wgrant> Nah, the extreme (but probably correct) position here is the EU should be involved purely because of batshit insane laws like this.
<wgrant> s/involved/dissolved/
<lifeless> actually, if we operated in NZ, we'd have similar constraints
<lifeless> EU is in no way a special snowflake
<stub>  hey, some of us like batshit insane laws protecting privacy :)
<bigjools> wgrant has a point - the EU should be dissolved for plenty of reasons :)
<wgrant> stub: Filtering source IP addresses from the engineers operating the service is hardly protecting privacy in any useful manner :)
<stub> Assuming benevolent engineers
<wgrant> Ah
<wgrant> The initial bugsummary population query counted assignees in viewed_by, but the triggers to maintain it did not.
<wgrant> That explains some of the drift.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 3.47*10^2
<adeuring> good morning
<cjwatson> wgrant: Thanks for your review of https://code.launchpad.net/~cjwatson/launchpad/gina-deb-vendor-debian/+merge/111120.  I made that correction; could I have "Status: Approved"?
<wgrant> cjwatson: Sorry, probably had too many MPs open so the request didn't make it through before I closed the tab. Fixed.
<cjwatson> Thought so, thanks.
<lifeless> wgrant: \o/
<lifeless> wgrant: (finding a/the bug)
<wgrant> lifeless: It's one of the bugs, but there's at least one other that there's probably no hope of ever finding.
<wgrant> I suspect it's gone now.
<ivory> frankban: could you review this? https://code.launchpad.net/~ivo-kracht/launchpad/bug-425934/+merge/110786
<frankban> ivory: sure
<ivory> frankban: thx
<cjwatson> Dear qastaging, would you care to be any slower?  I'm sure it's possible.
 * czajkowski hands cjwatson a baseball bat 
<cjwatson> wgrant: Re the pcj-reupload / PermissiveSecurityPolicy discussion from last night: the test that fails if I ditch that removeSecurityProxy is lib/lp/soyuz/stories/ppa/xx-ppa-files.txt:274.  Is it even possible to run PermissiveSecurityPolicy stuff within pagetests?
<cjwatson> It certainly isn't obvious how to e.g. change the layer to something zopeless.
<cjwatson> I suppose I could change that code to run Archive.copyPackage rather than running do_copy directly, but ISTM that that would just move the problem to one of how to run the job in-place with a permissive security policy
<mpt> "Visible only to users with whom the project has shared embargoed but you've stopped reading by this point anyway."
<jml> :)
<mpt> (reported as bug 1015509)
<wgrant> cjwatson: Yeah, pagetests will be a problem there.
<adeuring> frankban: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-29713/+merge/111212 ?
<frankban> adeuring: on it in a minute
<adeuring> frankban: thanks!
<adeuring> stub: since this MP adds a DB patch file, you might want to have a look too, though it's just a change of ftq() and _ftq(): https://code.launchpad.net/~adeuring/launchpad/bug-29713/+merge/111212
<stub> adeuring: Ta. I'll look at it later. Not sure if this counts as a db patch or a code patch really with all that python.
<adeuring> stub: yeah, but formally it's a DB patch ;)
<stub> adeuring: I think the old code duplication between _ftq and ftq was because we were generating it from a single source. Making one depend on the other now is a good move.
<adeuring> thanks :)
<StevenK> stub: Can you tag the bug related to r15444 as 'qa-bad bad-commit-15444' ?
<StevenK> Hmmm, and then it lands in r15448.
<cjwatson> What happened to http://lpqateam.canonical.com/qa-reports/deployment-stable.html?
<cjwatson> (404)
<james_w> looks like maybe the vhost is now served by something else on the same machine?
<james_w> http://lpqateam.canonical.com now looks suspiciously like https://lpbuildbot.canonical.com/
<cjwatson> Yes, quite
<cjwatson> (Only without HTTPS/openid)
<cjwatson> webops: lpqateam.canonical.com appears broken - see above
<cjwatson> james_w: weirdly, those two aren't on the same IP address
<gnuoy> I'll take a look
<mthaddon> cjwatson: interesting, I suspect that's me - checking
<cjwatson> mm, wrong VirtualHost in /etc/apache2/sites-enabled/lpbuildbot.canonical.com perhaps
<mthaddon> cjwatson: sorry about that, fixed
<cjwatson> thanks
<deryck> adeuring, https://plus.google.com/hangouts/_/698ee8339ef1f2f3cbdce4185991264b12204423?authuser=0&hl=en
<stub> StevenK: Did I break qatagger?
<sinzui> jcsackett, we have a busy day today. I think we have a check point in 20 minutes and I need to inform you of the revelations about creating a team
<jcsackett> sinzui: yeah, i talked a bit with wallyworld_ about it last night, but i def need more info.
<jcsackett> you want to talk post-checkpoint?
<sinzui> yes.
<jcsackett> cool. long round of meetings then. i better go make more coffee. :-P
<sinzui> Maybe we can hijack the hangout
<jcsackett> i'm fine with that.
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<gary_poster> stub, I suspect (hope for your sake) that you are long gone, but just in case, http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/15448 is causing failures on the parallel test machine.  the default flavor of stores that we are getting in the failing tests is master, not slave, which appears to be the source of the errors.  I can't duplicate locally, but I duplicate every time, including across new
<gary_poster> machines, on EC2.
<gary_poster> Failures look like this: http://ec2-184-72-208-228.compute-1.amazonaws.com:8010/builders/lucid_db_lp/builds/0/steps/shell_9/logs/summary
<gary_poster> I can duplicate on the EC2 machine trivially, just by running the first test in the lxc container in isolation.
<gary_poster> the store provides IMasterStore, not ISlaveStore.
<gary_poster> I'm digging in, but three reliably failing tests is a big step back for us
<gary_poster> so help would be appreciated.
<gary_poster> I'll send you a note at my EoD to let you know where we are with this, one way or the other
<stub> gary_poster: yo
<gary_poster> oh, hey stub!  I'm sorry you are still around :-)
<stub> so it went through ec2 and buildbot fine. I'm not sure why it fails there, but a start would be to confirm PG 9.1
<gary_poster> stub, yeah, I saw it passed everywhere else fine, and I have not been able to dupe locally at all
<gary_poster> and yes, just verified the verification: the machine is running 9.1
<gary_poster> well, to be precise--
<gary_poster> 9.1.4-1~lucid4
<stub> xx-dbpolicy.txt is the interesting one. Others will be fallout I assume.
<stub> We seem to be in master only mode
<gary_poster> right
<gary_poster> lp.services.webapp.tests.test_dbpolicy.LaunchpadDatabasePolicyTestCase.test_defaults is really just the same thing
<gary_poster> when I put a pdb in there, it says that the store implements master
<stub> getting timeouts on launchpad...
<gary_poster> yeah, me too, but it eventually loaded
<stub> gary_poster: It is valid to get the master store if you are asking for a slave store. We need to trace that to see why it is deciding to do that. For this, it would mean the lag is too high and it considers the slave useless.
<gary_poster> oh
<gary_poster> ok
<stub> +        streaming_lag = slave_store.execute(
<stub> +            "SELECT now() - pg_last_xact_replay_timestamp()").get_one()[0]
<stub> I think that is the only relevant line added
<gary_poster> stub, in dbpolicy.py, in getStore, self.default_flavor == 'master'.  Is that consistent?
<gary_poster> (this is in a pdb, I mean)
<stub> test_dbpolicy.py is probably better to debug this I guess.
<gary_poster> yeah, that's where I am
<gary_poster> I got the lp.services.webapp.dbpolicy.LaunchpadDatabasePolicy
<gary_poster> which in says the default_flavor is master, in code and in pdb, so I must have zigged when I should have zagged earlier, because it makes sense that the store should be a master once I'm here.
<gary_poster> s/which in/which/
<stub> gary_poster: So we attach the interface to the object we return from getStore? How could this happen then?
<gary_poster> stub, I'm currently exploring whether lib/lp/services/webapp/adapter.py(848)get() should not have a LaunchpadDatabasePolicy from StoreSelector.get_current()
<gary_poster> because once it does, afaict, we're on the wrong path
<stub> gary_poster: and in that test, yes, it should be default of slave because we are testing the Slave policy
<gary_poster> stub, so the following result is the wrong one--we should see some other policy class, right?
<gary_poster> ((Pdb)) p StoreSelector.get_current()
<gary_poster> <lp.services.webapp.dbpolicy.LaunchpadDatabasePolicy object at 0x134b76cc>
<stub>    def setUp(self):
<stub>         if self.policy is None:
<stub>             self.policy = SlaveDatabasePolicy()
<stub>         super(SlaveDatabasePolicyTestCase, self).setUp()
<stub> so yeah, should be a slave policy
<gary_poster> huh
<gary_poster> ok, I'll do my pdb in that setUp
<stub> I'd watch SlaveDatabasePolicyTestCase.setup
<stub> It will not set the policy if it is not None, to avoid stoping on subclasses having already set it.
<gary_poster> yeah that's what I meant; trying
<gary_poster> whoa!
<gary_poster> self.policy is not None
<gary_poster> it is LaunchpadDatabasePolicy.
<gary_poster> Who did that and why!
<gary_poster> oh
<gary_poster> it is a confusing test assembly
<gary_poster> stub, notice the failing test is actually lp.services.webapp.tests.test_dbpolicy.LaunchpadDatabasePolicyTestCase
<gary_poster> which has its won setUp
<stub> oh
<gary_poster> which sets the policy to LaunchpadDatabasePolicy
<gary_poster> So now I'm wondering why it is working everywhere else
<stub> DEFAULT_FLAVOR is decided on by the policy, it might be slave, might be master
<stub> line 246 of dbpolicy.py
<stub> That line should be hit, and I bet it isn't
<stub> the getReplicationLag call above it is the new code I landed.
<gary_poster> stub, the code I saw in pdb was getStore in the base class, not install
<gary_poster> line 100 (with line 92) were the pertinent bits
<gary_poster> oh
<stub> gary_poster: Can you run "SELECT now() - pg_last_xact_replay_timestamp()" on one of your test dbs if you have one available from a paused test?
<gary_poster> but then we set
<gary_poster> k
<stub> It should always be returning NULL since the database isn't replicated
<stub> Unless I've misunderstood something, and your dbs are being setup differently
<gary_poster> stub, I should be connecting to a db that looks something like launchpad_ftest_2391, yes
<gary_poster> ?
<stub> gary_poster: yes, that looks fine
<gary_poster> stub, I get 02:17:48.970632
<stub> 2 hours, 17 minutes.
<stub> Bum.
<stub> So how are you setting your databases up?
<gary_poster> using lpsetup, which call launchpad-database-setup or whatever it is called.  Lemme get that code for us...
<stub> nah, that is fine. it is the standard stuff.
<stub> Hmm...
<stub> I get null, lucid staging database 9.1.4...
<stub> But I think this means my landing needs to be reverted, and I need to detect if the slave database is using pg 9.1 replication before doing that check.
<stub> I'll double check the docs first.
<gary_poster> stub, fwiw, the database is setup in line 222 of http://bazaar.launchpad.net/~canonical-launchpad-branches/lpsetup/trunk/view/head:/lpsetup/subcommands/install.py
<stub> gary_poster: but parallel testing hasn't changed this?
<gary_poster> hasn't changed what, stub?  We're using the same LP tree as everyone else, if that's what you mean
<stub> database setup.
<stub> If you start with 'make schema', it should all be fine.
<stub> which is utilities/launchpad-database-setup, and that should be fine :-/
<gary_poster> stub, we do run make schema.  results of all of that per-build setup are here:
<gary_poster> http://ec2-184-72-208-228.compute-1.amazonaws.com:8010/builders/lucid_db_lp/builds/0/steps/shell_8/logs/stdio
<stub> Unless perhaps your database crashed at some point and started up in recovery mode? Which would mean it replayed logs, and that function would return something.
<gary_poster> We've gotten the LANG complaints forever.  We now know the cause but we haven't bothered to change anything
<gary_poster> because it's been working fine up to now
<stub> yer, I think I need to revert and not assume pg_last_xact_replay_timestamp() returns NULL on unreplicated systems.
<gary_poster> stub, where would I look to see if we have the crash you expect?  or do you not want to bother?
<stub> Don't bother.
<gary_poster> ok cool
<stub> If it can happen, it can happen.
<gary_poster> ok
<stub> This code needs to cope.
<gary_poster> gotcha
<gary_poster> stub, fwiw, I see this: http://pastebin.ubuntu.com/1051255/ .  That seems to align with the time frame that we are talking about.  I bet we are shutting down the lxc containers harder than postgres expects, so that's what it is complaining about.
 * gary_poster likes tidy explanations
<stub> gary_poster: Yes. So replication works the same was as recovery after a hard shutdown.
<stub> The wal files are replayed to ensure the data files are correct.
<gary_poster> gotcha
<stub> Which makes my replication lag check return a value
<stub> Bum.
<gary_poster> gotcha. :-/
<stub> I'll revert that again now
<gary_poster> ok thanks stub, and thank you very much for helping so far past EoD
<stub> np. I had a few hours in the middle off so am catching up :)
<gary_poster> :-) cool
<lifeless> stub: good $midnight?
<stub> 1am?
<lifeless> yowser :>
<stub> gary defeated my cunning plans
<lifeless> I see
<czajkowski> is launchad the team that never sleeps!
<lifeless> this is the team that goes on and on
<stub> gary_poster: If you haven't messed with your evil postgresql installation, can you run 'SELECT pg_is_in_recovery(), pg_last_xact_replay_timestamp()' for me?
<gary_poster> stub, sorry, you missed the window by seconds
<stub> doh.
<gary_poster> stub, although...
<stub> I'll try and break my local install then :)
<stub> unless...
<gary_poster> (1) I got trunk after your branch and things were fine again, and (2) if you give me about five minutes, I can log into a running instance (with tests running) and do stuff there if that helps
<gary_poster> or (3) I could kill the running tests and do stuff then.
<gary_poster> any of those of interest, stub?
<stub> Nah, I think I've reproduced it here (shutdown immediate with changes made but before checkpoint).
<gary_poster> cool
<wallyworld_> sinzui: wgrant: StevenK: jcsackett: link works now
<wgrant> sinzui: http://launchpad.net/~78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwfnr8jn6ln0m3jxne2k9x1ohg85w3jabxlrqbgs-cdpq7ln4t-a811i2i3ytqlsztthjth0svbccw8inm65tmkqp9sarr553jq53in4xm1m8wn3o4rlwaer06ogwvqwv9mrqoku2x334n7di44o65qze
<wgrant> (that's where 'launchpad' gets you)
<cjwatson> That's still the best URL ever
<wgrant> The 'cdpq7ln4t' is 'launchpad'
<wgrant> Mutated deterministically to avoid collisions.
<cjwatson> wgrant: My theory is that Launchpad is trying to beat bug 255161 for the funniest bug ever.
<_mup_> Bug #255161: I am unable to print from open office, I tried reinstalling open office but it did not work. I use a brother mfc240c printer and I am running Hardy. Printing from other apps has not been an issue. <OpenOffice:Invalid> <cupsys (Ubuntu):Invalid> < https://launchpad.net/bugs/255161 >
<wgrant> cjwatson: Ah yes, I remember that one.
<cjwatson> wgrant: Does http://paste.ubuntu.com/1051761/ look about right as a gina hack to make it not commit anything?
<cjwatson> For dogfood
<cjwatson> It has commits all over the place so it's a little awkward
<wgrant> cjwatson: I think that's correct.
<cjwatson> I won't try it until after I next regain consciousness, so there's a while to catch idiocies :-)
<wgrant> Yep.
<lifeless> wgrant: so, why don't we just count the matching things and add -(N+1) and be done with it ?
<wgrant> lifeless: Because seeding an RNG deterministically is more fun.
#launchpad-dev 2012-06-21
<StevenK> wallyworld__, wgrant: https://code.launchpad.net/~stevenk/launchpad/fix-branch-subscribe-open-team/+merge/111326
<wallyworld__> StevenK: otp, can look soon
<wallyworld_> StevenK: r=me
<wallyworld_> StevenK: i assume the existing tests look for the field error
<StevenK> wallyworld_: Yeah, that was added in branch-subscribe-aag
<wallyworld_> cool
<wgrant> cjwatson: I'll run gina with commits enabled in a sec to revert the DB archive to the state we have on disk.
<wgrant> Oh
<wgrant> gina might be even slower than I thought it was.
 * wgrant fixes.
<wgrant> Ah, no.
<wgrant>    ->  Seq Scan on distributionjob  (cost=0.00..3711.70 rows=1 width=70) (actual time=69.665..129.033 rows=4 loops=1)
<wgrant>          Filter: ((job_type = 3) AND (distroseries = 107) AND (json_data = '{"sourcepackagename": 72909, "parent_series": 50}'::text))
<wgrant> That seems... fragile
<wgrant> StevenK: wtf
<wgrant> ?
 * wgrant blames bigjools too.
<wgrant> Do either of you recall why we do that, and what will break when I make it stop doing that?
<StevenK> Context?
<bigjools> errr someone is filtering on json_data
<bigjools> ?
<wgrant> DSDJ creation seems to first check if there are any existing jobs
<wgrant> By byte-matching the JSON
<bigjools> sweet fuck
<StevenK> Twitch, there are columns for thatr
<StevenK> s/r$//
<wgrant> There aren't columns for that.
<wgrant> But it's also probably not necessary.
<wgrant> And it's slow and fragile.
<wgrant> So I think I might delete it and see what happens.
<bigjools> who is bzr blaming?
<bigjools> do not delete
<wgrant> No idea, I haven't found the code yet.
<bigjools> check the commit msg please
<wgrant> You ruin all my fun.
<bigjools> why the heck is lp-propose pushing a target branch if it doesn't exist?!
<bigjools> wgrant: your delete and revert reflex is well known
<wgrant> Oh, sneaky, DSDJ is in Soyuz while DSD itself is in Registry.
<StevenK> I think it was jtv
<StevenK> find_waiting_jobs in DSDJ's model
<wgrant>     # Look for identical pending jobs.  This compares directly on
<wgrant>     # the metadata string.  It's fragile, but this is only an
<wgrant>     # optimization.  It's not actually disastrous to create
<wgrant>     # redundant jobs occasionally.
<wgrant> Yeah
<wgrant> So it can be deleted without peril.
<StevenK> I can recall being a little dubious when that code landed.
<jtv> StevenK, wgrant: yes that was me.  Feel free to improve it.
<spm> o/ jtv
<jtv> Hi there spm!
<jtv> It's just a filter for curbing duplication of jobs.  The thinking was that with 1 item in the JSON, it's a pretty reliable match; with 2 items in the JSON, there's only going to be at most 2 jobs even if the keys are in random order; and with 3 items in the JSON, it's time to actually think what should be done.
<wgrant> Heh, true.
<jtv> If the filter is too costly, it can be removed â but first I'd assess the risk of duplicating DSDJs.  That was part of the work I was hoping to avoid with this simple check.
<jtv> Then again, 50ms doesn't seem like a huge cost.
<jtv> An index could be another way to go.
<lifeless> well
<lifeless> its racy
<lifeless> unless we up our isolation level
<wgrant> Only slightly, but yes.
<wgrant> Upping the isolation doesn't help, I don't think
<wgrant> Oh, during the running I guess it would, indeed.
<lifeless> yah
<lifeless> it would make read-its-there-don't-add-commit conflict.
<lifeless> unless of couse a specific DSDJ is idempotent
<lifeless> in which case, the race isn't important, as it cannot be stale, but OTOH a duplicate should be approx free because the worker can check whether the output is there already.
<wgrant> The jobs are fairly expensive due to librarian access.
<wgrant> But it's otherwise not a problem for them to be duplicated.
<lifeless> can they not determine that they completely finished and shortcut (assuming idempotency)
<wgrant> Indeed, they probably could avoid recalculating the base version if neither series' version has changed.
<czajkowski> morning
<czajkowski> jtv: hiya, heading offline for a few days, left message in pm
<czajkowski> see you all Monday
<cjwatson> wgrant: thanks for QAing that, I wasn't looking forward to it
<wgrant> It took about 3 hours for the first run, IIRC.
<wgrant> Subsequent runs take less than a minute.
<adeuring> food morning
<ivory> jelmer: could you review this? https://code.launchpad.net/~ivo-kracht/launchpad/bug-897571/+merge/111367
<jelmer> ivory: sure
<jam> wgrant: are you still around
<jam> hi jelmer
<jelmer> hellÃÂ¸ :)
<jml> is there an http equiv of 'bzr branch bzr+ssh://bazaar.launchpad.net/+branch/pkgme' (i.e. a fixed string that I can put before a project name to fetch its trunk?)
<jml> I don't want 'lp:' because it generates warnings about launchpad-login and this is an automated process.
<rick_h_> morning
<rick_h_> jml, so what I've done is branch with the lp: and bzr info to get the full url for that project
<rick_h_> is it for one or many projects?
<jml> rick_h_: thanks. that's a bit of a shame, because I'd rather depend on the slightly more stable lp:foo url equivalent.
<rick_h_> jml: it could be there for sure, I'm just so far from a bzr expert to say for sure. abently will be around in a couple of hours and he'd be able to tell for sure
<jml> rick_h_: many. thanks.
<mgz> jml: nope, there's no simple mapping, you need xmlrpc
<mgz> don't do this but...
<mgz> >>> from bzrlib.plugins.launchpad import lp_directory
<mgz> >>> lp_directory.get_lp_login = lambda: None
<mgz> >>> lp_directory.LaunchpadDirectory().look_up("er...", "lp:bzr")
<mgz> 'http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev'
 * jml would rather patch LP to have a proper rewrite map for http
<jml> but there are only so many patches to LP that I can justify.
<mgz> would be nice, wouldn't it. xmlrpc is an arse.
<jelmer> doesn't just http://launchpad.net/PROJECT work?
<jelmer> it seems to here
<jelmer> jml: ^
<jam> dpm: how's the translation stuff coming?
<jam> jtv, dpm: from what I can see the summary statistics pages between P and Q are virtually identical. So it looks like all the async jobs have done their work, and got Q into pretty good shape.
<jam> guess jtv isn't here now.
<jcsackett> is anyone else getting sporadically failing javascript errors on ec2? I keep getting hatemail with a ton of failures but when i run all YUI tests locally they all pass.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: jcsackett | Firefighting: - | Critical bugs: 3.47*10^2
 * jcsackett realizes it's thursday.
<rick_h_> jcsackett: been that way all week. Was so sad when the wife corrected me today that it's not friday
<dpm> jam, I'll talk to the ubuntu-translations-coordinators team, and unless they see anything, I'm aiming to open translations tomorrow
<sinzui> jcsackett, can you review https://code.launchpad.net/~sinzui/launchpad/bug-tags-edit-0/+merge/111410
<jcsackett> sinzui: just opened it, actually. :-)
<frankban> abentley: ping
<abentley> frankban: pong
<frankban> abentley: I am having troubles trying to fix a test isolation bug; if 3 tests are run in a predefined order I get a db DisconnectionError
<frankban> abentley: these are the 3 tests: http://pastebin.ubuntu.com/1052752/
<abentley> frankban: Okay, is that the order they must be run in?
<frankban> and the last raises "DisconnectionError: terminating connection due to administrator command. SSL connection has been closed unexpectedly."
<frankban> abentley: yes
<abentley> frankban: Can you post the traceback, please?
<jcsackett> sinzui: r=me.
<sinzui> faboo
<jcsackett> sinzui: so, given that a) i'm not that far in the new team form work, b) wallyworld_ seems to have all of that *well* in hand, and c) there's a critical in our next lane that's been sitting for awhile, i'm thinking of dropping my curent work and doing the critical.
<jcsackett> thoughts?
<sinzui> jcsackett. I do want it worked on...but do we know how to solve it at this moment?
<jcsackett> sinzui: i *think* so. want to chat about it?
<sinzui> We want to unsubscribe users that project does not share with, and are not in the bug-supervisor or security contact roles, but not break ubuntu
<jcsackett> right, but we had to put a flag in for the ubuntu case anyway. which means we can structure what happens around that.
<sinzui> jcsackett,  when the ubuntu community make a bug public, they expect all the subscribers to remain because they and Lp do not know who should have access.
<frankban> abentley: http://pastebin.ubuntu.com/1052761/
<jcsackett> sinzui: hrm; fair. i had not thought about the "return to public" part.
<sinzui> jcsackett, so solving the issue for everyone except ubuntu fixes 99% of the problem
<sinzui> jcsackett, community might toggle between public and private by accident. The ubuntu bug supervisor does not get subscribed (we know that for certain) so we just have to hope the community can mange the subscriptions themselves
<abentley> frankban: So, that exception is actually being raised on the Celery side, I believe.
<jcsackett> sinzui: so, on switch to a private type, do the remove all subscriptions except special roles thing, and just hope everyone sorts it out when it switches back?
<frankban> abentley: I agree, it is raised by a transaction.commit called by pop_remote_notifications
<abentley> frankban: Which probably indicates a bug in the Celery stuff.
<sinzui> jcsackett, not for ubuntu. their supervisor is never subscribed
<jcsackett> sinzui: right.
<jcsackett> but otherwise.
<sinzui> jcsackett, lets just have a guard for ubuntu...no unsubscribing. They continue in the compromised fashion
<jcsackett> sinzui: ok. so you're good with me tackling that and dropping the (now unecessary) new team form work?
<sinzui> every other project can be secure because Lp will unsubscribe everyone who is not in the legacy role or the info type is not shared
<sinzui> jcsackett, yep
<abentley> frankban: So, the Celery tasks do some hacky stuff to switch database users.  I'd assume that's the cause of the problem.
<jcsackett> cool stuff. :-)
<abentley> frankban: Are multiple tests doing pop_remote_notifications?
<sinzui> jcsackett, If my qa goes well today, I could return to bug tag manager. It requires a lot of markup that it could generate as progressive enhancement...this would be an incremental step to adding tag completion to +filebug and search which do not have the cumbersome markup.
<abentley> frankban: Also, you know that you can get the celeryd-side traceback from the result object, right?
<frankban> each test in that TestCase. and all the tests in that testcase call switch_dbuser(config.branchscanner.dbuser). But if I replace the last test with another one in the same TestCase everything is ok.
<frankban> abentley: yes
<jcsackett> sinzui: that would be awesome.
<jcsackett> now, i just have to figure out why js tests are failing on ec2 but passing locally for me.
<sinzui> jcsackett, what really?
<sinzui> jcsackett, lucid uses an older version of webkit that knows that ids are unique. It return null if you try to locate it. The webkit on out computers returns the first one
<jcsackett> sinzui: so, the require-id flag thing maybe the issue?
<sinzui> jcsackett, I am not sure. Recall we stopped the line a few months ago when wallyworld___ had a branch that always failed on lucid. The failure was legitimate because the page was creating bad ids, and the js only worked with the first item in a listing
<jcsackett> sinzui: i do recall that; but these are modules i haven't touched and that don't rely on pickers, so i feel like something else might be happening than the new picker stuff generating bad ids.
<sinzui> I saw the same problem in workitems a few weeks ago too
<sinzui> oh, which modules
<jcsackett> sinzui: http://pastebin.ubuntu.com/1052825/. all with JS timeout.
<jcsackett> can't get it to happen locally though. :-/
<sinzui> whoa
<jcsackett> i've sent of ec2 test -o '--layer=YUI' to see if it's a "whole suite" vs "yui suite" issue.
<sinzui> jcsackett, I am very surprised. gary_poster added incremental reporting to html5-browser that also resolved timeout issues where tests grew too large.
<jcsackett> sinzui: forget everything, i just figured it out, and i am an idiot.
<gary_poster> :-)
<sinzui> yui_xhr are slow even on my computer.
<gary_poster> yeah
<gary_poster> they are
<jcsackett> when i resent out the branch, not everything was pushed--it tested the old version, which had syntax issues.
<jcsackett> so, if this ec2 test run comes back green i'll lp-land, since it's all just YUI changes.
<sinzui> well done.
<jcsackett> yeah. i am smooth.
<jcsackett> note to self: always double check revid.
<gary_poster> sinzui, Easter Division?
<sinzui> jcsackett, I still use ec2 test and bzr pqm-submit in my working tree...they always abort if the revs are different
<jcsackett> sinzui: yeah. i'm not actually sure why ec2 land doesn't.
<jcsackett> perhaps that's something to change.
<cjwatson> Huh, I thought it did.  Doesn't it claim that it does?
<cjwatson> Certainly seems surprising for it not to
<sinzui> gary_poster, In the Five Star Stories manga, the heroes are members of the Easter division. I think it is a Japanese corruption on East
<gary_poster> it uses pqm-submits code to do that sort of thing
<gary_poster> :-) cool sinzui
<gary_poster> oh
<gary_poster> and ec2 land is just a wrapper around ec2 test
<gary_poster> at least that's what I saw when I looked at it last :-P
<sinzui> but you can run ec2 land from any directory. I think its goal it to land the approved version, not the current version
<gary_poster> oh
<gary_poster> maybe it changed then
<frankban> abentley: here is the traceback attached to the job: http://pastebin.ubuntu.com/1052839/
<abentley> frankban: So my guess is that it's committing using the old connection.  That seems to be the only way past operations could have an influence.
<frankban> abentley: I'll investigate it, thanks for the hint.
<replaceafill> hello everybody, i'm new to launchpad development, but i'd like to contribute fixing trivial bugs
<jelmer_> hi replaceafill
<replaceafill> i set up my environment with rocketfuel
<replaceafill> hi jelmer_
<replaceafill> and chose LP: #210821
<_mup_> Bug #210821: bug tracker list shows inactive projects <404> <bugwatch> <lp-bugs> <oops> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/210821 >
<jelmer_> replaceafill: great! How is it going?
<replaceafill> jelmer_, http://pastebin.ubuntu.com/1052858/
<replaceafill> that's my approach
<replaceafill> but i have a few doubts
<replaceafill> 1. it changes part of a model
<replaceafill> i'm not sure if it's better doing it at the "view level"
<replaceafill> (assuming using .active checks is ok)
<replaceafill> btw, sorry for my english, is not my first language :)
<jelmer_> replaceafill: using .active seems reasonable to me
<replaceafill> jelmer_, ah ok
<sinzui> replaceafill, that wont fix the bug
<sinzui> replaceafill, projects can be toggled from active to inactive, then back to active
<sinzui> It is okay for an inactive project to have a remote bug tracker
<replaceafill> sinzui, i tried it in launchpad.dev, marking the gnome terminal project as inactive, and it gets filtered in the bugtrackers view
<replaceafill> sinzui, ah
<jelmer_> sinzui: this is just about what projects are listed in the remote bug tracker's info page I think
<jelmer_> sinzui: wouldn't it make sense in that case to hide those projects that are inactive?
<sinzui> replaceafill, the problem is that queries to get the bugtacker/bugtask queries often forget to add Product.active == True
<sinzui> jelmer_, we do want to hide deactivate projects. That is either a query or a .active check in code iterating over a dumb result set.
<jelmer_> sinzui: isn't that the code replaceafill is changing though?
<sinzui> okay. I am mistaken. That works, but it is iterating over unused data. Both queries in the preceding lines could be fixed to make LP faster: http://pastebin.ubuntu.com/1052872/
<sinzui> jelmer_, replaceafill ^ is less work for Lp
<replaceafill> ah, i had the sense that it could be fixed that way :)
<replaceafill> but i don't know enough yet to change queries :(
<jelmer_> sinzui: that makes sense
<jelmer_> (does that really need raw SQL, btw?)
<sinzui> jelmer_, it should be storm, but this is very old code.
<replaceafill> sinzui, could i help writing a test for your solution?
<replaceafill> or maybe that's easy enough to write for you and i should move on to something else?
<sinzui> replaceafill, BugAlsoAffectsProductWithProductCreationView. _loadProductsUsingBugTracker is the last remaining origin of 404's for this case. BugTracker.product contains inactive products, and that view coverts the result set to a list without a filter
<sinzui> replaceafill, I think the test for your solution would be identical for either your solution of mine. We want to add a bugtracker to two products, make one inactive, then assert that .getPillarsForBugtrackers() returns one product
<replaceafill> sinzui, ah ok, can i help with that?
<sinzui> sure. I want to have lunch, so you can continue work on the bug. I did not intend to steal it from you.
<replaceafill> sinzui, :) np i understand
 * sinzui needs to do QA
<replaceafill> i'll dig into tests looking for examples to follow
<replaceafill> and will report back
<replaceafill> thanks sinzui and jelmer_
<rick_h_> ivory: the link in your email came up bad, can you dbl check it please?
* gary_poster changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: jcsackett | Firefighting: https://wiki.canonical.com/IncidentReports/2012-06-21-LP-codehosting-intermittent-hangs (gary_poster) | Critical bugs: 3.47*10^2
<jcsackett> sinzui: just found out i'll need to bail around 6:30 or 6:45 tonight. :-/
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: https://wiki.canonical.com/IncidentReports/2012-06-21-LP-codehosting-intermittent-hangs (gary_poster) | Critical bugs: 3.47*10^2
* gary_poster changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivory | On call reviewer: - | Firefighting: https://wiki.canonical.com/IncidentReports/2012-06-21-LP-codehosting-intermittent-hangs (lifeless) | Critical bugs: 3.47*10^2
* lifeless changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: https://wiki.canonical.com/IncidentReports/2012-06-21-LP-codehosting-intermittent-hangs (lifeless) | Critical bugs: 3.47*10^2
* lifeless changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<lifeless> doh, no sinzui
#launchpad-dev 2012-06-22
<lifeless> benji: around?
<lifeless> benji: would like to briefly talk about 'logs' w/you.
<lifeless> if not, thats fine too, we can sync another time, no rush.
<lifeless> ahahhaha
<lifeless> https://oops.canonical.com/oops/?oopsid=OOPS-c182f7f94210ede212ae5b835ef993c7#longstatements
<wgrant> lifeless: Hm?
<lifeless> 8.5 second query
<lifeless> to grab products
<lifeless> https://bugs.launchpad.net/launchpad/+bug/1016156
<_mup_> Bug #1016156: ProjectGroup:+index timeout due to slow query of subprojects <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1016156 >
<wgrant> lifeless: I don't think that's real
<wgrant> lifeless: Given that the same query took 4ms 6ms earlier.
<lifeless> indeed
<wgrant> xmlrpc-private
<lifeless> GIL interference is a possibility
<lifeless> is there a faq / updated docs for the new ec2 land setup ?
<StevenK> New as in lp-dev-utils?
<lifeless> yeah
<lifeless> time I fixed my landing setup
<lifeless> so I can land bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/sourcecode-meta
<StevenK> lifeless: Uses the same creds as when it was in bin/
<StevenK> lifeless: I have a checkout of lp-dev-utils in ~/launchpad and a symlink into it from ~/bin
<lifeless> I have questions like: is it a package, do we run it from source?
<lifeless> StevenK: and I'm sure other folk who join the team will need to know, which is why my *first* question was on docs ;)
<lifeless> StevenK: I do appreciate the direct help.
<lifeless> StevenK: I guess I"m trying to do two separate but overlapping things.
<StevenK> lifeless: https://dev.launchpad.net/EC2Test
<lifeless> thanks
<wgrant> lifeless' branch is cursed.
<wgrant> With an early MP, again
<wgrant> Hmmm
<StevenK> I've been waiting for the scanner before proposing
<StevenK> Which is annoying
<StevenK> Does dev log oops into /var/tmp/lperr?
<wgrant> If you kill rabbit, yep
<StevenK> And if rabbit is running?
<wgrant> You'll need to run an amqp2disk
<wgrant> So kill rabbit
<lifeless> or run an amqp2disk :P
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-bprc/+merge/111542
<wgrant> StevenK: Looking
<wgrant> StevenK: yay
<StevenK> wgrant: And https://code.launchpad.net/~stevenk/launchpad/drop-populate-branch-ap/+merge/111544
<wgrant> StevenK: You ran it on DF?
<StevenK> Running
<wgrant> Great
<wgrant> r=me
<StevenK> wgrant: Hm, I'm not sure it is useful to change samepledata for it. There are two private branches in it.
<wgrant> StevenK: Ah, indeed, that's a good point. Run it over both sets of sampledata.
<wgrant> And manually delete the scriptactivity rows that it creates.
<wgrant> (make sure you limit garbo to just that one job)
<StevenK> I did so for DF too
<wgrant> And do review the sampledata diff to make sure it's sensible.
<StevenK> Oh, default LPCONFIG and the other is testrunner or something?
<wgrant> LPCONFIG=development and LPCONFIG=test-playground
<wgrant> development is usually the default
<StevenK> wgrant: sampledata changes pushed to drop-populate-branch-ap
<wgrant> StevenK: LGTM
 * StevenK lp-lands
<rick_h_> lifeless: commented on the bug, will try to get a branch that nukes it out in the morning
<lifeless> rick_h_: \o/
<adeuring> good morning
<ivory> rick_h_: it's https://bugs.launchpad.net/launchpad/+bug/824292
<_mup_> Bug #824292: DistroSeries page: "Show uploads" vs "All uploads" <confusing-ui> <easy> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/824292 >
<cjwatson> StevenK: any further comments on https://code.launchpad.net/~cjwatson/launchpad/pcj-reupload/+merge/111124, or should it be "Status: Approved"?  I've given up on the removeSecurityProxy avoidance I wanted to do, although if you have any bright ideas on that I'm certainly listening.
<cjwatson> QA might be exciting.  I guess I'll need ops to create me a p3a on qas or something, when the time comes
<cjwatson> StevenK: I have http://paste.ubuntu.com/1053911/ in my local tree; but those are pre-existing bugs, I believe, and I think I might be better off fixing them in separate branches
<StevenK> cjwatson: I'm not certain the package copier even supports custom uploads.
<StevenK> cjwatson: However, I agree with you,the XXX's are fine for the moment, and you can address them in later branches.
<cjwatson> Those are uncommitted XXXs, but maybe I should commit them
 * cjwatson does so
<cjwatson> Direct copies don't support custom uploads, but I think delayed copies do.
<cjwatson> In fact I know they do.
<cjwatson> So it's sort of technically a regression; but it only matters if they publish d-i or dist-upgrader updates.
<cjwatson> Oh, hm, wait, does it matter for translation tarballs?
<cjwatson> Maybe I need to fix these in a separate branch *first* :-(
<cjwatson> Otherwise I can imagine hilarity if there's some kind of firedrill security update.
<jml> hello
<jml> with reference to my question yesterday,
<jml> 'bzr branch https://code.launchpad.net/$FOO' works
<cjwatson> Fortunately copying custom uploads was on my to-do list anyway, so I guess this just moves it around.
<lifeless> jml: we have a fake bzr branch metadata served by LP at that location.
<lifeless> jml: it will work for any series as well
<jml> lifeless: thanks.
* adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: adeuring | Firefighting: - | Critical bugs: 3.47*10^2
<jam> cjwatson: I've been tasked with trying to clean up some space from the librarian, and it was recommended that I talk to you about space being used by the primary archive.
 * cjwatson is on -ops too, so either way
<sinzui> yo, deryck, jcsackett, rick_h_. I am looking at the bug tag editor. It flashes green on failure, I think this should be red. It flashed red on cancel...even though the UI is restored
<sinzui> should I reverse the colours?
<jcsackett> sinzui: red on fail sounds good to me.
<jcsackett> was pretty sure that was our standard.
<rick_h_> +1 sinzui
<rick_h_> I think it's probably wired green because of successful ajax call or something like that. See that a lot
<sinzui> rick_h_, that is my conclusion as well
<deryck> yeah, agree with the above :)
<sinzui> gentleman, I have some confidence that I will add tag completion to +filebug and search today
<rick_h_> woot!
<jcsackett> sinzui: you're a hero. :-)
<sinzui> No, I am just tired is mis-spelling bug tags when I file and search for bugs
<flacoste> rick_h_: do you have a link to the morph component you want to send to the YUI gallery%?
<flacoste> rick_h_: codebrowse link would be fine
<rick_h_> flacoste: sure, sec
* adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
<mrevell> Hey deryck, got time for a quick call?
<mrevell> no worries if not
<deryck> mrevell, sure, I can.  Just you and I or someone else?
<mrevell> Just me :)
<deryck> mrevell, hangout?
<sinzui> jcsackett, I don't see reviewer about. Could you review this branch since it follows up on the branch you reviews yesterday. https://code.launchpad.net/~sinzui/launchpad/bug-tag-enhacement/+merge/111639
<jcsackett> sinzui: sure.
<rick_h_> sinzui: commented, thanks for that cleanup. Makes a bunch of sense.
<sinzui> thank you!
<sinzui> rick_h_, I look for support for passing a list of nodes
<sinzui> It did not seem to be supported
 * sinzui is happy to pass a list
<rick_h_> sinzui: yea, I thought you could just pass a list but I couldn't get that to work in the fiddle with 3.4, but a nodelist worked
<rick_h_> sinzui: the one tricky bit is that &nbsp; you had, so I wrapped that into a span so it was a node
<rick_h_> hopefully doesn't effect anything else
<sinzui> the nbsp is a hack. Maybe I want a css rule input + button {padding-left: .5em;}
<rick_h_> yea, probably better in the long run
<sinzui> I will try that
<rick_h_> so yea, if the NodeList works for you then you can also drop a bunch of the var XXXX at the top as well hopefully
<rick_h_> since I think they were just for HIDDEN class bits and then for your .append()'s
<sinzui> yep
<rick_h_> sinzui: can I bug you to peek at: https://code.launchpad.net/~rharding/launchpad/remove_mustache/+merge/111633 and sanity check that's all I need to do for removing that sourcedep
<jcsackett> sinzui: I see rick_h finished up whilst i was looking at it, but r=me.
<rick_h_> abentley: can you peek at the above MP ^ if you have a chance?
<abentley> rick_h_: sure.
<rick_h_> ty
<rick_h_> one liner ftw! but finding that one line...
<replaceafill> sinzui, i asked you yesterday about LP: #210821
<_mup_> Bug #210821: bug tracker list shows inactive projects <404> <bugwatch> <lp-bugs> <oops> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/210821 >
<replaceafill> sinzui, this is the test i wrote: http://pastebin.ubuntu.com/1054651/
<abentley> rick_h_: Do you think that forking Mustache is a good long-term solution?
<rick_h_> abentley: well, the long term solution is to replace handlebars with mustache in 3.5
<rick_h_> abentley:  possibly pybars, but in order to do combo loading we have ot subsume mustache into our build dir
<abentley> rick_h_: It was a symlink already.  There shouldn't be a problem putting a symlink into our build dir.
<rick_h_> abentley: right, but it's not a YUI module
<rick_h_> that's why we subsumed it, so it could be combo loaded/wrapped
<abentley> rick_h_: But you can turn it into a module as part of the build phase, right?
<rick_h_> I suppose we could generate the YUI file as a special case for that module.
<rick_h_> abentley: but in talks with deryck the decision was to move towards taking over/owning the JS we use such as the accoridion widget, mustache, etc
<abentley> rick_h_: I don't understand why that would apply to Mustache, but not to YUI, esp when YUI contains something some similar to Mustache.
<gary_poster> jml, do you mind if I link publicly to the fabulous but depressingly large checklist you are working on? https://docs.google.com/a/canonical.com/document/d/1A8qTOYZd7p9dhjmlnnEKW46h38ydt9yNbmfgX3yhPGw/edit
<jml> gary_poster: please go ahead
<gary_poster> thanks
<rick_h_> abentley: not following that last bit. Why we don't just copy YUI into tree?
<abentley> rick_h_: If we have a bug that we've forked Mustache, then I'm fine with landing this.
<jml> gary_poster: my desperate plea for help is only more likely to be answered for wider publicity
<sinzui> replaceafill, That looks good. I think `login(ADMIN_EMAIL)` should be `login_celebrity('admins')` which avoids using the sampledata users that is more than an admin
<gary_poster> jml, I thought as much. :-)
<abentley> rick_h_: Why are we taking over/owning Mustache, when we're not taking over/owning YUI?  They are both libraries that we use.
<sinzui> replaceafill, push your branch up and ask me to review it. I can land it with the change I requested
<replaceafill> sinzui, great, thanks!
<rick_h_> abentley: it's a matter of scope and YUI plays nice with the combo loader and mustache didn't. The original work to fork/take over mustache in tree is long gone/done. This just finishes the cleanup by removing the sourcedep
<rick_h_> this was gone over when I originally added Y.app.mustache and got the green light after those discussions
<rick_h_> abentley: I'll add a bug marking lp.app.mustache as needing to be deprecated as part of the yui 3.5 upgrade.
<rick_h_> and XXX the current lp.app.mustache
<abentley> rick_h_: I'd like a bug saying that we've forked mustache.
<abentley> rick_h_: Because there aren't any meaningful changes in our fork.  It's all just packaging.
<rick_h_> abentley: added 1016666
<rick_h_> sorry #1016666
<_mup_> Bug #1016666: launchpad contains a forked version of mustache.js <js> <yui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1016666 >
<abentley> rick_h_: It may be a bug that mustache is used at all, but that's a different bug from the fact that we have a spurious fork.
<rick_h_> abentley: sure, understand and added it. The two kind of fit together in my mind as I think about them together as a single issue
<rick_h_> but understand, bug added that we've got the fork at all
<abentley> rick_h_: to be clear: you've just now added two bugs?  What's the id of the second?
<rick_h_> abentley: no, one bug, just iddn't add the # for the bot so repeated
<rick_h_> I've only added a bug that we've forked handlebars and added a comment that there's a plan to move away from it.
<abentley> rick_h_: I've changed your bug to "launchpad uses mustache.js" and I'm filing a new "launchpad has a spurious fork of mustache.js".
<abentley> rick_h_: filed as #1016668
<_mup_> Bug #1016668: launchpad has a spurious fork of mustache.js <Launchpad itself:Triaged> < https://launchpad.net/bugs/1016668 >
<jcsackett> abentley, rick_h_: using mustache is a bug now?
<abentley> jcsackett: rick_h_ thinks we should deprecate it and move to handlebars.
<jcsackett> abentley: ah. okay, i'm down with that.
<abentley> rick_h_: r=me
<rick_h_> abentley: ty
<rick_h_> jcsackett: eventually it's the master plan with YUI 3.5 having handlebars included. Combo loader, YUI upgrade, no more mustache is the goal
<abentley> rick_h_: There's many a slip 'twixt the cup and the lip, i.e. we may not end up moving to handlebars.
<rick_h_> abentley: understand, why I'm not arguing the two bugs with you. They are separate concerns.
<abentley> rick_h_: Cool.
<abentley> rick_h_: In fact, we could stay forked, but push it up as a branch of lp:mustache.js.  That would be cleaner IMO, but would make sourcecode necessary again.
<rick_h_> abentley: right, this change is coming because lifeless wants to remove sourcecode and I'm working to help with that
* bac changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: bac | Firefighting: - | Critical bugs: 3.47*10^2
<abentley> rick_h_: What's the alternative to something like sourcecode?
<rick_h_> abentley: I'm not sure tbh. package all the things?
<abentley> rick_h_: We only have custom package for Python things, though.  I don't think we want python packaging of JS things.
<rick_h_> abentley: https://code.launchpad.net/~lifeless/launchpad/sourcecode-meta/+merge/111527
<rick_h_> abentley: well, there's work underway to package up yui itself, so that side will. Now outside of that yea, not sure how it'll be handled best
<abentley> rick_h_: package it up as python or deb?
<rick_h_> abentley: package yui 3.5 as a .deb
<rick_h_> abentley: I'm trying to ec2 land this and getting an error https://pastebin.canonical.com/68740/
<rick_h_> have you seen this with colo branches perhaps? It's my new dev setup though so might be something else I've not done right
<abentley> rick_h_: I think if you specify a non-local branch, you have to use the merge proposal url.
<rick_h_> I got the same error with the full url
<rick_h_> hmm, maybe I gave it the branch url, let me look
<rick_h_> ah yea, my bad
<replaceafill> sinzui, merge requested
<replaceafill> sinzui, let me know if you want me to change something
<replaceafill> sinzui, i'll look for another trivial bug :)
<sinzui> replaceafill, can you make a one line code change to login_celebrity('admin') so that sample data does not skew the results
<replaceafill> sinzui, oops, i forgot that, sorry
<replaceafill> will fix
<replaceafill> sinzui, done
<sinzui> replaceafill, I am landing the branch now
<replaceafill> sinzui, thanks!
<abentley> deryck: my mind is exploding because this assertion is failing: https://pastebin.canonical.com/68746/
<deryck> abentley, let me look....
<deryck> abentley, have you pdb'ed into it and looked at branch.unqiue_name vs what's in the keys?  maybe it's something as simple as one is unicode and one is a string, or some such?
<abentley> deryck: I've only been able to reproduce it in script runs so far, but they are actually different URLs.
<deryck> abentley, ah, hmmm, that is strange.
<deryck> abentley, so it looks it up by what's in the keys list, but then doesn't match after the fact?  that makes literally no sense. :)
<abentley> deryck: Here's an example: https://pastebin.canonical.com/68748/
<abentley> deryck: So branch unique name is u'~person-name-100007/product-name-100002/branch-100008', but it matched '~person-name-100007/product-name-100002/unique-from-test-rewrite-py-line267-100043'
<deryck> abentley, I'm not sure how that could happen. I'm sorry.  Maybe take a look at the entire result set rather than what is returned by the one call?  Just to get more ideas?
<abentley> deryck: The resultset is one item long.  ResultSet.one errors if there is > 1 result.
<abentley> deryck: (If you're expecting > 1, you use ResultSet.any)
<abentley> deryck: Aha!  transaction.commit fixes it.
<abentley> deryck: unique_name is kept up to date by a trigger, so I bet that's implicated.  Somehow.
<deryck> abentley, ah, yeah, that makes sense.
<deryck> changed out from under you.  yuck, that is hard to find.
<cjwatson> StevenK: I've thought some more about https://code.launchpad.net/~cjwatson/launchpad/pcj-reupload/+merge/111124, and tracked down bug numbers for the remaining XXXes; I have another branch up for review that addresses copying of custom uploads, and I expect I'll be able to fix the P-a-s handling first thing next week since I now understand what's going on there.  Regarding the change of the delayed default, all the ...
<cjwatson> ... non-test users are either explicit about what they want (in particular I was careful with Archive.syncSource(s)), are definitely OK with direct copies, or probably nobody has ever cared about whether they work with a private source archive (InitializeDistroSeries); so I'm confident at this point that the test suite will handle the rest.
<cjwatson> StevenK: So, if you'd be happy to update Status, I'd be happy to land it at this point.  I think the chances of a security update in the interim that needs to include a debian-installer or dist-upgrader custom upload are negligible, and even if it does happen the worst case is that we need to publish those manually (which we're already doing a fair bit of anyway, until such time as my copy-custom-uploads branch lands).
#launchpad-dev 2012-06-23
<cjwatson> And this would let us see for sure whether this is good enough for ubuntu-security, at which point it unblocks a huge pile of other stuff.
<cjwatson> (I will do a little dance if I get to land http://paste.ubuntu.com/1055207/.)
<StevenK> cjwatson: It has been tossed to Approved.
<cjwatson> StevenK: Thanks.  Off to EC2 now.
#launchpad-dev 2013-06-17
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/jobrunner-012/+merge/169344
<wgrant> StevenK: That already has my +1, doesn't it?
<StevenK> It does not
<StevenK> https://code.launchpad.net/~stevenk/lazr.jobrunner/only-run-if-canrun/+merge/169105 does
<wgrant> StevenK: William Grant code 2013-06-14 Approve on 2013-06-14
<StevenK> Bah
<StevenK> Should have refreshed before asking
<wgrant> That was 3 days ago!
<StevenK> Friday afternoon
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/export-pu-auditor/+merge/169714
<wgrant> StevenK: We need more infrastructure before we can do that
<wgrant> StevenK: It'll open two TCP connections for every packageupload.
<StevenK> wgrant: What sort of infra are we talking?
<wgrant> StevenK: Preloading with multiget
<StevenK> Hm
<StevenK> I think multiget will result in requiring changes to auditorclient
<wgrant> Oh, it didn't have it from the start? :/
<StevenK> Yeah, auditorclient is a very simple send/recieve
<StevenK> q = q.filter(operation__in=request.GET.getlist('operation'))
<StevenK> Hm, so it might work with a list of operations
<StevenK> wgrant: I'll have a play with multiget
<StevenK> (Pdb) client.receive(obj=pu, operation=('packageupload-accepted',))[]
<StevenK> Doesn't look good :-(
<StevenK> (Pdb) p '%s/fetch/?%s' % (self.auditor, urlencode(params))
<StevenK> 'http://localhost:54313/fetch/?object=lp-development%3APackageUpload%3A16&operation=%28%27packageupload-accepted%27%2C%29'
<StevenK> I think that's a miserable failure
<StevenK> wgrant: Of course, this is all confounded by auditorclient not having a testsuite
<StevenK> Blah
#launchpad-dev 2013-06-18
<jelmer> 'morning StevenK
<StevenK> jelmer: O HAI
<jelmer> how's things in Launchpad land?
<jelmer> I see you guys are finally making progress with the number of critical bugs now that you no longer have the rest of us breaking stuff.
<StevenK> Haha
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/new-auditorclient/+merge/169971
<wgrant> StevenK: Are there tests for multiple objects too?
<StevenK> wgrant: I can fiddle the test to have person-undeleted on a second obj
<StevenK> ... which fails
<StevenK> So, debugging after lunch then
<StevenK> wgrant: http://pastebin.ubuntu.com/5776023/
<wgrant> >>> isinstance([], (list, tuple))
<wgrant> True
<wgrant> >>> isinstance((), (list, tuple))
<wgrant> True
<wgrant> >>> isinstance(1, (list, tuple))
<wgrant> False
<StevenK> wgrant: http://pastebin.ubuntu.com/5776036/
<StevenK> wgrant: You fail, btw. () is no tuple
<wgrant> () is a tuple
<wgrant> (foo) is not
<StevenK> I thought tuples required a , as in (,)
<wgrant> () is unambiguous
<wgrant> (foo) requires a comma because it is ambiguous
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/new-auditorclient/+merge/169971 has updaed
<StevenK> *updated
<wgrant> StevenK: OK
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/sha256-indices/+merge/169995
<StevenK> wgrant: r=me, but what is your plan WRT NMAF archives?
<wgrant> StevenK: What do you mean?
<wgrant> This is all about NMAF
<StevenK> Er
<StevenK> non-NMAF archives is what I actually meant.
<wgrant> That's an apt-ftparchive thing
<wgrant> Not our problem, and I believe it's already fixed
<StevenK> Right
#launchpad-dev 2013-06-19
<StevenK> wgrant: http://pastebin.ubuntu.com/5779460/
<wgrant> +        result_set = IStore.using(*origins).find(
<StevenK> wgrant: It's already 833 lines, should I keep going in the same branch?
<wgrant> Well, landing this one won't work very wel
<wgrant> l
<StevenK> Oh?
<wgrant> IStore is an interface
<wgrant> Calling .using() on it isn't going to be entirely successful
<StevenK> Yeah, I fixed that
<wgrant> Also, I would consider moving IStore and co. to l.s.d.interfaces
<wgrant> lpstorm is a silly name.
<StevenK> wgrant: Do you want it all in one branch? The diff is 833 lines, and that's only 20 files.
<StevenK> There's another 91 to go.
<wgrant> I'm hardly going to be reviewing the whole thing either way
<StevenK> Oh?
<wgrant> That sort of thing is extremely tedious and almost entirely pointless to review
<wgrant> It's so similar that I would miss any mistakes.
<StevenK> wgrant: So, do it all in one-shot, and throwing through ec2 should catch a fair bit
<wgrant> Right, this is the sort of branch that you sed and then throw at ec2.
<wgrant> Once tests past, read the diff to make sure it makes sense
#launchpad-dev 2013-06-20
<StevenK> wgrant: 7000+ line diff
<wgrant> StevenK: Exactly my point
<StevenK> wgrant: I didn't quite eradicate all of it
<StevenK> wgrant: http://pastebin.ubuntu.com/5782604/
<StevenK> lib/lp/services/database/doc/storm.txt, lib/lp/services/database/interfaces.py and lib/lp/services/webapp/adapter.py can be ignored since they use and/or implement it
<wgrant> StevenK: Right
<StevenK> wgrant: But killing it from test_login, xx-private-builds and testing/factory would be nice, I just didn't see how to.
<wgrant> StevenK: I'm confused
<wgrant> Why would you kill it from them?
<wgrant> It's not using it in the way that there are replacements for
<wgrant> Their usage of it is not deprecated.
<StevenK> Right
<lifeless> wgrant: StevenK: seen githubs new repository-next UI ?
<lifeless> no longer shows the first para of README.md at the top of the page
<wgrant> Yeah
<wgrant> And they dropped LP 2.0-style facet tabs
<jtv> There's a branch here that used to be imported from sourceforge, but now the Launchpad version should become the "official upstream."  How do I do that?
<jelmer> jtv: push up a clone of the imported branch to a new location, then change that to be the development focus, then remove the import
<jelmer> (the latter might be hard because of stacking, but you should be able to rename it)
<jtv> jelmer: OK, thanks!
<StevenK> wgrant: http://pastebin.ubuntu.com/5785333/
<wgrant> StevenK: Are you in the right layer?
<StevenK> wgrant: Given the callsite was returning IStoreSelector.get() before I changed it, I guess the component arch is available
<wgrant> StevenK: Other callsites can do it fine
<wgrant> However
<wgrant> I would potentially not port that callsite.
<wgrant> StevenK: Look at that test
<wgrant> The problem is quite clear.
<StevenK> Hah
<StevenK> Right
<StevenK> It has to use IStoreSelector
<wgrant> Or the test needs to be changed
<wgrant> But for that level of infrastructure it makes sense to use IStoreSelector directly.
<StevenK> Yeah. I've reverted that back.
#launchpad-dev 2013-06-21
<StevenK> wgrant: http://pastebin.ubuntu.com/5785377/ is the last failure, but the tests are very strange
<wgrant> StevenK: Which test is that?
<StevenK> lp.testing.tests.test_layers_functional.LaunchpadTestCase.testLibrarianWorking
<wgrant> But anyway, the problem is fairly clear
<wgrant> It expects an AttributeError or ComponentLookupError when the ZCA is not loaded
<wgrant> But now it gets a TypeError
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bug-1193157/+merge/170722
<StevenK> wgrant: Looks good, r=me
<cjwatson> StevenK: Ooh, just noticed bug 1188034.  Excellent.  Do you have client code already or should I write one for ubuntu-archive-tools?
<_mup_> Bug #1188034: Should be possible to upload chroots over the API <distributions> <lp-soyuz> <qa-ok> <soyuz-ftpmaster-tools> <Launchpad itself:Fix Released by stevenk> <https://launchpad.net/bugs/1188034>
<cjwatson> (I'm happy to do so)
<cjwatson> StevenK: Archive owner is an interesting choice for the permission.  You didn't consider launchpad-buildd-admins instead, which I think was what infinity and I were expecting to use?
<StevenK> cjwatson: infinity requested archive owner
<StevenK> cjwatson: My client code was a horrible hack, so please write a better one
<cjwatson> OK :)
<cjwatson> Do we want an equivalent of 'manage-chroot.py remove'?
<cjwatson> I think that's the only piece missing to be able to kill that script
<cjwatson> BTW I have a test now that exercises the case of copies into private archives owned by private teams causing /builders to 403, so hope to be able to push up a branch fixing that soon
<StevenK> cjwatson: Hmmm
<StevenK> cjwatson: data=None, perhaps? I suspect it will need a small code change
<cjwatson> It might be clearer as a separate method
<cjwatson> Particularly since sha1sum is a weird thing to have to pass with None
<cjwatson> But just for clarity in general
<StevenK> Hmmm, indeed
<StevenK> cjwatson: The QA with that method wasn't exactly smooth on qas, so I'd like to see a test against prod with someone who doesn't suffer from Australian internet
<cjwatson> Maybe infinity fancies doing a chroot refresh
<StevenK> I'd rather something like hardy i386 or something known just in case it does go pear shaped, TBH, but your call
<cjwatson> EC2 gave a few errors for one of my recent branches, most of which I've fixed; they were unreliable tests rather than anything I'd caused, I think.  I don't understand http://paste.ubuntu.com/5787381/ though - is that one that's known to be unreliable?
<wgrant> cjwatson: Yeah, that one's known to be unreliable.
<wgrant> You might have also run into two ddeb failures that I think are arch-dependent but I haven't bothered to fix yet.
#launchpad-dev 2013-06-22
<maxiaojun> is bzr dead
<cjwatson> wgrant: Yep, I encountered and fixed one of those sets of ddeb failures, and the other failure was a timing-dependent failure in lib/lp/soyuz/stories/soyuz/xx-person-packages.txt.
<cjwatson> wgrant: The "Unembargoed builds" section of lib/lp/soyuz/stories/soyuz/xx-private-builds.txt is deliberately testing a property that my branch makes not true.  Do you think perhaps I ought to be testing just for visibility of the archive owner, rather than of the archive?
<cjwatson> Since a private team means the very existence of the archive is confidential, while a private archive whose builds are being copied elsewhere is not quite so confidential.
<cjwatson> I'm not sure whether this is test-highlighting-wrong-design or fundamentally-wrong-test.
<cjwatson> It is, I suppose, slightly weird that with my branch you can read the +build if you have the link, but /builders refuses to link to it ...
<cjwatson> Archive.name is always public, just not necessarily Archive.owner.name.
<wgrant> cjwatson: Right, your branch isn't strictly correct, but I felt it was correct enough that we should perhaps not overcomplicate things.
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/testfix-builders-visibility is more correct in that sense, if that's a direction you're happy with.
<wgrant> Checking visibility of archive.owner would work, but I'm not sure it's worth the complication.
<wgrant> Maybe
<wgrant> Really we need to have a definition of build privacy that isn't completely insane
<wgrant> But such is launchpad
<wgrant> cjwatson: Besides being slightly less obvious, that seems fine.
<cjwatson> Arguably this belongs in ViewBinaryPackageBuild or something, but then I'd have had to do something separate for recipes ...
<wgrant> It's a pretty unfortunate hack either way, so I'm glad it's restricted to the view :)
<cjwatson> And there's possibly an argument that it's the formatter that's trying to poke about inside the build's archive and therefore the formatter should also be the thing that checks whether it's OK to do so
 * cjwatson prods the MP button
<cjwatson> I suppose an alternative contention is that, if a build's source has been copied to a public archive, you should be able to see the build but not know about the private archive it's building in.  But then links are weird and the build log probably leaks the archive anyway ...
<cjwatson> Further alternatively: a saner way to unembargo things would possibly be to copy-and-mangle the build to excise references to its original archive.  (But then that's editing history and there's still the build log problem.)
<cjwatson> The definitional problem is kind of inherent to unembargoing, really.
<wgrant> cjwatson: Yeah, IMO you should be forced to disclose the existence of your team and archive once you copy anything out of it
<wgrant> Anything else is just too messy
#launchpad-dev 2013-06-23
<jelmer> wgrant, StevenK: does either of you have access to edit the launchpad blog?
<lifeless> jelmer: they should
<wgrant> jelmer: I do
#launchpad-dev 2014-06-16
<cjwatson> wgrant: I'm tempted to just disallow LiveFS:requestBuild for private archives for now, as the issue with visibility of dependencies is going to take a while to get right (it can't all be done in requestBuild and dispatch, because the set of people who can see a LiveFSBuild might change later).  Does that make sense to you?
<cjwatson> PES are likely to need it, but not immediately.
<wgrant> cjwatson: Agreed.
<wgrant> The virtualizedness is the bigger problem.
<cjwatson> Haven't got to that yet; working through your review in order ...
<wgrant> Oh
<wgrant> You're in for an unpleasant surprise :)
<cjwatson> Well, it's a problem for opening it up to wider groups such as PES
<cjwatson> It's not particularly a problem with the pretty restricted set of people who can create/edit livefses/livefsbuilds at the moment
<wgrant> That's true, but it might indicate that this model is fatally flawed.
<cjwatson> I want to say that only people who can upload to some devirtualised archive should be able to request livefsbuilds, although most requests will be done by a bot user
<wgrant> Though I guess the consumers are few enough that we can easily change it later.
<wgrant> Right, that might be a sensible rule.
<cjwatson> Sorry, should be able to request livefsbuilds for a devirt archive, I mean
<cjwatson> (actually not a problem for PES, since they basically all have access to devirt archives already)
<cjwatson> wgrant: So a "has any permission on any devirt archive" query is simple enough; what would you think of artificially giving ~ubuntu-cdimage-robot a devirt PPA just so that it passes this test and can run our normal image builds?
<cjwatson> It would keep the code simple at the cost of a bit of deployment WTF
<wgrant> cjwatson: Have you thought about a proper long-term solution for this? I haven't managed to come up with one.
<wgrant> The cross-archive permission query isn't sustainable long-term, so I'm hoping we aren't backing ourselves into a corner here.
<cjwatson> Long-term I'd hope we can do all livefs builds in scalingstack ...
<wgrant> True, but it's similar in many ways to the privacy issue.
<wgrant> But for the immediate term, that workaround sounds reasonable.
<cjwatson> The simplest alternative that comes to mind is to have livefs virtualisation be an admin-settable property of some kind, rather than deriving it.
<wgrant> Yup, but that doesn't solve the privacy issue which is basically the same problem.
<cjwatson> We'd then have to disallow building a given devirt LiveFS against a virt archive.
<cjwatson> Maybe it would simplify things if you were likewise not allowed to build a public LiveFS against a private Archive.
<cjwatson> The same kind of way a public Archive isn't allowed to have a private ArchiveDependency.
<cjwatson> So we could just say that if you're building against a private Archive then the LiveFS owner has to match.
<wgrant> The ArchiveDependency situation is just marginally less bad than what was there before (no restriction).
<wgrant> Right, I'd suggest that owner of the dependent object should have to be able to see the archive.
<wgrant> At build-time.
<wgrant> But for private PPAs that has to be slightly stricter.
<wgrant> Possibly the LiveFS owner must have upload permissions
<cjwatson> Requiring identity would mean that we don't have to worry about what happens if somebody gets added to one side or the other.
<wgrant> Which then also solves the virtness issue, so I've been wondering if we can reasonably make that a normalr estriction.
<cjwatson> For build logs later.
<wgrant> True.
<cjwatson> Any permission rather than just upload permission would be a bit easier logistically.
<cjwatson> But it winds up much the same.
<wgrant> You mean ArchivePermission, I assume
<wgrant> ie. some kind of write access
<cjwatson> Yes
<wgrant> That's what I intended, yeah
<wgrant> But "any permission" could be misconstrued to include ArchiveSubscriber, which is how ArchiveDependencies are now I think.
<wgrant> Maybe?
<wgrant> Anyway.
<cjwatson> It potentially restricts our ability to use new teams for LiveFSes, but maybe that's OK
<cjwatson> I'm just a bit worried about having to add ArchivePermissions for new teams to Ubuntu in order to be able to build livefses
<cjwatson> For non-distro archives, whatever
#launchpad-dev 2014-06-17
<cjwatson> One bot with queue admin on Ubuntu is quite enough thankyouverymuch and I'd like to try to keep privileges at least slightly least
<wgrant> It all becomes a lot simpler when scalingstack is everywhere, but we still need to make private archives non-hideous.
<wgrant> If possible.
<cjwatson> So maybe we can still have somewhat different rules for public and private archives
<cjwatson> But trying to have a common class of solution for private and devirt means either something that looks for any devirt archive, or adding an ArchivePermission to Ubuntu primary, or a special-case hack of some kind
<cjwatson> I'd managed to avoid adding a celebrity so far ...
<wgrant> https://docs.google.com/a/canonical.com/document/d/1F1wh8MxaxC-pSx5yMsFNpKFm5Mytsvn0Ugw2AIgQXzU/edit#
<cjwatson> I was just writing up something too, only in vim :P
<wgrant> vim sadly isn't easily multi-user.
<wgrant> As much as I'd prefer it :)
<cjwatson> Right, just lots more pleasant to use when my mirror sync and backups are both running
<cjwatson> But let's see
<wgrant> Heh
<cjwatson> Any LiveFS can be built against a public archive.
<cjwatson> To build a LiveFS against a private archive, the owners must match exactly.
<cjwatson>  => registrant is in common owner => registrant can see archive
<cjwatson> was what I had so far
<cjwatson> LiveFS gains a require_virtualized column, set by admins as for PPAs.  (This is a bit more cumbersome, but lets us vet owners, and LiveFSBuild : LiveFS :: PPA builds : Archive, after a fashion.)
<wgrant> Right, the require_virtualized thing is hideous, but hopefully ~temporary.
<wgrant> The private archive restriction is hopefully not terribly onerous.
<wgrant> And can always be relaxed later, I suppose, if we run into real problems with PES.
<cjwatson> Even though that means the answers to the two problems are quite different rather than paralleling each other, I think that's actually sufficient given the existing LiveFS.requestBuild security
<wgrant> Having such a security-sensitive flag duplicated on another table is awful, but hopefully of limited life due to scalingstack taking over the world.
<wgrant> So I'm not as far against it as I was late last year, when everyone was "omg we can't do scalingstack for Ubuntu the world will be on fire"
<cjwatson> It's sort of duplication but not entirely
<wgrant> It's another class of objects that we have to check for terrible security holes.
<wgrant> In terms of nagios checks for owners and such.
<cjwatson> Yes, that's true, I should dig those up for comparison.  Are they in puppet?
<wgrant> But I think those two solutions are workable for now.
<wgrant> I'm not sure if they actually exist in any particularly current fashion. There are RTs which suggest they might not actually work.
<cjwatson> Yay.
<wgrant> Anyway, sounds like this should be relatively easy to implement for you?
<wgrant> Just need to ensure that the permission checks occur at dispatch time (as well?)
<cjwatson> Trying to rationalise this: a write permission check on the archive helps for privacy (buildd secret), but is wrong for virtness because really we're only reading from the archive and might well need to do a livefs build on devirt hardware for make-it-work reasons but with a virt PPA as a dependency.
<wgrant> Though I guess the lack of retries means that's not such a huge issue, still.
<wgrant> Right, that sounds reasonable.
<cjwatson> Yes, I can do this tomorrow.  I have indeed got the message that it needs to be done at dispatch time. :-)  Worth doing at least lightweight checks (and probably all of this is sufficiently lightweight) in the model on requests as well.
<wgrant> Definitely.
<wgrant> It's all pretty lightweight now you're not doing a hideous query over every ArchivePermission evar.
<cjwatson> SSD DBs baby
<cjwatson> or maybe not
<wgrant> Maybe before the heat death of the universe.
<cjwatson> I've done the rest of your review, so will just need to go round again and make sure I haven't broken the browser code, and make sure it still works end-to-end
<wgrant> cjwatson: I'm just wondering how likely it is that people will shoot themselves in the foot by building some random PPA on a non-virt LiveFS.
<wgrant> s/themselvesk in the foot/us in the face/
<cjwatson> Well, the most important use case for building a LiveFS against a PPA is the CI engine stuff
<cjwatson> Secondarily, letting flavours run short-term experiments
<cjwatson> The first is already all devirt, and perhaps we can just say that for the second you get to copy the LiveFS to a require_virtualized=True flavour
<wgrant> Yeah, exactly.
<wgrant> The only cases in which it really makes sense to do a nonvirt livefs on a virt PPA are narrow
<cjwatson> And then say that if LiveFS.require_virtualised is False then so must Archive.require_virtualised be.
<wgrant> Arch-indep only changes, and old Xen kernels
<wgrant> And the latter is going to go away in a couple of weeks i hope.
<wgrant> So I think that restriction would be sensible.
<cjwatson> Certainly don't think it makes sense to design this around the Xen constraints
<cjwatson> Kubuntu want to do PPA-based livefs experiments in the not too distant future
<wgrant> Yes, mostly documenting that so I can review IRC logs when in 18 months I wonder why we made stupid decisions.
<cjwatson> But I think we can hold that off for a while
<cjwatson> The CI engine stuff can't really wait
<wgrant> CI is all non-virt
<wgrant> Presumably Kubuntu would have to be too.
<cjwatson> Exactly
<cjwatson> Well
<wgrant> Or they'll be missing powerpc packages
<wgrant> In which case they wouldn't want powerpc ISOs anyway
<cjwatson> I'm not sure they care about powerpc for the experiments in question
<cjwatson> I haven't really analysed it but I suspect they could go all virt
<cjwatson> Which would save us from having to deal with the devirt => Canonical restriction
<wgrant> Right, but the only interesting case is a mixed one.
<wgrant> And Kubuntu doesn't seem to require that.
<wgrant> Nor does CI
<wgrant> And I can't think of any that do.
<cjwatson> The ones I can think of are quick experiments - "what happens if I build an image based on this change", outside the CI system
<cjwatson> But we could have people copy the livefs for that
<wgrant> Right, and they already have to copy if they don't participate in the livefs owner.
<wgrant> So copies have to work well anyway.
<cjwatson> Or even just say that if you try to build a LiveFS against a virt archive then the build ends up virtualised too.
<wgrant> Ah, that would work, indeed.
<wgrant> A LiveFS build is non-virt iff its LiveFS and Archive both are.
<cjwatson> It's require_virtualized not require_devirtualized, so it can be implicit in that direction.
<wgrant> I think those were the only thorny issues in the review, weren't they?
<cjwatson> There were a few things I had to slightly guess at how to implement correctly, but nothing else was fundamentally hard, no.
 * cjwatson sleeps, thanks for the help
<wgrant> Night, thanks for working this out
<wgrant> I'll hopefully approve your UI branch today, now that we know model changes aren't required.
<wgrant> stub: https://code.launchpad.net/~wgrant/launchpad/ppa-reset-2.0-db/+merge/223395 could use a review some time tomorrow, if you've time.
<stub> wgrant: k
<wgrant> Oh, you're still alive.
<stub> wgrant: what does a null vm_reset_protocol mean?
<wgrant> stub: Same as null vm_host -- incomplete setup if the virtualized flag is set
<wgrant> We'll refuse to dispatch in that case, as we do with vm_host
<wgrant> I could add a CHECK constraint to that effect, but then I'd have to fix all the tests that violate that constraint with vm_host already.
<wgrant> And given this will hopefully all go away within 12 months...
<stub> Yup. and unlikely worth adding the constraints for that, if we can.
<wgrant> stub: Thanks.
<cjwatson> wgrant: I believe I've implemented all the livefs stuff from last night (including a db-livefs change) and fixed up livefs-browser to match.  Just running an end-to-end build now.
<cjwatson> wgrant: But should be ready for re-review of the changes.
<cjwatson> wgrant: End-to-end build test still works.
<wgrant> cjwatson: Lovely, let me see.
#launchpad-dev 2014-06-18
<cjwatson> wgrant: \o/
<wgrant> cjwatson: http://derived-archive.dogfood.content.paddev.net/ is now a thing, btw. So we should be set up for QA there
<wgrant> (though DF's all dead at the moment for some translations experiments)
<cjwatson> Nice, thanks.
<wgrant> cjwatson: So you intend to ndt 17048 some time today?
<cjwatson> Yep, basically as soon as it's passed both buildbots
<wgrant> Great
#launchpad-dev 2014-06-19
 * danilos -> lunch
#launchpad-dev 2015-06-15
<blr> wgrant: are webhooks at a point now where I could look at building out a view perhaps?
<wgrant> blr: Not at the moment -- hopefully later in the week. Is setbranch all done from your perspective/
<blr> wgrant: yep, the cherry picked link name changes are here: https://code.launchpad.net/~blr/launchpad/project-configuration-rename/+merge/261918
<blr> once that's landed, imagine it will be much easier to review the other branch
<wgrant> blr: Oh, can you flip that out of WIP?
<blr> wgrant: gah, sure thing
<wgrant> One day things like this will be sensible.
<wgrant> eg. bugs will say whether they're open or closed
<wgrant> And MPs will do the same, except with an additional state that is neither.
<wgrant> Rather than expecting everyone to know and notice the particular statuses...
<blr> that sounds sensible
<blr> can you think of anything that needs addressing atm? (otherwise I'll find some more bugs to poke at)
<wgrant> blr: One thing we need is a ref picker.
<wgrant> blr: For +register-merge, mostly.
<wgrant> Currently you have to type the target ref manually.
<wgrant> It's the most annoying bit about Git MPs atm.
<wgrant> https://app.asana.com/0/29290342588691/37033052254451
<blr> wgrant: great, will have a look after I tidy up this MP.
<lifeless> asana?
<wgrant> lifeless: Lightweight task tracking service.
<lifeless> wgrant: using it for LP development?
<wgrant> lifeless: Where it makes sense, yes.
<lifeless> seems a little ironic ;)
<wgrant> lifeless: No moreso than using kanban like in the old days.
<wgrant> LP bugs is good for tracking some types of things.
<wgrant> blr: Are you around to fix the broken test? If not, I'll do it.
<blr> wgrant: back sorry
<wgrant> blr: No worries.
<blr> wgrant: https://code.launchpad.net/~blr/launchpad/project-configuration-rename/+merge/261930
<wgrant> blr: Thanks.
<blr> wgrant: buildbot happy, will get the other branch merged.
<blr> morning
<wgrant> Morning blr
<wgrant> cjwatson: Do you believve the GitHostingClient regressions to all be fixed now?
<blr> wgrant: did something blow up, or are you just up early? :/
<wgrant> blr: Just an early meeting.
<blr> ah good
<cjwatson> wgrant: Yes - they weren't regressions (well, except from unmerged code) since they only affected the patch case, and I've confirmed that to be working now
<wgrant> Oh, yeah.
<wgrant> No empty responses yet.
 * wgrant gets a lot of coffee and dives back into blr's setbranch
<blr> wgrant: ah thanks
<wgrant> blr: Thanks for cutting the irrelevant bits out of the diff.
<wgrant> It's still 1700 lines, but that's a lot better than 2300 :)
<blr> hopefully easier to deal with now
<blr> yes :)
<wgrant> And the changes are mostly to a few files rather than EVERYWHERE.
<blr> in retrospect I should have made the renaming changes first... will try to think about decomposing branches before going down the rabbithole next time
<blr> wgrant: when looking at it again yesterday, I did wonder if it needs an additional test for the setDefaultRepository case
<wgrant> blr: It's often difficult to foresee quite how deep the rabbithole is.
<blr> wgrant: just lodging a bug suggesting that the test factories should use the first available Person.name rather than throwing an exception - is that reasonable?
<blr> introducing more randomness to the property is probably the only way to manage it without a db query
<blr> wgrant: if you have any comments https://bugs.launchpad.net/launchpad/+bug/1465438
<mup> Bug #1465438: Factories should ideally use first available Person.name <test-system> <Launchpad itself:Triaged> <https://launchpad.net/bugs/1465438>
<wgrant> blr: It shouldn't cause a serious performance regression; NameAlreadyTaken is raised when getByName returns something, so it's already performing an extra query.
<wgrant> blr: The login code uses generate_nick to create a unique name.
<blr> wgrant: a minor annoyance. I'll make a note about generate_nick thanks
<wgrant> Yeah, it's quite irritating, but usually only a problem in the harness rather than in tests.
<wgrant> Could also include a timestamp in the generated names (it doesn't just affect Person), or possibly only do that outside tests.
<wgrant> But tests shouldn't be depending on the specifics of the name generation, so always including something unique would probably be fine.
<blr> wgrant: yep, that seemed like the least obtrusive way to improve it
#launchpad-dev 2015-06-16
<blr> huh, `ag` can find zope templates with --plone
<blr> wgrant: thank you for the review (imagine that took some time!)
<wgrant> blr: Yup, mostly tiny things.
<blr> wgrant: do we want the target ref to default to heads/master?
<blr> context +register-merge ref picker
<wgrant> blr: It should default to the target repo's HEAD ref, which is GitRepository.default_branch as of yesterday.
<blr> ah cool, handy
<wgrant> I'm not completely sure how it should all work at this stage. I think we eventually want a two-stage picker that selects a repo and then a ref within it.
<blr> yep, would make sense to be within the same control
<wgrant> We can't have the target ref input prepopulated with the default_branch, because the user could change the selected repo and invalidate the branch.
<wgrant> So I think we for now have to go with a picker that asks for a vocab on the selected repo, but for the normal case accept that an empty ref input means a server-side fallback to default_branch.
<blr> would be easy enough to reset it if the content of the target repository input changes
<wgrant> Right, but reset to what? :)
<wgrant> It'd have to be empty.
<blr> well, it would have to be empty yes
<blr> unless we could make an api call
<wgrant> I think blank == HEAD makes sense in general.
<wgrant> Like, if you clone a repo, you get HEAD.
<wgrant> HEAD is a sensible thing for a null branch name to mean.
<blr> ok, perhaps we just need to add a note in that case that HEAD is assumed unless otherwise specified
<wgrant> I don't think it's worth improving on that until we can have the unified picker.
<wgrant> Yep
<mwhudson> wgrant, blr: just curious, have you thought at all about getting go get launchpad.net/foo to work if the foo project uses git?
<blr> mwhudson: I'm afraid I'm not that familiar with golang, but that certainly sounds like a useful feature given our increasing use of the language...
<mwhudson> yeah, there is support in the go tool for urls like that, but it assumes bzr
<mwhudson> would be nice to fix it somehow
<mwhudson> i'm not sure about all the intricacies of how go get works
<blr> welp, apparently there's a carshare service in melbourne called GoGet. Thanks google.
<mwhudson> i think one approach would be to put <meta> tags on the launchpad.net/project page
<blr> presumably the go source is on github somewhere
<mwhudson> (and delete the special casing for launchpad in the go tool)
<mwhudson> blr: docs here https://golang.org/cmd/go/#hdr-Remote_import_paths
<mwhudson> source is here https://github.com/golang/go/blob/master/src/cmd/go/discovery.go
<mwhudson> (and get.go, vcs.go)
<blr> mwhudson: thanks, will have a look this evening
<blr> have been meaning to spend some time with golang at any rate.
<mwhudson> blr: would it be useful to file a bug about this?
<blr> mwhudson: absolutely yes, please tag it 'git'
<mwhudson> if you do the lp bits, i'll do the go bits :-)
<wgrant> Yep, that definitely makes sense.
<wgrant> Inasmuch as go get itself does :)
<mwhudson> https://bugs.launchpad.net/launchpad/+bug/1465467
<mup> Bug #1465467: put <meta name="go-import"> tags on project, series pages <git> <Launchpad itself:New> <https://launchpad.net/bugs/1465467>
<mwhudson> well yeah
<mwhudson> i guess testing this against launchpad.dev will actually be easier than testing it against launchpad.net, unless you hack your own build tool
<mwhudson> er
<mwhudson> *go tool
<blr> thanks mwhudson, at a glance it seems easy enough
<mwhudson> blr: yeah, seems like it should be fairly easy
<mwhudson> blr: go 1.5 will be released in august, would be nice to be able to get the relevant change into that
<mwhudson> not completely sure i want to set up a launchpad dev environment just for this, but i guess i could...
<blr> mwhudson: could you mock it locally somehow and point me at your golang fork on gh perhaps?
<mwhudson> blr: sorry, i don't really understand
<blr> mwhudson: I'm not certain you're going to be able to avoid setting up LP heh, although it is reasonably painless these days
<mwhudson> blr: depends on whether i can con one of you guys into doing it first
<blr> mwhudson: I can have a poke at it at codecraft this evening
<mwhudson> blr: awesome
<mwhudson> i've put it on my todo list
<mwhudson> but not at the top :-)
<maozhou_> Can someone help me? how to init series?
<cjwatson> maozhou_: Um, what are you trying to do exactly?
<cjwatson> Project series don't require initialisation, and initialising a distribution series on launchpad.net is something we'd expect to have had prior notice of.
<maozhou_> i set up launchpad server on local, then i add a series.but when i init it ,it dosen't work
<cjwatson> maozhou_: This is not even slightly easy to get going, but https://wiki.ubuntu.com/NewReleaseCycleProcess may provide some clues.
<maozhou> cjwatson: i do it by this way.(https://dev.launchpad.net/LEP/DerivativeDistributions),but when i init it ,it dosen't work.
<cjwatson> You'll probably need to run 'cronscripts/process-job-source.py IInitializeDistroSeriesJobSource'
<cjwatson> But on a local instance you're mostly on your own, or at least will have to dig through log files yourself and generally provide more useful problem reports than "it doesn't work" :-)
<cjwatson> I suspect derived distributions will be pretty complex to get going on a local instance, as you'd have had to import some other distribution to derive from.
<cjwatson> You could in principle import Ubuntu using scripts/gina.py or something, but there'd be a lot of details for you to work out for yourself.
<maozhou_> cjwatson: my workmate said that he once init the "DerivativeDistributions"  succeed by run some script, but he can't remember which script he run. may be i need try to run 'cronscripts/process-job-source.py IInitializeDistroSeriesJobSource'
<maozhou_> cjwatson: thanks for your help, Let me try by your way.
<wgrant> maozhou_: Oh, are you working on the kylinos.com.cn Launchpad instance that was spamming people a few months back?
<cjwatson> ... and have you deleted sampledata yet? :-)
<maozhou_> wgrant: yes, may be. I am sorry about that. but how to stop it spamming people?
<wgrant> maozhou_: As far as we could tell, you took a normal Launchpad development instance, including all the development sample data, enabled outbound email, and started using it. The sample data includes some real email addresses, and is generally a terrible basis for a production deployment.
<wgrant> I don't know what your goal for your Launchpad instance is, but starting with sampledata is almost certainly the wrong way to go.
<cjwatson> Also may well have security holes of various kinds since the sample data includes people with e-mail addresses you don't control in privileged teams.
<cjwatson> The development sample data is meant to make it easier to experiment with changes to Launchpad itself locally, by giving you a bit of test data so that important pages will render something at least a little bit useful.  But it should never be included in production deployments.
<cjwatson> I don't know if there's a sensible way to excise it from an already-running instance; it would probably involve non-trivial database surgery, and might well be easier to start again from scratch.
<maozhou_> ok, i will tell my workmate to check it . may be we need to modification datebase.
<maozhou> if i alter datebase about mailaddress,is it effectivity?
<cjwatson> You really ought to delete the predefined non-team accounts, and there will be various other junk in e.g. your list of registered projects, your bug/branch/specification tables, etc. that you probably want to get rid of.
<wgrant> A fresh bootstrapped database would be a much better idea.
<maozhou_> there are many data already in datebase ,if do a fresh bootstrapped database, all the data will be losed , may be we can't do that.
<maozhou_> we will check it to stop it spamming people.
<maozhou_> cjwatson:when i run  'cronscripts/process-job-source.py IInitializeDistroSeriesJobSource' , it's error . the error log is âRunning <InitializeDistroSeriesJob for distribution: kylin, distroseries: dota, parent[overlay?/pockets/components]: utopic[False/Release/main], architectures: (), archindep_archtag: None, packagesets: None, rebuild: True> (ID 2) in status Waitingâ
<maozhou_> Job resulted in OOPS: OOPS-ea5bde41e13487cb6437263a6b26b44d
<maozhou_>  1 InitializeDistroSeriesJob jobs did not complete.
<maozhou_> what is âOOPS: OOPS-ea5bde41e13487cb6437263a6b26b44dâ ï¼
<cjwatson> You'll need to find the actual text of that OOPS.
<cjwatson> It should have been dumped out somewhere, but exactly where depends on your configuration.
<cjwatson> The error_reports lazr config section should have an error_dir key, which might be a place to start.
<maozhou_> okï¼i know .thank you.
<cjwatson> wgrant: Would you mind having another quick look over https://code.launchpad.net/~cjwatson/launchpad/git-repository-ui-edit-target/+merge/261232 ?  I know you already gave an Approve vote, but I made some fairly extensive changes to cope with the changes related to target_default.
<blr> morning
<blr> hmm would be nice to have the option to delete a comment within x minutes of creation for people that haven't had enough coffee.
<mwhudson> i think there is a bug with a very small number about that :-)
<blr> mwhudson: made a start on your request last night, have the import meta rendering on Product+index, will try to sort out the bzr meta and ProductSeries tag before eow
<mwhudson> blr: i just noticed, cool :-)
<mwhudson> i'll get set up to be able to test it
<mwhudson> is launchpad still running on precise?
<blr> mwhudson: yep
<mwhudson> okidoke
<blr> mwhudson: have another branch (which allows a project to have an associated 'default' vcs, amoungst other things) which needs to land before this branch. Not far away however
<blr> will govern whether a bzr or git meta tag is rendered
<mwhudson> blr: ah yes, that seems relevant :-)
#launchpad-dev 2015-06-17
<maozhou_> cjwatsonï¼ in the file of  "configs/development/launchpad-lazr.conf",  the path of  "error_dirâ is â/var/tmp/lperr",but I can't find the folder "/var/tmp/lperr" ,why?
<maozhou_> cjwatsonï¼and I canât find any file like âOOPS....â
<maozhou_> cjwatsonï¼ in the file of  "configs/development/launchpad-lazr.conf",  the path of  "error_dirâ is â/var/tmp/lperr",but I can't find the oops file ,why?
<wgrant> maozhou: If you're running a production instance with a development config, all hope is lost.
<wgrant> wut
<wgrant> pygit2 gives strs instead of bytes for a whole lot of things on Python 3.
<wgrant> eg. ref names
<wgrant> what does it do when there are invalid UTF-8 sequences, I wonder.
<Spads> or new points in the astral plane ð¨
<maozhou> where is oops file, is there anyone knows?
<wgrant> maozhou: We don't know how your installation is configured.
<wgrant> But it sounds like you're using the development config on production, which is terrifying.
<wgrant> Depending on how you've set it up, the OOPSes may have ended up in an ephemeral rabbitmq.
<maozhou> cjwatsonï¼ in the file of  "configs/development/launchpad-lazr.conf",  the path of  "error_dirâ is â/var/tmp/lperr", but I can't find opps file I need
<maozhou> wgrant:in the file of  "configs/development/launchpad-lazr.conf",  the path of  "error_dirâ is â/var/tmp/lperr", but I can't find opps file I need
<wgrant> maozhou: The development configuration is not intended for a production environment.
<wgrant> If you're using the default development config, OOPSes may end up in rabbitmq.
<wgrant> If you are using rabbitmq for nothing else, consider stopping it. Then further OOPSes will fall back to the filesystem.
<maozhou> wgrant: I am using the development config  for test.
<maozhou> wgrant:  but ,how to stopping rabbitmq?
<wgrant> maozhou: It depends on how you have set up the system. Examine your configuration file to identify where it expects to find rabbitmq, find that rabbitmq, and stop it.
<maozhou>  wgrant:  If you're using the default development config, OOPSes may end up in rabbitmq ? where are the  rabbitmq? I just wan't to know the error information when I run "cronscripts/process-job-source.py IInitializeDistroSeriesJobSource'â
<wgrant> maozhou: OOPSes will be sent to rabbitmq if it is configured and running. It can be challenging to get them out of rabbitmq, so it's easiest to find the rabbitmq that they're going to, stop that rabbitmq, then trigger the OOPS again. Then it will end up on disk.
<maozhou> wgrant: I  understand,but I don't konw where is the configuration file to control rabbitmq.
<wgrant> maozhou: The same configuration file that you mentioned earlier.
<maozhou> wgrant: ok, I find it, let me try
<maozhou> wgrant: I have stoped rabbitmq in file "configs/development/launchpad-lazr.conf",but when I trigger the OOPS again , I still can't find the OOPS file .
<wgrant> maozhou: Have you restarted the application?
<maozhou> wgrant: Yes I run "make run_all" again on my launchpad server
<wgrant> maozhou: Have you confirmed that rabbitmq is not running?
<maozhou> wgrant: how to confirmed that rabbitmq is not running? I have changed the value of  "launch:" to "False" which in the "configs/development/launchpad-lazr.conf"  of section "[rabbitmq]"
<wgrant> maozhou: Are there any processes running with "beam.smp" in the commandline?
<wgrant> If so, which user owns them?
<maozhou> wgrant: what is "beam.smp" ?
<wgrant> maozhou: beam.smp is the erlang process that hosts (among other things) rabbitmq.
<maozhou> wgrant: is it the console which run "make run_all", if there are rabbitmq output on this console, it means the rabiitmq have not been stopped?
<wgrant> maozhou: If make run_all starts rabbitmq, it would be reasonable to assume that rabbitmq has started.
<maozhou> wgrant: so ,if the rabbitmq has been stopped ,the console of "make run_all" won't output messages?
<wgrant> maozhou: run_all will always output some messages, but it shouldn't look like it's starting rabbitmq if it's not starting rabbitmq.
<maozhou> wgrant: processes running with "beam.smp" in the commandline can judge the rabbitmq, but ,which commandline ?
<wgrant> maozhou: In the output of ps
<maozhou> wgrant: if there is processes running with "beam.smp" ï¼ itâs mean  the rabbitmq have not been stooppedï¼
<wgrant> maozhou: Yes.
<maozhou> wgrant: I reboot my launchpad serverï¼before running âmake run_allâ, there is processes running with "beam.smp". need i run "invoke-rc.d rabbitmq-server stop" to stop  rabbitmq server?
<wgrant> maozhou: That is worth a try.
<maozhou> wgrant: I have ran "invoke-rc.d rabbitmq-server stop" to stop  rabbitmq server,and I have modified the value of  "launch:" to "False" which in the "configs/development/launchpad-lazr.conf"  of section "[rabbitmq]". But after run "make run_all", there is processes running with "beam.smp" . Nedd I kill it?
<wgrant> maozhou: Who owns that process?
<wgrant> It may also be the case that run_all disregards the "launch" option. I forget.
<wgrant>     if len(requested_services) == 0:
<wgrant>         return [svc for svc in SERVICES.values() if svc.should_launch]
<cjwatson> run_all hardcodes a bunch of service names, I think.
<wgrant> make run_all explicitly names all of the things it wants, so the launch option is not respected.
<cjwatson> It should be pretty clear from the Makefile.
<cjwatson> Yeah.
<maozhou> wgrant: the user who run "make run_all" owns that process
<maozhou> I killed the process running with "beam.smp" , then I  trigger the OOPS again.  I find the OPPS  file.
<maozhou> thanks for your help :)
<blr> morning
<wgrant> Morning blr.
<wgrant> blr: It turns out that turnip on Python 3 is more difficult than we anticipated, because pygit2 on Python 3 is quite brain-damaged.
<wgrant> >>> r.listall_references()
<wgrant> Traceback (most recent call last):
<wgrant>   File "<stdin>", line 1, in <module>
<wgrant> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 14: invalid start byte
<blr> oh lovely >.<
<wgrant> That's what happens if a repo has a ref that isn't in the filesystem's encoding.
<wgrant> The low-level library tries to force Unicode onto something that isn't Unicode.
<wgrant> https://github.com/libgit2/pygit2/issues/537, let's see what they say.
#launchpad-dev 2015-06-18
<blr> wgrant: can I render a value from services.config without setting the property on the view?
<wgrant> blr: modules/lp.services.config/config in TALES will get you the current configuration.
<blr> wgrant: ah excellent thanks
<maozhou_> wgrant: do you kown how to delete a series on my test launchpad server?
<maozhou_>  Is there anynone knows how to delete a series?
<maozhou> oh,the network is bad
<maozhou> is there anyone knows how to delete the series?
<wgrant> maozhou: It's not possible to delete a distroseries without complex DB surgery.
<maozhou> ok
<maozhou> and how to delete a builder ? It's not possible to delete a builder  without modification of DB?
<wgrant> no, builders can't be deleted, as they're referenced by builds that they have performed. when we decommission a builder, we unset "builder ok" and "publicly visible" to dispose of it.
<maozhou> but sometimes the builder is not available, if we can't delete it, the list of builders will be more and more.
<wgrant> maozhou: unchecking the "publicly visible" flag hides it from the list
<maozhou> wgrant: that's a good idea
<blr> wgrant: sorry, could you clarify what you mean by the view should have a _label_ of "configure code"
<blr> did you mean a heading in the template? where your inline comment was placed it seemed like you were asking for an update to the docstring in the view class
<blr> have added an h2 with context/name at any rate.
<wgrant> blr: Ah, sorry, views have two special properties.
<wgrant> page_title, and label.
<blr> wgrant: added a .note class, does this look okay? http://i.imgur.com/J1N7L4g.png
<blr> wgrant: ah I see, no need to have it in the template.
<wgrant> blr: label appears as the h1 on the page, page_title in the breadrumbs in the head.
<wgrant> Yep
<wgrant> blr: That looks fine, yep.;
<blr> wgrant: any thoughts on an appropriate name for the vcs inferring function if I push it into the model?
<blr> inferred_vcs or default_vcs?
<wgrant> blr: inferred_vcs makes sense to me.
<blr> thanks
<wgrant> blr: It won't be around for terribly long, I suspect, as we can fill in all existing rows once the UI to change it is in palce.
<maozhou> I create a series which derives from "utopic", but I can't find arch of 386, why? like this : http://imgur.com/8wEriuZ
<wgrant> maozhou: That architecture apparently doesn't exist in the previous series of that distribution.
<wgrant> Only the first series in a distribution can copy from another distribution; subsequent series always copy from the previous series in the same distro.
<maozhou> wgrant:It's the first series in a distribution, I have created a new distribution.
<wgrant> maozhou: Are you sure that your version of utopic has i386?
<maozhou> wgrant: Yes , I'm sure. like this: http://imgur.com/ytOFPQC
<wgrant> maozhou: Is there an i386 checkbox on the new distribution's primary archive's admin page?
<maozhou>  wgrant: what's the URL of the new distribution's primary archive's admin page??
<wgrant> maozhou: Like https://launchpad.dev/ubuntu/+archive/primary/+admin, except not ubuntu
<blr> wgrant: should productseries-setbranch.pt be in code/templates?
<wgrant> blr: I'd say so. It's on a Registry object, but it's a totally Code-specific view.
<wgrant> Registry is quite big enough :)
<blr> wgrant: ok, I'll take this opportunity to move it - I just cargoculted product-setbranch from that, so figured that was where it should live!
<wgrant> blr: By no means an unreasonable assumption.
<maozhou> wgrant: I can't find the i386 checkbox .http://imgur.com/NVI7ePR
<wgrant> maozhou: It would be immediately below the bottom of that image.
<wgrant> A checkbox for each architecture, if you're running modern code.
<wgrant> Or potentially none, if you're running old code.
<maozhou> wgrant:http://i.imgur.com/OXU5S1s.png . I still can't it ,may be I'm running old code.
<wgrant> maozhou: Right, in that case it's not relevant.
<wgrant> maozhou: I think I may know what the problem is, though.
<wgrant> If you're running a largely unmodified development config, your batch navigator settings will be configured to use very small batches.
<wgrant> +initseries assumes that all of the architectures will be returned in a single batch.
<wgrant> (because all production-like systems have a default batch size of 75, far higher than the number of architectures)
<wgrant> Try increasing config.launchpad.default_batch_size to 75 or so, restarting the application, and refreshing the page.
<maozhou> wgrant: what's the "batch size" mean?
<wgrant> maozhou: When a request is made for a potentially large collections of objects, only a fixed number are returned at a time. eg. https://bugs.launchpad.net/ubuntu/ only shows the first 75 bugs.
<wgrant> maozhou: In the development config, the number is 5, not 75.
<maozhou> I understand
<maozhou> wgrant: how to increasing config.launchpad.default_batch_size? need I modify DB or code?
<wgrant> maozhou: launchpad-lazr.conmf
<wgrant> -m
<maozhou> ok
<maozhou> wgrant: I have increased config.launchpad.default_batch_size to 75 ,after running "make run_all" , I still can't find arch of i386.
<wgrant> maozhou: Perhaps that was not the problem after all. Does https://launchpad.dev/api/devel/ubuntu/utopic/architectures list i386?
<maozhou> Butï¼I can find arch of i386 if the series is  derives from other seriesï¼ eg. 14.04 .....
<wgrant> SELECT architecturetag, enabled FROM distroarchseries WHERE distroseries = (SELECT id FROM distroseries WHERE name = 'utopic' AND distribution = 1);
<maozhou> wgrant: https://launchpad.dev/api/devel/ubuntu/utopic/architectures  doesn't list i386.
<wgrant> I suspect, then, that someone has set the disabled flag on it.
<wgrant> The SQL statement will confirm.
<maozhou> ok
<maozhou> wgrant: You are right , arch of i386 has been disabled by someone, I recovered it , then I find the arch of i386.  Thank you.
<wgrant> Great.
<wgrant> Disabling an architecture is meant to be permanent, so you may run into trouble. But it may be fine depending on when they did it.
<maozhou> yes
<maozhou> Only the first series in a distribution can copy from another distribution, Is it reasonable?
<maozhou> If the first series is incomplete, we can't delete it, and the subsequent series can't copy from another distribution, that's in trouble
<wgrant> Correct, you need to be careful and work out what you're going to do.
<wgrant> Launchpad can only do so much automatically, due to file conflicts.
<wgrant> Planning how a distro will operate before you start using it is absolutely essential.
<wgrant> In some cases you may want to initialise it without any packages and copy the relevant packages in yourself.
<maozhou> I understand.
<maozhou> wgrant: If I have created new distribution and series. how to upload the chroot-xxx-.tar.bz2 to the series? I can only upload it to the series which belong to ubuntu.
<wgrant> maozhou: manage-chroot defaults to Ubuntu, but you can tell it to use any distro.
<maozhou> ok,thans
<maozhou> thanks
<maozhou> wgrant: I have created new distribution and series, and the series is initialized successful. But  I can't find the series which belong to the new distribution. And upload thd chroot-xxx.tar.bz2 is error. http://imgur.com/zc8ijjW   http://imgur.com/mAbPLux
<wgrant> maozhou: You'll find the series if you click "All series". The distro page doesn't show series with status=Experimental by default; you probably want to set it to Development.
<wgrant> The manage-chroot error is a permission problem.
<wgrant> Specifically, you need to be either an admin or a member of the primary archive's owner team to manage its chroots.
<maozhou> wgrantï¼but I can upload the chroot file to "ubuntu:utopic" successful  in the same way
<wgrant> maozhou: The user you're logged in as is probably a member of the team that owns your Ubuntu's primary archive.
<maozhou> wgrant: how to add current user to the team that owns my distributionâs archiveï¼
<wgrant> maozhou: The primary archive will be owned by the distribution's original owner.
<wgrant> That may, in this case, be the ~name16 user.
<maozhou> wgrantï¼ but the current user is ~name16 ï¼admin@canonical.comï¼
<wgrant> maozhou: What does https://launchpad.dev/api/devel/kylin2/+archive/primary say about the owner_link?
<maozhou> wgrant : I'm sorry, there are something wrong with my notebook, it's panic just now , can you repeat it?
<wgrant> 17:56:31   wgrant | maozhou: What does https://launchpad.dev/api/devel/kylin2/+archive/primary say about the owner_link?
<maozhou> #wgrant:  http://imgur.com/p8y9tsa
<wgrant> maozhou: Mysterious. Are you sure your API client is authenticated as ~name16?
<maozhou> wgrant: the user who login launchpad.dev is ~name16, http://imgur.com/CPel393
<wgrant> maozhou: But manage-chroot uses an OAuth token, not your browser cookie directly. It depends who you were logged in as when you first ran an API client against launchpad.dev.
<wgrant> You can run seahorse and delete the launchpad.dev entry to force it to reauthenticate.
<maozhou> but how to authenticate my API client as ~name16?
<wgrant> If there's no token in gnome-keyring (which seahorse manages), running manage-chroot will prompt you to use your browser to authorise the client again.
<wgrant> It will authenticate as ~name16 if you authorise it from a browser logged in as ~name16.
<maozhou> what's seahorse?
<wgrant> maozhou: seahorse is the graphical GNOME Keyring manager.
<maozhou> i understand
<maozhou> wgrant: which passwd should I delete? It is "Password for 'System-wide: Ubuntu (kylin)@https://api.launchpad.dev/' on 'launchpadlib'" ?
<wgrant> maozhou: That's the one.
<maozhou> ok
<cjwatson> maozhou: You said name16's e-mail address was admin@canonical.com.  Please change that (or, better, create a new user to add to the same teams and deactivate name16); that's the address you spammed a couple of months ago, and you said yesterday you were going to change the sampledata addresses.
<cjwatson> maozhou: If you change the existing user's address, please do it manually with SQL so that LP doesn't send an address change notification to the old address.
<maozhou> cjwatson: I'm sorry about that, I will modify DB about the mail addresses now.
<maozhou> cjwatson: Which email addresses  spammed you?
<maozhou> cjwatson: I'm surprised about it , I haven't configured the email. The vaule of Mailer is stub in "devel/zcml/package-includes/mail-configure-normal.zcml"
<cjwatson> maozhou: I don't know what was the source address, because I'm not behind admin@canonical.com; it was reported by one of our sysadmins.
<cjwatson> blr: Fixed your code-in-swift nit; can I take your comment as an Approve vote?
<cjwatson> (and thanks)
<cjwatson> wgrant,blr: I've tested self-imports from LP git to LP bzr, and they work fine now, both https:// and (should anyone so desire) git://.
<cjwatson> So definitely no firewall problems there.
 * cjwatson creates https://code.launchpad.net/~cjwatson/launchpad/git-import-test to see what will happen
<cjwatson> wgrant,blr: Oh dear, this plan may work for turnip but apparently not for Launchpad itself.  http://launchpadlibrarian.net/209437153/cjwatson-launchpad-git-import-test.log
<cjwatson> Maybe code-import-worker could bump its recursion depth, but I have no idea what it would need to bump it to
<cjwatson> blr: code-in-swift is just the first pass, BTW; I've been looking through some other deployments and the wsgi-app charm, and working out how I want this to look.  wsgi-app (although we can't use it directly because we have significant non-WSGI components) does something not a million miles away from what deployment-manager does, and I think we want the same kind of thing: [current_symlink, previous_symlink, build_label] config ...
<cjwatson> ... entries, mojo spec looks at build_label and pushes a tarball for that rev into swift if it needs to, charm ensures that it has unpacked copies from swift of all the configured things and GCs old unused ones
<cjwatson> To upgrade we juju set previous_symlink to current_symlink, build_label and current_symlink to the new rev, and mojo run --manifest manifest-upgrade
<cjwatson> To rollback we mojo run --manifest manifest-rollback, which does juju set current_symlink=$previous_symlink
<cjwatson> (and the rest)
<cjwatson> And upgrades shouldn't have the scary thing of overwriting the running code in place any more
<cjwatson> Anyway, I'm EOD and on leave tomorrow, so will pick this up on my back burner next week if I get time
<blr> cjwatson: oh yes, of course :)
<blr> cjwatson: that sounds like an excellent improvement
<blr> hmm no comments on your issue yet wgrant, clearly you've put the fear in them.
<blr> wgrant: hmm, does it make sense for ProductSeries to also have inferred_vcs? the context is of course now different in ProductSetBranchView
<blr> I suspect I should just check for the existence of self.context.inferred_vcs in initial_values instead
<blr> ah ForbiddenAttribute :|
<blr> wgrant: how can I check for a property on a proxied object?
<wgrant> cjwatson: The LP repo is packed with depth=200 IIRC. I'm tempted to fix dulwich to not recurse, though.
<wgrant> blr: You need to add it to the interface (or configure.zcml manually, if that class names attributes directly rather than using its interfaces for security)
<blr> wgrant: I think I want the latter, I'm not sure it makes sense for ProductSeries to have the property
<wgrant> blr: Oh, I missed the first part of your question, sorry.
<wgrant> blr: ProductSeries.vcs doesn't make sense, so you always want to find Product.inferred_vcs.
<wgrant> blr: Consider always using self.context.pillar.inferred_vcs.
<wgrant> .pillar is on a lot of Registry objects as a way to get to the Product or Distribution that contains the object.
<wgrant> It even works on Product and Distribution themselves.
<blr> ah wasn't aware of that, useful thanks
<blr> wgrant: is it acceptable to call a private method from the view?
<blr> (in this case Product._default_git_repository)
<blr> well, a cachedprop.
<wgrant> blr: Not usually. Why?
<wgrant> If you want the default repo, I would define a property on the view that uses IGitRepositorySet
<blr> wgrant: for the configure code link, which currently only checks for development_focus.branch
<wgrant> blr: Ah, you misunderstood my recommendation.
<blr> quite possibly!
<wgrant> blr: The need for the check goes away if you fix it so the link isn't randomly different from the others.
<wgrant> None of the others change based on whether a branch is set or not.
<wgrant> s/a branch/the config/
<wgrant> If we keep the check, it needs to be fixed to consider the Git case too.
<wgrant> But I don't think we need to keep the check.
<blr> wgrant: ah, so you're happy for it to just read 'configure code' in either case
<blr> agreed, I don't think we're losing much by having a simple link
<wgrant> We don't lose much, and we gain simplicity, performance and consistency.
<blr> 'Configure code' and 'Configure code for this project' for text and summary work?
<wgrant> Link summaries are silly, as they only appear as tooltips, but that works for consistency.
<wgrant> cjwatson: I think your sampledata was out of date. Will testfix if you're not around.
<wgrant> Have fixed the dulwich issue, trying an import locally now.
<wgrant> Looks happy.
#launchpad-dev 2015-06-19
<wgrant> Yay GitHub
<wgrant> Trying to create a PR for the dulwich changes.
<wgrant> The "Compare" link works, but there's no way to create a PR.
<wgrant> The "Pull request" link 404s.
<lifeless> jelmer: ^
<wgrant> Oh
<wgrant> 404 on the pull request link means "you're not logged in", apparently.
<wgrant> My session must have expired.
<wgrant> Why would you show the link and then 404 :(
<maozhou> cjwatson, wgrant : Is it spamming you yet?
<wgrant> maozhou: I haven't had reports of it in some weeks.
<maozhou> ok
<blr> hmm that was weird, must have been logged out for some reason, but ended up with base template json errors
<blr> wasn't entirely obvious what was going on...
<wgrant> blr: Hm, that's not a good sign.
<wgrant> What was the error?
<blr> wgrant: https://pastebin.canonical.com/133538/
<wgrant> jelmer: Thanks.
<wgrant> blr: Hm, can you reproduce it now?
<blr> wgrant: nope
<blr> wgrant: just about have all these changes complete, but would like to merge the templates - can a tal:condition be a route/path, or refer to the view class? Ideally would like to avoid having a property set on the project setbranch view just for this purpose
<wgrant> blr: tal:condition can be any TALES expression. But I don't think it'd be bad to have a is_series or similar property on the view to determine the behaviour, if you had to.
<blr> wgrant: ok, if that's acceptable that would certainly be easy enough
<blr> wgrant: the zcml docs suggest the 'for' directive can take multiple classes, but I don't see any instances where we're doing that (other than *), is there are a reason for that?
<wgrant> blr: Hm, that may work, not sure. If it makes things nicer, I have no problem with it.
<wgrant> It would be handy if that did work.
<blr> wgrant: hmm no it crashes actually heh
<blr> may have been misinterpreting Multiadapters http://docs.zope.org/zope.component/zcml.html
<blr> effectively I want to share a macro between IProduct and IProductSeries
<wgrant> Yeah, unless you have a common interface you need to register it twice.
<wgrant> What's the macro?
<blr> push instructions and the ssh key reminder. Both could be inlined in the template, but they seem like they could potentially be used elsewhere
<wgrant> ah, yes.
<blr> right, think that's in reasonable shape now.
<blr> might merge this branch into the golang import meta branch and try to get that working tomorrow for mwhudson
<blr> ah, I didn't add the repository_location vocabulary... drat. Will look at that as well.
<blr> at any rate, ttyl wgrant
<wgrant> blr: Thanks, have a good weekend.
#launchpad-dev 2015-06-20
<wgrant> cjwatson: https://code.launchpad.net/~cjwatson/launchpad/git-import-test is running happily now.
 * wgrant tries to get https://code.launchpad.net/~cjwatson/launchpad/git-import-test to scan.
#launchpad-dev 2015-06-21
<blr_> morning
<mwhudson> blr: morning
<mwhudson> blr: would it be useful to test project-meta-go-import?
<blr> mwhudson: yep certainly -  I need to also update the ProductSeries template, but it is there now for Product
<blr> for the meta tag to render, it requires that the project has a default vcs set via Configure Code (product/+configure-code)
<blr> and a bzr/git branch/repo of course :)
<blr> mwhudson: it might be easier however if you wait until this is on QA staging
<blr> would save you setting up LP locally
#launchpad-dev 2016-06-22
<wgrant> cjwatson: I wonder if it's worth eliminating the bugsummary temporary journal.
<wgrant> (currently the bugsummary maintenance triggers write everything to a temporary journal table, then flush the aggregate of that temporary table to the main journal table, which is then rolled up into the non-journal table by garbo-frequently)
<wgrant> The implementation means that will be rather noisy (usually changes are implemented by subtracting the bug from all summaries to which it is relevant, making the change, then adding the bug back... so lots of writes)
#launchpad-dev 2017-06-20
<mz_> What's LOSAs?
<cjwatson> Launchpad Operational System Administrators - an old name for one of the sysadmin teams within Canonical.  Why?
<mz_> "https://dev.launchpad.net/Soyuz/TechnicalDetails/Building" , mentioned it, I understand, thank you.
<cjwatson> I've modernised that page slightly.
<mz_> ok :)
#launchpad-dev 2017-06-21
<mz__> What's librarian mean in "https://dev.launchpad.net/Soyuz/TechnicalDetails/Publishing" ?
<wgrant> mz__: librarian is Launchpad's blob store.
<mz__> What's blob? Binary Large Object?
<wgrant> mz__: Yes.
<mz__> Is it the field types in database?
<wgrant> mz__: The librarian is designed to store dozens of terabytes of data, so it stores it either as files on local disk or in OpenStack Swift. It does not store blob data in the database.
<mz__> Ok, I understand, thanks.
#launchpad-dev 2017-06-23
<xnox> Timeout error, please try again in a few minutes.
<xnox> OK
<xnox> is there a status page for launchpad availability?
<xnox> next downtime appears to be scheduled for the 26th
<cjwatson> There's no scheduled downtime at the moment.
<cjwatson> Nor unscheduled downtime.  A timeout is usually just a slow page.
<cjwatson> And if you're reporting one, you must quote the OOPS ID or the report is fairly useless.
#launchpad-dev 2018-06-24
<jelmer> hi cjwatson, wgrant
<jelmer> I've published a copy of bzr-builder that is standalone and uses breezy @ lp:brz-builder
<cjwatson> jelmer: Would you mind filing a bug against launchpad-buildd to use it?
<cjwatson> I think that's the only place we'd need to change.  Though presumably it really ought to wait for a stable breezy release
#launchpad-dev 2020-06-15
<SpecialK|Canon> https://code.launchpad.net/~doismellburning/launchpad/+git/launchpad/+merge/385644 please
<tomwardill> +1
<SpecialK|Canon> Thanks :)
<SpecialK|Canon> And uh https://code.launchpad.net/~doismellburning/launchpad/+git/launchpad/+merge/385741 to fix the test I broke please...!
<SpecialK|Canon> http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1334/steps/shell_9/logs/summary also shows TestRunMissingJobs.test_run_missing_ready_does_not_return_results failure but I can't reproduce it locally, can't see how my change would have impacted it, and have a suspicion that it's a transient timing issue?
<tomwardill> yeah
<tomwardill> that's a celery / rabbit failure manifestation
<SpecialK|Canon> tomwardill: an actual proper persistent failure, it seems? http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1335/steps/shell_9/logs/summary
<ilasc> SpecialK|Canon: nope, that's transient
<tomwardill> you just have the fun of it happening multiple times :)
<ilasc> that's one of "the usual"
<ilasc> right, as tomwardill said :)
<SpecialK|Canon> Ah ok I'm 2/2 so far - I recognise that's not a lot of data points but also figured safest to assume that I'd just broken stuff ;)
<SpecialK|Canon> Cheers!
<SpecialK|Canon> Can I kick off a restart somehow or do I just wait for someone else to merge something?
<tomwardill> http://lpbuildbot.canonical.com/force
<SpecialK|Canon> ...it's even right there on the front page isn't it - thanks
<tomwardill> it's non-ovious terminology
<tomwardill> also non-obvious place, you'd expect a 'retry' on the build page or something
<tomwardill> (fwiw, our cancel also doesn't work)
<pappacena> Did it fail again? I've retried right before the standup...
<SpecialK|Canon> pappacena: ah then I might have double-retried, sorry
 * pappacena not a problem :)
<tomwardill> iirc, if it's in a failed state, a merge won't cause a new build
<tomwardill> you have to force it once the merge shows up on the left column
<SpecialK|Canon> it definitely tried a new build of my merged test fixen
<tomwardill> https://usercontent.irccloud-cdn.com/file/9jd2SNIX/image.png
<tomwardill> not at that time difference it didn't, that was pappacena retrying it for you
 * SpecialK|Canon retracts the "definitely"
<SpecialK|Canon> I assumed that there would be visible indication in the waterfall of someone doing that
<tomwardill> nope
<tomwardill> force is a bit of a weird hack in this version of buildbot, it's not natively implemented
<tomwardill> but there was a standard 'recipe' for people to extend the web ui to create it
<SpecialK|Canon> no pressure re your buildbot work, but uh.... ;)
<tomwardill> hah
<tomwardill> SpecialK|Canon: you can see it on http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1335 fwiw
<tomwardill> under 'Reason'
<tomwardill> but the waterfall doesn't show it
<pappacena> ilasc, tomwardill these are the MPs for projects as pillar for OCI projects:
<pappacena> - https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/384506 (create & edit page)
<pappacena> - https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/384514 (search & list)
<SpecialK|Canon> http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1336/steps/shell_9/logs/summary - TestJobRunner.test_runAll_mails_oopses also a known flaky?
<pappacena> tomwardill: on https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/384514, you've mentioned a typo, but I cannot see your comment on the code :-(
<tomwardill> pappacena: sigh, bet I forgotr to save it
<tomwardill> I did
<tomwardill> pappacena: L590 in the diff, s/contains/contain
<tomwardill> (findByPillarAndName)
<pappacena> Ah! Thanks!
<tomwardill> pappacena: fairly sure it was my typo to start with :)
<pappacena> hahaha. Maybe... I will fix it there anyway :)
<tomwardill> ah poop, actual test failure
 * tomwardill gets to fixing
 * pappacena :-( 
<tomwardill> tricky as we don't actually have the git repository that has been created to check against it's IP
<tomwardill> mgiht just have to swap it for an assert 'id' in repo.keys()
<pappacena> Maybe using a matcher for the data type could do?
<pappacena> Like `'id': IsInstance(int)`
<tomwardill> yeah, that's probably better
 * tomwardill does that
<tomwardill> fix my whoopsie: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/385759
<tomwardill> curious as to how it got to be 'not 1' though, that does imply a test isolation failure somewhere
 * pappacena +1
<pappacena> I guess the tests do not delete the database (or it depends on the layer, or whatever). And increasing a sequence has effect regardless on rolling back a transaction...
<tomwardill> ta, landing
<tomwardill> and then poking buildbot
#launchpad-dev 2020-06-16
<wgrant> Yeah, if a test doesn't commit then the DB is reset by aborting the transaction. I had a branch at one point to randomise sequences to track down tests that cared, but there were too many failures to reasonably fix.
<tomwardill> wgrant: any idea how I can QA a garbo job?
<wgrant> tomwardill: I believe staging and qastaging have it cronned.
<tomwardill> wgrant: so check the logs?
<wgrant> tomwardill: Well, ideally also ensure a scenario exists that the job will act on.
<wgrant> But in some cases yes.
<tomwardill> hmm, neither of them have usable OCIProjects on them.
<tomwardill> boo
<cjwatson> tomwardill: I think dogfood has it cronned too
<cjwatson> check ~launchpad's crontab
<tomwardill> SpecialK|Canon: you should OpenGraph the MP page too
<SpecialK|Canon> tomwardill: I assume/hope it does already - do you happen to have a handy MP URL for staging/qastaging/dogfood handy?
#launchpad-dev 2020-06-17
<tomwardill> round up for a fun review! Using testunit output in buildbot master: https://code.launchpad.net/~twom/lpbuildbot/subunit/+merge/385929
<pappacena> Talking about buildbot and tests, I broke a couple of tests just now...
<pappacena> I'll self-approve a MP to fix that... quite trivial change: https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/385928 in case anyone would like to take a look...
<tomwardill> +1
<pappacena> tomwardill: I think this is the last MP left to finish the project-project-oci-project-project feature: https://code.launchpad.net/~pappacena/launchpad/+git/launchpad/+merge/384755
<pappacena> You have provided a comment, and it should be fixed now. Whenever you have time, can you check that again?
<tomwardill> pappacena: +1
 * pappacena thanks!
#launchpad-dev 2020-06-18
<tomwardill> going to attempt the loggerhead upgrade trello card
<tomwardill> never tried to get loggerhead running locally, so here goes...
<tomwardill> garbo OCI job appears to have DTRT on labbu
<SpecialK|Canon> Nice
<SpecialK|Canon> obligatory "wait my builds are gone, are you calling them garbage" etc. etc.
<tomwardill> ... yes, yes I a
<tomwardill> *am
<tomwardill> oh hey, a list of OCI Registry Credentials just appeared on my test instance!
<tomwardill> how do I bzr push to a local instance....
<tomwardill> found it, but bzr still expects /dev/ to be on launchpad.dev
<tomwardill> hacked it
<tomwardill> upgrade loggerhead to r501: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/386001
<ilasc> ð
<ilasc> +1
<tomwardill> ta, landing. Will give it a proper poke on staging as I've only got limited test data to hand in my dev instance
<ilasc> yep, makes sense
<ilasc> tomwardill: in relation to the list of OCI Registry Credentials: please start filing bugs with things you don't like, or think you  would like to see improved and subscribe me
<ilasc> not long and we should have the Edit OCI Registry Credentials functionality too ð
<tomwardill> ilasc: left some comments on your credentials edit MP
<ilasc> thanks tomwardill !
<tomwardill> happy to be wrong about the removeSecurityProxy things, but it just looked a little odd
<ilasc>  I'll have a loo, thanks Tom :-)
<tomwardill> "Total: 0 tests, 0 failures, 7 errors, 0 skipped in 7.570 seconds."
<tomwardill> ... going well then
<tomwardill> fairly sure there should be more tests than that
<tomwardill> "Total: 181 tests, 0 failures, 137 errors, 0 skipped in 33.517 seconds."
<tomwardill> unsure if progress
<tomwardill> " Path 'Question.<primary key>' matches no known property."
<tomwardill> tum-ti-tum
<pappacena> well, at least the error message changed... hehe
<tomwardill> okay, not the model I started with, but Converting questionsubscription to Storm: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/386031
 * pappacena reviewing
<tomwardill> thanks pappacena, I'll land that in the morning when I'm around to poke buildbot if required
<tomwardill> one step closer to Storm Questions.
 * pappacena ð
 * tomwardill -> EOD
<tomwardill> going to file the buildbot RT in the morning, because why not on a Friday?
#launchpad-dev 2020-06-19
<tomwardill> okay, I broke buildbot, working on it
 * tomwardill shakes fist at doctests
<ilasc> ð
<tomwardill> https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/386081
<tomwardill> ilasc: you still have the 'findByRecipe' method in your oci-registry-credentials MP. Is that used anywhere?
<tomwardill> ta for the review :)
<ilasc> tomwardill: well spotted! it will be used only in the next MPs for Push Rule viewing & editing - this is the result of splitting the big MP into smaller ones, I'll remove it now
<ilasc> done
<tomwardill> ilasc: +1
<ilasc> thanks tomwardill , I'll obviously wait until later today or at some point next week to land, give buildbot a chance to recover ð
<tomwardill> well, at least it's gone back to 'normal' errors now
<ilasc> lol
<tomwardill> storm.exceptions.PropertyPathError: Path 'QuestionMessage.<primary key>' matches no known property.
 * tomwardill has no idea what that means (and yes, it's similar to the one I got yesterday,b ut that one went away)
<ilasc> hmmm
<ilasc> tomwardill: this buildbot or local dev env ?
<tomwardill> local instance
<ilasc> ok
<tomwardill> trying to convert QuestionMessage to Storm
<tomwardill> there is a primary key on that model
<ilasc> yes, ok, used to get similar with the stom -> sqlalchemy but on the FK
<ilasc> this one sounds like it doesn't know what 'primary key' is for QuestionMessage
<tomwardill> `id = Int(primary=True)`
<ilasc> yep, that should be sufficient
<ilasc> is QuestionMessage declared StormBased or SQLBased ?
<tomwardill> aha, progress
<tomwardill> I've changed QuestionMessage to be StormBase, but Question still had a SQLBase style ForeignKey declaration
<tomwardill> changing that to a Reference has solved that particular problem
<ilasc> ð good!
 * tomwardill gets lunch before any more
<tomwardill> something something, DOC TESTS ARRGGHHH
<pappacena> There is no love in something that breaks even the editor's syntax highlighting...
<tomwardill> no
<tomwardill> this set is somewhat annoyiong
<SpecialK|Canon> is this the file that crashes your ide?
<tomwardill> no, that's factory.py
<tomwardill> although it's a lot better now
<tomwardill> this doctest is failing, but i cna't tell why, because all it will tell me is that the transaction has been aborted
<tomwardill> repeatededly
<pappacena> Can you send stacktrace here?
<tomwardill> oh, found it
<tomwardill> First exception had the relevenat stack trace
<tomwardill> but I had to turn on 'infinite scrollback' in my terminal to be able to see it
 * pappacena laughing, but worried ð
<tomwardill> anyone tried using lazr Snapshot with Storm ReferenceSet?
<tomwardill> looks like it's failing to snapshot it, my referenceset for the 'old' is the same as the 'new'
<tomwardill> hmm, doesn't appear to work
 * tomwardill replaces it with a list() property instead
 * tomwardill undoes all the replaceing of [-1] with .last()
<tomwardill> "Total: 226 tests, 0 failures, 0 errors, 0 skipped in 4 minutes 20.402 seconds."
<tomwardill> I think that'll do for the day
 * tomwardill -> EOD
