#launchpad-dev 2009-11-02
<mwhudson>   Tear down canonical.testing.layers.TwistedAppServerLayer in 3 minutes 0.938 seconds.
<mwhudson> this seems excessive :(
<lifeless> probably swapped out
<mwhudson> yeah
<thumper> mwhudson: got a few minutes to talk about test strategy for a change I ahve
<thumper> ?
<mwhudson> thumper: sure
<wgrant> As what DB user do the tests run by default?
<mwhudson> wgrant: launchpad
<wgrant> mwhudson: Ah. Thanks.
<wgrant> Hm.
 * thumper gains even more dislike of the librarian for tests
<wgrant> Because it takes FOREVER to start up, or something more sinister?
<lifeless> singleton.slow. functional-not-unit.
<mwhudson> or maybe because you have to commit all over the place when using it?
<thumper> failed to shutdown in this instance
<thumper> also for those other reasons, but in this particular time, for failing to die
<mwhudson> that's a lot of reasons for hate between us
<wgrant> There are more.
 * thumper EODs for dinner making
<mwhudson> programming can be terrible some times
<wgrant> What's going wrong?
<mwhudson> oh just zope.interface crap
<wgrant> Ah.
<stub> The librarian could be mocked for many tests - the transaction rules are well documented and the interface simple. Nobody has bothered though (since it is a per test-run cost rather than a per-test cost?)
<lifeless> stub: I think you mean stubbed (no pun intended) - Mocks are not suitable for keeping true to the transaction rules.
 * wgrant tries to remember the URL to that stubs vs. mocks. vs. fakes thing that was around a while ago.
<lifeless> wgrant: all are test doubles; mocks assert a sequence of calls with seeded result values
<wgrant> Ah.
<wgrant> I can never remember which is which.
<lifeless> stubs are generally limited functionality doubles that do stuff
<lifeless> and fakes do even less
<lifeless> anyhow, the rule to remember is 'mocks do not meet contracts' - each and every test has its own mock, or they aren't actually mocks.
<lifeless> on the upside you can create a mock by decorating a realobject and recording the input/output sequence it sees.
<lifeless> which can be a good way to make mocks, and with a little glue, turn them on/off on a per-run basis
<lifeless> to let one validate that the mock hasn't altered test validity asynchronously.
<mwhudson> it's on mumak.net somewhere i currently can't find
<mwhudson> _so many_ places reference next_mirror_time :(
<MTecknology> stub: hi
<MTecknology> stub: we were just talking about you earlier
<mwhudson> and mostly for lame, or at least non-intent revealing, reasons
<stub> Talking about me for lame and non-intent revealing reasons? O_o
<MTecknology> I decided I can either do more crappy documentation in the 30min I'll be awake, start reading an article I need to do a presentation on, tweak my bot some, install stumbleupon and use it, or..... do nothing.... Problem is that none of them sound appealing
<MTecknology> stub: I was asking about stu and thumper thought I was talking about you. But then we did talk about you and I realized we like the same kind of ice cream
<stub> stu flavoured ice cream? This is getting more dubious.
<MTecknology> lol
 * mwhudson eods
<wgrant> stub: Can you have a quick glance at http://pastebin.ubuntu.com/307257/ and tell me if i'm doing anything horrifyingly forbidden ? I'd really really like to get this in before 3.1.10, and need to discuss it with Soyuz people tonight. Basically, I need a way to restrict permissible source package formats by distroseries. This is it.
<stub> wgrant: We normally use a integer/enum instead of a string where there is a fixed set of allowed options, and strings for free text. I suspect 'format' is not free text and should be an enum.
<stub> wgrant: The table name sounds a little dubious, but that will shake out in discussions with soyuz. Is this table defining the formats that are allowed for a distroseries or the formats that have been used in a distroseries?
<wgrant> stub: Right, I wondered about that. But to be consistent, that would mean migrating a column (sourcepackagerelease.dsc_format) from a string to an enum too.
<wgrant> stub: It's the same scheme used for ComponentSelection and SectionSelection, but I will discuss it with Soyuz. It defines the formats allowed in a distroseries.
<stub> We are inconsistent in many places - if you are correct, adding a comment in comments.sql is fine rather than migrating the column now. We can't be as agile with the DB schema as we are with Python.
<stub> a string field might be fine in this case if it is 'whatever apt or tool x tells us' rather than some value Launchpad needs to actually understand.
<wgrant> All values in that column should currently be identical, but it still sounds messy.
<wgrant> It does have to be understood.
<wgrant> At the moment there is only one valid value. After my branch, there will be three, defining different sets of allowed files in the sourcepackagerelease.
<wgrant> The formats are defined by dpkg.
<stub> So in general with the Python code we want to avoid "if format == '1.0'" and instead have "if format == PackageFormat.WHATEVER"
<stub> If it turns out in discussion that a string is the right choice for this column, I'll want some vague justification.
<stub> But no babies will die either way.
<wgrant> What's the best way to do this? Define the DBEnumeratedType and then a dict mapping '1.0', '3.0 (quilt)' etc. to its items? Or does DBEnumeratedType have some builtin way to do this, like lazr.restful seems to use?
<sridher> got an error to run > make schema
<wgrant> sridher: Which error?
<sridher> http://paste.ubuntu.com/307272/
<wgrant> sridher: Python 2.6 is not supported.
<wgrant> sridher: Only 2.4 (and nearly 2.5, but it's not merged yet) at this point.
<sridher> ok thank you sir
<wgrant> sridher: Did you use the rocketfuel-setup script as described on https://dev.launchpad.net/Getting?
<sridher> yes sir
<sridher> i got error that "developer-dependency" package not found
<wgrant> sridher: Which version of Ubuntu are you using?
<sridher> karmic
<wgrant> That should just work.
<wgrant> Try running rocketfuel-setup again.
<sridher> ok sir
<sridher> Package `launchpad-developer-dependencies' is not installed and no info is available.
<wgrant> apt-get update
<sridher> ok
<sridher> http://paste.ubuntu.com/307275/
<wgrant> sridher: You have the launchpad PPA enabled (probably in /etc/apt/sources.list.d/launchpad-dev.list)?
<sridher> yes sir
<wgrant> sridher: pastebin the output of 'apt-get update'
<sridher> http://paste.ubuntu.com/307278/
<wgrant> I do not know what the problem is.
<wgrant> Unless you are using something other than i386 or amd64.
<sridher> iam amd64
<wgrant> What is in /etc/apt/sources.list.d/launchpad-dev.list?
<sridher> deb http://ppa.launchpad.net/launchpad/ppa/ubuntu karmic main
<sridher> deb http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main
<wgrant> That's fine.
<sridher> is it work with python 2.4?
<wgrant> sridher: Yes. I presume you modified your copy to use python2.6 manually?
<wgrant> al-maisan: Morning.
<al-maisan> good afternoon wgrant, how are things?
<sridher> after getting an 2.4 error i changed /devel/Makefile version to 2.6
<wgrant> al-maisan: Not too bad. Just finishing off my v3 source format branch. One thing I came across in the tests was that the 'absolutely-anything' upload policy doesn't accept PPA uploads -- do you know if this is deliberate? I need it to for my tests, and enabling PPA uploads in it doesn't break other tests in lp.archiveuploader.
<wgrant> But who knows what other tests might lurk elsewhere.
<al-maisan> wgrant: I don't know off-hand but will take a look.
<wgrant> al-maisan: (this branch recently became very important, as Debian has started uploading 3.0 (quilt) sources)
<al-maisan> wgrant: gotcha.
<al-maisan> wgrant: would the importance be diminished in any way of we sunc'ed lucid from testing as opposed from unstable?
<al-maisan> s/of/if/
<wgrant> al-maisan: That would delay it by ten days at most, and with the current backlash it's unclear exactly what is going to happen to Lucid syncs.
<al-maisan> wgrant: I see.
<al-maisan> wgrant: re. the 'absolutely-anything' upload policy I find this snippet in AbstractUploadPolicy.rejectPPAUploads():
<al-maisan>         """Reject uploads targeted to PPA.
<al-maisan>         We will only allow it on 'insecure' and 'buildd' policy because we
<al-maisan>         ensure the uploads are signed.
<al-maisan>         """
<al-maisan> wgrant: why would it be so important for PPA uploads to be signed?
<wgrant> al-maisan: Seems like absolutely-anything (only used in tests) should be safe, however.
<sridher> http://paste.ubuntu.com/307284/
<wgrant> al-maisan: I don't know. It works OK once I enable it in the policy, so the code is fine with unsigned uploads.
<wgrant> There is some special signature treatment with PPA uploads (stripping signatures from stored .changes), but that seems to handle a lack of signature fine.
<al-maisan> wgrant: if the policy in question is only used in tests then I guess "It's better to beg forgiveness than ask permission" applies ;)
<wgrant> al-maisan: OK. Thanks.
<noodles775> Morning
<wgrant> Morning noodles775.
<noodles775> Hi wgrant :)
<wgrant> noodles775: Almighty RM, I need your advice.
<noodles775> thumper: your branch is rc-approved... (just in case you didn't see the email).
<noodles775> wgrant: for the v3sources?
<wgrant> noodles775: Right. Debian has started uploading them, so it's pretty important no.
<wgrant> +w
<noodles775> yep, definitely. But it's also the kind of thing that I'd like to have tested on dogfood thoroughly before r-cing, which means it might be best as a CP... but bigjools will be around later...
<wgrant> noodles775: The branches are pretty much done (save a conversion from a string to an enum in one place, which I'm finishing off now), but they're bigish.
<wgrant> Right.
<wgrant> But there are DB changes (safe) that are needed.
<noodles775> He'll be able to check your branch on df...
<noodles775> ouch
<noodles775> btw: Thanks so much for gettin in there and fixing those!
<wgrant> But assuming those get into 3.1.10, there are CPs required on Soyuz machines and appservers.
<wgrant> Plus somebody needs to get the new dpkg on the buildds and Soyuz machines... not sure who to talk to about that.
<wgrant> But lucid won't open for a few days, and the v3 packages can be survived without for a while, so that's not quite as critical as the DB changes.
<noodles775> right. Have you got a branch with the db changes ready for review?
<wgrant> Not on its own yet, as I need to convert a column to an enum.
<wgrant> There is a new table, plus some new enum values on SourcePackageFileType that will cause appservers to explode if they don't know about them.
<wgrant> appservers and builddmaster, that is.
<noodles775> OK. So assign the db changes to stub+jml, bigjools can take a look when he's in and plan the best course of action.
<wgrant> Will do. Thanks.
<noodles775> Thank you!
<wgrant> noodles775: Any recommendations for what to rename the table listing permitted formats by distroseries? I've currently called it SourceFormatSelection (because of ComponentSelection and SectionSelection), but I'm sure that's suboptimal.
<noodles775> wgrant: re-name?
<wgrant> noodles775: Well, to you it's just naming it.
<noodles775> :)
* noodles775 changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.10 Release Critical - release manager is noodles775 (flacoste, BjornT)| I am Zero OOPS and So Can You! http://is.gd/4fkLl | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
* noodles775 changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.10 | PQM is release-critical Release Critical - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
* noodles775 changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.10 | PQM is release-critical only - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<noodles775> wgrant: SourceFormatSelection seems fine to me - if it's suboptimal, it's because of the other columns that you've based it on. The db reviewers will be in a better position to comment.
<wgrant> noodles775: Well, stub said to ask Soyuz...
<noodles775> wgrant: ah ok. so, as I said, SFSelection seems fine to me. bigjools will comment later I'm sure .
<thumper> noodles775: ta
<wgrant> What's the Storm equivalent of an EnumCol?
<wgrant> StormMigrationGuide has that as an unanswered question down the bottom.
<noodles775> wgrant: I haven't used it, but http://twistedmatrix.com/users/radix/storm-api/storm.properties.Enum.html
<noodles775> gr, no, that's not a db enum.
<wgrant> Looks like it is.
<wgrant> I see it used for that purpose in PackageCopyRequest, with a utility to convert the DBET.
<noodles775> Great.
<wgrant> Thanks.
<noodles775> np.
<wgrant> noodles775: Looks like canonical.database.enumcol.DBEnum is what I actually want, as it supported DBETs directly.
<noodles775> wgrant: good to know.
<noodles775> wgrant: can you update the migration guide (if you haven't already). Thanks!
<wgrant> noodles775: Will do.
<adeuring> good morning
<wgrant> Can I look up a DBItem given a DBEnumeratedType and the item's title?
<al-maisan> Good morning adeuring
<adeuring> hi al-maisan!
<noodles775> wgrant: why not use the actual enum? Like PackagePublishingStatus.PENDING? (but you can do PackagePublishingStatus.getTermByToken('Pending') too?).
<wgrant> noodles775: I need to convert a string representation of a format from the .dsc (eg. '3.0 (quilt)').
<wgrant> I'll try that.
<noodles775> Great.
<wgrant> I thought that would go the other way.
<wgrant> Oh, woops. Was wondering why my enum was throwing strange SyntaxErrors. Then realised the names started with digits.
<wgrant> FORMAT_1_0 rather than 1_0, perhaps?
<bigjools> wgrant: good <insert time of day here>, I hear you're working on https://bugs.edge.launchpad.net/soyuz/+bug/293106
<mup> Bug #293106: does not support debian v3 source formats <feature> <motu> <ppa> <soyuz-upload> <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/293106>
<wgrant> bigjools: Good morning. I am. Shall we discuss the plan?
<bigjools> please do
<bigjools> is it blocking anything in the distro that you know of?
<bigjools> I assume some v3 packages need to be synced now?
<wgrant> Not right now, but Lucid will open in a few days, and the first v3 packages have appeared in Debian.
<wgrant> Then it will be blocking things, although not critically to start with.
<wgrant> Now, the code is done and works.
<bigjools> ok can this bump to the 3.1.11 milestone?
<wgrant> I don't think it can.
<bigjools> massive changes 2 days before a release make me very nervous
<wgrant> However, if we get some safe bits in now, the rest can be CPed to Soyuz machines at your leisure.
<bigjools>  /o\
<bigjools> I love coming back from holidays!
<wgrant> All that needs to be applied everywhere is a new table, a few new SourcePackageFileType values, a trivial function on DistroSeries, and four lines in CopyChecker.
<bigjools> wgrant: what have you done to fix it?
<wgrant> The big invasive changes are only needed on cesium, germanium and cocoplum.
<bigjools> have you got a branch I can see?
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection has it all. I'm just finishing off a change recommended by Stuart.
<bigjools> and why can't it wait a month, do you expect a lot of v3 packages?
<wgrant> I don't know. But distro people were eeking about it, and Lucid will freeze very early.
<wgrant> I don't think anyone can say how many v3 packages will appear in the next month and a bit.
<bigjools> is there a workaround to sync them if we can't upload v3?
<wgrant> They should be convertible to 1.0 with a little effort, I guess.
<bigjools> wgrant: why the new table?
<wgrant> bigjools: To restrict which series can have the new formats.
<bigjools> why is that needed?
<bigjools> I don't know anything about the new format BTW :/
<wgrant> bigjools: Only >= karmic's dpkg supports the new formats.
<bigjools> ok
<wgrant> So the old chroots will be very unhappy (and maybe chrootwait; I'm not quite sure)
<al-maisan> could we control the source format selection w/o a schema change, at least initially?
<al-maisan> i.e. by a piece of python logic that ascertains "Only >= karmic's dpkg supports the new formats"
<wgrant> You could hardcode the series name, I guess.
<bigjools> ew
<wgrant> Yes.
<al-maisan> .. and clean up things in 3.1.11
<bigjools> I'm not sure what that would buy us, it's still an invasive change
<al-maisan> no question about that
<al-maisan> but we'd at least avoid "last minute" db schema changes
<wgrant> Absolutely awful timing, I know, but Debian only revealed its intentions at the end of last week :(
<bigjools> I am still -1 on this going in unless someone from distro steps in and says it must
<wgrant> Right.
<wgrant> I'm talking to them now.
<bigjools> where "someone from distro" = cjwatson :)
<bigjools> this is a 4k diff branch - that's just insane to put in at this stage
<bigjools> err wait
<bigjools> diff fail
<wgrant> Last I checked it was 1.6klines
<al-maisan> 1581 lines of diff
<al-maisan> still a massive change
<wgrant> And a tiny fraction of that is needed *right now*.
<wgrant> But yes, it is big.
<bigjools> ok now I am diffing against db-devel :)
<wgrant> Ahaha.
<wgrant> That would do it.
<bigjools> wgrant: your sample data change changes the archive table, why?
<wgrant> bigjools: external_archive_deps change wasn't in there previously.
<bigjools> ah you can blame me for that
<wgrant> Yup :P
<wgrant> SourceFormatSelection is changing to an enum at the moment.
<bigjools> you're removing the table?
<wgrant> No.
<wgrant> It's just going to use an enum rather than a string for the format.
<bigjools> wgrant: from canonical.database.enumcol import DBEnum
<wgrant> bigjools: By that I mean I'm rerunning the tests after completing the change.
<bigjools> wgrant: what's different with the 3.0 format then?
<wgrant> bigjools: New types of files.
<wgrant> And massive incompatibility.
<wgrant> And new compression formats.
<wgrant> (although we're following dak and not supporting lzma, only gzip and bzip2, at cjwatson's direction)
<bigjools> lzma dropped entirely?
<bigjools> interesting
<wgrant> Right. Trivial regex change to bring it back if we need to.
<wgrant> (lzma deb support remains)
<wgrant> Oh. Curses. SourcePackageFormat already exists.
<wgrant> Any suggestions on what to call it?
<bigjools> SourceUploadFormat?
<wgrant> That's even more of a lie than the old SourcePackageFormat.
<wgrant> It's a property of the source package, and nothing really to do with the upload.
<bigjools> yeah
 * bigjools -> OTP
<wgrant> OTP?
<wgrant> This #ubuntu-devel is going to get messy!
<wgrant> +discussion
<wgrant> (my best idea is to rename the old SourcePackageFormat to SourcePackageType or something like that, as it is only mentioned on 7 lines, but I guess that would get me even more murdered than I will anyway)
<bigjools> wgrant: ok back (On The Phone BTW)
<wgrant> bigjools: Oh, right.
<bigjools> wgrant: +1 on renaming SourcePackageFormat
<wgrant> bigjools: Great, 'cause I've done it.
<bigjools> heh
<bigjools> SourcePackageType ?
<wgrant> That's the one.
<bigjools> perfecto
<bigjools> one day we might even start using it for something useful and support RPM
<wgrant> Perhaps.
<wgrant> Hm.
<wgrant> FS corruption.
<wgrant> That is not good.
<wgrant> (product.py started complaining of SyntaxErrors, when I hadn't touched it. it is now full of garbage)
<wgrant> I wonder what else just got eaten.
<bigjools> ext4? :)
<wgrant> indeed.
<wgrant> But the file size changed.
<wgrant> So it's not the OMG KITTENS ARE DIEING BUT ONLY THREE PEOPLE HAVE REPORTED IT bug.
<wgrant> bigjools: Can I safely raise an UploadError in one of NU's error generators if I want to reject the upload and stop right there, or do I have to move the check to another function?
<bigjools> good question
 * wgrant traces it all the way up.
<bigjools> suck it and see I guess
<wgrant> Just this last failure case to handle...
<bigjools> wgrant: EarlyReturnUploadError
<wgrant> bigjools: Ah, handy. Thanks.
<wgrant> (NU is too convoluted)
<bigjools> no kidding
<bigjools> it was refactored big-time a couple of years ago
<wgrant> I recall.
<wgrant> But that didn't get finished.
<bigjools> and it improved things a lot, if you can beloeve that
<wgrant> I cannot.
<bigjools> :)
<wgrant> Maybe I will revert the tree later and realise how lucky I am.
<jml> ahh, nascentupload
<wgrant> jml: You were privileged to be able to experience it before leaving engineering.
<jml> wgrant, indeed.
<jml> wgrant, I hope I made an improvement
 * wgrant isn't touching those bits.
<wgrant> NU might not be so bad if it was properly tested, too.
<deryck> Morning, all.
<wgrant> bigjools: OK, it seems that most of the rest of my FS survived. I've just finished off that change.
<bigjools> wgrant: most of? :)
<wgrant> bigjools: Well, I've only found that one eaten file. All the packaged files check out fine, my branches are fine, and fsck is happy, so I presume most of it is OK.
<wgrant> But it seems odd that there would be only file, given that I haven't written to it.
<bigjools> yes
<wgrant> bigjools: Broken sample data is currently being fixed :/
<bigjools> wgrant: what's broken?
<wgrant> bigjools: There is a firefox-0.9.2.orig.tar.gz used in tests (note the '-' in place of '_'). That previously worked, as the type check just looked for endswith('.orig.tar.gz'). But now it uses a regex which needs a '_'.
<bigjools> wgrant: this is in the data/suite/* stuff?
<wgrant> bigjools: No, no. archiveuploader would never accept that. This is in a doctest that grabs an existing LFA and SPR.addFile()s.
<bigjools> ah ok
 * wgrant directs significant volumes of hate at even more crackful sampledata.
<leonardr> jml, you seem to know a lot about rickspencer3's whereabouts. do you know where in the world is rickspencer3 today?
<jml> leonardr, if I had to guess, I'd say he's in the west coast of the continental united states of america.
<leonardr> hmm
<leonardr> i will hunt him down!
<jml> leonardr, I only half-followed your recent conversation with al-maisan -- is exposing a dict over the API really that tricky?
 * al-maisan cocks up one ear
<leonardr> jml: it's never been done before. making the lazr.restful changes to expose a dict should not be *that* difficult
<leonardr> but al-maisan wanted to expose a dict _of lists_
<jml> leonardr, why is that important?
<wgrant> Wasn't it a dict of Entries?
<leonardr> al-maisan can correct be, but i believe it was a dict that mapped a person to a list of strings
<jml> there are now two interesting branches of this conversation :)
<jml> leonardr, isn't a dict of lists a fairly standard json construct?
<al-maisan> leonardr: actually it was a dict {pocket : object } .. let me check again
<jml> al-maisan, it was mapping a pocket enum to a branch object iirc.
<al-maisan> that would be right
<leonardr> i don't remember this. i remember people mapped to the types of reviews they'd been assigned to
<leonardr> anyway, the problem is that the arguments to a named operation have to be described in wadl, and recursively defined data types are difficult to define in wadl
<jml> leonardr, ahh, I see.
<leonardr> a recursively defined data type is a sign that things are getting too complicated and you should rethink your resource design
<jml> leonardr, I can see the point
<wgrant> Ahhhh. Tests pass a little better when one of the data files used in a few tests *hasn't* been corrupted by a stupid filesystem.
<jml> as much as I like "for your own good"-style design.
<jml> *dislike*
<jml> (one of them Freudian mothers)
<al-maisan> leonardr, jml: this is what I was trying at that time: http://pastebin.ubuntu.com/300148/
<leonardr> jml: what i've said is kind of seat-of-pants stuff. my knowledge of that part of lazr.restful is cached on disk.
<jml> leonardr, heh
<leonardr> somewhat literally, in that i would have to go into the code and re-learn it
<al-maisan> leonardr: : so, the problematic part in this case was the return type of the collection returned.
<al-maisan> anyway, since that did not work I resorted to: https://code.edge.launchpad.net/~al-maisan/launchpad/352094
<jml> leonardr, I know the feeling.
<mpt> What's a "package set"?
<mpt> I see <https://dev.launchpad.net/Soyuz/Specs/PackagesetsAndDistroseries> and <https://dev.launchpad.net/Soyuz/Specs/PackagesetsAndDistroseries/OriginalAnalysis>, but they don't seem to mention what a package set is
<al-maisan> mpt: https://dev.launchpad.net/VersionThreeDotO/Soyuz/StoryCards#packagesetacl
<mpt> ahh, security
<mpt> thanks al-maisan
<al-maisan> you are welcome :)
<Ursinha> hey bigjools-lunch :) can you tell me what's exactly the meaning of the pending milestone you have in soyuz? after getting back from lunch, of course :P
<bigjools> Ursinha: one word: "Celso"
<Ursinha> bigjools, oh crap
<Ursinha> bigjools, but do you know his rationale behind it?
<bigjools> Ursinha: it was a place for him to put stuff he didn't want to forget about
<Ursinha> bigjools, I see
<Ursinha> bigjools, you're not using it?
<bigjools> and it turned into a general pool of bugs that we need to sort out
<bigjools> sort of
<Ursinha> bigjools, I see
<Ursinha> bigjools, I asked because I have some bugs that are depending on others to be fixed, not our bugs, and as we don't have a Pending status I considered using a Pending milestone
<bigjools> Ursinha: as you know, things have changed focus recently so it's fallen to the side a little.  But its general concept will be re-used for LP generally soon.
<bigjools> ah
<bigjools> go for it
<Ursinha> cool
<Ursinha> thanks bigjools :)
<bigjools> any time
<gary_poster> bigjools: hi!  I have this on my QA page from Francis.  Am I right that I can push it over to your guys' page? "r9807 [r=adeuring][ui=none][bug=458833] Protect Build attributes using launchpad.Edit. (Landed on behalf of Julian.)"
<bigjools> gary_poster: yes please do
<gary_poster> bigjools: cool thanks
<bigjools> gary_poster: are you going to stick it on my QA wiki page?
<gary_poster> bigjools: sorry yes.  Trying to reply to mthaddon on a nother channel then was going to do that.  If you want to do it before me, feel free. ;-)
<bigjools> gary_poster: yeah I can do that, what's your test plan URL?
<gary_poster> bigjools: thanks https://dev.launchpad.net/FoundationsTestPlan/3.1.10
<bigjools> shoulda guessed that really :)
<gary_poster> :-)
<bigjools> gary_poster: umm release yer edit lock :)
<gary_poster> bigjools: sorry done
<bigjools> ta
<bigjools> done
<wgrant> bigjools: Have you a definitive answer from cjwatson?
<bigjools> wgrant: not definitive, but there's no clear requirement that I can see for putting this in 3.1.10
<bigjools> wgrant: if the default dpkg format changes then he points out that we're fucked
<wgrant> bigjools: Right. And that was planned to happen.
<wgrant> But might not be any more... hard to tell.
<sinzui> bac: EdwinGrubbs: barry: I was able to test the product-release-finder. 1 bug is fixed, the other is not. I am going to work on the bad item
<bigjools> wgrant: did stub review your db changes
<wgrant> bigjools: He commented that the table name was a bit odd, but that it was a Soyuz decision, and also that it should use an enum rather than a string.
<wgrant> The latter is done, and you seem fine with the former.
<bigjools> wgrant: yeah it follows some convention I guess
<bigjools> wgrant: so I am thinking that we should land the schema changes only, in case the brown smelly stuff hits the fan this month
<wgrant> The cut down (that is, without the big archiveuploader changes) diff (excluding sample data) is ~500 lines.
<barry> sinzui: cool
<bigjools> wgrant: we don't even need that, we can just land the sql patch on db-devel
<bigjools> that's the thing that would stop a CP if it came to that, so this increases our options
<wgrant> True.
<bigjools> so if you can make a separate branch with that, create an MP, get jml and stub to review it, I'll land it for you
<bac> sinzui, barry: here are the portlets called on the distroseries page:  http://pastebin.ubuntu.com/307675/
<bac> sinzui, barry: for karmic, all render but the third one:  https://edge.launchpad.net/ubuntu/karmic/@@+table-milestones
<sinzui> bac: I bet it is the status counts for bugs and blueprints
<sinzui> bac: barry: The counts are a loop in python, Maybe the code can be made faster, or we can change the model to provide the summary. I prefer the first option because the code is also used to count assignees
<sinzui> ps. you rock bac
<Ursinha> hi sinzui, about the BAD items, I see you're working on bug 415595. do you know about the others?
<mup> Bug #415595: Product release finder should import all files <story-product-release-finder> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/415595>
<sinzui> We are working on all bad items
 * sinzui has a fix for Bug #415595
<mup> Bug #415595: Product release finder should import all files <story-product-release-finder> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/415595>
<Ursinha> sinzui, right! in your opinion, any of these are release blockers?
<kfogel> http://paste.ubuntu.com/307767/
<kfogel> anyone know why my rocketfuel-get is failing as per above paste?
<barry> sinzui: i'm back now; what can i look at?
<kfogel> "bzrlib.errors.BzrError: xmlrpc protocol error connecting to https://xmlrpc.edge.launchpad.net/bazaar/: 502 Bad Gateway" ?
<sinzui> Ursinha: I think the timeouts on distroseries defintely are
<sinzui> Ursinha: the release finder could slip
<Ursinha> sinzui, right
<sinzui> The bad CSS must be fixed or we can revert the entire branch
<kfogel> jml: curious if you recognize the error (see my paste a few lines above)
<jml> kfogel, it rings a bell. are you behind a proxy?
<kfogel> jml: indeed I am.  But I could easily not be.
<jml> kfogel, try that.
<Ursinha> sinzui, are you working in the old bug numbers or have you opened new ones?
<Ursinha> by you I mean registry team
<sinzui> Ursinha: I reopened the old because because our changes do not represent a partial fix
<bac> barry: ping
<barry> bac: pong
<Ursinha> sinzui, and the others?
<sinzui> I am testing the one that I can test.
<sinzui> Ursinha: I do not know how to test salgado's change. I am not even sure it is a regisrty issue
<sinzui> barry: ping
<Ursinha> sinzui, actually I'm talking about the BAD items you say registry team is working on. I see you reopened the bug of yours, want to know about the others :)
<sinzui> They are inprogress I think
<barry> sinzui: pong
<sinzui> Ursinha: https://edge.launchpad.net/launchpad-registry/+milestone/3.1.10
<sinzui> ^ sort by status
 * Ursinha looks
<sinzui> barry: I was going to ask for you help with new member + mailing list tests, but I get an oops with my test candidate: https://staging.launchpad.net/~haibunku
<Ursinha> thanks sinzui
<barry> sinzui: that oops looks related to my team privacy portlet change
<sinzui> yes
<barry> sinzui: like. some code didn't get updated perhaps?
<sinzui> but I saw your style fix on edge it it worked
<barry> sinzui: does it work in lp.dev?
<barry> (i should try it)
<sinzui> barry: staging is borked
<barry> sinzui: gotcha
<sinzui> barry: db-deel has the right template, but not the right view
<sinzui> barry: did you land the fix in two branches?
 * sinzui sees the visibility_portlet_class in devel but not db-devel
<kfogel_> jml: urgk.  this one ring a bell to you? http://paste.ubuntu.com/307835/  "bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for parent_id_basename_to_file_id maps"  (I've gotten out from behind the proxy now.)
<barry> sinzui: i don't think so.  i had two bugs each with a branch but that particular feature should have landed in one branch
<barry> sinzui: oh, that's sick
<jml> kfogel_, umm, no, sorry.
<jml> kfogel_, that's #bzr territory
<barry> sinzui: i'm grabbing db-devel now
<sinzui> barry: ../db-devel/lib/lp/registry/browser/person.py line 3248 should be our missing method
<barry> sinzui: how did registry/templates/team-index.py get the change but not browser/person.py?
<barry> sinzui: oh wait
<sinzui> barry: gremlins from the kremlin? Aliens for Mars? The Illuminati?
<barry> sinzui: db-devel's team-index.pt did not get the change
<sinzui> oh, I revered the issue?
<kfogel_> jml: thought probably -- but you did so well with the last question that I wanted to ask :)-.
<barry> sinzui: i just bzr-tool-grep'd db-devel.  there's no visibility_portlet_class in the code at all
<barry> sinzui: so, wtf is staging running?!
<Ursinha> hi noodles775, are you there?
<Ursinha> hmm, no
<Ursinha> is it just me or edge is really slow?
<rockstar> Ursinha, I've noticed it's really slow as well.
<Ursinha> rockstar, maybe we should poke a losa
<rockstar> Ursinha, yeah, maybe.  Forking to do that is not something I have bandwidth for right now though.
<Ursinha> hehe
<Ursinha> mbarnett, hey
<mbarnett> Ursinha: heya
<mbarnett> Ursinha: we are in the midst of a rolling update
<mbarnett> Ursinha: which is why edge is a bit slow
<Ursinha> mbarnett, oic
<Ursinha> rockstar, ^
<Ursinha> mbarnett, for how long will it be like this?
<mbarnett> Ursinha: i'll check on the progress
<Ursinha> mbarnett, cool
<beuno> barry, can you think of any reason why we wouldn't want to expose what mailing lists people are subscribed to on ther profiles?
<Chex> Ursinha: sorry, one of the edge workers was slow to come up, should be OK now, let me know if it isn't?
<mbarnett> what he said!
<barry> beuno: for public lists?  i think it would be interesting to know that
<Ursinha> sure mbarnett and Chex
<Ursinha> thanks both
<Ursinha> :)
<beuno> barry, perfect answer
<barry> beuno: facepad!
<thumper> why has no-one booted buildbot into touch?
<thumper> the db_devel builder is still fubared
<thumper> and we're in testfix because of it
<thumper> it has been over 12 hours!!!!!!
<beuno> barry, don't reveal my secret plan!
<barry> beuno: as long as it's not spacepad :)
<beuno> barry, do we store ML messages on the db at all?
<barry> beuno: in general, no.  we only store ones that are held for moderation
<jml> thumper, hi
<thumper> jml: hi
<beuno> barry, how complicated would it be to store subjects and authors?
<jml> thumper, up for a skype call?
<thumper> jml: how long do you need?
<thumper> jml: have normal stand up soon
<jml> thumper, ~30 mins. When's your standup?
<thumper> in 10 minutes
<barry> beuno: you'd probably also want message-ids.  i think not complicated, but i don't know how expensive it would be (performance- or space- wise)
<beuno> barry, and date  :)
<barry> beuno: you probably just want to store successfully posted messages, so you'd need to hook into that process, but i think not hard to do
<barry> beuno: yep
<beuno> barry, so we could tell people what kind of traffic a ML has, and what the latest messages where
<beuno> barry, secondly, are there any good proprietary archiving softwares?  (you've said there aren't any open ones)
<barry> beuno: well, if you did that, you'd /have/ to give them links to those messages.  /me remembers our last conversation
<jml> thumper, is after the standup ok?
<thumper> jml: yeah, I think so
<barry> beuno: i'm not familiar with any commercial archivers
<jml> thumper, cool. can you ping me please?
<thumper> yep
<beuno> barry, ok, thanks
<mwhudson> morning
<james_w> morning mwhudson
<james_w> mwhudson: thanks for your work on branch-distro.py
<mwhudson> james_w: np, is everything working ok for you?
<james_w> seems to be fine
<mwhudson> cool!
<thumper> rockstar: want to host?
<rockstar> thumper, sure.
<rockstar> mwhudson, abentley, skype?
<mwhudson> rockstar: yep
<Ursinha> hello gary_poster, could you triage bug 423447, please?
<mup> Bug #423447: Trying to add an email that's already registered as a SSO account but not a person, oopses <oops> <Launchpad Foundations:New> <https://launchpad.net/bugs/423447>
<gary_poster> Ursinha: a bit busy on buildbot right now
<Ursinha> ok gary_poster, no problem
<thumper> jml: ready
<jml> thumper, oh sorry
<gary_poster> Ursinha: triaged
<Ursinha> gary_poster, thank you
<maxb> gary_poster: Hi! Good news: ztk-2.5 still fine as of my weekend re-test. Bad news: There's a new test, xx-resetpassword-for-sso-account.txt, on devel that *requires* canonical-identity-provider to pass - bit of a downer for me!
<maxb> (sorry to pick on you, but salgado isn't here, and it was [r=gary] :-) )
<gary_poster> maxb: heh and urg
<maxb> Let me know if you'd like it filed as a bug
<gary_poster> maxb: I'm swamped ATM.  Make a bug for me, and I'll assign it?  Yes please
<maxb> Filed as bug 471724
<mup> Bug #471724: Test lib/lp/registry/tests/../stories/foaf/xx-resetpassword-of-sso-account.txt fails if canonical-identity-provider absent <Launchpad Foundations:New> <https://launchpad.net/bugs/471724>
<wgrant> jml: Can you please review https://code.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection-db/+merge/14304?
<jml> wgrant, will do.
<jml> wgrant, if not tonight, first thing tomorrow
<wgrant> jml: Thanks.
<jml> wgrant, np
<mars> silly question: do we still use the lpwindmill.py script for windmill test authoring?
<mars> if so, where may I find the documentation on the wiki?
<mars> rockstar, if you are still around: have you written any windmill tests recently?
<rockstar> mars, I've ported some, am probably going to write some before the sprint, why?
<mars> rockstar, I've been trying to test the YUI 3 upgrade, and was going to run the windmill suite.  However, I can't run windmill, or start the test authoring tools
<rockstar> mars, how are you doing it?
<mars> and I can't find anything about writing windmill tests on the wiki
<rockstar> mars, I think we need to update them.
<mars> rockstar, $ bin/py lib/lp/scripts/utilities/lpwindmill.py firefox http://launchpad.dev:8085
<rockstar> mars, if you'd like, I can call you and get you up to speed on the new way of running windmill.
<rockstar> mars, yeah, it's changed recently.
<mars> rockstar, that's ok, I don't have skype.  Windmill starts using bin/test, but I'm wondering about authoring
<mars> ok
<rockstar> mars, no idea how to use the authoring tools.  I've never used them.
<mars> rockstar, how did you write windmill tests in the past?
<rockstar> mars, vim
<mars> rockstar, I mean, did you use the lpwindmill script to set up a test server on port 8085, then start poking at the DOM with the windmill interface?
<rockstar> mars, no, I just had my dev server and firebug, and would write the test, run it, fix it, etc.
<mars> ok
<mars> rockstar, after upgraded to Karmic, I'm getting SSL errors when I try to run the windmill suite.  It looks like the bin/test suite is not using port 8085, at least, not to log into the site
<rockstar> mars, turn off your dev server when you run the tests.
<rockstar> Windmill and the dev server have never really played well.
<mars> I thought we fixed that :(
<thumper> mwhudson: what do you need to qa your last two items?
<rockstar> mars, not that I've ever seen.
<mars> I'll try again, doubly sure
<mwhudson> thumper: oh i did one
<mwhudson> thumper: and the other i need some queries running i guess
<mars> "Error in sys.excepthook:"  oops.
<mars> :)
<wgrant> Can I access the API anonymously yet without patching waddlib to use the browser-accessible root?
<mars> wgrant, you used the word "yet" - have you heard of  a plan to allow such access?
<mars> just curious
<mars> I've been away for a while
<rockstar> wgrant, I'm not aware of any work done.
<wgrant> mars: Well, I filed a bug about it. It was scheduled for 3.0 (I think). It didn't happen.
<mwhudson> do we still have the problem where when a query times out the time for the query that tripped the limit is not added to the sql time in the oops report
<mwhudson> ?
<wgrant> I thought that was fixed for 3.0.
<wgrant> But I was about to ask about that.
<mwhudson> yeah i thought so too
<mars> rockstar, you were right, was the dev server running as well
<wgrant> mwhudson: What was the last query?
<rockstar> mars, yeah, the biggest problem with Windmill is knowing all its quirks.  Once you know all the errors it shows, you know why you made it mad.
<rockstar> mars, this is also a great example of Stockholm syndrome.
<mwhudson> wgrant: select person.* from BugSubscription, Person WHERE Person.id = BugSubscription.person AND BugSubscription.bug = 417842 AND (1=1) ORDER BY Person.displayname
<mwhudson> wgrant: unless there is a truly insane number of subscribers that doesn't seem like it should be that slow
<mars> rockstar, LOL
<wgrant> mwhudson: That shouldn't be too bad, but I have to wonder why it is executing that at all.
<wgrant> mwhudson: Since that is loaded via AJAX in a portlet later.
<mwhudson> wgrant: yeah
<mars> rockstar, the funny thing is, I know *exactly* where this tool fits into the big picture now.  I can start thinking of ways to make it better
<rockstar> mars, cool.  Let me know how I can help.
<mwhudson> wgrant: that query is fast apparently so who knows what's going on
<wgrant> mwhudson: Using the very handy ++oops++ locally, I can see some pretty bad looking queries (selecting all bugsubscriptions for bugs that are duplicates of the current bug, and prejoining all people referred to by those subscriptions)
<wgrant> There is absolutely no reason to do that.
<mwhudson> wgrant: indeed, but they're acutally pretty quick
<wgrant> Hm. sad.
<mwhudson> so it's bad, but not the problem
<mwhudson> wgrant: that terribly looking prejoin query takes 77.0 ms
<wgrant> mwhudson: Blah.
<mwhudson> and it would be so nice if a 77ms query was unacceptable but ...
<wgrant> While there are queries going around, how do I convince somebody to run 'SELECT DISTINCT dsc_format FROM sourcepackagerelease' on production? I'd like to see just how broken the data there is, as the early sample data is shocking.
<mars> :D
<mwhudson> wgrant: mbarnett is your friend i guess
<mwhudson> wtf
<mwhudson> factory.makeBug() in 'make harness' seems to be broken
<wgrant> mbarnett: Hi there. Can I convince you to give me the result of 'SELECT DISTINCT dsc_format FROM sourcepackagerelease' on production?
<mbarnett> wgrant: sure sure
<mwhudson> IntegrityError: duplicate key value violates unique constraint "bugnotification__bug__message__unq"
<wgrant> mwhudson: Awesome.
 * wgrant tries.
<wgrant> Confirmed.
<wgrant> WTF.
<mbarnett> wgrant: http://pastebin.ubuntu.com/308005/
<wgrant> mbarnett: Excellent, thanks.
<mbarnett> wgrant: welcome
<wgrant> It's wrong, but easily fixed when we need to.
 * mwhudson makes schema to see if that helps
<wgrant> mwhudson: It doesn't.
<wgrant> mwhudson: I've got a fresh one here and it's broken like that.
<mwhudson> wgrant: wtf
<wgrant> mwhudson: That too.
<mwhudson> it's used in the tests a fair bit
<mwhudson> maybe differences between test and dev databases
<mwhudson> what the heck!
<wgrant> I tried ftest-playground already.
<wgrant> Same problem.
<wgrant> That was my first guess.
<mwhudson> bugnotification is empty before this happens
<wgrant> A subscriber might be doubled up somewhere?
<mwhudson> no, the index is just on message and bug
<wgrant> subscriber meaning Z3 subscriber, not BugSubscriber.
<mwhudson> ah
<mwhudson> yes possibly
<mwhudson> wgrant: yeah, seems possible
<wgrant> I don't know too much about the subscription system, though.
<mwhudson> wgrant: yes, seems all subscribers are added twice in make harness
<wgrant> mwhudson: Ah, that's easy.
<mwhudson> wgrant: easy to fix>
<mwhudson> ?
<wgrant> mwhudson: If you look at canonical.database.harness._get_locals, you'll see it calls execute_zcml_for_scripts, and then executes script.zcml.
<wgrant> But execute_zcml_for_scripts appears to execute script.zcml itself.
 * wgrant tests.
<wgrant> But that seems far too easy.
<mars> bad smell in the windmill python suite: the client object has to be passed around everywhere.  Need a context object or something.
<wgrant> mwhudson: Commenting out the script.zcml invocation in canonical.database.harness._get_locals fixes it, and the appropriate bugnotification is still created.
<wgrant> But I must be missing something.
<mwhudson> oh i think maybe xecute_zcml_for_scripts executes more zcml than it used to
<wgrant> Possibly.
<wgrant> It certainly executes script.zcml itself.
 * wgrant annotates.
#launchpad-dev 2009-11-03
<mwhudson> i was thinking of devel r9693
<wgrant> harness is the only place that uses script.zcml directly, so I think it's the right fix.
<mwhudson> so what was i doing?
<mwhudson> ah yes, creating a bug with zillions of duplicates
<wgrant> 9693 looks innocent enough.
<thumper> wgrant: I have a bug for you :)
<thumper> wgrant: SourcePackage needs to be renamed to something like DistroSeriesSourcePackage
<wgrant> thumper: It does. Everyone knows that :(
<thumper> :)
<thumper> and no-one wants to do it
<thumper> my JFDI juice is all used elsewhere I'm afraid
 * thumper signs off for a while
<wgrant> mwhudson: My 300-dupe bug only takes a couple of seconds to render locally :(
<mwhudson> wgrant: same here!
 * wgrant checks subscriber counts on all of 417842's dupes.
 * mwhudson adds a couple hundred subscribers to the bug too
<mwhudson> wgrant: do you want to submit a branch that comments out the script.zcml from harness?
<wgrant> mwhudson: I will, once I work out if there are any tests.
<wgrant> I presume not.
<mwhudson> i think you're probably right
<mwhudson> thanks
<mwhudson> still <4s
<wgrant> There are only 454 (public) subscribers on the dupes.
<wgrant> And 254 on the master.
<mwhudson> well, i guess i'm going to apply my SEP field now
<mwhudson> wgrant: do you know if a bug is filed?
<wgrant> That sounds wise.
<wgrant> I haven't seen one.
<wgrant> Alright.
<wgrant> Why, exactly, has my query count jumped from 140 to 215 to 426.
<wgrant> The only change I made was unsubscribing from the master between the first two.
<wgrant> And now down to 294 again.
<mwhudson> frikking wifi
<mwhudson> wgrant: have you filed such a bug or shall i?
<wgrant> mwhudson: Please do. I can't see the details of the OOPS.
<mwhudson> ok
<mwhudson> huh wow the bug loaded on prod
<wgrant> How quickly?
<mwhudson> now was that because i'm not logged in on that machine, or because it's prod...
<mwhudson> heh
<mwhudson> aaadh
<mwhudson>     At least 116 queries issued in 19.42 seconds
<mwhudson> so it just sneaked in :-)
<wgrant> ++oops++ it!
<mwhudson> yeah, it timed out the second time
<wgrant>     At least 241 queries issued in 14.22 seconds
<mwhudson> i suspect the edge appservers are more loaded than the lpnet ones right now
<wgrant> lpnet got some more added recently, didn't it?
<mwhudson> yar
<mwhudson> hm now it loaded for me on edge
<mwhudson> still not logged in
<wgrant> My lpnet time was logged in.
<mwhudson> wgrant: https://bugs.edge.launchpad.net/malone/+bug/471974 fwiw
<mup> Bug #471974: bug timing out without much sql time <timeout> <Launchpad Bugs:New> <https://launchpad.net/bugs/471974>
<wgrant> Hmmm. I cannot branch devel locally; it fails with obscure bzr errors. I wonder if the FS corruption last night was more extensive than I suspected...
 * wgrant is amused that even UX people don't like indicator-applet.
<Ursinha> anyone online up to explain me a difference between zope.schema.Datetime and canonical.database.datetimecol.UtcDateTimeCol?
<Ursinha> can I make the first timezone aware? os is it already?
<Ursinha> s/a difference/the difference/
<wgrant> The first is an interface field, the second is a Storm/SQLObject type?
<wgrant> IIRC z.s.Datetime knows about timezones.
<wgrant> Actually, I guess it doesn't really have to care.
<Ursinha> wgrant, I thought that was the inverse
<Ursinha> UtcDateTimeCol aware of tz and Datetime not
<Ursinha> ?
<wgrant> Ursinha: They are not the same sort of thing,
<Ursinha> wgrant, I'm trying to understand this error: TypeError: can't compare offset-naive and offset-aware datetimes
<wgrant> Ursinha: Where does the z.s.Datetime come from?
<Ursinha> a field declared in an interface
<wgrant> But the data comes from a form?
<Ursinha> wgrant, no, it's calculated
<wgrant> Ursinha: I think your calculation is just returning a naive datetime.
<Ursinha> wgrant, let me see
<Ursinha> wgrant, my calculation is Max(POFile.date_changed)
<wgrant> Ursinha: And what are you comparing it against?
<Ursinha> wgrant, POFile.date_changed
<wgrant> Huh.
<wgrant> They should both be tz-aware! pastebin the code?
<Ursinha> hehe
<Ursinha> I'd have to paste a lot of code :) I'm doing a change in translations that adds a last changed column to the +translations list
<Ursinha> so I added a last_changed_date = Datetime in an interface and it carries that Max(foobar) later in the code
<wgrant> Unless you're doing something pretty strange, the interface will not matter.
<wgrant> ('strange' including using form infrastructure or lazr.restful)
<Ursinha> wgrant, self.assertEquals(type(psl.last_changed_date.tzinfo), type(pofile2.date_changed.tzinfo)) returned AssertionError: <type 'NoneType'> != <class 'pytz.UTC'>
<Ursinha> wgrant, mission now is understand why
<wgrant> Ursinha: What is this psl?
<Ursinha> wgrant, productserieslanguage
<wgrant> Ursinha: is last_changed_date a new property, or a new direct DB column?
<Ursinha> wgrant, a new property
<wgrant> Ursinha: What's the code behind it?
<wgrant> (timezones suck. hate hate hate)
<Ursinha> wgrant, the changes I did in interface/productserieslanguage.py: http://paste.ubuntu.com/308142/
<wgrant> Ursinha: In non-Zopey Python, the interface does not matter except for security.
<Ursinha> wgrant, this is the bit I've changed that calculates that property: http://paste.ubuntu.com/308148/
<Ursinha> wgrant, weird thing is that for one template, the first case, it doesn't fail, for the second it does, so Max modifies or returns something different of what's stored? I'm trying to figure out where I am losing that bit of timezone
<wgrant> Ursinha: Sorry, had to run off for a while. If you look at UtcDateTimeCol, you can see it injecting the timezone manually. I'm not sure how much of that class will be used when you're not actually retrieving an object, just a field.
<Ursinha> wgrant, I could see here that max returns the datetime without tz set
<Ursinha> wgrant, actually I thought that the tz info was stored in the database, but that in fact wouldn't make much sense, since the whole system is UTC
<wgrant> Ursinha: Right, they are all just stored timezoneless as UTC.
<wgrant> Ursinha: Aha. Have a look at lp.code.model.branch.BranchCloud.getProductsWithInfo
 * Ursinha looking
<wgrant> (particularly the XXX at the end)
<wgrant> That suggests that you're not doing anything wrong.
<Ursinha> wgrant, that was exactly what I thought I was doing
<Ursinha> wgrant, thank you muchly
<Ursinha> wgrant, are you going to uds?
<wgrant> Ursinha: I'm not.
<Ursinha> wgrant, I'll never meet you in person
<Ursinha> :/
<wgrant> Ursinha: I guess what I would do here is just set tzinfo to UTC before I use it.
<wgrant> Ursinha: Maybe another UDS, I suppose.
<Ursinha> wgrant, I wonder if setting this is needed to what I'm doing
<wgrant> Ursinha: If you want to compare two datetimes, you need them to be either both naive or both not.
<Ursinha> wgrant, I only do this in the tests
<Ursinha> wgrant, with the assert
<wgrant> stub: Around for a re-review?
<stub> wgrant: yes
<stub> wgrant: I'll need some details - don't see anything on my review list.
<wgrant> stub: I'm finding a link. I haven't worked out how I'm meant to request your review yet.
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/distroseries-source-format-selection-db/+merge/14304
<stub> wgrant: You click on 'request another review', type my name (so it appears on my list) and type 'db' (although that doesn't really matter)
<stub> And again for jml
<wgrant> stub: Ah, thanks. Done.
<stub> So integer enum was ok or better than text for format?
<wgrant> Yes.
<wgrant> There are only three possible values, and it seems neater.
<stub> Table name discussed with soyuz  and SourcePackageFormatSelection seems best?
<wgrant> Yes.
<wgrant> There is an existing SourcePackageFormat enum which will be renamed to SourcePackageType soon.
<wgrant> (it's DPKG vs. RPM, which is obviously not particularly useful at this point)
<stub> wgrant: All fine & reviewed. Need jml's approval too.
<wgrant> stub: Great. I should renumber the patch now?
<stub> Any time before landing is fine.
<stub> It won't worry jml's review
<wgrant> Thanks.
<noodles775> Morning
<adeuring> good moring
<jml> good morning
<jml> wgrant, patch reviewed.
<noodles775> yay, thanks jml.
<noodles775> (and thanks wgrant *again* for your work!)
<wgrant> Thanks jml.
<bigjools> wgrant: did you by any chance check to see if my fix for the security hole worked?  no worries if not, I am doing some Q/A anyway
<wgrant> bigjools: I didn't. I might as well try now.
<wgrant> bigjools: You didn't mark the fields readonly, then?
<bigjools> wgrant: no, just fixed the zope permissions
<wgrant> bigjools: Still an insufficient fix in my opinion, as I can cause OOPSes.
<wgrant> bigjools: staging is still broken, but I presume that's because db_lp is borked.
<bigjools> it's not an oops though is it?
<wgrant> https://staging.launchpad.net/ubuntu/+source/e2fsprogs/1.41.9-1ubuntu3/+build/1307119
<wgrant> It is.
<bigjools> permission failures should not be oops, they're routine
<wgrant> If I set the status to something bad, I can make my builds OOPS.
<wgrant> It's not a permission failure.
<wgrant> It's just that normal users have launchpad.Edit over their builds.
<bigjools> oh you can set your own builds
<bigjools> right
<wgrant> Which means they can corrupt internal LP state.
<bigjools> hmmm
<bigjools> ok it can be fixed next cycle
<wgrant> Yup.
<wgrant> The big issue is fixed now, at least.
<bigjools> thanks for checking!
<wgrant> np
<wgrant> Thanks for fixing.
<wgrant> bigjools: Thanks for the review. Are you going to land it?
<bigjools> wgrant: stub will
<wgrant> bigjools: Ah, great.
<bigjools> he sometimes does a mega-landing of loads of patches
<bigjools> actually that was before we had db-devel, so he's just being kind to you this time :)
<wgrant> I see.
<wgrant> noodles775: Thanks.
<noodles775> wgrant: ... to you too :)
<stub> What am I landing?
<stub> I've got nothing pending to land so no point blocking on me.
<bigjools> stub: wgrant's db patch
<bigjools> you said you'd land it in the MP ...
<wgrant> Presumably "[it] Can land", not "[I] Can land"
 * noodles775 lands it on db-devel.
<stub> Anyone can land. I'll do it since ...
<stub> or maybe someone will beat me too it while I'm typing ;)
<noodles775> stub: oh, I hadn't started, but can continue... :)
<bigjools> stub: I am too used to you landing db patches for us :)
<deryck> Morning, all.
<noodles775> Morning deryck
<wgrant> Thanks $LANDER!
<noodles775> heh
<jtv> What classes that we use are not in the lp or canonical modules but contain DBEnumVariable instances?
<Ursinha> hi noodles775 :)
<noodles775> Hi Ursinha!
<stub> jtv: Storm stuff
<stub> jtv: https://pastebin.canonical.com/24251/
<stub> Doen't seem that helpful :-P
<jtv> stub: because it strikes me that we simply don't have enough LP objects in memory to reference all those DBEnumVariables
<Ursinha> noodles775, I tagged the bugs that are rollout blockers properly, so we can check all status doing a lp search
<Ursinha> noodles775, do you think this is useful?
<Ursinha> noodles775, it's an old idea from matsubara and I
<jtv> stub: it does eliminate a lot, I guess.
<Ursinha> noodles775, I've added the search url to the wiki page
<noodles775> Ursinha: yeah, definitely - I've been using it since you updated the CRB page with the link. It's great for an overview, but doesn't allow me to annotate things (so I'm currently still finding the CRB page helpful)
<noodles775> s/helpful/helpful too
<Ursinha> noodles775, right, I thought so
<Ursinha> noodles775, so you think it's worthy doing?
<noodles775> Ursinha: yes definitely. It showed me which bugs I wasn't tracking! (the registry ones... :) ).
<jtv> stub: it's as if we're cleaning up the orm-backed objects but keeping their contents
<Ursinha> noodles775, cool :)
<Ursinha> noodles775, are you taking care of talking to people about their outstanding QA items, or do I still have to do that? :)
<noodles775> Ursinha: I'm trying, but there have been lots of other issues happening, so I would welcome any help, thanks!
<mars> salgado, around?
<Ursinha> noodles775, all right, I'll put a branch of mine for review and get to help you :)
<salgado> mars, yep
<noodles775> Ursinha: thanks!
<Ursinha> noodles775, np :)
<mars> hi salgado, my YUI3 branch fixes up client.js so most of the launchpad functionality works again: lp:~mars/launchpad/yui-3final-upgrade
<salgado> mars, cool.  what was the problem that we were seeing last week?
<mars> salgado, I've found that windmill works well in this case
<mars> the problem last week was a change to an object, from Object() to an object literal has.  Y.extend() didn't know how to deal with it
<mars> Y.Plugin became Y.Plugin.Base - passing Y.Plugin to extend() didn't work any more.
<noodles775> Ursinha: bigjools said that bug 293106 isn't a crb (just the related landing that you noticed).
<mup> Bug #293106: does not support debian v3 source formats <current-rollout-blocker> <feature> <motu> <ppa> <soyuz-upload> <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/293106>
<Ursinha> noodles775, I just saw that
<salgado> mars, oh, I see.  glad you've figured it out
<Ursinha> noodles775, a true case where we still need that page :)
<Ursinha> noodles775, I've removed the tag
<noodles775> :)
<noodles775> Ursinha: thanks.
<jml> allenap, pushed to lp:~jml/launchpad/ec2-parry
<allenap> jml: Thanks :)
<gmb> Dammit, why is the "m" key so close to the "e" key?
<mars> BjornT, just checking the jscheck buildbout output: there appear to be two errors, but they weren't picked up by the test runner?  https://lpbuildbot.canonical.com/builders/jscheck/builds/91/steps/shell_7/logs/stdio
<BjornT> mars: are you thinking of the 'test_results: ERROR' messages?
<mars> BjornT, yes, one ERROR and one INFO
<mars> BjornT, make that two ERROR messages
<mars> ah, three
<BjornT> mars: right. the ERROR ones aren't really errors. we need to find some way of surpressing those log messages. they happen when you do an assert, and tell windmill not to really assert, just to return the result
<mars> BjornT, ok, I understand.  Not straight-forward to fix though.  I was just looking at the code.
<leonardr> bac, you're in luck. i had to upgrade the lazr.restfulclient in launchpad to test my new launchpadlib branch. so i'll be able to at least run the test before i go on vacation. once pqm opens back up you should be able to just submit my branch that changes the required version
<bac> leonardr: great
<mars> BjornT, is there any way to control the options passed to the bin/test windmill test runner?
<BjornT> mars: what do you mean? how do you want to control them? and which options are you talking about?
<mars> BjornT, the '-e' option tells windmill to not exit at the end of the test run, allowing you to poke around in the test suite.  Can I simulate that with bin/test?
<BjornT> mars: no, you can't. so far i've been using -D, or using import pdb; pdb.set_trace(), but it'd would be nice to add a way to tell windmill not to shut down
<BjornT> mars: why do you need to poke around, btw?
<mars> BjornT, some tests are failing as a result of the YUI upgrade.  I can run individual tests, but they just die.  I need to jump through many hoops to recreate the windmill test fixture.
<mars> test running works fine - test authoring is a royal pain
<BjornT> mars: ok. the easiest atm is passing in -D, so that it breaks on the first failure
<mars> right
<mars> thanks BjornT
<BjornT> mars: it shouldn't be too hard to hack BaseWindmillLayer to do what you want, though, like checking for an environment variable, and not call windmill_teardown() if the variable is there
<mars> BjornT, ah, cool idea
<bac> sinzui: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1402EB656
<bac> EdwinGrubbs: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1402EB656
<mars> salgado, hit a problem, looks like the 3.0final JS doesn't build for production?  So windmill doesn't run properly.
<mars> something is off in launchpad.js itself
<salgado> mars, what do you mean by doesn't build for production?
<mars> salgado, we combine all the JS together into one big launchpad.js for production and testing.  That file is messed up right now, so windmill won't run.
<salgado> mars, oh, I see. and we don't do that with the default config?
<mars> nope, the test config uses launchpad.js
<mars> the dev config uses individual JS files
<salgado> right
<mars> it uses either testrunner or testrunner-appserver
<mars> not sure which
<BjornT> mars: have you tried removing the lazr-js directory, and then run 'make' again?
<sinzui> bac: Edwin: I still think bug 458169 is a dup of bug 455812. the root cause is the view/portlet that is working with the milestones. Should we mark the one as a dup?
<mup> Bug #458169: Distroseries:+index page timing out <current-rollout-blocker> <timeout> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/458169>
<mup> Bug #455812: distroseries milestone timeout <current-rollout-blocker> <timeout> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/455812>
<mars> BjornT, trying now
<bac> sinzui: i'm not sure what that buys us.  if one fix covers them both then that's great.  but it might be better to leave them both open to ensure any fix works for both scenarios.
<EdwinGrubbs> sinzui: that's fine
<sinzui> bac: okay
<mars> BjornT, did not work
<mars> BjornT, looks like YUI 3.0.0 is in launchpad.js, but none of the on-page javascript is getting activated
<mars> so, why would it work in dev, but not in testing?
<mars> launchpad.js vs. individual files is the only major difference I can think of
<BjornT> mars: are there any js error messages in the browser?
<mars> no
<mars> that would prevent processing the remainder of the file, true
<mars> but nothing in the console
<BjornT> that's odd
<mars> it is
<mars> another subtle API change?
<mars> I'll hack in on-page JS debug logging, might turn up something
<noodles775> Hi EdwinGrubbs and sinzui: can you guys pls take a look at the three registry bugs listed as In progress at https://dev.launchpad.net/CurrentRolloutBlockers and update the status?
<noodles775> or let me know and I'll update it - whichever.
 * sinzui updates
 * noodles775 just sees the scrollback...
<noodles775> thanks sinzui
<Ursinha> hey bigjools :)
<sinzui> noodles775: in summary, we are asking for an RC for the CSS that fixed some links in the side portlets. The entire team is working on the distroseries timeouts. The root cause relates to summarising milestones. there may be one  bug here.
<sinzui> barry: We are meeting to discuss the software center in 16:00 UTC
<sinzui> barry: see the standup notes
<barry> sinzui: +1  just saw that (email is starting to trickle in again)
<noodles775> sinzui: um, the MP there is targeted to devel? (and has a diff of 2600 lines)
<noodles775> https://code.edge.launchpad.net/~edwin-grubbs/launchpad/all-downloads-link-sprite/+merge/14333
<sinzui> barry, bac, EdwinGrubbs: Fixing the distroseries is our top priority. I think the milestone status summaries are the cause of the 90% python processing time. There may be gross inefficiencies in counting the status/assginees for bugs and blueprints. The caching of the bugs and blueprints in the view may not be good enough. Maybe we want to adapt the objects that are being counted into a simpler object that does less indire
<sinzui> noodles775: read the comment
<noodles775> yeah, just did...
<barry> sinzui: +1
<noodles775> sinzui: but that doesn't explain why it's targeted to devel... I'd expect it to still be targeted to db-devel?
 * sinzui struggle to organise two simultaneous meetings
<sinzui> noodles775: I am too busy, talk the the people involved
<noodles775> EdwinGrubbs: ^^^
<bigjools> Ursinha: hi
<Ursinha> bigjools, guess you know what I'll ask :)
<EdwinGrubbs> noodles775: what is targeted to devel instead of db-devel? BTW, have all the devel revisions finally been merged into db-devel?
<mars> BjornT, does the lpwindmill.py script still work?
<mars> I should say, is it supposed to work?
<Ursinha> bigjools, all untested items are from al-maisan, is he testing that or you split?
<al-maisan> Ursinha: I am about to finish a LP API test script that tests all the open items.
<Ursinha> al-maisan, that's cool! :)
<Ursinha> thanks al-maisan
<al-maisan> Ursinha: you are welcome.
<BjornT> mars: no, it's not. do want it to debug the js problems?
<bigjools> Ursinha: what he said :)
<noodles775> EdwinGrubbs: the above MP. And no they haven't, waiting for the next stable->db-devel.
 * noodles775 checks stable
<mars> BjornT, it used to be the way to start the windmill environment so you could author tests.  It would start a server and set up the windmill test layer.
<mars> BjornT, for some reason JS is not executing at all in firefox under windmill.  Great :(
<mars> that would explain why there are no errors!
<BjornT> mars: you're the first one i know that would actually author a test using the windmill gui :)
<BjornT> mars: does it work if you start the normal app server, with devmode turned off?
<EdwinGrubbs> noodles775: it just says devel instead of db-devel since I used the "bzr send" to submit the mp, and I was apparently missing an option.
<sinzui> barry: I have checked that db-devel does show teams. Staging is broken and it did not get a code update in the last 24 hours. Other than forcing an update we need to wait
<mars> BjornT, how do you turn off devmode?  I tried setting LPCONFIG, but that did not work.
<noodles775> EdwinGrubbs: yep, it just leaves me nervous seeing such a large diff ;), but r-c=me for the real diff that you've listed. Updating now.
<barry> sinzui, noodles775 don't you think it's too risky to roll out to production without a stable staging environment to do testing on?  i know my comfort level with the mailman upgrade will go way up if i have a chance to test it on staging first
<EdwinGrubbs> noodles775: I would like to land it after devel gets merge into db-devel. How long do I have before pqm is closed on db-devel?
<BjornT> mars: edit configs/development/launchpad.conf
<BjornT> mars: although setting LPCONFIG=testrunner-appserver should work as well
<noodles775> EdwinGrubbs: just over 8hrs (00:00 UTC). and the current devel should be landing in db-devel any minute now (lp is currently updating the branch).
<mars> testrunner-appserver dies complaining about the missing ftest db
<mars> BjornT, shutting off devmode did the trick - now I can do "make run", and see that the JS is not loading
<mars> still no errors though
<mars> BjornT, salgado-lunch, something broken in the module machinery again.  This could take a while.
<mars> salgado-lunch, for now you can just use devmode to test features by hand, should work fine.
<barry> bac: is it insane to run your script with arg1 == 500?
<bac> barry: no, why?
<barry> bac: just in an effort to get it to timeout
<sinzui> barry: bac: EdwinGrubbs: noodles775: I propose that we timebox the fix for distroseries. If we do not have a branch in the next 6 hours, we add an exception the the milestone logic to not count IBaseDistribution.
<barry> bac: locally
<bac> barry: yes, using 500 should be very snappy.  i ran with 1000.  it did not cause a timeout, though
<noodles775> flacoste: ^^^ as I'll not be around, I'll leave that in flacoste's hands.
<barry> bac: ah
<bac> sinzui: are we using the regular conf number?
<barry> bac: that's interesting
<bac> barry: which part?
<sinzui> bac: yes
<barry> bac: that it didn't timeout locally
<flacoste> sinzui: the branch need to be reviewed and QAed on staging, and if you land it through ec2 test, you haver less time than that
<sinzui> flacoste: okay, we make the hack now, If we solve it later this week we can ask for CP
<flacoste> sinzui: that's even better!
<kiko_work> hey there!
<kiko_work> flacoste, yo
<bac> sinzui: i'm in the call.   it is now, right?
<sinzui> yes
<flacoste> kiko_work: hi there (on the phone)
<kiko_work> flacoste, cool, ping me when you are free
<Ursinha> sinzui, is bug https://bugs.edge.launchpad.net/launchpad-registry/+bug/462742 supposed to be fixed?
<mup> Bug #462742: OOPS getting a project index page as an anonymous user <oops> <Launchpad Registry:Fix Committed by intellectronica> <https://launchpad.net/bugs/462742>
<Ursinha> sinzui, I've noticed several occurrences of the oops you mentioned in your comment in the last days, but none yesterday
<sinzui> Ursinha: I think you need to talk to intellectronica. The bug affects the registry, but the origin that I understand is bugs
<Ursinha> sinzui, ok, I'll talk to him
<Ursinha> thanks sinzui
<mars> sinzui, ping re: base-layout JS macros
<sinzui> hi mars
<mars> hi sinzui, I assume you wrote base-layout.pt?
<sinzui> I did
<mars> cool
<mars> first, nice job, it is a beautifully clean file :)
<Ursinha> bigjools, there are two items from wgrant that are soyuz
<Ursinha> bigjools, I've moved them from lp general testplan to soyuz one
<mars> sinzui, just wondering which macros, if any, that sub pages should use to put their own JS files onto the page
<sinzui> hmm
<mars> I see you have load-javascript
<mars> and page-javascript
<mars> (which calls load-javascript)
<sinzui> The first is for fragemetns/iframes, the later if for a full page
<sinzui> The latter does the extra work to activate common UI behaviours for classes
<sinzui> mars: I wonder if we could simplify that if we removed the old maintemplate
<mars> sinzui, is there a converted page in registry that loads it's own JS that I could look at?
<sinzui> The timeline is an example
<mars> ok
<mars> sinzui, so object-timeline-graph and timeline-macros?
<sinzui> Yes, those should like the two items
<sinzui> mars: The design permits users to embed the timeline on their site
<mars> right, I see that.  Nice.
<sinzui> The credit goes to EdwinGrubbs
<mars> sinzui, I see that productseries-index still uses the header-epilogue
<mars> to add the unique JavaScript files
<sinzui> Does the distroseries-index?
<mars> head_epilogue
<sinzui> oh no, because the there is no timeline for that
<sinzui> It would be nice to factor the epilogue out
<mars> hmm
<sinzui> It would be nice if I did not need to register the calendar widget on som many pages
<mars> My "nice to have" front-end list is growing quickly... :)
<mars> sinzui, I bet there is a nice general solution here, just need to think about it
<mars> and I wish it was easy to create custom TAL elements!  <lp:calendar>
<mars> or <tal:use-js src="blah/blah"/>
<sinzui> My reason for nice to have is that we will have fewer bugs about calendars not activating if we have a unified solution. If the calendar were in the compressed JS, it could connect itself every time time a calendar class is used
<mars> sinzui, or we could use our own loader.  That would work too
<sinzui> yes
<flacoste> kiko_work: i'm free
<mars> src="loader?launchpad.js=3.0&calendar.js=2.0" or something
<sinzui> mars: we had (maybe still have) a flag on the view that states if the calendar is required. We use the same solution for gmap. Maybe we could have a property on a view that base-layout knows to set the JS requirements
<sinzui> EdwinGrubbs: barry: one of us needs to create a branch to stop loading the summaries on IBaseDistribution. Can either of you do that in the next two hours? I can try, but I have only 90 minutes.
<EdwinGrubbs> sinzui: I can start on that as soon as I get my branch into pqm.
<mars> sinzui, I see, you store the "I need this JS" flags on the request object.  Works, but could be better.
<sinzui> Yes, the old calendar did that
<sinzui> I copied the approach for gmap
<sinzui> gmap is at the request level, I do not think the calendar is
<mars> nope, gmap and json are though
<sinzui> Let me restate that. gmap is *currently* at the request level because there are no cases where one fragment needs the gmap and another say it does not
<sinzui> OSM might be easier
<mars> OSM?
<sinzui> OpenStreamMaps. That would permit us to test the script in the test runner
<mars> ah
<sinzui> That would also be cheaper (secure gmap costs money)
<mars> I seems the JS-to-template component glue needs some work
<kiko_work> flacoste, you are so nice!
<kiko_work> how are you?
<mars> sinzui, ok, thanks for the overview.  Now I can try to figure out why the JS rollup is not loading.
<sinzui> EdwinGrubbs: I just realised that we have MilestoneView.is_distroseries_milestone. The template/portlet could use that attribute to know that it should not get the status counts.
<EdwinGrubbs> sinzui: ok, I'll use that
<flacoste> kiko_work: i'm great, now that the buildbot has started working again
<kiko_work> flacoste, what was the issue
<kiko_work> I am working on business papers
<flacoste> kiko_work: we don't know
<Ursinha> deryck[lunch], when you return, do you intend to fix bug 394097 for the rollout?
<mup> Bug #394097: accessing message 0 of bug 371281 gives wrong return type of IMessage <api> <bug-page> <oops> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/394097>
<flacoste> kiko_work: well, one of the issue is that the time it took to checkout the branch was longer than the timeout
<kiko_work> flacoste, sounds like a bzr optim issue!
<flacoste> kiko_work: sounds like a missing repo, yeah
<kiko_work> heh
<deryck> Ursinha, no, I hadn't thought about trying to get something for that in the rollout.  Let me look into it more and see what would take to fix it.
<Ursinha> thanks deryck
<bigjools> Ursinha: those two outstanding needstest are done
<Ursinha> bigjools, ice_cream++
 * bigjools eat++
<Ursinha> thanks bigjools :)
<bigjools> Ursinha: there will be one more later when al-maisan gets something landed
<Ursinha> bigjools, right
<Ursinha> deryck, there's also bug 453203 happening quite frequently, guess because of ubuntu release and people filing more bugs through apport
<mup> Bug #453203: UnicodeDecodeError in +filebug: unexpected code byte <oops> <Launchpad Bugs:New> <apport (Ubuntu):New> <https://launchpad.net/bugs/453203>
<deryck> Ursinha, ok, will look
<Ursinha> deryck, thanks
<deryck> Ursinha, for your second one, bug 453203, is there anything for us to do?  I thought allenap was going to talk to pitti about that as it was on apport's end.
<mup> Bug #453203: UnicodeDecodeError in +filebug: unexpected code byte <oops> <Launchpad Bugs:New> <apport (Ubuntu):New> <https://launchpad.net/bugs/453203>
<deryck> maybe I'm recalling wrong though.  allenap ?
<Ursinha> deryck, well, I don't know, that's what I asked in the bug :)
<allenap> deryck: Yes, I need to talk to pitti.
<deryck> Ursinha, oh, sorry. :)  Didn't look down that far.
<Ursinha> deryck, :)
<deryck> allenap, is there anything we can do on our end, or that's what is to be learned from chat with pitti?
<allenap> deryck: To be honest, I just need to ask him if he recognises the problem. Perhaps it rings a bell, or perhaps he can help me trace where it might be happening.
<deryck> allenap, ok, cool.
<deryck> Ursinha, so given that, I don't see us getting anything on that one today.
<allenap> deryck: I'm not sure that the problem is with apport, or that he'll be able to help, but it's a next step.
<deryck> gotcha
<Ursinha> ok deryck, thanks
<Ursinha> deryck, one last question :) who's testing the last item in bugs' testplan?
<deryck> Ursinha, I thought we were green. :)  Let me look.
<deryck> oh that's me :)
 * deryck draws the lucky short straw
<deryck> Ursinha, I'll catch that here in a minute.
<Ursinha> thanks deryck!
<deryck> leonardr, ping
<leonardr> hi deryck
<deryck> leonardr, trying a little api test script on .dev server locally.  I get this:  http://pastebin.ubuntu.com/308693/
<deryck> I see a bug that this should be fixed, right?
<leonardr> deryck: you need to update lazr.restfulclient to 0.9.10
<deryck> leonardr, ok, cool.  thanks.
<deryck> leonardr, there's no 0.9.10 in download-cache.  Should there be?
<leonardr> deryck: i'm working on that now
<deryck> ah, ok.
<leonardr> i'll probably commit it on monday when i come back from vacation and pqm is ope
<thumper> rockstar, abentley, mwhudson: I'll be a little late for the standup today as I'm going to watch Jessie present her project to the class (will go on for 2 minutes).
<thumper> I shouldn't be too late, but just a heads up
<rockstar> thumper, ack.
<rockstar> thumper, it's really odd to see you around in my AM.
<thumper> :)
<rockstar> mars, what is the python in lazr-js for?
<jml> g'night all
 * rockstar lunches
<mars> rockstar, the Python scripts?  Those are our build system.
<mars> And some useful utilities.
<sinzui> EdwinGrubbs: barry: bac: I just noticed that Specification.milestone is not indexed in the DB. Did anyone investigate this as a cause for the distroseries timeouts?
 * sinzui still thinks the issue is the counts in the python code
<EdwinGrubbs> sinzui: I was working on getting some explains for that query. There is an index for (distribution, milestone), so as long as the query specifies both of those columns in the WHERE clause, it will use that index.
<sinzui> Thanks for the explanation
<sinzui> EdwinGrubbs: bac: looking carefully at the oops that bac provided, I believe only 2 milestones were iterated through before the timeout. We have a long way to go to improve performance
<barry> EdwinGrubbs, sinzui how odd: 2 milestones?  this is not reproducible on launchpad.dev
<sinzui> barry: That is a lot of processing time for about 1000 bugs!
<sinzui> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1402EB656
<barry> sinzui: it sure is
<sinzui> barry: we are iterating over the bugs about 8 times each milestone. We are doing the same for blueprints, but on a lesser scale
<sinzui> I assume a lesser scale, I do not see how many blueprints are involved.
<bac> sinzui: i setup a test locally with 1 milestone and 100 bugs and 100 blueprints for that milestone.  i saw get_status_counts called twice, once for bugs and once for blueprints.
<barry> bac: that would make some sense given productseries-milestone-table-row.pt
<sinzui> bac: yes the calls are correct, what about the times? were some of the bugs private?
<bac> sinzui: no private bugs
<bac> sinzui: but your earlier statement made it sound like that method was called more often.
<bac> sinzui, barry: might be worth setting up multiple milestones to see while throwing in some private bugs
<barry> bac: your setup script doesn't create any bugs.  that might be interesting to add
<bac> barry: the one i shared on U1 does.  can you see it?
<barry> bac: only in my browser
<barry> bac: i'll run that one
<sinzui> bac: the get_status_counts is a loop oer all items passed.
<bac> you mean "o'er"
<bac> sinzui: right, but i saw it called only once for bugs and once for blueprints.  perhaps i misunderstood your earlier comment.
<sinzui> I think so. Iterating over all the bugs to get the status.
<deryck> Ursinha-food, when you're back, just an FYI, I don't think we'll get a fix for the message 0 bug today either.  I think there's more to it that just IMessage vs. IBugMessage.
<mars> sidnei, around?
<sidnei> mars: iam
<mars> hi sidnei, just wondering if you helped with the yui-patch.js file in lazr-js?
<barry> bac: just fyi: you should add "import _pythonpath" to the top of your script
<mars> sidnei, I'd like like to know if it does anything beyond simply short-circuiting the file download process
<sidnei> mars: i didn't help. but that's all it does (you can diff against loader.js)
<bac> barry: i was just invoking with 'bin/py'
<barry> ah
<barry> the #! threw me off
<bac> barry: sorry...
<bac> bad form
<barry> bac: TypeError: makeSpecification() got an unexpected keyword argument 'owner'
<barry> that makes no sense
<bac> barry: the version U1 worked for me.  don't know.
<mars> sidnei, you are right, thanks for the info
<mwhudson> good morning
<barry> bac: i'm not doubting you but that seems impossible ;)
<bac> barry: perhaps i put up the wrong version.  tweak at will.
<barry> bac: k
<mars> sidnei, ah, do you know if that patch is used directly by jsTestDriver in some way?  I can't find any string occurances of the yui-patch.js fine in lazr-js itself.
<mars> morning mwhudson
<sidnei> mars: yes, through widgets.conf, which loads all the javascript needed for the tests.
<mars> sidnei, and is that in JSTestDriver?  Such a file is also not in lazr-js :/
<mars> wait, maybe I have the wrong branch here....
<mars> this is Bjorn's buildout branch
<sidnei> mars: right. i added it to lazr-js in my branch
<rockstar> mars, is it really necessary to have our lazr-js build system given its own setup.py
<barry> losa ping
<barry> losa unping
 * mbarnett runs in then back out very quickly
<mars> rockstar, in order to use it via buildout, yes
<rockstar> mars, sad.
<mars> rockstar, we are kind of pretending that it is a python dependency instead of using some tarball or something
<abentley> thumper: ack.
<mars> rockstar, I don't know why
<mars> rockstar, may a been the simplest path to taking advantage of the buildout versioning capabilities?
<rockstar> mars, well, it just seems to clutter things up I guess.  It's not in my way or anything.
<mars> think about it this way: if it was a requirement of launchpad-developer-dependencies, we would be layering some debian/ stuff on top
<mars> instead, we are layering the Python eggs+zc.buildout stuff on top
<mars> they both serve the same role
<mars> rockstar, after hanging out for a few years on distutils-sig, I've come to the conclusion that there is no good solution to this stuff.  Only pragmatic ones.
<rockstar> mars, yeah, but it seems odd that a javascript library would be using a python specific build system.  Maybe that's not wrong, just odd.
<mars> rockstar, kind of odd - we are not used to thinking of Python as build system glue.  Maybe it would be more normal to have a wrapper that takes the lazr-js tarball and builds that into an egg instead.
<mars> instead of putting the egg guts into the lazr-js source tree
<rockstar> mars, well, that would require work. I prefer complaining.
<rockstar> :)
<mars> lol
<mars> you know, that would be pretty clean, put everything zc.buildout and setuptools related under a pdist/ directory, just like apt packages have a debian/ directory.
<mars> would hide setup.py, setup.cfg, .egg-info...
<mwhudson> abentley: hi
<abentley> mwhudson: Hi.
<rockstar> mwhudson, did you see that the puller blew up again?
<mwhudson> abentley: am i going blind, or is BranchMergeProposalJobDerived.iterReady missing a join like BranchMergeProposalJob.branch_merge_proposal = BranchMergProposal.id ?
<mwhudson> rockstar: yeah :/
<mwhudson> abentley: i guess the distinct=True on the results means this doesn't actually give the wrong results...
<abentley> mwhudson: I think you might be right.
<abentley> mwhudson: We can't be sure that the Branch is the source_branch of the BranchMergeProposal for the job using this query, right?
<mwhudson> abentley: right
<mwhudson> abentley: i think it's a cross join (is that the word)?  you'll get every ready branchmergeproposaljob of the right type for every branch that's up to date with mirroring
<mwhudson> *every branch that's the source of a merge proposal
<mwhudson> of course the reason i'm looking at this is that in my branch is that it doesn't return a job it should be returning
<mwhudson> oh, that's easy
<abentley> mwhudson: That might explain why sometimes the merge proposal creation job dies with NotBranchError.
<mwhudson> abentley: yes it might
 * mwhudson makes a new branch
<thumper> mwhudson: if it is getting it wrong, could be rc candidate
<thumper> abentley, rockstar: one of you want to host the conf call?
<abentley> thumper: I'll do it.
<mwhudson> thumper: i think it will be getting it wrong some of the time, will write some tests after the calls
<mwhudson> -s
<mars> sidnei, since it looks like JSTestDriver wants to load everything and completely ignore the YUI loader, it may be better to bypass it entirely.
<mars> sidnei, I've checked the source: in 3.0final you can unbundle the loader, and the module dependencies should still work
 * mars heads over to #yui to ask
<sidnei> mars: interesting.
<sidnei> mars: if you figure that out let me know, i haven't merged my yui3 branch of lazr-js yet.
<mars> sidnei, when you have some time, could you try out this patch against your JSTestDriver branch?  It should remove the YUI loader
<mars> sidnei, http://pastebin.ubuntu.com/308823/
<sidnei> mars: awesome. will do.
<sidnei> mars: also need to figure out how to submit my branches to pqm, they failed on friday.
<mwhudson> abentley: is there a bug for "sometimes the merge proposal creation job dies with NotBranchError" ?
<abentley> mwhudson: Yes, and I've linked your branch to it.
<mwhudson> abentley: oh right :)
<sinzui> Chex: ping
<Chex> sinzui: hello
<sinzui> Chex: can you run the cronsripts/product-release-finder on staging ?
<sinzui> barry, did you remove then recreate the Ubuntu Haiku list?
<barry> sinzui: nope
<Chex> sinzui: sure
<barry> sinzui, flacoste, mbarnett email to ~haibunku on staging looks great.  i feel good about the mailman upgrade
<mbarnett> barry: yay
<sinzui> barry: and new team members will get an email telling them *how* to subscribe to the list. Best update ever
<barry> \o/
<jml> barry, please fix the moderation link in emails.
<wgrant> elmo: Don't worry, Soyuz isn't as insane as you seem to have assumed.
<elmo> wgrant: well
<elmo> wgrant: it's not so much an assumption but history/experience; I'd be happy to be wrong about it
<wgrant> elmo: See my comment in bug #471076
<mup> Bug #471076: filecache-default corruption should not happen <Launchpad Auto Build System:New> <https://launchpad.net/bugs/471076>
<elmo> blah
<wgrant> Heh. Yes.
<elmo> fine, fine
<elmo> well I can't hardly diagnose it now since launchpad helpfully removed the "bad" version from the cache
<wgrant> Oops.
<wgrant> elmo: Are all the buildds on >= Hardy at the moment, or are the PPC ones still Dapper?
<wgrant> Because they are all going to need a dpkg backport very soon.
<elmo> wgrant: ppc are dapper still
<elmo> is dpkg-source run outside the chroot?
<wgrant> :(
<wgrant> It is.
<wgrant> Easiest might be to move it inside, then.
<elmo> well that's freedom hating
<elmo> I'm only running dapper because hardy epically fails on that hardware
<elmo> if karmic works, I'd run that
<wgrant> I'd really like to minimise the patches required against sbuild and run a less ancient unmaintained fork...
<elmo> so would I :)
<wgrant> I've just remembered that lp-buildd's ancient sbuild needs fixes to work with >= karmic dpkg-source, anyway. So it needs patching either way.
#launchpad-dev 2009-11-04
<Chex> sinzui: I have a output log for your script run on staging, not sure if it completed properly or not.  Do you want to see the log?
<wgrant> spm: Has gina been crashing for ~ a week, or is it robust enough to withstand the un-understandable packages which have turned up in the Debian archive?
<mwhudson> that was a weird oopsy crashy thing
<wgrant> Hm?
<mwhudson> my machine more or less hung
<mwhudson> only more or less though
<mwhudson> i could type into bash
<mwhudson> but the processes i started didn't get anywhere
<wgrant> Hmm. Disk access was hanging?
<lifeless> mwhudson: 'dmesg' next time
<mwhudson> lifeless: that was one of the things that didn't do anythign
<mwhudson> and i think it recovered actually, but i was into the reboot by then
<spm> wgrant: I'm not personally aware of any failures - but have been away for a few days...
<noodles775> EdwinGrubbs: thanks for getting that QA done on staging! Just to confirm, I'm guessing you also qa'd the 'distroseries milestone timeout' there (the CRB page only has the +index page entry as qa'd). Is that right?
<noodles775> Welcome back spm :)
<EdwinGrubbs> noodles775: The previous fix applied to both bugs, and I erroneously thought that new fix did also. it is not timing out on staging, but I don't know why not.
<spm> noodles775: :-)
<mwhudson> noodles775: _surely_ you should be asleep
<wgrant> It's only 4am!
<noodles775> mwhudson: normally yes, but I'm just a little keen to see that things go smoothly ;)
<mwhudson> i see
 * mwhudson reminds himself to never volunteer to be RM
<spm> mwhudson: but you had all that experience with BB. You should be poifect for the role!
<mwhudson> spm: itym "have enough mental scar tissue"
 * spm notes that trolling when the other person can't do Punch-In-The-Nose over TCP/IP is perferable
<spm> mwhudson: same same
<wgrant> Isn't the release ~6 hours away?
<noodles775> mwhudson: speaking of which, do you know where the two NEEDSTESTING remaining items on the code team testplan.
<wgrant> I don't see any announcements of it anywhere.
<thumper> noodles775: it seems that my fix didn't
<thumper> noodles775: I need to try again to work out what the problem is
<noodles775> wgrant: yes it is, I sent the schedule to lp-dev, I'm not sure whether mrevell has an email prepared...
<wgrant> mrevell won't be up before the release.
<thumper> mwhudson: up for the call?
<noodles775> thumper: ok, if your certain, move it to BAD (?), and let me know if you plan to get it in a re-roll.
<thumper> noodles775: not entirely certain
<thumper> noodles775: I'm going to try to reproduce locally
<thumper> noodles775: but apparently the revision has been rolled out and the page still oopses
<thumper> noodles775: so in that sense it is bad
<noodles775> thumper: ok
<thumper> noodles775: wait though
<thumper> noodles775: I think it could be the staging librarian masking it
<noodles775> :) yay, ok.
<thumper> spm: how do we get stuff from the production librarian to the staging librarian?
<thumper> spm: what is copied across?
<spm> thumper: istr it's proxied - kinda sorta handwavy.
<thumper> hmm...
<thumper> http://staging.launchpadlibrarian.net/30419774/mp-unicode.msg
<thumper> spm: gets processing failed
<thumper> spm: any ideas how or why?
<spm> huh. and the real one works. looking...
<mwhudson> thumper: one sec
<noodles775> Do you guys know about abentley's entry? bug 436325?
<mup> Bug #436325: Diffstat generation chokes on binaries (and others) <oops> <Bazaar:Fix Released by abentley> <Bazaar 2.0:Fix Released by abentley> <Launchpad Bazaar Integration:Triaged by abentley> <https://launchpad.net/bugs/436325>
<noodles775> wgrant: mrevell will be around *for* the release, but I take from what you're saying that there's usually an announcement email sent to LP-users before the release?
 * noodles775 looks through the archive
<mwhudson> thumper: ok call?
<wgrant> noodles775: Normally sent to launchpad-announce, I think, so that people know that LP will be unavailable.
<noodles775> gr... I don't even know if I'll have access to send there. Thanks wgrant... checking now.
<thumper> mwhudson: yep
<wgrant> noodles775: They've classically been sent 24-36 hours before.
<wgrant> However, I hope that lists.c.c announcement lists have a not-after-1am guard enabled.
<noodles775> wgrant: yeah - I'm not sure why it's not been sent this time, but I'll update the rm-wiki notes to put a check in there. Organising now.
<wgrant> noodles775: Thanks.
<wgrant> People get unhappy enough even when it is well announced.
<noodles775> :/
<spm> thumper: haha. the staging librarian is down atm; staging rollout in progress. give it another 10-20 and try try again.
<thumper> mwhudson: OOPS-1404EB48
<thumper> https://lp-oops.canonical.com/oops.py/?oopsid=1325EA163
<thumper> spm: is the staging librarian up?
<spm> nope. something looks nicely borked...
<thumper> mwhudson: http://pastebin.ubuntu.com/309108/
<spm> thumper: alas. it is still working - the restore is gradually building eggs. perhaps abnother 45 mins wait may be appropriate :-/
<thumper> noodles775: it may be fixed
<thumper> noodles775: it certainly works locally
<noodles775> great!
<thumper> noodles775: I clicked on the link in staging which took me to edge, not staging
<thumper> and I didn't notice
<thumper> :-|
<noodles775> thumper: do you know about abentley's bug listed there too?
<thumper> noodles775: yes, it is bad but not a blocker
<spm> wgrant: ahh. in answer to your Q earlier about gina. I (you probably already know...) present bug 449408
<mup> Bug #449408: Need scriptactivity monitoring of "gina" <Soyuz:Triaged> <https://launchpad.net/bugs/449408>
<wgrant> spm: I've seen that, yes.
<wgrant> gina is a very special breed of evil.
<spm> wgrant: that means a lot; hearing you say that as in. ;-)
 * spm has visions of gina owning it's own plane of the netherhells
<wgrant> Heh.
<wgrant> It'll need a bit of hacking soon :(
<spm> by the sound of; hacking with an axe
<spm> anywas. bbs. school run time.
<wgrant> Just with some v3 support.
<MTecknology> Is postgresql any faster than mysql?
<MTecknology> random question for the channel :P
<wgrant> Is that a valid question?
<MTecknology> I don't have any basis for knowing
<thumper> https://dev.launchpad.net/Soyuz/Specs/BuilddManagerUploadDecoupling
<thumper> https://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation
<noodles775> wgrant and others: I've been chatting with flacoste and we've decided to reschedule for Thursday 5th 09.00 UTC due to the lack of announcement.
<wgrant> noodles775: Probably a good idea.
<noodles775> potentially 90mins of complete offline (no read access) is too scary - and we still don't have anyone who can send to lp-announce atm.
<wgrant> But... 5:30am. Ouch.
<wgrant> Ooh.
<wgrant> That's bad.
<wgrant> No read only?
<noodles775> wgrant: nope - librarian machines are being updated.
<wgrant> Ah.
<wgrant> That really does need an announcement.
<noodles775> yup.
<noodles775> updating identica now...
<wgrant> Thanks for working that out.
<noodles775> ... and thanks for pointing that out :)
<jtv> jamesh, got time to help with a nagging problem I've been having that probably involves storm?
<jamesh> jtv: sure
<jtv> great, thanks.
<jtv> The problem is this: we have a script that's using up too much memory.
<jamesh> have you thought of buying more memory?
<jtv> It shouldn't be pinning anything in memory across iterations of a particular loop.
<jtv> yes, we considered that but I'm Dutch.
<jtv> The only class from the Launchpad source tree that it seems to be allocating in great numbers is DBEnumVariable.
<MTecknology> How hard is it to backup a launchpad server if you run everything on a single system?
<jtv> jamesh: There are no other LP objects in memory that could explain having so many of those around.
<jtv> MTecknology: not very, just back up the database.
<MTecknology> jtv: and file system?
<jtv> MTecknology: and... sleepwalking again?  :)
<jamesh> well, you generally get one Variable for each field of each live instance Storm object instance
<wgrant> And the bzr branches, and the librarian.
<MTecknology> jtv: hm?
<jamesh> and a few get created during expression compilation, but they shouldn't last long
<jtv> MTecknology: oh, that'd be codehosting.  I wouldn't know about that part.  :/
<jtv> jamesh: this is after transaction.commit() & gc.collect()
<jamesh> jtv: what code in particular are we talking about?
<jamesh> just so I've got some idea of what's happening
<jtv> jamesh: in the LP tree, lib/lp/translations/scripts/message_sharing_migration.py
<jtv> jamesh: in there, look for _mergeTranslationMessages
 * jtv sneezes
<jtv> damn this cold weather
<jtv> jamesh: that loop looks a lot more complex than it is
<jtv> most of it is just ways to avoid pinning ORM-backed objects in memory across iterations
<jtv> in essence it's "for potmsgset in potmsgsets: for tm in potmsgset.getAllTranslationMessages(False, prefetch=False): message.shareIfPossible()"
<jtv> The rest is just a dance to make it store only objects' ids across iterations.
<jamesh> jtv: http://paste.ubuntu.com/309149/ <- I haven't tested this code, but it should give you some idea of what objects Storm knows about
<jtv> jamesh: that may not tell us much...  there's >100Ã more of these DBEnumVariable objects floating around than there are objects in Store._alive
<jtv> (In the dump I have)
<jamesh> jtv: do you know what's referencing those objects?
<jtv> If only I did...
<jtv> This is where the storm internals start to matter, I suspect.
<jamesh> gc.get_referrers() can help
<jtv> Oh, I knew about get_referents but not this one
<jamesh> I don't have any good sample code to play with for it
<jtv> no worries, I can slot this right into our debug dumps.
<jamesh> you sometimes need to use it recursively though
<jamesh> the reference chain for most instance attributes is instance -> dict -> attr
<jtv> The current dumping code is ready to go through 2 levels
<jtv> so again no worries there
<jtv> I just thought there was only get_referents and not get_referrers
<jamesh> get_referrers is a fair bit more expensive
<jamesh> since you essentially need to ask all the container objects whether they reference the one in question
<jtv> Anyway, I have in memory: 92 POTMsgSets, 65 Languages.  That's about it for the real Launchpad objects.  Then, 19K DBEnumVariables, 334 CachedPropertys, 123 DBSchemaEnumCols, 99 PublicPersonChoices, 71 ContextTitles, and change.
<jtv> The Language objects may get cached somewhere; I haven't bothered to look.  I have no idea why the POTMsgSets would stick in memory unless Storm just likes to keep them in cache across transactions.  The rest is a complete mystery.
<jamesh> how many enum columns are on each of those classes?
<jamesh> e.g. if POTMsgSet has 5 enum cols, then it would account for 460 DBEnumVariables
<jtv> jamesh: might bools be implemented in this way?
<jamesh> no
<jamesh> jtv: DBEnumVariable has an _enum attribute pointing at the associated lazr enum
<jtv> then judging by the interfaces, there's one enum in Language and one in POTMsgSet.
<jamesh> might help to see what those variables are for
<stub> https://pastebin.canonical.com/24271/ has counts for non-lp objects too.
<jamesh> might be an easier way to lay the blame than get_referrers
<stub> (scroll down to the last section for the meaningful results - should have trimmed the earlier sections)
<jtv> jamesh: as per stub's clever suggestion we're dumping referrer info for some random samples
<stub> So 108423 storm.variable.IntVariables lying around, most likely buried in tuples or dictionaries along with functions (I suspect validators)
<jamesh> so, for a storm object you have a dual ObjectInfo that Storm manages behind the scenes
<jamesh> inside that object there is a tuple of Variable objects representing the primary key
<stub> (We are actually making a list of integer ids rather than some object that can be cast to an integer, aren't we?)
<jamesh> and a mapping from from columns to variables for the other fields
<jtv> stub: good question... I'm just getting [obj.id for obj in result]
<jamesh> so IntVariable in a 1-tuple would be primary_vars
<jamesh> Variables as values in a dict are probably other columns
<jtv> stub: I pushed a new version of the branch that, for the random samples, dumps referrers instead of referents.
<jtv> maybe run again with that?
<stub> k
<stub> Gimme a package ;)
<stub> jtv: Did you tweak the filter? We don't seem to have enough launchpad database objects to account for everything so I suspect we need to turn the object filter off like I did last run
<jtv> stub: package: eog
<stub> Gah. What is the branch again btw? Lost my shell history.
<jtv> stub: can do... it's a matter of removing the "filter" argument from the dump_mem call
<jtv> lp:~jtv/launchpad/debug-mem-2
<stub> stage 1 underway (using my existing local hacks for no filter and more readable sample, not that the sample turned out to be useful)
<jtv> I commented out the filter on my end as wel
<jtv> l
<jtv> what's the local hack for a more readable sample?
<stub> jtv: https://pastebin.canonical.com/24309/
<jtv> I don't think the exception handling on get_class is needed... hasn't barfed on me yet
<jtv> has it barfed on you?
<stub> jtv: It did when I was trying to inspect contents of tuples - some bzrlib late loading stuff. Not needed now I suspect.
<stub> I was trying to categorize the tuples but it was too noisy.
 * stub goes to the fly lice sho
<jtv> stub: bong appetit
<mrevell> Morning :)
<wgrant> Morning mrevell -- want to announce the downtime?
<mrevell> wgrant, This is me back after a couple of days holiday. Roll-out downtime looks as though it's on the status page to me.
<wgrant> mrevell: So that has completely replaced launchpad-announce as the announcement mechanism? There has always been an email to launchpad-announce 24-36 hours before.
<mrevell> wgrant, I'm just checking to see what's been done and what hasn't...
<mrevell> wgrant, We're still waiting on definite confirmation of the roll-out time. I'll announce as soon as I have it.
<wgrant> mrevell: OK, great. This one is particularly important to announce well ahead of time...
<al-maisan> wgrant: http://identi.ca/launchpadstatus
<wgrant> al-maisan: I know that, but most people don't.
<al-maisan> wgrant: great.
<wgrant> And people rightly get unhappy when LP goes down for 90 minutes without an announcement.
<al-maisan> wgrant: agreed.
<jtv> jamesh: one thing we keep seeing in the random samples of live objects is _break_on_local_diverged, so presumably we've got a lot of those
<jamesh> that'd be the Reference() caches
<jamesh> If you have a reference Foo.bar using Foo.bar_id => Bar.id, Storm caches the value of Foo.bar and sets up an event handler to break that connection when Foo.bar_id changes
<jamesh> (such as happens when you commit the transaction
<jtv> So what happens to those event handles when we commit?
<jamesh> on commit, all alive objects are invalidated
<jamesh> so non-primary vars are cleared
<jtv> You've mentioned primary vars before.. what are they?
<jamesh> the columns that make up the primary key
<jamesh> just "id" for pretty much everything in Launchpad, IIRC
<jtv> Yes
<jamesh> the primary vars aren't cleared because that's what is used to tie the Python object to the database row
<jtv> jamesh: that begs two questions: why is the Python object still alive?  And why so many more of these than live ORM-backed objects?
<wgrant> With the delay, is 3.1.10 absolutely frozen, or is there a possibility of getting in a trivial fix (in code only used by sync-source.py) to prevent crashes caused by v3 source packages breaking the autosyncer?
<jamesh> jtv: well, one possibility is that you've hit a bug.
<jamesh> but I don't really have enough info to say one way or the other
<jtv> I also saw one dict that just mapped the string "remote" to a POTMsgSet.  Would that be a Storm thing as well?
<jamesh> https://bugs.edge.launchpad.net/storm/+bug/435962 was reported recently about References from primary keys.  I don't suppose that would apply to your situation?
<mup> Bug #435962: Reference can return a cached invalidated object if the local key is the primary key <Storm:Confirmed> <https://launchpad.net/bugs/435962>
<jamesh> that'd be the cache inside the Reference, yes.
<stub> jamesh, jtv: Here is the final log from the eog export: https://pastebin.canonical.com/24314/
<stub> c/export/merge-job-thingy
<jtv> stub: thanks
<jamesh> we really need better __repr__() on the various Storm classes
<jtv> stub, jamesh: note that this dump shows referrers for the random sample, not referents
<stub> Thats a lot of EventSystem instances
<jtv> about one per objectinfo, so I'd guess those are the callback hooks for key changes
<jtv> about as many "function" objects... are those generated on the fly for each object?
<jtv> "frame" is a stack frame, meaning a local variable?
<stub> I think so. If they were frames stored in tracebacks we would see object count for that.
 * stub toys with the idea of disabling the cextensions implementation of EventSystem as a shot in the dark
<stub> jtv: Got another package for me to prepare, running the first two stages?
<jamesh> I'd be a bit surprised if the EventSystem impl was introducing bugs
<jtv> stub: file-roller
<jamesh> maybe users of the API, but not the actual implementation
<stub> So would I actually (everything else would be exploding too, although this is our most extreme job in the current code base as far as object loading is concerned)
<jtv> do we have an easy way to find out how many objects were loaded overall?
<jtv> stub: so... are you trying that idea of disabling cextensions?  Shot in the dark it may be, but it hits a lot of stuff we can't see.
<stub> I'm running stage 1&2 of fileroller in preparation for whatever we do next. That might be disabling EventSystem from cextensions, better logging of the events so we can see what is holding onto them, or whatever you come up with ;)
<stub> At the moment, I'm reading email.
<jtv> stub: I for one am way out of my depth by now
<jtv> And wtf would a stack frame be referencing all this stuff?
<jtv> you mentioned other ways of referencing frames...  could something be holding a whole stack incarnation in memory?
<jtv> like maybe a cextension forgetting to unregister something even?
<jtv> jamesh: is it possible for a C extension to pin a stack frame in memory, if it forgot to drop a reference or something?
<jamesh> jtv: I wouldn't think so.  Not any more than a Python stack frame
<jtv> I mean, could it pin a Python stack frame in memory?
<jtv> So many of these objects that should live inside Storm-backed objects seem to be referenced from the stack.
<jtv> stub said something interesting earlier: when I do [obj.id for obj in query_result], do I really get ints?  Or is there any chance that I might be getting IntVariables or something?
<stub> IIRC tracebacks and exceptions can leak stack frames if not handled correctly, but I don't think we are seeing that.
<stub> obj.id should be returning an integer - it always has in the past
<jamesh> jtv: well, the only Python stack frames that stick around outside of the C call stack are generators
<jamesh> all the others are essentially tied to C function calls
 * jtv is grepping for 'yield'
<stub> comprehensions are generators too I think
<jamesh> you can have references to local variables held in cell objects too when you work with closures
<jamesh> (i.e. a nested function that uses variables from its parent scope)
<jtv> stub: less likely that we're keeping references to those around though
<jtv> so that's what those cell objects are... not getting many of those though
<stub> I saw those cell instances but have no idea what they are.
<jtv> jamesh: oh,  you're saying a cell can hold actual stack frames in memory, as I was suspecting the cextensions of doing?
<jamesh> no
<jamesh> a stack frame can hold cells that can hold locals
<jamesh> (assuming you haven't been storing tracebacks and the like in local variables ...)
<jtv> (we haven't)
<jtv> the cells don't seem to be doing much in the scheme of things
<stub> So am I correct and the current thinking is we are somehow leaking ObjectInfo instances (ObjectInfo's have a self._event==EventSystem)?
<adeuring> good morning
<jtv> hi adeuring
<adeuring> hi jtv!
<jtv> stub: that's my impression, but the frames seem to be doing it.
<jtv> looking at https://pastebin.canonical.com/24314/ random sample 3, there also seems to be an EventSystem referencing an ObjectInfo.
<jtv> Through a frame, so the EventSystem seems to be holding a frame in memory.
<stub> None of our code will deal with ObjectInfo directly.
<stub> A Store has an EventSystem, as does each ObjectInfo
<jtv> oh, wait, it's a list containing a frame and an EventSystem.
<jtv> So we have [frame, EventSystem], where frame holds a weak reference to an ObjectInfo.
<jtv> hmm... weak ref shouldn't be the problem though
<jtv> but we are seeing lots of that pattern: a list containing a frame and one other reference.
<jtv> Is that something that happens in the ObjectInfo?
<jtv> Or is it something that python does for generators or closures?
<stub> prelim done
<jtv> cool... now to figure out what to do with it
<jtv> I notice that there are 5 twisted.Failure objects
<jtv> wonder if those might be holding stack frames?  It'd be odd for them to hold frames with Storm stuff, I guess
<jtv> I have no idea how Twisted would be involved in a script like this...  does storm talk to it?
<thumper> bigjools: hi
<bigjools> hey thumper
<thumper> bigjools: is it worth us having a call?
<bigjools> thumper: what do you want to talk about?
<thumper> bigjools: have you looked at your email?
<thumper> bigjools: I sent something about 4.5 hours ago
<bigjools> I have a shitload of email!
<bigjools> and kmail is a PITA, it hangs folder views when resuming >:(
<bigjools> one sec
<bigjools> thumper: there's no chance of this working at the end of 3.1.11
<bigjools> nor 3.1.12
<thumper> why?
<bigjools> there's too much work to do, and many of the key people are sprinting in November
<jtv> jamesh: we've got 5 Failure objects hanging around Twisted... any idea where those might come from and how we get rid of them?
<thumper> I'm not saying we migrate soyus code away
<thumper> just make it possible to run jobs in a protected environment
<stub> jtv: I imagine that is just noise from all the component architecture gumph that loads up
<bigjools> you need a VM for that
<jtv> jamesh: or are they just unrelated failures to connect to services that this script isn't using?
<thumper> bigjools: and that is how the soyuz code does it?
<stub> jtv: I'm going to drop to a debugger once the object count goes about 1.5 million
<bigjools> like the PPA VMs - they get totally reset after each PPA build
<bigjools> yep
<thumper> bigjools: what is the setup and teardown time of the vms?
<jtv> stub: cool
<bigjools> thumper: thankfully *very* quick, the IS guys did some great optimisation and it takes about 5 seconds IIRC
<thumper> bigjools: do we have some to test with in staging?
<bigjools> no, just dogfood
<bigjools> we have one only
<thumper> well that's going to be a problem
<bigjools> yeah
<thumper> we have only one vm?
<bigjools> one builder
<bigjools> whether it's a VM or not is irrelevant to how we talk to it
<thumper> a builder can run multiple vms?
<bigjools> other than resetting the machine
<bigjools> the one we have is virtual I think, but yeah we can run many builder VMs on a single box IIRC
<bigjools> thumper: do you want to jump on a call with me and danilos?
<thumper> bigjools: both translations and code will need arbitrary jobs to run in protected environments
<bigjools> I know :)
<thumper> I want to work out how we are going to make this happen
<thumper> my problem is it is 22:20 and my brain is fuzzy
<bigjools> yeah
<bigjools> I have the same problem and it's only 0920
<bigjools> so basically we already have an environment to do this - the problem is that it's built around soyuz
<thumper> bigjools: what I want is a flag on the job class that says "run me in a protected env" and have the job runner just do it
<bigjools> yes
<bigjools> https://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation
 * thumper ndos
<bigjools> and I have a plan --^
<bigjools> but there are a number of stumbling blocks - but by the end of this week I will have a big list of smaller issues to tackle to get us to the point where we can get this working for non-soyuz jobs
<thumper> bigjools: mwhudson will be looking at this over 3.1.11
<bigjools> fabulous
<thumper> bigjools: danilo has also offered some resources I think
<bigjools> so I think the best thing he can work on is to refactor IBuilder
<thumper> bigjools: and I gather that foundations may also be able to provide someone
<thumper> bigjools: if you can provide some break down of the work
<thumper> bigjools: we'll take a look at it
<bigjools> I will
<thumper> ta
<bigjools> in fact I am
<thumper> fantastic even
<BjornT> thumper: could we have a call about this some time? i'd like to learn more about the current job system, and how you want to extend it.
<thumper> BjornT: sure, but not right now :)
<thumper> BjornT: abentley is going to be doing some work on it next cycle
 * thumper signs off
<BjornT> thumper: yeah, i wasn't suggesting to do it now :)
 * thumper smiles
<bigjools> BjornT: I suggested 2-3h after the TL call
<BjornT> bigjools: ok, that's probably fine for me
<noodles775> Hey stub, would you be able to go through and update: https://dev.launchpad.net/FoundationsTestPlan/3.1.10 ? There's quite a bit of QA listed there.
<jtv> stub: any luck?
<stub> noodles775: I've moved one item. The other two of mine still need QA (one I'm not sure how to test, the other I don't think I have the resources to test without taking staging down for half a day)
<stub> jtv: So every leaked ObjectInfo I look at is similar to this:
<stub> (Pdb) random.choice([o for o in gc.get_objects() if isinstance(o, ObjectInfo)])
<stub> {<storm.references.Relation object at 0x2dfe550>: {'remote': <POTMsgSet at 0xa063c50>}, 'sequence': 1, <storm.references.Relation object at 0x2dfe850>: {'remote': <POTemplate at 0x5563c50>}, <storm.references.Relation object at 0x2dfe610>: {'remote': <Language at 0x790c190>}}
<noodles775> Thanks stub.
<stub> jtv: inf.get_obj() returns None, which seems to mean it really has been lost
<jtv> stub: and yet it keeps that reference to the POTMsgSet
<jtv> jamesh: were the "remote" entries meant to be cleaned up as the objects fell out of cache maybe?
<stub> Would the POTMsgSet be caching any back references I wonder?
<jtv> stub: not that I've seen so far
<jamesh> jtv: they should get cleared out when the key for the reference gets cleared
<jamesh> they are meant to be strong reference
<jtv> stub: and these POTMsgSets were not supposed to be in cache any more, right?
<stub> Pdb) pprint(inf)
<stub> {<storm.references.Relation object at 0x2dfe550>: {'remote': <POTMsgSet at 0xd2e6550>},
<stub>  <storm.references.Relation object at 0x2dfe610>: {'remote': <Language at 0x790c190>},
<stub>  <storm.references.Relation object at 0x2dfe850>: {'remote': <POTemplate at 0x545f810>},
<stub>  'sequence': 1}
<stub> It might well still be in the cache. Why are we interested in the POTMsgSet and not in whatever is holding a reference to this ObjectInfo?
<stub> Referrers are:
<stub> <type 'tuple'>
<stub> <type 'tuple'>
<stub> <type 'builtin_function_or_method'>
<stub> <type 'builtin_function_or_method'>
<stub> <type 'tuple'>
<stub> <type 'dict'>
<stub> (one of which will include my interactive session)
<jtv> So who's holding the tuples and dict?
<jtv> (I'm making a wild guess that the builtin_function_or_method is the debugger)
<stub> (Pdb) repr(refs[0])
<stub> "({<storm.references.Relation object at 0x2dfe550>: {'remote': <POTMsgSet at 0xd2e6550>}, 'sequence': 1, <storm.references.Relation object at 0x2dfe850>: {'remote': <POTemplate at 0x545f810>}, <storm.references.Relation object at 0x2dfe610>: {'remote': <Language at 0x790c190>}},)"
<stub> (Pdb) repr(refs[1])
<stub> "({<storm.references.Relation object at 0x2dfe550>: {'remote': <POTMsgSet at 0xd2e6550>}, 'sequence': 1, <storm.references.Relation object at 0x2dfe850>: {'remote': <POTemplate at 0x545f810>}, <storm.references.Relation object at 0x2dfe610>: {'remote': <Language at 0x790c190>}},)"
<stub> (Pdb) repr(refs[2])
<stub> '<built-in method _emit_object_deleted of storm.info.ObjectInfo object at 0xd9c9b60>'
<stub> (Pdb) repr(refs[3])
<stub> '<built-in method get_obj of storm.info.ObjectInfo object at 0xd9c9b60>'
<stub> (Pdb) repr(refs[4])
<stub> "({<storm.references.Relation object at 0x2dfe550>: {'remote': <POTMsgSet at 0xd2e6550>}, 'sequence': 1, <storm.references.Relation object at 0x2dfe850>: {'remote': <POTemplate at 0x545f810>}, <storm.references.Relation object at 0x2dfe610>: {'remote': <Language at 0x790c190>}},)"
<stub> A twisty maze of tuples, all of them alike.
<stub> Gah... that time of the day when my wireless connection starts dropping out :-P
<stub> jtv: So for that POTMsgSet, there are 229 referrers
<jtv> stub: are they (or some) TranslationMessages?
<stub> All dicts, so probably lost ObjectInfos like the above
 * jtv checks how many TMs we had in memory
<jtv> ah
<jtv> so maybe these are event handlers waiting for _foreign_ keys to be changed?
<jtv> Remarkable: we're not leaking _any_ TranslationMessages.
<jtv> at this point it'd be nice to have a neat graph plotted of that maze...
<stub> Unfortunately all I have is a pdb shell and a crappy connection ;)
<jtv> stub: maybe this is what we need: http://mg.pov.lt/objgraph/
<stub> Could be good. You can try that locally.
<jtv> It's on Launchpad, interestingly enough, but that's not showing up on google.
<jtv> stub: missed your message earlier... I'll try objgraph
<noodles775> hi danilos, there's one translation bug of yours tagged as needs-testing - bug 459132, have you qa'd it, or can you pls if not? Thanks.
<mup> Bug #459132: Clean up existing untranslated credits messages <qa-needstesting> <Launchpad Translations:Fix Committed by danilo> <https://launchpad.net/bugs/459132>
<danilo_> noodles775, haven't QAd it yet, won't run the script until it's QAd though, it shouldn't block the release
<noodles775> danilo_: great, thanks for the update.
<noodles775> BjornT: when you get a chance, would you be able to either QA or mark as untestable the item at https://dev.launchpad.net/LaunchpadTestPlan/3.1.10
<deryck> Morning, all.
<mrevell> mthaddon, Hi
<mthaddon> mrevell: howdy
<BjornT> noodles775: it is OK, but i can't seem to be able to edit the wiki page
<noodles775> Thanks BjornT - I'll update it.
<noodles775> BjornT: or not - Type error - I guess the same as you.
<BjornT> noodles775: yeah
<danilo_> noodles775, hey, when's the actual release being rolled out?
<danilo_> noodles775, ah, seen the email, never mind :)
<noodles775> :)
<jtv> jamesh, stub: http://rookery.canonical.com/~jtv/objects.png
<jtv> probably a bad example because this object has legitimate business being in memory
<stub> jtv: It does? I can generate a new one then ;)
<jtv> stub: seems to be the same in your graph, yes...  I did have to copy your trick of getting a random pick though.  I think get_objects may be in-order.
<jtv> stub: see latest version of my branch, it produces the objects.png at the very end
<stub> Why do you think the objects should be there?
<stub> I'm seeing gobs and gobs of ObjectInfo's linked to a lone POTemplate in the top left corner
<stub> oh - you mean on your graph
<jtv> well, same on yours: the POTemplates are supposed to be in memory because they're referenced
<jtv> there should probably be a potmsgset in the graph somewhere
<stub> Yes - the POTemplates are not the problem. The couple of million ObjectInfo's are.
<stub> I can't see a POTMsgSet - might have been cuttoff
<stub> (the graph took a few mins to generate as it is ;) )
<jtv> you could start plotting from one... mine is smaller & faster, but maybe doing it all the way at the end allows too much leakage to be cleaned up again
<stub> The dubious bit to me is the 'changed' link in my graph. There is a set of 4k objects, one of which the ObjectInfo I'm inspecting (after following a few links)
<jtv> actually... why should there be so many ObjectInfos attached to one POTemplate?  I thought jamesh said 2 per object.
<jtv> "changed" link?
 * jtv really needs that 52" monitor now
<jtv> oh, I have it
<stub> So these objects are registered on the 'changed' event on the POTemplate I think, and for whatever reason that isn't cleared over transaction boundaries?
<jtv> or maybe they're sets of objects that storm thinks have changed?
<jtv> given that committing seems to count as changing the primary key somehow...
<stub> The only dictionary in EventSystem is the hooks to call. So the 'changed' is the key into the _event._hooks dictionary.
<jtv> ahok
<jtv> so... the event hooks aren't cleaning themselves up?
<stub> Which would indicate that these objects are not unhooking themselves when they drop out of scope I guess?
<jtv> stub: we can ask niemeyer on #storm
<noodles775> Hi salgado ! Could you please take a look at your remaining foundations NEEDSTESTING entries when you get a chance (and possibly others there ;))?
<salgado> noodles775, I can take a look, but we don't test our own items in foundations, so I'll ask matsubara to do it
<salgado> matsubara, ^.  mine have instructions
<noodles775> salgado: fine - as long as they're updated I'm happy :)
<noodles775> Thanks!
<matsubara> salgado, ok
<bigjools> wgrant: you need to get r-c for your branch as well
<bigjools> I think we can get it in the regular rollout
<wgrant> bigjools: Ah, that would be handy.
<wgrant> noodles775: Can I please have an r-c for https://code.edge.launchpad.net/~wgrant/launchpad/sync-source-v3-fix/+merge/14417?
<bigjools> and ultimately I'd love it if this code was not in the LP tree
<wgrant> NSS!
<bigjools> *cough*
<bigjools> on that note, have you looked to see if Gina will blow up with 3.0 formats?
<wgrant> It will.
<bigjools> fuck
<wgrant> Yes.
<bigjools> well-factored code FTW
<wgrant> But buildds are higher on my priority list at the moment.
<wgrant> However, I can't do anything on them until tomorrow morning, so gina-time now.
<wgrant> It shouldn't be too difficult.
<wgrant> But I wonder how spectactularly it is blowing up at the moment -- is it crashing entirely, or just failing to import some stuff?
 * wgrant tries to remember how to get this thing running.
 * bigjools -> food
<stub> jtv: Was it the wordpress product or sourcepackage?
<jtv> product
<jtv> but part of that job is done now, I think?
<stub> jtv: I never attempted wordpress
<jtv> I thought you did, early on...
<stub> jtv: I've commented out the gc.collect() and object count stuff. Just reporting memory usage.
<stub> jtv: No - we did a smaller package first and it failed.
<jtv> so did you run against the package just now, or the product?
<stub> file-roller sourcepackage
<stub> Of course, there is still a coredump lurking in there somewhere ;)
<jtv> oh yes, that too  :(
<jtv> stub: so wait, you were saying that it was fixed for... file-roller I guess, but not for wordpress, right?
<jtv> stub: or did you mean to say "*Now* for that wordpress run"?
<stub> erm... yes. typo ;)
<jtv> phew!
<wgrant> bigjools: OK, great. The fix for gina is simple, and for now it will just skip the package.
<bigjools> wgrant: awesome, thanks
<sinzui> noodles775: The registry is OK to release now
<noodles775> Thanks sinzui !
<noodles775> matsubara: how are you going with the foundations QA?
<matsubara> noodles775, haven't touched that yet. it's next in my list though
<noodles775> matsubara: great.
<noodles775> matsubara: I'll do salgado's items that have instructions now.
<matsubara> noodles775, I'm doing r9779 feel free to take the next one
<noodles775> matsubara: yep, taking 9817
<noodles775> matsubara: actually, but I think I *need* you anyway - do you have access to staging's mailbox? If so, can you confirm the email as outlined there.
<matsubara> noodles775, btw, I'm syncing my local mailbox to have the staging email for those login workflows, I'll let you know when I get your email :-)
<noodles775> taa
<matsubara> noodles775, it'll take some time since the mailbox is huge
<noodles775> matsubara: ok, so pls just update that item too (I guess it wasn't so helpful for me to try to do some!)
<matsubara> noodles775, will do. no worries :-)
<Ursinha> matsubara, is it huge even with the daily cleaning?
<matsubara> Ursinha, the problem is that my offlineimap cache hasn't been updated in a long time
<Ursinha> matsubara, maybe you should use the regular imap now that the inbox is manageable
<matsubara> Ursinha, offlineimap is still a lot faster IMO
<Ursinha> matsubara, not for sporadic access, I guess
<Ursinha> but it's a matter of taste :)
<matsubara> Ursinha, mutt + imap sucks anyway
<wgrant> Can I grab an r-c for https://code.edge.launchpad.net/~wgrant/launchpad/sync-source-v3-fix/+merge/14417?
<Ursinha> noodles775, ^
<noodles775> looking
<noodles775> Thanks wgrant - rc approved.
<wgrant> noodles775: Thanks.
<noodles775> wgrant: is someone already organised to land that?
<wgrant> noodles775: Not sure. bigjools ^^?
<bigjools> I am OTP, would you mind landing it noodles775?
<wgrant> noodles775: If you can, thanks! Happy archive admins are good.
 * wgrant sleeps.
<noodles775> wgrant, bigjools: yep, no probs. Sleep well wgrant :)
<bigjools> g'night wgrant
<barry> reviewers, lurkers, beuno and anybody else -> #launchpad-meeting in 5m
<barry> reviewers -> #launchpad-meeting
<mpt> sinzui, hi, do you have time to supply those UDS discussion estimates today?
<sinzui> mpt: no. I am doing them today
<sinzui> mpt: I can give scale, not time. I may be able to say the number of releases
<mpt> sinzui, I'm mainly concerned with making it easy for robbiew to schedule UDS at the moment :-) Estimates for the actual implementation are interesting, but not required for that.
<sinzui> Oh, time needed to discuss. I really have not idea. Maybe I will when I have this work broken into stories with technical notes.
<mpt> sinzui, "not at all" is a possible valid answer for the Launchpad-related topics, if they would be better discussed outside UDS.
<sinzui> noodles775: ping
<noodles775> sinzui: hi
<mpt> sinzui, but if there are Registry + Soyuz people attending UDS, it would make sense to have face-to-face discussions on them since they'll be higher bandwidth than teleconferences
<sinzui> noodles775: I think we need another rc https://staging.launchpad.net/bzr/+milestone/2.0.1
<sinzui> https://staging.launchpad.net/gdp/trunk/0.2
<sinzui> Bug 473738 reports issues that may relate to recent CSS changes
<mup> Bug #473738: milestone page refuses to let me see the bug status (after resizing) <Launchpad Registry:New> <https://launchpad.net/bugs/473738>
 * noodles775 looks
<sinzui> noodles775: The issue appears to be the "Download files for this release" table. the page lays out correctly if you remove the release
<sinzui> or maybe the dual combination or .portlet .full-page-width
<sinzui> noodles775: I see the issue, The DOM shows the full-page-width is nested inside another. This is a pretty simple fix.
<noodles775> sinzui: great - we've still time to land on db-devel (especially as you won't need to run the suite if it's just a css change). Let me know when it's ready. Thanks!
<noodles775> oh, although that would be a template change? still time though.
<sinzui> the milestone tests are pretty robust. we do not test markup, only content. the class is not tested
<noodles775> Great.
<noodles775> matsubara: thanks for clearing the foundations QA!
<matsubara> noodles775, np :-)
<sinzui> noodles775: Another course is to add a CSS rule for .full-page-width .full-page-width that disables the width change.
<matsubara> noodles775, about the one item in the Launchpad itself queue, I already pinged mrevell about it and basically we can't check on staging yet as it's still is one revision behind
<Chex> sinzui: did you see my message on your staging script run last nite??
<noodles775> sinzui: sounds much safer - and would actually be useful too?
<sinzui> Chex: I did I would be a nice read
<mrevell> matsubara, you pinged me about that? I don't see it.
<noodles775> matsubara: yep, I can do that later - it's easy.
<Chex> sinzui: oki-doki, hang on
<barry> abentley: what rev of lp:bzr-pipeline works with bzr 2.0.1?  i just updated my plugin and it borkeded
<sinzui> noodles775: it is safer, and it clearly defines our intent
<matsubara> mrevell, I pinged you on another channel
<noodles775> mrevell: just the blog post update...
<mrevell> noodles775, all ok with it?
<abentley> barry: The lp:bzr-pipeline/stable branch works with bzr 2.0.x
<noodles775> mrevell: yep, should be, we just need to wait for staging to update before we can QA it, that's all.
<barry> abentley: cool, thanks
<abentley> barry: np
<Chex> sinzui: chinstrap:~stasik/tmp/
<noodles775_> um, is there a bug with the display of MP's?
<noodles775_> https://code.edge.launchpad.net/~matthew.revell/launchpad/whatsnew-3110/+merge/14231
<noodles775_> The original reviewer is not displayed there, it looks like it's reviewed by jtv, but by the comment seems to be abel.
<jtv> noodles775_: I updated its status only
<jtv> noodles775_: after the original reviewer voted Approve.
<noodles775_> jtv: ah, *phew*, thanks.
<jtv> noodles775_: sorry for the confusion,  I just wanted to get it off the "reviews you should be doing RIGHT NOW, slacker!" list
<noodles775_> np... good thing to do!
<jtv> Well yes and no... technically one could argue I might not have been qualified to decide whether the Approved vote was enough to approve the whole MP
<noodles775_> yes, I should be doing that when I do the rc right?
<noodles775_> I'm still confused why abel's not listed as a reviewer - maybe he just didn't select 'Approve' when reviewing it and I didn't notice before?
<sinzui> noodles775_: I am now weeping. I can see a programming error in the product-release-finder. I am inclined to purse a CP rather than fix it for an RC.
<sinzui> noodles775_: The CSS fix for the milestone page is simple, but there is something else wrong, the sidebar is in the wrong position. There must be a markup nesting error, but I cannot see it
 * sinzui blows a gasket.
<bigjools> poor gasket
<noodles775_> sinzui: no stress - I think it's not really critical - it doesn't stop people from seeing the information as you can always scroll.
<noodles775_> sinzui: what do you think?
<sinzui> yes that is true
<sinzui> The side bar is at the bottom of the page
<noodles775_> sinzui: can the simple css fix just be landed on its own and leave the side bar issue for later? Up to you.
<sinzui> absolutely
<sinzui> noodles775_: FF hates this <div style="clear:both" /> changing the markup to use an open and close tag fixes the side bar.
<noodles775_> Great!
<sinzui> oh there were only two cases in the whole tree
<sinzui> That that cause the nesting error in the DOM, but the XML checker passed it
<sinzui> It is same the reviewers meeting is over now
<noodles775_> yeah
<danilos> Ursinha, so, launchpad-2207-00-0.sql is a base database description from one point in time (judging how stub usually names them, that one is from 2.2 LP series from July)
<Ursinha> danilos, go ahead
<danilos> Ursinha, the changes you showed me in http://paste.ubuntu.com/309660/ are correct SQL-wise, but you need to make it into an incremental patch we can apply on the live database
<danilos> Ursinha, that means using ALTER TABLEs and similar
<danilos> Ursinha, look at examples in database/schema/patch-*.sql
<Ursinha> danilos, I'm doing that, creating one with alter table, that is
<Ursinha> I'd like to know if the changes were correct before doing that
<Ursinha> danilos, right, after that what should I do?
<danilos> Ursinha, create a patch and name it something like patch-2207-95-0.sql (use a number instead of 95 that is not used by existing patches; do update LaunchpadDatabaseRevision table with the number as well)
<Ursinha> right
<danilos> Ursinha, the reason why I am saying you started the wrong way is because you should be able to play with this on launchpad_dev DB, i.e. to construct relevant fields; just do "psql launchpad_dev", and issue ALTER TABLEs there, create new indexes if needed and whatnot
<Ursinha> danilos, oh, I see
<danilos> Ursinha, patching launchpad-*.sql is not something you'd ever want to do, at least not until you get promoted to being stub :)
<Ursinha> danilos, haha sure :)
<Ursinha> danilos, not my intention here :)
<danilos> Ursinha, also, doing direct modifications in 'psql launchpad_dev' is much faster than re-running make schema a few times
<Ursinha> danilos, indeed it is, but again, I just wanted to check with you if those changes were correct :)
<Ursinha> I couldn't think of other way :)
<Ursinha> danilos, sorry :P
<Ursinha> danilos, this is cool exercise, btw
<danilos> Ursinha, I am going back to not knowing anything about it :)
<danilos> Ursinha, in a call as well now :)
<Ursinha> danilos, sorry, will disturb you later then :)
<Ursinha> matsubara, did we have any problems with summaries generator script? I see no lpnet summary for yesterday
<matsubara> Ursinha, likely. if the html/txt summary is not in the oops-summaries/ directory, then it wasn't generated. can sort it ou?
<Ursinha> matsubara, ok, I'll try to run that again
<Ursinha> matsubara, thanks
<thumper> when are we getting yui3 release code into LP?
<thumper> I spent time chasing stuff last night hitting issues by reading online docs
<thumper> only to find that our code didn't have it
<thumper> the methods I want that is
<sinzui> Chex: I think the staging reset clobbered the product-release-finder test earlier that I anticipated. Can you run the script now *if* we are not updating the staging?
<Chex> sinzui: yes sure, hang on and let me check
<rockstar> thumper, I think mars is working on it (and I hope it's done in time for the sprint)
<salgado> sidnei-away, when you get back, I'd appreciate if you could have a look at bug 474459 and see if you've got any ideas on what could've caused that
<mup> Bug #474459: Text input in the picker's footer can't be focused <LAZR Javascript Library:New> <https://launchpad.net/bugs/474459>
<mpt> sinzui, robbiew is doing the UDS scheduling this week, so if you think those LP bits would benefit from discussing at UDS at all, I suggest either today or tomorrow :-)
<sinzui> mpt: I am struggling to get this done. I will try
<mpt> thanks
<Ursinha> danilos, ping me when you're available, please :)
<danilos> Ursinha, ping
<danilos> Ursinha, what's up? :)
<Ursinha> danilos, http://paste.ubuntu.com/309797/
<Ursinha> see if this is correct, please
<danilos> Ursinha, now, that looks much better
<danilos> Ursinha, you don't want to use 9 as the number though, at least until stub assigns you one, since it's much more likely you are going to merge with latest db-devel and somebody would have already used that number
<danilos> Ursinha, so, go with something like 99 or 92 or 97 or...
<Ursinha> danilos, I see, renaming..
<danilos> Ursinha, and, the next step is to add a field description to comments.sql and you can submit that for DB review
<Ursinha> danilos, ok, I'll do this right now
<danilos> Ursinha, basically, the DB patch + comments.sql modification should be all you've got in this branch
<Ursinha> danilos, sorry taking so long, had to figure out the problem with the oops summaries
<Ursinha> danilos, right.
<Ursinha> danilos, should I file an mp or just submit my branch to lp?
<danilos> Ursinha, excuses, excuses... it's not like we are releasing tomorrow!
<Ursinha> hehe
<danilos> Ursinha, file a new bug for this branch, attach a branch to it, make an MP and ask for stub's review
<Ursinha> danilos, I've filed, will do that
<danilos> Ursinha, make an MP against lp:launchpad (not launchpad/devel), and once you get approval and we are good to land stuff, land it on db-devel
<danilos> Ursinha, also, in general, you want to start branches like these by branching off db-devel, especially if you are going to land it like that, but it shouldn't be a problem now that you are going to land it after the rollout
<Ursinha> danilos, I see that in the docs it says to name the file patch-xx-99-0.sql, so my file should be named patch-90-99-0.sql?
<Ursinha> danilos, I did that already
<danilos> Ursinha, excellent :)
<danilos> Ursinha, no
<danilos> Ursinha, xx should be 2207 in that case
<Ursinha> oh *stupid*
<Ursinha> sorry
<Ursinha> lack of coffee here
<danilos> Ursinha, 99 is the least likely number to have been reached, but anything well far off from the latest number is good :)
<danilos> Ursinha, excuses, excuses, it's not like... :P
<Ursinha> danilos, I won't use 99 because it's likely someone had the same idea :P
<danilos> Ursinha, it doesn't matter, stub will give you a new number and then you'll bzr mv to that
<danilos> Ursinha, if anyone managed to land a patch with -99, you'd better let them know :)
<danilos> Ursinha, anyway, after that, the next step is to do the interface/model changes to add translation_focus to IProduct and Product, make it settable from +changetranslators
<danilos> Ursinha, and finally, modify primary_translatable to point to the translation_focus if it's defined, similar to what DistroSeries.primary_translatable is doing
<Ursinha> danilos, please, hold on for a moment :)
<danilos> Ursinha, I ain't, I am about to split off :) take notes of above, and work on it one step at a time
<danilos> Ursinha, don't be surprised if you don't finish it all by the end of the day, but do ask others about how this is done
<danilos> Ursinha, this ain't anything others can't help with
<Ursinha> danilos, excuses, excuses... it's not like it's EOD for you already :P
<danilos> Ursinha, it's not like it is, it's well beyond it :)
<Ursinha> danilos, :)
<Ursinha> danilos, I took notes, will add the patch to the branch, comment and open and mp
<Ursinha> hopefully will have other changes to discuss tomorrow
<Ursinha> *open an mp
<Ursinha> will bug everyone else to accomplish that :)
<sinzui> salgado: Chex Do you know if there has been any action on bug 458835
<mup> Bug #458835: download counters are a few days old <Launchpad Registry:Triaged by salgado> <https://launchpad.net/bugs/458835>
<danilos> Ursinha, of course :) thanks for the effort you are putting into it :)
<sinzui> salgado: Chex: is there a job being setup? is there an RT
<danilos> Ursinha, anyway, I am off
<Ursinha> danilo-afk, have a nice evening then :)
<salgado> sinzui, yes, we now have all logs available for the script to parse but we're missing a config change to point it to the new place.  the config change has landed and should be rolled out tomorrow
<sinzui> salgado: rock. should I mark the bug fix committed?
<salgado> sinzui, yeah, I think that's be reasonable
<sinzui> salgado: If I have just and SSO account though Ubuntu, can I log into Launchpad *now* and get a profile created automatically?
<salgado> sinzui, yes
 * salgado double checks
<sinzui> salgado: there is a test I can read?
 * sinzui is uncertain because of so many bugs about the transition from account to person
<Ursinha> I wonder why is bzr telling me I cannot push my branch to lp due to uncommitted changes when bzr status shows nothing
<salgado> sinzui, tests/test_login.py in lib/c/l/
<sinzui> thanks very much
 * thumper punches javascript in the face
<sinzui> barry: EdwinGrubbs: I just sent you an annotated summary of the the UDS software discussions about the software center. I need you estimates of the time needed to discuss each item.
<barry> sinzui: k
 * rockstar -> lunch/gym
<sidnei> salgado-afk: is that with yui 3.0.0?
<sidnei> salgado-afk: i suspect it might be a zIndex issue, but would have to look closely
<sinzui> barry: yes soyuz is going to the sprint. The mis-perception is because registry *owns* a package record, is must also know about the actual source package and the built package
<thumper> beuno: ping
<sinzui> barry: I'll ask bigjools about who should attend.
<thumper> rockstar: where is the spinner icon?
<rockstar> thumper, I believe /@@/spinner
 * rockstar really should go eat lunch now.
<thumper> rockstar: I'd like a call with you when you have finished eating
<rockstar> thumper, okay.  I was also hoping to hit the gym, but maybe I'll do that later tonight.
<barry> sinzui: +1
<rockstar> thumper, ready.
<wgrant> Bug #474593 looks LOSAish.
<mup> Bug #474593: Certain URLs don't redirect to SSL <Launchpad itself:Confirmed> <https://launchpad.net/bugs/474593>
<mwhudson> wgrant: very odd
<mwhudson> mbarnett: hi, can you look at the bug wgrant just mentioned?
<wgrant> mwhudson: Rather.
<beuno> thumper, hi
<thumper> beuno: too late, found my answer
 * mwhudson finds lib/canonical/buildd for the first time
<mwhudson> omg the horror
<Chex> wgrant: mwhudson: hi, looking at that bug.
<mwhudson> Chex: cool
<wgrant> Chex: Thanks.
 * thumper looks at the file too
<wgrant> mwhudson: RUN AWAY.
<wgrant> mwhudson: NOW.
<thumper> mwhudson: OMG even the directory contents looks scary
<mwhudson> wgrant: sadly i'm not sure that's an option
<mwhudson> if we're luckly maybe rm -rf will be
<thumper> heh
<wgrant> It doesn't quite run properly on Karmic, but can be made to with a bit of patching.
<wgrant> But to get it to actually build, it needs a bit more.
<mwhudson> some of it looks very similar to code i wrote actually...
<wgrant> Just remember to ignore all of the docs in there.
<mwhudson> (ProcessMonitorProtocol, to be specific)
<mwhudson> wgrant: heh ok
<wgrant> Some of them are a little bit correct.
<wgrant> But most of them are just speculation.
<mwhudson> wgrant: can i ask some basic questions about this code?
<wgrant> mwhudson: Sure.
<mwhudson> wgrant: aiui the buildd-manager runs from cron and basically looks through the list of builds pending and builders and assigns builds to idle builders
<wgrant> mwhudson: buildd-manager is a daemon which polls everything every 5 seconds.
<mwhudson> oh ok
<wgrant> It's possible the old slave-scanner was a cron job, but that was before my time.
<mwhudson> but the second half is correct?
<wgrant> That's part of its task.
<mwhudson> yeah, i think that changed recently
<mwhudson> so one thing i don't really get is what "assigns builds to idle builders" means
<wgrant> It also watches builders, and fucks them up if they are not doing what it things they should be.
<wgrant> And takes binaries from them when they finish.
<wgrant> What don't you understand about that?
<mwhudson> well i think i know what happens
<mwhudson> but i'm not sure
<mwhudson> so there's a process on the builder that talks xml-rpc
<wgrant> Right.
<mwhudson> when it is given a job to do, it sets up a chroot/fires up a guest vm >
<wgrant> That's the beast that lurks in lib/canonical/buildd.
<mwhudson> ?
<wgrant> Ah, not quite.
<wgrant> The way the VM stuff works is nothing to do with lp-buildd.
<mwhudson> or is it already running in the chroot/vm
<wgrant> lp-buildd lives *inside* the VM.
<mwhudson> aah ok
<wgrant> Before dispatching a build to a virtual builder, buildd-manager will fire a reset trigger to the host over SSH.
<wgrant> (this part is not public, but I know vaguely how it works)
<mwhudson> so a chroot is involved even when the builder is virtualized?
<wgrant> The guest, which contains the lp-buildd, then reboots with a fresh image.
<wgrant> Yes.
<mwhudson> ok
<wgrant> To ensure a clean environment.
<mwhudson> that's a good thing to learn :)
<wgrant> So, buildd-manager then tells the builder to get all the files it needs.
<wgrant> That is, the chroot tarball and all the source files.
<mwhudson> lp-buildd is the thing that runs on the slave?
<wgrant> (the builder then grabs them from the librarian directly)
<wgrant> Right.
<mwhudson> aka the builder
<mwhudson> ok
<wgrant> Once the builder has all the files cached, buildd-manager calls build(), and the builder does its stuff.
<wgrant> buildd-manager will notice when a builder has finished from its status message.
<mwhudson> the builder runs a subprocess something very vaguely like "chroot /srv/karmic make" ?
<wgrant> The status message contains the build status (successful, failed, dependency wait, etc.), and names and hashes of all of the built files.
<wgrant> Very very vaguely, but yes.
<mwhudson> how does the builder find the files?
<wgrant> (that's the sbuild thing you see there -- it's an ancient unmaintained fork of the standard Debian buildd tool)
<mwhudson> is this where that terrible perl script (sbuild) comes in?
<wgrant> Hm. Good question.
<wgrant> I presume sbuild dumps them into a directory that it just lists.
 * wgrant checks.
<jml> barry, you still around?
<barry> jml: yep
<mwhudson> oh so the subprocess is more like "chroot <chroot> sbuild" ?
<wgrant> mwhudson: sbuild does the chrooting itself.
<jml> barry, yay
<wgrant> It runs outside it, but jumps into it to do most of the work.
<jml> barry, I uploaded a patch to reproduce bug 394133
<mup> Bug #394133: Truncated links in Launchpad mailing lists automatic messages <mailing-lists> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/394133>
<mwhudson> wgrant: ok
<wgrant> lp-buildd actually just calls sbuild-package, which then calls sbuild.
<jml> barry, but I don't know how to fix it.
<barry> jml: sec..
<wgrant> mwhudson: So, the sbuild will result in one .changes file, which references the rest of the binary files. The .changes file has a known name, so canonical.buildd.debian.DebianBuildManager.gatherResults can use that to find the rest of the files.
<wgrant> It then sticks them in the file cache, at which point they can be retrieved by the buildd-manager simply by knowing the SHA1.
<barry> jml: do you want me to take a crack at it?
<gary_poster> sinzui: I'm sorry to bother you, but I'd like your thoughts.  I'm triaging and wondering what the heck to do with https://bugs.edge.launchpad.net/launchpad-foundations/+bug/70416 .  It seems like a legitimate, important bug, and even importantly tied to our 6-month theme.  Do you know of some already-present feature or initiative that would address the problem?  If not, I think I'll bring this up on the mailing list as an im
<gary_poster> consider.
<mup> Bug #70416: Gnome menu names not linked to packages <Launchpad Foundations:New> <https://launchpad.net/bugs/70416>
<jml> barry, yes please, or at least give me some guidance on how to fix it
<jml> barry, my last comment on the bug report has a lot of details.
<mwhudson> wgrant: i guess "sbuild does the chrooting" wasn't what i wanted to hear
<mwhudson> wgrant: you know why i'm asking about this?
<lifeless> mwhudson: is this about building source packages?
<wgrant> mwhudson: BFB, I presume.
<lifeless> mwhudson: if so, I have that happening in a schroot for another project, using bzr-builder
<sinzui> gary_poster: I cannot comprehend what is being asked for
<mwhudson> wgrant, lifeless: more about running jobs where we don't trust the code running
<mwhudson> of which BFB is a particular case yes
<gary_poster> sinzui: you don't know what I am asking of you, or what the bug report describes?
<lifeless> mwhudson: schroot can do sessions
<lifeless> its very useful
<sinzui> gary_poster: Ubuntu does *not* want users to report bugs via launchpad. they should use their desktop
<wgrant> It's already secure in the VM, but it's probably still good to run it in a chroot inside that.
<wgrant> But for build cleanliness reasons.
<gary_poster> sinzui: right, so I can close this and say "you should use apport now"?
<gary_poster> that's just crash reports, so not quite the same
<lifeless> mwhudson: I've sent you a mail
<lifeless> with some helpers I put together
<wgrant> I'm not sure there's much point using schroot, since we don't need sessions and need to grab tarballs dynamically. But lifeless' work could well be useful.
<sinzui> gary_poster: Oh, yes I think it should be closed, but if you hesitate, target to launchpad bugs, because that is only launchpad app involved i this use case
<gary_poster> sinzui: ok, cool.  thank you.
<sinzui> gary_poster: I stole 180 bugs from you a few weeks ago
<sinzui> I hope you don't miss them
<gary_poster> sinzui: I know and appreciate :-)
<wgrant> mwhudson: I imagine it will be pretty similar to the current sbuild setup, except without a few thousand lines of Perl.
<wgrant> Although... hmmm. We need to manage build-deps somehow.
<wgrant> And that is messy.
<wgrant> :(
<lifeless> wgrant: schroot with root user permitted
<lifeless> or sudo permission in the chroot for a separate user to run apt-get
<lifeless> mwhudson: you'll also want to see my bugs on bzr-builder filed recently
<wgrant> lifeless: Permissions are not the problem, sadly.
<lifeless> while its not bzr-builddeb, I was solving/facing the same issues you're thinking about, I suspect.
<lifeless> wgrant: oh?
<mwhudson> lifeless: i think first it probably makes sense to focus on rosetta's needs as i think they're a bit less exciting than BDB
<wgrant> lifeless: Resolving build-deps is a difficult problem.
<mwhudson> BFB
<mwhudson> but maybe that's just because i don't really know what they are
<wgrant> Although... it's unclear at this point how bzr-bp-time dependencies will be specified.
<wgrant> data and even you are a LP admin (Foo Bar) you, that field would be read-only.
<lifeless> dpkg-checkbuilddeps
<wgrant> Um.
<wgrant> Oops
<wgrant> lifeless: That checks. Does not help too much with installing.
<wgrant> sbuild has its own magic to do this.
<barry> jml: i added some ideas as a comment.  i can try them fairly quickly on your branch...
<jml> barry, cool. thanks.
<lifeless> wgrant: its trivial - you run dpkg-checkbuilddeps, apt-get install the resulting list
<wgrant> Hmm.
<barry> jml: i don't see a linked branch.  are you sure you pushed it?
<jml> barry, harumph....
<lifeless> wgrant: its not as simple as apt-get install `dpkg-checkbuilddeps`, but its not much more, really.
<lifeless> wgrant: you use a chroot session for efficiency
<jml> barry, I pushed it, but apparently the branch linking UI failed me
<barry> ;)
<lifeless> and thats why I talked about permissions ;)
<wgrant> Is there a spec for the Rosetta stuff?
<jml> barry, lp:~jml/launchpad/ml-links-bug-394133
<mwhudson> wgrant: i have the very vague notion that the initial plan is "if you have funny bzr-lp-build time deps your build will fail"
<barry> thx
<mwhudson> james_w would be a better person to ask that though
<wgrant> mwhudson: That seems unlikely.
<wgrant> mwhudson: But would make the initial implementation much much easier.
<james_w> if you have funny deps for the construction of a source package then your package is buggy
<james_w> it's a requirement to specify then in Build-Depends
<james_w> so installing those would work
<mwhudson> ah ok
<wgrant> Ah, so that's how it's working.
<wgrant> I see.
 * wgrant cries.
<jml> there there
<wgrant> So.
<thumper> rockstar: it seems that the pretty overlay doesn't have an easy way to replace the content
<wgrant> The slave should be pretty simple:
<wgrant> 1) Unpack chroot (with existing code)
<wgrant> 2) Mount chroot (also existing)
<wgrant> 3) Check out branch
<wgrant> 4) pbuilder-satisfydepends (or equivalent)
<wgrant> 5) bzr-buildpackage
<wgrant> 6) Find changes file
<wgrant> 7) Done
<wgrant> Most of the code can be extracted from the existing Debian slave.
<thumper> rockstar: or perhaps I'm looking in the wrong place
<wgrant> Except for like three commands.
<mwhudson> wgrant: the code for steps 1-2 is in perl though, right?
<mwhudson> it doesn't sound too scary though really
<wgrant> mwhudson: No. Those two bits are lp-buildd-specific shell scripts (mount-chroot and unpack-chroot)
<mwhudson> ah ok
<mwhudson> shell, way better than perl!
 * mwhudson coughs
<wgrant> They're also only 80 lines in total, not 4000
<mwhudson> wgrant: i don't suppose you know this for sure, but i guess the lp-buildd inside the vm is started by an @reboot crontab entry or similar?
<wgrant> mwhudson: The package has an init script.
<wgrant> I don't know if that's used in production, but I presume so.
<mwhudson> oh right
 * wgrant disappears for a while.
<mwhudson> wgrant: thanks for the input
<rockstar> thumper, yea, you may have to create it.
<thumper> rockstar: working prototype lp:~thumper/launchpad/popup-diff
<thumper> rockstar: very ugly right now
<rockstar> thumper, javascript is pretty ugly.  :)
<thumper> rockstar: but it has colours, dynamic loading on demand, and only load once
<thumper> rockstar: what I want is to have nice scrolling when diff appears, and limiting the visible size of the overlay
<thumper> rockstar: ideas?
<thumper> rockstar: although, I'll have to wait, I'm afk for a few hours
<rockstar> thumper, okay.
<wgrant> mwhudson: Anything else you want to know?
<mwhudson> wgrant: not right now thanks
#launchpad-dev 2009-11-05
<mwhudson> wgrant: if you could read my email and tell me which stuff i need to pay attention to, that would be great :/
<wgrant> mwhudson: Which email?
<wgrant> Oh.
<mwhudson> wgrant: my inbox :)
<mwhudson> nm
<wgrant> Right. Just worked that out.
<thumper> rockstar: I'm back if you are around
<Ursinha> hey wgrant, can you try to give me a little help here? :)
<wgrant> Ursinha: Sure.
<Ursinha> wgrant, I've added a field to the Product table, where on earth should I declare it to access it in translations side?
<wgrant> Ursinha: Are you getting ForbiddenAttribute exceptions?
<Ursinha> wgrant, yes sir
 * thumper away again
<wgrant> Ursinha: It depends on how the particular object is set up. In some cases there will be interfaces like IProductPublic, IProductEdit, IProductAdmin which have attributes protected by different permissions. In other cases, lib/lp/SOMETHING/configure.zcml will name a permission for each attribute.
<wgrant> Let's see which case IProduct is...
<wgrant> Ursinha: OK, so if you look at lib/lp/registry/configure.zcml around line 1040, you'll see security stuff.
<Ursinha> let me see
<wgrant> You'll see <require permission="launchpad.Foo" interface="IBar" />. That restricts access to the attributes and methods on Bar to those users holding the launchpad.Foo permission.
<wgrant> <require permission="launchpad.Foo" set_attributes="foo bar" /> restricts setting of attributes 'foo' and 'bar' to launchpad.Foo.
<Ursinha> argh, right, sorry wgrant, got lost here
<wgrant> Still lost?
<Ursinha> about other bit
<wgrant> Ah.
<Ursinha> wgrant, thank you very much.
<Ursinha> again.
<wgrant> np
<noodles775> sinzui: did you get that QA done?
<noodles775> I'm checking staging currently but the css issue still seems to be there :/
<noodles775> Ah, staging is on r8640... np.
<mwhudson> aargh soyuz is eating my brain
<lifeless> nom nom nom
<wgrant> mwhudson: Which bit?
<wgrant> (it is very effective at that)
<mwhudson> wgrant: the bit you told me to run away from
<wgrant> Ah, yes.
<wgrant> Got it to run or do anything yet?
<mwhudson> nah, haven't even tried that
<mwhudson> wgrant: there seems to be a "build style" concept, of which the only actual instance is debian style?
<wgrant> mwhudson: Correct.
<wgrant> mwhudson: You can see it registered in daemons/buildd-slave.tac
<wgrant> (that bit took a while to find)
<mwhudson> wgrant: right
<mwhudson> wgrant: soyuz builders get the sourcepackage bits that need to be built from the librarian right?
<wgrant> mwhudson: Sometimes.
<wgrant> mwhudson: Unless the files are private.
<mwhudson> oh right
<wgrant> In that case, they live in the restricted librarian and cannot be retrieved.
<mwhudson> so how does it work then, do you know?
<wgrant> buildd-manager instead gives them a private-ppa.launchpad.net URL with credentials, IIRC.
<mwhudson> oh ok
<mwhudson> wgrant: the chroots are in the librarian too?
<wgrant> Yup.
<wgrant> mwhudson: You can probably reuse the same chroots, and just install bzr-buildpackage inside them during the build.
<mwhudson> wgrant: yeah, not having to open that bear pit would be nice
<wgrant> It will make $UOSA happy.
<spm> wgrant: unlikely. he's a grumpy old bugger.
<wgrant> :(
<wgrant> I must try to catch him tomorrow morning, since I failed today.
<mwhudson> spm: s/happy/less grumpy/
<spm> tbh, I don't think I've ever seen lamont unhappy. irritated yes, unhappy, no.
<wgrant> I'm sure that maintaining an extra set of buildd chroots would tip him just over into unhappy :P
<mwhudson> spm: the baseline for admins is not especially happy after all
<spm> mwhudson: gah!!!! do you mean to imply that I'm -EHAPPY ????
<spm> wgrant: if you present them as hpia mk2's you could swing it with him being non the wiser? :-P
<wgrant> Ooh, true.
<wgrant> mwhudson: Deciphered the beast yet?
<mwhudson> wgrant: well, making progress
<mwhudson> wgrant: is there a difference between the slave scanner and the buildd master, or are they different names for the same thing?
<wgrant> mwhudson: slave-scanner was scaling badly, so was reincarnated as buildd-manager. Both are builddmasters.
<wgrant> slave-scanner is, I believe, only of historical value.
<mwhudson> wgrant: ahh ok
<mwhudson> wgrant: and aiui build scheduling is done be yet another process that manipulates the buildqueue table which is then consulted by builddmanager?
<wgrant> mwhudson: mrrrrrrh sort of but not really
<wgrant> mwhudson: There is this queue-builder thing.
<wgrant> It is run occasionally.
<wgrant> It only really needs to be run when you add a new arch or distroseries.
<wgrant> It creates new builds, and creates buildqueues for those that are missing.
<wgrant> But in the normal case, the Builds and BuildQueues are created and given a score at upload-time.
<mwhudson> does it need to be run when a build is rescored for example?
<wgrant> No.
<wgrant> Rescoring directly updates the BuildQueue, which findBuildCandidate then sees.
<mwhudson> or not, because the score is the output of the at-upload computation and rescoring overrides the result
<wgrant> Right.
<wgrant> queue-builder has a rescoring mode, where it will recalculate all scores.
<wgrant> But I don't think queue-builder is run regularly in production at the moment.
<wgrant> are bug subscriber sprites (not those in the "Also notified" section, though) slightly raised for anybody else)?
* spm changed the topic of #launchpad-dev to: LP down 0900 UTC - 1030 | This is Launchpad Development Channel | Week 4 of 3.1.10 | PQM is release-critical only - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<mwhudson> wgrant: sorry, i'll stop this soon
<mwhudson> wgrant: does the buildd-manager suck the built files off the slaves too?
<wgrant> mwhudson: Hey, it's more interesting than Java revision..
<wgrant> It does.
<wgrant> There's a handler in BuildQueue to do that, I think.
<wgrant> (the slave will report its state as WAITING when it is complete)
<wgrant> Hm, no, not in buildqueue.
<wgrant> But it's somewhere, anyway.
<wgrant> buildd-manager will grab the files, tell the slave to clean up, and then process the binary upload.
<mwhudson> wgrant: lp.buildmaster.buildergroup it seems
<wgrant> Something like that.
<mwhudson>         # XXX cprov 2007-07-11 bug=129487: untested code path.
<mwhudson> yay!
<wgrant> This is Soyuz.
<wgrant> There are altogether too many XXXs in that method.
<wgrant> But bigjools wants to offload most of the processing, anyway.
<adeuring> good morning
<wgrant> noodles775: What's wrong with the librarian? Disk upgrades?
<bigjools> mo disk IIRC
<wgrant> Makes sense.
<wgrant> bigjools: I wonder how much librarian disk would be saved if daily PPAs were configured to cull binaries after just a few days.
<bigjools> lots!
<bigjools> I might chop the current 28 day limit to much less
<bigjools> across the board
<bigjools> however the biggest culrprit by far is Ubuntu
<mrevell> Morning
<bigjools> eyup mrevell
<SteveA> 10:11 < SteveA> the launchpad offline page is really really annoying
<SteveA> 10:11 < SteveA> instead of serving a page up at the URL I want, but serving the offline page with a 5xx error
<SteveA> 10:12 < SteveA> it redirects me to an "offline" page
<SteveA> 10:12 < SteveA> the problem with this is, I lose any URLs I have in my browser when I refresh the URLs
<SteveA> 10:12 < SteveA> such as when I restart my firefox session
<SteveA> 10:12 < SteveA> this is a significant loss of state.
<SteveA> 10:13 < SteveA> the offline page is even a 200 OK
<SteveA> 10:13 < SteveA> it should be a 5xx
<stub> I guess if haproxy doesn't support what we want, we can open a bug and maybe submit a patch.
* mthaddon changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.10 | PQM is release-critical only - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<deryck> Morning, all.
<bigjools> james_w: fancy a trip to NZ in january?
<james_w> I wouldn't say no :-)
<bigjools> james_w: we're holding the build from branch sprint in Wellington
<james_w> nice
<bigjools> so it's up to you, it will be a mostly coding sprint I think
<bigjools> LCA is the week after, which is why we're doing it there
<james_w> thought so
<bigjools> jtv: around?
<jtv> bigjools: around
<jtv> bigjools: what's up?
<bigjools> jtv: howdy - I was wondering if you'd like to get involved in documenting how our build farm works, so you can learn a bit about it in preparation for the work we're doing to support translations jobs
<jtv> bigjools: sounds like a good idea
<jtv> bigjools: got any concrete plans for it?
<bigjools> al-maisan and noodles775 are looking into it as well, can you join up with them
<wgrant> mwhudson had fun with lp-buildd today, too!
<bigjools> I hear :)
<jtv> al-maisan, noodles775: how are you going about it?  (Or rather, how should I :-)
<noodles775> jtv: I haven't yet (still following the release issues), but plan to install soyuz locally using wgrant's instructions (I've never had time to do it yet)
<wgrant> noodles775: I rewrote them to be more useful this evening, but haven't uploaded them anywhere yet.
<al-maisan> jtv: I am also a bit busy but thought of reading the code and playing with the Soyuz dogfood system.
<jtv> noodles775: oh, I hadn't even realized that there was an additional installation procedure
<noodles775> wgrant: ah, please let me know when you do...
<bigjools> jtv: I'll include you on an email I am sending about the db schema
<noodles775> jtv: for actually running the build infrastructure locally.
<bigjools> jtv: you need to get a local buildd running
<jtv> wgrant: as you've guessed by now, please add me to the distribution list.  :-)
<jtv> bigjools, noodles775: were you deliberately completing each other's sentences there, or should I read you as two separate input channels?
<bigjools> heh, separate :)
<bigjools> prophecy - Guns n Roses "Back off Bitch" just started on my random playlist
<wgrant> noodles775, jtv: http://williamgrant.id.au/f/1/2009/running-soyuz.html is the new version.
<jtv> wgrant: thanks!
<bigjools> nice
<bigjools> wgrant: it would be good if you can put that on dev.launchpad.net
<wgrant> bigjools: I've been planning to.
<bigjools> coolio
<wgrant> Any hints as to where it should go?
<bigjools> somwhere under https://dev.launchpad.net/Soyuz
 * wgrant hunts out the Codehosting equivalent.
<noodles775> Thanks wgrant
<wgrant> https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally
<bigjools> \o/
<noodles775> Nice :)
<bigjools> thanks muchly
 * wgrant works out how to attach those two helper scripts.
<wgrant> Alright, that looks reasonable.
<danilos> wgrant, btw, I think it'd be great to have sampledata on launchpad_dev updated to reflect what's there after update-ubuntu-sane.py script
<wgrant> danilos: Probably, now that it's separate from ftest.
<danilos> wgrant, exactly :)
<wgrant> Amusingly, most of the tests use the 'ubuntutest' distro anyway, because the 'ubuntu' distro is too screwed.
<danilos> wgrant, yeah, we're also trying to use more and more of in-place object creation instead of relying on sampledata
<bigjools> but the jury's out on whether that's quicker than resetting the db each test
<wgrant> I should also probably get the archive.launchpad.dev stuff added to rf-setup, like I did with ppa.launchpad.dev a couple of months back.
<wgrant> bigjools: The external archive dependencies feature is very handy.
<bigjools> wgrant: that's its problem :)
<wgrant> I never like to see 'NIGHTMARE.' in specs.
<maxb> ooi, when does the db-stable->devel merge take place? Is it scheduled, or somewhat adhoc?
 * wgrant sadly compares the Codehosting and Soyuz local instance pages.
<bigjools> wgrant: NIGHTMARE is honesty :/
<wgrant> bigjools: Yep.
<al-maisan> wgrant: which ones do you like better?
<al-maisan> ;)
<EdwinGrubbs> bigjools: ping
<salgado> sidnei, around?
<bigjools> EdwinGrubbs: hey, wassup
<EdwinGrubbs> bigjools: you mentioned in bug 196322 that the +allpackages could be removed. Will this prevent google from crawling the source packages pages, and is that a problem?
<mup> Bug #196322: "All source packages" page empty when distribution doesn't use Launchpad for packaging <post-3-ui-cleanup> <trivial> <ui> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/196322>
<bigjools> let me check
<bigjools> EdwinGrubbs: blow it away
<Ursinha> matsubara, meeting in 15 mins?
<matsubara> stub, Chex, gary_poster, rockstar, bigjools, danilos, sinzui, allenap: LP production meeting in 14 min @ #launchpad-meeting
<danilos> matsubara, thanks for the reminder, I might not be a good participant (along with bigjools, gary_poster and sinzui) since we have a TL call at the same time
<bigjools> damn DST
<Ursinha> matsubara, oops summaries won't be completed in time as well
<bigjools> noodles775: can you attend the production meeting instead of me please!
<matsubara> danilos, I can cover for gary_poster
<danilos> bigjools, that's why I had complaints about moving the call last week, but forgot about the timing for this week :)
<gary_poster> matsubara: thank you.  what do you want to do about this longer term?  Presumably we hould move one or the other
<gary_poster> I was wondering if the team lead meeting on Thursday should be moved to the same time as Wednesday during DST
<matsubara> gary_poster, not sure what do about it in the longer term. we can try to re-schedule or send other people as QA contacts for each of the teams when the TL is not available
<gary_poster> matsubara: ok.  I'm sure we'll bring up the conflict on the TL call; I or someone else will then check up with you and Ursinha
<sidnei> salgado: sorta
<matsubara> gary_poster, all right. thanks
<salgado> sidnei, hi there. I was wondering if you could help me with that lazr-js bug I mentioned yesterday.  would you have some time for that today?
<sidnei> salgado: today is looking nasty, but if you have a reproducible case, let's schedule it for 4pm BRT
<salgado> sidnei, I do have one, yes.  thanks a lot!
<sidnei> salgado: deal. thanks!
<beuno> mrevell, I just realized I never replied to your tour email  (it dropped off my unread emails)
<mrevell> hey beuno, there's no rush. I had a meeting with Iain Farrell yesterday, as I happened to be at Millbank for some other meetings. I'll CC you on the writeup of what we spoke about.
<beuno> mrevell, thenks
<flacoste> abentley: around?
<flacoste> rockstar: around?
<abentley> flacoste: Yes.
<flacoste> abentley: we have a problem with codehosting can you help?
<abentley> If it's the fact that http access doesn't work, I'm already looking into that.
<flacoste> ok, what is the situation?
<flacoste> do you need any help?
<abentley> I don't know what's causing it.  It's hard to reproduce locally, but I'm getting there.
<flacoste> abentley: what info do we have on the issue?
<abentley> The best info I have is that some files which should be there are 404ing and others are not.
<flacoste> abentley: btw, mthaddon was looking for you on #launchpad-code
<abentley> flacoste, mthaddon: I lost my connection to irc.canonica.com, and it doesn't want to reconnect.
<mthaddon> abentley: hmm, weird - well we can work here if you like
<abentley> mthaddon: jam is reporting that not all files are 404ing, but I'm getting "connection timed out" when I try to access bazaar.launchpad.net.
<abentley> mthaddon: This URL is reported to not 404: http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/.bzr/branch-format
<mthaddon> yeah, I don't get a 404 for that one
<abentley> mthaddon: Is that possibly a caching issue?
<mthaddon> and http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/changes works for me too
<mthaddon> I can't see any caching options in the apache configs
<abentley> mthaddon: I guess it can always be a caching issue, with the prevalence of intercepting proxies.
<abentley> mthaddon: But I tried it on devpad and it worked there too.
<mthaddon> https://bazaar.launchpad.net/~launchpad-pqm/launchpad/production-devel/changes works for me too
<mthaddon> so who is reporting problems, and what kind of problems?
<abentley> mthaddon: Sure, but AIUI, we're worried about http codehosting, not codebrowse.
<abentley> mthaddon: jam is one of the people noticing problems.
<mthaddon> ok, so what specific bit is failing?
<mars> BjornT, ping re: lazr-js and buildout
<BjornT> hi mars
<mars> hi BjornT.  Did your lazr-js and LP buildout changes land successfully?
<BjornT> mars: yes, it landed quite a while ago
<mars> BjornT, ok, so is it safe to bump the version number on the lazr-js trunk, and start landing sidnei's branches?
<BjornT> mars: yes
<sidnei> \o/
<mars> BjornT, awesome, thank you
<sidnei> now, how do i land them?
<mars> ...
<sidnei> bzr pqm-submit?
<mars> sidnei, yes, that should work
<mars> it Works For Me
<sidnei> it didn't last time. maybe i have to be added to some acl somewhere
<mars> BjornT, ^ ?
<BjornT> sidnei: can you try again? or show me which error you got?
 * mars wonders who is the scrollkeeper for the PQM system, the repository of such arcane knowledge?
<sidnei> the error i got was []
<sidnei> Command failed!
<sidnei> All lines of log output:"Branch ['/home/pqm/archives/rocketfuel/lazr-js/toolchain', 'bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/lazr-js/toolchain'] not found in config"
<abentley> flacoste: We are seeing a gradual recovery.  Do you have any examples of branches that are still failing?
<BjornT> sidnei: to which pqm address did you send the request to?
<flacoste> abentley: not really, mthaddon: do you have any example HTTP branches still failing?
<sidnei> BjornT: launchpad@pqm.canonical.com apparently
<sidnei> BjornT: rockstar pasted me the config
<BjornT> sidnei: i would try again
<mars> BjornT, how much longer are you on for?
<cody-somerville> Am I reading correctly the codehosting issue is slowly recovering? My commit from 2 hours ago is still not available via launchpad yet.
<BjornT> mars: no idea. i'll probably be on and off the whole evening, and night since i need to prepare for my trip tomorrow
<mars> ok
<rockstar> Ursinha, sorry I missed the meeting.  I forgot my timezone changed.
<mars> rockstar, btw, we are working on YUI3 for LP, devmode is good, but the production JS build is broken.  If I can fix that before the sprint, then we can land it.
<mars> but we do have an integration branch for people to use.
<mars> thumper could try that branch if he is feeling adventurous :)
<BjornT> abentley: http://bazaar.launchpad.net/~launchpad-pqm/lazr-js/toolchain/
<Ursinha> rockstar, okay
<rockstar> mars, let me know how I can help.
<sidnei> uhm, my submission didn't even show up in pqm, no reply even. maybe it's just a delay
<rockstar> sidnei, there's funkiness in the web display recently.
<abentley> BjornT: Are you able to view bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/lazr-js/toolchain/ ?
<Ursinha> rockstar, it was asked someone from code to comment on https://bugs.edge.launchpad.net/launchpad-code/+bug/475394
<mup> Bug #475394: bzr http access broken after 3.1.10 rollout <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/475394>
<Ursinha> if you or abentley could do that it would be nice
<Ursinha> thanks
<rockstar> Ursinha, I'm trying to find out what's going on by reading the backchat.
<mars> abentley, https://code.edge.launchpad.net/~launchpad-pqm/lazr-js/toolchain works for me
<abentley> Ursinha: We don't know what's wrong.
<sinzui> barry: EdwinGrubbs: bac: can you close your 3.1.10 registry bugs?
<rockstar> abentley, shall we have a call to catch me up?
<abentley> rockstar: Okay
<barry> sinzui: i just ran close-my-bugs :)
<EdwinGrubbs> sinzui: sure
<sinzui> fab
<sinzui> Maybe I will find and fix the insane karma action for that action so people can safely close other people's bugs
<sinzui> barry: Do you think the rule is: use the assignee or the user doing the action?
<barry> sinzui: i think the person who sets fix committed should get the karma
<barry> fscking rich-roots kills me again!
<sinzui> I disagree. I find lots of bugs that you have indirectly fixed a long time ago. I assign it to you and mark it fix committed. you should get the credit
<barry> sinzui: oic.  assignee then
<abentley> rockstar: ringing...
<sinzui> barry: I think I gave you and edwin credit for about 30 bugs I discovered in foundations last month.
<sidnei> uhm, now it shows up, but it's out of order. how can i remove a script from pqm?
<barry> sinzui: awesome. i can go out to lunch now!  have you seen my new site www.karmaforgold.com?
<sinzui> \o/
<mars> losas, does someone have a moment to help sidnei with his PQM submission?
<BjornT> abentley: yes, the bzr+ssh url works, but not http
<abentley> BjornT: You don't have write access to that branch, do you?
<Ursinha> bigjools, can I mark bugs 470150, 470411, 472326, 472608, 472929 as Fix Released?
<Ursinha> hmm
<bigjools> Ursinha: some URLs would be nice :)
<Ursinha> bug 470150, bug 470411, bug 472326, bug 472608, bug 472929
<mup> Bug #470150: package set based archive permission checks need to consider the distro series <current-rollout-blocker> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/470150>
<mup> Bug #470411: please change the default source suite for Debian to 'testing' in _syncorigins.py <current-rollout-blocker> <trivial> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/470411>
<mup> Bug #472326: When adding and removing package subsets the distro series needs to be observed <current-rollout-blocker> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/472326>
<mup> Bug #472608: IPackagesetSet.getByName() is missing LP API metadata for the newly added 'distroseries' param. <current-rollout-blocker> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/472608>
<mup> Bug #472929: Archive +upload URLs for package sets need to be distro series aware <current-rollout-blocker> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/472929>
<Ursinha> :)
<bigjools> lol
<bigjools> Ursinha: well if you want to steal al-maisan's karma
<al-maisan> yeah, all of these were rolled out.
<Ursinha> bigjools, hm, I thought the assignee gets the karma
<bigjools> Ursinha: I think the person who marks it does
<Ursinha> bigjools, this kinda sucks
<al-maisan> yes indeed.
<bigjools> Ursinha: patches accepted I'm sure ;)
<Ursinha> bigjools, I'm learning, hopefully will be able to do that soon :P
<bigjools> \o/
 * bigjools closes his bugs
<bigjools> yay for the API
<rockstar> noodles775, are we having a second rollout?
<Ursinha> bigjools, so I won't close al-maisan's bugs so I won't steal his karma
<bigjools> rockstar: noodles775 has left for the day, he was up at 3am his time
<al-maisan> I can close them later if that suits
<bigjools> al-maisan: I have a script that will close your own bugs in 10 seconds
<mars> bigjools, I don't remember that part being in the Release Manager job description...
<al-maisan> bigjools: that's very handy.
<bigjools> mars: yeah it kinda sucks but he's a diligent man
 * beuno-lunch hugs barry for filing the bzr bug
<BjornT> abentley: no, i don't think i have write access to that branch. however, branches that i have write access to fail as well, i tink
<BjornT> abentley: for example http://bazaar.launchpad.net/~bjornt/garmin-sync/devel
<abentley> BjornT: Yes, it's just that as a debugging tool, we want to make sure the mirrored copy is alright, and the you see the mirrored copy over bzr+ssh if you don't have write access.
<Ursinha> gary_poster, are you working on bug 475550 for the reroll?
<mup> Bug #475550: _pythonpath whitelist of clean_modules is too fragile <Launchpad Foundations:In Progress by gary> <https://launchpad.net/bugs/475550>
<gary_poster> Ursinha: yes
<Ursinha> gary_poster, thanks
<gary_poster> Ursinha: np
<sidnei> BjornT, mars: pqm finally failed my branch, with the same error as before. i don't think it's because of release-critical though.
<Ursinha> sinzui, I'm sorry, I think I forgot asking in the meeting, you said bug 475433 was a good candidate for CP. you meant CP or reroll?
<mup> Bug #475433: oops leaving milestone code_name empty <oops> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/475433>
<sinzui> I meant CP, but now that I see the issue is lower, I am working on an RC
<sinzui> Ursinha: we may see many oops that end with:
<sinzui> AttributeError: 'NoneType' object has no attribute 'str
<sinzui> Ursinha: This may be happening with an not-required StrippedTextField
<sinzui> s/an/any/
<mars> sidnei, it looks like dealing with that will have to wait until after they get codehosting back to full speed :(
<sidnei> mars: thats fine. i assume lazr-js will be taken out of pqm as well?
<mars> sidnei, that I do not know
<barry> beuno-lunch: ;)
<sidnei> rockstar: are we getting your magic migrate-my-branch button this cycle or the next?
 * Ursinha stabs oops-tools
<rockstar> sidnei, probably the beginning of this next cycle, on edge.  The infrastructure got rolled out with the rollout, but the UI didnt'.
<sidnei> rockstar: awesome, thanks!
<BjornT> sidnei: you'll have to ask a losa about that error. we will only take lazr-js out of pqm if we set up a buildbot for it. but it might be better to run the tests before commit, rather than after, assuming it won't take long to run them.
<Ursinha> anyone else able to file bugs in lp?
<Ursinha> hi deryck, a user filed bug 475498 and I'm also getting consistently timeouts trying to do what he was doing
<Ursinha> deryck, I initially thought about another bug on dupefinder with long sentences but even a few words trigger the timeout
<deryck> Ursinha, ok, I'll take a look at it.
<Ursinha> thanks deryck
<mrevell> Night all
<sidnei> salgado: just got out of a meeting, and have to leave in 20, can you email me instead?
<salgado> sidnei, sure, I'll do that
<sidnei> salgado: thanks!
<salgado> sidnei, thank you
<sinzui> barry: close this bug I rediscovered and get the karma: https://bugs.edge.launchpad.net/launchpad-registry/+bug/102273
<mup> Bug #102273: product search shows description instead of summary <Launchpad Registry:Fix Committed by barry> <https://launchpad.net/bugs/102273>
<barry> yay karma!
<sinzui> EdwinGrubbs: I wrongly closed the bug I was looking for: bug 263223 relates to the layout you were looking at
<mup> Bug #263223: "Working on" and "Latest Memberships" inconsistencies on person page <trivial> <ui> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/263223>
<beuno> sinzui, EdwinGrubbs, FYI, +1 on all of sinzui's comments on the UI pre-review
<beuno> I have nothing to add  :)
<EdwinGrubbs> thanks
<EdwinGrubbs> sinzui: I wasn't sure if you wanted me to put the badges in a table like person-portlet-contributions.pt. Alternatively, I could just float the badges to the right.
<EdwinGrubbs> of course, that might be bad if the bug title should wrap.
<sinzui> EdwinGrubbs: I am not sure. I do not like the table, but badges are shown in tables on all pages except this one instance
<sinzui> EdwinGrubbs: I tried floating on the milestone page and later switched to a table column because users noticed that they were still displaying differently
<sinzui> EdwinGrubbs: In this case, the user will see two presentation and they should be the same. so if you do not want to alter the erson-portlet-contributions.pt, I would use a table just like that portlet
<EdwinGrubbs> ok
<tsimpson> is there a decided time scale on https://bugs.edge.launchpad.net/launchpad-foundations/+bug/385517 ("launchpadlib users made to authenticate unnecessarily")?
<mup> Bug #385517: launchpadlib users made to authenticate unnecessarily <Launchpad Foundations:In Progress by leonardr> <https://launchpad.net/bugs/385517>
<mwhudson> airport time
#launchpad-dev 2009-11-06
<noodles775> Hi jtv - do you know if anyone's working on the current buildbot failure? lp/translations/doc/gettext-check-messages.txt
<jtv> noodles775: no idea; let me check my irc backlog
<jtv> noodles775: nothing in my backlog... was late logging in.
<noodles775> jtv: thanks - I can't see a failure email either... do you? If not, I'll check why the email didn't get sent.
<jtv> I'm checking my mailbox, but I may already have deleted it if I got it.
<jtv> failure on jscheck...
<noodles775> yeah, saw that one.
<jtv> checking trash now
<noodles775> jtv: do you mind doing the testfix? It looks like it was danilos r8643... we need to get it fixed asap in case the other later changes have issues too :/
<jtv> noodles775: oh, it's a real test breakage?  Been a while since we saw one of those...
<noodles775> yep
<jtv> on db-devel?
<noodles775> https://lpbuildbot.canonical.com/waterfall
<jtv> noodles775: a sucky one, but definitely one in my neck of the woods
<noodles775> Thanks jtv!
<jtv> noodles775: bug 476218
<mup> Bug #476218: Buildbot failure on db-devel: unique violation on POTranslation <Launchpad Translations:In Progress> <https://launchpad.net/bugs/476218>
<noodles775> Thanks jtatum
<noodles775> jtv (sorry)
<jtv> jtatum: if I were you I'd figure out what IRC client(s) these folks are using, and file a bug for over-aggressive nick completion!
<jtv> noodles775: bad news.  The unique constraint that's being violated is on the primary key.
<noodles775> :/
<jtv> In other words, somehow it became possible to create objects with ids that were already taken.
<noodles775> Where's stub when you need him ;)
 * noodles775 pulls db-devel
<noodles775> jtv: running `bin/test -vv -t fix_translation_credits.txt -t gettext-check-messages.txt` gives a different error again.
<noodles775> as if these tests are dependent on being run in some order.
<jtv> as if they forget to mark the database as dirty maybe?
<noodles775> al-maisan: I just noticed this is a branch that you reviewed (not that that implies anything other than you might be familiar with it... do you mind taking a look too? It's very urgent that we get this fixed).
<al-maisan> noodles775: where's the branch you are alluding to?
<noodles775> al-maisan: r8643 db-devel
<noodles775> for bug 474874
<mup> Bug #474874: Missing DB permissions in fix_translations_credits script <current-rollout-blocker> <qa-needstesting> <Launchpad Translations:Fix Committed by danilo> <https://launchpad.net/bugs/474874>
<al-maisan> ah, OK.
<jtv> noodles775: I get exactly the same error when I run that combination
<noodles775> jtv: I just realised why though... I assume we should be running make schema (as there is a db schema security change).
<jtv> ah, that's what I did, yes
<noodles775> yep, my bad.
<jtv> I texted stub
<noodles775> Thanks.
<al-maisan> jtv: this is very funny .. danilo's branch merely changes schema security settings..
<jtv> I haven't spotted any recent changes so far that might've caused this
<jtv> al-maisan: I don't think there's any particular reason to suspect that branch in particular, apart from the fact that this happens to be rosetta
<al-maisan> hmm..
<noodles775> jtv: but it's 100% reproducible isn't it? which means it *has* to be that branch... or did I miss something?
<jtv> stub has an idea
<noodles775> yay
<jtv> there was a zcml branch from a community member that he saw a review pass by for few days ago
<jtv> sorry, a branch that affects zcml
<jtv> ah, he saw a patch fixing a problem like this (which may still not have gone in!)
<jtv> the problem seemed to be something along the lines of objects being created twice
<wgrant> I did a ZCML-related duplicate primary key thingy.
<wgrant> But it only affects "make harness".
<jtv> wgrant: that sounds like the one!
<noodles775> wgrant! tell us more...
<noodles775> wgrant: now it's affecting our test-run...
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/fix-duplicate-harness-zcml/+merge/14464 is the one.
 * noodles775 gets a patch
<noodles775> paste
<jtv> may be same cause
<wgrant> It's unlikely to be the same.
<wgrant> Hmm.
<noodles775> wgrant: http://pastebin.ubuntu.com/311315/
<noodles775> You can see it with db-devel r8643 by doing:
<wgrant> Ew.
<noodles775> bin/test -vv -t fix_translation_credits.txt -t gettext-check-messages.txt
<wgrant> Hm, it's not in a script, though.
<wgrant> What was the diff that killed it?
 * wgrant narrowly avoids swap-death.
<jtv> wgrant: we don't know... looks like the last build that _succeeded_ was triggered by a branch of yours, on Wednesday around 18:xx UTC
<wgrant> OK, pull nearly done.
<noodles775> wgrant: it's turned up on a buildbot run of r8643 - diff http://pastebin.ubuntu.com/311319/
<jtv> so looks like everything >8640 is suspect
<al-maisan> hmm .. running gettext-check-messages.txt alone passes
<jtv> I went through all of those though and saw nothing that might cause this.  :-(
<noodles775> al-maisan: yep.
<noodles775> wgrant: notice that the diff has a new doctest which *does* run a script.
<wgrant> Ah.
<wgrant> I wondered what gettext-check-messages.txt had to do with anything.
<al-maisan> so, something in fix_translation_credits.txt is breaking things, no?
<wgrant> Does fix_translation_credits.txt itself fail?
<wgrant> Or does it just break tests run after it?
 * al-maisan checks
<al-maisan> wgrant: no, it does not fail.
 * noodles775 applies wgrant's diff.
<wgrant> Probably not related, then.
<noodles775> oh, that's for harness.
 * jtv tries forcing a dirty database
<al-maisan> yeah .. it would seem fix_translation_credits.txt only updates existing records
<noodles775> but why does running it together with gettext-check-messages.txt cause the exact failure?
<jtv> noodles775: probably because something updates the database in a way that our test framework doesn't notice, but without calling force_dirty_database either
<jtv> Known, nasty problem.  Thing is, that shouldn't get us into this particular situation.
<wgrant> As long as the test runs in DatabaseLayer, it should be reset every time, right?
<jtv> Because this particular situation smacks of an id sequence being reset,
<jtv> but the objects that have been created not being cleaned up
<wgrant> I don't see code in DatabaseLayer that checks if the DB is dirty.
<wgrant> Hm, but it does run in DatabaseLayer.
<wgrant> Damn.
<adeuring> good morning
<jtv> hi adeuring
<al-maisan> oh, wait, I see `TranslationMessage` instances being created in potmsgset.updateTranslation()
<noodles775> hi adeuring
<adeuring> hi jtv!
<adeuring> hi noodles775!
<jtv> al-maisan: yes, that's what it does
<jtv> and that in turn will create a POTranslation (unless one for the exact same string already exists)
<al-maisan> i.e. this is not an update-only action
<jtv> but we should be looking at fix_translation_credits.txt
<al-maisan> jtv: does run_script() flush the db after completion?
<jtv> It runs a script.
<al-maisan> yes
<al-maisan> right.
<jtv> It flushes the db, but that's not the issue.  The issue is, there's no way for anyone to know that that script just modified the database.
<jtv> So what I'm trying right now is to mark that test as dirtying the db
<jtv> (I was a bit slow about this earlier because I was also still on the phone with stub :-)
<jtv> yup, that fixes it
<al-maisan> jtv: hmm .. sounds promising.
<wgrant> I don't see other tests that *just* run_script.
<al-maisan> Ah.
<jtv> wgrant: we may have them here or there... but they should tell the test setup that the db is dirty
<jtv> al-maisan: yes, more than just "promising." :)
<al-maisan> jtv: :)
<noodles775> Thankyou jtv, wgrant and al-maisan!
<jtv> and stub
<al-maisan> de nada
 * noodles775 's blood pressure drops drastically
<noodles775> (yes, and stub of course!)
<wgrant> When's the reroll?
<noodles775> wgrant: tentatively scheduled for 16:00UTC today
<wgrant> Ah.
<jtv> who wants to review the fix?
<jtv> (still working on the mp)
<noodles775> I'll r-c it :)
<jtv> noodles775: that was my next questionâ"are you authorised to RC?"
<noodles775> adeuring is todays OCR, but it might be quicker for al-maisan given that he's already familiar with the situation?
<noodles775> jtv: yep.
<jtv> Do I request a "release-critical" review for that?
<noodles775> yes please.
<jtv> noodles775: requested
<jtv> turns out searching for "noodles" gives 3 little pages of results
<jtv> none of which seems to be you
<jtv> still, if I did that on google it'd probably be a whole lot worse :)
<noodles775> yep, I've only started using that as a nick since starting here (although it was a nickname in high school).
<wgrant> Why the 775?
<noodles775> noodles is already in use, needed a number
<wgrant> Ah.
<noodles775> (and i was born in 75 which is probably why a 'random' number ended up being 775 ;) ).
<jtv> noodles775: maybe run_script (and equivalents, I think we have 2 of that name already) should just force a dirty db?
<noodles775> jtv: yeah, might be worth chatting with BjornT about a longer term solution.
<jtv> noodles775: I can think of some... I suspect it's mainly a matter of minimizing performance cost
<jtv> also, nowadays, postgres doesn't allocate an xid until you've made changes.  So that could be a way to keep track as well.
<jtv> Or maybe the existing mechanisms should have been enough to prevent this particular failure, and there's a bug in that somewhere.
<jtv> AIUI the id sequences should be reset to whatever is the highest that's still seen in the table, + 1
<jtv> in which case this problem _should_ never have happened.
<noodles775> yep.
<jtv> noodles775: ec2 instance is setting up
<noodles775> jtv: um, for a test-fix, I assumed you'd land it directly on db-devel?
<noodles775> jtv: in fact, I think time-wise, we have no option but to do so?
<jtv> noodles775: ah, of course.
<jtv> noodles775: blame the 'flu or my total inexperience with [testfix], your choice
<noodles775> yes, please submit to pqm to land on db-devel.
<jtv> the [testfix] comes before the [release-critical=], right?
<noodles775> Looking at the log, it seems so:  [testfix][release-critical][r=kiko][ui=rs] Reverting the exposure of
<noodles775>         Request and rc from kiko. Using testfix as there was a db init
<noodles775>   [testfix][release-critical][r=edwin][ui=none] [bug 320889] Fixed a
<mup> Bug #320889: Can't retarget a blueprint <Launchpad Blueprints:Fix Released by sinzui> <https://launchpad.net/bugs/320889>
<noodles775> woops, sorry for pasting there.
<jtv> hmm... got a regex error nonetheless.  :(
<jtv> Now to find why "[testfix][release-critical=noodles][r=adeuring][bug=476218] Fix\n\tbuildbot failure: test failed to force dirty db."
<jtv> does not match "(?is)^\\s*\\[testfix\\]\\s*\\[(?:release-critical|rs?=[^\\]]+)\\]"
<mrevell> Morning
<jtv> hi mrevell, had a chance to look at my scribblings?
<jtv> (that's his morning ruined in a hurry :-)
<mrevell> will do this morning jtv :)
<jtv> :)
<noodles775> jtv: it looks like that regex (ie. for testfix when in rc) doesn't want the =noodles bit.
<noodles775> that would also fit with the examples I pasted above too.
<jtv> noodles775: it says =[^\\]+ before the closing brackets though...
<jtv> oh, that's for rs only
<mrevell> I have to reboot, back in a sec
<noodles775> yep.
 * jtv resubmits
<jtv> ...and PQM accepts!
<noodles775> 3-points - thanks jtv!
<wgrant> bigjools: Is your bug #476316 not a dupe of bug #475172 which is also probably a dupe of bug #464161 which already has bug #475566 as a dupe?
<mup> Bug #476316: Slony-I: Table emailaddress is replicated and cannot be modified on a subscriber node <current-rollout-blocker> <oops> <Soyuz:In Progress by julian-edwards> <https://launchpad.net/bugs/476316>
<mup> Bug #475172: EnsurePerson fails for new maintainers during upload processing <Soyuz:Triaged> <https://launchpad.net/bugs/475172>
<mup> Bug #464161: authdb Store for scripts connecting to incorrect database <Launchpad Foundations:Confirmed for stub> <https://launchpad.net/bugs/464161>
<mup> Bug #475566: Slony-I: Table account is replicated and cannot be modified on a subscriber node <Launchpad Translations:Triaged> <https://launchpad.net/bugs/475566>
<bigjools> wgrant: sort of, but I still need a local fix
<wgrant> Ah.
<bigjools> file first, dupe later I say
<deryck> Morning, all.
<sidnei> salgado: ping
<salgado> hi sidnei
<sidnei> salgado: found the problem. thankfully i've been bitten by this before.
<sidnei> salgado: take a look at _bindUIPicker in picker.js
<salgado> sidnei, yay!  will do in a minute
<sidnei> salgado: there's a handler for the focus event that's doing e.halt() if e.target !== this._search_input
<sidnei> salgado: i think the order that the handlers get dispatched changed, so that handler, which is for the bounding box gets triggered before the more specific handler for the extra-input
<salgado> and that handler is to make the focus go to the search input whenever I click on the bounding box?
<sidnei> salgado: apparently yeah. i think it should also check if the target isn't an input or some other control. that should fix your issue.
<salgado> sidnei, cool, I'll give that a try.  at least now I know where the problem is. :)
<salgado> sidnei, thanks a lot!
<sidnei> iirc, all this mess is because firefox gets focus events wrong. there are lots of threads out there about it.
<Ursinha> bigjools, oi :)
<bigjools> Ursinha: oi
<Ursinha> bigjools, there's one oops that only  happens to one team, easily reproducible: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1405K1314
<Ursinha> bigjools, but only you may know the cause :)
<bigjools> Ursinha: I am looking
<bigjools> Ursinha: it's a data inconsistency issue, I've no idea why it would get like that
<bigjools> basically the build state says it's building but there's no buildqueue record
<bigjools> we need to reset the build state
<Ursinha> bigjools, yes, then it shows in the list and when you click, boom
<Ursinha> bigjools, resetting it is easy to do?
<bigjools> needs LOSA + SQL
<Ursinha> right
<Ursinha> bigjools, I'll file a bug just to keep track
<bigjools> BjornT: can you approve the DB query I just added to LPS please
<jtatum> lol jtv :)
<ursula> oh, I lost a joke from jtv :/
<noodles775> for Ursinha : <jtv> jtatum: if I were you I'd figure out what IRC client(s) these folks are using, and file a bug for over-aggressive nick completion!
<Ursinha> lol
<Ursinha> thanks noodles775 :)
<noodles775> :)
<jtatum> don't feel bad, Ursinha, it was ~6 hours ago ;)
<Ursinha> jtatum, oh! I thought that it happened in the moment I lost connection here :)
<bigjools> Ursinha: we didn't say anything bad about you, honest
<Ursinha> bigjools, oh, that I don't mind, kinda used to it
<Ursinha> :P
<bigjools> surely not :)
<jtv> jtatum: so... you still think I'm joking?  Make a note to read it again a few months later.  :-)
<jtatum> lol
<jtatum> jtv, noted ;)
<salgado> barry, when you encountered that different rich-root problem, I assume you fixed it by creating a new repo?  if so, what format did you use for it?  I used pack-0.92 but that doesn't support stacking AFAICT
<barry> salgado: i used pack-0.92.  i didn't notice if it stacked or not (i /think/ it did but i'm not positive)
<barry> salgado: yes, i created a new repo, and used diff/patch to get the change across.  kinda sucked ;)
<salgado> barry, indeed, I've just did that.  I think the first format that supports stacking is 1.6, though
<barry> salgado: yeah.  stacking's a lp optimization though.  when the format bug is fixed i'll happily let lp consume less disk space ;)
<salgado> barry, well, not just that, if you don't use stacking you'll have to push the whole history every time you push a new branch
<barry> salgado: true.  lp-dev-utils is small though and i have a fat pipe
<Ursinha> danilos, what's the difference between product and project?
<gary_poster> barry: do you know the procedure for adding a dependency to launchpad-dependencies (as in, python2.5-devel :-) )
<barry> gary_poster: i don't, but salgado-lunch does
<gary_poster> barry: I thought he did, and pinged him elsewhere, but was excited enough to ask you just in case while he is at lunch :-)
<gary_poster> thx
<barry> gary_poster: ;)
<Ursinha> I wish I could ask wgrant, he's a walking grep :)
<matsubara> Ursinha, product is a project and project is a project group. those are the old names
<Ursinha> danilos, sure :)
<Ursinha> matsubara, thanks!
<Ursinha> danilos, so, what's the difference between primary_translatable and translation_focus?
<danilos> Ursinha, translation_focus is explicitely set (or not) and primary_translatable can be defined even if translation_focus is not set (it's calculated otherwise)
<danilos> Ursinha, basically, translation_focus mirrors the DB value, and primary_translatable provides with a good default if translation_focus is not defined
<Ursinha> danilos, right, that leads to the second question
<Ursinha> danilos, if translation_focus is none, what should it be, development_focus?
<danilos> Ursinha, well, what do you think it should be? (a serious question)
<Ursinha> errr sorry
<Ursinha> hm, no, that's right
<Ursinha> danilos, I think it should be the dev focus
<danilos> Ursinha, the situation is this: project maintainer has not defined any release as the focus for translations
<danilos> Ursinha, and I agree, it makes most sense :)
<Ursinha> danilos, and after that, keep the same as it was
<danilos> Ursinha, and if that's not defined either, I'd pick the latest series
<danilos> Ursinha, well, it should default to development_focus already
<danilos> Ursinha, so, the change you are making should be "if there's translation_focus, then that, otherwise keep the same as it was" :)
<Ursinha> danilos, that's what I thought was the right thing to do :)
<Ursinha> danilos, other question
<Ursinha> danilos, what's the secondary_translatable_serieses?
<danilos> Ursinha, I don't know
<Ursinha> danilos, :)
<danilos> Ursinha,         """Return a list of IDistroSeries that aren't the translation_focus.
<danilos>         It only includes the ones that are still supported.
<Ursinha> danilos, just want to be sure that's what I imagine it to be, a list of translatable series that not the translation focus
<Ursinha> danilos, I see the comment, boss :P
<danilos> Ursinha, well, it seems pretty clear to me :)
<EdwinGrubbs2> deryck: ping
<deryck> EdwinGrubbs, pong
<EdwinGrubbs> deryck: I just sent you an email that I hope you can respond to now, so that I can land the branch today before I go to the lazr-js sprint.
<deryck> EdwinGrubbs, will look now.
<deryck> EdwinGrubbs, the design is fine that you sent.  I have no qualms at all. :)  I actually like that better than what I proposed.
<EdwinGrubbs> thanks
<sinzui> EdwinGrubbs: I sent you a reply to the design
<sinzui> EdwinGrubbs: If you don't was to fix the CSS, I'll do it right now to fix bug 465524. I know the problem affects more than the two portlets we are discussin
<mup> Bug #465524: ~name Latest Memberships portlet joined date is rendered in the incorrect style <post-3-ui-cleanup> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/465524>
<bigjools> Ursinha: attagirl for your reply to danilos :)
<Ursinha> bigjools, is that good? :)
<bigjools> yes very :)
<EdwinGrubbs> sinzui: I'm fixing the registered CSS class not to use italics. Can you look at https://edge.launchpad.net/ubuntu/+mirror/ftp.iinet.net.au-archive ? It seems like the "registered by" info should go in the top right corner even though it is really long on this page.
<sinzui> EdwinGrubbs: does that registration info *look* like it is correct? Where should that information be on the page?
<EdwinGrubbs> sinzui: I have no idea if it is correct. Normally, we put "registered by" just above the sidebar.
<sinzui> EdwinGrubbs: Yes it goes in the registration slot. Official belongs in the Archive information portlet
<sinzui> EdwinGrubbs: you should have an idea. Any time you want to know when an object was created and by whom, look to the right under the logout button
<sinzui> EdwinGrubbs: You do not need to fix this issue. I'll report and bug and get it fixed. There are several cases of these.
<danilos> Ursinha, you'll know who to ask any questions from now on, now that you hurt my feelings ;)
<Ursinha> danilos, lol, sorry boss
<Ursinha> danilos, you're emacs, gnome guy, but I love you anyway
 * Ursinha pets danilos
<danilos> Ursinha, :)
<Ursinha> :P
<Ursinha> danilos, how do I find out the currentseries of a project?
<Ursinha> err product
<danilos> Ursinha, 'currentseries'?
<danilos> Ursinha, do you mean development_focus series? :)
<Ursinha> oh, wait
<Ursinha> danilos, apparently product.currentseries is related to distros
<danilos> Ursinha, I don't see product.currentseries
<danilos> Ursinha, I only see distribution.currentseries
<Ursinha> danilos, sorry, I see targetseries = ubuntu.currentseries
<danilos> Ursinha, right... is there any reason why do you worry about the rest of that method? :)
<Ursinha> danilos, no, I'm just trying to understand how it works :)
<danilos> Ursinha, well... it's quite simple :) it returns either the development_focus productseries or the latest productseries which has translatable templates; if none of those exist, it returns any linked ubuntu sourcepackage instead, or if there's no ubuntu sourcepackage, any other sourcepackage
<danilos> Ursinha, btw, did you get the DB patch reviewed?
<Ursinha> danilos, nope :(
<Ursinha> guess people are busy this weel
<Ursinha> week
<mrevell> good weekend all
<maxb> abentley: around? I don't understand your assertion that "undecided" repositories would encourage pack-0.92?
<mars> flacoste, do you have a forecast as to when we will be dropping r-c?
<beuno> mars, how's the yui3ification?
<mars> beuno, moving along.  Everything works well, we've only had a few small changes to deal with.
<mars> beuno, I just fixed a bug that was preventing the production config from loading.  With that fix we are one big step closer to being able to land the branch.
<beuno> mars, that's great news
<beuno> and most widgets work with 3.0.0 and the updated lazr-js
<mars> beuno, we will at minimum have an integration branch to build off of for the sprint
<mars> beuno, most do
<beuno> mars, that's awesome, thank you
<flacoste> mars: it's already dropped, since ~4h ago
<mars> flacoste, ok, thanks
* mars changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 4 of 3.1.10 | PQM is open - release manager is noodles775 (flacoste, BjornT) | I am Zero OOPS and So Can You! http://is.gd/4fkLl |   https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<mars> sidnei, ^ it should be ok to start landing lazr-js branches, if you dare tempt the wrath of Friday :)
<sidnei> mars: i would prefer waiting until monday :)
<mars> hehe
<mars> sidnei, ok, we will get everything straightened out monday morning during the sprint opener
<beuno> sidnei, could you add the branch URL to the sprint page?
<mars> beuno, we'll have to land on trunk.  salgado and I have made a number of fixes on top of sidnei's and Bjorn's work
<beuno> mars, can we have the latest there?  it will take a while to get it into trunk, and it would be ideal not to stall on it
<mars> beuno, we can.  I can push my integration branch with everyone's changes rolled into it.
<beuno> mars, that would rock. Add it to the agenda wiki and we're set!
 * wgrant dislikes the new c-i-p UI.
<beuno> wgrant, c-i-p?
<wgrant> beuno: canonical-identity-provider
<beuno> wgrant, the one with the launchpad UI, or the ubuntu UI?
<wgrant> beuno: The Ubuntu one doesn't seem to be serving anything yet. Does that UI exist?
<wgrant> Maybe that one is less utterly sucky...
<wgrant> But I presume not, since 3.1.10 brought some changes to the Launchpad UI.
<beuno> wgrant, it's completely different
<beuno> and should stop sucking
<wgrant> beuno: Will it supplant the Launchpad UI entirely?
<beuno> wgrant, it will be a separate instance with the same db
<beuno> so it will be used for all ubuntu-related stuff
<beuno> at some point, we will either re-work the launchpad one, or replace it with the ubuntu one
<wgrant> Ahh.
<wgrant> Bug #476963, anyway.
<mup> Bug #476963: UI is pretty awful <Canonical SSO provider:New> <https://launchpad.net/bugs/476963>
<beuno> :)
 * thumper is at kiwipycon
<thumper> even has web access :)
<wgrant> Not once everyone else gets there.
 * wgrant has bad memories of UDS Jaunty.
<ajmitch> wgrant: it's not that bad, honest
<ajmitch> (yet)
<wgrant> Until 128 people get into the hotel, and the DHCP range fills up.
<ajmitch> we're not all at a hotel, thankfully
<mwhudson> yes, hotels insisting on doing the networking are the bane of conferences
<EdwinGrubbs> sinzui: so, do you just want me to move the milestone badge to be first, since it is the most often present?
<sinzui> EdwinGrubbs: I expect it to have the same order as this: https://edge.launchpad.net/launchpad-registry/+milestone/3.1.11
<EdwinGrubbs> sinzui: I'm also wondering if I should implement beuno's 0.5em padding on the <tr> or on the .registered stylesheet.
<sinzui> EdwinGrubbs: I think right-aligned is all that is needed.
<sinzui> EdwinGrubbs: I need to find my children. I'll be back in a bit.
<EdwinGrubbs> sinzui: huh, beuno asked for 0.5em padding below each item.
<maxb> I want to use bazaar to track source packages with modifications in the launchpad ppa. Would it make sense to register a launchpad-ppa project?
<wgrant> maxb: People use distro branches for that.
<maxb> for packages in a ppa?
<wgrant> Yes.
<wgrant> Not exactly semantically correct.
<maxb> hmm
<wgrant> But probably better than a project.
<wgrant> After all, they are (in most cases) branches of an Ubuntu package.
<maxb> lp:~maxb/ubuntu/karmic/python-defaults/launchpad-ppa ?
<wgrant> Right.
<maxb> mm, neat
<wgrant> Hm?
<maxb> oh, I'm just sitting here grinning that having twigged that package branches are the right way to do it, I can just push and all is happy :-)
<wgrant> Yup.
<wgrant> I'm not sure if you can manually stack on LP without playing with bzrlib.
 * wgrant tries.
#launchpad-dev 2009-11-07
<wgrant> Hm.
<wgrant> tellurium seems unhappy.
<wgrant> Yes, I just killed staging codehosting :(
<wgrant> Pushed https://code.staging.launchpad.net/~wgrant/ubuntu/karmic/bzr/test-1. tellurium:22 now rejects connections, and there is a rather sinister error message on that page since the branch got scanned.
<ajmitch> wgrant: that does look a bit worrying
<cody-somerville> maybe we should alert a losa?
<wgrant> staging's no production.
<wgrant> Given how broken production codehosting got during the rollout (and the cowboys required), I'm not entirely surprised.
<cody-somerville> mmm... it was very frustrating.
<lifeless> thumper: http://nb.io/hacks/csshttprequest
<wgrant> AJACSS. Nice.
<jml> wgrant, that's a very sinister looking error message
<lifeless> jml: hi! where are you?
<jml> wgrant, but AIUI, the prod codehosting issues had nothing to do with dependencies.
<jml> lifeless, in a hotel in Hong Kong
<lifeless> jml: doing a stop over?
<jml> lifeless, yeah. believe it or not it actually saves us money :)
<lifeless> jml: ?!
<lifeless> jml: hows doe that work? different carrier?
<jml> lifeless, round-the-world ticket
<jml> lifeless, I'm going on to Dallas after Queensland.
<lifeless> jml: ah, going to UDS?
<wgrant> jml: Right, it is a bit odd. Particularly because it did successfully scan the single revision in the branch.
<lifeless> jml: so, what are you doing?
<jml> lifeless, enjoying a post-breakfast malaise
<jml> lifeless, my flight leaves after 10pm, so I've got a bit of time to kill, and no room to do it in.
<lifeless> mmm
<jml> I also should do things like prepare a launchpadlib hacking session for UDS.
<lifeless> its the weekend
<lifeless> you should see hong kong a bit; get hotel to care for luggage
<lifeless> and update tribunal a bit too
<jml> heh heh
<wgrant> Oh *goody*. Somehow a bad version of c-i-p made it into the rollout.
<wgrant> (Bug #476963)
<mup> Bug #476963: UI is pretty awful <Canonical SSO provider:Incomplete> <https://launchpad.net/bugs/476963>
<lifeless> wgrant: bad versions of a lot of stuff, apparently :P
<lifeless> jml: hahaonlyserious :- be a tourist and do personal stuff too. Its *good for you*
<jml> lifeless, sure
<jml> lifeless, although, all things being equal I'd like to stay indoors and read :)
<jml> how did people ever do tourist stuff before google maps and wikitravel
<mzz> heh, you know about this ancient invention where they print a map on paper? :)
 * mzz had fun playing tourist in prague with a bunch of people where half of them had a mobile device and none of them could get a decent gps signal, so they were all lost
<mzz> fortunately one of us had an oldfashioned map
<lifeless> mzz: baked it in clay
<lifeless> jml: reading can be fun too
<lifeless> jml: perhaps you could find a by-the-hour hotel :)
<jml> lifeless, I'm actually in the right area of town for that
<jml> mzz, oh, I don't have a smartphone / gps thingy.
<jml> mzz, I consult maps before I head with nothing but my memory and a misplaced confidence in my own abilities to navigate
<mzz> sounds like a plan
<mzz> (I don't have one of those either, but I know I'd be overusing it if I did)
<lifeless> jml: its a canonical tradition to be in the right area of town
<jml> lifeless, heh
<jml> lifeless, I seriously doubt Mooloolaba has such an area
<lifeless> jml: its the sunshine coast.... it *is* such an area
<jml> naaah
<jml> gold coast, maybe
 * sinzui starts to fix barry's damage
 * wgrant prepares a few sed branches :(
<maxb> Does anyone know for any documentation of process for how we manage non-PQM-ed projects?
<maxb> I'm hoping to crib it as I write guidelines on how to make changes to meta-lp-deps
<wgrant> Given some of the things I've seen done, I'm not sure there are any.
<nhandler_> Is something gong on with the LP servers? I am not able to get the rocketfuel-setup script using the command on dev.launchpad.net
<thumper> hmm...
<thumper> listening to a talk on deliverance
<thumper> seems very interesting
<janklopper> Hi, quick (hopefully) question
<janklopper> can launchpad run on debian?
<lifeless> probably
<lifeless> thumper: repoze.org looks interesting :)
<thumper> lifeless: I haven't looked at it yet
 * thumper seconds lifeless's probably for killercow
<lifeless> http://zedshaw.com/blog/2009-11-6.html <- 1-5 ratings suck
 * thumper looks
<killercow> ill go and try it out next week then
<killercow> can't be too hard, if all the deps check out
<lifeless> :P
<lifeless> are you going to hack on it, or do you want to deploy it somewhere?
<killercow> deploy it locally, then hack our unittesting and linting frameworks in there
 * lifeless perks up at unittesting
<lifeless> what framework do you use?
<killercow> brb, gotta pick someone up
<killercow> lifeless: a couple, one for each language :P
<thumper> lifeless: he obviously doesn't want to talk to you :)
<thumper> gah, beaten to Enter yet again
<lifeless> killercow: how do they tie to launchpad, for you? [and have you heard of subunit ;)]
<lifeless> thumper: you're getting slow in your old age
<thumper> yeah
<ajmitch> it's the comfy couch that's slowing him down
<thumper> there is space next to me now :)
<lifeless> I've got my legs up and direct view on the project though
<lifeless> projector
<killercow> back :)
<killercow> lifeless: haven't heard of it
<killercow> our project does linting for various languages and displays stuffs the results in the database
<killercow> so, it adds linting + unittesting + "accountability" based on the repository changes
<lifeless> interesting
<wgrant> killercow: I was able to run LP on Lenny by convincing rocketfuel-get that it was actually running on Hardy (so it used the right PPA).
<killercow> hmm, sounds good enough
<wgrant> But I also had to manually grab a couple of extra packages like ubuntu-keyring and something else that I forget.
<killercow> will probably want to try this in a vm anyway
<wgrant> I meant to write proper instructions, but Karmic's KVM sucked and corrupted my disk image at that point.
<killercow> so I can hose the env and go back if needed
 * wgrant tries again.
#launchpad-dev 2009-11-08
<wgrant> Where is that mwhudson? I feel he's taking a bit too much liberty with his presentation about Twisted in LP -- buildd-manager reliable? ha ha ha
<ajmitch> last I saw, he was somewhere in this room
<lifeless> wgrant: were you at pyconnz?
<wgrant> lifeless: No.
<wgrant> The only Linuxy thing I've managed to get to is UDS Jaunty.
<jml> wuu
<jml> now have ec2 spitting test results into a test result object
<wgrant> jml: Is this so you can start a bazillion ec2tests and run the test suite in just a few minutes?
<lifeless> wgrant: components of, yes
<wgrant> Fancy.
<lifeless> wgrant: subunit
<lifeless> jml: +1
<lifeless> jml: when will tribunal show it?
<jml> wgrant, that's the next step, yes.
<jml> lifeless, some day :)
<robert_ancell> For the last few days when I try and access bug 429322 I get a timeout.  Other bugs work fine.
<mup> Bug #429322: seahorse-agent assert failure: ERROR:iop-profiles.c:606:IOP_generate_profiles: assertion failed: (obj && (obj->profile_list == NULL) && obj->orb) <apport-crash> <i386> <seahorse-plugins (Ubuntu):Confirmed for robert-ancell> <seahorse-plugins (Ubuntu Karmic):Confirmed for robert-ancell> <seahorse-plugins (Fedora):Confirmed> <https://launchpad.net/bugs/429322>
<robert_ancell> This bug is being frequently duplicated and is getting a lot of angry users complaining about being spammed.
<mars> robert_ancell, it may be a while before someone can look into that problem: 9am Australian time at least I'd think.  Many on the team are on sprints this week.  I suggest sending a mail to help@launchpad.net
<spm> mars: given robert's in sydney... ;-)
<robert_ancell> 10:40 in sydney :)   Didn't know everyone was off sprinting :(
<spm> UDS + couple of sprints. LS, pycon in NZ, few others.
<robert_ancell> mars, spm: can you guys confirm you get the same timeout?
<spm> robert_ancell: yeah.  i Saw it on edge. was looking at the oops I got. OOPS-1408C2473 weird.
#launchpad-dev 2010-11-08
<lifeless> wgrant: oh ?
<lifeless> :( State: Chroot problem
<wgrant> lifeless: Ew where?
<lifeless> sandpaperfig
<lifeless> natty
<wgrant> A recipe build?
<wgrant> natty-cat-lpbuildd doesn't exist yet..
<wgrant> So no recipes on natty until we get the new lp-buildd out.
<wgrant> Which is blocking on the Code bug being fixed.
<wgrant> Which probably needs a Julian.
<wgrant> Which I believe we will have tonight.
<lifeless> wgrant: yes
<lifeless> wgrant: anyhow, what do you mean by '12:27 < wgrant> Yeah, it looks like it has the same bug that r11812 fixed.
<wgrant> lifeless: There is a race in the fix for 669676.
<lifeless> oh, the 'submitted while starting up' thing ?
<wgrant> Yeah.
<wgrant> Bug #627608
<_mup_> Bug #627608: Got a 401 on a fresh purchase <qa-ok> <Software Center Agent:Fix Released> <Soyuz:Fix Committed by michael.nelson> <software-center (Ubuntu):Fix Released> <https://launchpad.net/bugs/627608>
<lifeless> wgrant: we are btw running 11858 now
<thumper> lifeless: I thought you fixed the recipe feature flag?
<wgrant> lifeless: :(
<wgrant> No recipes yet.
<lifeless> thumper:
<lifeless> thumper: what do you mean?
<thumper> lifeless: that the recipes will be visible on prod for a specified team
<lifeless> yes
<lifeless> once that code is deployed.
<lifeless> 11866 is the magic number to get qa'd and deployed
<thumper> ok
<lifeless> (bzr log devel :P)
<lifeless> \o/ my +packages change seems to have worked.
<lifeless> https://launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204 worked first time.
<StevenK> lifeless: Why does the deployment page sometimes show the revisions behind the one that is blocking rollout, and sometimes doesn't?
<lifeless> bad-commits show more
<lifeless> because they aren't a blocker *if* their rollback is present
<lifeless> unqaued stops the thing
<lifeless> its being altered to show more always
<StevenK> Excellent
<lifeless> oh OOPS-1773EA141 where art thou?
 * StevenK notes qastaging wasn't reset over the weekend
<wgrant> lifeless: getBuildStatusSummaryForSourceIds hasn't had its timeout lifted?
<lifeless> hard_timeout pageid:Archive:EntryResource:getBuildSummariesForSourceIds 1 20000
<lifeless> two possibilities
<lifeless> doesn't work on apis
<lifeless> its terrible
<lifeless> I'll check yesterdays oops
<wgrant> Hmm. It's not showing up in the debug footer's scopes.
<wgrant> in scopes {'pageid:BugTask:+create-question': False, 'pageid:Archive:+packages': True, 'pageid:Milestone:+index': False, 'pageid:Archive:+index': False, 'pageid:Person:+bugs': False, 'pageid:POFile:+translate': False}
<lifeless> that thing is a bit bustified
<lifeless> it shouldn't be showing all those for starters
<lifeless> SQL time: 19144 ms
<lifeless> Non-sql time: 1201 ms
<lifeless> Total time: 20345 ms
<lifeless> Statement Count: 155
<lifeless> conclusion : its terrible.
<lifeless> https://api.launchpad.net/devel/%7Exorg-edgers/+archive/ppa
<lifeless> source_ids=%5B%221356771%22%2C+%221356770%22%2C+%221356769%22%2C+%221349686%22%2C+%221349688%22%2 ... 1164%22%2C+%221348857%22%2C+%221348858%22%2C+%221348870%22%5D&ws.op=getBuildSummariesForSourceIds
<wgrant> Yeah.
<lifeless> wgrant: hey
<lifeless> do you have a script that grabs attachments of private bugs, by chance?
<wgrant> lifeless: No. All I know that does that is apport.
<wgrant> lifeless: Going to try the shiny new librarian?
<lifeless> yes
<lifeless> all reports on qastaging were positive
<wgrant> Yay.
<wgrant> Erm, there is no DNS.
<wgrant> But OK.
<lifeless> thats interesting ..
<wgrant> It's there for staging and qastaging, but not production.
<lifeless> yes
<lifeless> folk are busy
<lifeless> its queued, or something
<lifeless> I've pinged on the rt
<lifeless> wgrant: the scopes that you see are only those evaluated in order to do the page
<lifeless> others below that in thepriority list are not show
<lifeless> *shown*
<wgrant> lifeless: Ah, so it queries them one at a time?
<lifeless> one query to get the rules
<lifeless> then one rule evaluated at a time
<lifeless> of the rules for the flag being requested
<wgrant> Ah.
<lifeless> many rules do not require db querying to evaluate
<lifeless> so far only team: does, in fact
<thumper> wgrant: do you happen to know off the top of you head which template is used for binary builds?
<wgrant> thumper: Probably build-index.pt
<wgrant> Yes.
<wgrant> They weren't renamed when the class and interface were.
<thumper> ta
<wgrant> I should probably fix that at some point, but there's so much overhead in getting stuff landed :(
<bac> thumper: do you think your [testfix] will really work or just hopeful?
<spm> wgrant: you'd like access to directly edit the source on the prod servers? >:)
<wgrant> spm: pls.
 * spm tries to figure out how to express "no" in a polite, yet firm way.....
 * wgrant pouts.
<lifeless> spm: FOAD
 * spm avoids drinking, and hence avoids a replacement needing keyboard incident
<spm> lifeless: I was thinking it, i wasn't gunna say/type it
<lifeless> :)
<lifeless> spm: and here I was imagining you with a coffee
<spm> something about my cold dead corpse on the ground as well was needed
<wgrant> spm: That can always be arranged :)
<wgrant> Canberra is cold enough.
<spm> wgrant 1, spm 0
<lifeless> OTOH
<lifeless> you'd have to go to canberra
<wgrant> True.
<wgrant> I'm not sure I'd risk it.
<spm> I could take you for a tour thru NPH! Watch your elected pollies in action!
<lifeless> I want a jack point in the NBN switch room
<lifeless> stuff the pollies
<spm> I've been in their DC. About 15 years ago, was bloddy impressively large then.
<spm> heh, had to get them to convert a 9" reel tape onto ... blah, some DEC VMS tape format whose name eludes me.
<spm> same physical format as DLC tape, I think it is.
<lifeless> oh
<lifeless> uhm
<spm> fail. same size as LTO!.
<spm> gah. DLT!!!. ffs. fail fail fail.
<lifeless> rofl
<spm> TK50, TK70 tape.
<spm> http://en.wikipedia.org/wiki/Digital_Linear_Tape <== *1984* format size. still in use today (MUCH higher capacity)
<wgrant> Even I've used SDLT.
<spm> you old man you
<spm> come to join us greybeards eh?
<wallyworld> thumper: bug 671458. do you agree with it? all new mps will no longer have no reviewer set so is the behaviour described in the bug wrong? i can see both sides.
<_mup_> Bug #671458: Branch.addLandingTarget adds a review even when the caller requests there be none. <code-review> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/671458>
<thumper> bac: I expect it to wokrk
 * thumper needs to offload some kids
<thumper> anyone want any?
<bac> thumper: thanks, but no.
<lifeless> sinzui: ping
<lifeless> sinzui: you seem to (perhaps) have a script marking bugs as fix released... I think if you do that it will need some improvements to be reliably in RFWTAD world
<StevenK> lifeless: I spoke about that to sinzui at UDS.
<StevenK> I think in the short term feeding it a list of bugs would be good
<lifeless> Well I'm specifically thinking of:
<lifeless>  - incrementals
<lifeless>  - not deployed to
<lifeless> cases
<wgrant> s/Fix (Committed|Released)/Fixed/g
<lifeless> wgrant: EWHINE
<lifeless> http://www.slideshare.net/padday/the-real-life-social-network-v2
<wgrant> I saw that yesterday, got up to slide 40 or so, then gave up.
<wgrant> Is there anything interesting in the later bits?
<lifeless> not sure
<lifeless> The start is obvious but interesting
<lifeless> http://www.hideandseek.net/cant-play-wont-play/ is also entertaining
<wgrant> Wow that is a lot of Flash.
<wgrant> Yay, PQM doesn't want to eat my soul.
<thumper> http://pastebin.ubuntu.com/527981/ some recipe stuff
<thumper> sourcepackages being built by daily recipes that have successfully built
<lifeless> wgrant: its worth finishing the social network thing; its getting interesting around page 100
<wgrant> lifeless: Hm, I may, then.
<wgrant> After I finish playing with this ARM netbook.
<StevenK> Don't you have study to do? :-)
<wgrant> Pfft.
<lifeless> bah, soyuz test fail
<StevenK> lifeless?
<jelmer> lifeless: hi
<thumper> no danilo?
<adeuring> good morning
<jelmer> lifeless: still there?
<mrevell> Hi
<jtv> hi henninge!
<jtv> I thought you weren't here today?
<henninge> jtv: Hi!
<henninge> You thought wrong ... ;) I am just late.
<jtv> Ah.  I am late too, at least in terms of IRC login.
<jtv> I think I have the causes now: id change on devpad, another server freeze of the kind I've been getting since the lucid upgrade, a crappy new router, and some temporary weirdness with my internet connection.
<jtv> Not easy to debug so many variables.  :-)
<StevenK> Which id changed on devpad?
<henninge> jtv: that sounds hard :(
<jtv> StevenK: the ssh pubkey
<jtv> henninge: still not done tbh, but I've worked around the problems and that's something.
<LPCIBot> Project devel build (197): FAILURE in 3 hr 13 min: https://hudson.wedontsleep.org/job/devel/197/
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa] log files are created in a logs
<LPCIBot> subdirectory instead of the root directory
<LPCIBot> * Launchpad Patch Queue Manager: [r=jelmer][ui=none][no-qa] Calculate a distroseries' architectures
<LPCIBot> and components directly from the DB, not via lucilleconfig.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none][bug=671568] Remove the last imports from
<LPCIBot> canonical.launchpad.interfaces from lp.*.
<jelmer> wgrant: Ah, you finally managed to land your branch?
<wgrant> jelmer: Yes!
<wgrant> jelmer: PQM decided to jump out of testfix for a while.
<wgrant> Thanks for trying it a couple of times.
<wgrant> Now I just need to work out what to do about germanium's config's insanity, then the rest of the sequence can land, and lucilleconfig will evaporate.
<jtv> A commit by gmb triggered a successful devel build this weekend, but the fix didn't make it into db-devel.  So thumper did a manual merge this morning.
 * bigjools is getting frustrated with people targeting bugs to "soyuz" just because they have the word "package" in the title
<henninge> lifeless: ping
<henninge> jtv: danilo_ is asking about you ... ;)
<jtv> henninge: ah!
<jtv> I got no answer from him earlier
<danilo_> hey jtv
<danilo_> jtv, you should have gotten my away message :)
<jtv> hi danilo_!  No, I got nothing.  I couldn't log into the internal IRC.  :-(
<jtv> But I rebooted my server so I can get in now.
<danilos> jtv, ah, that's where my away message was :)
<jtv> Figures.  It was _not_ an easy day, IT-wise.
<danilos> jtv, why, what happened?
<wgrant> bigjools: And so it begins :(
<bigjools> wgrant: ?
<wgrant> bigjools: My work queue is already non-empty, and I don't start for a month :P
<bigjools> wgrant: :D
<bigjools> dude, the queue is massive, it's just that the first one is written down ;)
<wgrant> Haha.
<jtv> wgrant: stop laughing and start coding.  Welcome on board.  :-)
<wgrant> jml: Is the s/PPA/Software Archives/ part of an evil plan to introduce project-owned PPAs?
<jml> wgrant: not really.
<jml> wgrant: I mean, I have an evil plan to introduce project-owned PPAs
<jml> wgrant: but stage 1 of that plan is "introduce project-owned PPAs"
<bigjools> PPA is dead.  Long live LSA.
<jml> bigjools: please don't replace PPA with another acronym.
<bigjools> almost LSD, which seems eerily appropriate.
<wgrant> Then the primary archive can be a Launchpad Ordinary Software Archive.
<wgrant> And then we have awesome acronyms.
<bigjools> jml: I am not going to type "Software Archives" in full every time, your request is going to be pretty futile I'm afraid
<jml> bigjools: what's wrong with just typing "archive" for short
<bigjools> it's not specific enough
<bigjools> I suspect ppa will live on in dev circles, frankly
<jml> me too. I'm not too worried about that.
<jml> but introducing another obscure acronym will be an utter fail
<bigjools> it would only be in dev circles, not in the UI :)
<jml> hmm.
<bigjools> what do I do with ArchivePurpose.PPA for example
<bigjools> we could just leave everything as ppa, internally
<jml> well, that's going to be an interesting question anyway, once you get project archives
<bigjools> indeed
<wgrant> ArchivePurpose is probably crack.
<bigjools> why?
<jml> prefer polymorphism to type checking!
<bigjools> I agree, I just wish PG would do that ;)
<wgrant> I'm not entirely a fan of the way PPAs are modeled at the moment. It feels like we should have PersonalPackageArchive which references an Archive, and Distribution which references an Archive, but I'm really not sure how it would all fit together.
<wgrant> I guess that's probably the case for most of the Soyuz model, though.
<jml> bigjools: separate problem. even if the table has a flag/enum row, the API doesn't necessarily need to have is_ppa() etc
<jml> it might need to for other reasons
<bigjools> yes, it was badly modelled in that regard
<bigjools> I have a first class honours in hindsight
<wgrant> Also, the partner archive sort of screws that idea up.
<wgrant> But we can hopefully kill that soon, right?
<bigjools> nope :(
<wgrant> !?
<wgrant> Commercial PPAs don't win?
<bigjools> not until lucid server is EoL
<wgrant> bigjools: We can't repurpose commercial-compat.sh to mangle a PPA into something that looks like partner?
<bigjools> we'll be doing commercial PPA^Wsoftware archives AND partner
<wgrant> :(
<wgrant> I think I'd prefer a mangling script to keeping this cruft around for five years.
<bigjools> totally - if it can be done, then I am +1
<bigjools> I've not thought about it yet though
<wgrant> Sure.
<jml> hmm
<jml> I'm having trouble building from devel due to some conflicts in the vietnamese mailman po file
<jml> I wonder why.
<jml> hmm, not devel, but devel w/ c/l/interfaces/__init__ gutted
<deryck> Morning, all.
<jml> deryck: hello
<jml> ahhh... deeper, earlier causes
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (198): FIXED in 3 hr 14 min: https://hudson.wedontsleep.org/job/devel/198/
<LPCIBot> * Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa] Unit-test translation permissions.
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa] Use patched version of zope.pagetemplate
<LPCIBot> 3.5.0
<danilos> 3h14 mins? is it by any chance 15.9265 seconds as well?
<jml> bigjools: at some point this week I'd like to talk about uploading to personal archives and the CoC.
<bigjools> jml: ah yes, we decided to fold the PPA ToS into the CoC
<jml> bigjools: esp wrt https://code.launchpad.net/~jml/launchpad/allow-upload-without-coc/+merge/40218
<bigjools> or something with the right acronyms at least
<jml> bigjools: I was talking to sabdfl the other day, and he was fine w/ not requiring CoC for upload rights to PPAs.
<bigjools> jml: I still think we should make it part of the LP ToS, so it's just implicit
<jml> bigjools: make what part of the ToS?
<bigjools> the PPA ToS
<jml> bigjools: oh right.
<jml> bigjools: but that's a separate question from CoC
<bigjools> I probably mean the CoC :)
<jelmer> oh, there are separate PPA ToS ?
<jml> bigjools: you are either confused or being confusing.
<bigjools> "the thing you agree to when you sign up for a PPA"
<jml> bigjools: that's the terms of service. but there's also code in LP (see https://code.launchpad.net/~jml/launchpad/allow-upload-without-coc/+merge/40218) that prevents you from uploading unless you have signed the Ubuntu Code of Conduct
<bigjools> jml: right - what I mean is that we should amend the ToS to include anything we feel necessary from the CoC
<bigjools> there may be nothing of course, but we should check it over anyway
<jml> bigjools: I've just checked. There's nothing that should be specifically in the PPA terms of service.
<bigjools> ok great
<jml> bigjools: http://www.ubuntu.com/community/conduct ; https://help.launchpad.net/Legal (LP terms); https://help.launchpad.net/PPATermsofUse (PPA terms)
<bigjools> jml: I presume the PPA ToS will be implicit... ?
<jml> bigjools: the change I'm proposing doesn't touch the PPA ToS. You will still have to agree to them before creating a PPA.
<jml> bigjools: longer term, we should fold the PPA ToS into the Launchpad ToS.
<jml> but that's a discussion for another day.
<bigjools> ok
<deryck> gmb, hurrah!  Tests pass and it's done!  https://code.launchpad.net/~deryck/launchpad/rockstar-js-refresh/+merge/40329
<deryck> gmb, I'm going to land unreviewed since rockstar's already passed reviewed.  So one more ec2 run, and it should land.
<gmb> deryck: Let joy be unconfined.
<deryck> indeed!
<gmb> deryck: Cool. Many thanks for all the hard work :)
<deryck> np!  Glad to get it done. :-)
<jml> what actually does the registering of adapters for top-level IEntrys?
<jml> I've got a branch that breaks it for IPillarNameSet by changing import order around (yay), and I'd like to see what stable is doing differently
<bigjools> jml: I think there's some zcml that includes canonical/interfaces/__init__.py
<bigjools> or summat like dat
<jml> something to do w/ lazr.restful.metazcml
<jml> boo yah.
<jml> the webservice:register directive
<bigjools> yeah that one
<jml> sinzui: call?
<sinzui> one moment I am working with a user
<jml> sinzui: np
<sinzui> jml, this hour has fallen apart for me. I just finished with a user, but I also just got a call to pick up my son from school. I wll not be available for 30 more minutes
<jml> sinzui: np.
<jml> sinzui: another day then.
<lifeless> jelmer: hi
<lifeless> henninge: hi
<lifeless> morning y'all
<jelmer> lifeless: 'morning
<henninge> Good early morning lifeless ;)
<jelmer> lifeless: This time you are actually up early!
<lifeless> jelmer: 6am, nominal.
<lifeless> :)
<jelmer> lifeless: You mentioned on bug 627608 that you did some setup on qastaging to test generate-ppa-htaccess
<_mup_> Bug #627608: Got a 401 on a fresh purchase <qa-ok> <Software Center Agent:Fix Released> <Soyuz:Fix Committed by michael.nelson> <software-center (Ubuntu):Fix Released> <https://launchpad.net/bugs/627608>
<henninge> oh right, you are on that little Island
<lifeless> henninge: yes, the little island that could
<jelmer> lifeless: What did you do exactly? Does qastaging do any publishing at the moment ?
<lifeless> jelmer: we did
<lifeless> jelmer: we setup a db user that was missing
<lifeless> and fixed the librarian config for qastaging
<lifeless> because of resource constraints, the previously announced 'ask a losa to run specific scripts' still applies.
<abentley> lamont: what's the minimum RAM for an i386 virtual builder?  Is it different for AMD64?
<jelmer> lifeless: So publishing doesn't happen by default?
<lifeless> jelmer: I don't know what 'publishing' stands for. Can't answer the question.
<lifeless> jelmer: if something you need in order to qa isn't present/happening on qastaging, please ask a losa to arrange it for you, for the patch you're qaing.
<jelmer> lifeless: I'm referring to the Soyuz publish-distro.py script which is used to publish PPA's.
<jelmer> lifeless: I'm wondering though where the results end up if that script ran, as private-ppa.qastaging.launchpad.net nor ppa.qastaging.launchpad.net appear to exist.
<lifeless> jelmer: ok, specifics. \o/
<lifeless> jelmer: ask a losa though - its likely the first time we've qa'd something needing that.
<jelmer> lifeless: I figured the QA'ing of the generate-ppa-htaccess script had the same issue, as it would write to the same location.
<lifeless> jelmer: we asked the losa to look on disk for us.
<jelmer> lifeless: ahh, ok
<jelmer> having to ask a losa for that sort of thing will make QA'ing of Soyuz stuff quite a bit more painful though :-/
<lifeless> jelmer: so there are two things
<lifeless> firstly, ask for the domains etc via rt, cc francis and me if you would
<lifeless> secondly when we get the qa sscripting server, we'll move all the background tasks for staging and qastaging to it
<lifeless> so the staging web ui's will be more representative of a deployed web ui
<jelmer> lifeless: Ok
<jelmer> lifeless: I'd like to file an rt about ppa.qastaging.launchpad and private-ppa.qastaging.net, is that ok ?
<jelmer> private-ppa.qastaging.launchpad.net
<jelmer> (although I do see qastaging.net is still free ;-)
<bigjools> you're going to have much pain running the publisher on staging
<lifeless> jelmer: yes
<lifeless> henninge: is it really week 1?
<henninge> No, this is week 3.
<henninge> lifeless: what makes you think that?
<jelmer> henninge: See the topic :-)
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<henninge> jelmer: what topic?
<henninge> ;-P
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday | Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<bigjools> good night folks
<lifeless> night
<jml> g'night all
<lifeless> night
<lifeless> henninge: so I'd like to go with my proposed mechanism
<lifeless> henninge: but its only a preference
<henninge> henninge: I see the advantage of keeping the focus.
<henninge> lifeless: ^
<henninge> lifeless: is this still up-to-date?
<henninge> https://dev.launchpad.net/MergeWorkflow
<henninge> That's what I used for my propsal.
<lifeless> I don't think its particularly stale. Let me see
<lifeless> ah, its stale.
<henninge> Because you mentioned that we don't have production anymore
<lifeless> the big production circle is now reserved for private-patches only - there was a significant list thread about it
<lifeless> we're deploying straight from stable
<lifeless> the circle for production is not in the deployment path anymore its straight from stable to lpnet/edge once qa'd.
<henninge> I probably saw that thread but the new model had not yet registered well with me so I probably could not follow.
<lifeless> Ursinha: hi, I think you did that lovely graphic ?
<Ursinha> yes sir
<henninge> Hi Ursinha ;)
<Ursinha> hey henninge! :)
<henninge> lifeless: so in that case your workflow makes more sense.
<lifeless> Ursinha: do you have the source still? we need to update it ... I'll update the text right after breakfast
<henninge> I just wish we had a way to block developers less.
<Ursinha> lifeless, yes, I do
<Ursinha> have to find it, a minute, please
<lifeless> actually, updating the text on the page now
<henninge> lifeless: you go for breakfast, I will go for dinner. ;-) But I'd like to continue this afterwards. I need a clear picture of the roll-out, obviously ...
<lifeless> obviously ;)
<henninge> :)
<bdmurray> I got a windmill test failure when trying to build re lp/registy/.../test_team_index.py is this fixed now?
<lifeless> jelmer: so, how are yo ugoing on that qa?
<lifeless> jelmer: it is the most important thing for the team right nw
<lifeless> flacoste: ping
<lifeless> flacoste: the 'release process db-devel->devel 						
<lifeless> Inbox
<lifeless> thread - you noted a small omission on my part about handling urgent fixes during the stabilisation period, but you didn't indicate any other approval/disapproval
<Ursinha> lifeless, I've attached to the wiki page the latest version I have of the diagram
<lifeless> I think we just collidded :(
<lifeless> Ursinha: thanks
<lifeless> I've restored the end of the page
<Ursinha> argh
<Ursinha> silly question: I just attached the file, does that count as editing the page as if I was changing text?
<lifeless> Ursinha: I think so
<lifeless> but I don't know so
<Ursinha> that explains
<lifeless> Ursinha: it thinks you have an edit lock open ?
<Ursinha> lifeless, it's lying
<lifeless> ok
<lifeless> I'll redo thebottom half of my update
<Ursinha> lifeless, sorry
<lifeless> de nada
<lifeless> moin's th buggy thing
<lifeless> Ursinha: ok, its updated
<lifeless> Ursinha: would you be willing to update the diagram to match the logic ?
<lifeless> Ursinha: I'm rather awkward @ that
<Ursinha> lifeless, I'm in the middle of something right now, so if you want to have it updated now I can't
<Ursinha> but I can do that later, yes
<lifeless> Ursinha: there is no panic.
<Ursinha> cool
<lifeless> grah
<lifeless> Text conflict in database/sampledata/current-dev.sqlText conflict in database/sampledata/current.sql
<lifeless> jelmer: hi
<jelmer> lifeless: Hi
<jelmer> lifeless: I'm wrapping up a discussion of a related review with Aaron at the moment and that RT, am going to ask Chex about the private PPA next.
<lifeless> jelmer: I want to unblock the team - https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html
<lifeless> jelmer: I'm glad you're going to talk to chex next.
<lifeless> but I'd like to emphasis that once something lands on devel, the *most important thing you can do is to qa it*
<lifeless> unqa'd stuff on devel is our critical section
<lifeless> our GIL
<lifeless> etc
<jelmer> lifeless: I did QA it on dogfood, but I haven't QA'ed anything like this on qastaging before. Sorry it's taking so long.
<lifeless> we do 10 landings a day
<lifeless_> bah
<lifeless_> jelmer: what was the last line you saw ?
<jelmer> <lifeless> we do 10 landings a day
<lifeless_> right
<lifeless_> I hit my kill switch :)
<lifeless_> so a days lack of qa backs up 10 commits; 2 days 20 etc
<lifeless_> jelmer: also note that the key test is identifying whether the revision is *deployable*, not whether the *bug is fixed*
<lifeless_> s/*deployable*/*deployable to the nodowntime set*/
<lifeless_> flacoste: also https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-pageids.html is a little uhm, brief.
<lifeless_> flacoste: Its now running your code ...
<flacoste> lifeless: well, it wasn't
<flacoste> lifeless: it now will
<flacoste> lifeless: should all be sorted by tomorrow
<lifeless> flacoste: kk
<lifeless> flacoste: see my other question on db deploys
<flacoste> lifeless: yep, switching to inbox rsn
<lifeless> as opposed to zero ? :P
<flacoste> lifeless: i don't know the state of my inbox when i'm not observing it :-)
<lifeless> ahhh
<lifeless> its quantum
<flacoste> lifeless: are you talking about your reply to henninge? i don't see a reply to my questions
<lifeless> flacoste: well, I think we should discuss in real time
<lifeless> I hadn't replied to your reply, thats true
<flacoste> lifeless: we have a call in 1h15, would that be good?
<lifeless> doing so no
<lifeless> sure
<wallyworld_> abentley: rockstar: thumper: wanna do the standup now?
<abentley> wallyworld_: I can, but we're on standard time now, so an hour later would be fine.
<wallyworld_> abentley: np. today though i have to drop the kid to school a little earlier so i need to do it now :-)
 * thumper is here now
<abentley> thumper: hop on
<lifeless> flacoste: calling you
<lifeless> Ursinha-afk: hi - https://devpad.canonical.com/~lpqateam/qa_reports/deployment-db-stable.html - seems to be missing stuff
<lifeless> flacoste: https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=qa-needstesting,qa-bad
<henninge> lifeless: Hi! ;)
<lifeless> StevenK: could you qa https://bugs.launchpad.net/soyuz/+bug/656166 ?
<_mup_> Bug #656166: Cannot request an IPackageDiff for a DistroSeriesDifference via api <derivation> <qa-needstesting> <Soyuz:Fix Committed by michael.nelson> <https://launchpad.net/bugs/656166>
<jam> I'm being called a failure every 30minutes by Launchpad PQM. It is really hurting my self esteem...
<jelmer> jam: :-))
<jml> does fixing a conflict count as firefighting?
<jam> jml: shouldn't you be in testfix mode since the basic infrastructure is failing?
<jml> jam: me personally?
<jam> "you the launchpad team"
<jam> and possibly "you the guiding person of the launchpad team" :)
<jml> hah
<jam> Since I'm not on the team, "we" would have been a wrong term
<jml> jam: personally, I try to treat a conflict as a top-priority interrupt. But I'm busy doing personal stuff atm.
<jml> jam: "stop the line" is a horribly vague concept in a distributed team.
<jam> jml: well, it has been failing for 3hrs20min according to my logs
<jam> so nothing seems particularly stopped
<jml> merges from stable to db-devel are stopped
<jam> (part of your infrastructure is stopped, in such a way that it is spamming the whole team that it is unhappy, for >3hrs without obvious sign of anyone doing anything. Given that it spams every 20min, it is perceived that something should have been done by now)
<jml> jam: I agree. The situation is bad, and I admit that we have a problem.
<jml> jam: However, as the AA people say, that is only the first step.
<jam> probably the only reason *I* find it particularly annoying is because it spams to launchpad@ vs canonical-launchpad@ so my default filter doesn't put it in the right folderc
<jml> jam: well, I'd be genuinely grateful if you could channel your (quite justifiable) annoyance into some kind of solution.
<spiv> jml: heh.  Where in lean are the other steps, like making amends, I wonder? :)
<spiv> jml: or the "fearless moral inventory"...
<jml> spiv: inventory is waste
<spiv> That almost makes sense!
<lifeless> jml: its a firefight for sure
 * lifeless -> doctors
<henninge> lifeless: wait!
<lifeless> henninge: sorry have to go; flacoste can help you
<henninge> yes, talking to him now ;)
<wallyworld_> poolie: ping! can we have a quick chat when you are free? skype?
<thumper> jelmer: ping
<jelmer> thumper: hi
<thumper> jelmer: hey
<thumper> jelmer: ISTR that you were working on a bug where the build job would get stuck uploading
<thumper> jelmer: I came across a problem yesterday with a user where recipe builds had been requested that produced the same package version
<thumper> jelmer: and the resulting recipe builds where stuck uploading
<thumper> jelmer: does this sound familiar?
<jelmer> thumper: Yes, I fixed one of the causes of that a couple of weeks ago but there's another that's crept up recently that I investigated today.
<thumper> jelmer: ok, cool
<jelmer> thumper: It appears to be because of a missing database permission so shouldn't be too hard to fix.
<thumper> jelmer: sounds great
<LPCIBot> Project devel build (201): FAILURE in 3 hr 22 min: https://hudson.wedontsleep.org/job/devel/201/
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] Upgrade to r128 of testtools trunk,
<LPCIBot> pre-release version.
<LPCIBot> * Launchpad Patch Queue Manager: [r=mars][ui=none][bug=672519] Change ec2 land command to update the
<LPCIBot> merge proposal commit message attribute with the built commit
<LPCIBot> message with proper QA tags.
#launchpad-dev 2010-11-09
<lifeless> is anyone looking at Person:+commentedbugs ?
<poolie> hi wallyworld_, sorry for the lag, do you still want to talk?
<wallyworld_> poolie: ok
 * thumper stabs TAL in the face
 * thumper is getting confused by PackageBuildDerived
<thumper> wgrant: ping
<thumper> or StevenK ping
<wgrant> thumper: Hi.
<thumper> wgrant: nm
<thumper> I'm walking through build farm job derived
<thumper> gee, could it be more convoluted?
<wgrant> It is a little awful, yes.
<bac> thanks for the QA thumper, i was looking for it to land
<thumper> np
<wgrant> thumper: I've been meaning to excise the obsolete bits, but I should probably study instead.
<wgrant> thumper: In related news, trying to fit inheritance into an RDBMS sucks.
<thumper> wgrant: that it does
<rockstar> thumper, what are you looking at in the build farm jobs?
<wgrant> Pain and suffering, I imagine.
<rockstar> wgrant, that's not what you're looking for usually, but it's what you get.
<thumper> rockstar: yes
 * thumper stares with wonder at the steps he needs to build a test page
<thumper> so... a source package release is one of: a recipe build; or an upload?
<thumper> wgrant: ?
<mwhudson> well, "is the result of" surely?
<mwhudson> maybe not
 * mwhudson lets wgrant answer
<thumper> mwhudson: yeah, that's what I ment
<thumper> the result of a successful recipe build
<thumper> I was a little terse
<rockstar> Oh man, I can totally do the overlay reflowing if we trash our overlay and use Y.Overlay.
<thumper> gah
<thumper> stupid factory
<thumper> rockstar: how about morphing dialogs?
<rockstar> thumper, that's what I meant.
<thumper> rockstar: DO IT!
<rockstar> thumper, it's freakin' easy, assuming that the [beta] tag doesn't mean it's going to change much.
<rockstar> thumper, doing it now.  :)
<rockstar> I'm also trying to listen in on a panel entitled "The Future of Frontend Engineering."
<lifeless> stub: hi ?
<stub> lifeless: yo
<lifeless> I've been meaning to grab some time to talk about db evolution
<lifeless> are the technologies around that can do modest online DDL ?
<lifeless> things like add a table, drop a column ?
<lifeless> I'm wondering about an approach where we do small steps and use things like appserver code / triggers to update an old-format table and new-format table insync,
<wgrant> thumper: Right, a SourcePackageRelease is the result of an upload, whether it be by FTP or recipe.
<lifeless> use the garbo to migrate untouched rows across
<thumper> yep
<lifeless> and once we're fully synced stop writing to the old table, break all its references and discard it
<wgrant> lifeless: Surely we are the not the first people to encounter this problem.
<lifeless> wgrant: yes, but we're using slony
<lifeless> wgrant: single db server is trivial; sharded is trivial.
<stub> lifeless: There is a commercial product that claims to do modest online DDL, but I doubt it would work with replication.
<lifeless> wgrant: the drizzle folk claim nbd can do what we need, but I haven't dug in in detail - and thats not replication either.
<wgrant> lifeless: Ah, fair point.
<stub> lifeless: For simple DDL (add a table, add a new NULLable column, drop a column), we are much better off when we are running Slony 2.x or maybe some other replication. The stuff in PG 9.0 may well be suitable for us soon too (once we ween off using replication to ship data to ISD)
<stub> lifeless: So we are stuck where we are until we upgrade or switch replication.
<lifeless> ok
<lifeless> stub: with RFWTAD roughly done, this is now next up as TA work in kanban
<stub> TA?
<lifeless> technical architecture
<lifeless> I've given myself two slots in kanban; currently performance + continuous deployment
<StevenK> Does test suite runs fall under that, or is that seperate?
<lifeless> StevenK: I don't think it does
<lifeless> StevenK: the kanban limits don't stop me doing spikes and helping folk
<lifeless> StevenK: but they are where I'm focusing the vast bulk of my effort
<StevenK> lifeless: Well, sure, but you were doing some work with parallel tests, and then you just ... stopped :-)
<lifeless> StevenK: I did a spike
<StevenK> Ah
<lifeless> StevenK: so did Ian
<lifeless> stub: could you perhaps have a look at why https://bugs.edge.launchpad.net/malone/+bug/668138 is slow? I think its a 8.4 thing, and it would be nice to be able to tell what the issue is
<_mup_> Bug #668138: Person:+commentedbugs timeouts <dba> <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/668138>
 * thumper stabs soyuz repeatedly
<thumper> probably not entirely fair
<thumper> but it is the closest thing
<thumper> where is the code that schedules recipe builds?
<wgrant> thumper: 'schedules' meaning 'creates'?
<thumper> wgrant: yeah
<thumper> I'm messing around with the factory
<thumper> so I can test my page
<thumper> I created a source package recipe build
<wgrant> LOF, or the SPRB factory?
<thumper> but it has no buildqueue_record
<thumper> so no eta
<thumper> what creates the buildqueue_record?
<wgrant> Try calling sprb.queueBuild()
<wgrant> (this whole thing is scheduled for demolition)
 * thumper sighs
<wgrant> Hmmmmm?
<thumper> I  can't seem to get it to say anything other than "No suitable builders"
<lifeless> ahh LBYL, gotta love it
<lifeless> you'll need an i386 capable builder
<wgrant> i386 virt, no less.
<wgrant> (this is nothing to do with Soyuz; that message is purely a Code invention)
<rockstar> thumper, the tests for source package recipes should be a good template for how to get all this set up (and there's a lot of shit to set up)
<thumper> rockstar: yeah, I'm going through one, but it isn't helping
<thumper> I'm calling factory.makeBuilder()
<rockstar> thumper, do the distros/distroserieses match?
<thumper> rockstar: eh?
<rockstar> thumper, I'm pulling from memory here, but I remember some issues where the builder wasn't set up to build against the distro and/or distroseries the recipe build was for.
<thumper> rockstar: right now it is the processor doesn't match
<rockstar> thumper, ah yes.
<thumper> the buildqueue processor it is 10
<thumper> if I call makeBuilder I get a processor id 1
<thumper> the buildqueue that was created has a factory generated processor family
<thumper> that's why
<thumper> :(
<rockstar> thumper, yeah, was just about to say that.
 * thumper wants to smack this around
<rockstar> thumper, I have some model tests that set up what you want.
<thumper> now the question is what do I need to change?
<thumper> rockstar: can you point me at one plz?
<rockstar> thumper, one sec.
<wgrant> thumper: Create a builder for the processor in that family.
<wgrant> Or unfuck the factory.
<thumper> wgrant: I'd rather make the buildqueue work with the default builder
<thumper> I want to know what is making factory processors
<wgrant> thumper: Probably whatever is constructing a distroseries.
 * thumper thinks he has it
<rockstar> thumper, lib/lp/code/model/tests/test_sourcepackagerecipebuild.py line 62
<wgrant> thumper: Recipes should use the distroseries' nominatedarchindep distroarchseries' processorfamily's processor.
 * thumper need to unfuck the factory
<rockstar> thumper, that's a hell of a spike. Fair warning.
<LPCIBot> Project devel build (202): STILL FAILING in 3 hr 43 min: https://hudson.wedontsleep.org/job/devel/202/
<thumper> \o/
<thumper> finally have a page showing pending build
 * thumper EODs
<lifeless> wgrant: around ?
<lifeless> my improvement to https://bugs.launchpad.net/soyuz/+bug/662523 is on qastaging; I'm trying to find the page to use to reproduce ;)
<_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out <qa-needstesting> <timeout> <Soyuz:Fix Committed by lifeless> <https://launchpad.net/bugs/662523>
<wgrant> lifeless: Archive:+packages might use something like that method.
<wgrant> Otherwise there's the API.
<lifeless> yeah
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (203): FIXED in 3 hr 20 min: https://hudson.wedontsleep.org/job/devel/203/
<LPCIBot> Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=661441] Do checkwatches custom logging to
<LPCIBot> info instead of error to avoid duplicate OOPS reports.
<lifeless> wgrant: do you have an api script thats vaguely relevant here, handy?
<wgrant> lifeless: No. fta has one which uses it a lot.
<jelmer> 'morning
<mrevell> Well hello
<jtv> Why can't we "make" in db-stable?  It complains about a missing module, difftacular.
<jtv> Seems to be a bzr plugin that I don't have.
<henninge> Same for db-devel but not for devel.
<henninge> http://paste.ubuntu.com/528623/
<bigjools> jtv, henninge: utilities/update-sourcecode ?
<henninge> bigjools: dunno about that but I did rocketfuel-get
<henninge> but let me try
<jtv> What did I miss?
<bigjools> I sometimes need to do it separately
<henninge> jtv: bigjools suggested utilities/update-sourcecode
<jtv> But rocketfuel-get does that too, doesn't it?
<jtv> (Which naturally I ran)
<henninge> yes, what I said
<henninge> bigjools: lots of "no change"
<henninge> :(
<bigjools> I suspect that db-devel has got a dependency change that is not in devel
<jtv> I believe we were once supposed to have special sources.list entries for a bzr PPA, but those became unnecessary.
<bigjools> I'll see if I can re-create here
<jtv> bigjools: this is no time to play around with the missus, we need your help!
<bigjools> it. takes. too. damn. long. to. run. make.
<jtv> Hey, I just took a minute off "make schema."  If everyone tried to do the sameâ¦
<bigjools> really?
<henninge> jtv: make build now worked for me in db-devel
<henninge> It may really be related to udpate-sourcode ... strange
<henninge> trying recife now
<bigjools> ok re-created, now I am trying something
<jtv> henninge: I got an update to dulwich, but that must have come in minutes ago because I've been running rocketfuel-get
<bigjools> so
<bigjools> the update-sourcecode in db-devel has a change that is not in devel
<bigjools> "Getting difftacular from lp:difftacular at revision 6"
<bigjools> we've had this issue before
<jtv> Grr
<henninge> But why don't I see anything about it in the output of update-source code.
<wgrant> Hmm?
<wgrant> difftacular is a new dep in db-devel's sourcedeps.conf.
<henninge> and I guess rocketfuel-get uses the one from devel, right?
<jtv> Yes, I guess on the assumption that dependencies would appear there first.
<bigjools> yes
<jtv> Ah great: no module named _gpgme
<jtv> that's progress, right?  :)
<henninge> jtv: recife now builds for me.
<henninge> bigjools, wgrant: thanks
<jtv> I think I probably tried to do too much before all the downloads were finished.
<jtv> thanks bigjools!
<bigjools> :)
<maxb> matsubara: Do you think you could document that recipe build of bzr-pqm into the launchpad PPA you've got on dev.lp.net/LaunchpadPpa, fix it to build for maverick too, and maybe set its ownership to ~launchpad-committers?
<matsubara> maxb, what kind of documentation are you looking for?
<maxb> I guess a link to the recipe would be fine? Just enough for someone to figure out where the package came from and why it is there
<maxb> Ideally enough to ascertain when, at some point in the future, it is no longer necessary to maintain a separate package
<matsubara> maxb, will do
<deryck> Morning, all.
<deryck> mars, ping
<mars> Hi deryck, in the team standup, will ping back in a few minutes
<deryck> mars, ok, thanks
<flacoste> jml: Mumble?
<stub> difftacular? Did we grow a new dependency without buildout or lp packages love?
<stub> ahh... sorted. Needed to relink sourcecode/
<bigjools> stub: you need to use update-sourcecode in db-devel
<mars> Hi deryck, what's up?
<deryck> mars, hi, now I'm on call.  Will ping shortly.
<mars> deryck, heh, ok :)
<deryck> mars, off now
<deryck> mars, so windmill is driving me insane ;)
<mars> ah, I'm sorry to hear that
<deryck> I'm trying to get rockstar's branch to update to latest lazr-js landed....
<deryck> this is also the yui 3.2 upgrade
<deryck> but it's blocking our subscription story work and has been for a few weeks now
<jml> flacoste: ping
<deryck> so I need to get it done
<mars> ok
<mars> deryck, does the lazr-js work depend on the yui 3.2 work?
<mars> as in, they need to be upgraded at the same time?
<jml> flacoste: sudo apt-get install gobby-0.5
<deryck> mars, yes.  the branch updates lp to lazr-js rev 188, which is yui 3.2 plus other fixes.
<deryck> mars, I have a high degree of confidence in the upgrade it self -- windmill tests pass individually and playing with the dev site locally looks good....
<deryck> mars, but the tests will not pass in ec2
<mars> deryck, ok, do you have some sample failures
<mars> ?
<deryck> mars, yes, let me forward you the two recent logs in email
<mars> deryck, I'd like to see if the failures are new, or similar to existing ones
<mars> timing, or DOM
<deryck> mars, they appear to be similar to issues before where the tests collectively suddenly start to fail.  I'm hesistant to call that timing, but it smells the same.
<deryck> mars, email sent with logs and a bit more info.
<flacoste> jml: https://bugs.edge.launchpad.net/launchpad-code/+bug/634466
<_mup_> Bug #634466: bzr branch lp:subvertpy fails as it gets a private branch url. <codehosting-ssh> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/634466>
<mars> deryck, got it, give me a bit to browse them
<deryck> mars, sure.  thanks for the help!
<mars> deryck, ec2 lag between branches - weird.  I fixed the limit to 8 hours during UDS, so that should have been fixed (I hope!)
<deryck> mars, ah, well that would explain it then.  The js branch is just taking longer to complete.  Not sure why.
<mars> deryck, ugh, well, the js-refresh run took 7.6 hours, eh?
<mars> "Total duration of test run 27166.2 seconds"
<deryck> yeah
<mars> and the older- run, someone killed it?
<mars> KeyboardInterrupt
<deryck> mars, I didn't.  It seems to have died on it's own.
<deryck> hmmm
<deryck> I didn't notice that
<deryck> how would I have keyboard interrupted it?
 * mars shrugs
<mars> well, looking at the test run time
<mars> 12357.3 seconds.
<mars> 3.4 hours
<mars> deryck, so the older run, and the newer run - what is the difference between them exactly?
<mars> because there is a big difference in the suite run times
<mars> thanks to subunit, I can probably narrow that down
<deryck> mars, nothing that I know of.  I made no changes.  Same revno of my branch.  the first failed mysteriously, so I immediately fired off the second run.
<deryck> mars, up until this, I was getting long runs but pretty clear errors that I could progressively chip away at.  real failures.  After fixing the last of the real failures, then this mystery.
<mars> deryck, ah, so the keyboard interrupt - that was a headless ec2 run you fired off?  And I presume that you didn't ssh into the server and kill it...
<deryck> mars, right.  Just a normal ec2 land run.  I did nothing differently that I normally do.  Did not ssh in.
<mars> weird
<mars> deryck, did you receive summary mails for these runs? (sorry if you already told me, and I'm forgetting)
<deryck> mars, yes
<deryck> mars, for both.  I just sent you the logs from the mails rather than forwarding the mails.  Would you like the mails?
<mars> deryck, please
<deryck> mars, sent.  and I just noticed something....
<mars> ?
<deryck> mars, in the older KeyboardInterupt one, the error up further is Firefox IO error.  I get this locally everytime when running ./bin/test -cvv --layer=WindmillLayer
<deryck> I just quit running the tests this way, thinking I was doing it wrong.
<mars> deryck, which line, which file?
<mars> ah, the older one
<mars> ack
<deryck> ok
<mars> just read backwards in the older one
<mars> WARNING: A test appears to be hung. There has been no output for 600 seconds.
<mars> deryck, ok, in the older file, which line is the Firefox IO error?  I am looking at line 39084, which is the last output before the hang
<deryck> mars, line 39216
<mars> ah, ok, I get it.  So line 39084 is the last line of normal output.  Everything from there between there to line 39270 was caught in the output buffer.  Now I get it.
<mars> deryck, ok, so I am going to focus on the second run, since it at least completed
<mars> it may be worth comparing the execution times of both
<deryck> mars, ok
<mars> to reach the same point
<sinzui> bigjools, ping
<bigjools> sinzui: yo
<sinzui> bigjools, do we have a function that can compare to debversions and state if one version is newer than the other?
<bigjools> sinzui: of course
<bigjools> just digging up what we use
<bigjools> sinzui: apt_pkg.VersionCompare()
<sinzui> bigjools, I think I neglected to mention that I am actually comparing a spr version to a product release version. I expect to munge something to make this work
<sinzui> thanks!
<bigjools> see lib/lp/archiveuploader/nascentupload.py:_checkVersion()
<bigjools> sinzui: /o\
<bigjools> good luck
<sinzui> I think I will change scope if this is more than two hours of work
<james_w> sinzui, it should be simple with python-debian
<james_w> from debian.changelog import Version
<james_w> product_version = Version(<product-version> + "-1"); package_version = Version(<package-version>)
<james_w> print Version(product_version.upstream_version) > Version(package_version.upstream_version)
<sinzui> james_w, thank you very much.
<james_w> it looks odd, but I think that's the behaviour that you want
<sinzui> I want to tell packagers looking at a source package page that upstream has a new version, or if Ubuntu is ahead of upstream (presumably Lp is missing the a registered upstream release)
<flacoste> bigjools: mumble?
<bigjools> flacoste: yep
 * jelmer_ wonders why jml is naming branches after Greek islands
<jml> jelmer_: The Book of Revelation (aka "The Apocalypse") was written on the island of Patmos.
<mars> deryck, ping, I might have a test you can try
<deryck> mars, ok, cool.  What's up?
<mars> deryck, hmm - I noticed the YUI unittest runs were failing
<mars> deryck, lib/lp/testing/__init__.py:741]
<mars> deryck, the waits.forElement on that line has no time limit - is it an infinite wait?
<deryck> mars, which test?
<mars> deryck, three of them I think - lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest
<mars> lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest
<mars> lp.code.windmill.tests.test_yuitests.CodeYUIUnitTestCase.runTest
<mars> lp.bugs.windmill.tests.test_yuitests.BugsYUIUnitTestCase.runTest
<deryck> mars, ok, I see what you mean now.  I have no idea why there's isn't a time there.
<deryck> I didn't know that was possible actually.
<mars> so that is the immediate cause of three of the failures.  deryck, those YUIUnitTestCase classes pass locally?
<deryck> mars, yes, they do.  with ./bin/test -cvv -m bugs -t test_yuitests (for example)
<mars> deryck, and have you considered running just these testing in ec2?  "ec2 test -o '-vv --layer=FooWindmillLayer'"
<mars> ok
<deryck> mars, no, hadn't thought of that.  I can try that.
<sinzui> mars, you can have an indefinite wait in forElement. I did not know that and I certainly hope I did not write those lines of code
<mars> deryck, run it with --post-mortem.  If they fail, you can ssh in and run them with 'bin/test -D'
<mars> sinzui, well, deryck's tests are failing, so the wait does not appear to be inifinite
<mars> (inifinite?)
<deryck> mars, ok, I'll try some additional runs with these suggestions and see what I turn up.
<deryck> mars, thanks for your help!
<mars> that would tease out the problem with the YUI parts
<mars> looking at the others
<mars> deryck, np
<mars> deryck, also, both tests take roughly the same time to execute: up to 15 minutes
<mars> errm
<mars> deryck, rather, one took 2hours 55 minutes, the other took 3 hours 11 minutes.  That implies that they would have both run up to a full seven hour run.  Too long.
<deryck> mars, by "one" and "other" do you mean just the yuitests took that long?
<mars> deryck, sorry, the two subunit streams you gave me
<deryck> ah, right
<mars> here, I have a run from Nov. 3rd, I'll compare to that
<mars> 12591 seconds - so 3.5 hours for the full run
<mars> 2 hours and 15 minutes to finish the test_me_too bugs windmill test
<mars> deryck, so your branch has taken an average of 48 minutes more than mine to execute up to the test_me_too windmill test (assuming execution order is the same)
<deryck> ok, that's interesting
<deryck> I wonder why?
<mars> deryck, well, your and my test runs took 2:41 and 2:47 respectively to reach the first windmill test
<mars> deryck, could be networking - if your branch is choking on network timeouts or waits.forWhatever(), then that would take the time right out
<mars> 'could not resolve host' takes a while
<mars> but that should not be happening... hmm
<mars> deryck, your test run also has a lot of thread trash - I fixed that exact bug before, where the threads were from the testrunner trying to erroneously access an external web service provider
<mars> sorry, the windmill test proxy server was trying to do the access on behalf of the page
<mars> those connections timed out, and left thread trash
<mars> deryck, hi, back?
<deryck> hi mars, yes sorry.  Connection issues.
<mars> deryck, how much of the last bit did you see?
<deryck> mars, I saw mars> deryck, well, your and my test runs took 2:41 and 2:47 respectively to reach the first windmill test
<deryck> and that's it
<mars> deryck, ok, so I say your test run had a lot of thread trash - I fixed that exact error with the tests in the past
<deryck> ok
<mars> deryck, IIRC it means the page is trying to access web resources outside of the test server
<deryck> hmmm
<deryck> mars, ok, I'm poking at this a bit more now, trying to work it out.
<mars> deryck, I am going to run a test with ec2, see if I can track down the off-site connections.
<mars> (if that is what is happening)
<deryck> mars, ok, thanks!
<deryck> mars, would the hard coded ports cause any issue here?  (thinking of wallyworld's email recently about not having this anymore to get ready for parallel runs)
<deryck> well, couple weeks ago recently :-)
<mars> deryck, when was this branch last updated?
<mars> the ports should be transparent
<deryck> mars, merged devel daily, even yesterday.  but the ports are still there in the tests.
<mars> oh?
<deryck> mars, yeah.  but they're also there in devel, too.
<mars> was wallyworld's work in preparation for removing ports, or did it actually start to remove them?
<deryck> mars, I don't see the ports in db-devel.  So maybe this is completely inrequired for devel at this point.
<mars> deryck, ok, so you are saying the windmill test harness has drifted between devel and db-devel?
<bigjools> jml: would you have a moment to help me diagnose that latest problem in the buildd-manager please?
<deryck> mars, I don't know what I'm saying :-)  I'm asking. :-)  I am saying devel and db-devel are not the same for me.  db-devel does not have hard coded ports, devel does.
<deryck> mars, I don't know where wallyworld did his work.
<mars> checking the list for that
<jml> bigjools: I have half an hour.
<bigjools> jml: oddly, so do I
<bigjools> jml: thanks.  The traceback is in here: https://bugs.launchpad.net/soyuz/+bug/671242
<_mup_> Bug #671242: New buildd-manager disabling everything in sight <Soyuz:Triaged by julian-edwards> <https://launchpad.net/bugs/671242>
<jml> f'n wrapping
<bigjools> yes
<bigjools> someone should fix that :)
<deryck> mars, it was merged to db-devel.  so shouldn't be merged to devel at this point.
<mars> deryck, cool, so that should not factor into this
<deryck> mars, right.  Sorry, that was just noise.  Just trying to cover all bases.
<jml> bigjools: http://paste.ubuntu.com/528801/
<bigjools> jml: it's failing inside rescueBuilderIfLost() which is  bascically the first thing that scan() does
<mars> deryck, no problem, you just reduced the problem space.  That's a good thing :)
<deryck> cool :-)
<bigjools> jml: at some point the builder object goes wonky as wgrant seems to suggest
<bigjools> the behavioural code that it's using is not consistent with the fact that it's not got a current job
<mars> deryck, I need to take lunch, should be back in about an hour
<deryck> mars, np.  thanks, again!
<jml> bigjools: hmm.
<bigjools> jml: given that this is the very first piece of code that's run after it re-fetches the builder object in scan(), I am stumped.
<jml> bigjools: is that a given?
<jml> bigjools: I'm still paging in the cdoe
<jml> bigjools:         buildqueue = getUtility(IBuildQueueSet).getByJob(self.job)
<Ursinha> bigjools, hi :) do you know what are all those PoolFileOverwriteError oopses about?
<bigjools> Ursinha: not really :/  I need to investigate but there's about 100 other critical things to fix first :(
<Ursinha> bigjools, :/
<Ursinha> I see
<Ursinha> do you want me to file a bug about it?
<bigjools> Ursinha: there might be one already, but yes please
<Ursinha> bigjools, I couldn't find one, I'll try  a bit more and file one if not succeed
<bigjools> thank you
<bigjools> good night everyone
<bdmurray> I should be testing bug 666496 on qastaging right?
<_mup_> Bug #666496: findSimilarBugs() returns bug task you are using <qa-needstesting> <trivial> <Launchpad Bugs:Fix Committed by brian-murray> <https://launchpad.net/bugs/666496>
<lifeless> Ursinha: ping
<Ursinha> hi lifeless, I saw db-stable reports looks odd
<lifeless> cool
<Ursinha> I'll investigate that after finishing my ongoing task
<lifeless> thanks
<lifeless> jelmer_: ping
<lifeless> Ursinha: Could you bring back the count at the top ?
<lifeless> Ursinha: Its nice that we're showing all the revs, but the count was a great visual summary.
<Ursinha> the count?
<Ursinha> which count?
<lifeless> oh, its still there.
<lifeless> EMORNINGBRAIN
<Ursinha> hehe
 * Ursinha hands lifeless a cup of strong coffee
<lifeless> I hadn't quite scrolled to the top
<lifeless> jelmer_: I'm going to land a reversion of 11859 this evening if we can't get it QA'd in advance. I'll land one that makes the cowboy permanent.
<lifeless> jelmer_: we've been blocked for 3 days on that revision.
<lifeless> deryck: ping
<deryck> hi lifeless
<lifeless> got time for a voice chat? I want to talk bugtask:+index performance
<deryck> lifeless, can we do tomorrow?  I've only got a couple hours left and lots to do.  Heads down on lazr-js that has to get landed to unblock gmb.
<deryck> or end of day for me, even
<lifeless> deryck: I'm away tomorrow, friday
<deryck> ah right
<lifeless> deryck: we can cap it - 10 minutes
<lifeless> its just a little intricate because of comment imports
<deryck> lifeless, give me 30 minutes or so to finish what I'm doing, if that works for you.
<lifeless> great, thanks
<deryck> np
<jml> g'night all
<lifeless> night jm
<lifeless> l
<lifeless> thumper: we're definitely overriding merge proposal diffs from time to time - https://code.launchpad.net/~jelmer/launchpad/637507-subprocess-sigpipe
<lifeless> oh man, thats a branch.
<lifeless> I really do have morning brain.
<lifeless> Where is the hot water....
<deryck> lifeless, want to do a call now?
<lifeless> \o/
<lifeless> jelmer_: ping
<jelmer_> lifeless, hi
<lifeless> jelmer_: any progress on rev 11859 ?
<lifeless> jelmer_: we can deploy up to 11888 if 11859 won't break anything in nodowntime.
<jelmer_> lifeless: I discussed it with bigjools before he EODed. I'm going to double check it doesn't break anything on dogfood and then mark it qa-ok.
<lifeless> jelmer_: I know its past your EOD, but this is now critical
<jelmer_> lifeless: That's why I'm here.
<lifeless> \o/
<lifeless> can I help ?
<jelmer_> lifeless: Not for the moment, but I'll let you know if there is anything.
<lifeless> kk
<jelmer_> lifeless: I tried to qa on qastaging with Chex' help yesterday, but we couldn't find the right things to check.
<lifeless> jelmer_: once you have qa'd it please document somewhere the problem you ran into
<lifeless> jelmer_: most folk don't have dogfood access, so its imperative that we sort these qastaging issues out
<thumper> lifeless: what do you mean overriding merge proposal diffs?
<deryck> mars, ping
<jelmer_> lifeless: do you have a preferred place for that to be documented?
<jelmer_> lifeless: anyway, things are still working on dogfood so I've tagged qa-ok
<jelmer_> lifeless: it's really EOD for me now, I'll send a note about what's missing on qastaging tomorrow by email; if something else (wiki page?) is better, please drop me a note on irc.
<lifeless> jelmer_: thank you
<lifeless> jelmer_: uhm, an RT to get it fixed (if its a problem) is best
<lifeless> jelmer_: if its not a problem but just docs, write it up somewhere on dev.l.n
<deryck> mars, I think I found the fix for this branch.  *think* :-)
<mars> Hi deryck
<mars> deryck, cool, what did you find?
<mars> deryck, not much on this end - I have a full log of all HTTP traffic to sift through, and a reproducible failure, but it doesn't match what you were getting.
<deryck> mars, so remember when we chatted about 404s for CSS calls a couple days ago when not in devmode....
<mars> right
<deryck> mars, the tests were trying to fetch css for yui's CDN without fetchCSS: false.  a sed for all YUI() calls to include that, and tests are passing locally when run collectively now
<mars> deryck, did setting that in our global YUI_CONFIG value not fix all callsites?
<deryck> mars, I didn't know we add a global YUI_CONFIG.
<deryck> had, rather
<deryck> I don't think anywhere is using it anyway.  Wouldn't it be YUI(YUI_CONFIG).use()?
<flacoste> gary_poster: mumble?
<flacoste> gary_poster: or skype?
<mars> deryck, yes, I would think you would have seen it.  Maybe... part of the LPS sandbox code?
<gary_poster> flacoste, either.  your preference?
<mars> deryck, actually, that is a good question: why do we have YUI().use everywhere, when we also have the LPS variable defined for the very purpose of a global site-wide JS control pooint?
<flacoste> gary_poster: let's try mumble, i'm in foundations one-on-one
<gary_poster> ok
<deryck> mars, right.  That's what I did was add fetchCSS: false to the global LPS object, but there are a few places where we make a new one.
<mars> deryck, grep turned up 21 callsites.  Would it be too much trouble to change them?
<mars> deryck, grep -r 'YUI(' lib/lp/* | grep -v '/tests/'
<deryck> mars, I can change them all.
<mars> 24 if you include lib/canonical/launchpad/javascript
<deryck> mars, change them all to use LPS?
<mars> deryck, yes
<lifeless> rockstar: oh hiai
<deryck> mars, ok.  Do you think there's ever a legitimate reason to instantiate a new YUI?
<wallyworld> abentley: thumper: standup?
<mars> deryck, directly?  No, I think it would cause more problems than it solves.  We decided on the approach of 'one site-wide JS baseline'
<mars> deryck, and instantiating your own YUI is basically ignoring that baseline
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<deryck> mars, ok.
<mars> deryck, that causes problems because our JS use-case has special requirements - SSL being the big one.
<deryck> mars, well it can't be causing too many problems. :-)  It's never been fixed correctly it seems.
<mars> heh, true
<deryck> mars, it's cool, I can fix it.  Was just more wondering if anyone of the 24 sites would have been intentional, rather than accidental.
<mars> deryck, I think they were just hold-overs.  They 'just worked' because the JS they needed was on the page, and they did not require sandbox configuration parameters (until now)
<deryck> right
<LPCIBot> Project devel build (206): FAILURE in 3 hr 42 min: https://hudson.wedontsleep.org/job/devel/206/
<LPCIBot> Launchpad Patch Queue Manager: [r=abentley][ui=none][bug=672476] Always create a .htaccess file when
<LPCIBot> publishing a private PPA.
<thumper> abentley: Error in test lp.codehosting.scanner.tests.test_bzrsync.TestGenerateIncrementalDiffJob.test_create_on_new_revision
<thumper> abentley: https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/383/steps/shell_6/logs/summary
<deryck> mars, I need to EOD, but I'll do the conversions and send this off to ec2 again and see what happens.  Thanks for all your help today!
<mars> deryck, no problem.  Good luck, let me know how it went tomorrow
<deryck> will do
<deryck> g'night, all.
<cody-somerville> ummm...
<wgrant> Uhoh.
<cody-somerville> yea
<cody-somerville> I dunno exactly whats wrong but something is very wrong.
<cody-somerville> I get 'KeyError: 'is_edge'<br /> ' on a lot of pages.
<cody-somerville> directly adding edge subdomain seems to fix it
<cody-somerville> https://launchpad.net/oem-archive <-- this oopses
<cody-somerville> https://edge.launchpad.net/oem-archive <-- this works
<wgrant> Eep.
<wgrant> lifeless: ^^
<wgrant> It's approximately universal.
<wgrant> Even for non-betatesters.
<lifeless> yes, fixing
<lifeless> cody-somerville: fixed
<lifeless> thanks for letting us know
#launchpad-dev 2010-11-10
<lifeless> StevenK: hi
<StevenK> lifeless: Hello
<lifeless> bigjools asked me to chat with you about derivation logic
<lifeless> and proffer my modest assistance if needed
<lifeless> wgrant: are you around perchance?
<lifeless> james_w: hi
<james_w> hi lifeless
<lifeless> I was wondering if there are test helprs for populating ppas already?
<james_w> lifeless, SoyuzTestPublisher
<james_w> it's not very pleasant though
<james_w> the factory has some stuff too
<lifeless> I'm working on https://bugs.edge.launchpad.net/soyuz/+bug/672371
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/672371>
<lifeless> I want a query scaling test
<james_w> but you don't want "put 200 packages in this PPA"?
<lifeless> so I need to put some packages (in different staes I guess) into a test ppa
<lifeless> and I don't care about publishing an archive at all
<james_w> archive = self.factory.makeArchive(purpose=ArchivePurpose.PPA)
<james_w> for i in range(3):
<james_w>     self.factory.makeSourcePackagePublishingHistory(archive=archive)
<james_w> something like that will give you the db gunk, and not do any actual publishing or anything
<lifeless> sweet, giving it a spin now.
<james_w> bigjools has hinted in the past that using the factory for soyuz is a bad idea, but I don't really understand why
<lifeless> the factory is a bit monolithing
<lifeless> s/ing/ic/
<lifeless> I'd like to split into different fixtures
<lifeless> ZOMG
<lifeless> Difference: queries do not match: 25 != 38
<lifeless> adding one package.
<lifeless> thanks!
<james_w> np
<james_w> you can obviously change statuses with that
<james_w> but you might want to also add a scaling test for binary publications too
<james_w> though that might affect the main PPA page rather than +packages
<lifeless> Its like playing whackamole with dyanmite
<james_w> I think there's a makeBinaryPackagePublishingHistory or something too
<lifeless> as long as we plug each hole, its an upwards trend.
<lifeless> my first focus is unfucking +packages today
<james_w> right, but as you are there a test for binaries not scaling badly on +packages is probably worth 10 minutes
<lifeless> james_w: the problem isn't the test
<james_w> if it passes you can leave it to catch problems in the future, if it doesn't you can decide whether to fix it, or just defer
<lifeless> james_w: its the fix.
<lifeless> james_w: I appreciate the moral support, but I'm optimising this for landing, nothing else at this point.
<james_w> ok
<lifeless> james_w: wgrant: Is there any existing class that models 'set of SourcePackageRelease objects' ?
<james_w> lifeless, not that I know of, but that doesn't mean there isn't
<lifeless> \o/ flat query counts on +packages
<nigelb> whoa 6500 bugs..... O_O
<lifeless> heh
<StevenK> lifeless: Right, so I'm looking at this graphing -- looks like the changelog is easily grapable from the librarian without having to pull apart the source package to get it
<lifeless> whats the processing model for this
<lifeless> is it per-distroseriesdiff-package
<lifeless> or are you doing many packages at once ?
 * StevenK peers at the db-devel failure
<StevenK> lifeless: It will be per spr
<StevenK> It's been a while since I've played with the librarian, so I'm trying to recall how to get content out of a LFA
<lifeless> StevenK: look at the merge proposal code
<StevenK> lifeless: How does that help?
<nigelb> ouch, http://www.linuxjournal.com/content/ubuntu-bug-reporting
<wgrant> lifeless: There isn't.
<lifeless> wgrant: ECONTEXT
<lifeless> StevenK: the merge proposal code gets the diff from the librarian in-appserver, then marks it up
<wgrant> lifeless: There is no SPRSet.
<wgrant> Or anything like it.
<wgrant> StevenK: What are you graphing changelogs for?
<lifeless> wgrant: thanks
<StevenK> wgrant: Find the base version between Ubuntu and Debian
<wgrant> Right.
<wgrant> So we will need to fix gina and run my script of dogfood-murder.
<StevenK> If we want to pull the changelogs directly from the librarian, and I think we do
<lifeless> well
<wgrant> Unless you want to be unpacking source packages.
<lifeless> unpacking sp's is nuts
<StevenK> Source unpack just to graph versioning makes me cry
<lifeless> putting this in the db would be better than using the librarian, unless you *know* that you'll only hit 4-5 versions on most packages.
<lifeless> (because the db runs in-ram, the librarian doth not)
<StevenK> We can't say for absolutely certainy
<wgrant> The DB is crazy, though, as changelogs can be many megabytes.
<wgrant> I imagine we should only need to look at the latest changelog.
<wgrant> Unless someone's been doing Bad Things.
<StevenK> No, I can forsee us stepping back a few revisions
<wgrant> Howso?
<StevenK> Picture foo 1.4-1 in Debian, and foo 1.4-1ubuntu8 in Ubuntu
<wgrant> StevenK: The 1.4-1ubuntu8 changelog will have 1.4-1 in it.
<wgrant> Done.
<wgrant> One librarian read.
<lifeless> wgrant: we don't need the whole cl
<lifeless> wgrant: just the graph pointer to the next version that has a cl prefix
<wgrant> lifeless: We in theory have something like that with changelog_entry.
<wgrant> But that relies on people using -v properly.
<wgrant> Which they don't.
<wgrant> In the normal case we should need to read one SPR's changelog from the librarian.
<wgrant> I think.
<wgrant> Is that really bad?
<lifeless> well
<lifeless> its going to be diskio most of the time
<lifeless> probably tolerable
<lifeless> nigelb: thanks for pointing that out, I've replied on it
<nigelb> lifeless: \o/
<nigelb> Its nice to tell folks that people are working on the timeouts
<wgrant> OTOH it's not really nice that it was left to rot for years :)
<lifeless> well
<lifeless> I think the coding style being used was a big driver
<wgrant> Oh yes, certainly.
<lifeless> without that being identified *and changed* it wouldn't matter how much one tried, it wouldn't be fast.
<stub> test_bzrsync.py has exploded on db-devel putting us in testfix.
<wgrant> StevenK: Still around?
<wgrant> Ah, nevermind. gina is just confusing.
<adeuring> good morning
<stub> I can't see any particular change that would have caused the bzrsync test failure.
<lifeless> sweet - http://morepypy.blogspot.com/2010/11/snake-which-bites-its-tail-pypy-jitting.html
<mrevell> Hola
<bigjools> hey
<allenap> Morning.
<wgrant> bigjools: Is there a significant penalty in not caching the build behaviour?
<bigjools> wgrant: given the queries it's running, I expect so yes
<wgrant> :(
<bigjools> well, take a look for yourself ...
<wgrant> Can we use the existing cachedproperty stuff which has proper invalidate hooks?
<bigjools> that's exactly what I was thinking
<wgrant> That was all fixed recently to not suck.
<bigjools> I'm going to go over it with noodles
<wgrant> A good plan.
<bigjools> wgrant: actually it's not *quite* the same as a cached property
<bigjools> or is it ...
<wgrant> bigjools: Ideally anything that changed the current job would invalidate it.
<wgrant> But that sounds hard.
<wgrant> But in the new code we probably invalidate the object lots anyway.
<bigjools> wgrant: see _setCurrentBuildBehavior which is for that purpose
<bigjools> wgrant: so my theory is that although the storm properties of self.builder are getting refreshed, the other ones are not
<bigjools> which makes no sense since self.builder is re-assigned
<bigjools> but as Sherlock Holmes said ...
<wgrant> I forget exactly how Storm invalidation works.
<bigjools> it uses pixie dust
<jml> what's up with the test failure?
<deryck> Morning, all.
<bigjools> morning deryck
<bigjools> jml, I re-created the b-m problem in a test \o/
<jml> bigjools: yay
<bigjools> it's utterly bizarre though
<jml> bigjools: using actual APIs or by poking the db?
<bigjools> re-fetching the builder object doesn't reset the behaviour
<bigjools> actual API
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: test failure on db-devel | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<bigjools> jml: it's caused by another builder building the job and then when it destroys the buildqueue, the scan of the old builder goes bang
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: test failure on db-devel (since Nov 9, 1612UTC) | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> bigjools: interesting
<jml> bigjools: so is the problem that another builder builds the job, or that the old builder doesn't reliably handle another builder doing so?
<bigjools> jml: it could be one of 2 things:
<bigjools> 1. the job on the old builder is not getting reset properly
<bigjools> 2. the pseudo-cached current_build_behavior property is not getting reset on the builder object
<bigjools> I am highly suspicious of the latter, but now I have a test case I can work it out
<jml> bigjools: cool.
<bigjools> I can just switch it to a @cachedproperty since lifeless fixed that
<bigjools> jml: right, it's the latter.  Re-assigning the builder seems to retain a private property.  WTF?
<wgrant> wallyworld: Have you seen DecoratedResultSet? Does it not fulfill your needs?
<wallyworld> wgrant: i haven't come across that class yet. thanks for the pointer
<wallyworld> i'm clearly still getting across the full api
<wgrant> wallyworld: It's not actually in Storm core yet, AFAIK.
<wgrant> wallyworld: It's in the Launchpad tree somewhere.
<wgrant> lib/canonical/launchpad/components/decoratedresultset.py
<wallyworld> wgrant: cool thanks. i'll take a look. it perhaps should be moved to storm i guess
<wgrant> Indeed.
<lifeless> jelmer_: bigjools: My branch has a test failure - I failed to delete the check that the ArchivePublication delegate can return a false list of builds, which should be irrelevant now that its given a complete answer rather than injecting a cheap build callback.
<lifeless> I've forwarded the failure from ec2 to jelmer
 * lifeless goes back to sleep
<lifeless> (and I'm on leave tomorrow/friday)
<jml> bigjools: I'm not sure what that means
<bigjools> jml: me neither, I'm still debugging
<wgrant> bigjools: The object will just be reretrieved from Storm's cache.
<bigjools> I'm trying to work out if the current_build_behavior property is retained or recalculated
<jelmer_> lifeless: Thanks, mine is still in progress
<bigjools> wgrant, jml:
<bigjools> removeSecurityProxy(getUtility(IBuilderSet)[builder.name])._current_build_behavior
<bigjools> <lp.soyuz.model.binarypackagebuildbehavior.BinaryPackageBuildBehavior object at 0xd388650>
<bigjools> I guess storm is returning not only the cached data but the actual object I already have :/
 * bigjools plays with cache invalidation
<jml> bigjools: are you sure the thing returned by getUtility(IBuilderSet)[builder.name] is a different object?
<jml> (i.e. id(foo) != id(bar))
 * bigjools is checking
<bigjools> jml: it's the exact same object
<bigjools> even if I invalidate the cache it returns the same object
<jml> how do you invalidate the cache?
<bigjools> Store.of(builder).invalidate()
<jml> hmm. I wonder what that is supposed to do.
<bigjools> I guess storm is jsut refreshing the object it already has for performance reasons
<bigjools> so I should be able to fix this by invalidating that cached property
 * bigjools haxors
<jml>         This prevents Storm from returning the cached object without
<jml>         first verifying that the object is still available in the
<jml>         database.
<bigjools> it also re-reads the object from the db
<jml> well, it might, but that's not what the docs say
<bigjools> they're incomplete then
<bigjools> but then this is storm docs ...
<jml> right, looking at it the implementation it calls _mark_autoreload, which (at a guess) tells storm to reload column values when they are next asked for.
<jml> but there's absolutely no way storm could even know about user-maintained state
<bigjools> yup
<bigjools> but the end user has no way of knowing that storm is going to return you the same python object that it already knows about
<bigjools> so naive code like this will break
<jml> hmm.
<bigjools> now, how do I define an invalidation hook for storm
<jml> I was just asking that
<jml> self._run_hook(obj_info, "__storm_invalidated__")
<jml> that's in _mark_autoreload
<jml> bigjools: ^
<bigjools> just found it :)
<jml> how did this code work at all previously?
<bigjools> nfi
<bigjools> really, nfi
<jml> I guess we're out of testfix now.
 * jml pours the first bowl of wrath onto an unsuspecting code base
<bigjools> hurray, my test passes now
<jml> crap. apparently I have to re-authorize my ec2 app
<jml> leonardr: what does it mean when I have multiple applications with the same name & permission but different dates on https://launchpad.net/~jml/+oauth-tokens?
<leonardr> jml: it probably means you cleared out your .launchpadlib directory and your client forgot about the earlier token and had to create a new one
<jml> leonardr: ok, that makes sense. should Launchpad clean up old tokens?
<jml> I guess we don't know for sure they are now unused
<jml> or, different computers or what have you
<leonardr> jml: we've kicked around ideas, but none of them have been airtight enough to implement
<leonardr> once we release the code for desktop integration, the situation should improve significantly
<leonardr> since there will be fewer tokens
<jml> leonardr: ahh yes.
<bigjools> jml: I think my branch might benefit from a quick review, would you care to do the honours please?
<jml> bigjools: sure.
<jml> didrocks: hi
<jml> didrocks: I just landed a branch that removes the need to sign the CoC before uploading to a PPA
<didrocks> jml: hey! awesome, that will make everything so much easier! :)
<jml> didrocks: I know you don't have much time to work on quickly (unity unity unity unity), but I thought you'd like to know :)
<didrocks> jml: yeah, that's kind of you to hilight that :) (and you are correct about your assumption :))
<didrocks> thanks a lot :)
<jml> every fricking time I need to do something with datetime I need to look up the API docs
<jelmer_> jml: perhaps you should add another module for dealing with times and dates to Python ? :-P
<jml> good idea!
<jml> but I'll add it straight to the stdlib without actually using it in live code first
<jml> because standardizing on well-tested, commonly-used code is unpythonic
<jelmer_> ah, I see you've got experience in this area
<jml> ding ding ding!
<deryck> gmb, question about bug 485627, if you have a sec
<_mup_> Bug #485627: Prompt user to subscribe when marking a bug as duplicate <email> <qa-ok> <story-better-bug-notification> <ubuntu-qa> <Launchpad Bugs:Fix Released by gmb> <https://launchpad.net/bugs/485627>
<gmb> deryck: Shoot.
<deryck> gmb, do we now *not* subscribe a user to the master bug via the duplicate?
<mars> sinzui, ping
<gmb> deryck: Well, you're still showed as subscribed via dupes in the UI, but I don't think you get notification about the master bug unless you subscribe to it directly any more.
<sinzui> hi mars
<mars> Hi sinzui.  I was hoping you could help me with an account registration question
<mars> sinzui, I just created a second account, using my work address, for some work I am doing.  I used the login/register link to create the account - it seems to have gone fine
<deryck> gmb, ah, ok.  So we have some cleanup around that then.  I'm thinking of the bug from jam about not having a direct link any longer.
<gmb> deryck: Yes, I think there's some cleanup needed.
<mars> sinzui, I just received a mail that says "If this was you, perhaps you've forgotten your password?", otherwise "If not, someone may be trying to impersonate you.".  I have *not* received my mail with my confirmation code and/or link.
<deryck> gmb, I think we should drop the subscriber from the web ui, preserve the old text about "the dupe is at LINK", and then add the new additional notice:  "You are not subscribed, if you want to, go to LINK"
<sinzui> mars, are you saying that U SSO did not know about the other address, and Launchpad did not either, so Lp created a profile when U SSO said you logged in?
<sinzui> mars, okay,
<mars> sinzui, I hit "Log out" on Launchpad before doing this
<sinzui> mars, U SSO know a lot of email addresses. I believe it knows every email address that Lp knows via replication
<gmb> deryck: That would also mean that we don't have to hit the API for subs-from-duplicates for each bug page, which would be nice.
<sinzui> so you did not register
<deryck> gmb, would it?  We would still need to show current and past subscribers from dupes, right?
<mars> sinzui, ok.  I am worried, because from my perspective as a user, Launchpad just slapped me for trying to register a second account using my existing address.
<deryck> gmb, or are we not sending duplicate subscribers emails at all any more?
<sinzui> mars, you have not talked to Lp!
<gmb> deryck: OICWYM. What's a past subscriber from a dupe? Someone who was subscribed via a dupe but now isn't?
<sinzui> mars, as I keep saying, registration, login, and reset are U SSO. those pages and emails are not Lp
<gmb> deryck: Honestly, I'm not acutally sure now. Maybe we are but we're just sending it with the "Subscribe to the master bug" footer.
<mars> sinzui, ok, but I used the green Launchpad login screens on login.launchpad.net - again, from my perspective, I have only been working with Launchpad.
<sinzui> mars, yes, that is why I keep send emails to fix this. You have not and when users go directly there to register with lp, they are *not* redirected to real Lp to login...the profile is not created
<deryck> gmb, ok, so I think we need to find out for sure.  and then do cleanup.  Let's use bug 673203 to track this.
<_mup_> Bug #673203: Duplicate email sends link to +subscribe <email> <regression> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/673203>
<gmb> Ok
<deryck> gmb, since I'm stuck on lazr-js still, would you mind including that in your "bugs I'm working on until deryck is done" list? :)
<gmb> deryck: Sure, no worries. Can you add a card for it to the bugs lane? I'll take a look once I'm done with this blasted test that I'm trying to write.
<mars> sinzui, ok.  I am sorry to hear the experience has those problems, and glad to hear that you know what is happening.  I was just looking for some help finding a way to register my account, not to accuse.
<deryck> gmb, sure, thanks!.  on the bugs backlog, or the task lane for the story?  I'm not picky either way myself.
<sinzui> mars, visit U SSO and login as old identity, view the email address that you cannot use. logout. register with an unknown email address.
<gmb> deryck: Task lane would make more sense, actually
<deryck> gmb, ok will add it now
<gmb> cool
 * gmb -> afk to run an errand; bbiab
<mars> sinzui, ok, I'll do that.  Thank you for your help.
<jml> bigjools: you know that thing with twisted xmlrpc that you had to monkey patch?
<jml> bigjools: I just landed the fix upstream
<bigjools> jml: \o/
<bigjools> I forgot what it was patching now
<jml> the xmlrpc client
<jml> to make sure it disconnects properly
<bigjools> oh and hugs for fixing glob imports!
<bigjools> that's the badger
<jml> bigjools: ta, but I really just put the cherry on top of a cake made by others. sinzui's been working at it for months
<bigjools> my heroes
<deryck> gmb, allenap -- checkwatches hasn't run for two days according to the errors emails.  Should we be concerned?
<flacoste> jml: i'm on mumble
<allenap> deryck: Yeah, probably.
<allenap> deryck: Can you direct me to what you're looking at?
<deryck> allenap, do you get emails from script-failures?
<deryck> allenap, I fwd'ed you a mail.  Do you mind chasing it up with a losa?
<allenap> deryck: I do get them, but I mark them as read immediately! I keep them only for reference (one which I forgot I had until now).
<allenap> deryck: I'll chase it down.
<deryck> allenap, thanks, I appreciate it.
<gary_poster> sinzui, you probably saw my request for a reviewing mentor for benji.  Someone from Registry would be my first choice, if you or EdwinGrubbs are willing and able.  Is that a possibility?
<sinzui> gary_poster, I will be mentoring jcsackett next week. I will ask EdwinGrubbs. He is at a conference this week
<EdwinGrubbs> gary_poster: I can start on it after the openstack conference.
<gary_poster> EdwinGrubbs, thank you!
<gary_poster> and thank you sinzui
<sinzui> EdwinGrubbs, I just reviewed your branch.
<EdwinGrubbs> sinzui: thanks
<gary_poster> I'll reply to my email requesting a mentor.  Thanks, I really appreciate it, EdwinGrubbs.
<bac> gary_poster, EdwinGrubbs: hurrah.  thanks edwin.
<gary_poster> :-)
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (207): FIXED in 3 hr 43 min: https://hudson.wedontsleep.org/job/devel/207/
<rockstar> deryck, hi
<lifeless> jelmer: hi, I'm really not here, but how is it going ?
<deryck> hi rockstar
<rockstar> deryck, how goes the lazr-js battle?
<deryck> I hope you have windmill magic in your pocket
<rockstar> deryck, :(
<deryck> It's playing in ec2 again.
<lifeless> flacoste: do you know?
<deryck> rockstar, I think one of the issues was without fetchCSS: false used globally, yui was attempting to go offsite for CSS, this 404s, the test runner got into thread contention, and tests started failing.
<rockstar> deryck, ah, yeah, I just ran into that here.
<rockstar> deryck, I think this was a change in 2.2.
<deryck> 3.2?
<rockstar> Yeah, that's what I meant.
<rockstar> (I'm tired...)
<deryck> no worries :-)
<deryck> rockstar, how is yui conf?  Fun and useful, I hope.
<rockstar> deryck, very much so.  I wish we could have had more people go.  I went to a workshop yesterday that gave me the desire to delete most of lazr-js.
<rockstar> deryck, I'll be typing up a thorough report on the way home.
<deryck> yeah, I would have loved to have gone.
<deryck> we've learned a lot as we've gone along, but yui devs view js the crockford way.  Would be nice to absorb some understanding of that more fully.
<flacoste> lifeless: last i saw was the tag qa-ok being applied to the bugs
<rockstar> deryck, funny you should say that... I'm sitting next to the Crock right now...
<rockstar> (He's wearing purple sneakers)
<deryck> all the cool js hackers do
<rockstar> Well, at least the Yahoos
<jam> Chex: ping, for great justice! (question about lp-forking-server and getting testools upgraded on bzr's pqm)
<lifeless> flacoste: ok, so we haven't deployed it
<flacoste> lifeless: i don't think so
<flacoste> lifeless: not according to LPS
<lifeless> https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html doesn't show a fix
<lifeless> and its not in pqm
<lifeless> and there's no qa-ok on https://bugs.launchpad.net/soyuz/+bug/672371
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <regression> <timeout> <Soyuz:Triaged by lifeless> <https://launchpad.net/bugs/672371>
<lifeless> flacoste: I promised Lynne I'd really have a day off today.... are you able to figure out where its up to and either deploy 11887 (take the faulty rev off) or get the fix 'out there' ?
<flacoste> lifeless: 11887?
<lifeless> 11888 is faulty
<lifeless> I started a thread on lp-dev about it
<lifeless> its making ppa/+packages pages timeout
<flacoste> lifeless: is it deployed?
<lifeless> yes
<flacoste> lifeless: ok, i'll handle this
<lifeless> I'm checking devel in case buildbot is currently munching on it
<flacoste> lifeless: do you have any scripts to prepare for a nodowntime roll-out?
<lifeless> flacoste: I've sent a followup mail just now.
<flacoste> lifeless: adding bug numbers, moving stuff to Fix released, etc.?
<lifeless> flacoste: none at all; I've filed bugs for the bits I think are reasonable to automate.
<flacoste> ok, so it's all manual for now
<flacoste> thanks
<flacoste> lifeless: enjoy your day off
<lifeless> thanks
<lifeless> the thread name is 'possible regression on bug 672371'
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <regression> <timeout> <Soyuz:Triaged by lifeless> <https://launchpad.net/bugs/672371>
<flacoste> jelmer: around?
<lifeless> flacoste: I'll forward you the test result too
* flacoste changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<LPCIBot> Project devel build (208): FAILURE in 3 hr 16 min: https://hudson.wedontsleep.org/job/devel/208/
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none][bug=671242] Ensure that the custom
<LPCIBot> _current_build_behaviour property on Builder is invalidated
<LPCIBot> when Storm invalidates the database values. Fixes the
<LPCIBot> buildd-manager so that it doesn't disable builders erroneously
<LPCIBot> due to code tracebacks.
<LPCIBot> * Launchpad Patch Queue Manager: [r=gmb][ui=none][bug=667340] Add a new 'fixverified' status mapping
<LPCIBot> for Trac external bug trackers. Maps to 'Fix Released'.
<LPCIBot> * Launchpad Patch Queue Manager: [r=mars][ui=none][bug=621090] fix content type on two JSON views
<LPCIBot> * Launchpad Patch Queue Manager: [r=mwhudson][ui=none][bug=384831,
<LPCIBot> 539496] Do not re-export anything from
<LPCIBot> lib/canonical/launchpad/interfaces/__init__.py. Explicitly
<LPCIBot> register webservice code.
<LPCIBot> * Launchpad Patch Queue Manager: [r=mars][ui=none][bug=673015] Allow people to upload to PPAs without
<LPCIBot> having previously signed the code of conduct.
<deryck> bye, all.
<jml> thumper: ping
<thumper> jml: hi
<thumper> jml: I normally have the standup around now, but wallyworld doesn't seem to be here yet
<jml> thumper: that's ok, I need a few minutes anyway
<wallyworld> thumper: i'm here now :-)
<jml> anyone mind if I fix the test failures in devel?
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: devel failing, jml fixing | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<thumper> jml: no... go ahead
<thumper> abentley: around?
<abentley> thumper: hi.
<thumper> abentley: mumble
<lifeless> flacoste: no, I hadn't fixed the test - it was 0130 when I found that it was failing
<flacoste> lifeless: it's in PQM right now
<lifeless> flacoste: thanks
<abentley> thumper: BTW, I think we should switch the build delay stats to past-six-hours rather than past-24-hours.  I.e. data since last sample.
<rockstar> Heh.  I just ran into Mikael in the Yahoo cafeteria.  He says Windmill is dead.
<thumper> heh
<thumper> hmm
<thumper> rockstar: was he a windmill author?
<rockstar> thumper, he was one of them.  He doesn't do anything with windmill anymore.
<jml> :( :(
<thumper> rockstar: what did he suggest to use?
<rockstar> Aaron is apparently still taking patches, but isn't maintaining it.
<thumper> jml: mumble?
<jml> thumper: sure
<rockstar> thumper, I quote "Everything we complained about in Selenium is basically fixed now."
<rockstar> thumper, and YUITest now has a Selenium shim in it as well.
<rockstar> Although YETI was also brought up
<rockstar> thumper, I may as well show everyone what I've got now...
<rockstar> So, we currently have this pretty-overlay widget that's kinda messy but all our overlays inherit from it.  It looks like this: http://people.canonical.com/~rockstar/pretty-overlay.png
<rockstar> In ~20 lines of plugin js and some custom CSS, I've made the standard Y.Overlay do the exact same thing.  It looks like this: http://people.canonical.com/~rockstar/overlay.png
<rockstar> The latter loads from cold cache uncompressed 5x than the pretty overlay does completely compressed.
<rockstar> Er, 4-5x faster.
<jml> thumper: https://bugs.launchpad.net/launchpad-code/+bugs?field.searchtext=&orderby=-importance&field.status:list=NEW&field.status:list=INCOMPLETE_WITH_RESPONSE&field.status:list=INCOMPLETE_WITHOUT_RESPONSE&field.status:list=CONFIRMED&field.status:list=TRIAGED&field.importance:list=HIGH&field.importance:list=UNDECIDED&field.tag=recipe&
<jml> thumper: e.g. http://paste.ubuntu.com/529601/
<jml> thumper: https://code.launchpad.net/~testtools-dev/testtools/trunk/+new-recipe
<mars> rockstar, so it sounds like you are saying, it has been two years since we started this, time for a technology refresh?
<mars> rockstar, because everything we started with has been fixed (lazr to make up for gaps in YUI core) or shaken out (windmill v selenium)
<rockstar> mars, no, more like "we over engineered it, and now it's time to simplify it"
<mars> rockstar, that too :)
<rockstar> mars, the overlay code is a good example of us not really knowing what the capabilities of YUI were.
<rockstar> mars, my goal is to shrink the maintenance footprint of our code.
<rockstar> (because we don't have a lot of resources to maintain javascript)
<rockstar> mars, did you see my previous statement about windmill?
<mars> rockstar, I did.  It does not surprise me.
<rockstar> mars, yeah, he was talking about the difficulties in doing event driven stuff in python.
<mars> rockstar, I think you are right, going by what you are saying, we should have sent more people.  We are new at this: we get a lot out of it compared to, say, going to PyCon.
<mars> PyCon is cool, but what you get out of the conference changes as you gain more experience with the language, tools, and community
<rockstar> mars, yeah, maybe.  I don't think there's much interest in writing javascript on the team though.
<mars> rockstar, I was actually thinking of people from other projects, too.  They use YUI too.  (I'm surprised Sidnei isn't there)
<rockstar> Some of that might be that javascript (and YUI in particular) can be REALLY confusing.  :)
<rockstar> mars, sidnei has new twins.  I think he's tied up somewhere.
<mars> oh, yeah, I forgot.  (not that I would know /anything/ about that little life change...)
<jml> rockstar: event-driven stuff in Python is easy and fun!
<jml> Python is so awesome for events
<rockstar> jml, not when you're tied to an existing event loop (firefox, in this instance)
<jml> rockstar: well, that's not doing event-driven code in Python, surely
<jml> rockstar: otherwise, I don't see where the difficulty is. it's easy to write code that responds to events from external systems, regardless of how they're written
<rockstar> jml, yeah, there's definitely some bias there.
<rockstar> Er, I mean in the case of saying python isn't good for event driven development.
<jml> rockstar: next time someone says that, point them to #twisted :)
<mars> rockstar, well, I'm looking forward to seeing what comes out of this.
<rockstar> jml, yeah, yeah.  :)  Basically, he was telling me what I wanted to hear (windmill is dead) so I smiled and nodded to everything else he said.
<mars> rockstar, when can the rest of us expect your trip report? :)
<rockstar> mars, I'm going to type it up tonight after Douglas Crockford's verbal abuse.
<mars> closing keynote?
<rockstar> mars, I think it's just a Bayjax meeting that coincides with the end of YUIConf.
<jml> <lp.buildmaster.model.buildfarmjobbehavior.IdleBuildBehavior object at 0x88a8750> is not an instance of <class 'lp.buildmaster.model.buildfarmjobbehavior.IdleBuildBehavior'>
<jml> what am I missing? that seems nonsensical?
<rockstar> jml, instance v. class?
<mwhudson> jml: security proxies?
<jml> mwhudson: sterling concept
 * jml tries
<wgrant> But security proxies are meant to preserve isinstance, aren't they/
<wgrant> Er, instanceof.
<mwhudson> wgrant: not instanceof -- isn't that js?
<jml> wgrant: nope, zope has special thingummies
<wgrant> mwhudson: Uh, yes. Been dealing with too much Java at uni lately.
<jml> http://pastebin.ubuntu.com/529617/  â can someone please eyeball that testfix?
<jml> mwhudson: it was indeed rSP
<mwhudson> argh, my brain is reading that as "the register that contains the stack pointer"
<mwhudson> jml: wow, that assertIdentical looks optimistic
<mwhudson> jml: looks fine, i was vaguely under the impression that we had an assertIsInstance that did that already?
<jml> mwhudson: we do, but trial base class
<mwhudson> ah ok
<jml> mwhudson: I'm fixing that after I fix circular stuff :)
<mwhudson> r=me then, if you like
<jml> mwhudson: thanks.
 * jml submits
 * jml waits for pqm
<thumper> jml: I'm looking at the db-devel failure
<jml> thumper: thanks :)
<thumper> it seems intermittant
<jml> thumper: yeah, that's my guess. I didn't get it doing a few spins locally.
<thumper> I have
<jml> thumper: but I can't find the race in the code
<jml> thumper: cool. you've made more progress than me.
<jml> thumper: also, I can't see anything that's changed recently there. (but I had a pretty shallow & quick poke through the vcs log, so I probably missed something)
<thumper> hmm
<thumper> it seems the latest testtools is a bit tempermental
<thumper> I ran a test with -D to break on failure
<thumper> but it broke into the debugger on success
<jml> thumper: I don't know of anything that's changed in testtools along those lines
<jml> bigjools: hi
<jml> bigjools: don't worry about the test failure
<jml> bigjools: http://pastebin.ubuntu.com/529617/
<jml> bigjools: I've got a fix churning in PQM right now
 * jml continues to sit around, waiting for PQM to churn.
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: devel failing, jml fixing; db-devel failing, thumper fixing | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> jml: hi
<jml> lifeless: hello
<lifeless> I've pushed a big chunk of the testrepository test helpers into fixtures
<lifeless> and I've some matchers to push into testtools
<lifeless> what do you think of making testtools depend on fixtures
<jml> lifeless: I'm not exactly against it, although doesn't fixtures depend on testtools?
<lifeless> for its tests only
<jml> lifeless: what do you think of merging them?
<lifeless> the only thing that makes me hesitate is that fixtures are useful outside of tests
<lifeless> like we might move matchers out of testtools eventually
<jml> I can imagine the first with less stretching than it takes to imagine the second
<lifeless> heh
<jml> lifeless: in any case, I'm vaguely positive toward the idea, but I'd need to sleep on it
<jml> lifeless: could I persuade you to push a new release of testrepository into debian sometime soon?
<lifeless> thats what I'm heading towards
<lifeless> fixtures is in NEW
<jml> lifeless: yay
<jml> lifeless: I've made a ~testing-cabal team w/ a PPA. I want to be building dailies of stuff into that
<lifeless> yes
<jml> lifeless: unfortunately, I suck hard at packaging and haven't had a spare moment to actually fix them to not suck.
<lifeless> you might want to add that team to the recipes beta
<lifeless> so that I can see it :)
<jml> lifeless: you should be able to see it. launchpad-beta is in recipes beta
<lifeless> kk
 * jml ec2 submits database-apocalypse
<lifeless> ah there it is
<lifeless> \o/ success
<lifeless> jml: so, we can nuke edge now ?
<jml> lifeless: you tell me :)
<jml> lifeless: you don't need edge to get at recipes
<jml> that's all I know
<wgrant> So all we need edge for is API scripts for the next 4.5 years :D
<wgrant> Alternatively we could hunt down edge API scripts using the stats which probably don't exist.
<mwhudson> a redirect in the frontend doesn't sound like much burden
<mwhudson> (assuming scripts will work with that)
<wgrant> It won't work.
<wgrant> At least I don't think it will.
<wgrant> The WADL has edge stuff hardcoded.
<wgrant> Unless you translate the outgoing WADL too...
<jml> or
<jml> you could just leave api.edge as is, put a redirect up for all the other subdomains and bring the edge machine(s) into the lpnet pool
<wgrant> True.
<poolie> wallyworld: is bug 636930 now fixed in launchpad, or at least inprogress?
<wgrant> As well as looking for any authenticated edge API requests and severely chastising those users.
<_mup_> Bug #636930: Upgrading a repository fails with 'Inter1and2Helper' object has no attribute 'source_repo' <bazaar> <sru> <upgrade> <verification-done> <Bazaar:Fix Released by spiv> <Bazaar 2.2:Fix Released> <Launchpad Bazaar Integration:Triaged> <bzr (Ubuntu):Fix Released> <bzr (Ubuntu Maverick):Fix Released> <https://launchpad.net/bugs/636930>
<jml> silly conflict :(
 * jml runs ec2 land again
<poolie> jml, i think my dkim branch just needs to be submitted
<poolie> it may fail in ec2
<jml> poolie: cool.
<wallyworld> poolie: you mean upgrading lp to use bzr 2.2.1?
<jml> poolie: I was meaning to ask you
<poolie> would you be so kind as to send it for me, and i'll in parallel run the tests here and see what happens
<jml> poolie: I mean, in real time in addition to my earlier askinations
<jml> poolie: will do. you'll be CCd the results of the run
<spiv> 'bzr ping lp:bzr' still reports 2.2.0, so I don't think 636930 is fix released yet.
<wallyworld> poolie: spiv: i have tried for several days to land the @%@! fix but we seem to be permanently in textfix mode :-(
<wallyworld> ie my lp-land keeps getting rejected
<jml> poolie: conflicts
<jml> poolie: also, commit message
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: db-devel failing, thumper fixing | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<poolie> jml, ok, it's on my queue to fix them, test, and then i'll let you know
<jml> poolie: thanks. If I hear from you, I'll land it tomorrow
<lifeless> mwhudson: they won't
<lifeless> mwhudson: I filed a bug on foundations if you want the gory details
<lifeless> my current basic plan is - shrink edge as fast as we can
<lifeless> keep looking at how to fold the machines into the one cluster
#launchpad-dev 2010-11-11
<wgrant> lifeless: Do we know how many clients are using edge still?
<jml> fourth time lucky
<lifeless> wgrant: no, though if we slightly-unfold the edge ppr we could tell total hits
<jml> for Christmas, I would like tighter feedback loops
<wgrant> I'm getting tighter feedback loops for Christmas.
<jml> lucky :(
<jml> sixth time, perhaps
<thumper> jml: I think I've found the problem
<jml> thumper: what is it?
<jml> (I've spent 45 minutes waiting for computers to do things)
<thumper> factory.getUniqueDate() :(
<thumper> it adds a getUniqueInteger to an epoch
<thumper> the initial counter is intialized randomly
<thumper> the problem occurs when the bmp is created after NOW
<thumper> HORRIBLE
<jml> thumper: ugh, well spotted
<thumper> well debugged you mean
<jml> thumper: indeed I do.
<thumper> cause it wasn't initially spottable
<jml> thumper: I feel this is an opportune moment to point out that lp/testing/tests/test_factory.py exists.
<thumper> yes, but the factory is doing the right thing
<thumper> the test isn't
<thumper> it doesn't need a random date
<thumper> it needs to make sure the date is in the past
<jml> ahh ok
<jml> yay successful detach
 * jml flees
<jml> (maybe database-apocalypse will pass tests and land while we aren't in testfix mode and then pass on the buildbot and get pulled into stable before I wake up)
<wgrant> ROFL
<poolie> lifeless: mkanat tells me codebrowse is now behind a loadbalancer, with multiple instances?
<poolie> that's great
<thumper> testfix submitted
 * thumper is grumpy that it took two hours
<thumper> wallyworld: wanna chat?
<wallyworld> thumper: give me 5 mins
<thumper> ack
<poolie> thumper: wallyworld: anyone, are we now running multiple codebrowse processes?
<thumper> poolie: I believe so
<poolie> so that should help with stability and performance?
<poolie> do you want to nominate any other bugs for max to do next?
 * thumper thinks
<thumper> we will likely have something to do with bzr-history later
<thumper> but we were wanting some work done on the LP side first
<thumper> poolie: how about the default file render to show the text at that revision rather than an annotated view?
<wallyworld> thumper: mumble?
<lifeless> poolie: yes it is, part of the nodowntime effort
<lifeless> poolie: each loggerhead instance is still internally threaded
<poolie> excellent
<lifeless> poolie: I'm not really here today - am on a swap day
<poolie> i was asking max to push it but if it's already done that's great
<lifeless> poolie: single-threading the backends still needs to be done
<lifeless> but its not something I'm directly pushing for loggerhead [yet] because there are still some easier bigger fish to fry
<LPCIBot> Project devel build (209): STILL FAILING in 3 hr 40 min: https://hudson.wedontsleep.org/job/devel/209/
<thumper> lifeless: aren't you on leave?
<lifeless> yes
<lifeless> but IRC highlights still highlight
<poolie> so my suggestions to him were: make a stable branch; turn off annotate by default; allow http caching; profile cpu and memory for serving single pages
<wallyworld> poolie: spiv: the bzr upgrade just got merged in as rev 11902
<poolie> yay!
<spiv> wallyworld: hurrah!
<spiv> wallyworld: meanwhile, mkanat on #bzr just bumped into bug 668176, so you'll be wanting to do a bzr upgrade again fairly soon ;)
<wallyworld> sorry about the delay. the code was ready days ago. it took a while to find a window where we weren't in textfix mode :-(
<wallyworld> spiv: np. i assume you mean 2.2.2. ping me on the 18th when it is released :-)
<spiv> wallyworld: well, I personally don't care if it's 2.2.2 final or just tip of lp:bzr/2.2 or just 2.2.1 with that one tiny fix backported.   But yes :)
<wallyworld> spiv: 2.2.2 final is only a week away. can it wait till then?
<lifeless> wallyworld: remember that we have a scheduled downtime our thursday
<lifeless> wallyworld: so waiting a week means waiting a month
<lifeless> wallyworld: or scheduling extra downtime
 * lifeless runs away
<wallyworld> spiv: assuming 2.2.2 final codebase is complete by mid next week, i could take tip of lp:bzr2.2 and do it before the downtime?
<lifeless> wallyworld: you can't
<lifeless> wallyworld: we select the revision early in the week
<lifeless> wallyworld: trying to slide things in at the last minute is a guaranteed nightmare.
<lifeless> wallyworld: remember you need time to qa on qastaging/staging, deal with testfix, possible test failures trying to land it.
<wallyworld> lifeless: ok. i'll do it tomorrow. now fcuk off and enjoy your day off
<lifeless> lol ok
<lifeless> I guess what I'm trying to say is 'either do it now, or decide to (wait a month | schedule extra downtime)
<lifeless> ciao
<thumper> WHY?
 * thumper looks to the sky
<ajmitch> thumper: it's clearing up now, just a short bit of rain :)
 * thumper is referring to soyuz factory generated crap that won't render
<ajmitch> heh :)
 * thumper sighs some more
<lifeless> StevenK: please tell me you didn't land something without using ec2land ?
 * thumper just saw the failure
<thumper> I thought it was a general failure so forced it, but on reading the error it looks like StevenK
 * thumper pulls trunk
<lifeless> StevenK: (of course, it could be the known race condition with ec2 - and I am very curious which it is)
<thumper> lifeless: I get the same error trying to make devel
 * thumper testfixes
<thumper> done
 * thumper EODs
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (210): FIXED in 3 hr 16 min: https://hudson.wedontsleep.org/job/devel/210/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][r=mwhudson] Upcall correctly,
<LPCIBot> actually do the assertion in the test, fix the assertion.
<LPCIBot> Project devel build (211): FAILURE in 23 min: https://hudson.wedontsleep.org/job/devel/211/
<LPCIBot> Project devel build (212): STILL FAILING in 1 min 49 sec: https://hudson.wedontsleep.org/job/devel/212/
<Maha2> Hi frnds anybody hav black money?
<wgrant> Wow.
<wgrant> c.l.i and c.l.d are empty.
<lifeless> things are moving
<wgrant> Aren't you not here?
<lifeless> I'm hacking on testtools
<lifeless> lp:~lifeless/testtools/matchers if you're interested
<lifeless> personal time
<adeuring> good morning
<LPCIBot> Project devel build (213): STILL FAILING in 3 hr 39 min: https://hudson.wedontsleep.org/job/devel/213/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][rs=thumper] Fix the PPACreationError import in
<LPCIBot> lib/lp/registry/interfaces/webservice.py.
<mrevell> Hey up
<bigjools> helleau
<henninge> bigjools: It's "Hellau!" actually but not for another 50 minutes or so...
<bigjools> henninge: it's actually whatever I want it to be :)
<henninge> ;)
<henninge> Carneval season starts at 11:11 today in some parts of Germany
<bigjools> what's that?
<wgrant> bigjools: Hi.
<bigjools> hi
<wgrant> bigjools: https://launchpad.net/ubuntu/jaunty/+source/xserver-xorg-video-geode/2.11.10-1~jaunty1
<bigjools> :/
<henninge> Just to correct myself, it's "Helau"
<henninge> lifeless: Hi, still around?
<wgrant> bigjools: Someone accidentally uploaded it (was meant for a PPA)... and the Release pocket check is only for CURRENT/SUPPORTED, not OBSOLETE.
<bigjools> wgrant: how did that get there?
<bigjools> ah
<bigjools> crepe
<wgrant> bigjools: I think we should probably reject all uploads to an obsolete series (it would fix a PPA-related bug too).
<wgrant> But that needs to be checked with people.
<bigjools> wgrant: we can't do that - PPAs need to remain active
<bigjools> but obsolete distros, totally
<wgrant> bigjools: Hmm, why?
<wgrant> In the past the virtualisation flag has been turned off upon obsolescence.
<wgrant> Although not yet for Jaunty, it seems.
<bigjools> because some crazy people want to carry on using them
<bigjools> so I'd rather it was a switch than a lock :)
<wgrant> :(
<wgrant> I guess we'd best just extend the current/supported check to obsolete.
<bigjools> yeah
<wgrant> I had a look at the upload checking code.
<wgrant> Those pockets are hardcoded in a few places.
<wgrant> :(
<wgrant> Also, the archive permission API is.......
<bigjools> this surprises you ...
<bigjools> well if you dislike the API then feel free to be constructive
<wgrant> Yeah, I had a look at redesigning it yesterday.
<wgrant> (I need to change it to allow pockets for queue admins)
<wgrant> bigjools: There's just one thing blocking my lucilleconfig extermination: germanium's publisher's temp directory. The existing code uses the distro's lucilleconfig root (/srv/launchpad.net/ubuntu-archive/ubuntu-temp), which seems more than mildly insane for PPAs. The new code will use archivepublisher.root, which isn't set by germanium's config, so it will be /var/tmp/archive/ubuntu-temp.
<wgrant> Easy fix is to set archivepublisher.root in the ppa config.
<wgrant> But I wonder if we want the temp directory to be in a less insane place on germanium anyway.
<bigjools> I can't remember what it uses the temp dir for
<wgrant> It writes out Packages and Sources there, so it can atomically rename them into the real archive.
<wgrant> So it needs to be on the same FS.
<bigjools> right
<bigjools> /var/tmp is not useful then
<wgrant> No.
<wgrant> So it needs to be fixed.
<wgrant> But I think it should be fixed to something other than a lie.
<wgrant> ('ubuntu-archive' is the lie)
<lifeless> jml: ping
<lifeless> henninge: no, not around at all.
<lifeless> whats up?
<henninge> lifeless: two questions
<henninge> lifeless: what's the state of bug 672371?
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <regression> <timeout> <Soyuz:Triaged by lifeless> <https://launchpad.net/bugs/672371>
<henninge> lifeless: Are you working on that?
<lifeless> no, I'm on leave
<lifeless> the mailing list thread covers the current status
<henninge> right, ok
<lifeless> which is that a fix is landed, pending qa on qastaging, and deployments are blocked until the fix is qaed (and everything leading up to it)
<jml> lifeless: hi
<lifeless> jml: you might like https://code.launchpad.net/~lifeless/testtools/matchers/+merge/40606
<jml> lifeless: not as much as I like: "Test results: database-apocalypse => devel: SUCCESS"
<jml> lifeless: but I'll take a look
<henninge> lifeless: thanks
<henninge> second question: I see that a lot of revisions have landed on stable which have not been processed by the QA bot.
<henninge> no qa tags, no fix committed.
<henninge> Is the bot on hold because of the blockage?
<henninge> Or could it be broken or just slow?
<henninge> lifeless: ^
<jml> lifeless: actually, looking, I need coffee first. won't be a jiffy
<lifeless> henninge: no idea
<henninge> who could I ask about the bot?
<lifeless> henninge: jml/stub/flacoste/gary/urshina/mars/diogo/me
<lifeless> except I'm not here.
<henninge> ;-)
<lifeless> its in a service account that those folk (maybe more) have access too
<henninge> lifeless: thanks a lot, enjoy your leave ;)
<lifeless> de nada
<stub> losa, urshina or diogo I think. I don't know where or how the bot works.
<lifeless> right
<lifeless> access doesn't imply knowledge
<lifeless> stub: lookin the lpqateam crontab if you want to chase pointers.
<lifeless> henninge: qastaging is down
<lifeless> henninge: that will break the bot
<henninge> Oh!
<lifeless> I don't know why its down - on leave + no losa : not going to dig in myself ;)
<henninge> No, I'll have to wait for Chex .... :(
<henninge> thanks again
<bigjools> henninge: ask a GSA
<bigjools> it's too important to be left down
<henninge> on #is
<henninge> ?
<bigjools> yes
<henninge> will do
<thumper> \o/
<thumper> my testfixes worked
 * thumper heads to bed
<matsubara> hi henninge
<matsubara> do you still need help with the qa bot?
<henninge> matsubara: Hi!
<henninge> I already found out that qastaging is running its update script atm.
<henninge> Would that stop the qa bot?
<lifeless> henninge: its been down for hours
<lifeless> henninge: whatever its doing is going to fail
<lifeless> and yes it will interfere
<lifeless> the bot reads the active revno from the qastaging and staging web uis
<henninge> IS just told me that the qastaging-update script started only 21 minutes ago ...
<lifeless> yes
<lifeless> Nonetheless, its been down for hours
<matsubara> henninge, lifeless: yes, that's it: https://devpad.canonical.com/~lpqateam/qa_reports/logs/output-launchpad-stable.log
<henninge> lifeless: is "it" the bot or qastaging?
<lifeless> qastaging
<matsubara> sorry, still waking up. lifeless is right. qastaging is down then the bot can't do its job
<lifeless> henninge: jml: I've mailed the list the fragment from the backtrace ng dug up.
<jml> lifeless: I saw.
<lifeless> henninge: could you perhaps file an RT that that log location isn't documented/visible to us?
<henninge> lifeless: yes saw that. thanks
<jml> lifeless: has the problem been fixed? do you need help fixing it?
<henninge> lifeless: good idea
<lifeless> henninge: we should have knowledge and visibility to diagnose things like this
<lifeless> jml: I'm on leave.
<henninge> jml: I will fix it
<henninge> and come back to you if I need help ;)
<jml> henninge: thanks
<jml> henninge: it should be a simple matter of importing things from the right place
<henninge> yes, I expect
<lifeless> jml: so what did you think of the testtools patch ?
<jml> henninge: my worry is that lpnet has a similar zcml file
<lifeless> jml: I'm sure henninge will sed the entire production-configs ;)
<lifeless> there will be 40 or so files to change
<henninge> easy!
<henninge> ;)
<henninge> jml: no, I think we are safe. No reference to canonical.launchpad.interfaces in the production configs.
<jml> henninge: other than the qastaging reference, surely?
<lifeless> huh, perhaps its entirely in tree as it is
<henninge> jml: not even there
<henninge> lifeless: that's what I think
<jml> henninge: so where's the broken code?
<henninge> guys, give me time to figure it out ... ;)
<LPCIBot> Project db-devel build (132): FAILURE in 4 hr 2 min: https://hudson.wedontsleep.org/job/db-devel/132/
<henninge> jml: in ./scripts.zcml http://paste.ubuntu.com/529912/
<henninge> So, where are those files maintained?
<jml> lifeless: replied
<jml> lifeless: sorry it took so long. multitasking is ... SQUIRREL! ... what was I saying, oh yeah, multitasking is hard.
<lifeless> :)
<jml> lifeless: basically, two things
<jml> lifeless: why not 'Raises(FooError)'
<jml> lifeless: and the other is "%s/raises_value/raises_value_error/g"
<lifeless> jml: separation of concerns
<lifeless> jml: we could have a helper that curries the two together
<lifeless> MatchesException is trivially adaptable for use on Failures
<lifeless> Raises isn't
<jml> lifeless: I'm happy to have MatchesException be separate
<jml> lifeless: I just reckon Raises could construct one itself
<jml> lifeless: or, as you say, we could have a third class / helper that curries them. If so, I'd argue that the helper ought to be called Raises, since that's the prime real estate and almost all of the time everyone will want to be asserting that something raises an exception.
<jml> rather than, say, an exc_info tuple that is less than something else
<jml> gasp!
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: stable->db-devel conflict, jml fixing; qastaging busted, henninge fixing | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> jml: I have mixed feelings
<lifeless> jml: I don't particularly like argument inspection
<lifeless> jml: type inspection I mean
<lifeless> jml: but we could add it
<jml> lifeless: you mean, have the instructor take a matcher or an exception class?
<jml> constructor
<lifeless> yeah
<lifeless> taking a matcher is an important use case
<jml> lifeless: got an example?
<lifeless> in test_runtest
<lifeless>  Raises(MatchesException(Exception("foo"))))
<lifeless> its checking that the actual exception is passed on rather than some arbitrary exception
<lifeless> a small improvement that was easy to do here
<jml> well, that's not really passing in a matcher
<jml> you could, in theory, take whatever is passed to Raises() and pass that to MatchesException.
<lifeless> you could
<lifeless> if you want to bind the two together
<jml> well, it's not symmetric. MatchesException doesn't lose its independence.
<henninge> lifeless, jml: It seems to be an untracked file outside our tree and outside the configs. I'll have to talk to the losas to find out if it's really not maintined anywhere.
<lifeless> that makes it harder to do exact checks (vs subclass) without changing the core
<henninge> I'll see that I get it patched now, though.
<lifeless> henninge: thank you
<jml> lifeless: so, rather than type checking, I would rather, def __init__(self, exc_type=None, matcher=None): style shenanigans
<jml> lifeless: I would rather a helper than that.
<lifeless> jml: I think we're only seeing the thin edge of the wedge; once we open up things we'll start to see more sophisticated checks and usage.
<jml> lifeless: I don't :)
<lifeless> jml: We can add a helper (say 'raises') later, I don't feel one is really needed right now.
<spiv> lifeless: WHUI (yet)
<lifeless> spiv: that also applied to getDetails, but its taken off (e.g. see soupmatchers)
<spiv> I do strongly believe in making known common use cases convenient.
<spiv> But I also believe in sleeping :)
<jml> lifeless: it's not enough to block the landing, I'll grant, since we can add one later. But it is needed right now, because there's already stacks of Raises(MatchesException(FooError)) in testtools
<jml> spiv: :)
<spiv> So I'll look forward to reading the results of this discussion some time later.
<lifeless> jml: tell you what
<lifeless> jml: I've done the other variable rename and pushed while we chatted
<jml> lifeless: yay
<lifeless> jml: i leave it to you to land, in whatever shape you think best ;)
<jml> lifeless: did you do the MANUAL change?
<jml> I forgot to mention in IRC
<lifeless> jml: no, lp hadn't forwarded your mail
<jml> np
<jml> lifeless: anyway, it's a deal.
<lifeless> cool. Have a good day.
<lifeless> gnight jml, spiv, et al
<henninge> qastaging is back!
<jml> henninge: cool.
<jml> henninge: what was the problem?
<henninge> There is a zcml file out side the tree that referenced canonical.launchpad.interfaces.
<henninge> I still don't know where it is maintained (it should be!) but Ng patched it for me.
<henninge> jml: looks like this: https://pastebin.canonical.com/39629/
<jml> henninge: ok, makes sense
<henninge> jml: I think it should be put into the production configs branch.
<jml> henninge: that sounds sensible to me.
<jml> henninge: perhaps follow this up with a losa?
<henninge> that's my plan as soon as Chex wakes up. ;)
<jml> henninge: good good :)
<jml> ahh yes, conflict
<jml> I had forgotten in the dizzy excitement of email processing.
<jml> can someone help me QA https://bugs.launchpad.net/soyuz/+bug/673015 ?
<_mup_> Bug #673015: Code of Conduct requirement for PPA upload rights is unnecessary <ppa> <qa-needstesting> <Soyuz:Fix Committed by jml> <https://launchpad.net/bugs/673015>
<bigjools> jml: what sort of help do you need?
<jml> bigjools: well, I guess I need to try uploading without the CoC signed
<deryck> Morning, all.
<bigjools> jml: ok I can make that very easy for you
<jml> bigjools: of course, I've already signed it, so I need to figure what to do there
<bigjools> jml: make a new user
<jml> bigjools: and I don't really know how to upload to a PPA :)
<bigjools> and put your gpg key on it
<jml> bigjools: ok.
<bigjools> jml: the other option is that I hack your account on DF to remove the CoC :)
<jml> hmm
<jml> so I can't really test this on qastaging?
<bigjools> no
<jml> has to be DF?
<bigjools> yes
<jml> harumph
<jml> bigjools: do I have to do something special to get the right version of code onto DF?
<bigjools> jml: make a new user on DF - its emails go to the staging inbox
<bigjools> jml: yes - ask me :)
<jelmer> bigjools: fwiw I updated dogfood this morning
<jml> oh heavens
<bigjools> it runs db-devel, what revno was it in?
<bigjools> jml: I noticed
<bigjools> jelmer: I noticed
<wgrant> (you can also deactivate a CoC sig)
<jml> I'd forgotten about staging inboxes
<jml> bigjools: I'm not sure if it's made its way to db-devel. checking now.
<bigjools> jml: I can merge a devel revno if you want
<jml> bigjools: r11894 of stable. Not on db-devel yet.
<jml> will be once my conflict resolution branch gets through PQM.
<bigjools> jml: ok just ping me when it's ready then and I'll update DF's code
<jml> bigjools: will do. thanks.
<bigjools> can you believe that the existing code that downloads builder files has no code to deal with errors ...
<jml> readily
<jml> I really need to make an inventory tracker thingy for Launchpad.
<jml> jelmer: hi
<jml> jelmer: did you ever get around to testing opening 'o' and 'p' series?
<jelmer> jml: hellÃ¸
<jelmer> jml: No, but I do that on dogfood now or after you've QA'ed your change.
<jml> jelmer: super. thanks.
<jelmer> jml: Are you QA'ing now?
<jml> jelmer: no, waiting for db-devel to have the new goodness to test
<jml> jelmer: so you could go ahead right now :)
<jelmer> ok, great
<Ursinha-afk> henninge, hi
<Ursinha-afk> which bugs?
<henninge> Ursinha: Hi!
<henninge> which bugs?
<henninge> :-P
<bigjools> Ursinha: hi, I want to request a feature on the qa bot so that I can set something as previously QAed before it lands.
<jml> weird. just got a gpg validation rejection from pqm
<bigjools> oh - nobody told you?
<Ursinha> henninge, that weren't processed by the qa-tagger :)
<henninge> Ursinha: Oh, those ;)
<Ursinha> bigjools, I'd ask you to file a bug and I'll triage it :)
<bigjools> Ursinha: no worries - my next question was where to file it? :)
<Ursinha> bigjools, https://launchpad.net/qa-tagger
<bigjools> cheers
<Ursinha> :)
<jml> bigjools: nobody told me. Have I been sacked?
<bigjools> :)
<bigjools> Ursinha: done!
<henninge> stub: Hey!
<henninge> stub: I am trying to reproduce and OOPS I am getting from the poimport script.
<Ursinha> bigjools, thanks :)
<henninge> stub: but I don't see any. Could it be that they get ignored locally?
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: stable->db-devel conflict, jml fixing | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (214): FIXED in 3 hr 37 min: https://hudson.wedontsleep.org/job/devel/214/
<LPCIBot> Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=673874] Improve memcache caching of some
<LPCIBot> bugs pages
<stub> henninge: No - should be no difference with OOPS reports between local and production
<stub> henninge: If you get a WARNING or above, you should get an OOPS
<deryck> gmb, hi.  still not having much luck getting a clean test run with js upgrade branch.
<henninge> stub: yes, that's what I am trying to fix because we have too many WARNINGS. I just cannot find where the script issues the warning (which then becomes an  OOPS).
<henninge> And I don't see a warning in the log output when I run it loally.
<henninge> jml: I marked th bugs for your c.l.interfaces branch as qa-bad to prevent that revision from being deployed.
<stub> henninge: Ahh. If you want, there is more information available to the logger. You can set a breakpoint in canonical.launchpad.scripts.logger
<gmb> deryck: Okay. K
<gmb> ... er
<gmb> deryck: I mean: I'm still tied up with other bugs at the moment anyway, so not a big worry there. Can I help at all?
<stub> henninge: Oh.. no warning == ignore me.
 * henninge ignores stub
<jml> henninge: why does it need to be prevented from being deployed. I thought the issue was fixed?
<deryck> gmb, I don't think so, but thanks!  I've got a couple more ideas to try and if those prove unfruitful, then I'll cast about for more help.
<henninge> jml: on qastaging and staging but not on production.
<gmb> deryck: Okay.
 * gmb lunches
<henninge> I want to talk to a losa before I touch that.
<jml> henninge: ok. so I can I rely on you to change it back to qa-ok when prod is fixed?
<henninge> yes, you can ;)
<jml> henninge: thanks.
<stub> henninge: Doesn't the traceback in the OOPS tell you where the script issues the warning?
<henninge> I think not, let me try to see
<stub> Oh.... no traceback because it isn't in an exception handler.
<henninge> yes, I think that's it.
<stub> If you are bored of hitting your head against this wall, consider refactoring OopsHandler in c.l.scripts.logger -- you should be able to extract a lot more information from the log record in the emit() method and add it to the OOPS report, such as the line number and file name.
<henninge> stub: I'll do that when I get bored.
 * henninge is not bored today ...
<henninge> but now for some lunch, first
<henninge> thanks stub!
<jml> I hope the PQM submission that's running now fails
<jml> otherwise we'll be in a pickle
* jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<deryck> allenap, I think checkwatches is running again.  https://lpstats.canonical.com/graphs/BugWatchUncheckedOutdated/
<deryck> allenap, did you find out anything about this?  Or just happened to run again?
<jml> le sigh
<allenap> deryck: Chex sorted it out last night. Hung process, don't know why though.
<deryck> allenap, ok, cool.  At least we got it going again.  Thanks!
<deryck> And thanks Chex!
<gmb> allenap, deryck: Sometimes it gets stuck. I'd file a bug about it but the problem is so vague as to be unhelpful without some serious debugging in a production environment.
<deryck> yeah
<allenap> gmb: Do we run it multi-threaded at all, or was that turned off?
<gmb> allenap: I think we turned that off because it was doing something unsavoury to either our servers or someone else's. I could be wrong.
<jelmer> jml: So, having an O and a P series seems to work
<jml> jelmer: sweet.
<jml> jelmer: uploads to them are forbidden?
<jelmer> jml: Yeah, I haven't enabled any of that.
<jml> jelmer: I mean, have you tested that uploads to them are forbidden?
<jelmer> jml: Also, I had to name them o-series and p-series as "o" and "p" are too short. I haven't tried whether renaming the series works.
<jelmer> jml: Yes, although I now realize that it gave me a "Unknown distroseries" error, which seems weird.
<jml> jelmer: that's a bug, but probably not enough to block opening them.
<jml> jelmer: could you try a rename?
<jelmer> jml: yup
<jml> jelmer: ta
<jml> jelmer: how did renames work out?
<jelmer> jml: Still on it.
<jelmer> jml: It seems to be working (though I'm not entirely sure what I should be testing that might break): https://dogfood.launchpad.net/ubuntu/obvious
<jml> jelmer: it doesn't redirect. I wonder if that'll be a problem.
<bigjools> jml: can you join our mumble channel
<jelmer> jml: Redirect in what way?
<bigjools> there has to be a decent way of getting the trial TestCase and TestCaseWithFactory to co-exist
<jml> bigjools: yeah, there is
<jml> bigjools: but the branch where I do it has about eight failing tests.
<jelmer> hmm, mumble fail?
<bigjools> so I have a test that uses TestCase.makeTemporaryDirectory() but I need to change it to use the trial TestCase
<bigjools> should I wait for your branch or haxor something?
<henninge> So, the +packages timeouts seem not to be fixed. Any ideas what to do next?
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: Still no fix for timeouts in r11888 | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> bigjools: trial has something for that
<jml> bigjools: it's in the api docs
<bigjools> ok ta
<bigjools> jml: mktemp?
<jml> bigjools: yeah, that's it
<bigjools> it's not quite the same thing but I can fiddle
<jml> bigjools: fair enough.
<bigjools> jml: thanks for the pointer
<jml> I'll get my act in gear and land this testtools branch sometime soon
<deryck> mrevell, ping
<bigjools> is "act" a euphemism?
<mrevell> Hello deryck!
<jml> bigjools: yes. it's short for "arse".
<jelmer> jml: Just confirmed that targetting bugs works.
<jml> jelmer: thanks.
<jml> jelmer, bigjools: btw, could one of you please update dogfood to db-devel r9960 or later?
<jelmer> jml: sure
<jml> jelmer: thanks.
<jelmer> jml: dogfood should be on 9961 now
<bigjools> jml: can I pick your brains about Twisted again?
<jml> bigjools: sure
<bigjools> jml: so I am hoping there's a Twisted pattern for something
<bigjools> when we're fetching files from the builder it's currently a simple "for" loop
<bigjools> but if that's async then it needs changing - is there anything in Twisted that can help?
<jml> bigjools: uhh...
<jml> bigjools: what are you trying to do?
<jml> bigjools: fetch one file, then fetch a second file, then fetch... etc?
<bigjools> so it loops over all the files synchronously fetching them
<jml> bigjools: or are you fetching many files and you don't care about the order
<bigjools> ah good point
<bigjools> I can just request them all at once
<bigjools> dunno if that will make the builder barf
<jml> dl = gatherResults([fetchFileAsync(filename) for filename in filenames])
<bigjools> rejoice :)
<jml> dl.addCallback(when_all_files_are_gotten)
<jml> assuming fetchFileAsync returns a Deferred
<bigjools> it does
<jml> there are some quirks with error handling
<jml> specifically, it's not quite as obvious as when it's synchronous
<bigjools> well given I said there is no error handling at all right now ...
<jml> heh
<bigjools> out of interest if I did want to do them one at a time, is there another pattern?
<jml> hmm.
<jml> perhaps a DeferredSemaphore(1)
<jml> it's not very convenient though
<jml> bigjools: I reckon BuilderSlave could probably grow a method that takes a list of files and does this download. Makes it one step closer to putting a multi-file method on the slave itself
<bigjools> quite probably
<bigjools> It needs some careful thinking because there's a bunch of checking done one file names
<bigjools> s/one/on/
<jml> bigjools: not on the client side
<bigjools> jml: au contraire
<bigjools> it checks each file to make sure it's not going to write where it's not allowed to
<jml> bigjools: which of these three methods is doing checking on filenames? http://paste.ubuntu.com/530111/
<bigjools> jml: you're looking at the wrong thing there
<bigjools> you need to look at where we get files off the builder
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build (133): FIXED in 4 hr 2 min: https://hudson.wedontsleep.org/job/db-devel/133/
<jml> bigjools: oh right, you mean this instead: http://paste.ubuntu.com/530112/
<bigjools> jml: that's the bottom of the stack, but yes
<bigjools> the checking is done in _handleStatus_OK
<bigjools> for filename in filemap:
<bigjools> etc
<jml> bigjools: how is it any more complicated than finishing that loop and then having a list of known-good filenames to fetch?
<bigjools> jml: it's slighly more but not much
<jml> yay
<jml> there's a global state bug in my testtools-experiment branch
<mrevell> sinzui, I *think* I've removed the troublesome ACL from the Feedback page. I've also updated the page with a link to the SSO help. Could you please try editing the page to confirm the ACL removal has worked?
 * jml off
<cody-somerville> How would I go about trying out recipe builds?
<wgrant> cody-somerville: https://help.launchpad.net/Packaging/SourceBuilds/Recipes
<rockstar> abentley, is the standup in ~45ish?
<abentley> rockstar: yes.
<rockstar> abentley, thank you sir.
<abentley> rockstar: no problem.  Will you be joining us?
<rockstar> abentley, indeed I will.  I just got home.
<abentley> rockstar: welcome back.
<thumper> rockstar, abentley: mumble
<rockstar> thumper, on my way
<thumper> rockstar: https://bugs.edge.launchpad.net/launchpad-code/+bug/664234 needs qa
<_mup_> Bug #664234: Export the branch merge queue data through the API <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by rockstar> <https://launchpad.net/bugs/664234>
<thumper> wallyworld: mumble again?
<wallyworld> thumper: ok
<lifeless> wgrant: hi
<wgrant> lifeless: Hi.
#launchpad-dev 2010-11-12
<mwhudson> err!
<mwhudson> why is branchChanged hitting AssertionErrors?
<spiv> And no visible OOPS ID in the traceback sent to my 'bzr push' either...
<mwhudson> yeah
<spiv> On the other hand, LP did seem to successfully notice that my branch changed.
<mwhudson> thumper: hello :-)
<mwhudson> well
<mwhudson> the assertionerror is because the transaction is doomed
<mwhudson> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777XMLP119
<mwhudson> ah no, being doomed
<mwhudson> its in a timeout block
<mwhudson> wth, there's a gap of 15s between recorded queries
 * wgrant stabs qastaging.
<StevenK> wgrant: What did qastaging ever do to you?
<mwhudson> spiv: https://bugs.launchpad.net/launchpad-code/+bug/674305 <- feel free to hit the affects me too thing :-)
<_mup_> Bug #674305: bzr push occasionally reports AssertionError on terminal <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/674305>
<wgrant> StevenK: Timed out lots.
<wgrant> Although it may just be that those pages are broken now.
<wgrant> (Archive:+index, +packages, +delete-packages, that sort of thing)
<wgrant> Hmm.
<wgrant> It'd be nice if daily builds didn't all hit and DoS the build farm at the same time.
<spiv> mwhudson: done, thanks!
<lifeless> wgrant: https://bugs.launchpad.net/soyuz/+bug/672371
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <qa-needstesting> <regression> <timeout> <Soyuz:Fix Committed by jelmer> <https://launchpad.net/bugs/672371>
<thumper> mwhudson: hey
<thumper> mwhudson: whazzup?
<mwhudson> thumper: that bug
<thumper> mwhudson: I think
<mwhudson> thumper: https://bugs.launchpad.net/launchpad-code/+bug/674305
<_mup_> Bug #674305: bzr push occasionally reports AssertionError on terminal <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/674305>
<thumper> mwhudson: I think that may be the xmlrpc fuckage
<thumper> mwhudson: not sure why there are massive gaps
<mwhudson> thumper: the xmlrpc fuckage?
<mwhudson> the same as for getJobForMachine?
<thumper> mwhudson: all the timeouts on the xmlrpc server
<thumper> mwhudson: exactly
<mwhudson> hm, ok
<thumper> I've not been able to find out why we have 8s gaps
<thumper> with no obvious reason
<mwhudson> :/
<thumper> I spent almost a week chasing it
<thumper> and I've nothing to show for it
<wgrant> lifeless: Yeah, but isn't that in theory fixed?
<lifeless> wgrant: see my last comment
<wgrant> Oh.
<lifeless> iz single slow query
<lifeless> well
<lifeless> there are other slow queries
<lifeless> but thats the smoking gun
<wgrant> does that also take forever on a real DB?
<thumper> lifeless: ah... no
<thumper> it isn't a slow query
<thumper> it is the 15s gap between query execution and the next one that bothers me
<thumper> mwhudson: I'd love some help chasing that down as I've exhausted my understanding on that problem
<lifeless> wgrant: don't know
<lifeless> thumper: huh, what are you talking about?
<lifeless> thumper: I'm talking about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1776QS51
<thumper> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777XMLP119
<lifeless> query 33
<thumper> lifeless: ^^
<lifeless> thumper: I'll have a look
<lifeless> thumper: that looks like thread starvation to me
<thumper> lifeless: but it is only a guess
<thumper> lifeless: and why is it starved
<thumper> we don't know
<thumper> we are just guessing
<lifeless> thumper: the losas have the xml server split out as the highest ticket
<lifeless> thumper: when thats done we'll have more resources for xmlrpc
<lifeless> thumper: and after that the single threaded experiment will kick in
<lifeless> thumper: if you want to work on this today, I suggest implementing the per thread stats
<thumper> lifeless: no, I'm in the middle of something else
<lifeless> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/243554 for reference
<_mup_> Bug #243554: oops report should record information about the running environment <oops-infrastructure> <Launchpad Foundations:Triaged> <OOPS Tools:Triaged> <https://launchpad.net/bugs/243554>
<lifeless> wgrant: I have two problems answering for 'on a real db'
<lifeless> wgrant: firstly, we don't have the substituted ids to reproduce
<lifeless> wgrant: secondly don't have access and we're short staffed losa-wise.
<lifeless> wgrant: where are you up to exam wise?
<wgrant> lifeless: On the first day of a 12 day break.
<wgrant> So not doing much.
<lifeless> wgrant: Are you interested in tackling this perf issue? I have a trip on sunday for the cassandra training
<wgrant> lifeless: We should have a stub soon, shouldn't we?
<lifeless> and shoppinh/prep to do today
<lifeless> wgrant: in a few hours yes
<lifeless> wgrant: I'm strictly on leave, but I'm pretty bad at unwinding for < several-week periods.
<wgrant> Heh.
<lifeless> right now though, I have to do a shop-run. bbs.
<wgrant> So, 11888 made it bad, and the fix in iforgetwhat didn't help?
<lifeless> it helped
<lifeless> but not enough
<lifeless> we have two options
<lifeless> fix the query - its taking 200ms per SPPH at the moment.
<lifeless> rollback both 11888 and 11903(?)
<lifeless> note that rolling back leaves the page at 10 seconds and the ajax status updating timing out.
<wallyworld> thumper: hello, mcfly
<thumper> wallyworld: whazzup?
<wallyworld> i can't get branch lp:~wallyworld/launchpad/invalid-branch-link-message to merge properly
<wallyworld> it's not in the codebase either locally or on loggerhead and any merge attempts via pqm or lp-land claim there is nothing to do
<thumper> wallyworld: this is the revision that was backed out wasn't it?
<wallyworld> yes
<wallyworld> but i fixed it
<wallyworld> ie backed out the bad yui stuff
<thumper> right
<wallyworld> it's gone past ec2 again no probs
<thumper> did you reverse the reversed merge?
<wallyworld> no. not sure what to do
<thumper> right
<thumper> what you need to do is to merge devel into you branch
<thumper> then do a reverse merge of the revision that backed out your change
<thumper> the guts of the problem is that most of your branch has been merged
<thumper> and the files were then reverted
<thumper> so you need to revert the revert
<wallyworld> ok. noob alert. how do i do a reverse merge?
<thumper> do you know the devel revision that reverted your merge?
<thumper> wallyworld: it is a cherry pick merge
<wallyworld> i did just after it happened :-)
<spiv> wallyworld: merge -r NEW..OLD (rather than merge -r OLD..NEW)
<thumper> wallyworld: I
<wallyworld> i can see if i can find it
<thumper> wallyworld: I'll leave you in spiv's capable hands
<spiv> wallyworld: "bzr help revert" has an example:
<thumper> wallyworld: to test the merge locally
<spiv> âFor example, "merge . --revision -2..-3" will remove the
<spiv>   changes introduced by -2, without affecting the changes introduced by -1.â
<thumper> wallyworld: get an up to date devel, and go bzr merge --preview ../my-branch
<thumper> wallyworld: that way you can see what pqm will be attempting to merge into devel
<thumper> wallyworld: in the way of changes
<wallyworld> ok. i'll have a wee looksy. thanks. i'll grab a quick bite first. suddenly i'm hungry
 * thumper finally has the recipe index builds looking nice
<thumper> now for the tests...
<thumper> F**K ME - 150 / 1593  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<thumper> hard / soft timeouts
<thumper> 36 /  131  CodehostingApplication:CodehostingAPI
<thumper> mwhudson: ^^^ that'll be contributing to the push issues
<mwhudson> thumper: yep
<mwhudson> also :(
 * thumper has push failures like mwhudson had
<lifeless> wgrant: so ;)
<wgrant> lifeless: Hi. Just reinstalled and trying to get Launchpad running.
<lifeless> meep!
<wallyworld> poolie: ping
<wgrant> Desktp + Soyuz on amd64 with lp-buildd in a VM does not fit well in 4GiB. :/
<poolie> hi there wallyworld
<poolie> hi wgrant, lifeless
<wgrant> Afternoon poolie.
<lifeless> hi poolie
<wallyworld> hey, with the bzr 2.2.2 upgrade, we talked about doing it today from tip to avoid 2 lots of downtime. but i don't really think we should package trunk prior to official release. what downtime is involved? when i did the 2.2.1 upgrade, was there any downtime there?
<poolie> so two things:
<poolie> firstly, i wasn't really saying "you should package tip", just "it's safe to jump to tip if you want to"
<poolie> we shouldn't normally need to
<wgrant> There is a few seconds of downtime for codehosting upgrades.
<poolie> and if there's a bug there for which you need an urgent deployment, it could be better to just do a release immediately
<poolie> secondly i don't think it's really relevant to downtime
<spiv> wgrant: although if you are a user 90% of the way through an hour long push the cost to you will be more than a few seconds...
<poolie> i probably said "to avoid lag between us landing a fix and you running it"
<poolie> hm iwbni it didn't interrupt running connections
<wgrant> Hmm, true.
<spiv> poolie: hmm, and in this case hypothetically it wouldn't need to; we don't need to restart the ssh server, just provide a new bzr so that new connections will get a fixed lp-serve...
<wallyworld> so, me thinks it's better to wait for bzr 2.2.2 to be released next week deal schedule a small outage
<wallyworld> if needed at all
<wgrant> We have a downtime window next week for the DB upgrade anyway.
<lifeless> right
<lifeless> otherwise we have to schedule downtime
<lifeless> unless its zomg time
<lifeless> we will once the relevant RT is done have no-downtime deploys to codehosting.
<lifeless> but its (I think) third in the queue.
<lifeless> and we're getting one item done every 2-3 weeks.
<wallyworld> there's that cpu spin/wait issue that 2.2.2 fixes and a few people get hit by hit but not so many that we shouldn't wait till next week...
<spiv> Tangentially, I see https://lpstats.canonical.com/graphs/CodehostingPerformance/ looks a bit alarming ?
<lifeless> it does
<lifeless> fortunately its friday and noone will care about it till Monday
<lifeless> <ha ha ha>
<wallyworld> lifeless: you shouldn't care about it either. so much for you taking the day off. my wife would kill me if i worked too much on my "day off"
<poolie> hm, is is that a repeating pattern over the last 24h?
<poolie> spm, are you back at work?
<spm> poolie: I am, but seriously considering tking the rest off - having a horrible hayfever attack atm - has triggered a very nasty asthma response. :-/
<lifeless> spm: :(
<lifeless> spm: taken claratyne?
<spm> indeed
<lifeless> spm: saline solutionas suggested can help a lot - gets the pollen out
<spm> aye
<spiv> mmm, neti pots.
<lifeless> spiv: I ordered one wed
<poolie> nasonex is great (prescription only)
<wgrant> Hmm. It'd be nice if we had tracebacks for each SQL statement.
<lifeless> poolie: yeah, mine runs out in a few days
<lifeless> I've been given a (different) thing - I haven't read up to see if its equivalent yet.
<wgrant> rofl
<lifeless> 'allonase' or something like that
<wgrant> 'I also suggest renaming "incomplete" to "need info", as it's much more
<wgrant> descriptive. "Incomplete" makes it sound like the bug is in progress of
<wgrant> being fixed, but not yet done.'
<lifeless> wgrant: https://bugs.launchpad.net/launchpad-foundations/+bug/606959
<_mup_> Bug #606959: oops should record the short traceback that caused each query? <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/606959>
<spiv> lifeless: heh
<spiv> lifeless: what's nice about that idea is that although capturing tracebacks is a touch expensive, that shouldn't matter if you only do a reasonable number of queries ;)
<lifeless> spiv: http://ecoyogastore.co.nz/eco-yoga-gear/neti-pot
<lifeless> spiv: yeah
<poolie> i saw, linked from the discussion of Go, google have a final bug status of "unfortunate"
<poolie> that's nice
<lifeless> lol
<poolie> "suckstobeyou" :)
<wgrant> I thought they added that specially for the naming bug.
<spiv> lifeless: what web stores need for neti pots are photos more like http://www.flickr.com/photos/debrisdesign/502255811/
<wgrant> But I may be wrong.
<poolie> oh, maybe
<poolie> it could be freeform for all i know
<lifeless> spiv: yeah, I hope it has a manual
<poolie> but it's a bit more precise for some things than 'wontfix'
<spiv> lifeless: the internet can provide a guide or twenty, I'm sure.
<lifeless> what we need is a closure-space
<lifeless> N dimensions and a slider.
<lifeless> like the colour-space pickers
<wgrant> poolie: That's what Opinion is for!
<wgrant> *cough*
<lifeless> wgrant: thats an opinion!
<wgrant> lifeless: :(
<lifeless> seriously
<lifeless> its still an experiment as far as I've heard
<wgrant> Ah.
<wgrant> OK, with Unity defeated, it is now time to look at that query.
<lifeless> heh
<lifeless> wallyworld: if you want to discuss https://bugs.launchpad.net/bugs/674329 further I'm happy to do so - I didn't mean to prevent discussion about whatever symptoms you ran into.
<_mup_> Bug #674329: DecoratedResultSet eagerly fetches all results <performance> <Launchpad Foundations:Won't Fix> <https://launchpad.net/bugs/674329>
<wallyworld> lifeless: hmmm. seems at first glance the whole concept of iterable results sets which load records in batches is not supported?
<wallyworld> what is the query returns 10000000 records. and the user only wants to see 100 at a time?
<lifeless> wallyworld: thats what batch navigator is for
<lifeless> wallyworld: we do a count(*) [we should estimate instead, but thats orthogonal) and then use a slice (OFFSET X LIMIT Y in SQL) to only retrieve 100 at a time.
<wallyworld> i realise that's what it is supposed to be for, but isn't the pirpose defauted if __iter__ loads the whole lot anyway
<lifeless> wallyworld: __iter__ is /not/ for 'do partial work'
<lifeless> wallyworld: (neither in general, nor in this specific case)
<lifeless> wallyworld: in this specific case its because the database server will do all the work requested, always.
<lifeless> so we have to ask for the right amount of work up front rather than do some, do some more, and then say that we're done.
<lifeless> wallyworld: if you consider the implications of ORDER BY/GROUP BY on the work required in the db, this should make a lot of sense
<wallyworld> sorry for my dumbness, but isn;t the whole concept of yield to avoid eagerly realising the entire list?
<lifeless> uhm
<lifeless> so, iterators, generators and lazy evaluation
<wallyworld> why does the server do all the work? other databases don't enforce this?
<lifeless> wallyworld: good question. Pg definitely does; others I won't speculate on.
<wallyworld> sure, the database has to do some work to satisfy order by etc, but the step of extracting the data from the db into the result set needn't be done unless required
<lifeless> nevertheless
<lifeless> python-pgsql has a single large buffer with the results, no further network access occurs as we iterate the rows.
<lifeless> Or so I am assured by Smart People.
<lifeless> [specifically jamesh who dug into this in the past too]
<wallyworld> ok then.
<jamesh> by python-pgsql, you mean psycopg2?
<lifeless> jamesh: blah - yes
<wallyworld> lifeless: so to recap, if the result set has 10000000 rows, it's ok to do a list(rs) which effectively constructs an in memory data structure with all that data even if we only want to process 100 at a time?
<jamesh> wallyworld: if you stop reading the result set early, the only effort you're going to save is the conversion of the result buffer to Python objects on the client side.
<wallyworld> or am i missing something?
<wgrant> wallyworld: You'll slice first.
<wallyworld> yes, and for a large result set, that's significant and a potential performance issue
<wgrant> wallyworld: The slice affects the issued query.
<jamesh> if you know you will only need a subset of the rows, tell the database so that it can send you less info.
<wallyworld> jamesh: i'm talking about say batch navigator which allows the user to scroll through the results 100 at a time.
<wallyworld> we may want the whole lot eventually, but not all at once
<wgrant> That slices, so the DB only sends those 100 rows.
<wgrant> And only those 100 are turned into objects.
<wallyworld> wgrant: not if a list(rs) is done??
<wallyworld> which is what happens in DecoratedResultSet
<lifeless> wallyworld: no, to recap, slice the resultset.
<wgrant> wallyworld: __iter__ will only be called on the sliced version, right?
<jamesh> wallyworld: how do you know you'll want them all eventually?
<wgrant> slicing returns a new resultset.
<wgrant> And __iter__ is called on *that*.
<jamesh> for example, how often do people go to the second page of results from a bug search?
<wallyworld> jamesh: i said we *may* want them all eventually, say if the user scrolls to the end
<lifeless> wallyworld: general principle: specify all the work you want within a *transaction* - call it 2 seconds of processing time.
<wallyworld> :-)
<lifeless> wallyworld: and ask for, and process that. No more (would be wasted). No less (would result in additional queries - lowers efficiency)
<lifeless> wallyworld: the batch navigator does this slicing for you
<lifeless> wallyworld: how about we get concrete. 'I'm trying to do X, and Y is happening'
<wallyworld> ok. i think my problem is i misunderstood how the batch navigator works.
<wallyworld> thanks for setting me straight :-)
<lifeless> the batch navigator uses count() on the base result set to estimate the number of pages
 * wallyworld crawls back to his hole
<lifeless> and a slice to get the data for the current page
<wallyworld> makes sense
<lifeless> the count() is a performance issue with huge datasets
<lifeless> we need to switch to estimators
<wallyworld> yeah.
<lifeless> but thats orthogonal
<wallyworld> also, in my case, i had a query with a group by so had to override count()
<lifeless> erm
<wallyworld> the default storm rs barfs
<lifeless> :(
<lifeless> I thought that was fixed in 0.18
<wallyworld> you can't say select (*) from xxxx with a group by in it
<wallyworld> no
<wallyworld> i fixed it quite simply
<wallyworld> but i also found a bug in Count()
<wallyworld> it messes up count(distinct xxx)
<poolie> lifeless, do you go to the losa meetings?
<wallyworld> it leaves out () around the columns
<poolie> i don't know the speciic name for it, but i mean the one where francis asks them to do things
<wallyworld> s/select(*)/select count(*)
<lifeless> poolie: no, tz fail. I get minutes, and have a separate meeting with ISF
<poolie> k
<lifeless> poolie: I do when I'm in a workable tz
<poolie> i'll mail him then
<poolie> thanks
<poolie> jam, did you file an RT for starting lp-serve?
<poolie> bug 660264
<_mup_> Bug #660264: bzr+ssh on launchpad should fork, not exec <qa-ok> <Launchpad Bazaar Integration:Fix Committed by jameinel> <https://launchpad.net/bugs/660264>
<jam> I've had an rt for a while now, 41340 IIRC, but I'm not positive
<poolie> thanks, i'll check that
<jam> sorry, 42156
 * wallyworld goes to make a coffee and get his fire proof suit
<jam> poolie: https://rt.admin.canonical.com/Ticket/Display.html?id=41791
<poolie> that's not exactly the same as getting it running though
<poolie> is there a ticket or bug for that?
<poolie> iirc you need them to change some configuration scripts that you don't yourself have access to?
<lifeless> poolie: the lp-serve thing is moving; jam needed to land more code
<poolie> to do what?
<jam> poolie: there is one, but I keep shooting blind as to the rt number
<jam> Let me find the email
<poolie> thanks
<wgrant> Could someone run http://paste.ubuntu.com/530449/ on staging?
<poolie> lifeless, while jam's, looking, what do you understand the state of this to be?
<poolie> i'd just like to make the bug accurate and work out where if anywhere it's getting stuck
<lifeless> poolie: its in a back and forth discussion with the losas as they figure all the bits out
<lifeless> poolie: its low priority (relatively that is) so I wouldn't expect it to happen rapidly
<jam> poolie: 42199
<lifeless> poolie: mwhudson was landing the init script for jam, and with that it should be able to be enabled on staging
<lifeless> and then qad
<lifeless> epic fail
<lifeless> 3142  OOPS-1776B79    BugTask:+index
<poolie> so from that rt it looks like the next action is still 'get the service running on qastaging'?
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      238 /   35  Person:+commentedbugs
<lifeless>      150 / 1593  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       50 /  188  BugTask:+index
<lifeless>       36 /  131  CodehostingApplication:CodehostingAPI
<lifeless>       16 /    9  Person:+bugs
<lifeless>       14 /  352  Distribution:+bugs
<jam> poolie: right, this whole week there haven't been enough l-osas, and there have been some critical things going on
<lifeless>        9 /   70  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>        9 /    8  Archive:+copy-packages
<lifeless>        8 /  396  Distribution:+bugtarget-portlet-bugfilters-stats
<jam> today there was only Ch-ex
<lifeless>        7 /    0  BugTask:+addcomment
<lifeless> poolie: yes
<poolie> k, i don't want to preempt the critical things, of course, i just want it to not stay stuck after that
<lifeless> poolie: so in my queue its:
<lifeless>  - after RFWTAD stuff - thats important to finish getting single revs deployed and finish eliminating operation risk
<lifeless>  - after token librarian - thats old inventory which fixes timeouts for many private attachments (e.g. security builds)
<lifeless> in terms of LOSA time
<poolie> ok
<lifeless> short interrupts to move it along are of course reasonable
<poolie> so it's off john's plate until they get to it?
<lifeless> poolie: John can best answer that
<jam> lifeless, poolie: I'm at least pending them telling me what I need to do next
<jam> the last round I didn't know I needed until they asked for it
<poolie> mm there seem to be a few problems like that
<thumper> RFC: http://people.canonical.com/~tim/recipe-latest-builds.png
<thumper> it is using factory generated fake data, so I have multiple binary builds for the same arch
<thumper> but the basics are there
<thumper> this is up for review now
<thumper> poolie: hi
<thumper> poolie: we have another urgent need for committing to stacked branches
<poolie> hi thumper
<poolie> i think francis mentioned this...
<thumper> poolie: bzr-builder commits to the branch
<poolie> it was for.. right
<poolie> and why does it want a stacked branch not a checkout?
<thumper> poolie: and getting a branch for some big projects was using much more memory than the virtual builders had
<thumper> poolie: because it never pushes
<wgrant> thumper: Not a fan of the triplicated spr name and version, but apart from that it looks great.
<thumper> poolie: apparently an alternative solution is to change the merge code
<thumper> poolie: abentley wrote it all up
<poolie> onto the bug about commit?
<thumper> on the incident report
<thumper> for the buildd failures
<poolie> that was an email or a wiki page?
<thumper> wiki page I believe
<thumper> I could forward you the email if you like
<thumper> aaron wrote solutions up for me
<poolie> i can probably find it
<thumper> rockstar: ping?
 * thumper EODs
<poolie> thumper, is that https://wiki.canonical.com/IncidentReports/2010-10-28-LP-build-manager-not-dispatching ?
<thumper> poolie: ah, I see it isn't all on the incident report
<lifeless> thumper: if its not pushing
<lifeless> thumper: why commit at all?
<lifeless> stub: what do you think of the idea of capturing query params in oops
<lifeless> stub: it seems to me it will help reproducing issues  lot
<stub> lifeless: We will be logging private information, including information lp devs technically shouldn't have access to.
<stub> Some of that already leaks via the URL of course (so LP devs can learn about private teams they shouldn't know about)
<stub> But that hasn't been a problem so far, as private stuff has been company internal rather than private to a subset of the company.
<lifeless> stub: well, in theory :)
<lifeless> stub: so, we also manually create many queries today
<lifeless> so at least - today - we already leak that
<stub> Content of some of the private bugs could be an issue, as that would violate vendorsec
<lifeless> yeah
<lifeless> all disclosure stuff is serious
<lifeless> stub: when would we use content from a private bug in a query ?
<lifeless> stub: INSERT I guess
<lifeless> stub: + 'bugs like this'
<lifeless> stub: uhm, fo rthe INSERT case we could choose not to substitute
<lifeless> s/substitute/include/
<lifeless> stub: we're trying to figure out why https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777QS12 has multi second queries
<lifeless> stub: doing them by hand with plausible ids is extremely fast - 130ms for the main lookup in the page
<lifeless> stub: could it be the something odd like the isolation level (what level does appserver run as), or is it just the specific ids that will be at issue?
<lifeless> stub: ping
<lifeless> hmm, nvm for a sec
<lifeless> wallyworld: qastaging-slave vs main
<lifeless> perhaps
<lifeless> bah
<lifeless> wgrant: ^
<wgrant> lifeless: Could be, I suppose.
<wallyworld> lifeless: ECONTEXT
<lifeless> wallyworld: I was talking to wgrant ; tab fail.
<wallyworld> lifeless: np. i figured that when i saw the rest of the conversation come through :-)
<lifeless> stub: ping
<stub> lifeless: pong
<lifeless> hi
<lifeless> I need your help
<lifeless> we've got a very odd thing happening
<lifeless> have a look at these two oopses
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777QS12
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777QS19
<lifeless> this is the +packages page which is a current blocker for deploying
<stub> isolation level doesn't cause slowdowns
<lifeless> this page
<lifeless> https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204
<lifeless> in 1777QS12 query 34 takes  6.3 seconds
<lifeless> in 19 it takes 202ms
<lifeless> and 39 takes 20 seconds
<lifeless> we've shutdown cronscripts on asuka
<lifeless> so the load should be tolerable (about 2 I believe - spm can confirm ?)
<lifeless> running query 34 by hand, it takes about 200ms consistently, every time
<stub> lifeless: Are the oopses from the first batch? The way we currently do batching means that it you have a large set of results, the later batches will always timeout.
<lifeless> stub: same batch in both oopses
<lifeless> stub: same exact url
<lifeless> ahh
<lifeless> I think I've managed to get a slow query
<lifeless> \o/ finally
<wgrant> !!
<stub> OOPS-1777QS19 q39 is slow and comes with all the parameters (obviously we are not sanitizing the aborted query...)
<lifeless> stub: yeah, its also genuinely slow locally
<lifeless> by which I mean ro user on qastaging
<lifeless> stub: thanks
<stub> And that is slow because it is returning 1.35 million rows
<lifeless> \o/
<lifeless> wgrant: ^
<wgrant> Hmm. Is that the newer version query?
<lifeless> I think I just reused the existing grouped version of it
<lifeless> sounds like it was inefficient already
<lifeless> :)
<lifeless> or buggy
<wgrant> 1.35 million rows sounds buggy.
<lifeless> wgrant: this give you what you need to make a test, isolate n fix?
<stub> lifeless: its missing a join condition
<wgrant> lifeless: Maybe.
<wgrant> Hah, so it is.
<stub> lifeless: Its missing a 'AND sourcepackagename.id = sourcepackagerelease.sourcepackagename
<wgrant> SPN
<wgrant> Yeah.
<lifeless> stub: in the inner or outer?
<stub> The outer
<lifeless> 2.7 seconds
<lifeless> tolerable with just one
<stub> So every matched row is being expanded to 38k rows.
<wgrant> Ah.
<wgrant> I think in fact that it shouldn't be joining against SPN at all.
<lifeless> wgrant: still badly needs tuning
<lifeless> oh, I did chage that, I removed spn.... but I bet storm is putting it back in.
<lifeless> bastardo.
<lifeless> how do you disable autotables?
<lifeless> jamesh: ^
<wgrant> lifeless: It's still explicitly there.
<wgrant>             clauseTables=[
<wgrant>                 'SourcePackageName', 'SourcePackagePublishingHistory'])
<wgrant> s/Name/Release/, I suspect.
<stub> So we might be able to avoid the subselect using DISTINCT ON
<jamesh> lifeless: what's the context?
<lifeless> jamesh: nvm :)
<lifeless> jamesh: I was thinking storm was seeing a table ref from an inner query and autotables adding it to the outer FROM
<lifeless> jamesh: but I was wrong
<wgrant> lifeless: So, how does it go if you remove the SPN join from the query?
<jamesh> ah.
<lifeless> wgrant: fine
<lifeless> wgrant: what file is tha tin
<lifeless> wgrant: 2.6 seconds
<wgrant> lifeless: lib/lp/registry/model/distroseries.py
<wgrant> 2.6 seconds sounds sort of excessive.
<lifeless> http://pastebin.com/bJ2TxmFc
<stub> Hmm... distinct on makes it worse.
<lifeless> ok
<lifeless> thats up in PQM
<lifeless> immediate fix
<wgrant> And that hopefully makes it non-critical.
<lifeless> yeah
<lifeless> assuming theres nothing hiding behind it
<lifeless> let me get the change cowboyed to see
<wgrant> This explains why even trivial archives were timing out.
<lifeless> wgrant: indeed
<lifeless> ok its landed
<adeuring> good morning
<bigjools> morning
<mrevell> Hey up, by the way
<henninge> lifeless: It looks like your fix for bug 672371 did not help. +packages still times out on qastaing.
<_mup_> Bug #672371: Archive:+packages timeouts <ppa> <qa-needstesting> <regression> <timeout> <Soyuz:Fix Committed by jelmer> <https://launchpad.net/bugs/672371>
<henninge> What's next? Revert r11888?
<henninge> jml: Hi! Any chance you could QA bug 673015?
<_mup_> Bug #673015: Code of Conduct requirement for PPA upload rights is unnecessary <ppa> <qa-needstesting> <Soyuz:Fix Committed by jml> <https://launchpad.net/bugs/673015>
<henninge> allenap: Hi! Any luck figuring out bug 667340?
<_mup_> Bug #667340: Trac status of "Verified" confuses bug watcher <qa-bad> <trac-support> <trivial> <Launchpad Bugs:Fix Committed by allenap> <https://launchpad.net/bugs/667340>
<henninge> stub: Can you please QA bug 673874 before starting on your weekend?
<_mup_> Bug #673874: Improve bug comment caching <qa-needstesting> <Launchpad Bugs:Fix Committed by stub> <https://launchpad.net/bugs/673874>
<allenap> henninge: No, not yet. It hasn't caused any regressions, so it's actually safe to go.
<allenap> henninge: I'll mark it as qa-ok but continue to investigate.
<henninge> allenap: thanks a lot!
<henninge> gmb: can you please QA bug 672507 ?
<_mup_> Bug #672507: Add bug_notification_level to the structural +subscribe view <qa-needstesting> <story-better-bug-notification> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/672507>
<gmb> henninge: Sure.
<gmb> henninge: Done
<henninge> gmb: thanks a lot!
<bigjools> henninge: jml needs my help to QA that
<henninge> bigjools: thanks for offering it ;)
<lifeless> henninge: see the comment in the bug
<bigjools> he can't QA without it since it needs dogfood :)
<lifeless> henninge: we're waiting on https://lpbuildbot.canonical.com/waterfall
<henninge> lifeless: ah yes, thank you.
<jml> hello.
<jml> yes QA, I know I know
<jml> bigjools: where do I need to point .dput.cf at?
<bigjools> jml: http://pastebin.ubuntu.com/530615/
 * bigjools processes your upload
<bigjools> jml: rejected
<jml> bigjools: why so?
<bigjools> jml: can I help you make a dummy package that I know works
<bigjools> "Unable to find python-testtools_0.9.6.orig.tar.gz"
<bigjools> and it was a mixed upload it seems
<jml> meaning?
<bigjools> binaries and source
<bigjools> jml: I normally use the "hello" package
<bigjools> apt-get source hello
<bigjools> cd hello-2.5
<bigjools> dch -i
<bigjools> <add a revision>
<jml> yeah, that's what I did with testtools
<jml> (so far so good)
 * bigjools sighs at stuck keys
<jml> heh
<bigjools> ok, then you need to "debuild -S"
<jml> ahh
<jml> it's the -S that I didn't do
<jml> uploaded
<bigjools> accepting it this time
<jml> yay
<bigjools> you cleared the CoC from ~jml?
<jml> bigjools: I did, but I'd like to double check with getUtility(IPersonSet).getByName('jml').is_ubuntu_coc_signer
 * bigjools checks
<bigjools> False
<bigjools> qa-ok!
<jml> sweet.
<jml> bigjools: thanks!
<bigjools> my pleasure
 * bigjools goes to celebrate with caffeine
<jml> henninge: what's the word on the crazy non-vc managed file that refers to class paths?
<henninge> jml: It cannot be updated outside of a roll-out - at least not without Tom around ...
<henninge> jml: So I am preparing a branch that adds the required import to c.l.i again with an XXX to remove it again after the roll-out.
<jml> henninge: that seems unsatisfactory
<henninge> and a special roll-out requirement to update that file
<jml> henninge: can't we just add the requirement and leave c.l.i as-is?
<henninge> jml: only if we go without a further deployment today
<jml> henninge: so it needs a rollout-with-downtime?
<henninge> so spm says, yes.
<jml> henninge: did he say what it's needed for?
<henninge> jml: hang on, I'll forward the mail
<jml> henninge: thanks :)
<jml> henninge: ok. I find this whole thing colossally annoying, but it looks like you guys are making the best of a bad situation.
<henninge> jml: we are trying hard ... ;) thanks
<henninge> and yes, it is annoying
<lifeless> henninge: it passed buildbot
<lifeless> henninge: when 914 hits qastaging
<lifeless> then
<lifeless> https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204
<lifeless> should start working
<lifeless> that should be anytime now
<henninge> lifeless: thanks! But it will be another 4 hours or so ...
<lifeless> henninge: why?
<henninge> https://lpbuildbot.canonical.com/builders/lucid_lp/builds/355
<henninge> It just entered buildbot not passed it yet
<lifeless> oh crumbs 913 I see
<lifeless> ah well
<lifeless> gl
<lifeless> !
<lifeless> and gnight all
<henninge> lifeless: good night and thanks again.
<bigjools> hello - I am having trouble getting the webservice to work on dogfood.  When I try and log in, there's a rejection because it can't traverse to '1.0'.  HALP?
<jelmer> bigjools: Have you tried using 'devel' rather than 1.0 ?
<bigjools> yes, that's what I am using - which makes the error more odderer
<bigjools> launchpad = Launchpad.login_with('testing', 'https://api.dogfood.launchpad.net/devel/')
<jelmer> Shouldn't there be another /api/ in there?
<bigjools> yes
<bigjools> still fails!
<jelmer> then I'm out of ideas :-)
<bigjools> hmm using https://api.dogfood.launchpad.net/api worked
<bigjools> ah you need to write version='devel' in the login_with params
<bigjools> jml: got a sec?
<jml> bigjools: sure
<bigjools> jml: I'm probably doing something very very stupid but I have code blindness.   See  http://pastebin.ubuntu.com/530655/
<bigjools> there's a code snippet and a pdb session
<bigjools> the inner function callback can't see all of the outer method's variables....
<LPCIBot> Project devel build (220): FAILURE in 2 hr 4 min: https://hudson.wedontsleep.org/job/devel/220/
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Remove StartsWith matcher from
<LPCIBot> lp.testing.matchers in favour of one from testtools & fix some
<LPCIBot> assertions that always passed.
<LPCIBot> * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Really drop Sourcepackagename from getNewerSourceReleases - fixing massive timeouts on +packages.
<jml> ./me looks
<deryck> Morning, all.
<bigjools> morning deryck
<jml> bigjools: you are masking them in scope, I think.
<jml> bigjools: let me knock up a simpler example...
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is open | firefighting: Lots of timeouts on qastaging!! | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> bigjools: http://paste.ubuntu.com/530657/
<henninge> OK, qastaging is timing out left and right ... :(
<henninge> Ubuntu pages seem to work fine but any project page times out.
<jml> bigjools: in "if file_sha1 == 'buildlog':", you are overriding out_file, out_file_name and out_file_fd
<jml> bigjools: probably the thing to do is pass them in.
<jml> e.g.
<bigjools> jml: it's not got that far yet
<jml> d.addCallback(got_file, out_file_name, out_file)
<jml> bigjools: it doesn't matter.
<bigjools> ok
<jml> bigjools: run the python I pasted
<bigjools> that's special
<jml> bigjools: simply having an assignment in the scope masks the outer scope, whether or not the assignment has been evaluated.
<jml> bigjools: I'm not sure it would be sensible to do anything else.
<bigjools> jml: ok thanks , I'll pass 'em in
<jml> bigjools: np.
<henninge> Argh!
<henninge> I think I never realized how widespread the problems are that r11888 caused.
<henninge> Maybe it's just that.
<wgrant> henninge: It should be limited to pages on IArchive.
<jml> henninge: can you please subscribe me to whatever bug you file for the XXX in c/l/interfaces/__init__?
<henninge> jml: oh bug, right ... ;)
<wgrant> henninge: Anything outside Archive:+(index|packages|copy-packages|delete-packages) is probably not 11888.
<henninge> wgrant: thanks
<henninge> although I wish it was ... (because there is a fix coming)
<jml> bigjools: should I put that API gotcha on a wiki page somewhere?
<bigjools> jml: not yet - I can't get it working still
<jml> bigjools: ok.
<bigjools> jml: there's an error from wadllib about "Can't look up definition in another url"
<jml> I've not seen that one before
<bigjools> and I suspect I need leonardr
<bigjools> yeah, it's doing something weird so that the /api is stripped somewhere
<wgrant> The URL shouldn't have /api in it.
<bigjools> but later depends on it being there
<wgrant>  /api is used to traverse from the webapp to the API -- you don't use it on api.launchpad.net.
<bigjools> .........
<bigjools> and so it works
<bigjools> thanks wgrant
<wgrant> Heh.
<henninge> jml: bug 674476 I failed to mention it in the XXX, though... :/
<_mup_> Bug #674476: Files outside the LP tree reference LP code <Launchpad Foundations:New> <https://launchpad.net/bugs/674476>
<jml> henninge: thanks. that's ok.
<henninge> and you are subscribed
<henninge> jml: and thanks for reminding me about the bug
<jml> henninge: np.
<mrevell-lunch> #
<jml> Started in 15 minutes 27 seconds!
<bigjools> jml: ha - remember how we added Deferred to lp_sitecustomise.py?
<jml> yeah?
<bigjools> jml: looks like I need DeferredList too :)
<jml> bigjools: I thought DeferredList subclassed Deferred
<bigjools> ForbiddenAttribute: ('addCallback', <DeferredList ....
<jml> meh
<bigjools> I guess zope doesn't care so much about that
<jml> why might bugtask.date_closed be none, even though its status is one of Fix Released, Wontfix or Inprogress?
<shadeslayer> jam: was my qtwebkit build fix0red?
<gary_poster> henninge: where are we with the qastaging slowdown?  I see that only qastaging is affected; staging and production are fine.  The timeout exception I see is within database code, but since that's where we check for timeouts, that's not necessarily indicative.
<gary_poster> Has anyone looked at qastaging logs?  Has anyone looked at performance graphs?  Has anyone tried to correlate performance graphs with revisions deployed on qastaging?
<gary_poster> and, are we coordinating here or on -ops?
<gary_poster> Hm, no tuolomne graphs of qastaging AFAICT :-/
<gary_poster> Maybe I need to know the machine name(s)
<gary_poster> qastaging is the same machines as staging, and staging is not timing out (as badly?) so machine load doesn't seem likely...
<gary_poster> trying logs
<henninge> gary_poster: staging is a lot of revisions behind qastaging atm
<gary_poster> henninge: I figured it was something like that, yeah
<gary_poster> so what has been done?
<gary_poster> I saw stub's reply, but that didn't tell us much
<gary_poster> I was about to grope around in logs
<henninge> gary_poster: logs is good
<henninge> I was hoping that the authors of the revision could check if any of their code could be causing this.
<gary_poster> henninge: you identified the revision?
<henninge> no, it's just any of the later ones.
<gary_poster> ah :-)
<henninge> bug I could narrow down the range because it only started today.
<gary_poster> henninge: you up for that while I do log groping?  I can share log groping fun here.  So far the only thing that does not look like chatter in the qastaging librarian log  is "Exception KeyError: ((<class 'canonical.launchpad.database.librarian.LibraryFileAlias'>, (1890638,)),) in <function remove at 0x8f3ced8> ignored"
<gary_poster> seems to be mostly happy thoug
<gary_poster> h
<henninge> gary_poster: I am looking at the revs atm, yes.
<gary_poster> cool thanks
<gary_poster> There are boatloads of "No handlers could be found for logger "librarian"" things in logs, which do make me a nit nervous
<gary_poster> bit
<henninge> gary_poster: I noticed earlier that getting images from the librarian had a long delay.
<jml> mrevell: flacoste: http://paste.ubuntu.com/530734/
<gary_poster> henninge: yeah.  Maybe.  It doesn't smell like the cause to me.  This is interesting though: qastaging app log is *swamped* with these: http://pastebin.ubuntu.com/530735/
<henninge> gary_poster: What is a "DoomedTransaction"?
<gary_poster> a transaction that must not be restarted
<gary_poster> henninge: may be an unrelated problem.  This started after the the restart 2010-10-21T15:22:39, so it's been happening a looong time
<henninge> ah, ok
<abentley> jkakar: around? ResultSet.set is generating bad SQL.
<abentley> jkakar: http://pastebin.ubuntu.com/530742/
<jkakar> Can you paste the code that generates this, please?
<jkakar> abentley: ^^ Also, am on a call, will be laggy.
<abentley> jkakar: http://pastebin.ubuntu.com/530747/
<jkakar> abentley: What's the __storm_table__ for SourcePackageRecipeBuild.
<abentley> jkakar: __storm_table__ = 'SourcePackageRecipeBuild'
<jkakar> abentley: Is that right?
<abentley> jkakar: Yes.
<jkakar> abentley: Can you do a JOIN in an UPDATE statement?  It looks like you're building a bad query, not that Storm is generating a bad one.
<abentley> jkakar: I'm not an expert on SQL syntax.  It's possible that I'm asking Storm to do the impossible, but if I am, I expect Storm to tell me.
<jkakar> abentley: Storm won't tell you if you're trying to do the impossible.
<jkakar> abentley: It's reasonable to expect it, but Storm is just a "thin" *cough* layer with an expression compiler that generates SQL exactly as you specify
<jkakar> abentley: The database is telling you that you're trying to do the impossible.  Which, given that different backends have different definitions of "impossible", is probably right anyway.
<abentley> jkakar: This doesn't seem like it *should* be impossible.  Can't one do a subselect or something?
<jkakar> abentley: Sure.  The best thing to do is first, figure out what query you want to run.  The second step is to figure out how to make Storm generate it.
<jkakar> abentley: If you can write out the query you want I can help you figure out the second part.
<abentley> jkakar: If that is actually how it is done, why not write the query directly?
<jkakar> abentley: A few reasons:
<jkakar> - Storm will expand a class name into a series of column names in a query, such as in a SELECT.
<jkakar> - When you use Storm you get a result set that gives you powerful capabilities, like union, max, count, etc.
<jkakar> - When your class changes, because you added or removed a column, you don't have to change your queries unless they involve one of the modified attributes.
<jkakar> - Most of the time you already know what query you want, so it isn't hard to get from what you want to a store.find() call with Storm expressions.
<jkakar> abentley: This sounds like a case where the problem is not knowing what query you want to run.  With Storm you're always expected to know what query you want to run.
<abentley> jkakar: What makes you say that?
<jkakar> abentley: It was designed explicitly not to hide SQL from you, but in fact, to make it possible to generate the exact query you want.
<jkakar> abentley: Because the query you're generating doesn't work (according to the database)?
<jkakar> abentley: Sorry, that probably sounded offensive, but I mean no offense.
<abentley> jkakar: I didn't set out to generate a query.  I set out to use an existing function that returns a collection that provides functionality that should do what I want.
<jkakar> abentley: Okay.
<abentley> The query I actually want is, "Find all SourcePackageRecipeBuilds where recipe = X and set recipe to NULL", which I can work out in SQL if you like.
<jkakar> abentley: That's the next step, yes, working it out in SQL so you know what you need Storm to generate for you.
<abentley> jkakar: UPDATE SourcePackageRecipeBuild SET recipe = NULL WHERE recipe = 5
<abentley> jkakar: 5 actually being a variable.
<jml> css question
<jml> if I want to have a heading that has an image followed by some text aligned to the middle of that img, how do I do that?
<jkakar> abentley: store.find(SourcePackageRecipeBuild, SourcePackageRecipeBuild.recipe == $value).update(recipe=None)
<abentley> jkakar: I don't want to have two definitions of how you get the builds associated with a recipe, so how do I update getBuilds to return a ResultSet that works?
<jkakar> abentley: Let me read some PostgreSQL documentation for a sec...
<jkakar> abentley: Hmm, it looks like you could include multiple tables in an UPDATE... at least on PostgreSQL.
<jkakar> abentley: I'm not sure exactly what you need... but I think you would benefit by writing a specialized query for the case you have.
<jkakar> abentley: For two reasons, (1) it's a simpler query than the one from getBuilds and will probably run faster and (2) you'll run one less query than you do now (by specifying pending=True and pending=False).
<bigjools> henninge: how big do the tarballs get that are produced by the TTBJs on the builders?
<henninge> good question
<henninge> bigjools: well, it's all text files so they should compress nicely.
<abentley> jkakar: I disagree that it's a benefit.  I'd rather have clearer code than simpler queries, and I think two queries is acceptable, and if I cared, I could update getBuilds so that I could get all builds at once.
<bigjools> henninge: it's just that the code that jtv wrote reads them into memory ...
<bigjools> it obviously works but I'd rather not have a time bomb
<henninge> bigjools: They should not become very big, most projects don't have many templates
<henninge> and if they have many, they are each small
<jkakar> abentley: Okay.  Updating getBuilds to optionally include the pending clauses then would do what you want... ie: use it in a way that doesn't include the pending clauses.
<bigjools> henninge: typically what sort of size?
<henninge> I'd have to research that. danilos, do you have a figure off the top of your head?
<bigjools> henninge: the change I am making will mean we could potentially be reading as many of these as there are builders
<bigjools> in parallel
<danilos> henninge, 17
<henninge> thanks danilos
<bigjools> danilos is ever helpful :)
<danilos> henninge, uhm, let me read the backscroll then
<danilos> bigjools, if the tarball only includes translations, they should be small (never more than say 50M for the biggest case, but probably around 1M for most)
<abentley> jkakar: So it would only support set if pending was not supplied?
<jkakar> abentley: Yep.
<abentley> jkakar: gross.
<jkakar> abentley: Unless we change the way UPDATE statements are generated.
<jkakar> abentley: So you were probably right in the beginning, there probably is a bug in Storm.
<bigjools> danilos, henninge: aieeeee, I just looked at addOrUpdateEntriesFromTarball
<bigjools> tarball_io = StringIO(content)
<bigjools> if I have the file on disk is there a different method that will work?
<danilos> bigjools, well, by that time, they are already in memory :) where is "content" initialized?
<henninge> bigjools: actually, that's my code ;)
<bigjools> danilos: either in the upload processor or from the builder
<danilos> bigjools, we can as easily parse the file directly on-disk using the tarfile module, if I am not mistaken
<bigjools> sorry but arbitrarily sized files going into stringio scares me
<bigjools> I'm going to file a bug about this, it'll need fixes in a few places
<danilos> bigjools, uhm, what I am trying to say is that StringIO is a shallow wrapper, entire file is already in the memory
<bigjools> danilos: yes, it should not be :)
<henninge> I get it
<henninge> just some figures
<danilos> bigjools, agreed, perhaps we need to save it to a tmp file before we process it
<henninge> all of gimps templates are 736k
<henninge> all of gtk+ templates (2) are 264k
<bigjools> danilos: well I can make a tmp file available in the buildd-manager and the upload processor before it calls that method
<bigjools> it currently has to read the file into memory before passing it
<bigjools> if the template generation goes a bit wonky then it can easily take out the buildd-manager
<bigjools> which Is Bad (TM)
<danilos> bigjools, then it'd be a very simple fix on "our side"
<bigjools> excellent, I'll file the bug and put some pointers to soyuz/buildmaster code in it
<bigjools> cheers
<danilos> bigjools, don't do it before you make the tmp file available :P
<danilos> bigjools, also, note that we are using the same thing for actual Ubuntu package builds, so we'd want to fix that as well
<bigjools> danilos: yes, that's what I was referring to above about the upload processore
<danilos> bigjools, ok consigliere ;)
<bigjools> heh
<bigjools> was about to make a joke about an offer you can't refuse
<danilos> heh
<abentley> jkakar: Here's a version that seems to work: http://pastebin.ubuntu.com/530766/
<jkakar> abentley: Yeah, not surprisingly.  I wonder how that query performs compared to the other one, though?
<abentley> jkakar: For the cases where both work, I bet they both perform the same.  That's got to be trivial to optimize.
<jkakar> abentley: Probably, yes.  Though, in practice, I've occasionally seen dramatically different performance when a query uses a subselect vs. when it doesn't.
<jkakar> It's hard to understand when that will be the case or why, though.
<henninge> gary_poster: staging is timing out, too, now. It has been updated from 9955 to 9965
<gary_poster> henninge: well, that seem to point a pretty stong finger at code then, which simplifies things in some ways.  how is the revert going on qastaging?
<gary_poster> *strong
<henninge> gary_poster: it's taking it's time
<gary_poster> :-)
<henninge> its
<gary_poster> ok
 * gary_poster carefully replaces the second "it's" but leaves the first intact ;-)
<henninge> thanks for being careful ;)
<abentley> jkakar: filed as https://bugs.edge.launchpad.net/storm/+bug/674582
<_mup_> Bug #674582: Storm may generate SQL errors on ResultSets.set for otherwise-working ResultSets. <Storm:New> <https://launchpad.net/bugs/674582>
<jkakar> abentley: Thanks!
<sinzui> henninge, gary_poster: I agree that staging is now as useless as qastaging, but I do not see what has changed to make SPR/SPPH queries slower.
<gary_poster> sinzui, I am no longer actively investigating, because henninge's summary that revisions 11888 -> 11899 -> 11914 are a likely cause sounded like a good hypothesis.  We are waiting to see if reverting these clears up qastaging.
<sinzui> 11914 is not on staging, so I discount that
<gary_poster> yes, but it's part of the logical set
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build (221): FIXED in 4 hr 3 min: https://hudson.wedontsleep.org/job/devel/221/
<sinzui> gary_poster, henninge, I do not understand the "set" point. I do not see that revision on staging 11914. I suspect that 11914 fixes the issue. I think the origin of the issue is 11899
<henninge> sinzui: 11914 does not fix it, it had already been on qastaging and did not help
<henninge> sinzui: we are currently reverting 914 and 899
<henninge> gary_poster, sinzui: do you know if the revision display at the bottom of the LP page is dynamic or static?
<henninge> i.e. Does it need a "make build" to be updated or is the information straight from the branch?
<jml> people.c.c has an old launchpadlib :(
<sinzui> henninge, it requires make build
<henninge> so Chex just told me
<bigjools> jml: I added a test to directly use downloadPage against a real slave in a test and it gets a "405 Method not allowed".  Do you know if Twisted has the equivalent of urllib2.debug = True ?
<sinzui> EdwinGrubbs, I am looking at distroseries.getCurrentSourceReleases() I think the subquery for max(spph.id) is doing a full table scan of SPNs because there is no constraint to return only the SPNs passed to the method
<jml> bigjools: I don't know what urllib2.debug=True is.
<jml> bigjools: and I don't know of any debugging foo off the top of my head
<bigjools> jml: it dumps the http comms to stdout - I'm trying to work out what methods it's using that's not allowed
<sinzui> EdwinGrubbs, I suspect that moving 'SourcePackageRelease.sourcepackagename IN %s" into the subquery will make the query faster
<EdwinGrubbs> sinzui: that shouldn't be necessary since it looks like "spr.sourcepackagename = SourcePackageRelease.sourcepackagename" makes it search for all the spph/spr records for a single sourcepackagename.
<jml> bigjools: nothing obviously like that in t.web
<bigjools> jml: yeah, I looked too
<jml> bigjools: wireshark maybe?
<bigjools> tcpdump ... :)
<jml> bigjools: ooh, did you know about from launchpadlib.uris import DOGFOOD_SERVICE_ROOT?
<bigjools> yes
<bigjools> I think I put it there and shamefully forgot
<sinzui> EdwinGrubbs, that assumes that the query planner built that set first
<EdwinGrubbs> sinzui: I've never seen an instance where the query planner thought that it would be faster to run a correlated subquery first and then limit the results of the outer query.
<jml> why is it that bazaar.launchpad.net is so hard for dns servers to resolve?
<EdwinGrubbs> s/limit/filter/
<sinzui> EdwinGrubbs, since we are looking at a PG 8.4 change + the removal of the SPN table from the query. I think I should get sometimes based on where that constraint is placed
<jml> mrevell: still around
<jml> ?
<mrevell> Hi jml, sure am
<jml> mrevell: I don't know where best to put this link on the beautifully presented https://dev.launchpad.net/BugJam â http://mumak.net/lp-bugjam-2010/
<jml> mrevell: it's a count of the number of bugs fixed during the bug jam so far
<EdwinGrubbs> sinzui: how many sourcepackagenames are passed in as an argument to getCurrentSourceReleases()
<mrevell> jml, I love it :)
<mrevell> jml, I'll put a link under "Tracking progress"
<jml> mrevell: thanks.
<sinzui> EdwinGrubbs, 1, but  get get 38536 where we would expect 1 from natty, maybe 3 for maverick
<sinzui> Edwin 1, my move of the constraint does not fix the issue, 2 I feel pretty good that getting what looks like getting a match for every SPN in natty implies an open join
<EdwinGrubbs> sinzui: can I see the query plan?
<sinzui> I will get it for you
<bigjools> jml: well, that flushed out a nice bug in the tests we wrote a few weeks ago :)
<jml> bigjools: which was?
<bigjools> jml: it was constructing a url of the form /rpc/rpc
<jml> bigjools: heh
<jkakar> jml, abentley: Woah: http://paste.ubuntu.com/530794/
<jml> jkakar: yeah, it's filed as a critical bug.
<jml> jkakar: https://bugs.launchpad.net/launchpad-code/+bug/674305
<_mup_> Bug #674305: bzr push occasionally reports AssertionError on terminal <codehosting-ssh> <xmlrpc> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/674305>
<sinzui> EdwinGrubbs,  this is the plan to get the current release of bzr, 1 SPN provided and only 1 expected: http://pastebin.ubuntu.com/530797/
<jkakar> jml: Cool.
<jkakar> jml: Dunno if it helps debugging, but this was with a bound branch, it wasn't a push (explicitly).
<jml> jkakar: I'm not at all involved in fixing it
<jml> <- part of the problem
<jkakar> Heh
<sinzui> EdwinGrubbs, sourcepackagename is still listed in the FROM. It was removed several revisions ago
<sinzui> removing it from the query fixes everything
 * sinzui looks at code again
<EdwinGrubbs> sinzui: yeah, I was wondering where that table came from.
<EdwinGrubbs> sinzui: so, is the code not broken? Was it just an old oops?
<sinzui> EdwinGrubbs, It was removed a few days ago, lifeless removed it from clauseTables in r11914, but I suspect something else is putting the table in the from clause
<sinzui> Edwin to be clear, the SPN joins were removed a few days ago, Lifeless then landed another branch to remove it fix clauseTables. But this oops shows that the SPN table is still in the from clause
<sinzui> EdwinGrubbs, ^
<sinzui> EdwinGrubbs, sorry. I am looking at too may oopses. That oops was for an older revision
 * sinzui tries query from r11915
<sinzui> EdwinGrubbs, This is the correct plan for qastaging: http://pastebin.ubuntu.com/530801/
<mrevell> sinzui, Thanks for your post wrt strategies for the bug jam.
<sinzui> mrevell, your welcome
<mrevell> Have a wonderful weekend people. See you Monday.
<EdwinGrubbs> sinzui: ok, the problem is that there are 1138 spr records for a single sourcepackagename.
<sinzui> edwin I agree. I am looking for a constraint or a revised subquery that  removes the loop or 1138
<lifeless> moin
<lifeless> sinzui: rev 11914
<lifeless> EdwinGrubbs: ^
<sinzui> lifeless: yes, but 11915 still times out
<sinzui> lifeless we want to reduce the loop of SPRs in the query
<bigjools> good bye, have a nice weekend
<EdwinGrubbs> sinzui: since, there is only one valid spph record for all the spr records, you would get good performance by just eliminating the subquery and moving the conditions into the outer query. You will just have to eliminate the duplicates. DISTINCT won't let you choose the spph record with the max id, so you would have to do that in python, if it is important to get that spph record and not a random one.
 * sinzui nods
<lifeless> sinzui: works for me
<lifeless> https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204
<lifeless>     At least 782 queries/external actions issued in 17.77 seconds
<lifeless> little slower than ideal
<lifeless> trying again to remove cold cache effects
 * sinzui just went from 9695.618 to 33.446 ms using a subquery table of just current ids
<lifeless>     At least 782 queries/external actions issued in 12.63 seconds
<lifeless> sinzui: ^ https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204
<lifeless> sinzui: I welcome further improvements here
<lifeless> EdwinGrubbs: bringing too much back and filtering in python will almost always be slower
<sinzui> lifeless yes, we want to see a source package page load a single spr.
<lifeless> storm is (relatively) slow at deserialisation, due to the cache coherency logic
<sinzui> https://qastaging.launchpad.net/ubuntu/natty/+source/bzr
<lifeless>     At least 49 queries/external actions issued in 1.91 seconds
<lifeless> view-source:https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=1
<lifeless> interestingly that page is not flat yet, the binaries must be the cause because there is a test that its flat with sources...and the binary test seemed surprisingly low to me
<EdwinGrubbs> lifeless: it won't bring too much back since there is only one spph record for 1300 spr records that meets the condition. So, the filtering in python might only have to deal with eliminating a handful of records.
<sinzui> EdwinGrubbs, I essentially  did the reverse, of your suggestion. I converted the subquery to get the max id to be a table of only viable candidates: http://pastebin.ubuntu.com/530811/
 * sinzui now tries to do it the EdwinGrubbs approved way
<lifeless> ah right, deep history leading to a slow query
<lifeless> EdwinGrubbs: when we query for 200 rows
<lifeless> EdwinGrubbs: what would happen then
<lifeless> EdwinGrubbs: e.g. for  http://pastebin.com/7jC2vD7G
<EdwinGrubbs> lifeless: yes, I would like to do it in the database, but I don't know if getting just the max(spph.id) for each sourcepackagename is important or not. To do that in the database would require using a temp table in order to get rid of the subquery.
<sinzui> EdwinGrubbs, lifeless. I think this is the solution we want to achieve in the code http://pastebin.ubuntu.com/530817/
<EdwinGrubbs> sinzui: that only works for a single sourcepackagename.
<EdwinGrubbs> sinzui: oh wait
<sinzui> Edwin why? I see the table controls the SPNs
<sinzui> me tries a list
<sinzui> Edwin it does work with multiple SPNs
<EdwinGrubbs> sinzui: ok, that makes sense. I was thinking that you would run into problems with group by, but you are just grouping by the spr columns, so it all works out.
<sinzui> well, it certainly did not work until I add that
<sinzui> EdwinGrubbs, I do not need the outer "SourcePackageRelease.sourcepackagename IN ()" do I?
<EdwinGrubbs> sinzui: no
<sinzui> This is wicked fast
<sinzui> I am going to start a branch and watch the tests pass
<sinzui> gary_poster, henninge: I have a very fast query that fixes distroseries.getCurrentSourceReleases()
<lifeless> (37 rows)
<lifeless> Time: 186.584 ms
<lifeless> sinzui: thats for the big page
<lifeless> using your branch
<lifeless> sinzui: love your work
<sinzui> wow. I feel good
<lifeless> this is going to knock +packages right back to zilch on the timeouts chart I think
<sinzui> This will have to be stormified. I know how two write this in storm, but not sqlobject
<lifeless> hmm?
<lifeless> I mean thats great
<lifeless> but I can help you do it in situ if you want
<henninge> sinzui, lifeless: Is what you are doing related to the qastaging timeouts?
<sinzui> yes
<lifeless> henninge: do you mean on +packages?
<sinzui> this looks like it will also fix many other timeouts in production too
<henninge> no, the general timeouts we get on all kinds of pages.
<lifeless> henninge: no
<henninge> :(
<lifeless> henninge: we get timeouts because of a few reasons
<lifeless> a) cold cache effects in the db - its much smaller in memory that production
<lifeless> b) we have inefficient code and staging hardware shows this up
<lifeless> this is a case in point - sinzui is shaving many seconds off of a routine page
<lifeless> c) contention/thrashing in the appserver due to all the scripts running on the appserver staging host asuka
<lifeless> there is an rt open to address (c)
<lifeless> (a) - retry a few times, if it eventually works prod will probably chew it up happily
<lifeless> (b) - we need to fix our code. Which will help with (a) too
<henninge> but it seems to be related to certain revisions of the code
<henninge> it started on quastaging and when staging got updated with the same revisions it showed the same timeouts whereas before it (staging) was working fine.
<lifeless> henninge: what pages specifically
<henninge> all project homepages
<henninge> launchpad.net/anyproject
<henninge> all source packages
<lifeless> from 11888 to 11914 we had a very broken query for getCurrentSourceReleases
<henninge> launchpad.net/ubuntu/maverick/+sourece/anypackage
<lifeless> all the pages you're listing are covered by it; it should be tolerable now - the same as before 11888
<henninge> 11914 did not fix it, though
<lifeless> what EdwinGrubbs and sinzui are doing is  about to make it much better
<lifeless> henninge: this is one reason those pages are all slow on lpnet too
<sinzui> lifeless, henninge method is used in soyuz, translations, registry, and bugs pages. Anything that wants to know the current release of a package is going to be between 50 and 100 times faster
<lifeless> sinzui: yah
<lifeless> sinzui: note that production db is much faster
<lifeless> sinzui: so not all pages will zoom as much
<lifeless> https://launchpad.net/ubuntu/natty/+source/bzr
<rockstar> Does this error message in rabbitmq mean anything to anyone? It's preventing me from install launchpad-developer-dependencies. http://pastebin.ubuntu.com/530828/
<lifeless> but there are many pages which do this query that will benefit a great deal
<lifeless> rockstar: is it already running?
<rockstar> lifeless, no, it won't start.
<henninge> lifeless: I don't find those pages particularly slow but maybe I am just so accustomed to LP slowness ...
<henninge> ;-)
<rockstar> lifeless, when the package gets installed, it explodes and prevents anything else from being installed.
<lifeless> rockstar: on maverick ?
<rockstar> lifeless, yes.
<lifeless> hmm
<lifeless> I don't know sorry
<henninge> OK, let's wait and see the outcome of that work.
<lifeless> inet_tcp",{{badmatch,{error,duplicate_name
<lifeless> makes me think the socket is in use
<rockstar> lifeless, hm...
<lifeless> which would happen if you had a rabbit instance already running
<lifeless> e.g. if the devscript is buggy on upgrades
<henninge> sinzui: you have my r-c approval for landing that if it gets too late.
<rockstar> lifeless, oh!  Yeah, the other change on this laptop is the u1 setup, so I guess that makes sense. I completely spaced that.
<sinzui> henninge, thanks.
<henninge> PQM is scheduled to close in 3 hours
<rockstar> lifeless, everything are happy now.
<henninge> so you will need r-c ;)
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is closing at 22 UTC | firefighting: Lots of timeouts on qastaging!! | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<rockstar> lifeless, thanks for intervening between my head and the wall.
<lifeless> rockstar: that was it ?
<lifeless> rockstar: if so, please file a bug ... buggy package ;)
<rockstar> lifeless, well, I should also say that I run launchpad in a chroot.
<lifeless> rockstar: thats fodder for the bug report
<flacoste> henninge, sinzui: could these slow pages be related to the changes to add latest releases to the source package pages?
<flacoste> henninge, sinzui: that was added in a recent revision
<sinzui> flacoste, I do not think so. The method was unchanged this year except for lifelesses changes this week
<sinzui> flacoste I think this is PG 8.4
<flacoste> sinzui: hennige says that timeouts increased with a recent revision
<flacoste> as staging is now seeing the same timeouts than qastaging
<flacoste> whereas it wasn't until it was updated
<flacoste> and qastaging wasn't either yesterday
<sinzui> flacoste yes, we had an open join, but there were timeouts none-the-less
<sinzui> flacostewe had two landing to fix the issue, neither was substantial
<lifeless> I fluffed one
<lifeless> removed an unneeded table *constraint*, left the table in by mistake.
<lifeless> that went boom badly ;)
<flacoste> pages affected are:
<flacoste> all project homepages
<flacoste> all sourcepackages
<flacoste> according to henninge again
<lifeless> flacoste: we did just discuss this
<lifeless> 20 minutes ago
<flacoste> right, i read the backlog
<lifeless> kk
<flacoste> but it's not clear that have identified the issue
<lifeless> flacoste: we have an 8 second query
<lifeless> that will come down to 140ms
<lifeless> on qastaging
<flacoste> sure
<lifeless> we know they all use it
<lifeless> until its fixed we have no data about what lies behind it
<flacoste> well there is an alternative, which is to do a binary search to find the revision introducing the slowness
<lifeless> flacoste: 11888
<flacoste> lifeless: i was under the impression that we tried reverting 11888 and its two following fixes from qastaging, but that still resulted in all these pages timing out
<flacoste> but now, i'm not sure, it's possible that only the follow-up fixes were reverted...
<lifeless> flacoste: 11888 is a confounding factor
<flacoste> henninge: could you confirm/inform the above?^^^
<lifeless> flacoste: with 11888 present, any other flaws would have been magnified
<flacoste> right, but without it, we shouldn't see any more timeouts than before
<lifeless> flacoste: even if 11888 isn't the cause of all the issues, we can't be sure without running with 11888 reverted and the others present
<lifeless> flacoste: we're runnign 11887 live
<flacoste> i know
<lifeless> flacoste: so yes, I agree.
<lifeless> we've always seen more timeouts on qastaging
<lifeless> it has a 10 second timeout
<flacoste> but i thought we had made that tests on qastaging (no 11888, others present) and found that it was still timing out all over the place
<flacoste> but maybe, that's not the test that took place
<flacoste> let me check the branch...
<flacoste> lp:~henninge/launchpad/stable-revert
<flacoste> ah, no
<flacoste> only 11899 and 11914 were reverted
<flacoste> so your hypothese holds
<lifeless> ok
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777QS139 is qastaging.launchpad.net/bzr
<EdwinGrubbs> sinzui: can you look at these screenshots of the involvement portlet for the bugsupervisor versus the admins? https://devpad.canonical.com/~egrubbs/configuration/
<sinzui> Edwin It was easier to write an exception?
<sinzui> EdwinGrubbs, We wanted to hide the link once the tracker was configured
<EdwinGrubbs> sinzui: well, the progress bar doesn't make sense with just one link being shown, so an exception seems like the cleaner solution. It would also be odd to have a single link hidden under the "Configuration options" expander.
<sinzui> okay, I agree. Your approach is correct
<lifeless> flacoste: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777QS139
<henninge> flacoste: I did not revert 11888 in that branch because it has been on qastaging for a while without any trouble.
<lifeless> qastaging.launchpad.net/bzr - its the query sinzui is redoing
<henninge> flacoste: I am sorry for the misunderstanding
<flacoste> lifeless: ack
<sinzui> I am fixing the distroseries/source package problem illustrated by qastaging.launchpad.net/ubuntu/natty/+source/bzr
 * henninge has to go away for a bit again
<EdwinGrubbs> sinzui: can you review https://code.edge.launchpad.net/~edwin-grubbs/launchpad/bug-664788-configure-bugtracker-link-permission/+merge/40754
<sinzui> I will
<lifeless> thumper: perhaps we can cowboy in a squelch for the xmlrpc Fault
<lifeless> sinzui: I have a favour to ask
<lifeless> sinzui: add [rollback=11888] to your landing for the new query
<sinzui> yes lifeless
<sinzui> I will
<lifeless> sinzui: it will tell qatagger that https://bugs.launchpad.net/soyuz/+bug/662523 can be unblocked so the deploy report is accurate
<_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out <bad-commit-11888> <timeout> <Soyuz:Fix Committed by lifeless> <https://launchpad.net/bugs/662523>
<lifeless> sinzui: thank you!
<wgrant> lifeless: So that SPN fix worked?
<lifeless> wgrant: yes, and sinzui has an even more effective fix to make other uses of the query much more efficient
<lifeless> wgrant: https://qastaging.launchpad.net/~yavdr/+archive/stable-vdr/+packages?start=0&batch=204
<wgrant> lifeless: How fast is sinzui's?
<lifeless> 8700->100ms
<lifeless> wgrant: on production this is already tolerably fast, db server size yada yada yada
<wgrant> Nice.
<wgrant> Yep.
<lifeless> wgrant: but I expect a positive improvement all over.
<sinzui> wgrant: my mp has time summaries and SQL explains https://code.launchpad.net/~sinzui/launchpad/ds-getcurrentreleases/+merge/40756
<lifeless> sinzui: I'm really glad you guys dug into this
<sinzui> Making the SP and DS pages faster really has requires half a dozen engineers looking at the same number of objects
* henninge changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.11 | PQM is in release-critical mode | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> sinzui: we've 22 or so, looking all across the board
<sinzui> I fear milestones will be the last to fix :( I have time to return to that one next week.
<lifeless> +commentedbugs is the current most severe timeout
<lifeless> and stub has a fix \o/
<lifeless> I won't have time to do anything with it till week after next
<wgrant> Let me guess... it's querying badly to try to find comments with index != 0?
<lifeless> read the bug :)
<cody-somerville> http://www.jacobian.org/writing/buildbot/ci-is-hard/ <-- lmao. "Djangoâs big. The test suite is around 40,000 lines of code in something like 3,000 individual tests. We work constantly to speed up the test suite, but best case it still takes about 5 minutes to run. This means that our CI absolutely needs to be distributed â a single test server wonât cut it."
<lifeless> flacoste: ping
<flacoste> hi lifeless
<lifeless> flacoste: I think we need to treat this bzr thing as an emergency
<lifeless> flacoste: its very frequent
<poolie> lifeless, which?
<lifeless> poolie: the backtrace on push
<flacoste> lifeless: my understanding is that it's only annoying, not a real error
<lifeless> flacoste: our users don't know this
<lifeless> flacoste: perception
<poolie> this is the zope error being shown to the user?
<poolie> is there a bug?
<poolie> bug number
<lifeless> poolie: flacoste: https://bugs.launchpad.net/launchpad-code/+bug/674305
<_mup_> Bug #674305: bzr push occasionally reports AssertionError on terminal <codehosting-ssh> <xmlrpc> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/674305>
<wgrant> Also, doesn't it stop a scan from being requested?
<flacoste> lifeless: any idea of how we could fix this apart from escalating this RT?
<flacoste> wgrant: that would be new information
<lifeless> flacoste: here are the options I know about
<lifeless> wgrant: thats m understanding too
<lifeless> flacoste: a) escalate the RT
<wgrant> If it doesn't mean that no scan is requested, then we have bigger problems.
<lifeless> b) wedge in some retry code here - high risk
<lifeless> wgrant: there are multiple routes to trigger scans
<lifeless> wgrant: its possible a redundant route is saving us
<lifeless> wgrant: e.g. the disconnect hook
<wgrant> Possibly.
<lifeless> c) push the mailman improvement and hope its enough
<lifeless> d) disable other services like codeimport that use the same service
<flacoste> c and d looks like the main option at this time
<flacoste> can we get confirmation that scan isn't triggered?
<flacoste> i don't get any errors from here fwiw
<lifeless> it seems to be a couple of users an hour - which, because its not (appearing-to-be) localised to product/bug like other timeouts, particularly confusing and harmful to our users.
<lifeless> theres no obvious rationale they can connect it to
<lifeless> wgrant: we're talking with ops now
<lifeless> Time Out Counts by Page ID
<lifeless> Hard	Soft	Page ID
<lifeless> 570	7384	CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless> 211	32	Person:+commentedbugs
<lifeless> 164	561	CodehostingApplication:CodehostingAPI
<lifeless> 44	156	BugTask:+index
<lifeless> 8	10	ProjectGroup:+milestones
<lifeless> 6	305	Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless> 5	259	Distribution:+bugs
<lifeless> 5	14	Person:+bugs
<lifeless> 5	7	DistroSeries:+queue
<lifeless> 5	4	Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless> bah sorrry for the formatting
<lifeless> flacoste: ^ turning off code imports is probably the fastest thing we can do
<lifeless> mbarnett: how hard is it to disable all code imports ?
<lifeless> mbarnett: we should be able to see an immediate drop in that netstat over a couple of minutes if thats were to help
<lifeless> flacoste: 570 7384 CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless> flacoste: 164 561 164I561ICodehostingApplication:CodehostingAPI
<flacoste> lifeless: good suggestion
<lifeless> mbarnett: please turn off the importds
<lifeless> mbarnett: And keep watching that netstat
<lifeless> flacoste: https://devpad.canonical.com/~lpqateam/lpnet-oops.html#time-outs is where I'm looking
<mbarnett> no more imports should be fired off after any currently running complete.
<lifeless> flacoste: look at this:
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777XMLP1011
<lifeless> SQL time: 17 ms
<lifeless> Non-sql time: 15074 ms
<wgrant> Ow.
<lifeless> flacoste: this is why I want a) single threaded appservers and b) in the main cluster
<flacoste> right
<flacoste> the GIL hypothesis
<lifeless> yes
<lifeless> for instance
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1777XMLP4675
<lifeless> but i have another hypothesis
<lifeless> if we start the timer too early
<lifeless> a deep queue could look like this as well
<lifeless> see the 4675 in particular its a soft timeout
#launchpad-dev 2010-11-13
<lifeless> poolie: interesting blog-link
<poolie> thanks
<poolie> i think we/i could do with more of that kind of focus too
<poolie> there's a principle something like "if you don't do feature/project X, will you regret it in a year"?
<poolie> that probably doesn't precisely work, but you know what i mean
<poolie> perhaps
<lifeless> yeah
<poolie> whereas there really are other values Y that, if you don't get them done, you will regret
<poolie> X might seem smaller so you might as well just try to do it along the way, but ... they add up, and estimation is impossible
<lifeless> regret based scheduling
<lifeless> I like it
<poolie> :)
<thumper> turning off code imports is very simple
<thumper> we just stop the importers
<thumper> take the importers offline
<lifeless> thumper: a maintenance file has been created on disk
<lifeless> thumper: see -ops backlog for the stats we looked at
<thumper> lifeless: yeah, I see that there is only one import running
<lifeless> bbiab
<lifeless> flacoste: its not fixed
<lifeless> mtaylor just had one
<mtaylor> flacoste: http://paste.drizzle.org/show/165/
<lifeless> poolie: have you spoken with flacoste already?
<poolie> not on the phone and only briefly here
<wgrant> lifeless: Do we know what has caused the excessive load to appear?
<lifeless> wgrant: the machine itself is low load
<lifeless> wgrant: but the appserver has all four threads active all the time
<lifeless> wgrant: and we're seeing things like I pasted - with gaps like: 10.	37	2ms	SQL-launchpad-main-master	UPDATE Branch SET last_mirrored=CURRENT_TIMESTAMP AT TIME ZONE 'UTC', last_mirrored_id=%s WHERE Branch.id = %s
<lifeless> 11.	15075	0ms	SQL-launchpad-main-master	INSERT INTO Job (status, attempt_count) VALUES (%s, %s) RETURNING Job.id
<wgrant> lifeless: :/
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      571 / 7556  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>      212 /   32  Person:+commentedbugs
<lifeless>      169 /  590  CodehostingApplication:CodehostingAPI
<lifeless>       46 /  157  BugTask:+index
<lifeless>        8 /   10  ProjectGroup:+milestones
<lifeless>        6 /  311  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        5 /  262  Distribution:+bugs
<lifeless>        5 /   15  Person:+bugs
<lifeless>        5 /    7  DistroSeries:+queue
<lifeless>        5 /    4  Archive:EntryResource:getBuildSummariesForSourceIds
<cody-somerville> lifeless, I noticed your comment on LP #674305
<_mup_> Bug #674305: bzr push reporting AssertionError on terminal <codehosting-ssh> <oem-services> <xmlrpc> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/674305>
<cody-somerville> I just got the error myself... dunno if thats useful information or not but thought I'd let you know
<lifeless> cody-somerville: its widespread
<lifeless> cody-somerville: I get it 2/3 pushes.
<lifeless> we're working on it
<cody-somerville> lifeless, Is the docstring for branchChanged method on the BranchFileSystemClient class wrong? "Mark a branch as needing to be mirrored. ..." - shouldn't that say 'scanned' instead of mirrored since hosted branches don't need to get 'mirrored' anymore?
<lifeless> stale, yes
<lifeless> we do still mirror branches from other servers
<cody-somerville> right
<LPCIBot> Project db-devel build (141): FAILURE in 3 hr 43 min: https://hudson.wedontsleep.org/job/db-devel/141/
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11923,
<LPCIBot> 11924, 11925 included.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11918,
<LPCIBot> 11919, 11920, 11921, 11922 included.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11916,
<LPCIBot> 11917 included.
<LPCIBot> Project devel build (224): FAILURE in 3 hr 38 min: https://hudson.wedontsleep.org/job/devel/224/
<LPCIBot> Launchpad Patch Queue Manager: [release-critical=henninge][r=EdwinGrubbs][ui=none][bug=674672][rollback=11888]
<LPCIBot> Make distroseries.getCurrentReleases faster.
<LPCIBot> Project db-devel build (142): STILL FAILING in 3 hr 39 min: https://hudson.wedontsleep.org/job/db-devel/142/
<LPCIBot> Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11926
<LPCIBot> included.
<lifeless> morning
 * cody-somerville hugs lifeless.
 * cody-somerville hugs elmo.
<lifeless> cody-somerville: thanks
<lifeless> sinzui: hi
<lifeless> sinzui: just want to say, love what you're doing to bugs atm.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project db-devel build (143): FIXED in 3 hr 59 min: https://hudson.wedontsleep.org/job/db-devel/143/
<lifeless> jml: hi
#launchpad-dev 2010-11-14
<lifeless> jml: I'm off but - rev 130 removes the note about actually spinning the reactor; did you decide that that actually isn't relevant to users?
<lifeless> === Top 10 Time Out Counts by Page ID ===
<lifeless>     Hard / Soft  Page ID
<lifeless>      384 /  578  CodehostingApplication:CodehostingAPI
<lifeless>      193 /   41  Person:+commentedbugs
<lifeless>       94 /  832  CodeImportSchedulerApplication:CodeImportSchedulerAPI
<lifeless>       39 /  151  BugTask:+index
<lifeless>       11 /  192  Distribution:+bugs
<lifeless>       11 /    5  DistributionSourcePackage:+addquestion
<lifeless>       10 /   53  MaloneApplication:+bugs
<lifeless>       10 /   18  Archive:EntryResource:getBuildSummariesForSourceIds
<lifeless>        5 /  229  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless>        5 /    2  ProjectGroup:+milestones
<lifeless> wgrant: seen anyone complaining about assertion errors for bzr?
<wgrant> lifeless: There've been no complaints anywhere that I watch.
<wgrant> So it seems like it's all fixed.
<lifeless> cool
<lifeless> android tethering++
<lifeless> elmo: if you have'nt crashed yet - thanks heaps, seems we're all good.
<lifeless> n't
<lifeless> ok, boarding is up.
 * lifeless waves
<mwhudson> morning
<wgrant> thumper: Around?
<mwhudson> wgrant: not feeling well today, apparently
<wgrant> Ah :(
#launchpad-dev 2011-11-07
<wgrant> Someone linked to bug #1234 on the ISO tracker.
<wgrant> I expect.
<wgrant> Also, ubuntuqa uses edge.
<lifeless> another bug to file
 * StevenK grumbles at TALES being useless.
<lifeless> .
<StevenK> lifeless: http://pastebin.ubuntu.com/730574/
<lifeless> win
<lifeless> uhm, nothing jumps out at me
<StevenK> What's annoying me is that I saw it last week, and I fixed it, and I can't remember how.
<StevenK> wgrant: Do you have any clues from the above traceback?
<wgrant> Not really.
<StevenK> No traceback when not logged in. Curious.
<wgrant> Hardly.
<wgrant> It's in all probability the user info in the top right corner.
<wgrant> That's what uses link-display-id or whatever it is the most.
<StevenK> But factory.makePerson() should set all of that up, surely ...
<wgrant> Which layer are you in?
<StevenK> DatabaseFunctional
<StevenK> wgrant: Is DatabaseFunctional too low?
<wgrant> StevenK: shouldn't be.
<StevenK> wgrant: Same error with LaunchpadFunctional :-(
<StevenK> Oh, damn it, I know what I'm missing
<huwshimi> Is anyone else getting sporadic "Exception: RabbitMQ server is not running." errors when trying to run 'make run'?
<lifeless> huwshimi: not I
<lifeless> huwshimi: but make run isn't a common thing for me atm
<lifeless> huwshimi: poolie had that recently, but I don't know what was causing it for him
<wgrant> I use make start and make run frequently.
<wgrant> No trouble.
<wgrant> But this is lucid.
<huwshimi> hmmm
<StevenK> AssertionError: queries do not match: 0 != 1817
<StevenK> Whee!
<lifeless> can has review ? https://code.launchpad.net/~lifeless/python-oops-twisted/bug-873030/+merge/81427
<poolie> haha yes
<StevenK> And 200 SPPH queries, so success
<poolie> huwshimi, that's actually launchpad's cute way of saying "rabbitmq server *is* running" :)
<lifeless> bbiab, -> shopping
<poolie> i put up a patch to make it give a better message but i think it's not deployed yet
<poolie> look for a stray rabbit process and kill it
<huwshimi> poolie: Oh right. That is cute
<StevenK> poolie: It doesn't need to be deployed, it just needs to be on devel
<poolie> apparently our tone of voice is 'extremely sarcastic'
<poolie> StevenK, does that come from a branch?
<StevenK> poolie: lp:launchpad
 * poolie goes to look
<poolie> no, it needs to be fixed in rabbitfixture
<StevenK> Ah
<StevenK> Where does that live?
<poolie> huwshimi, was that in fact your problem?
<wgrant> It's an egg.
<poolie> if so that nudges me to make sure my thing lands
<huwshimi> poolie: I'm not sure, I'll try next time it complains
<wgrant> The only things that come from branches are bugs.
<poolie> by 'deployed' i meant 'packaged updated etc'
<poolie> huwshimi, if there's an orphaned instance i think it will happen again next time until you do kill it
<huwshimi> poolie: it hasn't complained for a while
<poolie> wgrant, ha ha
<poolie> but, what are the things in sourcecode/ then?
<StevenK> poolie: We have three ways of adding dependencies.
<wgrant> poolie: They're bugs.
<wgrant> poolie: They shouldn't be branches.
<poolie> i know this
<poolie> StevenK, i know there are 3 ways
<poolie> is this wgrant's way of saying that one of those ways is deprecated?
<wgrant> Or at least generally discouraged, yes.
<wgrant> (also, there's a fourth way: including them in the tree)
<poolie> that's what you mean by saying they're bugs?
<wgrant> Yes.
<StevenK> Oh, ugh, I didn't know we linked from lib into sourcecode.
<wgrant> That's how sourcecode works :)
<StevenK> That's disgusting, kill it.
<StevenK> wgrant: I have a test that works!
<StevenK> wgrant: It's a bit evil at the moment since it creates 100 BugNominations, but it proves the query count is excessive
<StevenK> Oh, it's even worse. lib/mailman is a copy of sourcecode/mailman
 * StevenK tries to work out how to prefetch the entire batch to seed Storm's cache
<huwshimi> I'm guessing in Launchpad there is no intelligent way to create user icons from there logos or mugshots right?
<wgrant> huwshimi: No.
<huwshimi> #$%
<wgrant> huwshimi: For projects we ask them to upload a separate 14x14 image.
<wgrant> Particularly since you can't sensibly scale much down to 14x14
<huwshimi> well you can, we just don't have the infrastructure
<lifeless> image scaling without jaggies is not -all- that simple. We might have been better off asking for a single svg.
<wgrant> O_o
<wgrant> not sure if serious
<huwshimi> No,
<huwshimi> this is ridiculous
<huwshimi> why do we have to overthink everything
<huwshimi> this the web
<huwshimi> people scale images ALL THE TIME
<wgrant> Yes.
<huwshimi> and they don't have massive scaling SVG mega frameworks, just to do the simplest of things
<wgrant> Using SVG here is entirely ridiculous.
<wgrant> But I'm not convinced that we can usefully scale normal sized images all the way down to our 14x14 icon format.
<lifeless> it all depends on what the criteria you're aiming for are. If we are happy with browser scaling, just set attributes on the image.
<wgrant> Anything below 32x32 tends to be pretty hideous, IME.
<lifeless> If we aren't happy with browser scaling, I think its a safe assumption we won't be happy with unattended resizing either.
<lifeless> (Because browsers keep iterating on their scaling logic to make things look good, so they are pretty solid)
<huwshimi> lifeless: Really and use all the extra bandwidth, you really think that's the best practice here?
<huwshimi> lifeless: And how about background images?
<huwshimi> lifeless: Shall we exclude older browsers from having scaled images too?
<lifeless> huwshimi: I don't know - I'm not *setting* the goals here -
<lifeless> huwshimi: you'll know much better than I the limitations of the browsers we're dealing with
<huwshimi> lifeless: Right, but this is a solved problem. Let's not invent a new ethos for handling images
<lifeless> huwshimi: thats great
<lifeless> huwshimi: I didn't realise it was so well solved :)
<wgrant> Hmm? Server-side thumbnailing isn't exactly an uncommon, new concept...
<huwshimi> wgrant: Thankyou
<StevenK> Perhaps lifeless is one of our four IE6 users.
<lifeless> wgrant: not down to 14x14
<wgrant> I recall implementing it in some of my early web projects in like 2002, and it was probably popular before then and 10-year-old me just didn't realise.
<wgrant> No.
<wgrant> Not down to 14x14.
<wgrant> I agree that we can't do that.
<lifeless> thumbnailing is one thing
<lifeless> but extremely large, or extremely small scales are damn tricky
<huwshimi> lifeless, wgrant: You seriously think we shouldn't scaled down to ~14 (or 20 or whatever we want for small icon sizes)? You seriously think most people aren't just going to scale it before they upload it anyway (if they're a project)
<lifeless> heck, Next machines in 1997 happily thumbnailed images in a directory listing, if memory serves.
<wgrant> It is a rare website that uses small icons like that.
<wgrant> We should probably stop.
<wgrant> huwshimi: No, I argue that we shouldn't be using 14x14 icons for projects and people.
<wgrant> I don't think anyone else does.
<wgrant> They use more sensible stuff of at least 20x20, from what I've seen.
<lifeless> huwshimi: I know that its hard to get a good representation of a logo at 14x14 without manual fixups - e.g. an icon editor
<lifeless> huwshimi: I don't know if our users care about that
<lifeless> huwshimi: and I don't know if we need 14x14 for any reason other than having it already.
<wgrant> Exactly.
<lifeless> huwshimi: I am not about to argue for complexity we don't need : see francis bug report for the cost of complexity on us as a team.
<huwshimi> right, but that shouldn't stop us from forcing users and ourselves into set sizes (192, 64, 14)
<lifeless> huwshimi: one answer, for instance, is to scale by default and allow overrides for folk to fix things up
<lifeless> stub: https://code.launchpad.net/~lifeless/python-oops-twisted/bug-873030/+merge/81427 (OCR penance time)
<lifeless> huwshimi: if your new design will permit a larger micro size, it may make a substantial difference to auto-scaled images [experimentation or citatin needed]
<StevenK> My prefetch query doesn't impact the query count :-(
<huwshimi> Ugh, it just turns my job into a pipe-dream, as everything I design requires weeks of work to implement the simplest thing (and would never get priority over other work)
<StevenK> It could just be a stupid query
<stub> lifeless: k
<lifeless> StevenK: might be the cache size as well
<wgrant> StevenK: dev runs with a storm cache of only 100 objects.
<wgrant> StevenK: Product uses 10000
<wgrant> Production, that is.
<lifeless> huwshimi: are there js only scaling libraries? I assume there must be
<lifeless> huwshimi: you could scale on the client and upload three things from one js form
<lifeless> huwshimi: s/you could/could you/?
<StevenK> wgrant: Right, so my test might just be too large at the moment
<huwshimi> lifeless: Well there kind of are, but they still require the client to download the full-size image
<lifeless> StevenK: or just about right
<StevenK> wgrant: TBH, I'm tempted to just blame my current query
<lifeless> huwshimi: I meant in the edit form for the person; do all the prep on the client
<StevenK> lifeless: I'm creating 100 bug nominations, and then browsing the view with ?batch=100. It seems excessive to me, and I wrote it.
<lifeless> huwshimi: that would avoid nearly-all backend changes
<huwshimi> lifeless: Oh, I see. Well there's no javascript version of PIL, Imagemagick etc. no.
<lifeless> huwshimi: which is where we are not really prepped to do dynamic stuff
<lifeless> huwshimi: we have reasonable tools to run e.g. imagemagick in the backend, but they aren't trivial to use if you're not familiar with them, which IIRC was one of your goals
<stub> lifeless: So you do want a grammar and coding style review, or wait for someone who knows a bit about twisted and wsgi?
<lifeless> stub: jamesh is eyeballing it
<lifeless> stub: I hope he's wiring it up to test in the u1 stack
<lifeless> stub: as much of a review as you want to give would be great.
<huwshimi> lifeless: I'm not sure quite what you mean. It's pretty easy to set up PIL for thumbnailing etc. (it even has .thumbnail())
<lifeless> huwshimi: I would expect that to be done in a 'job' straight after the upload succeeds, to avoid timeout issues.
<lifeless> huwshimi: which involves a new table; a new config entry to run under the shared job environment, some tests for your code, and a Job definition
<StevenK> And 64 million interfaces
<lifeless> we're iterating towards this being lighter still
<huwshimi> lifeless: Sure, or it is often set up as a service when required (e.g. launchpad.net/thumbnail-service/14/14/file.jpg)
<lifeless> and, pick yourself off the flow, it used to be much worse
<lifeless> huwshimi: s/flow/floor/
<lifeless> huwshimi: I'd have some scaling worries about a JIT service, though yes we could do one.
<stub> lifeless: what is the number after your name in NEWS?
<lifeless> stub: did I forget the # ? its a bug number
<huwshimi> lifeless: Right, but anyway, I'll stop talking about this now as it is only a frustrating mental exercise
<stub> lifeless: And the word 'Bug' :)
<lifeless> stub: not needed ;)
<lifeless> stub: the rest of the file - and in other projects too  - just does #1234
<stub> Yay for confusing
<lifeless> seen a Debian/changelog recently ? :)
<stub> 'Better than a Debian changelog' is damning with faint praise
<stub> But at least with the '#' it won't look like you are giving your body measurements or date of birth
<lifeless> :)
<StevenK> Hm, my query doesn't help with only 10 bug nominations either, still 197 queries.
<lifeless> StevenK: is your theory that they *can* come out of cache false?
<lifeless> StevenK: if the deeper code is doing anything *other* than attribute traversal, you can't optimise by cache loading.
<lifeless> StevenK: you would need to actually setify the logic (which is a good thing anyhow)
<StevenK> lifeless: I thought I could cache the bugnominatons themselves in the first case.
<StevenK> But it seems that is pointless
<StevenK> lifeless: So, I'm tempted to declare that cache seeding won't help, but then I'm stuck about what I can do instead
<lifeless> have you identified the cause of the problem ?
<StevenK> I've identified what is causing the masses of SPPH queries.
<lifeless> go on
<StevenK> I'm not sure if that is the cause, but it's one thread to tug on.
<wgrant> StevenK: This is for latest_published_component?
<lifeless> StevenK: ok, so can you pastebin the call stack ?
<StevenK> wgrant: So latest_published_component wants SPPH, and I then I think verifyUpload() does too
<wgrant> StevenK: How are you preloading it?
<wgrant> You'll need to turn it into a cachedproperty.
<wgrant> And populate it manually in a set-based manner.
<StevenK> wgrant: Currently, I was trying to cache the bug nominations themselves first, but that hasn't changed anything.
<wgrant> Are there lots of BugNomination queries?
<StevenK> wgrant: 2 per row
<wgrant> That doesn't make much sense.
<wgrant> Do you have a query log?
<StevenK> Yup
<StevenK> wgrant: http://people.canonical.com/~stevenk/test_query_log.txt
<lifeless> none of those are attribute access queries
<wgrant> Ah, so SPPH isn't really the problem at all.
<wgrant> The whole thing is.
<lifeless> they are all explicit lookups from procedural code AFAICT in a quick glance
<StevenK> Heh
<lifeless> cache cannot help you at all
<wgrant> Well, the built-in Storm cache can't.
<lifeless> ^storm ...
<wgrant> cachedproperty can
<wgrant> So, you are going to need to radically setify canApprove.
<wgrant> Unless you want to do it all manually beforehand, which I would discourage.
<StevenK> Right, so canApprove() is the crux of the problem. I certainly agree.
<lifeless> so what you need to do is write a set based version of it
<wgrant> s/crux of/
<lifeless> that takes many nominations (or something like that) and figures out the answer for all of them.
<wgrant> Exactly.
<lifeless> make the existing canApprove forward to that new function
<lifeless> and finally migrate this code path over to use the new function directly with all of the nominations for the page
<StevenK> But canApprove() is hideous :-P
<lifeless> yes
<lifeless> take small bits
<lifeless> *bites*
<StevenK> Hm, there is a BugNominationSet ...
<StevenK> So there are two issues I can see.
<StevenK> 1. My batch in the view is bugtasks.
<StevenK> 2. canApprove() sucks
<lifeless> the other alternative is to eliminate nominations; is possible that the new security rules mean they are not useful now.
<lifeless> so 1) is something that cachedproperty may help with
<StevenK> Oh, sure, eliminating nominations doesn't sound controversial *at all* :-P
<StevenK> wgrant: Can I beg you for a pre-impl about this new method?
<wgrant> It's pretty simple really, but OK.
<poolie> wgrant, re bug 435905, i wonder if you would agree we should just close it
<poolie> and keep wrapping roughly where we currently do
<poolie> until we get proper markup
<lifeless> 3K timeouts. Hmm
<wgrant> lifeless: Network outage overnight.
<lifeless> stub: so no review results ?
<lifeless> stub: or just the # is it ?
<wgrant> poolie: Do you really mean me?
<poolie> i do; istr you had some opinion about always using greasemonkey before
<poolie> and your 'fix the defaults' reminded me of it too
<wgrant> Ah.
<wgrant> Well, I think the fix for it is markup.
<stub> lifeless: Getting there, distractions distractions
<poolie> then i agree
<poolie> thanks
<lifeless> stub: ;)
<lifeless> aieee, gmail, what have you done
<wgrant> poolie: The current uploads work mostly. We shouldn't keep hacking it to make various special cases work. We should do what the rest of the world does.
<wgrant> We should follow what the rest of the world does for an awful lot of things.
<wgrant> We needed to do our own thing 7 years old.
<wgrant> s/old/ago/
<wgrant> But it's no longer 7 years ago, and the world has moved on.
<poolie> 'current uploads'?
<wgrant> Er.
<wgrant> Sorry, multiple conversations.
<poolie> current css?
<wgrant> Yes.
<poolie> i agree with the general thing anyhow
<poolie> i do wonder how much say feature flags or timelines are available externally in other frameworks
<lifeless> rpm.net
<poolie> ?
<poolie> it lapsed?
<lifeless> huh
<lifeless> let me find their current home
<poolie> did you mean rpmfind?
<wgrant> Was that the fork, or the not fork?
<wgrant> I can't remember.
<poolie> the description of the hardware on http://rpmfind.net/ is awesome
<poolie> though i bet we have some that is older
<lifeless> http://newrelic.com/
<lifeless> huh, they have added python since I last strolled by
<lifeless> http://newrelic.com/docs/python/instrumented-python-packages
<poolie> mm
<poolie> there may even be free versions
<poolie> s//alternatives
<lifeless> well, oops-tools & sentry are in that space, though neither is anywhere -near- as advance.
<lifeless> oops-tools is the most metric enabled open end-to-end one I know of for Python
 * stub wonders why magic asynchronous tests are a good idea
<lifeless> stub: magic?
<poolie> is there any kind of trick about $PYTHONWARNINGS?
<stub> test methods returning deferreds with callbacks that the test runner somehow executes because the test case class has a property set to a particular value
<lifeless> stub: well, you need *something*
<lifeless> stub: this has the nice property that its extensible to different async environments in one library (vs e.g. twisted trial)
<stub> because tests being nearly the ultimate in synchronous code require asyncronicity
<lifeless> yeah, they very much do when the code is callback based
<lifeless> and the callback engine doesn't guarantee pause/resume/single step facilities
<stub> but a helper invoked explicitly would remove the magic
<stub> And would even allow you to test post state without registering these tests as callbacks.
<lifeless> see under callback engine guarantees
<stub> d.addCallback(lambda _: self.assertEqual(expected, self.calls)) would be much more readable as just self.assertEqual(expected, self.calls)
<stub> That doesn't mean anything to me.
<poolie> but there's a difference ,right
<poolie> the assertion is: _eventually_ when this is called, there are N calls
<lifeless> stub: I agree that it would be nicer if it was single stepped and looked just like regular python
<stub> I don't see the difference between 'd.addCallback(lambda: blah blah assert blah); return d' and 'self.run_deferred(d); self.assert'
<lifeless> in principle inlineCallbacks could be used.
<huwshimi> A review for some lovely person, if possible: https://code.launchpad.net/~huwshimi/launchpad/avatars-everywhere-712894/+merge/81430
<stub> Yer, just the more I see twisted code the more I think that twisted was a bad idea and should never have been shoe horned into Python.
<stub> Code that makes you go 'blergh' when you haven't gone through the indoctrination process keeps you isolated.
<lifeless> huwshimi: I've given it a once over
<poolie> ah, the trick is it's new in 2.7?
<lifeless> poolie: what is new in 2.7 ?
<poolie> $PYTHONWARNINGS
<lifeless> ah, dunno.
<poolie> added by one B. Warsaw
<nigelb> *yawn*, morning
<poolie> hi nigel
<nigelb> Hey poolie :)
<poolie> is there some easy way to override buildout and make it use a dependency from a tree, while i'm futzing with it?
<poolie> i guess replace the egg with a symlink
<wgrant> poolie: You should be able to symlink the tree into the root of your LP tree, then add the dir name to the 'develop' setting in buildout.cfg.
<lifeless> poolie: bin/buildout buildout:develop=". ../path/to/tree"
<lifeless> wgrant: no need to symlink
<wgrant> Oh, or that.
<wgrant> Didn't know that worked.
<poolie> wow, ok
<poolie> so those two trees come before everything else?
<wgrant> develop should just be '.' by default... what do you mean?
<poolie> huwshimi: whoa!
<poolie> wgrant, i meant before things pulled in from eggs
<poolie> huwshimi, that would be a good thing to add behind a user 'labs' switch
<poolie> though i'm not suggesting you block on adding one
<wallyworld_> stub: hi, i have a sql question for you if you have a moment sometime
<huwshimi> poolie: Yeah, we'll see how much it breaks launchpad :)
<StevenK> wgrant: http://pastebin.ubuntu.com/730709/
<poolie> at least a flag would be good
<StevenK> wallyworld_: Just ask, it may not be stub that answers.
<huwshimi> poolie: Wouldn't labs be similar to launchpad-beta-testers?
<wallyworld_> StevenK: it's a performance related thing
<StevenK> wallyworld_: Then lifeless or wgrant would probably also have opinions
<wallyworld_> fair enough. although i didn't want to bug them :-)
<wallyworld_> but anyway, i think this will be ok, hopefully: https://pastebin.canonical.com/55351/
<wallyworld_> it is to check whether a given person owns any pillar
<wallyworld_> and i'm looking to use it amongst other places in a storm validator
<wallyworld_> so i want it to perform well
<wallyworld_> there's a similar query for isSecurityContact
<poolie> huw, it would be similar with a few differences
<poolie> - the ui is more obvious than joining/leaving a team
<poolie> - teams are kind of used for too many things
<poolie> - we can have more than one concurrent experiment
<stub> wallyworld_: That query seems very odd.
<poolie> - we could hook it up to feedback
<poolie> - oh, users would have a clearer idea what things were experimental
<wallyworld_> stub: maybe. the unit tests pass :-)
<wallyworld_> but if it can be improved...
<poolie> at the moment they just see everything (though i believe they're going to get some kind of systematic indication in future)
<wgrant> wallyworld_: I'd prefer that it was in a setter, not a storm validator.
<stub> Yes, but selecting the first row from the person table if the EXISTS matches, throwing it away and returning true...
<wallyworld_> stub: the only reason is use select from person was to keep dtore.find happy
<wallyworld_> stub: i can constuct sql without any table "select exists ...." but storm complains
<wallyworld_> i guess i could do a store.execute() with raw sql?
<wallyworld_> by complains i mean store.find() says "there's no table"
<wallyworld_> if i try and run something like "select exists (select something where condition)"
<wallyworld_> wgrant: storm validators are bad?
<wallyworld_> they seem to be used in the type of thing i'm doing already
<wallyworld_> eg is_valid_person and friends
<wgrant> Lots of things are already used for lots of types of things :)
<wgrant> I think Storm validators impede clarity.
<wgrant> And are often used to do very bad things.
<wallyworld_> i don't agree that they hinder clarity
<wallyworld_> anything can be misused
<wgrant> Storm validators become problematic when eg. you want to pass an argument into them.
<wgrant> For example, admins might be allowed to override some restrictions.
<stub> https://pastebin.canonical.com/55352/ might work better
 * wallyworld_ looks
<lifeless> huwshimi: I suspect it will need the eager loading changes I reference done before you can enable the flag :(
<lifeless> huwshimi: Its very cool to see it being done
<wgrant> lifeless, huwshimi: It also has an XSS hole or two.
<stub> wallyworld_: But what you have is just as fast despite looking a bit WTF?
<wallyworld_> stub: thanks. i'll use that. although the one i pasted also works, but your is easier to grok i guess.
<huwshimi> lifeless: Oh, sorry I mis-understood
<wallyworld_> stub: i guess i was trying to avoid the count(*)
<stub> wallyworld_: I'm not sure of how to shoe horn it into Storm
<lifeless> huwshimi: I was probably unclear
<huwshimi> lifeless: It's all good
<lifeless> huwshimi: Landing with a flag is fine, I just think its near-certain that enabling it will make things timoeut
<huwshimi> lifeless: Sure
<lifeless> huwshimi: so you could, if you think I'm wrong try it and possibly save yourself some time
<stub> wallyworld_: If you go for yours, change the UNION to UNION ALL
<wallyworld_> stub: i think because yours does a select from... i can get it in
<lifeless> (IMBW)
<lifeless> I don't think I am, in this case.
<huwshimi> lifeless: What's involved in making the eager loading changes?
<wallyworld_> stub: ok. i'll have a look at it. if it all goes pear shaped, i can always revert to my original sql since the performance is ok
<wallyworld_> thanks
<huwshimi> lifeless: Is it something I can take a look at?
<stub> wallyworld_: We are looking at <10ms, so whatever ;)
<wallyworld_> stub: cool. the perf figure is what i was after. thanks :-)
<wallyworld_> wgrant: i agree about the argument passing thing. so validators are good for invariant conditions
<wgrant> wallyworld_: It's *presently* invariant.
<huwshimi> wgrant: Where are the XSS holes? This is mostly just existing code, but I'm happy to fix it up a bit
<wallyworld_> yes. i don't see this one changing though
<wallyworld_> but perhaps it could
<lifeless> huwshimi: the key function is the one I mentioned in my review
<wgrant> StevenK: if reason is not None and not (is_new and getComponentsForUpload)
<lifeless> getPrecachedPersonsFromIDs
<lifeless> which calls
<lifeless> _getPrecachedPersons
<lifeless> the related but separate cacheBrandingForPeople already grabs all three sizes
<lifeless> so you don't need to touch that
<huwshimi> lifeless: Ah I see
<wgrant> huwshimi: The URL contains user-controlled content (in the filename), and it's now in the page unescaped. cgi.escape is also no good for attribute values -- it doesn't escape quotes by default.
<wgrant> huwshimi: I recommend that you use structured()
<wgrant> In c.l.w.menu
<huwshimi> lifeless: I have to go deal with a whinging baby, but I'll take a look at this tomorrow.
<huwshimi> wgrant: Thanks, I'll take  a look
<StevenK> wgrant: http://pastebin.ubuntu.com/730715/
<lifeless> huwshimi: cool. Please feature flag it *anyway* because we may well find that this change causes brand new late evaluations
<huwshimi> lifeless: Sure, no problems. Thanks for that
<lifeless> huwshimi: I may seem to be being overly cautious here, but believe me, LP is designed to be slow
<lifeless> so we're fighting all the time
<huwshimi> lifeless: Haha, no I totally understand
<poolie> lifeless,  thanks for the tip about setup.py
<poolie> the one in oops-repository was trivially broken
<poolie> i pushed a fix
<lifeless> oh thanks
<poolie> i get a warning about extras_require and install_requires
<poolie> but perhaps they're used by one of the other cluster of tools
<lifeless> yah
<lifeless> thanks for the review stub
<lifeless> stub: shall we catchup ?
<stub> lifeless: sure.
 * stub boots laptop
<poolie> lifeless, still here?
<poolie> i thought i would need to dpkg txfixtures so it could be installed on the buildd guests
<lifeless> poolie: ah possibly. I don't know enough there sorry :(
<poolie> pretty sure
<poolie> at any rate if we're going to split this out as a dependency of buildds it will need to get there somehow
<lifeless> stub: d/c ?
<adeuring> good morning
<jelmer> does anybody know what is up with strawberry (qastaging's code importer) ? It doesn't seem to be processing code imports.
<StevenK> jelmer: Ah, you're attempting to QA r14243?
<StevenK> Hm, no rvba today.
<jelmer> StevenK: no, I was hoping to QA bug 485932
<StevenK> jelmer: Yes, which is linked to stable r14243. :-)
<StevenK> allenap: Could you try and QA r14234, r14250 and r14260 today at some point?
<wgrant> jelmer: qastaging had some download-cache issues earlier. Might want to poke a LOSA to poke strawberry.
<bigjools> jelmer: are you able to do some QA please? http://launchpad.net/bugs/485932
<StevenK> bigjools: He just said he was ...
<bigjools> ah, /me reads backscroll
<jelmer> wgrant, StevenK: I'll check with a LOSA, thanks
<StevenK> bigjools: Have you bugged rvba for the three revisions he has to do? :-)
<bigjools> StevenK: I see you're on the same jihad as me
<lifeless> bigjools: what rt should I tweak to add the oopses exchange to the txlpongpoll permissions (on all three sites)
<StevenK> bigjools: Yes. The deployment report has been Red Squaded.
<lifeless> s/to add/to get losas to add/
<bigjools> lifeless: I'd do a new one
<bigjools> lifeless: but failing that, 48109
<lifeless> new one done
<lifeless> allenap: hi; why ez-setup ?
<lifeless> allenap: oh, I see - why don't we put that in the shared LP download cache ?
<allenap> lifeless: Otherwise it tries to get it from the network.
<allenap> lifeless: D'oh, ignore that.
<allenap> lifeless: I assume you're talking about txlongpoll? I recently added ez_setup to Storm too.
<lifeless> yes
<allenap> lifeless: For txlongpoll I want it to be standalone, and not depend on the LP download cache. It's meant for Landscape to use too, though I have no idea if they have plans to do so.
<lifeless> allenap: mmm sure
<lifeless> allenap: these things are not actually related.
<lifeless> allenap: e.g. LP deployment target can assume someone has done bzr checkout ... deployment cache
<lifeless> allenap: without compromising use of bootstrap + any other deployment cache.
<lifeless> allenap: separately, how would you feel about us dropping zope.testrunner from it (or at least making subunit.run a supported target, so I can use testrepository :))
<allenap> lifeless: So I should put ez_setup.py into download-cache/dist then refer to it there in Makefile.
<lifeless> allenap: I think it would avoid some duplication
<allenap> lifeless: Very happy about that.
<allenap> lifeless: I think moving ez_setup.py will make it harder for non-LP folk to use txlongpoll, but perhaps bootstrap.pt DTRT.
<allenap> lifeless: Also, Launchpad itself has ez_setup.py in it.
<allenap> But I guess that's a candidate for moving to the cache.
<lifeless> allenap: python ./bootstrap.py will download from the internet; only we care about that :)
<lifeless> allenap: (and other folk that can reasonably use the lp download cache, like landscape)
<lifeless> allenap: (or setup their own)
<lifeless> allenap: its only a thought
<allenap> lifeless: I don't see that it buys much.
<lifeless> allenap: I fear and loath ez_setup; having it in one place would reduce the amount of evil in the world
<lifeless> allenap: also I don't want to have to add it to oops, oops-amqp, oops-wsgi, oops-twisted, oops-datedir2amqp, oops-tools, ...
<lifeless> allenap: (I'm being selfish :P)
<lifeless> bigjools: rt 48999 if you didn't get a CC from rt
<bigjools> got it
<lifeless> allenap: the README claims manual egg location is needed, but it Just Worked for me; are the docs old, or does the LP downloadcache make things easier ?
<allenap> lifeless: Hehe, okay.
 * allenap reads the README
<jtv> We're getting silly warnings from the packaging_translations job that it's skipping translations for Lenny.  Anyone mind if I downgrade that to "info"?
<jtv> bigjools: (I know I said there was no time for me to get into a new item today, but there was only 1 bug to triage :)
<allenap> lifeless: Ah, yes, the README is slightly wrong, and it should Just Work with the LP download-cache.
<allenap> It should read tarball instead of egg.
<lifeless> allenap: in the options in the twisted plugin
<lifeless> allenap: how do you not do a short option - pass None instead of "o" ?
<allenap> lifeless: You mean you want a null OOPS prefix? Sorry if I'm being stupider than usual today, I had a rough night :)
<lifeless> I'm adding amqp oops support to txlongpoll
<lifeless> which is a bit fugly because its not a twisted publisher, but nevermind that :)
<lifeless> I am running out of single letters that are not used by twistd
<lifeless> e.g. -e is encrypted
<allenap> lifeless: Yes, None seems to do the trick.
<lifeless> coolio
<lifeless> uhm
<lifeless> Will need your advice on whether to test this or not in ~ 5 minutes
<lifeless> also hope you have a better day than night
<allenap> Thanks :)
<lifeless> allenap: https://code.launchpad.net/~lifeless/txlongpoll/oops-amqp/
<lifeless> allenap: I have no idea whether this is sanely testable or not.
<allenap> lifeless: Okay, looking.
<lifeless> thanks!
<lifeless> allenap: I'm going to halt()
<lifeless> allenap: that branch should work as-is, I'm sure you can play with it more to see ;)
<allenap> lifeless: Okay. There are a couple of syntax errors in there. I'm inclined to say that all it needs is something to verify that publishers are set up correctly after calling setUpOopsHandler.
<lifeless> ah there are aren't there
<lifeless> interesting that the plugin isn't tested at all then :)
<lifeless> allenap: perhaps we should move all this code out of the plugin, make it testable, and just import it from the plugin to activate it
<allenap> lifeless: Yeah, agreed.
<lifeless> allenap: would you have the bandwidth to run with this ? I wanted to solve the bits that had changed while you guys were building stuff
<allenap> lifeless: Yeah, I guess so. I'll need to run it by bigjools.
<lifeless> if you can, great. if you can't, thats ok too.
<bigjools> you rang
<lifeless> bigjools: I put amqp support for oopses in a branch of txlongpoll
<bigjools> cool
<lifeless> bigjools: this, by virtue of syntax errors uncovered txlongpoll's twistd plugin being untested - entirely untested.
<lifeless> bigjools: (because the test suite still passes)
<bigjools> awesomery
<lifeless> bigjools: I was asking allenap if he had the bandwidth to fix that (by moving its contents into the main module where its testable), incorporating my otherwise trivial branch as he goes.
<lifeless> bigjools: he said possibly-but-ask-you
<bigjools> I am ok with it as long as he is
<lifeless> more-or-less
<allenap> Yes, I'm good with it. Deal.
<lifeless> sweet, thanks!
<jtv> Who's up for a really easy review?  https://code.launchpad.net/~jtv/launchpad/bug-887063/+merge/81439
<jelmer> jtv: it seems you're just changing from warning to warn?
<jelmer> jtv: The proposed fix says you're changing to info
<jtv> Oh God I got that wrong didn't I?  Thinko.
<jtv> This is why we need reviews even on âtrivialâ changes.  :)
<jelmer> :)
<jtv> Fix is being pushed.
<jtv> Done
<jtv> jelmer: the diff has updated.  Note there's also a change to the imports; that's from the automatic imports-formatting script.
<jtv> Unsurprisingly, tests, still pass.
<jtv> Except for, very surprisingly, an unrelated test that breaks even on devel.  What is it _now_?
<jtv> And I can't really investigate with my system needing unrelenting restarts.
<jelmer> jtv: which test?
<jtv> test_splits_and_merges
<jelmer> jtv: r=me, if that failure isn't spurious I'm curious how your change breaks it..
<jtv> It's not my change.  I get the failure even in devel.  :(
<jtv> Argh!  No wait.  I applied the same change in devel.  Something's wrong with me today, clearly.
 * jtv reverts & reruns
<jtv> Reverting my change doesn't fix the breakage in devel.  WTF.
<jtv> Probably a test isolation failure then, I'm guessing.
<jtv> jelmer: are you getting a failure there?  ./bin/test -vvc lp.translations.tests.test_translationpackagingjob -t test_splits_and_merges
<jtv> thanks jelmer â I'll file a separate bug for that breaking test, assuming my own branch gets through EC2.
<jelmer> jtv:   Ran 1 tests with 0 failures and 0 errors in 3.503 seconds.
<jtv> Oneiric?
<jelmer> lucid
<jtv> That may be it then.
<jelmer> (I'm on precise but running a container with lucid)
<jtv> Ah.
<jtv> Call for Help: Could someone try this on oneiric?  ./bin/test -c lp.translations.tests.test_translationpackagingjob -t test_splits_and_merges
<StevenK> jtv: Ran 1 tests with 0 failures and 0 errors in 1.609 seconds.
<jtv> thanks StevenK
<jtv> Oh, StevenK: what revision is that on?
<StevenK> jtv: r14262
<jtv> Hmmâ¦ on 14261.
<jtv> Who knowsâlet's try.
<jtv> Ah no wait, I've got a bus to get onto.  :(
<jtv> nn folks!
<benji> https://dev.launchpad.net/ | On call reviewer: benji | Critical bugtasks: 266
<benji> heh
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugtasks: 266
<deryck> Morning, all.
<jelmer> Arvo, Deryck
<jml> jelmer: tee hee
<jelmer> jml: hmm?
<jml> jelmer: that's a very Australian way of wishing someone a good afternoon.
<jelmer> ah, hmm
 * jelmer wonders where he picked that up
<allenap> benji: Fancy a Storm review? It already has a +1, but needs another. https://code.launchpad.net/~allenap/storm/django-disconnection-errors/+merge/80717
<benji> allenap: my pleasure
<allenap> Thanks!
<benji> allenap: https://code.launchpad.net/~allenap/storm/django-disconnection-errors/+merge/80717 looks good
<allenap> benji: Thanks.
<bigjools> jelmer: still around?
<jelmer> bigjools: jup
<bigjools> jelmer: hurray! did you manage to QA r14243?
<jelmer> bigjools: no :-( the code importer on qastaging is down and I couldn't find a losa earlier
<jelmer> bigjools: I see mbarnett is around now, so I'll ask him
<bigjools> jelmer: mbarnett is around
<bigjools> jelmer: great, thanks!
<abentley> benji: could you please review https://code.launchpad.net/~abentley/launchpad/yui-navigator/+merge/81470 ?  It's overlength, but that's just from indentation changes.
 * mbarnett hides 
<timrc> bigjools, are there any plans of releasing a 2.0 api for this LTS?
<bigjools> timrc: I have not heard of any
<bigjools> but we should look at it
<timrc> bigjools, I don't think we have any pressing needing for it, devel suits us just fine, so it's more out of own personal curiosity ;)
<timrc> er need... damn you -ing
<benji> abentley: I was away for lunch when you asked, but I just finished your branch.
<abentley> benji: thanks.
<benji> my pleasure
<abentley> benji: Updated per your suggestions.
<benji> abentley: cool
<lifeless> morning
<timrc> howdy
<m4n1sh> is there a command line tool where I can access bug info from LP
<m4n1sh> and also initiate merge requests?
<lifeless> there is a thing called hydrazine
<lifeless> and bughugger
<lifeless> and lp-tools
<m4n1sh> thanks
<m4n1sh> lifeless: not in archives?
<m4n1sh> its in a PPA
<lifeless> sinzui: do you have time for a catch up today ?
<sinzui> I do.
<sinzui> maybe in 30 minutes when I complete this review
<lifeless> coolio
<jelmer> lifeless: lp-tools is in the archive
<jelmer> s/lifeless/m4n1sh/
<m4n1sh> ahh. it is called lptools
<jcsackett> hey, lifeless! i updated the MP you marked needing fixing. care for a second look? (https://code.launchpad.net/~jcsackett/launchpad/open-delegate-subscription-problems/+merge/81090)
<lifeless> sure thing
<lifeless> jcsackett: line 169 in the diff is very suspect
<lifeless> jcsackett: thats the bit where you delete a Makefile and replace it with a symlink to somewhere on your hard disk.
<jcsackett> lifeless: yeah, i just saw that.
<jcsackett> lifeless: fixing that now, sorry for the time waste. :-p
<lifeless> jcsackett: np, I've reviewed the rest :)
<lifeless> jcsackett: I have one suggestion to avoid duplication further
<jcsackett> lifeless: i'm all ears.
<lifeless> jcsackett: which should be in your mailbox by now :)
<lifeless> delegation!
<jcsackett> lifeless: oh, i do like that. we are likely to be doing that check a fair bit.
<jcsackett> consider it done, and thanks. :-)
<lifeless> de nada
<flacoste> lifeless: hi
<flacoste> lifeless: available for our call?
<flacoste> early this week
<lifeless> I sure am
<lifeless> sinzui: will in a little bit later be ok ?
<lifeless> sinzui: if I talk to the boss first :)
<lifeless> flacoste: skype ?
<sinzui> lifeless, ping me when you are ready
<flacoste> lifeless: yes, i'm trying a new headset, let's see how it goes
<lifeless> sinzui: great, will do
<bac> hi abentley, i'm getting an error from bzr lp-submit.  it's the first time i've tried it on a new oneiric machine.  are you familiar with any problems like: http://pastebin.ubuntu.com/731374/
<abentley> bac: Sorry, I'm not familiar with that.  It looks like a password/launchpadlib issue.
<maxb> dbus error talking to your gnome-keyring-daemon
<maxb> Though quite what's broken such that g-k-d is not responding, I don't know
<allenap> lifeless: lp:~allenap/txlongpoll/oops-amqp is what I did with your branch of the same name.
<allenap> lifeless: It depends on lp:~allenap/txlongpoll/bootstrap-without-net, and some revisions have been mixed up a bit between those branches. I'll remedy that now.
<allenap> lifeless: Actually, no, they're not mixed up, I was being paranoid.
<lifeless> sinzui: ping
<sinzui> hi lifeless
<lifeless> allenap: sounds great, do you need anything from me ?
<lifeless> sinzui: skype ?
 * sinzui starts skype
<thumper> flacoste: hi
<flacoste> hi thumper
<thumper> flacoste: busy?
<allenap> lifeless: Nope, I'll get it reviewed tomorrow unless you want to +1 it at some point :)
<lifeless> allenap: cool; I may, will see how the day pans out
<flacoste> thumper: more or less, general post-UDS catchup
<flacoste> thumper: what can i do for you?
<thumper> flacoste: can I call you for some questions?
<flacoste> sure, skype
<allenap> lifeless: Thanks, have a good day :) Night all.
<wallyworld___> sinzui: jcsackett: you guys around for the standup?
<jcsackett> oh right, time changed.
 * jcsackett gets on mumble
<sinzui> wallyworld___, sorry, my call with lifeless got very interesting
<wallyworld___> np :-)
<wgrant> sinzui: That's worrying.
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266
<timrc> 266 critical bug tasks, I guess it may be time for a new "super critical" category :)?
<thumper> HAHA
<timrc> Thank you, thank you, I'll be here all night
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugtasks: 275
<wgrant> timrc: It's even better :)
<timrc> ouch
<wgrant> jcsackett: http://webnumbr.com/launchpad-critical-bugs
<wgrant> https://lpstats.canonical.com/graphs/LPProjectCriticalFiled/20101108/20111108/
<timrc> July was a good month for you
<wallyworld___> sinzui: we are looking to finish the standup, we can talk again tomorrow?
<sinzui> wallyworld___, thanks.
<wallyworld___> np
<allenap> wgrant: I have a couple of txlongpoll branches that need reviewing. Do you fancy taking a look? The only catch is that I'm going to bed in a minute so I won't be around to discuss things.
<allenap> They are:
<allenap> https://code.launchpad.net/~allenap/txlongpoll/bootstrap-without-net/+merge/81323
<allenap> https://code.launchpad.net/~allenap/txlongpoll/oops-amqp/+merge/81504
<wgrant> allenap: Sure, I'll do them today.
<allenap> wgrant: Thank you :)
<lifeless> timrc: ubuntu has more ;)
<timrc> lifeless, ever have the urge to mark them all "Won't Fix" ;)?
<lifeless> timrc: no, but the rest, yes
<lifeless> allenap: http://pypi.python.org/pypi/python-subunit
#launchpad-dev 2011-11-08
<huwshimi> lifeless: Morning
<huwshimi> lifeless: I started to take a look at the caching
<huwshimi> lifeless: Is this what you meant? https://code.launchpad.net/~huwshimi/launchpad/avatars-everywhere-712894/+merge/81430
<huwshimi> (I've pushed changes which will be in the diff)
<huwshimi> lifeless: Is there more that I need to do?
<huwshimi> lifeless: You'll have to bear with me, as I really don't know what I'm doing here :)
<lifeless> heh, sure
<lifeless> huwshimi: thats pretty close
<lifeless> I'll give a longer answer in the review
<huwshimi> lifeless: Thanks :)
<lifeless> done, -> optometrist, bbiab
<wgrant> %#$@
<wgrant> die, postgres, die
<lifeless> ?
<StevenK> What about it?
<wgrant> The planner is making stupid decisions.
 * wgrant pastes.
<wgrant> lifeless: http://paste.ubuntu.com/731544/
<wgrant> First plan, for a single policy.
<wgrant> Very sensible.
<wgrant> Straight double nested loop down three indices.
<wgrant> Second one, also a single policy but not specifying the ID directly (this is how I initially tried it): terrible terrible plan doing a manual person sort afterwards. It should have known it would get at most one row, but perhaps the policy ID I hardcoded is in the stats so it gets special treatment.
<wgrant> Third one, hardcoding the three IDs.
<wgrant> Even worse plan.
<wgrant> (these are Ubuntu's three access policies, on real data)
<lifeless> wgrant: a request for these pastes in future
<lifeless> wgrant: please wrap the queries.
<wgrant> lifeless: I normally do, but was too grumpy this time.
<wgrant> I tried a CTE, but it doesn't work. Even if it returns just the one row.
<wgrant> I guess because it plans beforehand.
<wgrant> It's probably seeing that the 97822 policy is special and planning accordingly, I guess.
<lifeless> so id=97822 == policy=1 ?
<wgrant> But if it knows it's going to dominate, then why does it ignore it when I specify 3 IDs...
<wgrant> Ah, yes, sorry.
<wgrant> 97822 == distro 1, policy 1
<wgrant> Which is Ubuntu private non-security.
<lifeless> is policy an enum ?
<wgrant> Yes.
<wgrant> It may be better to explode it here.
<wgrant> But only because the planner is being obtuse.
<wgrant> It's perfectly feasible to do the same simple nested loop down three indices, just with an ANY on the apg filter.
<lifeless> look at the estimates
<lifeless> its estimating 2, then 45
<lifeless> 2nd plan
<wgrant> Yeah.
<lifeless> first one its estimating 57K rows - and considering 1.2M
<lifeless> but the actual execution is fast
<lifeless> this is on df ?
<wgrant> Yes.
<wgrant> I have the queries to set up the data in a minute or two, if you want.
<lifeless> the second plan has a lower cost estimate
<lifeless> and the third lowest of all
<wgrant> Yeah, because the row count estimates are off.
<lifeless> so the interesting question is why
<lifeless> have you analyzed ?
<wgrant> By a factor of 3.
<wgrant> Er, 3 orders of magnitude.
<wgrant> Yes.
<wgrant> VACUUM ANALYZE, ANALYZE, etc.
<wgrant> Several times.
<wgrant> No change after the initial one.
<lifeless> scale 1000 ?
<lifeless> or the 100 df is running ?
<wgrant> 100
<lifeless> care to try
<wgrant> I guess.
<wgrant> Worth a try.
<lifeless> its a thing to rule out
<lifeless> so the first one is potentially a bad plan too
<lifeless> but we expect the data sizes to stay modest
<lifeless> what does it do if you say offset 500 to it ?
<lifeless> [curiosity]
<wgrant> I'm not sure that's important, but sure.
<wgrant> 680ms, so it's pretty linear.
<wgrant> As expected from the plan.
<wgrant> Doing a sort-key-based offset is very fast, also as expected.
<wgrant> That plan is perfect for the main disclosure management view.
<wgrant> (unless the set of observers is small)
<wgrant> person_sorting_idx is only 57MB, and presumably very hot, so it may even be fast enough when there aren't many.
<wgrant> I suspect the first batch is actually slower than the rest.
<wgrant> Because it's going to have to walk through all the usernames that start with digits.
<wgrant> Roughly none of which are going to be relevant, because sensible people don't generally have usernames that start with digits.
<StevenK> UPDATE person set name = '8wgrant' where name = 'wgrant'; commit;
<wgrant> I actually mean displayname.
<wgrant> But yes.
<lifeless> rofl
<lifeless> https://github.com/juuso/BozoCrack
<wgrant> Heh
<wgrant> It is a useful strategy sometimes.
<wgrant> lifeless: So, any query suggestions?
<lifeless> sorry was sidetracked
<lifeless> uhm
<lifeless> (distribution, policy) is unique
<lifeless> but id is what shows up on the other tables
<wgrant> Yes.
<lifeless> where are your data generating scripts?
<lifeless> (and do they work on temp tables ;))
<wgrant> They are temp tables, yes.
<lifeless> wgrant: try this http://paste.ubuntu.com/731583/
<wgrant> Just dropped the fks and it all works.
<lifeless> bah
<lifeless> not limit 0
<lifeless> offset 0
<wgrant> http://paste.ubuntu.com/731585/ <- forgive the awful code to create artifact-specific permissions. There's probably a better way to do it, but everything else I tried was unbelievably slow on DF.
<wgrant> lifeless: Erm, explicit offset 0?
<wgrant> Isn't that redundant?
<wgrant> (also, that's still 3.5s)
<lifeless> optimisation barrier
<wgrant> I think that's the problem.
<wgrant> I suspect it chooses the index-scan-only plan because the ID it's looking for dominates the stats.
<wgrant> Except not.
<lifeless> what do you mean by 'dropped the fks and it all works'
<wgrant> The original table definitions had foreign keys to person/bug/product/etc.
<wgrant> But temp tables can't have fks to permanent tables.
<wgrant> So I dropped the constraints.
<lifeless> oh right, of course ;)
<lifeless> I presumed that (bugsummary experience most recently ;P)
<lifeless> hah
<lifeless> Time: 27666.227 ms
<wgrant> Oh?
<lifeless> first run of second plan
<wgrant> Ah
<lifeless>  SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.id=97822 ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<lifeless>  id
<lifeless> ----
<lifeless> (0 rows)
<wgrant> Your IDs will be different :)
<lifeless> ah, doh
<wgrant> SELECT id, policy FROM accesspolicyuse WHERE distribution=1;
<lifeless> we'll need to iterate on abels code
<lifeless> to paginate on a function sort
<wgrant> Yes.
<wgrant> I had already looked at that a little.
<lifeless> did you look at nuking the functional index?
<wgrant> Just using eg. person.name for these queries? Yes, didn't help.
<huwshimi> wgrant: Is this what you meant for fixing that XSS problem? http://paste.ubuntu.com/731598/
<wgrant> huwshimi: !!! someone did it right the first time. Yes, that's what I was meaning.
<huwshimi> wgrant: Hah! It might look right, but it provides a mangled output
<wgrant> Oh?
<wgrant> Ah
<wgrant> You may need to return .escapedtext, depending on how you're using it.
<huwshimi> wgrant: It returns this: http://paste.ubuntu.com/731600/
<wgrant> If it's in a TALES formatter, it probably will need .escapedtext.
<huwshimi> wgrant: Ah
<wgrant> Yeah.
<wgrant> Sorry, forgot to mention that.
<huwshimi> wgrant: That's fine :)
<huwshimi> wgrant: Perfect, working now :)
<wgrant> Great!
<lifeless> wgrant: .name wouldn';t help here, but for abels work...
<lifeless>  registrant                         |          1 |         4 |          0
<lifeless> we might want to drop that
<wgrant> ?
<lifeless> totally unused colum on person
<wgrant> s/drop/fix/
<lifeless> also language
<lifeless> 'meh'
<lifeless> wgrant: can you paste me
<lifeless> SELECT attname, most_common_vals, most_common_freqs FROM pg_stats WHERE tablename='accesspolicyuse';
<lifeless> actually, nvm
<wgrant> http://paste.ubuntu.com/731614/ anyway.
<lifeless> thats very interesting
<lifeless> I wonder why distribution is blank
<wgrant> Distributions are extremely rare.
<wgrant> 40 vs 33000
<lifeless>  SELECT attname, most_common_vals, most_common_freqs FROM pg_stats WHERE tablename='accesspolicyuse' and attname='distribution';
<lifeless>    attname    |                                         most_common_vals                                          |                                                                                                                                                                                                           most_common_freqs
<lifeless> --------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
<lifeless>  distribution | {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35} | {3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3
<wgrant> Even a stats target of 1000 might not catch a distribution.
<lifeless> ...
<lifeless> e.g. its not blank for me
<wgrant> You probably still have a huge target.
 * wgrant tries.
<lifeless> 105 apu entries with no product
<lifeless> 98K where it is
<lifeless> 0.1%
<lifeless> 1-0.001^1000 chance of not seeing one
<lifeless> IIRC
<wgrant> WTF
<wgrant> Now product is empty.
<wgrant> distribution has lots.
<lifeless> 1000 is the stats it keeps, not the rows it consults
<lifeless> to determine most common it has to consult more rows than it keeps
<wgrant> Oh.
<wgrant> Still doesn't explain it.
<lifeless> we're looking at the rows it keeps, and if that was random over the population we could estimate by frequency
<lifeless> but this is explicitly the most common values per attribute
<lifeless> so yeah, I dunno
<wgrant> And on APU it's 3 tuples for each non-NULL value.
<lifeless> tell me something, why does your script generate 98K policy uses
<lifeless> is that because all products get security ?
<wgrant> It currently just creates public, private, security for every pillar.
<wgrant> Until we have a sensible way to manage comment editing, we probably have to allow private bugs for all projects.
<wgrant>         4 / 1346  RootObject:+login
<lifeless>   EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant where policy IN (SELECT id from accesspolicyuse where distribution=1 and policy=1)) ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<lifeless> 780ms
<lifeless> wgrant: I think your first plan is a fluke the more I look at it
<wgrant> Oh?
<lifeless> wgrant: the indices are not in the same order
<wgrant> It may be a fluke, but the others are crap :)
<lifeless> one is person sort
<lifeless> the other is person
<wgrant> Yes.
<lifeless> it can't dual iterate
<wgrant> No.
<lifeless> it can?
<wgrant> But it happens that (in this case) the coverage is good enough that it's not a problem.
<wgrant> It can't, AFAIK.
<lifeless> double negs :)
<lifeless> so
<lifeless> I think a better question isn't 'why are the other plans slow'
<lifeless> its 'what do we want to actually query'
<lifeless> thus the query above
<lifeless> or
<lifeless> a less extreme
<lifeless> EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<wgrant> Which is 1180ms
<lifeless> interestingly
<lifeless> you realise that you are querying outside of both your partial indices
<wgrant> Yes.
<wgrant> I added a new non-partial one.
<lifeless> is NULL -> 4ms
<wgrant> Which would possibly be better done by making another one non-partial, but perhaps not.
<lifeless> is not NULL -> 2000ms
<wgrant> On what?
<lifeless>  EXPLAIN ANALYZE SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.policy = 1 AND artifact is NOT NULL ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<wgrant> artifact?
<wgrant> Well, yes.
<wgrant> There are far fewer where artifact is NOT NULL
<wgrant> That's not so much the indices as the volume of data.
<wgrant> Your thing now has to sort a total of 1 person.
<wgrant> Which is not a significant challenge.
<lifeless> different plans
<lifeless> not unsurprising
<lifeless> ok so to rephrase the problem
<lifeless> 'there are 57K people with explicit access within Ubuntu and we want to sort them by fields only on Person'
<wgrant> Yes.
<lifeless> by definition we need to either read *all people* in sort order filtering against the set of grants - which will for random names mean we read ~ all people
<lifeless> or we need to read all grants (57K) and then cherry pick people
<wgrant> Yes.
<lifeless> which is much more selective and what its doing
<lifeless> so why is it slow?
<wgrant> I don't know. I wouldn't expect materialising 57k rows from such a narrow table to be slow at all.
<lifeless>                ->  Nested Loop  (cost=3.39..66.16 rows=13 width=25) (actual time=14.369..1910.529 rows=57946 loops=1)
<wgrant> Which plan is this?
<lifeless> EXPLAIN ANALYZE SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.policy = 1 AND artifact is NOT NULL ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<lifeless>                      
<lifeless> but any of them start to look like this
<wgrant> eparse
<lifeless>  explain analyze SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.id=98234 ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;
<lifeless>                       
<lifeless> is weird
<wgrant> What about it is weird? The plan?
<lifeless> yes
<wgrant> Hmm.
<wgrant> I guess the non-indexed sort of 57k people is not going to be terribly fast, because person_sort_key is PL/Python.
<wgrant> But I'm pretty sure it happened with a straight string sort too.
<lifeless> I'll just reply to your mail then test ;)
<lifeless> I think materialising the sort column may well help
<wgrant> Did anything useful come from the discussion this morning?
<lifeless> yes, we cleared things up
<wgrant> FSVO
<wgrant> And for now.
<StevenK> Hah
<StevenK> Does that mean I can continue with my branch to hide private teams in subteam of?
<wgrant> ANd probably by trebling the scope of the project.
<lifeless> StevenK: yes, the confusion was that that page shows super and sub
<lifeless> StevenK: see bug 405277 which really covers the visibility rules
<_mup_> Bug #405277: Private teams are not able to join other teams (public or private) <disclosure> <lp-registry> <privacy> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/405277 >
<StevenK> wgrant: If it's a death march until March, what difference does December next year make? :-(
<lifeless> StevenK: you probably also want to hide proposed memberships from private teams, unless the private team has signed off already.
<StevenK> I really want to finish this branch that kills strict_component from {check,verify}Upload since it's close to done, but wgrant is ignoring me. :-P
<wgrant> Bah, that could have been said here rather than in PM.
<StevenK> You were busy here :-P
<wgrant> StevenK: I don't understand what you're saying.
<StevenK> So, there is two things
<StevenK> One test failure is test_checkUpload_without_strict_component. I'm concerned that is a buggy test since strict_component is dead and is possibily relying on behaviour that is removed.
<StevenK> Hm. It's expecting checkUpload() to pass, but it now raises NoRightsForComponent.
<StevenK> s/raises/returns/
<StevenK> Since returning exceptions is sexy or something
<wgrant> It's no longer a useful or possible test.
<wgrant> So it should be removed.
<StevenK> The other test failure is from the upload processor, which is not rejecting a package that should be.
<StevenK> I think it is because checkUpload() is returning an exception and we do not reject because the package is NEW.
<wgrant> pastebin the failure
<wgrant> And diff
<StevenK> AssertionError: !=:
<StevenK> reference = "Not permitted to upload to the RELEASE pocket in a series in the 'CURRENT' state."
<StevenK> actual = ''
<wgrant> Ah, indeed.
<wgrant> So, we need to catch the no privileges exception.
<wgrant> Well, "catch"
<wgrant> Check for it.
<StevenK> Yes.
<wgrant> If checkUpload has failed due to a lack of permissions, then do the NEW check.
<StevenK> This strikes me as ugly
<wgrant> Welcome to Soyuz.
<StevenK> wgrant: Any exception from checkUpload is lack of perms?
<wgrant> No.
<wgrant> eg. the pocket error that you quoted
<StevenK> NoRightsForArchive() == lack of permissions
<StevenK> NoRightsForComponent() == lack of permissions
<wgrant> NoRightsForArchive is a lie and should probably die, or be moved into NascentUpload.
<wgrant> NoRightsForComponent is stupid and too specific.
<StevenK> CannotUploadToPPA is arguably permissions
<StevenK> However, I agree ArchiveDisabled is not.
<wgrant> It's not a matter for agreement.
<StevenK> So, how should this code be structured?
<StevenK> And do you still need to see the diff?
<wgrant> I would probably rework verifyUpload to raise a single exception when there are insufficient upload rights.
<wgrant> Except perhaps preserving NoRightsForArchive, but implementing it in a non-buggy fashion.
<wgrant> NoRightsForComponent has no right to exist.
<StevenK> Isn't it used for, say, partner?
<wgrant> Partner doesn't need to behave well.
<wgrant> As long as it vaguely works for 6 months.
<StevenK> Hm, partner is a seperate Archive anyway
<wgrant> In some ways.
<StevenK> It isn't Archive.id 1, is what I mean
<StevenK> Which means it has its own ArchivePermission entries
<wgrant> Yes.
<wgrant> But that's almost all it means.
<StevenK> NoRightsForComponent is only checked due to checkArchivePermission(person, component), is that check pointless too?
<wgrant> That would be cheating.
<StevenK> I don't really want to rewrite verifyUpload() in this branch
<wgrant> It's only like 10 lines.
<StevenK> It's currently 30, so that sounds like an improvment
<StevenK> I'm tempted to say 'prove it' :-P
<lifeless> wgrant: ok, the call
<lifeless> wgrant: so the two bits we were getting stuck/confused on were:
<lifeless>  Person:+index shows sub and super; this was just confusion. super should not be shown. sub should always be shown.
<lifeless> StevenK: ^
<lifeless> and bug subscriptions should always be shown too
<lifeless> no spying!
<lifeless> wallyworld_: ^
<wgrant> That was my understanding.
<wallyworld_> lifeless: so we don't *ever* want to hide private subscribers?
<lifeless> super should not be shown because the teams a person are in should not be disclosed just because you know the person -> visibility is not granted to non-participants of a Person against their memberships.
<lifeless> subteams should always be shown (if you can see the team they are a member of) because the subteam could otherwise give you an invisible project owner. e.g. public project, public owning team, no members visible. But really there is a private team lurking inside the public team.
<lifeless> we don't want that-> we show the existence of the private team.
<lifeless> wallyworld_: the rule is, if you can see the bug, you can see the name, displayname, url, of its subscribers (including structural subs)
<wallyworld_> ok. but if you click on the link, it will 404?
<wgrant> lifeless: aaaaaaa
<lifeless> wallyworld_: there are a few drivers for this. One is the guiding principle that interacting with a public object discloses private objects.
<wgrant> lifeless: We're changing privileges based on admin status?
<lifeless> wallyworld_: it will give a 200 and a little details.
<lifeless> wallyworld_: which will require separate work.
<wallyworld_> ok,so the normal +index view with a bunch of stuff not rendered
<lifeless> wallyworld_: the second is that if anyone replies to the bug via mail, and they are subscribed via a private team, they are very likely to disclose the private team anyhow due to the footer we include, which 'Reply' in many modern mail clients includes.
<lifeless> wallyworld_: yes
<wgrant> lifeless: How do you propose that Person:+index knows if you have partial visibility to it?
<lifeless> wgrant: theres a something like 10 clause EXISTS union we need to write.
<lifeless> wgrant: AFAICT it should be fast.
<wgrant> I was about to say that that was not an answer.
<wallyworld_> only 10 clauses? surely we could try for 20 :-P
<wgrant> We would have to join bugsubscription->bug->accesspolicyartifact->accesspolicyuse->accesspolicygrant->teamparticipation, and that's just for one part of it.
<lifeless> wgrant: its not trivial unfeasible.
<wgrant> It is, however, trivial silly.
<lifeless> I have been thinking about this for a year or so
<lifeless> and I disagree with you :)
<wgrant> So.
<lifeless> 9
<lifeless> bah
<wgrant> Say that one day in future we decide that being a monolithic webapp is actually a stupid idea, and SOA is cool.
<lifeless> how about that huh
<lifeless> \o/
<wgrant> How does that work now?
<wgrant> We call out to 50 services?
<lifeless> in parallel, yes
<wgrant> ...
<StevenK> Like fun that scales
<lifeless> (and cancel the calls when one answers)
<wgrant> This also puts the LP services in a strongly privileged position.
<lifeless> StevenK: it actually scales very very well
<lifeless> StevenK: because each service can cache answers to these questions when appropriate
<lifeless> StevenK: 'can user X see object Y' is a -stock- query any service that owns data has to be able to answer efficiently
<StevenK> lifeless: But 50 parallel calls? Times 50 users?
<lifeless> StevenK: where do you get times 50 users ?
 * wallyworld_ gets out the popcorn
<StevenK> Out of the air
<lifeless> StevenK: so, this is the sequence:
<lifeless> traversal
<lifeless> oh this is private
<lifeless> check rules
<lifeless> -> denied
<lifeless> -> full
<lifeless> -> restricted
<lifeless> where denied is a failure to meet full or restricted
<lifeless> full depends on the object, say for Person, it is granted by: is_member or is_owner
<lifeless> restricted is granted by 'common ground', so we need to find *a* object where:
<lifeless> user.expand_through_team_participation() can see && object subscribed-to, owns, etc.
<lifeless> so one call to the team membership service, to get all the teams of user
<lifeless> thats fast, we know it is.
<lifeless> and then for each data store that might hold an object the Person we are looking at owns [note we don't need TP expansion of it]
<lifeless> we ask for exists(discloses=Person, viewer=TP_expanded_user)
<lifeless> will it be dead trivial to write? Probably not. Is it beyond our current schema, or an SOA - not at all.
<lifeless> its also very cachable if we need to.
<lifeless> though that might be terrifyingly large.
<lifeless> wallyworld_: StevenK: I've commented on both your bugs.
<wallyworld_> thanks
<wallyworld_> nice to see it clarified :-)
<lifeless> hopefully it makes sense
<wallyworld_> i'll read soon. don't want to context switch right at the minute
<lifeless> if it doesn't, I'm certain sinzui and I are in sync (we were very close to as it turned out), and I can clarify further as needed.
<lifeless> thats fine
<lifeless> tis EOD here, so I'm going to be in and out for a hiwle
<wallyworld_> i'll grok before tomorrow's standup
<StevenK> wgrant: I still can't see how verifyUpload() can be only ten lines :-(
<wgrant> 20, maybe.
<StevenK> wgrant: If it makes it easier and simplier, then I'm for it, I just can't envision where to start ripping out bits
<lifeless> wgrant: http://www.postgresonline.com/journal/archives/209-Manually-setting-table-column-statistics.html
<lifeless> wgrant: may help
<lifeless> wgrant: e.g. upgrade to pg 9
<StevenK> Oh sure, because the upgrade to 8.4 went so well :-P
<wgrant> We should really do that eventually.
<lifeless> stub and I began discussing logistics for it yesterday, with fdt nearly done
<lifeless> wgrant:  EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person.displayname LIMIT 75;
<lifeless> Time: 351.454 ms
<lifeless> wgrant: compare the plans
<lifeless>                ->  Index Scan using person_pkey on person  (cost=0.00..6.39 rows=1 width=14) (actual time=0.012..0.013 rows=1 loops=17313)
<lifeless> wgrant: its a bit variable, but as low as 200ms
<lifeless> wgrant: I suspect staging is getting a hammering just now ;)
<lifeless> wgrant: smaller sort too
<stub> It will be faster if you ORDER BY lower(displayname)
<stub> Or at least, it might be able to use an index then
<wgrant> lifeless: Hm, does that win for any reason beyond the DISTINCT being done first, and not using person_sort_key?
<wgrant> The DISTINCT only cuts it by a factor of 4.
<wgrant> The rest is pretty similar, apart from the sort key.
<stub> ORDER BY person_sort_key(displayname, name) might be good too
<lifeless> stub: thats twice as slow
<lifeless> stub: functional index
<lifeless> stub: wgrant can fill you in on the gory details :)
<stub> There used to be bugs with functional indexes, where the function would be called unnecessarily. Not sure if that has been fixed yet.
<stub> But if the index is usable, lower() should be fast since we have an index on lower(displayname) and the functional index bug does not occur for magic internal functions like lower
<lifeless> stub: its not using the index
<lifeless> http://paste.ubuntu.com/731585/
<lifeless> will get you the datatructure in temp tables, to play with
<lifeless> http://paste.ubuntu.com/731544/ is the problem wgrant put to me
<stub> With lower() it is not using the index?
<wgrant> It can't really use the index.
<lifeless> it could
<wgrant> Except for that one special case.
<wgrant> Well.
<wgrant> It could.
<lifeless> load it all up and walk
<lifeless> but its going to be inefficient
<wgrant> But it would have to walk the whole thing.
<wgrant> Right.
<wgrant> Like the first plan I provided.
<wgrant> Which lifeless may have just linked.
<lifeless> because we're likely to get a wide spread
<wgrant> Yes
<lifeless> only special cases will walk successfully
<wgrant> For Ubuntu it works very well.
<wgrant> For everyone else, probably not so much.
<lifeless> we're pulling back 17K / 1437K peoples
<lifeless> <2%
<stub> But anyway, if there is slowness with a person_sort_key query, it might be due to PG bugs and try a lower instead as there is an index on that too.
<lifeless> index walks will hit data pages, so an index walk becomes roughly equivalent to a table scan at this size of random-spread data
<lifeless> stub: yeah, got that
<lifeless> stub: I think there is (because the function is called per row, and there are [depending on query structure] 56K or 17K rows,.. and python is slow
<lifeless> I might put a extra column on person and give it a spin, I'm curious
<wgrant> I'm not quite sure why it's done in Python.
<wgrant> Possibly because why not.
<wgrant> In 2005.
<lifeless> because we didn't comprehend the cost of functional indices when they were introduced.
<wgrant> Right.
<stub> Its designed to be used to order results, so slowness in theory only matters on insert
<stub> Cause ordering would hit the index
<lifeless> stub: and in sorting
<wgrant> Ordering can hit the index.
<wgrant> But it's only effective if you're sorting *lots* of people.
<lifeless> ordering by (foo) requires foo to be calculated
<lifeless> wgrant: other way around
<lifeless> its only useful if you're doing an index walk
<lifeless> which requires the index to -very very- closely match the query
<wgrant> Right, which is only efficient if you're wanting lots of those people...
<wgrant> Yes.
<wgrant> Isn't that what I said?
<lifeless> intersecting sets
<lifeless> lots of people
<lifeless> or subsections of the index that are well defined can do it too
<wgrant> Walking the index is only efficient if you want a significant chunk of its values.
<lifeless> or a small subsection
<lifeless> with deep indices
<lifeless> we can go around this all day :)
<mwhudson> heh are you still talking about query plans?  i've reinstalled my machine and restored ~ from backup since you started on that
<wgrant> OK, "walking a portion of an index is only efficient if you want a significant chunk of the values from that portion"
<lifeless> yes, though really we should qualify that to 'in pg' - because other systems can walk on key prefixes ('loose scans')
<StevenK> mwhudson: Why?
<lifeless> qastaging may timeout for you for a little bit
<huwshimi> lifeless: If you happen to get a chance to have another look at that avatar branch it would be much appreciated.
<lifeless> huwshimi: is thursday ok ? I have tomorrow off for baby stuff, and doubt my ability to page it in properly tonight
<mwhudson> StevenK: mixture of really strange acpi bugs and wanting the disk space win7 was taking up
<huwshimi> lifeless: Yeah sure, no problems at all
<lifeless> huwshimi: perhaps you can get the OCR (wgrant?) to review it, and take my comments into consideration.
<huwshimi> lifeless: Sure, thanks :)
<wgrant> I started at 6am, so I was actually about to wander off. But if it's short...
<lifeless> mwhudson: my new intel 320 ssd is coming ... :)
 * wgrant looks.
<lifeless> mwhudson: I got the 300GB model
<huwshimi> wgrant: It's not a problem. Another time is fine
<mwhudson> lifeless: yeah, that's a bit tempting, but the 140 gigs i now have will take me a while to fill :)
<mwhudson> i don't keep photos or much music on this machine anyway
<lifeless> mwhudson: funny win7 stry
<lifeless> mwhudson: my desktop came with win7 - its a 16GB ram machine
<mwhudson> (about 1/3 of my disk space consumption is in ~/VirtualBox\ VMs/ ...)
<lifeless> mwhudson: I kept win7 for games and just-in-cases
<lifeless> mwhudson: gave it, I think 100GB or something.
<lifeless> mwhudson: imagine my surprise when it creates a 16GB swapfile.
<StevenK> Hah
<lifeless> wgrant: stub: 40 seconds to do:
<lifeless> select count(distinct person_sort_key(person.displayname, person.name)) from person;
<wgrant> lifeless: Swapfile or hiberfile?
<lifeless> wgrant: pagefile.sys
<wgrant> Huh
<wgrant> Odd
<lifeless> no, unchanged since NT 3.5
<lifeless> its daft, because more ram == more slowness from page file and less need
<wgrant> I thought it by default used a dynamically scaling one.
<lifeless> yes, thats the default size heuristic
<StevenK> Linux still wants swap with a large amount of RAM
<lifeless> StevenK: lots of sites that care about perf disable it these days for linux.
<StevenK> Sure, the kernel wants it, doesn't mean it gets it. :-)
<lifeless> StevenK: you still get VM backed shared pages etc; but there is precious little use for swap per se
<stub> launchpad_prod_3=# explain select count(distinct(person_sort_key(displayname,name))) from person;
<stub>                               QUERY PLAN
<stub> -----------------------------------------------------------------------
<stub>  Aggregate  (cost=51296.04..51296.30 rows=1 width=21)
<stub>    ->  Seq Scan on person  (cost=0.00..47617.83 rows=1471283 width=21)
<stub> (2 rows)
<stub> Time: 534.701 ms
<mwhudson> lifeless: hah
<lifeless> stub: garh what
<mwhudson> lifeless: that might have happened to me, i guess
<mwhudson> (8 gigs in this machine)
<stub> lifeless: Warm index
<lifeless> mwhudson: http://support.microsoft.com/kb/314482
<StevenK> I thought Windows had an option to twiddle for no swap
<lifeless> The default paging file size is equal to 1.5 times the total RAM. However, this default configuration may not be optimal in all cases
<wgrant> StevenK: It does.
<lifeless> so, yes, not 16. 24GB
<mwhudson> rofl
<stub> lifeless: actually, not hitting the index. just warm and gobs of ram I guess to hold the immutable result of the function :)
<wgrant> StevenK: Hm?
<wgrant> Er.
<wgrant> stub: That was an EXPLAIN.
<stub> ahaha
<wgrant> stub: Not an EXPLAIN ANALYZE...
<lifeless> hah yes
<lifeless> ECOFFEE
<wgrant> So how was it 500ms.
<wgrant> 1000?
<wgrant> But surely not on such a trivial query.
<wgrant> I mean, what planning is there to do...
<mwhudson> does pg 9.1 work for lp dev?
<lifeless> mwhudson: maybe ;)
<wgrant> mwhudson: Unlikely.
<wgrant> But possibly.
<stub> mwhudson: probably
<stub> mwhudson: Might need some tweaking of the various setup scripts
<lifeless> person update still going... probably should have chunked it
 * mwhudson chuckles
<mwhudson> i guess i'll find out sooner or later
<lifeless> at least its not waiting for anything
<wgrant> Well, yes, if you want to tweak stuff then of course it will work :P
<stub> lifeless: So the python method is about 3x slower than the builtin lower()
<lifeless> stub: yes :)
<lifeless> stub: which will be slower than an actual column
<lifeless> I'm making one on qastaging's Person table.
<lifeless> just because
<stub> lifeless: I'm thinking of whitelist for scripts we know have long running transactions, so we need to add them explicitly and open bugs.
<wgrant> stub: Hmm, but we often have to kill them early.
<lifeless> stub: so there are three cases I think
<wgrant> Because eg. karma sticks around for 10 minutes after we kill it.
<lifeless> stub: readonly safe to kill
<lifeless> stub: readonly unsafe (mutates disk unrecoverably)
<lifeless> stub: readwrite unsafe (mutates disk unrecoverably)
<lifeless> first case should be readonly/readwrite safe to kill (atomic, no external changes)
<stub> lifeless: safe to kill, but should we do it without making note that this task needs to be fixed?
<lifeless> stub: I like making a note
<lifeless> stub: I worry that you mean 'have the deploy abort when it finds a new one'
<stub> wgrant: But it won't stick around for 10 minutes if the full-update.py terminates the backend
<stub> lifeless: if we find a new one, we should abort and investigate.
<wgrant> stub: Are you sure?
<lifeless> stub: why do you say that? we know what code we have...
<wgrant> pg_cancel_backend isn't effective. I thought we tried pg_terminate_backend once too...
<stub> lifeless: sticking our head in the sand, crossing our fingers and full steam ahead will *normally* work
<stub> lifeless:  I mean defining EVIL_USERS = [], similar to fragile users, where we list the badly behaved db users along with a bug number. These users we ignore their long running transactions and let preflight.py terminate things.
<lifeless> wow, person is sparse on id
<stub> erm... full-update.py terminate things
<mwhudson> lifeless: all those shipit accounts that got deleted?
<lifeless> mwhudson: p'rhaps
<StevenK> lifeless: Out of interest, did you see my analysis on bug 618399?
<_mup_> Bug #618399: DistroSeries:+nominations timeouts <lp-blueprints> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618399 >
<lifeless> StevenK: yes, sounded plausible
<wgrant> They actually mostly didn't get deleted.
<wgrant> There's still a tonne around.
<lifeless> wgrant: check a histogram of person ids on 100K intervals
<lifeless> stub: what about WARNING in the script (so losas see it) if something isn't in EVIL, but not aborting.
<stub> lifeless: we used to be at >10million rows
<stub> lifeless: Yes, so we proceed into downtime hoping everything is working fine.
<stub> I'd rather not hope. I'd rather we paused and thought about things a little bit.
<wgrant> stub: But the highest person ID is only like 3.6m
<wgrant> And account ID is 4.8m.
<StevenK> DROP TABLE Account;
<stub> Could have sworn we got to 10million. Might be thinking of account?
<wgrant> 2.2m people no longer exist.
<stub> Graphs will have the answers if anyone wants to wait for them to load ;)
<wgrant> I suspect lots of the remaining ones shouldn't either.
<wgrant> Account retains 4.6m of its 4.8m rows :/
<wgrant> I suspect 3.4m of those are unused.
<wgrant> Now that shipit is deceased.
<StevenK> How can we tell, and can we just delete them?
<lifeless> stub: so you're placing a premium on us having *unidentified* code that is both (bad) and (harmful to interrupt).
<lifeless> stub: I'm placing a premium on failing to do a FDT being disruptive to velocity and our squad leads having identified (harmful to interrupt) cases already.
<lifeless> stub: how can we better refine the expected outcome of e.g. me being wrong and something horribly harmful and bad coming along
<lifeless> stub: one way would be to list everything not fragile as evil, going from the list of db users
<lifeless> anything we think interrupts could be harmful to wouldn't be evil.
<lifeless> (and would cause a stop)
<lifeless> but this seems, to me, to be equivalent to fragile
<stub> lifeless: I think the rare occurrence of delaying a FDT a day or two (without any downtime) is worth us not guessing about what is safe to terminate or not. We have done a lot of FDT now, and identified two processes that are a problem.
<stub> If it is not rare that some new is badly behaved or an existing process starts misbehaving, then we have a different problem.
<stub> Why has this process changed? Why is this new process behaving badly? Has the system become unstable enough that we shouldn't risk further changes?
<stub> Have the assumptions we used when designing fastdowntime deploys become invalid?
<lifeless> wgrant: 200ms is as fast as I can get it tonight; stub may do better
<stub> I haven't seen the query yet
<lifeless> stub: the pastebins ;)
<stub> I don't see the query we are trying to optimize in the pastebin I have.
<lifeless> stub: separately: functional index - 35s, cached sort column, 8s - count distinct over person (1.4M rows) - hot index
<stub> oic. found it
<lifeless> stub: two pastebins: http://paste.ubuntu.com/731585/ and http://paste.ubuntu.com/731544/
<lifeless> bah, yes
<stub> My first question is why on earth would we want to do this query? Being able to hide from security reports by changing your name to match someone elses doesn't seem a good idea.
<lifeless> stub: they can't do that :) - name is unique
<lifeless> stub: its the url component
<stub> So the distinct on is redundant
<lifeless> yes, but wgrants first query needs the distinct to eliminate 40K duplicate rows
<lifeless> my rephrased one focused on person only finds the 17K interesting rows inthe first place
<lifeless> this is the fastest I have (so far):
<lifeless> EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT distinct person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person.temp_sort_key LIMIT 75;
<lifeless> where temp_sort_key is just a materialised person_sort_key
<lifeless> it jumps around but hovers aorund 300ms
<lifeless> stub: i'm thinking about the other stuff; need to cook dinner now
<stub> yup
<stub> Bah. setup pastebin missing a bracket
<stub> Or gnome ate it
<stub> I can't see any obvious ways of improving those queries more. I suspect we don't care enough to materialize the person_sort_key and should just use displayname or lower(displayname)
<stub> (although lower(displayname) won't give unique ordering, blergh)
<stub> EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT DISTINCT ON (person) person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY lower(displayname),name LIMIT 75;
<stub> Seems a little faster actually... probably cache artifacts.
<stub> yer - performs the same as using temp_sort_key
<jtv> Who can help out with an openid-and-account-merging problem?  https://answers.launchpad.net/launchpad/+question/177539
<jtv> Come on people, not everybody at once please.  :)
<jtv> stub, lifeless: any ideas what to do about that one?
<lifeless> jtv: they need to merge them separately in SSO AIUI
<wgrant> stub: I was initially doing DISTINCT ON (Person.name). But that wasn't fast, so I tried using the other indexed values that we normally use (person_sort_key). Those aren't real queries, but they're close :)
<stub> wgrant: yer. surprised the standard join version comes in at twice lifeless' version though. But selecting 55k rows, distilling & ordering looks like it takes 2-300ms.
<wgrant> stub: Yeah :/
<lifeless> I was surprised at the slow too
<adeuring> good morning
<mrevell> Hi
<allenap> Hello.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 275
<mrevell> hiw bigjools
<mrevell>  or rather, "hi"
<bigjools> mrevell: yello
<bigjools> ok, how much do you know about copying packages in PPAs?
<mrevell> bigjools, Relatively little.
<bigjools> mrevell: ok I'll show you an example page
<bigjools> mrevell: https://launchpad.net/~launchpad/+archive/ppa/+copy-packages
 * mrevell looks
<mrevell> Righto.
<bigjools> mrevell: so at the moment, people click the boxes and the buttons and it tries to do all the copying inside a webapp request
<mrevell> Right, I see.
<bigjools> this frequently times out
<mrevell> I can imagine :)
<bigjools> because there's a million checks to make when copying
<bigjools> we now have an infrastructure to do copying in the background in a job
<bigjools> done as part of derived distros
<bigjools> so I'd like to switch over to using that
<bigjools> but there are two problems:
<bigjools> 1. some people still want to do "instant" copying
<bigjools> 2. if we do the background copying, we have no way of providing feedback if it fails
<bigjools> so my question is, do you feel up to providing some input on that?
<mrevell> I certainly do.
<mrevell> I'd like to ask some questions, first.
<bigjools> sure
<mrevell> wrt the first point: how important do you judge this need to be? We can look at it in more detail etc but what's your feeling right now?
<bigjools> sort of.  people are used to it, it would be hard to take away. I think wgrant had some strong opinions?
<maxb> Also, there are scripts which invoke API methods to copy packages, and may legitimately (IMO) assume that a successful method return indicates a successful copy
<bigjools> ah yes, I forgot about that
<maxb> Copying a single source package generally does not time out
<mrevell> bigjools, Would it be naive of me to ask if we can just hand the job to the longpoll infrastructure and then have a live status box on the page?
<bigjools> maxb: there's a new API call for background copying
<maxb> Just so long as the synchronous one doesn't go away
<bigjools> it won't
<mrevell> As for instant copying, I can imagine it might be fairly straightforward to have a UI that allowed for instant copying of a single package
<mrevell> but explained it'd be a batch job for anything more.
<bigjools> mrevell: longpoll would complement this for sure, but we'd need it to work without as well
<mrevell> Huw can help us design that.
<mrevell> bigjools, Forgive me for asking, but why does it need to work without it?
<bigjools> mrevell: basically lifeless is insistent that we don't depend on rabbit
<bigjools> I sort of agree
<mrevell> Okay, fair enough.
<mrevell> bigjools, So, we could we email people if there's a failure?
<bigjools> it's an option, but one that I dislike
<mrevell> Why?
<bigjools> email indicates you have a UI failure
<bigjools> we all get too much email already
<wgrant> Sorry, back.
<mrevell> Okay, fair points; particularly the second about getting too much email already.
<bigjools> now it *could* be justified in that we already get email notifications for uploads anyway
<wgrant> So, email is probably wrong.
<wgrant> We should provide an API method to poll.
<mrevell> bigjools, This is something we could fix with a personal dashboard.
<wgrant> And longpoll for the normal case.
<bigjools> wgrant: longpoll
<wgrant> Right.
<bigjools> we already have API to see if a package is present
<wgrant> We should identify syncSource callers and ask known ones to migrate.
<mrevell> bigjools, There'd be a notifications area. We could even push out a Twitter notification. "@bigjools Your package copy failed. See pad.lv/fail1234 for more detail."
<bigjools> well we can keep syncSource
<wgrant> Of course, there's no reason that reasonably sized copies can't be done in the webapp.
<wgrant> It's just that our schema and code are terrible.
<mrevell> wgrant, Do we have any idea of what is reasonable?
<wgrant> Mostly the code is terrible.
<wgrant> mrevell: Not really, no.
<bigjools> mrevell: I think these are all great - there are Rabbit â Twitter etc. gateways
<wgrant> mrevell: Nobody has spent much time optimising the copier.
<mrevell> Are we safe to say that, generally, a single package copy will work in the webapp?
<mrevell> bigjools, But we need a solution that doesn't rely on Rabbit, right?
<bigjools> however much it's optimised, it'll always fail given too many packages
<mrevell> And is a single copy the primary use case?
<bigjools> mrevell: for throw-away notifications I think it's fine. what's important is that the *only* way doesn't involve rabbit
<bigjools> single copy can fail actually
<StevenK> mrevell: Generally. And then the kernel team tries to copy 'linux' and it constantly times out.
<mrevell> bigjools, So, we need a notification area on a personal dashboard.
<bigjools> that --^
<lifeless> bigjools: mrevell: a bit of nuance on rabbit
<mrevell> Blimey, okay
<bigjools> mrevell: or a notification area on the PPA page
<lifeless> I'm saying we can't depend on it not being rebooted etc
<bigjools> mrevell: which is better in this case as others will want to see the copy
<lifeless> I think its fine for something *we can accept occasional loss on* to be rabbit only
<lifeless> but we need to explicitly make that choice.
<bigjools> lifeless: right.  Like twitter notifications
<lifeless> right
<mrevell> bigjools, The beginnings of a wall for the PPA, effectively
<lifeless> the reality is that such loss will be extremely rare
<lifeless> because a graceful reboot will keep persistent rabbit messages
<bigjools> mrevell: I was thinking of a notification area in the PPA page that shows in-progress and failed copy jobs
<lifeless> its only if ackee shits itself :) that we'll lose in-flight messages.
<mrevell> Thanks for the clarification lifeless.
<bigjools> and you can ack the failed ones which makes them go away
<mrevell> bigjools, Sounds ideal.
<mrevell> bigjools, Let's involve Huw.
<mrevell> bigjools, Is the Twitter thing realistic?
<mrevell> bigjools, Within the scope of a maintenance task, I mean.
<bigjools> mrevell: excellent - if he can design something that'd be fab.  I am away for 2 weeks from Thurs, but I can send an email describing what we need
<lifeless> so you could, for instance, do rabbit only notifications, and if we have a hardware failure on rabbit, we lose some
<bigjools> mrevell: sorta, depends on how hard it is
<StevenK> I don't like the idea of Twitter
<mrevell> bigjools, Please do. Huw can get something done this week.
<bigjools> we also want an RSS feed based on Rabbit notifications
<lifeless> mrevell: twitter - I'll need to talk to elmo about how we should interact with twitter, but in principle doable
<mrevell> StevenK, Or identi.ca. I'm looking for a way to do push notifications that doesn't involve email.
<bigjools> there's a million things that soyuz does that we need notifications for
<lifeless> mrevell: (firewall aspects basically)
<mrevell> lifeless, Sounds good. I'm not totally sold on it but "Why doesn't LP send me a message on Twitter when X happens?" seems to come up every now and then.
<StevenK> mrevell: I don't like it for the concept, not the use of Twitter.
<lifeless> mrevell: I'm not arguing for or against, just letting you know :)
<mrevell> StevenK, Not even as a back-up to a notifications area on the PPA npage?
<lifeless> mrevell: wearing my TA hat I want a scalable secure pub-sub to the internet facility for all of LP, which would let folk external to lp do interesting things like twitter notifications
<mrevell> lifeless, Yeah, thanks :) I mean, if there are mountains to be moved to get Twitter notifications, then I'd really rather spend to time ... and research ... looking at whether we really want to do this.#
<lifeless> mrevell: but we haven't built that
<lifeless> mrevell: I have a design in mind though, FWIW :)
<mrevell> lifeless, Right, precisely. I'd rather do it properly than just tack something on for this one use case.
<bigjools> lifeless: Rabbit has all sorts of useful gateways :)
<StevenK> mrevell: I seriously want notifications about stuff I'm doing within Launchpad to stay within Launchpad.
<mrevell> StevenK, There's no reason we couldn't make it opt-in.
<bigjools> yes there is the privacy issue
<lifeless> bigjools: its a likely component, I'm actually thinking pshb as the primary protocol, rabbit can be used to shove notifications around internally, and pingthesemanticweb.com externally; I have a rough idea for handling private objects even.
<mrevell> Like I say, this is something that people ask for but that I havne't necessarily given all that much thought.
<StevenK> The fact that I *despise* Twitter/identi.ca might be shining through ...
<mrevell> However, I'm generally in favour of Launchpad pushing notifications out to where people want to get them, rather than forcing them to come to LP.
<bigjools> #luddite
<mrevell> StevenK, In which case, don't opt-in
<mrevell> bigjools, Haha!
<mrevell> StevenK, I think it's right not to force people to use it.
<bigjools> +1 t o push
<wgrant> Does Twitter do private tweets?
<wgrant> We have lots of private notifications :)
<bigjools> DMs
<mrevell> wgrant, There's direct messaging but ... whether we consider that private or not is another matter.
<bigjools> ok thanks for the input everyone
<stub> I think it would be awesome if people could choose from a variety of notification methods, making email addresses in Launchpad optional.
<stub> email/jabber/twitter/identi.ca probably covers the bulk of our userbase.
<lifeless> facebook.
<lifeless> would cover nearly all :P
<allenap> jelmer: Can you help me with mardy in #launchpad. He's having problems doing a git import, http://launchpadlibrarian.net/84700393/mardy-webcredentials-libaccounts-glib-trunk.log
<allenap> ?
<bigjools> wgrant: did you ever look at the debtags bug?
<jelmer> allenap: looking now
<allenap> jelmer: Ta, thanks.
<stub> lifeless: Facebook can be accessed via email
<wgrant> bigjools: I mostly ignored it.
<bigjools> wgrant: I'm giving it an initial analysis - I'm a little confused as to the aims of the bug
<bigjools> bug 57418
<_mup_> Bug #57418: Support debtags in Packages.gz <escalated> <feature> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/57418 >
<bigjools> seems like we need another import :/
<wgrant> bigjools: Well, yes, we need to work out what they want. I strongly suspect that using Debian's DB directly will be unacceptable, so we'll need to DBify it or have a custom Ubuntu one.
<bigjools> indeed
<wgrant> (because we have lots of local package)
<wgrant> +s
<bigjools> which brings the question of merging upstream's
<wgrant> And that is the stage at which I play the midnight card and disappear :)
<bigjools> heh
<wgrant> For it is a mess.
<bigjools> it's an escalated mess
<wgrant> Very similar to P-a-s.
<bigjools> hence I am trying to clear it up
<bigjools> I am not doing another P-a-s
<wgrant> Yeah, but escalated Soyuz issues are all your problem now :)
<wgrant> Heh.
<wgrant> Well.
<wgrant> It'd be easy to.
<bigjools> that's the problem
<bigjools> it's soooooo tempting
<wgrant> As is sleep.
<wgrant> So I shall leave you to temptation.
<bigjools> jml: are you involved with the request to support debtags?
<bigjools> wgrant: good night
<jml> bigjools: yes. I asked Francis when we could expect it and whether there were any plans / specs in case we wanted to do the work.
<bigjools> jml: I am doing some analysis
<bigjools> jml: I wondered if the requirement was to use Debian's debtags database or to start a new one for Ubuntu?
<bigjools> it's not specified
<jml> bigjools: cool. If you need requirements from us, mvo & james_w are the best to ask.
<jml> bigjools: I *think* we'd need to start a new one
<jml> bigjools: but mvo would know for sure.
<bigjools> jml: ok cool - do you guys have an IRC channel of your own?
<jml> bigjools: oh boy, do we ever! #ubuntu-app-devel, #software-center on Freenode
<bigjools> ok I'll pop in after lunch
<bigjools> cheers
<jml> np
<sinzui> mrevell, Did you ever send StevenK the text for the team PPA emails?
<mrevell> StevenK, I did, yes, this morning.
<sinzui> fab
<nigelb> Gmail thinks every mail bigjools writes is very important :P
<sinzui> mrevell, did you see the email I sent to the list about open/closed teams. Maybe Dan can help solve the language issue?
<mrevell> sinzui, I did; I'm hoping to reply later. Dan'll be back tomorrow.
<sinzui> thanks
<bigjools> nigelb: gmail is clever
<nigelb> bigjools: :)
<sinzui> jcsackett, ping
<jcsackett> sinzui: pong.
<sinzui> jcsackett, do you have a few minutes to mumble?
<jcsackett> of course.
 * jcsackett fires up mumble.
<deryck> abentley, adeuring -- you guys see any reason we shouldn't turn on new listings for ourselves now?
<adeuring> deryck: sounds fine
<abentley> deryck: we might get timeouts, and we can't turn them off.
<deryck> abentley, are you opposed to trying, seeing what it's like, and if arise, disabling the flag?
<abentley> deryck: no.
<deryck> great, thanks!
<abentley> deryck: Just be sure to disable wallyworld's stuff, if it's enabled for us.
<mrevell> hey abentley, I just need to grab a coffee; time ran away with me. Five minutes.
<deryck> abentley, ok, will do.  I'll make the rules match qastaging.
<abentley> mrevell: no problem.
 * bigjools just deleted a bugtask.. Awesome
<abentley> deryck: dynamic bug listings are currently disabled on qastaging for me.
<deryck> abentley, ok.  I think you need to be in the demo group.  I'll look here in a sec.
<abentley> deryck: Also, the rule for disabling ajax.batch_navigator.enabled is set for orange squad, when it should be set for custom-buglisting-demo
<deryck> abentley, ok.  And I don't need that rule on lpnet if it's not turned on at all on lpnet, right?
<abentley> deryck: right.
<deryck> ok, cool.
<deryck> abentley, I added us back to the demo team.  Things seem to work ok, until I can change the flag for the other feature to the demo team.
<abentley> deryck: great.  Thanks.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<mrevell> Night all
<bryceh> deryck, quick question (raised by one of our upstreams) - do you know if the bugzilla comment importer is enabled for GNOME?
<deryck> bryceh, hmmm, I don't actually know.  I recall it being raised some time ago about wanting to get it working....
<deryck> bryceh, I may have dropped the ball on getting it enabled.
<bryceh> deryck, ok thanks
<deryck> bryceh, I think it would be better for one of the maintenance squads to check into it... perhaps yellow, which has gmb on it, since he knows a lot about that area.
<deryck> bryceh, I think that's why I dropped it before, just caught in feature work like I am now.
<bryceh> deryck, ok good to know.  Yeah at this point it's not a request, just question about the status of it
<deryck> ok, cool.
<deryck> abentley, you around still?
<bac> hi lifeless, the test i added to the branch you reviewed yesterday is causing other worker tests that run after it to fail.  can you see anything i have bungled?  http://pastebin.ubuntu.com/732539/
<lifeless> bac: getMirrorFailureForException is a helper, not a test itself
<lifeless> bac: what is the test that fails (and does it fail when run on its own ?)
<lifeless> bac: one possibility is that the test isn't twisted safe - the call to worker.mirror() may be assuming synchronous behaviour, which isn't a guarantee in any twisted environment.
<lifeless> bac: if that is the case, the helper will need to be reworked to use deferred and so forth
<lifeless> :(
<abentley> deryck: kinda.
<deryck> abentley, one question.... I assume I need to call change_fields with the right config for my integration work.  true?
<abentley> deryck: Right.
<deryck> abentley, awesome, thanks!  Have a good night.
<abentley> deryck: The list of fields is around browser/bugtask.py:2241
<deryck> abentley, thanks
<abentley> deryck: kp
<abentley> deryck: np
#launchpad-dev 2011-11-09
<wgrant> Ooh.
<wgrant> The new bug listings are mostly nice.
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 275
<wallyworld_> StevenK: you working on 884217?
<StevenK> wallyworld_: I have a branch for it
<wallyworld_> StevenK: the reason for asking is that i am thinking about implementing a security adaptor such that if the logged in user can view the context undet which team name visibility is required, they can see the team name
<wallyworld_> so if the team name needs to be displayed because the team is a bug subscriber, then the context would be the bug
<wallyworld_> these rules would replace any current security adaptor for private team visibility
<StevenK> I think we only have the TAL formatter
<wallyworld_> StevenK: so i wanted to be sure that we weren't going to do the same thing etc
<StevenK> Which returns <hidden> and soft OOPSes a MixedVisibilityError
<wallyworld_> i'll need to dig around and see what needs to be done
<wallyworld_> to fix it
<wallyworld_> but i think the above sounds ok?
<jtv1> StevenK: [BS]PPH.{binary,source}packagename have been fully populated now?
<bac> hi, lifeless, sorry i had to run
<StevenK> jtv: Not even close.
<jtv> Oh.
<jtv> Damn RaphaÃ«l and his clever ideas.
<bac> lifeless: the tests that actually failed are those that use the helper, such as testBadUrlBzrSshCaught
<jtv> StevenK: I guess you'll add not-null constraints when done.
<bac> and they do pass on their own.  they only fail if run after the new test i introduced, which makes me think i've done something Bad(tm) in that test
<jtv> hi bac :)
<bac> hi jtv
<StevenK> jtv: It is being blocked by long running transactions from the publisher, and if that isn't running, then it's the backups.
<lifeless> bac: thats very unusual then isn't it!
<bac> lifeless: yeppers
<lifeless> bac: uhm, I'm not entirely here today; can you drop me a mail and I will look first thing tomorrow ?
<jtv> StevenK: ironic, since I was also hoping to use the column to speed up the publisher.
<bac> lifeless: sure.  thanks.
<bac> jtv: staying dry?
<jtv> bac: moving around to do so, but yes.  In the house, last we know is it got up to just below the kitchen counter-top.
<jtv> The hip new word is revacuee.
<jtv> StevenK: perhaps there's more we can do to break up those long-running publisher transactions?
<StevenK> Fixing the dominator would be one, I think.
<jtv> Catch 22.
<jtv> StevenK: the updater code looks like it's probably slower than necessary.
<StevenK> jtv: You need my denorm work to make the dominator faster?
<jtv> It'd help, yes.  I've got about 5 branches waiting for rollout, and those will probably help, but it's getting close to the end of the low-hanging fruit.
<jtv> Some things that I think would speed up the updater:
<jtv>  * Don't order by id.
<jtv>  * Bulk-load the [BS]PRs.
<jtv>  * Don't load the [BS]PNs at allâall you need is their ids.
<jtv>  * More radical: don't load anything, just let the DB do the work.
<poolie> is this change to block reopening fixreleased bugs new?
<jtv> poolie: EWIN?
<poolie> no
<wgrant> poolie: It's at least several months old.
<wgrant> Maybe even a year now.
<poolie> perhaps i just never hit it
<wgrant> poolie: The reporter and bug supervisor can reopen.
<wgrant> But not eg. the assignee.
<poolie> hm
<poolie> i guess i'm not a supervisor for bzr-svn
<poolie> :/
<wgrant> Indeed.
<wgrant> That's ~bzr-svn, which has only three real members.
<poolie> ffs
<poolie> i see, i thought it changed recently because robert just answered a year-old question about it
<poolie> wgrant, StevenK, lifeless, what are we going to do with https://code.launchpad.net/~vorlon/launchpad/sbuild-multiarch-syntax/+merge/81385 ?
<wgrant> poolie: Land it.
<poolie> *hug*
<wgrant> poolie: Don't add [no-qa] to the commit message.
<wgrant> Rather use bzr lp-land --no-qa
<wgrant> Or ec2 land --no-qa
<wgrant> That adds it automatically.
<wgrant> And hopefully avoids confusing things.
<poolie> ok
<poolie> i wonder if there is a bug for it
<wgrant> I don't think so. I've discussed it with various people in person, but I don't recall a bug.
<poolie> not obviously
<poolie> and it's out-of-band qa anyhow
<micahg> poolie: I think we have security builds going for the next 9 hours
<micahg> eh, semi-wrong channel, meh
<wgrant> Hah.
<wgrant> There's no UNIQUE bugsubscription(bug, person)
<StevenK> The UI forbids it, though?
<wgrant> Probably mostly.
<wgrant> poolie: Not going to make that bug Critical+regression? :)
<poolie> i guess i could
<poolie> rebooting
<nigelb> jamesh: Hi, around?
<nigelb> Ah, its lunch time in Perth.
<jamesh> nigelb: hi
<nigelb> jamesh: Hey, are you the James listed as Uploader on PyPI for django-openid-auth? :)
<jamesh> nigelb: yeah
<nigelb> jamesh: Excellent, 0.4 was uploaded as django-openid-auth_0.4.tar.gz as opposed to -0.4 which leads to pip not finding it :(
<nigelb> It was fun tracking that down, pip search django-openid-auth says 0.4 is latest, but installing doesn't work.
<jamesh> nigelb: ah. I didn't actually roll the 0.4 release (achuni handled it, iirc)
<nigelb> jamesh: ah. I couldn't find achuni on IRC, so I thought best to let you know :)
<jamesh> nigelb: could you file a bug about it so we don't mess it up for 0.5?
<nigelb> cool, will do.  Is there a way to fix 0.4?
<jamesh> nigelb: I don't really use pip.  Where is the error occurring?
<nigelb> jamesh: let me get you a pastebin
<nigelb> jamesh: http://paste.ubuntu.com/732749/
<nigelb> Hrm. I guess I could try installing _0.4
<nigelb> Nope, didn't work.
<jamesh> nigelb: anything relevant in the log file that paste mentions?
<nigelb> yup
<nigelb> sec
<nigelb> jamesh: http://paste.ubuntu.com/732755/
<jamesh> nigelb: that's weird: Could not fetch URL http://launchpad.net/django-openid-auth/trunk/0.4/+download/django-openid-auth_0.4.tar.gz (from http://pypi.python.org/simple/django-openid-auth/): HTTP Error 404: Not Found
<jamesh> checking that here, I get "303 See Other" response pointing at the file in the librarian
<nigelb> Oh.
<nigelb> That is weird.
<wgrant> stub1: around?
<poolie> i think i will build a package of the buildd stuff now just to test
<poolie> and then do an update after vorlon's stuff lands
<wgrant> poolie: Sounds good.
<nigelb> jamesh: I'm guessing the usual download is directly from PyPI which it can't find now.  Anyway, filed bug 887882
<_mup_> Bug #887882: Can't install django-openid-auth 0.4 via pip <django-openid-auth:New> < https://launchpad.net/bugs/887882 >
<huwshimi> wgrant: I saw you said something about the new bug columns earlier. Are they live somewhere or did you just see a mockup?
<jamesh> nigelb: thanks
<nigelb> :)
<wgrant> huwshimi: Join https://qastaging.launchpad.net/~custom-buglisting-demo (it's Open) and you'll see them on qas.
<nigelb> oooh.
<poolie> hi nigelb
<poolie> wgrant, only qas for now?
<nigelb> Afternoon poolie :)
<huwshimi> wgrant: Ah, perfect, thanks
<stub1> wgrant: yo
<wgrant> poolie: Yes.
<nigelb> zomg. Can't wait. custom bug listings is *amazing*
<wgrant> stub: It was suggested that I try the new schema without the denormalisation (so, adding policy to AccessPolicyArtifact, meaning that AccessPolicyGrant has either policy or artifact set, not both).  I managed to get the equivalent Ubuntu query down to only 10% slower than the denormed version on DF (~650ms), but for products it's around ~100ms, which is slower than the original.
<wgrant> This seems to be because it always does a seq scan on accesspolicygrant.
<wgrant> Rather than indexing into it.
<wgrant> Can't work out why.
<wgrant> http://paste.ubuntu.com/732768/ is the setup and some sample queries.
<wgrant> The third query (one CTE and some subselects) is the fastest for Ubuntu. The others sit at 1-1.5s.
<wgrant> But I can't get it to not seq scan apg.
<poolie> that is nice
<poolie> i wish the colors were better chosen though
<wgrant> poolie: What in particular?
<wgrant> I think Low should not be bold and black.
<poolie> i agree
<poolie> i would like the importance things to be on one spectrum, so their ordering is reflected in the colors
<wgrant> Yeah.
<poolie> and i think i would like them not to overlap with the status colors
<wgrant> They're the same as they are now.
<poolie> huwshimi, what do you think?
<poolie> i like incremental changes
<wgrant> Red/Orange/Yellow... Black/Blue
<poolie> but for things like this it seems like it would be a shame not to make a better splash
<wgrant> Hmm.
<wgrant> Actually.
<wgrant> They're the same as the current *text* colours.
<wgrant> Which is Red/Orange/Green/Black/Blue.
<huwshimi> wgrant, poolie: We actually have a set of colours that the design team use for statuses
<poolie> i wonder if that icon next to the product name is really earning its keep
<huwshimi> wgrant, poolie: I haven't seen this black before
<wgrant> Rather than the icons, which are Red/Reddish/Orange/Yellow/Blue
<wgrant> Which probably makes more sense.
<wgrant> Except for Yellow being yellow.
<wgrant> Which is a bit ew.
<poolie> huwshimi, and are these colors on qas the official colors?
<huwshimi> poolie: Nope
<poolie> i guess specifically 'medium' has a very bold green which is more saturated (to my eye) than High
<poolie> and Low even more so
<poolie> and Wishlist too
<poolie> :)
<wgrant> Yeah, Medium and Low are really obvious.
<huwshimi> poolie: I believe these are the same colours we used to have, with the exception of the black
<poolie> i think that's true
<wgrant> Particularly since High/Critical are the same colour as Triaged, which is the usual status for bugs with an importance.
<wgrant> So there's no contrast.
<huwshimi> wgrant: I had thought the status colours were going to be removed
<wgrant> Hmm, interesting.
<poolie> also the small caps importance next to the regular status is a little jarring
<huwshimi> It's been a long time since I've seen this work and clearly I need to do a proper review
<poolie> and they are oddly aligned
<wgrant> There was a bug filed overnight about Fix Committed looking like an ajax link.
<poolie> 'ajax link' :)
<wgrant> Bug #887576
<_mup_> Bug #887576: new bug listing importance is difficult to read <bug-columns> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/887576 >
<wgrant> Bug #887575
<_mup_> Bug #887575: fix released and fix committed status color in new bug listing look like ajax action links <bug-columns> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/887575 >
<wgrant> Interestingly, the first bug's title mentions importance, but description says status.
<wgrant> Hmm.
<wgrant> GitHub's status styleing on the issue page itself is suspiciously similar to our new listings' importance styling.
<poolie> wgrant, yeah, and unfortunately there is a bit more logic to them styling them that way
<wgrant> Yes.
<poolie> which is that they are tag-type things that can be dragged around
<wgrant> Well, labels are separate, but all the category-related things seem to be styled like that.
<wgrant> Labels, statuses, filters.
<wgrant> And Google Code's issue tracker is just hilarious.
<wgrant> huwshimi: Are the desired colours shown somewhere?
<huwshimi> wgrant: Not that I know of
<huwshimi> wgrant: I have some hex codes
<wgrant> Do their values map reasonably well onto ours?
<huwshimi> wgrant: Yeah, let me grab a screenshot
<wgrant> poolie: What do you think of the general prominence of the importance?
<wgrant> poolie: It seems like it's massively emphasised over the status.
<wgrant> When the status is possibly more important.
<poolie> i find the stacked layout kind of hard to scan
<poolie> especially because the colors overlap
<wgrant> Yeah, that's a worry.
<huwshimi> wgrant: http://people.canonical.com/~huwshimi/status_colours.png
<wgrant> But maybe that's less important once we have good sorting and filtering.
<wgrant> huwshimi: Hmmmmmmmmm.
<poolie> i think we ought to ask: why are we showing this at all
<poolie> what kind of thing do people do with this information
<poolie> ...
<poolie> "oh, that's new/untriaged, i'll go and do it now"
<wgrant> Entirely desaturating the status was not the solution I had in mind.
<wgrant> Sort of the opposite :)
<wgrant> The importance is really more like a tag, IME.
<poolie> "of the bugs with this tag, 2 are in progress, 3 fixed, 3 triaged"
<wgrant> An unimportant tag.
<poolie> i think also whether projects accept it or not, there's "critical, high, other"
<poolie> oh, and null
<wgrant> Yep.
<poolie> i do wonder if the scale should reflect that by having nonlinear differences in color - http://people.canonical.com/~huwshimi/status_colours.png is a bit better in that respect but not quite there
<wgrant> Yeah.
<wgrant> "Other" is split.
<wgrant> But it's clearly separate from Critical and High.
<wgrant> By being cooler.
<poolie> so i would have eg "red, orange, yellow, faded yellow, dull yellow"
<wgrant> The colours in that are much more subtle, too.
<wgrant> Much better.
<wgrant> The blocks of colour also look pretty odd next to our detailed icons, I think.
<poolie> i do generally like the colors
<poolie> i think
<wgrant> It's all pretty simple and not busy, but then you get to these variously-coloured outlined detailed icons :(
<poolie> the statuses are less jarring because they are not colored at all
 * wgrant campaigns for monochrome icons.
<wgrant> Yeah.
<poolie> but it seems based on an assumption that the importance is much more salient than the status
<poolie> which i don't think is true
<wgrant> Indeed.
<poolie> ... and again that should probably come out of a theory about what people use this information for
<wgrant> Well.
<wgrant> It really depends.
<wgrant> Depending on the filter.
<poolie> i don't think it's universally true
<poolie> right
<wgrant> eg. we by default filter to open bugs.
<poolie> yeah
<poolie> you know i'd really like the filters to start having an overridable opinion about what you're using that filter for
<poolie> eg inprogress should show you to whom they're assigned
<wgrant> But it's still useful to distinguish between those that are untriaged (although they would have NULL importance), those that can be worked on, and those that are being worked, and those that are fixed but not released.
<wgrant> Yeah.
<wgrant> So.
<poolie> yes
<wgrant> Really the status is just noise normally.
<poolie> i think it at least some uses it'd be good to highlight "in play" statuses
<wgrant> Everything can be distinguished from importance and assignee.
<poolie> well, really just inprogress, and maybe fixcomitted
<wgrant> New: unset importance
<wgrant> Triaged: set importance
<poolie> well, that too
<wgrant> In progress: set assignee
<poolie> get rid of a few statuses
<wgrant> Incomplete... not sure
<wgrant> Fix Committed: pointless
<poolie> i think incomplete is an orthogonal bit
<wgrant> Yeah.
<poolie> "needs more input", ie developers can't work on it (and it will eventually expire) if an affected user doesn't provide more info
<wgrant> I've thought about abolishing status for issuetracker.
<poolie> i think this works pretty well  in answers
<wgrant> invalid/wontfix/opinion are all one status
<wgrant> fixcommitted is largely pointless
<wgrant> incomplete can apply to most states, so should be separate
<wgrant> new is implied by there being no importance
<wgrant> inprogress is implied by there being an assignee
<poolie> ... anyhow, one step at a time
<wgrant> Heh
<poolie> i think one good thing huw's retheme can hopefully do is provide a consistency about where color is used and how much
<wgrant> Yeah.
<wgrant> huwshimi: Do you have plans for our iconset?
<poolie> i would hope it says it's used for important things
<poolie> kill the goddamn pencil
<huwshimi> wgrant: Yeah, I have plans to redesign it.
<wgrant> The pencil was scheduled for demolition 2.5 years ago :)
<wgrant> huwshimi: Excellent.
<huwshimi> wgrant: I haven't done much work on it yet, but it's probably going to be monochrome
<wgrant> Eeeexcellent.
<huwshimi> wgrant: At least in the current mockups it has been
<poolie> grey plus purple anyhow
<micahg> ok, so I see the new delete task feature, but I'm not sure, if I delete a default task, does that remove the targetted tasks as well or do I have to target the devel release to delete the default task or is there still no way to delete the default task?
<wgrant> micahg: You can delete any task individually, as long as at least one task is left on the bug.
<wgrant> micahg: This does mean that you can have eg. a linux (Ubuntu Lucid) task without a linux (Ubuntu) task.
<micahg> right, but like in bug 887339, I don't need the default tasks, but I need the targetted ones
<_mup_> Bug #887339: Tracking bug for Firefox 8 transition <firefox (Ubuntu):Invalid> <mozvoikko (Ubuntu):Invalid> <ubufox (Ubuntu):Invalid> <firefox (Ubuntu Natty):Fix Committed by micahg> <mozvoikko (Ubuntu Natty):Fix Committed by micahg> <ubufox (Ubuntu Natty):Fix Committed by micahg> <firefox (Ubuntu Oneiric):Fix Committed by micahg> <mozvoikko (Ubuntu Oneiric):Fix Committed by micahg> <ubufox (Ubuntu Oneiric):Fix Committed by micahg> < https://la
<wgrant> I would discourage deleting them.
<wgrant> The parent target will still appear, see eg. bug #1234
<_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <lp-foundations> <Launchpad itself:Fix Released by debonzi> < https://launchpad.net/bugs/1234 >
<wgrant> Er.
<wgrant> Not that.
<wgrant> Bug #43150
<_mup_> Bug #43150: [SRU] maxima frontends fail to connect <wxmaxima (Ubuntu):Fix Released> <gcl (Ubuntu Dapper):Fix Released by wgrant> <maxima (Ubuntu Dapper):Fix Released> < https://launchpad.net/bugs/43150 >
<wgrant> The parent targets without tasks are still shown in the table.
<wgrant> Task deletion exists mostly to remove projects that shouldn't be on bugs.
<wgrant> It's not intended for what you suggest, though it will mostly work.
<micahg> right, that's what I'm thinking, to look something like that bug
<micahg> but if it's not advised, I'll leave it
<wgrant> (that bug was never intended to be like that, but it ended up being somewhat broken when series tasks were reworked in 2007)
* spm changed the topic of #launchpad-dev to: qas/staging down | https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugtasks: 275
* StevenK changed the topic of #launchpad-dev to: qas/staging down | https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<jtv> StevenK: oh crap, completely forgot to OCR today.  I'll just join you.
* jtv changed the topic of #launchpad-dev to: qas/staging down | https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugtasks: 275
<StevenK> Or not join me
<jtv> Well, replace you.
<jtv> Or something.
<StevenK> I sit replaced.
* spm changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv | Critical bugtasks: 275
<jtv> morning rvba
<rvba> \o jtv!
<poolie> hi rvba, jtv
<rvba> Hello poolie.
<poolie> so when launchpad runs its tests against the external buildd code
<poolie> it will need a copy of that
<wgrant> poolie: Probably best to put it in sourcecode/ for now.
<poolie> ok
<poolie> yeah it would be a bit weird to make it an egg
<jtv> hi poolie â back from getting some stuff
<jtv> wgrant: any objections to a dogfood update?
<wgrant> jtv: Go ahead.
<jtv> thx
<poolie> package name 'launchpad_buildd' or 'launchpadbuildd' or 'lpbuildd'?
<poolie> maybe the last
<huwshimi> Just heading out, but a branch for someone who is generous enough: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
<danhg> Morning all
<mrevell> Hello
<adeuring> good morning
<jtv> I think this is one of those rare cases where I need full tracebacks from test errors.  Anyone know how I get the full traceback instead of the normal abridged one?
<jml> jtv: it's a bit tricky.
<jml> jtv: try decorating the test with @run_test_with(FullStackRunTest)
<jtv> That doesn't sound too tricky!
<jml> jtv: from testtools import run_test_with; from testtools.tests.helpers import FullStackRunTest
<jtv> Thanks.
<jml> jtv: it won't give you the full stack though, it'll just reveal the testtools-specific stuff
<jtv> We'll see how far it gets me.
<jml> jtv: bug 881778 and bug 881780 have been filed already on testtools and are my next thing to do for it.
<_mup_> Bug #881778: frame hiding cannot be disabled, interferes with debugging <regression> <traceback> <testtools:Triaged> < https://launchpad.net/bugs/881778 >
<_mup_> Bug #881780: no way to transiently disable frame hiding <traceback> <testtools:Triaged> < https://launchpad.net/bugs/881780 >
<jtv> Alas, it didn't do the trick.
<jml> jtv: did it not work at all?
<jtv> Actually, the "traceback" consists of _only_ the exception description.  I've been assuming that that was just because the abridgment eliminated the whole thing, but perhaps there's some other way that could happen?
<jtv> Traceback (most recent call last):
<jtv> DNSLookupError: DNS lookup failed: <etc.>
<jtv> Just those two lines.
<poolie> can someone have a squiz at https://code.launchpad.net/~mbp/launchpad-buildd/rename-package/+merge/81692
<jtv> poolie: after my standup, sure
<allenap> jtv: Little review? https://code.launchpad.net/~allenap/launchpad/bug-export-schema/+merge/81699
<jtv> allenap: otp; would half an hour be OK?
<allenap> jtv: Absolutely, it's not urgent.
<allenap> Thanks.
<jtv> cool
<jml> hey, diffs seem to be taking a long time to generate.
<jml> since posting my MP, I've gone downstairs & filled up a bottle of water, greeted the postman (not a euphemism), and then put a bunch of cardboard boxes in the loft. Still no diff.
<lifeless> jml: single threaded, possibly behind something big.
<bigjools> that would be a great euphemism though
<lifeless> jml: your bug about reviews
<lifeless> jml: isn't https://code.launchpad.net/~lifeless/+activereviews roughly the view you ask for ?
<nigelb> ooh, it was jml  that asked for this at the launchpad session?
<nigelb> lifeless: NICE!
<nigelb> lifeless: I maybe seeing a regression. On the project page now it doesn't show the different categories "Reviews I'm waiting", "Reviews I can do", etc
<lifeless> https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> does for me
<nigelb> what the....
<nigelb> it didn't work 2 minutes ago.
<nigelb> It picks every review I'm associated with. Like.
<lifeless> check the url
<lifeless> bet you there is a difference
<nigelb> Probably my mistake
<nigelb> Anyway, this is nice :)
<lifeless> e.g. +reviews vs +activereviews
<lifeless> or something
<jml> lifeless: no.
<jml> lifeless: or rather, it's close, but I want reviews that aren't assigned to me to appear on it. If you submit an MP to, say, it won't show up on my page.
<nigelb> "Reviews I *can* do"
<jml> well
<jml> yes. although there's a lot of those :)
<nigelb> heh
<jml> restricting it to projects I'm interested in would be nice.
<jtv> allenap: you're done.
<allenap> jtv: Thanks.
<lifeless> jml: so, I think it just needs a way to say 'I may be a reviewer for X, but I don't care about it'.
<lifeless> jml: one way is to not be in the reviewer team for X.
<jml> uhh... no?
<jml> well, depends on what "it" is there
<lifeless> so, +activereviews shows reviews that aren't assigned to you. The heuristics is that they are on a branch you are in the review team for.
<lifeless> what you said above was '00:09 < jml> lifeless: or rather, it's close, but I want reviews that aren't assigned to me to appear on it. If you submit an MP to, say, it won't show up on my page.
<jml> lifeless: no, it doesn't.
<lifeless> '
<lifeless> and then '00:10 < jml> restricting it to projects I'm interested in would be nice.
<lifeless> jml: are you saying you want proposals that you're not an authoritative reviewer for to show up as well?
<jml> lifeless: "as well" is wrong. Person/+activereviews does not show reviews that aren't assigned to me. e.g. kampka's review on https://code.launchpad.net/testtools/+activereviews isn't shown on https://code.launchpad.net/~jml/+activereviews
<lifeless> jml: ah, or are you saying that 'reviews I can do' doesn't show up on your ~ +activereviews?
<lifeless> jml: This may be shallow. Thanks for helping me understand.
<jml> lifeless: yes, that's what I'm saying. That was the intended behaviour of the page when thumper made it.
<jml> tbh, if suddenly there was a "Reviews I can do" section, I'd want it grouped by project.
<jml> lifeless: np
<jml> another slow diff generation :\
<lifeless> jml: naturally
<lifeless> jml: would you care to note this in the bug ?
<jml> lifeless: done
<lifeless> thhanks
<StevenK> jtv: Are you still OCR, or are you EOD?
<jtv> StevenK: sort of eod tbh
<jtv> But!
<jtv> If your branch isn't too big I can trade you a speedup for the [BS]PPH.{source,binary}packagename populators.
* jtv changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<StevenK> In the garbo job?
<jtv> Yes
<StevenK> jtv: Problem is, my branch isn't done yet.
<jtv> Well that'd put a bit of a crimp on it.
<StevenK> A bit, yes
<jtv> bigjools: about a minute and a half â still seems to do the usual stuff though
<StevenK> jtv: Are you still up for that MP swap?
<deryck> Hi, all.
<flacoste> hi deryck!
<deryck> adeuring, ping for standup
<adeuring> deryck: oops, starting mumble now. sorry
<deryck> np
<sinzui> I believe there are 3 private new bugs in Launchpad that no-one can see. This has been the case for a few months
 * sinzui wonders if staging can show them
<deryck> adeuring, are you working from this mockup for the beta banner?  http://people.canonical.com/~deryck/beta_banner.png
<adeuring> deryck: yes
<deryck> adeuring, ok.  And it's that blue on black that's hard to read?
<adeuring> deryck: right
<deryck> adeuring, do you have the transparency, too?
<adeuring> deryck: yes
<deryck> adeuring, ok, cool.  I'd say just keep that match and point huw at the issues you see then.
<adeuring> deryck: ok. BTW, I suspect that the blue colour in the mockup is simply  not identical with the colour we use elsewhere for links
<nigelb> Custom bug lists feature looks amazing btw
<deryck> nigelb, thanks!
<nigelb> :)
<deryck> adeuring, that's what I was wondering.  let me open in gimp and get the exact color.
 * deryck thinks the Oranga Mafia are becoming design gurus
* abentley changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugtasks: 275
<deryck> adeuring, like the gray, it's hard to pin down from mockup, but I think it's #4884EF.
<adeuring> deryck: ok, I'll try it
<abentley> deryck:  could you please review https://code.launchpad.net/~abentley/launchpad/enable-anonymous-jsoncache/+merge/81741 ? (I'm the OCR)
<deryck> abentley, I can.  Give me 15-20 minutes to wrap up some emails for the day, and then I'll start.
<abentley> deryck: thanks.
<deryck> abentley, your MP looks great.  No comments really.  r=me.
<abentley> deryck: thanks.
<deryck> abentley, adeuring, hey look there's rick_h :)
<adeuring> rick_h: welcome!
<rick_h> sorry, should have thought to look for -dev a while ago
<abentley> rick_h: Hi!
<rick_h> howdy all
<deryck> rick_h will be new Orange Mafia next week, for those who don't know.
<jelmer> rick_h: welcome :)
<rick_h> thanks, looking forward to it
<rvba> welcome rick_h!
<nigelb> oooh! Hello rick_h!
<timrc> danhg, welcome aboard and thanks for doing the usability sessions at UDS/P
<danhg> hey, thanks tim
<danhg> was great to hang out
<abentley> deryck: back, forward and reload buttons now work.
<deryck> abentley, nice!
<deryck> abentley, well done.
<lifeless> cr3: hi
<cr3> lifeless: hola senor, what's up?
<lifeless> sorry otp now ;)
<cr3> lifeless: no worries, reping me when done
<lifeless> cr3: hi
<lifeless> cr3: wanted to touch base on the person access api we discussed
<mrevell> night all
<lifeless> cr3: however I have to run in a minute now - monthly allergy vaccine injection
<lifeless> cr3: will try around this time tomorrow
<cr3> lifeless: sure, looking forward to your ping tomorrow
<lifeless> wallyworld_: add member should't need changing
<lifeless> wallyworld_: non-open/delegated teams already prevent addition of open-delegated teams to the team.
<thumper> lifeless: the reason behind avoiding having reviews I can do on the person review page was the potential amount of them
<abentley> deryck: chat?
<deryck> abentley, sure.
<abentley> deryck: on mumble, I mean.
<mwhudson> +                    report.get('value', 'No execption value'))
 * mwhudson twitches
<bac> hi mwhudson
<bac> mwhudson: you afflicted by more than the typo?
<mwhudson> bac: no, just the typo
<mwhudson> bac: i guess i could just fix it myself, sorry
<bac> mwhudson: ok -- i'll fix it.
<bac> mwhudson: turns out 'execption' occurs in our code base three times!
<mwhudson> bac: hahaha
<Lifelrss> Statik: hi
<mwhudson> jelmer: so if the recipe memory problems are fixed, has anyone tried a kernel build yet? :-)
<jelmer> mwhudson: hmm, that'd be an interesting thing to try
<mwhudson> jelmer: john rigby from linaro actually wanted to do it, too
<mwhudson> ah, kernel imports don't scan correctly
<mwhudson> is that going to be a problem?
<jelmer> mwhudson: yeah, that might be a problem
<jelmer> it seems like 2.6 imports fine though
<mwhudson> ah, that's probably been around for long enough
<jelmer> and I have some in-progress work to improve the scanner performance
<jelmer> Branch.revision_history() is deprecated in bzr 2.5, which breaks lots of things in Launchpad, so that code will have to change anyway
<mwhudson> i think it's some new-ish time restriction on how long scanner jobs run for that prevents new imports from scanning
<jelmer> might as well use the new, fancy, APIs
<mwhudson> jelmer: does it involve deleting BranchRevision?
<jelmer> mwhudson: no
<mwhudson> sad fave
<mwhudson> *face
<jelmer> branchrevision scares me
<mwhudson> the things it does that aren't trivial to replace with bzr ops are ... i forget now
<mwhudson> per project revision feeds?
<mwhudson> revisioncache updates in general i guess
<mwhudson> really we should have a microservice for things like "which revisions are unmerged in this mp" blah blah blah
<mwhudson> jelmer: i remember that revision_history started to be bad news around when i started with canonical
<mwhudson> jelmer: some things move slowly :-)
<jelmer> mwhudson: the format moved away from it a long time ago
<mwhudson> yeah
<jelmer> mwhudson: but the API was still there
<mwhudson> there was a version of knits with it and another without it?
<wgrant> jelmer, mwhudson: I've had some luck with manually doing progressive scans of large new imports.
<poolie> huwshimi, hi, there were a couple of things at the google event that i wanted to briefly mention with you
<poolie> also thanks for fixing the tag cloud
<huwshimi> poolie: :)]
<StevenK> abentley: Hai, are you still up for reviewing a branch?
<sinzui> wallyworld_, from canonical.launchpad.webapp.authorization import (
<sinzui>     precache_permission_for_objects,
<sinzui>     )
<poolie> lifeless, re putting launchpad-buildd in an egg, that seemed a bit strange for something that's mostly not python, and has no python packaging at the moment
<poolie> it feels more like it should be a dpkg dependency
<jelmer> wgrant: that's by pushing the branch 10k or so revisions at a time?
<wgrant> jelmer: Yep.
<wgrant> jelmer: Sometimes much less :/
<abentley> StevenK: sorry.
* abentley changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<lifeless> poolie: I want to reduce the number of dependency systems we are using
<lifeless> poolie: for the forseeable future, eggs are needed, and so are dpkg packages.
<lifeless> poolie: so either is fine IMO
<poolie> i just need to make sure installing the package does not convert your machine into a buildd
<poolie> though that might be kind of useful :)
<poolie> especially leading up to the release
<lifeless> well
<lifeless> there are some options
<lifeless> we could mandate having it run in a container
<lifeless> for instance
<lifeless> or you could create a test fake like that sketched on the SOA pages
<poolie> i think eventually separating the tests along the interface will be good
<poolie> for now i'm testing the whole thing because it's not very expensive and there is risk of integration breakage
<poolie> do you know how hard it would be to run up a realistic buildd vm locally and do manual integration testing?
<poolie> my impression is it's quite hard and unautomated
<lifeless> https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally
<wgrant> poolie: It's reasonably unautomated, but not exactly hard.
<wgrant> It's also well-documented.
<wgrant> Reasonably.
<wgrant> I think people other than me have done it.
<lifeless> ripe for improvement
<mwhudson> i've set up an arm buildd
<mwhudson> and used it with launchpad
<mwhudson> .dev
<mwhudson> it's fiddly, but not hard
<poolie> thanks
<wgrant> That page suggests installing it on your LP system, but that only really makes sense if your LP is running in a VM.
<wgrant> Because it really shouldn't be running on a real machine.
<poolie> i'm surprised that will work because it seems to have a hardcoded "if not under xen, abort" check
<wgrant> poolie: Ah, for recipe builds, probably not.
<wgrant> We have some thoroughly stupid checks.
<wgrant> That I couldn't be bothered complaining about.
<wgrant> (that one is new)
<flacoste> iirc, it was a request from IS
<flacoste> (the if not under xen, abort)
<wgrant> Yes, and it was not a very sensible request :/
<wgrant> Because recipe builds are the safe ones.
<wgrant> Well, not safe.
<wgrant> But safer than binary builds.
<wgrant> If it's not running under Xen, we have far larger problem.s
<flacoste> i have to admin, i forgot the rationale behind them asking this
<flacoste> admit
<poolie> i can easily change it to be on by default but disableable
<poolie> if that's a word
<wgrant> Hopefully lamont won't notice if we remove it.
<wgrant> It's only very very slightly beneficial to security.
<poolie> that actually seems like a nice place to use one of those TCB features
<poolie> perhaps they're not in the stock kernel
<poolie> but, limiting the machine to exec'ing only particular binaries
<poolie> or for that matter just having some noexec filesystems
<wgrant> poolie: Doesn't quite work for a buildd :)
<StevenK> sinzui: I can't use celebrity_logged_in -- I need to pass the principal into the view.
<poolie> wgrant, why?
<StevenK> sinzui: I already thought of using it when I wrote the branch.
<poolie> hm
<wgrant> poolie: Because they construct unsigned binaries...
<poolie> i guess you would need something that allowed execution inside the chroot but not outside
<poolie> perhaps possible though
<wgrant> And that's not useful.
<wgrant> Because chroots are not secure.
<poolie> so the point is to make sure arbitrary code does not run outside of the vm
<poolie> perhaps the host outside of the vm should be more locked
<wgrant> No.
<wgrant> It's to make sure untrusted code doesn't run on the non-virt builders.
<wgrant> But it's a failed attempt to do that.
<wgrant> Because binary builds still work, and they're far easier to exploit.
<lifeless> uhm
<lifeless> so I'm not endorsing the check
<lifeless> but failure to prevent attacks A *and* B doesn't make a solution to attack A valueless.
<wgrant> No.
<wgrant> But in this case it's really not very useful.
<wgrant> And it's actively harmful to development.
<wgrant> Not useful + harmful == kill
<poolie> i'll do something about it so i can test
<poolie> i can understand how they would want some protection against things running in the wrong place through operator error, even if it is not 100%
<wgrant> It's not even 10%.
<wgrant> lifeless: http://paste.ubuntu.com/732768/ is an attempt without the denormalisation (AccessPolicyArtifact.policy exists, and AccessPolicyGrant's policy and artifact columns are mutually exclusive).
<wgrant> lifeless: It's about 10% slower for Ubuntu on DF.
<wgrant> lifeless: But much slower for other projects.
<wgrant> (the third form of the query seems to be the fastest)
<wgrant> Because it does a seq scan on APG.
<wgrant> Refuses to use the indices.
<wgrant> Was wondering if you could poke a bit.
<lifeless> sure, I need to have an urgent poke at a few other things first, but after that I will
<lifeless> (running out of disk space on carob would be bad)
<StevenK> Sure, not seeing 2,000 OOPSes would be so tragic.
<StevenK> Won't someone think of the OOPSes
#launchpad-dev 2011-11-10
<lifeless> wgrant: storebuildinfo, really?
<lifeless> (bug 828017)
<_mup_> Bug #828017: UnparsableDependencies raised by retry-depwait script <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/828017 >
<lifeless> wgrant: for my education, what makes this a dup?>
<wgrant> lifeless: Check the referenced builds.
<wgrant> lifeless: They don't have a builder, or datefinished, or...
<lifeless> wgrant: are you saying that if they did, it wouldn't raise that oops ?
<StevenK> It would not, since the information would be set.
<wgrant> lifeless: The error is raised when the dependencies are unparseable, which means either lp-buildd is buggy and returning bad data, or LP isn't handling it properly.
<wgrant> In this case, storeBuildInfo isn't being committed.
<wgrant> None of the metadata is stored.
<lifeless> ok, so they don't have crap in their headers, for instance.
<wgrant> No.
<wgrant> The data is just None.
<huwshimi> OK, JS hackers, what's the Launchpad way of moving the JS in this template into its own file? http://bazaar.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/view/head:/lib/lp/bugs/templates/bugtarget-portlet-bugtags.pt
 * wgrant nominates wallyworld_
<huwshimi> I assume I can just move it into a file in lib/lp/bugs/javascript
 * wallyworld_ looks
<huwshimi> but I want to make sure it gets namespaced properly and uses whatever global stuff we have
<wallyworld_> huwshimi: much of the js for the bugtask page lives in bugtask_index.js but you may want to use a new namespace for this
<huwshimi> wallyworld_: What would you recommend?
<wallyworld_> huwshimi: simply have the on domready in the tales call the method you have moved to the new namespace
<wallyworld_> perhaps lp.bug.bugtask.tagcloud
<wallyworld_> but i'm not the best at making up names
<wallyworld_> it depends on if you think there may be more stuff to add which would warrant a new namespace
<wallyworld_> i think that lp.bugs.bugtask_index has too much stuff at the moment
<wallyworld_> might be good to have the tag cloud stuff separate
<wallyworld_> typo, should be lp.bugs.bugtask.tagcloud, or even lp.bugs.tagcloud
<huwshimi> wallyworld_: Where abouts is that tales domready?
<wallyworld_> huwshimi: the existing js embedded in the tales have a domready call
<wallyworld_> just replace the js in there with a call to the new namespaced method
<wallyworld_> huwshimi: do you see what i mean?
<huwshimi> wallyworld_: I know what you mean, I'm just not sure which file it is in
<wallyworld_> huwshimi: the one from your link
<wallyworld_> bugtarget-portlet-bugtasks.pt
<huwshimi> wallyworld_: Ah right.
<huwshimi> wallyworld_: Is there a way I can modify things so I can remove the javascript from that page completely?
<wallyworld_> huwshimi: the on domready callback has to go somewhere
<wallyworld_> there's an argument that it belongs with the template
<wallyworld_> but it could also go on the template for the index page which contains the tag cloud i guess
<wallyworld_> so if it is removed from the template for the tag cloud, it would need to be put separately wherever the tag cloud is used
<wallyworld_> i guess the tag cloud is only used in the one place?
<wallyworld_> on the bug task index page
<huwshimi> wallyworld_:  At some stage we will be moving to having no javascript in our templates
<wallyworld_> \o/
<huwshimi> wallyworld_: As I was editing this code already I thought I would try and move this out of the template now
<wallyworld_> huwshimi: so as far as i know, we don't yet have a well defined plan to deal with that
<huwshimi> wallyworld_: I guess really it needs to happen with a proper js loader
<wallyworld_> yep
<huwshimi> wallyworld_: maybe I'll leave this for now then
<huwshimi> wallyworld_: Well actually I'll move it into its own file, but leave the call
<wallyworld_> huwshimi: the best you can do, afaik, is to just have the on domready stub in the template and the core business logic elsewhere
<wallyworld_> for now
<huwshimi> wallyworld_: Yep, ok that will set it up for the future complete removal. Thanks :)
<wallyworld_> i've used this exact pattern where i need to be able to invoke the js in an xhr callback
<wallyworld_> as well as the page loaf
<lifeless> benji: still around? I'm going to guess not ...
<wallyworld_> huwshimi: np. good luck with it
<huwshimi> wallyworld_: Thanks
<lifeless> wallyworld_: hi; did you see my ping earlier ?
<lifeless> thumper: number of possible reviews - performance or visualisation ?
<wallyworld_> lifeless: ah yes, thanks. i tested it and yes it does do the right thing
<thumper> lifeless: yes
<thumper> lifeless: both
<lifeless> thumper: so consider https://code.launchpad.net/launchpad-project/+activereviews
<lifeless> thumper: or ubuntu/+activereviews
<lifeless> thumper: for a developer in either project, thats going to be a large fraction - maybe 80 or 90 percent - of what we'd be showing on their ~/+activereviews, simply due to the activity of the projects
<lifeless> thumper: or do you think it would be a much smaller fraction ?
<thumper> approved merges from mid 2009 is a bit sad
<lifeless> it is
<thumper> lifeless: I have no metrics
<thumper> so I can't make any real call on this
<lifeless> one way to get some is to implement it, flagged, and see
<thumper> sure
<thumper> at the time of development we didn't have feature flags
<lifeless> yeah, they make things easier :)
<thumper> and I chose the solution that was less likely to explode with timeouts
<thumper> but feel free to "fix" it
<lifeless> :) I might scratch at the itch
<lifeless> jml's request for -a- page is reasonable, the main thing I didn't understand is why that shouldn't be on the existing per user page
<lifeless> since the data volume is ~the same either way.
<StevenK> lifeless: And how do you determine a person is involved in the project?
<lifeless> StevenK: current heuristic of being a reviewer for the branch
<lifeless> StevenK: can iterate on that, obviously.
<lifeless> StevenK: As a dashboard for reviews, +activereviews is pretty good. The only issue at the moment, for a single dev, is having to visit one per project
<StevenK> Indeed.
<lifeless> consider that looking at 'requested reviews I can do' for all projects in LP is equivalent to just seeing those you are a reviewer for - because the list is empty when you aren't an oficial reviewer.
<huwshimi> wallyworld_: Do you mind doing a quick once over of what I did? https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
 * wallyworld_ looks
<wallyworld_> huwshimi: for testability, you want to avoid Y.io
<wallyworld_> you want: lp_client = new Y.lp.client.Launchpad();
<wallyworld_> lp_client.io_provider(xxx)
<wallyworld_> a bit pseudo ish
<wallyworld_> but the above allows a mock io provider to be used for tests
<huwshimi> wallyworld_: Ah, I didn't do that part, but this is a good opportunity to fix it
<wallyworld_> see existing stuff in bugtask_index.js or elsewhere
<huwshimi> wallyworld_: Thanks, I'll take a look
<wallyworld_> huwshimi: also, set('innerHTML') i think we are trying to avoid where possible, can't recall exactly, but setContext() may work
<wallyworld_> huwshimi: or new_node = Y.Node.create(xxxx); old_node.replace(new_node)
<wallyworld_> or something like that
<huwshimi> wallyworld_: Sure, I'll fix that up too
<wallyworld_> i know that the server may generate only legit html and all that, but it is a potential security hole
<wallyworld_> other than that, looks ok
<wallyworld_> huwshimi: in case you don't notice it, the lp_client construction is usually inside a setup type function so the client is only constructed once
<huwshimi> wallyworld_: Yeah ok thanks
<wallyworld_> good luck :-) there should be several examples to cargo cult. let me know if you are stuck
<huwshimi> wallyworld_: No problems, will do
<huwshimi> wallyworld_: I think it should be fine in the setup function I already have there.
<wallyworld_> the lp_client construction? yes, so long as setup is only called once which i guess it will be
<wallyworld_> you will want to make your setup function take a config dict as an argument so you can pass in the io_provider stub for tests
<huwshimi> wallyworld_: Oh ok, in that case would it be easier to have its own function?
<huwshimi> wallyworld_: Easier for testing that is
<huwshimi> wallyworld_: I guess it would be
<wallyworld_> huwshimi: perhaps.  getClient(config) or something
<wallyworld_> i think setup_lp_client(config) is also used, which assigns a module scoped lp_client variable
<wallyworld_> then the code can just use lp_client and assume it's been all set up
<wallyworld_> and the test would call setup_lp_client() with the correct config to pass in the io+provider
<wallyworld_> whereas the prod code would call with no confif
<huwshimi> wallyworld_: Should that setup function check to see if a lp_client already exists and if so do nothing?
<wallyworld_> huwshimi: yes
<huwshimi> wallyworld_: Ah good
<wallyworld_> so var lp_client = null; somewhere in the module, and then the setup function sets it
<huwshimi> wallyworld_: Yup
<huwshimi> wallyworld_: Well, "var client;" is enough right?
<huwshimi> erm
<huwshimi> var lp_client;
<wallyworld_> i prefer to be explicit, but your call
<huwshimi> wallyworld_: I don't mind. I'd prefer to be consistent
<wallyworld_> sure. i'm not sure what our coding standard says for that to be honest
<huwshimi> wallyworld_: let me check
<huwshimi> wallyworld_: Most of these files seem to just declare the variable name
<wallyworld_> sounds good
<huwshimi> wallyworld_: The docs don't seem to say though
<wallyworld_> as you say, just use what's gone before :-)
<huwshimi> wallyworld_: Ok, here's where I'm at: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
 * wallyworld_ looks
<wallyworld_> huwshimi: i have led you astray a bit i'm sorry. if all you are doing is making a straight Y.io call and don't need the lp_client extras, you just do:
<wallyworld_> var io_provider = Y.lp.client.get_configured_io_provider(config)
<wallyworld_> where config is what's passed into your setup
<wallyworld_> it is either undefined or a dict with an io_provider key
<wallyworld_> if you do need the full lp_client, you do lp_client = new Y.lp.client.Launchpad(config);
<wallyworld_> so in your current code, you are missing the config param for constructing the lp_client, but as per the above, you may not need that
<lifeless> \o/ no oopses on asuka since the 2nd (all via amqp)
<huwshimi> lifeless: Oh, that config parameter did exist, I lost it at some point
<huwshimi> lifeless: sorry that was meant of wallyworld_
<huwshimi> wallyworld_: Or maybe it only ever existed in the code in my brain
<wallyworld_> huwshimi: perhaps :-)
<lifeless> older for tellurium.
<wallyworld_> so if you want, you can just get yourself the io_provider
<wallyworld_> good news about the oopses
<huwshimi> wallyworld_: So this is how it should look? http://paste.ubuntu.com/733801/
 * wallyworld_ looks
<wallyworld_> huwshimi: that looks ok to me. your test setup then just calls setup_io_provider(). if io_provider is only used the once in setup_taglist, you may consider ditching the separate setup
<huwshimi> wallyworld_: yeah ok
<wallyworld_> so make setup_taglist take the config param
<huwshimi> wallyworld_: ok sure
<wallyworld_> huwshimi: sorry about the false start before, i'm used to needed the whole lp_client object
<huwshimi> wallyworld_: No, all good, I've learnt some things
<wallyworld_> excellent
<wallyworld_> the mock io provider we have is pretty good
<wallyworld_> lp.app.javascript.testing.mockio.js
<huwshimi> wallyworld_: Should I still have a check for if a io_provider already exists?
<huwshimi> wallyworld_: Or is that really only important for the lp_client
<wallyworld_> huwshimi: it doesn't really matter in this  case because it's just grabbing a value from a dict
<huwshimi> wallyworld_: ok sure
<wgrant> lifeless: bugsummary is pretty impossible to modify :/
<wgrant> lifeless: There's like 5 patches, and they weren't flattened into trusted.sql.
<wgrant> And they aren't in order :*(
<huwshimi> wallyworld_: Ok, hopefully this is what you meant :) https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
<lifeless> wgrant: sorry :) - twitch.
<lifeless> hey, anyone know if there is a good 'touched recently' field on question ?
<wgrant> I guess we need a new baseline schema soon anyway.
<wgrant> lifeless: you probably have to take the max of all the date fields :/
<lifeless> just query response I think will do
<lifeless> they seem to combine to cover it
<wgrant> They even cover answers?
<lifeless> last-response
<lifeless> last-response + last-query forms the rule for who needs to act next
<wgrant> Right, but what about a proposed answer?
<lifeless> arguably doesn't need to cover messages - they are scanned separately
<wgrant> Erm, what does it cover, then?
<wgrant> Is datecreated OK?
<lifeless> whiteboard, title description
<wgrant> Oh, whiteboard.
<wgrant> Blah.
<lifeless> so datecreated isn't enough
<lifeless> down to 5s query
<lifeless> big improvement
<lifeless> (for 1 week)
<lifeless> seq scanning question
<wallyworld_> huwshimi: i would reverse the config and io_config names, otherwise ok i think. the setup config can contain more than just io_provider. and io_config seems better suited for use in the io call
<huwshimi> wallyworld_: Ok, no problems
<lifeless> 500ms with two simple indices
<lifeless> OTOH I could just limit those to their containing scope.
<lifeless> probably easier I think.
<wgrant> lifeless: For a full week?
<lifeless> yes
<huwshimi> wallyworld_: OK, done, https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
 * wallyworld_ looks
<wgrant> jtv: For arch-parallel a-f, see shelf 21 on DF.
<wallyworld_> huwshimi: one more improvement  - use start and end callbacks in to show/hide spinner
<wallyworld_> start and end callbacks in io_config
<wallyworld_> that way you don't need to call hide in success handler
<wallyworld_> and again in failure callback
<wallyworld_> use something like "end: Y.bind(namespace.hideSpinner, namespace)"
<huwshimi> wallyworld_: What does the bind give me that just doing "end: namespace.hide_spinner" doesn't?
<wallyworld_> huwshimi: ah, sorry. i referenced an example where i had to pass a parameter to the hide api call, and i think bind is required then
<huwshimi> wallyworld_: Ah right
<huwshimi> np
<wallyworld_> the js tests will ensure that it all works in any case
<StevenK> I'd like to return count(archive.id) in a SELECT query, but only if it's > 1
<StevenK> Oh, wait, they will not disappear.
<wgrant> StevenK: HAVING
<StevenK> So I need to filter on state
<wgrant> SELECT owner, COUNT(*) FROM archive WHERE status = 0 GROUP BY owner HAVING COUNT(*) > 1
<StevenK> wgrant: Can I do select count(...) AS foo .... HAVING foo > 1 ?
<wgrant> Probably.
 * StevenK tries
<StevenK> Probably not, that is.
<lifeless> ok
<lifeless> wgrant: what did you want me to look at ?
<lifeless> now I have my query for oops pruning sorted, I'll look at your thing
<wgrant> lifeless: Thanks.
<wgrant> lifeless: So, http://paste.ubuntu.com/732768/ is the suggested undenormalisation.
<wgrant> lifeless: It's only ~10% slower for Ubuntu.
<wgrant> But 2.5x slower for ubuntuone-servers, because this ends up doing a seqscan on apg.
<wgrant> Not sure why.
<lifeless> I guess I need to regen sample data
<lifeless> ah itsi nthe bin
<wgrant> The paste contains 4 queries at the end.
<wgrant> The third is fastest.
<StevenK> Sample data is in the bin?
<wgrant> It's ~600ms for Ubuntu, vs 1200-1500ms for the other three.
<wgrant> But it still does a seqscan.
<wgrant> And is 120msish for ubuntuone-servers, rather than the old denormed 30-40ms version.
<lifeless> the old version was slow on ubuntu ?
<lifeless> or is this just data that the oldver was better overall
<lifeless> ?
<wgrant> Your version was just under 600ms for Ubuntu on DF.
<wgrant> The new one is a little over.
<lifeless> whats changed in the schema ?
<wgrant> APG.{artifact,policy} are mutually exclusive, and APA.policy exists.
<wgrant> So each artifact's policy is stored on APA, instead of many times in APG.
<lifeless> ah yes
<lifeless> so this is a normalisation
<lifeless> not a denorm :P
<wgrant> undenorm, I said.
<wgrant> It's not normalising, it's reverting the denormalisation :)
<lifeless> double negative parsing fail
<wgrant> Heh
<lifeless> ok, which query sucks again ?
<wgrant> The four at the end all do the same thing. The third is fastest.
<wgrant> One CTE and a couple of subselects.
<wgrant> Try against product 9514 to see the greater slowness.
<wgrant> But the plans are the same.
<wgrant> Er, the plans for distro 1 and product 9514, that is.
<lifeless> wgrant: plan 3, product 9514 - 114ms
<lifeless> plan 4 65ms
<wgrant> Bah, really?
<wgrant> Indeed, 30% faster.
<wgrant> But slower for Ubuntu.
<lifeless> http://paste.ubuntu.com/733853/
<lifeless> thats the plans you had
<wgrant> #3 there has a different context (ubuntuone-servers rather than ubuntu), but yes.
<lifeless> bah
<jtv> wgrant: do you mean that parallel a-f has been sitting around as uncompleted work?  That would be a serious waste.
<lifeless> wgrant: 410ms for #3 on ubuntu
<wgrant> jtv: Well, we don't really know how much of a win it is.
<lifeless> wgrant: do you want the plan ?
<wgrant> lifeless: That's 650ms on DF. #4 is 1.1s. So it is behaving somewhat differently to qastaging :/
<wgrant> lifeless: Not really.
<lifeless> wgrant: try raising your stats
<jtv> wgrant: thing is, it won't be at all hard to make the MF manage a sensibly-sized pool of processes.  So if 2 parallel processes is a win but 5 aren't, we can do 2.
<huwshimi> wallyworld_: I made the change to start/end and also changed the show show/hide to using classes instead of styles. As the show/hide are one liners do you think they should still be in separate functions?
<huwshimi> wallyworld_: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
<wgrant> lifeless: Mmm, the slow bits of the plan aren't really different
<lifeless> wgrant: those were all hot btw
<wgrant> The plan for #3 on 9514 is identical, and performance is almost the same. But it still hsa the seq scan, which seems unfortunate.
<wgrant> But if it's only 410ms for Ubuntu then I guess that's not too bad...
<lifeless> wgrant: 123557 rows
<lifeless> wgrant: 16ms to seq scan
<lifeless> wgrant: (just to read the rows once through, no processing)
<wgrant> lifeless: How'd you measure that? SELECT *?
<lifeless> wgrant: yes
<lifeless> 59785 rows are ubuntu
<lifeless> wgrant: thats 50% - seqscan is always going to be fastest ;)
<wgrant> That takes 160ms on DF.
<wgrant> Hmm.
<wgrant> But that can't be right, so maybe not.
<wgrant> lifeless: Yeah.
<lifeless> http://paste.ubuntu.com/733866/
<wgrant> But U1 and other products will be smaller.
<lifeless> sure
<lifeless> 9514 was the 'slow' one right ?
<wgrant> 100ms instead of 40ms, yes.
<wgrant> 10294 is LP, which may be interesting.
<lifeless> 4K grants for 10294 and 9514 each
<lifeless> sorry, 60K for ubuntu, not 600K
<lifeless> my eyes, me eyes, my eyes
<lifeless> wgrant: still 5% of a table with narrow rows will hit ~ all pages unless its clustered on the selector
<wgrant> True.
<wgrant> It's also only a 9MB table and should be nice and hot.
<lifeless> wgrant: you can defeat the seqscan by unioning
<lifeless> if you want
<wgrant> Yeah, but that's really slow for Ubuntu, at least on DF.
<StevenK> distroseries-edit-datereleased => devel  [OK]     (up for 4:39:27.158866)
<StevenK> Oh, come on, ec2.
<lifeless> 800ms here
<lifeless> wgrant: 180ms in
<lifeless>                                  ->  Index Scan using accesspolicygrant__artifact__person__key on accesspolicygrant  (cost=0.00..7.14 rows=2 width=8) (actual time=0.005..0.006 rows=2 loops=33978)
<lifeless>                                        Index Cond: (pg_temp_43.accesspolicygrant.artifact = accesspolicyartifact.id)
<lifeless> rm, something like that
<StevenK>         # XXX cprov 2007-02-22 bug=87098:
<StevenK>         # Since we only allow one PPA for each user,
<StevenK> Ha. Hahaha. Hahahahahahaahahahahahaha
<poolie> hm
<poolie> so the buildd package can be installed pretty easily as a dpkg on developer systems
<wgrant> lifeless: OK, 800ms is better than I get, but still 100% slower than #3
<poolie> at the moment it automatically starts a buildd
<poolie> i suspect this is not very safe or desirable
<wgrant> poolie: It's a nice easy remote root vulnerability, right.
<poolie> i'm inclined to handle it by splitting off an "actually run it" package
<poolie> which can be installed on the real buildds
<poolie> with that still called launchpad-buildd
<poolie> and then eg launchpad-buildd-bin or python-lpbuildd
<poolie> or maybe both
<poolie> that has all the stuff that will be developer dependencies
<wgrant> Mmm.
<wgrant> I'd prefer not to have launchpad-buildd in any readily accessible apt repo.
<poolie> :)
<poolie> someone will run it
<wgrant> Yes.
<poolie> well, i could take this course and just not upload it
<wgrant> Huh.
<wgrant> Speaking of remote root vulnerabilities.
<wgrant> lolpython
<poolie> the other way these things tend to be handled is to have the daemon off by default and an 'are you really sure' in /etc/default/something
<poolie> ?
<wgrant> yaml.load isn't safe. It loads pickles.
<wgrant> What a sensible default.
<poolie> seriously?
<wgrant> Yeah.
<wgrant> You have to use yaml.safe_load for that.
<poolie> as a kind of dwim if you feed it a pickle?
<wgrant> Not sure. I guess they define a way for YAML to contain pickles.
<jtv> Say wgrant, how do I dput a package onto a PPA on dogfood?
<StevenK> wallyworld_: What happened to TPG?
<wallyworld_> ?
<poolie> wgrant, not pickles as such as far as i can see
<poolie> but you can call python objects from inside it
<StevenK> wallyworld_:  "rev.aanet.com.au"
<wgrant> Yeah, not actual pickles.
<wgrant> Its own implementation.
<wgrant> But still.
<poolie> it's similar to doing eval
<wgrant> Who could possibly think that was a reasonable idea.
<wallyworld_> StevenK: oh, i've relocated today to a different place, change of scenery
<poolie> anyhow
<poolie> > off by default and an 'are you really sure' in /etc/default/something
<poolie> the main drawback is
<wgrant> jtv: [df]
<wgrant> method = ftp
<wgrant> fqdn = upload.dogfood.launchpad.net:21
<wgrant> incoming = %(df)s
<wgrant> login = anonymous
<poolie> losas need to turn it on i guess
<wgrant> jtv: dput df:ubuntu something_source.changes
<poolie> and, some people will turn it on and get in to trouble
<wgrant> poolie: And it breaks the automated installation.
<lifeless> wgrant: plan three is 640ms
<StevenK> wgrant: That's to ubuntu, not a PPA
<jtv> wgrant: so no mention of the PPA at all?
<wgrant> lifeless: How fast was your query?
<lifeless> wgrant: uhm, variable. just got a 450
<poolie> so...
<wgrant> jtv: Ah, for a PPA. Use a PPA path.
<lifeless> wgrant: lots of noise
<wgrant> lifeless: Er, your query from a couple of days ago, that is.
<wgrant> I can't remember.
<wgrant> Was 300ms or something.
<lifeless> oh right
<lifeless> but that was old schema wasn't it ?
<wgrant> It was.
<jtv> wgrant: is a PPA path something like âppa:user/ppaâ, without changes for dogfood?
<wgrant> jtv: The FTP path is ~user/ppa/distribution
<wgrant> So df:~user/ppa/distribution
<jtv> Oh
<jtv> Thanks
<wgrant> lifeless: But I would like to get a real performance comparison between the two schemas.
<wgrant> lifeless: The queries in the denormed one are already slow.
<wgrant> StevenK: Erm.
<StevenK> wgrant: Hm?
<wgrant> Ah, I see.
<wgrant> Just somewhat bemused at the misspelt instance variables your rev added.
<wgrant> but there was actually a reason for it, and the misspelling wasn't yours.
<StevenK> Oh?
<jtv> stub: any luck with Q/A for bug 826653?
<_mup_> Bug #826653: preflight check could report patches to be applied <fastdowntime-later> <qa-needstesting> <Launchpad itself:Fix Committed by stub> < https://launchpad.net/bugs/826653 >
<stub> Needed to wait for a staging update... lets have a look
<wgrant> StevenK: "derivative" is not spelt "derivitive", and it's sort of odd to set the instance variables in setUpFields.
<stub> Probably should avoid a rollout today since we are doing the slony upgrade btw.
<wgrant> Do we have a GSA scheduled to do the package swap? :)
<jtv> stub: that's a shameâwe could roll out all 14 available revisions once your Q/A on this is done.
<stub> qa-ok
<stub> wgrant: I thankfully don't have to juggle those details now ;)
<wgrant> stub: Oh?
<jtv> thanks stub
<stub> Other people schedule the time
<wgrant> Well, as long as someone knew a GSA was required...
<StevenK> stub: What should I do with https://code.launchpad.net/~stevenk/launchpad/db-packageupload-archive-index/+merge/79903 ?
<poolie> biab
<wgrant> lifeless: So, do you have a feeling as to which one is better? Your numbers are probably better than mine. Is the denorm worth it?
<stub> StevenK: Land it?
<StevenK> stub: Last time I spoke to you, you wanted me to hold off a few days. It's fine to land and have the indexes dropped?
<wgrant> st[a-z]+/i: Doesn't that drop the index, which we were going to do once we determined it was unused?
<StevenK> Hahaa
<stub> StevenK: I'll look
<stub> I recall now
<lifeless> wgrant: still fiddling
<lifeless> wgrant: the full is trivial - 4ms
<wgrant> lifeless: Ah, k
<StevenK> stub: And sorry for dropping the ball :-(
<wgrant> lifeless: Huh?
<lifeless> wgrant: seeing if I can get a better restricted result
<wgrant> Oh.
<wgrant> Right.
<wgrant> Yes, there aren't many full observers.
<lifeless> wgrant: so I'm fiddling without the full case
<wgrant> It's possibly beneficial to split the table, but let's see what you find.
<lifeless> wgrant: denorming the sort key onto apg might be good
<lifeless> if you want to make it fly
<wgrant> lifeless: It would be.
<wgrant> But, uh, let's not go there...
<wgrant> yet, at least.
<stub> So packageupload.signing_key is never used. But probably should leave that.
<wgrant> stub: The index?
<wgrant> Hmmm.
<wgrant> Odd.
<wgrant> It should be.
<stub> package_copy_job_idx is not used much
<stub> yes
<wgrant> package_copy_job_idx?
<wgrant> What's that?
<wgrant> It doesn't match our usual format, and I can't see it on packagecopyjob
<stub> id_distroseries_archive is good to drop - it is unused
<wgrant> Oh.
<wgrant> On PackageUpload. i see.
<StevenK> stub: There are not many PCJs floating around
<StevenK> wgrant and I have a plan to make more
<stub> distroseries_status_idx is still used, but I suspect those uses will switch to the other index if it is dropped
<wgrant> id_distroseries__archive seems like a completely bad idea.
<lifeless> wgrant: denorming the sort key onto grant is insufficient
<lifeless> wgrant: these are all under 1s
<lifeless> wgrant: and all doing full set ops, so sustainable
<stub> http://paste.ubuntu.com/733885/ is before, http://paste.ubuntu.com/733886/ is after if anyone wants to see the raw stats
<lifeless> wgrant: my recommendation is this:
<stub> Timeframe was around when I last discussed this with StevenK :-)
<StevenK> packageupload__signing_key__idx
<lifeless>  - A) go with this normalised schema
<StevenK> That's totally unused, should we just drop it?
<lifeless>  - B) if its too slow create a fact table from it (like bugsummary) that can answer in a few ms.
<wgrant> Sounds good.
<wgrant> Thanks.
<stub> StevenK: If inserts or updates are not too slow, leave the signingkey index. It is good to always have indexes on foreign key references to avoid surprises.
<StevenK> stub: Fair enough, if you're happy dropping the two indices, let's land it, then
<stub> If there are still queries looking for distroseries without an archive, are they broken?
<StevenK> Yes
<wgrant> Hmm.
<StevenK> Since they'll conflate PPAs and the primary archive
<wgrant> Except possibly Distribution:+ppas.
<wgrant> Let's see what queries that does.
<StevenK> stub: I thought we seperated columns by __ in indicies, so I'm surprised by id_distroseries...
<stub> We still got 70k queries hitting the distro__status index. If we drop it, the alternatives are the archive__distro__status index and the distro index. Neither would be good if there is a valid use for queriying all archives for a distro,status
<wgrant> StevenK: It's a typo.
<stub> StevenK: Old naming conventions
<StevenK> Ah
<stub> and that would be a typo anyway
<StevenK> Heh
<stub> Possibly that is there to support a foreign key reference? Or maybe a failed attempt to optimize a query.
<stub> c/is/was/
<StevenK> Conjecture, hearsay ... :-)
<StevenK> But it looks like that index has been there for a long long while
<wgrant> No.
<wgrant> It's new.
<wgrant> IIRC lifeless added it.
<wgrant> To optimise DistroSeries:+queue for NEW/ACCEPTED.
<wgrant> I don't see how it could possibly work, though :)
<StevenK> Haha
<lifeless> which one?
<wgrant>     "packageupload__id_distroseries__archive__idx" btree (id, distroseries, archive) WHERE status = ANY (ARRAY[0, 1])
<wgrant> Primary key first in the index... not entirely useful.
<wgrant> I guess that packageupload(status) would do the same thing.
<wgrant> -- Partial index for queue processing view : the lopsided data makes this
<wgrant> -- necessary (millions of rows matching the archive, 100's of rows match the
<lifeless> made by me, landed by you
<wgrant> -- query).
<wgrant> Really?
<wgrant> No.
<lifeless> check annotate :P
<StevenK> But it's +queue, it's for a tiny amount of archives
<wgrant>         committer: Robert Collins <robert@canonical.com>
<wgrant>           Create a partial index to support queue processing queries more efficiently.
<lifeless> you put the db revno in it
<wgrant>         committer: William Grant <william.grant@canonical.com>
<wgrant>         branch nick: fix-patch-57
<wgrant>           Live-deployed patches need a non-zero patch number. Production is already fixed.
<lifeless> ah
<lifeless> ok
<lifeless> anyhow, if its not used, remove it; it would have been added based on the queries seen at the time
<wgrant> StevenK: Yeah, but as the comment says it's very lopsided.
<wgrant> lifeless: It probably is still useful.
<wgrant> But it's excessive.
<lifeless> note that a sort by id can trigger that index.
<stub> It hasn't been hit once in a few weeks. We don't need it.
<StevenK> stub: So, that's a +1 to toss that branch at db-devel?
<stub> I'm interested in if we really should drop the distroseries__status__idx index.
<wgrant> Distribution:+ppas uses SPPH, not PU.
<wgrant> It would be nice if we could find out what was using that index.
<stub> So it seems that if there are still queries not using archive as a filter on packageupload, they are broken? That would also explain why the id_distroseries index is never hit.
<stub> ignore that
<stub> So it seems that if there are still queries not using archive as a filter on packageupload, they are broken?
<wgrant> stub: Well, or the data is sufficiently skewed (archive 1 still has more than 50% of the table, probably) that the planner decides using archive is not useful.
<stub> It would still use the archive__distroseries__status index I think
<lifeless> what I've seen happen a couple of times is that transforming the query very slightly will cause indices not to be used
<wgrant> Very slightly :(
<lifeless> so it may just be a minor change during derived distros
<lifeless> stopped it being used
<wgrant> There have been no changes of that sort, really.
<wgrant> But maybe.
<stub> If inserts and updates are not causing us grief, I'm happy enough to leave the index in there. PG seems to still like using it for whatever reason.
<StevenK> Right, so just id_distroseries dies
<stub> Yup
<StevenK> stub: That's landed. r11140 on db-devel
<poolie> so if i want to get /usr/share/launchpad-buildd onto the pythonpath when launchpad runs, what would be a clean way to do that....?
<poolie> maybe just in the makefile
<lifeless> poolie: buildout
<poolie> buildout.cfg
<poolie> yeah got it
<lifeless> poolie: I'm curious why you say this isn't really python
<poolie> it contains perl :-P
<wgrant> It's not a Python package.
<poolie> and the package as a whole just seems oriented towards being a dpkg not an egg
<wgrant> It's an application.
<wgrant> Which happens to use an internal Python package.
<poolie> obviously python things can contain arbitrary stuff
<poolie> right
<lifeless> so, why do you want it on the python path then ?
<poolie> i want the python stuff on the path
<wgrant> poolie: I would just put it in sourcecode/ for now.
<wgrant> Rather than disabling it by default and not mangling /etc/sudoers and etc.
<poolie> i could split it into one part distributed as an egg and one part as a dpkg
<poolie> wgrant, i think i'm going to go for two dpkgs
<wgrant> :/
<poolie> why?
<poolie> lifeless semi-vetoed sourcecode
<wgrant> And I am mostly-vetoing dpkg.
<lifeless> why?
<wgrant> It's not a package that should be available to anyone.
<wgrant> It's terribly dangerous. It is badly behaved.
<lifeless> what should we use instead?
<wgrant> sourcecode.
<wgrant> I suspect.
<wgrant> Until we have a proper fake version.
<poolie> are my changes making it any more dangerous?
<poolie> people could run it today and get into trouble
<wgrant> You're proposing putting the package into a public archive and encouraging its installation.
<poolie> i'm happy to keep the part that actually starts the daemon out of a public archive
<wgrant> How?
<poolie> not uploading it?
<poolie> uploading a poisoned highly-versioned package that does nothing?
<wgrant> Having the source package sometimes build the evil package and sometimes not sounds like a recipe for trouble.
<wgrant> This is presumably only a temporary measure.
<wgrant> LP shouldn't need the buildd tree.
<poolie> perhaps we can detect if the package is running on a 'real' buildd and abort otherwise?
<poolie> wgrant, i'm trying to make lp depend only on the parts it actually needs to test
<poolie> which don't include anything running as root
<poolie> i guess we could do a further split of the 'actually really run it as root' scripts into a separate source tree
<wgrant> lifeless: What is the policy on test fakes these days?
<lifeless> requires for new microservices
<lifeless> this is in a grey area
<lifeless> I would vastly prefer one
<lifeless> but split out without one is better than not split out
<poolie> ...
<lifeless> and poolie has a record of continuing to tug on things
<poolie> so, meaning what exactly?
<poolie> launchpad tests do not need the actual buildd implementation at all?
<lifeless> ELOCAL, bbs
<wgrant> poolie: They shouldn't.
<poolie> i would prefer that
<wgrant> poolie: Why would they?
<wgrant> They need something that looks like it.
<wgrant> They clearly don't need the actual thing, since it can only run as root.
<poolie> there are a couple of answers
<wgrant> We care that it speaks the protocol correctly.
<poolie> one is to reduce risk while splitting things i want to keep as much test coverage as i can at least during the transition
<poolie> because there are no good automated overall integration tests
<wgrant> Right.
<poolie> so...
<poolie> i guess i could look at just redoing those tests now
<wgrant> I suggest that the cheapest thing to do right now, until we have a strategy to move forward, is to use sourcecode. I don't like it either, but I think it's the best thing for now.
<poolie> your main concern being that people will blithely install the buildd package and kill themselves
<wgrant> And LP doesn't really like using packages for Pythonish deps.
<wgrant> And lp-buildd doesn't make sense as an egg.
<wgrant> And lp-buildd deployment is fragile and obscure enough that I'd really really not like to change it right now by splitting the packages up.
<poolie> perhaps we should just look at the tests that do depend on it
<poolie> and see if they can easily be changed right now
<wgrant> Possibly, but we'd probably need to rewrite significant volumes of tests on both sides.
<wgrant> Worth a try, though,
<wgrant> jtv's been in this area lately.
<wgrant> Hi jtv.
<jtv> I deny everything.
<poolie> so the connections are
<poolie> BuilddSlaveTestSetup (a fixture)
<wgrant> jtv: You've been dealing with buildd integration and tests thereof lately, right?
<poolie> some things in translations that use generate_translations_templates and also write to the lp db
<jtv> One particular aspect.
<poolie> and the codehost
<poolie> the utility 'pottery-generate-intltool'
<poolie> and that's about it
<poolie> so the buildmaster things could perhaps be satisfied with a fake
<poolie> the translations tests really look like integration tests to do with how code inside the buildd runs
<jtv> Everything related to buildd looks like integration tests.  :/
<poolie> i guess they need to be split into tests that cover the two respective sides of the handoff
<jtv> The buildd infrastructure isn't easily split into âsidesâ of communication, unfortunately.
<jtv> (Also, there are at least three sides)
<wgrant> jtv: Master and slave?
<wgrant> What's the third side?
<jtv> Librarian.
<poolie> and codehosting too i think
<wgrant> Mm, slightly.
<lifeless> ideally three sets - in-lp, in-buildd, and a smattering of end to end tests run on jenkins [note that we don't have these today, because we don't run the LP test suite as root...
<wgrant> But the librarian and codehosting deps from the buildd are very weak and easily fakeable.
<lifeless> the more network fakes we build the easier it gets
<lifeless> not to be trite
<poolie> yeah
<jtv> But we have same very nasty issues to deal with there, such as âwhat if interopation with another builder aborts changes done in the context of your builder while you wait for the librarian?â  This is what I' trying to deal with at the moment.
<poolie> i just don't want one ginormo commit/deploy, since it will be hard to debug
<lifeless> jtv: that doesn't depend on the builder implementation nor the librarian implementation in any way that a fake can't simply simulate
<jtv> IMHO we need loads of unit test coverage built first, followed by a streamlining of the integration tests.
<jtv> Another problem compounding the situation though is that the whole buildmaster architecture urgently needs to be cleaned up.
<poolie> well
<wgrant> poolie: Right, I want small steps.
<wgrant> The easiest way to get the split done now is to use sourcecode. Then we can start severing connections, writing fakes, etc. And eventually drop the dep.
<wgrant> Rather than breaking deployments by reworking the package into a form that is usable in an ugly way until we have fakes.
<poolie> can you explain why using a deb will be breaking deployments, ugly, etc
<wgrant> Well, nobody knows how lp-buildd deployments work now.
<wgrant> The package mangles sudoers itself in ways that interact unobviously with puppet.
<wgrant> etc.
<wgrant> I don't like our chances of changing the package without breaking deployments in various ways.
<lifeless> another sequence you could use
<lifeless> is to write fakes in-lace
<lifeless> get it all copacetic
<lifeless> then remove the code
<poolie> fwiw i just now have lp passing its tests using the deb version of lpbuildd
<lifeless> that said, we have qastaging / staging to test deploys on
<poolie> what is in-lace?
<lifeless> in-place
<wgrant> lifeless: Hahahaha
<poolie> in which place?
<lifeless> so I'm not scared of undetected breakage hitting production
<wgrant> We saw how well staging testing worked last time...
<wgrant> poolie: In the LP tree.
<lifeless> wgrant: once we used it properly it seemed to work ok
<lifeless> wgrant: or did I miss something?
<wgrant> lifeless: Do we know what went wrong?
<wgrant> We broke prod last week.
<wgrant> Because staging QA didn't.
<lifeless> yes, AIUI we didn't actually end-to-end things the first time around
<lifeless> IMBW
<wgrant> We did.
<wgrant> But the builder was unbroken.
<lifeless> what was broken then ?
<wgrant> Quite how it was unbroken I don't think we know.
<poolie> lifeless, so you're saying, keep lp's own copy of the code
<poolie> and gradually rewrite the tests to not need a builder process?
<poolie> we could
<lifeless> poolie: well, we want a builder process
<lifeless> poolie: but that should be a network fake rather than a live instance
<lifeless> poolie: https://dev.launchpad.net/ArchitectureGuide/ServicesRequirements#Test_fake
<poolie> riht
<poolie> hm
<lifeless> I'm saying that separating the trees (into lp and a live buildd with included test fake tree) could be the last step, rather than the first.
<poolie> aside from the coupling to the deployment environment, i don't think testing against the actual buildd code is all that bad
<poolie> right
<poolie> however, i have actually done it in the other order
<poolie> so i'd kind of rather not abandon it unless there's actually something unavoidably bad
<poolie> it is meant to be a cleanup so i do want it to be something we're happy is at least incrementally cleaner
<lifeless> I'm fine with either order; I don't see a package in the LP PPA as terrible, particularly if you're going to keep poking at it - in fact, by getting closer to how we deploy (e.g. if we hint to lamont we want the same package and get his support), we may reduce friction
<poolie> i hope so
<lifeless> poolie: I only offer up the other order for consideration
<poolie> what do you mean by 'same package'?
<poolie> same binary, without a cat rebuild?
<lifeless> same build process
<lifeless> cat would rebuild, always does
<lifeless> but no hidden magic
<poolie> afaik that is what we did in the last deploy
<poolie> or perhaps it's just sufficiently hidden magic :)
<poolie> woo
<poolie> everything likely to have broken now seems to be passing
<poolie> so wgrant can i make you tolerably happy if i use the dpkg dependency but
<poolie> - split off the 'kill me now' bits into a separate package which is not in ppa:launchpad
<lifeless> man, TestOopsPrune is a littttle braindead
<poolie> - file bugs against the tests that depend on importing from it
<poolie> - perhaps try to rig the init script so it will start without disruption on the buildds but not on anyone else's machine
<wgrant> Perhaps.
<lifeless> it writes the same oops into recent and old directories, and then tests that its only deleted from some.
<lifeless> win
<lifeless> .
<wgrant> Hah
<poolie> wgrant, so just to be clear, the two things are
<poolie> - it will be horrible if people start offering buildds on untrusted networks
<poolie> - generally we don't like python in debs-
<poolie> (but we maybe like sourcecode less, depending who you ask)
<poolie> is that correct?
<poolie> well, https://code.launchpad.net/~mbp/launchpad/800295-delete-buildd/+merge/81815
<lifeless> wgrant: StevenK: jtv: whats a good mixin interface that product and distribution already inherit, to add a referenced_oopses(start, end) method to
<lifeless> 'There is None' is a valid if sad answer
<poolie> ok i'd better go
<poolie> may be back later
<lifeless> wgrant: we were talking about the API the other day
<lifeless> wgrant: what decorators will I need for findReferencedOOPS(start_date, end_date), do you think ?
<lifeless> wgrant: operation_parameters, export_read_operation, I presume
<wgrant> lifeless: Yep.
<wgrant> And operation_for_version
<wgrant> But that should be it.
<wgrant> Just leave the return type undeclared.
<lifeless> thanks
<lifeless> now, I'd like a sanity check on something
<lifeless> I plan to ignore security and scan everything, on a public anonymous API
<lifeless> but return *only* the OOPS strings themselves.
<lifeless> I'm assessing our regexps as tight enough to not disclose anything
<lifeless> Am I wrong?
<wgrant> Well.
<wgrant> That's a good question.
<wgrant> I don't currently like the way we permit whitespace.
<wgrant> But at most it can reveal one word.
<lifeless> do we actually linebreak OOPS-bar ?
<lifeless> perhaps we should make that not break, and tighten the refex
<wgrant> That's what I was thinking.
<lifeless> or
<lifeless> perhaps we should tighten the regex and file a low bug about linebreaking that
<wgrant> Even better.
<lifeless> if someone writes oops-tradesecrethere in a bug, we may disclose something; I think thats a fairly low risk
<stub> oops-i-did-it-again
<lifeless> will pick up'i' :P
<stub> Today Thailand celebrates Loy Kratong to pay respect to the spirit of the waters.
<wgrant> Heh
<nigelb> Irony right there.
<nigelb> stub: You're in a safe area I hope?
<stub> I'll be last to go
<nigelb> :)
<lifeless> jtv: where were you putting your 'object implements interface' tests ?
<lifeless> jtv: I'm adding a mixin interface to product and distribution; I went looking in lib/lp/registry/tests/test_product.py first
<lifeless> jtv: ah, I found it
<lifeless> jtv: I was searching for verifyObject :)
<mrevell> Hello
<adeuring> good morning
<StevenK> And AWS add a second US West region.
<StevenK> No, this isn't confusing AT ALL.
<nigelb> StevenK: Actually, there are 4 US locations :)
<nigelb> Virginia, Portland (new), California, and a Govt location which is only for US folks.
<nigelb> Morning bigjools!
<mrevell> ha
<nigelb> Hrm, did I scare him away?
<nigelb> :D
<allenap> Good morning everyone. Who knows about PPPAs?
<wgrant> allenap: What's up?
<allenap> wgrant: I can see a PPPA, and so can an esteemed colleague of mine, but get 401 when adding the suggested entries to sources.list. I noticed there's no user auth garbage in the URLs it suggests.
<wgrant> allenap: Those pages aren't meant to be used.
<wgrant> They were invisible until someone did something 18 months ago.
<wgrant> Use https://launchpad.net/~/+archivesubscriptions
<wgrant> instead
<allenap> wgrant: Ah ha, okay. So the PPPA needs to be granted to the user.
<wgrant> The URLs will probably magically become sensible once you click the "View" link on +archivesubscriptions.
<wgrant> That generates your unique access token.
<wgrant> Hah
<wgrant> And the CSS is broken, so you can actually see which of the View links are normal, and which are magical.
<allenap> wgrant: Thanks, that's great.
<nigelb> wgrant: Magical?
<wgrant> nigelb: JavaScript which requests an access token.
<nigelb> ah.
<nigelb> I don't see any of those. but then I only have one PPA listed there.
<jml> is it true that LP has instant oops availability?
<wgrant> jml: On staging and qastaging, but not on production yet.
<wgrant> We don't have the rabbitmq metrics completely done yet.
<jml> wgrant: ah right, thanks.
<wgrant> It all works fine, but we're not putting rabbit into production until we have good monitoring and such.
<jml> +1
<jml> is there a live feed of oopses too?
<wgrant> Not as yet.
<jml> or do you have to know the code before you can get to them?
<wgrant> But oops-tools is hackable now, so I expect we'll see some nice stuff soon.
<wgrant> And it's all split out into non-LP-specific projects, so others can use it if they so desire.
<wgrant> lifeless has done amazing work to that end.
<jml> yeah, that's why I'm asking, as it happens.
<lifeless> there is an index on date I think
<lifeless> so we can do an API to get latest N
<lifeless> and expose that in LP itself
<lifeless> or add it to the oops-tools console
<lifeless> I've also just got the OK from jane to LGPL the plumbing
<wgrant> Ah, excellent.
<wgrant> That'll make things a bit more sensible.
<lifeless> jml: lp:python-oops-tools if you want to go looking
<jml> lifeless: thanks
<lifeless> I am considering adding an explicit oops-reference table to LP
<lifeless> to let us have references into the separate oops-tools service
<lifeless> this might make the crashdump project better
<stub> lifeless: Any reason to have the oops-reference table in lp rather than in a database controlled by the oops service?
<adeuring> allenap: are you doing reviews today?
<lifeless> stub: yes, because LP knows what things hold references
<lifeless> stub: both ends may want to know actually; e.g. oops-tools knows 'referenced by url, url, url, ...' and lp knows 'bug X references oops, oops, oop', question Y references oop, oops, oops etc
<lifeless> stub: this is a new thought still getting assembled though; ask me tomorrow I may have a totally different idea
<allenap> adeuring: I am.
* allenap changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 275
<allenap> adeuring: Though I probably have enough on my plate already :-/
<allenap> adeuring: I am out much of the afternoon, working late this evening instead.
<adeuring> allenap: ah, ok. If you have some spare time nevertheless, could you have a llook at this mp: https://code.launchpad.net/~adeuring/launchpad/banner-for-beta-features-2/+merge/81833 ?
<allenap> adeuring: Sure :)
<stub> lifeless: LP doesn't really know, it holds a heap of information that needs to be scanned and processed. The way we do it at the moment is not scalable. If we picked up references at insert time we no longer need to scan. But where to store those references? Just as easy to send a message to rabbit as it is to store the reference in a db table.
<stub> lifeless: And that way the service that stores the OOPSes doesn't need to reach into the Launchpad database to determine what can stay or what can go.
* jcsackett changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugtasks: 275
<rvba> Could you guys tell me how to create a "mirrored" branch? I'd like to answer a question in #lp but I can't find how to change a branch type myself (https://help.launchpad.net/Code/MirroredBranches)
<rvba> jcsackett: I'm pinging you since you've OCR ;). Do you have any idea? how to created a mirrored branch? (See my question above). The doc in https://help.launchpad.net/Code/MirroredBranches really seems outdatedâ¦
<rvba> s/you've/you're/
<rvba> s/created/create/
<rvba> phew
<nigelb> rvba: on the code.launchpad.net/project page click on "Import a Branch" I think.
<nigelb> where project is whichever project it needs to be associated with.
<rvba> nigelb: but what about if you want to have a branch in ~me/+junk/my_mirrored_branch ?
<nigelb> rvba: I think mirrors aer associated with project not +junk. *checks*
<rvba> That's not what https://help.launchpad.net/Code/MirroredBranches says but this might be outdatedâ¦
<nigelb> Oh. Hrm. Interesting
<nigelb> Imported branches don't have a Mirrored status anyway.
<rvba> I need a code specialistâ¦ abentley maybe? Would you know the answer to my question above?
<abentley> rvba: I've never needed to, but...
<rvba> abentley: I suppose the right way to do that is to properly create a project and have the branch mirrored in that projectâ¦
<rvba> But that's not what https://help.launchpad.net/Code/MirroredBranches says.
<abentley> rvba: It looks like you go to the pillar page and click "Import a branch".
<rvba> abentley: I'm sorry but what is the pillar page (you have to know that I've been trapped inside Soyuz for a few months, I just escaped very recently).
<abentley> rvba: https://code.launchpad.net/launchpad/ for example
<nigelb> Trapped inside soyuz. Oh dear.
<rvba> abentley: ok, so no mirrored branch in /~me/+junk then?
<abentley> rvba: And probably anywhere else branches are listed.
<abentley> rvba: You might want to ask someone on the Bazaar team.  They're the last ones who touched this functionality.
<rvba> abentley: ok I'll do that, thanks for the help.
<deryck> jcsackett, are you reviewing adeuring's js branch?
<jcsackett> deryck: yeah, i was just looking through it.
<deryck> jcsackett, ok, cool.  Thanks.
<jcsackett> was about to tell adeuring i had left some questions on it.
<jcsackett> it seems a bit redundant now, but. adeuring, i left some comments on your MP. :-)
<deryck> heh
<adeuring> jcsackett: I deleted the redundant stuff (and thanks for the review)
<jcsackett> thanks, adeuring. r=me.
<adeuring> jcsackett: thanks!
<mrevell> danhg, We should aim to talk about bug 888596 before the end of the day; if not, tomorrow.
<_mup_> Bug #888596: Instructions for generating gpg keys are not provided for Unity <Launchpad itself:Triaged by danhg> < https://launchpad.net/bugs/888596 >
<danhg> I'm looking over it now
<abentley> allenap, jcsackett: could you please review https://code.launchpad.net/~abentley/launchpad/back-button-support/+merge/81875 ?
<jcsackett> abentley: looking now.
<jcsackett> abentley: r=me. looks fine.
<abentley> jcsackett: thanks.
<sinzui> jcsackett, do you have time to mumble?
<mrevell> Night all
<abentley> deryck[lunch]: I'm up for our chat anytime.
<deryck> abentley, ok, cool.  Give me just a bit more and I'll ping you.
<jcsackett> sinzui: sure. one moment.
<deryck> abentley, I can chat now, if you're still free.
<abentley> deryck: let's go!
<jcsackett> bac: r=me on your lplib MP.
<bac> just saw that, thanks jcsackett
<jcsackett> sinzui: i have to run out for a few moments; i'll ping you to mumble when i return.
<jcsackett> sinzui: i am again free to mumble whenever you are.
<deryck> jcsackett, are you reviewing still?
<jcsackett> deryck: i am. have a branch for me?
<deryck> jcsackett, indeed! https://code.launchpad.net/~deryck/launchpad/toggle-bug-fields-config-widget/+merge/81900
<deryck> jcsackett, more of my widgets for new bug listings work.
<jcsackett> deryck: looking at it now.
<deryck> jcsackett, awesome, thanks!
<jcsackett> deryck: r=me. looks like nice work.
<jcsackett> and it just taught me how the getter/setter attr stuff in YUI works. :-P
<deryck> jcsackett, awesome, thanks!
<deryck> heh
<deryck> good deal then.
<jcsackett> indeed. :-)
<deryck> it's pretty expressive when you get you're head around it.
<jcsackett> yeah, it certainly seems pretty cool.
<cr3> hi folks, there are broken links to the Twisted Coding Standard and CodingStyle for Zope 3 on https://dev.launchpad.net/PythonStyleGuide
<cr3> I suspect the first should point to: http://twistedmatrix.com/trac/browser/trunk/doc/core/development/policy/coding-standard.xhtml?format=raw
<cr3> and the second to: http://wiki.zope.org/zope3/CodingStyle
<cr3> not knowing what the original links looked like, these are only googled guesses
<lifeless> sounds good to me
<lifeless> cr3: so, did you want to talk ?
<cr3> lifeless: sure, I was mostly concerned about solving the problem of synchronizing user data not only internally but for community projects as well. do you think it would be reasonable to expose both a restful and xmlrpc interface, ie would this just cause more confusion?
<lifeless> I think thats premature generalisation then.
<lifeless> very easy to overbuild and underdeliver that way
<cr3> lifeless: as you said, concentrating on technology at this point might be premature but I feel there's already a strong preference, maybe because of precedence, for xmlrpc rather than restful, maybe because of launchpadlib experience.
<lifeless> we only truely know our requirements => and we're wrong most of the time :)
<cr3> lifeless: good argument, I'm already convinced :)
<lifeless> ok :)
<lifeless> so - a LEP soon ?
<lifeless> based on what *result tracker* needs
<cr3> lifeless: onto the next concern then: I'm not sure when I can deliver a LEP so I hope nobody else might be depending on exposing such a microservice, ie I wouldn't want to block anyone or duplicate their effort
<lifeless> I will work with you to refine things and make tech choices once the needs are really understood
<lifeless> cr3: writing the LEP doesn't imply implementing
<lifeless> cr3: its a discussion point for decision making
<lifeless> cr3: And I promise you, if/when someone else needs something similar, I won't queue them up behind you
<cr3> lifeless: ok, lets say for Monday then, any later would just be procrastination :)
<lifeless> cool
<lifeless> I shall be around :)
<cr3> broken links in PythonStyleGuide fixed, enjoy!
 * cr3 thinks that wikis should have a broken link crawler running periodically
<lifeless> I think they break enough on their own
<abentley> deryck: could you please review https://code.launchpad.net/~abentley/launchpad/dynamic-listing-robustness/+merge/81904 ?
<deryck> abentley, I can.  I'm about to go offline to switch work locations, but can do it when I return.
<abentley> deryck: Sure.
<jcsackett> abentley: just looked at your most recently put up MP. it is r=me.
<abentley> jcsackett: thanks.
<lifeless> allenap: would voice help ?>
<lifeless> allenap: we may be talking past each other
<allenap> lifeless: Yeah, let's do that.
<allenap> Skype okay?
<lifeless> allenap: now? Or next week? now is fine for me
<lifeless> skype is fine
<cr3> huwshimi, danhg: for the meeting, mumble or just irc?
<huwshimi> cr3: Hi, lets do voice if we can.
<allenap> lifeless: Yeah, now is good.
<huwshimi> danhg: Do you have mumble set up?
<lifeless> allenap: calling
<allenap> lifeless: Skype has crashed :-/
<lifeless> hah! win :(
<cr3> huwshimi: worst case, I can probably try to conference both of you in by phone
<huwshimi> cr3: Lets mumble for now and if that doesn't work for Dan we'll try something else.
<cr3> huwshimi: http://results-tracker.ubuntu.com
<cr3> huwshimi: http://results-tracker.ubuntu.com/ubuntu/oneiric/+testruns/2183
<deryck> abentley, hey, I see jcsackett beat me to the review.
<abentley> deryck: yeah.  He's feisty.
<deryck> indeed
<jcsackett> deryck, abentley: i just like keeping the queue clean. :-P
<lifeless> abentley: hi
<lifeless> abentley: why do you want to precache batches ?
<abentley> lifeless: I want to make the user experience fluid.
<lifeless> abentley: I don't think its a good use of money to compute results that will only rarely be used.
<abentley> lifeless: I don't follow.
<lifeless> abentley: there are what, 10 adjacent batches - first, last, next, prev, and 6 or so different sorts.
<lifeless> abentley: the user can only follow one of those, any results for the other 9 will be wasted.
<huwshimi> deryck: Just running a bit late. 5 mins?
<deryck> huwshimi, ok.  abentley and possibly adeuring will join us.  mumble ok?
<huwshimi> deryck: Fine
<abentley> lifeless: No, they are not wasted just because they are not used immediately.
<abentley> lifeless: The various sorts are always the first batch, no matter what batch you're on.
<abentley> lifeless: So they apply for the life of the page.
<lifeless> if the user doesn't change to all of those sorts, then the results do not benefit that user, right ?
<abentley> lifeless: yes.
<lifeless> We are running short of server resources as it is; I think we need a strong justification for doing speculative calculations like this - particularly while we have timeouts doing bug search batches
<lifeless> there is also the complication that if you don't serialise the requests you can easily DOS our serers
<lifeless> and if you do serialise, then you need to make sure you're still responsive when the user clicks on a batch that you haven't received a result for yet.
<deryck> huwshimi, abentley -- let's meet in Orange room.
<abentley> lifeless: If money is an issue, exactly how much money are we talking about?  We can't compute the tradeoff without knowing that.
<huwshimi> deryck: No problems, I'll be a minute
<abentley> lifeless: if 10 requests can DOS our servers, doesn't that indicate a problem with our infrastructure?
<lifeless> abentley: can I ask what is unfluid about the experience today ?
<lifeless> .
<lifeless> (sorry, my adsl is terrible at the moment)
<lifeless> .
<lifeless> tcp, please come back to me
<abentley> lifeless: on a call.
<lifeless> .
<lifeless> .
<lifeless> abentley: we have 64 concurrent service points, which can support all our users today. If you make bug search - our single most common operation - use 10 times the resources, we'll have to expand the cluster.
<lifeless> abentley: how much? I don't know.
<lifeless> abentley: but server scaling isn't cheap, and we should strive for efficiency. I don't see calculating results speculatively as being very efficient; I'm sorry but I'm going to ask that this not land - as a similar thing in the bug subscription area was backed out.
<huwshimi> http://people.canonical.com/~huwshimi/status_colours.png
* jcsackett changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<sinzui> wallyworld_, wgrant, StevenK: http://www.youtube.com/results?search_query=most+interesting+man+in+the+world&aq=0&oq=most+inter
<lifeless> bbiabm, dropping lynne up the street
* wallyworld_ changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugtasks: 275
#launchpad-dev 2011-11-11
<poolie> huwshimi, hooray for starting somewhere
<huwshimi> poolie: Haha.
<huwshimi> poolie: For too long I've been paralised by Launchpad not having the infrastructure to do things properly.
<huwshimi> poolie: Making the infrastrucuture right has resulted in a trade-off for good UX in Launchpad
<huwshimi> poolie: Now I don't care about the infrastructure, I just want the UX to be good
<huwshimi> *flame war, GO!*
<lifeless> infrastructure for its own sake is waste ;)
<lifeless> huwshimi: I totally applaud reasoned cutting-of-gordian-knots, which I think you are doing here.
<huwshimi> lifeless: And likewise, I'm all for proper infrastructure
<huwshimi> I actually really can't stand doing things poorly. I've just gotten to the point that I'd rather do things poorly than not at all at the moment
<lifeless> huwshimi: One thing launchpad has gotten stuck on in the past is defining 'right' - and ended up with bells and whistles we don't need.
<lifeless> We also tend to focus too much on doing it all ourselves, IMO
 * poolie cheers
<poolie> i still like the idea of accumulating even just a set of screenshots of things we think were done right
<poolie> by way of a cheap style guide
<huwshimi> lifeless: Also as this is the web we should be architecting for change
<lifeless> huwshimi: of course! you've seen my slides, right :)
<wgrant> Launchpad has never done any sort of, well, design or architecture.
<wgrant> Except when it designs things too much and badly.
<lifeless> wgrant: you may find yourself stepping on toes if you don't qualify things a little :)
<huwshimi> lifeless: Are they slides from Dallas?
<huwshimi> or dublin
<lifeless> huwshimi: prague and dallas
<huwshimi> lifeless: Are they online somewhere, I remember your talk, but don't remember all the details
<lifeless> let me get you a link
<lifeless> there is one in the wiki under ArchitectureGuide, but also ..
<lifeless> https://docs.google.com/present/edit?id=0AR8DGdpwOJuiZGdwZGNmbjlfMTFoY2Nwcm1ocw&authkey=CMiPvu0F
<lifeless> and
<lifeless> https://docs.google.com/a/canonical.com/present/edit?id=0AR8DGdpwOJuiZGdwZGNmbjlfNGZkNDZmZ2N6&authkey=CJWpj5EN&hl=en_US
<lifeless> blargh, if either don't work let me know :)
<huwshimi> lifeless: They, work. Just taking a look
<poolie> huwshimi, the question came up the other day of: why do the importance labels look like "tags"  or labels
<poolie> not sure if you were here for it but i'm curious if you know
<huwshimi> poolie: There's no real science behind it. It's a very common design pattern for representing importance or status. Both importance and status were displayed like that in early mockups
<poolie> it's ok with me
<lifeless> wgrant: sob
<lifeless> wgrant: 'TypeError: Could not serialize object set([u'OOPS-ABCDEF1234']) to JSON'.
<lifeless> wgrant: is this just cruft - e.g. if I list() it, its good ?
<lifeless> [list works]
<wgrant> omg optus finally fixed it.
<wgrant> lifeless: Yep, listing it should work fine.
<StevenK> wgrant: It's only taken them until 1pm
<wgrant> It's apparently been broken since 3am or so.
<StevenK> So 10 hours of international routing being broken. Epic.
<wgrant> Only for most of Europe.
<wgrant> Which nobody cares about, I guess.
<lifeless> and with that, oops-prune is dead.
<lifeless> I need a hero^Wreviewer https://code.launchpad.net/~lifeless/launchpad/bug-888375/+merge/81928.
<huwshimi> wallyworld_: Not sure if you saw my last message yesterday about that javascript I was working on. I changed the io events and also fixed things up so they use classes instead of styles for show/hide. Do you want to give that one last look? https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689
<wallyworld_> huwshimi: looking now
<huwshimi> wallyworld_: Thanks, no hurry
<wallyworld_> lifeless: i'll look at your mp too :-)
<lifeless> thanks
<wallyworld_> huwshimi: looks good. were you going to write or update any yui tests?
<huwshimi> wallyworld_: Erm, I didn't look
<huwshimi> wallyworld_: I guess I'll need to have a look as I've moved this code around (I didn't write most of it, just fixed it up)
<wallyworld_> huwshimi: understood. it's was naughty for it to be approved without any questions about tests :-)
<huwshimi> wallyworld_: :)
<wallyworld_> lifeless: do we want to make the search interval half closed as is common with date/time based searches?
<lifeless> wallyworld_: I don't think we care
<wallyworld_> or allow one of the start/end dates to be None, perhaps with a sensible default to disallow too broad an interval
<lifeless> I think thats overthinking it
<wgrant> SIGH
<wgrant> It is broken again.
<wallyworld_> i would care because if you want all oops for a particular day, you would normally do 1/1/2009 00:00 <= date < 2/1/2009 00:00 for example
<lifeless> our replication strategy alone means the former wouldn't be sufficient and the latter is unneeded - oops will always care about exact end dates as they won't be pruning very new things
<wallyworld_> ok
<lifeless> wallyworld_: between is fully open
<lifeless> wallyworld_: a BETWEEN x AND y
<lifeless> ->
<lifeless> a >= x AND a <= y
<wallyworld_> hmm. i checked the online. the example given was fully closed
<lifeless> wallyworld_: so naive use will Just Work anyhow (subject to replication lag etc etc if we ever get the API running on slaves)
<wallyworld_> but it could be wrong
<lifeless> http://www.postgresql.org/docs/8.4/static/functions-comparison.html
<wallyworld_> should there be asserts / ValueError checks to catch None dates being passed in then? also context_params is not checked for None. do we care?
<lifeless> 'addition to the comparison operators, the special BETWEEN construct is available:
<lifeless> a BETWEEN x AND y
<lifeless> is equivalent to
<lifeless> a >= x AND a <= y
<lifeless> '
<lifeless> context_params is internal code only
<lifeless> date ranges are the only thing clients are permitted to influence
<lifeless> if they pass none in, it will do date between NULL AND <other>
<wallyworld_> internal code can still be buggy :-) but fair enough
<lifeless> which will match nothing (as null operator foo -> false)
<lifeless> wallyworld_: it can be, which is why there are two ws tests that test the product and distribution glue is correct
<wallyworld_> ok. thanks for clarifying :-)
<lifeless> wallyworld_: I will teak the interface docstring
<lifeless> wallyworld_: to avoid folk getting confused and thinking its like a slice vis-a-vis None
<wallyworld_> cool. that would be good, since what's there is open to assumptions :-)
<wallyworld_> lifeless: i can't see the replacement pruning functionality. there's apis to search for oopses but the pruning capability has been deleted ans there's no replacement in this mp?
<lifeless> thats right
<lifeless> its no longer a launchpad concern
<lifeless> it should never have been part of the LP tree anyway :)
<wallyworld_> sure. just checking that we weren't losing expected functionality
<wallyworld_> s/expected/required
<lifeless> oh we are
<lifeless> just it won't be coming back to the LP tree
<lifeless> separate project now
<wallyworld_> and it's ready to go? or will we go without pruning for a bit?
<lifeless> python-oops-datedir-repo
<lifeless> its about an hours work
<lifeless> :)
<wallyworld_> ok
<wallyworld_> famous last words :-)
<lifeless> getting it deployed will take longer
<wallyworld_> so we can afford not having oops pruning for X interval till it gets done i guess
<lifeless> yes
<lifeless> carob suffers before the appservers do
<huwshimi> Does a + in one of our urls signify a static path (as opposed to dynamic data e.g. launchpad.net/dynamic/+static/)? Do all our URLs follow this pattern?
<lifeless> and right now, for staging oopses, it has no pruning, because this APi isn't here, but they are using the amqp oops path
<wallyworld_> thanks. i'm quite ignorant of our deployment requirements, so just wanted to check :-)
<lifeless> huwshimi: the + indicates a view of an object
<huwshimi> lifeless: Oh
<lifeless> huwshimi: so /~person/+index is the actual url for /~person
<lifeless> huwshimi: its a bit bong
<lifeless> huwshimi: it has a few reasons behind it, but it gets in the way too
<huwshimi> lifeless: So in a hypothetical world where we would map bugs.launchpad.net to a new URL would it be launchpad.net/launchpad/bugs or launchpad.net/launchpad/+bugs?
<lifeless> huwshimi: there are plenty of options
<lifeless> huwshimi: bugs.l.n has a distinct view from code.l.n, but you could do launchpad.net/+bugs-index and launchpad.net/+code-index
<lifeless> for instance
<huwshimi> lifeless: OK thanks, it's not a problem now, I was just trying to understand how the +s work
<lifeless> de nada
<wgrant> huwshimi: + is mostly used to disambiguate system- and user-controlled path segments. Since it's not permitted as the first character of a Launchpad name, we can support paths like /PROJECT/SERIES and /PROJECT/+bug/BUG without conflicts. It's also used in view names (eg. +bugs, +index), for similar reasons.
<wgrant> So yes, basically what you said.
<wgrant> There are some old paths where that doesn't hold true.
<wgrant> eg. /bugs/BUG
<wgrant> But it applies to most places.
<huwshimi> wgrant: Great, good to know.
<lifeless>  /bugs/BUG was deliberate
<lifeless> we don't need to use it for new stuff, though it is a safe default
<huwshimi> lifeless: I imagine if we did map our subdomains then bugs.launchpad.net/ would become launchpad.net/bugs/ so /bugs/BUG would make even more sense
<lifeless> huwshimi: if we consolidate, I'd really love to see the namespace not deepend
<lifeless> huwshimi: would launchpad.net/$project -> homepage; click on bugs there, be ok ? I think google code basically do this..
<lifeless> huwshimi: or perhaps make 'bugs' subordinate to the project context :)
<huwshimi> lifeless: Sorry, I'm not quite sure what you mean?
<huwshimi> lifeless: I would imagine you would have something like launchpad.net/launchpad/+bugs, launchpad.net/launchpad/+translations etc.
<huwshimi> lifeless: I was saying launchpad.net/bugs/ would become a homepage for bugs like bugs.launchpad.net is now (if we decided we still wanted it)
<huwshimi> lifeless: Is that what you were asking?
<wgrant> That's how it was originally, actually.
<wgrant> until 2006ish.
<huwshimi> wgrant: April, 2006
<wgrant> huwshimi: How'd you work that out?
<huwshimi> wgrant: http://www.markshuttleworth.com/archives/30
<wgrant> A-ha.
<nigelb> What happened to Applications on subdomains? :)
<wgrant> nigelb: That's what we have now.
<lifeless> huwshimi: I thought you meant e.g. bugs.l.n/launchpad/+bugs?field.. -> l.n/bugs/launchpad/+bugs?field..
<wgrant> And have had for several years.
<lifeless> huwshimi: which I think would be ugly
<nigelb> wgrant: Ah. Launchpad applications. Right.
<nigelb> I confused that with projects.
<huwshimi> lifeless: Oh, yeah, not at all. Let's make the project the base and tack the app url on after that somehow
<lifeless> huwshimi: I think that makes a lot of sense
<huwshimi> lifeless: :D
<huwshimi> wgrant: It's funny, there's a few things I think we got right a few years ago that I would like to revert to :)
<wgrant> huwshimi: Yeah...
<lifeless> wgrant: can you check I transcribed your fix for bug 884265 correctly ?
<_mup_> Bug #884265: req_vars header values may cause crash in dboopsloader <python-oops-tools:Triaged> < https://launchpad.net/bugs/884265 >
<mwhudson> let's all go back to arch!
<wgrant> mwhudson: A sound plan.
<lifeless> wgrant: and if so, approve my MP ?
<wgrant> lifeless: Your MP being rye's?
<lifeless> no
<lifeless> t'other one
<wgrant> Ah.
<wgrant> Which was created after I last looked.
<wgrant> approvated
<poolie> huwshimi, "a normal ux"
<poolie> awesome
<poolie> i think even google seems to be going away from having separate subdomains?
<lifeless> can has review? https://code.launchpad.net/~lifeless/python-oops-tools/bug-888866/+merge/81933
<poolie> it's hard to tell
<poolie> but the calendar is now google.com/calendary
<nigelb> huwshimi: I like this going back to no subdomains thing ;)
<poolie> me too
<wgrant> It's also pretty damn easy to move away from subdomains.
<poolie> i would like to see a mockup of what the navigation ought to look like
<wgrant> We've had so many different navigation UIs over the years.
<wgrant> None of them have worked :/
<poolie> huwshimi, one interesting thing from gdd was their stuff about breadcrumby/back-stack navigation in android
<wgrant> And now GitHub has basically reproduced LP 2.0
<poolie> it does not map totally directly
<poolie> but there are some ideas
<poolie> is that good or bad?
<poolie> specifically they have a plain 'back', crossing application or other activity boundaries
<poolie> and also an 'up'
<nigelb> wgrant: heh, yeah.
<poolie> taking you towards the top level of that particular application
<poolie> obviously for us its complex because there is browser back, and also going back towards the lp app root vs the pillar root
<poolie> but, perhaps we want to emphasize the latter
<lifeless> can has review ? https://code.launchpad.net/~lifeless/python-oops-datedir-repo/bug-888866/+merge/81937
<lifeless> well, EOD for me.
<lifeless> EOW even
<poolie> cheerio lifeless
<poolie> wgrant, ok, the package style is a bit more modern now
<StevenK> Does it make use of dh 7 yet?
<wgrant> hardy doesn't have dh7, but cat should if you want it.
<StevenK> Oh, drat. Hardy
<micahg> hardy-backports has dh7 :)
<poolie> it doesn't
<poolie> that doesn't seem high on the list of issues
<poolie> should i add this to just launchpad-dependencies, or something more specific?
<poolie> which one is used to run tests i wonder
<poolie> developer dependencies apparently
<huwshimi> Have a good weekend all!
<poolie> you too
<wgrant> poolie: launchpad-developer-dependencies, probably.
<poolie> yep, done and pushed
<poolie> do i _have_ to update the ec2 image for new dependencies?
<wgrant> Yes.
<poolie> can't it just install them when it boots?
<wgrant> It doesn't apt-get upgrade
<StevenK> No
<poolie> could it?
<poolie> so, to me it really does look like it does an apt-get upgrade
<poolie> but that may be a decoy
<StevenK> poolie: The only case that does is the AMI upgrade path
<poolie> yeah, but istm i can, and might as well, just do it when a real test is booting
<StevenK> I don't like that idea.
<StevenK> Some tests on ec2 are fragile enough without adding more changes.
<poolie> mm
<poolie> but this is fairly determinisitic
<poolie> and brings the tests closer to what people will experience elsewhere
<StevenK> We have been on rev 522 for over one month
<StevenK> And given it takes about an hour to make a new AMI, I don't think the cost is high at all.
<poolie> !!
<poolie> i think an hour of avoidable works' pretty high
<StevenK> It isn't an hour of work. It's one command and wait an hour
<StevenK> Much like our test suite
<poolie> that's worth cutting down too
<poolie> an hour of latency is also worth cutting out
<StevenK> I would seriously prefer you just make a new AMI and bring up changing it on the mailing list.
<poolie> i am running my test in this now
<poolie> which seems equivalent
<StevenK> And I submit that changing it and then announcing it is as much akin to what you almost accussed me of with +personal.
<poolie> oh
<poolie> i'm just going to put up an mp
<poolie> i'm not going to just change the ami
<poolie> in fact i'm pretty sure i am not trusted to change the ami, which is another reason i looked at just updating the single machine
<poolie> and i did ask about it here
<StevenK> poolie: Have a look at lib/devscripts/ec2test/account.py, line 25 on.
<poolie> as i suspected, it's out of date :)
<poolie> wrt henninge
<StevenK> We don't remove people
<StevenK> And I suspect adding your ID there would not cause much of an issue
<poolie> yeah i could
<StevenK> We can probably remove henning, salgado, jml
<poolie> i think current staff who are no longer on the lp team is a bit different
<poolie> but, maybe still least privilege
* wallyworld_ changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
<jml> actually, that reminds me, I need to delete my LP image
<jml> it's costing me US$0.53 and Â£2 each month in Amazon & then bank fees.
<StevenK> 2GBP is a bit steep
<mrevell> Hello
<danhg> Morning Everyone
<adeuring> good morning
* adeuring changed the topic of #launchpad-dev to:  https://dev.launchpad.net/ | On call reviewer: adeuring | Critical bugtasks: 275
<jam1> I thought the new fast-oops was deployed. I got an OOPS trying to list my branches in a project page, but the oops (with a hash) hasn't shown up after about 10 min
<jam> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-55ae33da0714197906011068d73e95f3 if people care
<rvba> jam: fast-oops is not yet deployed in production.
<jam> ah, I thought robert said that it will show up once we switch to hash-based ids
<jam> anyway, https://bugs.launchpad.net/launchpad/+bug/889042
<_mup_> Bug #889042: OOPS while viewing branch listing for ~user/project <Launchpad itself:Triaged> < https://launchpad.net/bugs/889042 >
<jam> browsing to a per-user branch listing tracebacks
<rvba> jam, hum, that's related to the new branch menu recently introduced, I think it's worth disabling the flag until the problem that you just had is solvedâ¦ I'll take care of thatâ¦
<jam> rvba: thanks. code.lpn/~jameinel works but code.lpn/~jameinel/project doesn't
 * nigelb wonders what mrevell was testing
<mrevell> nigelb, hmm?
<mrevell> nigelb, Oh, the mailing list archive seems to be delayed by a couple of days
<nigelb> ah, nice :)
<nigelb> I did ask wgrant about it some days ago.
<nigelb> err weeks
<nigelb> Back then it was hours, not days.
<rvba> jam: the flag has been removed so this should be back to normal, thanks for reporting this.
<jam> rvba: confirmed, my ~user/project page loads now
<deryck> Morning, all.
<rick_h> morning
<rvba> Hi adeuring, would you have time to have a look at this MP? https://code.launchpad.net/~rvb/launchpad/bug-889042/+merge/81988
<adeuring> rvba: sure
<flacoste> mrevell, sinzui, gnuoy, allenap: i realy doubt that the delays in mail being sent out is related to the delays in mail archiving
<rvba> adeuring: thank you. The diff should be available shortlyâ¦
<flacoste> mailman is using separate queue runner for all these aspects
<sinzui> flacoste, as wgrant remarked, there is a lot of list traffic in ubuntu-x-swat maybe
<flacoste> that's a more likely explanation
<flacoste> in the past we had issues with the xmlrpc calls timing out, but that was fixed i think
<flacoste> is mailman the bottleneck or our stmp server?
<flacoste> also, mrevell tests seem to have been near instantaneous
<gnuoy> flacoste there is no backlog in exim
<flacoste> is there a backlog in mailman?
<mrevell> Daviey, Is there still a backlog for the openstack list?
<mrevell> Daviey, I mean delay
<deryck> adeuring, https://dev.launchpad.net/QA/ExploratoryTesting/CustomBugListings
<deryck> abentley, https://dev.launchpad.net/QA/ExploratoryTesting/CustomBugListings
<Daviey> mrevell: The archive doesn't seem to be showing recent mail, https://lists.launchpad.net/openstack/
<Daviey> regarding actual mail, i'm not sure.
<sinzui> Why are there always 3 new bugs? I have checked for private new bugs that we might be be subscribed too, but there are none in production
<flacoste> Daviey: right, mailing list archives isn't expected to work, a work around is to visit mail-archives.org
<flacoste> all your lists are archived there in real-time
<Daviey> flacoste: for the nosey, why is that?
<flacoste> actually, it's mail-archive.com
<sinzui> excellent, staging is broken in the same way.
<flacoste> Daviey: because originally we didn't want to host archives ourself, because there is no good software for it
<Daviey> ah
<flacoste> down the line we decided to do a dirt-cheap solution
<flacoste> but that's what we have now
<flacoste> dirt-cheap
<flacoste> that doesn't work most of the time
<deryck> Seems odd to me we have the same yuixhr failure in both devel and db-devel.
 * deryck is investigating build failures
<Daviey> flacoste: thanks
<adeuring> rvba: r=me
<rvba> thank you adeuring.
<gnuoy> flacoste, there doesn't seem to be a  backlog in mailman
<flacoste> gnuoy: ok, so we think the delay problem is solved?
<gnuoy> flacoste, no, sorry, I was just answering your question from an hour ago
<flacoste> well, if we don't have a queue, there shouldn't be delays :-)
<gnuoy> unless the thing putting them on the queue it the problem
<gnuoy> s/it/is/
<deryck> abentley, lines are up again if you need to land stuff.
<abentley> deryck: Cool, thanks.
<deryck> matsubara, we using mumble for the call?  or skype?
<matsubara> deryck, skype so danhg can join us
<deryck> ok
<abentley> matsubara: online now, as aaron_bentley
<deryck> matsubara, I'm ready to go too.
<deryck> adeuring, you want in this call?
<matsubara> abentley, thanks. skype is taking forever to find your user id
<adeuring> deryck: sure. skype?
<deryck> adeuring, yup.  and tell matsubara your skype id if he doesn't have it.
<matsubara> deryck, can you host? I suppose you have aaron and abel already in your list?
<deryck> matsubara, sure.
<matsubara> do you see me online?
<adeuring> matsubara: adeuring on skype. give me two minutes to Åtart a machine where skype is installed
<flacoste> deryck: with spurious test failures, don't cross fingers, disable tests :-)
<allenap> gmb: Do you what's happening/happened with bug relationships?
<gmb> allenap: Not since I handed in the draft of the LEP. mrevell and flacoste will likely know where it's at.
<flacoste> allenap: your squad is likely to work on it
<flacoste> allenap: once gary squad is done with test suite parallelizatin
<allenap> flacoste: So something for 2012 then. Ta.
<flacoste> allenap: it's not clear what the exact scope will be yet
<flacoste> allenap: yep
<flacoste> early 2012 is what is on our agenda
<deryck> flacoste, sorry was otp.  I wasn't confident it was spurious enough to disable.  if it happens again, I will.
<allenap> Does anyone know why I can't find more than 1207 bugs at https://bugs.qastaging.launchpad.net/ares (via advanced search), even though I know that there are 1515 bugs, and launchpadlib confirms it? I assume it's something to do with bugs/bugtasks, yet they are all newly created tasks from an import.
<deryck> allenap, private bugs?
<allenap> deryck: There's one private bug. Checking anonymously via launchpadlib yield 1514 bug tasks.
<deryck> hmm, weird.  I remember something like that with a bug import for me.  either private bugs or closed bugs.
<allenap> deryck: I sense that I've figured this one out before, but I cannot quite access the memory.
<deryck> heh, yeah
<deryck> allenap, do you know what the user sees?
<deryck> allenap, also, expired maybe?
 * deryck is guessing :)
<allenap> deryck: The user and I see the same things: just 1207 bugs (well, he sees 1206). Ah, I have an inking...
<allenap> deryck: I've figured it out! There are 308 tasks with Unknown status, and that's not given as an option in the advanced search page.
<deryck> allenap, ah ha! :)
<allenap> deryck: Thanks for prodding in the right direction :)
<deryck> allenap, np!
<lifeless> allenap: evening
<rvba> Hi lifeless!
<lifeless> hi rvba
<lifeless> sad news about the oopses!
<rvba> yeah, but that's fixed, the branch is in ec2.
<lifeless> cool
<sinzui> lifeless,  do you know if the process that manages BugSummary ever pruges/rebuilds the stats? I am trying to understand why /launchpad always has three new bugs that the db says is not true
 * rvba EODs
<rvba> lifeless: I've got a perf related question on this MP, maybe you could have a look next weekâ¦ https://code.launchpad.net/~rvb/launchpad/builders-timeout-bug-887078-eager-load/+merge/82008
 * rvba really EODs now.
<lifeless> sinzui: it doesn't no. There is obviously a bug.
<sinzui> yes. I think one something odd gets into the data, it is stuck
<sinzui> lifeless, http://pastebin.ubuntu.com/735551/ show one of the bugs has a patch. I have been looking for such a bug in Lp. They are pretty rare.
<lifeless> sinzui: could you file a bug ?
<lifeless> I meant to last time I noticed, but I got distracted
<sinzui> lifeless, I will
 * micahg hugs sinzui for cleaning up null bug tasks
<sinzui> thank you.
<deryck> rick_h, ping.  you around?
<deryck> abentley, would you like to review my integration branch, which gets toggling of fields working?
<abentley> deryck: sure.
<deryck> abentley, thanks!  https://code.launchpad.net/~deryck/launchpad/listing-config-widget-integration/+merge/82024
<deryck> abentley, I also did a little drive by to clean up some of that CSS/design stuff we've been discussing.
<abentley> deryck: ah, okay.
<deryck> abentley, when we're nearly done, I'll do a pass to consolidate/polish CSS rules, and perhaps simplify.  But that should be closer to the mockup, and let everyone see we can get it right.
<abentley> deryck: It looks like you're specifying field visibility rather than using the values from LP.cache.  Is that right?
<deryck> abentley, yes, I'm getting the value from when the form overlay is submitted.
<abentley> deryck: I mean BugListingConfigUtil.field_visibility_defaults
<deryck> abentley, ah, yes.  I hard coded them in the widget.  so it echos the browser code.
<abentley> deryck: What is the purpose of echoing the browser code instead of using the values generated by the browser code?
<deryck> abentley, to me it seemed better because the widget can work in isolation of the cache.  or rather, I did build it separate from the cache....
<deryck> abentley, so there's no harm in pulling from cache in a later pass and not leaning on the hard coded defaults.
<abentley> deryck: I think this is a DRY violation.  If we change only one, then the bug listings (client-side or server-side) and the widget will disagree about which fields are displayed.
<deryck> abentley, I agree.  Does it matter for the sake of this branch, though?  Can I change that when I fix the persistence issue?
<abentley> deryck: And consider that the browser code will probably start setting these values according to query variables, per the LEP's URL requirements.
<deryck> right
<deryck> abentley, I'm completely agreed with you.  Just wondering if fixing is a subsequent branch is ok.
<abentley> deryck: I guess you could, but I don't understand why you'd want to.  Violating DRY takes more effort then following it.
<deryck> abentley, because, it's a larger change, and I want to think about how best to do it.  Assuming you don't have other issues, this could land now, and at least get the interaction live for people to see.
<abentley> deryck: It's not just a one-line variable assignment?
<deryck> abentley, ah, you mean just in the page hook up?  I was thinking refactoring so the hard coded ones weren't there at all.  fixing tests, etc.
<abentley> deryck: Oh, I see.
<abentley> deryck: Well, if the tests are holding it back, I guess that's okay then.  I just didn't understand your reluctance.
<deryck> abentley, ok, thanks.  I would like to fix it altogether, so the widget can't work without getting data from the cache.
<abentley> Right.  And then you could have a factory function for testing.
<deryck> it's pedantic I guess, but it makes more sense to me if the test describes what you should do with the widget.
<deryck> abentley, right
<deryck> I meant my reluctance is pedantic, not your suggestion, abentley :)
<abentley> deryck: r=me.
<deryck> abentley, thanks!
<rick_h> deryck: pong, been afk most of the day
<abentley> deryck: We should also look at the module naming.  I see "lp.bugs.buglisting" and "lp.buglisting_utils" and they don't match.  And I'm not sure whether ListingNavigator shouldn't also be a utility.
<deryck> abentley, yeah, I agree.  I'm not crazy about my namings of this and configutils.BaseConfigUtil.
<abentley> I'm not entirely sure what the style is supposed to be.  I've seen a lot of different things.
<deryck> we're inconsistent indeed.  I think everything should just live under lp, a la Y.lp.module.
<deryck> and probably a listings module and a utilities modules would let us divide these pieces up nicely.
<deryck> abentley, can you mark the mp approved too please?
<deryck> Later on, everyone.  Have a nice weekend.
#launchpad-dev 2011-11-12
<Ronnie> what version of python-launchpadlib is advised. the 1.6.0 from the 10.04.3 repo or the 1.9.9 from pip ?
<Ronnie> the software works with both versions
<Ronnie> so whats the best in terms of stability and performance ?
#launchpad-dev 2011-11-13
<poolie> good morning
<rick_h> afternoon poolie
<poolie> hi rick_h
<thumper> morning
<poolie> lifeless, could you look at https://code.launchpad.net/~mbp/launchpad/ec2-update/+merge/81946 (pretty small)
<lifeless> can has review ? https://code.launchpad.net/~lifeless/python-oops-wsgi/bug-888866/+merge/82088
#launchpad-dev 2012-11-05
<StevenK> wallyworld_: 200!
<wallyworld_> bugs?
<StevenK> wallyworld_: http://webnumbr.com/launchpad-critical-bugs
<wallyworld_> hooray. too bad that includes the whole project
<StevenK> I might be dropping it to 199, just investigating
<StevenK> Or not, given the output on neem
<nigelb> Wait, weren't we at 400 at one point?
<wallyworld_> yes, but we have been busy
<StevenK> nigelb: Yah.
<StevenK> nigelb: Purple is just that good.
<nigelb> StevenK: Dang!
<StevenK> nigelb: The graph tells the story
<nigelb> StevenK: link me?
<StevenK> nigelb: http://webnumbr.com/launchpad-critical-bugs
<nigelb> the hell!?!
<spm> nigelb: I wouldn't pay too much attention to the braggarts tbh; it was all easy low hanging fruit they picked off.
<spm> deliberate marking of trivial bugs as critical, over time.
<spm> and other assorted data juggling tricks
<StevenK> Suuuure
 * spm goes to greet StevenK @ the front door
<StevenK> Haha
<nigelb> haha
<adeuring> good morning
<czajkowski> morning
<ajmitch> evening
<rick_h_> morning LP peeps
<czajkowski> rick_h_: hiya
<czajkowski> rick_h_: all back to normal now ?
<rick_h_> czajkowski: almost. When I got back wed night the wife went away thurs morning. She just got back last night
<czajkowski> rick_h_: aww
<rick_h_> so will finally get back to normal this week I hope
<rick_h_> annual girls weekend kind of thing
<czajkowski> nods
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: ~200
<abentley> benji: Could you please review https://code.launchpad.net/~abentley/launchpad/info-type-adds-policy/+merge/132749 ?
<benji> abentley: sure
<abentley> benji: Thanks.
<abentley> benji: Could you please review https://code.launchpad.net/~abentley/launchpad/no-proprietary-packagings/+merge/132960 ?
<benji> abentley: sure
<abentley> benji: Thanks.
<bigjools> morning
<mwhudson> bigjools: apparently
<bigjools> mwhudson: haha :)
<bigjools> mwhudson: as I saw in a tweet earlier, doomed to be awake at 4am in every timezone
<lifeless> bigjools: o/
<bigjools> howdy lifeless!
<bigjools> lifeless: how's the new job?
<lifeless> bigjools: pretty fantastic
<bigjools> good to hear
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~200
#launchpad-dev 2012-11-06
<StevenK> IntegrityError: new row for relation "bug" violates check constraint "sane_description"
<StevenK> Success
<mwhudson> i bet there are plenty of bugs that pass that constraint but are still not sane
<StevenK> mwhudson: Hush
<lifeless> mwhudson: I don't believe bugs can be sane or not.
<mwhudson> lifeless: all observations depend on where you stand and all that?
<StevenK> Anyway, checking that description is under 50000 chars and then adding more characters to it === BAD
<lifeless> indeed
<mwhudson> :(
<mwhudson> https://launchpad.net/lava/+milestone/2012.11 is timing out
<mwhudson> doesn'
<mwhudson> doesn't for an icognito window
<mwhudson>  OOPS-c9dfbe1f7caf5911bd217ad90e6b1ea7
<mwhudson> i wonder if it will work if i leave beta testers...
<mwhudson> and then lp-oops.canonical.com said OOPS-2dbf673404dbb8692b14ac4ccb7133a6
<mwhudson> i think this is an indication to stop looking at this
<wgrant> mwhudson: oops.canonical.com
<wgrant> It's there
<wgrant> But no comment :)
<mwhudson> oh right
<mwhudson> raise TemplateDoesNotExist(name)
<wgrant> https://oops.canonical.com/oops.py/?oopsid=OOPS-c9dfbe1f7caf5911bd217ad90e6b1ea7
<mwhudson> LaunchpadTimeoutError: Statement: 'SELECT DISTINCT
<mwhudson> not a promising start
<mwhudson> looking at the query certainly makes me think it will work if i leave beta testers...
<mwhudson> ah yes
 * StevenK stabs zope.formlib
<wallyworld> StevenK: is the next FDT 08:00UTC?
<StevenK> wallyworld: We've been doing our FDTs whenever, due to pretty much no downtime
<wallyworld> ok
<StevenK> wallyworld: It's all good on staging?
<wallyworld> StevenK: yes, the new tables are as yet unused
<StevenK> wallyworld: How long did the patch take to apply?
<wallyworld> 0.0
<nigelb> Is today the day you guys get a holiday for a "sporting" event?
<StevenK> Only wgrant
<nigelb> Oh, so he's just gonna continue working :P
<StevenK> Melbourne has a public holiday for Melbourne Cup Day, the rest of Victoria and Australia do not.
<nigelb> ah
<nigelb> I thought the entire state got a holiday. Thought wrong I guess :)
<StevenK> I might wrong, but I thought it was just Melbourne itself.
<nigelb> You're right.
<bigjools> lazy b'stards
<nigelb> lol
<StevenK> bigjools: Speaking of those, Prince Charles and a horse were in the crowd watching
<StevenK> Oh, it wasn't a horse. It was Camilla
<bigjools> haha
<nigelb> haha
<nigelb> StevenK++
<nigelb> I can't stop laughing. Thank god I work from home.
<bigjools> in that case: lazy, privileged, tax-thieving b'stards
<StevenK> bigjools: Did you see https://plus.google.com/101824923181156392444/posts/1N6PcSMKL4g ?
<nigelb> That is brilliant!
<nigelb> play rugby (which has some similarities to American football, but does not involve stopping for a rest every twenty seconds or wearing full kevlar body armour like a bunch of nancies)
<nigelb> hahahah
<bigjools> I think that appears every year, with various details amended to reflect current reality
<nigelb> heh
<bigjools> 'We will tell you who killed JFK when you apologize for the abomination known as âTeletubbies"'
<bigjools> rofl
<nigelb> haha
<alicatux> $ bzr branch lp:launchpad
<alicatux> KilledkB   213kB/s | Fetching revisions:Inserting stream:Estimate 975008/131177
<alicatux> 3rd attempt... still getting bzr killed...
<StevenK> On your machine?
<StevenK> wallyworld__: O hai. Can haz review?
<wallyworld__> sure
<StevenK> wallyworld__: https://code.launchpad.net/~stevenk/launchpad/filebug-description-widget/+merge/133009
<wallyworld__> StevenK: 66	+ string = 'y' * 50001  ---- dont' you want to make the string nominally smaller than 50000 to ensure that the extra text tips it over an causes the error?
<StevenK> wallyworld__: I want it larger than 50000. I can, if you wish
<wallyworld__> yes, but you want the extra text to be the thing that makes it too large. that's the point of the branch
<wallyworld__> shoul the check on line 27 be > 50000 not > 50001 ?
<StevenK> wallyworld__: Yes, that's the extra check. The 'y' * 50001 will turn up as extra_description not the comment
<StevenK> wallyworld__: sane_description wants <= 50000
<StevenK> Therefore, > 50001
<wallyworld__> yes so > 50000 is the check
<StevenK> Right
<wallyworld__> > 50000 is the opposite of <=50000
<wallyworld__> ack on the y * 50001
<wallyworld__> so r=me with the 50000 change
<StevenK> wallyworld__: http://pastebin.ubuntu.com/1336641/
<wallyworld__> yes cool, thanks
<StevenK> wallyworld__: Thanks
<wallyworld__> np
<alicatux> yes, onmy machine
<StevenK> alicatux: How much RAM do you have?
<alicatux> 768
<StevenK> Wow
<StevenK> That might be why
<alicatux> old machine...
<alicatux> i see
<StevenK> alicatux: I'd recommend at least 4GiB if you want to run Launchpad for development, you probably need 2 to develop on it
<alicatux> resort to the cloud then, lol
<alicatux> i made it.
<alicatux> https://dev.launchpad.net/Running/LXC
<alicatux> step 2 "-a i386"
<alicatux> it's a 64 bit machine should I stick with i386?
<adeuring> good morning
<StevenK> dpm: You should check out translations for raring.
* rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: rick_h | Firefighting: - | Critical bugs: ~200
<StevenK> dpm: I think that should be everything
<adeuring> good morning frankban, could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/product-lp-limitedview/+merge/133046 ?
<frankban> hi adeuring, sure, right after grabbing some food
<adeuring> frankban: sure, enjoy lunch!
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban, rick_h | Firefighting: - | Critical bugs: ~200
<dpm> hey StevenK, looks good now. The contributors list seems to match the one from Q
<StevenK> Indeed.
<dpm> excellent, thanks
<StevenK> dpm: So you can handle everything else and I'm all done, until it's time to export langpacks?
<dpm> StevenK, I believe yes, your job is done. The next step will be to modify the cron jobs to do Raring language pack exports, but I'll file an RT for that.
<StevenK> dpm: You'll probably have to drop one series
<dpm> yes
<StevenK> steven@undermined:~/lp-production-crontabs% grep language-pack-export loganberry-launchpad | cut -d\  -f11 | tr '\n' ' ' ; echo
<StevenK> precise quantal lucid
<dpm> yeah, we can probably drop lucid, so that we've got at least the latest LTS and stable release
<nigelb> james_w: Happy Birthday!
<balboah> is my username supposed to include the superlong login hash in every place it is displayed?
<balboah> it doesn't fit the layout
<benji> gary_poster: once I fix some lint I'll have a tired, poor branch for you to look at... hmm, or maybe not, since it was based on one of Matt's branches; I'll need to figure out the right lbox way to do that
<gary_poster> cool benji.  I think you should just adopt it, myself
<czajkowski> balboah: can you show us what you mean ??
<gary_poster> benji, btw, notice channel :-P
<benji> pfft
<gary_poster> :-)
<balboah> czajkowski: I mean this https://dl.dropbox.com/u/2468164/Screen%20shot%202012-11-06%20at%2014.21.41.png and this: https://dl.dropbox.com/u/2468164/Screen%20shot%202012-11-06%20at%2014.23.26.png
<czajkowski> balboah: ok so if you go to edit your LP page you can change the LP display name
<czajkowski> your ac was autocreated
<czajkowski> balboah: https://launchpad.net/~/+edit
<balboah> czajkowski: aha. Hmm I wonder where the auto create happened. Oh well now it looks better, thanks
<czajkowski> sinzui: what is the difference in the tag easy and trivial
<sinzui> trivial can be fixed in one hour and probably does not need tests.
<sinzui> easy can be fixed in a day and definitely needs tests.
<mgz> that's a more concrete answer than I'd have managed :)
<sinzui> lots of text/UI changes are untestible
<czajkowski> sinzui: hmmm
<sinzui> easy was added years after trivial
<czajkowski> thats really not a great way for me to see it tbh, they both mean the same thing
<czajkowski> and a text change to me seemed trivial
<sinzui> czajkowski, we say hour because we observed in 2009 that an experienced Lp engineer can fix 8 in one day
<czajkowski> nods
<sinzui> czajkowski, I rarely use easy since most bugs can be fixed in day :)
<czajkowski> sinzui: yes but I'm not you :) and ti seemed trivial
<czajkowski> no point in arguing it just feels like semantics :s
<mgz> it seems more like there is a (semi) useful distinction, but then it becomes impossible to triage without understanding exactly how much work any particular minor change would take... which I certainly don't know at a glance
<czajkowski> and nobody had ever told me the difference almost 9 months in
<sinzui> czajkowski, there is no right answer. Lots of things people have marked easy or trivial were not. The only real way to judge them to for someone to outline the fix so that other can judge how long it would take them to complete
<czajkowski> nods
<james_w> thanks nigelb
<deryck> sinzui, hi.  got a sec?
<sinzui> yes
<deryck> I'd like to get some voice time today with you, just to chat about transitive privacy and other related things.
<deryck> sinzui, do you have a time that's best?
<czajkowski> deryck: have you seen https://bugs.launchpad.net/launchpad/+bug/1075365
<_mup_> Bug #1075365: Timeout when trying to visit previous sprint pages <fallout> <lp-blueprints> <private-projects> <regression> <timeout> <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/1075365 >
<deryck> czajkowski, indeed.  see abentley is already on it. :)
<abentley> czajkowski: Yes, and I'm investigating.
<sinzui> deryck, within the next 4 hours
<czajkowski> ah joys of not refreshing pages
<czajkowski> grand job
<czajkowski> shall move on to other bits :)
<czajkowski> thanks
<deryck> :)
<czajkowski> deryck: sorry would this be related to it ? https://bugs.launchpad.net/launchpad/+bug/1075569
<deryck> sinzui, how about 17:00 UTC, 11 for me, 12 for you, I believe?
<sinzui> okay
<deryck> sinzui, thanks.  I'll add a calendar event so we get this nice hangout setup now. :)
<sinzui> thank you
<deryck> czajkowski, that could be related, but there's a slightly different longer running query there.  abentley, can you glance at that bug in parallel to determine if it's related?
<nigelb> james_w: :)
<czajkowski> deryck: abentley thank you
<deryck> np!
<abentley> deryck: looking...
<deryck> abentley, thanks!  Feel free to triage and add a card to the board if it seems not directly related.
<abentley> deryck: Not clear whether it's related.  If we find a way of speeding up the privacy clause, that might affect both.  Added to board.
<deryck> abentley, ack, thanks.
<czajkowski> cheers folks
<mask19> I `make run`ed a new launchpad and I got "OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
<mask19> Unable to establish SSL connection."
<mask19> (wget result)
<mask19> how to fix?
<mask19> and Connection refused now...
<mask19> hmmm... it seems I cant wget it outside the container... solved
<rick_h> mask19: yea, by default apache only listened on localhost
<mask19> how do I expose it? simply change the apache config?
<rick_h> mask19: there's supposed to be a flag you can pass to a make command that lists the interfaces. looking
<deryck> You can `make install` in your lp tree to get Apache setup correctly.
<rick_h> deryck: right, but there's a BIND_ADDRESS or something you need to set to *
<rick_h> https://dev.launchpad.net/Running/RemoteAccess?highlight=%28make+install%29
<rick_h> there we go, knew there was a doc for it
<rick_h> LISTEN_ADDRESS
<jcsackett> sinzui: any chance you have time to chat when you're done chatting with deryck?
<sinzui> yes
<mask19> thanks~
<jcsackett> sinzui: excellent. ping me when you're available?
<sinzui> okay
<deryck> rick_h, mask19 -- but make install should do that for you.  If it doesn't, we should file a bug and fix it.
<rick_h> deryck: what it does is setup the apache config to only listen on 127.0.0.1 by default
<rick_h> deryck: so you have to override that if you want to access it remotely
<rick_h> e.g. server on lxc container but access with local browser
<deryck> oh wait, you mean via another machine on the same network.  sorry, ignore me.
<deryck> right, gotchas.
<abentley> deryck: Could you please run this with EXPLAIN
<abentley> ANALYSE: https://pastebin.canonical.com/77729/ ?
<deryck> abentley, sure.  getting it now....
<mask19> do you have to make run it as root after install?
<abentley> deryck: I just discovered that even though the "Long SQL Statements" portion of an OOPS does not show variable substitutions, the "SQL statement log" *does* show substitutions.
<deryck> ah, cool.
 * deryck is still waiting
<mask19> I got " Operation not permitted: '/var/tmp/mailman' "
<deryck> first runs on staging db are slowwwwww
<deryck> mask19, yeah, you need to run it as root
<deryck> oh wait
<deryck> mask19, you need to make install as root, but make run should not be.
<mask19> rm-ed "/var/tmp/mailman", and fine
<mask19> sudo make clean actually
<czajkowski> jcsackett: ello, can you help me with https://answers.launchpad.net/launchpad/+question/213444
<rick_h> czajkowski: yea, so when you auth an app you end up generating a token for that application you authorized
<czajkowski> rick_h: aye that I get
<czajkowski> the delay bit not seen before though
<rick_h> yea, not sure why it's not listed
<rick_h> I can't see his tokens to know if he's just not seeing it
<czajkowski> rick_h: ahh
<rick_h> the delay is just that the email is sent from the job system
<rick_h> so not unexpected, but guess I'm not sure why it was 16hr :/
<czajkowski> nods
<czajkowski> 16 is a bit much
 * rick_h isn't sure what the normal run time of that background email job stuff is
<czajkowski> <16 :)
<rick_h> lol
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: rick_h | Firefighting: - | Critical bugs: ~200
<mask19> where's zope.app.form?
<sinzui> jcsackett, we can talk now. I have a meeting at 2:00 EST
<jcsackett> sinzui: ok, be on g+ momentarily.
<jcsackett> sinzui: http://bazaar.launchpad.net/+branch/launchpad/+filediff/~jcsackett/launchpad/filediff-permission-denied
<jcsackett> sinzui: https://oops.canonical.com/?oopsid=787fce475f899a29b3241cc900517fb3
<lifeless> https://bugs.launchpad.net/launchpad/+bug/6 needs toggling back to invalid
<_mup_> Bug #6: "next 10 entries" at bottom of page <lp-translations> <Launchpad itself:Fix Released by carlos> < https://launchpad.net/bugs/6 >
<lifeless> apparently I didn't get the emeritus stuff anywhere as near setup as I thought I had
<lifeless> cause I can't :)
<rick_h> lifeless: updated
<lifeless> danke :P
<sinzui> jcsackett, how goes it? time for another hangout?
<jcsackett> sinzui: it goes. it's definitely a javascript error, though i have learned that the urls in the oopses are wrong. if you click one of the erroring urls in the javascript, the url accompanying the oops isn't a match.
<sinzui> \o/
<sinzui> lets hangout
<jcsackett> sinzui: ok, let me figure out where i've since put my phone. :-)
<jcsackett> sinzui: i'm on now, having responded to your invite.
<jcsackett> but you are gone.
<jcsackett> sinzui: cat disconnect you?
<sinzui> My phone thought I needed to get children
<jcsackett> ah. :-)
<StevenK> mwhudson: O hai, do you have a sec?
<mwhudson> StevenK: yes
<StevenK> mwhudson: Bug 820069 for reference, is BranchType 2 (MIRRORED) used anymore?
<_mup_> Bug #820069: BadUrlLaunchpad raised mirroring a branch <branch-puller> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/820069 >
<mwhudson> StevenK: i believe there are still branches of that type
<mwhudson> it is not possible to create them any more
<mwhudson> but the existing ones are still there
<StevenK> So the UI to create MIRRORED doesn't exist any more
<mwhudson> right
<StevenK> That would explain why I couldn't find it
<StevenK> mwhudson: So the puller is still running?
<mwhudson> yeah
<joey> OOPS-16745647cef3ea06f938ac0919f22630
<mwhudson> for mirrored and imported branches
<joey> let me rephrase that, sinzui if you are still interested in time-out oopes,  OOPS-16745647cef3ea06f938ac0919f22630
<sinzui> joey, I am reporting that oops as a bug
<joey> sinzui: I had a different code block fail again but it might be related.  OOPS-ffd854a1f01abf0eea6ae144fc7111ac
<joey> sinzui: they both use .endswith("blah")
<joey> sinzui: specifically, team.name.endswith("-r")
<joey> I can work around it. I'll just cache the team.names into python and iterate over that
<sinzui> joey, the timeout is before, the method. The timeout is not being able to get the next batch. The query is ambiguous and certainly looks expensive
 * joey nods.
<joey> I have to admit I've learned more about the API in the last few weeks than I ever did when I was on LP.
<joey> data-mining-r-us
<joey> sinzui: *sigh* oh trying to cache the names resulted in  OOPS-dba5f06bc8e0b8fd6186cf4131717dab
<joey> sinzui: I'm not using my privs to bypass the time outs (I'm using lp-shell)
<joey> the list size sinzui that it's traversing is 1518 if that helps at all
#launchpad-dev 2012-11-07
<StevenK> wallyworld__: *DB-*
<StevenK> You said stable, which is not right
<wallyworld__> shit
<StevenK> We can't fix it without reverting, so it's not worth it
<wallyworld__> it merged the right file though
<StevenK> Oh sure, it's just the commit message that's the problem
<wallyworld__> hopefully no one else will notice
<StevenK> wgrant will. Especially if I tell him first.
<wgrant> Heh
<StevenK> steven@undermined:~/launchpad/lp-branches% ls -1 | wc -l
<StevenK> 356
<StevenK> Hmmmm
<wgrant> I think you may have a problem.
<StevenK> /dev/mapper/sys-home  150G  105G   38G  74% /home
<StevenK> Not yet :-P
<wallyworld__> StevenK: ian@wallyworld:~/projects/lp-branches$  ls -1 | wc -l
<wallyworld__> 575
<StevenK> Haha
<wallyworld__> mine is bigger than yours :-P
<wallyworld__> StevenK: do you know where can look to see the preferred way to capture log messages or otherwise inspect what is logged during a job run so that i can write a test assertion?
<StevenK> Why are you asserting on log output, anyway?
<wallyworld__> the required job behaviour is to do a no-op and log a message
<wallyworld__> and i want to check tthat that occurs
<StevenK> wallyworld__: You probably want to create a logger with a Handler that backs onto StringIO
<StevenK> And then pass that in
<wallyworld__> yeah, i was hoping for an example to save me writing it :-)
<StevenK> BufferLogger from lp.services.log.logger
<lifeless> wallyworld__: StevenK: or the one in fixtures
<lifeless> you don't need to write it at all
<wallyworld__> thanks, will look
<lifeless> and the one in fixtures knows how to install itself and restore state afterwards
<wallyworld__> lifeless: so, BranchScan job does a "from lp.services.scripts import log" in it's run method. i guess i need to change that too?
<lifeless> shouldn't, no.
<wallyworld__> right, i'll read the code
<wallyworld__> thanks
<wallyworld__> StevenK: can haz quickie? https://code.launchpad.net/~wallyworld/launchpad/branchscan-deleted-branch-602323/+merge/133175
<StevenK> wallyworld__: Your test doesn't actually test anything
<StevenK> Oh
<StevenK> It's in the with, sneaky
<wallyworld__> yeah
<wallyworld__> i didn't even now that context manager existed
<wallyworld__> know
<StevenK> expected_message = 'Skipping branch %s because it has been deleted.' % (
<StevenK>     db_branch.unique_name,)
<lifeless> wallyworld__: you can use 'with foo as bar:'
<StevenK> Might work in terms of line length, too
<lifeless> so with try_advisory_lock(...) as lock:
<wallyworld__> can do
<wallyworld__> it will only have 1 or 2 loc, and i was copying an existing test
<StevenK> Refactor the other tests?
<StevenK> As a thought
<wallyworld__> for what purpose? they read fine as written
<StevenK> Save a few LoC?
<wallyworld__> a few? for what benefit?
<wallyworld__> if it were many, then sure
<wallyworld__> there's no readability issue or anything concrete that will benefit from changing thing here i don't think
<StevenK> wallyworld__: That's one difference between you and me -- I will refactor to save a few LoC, and you will not.
<wallyworld__> i will if there's a decent gain. but otherwise it make the mp diff harder to read and achieves very little benefit
<StevenK> wallyworld__: I don't think lines 62 and on don't need to be in the outer with
<wallyworld__> yes, good pickup
<wallyworld__> i saved 1 loc in the job method \o/
<wallyworld__> lp is now fixed
<StevenK> wallyworld__: This means you can probably reflow expected_message
<wallyworld__> sorry, too sarcastic
<StevenK> wallyworld__: Your review is now 'Needs Attitude Fixing'
<StevenK> :-P
<wallyworld__> hah
<wallyworld__> at least i leave no ambiguity as to my philosophy
<wallyworld__> sadly expected_message still neews 2 lines
<lifeless> shoot on sight ?
<StevenK> wallyworld__: Better than 3
<wallyworld__> it was only 2
<StevenK> No, it was three
<StevenK> 62	+ expected_message = (
<StevenK> 63	+ 'Skipping branch %s because it has been deleted.'
<wallyworld__> lifeless: shoot what?
<StevenK> 64	+ % db_branch.unique_name)
<wallyworld__> ah, sorry, still 2
<wallyworld__> 3
<wallyworld__> *3*
<StevenK> wallyworld__: Shoot the messenger
<wallyworld__> StevenK: https://pastebin.canonical.com/77799/
<wallyworld__> i changed the existing "lock = foo..." to "with foo():" to save 1 line
<lifeless> wallyworld__: your philosophy, is it 'shoot on sight' ?
<StevenK> wallyworld__: Yes, I got that :-)
<wallyworld__> lifeless:  shoot the messenger?
<lifeless> wallyworld__: forget it
<wallyworld__> sorry, slow today
<StevenK> wallyworld__: I guess the expectedLog really only needs to be around the job.run
<StevenK> Today?
<StevenK> What's your excuse the rest of the time?
<wallyworld__> well, always
<wallyworld__> i could move it down a few lines
<wallyworld__> StevenK: i made that change and pushed
<StevenK> Ah ha, I think I've worked out how to refactor this lot.
<StevenK> wallyworld__: r=me
<wallyworld__> StevenK: thanks.
<StevenK> wallyworld__: You have broken buildbot.
<wallyworld__> yeah - fixing. stupi celery
<wallyworld__> apparently it is illegal to get the branch unique name in the job init when running with celery
<StevenK> wallyworld__: https://code.launchpad.net/~stevenk/launchpad/local-codeimports-bad/+merge/133183
<StevenK> wallyworld__: "[testfix][r=wallyworld][no-qa] Test fix for r16242 - rever" Huh?
<wallyworld__> StevenK: looking now
<StevenK> And here I was, about to reboot so I can play HL2.
<wallyworld__> StevenK: r=me
<wallyworld__> now you can play :-)
<wallyworld__> my fingers sure messed up that commit message
<StevenK> Haha
<StevenK> wallyworld__: I can't, I have to pick up my wife soon :-)
<wallyworld__> hmmm. HL2 or wife
<wallyworld__> tough choice
<StevenK> I didn't know you'd played HL2?
<wallyworld__> i haven't but i am not ignorant of its existance
<adeuring> good morning
<wallyworld__> flacoste: hi, i'll be on leave to go to the cricket on friday, can we do it today?
<flacoste> wallyworld__: are you available now?
<flacoste> wallyworld__: or in the next 1h15?
<flacoste> wallyworld__: it's my wife's birthday today, so coming back in the evening is not an option ;-)
<wallyworld__> i have to take lachie to school in 10 minutes and have my standup in 1 hour. can i ping you after that? say in about 2hrs?
<wallyworld__> ah ok
<flacoste> wallyworld__: do you have any time between the school run and your stand-up?
<wallyworld__> what about sat morning my time?
<wallyworld__> i have 30 minutes
<flacoste> wallyworld__: that should be plenty
<wallyworld__> in about 45 minute's time
<wallyworld__> ok
<flacoste> wallyworld__: ping me when you're available
 * wallyworld__ nods
<flacoste> wallyworld__: that's better than sat morning, will the cricket game even be finished by then ? ;-)
 * flacoste likes hockey 2 hours and your done
<czajkowski> flacoste: yes but with hockey you get really serious injuries!
<czajkowski> cricket they just get sun burnt!
<flacoste> lol
<czajkowski> flacoste: I went to one game when I was over in Canada 3 years ago, and the amount of nose bleeds was unbelievable , not even in rugby is it that messy
<flacoste> czajkowski: yeah, hockey is a violent game
<flacoste> especially in North America
<rick_h> deryck: http://www.mikechambers.com/blog/2011/07/20/timing-issues-when-animating-with-css3-transitions/ and the last comment got me going http://jsfiddle.net/jyjXN/7/
<rick_h> deryck: just as an fyi spread the knowledge.
<bigjools> hello folks
<rick_h> howdy bigjools
<rick_h> bigjools: so was that starts in the plane a sunroof glass kind of thing or was it projected/faked?
 * rick_h goes completely off topic 
<bigjools> rick_h: LEDs in the ceiling
<rick_h> ah, less impressed now :(
<bigjools> picture didn't do it justice though
<bigjools> very well done
<bigjools> first plane I've been on with orange lighting too!
<deryck> rick_h, interesting.  I guess setTimeout is unavoidable.  But I guess that makes sense when we're dynamically constructing nodes.
<rick_h> deryck: that last comment about forcing a repaint by resetting display on the DOM node works
<rick_h> so it's ugly, but not set timeout interval and should work peachy
<czajkowski> bigjools: how long was the flight ?
<rick_h> see in the jsfiddle: dom_node._node.style.display = document.defaultView.getComputedStyle(dom_node._node)['display'];
<bigjools> czajkowski: "too"
<deryck> rick_h, yeah, that's better than the setTimeout.
<wallyworld__> flacoste: google+ hangouts is screwing me over - it keeps wanting to install the google talk plugin over and over even though the hangout initially starts ok
<flacoste> wallyworld__: want to use skype?
<wallyworld__> ok
<sinzui> wallyworld__, StevenK, jcsackett, I will be 30 minutes late to our meeting. Feel free to post pone it.
<wallyworld__> i have no problem starting 30 minutes late
<jcsackett> fine by me.
<StevenK> Okay, that's fine.
#launchpad-dev 2012-11-08
<StevenK> BLEH
<StevenK> I wanted buildbot done when I got back from lunch, but it failed PQM due to testfix
 * StevenK stabs testfix
<StevenK> wallyworld, wgrant: The Top 10 Durations make me very sad. Product, Bug, Bug, Bug, Bug, Bug, Bug, Bug, Distribution, Bug
<wallyworld> there's more to do
<StevenK> wallyworld: It's LP. There is *always* more to do.
<wallyworld> yep
<adeuring> good morning
<dimitern> adeuring: morning :)
<StevenK> adeuring: Could you please please get Aaron to do his QA today? It's been holding up deployments for two days.
<adeuring> StevenK: sure
<smartboyhw> Hi guys, is the on call reviewer rick_h here?
<czajkowski> smartboyhw: nope he's stateside atm
<czajkowski> think thats left over from yesterday
<smartboyhw> czajkowski, ouch. Want to fix a small bug
<smartboyhw> Very small, just a typo thing
<czajkowski> adeuring: know who is the on call reviewer today ?
<adeuring> czajkowski: nobody right now: https://dev.launchpad.net/ReviewerSchedule
<czajkowski> adeuring: cheers
<adeuring> smartboyhw: I am a reviewer too. what's up
<smartboyhw> adeuring, bug 978002
<_mup_> Bug #978002: Incorrect wording in 'Your Launchpad Karma'. <karma> <lp-answers> <trivial> <Launchpad itself:In Progress> < https://launchpad.net/bugs/978002 >
<smartboyhw> very small one:P
<czajkowski> adeuring: thanks
<adeuring> smartboyhw: As Curtis noted in a bug comment, you patch unfortunately does not fix the problem in production
<smartboyhw> oh?
<adeuring> We need to manually run an SQL command on the production database
<smartboyhw> adeuring, eh
 * smartboyhw goes crying
<smartboyhw> Let me go find another bug to fix then
<adeuring> smartboyhw: sure, sounds good. if you have any qestion, feel free to ping me
<StevenK> smartboyhw: I will mention that bug to Curtis when I speak to him, too, since he hasn't done it.
<smartboyhw> adeuring, StevenK thanks
<ev> Hiya. Would someone please create a project group for me: https://answers.launchpad.net/launchpad/+question/213666
<czajkowski> ev: sure
<ev> czajkowski: cheers
<czajkowski> ev: can you pm me a project group summary and a description please.
<rick_h> heh, sorry that was Tues
<czajkowski> rick_h: morning
* rick_h changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~200
<rick_h> morning czajkowski
<czajkowski> rick_h: whats the state of the nation over there with the electricity
<rick_h> czajkowski: came back in the afternoon and all the clocks reset and hopefully the UPS is charged back up
<rick_h> kind of strange, we don't lose power much here
<czajkowski> sweet
<czajkowski> back home home it does reguarly
<czajkowski> for hours, and nobody really does the back up generator
<czajkowski> not sure why
<czajkowski> remember one christmas it was gone for 6 days
<czajkowski> pita
<rick_h> ouch
<rick_h> sucked just to lose it for half a day
<rick_h> I think 6 days would kill off the fish
<czajkowski> most of the country was hit that year.
<dpm> hi Launchpad devs, what's the best way to reassign ownership of a branch in LP from an individual to a team?
<bac> rick_h: why did the power go out?
<rick_h> bac: no idea. The website showed it hit a couple sqmi but no reason I could find
<rick_h> so guess something just went boom
<czajkowski> ev: almost there
<bac> rick_h: a friend of mine had a *huge* salt water aquarium with lots of exotics and coral.  lost power for 7 days and they were all dead.
<czajkowski> just having issues with the name
<czajkowski> :/
<bac> he went out and bought a $10K generac as soon as possible
<rick_h> bac: yea, I think this was our first outage of more than 10min in years so got me thinking about the squarium yesterday
<czajkowski> dpm: hmmm
<czajkowski> unsure
<rick_h> freshwater though, so not that much $$ into it
<czajkowski> rick_h: any ideas re dpm
<rick_h> but still a couple grand
<rick_h> czajkowski: dpm I'd just take the branch and push it to the team if you've got access.
<rick_h> bzr push lp:~team/project/branch
<dpm> ok, let me try that
<dpm> thanks
<rick_h> dpm: off the top of my head, errors may ensue :)
<czajkowski> bah naturally the first time I go to create a project group it wont work smoothly
<czajkowski> keep having an issue at Invalid name 'ubuntu error tracker'. Names must be at least two characters long and start with a letter or number. All letters must be lower-case. The characters +, - and . are also allowed after the first character.
<czajkowski> but cant se any other project with the name so no idea why it's clashing :/
<czajkowski> ev: I'm gonna wait till sinzui comes online to ask him but it'll be done in the next few hours.
<dpm> rick_h, czajkowski, pushing the team branch and then setting the location in the project page worked, thanks!
<rick_h> dpm: awesome, glad to hear it
<czajkowski> dpm: lovely jubbly
<czajkowski> now if all questions could be as simply solved today that would also be great
<czajkowski> I know full well sinzui is gonna come along and say someting very simple as to why I cannot create this project group
<czajkowski> but for the life of me I've no idea why :/
<ev> czajkowski: thanks
<czajkowski> Grabbing lunch bbiab
<ev> thanks czajkowski, sinzui
<czajkowski> sinzui: you did it ?
<czajkowski> bah wanted to learn why it wasnt doing it for me :/
<sinzui> the form requires a summary and description, and since none was provided in the question, I pasted the display name
<sinzui> czajkowski, We don't uses title, but it too is required, so we paste that name in there
<czajkowski> yes but I had that pasted in
<czajkowski> I pasted the name in there also :/
<sinzui> what error did you see? the form will usually state there is a form error?
<sinzui> czajkowski, repeat what you did here: https://qastaging.launchpad.net/projectgroups/+new
<czajkowski> ok
<czajkowski> sinzui: same issue
<czajkowski> Invalid name 'ubuntu error tracker staging'. Names must be at least two characters long and start with a letter or number. All letters must be lower-case. The characters +, - and . are also allowed after the first character.
<czajkowski> A unique name, used in URLs, identifying the project group. All lowercase, no special characters. Examples: apache, mozilla, gimp.
<czajkowski> wit the naming
<sinzui> name is launchpad-id
<sinzui> ubuntu-error-tracker
<czajkowski> ahhhhh
<czajkowski> https://qastaging.launchpad.net/launchpad-id
<sinzui> ev, wrote the display name and launchpad-id in reverse of what Lp does
<czajkowski> *headdes*
<czajkowski> ok
<czajkowski> will know for future
<czajkowski> couldnt see what the issue was
<czajkowski> sinzui: thanks for the help
<deryck> Morning, all.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: ~200
<tim_> sinzui, any plans on folding blueprints into the new sharing stuff?
<sinzui> tim_, deryck's squad have already done that
<sinzui> tim_, the porject's +sharing page lets you set the blueprint sharing policy
<tim_> sinzui, awesome
<deryck> yup, just need to be in beta testers group.
<tim_> I think I'm in that group
<adeuring> jcsackett: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/product-lp-limitedview-2/+merge/133481 ?
<rick_h> deryck: ping
<deryck> rick_h, yo
<rick_h> deryck: hey so wiring this into blueprints and noticing the fun part.
<rick_h> so if the project is private we show the privacy banner
<rick_h> and so when you go to create a new blueprint the privacy banner is already showing
<jcsackett> adeuring: you got it.
<rick_h> while the type shows public at the moment to start with
<adeuring> jcsackett: thanks!
<rick_h> have we talked about how we want banners to work in mixed cases?
<deryck> rick_h, I think that's correct, right?  If you have a proprietary or embargoed project, the artifacts should be that by default anyway, which means we start with the banner.  right?
<rick_h> right, maybe I've just got my self into a strange project setup state.
<rick_h> ok, will double check the blueprints widget since it shold have defaulted to non-public since blueprints were configured in sharing to be proprietary only
<deryck> rick_h, we may not be consistent in this yet, see bug 1076412 that I just filed.  But what you described sounds like what we expect.
<_mup_> Bug #1076412: Proprietary projects should only allow proprietary policies on its artifacts <private-projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1076412 >
<rick_h> deryck: yea, I see what I did. I have a prop project and I setup in sharing to have blueprints that were public, but could be prop.
<rick_h> deryck: so I just got thrown off when I saw the privacy banner, but the information type widget was defaulting to public
<jcsackett> adeuring: r=me.
<czajkowski> cjwatson: ello any idea when Bug 1071562  might get a look at ?
<_mup_> Bug #1071562: UEFI signing failures cause binaries to be republished continuously <oops> <regression> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1071562 >
<adeuring> jcsackett: thanks!
<cjwatson> czajkowski: yeah, sorry - I plan to get to it tomorrow
<czajkowski> cjwatson: excellent thanks colin
<deryck> adeuring, I added a hangout to the calendar event for our call.
<adeuring> deryck: coming...
<benji> bac: sure; when will my new one be?
<benji> pfft
<jcsackett> sinzui (or anyone else) any idea where bzr branches go when pushed to lp://dev on the local filesystem?
<jcsackett> abentley, you might know? ^
<abentley> jcsackett: I think it's /var/tmp/bazaar.launchpad.dev/mirrors
<abentley> jcsackett: it will be in a numbered subdirectory in there.
<jcsackett> abentley: hrm. i looked there earlier i thought.
<jcsackett> oh! i didn't check dotfiles.
<jcsackett> just checked those in the cache...
<abentley> jcsackett: The numbered subdirectories are derived from the hex representation of the branch db id.
<jcsackett> abentley: ah, dig. thank you. :-)
<abentley> jcsackett: np.
<jcsackett> sinzui: i am running to my appt. shall i set you as OCR in the title?
<jcsackett> sinzui: i'll review your branch as soon as i can.
* sinzui changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call sinzui | Firefighting: - | Critical bugs: ~200
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: jcsackett | Firefighting: - | Critical bugs: ~200
<jcsackett> sinzui: i am looking at your branch. i am a little confused about what's going on in line 62 of your diff.
 * sinzui looks
<jcsackett> sinzui: it looks like you call get_msg_ids_to_remove, which calls _get_msg_ids...; but then you call _get_msg_ids again on the return?
<sinzui> Yes, by passing a subset of ids, the method will filter out those that became used inbetween transaction calls
<sinzui> The list of ids is cached and it contains duplicates
<jcsackett> ah, ok.
<jcsackett> just making sure i was reading things right.
<sinzui> Since the cache is can be invalid the moment we call commit() at the end of the loop, we need to ensure we still only have unused potmsgset ids
<jcsackett> dig. r=me, sinzui.
<sinzui> thank you
<sinzui> jcsackett, do you have a few minutes to talk about this http://paste.ubuntu.com/1343678/
<sinzui> its in regards to bug 872335
<_mup_> Bug #872335: TypeError raised by rosetta-poimport.py script <lp-translations> <oops> <rosetta-imports> <Launchpad itself:Triaged> < https://launchpad.net/bugs/872335 >
<czajkowski> sinzui: is it possible to have private project groups, never heard f them but am getting all sorts of unusal requests today
<sinzui> no, we have no plans to make them since project groups are broken by design
<czajkowski> nods
<czajkowski> grand
<sinzui> They don't work for organizations or communities
<sinzui> We would remove them if we had time
<czajkowski> sinzui: that might upset a lot of people
<sinzui> They are upset now that they don't work
<sinzui> Organisations cannot trust them since anyone can place their project into your group, and since s project cannot be in more than one group, communities cannot share and plane.
<czajkowski> nods
<czajkowski> logically it makes sense
<lifeless> flacoste: o/
<sinzui> StevenK, jcsackett: http://paste.ubuntu.com/1343678/
<StevenK> sinzui: http://pastebin.ubuntu.com/1343825/
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: - | Firefighting: - | Critical bugs: ~200
#launchpad-dev 2012-11-09
<flacoste> lifeless: o/
<StevenK> wallyworld: Your rev is qa-bad. It doesn't work, but it doesn't hurt either.
<lifeless> StevenK: isn't that qa-ok
<StevenK> Well, I'll be marking it as such
<StevenK> We need a tag for failed QA but safe to deploy
<wgrant> StevenK: That's qa-ok
<wgrant> qa-FOO refers to deployment safety
<wgrant> Nothing about functionality of the change itself
<adeuring> good morning
<czajkowski> morning
<czajkowski> Does anyonek know anything about loggerhead bugs ? https://bugs.launchpad.net/loggerhead/+bug/1075907
<_mup_> Bug #1075907: Loggerhead - serving two directories from parent <loggerhead:New> < https://launchpad.net/bugs/1075907 >
<mgz> czajkowski: not really.
<czajkowski> mgz: nobody seems to be :(
<rick_h> czajkowski: so I'll confirm the bug as abently and I hit it when working on helping the guy with loggerhead JS stuff back on wed
<rick_h> czajkowski: but no idea tbh as to the fix
<rick_h> czajkowski: I can say the whole profile thing is just a side show. It's a tool to help debug performance and it's not that, just isn't working atm
<czajkowski> rick_h: ahhh
<czajkowski> thank you
<jcsackett> morning, all.
<rick_h> morning jcsackett
* bac changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: bac | Firefighting: - | Critical bugs: ~200
<bac> hi jcsackett
<rick_h> deryck: abentley just want to make sure this encapsulates/makes sense to others https://bugs.launchpad.net/launchpad/+bug/1076540/comments/1
<_mup_> Bug #1076540: branch +edit doesn't load picker JS for pretty UI elements <javascript> <ui> <ui-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1076540 >
<deryck> rick_h, fine with me.  just meant to capture the idea, so it works for that.
<abentley> rick_h: +1.
<abentley> rick_h: Just had a thought that maybe we could add that functionality to the base template, instead.
<rick_h> abentley: ah ok, and do a conditional if the attribute is defined add the YUI block?
<abentley> rick_h: Right.
<abentley> rick_h: For ultra-mega-consistency.
<rick_h> abentley: yea, that makes sense and makes for even less new code/changes
<rick_h> noted in a follow up comment
<abentley> bac: Could you please review https://code.launchpad.net/~abentley/lp-dev-utils/cs-test/+merge/133712 ?
<bac> abentley: i'll be glad too after this meeting and lunch
<abentley> bac: Thanks.
<sinzui> bac, do you have time to review https://code.launchpad.net/~sinzui/launchpad/poimport-typeerror/+merge/133713
<bac> sinzui: i'll be glad too after this meeting and lunch
<bac> s/too/to
<sinzui> thank you
<bac> abentley: how long do these tests take on canonistack?
<sinzui> enail?
<sinzui> thanka bac
<sinzui> I think I want enails. With programmable nails I could change the colour or pattern in seconds
<jcsackett> sinzui: ooh! you could have fractals at that point. :-P
<abentley> bac: ~391 minutes.
<abentley> so 6:31
<bac> speedy
<bac> abentley: i guess lpsetup takes a long time too since it has to fetch the whole lp branch.  ie it isn't preseeded in the ami like our ec2 runs, correct
<abentley> Well, the whole install takes about 10 minutes.
<abentley> So it's not a big percentage, so far.
<abentley> But m1.small are multi-core, so parallel-testing should be possible in theory.
<abentley> Using a larger instance type would also help.
<bac> abentley: i guess since we're in the data center the fetch of the branch is fast
<abentley> bac: I checkout, rather than fetch the branch, so it's just creating the working tree.  It's about a minute.
<bac> abentley: are you providing instructions for use anywhere?  perhaps with reference to canonistack setup page.
<abentley> bac: I just wanted to get the code out there, to start, so that we can iterate on it.
<bac> abentley: ok.
<bac> abentley: done
<bac> thanks
<abentley> bac: Thanks you!
<abentley> bac: Could you please review https://code.launchpad.net/~abentley/launchpad/optimize-spec-query/+merge/133716 ?
<abentley> bac: The code isn't actually obsolete yet.  I haven't moved every get_specification_privacy_filter call over to visible_specification_query yet.
<bac> abentley: ok.  then please DO NOT remove it after QA.  :)
<bac> sorry for the misread
<abentley> bac: roger.
<abentley> bac: Thanks for the review.
<rick_h> jcsackett: or sinzui I've got a series of very JS branches coming for the updated banner work and would love to get some more UI special reviews for these if you don't mind. https://code.launchpad.net/~rharding/launchpad/add_new_banner/+merge/133736 is the first branch that adds the new code
<jcsackett> rick_h, sinzui: i can take it, as long as about 30 min delay is alright.
<rick_h> yea definitely. No rush.
<rick_h> thanks jcsackett
<jcsackett> cool.
<sinzui> rick_h, I think you would enjoy this when you get a break. It's David Mitchells rant on modern furniture: http://www.youtube.com/watch?v=syii9DKnb2M
<rick_h> sinzui: <3 on all accounts
<jcsackett> rick_h: is this still going to work with the static html version for no javascript browsers? To my knowledge we still have to support that case.
<rick_h> jcsackett: no, it doesn't currently render w/o JS
<rick_h> jcsackett: didn't realize that was a requirement for the banner since it would fail to work on the informatino type change w/o JS
<jcsackett> rick_h: yeah, we accepted just at least having the banner on reload, since no one expects on the fly change when there's no JS.
<rick_h> k, thinking of how to try to incorperate that without duplicating the html template and such in multiple places
<rick_h> and ugh :P
<jcsackett> sinzui: do we still have a limited case for needing the banner on no-js browsers?
<jcsackett> rick_h: i'm sorry i didn't notice this in your last sanity check--i wasn't thinking then. :-/
<sinzui> YES
<jcsackett> that was emphatic. :-p
<sinzui> LTS headless requires it
<rick_h> heh, so that corrects my misconception that there was a limited set of features for non-js such as filing a bug and such which banners wouldn't get involved with
<abentley> rick_h: You could use mustache-- I added it specifically so we could use the same template for HTML and js.
<rick_h> sinzui: LTS headless?
<rick_h> abentley: yea, the tempalte was small enough I didn't use mustache so will look that route perhaps
<jcsackett> rick_h: the banners don't need to do hot updates or even be totally accurate to infotype. i think a banner of "information on this page is restricted by info type" that never changes or the like satisfies the need.
<rick_h> but you still end up with collisions of message text generatoin and such
<jcsackett> sinzui, that sound right? ^
<rick_h> jcsackett: yea, that makes sense. I noticed they have more vague default messages
<sinzui> precise will live for years with old browsers, and browsers from servers that need to do a confidential actions need to know the action is trusted
<rick_h> but yea, didn't think banners were in any way functional and required to work non-js
<jcsackett> rick_h: yeah, the "stuff here is hidden, somehow".
<jcsackett> i *think* the boring html version is already there; you may be fine.
<jcsackett> at least this branch may be--the wiring just needs to not kill it.
<rick_h> jcsackett: well I rip it out in order to remove duplication and making it easier to edit/adjust
<rick_h> jcsackett: so yea, it's something I'll have to hea back with and think on for a minute
<jcsackett> rick_h: right, i think you can not rip it out and make sure the banners use the built in HTML_PREPROCESSOR so your other stuff works.
<jcsackett> *this* branch though i don't believe is impacted.
<rick_h> jcsackett: true, this branch is totally unconnected atm
<jcsackett> again: i'm sorry i didn't notice this when i last looked.
<rick_h> but if I move to mustache then the generation here will alter
<rick_h> though not in any killer way
<jcsackett> rick_h: i'm approving this branch, at least. you may need to do further changes against it in wiring, but the new modules are all good.
<jcsackett> so r=me on the first one.
<rick_h> jcsackett: ok, thanks
<jcsackett> rick_h: no problem.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On Call Reviewer: - | Firefighting: - | Critical bugs: ~200
<abentley> deryck: starting to work on bug #1076412.  Quick pre-imp?
<_mup_> Bug #1076412: Proprietary projects should only allow proprietary policies on its artifacts <private-projects> <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/1076412 >
<deryck> abentley, I have a couple things I'm trying to finish before EOD.  Wanna wait until Monday? Or maybe chat with rick_h about it?
<abentley> deryck: The basic thing is we're going to draw a line between Poprietary and Embargoed and Public types, WRT sharing and product info type.
<abentley> deryck: So Embargoed product with proprietary bugs is okay, and vice versa.
<abentley> Right?
<deryck> abentley, right.
<abentley> deryck: Great.
#launchpad-dev 2012-11-10
<yofel> hi, does someone know if I can enable/disable a release in a daily build recipe over the API? I tried:
<yofel>     recipe.distroseries.append(ubuntu.getSeries(name_or_version="raring"))
<yofel>     recipe.lp_save()
<yofel> wrong one
<yofel>     recipe.distroseries.append(ubuntu.getSeries(name_or_version="raring").self_link)
<yofel> but lp_save() doesn't do anything
#launchpad-dev 2012-11-11
<cjwatson> Hmm, apparently r16253 left buildbot broken over the weekend :-(
<czajkowski> oops
<czajkowski> cjwatson: afternoon
<cjwatson> Hi.  Not here for long; pub lunch beckons.
<czajkowski> enjoy
<wallyworld_> wgrant: any ideas you have for speeding up that garbo job (PopulateLatestPersonSourcepackageReleaseCache) would be great. i essentially just replicated the live query but added in maintainer or creator to the mix. perhaps i should drop out maintainer/creator from the distinct and select all distinct archive/series/spn records into a temp table and then requery the temp table?
<wgrant> wallyworld_: Well, I'd probably just slice into the recent pubs, pull out their data, grab the relevant cache records, and compare the times.
<wgrant> We coalesce non-distinct records into a single row in the cache table based on the timestamp anyway, so the DISTINCT ON in the query is pointless
<wallyworld_> so a simple query order by date desc
<wgrant> Right
<wallyworld_> ok, i'll rework it and see how it turns out
<wgrant> I'm not sure I understand what purpose your SPR watermark serves today
<wgrant> I'd just slice into the SPPHs, grab the SPRs and their earliest publication IDs, exclude any SPPHs that aren't the latest publication, grab the relevant cache rows, update any cache rows that need updating
<wallyworld_> to limit the sprs considered
<wallyworld_> earliest publication ids?
<wgrant> s/latest publication/earliest publication/
<wallyworld_> s/earliest publication/latest publication/ maybe?
<wallyworld_> since we only want the most recent ones
<wgrant> Don't we only care about the first publication of an SPR?
<wgrant> The latest publication for the cache key, but the first publication for the SPR
<wallyworld_> um. perhaps i misunderstand the data model
<wgrant> What is your understanding of it?
<wallyworld_> each time a source package is published, a spr is created and also a spph record is created
<wallyworld_> ?
<wgrant> No
<wallyworld_> :-(
<wgrant> A source may be copied, in which case there's a new SPPH but no new SPR
<wallyworld_> when does a copy occur and where it is copied to?
<wgrant> when someone requests that a copy happens :)
<wgrant> It may be copied to anywhere
<wgrant> And archive, any distroseries, any pocket
<wallyworld_> uploaded_archive is the original archive published to, not any copied one, right?
<wgrant> Right
<wallyworld_> so, the main garbo job query could be on spr?
<wallyworld_> is each record in spr table known/guaranteed to be published?
<wallyworld_> if so, why the join to spph?
<wgrant> Because we need to only show SPRs that have ever had an SPPH
<cjwatson> Did you folks notice the buildbot failure?  I guess Abel won't be around for a while yet
<wgrant> Blah
 * wgrant reverts
<wallyworld_> wgrant: so when would a spr record be created when there is no publication?
<wgrant> wallyworld_: When it's waiting in a queue to be approved
<wgrant> Or rejected
<wallyworld_> ok
<wallyworld_> can i easily tell which spph are for copied sprs?
<wallyworld_> to filter out those in any spph query?
<wgrant> No
<wgrant> But even if you could it probably wouldn't be worth it
<wgrant> The extra expense in filtering afterwards is going to be minimal
<wgrant> Because we have the SPPH.id watermark
<wallyworld_> so the main query should be on spph joined to spr?
<wgrant> I think the set of three SELECTs I outlined earlier is reasonable
<wgrant> You could possibly grab the SPRs in the first SELECT, but it doesn't really matter
<wallyworld_> ok, will see what falls out, thanks
#launchpad-dev 2013-11-04
<cjwatson> wgrant: Sort of.  I was looking into how that kind of thing works to figure out whether I have a hope of understanding it well enough to extend it.
<wgrant> cjwatson: Ah, probably not worth you doing that...
<cjwatson> Right, wasn't sure :)
<cjwatson> It was going to be a plane project since I had nothing else to do, but then I unaccountably slept instead.
<wgrant> It's reasonably horrific, and there's not a whole lot of code to be written.
<wgrant> !
<wgrant> How awful.
<cjwatson> I think the slave side is reasonably ready now, in any case.  I don't remember what we decided we needed to improve in longpoll-land ...
<cjwatson> Oh, though I should do a bit more work on failure handling
<cjwatson> Though maybe not worth it, recipes don't have any special handling for "fundamental recipe-building packages not installable"
#launchpad-dev 2013-11-07
<bigjools> Every time I add or change work items on a blueprint, it discards all the data the first time.
<StevenK> bigjools: All data, or all changes?
<bigjools> StevenK: changes
<bigjools> so if I start at nothing, type a load, and hit the ok button, the lot gets lots
<StevenK> work items backs on a cached property, so it's possible something isn't invalidating
<bigjools> lost*
<bigjools> I can paste the same thing in and next time it sticks
<StevenK> Bleh, ISpecification.updateWorkItems does delete work_items from the property cache
#launchpad-dev 2013-11-09
<xnox> PPA builders seem to have odd balancing: it's equal split between i386 & amd64, yet there are more jobs that are run on i386-only
<cjwatson> xnox: We rebalance manually based on queue length
<cjwatson> xnox: It's not even a little bit static :-)
<cjwatson> xnox: (The correct fix is builder pooling, so that the multiple-architecture-capable builders just suck down whichever build is next regardless of architecture)
<wgrant> Hopefully xnox won't see that for at least a couple of days :P
<cjwatson> as in because he's hopefully recovering from DC action last night?
<wgrant> Indeed.
<cjwatson> I read the IR, though it kind of petered out
<wgrant> He ended up leaving around 6 hours ago.
<cjwatson> Sounds grim
<xnox> yeah, it wasn't the most efficient visit =] but it was fun.
#launchpad-dev 2014-11-03
<kb9vqf> wgrant: ping
<kb9vqf> I have the production system upgraded to the latest source and have been squashing remaining bugs but the build farm itself is giving me quite a bit of grief
<kb9vqf> I have a test package that's built but stuck "Uploading build"
<kb9vqf> If I understand correctly this means the builder already uploaded the built files but the script responsible for publishng them is not running properly?
<kb9vqf> I thought process-accepted.py was responsible for publishing the uploaded build files, but now I am not so sure
<kb9vqf> Help? :-)
 * kb9vqf is also trying to reenable the build farm on non-virtualized builders...seems quite a lot changed over 4 years
<kb9vqf> Hmmm...finally came across a page detailing some of the upload process (https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally).  I think poppy isn't even receiving the packages; let me dig a bt further
<wgrant> kb9vqf: Ah, I forget when some of this changed.
<wgrant> kb9vqf: build uploads don't actually use poppy.
<kb9vqf> Ah, ok.
<kb9vqf> Where do they go?
<kb9vqf> I don't even see any log files
<wgrant> They should end up automatically somewhere like /var/tmp/buildmaster/incoming
<wgrant> That path will be logged in the buildd-manager log.
 * kb9vqf goes to look
<wgrant> alphecca.canonical.com-lp_buildd:* * * * * /srv/launchpad.net/codelines/current/scripts/process-upload.py -C buildd --builds /srv/launchpad.net/builddmaster/ -q --log-file=DEBUG:/srv/launchpad.net/production-logs/process-build-uploads.log
<wgrant> That's how we process them on production.
<kb9vqf> at least the builder is sort of working: "Grabbing file:"...
<wgrant> That changed a couple of times in 2010.
<wgrant> If it got to "Uploading build", the builder worked fine and it's in the upload queue.
<kb9vqf> ok, let me check a couple things real quick
<wgrant> You just need to find where the upload queue is, and ensure process-upload -C buildd --builds is running over it.
<kb9vqf> yeop, my process-upload is running againast poppy
<kb9vqf> *against
<wgrant> You need both.
<kb9vqf> gotcha
<wgrant> One with -C insecure for the untrusted source uploads from poppy, and one with -C buildd --builds for the trustworthy binary uploads from buildd-manager.
<wgrant> (though -C insecure might be the default; I forget)
 * kb9vqf confirms the second was missing
<kb9vqf> the 2010 timeframe sounds right
<wgrant> buildd-manager used to do that inline, but it doesn't scale very well when you're processing jobs from 150 buildds.
<wgrant> So it just writes them to a normal upload queue and waits for a cronjob nowadays.
<kb9vqf> that should help--I upgraded because the build farm was not scaling well
<kb9vqf> so glad to see that improvements were made on that end :-)
<kb9vqf> Any ideas though on what to do about this error: Scanning eostre failed with: Allegedly clean slave not idle ('BuilderStatus.WAITING' instead)
<kb9vqf> I can't virtualize the farm, and am contemplating hacking the buildd-manager to force it to work on my setub, but want to make sure there isn't  flag I can set instead
<kb9vqf> wow my keyboard is behaving badly tonight
<kb9vqf> sorry :-P
<wgrant> Hmmm.
<kb9vqf> basically it builds one package and uploads it, cleans the slave and then dies
<wgrant> Yeah
<wgrant> So all your builders are marked virtualized in Launchpad, but aren't actually reset between builds because they're real hardware?
<kb9vqf> Right
<wgrant> We used to do that for ARM too, but not any more so I added some safety checks.
<kb9vqf> Yep arm here is the problem
<wgrant> You have two options.
<wgrant> a) Set all the builders and the relevant PPAs to non-virtual, and hack the database to ensure the existing builds are non-virtual (trivial).
<wgrant> Or b) Adjust lib/lp/buildmaster/interactor.py's cleanSlave method to always use the non-virtual path.
 * kb9vqf thinks b) would be best as a) causes user-visible changes
<wgrant> The virtualized path is optimised to not bother cleaning after a build, since obvious a VM reset is going to result in a clean VM.
<kb9vqf> you guys still using Xen?
<wgrant> We switched to an KVM+OpenStack solution in August.
<kb9vqf> ok, that sounds better
<kb9vqf> kvm works on arm IIRC so that might even be an option now
<wgrant> https://insights.ubuntu.com/2014/10/30/scalingstack-2x-performance-in-launchpads-build-farm-with-openstack/ outlines the old and new solutions.
<kb9vqf> great!  I'll give that a read
<wgrant> But I would recommend a) for your case. It shouldn't cause any user-visible changes unless you're doing very weird things.
<kb9vqf> first thing that happens is the build farm status page shows all the non-virtualized builders
<wgrant> Well, they just use a different icon and move to a slightly different place on the page.
<wgrant> Is that a problem?
<kb9vqf> switching the builders to non-virtual puts them into the "officia" build queue and leaves "0 builders" for PPAs
<kb9vqf> not a big problem, but ugly
<wgrant> Yeah, we've needed to change those descriptions for a while, but to avoid confusing users in the normal case we've left them that way.
<kb9vqf> actually if you're on KVM now I might look into the virtualization
<wgrant> You could easily hack the template to only show one set of stats, though.
<wgrant> Our ARM stuff is still unvirtualised.
<kb9vqf> It was a no-go on Xen due to arm but now...
<wgrant> ARM PPAs use qemu-user :/
<kb9vqf> oh, ok
<wgrant> But if you have supportable hardware, it works.
<kb9vqf> real arm hardware here
<wgrant> Sadly virt-capable server-class ARM hardware is just now coming available.
<wgrant> You need at least Cortex-A15.
<wgrant> An A8 or A9 will not do.
<kb9vqf> Cortex A9 here so no-go
<wgrant> Yeah :/
<kb9vqf> well looks like an A15 upgrade will be needed in the future then
<wgrant> We hope to move to a fully virtualized pool within a year or so.
<wgrant> Once we have enough virt-capable ARM hardware in the pool.
<wgrant> Currently the non-virtualised ARM buildds are quad-A9s.
<kb9vqf> I was hoping I could virtualize as I get tired of having the build pool go offline under heavy load
<kb9vqf> but oh wll
<kb9vqf> *well
<kb9vqf> Yeah, quad A9s here too
 * kb9vqf wonders if we're using the same chip :)
<kb9vqf> wgrant: upload accepted, thanks for the help! :-)
 * kb9vqf wonders if he should update that Wiki page
<wgrant> kb9vqf: Excellent.
 * kb9vqf reads the Wiki again and sees the info clearly there--must have been stuck on the old system too long :-P
<wgrant> kb9vqf: Oh, also, manage-chroot.py is no longer a script in the LP tree. It's an API client in lp:ubuntu-archive-tools.
<kb9vqf> that would have tripped me up; thanks for the info
<wgrant> And if you want to create non-Ubuntu PPAs, you need to use the API. The +active-ppa form still forces Ubuntu today.
<wgrant> s/active-ppa/activate-ppa/
<kb9vqf> wgrant: for now we'll just continue with our hack; no need to change whats working until the other solution is better
<wgrant> Yep.
<kb9vqf> wgrant: calling convention for the new manage-chroot API client is documented somewhere?
<wgrant> kb9vqf: Should be pretty much the same as before, IIRC. It should also be on that page, and in --help
<kb9vqf> ok, just thought I'd verify while you were on the line so to speak
<wgrant> You'll need to give it something like '-l https://api.quickbuild.pearsoncomputing.net/' or so as well, though.
<kb9vqf> sure, makes sense
<wgrant> The manage-builders script in the same tree is probably also of interest for bulk operations on the build farm.
<kb9vqf> that would be very nice--I was looking into writing something like that
<kb9vqf> wgrant: one more little question...I noticed the Google Maps stuff was ripped out.  We were using that and I kind of miss it; any chance of it coming back?  I filed a bug report on it yesterday.
<wgrant> kb9vqf: It was removed a few years ago because Google Maps was no longer unusable for various reasons. It would probably be possible to reimplement it on top of OpenStreetMap, but it's not something we're interested in today.
<kb9vqf> ok
<kb9vqf> wgrant: is there a document available that details the codehosting setup/how to get bzr imports working?  I am looking at activating it but immediately see references to internal Canonical machines running unknown software
<wgrant> kb9vqf: Codehosting is a fair bit of work to set up and maintain. I'd avoid it unless you have no other choice, particularly given that things will be radically changing there over the next few months.
<kb9vqf> OK, we can hold off.  Changing for the better I hope?
<wgrant> That's the plan.
<kb9vqf> :-)
<kb9vqf> What I'd really like to see is Launchpad's translation system interface with a GIT module, but I don't know if that's possible
<kb9vqf> s/module/repository/
<kb9vqf> Well I converted the database over to allow non-virtualized builds...test builder is active, let's see how it goes :-)
 * wgrant watches.
<kb9vqf> wgrant: Does the new PPA creation code force require_virtualized to True or does it use the database default?
<wgrant> kb9vqf: That hasn't changed. They always default to virtualized.
<wgrant> You can tweak it per-PPA on +admin, and should probably leave the default alone for security reasons, unless you really need to create lots of PPAs regularly.
<wgrant> Given that anyone who can log in can create a PPA.
<kb9vqf> wgrant: Right, I wasn't clear.  I mean since I changed the database to default to non-virtualized do I need to also change the code?  I have users that do create PPAs and I don't want to go into the database constantly to allow builds
<wgrant> kb9vqf: Yes. If you really want to change that default, despite the security implications, you'll probably want to change Person.createPPA to call ArchiveSet.new with require_virtualized=False
<kb9vqf> Thanks, that's exactly what I needed to know
<kb9vqf> I've been running it this way for all these years with no problems (I dealt with the security holes in the infrastrucutre)
<kb9vqf> *infrastructure
 * kb9vqf notes with some satisfaction that the previously uploaded test build has now been fully published to the archive.  Still waiting on non-virtualized build tests.
<wgrant> Excellent.
<kb9vqf> So aside from codehosting any other major changes coming down that I should be aware of?  I won't be using the bugtracker at any point as we have a well-entrenched Bugzilla but some of the other features are interesting
<wgrant> Nothing scheduled right now that should make your life hell.
<kb9vqf> Sounds good
<wgrant> Though there will probably be some Soyuz changes next year some time that might have ramifications, they'll be nothing that's terribly difficult to migrate to.
<kb9vqf> Ah, this wasn't hell, it was just a .... challenging .... upgrade with .... lots of downtime .... hmmm, on second thought let's not repeat it :-)
<wgrant> Though I am doing a BinaryPackageBuild schema change now that has a Python migration, but I'll include an SQL patch to do the same thing :)
<kb9vqf> thanks for thinking of the only other Launchpad instance; I appreciate it :-)
<kb9vqf> so far the non-virtualized test builder is working perfectly, even with build failures, so I'm going to cross my fingers and activate additional builders
<wgrant> Lovely.
<kb9vqf> base system looks to be running well (much faster than the old Launchpad instance actually); I'm going to sign off for tonight (wee hours of the morning here).  Thanks again!
<kb9vqf> wgrant: Just ran into a problem with a builder that was functional until the builds were accidentally interrupted
<kb9vqf> log now states "Build log: /home/buildd/.lp-sbuildrc is corrupt; builder needs repair work"
<kb9vqf> have you run into this before?
<kb9vqf> the file mentioned doesn't exist on the system
<kb9vqf> reinstalling the package fixed it, nevermind :)
#launchpad-dev 2016-11-07
<Pali> hi! it is possible to have mirror of git repository on launchpad?
<cjwatson> Pali: Very very soon
<cjwatson> Pali: The (hopefully) last set of necessary patches is in the review queue now
<Pali> as workaround bug https://bugs.launchpad.net/bzr-git/+bug/402814
<mup> Bug #402814: Importing revisions with submodules is not supported <feature> <git> <lp-code> <udd> <Bazaar:Triaged> <Bazaar Git Plugin:Fix Released by jelmer> <cloudfoundry:Invalid> <Launchpad itself:Triaged> <https://launchpad.net/bugs/402814>
<cjwatson> Pali: You can watch https://bugs.launchpad.net/launchpad/+bug/1638219
<mup> Bug #1638219: launchpad: remove headers from Contents files <api> <lp-soyuz> <qa-ok> <soyuz-publish> <trivial> <Launchpad itself:Fix Committed by cjwatson> <https://launchpad.net/bugs/1638219>
<cjwatson> Er, not that one
<cjwatson> Pali: You can watch https://bugs.launchpad.net/launchpad/+bug/1469459
<mup> Bug #1469459: import external code into a LP git repo (natively) <feature> <git> <qa-ok> <Launchpad itself:In Progress by cjwatson> <turnip:In Progress by cjwatson> <https://launchpad.net/bugs/1469459>
<Pali> I think I saw some git mirror support in launchpad API
<cjwatson> Some bits are there but it's not quite all landed
<Pali> I cannot subscribe to that bug :-(
<Pali> Timeout error
<Pali> cjwatson: when is expected that git mirror feature to be published?
<cjwatson> Pali: Should go away in 10 minutes or so, there's some kind of periodic database maintenance job that causes us occasional trouble
<Pali> ok, I will try in next 10 minutes
<cjwatson> I think it's OK now
<Pali> and another question, is dumb http transport for git going to be supported in that git mirror?
<Pali> because currently git-to-bzr import does not support it
<cjwatson> Pali: Should work pretty much as long as we can 'git clone' it
<Pali> ok
<cjwatson> Pali: If you give me an example I can make sure to QA it once we're ready
<cjwatson> Pali: when> next week or two, I expect
<cjwatson> I finished the last hard bit on Friday
<Pali> ok, going to find that git repo which I was not able to import on bzr launchpad
<Pali> cjwatson: this is that failed git-->bzr import: https://code.launchpad.net/~pali/ocl-icd/trunk
<Pali> "git clone https://forge.imag.fr/anonscm/git/ocl-icd/ocl-icd.git" is working fine on my local machine, but launchpad was unable to import it
<Pali> you can see failed logs
<cjwatson> right, I'm pretty certain that will be fine
<cjwatson> and if there is some minor bug then it will be much simpler to fix
<cjwatson> but I don't see anything there that would cause a problem
<cjwatson> the only hard repository-dependent bit really is syncing the HEAD symbolic ref across; everything else is just straightforward git operations
<Pali> and... after that all new git imports will be imported to git? or will there be option to import external git repos as bzr (like it is now)?
<cjwatson> Pali: You'll be able to choose, at least to start with
<cjwatson> https://code.launchpad.net/~cjwatson/launchpad/codeimport-git-configure-code/+merge/308577 implements that
<Pali> ok, thank you for information... now I'm just waiting until git mirror feature will be published :-)
<cjwatson> If like literally everyone switches to git-to-git then we might decide that it isn't worth the trouble to continue to maintain git-to-bzr
<cjwatson> But realistically I imagine that's a while off
<cjwatson> git-to-bzr will at a minimum remain useful for transitional purposes for a while
<cjwatson> We'll probably try to gently steer people towards git-to-git though
#launchpad-dev 2016-11-09
<cjwatson>             view = create_initialized_view(
<cjwatson>                 repository, '+addsubscriber', pricipal=owner, form=form)
<cjwatson> wat
<cjwatson> I guess that doesn't do anything ..
#launchpad-dev 2016-11-11
<Spads> https://launchpad.net/ubuntu/+mirror/ubuntu.cybercomhosting.com-archive <-- hey so this mirror's prober log is a giant 404 party.  Why is it listed as up-to-date?
#launchpad-dev 2017-11-10
<cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad-buildd/backend-run-cwd/+merge/333569 fixes a regression in the undeployed launchpad-buildd 155, so would be good to get in place
<wgrant> cjwatson: r=me
<cjwatson> ta
