#launchpad-dev 2009-09-28
<lifeless> mwhudson: whats the lightest weight html-suitable template engine for python that you know of?
<mwhudson> lifeless: i don't know really
<mwhudson> lifeless: mako, perhaps/
<mwhudson> ?
 * wgrant would suggest genshi
<mwhudson> i don't really go in for light weight templating engines
<mwhudson> wgrant: genshi is nice, but not particularly lightweight?
<mwhudson> lifeless: what do you actually mean by lightweight?
<lifeless> by lightweight I mean 'easy for users of $somethingIwrite' to install -and- doesn't write to disk
<mwhudson> i guess genshi fits those
<lifeless> mwhudson: I'm asking in the context of
<lifeless> https://bugs.edge.launchpad.net/subunit/+bug/426283
<mup> Bug #426283: HTML formatter <html> <samba> <subunit:Triaged> <https://launchpad.net/bugs/426283>
<mwhudson> pretty sure it's easy_installable, it's in debian and ubuntu, don't *think* it writes to disk
<wgrant> It doesn't write to disk.
<wgrant> That would be a bit ew.
<lifeless> wgrant: you haven't seen much of the templaters around then :P
<wgrant> lifeless: I have. But I ran away from those.
<lifeless> so what I concretely need is something that can output large (20K+ tests, some fraction of which have detailed diagnostics) collections, smoothly
<mwhudson> lifeless: give genshi the ten minute trial
<lifeless> mwhudson: thanks
<mwhudson> feedback@launchpad.net gets the oddest selection of random mails
 * mwhudson eods
<adeuring> good morning
<al-maisan> moin abel
<jtv> mwhudson, still here?
<jtv> ah, no he isn't
<mwhudson> jtv: no
<jtv> mwhudson: right, nmâsorry to bother you.
<mwhudson> s'ok :)
<henninge> jtv: do we have a bug about the bzr upload failure? I am not talking about "stuck in running" but the root cause to that.
<jtv> henninge: the root cause as in, the reason why these particular jobs were getting oopses?
<henninge> jtv: yes
<jtv> henninge: if so, I don't know; ask Code people
<henninge> don't think so
<jtv> henninge: I think it's an error we've been seeing in unrelated code oopses
<jtv> but then, those code oopses tend to look like Greek to me
<jtv> (meaning "I'm supposed to understand them but I can't, or at least I can't be arsed)
<henninge> jtv: I am not sure if it's a code bug or if I am using bzrlib wrong. I am still looking at that.
<henninge> jtv: I should at least be able to provide a work-around quickly.
<jtv> henninge: it's a missing file, right?  It sounded highly Code/Codehosting-internal.
<henninge> jtv: ah no, the missing file was the configuration for oopses missing on dev. Not the real cause.
<henninge> jtv: the current problem is that I am doing "get_file_text" on the root folder ...
<jtv> henninge: oh, I wasn't aware of that one
<jtv> then that sounds like you're not getting the right id... lifeless knows this stuff, though this may not be a good time to trouble him
<lifeless> 'sup ?
<lifeless> (btw, use lifeless: to highlight my nick)
<jtv> lifeless: henninge's script is doing get_file_text on the root folder for a branch, when presumably he's trying to read an actual file.
<henninge> jtv: wait! you don't know all the detail ... ;-)
<jtv> henninge: don't talk to me, un-confuse lifeless about what we're trying to say.  :)
<henninge> lifeless: wait a second
<lifeless> do you have some elevator muzak?
<henninge> lifeless: ;-)
 * jtv hums Greensleeves in an infuriating monotone
<henninge> lifeless: ok, I am doing "iter_changes" with a "specific_files" parameter
<henninge> lifeless: the specific_files only contains one file.
<henninge> lifeless: the file is in a subdirectory
<henninge> lifeless: I get three changes back: The file, the directory, the root. Does it always do that? Has it always been doing that?
<henninge> "always" = within the last 6 months, I guess
<lifeless> henninge: since about 1.18
<lifeless> henninge: it returns enough data to be sure that a delta made from the changes will be consistent
<lifeless> henninge: so, I can infer that the root and containing directory are new.
<henninge> ah, so if they weren't, if only the file was changed, that's all I'd get, right?
<henninge> just the file
<lifeless> right
<henninge> lifeless: That would explain why this doesn't always file.
<lifeless> we have had many repeated bugs where new directories (and variations on a theme) do the wrong thing, so (and it is contentious) I fixed it by including more data when needed.
<henninge> it fails btw. because I do get_file_text on all three of them ...
<lifeless> henninge: check kind :) - I mean, someone could have a directory en.po
<lifeless> or a symlink
<henninge> lifeless: I was going to do that now. When I build the specific_files list, I check file_type and had thought that was enough.
<henninge> lifeless: thanks
<henninge> file_type is kind, I think
<lifeless> I'm sorry about the fallout; it was in the NEWS :P
<Fly-Man-> Morning
<Fly-Man-> mwhudson: *ping*
<mwhudson> Fly-Man-: hi
<mwhudson> Fly-Man-: it's very late for me i'm afraid
<Fly-Man-> mwhudson: O, I'm in no hurry
<Fly-Man-> but did you find something out of the pastebins I made for you ?
<mwhudson> Fly-Man-: not really no :/
<Fly-Man-> Hmm, so what would be wise to do ?
<Fly-Man-> install the bzr first, before I do a apt-get update ?
<Fly-Man-> because i'm not sure if bzr 2.x works with puthon 2.6
<wgrant> Wow.
<wgrant> Even Canonical people file Ubuntu bugs against Launchpad.
<Fly-Man-> wgrant: *lol*
<lifeless> wgrant: we do?
<wgrant> lifeless: Some of you.
<Fly-Man-> But any ideas on how to get 1.18
<Fly-Man-> maybe that'll not blow up
<henninge> lifeless: Hi again ;-)
<henninge> lifeless: nm
<henninge> lifeless: Ok, I got it fixed by checking kind.
<henninge> lifeless: but I investigated further into why this only failed sometimes and I found this: http://paste.ubuntu.com/280287/
<henninge> lifeless: do you have an explanation? I can provide you with the branch that causes the oops.
<Fly-Man-> Is there an Ubuntu server admin awake ?
<Fly-Man-> Keyserver for Ubuntu is failing
 * Fly-Man- found the solution
<Fly-Man-> pool.sks-keyservers.net
<Fly-Man-> Now how do I make a bug for the rocketfuelsetup ?
<maxb> It seems to be failing a lot lately
<Fly-Man-> Yeah, the keyserver seems to be getting it as well
<Fly-Man-> so i just gotten the replacement
<Fly-Man-> so it will try and find another one
<Fly-Man-> but I hope someone can include that in the rocketfuelsetup
<maxb> oh, I meant the keyserver. I've never run rocketfuel-setup
<Fly-Man-> so that it doesn't blow
<Fly-Man-> mrevell: Morning
<mrevell> hi there Fly-Man-
<Fly-Man-> mrevell: Who creates the rocketfuel-setup file ?
<Fly-Man-> I would like to have them change 2 bits in it
<Fly-Man-> change the keyserver that's in there to pool.sks-keyservers.net
<mrevell> Fly-Man-: Could you please file a bug at https://bugs.launchpad.net/launchpad
<Fly-Man-> mrevell: Added
<Fly-Man-> https://bugs.launchpad.net/launchpad/+bug/438104
<mup> Bug #438104: Changes for the rocketfuel-setup <Launchpad itself:New> <https://launchpad.net/bugs/438104>
<mrevell> thanks Fly-Man-
<jtv1> cprov, heads up: looks like the displayname for source packages was just changed and it's breaking tests in devel.
<jtv1> Tests on devel are breaking.
<cprov> jtv: uhm, bad ... did I break it in one of my last revisions ?
<jtv> cprov: looks like source package display names have changed.
<jtv> ah, title, not displayname.
<cprov> yes, it's in the view -> r9580
<jtv> In the view?  It seems to be a change in the model code...  by Edwin?
<jtv> EdwinGrubbs2: did you just change the title for sourcepackages?  It seems to be breaking tests in devel.
<EdwinGrubbs2> EdwinGrubbs2: yes, I'm sorry, I'll fix that.
<jtv> EdwinGrubbs2: here's the output: http://pastebin.ubuntu.com/280390/
<cprov> jtv: you are right, I can't read.
<jtv> cprov: I didn't say you can't read, but if you can't read, you may not have a way of knowing that.  :-P
<cprov> jtv: exactly what happened ...
<cprov> \action hides
<jtv> cprov: another thing...  I've got a script to look up and re-upload Soyuz translations tarballs to Rosetta, but AFAICS no sample data to debug it with.  I don't suppose there's an easy way to get that?
<jtv> Maybe get a package from somewhere, upload it to launchpad.dev somehow?
<cprov> jtv: yes, re-uploading something thin is your best bet.
<cprov> jtv: lib/lp/soyuz/doc/distroseriesqueue-translations.txt
<jtv> cprov: testing my work is going to be hell if I have to reproduce most of that...  Do you think I could sneak a "find Translations tarballs for last upload" into SourcePackage and check it from this doctest?
<jtv> It'd save me a lot of test setup for what's basically Registry and Soyuz code.
<cprov> jtv: yes, that makes sense for a `SourcePackage`
<noodles775_> allenap: I submitted a testfix branch to devel 0.5hrs ago now, but according to the following, the branch is still being updated:
<noodles775_> https://code.edge.launchpad.net/~launchpad-pqm/launchpad/devel
<noodles775_> Any ideas?
<noodles775_> I'm not sure why it would take so long to be scanned? (Unless there was something wrong with my branch?)
<allenap> noodles775_: I'm looking at the buildbot stuff, trying to understand that (!). I've seen the scanning problem too, I think we need a code ace. abentley, rockstar: branch scanning seems to be broken for lp:launchpad/devel, can you help out?
<noodles775_> allenap: looks like it's a general scanning issue too (see #lp)
<abentley> allenap: I can try.  This stuff is usually handled by antipodeans.
<allenap> abentley: Heh, thank you.
<abentley> allenap: What are the symptoms?
<allenap> abentley: https://code.edge.launchpad.net/~launchpad-pqm/launchpad/devel shows branch updating for a very long time.
<noodles775_> abentley: I've sent a testfix r9581 to devel over 0.5hr ago, but on #lp it looks like it's not specific to the devel branch.
<allenap> abentley: And people (for DBO in #launchpad) cannot pull his latest code, even though bzr says it's been pushed.
<abentley> noodles775_: They can't get the latest revision in your branch?
<noodles775_> abentley: sorry, no I was just saying that the issue is not related to one specific branch, others seem to be experiencing it with other branches too.
<noodles775_> abentley: my main issue is that the testfix revision is not being picked up by buildbot :)
<abentley> It looks like 9581 hasn't been pulled into the mirrored area.  That needs to happen before it can be scanned.
<abentley> noodles775_: It's unlikely that it's a problem with your branch, given that PQM was able to merge it and push to lp
<noodles775_> abentley: good-to-know - thanks!
<abentley> losa: Please update the puller log on devpad.
<Chex> abentley: hi, ok, looking for you
<thumper> abentley: hi
<abentley> thumper: hi.
<thumper> abentley: stub mentioned that the preview generation is being done within a transaction
<thumper> abentley: can I get you to look at that?
 * thumper is being ordered to close the laptop
<stub> That is my best guess. Either that or the script hung.
<abentley> thumper, stub: Each diff is generated within its own transaction, so that seems reasonable.
<thumper> abentley: some diffs are taking over 30 minutes
<thumper> see a mysql one
 * thumper has to close it now
<abentley> thumper: We should not spend that kind of time on diffs.
<XDevHald> Good morning everyone.
<XDevHald> Any news on the report a bug link being re-linked on the pad again or has this been migrated into the application it self?
<abentley> XDevHald: Is this specifically for Ubuntu?
<XDevHald> abentley: That is correct.
<abentley> XDevHald: That was removed at the request of Ubuntu.
<XDevHald> Are their future plans of this being re-added for those who do not run the development distro and want to report a bug?
<XDevHald> there*
<stub> abentley: So the solution might just be to abort diff generation if it is taking too long rather than move the diff generation outside of the database transaction.
<abentley> stub: That seems right to me.  More complicated, but it reduces the DoS surface area.
<stub> No hurry from my end then - that is exactly what is happening as far as the DB is concerned (transaction sits idle for 30 mins and we kill it)
<wgrant> XDevHald: What does running the development release have to do with anything?
<abentley> stub: Cool.
<abentley> XDevHald: There should be "Report a bug" in the help menu, even in non-development releases.  I see it, and I'm running Jaunty.
<XDevHald> My apologies, not the development release, but all developments of Ubuntu.
<james_w> XDevHald: that is a question for #ubuntu-bugs
<XDevHald> Excellent, thank you james_w
<lifeless> henninge: one is rich root
<lifeless> henninge: the 'content' of a dir is ''
<lifeless> gnight
<henninge> lifeless: thanks
<henninge> abentley: ^^
<abentley> henninge: If we follow the rule that "the content of a dir is ''", then the bug is in non-rich-root branches.
<henninge> henninge: yup, my interpretation, too.
<henninge> abentley: ^ ;)
<henninge> talking to myself again
<abentley> henninge: But your code shouldn't be doing get_file_text on a directory anyhow.  Why are you doing that?
<henninge> abentley: I know, that is a bug and I fixed it. Checking kind.
<abentley> henninge: Ah.  Thanks for reporting the issue anyhow.
<abentley> henninge: What's the bug #?
<henninge> abentley: I did not know (and if I understand it right it's a new behaviour) that iter_changes with specific_files returns the directories the files are stored in , too.
<henninge> abentley: haven't reported yet, got distracted ... ;-)
<henninge> I'll let you know
<abentley> henninge: This was a change introduced by lifeless over my strong objections.
<lifeless> abentley: I don't think its intentional that get_file_text works on a dir; I was just explaining how it worked at all
<abentley> Chex: Any luck?
<Chex> abentley: sorry, having trouble finding it, do you have any suggestion on devpad where to look?
<abentley> Chex: It's mirrored to /x/launchpad.net-logs/scripts/crowberry/puller.log
<jtv> bigjools: I'm not entirely sure this getSourceBySourcePackageReleaseIDs thing you sold me is working.
<abentley> Chex: How's it going?
<Chex> abentley: sorry for the delay, the crowberry/scripts dir logs have been synced now
<Chex> updated puller.log file
<abentley> Chex: Thank you.
<Chex> abentley: your welcome
<abentley> Chex: Is there a long-running branch-puller process on crowberry?
<Chex> abentley: looking
<abentley> Chex: I believe the script name is supermirror-pull.py
<Chex> abentley: thanks, I see it, yes there is a process of that name running for 2 hours now
<abentley> Please kill it.  This is causing an operational problem.  Please monitor the next run to ensure it completes successfully.  I suggest starting with SIGINT, but do whatever you have to.
<abentley> Chex: any news?
<Chex> abentley: process killed, and it automagically restarted, watching it do work in the logfile now
<abentley> Chex: Sorry I forgot to use your nick before.
<Chex> abentley: yes sorry, bit of a delay back to because of that, sorry too
<Chex> (to you)
<Chex> abentley: I just rsynced the puller.log over to devpad for you by hand, so you can see what it is doing now
<abentley> Chex: Thank you.  It looks like it's no longer stuck.
<Chex> abentley: ok, great, your welcome
<abentley> Chex: There may still be a problem, because I would expect the revno for http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel to be 9581 if the puller completes successfully.
<Chex> abentley: ok, sure, I will keep an eye on the running job on crowberry, make sure it finishes cleanly or see if it has problems.
<abentley> Chex: When it says "Stopping factory <twisted.web.xmlrpc._QueryFactory instance at 0x31fb560>" I believe this means that a script run has terminated.
<abentley> thumper: around?
<thumper> abentley: not really
<abentley> thumper: The puller is broken.
<Chex> abentley: well ok, I am seeing that a lot, and constantly
<thumper> abentley: do we know in what way?
<abentley> thumper: No.  It looks like it's running to completion, but http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel is stuck at revno 9580 instead of 9581
<abentley> thumper: I've manually taken a write lock to try and fix it, but that doesn't seem to have worked.
<thumper> abentley: :(
<thumper> abentley: I'm having to close the laptop again
<thumper> abentley: plz email details to mwhudson so he is aware
<thumper> abentley: anything in the logs?
<abentley> thumper: This is affecting all of lp.  Are you sure you need to close the laptop?
<thumper> are the logs synced?
<abentley> thumper: Yes.  I got Chex to kill the puller, then it ran and he synced the logs after a successful run.
<thumper> I've unplugged and moved to the kitchen
<abentley> thumper: I'm sorry, but it does seem urgent.  I haven't been able to determine a cause in the logs.
<thumper> the logs don't look that helpful
<abentley> thumper: It would help if they indicated which *branch* mirrored successfully, or ideally, which branches were still pending.
 * thumper nods
<abentley> thumper: I'm filing a bug about that.
<thumper> Chex: can I get you to sync the puller logs again please?
<abentley> thumper: There was a long-running process, which grabbed the lock at /var/lock/launchpad-branch-puller.lock at 13:14:03 and didn't let go.  I had Chex kill it at 15:32:04
<thumper> hmm...
<thumper> do we know what it was grabbing?
<thumper> probably not
<abentley> thumper: It's hard to infer, but I can try.
<thumper> do we know if it is just the launchpad branch that is failing to be mirrored properly
<thumper> or if there are others?
<abentley> thumper: We know there are others.
<abentley> thumper: https://code.edge.launchpad.net/~matsubara/oops-tools/package-diff-oops is an example.
<Chex> thumper: abentley : logs rsynced to devpad again
<thumper> Chex: thanks
<abentley> thumper: https://code.launchpad.net/~gagern/bzr-cvsps-import/lp437295 is another example.
<thumper> crap
 * thumper wonders WTF is going on 
<thumper> what has changed?
<thumper> what happened at 13:14?
<abentley> thumper: I believe the puller code has been significantly rewritten in this release.
<abentley> thumper: I can try to work out which branches weren't mirrored.
<thumper> it was
<thumper> but it was working earlier
<thumper> so something changed very recently
<thumper> it seems the scanner also seems a small bit fubared
<abentley> thumper: Or we've encountered a corner case.
<thumper> bzr revno http://bazaar.launchpad.net/~allenap/launchpad/itch
<thumper> 9582
<thumper> but https://code.edge.launchpad.net/launchpad shows the branch as not pushed to yet
<thumper> not "not scanned"
<thumper> which is suspicious
<thumper> Chex: can you pastebin "select * from branch where unique_name="~launchpad-pqm/launchpad.devel" for me please?
<abentley> thumper: Don't understand.  https://code.edge.launchpad.net/~allenap/launchpad/itch appears up-to-date.
<thumper> abentley: it wasn't when I looked just a minute ago :)
* bac changed the topic of #launchpad-dev to:  This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<rockstar> thumper, didn't spm apply a cowboy db permission?
<thumper> Chex: in the renaming of the puller, we now don't get the log file rolled
<thumper> rockstar: yes, I think I did see something about that
<Chex> thumper: I received an error on that, pmsging you
<thumper> hmm
<Chex> thumper: https://pastebin.canonical.com/22671/
<Chex> thumper: ok, I will take a look at the puller.log logrotate
<thumper> Chex: is the puller still running?
<Chex> thumper: yes it is
<abentley> thumper: There are too many PQM commits for me to determine which went with which branch.
<thumper> abentley: yeah
<abentley> thumper, Chex: are logs in UTC or local london time?
<thumper> abentley: I think they are UTC
<abentley> Chex: If there's a long-running update_preview_diffs process on loganberry, could you please SIGINT it and then sync the logs and oopses?
<Chex> abentley: ok sure
<jml> thumper, http://pastebin.ubuntu.com/280563/
<thumper> abentley: we need a way to kill diffs that take too long to generate
<abentley> thumper: I agree.  We could change the job runner so that it forks before running the job, and then the moitor could kill it if it takes longer than the timeout.
<abentley> s/moitor/monitor
 * thumper nods
<jml> +1
<jml> it would be nice if there were a really easy-to-use helper for that too, since it's probably a common thing for jobs.
<rockstar> jml, yeah, like the way imports work.
<jml> right, or the branch puller
<jml> modulo certain existing issues
<Chex> abentley: ok, process killed, restarted automagically, and rsynced logs and oops, altho that gets done there every 3 minutes..
<abentley> Chex: Thanks.  I didn't realize that was done so frequently.
<abentley> Chex: /x/launchpad.net-logs/scripts/loganberry-bzrsyncd/updatepreviewdiffs.log does not look like it's been updated since 2009-09-28 00:07
<Chex> abentley: there are many varied syncs happening; its difficult to track what and when things run, no worries
<Chex> abentley: looking
<abentley> jml: If it's the job runner, then it will work for anything that implements IJob.
<jml> abentley, cool.
<Chex> abentley: sorry, I haven't found that rsync yet, and I need to work on a rollout for Landscape right now
<abentley> thumper: One thing that worries me is the possibility that if we fork, we'll end up running the cleanups in the child thread.
<Chex> abentley: I will get back to it as soon as I can, or another losa could help you
<abentley> losa who is not Chex: Please update /x/launchpad.net-logs/scripts/loganberry-bzrsyncd/updatepreviewdiffs.log
<mthaddon> abentley: should be better now?
<abentley> mthaddon: Thank you.
 * mbarnett thinks mthaddon is too quick 
<abentley> thumper: there is a timeout on db connections, so the script will be killed after 30 minutes, but this is not a good long-term solution.
<thumper> abentley: please put a bit of time into thinking about a good solution
<thumper> abentley: poolie suggested filing a bug on bzr as it shouldn't take that long to generate the diff
<thumper> abentley: the puller had a complete success run
<thumper> abentley: I'm composing an email for mwhudson now
<jml> thumper, https://bugs.edge.launchpad.net/launchpad-code/+bug/438292
<mup> Bug #438292: startMirroring called twice for each branch being mirrored <branch-puller> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/438292>
<abentley> thumper: I agree we should file a bug on bzr, and then probably I should fix that bug.  And we should also support timeouts in the job system.
<thumper> yeah
<jml> thumper, https://bugs.edge.launchpad.net/launchpad-code/+bug/438298
<mup> Bug #438298: Branch puller triggers false alarms <branch-puller> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/438298>
 * rockstar lunches
<mwhudson> good morning
<mwhudson> rockstar, abentley: standup soon?
<rockstar> mwhudson, not for another hour, ent?
<abentley> mwhudson: Sure, anytime.
<mwhudson> rockstar: oh right, i'm in DST now
<rockstar> mwhudson, ah, okay.  We can have the standup now, that's fine.  I just wanted to make sure my computer wasn't the confused one.
<mwhudson> rockstar, abentley: when is most convenient for you guys this week?
<abentley> mwhudson, rockstar: I believe the plan was to stick with 9:15 NZ time.
<mwhudson> ok
<mwhudson> i'll be missing it in two days time, fwiw
<rockstar> mwhudson, on Fridays, abentley and I like to do it about now anyway, so he has his evening.
<mwhudson> abentley: want to host?
<abentley> mwhudson: sure
<abentley> mwhudson, rockstar: don't see you on Skype.
<mwhudson> i
<rockstar> abentley, booting the jaunty box, one sec.
<mwhudson> 'm online
<mwhudson> i can restart it i guess...
<abentley> mwhudson: https://code.edge.launchpad.net/~thumper/launchpad/fix-puller-logging/+merge/12537
<Fly-Man-> herb: *ping*
<Fly-Man-> wrong window ..
<Fly-Man-> herb: If you have time, could you update the launchpad source in your folder ?
<Fly-Man-> http://people.canonical.com/~herb/
<Fly-Man-> It's kinda ... old
<Fly-Man-> and also it's the dbdevel one
<herb> Fly-Man-: it was kinda meant as a stopgap after release. Maybe someone from the LP team can come up with a more manageable solution?
<Fly-Man-> herb: Okay :)
<herb> Fly-Man-: if you don't get any feedback over the next couple of days, ping me again and I'll see if I can help you.  Might be worth filing a bug against launchpad to track.
<Fly-Man-> Will do ...
<herb> Fly-Man-: cool. thanks!
<Fly-Man-> Nice to have it there, but it's an other version
<herb> yeah. definitely. I didn't expect it to still be in the documentation. :)
<Fly-Man-> Yupz
<Fly-Man-> and since my bzr doesn't like the rocketfuel setup script
<Fly-Man-> I was able to download that tarball
<Fly-Man-> but then there's no lp-dependencies
<Fly-Man-> so it's kinda useless :|
<wgrant> FWIW, I don't think that tarball is necessary any more. I can check out devel in less than half an hour, even from .au...
<maxb> Presumably the tarball will become even less necessary after the LP su
<maxb> source gets a post-2.0 bzr pack done on it
<maxb> or has that been done already?
<wgrant> It hadn't a few days ago.
<wgrant> But it's not too badly fragmented.
<wgrant> (it was packed fourish weeks ago)
#launchpad-dev 2009-09-29
<lifeless> spm: here?
<spm> lifeless: am now
<lifeless> nvm
<lifeless> sorry
<spm> np
 * mwhudson lunching
 * mwhudson has a nasty feeling that it's possible to have two concurrent processes trying to pull the same branch now
<mwhudson> that doesn't seem like it's gong to work well
<lifeless> oh man
<lifeless> I have a dirty mind
<purserj> two processes one pid?
<al-maisan> Good morning!
<thumper> mwhudson_: still here?
<mwhudson_> thumper: not really
<thumper> mwhudson_: ok, np
<adeuring> good morning
<al-maisan> moin adeuring
<adeuring> hi al-maisan!
<mrevell> Morning
<henninge> jtv: Hi!
<jtv> henninge: hi
<jtv> again
<jtv> :)
<henninge> jtv: could you help David?
<henninge> ;)
<jtv> henninge: david who?  david where?
<henninge> jtv: I am looking at the RemoveTranslations script.
<henninge> jtv: glad to see you are on top of things
<henninge> :-)
<henninge> jtv: it has an option "origin". What is that?
<henninge> jtv: I mean, what is the "origin code" ?
<jtv> henninge: that's that field saying "this came from the web UI" or "this came in from bzr"
<henninge> ah!
<jtv> RosettaTranslationOrigin
<henninge> jtv: there seems to be no way of restricting to a certain Project/Distro ?
<jtv> in lp.translations.interfaces.translationmessage
<henninge> jtv: yes, I remember now
<jtv> of restricting deletions to a project or distro?
<jtv> no, the script wasn't particularly made for real mass deletions
<jtv> but if you have a particular need, feel free to extend it.
<jtv> The options processing is still broken, which is a shame because it'd be really nice to have elsewhere too.
<henninge> jtv: https://answers.edge.launchpad.net/rosetta/+question/83585
<jtv> It really needs better ways to select templates.
<jtv> henninge: that's not currently something that the remove-translations script can do.  :(
<henninge> jtv: well, I'd have to look up all those chinese translators and do a run for each, restricting to ace.
<jtv> henninge: this doesn't look urgent; it may be better in the long run to extend the script.
<henninge> jtv: ok, I'll file a bug and make mention of it in the question.
<jtv> henninge: ok
<jtv> rockstar: got some more code import questions... can you handle them?
<jtv> Well, one code import question and one problem with bzr send.
<jtv> henninge: are you free for a pre-imp?
<henninge> jtv: I have 10 minutes ..
<jtv> henninge: should be enough!
<jtv> henninge: skype doesn't show you as online though
<jtv> henninge: ah, sound problem on your end
<henninge> jtv: should work now
<jtv> cprov: still fighting the "what is the latest translations upload for this package" issue...  is it normal that I'm not finding any upload sources in distroseriesqueue-translations.txt?
<cprov> jtv: probably, if you need consistent data you better create it, paste me some code, let me help you.
<jtv> cprov: very kind, thanks.  Maybe the easiest is if I point you to my branch: lp:~jtv/launchpad/redo-upload
<Ursinha> adeuring, hi
<danilos> jtv, hi, have you seen the bzr export problem? henninge, can you please look at it? (and thanks for requesting the CP)
<jtv> danilos: yes, I mentioned it in the call yesterday.
<cprov> jtv: k
<jtv> danilos: some branches still locked from when things were failing.
<danilos> jtv, "some"? 71 out of 78? :)
<jtv> danilos: yes, some.  :-P
<danilos> jtv, what we need to do to fix it?
<jtv> danilos: break the locks.
<danilos> jtv, ok, so who has the privileges to do that, and who'll do it? :)
<jtv> danilos: some users seem to have done it for themselves.  I wanted to ask the losas about it.
<danilos> henninge, can you help with this while jtv is busy figuring out the re-uploading stuff?
<henninge> danilos: sure, I'll see what I can do!
<danilos> jtv, is henninge around at all (I didn't see anyone online a few hours ago, and I have trouble being online at all times)
<jtv> (and be careful only to break locks that are several days old)
<danilos> henninge, excellent, thanks a lot
<jtv> danilos: no, he's not around, you're talking to a bot
<cprov> jtv: is your branch pushed ?
<jtv> cprov: think so... hang on
<danilos> jtv, go away :)
 * jtv goes away
 * henninge too
* mrevell changed the topic of #launchpad-dev to: Code hosting offline 10.00-10.30 UTC 1st Oct -- http://blog.launchpad.net/general/code-hosting-offline-2009-10-01 | This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
* mrevell changed the topic of #launchpad-dev to: Code hosting offline 10.00-10.30 UTC 1st Oct -- http://is.gd/3MyHY | This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<jtv> cprov: that branch should be up there... look at what I did in model/sourcepackage.py, with julian's guidance
<cprov> jtv:k
<jtv> cprov: ohhhh, I'm assuming main_archive... is that a problem?
<cprov> jtv: not necessarily
<cprov> jtv: in getLatestTranslationsUploads() implementation 'histories' is wrong, we can have multiple publications of the same SPR in the same distroseries in different pockets.
<jtv> cprov: hang on, in standup!
<jtv> sorry!
<cprov> jtv: the assertion is too strict, in fact
<cprov> jtv: once you have a published SPPH you can user pub.sourcepackagerelease.package_upload.customfiles directly, no need to user the PUSet lookup
<mrevell> mrjazzcat: https://edge.launchpad.net/~chromium-daily/+archive/ppa
* mrevell changed the topic of #launchpad-dev to: Launchpad's slow right now -- we're looking into it | Code hosting offline 10.00-10.30 UTC 1st Oct -- http://is.gd/3MyHY | This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<beuno> mpt, how do you feel about dropping the "overview" tab?
<beuno> re: bug 185030
<beuno> I'm 50/50 on it...
<jtv> cprov: sorry to be away at that crucial time.  Back now.
<mup> Bug #185030: "Overview" is an awkward name for the first tab <post-3-ui-cleanup> <Launchpad Registry:In Progress by beuno> <https://launchpad.net/bugs/185030>
<cprov> jtv: no worries, did you follow my comments ?
<jtv> cprov: yes, though somehow these long identifiers always make my head swim  :)
<jtv> Matching them up to code now.
<mpt> beuno, I don't think the project icon alone would be obviously mutually exclusive with the tabs.
<mpt> beuno, and it would worsen the problem of navigation in distribution packages vs. distributions.
<beuno> hm
<beuno> yeah
<mpt> beuno, because (for example) you'd be clicking on an Ubuntu icon to go to a source package page.
<mpt> (Or to go to a distribution series page.)
<beuno> agreed
<beuno> I'll keep chewing on it then
 * mpt can load other Bugs pages, but can't load that bug report
<beuno> yeah
<beuno> LP is slow again
<beuno> something off with apache I think
<mpt> beuno, I think renaming it to "Profile", "Package", "Project", etc would work pretty well
<mpt> That's what I was working on (but didn't finish) when I left the LP team. :-)
<jtv> henninge: about that pre-imp... is 30 minutes from now ok for you?
<beuno> mpt, that approach itches for a reason. I should mock it up and see how it looks
<henninge> jtv: so at 15:59? OK.
<henninge> ;)
<jtv> henninge: roughly :)
<henninge> jtv: about the locked branches - where do I see them?
<jtv> henninge: error-reports.  Hang on, I'll paste you the latest.
<jtv> henninge, sanitized list: https://pastebin.canonical.com/22715/
<jtv> cprov: looks like SPPH.sourcepackagerelease.package_upload can be None..?
<cprov> jtv: in the sampledata, yes
<jtv> cprov: ah.  Still getting the exact same symptoms in that test...  Do you think splicing into distroeseriesqueue-translations.txt is the wrong approach for exercising this?
<cprov> jtv: I dunno, let me look at the test.
<henninge> jtv: how do I get information about the locks, like the lock age?
<jtv> henninge: normally, if they were your branches, you'd use bzr break-lock.  As it is, I don't know.  This one needs help from codehosting folks, I think
<henninge> jtv: ah, ok. So there is not an easy way.
<jtv> henninge: they may have a trick, like just deleting a bunch of lock files.
<henninge> abentley: ping
<abentley> henninge: pong
<henninge> abentley: Hi!
<henninge> abentley: We have a number of branches that were left locked by the translations export script.
<henninge> abentley: we are looking for an easy way to break them while making sure, the locks are really stale.
<henninge> abentley: do you have a suggestion?
<abentley> henninge: I can have a look at bzrlib.
<mpt> thumper, reported bug 438711
<mup> Bug #438711: Can launchpad-login but not launchpad-logout <Bazaar:New> <https://launchpad.net/bugs/438711>
<henninge> abentley: I mainly thought of the permissions issue. I cannot break other people's locks, can I?
<abentley> henninge: You can break a lock on any branch you have write access to.
<abentley> henninge: If we get a losa to run a script on crowberry, then you can break any lock.
<henninge> abentley: Ok, that is what we'll need, I guess.
<henninge> abentley: Does bzr break-lock detect if a lock is really dead or not?
<abentley> henninge: I'm investigating that.
<henninge> abentley: cool, thank you
<abentley> henninge: There doesn't appear to be any check that the holder is dead.
<abentley> henninge: In the sftp case, the holder will be a process on a remote machine.
<abentley> henninge: I am not certain about the bzr+ssh case.
<henninge> abentley: is there a way to read the age of a lock?
<abentley> henninge: Yes.
<henninge> abentley: oh cool, that should help. How?
<abentley> henninge: For a bzr+ssh lock, the process id is the server-side process.
<henninge> abentley: oh cool, too, I think.
<henninge> abentley: these are all designated as lp-hosted. What protocol is that?
<abentley> henninge: calling "peek" on the lock will tell you pid and start_time.
<abentley> henninge: For the case where they were created by a script, they will be accessed as local files.
<abentley> henninge: So your script didn't use a protocol, but users can access them via bzr+ssh or sftp.
<abentley> I think if you ensure the PID isn't active and the lock is old, the risk of breaking an active lock is low, but not 0.
<henninge> abentley: yes, I guess I should just get a listing of the locks, first.
<jtv> cprov: code's a lot simpler and cleaner now, but still not finding anything in that test.
<jtv> cprov: I've pushed an update.
<cprov> jtv: will update in one sec
<jtatum> intellectronica: ping :)
<EdwinGrubbs> sinzui: ping
<sinzui> Hi EdwinGrubbs
<intellectronica> jtatum: hi. i got some test failures for your branch, but it looks unrelated to your change. let me forward them to you
<EdwinGrubbs> sinzui: I had some difficulties running close_bugs_from_commits.py, and matsubara informed me that he can run it for all the launchpad projects, but that some of the team leaders had opted out. Can we opt in?
<sinzui> EdwinGrubbs: I do not want to
<sinzui> EdwinGrubbs: The script does not give you credit for your work. Barry was put out by it.
<sinzui> EdwinGrubbs: Did you get the script to work for yourself?
<sinzui> EdwinGrubbs: I ran it (or a derivation of it) Last Wednesday and Friday.
<EdwinGrubbs> sinzui: I got it to work, but it only limits changes to given projects not people, so I actually Fix Committed some other people's bugs.
<EdwinGrubbs> sinzui: when you ran it, it didn't close any of my Blueprint bugs.
<sinzui> oh
<sinzui> EdwinGrubbs: I'll send you my script
<EdwinGrubbs> sinzui: when you say you don't get credit for closing the bug, do you mean that you don't get karma?
<sinzui> yes, + we loose tracking of that...we want to use karmaevents to report person's activities, even if there are no point. That script (or our API) wrongly reports who did the work
* mrevell changed the topic of #launchpad-dev to: Code hosting offline 10.00-10.30 UTC 1st Oct -- http://is.gd/3MyHY | This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<jtv> cprov: had a chance to look at why that test has no package uploads?
 * rockstar lunches
<Ursinha> hi abentley
<Ursinha> abentley, do you know if it's a known issue that lp doesn't support http branch urls for git?
<abentley> Ursinha: hi
<Ursinha> issue or limitation
<abentley> Ursinha: I think I've seen a bug for that...
<Ursinha> abentley, I just searched and couldn't find one
<Ursinha> maybe I'm just doing it wrong :)
<abentley> Ursinha: It may have been a question.  Still looking.
<abentley> Ursinha: I can't find anything.  I don't know whether it's a known issue.
<jtv> cprov: not having much luck deciphering the situation in distroseriesqueue-translations.txt...  Does the SPR get created in that test?
<Ursinha> abentley, well, I'll file a bug then
<Ursinha> thanks abentley
<abentley> Ursinha: np
<cprov> jtv: uhm, forgot about it.
<jtv> I'm really too tired to work on this, but it needs to get done.  :/
<cprov> jtv: duderino, translations come from binaries, not sources (!)
<rsalveti> hi, I'm trying to import a git repository inside launchpad, but it's just available in http (webdav) and currently launchpad doesn't support it
<jtv> cprov: so this test doesn't upload a source package?
<cprov> jtv: no, it creates a base source, so we can upload a binary with a translation tarball
<rsalveti> I'd like to know why exaclty this is not supported, and if possible, where is the code that actually implements the git import from launchpad
<cprov> jtv: your lookup on SP is wrong.
<rsalveti> so I could take a look and maybe hack it to make the support
<jtv> cprov: argh
<rsalveti> don't know if it's a bzr problem or launchpad itself
<jtv> cprov: I should be finding the binary packages for the source package?
<Ursinha> abentley, ^
<cprov> jtv: looks for custom files uploaded with binaries built by the context SPR
<Ursinha> rsalveti, I'm on my way to file a bug about it, abentley, with whom should rsalveti talk to get to know what to do?
<rsalveti> I'm quite new to both launchpad and bzr, but would like to know what could I do to have this support :-)
<cprov> jtv: yes,  history.getBuilds() -> build.package_upload.customfiles
<Ursinha> rsalveti, abentley and rockstar are the people that would be available in a reasonable timezone :P
 * jtv cries quietly into his keyboard
<rsalveti> :-)
<abentley> rsalveti: launchpad uses bzr-git to do git imports.  The launchpad code is will be in launchpad/stable/lib/lp/code/
<abentley> sorry, "lib/lp/code".
<rsalveti> abentley: cool, will take a look at it
<rsalveti> going to a meeting now =\, in one hour will be back, thanks
<abentley> rsalveti: There will be more in lib/lp/codehosting/codeimport
<jtv> cprov: will getBuilds give me any superseded builds?
<cprov> jtv: all builds for the context source are relevant, if they got superseded their 'package_upload' property is None.
<jtv> cprov: so at least that's a mercy
<jtv> julian said this was easy, damn his salad tongs
<jtv> cprov: progress!  I'm seeing the upload!
<cprov> jtv: cool
<jtv> cprov: ooooh, I have passing tests!  thank you thank you thank you
<cprov> jtv: you know my address, send cookies ;)
<jtv> cprov: how do I explain this...  I know your _email_ address.
<jtv> That's not useful for this type of cookie.
<cprov> jtv: :)
<cprov> jtv: you know iterating over builds and getting their corresponding package_upload is somehow expensive, right ?
<jtv> cprov: no I don't know, and I'm not sure I care much.  :)  This is for a one-off, not production use.
<cprov> jtv: will your script dealing with multiple sources ?
<jtv> cprov: a few dozen source packages.
<cprov> cool, no worries then
<mwhudson_> morning
<Ursinha> rsalveti, I've filed bug 438929 for your import problem
<mup> Bug #438929: Launchpad doesn't accept git imports with http <codehosting> <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/438929>
<rsalveti> Ursinha: cool, thanks, going to take a look
<Ursinha> rsalveti, np
<rockstar> Ursinha, I didn't see your PM earlier.  What's up?
<mwhudson> rockstar, abentley: morning
<mwhudson> rockstar, abentley: standup now?
<rockstar> mwhudson, yo
<mwhudson> ish
<rockstar> mwhudson, sure.
<abentley> mwhudson: Okay.
 * mwhudson away for a speel
<mwhudson> speel
<mwhudson> spell
<thumper> mwhudson: hi
<thumper> rockstar: hi
<rockstar> thumper, hi
<rockstar> thumper, I think mwhudson is off to pick up a suit.
<thumper> the hotel internet is flirting with me
<thumper> first it is there
<rockstar> thumper, is it there now?
<thumper> not really
<thumper> it lets me get one irc message in
<thumper> and then dies
<thumper> OMG three in a row!
<ajmitch> you haven't appeared to disconnect yet
<mwhudson> thumper: good evening
<mwhudson> err
<mwhudson> from https://answers.edge.launchpad.net/launchpad-code/+question/84184, how can i see the current status of the question?
<mwhudson> oh, there it is
<thumper> omg this internet connection sucks
<mwhudson> thumper: hi
#launchpad-dev 2009-09-30
 * mwhudson afk for a couple of hours (but working, at least a bit)
<lifeless> hi thumper
<lifeless> solved world peave yet?
<masom> Hi, any tips or howto properly setup launchpad for a local use (non-hosted)? It kinda runs fine with the debug branch but is there anything that should be modified to have a ... let say not devel launchpad instance?
<intellectronica> masom: there isn't really any reason to run launchpad in any capacity other than for development, so nobody puts any effort into that
<masom> intellectronica: thats what i figured out since nobody made a howto on how to host it by himself.
<masom> intellectronica: but technically, from dev to "production", the main changes are in the config files?
<masom> intellectronica: (such as the ability to actually send outbound emails)
<intellectronica> masom: the production environment for launchpad is a _very_ complex affair (consider that there are several administrators and engineers working full-time just to keep launchpad.net going). it's the configuration and a setup of quite a few machines and many running processes
<intellectronica> masom: it's not impossible to replicate it if you're willing to commit the same resources, but it would seem a pointless exercise, given that launchpad.net is already there for you to use, and that you can help develop it, it being free software
<masom> intellectronica: yeah, i understand that :) But it's just for the kick of actually having it running as the real deal. If nobody outside canonical is able to reproduce it, its dangerous.
<intellectronica> masom: there's nothing stopping you from running 'the real deal'. many people run it locally for development, and it's essentially the same software
<masom> intellectronica: yeah thats what i figured out when it launched locally, i just wanted to poke around to know how hard it would be to make the devel react like the launchpad.net deployment. And i'm glad that it looks like its mostly config files to modify.
<lifeless> does anyone else find that malone bug pages run _very_ slowly once you have 30 to 40 open?
<lifeless> just typing in the comment field feels like molasses
<masom>  mmm stupid question, what would trigger this: lazr.config.interfaces.ConfigErrors: ConfigErrors: launchpad does not have a localhost http key. ?
<masom> Oh also a very weird thing
<masom> make run_all with a differnt host (let say dev.local instead of dev), apache tries to connecto port 8086, moving apache to proxy 8085 works, any reason why it's attempting to use 8086 ?
<lifeless> any new on the red border problem?
<lifeless> [6 edits so far on this bug summary :(]
<al-maisan> Good morning
<thumper> hi hi
<mrevell> hi
<thumper> mrevell: dude, you're across the table!
<mrevell> :)
<jtv> spm: system reboot, distractions...  I think we'll have to let the jp weirdness lie for the moment.
<spm> jtv: :-) oki
<jtv> spm: thanks though!
<spm> jtv: anytime* (* subject to core hours, laptop availability and other terms & conditions. please see your contract)
<jtv> spm: would that be the contract kept in the data centre managed by you untrustw^^^NO CARRIER
 * spm returns the lart to it's place of rest
 * jtv hates to think what place that is
<spm> pile of shrunken pc's from devs that were .... larted
<stub> >>> None < 44
<stub> True
<stub> Grrr...
<lifeless> \o/
<thumper> stub: ping
<stub> thumper: pong
<thumper> stub: it seems we are *still* missing a db permission https://lp-oops.canonical.com/oops.py/?oopsid=1368BM2
<thumper> stub: I thought it was fixed
<thumper> stub: can you add it to your pending branch and fix on production?
<stub> We granted select permissions. looks like it is trying to update.
<stub> (?)
<stub> no....
<thumper> huh?
<thumper> outgoing email shouldn't update anything
<stub> thumper: Is this the branchscanner?
<thumper> stub: no this is the revision email sending script
<stub> thumper: What db user does it connect as?
 * thumper looks
<thumper> send-branch-mail
<stub> send-branch-mail?
<thumper> yes
<maxb> What bit of LP actually requires memcached? "make run" seems happy without
<thumper> maxb: nothing just yet
<stub> chicken/egg
<stub> thumper: perms granted on production. Updating my branch now.
<thumper> stub: thanks
<poolie> thumper: http://bazaar-vcs.org/SmartPushAnalysis1.4
<poolie> sudo tc qdisc add dev lo root netem delay 500ms
<poolie> go on, paste in sudo commands from irc :)
<poolie> you know it makes sense
<Fly-Man-> Who handles the bugs related to the source pulls?
<spiv> Fly-Man-: https://bugs.launchpad.net/launchpad-code
<Fly-Man-> spiv: Is that also the place for this bug: rocketsetup has wrong keyserver mentioned ?
<spiv> Just file a bug on the launchpad project for that, IIRC.
<Fly-Man-> spiv: Already did
<barry> reviewers and lurkers -> #launchpad-meeting in 4m
<stub> How to I get the launchpad-developers-dependencies package rebuilt? I got a change landed to the branch meta-lp-deps branch but from there I am clueless.
<Fly-Man-> ./rocketfuel-setup: line 372: 19632 Segmentation fault      bzr branch http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ $LP_TRUNK_NAME
<Fly-Man-> ERROR: Unable to create local copy of Rocketfuel trunk
<Fly-Man-> Anyone want to elaborate on that one ?
<Fly-Man-> This keeps happening over and over again with the bzr 2.0.0.1 from the ppa
<barry> Fly-Man-: i've never seen that before.  can you gdb the core file and see what it says?
<Fly-Man-> gdb-python or gdb itself ?
<barry> Fly-Man-: gdb i'm not sure what gdb-python is
<Fly-Man-> python variant
<Fly-Man-> Any command lines I need to add ?
<barry> Fly-Man-: actually, start with 'file core'
<Fly-Man-> file core ?
<barry> Fly-Man-: that will tell you which binary is crashing, though i'm going to bet it's python.  and if that's the case, it's almost certainly a bug in some bzr extension
<barry> Fly-Man-: what platform are you on?
<Fly-Man-> Vmware Jaunty 2 Gb mem, 15 Gb hard drive
<barry> Fly-Man-: i'm on karmic now and bzr 2.1.0dev.  i'm having other problems but dumping core isn't one of them ;)
<maxb> Random question: what exactly is rocketfuel? Is it just a cute name for devel?
<barry> Fly-Man-: try running that bzr command manually at the shell and see what happens
<Fly-Man-> maxb: it's the script that installs the devel branch on a machne and setup the dependencies
<barry> maxb: basically yes.  it's historical
<barry> maxb: the branch used to be called rocketfuel (get it, launchpad... rocketfuel... soyuz... there's a theme :) but the only traces now are in the setup scripts
<maxb> <Fly-Man-> Who handles the bugs related to the source pulls?
<maxb> Fly-Man-: In general, people don't handle bugs (on any open source project). Groups of people pick up various bugs depending on their availability. Hence the use of bug trackers. :-)
<Fly-Man-> maxb: true
<Fly-Man-> maxb:that's why ppl always are impatient when a bug is still not fixed ;)
<maxb> stub: You build the package in the normal way of working with Debian source packages and upload it to the launchpad PPA. (Or, if you like, I'll do it and find someone in ~launchpad to copy it there)
<stub> Given I don't know where to start with working with Debian source packages, you or someone who understands had better do that part.
<maxb> stub: I'm afraid trunk doesn't build - the trailer line on your changelog entry is malformed
<jtv> abentley: looking into the workaround for the branch commits... whaddya think, should I do it right there in DirectBranchCommits?  Or in the script that calls it, and add a check in DBC?
<jtv> abentley: The advantage of the latter is that the branch conversion may take a while, and it's polite to keep it in a separate transaction
<maxb> stub: There must be two spaces between the email and the date, and the date format must exactly match the format demonstrated in other changelog entries
<abentley> jtv: Do what?  Convert to unstacked?
<jtv> abentley: right
<stub> maxb: fixed
<abentley> jtv: I would do it in the script that calls it.  I'm not really happy with converting to unstacked, though.
<jtv> abentley: I'm not completely happy about it either, but I think it beats alternatives like reverting to regular branch/commit/push or putting the burden on the user.
<maxb> stub: not fixed - must be day month year, not month day year
<stub> maxb: and again?
<maxb> stub: that's good, built source, uploaded to ~maxb/+archive/launchpad, waiting for process-upload and build
<stub> ta
<Fly-Man-> barry: Strangely when I try it with the command line, it doesn't fail so far ...
<barry> Fly-Man-: very weird!
<jtv> abentley: I don't suppose you guys have a helper function to say whether a branch is stacked?
<abentley> jtv: That information is stored in the database, and can be easily retrieved with bzrlib.
<jtv> abentley: it turns out get_stacked_on_url is a very roundabout way of finding out.  :)
<jtv> abentley: Branch.stacked_on is None, right?
<abentley> jtv: Yes.
<abentley> jtv: An alternative is to embrace "Easier to ask forgiveness than permission", and handle the exception generated by committing to a stacked branch.
<jtv> abentley: wayyyy too much code for the "nothing's going on" case.  :)
<jtv> I tried.
 * Ursinha looks around for bugs people
<Ursinha> intellectronica, hi :)
<Ursinha> intellectronica, could you please set the importance on bug 439449?
<mup> Bug #439449: On a bug, entering +edit page shows the wrong bug number <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/439449>
<Ursinha> thanks :)
<intellectronica> Ursinha: done
<Ursinha> intellectronica, thanks again
<intellectronica> Ursinha: thanks for bringing this to my attention
<Ursinha> intellectronica, no problem. I found this when trying to workaround bug 423924
<mup> Bug #423924: Entity-body was not a well-formed JSON document when updating bug description <post-3-ui-cleanup> <Launchpad Bugs:Triaged by deryck> <https://launchpad.net/bugs/423924>
<Fly-Man-> barry: It's stopping at the same place everytime
<intellectronica> jtatum: rejoice, your patch has landed!
<intellectronica> jtatum: thanks again for picking this up, you rock
<jtatum> intellectronica: thank you very much! the celebrations have begun here in my lair. champagne for everyone.
<ScriptRipper> hi kiko
<Fly-Man-> barry: *ping*
<barry> Fly-Man-: *ponngggggg!!!* :)
<Fly-Man-> barry: What repo are you usin for 2.1
<barry> Fly-Man-: bzr?
<Fly-Man-> yes
<barry> it's either:
<barry> deb-src http://ppa.launchpad.net/bzr-beta-ppa/ubuntu karmic main
<barry> deb http://ppa.launchpad.net/bzr-beta-ppa/ubuntu karmic main
<barry>  
<barry> or
<barry> deb http://ppa.launchpad.net/bzr-nightly-ppa/ppa/ubuntu karmic main
<barry> deb-src http://ppa.launchpad.net/bzr-nightly-ppa/ppa/ubuntu karmic main
<barry>  
<barry> probably the latter and i probably shouldn't be using the nightlies ;)
<Fly-Man-> Let's see if that fixes the segmentation fault
<Fly-Man-> barry: nope, same issue
<Fly-Man-> wth is happening here
<Fly-Man-> it worked with 1.16 and 1.18
<Fly-Man-> then the ugrade to 2.x makes it shakey
<barry> Fly-Man-: are you on a 64bit system perhaps? (i am tho and haven't seen this)
<Fly-Man-> x86 machine
<Fly-Man-> it's latest vmware
<Fly-Man-> machine has 4 gb
<barry> i can't imagine vmware has anything to do with it, but you never know
<barry> Fly-Man-: you might ask around on #bzr though
<Fly-Man-> They blamed the mem
<Fly-Man-> the memory is brand new on that machine
<Fly-Man-> build that machine 4 days ago
<Fly-Man-> Hmm, and there's no tree where there's 1.18 bzr anymore ....
<barry> i always get suspicious when the hardware is blamed.  so much more rare than a missing incref in my experience ;)
<Fly-Man-> that's my idea as well
<Fly-Man-> why would it work on 1.18
<Fly-Man-> and not on 2.x
<Fly-Man-> barry: Could you try something for me ?
<Fly-Man-> Could you create a tarball from devel ?
<Fly-Man-> or someone else
<barry> Fly-Man-: why not try to grab the latest bzr trunk and build it from source?
<Fly-Man-> I just pulled the nightlies
<Fly-Man-> same segmentation fault
<Fly-Man-> so it seems to be something in 2.x
<Fly-Man-> which wasn't there in 1.18
<barry> Fly-Man-: maybe abentley or rockstar knows?
<Fly-Man-> Good idea, abentley , rockstar : Any ideas ?
<rockstar> Fly-Man-, reading backchat
<Fly-Man-> rockstar: Thanks
<rockstar> Fly-Man-, the nightlies are working fine for me.  Upgraded just this morning.
<Fly-Man-> rockstar: In short: I've been using Launchpad.dev for some time, until the latest release of the bzr 2.x. After that, during reinstalls the bzr just quits with a segmentation fault while trying to pull the latest
<rockstar> Fly-Man-, what do you mean by "reinstalls" ?
<Fly-Man-> rockstar: virtual machine reinstallations
<Fly-Man-> Every 5 days I need to clean up Vmware machines
<Fly-Man-> while the server is getting too burdened
<Fly-Man-> so i cleaned up last Saturday
<Fly-Man-> after that, it's a mess while trying to get the initial tree
<rockstar> Fly-Man-, it might very well be "hardware" because VMware's memory is still software.
<rockstar> Fly-Man-, I suggest branching bzr.dev, building the C extensions, and trying that.
<Fly-Man-> rockstar: is there a trunk for that ?
<rockstar> Fly-Man-, bzr.dev is trunk
<Fly-Man-> rockstar: Same error
<rockstar> Fly-Man-, try it outside a VM.
<Fly-Man-> rockstar: I'll have a look when there's a machine available in the datacenter
<Ursinha> hey cprov, are you there?
<rockstar> barry, so, I think I'm the only one around for the AsiaPac reviewer meeting today.  Is it even worth having one?
<barry> rockstar: probably not, the most meaningful thing we did in ameu was to wish cprov well :)
<cprov> Ursinha: yup
<Ursinha> cprov, hi :) can you help me with https://answers.edge.launchpad.net/launchpad/+question/84264, please?
<cprov> Ursinha: sure, in a bit
<Ursinha> thanks cprov
<cprov> Ursinha: done
<Ursinha> cprov, thanks :)
<barry> bac: ping
<bac> hi barry
<barry> bac: hi, are you busy?
<bac> barry: i'm always busy and productive.  but i have a few moments to spare.
<barry> bac: ;) can we skype?
<bac> sure
 * bac launches
<maxb> Thanks whoever copied launchpad-dependencies 0.55 :-) I was just about to send an email requesting it :-)
<maxb> err
<bac> barry: ready
<maxb> whoever copied it to hardy, shouldn't have
<barry> bac: calling...
<barry> bac: bug 403606
<mup> Bug #403606: ExpatError errors should be handled to not generate the OOPSes <oops> <Launchpad Registry:In Progress by barry> <https://launchpad.net/bugs/403606>
<barry> bac: https://lp-oops.canonical.com/oops.py/?oopsid=1306XMLP2
<barry> bac: http://paste.ubuntu.com/282468/
<maxb> Hmm. The launchpad PPA for hardy is currently uninstallable.
<maxb> Is there anyone who needs to know this?
<barry> bac: eggs/zope.publisher-3.5.6-py2.4.egg/zope/publisher/xmlrpc.py
<Fly-Man-> rockstar: *ping*
<Fly-Man-> A new machine just got delivered in the datacenter
<Fly-Man-> so I;m setting that one up with Jaunty now
<Fly-Man-> to see if it works then
<rockstar> Fly-Man-, if you continue to get the segfault, #bzr is going to be able to help much more than I can.
<maxb> I wish the meta-lp-deps branches were somewhere I had write access :-/
<maxb> Please can I enlist a member of ~launchpad to add this missing tag:
<maxb> bzr tag -d lp:~launchpad/meta-lp-deps/trunk -r revid:maxb@f2s.com-20090910075602-i7m5vu8dicck9xcu 0.54
<maxb> bzr tag --force -d lp:~launchpad/meta-lp-deps/trunk -r revid:stuart.bishop@canonical.com-20090930150127-02ubq4ceny88higo 0.55
<maxb> also
<maxb> and yes I do mean --force in the second
<mwhudson> good morning
<maxb> good evening :-)
<maxb> To what extend do we care that the launchpad PPA is not currently installable on hardy?
<mwhudson> maxb: it's pretty bad
<maxb> I'm in no position test anything more than the installability of packages on hardy, but I could prepare something that is at least better than what is there.
<maxb> Can I enlist you to run those two bzr tag commands from scrollback ~30 mins ago before they get forgotten about, or shall I email them to the list?
<mwhudson> maxb: ok
<mwhudson> maxb: what's the problem on hardy currently?
<maxb> Someone copied the new jaunty/karmic launchpad-dependencies build to hardy too, but hardy was using a branched version
<maxb> so the hardy-specific changes were removed
<mwhudson> oh right
<maxb> On a related note, we should probably change the description of the PPA to say that Intrepid is no longer supported.
<maxb> OK, in two minutes, once it's published, please could someone copy-with-binaries 0.55hardy1 from maxb/launchpad to launchpad/ppa (hardy)
<maxb> And also push lp:~maxb/meta-lp-deps/hardy into lp:~launchpad/meta-lp-deps/hardy
<maxb> right, it's published
<beuno> maxb, I think I have the power to do so
<maxb> thanks
<Fly-Man-> rockstar: It seems that there's an issue with the latest Vmware server
<Fly-Man-> I have alerted the Vmware staff to have a look at it
<beuno> maxb, done
<maxb> Thanks, hardy lp devs will thank you :-)
#launchpad-dev 2009-10-01
<mwhudson> aaaaa
<lifeless> bbbbb
<mwhudson> want to make /bin/py start up in 0.07 seconds (ish) rather than 0.5 ?
<mwhudson> touch lib/site.py
<mwhudson> in other news: setuptools is bad
<lifeless> orly
<lifeless> does lib/site.py exist normally?
<mwhudson> it has some behaviour that is quadratic in the length of sys.path
<mwhudson> lifeless: no, touching it prevents setuptools' site.py from loading
<lifeless> hah
<maxb> The other reason setuptools is bad? I can't find any way to make its namespace package support compatible with having system installed lazr.* and buildoutified lazr.*
<lifeless> well
<lifeless> eggs are bad
<lifeless> the rest follows
<mwhudson> maxb: "the" other? :)
<lifeless> someone should write a rant
<maxb> well, I was thinking more *immediate* reasons :-)
<mwhudson> the overriding problem with setuptools is that people use it
<mwhudson> which is not the correct thing to do with software from pje
<lifeless> LOL
<lifeless> quotes page....
<mwhudson> the correct thing is to be inspired by it
<mwhudson> heh
<al-maisan> Good morning!
<thumper> morning
<spm> yo
<thumper> spm: how goes the puller?
<spm> thumper: as expected. problem free now we have sufficient logging to detect the actual problem.
<thumper> spm: excellent
<thumper> mwhudson: hi
<mwhudson> thumper: hello
<thumper> mwhudson: were you able to glean any extra info from that puller log?
<mwhudson> thumper: no
<mwhudson> thumper: it looks totally impossible :/
<thumper> :(
<thumper> did you cowboy any more logging?
<mwhudson> no, no more yet
<spm> sun spots?
<bigjools> I hear that Dallas has the right kind of hats for us
<thumper> spm: what is the branch puller delay?
<spm> thumper: I think it's this one: select greatest(0.0, extract(epoch from (CURRENT_TIMESTAMP AT TIME ZONE 'UTC' - min(next_mirror_time)))) from branch where branch_type <> 4; or a variant of.
<jtv> mrevell: why does staging *still* say it's 2.2.7?  To confuse the enemy?
<jtv> Same for edge, actually
<mrevell> jtv: I think danilos or beuno have a branch to fix that
 * beuno has no suck branch
<beuno> *such
<beuno> wow
<beuno> I have filed a bug though
<jtv> We could file one automatically for every cycle.
<jtv> (No, _not_ serious)
* mthaddon changed the topic of #launchpad-dev to: Code hosting offline 10.00-10.40 UTC 1st Oct -- http://is.gd/3MyHY | This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
* mthaddon changed the topic of #launchpad-dev to: This is Launchpad Development Channel | Week 00 of 3.1.10 | PQM is OPEN | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in #launchpad-reviews | Use http://paste.ubuntu.com/ for pastes | This channel is logged: http://irclogs.ubuntu.com/
<Fly-Man-> rockstar: *ping*
<Fly-Man-> I managed to pinpoint the error back to Python that doesn't like to be inside a Vmware Server 2.0.1
<Fly-Man-> The people from VMware were shocked to see that this could happen and are thinking about ways to make sure this keeps working
<barry> jtv: or we could just not /have/ a version number in the footer <wink>
<barry> Fly-Man-: wow.  what did vmware say was the problem?
<Fly-Man-> Python version 2.6 from Jaunty
<Fly-Man-> It seems that over the period of time
<Fly-Man-> with the rocketfuel setup
<Fly-Man-> it sets up 2.4, 2.5 and 2.6
<Fly-Man-> and those seems not to be happy with each other and the tools that VMware released
<barry> huh.  interesting.
<Fly-Man-> at least, not with the machine i'm running it on :p
<stub> select milestone.name
<stub> from milestone,product
<stub> where product.name='launchpad'
<stub> and product.id = milestone.product
<stub> and dateexpected < current_timestamp at time zone 'utc'
<stub> order by dateexpected
<stub> limit 1;
<stub> which would work if we bothered to keep our data up to date ;)
<stub> oh - we do. I just forgot the desc
<stub> select milestone.name
<stub> from milestone,product
<stub> where product.name='launchpad-foundations'
<stub> and product.id = milestone.product
<stub> and dateexpected < current_timestamp at time zone 'utc'
<stub> order by dateexpected desc
<barry> kfogel: i am an idjit :)  see my last follow up on that thread.  i think there's no bug there
<kfogel> barry: so, uh, what was the bug Henning saw, though?
<barry> kfogel: i don't know
<thumper> abentley: hello
<thumper> ?
<abentley> thumper: Hi.
<thumper> abentley: hey, how's it going?
<abentley> It's going fine.  Today just started, but I'll be trying to get an improved bzr cherrypicked.
<abentley> thumper: And then back to the inline-commenting stuff.
<thumper> abentley: how much faster is the preview diff generation with your patch?
<thumper> abentley: and has that patch gone/going into bzr.dev?
<thumper> abentley: it is bzrlib isn't it?
<abentley> thumper: On my example test, it is 22x faster, but that will vary a lot because it changes O(n^2) scaling to O(n) scaling.
<thumper> excellent
<thumper> example test was what?
<thumper> not the mysql test?
<thumper> I have the mysql branches locally
<thumper> if you want me to compare
<abentley> thumper: It has landed in the 2.0 branch, and will propagate to bzr.dev and appear in the next release.
 * thumper nods
<thumper> has an official tarball available, or are we going to add our own?
<abentley> thumper: I don't have the patience to run the sql merge without the patch, but with the patch, it completes in about a minute.
<thumper> awesome
<thumper> without the patch it is over an hour
<thumper> (on my machine)
<abentley> thumper: Just under an hour on loganberry.
<thumper> that's awesome
<thumper> so...
<thumper> adding a new bzr tarball to the download cache?
<thumper> how is the inline editing widget work coming along?
<jtv> code folks!  How do I create a stacked branch in a test?  Is passing stacked_on to factory.makeBranch enough?
<abentley> thumper: Yes, and setting up a corresponding branch.
<thumper> jtv: yes, if you just want a stacked test branch
<thumper> jtv: it won't create the branch on disk
<abentley> thumper: I've got it working just for comments, and it puts the comment in the wrong place, and it doesn't support replies.
<thumper> oh kay...
<jtv> thumper: I want to reproduce bug 375013 in a test, basically
<mup> Bug #375013: Commit to a stacked branch does not work <commit> <stacking> <Bazaar:Triaged> <https://launchpad.net/bugs/375013>
<thumper> jtv: ah
<thumper> jtv: you want to create real branches
<jtv> yes
 * thumper tries to remember test examples
<thumper> jtv: lib/codehosting/scanner/tests/test_bzrsyncd.py:537
<thumper> jtv: is an example of creating a real stacked branch
<jtv> thumper: just what I was hoping for, thanks.
<jtv> separate call to set_stacked_on_url, I see.
<jtv> That'd do it, yes  :)
<jtv> thumper: stacked_on_branch is kinda sucky as a variable name... why not call your branches "pea" and "mattress"?
 * thumper shoots jtv with a pea shooter#
<jtv> (Of course since I'm testing a stacking limitation, my test would be called Princess)
<thumper> jtv: i'd have to reject that branch
<jtv> On *no* account look up "It's Princess!" (by the makers of South Park) on the intertubes
<jtv> thumper: but you'd be grinning while doing so
<gmb> Anyone got any idea why I'd be seeing timeouts in googletestservice.py during a test run?
<thumper> ah netsplit
<thumper> gosh isn't bigjools annoying
<thumper> heh
<thumper> gmb: ping
<gmb> thumper: Hi
<thumper> gmb: https://code.edge.launchpad.net/launchpad/+activereviews
<thumper> gmb: you have two old approved branches
<thumper> gmb: have they landed?
 * gmb looks
<thumper> gmb: they may be not identified correctly as we switched formats around then
<gmb> thumper: No, they're branches that are waiting on subsequent branches before they can be landed. They're still unmerged.
 * gmb wishes there was a way to express that that +activereviews could understand.
<thumper> gmb: why can't they be landed?
<thumper> gmb: what would happen now if you land them?
<thumper> gmb: will it break something, or just be unused code?
<gmb> thumper: Things would break.
 * thumper wonders how to show this
<gmb> thumper: They're large branches, or rather they're logical blocks of large branches.
<gmb> Which I had to leave because of more pressing work.
<thumper> hmm..
<thumper> ideally we'd want to have a way to say "this is approved but blocked"
<thumper> so you won't get people like me bitching about it
<gmb> Heh.
<gmb> Right :)
<gmb> thumper: Want me to file a bug for this so that you can get on with team leadery things?
<thumper> sure
<gmb> Will do.
<gmb> thumper: https://bugs.edge.launchpad.net/launchpad-code/+bug/440011
<mup> Bug #440011: There should be a way to say that a merge proposal is approved but blocked <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/440011>
<thumper> ta
<henninge> maybe I should ask here ...
<henninge> I want to use a TAL formatter in a view and I found out I can do it this way: https://pastebin.canonical.com/22831/
<henninge> Is that the right way to do it? Or should I go through the registration somehow? I remember seeing an example once that was different ...
<rockstar> Ursinha, sorry about missing the meeting.  Drop off at the vet took longer than expected.
<Ursinha> no problem rockstar
<Ursinha> do you know why is update_review_diffs script failing?
<rockstar> Ursinha, yes, and abentley had a hack to fix it, and I think a correct fix as well.
<abentley> rockstar: The "correct" fix stops it failing in this case, but not all cases.
<rockstar> abentley, yeah, I think you mentioned that last night.
<abentley> Ursinha: It's been discussed on the list.  See the threads under Scripts failed to run: loganberry:update_preview_diffs
<Ursinha> thanks abentley and rockstar
<poolie> my first MP for launchpad! :-) https://code.edge.launchpad.net/~mbp/launchpad/mbp-trivial/+merge/12732
<thumper> beuno: https://code.edge.launchpad.net/~loggerhead-team/loggerhead/trunk-rich
<beuno> thumper, I LOVE YOU
<kees> gmb: /me is here now.  :)
 * kees waves to bryce__ 
<bryce__> whew
<bryce__> debug output and test script posted to https://bugs.edge.launchpad.net/malone/+bug/329917
<mup> Bug #329917: Changing a task's target using the API OOPSes as NotImplementedError <api> <oops> <ubuntu-qa> <Launchpad Bugs:Confirmed> <https://launchpad.net/bugs/329917>
<gmb> kees: I think we might have discovered a race condition here.
<gmb> kees: Normally, that NotImplementedError will only get called if you try to access the bug using a target on which it isn't reported.
<gmb> kees: e.g. https://bugs.launchpad.net/evolution/+bug/42 when bug 42 isn't reported on evolution.
<mup> Bug #42: Bug description listed in task is not the correct description <Launchpad Bugs:Fix Released by bradb> <https://launchpad.net/bugs/42>
<mup> Bug #42: Bug description listed in task is not the correct description <Launchpad Bugs:Fix Released by bradb> <https://launchpad.net/bugs/42>
<gmb> kees: So it's almost as if the new bugtask is being accessed before it actually exists, which seems strange.
<kees> gmb: I haven't caught any OOPSes myself (how can I find an OOPS via the API?) but I always see the HTTP-500 error.
<gmb> Hrm.
<gmb> kees: OOPSes aren't available via the API because they're not part of LP, as such; that's a separate system that Launchpad links to.
<kees> ah-ha.  do they show up in the html test from the API failure?
<bac> barry: have you run ec2 lately?
<gmb> kees: I'm not sure; I'll find out for you.
<gmb> kees: So, I don't know what's causing the problem but I at least understand the root cause of the problem. I'll update the bug and we'll get someone on it tomorrow.
<gmb> (since we're all in the UK atm)
<kees> gmb: ok, great, thanks!
<maxb> bzr 2.0.0-lp-1?
<EdwinGrubbs> BjornT: ping
 * kees hugs matsubara for r9611 :)
<matsubara> kees, it's probably not me who you want to hug :-)
<kees> matsubara: oh, er, well, you touched the bug.  :)
<matsubara> kees, I just run a script that parses commit logs for bug and mark them as fix committed
 * kees looks for Muharem
<mwhudson> good morning
<kees> intellectronica: uhm... bug 329917 is certainly not invalid.
<mup> Bug #329917: Changing a task's target using the API OOPSes as NotImplementedError <api> <oops> <ubuntu-qa> <Launchpad Bugs:Invalid> <https://launchpad.net/bugs/329917>
<intellectronica> kees: read my explanation? it's a different bug from the one described originally and marked as duplicate, and needs to be fixed client-side
<intellectronica> kees: i'm looking at the real bug as we speak
<kees> intellectronica: which should be the master for the issue pitti, me, bdmurray, and bryyce are seeing?
<kees> intellectronica: aH, 342355
<intellectronica> kees: i think bug #342355 is the real bug. the one Ursinha-afk filed is a red herring. it would be nice to fix it (in launchpadlib, not launchpad) at some point, but it's not a show stopper
<mup> Bug #342355: transitionToTarget for an Ubuntu package yields HTTP 500 error <Launchpad Bugs:New> <https://launchpad.net/bugs/342355>
<intellectronica> i also described a workaround
<kees> intellectronica: ok, cool.  I'm interested in 342355 then.  :)  can you milestone/confirm/high that one instead?
<mwhudson> maxb: you there?
<intellectronica> kees: i'm investigating that one now and will update as soon as i know what's involved. i suspect it's an easy one and can get a fix out quickly
<kees> intellectronica: rockin'!  thanks :)
<intellectronica> kees: do you happen to have the foo to make me part of the ubuntu-bugs team (on staging)?
<kees> intellectronica: let me see
<kees> intellectronica: doesn't look like it, let me poke bdmurray
<abentley> rockstar: Have you tried "make sync_branches" recently?
<kees> intellectronica: bdmurray is afk for lunch, it seems... let me see if I can find jcastro
<rockstar> abentley, no, but I remember that thumper and I had an issue with it when we were messing with the branch rewrite stuff.
<intellectronica> matsubara: can you? ^^^^^ i think you've got admin foo powers?
<rockstar> abentley, I think there was a configuration issue.
<rockstar> abentley, although it looks like mwhudson is awake and alert, so he might have some insight.
<abentley> mwhudson: Have you tried "make sync_branches" recently? (It's not scanning the branches for me)
 * mwhudson tries to work out what's in trunk
<matsubara> intellectronica, I don't have lp admin powers
<mwhudson> abentley: it's possible the "pull_branches" rule isn't right
<mwhudson> abentley: does 'make pull_branches' seem happy?
<kees> intellectronica: hrmpf, all the admins are idle.  :P  is there some action you need to take that I can help with instead?
<mwhudson> abentley: another problem can be that /var/tmp/bazaar.launchpad.dev/mirrors/ ends up 0700 for some reason
<mwhudson> kees: mbarnett should be here
<abentley> mwhudson: Yes.  Earlier runs of sync_branches reported successful mirroring.
<mwhudson> abentley: odd
<intellectronica> kees: no big deal. i'll need to reproduce it locally anyway, just wanted to see what actually happens one you run your script and get a stack trace, but i can get that from the oops report
<kees> intellectronica: I take it back, ogasawara fixed it.  you should be in now
 * mbarnett looks around
<kees> mwhudson, mbarnett: sorry, I meant ubuntu-bugcontrol admins.  :)
<intellectronica> kees: awesome, thanks
<mwhudson> mbarnett: ah, sorry
<abentley> mwhudson: This looks like 0755
 * mbarnett wanders off again 
<mbarnett> mwhudson: np
<mwhudson> abentley: when you say "not scanning" do you mean the scanner isn't even looking at the branches you think it should?
<abentley> mwhudson: the branch scanner script reports 2009-10-01 18:54:29 INFO    OOPS-1370BS1: Not a branch: "lp-mirrored:///~mark/bzr/bzrtools-silly". (~mark/bzr/bzrtools-silly)
<abentley> mwhudson: I think the puller is putting the branches in the wrong place...
<mwhudson> abentley: or cocking up the stacked_on_url
<abentley> mwhudson: No stacking enabled for this project.
<mwhudson> ok, that's one thing down :)
<abentley> mwhudson: I looked at /var/tmp/bazaar.launchpad.dev/mirrors/ and the branches weren't there.  (there were two presumably very old branches)
<mwhudson> ok that's screwy
<mwhudson> huh, the puller isn't working at all for me for some reason...
<mwhudson> abentley: need to have coffee before looking at this more, brb
<abentley> mwhudson: cool
<abentley> mwhudson: I have an idea-- I bet you apache's mad at me.
<rockstar> abentley, you could say that all the time and 60% of the time you'd be right.
<intellectronica> kees: i have a fix for your bug. review and test run, and if everything goes ok it should be fixed on edge after the next rollout
<mwhudson> abentley: working now
<mwhudson> ?
<abentley> kinda working.  It's got the wrong branch data, but that might be my fault.
 * kees hugs intellectronica
<abentley> mwhudson: It may be because I didn't have an /etc/hosts entry for bazaar-internal.launchpad.net
<mwhudson> yes, that would certainly prevent the scanner from working
<mwhudson> (i presume you mean bazaar-internal.launchpad.DEV ?)
<abentley> mwhudson: Yes.
<abentley> mwhudson: Up for a chat about jobs and code import?
<mwhudson> abentley: ok, one sec
<Ursinha> intellectronica, I understand what you mean in that bug, I think we should create one bugtask for launchpadlib, and mark the bug as dupe in case we find the lplib bug you mentioned
<intellectronica> Ursinha: i think that's a good idea
<Ursinha> intellectronica, done, then :)
<intellectronica> Ursinha: great, thanks
 * rockstar lunches
<leonardr> james_w, version 1.5.2 of launchpadlib has been released for your packaging enjoyment
<matsubara> cprov-afk, when you're around could you take care of https://answers.edge.launchpad.net/launchpad/+question/81974 please?
<cprov-afk> matsubara: uhm ... sure
<matsubara> cprov-afk, thanks!
<cprov-afk> matsubara: actually, I can't
<cprov-afk> matsubara: I'm not a soyuz-member anymore :`(
<matsubara> cprov-afk, oh!
<matsubara> well, since the user is OK with the team being deleted, I think I'm going to suggest that.
<matsubara> cprov-afk, do you know if merging the team away be blocked by the ppa linkage?
<cprov-afk> matsubara: merging is allowed
<matsubara> cool. I'll ask the losas then
<matsubara> thanks cprov-afk
<cprov-afk> matsubara: np
<mwhudson> abentley: lib/canonical/twistedsupport/processmonitor.py
<wgrant> Oh, cprov's leaving? :(
<mwhudson> left, even :)
<mwhudson> :( rather
<mwhudson> i think
<rockstar> Yea, his last day was yesterday.
<wgrant> Wow.
<mwhudson> however what with launchpad now being open source, i'm sure he won't escape that easily :)
<wgrant> Heh.
<barry> rockstar: ping
<barry> EdwinGrubbs: ping
<rockstar> barry, yo
<barry> rockstar: hi.  do you have a few minutes to look at some screenshots?
<rockstar> barry, yes.  I still plan on being around for a bit.
<barry> rockstar: cool, thanks.  i'm playing around with some fixes for bug 440220
<mup> Bug #440220: registering slot interferes with watermark <post-ui-3-cleanup> <Launchpad Registry:In Progress by barry> <https://launchpad.net/bugs/440220>
<barry> rockstar: there are some screen shots attached to that bug report.  the first one is a snap of the current problem, then three snaps of some fixes
<barry> rockstar: i'm very interested to know what you think.  note that this approach is keeping the registering slot where it currently is and just playing with layout and nowrap
<barry> rockstar: but another possibility of course is to move the registering slot to a different location
<barry> rockstar: anyway EOT
#launchpad-dev 2009-10-02
<barry> rockstar: thanks for the comments.  i feel the same way, but i wanted to explore this option.
<rockstar> barry, yea, exploring this is pretty cheap, so we should see all the options.
<maxb> mwhudson: Hi
<mwhudson> rockstar: how much work is involved in getting some bug-status like love for merge proposal status?
<mwhudson> rockstar: (just curious, really)
<mwhudson> maxb: hello
<mwhudson> maxb: i wonder why i was pinging you :-)
<maxb> :-)
<mwhudson> oh right, that was it
<rockstar> mwhudson, thumper's working on it, and I was told not to do it for him.  :)
<mwhudson> maxb: i think it would be a good thing to copy a few packages from karmic to the ~launchpad ppa (python-launchpadlib & dependencies) and have launchpad-developer-dependencies depend on them
<mwhudson> maxb: does that sound sane?
<mwhudson> i've just installed a bunch of karmic packages on jaunty, seems to be working, dunno about hardy...
<mwhudson> rockstar: heh heh
<maxb> hm. What are they needed for?
<wgrant> That doesn't sound like a good idea.
<wgrant> Won't it just break everything on the other releases with the lazr.* setuptools mess?
<rockstar> mwhudson, I know he's got the branch status stuff in review right now, and that deryck has reviewed it.
<mwhudson> maxb: 'ec2 land' does stuff with launchpadlib and can't use the eggs we have because launchpad is still stuck on 2.4
<mwhudson> maybe the real fix is to get launchpad on 2.5 asap
<rockstar> maxb, mwhudson, wgrant, I might point out that it's been a habit that l-d-d tends to break in awkward ways every time we switch versions.
<mwhudson> barry: hello
<mwhudson> barry: Launchpad.  Python 2.5.  know what's going on there?
<rockstar> mwhudson, I think gary was working on that.
 * wgrant thought maxb was.
<maxb> mwhudson: erm "can't use the eggs we have" - hmm?
<wgrant> Current 2.5 test failures are at https://dev.launchpad.net/LaunchpadOnKarmic
<rockstar> Although it seems like gary is working on a lot of things, which may explain why it's slow going.
<mwhudson> wgrant: oh that's a very useful page :)
<maxb> rockstar: um, you can point that out, but what does that mean for the current discussion?
<mwhudson> maxb: well, can't us /bin/py at least, because boto depends on python 2.5
<barry> mwhudson: hi.  i don't.  i'd love to help this along
<barry> mwhudson: /me -> dinner
<maxb> mwhudson: ah, ok- that makes sense
<maxb> Status of LP on Python 2.5 is that people, me included, need to work on whittling the list on LaunchpadOnKarmic
<maxb> If 'ec2 land' requires a system-python launchpadlib newer than available in older ubuntus, normally I would be all for a backport.
<rockstar> maxb, LaunchpadOnKarmic is a misleading name, since I'm running Launchpad on Karmic right now.
<maxb> However, the namespace package mess is a major blocker
<maxb> You are right, the wiki page is now poorly named.
<maxb> When I started it, I assumed that 2.4 on karmic was not a possibility
<rockstar> maxb, I point this out in saying that we did this exact same thing with this same package on the Intrepid->Jaunty jump that caused breakages, but we weren't too concerned since it worked for Launchpad.
<maxb> huh? 'exact same thing'
<maxb> ?
 * mwhudson lunches
<mwhudson> wgrant: i think i have just thumped into the "setuptools lazr namespace package" issue
<mwhudson> wgrant: do you know a workaround?
<mwhudson> oh, i think there's one on that handy wiki page
<mwhudson> oh man W T F
<mwhudson> apache is broken on my machine because fricking WINDMILL chokes on import if $PATH isn't set
<mwhudson> well not broken, but unhappy
<wgrant> mwhudson: I don't have any real breakage from the lazr setuptools issue.
<wgrant> Apart from a few tests.
<mwhudson> wgrant: i didn't have lazr.enum installed system wide
<mwhudson> which seemed to break scripts/mirror-branch.py rather thoroughly
<wgrant> mwhudson: Yep, codehosting script tests are broken.
<wgrant> Fortunately I haven't had cause to touch codehosting much lately.
 * wgrant lunches.
<mwhudson> thumper: come back, it's too quiet here
<MTecknology> You should all feel sorry for me..
<MTecknology> I'm about to make a sync from Launchpad to Drupal...
<MTecknology> for 4 projects
<wgrant> This side of the world is rapidly running out of LP engineers.
<MTecknology> wgrant: I'll be trying to do some LP dev pretty soon
<wgrant> MTecknology: You are not on my side of the world!
<MTecknology> wgrant: I could be
<MTecknology> more and more, cvs irks me
<MTecknology> I know I made changes - but cvs won't listen
<mwhudson> hmm
<mwhudson> make build is failing on the python2.5 branch
 * mwhudson EOWs
<al-maisan> Good morning!
<mwhudson> hello al-maisan
<al-maisan> hello mwhudson :)
<adeuring> good morning
<thumper> morning
<thumper> mwhudson: back soon :)
<wgrant> How is the team lead thingy going?
<thumper> wgrant: interestingly
<thumper> wgrant: there'll be some email sent to the dev list at the end of today I expect
<mrevell> Morning
<thumper> mrevell: morning
<henninge> Hi, I found out that I can do this to use a TAL formatter in view code: http://paste.ubuntu.com/283641/
<henninge> Is that a "legal" use of the API or should I access the formatter in some other way?
<henninge> jtv: ^?
<jtv> henninge: all I can say is it's clever and simple.  :)
<henninge> jtv: thanks ;) So you'd let that through on a review? I guess that is what I am asking ...
<jtv> henninge: curtis is the real person to ask imho
<jtv> but I would allow that, yes.
<henninge> jtv: oh, and he's in our time zone atm ... ;)
<jtv> or close to it
<henninge> sinzui: Hi! Is it OK to do this in a view? http://paste.ubuntu.com/283641/
<bigjools> wgrant: hey there - did you ever get anywhere with the download counters for PPAs?
<wgrant> bigjools: The refactoring of the existing log parser is done, so the PPA backend is very very easy.
<wgrant> But it needed discussion, so I decided to wait until post-3.0.
<wgrant> (the refactoring is done and merged, that is)
<bigjools> awesome
<bigjools> we'd like it to work with the partner archive too :)
<wgrant> So I saw.
<al-maisan> hello code hosting guys :)
<al-maisan> diffs in merge proposals seem to be off quite a bit
<al-maisan> see e.g. https://code.edge.launchpad.net/~al-maisan/launchpad/cmsg-425800/+merge/12774
<al-maisan> and http://pastebin.com/m4c94727b
<sinzui> henninge: sorry I am sprinting. I think your pastebin, if fine. I have used tales in code several times.
<henninge> sinzui: I am aware of you sprinting ... ;) Thanks for taking the time to look at my pastebin.
<noodles775> al-maisan, you're aware that you've targeted db-devel there?
<al-maisan> noodles775: oops, my fault then.
<al-maisan> I apologise to the code hosting guys ;P
<noodles775> al-maisan, I've done that too... start a branch when devel is closed and forget later :)
 * al-maisan needs some coffee
<al-maisan> bbiab
<wgrant> bigjools: My quick PPA BPR counter implementation from several weeks ago is only about 300 lines of diff.
<noodles775> wgrant or bigjools: could one of you take a look at my comments at the bottom of https://answers.edge.launchpad.net/launchpad/+question/84457 ?
<noodles775> I'm confused why the Packages file would be empty rather than simply containing the previous versions?
<wgrant> Lessee.
<wgrant> Hmm.
<wgrant> That is a good question.
<wgrant> Hm.
<wgrant> Looks like it never built on Hardy.
<wgrant> I can't find a successful Hardy build (they're all superseded), while the rest are fine.
<wgrant> I presume it keeps depwaiting.
<wgrant> We'll see in a minute or two.
<noodles775> wgrant, https://launchpad.net/~inkscape-nightly/+archive/ppa/+build/841006
<noodles775> (and thanks for taking a look)
<wgrant> noodles775: Ah.
<wgrant> So it did succeed months ago.
<wgrant> But.
<wgrant> It was deleted.
<wgrant> 0.46+devel~nightly8.04-7213-1 was the last success.
<wgrant> The subsequent upload failed.
<wgrant> Both were deleted.
<wgrant> So the binaries vanished.
<wgrant> No later upload succeeded, so there are still no binaries.
<noodles775> Ah - great.... thanks wgrant .
<wgrant> The DASBP views make that pretty easy to see, but it's rather more convoluted for a PPA.
<noodles775> Yeah.
<barry> noodles775: hi!  would you be able to do a quick ui review of some screen shots?
<noodles775> barry, sure - if it gives me a break from CHR project reviewing :)
<barry> noodles775: :)
<barry> noodles775: can you take a look at the last two screen shots in bug 440220
<mup> Bug #440220: registering slot interferes with watermark <post-ui-3-cleanup> <Launchpad Registry:In Progress by barry> <https://launchpad.net/bugs/440220>
<noodles775> looking
<EdwinGrubbs> lifeless: ping
<EdwinGrubbs> beuno: ping
<beuno> EdwinGrubbs, hi
<beuno> jml, http://paste.ubuntu.com/283879/
<EdwinGrubbs> beuno: can you give me your feedback on the mockups in bug 399554
<mup> Bug #399554: timeline series data is very hard to access <post-3-ui-cleanup> <ui> <Launchpad Registry:In Progress by edwin-grubbs> <https://launchpad.net/bugs/399554>
<beuno> let me try
<beuno> EdwinGrubbs, it's nice
<beuno> it's a good idea
<beuno> although removing the dates is... well, I don't know
<jml> beuno, thanks. as discussed, mainline commits are more interesting.
<beuno> "patches welcome", I hear
<EdwinGrubbs> beuno: statik had the idea of just hiding the dates when the graph is in a portlet. I could also do something where depending on your zoom level it would hide/show the dates, or I could add a checkbox next to the zoom buttons.
<beuno> EdwinGrubbs, ah, could be a good idea
<beuno> EdwinGrubbs, lets have a call next week about it
<beuno> would that work for you?
<EdwinGrubbs> beuno: sure. I'll set that up after I get some feedback from lifeless.
<beuno> EdwinGrubbs, great, thanks
<thumper> can anyone swap CHR with me for the 7th?
<mrevell> thumper: I'm down for the 14th. Do you wanna take that and I'll take your 7th?
<thumper> mrevell: sounds great, thanks
<mrevell> thumper: I'll update the calendar
<thumper> mrevell: ta
<mrevell> Hmm, I'm getting a traceback on ec2 test today but it was fine yesterday. Anyone else seen that?
<gmb> Argh. Why am I getting "AttributeError: 'thread._local' object has no attribute 'interaction'"?
<lifeless> EdwinGrubbs: yo?
<EdwinGrubbs> lifeless: hi, did you see the mockups in the bug you opened
<lifeless> EdwinGrubbs: hi, yes been thinking
<lifeless> EdwinGrubbs: I think they are better
<EdwinGrubbs> lifeless: The main differences between the mockups and the old textual display of series/milestones/releases are that the graph doesn't wrap milestones/releases in large series and that it limits both the height and width of the displayed data.
<EdwinGrubbs> lifeless: it might improve the usability of the graph it was possible to optionally hide/display milestones and releases based on what you interested in.
<kiko> ahoy there
<kiko> it's KARMIC UPGRADE day
<kiko> wooo
 * wgrant quickly uploads some change to break kiko's machine.
<kiko> somebody already did that
<wgrant> :(
<kiko> maybe you could upload a fix though!
<kiko> what's with the splash screen coming after all the boot messages
<wgrant> It's probably because it has to wait for KMS to initialise. Keybuk was working on that sort of thing, and it's a lot better than it was a month ago.
<wgrant> I presume it will get quieter before release.
<james_w> you get all boot messages?
<james_w> or just some apparmor/modprobe/fsck stuff
<james_w> and then a few "starting" messages?
<Fly-Man-> Hooray, Karmic is here
<wgrant> Maybe I misunderstood the full magnitude of the issue. I just get a few lines at the start.
<kiko> james_w: I only get like 10 lines after some 2 screen flashes, but it kinda upsets the smooth experience
<kiko> james_w: starting with fsck and then some other stuff
<james_w> yeah
<kiko> the boot is so fast
<james_w> I think true smoothness will be next release
<kiko> that having the flash and switch just steals victory
<james_w> I think there's work to do to quieten things down without giving no information when the boot fails
<kiko> sure, but the boot doesn't fail for me!
#launchpad-dev 2009-10-04
<mwhudson> thumper: i guess you're still in transit today?
<jml> mwhudson, ping
<mwhudson> jml: hi!
<jml> mwhudson, hi :)
<jml> mwhudson, up for a quick & non-official skype call?
<mwhudson> jml: sure thing
<mwhudson> jml: call away
 * jml doing the sound dance
<mwhudson> jml: https://dev.launchpad.net/BranchingUbuntu
<jml> mwhudson, https://wiki.ubuntu.com/NewReleaseCycleProcess
#launchpad-dev 2010-10-04
<wgrant> lifeless: How old is staging's DB?
<wgrant> (I need to get a rough idea of the publications demolished by bug #653382)
<_mup_> Bug #653382: BinaryPackagePublishingHistory._getOtherPublications fails to restrict the distroseries context <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/653382>
<lifeless> wgrant: I'm not sure
<wgrant> It appears to be from the 16th.
<wgrant> lifeless: Could you please run http://paste.ubuntu.com/505395/?
<wgrant> (hm, it has a shiny new theme)
<lifeless> wgrant: terrible query
<lifeless> wgrant: not exists is probably better than count(*) == 0;
<wgrant> lifeless: Fair point.
<lifeless> also its probably flattenable into a single query rather than correlated subquery
<lifeless> which will tend to be much faster
<wgrant> How could I flatten that?
<lifeless> using group by?
<wgrant> Oh, true.
<lifeless> or possibly
<lifeless> just use a left outer where bpph.id is NULL
<lifeless> but I'd have to actually think to suggest more
<lifeless> wgrant: its going to be a while, the query is expressed poorly I think.
<lifeless> wgrant: by a while, I mean its running and has been for 10 minutes
<lifeless> I'll be back in a bit
<nigelb> 25
<wgrant> lifeless: Yeah, 3am SQL isn't optimal, as it turns out.
<wgrant> http://paste.ubuntu.com/505410/ removes the subquery, and gets rid of one of the BPPH scans.
<wgrant> And is generally a whole lot less hostile.
<wgrant> But it will still probably take a while, given what it's doing.
<wgrant> I could reduce the dataset to publications superseded since the bug was introduced, but I know there have been similar bugs in the past, so I'd like to see that there is no older broken data too.
<wgrant> http://paste.ubuntu.com/505415/
<lifeless>  123744
<lifeless> (1 row)
<lifeless> Time: 768829.442 ms
<lifeless> thats using 'not exists (select binarypackagepublishinghistory.id ...
<wgrant> Hm, that's a few.
<lifeless> 505415 is running now
<wgrant> Thanks.
<wgrant> Er, 505415 doesn't have a COUNT, so it's going to give a crapload of output.
<lifeless> when it finishes :P
<lifeless> I'll hit END
<lifeless> wgrant: the 505415 query is odd
<wgrant> Howso?
<lifeless> wgrant: if you want 'no other_bpph' then use join, not left join
<lifeless> and drop the having COUNT
<wgrant> Er. Won't a JOIN against something that doesn't exist result in... nothing?
<lifeless> right, so those rows are skipped.
<lifeless> think sets
<lifeless> not procedure
<lifeless> 3662 rows
<wgrant> Hmm. I wonder what the other 120000 are.
<lifeless> mwhudson: 'test' passed ec2, of course testfix then fcked us.
<mwhudson> lifeless: \o/, at least a little bit
<lifeless> yeah
<lifeless> will bomb them in at EOD
<wgrant> lifeless: How would you rewrite that query to not left join? I want the IDs, so joining against an empty set is not going to help me much.
<lifeless> I'd start by writing out in english what I want
<lifeless> I'm not clear what you want
<lifeless> the two versions I've seen are doing different things. (thus the different results)
<wgrant> I want to find superseded binary publications where the dominant build has no publications in that context.
<wgrant> (context == (archive, distroarchseries, pocket))
<wgrant> Er, (archive, distroseries, pocket) for this particular case.
<lifeless> so 'superseded but nothing superseding it' ?
<wgrant> Right.
<wgrant> We record the build that supersedes each publication.
<lifeless> so
<lifeless> to first address the left join/join lemma
<lifeless> if I have two tables
<lifeless> SUPERSEDED SUPERSEDES
<lifeless> (imaginary)
<lifeless> with supereseded.id == supersedes.superseded_id
<lifeless> then ED JOIN ES group by ED.id having count(ES.id) == 0
<lifeless> is equivalent to
<lifeless> ED LEFT JOIN ES group by ed.id
<lifeless> bah
<lifeless> got the LEFT in the wrong example
<lifeless> then ED LEFT JOIN ES group by ED.id having count(ES.id) == 0
<lifeless> is equivalent to
<lifeless> ED JOIN ES group by ed.id
<wgrant> Won't ED JOIN ES return the opposite of what we want?
<lifeless> oh right
<wgrant> It's the same as HAVING COUNT(ES.id) > 0
<lifeless> left join where ed.id is NULL
<wgrant> I guess that might work.
<lifeless> now, one question is whether ES is better correlated or not; if we can make it very fast to satisfy for a given ED
<lifeless> wgrant: I meanT Left join where es.ID IS null
<wgrant> Right, that makes more sense than JOIN.
 * rockstar slaps buildout
<lifeless> now one issue here
<lifeless> there is what, 100 times as many ED as ES
<lifeless> so it might be better to start with dominant builds
<lifeless> and for each dominant build return: content, publication, (a) superseded build
<lifeless> which should look at 1% of the data we're considering today, no ?
<lifeless> awol for a bit
<lifeless> bbiab
<wgrant> lifeless: Where did you pull the 100 figure from?
<wgrant> I guess starting at BPB might be a little smaller, but we would still need to select every single superseded BPPH.
<wgrant> ES's intersection with ED is massive.
<lifeless> wgrant: well, gimme a query to get stats
<lifeless> wgrant: we'll see what my ass looks like
<wgrant> It's not clear what the stats should be.
<wgrant> It's also not clear why we're trying to optimise a query that will hopefully only be run once :P
<michaelh1> yao, could you put LP: 653316 on your list please?
 * michaelh1 types on the wrong channel
<rockstar> wallyworld__, how's things?
<wallyworld__> rockstar: pretty quiet. everyone's away
<wallyworld__> i'm having "fun" with YUI
<rockstar> wallyworld__, me too!
<wallyworld__> :-)
<rockstar> wallyworld__, it's quite possible my fun will be breaking yours.  :)
<rockstar> wallyworld__, are you working on the thing you and I talked about last week?
<wallyworld__> yeah, got the endpoint all set up. just having fun figuring out the YUI object model so I can get the info I need from client side and send it to the server
<wallyworld__> rockstar: i just got a call from my son who is sick at school. i have to duck out briefly to pick him up
<rockstar> wallyworld__, okay.  I just wanted to make sure you had someone around with thumper out this week.
<lifeless> wgrant: several reasons:
<lifeless>  - if the data model doesn't support this sort of query, we're going to be doing it more than once, eventually.
<lifeless>  - e.g. in the garbo. And 10 minute queries are flat bad [in the absence of dedicated data mining clusters]
<lifeless>  - You're not going to want to fix up the prod data? We can't sensibly run a 10 minute query on lpprod
<wgrant> True.
<lifeless> hmm
<wgrant> lifeless: How long did 505415 take on staging?
<lifeless> 600941.564 ms
<lifeless> statik: don't cringe at the ui, but - https://edge.launchpad.net/+feature-rules
<lifeless> statik: thats our 'takes effect immediately' config system, up and live
<lifeless> statik: we can also use it to drive A/B tests and similar experiments
 * jtv peeks as well
<wgrant> lifeless: Hmm. Not fantastic. But it'd probably be faster on a production slave, and we do already have some regular multi-minute queries (on the master, no less).
<lifeless> wgrant: please file bugs for all those queries that you know of.
<lifeless> wgrant: any query over 5 seconds is a bug.
<wgrant> (although I know how to optimise those queries down to a few seconds, I haven't got around to doing it yet)
<wgrant> They're in the publisher.
<lifeless> wgrant: any *write* query over 1 second is a bug.
<lifeless> wgrant: doesn't matter.
<lifeless> wgrant: I should say, any write transaction, from the first update/insert/delete through to commit, taking more than 1 second, is a bug.
<wgrant> Right.
<statik> awesome
<wgrant> The publisher is itself pretty much a bug. We all know that.
<statik> hey jtv! how about a phone call?
<lifeless> wgrant: I like having bugs if *someone* knows there is a problem.
<jtv> statik!  Not sleeping yet?
 * jtv scrambles around for headsets
<statik> jtv: nosir, i hung around specially for you
<jtv> :)
 * jtv takes leisurely sip of coffee
<lifeless> wgrant: so, please file them?
<lifeless> wgrant: if you get pushback, I'll go to bat over their existing.
<wgrant> I suppose I might be able to use that same optimisation in the repair query.
<wgrant> Mm, no, not really.
<wgrant> It makes too many assumptions that are now broken.
<lifeless> stub: https://lpbuildbot.canonical.com/builders/prod_lp/builds/120/steps/shell_7/logs/summary
<lifeless> stub: thats a prod-lp run, the same passed ec2 earlier
<lifeless> stub: I'm wondering if its a python/twisted version issue?
<stub> lifeless: No such file lib/lp/services/scripts/tests/cronscripts.ini
<stub> lifeless: That is a relative path, so a current working directory issue
<lifeless> *blink*
<stub> No idea how it got through ec2 though if that is really the case\
<lifeless> stub: I landed the CP using ec2land
<lifeless> stub: its 50 or so revs that have all passed qc
<lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/cp/+merge/37345
<stub> lifeless: maybe a isolation/test ordering issue with an earlier test changing the cwd?
<lifeless> stub: could be; maybe BaseLayer should check that cwd hasn't been fucked with
<lifeless> in bzrlib we ensure its not messed up in the base class, *and* we don't use chdir except in very controlled circumstances (because its not nice as a library to use it :P)
<stub> It does already :-/
<lifeless> ><
<lifeless> mwhudson: wow, something chomped on the encoding in that mail
<stub> Does that file still exist in the branch that failed?
<lifeless> its production-devel
<lifeless> checking
<lifeless> stub: yes
<lifeless> bzr ls lib/lp/services/scripts/tests
<lifeless> lib/lp/services/scripts/tests/__init__.py
<lifeless> lib/lp/services/scripts/tests/cronscripts.ini
<lifeless> ...
<stub> Can add the cwd to the error if that file isn't found to check my initial theory (although I still can't see how it would be possible, it is my best guess).
<lifeless> it could be a dict order thing between python versions
<lifeless> causing the layers to be grouped in a different order.
<lifeless> stub: alternatively, where are we at in terms of getting to python 2.6 everywhere? [db server upgrades]
<lifeless> forwarded you the ec2run results for the branch
<stub> We have done hackberry, we have chokecherry, wildcherry, poha and plantain to go. As far as Python 2.6 everywhere, the important one is wildcherry
<lifeless> whats the eta on wildcherry?
<lifeless> I've never heard of poha and plaintain :(
<stub> I was planning to do it after chokecherry, but there is nothing stopping us doing it next now I think about it.
<stub> poha and plantain are the SSO database servers.
<lifeless> ah kk
<lifeless> stub: rs=me if you wanted to land something to debug this, but I think upgraded bb to python2.6 for prod_lp is probably best, if we can get wildcherry done.
<lifeless> stub: its past EOD for me, so I'll only be sporadically around now.
<lifeless> (you'd need to send it straight to production-devel)
<jtv> lifeless: you wouldn't happen to know where the branch scanner does its get_r[ow]_server() and start_server(), would you?
<jtv> and hi, btw :)
 * jtv now sees the eod note
<noodles775> Hi jtv1, is there one particular view in translations that displays multiple selectable items for form processing which you'd recommend looking at? I want to compare with teh way we do it for copy/delete packages in case there are better examples.
<noodles775> Specifically, creating a vocabulary based on the current view's search terms...
<danilos> noodles775, we've got some stuff like that, but none of that is, as you put it, "recommended to look at" :)
<noodles775> heh, OK. Thanks danilos.
<danilos> noodles775, looking at import queue stuff is of that type (lp.translations.model.translationimportqueueentry and on from there)
 * noodles775 looks
<adeuring> good morning
<jtv1> noodles775: whatever you do, don't look at our languages list.  :)
<jtv> hi adeuring
<adeuring> hi jtv!
<noodles775> danilos, jtv: OK, it looks like you guys similarly use a VocabularyFactory when its dependent on the context object, ah, but you pass the view into __init__, that's what I was looking for. Thanks!
<noodles775> Morning adeuring, bigjools
<adeuring> hi noodles775!
<jtv> danilos: uh-oh, that sounds like he may imitate something we did.  Should we stop him?
<bigjools> morning all
<jtv> morning bigjools!
<danilos> jtv, heh, in general yes, but let's run an experiment and see where it takes him :)
<danilos> bigjools, morning
<noodles775> jtv: heh, afaics, it's the most zope-like way to do it (using a vocabulary factory which is evaluated when the view is initiated etc.
<jtv> danilos: good pointâ¦ this way he gets the blame or we get the credit
<noodles775> lol
<jtv> Take.
<jtv> Did I say "get" the credit?
<jtv> mwhudson: do you know if getting, starting, and stopping servers as returned by get_ro_server() is something we can do very frequently?
<gmb> hurrah for code that fixes things for vague and (to me) confusing reasons.
<lifeless> jtv: hi, 'sup ?
<jtv> lifeless: hi, I _think_ I found the answer more or less the second you answered.
<lifeless> hah, kk
<jtv> I was trying to figure out whether a transport or server (I'm not sure of the terminology) for lp-internal:/// branch URLs was running when the branch scanner triggers a particular kind of work.
<bigjools> lifeless: hi
<lifeless> bigjools: evening!
<bigjools> lifeless: did you fix prod_lp by any chance?
<lifeless> the builder yes; the build decided to fail on some of stubs new stuff; looks like a test that is doing chdir or something
<lifeless> we *think* upgrading the builder to py 2.6 will fix it (because the cp passed ec2)
<bigjools> lifeless: I've been trying to get a CP done for a week now, this is crazy.  I am going to do a cowboy instead.
<lifeless> bigjools: have you landed the revision ?
<bigjools> yes
<lifeless> then it should be deployable
<bigjools> from prod-devel?
<lifeless> we can deploy from any branch
<bigjools> ok
<lifeless> if the rev is specified
<bigjools> lemme find it
<lifeless> but, I thought your stuff hit prod-stable last week
<lifeless> rev 9777 in prod-stable
<lifeless> isn't that your thing?
<bigjools> no
<bigjools> 9779 in devel
<lifeless> oh, 9778 and 9779
<bigjools> yeah, 9779 had the oops isolation fix in it to see if that helped
<bigjools> that fix is critical for the platform team
<lifeless> got a link to a failure from last week?
<lifeless> did you land your cp's via ec2? [If so that implies real seriously annoying fuckage in the hardy builder]
<bigjools> lifeless: https://lpbuildbot.canonical.com/builders/prod_lp/builds/119/steps/shell_7/logs/summary
<bigjools> no, withoyt
<bigjools> without
<lifeless> ahhh
<bigjools> I don't use ec2
<lifeless> I -hate- the openid glue on buildbot
<mrevell> Morning
<lifeless> that does look like the oops issue doesn't it
<bigjools> aye
<lifeless> bigjools: so my branch included your stuff and passed ec2
 * bigjools brb
<lifeless> bigjools: but i'd -really- like it if we consistently use ec2 when landing stuff on production-stable
<lifeless> bigjools: anyhow
<lifeless> bigjools: I'm +1 on a cowboy of your revisions (e.g. deploy from prod-devel rev 9779)
<lifeless> bigjools: stub: is working on wildcherry now I think.
<lifeless> I'll send mail then off again.
<wgrant> bigjools: Feel like some DB surgery?
<stub> eh?
<lifeless> stub: well, 'you' meaning you're coordinating, no ?
<bigjools> lifeless: I've never used ec2, and I'm not about to start :)
<bigjools> wgrant: wassup?
<lifeless> bigjools: that sounds unhelpful and negative
<bigjools> lifeless: what?
<lifeless> bigjools: but its after 9pm, so I'm going to ignore it and go unstrap more boxes
<bigjools> that's a particularly unhelpful comment yourself
<lifeless> blah it is
<lifeless> sorry
<wgrant> bigjools: See https://edge.launchpad.net/ubuntu/lucid/i386/python-imaging-doc -- ignoring the new Proposed publication, spot the issue.
<lifeless> so, ec2test gives us important guarantees for bottleneck branhces
<bigjools> lifeless: I run the test suite locally, always have done.
<lifeless> bigjools: so one of the things that ec2 does is run in a consistent clean state each time; do you arrange that locally as well?
<bigjools> lifeless: yes
<lifeless> bigjools: I'm fine with folk running the test suite locally, it just seems particularly hard to make local match 'what buildbot does'
<lifeless> hell its hard to even make buildbot match buildbot at the moment.
<bigjools> lifeless: if nobody ran it locally (there's 2 of us) then you'd not see local issues either
<lifeless> bigjools: I think you should ignore my previous snarky comment; am very tired, been battling a sinus infection all week, terrible beds at the hotel in sydney, and *yawn*
<lifeless> bigjools: -sorry-
<bigjools> wgrant: groan
<bigjools> lifeless: no worries, it's not good to work when tired eh? :)
<wgrant> bigjools: Yes. My fault, due to unobvious Storm quirks plus shitty old tests which missed it. Bug #653382.
<_mup_> Bug #653382: BinaryPackagePublishingHistory._getOtherPublications fails to restrict the distroseries context <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/653382>
<wgrant> Easy enough to fix, fortunately.
<lifeless> bigjools: I wasn't ;) I wandered past and saw a ping... then you pinged as well ;)
<bigjools> ah yes I saw that
<bigjools> lifeless: ok :)  thanks for the +1 on the cowboy anyway
<bigjools> wgrant: how many packages are affected?
<wgrant> bigjools: lifeless ran a query for me on staging (which has data from 3ish weeks ago), and there were around 3000 publications. I'm not sure if the query was perfect, nor how many of those were in the primary archive.
<wgrant> For those in the primary archive we just need to set them back to Published, and unset datesuperseded and supersededby, since they'll all be in Release.
<wgrant> And there should be just about no PPA publications affected, since the use cases there are different.
<bigjools> this is .... not good
<wgrant> ... yes.
<bigjools> can you show me the query please?
<wgrant> http://paste.ubuntu.com/505415/
<wgrant> (the code rolled out on 2010-08-11 or so)
<wgrant> It finds publications superseded by a build that's not published in the context.
<bigjools> what about deletions?
<bigjools> that'll pick up valid deletions
<wgrant> It won't.
<wgrant> Deletion doesn't set supersededby.
<bigjools> aha
<bigjools> umm has death row reaped these?
<wgrant> When I discovered this on Saturday, I initially thought that mass expiration last week would have killed lots of binaries that were hit by this. But it turns out that it wouldn't have.
<wgrant> No.
<wgrant> Because the dominator doesn't run over frozen pockets.
<wgrant> So they're Superseded, but have not been scheduled for removal.
<bigjools> that's lucky :)
<wgrant> If they were deathrow candidates, we would have noticed almost immediately that something was wrong.
<bigjools> what about maverick though?
<wgrant> Release pocket changes can only happen in maverick.
<wgrant> So it doesn't matter that they leak into maverick through this bug, because they were performed in maverick anyway.
<wgrant> So only frozen release pockets (and possibly some PPAs, but that's probably tiny) are affected.
<bigjools> ah maverick only has a release pocket at the moment
<wgrant> The query still respected pockets.
<wgrant> Maverick has -proposed too.
<bigjools> really? so why didn't the dominator kill maverick's packages then?
<wgrant> It did!
<wgrant> But they were meant to be killed.
<bigjools> true
<wgrant> All of the problematic dominations were meant to remove things from maverick, since that's the only place they can happen.
<bigjools> let's get the code fix in before doing this then
<wgrant> I think this is about the fourth time this code has broken :/
<wgrant> Certainly, yes.
<wgrant> Was just letting you know the details.
<bigjools> wgrant: thanks
<bigjools> the dominator code scares me :)
<wgrant> It is a bit like that.
<wgrant> This was only noticed when a lucid-proposed upload landed in NEW, which strongly suggests that nothing bad made it further than the DB.
<jml> hello
<bigjools> morning jml
<jml> bigjools: good morning.
<jml> What merriment and hijinx are in store for me today, I wonder.
 * wgrant lures jml back to engineering.
<jml> I suspect I've got more hiring than engineering to be doing.
<wgrant> There seems to have been a bit of that lately.
<jml> it never seems to stop
<wgrant> bigjools: Thanks.
<bigjools> np
<bigjools> wgrant: I'm trying to figure out if we can make your query faster
<wgrant> bigjools: So was I.
<bigjools> maybe a subselect?
<wgrant> It's possible we could use an optimisation like the one in Dominator.judgeAndDominate/
<wgrant> What sort of subselect?
<wgrant> My initial version had a NOT EXISTS subselect, and was even slower.
<bigjools> urgh
<bigjools> maybe stub has some time to help :)
<wgrant> bigjools: Does the query look otherwise correct?
<bigjools> wgrant: it's hard for me to tell, to be frank
<wgrant> Heh, yes.
<bigjools> the left join is confusing me
<bigjools> and it doesn't help that my sql is shite
<wgrant> The subselect was a bit more obvious.
<wgrant> But I'm taking each publication and left joining any publications of the supersededby build in the original publication's context.
<wgrant> Then finding those that have no publications.
<jtv1> danilos: just to let you know, the bzr plugin does what we expect (even more so than other API stuff): uses production xmlrpc on production, edge xmlprc on edge, and staging xmlrpc on staging.
<danilos> jtv1, ok, cool, thanks for checking
<jml> bigjools: has demand for PPA builds dropped or is jelmer's improvement really as awesome as the graphs make it look?
<bigjools> jml: the latter
<wgrant> It could easily have increased throughput 300%.
<bigjools> bear in mind that we used to spend up to around 20 minutes blocking on uploads per scan...
<bigjools> it's the most awesome change on soyuz I've seen in 3 years
<jml> wgrant: I've only really been looking at queue length. it's much shorter, which presumably means that people are getting what they want much faster.
<jml> bigjools: yeah. it's amazing.
<wgrant> It will be interesting to see how it copes in a couple of days when all the builders disappear.
<jml> bigjools: also shows that sabdfl was probably right to refuse more hardware
<bigjools> jml: once we get this change done that we're working on too, then it'll be more awesomererererer than anything
<bigjools> jml: totally
<bigjools> we always knew utilisation was crap
<jml> although, as wgrant says, we'll find out if we need more *dedicated* hardware fairly soon.
<bigjools> yip
<wgrant> So the graphs all look healthy? My occasional /builders checks have had the queues pretty much constantly empty.
<jml> wgrant: if bigjools doesn't mind, I'll post a public screenshot.
<bigjools> jml: I was going to blog about it
<bigjools> meant to do it Friday and forgot
<jml> bigjools: I reckon it's definitely blog-worthy.
<wgrant> I have seen a couple of reports of builds stuck in Uploading, though.
<bigjools> jml: if you want to paste something for wgrant that's ok
<bigjools> wgrant: yeah, known issue, it's when they fail to upload
<wgrant> Ah, k.
<wgrant> Apart from that, it seems to have gone incredibly smoothly.
<bigjools> it had 2 months of dogfooding :)
<jml> wgrant: http://people.canonical.com/~jml/Active-Builders.png
<jml> wgrant: we changed the way the green bit was calculated on the 30th
<wgrant> jml: It was originally the total?
<jml> yeah.
<bigjools> the graphing app is a little restrictive
<wgrant> That certainly looks blog-worthy.
<jml> bigjools: I spent a few minutes doing some research the other day. Looks like the only decent web-based graphing app is hosted in a google data center.
<bigjools> heh
<wgrant> Although the graph would be more impressive if the old data was corrected, so the reduction in utilisation was more obvious.
<wgrant> Hmm. I suppose this also means that we'll be able to do rebuilds without destroying the world for two weeks?
<bigjools> the important bit is the queue length
<wgrant> It is.
<bigjools> we want as much utilisation as possible
<bigjools> rebuilds will certainly be better!
<bigjools> yay for codebounce being down
<jml> bigjools: that seems a non sequitur
 * bigjools has secateurs and is not afraid to use them
<jml> bigjools: if you wanted as much utilization as possible, then the important thing would be the red bit â not seeing any green.
<bigjools> jml: yes - however, the current graph will never do that since the builders are not marked as building until the files are dispatched
<bigjools> that'll change when we release the stuff we did
<wgrant> How well does your new thing survive restarts?
<jml> bigjools: right, but that means that the queue length is *not* the important part.
<bigjools> jml: it was, but as we get better at utilisation, I agree
<jml> bigjools: well, I happen to only care about increased utilization insofar as it reduces the wait time for our users.
<wgrant> I can't say I trust tuolumne's data, though... how is the maximum number of active builders not integral?
<wgrant> jml: But wait time is better estimated by queue size, as bigjools says.
<bigjools> jml: yes - but remember the graph works off the data in the DB - so although we are making great utilisation of the builders now, the graph is incorrect.
<jml> wgrant: yeah, I know. that's what I'm getting at: wait time is the important thing; queue length is a good proxy measure; statements about utilization do not follow from this
<jml> e.g. we can easily increase utilization by reducing the number of builders
<wgrant> jml: Oh, right. Misread, sorry.
<deryck> Morning, all.
<jml> brb. network issues.
<stub> Anyone still on Lucid? I'm longer able to build Launchpad on my Lucid desktop and am still sorting Maverick issues on my laptop.
<wgrant> What does it whinge about?
<stub>     ImportError: cannot import name SAFE_INSTRUCTIONS
<wgrant> update-sourcecode
<wgrant> Your bzr-builder is out of date.
<stub> Ok. Not sure how I got that out of sync.
<wgrant> What's the Maverick trouble? Not Launchpad-specific?
<stub> No, other stuff. Now I've tracked down and reported the continual Firefox crashes I should be ok but haven't tried for a few days.
<bigjools> http://blog.launchpad.net/cool-new-stuff/more-build-farm-improvements
<bigjools> jml: the graph description is wrong :)
<wgrant> bigjools: Is that a challenge to the community to make the build farm collapse under the load of more daily builds?
<jml> bigjools: was it Shakespeare who once said "Then fix it, dear Henry"
<bigjools> ummm :)
<stub> So its trying to import from bzrlib.plugins.builder.recipe, and my egg doesn't have plugins/builder, and I've rebuilt it too.
<wgrant> stub: Egg? You mean tree?
<wgrant> Oh.
<wgrant> There should be a sourcecode/bzr-builder symlink.
<wgrant> Is there not?
<bigjools> jml: did you get a chance to look at my b-m changes?
<stub> wgrant: Its a directory. Is it a symlink in your tree?
<wgrant> stub: It is a symlink.
<stub> Ahh... probably old cruft stopping update-sourcecode from doing its thang
<stub> Dunno how bzr-builder and bzr-hg got that way - it isn't *that* old.
<jml> bigjools: no, I didn't. I'll take a look now after I finish reading through sinzui's privacy mails
<bigjools> jml: ok cheers. I want to try and keep coding while I have momentum.
<jml> also, I'd just like to note for the record that even though I am incredibly bad at doing push-ups, I am trying.
<jml> off to grab a bite. back soon.
<bigjools> good plan
<wgrant> bigjools: Did you get anywhere with making my query suck less?
<jml> bigjools: it's still builderslave-resume?
<wgrant> I looked at the size of the diff and ran away.
<jml> yeah. it's a big, scary branch.
<wgrant> The size makes it scary, the changes make it scary, and the part of the system that it touches makes it scary.
<jml> yes.
<jml> Using saved parent location: bzr://stinkpad.local/builderslave-resume/
<jml> that's not going to work
<wgrant> jml: Ooh, we're getting a web designery person?
<jml> wgrant: yes.
<wgrant> Excellent news.
<jml> visual design + CSS/JS/HTML
<wgrant> Perfect.
<jml> http://webapps.ubuntu.com/employment/canonical_LPSWD/
<jml> still interviewing.
<bigjools> wgrant: bi
<bigjools> sigh
<bigjools> wgrant: no
<bigjools> jml: yes
<mars> bigjools, ping, did your cherrypick from Friday get through prod_lp?
<bigjools> mars: no
<mars> bigjools, :(
<bigjools> quite :/
<bigjools> see lifeless's email to the list
<mars> bigjools, anything I can do to help?  Is prod_lp even viable right now?
<mars> yes, I saw that.  But I don't think there is anything for me to do then besides trying to pull the py2.6 code out myself
<bigjools> mars: I've not really been following it closely, I've asked for a cowboy until it's fixed
<mars> makes sense
<mars> man, that failure in prod_lp is scary
 * jml finally gets around to filing a bzr bug from the worcestershire sprint
<bigjools> saucy
<jml> hruh
<jcsackett> danilos: ping.
<danilos> jcsackett, hi
<wgrant> bigjools: So, you're landing my domination branch?
<bigjools> yes
<jcsackett> danilos: just saw your comment on the bug 652256 (about configuring translations from front page via action menu).
<_mup_> Bug #652256: Cannot configure translations from translations front page <bridging-the-gap> <Launchpad Translations:In Progress by jcsackett> <https://launchpad.net/bugs/652256>
<wgrant> Great.
<wgrant> I shall depart for the evening, then.
<jml> stupid crappy buggy java crap.
<bigjools> s/crappy buggy//
<danilos> jcsackett, right, it seems we are working towards a similar thing from a different direction :)
<danilos> jcsackett, granted, we've been a bit stuck on this
<noodles775> Night wgrant
<jml> actually, it's threads.
<jcsackett> danilos: really, i think the focus of the bug i'm no is the configuration of the using launchpad enum; i'm happy to avoid treading on toes re: permissions.
<jcsackett> danilos: when you refer to action menus, you mean the sidebar menus we see on some applications, like answers (e.g. the menu for gdp: https://answers.edge.launchpad.net/gdp)
<danilos> jcsackett, right, I am not sure what the scope is, and it'd probably be best to discuss this in a call which includes Curtis as well
<jcsackett> danilos: agreed; just getting my information in a line right now as i figure out my day. :-)
<danilos> jcsackett, yeah for "action menus"
<jcsackett> danilos: so the intent is to not use those? i was under the impression those were the current preferred way.
<danilos> jcsackett, heh, I don't know, when we did the initial 3.0 design ~15 ago we said that we are going to avoid them, which is why those translation pages have none
<danilos> jcsackett, we later agreed to have them "where we must", but we already had a sufficiently good solution for translations pages so we didn't have to use them
<jcsackett> danilos: okay. curtis and i have a meeting in 5 (which he will hopefully be online for). i'll bring this up with him and we'll try and get back to you in 35 or so. that work?
<danilos> jcsackett, sure, though I should be on another call at about that time
<jcsackett> danilos: ok. what timezone are you (or when do you EOD) so i can make sure we get back to you in a timely fashion?
<danilos> jcsackett, heh, I am UTC+2, and I'll probably stick around for another 1h30mins
<jcsackett> danilos: dig. thanks!
<danilos> somebody at the door, brb
<jml> bigjools: you could separate out the change to logger. (it needs at least a comment and maybe a test and then it could land)
<bigjools> jml: yeah
<jml> bigjools: doing this to handleTimeout makes it a bit simpler: http://pastebin.ubuntu.com/505740/
<deryck> so are we in testfix until the python 2.6 issues sinzui referred to in reply to a couple broken build emails.
<deryck> hmmm, ok, so the lucid builders are green.
<bigjools> jml: yes thanks - it was a bit of old factoring I'd not got around to yet, thanks for the reminder :)
<jml> bigjools: so much deleted code :)
<bigjools> jml: \o/
<bigjools> jml: it's basically all the recording slave stuff
<bigjools> the b-m is looking quite simple again
<deryck> I need a team lead to sign off on my script to disable bug expiry option and notify users we'll be re-enabling this.
<deryck> maybe bigjools or sinzui could help here? ^^
<jml> bigjools: I've sent you an email with the remainder of my thoughts for now.
<deryck> jml, could I get your sign off on my re-enable bug expiry notification script?  You looked once before for email clarity.
<deryck> Now just need sign off for running on staging then lpnet.
<jml> deryck: where can I find it?
<deryck> jml, http://pastebin.ubuntu.com/505754/
<deryck> jml, or lp:~deryck/+junk/lpjunk
<jml> deryck: the script looks sound to me. I'm a bit nervous about the queries, since the last time I approved something like this, I was highly mistaken.
<deryck> jml, nervous that they'll take forever?
<jml> deryck: nervous that they'll spam everyone / do the wrong thing
<jml> deryck: I guess the debug options are a good check on that.
<deryck> jml, that's why I have the verbose/report options.  I'm going to check the data before sending email.
<deryck> jml, so I can claim your approval for staging at least?  And ping back after I have the data in hand?
<jml> deryck: deffo
<deryck> cool, thanks!
<jml> c.l.webapp.testing.verifyObject is ridiculous.
<cr3> noodles775: hi there, did you write lp_dynamic_dom_updater.js?
<noodles775> cr3: yeah, a long time ago.
<noodles775> cr3: what are you wanting to do? (I'm not sure how useful that module is generally)
<cr3> noodles775: question for you: under what circumstances will will the actual_interval be halved? I don't quite see how this could ever be true on line 319: if (new_actual_interval >= config_interval) {
 * noodles775 looks
<cr3> noodles775: as far as I can see, that will only ever be true in the event the actual_interval is doubled
<noodles775> cr3: If there are a few xhr requests that take a long time, line 308 will double the interval used between requests each time. ..
<noodles775> :)
<cr3> noodles775: so, perhaps my misunderstanding is that I was expecting the dynamic updater to also potentially decrease request interval
<cr3> noodles775: however, I now see the interval will never be lower than the config interval, only potentially higher
<noodles775> cr3: yes, it will, but looking at that code, it will never be l....
<noodles775> Yes.
<cr3> noodles775: is that the intended behavior?
<noodles775> cr3: Yes - why would you want it to go below the interval which you configured?
<cr3> noodles775: because more is better :) but I can appreciate the motivation, thanks for the explanation though
<noodles775> cr3: np. Maybe you've got a different use-case. When I wrote that, it was after we'd had lots of people with the same page open hitting the server every 5 seconds (or whatever we had configured). We were only interested in reducing load on the server if it's not responding as expected, not increasing the frequency which we'd already decided was sufficient for page updates.
<cr3> noodles775: yeah, sounds perfectly reasonable and I'll actually change my use case to that :)
<bigjools> jml: thanks, making some changes as you suggest, they all look good.
<jml> g'night all.
<jml> will be back a bit later to talk to the kiwis
<lifeless> moin
<lifeless> abentley: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html hi; could you perhaps QA bug 613958
<_mup_> Bug #613958: upload failure emails should include the upload log <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by abentley> <https://launchpad.net/bugs/613958>
<lifeless> ?
<abentley> lifeless: maybe.  I keep hitting failure modes like staging restores and buildfarm weirdness.
<lifeless> ugh
<lifeless> abentley: whats needed to be confident in deploying it ?
<lifeless> (given that qa-ok really means 'ok to release with it')
<abentley> lifeless: I don't know.  I only think about qa-ok meaning that "behaves as intended".
<lifeless> gary_poster: is https://code.edge.launchpad.net/~stub/launchpad/production/+merge/37482 on the way to pqm or does it need someone to do that? I'm happy to do the legwork...
<abentley> lifeless: I don't think there's a test for deployability that I could do that would be much simpler than testing that it behaves as expected.
<lifeless> kk
<lifeless> anything I can do to help us check that?
<abentley> lifeless: I don't think so.
<lifeless> ok; I'd like to do more continuous-ish deployment drops this week, so if there is anything I can do to help please let me know.
<gary_poster> lifeless: sorry, was talking to mars about status.  stub's branch was rejected when stub submitted it.  I don't know why it was rejected, so I just retried it, and PQM is just now getting to it.  (https://pqm.launchpad.net/).
<gary_poster> mars and I think that it will not address at least one of the five errors, the one from ec2 test.  mars is going to get a 2.5 version running and will report back our status on those five failures as soon as he knows.
<gary_poster> I'm going to switch to launchpad-ops for some questions to losas about this.
<lifeless> kk
<lifeless> fooding, brb
<sinzui> Has anyone gotten spam from jscrambler? I think they harvested names. I cannot find any suspects in our two-week-old staging db
<gary_poster> mars, stub's branch made it through pqm.  any news on test failures?
<mars> pulling prod-devel onto the laptop now
<gary_poster> ok
<gary_poster> abentley: you pointed foundations at https://bugs.edge.launchpad.net/launchpad-code/+bug/644788 but I don't know what aspect of foundations is involved.  Does the librarian serve that file?  Or is the lack of an oops id the foundations problem you identified?  Depending on what is going on, that might or might not be unexpected.
<_mup_> Bug #644788: error fetching large diff "There was a problem fetching the contents of this file." <Launchpad Bazaar Integration:Triaged> <Launchpad Foundations:New> <https://launchpad.net/bugs/644788>
<abentley> gary-brb: can't it be both?
<gary_poster> abentley: both what?
<abentley> gary_poster: the librarian is failing to serve the file, and failing to give an oops.
<lifeless> flacoste: hi; I have a call in my calendar now, with you.
<gary_poster> abentley: ah, ok.  I'll update the bug with that then.  does that have anything to do with the librarian token stuff Robert did--I mean, might it be addressed by this?
<gary_poster> lifeless, I didn't understand your concern about lucid_lp_production--do you agree that I can make an RT about this and ask flacoste to give it a high priority?
<gary_poster> We may want to get rid of it, but we're not getting rid of it this instance
<gary_poster> instant I mean
<lifeless> gary_poster: of course, RT away.
<gary_poster> cool thanks
<lifeless> gary_poster: I 100% agree it needs fixing ;)
<gary_poster> heh ok
* lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | Launchpad Development Channel | Week 2 of 10.10 | PQM is open for business | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<flacoste> hi lifeless
<lifeless> flacoste: hiya
<flacoste> lifeless: luckily, it's also on my calendar! let's skype
<mars> gary_poster, stub's branch did not fix the ec2 errors
<gary_poster> mars, any of them?
<mars> it did fix at least one of the others
<mars> going through now
<gary_poster> oh I see
<gary_poster> ok
<lifeless> flacoste: http://wiki.postgresql.org/wiki/Postgres-XC
<lifeless> flacoste: https://edge.launchpad.net/+feature-rules
<mars> gary_poster, I can reproduce the prod_lp ec2 test failure locally, am working on a fix now
<gary_poster> mars, great.  Everything else is addressed?
<mars> (locally, as in maverick + py2.6)
<mars> yes, everything else is fixed by stub's branch
<gary_poster> awesome.  Oh, you mean that ec2 test tests fail in maverick/py 2.6, mars?
<mars> yep - weird
<gary_poster> hm
<gary_poster> very
<mars> on devel: $ bin/test devscripts.ec2test.tests.test_remote -t test_email_body_is_format_result
<gary_poster> OK, I'll make a reply to stub's reply.  Thank you.
<abentley> gary_poster: We proxy the librarian because we need to control access in case it's a private merge proposal.
<gary_poster> abentley, so lifeless' token work would address this, right?
<jml> lifeless: ping
<abentley> gary_poster: doesn't lifeless' token work address all cases where we proxy?
<lifeless> jml: otp
<jml> lifeless: ok. let me know when you're off & available for a call.
<gary_poster> I believe so, yes, abentley, but I was hoping for confirmation from you that you thought that approach would work for the code team's use case.
<abentley> gary_poster: We're using the standard approach for serving files at a URL.  I don't know enough details about lifeless token approach to know whether it would work for us.
<gary_poster> fair enough.  thanks abentley
<lifeless> gary_poster: the code teams use case involves parsing the librarian content in the appserver, for now.
<lifeless> gary_poster: if its MP's
<gary_poster> lifeless: oh. :-(
<lifeless> I've raised this with tim as a possible thing to evolve in the future.
<abentley> gary_poster: however, that isn't the use case shown in the bug.
<gary_poster> lifeless: I'll record this in the bug comment then.  It sounds like Tim, you, and somebody on Foundations should figure out a way to make the librarian able to do everything over in the librarian
<gary_poster> abentley: oh, sigh.  trying to do too many things at once
<abentley> lifeless: the use case in the bug is just retrieving the raw bytes of the diff.
<lifeless> abentley: ah, I see
<abentley> lifeless: using this url: https://code.edge.launchpad.net/~jameinel/launchpad/lp-service/+merge/35877/+preview-diff/+files/preview.diff
<lifeless> gary_poster: abentley: this is a dupe of adeurings bug I think
 * gary_poster would be happy to hear that
<lifeless> this particular case will be helped by the token stuff
<gary_poster> cool
<wallyworld> morning
<wallyworld> rockstar, abently - ready for standup when you are
<rockstar> wallyworld, great.
<rockstar> abentley, skype?
<gary_poster> sinzui: would love your comment/thoughts on adding a check for bare excepts in our linter, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/636325
<lifeless> jml: ready in a few
<_mup_> Bug #636325: please add a commit ratchet for bare excepts <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/636325>
<jml> lifeless: ok.
<gary_poster> running away, back soon.
<mars> alright! we have a fix for prod_lp
<wgrant> NonZeroExitCode: Test process died with exit code 2, but no tests failed.
<wgrant> Thanks, ec2, I love you too.
<benji> heh
<jam> jml: since I see you here, when is the next lp db-devel rollout? (I have a patch based on db-devel accidentally, and I want to get it cleaned up, but don't want to pull in the next cycle's changes and have it delayed that much longer)
<lifeless> jml: 13th or so
<lifeless> jam: ^
<jam> lifeless: thanks
<jam> btw, *I* find your performance emails interesting to read
<lifeless> jam: cool, thanks ;)
<mars> lifeless, would you be willing to review this prod_lp fix?  I need to sign off soon, possibly before gary_poster returns: https://code.edge.launchpad.net/~mars/launchpad/fix-ec2test-on-production-devel/+merge/37526
<lifeless> seems plausible
<mars> lifeless, thanks
<gary_poster> mars, do you ned me to shepherd this through?
<gary_poster> and thank you
<mars> gary_poster, please - lp-land is busted on the system I'm using :(
<gary_poster> mars, ack, will do.
<mars> thank you.  gary_poster, I need to sign off now.  talk to you tomorrow
<gary_poster> bye mars
<wgrant> Can someone please re-ec2 lp:~wgrant/launchpad/bug-629921-packages-empty-filter? The last run died spuriously.
<wgrant> StevenK: When you're around, can I convince you to look at some PPA publisher logs for me?
<wgrant> StevenK: unping.
<lifeless> gary_poster: hi
<gary_poster> lifeless: hey, will be leaving in 60 secs
<lifeless> grah :P
<gary_poster> sorry :-/
<lifeless> I wanted to talk about 'definition of critical' and its relation to bugs.
<gary_poster> fair enough
<gary_poster> tomorrow?
<lifeless> sure
<gary_poster> cool.  ttyl.
<lifeless> no panic, I just suspect we have wiki pages to edit and fix up etc
<gary_poster> cool
<wgrant> lifeless: Could you ec2 https://code.edge.launchpad.net/~wgrant/launchpad/bug-629921-packages-empty-filter/+merge/37339?
<lifeless> wgrant: looks like henninge is landing it ?
<wgrant> lifeless: ec2 got angry and exploded.
<wgrant> NonZeroExitCode: Test process died with exit code 2, but no tests failed.
<wgrant> The other one went through fine.
<lifeless> sending it
<wgrant> Thanks.
<poolie> hi lifeless, wgrant
<wgrant> Morning poolie.
<lifeless> hi poolie
#launchpad-dev 2010-10-05
<lifeless> matsubara-afk: the timeout feature flag was harder than anticipated, its all landed now, though
<wgrant> lifeless: Julian wrote a faster query last night, but it was slightly wrong. Could you try http://paste.ubuntu.com/506074/ and see if it doesn't take 10 minutes?
<lifeless> hmmm tmp tables :)
<lifeless> :( I mean
<wgrant> Yes.
<wgrant> It's a similar optimisation to the one we use in the publisher.
<wgrant> Which cuts 2.5 minutes down to less than two seconds.
<lifeless> 6 seconds to populate the temp table
<lifeless> should be able to express it directly. Nevertheless, if it works, it works.
<wgrant> One would think so.
<lifeless> perhaps the select needs a supersededby IS NULL in there ?
<wgrant> The subselect?
<lifeless> is supersededby indexed ?
<wgrant> No.
<lifeless> yes, the subselect
<wgrant> It's valid for something to be superseded by something that's also itself superseded.
<wgrant> It appears to be indexed.
<lifeless> yes, but we don't care about those do we?
<lifeless> only the tip of the chain
<wgrant> We do care about those. We are looking for things which should not be superseded but are.
<wgrant> It doesn't matter if the thing that superseded them is superseded or not -- it should not have superseded them.
<lifeless> ok
<wgrant> The query is still going?
<lifeless> yes
<wgrant> :(
<wgrant> I guess I did just increase the data size by several times.
<lifeless> 3662 rows
<lifeless> 465000.849 ms
<wgrant> Comfortingly an identical number to yesterday's.
<wgrant> Thanks.
<lifeless> de nada
<wgrant> Hmmm
<wgrant> http://paste.ubuntu.com/506096/ might be less crap.
 * StevenK grumbles about http://paste.ubuntu.com/506100/
<wgrant> StevenK: bin/kill-test-services
<wgrant> Or was that on ec2?
<wgrant> Hm, looks ec2.
<StevenK> wgrant: On ec2, yes
<wgrant> Sad.
<wgrant> I had a similarly opaque error on one of my runs this morning.
<wgrant> But it was completely different.
<lifeless> well the librarian is definitely gone :P
<lifeless> wgrant: 21 seconds to make the temp table
<lifeless> 3662 rows
<lifeless> 121149.515 ms
<wgrant> Aha!
<wgrant> Much better. Thanks.
<wallyworld> lifeless: what's SOP for turning on sql trace for local debugging? i can add "from storm.tracer import debug; debug(True)" to bin/run (for example). is there cmd line arg i can use?
<lifeless> LP_SQL_DEBUG=1
<lifeless> LP_SQL_DEBUG_EXTRA=1 to get backtraces
<wallyworld> lifeless:  excellent. thanks
<lifeless> those are environment variables
<lifeless> wgrant: librarian fail
<wgrant> lifeless: Hm?
<lifeless> see email
<wgrant> lifeless: So, that's two attempts that have failed in two different spurious ways.
<wgrant> Awesome.
 * StevenK peers at http://ppa.launchpad.net/launchpad/ppa/ubuntu/dists/lucid/main/binary-amd64/Packages
<StevenK> No launchpad-dependencies at all? WTF?
<wgrant> Heh, is this what I think it is?
 * wgrant checks.
<lifeless> wgrant: its almost certainly my layers work
<lifeless> wgrant: we'll need to figure out whats up
<StevenK> lifeless: Does that mean http://paste.ubuntu.com/506100/ is probably you too?
<lifeless> wow, Product:+bugtarget-portlet-bugfilters-stats is unhappy
<lifeless> stub: ^
<wgrant> StevenK: I suspect you have found one of the few PPA cases of bug #653382....
<_mup_> Bug #653382: BinaryPackagePublishingHistory._getOtherPublications fails to restrict the distroseries context <Soyuz:In Progress by wgrant> <https://launchpad.net/bugs/653382>
<StevenK> wgrant: You mean, the publisher has?
<wgrant> StevenK: Indeed.
<lifeless> StevenK: that pastebin may be related yes.
<lifeless> StevenK: however that looks like a racey test to me
 * StevenK grumbles
<stub> eh?
<lifeless> stub: I suspect a fat index or something - 970 timeouts on the bugstats portlet
<stub> Got an OOPS handy?
<wgrant> So, I'd like to get http://paste.ubuntu.com/506212/ run on prod to see how widespread the carnage is.
<lifeless> lpmain_staging=> SELECT COUNT(*) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.id IN ( SELECT BugTask.id ... taking 15 seconds
<lifeless> stub: theres tonnes in the oops report, but e.g. https://lp-oops.canonical.com/oops.py/?oopsid=1738C1035
<StevenK> lifeless: Count(*) is expensive as hell anyway?
<lifeless> StevenK: yes but
<lifeless> StevenK: this is normally a subsecond query.
<lifeless> StevenK: count(*) cost is a nuanced topic, its not guaranteed cheap is the main thing to remember.
<StevenK> wgrant: Is that the same SQL lifeless ran on staging?
<wgrant> StevenK: Except that it tells me how many are in each archive, yes.
<StevenK> Does it give the archive name, or just the id?
<wgrant> Just the id.
<StevenK> wgrant: I wonder if it's worth twiddling the SQL for the archive name?
<wgrant> StevenK: I'm not allowed to possess that.
<StevenK> Oh, right
<wgrant> The only archive that I really care about the identity of is archive 1, and I know which that is.
 * StevenK prods lifeless towards the SQL wgrant is talking about for a +1
<lifeless> +1
<lifeless> you'll need to put it on LPS etc etc
<wgrant> LPS for a SELECT!?
<lifeless> oh, ebrain
<spm> 506212?
<lifeless> StevenK: you don't need cross-check approval for readonly queryies
<wgrant> That.
<lifeless> unless the losa in question looks at you funny.
<StevenK> Ah
<spm> vs the losa in question looking funny. which is a different problem.
<StevenK> spm: Pls fix and make wgrant happy?
<spm> StevenK: see that's the problem. the 2nd part of that sentence. I'm not sure I wish to comply.
<spm> esp when the alt is far more entertaining for me!
<StevenK> s/happy/less angsty/ ?
<spm> it amounts to the same thing from my perspective.
<wgrant> StevenK: Hey, this is my most monumental screwup ever :P
<spm> ^^ young and experienced. Bigger and more spectacular screwups will come....
<spm> *in*experienced.
<spm> gah. troll typo fail :-)
<StevenK> Bwahaha
<wgrant> Heh.
<spm> wgrant: http://paste.ubuntu.com/506216/
<wgrant> Oh thank god.
<spm> yes. you should thank me.
<wgrant> Hah.
<StevenK> 3781 in id 1?
<wgrant> Yep.
<wgrant> That's a really good thing.
 * StevenK wonders how to get the id of the Launchpad PPA
<wgrant> Because every broken publication in the primary archive is trivially revertable with a single query.
<wgrant> Since there are so few archives, I might just join against archive, and get owner.name, archive.name and archive.private.
<StevenK> wgrant: You were afraid it was going to be every archive?
<wgrant> StevenK: More than 37.
<StevenK> Why 37? That was the number on staging?
<stub> lifeless: That query is performing very poorly under PG 8.4
<wgrant> No, I just expected there'd be more than we got.
<wgrant> spm: http://paste.ubuntu.com/506220/
<wgrant> Hopefully there'll be no private archives. Or at least only private archives in public teams.
<stub> bug at over 7 seconds under 8.3, the query sucks already.
<lifeless> stub: it takes 780ms on staging
<lifeless> stub: isn't staging 8.4 ?
<stub> Yes. Hmm...
<stub> The production db is all freshly packed - it is only a few days old.
<stub> production 8.4 one that is (which I think is where all the timeouts are coming from - the OOPS is talking to the slave, and we have only one slave atm)
<stub> But 272 seconds is a little extreme...
<lifeless> rotfl
<StevenK> That's only nearly 5 minutes, what's the problem? :-)
<stub> Comments in the SQL like "                -- We create this rather bizarre looking structure" don't inspire confidence ;)
<lifeless> yeah, indeed.
<lifeless> stub: so, staging is 8.4 and fast, prod slave is 8.4 and slow ?
<stub> yup. staging data is from about 6-7 days ago though.
<wgrant> spm: http://paste.ubuntu.com/506220/
<wgrant> Maybe my connection will survive this time.
<lifeless> so one possibility is a something thrown out by new data?
<StevenK> wgrant: Wishful thinker
<StevenK> Hahaha
<lifeless> it seems improbably to me, given the depth of history, it would need a huge seachange to alter the right query structure
<StevenK> spm: Quick! Run the query while he isn't looking.
<lifeless> stub: any clues from analyze?
<stub> yes. The slow bit is an expensive nested loop the planner thought would get a single row but actually got over 5000, so 5000 times slower than expected.
<spm> I've already done it. just trying to confirm the results for him :-D
<stub> lifeless: http://paste.ubuntu.com/506228/
 * stub runs an analyze on hackberry for a laugh
<lifeless> stub: compare the plan on staging
<lifeless>          ->  Nested Loop Anti Join  (cost=19392.56..26967.75 rows=1 width=4) (actual time=766.504..860.778 rows=5576 loops=1)
<lifeless> if thats what you were saying is the issue, staging has it too
<lifeless> stub: this seems to be a key, slow, but
<lifeless> (bit(
<lifeless> Bitmap Heap Scan on bugtask  (cost=295.98..20874.68 rows=15471 width=24) (actual time=9.812..139.968 rows=19011 loops=1)
<lifeless>                                                    Recheck Cond: (status = 15)
<lifeless>                                                    Filter: ((assignee IS NULL) AND (milestone IS NULL))
<lifeless>                                                    ->  Bitmap Index Scan on bugtask__status__idx  (cost=0.00..292.11 rows=23310 width=0) (actual time=5.382..5.382 rows=24160 loops=1)
<lifeless>                                                          Index Cond: (status = 15)
<mwhudson> query plans and irc, a match made in heaven
 * StevenK grumbles at the continued hiding of launchpad-dependencies in the PPA
<stub> so the estimates are in the same ballpark, which means the planner meant to choose this bogus plan
<stub> full analyze hasn't helped anyway (just in case autovacuum stuffed up or the statistics were screwed after the restore)
<stub> Of course, if I was a query planner I'd take one look at that query and run away screaming
<lifeless> stub: I need to pop out; I'll leave this with you
<stub> gee, thanks ;)
<lifeless> if you need a hand I'll be back after dinner ;)
<lifeless> I do wonder why its choosing this plan
<lifeless> still, 800ms is slow anyway, perhaps the best thing is just to:
<lifeless>  - stop filtering private bugs from the count
<stub> The nested loop I pointed out earlier - it wasn't expecting to execute that sub query so many times.
<lifeless>  - stop trying to match up all the rules
<stub> Yes, overly precise  counts have bitten us before.
<lifeless>  - and just do a group by/count
<wgrant> StevenK: You can revive it if you want.
<wgrant> Delete it and copy it back.
<StevenK> wgrant: Can't we just fix it?
<wgrant> StevenK: Hopefully.
<wgrant> Need to get the fix cowboyed, check if any of them have been deathrowed yet, and then hopefully just mass-revert the status.
<wgrant> *Hopefully*.
<StevenK> lifeless: I think EdwinGrubbs' post on the dev list points out the problem -- is that your doing too?
* bac changed the topic of #launchpad-dev to: Performance Tuesday! | Launchpad Development Channel | Week 3 of 10.10 | PQM is open for business | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> back
<StevenK> lifeless: https://code.edge.launchpad.net/~stevenk/launchpad/librarian-teardown-failure/+merge/37561
<StevenK> (Per the discussion on the -dev list)
<lifeless> no, thats wrong.
 * StevenK goes to get a drink, instead of snapping
<lifeless> I'm replying to the list now.
<lifeless> StevenK: I'm sorry that its wrong - I know infrastructure problems are frustrating. When I say wrong, I don't mean 'unstylistic', I mean 'it will cause other insidious test failures.'
<StevenK> lifeless: Sorry, I'm only frustrated since I've seen errors like this since Saturday, but had no time to look
<lifeless> StevenK: can't have been since saturday, the code landed monday night.
<lifeless> revno: 11667 [merge]
<lifeless> committer: Launchpad Patch Queue Manager <launchpad@pqm.canonical.com>
<lifeless> branch nick: launchpad
<lifeless> timestamp: Mon 2010-10-04 07:20:04 +0100
<lifeless> message:
<lifeless>   [r=mwhudson][ui=none][no-qa] Start consolidation and rationalisation of Librarian test helper code.
<lifeless> whatever is causing you grief since saturday is different.
<henninge> Hi wgrant! ;)
<StevenK> General warnings.
<StevenK> /usr/lib/python2.6/atexit.py:24: DeprecationWarning: Attempt to tearDown inactive fixture.
<StevenK> lifeless: With your patch ^
<lifeless> StevenK: thats fine
<lifeless> belt and braces
<lifeless> we'll get rid of that when we get rid of the atexit
<lifeless> which is another iteration
<bac> lifeless: i got the following errors with your patch: http://pastebin.ubuntu.com/506298/
<lifeless> bac: ok, I thought it might be incomplete.
<lifeless> thanks
<lifeless> bac: can you add a print cls.__bases__ into my patch and show me the result ?
<bac> (<class 'canonical.testing.layers.BaseLayer'>,)
<lifeless> oh
<lifeless> bac: can you do that in LaunchpadScriptLayer.tearDown ?
<bac> (<class 'canonical.testing.layers.BaseLayer'>,)
<lifeless> really?
<bac> lifeless: yeah,it was called twice
<lifeless> oh right
<bac> lifeless: the prints don't show up, so i put in a breakpoint
<lifeless> I thought you were answering the script case
<lifeless> bin/test -vvt old-testing.txt -t spec-mail-exploder.txt
<lifeless> with
<lifeless> --- lib/canonical/testing/layers.py     2010-10-04 06:20:04 +0000
<lifeless> +++ lib/canonical/testing/layers.py     2010-10-05 07:36:47 +0000
<lifeless> @@ -1338,6 +1338,7 @@
<lifeless>      @classmethod
<lifeless>      @profiled
<lifeless>      def tearDown(cls):
<lifeless> +        print cls.__bases__
<bac> lifeless: i don't mind helping, but do you not have a dev env setup?
<lifeless> bac: I don't experience the problem.
<bac> really?  wow!
<lifeless> as I said on list
<lifeless> it passed ec2test
<bac> (<class 'canonical.testing.layers.ZopelessLayer'>, <class 'canonical.testing.layers.LaunchpadLayer'>)
<bac> it died on ec2 for me twice
<lifeless> so, its not __bases__, that appears to be constant and fine.
<lifeless> ...
<lifeless> bac: bah, isp failure
<lifeless> EdwinGrubbs: around ?
<bac> lifeless: it is 3am for EdwinGrubbs
<lifeless> ah
<lifeless> for some reason I thought he was eu
<lifeless> ohhh
<lifeless> I wonder
<lifeless> I'm trying to figure out hth this got into the system
<wgrant> henninge: Hi.
<henninge> wgrant: Hi, otp now ;)
<wgrant> bigjools: http://paste.ubuntu.com/506231/ is the list of casualties and their archives (from prod a few hours ago)
<wgrant> So it's mostly the primary archive, which is handy.
<bigjools> wgrant: context?
<wgrant> bigjools: the _getOtherPublications bug.
<bigjools> ok
<bigjools> I think we'll run the fix on staging first for this one
<wgrant> I'd say so.
<bigjools> wgrant: so, the new query looks the Thing.
<wgrant> bigjools: It also returns the same result as my original one.
<bigjools> always a bonus
<bigjools> I guess it's trawling quite a few rows
<wgrant> Just a few.
<bigjools> the question is, do we want to fix PPAs too
<wgrant> We should do something. Even if it's marking them Deleted instead.
<bigjools> hmmm
<bigjools> so my questions are, what happens if we do nothing, and what happens if we run this fix on them?
<wgrant> If we do nothing, we end up with confusing old data.
<wgrant> A few PPAs end up with some missing packages.
<wgrant> But most of those are probably obsolete anyway.
<wgrant> If they haven't been deathrowed already, we can just set them back to Published and everything will work.
<wgrant> If they have been deathrowed, we should probably just mark them Deleted, because it's messy and it's probably that nobody will notice.
<wgrant> However, we should get the fix cowboyed or CP'd and make the primary archive consistent first, I suspect, given what time it is.
<wgrant> The whole release thing on Sunday.
<allenap> bigjools, wgrant: Is what you're talking about related to the broken ppa:launchpad/ppa?
<wgrant> allenap: Yes.
<allenap> wgrant: Okay, cool. Do you know of any PPA mirrors I can use that are out of date, and thus not broken? I'm going to do a lot of twiddling thumbs today otherwise :)
<wgrant> allenap: You could always grab launchpad-dependencies manually.'
 * wgrant finds the link.
<allenap> wgrant: Ah, it's in the PPA, just not in the manifest or whatever it's called?
<bigjools> ppa:launchpad/ppa got broken too?
<jtv> henninge, you'll want to know about this: a conflict has crept into your convert-imports branch.
<jtv> henninge: lib/lp/translations/stories/translations/30-rosetta-pofile-translation-gettext-error.txt
<wgrant> bigjools: Lucid's launchpad-dependencies were vaporised a couple of days back, yes.
<bigjools> :/
<mrevell> Hello
<wgrant> bigjools: It's the only report of breakage, interesting enough.
<wgrant> allenap: https://edge.launchpad.net/~launchpad/+archive/ppa/+sourcepub/1263354/+listing-archive-extra has the binaries you need.
<bigjools> I've just started the query on staging to see how many rows there are when restricting to archive=1
<allenap> wgrant: Thanks.
<wgrant> bigjools: http://paste.ubuntu.com/506220/ was the last query I had run on production.
<wgrant> Gives the count for each archive.
<bigjools> yeah I have tjat
<bigjools> wgrant: would you mind doing me a favor and making a query to fix the PPA publications while I do this?  I think we can filter on dateremoved
<wgrant> bigjools: That was my thinking too.
<bigjools> great minds ...
<wgrant> Anything with dateremoved IS NOT NULL should be safe to just flip back.
<wgrant> Which should cover all of the primary archive cases too, hopefully.
<bigjools> ah very good point
<bigjools> nm then
<bigjools> I'll pop that in
<henninge> jtv: thanks
<wgrant> Just set the status, and unset datesuperseded, datemadepending and scheduleddeletiondate
<wgrant> I think that should do it.
<wgrant> Although the last two aren't relevant for the primary archive.
<wgrant> If anything has those set we have other bugs.
<henninge> wgrant: Did you see that test failure?
<bigjools> and supersededby
<wgrant> bigjools: erk, good point.
<wgrant> henninge: Yes. lifeless ran it again, and it failed spuriously another way.
<bigjools> :)
<wgrant> Along with a few other branches.
<wgrant> So there's something wrong with devel.
<henninge> who ones merge proposals code?
<henninge> s/ones/owns/
<henninge> s/who/who in the code team/ ;)
<lifeless> wgrant: should be fixed soon, branch is in pqm atm
<wgrant> lifeless: Ah, great.
<jtv> danilos, henninge: we've long wanted to use the permissions system for translations, but I find myself really not wanting to do that now.  What would permissions be _on_?   It seems weird to invent a DummyPOFile just to check permissions on it, for instance.
<danilos> jtv, what do you mean with 'now'? do not want to use "full" zope machinery with this particular branch?
<jtv> danilos: I mean "now that we're sharing translations so widely."
<wgrant> bigjools: Should we run a query on prod to see how many have already been removed?
<bigjools> doing it now
<wgrant> With the per-archive counts?
<bigjools> quite a lot it seems, I get a count of 842 in the query now
<bigjools> after adding "bpph.dateremoved IS NOT NULL"
<wgrant> On staging?
<bigjools> yes
<jtv> danilos: if it were just "are you one of the owners of this product," for instance, I'd have no problems.  But if we go pofileâpotemplateâproductseriesâproductâprojectâtranslationgroupâtranslator, that's a lot for a permissions check.
<wgrant> Are any of those in the primary archive?
<bigjools> one sec
<danilos> jtv, well, sharing aspects of it will be relatively hard to map in the zope permissions system, but ideally, we'd at least make it a simple check: "can-do-something-here" and "can-do-something-on-the-other-side"
<wgrant> The publications I checked hadn't had removal scheduled, so I doubt it.
<danilos> jtv, ideally, we'd just check permissions on the pillar level (product/distribution), because that's where we get them from
<danilos> except for pofile, of course
<jtv> danilos: sure, that part's fine for me and may even hide a bit of the "if productseries" ugliness
<jtv> danilos: that's the other thingâI really, _really_ don't want to give edit rights to pofile owners.
<danilos> jtv, I am sure you can come up with a reasonable design, and you might be right that permissions system is not the right solution
<danilos> jtv, right, we want maintainers to have edit rights on them, but other than that, just translators for appropriate language
<jtv> danilos: I appreciate the trust.  :)  I think POFile.owner is a bit of an historical mistake; it's really "creator" except we always set it to rosetta-admins if the creator doesn't have edit rights already.
<danilos> jtv, fwiw, do note that you will see tests breakage if pofile.owner can't edit a pofile (I am sure that in all our laziness we used that often :)
<allenap> wgrant: Thanks for that link; my bacon has been saved.
<wgrant> allenap: Yeah, uh, sorry about that.
<danilos> jtv, oh, pofile.owner is bordering on the useless, I'd say
<jtv> danilos: there's a good pointâ¦ another reason to clean up, probably.
<allenap> wgrant: Ah, are you the culprit? :)
<jtv> danilos: if we stop giving it any privileges, we can also set it to the actual creator rather than "set to creator _if_ it doesn't conflict with current permissions."
<wgrant> allenap: Yes :/
<danilos> jtv, then it'd be nice to rename it as well
<jtv> danilos: it may also avoid some of those cases where someone who's not in the translation team can continue to submit bad translations.
<allenap> wgrant: Hehe, oops.
<jtv> danilos: yes, I'd like that too.
<wgrant> allenap: Rather.
<danilos> jtv, true, true
<danilos> jtv, so, basically, you want to drop "owner" field and create a different "created_by" field
<danilos> jtv, I agree that'd be nice, but this is a cleanup job for later :) for now, you can simply ignore the owner for permissions
<jtv> danilos: that's what I had in mind, yes.
<danilos> jtv, cool, sounds good
<jtv> The main thing right now is not to give special rights to pofile.owner, in which case the only things that matter are the person, the language, and the pillar.
<jtv> I was thinking of making this a method in ITranslationsPerson, since we have so many functions about this spread out.
<lifeless> ok, devel should be fixed
<wgrant> lifeless: Could you fire of that branch again, or shall I convince someone else?
<lifeless> someone else per favour
<jml> hello
<bigjools> hello
<wgrant> henninge: Could you please ec2 that branch again?
<wgrant> bigjools: Any progress?
<bigjools> query still running
<wgrant> Er.
<henninge> wgrant: just test it or try to land it?
<wgrant> Doesn't it only take three minutes?
<wgrant> henninge: The latter.
<henninge> wgrant: ok
<bigjools> wgrant: probably on production
<wgrant> bigjools: This was on staging.
<bigjools> hmm
<wgrant> Although it didn't have the archive/person join, that really shouldn't be that bad.
<stub> So 250k of our 330k branches are owned by 'ubuntu-branches', which is rather lopsided and will certainly be triggering timeouts in some areas.
 * bigjools kills it
<wgrant> Who is buildbot angry with?
<bigjools> wgrant: http://pastebin.ubuntu.com/506369/
<lifeless> stub: any luck with the bug portlet stuff?
<bigjools> wgrant: which means 1100 or so packages are lost in Ubuntu
<wgrant> bigjools: Uh, is that dateremoved IS NOT NULL?
<bigjools> yes
<bigjools> wgrant: sorry
<bigjools> :/
<stub> lifeless: I'd say 1) work out what the query is trying to do 2) rewrite it.
<lifeless> 1) might be hard :P
<bigjools> wgrant: I meant to say, that's bpph.dateremoved IS NULL
<wgrant> bigjools: Is that prod or staging?
<bigjools> wgrant: staging
<wgrant> bigjools: The 3700 was from prod.
<bigjools> ok
<bigjools> I need to land your fix on prod
<wgrant> What does http://paste.ubuntu.com/506220/ say on staging? I suspect that will show that we haven't lost anything from the primary archive.
<wgrant> Alternatively, what does your dateremoved IS NULL query say on prod?
<wgrant> But yes, germanium and cocoplum want that patch.
<bigjools> argh
<stub> lifeless: Indeed. btw. the query I'm testing is slow on staging too, although better at a little under 30s. I'd suggest it is working at all under PG 8.3 purely by accident (if you consider 7 seconds working)
<bigjools> I cna't remember if that query was with or without the NOT
<lifeless> stub: thats odd; the one I copied from the OOPS I pasted takes 800ms on staging.
<wgrant> bigjools: I'm fairly sure it must have been dateremoved IS NULL.
<lifeless> stub: perhaps I'm connected to 8.3 on staging?
<stub> There is no 8.3 on staging
<stub> http://paste.ubuntu.com/506383/ is what I was looking at
<lifeless> http://pastebin.com/w3h72qvC
<bigjools> wgrant: right, nothing shows for NOT NULL in the main archive
<wgrant> bigjools: So I'm not completely insane.
<wgrant> This is good.
<bigjools> wgrant: just slightly
<stub> lifeless: I can confirm your query is fast on staging, fast on production 8.3, and slow on production 8.4.
<lifeless> stub: cool
<lifeless> stub: bizarre, but cool
<wgrant> bigjools: How about the rest of the archives?
<stub> lifeless: maybe data, or maybe less ram causing different query plans
<bigjools> wgrant: http://pastebin.ubuntu.com/506387/
<wgrant> bigjools: That's staging dateremoved IS NOT NULL?
<bigjools> wgrant: yes
<wgrant> OK, so we can take care of most of them trivially.
<bigjools> wgrant: can you help me QA your fix on dogfood
<wgrant> Once we've fixed the dateremoved IS NULL, I'll get a list of the remaining pubs and work out what to do with them.
<wgrant> bigjools: Sure.
<wgrant> Let's see how...
<bigjools> wgrant: I'm updating its code, we need a test scenario
<wgrant> bigjools: Easy enough. Just upload a new launchpad-dependencies to maverick in ppa:launchpad, run the publisher, and confirm that the old version is superseded in maverick but still alive in lucid.
<wgrant> bigjools: Ah, I see that dogfood's PPA doesn't have a maverick launchpad-dependencies yet.
<bigjools> yeah
<bigjools> easily fixed
<bigjools> grar, I need a no-sign mode for publishing PPAs
<wgrant> Just unset the signing key, I suppose.
<bigjools> still a pita
<wgrant> How do you normally do it?
<bigjools> either than or copy some .gpg files around
<bigjools> that*
<bigjools> I need a single ppa mode too
<wgrant> 'UPDATE archive SET publish=false' FTW.
<bigjools> heh
<wgrant> bigjools: Er, why is the maverick publication Published but without a datepublished?
<wgrant> Did you manually set that?
<wgrant> Or is something even more broken?
<bigjools> I've been hacking it
<bigjools> looks like it worked ok
<StevenK> bigjools, wgrant: Ah, is this to fix the no launchpad-dependencies in lucid problem?
<wgrant> StevenK: Yes.
<bigjools> -ish
<wgrant> Right.
<lifeless> mrevell: is there any reason we don't show the revno and 'get the code' link on launchpad.net ?
<lifeless> mrevell: (compare the footer on  launchpad.net and edge.launchpad.net)
<wgrant> production-stable isn't public.
<lifeless> wgrant: prod stable isn't referenced.
<wgrant> lifeless: It should be by prod...
<lifeless> wgrant: and its going anyhow.
<lifeless> wgrant: base-layout-macros.pt, or look at the two pages I referenced.
<wgrant> launchpad.net's revno is on production-stable, which mortals cannot see.
<lifeless> wgrant: imagine an epic shrug from me.
<wgrant> Huh?
<bigjools> lifeless: wow you're on the ball with my CP
<bigjools> thanks
<lifeless> bigjools: de nada
<lifeless> wgrant: I'm making changes to match up with edge going, I don't care about the prod-stable stuff; I care about the UI
<wgrant> lifeless: How can it work if production-stable isn't public? Unless you link to the completely wrong code...
<lifeless> wgrant: we're deleting production-stable
<bigjools> grrr lp-land not playing ball on that branch
<wgrant> lifeless: Well, sure, once that's done.
<mrevell> lifeless: Before it was the revno, it was also the version number. I think it was on edge, and staging, to help people see that edge was running different code to production and also to help with reporting of bugs. I don't see any reason to keep "Get the code" off production. The revno maybe feels to me like something you wouldn't see on a production web app but I don't have a strong opinion on it.
<lifeless> wgrant: I can tolerate some imperfection
<mrevell> lifeless: tl;dr Old reasons that probably don't apply now.
<lifeless> mrevell: thanks
<deryck> Morning, all.
<wgrant> bigjools: So the CP is happening?
<bigjools> wgrant: OTP
<wgrant> k
<lifeless> wow, isn't ohloh the hot potato
<wgrant> Because it keeps failing to import LP?
<wgrant> Oh, I see.
<wgrant> Indeed.
<wgrant> They can't seem to format their emails properly.
<lifeless> mrevell: so, my blog post
<lifeless> mrevell: could you hit 'go' on it before you EOD, or reply to me ?
<lifeless> gnight all
<matsubara> lifeless, thanks for landing that. :-)
<wgrant> bigjools: I suppose we can do little more tonight, since the CP will not happen for ages?
<allenap> gmb: There was a question directed at you in bug 605783, but I think it's something that deryck should take a look at.
<_mup_> Bug #605783: SourceForge bugwatch updates are broken <bugwatch> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/605783>
<leonardr> dumb zope browser/mechanize question: how can i iterate over all the buttons in a form?
<leonardr> benji, maybe you know
<leonardr> <leonardr> dumb zope browser/mechanize question: how can i iterate over all the buttons in a form?
<leonardr> _get_all_controls is really particular about only letting me get _one_ control
<leonardr> aha! .mech_form.controls
<benji> leonardr: that'll work (but is undocumented/unsupported); you can also use beautifulsoup
<leonardr> what is this "beautifulsoup"?
<deryck> allenap, I'll comment there.  I saw it but was trying to figure out if we could get this scheduled.
<leonardr> benji: more seriously, is there a shortcut for making a soup of the current browser page?
<leonardr> i see some examples in pagetests but they do it from scratch
<allenap> deryck: Yeah, that's what I meant really.
<deryck> allenap, understood.  I feel bad because we really should get this fixed.  But it can't happen anytime soon.
<benji> leonardr: heh; BeautifulSoup(browser.contents)
<leonardr> benji: ah! i was thinking of helper *functions* like find_tags_by_class, which make a BS object and do something else
<deryck> So nice to not see [object Object] when I comment on bugs now.
<leonardr> benji: actually, it looks like the right thing to do is to call find_tag_by_id(browser.contents, 'maincontent')
<leonardr> that gives you a bs tag
<leonardr> i don't think we create a bs object unless we really need the whole page
<bigjools> wgrant: nope, we need to wait for it to land
<bigjools> wgrant: then we can roll it out and do the SQL to fix everything
<wgrant> Sounds good.
<bigjools> where "everything" is followed by a sarcasterisk
<jml> bigjools: could you tell me why this query returns 0 rows?
<jml> http://paste.ubuntu.com/506504/
<jml> it used to return lots
<bigjools> jml: what are you trying to do?
<jml> https://lpstats.canonical.com/graphs/UddSourcePackagesWithoutBranches/
<bigjools> jml: DistroSeries.releasestatus = 2 ?
<jml>         # DistroSeries.releasestatus == 2 => under active development
<bigjools> jml: it's not any more
<bigjools> it's frozen
<jml> bigjools: that'll be it then.
<jml> bigjools: thanks.
<bigjools> jml: np!
<bigjools> glad to be helping you for a change :)
<jml> uhh :(
<jml> SeriesStatus.DEVELOPMENT == 2 :(
<StevenK> jml: That isn't what bigjools meant.
<StevenK> jml: Not that 2 isn't Development; that Maverick itself is Frozen. It won't thaw until release.
<bigjools> jml: maverick is in "Pre-release Freeze"
<bigjools> not development
<jml> bigjools: but I'm not searching for maverick in that query.
<bigjools> jml: there are no other series in development
<bigjools> it would have been matching only maverick
<jml> ahh ok. now I get it.
<jml> I guess I need to make it say "releasestatus IN (2, 3)"
<bigjools> when natty is opened it'll be experimental for a bit too
<leonardr> jml, can i get your buy-in on https://dev.launchpad.net/LEP/DesktopWideLaunchpadIntegration? i already have buy-in from kees, lifeless, and other luminaries
<jml> leonardr: almost certainly
<jml> leonardr: I feel obliged to read it before I approve it though.
<leonardr> please do
<jml> leonardr: but I've been following the ML discussion so I reckon it'll be smooth
<allenap> deryck: What do we do with feature suggestions normally? https://answers.edge.launchpad.net/malone/+question/127685
<allenap> Convert to a wishlist bug?
<jml> allenap: orbital lasers.
<allenap> jml: Lol. Getting a GPS fix now.
<deryck> allenap, yeah, but I think this "have pretty graphs" bug already exists somewhere.
<allenap> deryck: I can't find it, but I can reply saying that, yes, we want it too, and perhaps paste kfogel's contributor bait paragraph too?
<allenap> If I can find *that*.
<jml> can I persuade someone to write a script that shows all of the in-progress work on Launchpad at a branch/revision/bug level, broken down by stage (e.g. needs-reviews, needs-qa, being-tested, awaiting-deployment etc)
<jml> ./utilities/branch-flow is something of a start, but isn't really that great.
<jml> leonardr: contrariwise, could you please look at https://dev.launchpad.net/LEP/ActivatingDevelopmentFromDesktop
<leonardr> jml: there's a small chance that my lep obsoletes that one, but if not, there's not much overlap.
<leonardr> ActivatingDevelopmentFromDesktop is currently the one remaining case where we authorize an app-specific token
<leonardr> (obsoletes that one == obsoletes the part about time-limited authorization keys)
<jml> leonardr: ahh ok.
<jml> leonardr: I've just approved DesktopWideLaunchpadIntegration. Thanks for the clear write-up.
<leonardr> i have not done exhaustive research, but it's possible that a time-limited authorization key will not really provide much additional security
<leonardr> sure
<mars> benji, did you see the mailing list thread about the LayerIsolationError?  Is that what you are experiencing?
<danilos> jcsackett, hi (btw, I don't see you on #launchpad-reviews)
<jcsackett> danilos: hi, no, i'm CHR today and got lost in that and now a testfix; completely forgot to join #launchpad-reviews.
<jcsackett> i'll join that channel if you would prefer.
<danilos> jcsackett, just one question about the branch you asked me to review? what is the goal behind it? just switching from official_rosetta to the usage enum?
<danilos> jcsackett, nah, this is fine, just thought I'd mention it if you want to add it to your auto-join channel list :)
<jcsackett> no, that goal has been met, this is dealing with consistency in configuration for usage_enums.
<jcsackett> other services have a config option even when set to ServiceUsage.LAUNCHPAD; translations did not, as it has no action menu to grab the global action.
<jcsackett> so, for consistency's sake, the branch adds one there (so you could, say, disable the enum for a product if it's enabled).
<danilos> jcsackett, oh, so you mean it's about adding it to the translations.lp.net pages?
<jcsackett> danilos: not entirely sure what you mean.
<jcsackett> the branches sole goal was to add a configuration link for the usage enum to translations when translations is already enabled (via enum) on a product.
<danilos> jcsackett, well, it was possible to set it on "registry" pages
<jcsackett> ah, i see.
<jcsackett> right, the translations front page; so translations.launchpad.net/<product>
<danilos> jcsackett, right, so this is exactly the thing that I wanted to move to the global "configure translations" page :)
<danilos> jcsackett, thanks for explaining the reasoning, I'll comment on the bug
<danilos> jcsackett, we can get this landed, but it just makes stuff more confusing imo
<danilos> jcsackett, also, if all the other apps are using action menus, we should probably do that for translations too
<jcsackett> danilos: that's why i requested you review it; it's possible the bug it's filed against isn't really valid.
<jcsackett> all other apps are dealing with collections and have the same basic interface (list of related items). translations is an exception to that, so it's possible that the configuration option belongs in +settings or something, as those other bugs you pointed out describe.
<danilos> jcsackett, I am not sure I agree with that statement (translations pages deal with collections as well, and we quite specifically show a list of languages on the product frontpage); there are ways to get to a list of templates, or list of imports, so I think we are not that special case, but might just be out of date on the design side
<danilos> jcsackett, and if we are, I want to fix that :)
<danilos> jcsackett, also, why do we use the camel case capitalization for the title on "Configure Translations" pages? I thought the agreed norm was to not use it
<danilos> salgado-lunch, hey, UI reviewer, what's the recommended capitalization for web page titles? :)
<jcsackett> danilos: what part of this branch makes it more confusing? just the addition of yet another configuration option?
<danilos> jcsackett, yeah (i.e. instead of fixing it all to be on the same page, we are spending effort to introduce the new page)
<jcsackett> danilos: fair. it's a bit of a stopgap.
<jcsackett> if you could write up a good reason we don't need this in favor of a better settings page on the MP, i'm fine to abandon the branch and mark the related bug as wontfix.
<jcsackett> danilos: we just need an explanation for leaving it as is when we go to mark the bridging-the-gap feature work as done.
<danilos> jcsackett, right, and as such, I don't have a big problem with the branch as-is
<jcsackett> danilos: less of course fixing the issues with title &c?
<danilos> jcsackett, well, it's ultimately your call: I will write up an explanation, but other than a minor issues, I am fine with it landing as well
<danilos> jcsackett, so, I'll approve the branch as well
<jcsackett> danilos: cool. sounds good. :-)
<jcsackett> thanks, danilos.
<jcsackett> danilos: FWIW I would love to see (and help with) a more unified translations settings page. :-)
<jcsackett> (though the help with part may have to be mostly on my own time.)
<danilos> jcsackett, heh, sure, we'll work on it pretty soon now as well: we want to streamline translations setup as our next step
<jcsackett> danilos: awesome. :-)
<sinzui> flacoste, meeting on mumble?
<flacoste> sinzui: yes
<salgado> danilos, according to https://dev.launchpad.net/UserInterfaceWording, it should be sentence case
<danilos> salgado, right, thanks
<danilos> jcsackett, ^ (link by salgado :)
<jcsackett> danilos: dig; i'll make that change.
<lifeless> moin
<jelmer> lifeless!
<lifeless> hi jelmer
<deryck> morning, lifeless.
<lifeless> hey deryck
 * tyarusso sighs
<lifeless> ?
<tyarusso> Has *anyone* succeeded in setting up their own production instance of Launchpad + CIP?  I'm getting failures finding things from pip with CIP currently.
<tyarusso> * anyone other than Canonical, of course
<tyarusso> I wish Canonical offered hosted Launchpad instances for businesses, or at least a consulting service to set it up for you.
<jpds> tyarusso: I would talk to bac.
<tyarusso> jpds: do you know when s/he is normally around?
<lifeless> jpds: bac doesn't do that anymore, mrevell does
<lifeless> tyarusso: hi; we do run private content for businesses that want it, on the production LP.
<lifeless> tyarusso: matthew.revell at canonical.com would be good to talk to about your needs.
<sinzui> flacoste, PS 4198 pillar/person name collisions, 1509 of these are actually teams
<flacoste> sinzui: you mean we have 4.2k project that have the same name than a team?
<sinzui> 1509 have the same name as a team
<tyarusso> lifeless: Well, that situation we would want is to have it under our domain name and branding, which I was previously told isn't possible currently with your stuff.  Still true?
<sinzui> 2.5k are users with the same name
<flacoste> ack
<lifeless> tyarusso: thats true, launchpad itself isn't capable of being branded multiple ways in a single deployment.
<tyarusso> lifeless: Do you think revell would be useful for helping us set up our own, or is that outside of his scope?
<lifeless> I'd just ask on the launchpad-dev mailing list
<tyarusso> k
<tyarusso> It's being rather hard to convince myself to do this instead of 'apt-get install gforge', which is a shame, since LP is rather nice once it's up.
<flacoste> deryck: did we deselect the automatic expiry before sending the notifications?
<flacoste> tyarusso, lifeless: it's outside of his scope
<flacoste> tyarusso: the only instance i know of a rebranded LP is quickbuild
<deryck> flacoste, actually, no.  Send then deselect.  That was a bad choice in hindsight.
<flacoste> deryck: it is!
<flacoste> deryck: i just went to my project to enable it
<flacoste> and saw it was already enabled
<deryck> yeah, it's a bad choice.  not sure if we should do anything to correct it now.
<flacoste> tyarusso: https://quickbuild.pearsoncomputing.net/
<flacoste> deryck: i'd say send a second email once you deselect it for real
<flacoste> deryck: because anybody who acted on your email right away will be screwed
<flacoste> or at least very confused
<deryck> flacoste, sure.  I can do that.
<deryck> hate to spam, but it's probably the reasonable thing to do.
<tyarusso> flacoste: hmm, interesting.  Who runs that?
<flacoste> tyarusso: be aware thought that it's not a full deployment, there are stuff that is disabled
<flacoste> tyarusso: i can't remember
<flacoste> but he's the maintainer of the trinity project
<flacoste> (continue to support packaging of KDE 3 series)
<tyarusso> well, out of time for today.  Maybe better luck tomorrow...
<flacoste> gary_poster: thanks for your branch on bug 650343, but the header name actually changed
<_mup_> Bug #650343: Add X-Launchpad-Original-To to recipient lists <Launchpad Foundations:Triaged by gary> <https://launchpad.net/bugs/650343>
<flacoste> gary_poster: it's now X-Launchpad-Original-To, (not X-Original-To)
<gary_poster> flacoste: ah, misunderstood.  Not merged for other reasons.  Will make a change
<bdmurray> sinzui: I'd like to edit the permissions for OfficialBugTagsManageView so that bug supervisors can also edit them.  Would I add a class for this to security.py?
<sinzui> bdmurray, isn't that also security on the context object. I think the rules are in there, and the rules on the view just work
 * sinzui looks
<lifeless>  we can't use views for security though
<sinzui> have I gone mad? I thought there was a bugtag interface and model
<bdmurray> lp.bugs.interfaces.bugtarget.IOfficialBugTagTargetRestricted ?
<sinzui> bdmurray, thanks so we do have an interface, now I will check the zcml
<sinzui> bdmurray, there is nothing to edit
<bdmurray> sinzui: it is just launchpad.Edit now right?  I want to extend this to bug supervisor.
<sinzui> bdmurray, IOfficialBugTagTargetRestricted is only used to define security on the view and since there is no rule for IOfficialBugTagTargetRestricted, it falls back to target types lp.Edit (distro|product|dsp|dseries|pseries)
<bdmurray> sinzui: yes, so I'm asking how to make this rule
<sinzui> bdmurray define a security checker for IOfficialBugTagTargetRestricted that checks the admin, target.owner, target.bug_supervisor
<bdmurray> sinzui: okay, that's what I'd thought and it what would the value of usedfor be?
<sinzui> bdmurray, in the case of a project or any kind of series, I think checking target.driver is also correct. This addition ensure Release managers can tag their bugs
<sinzui> bdmurray, usedfor = IOfficialBugTagTargetRestricted
<bdmurray> sinzui: okay, thanks for the help
<lifeless> flacoste: I intend to just Edit the wiki pages surrounding production changes to match RFWTAD - do you want any editorial oversight ?
<lifeless> flacoste: or will you just look and tweak post-hox
<flacoste> lifeless: please be my guest, i'll tweak post-hoc if needed
<negativezero> hey guys, is is appropriate to ask about the launchpad-developer-dependencies package in here? #launchpad is awfully quiet
<sinzui> negativezero, this is the correct place
<negativezero> i'm trying to install launchpad locally, but apt can't seem to find the aforementioned package.  i've used both rocketfuel-setup and added the ppa's to my apt sources
<sinzui> negativezero, :( rocketfuel-setup does add the archives to your sources list
<sinzui> what version of Ubuntu do you have?
<negativezero> 10.04
<rockstar> sinzui, how do I handle database records that reference Person?
<sinzui> negativezero, did you also update your apt-cache to see what is in those soruces
<sinzui> rockstar, ?
<rockstar> sinzui, I should have put quotes around "handle."
<rockstar> sinzui, I get a nice "NotImplementedError: branchmergequeue.owner reference to person.id is in a UNIQUE index but has not been handled"
<sinzui> rockstar, what are database records?
<rockstar> sinzui, database patches.
 * rockstar needs to pull his head out...
<mwhudson> rockstar: is this in the person merge tests?
<rockstar> mwhudson, yes.
<mwhudson> you need to add code in the person merge code to decide what to do
<rockstar> mwhudson, well, no, it's in all sorts of tests, but I assume it has to do with the person merge stuff.  That's why I asked sinzui in the vaguest way possible.
<mwhudson> oh
<rockstar> mwhudson, but I assume what you say is correct.
<rockstar> mwhudson, where is the person merge code?
<negativezero> sinzui: i ran an apt-get update. is there a command to update apt-cache specifically?
<rockstar> negativezero, that's what apt-get update does.
<sinzui> rockstar, I think I understand. in cases where this could happen, we often check that it can happen, then ask the user to either destroy the object to prevent a unique key violation, or we add rules that make the new items unique
<rockstar> negativezero, I suspect that the PPAs are out of date with Lucid now, since we've all switched to maverick.
<sinzui> negativezero, you did the right thing
<rockstar> sinzui, can't we just change the order?
<rockstar> s/order/owner...
<negativezero> there's some launchpad-integration packages in my cache
<rockstar> Dammit I should go back to bed and hope tomorrow gets better.
<sinzui> rockstar, I saw an oops like this in translationqueueentries recently. I decided to to change the owner. The other option is to tell the user he must wait until the queue is cleared
<sinzui> rockstar, in the translationqueue issue, we discussed there is a chance that permissions will be lost. I do not know what to do in the case of  branchmergequeue.owner, but I think you want to add a rule to change it to the new owner
<rockstar> sinzui, okay, so where's the code that "handles" this kind of problem?
<rockstar> (Basically, how do I fix these tests)
<sinzui> rockstar, lp/registry/model/person.py look for the for "def _merge"
<rockstar> sinzui, ah, okay.
<sinzui> rockstar, you may want to look at lp/registry/browser/peoplemerge.py if you decide you need to tell the user to wait for the queue to empty.
<rockstar> sinzui, this queue is like the pqm queue.  If you're lucky, it'll NEVER be empty.  :)
<sinzui> rockstar, can I have more than one item in the queue?
<rockstar> sinzui, yes, but they already have owners.
<rockstar> (The items are branch merge proposals)
<sinzui> excellent just look at the model you only need to add or update a rule to change their owners
<rockstar> sinzui, it does seem a bit odd that I get this on a failure in a database patch branch.
<sinzui> rockstar, I think salgado/stub is pretty clever I suspect they check that for everything a person links to has a rule in person merge
<rockstar> sinzui, okay.  Thanks for your help.  I think that'll be the last of my failing tests.
<lifeless> flacoste: ok, I think I'm done.
<wgrant> Grararrara
<rockstar> abentley, wallyworld, is it skype time yet?
<abentley> rockstar: sure.
<wgrant> I had a whole lot of translation-related failures in one of my branches overnight.
<wgrant> Are those known?
<wallyworld> rockstar: yep
<rockstar> abentley, wallyworld, my mic no longer works, because it accidentally got unplugged and pulse is a stubborn bugger.
<rockstar> abentley, wallyworld, thanks!
<lifeless> rockstar: pkill npviewer.bin + check in the sound prefs that its not muted.
<rockstar> lifeless, yeah, but when it already thinks there's a connection, it won't reset it.
<flacoste> lifeless: done with what?
<lifeless> flacoste: edits to the production process docs
<flacoste> lifeless: ah, cool, i'll review the notifications email later :-)
<flacoste> abentley: did you sort out your QA issue?
<lifeless> flacoste: we'll need to do another pass once qastaging is actually live.
<maxb> Anyone: what's the status of the erroneous-superseding issue?
<lifeless> in buildbot
<lifeless> 40 minutes to go
<lifeless> when that completes we can CP it
<lifeless> and then we need to do a db update to change data to be right
<lifeless> wgrant has details
<wgrant> I don't know the CP status, obviously.
<wgrant> We will revert most of the publications to Published this evening. But some older PPA ones will need more manual recovery.
<lifeless> wgrant: naturally, I mean the db repair
<wgrant> But the ~launchpad PPA is fixed now, anyway.
<lifeless> bugger
<lifeless> 	twisted.internet.error.ProcessExitedAlready:
<lifeless> where was the fix for that?
<wgrant> I've only seen that from a buildd.
<lifeless> bigjools mentioned it as a known test failure in devel, which was fixed
<lifeless> we need to port the fix to prod-devel.
<gary_poster> deryck: ping?
<deryck> hi gary_poster
<gary_poster> hey deryck.  I'm trying to get my triaging in order.  What's the story of this one: https://bugs.edge.launchpad.net/lazr.restful/+bug/443217
<_mup_> Bug #443217: Changing a bug's affected project or description doesn't ever finish <bug-page> <dhrb> <javascript> <ui> <lazr.restful:New> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/443217>
<gary_poster> it looks like Tom was working on it and had a handle on it, but of course, Tom is off doing other Tom things now
<deryck> gary_poster, it's actually very close to next on our queue.  It's the next card on the backlog.
<gary_poster> ah ok cool
<gary_poster> is it a lazr.restful bug?
<deryck> gary_poster, I think it can be fixed in lp.  But I left the task new until we could look more closely at it.
<gary_poster> ok deryck.  um...since I'm trying to get rid of "New"s...would "Incomplete" be a reasonable description of the lazr.restful part of the task?  I guess that seems kind of artificial, but it reflects reality in that I shouldn't have anyone look at it yet, AFAICT.  Would that be ok?
<deryck> gary_poster, yeah, fine with me.  We should make a call on it before it expires.
<gary_poster> cool, thanks deryck
<deryck> np!  Thank you.
<wgrant> Were there any known translations failures overnight?
<lifeless> wgrant: could you perhaps look for that bugfix he talks about ?
<lifeless> wgrant: and I will shepard it into prod devel
<wgrant> lifeless: Any idea when it was from?
<lifeless> his sprint with jml I think
<lifeless> or since
<wgrant> Which rev was last mass-CP'd?
<lifeless> we're fully deployed up to (inclusive) 11609
<lifeless> flacoste: no, abentley oculdn't qa completely, blocked on puloads on staging not working
<lifeless> flacoste: lamont was going to look at it
<wgrant> lifeless: I see nothing particularly relevant since then.
<wgrant> There were a couple of things moved around due to the sprint.
<wgrant> But that's about it.
<lifeless> ah
<lifeless>  r11594
<lifeless> which we have
<wgrant> Ah, yes, that one.
<sinzui> bryceh, ping
<bryceh> sinzui, yep
<sinzui> bryceh, do you see an administer link on https://edge.launchpad.net/~bryce ?
<bryceh> sinzui, yes
<bryceh> sinzui, 'Administer' and 'Administer Account'
<wgrant> deryck: Woah, I just got seven emails from you.
<sinzui> bryceh, you can use Administer to change that name...
<lifeless> StevenK: I've just mailed canonical-lp about the prod-devel bb failure, can you perhaps investigate? you know more than I about that area.
<deryck> wgrant, you must own 7 projects on lp. :-)
<wgrant> Ah, missed that detail.
<wgrant> Indeed.
<sinzui> bryceh, but to change your id to bryce, you need to *delete* all you PPAs. You can create new ones, but they cannot have the exact same name.
<sinzui> bryceh, that may be too drastic of a step for you to take
<lifeless> afk- lunch and firewood stacking
<bryceh> sinzui, ok, that may be ok.  Would it also require re-joining groups / mailing lists, or will those memberships get transitioned?
<sinzui> bryceh, nothing else. changing your launchpad id does not change your db id.
<sinzui> bryceh, the PPA restriction is because you have a public URL attached to the PPA
<sinzui> bryceh, I know of one user who deleted his PPAs, then a day later we got complaints that those PPAs went missing
<bryceh> sinzui, ok
<bryceh> sinzui, since I've been on launchpad the past 5 months I think all my ppas are basically stale by now.
<bryceh> anything "permanent" I've tended to put into a team ppa rather than my own.  So I think I'm safe, but that's good to know.
<sinzui> okay.
<sinzui> I wish we could know how many subscribers we have.
<wgrant> The code is all there for number of package downloads.
<bdmurray> sinzui: I'm having some issues with my branch regarding permissions for official bug tags - it seems to be the security checker is never used
<sinzui> bdmurray, :( So while the view has always had a rule for IOfficialBugTagTargetRestricted, the security checker for IOfficialBugTagTargetRestricted is not used. I assume the IProduct launchpad.Edit is used?
<sinzui> wgrant, the code is there, but the logs are not being parsed?
<wgrant> deryck: You didn't notify owners of projects where the flag was disabled?
<wgrant> sinzui: Correct.
<wgrant> I landed the code in March, but some performance issues took a while to fix.
<deryck> wgrant, I notified bug supervisors if there was one at first, thinking if one was set that was the preferred contact for something like this.
<deryck> wgrant, stupid me, the bug supervisor can't change the setting.
<wgrant> deryck: I didn't get an email for some of my team-owned projects.
<wgrant> But I suppose they might have had the flag disabled.
<deryck> wgrant, ah right.  Yes, if they already had this disabled, you would not have been contacted at all.
<bdmurray> sinzui: I'm adding the security checker for IOfficialBugTagTargetRestricted so yes I believe IProduct launchpad.Edit is currently used
<sinzui> bdmurray, can you pastebin the checker addition?
<wgrant> It will use the most specific adapter.
<wgrant> There is no way around it.
<bdmurray> sinzui: http://bazaar.launchpad.net/~brian-murray/launchpad/modify-official-bug-tags-permissions/revision/11593
<wgrant> (unless you have the tag stuff on a separate object, or use a different permission name)
#launchpad-dev 2010-10-06
<bdmurray> okay, so creating a new permission name and adding it to permissions.zcml works
<sinzui> bdmurray, we do not want to invent permission names
<bdmurray> sinzui: that makes sense but where does that leave us?
<deryck> Later on everyone.  until tomorrow
<sinzui> bdmurray, I do not have a clear path. It took us more than a year to sort the mess out with IProduct, and CommercialAdmin still does not belong the the flacoste cannon.
<sinzui> or canon
<sinzui> either works for me
<sinzui> bdmurray, I am strongly in favor of a Permission like BugEdit that grants supervisors permission to change bugtags and configure bug-trackers
<sinzui> bdmurray, I believe the project owner is delegating responsibility to the supervisor to manage the bugs domain. I would expect this person to have the power to do his job.
<bdmurray> sinzui: so a new permission that was used in multiple places would be agreeable then?
<sinzui> I think we need more than me to agree on this.
<sinzui> bdmurray, we have a launchpad.Driver permission
<bdmurray> sinzui: from a social standpoint or a technical standpoint?
<sinzui> social. The driver is a delegated role, and there is a launchpad.Driver. For the sake of symmetry, I expect launchpad.BugSupervisor
<bdmurray> okay, fwiw I've spoken to deryck and the Ubuntu Tech Board about this
<sinzui> bdmurray, when we talk about roles, responsibility, and permission, the bug supervisor and driver roles are not very clear, and in 6 months, they could be indistinguishable because of changes to access and notifications...
<sinzui> but if there was a permission like BugSupervisor, there were be a clear way to discuss what that role does
<bdmurray> yes, I think using a BugSupervisor permission makes sense too
<sinzui> and I think if we updated Lp to use launchpad.BugSupervisor, we will discover that driver and supervisor will never converge after the privacy changes are mae
<sinzui> bdmurray, lets assume launchpad.BugSupervisor will happen. update your checker and the zcml
<sinzui> !!
<sinzui> I now know why I am being inundated by requests to close projects. I have seen these projects listed in deryck's script. We have inadvertently reminded people they have dead projects.
<bdmurray> yes, not notifying inactive projects might have been a good constraint
<sinzui> how do we know they are inactive? if I knew that, I could write a process to deactivate them after 6 months or a year
<bdmurray> I don't know but I went to one's page and it said it was inactive ...
<bdmurray> https://edge.launchpad.net/10n-linuxme
<sinzui> bdmurray, I search for those every 3 months
<sinzui> I also deactivate about 2k of projects this year that had no license or artefacts
<sinzui> karma cannot be used because milestones and releases to not generate karma
<sinzui> and I know if the project is linked to another community like ubuntu, it cannot ever be deactivated,
 * sinzui has 9 months to solve this before his goals are up for review
<bdmurray> sinzui: if I made a merge proposal for my branch is that something you'd be willing to review?
<wallyworld> lifeless: or someone? 2 days ago i landed a branch via pqm, got a success email, but the bug tag is still not updated to qa-needstesting and it's not merged into devel yet. is this expected or have i missed something?
<lifeless> wallyworld: what branch did you merge it into ?
<wallyworld> lifeless: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/
<wallyworld> that was the submit target
<lifeless> whats your branch?
<wallyworld> bzr+ssh://bazaar.launchpad.net/~wallyworld/launchpad/windmill-1.3r1544
<lifeless> wallyworld: no idea; send it again ?
<lifeless> forward me the success message, I might be able to get some idea from pqm logs (they are on sodium, if you're interested in looking yourself)
<wallyworld> lifeless: ok, i'll give it a go. that one from yesterday (sql logging) also needs to be resubmitted because of translation test failures on ec2. are these fixed now?
<lifeless> 2010-10-05 20:14:45 INFO    Database replication lagged 11:06:29.567146. Sleeping up to 10 minutes.
<wallyworld> lifeless: i'll look at the logs on sodium as well.
<lifeless> bwahaha
<lifeless> wallyworld: dunno about test failures
<lifeless> wallyworld: I assume its all working all the time ;)
<wallyworld> lifeless: yeah, should be but, AssertionError: Failed doctest test for 10-distro-translation-group.txt
<wallyworld>   File "lib/lp/translations/tests/../stories/translationgroups/10-distro-translation-group.txt", line 0
<wallyworld> can't see how it's at all related to a sql logging change that isn't even activated if the env variable is not set
<lifeless> the failure details are later ;)
<wallyworld> lifeless: canonical.testing.layers.LayerIsolationError: Librarian has been killed or has hung.Tests should use LibrarianLayer.hide() and LibrarianLayer.reveal() where possible, and ensure the Librarian is restarted if it absolutely must be shutdown: [Errno socket error] [Errno 111] Conne
<wallyworld> looks like librarian layer has hung or something?
<wallyworld> i knew the failure was later, just didn't want to cut n paste too much :-)
<lifeless> that was discussed on the list
<lifeless> its fixed
<wallyworld> oh ok. thanks. i'm a bit behind in my email reading :-(
<lifeless> de nada
<lvh> Hello!
<lvh> https://dev.launchpad.net/Running/VirtualMachine
<lvh> Under Alternatively
<lvh> Why does the vm need to be built as root
<lifeless> I don't know! poolie came up with that recipe.
<lifeless> he might know
<poolie> lvh, i think that's just an implementation limitation of vm-builder
<poolie> i think because it creates the image using debootstrap, and that creates a chroot, and that can only be done as root
<poolie> imbw
<wgrant> It needs to chroot, right.
<lvh> Oooh. Okay, thanks.
<wgrant> vm-builder doesn't use a VM.
<lvh> wgrant: Before this recipe I used Virtualbox, hence the confusion.
<wgrant> vm-builder doesn't install from an ISO, so it needs to be able to chroot in and install packages.
<lvh> right
<poolie> wgrant, well, it creates a chroot and then it moves it into a vm
<wgrant> poolie: Right. But other VM solutions don't need root because they run the installer in a VM.
<poolie> right
<poolie> in theory vm-builder could support the same external interface and work by booting a network image within the vm
<poolie> if you don't have root on the machine you want to use as the vm host i would suggest following the first set of instructions
<poolie> they're similar, just a bit more manual
<lvh> It's not really a big deal, I was just wondering :-)
<lvh> It's still super-awesome that you people open-sourced lp.
<wgrant> Why are you running it in a VM? To avoid destroying your system, or because you don't run Ubuntu?
<lvh> wgrant: Former
<lvh> wgrant: I virtualize pretty much everything all the time
<wgrant> Sounds like a good policy,.
<lvh> Well, I'm a Python guy
<poolie> mm
<lvh> Setuptools is a hard and cruel teacher.
<lifeless> wallyworld: replied to your perf tuesday mail; backt o stacking wood for me.
<wallyworld> lifeless:  thanks. i'll have a look
<lvh> wgrant: Oh by the way, I noticed you guys are starting to do CD
<lvh> That's really awesome, I used to do CD at my job and am now blogging about doing it in a reusable fashion.
 * wgrant isn't actually quite one of those guys.
<lvh> Has whoever it is responsble considered documenting this CD process?
<lvh> wgrant: Whoops. Yeah, that "you" was plural
<lvh> By which I mean whoever it is now doing CD on lp.
<wgrant> lifeless knows a lot about that effort.
<wgrant> But he's just run away.
<wgrant> Most of it should be documented on dev.launchpad.net.
<wgrant> https://dev.launchpad.net/LEP/ReleaseFeaturesWhenTheyAreDone is the LEP.
<wgrant> https://dev.launchpad.net/MergeWorkflow, https://dev.launchpad.net/QAProcessContinuousRollouts are relevant.
<poolie> lifeless, nice public blog post btw
 * poolie works out which of the 20 meanings of "CD" this is :)
<poolie> lvh, i'm really pleased we're doing https://dev.launchpad.net/LEP/FeatureFlags now too
<poolie> which is subsidiary to CD
<lvh> poolie: I meant continuous deployment.
<lvh> poolie: Not sure what the other ones are
<lifeless> lvh: hi, yes, we are documenting it
<lifeless> the LEP documents some specific changes, and links to the MergeWorkflow which links to the qa process and related automation
<lifeless> we'll be open sourcing the qa toolchain soon
<lifeless> wgrant: yes, rosetta stuff is failing
<lifeless> wgrant: did you want the error report?
<lvh> lifeless: Awesome!
<lvh> lifeless: The only real problem I had implementing this myself was push notification
<wgrant> lifeless: No, just wondering about the ec2 failure I had overnight.
<lifeless> kk
<poolie> lifeless, it would be nice to document the terminology around tasks, events, etc into the arch guide if it ever settles
<rockstar> lifeless, https://code.edge.launchpad.net/~rockstar/launchpad/merge-queues-db/+merge/37686
<rockstar> lifeless, (please)
 * rockstar should learn some manners
<lifeless> rockstar: it has a *huge* diff in it.
<lifeless> mwhudson: hey
<lifeless> mwhudson: do you think a third line by line review is needed of lp-service?
<lifeless> mwhudson: spiv went at it hammer and tongs
<rockstar> lifeless, the diff is huge because of sampledata.
<rockstar> lifeless, the sampledata files are the bane of my existence.
<lifeless> right, you need to use pg 8.4 nowadays
<lifeless> rockstar: I hates them too.
<mwhudson> lifeless: well, maybe not of the code, but i spotted a few little things that could do with fixing in comments & such
<mwhudson> lifeless: i'll try to find some time to root them all out
<lifeless> mwhudson: needs or nices
<lifeless> mwhudson: if its nices, perhaps we can land and iterate?
<mwhudson> one need
<rockstar> lifeless, I used pg 8.4.  The diff would be a lot bigger otherwise.
<rockstar> Oh crap.  Is it SERIOUSLY that big of a difference between 8.4.3 and 8.4.4?
<lifeless> apparently
<lifeless> stub: hey
<stub> yo
<lifeless> stub: I was wondering if its 8.4.3 vs 8.4.4 (the bug portlet query thing)
<stub> Shouldn't be with a patch release, and we are running 8.4.4 everywhere anyway.
<lifeless> ok
<rockstar> stub, I put a review in for you.  Apparently there's something wrong with the sampledata that I'm trying to fix now, but the patch is ready for your eyes.
<stub> rockstar: Sampledata isn't that bad - only one spurious table moving position. The other changes are for tables your patch touches.
<rockstar> stub, yeah, I usually see changes like that in sampledata, but lifeless said in Needs Fixing, so I'm trying to Fixing.
<stub> rockstar: The json configuration is a bit of a worry. Soyuz did the same thing several years ago with 'lucille_config' and have been bitching about it ever since.
<stub> rockstar: You are running 8.4.3, latest is 8.4.4. That is the likely cause.
<rockstar> stub, yeah, this json config thing will not be available to users on the frontend.  It's specifically for Tarmac.
<rockstar> stub, basically, it's a way for Tarmac to get configs from Launchpad instead of having to configure it locally.
<stub> rockstar: So should it be tarmac_configuration? Or is tarmac going to be mushed into Launchpad and lose its identity as a separate product?
<rockstar> stub, well, I don't want to shut out something better from coming along (if something better CAN come along)
<stub> rockstar: Is the name column for use in URLs, or is it the displayname?
<rockstar> stub, ah, yeah, we _might_ need it in the urls (I'm still only doing the data model right now).  I guess that means that it needs an index.
<stub> I'm thinking about a valid_name() CHECK constraint
<stub> Shouldn't need an index, as we will be looking it up in conjunction with the owner and there is already an index for that
<rockstar> stub, hm, I've never seen a valid_name CHECK constraint, but I'm happy to add it if you can give me an example.
<stub> Sure. I'll add it with the needed indexes.
<stub> rockstar: So there is no comment on Branch.merge_queue_config. I'm wondering if that is redundant, or why it is different from BranchMergeQueue.configuration
<rockstar> stub, so the branch and the queue can both have config values.
<stub> And tarmac munges them together into a single config?
<rockstar> stub, yeah, kinda.
<rockstar> Currently, it actually un-munges them apart for certain contexts.
<rockstar> Huh, launchpad-developer-dependencies is broken in the PPA for maverick...
<rockstar> *sigh*
<rockstar> stub, there's a comment for Branch.merge_queue_config but wasn't one for Branch.merge_queue - I'm remedied this.
<stub> oic
<rockstar> stub, does this look right? http://pastebin.ubuntu.com/507002/
<stub> I've just put in the review
<stub> rockstar: Mostly fine. Its missing a comma on the preceding line.
<rockstar> stub, yeah, I just found that the hard way.
<lifeless> wallyworld: so that test failure looks like the ff *package* rather than ff *product* is showing up
<stub> Possibly an ordering problem
<rockstar> stub, why do you need an index on BranchMergeQueue.registrant?
<wgrant> rockstar: Missing python-codespeak-lib?
<stub> rockstar: Because we need an index on every reference to the Person table or the person merge code becomes glacial.
<lifeless> stub: as opposed to ? :P
<rockstar> stub, ah, okay.
<stub> rockstar: Similar with references to LibraryFileAlias and the librarian garbage collector.
<wallyworld> lifeless: if the doc test is actually incorrect, how did it get so that it was included in my test run? wouldn't the landing of that other branch have been rejected?
<lifeless> wallyworld: my experience has been that usually my change managed to break something nonobvious
<stub> wallyworld: There are some tests that rely on ordering returned by PostgreSQL rather than. These can explode at any time.
<rockstar> wgrant, it looks related to python-pocket-lint
<wgrant> rockstar: There is a known issue with python-codespeak-lib having been renamed to python-py.
<wgrant> launchpad-developer-dependencies depends on that.
<rockstar> wgrant, http://pastebin.ubuntu.com/507006/
<wallyworld> lifeless: ok, i'll recheck. but the my branch is for the sql logging enhancement and the code isn't even activated unless an env var is set. still, i may have missed something
<rockstar> wgrant, apparently python-pocket-lint has no installation candidate.
<lifeless> wallyworld: bzr diff -r ancestor:../devel | less
<lifeless> wallyworld: is a useful way to see whats really different in your branch
<rockstar> wgrant, this is on a fresh maverick chroot and rocketfuel-setup
<wgrant> rockstar: sinzui uploaded a new pocket-lint yesterday, and the bug ate maverick's copy.
<wgrant> We can probably just copy the new one up from lucid.
<rockstar> wgrant, okay, is there a fix planned?
 * rockstar should probably go read the list...
<wgrant> sinzui: Is there a reason that pocket-lint 0.5.5 is only in lucid?
<mwhudson> surely launchpad-developer-dependencies doesn't need to depend on py.test any more?
<wgrant> How is production-devel looking?
<wgrant> Has buildbot blessed it since my fix landed?
<lifeless> wgrant: unchanged
<lifeless> wgrant: your fix?
<wgrant> lifeless: The fix to unfuck Soyuz.
<wgrant> _getOtherPublications blah blah.
<lifeless> the last I heard was this ProcessAlreadyExited thing
<lifeless> is that what you're talking about?
<lifeless> twisted.internet.error.ProcessExitedAlready
<rockstar> wgrant, Soyuz can be unfucked?
<rockstar> :)
<wallyworld> lifeless: just checked - the diff is the same as shown on the original mp. ie some extra coding code inside an if block. nothing else. sigh.
<lifeless> wgrant: this is what rev 11595 or whatever was meant to fix, but thats in prod-devel and it still fails.
<lifeless> wallyworld: throw it at ec2 again :(
<wallyworld> s/coding/logging
<lifeless> wallyworld: if the test passes locally.
<wallyworld> lifeless: will do 4th time lucky hopefully
<lifeless> (the failing test I mean)
<wallyworld> yeah, i knew what you meant :-)
<lifeless> try merging devel into your branch and running the failing test.
<wallyworld> just did that before i ran the diff :-)
<lifeless> k
<wgrant> rockstar: Hopefully, yes.
<wgrant> lifeless: bigjools CP'd something last night for me. That's all I know.
<rockstar> wgrant, does it involve a Delorian?
<wgrant> rockstar: That would be nice, but I think we can work around the lack of one.
<rockstar> wgrant, reminds me of this tweet: http://twitter.com/#!/rockstar_/status/26408378955
<wgrant> Heh.
<wgrant> Sometimes it would be nice to go back and fix the data model, but we're a few years too late :(
<wallyworld> lifeless: test passes locally. will try ec2 again. wish me luck :-)
<rockstar> wgrant, yeah. I find myself saying that about a lot of things, so much so that I'm afraid to make NEW data models, for fear that I'll be cursing the person who created the data model in a year.
<wgrant> Eh, Code's is trivial in comparion. And it's fixable.
<rockstar> wgrant, yeah, but source package recipes isn't really "code."  It's kinda the fence between the US and Mexico.
<rockstar> In the "kinda broken, probably full of holes, but politicians sure love talkin' about it"
<rockstar> ...sense.
<wgrant> We'll get there eventually.
<wgrant> lifeless: So production-devel is still broken, so we probably won't be rolling out any fixes from it tonight?
<mwhudson> lifeless: i just did a line by line review of lp-service
<lifeless> wgrant: until someone figures out whats wrong, no.
<lifeless> wgrant: hang a sec
<lifeless> mwhudson: thank you!
<lifeless> wgrant: lp:~lifeless/launchpad/cp
<wgrant> The rev I'm interested in isn't in that lot.
<lifeless> I just pushed r 9785
<lifeless> and forwarded you mail
<wgrant> Ah, 9784.
<wgrant> Hm, I see.
<wgrant> That is odd.
<wgrant> Which was the last blessed rev?
<wgrant> 9779?
<lifeless> hit launchpad.net
<lifeless> press ctrl-U
<wgrant> .... what.
<wgrant> So it hasn't broken since then. It already was. Great.
<lifeless>     r9783
<wgrant> Yep.
<lifeless> is blessed and deployed
<wgrant> And neither 9784 nor 9785 could have caused that failure.
<lifeless> I can press 'build again' for kicks.
<wgrant> Hm, U1 is actually getting somewhere.
<wgrant> Finally!
<poolie> mm?
<wgrant> The website no longer sucks and they are offering new features.
<wgrant> And contact syncing is turned on again.
<wgrant> And I haven't found any critical security flaws in a while.
<poolie> heh
<lifeless> wgrant: glad to see you looking on the bright side
<bac> hi danilos
<wgrant> lifeless: Well, there *is* a bright side now :)
<poolie> hi bac
<wgrant> For example, I can no longer erase your filesystem by getting you to click a link.
<poolie> there's alwaysa  bright side
<bac> hi poolie
<bac> bye poolie
<wgrant> Heh.
<poolie> however, C-w in xchat being bound to 'close tab' is not a bright side :)
<wgrant> That always hits me when filing bugs.
<poolie> argh, i just remembered that when i used emacs and firefox, meta-q was fatal
<poolie> soooo awful
<poolie> i think they eventually removed it
<lifeless> I think I'll use this evening...
<lifeless> to ...
<lifeless> write some code!
<wgrant> !!!
<wgrant> How is the tokenised staging librarian going?
<lifeless> wgrant: you know what happens when you've got more items to service than servicers?
<lifeless> wgrant: thrashing... thats what.
<lifeless> wgrant: [its up there with the other highest-priority-we-have LP tickets]
<lifeless> wgrant: my first couple of weeks in the job could be paraphrased as 'saturate the losa queue till christmas. Thanks.'
<wgrant> Hah.
<lifeless> spm: am I wrong?
<spm> which year, for christmas, has been left undefined.
<wgrant> We don't have that new LOSA yet? :(
<lifeless> spm: lol
<poolie> the last thing we need is more code :)
<spm> wgrant: not yet, we had 3 interviews "today" tho
<lifeless> wgrant: I've got >10 tickets at or above pri 80
<lifeless> wgrant: nearly all are RFWTAD items
<lifeless> wgrant: rt 41202 is the tokenised librarian, its top equal with 3 others, in the LP queue.
<lifeless> spm: can I get profiling on staging pretty please?
<spm> yah, give us a sec tho. just kick off the slow bit on this update
<lifeless> wgrant: can you reproduce the failure?
<lifeless> wgrant: it might be a python 2.5 thing.
<wgrant> lifeless: I don't know where the failure is.
<lifeless> WOW
<lifeless> bah
<lifeless> wow
<wgrant> buildbot emails don't detail the failure.
<lifeless> Person:+branding 65 1845.67 411.42
<wgrant> Which is pretty awesome.
<lifeless> wgrant: huh, sure it did
<lifeless> mmm
<wgrant> I know that some test failed with that error, but only because Julian said so.
<StevenK> They don't, you need access to buildbot to see it
<StevenK> FWIW, I can't reproduce it locally
<lifeless> StevenK: using production-devel?
<StevenK> Yup
<lifeless> wgrant: privmsged you
<wgrant> I don't see any actual errors in that.
<wgrant> Just a lot of noise...
<lifeless> wgrant: StevenK: I suspect a python 2.5 issue
<lifeless> like the one stub found with filenotfound errors raising different exceptions
<wgrant> So, it's unrelated to the one fixed in r11594
<wgrant> This error happens when the slave breaks horribly.
<StevenK> Bleh, you say after I purged python2.5 yesterday
<wgrant> lifeless: It's not showing up in lp or db_lp, though?
<lifeless> because they are fucked
<lifeless> and don't work at all
<wgrant> Well, that's encouraging.
<lifeless> yeah
<lifeless> we're || close to having wildcherry migrated
<wgrant> Wildcherry's happening tonight, yeah?
<lifeless> downtime in 7 minutes or so and then tomorrow to switch back
<lifeless> spm: tell me when ;P
<StevenK> lifeless:   File "/home/steven/launchpad/lp-branches/production-devel/parts/scripts/site.py", line 143
<StevenK>     with f:
<wgrant> StevenK: You probably need to rebuild your tree for 2.5.
<StevenK> Fixing that gives: TypeError: bzrlib._static_tuple_c.StaticTuple is not a type object
<wgrant> site.py isn't under version control.
<StevenK> Right
 * StevenK does so
<spm> lifeless: gah. sorry. is restarting now
<wgrant> WTF
<wgrant> 2010-10-06 12:32:14+0530 [HTTPChannel,1,127.0.0.1] 127.0.0.1 - - [06/Oct/2010:07:02:14 +0000] "POST /rpc/rpc HTTP/1.0" 200 148 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
<wgrant> When I run the test locally, that ends up in /var/tmp/buildd/build-slave.
<wgrant> Er, /var/tmp/build-slave.log.
<wgrant> That's a very odd timezone.
<poolie> that control to turn on expiry is kind of hard to find
<poolie> being under "Bugs are tracked where"
<lifeless> poolie: file a bug :)
<StevenK> wgrant: +0530 is a twisted's default timezone, I think
<StevenK> wgrant: I came across it during sftp hacking, and jml told me why, I just can't recall
<wgrant> To try to break everything that exists, probably...
<poolie> somewhere in india?
<StevenK> I think so
<lifeless> +1230 would be better
<poolie> right
<StevenK> +0530 is Indian Standard Time
<StevenK> +1230 doesn't exist, +1245 is the closest
<lifeless> spm: still here?
<spm> yup; sorta. doing the incident report for forster/mailman
<lifeless> do the profile files rsync on cron
<lifeless> or is it totally manual-when-asked?
<lifeless> spm: ^
<StevenK> lifeless: Can't reproduce with python 2.5 either
<wgrant> StevenK: What does your slave log say?
<StevenK> wgrant: http://pastebin.ubuntu.com/507065/
<wgrant> Right, same here.
<wgrant> The one in the failure finishes INIT and starts on UNPACK.
<stub> We set the timezone in the test runner to India to catch bugs.
<wgrant> It's possible that the production-devel buildbot sucks at timing.
<wgrant> The test is probably racy.
<lifeless> it was run twice in a row
<lifeless> is it the new test?
<StevenK> Bleh! This branch is cursed
<wgrant> Aha.
<wgrant> Adding a time.sleep(1) makes it explode. Not the same way, though.
<wgrant> So the test is racy.
<wgrant> I think the buildbot might just manage to run the buildd further than my laptop can, so it dies before the test can abort it.
<wgrant> That error is well-known by anyone who runs the buildd locally, so it's been around forever.
<lifeless> ok
<lifeless> so you can fix?
<wgrant> Is the test new, do you know?
<wgrant> I suspect it is.
<wgrant> It is.
<lifeless> its new in 9780
<lifeless> which is passed and deployed
<wgrant> That only means it needed to succeed once, though.
<lifeless> 4 times
<lifeless> I'll force a build
<wgrant> If this doesn't work, I'd disable it. It's not testing anything new, and the whole slave testing framework is broken anyway.
<StevenK> wgrant: Isn't Julian fixing that with his new branch
<StevenK> ?
<wgrant> He's not fixing the slave, but he's probably replacing that bit of the master.
<wgrant> Basically, the slave test config works OK for the first couple of hundred milliseconds of a build.
<wgrant> After that it explodes spectacularly.
<StevenK> And most of the tests are probably touched/re-written
<wgrant> We could probably fix it by replacing the test config's unpack command with a sleep.
<wgrant> And probably cleanup too.
<lifeless> spm: ping
<spm> lifeless: sorry, the logs should sync every so often; but iof not there; I'll prod
<lifeless> please do; if you can tell me the frequency and phase I'll bother you less
<spm> normally every 3 or 6 minutes
<lifeless> well, I left it 20 minutes
<adeuring> good morning
<mrevell> Hello
<danilos> bac, hey-hey (sorry, I start late on Wednesdays :)
<wgrant> bigjools: Morning.
<bac> np, danilos
<danilos> bac, how's it going?
<bac> danilos: i was wondering if you could look at a patch for me?
<bac> http://pastebin.ubuntu.com/507036/
<danilos> bac, sure
<bac> danilos: i'm trying to insert the robots "noindex" conditionally.  for some reason, even when context/translatables returns products, the noindex is inserted.
<bigjools> morning
<bac> danilos: so the last test at line 66 fails
<bac> hi bigjools.
<bigjools> hey bac, how's the orient?
<bac> bigjools: having a good time,thanks
<lifeless> wow 1 second in 'currentseries'
<danilos> bac, right, give me a minute, otp
<bac> danilos: ok
<bigjools> bac: cool - have you been before?
<bac> bigjools: yeah.  this is my fifth trip since 2000.
<bac> bigjools: the smog is about to choke me to death, though.  :(
<bigjools> bac: urgh :(
<bigjools> jml: around yet?
<bac> it's breezy and temperate out but i've had to seal up the apartment and turn on the AC.
* lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is open for business | firefighting: - | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<wgrant> Should I be concerned that my first thought upon seeing "breezy" was "must... kill... sampledata"?
<lifeless> losa: can disable staging profiling - thanks.
<bigjools> wgrant: it must die but approx 1 million tests will need fixing
<wgrant> bigjools: Speaking of tests, have you seen the wonderful probably-race in production buildbot?
<bigjools> wgrant: yes
<bigjools> currently trying to work out wtf is going on
<wgrant> I'm pretty sure it's a race. Try adding a time.sleep(1) in that test.
<wgrant> Boom.
<wgrant> Different, but very similar, failure.
<wgrant> The test slave setup is known to be crap.
<bigjools> problem is, why is it happening now?
<bigjools> nothing's landed there for ages
<wgrant> The test has only been around for a couple of weeks.
<wgrant> It last passed the production buildbot two revisions ago. And it's clear that neither of them broke it.
<bac> wgrant: which test are you discussing?
<wgrant> lp.buildmaster.tests.test_builder.TestSlave.test_abort
 * bigjools stabs layers
<bigjools> ah that is one of the new tests indeed
<wgrant> It was added only two weeks ago.
<bigjools> I know what the problem is then
<wgrant> Oh?
<bigjools> race condition
<bigjools> same sort of thing that I fixed recently in the buildd-manager
<wgrant> Sort of.
<wgrant> There are two issues.
<wgrant> The ProcessExitedAlready thing is not the main one.
<wgrant> It will still break if you fix that.
<bigjools> really?
<wgrant> The test slave will pretty quickly fail and end up no longer BUILDING.
<wgrant> So the abort will fail.
<wgrant> That's what happens if you add the sleep into the test.
<wgrant> The ProcessExitedAlready probably happens when the test aborts it at the wrong time. When I run the unmodified test locally, it never makes it out of the INIT stage.
<wgrant> The buildbot log I've seen showed it get into and fail UNPACK, then get into CLEANUP, start to fail that, then get aborted.
<bigjools> I don't understand why you think catching that exception won;t work
<wgrant> The root cause is that the test slave config is only good if you abort it before the build gets anywhere.
 * bigjools files bug and disables test
 * bac should pay more attenting to those LP downtime messages...
<wgrant> We've overrun the window by a long time.
<wgrant> So that wouldn't have helped.
<lifeless> stub: ping
<stub> lifeless: pong
<wgrant> bigjools: 	exceptions.ValueError: Slave is not BUILDING when asked to abort
<lifeless> what do you think of the idea of caching 'distro.currentseries' rather than calculating it every time?
<bigjools> hmm didn't see that
<wgrant> bug #236925
<_mup_> Bug #236925: Make Distribution.currentseries an explicit DB column <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/236925>
<wgrant> lifeless: ^^
<wgrant> bigjools: That's what remains of the race if the buildbot exception doesn't occur.
<lifeless> wgrant: hah
<bigjools> wgrant: ok
<lifeless> wgrant: well, see my mail just sent.
<lifeless> wgrant: 1 second of bug one is in figuring that out and returning the series object.
<lifeless> wgrant: for 6 distros
<danilos> bac, you seem to be setting usage on product, whereas a view is on project group, could that be related?
<bac> danilos: the usage for a project group is determined by its products
<bac> danilos: i'm really stumped, b/c the test at line 49 passes, showing the PG has translatables and that is the condition in the TAL expression
<wgrant> lifeless: How is that query so awful?
<danilos> bac, I don't think we are doing anything special in there for project groups in translations
<lifeless> wgrant: 200ms on avg
<wgrant> lifeless: That sounds like a bug.
<lifeless> (1sec/6 actually)
<wgrant> Hm.
<danilos> bac, have you seen an XXX by jcsackett on projectgroup.translatables though (though, I don't expect that'll help much)
<wgrant> Something is fairly broken.
<lifeless> it load all the series
<wgrant> We seem to be out of read-only, but sessions aren't working.
<bac> danilos: i did.  i've checked that the two states are in sync.
<wgrant> Ah, there we go.
<danilos> bac, so I'd say something is wrong with usage determination for project groups (I don't know how's that done)
<bac> danilos: ok, thanks for looking.  i thought there might've been something translation-specific i was overlooking, such as earlier when i didn't create a POTemplate for the product
<gmb> Anyone else getting "bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist." when trying to push to LP?
<gmb> Hmm. SSH key failure. That's weird...
<gmb> Oh.
<gmb> Wait.
<gmb> never mind.
<wgrant> We just came out of readonly a few minutes ago.
<wgrant> Doesn't quite explain it, though.
<gmb> wgrant: Yeah, I know what hte problem is.
<gmb> Basically, bits of my disk have gone away. This happens.
<bac> wgrant: web is back but code hosting isn't
 * gmb reboots; bbiafm.
<gmb> Aha.
<gmb> So, separate problem.
<danilos> bac, sorry, but I don't know how you decide if robots meta tag needs to be added or not, and I think that's what's important here; in general, a requirement for a template in a product is still there, but that depends on what condition exactly are you using when generating robots stuff
<wgrant> bac: "Permission denied" doesn't seem like an excellent failure mode for that.
<bac> wgrant: nope
<bac> wgrant: you said bazaar.lp.net takes longer to come back.  why and how much?
<bac> danilos: the test is just condition="context/translatables" but i assume you see that in the patch.  are you asking something else?
<wgrant> bac: I'm not sure why. It may be because the LOSAs just take a while to start it back up...
<bac> it is back
<gmb> \0/ login dance
<wgrant> Yes, the sessions seem to have evaporated.
<danilos> bac, right, I am not sure what's going on and it looks very, very weird
<wgrant> And authentication didn't work for a while.
<bac> danilos: yeah, thanks for looking.
<jml> bigjools: yeah, around now, sort of.
<bigjools> jml: can you look at the prod_lp buildbot output, the test_abort that we added is failing.  I'm going to disable it as I don't think there's a quick fix but you might see differently.
<jml> bigjools: at a first glance it looks like a race: a bug in the slave maybe
<bigjools> jml: we're not waiting for the build to start before trying to kill it
<wgrant> The issue is more that you're waiting too long, so the test config crashes.
<jml> in either case, bug in the slave
<jml> it's a server, it should handle people asking it to do things
<bigjools> yes
<bigjools> as I discussed with wgrant above, we need to catch that ProcessAlreadyExcepted and also deal with trying to abort something that's not started
<jml> yeah, that sounds about right
<wgrant> Or just fix the test config to sleep a lot.
<bigjools> no :)
<wgrant> The problem is that it gets too far. Not that it doesn't get far enough.
<wgrant> Adding sleeps won't delay the test suite.
<bigjools> I filed a bug 655559 about it
<jml> wgrant: even so, the slave should catch the ProcessAlreadyExited exception
<_mup_> Bug #655559: test_builder.py / test_abort is failing on buildbot, but not locally <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/655559>
<wgrant> jml: Indeed. It happens a bit locally.
<wgrant> But blah twisted blah confusing.
<jml> not at all.
<jml> try: kill_process(); except ProcessAlreadyExited: pass # yay less work for me
<bigjools> I wonder if in this case it should not be raising exceptions here, the vast majority of use cases don't care if it's already exited.
<lifeless> gmb: so talk to me
<bigjools> a return value would be sufficient if you need to know
<jml> bigjools: I don't know the use case here, but lots of similar code just swallows the exception
<jml> bigjools: it's not so much "kill the process" as "ensure that it's dead"
<bigjools> jml: yes - that points to a flaw in the API IMO
<jml> bigjools: file a ticket on Twisted if you'd like
<bigjools> mebbe
<wgrant> jml: OK then, "but blah twisted blah confusing blah slave blah untested."
<jml> wgrant: the slave is not going to get any better thinking that way
<jml> noodles775: I can't see the ec2 test details you mentioned in your email
<noodles775> jml: in the attached file on the message to launchpad-dev? (652838-select-diffs-for-syncing-r9844.subunit.gz)
<jml> noodles775: subject?
<noodles775> jml: '"ec2 land" and "ec2 test" results should be truthful once more' sent 18 minutes ago.
<jml> noodles775: nope, not there.
<wgrant> There were ML issues a few hours ago.
<wgrant> May still be broken.
<noodles775> Seems so, I'll forward it directly.
<jml> noodles775: thanks.
<gmb> lifeless: So, have you looked at the diff attached to that merge proposal (I'm assuming that that's what you want me to talk to you about)?
<lifeless> gmb: yes, and replied.
<lifeless> gmb: I suspect your best bet is
<gmb> Ah, just saw your reply.
<lifeless> bah
<lifeless> I suspect your best bet is not splitting the code up
<jml> noodles775: which windmill test failed?
<lifeless> -or-
<lifeless> change the contract for getSFP to not return a resultset
<lifeless> and then store the result in a cachedpropery
<lifeless> like
<lifeless> @cachedproperty
<lifeless> def _subscribers_for_persons(self): return {}
<noodles775> jml: sorry, I should have included that in the email: test_diff_extra_details_blacklisting
<jml> noodles775: interesting
<jml> noodles775: something is really borked here. there's way more failures than just that in this test run
<lifeless> gmb: and then assign results into that if person is not in self._subscribers_for_persons, or roughly that.
<gmb> lifeless: Hmm. Well, we *need* to split the code up for the sake of finishing the better-bug-subscriptions story, so I guess changing the way getSFP works would have to be the way to go.
<lifeless> gmb: alternatively, making sure that you don't use *any* subscribers etc other than in the new view's template.
<noodles775> jml: what's an example I can search for?
<lifeless> the problem is that multiple views is a sure fire way to get potato programming
<gmb> lifeless: I don't think that leaving the loop / list() in there will slow the old view down though (that's where the XXX is, btw) because the loop is already there; I've just removed the contents.
<jml> noodles775: well, just search for 'failure:' or 'error:'
<lifeless> gmb: it will
<jml> noodles775: or 10-distro-translation-group_txt
<jml> umm, the regex ^error: will work better
<lifeless> gmb: well, I'm assuming you're calling into the new template ?
<noodles775> jml: I did searcth for failure... (lots of tests have that in their name). I'll take a look soon (just otp)
<gmb> lifeless: Let's back up a second...
<lifeless> gmb: if you're not, then just list() the getSFP and you're done
<gmb> lifeless: Right.
<lifeless> (but it would be a good idea to stop generating subscribers *at all* for the non-javascript version.
<gmb> Agreed.
<lifeless> that or get rid of the separate javascript version.
<jml> noodles775: the colon is important.
<lifeless> because we're doing the work twice at the moment.
<gmb> lifeless: That's what splitting the code will allow us to do.
<jml> lifeless: subunit is buggy.
<lifeless> jml: all code is.
<lifeless> jml: I'd love to talk more about this right now, but if I don't halt(), I'll have had way too little sleep at 6am for the tl meeting.
<jml> lifeless: understood.
<jml> lifeless: sleep well.
<lifeless> gmb: may the force be with you. Or something.
<gmb> lifeless: Thanks. Enjoy your evening and may LP not be too broken when you awake.
<jml> noodles775: I think I've figured it out.
<wgrant> bigjools: Have all the sparc/ia64 pubs been fixed for good? The archive freezes rather soon.
<jml> noodles775: let me know when you're off the phone. I'd like to talk it through.
<noodles775> jml: back... did you want to mumble, or chat here?
<jml> noodles775: chat here is good.
<jml> noodles775: if you grep for the LayerIsolationError, you'll see that there are no actual errors before then, and any errors after them were not reported
<jml> noodles775: something about it is messing up subunit's parsing
<noodles775> Right.
<jml> noodles775: if you look in lib/devscripts/ec2test/remote.py, in the  test() method (line 274) you'll see the guts of ec2 test
<jml> noodles775: there's some code there like:
<jml>             # The process could have an error not indicated by an actual test
<jml>             # result nor by a raised exception
<jml>             if result.wasSuccessful() and retcode:
<jml>                 raise NonZeroExitCode(retcode)
<noodles775> I'm hoping the windmill failure that I had (which was reproducible locally with FF, but not Chromium) was the only real error (ie. not related to the librarian failure)
 * noodles775 checks buildbot waterfall.
<jml> I guess I'd like to know how bin/test managed to have a failed test result but still returned a zero exit code
<jml> or if remote.py is buggy in this regard.
<bigjools> wgrant: no it's still screwed
<bigjools> I've no idea why at the moment
<bigjools> a-f keeps publishing the Packages file after it's deleted
<noodles775> OK, so it seems the windmill test was the only real failure - great (as this branch obviously landed).
<wgrant> bigjools: I guess I'll try to fix it locally. The key bit is getting it out of Release.
<jml> noodles775: in any case, I'm going to try to write a fix for this in subunit
<wgrant> Everything else is just a bonus, since we can delete Packages and Sources later.
<bigjools> wgrant: cheers
<jml> noodles775: unfortunately I have no idea how to deploy that fix to our ec2 instances.
 * bigjools sighs at PQM taking *so freaking long* to land a branch
<wgrant> bigjools: Disabling that test?
<bigjools> yes
<bigjools> there was a branch in front of mine
<wgrant> Yay.
<bigjools> and mine's still getting processed
<bigjools> this is ridiculous
<jml> bigjools: I've filed an RT about it.
<bigjools> to land something in PQM is now taking longer than the entire test suite when I first started on LP
<bigjools> jml: yay - but do you know what can be done?
<wgrant> What takes all the time?
<jml> bigjools: well, the RT is mostly asking to give me & other developers access to the config so we can even begin to debug the problem.
<wgrant> sn't it just... a merge and grep?
<bigjools> I think it runs a make before merging
<jml> who knows!
<jml> that's the point
<jml> silly to talk about what it might do when there's a file on a computer somewhere that could end all speculation
<wgrant> But even that won't take very long, unless prasÃ©'s a 386...
<noodles775> jml: From memory henninge wrote up some instructions after he updated some package versions for ec2, or did you mean something else?
<jml> noodles775: oh, mostly I just mean that it's another complex step that I guess I'll figure out when the time comes ):
<jml> that was meant to be a smile
<noodles775> Yep.
<wgrant> bigjools: Is there a bug?
<bigjools> wgrant: bug 648715 might fit
<_mup_> Bug #648715: Binary publications should not be created for disabled architectures <qa-ok> <soyuz-publish> <Soyuz:Fix Committed by julian-edwards> <https://launchpad.net/bugs/648715>
<bigjools> I verified that new Packages entries are not written
<wgrant> I don't think so. I'll file a new.
<bigjools> yeah
<wgrant> +one
<bigjools> but the Packages file itself is renewed, for some reason
<wgrant> Well, it's fairly clear why. It'll be a simple fix.
<bigjools> I haven't looked into it yet because I'm busy cleaning up your bug ;)
<wgrant> Er, yeah.
<jml> jelmer: is there a PPA w/ latest releases of testtools, subunit etc?
<jml> actually, let me try to use Launchpad to find out
<jml> that was only a little hard.
<maxb> jml: Yes, and as a Launchpad developer, you already should have it enabled by rocketfuel-setup :-)
<jml> maxb: no, the release of testtools in that PPA is 0.9.2, latest is 0.96
<jml> 0.9.6, rather.
<maxb> However, see the ~bzr PPA
<jml> interesting
<jml> I wonder why https://edge.launchpad.net/ubuntu/+source/python-testtools doesn't list that
<jml> https://edge.launchpad.net/ubuntu/+ppas?name_filter=python-testtools times out
<wgrant> bigjools: The a-f tests are... special.
<bigjools> wgrant: yes.  and one of them fails on maverick
<wgrant> What is all this FakeSelectResult crap...
<wgrant> Oh, it lives in that module.
<wgrant> Even more special :(
<bigjools> wow my branch finally landed 1.5 hours after I submitted it
<jml> RT #41739, fwiw
<_mup_> Bug #41739: Increase number of Retry attempts <Launchpad Foundations:Fix Released by stub> <https://launchpad.net/bugs/41739>
<wgrant> bigjools: So we might be able to fix things eventually. Yay.
<bigjools> wgrant: I hope so
<bigjools> wgrant: I'll assign that bug to you I guess
<wgrant> Er, yes, please do.
<wgrant> It'd be nice if it let me do that while filing.
<bigjools> I thought you could?
<wgrant> Only if you're a bug supervisor.
<bigjools> sigh - the person picker is timing out
<bigjools> I'm not sure if I'm just angry today or if LP is getting on my tits, but this is incredibly frustrating
<bigjools> non-js FTW
<wgrant> We are no longer on wildcherry.
<wgrant> So it's not unthinkable that it might be slow.
<bigjools> I see jml filed a bug too
<bigjools> jml: bug 655620 made me laugh
<_mup_> Bug #655620: "Other versions" list doesn't include interesting other versions <Soyuz:New> <https://launchpad.net/bugs/655620>
<bigjools> I had other people asking "why doesn't it show X"
<bigjools> the answer is because LP is not a mind-reader :)
<wgrant> Tree builds take too long.
<bigjools> yes
<jml> bigjools: well, sure, Launchpad can't be expected to read minds. However there are two obvious improvements here
<bigjools> jml: it sorts by karma.
<jml> bigjools: one is that it could act less like a mind reader and explain why it picked those, perhaps having a link to "more" rather than a search...
<jml> bigjools: the second is that PPAs w/ newer versions of the package are probably more interesting to people
<bigjools> yeah
<bigjools> your bug might be a dupe.
<allenap> gmb: Do you have any time today or tomorrow to start work on some basic UI for subscribe to search?
<gmb> allenap: Define "start work"
<jml> noodles775: this subunit test reproduces the bug minimally: http://paste.ubuntu.com/507163/
<jml> it looks like it's actually two bugs :(
 * noodles775 looks
<jml> 1. zope.testrunner's subunit formatter should format KaboomError as a genuine subunit error
<jml> 2. subunit's parser should be more robust in the face of stupid input
<allenap> gmb: I think there's enough subscription filter API under the hood now, and in db-devel, to write something to subscribe to an advanced search, albeit if that advanced search only filters on statuses, importances and the presence or absence of tags.
<allenap> gmb: So it would be cool to have a button on the advanced search page saying "Subscribe to this mofo" that does something.
<gmb> allenap: Okay. At the moment my focus is on getting the Wizard integrated with devel (good luck with that) and making it possible to filter individual subscriptions by notification level.
<deryck> Morning, all.
<gmb> Which seems simpler for Weds wk3, though with continuous deployment being more and more practical maybe that's an unnecessary concern.
<gmb> allenap: Are you asking me to take the work on or are you saying you want to talk about how best to do it?
<gmb> Morning deryck.
<allenap> gmb: Take the work. I might have time to do it if I get a move on. But no worries, just wondering. Ping me if you find you have some time and you're interested.
<gmb> allenap: Okay, will do. At this point though I'd be surprised if I have time before next week,
<wgrant> bigjools: The a-f test failure in maverick seems to be caused by it becoming a bit more pedantic.
<wgrant> bigjools: The test uses a binary named 'foo.deb', which a-f just ignores. Renaming it to 'foo_1.0_i386.deb' fixes it.
<bigjools> wgrant: ha
<bigjools> nice
<wgrant> Heh. I was just going to file a bug about the failure, and entered the summary "test_ftparchive failing on Maverick". The first result the dupefinder found was "test_ftparchive failing on Lucid". So it isn't completely useless.
<jml> :)
<wgrant> This code is the stuff of nightmares...
<bigjools> enjoy :)
<wgrant> bigjools: Parts of the a-f code look at DS.architectures, while others look at archtags in lucilleconfig.
<bigjools> lucilleconfig needs to die in a furnace
<wgrant> Yes.
<bigjools> it's absolutely hideous
<wgrant> bigjools: Should I fix it for PPAs too? They aren't so critical, and they're a completely different part of the code.
<wgrant> NMAF is also used for partner, but partner's release pocket isn't frozen, so that should be fine.
<bigjools> wgrant: fix what for PPAs?
<wgrant> bigjools: Not publishing disabled archs.
<wgrant> PPAs have sparc/ia64 indices.
<bigjools> wgrant: I think so
<wgrant> As does partner.
<bigjools> yeah that's always been wrong, it ignores supported_arches
<wgrant> There's no critical need to get it fixed in the next 48 hours, so I'm tempted not to throw more into this CP.
<bigjools> +1
<wgrant> So they'll drop out of Release (because that code is shared), but the indices will still exist.
<jml> anyone able to help twb on #launchpad?
<bigjools> I hate test_translationtemplatesbuildbehavior.py
<wgrant> It's interfering with your async stuff?
<wgrant> Because Translations actually tests stuff? :P
<bigjools> no, because the way it tests is crap
<bigjools> millions of fake/stub
<wgrant> FFS
<wgrant> The native publisher still goes through FTPArchiveHandler
<jml> how do I get the return code of something I run in bash?
<wgrant> jml: $?
<jml> wgrant: thanks.
<cr3> deryck: hi there, I received a few messages from you about re-enabling auto expiring bugs. usability might've been better sending one message per person listing all their projects, instead of one message per person per project :)
<deryck> cr3, yes, I agree.  It was a mistake on my part.  My apologies for the noise.
<cr3> deryck: no problem, it was mostly a suggestion for next time rather than whining :)
<deryck> cr3, I understand.  Thanks for the suggestion. :-)
<cr3> deryck: my concern about next time is that these emails are sent on an adhoc basis, I suspect, so perhaps there should be a system in place to help the next person do the right thing, because it might not necessarily be you
<deryck> cr3, yeah, that's my feeling, too.  That we shouldn't have to write ad hoc scripts to email people, which a small mistake has huge ramifications.
<cr3> deryck: learning from ones experience is good, learning from other's experience is divine :)
<deryck> cr3, or else, we really shouldn't send email at all.
<allenap> jml: Are we inextricably tied to zope.test*?
<jml> allenap: as long as we use layers, yes.
<jml> allenap: but that's the only tie.
<allenap> jml: Do you know what it would take to move those to testresources or something like that? I ask because I'm interested in doing it.
<allenap> If it's worthwhile.
<jml> allenap: roughly speaking, yes. there's been some discussion about this on the mailing list, and I've filed a bug on launchpad-foundations somewhere
<jml> allenap: the discussion is quite recent, iirc. Sep maybe.
<allenap> jml: I'll try and find it. It rings a bell so I probably read it ;)
<wgrant> bigjools: Can we ensure that ia64 and sparc have 0 active publications in the primary archive upon release?
<wgrant> bigjools: If we can't, I need DB changes or a set of hacks to prevent indices.
<jml> thank you surveymonkey for losing my carefully constructed thoughs.
<wgrant> Because ftparchive loves its views.
<jml> *thoughts.
<wgrant> I guess I'll go with the extra layer of hacks.
<bigjools> wgrant: it's possible to run a query to delete all the publications, yeah
<wgrant> bigjools: The problem is this: ftparchive dumps BinaryPackageFilePublishing for the relevant (series, pocket), and marks everything it sees as needing a release file.
<wgrant> The distroarchseries is hidden behind the view, so I can't filter out disabled stuff.
<wgrant> So I can't remove the release file requests -- I have to go through and filter them later.
<wgrant> And they are at that point thoroughly stringified.
<stub> allenap: There is no longer anything tying us to layers. Teardowns need to be added to the layers that claim they can't be torn down (this is no longer correct). The layers would need to be broken apart into resources. All the tests need to declare their combination of resources rather than their layer, or some sort of compatibility layer added so layer declarations work in the new world order.
<allenap> stub: There's hope then! I assume I'm not the only one who thinks that zope.testing is byzantine.
<stub> allenap: sure. It never was a particularly good fit for us, but the only option available at the time.
<bigjools> wgrant: so we need to unpublish all the BPPHes
<wgrant> bigjools: And then really hope that none come back, or we're screwed.
<allenap> stub: Interesting. (Also, I wasn't around when it was chosen, and I certainly don't mean to knock the choices that were made.)
<bigjools> wgrant: well I'm preventing them in every place they get created now
<wgrant> I guess if we do a final check that everything's OK before we freeze the pocket.
<bigjools> wgrant: so to summarise: a) stop a-f publishing disabled DASes, b) unpublish them
<wgrant> bigjools: It's easy to stop it from publishing empty disabled DASes. Non-empty disabled DASes require a hack which I'm testing now.
<wgrant> I think it will work.
<bigjools> mmmkay
<wgrant> Still, the hack is mostly stolen from another rather hackish piece of work in the Release generator.
<bigjools> :'(
<wgrant> Which parses string pockets and does things like architecture[7:]
<wgrant> (that's to strip 'binary-' of the front)
<bigjools> we should not do that kind of hack
<wgrant> No.
<bigjools> I'd much rather run SQL to delete the publications
<wgrant> But that needs refactoring of lots of code and the a-f views.
<wgrant> Right.
<wgrant> We don't need to actually DELETE the publications. Just Delete them.
<wgrant> But I'm worried that some might spring up while we're not looking.
<bigjools> you mean status=deleted
<wgrant> Right.
<wgrant> They need to not be Published.
<bigjools> and I'm not too worried about them re-appearing
<bigjools> as long as we prevent builds getting created
<wgrant> You did fix copies as well, right?
<bigjools> yes
<wgrant> bigjools: I just pushed up the various changes to https://code.edge.launchpad.net/~wgrant/launchpad/bug-655614-disabled-arch-indices... see what you think.
<wgrant> I need to rewrite the tests properly, since they don't catch many a-f cases.
<wgrant> Of which there are far too many.
<bigjools> wgrant: can you do a WIP MP please
<bigjools> gets me a diff :)
<benji> hey guys, how I can run a script with the full LP configuration loaded? (adapters registered, etc.)
<bac> abentley, adeuring, allenap , bac, danilo, sinzui, deryck, EdwinGrubbs, flacoste, gary, gmb, henninge, jelmer, jtv, bigjools, leonard, mars, noodles775: Reviewer Meeting in 2 minutes
<wgrant> bigjools: The four revs are deliberately pretty independent. Viewing them in isolation is probably better.
<wgrant> We may only want one of them.
<bigjools> benji: execute_zcml_for_scripts()
<wgrant> And they're tiny.
<bac> wgrant: ^^
<bigjools> wgrant: ok
<benji> bigjools: ah ha!
<wgrant> I *think* that branch completely fixes a-f.
<wgrant> But the code is so convoluted that it's really hard to tell.
<bigjools> benji: yeah, in soyuz world it's a bit ubiquitous :/
<benji> heh
<deryck> bac, apologies, but I'll miss today.
<bac> deryck: thanks
<Ursinha> bigjools, hi, could you tell me why did you remove the tag and milestone from bug 566339?
<_mup_> Bug #566339: Cannot accept package which would notify private email addresses <boobytrap> <Soyuz:Triaged by julian-edwards> <https://launchpad.net/bugs/566339>
<bigjools> Ursinha: because its not fixed
<Ursinha> bigjools, did you consider using --incremental?
<Ursinha> removing tags will break the script
<bigjools> Ursinha: I attached the bug to the branch incorrectly, so I'm just reversing the change
<Ursinha> ah, ok
<bigjools> I removed the branch ages ago
<bigjools> I just noticed it was still in the milestone
<Ursinha> in this case removing the branch should work
<Ursinha> thanks bigjools :)
<bigjools> Ursinha: np!
<jml> allenap: did you find those discussions?
<allenap> jml: No, I didn't unfortunately. stub replied here with some interesting information.
<jml> allenap: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/419691
<_mup_> Bug #419691: Many tests use unnecessarily expensive layers <build-infrastructure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/419691>
<tyarusso> Hi, I'm having some trouble installing canonical-identity-provider, at the syncdb step.  Any ideas on this error?  http://pastebin.ca/1955643
<jml> allenap: "stories for external test helpers (librarian, memcache, etc)" on launchpad-dev@ (Sep 26)
<allenap> jml: Yep. Okay, the priority of that bug is now low, so maybe I should walk away.
<bigjools> the person picker on bug pages is constantly timing out today
<jml> allenap: the priority doesn't mean very much
<jml> allenap: I gave it a priority for the sake of giving it a priority.
<allenap> bigjools: Wfm. Any particular names?
<wgrant> tyarusso: It used to always default to False. Add 'READ_ONLY_MODE = False' to settings.py.
<bigjools> allenap: see if you can assign https://bugs.edge.launchpad.net/soyuz/+bug/55774 to "jelmer"
<_mup_> Bug #55774: Soyuz tests explore "Building from Accepted" when the production doesn't <tech-debt> <Soyuz:Triaged> <https://launchpad.net/bugs/55774>
<wgrant> bigjools: That bug is wrong, isn't it?
<wgrant> Oh, no, it's right.
<bigjools> wgrant: :)
<tyarusso> wgrant: Thanks.
<bigjools> ancestry from the queue is bong
<wgrant> I had that stupid difference between source and binary publications.
<wgrant> Custom uploads break everything :(
<allenap> bigjools: A bit slow, but it worked.
<bigjools> allenap: hmmm ok.  must be borderline then, thanks
<allenap> bigjools: It is slow though. That kind of thing should be near instant.
<bigjools> allenap: it's sometimes very quick, sometimes crawls for me
<tyarusso> wgrant: So I got past that step, was prompted to create a superuser, selected yes, gave my info, and after that got this:  Failed to install custom SQL for identityprovider.Account model: must be superuser to create procedural language "plpythonu"  - should I have said no?
<wgrant> tyarusso: It's expecting your postgres account to be a superuser.
<wgrant> You'll need that either way.
<wgrant> I'd "sudo dropuser $USER" and "sudo createuser -a $USER"
<wgrant> Then it should work.
<wgrant> Or I suppose you can alter the existing user, but I don't know how to do that offhand.
<tyarusso> dropuser: removal of role "tyarusso" failed: ERROR:  role "tyarusso" does not exist
<tyarusso> I'm guessing you don't actually mean my shell $USER, but the username defined in settings.py then.
<wgrant> Well, in a lot of cases that probably is $USER.
<wgrant> But if you've configured it differently, then indeed not.
<tyarusso> dropuser: removal of role "launchpad" failed: ERROR:  role "launchpad" cannot be dropped because some objects depend on it - argh
<wgrant> You don't want to play with the 'launchpad' user if you want to run Launchpad.
<tyarusso> There's an actual launchpad user?  Well frick.
<wgrant> A PostgreSQL user, yes.
<tyarusso> sigh - that's what I called mine :S
<wgrant> Launchpad handles its DB users itself, by making your shell account a postgres superuser.
<tyarusso> So I should be putting tyarusso in the two DATABASE_USER fields of settings.py then?
<wgrant> Unless you have a pressing need to use another user.
<tyarusso> mmk
 * tyarusso notes that documentation is lacking
<wgrant> Setting up a database depends a lot on the given environment, so any specifics
<wgrant> are left out, and what is here presented should only serve as a guideline.
<wgrant> ::
<tyarusso> Yeah.....not helpful.
<tyarusso> Only one of many things though - I had to manually finagle a bunch of the python modules too.
<wgrant> There's unfortunately no public ISD dev channel, so the relevant team is not accessible.
<wgrant> And I haven't run it for a while.
<wgrant> (c-i-p isn't developed by the Launchpad team)
<tyarusso> I *think* I might have identity-provider running now.  Maybe.  Running through the actual LP install again to see what happens.
<tyarusso> btw, is there a way to get db-stable directly, or do I have to get devel first and add it?
<wgrant> You could probably hack rocketfuel-setup to grab it directly.
<wgrant> But that's fairly pointless.
<tyarusso> k
<flacoste> am i the only one for whom the person or team ajax selector doesn't work?
<flacoste> it always fails with 'Failed to load results'
<wgrant> That's the third report of it, and I've seen it too.
<sinzui> leonardr, please retarget questions before you answer them. The question will be reopened by the user 50% of the time and getting a different person answers each question is very confusing
<leonardr> sinzui: ok, what is the proper target for a 'delete my project' question?
<sinzui> launchpad-registry
<sinzui> I a doing them now
<leonardr> thanks
<sinzui> leonardr, you only need to retarget. the teams get to do the work
<leonardr> oh, has that changed recently? i thought that was only for bugs
<leonardr> (sending out those reminder emails really brought people in to disable their projects)
<bigjools> flacoste: yes me too, although when someone else tried it it worked, albeit slowly
<tyarusso> Once I get this up, am I going to have to do anything special to get LP to talk to canonical-identity-provider?
<tyarusso> Also, is the python virtualenv stuff necessary if I actually want a system install, not just a development environment?
<bigjools> wgrant: still up?
<mars> sinzui, small annoyance - a recent maverick upgrade has Launchpad thinking that I am logged out, but the original LP login service says I am logged in.  Might come up with other Maverick users.
<mars> sinzui, hitting the Login link a second time made the site do the right thing
<sinzui> mars, if Lp thinks you are logged in, I think you are, gary or benji could confirm that
<mars> maybe it was browser caching
<tyarusso> Why do I have to have my SSH key registered with LP just to do a checkout of rocketfuel stuff?
<mars> tyarusso, that is so rocketfuel-get can use the bzr+ssh protocol.  IIRC it is not a requirement to fetch the source code
<lifeless> moin
<mars> jml, you must keep an +activereviews window open :)
<jml> mars: email. I basically live in my mail client.
<jml> also
<jml> how is it at all possible that these tests are failing in stable?
 * mars shrugs
<mars> jml, I first noticed the failure in production-devel
<mars> I have no idea how it got that far
<mars> before failing
<jml> failing tests in stable is pretty serious.
<jml> unfortunately I have to go.
<jml> good bye.
<mars> good night
<jcsackett> danilos: you around?
<lifeless> deryck: https://bugs.edge.launchpad.net/malone/+bug/607958
<_mup_> Bug #607958: timeouts on Distribution:+bugtarget-portlet-bugfilters-stats <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/607958>
<lifeless> deryck: its now failing much more often
<lifeless> 1509 /    0  Product:+bugtarget-portlet-bugfilters-stats
<deryck> lifeless, ok.  I'll add it to our backlog and have the next person free take a look.
<lifeless> thats > 1 / minute :(
<deryck> yeah, that's pretty ridiculous.
<lifeless> sinzui: / bigjools: 794 /   98  Distribution:+ppas may be a similar fallout
<lifeless> deryck: also 432 /  323  Distribution:+bugtarget-portlet-bugfilters-stats
<lifeless> We could raise the timeout, but it won't help - its been taking multiple minutes.
<lifeless> flacoste: hi
<lifeless> flacoste: downtime window - I think we should move to a budget based approach now :)
<flacoste> lifeless: i agree
<lifeless> 5 hours as an estimate scared me
<bigjools> eek
<lifeless> bigjools: see canonical-launchpad
<bigjools> irc fail, ignore that
<lifeless> ll
<lifeless> kk
<flacoste> lifeless: i had the same reaction
<Ursinha> hi abentley, is bug 613958 already QAed?
<_mup_> Bug #613958: upload failure emails should include the upload log <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by abentley> <https://launchpad.net/bugs/613958>
<abentley> Ursinha, no, but it's not for lack of trying.
<abentley> Ursinha, I am waiting on jelmer to fix the staging build farm.
<Ursinha> abentley, oh, I see.
<Ursinha> abentley, I made a note on the bug. Thanks for the info
<lifeless> elmo: hi
<jml> can someone do me a favor & run "./bin/test -cvv devscripts.ec2test.tests.test_remote" in stable? (r11681 ideally)
<jml> it won't take very long
<lifeless> gary_poster: hi
<lifeless> uhm bugs
<gary_poster> hi :-)
<lifeless> pageid + hard_timeout feature flags needs qa
<lifeless> needs a losa though, to set the flag on staging
<gary_poster> ok
<lifeless> e.g. pick a page thats timeing out, ask for a feature-rule with that pageid and (say) 40 second timeout
<lifeless> 40000 would be the figure (its ms)
<gary_poster> since you are giving instructions, does that mean you are asking for us to do QA? :-)
<lifeless> no
<gary_poster> (which is fine, just being clear)
<gary_poster> oh ok
<lifeless> I've been trying to line up the stars with a losa to do it
<gary_poster> oh ok cool
<lifeless> bug https://bugs.edge.launchpad.net/launchpad-foundations/+bug/395960
<_mup_> Bug #395960: proxying user supplied libarian files via the launchpad appserver domain has security and performance issues <librarian> <qa-ok> <Launchpad Foundations:Fix Committed by lifeless> <https://launchpad.net/bugs/395960>
<lifeless> we have the code we haven't qa'd
<lifeless> we can't till rt 41202 or something moves forward
<gary_poster> so...it sounds like we should remove the qa tag entirely
<lifeless> so it will stay fix committed till we've qa'd and flipped the feature flag to enable it.
<lifeless> gary_poster: I doubt that that will make the qataggerhappy ;)
<lifeless> this raises another subtle workflow change
<gary_poster> maybe not...qa-untestable then perhaps
<lifeless> 'deploying code' is no longer 'releasing a fix'
<lifeless> so the thing that checks 'is merged to prod-stable' is now only right most of the time ;)
<gary_poster> heh
<lifeless> (because with feature flags we may deploy, but not enable - e.g. due to external constraints, doing a shock n awe thing, etc)
<lifeless> dunno if we need to change anything, but certainly should be aware of this difference.
<gary_poster> qa-flagged?  qa-deployable?
<lifeless> dunno
<lifeless> qa does seem all about deployment
<lifeless> perhaps we should revamp all the tags
<gary_poster> sinzui, leonardr, is there a way around stuartm's concerns?  is this a registry thing, more than a foundations thing? https://bugs.edge.launchpad.net/launchpad-foundations/+bug/655565
<_mup_> Bug #655565: Immutable reference to users in API <Launchpad Foundations:New> <https://launchpad.net/bugs/655565>
<lifeless> however I don't think we need to move rapidly here - the failure mode is that whoever is driving the thing toggles it back to fix-committed.
<lifeless> and most things will be usable immediately, its a rare corner case.
<sinzui> gary_poster, There is not and it this issue is much bigger than api
<gary_poster> lifeless, ack.  I'll file a bug about the general issue and ask for discussion particularly from matsubara and Ursinha .  For now, for the release, do you agree on qa-needstesting?  Or is there some other gesture better?
<lifeless> I set it to qa-ok
<gary_poster> cool, lifeless, thanks
<bdmurray> How does one set the bug_tracker_link for a project / distro to launchpad for a test?
<lvh> Hi
<lvh> I'm running rocketfuel-setup and I've gotten this error message twice in a row now:
<lvh> ./rocketfuel-setup: line 308:  2072 Killed                  bzr branch lp:~launchpad-pqm/launchpad/devel $LP_TRUNK_NAME
<lvh> ERROR: Unable to create local copy of Rocketfuel trunk
<lvh> What gives?
<gary_poster> sinzui, ack.  so...does that mean this is registry, or...keep it in foundations because it is just a big honking pervasive problem in a way that I don't understand yet but that you don't have to explain to me now? :-)
<sinzui> gary_poster: leonardr. users can change their launchpad id while there are emails with links to the old id. team admin see this in pending members requests. Users also complain when someone removes a PPA so to allow a name change
<lifeless> lvh: check dmesg, you may be getting OOM killed
<lvh> lifeless: Indeed I am! Thanks.
<lvh> lifeless: I was running this on a very limited VM
<lvh> how much memory do I need?
<sinzui> gary_poster, we decided not to expose user db ids, and we are not about to expose this person as long as our engineers reports bugs about the items that wrongly expose ids
<lifeless> lvh: a couple of GB is ideal
<leonardr> bdmurray: can you find a project that already uses launchpad and use its value for .bug_tracker to set .bug_tracker on the object you're testing?
<sinzui> gary_poster, There is a small piece of irony in this bug. Version one of SSO had immutable openid identifiers. Users hated it.
<bdmurray> leonardr: that'd be smart but the couple of projects I've checked have null values for it
<sinzui> gary_poster, do you have a few minutes to talk about identity on mumble? I think we can come to common understanding that may help us find a sane path through this
<gary_poster> sinzui, I'm happy to give it a try, sure.
<gary_poster> (lvh, I use 2 MB.  I think I've had success with 1.5 and maybe even 1 in the past.)
<leonardr> bdmurray: judging from the apidoc, "null" might actually mean "launchpad"
<leonardr> since there's no option for a bug_tracker with a bug_tracker_type of "launchpad"
<bdmurray> leonardr: hmm, okay.  maybe I just want to enable bug tracking for the item then
<mars> lifeless, fyi, I found out what was wrong with the tests in my feature flags helper branch: LaunchpadBrowserRequest has not .features attribute yet
<mars> LaunchpadFunctionalLayer doctests don't use LaunchpadTestRequest, which does have the attribute
<lifeless> popping out to the hospital, back in a few hours
<lifeless> mars: its set by a request started hook
<lifeless> I think.
<mars> lifeless, thanks, I'll take a look for it
<jam> mwhudson: hi michael. Thanks for the review. I think I've done the changes you requested, and I responded to your email.
<mwhudson> jam: ok, cool
<jam> Is there anything else I can do for you in the next hr or so before the end of my day?
<mwhudson> jam: hope i didn't come across as too picky
<jam> mwhudson: it was nice to get actual feedback vs silence
<mwhudson> yes, i can see that :)
<jam> there are lots of decisions I had to just sort of come up with
<jam> so having someone go over it closely is nice
<jam> mwhudson: do you know how "lp-production-configs" is set up?
<mwhudson> jam: um in what way?
<jam> I'm supposed to propose a patch to start rolling this out on staging, but I'd like to set the flag in the right file
<jam> I can set "staging-lazr.conf" staging/* lpnet* or ?
<jam> All I can surmise is that it is somehow overlay configurations
<jam> but I don't really know which ones are active on which servers.
<mwhudson> jam: you appear to be subscribed to https://code.edge.launchpad.net/~launchpad-pqm/lp-production-configs/trunk
<mwhudson> ah right
<mwhudson> yeah, just edit staging-lazr.conf
<mwhudson> jam: if you look in staging/launchpad-lazr.conf, you'll see that the first line is "extends: ../staging-lazr.conf"
 * mwhudson hopes; going on memory here
<jam> sure, but why would one edit "staging-lazr.conf" vs "staging/launchpad-lazr.conf"
<jam> I assume one is not always active, etc.
<jam> anyway
<jam> staging-lazr.conf has a clear [codehosting] section already
<jam> so I'll put it in there.
<mwhudson> jam: certainly at some point there was more than one staging config
<mwhudson> there was staging and bazaar-staging and ... that all extending staging-lazr.conf
<mwhudson> this might not be true any more
<jam> mwhudson: before I push my patch back to lp, I assume that any branches named "*/lp-production-configs/*" is still private?
<jam> I don't want to accidentally publish the secrets in trying to get a merge proposal put up
<mwhudson> jam: https://code.edge.launchpad.net/lp-production-configs says "
<mwhudson> New branches you create for Launchpad Production Configs are private initially." for me
<jam> says "public" for me :(
<jam> I'm glad I asked
<jam> any idea how to change that?
<mwhudson> jam: by saying "losa ping" i guess
<mwhudson> i can't see privacy policies
<jam> my dearest losa, it seems I have finally been given access to the all-important lp:lp-production-configs and want to submit a merge proposal against it, however it would seem that by default my submission would be public, and would leak all of the secretive goodness out into the world
<jam> could you help me prevent that?
<jam> Sorry I have to go for now, it seems my son fell in the playground and isn't feeling well.
<jam> I'll try to respond to request later on tonight.
<mwhudson> jam: sure, i'll reply to your reply in the next couple of hours
<mbarnett> jam: if you need a private branch, feel free to create it and an i privatize it for you.
<mwhudson> jam: hope your son is ok
<wallyworld> abentley: skype?
<jml> can someone do me a favor & run "./bin/test -cvv devscripts.ec2test.tests.test_remote" in stable? (r11681 ideally)
<mwhudson> jml: ok
<mwhudson> jml: i expect i'll have to run 'make' in stable so it'll be a while :-)
<jml> mwhudson: 'make compile'
<mwhudson> yeah true
<jml> which still takes too long
<mwhudson> jml:   Ran 68 tests with 7 failures and 0 errors in 21.151 seconds.
<abentley> wallyworld: https://pqm.launchpad.net/
<poolie> jam, gl
<poolie> hello jml, abentley, mwhudson, wallyworld
<abentley> poolie: hi
<wallyworld> poolie: guten morgen
<jml> mwhudson: thanks.
<mwhudson> jml: btw zope.testing aaaaaaaaaaaaaaaaaaaa
<jml> mwhudson: that's sad news. it means that test failures have made their way into stable :(
 * mwhudson just got to that thread in his email
<jml> let me know if you have the motivation to actually fix it. :)
<mars> mwhudson, 7 failures?
<mwhudson> jml: heck no
<mwhudson> (sorry)
<mars> jml, did you try my patch for the issue?
<jml> mars: sorry, I thought you ran the whole suite
<mars> no, just the test that prod_lp said was failing
<mars> that is why this is even weirder now
<jml> well, I've just hit the big red button.
<jml> lifeless: ^^
<jml> mwhudson: I'd patch upstream if it weren't for the z.testrunner move
<jml> mwhudson: as it is I'd have to upgrade Launchpad to use z.testrunner, which is simply more hacking than I have time for right now.
<jml> anyway, I have to sleep. Sorry to raise havoc and run.
<mwhudson> jml: we can just put a patch in the zope.testing we use?
<jml> mwhudson: yeah, that's probably the simplest thing. we then have to chose whether to do it right or not.
<lvh> Hm. Is it normal that make schema occasionally takes pretty long?
<jml> mwhudson: in any case, I'm pretty sure I've left enough info in the email that anyone on the Launchpad team could do it.
<mwhudson> jml: thank heavens i'm no longer on the launchpad team!
<mars> lvh, a few minutes at least, and it takes even longer if you stare at the console :)
<mwhudson> >:)
<jml> mwhudson: bah.
<jml> mwhudson: you'll be back.
<lvh> mars: Ah, okay
<lvh> mars: it's in a VM so I'm going to inflate that number by a factor of ten
<EdwinGrubbs> jml:  which seven tests are failing in stable?
<jam> mwhudson, poolie: my son seems ok, just a little stunned and unusually quiet (aka, not running around at full speed :). Anyway, thanks for your help in getting this pushed forward, I'll try to be on later tonight to keep some momentum up
<jml> EdwinGrubbs: the ones mentioned in my email
<poolie> thanks
<mars> EdwinGrubbs, ./bin/test -cvv devscripts.ec2test.tests.test_remote
<poolie> i'll file the rt for you
<jml> EdwinGrubbs: but maybe there's more
<jml> EdwinGrubbs: I mean, who knows?
<mars> jml, you may have to wait for AsiaPac or morning EU for this to get resolved - we are all close to EOD here
<mars> jml, but thank you for raising the alarm
<jml> mars: sure.
<jml> mars: it's well and truly time for me to go to bed
<mars> well, then good night jml!
<jml> mars: but one thing I'm keen on chasing up later is this: why didn't alarm bells go off when I mentioned this four or five hours ago?
<jml> good night all.
<jml> and good luck!
<mars> seven failures in devel as well
<mars> my patch fixes four of those: https://code.edge.launchpad.net/~mars/launchpad/fix-ec2test-utf-in-devel/+merge/37760
<poolie> night
<mars> EdwinGrubbs, I'm working on a patch for that problem now
<EdwinGrubbs> ok, cool
<mars> EdwinGrubbs, could you please review  https://code.edge.launchpad.net/~mars/launchpad/fix-ec2test-utf-in-devel/+merge/37799 ?
<mars> mwhudson, ^ would you be able to help?
 * mars really really has to leave
<mwhudson> mars: reviewed
<mars> mwhudson, thanks!
<mwhudson> mars: you haven't set a commit message btw
<mwhudson> (in case you were about to ec2 land)
<mars> mwhudson, I was going to run it through lp-land
<mwhudson> ok
<mars> sent!
 * mars runs
<wgrant> How is prod_lp looking?
<wgrant> Did my CP finally make it out last night?
<wgrant> tyarusso: Did you have any luck with c-i-p?
#launchpad-dev 2010-10-07
<lifeless> wgrant: will look for you in a minute
<lifeless> mars: oh, you're gone.
<lifeless> wgrant: cp has been done
<lifeless> "Julians: Fixes critical bug where arch-independent files are
<lifeless> getting erroneously superseded across series boundaries, resulting in
<lifeless> loss of data for PPA users and incorrect package status in Ubuntu
<lifeless> SteveKs: Don't copy disabled DASes to the child distroseries during
<lifeless> InitialiseDistroSeries."
<wgrant> lifeless: They're both on cocoplum and germanium?
<lifeless> dunno
<Hudson> Project devel build (95): STILL FAILING in 4 hr 1 min: https://hudson.wedontsleep.org/job/devel/95/
<wgrant> !!
<StevenK> \o/
<mwhudson> me likey the shouting
<StevenK> It can also do e-mail, a'la buildbot, I've just not configured it
<lifeless> .oO a hudson
<lifeless> StevenK: I like twitter myself
 * StevenK is ignoring twitter in the hopes it all just goes away
<lifeless> StevenK: we're organising openid
<StevenK> lifeless: For 1.378 or 'soon'?
<lifeless> soon
<wgrant> OpenID for what?
<lifeless> hudson
<lifeless> StevenK: please pick a different irc nick for our hudson
<wgrant> It doesn't support it? That's a bit odd.
<StevenK> lifeless: Suggestions?
<lifeless> lpci ?
<lifeless> 42?
<lifeless> 101010?
<lifeless> back soon
<LPCIBot> Project db-devel build (53): STILL FAILING in 15 min: https://hudson.wedontsleep.org/job/db-devel/53/
<StevenK> Oh, damn, I know why
<LPCIBot> Project db-devel build (54): STILL FAILING in 32 sec: https://hudson.wedontsleep.org/job/db-devel/54/
<wallyworld> StevenK: you are going to install the Chuck Norris plugin for the Hudson instance aren't you? :-)
<wallyworld> http://wiki.hudson-ci.org/display/HUDSON/ChuckNorris+Plugin
<StevenK> I wasn't planning on :-)
<wallyworld> but it's soooo cool
<wallyworld> that default butler guy is too boring. chuck makes it a lot more interesting
<lifeless> brucie!
 * StevenK blinks at subunit
<lifeless> StevenK: the bruce schnier plugin ftw
<StevenK> Both of which are completly pointless? :-)
<lifeless> wallyworld: you have mail
<wallyworld> lifeless: thanks
<poolie> lifeless: thanks for your post about a flags remote api; but let's put it on that bug?
<poolie> that mp is big enough already
<lifeless> poolie: sorry :)
<poolie> np :)
<poolie> it's an interesting point
<poolie> i just don't want it lost
<lifeless> poolie: perhaps you could copy it across?
<poolie> sure :)
<lifeless> thank you
<poolie> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/656031 in case you think of anything else or want to prioritize it
<_mup_> Bug #656031: want non-storm way to get feature rules <feature-flags> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/656031>
<StevenK> lifeless: Are you up for a quick testfix review for db-devel?
<wgrant> StevenK: You wouldn't happen to know the publisher code, would you?
<StevenK> wgrant: What about it?
<wgrant> StevenK: I think the way that it determines which pockets and archs to generate a Release file for is far more complicated than it needs to be.
<wgrant> And I'm hoping that someone who still exists has dealt with it before.
<wgrant> But I doubt it :(
<StevenK> I suspect Julian, maybe al-maisan
<wallyworld> lifeless: well, 5th time lucky on that sql logging branch. it's in the hands of the ec2 gods again. i've made a sacrifice on an old windows pc to appease them
<wgrant> Speaking of landing... I have a branch that ec2 has thrown a fit at three times. Could someone try it again?
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/bug-629921-packages-empty-filter/+merge/37339
<StevenK> wgrant: Doing so
<wgrant> StevenK: Thanks.
<lifeless> StevenK: url
<StevenK> lifeless: https://code.edge.launchpad.net/~stevenk/launchpad/db-fix-disabled-dases-test/+merge/37819
<lifeless> doesn't edwin have the same thing up for review?
<StevenK> His doesn't do the clean-up that mine does
<StevenK> (And I'm all for making tests shorter)
<StevenK> lifeless: And I already spoke to Edwin about it in -reviews
<lifeless> so InitialiseDistroSeries was the helper?
<StevenK> lifeless: The helper removed was _create_distroseries()
<lifeless> StevenK: does it work?
<StevenK> lifeless: Yes, I verified the test failed before fixing it, and that it passed after fixing it.
<LPCIBot> Project devel build (96): STILL FAILING in 3 hr 25 min: https://hudson.wedontsleep.org/job/devel/96/
<LPCIBot> * Launchpad Patch Queue Manager: [r=EdwinGrubbs][ui=none][bug 645702] Forward small messages to the
<LPCIBot> moderation queue.
<_mup_> Bug #645702: oops in holdMessage storing large message  <mailing-lists> <oops> <Launchpad Registry:Fix Committed by sinzui> <https://launchpad.net/bugs/645702>
<LPCIBot> * Launchpad Patch Queue Manager: [r=thumper][ui=none][bug=652626] Upgrade windmill from r1440 to r1544
<LPCIBot> to fix an issue preventing some new launchpad tests from running
<wgrant> Botwar!
<StevenK> Haha
<spm> don't encourage them!
<wgrant> What's with those utf8 failures?
<StevenK> Mmmm. The SCM messages are not so handy
<poolie> do i need to mark a bug fixreleased after qa, or just qa-ok?
<poolie> does a bot do the former?
<wgrant> poolie: Just qa-ok
<poolie> great, bug
<wgrant> Fix Released should only happen once it's on prod.
<poolie> bug 615740 is done then
<_mup_> Bug #615740: test_on_merge.py doesn't handle eintr <qa-ok> <Launchpad Foundations:Fix Committed by mbp> <https://launchpad.net/bugs/615740>
<wgrant> Parts of NMAF look far too much like are an initial hacked-together implementation that was meant to be immediately followed by a proper one.
<wgrant> Hacks hacks hacks.
<wgrant> EVERYWHERE.
<LPCIBot> Project db-devel build (55): STILL FAILING in 3 hr 53 min: https://hudson.wedontsleep.org/job/db-devel/55/
<adeuring> good morning
<mrevell> Hallo
<wgrant> Hullo.
<poolie> lp is readonly? no warning? ffs.
<wgrant> It was on identi.ca.
<wgrant> And the blog.
<wgrant> But no in-app warning that I saw.
<poolie> tweeting 17h in advance is kind of missing the point of the medium
<wgrant> Heh.
<poolie> i'll do it now
<wgrant> bigjools: Sorry I missed your ping last night; I'd left about three minutes earlier.
<bigjools> nae bother
<wgrant> I believe the CPs are done?
<bigjools> yes
<wgrant> Phew.
<bigjools> although my fixit query didn't work, I need to look at it again
<wgrant> What was the query?
<bigjools> http://pastebin.ubuntu.com/507347/
<bigjools> we cancelled it after a couple of hours :/
<wgrant> Ah, that kind of not working.
<wgrant> What if you just do the usual 2.5-minute SELECT into a temp table, then UPDATE from that?
<wgrant> Also, dateesuperseded.
<wgrant> You're not unsetting that.
<bigjools> ah true
<bigjools> yeah I'll do the extra temp table
<wgrant> You might also want to check datepublished IS NOT NULL
<wgrant> (since it's possible that it superseded a Pending pub)
<wgrant> Setting that to Published would be confusing, since it would leave a file missing from disk, but it wouldn't really break anything.
<bigjools> hmmm yes
<allenap> Staging was timing out on pillar pages yesterday, same today. Is this a known problem, or is there some nastiness in db-stable?
<bigjools> I hate our model
<wgrant> I mean, it's unlikely, but possible.
<wgrant> bigjools: Sometimes I wonder why status is explicit.
<bigjools> wgrant: to help query performance I would guess
<bigjools> I'd rather have more states
<wgrant> More!?
<bigjools> yes
<bigjools> it's a farce at the moment - superseded, but then you also have to look at dateremoved to see if it's really gone, for example
<wgrant> Was ArchiveRemovalRedesign before your time?
<wgrant> I think it might have been.
<bigjools> yes
<wgrant> Back in the good old days we had Pending, Published, PendingRemoval, Removed.
<wgrant> Then ArchiveRemovalRedesign introduced the three removal candidate states, and removed the explicit on-disk state.
<wgrant> So we are now stuck half-way between having an explicit status field, and implying status from dates.
<wgrant> I'm not sure if that was always the plan.
<wgrant> Or if it only got taken half-way.
<bigjools> exactly
<wgrant> I guess removing Pending would make it pretty much sensible.
<wgrant> Having Pending but not Removed is stupid.
<wgrant> But having Published/Superseded/Deleted/Obsolete makes sense.
<jml> hello...
<bigjools> wgrant: ok I've fixed staging, let's check it out
<wgrant> The publication which alerted me to everything is fixed.
<wgrant> That is encouraging.
<bigjools> hurray
<wgrant> That leaves no borked pubs in the primary archive?
<bigjools> I hope so
<bigjools> we're down to 20k rows that match the first query
<wgrant> ie. the first temporary table containing all supersed arch-indep pubs since 2010-08-09?
<bigjools> yes
<bigjools> shockingly the 2nd query now returns zero :)
<wgrant> I hope that's the one with dateremoved IS NULL
<bigjools> yes
<jml> anyone fixing the test failure in devel?
<jml> or the one in db-devel, for that matter.
<wgrant> StevenK had a branch for the IDS db-devel failure.
<wgrant> Not sure about devel.
<jml> wgrant: thanks.
<jml> I've forced a rebuild of both db-lp buildslaves
<jml> I guess I'll fix devel
<StevenK> jml: I have a branch for devel, I was waiting for codehosting
<StevenK> db-devel is in buildbot now
<jml> StevenK: oh cool. the test_remote failures?
<StevenK> jml: Yes
<jml> StevenK: yeah, but buildbot died :\
<jml> StevenK: sweet. may I see the merge?
<StevenK> jml: Pushing it now
<jml> StevenK: thanks.
<wgrant> Oh, so Hudson wasn't on crack? Those were real failures?
<jml> that reminds me.
<jml> time to run the full test suite against stable & db-stable to see if there's anything else to be done
<StevenK> jml: https://code.edge.launchpad.net/~stevenk/launchpad/fix-ec2test-utf8/+merge/37831
<jml> looking
<StevenK> jml: But that looks a lot bigger than what I had locally ..
<jml> yeah.
<jml> StevenK: it also looks suspiciously like mars's branch.
<StevenK> jml: Yes ...
<jml> StevenK: why is that? I thought that mars's fix had already landed...
<wgrant> Wrong target...
<StevenK> So did I
<jml> wgrant: ta
<jml> StevenK: ok. get the right diff & we'll figure out what's making this more interesting than it needs to be.
<wgrant> Hm.
<wgrant> We can now kill 2.5 off, can't we?
<wgrant> Since wildcherry no longer fails.
<wgrant> !@!!@!
<wgrant> I just had a branch fail EC2 for the fifth time.
<wgrant> Each time with a different error.
<wgrant> Grrr.
<jml> wgrant: I'm sorry.
<jml> wgrant: lifeless is going to make it better.
<wgrant> Heh.
<jml> wgrant: until then, can I retry your branch for you.
<wgrant> jml: devel's still broken, isn't it?
<jml> (also, what's the deal with this paramiko error)
<jml> wgrant: I'll run it against stable
<jml> of course
<jml> stable might be broken
<wgrant> Heh.
<wgrant> Yes...
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/bug-629921-packages-empty-filter/+merge/37339
<StevenK> jml: I'm not sure why that diff is so wrong
<wgrant> StevenK: Targetted at lp:launchpad.
<wgrant> And db-devel is missing three devel revs.
<jml> StevenK: it's targeted to db-devel rather than devel.
<StevenK> Oh, ...
<StevenK> jml: https://code.edge.launchpad.net/~stevenk/launchpad/fix-ec2test-utf8/+merge/37832
<StevenK> (That's what I get for filing MPs while on the phone to bigjools)
<jml> StevenK: uhh ok.
<jml> StevenK: do those tests pass locally for you?
<StevenK> jml: Yup
<jml> hmm.
<jml> I suspect platform variation. mars changed the 'utf-8' to 'utf8' to fix a test failure
 * jml tries
<jml> anyone getting paramiko errors? http://paste.ubuntu.com/507903/
<StevenK> Oh, are you kidding me? Now it wants utf8, rather than utf-8
<jml> yeah. let me look at the tests.
<bigjools> wgrant: I'm going to run that fixit query in an hour or so, I think it's ok unless you have any further misgivings?
<jml> I reckon we should relax the constraint somehow.
<wgrant> bigjools: pastebin pls.
<wgrant> Just to be safe.
<jml> StevenK: ok. here's what we should do.
<wgrant> "charset=utf8" is wrong, though.
<wgrant> Anything that wants that is broken.
<jml> StevenK: split body['Content-Type'] by ';', assert that the first part is 'text/plain'
<jml> StevenK: don't care about the second part.
<bigjools> wgrant: http://pastebin.ubuntu.com/507877/
 * wgrant WTFs at the differing theme between paste.ubuntu.com and pastebin.ubuntu.com
<jml> wgrant: for this, we just don't care. if people start having encoding issues with their ec2 mail then we'll worry then.
<wgrant> bigjools: +1 doit.
<StevenK> jml: MP updated
<jml> StevenK: thanks.
<jml> StevenK: have you run the tests locally?
<StevenK> Yes
<StevenK> bin/test -vv -m devscripts -t test_remote is happy
<jml> StevenK: humour an old man, run bin/test -vv devscripts
<StevenK> ... Aren't you younger than me?
<StevenK> jml: Ran 146 tests with 0 failures and 0 errors in 15.604 seconds.
<wgrant> Almost 10 tests a second? What sorcery is this?
<StevenK> ... A fast machine?
<StevenK> You should try it
<wgrant> Ah, and no DB too, I guess.
<jml> yeah. they are tests for the devscripts stuff
<StevenK> wgrant: Only unittest layer, so quick
<jml> actually the test_remote tests are quite slow because they make real branches
<StevenK> jml: Should I land that as a testfix?
<jml> StevenK: yes please
<jml> StevenK: sorry for the delay
<StevenK> jml: Do you want to stamp it?
<jml> StevenK: sure.
<jml> StevenK: done
<jml> sorry about the delay. multitasking way too much.
<StevenK> jml: Tis cool. I've tossed it at PQM
<jml> no one knows anything about that paramiko failure
<wgrant> Hm, that's an odd one.
<jml> yeah. I haven't sat down to serious debugging yet. Want to flush out my mail queue first.
<bigjools> wgrant: production fixed
<wgrant> bigjools: Looks good, thanks.
<wgrant> can you grab counts of remaining broken publications?
<bigjools> yeah already done a while ago
<deryck> Morning, all.
<LPCIBot> Project db-devel build (56): STILL FAILING in 3 hr 42 min: https://hudson.wedontsleep.org/job/db-devel/56/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][r=lifeless][ui=none][no-qa] Fix the InitialiseDistroSeries
<LPCIBot> test for disabled DASes.
<bigjools> funk6
<bigjools> err funky, even
<wgrant> jml: Did you end up sending my branch off?
<wgrant> It may have some hope..
<bigjools> wgrant: can we talk about your bug-655614-disabled-arch-indices branch
<wgrant> bigjools: I opened that 30 seconds ago and was about to ask you the same thing.
<bigjools> :)
<wgrant> It's not so much a branch as a set of a few patches that we might want bits of.
<bigjools> r9787 is confusing me
<wgrant> It's probably too revolting to land (but no worse than parts of the existing code :/)
<wgrant> Hm?
<danilos> henninge, so, in what sense does +translations on a distroseries fail?
<wgrant> bigjools: I removed a duplicate loop and added an arch.enabled guard. Previously it was looping across the archs once to publish indices, and again to request Release files.
<bigjools> wgrant: why the changes other than the .enabled check?
<bigjools> ah
<wgrant> It was that or add the check twice.
<bigjools> got it
<bigjools> is it all tested?
<wgrant> 9788 has some quick partial tests, but they're not thorough or pretty.
<bigjools> not series.getDistroArchSeries(architecture[7:]).enabled):
<bigjools> eeeurrrrghh
<wgrant> bigjools: I stole the architecture[7:] from existing code!
<bigjools> I know :(
<wgrant> I originally did .split('-')[0], but decided to go with the... preferred method.
<henninge> danilos: on <a tal:attributes="href context/menu:navigation/templates/url">
 * bigjools puts fingers in ears and goes blahblahblah
<henninge> danilos: 'templates' does not exist
<wgrant> bigjools: I have formulated a plan to remove all that stupidity, but it will require a bit of a-f disentanglement.
<wgrant> So it can't be done quickly.
<jml> wgrant: yes
<wgrant> jml: Great, thanks.
<henninge> danilos: that's when rendering DistroSeriesView.
<jml> StevenK: build failure just sent to the list
<bigjools> wgrant: if you get this branch finished I will run it on dogfood while you get it reviewed
<henninge> danilos: that's the test that's failing, actually the last "test_" method im the file.
<henninge> danilos: http://pastebin.ubuntu.com/507871/
<wgrant> bigjools: I'll write the tests properly now it's no longer 3am.
<StevenK> jml: That's lp, not lucid_lp
<wgrant> But I think it's good, so please try on DF, yes.
<jml> StevenK: I'm a little bit behind ... does lp build slave still matter?
<wgrant> wildcherry was the end of 2.5, I believe.
<wgrant> So we may be safe.
<henninge> danilos: read the backscroll on #lp-reviews for more detail ;-)
<StevenK> We're all on lucid now, I think
<bigjools> wgrant: :)
<henninge> danilos: bac has ended his day in defeat, so it is not super-urgent atm.
<danilos> henninge, fwiw, I can simply create a distribution and distroseries using factory methods and browse to +translations on it
<henninge> danilos: that's odd. How do you do that exactly?
<henninge> (I know how to use the factory methods)
<wgrant> bigjools: I'd test PPA and primary archives both with and without active publications in the disabled DASes. Check that the DAS' directories aren't recreated, and confirm that the series' Release doesn't include those arches.
<wgrant> I will return in 20.
<danilos> henninge, in 'make harness' using http://paste.ubuntu.com/507948/
<bigjools> wgrant: yup
<danilos> henninge, oh, if you know how to use factory methods, perhaps you were asking how do I browse to it? :P
<henninge> danilos: exactly ;)
<danilos> henninge, well, I open a web browser and go and look at it :)
<henninge> danilos: I was not sure which changs to the db are presereved after make harness.
<henninge> danilos: ok, simply 'make run'
<danilos> henninge, all changes that you've committed are preserved
<danilos> henninge, right, but I do that even before 'make harness': it's basically like two appservers talking to the same DB
<henninge> I see
<danilos> henninge, I've already looked at that test for bac, but now that I think about it, it's probably a missing commit somewhere
<danilos> or at least a .sync()
<henninge> danilos: we were also wondering why DistroSeriesTranslationsMenu.templates is @enabled_with_permission('launchpad.Edit') when still it is shown to everybody.
<danilos> henninge, probably because it's not useful to anybody else all that much
<danilos> henninge, i.e. perhaps shouldn't be like that
<henninge> danilos: no, I am saying that the link (and the page) is shown to unauthorized users, too.
<henninge> danilos: *although* it is @enabled_with_permission('launchpad.Edit') ...
<danilos> henninge, oh, then the decorator needs to go
<henninge> danilos: you mean it has no function any more?
<henninge> danilos: btw, a commit makes the test pass.
<henninge> ;-)
<henninge> danilos: where would the sync() go?
<danilos> henninge, where the commit is, on the object that needs syncing (sometimes a sync() is not enough, but you could try self.distroseries.sync())
<danilos> henninge, that also means that all the other tests (for non-LAUNCHPAD usage values) pass by accident :)
<bigjools> sync() is useless now I thought?
<jml> indeed
<bigjools> store.flush() FTW
<henninge> good to know ;)
<henninge> danilos: A commit does *not* help. btw. I had forgotten that I had commented out the failing parts of the template ...
<henninge> :(
<danilos> bigjools, ah, right, thanks :)
<danilos> henninge, sorry to hear that
<danilos> henninge, does removing the permissions decorator on the property help?
<henninge> nope
<henninge> danilos: if I comment out the context/menu:navigation/templates line in the tmplate
<henninge> danilos: it fails on context/menu:navigation/langpacks
<henninge> ...
<henninge> danilos: My suspicion is it has the wrong menu. But why and how do I prove that?
<danilos> henninge, right, we should just get rid of the context/menu:navigation use anyway, but... that doesn't help with this branch
<danilos> henninge, hum, "wrong" menu is a potential issue if the entire page is included into distribution:+translations, but I am pretty sure it isn't
<danilos> henninge, fwiw, I'd suggest trying setting a breakpoint and going through the test with pdb (or trying it manually in 'make harness')
<jml> bigjools: so you've got the buildd-slavescanner work now?
<jml> bigjools: I can remove it from my todo?
<bigjools> jml: I'll sort it, yes
<jml> bigjools: ta
<bigjools> jml: btw have you been playing Uncharted2 then? :-)
 * wgrant returns.
<jml> bigjools: not recently. and actually it was mostly watching two of my flatmates play it
<bigjools> F1 2010 is keeping me up way too late
<wgrant> bigjools: Hm, the Release update didn't work?
<wgrant> Or have you just deleted but not republished yet?
<bigjools> wgrant: still trying to get the f***ing publisher to do something
<wgrant> Heh.
<wgrant> â¥ dogfood
<bigjools> I've got a pending publication but p-d ignores it
<wgrant> Pending publications, or pending uploads?
<wgrant> It's easy to forget p-a...
<bigjools> already ran p-a
<bigjools> there's a pending source and binary now
<wgrant> So even phase A doesn't see them?
<bigjools> nup
<bigjools> this is w/o your patch too
<wgrant> Yeah. I didn't touch phase A.
<wgrant> Come on DF... show me your queue.
<wgrant> Uh, is DF's appserver hung?
<wgrant> There is little response.
<bigjools> +queue?
<wgrant> Ah, there we go.
 * bigjools restarts DF
<wgrant> Yeah, that.
<wgrant> It's alive now.
<bigjools> +queue is slow slow slow
<wgrant> It was going for well over a minute.
<wgrant> Which should have timed out.
<wgrant> But it's rendered now.
<bigjools> bizarrely your branch conflicts in lib/canonical/launchpad/browser/launchpad.py
<wgrant> Mine is based on production-devel.
<wgrant> So that's possible, I suppose.
<bigjools> don't do that
<bigjools> not yet anyway
<bigjools> where did you branch from?
<wgrant> A production-devel branch. I'll rebase on devel. Gimme a sec.
<bigjools> ok now I'm getting hacked off, why is it not publishing
<wgrant> Well, there is no appserver running. Is there a librarian? p-d needs one...
<wgrant> bigjools: That branch is now based on devel.
<bigjools> I have to run a db patch on DF
<wgrant> Yay.
<bigjools> which is always painful
<wgrant> It's not the branchrevision one, I hope.
<bigjools> HWSubmissionDevice getting a new index - ffs
 * bigjools kills it
<bigjools> and 2 others
<jml> wgrant: btw, I've seen a thread isolation error in the layers tests in your branch.
<wgrant> .... #6.
<jml> wgrant: it might not fail
<jml> layer tests are hella weird
<wgrant> Hm, Ok.
<bigjools> wgrant: of course it would have helped to have publish=True for the main archive
<wgrant> bigjools: Uh, DAS.enabled is in production-devel but not devel?
<wgrant> bigjools: Haha.
<bigjools> wgrant: yes, use db-devel
<wgrant> Well, that's a bit special.
<bigjools> it's because of the late branch for the last rollout
<bigjools> the DB is patched in prod without a patch file in devel
<wgrant> Ah, I thought that did make it into devel anyway.
<wgrant> I recall the late patch.
<bigjools> and now I am going to get some food while the publisher runs
<bigjools> we could do with using your domination speedup thing
<bigjools> back in 30 mins
<wgrant> I'm going to see if I can remove the temp table first.
<wgrant> The whole going behind Storm's back thing makes me a bit wary of duplicating it.
<jcsackett> henninge, danilos: hi.
<danilos> jcsackett, hi
<jcsackett> danilos: i have two more bugs in translations (filed by registry) that i would like input on, if you have time.
<jcsackett> bug 652296 and bug 652301
<_mup_> Bug #652296: distribution translations front page contradicts itself for owner <bridging-the-gap> <confusing-ui> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/652296>
<_mup_> Bug #652301: non-soyuz distro has contradictory involvement portlet <bridging-the-gap> <confusing-ui> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/652301>
<jcsackett> i think the one about distribution contradicting itself falls into the category of things you guys are working on as part of other bugs, and the bug can be discarded. danilos, do you agree?
<rockstar> abentley, hi
<abentley> rockstar: hi.
<rockstar> abentley, so, in my merge-queues-db branch, I deleted the tests because I needed the tests to run.  thumper and I had come to the conclusion that making the db patch as small as possible was a good idea.  The actual model code gets deleted in the next pipe.
<abentley> rockstar: does the model code work?
<rockstar> abentley, not without the tables underneath it, no.
<rockstar> abentley, but there are imports and such that I didn't want to also have to go chase just to get the db-patch in.
<rockstar> (although I have some failing tests which may force me to do that)
<abentley> rockstar: so you're reducing the test coverage in order to allow some code to be broken?
<rockstar> abentley, no, just trying to keep the diff manageable.
<abentley> rockstar: you're reducing the test coverage in order to allow some code to be broken so that the diff is manageable?
<rockstar> abentley, yes.  Code that isn't used (and never was) is now broken in this branch.  It is fixed in the next one.
<abentley> rockstar: while there may be some exceptions, I don't think this is a good general policy.
<abentley> rockstar: would it have been possible to remove the model code and tests first, then do the db patch?
<rockstar> abentley, no, not at all.  I am contending though, that this is probably an okay exception.  As far as I'm concerned, the current merge queue code is nothing but technical debt.
<rockstar> abentley, I guess, but in this next branch, I'm replacing the model code with the correct code.
<rockstar> abentley, this is really not ideal.  Instead of creating new functionality, I get to remove old, half-baked functionality first.  By today's policy, the current branch merge queue code never would have been landed.
<abentley> rockstar: all of this doesn't actually matter because you won't be landing this branch by itself, right?
<rockstar> abentley, well, it's getting to the point that I need to land it by itself, because PQM is closing.
<rockstar> abentley, but there are still broken tests, so I may need to delete it anyway.
<abentley> rockstar: I understand that you're frustrated.  I'm sorry to be sticky about this.
<rockstar> abentley, no, I'm not frustrated, just explaining.
<rockstar> abentley, or rather, I'm frustrated that we have old stale code just festering in Launchpad.
<abentley> rockstar: I'll approve this, but please try to avoid reducing test coverage in the future.
<rockstar> abentley, yeah, that's the last thing I want to do.
 * rockstar likes tests.
<rockstar> abentley, it's still quite possible that I'll need to delete the models in this pipe, but I can't get the Librarian to run anymore to find out.
<allenap> adeuring, gmb: Do either of you have time to review a branch? It's mostly a search-n-replace job. https://code.edge.launchpad.net/~allenap/launchpad/subs-for-bugtask-bug-656194/+merge/37835
<adeuring> allenap: I'll do it
<allenap> adeuring: Thanks.
<wgrant> bigjools: The publisher is *still* not done?
<wgrant> Or did I explode it?
<bigjools> wgrant: dogfood is slow at publishing
<wgrant> .... apparently.
<bigjools> the "Calculating binary filelist." part is REALLY slow
<wgrant> Anyway, I have some nonawful tests now.
<bigjools> I expect it'll be another hour yet
 * rockstar reboots
<wgrant> bigjools: I wonder how it can be so bad. Maverick should have fewer than 10000 binary lines.
<wgrant> Er, 100000.
<wgrant> Still not many.
<wgrant> bigjools: https://code.edge.launchpad.net/~wgrant/launchpad/bug-655614-disabled-arch-indices/+merge/37849 <-- besides the greasy stolen hack, does it look OK?
<wgrant> I will throw it at a reviewer if you have no immediate objection.
<wgrant> The tests now verify that the indices aren't created by either publisher, that the arch directory isn't created, and that it's not in the serise' Release.
<wgrant> That should be everything.
<bigjools> wgrant: can you put the enabled flag as a param to makeDistroArchSeries
<wgrant> bigjools: Good point.
<wgrant> jml: Failure #6!
<wgrant> Awesome.
<wgrant> This is a record for me.
<jml> wgrant: yeah.  :(
<jml> wgrant: and that was against stable too.
<wgrant> â¦..
<wgrant> Well then.
<jml> wgrant: which is no longer a known good branch :(
<jml> wgrant: my run against stable didn't seem to work in the way I expected.
<jml> as in, my test run of just stable nothing merged in.
<wgrant> How do you mean?
<jml> well, it only ran devel
<wgrant> Hm, convenient.
<jml> so, going to try again
<jml> also, running those failing tests on my stable branch
<jml> ffs.
<jml> make schema :(
<jcsackett> danilos: any thoughts regarding those bugs?
<flacoste> mars: what's the status of the failing tests on the stable branch?
<mars> flacoste, jml would know.  StevenK had a fix that was in buildbot
<danilos> jcsackett, sorry, it usually useful to ping me every time or I can easily miss out on a followup to a discussion (IRC is famous for having multiple conversations over multiple channels all happening at the same time :)
<flacoste> mars: do we know why that happened?
<danilos> jcsackett, I am taking a look now
<tyarusso> wgrant: Maybe.  I appear to have gotten both c-i-p and lp to run without errors.  Still no idea how to actually connect them to each other such that a "Register" link appears on LP.
<jcsackett> danilos: ah dig, sorry for not pinging you on all that. :-)
<flacoste> mars: and the fact that it landed in buildbot doesn't mean it's fixed, given that the failures weren't noticed by the builder in the first place
<mars> flacoste, the accute symptom appears to be a difference between how platforms handle charset encodings: some say the mail was written as 'utf8', other say 'utf-8', and others say 'quote-printable'
* flacoste changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is open for business | firefighting: stable has test failures | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> flacoste, mars: see the mail I just sent
<wgrant> tyarusso: Grep around in the LP configs for 'testopenid.launchpad.dev', and change it to your c-i-p domain.
<jml> mars: it's deeper than that
<wgrant> tyarusso: But I am asleep.
<mars> jml, yep, hence the 'accute symptom' - stating that "This man died from blood loss!" while ignoring the stab wounds
<danilos> jcsackett, I agree about bug 652296, we can probably wontfix it
<_mup_> Bug #652296: distribution translations front page contradicts itself for owner <bridging-the-gap> <confusing-ui> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/652296>
<jml> mars: uhh, to stretch the analogy, blood loss, internal bruising and liver collapse
<jml> mars: there are two other failures that have nothing to do with encoding
<jcsackett> danilos: i cannot express how much i wanted to hear that. :-)
<mars> jml, great :(
<jcsackett> danilos: the other one is less a "can we wontfix this" and more a "there are plans for dealing with distros like this in the future, what do you want to do now?" sort of thing.
<jcsackett> which is a far more complicated question.
<flacoste> mars,jml: can't these failures be not noticed because of the subunit format failure we saw earlier?
<jml> flacoste: that would be my best guess. I haven't done *any* analysis though. my first step was trying to measure the extent of the problem...
<flacoste> mars: given that these failures are reproducible locally, can we do a binary search to find the revision at which point they started failing and went unnoticed in buildbot?
<jml> flacoste: still waiting on db-stable results
<danilos> jcsackett, yeah, the other one (bug 652301) is useful in itself and should probably stay... I'd say leave it open and we'll worry about it (i.e. I don't think it's got anything to do with the bridging the gap theme)
<_mup_> Bug #652301: non-soyuz distro has contradictory involvement portlet <bridging-the-gap> <confusing-ui> <Launchpad Translations:Triaged> <https://launchpad.net/bugs/652301>
<mars> flacoste, yes, that would work
<jcsackett> danilos: okay, thanks. i'll remove the briding-the-gap tag and take it out of registry's backlog.
<sinzui> \o/
<flacoste> mars: can you take care of that analysis, pretty please?
<danilos> jcsackett, thanks
<mars> flacoste, sure :)
<flacoste> thanks mars
<EdwinGrubbs> rockstar: ping
<rockstar> EdwinGrubbs, hi.
<EdwinGrubbs> rockstar: On the off chance that you aren't busy, can you look at the launchpad-code bugs that are still in qa-needstesting?
<rockstar> EdwinGrubbs, I can.  I'm about to head out for a second.
<LPCIBot> Project devel build (97): STILL FAILING in 3 hr 52 min: https://hudson.wedontsleep.org/job/devel/97/
 * rockstar afks
<gmb> jml: I cannot get xx-person-subscriptions to fail locally in stable. Neither can jcsackett, from what I understand.
<gmb> So why it's failing in buildbot I don't know.
<jml> gmb: interesting.
<jml> gmb: it fails for me.
<jcsackett> gmb: that's right. i run xx-person-subscriptions on ec2, it fails as you demonstrated. i run it locally, it works (as the code currently is).
<stub> xx-person-subscriptions.txt looks like a data ordering issue - the sort that occasionally pops up with sampledata and PG updates.
 * gmb `make schema`s
 * jcsackett groans and also makes schema.
<stub> I've just emailed, but summary is I think +subscriptions displays whichever bugtask it finds 'first', but doesn't tell the database what 'first' means and we get an arbitrary bugtask.
<gmb> ARGH.
<jml> actually, a thing we should do is look at the buildbot logs, see if the failures are in the output but not detected...
<gmb> stub: So it doesn't use bug.default_bugtask.
<gmb> And I'd just assumed it did.
<gmb> Arseholes.
 * stub bets switching to default_bugtask causes a performance problem ;)
<gmb> bdmurray: So, re: your email is it actually an ordering issue as you suggest or is it the issue that stub suggested above.
<gmb> stub: Shouldn't any more.
<gmb> I fixed that.
<gmb>     def default_bugtask(self):
<gmb>         """See `IBug`."""
<gmb>         return Store.of(self).find(
<gmb>             BugTask, bug=self).order_by(BugTask.id).first()
<stub> This is just my guess btw. I haven't looked at the code.
<bdmurray> gmb: well the bugtasks could have different date_last_updated fields but I don't think the sample data should be changing ... right?
<gmb> bdmurray: Well, no, but if we're just describing the subscriptions a user has why not just use default_bug_task?
<bdmurray> gmb: that seems like a fine solution to me
<gmb> bdmurray: Cool. Are you working on it now or should I put a branch together?
<bdmurray> gmb: No, I'm not working on it now but I could be.
<gmb> bdmurray: I'll take care of it since I'm in the code already.
<bdmurray> gmb: okay, thanks!
<gmb> np
<stub> sorting by date_last_updated can be dangerous, as if you modify two bugtasks in the same transaction you get identical timestamps and the ordering is undetermined.
<bigjools> yay postgres vuln
<dobey> is lp taking a horrendously long time to rescan branches for anyone else?
<maxb> The branch scanner was broken yesterday. Has it been fixed?
<mars> abentley or rockstar, ^ ?
<mars> if there is a problem, should it be listed in the channel topic?
<abentley> maxb: I didn't hear there was a problem.
<mars> dobey was asking too
<maxb> The branch scanner was throwing SQL permission errors.
<abentley> maxb: any oops ids?
 * maxb hunts irclogs
<mars> flacoste, jml, you are going to love this: revision 11634 is the culprit
<jml> A wise man once said, All you need is love.
<flacoste> mars: and what did this revision introduced?
<mars> "Move read_transaction & write_transaction to lp.services."
 * jml looks at the revision
<flacoste> really!?!
<mars> flacoste, jml, just to be clear, that is devel r11634
<abentley> mars: (and also stable r11634)
<maxb> the branch scanner issue was http://launchpadlibrarian.net/57189808/jR3X1uFDWYhXPet14K6N1hWzrkG.txt
<flacoste> mars: that's the revision where the tests started failing? does it give also the explanation why buildbot didn't spot the failures?
<jml> mars: test_remote passes on r11634 for me
<mars> fails here
<jml> interesting
<maxb> http://irclogs.ubuntu.com/2010/10/06/%23launchpad.html#t22:27
<jml> are we looking at a config issue or a timing issue or what
<mars> jml, well, I figured it must have passed for you - how else would you have done your TDD loop?
<jml> mars: to be clear, test_remote does not work for me with tip of stable
<jml> mars: and that revision doesn't do anything with devscripts
<mars> ... jml, does that include StevenK's fix?
<abentley> maxb: ack
<jml> mars: no, not including StevenK's fix
<mars> jml, I did this (sandboxed devel setup): $ bzr branch -r 11634 devel test && cd test && bzr checkout && utilities/link-external-sourcecode ../devel && make && bin/test -cv devscripts.ec2test.tests.test_remote
<mars> jml, are you running Maverick or Lucid?
<abentley> maxb: AFAICT, the scanner is running normally on most branches.  It appears to be a problem for this particular branch.
<jml> mars: Maverick
<mars> same
<jml> hmm.
<mars> flacoste, does it fail for you?
<mars> a larger sample would really help here
<mars> I'll run it on my Lucid laptop
<jml> mars: I did bzr revert -r 11634... I'll try a fresh branch just in case
<allenap> bdmurray: Btw, I also tried to investigate bug 654585, but was prevented by staging timing-out all the time.
<_mup_> Bug #654585: Line break stripped from bug reported acknowledgement <Launchpad Bugs:Incomplete> <https://launchpad.net/bugs/654585>
<jml> mars: yeah. fails for me.
<jml> mars: trying again, and then with previous revision
<abentley> maxb: this is bug 634451 and a fix was applied to production around 2010-09-12.  Apparently, it didn't stick.
<_mup_> Bug #634451: launchpad code rescans failing <branch-scanner> <oops> <qa-ok> <Launchpad Bazaar Integration:Fix Released by thumper> <https://launchpad.net/bugs/634451>
<flacoste> mars: what's the test i should run?
<mars> flacoste, bin/test -cv devscripts.ec2test.tests.test_remote
<bdmurray> allenap: you must have some ubuntu bug you could report ;-)
<allenap> bdmurray: No, it's perfect :)
<flacoste> mars: i cannot make that revision at all :-/
<flacoste> mars: compile_templates fails
<flacoste> ImportError: cannot import name SAFE_INSTRUCTIONS
<mars> flacoste, on Lucid - strange.  Have you run 'apt-get update && apt-get install' recently?
<mars> flacoste, and I assume you ran 'rocketfuel-get' to ensure sourcecode is up to date?
<mars> flacoste, and this is in a fresh branch?
<flacoste> it's a fresh branch
<flacoste> i did update this week
<flacoste> but i didn't run rocketfuel-get
<flacoste> only updated the download cache
<mars> ok, it is building on Lucid here
<mars> but is is not finished yet
<mars> flacoste, builds for me on Lucid
<mars> and the tests fail
<jml> mars: I can confirm tat revision introduces the bug
<jml> that*
<mars> ok, so we still have a "wtf" moment here though
<mars> :)
<mars> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/11634
<dobey> hrmm
<jml> mars: have you tried with the other tests?
<mars> jml, no, I shall do so - they involved database stuff too?
<jml> mars: umm... the xx-person-whatever thing?
<flacoste> mars: still fails
<mars> flacoste, you may want to talk to bigjools about using the vm builder to get a Maverick instance then
<jml> mars: http://paste.ubuntu.com/508043/
<flacoste> mars: so i cannot that revision
 * flacoste needs to lunch
<mars> ok
 * jml meeting
<abentley> maxb: I am trying to get an answer on whether the permissions have been fixed, but it's taking a while.
<lvh> Hi :-)
<lvh> What's the right place to make or discuss feature requests for lp?
<lvh> I was just thinking it would be really awesome if merge requests had diffs that looked like bzr qdiff (side by side). A bit like rietveld.
<mars> lvh, best file a bug with the request, or ask about the feature on the launchpad-dev mailing list.  For MPs specifically, you would likely end up talking to abentley or rockstar.
<benji> I'm upgrading to Maverick and my LP tests are now erroring out with "ImportError: No module named mailman.monkeypatches.defaults"; anyone seen this before?
<benji> nope, I've not done anything with that yet
<benji> I just skimmed some KWallet stuff and it seems to be about the same design/complexity as the Gnome Keyring.
<benji> oops, wrong chan
<mars> benji, I have not checked - have you run rocketfuel-get, merged devel in the branch you are working on, and run 'make clean && make'?
<benji> mars: nope; I'll do that.
<benji> (well, I was running the tests on devel, but the make clean I haven't tried)
<mars> bac or sinzui, ping, you both rebuilt your systems for maverick, correct?
<sinzui> no, I upgraded
<mars> ok, it was Brad then who did a rebuild
<benji> mars: the make clean helped, but now I get a PG error; I tried this https://pastebin.canonical.com/38344/ but got the error you see there
<dobey> mars, maxb, abentley: did you guys figure out anything with the branch rescans yet?
<abentley> dobey: What I know so far is we already had this problem, it was fixed on production, and the database upgrade seems to have lost the fix.
<abentley> dobey: so I am trying to get it re-applied.
<abentley> dobey: it was bug 634451
<_mup_> Bug #634451: launchpad code rescans failing <branch-scanner> <oops> <qa-ok> <Launchpad Bazaar Integration:Fix Released by thumper> <https://launchpad.net/bugs/634451>
<dobey> abentley: ok, thanks.
<mars> benji-lunch, looking
<mars> benji-lunch, you need to open Synaptic or Update Manager and re-enable the launchpad PPA software source
<mars> benji-lunch, the upgrade disables all third-party repos
<mars> benji-lunch, then, after that, 'sudo apt-get update && sudo apt-get install'
<mars> gary_poster, flacoste, how did the Python 2.6 upgrade go?  Can we kill the Hardy builders yet? :)
<mars> err, database upgrade
<gary_poster> mars, it went well; see my brief email to the list for remaining step.  yes, we can kill the Hardy builders, which is part of what I have in my RT for the last step.
<mars> \o/
<dobey> benji-lunch, mars: i think he meant s/install/upgrade/ for that second one. install without args won't do much :)
<mars> thanks gary_poster.  It will be nice to see those mails stop
<gary_poster> agreed
<mars> dobey, I thought 'apt-get install' offered to install new versions of all of the upgradable packages?
<mars> and that 'upgrade' did the base system
<maxb> No, that's not how it works
<dobey> upgrade installs new versions
<maxb> AFAIK apt has no "just the base system" thingy at all
<dobey> indeed
<mars> so what was the apt command to upgrade Debian (or Ubuntu) to the next major version?
<mars> do you have to explicitly tell it to do so, or does it just happen when the new version is released?
<maxb> There is no single apt command for that
<maxb> Ubuntu: Use the release-upgrader tool.
<maxb> Debian: Follow the full instructions in the release notes
<mars> Funny, I have been using Linux for more than 12 years, and never knew that
<flacoste> mars: what was the name of a devscripts failing test?
<mars> flacoste, test_remote
<flacoste> mars: more specifically, there are a bunch of test_remote
<flacoste> mars: i'm trying to assess if buildbot ran the failing tests successfully, it appears so
<flacoste> mars: https://lpbuildbot.canonical.com/builders/lucid_lp/builds/195/steps/shell_7/logs/stdio
<mars> flacoste, bin/test -cv devscripts.ec2test.tests.test_remote
<flacoste> mars: i can't reproduce
<mars> hmmm
<flacoste> mars: so i'm looking for something like a pastebin of the failing tests output
<mars> that is not necessarily a bad thing
<mars> ok
<flacoste> mars: i can't reproduce because my env is not working, not because i don't have the error :-)
<mars> oh :p
<mars> oh, to have a pastebin grep
<mars> flacoste, http://pastebin.ubuntu.com/508177/
<flacoste> mars: thanks, these tests pass successfully on build 195
<flacoste> so i think, this is an environmental thing
<flacoste> diff between dev env and the builder env
<mars> I can't think of how to debug that though
<mars> would have to dig into the code to see where those variables are being generated
<mars> and what the heck is with the revno that caused everything to break?  It does not look related
<flacoste> not sure how much we should chase this up at this stage
<flacoste> the utf-8, utf8 things might be due to a package update
<flacoste> that isn't in the builder yet
<flacoste> iow, these were all fragile tests
<flacoste> are all tests passing in stable locally and buildbot now?
<flacoste> as long as we have deltas between the dev / test / production environment, we'll suffer from fragile / env tests
<lifeless> environment failures sure
<lifeless> but tests should be isolated from the environment, OR, we should ask all devs to develop in $prod-target-vms
<jml> we fixed the utf-8 / utf8 things
<jml> there's no easy way to learn that it's env dependent
<LPCIBot> Project devel build (98): STILL FAILING in 3 hr 50 min: https://hudson.wedontsleep.org/job/devel/98/
<jcsackett> EdwinGrubbs: what time is PQM closing today? can't seem to find the email...
<EdwinGrubbs> jcsackett: 22:00 UTC, so in 3 hours
<jcsackett> thanks.
<LPCIBot> Project db-devel build (57): STILL FAILING in 4 hr 8 min: https://hudson.wedontsleep.org/job/db-devel/57/
<lifeless> off to a doctors visit, back in a few hours
<mwhudson> jml: hey
<mwhudson> jml: have you looked at the lp-service branch at all?
<rockstar> sinzui, hi.
<sinzui> hi rockstar
<rockstar> sinzui, I wonder if you might be able to help me sort out some things with the account merger, or point me to someone who might.
<sinzui> rockstar, does this related to login or passwords
<rockstar> sinzui, it relates to how the table list gets generated for finding references to Person.id
<rockstar> sinzui, it's somehow finding a table I dropped in this branch.
<sinzui> interesting
<sinzui> I do not know, but I want to know so you have my attention
<sinzui> which table
<rockstar> sinzui, branchmergerobot
<sinzui> and you get merge errors because the master rule believes there is a condition we do not have a rule for?
<rockstar> sinzui, I WAS getting errors for the table I created that referenced it.  Now I'm getting errors because it's trying to update a table that no longer exists.
<sinzui> I wonder is make schema generates a no-no list for merge
<rockstar> canonical.database.postgresql.listReferences seems to be returning branchmergerobot, which this branch drops.
 * sinzui starts reading code
<rockstar> sinzui, lp.registry.model.person:4004
<rockstar> sinzui, somehow, in the references var, branchmergerobot is showing up.  I have no idea why, since the table is deleted.
<sinzui> rockstar, I see 4004 as the the line that changes the user name
<rockstar> sinzui, oh wait! It's not deleted, it's just moved to to_drop (since dropping it would break replication)
<sinzui> oh that sucks
<rockstar> Oh yeah, I had some local changes, so the lines wouldn't match up.
<sinzui> listUniques or Handle all simple cases>
<sinzui> ?
<mwhudson> ow
<mwhudson> is the fkey constraint still on the table when it's in to_drop?
<sinzui> rockstar, I think listReferences needs to verify the table is not going to be dropped, or merge needs to filter that list by checking some table/column really needs updating
<rockstar> mwhudson, no, the column that references it is also dropped.
<mwhudson> oh, that's something at least
<rockstar> sinzui, handle all simple cases.
<jcsackett> can someone tell me why factory.makeBranch sets url to None if the branchtype is IMPORTED?
<sinzui> rockstar, this is really old code. I think though that replication is the real issue. This is probably the first we have ever dropped a person column since replication was added
<rockstar> jcsackett, because an import branch has a code import that points to where it imported the branch from.
<rockstar> jcsackett, er, it has an ICodeImport...
<rockstar> sinzui, yeah, so I'm not entirely sure how to proceed.  I could do a real hack job and just add branchmergerobot to the skip list.
<rockstar> sinzui, replication is the real issue.
<sinzui> rockstar, I try filtering the out the tuple  and leave an XXX to remove the hack after the DB update
<rockstar> I guess I need to commit hairy query.
<jcsackett> rockstar: thanks, i missed makeCodeImport when I read through factory.py
<sinzui> rockstar, or you add a rule to your sql to add UPDATE CASCADE before your drop and trick the sanity check
<rockstar> sinzui, yeah, that might be a good idea.
<rockstar> sinzui, or I could just drop the column.
<mwhudson> jcsackett: "because our data model is a bit messed up"
 * jcsackett laughs.
<mwhudson> (the answer to many questions about any sufficiently old and large project i guess!)
<mwhudson> jcsackett: branch.url is only about mirrored branches
<rockstar> mwhudson, yeah, I was just talking to wgrant a few days ago about the prospect of Delorean Driven Development.
<mwhudson> jcsackett: really, the information about where the branch is mirrored from should be separate from the branch table, as it is for code imports
<mwhudson> jcsackett: but (a) chunk of work to change for no real benefit (b) ~noone uses mirrored branches any more
<jcsackett> mwhudson: yeah, i should have actually known that since i was doing some work with imports prior to looking at factory; i just didn't see makeCodeImport and assumed makeBranch did something magical.
<jcsackett> and then was sadly disappointed. :-P
<mwhudson> heh
<rockstar> jcsackett, "there is no disappointment in no code."
<rockstar> sinzui, what if I did this: http://pastebin.ubuntu.com/508280/
<sinzui> rockstar, does it work?
<rockstar> sinzui, yessir.
<sinzui> then I love it
<rockstar> sinzui, yay.
<wgrant> Yay, my branch finally worked on DF.
<wgrant> We can release Maverick.
<flacoste> gary_poster, lifeless: do we know why the weekly and monthly performance reports are not generated?
<gary_poster> no
<flacoste> that's the https://devpad.canonical.com/~stub/ppr
<gary_poster> on call
<poolie> hi all
<flacoste> hey poolie!
<poolie> hi flacoste
<poolie> i'll still try to get you some bugs-closed numbers
<poolie> it's not as easy as it should be :/
<flacoste> well, it's actually not an easy problem
<flacoste> poolie: i've tried gathering the data at the DB level, and found it hard
<poolie> i can see that having some metric is useful
<poolie> mm that's what i was going to do next
<flacoste> what should we count?
<poolie> "line written" :-P
<flacoste> bugs on the bzr project
<flacoste> across the bazaar project
 * sinzui checks his query data for closed bugs by developer and milestone
<flacoste> fwiw, there were 4 bugs tagged udd closed during the quarter
<poolie> i was going to try in bzr, udd, bzr-*
<poolie> mm
<flacoste> seems low
<poolie> it seems a bit low
<poolie> i suspect the tag is not always applied, especially when people split a bug into several etc
<flacoste> that's surprinsgly low given the total number of bugs fixed
<flacoste> so not sure i wanted to report that data :-)
<flacoste> top was 'easy' bugs
<flacoste> i didn't include bugs in the udd project
<flacoste> since i suspect they weren't tagged udd
<flacoste> i was going to add a UNION for those
<flacoste> poolie, if you could only run the hottest script for main, that would be good
<flacoste> i can probably get the bugs data myself
<flacoste> so don't spend too much time on that
<poolie> one way to do it is to count things from the release notes
<poolie> manually classifying them now as udd relevant or not
<flacoste> right
<flacoste> but that excludes work that was done outside of bzr
<flacoste> itself
<poolie> from http://doc.bazaar.canonical.com/bzr.2.2/en/whats-new/whats-new-in-2.2.html and http://doc.bazaar.canonical.com/bzr.2.2/en/release-notes/index.html
<poolie> right, and both of them exclude non-code-changing work
<poolie> for example john has put probably a couple of months into lp-serve
<poolie> which is moderately relevant
<flacoste> yep
<poolie> but not in bzr at all
<rockstar> What time is PQM closing again?
<flacoste> poolie: could we run hottest100 from devpad weekly?
<flacoste> poolie: and have the reports available over HTTP, that could even be the first step to graphing this stuff
<flacoste> if you set up a cron job on devpad, i'm willing to do the graphing
<poolie> that would be good
<poolie> when i ran it last time, too many lp rpc requests failed
<poolie> i'm not sure what's changed that it's seeing that
<rockstar> Edwin-afk, when you return, ping.
<rockstar> (I'm probably going to need an RC)
<flacoste> poolie: well you ran it over a DB ugprade
<poolie> mm, i think it was failing even earlier in the day, but then there were a couple of outages
<LPCIBot> Project devel build (99): STILL FAILING in 3 hr 24 min: https://hudson.wedontsleep.org/job/devel/99/
<LPCIBot> * Launchpad Patch Queue Manager: [r=mars][ui=none][bug=654795] Allow a distribution's bug supervisor
<LPCIBot> to set source package bug reporting guidelines and acknowledgment.
<LPCIBot> * Launchpad Patch Queue Manager: [r=jml][ui=none] An ordering issue in Person:+subscriptions has been
<LPCIBot> fixed.
* Edwin-lunch changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is Release-Critical | firefighting: stable has test failures | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
* Edwin-lunch changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is Release-Critical (Release manager: EdwinGrubbs) | firefighting: stable has test failures | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<mwhudson> losa ping
<spm> mwhudson: yo!
<mwhudson> spm: the branchscanner db user can't access the distributionsourcepackage table
<mwhudson> spm: as seen in http://launchpadlibrarian.net/57215542/jR3X1uFDWYhXPet14K6N1hWzrkG.txt
<spm> ugh
<spm> ho ho ho. that sounds like cowboy #1 from the last release hasn't yet landed.
<mwhudson> spm: well, it's in devel
<mwhudson> spm: but i think which db was master changed recently?
<spm> heh. *twice*. yes. :-)
<mwhudson> so it's possible the cowboy didn't propagate with the master switcheroo?
<mwhudson> spm: anyway, can you recowboy?
<spm> yarp. so doing.
<mwhudson> thanks
<mwhudson> i don't know how to rescan the branches this affected...
<spm> https://pastebin.canonical.com/36984/
<spm> already done
<spm> UPDATE 33
<mwhudson> spm: heh, this has happened before i guess? :)
<spm> istr stub did run sec.py, so I'd assume that zotted all the grnats nicely
<mwhudson> because otherwise that was some quick typing :)
<spm> mwhudson: https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus <== cowboy numero uno
<spm> I am teh awesome
<LPCIBot> Project db-devel build (58): STILL FAILING in 3 hr 43 min: https://hudson.wedontsleep.org/job/db-devel/58/
<LPCIBot> * Launchpad Patch Queue Manager: [r=stub][ui=none][bug=654882] Add remote_name to component and
<LPCIBot> component group tables to track the original name from bugzilla,
<LPCIBot> so we can supply it when forwarding bugs.
<LPCIBot> * Launchpad Patch Queue Manager: [r=stub][ui=none][bug=644794] Switch out a link to
<LPCIBot> DistroSourcePackage,
<LPCIBot> in favor of separate links to Distribution and SourcePackage.
<LPCIBot> * Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][bug=655614] Don't publish indices for
<LPCIBot> disabled architectures.
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11683,
<spm> I just thought; ya know, this'll effect branches that need rescanning and just typed that query in. magic.
<LPCIBot> 11684, 11685, 11686 included.
<mwhudson> spm: is the scanner chewing on the backlog now?
<lifeless> nom nom nom
<spm> not sure yet - just getting the slaves updated as well
<mwhudson> pretty sure the scanner doesn't talk to the slaves
<spm> mwhudson: ahh it's only having issues on *some* branches; so it was still mostly (for values of most) working
<mwhudson> spm: right
<mwhudson> i don't understand why "        and job.date_created > (current_timestamp at time zone 'UTC' - interval'1 day')" is in that query
<lifeless> spm: bugger, can you cancel my pqm submission? plox?
<spm> lifeless: sure
<lifeless> mwhudson: https://code.launchpad.net/~lifeless/launchpad/db-code
<lifeless> mwhudson: its just a revert; are you willing to rs it ?
<lifeless> I propose to throw it straight at db-devel and let bb sort it out
<spm> lifeless: sorry - too late; it's already gone thru.
<lifeless> spm: no way, it takes longe rthan that to apply
<lifeless> perhaps it didn't get sent . \o/
<spm> https://pastebin.canonical.com/38360/ <== that one?
<lifeless> yeah, it didn't land. thanks for looking
<wallyworld> rockstar: you ok for a chat?
<rockstar> wallyworld, sure are.
<rockstar> Lemme slap skype into submission.
<wallyworld> skype?
<lifeless> mwhudson: gnip
<mwhudson> lifeless: it seems reasonable
<poolie> i wonder if we should implement "like this" in lp by linking out to one (or more) praise-giving web services?
<poolie> rypple, ohlo, ... probably more
<poolie> flattr
<lifeless> if we want that feature that might be a good way to do it.
<rockstar> wallyworld, can you hear me?
<poolie> i'm not saying i urgently want it, but adding the integration seems more tractable than writing the whole thing
<wallyworld> rockstar: nope
<lifeless> poolie: its probably more complex to integrate
<poolie> than to write from scratch?
<lifeless> because it raises questions we don't have if we do it inhouse
<lifeless> yes
<lifeless> like, off the cuff:
<lifeless>  - how do we deal with liking private objects
<lifeless>  - or private people liking things for that matter
<lifeless>  - how do we deal with objects being renamed
<lifeless>  - how do we ensure it themes in well on an ongoing basis (not just when we do the integration)
<poolie> sure, of course
<mwhudson> "be the first of your facebook friends to like this!"
<mwhudson> i can see that going down well :)
<lifeless> as long as it only shows up on oem services projects ;)
<poolie> needs a feature scope for "hates non-source-released web services"
<wgrant> sinzui: Can we switch off lp and db_lp, then?
#launchpad-dev 2010-10-08
<sinzui> wgrant, I do not know. mars and gary will know
<wgrant> Can someone mentor StevenK's code* of https://code.edge.launchpad.net/~wgrant/launchpad/bug-655648-a-f-maverick/+merge/37820?
<lifeless> wgrant: yes we can switch it off
<lifeless> spm: deathrow fialing to run ?
<spm> what's the logs say?
<wgrant> Erk.
<wgrant> When did it start to fail?
<wgrant> Is it complaining that it can't remove files because LFCs are missing?
<lifeless> wgrant: one day ago
<wgrant> One day ago, or 13 hours ago?
<lifeless> spm: The script 'process-death-row' didn't run on 'cocoplum' between 2010-10-07 22:07:04 and 2010-10-07 23:07:04 (last seen 2010-10-06 07:13:08.625020)
<wgrant> Oh, cocoplum.
<lifeless> wgrant: first error was The script 'process-death-row' didn't run on 'cocoplum' between 2010-10-06 15:07:04 and 2010-10-06 16:07:04 (last seen 2010-10-06 07:13:08.625020)
<lifeless> and
<lifeless> The script 'process-death-row' didn't run on 'germanium' between 2010-10-06 11:07:03 and 2010-10-06 13:07:03 (last seen 2010-10-06 07:33:51.715570)
<wgrant> Is germanium's OK?
<spm> lifeless: no, the logs from cocoplum for that script. they'll be on devpad
<wgrant> Crap.
<wgrant> So it predates the DB surgery last night.
<wgrant> That's a good thing.
<lifeless> spm: where would I look?
<lifeless> wgrant: publish queue or pload ?
<spm> devpad? heh, what I usually do is 'locate death' and infer from there :-)
<wgrant> lifeless: pload?
<lifeless> 2010-10-07 20:33:04 INFO    Creating lockfile: /var/lock/launchpad-process-death-row.lock
<lifeless> 2010-10-07 20:45:06 ERROR   Unexpected exception while doing death-row unpublish
<lifeless>  -> http://launchpadlibrarian.net/57251973/rbZP27X6hCTbVYqO1d4X8iLXCSg.txt (terminating connection due to administrator command
<lifeless> server closed the connection unexpectedly
<lifeless>         This probably means the server terminated abnormally
<lifeless>         before or while processing the request.
<lifeless> )
<wgrant> Well, that is awesome.
<lifeless> spm: there are death-row entries in the log file, what does the 'failed to run' stuff look for ?
<lifeless> wgrant: it continues:
<lifeless> )
<lifeless> 2010-10-07 20:45:15 INFO    Processing http://ppa.launchpad.net/soren/ppa/ubuntu
<lifeless> 2010-10-07 21:03:05 INFO    Creating lockfile: /var/lock/launchpad-process-death-row.lock
<wgrant> So it mysteriously died without saying anything?
<spm> that last line is a continual source of useless information across a bunch of scripts. it's not creating a lock file; it's starting to try and create a lock file and may or not have succeeded.
<spm> it looks to be nicely hung, just awaiting being drawn and quartered.
<spm> Process 32346 attached - interrupt to quit
<spm> restart_syscall(<... resuming interrupted call ...>
<wgrant> Is it running with maximum verbosity?
<spm> -vv
<wgrant> Does that show DEBUG? I forget.
<spm> i've killed cocoplum
<spm> yes
<wgrant> Where was cocoplum's hanging?
<wgrant> (in the log)
<spm> this is where it gets tricky to tell
<spm> but if I hazard a guess: 2010-10-07 22:10:11 DEBUG   Unpublishing death row for Primary Archive for Ubuntu.
<wgrant> Wll, that's handy.
<wgrant> Can you set that LP_DEBUG_SQL thingy and run it again?
<wgrant> Since it's not being very verbose...
<lifeless> spm: does nagios alert on this condition?
<spm> lifeless: no; that's what scriptactivity is for
<lifeless> perhaps we should feed that into nagios
<spm> given the number of apparently false alarms we're getting from scriptactivity; I'd be unhappy at doing that atm :-(
<lifeless> ok
<lifeless> are there any false alarms without bugs?
<spm> basically we've been doing a major ... redo(?) of nagios alerts the past few months. so critical alerts really are critical; with a special subset of those being ZOMG critical. everything else is warning; with the idea that they get manually scanned, vs an actual alert. "Red Fatigue" is a real problem.
<lifeless> I get that
<lifeless> I'm asking if the false alarms have had bugs created for them
<wgrant> lucid_db_lp hates me.
<wgrant> ...
<wgrant> Somehow production-devel got merged into db-devel with my branch. Despite not showing up in the MP diff.
<lifeless> wgrant: heh; did you  build your branch on my cp branch ?
<wgrant> Yes, but then I replayed on top of db-devel. Apparently that created a merge from production-devel.
<wgrant> And somehow the diff doesn't include the merged changes.
<lifeless> what rev
<lifeless> of db-devel
<wgrant> 9875
<wgrant> https://code.edge.launchpad.net/~wgrant/launchpad/bug-655614-disabled-arch-indices/+merge/37849
<spm> lifeless: (sorry distracted a bit there) I'm not aware of bugs being filed; I'd assume that those who are responsible for the scripts in question are reading the logs on devpad for same and filing bugs from that? :-)
<lifeless> wgrant: the diff doesn't update once merged.
<lifeless> spm: I make no such assumption.
<wgrant> lifeless: I know.
<wgrant> lifeless: So the diff should show the result of the merge.
<wgrant> But it ignores the fact that there's a production-devel in there.
<lifeless> spm: can you start either filing drive-by bugs when you get a false alert - any false alert, or failing that at least email me and I'll file the bugs.
<lifeless> wgrant: the content of the commit on db-devel is definitely fine.
<wgrant> lifeless: bzr cdiff -c9875 doesn't show a windmill change in versions.cfg?
<lifeless> wgrant: well it has lib/devscripts/ec2test/tests/test_remote.py changed
<wgrant> Right, that's another one. Note how that doesn't show up in the MP diff.
<lifeless> wgrant: and versions.cfg
<lifeless> -# r1544 of lp:windmill (the tip revision at the time of packaging).
<lifeless> +# r1440 of lp:~bjornt/windmill/1.3-lp. It includes our patches to make test
<spm> lifeless: heh, by false alerts; i mean that some of these generate errors semi frequently; and those failure alerts go to the LP list; and no one in turns says "hey this is a problem! the logs say X, can a losa investigate Y; I've filed Bug Z to track this" ergo, false alarm.
<mwhudson> like oops-prune, for example?
<spm> right
<spm> tho that one did exactly as I described. diogo did chase it up and file a bug.
<mwhudson> although for that one, i thought it was a conscious decision that it didn't warrant a CP
<wgrant> lifeless: So why isn't that in the MP diff?
<mwhudson> so the monitoring arguably should have been disabled
<lifeless> spm: ergo a bug should be filed that it should not generate that error
<wgrant> The MP diff isn't very useful if it doesn't show the real diff.
<spm> otherwise we're just middlemanning the information for developers. we as losas are going to be looking at the exact same info that the developers already have access to. so we're duplicating for no reason.
<lifeless> spm: if the bug gets marked 'invalid' clearly it *is* an error and *should* be addressed.
<spm> mwhudson: yeah. that's a trap. disabling is very easy to forget to re-enable.
<spm> so you end up with convoluted loops like we have with the staging restores. we have a text lock file that stops stagnig alerts during updates; but have anotehr alert that checks for the age of that lock. meta alerts ftw.
<mwhudson> at '+ 1 month' mail ...
<lifeless> spm: there is a difference in focus between day to day reactive and feature work
<lifeless> spm: I'm not advocating lots of handoffs; I'm saying that operational issues should:
<lifeless>  - trap to the monitoring system [nagios]
<lifeless>  - not trap if nothing is wrong
<lifeless> and that when you see a trap, if its ignorable, we should change things to not trap.
<lifeless> I appreciate that the scriptactivity stuff isn't hooked to nagios, but it would be better if it was, and if we can't because its too noisy, we need to fix that.
<lifeless> and the losas are best placed to judge.
<spm> right. And I'm saying we're so flat out reacting to ignorable alerts; or puppetting requests that other salready have all the info they require; they we have approx zero time to focus on all tyhe feature reqeusts that are asked of us.
<lifeless> spm: so the way to dig out of that is to fix things one at a time; ignorable alerts seem straight forward. I won't know if its one of those unless you or someone else tell me.
<spm> it's not an us vs them situation here. its "our" system, and as we move more to regular rollouts etc, devs NEED to start being more aware of operational stuff happening. they can't shunt it aside as "oh I'm working on a feature"
<lifeless> spm: I agree
<lifeless> spm: who *should* I be asking to ensure that ignorable mails are turned into bug reports?
<spm> so personal take - if the scriptactivity reports, which are really operationally lite compared to alerts; are the first step devs have to see an operation area and fix it, so it is less noisy and more useful for them.
<spm> scriptactivity is purely under the control of dev's.
<spm> our involvement tehre is around ensuring things are added to be checked for failure. which is itself a failure state :-)
<spm> so look at cocobanana and germanium with the process-death-row alerts.  they're generating emails every hour. is this appropriate? it needs fixing, sure, manual intervention was required. But how long can it remain broken before we need to look? Were the logs recorded sufficient to identify what was the problem?
<spm> is it a critical service like, eg the branchscanner which is time sensitive and IMHO should be nagios'ed; or more "important, not critical"
<spm> and if so; how can we modify the logic in the activity report so we get timely alerts without dying under noise
<spm> ideally, every one of those scriptactivity responses would get a personal ACK, "this is noise, bug X; this is a problem losas do Y"
<lifeless> spm: so
<lifeless> spm: things like nagios understand the idea of edge/level triggering of notifications
<lifeless> spm: reproducing that in other tools seems pointless
<spm> hrm. sorta?. In that you have 4 states: unknown. critical. warning. ok. And nagios is geared to aprporpaitely deal with that. How those states are defined, is not a nagios thing; we have bunches of scripts to get these for funkier corners of losaville. I put SA into the same boat.
<spm> but who gets the alerts, and how timely they are is a BIG component here.
<spm> or rephrase. Critical bug reports. should they be tied to nagios? (going for absurd by way of demonstratin')
<spm> so a critical security bug gets an sms sent to The Developer On Duty; and if they don't ACK within 10 minutes it goes to a team lead; and if they don't ACK in 10, it goes to francis.
<spm> if X breaking is more a case of "we can wait a few hours, or days" then SMS alerts is the wrong way; and just get the script in question to send a more descriptive and full email to a generic list; nagios is overengineered and heavy weight for this case.
<spm> ie. it's friday. a week into our shift to daylight savings, and I personally still haven't adjusted the hours I get SMS alerts. multiply that by more people? urg.
<lifeless> spm: there's a pretty strong argument we should be running a 24x7 shop for security issues
<lifeless> spm: so I don't think thats particularly absurd
<lifeless> spm: I also get and agree that we shouldn't be alerting everyone on every fault, but again, reimplementing filtering and routing of that in an adhoc lp specific script seems unoptimal
<spm> as in adjusting how often we send out alerts and who they get escalated to etc? shrug, 6 of 1. for security alerts; if you really want to go that way, sure. nagios would be best; but if we just want to send a simple email; it's overkill. The logic on what is/not an alert will still need to be written. that part must happen.
<lifeless> yes
<lifeless> and my first comment was that when that logic is wrong its a bug, and it needs to be filed.
<lifeless> which I stand by.
<spm> at the moment that logic is (simplisticly) in the scriptactivity script; so that's the obvious place to fix it.
<lifeless> sure, I made no comment at the start of this conversation about where to fix it.
<spm> it has been a wide ranging discussion full of interesting and pertinent ideas and concepts. :-)
<lifeless> hah
<lifeless> :)
<spm> but yes, back to the start. atm, for better or worse, and for a while longer; reports out of scriptactivity are going to still go to the LP list. If those messages atm are noisy, or not as helpful, or indicative of bugs needing to be filed, then my 2c is that the recipients - the LP list - should be filing bugs about them. if we want losas to do bug filing; then the obvious fix is send the results to losas@. Which'd get me crucified by the others, but
<spm> anyway.
<spm> or phrased another way - losas are responsible for real time alerts; lp-devs are responsible for not-realtime alerts.
<spm> grotesquely simplified.
<lifeless> spm: scriptactivity failures are realtime IMNSHO, thats also a massive simplification
<spm> I'm not sure they are.
<spm> some of those scripts run every 5 minuyes; but we only care about alerts after 12+ hours.
<spm> checkwatches springs to mind as culprit #1
<spm> the poorly named nightly.sh, has a whole bunch of scripts in it. some that take 5+ hours to run. an additional hour or 3 on waiting for a success/fail result isn't urgent.
<lifeless> so simplisticly, scriptactivity should only be looking for a run every 12 hours
<lifeless> theres a mismatch there.
<spm> made worse that the scripts in nightly.sh are serial; and get bounced around start/stop times by other scripts being fast or slow
<spm> hrm. possibly comms misunderstanding; we have 22 separately timed runs (crontab entries) for scriptactivity; which in turn measures some 22 x ?? scripts.
<spm> eg: nightly.sh:
<spm> 0 5 * * * $LP_PY /srv/launchpad.net/production/launchpad/scripts/script-monitor.py -U statistician 1440 loganberry:karma-update loganberry:allocate-revision-karma loganberry:launchpad-stats loganberry:expire-questions loganberry:checkwatches loganberry:productreleasefinder loganberry:update-cache loganberry:launchpad-targetnamecacheupdater loganberry:flag-expired-memberships loganberry:update-personal-standing
<spm> ie. check for a successful run of those scripts on those server(s) once a day at 5am, for activity in the last 24 hours.
<spm> which gets busted nicely; if said script in total takes 24.1 hours to run.
<lifeless> well shannon has something to say about that
<lifeless> have to sample at twice the frequency you want to measure
<spm> I think my head imploded trying to appreciate that; so I'll just go for the wise 'nod'
<lifeless> so we need 12 hourly probes
<lifeless> looking for a run in the last 24
<lifeless> actually thats not quite right
<lifeless> we want 12 hourly probes and an error message if we have two successive probes with no activity, we also need that when those long scripts take forever, we a) consider them as one group and b) report on activity as it goes
<LPCIBot> Project devel build (100): STILL FAILING in 3 hr 28 min: https://hudson.wedontsleep.org/job/devel/100/
<LPCIBot> Launchpad Patch Queue Manager: [r=EdwinGrubbs][ui=none][bug 652007] Switch mlist-sync to use
<LPCIBot> LaunchpadScript as its base.
<_mup_> Bug #652007: scripts/mlist-sync.py should use LaunchpadScript as a base <mailing-lists> <oops> <Launchpad Registry:Fix Committed by sinzui> <https://launchpad.net/bugs/652007>
<wgrant> Hm.
<wgrant> bin/kill-test-services is broken.
<wgrant> The librarian stuff has been changed.
<lifeless> ah fnord
<lifeless> a) its untested
<lifeless> b) its untested
<lifeless> c) we should really move to the model I wrote up :)
<LPCIBot> Project db-devel build (59): STILL FAILING in 3 hr 41 min: https://hudson.wedontsleep.org/job/db-devel/59/
<LPCIBot> Launchpad Patch Queue Manager: [testfix][rs=sinzui][ui=none] Resolved adapter conflicts in
<LPCIBot> configure.zcml.
<sinzui> spm: I want to move productreleasefinder to a separate script or run it in parallel. Half of those scripts do not need exclusive use of the database
<bac> spiv:  the test suite has some known spurious tests and i think your branch got hit by them all.  :(
<spiv> bac: I'm a lucky guy!
<bac> yep.  so i'm going to wade through the discussion and see where we are.  will land after things return to normalcy but will probably miss today's PQM deadline.  no big deal since this isn't a rush by any means.
<spiv> No, clearly not :)
<spm> sinzui: sure. probably as part of the current release? or maybe as a CP or something? I guess the only Q I'd have - did you have a particular time you wanted it to run within; or just leave it to us to try and pick a "not as busy" time?
<LPCIBot> Project db-devel build (60): STILL FAILING in 1 hr 29 min: https://hudson.wedontsleep.org/job/db-devel/60/
<LPCIBot> * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11689,
<LPCIBot> 11690 included.
<LPCIBot> * Launchpad Patch Queue Manager: [release-critical=edwingrubbs][rs=mwhudson][ui=none][no-qa] Rollback dropping of BranchRevision.id
<lifeless> spm: deadrows still dead
<spm> i think that's a lie; the logs suggest stuff is happening.
<lifeless> so the scriptactivity check for it is wrong?
<spm> overly aggressive *atm*; probably
<lifeless> I'll file a bug
<spm> dunno if a spike in actions causing it to appear to fail or something funky
<lifeless> if the script is running
<lifeless> and scriptactivity is complaining, then our detection logic is flawed.
<spm> (last seen 2010-10-06 07:33:51.715570) <== that worries me
<spm> I know we've seen one script previously (don't recall which) that was working, to a point; and would then die. so stuff was happening but the magic record "I have finished successfully!" wasn't getting written.
<spm> StevenK: ^^ can you offer any insights as to what/where/why we can look for what's up?
<mwhudson> you can also have the fun where the outage causes so much work to build up that it takes longer than the check period to complete
<mwhudson> really scriptactivity is pretty awful
<spm> https://pastebin.canonical.com/38362/
<spm> vis last current entry: 2010-10-08 02:58:28 INFO    Processing http://ppa.launchpad.net/mozillateam/ppa/ubuntu <== germanium
<spm> so stuff is happening; but we're not getting completion.
<spm> *successful* completion.
<mwhudson> can you see when the currently running process started?
<spm> ~ 3.5 hours ago
<mwhudson> ah
<mwhudson> so i think my guess seems likely
<spm> hrm. maybe. I suspect there's more to it than that tho.
<spm> the current process, via the log, has been "stuck" for an hour.
<spm> ugh. there's whole bunches of oop's on germanium which aren't (apparently) on lp-oops
<spm> bah. the oops themselves are on devpad tho
<spm> eg /x/launchpad.net-logs/production/germanium/lp_publish/2010-10-06/85613.PPAPUBLISH854
<StevenK> I wonder if they all are PoolOverwrite
<spm> appear to be
<spm> based on the random 4 I've catted
<LPCIBot> Project devel build (101): STILL FAILING in 3 hr 25 min: https://hudson.wedontsleep.org/job/devel/101/
<LPCIBot> Launchpad Patch Queue Manager: [r=danilo][ui=none][bug=652256] Adds a configuration link for
<LPCIBot> translations_usage on the product translations front page.
<LPCIBot> Project db-devel build (61): STILL FAILING in 17 min: https://hudson.wedontsleep.org/job/db-devel/61/
<wgrant> StevenK, spm: deathrow is failing with PoolOverwriteErrors?
<spm> wgrant: not sure. something is failing with those errors; the oops isn't actually recording what script is doing the failing.
<wgrant> Ah, that would be p-d, then.
<wgrant> publish-distro, that is.
<wgrant> Although those shouldn't be happening any more.
 * StevenK kicks Firefox and Thunderbird
<wgrant> StevenK: Chromium! Evolution!
<StevenK> wgrant: Evolution doesn't like IMAP mailboxes with >10,000 messages for me
<wgrant> StevenK: Even in Maverick?
<wgrant> It was a bit bad in Lucid, but works OK in Maverick.
<StevenK> Evolution still makes me twitch
<wgrant> :(
<lifeless> StevenK: Ever hacked on it ?
<StevenK> lifeless: On Evolution? Nope
<lifeless> do so, *then* you'll have reason to twitch
<StevenK> Don't make me look at the bad code
<lifeless> its not 'bad'
<spm> isn't this the place someone says "patches accepted"?
<lifeless> lies
<spm> *where* someone says. gah
<wgrant> StevenK: i-d-s copies packagesets now, doesn't it?
<StevenK> wgrant: Yes
<StevenK> wgrant: I have a branch to limit which ones are copied, too
<wgrant> StevenK: NewReleaseCycleProcess still says to copy them manually.
<wgrant> And it suggests that -a will be needed to exclude disabled archs.
<wgrant> But I think that was CPd a couple of days back.
<StevenK> Yes, I know, I need to talk to Colin again
<wgrant> OK, great.
 * wgrant gives the publisher a couple of good stabs.
<StevenK> Which tentacle are you trying to cut off?
<wgrant> Refactoring the crap I was dealing with last night. Mostly Release generation.
<wgrant> It has this awesome reorder_components function, which looks safe enough (it creates a new list).
<wgrant> But of course it also calls .remove on the old list.
<wgrant> So it looks like it's going to be non-destructive to its input, but then completely obliterates it.
<wgrant> Hm, that's actually a bit interesting. It will have been mutating the publisher's internal config.
<wgrant> Surprising that it works.
<wgrant> Even better is that the tests don't care.
<wgrant> bigjools: Morning. dogfood looks happy!
<bigjools> morning
<wgrant> The publisher tests do not whinge if you disable publication for non-Release pockets :(
* bigjools changed the topic of #launchpad-dev to: INCIDENT: process-death-row failing for PPA and Ubuntu || Launchpad Development Channel | Week 3 of 10.10 | PQM is Release-Critical (Release manager: EdwinGrubbs) | firefighting: stable has test failures | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
* bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is Release-Critical (Release manager: EdwinGrubbs) | firefighting: stable has test failures, process-death-row failing for PPA and Ubuntu  | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<lifeless> stub: are you going to be able to make the cassandra training? will you be staying for the end of the week ? [I'll stay for the 2 end days if you're going to stay, otherwise I'll just go for the 3 days training.
<lifeless> stub: also, can tables be renamed in postgresql?
<stub> I think I'm going for the 3 days training.
<stub> Yes, tables can be renamed in PostgreSQL
<jml> mwhudson: hi
<jml> mwhudson: only a little
<jml> mwhudson: I was a little surprised that the service wasn't written with Twisted
<lifeless> stub: when will you know ?
<stub> Early next week - I need to ensure I can sort out paperwork and stuff earlier than expected.
<lifeless> ok
<lifeless> I'll hold off booking for now
<lifeless> doing the extra 2 days experimenting with stuff for LP with you could be great.
<stub> Come home via Bangkok ;)
<lifeless> the routing on that would be nuts I think, now that I'm in NZ
<poolie> hello stub, jml, lifeless
<jml> poolie: hi
<jml> poolie: replying to some of your emails
<jml> poolie: wrt your dkim branch... I don't think you should try too much harder to refactor that bit.
<jml> poolie: I think someone should though. in a different branch.
<poolie> your reply about notifications the other day made my day
<jml> :)
<spiv> bac: heh, a completely different failure this time
<bac> spiv: well, that keeps things interesting...
<poolie> jml we all seem to be pretty much in agreement then
<poolie> i may put up a change
<poolie> just wondered if i was missing something
<wgrant> spiv: Ah, you're trying to compete with my 6 unique failures for a single branch?
<spiv> wgrant: well, 2 from 2 tries so far.
<spiv> This time it's a spurious windmill failure.
<spiv> Last time it was 4 (non-windmill) failures.
<poolie> this is really sweet
<poolie> just add "variations = [VaryByHttpClientImplementation()]" to a test class and it spans out
<poolie> better than writing load_tests
<jml> poolie: that is cool
<jml> poolie: what does that?
<poolie> http://bazaar.launchpad.net/~mbp/bzr/597791-http-tests/annotate/head:/bzrlib/tests/test_variations.py
<poolie> jml, and http://bazaar.launchpad.net/~mbp/bzr/597791-http-tests/annotate/head:/bzrlib/tests/test_http.py
<poolie> i may split/copy it into testtools
<jml> yeah.
<jml> for some reason, testscenarios is a separate project
<poolie> oh, or testscenarios
<poolie> hard to keep up
<poolie> and testfixtures is another one again i think
<jml> yeah
<jml> "do one thing and do it well" vs "make it easy to find, discover & maintain these things"
<jml> I guess if I had more energy I'd try to come up with a good over-arching name for the project of high quality testing tools for Python
<jml> and then make a cool website and stuff
<lifeless> poolie: I'd love a patch for testscenarios to do that
<poolie> yeah, it's a funny thing
<lifeless> jml: tip ? :P
<poolie> activation energy to do and test it in another library, and to bump the dependency
<poolie> how about if you review it here and if we're happy, i'll move it
<poolie> is the general plan to copy or to move?
<lifeless> I don't think bzr uses testscenarios itself, yet.
<lifeless> will chat more next week; gnight.
<jml> g'night.
* bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.10 | PQM is Release-Critical (Release manager: EdwinGrubbs) | firefighting: stable has test failures | https:/â/âdev.launchpad.net/â | Get the code: https:/â/âdev.launchpad.net/âGetting
<jml> if we wanted to have future distroseries for Ubuntu open on Launchpad so that people could assign bugs to those future distroseries, what would we need to do?
<wgrant> It should mostly work now.
<wgrant> (I talked about this a bit with cjwatson earlier)
<wgrant> We can have a new series set to Experimental, and it will happily coexist uninitalised.
<jml> hmm.
<wgrant> I think all uploads will be held in NEW. Perhaps they should be rejected.
<wgrant> (until a couple of months back it would have caused publisher carnage, but that was fixed)
<jml> Arguably that should be independent of the series status
<jml> right, but maybe code other than the publisher will break if we start doing that
<jml> also, "Experimental" isn't such a great status for Natty
<wgrant> It's not, no.
<sinzui> jml: I agree with wgrant. We can do it now, but it will not be open for building and translations.
<wgrant> Oh, there's also Future.
<wgrant> Forgot about that status.
<jml> that's a much more appropriate status
<jml> who would have permission to try an experiment on staging?
<jml> (I know staging isn't up right now)
<jml> and by permission I mean computer permisson
<wgrant> It's possible that ~ubuntu-drivers could do it, but it may well need an ~admin.
<jml> I'm surrounded by people who have the authority to do so :)
<wgrant> (I'm not sure I've ever seen Future used outside Debian)
<wgrant> (so it may not even be exposed in the UI)
<jml> right
<wgrant> Yeah, it needs an admin.
<jml> so "mostly work" might be more accurately phrased "doesn't work"
<wgrant> Well, Experimental would work fine.
<wgrant> Gr no staging.
<jml> another question, when is the psycopg problem going to be fixed so I can upgrade in peace?
<sinzui> jml: the bug was marked fixed a few days ago
<jml> sinzui: what do I have to do to get the fix?
<wgrant> It's not fixed..
<wgrant> launchpad-dependencies depends on the old version.
<sinzui> maybe wait for deps to update
<jml> wgrant: yeah, that's what I mean :)
<wgrant> Forcing installation of the old version is not a fix. It breaks upgrades and introduces all sorts of madness.
<wgrant> The real fix is inhibited by an argument over who is at fault.
<wgrant> sinzui: I can't view bug listings at all for a distro that doesn't use Launchpad Bugs. Is this intentional?
<wgrant> Not even +bugs works.
<sinzui> wgrant,  not exactly. I expected hacking the url to work
<wgrant> jml: Release nominations work as expected for Experimental and Future series.
<jml> wgrant: thanks.
<jml> wgrant: I'm just mucking around on a launchpad.dev instance now
<jml> I notice that we can't register a series as "Future". It seems to default to Experimental, but I'm not sure why that is.
<sinzui> jml: there are some old rules about the status of a new series and how it can progress through the status
<wgrant> Hah, newSeries does, yes. How very odd.
<jml> sinzui: that makes me want to ask a bunch of questions:
<jml> sinzui: 1. what are these rules?
<jml> sinzui: 2. are these rules still appropriate? what was the justification for them?
<wgrant> Some of the rules are still critical to Soyuz operation.
<jml> sinzui: 3. why are these rules not obvious on the series registration page?
<wgrant> The one about starting in Experimental... not so much.
<sinzui> the exist to ensure you cannot accidental move an ubuntu series back in status. If you make a mistake, god speed, find a losa and hack the db!
<jml> wgrant: it would be nice to decouple soyuz operation and distroseries status, I think
<wgrant> jml: Probably.
<wgrant> We need a tristate column to control the state of the RELEASE pocket, mainly.
<sinzui> jml, wgrant Only losas are creating series. Drivers cannot update the statues. We reply on a losas to do this. This is an Ubuntu exception
<jml> sinzui: I don't understand... how can rules about the status of a *new* series prevent accidentally moving a series back in status
<jml> sinzui: why are only losas allowed to create series?
<wgrant> Because they can seriously break Soyuz if done badly.
<sinzui> Ubuntu wants it that way
<jml> sinzui: pretend they don't.
<sinzui> I send an email to several people asking why this is
<sinzui> bdmurray, deduced the issue is that Ubuntu made its driver team the owners. There are too many untrusted people who would have permission to create series
<jml> sinzui: where is this documented?
<sinzui> bdmurray, sent an email last month explaining the issue and suggested that Ubuntu have an separate owner team
<jml> sinzui: how would I have found out about this if you were asleep?
<sinzui> jml: There is a bug about this. I have sent many emails to the Lp list overt the last two years. mdz also asks about what a driver can do (Ubuntu driver team or driver role?), and the Ubuntu council is where the recent emails were
<wgrant> (there is one thing that ~ubuntu-drivers can (but shouldn't be able to) do that nobody has mentioned yet)
<sinzui> jml: bug 174375
<_mup_> Bug #174375: Distribution drivers permissions may need redesign <Launchpad Registry:Triaged> <ubuntu-community:Confirmed for techboard> <https://launchpad.net/bugs/174375>
<jml> wgrant: which is?
<wgrant> jml: Well, check the owner of the Ubuntu primary archive...
<wgrant> And weep.
<jml> wgrant: I don't know how to do that.
<wgrant> Well, it's ~ubuntu-drivers.
<wgrant> So they can copy stuff in and probably add permissions.
<wgrant> (just noticed this now; it's not really related to the distribution.driver field, besides being the same team)
<jml> do I have to bug a losa if I want to experiment on staging?
<Chex> jml: if you mean working on the servers behind the web-interface of staging, then yes, through us
<jml> Chex: in this particular context I mean creating a new series of Ubuntu
<jml> once staging has restored itself to its usual glory, of course
<wgrant> It has been down for hours... I fear that it will not.
<jml> oh. pity.
<wgrant> I am hopeful that it's because someone is secretly setting up qastaging.
<wgrant> But I doubt it.
<sinzui> Chex, wgrant, jml: Edwin reported that staging has been down now for at least 12 hours
<sinzui> Chex, is staging stuck?
 * sinzui has qa to do on staging
<dobey> hey abentley
<abentley> dobey: hey.
<dobey> how's it going?
<abentley> dobey: alright.  Yourself?
<dobey> pretty good
<dobey> any luck with the branch scanning db issue?
<sinzui> does any one have power to moderate launchpad-feedback? flacoste?
<flacoste> sinzui: i have
<flacoste> sinzui: i'll take care of it
<sinzui> thanks
<abentley> dobey: I am waiting for Chex to tell me what the procedure is to reinstate the needed db permissions.
<dobey> abentley: ok
<abentley> dobey: the fix will be permanent when we do our next rollout, but I'm trying to get it reinstated now.
<bigjools> jml: what do you think about this: http://bazaar.launchpad.net/~julian-edwards/launchpad/builderslave-resume/revision/11661
<dobey> abentley: ok. i'm just asking about it constantly because tarmac requires metadata from the LP API that is only available when rescan succeeds, and we have some u1 branches that are not getting rescanned properly
<jml> bigjools: the comment and the logic look ok. I'm surprised that transaction.commit() doesn't do the reset though
<abentley> dobey: I understand that you need it.  I think it's unacceptable that it's breaking, but I don't have the power to fix it myself.
<jml> bigjools: also, I need to talk about packagesets
<jml> but maybe that's Monday
<flacoste> sinzui: done, there were 3 messages not spam
<bigjools> jml: the tests pass... not sure what else to say about it
<bigjools> jml: when would you like to talk about packagesets?
<bigjools> oh Monday
<jml> bigjools: what does write_transaction do that's different to transaction.commit()?
<bigjools> I think the firefighting this week has made me blind
<dobey> abentley: understandable
<bigjools> jml: the decorator is itself decorated with the reset_store one
<sinzui> thanks flacoste
<jml> I'm pretty sure I have more grey hairs today than I did Monday.
<jml> bigjools: ahh, ok.
<bigjools> jml: which is a little nasty for partial commits
<jml> bigjools: yeah
 * bigjools now intrepidly steps forward into the buildd-slavescanner.txt wilderness
<jml> bigjools: perhaps write_transaction shouldn't have the reset_store decorator, and the librarian should have reset_store explicitly on methods that currently have write_transaction
<bigjools> jml: yeah I did raise that with gary_poster, I think it warranted some careful testing to see if that reset is still needed, since it might be a hangover from sqlobject
<gary_poster> yes, removing that is a bit of a big deal
<gary_poster> since we also do that for transactions
<gary_poster> in the webapp
<gary_poster> Landscape doesn't reset store
<bigjools> interesting
<gary_poster> But it's been like this for us for so long that I'm afraid that we might rely on it
<bigjools> I wonder how much of this stuff is slowing us down
<jml> bigjools: but what I'm saying is that we don't have to worry about that
<gary_poster> so changing it would be one of those "who knows what will fall over when we switch it on"
<jml> bigjools: we split it off write_transaction, and then change the places where write_transaction is used so that it uses reset_store explicitly
<bigjools> gary_poster: indeed!
<jml> absolutely 0 risk.
<bigjools> jml: yes, it's only used in one other place
<gary_poster> but then we expose something that has untested semantics?
<gary_poster> sorry, I had a nick ping, but am not really fully in context yet
<gary_poster> IOW, why do we want to remove it out of write_transaction, if we are worried it might be necessary in write_transaction?
<bigjools> gary_poster: 'tis ok, it's not urgent by any means, and jml is talking about something different.  But I reckon we should spend some time examining if the reset is needed at some point.
<gary_poster> completely agree
<gary_poster> I've wondered about that since I started, actually
<gary_poster> (ish ;-) )
<bigjools> gary_poster: it may have been needed by the sqlobject compat layer, I wonder if jamesh would know
<gary_poster> he did
<gary_poster> he told me that it was
<bigjools> aha
<gary_poster> so like I said, we probably don't need it
<gary_poster> except that it may be a pervasive, underlying assumption for some of our code now
<bigjools> right
<bigjools> yeah :(
<bigjools> but our 100% test coverage would root that out!
 * bigjools runs
<gary_poster> :-)
<sinzui> henninge, noodles775: have you discussed UI review graduation
<henninge> sinzui: no and noodles775 is not my mentor ... ;)
<sinzui> henninge, thanks, sorry about my incompetence.
<henninge> sinzui: np
<noodles775> :)
<rockstar> Edwin-afk, ping
<jcsackett> sinzui: you encountered oddities with metal:head_epilogue and conditional content, right? specifically the condition being ignored and it always being there?
<sinzui> bac did
<jcsackett> ah. do you remember what he did to get around that?
<jcsackett> alternatively: bac? you still around?
 * sinzui checks pastebin
<sinzui> jcsackett, http://pastebin.ubuntu.com/507152/
<jcsackett> sinzui: thanks.
<dobey> abentley: still around?
<abentley> dobey: yes.
<dobey> abentley: so it looks like Chex did the patch, but am still seeing a couple branches (that were already pushed), waiting to be rescanned. any ideas on how to get them to 'finish' the rescan?
<abentley> dobey: Yes, the branches need to be write-locked and unlocked to generate a new scan job.  Pushing to them should do the trick, or I can do a manual thing.
<dobey> abentley: can you do the manual thing? the owner of the two branches i'm looking at isn't around, so i can't ask him to repush at the moment
<abentley> dobey: sure.  What branches?
<dobey> https://code.edge.launchpad.net/~rodrigo-moya/libubuntuone/dont-depend-on-gconf
<dobey> https://code.edge.launchpad.net/~rodrigo-moya/ubuntuone-client/dont-use-dialog-vbox
<dobey> those two
<abentley> dobey: okay, those should get scanned in the next minute.
<dobey> abentley: great. thanks much!
<abentley> dobey: you're welcome.  Sorry for the glitch.
<mars> A good Friday laugh: http://people.canonical.com/~mars/performance.png :)
<bigjools> I feel like the guy at the bottom this week
<mars> lol
<bigjools> g'night all
<mars> good night bigjools
<gary_poster> sinzui: I'm trying to figure out an immediate way to help Darwin @ https://answers.edge.launchpad.net/launchpad-foundations/+question/128430 .  I was going to suggest that he merge his accounts (pointing to https://help.launchpad.net/YourAccount/Merging).
<gary_poster> However, to do that without screwing up his PPA he would need to log into his primary account.  I could suggest merging his account to the secondary account, and then changing his name for him back to his primary account.
<gary_poster> Thoughts or ideas?
<gary_poster> Alternatively, I could try to construct SQL to try to figure out what is going on and ask the LOSAs to run it, but that's going to be such an expensive flail that I was hoping there would be another way
<sinzui> gary_poster, The reverse merge and rename is viable. I have suggested it to users as well.
<gary_poster> ok thanks sinzui.  I'll give it a whirl, and hope stub can investigate the underlying problem
<sinzui> gary_poster, deleting a PPA is very destructive, you cannot get it back...the name is taken forever
<gary_poster> sinzui: does that mean that the reverse merge/rename is dead for him?
<gary_poster> (cause he has a PPA in the primary account)
<sinzui> We cannot rename someone with a PPA. It must be deleted first
<sinzui> I do not think you can merge a profile with a PPA
<gary_poster> :-(
<gary_poster> So not an option
<gary_poster> So DB fiddling is the only real option
<sinzui> gary_poster, are the account ids wrong/switched?
<gary_poster> he doesn't have access to his primary account anymore.  He reports that he tried a password reset and never got an email back.  Unless something is seriously hosed in the LP database, that's probably what I need him to do.
<gary_poster> (I mean, work with ISD to get the password reset working for him)
<sinzui> gary_poster, did the user delete an email address on his baudm profile
<gary_poster> sinzui: he didn't report doing so, but I don't see how he could be where we are without doing so
<gary_poster> (I have not asked, but would be happy to if you thought it were worthwhile)
<sinzui> password reset is automatic from a form. it sends it to your email address. Follow the link and you are done.
<gary_poster> he says he didn't get the email
<gary_poster> I'll pursue that
<flacoste> gary_poster: do you know if stub qa-ed rev 9762?
<gary_poster> not by number :-) looking...
<flacoste> gary_poster: see https://devpad.canonical.com/~lpqateam/qa_reports/deployment-db-stable.html
<gary_poster> k
<flacoste> we are now on 9880
<cr3> when should an interface inherit from another interface or the model implement that interface?
<cr3> for example, the Product class implements IHasLogo and the IProduct interface inherits from IHasLogo too
<gary_poster> it's a semantic question cr3.  If an interface semantically subsumes another interfaces, it should inherit.  OTOH, if a class happens to implement two semantically different interfaces, the interfaces themselves should not inherit
<gary_poster> flacoste: what you see is an instance of bug 640566.  The old bug was added into the mix erroneously.  The new revision was in fact QAd according to the other bug: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/644975
<_mup_> Bug #640566: qa-tagger should not consider linked bugs to branches when bugs are already Fix Released <new-merge-workflow> <qa-tagger:In Progress by ursinha> <https://launchpad.net/bugs/640566>
<_mup_> Bug #644975: Export OpenIdIdentifier to Canonical SSO <qa-ok> <Canonical SSO provider:New> <Launchpad Foundations:Fix Committed by stub> <https://launchpad.net/bugs/644975>
<gary_poster> I am determining with Ursinha now what we should do practically about that old bug
<cr3> gary_poster: in the case of IHasLogo though, if the IProduct interface already inherits from IHasLogo, does it make semantic sense for the Product implementation to explicitly call implements for the IHasLogo interface, shouldn't it be implied by calling implements for the IProduct interface?
<flacoste> gary_poster: ah, ok, that means the report isn't useful at this stage then
<flacoste> gary_poster: to determine which revision can be rolled out
<Ursinha> flacoste, this kind of situation is an exception
<gary_poster> flacoste: correct, when people use the same branch repeatedly, it falls over until that bug is fixed
<flacoste> which is common in our team at this stage
<gary_poster> flacoste: I'm going to mark that qa-ok, on Ursinha's suggestion
<flacoste> stub, lifeless and others do this
<flacoste> ok
<flacoste> staging is down btw
<flacoste> we are stuck on a replication error
<gary_poster> They are the only two AFAIK, but in any case Ursinha is working on it
<flacoste> I've been known to reuse branches also
<flacoste> not that i'm a big commiters anymore
<gary_poster> :-)
<gary_poster> ok report should be happy again next time it runs
<gary_poster> flacoste: staging replication error: my only remediation options are to ask LOSAs to review the kinds of things stub have asked them to review in the past, or wake up stub, or send stub an email.  My inclination is to send stub an email, since I assume LOSAs have already done "the usual" whatever that is.  Agree/disagree?
<flacoste> gary_poster: it also means that it's likely that we won't have an estimate the time the DB upgrade take
<gary_poster> flacoste: yes
<flacoste> Chex: have you gone through the usual slony debug checklist?
<Ursinha> gary_poster, flacoste, it's updated now: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-db-stable.html
<gary_poster> Thanks Ursinha
<flacoste> Ursinha: do we have a bug to include the revision until tip?
<flacoste> Ursinha: for example, on https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html, i have no idea how many revisions follow that not-qaed revision
<flacoste> if it's only 1, it's not the same things as if there are 10 other revisions after that
<Ursinha> flacoste, no, there's not
<Ursinha> makes sense
<flacoste> i'll file one
<flacoste> what project?
<Ursinha> flacoste, qa-tagger
<gary_poster> Ursinha: I was pretty sure that these are shown
<gary_poster> But maybe I misremember :-/
<Ursinha> gary_poster, iirc they were never there
<Ursinha> mars knows better
<flacoste> Ursinha: bugs 65712 and 657015
<gary_poster> Ursinha: ok, you would probably know :-) .
<flacoste> bug 657012
<_mup_> Bug #657012: Display the revision number of tip on the report <qa-tagger:New> <https://launchpad.net/bugs/657012>
<flacoste> and bug 657015
<_mup_> Bug #657015: Deployable revisions report should go until tip <qa-tagger:New> <https://launchpad.net/bugs/657015>
<mars> Ursinha, ? reading backscroll
<mars> Ah, yes.  Ursinha, they are included in the fake report but the real application does not produce them
<Ursinha> thanks flacoste
<Ursinha> and mars
<sinzui> gary_poster, I updated the bug. We should request a losa to try a merge from edge to see if it fixes the users login issue
<gary_poster> I was just reading your reply sinzui, thank you.  OK, I'll try that.
<gary_poster> sinzui: djclue917 got an email address when he used that email address again, somehow, I think.
<gary_poster> oh, duh, you said this
<sinzui> login is so very confusing that I cannot remember what I say about it either
<sinzui> gary_poster, since the address is not preferred, I wonder if it is possible that the address was taken from the Lp profile...Lp profile -> 3 addresses, but two accounts with different email addresses.
<gary_poster> sinzui: does the lock on this page show that the email address is preferred now? https://edge.launchpad.net/~djclue917
<sinzui> The lock means it is hidden.
<gary_poster> oh ok
<sinzui> gary_poster, you have demi-god powers to see it
<gary_poster> go, me!
<mars> gary_poster or benji, API question for you: do you know what happens if you log in twice, passing different parameters to allow_access_levels?
<mars> login_with(allow_access_levels=["READ_PUBLIC", "READ_PRIVATE"])
<mars> then, log in again with: login_with(allow_access_levels=["READ_PUBLIC", "WRITE_PUBLIC"])
<benji> mars: I don't know definitively, but my strong suspiction would be taht the second superceeds the first
<gary_poster> not offhand, mars.  if benji doesn't, can look around.
<gary_poster> well, I'd hope so, but that doesn't mean it's so :-)
<mars> benji, by supercedes, do you mean it will ask you to authorize the application again?
<mars> since you changed the token's authorization
 * gary_poster hopes so
<mars> yes, that is what I suspected
<gary_poster> have you tried, mars?  that's what I'd do, with staging, since we are coming up empty
<mars> gary_poster, not yet, will do so.  benji, gary_poster, thanks for the help
<gary_poster> such as it was, in my case :-)
<benji> mars: yep, that is my expecation (and hope)
<rockstar> abentley, you know what would be cool?  A pump command that could work while I'm in the pipe I want changes to get pumped to.
<abentley> rockstar: yeah, I sometimes want that.
<abentley> rockstar: I think you can abuse pump --from-submit for that, but that will also apply any changes from the submit branch.
<rockstar> abentley, yeah, and then we get weird criss-cross merges too.  I always have to be careful with that.  :)
<abentley> rockstar: err, --from-submit will rarely give you criss-cross.
<abentley> rockstar: only if, say, you submit your first pipe, then merge & commit, then merge again after the pipe lands.
<mars> gary_poster, do you remember the conditions under which a staging code update happens?
<mars> I am looking at the graphs and there appears to be no regular pattern there
<gary_poster> mars, I think code updates happen as soon as a new revision is available, as checked by some time-regular cron-like thing; and db updates are on the weekend
<mars> gary_poster, ok, thanks.  I'll just say 'is down regularly'
<james_w> mars, I don't think it asks you to reauth
<james_w> mars, I think you just get 403 if you now want to do something that you didn't ask for permission for earlier
<mars> james_w, when escalating or dropping permissions?
<mars> ah
<james_w> mars, neither way does anything to existing tokens I don't think
<mars> ok.  Thanks james_w
<james_w> dropping is clearly not going to cause functional issues
<james_w> I may be wrong though, but I think that allow_access_levels is just sent to launchpad in the GET request to +authorize-token
<benji> gary_poster: on a freshly installed kubunto machine plus apt-get install python-keyring-kwallet I get "OSError: can't get access to dbus" when trying to store a password
<gary_poster> bah
<gary_poster> benji, sounds like it's time to trash keyring with prejudice
<benji> I concur.
<gary_poster> cool, go for it
<james_w> with what in preference?
<benji> gary_poster: do you want me to whip up a KWallet integration to go with our Gnome keyring support or go with what we have until the time comes?
<dobey> why is the prerequisite on a merge proposal a link to a branch, and not a merge proposal?
<gary_poster> benji: assuming KWallet integration is on the order of half a day to a day, go for it
<benji> sounds good
<gary_poster> cool
<dobey> gary_poster, benji: if this is in Python, which I presume it is, why not use python-keyring?
<gary_poster> dobey: that's exactly what we are talking about.  We've encountered several annoyances with it.  The straw that broke the camels back was that it doesn't seem to work on Kubuntu, which, other than Ubuntu/Gnome, is the other keyring that we care about.
<james_w> have you filed a bug?
<benji> james_w: not yet, but it's on my list
<dobey> gary_poster: that's odd
<gary_poster> dobey: "why is the prerequisite on a merge proposal a link to a branch, and not a merge proposal?" I don't understand why it would be a merge proposal, so I don't understand your perspective yet
<rockstar> abentley, shall we have our standing up?
<dobey> gary_poster: so we just ran into, and i filed bug #657038 with tarmac. and i'm working on a fix to tarmac to deal with it, but it doesn't seem like prerequisites work the optimal way for that. it seems like i would want to specify a merge proposal, because i am requiring the prerequisite to be merged into its target, before my proposal can land.
<_mup_> Bug #657038: Branches with prerequisites can land without prerequisite having landed <Tarmac:In Progress by dobey> <https://launchpad.net/bugs/657038>
<abentley> rockstar: sure.
<dobey> gary_poster: and since a branch can have many proposals with different targets associated, there is no easy way to determine what the developer meant by specifying a prerequisite
<gary_poster> dobey, ah I see.  AFAIK, prerequisites are only about presenting diffs from LP's perspective.  I can see why you would want that, and I expect the difference simply comes from the fact that the prerequisites weren't originally envisioned for that use case.
<dobey> gary_poster: so total usability fail then :)
<gary_poster> for this use case, yes, sounds like it
<dobey> well for both use cases
<dobey> because in the "show a different diff on LP" case, the diff doesn't necessarily show what will actually land
<gary_poster> MPs weren't made for tarmac
<dobey> ignoring tarmac
<dobey> if LP is showing a diff based on the prerequisite, and that proposal is approved/landed before the prerequisite, it might actually result in code being landed that wasn't shown in the diff, or finished being reviewed
<gary_poster> (not that I don't think tarmac ought to be a driver, actually)
<gary_poster> approved out-of-order is fine, and even nice to have IME.  Comes in handy for trivial branches after big honking branches.
<gary_poster> But anyway, I see your point.
<gary_poster> Make a bug. See where it goes. :-)
<dobey> well, it's good if the "trivial" branch doesn't also include the changes from the "big honking" branch. but if it doesn't, then there probably isn't much need for specifiying a prerequisite :)
<gary_poster> it did
<gary_poster> they did
<dobey> as it is now though, there's no easy way to implement a check to ensure that the 'big honking branch' lands first, to avoid having the 'trivial branch' land with unapproved code
<dobey> no?
<bryceh> I've been getting spurious warnings about LPModerate and XMLRPCRunner import errors - I am guessing I need to have something installed that isn't?  (This is lucid, dist-updated to current, with db-devel head)  http://pastebin.ubuntu.com/509048/
<gary_poster> dobey, right, which is the use case that tarmac has that I didn't have when I was doing things manually.  MPs were a way of social communication for me, rather than a mechanism for enforcement and automation, and I think that's how they started in vision.  I landed things myself, as it was appropriate to do so.  But anyway, like I said, file a bug!
<dobey> yeah i will
<gary_poster> bryceh: smells like Mailman failed to build to me.  Possible?
<bryceh> gary_poster, it's set to the stock defaults, I've not touched it in any way... does it require configuration for launchpad?
<gary_poster> bryceh: I mean, the mailman that launchpad builds itself in the Make
<gary_poster> not system mailman
<bryceh> oh, well I notice in the db-devel bzr log there's been some changes to it, but I've not made any changes
<gary_poster> bryceh: did you look at the output of make when you built this branch?  If not, try make clean && make, and see if it seems ok
<gary_poster> if so, retry tests
<bryceh> ok
 * gary_poster hopes it will then work, because he hasn't seen this problem
<bryceh> much better, thanks gary!
<gary_poster> yay, bryceh :-)
#launchpad-dev 2010-10-09
<lifeless> mars: staging resets weekly, not nightly.
#launchpad-dev 2010-10-10
<jml> hello
<wgrant> Evening.
<jml> I am sitting next to the people who are this very moment working on releasing Ubuntu
<jml> it's exciting
<wgrant> Excellent.
<wgrant> Launchpad hasn't broken yet?
<jml> not that I know of
<jml> but I'm around all day in case it does
<jml> it turns out that when you take twenty computers away from the build farm, the build queue starts to build up
<thumper> jml: good luck
<wgrant> Although this barely counts as building up.
<jml> thumper: thanks.
 * thumper is back tomorrow
<thumper> although I've been thinking I'd like longer off
<thumper> life a few years
<jml> wgrant: it's a brave new world
<jml> thumper: a few years off would be nice
<thumper> but I've got bills to pay
<wgrant> I expect some explosions from i-f-p, but I guess we'll see.
<jml> now that shipit is not the same thing as Launchpad, I think we're going to be alright
<jml> but, touch wood
<wgrant> The DB split has happened?
<jml> not afaik, but the app servers are split
<wgrant> True.
<jml> time for a cup of the brown muck that comes out of the espresso machine
<thumper> jml: how's the press ups going?
<jml> thumper: lousy
<jml> thumper: only being able to do three at a time being the root cause
<thumper> so you need to do 20 sets?
<jml> but aggravated by working in the office a lot recently
<thumper> what?
<thumper> people not dropping to do pressups all the time?
<jml> no. they seem to be worried about releasing this Ooboo-thingy
<thumper> surely they have time between builds and things :)
<jml> heh
<jml> although other office worker's are managing, it seems, so it's no excuse
<jml> s/'//
<mwhudson> i have managed about 10 so far i think
<thumper> mwhudson: I've noticed that you've not updated the wiki :)
<thumper> I was going to hassle you tomorrow about it
<mwhudson> yeah, i keep forgetting
<wgrant> Where is SSO support these days?
<wgrant> All links have mysteriously vanished.
<jml> I don't know.
<lifeless> moin
<mwhudson> good morning
<thumper> morning
<lifeless> hi thumper
<thumper> hi rob
<thumper> we are in release freeze now aren't we?
<thumper> lifeless: anything particularly interesting happen last week that I should know about?
<lifeless> yes
<lifeless> we had to rollback the branchrevision patch
<lifeless> it was going to take 3 hours to run
<lifeless> I have some ideas for how we might get it done for november
<thumper> lifeless: just the db patch?
<thumper> all the code should still run fine
<lifeless> thumper: the primary one is: create a new table, branchrevision2, which we don't query, but we insert all the same data into, and modify the code to keep totally up to date. Have a garbo job sync them up, and then during downtime rename branchrevision2 to branchrevision.
<thumper> lifeless: I'd love to hear how we can do it
<lifeless> thumper: I reverted the db-devel commit
<lifeless> thumper: if that was more than the db patch, then more was reverted
<thumper> arse
<thumper> there is much more than just the db patch there
<thumper> well
<thumper> not much more
<thumper> but more than just the patch
<thumper> the rest was test fallout
<thumper> lifeless: I plan to spend some of this cycle working out how to store less in the branchrevision table
<lifeless> thumper: have you spoken with jam yet?
<thumper> lifeless: I had planned to before I took a break but didn't
<lifeless> no worries
<thumper> I plan on catching up with him early this week
<lifeless> cool
<jml> lifeless: thanks for the review.
<lifeless> de nada
<jml> lifeless: do you want to see it again before it lands?
<jml> with any luck I'll be able to delete a couple of layers and a pile of test boilerplate from Launchpad once we're using a more recent testtools
<jml> lifeless: I'd appreciate your thoughts on https://bugs.edge.launchpad.net/testtools/+bug/657780 too
<_mup_> Bug #657780: Per-test decorator syntax for test runner <testtools:New> <https://launchpad.net/bugs/657780>
<jml> anyway, I'm off. so amazingly tired.
<lifeless> jml: will look at that bug
<lifeless> if you make radical changes
<lifeless> or if you'd like me to look at the tests in detail, then I'm happy to review it again
<lifeless> otherwise I say JFDI
<lifeless> jml: ^
<lifeless> man, its cold n wet today
 * thumper goes to make a coffee before wallyworld turns up
<wallyworld> morning
<thumper> wallyworld: morning
<wallyworld> thumper: welcome back, let me know if you want to skype sometime
<thumper> wallyworld: I do, and soonish
<wallyworld> ok, you know where i am
<lifeless> losa ping
<lifeless> losa unping
<spm> :-)
<thumper> wallyworld: https://code.edge.launchpad.net/~jameinel/launchpad/lp-service/+merge/37531
#launchpad-dev 2011-10-03
<mwhudson> the 503 page change was the one that uses flash for xdr?
<wgrant> mwhudson: Yes.
<mwhudson> boggle
<wgrant> mwhudson: To get at identi.ca/twitter feeds.
<mwhudson> oh
<wgrant> Yes, the Web is broken... but Flash is not a solution :)
<lifeless> mwhudson: wgrant: it uses flash as a fallback, not by default
<lifeless> modern browsers with javascript enabled will just work without the flash
<lifeless> however failing tests is a good reason to rollback :)
<wgrant> I wonder if those profiling changes weren't meant to land at all.
<wgrant> They seem somewhat insane and irrelevant.
 * wgrant rolls it all back.
<huwshimi> wgrant: How did you figure out which js was using mochikit? Or did you just remember?
<wgrant> huwshimi: I remembered seeing some bits of it around... grepping for it is going to be difficult :/
<lifeless> heh
<lifeless> explain select * from oops_oops;
<lifeless>                                QUERY PLAN
<lifeless> ------------------------------------------------------------------------
<lifeless>  Seq Scan on oops_oops  (cost=0.00..1922179.42 rows=27723142 width=464)
<lifeless> 27M rows.
<wgrant> Hmm.
<mwhudson> ah, i was wondering why you needed to see the query plan for that statement :-)
<lifeless> mwhudson: faster than count(*) :)
<mwhudson> different answer too, i guess?
<mwhudson> statistics vs actual
<lifeless> the stats at last update know the total live row count
<lifeless> vs instantaneous row count
 * mwhudson slams head into desk about interactions in lp's tests
<lifeless> mwhudson: welcome !
<wgrant> Sensible interfaces? What are they?
<lifeless> mwhudson: what got you onto this?
<mwhudson> https://bugs.launchpad.net/launchpad/+bug/631884 basically
<_mup_> Bug #631884: multiple variables pointing to FeatureController get out of sync easily <feature-flags> <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/631884 >
<lifeless> ah yes
<lifeless> there is another one about get_request_timeline
<lifeless> same issue
<mwhudson> yeah, features are a bit simpler, more localized
<mwhudson> so attacking them first
<lifeless> I like the sound of that
<mwhudson> and my change isn't directly related to the problem i was head desking about, or at least my change doesn't make any difference
<mwhudson> but for example, it doesn't seem we make any systematic effort to ensure that an interaction doesn't outlive a testcase
<mwhudson> also, TestCaseWithFactory sets up an interaction, TestCase doesn't
<mwhudson> which is a bit ... dunno, something
<mwhudson> anyway!
<lifeless> doesn't suprise me
<lifeless> if I may make a suggestion
<lifeless> a fixtures.Fixture to push and pop interactions would be very useful
<mwhudson> yes probably
<lifeless> s/suggestion/observation/
<mwhudson> i think also a pithy email and wiki page along the lines of "10 things every launchpad developer absolutely positively has to know about interactions and participations or else" is forthcoming
<lifeless> \o/
<lifeless> https://code.launchpad.net/~lifeless/python-oops/docs/+merge/77849
<lifeless> wgrant: mwhudson: can has review? ^
<mwhudson> oh right, profiling tests are/were broken on devel, right?
<lifeless> yes
<mwhudson> and guess what!
<mwhudson> thread local variables are involved
<lifeless> orly
<mwhudson> yrly
<mwhudson>     def assertCleanProfilerState(self, message='something did not clean up'):
<mwhudson>         """Check whether profiler thread local is clean."""
 * mwhudson chucks his feature flag thing into ec2
<lifeless> ok pgsql, thats just daft
<lifeless> or rather, like queries really really suck
<wgrant> Howso?
<lifeless> explain select pathname from oops_oops where pathname like '/srv/launchpad.net-logs/production/wampee/2010-05-24/%' order by pathname;
<lifeless>                                               QUERY PLAN
<lifeless> ------------------------------------------------------------------------------------------------------
<lifeless>  Sort  (cost=2000294.79..2000301.89 rows=2841 width=68)
<lifeless>    Sort Key: pathname
<lifeless>    ->  Seq Scan on oops_oops  (cost=0.00..2000131.82 rows=2841 width=68)
<lifeless>          Filter: ((pathname)::text ~~ '/srv/launchpad.net-logs/production/wampee/2010-05-24/%'::text)
<lifeless> possibly too-few statistics for the table size
<wgrant> Is there an index?
<lifeless> yes
<lifeless>     "oops_oops_pathname_key" UNIQUE, btree (pathname)
<lifeless> of course, its 5.5GB due to bloat.
<lifeless> wow, disk sorts. funky.
<lifeless>  explain select pathname from oops_oops where pathname like '/srv/launchpad.net-logs/production/wampee/2010-05-24/%' and date between '2010-05-24'::timestamp - '1 day'::interval and '2010-05-24'::timestamp + '1 day'::interval order by pathname;
<lifeless> seems to do whats needed.
<lifeless> we may be better offer with an IN statement or a temp table.
<lifeless> though they can degrade to seq scans horribly quickly.
<lifeless> temp table is at leat reasonabl
<lifeless> e
<lifeless> any reviewers in the house ? 8 liner ...
<mwhudson> lifeless: ah yes, my machine crashed after you pasted the link
<mwhudson> lifeless: link me up again?
<lifeless> https://code.launchpad.net/~lifeless/python-oops/docs/+merge/77849
<lifeless> thanks!
<mwhudson> done
<mwhudson> er, lunch time
<wgrant> lifeless: So, do we have any plans to fix all these timeouts?
<lifeless> wgrant: yes.
<wgrant> lifeless: Any *specific* plans? :)
<lifeless> wgrant: yes :)
 * StevenK waits for "* wgrant shakes lifeless until some plans fall out."
<wgrant> Pretty much!
<wgrant> 'cause we haven't made any progress in quite a while.
<wgrant> Apart from the batchnav thing.
<lifeless> bug search is a prime candidate for solr
<wgrant> And that's at least 18 months away.
<lifeless> thats a different discussion
<wgrant> Is it?
<lifeless> broad strokes: - tune implementations now (e.g. denorm, search schemas etc), and do a dedicated search engine if/when we reach the limits of that approach
<wgrant> We have some debilitating, embarrassing timeouts. We're making little to no progress on them.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 253 - 0:[#######=]:256
<lifeless> we have no engineers currently hacking on them
<lifeless> thats the proximate cause of no progress
<lifeless> francis is analysis *why* thats happening
<wgrant> That's been happening for months now.
<lifeless> analysing.
<wgrant> We have not made critical bug progress for 4.5 months now.
<wgrant> We adopted the new critical bug process 8.5 months ago.
<wgrant> More than half the time the new process has been working, it has been failing,.
<lifeless> .
<lifeless> yes, thats the evidence.
<lifeless> now, do you have an analysis of causes?
<wgrant> Well, the evidence points to one.
<wgrant> It's not all of it.
<wgrant> But it is something.
<lifeless> too many new criticals?
<wgrant> No.
<lifeless> <...>
<wgrant> The maintenance squad configuration.
<lifeless> please expand on that :)
<wgrant> In particular, the week Teal stopped taking criticals, the downward trend stopped.
<wgrant> http://webnumbr.com/launchpad-critical-bugs
<wgrant> We stopped taking criticals just after the middle of May.
<lifeless> the problem started earlier
<wgrant> Certainly.
<lifeless> so, that simply speaks to our ability to tread water
<wgrant> Sure, but we're no longer even close to treading water.
<lifeless> right
<lifeless> the key question is where are all the criticals coming from
<wgrant> It was clear we were in a terrible situation 3 months.
<wgrant> 3 months ago
<lifeless> until we rule out maintenance efforts, we can't even thing about things like 'add more maintenance folk'
<wgrant> How long is this going to take?
<wgrant> We are getting into a steadily worse situation.
<wgrant> And I've seen no acknowledgement that we actually have a crisis here.
<lifeless> I don't think its a run around in circles crisis
<lifeless> its a significant issue
<wgrant> What is a run around in circles crisis, then?
<lifeless> and francis is working on the analysis
<lifeless> wgrant: one where stopping and thinking will not help the outcome
<wgrant> The new definition and process are clearly not working. Reasonable functioning of the new structure of the LP team depends entirely on the critical queue being exhausted.
<wgrant> The critical queue is being the opposite of exhausted.
<lifeless> wgrant: yes, yes. Yes.
<wgrant> Therefore the new structure of the LP team will not provide reasonable response and functionality.
<wgrant> It's been nearly a year.
<lifeless> uhm
<wgrant> Something needs to be done.
<lifeless> so, lets detangle some things
<lifeless> firstly, the delta from old structure to new is *still*, even with this issue, positive: we -are- progressing both maintenance and development efforts with less scheduling headaches.
<wgrant> Hmm? For some definitions of maintenance, perhaps.
<lifeless> we haven't solved /all/ the issues we hoped to, but we have some.
<wgrant> I would posit that maintenance is, on the whole, substantially worse than it was.
<wgrant> Unless you restrict "maintenance" to "fixing timeouts and OOPSes"
<lifeless> wgrant: and escalated bugs, which previously were a real headache, we are doing them more reliably and with less delay than previously.
<wgrant> In some cases, yes.
<lifeless> wgrant: welcome to statistics.
<lifeless> now,
<lifeless> as for doing something.
<wgrant> Now, I'm not saying that the new structure is bad. I'm saying that the current implementation is objectively not working.
<lifeless> Do you propose we go off half cocked without analysing the situation ?
<wgrant> No, we need analysis.
<wgrant> But we needed analysis 3 months ago.
<wgrant> There is no analysis :(
<lifeless> francis is doing one.
<lifeless> Whats your beef with it ?
<lifeless> I posit that you want it complete yesterday.
<wgrant> Is it going to come to a conclusion?
<lifeless> of course
<wgrant> It has been several weeks.
<lifeless> of course
<lifeless> its not a trivial undertaking, and the person doing it has other things (like .e.g qbr prep) that are higher priority
<wgrant> Sure.
<lifeless> it took 4 or so months to do the analysis around the restructure
<wgrant> But it's something that needs to be done quickly, so perhaps the terribly busy person isn't the ideal person to do it :)
<wgrant> We have a serious problem here, and the longer we take to analyse it, the worse it is getting.
<wgrant> It took months for people to admit that we had a problem.
<lifeless> the scale is changing slowly, the structure isn't.
<wgrant> And now it is going to take months to analyse.
<lifeless> well, it may not take months, but its going to take more than a couple of days :)
<wgrant> It has already taken months.
<lifeless> What is your objective in this discussion? To convince me its drop-dead urgent? It doesn't seem to be to me. Important - yes. Important enough that its being done right now? Yes. Run-with-knives urgent? no.
<lifeless> the situation we're in is broadly 'too much to do and not enough hands to do it with'; thats -normal-. Whats interesting here is that we have things we believe we -must- do, which we don't have enough hands to do.
<wgrant> And the number of hands is decreasing :)
<lifeless> And that looks abnormal when you consider the definition of the things we -must- do is pretty modest [plug regressions, plug scaling issues, do escalated tasks]
<wgrant> Indeed.
<lifeless> the stuff francis has noted so far is that its mainly self inflicted
<lifeless> IIRC > 1/2 the bugs analysed so far have been caused by feature work
<lifeless> but he hasn't analysed enough to be confident that its indicative in the big picture
<lifeless> *IF* it is, then the thing to point at is cost of change: particularly hidden costs.
<wgrant> Hm, is this analysis progress visible somewhere?
<lifeless> And that will come back to code structure, which gets back to what reviews are [perhaps failing] to do. For instance.
<wgrant> I'd not heard any of this.
<lifeless> I don't know where his working notes are
<wgrant> It was not clear that any progress had been made whatsoerver.
<lifeless> I talk with him weekly
<lifeless> StevenK: ++
<StevenK> Hm?
<lifeless> nuking sampledata. Love it.
<StevenK> I'd remove all of it if I think it had any hope in hell of passing ec2.
<lifeless> oops-tools is truely painful to hack on
<lifeless> slow tests + no way (that I can see) to run just one.
<lifeless> wgrant: StevenK: I need a brief teddy bear about removing oops prefixes and the impact on fauly analysis
<lifeless> could either of you spare 10 minutes?
<wgrant> lifeless: Sure.
<StevenK> Public holiday and about to brave the daystar
<lifeless> wgrant: skype ?
<huwshimi> wallyworld: Do you know much about the names of the permission levels on the +managedisclosure page (e.g. Observer, Restricted Observer)? Are they used elsewhere in Launchpad?
<wallyworld> huwshimi: no, this is new terminology afaik
<huwshimi> wallyworld: OK, I'm just trying to figure out if there are any better labels we could come up with. Something a bit more descriptive. E.g. Restricted Observer can only view subscribed so they could be labelled "Subscriber"
<huwshimi> wallyworld: Any suggestions?
<wallyworld> huwshimi: i think we are trying to get away from conflating visibility with subscriptions
<huwshimi> wallyworld: OK, that makes sense. I think I mostly have reservations about the word "observer".
<wallyworld> it's hard to come up with a one or two word label that conveys the intended meaning
<huwshimi> yeah
<wgrant> Same with "disclosure"
<wallyworld> i think though that once the terminology becomes socialised, it will be understood
<wgrant> We're very good at using bad terminology.
<lifeless> you could just not label them.
<lifeless> and instead describe their rights
<wallyworld> perhaps we need good inline or popup help to explain what it means
<lifeless> [full access]\n* person\n* person ...
<lifeless> [access to listed objects only]\n* person\n* person
<wallyworld> labels are needed if we want to do a choice list selection widget though
<huwshimi> wallyworld: Yeah I was working on that when I decided I really wanted to think about the labels
<huwshimi> wallyworld: Yeah that's right
<lifeless> wallyworld: do we need a label for 'class of role' really?
<huwshimi> wallyworld: Or for things like "Add an Observer" links
<wgrant> "Grant access"?
<lifeless> wallyworld: or just a phrase that explains
<lifeless> right
<wallyworld> i think we need 3 things: 1. a good label, 2. a short sentence that can go in brackets after, and 3. verbose popup help text
<wallyworld> there are 2 role classes that can see stuff because of access being explicitly granted, vs one access class that can implicitly see everything because of the project role
<wallyworld> so that makes it a bit harder to come up with concise labels
<lifeless> there are?
<wgrant> "Project role => access" is a myth that I hope to dispell soon.
<lifeless> sorry, thats opaque
<wallyworld> observer and restricted observer are granted access roles
<lifeless> wallyworld: so is e.g. driver, and owner.
<wallyworld> but if you are a project owner, then you can see stuff
<lifeless> wallyworld: for performance we almost certainly want the owner etc in the one system, with checks that they are in sync.
<wallyworld> but if you register a project, you are the owner without it being granted
<wallyworld> ie visibility is implicit in that case
<lifeless> wallyworld: there is a difference between 'the user does not click to grant it' and 'it is not granted'
<wallyworld> i know ownership can be assigned
<wgrant> I think it should be the default, but it should not be implicit.
<wallyworld> lifeless: agree on the difference
<wallyworld> but if i register a project, i want to implicitly be able to see everything by default
<lifeless> wallyworld: its my understanding that owners etc won't be checked during lookup: if its not in the table, access denied.
<lifeless> wallyworld: no, you want to see everything by default. The word implicit isn't needed for the use case :)
<lifeless> wallyworld: I am asserting that your needs are met by:
<lifeless>  - defaulting project owner to full access (much like we default team owners to administrator status)
<wallyworld> lifeless: the balsamiq mock ups imply implicit visibility due to project roles
<lifeless>  - possibly having a rule that the owner must retain full access
<wgrant> Old mockups are not gospel :/
<wallyworld> sure, but they are the current starting point
<wallyworld> if we decide to do things differently so be it
<wgrant> Mm, I think they are hints.
<wgrant> Nothing more.
<wallyworld> sure
<wallyworld> now we have an interactive demo based on the mockups  which can be used to try and help clarify these sorts of requirements
<wgrant> Yep.
<lifeless> anyhow, my point is this: you are saying its complex to describe because you have multiple implicit accesses + granted access.
<wallyworld> yes, that's my current understanding
<lifeless> one way to address that, and a way I would encourage for reliability and performance in the system, is to drop implicit access. Instead use business rules to populate granted access when something should be automatic.
<wallyworld> no problem with that, i agree
<wallyworld> but we may still want to distinguish to the user that is the case
<wallyworld> ie someone has access due to automatic propogation based on business rules
<wallyworld> rather than an explicit grant of access by an authorative user
<lifeless> thats an interesting case
<wallyworld> so that if someone needs to be removed from the bug supervisors team, they ca nbe'
<lifeless> I think that the authoritative user still exists (they added someone to the role)
<wallyworld> what about the case where a team is assigned as bug supervisor.
<wallyworld> and someone in the team should be cut iff
<wallyworld> cut off
<lifeless> and the grant is a consequence of being added to the role - but that doesn't imply revoking when removed from the role
<wallyworld> but not the whole team
<lifeless> wallyworld: I don't think we should support that
<lifeless> wallyworld: let me be more clear: supporting that will be extremely hard.
<wgrant> You can't just remove them from the team?
<lifeless> wallyworld: either remove them from the team, or use a team that accurately models the supervisors desired.
<wallyworld> wgrant: yes, but you need to see why and how someone has acess so you know what you need to do to remove visibility
<wallyworld> lifeless: see above
<lifeless> wallyworld: so in this situation,it would show 'user FOO has access because they are in team BAR which has FULL ACCESS'
<wgrant> wallyworld: Sure, you'll see that they're a member of Ubuntu Bug Control, and that team is granted access.
<lifeless> I presume there is a free text field related to each observer, for explanations ?
<wallyworld> not that i am aware, but i could be wrong
<lifeless> e.g. 'fred is auditing us for iso9008:2000 status'
<wallyworld> that may be something to add
<wallyworld> if it is not already there
<lifeless> that would supply a way to explain business rule added access too - 'BAR added when set as bug supervisor'
<wallyworld> lifeless: with your example above, it may be that we want to say ".. in team BAR which has FULL ACCESS because team BAR is the bug supervisor"
<wgrant> Of course, ideally all access control would be managed through this view...
<lifeless> that isn't enough to support automatically removing BAR when changing bug supervisor (but that may be undesirable anyway)
<wgrant> So it would show the bug supervisor privilege for that team as well.
<lifeless> wallyworld: understood, I can't comment about whether we need that or not :)
<lifeless> wallyworld: I will comment that for teams we started with implicit stuff for owners
<lifeless> wallyworld: and have slowly, to great rejoicing, removed it.
<lifeless> wallyworld: so my *hunch* is that it will be a problem.
<wgrant> As Curtis and I discussed last week about bug supervisors.
<wallyworld> i agree 100% that the model will be much simpler to have one access lookup meachanism, and that owners etc should be put there via business rules
<wallyworld> but i think we need to record that, is all
<wallyworld> so we can show it to the users
<wallyworld> to explain *why*
<wallyworld> someone has access
<wgrant> You might recall that we discussed this on maybe Wednesday's call. The thought was that ideally the roles would not be conflated at all, but the initial project setup wizard would guide you through setting probably the same people into each role.
<wgrant> The explanation only needs to exist if the roles are configured in separate places and somewhat conflated.
<wgrant> Ideally they would be separated.
<wgrant> And manageable from the same place.
<wgrant> So it's obvious what's what.
<wgrant> Rather than having the bug supervisor on one page, security contact on another, and then a huge list of (restricted) observers.
<wgrant> All separated.
<wallyworld> yes, i recall. but relying on the setup wizard is not enough.
<wgrant> With no single view of access control.
<wallyworld> they would be separated
<wallyworld> not
<wallyworld> all users with access would be shown on one page
<wallyworld> regardless of how they got that access
<wgrant> I speak of more than just project-wide observer access
<wgrant> The same page really should list everyone with bug supervisor access, security contact membership, etc.
<wgrant> It probably won't in the first iteration.
<wgrant> But we should certainly not design ourselves into a corner.
<wallyworld> you mean for a given project though?
<wgrant> Yes.
<huwshimi> ... and this is why I should just JFD things instead of asking questions on IRC :P
<wallyworld> and that's what the mockups do now
<wgrant> wallyworld: Hm?
<huwshimi> wallyworld: Is "member" a confusing word to use?
<wgrant> wallyworld: They list everyone *with observer access through those roles*.
<wgrant> wallyworld: They don't take into account the privileges that those roles actually exist to grant.
<huwshimi> wallyworld: Actually on second thoughts I guess it probably is
<wallyworld> huwshimi: i can't think of a better term now
<wallyworld> wgrant: what page mockup are you looking at?
<huwshimi> wallyworld: Better than "Observer"?
<wgrant> wallyworld: None.
<wallyworld> huwshimi: maybe i don't have the context - you talking about team members?
<wallyworld> wgrant: when you said "they list everyone.....", what does "they" refer to then?
<wgrant> wallyworld: The last version of the mockups that I saw.
<huwshimi> wallyworld: definitely confusing then... I'll try and come up with something else
<wgrant> They only deal with observer access.
<wgrant> Not any other kind of access, like, say, bug supervisor write privileges.
<wallyworld> wgrant: and restricted observer, and access via a project role
<wgrant> They're just different paths to the same privilege.
<wallyworld> not really? restricted observer != observer
<wgrant> Restricted observer is the same sort of privilege, just over a smaller scope.
<wgrant> The privilege is visibility.
<wgrant> There are other privilegs.
<wallyworld> like ability to assign/triage bugs?
<wallyworld> you mean?
<wgrant> Right.
<wgrant> Which is what bug supervisor really means.
<wallyworld> sure, but that stuff we didn't do for this first cut
<wgrant> Indeed.
<wgrant> But it's very relevant for if you want to describe why someone has a permission.
<wgrant> Because then their access management page will say that they are the owner and an observer.
<wallyworld> yes, so there is a project role filter
<wgrant> Rather than just an observer because they are the owner.
<wallyworld> i think we sort of agree
<wgrant> So, your explanation suggestion is only needed because A) I haven't convinced you yet that project roles have nothing to do with observer privileges, except for being sensible defaults during the setup wizard; and B) the UI that is in scope right now will only cover observer privileges.
<wgrant> If A becomes no longer the case, then "PRIVILEGE because ROLE" is no longer the case.
<wgrant> If B becomes no longer the case, then the team's access management page will list ROLE anyway.
<huwshimi> wallyworld: So is an Observer part of a team or are they just directly connected to a project somehow?
<wallyworld> huwshimi: an observer is a person or a team who have been granted access to view a project's private artefacts
<wallyworld> so a person may be an Observer via their team membership
<wallyworld> huwshimi: a Restricted Observer can just see private bugs and branches
<wallyworld> huwshimi: an Observer can also see milestones, blueprints etc etc
<wallyworld> make sense/
<wallyworld> ?
<lifeless> wallyworld: thats temporary
<huwshimi> wallyworld: But an Observer can also see a projects private artefacts without being part of a team that gives them that permission right?
<lifeless> wallyworld: long term restricted will be able to be granted milestones blueprints etc
<lifeless> wallyworld: there is no intention to arbitrarily limit the things restricted observers can be given access to see
<wallyworld> ok
<wallyworld> huwshimi: yes, a person may be individually made an observer
<huwshimi> wallyworld: OK great thanks
<lifeless> huwshimi: or a team they are in (standard LP idiom there)
<wallyworld> lifeless: i said that above :-)
<lifeless> ah ll
<lifeless> kk
<wallyworld> lifeless: i assume the long term then is to allow projects to define what 'restricted' means
<lifeless> wallyworld: I don't know what you mean
<wallyworld> lifeless: so you said "restricted = bugs and branches only" is temporary
<wallyworld> and that later
<wallyworld> restricted would be gratned access to blueprints etc
<lifeless> restricted would *permit* granting access to blueprints etc
<lifeless> the limitation to bugs and branches is an implementation limit
<wallyworld> yes, that's what I was asking, thanks
<wallyworld> just wanted to be sure i understood
<lifeless> AFAIK there is no intent for restricted to ever change its meaning: you will always need restricted *to a concrete thing*
<lifeless> its just that the things we can put in the concrete list will expand as we implement more
<wallyworld> yes, that's how i understood it from what you said
<lifeless> kk
<lifeless> we're synced then :)
<wallyworld> thanks
<wallyworld> for now until more people start to discuss it :-)
 * wallyworld runs off tp pick up kid from school
<huwshimi> wallyworld: haha
<lifeless> can has review? https://code.launchpad.net/~lifeless/oops-tools/nomoreprefix/+merge/77858
<lifeless> wgrant: diff is there, if I can tempt you :)
<wgrant> lifeless: prefix.upper?
<wgrant> That seems unnecessary, and will break eg. SSO.
<wgrant> Ah.
<wgrant> It turns out that cloning a container works better if you don't have the MAC overridden in the config to a constant value.
<lifeless> wgrant: its moved lines, it is necessary
<wgrant> lifeless: Ugh, so it is. That's horrible.
<wgrant> lifeless: Can we try using a sensible one first?
<wgrant> Then fall back to the uppercase one.
<wgrant> Then create one in sensible case.
<lifeless> wgrant: thats a rabbit hole and orthogonal
<lifeless> wgrant: e.g. searches probably do upper() at the moment.
<lifeless> I'd love to fix oops tools up in general, but I don't fancy cascading tweaks when working on a different focus
<wgrant> lifeless: Yeah, I get what you're doing now.
<wgrant> (sorry, lightdm decided that sharing was caring, so it donated lots of my keystrokes to the underlying getty)
<lifeless> :)
<lifeless> win
<lifeless> wgrant: I take it my cover letter wasn't helpful enough ?
<wgrant> lifeless: I don't understand your req_vars and statements changes in models.py, or your second hunk in oops.txt.
<wgrant> Was it creating a duplicate QSMMX?
<lifeless> higher up in oops.txt it calls load_oopses
<lifeless> and loaded them all
<lifeless> the sameple QSMMX oops was previously silently ignored (due to the prefix not existing)
<wgrant> Ah!
<wgrant> Of course.
<lifeless> so this change makes it get loaded by that call (line 8 I think)
<lifeless> the req_vars and statements changes are future proofing for oopses from scripts that don't have req_vars, or statements.
<wgrant> So they're unrelated?
<lifeless> snuck in by testing with a minimal plausible oops.
<wgrant> k
<lifeless> totally.
<wgrant> Thought I must have been missing something.
<lifeless> nah, I failed to write that up. mea culpa
<wgrant> Approved.
<lifeless> thanks!
<wgrant> Oh right, HA librarian tonight.
<lifeless> btw
<lifeless> want something scary?
<wgrant> I am terrified.
<lifeless> oops-tools will discard oopses if rsync write them to disk out of order & loading runs during the rsync
<lifeless> out of order being what ext3 dirindex does, of course.
<wgrant> Heh.
<wgrant> "discard" meaning "not see", or actually delete?
<lifeless> not load
<wgrant> Right.
<lifeless> if there are 10 oopses when it finishes a run, but they are not range(1,11), then when it backfills, the optimiser which records len() will skip the early numbers
<wgrant> Yep.
<lifeless> and this is why I was looking at getting set data from the db :)
 * StevenK hopes his last ec2 run finishes quickly.
<StevenK> Given it started first, it's a little worrying.
 * nigelb wonders if buildbot will also be public.
<StevenK> I'm hoping for our CI infrastructure to be public, be it buildbot or jenkins.
<StevenK> But I'd prefer the latter.
<bigjools> good morning all
<nigelb> Good morning bigjools
<lifeless> bigjools: 'lo
<bigjools> howdy hackers
<lifeless> bigjools: hey, so what was wrong with that backported option ?
<bigjools> lifeless: it's awful
<bigjools> for starters, referring to python functions on a command line option is evil
<lifeless> with you so far
<bigjools> 2nd, you have to hard code the log name in the function. I wrote code to work around that but again it's evil
<lifeless> urgle, so you can't pickup the twistd option normally?
<bigjools> in the end --logfile=/dev/null is much easier :)
<bigjools> nope, remember that there's 2 sets of options
<bigjools> one for twisd, the others for the plugin
<lifeless> bigjools: ah, headdesk. Separately... I closed the bug you filed, but wanted to touch base with you to make sure you understand why
<bigjools> twistd*
<bigjools> which bug?
<lifeless> on oops-twisted
<bigjools> not read my email yet
<bigjools> oh, the fallback
<lifeless> yeah
<bigjools> I must admit, that way of doing it surprised me
<lifeless> because twisted's log mechanism sends to all observers, having one observer be a log file and another be the oops system causes failures to be logged twice - once in the log as a default thing, and once as an oops
<lifeless> so you have to have only one observer - the oops one, and have it *translate* the failures for logging as 'oops id XXX created'
<bigjools> that's not how I expected it to work either
<bigjools> I was expecting that the oops observer would *only* add the oops info
<bigjools> why does it need to translate?
<lifeless> Yeah, I can see how you might expect that - but can you see how its constrained by the broadcast setup they have ?
<bigjools> not really :) but I am listening
<lifeless> ok, so the default is to log the backtrace
<bigjools> yup
<lifeless> 30 or 40 lines or whatever.
<lifeless> we don't want that, we want to log 'created oops id XXX ValueError: foobar'
<bigjools> why don't we want that?
<lifeless> because it destroys readability of the log, and the exception and all its data is in the oops
<bigjools> I'd say that was subjective, but ok, it would be consistent with what LP does already
<lifeless> of course, if oops reporting is disabled, we want the traceback
<lifeless> I'll agree that its subjective
 * bigjools 's inner KDE user is screaming "make it configurable!"
<bigjools> anyway it all works hunky dory now
<lifeless> ok great
<bigjools> if we get time we'll add another buildout recipe to wrap all this up into a nicer single script
<lifeless> \o/
<bigjools> since the "--logfile /dev/null txlongpoll" is always required
<lifeless> headdesk. We really should lock down old fixed bugs.
<lifeless> (bug 7 comment..)
<_mup_> Bug #7: Need help for novice translators <lp-translations> <Launchpad itself:Fix Released by danilo> <Launchpad Documentation:Fix Released by matthew.revell> < https://launchpad.net/bugs/7 >
<nigelb> *headdesks*
<bigjools> at least it's not pornspam
<nigelb> although the more fun one was  - "Dear bug, I'd like to add you to my network"
<lifeless> \o/
<lifeless> wgrant: care for a follow on review to the prior ?
<lifeless> wgrant: https://code.launchpad.net/~lifeless/oops-tools/outoforder/+merge/77887
<lifeless> the diff is there, the warning is bogus
<wgrant> lifeless: Not right now, sorry.
<lifeless> who else is around, not on leave, and a reviewer....
<lifeless> oh look, stub.
<stub> shhh.....
<stub> lifeless: private branch?
<lifeless> stub: private project, pending some feedback I've asked for today from various folks
<lifeless> stub: open sourcing it should happen soon, when we will blind anyone that reads it ;)
<lifeless> think men in black, done via the code.
<lifeless> stub: you should have access though
<bigjools> lifeless: your change to the title on https://bugs.launchpad.net/launchpad/+bug/297471 is not English!
<_mup_> Bug #297471: PPA upload notifications always and only go to the signer <email> <lp-soyuz> <oem-services> <ppa> <Launchpad itself:Triaged> < https://launchpad.net/bugs/297471 >
 * bigjools fixorates
<wgrant> bigjools: It was fine!
<stub> lifeless: Do we cope with partial OOPS files now, such as an rsync in progress or being killed midrun?
<wgrant> bigjools: They always go to the signer, and they only go to the signer.
<bigjools> wgrant: well I misread it a few times, so no it wasn't
<bigjools> the English is awkward
<wgrant> Heh
<lifeless> stub: rsync does a tempfile + rename
<lifeless> stub: so we ... probably do see the tempfile, which will barf a oops parsing error, which is silently caught and discarded
<stub> We might do it that way...
<nigelb> wgrant: I just found out the reason for Ubuntu Mono being smaller :|
<wgrant> nigelb: Oh?
<stub> lifeless: Yer, but now when we rescan the file that previously barfed should complete.
<nigelb> wgrant: bug 727733
<lifeless> stub: yes
<lifeless> stub: it takes 3.5 seconds to get 10K rows (for a moderately busy day, one appserver) out of the db on carob
<stub> Which we do every 10 minutes, as that is how often the files get synced
<wgrant> nigelb: Hmm.
<lifeless> well, no, there is a cronscript, and it walks all active (cronscripts + appservers) at once
<lifeless> but -most- dirs are much much much smaller
<lifeless> much
<lifeless> and a rabbit injector won't need to walk dirs at all
<lifeless> (which is what this is all leading up to)
<nigelb> wgrant: Launchpad now looks much smaller and there's no clean way I can think of to fix it that falls back gracefully.
<wgrant> nigelb: Yeah, it's pretty small :(
<nigelb> by 2 pts! :(
<wallyworld_> stub: there was going to be a restore of the qas db from prod over the weekend i think. Do you know if it succeeded?
<wgrant> wallyworld_: It did.
<wgrant> wallyworld_: I finished it off with bradm this morning.
<wallyworld_> wgrant: cool. i'll land that db patch
<wallyworld_> thabnks
<wgrant> It is now nice and shiny from a Thursday or Friday backup.
<stub> If landing a db patch was blocked on refreshing production data, does that mean we are not running garbo jobs on qastaging? Or was it a manual fix of production data?
<wallyworld_> stub: we were running garbo but i landed a branch to delete it a bit too soon. cause i didn't realise we ran garbo on qas
<wallyworld_> i thought we used a db restore to update qas
<wallyworld_> i think until recently we did indeed not run garbo on qas?
<stub> We do use a db restore to update qas. If we are not running garbo jobs on qas, we should be.
<wgrant> We do run them on qas.
<wallyworld_> stub: we are running them now. but i thought we weren't
<stub> yup
<wallyworld_> i *think* it was a recent server upgrade which know allows us to
<stub> I dunno what was.
<wallyworld_> now
<wallyworld_> or at least that's what i think i was told
<wgrant> garbo's been running for a while.
<wgrant> But most other stuff doesn't.
<wallyworld_> ok
<lifeless> stub: thanks!
<bigjools> wgrant: I am starting to look at bug 34086, if you have any thoughts already let me know
<_mup_> Bug #34086: removal of arch-all packages while there are arch-specific packages dependent on it results in uninstallable binaries <escalated> <feature> <lp-soyuz> <rls-mgr-o-tracking> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/34086 >
<wgrant> bigjools: Urgh.
<bigjools> indeed
<bigjools> when I say "starting to look" I mean evaluate what can be done
<wgrant> This is pretty related to the sources thing.
<bigjools> similar
<wgrant> We should probably approach them in similar ways.
<bigjools> the key part is working out what the dependent packages are
<wgrant> Well.
<wgrant> There are two ways.
<wgrant> I mean, two ways to approach this.
<wgrant> 1) Don't remove packages if something is dependent on them.
<wgrant> 2) Remove all binaries for a source and arch at once.
<wgrant> Method 1 is just about guaranteed to drive you to insanity and terrible, terrible code.
<wgrant> And terrible, terrible cruft.
<wgrant> And won't catch other uninstallability cases.
<wgrant> So is mostly pointless.
<wgrant> 2 is easy, has been listed as a TODO since 2005, and I believe it fixes the important cases here.
<bigjools> yes
<bigjools> "Need to make binaries go in groups" is the comment in the code :)
<wgrant> So, talking about dependent packages is probably going to lead you nowhere good.
<wgrant> Best to ignore that line of investigation, I think.
<bigjools> well, no
<bigjools> I am considering all avenues, we'll use the best one
<wgrant> Well, I consider 1 to be not much better -- but significantly harder -- than 0) Don't ever remove anything at all.
<bigjools> you don;t understand my point
<bigjools> anyway, lunch calls
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 253 - 0:[#######=]:256
<nigelb> Is jtv off/away today?
<jelmer> nigelb: Yep
<nigelb> aha, that explains that
<rvba> Hi benji.  Would you be kind enough to mentor 2 of my reviews? (I've done the first branch and I'm currently reviewing the second one)
<benji> rvba: sure, what can I do fo you?
<rvba> benji: Thanks, here is the first one: https://code.launchpad.net/~allenap/launchpad/longpoll-associate-consumer-not-now/+merge/77894
<jelmer> benji: hi
<benji> hi jelmer
<jelmer> benji: I've got a fairly long merge proposal that's already been reviewed by a couple of others. I think I've now adressed all issues that were raised, can you have another look at it?
<benji> jelmer: sure
<jelmer> it is https://code.launchpad.net/~jelmer/launchpad/bzr-code-imports-ui/+merge/73153
<jelmer> benji: great, thanks :)
<rvba> benji: Here is the second review if you still have time to mentor it https://code.launchpad.net/~allenap/launchpad/longpoll-finish-read-only-request/+merge/77915
<benji> rvba: sure
<rvba> ta
<allenap> rvba, benji: Thank you for the reviews :)
<benji> np
<deryck> abentley, I saw you marked bug 758896 OPINION.  Have we started using this status for ourselves now?
<_mup_> Bug #758896: Checklist items and flashes send mixed messages <exploratory-testing> <upstream-translations-sharing> <Launchpad itself:Opinion> < https://launchpad.net/bugs/758896 >
<abentley> deryck: I think OPINION describes the status better.  When jml was product strategist, his opinion had more weight, but I think he was wrong then, too.
<deryck> abentley, well, OPINION is a closed bug though.  I don't think this bug should be closed.
<deryck> abentley, OPINION says, "closing this bug acknowledging we have a difference of opinion."  I don't think you mean that, do you?
<abentley> deryck: No, I meant "We have a difference of opinion about whether this is a bug".
<abentley> deryck: The implication that this is a closed bug escaped me.
<deryck> abentley, hmmm, so you're reading of the status isn't far off, but I think it being closed means it shouldn't be an opinion bug.
<abentley> deryck: Are you saying the opinion status shouldn't imply a closed bug?
<deryck> abentley, no, I'm saying opinion status shouldn't be used here because it is a closed bug.
<deryck> sorry, I wasn't clear
<abentley> deryck: it was a triaged bug, not a closed one.
<deryck> abentley, again, not being clear, sorry. :)  please change it back to triaged so the bug will not be closed. :)
<deryck> I don't think we should close this bug.
<abentley> deryck: I follow.
<deryck> abentley, but I wanted to understand your thinking and not come along and change it back without chatting with you about it.
<abentley> deryck: ISTM that opinion is actually neither open or closed.  It is "closed to some, open to some, depending on their opinion".
<deryck> abentley, yes, no one is clear on what OPINION means as a status.
<jelmer> I've only seen "Opinion" being used as a polite format of "Won't Fix".
<jelmer> s/format/form/
<abentley> jelmer: That kind of applies here.
<benji> jelmer: I just finished with https://code.launchpad.net/~jelmer/launchpad/bzr-code-imports-ui/+merge/73153.  I included a lint report for you.
<jelmer> benji: Thanks!
<benji> np
<jelmer> benji: any chance you can repeat your comment and vote at https://code.launchpad.net/~jelmer/launchpad/bzr-code-imports-ui/+merge/77961 ? I resubmitted the MP while you were reviewing, it seems (no new revisions).
<benji> jelmer: done
<jelmer> benji: Teh awesome. Thanks again.
<deryck> Hi, benji.  I have a branch for review if you're available.
<benji> deryck: sure
<deryck> benji, thanks!  https://code.launchpad.net/~deryck/launchpad/target-series-bug-834082/+merge/77966
<deryck[lunch]> benji, I'm lunching early but here if you have questions about that branch.
<benji> deryck[lunch]: ok
<jcsackett_> sinzui: you around/free to chat?
<sinzui> I am
<benji> deryck[lunch]: your branch looks good
<deryck[lunch]> benji, awesome, thanks!
<sinzui> jcsackett_, https://bugs.launchpad.net/launchpad/+bug/865467
<_mup_> Bug #865467: bug task missing from affects table because bug supervisor is not subscribed <disclosure> <regression> <Launchpad itself:Triaged by jcsackett> < https://launchpad.net/bugs/865467 >
<sinzui> jcsackett_, bug 375331 is a similar permission issue. bug 113825 describes the ambiguity in the ui. I doubt the later is in scope, but it does help explain the confusion about this issue
<_mup_> Bug #375331: Some extra filebug options don't show up for projects not having a bug supervisor <bugs> <projects> <Launchpad itself:Triaged> < https://launchpad.net/bugs/375331 >
<_mup_> Bug #113825: Null bug supervisor is confusing <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/113825 >
<barry> gary_poster: ping
<lifeless> morning
<nigelb> woah
<nigelb> Its not that late yet.
<nigelb> lifeless: early start?
<lifeless> 0734
<lifeless> a little late
 * nigelb hasnt seen 6 am to 8 am in a while :P
<gary_poster> barry, hi!
<barry> gary_poster: hi!  i *think* i've answered my own questions, re: autoppa and doing a storm release.  or let's say, i've worked around my issues :)
<gary_poster> barry, ok cool-ish :-)  sorry wasn't around
<barry> gary_poster: no worries.  at least it's python code so i can always trace what it thinks it should be doing and then make it dtrt :)
<gary_poster> :-) cool
<lifeless> flacoste: hi; call timeish ?
<lifeless> sinzui: medium bugs down to < 1 page
<sinzui> fab
<flacoste> lifeless: sure
<flacoste> lifeless: skype me when ready
<mwhudson> hm, bin/retest appears not to work with ec2 mails any more?
* benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 253 - 0:[#######=]:256
 * mwhudson smacks head again
<mwhudson> why do the feature flag context manager thingies install a new feature controller?  afaics it's only because of the caching
<mwhudson> that FeatureController does
<cjwatson> hmm, so I was wondering about an issue relating to writing the Debian->Ubuntu mass-sync API script on top of the new Archive.copyPackages method
<wgrant> What in particular concerns you?
<cjwatson> when we did this with 'sync-source.py -a', the syncs showed up as having been performed by "Ubuntu Archive Auto-Sync <archive@ubuntu.com>"
<cjwatson> this was generally a good thing - crediting to the admin who performed the auto-sync would give them way too much credit and end up with people assuming they knew things about particular packages and such
<wgrant> Yep.
<cjwatson> I'm wondering how we can do that with the new method, given that:
<cjwatson> (a) bug 827555 isn't done yet
<_mup_> Bug #827555: native syncs have no way to indicate sponsorship <derivation> <Launchpad itself:Triaged> < https://launchpad.net/bugs/827555 >
<cjwatson> (b) the ~katie user apparently no longer exists?
<wgrant> katie is suspended. Still exists, but can't be browsed to.
<StevenK> And she is still a celebrity, right?
<cjwatson> (c) I'd prefer auto-syncs to be performable by any archive admin without having to pass around credentials for an LP account
<wgrant> Still a celebrity, unfortunately, yes.
<cjwatson> (yes, I should have thought of all this ages ago)
<wgrant> cjwatson: Indeed. I don't think Red was really aware of how syncs are used.
<cjwatson> I don't want to apportion blame ...
<wgrant> Some could say that this is a problem that comes from having random people develop software like this :P
<cjwatson> I'd rather solve the problem
<lifeless> cjwatson: so, one option is to implement (a)
<cjwatson> right
<wgrant> I think that's the best option.
<lifeless> cjwatson: I suspect its tractable
<cjwatson> if people think that's the best option, and if it doesn't have too many tentacles, I'd be happy to have a go
<cjwatson> I wouldn't mind advance advice on the likely tentacles, though
<lifeless> avoid the blue ones, they itch.
<cjwatson> so we'd then basically do copyPackages(person=lp.people['katie'], ...), yes?
<lifeless> so things to think about
<lifeless> should there be limits on who you can credit ?
<cjwatson> The last time I thought about that, I decided it wasn't a problem since any uploader can already write random crap into the changelog trailer line if they like, and this should be just the same.
<lifeless> cjwatson: I think you'd want to lookup the person by email rather than hardcoding katie in there.
<cjwatson> I was abbreviating, but OK :)
 * StevenK kicks HE and Fremont 1.
<wgrant> StevenK: v6 is working fine.
<lifeless> uhm, you will need to care for backwards compat when you change the job metadata (because we don't rollout atomically)
<wgrant> v4 seems to still not have routes.
<StevenK> wgrant: Indeed.
<StevenK> Which is how I'm connecting.
<wgrant> But v6 was down for ages too :(
<cjwatson> lifeless: sorry, is that to me?
<lifeless> other than that, I presume the output fields already exist
<lifeless> cjwatson: yes
<lifeless> cjwatson: jobs run via a json blob of instructional metadata
<wgrant> So, this whole thing is pretty awful.
<cjwatson> I hadn't got as far as looking at the existing schema
<cjwatson> ah
<wgrant> Because it's only partially implemented.
<StevenK> PCJs is .. sort of horrible
<wgrant> Build notifications etc. still only take the SPR's properties into account.
<StevenK> But it's better than what we had before.
<wgrant> It's not.
<wgrant> It's the same as we had before, just for the third time.
<lifeless> cjwatson: that persists in the db between submission and completion, so changes to the code need to assume (A) old submissions are yet to be processed and (B) new submissions may get picked up by an old processor
<cjwatson> compat both ways, ok
<StevenK> At least it isn't delayed copies madness.
<lifeless> cjwatson: does the output field you want to put the sponsor into already exist ?
<lifeless> cjwatson: if so I think that that is the entire list of tentacles
<StevenK> I doubt it.
<cjwatson> it should do
<wgrant> It doesn't.
<wgrant> It's not clear what the sponsor actually means.
<cjwatson> I think it should act the same as citing somebody else's name in the changelog and then signing it yourself
<StevenK> There is requestor, but not sponsor
<wgrant> What does it mean? :)
<lifeless> well, how does the older sync do its job ?
<cjwatson> why isn't that the same?
<wgrant> cjwatson: Heh heh heh.
<wgrant> cjwatson: That's implemented as properties of the SPR.
<cjwatson> lifeless: the older sync is admin-only and can pretend to be whoever it likes
<wgrant> SPR.creator and SPR.signing_key
<cjwatson> wgrant: oh right
<cjwatson> and of course now we have the same SPR as Debian
<wgrant> cjwatson: This should have been fixed as part of DDs, but was conveniently swept under the carpet.
<StevenK> Indeed.
<wgrant> So much so that until last week we didn't even track who did syncs.
<cjwatson> bugger.  then this is actually quite hard.
<wgrant> Yes, it needs new modelling.
<lifeless> ok, so a schema change as well, and a little code to treat NULLs there as the current behaviour. New uploads to set the field, and the sync change to let it be controlled.
<wgrant> Schema changes are cheap now, but we need to actually think about it.
<cjwatson> I might just have to keep using sync-source.py -a for a while in P
<lifeless> cjwatson: doing the schema change is now straight forward
<cjwatson> which I'd hoped not to do, but isn't the end of the world
<wgrant> (and because it wasn't done as part of the project, we can't now have a simple, coherent model... we have to hack this together on top of existing crap :D)
<StevenK> If it's just a JSON field, it requires no schema change.
<wgrant> StevenK: We can record it in the job.
<wgrant> But that's all.
<cjwatson> lifeless: right, but trying to do data remodelling at the same time as an Ubuntu release may be a little beyond my cognitive capacity
<wgrant> And recording it in the job isn't very useful.
<StevenK> Hopefully having MOTUs and core devs do their own syncs hasn't been horrible.
<cjwatson> that's been largely fine
<wgrant> StevenK: Depends on your definition of horrible...
<StevenK> Hah
<wgrant> It seems to be working OK now.
<cjwatson> some teething trouble but the basic facility is great
<wgrant> And we even have some form of auditing as of a week ago.
<cjwatson> can't switch over completely to it until sponsorship is there though
 * cjwatson gets dragged off by toddler, brb
<wgrant> sinzui: No call?
<jelmer_> has anybody seen spurious failures in lib/lp/registry/browser/tests/poll-views_0.txt ?
<wallyworld_> sinzui: wgrant: unity issues, be on call soon
<wgrant> jelmer_: I haven't.
<sinzui> my audio appears to be knackered again
<wgrant> sinzui: I think we heard you for a moment.
<StevenK> I did.
<jelmer_> wgrant: guess it might be something else, I'll look further
<sinzui> StevenK, 853710
<sinzui> StevenK, bug 853710
<_mup_> Bug #853710: Teams that have ever had a PPA cannot be renamed <Launchpad itself:Triaged> < https://launchpad.net/bugs/853710 >
#launchpad-dev 2011-10-04
<mwhudson> what: http://paste.ubuntu.com/701996/
<mwhudson> haha uh, why have i had an ec2 t1.micro running since march?
<StevenK> Haha
<StevenK> Aren't they effectively free now?
<mwhudson> only for new signups i think?
<mwhudson> anyway it hasn't been costing me very much or i'd have noticed within a month i guess :-)
<mwhudson> sigh, 4 attempts to get ec2 test running :/
<mwhudson> (merge failure, odd ssh failure, local net glitch)
<StevenK_> wgrant: Didn't we remove polls?
<wgrant> StevenK_: Yes, but they were revived.
<lifeless> we tried
<wgrant> Because Ubuntu use them.
<StevenK_> Really? They haven't been used for the last few elections
<nigelb> Morning
<nigelb> wgrant: Ubuntu uses them for what?
<lifeless> StevenK_: ubuntu womens
<lifeless> IIRC
<nigelb> Most elections are condorcet
<nigelb> Ubuntu Women also used that, not LP polls
<StevenK> So perhaps we should ask Ubuntu again?
 * StevenK 
<StevenK> Sigh
 * StevenK searches for a single word that means 'person or team'
<wgrant> person
<wgrant> Or Person.
<mwhudson> so, feature flags
<StevenK> wgrant: IE, I want a class name for the refactored code.
<mwhudson> err hm
<mwhudson> why does LaunchpadTestRequest set .features to NullFeatureController?
<mwhudson> i guess LaunchpadTestRequest is used in some tests that don't have a db connection, perhaps
<lifeless> StevenK: Atom
<wgrant> lifeless: Any idea what's going on with the lazr.restful OOPS fix?
<lifeless> ECONTEXT
<wgrant> lifeless: The one you mailed IIRC benji about a week or so ago.
<wgrant> The __traceback___ one.
<lifeless> I thought he fixed it already
<lifeless> he acked it and made a branch immediately
<wgrant> Yeah.
<wgrant> But it apparently never landed.
 * wgrant hunts.
<wgrant> https://code.launchpad.net/~benji/launchpad/bug-854695/+merge/76285
<stub> wallyworld_: I don't think you want the trigger to initialize transitively_private in the BEFORE trigger, as it will be overwritten in the AFTER trigger anyway. Just 'ALTER TABLE Branch ALTER COLUMN transitively_private SET DEFAULT TRUE;' should be fine?
<wallyworld_> stub: sounds better. for some reason i thought we frowned upon default values for non null columns though?
<wallyworld_> and i think the default needs to be false
<wgrant> False is bad.
<stub> We frown on adding a new column with a DEFAULT, as that causes all rows in the table to be rewritten with the new DEFAULT.
<stub> Adding a DEFAULT to an existing column is fine
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 255 - 0:[#######=]:256
<wallyworld_> stub: ok
<wallyworld_> stub: if the trigger fails, will the row be rolled back?
<stub> wallyworld_: I chose DEFAULT TRUE because if the trigger is broken (it isn't), we default to private. Either TRUE or FALSE should be fine as the DEFAULT>
<stub> wallyworld_: Yes, if the trigger fails the whole transaction will be rolled back.
<wallyworld_> stub: wgrant: the reason for me asking for default = false is that i need to check the short circuit rules in the trigger
<stub> ok
<wallyworld_> if transitively_private is true, then it may not bother doing anything
<stub> short circuit checks private, not transitively_private
<stub> But double checking is good :-)
<stub> Tests are better...
<wallyworld_> yes, agreed. i will write one now that we are taking this approach
<stub> Hmm... the MP has gone
<stub> lifeless did not realize the field is already maintained with the AFTER trigger
<wallyworld_> stub: i renamed the branch
<wallyworld_> in anticipation of landing it in two parts
<stub> You can rename branches?
<wallyworld_> i'll rename it back
<wallyworld_> yes, apparently :-)
<stub> I managed to misspell logintoken as lockfile in my branch name the other day...
<wallyworld_> stub: renamed back now
<wallyworld_> go to the branch details link
<wallyworld_> and you can rename
<StevenK> Can haz review? https://code.launchpad.net/~stevenk/launchpad/refactor-team-rename/+merge/78037
<StevenK> Obviously, the answer to that question is "no".
<wgrant> Indeed.
<wgrant> So.
<wgrant> You've moved it into the model, apparently.
<wgrant> Which is probably where it should be.
<wgrant> But...
<wgrant> Person and Team don't need a model mixin, because they're both the same class.
<StevenK> So make it a method in IPerson itself?
<StevenK> reason = self.context.can_be_renamed() or so?
<wgrant> Quick possibly. checkRename or something like that.
 * StevenK refactors the refactor
<wgrant> StevenK: Also, can you put the "has" in the template?
<wgrant> Removes the has_ppa check under has_mailing_list.
<StevenK> But I can't ...
<wgrant> Why not?
<StevenK> Because it changes the prose for the both check
<wgrant> This %s has %s and may not be renamed.
<StevenK> "has a PPA ... and has a mailing list" versus "has a PPA and a mailing list"
<wgrant> %s is "an active PPA with packages published", "a mailing list", or "an active PPA with packages published and a mailing list"
<StevenK> Oh
<StevenK> Right
<wgrant> Also, possibly s/user/person/
<StevenK> Also fixed
<lifeless> is there a reviewer in the house ?
<lifeless> https://code.launchpad.net/~lifeless/python-oops-datedir-repo/hashing/+merge/78039
<wgrant> lifeless: Why BSON? :(
<wgrant> These are going to be even less readable than SSO OOPSes.
<lifeless> wgrant: decoders exist; handles binary crud (which we get like it or not)
<wgrant> And makes it just about impossible to view OOPSes unless oops-tools is agreeable and fast.
<wgrant> neither of which is true.
<lifeless> this is a component in fixing that
<wgrant> And until then our error analysis capabilities are crippled.
<lifeless> how so ?
<wgrant> It becomes extremely difficult to respond in realtime to complaints involving OOPSes.
<lifeless> details dude
<lifeless> give me the details
<wgrant> I don't speak BSON.
<wgrant> text editors don't like BSON.
<wgrant> oops-tools is slow to render, and even slower to pick up OOPSes, and often doesn't ever see them at all.
<lifeless> import bson; bson.loads(open('foo', 'rb'))
<wgrant> Mmm, perhaps.
<wgrant> OK.
<lifeless> slow to render is as yet undiagnosed. Slow to pick up is what I'm on an arc to fix.
<lifeless> not seeing them is fixed as soon as that chroot comes around.
 * StevenK gets spammed by wgrant
<wgrant> StevenK: Oh?
<wgrant> bugmail?
<StevenK> Yeah
<wgrant> lifeless: Also, what about eg. dev/DF OOPSes?
<StevenK> Damn it, come on MPJ!
<lifeless> wgrant: DF - install oops-tools.
<wgrant> hahahaha
<lifeless> wgrant: dev - oops-tools is about to be opened
<lifeless> wgrant: tests can use the programmatic interface
<wgrant> So everyone has to run their own Django app to see oopses?
<lifeless> wgrant: or you can set the serializer to rfc822 - I kept compat for a reason.
<wgrant> We need a sensible dev story for this before it becomes the default.
<StevenK> +2,000,000
<lifeless> no, we need to get this shit sorted
<lifeless> that django app is easier to install than LP
<wgrant> Yes, but sorting shit so it breaks other shit is not ideal.
<lifeless> 2 branches + buildout and a postgresql cluster
<lifeless> wgrant: its not breaking other stuff
<StevenK> It's another barrier to contribution
<StevenK> Which is already too high
<StevenK> So, fail
<wgrant> It's breaking the ability to debug a local Launchpad instance.
<wgrant> And to see test failures.
<lifeless> no, its not - to either.
<lifeless> the test suite attachOopses uses rfc822 serializer - it receives the oops dicts.
<wgrant> Ah
<lifeless> oopses written by subprocesses will change format, yes. But they are *already* a PITA to get at - and trivially decoded using... bson
<wgrant> It is trivial.
<lifeless> StevenK: its a barrier to -some- forms of contribution, but I look at a local oops about once a month, if that.
<StevenK> I tend to look at OOPS on carob using cat, now I won't be able to. :-(
<lifeless> StevenK: pages will still show backtraces to local devs, the console backtrace will still be plain text
<wgrant> But analysis of dev and prod OOPSes at the moment is now often done by means of cat, grep and cut
<StevenK> I hate our oops-tools app
<lifeless> so fix it
<StevenK> Sure, in the oddles of free time I have?
<wgrant> Right, we should fix it.
<wgrant> But *before* we break everything that isn't it.
<wgrant> I don't see why BSON has to happen at the same time as hashes.
<lifeless> we are -stalled- on multiple important improvements to our data gathering - to whit, extra info on appserver resource usage, clearer metadata, evolving the damn thing.
<wgrant> It seems unrelated.
<lifeless> they are separate fields in the repository object, and we can configure them however we want.
<lifeless> e.g. dev mode can use rfc822 if you don't mind extra data being dropped.
<StevenK> wgrant: Can you have another look at my MP?
<wgrant> StevenK: Was just doing so.
<wgrant> lifeless: What is the benefit in moving to BSON at the same time?
<lifeless> again, claiming this is breaking things is very exaggerated. Yes, the shell pipelines you used won't work. Are they intractably different with a new serializer? No.
<StevenK> +71 -44 makes me a little sad. I prefer removing more code than I add.
<lifeless> wgrant: A) I haven't moved anything to bson, B) see above.
<wgrant> 87+ Defaults to using serializer_bson.
<StevenK> Hm, this might be able to be refactored further.
<lifeless> yes
<lifeless> but our packages are version locked, except perhaps txlongpoll which I haven't looked into, so when bumping the version, if we want rfc822 explicitly we can do that. Its not coupled to *this* merge proposal.
<lifeless> (Separately, yes, I plan to use bson when I update the dep in LP, I'm being pedandtic because of the hyperbole you're tossing at this)
<lifeless> wgrant: so, what are you asking be written before we change to bson ?
<wgrant> Well, ideally we would have OOPS analysis tools that didn't involve obscene shell pipelines.
<wgrant> But that seems to not be practical.
<lifeless> wgrant: (Note that 'do not break anything' is incompatible and I reject that out of hand) So it 'provide a full text index' - we have plenty of ability to search with os.walkdirs + bson in an adhoc manner, IF/when oops-tools fails to meet our needs.
<lifeless> wgrant: I would *love* to run up solr etc and JFDI
<lifeless> wgrant: but thats a long pipeline, and I'm tired of getting significantly better data from oopses being blocked - when the change needed to do it differently is not all /that/ big
<lifeless> again though, policy vs requirements - this branch lets us choose
<StevenK> Blah, too hard to refactor further. Mainly because browser/person.py is *obscene*
<lifeless> wgrant: your smoketest branch - +1
<wgrant> lifeless: Thanks, I'll propose it properly in a sec.
<StevenK> wgrant: Can haz a vote?
<wgrant> lifeless: A couple of questions, but yours is approved.
<lifeless> wgrant: oops-pruner
<lifeless> wgrant: thanks
<wgrant> lifeless: But they're normally on disk as 1234.ABC1234
<wgrant> Not OOPS-*
<StevenK> wgrant: I've just pushed a change to make {Person,Team}EditView.setUpWidgets() identical, but it's very hard to refactor further.
<wgrant> StevenK: REFACTOR FURTHER
<wgrant> Always refactor further.
<lifeless> wgrant: yes, but the *id* is normally OOPS-blah. And using the id directly is simpler than using the id minus OOPS-
<lifeless> wgrant: also, the name on disk at the moment is all to do with sorting
<wgrant> True.
<wgrant> Go ahead, then.
<StevenK> wgrant: I tried. TeamEditView can't use BasePersonEditView as a parent due to circular import hell.
<wgrant> Oh?
<wgrant> browser.person import browser.team?
<StevenK> wgrant: http://pastebin.ubuntu.com/702084/
<wgrant> StevenK: WTF, why does browser.person import from browser.team?
<wgrant> Move Team*Menu into browser.team
<StevenK> classImplements(TeamIndexView, ITeamIndexMenu)
<StevenK> classImplements(TeamEditView, ITeamEditMenu)
<StevenK> But yes, that's a little screwed.
<lifeless> jamesh: hi
<rvba> Morning.
<rvba> wgrant: Hi William, thanks for QA'ing my +initseries branch ;).
<wgrant> It turned out to not be so important, but it was easy enough :)
<jamesh> lifeless: hi
<rvba> Always nice to start my day with a qa-ok message from the other side of the planet ;)
<lifeless> jamesh: ola~
<lifeless> jamesh: so, two things; oops-datedir-repo can write bson reports now (and trunk oops-tools can read them)
<lifeless> jamesh: this makes extension and handling a -lot- easier, I think you'll agree ;)
<jamesh> yep.
<lifeless> jamesh: secondly, I was wondering if you'd tried integrating stuff again? [I'd love to have an answer for getting oopses into the oops-tools django app itself, for instance]
<jamesh> lifeless: I haven't done much since we last talked.  My test u1 branch does log reports with unmodified Django (by subclassing Django's WSGI handler)
<lifeless> perhaps that subclass should be in oops-* somewhere? I don't know enough django to know if that makes sense
<lifeless> jamesh: also, thats awesome - we're in business ;)
<lifeless> wgrant: as promised (and as penalty you get the review :P) https://code.launchpad.net/~lifeless/python-oops/hostname/+merge/78053
<jamesh> lifeless: http://paste.ubuntu.com/702108/ <- Using this instead of the standard WSGIHandler should be enough.
<jamesh> actually, that plus oops_on_status=['500']
<wgrant> lifeless: Approved.
<jamesh> for the oops-wsgi middleware
<wgrant> With missing apostrophe.
<lifeless> wgrant: and aggregate->aggregation
<lifeless> wgrant: do you mean 'machines''s' ?
<wgrant> Right.
<lifeless> wgrant: on thursday, can I pick your brain on wabbit ?
<wgrant> lifeless: Of course.
<lifeless> jamesh: is there a preferred way to receive rabbit events in django ?
<jamesh> lifeless: I don't think so.  We've been handling them in a Twisted reactor running as a separate thread
<lifeless> aieeee
<lifeless> jamesh: how have you been handing off back to django code? or are you calling it from deferToThread sort of things?
<lifeless> bbiab
<jamesh> lifeless: I haven't poked too deeply in this area, but I think we're usually pushing out messages from Django views rather than receiving them
<jamesh> Django is not that well suited to long-poll style request handling, which is the main type of request I'd expect to want to receive messages
<lifeless> jamesh: I want to add oopses to the django db as they happen
<lifeless> jamesh: and write them to disk likewise
<wgrant> I smell a daemon...
<wgrant> Doing that from the WSGI app seems, er, aaaaaaa
<lifeless> wgrant: we have one
<lifeless> wgrant: doing that from outside the django transaction is also aaaaaa
<wgrant> Huh?
<StevenK> OMG, there's so much Team stuff in browser/person.py :-(
<jamesh> lifeless: okay.  So I probably wouldn't try to squeeze that into a Django web app.  You could certainly use the Django ORM from a separate daemon that receives the messages.
<wgrant> That would seem like the least insane way to go about things.
<wgrant> Unless we're using SQLite, running it in a separate process should really be less than a problem.
<StevenK> wgrant: I've moved the classes that caused the circular import, shall I explode the size of this branch and move everything that they depend on too?
<jamesh> wgrant: why would sqlite be a problem?
<wgrant> jamesh: SQLite doesn't do multiple writers, does it?
<jamesh> wgrant: pretty sure it does.
<lifeless> it does
<jamesh> I'm sure it is possible to configure the locking out of it, but I'm pretty sure it is in by default
<lifeless> but you need latest shiny
<lifeless> and its still got a large lock even tuned optimally
<lifeless> we can do a separate daemon, though I have an aesthetic dislike - I'd rather have an API on the one app and not poke into its backend
<jamesh> I don't think you're in any worse situation by having two processes accessing the database than a single process accessing the database from multiple threads with multiple connection objects
<lifeless> OTOH I don't want to make this a pile of yak shaving
<lifeless> jamesh: we're using pgsql anyhow
<lifeless> 27M oops reports would give even sqlite a bit of a hernia
<lifeless> without -very- careful tuning
<jamesh> lifeless: well, you can use your Django ORM classes outside of a web app.
<jamesh> I'm not suggesting that you duplicate all the application logic
 * StevenK prods wgrant.
<lifeless> jamesh: sure, I know :)
<lifeless> jamesh: like I said, its an aesthetic. We'd avoid a lot of issues in LP if we did that there, for instance.
<wgrant> StevenK: Sorry, missed that. Could you possibly move it in a prereq?
 * lifeless shrugs, we can start with a daemon and db access and switch to daemon + web api later
<StevenK> wgrant: BLEH
<jamesh> you'll just have to make sure your separate daemon process sets $DJANGO_SETTINGS_MODULE to something sensible before importing django
<StevenK> wgrant: This pipe is already 1885 lines and I'm not finished yet.
<wgrant> :D
<StevenK> wgrant: Right, every {I,}Team class moved is 2006 lines.
<StevenK> Now to fix __all__ and imports
<StevenK> And configure.zcml
<nigelb> StevenK: Nice cleaning :)
<StevenK> OH GOD, IT BURNS
<StevenK> PersonNavigation needs TeamMembershipSelfRenewalView, PersonView needs TeamJoinMixin and TeamOverviewMenu.
<StevenK> circular imports a-go-go
<StevenK> wgrant: ^
<StevenK> Oh damn, and the Mixin is a parent class.
<wgrant> Hah.
<StevenK>  /wrists
<wgrant> But why would PersonView need TeamJoinMixin?
<wgrant> Oh no.
<wgrant> I think PersonView might be used for teams as well.
<bigjools> StevenK: surely your wrists are used to the strain
<StevenK> wgrant: Looks like it.
<wgrant> Ah, TeamIndexView
<wgrant> Hopefully you can export TeamIndexView to browser.team, and take TeamJoinMixin over there.
<StevenK> I've already moved it.
<StevenK> I moved every {I,}Team class.
<StevenK> Now I'm trying to make pyflakes happy
<wgrant> Change TeamIndexView to inherit TeamJoinMixin, instead of PersonView inheriting it.
<wgrant> Hopefully only the team views need it.
<bigjools> StevenK: I mean from playing WoW of course... :)
<StevenK> bigjools: I quit WoW 3 months ago.
<bigjools> StevenK: good grief, are you feeling ok?
<StevenK> And that is why I didn't tell many people.
<bigjools> online games are like a drug addiction - I've gone cold turkey twice now
<jamesh> lifeless: fwiw, the Django bug report about fixing the WSGIHandler to not need that fix is no longer in the "design decision needed" black hole
<lifeless> jamesh: did it win or lose ?
<jamesh> lifeless: https://code.djangoproject.com/ticket/16674 <- they moved it to accepted, and wanted more verification that the change wouldn't cause problems.
<lifeless> \o/
<StevenK> wgrant: pyflakes clean, 2287 lines
<wgrant> StevenK: But does it run? :)
<StevenK> No, not yet.
<StevenK> Fighting with ZCML
<StevenK> wgrant: I have a small subset of tests passing, 2477 lines.
<wgrant> Heh.
<StevenK> Let me toss me it at ec2 to find the rest.
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 255 - 0:[#######=]:256
<poolie> stub, hi?
<stub> yo
<poolie> stub, can you suggest what to do on bug 866160?
<poolie> sorry bug 866100
<_mup_> Bug #866100: bug search with affects_me=on times out <affectsmetoo> <bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/866100 >
<stub> poolie: Need the OOPS to sync
<poolie> hm and now it's not timing out
<poolie> how uncooperative
<nigelb> poolie: I think it times out on first try
<nigelb> (and then caches)
<poolie> that's likely
<poolie> well, the oops will sync later
<nigelb> It timed out once for me, refreshed later.
<nigelb> I'm looking forward to 2 days of lp hacking \o/ Holidays!
<poolie> :)
 * gmb -> lunch
<poolie> stub, i put an example failing query there, in case that helps
<poolie> good night now
<nigelb> g'nite poolie
<stub> poolie: We don't know if that is the slow query. All we know is that it is the query being run when the timeout was hit.
<poolie> that is true, but that's similar to the query that was sometimes very slow when run on qas and by losas on prod
<poolie> anyhow no rush
<stub> If it is, things to try are to move the BugTask.id IN () subquery using BugAffectsPerson into a normal join, or to enforce it is actually run as a subquery rather than the planner turning it into a normal join by using a WITH clause.
<stub> I can reproduce with a simpler query on production.
<stub> I don't think either of those options I mentioned will help greatly here
<stub> We are ordering results using a field on Bug, but all the selection is going on in bugtask.
<stub> Not sure what we can do here. We can optimize for the case where we know we will not match many BugTask rows, but if we wanted to do 'All bugs targetted to Launchpad, order by date_last_updated' it would likely fail.
<bac> hi gmb, can i get a review when you have time?  i apologize it is way over our agreed size limit
<gmb> bac: Sure. I should be able to take a look at it within the next hour or so.
<bac> thanks gmb
<deryck> abentley, hey, hey.  Sorry the vet visit toook *much* longer than I expected.
<abentley> deryck: that's okay.
<abentley> deryck: lemme grab my coffee, and we can chat.
<deryck> abentley, let me just get settled in and we can chat briefly for a standup.
<deryck> abentley, yeah, give me 10 minutes.
 * beuno hands gmb bug #867529
<_mup_> Bug #867529: Dynamic loading comments load on top of page content <Launchpad itself:New> < https://launchpad.net/bugs/867529 >
<deryck> abentley, heading into mumble now.
<gmb> beuno: Thanks
<rvba> gmb: Would you have time to mentor a review of mine?
<gmb> rvba: I've got a large review to do for bac first, so you might be better served finding another reviewer to mentor you on this one.
<rvba> gmb: Okay ;)
<sinzui> gmb, do you have time to review https://code.launchpad.net/~sinzui/launchpad/dsp-historic-attributes/+merge/78106
<gmb> sinzui: I'm currently reviewing a large branch for bac, but I'll try to get to it this afternoon if I have time.
 * bac hangs head
<sinzui> oh. damn. I should of claimed that this morning because I read half of it
<sinzui> jcsackett, do you have time to mumble?
<deryck> skaet, ping
<skaet> deryck, pong
<gmb> sinzui: I'm afraid I won't be able to get to your branch today as I have to get a few other things done before my rapidly-approaching EoD. I believe StevenK is next on the list of reviewers.
<sinzui> thanks
* gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 255 - 0:[#######=]:256
<deryck> skaet, I'm trying to QA bug 834082.  Can you try poking the issue you reported at our qastaging server and see if it's fixed for you?
<_mup_> Bug #834082: Cannot target a bug to a series from an existing series task <critical-analysis> <qa-needstesting> <regression> <Launchpad itself:Fix Committed by deryck> <Ubuntu:New> < https://launchpad.net/bugs/834082 >
<gmb> bac: r=me (with comments). Excellent work, thank you.
<bac> thanks gmb
 * bac didn't do most of it, of course
<skaet> deryck, will try.
<deryck> skaet, thanks!  Here's a starting point, if you're not familiar with qastaging:  https://bugs.qastaging.launchpad.net/ubuntu/oneiric/+bugs
<deryck> skaet, qastaging can timeout so it's worth a reload or two if you hit a timeout.
 * skaet gets the timeouts....
<deryck> skaet, do you even get to a bug page?
<skaet> deryck,  get the bugs in oneiric page from the link you gave just fine.   However not being able to get to 834082
<deryck> ah, that sucks.
<deryck> skaet, will another bug work?
<skaet> deryck,  can you give me a link to that bug page directly?   May be faster than searching.
<deryck> skaet, https://bugs.qastaging.launchpad.net/launchpad/+bug/834082
<_mup_> Bug #834082: Cannot target a bug to a series from an existing series task <critical-analysis> <qa-needstesting> <regression> <Launchpad itself:Fix Committed by deryck> <Ubuntu:New> < https://launchpad.net/bugs/834082 >
<skaet> deryck, that got it.    will poke around.
<deryck> skaet, thanks!
<jcsackett> sinzui: available to talk for a few?
<sinzui> okay
<gmb>  e
 * gmb chuckles at rockstar's response to https://bugs.launchpad.net/launchpad/+bug/867661.
<_mup_> Bug #867661: Useless Comments on Bug Reports <Launchpad itself:Opinion> < https://launchpad.net/bugs/867661 >
<rockstar> gmb, he left it open.
<gmb> rockstar: Oh, indeed. In fact, I was half tempted to reply in the same vein until I saw you'd beat me to the punch :)
<sinzui> jcsackett, https://bugs.launchpad.net/launchpad/+bug/867718
<_mup_> Bug #867718: DSP picker entries are redundant <target-picker> <ui> <Launchpad itself:Triaged by sinzui> < https://launchpad.net/bugs/867718 >
<deryck> skaet, any luck checking that out?
<sinzui>  I see txlongpoll is taking 8% of my CPU all the time. That does not bode well. Nothing should preform worse than xorg
<skaet> deryck, got pulled into release issues while still trying,  will take a pass at it later this afternoon.
<deryck> skaet, ok, no problem.  Thanks!
<abentley> gary_poster: bzrlib.branch.Branch* is never security-proxied, but LoomBranch* is security-proxied.  I think they should be the same, but I've had trouble figuring out why bzrlib.branch.Branch isn't security-proxied.
<gary_poster> abentley, mwhudson put in some code for that in lib/lp_sitecustomize.  I suspect that's what you are looking for.  Look for dont_wrap_class_and_subclasses in that module, and its uses.
<abentley> gary_poster: thanks.
<gary_poster> welcome
<mwhudson> oh yes, that fun
<mwhudson> oh dear, my ec2 failure mail is so large emacs crashes opening it
<abentley> gary_poster, mwhudson: my fix: http://pastebin.ubuntu.com/702418/
<gary_poster> abentley, interesting.  Comments make it good AFAI am concerned.
<mwhudson> abentley: i wonder if moving the code that sets up BZR_PLUGIN_PATH to lp_sitecustomize might be better in some ways?
 * gary_poster defers to mwhudson :-)
<mwhudson> then there are a few random imports of lp.codehosting that can just go away
<mwhudson> gary_poster: i haven't thought about it very hard!
<gary_poster> :-)
<abentley> mwhudson: I think I was on crack to make an import have side effects like enabling plugins.
<mwhudson> abentley: i think this crack predates you on the lp team doesn't it?  but yes
<mwhudson> there is a slight argument that say, the appservers, should not import lp.codehosting at all and so won't have plugins enabled
<mwhudson> but if that's the way we want to go, they should be separate code bases (hi rob!)
<abentley> mwhudson: No, it was basically the first thing I did on the team.
<mwhudson> ah ok
<abentley> mwhudson: as long as the appservers are the private xmlrpc servers, they need to support loom branches.
<mwhudson> err how so?>
<abentley> mwhudson: Oh, I guess they operate at a lower level, providing transports, so they don't actually need to know about branch formats.
<mwhudson> yeah, just paths on disk
<mwhudson> so my ec2 test run appears to have failed, basically because the db started erroring all the time about half way through
<mwhudson> ah, it's hard_timeout checking
 * jelmer waves to abentley, mwhudson
 * abentley waves at jelmer
 * mwhudson waves back
<mwhudson> oh god everything is broken
<mwhudson> # XXX Julian 2009-05-13, bug=376171
<mwhudson> # Temporarily disabled because of intermittent failures.
<mwhudson> 2.5 years of temporary
<jelmer> mwhudson: 2.5 years is nothing compared to what the early Launchpad and Ubuntu specs planned to do in a single cycle
<mwhudson> haha true
<mwhudson> launchpad.canonical.com from about 2006 should be put online as a kind of massive example of how not to do project management
<jelmer> heh
<jelmer> http://web.archive.org/web/20060611133645/https://launchpad.net/
<mwhudson> ploooooooooooooone!
<mwhudson> or at least, it's css
<jelmer> heh, indeed
<lifeless> mwhudson: yeah there is a little bootstrap issue with hard timeouts
<lifeless> mwhudson: will be if/when we make soft ones flagged too
<lifeless> mwhudson: mmm, perhaps not. Anyhow, hard for sure
<mwhudson> lifeless: yeah, the permit_timeout_from_features flag is now on the request too in my branch
<mwhudson> lifeless: it's not reliably reset between tests in devel currently, so if it gets set (say by a test publication) and a test runs that sets up a feature controller before it gets reset... hilarity ensues
<lifeless> mwhudson: win
<mwhudson> (and one of the parts of my branch that i'm not 100% sure about is that a lot more tests set up a feature controller now)
<lifeless> so you asked yesterday about null feature controller and ltr
<lifeless> IIRC LTR is used in layers that don't have all the bits assembled, was the reason.
<mwhudson> yeah, that makes some kind of sense
<mwhudson> but i bet those tests don't actually access the feature controller
<mwhudson> (bit fragile of course)
<lifeless> you'll find out, or go mad trying.
<lifeless> I'm betting on mad.
<mwhudson> i'm hoping ec2 test will find out :-)
 * mwhudson might actually claim for his ec2 expenses this month...
<lifeless> if its more than the cost of doing expense claims ? :)
<lifeless> flacoste: http://h30565.www3.hp.com/t5/Feature-Articles/Innovate-or-Suffer-Slow-Brain-Asphyxiation/ba-p/499 is good reading
<mwhudson> haha, if you wiggle things around so that feature flags are active most of the time, a few sql effort tests fail
#launchpad-dev 2011-10-05
<cody-somerville> lol
<cody-somerville> Just got this error trying to instantiate a Launchpad object:
<cody-somerville> TypeError: __init__() takes at least 4 arguments (4 given)
<poolie> :)
<poolie> wow
<nigelb> Morning!
<poolie> hi nigel
<nigelb> o/
<nigelb> *yawn*
<StevenK> nigelb: Diff against target: 2522 lines (+1091/-1038) 8 files modified
<nigelb> StevenK: You, sir, are a master :-)
 * StevenK still needs to find someone to review it ... :-)
<nigelb> Too bad you're OCR today :P
<StevenK> I'm so not self reviewing a branch with a 2,500 line diff.
<nigelb> lol
<nigelb> Look for victims like wgrant :P
<StevenK> nigelb: lp:~nigelbabu/launchpad/create-description-5283 is still hanging around, do you need that tossed through ec2 again?
<nigelb> StevenK: There's a conflict
<nigelb> Working on it today after breakfast and coffee.
<nigelb> (Just need to create new sample data again
<nigelb> )
<StevenK> Like fun you do.
<nigelb> :D
<StevenK> Sample data needs to die, not be created.
<nigelb> Well, sure.
<nigelb> I don't want to break all dev machines though.
<nigelb> But, what's the alternative to sampledata?
<StevenK> Hm? How does no sampledata break dev machines?
 * StevenK actually reads the diff.
<nigelb> I thought there were legacy tests depending on sample data.
<StevenK> I can't see a conflict in the MP
<StevenK> And your target should be db-devel
<nigelb> I'm pretty sure there's a conflict.
<nigelb> I landed a db patch right before this
<nigelb> anyway, 20 minutes and I'll post it again :)
<StevenK> Uh? That branch *is* the DB patch?
<sinzui> StevenK, I added a UI branch that extends the DSP vocab branch I have in review.
<StevenK> sinzui: I have reviewed your first branch.
<StevenK> sinzui: My mega-branch is up for review
<sinzui> fab I will do that now
<lifeless> nigelb: you -never- need to add sample data
<lifeless> nigelb: you may need to tweak existing (but better to update the test that used it to not use it)
<nigelb> lifeless: I only need to run make sampledata and do the replacement thing.
<sinzui> nigelb, there are two sampledatas
<nigelb> normal and dev?
<StevenK> Testing and dev, right
<StevenK> I think
<sinzui> one (sampledata) is for tests and was frozen 2.5 years ago. No additions, but removals are okay. The other is sampledata-dev and it provides the data for your browser. It is good to provide sane data to look at Lp
<StevenK> sinzui: FSVO 'sane'
<StevenK> The publication records in sampledata are *wrong*
<sinzui> yes!
<nigelb> sinzui: "additions" in terms of adding new data or adding new fields?
<StevenK> The former
<sinzui> note in my last MP I gave instructions to populate the DSP table with proper data
<StevenK> sinzui: I'd like to chat tomorrow about my denorm garbo job, by the way.
<sinzui> okay
<StevenK> It is just making no progress at all.
<StevenK> If it isn't postgres backups, it's replication lag, and if it isn't those two, it's process-death-row.
<lifeless> StevenK: is pdr doing long transactions ?
<StevenK> lifeless: So before Dapper was cashiered out of the librarian, p-d-r took about 30 seconds. Now it takes 90 minutes.
<StevenK> And we haven't done any work on it due to lol-maintaince.
<StevenK> lifeless: But to be fair, p-d-r is a long third behind slony, backups and replication lag
<lifeless> StevenK: ok, so it was fast enough that noone at the time considered one big xaction a problem, and now its not ;)
<StevenK> lifeless: We probably need to just fix what it is choking over, TBH.
<StevenK> Our represtantion of the archive is not clean in our prod db.
<mwhudson> lifeless: so there is exactly one test that breaks if LaunchpadTestRequest sets up a real feature controller
<mwhudson> (it's testMemcachedWorking (canonical.testing.ftests.test_layers.ZopelessTestCase), excitingly)
<sinzui> StevenK, I could remove the maintainer. We are showing the maintainer because it is assumed that that is the person/team that will like be working with a private bug that you report
<sinzui> StevenK, We are showing all the binary package names in the expanded details so that you can see why the source package matched
<StevenK> I just think there is more useful information to show besides the maintainer.
<sinzui> StevenK, we do not plan to distinguish between package, the bugs about assign to a package make it clear users do not want to think about source and binary
<StevenK> sinzui: Oh, is it supposed to be an icon rather than the word?
<sinzui> Just the word, to be consistent with the case when we show project group, project, and distribution in target pickers
<StevenK> Right
<sinzui> Since we are going to show all the binary names, I think the maintainer is the big issue here.
<sinzui> StevenK, That value changes if the uploader changes right?
<StevenK> Which property of bpr/spr are you using?
<sinzui> StevenK, dsp.currentrelease.maintainer.displayname
<sinzui> That is the SPR for the current series
<StevenK> If maintainer is pulled from the source package itself, it's going to be "Ubuntu Core Developers" for an awful lot of the archive.
<sinzui> StevenK, that is interesting. If I were reporting a bug in a future private distro derived from oneiric, are "Ubuntu Core Developers" the group who will be seeing the bug besides the private distro owners?
<lifeless> mwhudson: win
<StevenK> sinzui: Perhaps. We haven't designed private distros yet. :-)
<sinzui> We may need to fix the dsp bug supervisor role/policy which is an old bug
<nigelb> Ok, breakfast has been had. Now to finish off those MPs.
<sinzui> StevenK, Do you want me to remove the maintainer? Force it to None so that it can be discusses at a later time, or keep it as it is?
<StevenK> sinzui: I like the idea of removing it so we can discuss it later.
<sinzui> I will remove it
<sinzui> StevenK, do you need any more information?
<StevenK> sinzui: I think that's it.
<sinzui> thanks. I will do this now.
<mwhudson> argh everything is horrible
 * mwhudson gives up for today
<nigelb> Launchpad - "argh everything is horrible"
<StevenK> Sounds right to me.
<nigelb> grar
<nigelb> I should redo this entire branch.
<StevenK> It sometimes comes to that.
<StevenK> bzr di > /tmp/foo ; cd .. ; rm -rf <branch>
<nigelb> hrm. Ouch
<nigelb> I deleted that as well.
<StevenK> The fact that I typed it quickly is in no way a sign of how often I do that ...
<nigelb> I did, however, save the db patch file.
<nigelb> So, if I'm not mistaken, all is not lost :D
<StevenK> If the branch is still on LP, you can grab it again
<nigelb> It is \o/
<nigelb> DVCS++
<nigelb> Reminds of the poor soul in this one http://programmers.stackexchange.com/questions/112270/small-company-15-developers-doesnt-use-managed-source-version-control-is-thi
<StevenK> nigelb: Cold sweat.
 * StevenK resubmits his MP, due to the diff going from 243 lines to 2788 ...
<nigelb> when I add a column, do I force default NULL or do I just leave it alone?
<nigelb> ohwait. This has db ack.  I should not be worrying about this.
<nigelb> help -> http://paste.ubuntu.com/702588/
<StevenK> bin/py doesn't exist, run make
<nigelb> ah
<nigelb> grr
<nigelb> Getting distribution for 'txlongpollfixture==0.1.3'.
<nigelb> make: *** [/home/nigelbabu/launchpad/lp-branches/create-description-5283/bin/py] Error 1
<StevenK> rf-get
<nigelb> already started :)
<StevenK> Or update your download-cache
<StevenK> Either way, blame Red Squad
<nigelb> I was about to say #blame-bigjools but that works :D
<StevenK> Right, I think that MP is squared away until the previous pipe lands and wgrant re-surfaces.
<nigelb> Is he off this week?
<StevenK> No, just today.
<nigelb> He should be online in an hour :P
<StevenK> Perhaps. I know what he's doing today, and he may not be.
<nigelb> Ah.
<nigelb> Do you both live in the same city btw?
<StevenK> No, wgrant is in Melsbum.
<nigelb> And you?
<StevenK> Sydney
<nigelb> AH
<wgrant> StevenK: Is the refactoring pipe reviewed?
<poolie> stub, if you can also give any suggestions on how to improve bug 868021 i'd appreciate it
<_mup_> Bug #868021: +affectingbugs times out <affectsmetoo> <bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/868021 >
<stub> poolie: I've already added some reworked queries there.
<poolie> :)
<stub> poolie: Bringing a 9s query down to subsecond. I haven't gone over the OOPS report now it should have synced.
<poolie> really 868021?
<poolie> the other similar one, 866100, we tried your queries and they're generally good but intermittently slow
<poolie> perhaps in a way that varies by db server
<stub> Sorry, thinking of the other bug
 * stub looks
<StevenK> wgrant: The pre-req branch was +1'd by Curtis.
<StevenK> wgrant: I had to resubmit the second branch because of it, so it needs your +1 again.
<stub> poolie: I suspect fixing Bug 866100 would address Bug 868021 too
<_mup_> Bug #866100: bug search with affects_me=on times out <affectsmetoo> <bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/866100 >
<_mup_> Bug #868021: +affectingbugs times out <affectsmetoo> <bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/868021 >
<wgrant> StevenK: Ah, great. Will hopefully look later tonight.
<poolie> stub, yes i wasn't sure whether it made sense to split them or not
<StevenK> wgrant: I'm in no rush, since I want the pre-req to land first.
<poolie> but since the conversation was getting a bit long i thought i would split
<stub> poolie: separate bugs could be fine, but I'd tackle the simpler case first.
<poolie> and 866100 is simpler?
<poolie> ok great
<stub> poolie: 866100 is simpler because I've done diagnosis :-)
<stub> poolie: I'm just assuming 868021 may be fixed as a side effect, as the trigger condition is the same (bugaffects clause)
<nigelb> StevenK: lol, wgrant did come online in about 1.5 hours :D
<wgrant> Indeed!
<allenap> nigelb: Your download-cache is out of date.
<nigelb> allenap: yeah, fixed that.
<allenap> nigelb: Cool :)
<bigjools> morning everyone
<nigelb> Hello bigjools!
<bigjools> hey nigelb
<nigelb> Hrm
<nigelb> someone changed 2010 to 2010-2011 in current.sql
<nigelb> Is a better fix to fix whatever generates newsampledata.sql?
<mrevell> Hello
<nigelb> Hey mrevell :)
<nigelb> StevenK: ohai, still around?
<StevenK> nigelb: Barely
<nigelb> StevenK: oh. I was just about to put that MP into your queue. Nevermind, I'll catch the euro train :)
<nigelb> allenap: Hi, OCR'ing today? :)
<allenap> nigelb: My day is tomorrow... let me see who's on call...
<allenap> nigelb: danilos is down for the EU slot today (jtv too, but he's away). Aaron is on later. If it's something small I don't mind taking a look though.
<nigelb> allenap: bah, right. My bad. I looked at Thu instead of Wed.
<nigelb> Its an approved db patch which I couldn't land because of conflicts - https://code.launchpad.net/~nigelbabu/launchpad/create-description-5283/+merge/76818
<nigelb> I made one extra change in database/schema/Makefile
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilos | Critical bugs: 255 - 0:[#######=]:256
<danilos> allenap, thanks for the reminder :)
<danilos> nigelb, btw, why did you not just merge the db-stable instead of starting from scratch?
<nigelb> danilos: Well, I screwed up massively :D
<danilos> nigelb, right, but in this case, I'd have to ask you to make a new merge proposal and get patches reviewed again, because we can't tell if they are the same or not
<nigelb> ahhh.
<nigelb> db review?
<danilos> nigelb, yeah, db review as well
 * nigelb does that.
<nigelb> *faceplam*
<nigelb> why did I just delete the branch.
<danilos> nigelb, btw, did you use "bzr push --overwrite" for this?
<nigelb> danilos: I did.
<danilos> nigelb, right, well, that makes any merge proposals you've got invalid since you are changing the branch history behind LP's back :)
<danilos> nigelb, whoops :)
<nigelb> danilos: Ah. Drat. Anyway, its nicer this way.
<danilos> nigelb, nicer? if you are refering to "bzr push --overwrite", it's not, just a simple "bzr merge lp:~launchpad-pqm/launchpad/db-stable && resolve conflicts && bzr ci -m 'Merge db-stable.'" is the best thing to do
<danilos> nigelb, the thing you did was basically rebasing, bzr doesn't require that at all :)
<danilos> nigelb, and your revision will end up as one revision when it's finally merged anyway
<nigelb> well, at some point of cleaning this up, I lost of track of what I was doing.
<nigelb> But yeah, noted for next time.
<nigelb> stub: hi, around?
<nigelb> stub: Need a re-ack of a db patch because I messed it up :)
<danilos> nigelb, I am sorry to cause you more work :/
<nigelb> danilos: heh, np. I think I shot myself in the foot there :D
<nigelb> https://code.launchpad.net/~nigelbabu/launchpad/create-description-5283/+merge/78220
<nigelb> mrevell: around?
<mrevell> I certainly am nigelb
<nigelb> mrevell: I was reading about Open Badges, it certainled seemed interesting to think of LP giving out badges for certain things - http://openbadges.org/About/
<nigelb> *certainly
 * mrevell looks
<mrevell> Hmm, interesting.
<G> nigelb: do systems like that really do anything though?
<nigelb> G: Well, this is the first such system.
<G> nigelb: I can understand in the context of Four Square, but in software development, I'm not sure
<nigelb> Its about showing your achivements in learning/coding.
<G> nigelb: the way I personally see it (and this is my own warped mind) but I think that sorta stuff is just 0 value fluff
<nigelb> G: Well, karma is 0 value as well.
<nigelb> I'd like to able to show off my skills
<nigelb> Like coderwall
<nigelb> Or like stackoverflow
<nigelb> Hrm, what does fixing bug 184737 involve?
<_mup_> Bug #184737: No tag autocomplete or official list in +filebug extra options <bugtag> <filebug> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/184737 >
<nigelb> I think there's some code I can reuse from the tag autocomplete in the bug's index page
<nigelb> gary_poster: tech-debt helps. I often look into it for bugs to fix :)
<gary_poster> nigelb, cool :-)  please reply with that note then
<nigelb> Whoever did the logpoll documentation has a *lot* of patience.
<nigelb> The diagram is ascii is pretty awesome
<rvba> nigelb: ;)
<deryck> Hi adeuring.  I didn't ping for the standup.  I assumed you weren't around until top of the hour.  Sorry.
<deryck> adeuring, just noticed you here :)
<adeuring> deryck: ok, I'll join
<deryck> adeuring, oh, we're done now.
<adeuring> deryck: yeah, noticed that too :) no problem
<deryck> adeuring, I was apologizing for not inviting you in. :)
<deryck> abentley, adeuring -- ready for skype?
<adeuring> deryck: yes
<abentley> deryck: yes
<deryck> restartingâ¦ you guys don't show for me
<abentley> deryck: I've had that plenty.  Just call me anyway.
<deryck> abentley, we don't hear you.  hear us?
<abentley> deryck: yes.
<abentley> deryck: grr
<deryck> abentley, should we drop you and add you back?
<abentley> deryck: sure.
<mrevell> Hey sinzui?
<danilos> sinzui, hi, did you ever figure out the oneiric librarian problem?
<sinzui> no I did not
<sinzui> matsubara, http://pastebin.ubuntu.com/702790/ the top 12 bugs
<matsubara> thanks sinzui
<deryck> abentley, mumble?
* danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 255 - 0:[#######=]:256
<bac> abentley: are you OCR today?
<sinzui> gary_poster, I do not think anything should high in launchpad-results. cr3 created it we may maintain it, but we never hack on it
<timrc> approximately how long do the code updates on staging take?
<timrc> and is there a notification system for updates so I can plan around it?
<gary_poster> sinzui: oh, we maintain it?  I thought this was kind of an odd triaging experiment for traiging someone elses code
<gary_poster> where "odd" means "I don't care for it" :-)
<sinzui> oh, we do not maintain it
<gary_poster> timrc, I'm not sure, but IIRC staging is having a long outage because of some db issues
<gary_poster> where long means 24 hours AFAIK
<sinzui> gary_poster, This is a case of community vs organisation. Like most canonical project groups. ours is intended to be what we own. but anyone can add their project to someone elses project group.
<timrc> noooo
<gary_poster> normally I assume updates are similar to qastaging: just a few minutes
<sinzui> gary_poster, The consequence of this is that we are triaging their bugs.
<sinzui> This is bad
<gary_poster> agreed sinzui
<sinzui> ~launchpad-results needs to do this, maybe he should remove this project from our project group
<gary_poster> sinzui, if we want the policy of "just triage other peoples bugs as low" I guess that is fine with me, though I think that policy should be publicized.  I was going with the policy of "why the heck is this in front of me?  I'll make arbitrary decisions."  I thought Francis spoke with Marc about this already and for some reason they agreed it would stay where it is
<sinzui> okay
<gary_poster> but I really just agree with you: I wish it were not
<cr3> sinzui: someone suggested that maybe launchpad-results shouldn't be under ~launchpad, but there was a wiki page about projects related to launchpad saying otherwise. I've been meaning to touch base with lifeless about it but haven't gotten around to it
<gary_poster> cr3, the only reason I care in this case is that it means we triage your bugs, because we are supposed to drive the count to zero
<gary_poster> So I stomp on you
<gary_poster> And don't like it :-(
<sinzui> cr3, This is a common problem. The problem is that project groups look like communities, but all the code implies it is organisations. We are still discussing the removal of groups
<cr3> gary_poster: since this is a recurring problem, I could remove launchpad-results being "part of" launchpad-project and maybe we could change that again later
<gary_poster> bigjools, you around?  I've tried to figure out what to do with https://bugs.launchpad.net/launchpad/+bug/868366 but simply don't understand enough.  Could you let me know what you think about triage?
<_mup_> Bug #868366: Diff from current Ubuntu version to current Debian version pending sync missing from +localpackagediffs page <Launchpad itself:New> < https://launchpad.net/bugs/868366 >
<cr3> done, I hope that helps a bit drive your bug count to zero... a good source of inspiration for me too :)
<gary_poster> cr3, thank you. :-) we drive *triage* count to zero every day, which is what this is about.  sadly, not so much for the bug count
<cr3> gary_poster: an unfortunate side effect is that your drive was a bit contagious being part of your group :)
<bigjools> gary_poster: sure thing
<gary_poster> cr3 lol
<gary_poster> thanks bigjools
<cr3> gary_poster: I agreed to try and adhere to your policy of driving the triage to zero for launchpad-results but I don't quite have the same level of vanguard as your team :(
<gary_poster> cr3, you are a team of 1 for this, right?  Understandable, I think.
<cr3> gary_poster: an army of 1! :)
<gary_poster> :-)
 * gary_poster must go babysit during lunch. :-)  biab
<abentley> bac: theoretically, but we have three job interviews today, so I haven't been able to do any.
<deryck> abentley, Skype time.
<abentley> deryck: online.  Even if it doesn't look like it.
<abentley> deryck: had to restart skype.  b
<abentley> deryck: back online.
<sinzui> jcsackett, mumble?
<jcsackett> sinzui: sure.
<deryck> abentley, he called me on regular skype
<deryck> abentley, adeuring adding you back now
<abentley> deryck: cool
<adeuring> ok
<timrc> has any one experienced random failures with syncSources()? I tried syncing a package between two ppas it failed the first time and succeeded the second time.
<timrc> this is on production
<timrc> http://paste.ubuntu.com/702853/
<timrc> it looks like a BadRequest exception was raised
<timrc> hm, actually, there is an oddity and so now I'm leaning towards a bug in my code
<timrc> wait, was looking at wrong ppa, back to thinking it's a problem with launchpad
<sinzui> gmb r=me
<lifeless> yawn, morning
<lifeless> cr3: hi
<cr3> lifeless: hola senor
<lifeless> you invoked me earlier ;)
<cr3> lifeless: you might've noticed I mentionned your name earlier, do you think launchpad-results should be a "part of" launchpad-project?
<cr3> lifeless: the problem we've encountered is that the ~launchpad folks are driving bug triage to zero more quickly than me, so the bugs for launchpad-results are appearing on your radar
<lifeless> cr3: given your goals, definitely :) - that is, becoming a component, with ui in the main tree, backend as a microservice etc.
<lifeless> cr3: what makes that a problem ?
<lifeless> (it sounds like a benefit to me :))
<cr3> lifeless: the "radar" part seems to be a problem where folks on launchpad have felt the need to set the status to triaged but feeling uncomfortable doing so on behalf of a project they are not familiar with
* abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Critical bugs: 255 - 0:[#######=]:256
<cr3> lifeless: it has more than benefited me, it has actually inspired me to keep up with the same level of bug management as launchpad, but I'm afraid about the making others uncomfortable part which makes me uncomfortable as a result :)
 * lifeless blinks
<lifeless> cr3: the lp CHR gets uncomfortable ?
<cr3> lifeless: CHR?
<lifeless> the rota
<lifeless> (the folk doing bug triage do so on a rota basis, sharin the load around)
<cr3> lifeless: still lost me, rota == person on rotation, ie vanguard?
<cr3> lifeless: it seems like they're seeing bugs against launchpad-results and the general reaction is: wtf!
<lifeless> ok
<lifeless> have you read https://dev.launchpad.net/BugTriage ?
<cr3> lifeless: I think I did but I'll read it again. I remember at some point I read something and agree to follow the same process as launchpad, not to make you guys happy as much as because it would benefit the launchpad-results project to follow such a process :)
<lifeless> righto
<lifeless> so the next question was, have you been happy with the results of the lp team doing triage in -results ?
<cr3> lifeless: however, I haven't been quite as fast at triaging bugs as the CHR. there's also the importance of bugs to consider, "high" just for launchpad-results might mean something different than "high" for the launchpad project as a whole
<lifeless> e.g. are they identifying high/critical/low for you sensibly ?
<cr3> lifeless: I've been happy and I've tried to return the favour as best I can, but there's also the above mentionned problem about importance that might need some tweaking
<lifeless> cr3: I think we want to avoid having different rules; but the rules in BugTriage should be broadly compatible.
<lifeless> cr3: if (say) most of the time the triager gets it right, and you need to tweak some to be accurate - thats fine [that happens within LP too]
<cr3> lifeless: ok, so "high" would mean the same to all of us: likely to get attention within the next six months, no matter which team actually does the fixing
<lifeless> yes; the folk doing triage are expected to be *broadly* aware of projects underway across LP (otherwise we could use a bot and make everything not-critical low :P)
<cr3> lifeless: sounds good to me, I removed launchpad-results from launchpad-project earlier but I'll add it again. thanks for making sense of all this! :)
<lifeless> so I don't think we would want - or be able to sustain - different rules for -results. OTOH I think the same rules should work for -results :)
<lifeless> cr3: have you announced on -dev that it was part of -project, and the consequences? (Or would you like Francis or I to do so ?)
<cr3> lifeless: perhaps I should do that myself so that I could answer most questions as a result
<cr3> thanks for offering though, it makes the project feel welcome :)
<lifeless> good!
<lifeless> flacoste: did you see http://h30565.www3.hp.com/t5/Feature-Articles/Innovate-or-Suffer-Slow-Brain-Asphyxiation/ba-p/499 ?
<flacoste> lifeless: i saw, didn't read it yet though
<lifeless> worth a quick read, I think you'll nod in agreement all the way through; some nice physiological references connected to reward programs.
<lifeless> flacoste: you may want to chase those for your upcoming task ;>
<flacoste> lifeless: i actually start researching the issue, and found the evidence-based management work of Pfeffer and Sutton very interesting
<sinzui> jcsackett, gary_poster is bragging about his madd html5 cross-domain scripting skillz. Maybe you want to talk to him about bug 823471 and get his comments on to the bug
<_mup_> Bug #823471: privacy ribbon does not appear on secondary sites <disclosure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/823471 >
 * gary_poster snorts
<jcsackett> sinzui: did you just actually use "mad skillz"?
<jcsackett> :-P
<sinzui> My fingers did
<jcsackett> sure, blame it on the fingers. :-P
<jcsackett> gary_poster: you likely to be free tomorrow morning-ish to talk about cross-domain scripting?
<gary_poster> jcsackett, on call, and not quite clear on what you want, but happy to talk later.  Sure, sounds good.  It would be late morning--I have a call at 9 and one at 10, so 10:30 or 11
<jcsackett> works for me. i'll ping you tomorrow then. :-)
<gary_poster> cool jcsackett, I look forward to it
<sinzui> lifeless, may I call you on skype?
<lifeless> sinzui: I will ping you in ~5, gotta wrap ELOCAL with the cats.
<sinzui> okay
<lifeless> sinzui: ping
<sinzui> calling
<lifeless> sinzui: ping
<lifeless> sinzui: I am back, should you wish to continue
<lifeless> sinzui: I am going to start digging into updating LP for the latest round of oops changes, so just call me on skype whenever
<mwhudson> any reviewer around? https://code.launchpad.net/~mwhudson/launchpad/unconditionally-endInteraction-after-test/+merge/78324
<mwhudson> i guess abentley is no longer on call, seeing as he is not online any more
* StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 255 - 0:[#######=]:256
<StevenK> mwhudson: r=me
<StevenK> mwhudson: Sorry, I was waiting for the diff to update.
<mwhudson> ah right yeah
 * lifeless closes the window
<mwhudson> the update was removing exactly two empty lines :-)
<mwhudson> thanks!
<mwhudson> i wonder why ec2 test remangles the postgres config each time
<mwhudson> surely that could be done in the image?
<mwhudson> ah well
<lifeless> sinzui: (hi)
<sinzui> lifeless, hi. wgrant and I will want to talk to you on skype shortly
<lifeless> ok
<micahg> so, I need to have multiple teams be non-members so as to not grant upload rights, is the only way to do this currently to make a single team containing the others set as owner?
<wgrant> micahg: The owner of a PPA primarily indicates upload rights.
<wgrant> micahg: If people should not have upload rights, they are not the owner.
<micahg> wgrant: sorry, this is for the archive, not PPA
<micahg> owner of a team w/upload rights  doesn't grant upload rights to the archive
<wgrant> "I need to have multiple teams be non-members so as to not grant upload rights"
<micahg> right :)
<wgrant> I'm confused.
<wgrant> Just don't add them to the team?
<micahg> wgrant: so, these teams need to administer, but not gain rights
<micahg> example: DMB and Edubuntu Council can approve edubuntu-dev, but neither team should have upload rights to the edubuntu packageset
<wgrant> You'll need to create a new team, right.
<micahg> is it worth filing a bug for a feature request to have multiple non-member admins?
<lifeless> sinzui: I am here when you want me
<sinzui> lifeless, I will start dinner near. wgrant and wallyworld_ will be ready in about an hour
<lifeless> kk
<lifeless> stub: how goes slonyI-2?
<stub> Migration process all thought through. Mulling over how to best do the migration script so it runs with as little modification on production after testing on staging.
<lifeless> woo
<wgrant> We're not just going to dump and rebuild the slaves?
<wgrant> s/dump/nuke/
<stub> We could, or we could make use of the Slony-ii 2.0 feature that lets us skip that.
<lifeless> what are the tradeoffs?
<stub> 'subscribe, and btw. all the data is already there and up to date so no need to recopy it'
<wgrant> Oh.
<stub> lifeless: If we rebuild the slaves, they are freshly packed. But better to do that one at a time after the update anyway.
<wgrant> Didn't know it could do that.
<lifeless> stub: if a slave is not identical, will slony as a whole barf, or just that node ?
<lifeless> stub: (at the first inconsistent update, I mean)
<stub> lifeless: We won't know about it until the first inconsistent update, at which point that slave will stop replicating.
<lifeless> ok
<wgrant> But that's no worse than would happen if it were inconsistent already.
<stub> The migration scripts are more complex though, as we will be dealing with all 7 nodes instead of just the 2 master nodes.
<wgrant> Will we have to run SSO from a detached slave for the duration, or can we install it live?
<stub> SSO has some sort of read only mode it uses, and the downtime should only be a minute or two anyway.
<wgrant> Right, but it presumably needs an accessible slave :)
<stub> I dunno. It always seemed problematic before and we do so few updates to that system it is always new and exciting.
<wgrant> Heh.
<stub> Might be pulling stuff in from cache.
<wgrant> I don't know what the SSO SLA is these days.
<wgrant> I suspect its availability hardly exceeds Launchpad's new process, though.
<lifeless> hmm, need a new storm code drop
<lifeless> not today.
<wgrant> 0.19's about to be released, did you see?
<wgrant> lifeless: Also, set_up_tacfile_logging is terribly buggy and responsible for a number of poppy-sftp issues.
<wgrant> So any attempt to remove it would be most welcome.
<wgrant> Oh, 0.19 is actually out now.
<StevenK> They've actually made a storm release?
<wgrant> Not quite a year since the last one.
<lifeless> sure, I've made some notes about how in the bug
<lifeless> I think a good twisted component is the first step
<lifeless> .
<StevenK> wgrant: You were saying something about parent class ordering for my branch?
<wgrant> Oh right.
<wgrant> Bear with me, Optus is only routing some of the DC subnets.
<wgrant> StevenK: BasePesronEditViewWidget? Why not just put it in BasePersonEditView?
<wgrant> Otherwise it should be something about checking renames, not BasePersonEditViewWidget.
<StevenK> Because TeamEditView also needs it
<wgrant> Also, remember that the MRO is first base class to last.
<wgrant> So you probably want that setUpWidgets mixin early in the base list.
<StevenK> And I wasn't sure about the schema set in BasePersonEditView
<StevenK> wgrant: So if you think using BasePersonEditView is okay for both, I'm happy to try it.
<wgrant> I'm not sure what's in it.
<wgrant> But the current name is silly, and it would be handy if we didn't have a dozen base classes shared between those views when we only need one.
<StevenK> A schema, fields, action_save and next_url
<wgrant> StevenK: Perhaps a PersonRenameFormMixin?
<StevenK> wgrant: I've put it into BasePersonEditView, what could possibly go wrong.
<StevenK> Bah
<StevenK> BasePersonEditView defines field_names as [], and TeamFormMixin as a large array
#launchpad-dev 2011-10-06
<wallyworld_> just saw that steve jobs has died
 * StevenK adds 1 to his internal number of channels that have mentioned it.
<wallyworld_> lifeless: did skype drop out or is it just me?
<StevenK> wgrant: Can haz another look at my MP?
<wallyworld_> thumper: ping!
<lifeless> wgrant: sinzui: did skype hiccup ?
<wgrant> Oh :(
<wgrant> So yeah, I think we know what we're doing.
<wgrant> Almost.
<wgrant> Multipillar bugs with disclosure views are still a bit of an issue.
<wgrant> But I think we have better ideas now.
<wgrant> Email probably works.
<wgrant> I need to experiment with how they interact with policies.
<wgrant> They may make it easier.
<sinzui> wgrant, yes. they do need more work, but I feel we have a path to solve the issue
<wgrant> I am tempted to go with policies now and then see how multipillar bugs fall out.
<sinzui> wgrant, I was thinking of adding a report task to the kanban. How many pillars share private bugs?
<wgrant> sinzui: I was running reports on Tuesday to work that out.
<wgrant> sinzui: Using all historical data, too.
<wgrant> sinzui: (any bug that is private now, or has ever had the visibility changed)
<wgrant> sinzui: I didn't end up finishing them, but I can finish the queries if you do want to know this.
<sinzui> oh, I had not considered that
<sinzui> I think we should know this because I am certain someone will ask us
<wgrant> sinzui: It is relevant because many security bugs end up public.
<wgrant> sinzui, lifeless: I have a set of scripts to populate an empty DB.
<wgrant> With celebrities.
<wgrant> And an initial user.
<sinzui> oh sweet
<wgrant> And then another one to create Ubuntu and set it up for Soyuz.
<wgrant> I wrote them around two years ago, but updated them a few months ago.
<wgrant> https://code.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch\
<sinzui> mailing lists require a script since it is not 100% in the db
<wgrant> http://bazaar.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch/revision/10987 is the guts of it.
<wgrant> karma and celebrities are the ugly bits.
<wgrant> No.
<wgrant> That would be nice, but slow and require lots of test fixes.
<wgrant> right.
<wgrant> These create a minimal DB that you can use a harness on to create more stuff.
<wgrant> wallyworld_, lifeless, StevenK: eg. I have my base bootstrap-lp-db script which just creates minimal celebrities and a user for the caller, then make-ubuntu-sane to create sensible Ubuntu series. I imagine there would be lots of scripts like make-ubuntu-sane.
<wgrant> We use make initscript-start on production.
<wgrant> Not make run.
<wgrant> *Especially* using the factory.
<StevenK> wgrant: Can haz review?
<StevenK> wgrant: bootstrap-lp-db was what lifeless was talking about it, which sounds like a good idea to me.
<wgrant> Not quite what lifeless was talking about.
<wgrant> I believe lifeless was talking about a script to generate something like the current sampledata.
<wgrant> With bugs and projects.
<wgrant> But not with SQL.
<lifeless> yes
<StevenK> wgrant: I was hoping my refactoring could remove more lines than it added, but alas.
<thumper> wallyworld_: pong
<wallyworld_> thumper: that utf-8 branch has a test failure on ec2 but not locally and my encoding foo is not that great https://pastebin.canonical.com/53926/
 * thumper looks
<wallyworld_> thumper: i'll paste the code snippet
<wallyworld_> thumper: https://pastebin.canonical.com/53927/
<thumper> and it works locally?
<wallyworld_> the string 'hello \xce\xa3' is the correct non-unicode representation of the unicode string
<wallyworld_> yes
<wallyworld_> in the debugger, both the unicode and non-unicode versions appear correctly as hello (sigma)
<wallyworld_> so i know the strings are correct
<thumper> damn
 * thumper has no idea
 * wallyworld_ also has no idea
<thumper> wallyworld_: ask someone who uses utf-8 more, like jtv or jelmer :)
<wallyworld_> will do
<wgrant> lifeless: http://paste.ubuntu.com/634326/
 * StevenK prods wgrant about his MP so he can land it
<wgrant> StevenK: Much better, thanks.
<nigelb> Hrm, no stub yet :(
<StevenK> stub has been awake for hours.
<StevenK> wgrant: Thanks! Tossing it at ec2.
<nigelb> StevenK: I managed to screw up my MP enough that I had to redo it :D
<nigelb> So, now I need approval again :(
<StevenK> nigelb: Been there, done that.
<nigelb> StevenK: That's comforting :)
<stub> nigelb: Where is the mp?
<nigelb> stub: Nevermind. I just noticed you approved it.
<nigelb> I must've lost the email :(
<nigelb> lifeless: ohai, I see that you're OCR today, could you land something for me? :)
<lifeless> nigelb: maybe; I'm EODish
<StevenK> Point me at it
<nigelb> lifeless: ah
<nigelb> StevenK: https://code.launchpad.net/~nigelbabu/launchpad/create-description-5283/+merge/78220
<nigelb> lifeless: It Thu that you start really early and end early?
<StevenK> nigelb: Is there a bug for that work?
<nigelb> StevenK: ah, right. Let me link.
<StevenK> It's just a db patch, so it doesn't matter much
<nigelb> StevenK: (In case it isn't apparent 5283)
<nigelb> oh no. Someone removed that feature
<wgrant> Not quite.
<nigelb> when linking a bug it would suggest the number that's in the branch name
<wgrant> It only works with numbers of 5 or more digits.
<nigelb> bah!
<wgrant> To minimise false-positives.
<StevenK> lol
<nigelb> StevenK: linked :)
<nigelb> I'm going to change the dev wiki db patch page to say the right year.
<StevenK> nigelb: lp-landed to db-devel
<StevenK> r11051 of db-devel
<nigelb> StevenK: that was fast.
<nigelb> You didn't need a test run?
<StevenK> It's a DB patch!
<nigelb> heh
<StevenK> What could possibly go wrong, etc, etc
<nigelb> Famous last words etc
<wallyworld_> StevenK: i had a db-patch fail last week. it got picked up by ec2 luckily. it was a combination of not null column and a trigger processing issue. so it depends on the nature of the change whether it should go through ec2
<nigelb> wallyworld_: Its a new column, probably less risk :)
<wallyworld_> nigelb: understood. i was just making a point about db patches sometimes needing ec2 :-)
<lifeless> nigelb: earlier , not really early atm
<lifeless> nigelb: but ye
<lifeless> s
<lifeless> wallyworld_: everything gets ec2 by default :)
<wallyworld_> lifeless: yes. i was trying to be gentle :-)
<wallyworld_> with my assertion that ec2 be used
<StevenK> wgrant: So, are you happy enough for populate-bprc to land, and if it starts being crap, we change the query?
<wgrant> StevenK: I'd prefer if we knew it wasn't crap, and that it was the way we wanted to go.
<StevenK> wgrant: My timing of the query was 'acceptable', TBH
<StevenK> And it didn't kill DF
<StevenK> wgrant: So given the two choices of "It will make me sad" or "I won't abide, and will revert it." is it? :-)
<wgrant> StevenK: Well, we need to decide whether we want it in our main DB right now.
<wgrant> And whether the query doesn't suck.
<StevenK> I thought we had these arguments already?
<StevenK> 51 polls have been created this year. I am disappoint.
<lifeless> StevenK: land
<nigelb> StevenK: Kill it :D
<StevenK> lifeless: land populate-bprc?
<nigelb> StevenK: :(
<nigelb> buildbot failed.
 * StevenK waits for SSO
<StevenK> Huh, weird.
<StevenK> lifeless: land populate-bprc?6170 tests
<StevenK> 1334 passed
<StevenK> 76 failures
 * StevenK waits for SSO[B[B
 * StevenK glares at his mouse
 * wgrant glares at StevenK.
<wgrant> Ah, I see the buildbot failure is noticed already.
 * wgrant waits for SSO.
<wgrant> Heh, you broke all the triggers.
<StevenK> Ah. Shall I revert?
<wgrant> Yes.
<wgrant> And ec2 things in future...
<StevenK> Doing so.
<nigelb> What'd I break?
<StevenK> All of the triggers.
<nigelb> Ugh.
<nigelb> How?
<nigelb> Or rather, how do I fix.
<StevenK> wgrant: Reversion tossed at PQM.
 * StevenK breaks PQM into tiny pieces and resubmits
<wgrant> nigelb: The lp_person table and the lp_mirror_person_* triggers in trusted.sql. You need to replace them in a patch, like you did with bugtask.statusexplanation's FTI trigger.
<wgrant> Wow.
<wgrant> They depend on column order.
<wgrant> stub: Make them go away :(
<StevenK> Haha
<StevenK> lp_person and lp_mirror_person_* are parasites.
<wgrant> Actually.
<wgrant> We rerun trusted.sql each time.
<wgrant> So you can probably just edit the functions in-place.
<StevenK> wgrant: Revert is r11053
<StevenK> wgrant: It has CREATE OR REPLACE TRIGGER?
<stub> yes, they can be edited in place. But we can't alter the lp_* table definitions.
<stub> They will go away when ISD obtains that information through other channels, or decides it isn't needed at all.
<wgrant> But they've got no motivation to fix this :/
<stub> nigelb: How is your PL/pgSQL? I might need to take over that patch.
<stub> They do, as their schema updates are a pain in the arse because their databases are tied into ours.
<nigelb> stub: I suck :)
<nigelb> stub: I'm glad to defer to you :)
<nigelb> Hrm, awkward timing for that quit.
<nigelb> :P
<nigelb> wgrant / StevenK - I seem to be opening quite a can of worms lately :)
<stub> bouncy bouncy
<nigelb> stub: In case you missed that, please take over :)
<stub> nigelb: ok
<nigelb> stub: hey, can you let me know once its done, so I can work on the follow up branches? :)
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 262 - 0:[#######=]:256
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 262 - 0:[########BOOM
<lifeless> whats the status of lazr-js ?
<wgrant> Merged into LP and abandoned.
<wgrant> But I think launchpad-results still uses a tiny bit of it.
<lifeless> wondering whether to mass close its bugs
<lifeless> or move them to lp
<lifeless> or ...
<wgrant> Ignore them forever :)
<lifeless> jelmer: bug 515918 - do we close it ?
<_mup_> Bug #515918: package_name and suite not passed to dailydeb <lp-soyuz> <recipe> <Launchpad itself:Triaged> < https://launchpad.net/bugs/515918 >
<nigelb> wgrant: I think those numbers are lying.
<wgrant> nigelb: Which?
<nigelb> wgrant: /topic
<nigelb> Mainly because it doesn't account for the fact that new bugs are opened and old ones are closed.
<wgrant> Well, it depends what you want to measure.
<lifeless> nigelb: its not reporting rate
<nigelb> lifeless: shouldn't it?
<nigelb> Or should it be a fraction with closed numbers as well to give a comparison.
<nigelb> Now it looks like nothing's happening :P
<lifeless> nigelb: its not, and never has been, a fraction
<lifeless> nigelb: perhaps I'm missing your point
<wgrant> Hm, only 57 launchpad-project criticals from before this year.
<nigelb> lifeless: Well, wht I'm trying to say is, well, it looks like work isn't being done. Which is wrong.
<wgrant> Work is being done, but progress is not being made.
<wgrant> The intention is to drive the critical queue to 0.
<wgrant> That's what that graph measures.
<nigelb> Ok, that explains it :)
<wgrant> (it was ~250 when it started)
<wgrant> Actually, started at 210.
<nigelb> I should agree to take a pie to the face at spring UDS.
<nigelb> Seeing the pace at which its going I may never have to :D
<wgrant> Yep :)
<wgrant> Also, Launchpadders aren't really at UDS any more, so there'll hardly be anyone to pie you :P
<nigelb> I thought there's always a small representation?
<nigelb> Anyway, there's ex-launchpadders in plenty.
<nigelb> I'm sure joey or kiko or jml would be happy to have the pleasure :P
<jml> wgrant: that's great (the only 57)
<jml> wgrant: although, huh, we're good at adding critical bugs, it seems
<wgrant> jml: I thought it would be well over 100.
<wgrant> But apparently not.
<wgrant> jml: Note that lots are old bugs, like timeouts that only appeared as we dropped the timeout.
<wgrant> There are 81 timeout bugs, and I would be surprised if even 10 of them were actually new bugs.
<jml> ahh right.
<jml> what's the problem with those? are they unusually hard to fix?
<wgrant> But I believe flacoste is currently analysing this sort of thing.
<wgrant> No.
<wgrant> I think the maintenance squads are suffering from a combination of taking too many big tasks at once, and possibly having too many people leaving :)
<nigelb> leaving?
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba* (allenap) | Critical bugs: 262 - 0:[########BOOM
<lifeless> jml: wgrant: the remaining old timeouts are hard to fix
<lifeless> we nabbed most of the low hanging fruit already
<wgrant> lifeless: Some of them are, yes. But lots are not.
<lifeless> wgrant: can you parse bug 397165 for me
<_mup_> Bug #397165: Upload queue changes should be more restricted <lp-soyuz> <soyuz-upload> <Launchpad itself:Triaged> < https://launchpad.net/bugs/397165 >
<wgrant> lifeless: The web UI is pretty good (I think everything matches what's there, except possibly that Rejected->Accepted might not be exposed), but the internal API and command-line tools allow many transitions that should not be permitted.
<lifeless> bug 405805 might qualify as critical even
<_mup_> Bug #405805: Upload processing may reach a transaction deadlock when closing bugs <lp-soyuz> <soyuz-upload> <Launchpad itself:Triaged> < https://launchpad.net/bugs/405805 >
<wgrant> lifeless: The upload processor doesn't really do transactions.
<wgrant> For example, it's the reason "zopeless" mail isn't queued until the end of the transaction.
 * StevenK tries to figure out the xx-ppa-workflow.txt error
<StevenK> I think the test is amusing the buggy case of team renaming.
<stub> Does a branch rolling back a rollback get a rollback= stanza in the commit message?
<lifeless> no
<lifeless> the rollback stanza is for the qa tagger
<lifeless> to tell it that it fixes the bad-revision-xxxx
<lifeless> so the range of revisions that can't be deployed is known
<wgrant> We really need to fix the branch scanner.
<wgrant> Needs to be faster, probably needs to do partial progressive scans of fresh branches (like we do with codeimports), and needs to destroy branchrevision :(
<jelmer> wgrant: progressive scans would be a good start
<wgrant> It can fortunately be done manually, but it's awkward and slow.
<wallyworld_> jelmer: are you free to help me with a unicode/utf-8 problem?
<jelmer> wallyworld_: hi
<jelmer> wallyworld_: sure, what's up?
<wallyworld_> hi. i'm not an expert sadly
<wallyworld_> here's the issue - i have changed mp and branch emails to utf-8 encode the diff
<wallyworld_> a test fails on ec2 but passes locally
<wallyworld_> i'll pastebin the test, just a sec
<wallyworld_> the test: https://pastebin.canonical.com/53927/, the failure: https://pastebin.canonical.com/53926/
<wallyworld_> i'm not sure why is passes locally and fails on ec2
<wallyworld_> the strings are correct - in my debugger, both the unicode and utf-8 versions are correctly represented
<jelmer> wallyworld_: have you tried running locally with LC_ALL=C ?
<wallyworld_> jelmer: what does that option do?
<jelmer> wallyworld_: It's an environment variable
<wallyworld_> sure :-) what does it affect?
<jelmer> wallyworld_: It's the variable with the current language and encoding
<wallyworld_> ok. and i assume that's what ec2 uses?
<jelmer> wallyworld_: it influences, among other things, how bzr will encode things for the terminal
<wallyworld_> the environment we run under ec2 i mean
<wallyworld_> ok
<jelmer> wallyworld_: We've seen some issues related to this when bzr is run with LC_ALL=C (ascii only) versus LC_ALL=en_AU.UTF-8 (UTF-8)
<wallyworld_> ok. i'm trying it now
<wallyworld_> jelmer: it still passes locally
<jelmer> wallyworld_: the odd thing appears to be that you specify decode=True (which I assume means "return unicode") but you're getting back a plain string
<wallyworld_> jelmer: decode=True tells the Python email library to reverse any encoding used when the message was constructed
<wallyworld_> yes, wonder why a plain string is returned
<wallyworld_> i'll have to dig around the email libs a bit i guess
<wallyworld_> jelmer: btw, i only modified that test. the test used to only use non-unicode strings and i added the u'hello ...' to test the utf-8 encoding
<wallyworld_> thanks for the help btw
<jelmer> wallyworld_: Sorry, still digging..
<wallyworld_> oh ok. thanks!. i've fired up the debugger and am looking to see if i missed anything
<jelmer> wallyworld_: do you understand what the :3 comes from?
<jelmer> I mean, not that the result is different, but why does it come up with ":3" ?
<wallyworld_> jelmer: the unicode string i used is u'hello (sigma)' where (sigma) is the greek letter
<wallyworld_> i guess that's what the decoded string comes out as on ec2
<jelmer> but why would sigma become :3 ?
<wallyworld_> i wish i knew
<jelmer> It would be more understandable if it replaced it with \uffff or something
<wallyworld_> yeah. locally, both strings print as the correct thing
<wallyworld_> by "print", the debugger does a str() on them any they display correctly
<cjwatson> anyone want to comment on the thoughts I posted in bug 572128?
<_mup_> Bug #572128: Ubuntu Archive translations are missing Index metadata file <Launchpad itself:New> <debmirror (Ubuntu):Confirmed> < https://launchpad.net/bugs/572128 >
<danilos> jelmer, wallyworld_: locale stuff can also support transcription to ASCII, though you usually have to explicitely ask for it
<wallyworld_> danilos: i can't correlate what's happening here though. why would it pass locally and fail on ec2?
<danilos> though, this doesn't sound like a reasonable transcription
<danilos> wallyworld_, I would attribute it to ec2 developing feelings towards you (iow, sounds weird, yes :))
<wallyworld_> i have a unicode string u'hello Î£' which is converted to utf-8, encoded as quoted-principal, decoded, and compared
<wallyworld_> all the encoding etc happens in the python email libs
<wallyworld_> i'm at a loss
<wallyworld_> danilos: i have feelings for ec2 as well but it's not love
<nigelb> heh
<danilos> wallyworld_, judging from how it's working out for you, I'd say it's mutual :P anyway, I did see some differences between python on ec2 image and recent releases; have you had someone try it out locally on lucid as well?
<wallyworld_> danilos: ah no. but that's a good idea, thanks. something environmental sounds like a reasonable culprit. and if it is, i wonder how to solve it
<danilos> wallyworld_, "unset LANG LC_ALL" locally?
<wallyworld_> i guess get it to fail first and go from there :-)
<wallyworld_> danilos: jelmer suggested LC_ALL=C which i tried
<wallyworld_> i'll try unset
<danilos> wallyworld_, check with "locale" what the resulting settings are (sometimes LANG can override stuff on GNU systems)
<cjwatson> LANG never overrides LC_ALL with GNU software
<cjwatson> some crappy non-GNU software gets it wrong though :)
<StevenK> I can never remember the order
<wallyworld_> LANG=en_AU.UTF-8 for me
<StevenK> LC_ALL LANG LANGUAGE ?
<danilos> cjwatson, ah, ok, so how about LANGUAGE? I know there's a specific GNU extension
<cjwatson> LANGUAGE > LC_ALL > LC_* > LANG
<cjwatson> although only for the purposes of message categories; for all other locale categories, LC_ALL > LC_* > LANG
<wallyworld_> with all those unset, it still works locally
<danilos> wallyworld_, just trying out your test gives me https://pastebin.canonical.com/53948/, but I guess that's a bug you are fixing :)
 * wallyworld_ cries
<cjwatson> perhaps EC2 has a non-existent locale set in its environment or something
<wallyworld_> could be, i'll have to look at how our images are generated
<StevenK> ec2test calls os.environ.pop() on LANG, LC_ALL and LC_TIME
<wallyworld_> StevenK: danilos suggested it may be a lucid thing
 * wallyworld_ needs to reinstall lxc again
* jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba* (allenap), jcsackett | Critical bugs: 262 - 0:[########BOOM
<deryck> Morning, all.
<rvba> Hi jcsackett!
<rvba> Morning deryck.
<jcsackett> morning rbva. :-)
<jcsackett> er, rvba. :-P
<jcsackett> sinzui: we're good to land the branch that kills the privacy banner feature flag now, right?
<sinzui> yes
<jcsackett> excellent.
<jcsackett> out it goes.
<flacoste> gary_poster, benji: (not urgent) can one of you have a look at https://code.launchpad.net/~mwhudson/launchpad/permit_timeout_from_features-on-participation-bug-861510/+merge/78355
<benji> flacoste: I will momentarily.
<gary_poster> thanks benji
<benji> np
<gary_poster> jcsackett I am finished with my regular morning calls, and can talk when you are ready.
<gary_poster> no rush
<jcsackett> gary_poster: awesome. i am actually free now, if you like. potentially this will be a short conversation. :-P
<jcsackett> mumble work for you?
<gary_poster> heh ok cool jcsackett.  skype usually works better, but I can do the mumble thing.  1 sec and I'll get on
<danilos> mrevell, btw, any idea how did this break: bug 826634?
<mrevell> bug 826634
<mrevell> Come on robot, give me a link
<mrevell> danilos, Crumbs. I've no idea
<bigjools> mrevell: amazing that nobody asked for wikis on your survey
<danilos> mrevell, oh, so the help files are on translations.launchpad.net and the links point to launchpad.net
<mrevell> danilos, Oh, strange
<mrevell> bigjools, Yeah, well, it was only a handful of people replied. I bet if we asked more people then wikis would come up.
<cjwatson> is there a way to stop /var/tmp/archive/ from being cleaned up after failing archivepublisher tests?
<bigjools> cjwatson: post-mortem debug
<cjwatson> as in pdb?  ok
<bigjools> yeah, there's a test runner option
<bigjools> or you can remove the cleanup code
<sinzui> jcsackett, mumble?
<jcsackett> sinzui: sure, one moment.
<benji> flacoste: I don't know if you want to engage further but FYI: I just added a comment to https://code.launchpad.net/~mwhudson/launchpad/permit_timeout_from_features-on-participation-bug-861510/+merge/78355 (didn't feel good about approving it, so I just commented)
<flacoste> benji: ok, thanks
* rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 262 - 0:[########BOOM
<gary_poster> bigjools, if you have a moment to look at https://bugs.launchpad.net/launchpad/+bug/869185 I'd appreciate it.  Please triage it, or if it is faster to tell me what to do about it, do that instead and I'll follow up.
<_mup_> Bug #869185: P-a-s file ignored (even on cocoplum) <Launchpad itself:New> < https://launchpad.net/bugs/869185 >
<bigjools> gary_poster: ok
<gary_poster> thank you much
<bigjools> gary_poster: gah it's a mix of more than one bug
<gary_poster> oh :-/
<bigjools> gary_poster: I can explain for your delectation and delight if you like
<gary_poster> bigjools, I'm happy to listen, sure :-)
<bigjools> do you know about the P-a-s file?
<gary_poster> and learn even
<gary_poster> no
<bigjools> ok, it's used when creating builds for sources and acts as an override to whatever the source thinks it should build on
<bigjools> so it can say !armel to exclude armel for example
<gary_poster> ok
<bigjools> the ubuntu guys rely on it so that they can take packages from Debian with no changes
<gary_poster> I figured it was somehinng like that from context
<bigjools> so bug #1 is that when we sync files from Debian using the new derived distros stuff, it doesn't use the P-a-s file at all
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <GenOS:In Progress by gen-os> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:In Progress> <The Linux OS P
<bigjools> argh go away mup
<nigelb> heh
<gary_poster> :-)
<gary_poster> I se
<gary_poster> e
<bigjools> bug two (ha! screw you mup) is that it also didn't appear to work for a regular upload of the libx86 package
<gary_poster> :-)
<gary_poster> ah ok.  So the first is a non-critical feature bug (?) and the second is a critical regression?
<bigjools> he's talking about cocoplum since that's where the ubuntu uploads go
<gary_poster> gotcha
<bigjools> so I need to split this into two
<bigjools> bear with me
<gary_poster> cool
<bigjools> gary_poster: ok I updated the original bug a bit and reference the new one in it
 * gary_poster looks
<gary_poster> bigjools, why is first one (869185) not regression/critical?
<bigjools> gary_poster: I was just thinking about that
<bigjools> it's only a single package, hence my hesitation
<gary_poster> right
 * gary_poster leaves the decision to you :-D
<bigjools> thanks :)
<gary_poster> fwiw, based on your explanation, if you had left it to me I would have said critical
<gary_poster> (on the basis of "the rules")
<bigjools> rules are there to bend
<gary_poster> :-) fair enough
<bigjools> I am just finding out if it's more widespread, if so it's a critical regression
<gary_poster> ok
<bigjools> no idea *how* since nobody changed that code lately ...
<cjwatson> I think I have a fix for bug 572128; would anyone care to have a *cough* post-implementation chat about it? :-)
<_mup_> Bug #572128: Ubuntu Archive translations are missing Index metadata file <Launchpad itself:New> <debmirror (Ubuntu):Confirmed> < https://launchpad.net/bugs/572128 >
<gary_poster> cjwatson I was just seeing if danilos were around to look at that bug
<gary_poster> I don't think he is :-P
<cjwatson> I posted general approach in the bug comments, and http://paste.ubuntu.com/703479/ appears to pass tests for me
<bigjools> cjwatson: I'd love to but kinda busy, I can chat tomorrow if you don't find anyone before then
<cjwatson> sure, that will be fine, pretty tired now anyway
<bigjools> I hear ya
<bigjools> cjwatson: since you're here, are you aware of any other failures to use P-a-s for uploaded packages?
<gary_poster> cjwatson, from a pure triaging perspective, is this a regression
<cjwatson> I'll assign myself in the meantime
<gary_poster> (572128) on the LP side
<cjwatson> gary_poster: yes, albeit one that's my fault
<gary_poster> :-) ok
<gary_poster> I'll triage as such.  thanks cjwatson
<cjwatson> (and a regression by cooperation between LP and mirroring tools)
<cjwatson> bigjools: I haven't *noticed* any, although that doesn't necessarily say much - doko is much better informed here than I am
<cjwatson> due to his work on test rebuilding
<bigjools> yeah, I was trying to reach him
<cjwatson> TBH it's the kind of thing I habitually write off since historically it's been lost in the noise of real failures
<bigjools> cjwatson: the other side of the coin - do you know any that are ok that are in p-a-s?
<cjwatson> now that we're getting those down to reasonable levels ...
<cjwatson> bigjools: grub2 was fine last time I uploaded it, although it also has an explicit Architecture line
<bigjools> ok - I wonder if p-a-s has a fault itself for libx86 then
<cjwatson> easy to check
<cjwatson> %libx86: !armel !hppa !ia64 !m68k !mips !mipsel !powerpc !sh4 !sparc  # <sys/io.h>
<cjwatson> looks right to me
<bigjools> me too
<bigjools> weird
<cr3> benji: quick lazr.restful/wadllib question for you, does this error ring a bell: UnsupportedMediaTypeError: This resource doesn't define a representation for media type text/plain
<benji> cr3: I don't think I've seen that one before but I would suspect there's an adapter you can provide that would make it work.
<cr3> benji: it seems to happen when an object is returned with a 304
<cr3> benji: the same api url, when returning a 200, works fine. but, when it has not been modified, that's when I get: https://pastebin.canonical.com/53985/
<benji> cr3: that's weird; since a 304 can't have a body then there's no reason to be looking for a representation (or for it to have a content-type at all)
<cr3> benji: indeed, the appserver says it's returning 0 bytes, the representation might be in the header though
<benji> cr3: well, 304 is Not Modified so there shouldn't be any representation anywhere
<jelmer> bug 369752
<_mup_> Bug #369752: Don't allow branch type to be selected on branch add form <easy> <lp-code> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/369752 >
<sinzui> jcsackett, mumble?
<sinzui> wallyworld__, https://launchpad.net/launchpad/+milestone/3.0
<wallyworld__> wgrant: here's the branch lp:~wallyworld/launchpad/utf8-encode-diffs-861979 and here's the test lp.code.mail.tests.test_branch.TestBranchMailerDiff.test_generateEmail_with_diff
<wallyworld__> wgrant: https://pastebin.canonical.com/53948/
* wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugs: 262 - 0:[########BOOM
<wgrant> wallyworld_: AssertionError: 'hello \xce\xa3' != 'hello :3'
<wallyworld_> wgrant: yes, that's what ec2 says too
<wallyworld_> so it's a lucid vs oneiric issue
<wgrant> Yep.
 * wgrant adds D
<wgrant> -D
<wallyworld_> bollocks
<wallyworld_> i could force the encoding to base64 to see if that helps
<wgrant> Content-Transfer-Encoding: quoted-printable
<wgrant> hello =3A3
<wgrant> Is that as expected?
<wgrant> Apparently not.
 * wallyworld__ sighs. another unity/compiz crash
<wgrant> you don't just restart unity in that case?
<wallyworld__> wgrant: it took out quassel as well this time
<wallyworld__> everything went belly up
<wgrant> Hah
<wallyworld__> wgrant: i've pushed a change, can you try again?
<wgrant> wallyworld__: Have you seen the hack at the top of sendmail.py?
<wgrant> To force quoted-printable?
<wgrant> I wonder if that might be relevant.
<wgrant> Because set_payload is indeed writing =3A3, which is :3.
<wallyworld__> wgrant: you mean in encodeOptimally()
<wallyworld__> ?
<wgrant> wallyworld__: No, at module load time.
<wgrant> It overrides the default encoidng.
<wgrant> del Charset.CHARSETS['utf-8']
<wgrant> Charset.add_charset('utf-8', Charset.SHORTEST, Charset.QP, 'utf-8')
<wgrant> Charset.add_alias('utf8', 'utf-8')
<wallyworld__> oah, didn;t notice that
<wallyworld__> well that sucks
<wgrant> http://paste.ubuntu.com/703671/
<wgrant> That is indeed relevant.
<wgrant> set_payload actually crashes until you do that.
<wgrant> Interestingly, in python2.7 it doesn't crash initially.
<wgrant> It encodes UTF-8 with base64.
<wallyworld__> wgrant: so on python 2.6, base64 encoding of utf-8 crashes?
<wgrant> And with the charset hack installed, 2.7 encodes to =CE=A3
<wgrant> wallyworld__: It seems so.
<wallyworld__> but i just tried pythin 2.6 and base64 locally and it worked
<wgrant> wallyworld__: I wonder if set_payload doesn't actually like unicode in 2.6.
<wgrant> Perhaps it takes the charset that the payload is encoded in, not a charset in which the payload should be encoded.
<wgrant> The docs are not clear.
<wgrant>         if str(charset) != charset.get_output_charset():
<wgrant>             self._payload = charset.body_encode(self._payload)
<wgrant> I think the payload is meant to be pre-encoded.
<wallyworld__> argh, ffs. another unity crash
<wallyworld__> i don't think the payload is meant to be pre-encoded
#launchpad-dev 2011-10-07
<wallyworld__> it works locally without doing that
<wgrant> 2.7's set_charset (called by set_payload) encodes it if it's a unicode.
<wgrant> 2.6 doesn't have that check.
<wallyworld__> huh? my 2.6 does
<wallyworld__>         if isinstance(self._payload, unicode):
<wallyworld__>             self._payload = self._payload.encode(charset.output_charset)
<wgrant> Ahhh.
<wgrant> Lucid's does not.
<wgrant> I don't think I have 2.6 installed on oneiric... let me check.
<wallyworld__> it's all thumper's fault
<thumper> it must be
<wallyworld__> it was going to be a quick patch to help out my old kiwi mate and look what it's turned into
<StevenK> wallyworld__: Welcome to Soyuz.
<StevenK> Oh. Wait.
<wallyworld__> :-)
<wgrant> wallyworld__: See the latest rev on http://bazaar.launchpad.net/~python-dev/python/2.6/changes?filter_file_id=43136%406015fed2-1504-0410-9fe1-9d1591cc4771%3Apython%252Ftrunk%252FLib%252Femail%252Fmessage.py
 * wallyworld__ looks
<wgrant> http://bazaar.launchpad.net/~python-dev/python/2.6/revision/42357
<wgrant> Crucially, from the old email.message docs:
<wgrant> 139
<wgrant>  
<wgrant>       The message will be assumed to be of type :mimetype:`text/\*` encoded with
<wgrant> 140
<wgrant>  
<wgrant>       *charset.input_charset*.
<wgrant> The new:
<wgrant> 139
<wgrant>       The message will be assumed to be of type :mimetype:`text/\*`, with the
<wgrant>  
<wgrant> 140
<wgrant>       payload either in unicode or encoded with *charset.input_charset*.
<wallyworld__> wgrant: well, the only thing to do then is to assume it's broken and code around it in our stuff
<wallyworld__> what's the policy with upgrading python on our prod environments? i think it will just be easier to hack around it in our code?
<wgrant> Not hack around it.
<wgrant> The old Python behaviour is not buggy.
<wallyworld__> you sure? "Also makes the way in  which unicode gets encoded to quoted printable for other charsets more  sane (it only worked by accident previously)"
<wallyworld__> seems like they made some changes to fix broken/unintuitive behaviour
<wgrant> Right, but set_payload taking Unicode is behaviour that is new in 2.6.6 or 2.6.7
<wgrant> Passing Unicode into Lucid's version violates expectations set out in the docstring.
<wgrant> It's certainly an unintuitive response to interface abuse, but it's still interface abuse :)
<wallyworld__> which means i need to encode outside of set_payload, which is hacking around it
<wgrant> It's not hacking around it.
<wgrant> It's using set_payload properly.
<wgrant> set_payload, until a year ago, was intended to be given encoded data.
<wallyworld__> except that it accepts a charset parameter which implies it will be doing the encoding based on that charset
<wgrant> The only reason unicode sometimes worked is because Python 2.x's Unicode support is terrible.
<wgrant> How does it imply that?
<wgrant> It
<wgrant> indicates the encoding of the message.
<wgrant> But the docstring is pretty clear.
<wgrant> It expects the message to be in text/*, encoded in the given encoding.
<wallyworld__> sure.
<StevenK> So it wants UTF-8, not unicode
<wallyworld__> perhaps i was misusing the term "hacking"
<wallyworld__> i was being loose with my definition
<wgrant> Perhaps.
<wallyworld__> i just meant i have to make changes to accommodate the old interface
<wgrant> Right.
<wallyworld__> even though the new one supports what we want
<wallyworld__> is was using hacking the the sense of "code hacking" ie code writing
<wallyworld__> anyways
<wgrant> Anyway, easy fix, hopefully.
<wallyworld__> yep
<wallyworld__> so long as my system stays up
 * wallyworld__ wishes he had an SSD
<wallyworld__> wgrant: can you try it now?
<wallyworld__> wgrant: i wasn't sure whether to use content.encode(charset) or content.encode(charset.output_charset). i used the first one
<wgrant>   Running:
<wgrant>  lp.code.mail.tests.test_branch.TestBranchMailerDiff.test_generateEmail_with_diff
<wgrant>   Ran 1 tests with 0 failures and 0 errors in 0.537 seconds.
<wgrant> wallyworld__: ^^
<wallyworld__> wgrant: excellent! thanks for your help
<wgrant> wallyworld__: How does this work with non-UTF-8 files?
<wallyworld__> wgrant: it works fine. there's other tests which use plain ascii diffs
<wgrant> ASCII diffs are also UTF-8.
<wallyworld__> perhaps i misunderstood your question
<wgrant> What happens if I try to diff a UTF-16 file?
<wallyworld__> the charset parameter to addAttachment() is optional
<wgrant> Yes, but BranchMailer always uses utf-8.
<wallyworld__> the check is isinstance(payload, unicode)
<lifeless> bzr treats all files as bytestreams for diff
<poolie> mm kinda
<poolie> i'm pretty sure a utf-16 file will be reported as 'binary files differ'
<lifeless> right, but we don't do the diff as unicode
<lifeless> so diffing a slavic codepage file against the same text in utf8 will show the encoded difference, not the decoded difference.
<lifeless> Unless that has changed?
<poolie> that's correct
<poolie> it could usefully be changed, perhaps, but that's true at the moment
<lifeless> and will include slavic codepage + utf8 in the output
<lifeless> so the output of diff, today, doesn't have a 'safe correct' encoding. File paths we encoded to the supplied encoding, but thats the only bit of user data we have a clear content type for
<wgrant> That is my concern.
<wallyworld__> wgrant: so the isinstance(xxx, unicode) check prevents utf-16 strings from being touched by any utf-8 stuff
<wgrant> Perhaps, but how does it get a unicode in the first place?
<wgrant> There's no unicode-safe way to handle diffs. They have to be treated as bytestreams.
<lifeless> erm, it stops unicode strings being processed, but the intputs are not decoded, so utf16 would still be there as utf16 (if it doesn't trip the binary check poolie mentioned)
<wallyworld__> not sure. maybe thumper knows since he raised the bug
<wgrant> And yes, we're probably still going to be declaring UTF-16 as UTF-8, which would be Bad.
<lifeless> I think he was hoping it was easy; its not.
<wgrant> It's easy unless you've dealt with encodings before :)
<lifeless> personally, I'd triage it low, add a task asking for a way to get *a* defined content coding out of bzr diff, and then let the discussion happen.
<wallyworld__> wgrant: why would we be declaring utf-16 as utf-8?
<wallyworld__> actually, we are, but that's a bug i will fix
<wgrant> Fix how?
<wgrant> We need to investigate how we are getting decoded Unicode diffs in the first place.
<wgrant> Since that doesn't really make sense.
<wallyworld__> wgrant: the utf-8 encoding only occurs if the payload is a unicode instance
<wallyworld__> and since utf-16 is not a unicode instance, we are not declaring the payload to be utf-8
<wallyworld__> via the content type header
<wgrant> Yes, but there's no way that can correctly be a unicode instance.
<wgrant> So the special case will either never be used, or there is another bug creating "unicode" diffs.
<wallyworld__> i'm not sure how the diffs get generated etc. the request was "when we attach diffs to outgoing code review and branch emails can we please utf-8 encode"
<wallyworld__> so if the diffs are never unicode in the first place, that's a separate issue to solve i guess
<wgrant> I presume he wants them *declared* as UTF-8.
<wgrant> They were probably already mostly encoded in it.
<lifeless> the committer and paths will be utf8
<lifeless> whats probably happening is that we have a TEXT column note a BYTEA column.
<wgrant> Diffs are in the librarian, or not stored at all.
<wgrant> Not in the DB.
<lifeless> or we're treating the text from the librarian as utf8 and decoding it, we'll be getting punning (and sporadic failures when smeone has garbage in their tree
<wallyworld__> the last part of the request was "we have people with weird names committing or .po file stuff et al"
<wgrant> Storm will generally correctly kick you in the face if you try to do bad Unicode stuff.
<wallyworld__> so yes, it's to do with the committer etc
<wgrant> Sounds like file content: PO headers.
<wallyworld__> yes
<wallyworld__> and such stuff would be in the diffs. but if the diffs never arrive at the send mail method as unicode.....
<poolie> lifeless, on the command line you do get a defined decoding out of bzr diff
<lifeless> poolie: ah, good. I though it was still a bit hairy.
<poolie> i'm 90% sure if you have a utf-8 locale and you diff from utf8 to cpwhatever, the + lines will have questionmarks
<poolie> mm
<poolie> i have seen the bug traffic but i'm not sure what the actual motivating case is
<wgrant> poolie: Hm, that's pretty unpleasant.
<wgrant> The diff not being a diff.
<lifeless> wgrant: it has the great advantage of not turning peoples terminals into mojibake
<wallyworld__> poolie: you mean for bug 861979?
<_mup_> Bug #861979: utf-8 encode diffs attached to outgoing email <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/861979 >
<poolie> well, i think you can choose between "accurate but not in the right encoding" and "in the right encoding but not complete"
<poolie> at the api level you can
<wgrant> lifeless: That's true, but it has the disadvantage of being wrong.
<lifeless> wgrant: given there is no right answer for a unicode terminal, I find asserting its wrong to be...interesting :)
<poolie> lifeless, thanks, i have seen that but it's not very complete
<wgrant> lifeless: It is not a correct diff.
<poolie> if the actual thing is that they have validly encoded file names
<poolie> then utf-8 encoding the output diff would be a reasonable improvement
<poolie> it's a simpler case than having arbitrary binary transforms
<poolie> wgrant, the question is whether you are generating the diff to feed to patch (to apply it as a binary diff) or to look at it
<poolie> the two things need different results
<wgrant> Yes.
<wallyworld__> so, given all this, am i correct in thinking lp is not given a unicode diff to attach to its emails?
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugs: 270 - 0:[########Segmentation fault (core dumped)
<wgrant> lifeless: Shouldn't bug #869064 be High? It's not a released feature yet.
<_mup_> Bug #869064: Batched comment loading should update the page URL to include ?comments=all <story-batched-comment-loading> <Launchpad itself:In Progress by gmb> < https://launchpad.net/bugs/869064 >
<wgrant> Bug #867529, too.
<_mup_> Bug #867529: Dynamic loading comments load on top of page content <regression> <story-batched-comment-loading> <Launchpad itself:Triaged> < https://launchpad.net/bugs/867529 >
<wgrant> Or do they inherit criticality?
<poolie> wallyworld__, didn't you already land a fix for this?
<wallyworld__> poolie: it failed ec2 due to python differences on lucid vs oneiric
<wallyworld__> so it's still not landed
<wallyworld__> those issues are now fixed
<wallyworld__> but is there any point?
<wallyworld__> if the diffs passed to lp are never unicode?
<poolie> 1- you need to be more precise about 'unicode'
<poolie> 2- it depends on how it's calling bzr (external 'bzr diff', through the api, etc) and what locale it is in
<wallyworld__> poolie: unicode as in a python string u'xxxxx' ie isinstance(diff, unicode). i'm not sure what the underlying encoding details would be
<lifeless> wgrant: its behind a feature flag? I thought the flag was toggled ?
<poolie> it's called inprocess?
<wgrant> lifeless: I hope it's only released for betatesters...
<wgrant> bugs.batched_comment_loading.enabled	team:launchpad-beta-testers	1	on
<wgrant> bugs.batched_comment_loading.enabled	team:launchpad	0	on
<lifeless> fixed
<wallyworld__> poolie: 2. i'm not sure how it's being called, will have to check. it's for when lp sends out codereview and branch emails. po files have committer info in the header and we want these to come through properly for international names
<wgrant> lifeless: However, it could possibly be priority inheritance.
<lifeless> wgrant: possibly; if so gmb can and (hopefully will) say 'its critical cause I'm using it to fix bug ABCDE'
<poolie> wallyworld__, i think if you call the right api you will get unicode diff results
<poolie> if there is now way to get them file a bzr bug and we'll help
<poolie> but i would be surprised, because it works from the command line
<wallyworld__> poolie: i think the show_diff_trees api is being used
<poolie> hm apparently that returns a byte stream, probably utf8 encoded
<poolie> the way it is handled in lp seems to suggest it's expected to be a byte stream too
<wallyworld__> poolie: so the change in lp is to essentially make the Content-Type header say "text/x-diff; charset="utf-8""
<wallyworld__> i think based on what you said that will be ok?
<poolie> i think so
<poolie> does it work?
<poolie> out for a bit
<nigelb> Morning!
<spm> o/ nigelb
<nigelb> ohai spm :)
<nigelb> wallyworld__: How's your love affair with ec2 going? :P
<wallyworld__> nigelb: well, the problem is fixed. it was a python 2.6 version issue. python email encoding on lucid behaves differently to on oneiric
<nigelb> wallyworld__: Oh.
<nigelb> In a good way or bad way?
<wallyworld__> nigelb: different. here a link to the diff http://bazaar.launchpad.net/~python-dev/python/2.6/revision/42357
<wallyworld__> it now accepts unicode payloads and checks that a payload is unicode before it encodes it using the supplied charset
<nigelb> \o/
<nigelb> Nice find :)
<wallyworld__> nigelb: wgrant looked at his copy of python 2.6 on lucid to discover the difference in behaviour and provided me the diff
<nigelb> Neat
<lifeless> aieieeeeeee
<lifeless> https://bugs.launchpad.net/launchpad/+bug/869631/comments/6
<_mup_> Bug #869631: BugTask:+editstatus timeout closing bug 353126 due to bug subscription lookups <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/869631 >
<wgrant> lifeless: So the query is returning unrelated mutes?
<lifeless> yes
<lifeless> muting any bug will mute you for all notifications via duplicates
<wgrant> That's the production query that's insane?
<wgrant> Oh.
<wgrant> Isn't that intended?
<lifeless> *all*
<lifeless> mute bug 1.
<_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <GenOS:In Progress by gen-os> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:In Progress> <The Linux OS P
<lifeless> then for 12134 a dupe of 456678 you won't get notified *if you would have previously*
<wgrant> Oh!
<wgrant> Unrelated duplicates.
<wgrant> I see.
<lifeless> constraining the subquery makes it a 2ms query
<wgrant> Heh
<wgrant> What if you just s/Bug,//?
<wgrant> That should fix it too, though it'll still be a subquery.
<lifeless> 3ms
<lifeless> all in the bug
<lifeless> but - wow
<wgrant> Yes.
<lifeless> its also worrying that its doing this multiple times
<wgrant> Most of the advanced subscriptions work is like that.
<wgrant> Repetitious.
<lifeless> all dupes are handled by one of these queries, why does it need to do it more than once
<huwshimi> Have a good weekend all
* wallyworld__ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 270 - 0:[########Segmentation fault (core dumped)
<mrevell> Hey there
<lifeless> poolie: wow, silentghost, wow.
<lifeless> allenap: the reason its slow to kill the machine is that its a infinite loop
<lifeless> allenap: but failure in err() isn't the same as failure in a deferred
<bigjools> lifeless: yeah we figured that out just now in our call
<lifeless> allenap: so when it fails after an oops, thats in a deferred. It has no errback()
<lifeless> allenap: so that triggers a __del__ failure, and round it goes.
<lifeless> bigjools: cool
<bigjools> lifeless: Python dynamic typing sucks :)
<poolie> lifeless, oh in the python bug?
<lifeless> poolie: yeah
<poolie> i agree it's not the nicest part of python
<lifeless> bigjools: s/Python// and then yes, sure ;)
<poolie> also by analogy to other things, you might think that 'callable(x)' *coerces* x to be callable
<lifeless> bigjools: a friend of mine, erik de casto lopo, *loathes* dynamic typed languages.
<poolie> as for dict, int, etc
<bigjools> it also is the best thing since sliced bread
<lifeless> poolie: I believe thats one of the criticisms of callable
<bigjools> yeah, it requires a different mindset
<bigjools> was hard for me to adjust coming from C++
<lifeless> bigjools: he loathes c++ too :P [like most of us, he is opinionated ;P]
<bigjools> :)
<StevenK> Erik is a good guy
<bigjools> what does he use, $obscure_language?
<StevenK> OCaml
<poolie> yes, ocaml
<StevenK> And Haskell
<lifeless> and C :)
<lifeless> http://mstation.org/erikdecl.php
<poolie> i was writing some python just now and thinking how much better it would be in a functional language, up to the point i needed a library
<lifeless> a little dated interview, before his functional phase AFAICT, but still gets him about right ;)
<poolie> oh i thought that was meant to convey that he liked php :)
<bigjools> I remember using proof by induction with functional languages.  Head fuck.
<bigjools> php is right up there with perl
<bigjools> or is that down there... :)
<lifeless> bigjools: btw, buikld status 6 + log -> what does that mean ?
<bigjools> lifeless: need a bit more context than that!
<StevenK> Build Status 6 == BUILDING
<StevenK> But it has a log set
<StevenK> And date_finished
<lifeless> bigjools: there was a bug filed by spm yesterday
<bigjools> it's bug 717969
<_mup_> Bug #717969: storeBuildInfo is sometimes ineffective <boobytrap> <buildd-manager> <qa-untestable> <Launchpad itself:In Progress by jtv> < https://launchpad.net/bugs/717969 >
<lifeless> bigjools: cause nagios etc
<lifeless> StevenK: thanks
<bigjools> lifeless: dupe it to that one then
<bigjools> jtv is working on it
<lifeless> bigjools: I figured jtv would want to look at it, but I was also curious about what it means
<bigjools> it's a transaction issue across Deferreds
<StevenK> But how? storeBuildInfo() sets the log and so on, doesn't it?
<lifeless> bigjools: well, I think the bug is about data cleanup - or will you sort them all out at once once root cause is fixed ?
<bigjools> what's spm's bug?
<lifeless> bug 868917
<bigjools> StevenK: we found a load of places where a commit() in one callback was committing more than it should
<_mup_> Bug #868917: broken build alert <canonical-losa-lp> <ops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/868917 >
<bigjools> lifeless: I say dupe it for now unless proven otherwise, 717969 is the most likely cause.  /me dupes
<lifeless> bigjools: do we need to do the data cleanup for it ?
<bigjools> yes
<allenap> lifeless: Cheers for the explanation. I thought it was probably something like that, but it's good to know exactly what.
<lifeless> allenap: (thats a theory, but it fits all the facts I know about this)
<bigjools> lifeless: I believe the losas have a script
<cjwatson> bigjools: are you up for a chat about bug 572128 nowish?
<_mup_> Bug #572128: Ubuntu Archive translations are missing Index metadata file <regression> <Launchpad itself:Triaged by cjwatson> <debmirror (Ubuntu):Confirmed> < https://launchpad.net/bugs/572128 >
<bigjools> cjwatson: sure, good timing
<bigjools> cjwatson: want to jump on voip/skype?
<wgrant> bigjools, cjwatson: Did you get that sorted out? I guess it's pretty urgent.
<bigjools> yes
<cjwatson> bigjools: there's an XXX from cprov suggesting using a generic Release parser, and I really don't want to perpetrate even more similar horrible code - would it be OK to use debian.deb822.Release to test the contents of Release files?
<bigjools> cjwatson: hell yes
<cjwatson> then I might fix the existing code while I'm here, too
<bigjools> champion :)
<bigjools> AttributeError: type object 'ZopelessDatabaseLayer' has no attribute 'switchDbUser'
<bigjools> waaaaaa
<wgrant> bigjools: You need to use LaunchpadZopelessLayer for switchDbUser
<wgrant> bigjools: BUT
<wgrant> bigjools: As of two weeks ago that's a deprecated API.
<wgrant> Use lp.testing.dbuser instead.
<wgrant> Works in any layer.
<bigjools> I am not using it, my parent test class is
<bigjools> deep in the publisher
<wgrant> Use LaunchpadZopelessLayer, or fix your parent to use lp.testing.dbuser directly.
<wgrant> switchDbUser is now just a trivial wrapper.
<bigjools> going former for now, although startup time PAIN
<wgrant> The Zopeless layers will soon be abolished, but I need to sort out different security policies and blah.
<bigjools> why?
<wgrant> Because there's nothing special about them any more.
<wgrant> Basically, they run in PermissiveSecurityPolicy instead of LaunchpadSecurityPolicy, and they don't have WSGI interceptors installed.
<wgrant> Apart from that they are interchangeable.
<bigjools> so I can just have a DatabaseLayer?
<wgrant> You probably actually want DatabaseFunctionalLayer, but yes.
<wgrant> Watch out for the security policy difference, though.
<bigjools> I probably don't
<wgrant> You don't need the CA at all?
<bigjools> CA?
<wgrant> Component Architecture
<bigjools> yes, but I don't want the bloody librarian
<wgrant> DatabaseFunctionalLayer.
<wgrant> Unless the layer says "Launchpad" or "Librarian", there's no librarian.
<bigjools> good
<bigjools> wfm
<wgrant> DatabaseFunctionalLayer is basically the same as ZopelessDatabaseLayer, except that the order of the name is inexplicably different and the security policy is different.
<wgrant> And, until two weeks ago, stuff needed to use *Zopeless* if it wanted to change DB users.
<wgrant> But that was all BS historical crap.
<wgrant> Which I deleted.
<bigjools> yeah
<bigjools> man, storm is really frustrating sometimes
<wgrant> Oh?
<bigjools> "expected int, found Reference"
<wgrant> Ah.
<wgrant> Well, that's sort of reasonable.
<wgrant> What's unfortunate is that you can't do Reference == Reference
<bigjools> nonetheless frustrating
<wgrant> s/unfortunate/pointless/
* bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: bac | Critical bugs: 270 - 0:[########Segmentation fault (core dumped)
<jml> wgrant: fwiw, ZDL came after DFL
<jml> wgrant: so now the only difference between "zopeless" and "functional" is default security policy, right?
<danilos> jcsackett, hi, I have a feeling that bug 627631 was not really fully solved, and bugtasks for other applications have gone (like translations) in our reorg: can you confirm that suspicion I have? :)
<_mup_> Bug #627631: Queries need to be updated to use usage enums, where possible <bridging-the-gap> <lp-registry> <qa-ok> <Launchpad itself:Fix Released by jcsackett> < https://launchpad.net/bugs/627631 >
<sinzui> danilos, many were reintroduced after the branch landed.
<danilos> sinzui, fwiw, I only saw one for official_rosetta with an XXX from jcsackett pointing to the bug above (other usages of official_rosetta seem to be in tests)
<sinzui> I added two official_malone recently. I was reading the schema. I think we need to change the schema to make it obvious that official_* is not always correct
<danilos> sinzui, I think we can remove the official_rosetta with that one fix if that was the ultimate goal
<sinzui> yes I think so to
<sinzui> jcsackett, pointed out my mistake and said we might be able to fix everything on in a single branch
<cjwatson> bigjools: https://code.launchpad.net/~cjwatson/launchpad/i18n-index/+merge/78618 for your delectation and delight
<bac> sinzui: do you know how the mechanism for automatically hiding the notification box works?
<sinzui> bac: no. I do not think I have ever seen a notification hide
<sinzui> bac: do you mean the click to hide that benji had worked on?
<bac> sinzui: or maybe it hid so fast you didn't notice!
<bac> sinzui: no, it is autohiding now, at least for the bug_acknowledgement_message
<bac> sinzui: i've searched for the JS that is doing it but haven't found it yet
<sinzui> I am clueless
<bac> sinzui: as described in bug 846163
<_mup_> Bug #846163: bug filing notice is removed before the user can read it <regression> <ubuntu-qa> <Launchpad itself:In Progress by bac> < https://launchpad.net/bugs/846163 >
<bac> sinzui: thanks anyway
<bac> i thought it would be somewhere like client.js...
<benji> sinzui: unfortunately I was never able to get the click-to-hide stuff just right, so I took it out
<bac> now it is blink-to-hide
 * danilos blinks
<sinzui> benji, yes, I recall that. I assumed that bac was talking about a replacement implementation
<benji> ah
<abentley> flacoste: would it be terrible if CanonicalConfig.process_name was None when sys.argv is unavailable?
<cjwatson> bigjools: incidentally, I've run the soyuz tests now, as expected no problems there
<bigjools> cjwatson: excellent
<bigjools> I wish mine would pass
<dobey> is there any way you guys can get oops logs synced faster? like on-demand? :)
<dobey> or on-crash as it were
<deryck> abentley, heyâ¦. I have to take a long lunch today to help Wendy with something shop related.  But I'll be around longer this afternoon.
<cjwatson> bigjools: is that merge looking plausible?  sorry to rush, I just really want to get this into ec2 today if possible so that there's some chance of deploying it on Monday ...
<abentley> deryck: okay.  I'll be taking lunch later than usual, so we may still miss :-)
<deryck> ah, ok :)
<bigjools> cjwatson: yeah, nearly done reviewing it
<bigjools> cjwatson: if you want to start making changes now, you need to fix the tests to make sure all the uses of open() close the file
<bigjools> so lots of with: :)
<cjwatson> hm, right
<cjwatson> can I get away with just changing the ones I was touching anyway?  it's an endemic problem in that test file
<bigjools> yeah I see :/
<bigjools> yeah fix those
<bigjools> cjwatson: also, we generally use assertEqual(expected, observed) so that the matcher debug output is correct
<cjwatson> aha, ok, I couldn't see docs of which way round it went
<cjwatson> cool
<bigjools> pycharm tells me :)
<bigjools> cjwatson: let me know when you make the changes and I'll land it
<cjwatson> ok, working on those now
<dobey> eww
<dobey> AttributeError: 'LaunchpadTimeoutError' object has no attribute '__traceback__'
<bigjools> cjwatson: I need to dash, either get someone else to ec2 land it or leave me an email
<cjwatson> I've made the changes bigjools requested to https://code.launchpad.net/~cjwatson/launchpad/i18n-index/+merge/78618; could somebody send it to ec2 land for me?
<flacoste> abentley: no, it wouldn't be perfectly fine, it's only used to find a config file and defaults to launchpad-lazr.conf if the conf file doesn't exist
<flacoste> abentley: i'm curious though, what can make sys.argv not available?
<bac> hi flacoste, do you know of any work recently to make request.response.notifications hide themselves?  i'm trying to find the JS that does it but can't locate it.
<flacoste> bac: you mean the code that allows you to dismiss notifications?
<bac> flacoste: no, the blue box shows up but disappears after 1/2 second or so
<bac> on its own
<flacoste> bac: hmm, no, i'm not aware of anything that might have introduced this
<cjwatson> bac: can I have a quick review of https://code.launchpad.net/~cjwatson/launchpad/precise-sync-testing/+merge/78646 ?
<bac> cjwatson: sure
<bac> flacoste: ok, maybe it is an unintended consequence of something else, like maybe the privacy notifications
<bac> cjwatson: approved.  do you need me to land it for you?
<cjwatson> bac: yes please (sorry for delay, was sorting out dinner)
<bac> cjwatson: will do
<cjwatson> bac: if you could also land https://code.launchpad.net/~cjwatson/launchpad/i18n-index/+merge/78618 (already approved by Julian) that would be marvellous
<bac> cjwatson: man, now you're getting pushy!  :)
<bac> cjwatson: could you please set commit messages on both merge proposals, if not done already?
<cjwatson> already done on i18n-index, will do so on precise-sync-testing now
<cjwatson> done; don't know if the latter needs [no-qa] or similar
<cjwatson> or if that gets added by tools
<bac> cjwatson: i've sent them both off to ec2.  i forgot to mark the one [no-qa] but you can tag the bug after it gets deployed to qastaging
<bac> cjwatson: after today i'm gone for a while, so you'll have to get someone else to follow up with these branches if they don't land
<cjwatson> bac: thanks!
<bac> np
<abentley> flacoste: When run under mailman, apparently.  If anything in lp_sitecustomize.py imports canonical.config, mailman starts spewing errors.  AIUI, this is expected behaviour for embedded interpreters: http://bugs.python.org/issue839151
<flacoste> abentley: ah ok, so yes, guarding the assignment of process_name based on sys.argv is the right thing to do
<flacoste> everything should still work if self._process_name is None
<benji> I have an open question which is about changing the maintainer of a project.  Any hints as to how to do that?
<lifeless> bac: flacoste: its the ajax notification support added by wallyworld
<lifeless> bac: the first ajax request completes, has no notifications in it, and so the existing ones are nuked
<benji> I suspect I need a registry team member.
<flacoste> benji: click on the maintainer and select somebody else?
<flacoste> benji: you are one
<flacoste> by virtue of ~launchpad
<lifeless> bac: I think I suggested in the bug, to put a X button on the notifications and have the ajax embedded notifications only ever *add* to them.
<sinzui> benji, click on the edit icon next to the maintainer name
<benji> flacoste: the maintainer is a link to the user's page and there isn't an edit icon for maintainer
 * sinzui looks
<flacoste> benji: on https://launchpad.net/launchpad-project, do you see the edit icon on the maintainer?
<sinzui> benji, sorry. I was thinking team
<sinzui> benji, only a LOSA can change a project maintainer
<flacoste> or if you already the maintainer
<flacoste> which explains why i see it on launchpad-project
<bac> lifeless: do you know when that change landed?
<benji> flacoste: I do there but not on https://launchpad.net/fusionforge
<flacoste> yep, what sinzui says: ask a losa
<lifeless> bac: end of the recipes project
<benji> sinzui: awesome, thanks
<lifeless> bac: they were doing ajax build queueing or something like that and needed to show a warning / notification as a result of the call
<bac> lifeless: so that was a good while ago
<lifeless> yes; we have a deluge of bugs ;)
<bac> lifeless: i've been going through our JS code but cannot find the relevant code.
<lifeless> bac: wallyworld__ tried to fix it at the time, and I think he did, but something unfixed it shortly after and we didn't notice immediately
<lifeless> bac: huh.
<bac> there is a section in client.js that deals with notifications
<sinzui> benji, I do not know why I advised you badly the first time. I was thinking a few hours earlier that If I could temporarily change project maintainer, I could fix 777 licenses
<benji> sinzui: heh :)
<lifeless> bac: rev 12779
<bac> lifeless: thanks!
<lifeless> bac: is the most recent thing - I remember now. wallyworld__ made it so that choosing to reset the notifications is explicit (rather than them always resetting)
<bac> wow, april
<lifeless> bac: this implies that the reset that is happening is explicit (and wrong) - it may be right other times, which means we have an unreconcilable situation and will need to find some way to cut the knot
<lifeless> bac: I found it by doing bzr log and looking for 'notification' until I saw one that looked relevant ;)
<sinzui> statement_timeout I should use when suggesting queries to in production? 2 seconds
<lifeless> yes
<lifeless> 2000 ms
<sinzui> thank
<sinzui> s
<lifeless> n
<lifeless> o probs
<lifeless> I'm reminded of captain kirks speech patterns
<sinzui> :)
<timrc> sinzui, Did cody-somerville point you at our automated PPA creation code yet?
<cody-somerville> timrc, No.
<timrc> :(
<lifeless> flacoste: bug 757298 - I've just commented - it was critical by inheritance
<_mup_> Bug #757298: database bloat report ordering needs tuning <critical-analysis> <qa-ok> <Launchpad itself:Fix Released by stub> < https://launchpad.net/bugs/757298 >
<flacoste> lifeless: ack
<lifeless> flacoste: I've commented to that effect (via email so not there yet) but not changed the status
<lifeless> flacoste: I think it could go either way
<flacoste> lifeless: i'll change the status (and the category in my analysis) if I'm swayed by your argument :-)
<lifeless> flacoste: :)
<lifeless> flacoste: so the argument is that index bloat had caused timeout storms twice already by april
<lifeless> flacoste: and that the report helps us get on top of it earlier
<flacoste> that's fair
<sinzui> lifeless, I wanted to run this in production, but it timesout under cold cache: http://pastebin.ubuntu.com/704176/ Do you have thoughts about making this faster? Should I take the extra pain of writing a garbo job, land, released, revert?
<lifeless> garbo please
<sinzui> okay
<lifeless> or
<lifeless> and this is a little radical
<lifeless> could an API script do it ?
<lifeless> sinzui: the second half - the 77 rows - that should be totally fine on prod
<sinzui> well interesting proposal. I the the first part that is fine under cold cache is doable over api
<sinzui> I do not think the second part that is slow is doable. But I have yet to think about all the cases that those 77 packages are missing
<lifeless> sinzui: ah I see your times in -ops
<lifeless> sinzui: your methodology has a hole in it :)
<sinzui> yep
<lifeless> sinzui: try this: set the statement timeout to 15 seconds; run the *select clauses only* for both queries outside a transaction; set the statement timeout to 2000; BEGIN; do the INSERT queries; ROLLBACK;
<sinzui> ah, build a hot cache
<lifeless> we care about the duration of locks, not the overall load [well we care about both, but you see what I mean]
<sinzui> I do
<lifeless> there is some overhead you can't pre-load when inserting into the table, because indices have to be updated (which means reading them in)
<lifeless> so to test this a second time, use a different slave
<lifeless> as long as we rollback, testing something like this on a slave is fine
<sinzui> okay thank is very helpful to know
<lifeless> sinzui: another strategy is to run the exact query in a loop with rollback and low timeout until it works
<lifeless> we'll release any locks every (say) 1 second [the timeout] and thus won't cause timeouts
<flacoste> lifeless: should yuo be assigned to bug 788518?
<_mup_> Bug #788518: staging services share OOPS prefix <Launchpad itself:Triaged> < https://launchpad.net/bugs/788518 >
<flacoste> lifeless: as your current work will fix this
<lifeless> flacoste: it won't really; it will just mean we get the reports.
<lifeless> flacoste: we will still want to identify each script properly, which means a different set of work to auto-assign reports for scripts by the script name
<lifeless> flacoste: which my work will enable but not itself do.
<flacoste> ok
<lifeless> flacoste: I may, later, do that
<flacoste> but the fact that we get the report means that the priorit drop at least
<lifeless> yes, I agree - once my work lands :)
<flacoste> lifeless: well, now it's on your radar ;-)
<cjwatson> hmm.  lp:~cjwatson/launchpad/i18n-index failed EC2.  The end of the log goes:
<cjwatson> successful: lp.archivepublisher.tests.test_publisher.TestPublisher.testReleaseFile
<cjwatson> WARNING: A test appears to be hung. There has been no output for 600 seconds.
<cjwatson> but I've run the next test along locally (testReleaseFileForPPA) and it succeeds - can anyone suggest a possible gotcha I might be running into?
<lifeless> might be a later transition, or a layer test teardown - they run after tests complete
<lifeless> (also layers are evil, this is one of the data points for why)
<wgrant> sinzui: I think we are nearly getting somewhere!
<wgrant> cjwatson: Could you forward me the output?
<wgrant> cjwatson: Is it possibly the apt-ftparchive hang in mvo1? I don't think the ec2 images have mvo2.
<wgrant> But I'm not sure if you've changed the test to actually exhibit the hang...
<cjwatson> I think the test is testReleaseFileForPPA, which would not be a-f
<wgrant> testReleaseFileForI18n will probably hang.
<wgrant> It sets include_long_descriptions = False, and then calls a-f.
<cjwatson> forwarded to you
<wgrant> Thanks.
<cjwatson> oh
<cjwatson> that would indeed be next alphabetically
<cjwatson> yep, in a local run it's
<cjwatson> successful: lp.archivepublisher.tests.test_publisher.TestPublisher.testReleaseFile
<cjwatson> test: lp.archivepublisher.tests.test_publisher.TestPublisher.testReleaseFileForI18n
<wgrant> Great.
<cjwatson> so this is artificial and we need to get ec2 images upgraded
<wgrant> I'll get the fixed version onto buildbot while we have a LOSA, and sort of buildbot images today.
<cjwatson> thanks; does that indicate bypassing ec2 then, or do you want to upgrade the ec2 images too?
<wgrant> Er.
<wgrant> s/sort of buildbot/sort out ec2/
<cjwatson> aha
<wgrant> We're in a fairly good position QA-wise, so as long as I can get buildbot sorted out before thedac disappears, we should be able to deploy this at 0830 on Monday :)
<cjwatson> Awesome
<cjwatson> That will make my next week suck a bit less
<wgrant> That's the hope.
<wgrant> KeyboardInterrupt lp.archivepublisher.tests.test_publisher.TestPublisher.testReleaseFileForI18n
 * wgrant upgrades apt-utils to lucid-proposed
<wgrant>   Ran 30 tests with 0 failures and 0 errors in 27.679 seconds.
<wgrant> Success.
* wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 269 - 0:[########Segmentation fault (core dumped)
#launchpad-dev 2011-10-08
<cjwatson> wgrant: I admit to being a little surprised that we apparently didn't have any tests that called a-f with include_long_descriptions=False before.
<cjwatson> Probably my mistake - I think we just checked that apt.conf looked right.
<wgrant> cjwatson: Indeed.
<wgrant> cjwatson: Anyway, buildbot is fixed, and ec2 is updating.
<wgrant> Will re-ec2land it once it's done.
<cjwatson> yay, ta
<nigelb> Morning *yawn*
<wgrant> cjwatson: Still around?
<wgrant> If so, could you apply https://pastebin.canonical.com/54077/ to your branch?
<cjwatson> looking
<cjwatson> why does changing that in my branch do anything? :)
<wgrant> That list specifies valid AMI owners.
<wgrant> I can change it locally, but once this lands everyone needs to be able to see the new AMI.
<wgrant> So ideally they land together.
<wgrant> I could just land it separately, I suppose.
<cjwatson> I can do it
<cjwatson> Given you're landing it you get to say whether it's policy-OK :)
<wgrant> Yep.
<cjwatson> done
<wgrant> Thanks, it's back in ec2, with the new image.
<wgrant> If it fails again, I'll try to sort it out.
 * cjwatson wonders why http://lpqateam.canonical.com/ was last generated over a day ago
<cjwatson> (though nice website)
<nigelb> wow, I didn't think I could see that.
<nigelb> That is neat.
<cjwatson> It used to be in a private location.  I think it was made public quite recently.
<cjwatson> Or at least the deployment report did.
<nigelb> Yeah, I remember jumping with joy for that one :)
<cjwatson> http://blog.launchpad.net/general/deployment-reports-are-now-public
<nigelb> heh
<nigelb> <3 the incident report section
<wgrant> cjwatson: Good question. Possibly related to staging being a bit broken lately.
 * wgrant checks.
<wgrant> The incident report finder is broken.
<wgrant> Ah, wiki.c.c seems to be kicking people out of teams quickly, or something.
<wgrant> Ah, yes, mentioned in #is. The ACL keeps getting erased.
<nigelb> hah
<wgrant> Anyway, dashboard robustified and working again.
<cjwatson> hm, https://lpbuildbot.canonical.com/changes/3807 probably should've been credited to me, not wgrant
<cjwatson> (the last commit was author=wgrant committer=me)
<cjwatson> (but all the rest were mine)
<cjwatson> I wonder what the deployment report will make of that
<cjwatson> and Contributions for that matter
<jelmer> cjwatson: I think this is just the person that landed it
<jelmer> since most contributors can't land stuff, I suspect the Contributions page uses something different
<cjwatson> jelmer: No, I'm always in this boat and my other landings are generally credited to me.
<cjwatson> It's definitely because the last commit was requested by wgrant so I --author-ed it
<cjwatson> I'll remember not to do that in future, I guess :-)
<jelmer> heh
 * jelmer doesn't get credits on the Contributions page because he was once in the Launchpad team
<wgrant> cjwatson: Contributions looks through the entire history.
<wgrant> cjwatson: So it will work fine.
<wgrant> Deployment report I'm not quite sure about.
<wgrant> jelmer: Ah, it works for people like mwhudson, because they now use linaro addresses.
<wgrant> You need a date threshold, I guess.
<jelmer> I might contribute such a change to the contributors script at some point.. :)
<wgrant> cjwatson: Both of your landings from today/yesterday are one https://dev.launchpad.net/Contributions#colin_watson
<wgrant> s/one/on/
<cjwatson> wgrant: cool, thanks
<cjwatson> not that it's more than vanity :)
<wgrant> Heh
<wgrant> sigh, buildbot has gotten slow again.
<lifeless> oh, so its working?
 * jelmer waves to lifeless 
<wgrant> lifeless: What's working?
<lifeless> bb
<wgrant> Yes.
<wgrant> It even passed once today.
<wgrant> After failing because of nodowntime.
<lifeless> how did ndt fail it ?
<wgrant> Ah, you must have been away the day I realised this.
<wgrant> Graceful appserver restarts only watch the main haproxy page.
<wgrant> Not xmlrpc-private.
<wgrant> So xmlrpc requests can be killed.
<wgrant> And result in a 502.
<lifeless> I don't recall a bug on this
<lifeless> or an RT
<wgrant> I meant to file an RT.
<wgrant> Only discovered it last week.
<lifeless> cool
<lifeless> nn
<wgrant> Night.
 * nigelb waves
<wgrant> Evening nigelb.
<nigelb> wgrant: ohai
<nigelb> Oooh. Need 3 more landings to be third :)
 * wgrant WHUIs the vhost config system.
<elmo> WHUI?
<wgrant> elmo: Past form of YAGNI.
<elmo> ah
#launchpad-dev 2011-10-09
<lifeless> elmo: 'we haven't used it'
<wgrant> Er, yes, forgot to actually expand it, sorry.
<wgrant> errrm
<wgrant> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2107AP52 is a worry
<wgrant> Erk erk erk.
<wgrant> BugTaskStatus is a lie.
<wgrant> 13/14 are INCOMPLET_WITH(OUT)_RESPONSE
<wgrant> But they're not in BugTaskStatus
<wgrant> Only in BugTaskStatusSearch
<wgrant> Fortunately the migrator is buggy and crashing, so not too many bugs are broken.
<lifeless> meep
<lifeless> (erm, what do you mean)
<wgrant> lifeless: It looks like only Product:+series is actually crashing due to this enum strangeness, but it's still pretty horrible.
<wgrant> Bug #871076
<lifeless> thanks, have commented
<wgrant> Have you?
<lifeless> email
<wgrant> Ah
<lifeless> so the incomplete patch teachs enumvariable about multiple schemas
<lifeless> so we can do evolutions like this
<lifeless> on load things not found in the first schema are looked up in the second etc
<wgrant> Ah
<lifeless> I haven't checked the view in question specifically but it sounds like its grouping by and then manually making a dict of the types expected
<wgrant> Right, it manually does BugTaskStatus.items[somevaluefromthedb]
<lifeless> one way would be to change that to use search
<lifeless> another would be to generate a mapping dict manually (or better yet teach the column now to do it)
<lifeless> s/now/how/
<lifeless> BugTask._status.load(somevaluefromthedb, from_db=True) or some such, -might- do it already.
<lifeless> but I suspect the code is currently too coupled to being a descriptor rather than mapping + description
<lifeless> .
<wgrant> Morning.
<jelmer> g'morning wgrant
<mwhudson> good morning
<wgrant> Morning jelmer, mwhudson.
<lifeless> morning early-grant
<wgrant> Hi lifeless.
 * mwhudson forgets things
<mwhudson> can i land changes to the config branch with pqm or does a losa have to do that?
<lifeless> pqm
<mwhudson> cool
<wgrant> https://dev.launchpad.net/WorkingWithProductionConfigs
<lifeless> much of that should probably be in-tree in a README
<lifeless> (or at least, a link to that wiki page)
<wgrant> mwhudson: It's been deployed *everywhere*?
<mwhudson> wgrant: https://devpad.canonical.com/~wgrant/production-revnos says so
<wgrant> Great.
<wgrant> Just wanted to be sure, as this week is a bit special.
<mwhudson> ah, release week & all that?
<wgrant> Yes.
<wgrant> and we need to deploy a fix to cocoplum before oneiric freezes for good.
<wgrant> And hope that nothing else is broken.
<mwhudson> it only took 12 days from (re-)landing to deployment
<mwhudson> felt longer
<mwhudson> how did we cope in the old deploy once a month world?
<wgrant> Yeah.
<wgrant> The librarian took a little while to get HA.
<wgrant> But it's pretty much done now.
<wgrant> Finally.
<lifeless> \o/
<mwhudson> cool
<wgrant> Just need to add graceful shutdown.
<wgrant> But everything is through haproxy, which is the hard bit.
<wgrant> Which leaves poppy :(
<wgrant> Hmm, I wonder what happens if we just push the FTP control connection through haproxy, but leave the data connection to go directly.
<wgrant> I suppose that would work.
<wgrant> And be trivial.
<lifeless> wgrant: thats what I've been telling you :)
<lifeless> wgrant: as a first iteration
<lifeless> second iteration will need more thought
<wgrant> Well, I'd been missing the fact that we don't care about the data connection.
<wgrant> Because normally I'm doing this sort of thing to firewall off or NAT the real host :)
<lifeless> for that you need frox
<lifeless> and we may need it for the second iteration
<lifeless> I haven't put a plan together for the details there yet, because theres no point planning when we're as far out as we are from execution
<mwhudson> jelmer: do you have a plan for converting existing mirror branches to bzr-code-import branches? (just curious)
<jelmer> mwhudson: I was actually just looking at the numbers
<mwhudson> jelmer: i have this old memory that says "roughly 3k mirrored branches"
<lifeless> that was gnome wasn't it ?
<mwhudson> possibly
<jelmer> mwhudson: it's more like 2k
<lifeless> we had a separate machine doing imports of gnome, and jelmer registered all of it one day
<mwhudson> ah no, that was a lot more than 3k
<jelmer> and there's about 300 remote branches
<jelmer> lifeless: IIRC that was all code imports though, not mirrors
<mwhudson> jelmer: i did some log grepping once and worked out that about 17 out of however many mirror puller runs actually imported anything in a week
<cjwatson> wgrant: so I have a short while now if you want me to QA r14123; although I'll need help with dogfood setup
<wgrant> cjwatson: I just updated it.
<wgrant> Takes ~forever.
<cjwatson> Will take or has taken?
<wgrant> Has taken.
 * wgrant dirties and publishes a PPA to start.
<jelmer> mwhudson, lifeless: I don't have plans to work on converting all remaining mirror/remote branches to import branches, and am mainly focussing on bfbia at the moment. Do you think we should worry about converting the existing mirror/remote branches ?
<mwhudson> jelmer: nah
<mwhudson> bfbia seems more important
<wgrant> cjwatson: Do we perhaps want to rsync the real primary archive's dists across?
<lifeless> jelmer: mwhudson: what would conversion entail ?
<mwhudson> i guess a bunch of fiddling -- db work to create a code import for each branch and change the type, an rsync script to prepare the staging area on escudero
<lifeless> jelmer: mwhudson: I'm a -huge- fan of finishing things off, so unless its particularly hard, yes, I think we should finish the work.
<wgrant> mwhudson: We don't actually need to populate the staging area, do we?
<lifeless> mwhudson: won't the staging area auto-populate, same as a new import ?
<wgrant> All we lose is a bit of pear time by not doing it.
<mwhudson> wgrant: well no, but that would imply a reimport of everything
<lifeless> mwhudson: known as a fetch :P
<mwhudson> i guess it depends if any of the branches being mirrored are large
<wgrant> Not cheap, but meh.
<cjwatson> wgrant: I thought there was a plausible distroseries somewhere already with a forked hello package in it
<mwhudson> mysql were using mirrored branches at one point... are they still?
<wgrant> cjwatson: DF was erased a couple of weeks back.
<mwhudson> (should assess disk space impact on escudero too i guess)
<cjwatson> wgrant: what would rsyncing the real dists across achieve?  if we actually tried to modify any of those distroseries, surely LP would clobber them anyway
<cjwatson> wgrant: oh, Translation-* custom uploads I suppose
<wgrant> cjwatson: Well, we don't have ddtp files.
<wgrant> Right.
<cjwatson> yeah, that might be an OK plan
<wgrant> But we can easily forge those, I guess.
<cjwatson> real data wouldn't hurt
<cjwatson> although I think the DS with include_long_descriptions=False was querulous or something?
<jelmer> wgrant, mwhudson: some of the existing mirror branches have data but the location they mirror no longer exists. I think we should make sure that data is kept around.
<lifeless> the risk of not completing the transition is that we have (yet another) ambiguous and stubby code area
<wgrant> cjwatson: Right, but that's gone now.
<wgrant> I'll have to upload something new.
<lifeless> mwhudson: jelmer: can the imports *read* from b.l.n ?
<wgrant> Or, well, copy it across.
<wgrant> That would work too.
<wgrant> But first I need to work out why the PPA publisher is trying to publish every PPA...
<mwhudson> lifeless: probably
<mwhudson> lifeless: not sure though, they don't currently
<lifeless> so one way you could solve this is to clone from b.l.n to escudero if the branch is populated and no staging area exists
<jelmer> lifeless: at the moment they only read from the data dir on escudero AFAIK
<lifeless> given all imports are public
<jelmer> are imports necessarily public?
<lifeless> today, yes.
<lifeless> (mwhudson will shout out if I'm wrong :P)
<jelmer> hmmok, I've been working on the assumption that they aren't necessarily public (which should only be a good thing for the future)
<mwhudson> i'm not sure imports are necessarily public
<mwhudson> i think they probably *are* though
<mwhudson> there was a bug about them at some point
<wgrant> 2011-10-09 21:16:18 DEBUG   Writing Index file for maverick/main/i18n
<wgrant> Let's see if it actually did anything...
<mwhudson> yay bug search
<wgrant> ppa:launchpad/ppa, this is.
<wgrant> No index there, which sounds right.
<cjwatson> That's a buglet - it shouldn't log that until it's sure it's actually going to do so.
<cjwatson> Not fatal though
<lifeless>  private | count
<lifeless> ---------+-------
<lifeless>  f       |  5593
<lifeless>  t       |     5
<lifeless> there are 5 private imports.
<lifeless> mwhudson: jelmer: however, as the import *details* are public (IIRC) this is a bit strange.
<wgrant> cjwatson: Bah, mawson can't connect to rsync on (rsync.)archive.ubuntu.com, and cocoplum's rsyncd forbids it.
 * wgrant creates fake files.
<mwhudson> lifeless: how so?  if the branch is private, how are you going to see the details?
<lifeless> mwhudson: only if the codeimport, which is directly traversable inherits the privacy rule
<mwhudson> lifeless: are you talking about api access?
<lifeless> mwhudson: well, maybe its changed, but imports used to have their own url
<lifeless> anyhow
<lifeless> all 5 have public source urls
<lifeless> I believe the only reason they are private is they were created on private-by-default projects
<mwhudson> lifeless: that changed a _long_ time ago
<mwhudson> yeah, i agree
<jelmer> lifeless: how many private mirror branches?
<lifeless> jelmer: give me the query to id mirror branches and I'll tell you
<mwhudson> lifeless: branch_type = ... 2?
<mwhudson> yes, 2
<lifeless>  private | count
<lifeless> ---------+-------
<lifeless>  f       |  2112
<lifeless> (1 row)
<lifeless> (0)
<mwhudson> \o/
<lifeless> need to make sure we don't permit by-id direct-apache access to the importds (would permit public imports of random private branches), but other than that, letting importds read from b.l.n makes a lot of sense to me anyway
<lifeless> e.g. to remove the staging area for imports that don't need it
<mwhudson> unless things have changed a bunch, we don't provide by id access to the importds
<lifeless> right, and shouldn't
<mwhudson> it's locked down to crowberry itself & loganberry by ip address
<wgrant> It's forbidden in Apache, and the opener should forbid it too.
<lifeless> I was just saying we need to keep it closed :)
<lifeless> I'd like to remove the importd non-xmlrpc-api code from the codebase
<wgrant> We can finally revoke DB access from the importds once we have a rabbit OOPS transport :)
<mwhudson> yes, that was a disappointing discovery
<lifeless> mwhudson: what that ?
<mwhudson> i did a bunch of work so we could revoke db access from the importds
<mwhudson> and then oops-prune fell over :-)
<cjwatson> wgrant: well, let me know if there's something I can actually do to help, or whether I'll just blunder around and get in the way :-)
<wgrant> cjwatson: Well, I know mostly what I'm doing here, so QA on dogfood mostly involves sitting around and watching bzr st bring it into 50% iowait.
<cjwatson> OK
 * wgrant is beginning to think that there may actually be something wrong with mawson.
<wgrant> 2011-10-09 21:37:40 DEBUG   Publishing custom translations_multiverse_20110922.tar.gz to ubuntu/oneiric
<wgrant> We should now have some files.
<cjwatson> wgrant: OK, well, I'm off to bed; I see publish-distro is running, so if it looks like it's all gone wrong in the next couple of hours, feel free to SMS me (number's in the directory)
<wgrant> cjwatson: I'll hopefully sort it out if stuff does go wrong :)
<wgrant> Night, and thanks for fixing this quickly!
<cjwatson> The knowledge that it will be worse not to fix it quickly is a wonderful incentive.
<wgrant> That's true.
<lifeless> wgrant: ians work may be a candidate
<lifeless> wgrant: the queries are consulting both private and transitively_private
<wgrant> lifeless: Yes, I noticed that.
<lifeless> not to mention that its buggy
<wgrant> All attempts to implement sensible privacy on top of non-sensible privacy have so far been buggy.
<wgrant> So of course it is buggy.
<lifeless> branches that are only transitively private and owned by the viewer won't be shown
<lifeless> well, specifically buggy
<mwhudson> yeah, i was wondering about that, surely all checks should be for transitively_private ?
<lifeless> yes
<wgrant> cjwatson: Get Translation files ...
<wgrant> [  0%] Getting: dists/oneiric/main/i18n/Translation-en.bz2ok
<wgrant> [  0%] Getting: dists/oneiric/multiverse/i18n/Translation-bg.bz2ok
<wgrant> cjwatson: debmirror is even happy with the output.
<cjwatson> awesome!
<cjwatson> looks fine visually too.
<wgrant> Yep.
<wgrant> Hm. LP derived distros are going to be published at derived.archive.ubuntu.com? :/
<wgrant> That seems a bit odd.
 * cjwatson is uncomfortable with that.
<wgrant> Yes.
<wgrant> Just a bit.
 * wgrant will convince Julian that it's a terrible idea tonight.
<elmo> wgrant: reference?
<elmo> nm, got it
<wgrant> In the DD RT.
 * wgrant just closed it.
<wgrant> eg. https://rt.admin.canonical.com/Ticket/Display.html?id=48314
 * elmo reopens
<lifeless> wgrant: you closed the url, or the ticket ?
<elmo> well, not that
<wgrant> Er.
<wgrant> I mean I closed the tab.
<wgrant> So didn't have the URL handy.
<lifeless> ma
<lifeless> man
<lifeless> i get disproportionately annoyed by 'N time has passed, whats up' in bugs.
<wgrant> Well, in their defense the bzr upgrade is taking its time :)
<wallyworld_> lifeless: just saw your comments about transitively_private. all queries are supposed to look only at transitively_private. if any still look at private, that's a bug and needs to be fixed. do you have an example?
<elmo> lifeless: paste large chunks of some random dude's biography as a reply
<lifeless> wallyworld_: the OOPS referenced in #launchpad
<wgrant> elmo: Thanks.
<elmo> "what's up?  well, let me tell you..."
<wgrant> elmo: I didn't see this name until today either.
<lifeless> wgrant: I meant in bug 294159
<wgrant> I wonder where it was discussed.
<lifeless> (naming - me neither)
<wgrant> "not", I guess.
<wallyworld_> thanks. i'll look. and fix it. it was meant to be s/private/transitively_private
<wgrant> Let's not make the PPA mistake again, either.
<wgrant> lifeless: Oh, a different one.
<wgrant> lifeless: I see.
<elmo> upload?  yeah
<wgrant> elmo: I mean having stuff under launchpad.net.
<wgrant> And that too.
<wallyworld_> lifeless: bollocks. can you paste to oops num. i had been disconnected from #launchpad for some reason
<lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=2108CP126
<wgrant> wallyworld_: https://pastebin.canonical.com/54081 is a query
<wallyworld_> thanks
<wallyworld_> lifeless: why is the oops shown as being linked to an old bug 638924?
<lifeless> because that correlation generates many false positives
<wgrant> StevenK: Around yet?
<wallyworld_> ok. i'll raise a bug to fix the query column. the timeout is a separate issue which won't be affected by using the correct column
<wgrant> wallyworld_: Why not?
<wgrant> wallyworld_: It'll have to query two indices, separately.
<wgrant> It may very well negatively influence the plan.
<wallyworld_> perhaps. but sad if that's the case. poor reflection on postgres
<wgrant> Only slightly.
<wallyworld_> a decent db should be able to efficiently query on two indexed boolean columns
<wgrant> On large, skewed tables, where the query was optimised for a single one?
<wallyworld_> i'm not familar with postgres's query optimisation so am not sure
<wgrant> Let's delete Launchpad and start again.
<wgrant> Changes that will be made are A) destroying bug heat, B) making bug subscription queries not terrible.
<wgrant> lifeless: can you explanalyse 'SELECT Bug.heat FROM Bug, Bugtask, Product WHERE Bugtask.bug = Bug.id AND Bugtask.product = Product.id AND Product.project IS NOT NULL AND Product.project = 82 ORDER BY Bug.heat DESC LIMIT 1' on qastaging?
<lifeless> wallyworld_: so, querying on two unrelated indices requires either A) a hash join on the indices or B) a bitmap join on the indices
<lifeless> wallyworld_: A) is done by generating the hash during processing, and B) requires reading every row in the index
<lifeless> wallyworld_: the only other form of join I'm aware of that could benefit is nested loops, and I've never seen postgresql do that within one table
<lifeless> wallyworld_: lastly, for any DB, using more indices (usually) implies more potential disk IO (and usually random at that), which the planner will avoid as its costly
<lifeless> wallyworld_: so yes, using two separate fields that are index separately is both more costly (in theory) and a candidate for causing performance issues
<wallyworld_> sure, but surely it could narrow the results by using each index sequentially. there are lots of queries we do which use more than one indexed column
<lifeless> yes, but check the plans - pg picks one index and runs with it
<lifeless> wallyworld_: some reasons for this are that indices can't tell you liveness of rows - you have to consult the table itself, and that cross-index statistics are (AFAIK) not well defined
<wallyworld_> ok
<lifeless> wallyworld_: mostly though, its the liveness I suspect: finding the candidate rows from the most selective index means you'll be reading the actual rows to do that; and checking a field in those rows is usually about as cheap or much cheaper than paging in index pages from a separate index
<lifeless> now, the math says that in very large tables with lopsided data (which we have) that the CPU time will become more expensive than the time to grab a second index and refine, but I suspect its so marginal that the planner code doesn't even permit it
<lifeless> if someone were to do a N-index without-consulting-the-rows filter, that would likely be more useful
<wallyworld_> so you seem to be implying we could run into issues for *any* query which filters on more than one indexed column
<lifeless> wallyworld_: we do
<wallyworld_> which seems absurd!
<lifeless> wallyworld_: its particularly bad on very big tables (like BPPH and SPPH)
<wallyworld_> that a db can't handle that scenario
<lifeless> wallyworld_: for precision: queries which filter on more than one column, where the most selective index is not very selective, in big tables, will have issues
<lifeless> wallyworld_: the db provides tools like N-column indices to handle such scenarioes
<wallyworld_> sure
<lifeless> wallyworld_: bitmap filters across multiple indices are great for hot-indices in moderate size tables
<wallyworld_> in this case, private|transitively_private = true is very selective
<wgrant> But that's two indices.
<wallyworld_> yes, and to me the db should be good at that, but it seems not so
<lifeless> wallyworld_: profiling in db's is the same as in regular code: unless you measure, you can't predict reliably.
<lifeless> wallyworld_: saying 'should' here is something I guess I object to: if someone writes the code to model the costs reliably, from table stats, then they can write the executor, and away we go.
<lifeless> wallyworld_: but in the absence of a model, we can't even say *hypothetically* that it should be good at it.
<wallyworld_> sure. i have no knowledge of how postgres' stats are calculated or how we have parameterised the analysis engine
<wallyworld_> past experience with oracle != postgres clearly
<lifeless> wallyworld_: we have 450K branches in the system; the indices are 10MB each, we have 22K private branches - hugely lopsided - so in principle the performance question is, is 'how expensive is consulting 22K rows of a cold index'
<lifeless> wallyworld_: oracle itself is also sensitive and needs continual review and tuning, different precise rules but similar issues, from my experience
<lifeless> bottom line is, you can't assume anything is fast until its been tested - both for python and for postgresql; things will always surprise us
<wallyworld_> yes
<wallyworld_> i know how lopsided the private branches numbers are. you wouldn't expect looking at "only" 22k rows from a cold index would be too bad, but you never can tell
<lifeless> fwiw - doing the query wgrant asked about above on bugs
<lifeless> which chose a nested loop plan, examining 27K bugs
<lifeless> took 25 seconds
<lifeless> you're looking at 1-2ms per row when IO is involved
<lifeless> so I expect that looking at 22k rows in a cold index to be moderately painful at best.
<wgrant> lifeless: Why do we have an haproxy status page on each service, rather than just closing the listener?
<wgrant> Seems like it would be much easier if processes just closed their listener and died when there were no connections left.
<wgrant> Rather than the haproxy dance in appserver initscripts, and adding an HTTP listener to every service.
#launchpad-dev 2012-10-01
<lifeless> StevenK: auditor has no buildout now ?
<nigelb> I just discovered an old branch I was working on , on this laptop.
<nigelb> I wonder if I can finish it. Hrm.
<StevenK> lifeless: auditor itself is not buildout enabled, no.
<StevenK> lifeless: Actually, what do you mean 'now'? auditor never had buildout.
<LPCIBot> Project devel build #1099: STILL FAILING in 1 min 23 sec: https://lpci.wedontsleep.org/job/devel/1099/
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #1100: FIXED in 3 min 26 sec: https://lpci.wedontsleep.org/job/devel/1100/
<StevenK> Jenkins, you lying scoundrel
<rick_h_> never trust it :P
<StevenK> rick_h_: Hmph.
<StevenK> Right, build 1101 is off and looking good, so time for some DEHR
<lifeless> StevenK: I assumed it would
<lifeless> StevenK: as its what we use for deploys
<StevenK> lifeless: No, that's production-auditor
<lifeless> wgrant: you were doing something about       263 /    0  POFile:+translate
<lifeless>  were you not ?
<LPCIBot> Project devel build #1101: FAILURE in 1 hr 29 min: https://lpci.wedontsleep.org/job/devel/1101/
<wgrant> lifeless: It's a terrible page in general, it's not just the query that I was looking at. But I eventually established that we probably can't optimise that query significantly without changing to something sensible like FTI, rather than substring matching.
<wgrant> (but that has obvious problems, given all the languages)
<lifeless> bllllar, pypi login is very odd
<lifeless> it had basic creds
<lifeless> so knew who I was
<lifeless> but wasn't showing my logged in
<lifeless> s/my/me/
<lifeless> click on 'login' and it changed to logged in.
<lifeless> ahhaha
<lifeless> http://pypi.python.org/pypi/Ojota/0.4.2
<lifeless> ' License: LICENSE.txt '
<lifeless> I am filled with confidence
<lifeless> linear operation dispatch
<lifeless> sigh, they need mentoring.
<lifeless> bah, codus interruptus. I will try to get back to this later.
<StevenK> Neat, concurrency 4 is 30 minutes quicker.
<StevenK> lifeless: https://lpci.wedontsleep.org/job/devel/lastBuild/ has your build artifacts
<tumbleweed> wgrant: https://code.launchpad.net/~stefanor/launchpad/packageset-destructor/+merge/127167 dropped the export_operation_as. Care to land it?
<wgrant> tumbleweed: Sure
<wgrant> Thanks
<tumbleweed> btw, that export_operation_as was there because I saw it in some other places. Should they be cleaned up?
<wgrant> Hm, which other places? export_destructor_operation has only been around for a couple of years, so some may have been incorrectly ported.
<tumbleweed> ack-grep says lib/lp/blueprints/interfaces/specificationbranch.py lib/lp/registry/interfaces/milestone.py
<tumbleweed> oh, and lib/lp/registry/interfaces/productrelease.py
<wgrant> Hmm
<wgrant> Those are possibly relevant to old versions of the API
<wgrant> tumbleweed: Can you set a commit message?
<tumbleweed> wgrant: done
<wgrant> tumbleweed: Thanks. That's landed.
<tumbleweed> thank you :)
<adeuring> good mmorning
<lifeless> ok
<lifeless> so why does django test discovery not look in submodules
<czajkowski> adeuring: ello ello
<StevenK> lifeless: Can haz another peek at jenkins?
<wgrant> tumbleweed: The destructor change is through buildbot, and should be in the next production deployment.
<tumbleweed> yup, got an e-mail, thanks
<lifeless> StevenK: remind me tomorrow, its late and i've been kicking django basics around
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji  | Firefighting: - | Critical bugs: ~300
<deryck> adeuring, abentley -- hey, having trouble connecting to the hangout.  still trying to get in...
<abentley> deryck: ack.
<sinzui> My house is full of plague. I an not getting our of jim-jams today
 * czajkowski passes a hot whiskey to sinzui 
<sinzui> I please no
<sinzui> I don't think it will stay down
<sinzui> I am trying not to ingest
<czajkowski> sinzui: any reason why a person cant sign the CoC with a subkey not their primary key ?
<sinzui> czajkowski, very good reasons
<czajkowski> mind telling them to the chat in -lp please :)
<czajkowski> *chap
<sinzui> czajkowski, Last week we downgraded gpg on several of our servers because an unneeded security fix for dpkg was applied.
<czajkowski> sinzui: https://bugs.launchpad.net/launchpad/+bug/1053568/comments/4  <-- that
<_mup_> Bug #1053568: PPA uploads signed with subkeys silently fail <gpg> <soyuz-upload> <Launchpad itself:Fix Released> < https://launchpad.net/bugs/1053568 >
<sinzui> czajkowski, We downgraded the machines we knew of.
<sinzui> czajkowski, right
<czajkowski> ah so I was right
<sinzui> if the problem returned, we need to ask if webops applied the security fix again...out machines were not affected by the issue
<cjwatson> security fix to gnupg, wasn't it, not dpkg?
<cjwatson> https://launchpad.net/ubuntu/+source/gnupg/1.4.10-2ubuntu1.1
<sinzui> cjwatson, yes
<deryck> abentley, adeuring -- do you guys expect any db patches this week, beyond my patch for Product.information_type?
<adeuring> deryck: can't think of any right now
<deryck> adeuring, ack, thanks
<abentley> deryck, adeuring: Are we going to need to extend AccessArtifact to support Products?
<adeuring> abentley, deryck: can't we use simply policy grants?
<deryck> yeah, I was under the impression from adeuring that this was a little simpler for products and didn't require AccessArtifact.
<abentley> adeuring: Yes, I think that makes sense.
<adeuring> after all, policy grangts already give access to branches, bugs, specs, so its "natural" to extend this to give access to the pillar/product itself
<deryck> adeuring, but no new db patches needed, though, right?
<adeuring> deryck: right, i think we don't need any.
<deryck> awesome.
<deryck> trying to coordinate since webops are sprinting.
<cjohnston> deryck[lunch]: ping when you get back
<deryck> hi cjohnston. have a call now, but I can ping after that.
<cjohnston> ok deryck
<deryck> cjohnston, hey. Free now.  What's up?
<cjohnston> deryck: I want to make sure I understand.. So as Summit is not authenticated, it will not see any private BPs that are proposed to the UDS sprint?
<deryck> cjohnston, correct.
<cjohnston> Ok.. so your comments confuse me a little.. If Summit doesn't have the ability to see it, is there a reason for the ability for a BP to be proposed to a sprint to exist?
<cjohnston> otherwise it still seems like a confusing UI as I propose my blueprint to a UDS sprint, I assume that more than likely the people who approve blueprints for the sprint can't see the blueprint to approve it for the sprint, even if they can, Summit can't see the blueprint to create the (private) meeting
<cjohnston> So either the functionality needs to be extended to work correctly, or removed from private BPs, IMO
<deryck> cjohnston, what do you call "correctly?"  What needs to happen to be correct?
<cjohnston> Where if a blueprint is proposed to a sprint, a 'sprint manager' can approve the blueprint for the sprint, and Summit sees the approved blueprint and creates a (private) meeting for the blueprint
<cjohnston> Otherwise it doesn't perform as expected
<cjohnston> by the user, based on how other blueprints perform
<deryck> The point about the sprint manager not being able to see it is a good point.  I'm not sure its a bug or not yet.  I need to think through that case.
<cjohnston> but even still, if the sprint manager could see it, Summit couldnt
<deryck> As for summit, that's different.  it's an add-on bit to launchpad.  so I don't think we want to automatically trust anonymous bots.
<deryck> ah, ok, I guess I don't understand how summit gets access.
<deryck> I thought it was anonymous API script, based on what you said earlier.
<cjohnston> 1 sec... trying to find the link
<cjohnston> https://launchpad.net/sprints/uds-r/+temp-meeting-export
<deryck> cjohnston, ok, so I think it's good to chat and work out any kinks here, but I don't think this anything we're going to change on our end before UDSâ¦.
<deryck> cjohnston, the people who will be bitten by this are a small group....
<cjohnston> well.. the new stuff is only in beta currently correct?
<deryck> cjohnston, only people who are both lp beta testers and have commercial subscriptions for lpâ¦..
<cjohnston> oddly enough I actually fall into it.. hehe
<deryck> fair point :)
<cjohnston> but yes.. it is a small group, and for this UDS I think we are ok.. but I'd like to have it correct before we start the planning for the next UDS when all of this is live
<deryck> cjohnston, so people who do fall in that, need to be careful about using private blueprints linked to UDS.
<cjohnston> so somehow we would have to get that info out to them
<deryck> as to the larger point, everything private in lp works the way blueprints do now.  If it's proprietary, then no one gets access accept those people explicitly given access.  I don't think we should change this.
<deryck> maybe that means that proprietary blueprints shouldn't be able to be linked to a sprint.
<deryck> I'm not sure agree, but its a fair point to argue.
<cjohnston> That's my thought.. or that if they are, then everything else needs to be changed to accomidate the fact that they can link it
<cjohnston> otherwise it gets stuck in some foriegn world
<deryck> well, no, it's private.  As they the word "private" implies. :)
<cjohnston> correct.. its private.. but they propose it to a sprint, but it never makes it onto the schedule
<cjohnston> thats where the problem as I see it lies
<deryck> are sprints proposed, or just purely linked to the sprint object?
<deryck> I assumed it was a straight link, but if it is an proposed relationship that can never be approved or rejected, then yes, that's a clear bug.
<cjohnston> I don't understand the question? A sprint itself is created, and a BP is proposed against a sprint
<deryck> ok, so I just checked.  You get the link on your blueprint, but it's not listed without approval.
<deryck> so you have kind of an unofficial link to a sprint
<cjohnston> so how would it get approved
<deryck> it wouldn't.  but why is that bad if it's a private object?
<cjohnston> then it shouldn't be able to be proposed.. what is the point of proposing a blueprint to a sprint if you cant get the blueprint approved for the sprint?
<deryck> I don't know.  It's hypothetical, sure.  I don't know that anyone wants this relationship.  But I imagine someone *could* be working on something private while at UDS, and like the link to the sprint to see to establish the relationship, i.e. as a point of navigation to the sprint for other stuff.
<deryck> I don't know this, I'm just speculating.
<deryck> But without talking to people who might use this, we can never know.
<cjohnston> I guess to me, I would rather that if the feature not work the same as it works for everything else, it shouldn't exist. I am not opposed to making it possible to propose a private bp, get it approved (somehow?), we want to make summit authenticate anyway, so then we could create a private meeting (a feature that already exists in Summit) based upon a private blueprint
<deryck> if people's expectations are that proposing would open this to approval, but not reveal the object otherwise, we could make that possible.
<deryck> Though personally, I don't know that an approved, but hidden blueprint, is anymore valuable that an unapproved and hidden one.
<deryck> but sure, if this was tied into the functionality to great private meetings that would be nice.
<cjohnston> approved would allow (as long as Summit can get the info that it needs) for the blueprint to make it on the meeting schedule
<cjohnston> as a private meeting
<deryck> right.  which is a fair point.
<deryck> but we don't have capacity to do all that extra work.
<deryck> someone else will need to build on what we've done.
<deryck> I'm not sure it's so broken that we need to remove the ability to link to the sprint at this point, but I'm open for arguments, obviously.
<cjohnston> then IMO, it's a bug currently that you can propose a private blueprint to a sprint but not allow it to get on the schedule.
<cjohnston> how hard would it be to add if bp=private hide sprint stuff
<deryck> cjohnston, not that hard, but it is some work, and I'm not convinced we need to do it.  let's just see what happens at this UDS.
<cjohnston> I'm not sure that this UDS is a good test as, as you said, its beta and required to be a member of one of these projects
<lifeless> flacoste: hi; can we push the meeting back a bit ?
<lifeless> deryck: cjohnston: I don't think being linked is a bug per se: the lack of reciprocal disclosure is (the rule is, if you link something private with something less private, the private thing becomes visible to the same degree)
<cjohnston> so then it would become a publicly viewable blueprint?
<lifeless> deryck: cjohnston: But for proprietary we want to prevent people making such links outside the things that the project owner controls, so that bug #2.
<lifeless> deryck: cjohnston: as for summit, +temp-meeting-export should DIAF and be replaced by an API call, and the API call can be authenticated by the bot, just have the summit bot be granted access to the relevant private project blueprints/bugs through access policy grants.
<deryck> this is what I was thinking, when I thought it was a bot.  just grant access to the bot for the blueprints you want scheduled.
<cjohnston> lifeless: we aren't against that.. does the API stuff already exist to do that?
<lifeless> cjohnston: you wouldn't want the spider approach the current API has for this
<lifeless> cjohnston: you'd want to literally port the meeting export code to a single API call (e.g. sprint.export())
<lifeless> deryck: yeah, +1.
<deryck> lifeless, also, I'm not sure that I agree with the rile that the thing should gain the visibility of the thing linked, but if we've been consistent with that, I'll abide by the rule. :)
<lifeless> The thing that will bite you though, is the idea of linking proprietary blueprints to public sprints either needs to violate the rule for interaction of objects, or needs to disclose the existence of the blueprint and the project.
<deryck> right
<lifeless> deryck: we're trying to be consistent with it, because it makes reasoning about UI and behaviour a /lot/ easier.
<lifeless> deryck: for all that it raises issues, it would have avoided disasters too - like the public/private bug dupe one.
<cjohnston> if everything was authed, summit could gather the info, and keep the info private in Summit, if the ability existed in LP, though it sounds like it doesnt
<lifeless> the sort of thing that this rule solves is:
<lifeless>  - invisible actions
<lifeless>  - spying
<lifeless>  - inexplicable failures (e.g. timeslot X is used. Not its not! yes it is!)
<deryck> I don't disagree it solves all those issues.  But it's also very easy to leak private data.  And I'm not sure which is worse.
<deryck> but I know we've had all these discussions before.  I'm not trying to dredge them back up.
<deryck> not we == lifeless/deryck but we == lp devs.
<lifeless> yeah :)
<flacoste> lifeless: i need to head off in a little less than 1h, maybe we would be better to postone to later in the week then?
<lifeless> flacoste: ok, tomorrow?
<lifeless> flacoste: or your wed, can't do later than that, on leave.
<flacoste> lifeless: yeah, let's do wed after the TL call
<lifeless> hokaydokay
<cjwatson> wgrant: Do you think https://bugs.launchpad.net/launchpad/+bug/3456 is still an issue?  (The bug title is outdated; read at least comment #1.)
<_mup_> Bug #3456: queue doctest removes security proxies <lp-soyuz> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/3456 >
<cjwatson> I thought it was acceptable that the uploader was zopeless, though perhaps I'm wrong.
* benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300
<wgrant> cjwatson: Not worth having a bug for
<StevenK> lifeless: https://lpci.wedontsleep.org/job/devel/lastFailedBuild/ hint hint
<lifeless> grrr at your cert
<StevenK> Yeah, sorry, I should fix that.
<lifeless> StevenK: its got crud in it:
<lifeless> successful: lp.services.scripts.tests.test_cronscript_enabled.TestCronscriptEnabled.test_enabled_cronscript
<lifeless> teskill: 186: No such process
<lifeless> Stopping lxc
<lifeless> t: lp.codehosting.vfs.tests.test_branchfs.TestLaunchpadTransportAsync.test_rmdir
<lifeless> see how test: lp.codehosting.vfs.tests.test_branchfs.TestLaunchpadTransportAsync.test_rmdir
<lifeless> and kill: 186: No such process\nStopping lxc
<lifeless> are intermingled
<StevenK> lifeless: Hmm, maybe I need to redirect stderr away?
<sinzui> wgrant, StevenK, wallyworld: can you see http://blog.launchpad.net/?p=3742&preview=true
<lifeless> StevenK: stderr /must/ be a separate fd, is it shared with stdout atm ?
<StevenK> lifeless: At the moment, I'm not redirecting them at all
<lifeless> StevenK: can i see the config ?
<StevenK> lifeless: Jenkins config, builder config? Which?
<lifeless>  yes
<lifeless> the jenkins project config
<lifeless> to start with
<lifeless> brb
<StevenK> lifeless: You should have access to configure the project on jenkins
<StevenK> wgrant: My test in the LP branch passes with the new lazr.restful hacked in.
<lifeless> StevenK: I don't have a login ?
<StevenK> lifeless: Login via SSO
<StevenK> That should work
<wgrant> StevenK: Great
<lifeless> StevenK: did you just change to concurrency=1 ?
<StevenK> Yes, wgrant's suggestion.
<lifeless> ok, so sudo /usr/local/bin/lp-setup-lxc-test lptests
<lifeless> is buggy
<lifeless> bbs, C need sme
<StevenK> lifeless: That is based on the example shipped in lpsetup. I can pastebin it if you wish
<StevenK> wgrant, wallyworld: https://code.launchpad.net/~stevenk/lazr.restful/dict-unmarshaller-none/+merge/126615
<StevenK> wgrant: Do you think the test I've added is valuable?
<wgrant> StevenK: Hm
<wgrant> StevenK: Don't we care mostly about marshalling?
<wgrant> Unmarshalling might be useful too, but it's less important here.
<wgrant> The bug you're trying to fix just cares about marshalling.
<StevenK> No, the bug I'm fixing is about unmarshalling
<StevenK> It gets None and tries to unmarshall that as a dict
<StevenK> wgrant: And sorry, I meant the LP test, not the lazr.restful test.
<wgrant> Ah, in the etag, right
<wgrant> StevenK: Does that test fail without the lazr.restful upgrade?
<wgrant> It doesn't look like it'll exercise the ETag path.
<wgrant> Because it's unconditional.
<StevenK> wgrant: http://pastebin.ubuntu.com/1255044/
<wgrant> Ah, so it does a conditional anyway, great.
 * wallyworld sighs, another hard lockup :-(
<StevenK> wgrant: So it's fine?
<wgrant> StevenK: yes
<StevenK> wgrant: Can you approve the lazr.restful MP then so I can land it?
<StevenK> wgrant: If you think the test I've added to LP adds value, I can leave it in, or I can just bump versions.cfg and move on.
<wgrant> StevenK: Does marshalling None as a dict work?
<StevenK> wgrant: Obviously, due to the lazr.restful test
<wgrant> StevenK: That tests *un*marshalling.
<StevenK> Yeah, I should learn to read at some point.
<StevenK> Let me check.
<StevenK>     >>> print dict_marshaller.marshall_from_json_data(None)
<StevenK>     None
<StevenK> There's an existing test
#launchpad-dev 2012-10-02
<wgrant> Right.
<lifeless> StevenK: its not part of lp-setup ? yes, I'd like to see it
<StevenK> lifeless: It is, it's shipped as an example.
<StevenK> lifeless: http://pastebin.ubuntu.com/1255107/
<lifeless> ok
<lifeless> can you pastebin the lxc-start-ephemeral script you have ?
<lifeless> StevenK: and what version of testtools and subunit and testrepository do you have (outside the lxc containers)
<StevenK> lifeless: Versions http://pastebin.ubuntu.com/1255115/
<lifeless> wgrant: heat cam eback, the very next year
<StevenK> lifeless: lxc-start-ephemeral is from precise, lxc 0.7.5-3ubuntu63
<lifeless> ugh
<lifeless> its got an exit 0 in there. That will cause bad signal, if ssh exits non-zero, we want that to propogate.
<lifeless> Someone has been 'fixing' it I bet.
<lifeless> wgrant: ^
<lifeless> that said
<lifeless> StevenK: Stopping lxc goes to
<lifeless> &2
<StevenK> I wonder if the lxc from precise is not right and I need a different version?
<lifeless> StevenK: we need to figure out why its ending up in stdout
<lifeless> StevenK: I wonder, perhaps jenkins is insane
<StevenK> wallyworld: How did you build the 0.19.7 tarball?
<wallyworld> python setup sdist
<wallyworld> setup.py even
<StevenK> wallyworld: So did I, but I have 3 more files than you
<wallyworld> in the dist directory?
<StevenK> No, in the tarball
<wallyworld> which ones?
<StevenK> +/setup.cfg
<StevenK> +/src/lazr.restful.egg-info/not-zip-safe
<StevenK> +/src/lazr.restful.egg-info/PKG-INFO
<wgrant> lifeless: Heat came back?
<wallyworld> i should have had the pkg-info in mine
<wallyworld> not sure about the others
<wallyworld> i don't know why there's a difference
<lifeless> wgrant: see the bug bryceh filed
<wgrant> lifeless: Ah yeah, that
<wgrant> lifeless: That's why I'm interested in locks :)
<lifeless> indeed
<wgrant> lifeless: However
<wgrant> lifeless: Note the time there.
<wgrant> lifeless: I don't think this particular case is mostly DB locks, but I don't know for sure which is why I wanted to get reports rolling :)
<wgrant> I suspect this is just Python
<wgrant> As the statement timeout should mean the query times out in well under 23 seconds...
<lifeless> wgrant: query times are not checked during lock waits
<lifeless> or weren't, its possible that 9.1 fixed that.
<wgrant> lifeless: Are so.
<wgrant> In 9.1 they definitely are
<lifeless> ok, so thats nice.
<wgrant> I'm pretty sure they were in 8.4 too
<lifeless> 24 seconds is slow for python, and there is no data to be returning there.
<wgrant> But I don't remember for sure
<wgrant> By "Python" I mean "someone else probably had a lock"
<wgrant> Because as you say, there is no data.
<wgrant> It may be interesting to check postgres logs for the killed transaction.
<wgrant> See when it actually happened.
<wgrant> We log errors, right?
<lifeless> pretty sure
<lifeless> StevenK: jml: jelmer: so the testrepository you have is old, the testing cabal builds are failing.
<StevenK> lifeless: Ah
<lifeless> thats not the cause of the issue AFAICT, but its a nuisance.
<StevenK> wallyworld: How did you add the 0.19.7 milestone when https://launchpad.net/lazr.restful/trunk times out?
<wallyworld> StevenK: url hack
<wgrant> Huh, that times out?
 * wgrant guesses workitems
<wgrant> No
<wgrant> Maintenance is not allowed to URL hack as the first step
<wgrant> Fix timeout instead :)
<StevenK> wgrant: OOPS-539a88c269cd05876823126d6dbb9e50
<wallyworld> wgrant: i'm not going to fix a timeout to delay adding a release
<wallyworld> add release first
<wgrant> Yeah, it's the workitems regression :(
<wgrant> 23 150ms queries
<StevenK> wallyworld: What's the URL then?
<wgrant> +addmilestone
<wgrant> IIRC
<wallyworld> hmm. +addmilestone
<wallyworld> i think
<lifeless> so, working through this
<lifeless> lxc-start-ephemeral, assuming my copy == stevenk's copy, as he did not pastebinit :), outputs 'Stopping lxc' to &2, stderr
<lifeless> testrepository, which spawns the lp-setup-lxc-test processes uses stdout=subprocess.PIPE, stdin=subprocess.PIPE)
<wgrant> lifeless, StevenK: It's odd that we don't see this problem in the DC
<wgrant> Unless lp-setup-lxc-test does it
<wgrant> We use a puppeted version of that in the DC
<lifeless> so we'd expect stderr to be the jenkins provided stderr fd inherited from testrepository
<StevenK> echo "Stopping lxc" >&2
<lifeless> however
<lifeless> we see all the ephemeral chatter in this file
<lifeless> Setting up ephemeral container...
<lifeless> Starting up the container...
<lifeless> ah, thats on stdout
<StevenK> That's from lp-setup-lxc-build I think
<wgrant> http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/70/steps/shell_9/logs/stdio
<wgrant> red == stderr
<wgrant> black == stdout
<lifeless> StevenK: its from lxc-start-ephemeral
<lifeless> StevenK: see 'setup_container'
<StevenK> Ah, indeed
<StevenK> wallyworld, lifeless: Can haz pypi rights for lazr.restful?
<wallyworld> sure
<lifeless> so in http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/70/steps/shell_9/logs/stdio, Stopping lxc is on stderr
<lifeless> as I pasted above, testrepository doesn't capture stderr at all, it does the default for subprocess
<wallyworld> StevenK: user name is SteveK?
<wallyworld> StevenK even
<wallyworld> ?
<StevenK> wallyworld: Yeah, stevenk
<wallyworld> StevenK: should be done
<wgrant> Hmmm
<wgrant> Postgres really really seems to hate BitmapOr
<wgrant> Even when the alternative is two seqscans.
<lifeless> a quick re-check of subprocess doesn't given any obvious bugs
<lifeless> StevenK: can I login on the executor, will need root access
<StevenK> lifeless: ubuntu@184.73.75.214
<StevenK> I've fed it your lifeless-64 ssh key
<lifeless> waaaay but its busay
<lifeless> or
<lifeless> StevenK: is port 22 open in the sec group ?
<StevenK> I'm not sure if it's open to 0/0
<lifeless> ssh: connect to host 184.73.75.214 port 22: Connection timed out
<StevenK> Right
<StevenK> lifeless: Should I add just your IP or a netblock?
<lifeless> shrug
<StevenK> lifeless: You should be good
 * StevenK drops lazr.restful-0.19.[2-5].tar.gz from the download-cache
<StevenK> lifeless: Ah, nice. You're right for the mo?
<lifeless> jenkins, you mad bro
<lifeless> testr     12536         jenkins    1w     FIFO                0,8      0t0      65754 pipe
<lifeless> testr     12536         jenkins    2w     FIFO                0,8      0t0      65754 pipe
<StevenK> Oh
<lifeless> testr     12536         jenkins    0r     FIFO                0,8      0t0      65753 pipe
<lifeless> testr     12536         jenkins    1w     FIFO                0,8      0t0      65754 pipe
<lifeless> testr     12536         jenkins    2w     FIFO                0,8      0t0      65754 pipe
<lifeless> testr     12536         jenkins    3u      REG              202,1  6975862      32201 /home/jenkins/launchpad/lp-branches/workspace/devel/.testrepository/tmp2CwhL_
<lifeless> testr     12536         jenkins    5r     FIFO                0,8      0t0      69881 pipe
<lifeless> thats the full set
<lifeless> testr's in, out, err, the temp file its stream its inputs too, 4 would have been the stdin for the child process, which is closes, and 5 is where its reading hte child output from
<StevenK> Hah, so it's Jenkins bug/misfeature
<lifeless> well
<lifeless> we have correlation
<lifeless> not causation
<lifeless> StevenK: so, we run sudo /usr/local/bin/lp-setup-lxc-test lptests /home/jenkins/.ssh/id_rsa $IDOPTION $LISTOPT ?
<lifeless> what does the DC run ?
<StevenK> sudo /usr/local/sbin/lp-setup-lxc-test I think
<StevenK> I think the only difference is the DC script has stuff hardcoded
<lifeless> fd's for lp setup look ok
<lifeless> lp-setup- 12540            root    0r     FIFO                0,8      0t0      69880 pipe
<lifeless> lp-setup- 12540            root    1w     FIFO                0,8      0t0      69881 pipe
<lifeless> lp-setup- 12540            root    2w     FIFO                0,8      0t0      65754 pipe
<lifeless> err is as we expect, out is the pipe
<lifeless> lxc-start 12541            root    0r     FIFO                0,8      0t0      69880 pipe
<lifeless> lxc-start 12541            root    1w     FIFO                0,8      0t0      69881 pipe
<lifeless> lxc-start 12541            root    2w     FIFO                0,8      0t0      65754 pipe
<lifeless> that looks fine too
<lifeless> and finally ssh
<lifeless> ssh       13831            root    0r     FIFO                0,8      0t0      69880 pipe
<lifeless> ssh       13831            root    1w     FIFO                0,8      0t0      69881 pipe
<lifeless> ssh       13831            root    2w     FIFO                0,8      0t0      65754 pipe
<lifeless> so, theres nothing down through to the ssh layer that can explain stderr output ending up on stdout for testr
<lifeless> other stderr output *does* end up in the jenkins console
<lifeless> the added address to .ssh/known-hosts specifically
<lifeless> https://lpci.wedontsleep.org/job/devel/1102/console
<lifeless> Warning: Permanently added '10.0.3.39' (RSA) to the list of known hosts.
<StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/preview-diff-none-type/+merge/126601
<wallyworld> StevenK: perhaps the assertVoteReference could be _assertVoteReference?
<wallyworld> otherwise looks good
<lifeless> StevenK: kill the build? run again as concurrency 4 ? the stream written so far is intact.
<LPCIBot> Project devel build #1102: ABORTED in 2 hr 23 min: https://lpci.wedontsleep.org/job/devel/1102/
<StevenK> lifeless: #1103 away
<StevenK> wallyworld: http://pastebin.ubuntu.com/1255162/
<wallyworld> +1
<StevenK> Bleh, my oops got pruned again
<StevenK> Clearly, I need to log into neem and make copies
<wgrant> lifeless: See also OOPSes like https://oops.canonical.com/oops/?oopsid=OOPS-c182f7f94210ede212ae5b835ef993c7
<wgrant> lifeless:  12s query that couldn't be blocked by any normal lock
<wgrant> We have a general problem, and I suspect that Bryce's bug is mostly that general problem.
<lifeless> wgrant: sounds plausible
<lifeless> this is mystifying
<StevenK> What is?
<lifeless> I was having trouble finding any evidence of a problem
<lifeless> xvfb-run   8494         jenkins    0r     FIFO                0,8      0t0   11112539 pipe
<lifeless> xvfb-run   8494         jenkins    1w     FIFO                0,8      0t0   11112540 pipe
<lifeless> xvfb-run   8494         jenkins    2w     FIFO                0,8      0t0   11112540 pipe
<StevenK> Ah
<lifeless> but - this is such evidence
<lifeless> xvfb-run has stderr and stdout linked
<lifeless> so we can corrupt a stream within the subprocess
<lifeless> across the ssh barrier
<StevenK> lifeless: You think this is jenkins level or lower?
<StevenK> I'm not sure if Jenkins does something like exec 2>&1 that it will hit processes it spawns
<lifeless> well, I'm still only at correlation stage
<lifeless> jenkins definitely is doing 2>&1
<lifeless> but that shouldn't have any way of affecting children of children if they make their own pipes
<lifeless> which testr does for stdout
<lifeless> ssh normally preserves stderr as a separate fd
<lifeless> e.g.
<lifeless> ssh localhost 'echo true >&2' 2>/dev/null
<lifeless> ssh localhost 'echo true >&2'
<lifeless> compare the output
<lifeless> it knows whats stdout and stderr on the slave, and faithfully lines that up on the invoking side
<lifeless> so WTF would xvfb-run be seeing stderr=stdout
<lifeless> ssh -n -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/jenkins/.ssh/id_rsa jenkins@10.0.3.132 -- env -u LANG xvfb-run --error-file=/var/tmp/xvfb-errors.log --server-args='-screen 0 1024x768x24' -a /home/jenkins/launchpad/lp-branches/workspace/devel/bin/test -vvv --shuffle --subunit --load-list /home/jenkins/launchpad/lp-branches/workspace/devel/temp/tmp0bW4iD
<lifeless> is the command being run
<lifeless> ssh -n localhost 'env echo true >&2'  2>/dev/null
<lifeless> still good
<lifeless> I'll file a bug on the silly exit 0 / exit 1 stuff
<lifeless> https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1059943
<_mup_> Bug #1059943: lxc-start-ephemeral masks process exit code <lxc (Ubuntu):New> < https://launchpad.net/bugs/1059943 >
<wgrant> lifeless: I already filed and fixed that in quantal
<lifeless> wgrant: oh, can you dupe it appropriately please then ?
<wgrant> And I have a precise backport branch.
<lifeless> DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1
<lifeless> ^ xvfb-run
<lifeless> StevenK: so, smoking gun.
<lifeless> StevenK: this may not explain all the corruption we're encountering, but I think it covers a pretty good chunk thereof
<lifeless> StevenK: are you running lucid or precise in the lxc containers?
<lifeless> StevenK: https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1059947
<_mup_> Bug #1059947: xvfb-run munges stdout and stderr together <xorg-server (Ubuntu):New> < https://launchpad.net/bugs/1059947 >
<lifeless> StevenK: it doesn't explain HTF 'Stopping lxc' got from lxc-start-ephemeral &2 into &1
<wgrant> blink
<wgrant> Ah
<lifeless> wgrant: ?
<wgrant> MilestoneTag.specifications had me confused for a while
<wgrant> Why it was intersecting all the resulting specs
<wgrant> But it makes some sense now
<LPCIBot> Project devel build #1103: STILL FAILING in 1 hr 32 min: https://lpci.wedontsleep.org/job/devel/1103/
<StevenK> lifeless: Whatever lp-setup does, which I guess is lucid?
<wgrant> Right, inside the containers is lucid
<StevenK> xvfb-run bug? :-(
<lifeless> StevenK: contributing factor at most
<lifeless> I've just edited the bug out of the root container and started another run
<StevenK> So buildbot doesn't xvfb-run, then?
<lifeless> timing bugs are terrible
<wgrant> buildbot does
<wgrant> But isn't the problematic stderr coming from *outside* xvfb-run?
<lifeless> wgrant: one case
<lifeless> which as I said
<lifeless> 15:52 < lifeless> StevenK: it doesn't explain HTF 'Stopping lxc' got from lxc-start-ephemeral &2 into &1
<wgrant> Ah, right
<lifeless> but
<lifeless> it may be confused signals
<lifeless> see
<lifeless> https://lpci.wedontsleep.org/job/devel/1103/console
<lifeless> where we have test data in the jenkins console
<lifeless> that happens when tests are mangled at a lower layer
<lifeless> and subunit's parser decides its seeing junk rather than tests
<lifeless> e.g. the subprocess race/rhing
<lifeless> which we need to get a reproducable test case for
<lifeless> _StringException: lost connection during test 'lp/registry/javascript/tests/test_milestone_creation'
<lifeless> the end of https://lpci.wedontsleep.org/job/devel/1103/console
<lifeless> shows
<lifeless> test: lp.codehosting.pukill: 186: No such process
<lifeless> Stopping lxc
<lifeless> ller.tests.test_scheduler.TestPullerWireProtocol.test_methodDispatchWithArguments
<lifeless> which is, I think, testr's stdout intermingled with the shared stderr, and - *this is fine*.
<lifeless> Its expected given jenkins mangling of the two onto one stream
<lifeless> StevenK: you probably want two forked packages; lxc (for the lxc-start-ephemeral fix from wgrant) and a xvfb-run with fixed stderr handling
<wgrant> https://code.launchpad.net/~wgrant/ubuntu/precise/lxc/bug-1050351
<lifeless> StevenK: so we'll see if thats any better, and if it isn't we'll see about making a reproducable test case.
<lifeless> for now, I'm going to context switch a bit
<wgrant> StevenK: Thoughts on bug #810716 and bug #690834?
<_mup_> Bug #810716: Translations section builds have highest priority in copy builds <buildfarm> <Launchpad itself:Triaged> < https://launchpad.net/bugs/810716 >
<_mup_> Bug #690834: Language packages in P3As are also scored 0 <lp-soyuz> <oem-services> <soyuz-build> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/690834 >
<wgrant> For context, if section == 'translations' then the score short-circuits to 0
<wgrant> Ignoring any archive offset
<StevenK> Ywah
<wgrant> I suspect we just want to apply the archive offset to the 0
<StevenK> I'll be hitting that code to
<wgrant> So they'll sit at the bottom of the archive
<wgrant> Oh?
<wgrant> Oh
<wgrant> I see you took the card
<wgrant> I was just collecting those three bugs to fix
<StevenK> Oh, you can do it if you want
<wgrant> If you've been busy with jenkins and haven't actually started...
<StevenK> I've created a branch and found the code, so 'not really'
<wgrant> Ah, well I just wrote a test, so I'm probably slightly ahead.
<StevenK> Nothing of consequence, so feel free to steal the card and I'll find another bug
<StevenK> Or re-package lxc
<wgrant> Thanks
<wgrant> Heh
<StevenK> Haha
<StevenK> I can see the edit page in bug 1059324, but the reporter can't
<_mup_> Bug #1059324: Can't edit branch import <403> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1059324 >
<wgrant> StevenK: ~vcs-imports can see +edit-import, mortals can't.
<wgrant> It's likely that the link should just be hidden.
<wgrant> It may just be a matter of adding one line of decorator.
<StevenK> Right
<StevenK> @with_permission or so?
<wgrant> @enabled_with_permission, IIRC
<StevenK> Bleh, the qa-tagger hates multi-task bugs? :-(
<wgrant> Should work, normally.
<wgrant> What's the issue?
<StevenK> It stated orphaned for my rev
<StevenK> Maybe it insists on default_bugtask, but that would be silly
<wgrant> Hm
<wgrant> That's odd
<StevenK> wgrant: Hmm, lib/lp/code/browser/branch.py edit_import checks perms already. Is it checking the wrong thing?
<wgrant> StevenK: That's a very good point
<wgrant> Although I wouldn't trust that function to not be insane
<wgrant> Given that it sets enabled = True a line earlier
<StevenK> Haha, I wonder if that's the fix
<wgrant> It's just a normal local
<wgrant> So I doubt it
<StevenK> 6 Unauthorized: (<CodeImport for ~rudloff/chaton-cms/0.1>, 'updateFromData', 'launchpad.Edit')
<StevenK> That seems more like he can view the page and the submit is blowing up
<wgrant> Right, that means the user doesn't hold launchpad.Edit on the CodeImport
<wgrant> What's the view permission?
<StevenK> launchpad.Edit on IBranch, looks like
<wgrant> Ah
<wgrant> That doesn't explain why there's a link, though
<wgrant> Perhaps there is no link
<wgrant> And the user URL-hacked.
<wgrant> Try to reproduce locally or on qastaging?
<StevenK> How did he POST, then?
<wgrant> The view is Branch:+edit-import, so the permission is wrong
<wgrant> It'll load for the branch owner
<StevenK> We have a few OOPSes, so we should be able to work it out
<wgrant> But the edit_import menu item correctly checks against branch.codeimport
<StevenK> OOPS-1d2dccea10b7b7c5f4bc11cf0eaa9ebf
<wgrant> The OOPS won't say
<wgrant> It'll show the referer as +edit-import either way
<StevenK> Right, so the ZCML is wrong
<wgrant> And can't really be right.
<StevenK> Not if we use for="...", right
<StevenK> wgrant: I'm not clear how to create a branch import
<wgrant> +new-import
<wgrant> http://code.qastaging.launchpad.net/launchpad/+new-import
<StevenK> wgrant: Right, so no link. URL hacking as the branch owner shows the form and submitting gives Not allowed here
<StevenK> Maybe we want to redirect away in initialize()
<StevenK> "You are not permitted to edit the import for this branch" etc
<wgrant> Or raise a Forbidden or so
<wgrant> No point being pretty if you have to URL-hack to get there.
<StevenK> Indeed
<StevenK> It already does for anonymous, indeed
<wgrant> If you waste more than two lines of code on this, you're probably doing it wrong.
<wgrant> StevenK: I think I'll take this opportunity to finally eliminate time-based scoring.
<wgrant> Any objections?
 * wgrant murders.
<StevenK> wgrant: We have time based scoring?
<StevenK> If it will complete quickly, score it higher?
<wgrant> No
<wgrant> If it's been in the queue for 5 minutes, give it a score bump of 5
<wgrant> an hour is a bump of 20
<StevenK> Oh, right
<wgrant> This ignores the fact that the tiebreaker is id, which is age anyway
<wgrant> And we don't run queue-builder any more
<StevenK> wgrant: Yes, murder it.
<wgrant> So the score is never calculated except when the build is 0 seconds old.
<wgrant> Thanks
<StevenK> Can we destroy queue-builder too?
<wgrant> I already got rid of most of it a while ago
<StevenK> wgrant: I'm at -4 with this branch with some cleanups and no test
<wgrant> :)
<wgrant> We have a lot of crap in our code.
<StevenK> Not sure if raise Forbidden is okay
<wgrant> Unauthorized may be more common, I think
<wgrant> But I forget
<lifeless> StevenK: ok, so the xvfb-run isn't the root cause, and whether it contributed is an open q
<lifeless> StevenK: tomorrow I shall poke further, the artifacts gathered should be sufficient
<StevenK> Ah
<lifeless> https://lpci.wedontsleep.org/job/devel/1104/console
<lifeless> shows subunit
<lifeless> so its escaped the parser again
<StevenK> lifeless: So I can kill the executor ec2 when the current one fails?
<StevenK> wgrant: Do you think a test is worth it?
<wgrant> StevenK: If you've deleted enough to still leave you net-negative :)
<LPCIBot> Project devel build #1104: STILL FAILING in 1 hr 28 min: https://lpci.wedontsleep.org/job/devel/1104/
<wgrant> StevenK, wallyworld_: https://code.launchpad.net/~wgrant/launchpad/bug-1058310/+merge/127414 https://code.launchpad.net/~wgrant/launchpad/bug-1056617/+merge/127412
<czajkowski> StevenK: wgrant kicking ass on bugs I see :) very nice
<wallyworld_> wgrant: r=me on 2nd one
<wgrant> Today should take us below 300 criticals :)
<wgrant> wallyworld_: Thanks
<StevenK> wgrant: And r=me on the first
<wgrant> Did we implement ParallelReviewing while I wasn't watching?
<czajkowski> wallyworld_: ah and you
<czajkowski> tab complete and me are not working at this hour
<wallyworld_> czajkowski: well, today i'm removing an old feature flag for dynamic bug listings which was released a while back. not sure if that's critical, but it needs to be done to keep the code clean and maintainable
<wallyworld_> tedious work
<czajkowski> which is needed
<wgrant> That's probably our biggest single bit of tech debt :)
<wallyworld_> i'm almost finished. a few more doc tests to go
<wallyworld_> maybe a bit optimistic to say "almost". "substantially" is better
 * wgrant plays buildbot roulette
<czajkowski> ah hat reminds me I need to find the bug on signing the CoC to make it easier to see if we can get some sort of check box implemented :D
<wgrant> Heh
<wgrant> StevenK: You have a couple of ancient branches on https://code.launchpad.net/launchpad/+activereviews
<StevenK> No I don't. :-)
<wgrant> Thanks.
<StevenK>  5 files changed, 24 insertions(+), 32 deletions(-)
<StevenK> I think that's far enough
<wallyworld_> StevenK: wgrant: have you noticed that the branch badges on the dynamic bug listing are no longer anchors linking to the associated branch?
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/no-edit-import-for-you/+merge/127417
<wallyworld_> StevenK: r=me
<StevenK> wgrant: I wonder if bug 1022334 can be closed due to YUI 3.5.1
<wgrant> StevenK: No
<wgrant> StevenK: The bug *listing* back behaviour may be fixed, but the bug reporting one probably isn't
<StevenK> I thought zope handled at this stuff, TBH
<wgrant> Our JavaScript is not Zope
<StevenK> Oh, it's a JS bug?
<wgrant> Well, +filebug is very JSey, so it's probably not unrelated.
<StevenK> Yeah
<wgrant> Or it could be that the view is too custom and doesn't let initial_values through properly.
<StevenK> wgrant: In https://oops.canonical.com/oops/?oopsid=OOPS-91e85c5a76936c13c1993df1985491e5 I guess the branch is None?
<wgrant> StevenK: Potentially because the requesting user can't see it
<wgrant> But it looks like it.
<wgrant> Yeah
<wgrant> StevenK: Yeah, it happens whenever the development focus is an import that isn't visible.
<StevenK> Heh, right
<StevenK> wgrant: So we should skip that bit?
<wgrant> That may be appropriate.
<StevenK> wgrant: It seems I broke buildbot. But it doesn't look to be my fault.
<wgrant> It just has a personal vendetta against you
<wgrant> Ah, yeah
<wgrant> We see that one sometimes :(
<StevenK> wgrant: QA! :-P
<wgrant> Did that like 90 seconds ago
<StevenK> lib/lp/code/templates/product-branch-summary.pt makes me sad
<StevenK> wgrant: I should force away?
<wgrant> StevenK: Yep
<adeuring> good morning
<wgrant> StevenK: tsk tsk
<StevenK> wgrant: Hm?
<wgrant> StevenK: You're not very good at deleting views :)
<StevenK> I was supposed to delete one?
<wgrant> You deleted +daily-builds
<StevenK> wgrant: Sure, but?
<wgrant> You missed 300 lines of code that was used only by it :)
<StevenK> Have you binned them, or shall I have at it?
<wgrant> They're miiiiiiine
 * StevenK makes a note to talk to wgrant's mother about sharing
<wgrant> We already finished sharing last month :)
<bigjools> s/sharing/closing legs/
<wgrant> No need for any more.
<wgrant> Ah, make that 400 lines.
<StevenK> wgrant: Meh, I'm -3500 and you're -1235 :-P
<wgrant> I won't be dictated by fact-checkers.
<wgrant> Also, http://webnumbr.com/launchpad-critical-bugs
<wgrant> We're 3 bugs away from where we were at the start of December.
<wgrant> Getting there.
<wgrant> StevenK: Your new test seems to be bad
<wgrant> If you're in AppServerLayer, you're probably trying to catch the exception over HTTP
<wgrant> Which isn't likely to be a gigantic success.
<StevenK> Hmmm, I thought it was passing locally
<StevenK> Let me roll it back
<wgrant> stub: Huh
<wgrant>     "specification_product_name_uniq" UNIQUE CONSTRAINT, btree (name, product)
<wgrant> Why is it that way around? :/
<stub> lookup by name?
<wgrant> Possibly, but it seems pretty odd
<wgrant> I prefer the theory that someone tried to make it as useless as possible.
<stub> it is the only suitable lookup by name, which would be for traversal
<wgrant> Right, but the URL is /product/+spec/foo
<wgrant> I don't know much that looks up by unqualified name.
<stub> yeah, so it can use that index.
<wgrant> Right, but the distribution index is around the right way :)
<wgrant> The product one is just inverted for probably no reason
<stub> it would have been inverted to look up by unqualified name. I see that is probably pointless, yes.
<wgrant> I'm not sure productseries and distroseries indices are useful, but I guess we might as well
<wgrant> The table isn't exactly high-write...
 * wgrant adds three
<wgrant> I didn't even consider that product might have been unindexed
<wgrant> Since that's basically the primary thing we've queried on for 7 years...
<wgrant> w
<wgrant> stub: I've added new indices (including UNIQUE (product, name) so we can kill off the other one), though the diff is taking its time to update
<wgrant> It's possible we want one on just name, but I don't think so
<stub> yup
<stub> unqualified doesn't make sense
<wgrant> It certainly doesn't make sense.
<wgrant> But it still might happen
<wgrant> Because a lot of blueprints is a little behind its sense quota.
<wgrant> But let's go without
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban | Firefighting: - | Critical bugs: ~300
<wgrant> stub: If that looks good I'll land it
<wgrant> frankban: Hi. I'm trying to clean up the review queue, and you have three branches on https://code.launchpad.net/launchpad/+activereviews. I think they're all for stuff that's now in lp:lpsetup, so they can be rejected from LP, right?
<stub> wgrant: yeah, that looks good. It might make more sense for these to be partial indexes, but I don't think we know enough yet to be sure.
<frankban> wgrant: yes
<wgrant> stub: Given the data volumes and rates we're looking at, it won't matter for another few years :)
<wgrant> frankban: Thanks
<wgrant> frankban: Can you set https://code.launchpad.net/~frankban/launchpad/setuplxc-remove-sleep/+merge/96422 from Approved to Rejected to get it off the list? It's proposed to a branch that I don't have review privs on.
<frankban> wgrant: thank you! and done
<wgrant> Great, thanks.
<lifeless> stub: should francis be the 2nd db reviewer going forward, or someone else ?
<czajkowski> morning folks
<stub> lifeless: Depends if Francis is comfortable giving the go ahead for patches. One feature of the reviews is to avoid hard to recover failures and dataloss.
<lifeless> stub: Indeed.
<lifeless> gnight
<wgrant> Night lifeless.
<adeuring> frankban: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/product-sharing-sec-adapter/+merge/127473 ?
<frankban> adeuring: sure
<adeuring> thanks!
<wgrant> adeuring: A couple of notes:
<wgrant> Product.information_type exists on production now
<wgrant> Though not on qastaging; you might be able to convince ops or stub to fix that
<wgrant> And also, LimitedView is probably not what you want, unless you've discussed that with Curtis/Robert already
<adeuring> wgrant: makes sense
<wgrant> View is what makes sense here
<adeuring> wgrant: i discussed this with sinzui for Ispecification
<adeuring> there were some pitfalls with lp.View
<wgrant> adeuring: LimitedView is intended to give people enough to know that something exists, but basically nothing else.
<sinzui> correct
<wgrant> Oh, sinzui's here
<czajkowski> sinzui: would you be free later on for a call re maintenance, have stand up now.
<sinzui> LimitedView reveals the spec name, displayname, and project, .public, and .information_type. They do not have icons that need to be revealed
<sinzui> yes
<sinzui> yes czajkowski
<wgrant> It's not clear that LimitedView is necessary for products at this point
<wgrant> But even if it is, it would probably expose name, displayname, owner, and that's about it
<czajkowski> sinzui: thanks gew Q's need to be handed over and want to go through them first
<sinzui> wgrant, subscribing someone to a bug, blueprint, or branch will require limited view to the product and possibly series
<wgrant> sinzui: I think those are likely to give full View to the product.
<wgrant> But that may be yet to be established.
<sinzui> wgrant, adeuring, without limited view, you cannot traverse to get to the artefact.
<wgrant> Possession of View implies possession of LimitedView by default
<sinzui> wgrant, We are not here to establish that, Deryck's squad are. We know that traversal requires access to .name to get the a subscribed private arttfact
<wgrant> Sure
<wgrant> But that doesn't mean that 90% of the attributes that were zope.Public should be launchpad.LimitedView now
<wgrant> 89% should be launchpad.View, 1% should be launchpad.LimitedView
<wgrant> Otherwise LimitedView is meaningless.
<sinzui> wgrant, I agree. I think 7 attributes are about the right number
<wgrant> Right :)
<sinzui> limitedview is intended to convey enough information to allow someone to identify the thing they are looking at. It deters spying and deceit when a private something is in public relationship
<sinzui> adeuring, OEM projects are only willing to reveal their name, displayname, icon, logo, .public, .information_type, and maybe .active to contractors that they subscribe to bugs and branches
<adeuring> sinzui: ok; just working replacining lp.LimitedView with lp.View for the current branch
<sinzui> okay.
<frankban> adeuring: should I wait for you to push these changes?
<adeuring> frankban: make sense, I think
<frankban> adeuring: ok
<adeuring> frankban: i renamed the permission to lp.view now. I'll add lp.limitedview in a later branch for the properties that need it. Testing both permissions would let the diff size explode ;)
<frankban> adeuring: ok
<deryck> abentley, adeuring -- always have troubles getting into the hangout the first time, still trying....
<abentley> deryck: I'm confused about where we stand wrt NULLs in Product.information_type.
<deryck> abentley, we need a garbo job to fill in values for existing products. nulls are currently allowed.
<abentley> deryck: I think we don't have a card for the garbo job.  And we also need to fix the sampledata.
<deryck> abentley, I lumped the garbo job into your current card back when I created it.  And I can add a card now for sampledata.
<deryck> abentley, and I don't mind if you split off that garbo stuff into a new card, too.
<abentley> deryck: I read my card as stuff to do "after information_type + garbo job for current values"
<deryck> abentley, yeah, I probably phrased it badly. :)
<abentley> deryck: I see what you meant now.
<abentley> deryck: Do you know whether sampledata updates can land in stable?  ISTM there would be no value to landing them in db-stable.
<deryck> abentley, I believe they can indeed land in stable.
<stub> Nothing except db patches lands on db-devel/db-stable now
<stub> Occasionally some tests to go with db patch, but that is really rare.
<abentley> stub: Thanks.
<abentley> stub: But if a db patch requires sampledata changes, we should still land both as one branch, right?
<stub> yeah
<stub> The bare minimum required to keep the tests passing and the build working, or if your db patch contains stored procedures tests to prove the code works.
<deryck> abentley, what was the feature flag you used for the new project UI?
<abentley> deryck: 'disclosure.private_projects.enabled'
<deryck> abentley, thanks
<xnox> what is initial LOC credit for somebody who never committed to lp?
<xnox> -100, 0, 100 ?
<sinzui> 0
<adeuring> frankban: if you are  still around, could you do another review? https://code.launchpad.net/~adeuring/launchpad/correct-permission-check-for-iproduct/+merge/127518
<frankban> adeuring: on a call now, but I will take a look in 30 or so...
<adeuring> frankban: thanks! (this MP is a bit smaller, BTW)
<deryck> abentley, adeuring, just to confirm, we do still need PROJECTGROUP/+newproject updated to match projects/+new
<deryck> so I'll leave the card on the board, obviously.
<abentley> deryck: roger.
<adeuring> ack
* frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300
<deryck> abentley, adeuring -- qastaging is updated for Product.information_type now.
<adeuring> deryck: cool
<xnox> sinzui: ok, thanks.
<sinzui> xnox, do you need help finding cruft to remove?
<xnox> sinzui: well there is a link to bug reports that are trianged with the cruft-removal-bug tag ;-) so I'm good.
<sinzui> okay. Ping me if you need help.
<xnox> trying to pick something easy, as I haven't done lp development yet, but there are a couple bugs that hinder my productivity ;-) =)))))
<xnox> but are "low" priority for now.
<abentley> deryck: There's no OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/model-product-info-type/+merge/127558 ?
<deryck> abentley, I will shortly.  Just grabbing me a bite to eat.
<abentley> deryck: Actually, I missed a spot.  Fixing now.
<abentley> deryck: ready for review again: https://code.launchpad.net/~abentley/launchpad/model-product-info-type/+merge/127558
<deryck> abentley, ack
<abentley> deryck: Garbo job was really easy with your suggestion: https://code.launchpad.net/~abentley/launchpad/product-info-type-garbo/+merge/127579
<deryck> oh nice
<deryck> abentley, r=me for both.
<abentley> deryck: thanks.
<deryck> np
<abentley> deryck: How do we decide whether EMBARGOED/PROPRIETARY are reasonable information_types for a given Product?  Do we look for a commercial subscription or something?
<deryck> abentley, yes, you have to have a commercial subscription to use them.
<abentley> deryck:
<abentley> deryck: I'm going to restrict it in the model for now, and we can add db constraints afterwards.
<deryck> abentley, sounds good.
 * deryck goes AFK for a little bit
<StevenK> lifeless: Moar Jenkins?
<wallyworld_> wgrant: StevenK: sinzui: you seen bug 1055766?
<_mup_> Bug #1055766: grep -R doesn't automatically search amazon <apport-bug> <i386> <oneiric> <Ayatana Design:Invalid> <command-not-found (Ubuntu):Invalid> <gnome-terminal (Ubuntu):Invalid> < https://launchpad.net/bugs/1055766 >
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/no-edit-import-for-you-redux/+merge/127613
<wallyworld_> StevenK: the test comment should be something like "Unauthorised users are forbidden....." since it's not just the branch owner who is not allowed
<StevenK> wallyworld_: http://pastebin.ubuntu.com/1257063/
<wgrant> StevenK: Given lifeless is probably terribly busy, might be better to throw broken Jenkinses at me sometimes
<lifeless> hi
<lifeless> I'm going to be poking it
<wallyworld_> r=me
<lifeless> StevenK: can you setup an executor ?
<StevenK> lifeless: I thought you only needed the build artifacts, but sure.
<lifeless> StevenK: reproducability is key, until I have a core recipe its an open question what will be needed
<lifeless> running just http://paste.ubuntu.com/1257072/ should give us a fail single-threaded.
<lifeless> StevenK: so what I want is an executor, and jenkins told not to run anything
<lifeless> StevenK: run one build, to seed it and recor da failure
<lifeless> and then I'll poke around as jenkins and reproduce etc
<lifeless> StevenK: so I want ssh, root access etc
<StevenK> lifeless: Said executor is up and in the middle of lp-setup
<StevenK> lifeless: Right, so I should add it to Jenkins and kick off a devel build too?
<sinzui> StevenK, I pulled the latests critical data. The chart is updated
<lifeless> StevenK: please, and we can configure jenkins to only do manual builds
<StevenK> lifeless: I already have
<lifeless> coolio
 * StevenK waits for Firefox to choke down sinzui's JS
<lifeless> StevenK: will you prep a fixed xvfb for the lp PPA
<sinzui> Not the js, it is 5 megs of bug data
<StevenK> I can, but as you say, it's no smoking gun
<lifeless> StevenK: it a necessary fix for reliability
<StevenK> For lucid, I guess
<lifeless> StevenK: any stderr IO from anything in the test suite or under it can corrupt a stream w/out that fixed.
<lifeless> StevenK: for precise too
<StevenK> I'm not sure if it's inside the container or outside that is the important one
<lifeless> inside
<lifeless> xvfb isn't present outside the container
 * StevenK stares at the xorg source package
<lifeless> xorg-server ?
<StevenK> Bleh
<StevenK> lifeless: http://pastebin.ubuntu.com/1257092/
 * StevenK stabs Contact this team's admins
<lifeless> StevenK: looks good to me
<StevenK> lifeless: If you have no complaints about the version, I'll upload it
<lifeless> StevenK: seems fine to me
#launchpad-dev 2012-10-03
<lifeless> StevenK: is https://lpci.wedontsleep.org/ still the jeknins?
<lifeless> StevenK: I don't see an attached executor
<StevenK> lifeless: That's because I haven't done it yet because I was noming
<lifeless> StevenK: ah kk
<StevenK> lifeless: Your key has been added
<StevenK> lifeless: ubuntu@23.22.38.59
<StevenK> lifeless: I've not updated xvfb did you want to hack that in before we kick off a build?
<lifeless> sure
<lifeless> win: $ lxc-ls
<lifeless> lptests
<lifeless> lptests
<lifeless> two for the price of one
<StevenK> Haha
<lifeless> StevenK: ah, lxc-start is still running
<lifeless> lptests setup still happening ?
<StevenK> No ...
<lifeless> the android-in-magazine thing is kindof cool
<lifeless> ok, well the base is running
<lifeless> so I'll just use that
<lifeless> StevenK: whats the password for jenkins, do you know ?
<lifeless> StevenK: doesn't seem to have passwordless sudo within the container
<StevenK> It ought to
<lifeless> jenkins@lptests:~$ sudo apt-get install xvfb
<lifeless> [sudo] password for jenkins:
<StevenK> It certainly does outside the container
<StevenK> lifeless: And it doesn't have one, the user is created with --disabled-password
<lifeless> StevenK: I am confuse
<lifeless> StevenK: how does lpsetup install stuff within it then ?
<StevenK> I have no idea how lpsetup does its thing
<lifeless> anyhow I've edited the file from outside
<lifeless> so xvfb-run is fixed
<lifeless> should I stop the container ?
<StevenK> Yeah, let's do that and I'll kick off a build
<lifeless> done
<StevenK> and done
<lifeless> what concurrency did you use ?
<StevenK> 4
<lifeless> hmm
<StevenK> That hasn't changed at least
<lifeless> lxc-ls shows 8
<lifeless> I detect a bug in lxc-ls
<lifeless> ls /var/lib/lxc/
<lifeless> lptests  lptests-temp-1FaKWwb  lptests-temp-c8jcW0W  lptests-temp-qoZ6ncr  lptests-temp-wbvIQLE
<lifeless> $
<StevenK> testr run --parallel --concurrency=4
<lifeless> yeah
<lifeless> I know
<lifeless> check out lxc-ls on the executor though :P
<StevenK> Haha
<StevenK> lxc has bugs? I'm SHOCKED
<StevenK> lifeless: It could be once is for the container and another to show its running?
<lifeless> argh
<lifeless> cp's should really be cp dest source
<lifeless> cp -t . <- awkward
<StevenK> Hah
<lifeless> StevenK: 'ps fax | grep python.*load-list | grep -v resume-layer | awk '{ print $14 }' | xargs cp -t .'
<lifeless> StevenK: captured the lists being executed.
<StevenK> You can't get that from the temp directory inside workspace/devel?
<abentley> lifeless, StevenK: I know little, but read this post that said "The lxc-list command lists anything thatâs a directory in /var/lib/lxc": http://jonathancarter.org/2012/09/29/llxc-my-little-python3-lxc-based-project/
<StevenK> Ah ha!
<StevenK>     raise LocationError(subject, name)
<StevenK> LocationError: (None, 'branch_type')
<lifeless> abentley: it also looks in cgroups
<lifeless> abentley: and the mount table - its just a shell script if you want to look at it, it is AFAICT just confused by the unionfs going on. Or something like that.
<abentley> lifeless: So far, I haven't used lxc.  Tried to set up juju-on-lxc last week, ran into some glitch with zookeeper.
<lifeless> abentley: ah, so yeah - juju-on-lxc does things more differently than juju-on-openstack, for instance, which I find odd.
<lifeless> abentley: lxc on its own is pretty nice, and lpsetup does a decent job of configuring it.
<lifeless> wgrant: / StevenK: I think a 'here is how I did it' post to the -dev list might be useful for folk.
<StevenK> lpsetup needs a -y :-(
<wgrant> lifeless: Once we've actually done it, yeah :)
<lifeless> the new shiny stuff hasn't been socialised nearly enough.
<wgrant> lifeless: Unrelatedly, I'd like to basically abolish the explicit slave/master store selectors from normal code.
<wgrant> Do you see any reason to keep them?
<wgrant> A few places explicitly use a slave store for searching, but that seems pretty pointless.
<lifeless> I haven't done an audit but the idea sounds fine to me
<lifeless> I agree that forcing a slave for searching is unnecessary
<wgrant> lifeless: Thanks.
<wgrant> If basically everything uses the policy's default store, then we can avoid terrible fallback hacks in the DB policies.
<wgrant> Plus code gets shorter
<wgrant> eg. a lot of scripts use IMasterStore everywhere
<wgrant> When scripts always default to the master anyway
<wgrant> wallyworld_: Looks like you may have broken the build
<wgrant> But that's impressively few test failures considering what you've done.
<wallyworld_> iy passed ec2
<wallyworld_> it
<wallyworld_> hmmm
<wallyworld_> looking
<wgrant> http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/83/steps/shell_9/logs/summary
<wallyworld_> i can click that myself you know
<wgrant> Yeah, but it's like a buildbot bookmark and 3 clicks away
<wallyworld_> i already had bb open to monitor the build
<wgrant> Ahh
<StevenK> wgrant: Do you know about the code.simplified_branches_menu.enabled FF ?
<wgrant> StevenK: It's been on for everyone for more than a year, I'm pretty sure
<wgrant> It can probably be removed.
<StevenK> Yeah
<StevenK> Lets do that
<wgrant> Even better if you remove the view code but forget the model stuff, so I can delete it myself later :P
<StevenK> Hah
<StevenK> I don't think this includes any model code
<StevenK> wgrant: PersonBranchesMenu makes me sad. It has these link properties and they're so generically named that I can't tell if they're used
<StevenK> wgrant: So if a method returns a link a template still has to reference it? I can't any results for '/owned'
<wgrant> StevenK: Most of the time
<wgrant> StevenK: But sometimes entire menus are probably rendered
<StevenK> This is PersonBranchesMenu and I want to destroy more code
<wgrant> Delete them and see what breaks, perhaps
<StevenK>  6 files changed, 66 insertions(+), 180 deletions(-)
<StevenK> I think I've deleted enough to make up for I added
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/skip-private-dev-focus/+merge/127622
<LPCIBot> Project devel build #1105: STILL FAILING in 1 hr 28 min: https://lpci.wedontsleep.org/job/devel/1105/
<StevenK> That's wallyworld_'s fault too
<wallyworld_> StevenK: this is where I think some pragmatism is needed with the LOC thing - your branch combines 2 totally unrelated changes just for the sake of artificially reducing LOC when really 2 branches would have been better, and now it makes it harder to review because you have to constantly ask what unit of work the various parts of the diff are for
<wgrant> The LoC policy doesn't say anything about single branches being LoC-negative.
<wallyworld_> i know
<wgrant> That would be perfectly acceptable as two separate branches
<lifeless> it explicitly avoids that in fact
<wallyworld_> which i why i questioned it
<wgrant> So there's no pragmatism required :)
<wallyworld_> well, that's my bias coming through
<StevenK> I can delete the MP and blow them apart
<wallyworld_> if a better solution requires slightly more zLOC, then so be it
<wallyworld_> LOC should be a secondard consideration, always
<wallyworld_> secondary
<wgrant> Sure
<wgrant> But in most cases there's an easy win
<wallyworld_> StevenK: that would be great if you could
 * StevenK stops glaring daggers at LibrarianFormatter
<lifeless> StevenK: ok, so what I'm doing to reproduce is this:
<lifeless> cd /home/jenkins/launchpad/lp-branches/workspace/devel/
<lifeless> jenkins@ip-10-218-67-147:~/launchpad/lp-branches/workspace/devel$ sudo /usr/local/bin/lp-setup-lxc-test lptests /home/jenkins/.ssh/id_rsa --load-list /home/jenkins/runs/tmpzqc6_R > ~/runs/stream 2>~/runs/stream_err
<lifeless> using the captured load-list
<lifeless> which contained one of the tests that got spewed to https://lpci.wedontsleep.org/job/devel/1105/console
<lifeless> StevenK: when it completes, if the stream is corrupt, we'll have reproduction.
<StevenK> Nice
<lifeless> StevenK: if it isn't, we'll need to keep trying harder.
<lifeless> Once we have reproduction, then I'll start on trimming down the size neede dto trigger the failure: first delete all the tests that ran *after* the corruption (allowing 2- or 3 to stay for buffer flushing just-in-case)
<lifeless> then delete all the tests before it, leaving enough to trigger the fail
<lifeless> rinse and repeat - binary search needed for some of these 'delete all the' bits.
<wgrant> wallyworld_: Why don't we just restrict +bugs and searchTasks to users with launchpad.View?
<lifeless> because every alteration will permute things
<wgrant> Or zope.Public for public artifacts.
<wgrant> wallyworld_: Unauthorized is the correct exception to raise; it triggers the 403 page like any other forbidden page
<wgrant> We don't need a more user-friendly message than that.
<wallyworld_> wgrant: no, what's there in the mp is restoring existing behaviour which i broke with the ff removal
<wgrant> Or if we do, then we need it generally.
<wgrant> wallyworld_: Right, but is it valuable existing behaviour?
<wallyworld_> yes
<wgrant> Oh, is this the "this information is not shared with you" thing?
<wgrant> That we added recently?
<wgrant> for LimitedView?
<wallyworld_> it was specifically asked for during disclosure development
<wgrant> Right, that makes more sense
<wgrant> Not actual sense
<wallyworld_> yes, the "this info is no shared " thing
<wgrant> But more :)
 * wallyworld_ doesn't want to speculate on the usefulness of the feature, just implementing what's been asked for
<wgrant> r=me
<wgrant> Sure
<wallyworld_> thank you
<wgrant> I just thought it was probably some terrible ancient thing
<wallyworld_> fair enough :-)
<wgrant> And terrible ancient things that I don't know about can usually be deleted without anyone noticing.
<wallyworld_> the recent bb breakage was me attempting to fix the failing test
<wallyworld_> so i neutered the test to get bb going
<wgrant> Right, I saw you'd dropped the +bugs team check thingy
<wallyworld_> and then provided the proper fix
<lifeless> StevenK: general design rule testr honours is to expose data / state so folk can go digging :)
<StevenK> But not quite enough data in this case
<lifeless> StevenK: what would you like added ?
<lifeless> /usr/bin/python 27917         jenkins    0r     FIFO                0,8      0t0   18555269 pipe
<lifeless> seems saner
<lifeless> /usr/bin/python 27917         jenkins    1w     FIFO                0,8      0t0   18555270 pipe
<lifeless> /usr/bin/python 27917         jenkins    2w     FIFO                0,8      0t0   18555271 pipe
<lifeless> so the xvfb-run fix is intact
<wallyworld_> wgrant: installing the cert works great for chrome but now ff constantly complains about (Error code: ssl_error_bad_cert_domain) even if i add an exception. any idea?
<StevenK> lifeless: Merely pointing out that testr does not provide quite enough state or data to be able to tell what's leaking
<lifeless> StevenK: but it does, AFAICT
<lifeless> StevenK: I mean, I'm doing this because I've lots of experience and I have the time atm
<lifeless> StevenK: but there was nothing hidden that would make it hard to reproduce
<lifeless> no need for a debugger, for instance.
<lifeless> or extra print statements
<lifeless> StevenK: I mean, clearly its going wrong :P
<StevenK> wallyworld_: https://code.launchpad.net/~stevenk/launchpad/skip-private-dev-focus/+merge/127629 should be more to your liking
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-simplified-branch-ff/+merge/127630
<wallyworld_> StevenK: bit of a nitpick on line 52 - s/an/a
<StevenK> Ah yes. It made sense before I added 'private' in :-)
<wallyworld_> StevenK: so thst i understand, will self.branch be None if the user cannot see the devfocus branch?
<StevenK> wallyworld_: Yes, I won't paste the code block, but the development_focus_branch property will only return the branch if the user has permission to see it. In the cases there is no devfocus or it's invisible, it returns None.
<wallyworld_> StevenK: right, and this is the self.branch attribute used in external_visible?
<StevenK> wallyworld_: Right, self.branch backs onto self.development_focus_branch
<wallyworld_> ok, thanks
<wallyworld_> r=me
<StevenK> wallyworld_: I'll push up that nitpick before landing too
<wallyworld_> StevenK: and now you get to close a second bug :-)
<wallyworld_> since there are 2 x mp
<wgrant> wallyworld_: Which domain are you going to?
<wallyworld_> wgrant: bugs.launchpad.dev
<StevenK> wallyworld_: Which second bug?
<wallyworld_> now works without complain in chrome
<StevenK> Very tempted to just Won't Fix bug 393407
<wallyworld_> StevenK: the one you raise for removing the feature flag
<_mup_> Bug #393407: PPA doesn't allow signed contributed builds <feature> <lp-soyuz> <ppa> <soyuz-upload> <Launchpad itself:Triaged> < https://launchpad.net/bugs/393407 >
<StevenK> wallyworld_: Bleh
<wgrant> wallyworld_: Does the Firefox cert error show the right cert?
<wgrant> It's possible you're fighting with SNI and losing, or something like that
<wallyworld_> hmm. not sure, let me look
<wallyworld_> wgrant: it's labelled as "Invalid Certificate" when i use ff to get the cert from the error screen. not sure how to know what the right one is
<wallyworld_> wgrant: i actually ran the entire rf-setup-certs.sh script once, but it complained about an invalid file or dir so i just ran the last 2 lines and that seemed to work
<wallyworld_> which it did for chrome
<StevenK> wgrant: I wonder if bug 583392 is fixed now
<_mup_> Bug #583392: IntegrityError raised setting a branch for a project series. <branches> <easy> <lp-code> <oops> <series> <Launchpad itself:Triaged> < https://launchpad.net/bugs/583392 >
<wgrant> wallyworld_: Try adding the cert to Firefox's certstore manually, I guess.
<wgrant> StevenK: It appears to be, indeed.
<lifeless> ok, it reproduced on that one stream
<lifeless> now to divide and conquer
<lifeless> cp tmpzqc6_R{,.orig}
<lifeless> and delete all but the 50 or so tests before the glitch
<lifeless> stream_err has only this:
<lifeless>  less stream_err
<lifeless> Warning: Permanently added '10.0.3.8' (RSA) to the list of known hosts.
<lifeless> kill: 186: No such process
<lifeless> Stopping lxc
<lifeless> and a blank line
<wgrant> Hm
<wgrant> Did someone disable qastaging updates?
<lifeless> it can mess things up, but there is so little of it, and its all (AFAICT) outside of the xvfb-run anyhow, which is why xvfb-run isn't directly griefing us
<lifeless> s/directly/frequently/
<lifeless> StevenK: http://paste.ubuntu.com/1257313/ list of tests
<StevenK> lifeless: Nice
<lifeless> StevenK: and for tracking progress https://bugs.launchpad.net/launchpad/+bug/1060616
<_mup_> Bug #1060616: subunit stream corruption with jenkins test runs <Launchpad itself:Triaged> < https://launchpad.net/bugs/1060616 >
<StevenK> lifeless: So, thanks for filing that, but not another critical :-(
<lifeless> StevenK: https://bugs.launchpad.net/launchpad/+bug/1060616/comments/1
<_mup_> Bug #1060616: subunit stream corruption with jenkins test runs <Launchpad itself:Triaged> < https://launchpad.net/bugs/1060616 >
<wallyworld_> wgrant: got it working. had to convert the x509 cert for chrome to a pkcs12 one for firefox. i have no idea about all this stuff
<lifeless> StevenK: now that we have an unadulterated stream, we can see pretty clearly whats fucked up
<StevenK> Indeed
<lifeless> I'll file a testr bug about making it easier to get said stream.
<wgrant> wallyworld_: Hm, Firefox should be able to import plain x509 too
<wallyworld_> wgrant: it didn't seem to give me that option, only pkcs12
<wallyworld_> i tried to import the x509 and it complained
<wallyworld_> at least it all works now, and the other ff lp.dev issue is gone now too
<lifeless> StevenK: so this is critical, because if it breaks on buildbot it will equally badly and mysteriously.
<wgrant> wallyworld_: Ah, were you trying to import into the Personal section?
 * wallyworld_ shrugs
<wgrant> that needs a private key too, so it requires PKCS#12
<wallyworld_> yes, looks like i was
<wgrant> Servers/Authorities should work with x509
<wallyworld_> ok, will try that next time it breaks
<StevenK> lifeless: Indeed
<wgrant> "USN-1551-1 fixed vulnerabilities in Thunderbird. The new package caused a
<wgrant> regression in the message editor and certain performance regressions as
<wgrant> So it wasn't just me!
<wgrant> well. This update fixes the problems."
<lifeless> StevenK: its worse than I thought:
<lifeless> see comment 2
<StevenK> Wheee
<wgrant> That's normal
<wgrant> If the subprocess dies, the rest of the layer goes with it.
<wgrant> And the parent often doesn't notice.
<StevenK> There's a bug for that
<StevenK> Subprocess death unheeded by parent or so
<wgrant> Right
<lifeless> well
<lifeless> so the question is
<lifeless> is python failing to process a finally:
<lifeless> or is something else fubared
<wgrant> Which test is it dying on?
<wgrant> Probably something that uses webkit
<lifeless> wgrant: lib/lp/code/javascript/tests/test_productseries-setbranch.html and lp/testing/tests/test_yuixhr_fixture_facet
<wgrant> Ah yes
<wgrant> So it is
<wgrant> So it'll be segfaulting.
<wgrant> You can reproduce that easily
<lifeless> so, subunit layer, it may make sense to have a 'close off the stream no matter what' escape clause
<wgrant> It usually happens when run outside xvfb-run, though
<lifeless> which the parent could use
<lifeless> however
<lifeless> need to determine first how the parent-child stuff works
<lifeless> it may be that the parent is just being stupid and not reporting a close when it knows already
<lifeless> StevenK: so, you know now whats wrong - webkit fuckage
<lifeless> StevenK: and I have the data on how the system was failing, so can reproduce externally.
<StevenK> Then why doesn't buildbot fall over into a screaming heap?
<lifeless> its webkit isn't segfaulting
<lifeless> (duh)
<StevenK> But why not? :-(
<wgrant> Run it manually
<lifeless> dunno
<wgrant> You might get a GTK error
<StevenK> And I'm curious what is causing it to SEGV
<wgrant> It normally segfaults because there's no X
<lifeless> nothing on stderr
<wgrant> But the xvfb-run should sort that out
<StevenK> Right
<wgrant> gdb :)
<lifeless> indeed
<lifeless> start an ephemeral container
<lifeless> ssh into it
<lifeless> run one xvfb-run bash
<lifeless> run gdb python --args -- bin/test lp/testing/tests/test_yuixhr_fixture_facet
<lifeless> (or thereabouts)
<wgrant> Yep
<StevenK> lifeless: You're still on the executor, are you going to try that?
<wgrant> The stacktraces tend to be reasonably self-explanatory, fortunately.
<lifeless> StevenK: no
<StevenK> Aww
<lifeless> StevenK: a) cynthia time, b) I'm pursuing the test reporting side of it, to make future runs less likely to fail in this mysterious manner
<lifeless> StevenK: daylight saving kicked in here last weekend, we're 3 hours out atm
<lifeless> StevenK: https://bugs.launchpad.net/subunit/+bug/1060628
<_mup_> Bug #1060628: cannot 'fix' a subunit stream that may be corrupted <subunit:Triaged> < https://launchpad.net/bugs/1060628 >
<StevenK> lxc-start -n lptests will start the ephemeral container?
<lifeless> StevenK: no
<wgrant> lxc-start-ephemeral
<lifeless> StevenK: lxc-start-ephemeral -d -n lptests
<wgrant> lxc-start -n lptests will start the real container
<lifeless> StevenK: so you can ssh in and not have the awful console emulator
<StevenK> root@ip-10-218-67-147:~# lxc-start-ephemeral -d -n lptests
<StevenK> getopt: invalid option -- 'n'
<StevenK> And the usage message isn't very helpful either
<lifeless> oh blah,
<wgrant> -o maybe?
<wgrant> I can't remember
<wgrant> May just be a positional arg
<StevenK> -o orig is in the usage
<lifeless> yes
<lifeless> -o lptests
<lifeless> I was remembering base container startup
<StevenK> A consistent UI is a bad UI
<lifeless> this isn't inconsistent
<wgrant> Back in my day it only had positional args, and we liked it.
<lifeless> because you wouldn't be naming it
<lifeless> it got 'fixed'
<StevenK> Bleh, and I run right into the 'jenkins has a password?' thing :-(
<lifeless> muhhaha
 * StevenK forcibly edits the lxc's sudoers
<StevenK> jenkins@lptests-temp-k0hHY4M:~$ sudo apt-get install gdb
<StevenK> Reading package lists... Done
<wgrant> stub: Do you recall if there was a reason for translations-export-to-branch to do its search on the slave DB?
<wgrant> It then casts the branch up to a master object to write to it
<wgrant> Which seems odd.
<StevenK> jenkins@lptests-temp-k0hHY4M:~$ xvfb-run bash
<StevenK> xvfb-run: error: Xvfb failed to start
 * StevenK grumbles
<wgrant> It does like its helpful error messages
<wgrant> Beware, 'tis a hideous shell script
<StevenK> Yes, and I'm trying to avoid bash -x
<stub> wgrant: Sounds like one I would have nagged about because of performance and/or long running transactions
 * StevenK stabs xvfb-run
<stub> wgrant: IIRC it had long running transactions, and when fixing that it was easy to switch it to using the slave for most of the work too.
<lifeless> wgrant: that was avoiding lock driven bloat on the master
<wgrant> The thing is the DB work is really really trivial
<wgrant> Hmm
<lifeless> wgrant: but without slony the bloat impact happens no matter what slave you are reading from
<lifeless> so without slony there is no benefit to that jumping around
<wgrant> It's the only remaining case of things that need to write querying from the slave store
<stub> wgrant: Yeah, but if you look sideways at a Storm object at the wrong time it gets refreshed from the DB and suddenly you have a new, open transaction sitting idle.
<stub> jtv did the fixup IIRC, I might be mistaken for the rationale.
<wgrant> It's used the slave store since the script first landed in the tree
<stub> But the basic principal of 'thou shalt use a slave instead of the master whenever possible' still applies.
<stub> Excellent. We should do more of that.
<wgrant> Should we?
<wgrant> The DB load from the two things that do that is miniscule.
<wgrant> And it's a fair bit of added complexity
<stub> Yes, the master isn't horizontally scalable.
<stub> Why is it complex?
<stub> I suspect it is complex because it is bending over backwards to ensure a transaction doesn't get held open while doing possibly lengthy operations, like anything to do with importing or exporting pofiles or anything to do with bzr.
<wgrant> Well, most things are very well served by just using their policy's default store
<wgrant> We have a lot of code that uses IMasterStore/ISlaveStore/IMasterObject unnecessarily.
<stub> Yeah, but something like exporting translations you can say 'this always uses the slave store' because we never care that the information is up to date.
<stub> The scripts policy could encode this, but other callers to the code don't get that policy.
<stub> Do we care now if things are a little out of date?
<wgrant> Mmm
<wgrant> I'm trying to think of a cleaner way to do the slave fallback. Currently you get a slave object that's actually connected to the master, which is a bit ugly.
<wgrant> Because we need master->slave fallback to some extent.
<wgrant> Though I guess if I get a fake-slave object and keep it around over a connection reset, it could end up reattached to a real slave
<wgrant> Our Storm object lifetimes are sometimes a little odd
<wgrant> stub: Does our postgres config log statement timeouts?
<stub> Yes
<wgrant> https://oops.canonical.com/oops/?oopsid=OOPS-b01fe5e323fdcb85e9bfda27beca4466
<wgrant> Is it easy enough to find out the second that the final statement there was terminated?
<wgrant> Since it took 24 seconds
<wgrant> I suspect the delay is on the Python side
<wgrant> But I'd like to know for sure
<wgrant> Since this goes back to the random delays we see sometimes that can't all be explained by GIL contention
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/assertion-review-token/+merge/127639
<wgrant> They usually exhibit themselves as very long DB queries
<stub> From my POV, there is nothing wrong with ISlaveStore returning a Store connected to the master db. However, the Zope adapter mechanics require that IFoo(bar) return an object that provides the IFoo interface.
<wgrant> Oh, it actually checks?
<stub> I've thought of making all objects that provide IMasterStore also provide ISlaveStore
<stub> I think it does in some cases? All cases? I forget the details
<wgrant> I guess having the slave store connected to the master DB isn't too bad, but it might be useful to log the current connection string somewhere.
<stub> We don't need to use the adapter spelling. It was the nicest spelling that met our needs at the time.
<wgrant> Currently in OOPSes we log whether it was master or slave, which can be helpful for diagnosing slowness.
<wallyworld_> StevenK: i'll pass since i'm not fully across the subtlties of the oauth stuff
<wgrant> I'll look in a sec
<StevenK> wallyworld_: Pft. It's subtle as a housebrick to the face.
<stub> We need to, at a minimum, switch from using dbname to dbname+hostname. Full connection string might be too noisy in some contexts.
<wallyworld_> StevenK: not really, there's subtle changes to the error handling and flow
<wallyworld_> and i don't know what might break
<stub> Our statement sanitation code isn't hiding the integer id in that oops
<wgrant> stub: We might just want to log timeline events on reconnection, and at connection start.
<stub> Doesn't want to handle function calls
<wgrant> The request started at 21:26:56
<wgrant> The fatal query should have started about 2 seconds after that
<wgrant> And terminated between 5 and 23 seconds after that.
<wgrant> I'm interested whether it was 5 or 23.
<stub> <lpnet55@launchpad_prod:25567> 2012-10-01 21:27:22 UTC ERROR:  canceling stateme
<stub> nt due to statement timeout
<stub> <lpnet55@launchpad_prod:25567> 2012-10-01 21:27:22 UTC STATEMENT:  UPDATE Bug SET heat=calculate_bug_heat(1047345), heat_last_updated=CURRENT_TIMESTAMP AT TIME ZONE 'UTC' WHERE Bug.id = 1047345
<wgrant> Hm
<stub> I'm seeing some odd replication stuff around that time, may be unrelated
<wgrant> So did postgres sit on it for 20s too long, or did we delay sending it for ages.
<wgrant> I wonder
<stub> We can't tell now
<wgrant> Indeed.
<wgrant> StevenK: You can't really just convert all AssertionErrors into notifications...
<wgrant> StevenK: And you can't just str() or unicode() things.
<wgrant> stub: Thanks for poking.
<wgrant> Could be Python, pgbouncer, or postgres :(
<StevenK> wgrant: So I can convert the 5 AssertionError's in the model into OAuthValidationError or something ?
<wgrant> StevenK: Something like that. There may already be a relevant exception,.
<StevenK> Not that I can see.
<lifeless> wgrant: could also be TCP
<StevenK> wgrant: http://pastebin.ubuntu.com/1257370/
<StevenK> lifeless: There. Fixed. ^
<bigjools> god I hate dpkg
<bigjools> once you get a bug in a postinst you're screwed
<wgrant> lifeless: Shhh, I only blame TCP when I want elmo to investigate
<wgrant> It's not at that stage yet :)
<wgrant> bigjools: What sort of bug?
<bigjools> it hangs
<wgrant> Oh good
<StevenK> bigjools: That's when you edit /var/lib/dpkg/info/foo.postinst :-P
<bigjools> package removal fails, reconfigure fails...everything bloody fails!
<bigjools> StevenK: yes, that's what I do :/
<bigjools> ctrl-c not working either
<StevenK> Ctrl-\ ?
<StevenK> ctrl-c is SIGINT, ctrl-\ is SIGQUIT
<StevenK> bigjools: Failing that, ctrl-z ; bg ; kill -9 ?
<bigjools> oh GREAT.  now the kernel bug that disconnects usb devices hit me.  no mouse
 * bigjools ragequit
<wgrant> StevenK: How formy is it?
<StevenK> wgrant: EPARSE
<wgrant> StevenK: Redirecting to the success page and adding a notification to say that it failed is somewhat unpleasant.
<wgrant> Is it close enough to a form that you can display an error message on it?
<wgrant> Rather than a notification?
<StevenK> Not really
<StevenK> There are no fields
<StevenK> We show 6 buttons and a lot of text
<bigjools> StevenK: kill -9 after searching for all the processes that apt-get started yeah :/  it's hanging on a very innocuous prerm ...
<StevenK> wgrant: I can redirect to '/' or something if you wish?
<StevenK> '~' might work too
<wgrant> StevenK: I'm not sure.
<wgrant> Redirecting to the callback is definitely wrong
<wgrant> Redirecting to anything and displaying a pretty blue success notification for failure is also a little wrong.
<StevenK> So we should continue to OOPS? :-)
<wgrant> No, ideally we'd display an error message.
<wgrant> Not just redirect with successful failure
<StevenK> wgrant: So what about response.addErrorNotification ?
<wgrant> StevenK: That might be an acceptable compromise.
<wgrant> It's probably easiest.
<StevenK>  wgrant http://pastebin.ubuntu.com/1257382/
<wgrant> StevenK: Sounds more reasonable. I don't think we use the term "request token" anywhere else in the UI, though. I'd try to find a better term by looking at the existing template.
<StevenK> What about 'token' ?
<StevenK> wgrant: request token is already shown for the token.permission == OAuthPermission.UNAUTHORIZED case?
<wgrant> StevenK: Isn't that the response to +access-token, which is given to the client, not the user?
<StevenK> I have no idea
<StevenK> Hm, I can think OAuthBase can die a quick death
<StevenK> Replace OAuthNonce.getStore with IMasterStore
<wgrant> I might mean IStore
<wgrant> s/I/You/
<wgrant> IMasterStore should be used only sparingly.
<StevenK> It uses MASTER_FLAVOR
<StevenK> So ...
<wgrant> So do a lot of things
<wgrant> A lot of things are wrong :(
<StevenK> And it has a comment saying "We want all OAuth classes to be retrieved from the master flavour.
<wgrant> Webservice requests always end up on the master today.
<wgrant> Mmm
<wgrant> I guess leave it. OAuthNonce will hopefully die soon anyway.
<StevenK> Ah
<StevenK> wgrant: So 'request token' is good enough, or you object?
<wgrant> The template doesn't mention tokens at all.
<wgrant> But I can't think of much better
<StevenK> Neither
<StevenK> And it's for an erorr that shows up what, once a week?
<wgrant> Anyway, you also might want to inhibit the callback
<StevenK> wgrant: Good point. That change is pushed.
<StevenK> wgrant: Can haz approval, or you still object? :-)
<wgrant> StevenK: Done
<StevenK> Blink. How did I miss that. I even re-indented that block. :-(
<adeuring> good morning
<wgrant> Hm
<wgrant> I think there's an infinite loop in the dominator
<wgrant> Oh, or it's a transaction conflict...
<wgrant> Heh, yes
<wgrant> Evil massive Soyuz doctests
<wgrant> StevenK: You have test failures
<StevenK> My testfix fixed all of them bar one, too. Sigh.
<wgrant>  10 files changed, 6 insertions(+), 1053 deletions(-)
<StevenK> Nice
<wgrant> aka. burning soyuz for fun and profit
<wgrant> As a precursor to making us not suck at transactions.
<czajkowski> wgrant: StevenK wallyworld_ can one of you please look at https://answers.launchpad.net/launchpad/+question/209658
<wallyworld_> czajkowski: i'm somewhat clueless when it comes to such email issues sadly :-(
<adeuring> StevenK: could you have another look at my MP?
<czajkowski> wallyworld_: it's now baffling
<czajkowski> I got him to add his key to LP
<czajkowski> but still having issues
<wallyworld_> czajkowski: curtis i think has knowledge in this area
<wallyworld_> he should be online soon
<StevenK> adeuring: r=me
<adeuring> StevenK: thanks
<wgrant> StevenK: Hm, which LoC counter did you use to determine you had more credit than me?
<wgrant> StevenK: I'm well ahead of everyone on 1-month, 6-month, 12-month and 18-month intervals.
<wgrant> By 'bzr high-scores'
<mgz> now now wgrant, no need to boast :)
<jelmer> wgrant: where does 'bzr high-scores' live?
<wgrant> jml's bzr-damage, probably
<wgrant> Yeah
<cjwatson> high-scores since r14780 (loc-contributions' default) is 1) William, 2) Curtis, 3) me, 4) Aaron, 5) Steve
<wgrant> Hm, loc-contributions seems to hate me
<cjwatson> Might be counting DB patches differently
<wgrant> Yeah
<wgrant> I merge db-stable a lot, so it might count them against me
<cjwatson> Yeah, quite possible
<wgrant> Whereas high-scores might just count each rev individually
<cjwatson> When I was writing loc-contributions I found both got things wrong in different ways
<wgrant> Yeah
<cjwatson> If high-scores does as you say, I expect it will miscount branches which were merged from devel before landing
<cjwatson> Which I reckoned was a commoner case
<wgrant> Hm, it may
<StevenK> wgrant: loc-contributions for me says -3500 or so, and like -1200 for you
<StevenK> Last I checked
<StevenK> I'm approaching cjwatson :-)
<cjwatson> Curtis still has me beat
<cjwatson> Though if I can get some ops time next week to deploy another PCJ runner or similar, I can flush a big negative queue then ...
<wgrant> Indeed.
<StevenK> cjwatson: There's a bunch of criticals you get to close too, which is nice.
<StevenK> wgrant: Don't you get to close bug 624078 now?
<_mup_> Bug #624078: soyuz-set-of-uploads.txt has unnecessary tests that are tricky to remove <lp-soyuz> <tech-debt> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/624078 >
<StevenK> In truth, soyuz-set-of-uploads.txt should just die in a large fire
<czajkowski> cjwatson: will you be doing more LP work next cycle ?
<cjwatson> Not as much
<cjwatson> Probably bits here and there
<czajkowski> will be running 2 more LP clinics at uds so hopefully we can get more pople to scratch their itches
<StevenK> cjwatson: You're still going to destroy delayed copies?
<wgrant> StevenK: Indeed, forgot to reference it in the commit message.
<cjwatson> StevenK: Yes
<cjwatson> Blocked on removing the synchronous path from Archive:+copy-packages, which is blocked on a bit more PCJ running capacity so that they're more responsive
<StevenK> I've been looking forward to delayed copies dying a slow and horrible death.
<cjwatson> But nothing hard at this point
<StevenK> Removing the synchronous path closes at least two criticals. Possibly a third.
<StevenK> Death of delayed copies is another one or two.
<cjwatson> And nearly 1500 LOC.
<wgrant> Aha
<xnox> czajkowski: hmm.... I have a few new small bugs I'd like to tackle. But since you need LOC to land those, maybe you can do a clinic on gaining LOC credit?
<wgrant> loc-contributions indeed implicates me in db-stable merges
<wgrant> And, even better, it actually double-counts me for about 2000 lines where I did both the merges
<wgrant> Excluding those gets me to -4500
<jelmer> bleh, bzr high-scores doesn't like ghosts
<wgrant> Sadly cjwatson still beats me
<cjwatson> xnox: Mine has been gained from (a) rewriting scripts as API facilities and removing the scripts (hard to duplicate) (b) removing completely useless unused stuff (c) rewriting doctests as unit tests
<czajkowski> xnox: people will land them if asked and they approve of the fixes
<czajkowski> but yes it could be brought up in the clinics
<cjwatson> (b) just requires finding them; (c) isn't a reliable source but there is a vast mine of crappy doctests to refactor
<xnox> =)))) lolz
<StevenK> But don't be scared away by LOC. Nothing says the removal has to come first.
<xnox> StevenK: not scared. Just playing the game ;-)
<StevenK> And if your change provides a maintaince benefit, lifeless or flacoste may grant an LOC exception.
<xnox> nah, I'd rather have fun with LOC. I was trying to do that with ubiquity a little bit, as self discipline.
<StevenK> xnox: So please, don't just groan about LOC, *talk* to us about it.
<mgz> killing doctests is always good.
<StevenK> I enjoy killing doctests
<StevenK> Soyuz has too many of them
<StevenK> cjwatson, wgrant and myself have helped drop the number
<wgrant> Hm
<wgrant> We have 10 Fix Committed criticals
 * wgrant curses sprints
 * StevenK does QA for his oauth fix the easy way
<StevenK> (Leaving a browser window hanging for >2 hours)
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: ~300
<deryck> rick_h_, are you back today?
<czajkowski> sinzui: so what's your favourite kinda cake :)
<sinzui> Chocolate with coconut and pecan frosting.
 * sinzui might make that for lunch
<czajkowski> sinzui: nobody seems tobe able to work this out, so it falls I'm afraid to you. https://answers.launchpad.net/launchpad/+question/209658
<czajkowski> I sorted out webops folks yesterday with a curly wurly cake :)
<sinzui> czajkowski, Looks like a malformed signature per
<sinzui> https://answers.launchpad.net/launchpad/+question/147254
<sinzui> and
<sinzui> https://answers.launchpad.net/launchpad/+question/86235
<sinzui> czajkowski, I think steve complete borked his setup be removing his keys and possibly changing his client
<slank> czajkowski: yes, that was delicious. thank you!
<czajkowski> slank: :)
<sinzui> czajkowski, the common cause of this (also seen in signing the CoC) is that the part after
<sinzui> -----BEGIN PGP SIGNATURE-----
<sinzui> got wrapped
<sinzui> or lost characters.
<czajkowski> how does one lose characters
<czajkowski> :s
<sinzui> It could be a copy+paste error of his client wrapped the message before sending it
<sinzui> bad copy or bad correction of a wrap will loose characters
<czajkowski> nods
<czajkowski> ok
<czajkowski> sinzui: 2 more things, 1) commericla renewals I've had 2 mails sent to the RT about them, I would expect they would get an email notification to renew and a link in that where to go and pay ? No?
<sinzui> czajkowski, I think we can tell steve that email clients are often the problem. I use shitehawk because it does signing properly with PGP/MIME
<czajkowski> 2nd thing will send to pm as it ha sa link
<czajkowski> sinzui: fair enough, not seen this issuebefore
<czajkowski> thank you though
<czajkowski> I shall send the cake :p
<sinzui> czajkowski, the emails do not contain the link because the link it not to launchpad.
<czajkowski> ah so how do they find where to go to renew ?
<sinzui> the email correctly state that the user must visit the project page and follow the instruction to renew
<abentley> sinzui: product_licenses_modified sends mail to the logged-in user.  Is this correct?  Wouldn't it make more sense to send it to the product owner or something?  I noticed because I called createProduct with UnauthenticatedPrincipal logged in.
<sinzui> which is purchase from a non-canonical site, then wait 30 to 300 minutes, then apply the voucher that took forever to get to Lp
<sinzui> abentley, it sends it the the maintainer, not the logged in user...the emails are created by a job once a day
<abentley> sinzui: I have a traceback that indicates otherwise.
<sinzui> abentley, the case you are reporting is a different email, and yes, that could use the maintainer
<sinzui> czajkowski, we have seen a lot of projects fail to renew because the company set a person as the maintainer, then they left the company.
<sinzui> We will state that we think users should never maintain projects owned by commercial organisations
<abentley> sinzui: http://pastebin.ubuntu.com/1258172/
<sinzui> abentley, yes, I understand. You can change the method to use the product.owner, but you may still need to update the that specific email to know about the registrant because the owner might know know he owns the project
<abentley> sinzui: s/know know/not know ?
<sinzui> abentley, you are correct
<abentley> sinzui: Okay, cool.
<sinzui> abentley, I have a bug I was thinking of fixing on friday to make czajkowski and my days easier
 * czajkowski hugs sinzui 
<sinzui> I think I need to edit that subscriber method
<sinzui> So I will find update the code to use the maintainer.
<abentley> sinzui: Okay, cool.  I'll work around it in my test.
<abentley> deryck: I'm the OCR (which I just noticed is an anagram for orc).  Could you please review https://code.launchpad.net/~abentley/launchpad/limit-product-info-types/+merge/127814
<deryck> heh
<deryck> abentley, sure.
<abentley> deryck: Thanks.
<deryck> np!
<abentley> deryck: For my next task, I was thinking of "Private project creation should enable appropriate apps".
<deryck> abentley, sounds great, thanks!
<abentley> deryck: There's currently a policy that if the license is Other/Proprietary, you get PROPRIETARY bugs/branches.  I'd like to change it so bug/branch/blueprint policy follows product.information_type.
<deryck> abentley, that sounds right.
<abentley> deryck: And I guess the other part is disabling Answers?
<deryck> abentley, indeed.  I had forgot about that.
<abentley> deryck: Cool.
<deryck> abentley, r=me for the MP.
<abentley> deryck: thanks.
<deryck> abentley, I just found a bug in project registration.  the js hides the license choice for you, but only selects proprietary/other and leaves the description blank, which raises form validation errors.
<abentley> deryck: I don't follow.
<abentley> deryck: I am seeing a "There is 1 error." message with no obvious error, though.
<deryck> abentley, when you select Proprietary information type, the js code hides the license stuff and automatically sets the license to "other/propretary"
<deryck> abentley, yeah, I believe that's because the form needs the extra "description" field for "other/proprietary" filled in, but it's now hidden.
<deryck> abentley, toggle it back to public, and select other, and you can see the option to fill in that field, which I did, toggled back to proprietary and the form will submit.
<deryck> abentley, however, I don't think the save step actually sets the information_type.
<abentley> deryck: Is this on qastaging?  It looks like information_type isn't selectable there.
<deryck> abentley, I'm doing it locally.  with the appropriate flag enabled.
<deryck> I really thought info type was saved.  could of sworn I checked that already.
<abentley> deryck: I don't see a description area for other/proprietary on qastaging.
<deryck> hmmm
<deryck> abentley, weird, I do.
<deryck> abentley, if I click "other/proprietary" then the text area expands open.
<abentley> Yeah, I don't get a text area, just the three checkboxes.
<deryck> weird.
<deryck> I wonder what's different about you.
<deryck> abentley, FWIW, I'll fix project creation to save info type now.
 * deryck branches
<abentley> deryck: I get the same behaviour on chromium and firefox.
<deryck> abentley, do you see any errors in either browser's console?
<abentley> deryck: Not in firefox...
<abentley> deryck: Not in chromium.
<abentley> deryck: (firefox actually has some CSS errors)
<deryck> hmm, weird.
<deryck> abentley, I'm stumped.  I have no idea why you wouldn't see it.  I've tried on 4 browsers across 2 laptops now and works fine for me.
<abentley> deryck: So, if I choose proprietary, then submit, the error page has a license description text box.
<abentley> deryck: But the initial page doesn't.
<deryck> oh weird, abentley.  It errors for me, with no text box.  But I get the text box normally when information_type is public.
<deryck> so I'm reversed of you.
<abentley> deryck: I didn't understand you properly.  I expected the text area to be shown when I clicked the "Other choices" expander, but it's actually shown when clicking "other/proprietary".
<deryck> abentley, ah, ok.  So we had a language fail then.  IRC communication failure, sorry.  What you're seeing is correct, and also what I'm seeing.
<deryck> abentley, the text area should not appear until you actually click the "other/proprietary" checkbox.
<abentley> deryck: My fail.  And yes, that's the behaviour locally and on qastaging for me.
<deryck> ok, cool.
<deryck> one less bug to sort out. :)
<abentley> deryck: So we've still got the problem that the license description is mandatory but not filled in.
<abentley> deryck: When auto-selected by information_type.
<deryck> abentley, exactly.  and in looking for where to fix the saving of information_type, I just found the spot to fix this.
<deryck> I tired to break rick_h_ of short, crazy variable names when we paired in Auburn, but I see he's still at is. :P
<deryck> at it, rather
<abentley> :-)
<rick_h_> deryck: I've tried! What did I short cut on? and sorry for the bug. I meant to chat with you about how to handle that with the wiring up of the field and see if we should put some standard text there.
<deryck> rick_h_, no worries. :)  and I can't decide if license_cont reads license_continue or license_container :)
<deryck> you could even have gone with license_div and stayed short but clear :)
<rick_h_> deryck: well it's not the license div, it's the container of the license div :)
<deryck> so license_div_div is still better than license_cont ;)
<rick_h_> but point noted. I'll work to unwire the cont/container abbrev
<abentley> deryck: I'm not sure if it makes sense to require a description of other/proprietary.  (In tests, I sometimes put "mine, all mine!")
<deryck> abentley, I was adding "Launchpad 30-day trial commerical license" to not have to change too much.
<deryck> it's in js, so nothing else changes.
<abentley> deryck: There's an argument that if Project.information_type is set to EMBARGOED, the branch/bug/spec policies should be PROPRIETARY, not EMBARGOED_OR_PROPRIETARY, just to be cautious.  What do you think?
<deryck> abentley, I need to think on that.  and about to host TL call.  let me come back to that question shortly.
<abentley> deryck: okay.
<deryck> abentley, coming back to your question, I think that makes sense.  Though I'd guess if people use embargoed for a project, they mean to go public later.  Just guessing.
<deryck> abentley, rather than the other-level-of-proprietary that it's used for in PES.
<abentley> deryck: If you're PES, you would set information_type=Embargoed to share milestones and productseries, but I don't know if they want to.
<abentley> deryck: I wonder if it'll be a problem that branches are associated with productseries?
<deryck> abentley, hmm, could be.
<deryck> abentley, so let's just keep it embargoed == embargoed_or_proprietary for now.  for consistency, and we can adjust as we get feedback.
<abentley> deryck: Okay.
<sinzui> jcsackett,  how goes your attachment branch?
<jcsackett> sinzui: i have discovered it is far more complicated than i thought. care to chat?
<sinzui> okay. I just wrote a test case for apport/blobs and attachments.
<sinzui> my concern was new bugs though
<lifeless> sinzui: hi
<lifeless> sinzui: did that log rotate fix get applied to LP?
<sinzui> No. we discussed patching Lp yesterday...then aggressively upstreaming the fix
<deryck> abentley, I have to go now for a birthday party for my little one tonight.  But I have a branch for review for those bugs, if you don't mind reviewing in my absence.
<deryck> abentley, if not, I can get it later tonight from someone when I'm back.
<abentley> deryck: sure.
<deryck> abentley, awsome, thanks! https://code.launchpad.net/~deryck/launchpad/actually-save-info-type-on-project-create/+merge/127863
<deryck> be back later on to finish up milestones/series.
* abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300
<sinzui> jcsackett, Bug #174794
<_mup_> Bug #174794: Uploading file with Internet Explorer uses full path name <ie> <lp-foundations> <oem-services> <Launchpad itself:Triaged> < https://launchpad.net/bugs/174794 >
<lifeless> StevenK: so lets get jenkins going today
<lifeless> owww ugh
<lifeless> the zope subprocess stuff is heinous
<lifeless> its not semantic at all
<lifeless> also, look at runner.py,
<lifeless> "
<lifeless>         # Display results in the order they would have been displayed, had the
<lifeless>         # work not been done in parallel.
<lifeless> "
 * lifeless isn't sure what to make of that
<sinzui> wallyworld_, the broken approve/decline js related to Bug #271697 and I think you get to close Bug #120357 and maybe this in now invalid Bug #629291
<_mup_> Bug #271697: Javascript for approving / declining nominations is confusing  <bug-nomination> <confusing-ui> <javascript> <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/271697 >
<_mup_> Bug #120357: Improve the accept/decline/nochange column on +nominations <lp-bugs> <Launchpad itself:Triaged> < https://launchpad.net/bugs/120357 >
<_mup_> Bug #629291: Missing column borders after expanding nominate <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/629291 >
<wallyworld_> ok
<lifeless> oh man, more bugs.
<lifeless> StevenK: wgrant: so I'm not going to file a bug for this, but the zope runner will corrupt a subunit stream if it gets keyboardinterrupt/memoryerror/systemexit
<lifeless> because it doesn't use finally: around outputing completing for layers.
<lifeless> you might want to address that if you have cause to touch it.
<lifeless> gary_poster: another FYI^
<wgrant> The zope runner generally just fails pretty terribly if it gets a keyboardinterrupt, yeah
<wgrant> The subprocess stuff is all a tad fragile
<lifeless> I'd like to get us totally away from it
<lifeless> just needs elbow grease
<StevenK> lifeless: Okay, creating a executor
<lifeless> StevenK: I have one hour to help you out
<StevenK> That's disappointing, because that's how long it will take to bring up the executor
<StevenK> I failed with xvfb yesterday
<lifeless> I have a 4pm flight
<lifeless> 2:30 at the airport
<lifeless> 1:30 leave here
<lifeless> its 12:36
<StevenK> lifeless: Ah, so you're coming to help in person? :-P
<wgrant> StevenK: Does ec2 hate your simplified branch menu branch?
<StevenK> It did, yes
<StevenK> FAILED (id=0, failures=31, skips=37)
<wgrant> Anything terrible?
<StevenK>         linkdata = method()
<lifeless> StevenK: if you want to come up to gosford with me, sure.
<StevenK>     TypeError: 'LinkData' object is not callable
<StevenK> wgrant: Most failures are caused by that
<wgrant> Ah
<StevenK> So I need to work out WTF
<StevenK> lifeless: Gosford? That's no fun
<lifeless> StevenK: http://miller.emu.id.au/pmiller/codecon/2012/index.html
<lifeless> StevenK: wgrant: anyhow, since zope.testrunner would need a massive overhaul to detect corrupt streams at source and fix, my hack-in workaround seems to be whats needed. I've queued that up to do, which will improve robustness in this sort of failure mode in future.
<StevenK> lifeless: So I still need to figure out why webkit SEGvs?
<wgrant> Right, the probable segfault is the big issue
<wgrant> You can often reproduce in bin/py pretty easily
<wgrant> Often just by importing it...
<StevenK> It would be nice to get xvfb working
<lifeless> StevenK: apt-get install didn't work ?
<StevenK> lifeless: Running xvfb on the executor inside LXC did not
<lifeless> StevenK: in what way didn't it work ?
<StevenK> It didn't start and didn't give any helpful hints as to why that was.
<StevenK> I thought these properties were supposed to return Link(...) :-(
<StevenK> Oh, no. It wants to call it
<StevenK> FAILED (id=1, failures=4 (-27), skips=2)
#launchpad-dev 2012-10-04
 * wgrant experiments with BranchRevision
<mwhudson> \o/
<wgrant> It may not come out of this very healthily
<wgrant> But then again, neither may mawson...
<lifeless> bai
<wgrant> Bye lifeless
<lifeless> wgrant: can you give StevenK a hand on the jenkins thing?
<lifeless> wgrant: I'd love to see it work :)
<wgrant> I was intending to, yes :)
<wgrant> StevenK: Shell pls
<wgrant> StevenK: So, xvfb-run still won't start?
<StevenK> wgrant: It didn't for me when doing it by hand
<wgrant> Let me see
<wgrant> StevenK: What are the creds for authing to the container?
<StevenK> wgrant: jenkins@
<wgrant> Oh, right
<StevenK> wgrant: I will usually sudo -u jenkins -i and go from there
<wgrant> Yep
<wgrant> Do you normally hack the container's sudoers?
<StevenK> I did yesterday
<StevenK> But not normally
<wgrant> Hm
<wgrant> So under xvfb-run it passes
<wgrant> In the root container
<wgrant> Spews locale errors, but passes
<StevenK> Sigh, there are tests that insist that the Registered branches link doesn't show up if there aren't any
<StevenK> Which makes it the only link to do so, so I crippled that
<wgrant> Right
<wgrant> Fix the tests :)
<wgrant> Or, possibly better still, delete them.
<wgrant> test: lib/lp/app/javascript/tests/test_multicheckboxwidget.html
<wgrant> Segmentation fault (core dumped)
<StevenK> \o/
<wgrant> Hm
<wgrant> But that same thing passes in the root container
 * wgrant blames aufs
<StevenK> Haha
<wgrant> aufs and I have history
<wgrant> Ah
<wgrant> Looking at syslog, there are some instances of what look like the symlink/hardlink restrictions
<StevenK> Ah ha
<StevenK> Mmmm, how do we turn that off?
<wgrant> Well
<wgrant> I just had another failure and didn't get the same message
<wgrant> And I don't actually know what the message was
<StevenK> Ah
<StevenK> [1236262.033939] /usr/bin/python[15312]: segfault at 14 ip 00000000f5496411 sp 00000000ff8a7160 error 4 in libgtk-x11-2.0.so.0.2000.1[f5310000+3cd000]
<wgrant> Ah right
<wgrant> dmesg is intact, indeed
<StevenK> wgrant: Right, so all tests pass. I'm going to toss this FF death branch at buildbot
<wgrant> So it is the hardlink restrictions
<wgrant> StevenK: Great
<wgrant> lp-setup was meant to disable that restriction, I thought
<wgrant> Or did we get a kernel fix that isn't in this ec2 image...
<wgrant> that would be pretty hilarious
<wgrant> Hm, the kernel's not even a month old
<wgrant> Yeah, the hardlink restrictions are still enabled.
 * wgrant disables
<StevenK> If you give me the runes, I can edit my setup script
<wgrant> Testing that it stops segfaulting now
<wgrant> It's pretty nice of xvfb-run to just carry on even though its setup failed
<wgrant> successful: lib/lp/app/javascript/tests/test_multicheckboxwidget.html
<wgrant> sudo sysctl -w kernel.yama.protected_nonaccess_hardlinks=0
<StevenK> Hah, right
<wgrant> is what you want
<wgrant> So, with that enabled, I'll shut down the containers and we can try a full --concurrency=4
<StevenK> Yeah, I'll need to add the executor to Jenkins, so tell me when you're done
<wgrant> StevenK: All torn down
<wgrant> Fire it up
<StevenK> wgrant: Done. #1106 kicked off
<wgrant> concurrency=4?
<wgrant> I don't want to wait all day, idaelly
<StevenK> Yeah, it's still 4
<wgrant> Great
<wgrant> Thought it might still be down to 1
<StevenK> It changed back days ago
<wgrant> Better safe than waiting for four hours :)
<wgrant> StevenK: Have you restored subunit2junitxml etc. after the debugging?
<StevenK> I have not
<wgrant> k
<wgrant> Let's see how it goes
<StevenK> It's fairly easy to restore and it only tells Jenkins which tests fail
<wgrant> Oh
<wgrant> I can just tail the stream directly now
<wgrant> :)
<StevenK> Haha
<StevenK> How long did it take you to realise that? :-)
<wgrant> ew
<wgrant> There are jars in ~/launchpad/lp-branches
<wgrant> die
<StevenK> Those are for jenkins, hands off
<wgrant> I know, but still ew
<StevenK> Haha
<StevenK> ~/launchpad/lp-branches is the base dir for jenkins, due to shared repo
<wgrant> Launchpad is a Java-free workplace :)
<wgrant> Ah
<StevenK> Because then it goes ahead with ~/launchpad/lp-branches/workspace/devel
<wgrant> This should take about 90 minutes, right?
<StevenK> Roughly
<wgrant> I guess it might be a bit longer, since the testrunners won't all drop dead when they start doing JS tests
<StevenK> Haha
<StevenK> wgrant: You said you understood the issue with bug 828133 ?
<_mup_> Bug #828133: Archive:+edit-dependencies OOPSes when Ubuntu dependency changed via API <critical-analysis> <oem-services> <oops> <ppa> <Launchpad itself:Triaged> < https://launchpad.net/bugs/828133 >
<wgrant> StevenK: Yeah
<wgrant> StevenK: IIRC the UI only exposes None/multiverse options for the primary archive dep
<wgrant> But you can set eg. main through the API
<wgrant> causing the UI to crash when it tries to render the current value
<StevenK> Haha
<wgrant> but reading the bug now to see if I remember anything else
<StevenK> Why would the UI do that? It's daft
<wgrant> Right, yeah, what I said
<wgrant> The UI gives "the source's component in the primary archive" and "all components" options
<wgrant> None and multiverse respectively.
<StevenK> Right, this should be pretty easy to test
<StevenK> ... unless Archive:+edit-dependencies is tested via a doctest ...
<wgrant> StevenK: The jenkins console thingy will show stream bits if the subunit breaks, right?
<wgrant> Because testr will let it through
<StevenK> I think testr will only show it right at the end
<StevenK> It doesn't seem to print out during
<wgrant> Ah
<wgrant> No subprocess deaths yet, anyway
<StevenK> Sigh, it's tested via two doctests
<StevenK> Which both make up 2000 lines
<StevenK> :-(
<lifeless> ahoy there
<wgrant> lifeless: Hi
<wgrant> We fixed Jenkins
<wgrant> And found that Launchpad itself has more than 900 million branchrevisions
<wgrant> That about covers the horrors that you missed :)
<lifeless> 1billion!
<wgrant> dollars, yes
<lifeless> cool
<wgrant> Flight not delayed by 12 hours yet?
<lifeless> not yet
<lifeless> what was causing the segfaults ?
<StevenK> hardlink restrictions
<lifeless> orly
<lifeless> -fun-
<wgrant> Yeah
<wgrant> xvfb-run fails to start X properly
<wgrant> So it just continues
<wgrant> Without X...
<lifeless> nice
<lifeless> did you file a bug ?
<wgrant> Well, I'm not sure if it actually sees an error from the syscall
<wgrant> It probably does, but I'm waiting to see if the build succeeds first
<StevenK>     ProgrammingError: permission denied for relation archivedependency
<StevenK> Aw fer ...
<StevenK> wgrant: So I should add main/restricted/universe to the vocab as well?
<wgrant> StevenK: Maybe.
<wgrant> StevenK: How would the UI show that?
<StevenK> Hmmm
<StevenK> That is a good point
<wgrant> You could perhaps add an extra option if the component is set to an unsupported value
<wgrant> Let any value be preserved by the UI, but not necessarily set
<StevenK> Right
<StevenK> I think I'm going to be forced to rewrite this doctest, it's annoying me.
<wgrant> wallyworld_: I think your MP fixes bug #618399
<_mup_> Bug #618399: DistroSeries:+nominations timeouts <bug-nomination> <lp-blueprints> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618399 >
<wallyworld_> oh, ok. thanks
<wallyworld_> i guess it does if it's not there anymore :-)
<wgrant> Right, it's no longer going to call canApprove on 75 nominations
<wgrant> Because it doesn't need to decide whether to show the control :)
<wallyworld_> yep
<StevenK> I note Development:Coding has been wgrant'd
<StevenK> It doesn't help that wallyworld_ has 3 cards in Review
<wgrant> 3503073kB   369kB/s | Fetching revisions:Inserting stream:Estimate 2710795/2795277
<wgrant> <3 lp:chromium-browser
<lifeless> wait, 3.5GB ?
<wgrant> Yes...
<lifeless> binary blobs in bzr?
<wgrant> Recall that chromium includes all its deps
<wgrant> It's not much overhead, either
<wgrant> 2.9G	/tmp/chromium-browser/.bzr/repository/upload/
<StevenK> wgrant: Why are you branching it?
<wgrant> To perform a manual incremental scan
<wgrant> To unbreak the LP DB state
<wgrant> hopefully
<wgrant> I could do it by manually hacking last-revision, but it's easier this way
<wgrant> Not quicker
<wgrant> But easier..
<lifeless> uhm
<lifeless> so this is in the DC?
<wgrant> No
<lifeless> why not branch --stacked; bzr push ?
<wgrant> It's local
<wgrant> I need to be able to examine history to chunk it properly
<lifeless> ah
<lifeless> well, gl.
<wgrant> I guess network latency in the DC wouldn't be too bad, though
<lifeless> hey, its your internets quota.
 * bigjools strokes unlimited internets
 * wgrant strokes 500GB Internets
 * lifeless strokes it to the left
 * bigjools doesn't want to see a pelvic thrust
<StevenK> I have 60GB, but that's peak. Off peak is unlimited.
<StevenK> My plan has been grandfathered about seven times and Exetel keep bugging me to switch
<lifeless> StevenK: let me guess, you told them to stroke it ?
<StevenK> Last time they rang me, I told them I'd think about it and get back to them.
<StevenK> I did the first part for about a second and have yet to do the second.
<wgrant> lifeless: Mmm
<wgrant> lifeless: Aren't DB reviews different because they have a habit of being catastrophic and difficult to fix?
<lifeless> wgrant: so entry to the club may be harder
<lifeless> wgrant: only folk in the club can sign more folk in :)
<lifeless> wgrant: we have a qa pipeline to catch gross problems
<wgrant> True.
<lifeless> ok, flap flap
<wgrant> One of the slaves has finished
<wgrant> No errors, failures, or subprocess deaths :)
<StevenK> There's two slaves done
<wgrant> three
<StevenK> Three
<StevenK> Haha
<wgrant> So
<wgrant> Now we just need to make sure result reporting works
<StevenK> Four
<wgrant> ideally getting failure counts etc.
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project devel build #1106: FIXED in 1 hr 26 min: https://lpci.wedontsleep.org/job/devel/1106/
<wgrant> Then we need a tarmac config and to tie up hloeung and spm next week
<wgrant> :D
<StevenK> Right, let me fix the script to tie subunit2junit back in
<wgrant> 5126232kB   665kB/s | Fetching revisions:Inserting stream:Estimate 2925464/3035215
<wgrant> I'm glad I didn't try to branch this on a FAT32 volume
<wgrant> -rw-rw-r-- 1 wgrant wgrant 4439527424 Oct  4 12:43 /tmp/chromium-browser/.bzr/repository/upload/oclqmgbw90pn25rdjjp0.pack
<StevenK> Haha
<StevenK> wgrant: Right, done. Shall we force another build?
<wgrant> StevenK: Do I have enough privs to examine the config?
<wgrant> I'm going to compare it to buildbot and what I know about production
<wgrant> So we get something as close to prasÃ© as possible
<StevenK> I'm not sure
<StevenK> Try? :-)
<wgrant> I have a Build Now button
<StevenK> https://lpci.wedontsleep.org/job/devel/ and 'Configure'
<wgrant> But not much else
<wgrant> No Configure link
<StevenK> wgrant: http://pastebin.ubuntu.com/1259265/
<StevenK> That's what Jenkins runs
<StevenK> Bleh, this Archive:+edit-dependencies code is horrible
<wgrant> Ah
<wgrant> So the only big missing thing is the .testrepository preservation
<StevenK> Nah, it will be
<wgrant> (that's init_testr.py)
<StevenK> It's ignored by bzr
<wgrant> Oh, will it?
<StevenK> So it won't get reverted out
<wgrant> Oh
<wgrant> I see
<wgrant> That sounds sensible, then.
<StevenK> And it will only run testr init if it needs to
<StevenK> The script might need fixing for parameterized builds but eh
<wgrant> So, fire off 1107
<wgrant> Great
<wgrant> + [ ! -d .testrepository ]
<wgrant> + [ -d download-cache ]
<wgrant> So it is indeed preserved
<wgrant> StevenK: You've added the sysctl call to the instance setup script?
<wgrant> Not that it's relevant to production.
<StevenK> I did
<StevenK> I'm tempted to try a m2.4xlarge with concurrency 8
<wgrant> Not much point
<wgrant> Does the maas jenkins+tarmac stuff require parameterised builsd?
<StevenK> wgrant: This bug is making me very sad.
<wgrant> StevenK: https://wiki.canonical.com/UbuntuEngineering/QA/UnityAutolandingSetup
<StevenK> On jenkins create a freestyle job with 3 string parameters - dir, trunkrev and branch - this job will be duplicated and renamed for all the projects.
<StevenK> So yeah
<StevenK> I suspect we only need branch
<wgrant> The pack is now 5.5GB...
<wgrant> This is a few times bigger than I expected
<wgrant> Google, what have you done
<wgrant> blargh what
<wgrant> JUnit XML doesn't stream?
<wgrant> :(
<wgrant> StevenK: Has the test count graph disappeared because the last $LOTS runs didn't return any XML?
<StevenK> wgrant: It has been streaming
<StevenK> wgrant: The test count graph disappeared because I turned off that bit due to no XML.
<StevenK> It's back on for this build, so we'll see
<wgrant> 8.4GB of download later, it's finally building the tree
<wgrant> RSS of 2GB, as well
 * StevenK stabs this bug
 * StevenK gives up and finds something that doesn't make him want to scream
<StevenK> Hmmm. A test for bug 1061374 might be complex
<_mup_> Bug #1061374: PlainPackageCopyJob.attemptCopy() crashes if only binaries were copied <easy> <oops> <package-diff> <regression> <soyuz-core> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1061374 >
<wgrant> StevenK: Not hugely
<wgrant> Publish source in one pocket
<wgrant> Publish that source + a new binary in a second pocket
<wgrant> Copy from second to first
<wgrant> boom
<StevenK> wgrant: 25 line test, though ...
<wgrant> :(
<StevenK> Now 30
<StevenK> Hmmm, no boom
<StevenK> wgrant: And see the graph is back is all its graphiness
<wgrant> Indeed
<StevenK> wgrant: So what is next for Jenkins?
<wgrant> StevenK: A good question
<wgrant> StevenK: Perhaps try setting up tarmac against qastaging or something, so we can know how it works?
<StevenK> wgrant: I can set up a parameterized build while you tackle tarmac if you wish?
<wgrant> StevenK: Sounds reasonable.
<StevenK> (Pdb) p error_text
<StevenK> u"copyme 2.8-1 in breezy-autotest (The signer of this package has no upload rights to this distribution's primary archive.  Did you mean to upload to a PPA?)"
<StevenK> Thanks for nothing, mr test
<StevenK> copyme 2.8-1 in breezy-autotest (No such distro series breezy-autotest in distribution ubuntutest.)
<StevenK> :-(
<StevenK> wgrant: I'm done. Do we have an LP branch on qas to test with?
<wgrant> StevenK: Nothing that's reasonably up to date
<wgrant> I'm trying to fight with Canonistack
<wgrant> Since I only have 512Kbps upstream
<wgrant> Ah, it works now
 * StevenK is also wondering why breezy-autotest is the distroseries
<wgrant> StevenK: It's in the PCJ JSON metadata, I believe
<wgrant> If you're using makePCJ then it probably defaults to it
<StevenK> I'm using STP.getPubSource and friends, so I don't see why the copy is failing with no such series
<wgrant> How're you creating the PCJ?
<StevenK> wgrant: http://pastebin.ubuntu.com/1259368/
<StevenK> wgrant: Oh, blah, the branch doesn't need to be on qas. Something simple kicking around that isn't merged?
<wgrant> StevenK: Oh, you just want to test the parameterised job now?
<StevenK> Right
<wgrant> A good start
<StevenK> See if it starts and then kill it
<wgrant> lp:~wallyworld/launchpad/dynamic-bug-listing-nominations2-1060015
<LPCIBot> Project test-branch build #1: FAILURE in 7.6 sec: https://lpci.wedontsleep.org/job/test-branch/1/
<StevenK> Heh heh
<StevenK> + bzr merge http://bazaar.launchpad.net/~wallyworld/launchpad/dynamic-bug-listing-nominations2-1060015
<StevenK> bzr: ERROR: No WorkingTree exists for "file:///home/jenkins/launchpad/lp-branches/workspace/test-branch/.bzr/checkout/".
<StevenK> :-(
<LPCIBot> Project test-branch build #2: ABORTED in 3 min 48 sec: https://lpci.wedontsleep.org/job/test-branch/2/
<LPCIBot> Project test-branch build #3: ABORTED in 31 sec: https://lpci.wedontsleep.org/job/test-branch/3/
<wgrant> Did it work?
<StevenK> Yes
<StevenK> copyme 2.8-1 in breezy-autotest (source has no binaries to be copied)
<StevenK> LIES!
<wgrant> The binaries are in the wrong pocket.
<StevenK> Silence.
<StevenK> But thanks.
<StevenK> wgrant: Same message :-(
<wgrant> paste test
<wgrant> Wrong archive?
<StevenK> wgrant: http://pastebin.ubuntu.com/1259385/
<wgrant> Wrong archive.
<StevenK> Putting archive=archive into getPubBinaries changes nothing
<wgrant> Hm
<wgrant> Have you checked that it actually creates any binaries?
<wgrant> Remember that it respects createMissingBuilds
<StevenK> Hmmm
<StevenK> It returns two BPPHs
<wgrant> pdb
<wgrant> compare the BPPHs and SPPH
<wgrant> Check that SPPH.getBuiltBinaries() returns something useful
<wgrant> That's what the copier uses
<wgrant> It should return a BPPH for every distinct BPR built from that source and published in the same suite and archive
<StevenK> [<BinaryPackagePublishingHistory at 0x10a747d0>]
<wgrant> Oh
<wgrant> Heh
<wgrant> It's picking the wrong SPPH
<wgrant> Um
<wgrant> You need to create them the other way around
<wgrant> Create the target source first
<wgrant> We ran into this this morning
<wgrant> It determines the source by a simple key of (archive, name, version)
<wgrant> No suit
<StevenK> Hahaha
<wgrant> So it just takes the latest
<StevenK> copyme 2.8-1 in breezy-autotest (a different source with the same version is published in the destination archive)
<StevenK> Bleh
<wgrant> Make sure it's the same SPR
<wgrant> It may be better to use the factory here
<wgrant> Since you really only need one binary
<StevenK> (Pdb) s
<StevenK> ForbiddenAttribute: ('sourcepackagerelease', <BinaryP...106b7150>)
<StevenK> Boom
<wgrant> :)
<StevenK> But the test passes
<wgrant> :(
<StevenK> wgrant: It's been a while since I used STP, I forgot that getPubSource is jealous of any other SPR and will create one everytime
<wgrant> Yes
<StevenK>                 # Otherwise, rely on the job runner to do the final commit,
<StevenK>                 # and do not consider a failure of a copy to be a failure of
<StevenK>                 # the job.
<StevenK> Hmmm
<StevenK> So if runJob doesn't blow up, how can I die?
<wgrant> Hm
<wgrant> You might have to get the exception from elsewhere
<StevenK> Like where, though?
<wgrant> Where'd you get it before?
<wgrant> In pdb?
<StevenK> Yeah
<StevenK> Hm, but run re-raises it
<StevenK> Right, test fails
<StevenK> reference = <DBItem JobStatus.COMPLETED, (2) Completed>
<StevenK> actual    = <DBItem JobStatus.FAILED, (3) Failed>
<StevenK> wgrant: How is tarmac?
<wgrant> Nearly working
<wgrant> Not talking to jenkins yet, but it seems to be otherwise OK
<StevenK> I think this branch is done too
<StevenK> Just seeing if test_pcj works
<wgrant> Ah
<wgrant> bzr is happier with lp://qastaging URLs when upgraded to a version from after qastaging was created, it turns out
<StevenK> Hahaha
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/attemptcopy-no-source/+merge/127932
<StevenK> Because wallyworld_ and Soyuz == unhappy Ian
<wallyworld_> :-)
<StevenK> Hahah, Deployment:Ready is quite scary
<wgrant> StevenK: Did you consider ignoring STP entirely?
<wgrant> Also, it might be nice to check for pending publications afterwards
<wgrant> To confirm that it indeed only created a single new BPPH, and no SPPH
<wgrant> Because if it created an SPPH (eg. because overrides happened to differ), the test is invalid.
<StevenK> wgrant: To be honest, the test was cargo-culted
 * StevenK looks for the IArchive method
<StevenK> Oh, it's on DistroSeries. Sneaky
<wgrant> Use the one on IArchive
<wgrant> getPublishedSources and getPublishedOnDiskBinaries, IIRC
<wgrant> (the latter is exposed as just getPublishedBinaries to the API)
<StevenK> Why not IDistroSeries.getPendingPublications() ?
<wgrant> Oh
<wgrant> Is that the one that's used by publish()?
<wgrant> If so, that works.
<wgrant> You'll probably also need DAS.getPendingPublications() too, though
<StevenK> Oh bleh, I was wanting one call!
<wgrant> You can't have one call
<wgrant> You can have two calls, though
<wgrant> By using getPublishedSources/getPublishedOnDiskBinaries on Archive
<wgrant> Rather than digging out the DAS and calling stuff on that and hoping you get the right one.
<StevenK> Right
<StevenK> Yeah, I just saw IDistroSeries.getPendingPublications will only return sources and thought, "Oh bleh I don't want to dig up the DAS."
<wgrant> I fear that mawson may have aspirations of being Deep Thought.
<StevenK> Haha
<wgrant> It's been chewing on this query for more than 5 hours
<StevenK> wgrant: http://pastebin.ubuntu.com/1259441/
<wgrant> StevenK: I'd drop the name/pocket restrictions
<wgrant> StevenK: You should be working in a clean archive anyway
<StevenK> wgrant: Done, anything else?
<wgrant> Does the test pass?
<StevenK> Yes
<wgrant> Then that looks reasonable. Push and I'll check and approve.
<wgrant> Evening lifeless
<wgrant> Welcome to the right side of the world
<StevenK> wgrant: Tarmac? I say, hopefully
<wgrant> Heh
<wgrant> Still trying to coerce qastaging into not timing out on branch scans
<wgrant> Since Tarmac relies on branch scans
<StevenK> Hah
<StevenK> wgrant: Diff updated
<wgrant> StevenK: Hm, you're still using STP and specifying some args you don't care about, but I guess it's not too bad.
<StevenK> wgrant: Did you want to stop using STP?
<StevenK> want me, even
<wgrant> STP is terrible and unnecessary here
<wgrant> LOF should be a pretty trivial replacement
<wgrant> And end up slightly shorter.
<wgrant> StevenK: Can you give me Jenkins superpowers so I can try to configure an auth key and stufF?
<StevenK> I was trying to avoid that :-P
<wgrant> Can I have an auth key for the job, then? :)
<wgrant> Merging lp://qastaging/~me+testing/launchpad-tarmac-test/omg into lp://qastaging/~me+testing/launchpad-tarmac-test/tarmac-trunk is ignored: The Jenkins server https://lpci.wedontsleep.org/job/test-branch is not reachableHTTP Error 403: Forbidden.
<StevenK> wgrant: You have superpowers
<wgrant> Thankyou sir
<nigelb> StevenK: Didn't he already have them? :)
<StevenK> Nope
<StevenK> Only me, and lifeless from yesterday
<nigelb> ah
<StevenK> wgrant: LOF now results in a failing test
<StevenK> And it's as slow
<wgrant> :(
<StevenK> Oh, no, it's 0.3 seconds faster
<StevenK> But it fails :-)
<wgrant> It fails because you're calling it incorrectly :P
<StevenK> makeBPPH(status=PackagePublishingStatus.PUBLISHED, pocket=PackagePublishingPocket.UPDATES, archive=archive,
<StevenK> source_package_release=spph.sourcepackagerelease)
<StevenK> That should be fine ...
<wgrant> distroseries
<StevenK> makeBPPH wants a DAS
<StevenK> And I didn't want to look that up, remember?
<wgrant> Hm, I thought it would take a DS and automatically make a DAS
<StevenK> It doesn't take a DS argument, no.
<wgrant> Ah
<wgrant> This Unity Tarmac fork is a bit special
 * StevenK makes a DAS
<StevenK> wgrant: How so?
<wgrant> StevenK: Well, you know how the document passed a dir= arg?
<wgrant> Which didn't seem to make much sense...
<StevenK> Oh
<StevenK> It's asuming tarmac and jenkins on the same machine?
<wgrant> It assumes that Tarmac and the Jenkins slave are on the same FS
<wgrant> Yeah
<wgrant> Fixing
<StevenK> wgrant: Still a failure with passing a DAS
<wgrant> StevenK: What's the failure?
<StevenK> No PENDING binary
<wgrant> The DAS is in the right DS?
<StevenK> das = self.factory.makeDistroArchSeries(distroseries=self.distroseries)
<wgrant> If so, paste it and I'll poke it with a stick
<StevenK> Oh, I bet I know what it is
<StevenK> Yeah
<StevenK>  1 file changed, 9 insertions(+), 9 deletions(-)
<StevenK> Not shorter too
<wgrant> But significantly less 2005
<StevenK> Yeah, it's 2006 now
<StevenK> wgrant: Diff updated again
<wgrant> StevenK: r=me with a trivial comment
<wgrant> Thanks
<StevenK> Tarmac!
<StevenK> :-)
<wgrant> Just tweaking the Jenkins job...
<StevenK> wgrant: Right, I did consider that, but I have to pass them to the job creation
<wgrant> StevenK: Oh, that's a good point and I'm stupid
<wgrant> Carry on :)
<StevenK> And spph.sourcepackagename and spph.source_package_version are long
<StevenK> wgrant: Tweaking how?
<wgrant> Adding a revid
<wgrant> Unity (and probably MAAS) use it rather strangely
<StevenK> I was assuming tip
<wgrant> They use Tarmac's staged tree with trunk + uncommitted merge
<wgrant> Pass that tree to the Jenkins job
<wgrant> And branch= is set to the packaging branch, to additionally merge in I guess.
<StevenK> Yeah
<wgrant> Jenkins is careful about using a revision ID
<wgrant> s/Jenkins/tarmac/
<wgrant> I wonder how much chaos I can cause with a crafted revid...
<StevenK> That's one reason I ignored revid
<StevenK> wgrant: Your changes look pretty good
 * wgrant crosses fingers
<StevenK> Should I be seeing a build?
<wgrant> Once I actually fix it to not crash
<wgrant> Aha
<wgrant> I see a build
<LPCIBot> Project test-branch build #4: STILL FAILING in 19 sec: https://lpci.wedontsleep.org/job/test-branch/4/
<wgrant> Started by remote host 127.0.0.1
<wgrant> orly
<StevenK> Bang
<wgrant> Ah, source URL is none
<wgrant> odd
<wgrant> Oh
<LPCIBot> Project test-branch build #5: STILL FAILING in 10 sec: https://lpci.wedontsleep.org/job/test-branch/5/
<wgrant> Heh
<wgrant> This is why we don't use string formatting to construct URLs.
<StevenK> lp://qastaging/~me testing/launchpad-tarmac-test/omg
<StevenK> Hahahahaha
<wgrant> Not dead yet...
<StevenK>  M  COPYING
<wgrant> And it has the branch merged
<StevenK> Haha
<wgrant> Yeah
<StevenK> What's the diff? :-)
<wgrant> -See the file LICENSE for copyright information.
<wgrant> +It's all mine.
<StevenK> Hahaha
<StevenK> I really do need to talk to your mother about sharing.
<wgrant> I didn't notice a timeout, so I wonder if it'll wait long enough...
<LPCIBot> Project test-branch build #6: ABORTED in 2 min 15 sec: https://lpci.wedontsleep.org/job/test-branch/6/
<StevenK> Blink. Was that you?
<wgrant> Yes
<wgrant> Going to trim the actual test step off
<wgrant> And retry
<StevenK> Heh
<wgrant> https://code.qastaging.launchpad.net/~me+testing/launchpad-tarmac-test/omg/+merge/94324 has some helpful comments
<StevenK> wgrant: How different is this Unity one from upstream?
<wgrant> It might just have the extra plugin, but I'm not sure
<LPCIBot> Project test-branch build #7: STILL FAILING in 18 sec: https://lpci.wedontsleep.org/job/test-branch/7/
<StevenK> Yeah, you can't do that.
<wgrant> I touched it
<wgrant> Should work now
<StevenK> In the build step? Naughty
<LPCIBot> Project test-branch build #8: STILL FAILING in 18 sec: https://lpci.wedontsleep.org/job/test-branch/8/
<StevenK> Hahaha
<wgrant> I'm confused
<wgrant> All the file's timestamps are within the last 5 minutes
<adeuring> good morning
<StevenK> wgrant: It said 46 seconds ago
<StevenK> I think you'll need to touch it in the build step
<wgrant> Oh, was looking at wrong log, indeed
<StevenK> Or just remove that post build action
<wgrant> But that's more annoying to revert
<StevenK> Not really
<LPCIBot> Yippie, build fixed!
<LPCIBot> Project test-branch build #9: FIXED in 23 sec: https://lpci.wedontsleep.org/job/test-branch/9/
<wgrant> Hm, taking its time to push
<StevenK> Or not pushing at all
<wgrant> It is pushing
<wgrant> 17MB so far
<wgrant> Not sure why
<StevenK> No stacking?
<wgrant> It is stacked
<wgrant> And it's allegrdly done
<wgrant> Yay
<wgrant> It worked
<wgrant> https://code.qastaging.launchpad.net/~me+testing/launchpad-tarmac-test/tarmac-trunk
<StevenK> Neat
<StevenK> Can I destroy the executor now?
<wgrant> Surew
<lifeless> o/
<wgrant> Hi
<lifeless> review plox - https://code.launchpad.net/~lifeless/python-timeline/bug-891989/+merge/127951
<wgrant> StevenK: Most of Unity's changes are irrelevant, and the Jenkins plugin needs a bit of a rewrite.
<wgrant> Might make a proper branch (extending lp:lp-tarmac-config, perhaps) tomorrow
<StevenK> \o/
<StevenK> r=IRC Nick would be a neat trick, too
<wgrant> lp:lp-tarmac-config takes a different approach
<wgrant> It just matches the usual commit regexp
<wgrant> So you have to put [r=] etc in yourself
<wgrant> Which is not ideal, but we could always autogenerate.
<wgrant> But then no-qa becomes hard
<StevenK> Right
<StevenK> That would work
<lifeless> wgrant: StevenK: please?
<lifeless> ciao for now
<StevenK> lifeless: r=me
<StevenK> And now for some DXHR
<bigjools> wgrant: do you know if there's a gotcha amending system .py files used in a wsgi app?
<wgrant> bigjools: Amending how?
<wgrant> "system" maening "packaged"?
<bigjools> yes
<bigjools> hacking in some logging
<bigjools> my changes aren't being used
<wgrant> Make sure you get the right copy
<bigjools> haha
<wgrant> Are you using dh_python2, python-support, python-central..??
<bigjools> dh
<wgrant> I assume dh_python2, given you're only >=precise
<wgrant> bigjools: Which file are you editing?
<bigjools> there is only one maas/settings.py on here and the .pyc happily sits there not updated
<wgrant> The pyc won't be updated, as it's root-owned
<wgrant> But Python should ignore it if the .py's mtime is newer
<bigjools> right
<wgrant> dh_python2 symlinks, rather than the traditional copying around that python-support and python-central did, so there should only be one copy
 * bigjools inserts deliberate syntax errr
<wgrant> But perhaps your settings.py is magical?
<wgrant> Heh
<bigjools> aha
<wgrant> Good aha, or bad aha?
<bigjools> well it proved it's picked up changes
<wgrant> Ah
<wgrant> Yeah, dh_python2 makes it pretty hard to get wrong
<abentley> deryck: This looks broken to me, because I can't add myself to proprietary , even though specs default proprietary.: https://qastaging.launchpad.net/bzrtools/+sharing
<abentley> deryck: Aah, I just "fixed" it by making bugs proprietary.
<deryck> abentley, but still there's a bug there, right?  you should be able to do so with just specs being proprietary, right?
<abentley> deryck: right.
<deryck> abentley, will you file a bug about that?  And add a card to the beta lane?
<abentley> deryck: sure.
<deryck> abentley, thanks!
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: ~300
<rick_h_> jcsackett: if you get a sec please https://code.launchpad.net/~rharding/launchpad/fix_banner_cleanup/+merge/128026
<jcsackett> rick_h_: r=me.
<abentley> jcsackett: Could you please review https://code.launchpad.net/~abentley/launchpad/info-type-app-config/+merge/128064 ?
<jcsackett> abentley: sure.
<jcsackett> abentley: the MP is showing you have a text conflict.
<abentley> jcsackett: It's just duelling insertions.
<jcsackett> abentley: e.g. you intend to keep all code in the TREE and MERGE-SOURCE bits?
<abentley> jcsackett: Yes.
<jcsackett> abentley: ok. r=me.
<rick_h_> abentley: ping, have a sec?
<abentley> rick_h_: pong.
<rick_h_> abentley: hey, I think jcsackett has me ok atm. At least more to dig into. Trying to get the right .pt for hte right context/project/product
<abentley> rick_h_: Cool.  Sorry, I had a late lunch.
<abentley> rick_h_: The only remaining alpha card is "Add information type to ProjectGroup $project/+newproject" so I'll take it, unless you've started.
<rick_h_> abentley: no, I was going to ask though since I thought it was said we didn't want to add information type to projectgroup?
<abentley> rick_h_: This is about creating products from a projectgroup page.
<rick_h_> abentley: ah ok, so not the projectgroup itself, gotcha
<rick_h_> abentley: have time to do a quick hangout?
<abentley> rick_h_: sure.
<abentley> rick_h_: I got it working by disabling your IProductSet checks.  Do you remember what those are for? http://pastebin.ubuntu.com/1260787/
<sinzui> jcsackett, do you have time to review a quick fix to not show bug listings if the project does not use Lp to track bugs? https://code.launchpad.net/~sinzui/launchpad/external-bug-tracker/+merge/128114
<jcsackett> sinzui: absolutely.
<jcsackett> sinzui: out of curiosity, do you have a credit of 3000+ lines as of this week, or have you earned 3000+ lines in this week alone? :-P
<sinzui> good question. I must confess I was guessing at 3000
<sinzui> I may have 4.5k to 5
<sinzui> do you hate me?
<sinzui> jcsackett, I gave wallyworld_ a pointer to two files in utilities that would give him -2500, but he has not use it
<jcsackett> sinzui: i don't hate you, just impressed. :-P
<jcsackett> i am not bitter about the LoC req. just occasionally inconvenienced.
<jcsackett> sinzui: i may well use those utilities changes if he doesn't though.
<sinzui> jcsackett, formatdoctest.py is shipped with pocket-lint. we don't needed it.
<abentley> sinzui: ISTM that if we want to reduce LOC, then LOC credit should decay.  But that does incent people to not remove LOC until they need credit.
<sinzui> also we don't use the porn report anymore, so you can delete the xxxreport.py
<jcsackett> abentley: that's a thought. there's also the fact that we seem to use raw line counts, and whitespace/comments probably shouldn't count.
<sinzui> abentley, it gets reset. Francis had Purple's LoC reset when we returned to maintenance
<abentley> sinzui: Oh.
<jcsackett> sinzui: r=me, though there is a typo to fix.
<abentley> sinzui: So you were counting LOC for work on private projects?
<abentley> or disclosure, etc?
<abentley> Actually, I think we want an incentive to maximize our LOC credit.
<cjwatson> View it as a way of keeping score, not as currency to spend?
<sinzui> abentley, we are damned if we do, though three of us have credits of about 40,000, the other two have 70,000. sharing + pickers + privacy-transitions added 30k
<sinzui> I don't think the deletion of BVP and bug subscription rules will amount to 30k
<wallyworld_> sinzui: i haven't done those files yet, but it's on my todo list
<sinzui> wallyworld_, I hope the file a pastebin works for you
<abentley> deryck: This code appears to have no effect (does not set default): self.widgets['information_type'].value = InformationType.PUBLIC
<sinzui> s/a pastebin/i pastebined/
<wallyworld_> sinzui: the chrome certificate issue is fixed, yes :-) \o/
<deryck> abentley, did I write that?  Or can you give me a larger snippet for context?
<abentley> deryck: I thought it was from your branch yesterday, but I was wrong.
* jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-dsp_picker-ff/+merge/128138
<wgrant> Already looking :)
<wgrant> StevenK: r=me
<StevenK> Hahah and wallyworld_ approved too
<wgrant> Hm
<wgrant> 14 Fix Committed criticals
<wgrant> That may be a record since the new definition began
<wallyworld_> doh, bb failure
 * wallyworld_ fixes
<StevenK> Bah, did I just hit testfix?
<lifeless> oh noes :P
<StevenK> Yes!
<StevenK> It's such a large problem that I'm going to destroy testfix entirely
<sinzui> wgrant, jcsackett sanitised the name because zope's traversal see the ? or the escaped ? and strips it. The librarian does have the name, Lp's own code does not
<wgrant> NotFound: Object: <lp.services.webapp.publisher.RootObject object at 0x968d82c>, name: u'foo?'<br />
<wgrant> Zope sees it fine, hm
<wgrant> I wonder if it may be our Apache config
<wgrant> Since it works fine on launchpad.dev
<wgrant> Without jcsackett's branch
<StevenK> Is this the bug attachment with a ?
#launchpad-dev 2012-10-05
<wgrant> I just uploaded and downloaded an attachment named 'huh?'
<sinzui> wgrant, https://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/983116/+attachment/3077581/+files/HUD-to-do-what%3F-de.png
<_mup_> Bug #983116: [Component: Keyboard] HUD is <Alt> not <Alt L> <amd64> <apport-bug> <hud> <precise> <gnome-control-center (Ubuntu):New> < https://launchpad.net/bugs/983116 >
<wgrant> Yeah, prod strips them
<wgrant> Compare the exception on https://launchpad.net/huh%3F and https://launchpad.dev/huh%3F
<wgrant> https://dogfood.launchpad.net/huh%3F shows the correct behaviour
<wgrant> qastaging is bad
<wgrant> So something on the prod/qas frontend configs is molesting our URLs; there's no LP bug here.
<sinzui> yes, jcsackett confirmed that qa and prodct beahve the same
<wgrant> So I would suggest replacing that MP with an RT
<wgrant> Who knows what else the frontends are doing :/
<sinzui> and watch it for 5 years?
<wgrant> Nah, I'll coerce APAC webops to give me the configs next week and then give them a diff :)
<wgrant> Diffs are a good way to accelerate RTs
<wgrant> I am glad it is reproducible on qastaging; that makes things much easier.
<wgrant> Since there's no squid or haproxy, AIUI
<StevenK> steven@undermined:~% wget -Sq https://qastaging.launchpad.net 2>&1 | grep X-Cache
<StevenK>   X-Cache: HIT from arsenic.canonical.com
<wgrant> Oh
<wgrant> So *that's* what arsenic does...
<wgrant> s/Oh/Oh god why/
 * wgrant files the RT
<StevenK> arsenic for staging too
<wgrant> We may need to do the same tweak we did to get the restricted librarian working
<wgrant>     # nocanon per RT#42560, woe with LP's handling of eg %2B
<wgrant>     ProxyPass / balancer://launchpad-librarian-cluster/ nocanon
<_mup_> Bug #42560: /var is mounted over /var/run and /var/lock <sysvinit (Ubuntu):Invalid> < https://launchpad.net/bugs/42560 >
<wgrant> That shouldn't be it, since decoding the %3F is not a legal part of canonicalization, but it's possible.
<wgrant> Whatever it is, either apache or squid is in the wrong here, not LP
<wgrant> It looks like it's probably our squid config, but I don't think I have a copy.
 * StevenK glares at DF
<StevenK> With no builder how am I supposed to QA? :-(
<wgrant> StevenK: DB hacking or qa-meh
<StevenK> Yeah, qa-meh might be the way
<StevenK> wgrant: I wonder if bug 834293 is fixed, we did a bunch of work on branch privacy
<_mup_> Bug #834293: Product:+code-index times out <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/834293 >
<wgrant> StevenK: No
<wgrant> We made it better, but not much better
 * wgrant finds data
<wgrant> https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-monthly-pageids.html
<wgrant> 99% under 7.26s
 * StevenK made the mistake of searching OOPS on neem with 'AssertionError'
<StevenK> 1648 OOPSes later ...
<StevenK> Almost all of them appear to be AssertionError: Bug #504291: Store left in a disconnected state.
<wgrant> StevenK: Why?
<_mup_> Bug #504291: DisconnectionErrors (already disconnected) happening again <fastdowntime> <lp-foundations> <oops> <Launchpad itself:Incomplete by stub> <Storm:Invalid> < https://launchpad.net/bugs/504291 >
<wgrant> We know the precise circumstances around +edit-dependencies
<wgrant> So we don't need to grep for it
<StevenK> I wasn't
<StevenK> That grep was for bug 808950
<_mup_> Bug #808950: AssertionError: You exported name as an IChoice based on an SQLObjectVocabularyBase, you should use lazr.restful.fields.ReferenceChoice instead <api> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/808950 >
<wgrant> Ah
<StevenK> But doesn't seem to show up
<wgrant> StevenK: Have you tried to reproduce it?
<StevenK> I was looking for an OOPS as the first step.
<wgrant> "AssertionError: You exported milestone_assignment as an IChoice based on an SQLObjectVocabularyBase, you should use lazr.restful.fields.ReferenceChoice instead." on Product:EntryResource:searchTasks
<wgrant> is pretty clear
<wgrant> And the fix is obvious
<wgrant> (hint: grep the entire tree for the field name)
<StevenK>         # The vocabulary should be HWBus; this is fixed in
<StevenK>         # _schema_circular_imports to avoid circular imports.
<StevenK> Didn't you kill that?
<wgrant> _schema_circular_imports continues to live
<wgrant> Its death will come only from lazr.restful's demise, or being split up into app-specific _schema_circular_importses.
<StevenK> wgrant: Hmm, I don't get it. Since bugtasksearch has assignee as a Choice
<wgrant> StevenK: Look at the other places that use it, and it should become clear
<StevenK> Hah. Assignee is a Reference
<StevenK> In bugtarget
<wgrant> That's fine.
<StevenK> But milestone_assignment isn't
<wgrant> We only care about milestone_assignment, don't we?
<StevenK> We shouldn't use copy_field, but use Reference?
<wgrant> I'd look at how the other code that goes near that field uses it.
<StevenK> wgrant: What other code? milestone_assignment turns up in two places.
<wgrant> You're on the right track.
<StevenK> That wasn't the answer I was expecting. :-)
<StevenK> wgrant: Oh, it isn't used at all./
<StevenK> search params never assigns it in __init__
<wgrant> Bingo
<StevenK> Should I drop it entirely then?
<StevenK> It's obviously exported, but use of it will return 500s
<wgrant> Right, it's probably in 1.0, but it's also likely that any attempt to use it crashes
<wgrant> Check that, and if it always crashes then obliterate it
<wgrant> If it doesn't always crash, think about it a big and then obliterate it anyway.
<wgrant> bit
<StevenK> Hah
<StevenK> So it wants a milestone name ?
<wgrant> I think it wants a milestone reference.
<wgrant> Given that it's using the Milestone vocab
<wgrant> But it's exported as a Choice, which means it wants an enum value
<wgrant> Which is why it crashes
<wgrant> Try giving it references and names and see if there's anything that doesn't OOPS
<StevenK> OOPS-a8ab241f5057264be5073a00263b04fb and OOPS-2c4ba13f50161b4d3ae38ccb94059308
<wgrant> Ah... ha...
<wgrant> The archeology of this is interesting
<wgrant> It actually dates back to batch edit functionality added in mid-2005.
<wgrant> In r1860
<wgrant> If milestone_assignment was set in search parameters, it looks like it set the milestone of the found bug tasks.
<StevenK> HAHAHA
<StevenK> It's only referenced in a doctest that has been disabled since 2005
<wgrant> Ah, no, there was a separate batch assignment form on the search page, but it all used that same "search" schema
<StevenK> # XXX Bjorn Tillenius 2005-08-01:
<StevenK> # This test is disabled in wait for SQLObject being fixed to accept
<StevenK> # both distinct=True and an orderBy argument.
<wgrant> xx-bugattachments.txt is disabled?
<StevenK> No, just the last test of it
<wgrant> Ah
<StevenK>   XXX print http(r"""
<StevenK>   ... GET /bugs/fir...
<StevenK> wgrant: What about name in that bug? Or is that long dead?
<wgrant> StevenK: It may still exist, but the pageid is bad
<wgrant> Perhaps search the apidoc for parameters named name
<wgrant> 'cause the method mentioned in the bug doesn't have onje
<wgrant> And I don't think it ever did
<wgrant> Huh
<wgrant> That code persisted until June 2006
<wgrant> But I'm pretty sure the bulk edit form was long gone by then, or I'd remember.
 * StevenK ignores name
<StevenK> wallyworld__: Read https://code.launchpad.net/~stevenk/launchpad/obliterate-milestone_assignment/+merge/128158 and weep
 * wallyworld__ reads
<wallyworld__> there were some cobwebs there
<StevenK> I think the cobwebs are covered in dust
<StevenK> Maybe this branch will land now
<StevenK> wallyworld__: Did you want to approve that MP, or are you afraid of the spiders that own said cobwebs?
<wallyworld__> StevenK: i'll aprove it, i just thought you wanted me to read it
<StevenK> wallyworld__: The reading was first, yeah
<StevenK> wallyworld__: I was curious about your reaction before you reviewed it
<wallyworld__> StevenK: so, you saying that if the api is used now, as it, it oopses?
<wallyworld__> as is
<StevenK> wallyworld__: searchTasks() works fine. If you pass in milestone_assignment as one of your arguments, you will get an OOPS.
<wallyworld__> ok, so no harm in removing it then
<StevenK> Right, and I doubt any one is using it
<StevenK> If they are, they move from a 500 to what, 404 or a lazr.restful error?
<wallyworld__> you may have won the prize for removing the oldest code
<wallyworld__> 40x for sure, not sure what
<wgrant> 400
<wgrant> Well, actually, once the WADL updates launchpadlib will probably tell them they're wrong before even sending the request
<wgrant> But still, if they were relying on getting a 500 from that then they're pretty strange :)
<StevenK> Let's find out in about an hour
<wgrant> Hm?
<wgrant> Why an hour?
<StevenK> buildbot and qas updating
<wgrant> Ah
<StevenK> Current run has 15 minutes to go
<StevenK> Unless wallyworld__ breaks it again
<wallyworld__> it was only once :-(
<StevenK> Twice
<wallyworld__> once
<wallyworld__> the other was suprious
<StevenK> Once this morning and the feature flag related changes a few days ago
<wallyworld__> same old thing we'vebeen seeing lately
<wallyworld__> i thought you meant today
<StevenK> Why did we add rf-setup-certs to LP?
<wallyworld__> curtis said to
<StevenK> I thought it was supposed to go into lp-dev-utils ?
<wallyworld__> oops, maybe, let me check
<wallyworld__> StevenK: the email said utilities
<StevenK> Hmmm
<wallyworld__> i can move it
<StevenK> wgrant: What do you think?
<StevenK> I think it should move
<wallyworld__> wgrant: StevenK: +expirable-bugs oops on a dsp. i can't see a link so it must have been via url hacking. i guess it should return a 404 instead of a 500? or did we want to implement expirable bugs on a dsp?
<StevenK> wallyworld__: So the ZCML has +expirable-bugs on IBugTarget
<StevenK> A DSP is a bug target, but it doesn't support it
<wallyworld__> ah. but there's no link on the dsp bugs page
<StevenK> Curtis and William are of the opinion that it should support it
<wallyworld__> ok, will look at that, thanks
<wallyworld__> and then we can add the link to the page
<wgrant> wallyworld__: I'm not really of the opinion that it should support it
<StevenK> wgrant: Do you think Colin's work is far enough long to close bug 556839?
<wgrant> I'm of the opinion that it might be easier to fix it to support it than it is to remove the page
<wgrant> Given it's registered for IBugTarget
<wgrant> StevenK: No
<StevenK> We don't use async by default?
<wgrant> wallyworld__: I'd put the cert gen script in lp-dev-utils, I think
<wgrant> As StevenK suggests
<wallyworld__> wgrant: why wouldn't we want to find expirable bugs on a dsp?
<wgrant> wallyworld__: 'cause even Distribution:+expirable-bugs had only 136 hits last month
<StevenK> Hah
<StevenK> And how many of those were Google?
<wallyworld__> hmmm.
<wgrant> Most were probably search engines, yeah
<wallyworld__> maybe we want to redo the zcml then to stop lying?
<wgrant> +expirable-bugs was mostly introduced to show what would happen when expiration was turned on
<wgrant> wallyworld__: Right, it's possibly best to just register it on IProduct and IDistribution
<wgrant> Not sure if IProductSeries or IDistroSeries or IProject have it linked
<wallyworld__> ok, i'll check that out. i think that's the better solution
<wallyworld__> rather than doing something no one will use
<wgrant> IProductSeries and IDistroSeries have about as many hits as I'd expect from Google
<wgrant> But it's difficult to say
<wgrant> wallyworld__: Right
 * wallyworld__ first gets out the hedge trimmer
<wgrant> In some situations like this it's a one-liner to fix it, so it's easier to just make it work
<wgrant> But this is not one of them
<wallyworld__> yeah, dicking eith _getTargetJoinAndClause() for dsp would be messy
<StevenK> wgrant: So, for bug 1050191, I agree with Curtis. If it was in +junk first, we don't care.
<_mup_> Bug #1050191: allocate-revision-karma.py is too slow to complete <branches> <karma> <lp-code> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1050191 >
<wgrant> StevenK: It's on my chopping block for the next few days
<wgrant> As it's directly implicated in the BranchRevision drama
<StevenK> Right
<StevenK> I'll leave it alone then
<wgrant> Fixing it is probably a prereq, but in which way I'm not sure
<wgrant> Right :)
<StevenK> Is mawson back to crying about your queries?
<wgrant> No
<wgrant> I have the data I need for now
<wgrant> You can molest mawson if you so desire
<StevenK> I have no need to, I was curious.
<wgrant> Oh right, you qa-meh'd the PCJ thing
<StevenK> I did, yes.
<StevenK> Bug 1031751 is interesting. I've seen a test fail in that way too
<_mup_> Bug #1031751: KeyError: 'primary_vars'  raised setting branch for a project <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1031751 >
<wgrant> Right, some of Orange's work triggered that a week or so ago
<wgrant> An unrelated cause, but similar exception
<StevenK> Yeah. I'm not sure about the cause for the bug
<StevenK> Almost at 45 revisions needing deployment
<wgrant> StevenK: I suspect a branch was being created with a private team as the registrant
<wgrant> StevenK: Perhaps registering a codeimport
<wgrant> But the OOPS is gone and the traceback severely truncated
<wgrant> Care you try what I suggested on qastaging?
<StevenK> +new-import with a private team?
<wgrant> That shouldn't set the registrant to the primary team, but it might
<wgrant> So it's worth a try
<StevenK> PrivatePersonLinkageError: Cannot link person (name=rhinos, visibility=PRIVATE) to <CodeImport for ~rhinos/launchpad/test-import> (name=None)<br />
<wgrant> Indeed
<wgrant> I can't see an obvious way to set the registrant to a private team
<wgrant> And you can't log in as a team any more, so that won't work...
<wgrant> Oh, hmmm
<wgrant> That OOPS would make it obvious :(
<lifeless> .oOo.
<wgrant> StevenK: Ah, it would have helped if I'd read the summary
<wgrant> OOPS-07a7ee485f0ae949f3e103cc1db1c082
<StevenK> Nice.
<StevenK> What was the cause, then?
<StevenK> Hmmmm, I think I get it. Creating an import for a series owned by a private team
<lifeless> is sso glitching
<lifeless> I'm having trouble logging into oops.c.c
<wgrant> lifeless: It's working for me
<lifeless> I'm getting The connection was reset
<lifeless> on
<lifeless> https://oops.canonical.com/openid/+return?janrain_nonce= ...
<wgrant> StevenK:                     code_import = getUtility(ICodeImportSet).new(
<wgrant>                         registrant=branch_owner,
<wgrant> From ProductSeriesSetBranchView.update_action
<StevenK> wgrant: So we should croak in validate if branch_owner is private?
<wgrant> StevenK: No
<wgrant> You should specify the registrant
<StevenK> Oh, registrant=self.user ?
<wgrant> Yeah
<StevenK> wgrant: But I thought the form was asking who should own the branch?
<StevenK> So isn't that ignoring them ...
<wgrant> own != register
<wgrant> So owner != registrant
<wgrant> I think you'll find that all the other branch creation callsites use the user as the registrant, with the provided owner as the owner.
<wgrant> That's certainly the intent.
<wgrant> I suspect that Registry got confused when producing this view
<StevenK> Right
<StevenK> Ugh, all of the tests for it are in doctests
<StevenK>  I chose SHA-512 because everyone knows it's 512 times more secure than SHA-1.
<StevenK> â Rusty Russell
<wgrant> :)
<StevenK>         obj_info["primary_vars"])
<StevenK>     KeyError: 'primary_vars'
<StevenK> Fantastic
<wgrant> In a test?
<wgrant> I wouldn't have a test for this issue directly.
<wgrant> Rather amend an existing test to check that the registrant is the user, not the owner
<wgrant> Making the branch approximately -1/+2
<StevenK> But I just wrote one :-(
<wgrant> StevenK: It may be useful for checking that the issue is really gone, but it's not something that's worth the test suite time and LoC to keep forever
<StevenK> Raaaargh
<StevenK> This entire test uses the product.owner as the registrant as well as the logged in the user
<wgrant> Change one of them to be the other, I guess
<StevenK> Changing the whole test to make a driver and set them as the owner of the series has lots of ('Invalid value', InvalidValue("token u'person-name-100001' not found in vocabulary"))
<wgrant> StevenK: Ensure that the user is a member of the team
<StevenK> What team
<StevenK> ?
<wgrant> You can only create a branch owned by yourself or a team in which you participate
<wgrant> So the owner has to be a superteam of the user
<StevenK> Right, large diff
<StevenK> wgrant: Did you want to review?
<wgrant> Sure
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/no-private-registrant-setbranch/+merge/128173
<jpds> StevenK: Do you use ROT26 from time to time?
<StevenK> ROT26 is the alphabet?
<adeuring> good morning
* adeuring changed the topic of #launchpad-dev to:  http://dev.launchpad.net/ | On call reviewer: adeuring | Firefighting: - | Critical bugs: ~300
<rick_h_> morning
<czajkowski> rick_h_: hey there
<czajkowski> you're on early
<rick_h_> czajkowski: trying to get back on time
<czajkowski> rick_h_: can you pop into #lp please if you're free?
<rick_h_> adeuring: ping, can you help czajkowski debug a blueprint issue? https://oops.canonical.com/?oopsid=OOPS-51c8161b2e479d7ed62ba67c1000bd19
 * adeuring is looking
<rick_h_> adeuring: looks like a user hit a forbidden setting it to embargoed or something.
<adeuring> 383264
<rick_h_> adeuring: ? typo'd bug number or something?
<adeuring> rick_h_: PEBCAK: I tend to forget to click into a wondow before typing something. I this I wanted to add a 2-fact auth number into a browser window, while irc IRC window still had the focus...
<rick_h_> adeuring: ah gotcha cool
<rick_h_> thanks for looking at the oops
<adeuring> rick_h_: https://bugs.launchpad.net/launchpad/+bug/1062207/comments/1
<_mup_> Bug #1062207: Unable to raise blueprint <Launchpad itself:New> < https://launchpad.net/bugs/1062207 >
<rick_h_> abentley: I missed you yesterday about setting the field value = PUBLIC
<rick_h_> abentley: that was my doing when I setup the UI because I needed to set a value to the form field so that choicepicker stuff would pick it up
<rick_h_> now that the field exists, model sets, etc it can probably just go away
<deryck> adeuring, thanks for the review.  Agree on both counts.  We can hold the individual adapters until after alpha though, I think.
<adeuring> deryck: right
<abentley> rick_h_: I've removed it, and there seem to be no ill effects.
<rick_h_> abentley: right
<rick_h_> and the IProductSet checks were because there were tests that hit that View that weren't of that interface and weren't getting information_type added onto it
<rick_h_> abentley: I had talked to deryck about that one when I thit it
<abentley> rick_h_: Strange.  We don't seem to have any tests like that now.
<rick_h_> abentley: it was something I hit. I had a chat with deryck about it asking if there was a better way than the interface checks
<rick_h_> abentley: ProjectGroup perhaps gets hit in there?
<abentley> rick_h_: I don't understand.
<rick_h_> abentley: I'll bring it up on the call in a sec
* bac changed the topic of #launchpad-dev to:  http://dev.launchpad.net/ | On call reviewer: adeuring, bac | Firefighting: - | Critical bugs: ~300
<abentley> After 3 years, I think I'm starting to understand buildbot's waterfall display.
<jelmer> abentley: lol
<abentley> jelmer: It's funny 'cause it's true.
 * deryck switches work locations, back online shortly
<sinzui> adeuring, bac, abentley: do either of you have time to review https://code.launchpad.net/~sinzui/launchpad/contact-new-maintainer/+merge/128273
<adeuring> sinzui: i'll look
<adeuring> sinzui: if the maintainer is a single person, not a team, user_address in LicenseNotification.send(), while it is a list of strings if the maintainer is a team. shouldn't a one-element list be used for "person is maintainer" case too?
<adeuring> sinzui: never mind, i see now taht a single address worked before
<sinzui> adeuring, indeed that is confusing. I had to read the code of our send mail helper to see that it would work
<adeuring> sinzui: right, I an often a bit worried when a function(method argument can have different types. ANyway, r=me
<sinzui> thank you adeuring
<rick_h_> adeuring: branch for you if you get a sec, I noted a 'known issue' for the backend stuff that needs updating will be coming in a second branch. https://code.launchpad.net/~rharding/launchpad/edit_product_info_type/+merge/128081
<adeuring> rick_h_: I'll look
<abentley> adeuring: Do we have an equivalent for get_specification_privacy_filter that works for product listings?
<adeuring> abentley: not yet...
<adeuring> rick_h_: r=me
<rick_h_> adeuring: ty much
<deryck> rick_h_, abentley -- I'm going to have to leave here shortly, a little earlier than expected.
<deryck> rick_h_, abentley -- but I'll obviously be around tonight and tomorrow as I finish tests and get my last branch landed.
<rick_h_> deryck: rgr
<abentley> rick_h_: I'm having trouble with a DB query-- getting duplicate results back.  Can you have a look at it with me?
<rick_h_> abentley: sure thing
<abentley> rick_h_: let's hangout
<rick_h_> abentley: rgr
<rick_h_> abentley or bac if you're still around. Would appreciate a review of https://code.launchpad.net/~rharding/launchpad/store_edit_product_info_type/+merge/128288
<abentley> rick_h_: looking.
<rick_h_> abentley: ty much
<abentley> rick_h_: Where is _setLicense being invoked?
<rick_h_> abentley: it's set by the ProductEditView when all the fields are updated.
<rick_h_> abentley: there's an assumption there because changing the information_type changes the license I guess...
<abentley> rick_h_: I think this change leaves a hole in the validation, because it allows information_type to change without setLicense being invoked.
<rick_h_> abentley: yea, right
<abentley> rick_h_: Can we do setLicense first?  Or add the subscription in _set_information_type?
<rick_h_> abentley: thinking, really hate to call setLicense from set_information_type
<abentley> rick_h_: Not set_license, but adding the subscription.
<rick_h_> so maybe the best thing is to try to pull out the block that does the commerical sub generation and share it between teh two
<rick_h_> what about a check in set_information_type that if the information_type is EMBARGOED/PROPRIETARY it sets the license manually there, overrides so that it makes sure it's getting the OTHER_PROPRIETARY value
<abentley> rick_h_: That would make sense.  Either the license or the information_type could be seen as proprietary intent.
<rick_h_> yea, it's just more a functino of license so trying to think of a good way to keep it there
<abentley> rick_h_: r16102 provides make_product_form which should simplify your code.
<rick_h_> the root issue is you can set the information type, and the license might not kick in.
<rick_h_> abentley: cool, thanks for the heads up on that.
<rick_h_> abentley: but can the license field ever not be supplied? It's a required field of data
<abentley> rick_h_: The view code should never be responsible for the sanity of the data.  Especially given we have a web service API.
<rick_h_> abentley: yea, sorry I thought it was required in the model via the validate method, but that's just worried about resetting the license review
<abentley> rick_h_: So, I think the subscription is the key thing.  The license doesn't matter for information types.  A product could have an open source license, but not be announced yet.  Which is fine, as long as they're paying for the right to secrecy.
<rick_h_> abentley: yea, so I'll pull that out into a new create_complimentary_subscription method and call from both places. I need to make sure it'll be safe to run twice
<rick_h_> since it could get called via both the license setter and the information type setter
<abentley> rick_h_: Sounds good.  We seem to be calling such methods "ensure_FOO", rather than create_ or make_.
<rick_h_> abentley: not sure ensure fits here like it does fixing policies
<rick_h_> it's just creating an initial compl. commercial sub
<abentley> rick_h_: It's like it in the sense that it doesn't try to create things that already exist.
<rick_h_> ok, fits my head strange but I can buy in :) Pushed an update to the MP when it processes
<rick_h_> bah, I need to update the test though to verify the subscription exists as well
<abentley> rick_h_: You don't need line 116 because it duplicates line 148.
<rick_h_> abentley: ah true thanks.
<abentley> rick_h_: I would do http://pastebin.ubuntu.com/1262684/ but I believe your code's equivalent.
<rick_h_> abentley: thanks for the suggestino
<abentley> rick_h_: r=me either way, but I'm surprised you don't have to change the view in order to solve the bug.
<rick_h_> abentley: the view was in the first branch to get the field to show up
<rick_h_> abentley: it's listed as a pre-req to this MP
<abentley> rick_h_: Gotcha.
<rick_h_> thanks for the late day review abentley
<abentley> rick_h_: np
<abentley> rick_h_: still before my EOD.
<rick_h_> still, late Friday bonus reviews yay ...not :)
<abentley> rick_h_: So thanks for the advice on debugging the privacy filter.  It turned out that the And() needed to be in a sub-select, because that Product._information_type == InformationType.PUBLIC evaluates true for most rows.
<rick_h_> abentley: cool, glad it worked out. Is that grantee thing still an issue or was that nothing?
<abentley> rick_h_: The grantee thing was not really part of it.  It was just generating rows where AccessPolicyGrantFlat.grantee_id != TeamParticipation.teamID but Product._information_type == PUBLIC, so such rows were being yielded in the result.
* bac changed the topic of #launchpad-dev to:  http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300
#launchpad-dev 2012-10-07
<wgrant> wallyworld_: Morning
<wallyworld_> hello
<wgrant> wallyworld_: Not sure if you saw, but we both landed testfixes yesterday morning
<wgrant> We should probably revert one of them
<wallyworld_> yes :-( i wish you had let me do it
<wallyworld_> we should get a deployment done first perhaps to clear the backlog
<wgrant> Well, I waited 10 hours after the initial failure so decided you were probably doing something more weekendy during the weekend :P
<wgrant> We probably don't have webops until the UK wakes up
<wallyworld_> i did it before i went to bed. you can't expect me to do it during the night
<wallyworld_> i did it when i woke up
<wallyworld_> it wasn't your issue to fix
<wgrant> Hm?
<wgrant> Build failures are *everyone*'s issue to fix
<wgrant> That's why they block landings
<wallyworld_> sure, but i caused it so i was fixing
<wgrant> And I had a one-liner trivial fix that would mean you wouldn't have to care, so I thought I'd help you out and unbreak it
<wgrant> Sorry
<wallyworld_> me too.  i did check and i didn't see anyone else landing anythng
<wallyworld_> so we sort of did it at the same time.
<wgrant> Yeah, you were only about half an hour later
<wgrant> Your fix is a little more comprehensive, but I think mine is more in the style that we've used elsewhere.
<wgrant> I'm not sure which wins
<wallyworld_> i reverted the caching because it can have unintended issues
<wallyworld_> but the one liner may be worth sticking with
<wgrant> Right
<wallyworld_> i do dislike how we put sql inside properties
<wgrant> To be completely correct we'd also need to invalidate on deletions and milestone retargetings
<wgrant> But I think those are probably OK to not care about here
<wallyworld_> yeah
<wallyworld_> my solution means that everything else except my use case uses the old way
<wallyworld_> there's a bigger design issue here though
<wallyworld_> which is out of scope for right now
<wallyworld_> so, we need to divvy up the bvm removal work
<wallyworld_> bvp
<wgrant> There's actually not that much to remove
<wgrant> private_bugs needs to go too
<wgrant> I forget if StevenK's here this week
<wallyworld_> not sure
<StevenK> I was going to slaughter private_bugs
<StevenK> It's next week I'm off
<wgrant> Ah, right
<wallyworld_> holiday?
<StevenK> I'll be in Adelaide from Wednesday until the following Monday
<wgrant> Wow, Deployment-Ready is looking a bit sad
<wallyworld_> wineries :-)
<wgrant> So, BVP and private_bugs are already limited to well-isolated chunks of code.
<wgrant> We'll want one person on each, I suspect
<wgrant> And then a cleanup pass afterwards, particularly around BranchNamespace and co
<StevenK> Right
<StevenK> private_bugs can't die completly
<StevenK> For API compat :-(
<wgrant> Indeed
<wgrant> But it becomes 'private_bugs = None'
<wgrant> Er, False, I guess
<wallyworld_> wgrant: i qa-ed that last remaining bug so we can deploy. i assume everyone's back on board this week?
<wgrant> wallyworld_: Pretty much, though I suspect we'll have no APAC today
<wallyworld_> too bad
<StevenK> It's the fly-home-from-London usual bug
<StevenK> You leave on Saturday afternoon and land Monday morning
<wgrant> Ouch
<wgrant> Lots of new bugs
<StevenK> OMG, private_bugs is everywhere
<bigjools> morning daylightsaverers
<wgrant> StevenK: Shouldn't be
<wgrant> StevenK: In FileBugGuidedView, a few bits in the Product model and forms
<wgrant> Hm, maybe one bit in person as a precursor to the commercial-owning check?
<wgrant> Yeah
<StevenK> % bzr st | wc -l
<StevenK> 11
<StevenK> So far
<StevenK>  10 files changed, 20 insertions(+), 268 deletions(-)
<StevenK> lib/lp/bugs/scripts/bugimport.py:        # If the product has private_bugs, we force private to True.
<StevenK> wgrant: ^
<StevenK> For example
<wgrant> Oh right, that one
<wgrant> I never quite worked out what to do with that
<wgrant> but still
<wgrant> wgrant@lamuella:~/launchpad/lp-branches/devel$ bzr grep -l private_bugs | grep -v -E '/(tests|doc|stories)/' | grep -v ^database | wc -l
<wgrant> 9
<wgrant> There's not much there
<wgrant> bigjools: Are you glad your curtains are unfaded, and your cows remain undistressed?
 * bigjools blinks
<bigjools> do I detect a hint of resentment of DST ...
<wgrant> DST is stupid, but the classical arguments against it are too :)
<bigjools> FWIW I wish we had it here
<wgrant> Aw
 * wallyworld_ likes DST too
<bigjools> sun comes up at 5:20am and sets at nearly 6pm.  Would make more use of evening light.
<wallyworld_> anyone know how to debug a unity launcher fu? ever since latest update, i can't some some apps eg konsole
<bigjools> although at 38c like it was yesterday, hiding in my  basement makes no difference :)
<StevenK> wgrant: Given I just ripped out the only use of private_project in lib/lp/bugs/model/tests/test_bug.py, I guess one of TestBugPrivateAndSecurityRelatedUpdatesP{rivate,ublic}Project can die too
<wgrant> 38!?
<wgrant> It never gets to 38 in October there, does it?
<wallyworld_> it does with global warming
<bigjools> lol
<StevenK> It hit 34 here last week
<wgrant> It was 18 here :P
<bigjools> I wonder why when temps are below averge nobody ever goes "global cooling!"
<wallyworld_> because there's a monotonic, systemic warming going on
<bigjools> wrong, temps constaint globally for 10+ years
<bigjools> constant*
<wallyworld_> my data is bigger than your data
<bigjools> leave it zipped
<StevenK> I think I'm done.
<StevenK>  19 files changed, 32 insertions(+), 524 deletions(-)
<wgrant> Plausible
<StevenK> wgrant: private_bugs still appears 79 times in the tree -- it still has to be eradicated from database/, it's still in IProduct, and lots of tests that play with private bugs use private_bugs in their function name.
<wgrant> Yeah
<StevenK> I may have deleted too much, but that's why we have version control.
<StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/death-to-private_bugs/+merge/128398
<wgrant> StevenK: Looking
<wgrant> Note that we shouldn't land this quite yet, until we've confirmed that everything has both set
<wgrant> eg. newyork is still not done
<StevenK> Oh sure
<StevenK> I was going to throw it at ec2 test and see how much carnage there is.
<StevenK> Who's killing BVP?
#launchpad-dev 2013-09-30
<StevenK> wgrant: My only concern with that branch was:
<StevenK> 291	- u'There is no project in Launchpad named "N\xf6 Such Product&amp;&lt...]
<StevenK> 294	+ There is no project in Launchpad named "N... Such Product&amp;&lt...
<StevenK> I couldn't get print_feedback_messages() to show me \x..., it always printed the codepoint
<wgrant> That's correct.
<wgrant> Because it prints the string, not its repr.
<StevenK> Right
<StevenK> wgrant: Hm, I seem to have tripped over https://code.launchpad.net/~stevenk/launchpad/drop-kick-getBugTasks/+merge/74941 ; getBugTasks isn't in the monthly PPR
<wgrant> StevenK: What do you mean?
<StevenK> wgrant: I had forgotten all about that MP -- looking at the latest monthly PPR, I can't see getBugTasks at all, but it's still exported
<wgrant> Ah, didn't notice it was old.
<wgrant> If it's not in the PPR that shows the other methods, then it wasn't called at all.
<StevenK> wgrant: As long as pageids is the right page to be searching
<wgrant> StevenK: I believe so. Do the other methods appear there?
<StevenK> IDistribution.getBranchTips shows up as Distribution:EntryResource:getBranchTips, so looks like. (DSP doesn't have any other exported methods, but has other exported properties)
<wgrant> Right
<StevenK> It was called 38 times in March, that is the only time I've seen it
<StevenK> But other monthly PPRs have been eaten
<SpamapS> Not sure anymore where to ask, but is there any way we can get a user unsubscribed from bugs on the heat project? Their out of office mail is spamming every change we make.
<lifeless> wgrant: StevenK: ^ halp.
<wgrant> Have you adequately slapped the user in the face for having a moronic autoresponder?
<wgrant> Which user?
<lifeless> SpamapS:
<SpamapS> wgrant: https://launchpad.net/~faramir
 * SpamapS is quite certain this, as most problems, is Lotus Notes' fault.
<wgrant> SpamapS, lifeless: lp-shell production devel; lp.load('/heat/+subscription/faramir').lp_delete()
<wgrant> The project owner should be able to do that.
<SpamapS> wgrant: maintainer? as in, ~heat-drivers?
<wgrant> Yes
 * SpamapS tries it
<SpamapS> wgrant: lovely, thank you. subscription removed.
<wgrant> Excellent.
<wgrant> Do stab the user and/or their email admins as well, though. :)
<StevenK> wgrant: So, do you think that MP is landable?
<wgrant> StevenK: I don't see why not.
<StevenK> The only thing I can come up with is it drops the method from 1.0
<StevenK> But we probably don't care
<wgrant> If it's not called frequently then we do not care.
<SpamapS> wgrant: right, slapping subscribed. :)
<cjwatson> Ursinha-afk: Just having a go at your QA now in order to unblock myself
<cjwatson> Copied https://dogfood.launchpad.net/%7Eubuntu-unity/+archive/daily-build/+sourcepub/3476923/+listing-archive-extra to the primary archive on DF
<cjwatson> wgrant: I restored the dogfood-publish config and its associated paraphernalia on labbu
<cjwatson> Ursinha: This looks like a failure, I'm afraid - notes in the bug
<Ursinha> cjwatson, will look
<cjwatson> I wonder if this is DF-specific, since I thought you mostly just moved that code around ...
<Ursinha> (sorry the delay in the answers, my ISP is not helping today)
<cjwatson> Ah, but I guess that code never had to deal with copies from a PPA before
<cjwatson> Odd, though, since _publishCustom uses self.packageupload too and it's clearly never None in those cases
<cjwatson> So the PackageUploadCustom mustn't be being constructed in the right way, but why ...
<Ursinha> cjwatson, yeah, I recall talking to wgrant about such cases and that never should be none
<Ursinha> when I was transforming that code in a job
<cjwatson> And for that matter how
<cjwatson> I don't see anything other than PackageUpload.addCustom that constructs new PackageUploadCustom objects, and that sets packageupload=self
<cjwatson> But the whole publisher looks kind of unhappy
<Ursinha> cjwatson, have you tried to copy any other kind of package other than from a PPA?
<cjwatson> No
<cjwatson> That's the only case that matters though :)
<Ursinha> cjwatson, yeah :/
<cjwatson> In fact this makes zero sense because the PUC is surely looked up from the PU in the first place
<cjwatson> Oh, they have a PU but no SPR, I misread
<cjwatson> Ursinha: OK, so PackageUploads that refer to copies (PackageCopyJobs) don't have a sourcepackagerelease - only direct uploads do
<cjwatson> Ursinha: I think you need to do the same kind of things that PackageUpload.package_name and friends do, and look at either bits of packageupload.package_copy_job or packageupload.sourcepackagerelease
<cjwatson> Ursinha: And of course have a test that exercises this in the context of copies
<Ursinha> cjwatson, yeah.. I should have tested the ppa copies, I felt that was missing after submitted to pqm
<Ursinha> I'm into it
<cjwatson> Cool - I think you can use package_copy_job.target_distroseries (I think) or package_copy_job.component_name
<cjwatson> For the various cases
<cjwatson> But then you have to do something with the PackageTranslationsUploadJob creation ...
<Ursinha> cjwatson, right, thanks for the huge help
<Ursinha> yeah, I'll have to adapt it all
<cjwatson> But I guess you know what to do there :)
<Ursinha> :)
<cjwatson> Yikes, attachTranslationFiles lives on SPR
<cjwatson> I guess maybe there needs to be a similar thing for PCJ?  Not sure
<Ursinha> cjwatson, hm. that sounds ugly but might be the way to go
<Ursinha> as pu doesn't have a spr it doesn't make sense to use the one it's there for such cases... but I'll check if there any other (less intrusive) way
<cjwatson> I think you have to go through PCJ - I remember dealing with this for the queue API exposure
<cjwatson> Ursinha: (see also my comments in https://bugs.launchpad.net/launchpad/+bug/851562, tangentially)
<_mup_> Bug #851562: Diffs not available for syncs on +queue page like for regular uploads <derivation> <package-copies> <package-diff> <Launchpad itself:Triaged> <https://launchpad.net/bugs/851562>
<Ursinha> cjwatson, yep, I was reading that right now
<Ursinha> cjwatson, I couldn't understand well in the call, internet is crappy, are you working on fixing that?
<cjwatson> Ursinha: Not actively
<cjwatson> Ursinha: I don't think it's a blocker for this, just context
<cjwatson> You don't really *need* the SPR here, it's just that that's where the code currently lives and it uses some attributes from it
<Ursinha> cjwatson, that's okay, I ask because I could look at that next, as the context is similar
<cjwatson> Awesome
#launchpad-dev 2013-10-01
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bi-dependencies-1/+merge/188489 and https://code.launchpad.net/~wgrant/launchpad/bi-dependencies/+merge/188493
<StevenK> Ursinha, wgrant: What do you think about reverting r16775?
<wgrant> The fix is probably very simple
<wgrant> Not sure how Ursinha's going with that, though.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/drop-kick-getBugTasks/+merge/188490
<StevenK> wgrant: Oh, is that a minus one, as opposed to dash one?
<wgrant> StevenK: k
<wgrant> StevenK: Er, yeah, I missed a -
<Ursinha> wgrant, it depends on how much you guys need to deploy new stuff today
<wgrant> Was meant to be dash-minus-1
<StevenK> Ursinha: It would be nice to deploy, but we can wait a bit
<StevenK> Ursinha: I have my own QA to finish off, so please don't rush on my account. :-)
<Ursinha> okay
<StevenK> 471+ self.assertEqual(('snap crackle pop', ''), output)
<StevenK> wgrant: You must answer for your crimes.
<wgrant> StevenK: Hey, the test had 'pop' in the final position, and there was a hole in the test coverage for buildd_name, so the test string ended up with three elements, 'pop' in the last. My hands were tied.
<StevenK> wgrant: Tell it to the judge.
<StevenK> wgrant: r=me for -1, with one niggle
<Ursinha> wgrant, so, PackageUploads that refer to copies don't have a SourcePackageRelease, and attachTranslationFiles is a spr method
<Ursinha> should I move it somewhere else? or add one for copies?
<wgrant> Ursinha: It might be worth moving somewhere else, but you will still need to find a SourcePackageRelease.
<wgrant> I suspect.
<wgrant> Unless you can just use the source name, let me see.
<wgrant> Hmm
<wgrant> Ursinha: I'd move attachTranslationFiles elsewhere, yeah. It doesn't really need a SourcePackageRelease; you have all you need on the PackageCopyJob too.
<wgrant> I'd just put it in a non-method function somewhere, I think.
<wgrant> StevenK: Thanks.
<Ursinha> wgrant, what I did so far is to add a component_name and target_distroseries properties in PackageUpload, following the pattern of other properties that vary according to them being copies or not
<wgrant> Ursinha: PackageUpload.distroseries already exists
<Ursinha> wgrant, I saw that, but it made no sense to me that other parts of the code query sourcepackagerelease to know the target
<wgrant> But you'll need to look up the component name from the relevant SourcePackagePublishingHistory in the target. I'm not sure if there are existing methods to help with that.
<Ursinha> so I thought this might mean something else
<wgrant> Ursinha: It used to make sense, sort of.
<Ursinha> haha
<wgrant> This code was already buggy in some copy circumstances.
<Ursinha> can I safely use PackageUpload.distroseries as target then?
<wgrant> Yes.
<Ursinha> okay.
<wgrant> PackageUpload.archive, PackageUpload.distroseries, PackageUpload.pocket are accurate
<wgrant> But you can't use sourcepackagerelease.component. You'll probably need to find the latest SPPH for the relevant (archive, distroseries, pocket, sourcepackagename), and look up the component from that.
<wgrant> That component lookup was already buggy for some security updates, but it is a lot more important now.
<Ursinha> wgrant, so it was buggy before?
<wgrant> Yes, for source copies.
<wgrant> This is probably actually more of a problem than I thought.
<Ursinha> hehe
<wgrant> Or maybe not.
<wgrant> The component check only applies to pre-oneiric, so its brokenness isn't too bad.
<wgrant> But it's still only a few lines to fix, so we probably should.
<wgrant> StevenK: Do you have a testfix?
<Ursinha> wgrant, should I use the SPR.publishings or query the archive with getPublishedSources?
<wgrant> Ursinha: You'll have to use getPublishedSources.
<StevenK> wgrant: Looking now, I was grabbing lunch.
<StevenK> Sounds like a few more deletions are needed.
<StevenK> wgrant: testfix == r16787
<StevenK> wgrant: With the death of proc_families, should I QA to packagecloner on DF?
<StevenK> s/to/the/
<wgrant> StevenK: Yeah. Can hopefully do just a small PS.
<StevenK> IDS{,J} got the same treatment, but I don't really want to do that.
<wgrant> StevenK: IDSJs are easy and cheap to test
<wgrant> StevenK: Just create one with a tiny PS
<wgrant> I normally create a Âµbuntu containing only bzr packages, IIRC
 * StevenK stabs SSO until it bleeds to death
<wgrant> Why?
<StevenK> wgrant: Because of the stale page error if you mis-type or don't hold down the yubikey long enough and it puts the default token in, and because once it logs me in it wants me to update my details and there's no way to say "Go away and auth to the site I was originally at"
<wgrant> Ah, handy
<wgrant> Filed a bug?
<StevenK> Not yet
<Ursinha> wgrant, is that ok if I move the content of attachTranslationFiles to run() in the PackageTranslationsUploadJob? having a packageupload at that point is enough (it seems), so I'd provide the packageupload and the libraryfilealias to the job
<wgrant> Ursinha: It might be useful to have it in a separate function in the job's module so you can continue to test it directly. But otherwise that sounds fine.
 * StevenK tries to remember how to create an IDSJ
<wgrant> StevenK: Create a new distro, create a series within it, initialise series.
<StevenK> Hm
 * StevenK anoints himself
<Ursinha> wgrant, okay, done :) now, the importer is currently sourcepackagerelease.creator, who should it be if it's a copy?
<StevenK> PCJ.requester
<StevenK> Or something close
<wgrant> Probably PU.findPersonToNotify
<wgrant> Which uses PCJ.requester when appropriate
<wgrant> I think we already use that for the job creator, don't we?
<StevenK> lib/lp/soyuz/model/packagecopyjob.py doesn't mention findPersonToNotify
<wgrant> Not PCJ, PTUJ.
<StevenK> lib/lp/soyuz/model/packagetranslationsuploadjob.py also doesn't mention findPersonToNotify
<StevenK> And findPersonToNotify wants an SPR, it doesn't deal with PCJs
<wgrant>         blamee = packageupload.findPersonToNotify()
<wgrant>         getUtility(IPackageTranslationsUploadJobSource).create(
<wgrant>             sourcepackagerelease, libraryfilealias, blamee)
<wgrant> Ah right
<wgrant> At this point we have an SPPH, though
<Ursinha> wgrant, attachTranslationsUpload uses the requester, it's another thing
<wgrant> So it'll use SPPH.creator, which should be the PCJ requester.
<Ursinha> I recall the same confusion during the sprint
<Ursinha> it rendered a testfix that time
<Ursinha> :)
<StevenK> Grr.
 * StevenK pokes around for pubconf
<wgrant> Ursinha: Use PTUJ.requester, rather than SPR.creator.
<wgrant> StevenK: Pubconfs can be set in the UI
<StevenK> wgrant: But the details don't matter?
<wgrant> StevenK: Not particularly, unless you want to publish it.
<StevenK> The initialization of Turgid Turtle has been scheduled and should run shortly.
<StevenK> I blame infinity
<wgrant> ...
<wgrant> Âµbuntu
<wgrant> Not microbuntu!
<wgrant> We aren't Fedora
<wgrant> Our infrastructure likes Unicode.
<StevenK> wgrant: It didn't for the name
<wgrant> Not for the name, no. :(
<StevenK> wgrant: How about that, then
<wgrant> Superior.
<StevenK> wgrant: Hm, where do OOPSes go? :-(
<wgrant> StevenK: Should be /srv/launchpad.net/dogfood-logs/
<StevenK> ProgrammingError: permission denied for relation packageupload
<StevenK> WAT
<wgrant> Oh
<wgrant> I was trying to remember why I had that diff on DF
<wgrant> That was it
<StevenK> So it was your fault? :-(
<wgrant> Not exactly, I just locally patched that permission in last time I was testing this
<wgrant> Because it had broken due to PCJ changes, IIRC.
<wgrant> Specifically CUC
<wgrant> I think
<StevenK> Do we want to land a security.cfg change?
<wgrant> Test on DF and land, yeah
<StevenK> Yup
<StevenK> Digging out the job so I can mangle it back to Waiting
<wgrant> I'd just create a new one.
<StevenK> Hm, IDS has SELECT on packageupload*
<StevenK> Ah
<StevenK> It wants INSERT on it and PUC
<StevenK> OH
<StevenK> And UPDATE
<StevenK> Damn searchable_names :-P
<StevenK> wgrant: https://dogfood.launchpad.net/microbuntu/turgid
<StevenK> wgrant: That's IDSJ run, and then harness'ing up updatePackageCount()
<wgrant> eh
<wgrant> Heh
<wgrant> So you know how I said our Unicode support was good
<wgrant> https://dogfood.launchpad.net/microbuntu/turgid/+source/2ping
<StevenK> HAHA
<wgrant> Fortunately Python 3 prevents people from writing such retarded code.
<StevenK> wgrant: How tasty are those words you're now eating?
<wgrant> Not entirely.
<StevenK> wgrant: Landing security.cfg change
<lifeless> turgid eh
<StevenK> lifeless: It was infinity's suggestion for what T would be
<StevenK> And it's a crappy test series on DF, so meh? :-)
<lifeless> I think you've been trolled
<StevenK> wgrant: Are you bouncing the appserver?
<wgrant> I am
<wgrant> It didn't come back up...
<StevenK> Haha
<StevenK> Hm, that looks better
<StevenK> wgrant: Facepalm at the diff
<wgrant> It won't affect many places
<wgrant> Because .format is backported from 3, it is slightly more Unicode-pedantic.
<wgrant> % works fine
<StevenK> wgrant: So I'm happy enough to qa-ok it, any objections?
<wgrant> Sure
<wgrant> As long as you've tested the other changed bits
<StevenK> wgrant: IArchive API bits, editing an archive and changing restricted processors, creating a distribution and doing the same, IDSJ and package cloner
<wgrant> Right
<wgrant> Sounds reasoanble.
<StevenK> I don't think there is anything else
<StevenK> wgrant: Is http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/892/steps/shell_9/logs/summary your fault?
<wgrant> StevenK: I don't believe so.
<wgrant> I haven't touched BuilderSlave, and that one has been occasionally failing 4eva.
<Ursinha> cjwatson, I'm going to take a nap and should return soon, hopefully the fix for bug 1201485 will be able to land today
<_mup_> Bug #1201485: Need to import translations for the unity daily builds <qa-bad> <langpack-o-matic:Triaged> <Launchpad itself:In Progress by ursinha> <Ubuntu Translations:Triaged> <https://launchpad.net/bugs/1201485>
<cjwatson> OK, great
<cjwatson> Happy to help reviewing
<lifeless> wgrant: https://answers.launchpad.net/launchpad/+question/236697 when you get a chance
<lifeless> wgrant: some of them I can do and am doing
<lifeless> wgrant: but I think a little help will be needed
<wgrant> lifeless: Do you just need the tuskar -> tripleo merge done?
<lifeless> wgrant: and python-tuskarclient reowned
<wgrant> lifeless: You can't ask the current owner to do that?
<lifeless> wgrant: sure, they're at an offsite teambuilding thing today :P
<lifeless> wgrant: but one thing at a time; the team merge first
<wgrant> lifeless: tuskar has a mailing list.
<lifeless> gnar
<lifeless> ok, I'll just manually add the people I guess
<StevenK> lifeless: API it?
<wgrant> lifeless: Note that it still owns a couple of trunks, I think.
<lifeless> wgrant: meh, mirrors from git
#launchpad-dev 2013-10-02
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bi-uB-no-bfjb/+merge/188766
<StevenK> wgrant: r=me
<wgrant> StevenK: Thanks
#launchpad-dev 2013-10-03
<wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/ss-get-builder-as-needed/+merge/188982
<StevenK> wgrant: r=me
<StevenK> wgrant: Landing the DB patch you approved yesterday
<StevenK> wgrant: Test failure is legimate?
<wgrant> No
<wgrant> That's the usual spurious one.
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/spec-notification-respect-sharing/+merge/188988
<wgrant> StevenK: I'd add an extra assertion to the test to verify that the drafter appears before the revocation.
<StevenK> wgrant: http://pastebin.ubuntu.com/6186821/
<wgrant> StevenK: Rgiht.
<StevenK> wgrant: MP updated
<StevenK> I love how fast that is now.
<wgrant> It's still really slow, mostly because of the scanner.
<wgrant> StevenK: r=me
<stub> I've been backporting, and been in awe how fast my recipes have been going through too.
#launchpad-dev 2013-10-04
<StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/fix-linkify-email-multiple-icons/+merge/189212
<StevenK> cjwatson: I fixed your multiple icon changelog bug: https://code.launchpad.net/~stevenk/launchpad/fix-linkify-email-multiple-icons/+merge/189212
<cjwatson> Ta
<StevenK> cjwatson: Can haz review?
<cjwatson> Oh, right, sorry, still waking up :)
<StevenK> cjwatson: :-)
<cjwatson> StevenK: The expected_html looks weird - why isn't the <<a href="...">foo@example.org</a>> business a bug?
<cjwatson> There's probably a more efficient algorithm that collects all the distinct e-mail addresses first and then does a single bulk query for all the ones it doesn't have preloaded, but that isn't a problem introduced by your patch
<cjwatson> (In fact I bet that contributes to +changelog slowness)
<cjwatson> Oh, I think it's just confusing input data, will leave a comment
<cjwatson> r=me
<StevenK> cjwatson: The function already allows you to stuff in a preload
#launchpad-dev 2015-09-29
<wgrant> cjwatson: If you're still around, could you please look at https://code.launchpad.net/~wgrant/launchpad/bug-1500576/+merge/272676?
<cjwatson> wgrant: Right, I'd wondered if that was going to require ILaunchBag in the model, but when it was reported it was a bit late for me to start in on it.  Thanks.  r=me
<wgrant> cjwatson: Thanks.
<wgrant> Yeah, no choice here, unfortunately.
<StevenK> wgrant: You mean you haven't abolished the bag? :-(
<wgrant> StevenK: I removed two callsites today.
<wgrant> But added one...
<StevenK> Net -1!
<blr> morning
<cjwatson> blr: morning :)
<blr> cjwatson: morning!
<blr> just found a bug in my token invalidation that did have a test, but the test was too naive..
<blr> tests can certainly engender a false sense of security heh
<cjwatson> Earlier today I had a bug where I'd left out "self" in a parameter list - almost all the tests passed anyway due to luck, but I was saved by one that now had a *very* confusing traceback ...
<blr> cjwatson: wgrant: in the token purge script called by cron, is there any harm in reporting the number of tokens purged to stdout? (useful for a manual test)
<wgrant> blr: You could also log it at INFO or something and have that not shown by default.
<cjwatson> Yep, I'd do that
<cjwatson> Lots of our cron scripts have that kind of status output
<blr> right, thanks
#launchpad-dev 2015-09-30
<lifeless> blr: tdd = developing by bouncing off the rails
<lifeless> blr: I *love* that quote [for all that I also love tdd]
<blr> heh yes
<cjwatson_> wgrant: I think I'm up to date with your reviews now.
<wgrant> cjwatson_: Thanks.
<wgrant> cjwatson_: I went back and forth between doing the unicode case in the xref layer and not.
<wgrant> cjwatson_: We could certainly cast on the way in, but how would it work on the way out?
<wgrant> In findFromMany, would we magically turn anything that looked like an int into an int?
<wgrant> I expect it will eventually be covered by an intermediate layer that knows how to resolve LP DB objects.
<wgrant> For example, if we had a type which used a name as a key, what would happen if the name looked integral?
<wgrant> I'd query for [('project', 'foo'), ('project', '123')], and get [('project', 'foo'), ('project', 123)] back and crash with a KeyError.
<cjwatson_> wgrant: True, that would be problematic.
<cjwatson> well, after spending most of a day on it, I think I now have parallel testing working (vivid host, precise guest)
<cjwatson> things I had to do: https://github.com/lxc/lxc/pull/667; use venv with testrepository 0.0.14, since newer versions expect subunit v2 and we aren't emitting that yet; https://code.launchpad.net/~cjwatson/launchpad/testrunner-uuid/+merge/272942; minor fiddling with .testr.conf; install python-gi and dbus-x11 in base container to make html5browser happier
<lifeless> cjwatson: you can use subunit1to2
<lifeless> cjwatson: to get v2 into newer testr's
<cjwatson> I have a working solution now though :)
<cjwatson> I think I would rather it continue to be a little bit inconvenient (but not too much) for me to use v1 locally, because in that case there's at least a chance I'll get round to hiking us up to v3
<cjwatson> er, v2
<lifeless> cjwatson: your call :)
<cjwatson> Probably need to get better dev control of buildbot first though
<lifeless> cjwatson: I just wouldn't want you to miss out on shiny new testr features
<lifeless> cjwatson: e.g. https://rbtcollins.wordpress.com/2015/09/18/testrepository-roadmap-201516/
<cjwatson> certainly don't want to be held back forever, but we see occasional reliability problems due I think to being on subunit v1, so that's some kind of priority in its own right
<cjwatson> alleged buildbot passes with failed tests and a low test count are particularly worrisome
<lifeless> what testr version are you using ?
<lifeless> (on buildbot)
<cjwatson> I don't think I know for sure, but I believe it's precise's, which is 0.0.5
<lifeless> ok so
<cjwatson> could be the one in https://code.launchpad.net/~yellow/+archive/ubuntu/ppa
<lifeless> I think (set -o pipefail; existing_command | subunit1to2) and a testr build including 83dff0a51b3ed3f66d5c5f2313c5057234312663
<lifeless> should fix that for you
<cjwatson> ok, that may be worth trying out, thanks.  would need to hunt down exactly what version we're using
<lifeless> that was fixed by 0.0.8 (perhaps earlier, but def 0.0.8)
<cjwatson> what I really want to do is to convert buildbot into a juju-deployed thing on the new developer-managed stagingstack, for multiple reasons: (a) direct access so that we can see just what's going on with these kinds of problems (b) way easier to upgrade (c) brownie points for jujuifying stuff (d) could very likely distribute builds across workers on multiple compute nodes rather than needing a couple of enormous ones
<lifeless> cjwatson: testr has hooks for distributing work itself now
<cjwatson> there'll be a bit of a cost to syncing the tree out to remote workers but not a prohibitive one
<cjwatson> ah, cool
<lifeless> cjwatson: but you could also schedule the work yourself and use testr load to coalesce the results
<lifeless> cjwatson: https://rbtcollins.wordpress.com/2013/01/14/multi-machine-parallel-testing-of-nova-with-testrepository/ shows one approach
<cjwatson> I think it would be sufficient to prepare the workers before calling testr and then just have a suitable test_command
<lifeless> yes; testr's current setup glue for this is serialised, so probably want a hybrid approach
<cjwatson> right, a bit like that approach but for the time being with manual provisioning/cleanup
<lifeless> the next gen stuff (see the roadmap blog post) will want more visibility, and that should enable really tuned things like optimising dispatch of per-layer things
<wgrant> cjwatson: Oh, you're sharing a non-COW working tree between concurrent test runs? Ambitious.
<wgrant> But yes, your buildbot plan roughly matches mine.
#launchpad-dev 2015-10-01
<cjwatson> I may be able to make it more robust, but it seems to more or less work.
<cjwatson> Do you use -c then?
<wgrant> cjwatson: I haven't used parallel tests locally in a while, but when I did I always used aufs2 to isolate the working dirs.
<wgrant> I don't know that there's much state in them, but I'd be surprised if there were none.
<wgrant> cjwatson: My other concern with the UUID change is that I can currently easily get into a running test's DB just by checking its PID.
<wgrant> It may be worth the tradeoff if we don't need COW on the working tree, though.
<cjwatson> wgrant: We could prepend the PID as well, presuming there's no particularly pressing length limit and that you could tab-complete the DB name.
<cjwatson> (I'm not in a rush if it's controversial, can always apply it locally as needed)
<wgrant> cjwatson: I wonder if Archive:+edit should hide checkboxes with impossible virtness.
<wgrant> cjwatson: Currently it lets my unprivileged account enable ia64 and sparc, which won't do anything.
<wgrant> Or I guess we could just restrict them, now, actually.
<cjwatson> wgrant: I thought I did hide impossible architectures.  Are you testing this on prod?
<wgrant> cjwatson: This was on dogfood.
<wgrant> Running devel tip.
<wgrant> Archive.available_processors doesn't check virtness at all.
<wgrant> (if the widget still existed on +admin it would make sense for that variant to show them all, but it doesn't so that's sort of moot)
<cjwatson> wgrant: Well, sure, but you won't see ia64 or sparc on prod because we hide arches that don't exist for any non-obsolete series.
<wgrant> cjwatson: Oh, of course.
<wgrant> That particular case indeed doesn't exist on prod.
<cjwatson> It's true that we show non-virt-capable arches, but I thought it was overcomplicated to try to handle that - would've needed either a separate view on +admin, or JS on +edit to show the hypothetical unrestricted non-virt-only non-obsolete arches when you uncheck require_virtualized
<wgrant> cjwatson: Well, +edit doesn't have the virt checkbox in the first place, does it?
<wgrant> If the arch widget were still on +admin, it would need to be fancy or just skip the check.
<cjwatson> oh, true, so  perhaps we could.
<wgrant> But on +edit the virt status can't change.
 * wgrant takes a hatchet to DatabaseLayer
<cjwatson> Oh?
<wgrant> It has ridiculous overhead.
<wgrant> Per-test overhead, that is.
<wgrant> Even in the fast path.
<wgrant> 10 tests that just each create a person take a total of 1200ms to execute
<StevenK> Welcome to Launchpad, I believe you've been here before?
<wgrant> I haven't looked at this problem in more than 18 months.
<wgrant> But I have enough wine that I can probably get through the test fallout this time.
<StevenK> wgrant: Aiming for the Ballmer Peak?
<wgrant> It's more that fixing tests that whine about ID differences is INCREDIBLY TEDIOUS.
<StevenK> Ah yes
<cjwatson> wgrant: I'm wondering what the path-like entry in the payload of merge proposal webhooks should be, and this makes me wonder whether we should consistently include a webservice link in other payloads.  (Perhaps as an addition; I can see how it's useful to have just the path sometimes.)
<wgrant> cjwatson: We can't include a full link, as the version and root may vary.
<wgrant> But we could include the force_local_path version, indeed.
<cjwatson> Right.
#launchpad-dev 2015-10-04
<mapreri> cool being able to enable/disable build arch in archives â¥
<cjwatson> It only really lets you usefully disable amd64 or i386 right now, but hopefully other arches will be along soon.
<mapreri> still, even if it has been some time now, it's really nice to see work and love on lp again ^^
<mapreri> anyway, I got a bug for you about this new feature :P
<cjwatson> Sure
<cjwatson> mapreri: The checkboxes are greyed out here
<cjwatson> mapreri: screenshot please
<mapreri> also here.  the point is that it's too hard to notice, and I only notice it if I untick another box and compare them
<mapreri> feel free to just mark it as very-low-resolve-it-lasts severity and move on, just wanted to share this :)
<cjwatson> OK, can you just follow up saying that the checkboxes are indeed greyed out but that it's too subtle?
<mapreri> sure
<cjwatson> I did think about not showing them at all if you can't enable them (vs. if they're restricted but already enabled), but I thought that having them present was a useful hint that you could ask for them
<cjwatson> Although there's a known bug that if the archive is virtualised (as most are) then we should only show the arches that we can build virtualised
<mapreri> cjwatson: would you prefer it in the description or as a comment?
<cjwatson> Don't mind
<mapreri> there you go
<cjwatson> ta
#launchpad-dev 2016-10-03
<cjwatson> Hm, turnip authentication is suboptimally-designed for code imports: it authenticates before it knows the repository, and the only state it retains between authentication and path translation is Person.id.
<cjwatson> I might have to change it to pass the username again if the user ID is None, and make up a username that's something like +code-import-job/<job_id> (to fit the blacklist)
<wgrant> cjwatson: Hm, do we really pass the ID as an int?
<wgrant> I think it's reasonable for authentication to be contextless; after all, in the SSH case we don't have context until authentication completes.
<cjwatson> We really pass the ID as an int.
<wgrant> :(
<cjwatson> I agree it's fine for it to be contextless in general, but in this case we need to either know what job the push relates to when authenticating, or we need to persist a bit of state until we do.
<cjwatson> More generalised git access tokens would suggest the latter approach.
<wgrant> Indeed, we need more than just the user ID now.
<cjwatson> And it's a more standard authn/authz style.
<wgrant> The current structure is fine, it's just unfortunate that the opaque data passed between the authn and authz stages is an int.
<cjwatson> (OK, we already do authz, it's just that at the moment we can do that with just the Person)
<cjwatson> Yeah, maybe I should just uplift that to a JSON dict.
<wgrant> AN ASSERTION!
<wgrant> Or a string.
<wgrant> Either way.
<wgrant> I don't think turnip needs to know anything about it, does it?
<wgrant> Other than maybe for logging.
<cjwatson> Well, yeah, a dict encoded as a string :)
<wgrant> So simply saying "any string will do" should work.
<cjwatson> No, turnip doesn't need to, though it's also not secret from it.
<wgrant> Certainly.
<cjwatson> We're already passing around a user/uid pair and only using the uid.
<wgrant> Indeed. I'm pretty sure I intended the username as temporary but then we never dropped it once you added the UID.
<wgrant> Let's add a third and never drop the other two!
<cjwatson> I was going to just go back to the username :)
<cjwatson> (I mean, not in general, there was a good reason we switched to UID.)
<wgrant> Eh, it needs changes on both ends anyway, so we might as well call it something unconfusing.
<cjwatson> It would be an annoying flag-day problem were it not for the fact that authenticateWithPassword is as yet unimplemented.
<cjwatson> So we can just declare that it now returns a dict.
<wgrant> One but not the other?
<wgrant> I thought the existing SSH key method already returned a dict.
<cjwatson> And translatePath can be changed to temporarily accept either form.
<wgrant> Maybe you changed that but didn't bother with the other.
<cjwatson> We pass a params dict to the backend in both cases.
<cjwatson> But the SSH backend just picks the username/uid out of its "avatar" directly, since lazr.sshservice deals with the authserver interaction.
<wgrant> Oh, SSH auth uses the authserver API
<wgrant> Right
<wgrant> Had forgotten that split.
<wgrant> I suppose we can leave that for now and emulate the dict on the turnip side.
<cjwatson> Yeah, that's easy.
<wgrant> And sort it out later if we need it, without any serious compat issues.
<wgrant> that/it == the SSH side of things, to be clear
 * wgrant sleeps.
<cjwatson> Yep
<cjwatson> Night.
<tsimonq2> I plan on writing some code that allows certain email addresses to be subscribed to a packageset (like yakktey-changes but filtered)
<tsimonq2> (or so I hope to)
<tsimonq2>  1. Is anyone opposed?
<tsimonq2>  2. Should I hardcode the email addresses in or should an administrator be allowed to edit the email addresses through a simple web interface?
<cjwatson> tsimonq2: no hardcoding
<tsimonq2> that's what I thought :P
<tsimonq2> cjwatson: but y'all aren't opposed to it?
<cjwatson> tsimonq2: it also shouldn't be email addresses IMO, it should be a mechanism to allow people/teams to subscribe to package upload notifications, much as they can to other kinds of notifications
<tsimonq2> oh?
<cjwatson> I would be opposed to notifying arbitrary addresses, but a subscription mechanism here seems reasonable
<tsimonq2> I see, ok
<cjwatson> lib/lp/soyuz/mail/packageupload.py would be the place that sends out the mail, but you'd need to figure out how to model the subscriptions
<cjwatson> possibly/probably including a way to subscribe to an entire packageset
<cjwatson> I'd suggest that it should use/extend the existing structural subscription mechanism
<tsimonq2> it would certainly help if there was a page that listed all packages in the Ubuntu archive allowing sorting by packageset
<tsimonq2> ok
<cjwatson> which doesn't support packagesets, but probably could
<cjwatson> well, you don't want to have to subscribe to all the packages individually in any event, so I don't know that such a page would be really the best approach anyway
<tsimonq2> well no, it would have checkboxes on the left of each item
<cjwatson> firstly, it would be an annoying amount of clicking; secondly, the packageset might change later and you probably want to track that
<tsimonq2> then the packageset name would be in bold and if you select that, it selects all the respective packages
<cjwatson> if you want to subscribe to a packageset, model that directly, don't record subscriptions to each element of it
<tsimonq2> oh yeah good point
<tsimonq2> ok
<tsimonq2> cjwatson: thanks, I'll get an MP ready within a week or two :)
<cjwatson> I think this is likely to be a moderate amount of work involving quite a bit of fiddly hacking, and it will require DB patches, so don't underestimate the work involved.
<cjwatson> To begin with given that it touches the DB it will certainly require at least two MPs, probably more.
<tsimonq2> yes
<tsimonq2> of course
<tsimonq2> that's just *my* estimate
<tsimonq2> it could be a lot longer
<tsimonq2> ok I have to go, I'll look into this later, thanks again!
<tsimonq2> win 12
<tsimonq2> whoops
<cjwatson> I don't want to discourage you, it's just a tough starter project :)
<cjwatson> So as long as you're aware of that.
<tsimonq2> cjwatson: I am :)
#launchpad-dev 2016-10-04
<wgrant> cjwatson: https://oops.canonical.com/oops/?oopsid=OOPS-4a0aaf3e8e9d686dd25943a55533b842 and https://oops.canonical.com/oops/?oopsid=OOPS-144400735e2b864ade9a821b80988e40 look like regressions from the target picker work. Possibly easier for you to grab them since you have context, but otherwise I'll have a look.
<cjwatson> wgrant: The latter is bug 1628091, the former is new.  Will look, thanks.
<mup> Bug #1628091: oops when loading bug <Launchpad itself:New> <https://launchpad.net/bugs/1628091>
<mkayaalp> I'm running a local launchpad.dev. When building a livefs, it always gets stuck at BuildStatus.UPLOADING
<mkayaalp> Should I be starting process-upload script manually?
<cjwatson> mkayaalp: Yes, that won't run unless you do so.
<mkayaalp> The soyuz docs mention some cron job that launches the process-upload
<mkayaalp> Where would that be
<mkayaalp> I don't see anything related under cronscripts
<mkayaalp> I can run it: scripts/process-upload.py --builds /var/tmp/builddmaster
<mkayaalp> But it exits when it's done processing all under incoming
<mkayaalp> So, in production does the ubuntu-cdimage wait until the cron job starts the process-upload?
<cjwatson> mkayaalp: The production cron job is:
<cjwatson> * * * * * /srv/launchpad.net/codelines/current/scripts/process-upload.py -C buildd --builds /srv/launchpad.net/builddmaster/ -q --log-file=DEBUG:/srv/launchpad.net/production-logs/process-build-uploads.log
<cjwatson> (obviously path-dependent)
<cjwatson> mkayaalp: And yes, it will exit when it's finished processing what's in incoming; it's a cron job, not a daemon
<cjwatson> mkayaalp: ubuntu-cdimage waits until the build's buildstate attribute on the webservice API reaches "Successfully built", which happens when the upload from the builder is processed.
<cjwatson> mkayaalp: See lib/cdimage/livefs.py:run_live_builds in lp:ubuntu-cdimage
<cjwatson> mkayaalp: (You must use -C buildd; processing builder uploads without that isn't going to work, since the default upload policy only accepts sourceful uploads and requires a signature.)
<mkayaalp> cjwatson: thanks a lot
<wgrant> mkayaalp: Are you familar with https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally? It's more oriented towards binary package builds, but many of the same pieces apply.
<mkayaalp> wgrant: I am running soyuz locally too
<mkayaalp> wgrant: That's what gets the build into the UPLOADING state after building it
#launchpad-dev 2016-10-06
<cjwatson> wgrant: I'd quite like to get https://code.launchpad.net/~cjwatson/launchpad/git-xmlrpc-auth-params/+merge/307694 into a deployment todayish if at all possible, so that I can move on with the turnip-side changes
<cjwatson> quite a bit of fiddly gated-on-deployments stuff going on here
<wgrant> cjwatson: Hm I'm almost certain I approved that this morning.
<cjwatson> doesn't seem so.  I pushed a couple of extra changes to it today as well, mostly as a result of thinking in more detail about the turnip changes
<cjwatson> (I plan to map turnip-authenticated-* to the auth_params dict fairly directly, except that uid gets int())
<cjwatson> (and can-authenticate is added on top)
<wgrant> Yep
<cjwatson> so then authenticateWithPassword will start returning a dict instead of a tuple, {"macaroon": password} in this case if it verifies that the macaroon at least belongs to it, and that will get mapped to turnip-authenticated-macaroon between pack-http and pack-git and then pack-git will send that as auth_params["macaroon"]
<cjwatson> I think that all works without too much pain
<cjwatson> when we get round to tokens issued for use by actual users we'll want something more compact, but for now just passing the serialised macaroon directly as the password is good enough
<wgrant> s/pack-git/pack-virt/?
<cjwatson> er yes.
<cjwatson> turnip.pack.git :-P
<wgrant> I do wish the native protocol had a name.
<mkayaalp> Why is ubuntu-cdimage trying to download "http://people.canonical.com/~evand/usb-creator/%s/stable" % series
<mkayaalp> it goes: project not in ("livecd-base", "ubuntu-base", "ubuntu-core", "ubuntu-desktop-next", "edubuntu"):
<mkayaalp> and then: ... or if (series >= "maverick"):
<mkayaalp> but there is nothing newer than natty in http://people.canonical.com/~evand/usb-creator
<cjwatson> legacy stuff we never removed; BTW this isn't a Launchpad question
<cjwatson> fetch failures there are ignored so should be harmless
<mkayaalp> ok sorry, the error was right after trying to download that
<mkayaalp> trying to execute build_all.sh and failing to find it
<cjwatson> that's in debian-cd, you need the various bits in configs/devel.  this is all notoriously challenging to set up locally
<cjwatson> further questions about cdimage to #ubuntu-devel or something please though, not here
<mkayaalp> ok got it thanks a lot
#launchpad-dev 2017-10-03
<cjwatson> tsimonq2: I really do not think it should be done via __getitem__ at all.
<cjwatson> tsimonq2: If you do that then GET requests will mutate the database, which is wrong.
<cjwatson> tsimonq2: It should be a new factory method and *not* called via __getitem__.
<tsimonq2> cjwatson: Ok, understood.
<cjwatson> (One practical reason that a new design shouldn't involve mutating the database in a GET request is that at some point we're likely to remove the workaround that does a commit after a webservice GET, since that slows things down and should be unnecessary with correct designs.)
#launchpad-dev 2017-10-04
<saigkill> Hello channel. Is it possible to rename a project and team in LP?
<cjwatson> saigkill: Sure.  File a ticket on https://answers.launchpad.net/launchpad/+addquestion if you can't do it yourself.
#launchpad-dev 2017-10-05
<stub> Hmm, we could use https://git-scm.com/docs/git-remote-helpers to configure git to understand lp: urls , and package it as a deb and/or classic snap to easily install or include as a dependency
