[00:18] <lifeless> jelmer: I find it amusing that a feature designed to reduce spam still spams when used via email.
[00:20] <jelmer> lifeless: oh, you're getting the affectsmetoo emails?
[00:20] <jelmer> sorry about that, I'll avoid using that via email
[00:20] <lifeless> jelmer: https://bugs.launchpad.net/launchpad/+bug/925224
[00:20] <_mup_> Bug #925224: affectsmetoo emails still spam all subscribers of a bug <Launchpad itself:Triaged> < https://launchpad.net/bugs/925224 >
[00:40] <rick_h> wallyworld_: yes, submitted and missed 3 tests
[00:40] <rick_h> wallyworld_: I've got to fix those and resubmit
[00:41] <rick_h> StevenK: howdy
[00:41] <wallyworld_> rick_h: ok. i've had to merge that branch locally inro my own own. if you could ping me when you've finished updating, that would be great
[00:41] <wallyworld_> s/own own/own work
[00:41] <rick_h> wallyworld_: will do, might be a bit. Out with some people atm
[00:41] <wallyworld_> rick_h: np. i didn't mean now, tomorrow if fine :-)
[00:42] <rick_h> wallyworld_: k, sounds good. Yea, just some stray doctests checking for YUI() vs LPJS so should be quick updtes
[00:42] <wallyworld_> cool :-)
[00:52] <huwshimi> lifeless: I think I'm getting bug 775214 from one of the Launchpad doc tests. I think the test passing but testr just ouputs "String or Integer object expected for key, unicode found". Is this actually a failure?
[00:52] <_mup_> Bug #775214: On python 2.7: String or Integer object expected for key, unicode found <Testrepository:Fix Committed by lifeless> < https://launchpad.net/bugs/775214 >
[00:55] <lifeless> huwshimi: where are you seeing this?
[00:56] <huwshimi> Output from "testr run -- -t sprites" (in the file lib/lp/services/doc/sprites.txt)
[00:56] <huwshimi> lifeless: ^
[00:57] <huwshimi> lifeless: But only from my modified version of that file
[00:58] <lifeless> huwshimi: probably not that bug
[00:58] <lifeless> huwshimi: does 'bin/test -t sprites' error ?
[00:59] <huwshimi> lifeless: nope: http://paste.ubuntu.com/825853/
[01:00] <lifeless> ok, could you run
[01:00] <lifeless> bin/test --subunit -t sprites > foo.log
[01:00] <lifeless> and then mail me that log file ?
[01:01] <huwshimi> lifeless: sure
[01:02] <huwshimi> lifeless: Mailed
[01:03] <lifeless> ok, so possibly you are seeing that
[01:03] <lifeless> what testr version are you running ?
[01:05] <lifeless> jelmer: was that irony?!
[01:05] <huwshimi> lifeless: testrepository 0.0.5-1.1 I think
[01:05] <lifeless> add the testing-cabal PPA
[01:05] <lifeless> that should sort things out for you
[01:06] <huwshimi> lifeless: Great, I'll try that
[01:11] <huwshimi> lifeless: Perfect, works now :)
[01:14] <lifeless> cool
[01:14] <lifeless> I have on my todo one more patch then a release for testr
[01:16] <huwshimi> lifeless:  I assume that ec2 test won't be confused by this somehow?
[01:27] <lifeless> it won't be
[01:34] <huwshimi> awesome
[03:04] <wallyworld_> StevenK: do you know how the lp credentials/ssh keys get into the ec2 instances, such that bzr commands against lp "just work" inside the instance
[03:06] <wgrant> wallyworld_: You might want to avoid thinking about it.
[03:06] <wgrant> ssh -A
[03:06] <wgrant> WCPGW
[03:06] <wallyworld_> wgrant: i've got a canonistack instance i'm playing with
[03:06] <wallyworld_> and i want to run some commands to pull lp trunk and run tests etc
[03:07] <mwhudson> i think ec2 test just uses ssh -A
[03:07] <wgrant> RIght.
[03:07] <wgrant> It uses ssh -A
[03:07] <wgrant> Which is why it's really not healthy to think about it for long.
[03:08] <wallyworld_> so i do "ssh -A <ip>" ?
[03:08] <wgrant> However you normally ssh into the instance, add -A
[03:08] <wallyworld_> right
[03:08] <wallyworld_> will try that, thanks
[03:08] <wgrant> However
[03:08] <wgrant> Depending on your SSH config that may not work.
[03:09] <wallyworld_> but it works when ec2 scripts do it
[03:09] <wgrant> Right, but they don't go through the DC.
[03:09] <wallyworld_> and the DC is more locked down?
[03:10] <wgrant> It is, but that's not relevant here. I think it was recommended at one point to locally disable forwarding for some hosts.
[03:10] <wgrant> But I don't have anything in mine, so I may have been misremembering.
[03:10] <wallyworld_> ok, will try it and see :-)
[03:10] <wallyworld_> thanks
[03:30] <StevenK> wgrant: How do I QA the TTBJ failure on DF?
[03:30] <StevenK> Since I've managed to avoid TTBJs completly until now
[03:34] <wgrant> StevenK: Difficult
[03:34] <wgrant> StevenK: TTBJs are normally done on staging
[03:34] <wgrant> But the code may not be there yet.
[03:35] <StevenK> wgrant: Happy enough to qa-untestable it, since let's face it, it's 3 lines
[03:35] <wgrant> Yeah
[03:35] <wgrant> And they were broken for a year without anybody noticing.
[03:35] <wgrant> So I'm not too concerned.
[03:35] <StevenK> I'm not interested in deploying either, given it's effectively the only change.
[03:43] <wgrant> StevenK: We can't deploy for a while anyway.
[03:43] <wgrant> stub landed some DB user changes that require production changes.
[03:44] <StevenK> They're still going through buildbot
[03:44] <wgrant> Ah
[03:44] <StevenK> Not interested in deploying 3 revisions with one change.
[03:44] <wgrant> No :)
[03:46]  * StevenK stabs gnome-terminal
[03:47] <rick_h> StevenK: hey, got a sec?
[03:47] <StevenK> rick_h: Sure do
[03:47] <rick_h> hey, so I started working on getting the YUI tests back up and linked right away from icing
[03:47] <rick_h> and to do that, I think I need the lp js files in the build/js/lp dir like we started out with
[03:48] <rick_h> and then just make all tests walk up ../../../build/...
[03:48] <rick_h> and then our convoy dir would be a symlink from build/js/lp off to the convoy root instead of building into the convoy root as we do now
[03:48] <StevenK> Why would we do that?
[03:49] <StevenK> I'm uncomfortable with out-of-tree symlinks
[03:49] <rick_h> for a couple of reasons. The YUI code is there now, and I want to keep that so we can swap out YUI versions in tests with a new make command that would swap out a symlink build/js/yui -> 3.3.0/3.4.0
[03:50] <rick_h> and I don't want to build twice, once into build/js/lp and once into $convoy_root/js/lp
[03:50] <rick_h> since we do things like min and such
[03:51] <rick_h> and if we create a build/js/lp then all the tests can point there and you it provides one place for all the files to be and if they're missed in the build step, the tests will fail
[03:51] <StevenK> rick_h: So, we don't currently build the rootdir automatically since it will blow up on qas/staging/prod, so I'm not sure about this plan.
[03:51] <rick_h> StevenK: right, so we don't currently build into the convoy dir
[03:52] <StevenK> make build is used by too many things currently
[03:52] <rick_h> but there's nothing that prevents us from the build/js dir
[03:52] <StevenK> rick_h: But we already build into icing
[03:52] <lifeless> rick_h: so the way we do it for icing is that the deploy process makes one symlink *into* our tree
[03:52] <rick_h> we build a single .js file into icong
[03:52] <rick_h> lifeless: and that could still do that right? It would symlink into build/js/lp
[03:52] <lifeless> rick_h: there are several symlinks in a farm in the root directory of the icing area on the apache server
[03:53] <rick_h> right now, the combo-rootdir places files into a $convoy_root dir but I want them to exist in build
[03:53] <lifeless> rick_h: yes, thats my proposal. So the symlinks point *into* our tree
[03:53] <StevenK> I don't think convoy follows symlinks
[03:53] <rick_h> lifeless: right, that's what I'm trying to change to
[03:53] <lifeless> StevenK: well, thats fixable :)
[03:53] <lifeless> rick_h: +1
[03:53] <rick_h> StevenK: ah right, I think that was the issue wasn't it. Well maybe I need to take a stab at fixing that then
[03:53] <StevenK> But I'm happy to investigate that change
[03:54] <lifeless> I'd like this to be as close to icing as possible, given they are so conceptually similar
[03:54] <lifeless> less surprises for new folk staring at it
[03:54] <rick_h> StevenK: ok, well I wanted to bring it up, see what we thought. Like lifeless says, I'd like the files to exist in our tree for the test purposes and then work out from there
[03:54] <rick_h> I don't want the tests knowing about $convoy_root and I'd like the tests to look at the build files in case the build is broken to know at the test level
[03:54] <StevenK> So we can populate build/js and then make /var/tmp/convoy be a symlink to that dir
[03:55] <rick_h> StevenK: yea, and if convoy doesn't follow symlinks I'll work on fixing that
[03:55] <rick_h> I think that'll be closer to what we do now with icing and would be cleanest.
[03:55] <StevenK> I'll see about doing that
[03:55] <rick_h> ty much
[04:10]  * StevenK grumbles
[04:11] <StevenK> Can't create a yui symlink under build/js since yui is already a directory
[04:11] <rick_h> StevenK: I thought there was a new command for installing the alt yui versions and such working
[04:13] <StevenK> The yui directory has changed from yui to build/js/yui
[04:16] <rick_h> StevenK: yea, that's part of what needs moving because if I want to run all the JS tests from YUI 3.3 to 3.4 then I need to update the seed file on all the tests
[04:16] <rick_h> so I want to write the tests to build/js/yui and have a make command to swap that symlink to the "active" yui version running
[04:17] <StevenK> Can you deal with build/js/yui/yui ?
[04:17] <StevenK> I can give you that easily
[04:17] <rick_h> yea, it can be whatever we want
[04:17] <rick_h> we just need to figure out what we want and stick with that standard
[04:18] <rick_h> but it needs to be coordinated between the make script, the test's seed file, and potentially convoy
[04:18] <StevenK> I might change the buildout yui stuff
[04:19] <rick_h> that's why I wanted to chat with you on it? :) I'm reading the Make book, but still researching make/buildout
[04:20]  * StevenK learnt make out of sheer self-defense
[04:20] <rick_h> anyway, time for bed here, really appreciate the help and let me know what goes wrong and how I can work on getting it going
[04:20] <rick_h> sorry to keep changing things over and over
[04:22] <StevenK> And now the icing yui links to the convoy rootdir?
[04:22] <StevenK> BAH
[04:22]  * StevenK shakes his fist at wallyworld_ 
[04:22] <wallyworld_> why?
[04:23] <StevenK> You can't break icing like that
[04:23] <StevenK> icing is served statically on banana/nutmeg and they won't have convoy installed
[04:24] <StevenK> I'd prefer icing's js stuff is left entirely alone
[04:47] <wallyworld_> the original build scripts from ages ago put yui stuff in icing
[04:48] <StevenK> I'm trying to sort this out, so I can bend build/js to my will
[04:50] <wallyworld_> cool
[04:51] <StevenK> wgrant: I can use inplace, since that is the default make target, but qas/s/prod don't call it
[05:28] <wgrant> huwshimi: That mockup is sort of along the lines of what I'd been thinking, but I was trying to work out how to distinguish some and all. Possibly all has a border or something.
[05:28] <wgrant> Because distinguishing them is reasonably important.
[05:30] <huwshimi> wgrant: Ah ok. Maybe we can think about colouring each status instead of all being grey. A bit like what Curtis had done for the text, but for the background.
[05:31] <wgrant> That's a possibility.
[05:31] <wgrant> But it seems like that might make more sense to indicate the policy.
[05:31] <wgrant> But I guess that's not so important if we show Nothing.
[05:31] <wgrant> I was assuming we'd not show policies with no access.
[05:32] <huwshimi> wgrant: Yeah, me too
[05:32] <huwshimi> wgrant: I guess then you'd have to have a "Add policy" button
[05:32] <wgrant> Right, there'd probably be an Add/+ link at the right which lets you select a new policy.
[05:33] <wgrant> But given that there's probably only three policies, it might be nicer to have them more tabular like this.
[05:33] <huwshimi> wgrant: Yeah.
[05:33] <wgrant> Then it becomes very scannable, particularly if we visually distinguish All and Some.
[05:34] <huwshimi> wgrant: It might be sensible if the reality is that there are a lot of "nothings"
[05:34] <wgrant> There are loooots of nothings.
[05:34] <huwshimi> wgrant: I might actually mock something up and send it to the list. I don't remember ever hearing why that changed (my original intention had been to hide the nothings)
[05:37] <huwshimi> wgrant: I guess it would be slightly less scannable though
[05:43] <huwshimi> wgrant: Also I guess it's slightly less explicit
[05:46] <huwshimi> wgrant: How often is it that a user/team will have more than one policy?
[05:48] <wgrant> huwshimi: Most projects will have either just proprietary or both embargoed security and user data. Some projects will have all three. Of those, project owners will generally have all of the available policies, most other users will only have user data, but the occasional user will probably have both security and user data.
[05:49]  * StevenK blinks
[05:49] <StevenK> Why is os.path.exists(full) returning False? :-(
[05:49] <wgrant> StevenK: Note that it follows symlinks.
[05:50] <huwshimi> wgrant: Ok, so in most cases we're talking about one or two policies being used for the whole project (so there won't be a jumble of all three very often)
[05:50] <wgrant> So will return False if the path points at a broken one.
[05:50] <wgrant> huwshimi: Right.
[05:50] <huwshimi> wgrant: Great.
[05:50] <StevenK> lrwxrwxrwx 1 steven steven 53 2012-02-02 16:22 /var/tmp/convoy -> /home/steven/launchpad/lp-branches/combo-url/build/js
[05:50] <StevenK> That looks fine to me
[05:56] <StevenK> [Thu Feb 02 16:55:41 2012] [error] [client 127.0.0.1] IOError: [Errno 13] Permission denied: '/var/tmp/convoy/yui/yui/yui-min.js'
[05:56] <StevenK> Hm, the plot thins
[05:59] <StevenK> Hmm, /var/tmp is 1777, so perhaps you can only cd if the owner matches
[06:00] <StevenK> Which sounds likely
[06:00] <StevenK> Since root can't cd either
[06:02] <wgrant> huwshimi: I think the scannability may recover if each policy type gets a colour.
[06:02] <wgrant> As lots of sites do with tags nowadays.
[06:03] <huwshimi> wgrant: I guess you could still do a lighter/darker shade of that colour to show "some" or "all"
[06:03] <wgrant> Exactly.
[06:09] <wallyworld_> StevenK: why ~/launchpad/lp-branches/combo-url?  is that local or on prod?
[06:09] <wallyworld_> i'd think we'd want it to point to somewhere in the tree
[06:12] <StevenK> That is the tree!
[06:12] <wallyworld_> not the tree where the lp source code is checked out to
[06:13] <wallyworld_> ie why not somewhere under lp-branches/my-lp-checkout/...
[06:13] <StevenK> Because I use seperate directories per branch
[06:14] <wallyworld_> :-(
[06:14] <wallyworld_> that's even more reason to put it under the proper source tree
[06:14] <wallyworld_> so different branches have their own copy
[06:15] <StevenK> It is in the source tree!
[06:15] <StevenK> That's the point of the build/js at the end
[06:15] <StevenK> The symlink is so I don't have to rewrite /usr/share/convoy/convoy.wsgi and reload apache everytime the branch changes
[06:15] <wallyworld_> so if your working tree for launchpad is lp-branches/my-lp-sandbox, how is lp-branches/combo-url in the launchpad working tree?
[06:16] <StevenK> What? You've lost me
[06:17] <wallyworld_> so the lp source code is checked out to lp-branches/some-working-dir
[06:17] <wallyworld_> and under there is lib/lp etc etc ie all the lp source
[06:17] <wallyworld_> so i would expect all build artifacts for that checkout to be located under that source dir
[06:17] <wallyworld_> not a level above
[06:18] <StevenK> wallyworld_: http://pastebin.ubuntu.com/826045/
[06:19] <wallyworld_> StevenK: oh, that is the name of a branch
[06:19] <wallyworld_> i thought it was just a directory where stuff got built into
[06:19] <StevenK> Yes!
[06:20] <wallyworld_> sorry, misunderstanding
[07:50] <lifeless> anyone up for a small revu ?
[08:25]  * lifeless makes a note to ring telecom tomorrow and complain about shocking performance occuring again
[08:46] <mabac> I'd like to submit a really small fix and wonder if a change to lib/lp/testing/_login.py should be accompanied by unit tests?
[08:47] <mabac> http://paste.ubuntu.com/826108/ is what I'd like to do btw since that would have let me know about trying to use the wrong function.
[08:55] <adeuring> good morning
[09:25]  * wgrant declares victory over triggers.
[09:25] <StevenK> wgrant: Oh, so you didn't win, you're just declaring victory?
[09:26] <wgrant> I think I've ironed out the last bugs.
[09:26] <wgrant> And they're not too slow.
[09:26] <wgrant> And they seem to do the right thing.
[09:26] <wgrant> (mapping legacy bug disclosure onto modern disclosure, and keeping it up to date across task/subscription/bug transitions and removals.
[09:37] <StevenK> wgrant: Can't we just unilaterally move the legacy stuff to modern terms?
[09:38] <wgrant> StevenK: Not in 20 seconds, no.
[09:39] <StevenK> wgrant: Hmm. And I guess a garbo job won't help?
[09:39] <wgrant> StevenK: The source data changes constantly.
[09:39] <wgrant> And we need to have it consistent eventually so we can flip the switch.
[09:40] <wgrant> The switch will tell the appservers to update the new schema instead, and tell the triggers to stop mirroring.
[09:41] <wgrant> (although some of the triggers will remain forever, the really ugly bits are only for the migration)
[09:42] <wallyworld_> wgrant: question, i have a test failure where it says httplib.HTTPSConnection does not exist, yet from the command line i can dir(httplib) and see that class is there. any ideas?
[09:43] <wgrant> wallyworld_: Hm, it exists in both 2.6 and 2.7.
[09:44] <wallyworld_> yes, hence mu confusion, although this error is in a canonistack instance
[09:44] <wallyworld_> running oneiric
[09:45] <wallyworld_> i've hand crafted some scripts to set up the instance, based on those used for aws. only a few test failures so far
[10:00] <StevenK> wallyworld_: Some scripts, or changes to lib/devscripts/ec2test?
[10:01] <wallyworld_> StevenK: some hand crafted python scripts to set up the instance and prepare launchpad, based on what's in devscripts
[10:02] <wallyworld_> StevenK: so far, about 1/2 way through, 3 test failures due to httplib.HTTPSConnection not being found, plus
[10:02] <wallyworld_> the only 2 other failures so far - one of them has output that includes a SHA512 checksum, causing the assert equals to fail https://pastebin.canonical.com/59271/
[10:02] <wallyworld_> so i'm not too disappointed for a first attempt
[10:08] <StevenK> wallyworld_: Nice. So perhaps the next step is to integrate your scripts into devscripts?
[10:19] <wallyworld_> StevenK: that is the plan - there's a cloud instance object that i'm thinking about factoring out and making pluggable, depending on whether --aws or --canonistack is specified
[10:32] <StevenK> wallyworld_: Hmmmm. I'd prefer it try canonistack first and fall back if there is no credientals or it can't start an instance.
[10:33] <wallyworld_> StevenK: details, details :-) i was merely talking generally
[10:33] <wallyworld_> the test run just blew up with an out of memory error, so need to arrange for a bigger instance
[10:34] <StevenK> You can specify the size when you start it
[10:35]  * wallyworld_ goes to find doco
[10:35] <StevenK> Perhaps you should find a lack of work :-P
[10:36] <StevenK> wallyworld_: You may need to ask IS about which sizes are configured, too
[10:36] <wallyworld_> yeah, might call time for today. but you can't be too critical yourself :-P
[10:36] <wallyworld_> i've been talking to andrew who has been most helpful
[10:36] <StevenK> I'm waiting for Steam to install a game, so speak for yourself. :-P
[10:38] <wgrant> wallyworld_: Is this on the private LP canonistack?
[10:38] <wgrant> Or the main canonistack?
[10:38] <wallyworld_> wgrant: i have no idea tbh
[10:38] <wallyworld_> perhaps the private one if i had to guess
[10:50] <stub> Anyone recall if there is a reason we are still using psycopg2 2.2.2?
[10:56] <stub> Nope... that's not it. Seems we or Storm are breaking string quoting
[10:57] <stub> >>> c.execute("SELECT %s", ("Hello\nMom",))
[10:57] <stub> >>> c.fetchone()[0]
[10:57] <stub> 'Hello\\nMom'
[10:57] <wgrant> stub: I thought it was psycopg2 that was the problem.
[10:57] <wgrant> Sure you upgraded the right one?
[10:58] <stub> With raw psyocpg2
[10:58] <stub> >>> cur.execute("SELECT %s", ("Raw\nPsycopg2",))
[10:58] <stub> >>> cur.fetchone()[0]
[10:58] <stub> 'Raw\nPsycopg2'
[10:59] <stub> Works. Thought it was I had 2.4.2 installed and lp was using 2.2.2. Upgraded lp and still happening. Suspect a dodgy Storm decision in the past.
[10:59] <wgrant> Sure you reran buildout etc. sufficiently?
[10:59] <stub> Or maybe we broke it for some reason
[11:00] <stub> Yes... psycopg2.__version__ reports what I expect
[11:00] <wgrant> :(
[11:03] <stub> Hmm...
[11:04] <stub> So 'make harness', 'from lp.services.database.sqlbase import cursor' gives the bug
[11:04] <stub> But 'make harness', 'import psycopg2; connect; cur=con.connect' etc. works
[11:04] <stub> And then the cursor() helper in sqlbase gives the right answers
[11:05] <wgrant> Huh
[11:06] <stub> nah... mixed up my cursors.
[11:07] <stub> raw psycopg2 works in make harness, cursors returned through that helper are busted.
[11:08] <stub> Yay. Think I'm about to wade through connection and cursor wrappers...
[11:08] <wgrant> :D
[11:09] <wgrant> It's a bit of an awful mess.
[11:09] <stub> oic. The 'cursor' does its own interpolation for some reason, using sqlvalues. And sqlvalues won't know about the stricter quoting I want.
[11:09] <wgrant> Heh
[11:09] <stub> And fixing sqlvalues should fix gobs
[11:11] <stub> I guess we do that so we get sane statement logs containing interpolated values
[11:18] <stub> And the root quoting is done by... sqlobject.sqlbuilder.sqlrepr
[11:18] <wgrant> Which is Storm :)
[11:28] <stub> Nope... looks like us in lib/sqlobject.py
[11:28] <wgrant> Huh
[11:28] <wgrant> Indeed
[11:28] <wgrant> WTF is that still there
[11:28] <stub> circa 2007
[11:28]  * stub puts on his fedora
[11:28] <wgrant> I thought lib/sqlobject was just importing stuff from Storm :/
[11:30] <stub> I have a feeling fixing that to do strict E quoting is going to have a lot of doctest fallout...
[11:30] <wgrant> We'll find out soon :)
[11:31] <stub> I could still chicken out and leave it 'for later'
[11:47] <rick_h> morning
[12:32] <rick_h> wallyworld_: heads up, relanding the LPJS stuff so hopefully will be merged when you get going again
[13:15] <mabac> I just proposed a tiny change and I haven't really pestered anyone to review it earlier. :) would anyone want to have a look at this? https://code.launchpad.net/~mabac/launchpad/login-raises-non-string/+merge/91265
[13:18] <rick_h> mabac: is this in response to a bug? Isn't there already email validation?
[13:20] <mabac> rick_h, I didn't file a bug (yet). this function gives me confusing zope errors when I passed a Person to it by mistake. I figured that it might help someone else to debug tests if it would give a better error.
[13:20] <mabac> rick_h, it's not related to production code btw, it's in testing/
[13:26] <rick_h> mabac: do you know off hand what a "participation" is?
[13:26] <rick_h> mabac: the docstring there suggests you can pass an email, a const, and a participation
[13:27] <rick_h> mabac: and it appears the participation can implement IPublicationRequest so that would be some object that would fail your str test
[13:28] <rick_h> bah, too early in the morning
[13:28] <rick_h> that's a second param
[13:28] <rick_h> sorry mabac, sending you in wrong circles this morning
[13:28] <mabac> rick_h, thanks, that saved me some typing ;)
[13:29] <mabac> rick_h, no problem
[13:29] <rick_h> so the only concern is what the ANONYMOUS constant is, and it appears that's set to a string as well
[13:29] <mabac> rick_h, yup, I think that constant is used as if it would be an email address
[13:30] <mabac> rick_h, seems to be mostly (only?) used in calls to this function
[13:30] <mabac> rick_h, rather these login* functions
[13:31] <salgado> mabac, rick_h, we can't use 'str' there because most callsites are actually passing in unicode. so basestring, maybe?
[13:31] <mabac> argh, good catch salgado. thanks
[13:32] <rick_h> salgado: sounds like a good plan
[13:32] <salgado> a full test run, for those who can afford it, would've caught that as well, though :)
[13:33] <rick_h> salgado: yea, definitely, but still nice to get it before several hours of ec2'ing
[13:33] <mabac> rick_h, salgado I'll fix that and try to survive running the tests locally again ;)
[13:34] <rick_h> mabac: so noted in the MP, let me know when the change is up and I'll ok and pass it to my mentor
[13:34] <mabac> rick_h, will do. thanks for you time!
[14:27] <mabac> rick_h, I've pushed a new revision https://code.launchpad.net/~mabac/launchpad/login-raises-non-string/+merge/91265
[14:28] <rick_h> mabac: ok, looks good thanks
[14:46] <rick_h> abentley: yea, that wiki update looks good to me. Thanks!
[14:46] <abentley> rick_h: cool.
[16:02] <cjohnston> Is there any sort of status on bug #881019? This is causing problems with Summit where users aren't able to retrieve their schedules and could cause them to miss private business meetings that they can only see if they are logged in properly.
[16:02] <_mup_> Bug #881019: Lp login is broken after account merge <merge-deactivate> <openid> <regression> <users> <Launchpad itself:Triaged> < https://launchpad.net/bugs/881019 >
[16:20] <sinzui> cjohnston, We are not working on it really.The current understanding is that the issue with offsite systems misunderstanding how OpenId works and that Lp supports multiple identifiers
[16:21] <sinzui> cjohnston, You should comment on the bug to provide information about what is broken for the summit site
[16:21] <cjohnston> sinzui: ty
[16:30] <adeuring> rick_h: fancy a review? https://code.launchpad.net/~adeuring/launchpad/bug-829074/+merge/91302
[16:31] <rick_h> adeuring: sure thing
[16:31] <adeuring> thanks
[16:59] <salgado> does it really take ages to run 'make newsampledata'?
[17:16] <salgado> 30 minutes at 100% CPU and counting
[17:18] <lifeless> well sampledata needs to diaf
[17:18] <lifeless> salgado: you're not adding sample data are you ?
[17:18] <lifeless> jcsackett: ola! - https://code.launchpad.net/~lifeless/loggerhead/bug-594591/+merge/91212
[17:19] <lifeless> salgado: (but 30 minutes is 5-6 times too long)
[17:20] <salgado> lifeless, I am!  (I probably won't commit it but I did newsampledata just so that I can restore my sampledata changes after a make schema)
[17:20] <salgado> maybe there's an easier way to do that?
[17:24] <salgado> lifeless, while I have your attention... I'm working on a script to migrate work items from the whiteboard of blueprints into a separate table.  it uses a fresh transaction for every BP because it rolls back if anything goes wrong.  is it worth using DBLoopTuner for that (also, we're talking about 2500 BPs that have work-items on their whiteboard)
[17:27] <lifeless> salgado: (I hate my internets)
[17:28] <lifeless> salgado: so, sampledata is officially reviled; the current practice is to use the factory to add test data
[17:28] <lifeless> salgado: and do it per test
[17:28] <lifeless> salgado: if you need the same exact test data in a number of tests, wrap it in a Fixture
[17:28] <salgado> lifeless, I know that; I once gave a talk on why sampledata was bad ;)
[17:29] <salgado> and that's why we now have two sets of sampledata
[17:30] <lifeless> righto :)
[17:30]  * lifeless wasn't sure how much you'd forgotten over in the land of django
[17:30] <salgado> I'm only changing the -dev sampledata, and even then I probably won't submit those changes
[17:31] <lifeless> whew ;>
[17:31] <salgado> it has the ~2500 BPs from production that have work-items on their whiteboard, and I'm testing my new script against that
[17:32] <salgado> and I wanted to restore those after a make schema, but I can probably avoid a make schema or dump just the one table I care about
[17:32] <salgado> but anyway, I have one important question for you... :)
 lifeless, while I have your attention... I'm working on a script to migrate work items from the whiteboard of blueprints into a separate table.  it uses a fresh transaction for every BP because it rolls back if anything goes wrong.  is it worth using DBLoopTuner for that (also, we're talking about 2500 BPs that have work-items on their whiteboard)
[17:33] <lifeless> yes
[17:33] <lifeless> dblooptuner is trivial and solves plenty of issues ;)
[17:33] <lifeless> shove it in the garbo hourly
[17:34] <lifeless> if there is no native way to show 'BP x was done' you can use memcache to store the highest migrated whiteboard (o
[17:35] <salgado> oh, but we I don't think we want that as a cronjob... we should run it just once
[17:35] <lifeless> salgado: so, you could manually arrange it
[17:35] <lifeless> but garbo means no sysadmin intervention
[17:35] <lifeless> you add it, watch the logs, remove it all without having to ask a single question
[17:36] <salgado> and it shouldn't hurt to run it a few extra times, so maybe that's a good idea
[17:39] <lifeless> we need a data migration specific helper I think
[17:39] <lifeless> one that will ignore cluster lag, has a dedicated storage area for point-migrated-to
[17:39] <salgado> yeah, that'd be nice
[17:39] <lifeless> but this is orthogonal to using it or not, just would make it easier to do each time
[17:50] <jml> lifeless: where's that thing you wrote about what to do instead of using an ORM?
[17:52] <jml> I can see https://dev.launchpad.net/LEP/PersistenceLayer, but ISTR something with implementation notes
[17:52] <rick_h> adeuring: man, you know how to build a MP phew
[17:52] <adeuring> rick_h: how dow mean ;)
[17:53] <rick_h> brain melted I think
[17:53] <lifeless> jml: you might mean two things
[17:53] <adeuring> sorry... let me know if I can help to up the mess#
[17:53] <lifeless> jml: do you mean having the template service depeend wholly on services
[17:54] <jml> lifeless: no, I think  I mean the other thing
[17:54] <lifeless> jml: or do you mean a rethink of how appservers that do both data access and business logic would get at their data?
[17:54] <jml> lifeless: that one
[17:54] <lifeless> I think that LEP is what there is
[17:54] <jml> hmm.
[17:54] <lifeless> there was a list thread with science fiction in it
[17:54] <jml> I'm sure I remember you & wallyworld talking more about it
[17:54] <jml> maybe the list thread is it
[17:54]  * jml looks
[17:55] <lifeless> jcsackett: hi
[17:59] <jml> lifeless: thanks.
[18:01] <lifeless> jml: you founds it ?
[18:02] <jcsackett> lifeless: hi.
[18:02] <lifeless> jcsackett: still reviewing ?
[18:02] <jcsackett> lifeless: all day. :-)
[18:02] <lifeless> jcsackett: oh, nvm, jam got to it
[18:02]  * lifeless fishslaps himself
[18:03] <jcsackett> lifeless: righto. :-)
[18:03] <jcsackett> you have a fish that near at hand? that's useful. :-P
[18:03] <jml> lifeless: yeah, over a couple of threads.
[18:03] <lifeless> no comment :P
[18:03] <jml> lifeless: think my head was conflating some of the code snippets from the API thing that Foundations was doing at roughly the same time
[18:04] <lifeless> yeah, some similarities
[18:04] <jml> lifeless: the "Simple made easy" talk has had me thinking about what I would want to use instead of ORMs
[18:04] <lifeless> aiming at similar sorts of overheads
[18:04] <lifeless> jml: cool
[18:04] <lifeless> e.g. scala? :)
[18:04] <jml> lifeless: well, Hickey would say Clojure, I think.
[18:05] <jml> lifeless: mostly I'm thinking about stuff within Python for now.
[18:05] <jml> lifeless: since most of my output needs to be deployed to Canonical DC
[18:06] <jml> and I'm disappointed with Go not having Haskell-like type classes. (I blame niemeyer for raising my hopes)
[18:07] <jml> I've got to head out. Maybe back in 4 hrs.
[18:07] <lifeless> jml: I'm underwhelmed by go fwiw
[18:08] <jml> lifeless: the concurrency thing seems kind of nice
[18:08] <adeuring> rick_h: thanks for the review!
[18:09] <jml> lifeless: but I'd agree it's mostly a bit meh.
[18:10] <jml> lifeless: if it ends up as "write code almost as easily as in Python but have it run 5x faster" then that's still pretty interesting
[18:10] <lifeless> that would be interesting
[18:11] <lifeless> I seem to have chosen cases that python is faster at every time so far, though
[18:11] <lifeless> which may just be bad luck
[18:11] <jml> heh
[18:11] <jml> My SMH target solver is faster in Go :)
[18:12] <jml> anyway, I really do have to go.
[18:12] <lifeless> smh?
[18:12] <jml> lifeless: sydney morning herald
[18:12] <lifeless> shoo
[18:12] <rick_h> jml: you played with pypy with it?
[18:13] <rick_h> I don't seen to find a ton of CPU bound issues that make pypy awesome, but wonder if something like that would fit
[18:13] <jml> rick_h: no, but my #twisted friends all love PyPy and want to marry it
[18:13]  * jml gone
[18:42] <salgado> lifeless, so, IIUC my ITunableLoop.__call__(chunk_size) should process up to chunk_size items in a single transaction and the loop tuner will tweak chunk_size on the next call according to the time my transaction takes. is that right?
[18:43] <lifeless> yes
[18:46] <salgado> lifeless, because of the way my code is structured, though, it would be more natural to process each item in a separate transaction (like http://paste.ubuntu.com/826728/), so that I can rollback if anything goes wrong when parsing the whiteboard of a given item.  that should play fine with DBLoopTuner, right?
[18:48] <lifeless> I don't know why it wouldn't, though you may find you need to ensure there is an open xaction etc
[18:50] <salgado> oh, right, other ITunableLoops seem to begin() a transaction when they're done processing a chunk
[19:01] <lifeless> salgado: yes, the timing of chunks is to prevent impacting webapp requests
[19:03] <salgado> I see
[19:05] <lifeless> sinzui: is ie8 'supported' for us ?
[19:07] <sinzui> :( yes. an a-level browser for YUI
[19:07] <sinzui> lifeless, we have a several IE bugs too
[19:07] <lifeless> so, per bugtriage I think 925154 should be critical ?
[19:07] <lifeless> page-worked, now errors ?
[19:08] <sinzui> lifeless, I think so, somone needs to fix the other IE bugs too. We have sat on the fence about the issue because we claim they can always use the featurewithout JS...which has not been true for a year
[19:10] <lifeless> abentley: ^ a heads up - per BugTriage supported browser issues are critical
[19:10] <sinzui> lifeless, Lets ask deryck and rick_h to look at them and determine if these are critical because we do not offer an alternate for IE
[19:10] <abentley> lifeless: that's a supported browser?
[19:10] <lifeless> abentley: per sinzui, yes - A-level
[19:12] <sinzui> lifeless, this is YUI's doc on graded browser experience: http://yuilibrary.com/yui/docs/tutorials/gbs/
[19:12] <abentley> lifeless, sinzui: To my knowledge, no testing was performed on IE.  I was under the impression that our support was C grade.
[19:12] <sinzui> abentley, I sure that is true. I do not have windows to ever do such a test
[19:13] <abentley> sinzui: A-grade for YUI does not imply A-grade for us, does it?
[19:13] <rick_h> yea, we've had broken IE for a while. I filed a bug a while ago on the portlets failing in IE causing JS to stop running
[19:14] <sinzui> abentley, I certainly does not, we have always said. the feature gracefully degrades. We have written a lot of JS in the last year that does not degrade...that does not even have a html-form equivalent to fall back to
[19:14] <lifeless> so, we have a business rule that IE needs to work for the core functionality for OEM
[19:14] <lifeless> whether we call it A grade or not is by-the-by
[19:17] <sinzui> lifeless, the rule is not that simple. We heard IE6 which is very difficult to support in modern libraries. Our own browser reports indicate that IE very minute. We have not had a report from OEM in two years that IE is broken.
[19:17] <lifeless> sinzui: I requested a confirmation that they still care about IE6 a few months back
[19:17] <lifeless> I will send another
[19:17] <lifeless> YUI still claims A for IE6, interestingly
[19:19] <rick_h> hmm, i thought they announced dropping that a year ago
[19:19] <rick_h> even Google apps all fail to work in IE6 any more
[19:19] <rick_h> my wife's medical practice just finally got chrome this month because of it
[19:19] <lifeless> maybe I am misreading that gbs page?
[19:19] <lifeless> ah
[19:19] <lifeless> http://www.yuiblog.com/blog/2011/07/12/gbs-update/#grades-deprecated
[19:19] <lifeless> 'graded browser experience with deprecated grades'
[19:20] <sinzui> rick_h, out Chinese counter-parts couldn't give a toss about what MS supports
[19:20] <rick_h> http://www.yuiblog.com/blog/2011/07/12/gbs-update/
[19:20] <rick_h> nvm, you found it
[19:20] <rick_h> so it's no longer a/b/c
[19:20] <rick_h> but "tested" with failback
[19:21]  * lifeless mails stakeholders 
[19:21] <rick_h> sinzui: sorry, just meant that as a "big web players" moving away from IE6
[19:21] <sinzui> rick_h, right. IE was very difficult to support even when it was an A-grade browser
[19:23] <rick_h> sinzui: definitely not against getting things working in IE at all. Don't want to seem that way
[19:24] <sinzui> lifeless, maybe you missed rick_h's and I's exchange about JS support: https://lists.launchpad.net/launchpad-dev/msg08750.html
[19:24] <sinzui> We seem to be talking it now
[19:25] <sinzui> rick_h, I had a round of fixes about 6 months ago when we discovered that IE's event bubbling failed because something like YUI.Event was not being loaded for them
[19:26] <rick_h> hmm, the one I hit was that the portlets load into a <tbody> which IE won't allow and throws an error so those ajax loads fail
[19:26] <rick_h> sinzui: but yea, I'm sure there's a bunch of issues that'd take some time to work through
[19:26] <rick_h> especially with buglistings as I don't know it's ever been IE tested
[19:29] <lifeless> sinzui: I saw that yes
[19:29] <sinzui> rick_h, we do not have a written doc about which browsers we support, and what happens when it is not supported.
[19:29] <lifeless> sinzui: I'm a big fan of inline rendering :)
[19:32] <rick_h> so I'm guessing that a landing page linked to Chrome Frame isn't going to fly? :)
[19:33] <lifeless> we might get away with chrome frame activeX control
[19:33] <lifeless> if the factories can use it
[19:38] <lifeless> sinzui: I probably got all the facts wrong; please correct my mail if so :)
[19:39] <sinzui> okay
[19:39] <abentley> sinzui: What do we do when one of our upstreams asks for their project to be deleted? https://answers.launchpad.net/launchpad/+question/185361
[19:41] <abentley> sinzui: nm, I see you're already on it.
[19:41] <sinzui> We deactivate if it is not being used by other communities. this one is. It is used by Ubuntu and the request is disputed. I did what I could by disabling answers and bugs
[19:41] <sinzui> ^ abentley
[19:41] <sinzui> The question answer I gave was final
[19:46] <rick_h> lifeless: so if we go the route of the U1 response with full graded support, is there any chance we'd be able to grab some sort of test bed for tests or QA of these?
[19:47] <rick_h> lifeless: sinzui at least running the js test .html files via something like webdriver?
[19:47]  * rick_h has been secretly looking to do something with just FF and Chrome, but IE gives a bigger excuse
[19:48] <sinzui> rick_h, We have talks about browser shots and a selenium test suite in the past.
[19:48] <rick_h> sinzui: yea, I've assumed so, I don't know the history so forgive repeat questions
[19:49] <sinzui> rick_h, our YUI tests are webkit because gecko is a bitch to compile and keep patched
[19:50] <sinzui> rick_h, note the irony in the webkit selection; it does not provide the JS engine used by firefox and chromium users
[19:51] <cjohnston> /wc/48
[19:51] <rick_h> chromium is webkit base at least, but yes. What I meant was I was trying to tinker with things like yeti and sst
[19:52] <rick_h> things that fire up real browsers
[19:54] <sinzui> oh yuck. I ran away from that level of testing. I do not disagree with it. I just think I should not be writing those tests are maintaining the infrastructure. I use libraries and standards as my guide to writing yui tests
[19:55] <rick_h> sinzui: definitely, but I know in buglisting we hit issues between FF and chrome alone and I was trying to think of a way to run just the JS tests in both.
[19:55] <sinzui> yes. I saw some comment about chromium oddities in the tests.
[19:56] <rick_h> it's something I'll continue to try to get running locally, but wanted to bring it up
[20:06] <lifeless> so jenkins + grid + browser tests yes
[20:06] <lifeless> or whatever - that sort of conceptual setup
[20:37] <lifeless> james_w: I've now chatted with mthaddon, and the consensus is to go with one shared oops-tools instance
[20:37] <lifeless> james_w: he is going to run it past james first
[20:37] <james_w> lifeless, ok, what were the reasons OOI?
[20:38] <lifeless> james_w: user/dev perspective: single shared service, no need to figure out which service instance a particular oops-id belongs too
[20:39] <lifeless> james_w: ops perspective - one instance to care for and feed; consumers can easily split out oopses on disk so resourcing can be partitioned off per-department if needed
[20:39] <lifeless> james_w: dev perspective - less instances to run around upgrading when there are security issues or whatnots
[20:40] <james_w> ok
[20:41] <james_w> we unfortunately have rather a long deployment pipeline before we can send the oopses anyway, but at least this probably shortens it a bit
[20:41] <lifeless> The mild overhead of going multitenant was an already expected cost as LP gets more SOAised
[20:41] <lifeless> ssltunnel to rabbit should let you start pretty quickly
[20:43] <james_w> it's not that
[20:43] <lifeless> oh, ok :)
[20:44] <james_w> there's branches queued up, but it needs code deployes
[20:44] <james_w> and packages in to CAT in co-ordination
[20:44] <lifeless> ah
[20:44] <james_w> then settings updates via puppet to turn it on
[21:49] <abentley> lifeless: could you explain what you mean about the script being misconfigured?
[21:54] <abentley> lifeless: Is it the fact that you're forcing the logs to INFO?  IMO, that's a stopgap, because the oops-id and message-id still aren't being logged at the same level as the error that they correspond to.
[21:54] <lifeless> abentley: the script was running with stdout/stderr capturing rather than with a --log-file=INFO:/... parameter
[21:55] <abentley> lifeless: Why include the -q parameter? is that for stdout/stderr?
[21:55] <lifeless> yes
[21:56] <abentley> lifeless: So the information that gets logged to stdout/stderr will still be deficient.
[21:56] <lifeless> abentley: it won't log anything to stdout/stderr
[21:56] <abentley> lifeless: Okay, so why include the -q parameter?
[21:57] <lifeless> so that it doesn't
[21:57] <lifeless> uhm, I don't mean this to be like playing 20 questions
[21:57] <abentley> lifeless: AIUI, the -q parameter prevents stdout/stderr from emitting INFO messages.
[21:57] <lifeless> abentley: yes; the goals for the setup are:
[21:58] <lifeless>  - we log direct to a file anything we want to log
[21:58] <lifeless>  - WARNING and above messages generate oopses for aggregation and analysis
[21:58] <lifeless>  - stdout/stderr are empty except when the script runner itself catastrophically fails (and we then get immediate notification on lp-error-reports)
[22:00] <lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=20e752ffc3566d6bee8c71afcadaa5a7 is an example of the oopses we get
[22:00] <lifeless> which seems to have a decent backtrace
[22:00] <abentley> lifeless: That all sounds good, though I'm not sure we really want to be logging at INFO level.
[22:00] <abentley> lifeless: Not with our volume.
[22:01] <lifeless> abentley: we can push less common stuff down to DEBUG
[22:02] <lifeless> abentley: I think, given that email has no guarantee of showing the user, that we probably do want to capture the messageid and oopsid every time.
[22:04] <abentley> lifeless: Yes, I can agree with that.
[22:08] <lifeless> we log what, 8M web requests a day, on apache + on the zope access logs, and email is a tiny % of that
[22:08] <lifeless> so we could be quite fat and it wouldn't be an issue
[22:09] <lifeless> thats not a reason to log more (or less) just a data point from the ops aspect
[22:09] <abentley> lifeless: It still seems wacky *not* to emit the oops-id and message-id at error level, since we do want to know that about errors.
[22:10] <lifeless> the problem is how to notify @ 'ERROR' when notifying @ 'ERROR' triggers an OOPS
[22:10] <lifeless> in regular logging parlance, the use of different levels is to permit filtering of output
[22:10] <abentley> lifeless: You stick it in the error that we're already emitting.
[22:12] <lifeless> abentley: how? that error is what becomes the oops - it shouldn't be getting written to the log file at all
[22:14] <abentley> lifeless: The message-id, I mean.  When the ERROR->OOPS handling is in place, the oops-id automatically gets logged at ERROR level, no?
[22:16] <lifeless> abentley: no, the oops-id generally isn't logged at all
[22:16] <lifeless> abentley: (because most of our scripts have no concept of UI - we find out via the oops reports)
[22:17] <lifeless> abentley: process-mail seems to do a lot of special handling that makes this worse than for other scripts
[22:18] <abentley> lifeless: Jobs that use the standard runner log their oops ids.
[22:18] <abentley> lifeless: I'm not sure how "concept of UI" relates to logging.
[22:19] <abentley> lifeless: I never read the oops reports, and when I get a bug report, I hit the logs to try and find out what happened.
[22:20] <lifeless> abentley: so the job runner does it's own generation and reporting - it doesn't depend on error behaivour
[22:20] <lifeless> abentley: the jobrunner reports oops ids at INFO
[22:20] <abentley> lifeless: true, true.
[22:21] <abentley> lifeless: I was just giving an example that contradicts "the oops-id generally isn't logged at all"
[22:22] <lifeless> ah right
[22:22] <lifeless> so, the logging OopsHandler
[22:22] <lifeless> exists to ensure we capture the error for the reports
[22:22] <lifeless> which are (currently) our primary signal for the health of the system in advance of user initiated reports
[22:23] <lifeless> I'm not sure if it is possible to emit a logging message from a logging handler, but if it is, logging the oops id at info would match what our twisted equivalent, and the job runner equivalent, do.
[22:25] <abentley> lifeless: I would imagine the logging OopsHandler could also append it to the message.
[22:26] <abentley> lifeless: Though that runs the risk of losing the whole thing if oops handling fails.
[22:30] <lifeless> abentley: the oops config is built with a couple of fallbacks, but yes that is a risk
[22:49] <StevenK> wgrant: Oh, how should that query change?
[22:49] <wgrant> StevenK: It should change to not return a list of people, but existence of a matching row for a particular person.
[22:50] <StevenK> WHERE cs.date_expires < NOW() AND u.id = self.user (effectively) ?
[22:50] <wgrant> Something like that.
[22:51] <StevenK> And drop the distinct, hand wave, hand wave
[23:55] <lifeless> ooh pub holiday monday \o/
[23:55] <StevenK> What's the occasion?
[23:56] <lifeless> waitangi
[23:57] <StevenK> lifeless: Have you yelled at your ISP today?
[23:59] <lifeless> not yet
[23:59] <lifeless> our ajax detection log has broken :(