/srv/irclogs.ubuntu.com/2012/02/02/#launchpad-dev.txt

lifelessjelmer: I find it amusing that a feature designed to reduce spam still spams when used via email.00:18
jelmerlifeless: oh, you're getting the affectsmetoo emails?00:20
jelmersorry about that, I'll avoid using that via email00:20
lifelessjelmer: https://bugs.launchpad.net/launchpad/+bug/92522400:20
_mup_Bug #925224: affectsmetoo emails still spam all subscribers of a bug <Launchpad itself:Triaged> < https://launchpad.net/bugs/925224 >00:20
rick_hwallyworld_: yes, submitted and missed 3 tests00:40
rick_hwallyworld_: I've got to fix those and resubmit00:40
rick_hStevenK: howdy00:41
wallyworld_rick_h: ok. i've had to merge that branch locally inro my own own. if you could ping me when you've finished updating, that would be great00:41
wallyworld_s/own own/own work00:41
rick_hwallyworld_: will do, might be a bit. Out with some people atm00:41
wallyworld_rick_h: np. i didn't mean now, tomorrow if fine :-)00:41
rick_hwallyworld_: k, sounds good. Yea, just some stray doctests checking for YUI() vs LPJS so should be quick updtes00:42
wallyworld_cool :-)00:42
huwshimilifeless: I think I'm getting bug 775214 from one of the Launchpad doc tests. I think the test passing but testr just ouputs "String or Integer object expected for key, unicode found". Is this actually a failure?00:52
_mup_Bug #775214: On python 2.7: String or Integer object expected for key, unicode found <Testrepository:Fix Committed by lifeless> < https://launchpad.net/bugs/775214 >00:52
lifelesshuwshimi: where are you seeing this?00:55
huwshimiOutput from "testr run -- -t sprites" (in the file lib/lp/services/doc/sprites.txt)00:56
huwshimilifeless: ^00:56
huwshimilifeless: But only from my modified version of that file00:57
lifelesshuwshimi: probably not that bug00:58
lifelesshuwshimi: does 'bin/test -t sprites' error ?00:58
huwshimilifeless: nope: http://paste.ubuntu.com/825853/00:59
lifelessok, could you run01:00
lifelessbin/test --subunit -t sprites > foo.log01:00
lifelessand then mail me that log file ?01:00
huwshimilifeless: sure01:01
huwshimilifeless: Mailed01:02
lifelessok, so possibly you are seeing that01:03
lifelesswhat testr version are you running ?01:03
lifelessjelmer: was that irony?!01:05
huwshimilifeless: testrepository 0.0.5-1.1 I think01:05
lifelessadd the testing-cabal PPA01:05
lifelessthat should sort things out for you01:05
huwshimilifeless: Great, I'll try that01:06
huwshimilifeless: Perfect, works now :)01:11
lifelesscool01:14
lifelessI have on my todo one more patch then a release for testr01:14
huwshimilifeless:  I assume that ec2 test won't be confused by this somehow?01:16
lifelessit won't be01:27
huwshimiawesome01:34
wallyworld_StevenK: do you know how the lp credentials/ssh keys get into the ec2 instances, such that bzr commands against lp "just work" inside the instance03:04
wgrantwallyworld_: You might want to avoid thinking about it.03:06
wgrantssh -A03:06
wgrantWCPGW03:06
wallyworld_wgrant: i've got a canonistack instance i'm playing with03:06
wallyworld_and i want to run some commands to pull lp trunk and run tests etc03:06
mwhudsoni think ec2 test just uses ssh -A03:07
wgrantRIght.03:07
wgrantIt uses ssh -A03:07
wgrantWhich is why it's really not healthy to think about it for long.03:07
wallyworld_so i do "ssh -A <ip>" ?03:08
wgrantHowever you normally ssh into the instance, add -A03:08
wallyworld_right03:08
wallyworld_will try that, thanks03:08
wgrantHowever03:08
wgrantDepending on your SSH config that may not work.03:08
wallyworld_but it works when ec2 scripts do it03:09
wgrantRight, but they don't go through the DC.03:09
wallyworld_and the DC is more locked down?03:09
wgrantIt is, but that's not relevant here. I think it was recommended at one point to locally disable forwarding for some hosts.03:10
wgrantBut I don't have anything in mine, so I may have been misremembering.03:10
wallyworld_ok, will try it and see :-)03:10
wallyworld_thanks03:10
StevenKwgrant: How do I QA the TTBJ failure on DF?03:30
StevenKSince I've managed to avoid TTBJs completly until now03:30
wgrantStevenK: Difficult03:34
wgrantStevenK: TTBJs are normally done on staging03:34
wgrantBut the code may not be there yet.03:34
StevenKwgrant: Happy enough to qa-untestable it, since let's face it, it's 3 lines03:35
wgrantYeah03:35
wgrantAnd they were broken for a year without anybody noticing.03:35
wgrantSo I'm not too concerned.03:35
StevenKI'm not interested in deploying either, given it's effectively the only change.03:35
wgrantStevenK: We can't deploy for a while anyway.03:43
wgrantstub landed some DB user changes that require production changes.03:43
StevenKThey're still going through buildbot03:44
wgrantAh03:44
StevenKNot interested in deploying 3 revisions with one change.03:44
wgrantNo :)03:44
* StevenK stabs gnome-terminal03:46
rick_hStevenK: hey, got a sec?03:47
StevenKrick_h: Sure do03:47
rick_hhey, so I started working on getting the YUI tests back up and linked right away from icing03:47
rick_hand to do that, I think I need the lp js files in the build/js/lp dir like we started out with03:47
rick_hand then just make all tests walk up ../../../build/...03:48
rick_hand then our convoy dir would be a symlink from build/js/lp off to the convoy root instead of building into the convoy root as we do now03:48
StevenKWhy would we do that?03:48
StevenKI'm uncomfortable with out-of-tree symlinks03:49
rick_hfor a couple of reasons. The YUI code is there now, and I want to keep that so we can swap out YUI versions in tests with a new make command that would swap out a symlink build/js/yui -> 3.3.0/3.4.003:49
rick_hand I don't want to build twice, once into build/js/lp and once into $convoy_root/js/lp03:50
rick_hsince we do things like min and such03:50
rick_hand if we create a build/js/lp then all the tests can point there and you it provides one place for all the files to be and if they're missed in the build step, the tests will fail03:51
StevenKrick_h: So, we don't currently build the rootdir automatically since it will blow up on qas/staging/prod, so I'm not sure about this plan.03:51
rick_hStevenK: right, so we don't currently build into the convoy dir03:51
StevenKmake build is used by too many things currently03:52
rick_hbut there's nothing that prevents us from the build/js dir03:52
StevenKrick_h: But we already build into icing03:52
lifelessrick_h: so the way we do it for icing is that the deploy process makes one symlink *into* our tree03:52
rick_hwe build a single .js file into icong03:52
rick_hlifeless: and that could still do that right? It would symlink into build/js/lp03:52
lifelessrick_h: there are several symlinks in a farm in the root directory of the icing area on the apache server03:52
rick_hright now, the combo-rootdir places files into a $convoy_root dir but I want them to exist in build03:53
lifelessrick_h: yes, thats my proposal. So the symlinks point *into* our tree03:53
StevenKI don't think convoy follows symlinks03:53
rick_hlifeless: right, that's what I'm trying to change to03:53
lifelessStevenK: well, thats fixable :)03:53
lifelessrick_h: +103:53
rick_hStevenK: ah right, I think that was the issue wasn't it. Well maybe I need to take a stab at fixing that then03:53
StevenKBut I'm happy to investigate that change03:53
lifelessI'd like this to be as close to icing as possible, given they are so conceptually similar03:54
lifelessless surprises for new folk staring at it03:54
rick_hStevenK: ok, well I wanted to bring it up, see what we thought. Like lifeless says, I'd like the files to exist in our tree for the test purposes and then work out from there03:54
rick_hI don't want the tests knowing about $convoy_root and I'd like the tests to look at the build files in case the build is broken to know at the test level03:54
StevenKSo we can populate build/js and then make /var/tmp/convoy be a symlink to that dir03:54
rick_hStevenK: yea, and if convoy doesn't follow symlinks I'll work on fixing that03:55
rick_hI think that'll be closer to what we do now with icing and would be cleanest.03:55
StevenKI'll see about doing that03:55
rick_hty much03:55
* StevenK grumbles04:10
StevenKCan't create a yui symlink under build/js since yui is already a directory04:11
rick_hStevenK: I thought there was a new command for installing the alt yui versions and such working04:11
StevenKThe yui directory has changed from yui to build/js/yui04:13
rick_hStevenK: yea, that's part of what needs moving because if I want to run all the JS tests from YUI 3.3 to 3.4 then I need to update the seed file on all the tests04:16
rick_hso I want to write the tests to build/js/yui and have a make command to swap that symlink to the "active" yui version running04:16
StevenKCan you deal with build/js/yui/yui ?04:17
StevenKI can give you that easily04:17
rick_hyea, it can be whatever we want04:17
rick_hwe just need to figure out what we want and stick with that standard04:17
rick_hbut it needs to be coordinated between the make script, the test's seed file, and potentially convoy04:18
StevenKI might change the buildout yui stuff04:18
rick_hthat's why I wanted to chat with you on it? :) I'm reading the Make book, but still researching make/buildout04:19
* StevenK learnt make out of sheer self-defense04:20
rick_hanyway, time for bed here, really appreciate the help and let me know what goes wrong and how I can work on getting it going04:20
rick_hsorry to keep changing things over and over04:20
StevenKAnd now the icing yui links to the convoy rootdir?04:22
StevenKBAH04:22
* StevenK shakes his fist at wallyworld_ 04:22
wallyworld_why?04:22
StevenKYou can't break icing like that04:23
StevenKicing is served statically on banana/nutmeg and they won't have convoy installed04:23
StevenKI'd prefer icing's js stuff is left entirely alone04:24
wallyworld_the original build scripts from ages ago put yui stuff in icing04:47
StevenKI'm trying to sort this out, so I can bend build/js to my will04:48
wallyworld_cool04:50
StevenKwgrant: I can use inplace, since that is the default make target, but qas/s/prod don't call it04:51
wgranthuwshimi: That mockup is sort of along the lines of what I'd been thinking, but I was trying to work out how to distinguish some and all. Possibly all has a border or something.05:28
wgrantBecause distinguishing them is reasonably important.05:28
huwshimiwgrant: Ah ok. Maybe we can think about colouring each status instead of all being grey. A bit like what Curtis had done for the text, but for the background.05:30
wgrantThat's a possibility.05:31
wgrantBut it seems like that might make more sense to indicate the policy.05:31
wgrantBut I guess that's not so important if we show Nothing.05:31
wgrantI was assuming we'd not show policies with no access.05:31
huwshimiwgrant: Yeah, me too05:32
huwshimiwgrant: I guess then you'd have to have a "Add policy" button05:32
wgrantRight, there'd probably be an Add/+ link at the right which lets you select a new policy.05:32
wgrantBut given that there's probably only three policies, it might be nicer to have them more tabular like this.05:33
huwshimiwgrant: Yeah.05:33
wgrantThen it becomes very scannable, particularly if we visually distinguish All and Some.05:33
huwshimiwgrant: It might be sensible if the reality is that there are a lot of "nothings"05:34
wgrantThere are loooots of nothings.05:34
huwshimiwgrant: I might actually mock something up and send it to the list. I don't remember ever hearing why that changed (my original intention had been to hide the nothings)05:34
huwshimiwgrant: I guess it would be slightly less scannable though05:37
huwshimiwgrant: Also I guess it's slightly less explicit05:43
huwshimiwgrant: How often is it that a user/team will have more than one policy?05:46
wgranthuwshimi: Most projects will have either just proprietary or both embargoed security and user data. Some projects will have all three. Of those, project owners will generally have all of the available policies, most other users will only have user data, but the occasional user will probably have both security and user data.05:48
* StevenK blinks05:49
StevenKWhy is os.path.exists(full) returning False? :-(05:49
wgrantStevenK: Note that it follows symlinks.05:49
huwshimiwgrant: Ok, so in most cases we're talking about one or two policies being used for the whole project (so there won't be a jumble of all three very often)05:50
wgrantSo will return False if the path points at a broken one.05:50
wgranthuwshimi: Right.05:50
huwshimiwgrant: Great.05:50
StevenKlrwxrwxrwx 1 steven steven 53 2012-02-02 16:22 /var/tmp/convoy -> /home/steven/launchpad/lp-branches/combo-url/build/js05:50
StevenKThat looks fine to me05:50
StevenK[Thu Feb 02 16:55:41 2012] [error] [client 127.0.0.1] IOError: [Errno 13] Permission denied: '/var/tmp/convoy/yui/yui/yui-min.js'05:56
StevenKHm, the plot thins05:56
StevenKHmm, /var/tmp is 1777, so perhaps you can only cd if the owner matches05:59
StevenKWhich sounds likely06:00
StevenKSince root can't cd either06:00
wgranthuwshimi: I think the scannability may recover if each policy type gets a colour.06:02
wgrantAs lots of sites do with tags nowadays.06:02
huwshimiwgrant: I guess you could still do a lighter/darker shade of that colour to show "some" or "all"06:03
wgrantExactly.06:03
wallyworld_StevenK: why ~/launchpad/lp-branches/combo-url?  is that local or on prod?06:09
wallyworld_i'd think we'd want it to point to somewhere in the tree06:09
StevenKThat is the tree!06:12
wallyworld_not the tree where the lp source code is checked out to06:12
wallyworld_ie why not somewhere under lp-branches/my-lp-checkout/...06:13
StevenKBecause I use seperate directories per branch06:13
wallyworld_:-(06:14
wallyworld_that's even more reason to put it under the proper source tree06:14
wallyworld_so different branches have their own copy06:14
StevenKIt is in the source tree!06:15
StevenKThat's the point of the build/js at the end06:15
StevenKThe symlink is so I don't have to rewrite /usr/share/convoy/convoy.wsgi and reload apache everytime the branch changes06:15
wallyworld_so if your working tree for launchpad is lp-branches/my-lp-sandbox, how is lp-branches/combo-url in the launchpad working tree?06:15
StevenKWhat? You've lost me06:16
wallyworld_so the lp source code is checked out to lp-branches/some-working-dir06:17
wallyworld_and under there is lib/lp etc etc ie all the lp source06:17
wallyworld_so i would expect all build artifacts for that checkout to be located under that source dir06:17
wallyworld_not a level above06:17
StevenKwallyworld_: http://pastebin.ubuntu.com/826045/06:18
wallyworld_StevenK: oh, that is the name of a branch06:19
wallyworld_i thought it was just a directory where stuff got built into06:19
StevenKYes!06:19
wallyworld_sorry, misunderstanding06:20
=== almaisan-away is now known as al-maisan
lifelessanyone up for a small revu ?07:50
* lifeless makes a note to ring telecom tomorrow and complain about shocking performance occuring again08:25
mabacI'd like to submit a really small fix and wonder if a change to lib/lp/testing/_login.py should be accompanied by unit tests?08:46
mabachttp://paste.ubuntu.com/826108/ is what I'd like to do btw since that would have let me know about trying to use the wrong function.08:47
adeuringgood morning08:55
* wgrant declares victory over triggers.09:25
StevenKwgrant: Oh, so you didn't win, you're just declaring victory?09:25
wgrantI think I've ironed out the last bugs.09:26
wgrantAnd they're not too slow.09:26
wgrantAnd they seem to do the right thing.09:26
wgrant(mapping legacy bug disclosure onto modern disclosure, and keeping it up to date across task/subscription/bug transitions and removals.09:26
StevenKwgrant: Can't we just unilaterally move the legacy stuff to modern terms?09:37
wgrantStevenK: Not in 20 seconds, no.09:38
StevenKwgrant: Hmm. And I guess a garbo job won't help?09:39
wgrantStevenK: The source data changes constantly.09:39
wgrantAnd we need to have it consistent eventually so we can flip the switch.09:39
wgrantThe switch will tell the appservers to update the new schema instead, and tell the triggers to stop mirroring.09:40
wgrant(although some of the triggers will remain forever, the really ugly bits are only for the migration)09:41
wallyworld_wgrant: question, i have a test failure where it says httplib.HTTPSConnection does not exist, yet from the command line i can dir(httplib) and see that class is there. any ideas?09:42
wgrantwallyworld_: Hm, it exists in both 2.6 and 2.7.09:43
wallyworld_yes, hence mu confusion, although this error is in a canonistack instance09:44
wallyworld_running oneiric09:44
wallyworld_i've hand crafted some scripts to set up the instance, based on those used for aws. only a few test failures so far09:45
StevenKwallyworld_: Some scripts, or changes to lib/devscripts/ec2test?10:00
wallyworld_StevenK: some hand crafted python scripts to set up the instance and prepare launchpad, based on what's in devscripts10:01
wallyworld_StevenK: so far, about 1/2 way through, 3 test failures due to httplib.HTTPSConnection not being found, plus10:02
wallyworld_the only 2 other failures so far - one of them has output that includes a SHA512 checksum, causing the assert equals to fail https://pastebin.canonical.com/59271/10:02
wallyworld_so i'm not too disappointed for a first attempt10:02
StevenKwallyworld_: Nice. So perhaps the next step is to integrate your scripts into devscripts?10:08
wallyworld_StevenK: that is the plan - there's a cloud instance object that i'm thinking about factoring out and making pluggable, depending on whether --aws or --canonistack is specified10:19
StevenKwallyworld_: Hmmmm. I'd prefer it try canonistack first and fall back if there is no credientals or it can't start an instance.10:32
wallyworld_StevenK: details, details :-) i was merely talking generally10:33
wallyworld_the test run just blew up with an out of memory error, so need to arrange for a bigger instance10:33
StevenKYou can specify the size when you start it10:34
* wallyworld_ goes to find doco10:35
StevenKPerhaps you should find a lack of work :-P10:35
StevenKwallyworld_: You may need to ask IS about which sizes are configured, too10:36
wallyworld_yeah, might call time for today. but you can't be too critical yourself :-P10:36
wallyworld_i've been talking to andrew who has been most helpful10:36
StevenKI'm waiting for Steam to install a game, so speak for yourself. :-P10:36
wgrantwallyworld_: Is this on the private LP canonistack?10:38
wgrantOr the main canonistack?10:38
wallyworld_wgrant: i have no idea tbh10:38
wallyworld_perhaps the private one if i had to guess10:38
stubAnyone recall if there is a reason we are still using psycopg2 2.2.2?10:50
stubNope... that's not it. Seems we or Storm are breaking string quoting10:56
stub>>> c.execute("SELECT %s", ("Hello\nMom",))10:57
stub>>> c.fetchone()[0]10:57
stub'Hello\\nMom'10:57
wgrantstub: I thought it was psycopg2 that was the problem.10:57
wgrantSure you upgraded the right one?10:57
stubWith raw psyocpg210:58
stub>>> cur.execute("SELECT %s", ("Raw\nPsycopg2",))10:58
stub>>> cur.fetchone()[0]10:58
stub'Raw\nPsycopg2'10:58
stubWorks. Thought it was I had 2.4.2 installed and lp was using 2.2.2. Upgraded lp and still happening. Suspect a dodgy Storm decision in the past.10:59
wgrantSure you reran buildout etc. sufficiently?10:59
stubOr maybe we broke it for some reason10:59
stubYes... psycopg2.__version__ reports what I expect11:00
wgrant:(11:00
stubHmm...11:03
stubSo 'make harness', 'from lp.services.database.sqlbase import cursor' gives the bug11:04
stubBut 'make harness', 'import psycopg2; connect; cur=con.connect' etc. works11:04
stubAnd then the cursor() helper in sqlbase gives the right answers11:04
wgrantHuh11:05
stubnah... mixed up my cursors.11:06
stubraw psycopg2 works in make harness, cursors returned through that helper are busted.11:07
stubYay. Think I'm about to wade through connection and cursor wrappers...11:08
wgrant:D11:08
wgrantIt's a bit of an awful mess.11:09
stuboic. The 'cursor' does its own interpolation for some reason, using sqlvalues. And sqlvalues won't know about the stricter quoting I want.11:09
wgrantHeh11:09
stubAnd fixing sqlvalues should fix gobs11:09
stubI guess we do that so we get sane statement logs containing interpolated values11:11
stubAnd the root quoting is done by... sqlobject.sqlbuilder.sqlrepr11:18
wgrantWhich is Storm :)11:18
stubNope... looks like us in lib/sqlobject.py11:28
wgrantHuh11:28
wgrantIndeed11:28
wgrantWTF is that still there11:28
stubcirca 200711:28
* stub puts on his fedora11:28
wgrantI thought lib/sqlobject was just importing stuff from Storm :/11:28
stubI have a feeling fixing that to do strict E quoting is going to have a lot of doctest fallout...11:30
wgrantWe'll find out soon :)11:30
stubI could still chicken out and leave it 'for later'11:31
rick_hmorning11:47
=== al-maisan is now known as almaisan-away
rick_hwallyworld_: heads up, relanding the LPJS stuff so hopefully will be merged when you get going again12:32
mabacI just proposed a tiny change and I haven't really pestered anyone to review it earlier. :) would anyone want to have a look at this? https://code.launchpad.net/~mabac/launchpad/login-raises-non-string/+merge/9126513:15
=== rick_h changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10^2
rick_hmabac: is this in response to a bug? Isn't there already email validation?13:18
mabacrick_h, I didn't file a bug (yet). this function gives me confusing zope errors when I passed a Person to it by mistake. I figured that it might help someone else to debug tests if it would give a better error.13:20
mabacrick_h, it's not related to production code btw, it's in testing/13:20
rick_hmabac: do you know off hand what a "participation" is?13:26
rick_hmabac: the docstring there suggests you can pass an email, a const, and a participation13:26
rick_hmabac: and it appears the participation can implement IPublicationRequest so that would be some object that would fail your str test13:27
rick_hbah, too early in the morning13:28
rick_hthat's a second param13:28
rick_hsorry mabac, sending you in wrong circles this morning13:28
mabacrick_h, thanks, that saved me some typing ;)13:28
mabacrick_h, no problem13:29
rick_hso the only concern is what the ANONYMOUS constant is, and it appears that's set to a string as well13:29
mabacrick_h, yup, I think that constant is used as if it would be an email address13:29
mabacrick_h, seems to be mostly (only?) used in calls to this function13:30
mabacrick_h, rather these login* functions13:30
salgadomabac, rick_h, we can't use 'str' there because most callsites are actually passing in unicode. so basestring, maybe?13:31
mabacargh, good catch salgado. thanks13:31
rick_hsalgado: sounds like a good plan13:32
salgadoa full test run, for those who can afford it, would've caught that as well, though :)13:32
rick_hsalgado: yea, definitely, but still nice to get it before several hours of ec2'ing13:33
mabacrick_h, salgado I'll fix that and try to survive running the tests locally again ;)13:33
rick_hmabac: so noted in the MP, let me know when the change is up and I'll ok and pass it to my mentor13:34
mabacrick_h, will do. thanks for you time!13:34
mabacrick_h, I've pushed a new revision https://code.launchpad.net/~mabac/launchpad/login-raises-non-string/+merge/9126514:27
rick_hmabac: ok, looks good thanks14:28
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h*, jcsackett | Firefighting: - | Critical bugtasks: 4*10^2
rick_habentley: yea, that wiki update looks good to me. Thanks!14:46
abentleyrick_h: cool.14:46
=== matsubara is now known as matsubara-lunch
=== salgado is now known as salgado-lunch
cjohnstonIs there any sort of status on bug #881019? This is causing problems with Summit where users aren't able to retrieve their schedules and could cause them to miss private business meetings that they can only see if they are logged in properly.16:02
_mup_Bug #881019: Lp login is broken after account merge <merge-deactivate> <openid> <regression> <users> <Launchpad itself:Triaged> < https://launchpad.net/bugs/881019 >16:02
sinzuicjohnston, We are not working on it really.The current understanding is that the issue with offsite systems misunderstanding how OpenId works and that Lp supports multiple identifiers16:20
sinzuicjohnston, You should comment on the bug to provide information about what is broken for the summit site16:21
cjohnstonsinzui: ty16:21
=== salgado-lunch is now known as salgado
adeuringrick_h: fancy a review? https://code.launchpad.net/~adeuring/launchpad/bug-829074/+merge/9130216:30
rick_hadeuring: sure thing16:31
adeuringthanks16:31
=== matsubara-lunch is now known as matsubara
salgadodoes it really take ages to run 'make newsampledata'?16:59
salgado30 minutes at 100% CPU and counting17:16
lifelesswell sampledata needs to diaf17:18
lifelesssalgado: you're not adding sample data are you ?17:18
lifelessjcsackett: ola! - https://code.launchpad.net/~lifeless/loggerhead/bug-594591/+merge/9121217:18
lifelesssalgado: (but 30 minutes is 5-6 times too long)17:19
salgadolifeless, I am!  (I probably won't commit it but I did newsampledata just so that I can restore my sampledata changes after a make schema)17:20
salgadomaybe there's an easier way to do that?17:20
salgadolifeless, while I have your attention... I'm working on a script to migrate work items from the whiteboard of blueprints into a separate table.  it uses a fresh transaction for every BP because it rolls back if anything goes wrong.  is it worth using DBLoopTuner for that (also, we're talking about 2500 BPs that have work-items on their whiteboard)17:24
lifelesssalgado: (I hate my internets)17:27
lifelesssalgado: so, sampledata is officially reviled; the current practice is to use the factory to add test data17:28
lifelesssalgado: and do it per test17:28
lifelesssalgado: if you need the same exact test data in a number of tests, wrap it in a Fixture17:28
salgadolifeless, I know that; I once gave a talk on why sampledata was bad ;)17:28
salgadoand that's why we now have two sets of sampledata17:29
lifelessrighto :)17:30
* lifeless wasn't sure how much you'd forgotten over in the land of django17:30
salgadoI'm only changing the -dev sampledata, and even then I probably won't submit those changes17:30
lifelesswhew ;>17:31
salgadoit has the ~2500 BPs from production that have work-items on their whiteboard, and I'm testing my new script against that17:31
salgadoand I wanted to restore those after a make schema, but I can probably avoid a make schema or dump just the one table I care about17:32
salgadobut anyway, I have one important question for you... :)17:32
salgado<salgado> lifeless, while I have your attention... I'm working on a script to migrate work items from the whiteboard of blueprints into a separate table.  it uses a fresh transaction for every BP because it rolls back if anything goes wrong.  is it worth using DBLoopTuner for that (also, we're talking about 2500 BPs that have work-items on their whiteboard)17:32
lifelessyes17:33
lifelessdblooptuner is trivial and solves plenty of issues ;)17:33
lifelessshove it in the garbo hourly17:33
lifelessif there is no native way to show 'BP x was done' you can use memcache to store the highest migrated whiteboard (o17:34
salgadooh, but we I don't think we want that as a cronjob... we should run it just once17:35
lifelesssalgado: so, you could manually arrange it17:35
lifelessbut garbo means no sysadmin intervention17:35
lifelessyou add it, watch the logs, remove it all without having to ask a single question17:35
salgadoand it shouldn't hurt to run it a few extra times, so maybe that's a good idea17:36
lifelesswe need a data migration specific helper I think17:39
lifelessone that will ignore cluster lag, has a dedicated storage area for point-migrated-to17:39
salgadoyeah, that'd be nice17:39
lifelessbut this is orthogonal to using it or not, just would make it easier to do each time17:39
jmllifeless: where's that thing you wrote about what to do instead of using an ORM?17:50
jmlI can see https://dev.launchpad.net/LEP/PersistenceLayer, but ISTR something with implementation notes17:52
rick_hadeuring: man, you know how to build a MP phew17:52
adeuringrick_h: how dow mean ;)17:52
rick_hbrain melted I think17:53
lifelessjml: you might mean two things17:53
adeuringsorry... let me know if I can help to up the mess#17:53
lifelessjml: do you mean having the template service depeend wholly on services17:53
jmllifeless: no, I think  I mean the other thing17:54
lifelessjml: or do you mean a rethink of how appservers that do both data access and business logic would get at their data?17:54
jmllifeless: that one17:54
lifelessI think that LEP is what there is17:54
jmlhmm.17:54
lifelessthere was a list thread with science fiction in it17:54
jmlI'm sure I remember you & wallyworld talking more about it17:54
jmlmaybe the list thread is it17:54
* jml looks17:54
lifelessjcsackett: hi17:55
jmllifeless: thanks.17:59
lifelessjml: you founds it ?18:01
jcsackettlifeless: hi.18:02
lifelessjcsackett: still reviewing ?18:02
jcsackettlifeless: all day. :-)18:02
lifelessjcsackett: oh, nvm, jam got to it18:02
* lifeless fishslaps himself18:02
jcsackettlifeless: righto. :-)18:03
jcsackettyou have a fish that near at hand? that's useful. :-P18:03
jmllifeless: yeah, over a couple of threads.18:03
lifelessno comment :P18:03
jmllifeless: think my head was conflating some of the code snippets from the API thing that Foundations was doing at roughly the same time18:03
lifelessyeah, some similarities18:04
jmllifeless: the "Simple made easy" talk has had me thinking about what I would want to use instead of ORMs18:04
lifelessaiming at similar sorts of overheads18:04
lifelessjml: cool18:04
lifelesse.g. scala? :)18:04
jmllifeless: well, Hickey would say Clojure, I think.18:04
jmllifeless: mostly I'm thinking about stuff within Python for now.18:05
jmllifeless: since most of my output needs to be deployed to Canonical DC18:05
jmland I'm disappointed with Go not having Haskell-like type classes. (I blame niemeyer for raising my hopes)18:06
jmlI've got to head out. Maybe back in 4 hrs.18:07
lifelessjml: I'm underwhelmed by go fwiw18:07
jmllifeless: the concurrency thing seems kind of nice18:08
adeuringrick_h: thanks for the review!18:08
jmllifeless: but I'd agree it's mostly a bit meh.18:09
jmllifeless: if it ends up as "write code almost as easily as in Python but have it run 5x faster" then that's still pretty interesting18:10
lifelessthat would be interesting18:10
lifelessI seem to have chosen cases that python is faster at every time so far, though18:11
lifelesswhich may just be bad luck18:11
jmlheh18:11
jmlMy SMH target solver is faster in Go :)18:11
jmlanyway, I really do have to go.18:12
lifelesssmh?18:12
jmllifeless: sydney morning herald18:12
lifelessshoo18:12
rick_hjml: you played with pypy with it?18:12
rick_hI don't seen to find a ton of CPU bound issues that make pypy awesome, but wonder if something like that would fit18:13
jmlrick_h: no, but my #twisted friends all love PyPy and want to marry it18:13
* jml gone18:13
salgadolifeless, so, IIUC my ITunableLoop.__call__(chunk_size) should process up to chunk_size items in a single transaction and the loop tuner will tweak chunk_size on the next call according to the time my transaction takes. is that right?18:42
lifelessyes18:43
salgadolifeless, because of the way my code is structured, though, it would be more natural to process each item in a separate transaction (like http://paste.ubuntu.com/826728/), so that I can rollback if anything goes wrong when parsing the whiteboard of a given item.  that should play fine with DBLoopTuner, right?18:46
lifelessI don't know why it wouldn't, though you may find you need to ensure there is an open xaction etc18:48
salgadooh, right, other ITunableLoops seem to begin() a transaction when they're done processing a chunk18:50
lifelesssalgado: yes, the timing of chunks is to prevent impacting webapp requests19:01
salgadoI see19:03
lifelesssinzui: is ie8 'supported' for us ?19:05
sinzui:( yes. an a-level browser for YUI19:07
sinzuilifeless, we have a several IE bugs too19:07
lifelessso, per bugtriage I think 925154 should be critical ?19:07
lifelesspage-worked, now errors ?19:07
sinzuilifeless, I think so, somone needs to fix the other IE bugs too. We have sat on the fence about the issue because we claim they can always use the featurewithout JS...which has not been true for a year19:08
lifelessabentley: ^ a heads up - per BugTriage supported browser issues are critical19:10
sinzuilifeless, Lets ask deryck and rick_h to look at them and determine if these are critical because we do not offer an alternate for IE19:10
abentleylifeless: that's a supported browser?19:10
lifelessabentley: per sinzui, yes - A-level19:10
sinzuilifeless, this is YUI's doc on graded browser experience: http://yuilibrary.com/yui/docs/tutorials/gbs/19:12
abentleylifeless, sinzui: To my knowledge, no testing was performed on IE.  I was under the impression that our support was C grade.19:12
sinzuiabentley, I sure that is true. I do not have windows to ever do such a test19:12
abentleysinzui: A-grade for YUI does not imply A-grade for us, does it?19:13
rick_hyea, we've had broken IE for a while. I filed a bug a while ago on the portlets failing in IE causing JS to stop running19:13
sinzuiabentley, I certainly does not, we have always said. the feature gracefully degrades. We have written a lot of JS in the last year that does not degrade...that does not even have a html-form equivalent to fall back to19:14
lifelessso, we have a business rule that IE needs to work for the core functionality for OEM19:14
lifelesswhether we call it A grade or not is by-the-by19:14
sinzuilifeless, the rule is not that simple. We heard IE6 which is very difficult to support in modern libraries. Our own browser reports indicate that IE very minute. We have not had a report from OEM in two years that IE is broken.19:17
lifelesssinzui: I requested a confirmation that they still care about IE6 a few months back19:17
lifelessI will send another19:17
lifelessYUI still claims A for IE6, interestingly19:17
rick_hhmm, i thought they announced dropping that a year ago19:19
rick_heven Google apps all fail to work in IE6 any more19:19
rick_hmy wife's medical practice just finally got chrome this month because of it19:19
lifelessmaybe I am misreading that gbs page?19:19
lifelessah19:19
lifelesshttp://www.yuiblog.com/blog/2011/07/12/gbs-update/#grades-deprecated19:19
lifeless'graded browser experience with deprecated grades'19:19
sinzuirick_h, out Chinese counter-parts couldn't give a toss about what MS supports19:20
rick_hhttp://www.yuiblog.com/blog/2011/07/12/gbs-update/19:20
rick_hnvm, you found it19:20
rick_hso it's no longer a/b/c19:20
rick_hbut "tested" with failback19:20
* lifeless mails stakeholders 19:21
rick_hsinzui: sorry, just meant that as a "big web players" moving away from IE619:21
sinzuirick_h, right. IE was very difficult to support even when it was an A-grade browser19:21
rick_hsinzui: definitely not against getting things working in IE at all. Don't want to seem that way19:23
sinzuilifeless, maybe you missed rick_h's and I's exchange about JS support: https://lists.launchpad.net/launchpad-dev/msg08750.html19:24
sinzuiWe seem to be talking it now19:24
sinzuirick_h, I had a round of fixes about 6 months ago when we discovered that IE's event bubbling failed because something like YUI.Event was not being loaded for them19:25
rick_hhmm, the one I hit was that the portlets load into a <tbody> which IE won't allow and throws an error so those ajax loads fail19:26
rick_hsinzui: but yea, I'm sure there's a bunch of issues that'd take some time to work through19:26
rick_hespecially with buglistings as I don't know it's ever been IE tested19:26
lifelesssinzui: I saw that yes19:29
sinzuirick_h, we do not have a written doc about which browsers we support, and what happens when it is not supported.19:29
lifelesssinzui: I'm a big fan of inline rendering :)19:29
rick_hso I'm guessing that a landing page linked to Chrome Frame isn't going to fly? :)19:32
lifelesswe might get away with chrome frame activeX control19:33
lifelessif the factories can use it19:33
lifelesssinzui: I probably got all the facts wrong; please correct my mail if so :)19:38
sinzuiokay19:39
abentleysinzui: What do we do when one of our upstreams asks for their project to be deleted? https://answers.launchpad.net/launchpad/+question/18536119:39
abentleysinzui: nm, I see you're already on it.19:41
sinzuiWe deactivate if it is not being used by other communities. this one is. It is used by Ubuntu and the request is disputed. I did what I could by disabling answers and bugs19:41
sinzui^ abentley19:41
sinzuiThe question answer I gave was final19:41
rick_hlifeless: so if we go the route of the U1 response with full graded support, is there any chance we'd be able to grab some sort of test bed for tests or QA of these?19:46
rick_hlifeless: sinzui at least running the js test .html files via something like webdriver?19:47
* rick_h has been secretly looking to do something with just FF and Chrome, but IE gives a bigger excuse19:47
sinzuirick_h, We have talks about browser shots and a selenium test suite in the past.19:48
rick_hsinzui: yea, I've assumed so, I don't know the history so forgive repeat questions19:48
sinzuirick_h, our YUI tests are webkit because gecko is a bitch to compile and keep patched19:49
sinzuirick_h, note the irony in the webkit selection; it does not provide the JS engine used by firefox and chromium users19:50
cjohnston/wc/4819:51
rick_hchromium is webkit base at least, but yes. What I meant was I was trying to tinker with things like yeti and sst19:51
rick_hthings that fire up real browsers19:52
sinzuioh yuck. I ran away from that level of testing. I do not disagree with it. I just think I should not be writing those tests are maintaining the infrastructure. I use libraries and standards as my guide to writing yui tests19:54
rick_hsinzui: definitely, but I know in buglisting we hit issues between FF and chrome alone and I was trying to think of a way to run just the JS tests in both.19:55
sinzuiyes. I saw some comment about chromium oddities in the tests.19:55
rick_hit's something I'll continue to try to get running locally, but wanted to bring it up19:56
lifelessso jenkins + grid + browser tests yes20:06
lifelessor whatever - that sort of conceptual setup20:06
=== matsubara is now known as matsubara-afk
lifelessjames_w: I've now chatted with mthaddon, and the consensus is to go with one shared oops-tools instance20:37
lifelessjames_w: he is going to run it past james first20:37
james_wlifeless, ok, what were the reasons OOI?20:37
lifelessjames_w: user/dev perspective: single shared service, no need to figure out which service instance a particular oops-id belongs too20:38
lifelessjames_w: ops perspective - one instance to care for and feed; consumers can easily split out oopses on disk so resourcing can be partitioned off per-department if needed20:39
lifelessjames_w: dev perspective - less instances to run around upgrading when there are security issues or whatnots20:39
james_wok20:40
james_wwe unfortunately have rather a long deployment pipeline before we can send the oopses anyway, but at least this probably shortens it a bit20:41
lifelessThe mild overhead of going multitenant was an already expected cost as LP gets more SOAised20:41
lifelessssltunnel to rabbit should let you start pretty quickly20:41
james_wit's not that20:43
lifelessoh, ok :)20:43
james_wthere's branches queued up, but it needs code deployes20:44
james_wand packages in to CAT in co-ordination20:44
lifelessah20:44
james_wthen settings updates via puppet to turn it on20:44
=== salgado is now known as salgado-afk
abentleylifeless: could you explain what you mean about the script being misconfigured?21:49
abentleylifeless: Is it the fact that you're forcing the logs to INFO?  IMO, that's a stopgap, because the oops-id and message-id still aren't being logged at the same level as the error that they correspond to.21:54
lifelessabentley: the script was running with stdout/stderr capturing rather than with a --log-file=INFO:/... parameter21:54
abentleylifeless: Why include the -q parameter? is that for stdout/stderr?21:55
lifelessyes21:55
abentleylifeless: So the information that gets logged to stdout/stderr will still be deficient.21:56
lifelessabentley: it won't log anything to stdout/stderr21:56
abentleylifeless: Okay, so why include the -q parameter?21:56
lifelessso that it doesn't21:57
lifelessuhm, I don't mean this to be like playing 20 questions21:57
abentleylifeless: AIUI, the -q parameter prevents stdout/stderr from emitting INFO messages.21:57
lifelessabentley: yes; the goals for the setup are:21:57
lifeless - we log direct to a file anything we want to log21:58
lifeless - WARNING and above messages generate oopses for aggregation and analysis21:58
lifeless - stdout/stderr are empty except when the script runner itself catastrophically fails (and we then get immediate notification on lp-error-reports)21:58
lifelesshttps://lp-oops.canonical.com/oops.py/?oopsid=20e752ffc3566d6bee8c71afcadaa5a7 is an example of the oopses we get22:00
lifelesswhich seems to have a decent backtrace22:00
abentleylifeless: That all sounds good, though I'm not sure we really want to be logging at INFO level.22:00
abentleylifeless: Not with our volume.22:00
lifelessabentley: we can push less common stuff down to DEBUG22:01
lifelessabentley: I think, given that email has no guarantee of showing the user, that we probably do want to capture the messageid and oopsid every time.22:02
abentleylifeless: Yes, I can agree with that.22:04
lifelesswe log what, 8M web requests a day, on apache + on the zope access logs, and email is a tiny % of that22:08
lifelessso we could be quite fat and it wouldn't be an issue22:08
lifelessthats not a reason to log more (or less) just a data point from the ops aspect22:09
abentleylifeless: It still seems wacky *not* to emit the oops-id and message-id at error level, since we do want to know that about errors.22:09
lifelessthe problem is how to notify @ 'ERROR' when notifying @ 'ERROR' triggers an OOPS22:10
lifelessin regular logging parlance, the use of different levels is to permit filtering of output22:10
abentleylifeless: You stick it in the error that we're already emitting.22:10
lifelessabentley: how? that error is what becomes the oops - it shouldn't be getting written to the log file at all22:12
abentleylifeless: The message-id, I mean.  When the ERROR->OOPS handling is in place, the oops-id automatically gets logged at ERROR level, no?22:14
lifelessabentley: no, the oops-id generally isn't logged at all22:16
lifelessabentley: (because most of our scripts have no concept of UI - we find out via the oops reports)22:16
lifelessabentley: process-mail seems to do a lot of special handling that makes this worse than for other scripts22:17
abentleylifeless: Jobs that use the standard runner log their oops ids.22:18
abentleylifeless: I'm not sure how "concept of UI" relates to logging.22:18
abentleylifeless: I never read the oops reports, and when I get a bug report, I hit the logs to try and find out what happened.22:19
lifelessabentley: so the job runner does it's own generation and reporting - it doesn't depend on error behaivour22:20
lifelessabentley: the jobrunner reports oops ids at INFO22:20
abentleylifeless: true, true.22:20
abentleylifeless: I was just giving an example that contradicts "the oops-id generally isn't logged at all"22:21
lifelessah right22:22
lifelessso, the logging OopsHandler22:22
lifelessexists to ensure we capture the error for the reports22:22
lifelesswhich are (currently) our primary signal for the health of the system in advance of user initiated reports22:22
lifelessI'm not sure if it is possible to emit a logging message from a logging handler, but if it is, logging the oops id at info would match what our twisted equivalent, and the job runner equivalent, do.22:23
abentleylifeless: I would imagine the logging OopsHandler could also append it to the message.22:25
abentleylifeless: Though that runs the risk of losing the whole thing if oops handling fails.22:26
lifelessabentley: the oops config is built with a couple of fallbacks, but yes that is a risk22:30
StevenKwgrant: Oh, how should that query change?22:49
wgrantStevenK: It should change to not return a list of people, but existence of a matching row for a particular person.22:49
StevenKWHERE cs.date_expires < NOW() AND u.id = self.user (effectively) ?22:50
wgrantSomething like that.22:50
StevenKAnd drop the distinct, hand wave, hand wave22:51
lifelessooh pub holiday monday \o/23:55
StevenKWhat's the occasion?23:55
lifelesswaitangi23:56
StevenKlifeless: Have you yelled at your ISP today?23:57
lifelessnot yet23:59
lifelessour ajax detection log has broken :(23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!