/srv/irclogs.ubuntu.com/2010/09/03/#launchpad-dev.txt

=== spiv_ is now known as spiv
michaelh1Speaking of me,00:35
michaelh1is staging up at the moment?  The last entry on successful-updates is from four hours ago00:36
lifelessmichaelh1: see my question to spm just before the reference to you :P00:39
michaelh1Thanks.00:40
* spm quickly unignores lifeless, reads, adds the ognore back and crawls into a depper hole.00:40
michaelh1A different question: is there a way to edit and/or delete bug messages in LP?  If not, can I edit them through the API?00:40
lifelessognore eh?00:40
spmand adds extra typos for maximal confusions00:40
lifelessmichaelh1: no00:40
lifelessmichaelh1: they can be hidden I believe, in extremis00:41
michaelh1No to both the UI and the API?00:41
lifelessand the DB00:41
spmedited for super awesomely special cases as well; but *really* needs to be special.00:41
spmwich is another way of saying I can probably count on one hand the numbers of times it's been done.00:42
michaelh1spm: nope, I don't count for that.  We're using messages to add meta data to a bug.  The problem is when the meta data is wrong...  tags are inappropriate.00:42
wgrantWhy are tags inappropriate?00:43
michaelh1wgrant: we're using tickets to also track the upstream status of a bzr revision00:43
michaelh1wgrant: the revisions are fully qualified as in lp:gcc-linaro/4.5,revno=1234 and there are up to 20 entries per bug00:44
wgrantmichaelh1: You could keep them in the description.00:44
michaelh1wgrant: I'd like to have a timestamp and author that is easy to review.  Descriptions not a bad idea though...00:45
lifelessmichaelh1: why are tags inappropriate?00:51
michaelh1lifeless: see the reply to wgrant above00:51
lifelessmichaelh1: I asked this before and you linked to a wiki page which didn't help me understand your design constraints or needs at all00:51
lifelessso, I'm open to the idea of building a generic lookaside store, but I want the use cases pinned down very carefully00:52
lifelesssince there are so many ways such a thing could go wrong00:53
michaelh1lifeless: yip.  I'll try and describe it better over the next few weeks00:58
lifelessmichaelh1: some specifics - please look at https://dev.launchpad.net/ArchitectureGuide for my big picture concerns01:05
lifelessmichaelh1: and for this thing - do we need:01:05
lifeless - search facilities (e.g. metadata values, modifiers, modification times)01:06
lifeless - logs/journalling01:06
lifeless - display UI in web pages01:06
lifeless - would this be better served by specific integration-information e.g. 'patches to forward upstream'.01:06
pooliegary_poster: you could link that lep from the relevant bug01:16
pooliehi lifeless01:16
lifelesshi poolie01:16
gary_posterpoolie, I'm trying to link all sub-bugs to LEP, but yeah, you are right, there is that big one, I forgot, thanks01:17
gary_posterCurrently fighting moin syntax though :-P01:17
lifelessspm: so, in all seriousness - what is the status of staging01:18
spmgah. sorry distracted. looking...01:18
* lifeless offers spm some eye for the eyeballs01:19
spmit's building an importd atm; so "soon". I'd suggest counted in smller minutes. maybe 15-30?01:19
lifelessgary_poster: oh you're still around01:20
lifelessexcellent. I need help.01:21
gary_poster:-P01:21
lifelesscan you do voice? just so that I can have the code full screen01:21
gary_posterI'm here because I didn't have time to get the stuff I needed to get done today, done01:21
lifelessL<01:21
lifeless:<01:21
gary_poster:-)01:21
lifelessLet me sketch my issue, and you can decide if mercy is appropriate01:21
lifelessI have a patch, lp:~lifeless/launchpad/oops01:21
gary_posterok looking01:21
lifelesswhich replaces the request_statements list with a Timeline object01:22
lifelessits failing one test. xx-opstats.txt line 18601:22
gary_posterand trunk doesn't, I assume?01:22
lifelessright01:23
gary_posterok01:23
lifelessI've thrown a bunch of print statements at timeout and soft timeout setting code01:23
lifelessand patched SoftTimeoutView in an attempt to understand it01:23
lifelessbut I'm getting less clue not more01:23
lifelesshttp://pastebin.com/xBnLuiPR has my 'wth' edits01:24
lifelessOne of the things confusing me is that AFAICT the test should *never have passed*01:24
lifelessthe page it requests does not DB access, and only DB access can set a hard timeout01:24
gary_posterheh, one of those01:24
lifelessI added DB access in, but that didn't work01:25
lifelesssetting the soft time out to 200ms seems to work01:25
lifeless20 doesn't reliably01:25
gary_posterhave to get boys out of bath, but back in a bit01:25
lifelessI'd be ok with a 'does not blow up in the next devs face' tweak01:25
lifelesskk01:25
lifelesswill rabbit on a bit01:25
lifelessIdeally I'd actually fix it01:26
lifelessone thing i'm sure is happening is this:01:26
lifeless the db code I addedfire, a timeout is raised01:26
lifelessbut the end request hook sees the request as having no oops id01:27
lifelessand the oops code that puts the oops id into the request isn't running until *after* the end request hook has set a soft request oops id01:27
gary_postersame request?01:33
lifelesspretty sure01:34
gary_posterfly by--still putting kids to bed01:34
lifelesswill check01:34
lifeless    + I (<class 'storm.exceptions.TimeoutError'>, TimeoutError(), <traceback object at 0xf1a16c8>)01:42
lifeless    + raising request= <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312 (<class 'storm.exceptions.TimeoutError'>, TimeoutError(), <traceback object at 0xf194b48>)01:42
lifeless    + G01:42
lifeless    +             UPDATE SessionData SET last_accessed = CURRENT_TIMESTAMP01:42
lifeless    + end-hook-set-soft <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 28005331201:42
lifeless    + raising request= <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312 (<class 'canonical.launchpad.webapp.errorlog.SoftRequestTimeout'>, SoftRequestTimeout(None,), None)01:42
lifeless    + save fool01:42
lifeless    + raised-soft <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 28005331201:42
lifelesssave fool is where the oops is set on the reuqest01:42
lifelessso - same request01:42
lifelessand raising is being called into01:42
thumperhmm... lunch is caling01:49
lifelessso _makeErrorReport is barfing01:49
lifelessand its silently swalloed01:49
lifelessgary_poster: thanks01:49
gary_posterlifeless, oh solved01:50
gary_posterawesome01:50
lifelessgary_poster: well not solved01:50
lifelessbut progress01:50
gary_posterI had actually built a branch and was proceeding to start a test01:50
gary_postergreat01:50
lifelessgary_poster: its failing to call sys.exc_info() ><01:51
gary_posterheh01:51
lifelesss/call/print the exc_info/01:51
lifelessI suspect a securityProxy stabbing me01:51
lifelessor something01:51
lifeless    + 'NoneType' object has no attribute 'microseconds'01:51
lifeless*blink*01:51
lifelessgary_poster: is there something that one can raise, which *will* escape the publication machinery and give a sensible diagnostic in pagetests?01:53
gary_posterdo not understand "escape the publication machinery"01:54
gary_poster(IOW, the goal)01:55
gary_posterlifeless ^^^ (and I need to get off the computer in 4 min for mental health :-) )01:56
lifelessgary_poster: I was thinking of ways to make this easier to debug in future01:56
lifelessgary_poster: thank you very much for helping me escape my own mental trap01:57
gary_posterheh, I'm not sure what I did to help, but I'm glad I did01:57
lifelessgary_poster: I think I'm good now. May brainstorm later about how this might have been easier to diagnose.01:57
lifelessgary_poster: you were patient; asked good questions, and sympathised.01:57
lifelesscombination let me take enough of a step back to apply my own awesome skillz :>01:58
lifelessnow to find this ztrace log thing01:59
gary_poster:-) cool.02:00
gary_posterhave a good day02:00
spmlifeless: if you haven't noticed; staging lives102:09
lifelessspm: \i/02:11
lifelessspm: erm02:11
lifelessspm: something is wrong02:11
spmoh?02:11
lifelessis https://staging.launchpad.net/successful-updates.txt meant to match with the revision in the footer of staging ?02:12
spmshuold02:12
lifelessspm: ...02:13
spmso I see02:13
spmis def revno 9738 in the live tree.02:14
* lifeless is skeptical02:15
spmbzr-version-info.py is 9710 tho. how... curious02:15
spmlaunchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revno02:15
spm973802:15
spm^^ bug in bzr then? :-P02:15
lifelessbug in the lp build thingymajig02:16
lifelessfile a bug with the details please? lp-foundations02:16
spmwait a sec... something's not right.02:17
spmhttps://pastebin.canonical.com/36696/ <== build date vs date02:18
spmand the rev id for that matter02:18
lifelessI agree02:18
lifelesssomething is wrong :)02:19
lifelessrun generate_version_info02:19
lifelesssee what happens02:19
spmheh, was kinda hoping that was "Oh That's *X*"02:19
lifelessgary_poster: a) thanks again; its all sorted and I'm much happier. b) what happened to your mental health :)02:19
gary_posterlifeless: (a) yay! (b) it appears to be hosed ;-)02:20
lifelessgary_poster: ah, welcome to my world... we can be insane together :)02:20
gary_posterlol :-)02:20
spmum. find . -name 'generate_version*' <== nothing found. ????02:20
lifelessgenerate_version_info02:21
lifelesshmm02:21
lifelessgrepping02:21
lifelessgary_poster: you may like knowing that the fix to my issue also means that every sql statement *attempted* on 'LaunchpadDatabase' will be logged.02:22
gary_posterthat sounds cool02:22
lifelesschanging the confusion that existed around attempted-but-not-executed02:22
gary_posterright02:23
lifelessfor now I'm logging them with a duration of 99999902:23
gary_posterheh02:23
gary_posterthat'll stand out, I hope02:23
lifelesscan dig into the oops stack to see if we can do '-' or something in future02:23
gary_postersure02:23
lifelessspm: scripts/update-bzr-version-info.sh02:24
spm*blink*02:25
spmlaunchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ scripts/update-bzr-version-info.sh02:25
spmSkipping bzr-version-info.py update; already at revno 971002:25
lifelessbzr 2.202:25
lifelessbet you its a bug02:25
lifelessplease file; launchpad-foundations + bzr tasks; losa tag, critical (it will really mess us up if that doesn't right itself)02:26
spmbzr --version ==> Bazaar (bzr) 2.2.002:26
thumperwallyworld_: best tales docs I've found http://www.owlfish.com/software/simpleTAL/tal-guide.html02:26
thumperlifeless: that's that all about?02:27
lifelessspiv: can you please assist ^ >02:27
lifeless?02:27
lifelessthumper: the thing that updates the revision in the bottom right of staging and edge isn't (updating)02:27
thumperah02:27
thumperok02:27
lifelesswhich suggests, very strongly, a bzr bug02:28
pooliethis twitter oauth thing strongly reminds me of the "we must have unique per app tokens" launchpad used to have02:30
pooliei think we've now seen the light, fortunately02:31
wgrantFor anonymous access, sure.02:31
poolieyes02:32
pooliethis was proposed for a long time as a reason we couldn't allow anonymous access02:32
wgrantIndeed.02:33
wgrantBut Twitter's situation is different, as it's authenticated.02:33
spmlifeless: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/62921202:36
_mup_Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>02:36
lifelesswe can have multiple apps per token, its just a config + trust thing02:36
lifelessspiv: if you could look at bug 629212 that would be awesome02:37
_mup_Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>02:37
lifelessspiv: I realise its an interupt; but we're hitting release-week on Monday, and thats already pretty harsh for lp :(02:37
poolielifeless: he's out atm (or he was)02:40
lifelesspoolie: oh sure, it can wait for later today02:43
lifelesspoolie: I just wanted to be clear about it all02:43
lifelessok, thats pretty impressive, I admit it: 2455  OOPS-1706L64    ProductSeries:+bugs02:46
lifelessthe first number is sql queries02:46
wgrantYou should have seen Soyuz a year or so ago.02:51
wgrantIt was better than that :)02:51
lifelesswgrant: without timing out ?02:52
wgrantlifeless: ... just.02:53
lifelessfoods time02:54
mwhudsonlifeless: well, that's why we have such beefy db servers :-)03:19
lifelessmwhudson: ><03:24
lifelesswtf03:29
lifelesswe still query AccountPassword?03:29
wgrantYup.03:30
wgrantIt's still needed for basic auth.03:30
wgrantBut I don't know why it's queried every time.03:30
lifelesswe don't support basic auth anymore do we? I mean, I know its *enabled*, but *support* is different03:31
wgrantRight.03:31
wgrantIt's enabled and dangerous.03:31
wgrantSo should be deleted.03:31
lifelesspropose a patch03:31
lifelessDoIt03:31
lifelessmake it controlled by a feature flag03:31
lifelessif after the rollout folk are screaming03:31
lifelesswe can add a rule to enable it03:32
wgrantOoh, good idea.03:32
lifelessand transition the remaining user(s)03:32
lifelessif noone screams by 10.10, we nuke the code completely.03:32
wgrant(and the table)03:35
lifelessspm: ping03:35
spmyo03:35
lifelessspm: on staging03:35
lifelessI have some evidence suggesting it *is running 9710*03:35
lifelessspm: I'd like to dig deeper03:35
lifelesshere is my evident03:35
lifelessmy patch 11483 in stable preloads is_valid_person for https://staging.launchpad.net/ubuntu/+assignments03:36
stubWe still have AccountPassword to support the test OpenId provider the testsuite uses. Dropping it is part of the roadmap I drafted yesterday.03:36
lifelessbut03:36
lifelessstub: awesome03:36
lifelessspm: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1707S1903:36
lifelessshows 191 SELECT ValidPersonCache.id FROM ValidPersonCache WHERE ValidPersonCache.id = %s LIMIT 1 calls03:37
lifelessspm: on edge, when I checked yesterday, the page didn't have that03:37
wgrantstub: I guess it really just needs a place to enter an email address?03:37
wgrantAnd maybe a button to fail auth, to make testing easier.03:37
stubwgrant: Yes, and a box to override the OpenId Identity to test some edge cases.03:38
wgrantSince that's difficult at the moment.03:38
lifelessspm: db-stable 9738 is past 1148303:38
wgrantAh, true.03:38
lifelessspm: so, in theory, it should not be issuing those queryies03:38
lifelessspm: so I think that staging right now is still useless03:38
spmmmmmmmmmmmm03:39
spmfrown evem03:39
lifelessspm: "grep 'def _specification_sort' lib/lp/blueprints/model/specification.py"03:39
lifelessspm: if that finds a function, lets cross check that whats showing on the web *is running from where you think its running*03:40
lifelessspm: if it doesn't, then what we're seeing on the web is consistent with the source code you're looking at, and we can look at bzr etc03:40
spm was going to ask do you have a way to confirm existance... ta03:42
spmlifeless: most curious. zero results found. ??03:42
lifelessthen we're not running 973803:42
lifelessok03:42
spmorsum03:42
spmbzr missing; perhaps?03:42
lifelessbzr revision-info --tree03:42
lifelessbzr revision-info03:42
lifelessrun both please03:42
spmoki03:42
lifelessbug 62921203:43
_mup_Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>03:43
spmwow03:43
spmlaunchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info --tree03:43
spm9710 launchpad@pqm.canonical.com-20100827184922-1f7jszjyrqw4dasv03:43
spmlaunchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info03:43
spm9738 launchpad@pqm.canonical.com-20100902161314-dy8vhr3l8epgpo2s03:43
lifelessspm: ok, the tree is out of date03:43
lifelessI'll update 629212 with that03:43
lifelessspm: please run 'bzr update'03:43
lifelessand make run, or however you kick things off03:43
spmi wonder if this'll break staging rather spectacularly - the DB would have been set via 9710....03:44
lifelessspm: almost certainly03:44
spmas we just rsync the code around. whee.03:44
lifelesswell03:44
lifelesswhere does this code start?03:44
lifelesswhats the source of the rsync03:44
lifelessand if you say sodium, el crasho, I'm going to cry.03:44
* spm hates to see a grown man cry and says nothing03:45
lifelessok, so you probably just need to run 'bzr update' on sodium03:45
spmlaunchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr update03:45
spmbzr: ERROR: This tree contains left-over files from a failed operation.03:45
spm    Please examine /srv/staging.launchpad.net/staging/launchpad/.bzr/checkout/limbo to see if it contains any files you wish to03:45
spm    keep, and delete it when you are done.03:45
lifelessand kick the whole thing off again03:45
* spm puts head in hands and tears up a little03:45
lifelesssodium will be like that too03:46
lifelesslets fix at root03:46
spmyeah, that's where I'm heading03:46
spmdamn. sodium is really futzed with that tree.03:48
lifelessbug updated03:48
lifelesswhats the sodium rt ? I cannae see it03:48
spmthe h/w fail one? not sure if it even has one tbh03:49
lifelessit had one03:50
lifelessit may have been closed after the chassis was replaced.03:50
spmhave moved the existing db-stable to *.busticated; doing a brand new branch et al to be sure to be sure03:51
lifelessthanks03:52
lifelesssorry for the interrupt03:52
lifelessspm: I've updated the bug and the bodyswap RT ticket03:58
lifelessstub: sodium has less CPU and RAM than it did, its why the PPR may be running into swap when it wasn't before.03:59
spmoh no, is fine; and yeah that sounds about right.03:59
lifelesshah04:01
lifelessrt fail04:01
lifelessit mades a new ticket04:01
lifeless41200 ><04:01
lifelessspiv: unping04:01
stublifeless: ahh04:09
lifelessstub: I only found out reading closed rt tickets looking for the sodium thing04:10
lifelessspm: it would be nice to share those things btw, just saying04:10
lifeless(and I know it wasn't up to you :>)04:11
lifelesshttp://ayende.com/Blog/archive/2010/08/31/it-really-happened-legacy-programmers-tales.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AyendeRahien+(Ayende+%40+Rahien)04:13
stubIs sourcepackage one word or two in the Launchpad vocabulary? I can never remember. I think I even asked this yesterday...04:17
lifelessone I think04:17
stubBinaryPackageBuild thinks it is two, pretty much everything else thinks its one. But BPB is new and perhaps the new way forward?04:19
wgrantstub: Two.04:20
wgrantWhat says it's one?04:20
lifelessspm: Segmentation fault in cron still waorries me04:20
stubwgrant: hyphenation of database column names... haven't looked for capitalisation in Python source code.04:24
stubbut hyphenation can be wonky from when we didn't underscore between words.04:25
wgrantstub: Pretty much everything except BPB predates the hyphenation.04:25
wgrantExactly.04:25
stubTa.04:25
wgrantIt's been capitalised SourcePackage since like 2005.04:25
spm`pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info --tree04:30
spm`9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x04:30
spm`pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info04:30
spm`9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x04:30
spm`lifeless: ^^04:30
lifelessspm`: \o/04:30
lifelessspm`: time for the acid test04:31
spm`:-)04:31
thumper:(05:05
thumper'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout' spurious test failure05:05
spm`lifeless: right. timing worked against us a bit there; have stabbed the (still doing it wrong) restore;  manually ensureed we have the latest pony; and the next restore in .... 4 minutes, should push out 973905:10
lifelessspm`: if my fingers cross any harder they will break05:10
spm`:-)05:10
lifelessstub: did you remember where our new storm base base class is ?05:10
stublifeless: I don't think anyone has created it yet.05:11
lifelessstub: is it ok to use storm.base.Storm ?05:12
lifelessstub: on completely new things?05:12
lifeless(TimeLimitedToken for the librarian )05:13
stubI use storm.locals.Storm, but yet. Better to create a three line subclass in the LP tree though and use it. I don't think anyone knows where it should live though.05:13
stubc/yet/yes/05:13
lifelessstub: I want to get this branch in shape for Abel05:13
lifelessif we're already using Storm directly, anywhere, we'll need to do a mass migration, so its no harder.05:14
stubWe already use Storm directly elsewhere.05:14
lifelessthanks05:14
=== almaisan` is now known as al-maisan
lifelesstime to rebuild my session db05:18
lifelessstub: will you want a sql patch for the tokens05:18
lifelessstub: or just grab the table defn I put in session.sql05:18
lifelessspm`: staging mail came through, but staging is down.05:24
lifelessspm`: is that normal, monsieur backtick ?05:24
spm`yeah. I manually rekicked it.05:24
lifelessspm`: just a minute ago ?05:24
spm`about 10ish05:24
spm`15 I kicked; it should have restarted abot 10 ago05:24
spm`and isn't. sigh.05:25
lifelessspm`: I saw mail 6 minutes or so ago05:25
spm`yeah. something funkies up05:26
stublifeless: I need to manually apply things, so no need for a database patch. Needs a note on 'unusual deployment' on the rollout page, because things will explode if I forget.05:26
stublifeless: Staging will explode too if it lands without work, so I guess I should make the changes to prod and staging just before you land.05:27
spm`wowo. something really blew up.05:31
spm`Fri Sep 3 04:22:22 UTC 2010 Applying database updates and permissions to DB05:31
spm`Traceback (most recent call last):05:31
spm`  File "./upgrade.py", line 13, in <module>05:31
spm`    import _pythonpath # Sort PYTHONPATH05:31
spm`ImportError: No module named _pythonpath05:31
spm`stub: ^^ staging restore just now05:31
lifelessbuildout05:31
stubYay, it isn't me!05:32
spm`although this looks bad too: urllib2.URLError: <urlopen error [Errno 2] No such file or directory: '/srv/staging.launchpad.net/staging/launchpad/download-cache/dist/setuptools-0.6c11-py2.6.egg'>05:32
spm`ha05:32
lifelesswhats the staging librarian url ?05:32
lifeless        ZopeXMLConfigurationError: File "/home/robertc/launchpad/lp-branches/working/lib/canonical/shipit/browser/configure.zcml", line 7.4-12.4905:34
lifeless        ImportError: No module named cachedproperty05:34
lifeless*hate hate hate stab stab stab*05:34
spm`lifeless: librarian.staging.launchpad.net ? or did you mean something else...05:37
wgrantlifeless: Split split split.05:37
wgrantspm`: Er, really?05:37
wgrantI really hope not.05:38
lifelessspm`: the staging equivalent to launchpadlibrarian.net05:38
spm`oh that one. right. staging.launchpadlibrarian.net05:39
wgrant(phew)05:39
spm`you may begin to see a pattern here... :-)05:39
lifelessthumper: can you please put rt 41202 to pri 8905:40
thumperlifeless: possibly, trying to login05:41
lifelesswhere do shipit bugs get filed?05:41
lifelesslike 'we can't rollout until this is fixed' style shipit bugs05:42
wgrantshipit05:42
wgrantBut they're normally fixed by LP...05:42
lifelessyeah05:42
lifelessI have a sketchy patch05:42
thumperlifeless: done05:42
wgrantBut it can't be broken, can it?05:42
wgrantec2 tests it.05:42
wgrantSo does buildbot.05:42
lifelesswgrant: guess what05:43
lifelesswgrant: go on guess05:43
lifelesswgrant: I dares you05:43
wgrantMy guess would be python version differences, but that doesn't seem relevant here.05:45
wgrantOr buildbot wasn't update-sourcecoding.05:45
wgrantOr prasé isn't.05:45
* spm` notes the 'prasé' and smiles :-)05:45
lifelesswgrant: zcml barfs in includes that don't affect the core test suite don't break it.05:46
lifelesswgrant: or something ~= to that05:46
wgrantlifeless: O_o05:46
lifelesswgrant: anyhow, I dunno.05:46
lifelessmaybe allenap cheated and didn't ec2land, in which case its going to go boom and we'll be in testfix in, oh, 2.5 hours05:47
thumperec2 giving WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!05:47
thumperanyone else seeing this?05:47
thumperis there a reason?05:47
lifelesswgrant: https://bugs.edge.launchpad.net/shipit/+bug/629259 for your pain05:48
_mup_Bug #629259: cachedproperty was imported by shipit <canonical-losa-lp> <ShipIt:Confirmed> <https://launchpad.net/bugs/629259>05:48
mwhudsonlifeless: have you run utilities/update-sourcecode?05:49
lifelessmwhudson: I'm checking that now, but pqm barfed at me with the same error earlier05:49
mwhudsonah05:49
lifelessmwhudson: on a production config change05:49
mwhudsonoh there's probably some config-manager branch that needs to change05:50
mwhudsonmaybe05:50
lifelessyes05:50
lifelessI suspect so now having checked05:50
lifelesssourcecode has been udpated05:50
wgrantthumper: That just means you're lucky.05:50
wgrantthumper: You've got an instance reusing an old IP.05:51
mwhudsonthe config-manager config is in the lp-production-configs branch iirc...05:51
thumper:(05:51
lifelessmwhudson: yes, had a lovely race condition there last night05:51
thumperwgrant: three times in a row05:51
thumpermust be extra lucky*05:51
lifelessactually its a bug05:51
lifelessif you're using ec2 land/test05:51
mwhudsonthumper: rm ~/.ec2/known_hosts05:51
lifelessthat is meant to zap stuff and set it up for you05:52
wgrantAh, right.05:52
thumperlifeless: using ec2 test05:52
mwhudsonec2 test should probably use the ssh -RogerMeHarder options to not get it to check keys at all05:53
mwhudsonif it has them even05:53
lifelessmwhudson: every now and then your uk heritage shines through ;)05:54
thumperso I can fix this by removing the known_hosts05:54
thumper?05:54
mwhudsonthumper: yes05:55
mwhudsonuntil next time05:55
mwhudsonlifeless: >:)05:55
* thumper sighs05:55
lifelessspm`: when is the next edge update ?06:01
lifelessspm`: can we abort it ? tip of stable is not safe to rollout.06:01
=== ]reed[ is now known as [reed]
lifelessI really dislike this Robert Collins(bugnumnber)...06:07
lifelessin emails06:07
wgrantAs in the fake bug email address?06:08
lifelessyes06:08
wgrantYes :(06:09
lifelessok -> airport. bbiab06:12
spm`lifeless: about one hour, and yes.07:02
=== spm` changed the topic of #launchpad-dev to: Code hosting offline 8.00-9.30 UTC on Friday 3rd September for unexpected hardware maintenance. http://is.gd/eRMxF | Launchpad Development Channel | Performance Tuesday | Week 3 of 10.09 | PQM is OPEN | firefighting: edge rollouts disabled | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
* StevenK tries to work out what Zope is telling him07:37
wgrantStevenK: What's it complaining about?07:37
StevenKwgrant: ComponentLookupError07:40
* StevenK grumbles at zcml07:42
lifelessspm`: can we stop it ?07:59
spm`lifeless: it is: https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus#Edge%20Updates07:59
lifelessspm`: thanks07:59
spm`np.07:59
spm`I should relaly restart bip to get rid of that '`'07:59
=== spm` is now known as spm
lifelessI'll put the rollback into pqm in a few minutes08:00
spmcoolio08:00
lifelessspm: hows staging?08:01
spmstill unwell I assume, haven't chased tbh.08:03
=== henninge_ is now known as henninge
spmlifeless: revno 9741 should be being rolled out in about 5 mins. (fingers crossed etc etc)08:38
spmon staging, as in.08:39
lifelessspm: great, thanks08:46
adeuringgood morning08:46
lifelessspm: bugger08:56
lifelessspm: can you stamp on pqm for me08:56
spmwich part?08:56
lifelessthe job I just mistargeted to prod-devel08:56
spmoops08:56
spmlifeless: lucky you08:58
StevenKIs there some way I can cleanup my lp-sourcedeps/eggs directory?08:58
spmCommit message [[rs=deryck][ui=none] Rollback rev 11491.] does not match commit_re08:58
lifelessspm: thanks08:59
lifelesshopefully this one will get through09:02
lifelessadeuring: I have a rollback landing09:02
adeuringlifeless: I've seen your mails (but haven't had yet enough coffee)09:03
lifelessadeuring: thats cool09:04
lifelessadeuring: I"m starting to update bugs and stuff now that I'm back home from the airport09:04
lifelessadeuring: simply letting you know so that we don't roll it back twice :P09:04
adeuringlifeless: ok. so, how shall we proceed regarding the need to get the retracers working again?09:04
lifelessthe firewall port has been opened09:05
lifelessadeuring: if lp is serving the restricted librarian urls out now, it should be working09:05
adeuringlifeless: right09:05
lifelessadeuring: have your coffee; catch up on mails09:05
adeuring;)09:06
lifelessthat will give me time to update bugs to make the plan as clear as I can09:06
spm<adeuring> lifeless: I've seen your mails (but haven't had yet enough coffee) <== Man after my own heart. I too find dealing with lifeless and his emails to be much smoother post coffee. ;-)09:12
allenaplifeless: Have I broken something?09:21
lifelessallenap: I'm not sure09:22
allenap:)09:22
lifelessallenap: have you landed the shipit change?09:22
allenaplifeless: Yeah, that landed over a week ago I think.09:22
lifelessallenap: for cachedproperty use?09:23
stubGah. I'm always bitching about other people adding stdout noise to the test suite, and I just notice some from one of my recent landings :-P09:23
allenaplifeless: Yep.09:23
lifelessallenap: ah, but your lp branch only landed last night?09:23
allenaplifeless: Yeah, but versions.cfg only got bumped in my branch too.09:23
lifelessallenap: so landing a production config change barfed this morning on shipit zcml09:24
allenaplifeless: Sorry, utilities/sourcedeps.conf09:24
lifelessallenap: because that doesn't use versions.cfg, we have to land branches that cause incompatibilities in lockstep, not way apart.09:25
lifelessallenap: it may be fixed now, I'll toss a test change at it.09:25
allenaplifeless: Was shipit tip pulled into production?09:26
lifelessallenap: yes, thats how the older stuff works09:27
allenaplifeless: Gah, I didn't realise that could happen.09:27
lifelessallenap: not live production, but the mechanism by which production config changes are made09:27
lifelessallenap: its a hangover, it should be changed.09:28
allenaplifeless: Okay. I'm sorry about that. I meant to land the two branches close to one another, then realised there were problems with the Launchpad part, and thought it would be okay - because of sourcedeps.conf - to leave shipit as is.09:28
lifelessallenap: no worries, now I know what went on it makes sense. I'm checking now that there isn't a persisting problem09:29
wgrantWhy doesn't production use sourcedeps.conf?09:29
allenaplifeless: Cool, thanks for sorting this out while I slept unaware :)09:29
lifelesswgrant: pqm landing production-configs uses config-manager09:29
lifelessallenap: I think your branch landing has sorted it09:29
wgrantlifeless: Yes, but why?09:29
lifelesswgrant: 20:30 < lifeless> allenap: its a hangover, it should be changed.09:30
wgrantlifeless: Your clock is wrong.09:31
wgrantBut, ah.09:31
StevenKwgrant: Pedant.09:32
lifelessoh foo, I can't land stuff till codehosting is back.09:32
* lifeless files a bug for redundandcy09:32
StevenKA redundant bug?09:33
jkakarI think the diff on the merge proposal page would be more readable if it used a fixed-width font.09:42
wgrantjkakar: It does.09:43
wgrantjkakar: Are you using edge, and also don't have the UbuntuBeta font installed?09:44
jkakarwgrant: I'm using edge and I have the UbuntuBeta font installed.09:45
wgrantHmm.09:46
jkakarThe diff is shown using the UbuntuBeta font.09:47
wgrantNot for me :(09:47
wgrantMaybe my UbuntuBeta is old.09:48
wgrantHm, no.09:48
lifelessthumper filed a bug for this on launchpad-web today, I think.09:50
lifelessor someone did.09:50
allenap+1 for diffs in cursive.09:50
StevenKArgh, no09:50
StevenKallenap: Evil!09:50
wgrantlifeless: There is a bug for it, yes.09:50
wgrantBug 62918109:50
_mup_Bug #629181: code review diffs are in a proportional font on edge <launchpad-web:New> <https://launchpad.net/bugs/629181>09:50
wgrantHence the question about whether the font was installed.09:51
jkakarallenap: p:first-letter { font-size: 200% }09:51
lifelesswgrant: ah, it was you ;p09:51
jkakarMight as well make the diff look like an old manuscript. :)09:51
allenapjkakar: Oh, oh, that's so brilliant.09:51
wgrantlifeless: It wasn't me.09:51
wgrantI just saw it fly past.09:51
lifelessoh, someone.09:52
StevenKlifeless: mwhudson09:52
lifelessah yes thanks09:53
lifelessadeuring: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/39596010:02
_mup_Bug #395960: proxying user supplied files via the launchpad appserver domain has security and performance issues <librarian> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/395960>10:02
lifelessadeuring: thats the bug we'll be fixing10:03
adeuringright10:04
lifelessadeuring: codehosting is back if you want to pull the private-librarian branch10:09
* adeuring is looking10:09
lifelessadeuring: there are instructions on https://code.edge.launchpad.net/~lifeless/launchpad/private-librarian/+merge/31020 about getting going with it10:10
lifelessadeuring: hey, so I've updated the bug and the mp and chased a few things around10:31
adeuringlifeless: ok10:31
lifelessadeuring: so I'm going to work on this monday, tuesday etc10:32
adeuringcool10:32
lifelessadeuring: if you'd like to collaborate that would be awesome10:32
lifelessadeuring: I understand deryck is giving you a slab of time to do this, if you're interested.10:32
adeuringlifeless: sure, I am10:32
lifelessawesome10:32
lifelessso, its heading towards late for me; I'd love to give you a brain scan to get my internal state on it10:33
lifelessbut failing that, perhaps you can poke around and ask / discuss to get some info before I crash?10:33
adeuringlifeless: thanks :) Are the scanners yet internet ready?10:33
lifelessadeuring: the scanners?10:34
lifelessoh10:34
adeuringlifeless: for brain scans10:34
lifelessfor the scan. No. Thats why I can't10:34
lifelessI have them in the doorways at the house; instant backup10:34
adeuringlifeless: cool. Anyway, I'm reading the MP, but I'm not that fast ;)10:34
michaelh1Hey, is the API up at the moment?  I'm getting strange errors using launchpadlib...10:45
=== al-maisan is now known as almaisan-away
=== mthaddon changed the topic of #launchpad-dev to: Launchpad Development Channel | Performance Tuesday | Week 3 of 10.09 | PQM is OPEN | firefighting: edge rollouts disabled | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
lifelesshmm, its not perf tuesday anymore ;P10:55
=== lifeless changed the topic of #launchpad-dev to: topic
lifelessbah10:55
=== lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.09 | PQM is OPEN | firefighting: edge rollouts disabled | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
seb128hi10:58
seb128is there any known bug with untagging bugs with the launchpablib api recently?10:58
seb128the retracers are running but seems untagging is not working10:58
seb128though the recent retracer crashed on a10:59
seb128lazr.restfulclient.errors.HTTPError: HTTP Error 500: Internal Server Error10:59
seb128...10:59
seb128      Sorry, you can't upload or download files from Launchpad at the moment,10:59
seb128      because we're doing feng shui in the server room.  Normal service will10:59
seb128      resume within an hour.10:59
lifelesshmm, thats a librarian related error10:59
seb128I guess that's different from the untagging10:59
seb128it failed to add the stracktrace10:59
lifelessmthaddon: sorry for the interrupt; have both librarians been behaving ?11:00
lifelessseb128: yeah, thats the upload process11:00
mthaddonnot aware of any librarian problems11:00
seb128ok, I've restarted the retracer11:01
seb128let's see if that was a one time issue11:01
lifelessadeuring: well I hope it made sense ;)11:21
adeuringlifeless: the discussion in the mp? sure11:21
lifelessadeuring: gnight11:36
lifelessadeuring: I'll probably poke at this in the weekend; please do push up any stuff you do on top of it and mention in the MP or something11:37
adeuringlifeless: ok, will do. nice weekend!11:39
noodles775Night lifeless11:42
jtvArgh.  This familiar to anyone?  It doesn't seem related to anything I did in my branch.11:49
jtvZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/ftesting.zcml", line 17.4-17.3511:49
jtv    ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/configure.zcml", line 147.4-148.4211:49
jtv    ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/shipit/configure.zcml", line 7.4-8.2811:49
jtv    ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/shipit/browser/configure.zcml", line 7.4-12.4911:49
jtv    ImportError: cannot import name cachedproperty11:49
deryckMorning, all.11:57
noodles775jtv: bug 629259 ?11:57
_mup_Bug #629259: cachedproperty was imported by shipit <canonical-losa-lp> <ShipIt:Fix Released> <https://launchpad.net/bugs/629259>11:57
jtvnoodles775: ah thanks!11:57
jtvhi deryck11:57
seb128re11:57
seb128retracer crahed again the same way11:58
seb128lazr.restfulclient.errors.HTTPError: HTTP Error 500: Internal Server Error11:58
deryckhi seb128.11:58
seb128      Sorry, you can't upload or download files from Launchpad at the moment,11:58
seb128      because we're doing feng shui in the server room.  Normal service will11:58
seb128      resume within an hour.11:58
seb128hey deryck11:58
seb128deryck, seems we progress, the retracers have access to the crashes now11:58
jtvnoodles775: I wonder why suddenly this started affecting me around the time I renamed my branch directory…11:59
seb128they just get an error when trying to upload the retraced stacktrace11:59
deryckseb128, ok, let me check on that.11:59
jtvnoodles775: ah, a devel merge was also needed.12:04
=== matsubara-afk is now known as matsubara
=== salgado is now known as salgado-brb
=== almaisan-away is now known as al-maisan
=== salgado-brb is now known as salgado
=== al-maisan is now known as almaisan-away
=== Ursinha is now known as Ursinha-lunch
=== salgado is now known as salgado-lunch
marsbeuno, something interesting for you and the design group: http://uxmovement.com/design-articles/faster-with-top-aligned-labels17:11
beunomars, based on research, that's how I defined this: https://wiki.canonical.com/UserExperienceDesign/WebGuidelines/Forms17:13
beunonot that I'm allowed to do design anymore (?)17:14
marsbeuno, that's awesome17:14
marsthe page I mean :)17:14
beunoheh17:15
beuno(apologies for all you lurkers not being able to see that page)17:15
beunomars, there's a bunch of stuff there: https://wiki.canonical.com/UserExperienceDesign/WebGuidelines/17:15
beunoI don't think anyone has picked up that work17:15
=== matsubara is now known as matsubara-lunch
=== Ursinha-lunch is now known as Ursinha
=== benji is now known as benji-lunch
=== lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
=== beuno is now known as beuno-lunch
=== matsubara-lunch is now known as matsubara
=== salgado-lunch is now known as salgado
=== matsubara is now known as matsubara-dr
=== beuno-lunch is now known as beuno
=== benji-lunch is now known as benji
=== james_w` is now known as james_w
=== deryck is now known as deryck[lunch]
=== matsubara-dr is now known as matsubara
=== deryck[lunch] is now known as deryck
lifelessgary_poster: timeline has landed21:44
gary_postergreat, lifeless21:44
gary_posterthat will be nice to see21:44
lifelessgary_poster: do you know of anything other than (sql, email, memcache) in production that can block atm ?21:44
gary_posterdo the oops tools need a parallel update?21:44
lifelessgary_poster: it would be nice to teach them to do more with the info and look at a format change; for now I worked in the current constraints as expressed in the LP code21:45
lifelessso the database column is now category21:45
gary_posterfair enough21:45
lifeless'launchpad-master' will be SQL-launchpad-master21:45
lifelessso the existing group-and-report-longest should still work21:45
gary_posterblocks:I'm told that librarian blocks already, and I do seem to recall this.21:46
lifelessah yes, I knew that one but forgot. Thanks!21:46
lifelessI'm looking at doing a tiny patch to add categories and reporting for all of these,21:46
lifeless.21:46
lifelessand propose it for RC so we get better reporting during the next cycle21:47
gary_posterbefore I forget, lifeless: in regards to your "simplify OOPS emails" email from a while ago, that fits in line with things matsubara and Ursinha had discussed.  In a couple of weeks I'm planning to start focusing them + mars on that.  We'll be in touch soon with our thoughts so we can all be aligned.21:47
lifelessgary_poster: great, thanks for letting me know.21:48
gary_posterof course21:48
lifelesshttp://queue.acm.org/detail.cfm?id=185404121:59
lifelessslightly tweaked version of his blog post21:59
marsheya lifeless, have a sec for a testrepository question?22:19
lifelessshoot22:19
marsok22:19
marsI was looking at using testrepository with the Zope testrunner22:19
marsand thinking about how zope can take test IDs in the form 'bin/test id1 id2 id3'22:20
=== _thumper_ is now known as thumper
thumperlifeless, any quake damage where you are?22:20
lifelessthumper: rangiora; haven't gone around the town looking but not expecting to find much : old riverbed, not reclaimed land22:20
lifelesschc turned to jelly after the first shockwave passed through22:21
thumperI've seen some pictures22:21
lifelessthumper: my place seems fine so far, haven't gone into the weather to audit the outside yet.22:21
thumperpersonally I slept through it22:21
lifelessthe press has some pretty stunning ones22:21
thumpercool22:21
marslifeless, so testrepository has an IDFILE and test_id_option... ok, just thought of something22:22
lifelessmars: the zope test runner should support --load-list22:23
lifelessmars: jml patched it up22:23
lifelessmars: naively, I'd expect the .testr.conf for launchpad to work with it unmodified22:23
marslifeless, hasn't landed in upstream.  I have the latest22:23
lifelessah22:23
lifelessmaybe zope.testing is ahead of zope itself?22:24
marszope.testing + zope.testrunner22:25
mars(now)22:25
marsactually22:25
marszope.testing + zope.testrunner + zc.recipe.testrunner22:25
lifelessok22:26
lifelessanyhow, --load-list is terribly simple.22:26
lifelessbest thing to do would be to get that upstream22:26
lifelessit is 'load all the tests. Perform a set intersection.'22:26
marslifeless, this should work: deoesn't though:22:26
mars[DEFAULT]22:26
marstest_command=./bin/test --subunit $IDOPTION22:27
marstest_id_option= $IDLIST22:27
lifelesswhat dos it do ?22:27
marsprints the string 'IDLIST'22:27
mars$ testr run --failing22:27
lifelessthere are a couple of testr bugs in this area22:27
mars'IDLIST'22:27
lifelessup22:27
lifelessuhm22:27
lifelesstry test_command=./bin/test --subunit $IDLIST22:27
marsok, that sort of works, but it looks like it only accepts one module or test22:29
marsbin/test that is22:29
marsok, so upstream is still the best option22:29
=== matsubara is now known as matsubara-afk
=== salgado is now known as salgado-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!