), which I guess I can grab with var div = Y.one('.yui3-lazr-formoverlay-actions'); [06:31] StevenK: you can grab a button using attribute matching also eg "button.ok-btn[type='submit']" or something like that === jtv is now known as jtv-eat === jtv-eat is now known as jtv [08:02] good morning [08:05] Good Morning adeuring & mrevell [08:05] yo [08:16] stub: Hmm, so it turns out that all but two of the storm_cache and storm_cache_size directives in the configs are unused. [08:18] wgrant: Just the appserver overriding? [08:19] wgrant: We currently have a huge value for the appserver because it hasn't been instrumented and are scared of reducing the size. [08:20] wgrant: And are scared of using high values in scripts to stop memory ballooning. [08:20] stub: Well, that's what I thought, but it's a little bit simpler than that. [08:21] Its busted and only works in rare cases? [08:21] The 'launchpad' section is used for everything, except for the librarian (which uses 'librarian'). [08:21] \o/ [08:21] launchpad is 10000 on production, 100 on dev. [08:22] Someone somewhere along the line stopped calling setConfigSection in production code, except for the 'launchpad' default in lib/canonical/config/__init__.py, and one other case in librarian.tac. [08:22] librarian seems to really only care about overriding the dbuser and isolation_level. [08:22] Which, conveniently, is what everybody else cares about overriding too. [08:23] Do we currently have any script memory hogs? [08:23] Apart from nasty gina and process-death-row worstcases recently, not really. [08:24] That I know of. [08:24] dbuser and isolation_level should really be hardcoded anyway, isolation because things break if you change the config and dbuser because we never make use of the fact we can change the config. [08:24] Yeah, apart from in tests. [08:24] ... and formerly in buildd-manager, but that was fixed a year ago. [08:25] So, I'm probably removing config section support, merging launchpad's settings into the main database section, changing librarian.tac to override directly. And due to the fix I landed this morning, that means that dbconfig.override() is pretty much a direct replacement for initZopeless, and ZTM can finally BURN.; [08:26] And this stuff that has been a shim for 4 years can finally die :/ [08:34] Uhoh. [08:34] canonical.lp won't die. [08:34] It somehow got revived today. [08:34] * wgrant blames jtv. [08:34] whuwhuwhu? [08:34] message: [r=benji][no-qa] Fix lots of lint in recently-changed files. [08:34] added: lib/canonical/lp/ [08:34] :( [08:34] * wgrant remurders. [08:34] Please do. [08:42] Bah. [08:42] Looks like I need to keep respecting config.launchpad.dbuser, as it's what all the appserver configs use. [08:44] jtv, i think you mean "symptom, search internet, selfdiagnosis, treatment, conspiracy theories" [08:45] poolie: I am _not_ taking you to any more coffee shops. [08:45] haha :) [08:46] lifeless: how's the python-oops-twisted thingamujig? [09:09] bigjools: good news and bad news [09:09] bigjools: the good news is I figured out what all the tech debt meant [09:09] bigjools: the bad news is that the new project isn't done yet, because of the good news [09:09] bigjools: pretty close though, just tests to write now [09:12] lifeless: well, thanks, you've saved me a bunch of work so it's all good AFAIAC: ) === gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 256 - 0:[########*** stack smashing detected ***: kees pinged [09:26] jelmer, hey that's a really interesting point about the code ui [09:28] poolie: thanks; I think it's one of those things that are hard to notice once you've gotten used to the existing UI. [09:29] yeah [09:29] i was actually talking to huw about this the other day [09:29] because we did a little on it at the o'rally and it's been idle since then [09:29] i would love to see it reactivate [09:29] Once the last code-import<->mirror integration branch lands, I'd really like to kill the "Register a branch" button. [09:29] i think it's one of those topics that can hurt user experience a lot, but that won't clearly come up in bug reports or user requests [09:30] s//hurt or help [09:30] yeah === StevenK changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 256 - 0:[#########]:256 [09:50] bigjools: turns out our current twisted integration can silently block the librarian etc [09:50] bigjools: we're calling blocking apis without deferToThread [09:50] lifeless: awesome [09:50] I will be fixing one of the two cases [09:50] what scenarios? [09:50] oops creation writes files to disk [09:51] and can trigger a re-read of the directory [09:51] but the key bit is we thunk logObserver to python logging [09:51] and thats documented in the twisted docs as 'can block' [09:51] we should be doing deferToThread [09:52] and then doing the may-block stuff in a thread [09:52] this just became blindingly obvious when I started asking 'how to extract this nicely' [09:52] for bonus points we have two distinct twisted integrations atm [09:52] I normally don't mind the spider, but it should get off my damn screen! [09:52] roughly equally used. [09:53] I'm not going to try and fix that, but I'm going to bless one - the one I extract :) [09:53] lifeless: well at least you spotted it [09:54] lifeless: which is the blocking API being called? [09:54] We need to append pending log messages to a list, and deferToThread something that logs them in the correct order. [09:55] Logging a message doesn't normally need to be treated as a blocking operation. [09:55] jml: PythonLoggingObserver [09:56] jml: it does, and this is documented in the twisted docs as well as the logging docs. [09:56] lifeless: where? [09:56] jml: on the web page :) - sorry, past EOD and got stuff to do [09:56] that's lame [09:56] but I understand. [09:56] the twisted.log api docs specifically [09:57] wherever they are [09:57] Oddities I foresee are log messages being spit out in the wrong order, timestamps being slightly off, and bound variables and stacks being mutated as the callsite proceeds and before the log message is generated. [09:57] lifeless: thanks [09:58] stub: failures already materialise the backtrace IIRC [09:58] they do. [09:58] stub: the rest are already happening because we're in a reactor environment [09:58] cool. timestamps will be within precision. [09:58] just thinking of why it is blocking in the first place :-) [09:59] its blocking because the python 'logging' module has a great big lock in the middle [09:59] (and because file IO and directory listing can block too) [10:06] bigjools: do we have any way to watch the debian domination on production? [10:06] jtv: the log file? [10:06] I mean the effects of. [10:07] Seeing whether the resulting status of Debian makes sense. [10:07] jtv: yeah, find a package that's dead in Debian but not in LP's import [10:07] also see the count of packages in +missingpackages [10:07] Oh, we derive Ubuntu from Debian in production now? [10:08] have done for a while! [10:08] see oneiroc [10:08] oneiric, even [10:08] that's how this bug was noticed IIRC [10:09] Ooh nice little beta bar [10:10] Mind you, this is transitional domination so it shouldn't mark dead packages as deleted yet; it'll keep the last release alive for permanent domination to deal with. [10:11] Also, gina's still dealing with squeeze. :/ [10:11] did it do sid yet? [10:11] No, don't think so. [10:11] if not, take a note of the number of packages now [10:11] compare after [10:11] Good one. We also did that on dogfood. [10:12] Packages and statuses, I think. [10:12] * jtv cooks up some SQL [10:21] jtv: you can just look at the package counts in the UI [10:21] The ones that need linking? [10:22] [11:07:42] also see the count of packages in +missingpackages [10:23] That includes the DSD machinery as well… since I'm not expecting any package deletions, I'll want to watch the DB as well. [11:07] gmb: clearly, you're not bitter but just a tad disappointed in my review style? [11:12] jtv: No, I've just formed an emotional attachment to the branch because I've been working on it for a week and a half. [11:12] ] [11:12] It's been a royal pain in my backside. [11:12] jtv: One of your in depth reviews is always appreciated, unless the reviewee is feeling particularly lazy. [11:12] Says the self-proclaimed Lazy Reviewer. :) [11:13] We ought to form a duo à la Laurel & Hardy. [11:13] :) [11:13] Hippie & Nazi, Reviewers™ [11:13] Yeah, I can see that going down a storm :) [11:16] lifeless: I need to do a txlongpoll test fixture - any opinion on whether that should be a separate project? [11:17] gmb: compared to what I got away with last week, it ain't so bad. [11:18] bigjools: I'm torn [11:19] bigjools: on the one hand it should clearly be reusable, so not part of LP [11:19] lifeless: we could not decide either :) It'll add more dependencies of course [11:20] on the other hand if its part of txlongpoll it ties the client language to the server implementation, and implies that a client that can use the fixture could get at the server innards - thats undesirable [11:20] on the gripping hand adding a new project has some cognitive overhead [11:22] I think we should try separate-project: we can always massively consolidate later [11:23] I was erring that way [11:25] gmb: your change to initialize() puzzles me. Why assign a variable you're not going to use? Are you trying to force evaluation of a property? [11:27] jtv: Yes; activity_and_comments is a cachedproperty. If it's not loaded at initalize() time you end up with recursion problems in tests (not in actual usage though). I can drop the variable assignment; it just looked odd to me without it. [11:30] jtv: I'm going to go and grab some lunch; feel free to stick any questions in the MP and I'll reply on my return. [11:31] Aye-aye. [11:38] flacoste, lifeless: we can't triage bugs for launchpad-results (like https://bugs.launchpad.net/launchpad-results/+bug/854635), meaning that it's not really our duty, yet it's part of launchpad-project and we can't drive untriaged bugs count to zero: what shall we do about it? [11:38] <_mup_> Bug #854635: unit model names have leading/trailing spaces < https://launchpad.net/bugs/854635 > [11:41] bigjools, the following bug seems to be simple to fix (missing DB user), yet pretty critical: bug 854449. Do you think we should treat it as an incident and start working on it asap? [11:41] <_mup_> Bug #854449: generate-contents-files.py failing on Ubuntu archive since September 11/12 < https://launchpad.net/bugs/854449 > [11:41] danilos: yes [11:42] danilos: I suspect fallout from recent db user renamings [11:42] danilos: also it needs a script monitor [11:43] bigjools, right, thanks, I'd appreciate it if you (as someone with more authority on the subject) adds a note to the bug itself, and I'll ensure we get started on it soon [11:43] soon == today [11:44] ok :) [11:45] danilos: It is a pretty trivial puppet task. [11:45] An RT and a LOSA-poke may be in order. [11:45] wgrant, I can guess it is for someone who knows all about the script [11:46] Hm? It's not got much to do with the script :) [11:46] wgrant, (eg. what DB user it should use) [11:46] if __name__ == '__main__': [11:46] script = GenerateContentsFiles( [11:46] "generate-contents", dbuser='generate_contents_files') [11:46] Problem solved :) [11:47] wgrant, I'd appreciate if you add that to the bug as well, since I am doing my CHR rotation right now [11:49] Done. [11:50] thanks === almaisan-away is now known as al-maisan [12:27] bigjools: You might recall that I removed txlongpoll a while back. [12:27] bigjools: lifeless forbids its inclusion in our setup.py. [12:27] FFS [12:27] For it is to be a microservice. [12:27] we need it for make run [12:27] that is all [12:28] I don;t know how else to set it up [12:28] Is there an RT for Bug #854449 yet? [12:28] <_mup_> Bug #854449: generate-contents-files.py failing on Ubuntu archive since September 11/12 < https://launchpad.net/bugs/854449 > [12:28] bigjools: Neither do I. [12:28] danilos: -results is in a bit of limbo; we should either have it meet our governance rules or remove it from being part of 'launchpad-project'. [12:29] danilos: chat to flacoste when he's up, and/or cr3 about which route [12:29] danilos: cr3 wants to have it properly integrated as a microservice etc [12:29] danilos: so, I think the right thing is getting it under our regular rules for bug supervisor etc [12:29] lifeless, right, thanks for updating me on it [12:30] wgrant: bigjools: you use a buildout recipe [12:30] wgrant: that gets it executable with a bin/txlongpoll entry point, but not importable [12:30] This is going to get very ugly. [12:30] But maybe. [12:30] wgrant: bigjools: my use-gpgfixtures branch on lp has/had an example of doing that for the new keyserver. [12:30] which flacoste gave me. [12:30] We'll need a separate buildout target for each dep. [12:31] lifeless: there is no bin/txlongpoll entry point [12:31] it's a TAP [12:32] so, you need the egg unpacked (buildout can do that), and you need a script (the fixture) that knows how to run the tap [12:32] no ? [12:32] just txlongpoll itself doesn't end up on our PYTHONPATH [12:33] can you explain your aversion to setup.py changes? [12:33] Ah, that's right. You'll need to use buildout to create a second twistd. [12:33] That has the egg. [12:33] I love setup.py changes [12:33] in this context :) [12:33] https://dev.launchpad.net/ArchitectureGuide/ServicesRequirements#Background_reasoning [12:34] I'm not sure that answers the question [12:34] thats the background reasoning [12:35] I'm not really au fait with buildout so this is a learning exercise [12:35] for why I want the microservices to not be importable [12:35] happy to be guided on this [12:36] wgrant: yes, buildout a z3c.recipe.script that has the additional egg, probably a defaults=... and tada [12:36] jtv: I hadn't noticed the conflicts with devel; have pushed a fix. [12:37] gmb: thanks. I'm… collating notes. [12:37] ok [12:37] bigjools: basically I want an ironclad avoidance of the sort of back-door dealing with API's we've had in LP with in-process stuff [12:37] ok [12:37] bigjools: so that folk can *confidently* work to the interface, not the implementation [12:37] this has several corollaries: [12:38] I'm still unsure what setup.py has to do with that [12:38] - there will be some form of split-outs that we can't do, because we don't know how to write an interface that is fast+ etc etc etc [12:38] I perfectly understand the rationale, just not the "how" :) [12:38] oh [12:38] so when you add something to setup.py, buildout adds the resulting egg to the PYTHONPATH [12:38] it does this transitively. [12:38] and a simple buildout recipe does not? [12:38] if you have setup.py reference package A [12:39] and A references B [12:39] B will be in the PYTHONPATH setup by bin/py [12:39] bigjools: the 'scripts' recipe can specify its own additional things [12:39] how are we going to make a fixture figure out how it can run the TAP then? [12:40] gmb: done. Frankly it was the name of the branch that attracted my attention in the first place. [12:40] bigjools: thinking [12:40] bigjools: got it [12:40] bigjools: the fixture needs to know the command line to run the thing [12:40] jtv: Heh, thanks. [12:41] yes, but it also needs to know where the twistd binary and plugin lives to do that [12:41] bigjools: this doesn't mean importing, it just needs to know that 'bin/twistd-longpoll' will run a twistd invocation that can import longpoll, and it needs the tap file [12:41] bigjools: yes, this will mean buildout.cfg stuff (sorry to be handwavy, nearly 1am etc) [12:42] ok, thoroughly confused here :) [12:42] wgrant: are you less confused? [12:42] maybe someone who knows buildout better can do this [12:42] jtv: Once again, your review has me saying "Wait, what?" I never knew about zip() (or knew and have forgotten). I shall go and learn about it forthwith. [12:42] gmb: good sport [12:43] def sudo_make_me_a_fixture(twistd='bin/twistd-longpoll', tap='bin/longpoll.tap'): [12:43] return LongPollFixture(twistd=twistd, tap=tap) [12:43] ^ thats the thunk code to inject the custom twistd script and tap location [12:44] the other bit is teaching buildout to create the tap and twistd-longpoll [12:44] bigjools: See txlongpoll's buildout.cfg. [12:44] The 'interpreter' section there does sort of what we want. [12:44] Except we probably want to use a script target directly, without an interpreter, if possible. [12:45] lifeless: how would it know where bin lives in your example? [12:45] bigjools: in LP it will always be bin/twistd-longpoll [12:45] even if we have a script that encapsulates the invocation [12:46] ah, so you want it in LP's bin [12:46] thats what buildout is for [12:46] its responsible for making that file [12:47] this is what flacoste showed me: http://bazaar.launchpad.net/~lifeless/launchpad/usegpgfixtures/revision/13758 [12:47] twistd is not entirely flexible in how it looks for TAPs is it ... [12:47] wgrant: ^ [12:47] bigjools: Not at all. So we'll need a second twistd that has basically just txlongpoll in its path. [12:47] right, we have something in the txlongpoll buildout.cfg that builds itself [12:48] Ahh, z3c.recipe.scripts, handy. [12:48] Although that still generates an interpreter. [12:48] wgrant: it didn't [12:48] I don't know why the line is needed ;) [12:50] what does that code put in bin/ ? [12:50] s/code/config/ [12:51] it creates bin/gpgfixtures [12:51] which has the buildout glue to set pythonpath etc [12:51] not in my dev area it doesn't :) [12:51] and ends up running gpgfixtures.keyserver.main(argv) [12:51] bigjools: what do you mean ? [12:52] there is no bin/gpgfixtures when I build latest trunk [12:52] of course [12:52] thats in a branch [12:52] unlanded? [12:52] ok [12:52] its demonstrating what you need to achieve, for a different but similar case [12:53] insomuch as I understand roughly what one of those lines is doing :) [12:53] right [12:53] this is why I pointed wgrant at it [12:53] he has more overlap with you [12:53] I'm dead :) [12:53] FSVO overlap :)_ [12:53] Well, we actually both have zero overlap. [12:53] But yes. [12:54] yeah, you were up at crazy o'clock [12:54] thanks for the advice so far lifeless, rest well [12:54] Night lifeless. [12:56] wgrant: ok so we might be able to get away with the one in txlongpoll, the only issue is what to do with the TAP directory abomination [12:56] bigjools: What about it? [12:56] That's all in the egg. [12:57] wgrant: yes, but it needs to end up in $CWD [12:57] Nope. [12:57] otherwise twistd doesn't find it [12:57] It's in twisted/plugins/blah [12:57] is there a way of telling twistd where it is then? [12:57] Just try that buildout rul.e [12:57] Should work fine.l [12:57] I can type, see. [12:57] you miss my point I think [12:57] twistd looks in twisted.plugins [12:58] The plugin is in twisted.plugins in the egg. [12:58] where "twisted" is in $CWD [12:58] No. [12:58] ah [12:58] In pythonpath. [12:58] ah [12:58] * bigjools experiments [12:58] The egg is all set up. It works fine. [12:58] If you add it to setup.py and rerun buildout, you'll see it listed when you run bin/twistd [12:58] So we just need to make a *second* twistd, that only has txlongpoll. [12:58] we do? [12:59] Since our normal twistd uses the deps from setup.py [12:59] ah right [12:59] It's for running LP stuff. [13:00] how do you rename the generated script name? [13:00] ah never mind [13:02] Morning, everyone. [13:02] hi deryck [13:02] morning deryck [13:02] wgrant: know what's up here? "Error: Picked: Twisted = 11.0.0" [13:03] bigjools: Possibly because you're using include_site_packages. [13:03] ok let's try false [13:03] nope [13:03] I have no idea what that error even means [13:04] I'd not know it was an error if it were not preceded with "Error" ... [13:05] ah think I found it [13:05] and no [13:14] abentley, adeuring -- can we standup at 20 after? I have another call at the bottom of the hour today. [13:15] deryck: sure [13:15] deryck: sure [13:16] great, thanks guys. === al-maisan is now known as almaisan-away [13:35] bigjools: i'm hopping on call with deryck [13:36] bigjools: but I could help you with that buildout after it, you could also grab gary_poster or benji to help out in the mean time [13:37] bigjools: you need to set versions.cfg to specify Twisted = 11.0.0 [13:39] jelmer: Did you intend to bump the bzr-git revision when you regenerated sourcedeps.cache? [13:45] wgrant: argh, sorry. That should have been in the original commit. [13:46] Ah, sure. [13:47] flacoste, hi, do you have any suggestions on what we should do with launchpad-results (it's part of launchpad-project, but we don't have privileges to triage bugs)? lifeless believes we should make it match lp-project standards for bug supervision [13:54] Hey deryck or gary_poster, do you think you could spare someone to finish implementing Huw's 503 page? (https://code.launchpad.net/~huwshimi/launchpad/503-error-844631) Huw tells me he has taken it as far as he can. [13:55] * gary_poster looking, mrevell [13:57] mrevell, sure, I can put it in our (pretty short) backlog. bonus points if you mark the bug escalated or critical :-P but not necessary [14:01] Thanks gary_poster! We consider Launchpad to be a stakeholder, so perhaps I could escalate it. It's producing an OOPs, which we want to continue logging, but I suppose that makes it critical. I'll mark it critical and ask lifeless' forgiveness if need be :) [14:01] mrevell :-) ok cool [14:03] mrevell, gary_poster: It's fastdowntime-related and an OOPS, so it can be critical. Doesn't need escalation :) [14:03] superb [14:03] wgrant, cool :-) === almaisan-away is now known as al-maisan [14:15] gary_poster: hello! [14:15] my buildout saviour [14:15] hey bigjools [14:15] heh [14:15] gary_poster: what I am doing is adding a script recipe to generate a "microservice" [14:16] versions.cfg already has Twisted = 11.0.0 [14:16] bigjools, script recipe from what [14:16] this is in Launchpad [14:16] 's setup.py I assume [14:16] not in setup, I am taking the txlongpoll egg and writing some buildout config to try and make it generate me a script [14:17] lifeless has banned changing setup.py for external scripts [14:17] which is fair enough [14:17] bigjools, ok. Oh, interesting. (I don't think I'm breaking that in my current branch..._ [14:17] ) [14:18] bigjools, my memory is a bit fuzzy on all this, but I bet if I look at it I'll have an idea. Do you have a branch for me? [14:18] I'll paste you my change, one sec [14:19] gary_poster: http://pastebin.ubuntu.com/693799/ [14:19] gary_poster: argh [14:19] sorry hang on [14:19] gary_poster: http://pastebin.ubuntu.com/693800/ [14:19] the first one is the whole file, that's just the diff [14:19] bigjools, An error with "Picked" is definitely complaining about "You didn't tell me which version to use, so I had to guess, and you told me to complain, so I'm complaining". So the question is, why isn't versions.cfg good enough [14:19] ok [14:20] gary_poster: oh I wonder if it doesn't like the txlongpoll egg then [14:21] since that's the part that requires twisted [14:22] um [14:22] maybe. I wouldn't have guessed so. [14:22] lemme try applying this. 1 sec [14:22] you'll need to update your eggs [14:22] well, dists [14:23] k [14:26] bigjools, I see "Error: Picked: txlongpoll = 0.2". Fixing versions.cfg for that... [14:26] gary_poster: weird, I don't see that [14:28] bigjools, ok now duped Twisted problem. Looking more... [14:28] gary_poster: ah right sorry, I had already applied the change to versions.cfg and didn't tell you :) [14:28] :-) cool [14:28] gmb: Could you mentor a review of mine? (the branch it self is from allenap) [14:33] bigjools: txlongpoll says it depends on twisted not Twisted. The name is Twisted, and everything else spells it as such. Apparently this confuses setuptools and/or buildout code. You can verify by editing eggs/txlongpoll-0.2-py2.6.egg/EGG-INFO/requires.txt to have Twisted, not twisted. Then build will complete successfully. I suggest changing setup.py in txlongpoll and rolling a new release. [14:33] argh [14:33] I thought it might be something like that [14:33] thanks gary_poster [14:33] welcome bigjools [14:34] I can't do enough releases [14:34] :-) [14:36] danilos: i agree with lifeless [14:36] flacoste, cool, so what do we do about it? ask cr3 to fix it all for us? :) [14:37] danilos: oh oh, what's this I hear about work? [14:38] danilos: or ask a losa if cr3 isn't responsive enough :-) [14:39] flacoste: good assumption! it would indeed make sense to have danilos on /ignore [14:39] rvba: Sorry, was otp. Certainly, I'd be happy to. [14:39] gmb: Thanks, here you go: https://code.launchpad.net/~allenap/launchpad/longpoll-storm-events/+merge/76215 [14:40] rvba: Okay, I'll take a look shortly. [14:40] flacoste: seriously though, is this about builders, hardwaredb, results tracker or something else? I'm not seeing an obvious thread by scrolling above [14:41] cr3: it's about the bug supervisor for launchpad-results [14:41] gmb: if you get a minute to look at a simple lazr.restful branch, I'd appreciate it: https://code.launchpad.net/~benji/lazr.restful/bug-854695/+merge/76230 [14:41] aha! I see now, bugs in launchpad-results. where can I familiarize myself with lp-project standards for bug supervision, I'm all for following that too [14:41] benji: I'll add it to my list. [14:42] thanks [14:42] flacoste: someone did ask me to triage them more quickly before, which I've been doing so far. [14:42] cr3: we'd like to change the bug supervisor so that the lp team can triage the bugs [14:42] flacoste: I suspect that "critical" might appear as a big red light on your radar though [14:43] also [14:43] flacoste: if you change the bug supervisor to launchpad itself and I'm not a member of launchpad engineers, won't that case problems for me? [14:43] s/case/cause/ [14:44]