[02:08] \o/ [02:08] 10000 OOPS-fb8c41fc98e9a541a1175a570450f288 Unknown [02:08] wgrant, wallyworld_: https://code.launchpad.net/~stevenk/launchpad/auditor-on-packageupload-reject/+merge/119291 [02:08] StevenK: how goes auditor ? [02:09] lifeless: Still wondering what the next step is. [02:09] go on [02:11] lifeless: So we have auditor, its fixture, its client and now two callsites in LP [02:12] is there a staging auditord yet ? [02:12] Nope [02:15] lifeless: So I guess we need to plan to deploy auditor for staging/qas and then slap it into an RT. [02:15] yes. [02:15] this might be a good time to do the refactoring I blathered on about [02:16] Oh? :-) [02:16] voice? [02:17] Skynet? Mumble? [02:17] the robots are coming! [02:19] (skype) [02:20] Right, let me get that set up [02:29] lifeless: Apparently, you're red [05:12] lifeless: Firebug is making Firefox do the crash-restart-crash loop [05:15] StevenK: use in-built dev tools [05:15] Oh, is that what Mozilla is trying to tell me? [05:15] And now it's lost my session details [05:15] I guess it got too embarrased [05:16] lol [05:16] Maybe I'll just switch to Chrome [05:18] * lifeless thought firebug *was* the built in dev tool [05:19] Haha [05:19] There's been built-in dev tools for ages in Firefox now. And it's been getting really good. [05:19] Whoosh. Missed the irony. [05:34] StevenK: bug 1025011 [05:34] <_mup_> Bug #1025011: Firebug extension causes firefox to crash (can be triggered by opening HUD)

< https://launchpad.net/bugs/1025011 > [05:35] Handy [05:35] so basically, Chris fixed it, but the next firebug update discovered another latent bug in the code [05:36] wgrant: I don't understand the remaining 5 failures. :-( [05:36] StevenK: so, either use 1.14.0 with Firefox 14 in the release pocket, 1.14.1 with firefox in -proposed, 1.14.2 and you're on your own [05:36] StevenK: 'sup? [05:37] StevenK: This is the testbrowser upgrade? [05:37] micahg: Or switch to Chromium in anger. [05:37] Which I did. [05:37] wgrant: Yah. [05:37] StevenK: and be open to lots of CVEs, have fun :) [05:37] StevenK: What are the failures? [05:37] wgrant: http://pastebin.ubuntu.com/1144330/ [05:38] I've fixed the third failure in xx-person-subscriptions.txt, but I'm not sure about the first two. [05:38] StevenK: goBack [05:39] wgrant: Which is only in two of them [05:40] :( [05:40] goForward? :) [05:40] None of them :-P [05:41] logging-in.txt is pretty obvious [05:41] micahg: which code is at fault ? [05:42] lifeless: globalmenu-extension [05:42] xx-person-subscriptions.txt is odd [05:42] Hm, Chromium doesn't want / for search :-( [05:42] micahg: ah well, we knew it was hairy. [05:42] that's the other option, disable the globalmenu :) [05:42] StevenK: Oh [05:42] StevenK: Try to think through the xx-person-subscriptions.txt failure [05:43] The first one, that is [05:43] wgrant: I'm still trying to work out why you say logging-in.txt is obvious [05:43] wow [05:44] postgresql schema patch downtime -> the dev wiki page on live patching is 5th in google. [05:44] StevenK: It's not using normal browser methods [05:44] StevenK: It uses some OpenID thing [05:45] lifeless: Indeed, we seem to have #5 and #6, from a couple of hopefully unbiased browsers. [05:46] StevenK: So, the first xx-person-subscriptions.txt failure is because it's now sending a sensible Referer [05:46] StevenK: The view uses Referer to set cancel_url [05:46] So it now ends up on the right page [05:46] Not the one it expects. [05:46] Which I thought I removed in anger [05:47] http://postgresql.1045698.n5.nabble.com/ChronicDB-Live-database-schema-updates-with-zero-downtime-td1901036.html [05:48] also lol - 'ChronicDB is currently a private technology' [05:48] moving right along [05:48] have we done a second schema patch without slony now ? [05:48] lifeless: No [05:48] lifeless: Hopefully Colin will have one for us tomorrow. [05:50] wgrant: logging-in.txt is making use of testbrowser to do stuff, it just uses the extra methods to parse what came back [05:52] anyone remember the statistic from flacoste about velocity improvement from FDT ? [05:52] StevenK: It passes the URL into consumer.complete, which looks a bit sus to me. [06:03] wgrant: It does? Where? [06:04] StevenK: In complete_from_browser [06:05] Ah [06:08] wgrant: So I guess xx-person-subscriptions.txt is the test expecting behaviour which is now fixed [06:09] StevenK: The first failure, yes. [06:09] The others I haven't looked at. [06:19] wgrant: How does goBack break the other two? [06:20] StevenK: I don't know. [06:21] I don't even know if it does. [06:21] But it seems very likely. [06:21] * StevenK sigh [06:21] I broke buildbot [06:26] That's not very nice of you. [06:26] Oh [06:26] I assumed that was local breakage [06:26] My build just failed the same way [06:26] I'm just about to land a testfix [06:26] As soon as bzr push completes [06:36] StevenK: wgrant: either of you guys adding the new Embargoed information type to the enum? i'll do it if no-one else is [06:37] I'm not. [06:38] Not sure it's worth adding yet, but maybe :) [06:38] I'm swearing at testbrowser and buildbot [06:38] i figured we'd need it done first so things could use it [06:38] Well [06:38] We should be sure it's going to work first [06:39] We are still discussing it, so I'm not sure it's worth leaping yet [06:39] I'm currently finishing off sharing policies for Proprietary. [06:39] StevenK: Right. [06:39] ok, i'll hold off. i thought we had agreed to do it [06:39] wallyworld_: Do you also do "Ready, fire, aim" ? [06:39] huh? [06:40] As opposed to 'Ready, aim, fire' [06:40] i must be slow today, don't get the reference? [06:50] its slightly whacky humour [06:51] the instructions called out by WWI officers - ready (guns loaded etc), aim - aim at the folk running across no mans land, fire - pull the trigger. [06:51] StevenK is suggesting that you are doing (firing) before aiming (finishing the discussion). [06:51] Personally, I think its a little harsh. [06:59] especially since i thought we had converged on a solution, but anyway [07:01] wallyworld_: We hopefully have, but eg. mrevell was still unclear about it on Friday [07:01] And it probably needs to be confirmed with PES [07:01] The solution evolved on the Friday call, and Curtis was away on his Friday, so it hasn't had much chance to settle. [07:01] should I put my nose in a little? [07:01] No. [07:02] if you want [07:02] wgrant: there was a card on the board in the next column, hence my thinking we had agreed to it also [07:03] if it's not final, we shouldn't put the card up yet :-) [07:03] wallyworld_: Bah [07:03] True [07:03] I put that there :) [07:03] It's agreed, but not final. [07:03] np, glad we could clarify [07:03] And mrevell appears concerned [07:04] oh? is there an email i missed? [07:05] * wallyworld_ has to go to soccer, back later [07:06] Well, just the steps to sharing beta one. It seems nobody outside the team is clear on what's happening. [07:06] we'll see over the next couple of days, I expect [07:16] * StevenK stabs Unity [07:16] Firefox crashes, and brings up a crash dialog, so you click Restart and then Unity loses track of the main Firefox window until you fiddle with the dock [07:23] good morning [07:40] lifeless: You missed a failure -- we tickled a bug in Slony when we dropped a column, so the sequence was killed and Slony got very unhappy with the world. [07:41] lifeless: Unless that's what you meant with two tables at once? We didn't drop the tables, we dropped their id columns. === almaisan-away is now known as al-maisan [07:46] I believe that's the failure to which he referred in his reply [07:46] Although misstating it, as you say. [08:17] StevenK: bah, you are right. [08:18] lifeless: Now you can edit your blog post and thank me. Or something. [08:24] Oh, there's a blog post? [08:24] Ah, there [08:25] where.. [08:25] http://rbtcollins.wordpress.com/2012/08/13/minimising-downtime-for-schema-changes-with-postgresql/ [08:25] ahh [08:54] StevenK: or I can let it stand wrongly vague :P [08:54] lifeless: "Or something." [08:54] :) [09:01] lifeless: StevenK wgrant anyone any thoughts on https://lists.launchpad.net/launchpad-dev/msg09565.html [09:05] Yes. None are printable. [09:05] wgrant: Thanks. [09:25] czajkowski: blueprints are for summit basically [09:26] czajkowski: so 'just for summit' is sufficient as an answer [09:26] czajkowski: though there may be a deeper question that isn't clear yet [09:28] StevenK: wgrant: so we dropped the id column from one or two tables ? [09:28] lifeless: I suspect cjohnston is working on summit stuff and wants to make a change, but hasn't explained what besides wording [09:32] czajkowski: I share your suspicion. [09:33] lifeless: which doesnt help get him an answer either :/ [09:33] czajkowski: and thus noone has answered ;) [09:33] Presumably I should be using ETags and gzip when communicating with Launchpad from another webservice, right? Anyone know if a thread-safe alternative to httplib2 for that? [09:34] httplib2 is threadsafe with the patch jml did, we think. [09:34] oh awesome [09:34] lifeless: true, I'm just following up on stuff from my week off [09:34] s/thread/concurrency/ [09:35] ISTR someone pushing for a release and getting it into 12.04 [09:36] I was asking for one [09:36] But wasn't that lazr.restfulclient not httplib2? [09:39] lifeless: We dropped an ID column from two tables. [09:40] wgrant: replacing it with some natural PK ? [09:40] s/it/them/ [09:40] lifeless: Right [09:40] Dropping a serial column automatically drops the underlying sequence [09:40] Slony automatically polls sequence values [09:40] => slony broke [09:41] The only ID columns we've dropped in recent history are BranchRevision.id and OAuthNonce.id [09:41] And the latter wasn't replicated [09:41] and I don't remember when we did BR.id [09:41] So it's possible that this isn't a regression [09:43] fark HUD. [09:43] DIAF. [09:43] What's it doing now? [09:43] killing FF [09:44] Ah [09:44] Disable the globalmenu extension? :) [09:45] * cjwatson looks at database/schema/comments.sql and wonders how he's supposed to "keep [it] alphabetical by table" [09:45] lalalalal [09:45] Should the first part of my branch be to sort it? :-P [09:45] Well [09:45] comments.sql is sort of dead to us now [09:45] https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess [09:46] Just saying [09:46] Whether it should continue to exist at all is not exactly clear [09:46] "If you have removed or renamed a table or column, update database/schema/comments.sql." [09:46] so, comments.sql is no longer applied in production. [09:46] Right, comments.sql needs updating in that case or make schema will fail [09:46] It doesn't say anything about adding new ones :) [09:46] We need to move: [09:47] - comments.sql [09:47] - fti [09:47] - sampledata [09:47] wgrant: "Comments on new tables and columns need to be added to database/schema/comments.sql. " [09:47] into the lazr.postgresql patch system [09:47] Curses. [09:47] and drop the LP one entirely. [09:47] cjwatson: don't stress on the alphabetical, please do add comments until we transition. [09:47] In the short term should I just put COMMENT statements in the patch itself? [09:48] no, comments.sql. [09:48] it won't apply in prod, but we don't care. [09:48] There was some discussion a while back about moving them to patches [09:48] As we did with trusted.sql [09:48] Have you dropped that idea? [09:49] wgrant: not at all, just we haven't actually done it AFAICR, and unlike trusted.sql its not functional. [09:50] Yeah [09:50] StevenK: thanks for pointing that out. Updated. [09:56] what is this a symptom of does anyone know? can't get launchpad.dev to respond this morning... [09:56] 2012-08-13T09:55:34 INFO root Startup time: 9.164 sec real, 9.110 sec CPU [09:56] channel 3: open failed: connect failed: Connection refused [09:56] channel 4: open failed: connect failed: Connection refused [09:57] could be rabbitmq not running ? [09:57] That's an error message from ssh [09:57] If that helps [09:58] ah, so I borked my ssh forwarding perhaps [09:58] will also poke the rabbit. [10:11] okay, so it seems nothing is actually listening on 8080, zope.server.http came up on 8085, not sure if that's normal, haven't got a log from when it was working earlier to compare against [10:11] ...or have I [10:12] hm, did that last time as well. [10:12] but 8080 is what used to work... [10:14] * mgz looks at apache setup [10:14] mgz: The normal launchpad.dev Apache config listens on 80 [10:15] (and 443) [10:15] 8080 is usually unused [10:18] well, apache is not running... >_< [10:18] what changed... [10:19] manually starting it and all is well [10:20] so close to not having to interveen, but need manual fixup too often, half the time it's me being dumb but this is too much of a pain [10:21] so, I think apache doesn't get added to the things that should be run on startup perhaps? install, use launchpad is fine, didn't come back after reboot... [10:23] lifeless: https://code.launchpad.net/~cjwatson/launchpad/queue-filter-source/+merge/119225 - I'd be happy to try running this on dogfood. Is there a sensible way to extract a compiled version of the query there (some kind of profile mode or something) or do I have to work it out by hand? [10:26] cjwatson: bzr merge lp:~cjwatson/launchpad/queue-filter-source [10:26] make initscript-stop initscript-start [10:26] Browse :) [10:27] Ah, but you won't get in-page query logs unless you're in ~launchpad... [10:27] And look at dogfood-nohup.out or something? [10:27] You could change the feature flag to allow you too :) [10:27] Oh right. I could add myself to ~launchpad on DF :P [10:27] The visible_render_time flag shows a partial query log at the end of the page [10:27] Or that, yes [10:27] How about API queries? [10:28] Good luck with that :) [10:28] Can't even use ++oops++ with the API [10:28] Perhaps use LP_DEBUG_SQL=1 bin/harness? [10:28] Actually I guess DistroSeries:+queue lets me do basically the same query, so not necessary in this case [10:28] Yeah [10:28] That's what I assumed you'd test [10:39] morning [12:04] blue - jam jelmer mgz vila https://bugs.launchpad.net/launchpad/+bug/1035338 [12:04] <_mup_> Bug #1035338: An unexpected error has occurred while updating a Launchpad branch < https://launchpad.net/bugs/1035338 > [12:05] just jam/mgz czajkowski :) [12:05] ah ok [12:05] thanks [12:16] czajkowski: that oops won't load for me, just spinns during lookup, but have responded on the bug === al-maisan is now known as almaisan-away [12:24] morning [12:43] Hmm. Upgrading dogfood says: [12:43] lazr.config.interfaces.ConfigErrors: ConfigErrors: database does not have a auth_master key. [12:43] database does not have a auth_slave key. [12:46] Something to do with r15782 I guess [12:47] ah, the dogfood config needs to be updated [12:48] * cjwatson will sort it out === benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 4.0*10^2 [12:56] mgz: I did eventually get the oops to load, though I did it by going to oops.canonical.com directly and then doing a search from there (still took a while) [12:56] interestingly, the content there is an XMLRPC Fault that contains yet-another OOPS [12:56] xmlrpclib.Fault: [12:56] look at that one: https://oops.canonical.com/?oopsid=OOPS-394216d625e87559cda9d8b92d70b90f [12:56] It ends up as RequestExpired. [12:57] I wonder if they hit downtime while he was trying to commit? [13:00] jam: No, planned DB downtime does not result in timeouts. [13:00] wgrant: even for bzr codehosting? [13:01] The underlying OOPS (the RequestExpired) is from the appservers, not codehosting. [13:01] this is a Request timeout in the Storm layer [13:01] But yes, even for codehosting. [13:01] now, it does seem to say that it was taking 7s to get a Feature flag looked up... [13:01] which seems very odd [13:02] Note also the 4s lookup for a person by ID [13:02] yeah [13:02] It's probably GIL contention or similar. [13:02] The appservers are single-threaded, except not. [13:02] Isn't that SQL time? [13:03] Or the timing code has enough python in it to cause some time to be in python-land [13:03] It's "SQL time" [13:03] But measured by Python [13:03] So if the interpreter doesn't run for 4s, it'll appear to have taken 4s longer. [13:03] It's possible that it's connection latency or firewall breakage. [13:03] But it's very similar to the behaviour we saw more frequently before we went to single-threaded* appservers [13:04] (* each instance actually has two threads, one to serve webapp requests and the other to serve xmlrpc-private (this is xmlrpc-private), so there can still be contention) [13:04] The only theory we have that does not involve persistent network flakiness is threading contention [13:04] So threading contention is the going assumption. [13:05] It's supported by the fact that we mostly see these on xmlrpc-private requests [13:06] And xmlrpc-private requests are rarely CPU-intensive on the appserver. [13:06] (so webapp requests have a smaller chance of being starved) [13:06] Now [13:06] 7s is pretty huge [13:06] Bordering on implausibility, really [13:07] But the 4s query afterwards means it wasn't just a momentary disruption [13:07] We often just see the slow first query [13:23] benji: Could I have a review of https://code.launchpad.net/~cjwatson/launchpad/report-pcj-oops-2/+merge/119209 ? I'd like to unstick the chain of work that's part of [13:24] cjwatson: sure, but I'm on an interview call at the moment; it may be an hour [13:25] OK, thanks - I probably need to head to a more congenial network environment anyway [14:35] cjwatson: review done [14:53] deryck: left the file as a .odg file and uploaded with images/set up. [14:53] deryck: let me know if you don't see it in the shared U1 folder [14:59] rick_h_, ok, will check here shortly [15:11] benji: thanks [15:12] congrats frankban [15:17] thanks rick_h_ [15:17] how do I set a feature flag to default to on? [15:18] doc says should always use "enabled", but the current state is that enabled should be True [15:18] mgz: feature flags are not truly bools [15:19] We set the scope to "default" and add the string "on" to indicate we want everyone to get the feature [15:19] but the codebase must have some idea of which ones are on at present [15:19] mgz we remove "on" to disable it. Any string will evaluate to True in that position [15:19] or all tests, and the testing config, set flags as needed? [15:19] sure, we have a page for that [15:20] https://launchpad.net/+feature-rules [15:20] mgz: tests must set flags to use them. They are off in the test suite [15:21] ...so landing this will inevitably change the current state till someone sets the new flag on production? [15:22] and everyone's local test setups will also change? [15:22] (not that anyone will mind, I'd think, in this case) [15:24] mgz: no one setup changes with the addition of a flad, nor does production. That is why we know bugs.dynamic_bug_listings is not finished. I have to enable it to see in in dev, and it must be on in production to see it. People only see the new feature because we enable it for them, or we remove the flags from the code [15:25] okay, so it's a bit of a hack using a feature flag for this then, but I'll trust that it's the most sensible option [15:25] mgz, So reading back to what you first asked...the answer is feature flags cannot be default on because by their nature, they are incomplete. [15:26] thanks for explaining the details for me sinzui :) [15:39] hm, seems it's not possible to do boolean logic in a tal expression [15:40] | relates to path lookup, not evaulation of the objects found [15:49] mgz: we use tales adapters to create bools. We also use "|Nothing" to safely adapt to false when an exception is raised [15:51] mgz: what path do you want to know is true or false? [16:30] ...darn disconnects >_< [16:31] sinzui: basically I want to change not:view/user into not:view/user||not:features/app.root_blog.enabled [16:31] use python:? [16:31] No [16:31] I think the shortest path to that is adding an attribute to the python class that contains that combination [16:31] We do not want logic in the templates. [16:32] python: is bad because it makes everything after it python [16:32] fair enough, I'll shut up :) [16:32] can't just use it to do the boolean logic correctly on path lookups [16:32] mgz, move the logic into the view so that we can write a unit test [16:33] right, that's pretty much where I'd arrived at. the test I've written is just looking at the html output [16:33] mgz: tales makes testing hard, trying to make it support logic ultimately leads to more hours spent trying to test it [16:33] is there somewhere else I can do this that doesn't involve the databasefunctionallayer? [16:34] mgz: I use view = create_initialized_view() then test the bool value of view.show_root_blob [16:35] I'll grep for some examples, thanks [16:36] mgz, given the logic you tried to write, I think you want to add tests to LaunchpadRootIndexViewTestCase [16:36] that's where I've got 'em [16:36] still painful for tdd, with multi-second setup time for the layers [16:37] mgz, You can try to write the test on the FunctionalLayer, but that wont be must faster [16:39] ./bin/test -vvc -t LaunchpadRootIndexViewTestCase lp.app.browser.tests.test_launchpadroot [16:39] will be the fastest command [16:40] oo, and colourful. [16:41] I've been using -m... apparently that's the same as just giving an arg? [16:41] mgz, gary_poster told me no, passing the module path as the last arg is different and faster === skaet_ is now known as skaet [16:42] ehehe === almaisan-away is now known as al-maisan [16:51] okay, this all looks workable [17:07] sinzui: './bin/test -vvc lp.app.browser.tests.test_launchpadroot LaunchpadRootIndexViewTestCase' might be faster still === deryck is now known as deryck[lunch] [17:07] oh [17:07] I look forward to faster tests [17:08] when layer setup takes nearly 10 seconds, it's hard to see a difference... [17:08] mgz :( [17:09] is there any distinction between `with anonymous_logged_in() as user: view = create_initialized_view(..., principle=user)` and just `create_initialized_view(...)`? [17:09] mgz yes :( [17:09] barry wrote the helper to ignore the user we alreay placed in the interaction with login... [17:10] if I want to test what someone sees if they arrive at launchpad.net with no cookies, which do I use? [17:11] principal is rarely needed though. Generally principal is only needed when we render the view and the page renders the "logged in as" portlet [17:11] hm. hard to tell what's needed and what's cargo culting when using other tests as templates. [17:12] just using the minimum that works is tempting, but sometimes that ends up asserting useless things. [17:13] mgz, you need to construct the request manually I think. The helper accepts an optional request object (which you may have already setup for the login user. Otherwise, the help will create it's own user and request [17:16] mgz, actually, the default request has no cookies, so you do not need to create it [17:17] mgz, if the method you wrote is looking at the self.request, pass the logged in user at the principal [17:21] hm, nope, test method I'm working from just goes getUtility, makePerson, login_person, create_intialized_view, call view to get result [17:55] sinzui: free to chat? [17:55] yes [17:55] excellent. [17:56] invite sent. [17:58] phone is being odd [17:58] It know I got the invite, but says I am not logged in. [17:58] something needs a whack upside the head [17:59] jcsackett: hold on a moment [18:00] Google doubts that I exist [18:00] sinzui: sure. i've cancelled the hangout so i can resend whenever. === deryck[lunch] is now known as deryck [18:16] cjwatson: you can export LP_SQL_DEBUG (IIRC) to get queries logged to stdout, and then interactively fiddle around till you see the one yo want [18:17] cjwatson: or you use ++oops++ if its a regular view to trigger an oops and get timing data; if you're in the right group (~launchpad) you'll get timing data on the initial queries in web views by default [18:38] deryck: ping, how we doing? I'm coming up on EOD in 30 and wanted to check in [18:39] benji: have a sec to look over https://code.launchpad.net/~rharding/launchpad/yuiv5/+merge/119406 sometime? [18:39] lifeless: this is per our conversation, if you know of a better way to do it via makefile magic appreciate feedback/tweaks. ^^ [18:44] rick_h_, on call, just a minute sorry [18:44] deryck: np, knew it would be tough to get time today. [18:45] deryck: o/ Did I miss our call ? [18:45]