[00:22] wallyworld_: How can you deserialise an enum? [00:22] wgrant: using the lazr.json lib [00:22] But you can't know what type it is. [00:22] So it's impossible. [00:23] the type is encoded in the serialisation [00:23] Ah [00:23] That sounds absolutely disastrous. [00:23] ly exploitable [00:23] it's how most serialisation scheme i've seen work [00:24] And most software is fucking terrible. [00:24] What's your point? :) [00:24] my point is i'm using common patterns and not reinventing any wheel [00:39] wallyworld_: Ew [00:39] Are you trying to turn JSON into pickle? :) [00:39] wgrant: i'm not going to argue with you. the approach has been approved by curtis [00:39] Letting user-specified JSON import arbitrary things, double-encoding stuff, embedding code paths in JSON... [00:40] wallyworld_: I haven't seen any suggestions from wgrant either [00:41] I don't know if there is a good solution. [00:41] JSON is too untyped to do this cleanly. [00:41] We could potentially do something with the enum name. But embedding the class path is extremely onerous and completely inappropriate. [00:42] we need to be able to handle arbitrary complex objects - it's not just about enums [00:42] JSON isn't for deserializing directly into complex objects. [00:43] it provides hooks to do so [00:43] and we use a pattern where we shove arbitrary stuff down rabbit [00:43] so we need to be able to handle that [00:43] That pattern was simply convenient. [00:44] We have not studied its validity or correctness. [00:44] My inclination would be to fix that. [00:44] Rather than turn JSON into pickle. [00:44] With the various security and compatibility concerns that that entails. [00:51] * lifeless notices the conversation [00:51] someone want to give me a quick overview? [00:51] '{"BaseItem,lazr.enum._enum.Item": "{\\"type\\": \\"Foo\\", \\"name\\": \\"BAR\\"}"}' [00:52] lifeless: ^^ [00:52] that looks a little like JSON [00:53] but whats the story around this? [00:55] I would love a review of https://code.launchpad.net/~cjwatson/launchpad/init-packageset-fixes/+merge/103036 [00:56] lifeless: lazr.json is designed to serialize and deserialize arbitrary types from JSON. [00:56] (and QA suggestions as per my MP comments) [00:56] lifeless: The initial case is lazr.enum. [00:56] lifeless: It does this in the format shown above. [00:56] wgrant: why are we doing this? wallyworld_ ^ ? [00:57] sorry, what's the question? [00:57] wallyworld_: I'd like to understand why we're doing arbitrary type encapsulation in JSON; its rather antithetical to JSON and thus has me confused. [00:58] lifeless: give me a sec otp [01:01] cjwatson: You could just say Packageset.distroseries_id.is_in(...) [01:01] Oh, that works? OK, will try [01:02] Ah, yes, I see that pattern elsewhere [01:03] nukes a circular import too [01:03] Yay [01:08] wgrant: Updated [01:11] cjwatson: Approved, thanks. [01:12] Wonderful. 'ec2 land --incremental' is right for this, isn't it? [01:12] I haven't actually checked the state of the DB here [01:12] Was going to tell you to use incremental, yep [01:12] Since we should also add DB constraints [01:12] Later [01:12] Right [01:13] OK, off to EC2 then [01:13] Great. [01:13] With any luck we can squeeze in a deployment pre-Q-opening ... [01:13] It's nodowntime now, so we'll hopefully have two or three before release. [01:22] lifeless: with the json thing - we are currently serialising arbitrary objects and if there's a particular object class that is not supported, we oops. enum is one but there will be others. the approach taken is to allow object encoders/decoders to be plugged in and used to handle the cases json doesn't know about [01:28] wgrant: on init-packageset-fixes, do you know if running initialize-distroseries on dogfood is going to be a saneish thing to do? [01:29] sub-day runtime, that sort of thing :P [01:29] cjwatson: It will take forever. We could initialise/create a small distro and do it there [01:29] Or we could run on qastaging, actually [01:29] Will still take a while, but probably 100x faster or so [01:29] I'll need help for that, but that sounds like a better plan [01:30] IIRC it's ten minutes or so on production [01:30] Didn't we use the web UI last time? [01:30] good point, I believe we did [01:30] So we shouldn't need webops to run stuff manually, hopefully. [01:31] that's what the NewReleaseCycleProcess notes say, anyway [01:31] Unless the runner isn't cronned, which is plausible. [01:31] * wgrant checks. [01:32] Might still need somebody to deal with creating a test series on qastaging for the purpose [01:32] Though I suppose we could just use q-series [01:33] I'd just rename q-series to Quarrellous Quokka and use that, yeah. [01:33] Does it need a rename? [01:33] wallyworld_: Are you still OTP? [01:34] I mean, in order to test this [01:34] StevenK: no [01:34] wallyworld_: Can haz mumble, then? [01:34] cjwatson: Not really, true. [01:34] sure [01:36] wallyworld_: http://pastebin.ubuntu.com/941907/ [01:36] wallyworld_: why are we serializing arbitrary objects? [01:37] wallyworld_: and serialising arbitrary object doesn't imply *deserialising* them. [01:37] wallyworld_: why is deserialising needed? what are we doing with this json ? [01:37] lifeless: you tell me and we'll both know. poor design? i think we are listening to object modified events and shoving the result down rabbit [01:38] lifeless: if we don't handle deserialisation, tests will fail. and i do think we deserialise at the other end of the mq? [01:38] wallyworld_: We don't have Launchpad at the other end of the mq [01:38] wallyworld_: which tests will fail if we don't deserialise? [01:38] (there's an ObjectModifiedEvent hook on the Storm baseclass which shoves diffs into RabbitMQ) [01:38] (this should probably be stopped0 [01:39] wgrant: shoves to rabbit for what? txlongpoll ? [01:39] lifeless: i would have to check and get back to you, can't recall exactly [01:39] lifeless: Yes. [01:39] lifeless: Currently only one thing is listened to: MP diff changes. [01:39] wallyworld_: so, I'm going to raise a red flag here, I think this is heading into a world of pain and information disclosure. [01:39] Specifically the job status, I think [01:39] YAY [01:40] wallyworld_: can you please schedule a call with curtis, you and I (more optional, but we three must be there) to review this. [01:40] lifeless: another potential use case - sending arbitrary data to jobs via rabbit [01:40] lifeless: so, you don't agree we should support serialising arbitrary objects? json or not [01:41] I don't agree. [01:41] lifeless: so the only alternative is to make each bespoke use case do it's own flattening [01:42] or rather, I don't think adding an additional typed-encoding in json makes any sense; mapping from rich objects to basic types is something we already do in the API via the interfaces. [01:42] lifeless: how about we have a call tomorrow after our standup [01:42] say around 9am AEST [01:42] or you could join our standup [01:42] starts at 8am AEST [01:43] wallyworld_: there is an existing protocol for flattening objects; if we do want to do this, that would be the right way forward; AFAICT though, we don't /want/ this, we've stumbled on it via adding something to an existing code path [01:43] wallyworld_: I will be here at both times, ping me :) [01:43] lifeless: ok, will do. talk tomorrow [01:47] (it also pretty much destroys our compliance to any of JSON's goals, and is likely to be insecure) [01:50] wallyworld_: http://pastebin.ubuntu.com/941914/ [01:51] wgrant: well we need to decide if there's a use case for arbitrary object deserialisation. most enterprise systems support it and provide mechanisms to do so. it's a very common pattern. [01:52] not sure why there's so much hostility given it's not exactly a new idea [01:53] I am not completely averse to the entire idea. [01:53] But this is without a doubt not the right way to do it. [01:54] wallyworld_: well, AIUI you're solving an OOPS we're seeing [01:55] wallyworld_: I think the OOPS has uncovered us sending confidential object change metadata to arbitrary unauthenticated clients on the internet [01:55] lifeless: yes, and in a way that handles arbitrary objects, which will happen, not just the oops we are seeing [01:55] wallyworld_: not in practice, but in potentia, we have a dangerous API [01:55] writing a system to let that dangerous API be more powerful isn't a great use of time IMNSHO [01:56] trimming that system back to be less dangerous, which will also avoid the OOPS, will be. [01:56] the question of 'can we serialise everything' is tangential. [01:56] serialise-everything opens inappropriate-disclosure holes; deserialise-everything opens bad state injection holes [01:58] lifeless: sure, there's a balance to be achieved. we can discuss tomorrow on the call. too hard via irc [01:59] yup [01:59] totally agree, thats why I suggested a call :) [02:01] lifeless: btw, the arbitrary objects thing - the stuff you mention just above is solved in the decoder/encoder classes, so it's not as bad as you probably are thinking [02:04] wallyworld_: I don't see how its solved [02:07] lifeless: an arbitrary object is only serialised if there's been a decoder registered for it's type (and it will oops if there's none). i guess it is possible to register a fake decoder and inject bad data, but that's an issue not unique to this case and we would need to deal with that i guess [02:08] consider a privileged field [02:08] how do you deserialise it *and* trust the result. [02:08] secondly, object graphs [02:08] lifeless: the implementation handles object graphs just fine [02:08] that's the point of it [02:08] I'm sure it does [02:09] but thats where the vulnerabilities in these systems come in [02:09] lifeless: I've been on Mumble with wallyworld_ for 30 minutes now, and you keep interrupting every 3 minutes! [02:09] I would have been done by now! [02:09] StevenK: hey, I'm just replying. Tell wallyworld_ to stop :P [02:09] I have been! [02:10] i will now. till tomorrow :-) [02:10] Now I'm tell you BOTH to stop. [02:10] in soviet russia, words speak you === statik_ is now known as statik [03:46] StevenK: how goes the yui test writing? [04:18] wallyworld_: Trying to muddle along [04:20] sounds like fun [04:21] wallyworld_: Firebug complains YUI isn't defined. [04:21] And make lint is complaining about undefined entity in the HTML. [04:21] hmmm. what are you trying to load? [04:21] sounds like a syntax error in the html [04:22] you may have stuff from the template left over [04:22] Let me just get my current thought out of my head and into Vim and I'll repaste the diff [04:22] ok [04:24] wallyworld_: http://pastebin.ubuntu.com/942011/ [04:24] one sec [04:29] StevenK: there's nothing obviously wrong with the html and test js that i can see. are you trying to load the html into your browser? [04:29] Yah [04:30] one thing is that you may not have included all the necessary dependencies in the html [04:30] all the js files required need to be specified manually [04:30] so you are missing choiceedit.js and friends i thibk [04:31] It's just strange it's manifesting as YUI being undefined. [04:31] Dpesm [04:31] Er [04:32] Doesn't one normally use LPJS from firebug? [04:32] This is a YUI test. [04:33] StevenK: if yui can't load all the dependencies it won't initialise [04:33] And therefore isn't defined. Right. [04:33] so if you say to load information_type.js and that required xxx.js and xxx.js isn't available, boom [04:34] so it's a pita but you need to add in the things you think you need by hand [04:34] to the html test harness [04:45] That's choiceedit, privacy and bugtask_index. Now to work out what else I need. [04:49] wallyworld_: I guess base == yui.js itself, what about oop, node, event and io-base? [05:24] wgrant: https://bugs.launchpad.net/bugs/+bugs?field.searchtext=foo+bar [05:24] wgrant: and https://twitter.com/#!/pipesec/status/194202104582258688 [05:24] wgrant: look at only one [05:28] lifeless: What about it? [05:28] Works fine for me. [05:28] Returns a plausible number of results [05:28] etc [05:29] The FTI index will take a while to page in on all 3 nodes. [05:29] wgrant: times out for me, 3 out of three times [05:29] Maybe you got a bad slave. [05:29] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-f28e0087ffa2474538d8d21683092d1d [05:29] Just got one in 8.6s rather than the normal 2.1s [05:29] So one slave may just be almost there. [05:29] could be [05:30] Reliably <2s now [05:30] (note that this is still GiST... will improve with GIN in a couple of weeks :)) [05:33] StevenK: you need node etc in your test js, not the html as yui as a whole is loaded but you still need to specify the dependencies in the js as like with all yui modules [05:34] lifeless: Currently the tiebreak sort key is always bugtask ASC or bug ASC. I've currently got the indices set up to handle the default direction for each of the common sort orders, but they doesn't work for the reverse because the tiebreaker is the wrong direction. [05:34] wallyworld_: http://pastebin.ubuntu.com/942063/ [05:34] lifeless: Do you think the tiebreak order is significant enough that I should double the number of indices, or can I reverse the tiebreak order in that case? [05:34] (currently reversing the order doesn't reverse the tiebreaker) [05:35] StevenK: so you've included overlay.js in your html test harness? [05:36] wallyworld_: http://pastebin.ubuntu.com/942064/ is the current test [05:36] StevenK: /build/js/yui/yui/overlay/overlay.js is bogus [05:36] it's more like lp/app/javascript/overlay/overlay.js [05:37] /build/lp/app/...even [05:37] That's there too [05:37] A few lines down [05:37] ah so it is. you don't need any yui stuff in there though [05:38] looks like you're missing anim [05:38] probs also extras [05:38] you sort of need to look at the requires bits of the things you are importing and import those dependencies also [05:39] wallyworld_: Right. I was unsure if I needed to care about the YUI bits included too, like oop and event. [05:39] not in the html, just the requires of the test js [05:40] if required [05:42] wallyworld_: But overlay only has: "requires": ["oop", "overlay", "event", "widget", "widget-stack", "widget-position"] ? [05:42] wgrant: so you have foo bar quux id all asc, and reversed foo desc bar desc quux desc id desc ? [05:43] StevenK: from memory, choiceedit requires those other things i mentioned [05:43] yes, lp.anim [05:43] lifeless: Currently we default to (importance ASC, bugtask DESC). When you reverse that listing, you get (importance DESC, bugtask DESC) rather than (importance DESC, bugtask ASC) [05:44] StevenK: and lp.anim requires lp.extras [05:44] wallyworld_: But choiceedit isn't complaining, and it's imported above overlay? [05:44] wgrant: ok, thats a bug [05:44] lifeless: Arguably. [05:44] StevenK: such are the mysteries of yui. i'm not sure why that would be the behaviour [05:44] So I can change it? [05:44] yes [05:45] Thanks. [05:45] reversing the sort should reverse all the rows, which it won't be atm [05:45] wallyworld_: http://pastebin.ubuntu.com/942069/ is a new diff [05:46] StevenK: looks reasonable, but only way to know is to try [05:48] wallyworld_: It still gives the YUI not defined errors [05:48] hmmm. still missing dependencies i would guess. what's the branch? [05:49] wallyworld_: Let me commit this test and push [05:49] ok [05:50] wallyworld_: lp:~stevenk/launchpad/bugs-information_type-ui-portlet [05:50] ok, let me save my current branch [05:55] StevenK: you appear to have one to many ../ in your js paths [05:55] s/to/too [05:56] wallyworld_: All of them? [05:56] all the ones referencing build/*** [05:57] my IDE told me that immediately :-P [05:57] Bah [05:57] I copied from the lp/app/testing/testrunner.js line [05:58] StevenK: our javascript dirs are entirely 100% consistent [05:58] StevenK: also, you will need [05:58] [05:58] [05:58] effects I added [05:58] definitely lazr.js then also [05:59] and i think that solves the initial bootstrapping of the test [05:59] so now you just have to make everything pass :-) [06:00] wallyworld_: But I need to drop a '../' from every src line involving build/js ? [06:00] yes [06:00] StevenK: but not the lp/app ones [06:00] the template is correct for those includes [06:00] Or the YUI one? [06:01] correct [06:01] Looks like only bugtask_index.js then [06:02] Bah, I still get YUI is not defined [06:02] ah, sorry i meant the top 2 lp/apps lines [06:02] id didn't notice the ones near the bottom were lp/app [06:03] should 5 5 changes [06:03] be [06:03] 1 bugs one and 4 lp app ones [06:03] You mean the one under Dependencies? [06:03] yes [06:04] the top 2 lp/app lines under dependencies are ok [06:04] the next 4 aren't [06:04] the next 2 are ok [06:04] and then bug task isn't [06:05] wallyworld_: So extras, anim, effects, lazr.js, overlay and expander? [06:05] StevenK: https://pastebin.canonical.com/64716/ [06:05] i should have pasted the diff in the first place [06:06] Haha, yes. [06:06] sorry [06:06] got myself confused [06:07] the only correct lines were overaly and expander [06:08] Oooh, it loads the test now. [06:08] Y.lp.testing.mockio is undefined. [06:09] should be 5 ../ [06:09] you may also find you need to include the subscription_list from lp/app [06:12] wallyworld_: It's actually calling the JS test now. [06:12] yay :-) [06:13] Testing completed at Mon Apr 23 2012 16:13:21 GMT+1000 (EST). [06:13] Passed:1 Failed:0 Total:1 (0 ignored) [06:13] progress! [06:13] Mind you, the test doesn't assert anything yet. :-P [06:13] But yes, progress! [06:16] lifeless: Barring massive issues I was aiming to turn it on for ~launchpad-beta-testers tomorrow morning. [06:16] But maybe earlier is good. [06:16] lifeless, wgrant: But I don't think perfect behaviour justifies doubling the number of indexes. Do real searches actually use the tiebreaker, or is it only to keep things perfectly stable for the test suite? [06:16] stub: Real searches use them. [06:16] stub: eg. Launchpad's default bug listings [06:17] stub: We only use three statuses, so the tiebreaker is used for the majority of the sorting. [06:17] ok [06:17] s/statuses/importances/ [06:19] wgrant: tomorrow is a bad day to make changes [06:19] wgrant: as wed is a pub hol [06:21] Mmmm, I guess. [06:21] Will run some more tests and do it this evening. [06:24] wallyworld_: I can't see how to grab hold of the choicesource after I simulate the click on the edit link. [06:24] StevenK: bit more info? [06:25] wallyworld_: I've got an edit link in the test HTML, which has href='#'. That is passed into setup_information_type_choice() so it should bring up the choicesource overlay when it's clicked. [06:26] wallyworld_: I want to click one of the options on the choicesource [06:26] right, i see. look at test_shareetable.js where i test the choicesource popups for changing info type for a sharee === mthaddon` is now known as mthaddon [06:27] StevenK: test_permission_update_click [06:28] lines 277-282 [06:28] wallyworld_: Right. I'm doing the same thing [06:28] figured as much :-) [06:29] var choicesource = Y.one('.yui3-ichoicelist-content'); [06:29] And choicesource is null [06:29] the click to open the choice source popup must be wrong [06:32] Hmm, I think I'm missing a div [06:33] have you looked behind the couch? [06:33] Hah [06:34] one of my favourite t-shirts is "I found Jesus. He was hiding behind the sofa" [06:35] Right, that gets the choicesource working [06:36] hi [06:36] is possible to browse launchpad project by program language? [06:37] you could use the API, but we don't have a built in thing for that AFAIK [06:39] lifeless, and through web page? [06:39] for example, to list all Python project [06:39] StevenK: the test passes? [06:40] wallyworld_: I have it working, I just need to figure out how to get the content of the so I can assert against it. [06:40] node.get('text') [06:41] StevenK: btw, while you're there, care to fix the capitalisation in BugSecrecyEditView.label? [06:41] Which capitalisation? [06:41] if bool(getFeatureFlag( [06:41] 'disclosure.show_information_type_in_ui.enabled')): [06:41] label += 'Information type' [06:41] else: [06:41] label += 'visibility and security' [06:41] The first has a capital, the second does not. [06:41] Neither should. [06:41] Right [06:43] lifeless: Meh, shall I play dangerously and switch it on now? [06:43] I can't see any issues, apart from the rare index usage due to tiebreaker, but it's still strictly better than the old code. [06:44] wgrant: for all beta +1 [06:44] Thanks. [06:44] wgrant: lets hit it fully on thurs. [06:45] Sounds good. [06:45] We may see particular issues with searching by structsub or tag, but it should be slightly better than the old one. [06:52] wallyworld_: Y.lp.bugs.bugtask_index.portlets is undef :-( [06:52] And adding "lp.bugs.bugtask_index.portlets" to requires didn't help [06:59] StevenK: did you include subscription_list? [06:59] wallyworld_: Yup [07:00] hmmm. must be something else. i'll see if i can see what it is [07:00] StevenK: lp.app.errors, lp.app.confirmationoverlay [07:01] For portlets? [07:01] lp.bugs.javascript.subscription [07:01] i think portlets renders subscriptions? [07:01] haven't checked [07:01] but if so, then subscriptions uses those things [07:01] So I should include lp/bugs/subscription.js too? [07:02] yes [07:03] subscribers_list is the other one I was thinking off [07:03] of [07:05] wallyworld_: Firebug showing NOT loaded is stuff I'm including and don't have to? [07:05] can't remember. example? [07:06] There's one thing that always appears there [07:06] widget-position-ex or something [07:06] yui: NOT loaded: lp [07:06] yui.js (line 4009) [07:06] that's spurious, ignore it [07:07] the lp one shouldn't appear [07:07] that means there is a missing dependency [07:10] StevenK: i'm off to soccer, back later. i'll check in to see how you're going [07:10] wallyworld_: I should stop working anyway [07:11] you left off " and start drinking" [07:11] Hah [07:11] i might have a few after soccer [07:11] I've been dealing with JS all day, that's *implied*. [07:11] yeah, i'm stuck in js as well atm [07:12] wallyworld_: Besides s/drinking/drinking heavily/ [07:12] almost got it sorted [07:12] Didn't you hear, we're rewriting LP in node.js. [07:12] luckily i know you're joking [07:30] good morning [09:24] stub, lifeless: https://code.launchpad.net/~wgrant/launchpad/bugtaskflat-more-indices/+merge/103063 [09:24] Heh [09:24] The backup completed 5s before I proposed the live index changes. [09:24] Handy. [09:26] stub: you've got that ? [09:26] yer [09:28] Will ordering by date_last_updated without a tie-breaker break tests? I guess fixing the tests would be better. [09:29] We could fake date_created by using id instead [09:31] I wonder if this would be better accomplished by dropping the IS NOT NULL on the existing partial indexes? Not for this live update, but that might work. [09:40] No obvious errors. No point me questioning the utility of the indexes - I understand they are driven by the searches we allow [09:46] stub: date_last_updated and date_created have always been regarded as unambiguous. We've never used a tiebreaker with them, despite the fact that they could technically be ambiguous. [09:47] We have, but only to keep tests happy and that was probably a mistake. [09:47] The existing bug search code did not use a tiebreaker. [09:47] better data is a better option [09:47] So the new one does not. [09:48] Yer, that is all fine. [09:48] stub: Dropping the IS NOT NULL doesn't work reliably. [09:48] These indices are just for sorting. [09:49] All the existing ones work marvellously :) [09:50] k. Would be nice if they did. My guess would be they would often be used, but occasionally odd stats would cause them to be skipped. But empirical results override my assumptions :) [09:51] stub: Dropping the NOT NULL doesn't help contextless sorts, as the index will still be primarily ordered by context. [09:51] oh, right [09:53] Unfortunately COUNT(*) over Ubuntu's open bugs still takes slightly over a second. [09:54] But better than the 4s it used to take. [10:01] jtv: ping [10:02] wgrant: At least it isn't EOVERFLOW yet [10:02] wgrant: Those indexes are live on staging & qastaging [10:02] hi czajkowski [10:02] stub: Let me do a few checks. [10:03] Marvellous, https://bugs.qastaging.launchpad.net/bugs/+bugs in 1.5s [10:04] Anything stopping us taking this live? [10:04] Just checking Ubuntu. [10:04] But it looks good. [10:04] jtv: sorry I'm ssure it's late over there but a bit unsure about https://bugs.launchpad.net/launchpad/+bug/987199 dpm reported on translations [10:04] Yeah, all more than fine. Enlivenate. [10:04] <_mup_> Bug #987199: Automatic translation exports fail to commit latest translations < https://launchpad.net/bugs/987199 > [10:08] czajkowski: I don't think it's that changes weren't detected. More likely is that the exports are breaking. You'll want to watch the launchpad-error-reports mailing list for error output from the translations-to-branch export script. Often it's a lingering branch lock. Another possibility is that imports and exports cross, in which case the export script backs off, for fear of overwriting something. [10:08] There's only one perma-locked branch. [10:08] The export script hasn't complained about ubuntu-doc ever AFAICR [10:08] nautilus-pastebin is the only regular victim. [10:09] hmm need to check folders for launcpad-error-reports ml [10:10] czajkowski: You're not subscribed by default. [10:10] nope I'm not [10:10] just checked all my folders [10:17] czajkowski, wgrant: I suppose it's not unthinkable that the translations branch was broken _for a while in the past_, and changes accumulated during that time. [10:18] I'm embarrassingly ignorant of how this stuff works. [10:19] If you know anything at all, that's far from embarrassing. As I recall, the exporter—at least at the time I wrot eit—lacked a way to see whether a particular change had already been exported. [10:19] Instead, it just exported changes from some fixed time window — even if that meant occasionally exporting something twice. [10:19] The branch is clearly getting translation updates committed to it. [10:20] Another possibility is might be that the translations are activated in such a way that their timestamps aren't updated. [10:21] czajkowski, wgrant: errr… there _was_ a discrepancy, right? Because I see 337 untranslated strings on both the Ubuntu side and the upstream side now. [10:22] I trusted the bug, didn't check myself. [10:22] Oh, need to check the branch contents apparently, not the translation stats. [10:22] Should've known; David's very thorough. [10:22] That's why I didn't bother checking :) [10:26] Good call, as it turns out. Yes, the branch has a bunch of untranslated strings that are translated in Launchpad. [10:26] So it sounds as if sharing is to blame. [10:27] Not certain, of course. [10:28] nods [10:28] thanks jtv [10:29] So the suspicion is that a string can become current, without updating timestamps in such a way that the translations-to-branch exporter notices that anything new needs exporting. [10:29] jtv, yes, as mentioned on the bug, it seems the stats in LP are correct, it's just the branch that does not seem to get them committed. [10:29] Hi dpm [10:29] jtv, hey, thanks for looking at this :) [10:30] I *think* it's the timestamp on the POFile that the translations-to-branch exporter looks at, and which most likely does not get updated. [10:30] jtv, could it be that translations have been done offline, uploaded, and LP hasn't noticed there have been some changes? [10:30] Shouldn't make a difference in itself. [10:30] ah, good to know [10:31] Any idea whether those changes were made in the Precise series itself, or made in e.g. Oneiric and shared with Precise? [10:31] in Precise, afaik, which is what the French team were translating [10:31] Because I don't _think_ (mostly guesswork for me at this point) that we update the POFile timestamp when we share translations. It's actually pretty costly to figure out which POFiles might be affected. [10:31] Hmm… if the work was done in Precise itself, you'd expect that timestamp to update. [10:32] they might have been doing the work on the upstream project too [10:34] Ah [10:34] That'd do it. [10:35] what do you mean exactly? [10:35] Well, sharing an individual string's translation (a TranslationMessage) between upstream and the Ubuntu package is a matter of setting an extra “this is the current translation for Ubuntu” flag on the TM. [10:36] what's a TM and who or what sets that flag? [10:36] ah, TranslationMessage [10:37] but the sharing seems to happen, just the commit doesn't seem to [10:37] TranslationMessage. LP sets that flag to make it the current translation in a given context; each TM has one such flag for current-upstream and current-in-Ubuntu. Setting that flag may affect Ubuntu-side POFiles for the same language in whose POTemplate the TM's POTMsgSet participates, _and_ which don't have another, newer translation to that language _and_ which don't have a diverged translation for that language. With me so far? [10:37] Never mind, I'm pulling your leg. :) [10:37] It's a bit complicated, and that's exactly my point. [10:38] When LP sets that flag, it's doing something relatively simple in itself. But once you start looking at all the POFiles that it might possibly affect, it gets a bit complicated. [10:38] And so I suspect we don't really seek out those affected POFiles to update their timestamps. [10:39] And if we don't update their timestamps, chances are the translations-to-branch exporter doesn't notice that anything's changed in its version of the POFile. [10:39] oh, I thought the point of having translation sharing between upstreams and Ubuntu was to be able to commit Ubuntu translations to the upstream projects. Do I understand it correctly that that actually does not happen by design? Or is it rather a bug? [10:40] It goes both ways, with exceptions. [10:42] (The exceptions are: Ubuntu translations only go automatically into upstream if the reviewer happens to have review privileges upstream as well; diverged messages, which override a message's translation in just one upstream release series or in just one Ubuntu release, stay diverged; and I think changes on either side don't override those on the other side unless the translation on the other side is either missing or identical to the old translatio [10:42] All that seems to work just fine, but it all clouds our view of which POFiles just received a new translation. It's a lot of work to get the exact set. [10:43] And so, I don't think we actually do all that work. [10:44] A fresh translation upstream will probably automatically go into all Ubuntu releases of the package that have the same English string, but probably without updating POFile.date_changed for all those places it ends up. [10:45] Ok, so the summary is that I should stop recommending maintainers to use sharing + automatic exports and tell them to request tarball exports instead. Does that sound reasonable? [10:45] It'd be a sad step back. :( [10:46] absolutely, and I'll be the first to regret it :/, but now I understand that sharing + exports does not work as expected, which I didn't know before. [10:47] Another equally sad way to cope would be to make some arbitrary change to the Ubuntu-side POFile. [10:47] I didn't know it either. I'm not even entirely sure. But I did just verify that POFile.date_changed is what drives the exports. [10:57] jtv, thanks a lot for the research and the help. I think it might make sense to add the relevant parts of the conversation to the bug. I'd volunteer for it, but you're the one who knows how everything fits together, would you mind adding your findings so far? [10:57] OK [11:01] dpm: I've updated the bug. I hope it's clear enough; if not, please say. [11:07] thanks a lot jtv, it looks clear enough to me for the parts I can understand :) [11:08] OK. Remember, if you really need one of these exports to happen, just make an arbitrary change and then re-active the original translation. [11:08] Not very nice, but it should do the trick for that POFile. === jamestunnicliff_ is now known as jamestunnicliffe [11:12] jtv, gotcha, that's good to know, as a workaround, thanks. [11:13] jtv: marked triage and low, should it be higher? [11:13] czajkowski: I would call David's problem a high priority; the one about the wrong credits a low priority. [11:14] nods [11:14] jtv: fixered cheers [11:14] ok === Pendulum_ is now known as Pendulum [11:52] rick_h: are you alive over there yet? if not ping when you come on thanks. [11:52] czajkowski: yep, I'm alive [11:53] rick_h: did you do stuff with OAuth mails last week ?https://answers.launchpad.net/launchpad/+question/194281 [11:53] czajkowski: yes I did [11:53] czajkowski: I'll reply. [11:54] lovely jubbly [11:54] cheers and happy monday to you :) [11:55] you too! [12:01] czajkowski: answered [12:01] and doh, newline fail on my twitter reply :/ [12:03] rick_h: :) I did say please after all === benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2 [12:03] czajkowski: yea, supposed to be a nice bash response: command not found [12:03] sounds like monday [12:03] rick_h: good weekend ? [12:04] busy, but yea can't complain [12:05] now monday...she's really starting to get irritating [13:00] Morning, all. [13:01] hi [13:01] gary_poster: fwiw, I haven't had a chance to look at bug 986421 and bug 986434, and probably won't get a chance soon. [13:01] <_mup_> Bug #986421: In parallel tests, global tags eventually become misapplied < https://launchpad.net/bugs/986421 > [13:01] <_mup_> Bug #986434: global tags unnecessarily inflate concurrent test results < https://launchpad.net/bugs/986434 > [13:03] jml, hey. np. the inflation one is no biggie. The global misapplication is actually not a showstopper for us either right now but we may want to investigate it and propose something if we make progress elsewhere. In terms of official release of the packages, I think the global tag one should be addressed before a release. The file inflation one might be invalid as far as you are concerned. [13:04] well, I think a strong case can be made for it being a genuine bug. [13:05] cool [13:26] wgrant: is bug 983420 really fix released? [13:26] <_mup_> Bug #983420: Celery branch scan jobs intermittantly fail to show revisions for new branches < https://launchpad.net/bugs/983420 > [13:26] wgrant: Oh, nm. [13:28] abentley: Yeah, sorry, misclicked and reverted. [13:28] wgrant: np. [14:20] sinzui: did you just spring clean license review? [14:21] czajkowski, I send a license-conflict email to one that was in review [14:21] sinzui: ah that was the one I'd left to check with you [14:21] just wondered refreshed page and was gone [14:21] good day :) [14:26] deryck: ping [14:26] rick_h, yo [14:26] deryck: so I think the issue is that the distro source pkg isn't 'published' so using the factory.makeSourcePackgePublishingHistory [14:27] deryck: do you know if I should make sure it's published for each series then? [14:27] rick_h, honestly, I don't know. sorry. [14:27] I guess does that click/make sense [14:27] ok, will test it out thanks [14:27] rick_h, yeah, I'd guess you need it in each series first. but don't know for sure. [14:30] rick_h: I'm on my way to bed, but an SPPH is per series and has a status. I think the default is PENDING. [14:31] The enum is in soyuz or archivepublisher. Probably the former. [14:31] StevenK: ok, thaks. [14:38] benji: Could you please review https://code.launchpad.net/~abentley/launchpad/celery-everywhere-4/+merge/102918 ? [14:39] abentley: sure, how urgently do you need it? [14:39] benji: not urgent. [14:39] k [15:16] jelmer, if you have a minute, we need a build of jml's subunit filter branch in our PPA, and we are not having success with our recipe. We thought we copied all the relevant bits from your recipe (https://code.launchpad.net/~testing-cabal/+recipe/subunit-daily) to ours (https://code.launchpad.net/~yellow/+recipe/subunit-daily) but debhelper is not installed for precise or lucid. Can you take a glance and give us a [15:16] ny hints as to what we're doing wrong? If you don't have time, np, understood of course. can try someone else [15:16] gary_poster: you're missing python-testtools [15:17] gary_poster: you'll need an updated version - you can either also add the testtools recipe, or at the testing-cabal PPA as a dependency for yours [15:18] jelmer, ah ok. to add the testing-cabal PPA as a dependency, I need to fork your packaging branch, right? [15:18] gary_poster: no, that should be a setting in your PPA [15:25] If you're using debhelper, you should explicitly build-depend on it, anyway [15:25] (Or your packaging branch should [15:25] jelmer, I see. We do have a recent python-testtools in our PPA already though (0.9.14-bzr267~ppa40~precise1 in https://code.launchpad.net/~yellow/+archive/ppa , from the April 19) [15:25] ) [15:25] cjwatson, yeah that's what I was wondering [15:26] so we could fork jelmer's packaging and add it explicitly and try that... [15:26] will try [15:26] The only things you're allowed to assume are Essential: yes + build-essential [15:27] i.e. libc6-dev, gcc, g++, make, dpkg-dev [15:27] ack [15:27] thanks [15:31] cjwatson, gary_poster: there already is a dependency on debhelper >= 9 in the subunit package [15:31] Dependency or build-dependency? [15:32] sorry, build dependency [15:32] as far as I can tell this is merely pbuilder-satisfy-depends giving a confusing error [15:32] gary_poster: does your testtools ship python3-testtools? [15:33] jelmer, no. we found that part of the build to be broken [15:33] Ah, yeah [15:33] maybe we disabled it prematurely [15:33] we really need better CA stuff. [15:34] gary_poster: I suspect that's the real cause of the error - the subunit python3 package depends on the testtools python3 package, and the source package depends on it too [15:34] CI [15:34] ah :-/ [15:34] jelmer, ok so we need to go back to why python-testtools was not building for us [15:34] on oython 3 [15:35] gary_poster: do you actually care about ubuntu releases all the way back to lucid? [15:35] because if you do that might be the cause [15:37] jelmer, it is necessary to have this available in precise and it is convenient to have it available in lucid. don't care about the rest. [15:39] jelmer, so for python-testtools your suggestion is to change that to only precise, and return to the original packaging branch, and see if that works, yeah? trying that [15:40] gary_poster: that's what I was going to suggest, but looking at the testing-cabal recipe now.. [15:40] it looks like testtools is broken for python3 at the moment because it uses some python2-specific code in trunk [15:40] jml: ^ [15:40] https://launchpadlibrarian.net/102600248/buildlog_ubuntu-natty-i386.python-testtools_0.9.14%2Bbzr254~ppa37~natty1_FAILEDTOBUILD.txt.gz [15:45] gary_poster: you can either add a dependency to the testing-cabal in your PPA, copy the testtools package or your own recipes (perhaps based on the ~testing-cabal ones) that disable the python3 support [15:45] or of course contribute a fix for the python3 build issue in trunk :) [15:48] jelmer, :-) I'll plan to contribute fix but the other ideas help us *now* which is always nice. But we already have a working python-testtools build that copies the cabal packaging branch and recipe and disable the python 3 support; I guess you mean we wouls also then need to disable the python3 support in subunit to go down that road? [15:48] it struck me that we could also be gross as a temporary fix [15:48] well doesn't even have to be gross [15:48] gary_poster: ah, yes - if you have a working testtools (that's new enough) then just disabling the python3 support in the subunit package should be sufficient too [15:49] but we coud have our own testtools branch for now with the fix. OK, I see some roads out of this. Thanks jelmer! === jcsackett_ is now known as jcsackett [16:43] jcsackett, are you aware of BaseBreadcrumbTestCase [16:44] I think we want to extend it if it does not work to test your changes [16:45] * jcsackett looks [16:46] yeah, i remember taking a look at this, and discarding it for some reason. [16:46] i'll double check why i discarded it, and update it. [16:47] I think other developers will be confused if they use this test, then spend hours to discover it does not support their needs. [16:47] jcsackett, you have been frustrated by cases like this in the past. I think I should be able to add the view to the list of traversed objects to get breadscrumbs === salgado is now known as salgado-lunch [16:49] jcsackett, I think getBreadcrumbsForUrl() is the method that needs improvement [16:49] sinzui: yeah, that seems right. [16:50] jcsackett, I think getBreadcrumbsForObject() might work, but it is not obvious that the object can be a view. [16:52] I think I passed the view in lp.registry.browser.tests.test_breadcrumbs to verify some views were correct. === salgado-lunch is now known as salgado === matsubara is now known as matsubara-lunch [17:37] james_w: I'm trying to migrate all Jobs to be able to run under Celery, but it seems like CopyArchiveJob is unused. Is that correct? [17:53] benji: thanks for your review. [17:53] abentley: my pleasure === matsubara-lunch is now known as matsubara [19:11] morning [19:38] benji: Could you please review https://code.launchpad.net/~abentley/launchpad/celery-everywhere-5/+merge/103161 ? [19:40] abentley: sure [19:58] * sinzui has exceeded his writing about non-interesting problems again [19:58] * sinzui considers the last beer as crutch [19:59] flacoste: ping [19:59] flacoste: my calendar claims now is our catchup [20:25] abentley: done with your latest review [20:25] benji: Thanks. [20:25] np [21:08] lifeless: yeah, i apologize, i had a conflicting appointment with my osteopath, do you mind if we catch-up tomorrow instead? [21:11] sure === benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2 [22:14] StevenK: lib/lp/bugs/javascript/bug_subscription_portlet.js [22:16] wallyworld_: Thanks. [22:25] wallyworld_: sinzui: let me know when you want me [22:25] lifeless: will do, soon :-) === matsubara is now known as matsubara-afk [22:37] lifeless, we are on mumble [22:47] wallyworld_: lib/lp/services/longpoll/adapters/storm.py [22:47] wallyworld_: end of the file [23:02] jcsackett, r=me [23:22] wallyworld_: http://www.ubuntu.com/devices/android [23:42] wallyworld_: Branch updated, lp:~stevenk/launchpad/bugs-information_type-ui-portlet [23:42] cool. will test [23:43] wallyworld_: Ta. Just pastebin me the output of bin/test -vvt test_information_type_choice [23:43] yup [23:52] StevenK: https://pastebin.canonical.com/64795/ [23:53] StevenK: doing a make resulted in this error though [23:53] ERROR: LP_BUGS_BUGTASK_INDEX_PORTLETS is not mentioned in the meta file [23:53] make: *** [combobuild] Error 1 [23:55] That is probably me. [23:55] wallyworld_: Do I need to worry about the NOT loaded messages? [23:56] don't think so [23:56] wallyworld_: I've fixed the issue that caused the LP_BUGS_BUGTASK_INDEX_PORTLETS error. [23:57] cool [23:58] yes, now make is clean [23:59] Yeah, if you reference stuff that doesn't exist in requires, convoy will scream.