/srv/irclogs.ubuntu.com/2012/04/23/#launchpad-dev.txt

wgrantwallyworld_: How can you deserialise an enum?00:22
wallyworld_wgrant: using the lazr.json lib00:22
wgrantBut you can't know what type it is.00:22
wgrantSo it's impossible.00:22
wallyworld_the type is encoded in the serialisation00:23
wgrantAh00:23
wgrantThat sounds absolutely disastrous.00:23
wgrantly exploitable00:23
wallyworld_it's how most serialisation scheme i've seen work00:23
wgrantAnd most software is fucking terrible.00:24
wgrantWhat's your point? :)00:24
wallyworld_my point is i'm using common patterns and not reinventing any wheel00:24
wgrantwallyworld_: Ew00:39
wgrantAre you trying to turn JSON into pickle? :)00:39
wallyworld_wgrant: i'm not going to argue with you. the approach has been approved by curtis00:39
wgrantLetting user-specified JSON import arbitrary things, double-encoding stuff, embedding code paths in JSON...00:39
bigjoolswallyworld_: I haven't seen any suggestions from wgrant either00:40
wgrantI don't know if there is a good solution.00:41
wgrantJSON is too untyped to do this cleanly.00:41
wgrantWe could potentially do something with the enum name. But embedding the class path is extremely onerous and completely inappropriate.00:41
wallyworld_we need to be able to handle arbitrary complex objects - it's not just about enums00:42
wgrantJSON isn't for deserializing directly into complex objects.00:42
wallyworld_it provides hooks to do so00:43
wallyworld_and we use a pattern where we shove arbitrary stuff down rabbit00:43
wallyworld_so we need to be able to handle that00:43
wgrantThat pattern was simply convenient.00:43
wgrantWe have not studied its validity or correctness.00:44
wgrantMy inclination would be to fix that.00:44
wgrantRather than turn JSON into pickle.00:44
wgrantWith the various security and compatibility concerns that that entails.00:44
* lifeless notices the conversation00:51
lifelesssomeone want to give me a quick overview?00:51
wgrant'{"BaseItem,lazr.enum._enum.Item": "{\\"type\\": \\"Foo\\", \\"name\\": \\"BAR\\"}"}'00:51
wgrantlifeless: ^^00:52
lifelessthat looks a little like JSON00:52
lifelessbut whats the story around this?00:53
cjwatsonI would love a review of https://code.launchpad.net/~cjwatson/launchpad/init-packageset-fixes/+merge/10303600:55
wgrantlifeless: lazr.json is designed to serialize and deserialize arbitrary types from JSON.00:56
cjwatson(and QA suggestions as per my MP comments)00:56
wgrantlifeless: The initial case is lazr.enum.00:56
wgrantlifeless: It does this in the format shown above.00:56
lifelesswgrant: why are we doing this? wallyworld_ ^ ?00:56
wallyworld_sorry, what's the question?00:57
lifelesswallyworld_: I'd like to understand why we're doing arbitrary type encapsulation in JSON; its rather antithetical to JSON and thus has me confused.00:57
wallyworld_lifeless: give me a sec otp00:58
wgrantcjwatson: You could just say Packageset.distroseries_id.is_in(...)01:01
cjwatsonOh, that works?  OK, will try01:01
cjwatsonAh, yes, I see that pattern elsewhere01:02
cjwatsonnukes a circular import too01:03
wgrantYay01:03
cjwatsonwgrant: Updated01:08
wgrantcjwatson: Approved, thanks.01:11
cjwatsonWonderful.  'ec2 land --incremental' is right for this, isn't it?01:12
cjwatsonI haven't actually checked the state of the DB here01:12
wgrantWas going to tell you to use incremental, yep01:12
wgrantSince we should also add DB constraints01:12
wgrantLater01:12
cjwatsonRight01:12
cjwatsonOK, off to EC2 then01:13
wgrantGreat.01:13
cjwatsonWith any luck we can squeeze in a deployment pre-Q-opening ...01:13
wgrantIt's nodowntime now, so we'll hopefully have two or three before release.01:13
wallyworld_lifeless: with the json thing - we are currently serialising arbitrary objects and if there's a particular object class that is not supported, we oops. enum is one but there will be others. the approach taken is to allow object encoders/decoders to be plugged in and used to handle the cases json doesn't know about01:22
cjwatsonwgrant: on init-packageset-fixes, do you know if running initialize-distroseries on dogfood is going to be a saneish thing to do?01:28
cjwatsonsub-day runtime, that sort of thing :P01:29
wgrantcjwatson: It will take forever. We could initialise/create a small distro and do it there01:29
wgrantOr we could run on qastaging, actually01:29
wgrantWill still take a while, but probably 100x faster or so01:29
cjwatsonI'll need help for that, but that sounds like a better plan01:29
cjwatsonIIRC it's ten minutes or so on production01:30
wgrantDidn't we use the web UI last time?01:30
cjwatsongood point, I believe we did01:30
wgrantSo we shouldn't need webops to run stuff manually, hopefully.01:30
cjwatsonthat's what the NewReleaseCycleProcess notes say, anyway01:31
wgrantUnless the runner isn't cronned, which is plausible.01:31
* wgrant checks.01:31
cjwatsonMight still need somebody to deal with creating a test series on qastaging for the purpose01:32
cjwatsonThough I suppose we could just use q-series01:32
wgrantI'd just rename q-series to Quarrellous Quokka and use that, yeah.01:33
cjwatsonDoes it need a rename?01:33
StevenKwallyworld_: Are you still OTP?01:33
cjwatsonI mean, in order to test this01:34
wallyworld_StevenK: no01:34
StevenKwallyworld_: Can haz mumble, then?01:34
wgrantcjwatson: Not really, true.01:34
wallyworld_sure01:34
StevenKwallyworld_: http://pastebin.ubuntu.com/941907/01:36
lifelesswallyworld_: why are we serializing arbitrary objects?01:36
lifelesswallyworld_: and serialising arbitrary object doesn't imply *deserialising* them.01:37
lifelesswallyworld_: why is deserialising needed? what are we doing with this json ?01:37
wallyworld_lifeless: you tell me and we'll both know. poor design? i think we are listening to object modified events and shoving the result down rabbit01:37
wallyworld_lifeless: if we don't handle deserialisation, tests will fail. and i do think we deserialise at the other end of the mq?01:38
lifelesswallyworld_: We don't have Launchpad at the other end of the mq01:38
lifelesswallyworld_: which tests will fail if we don't deserialise?01:38
wgrant(there's an ObjectModifiedEvent hook on the Storm baseclass which shoves diffs into RabbitMQ)01:38
wgrant(this should probably be stopped001:38
lifelesswgrant: shoves to rabbit for what? txlongpoll ?01:39
wallyworld_lifeless: i would have to check and get back to you, can't recall exactly01:39
wgrantlifeless: Yes.01:39
wgrantlifeless: Currently only one thing is listened to: MP diff changes.01:39
lifelesswallyworld_: so, I'm going to raise a red flag here, I think this is heading into a world of pain and information disclosure.01:39
wgrantSpecifically the job status, I think01:39
wgrantYAY01:39
lifelesswallyworld_: can you please schedule a call with curtis, you and I (more optional, but we three must be there) to review this.01:40
wallyworld_lifeless: another potential use case - sending arbitrary data to jobs via rabbit01:40
wallyworld_lifeless: so, you don't agree we should support serialising arbitrary objects? json or not01:40
lifelessI don't agree.01:41
wallyworld_lifeless: so the only alternative is to make each bespoke use case do it's own flattening01:41
lifelessor rather, I don't think adding an additional typed-encoding in json makes any sense; mapping from rich objects to basic types is something we already do in the API via the interfaces.01:42
wallyworld_lifeless: how about we have a call tomorrow after our standup01:42
wallyworld_say around 9am AEST01:42
wallyworld_or you could join our standup01:42
wallyworld_starts at 8am AEST01:42
lifelesswallyworld_: there is an existing protocol for flattening objects; if we do want to do this, that would be the right way forward; AFAICT though, we don't /want/ this, we've stumbled on it via adding something to an existing code path01:43
lifelesswallyworld_: I will be here at both times, ping me :)01:43
wallyworld_lifeless: ok, will do. talk tomorrow01:43
wgrant(it also pretty much destroys our compliance to any of JSON's goals, and is likely to be insecure)01:47
StevenKwallyworld_: http://pastebin.ubuntu.com/941914/01:50
wallyworld_wgrant: well we need to decide if there's a use case for arbitrary object deserialisation. most enterprise systems support it and provide mechanisms to do so. it's a very common pattern.01:51
wallyworld_not sure why there's so much hostility given it's not exactly a new idea01:52
wgrantI am not completely averse to the entire idea.01:53
wgrantBut this is without a doubt not the right way to do it.01:53
lifelesswallyworld_: well, AIUI you're solving an OOPS we're seeing01:54
lifelesswallyworld_: I think the OOPS has uncovered us sending confidential object change metadata to arbitrary unauthenticated clients on the internet01:55
wallyworld_lifeless: yes, and in a way that handles arbitrary objects, which will happen, not just the oops we are seeing01:55
lifelesswallyworld_: not in practice, but in potentia, we have a dangerous API01:55
lifelesswriting a system to let that dangerous API be more powerful isn't a great use of time IMNSHO01:55
lifelesstrimming that system back to be less dangerous, which will also avoid the OOPS, will be.01:56
lifelessthe question of 'can we serialise everything' is tangential.01:56
lifelessserialise-everything opens inappropriate-disclosure holes; deserialise-everything opens bad state injection holes01:56
wallyworld_lifeless: sure, there's a balance to be achieved. we can discuss tomorrow on the call. too hard via irc01:58
lifelessyup01:59
lifelesstotally agree, thats why I suggested a call :)01:59
wallyworld_lifeless: btw, the arbitrary objects thing - the stuff you mention just above is solved in the decoder/encoder classes, so it's not as bad as you probably are thinking02:01
lifelesswallyworld_: I don't see how its solved02:04
wallyworld_lifeless: an arbitrary object is only serialised if there's been a decoder registered for it's type (and it will oops if there's none). i guess it is possible to register a  fake decoder and inject bad data, but that's an issue not unique to this case and we would need to deal with that i guess02:07
lifelessconsider a privileged field02:08
lifelesshow do you deserialise it *and* trust the result.02:08
lifelesssecondly, object graphs02:08
wallyworld_lifeless: the implementation handles object graphs just fine02:08
wallyworld_that's the point of it02:08
lifelessI'm sure it does02:08
lifelessbut thats where the vulnerabilities in these systems come in02:09
StevenKlifeless: I've been on Mumble with wallyworld_ for 30 minutes now, and you keep interrupting every 3 minutes!02:09
StevenKI would have been done by now!02:09
lifelessStevenK: hey, I'm just replying. Tell wallyworld_ to stop :P02:09
StevenKI have been!02:09
wallyworld_i will now. till tomorrow :-)02:10
StevenKNow I'm tell you BOTH to stop.02:10
lifelessin soviet russia, words speak you02:10
=== statik_ is now known as statik
wallyworld_StevenK: how goes the yui test writing?03:46
StevenKwallyworld_: Trying to muddle along04:18
wallyworld_sounds like fun04:20
StevenKwallyworld_: Firebug complains YUI isn't defined.04:21
StevenKAnd make lint is complaining about undefined entity in the HTML.04:21
wallyworld_hmmm. what are you trying to load?04:21
wallyworld_sounds like a syntax error in the html04:21
wallyworld_you may have stuff from the template left over04:22
StevenKLet me just get my current thought out of my head and into Vim and I'll repaste the diff04:22
wallyworld_ok04:22
StevenKwallyworld_: http://pastebin.ubuntu.com/942011/04:24
wallyworld_one sec04:24
wallyworld_StevenK: there's nothing obviously wrong with the html and test js that i can see. are you trying to load the html into your browser?04:29
StevenKYah04:29
wallyworld_one thing is that you may not have included all the necessary dependencies in the html04:30
wallyworld_all the js files required need to be specified manually04:30
wallyworld_so you are missing choiceedit.js and friends i thibk04:30
StevenKIt's just strange it's manifesting as YUI being undefined.04:31
wgrantDpesm04:31
wgrantEr04:31
wgrantDoesn't one normally use LPJS from firebug?04:32
StevenKThis is a YUI test.04:32
wallyworld_StevenK: if yui can't load all the dependencies it won't initialise04:33
StevenKAnd therefore isn't defined. Right.04:33
wallyworld_so if you say to load information_type.js and that required xxx.js and xxx.js isn't available, boom04:33
wallyworld_so it's a pita but you need to add in the things you think you need by hand04:34
wallyworld_to the html test harness04:34
StevenKThat's choiceedit, privacy and bugtask_index. Now to work out what else I need.04:45
StevenKwallyworld_: I guess base == yui.js itself, what about oop, node, event and io-base?04:49
lifelesswgrant: https://bugs.launchpad.net/bugs/+bugs?field.searchtext=foo+bar05:24
lifelesswgrant: and https://twitter.com/#!/pipesec/status/19420210458225868805:24
lifelesswgrant: look at only one05:24
wgrantlifeless: What about it?05:28
wgrantWorks fine for me.05:28
wgrantReturns a plausible number of results05:28
wgrantetc05:28
wgrantThe FTI index will take a while to page in on all 3 nodes.05:29
lifelesswgrant: times out for me, 3 out of three times05:29
wgrantMaybe you got a bad slave.05:29
lifelesshttps://lp-oops.canonical.com/oops.py/?oopsid=OOPS-f28e0087ffa2474538d8d21683092d1d05:29
wgrantJust got one in 8.6s rather than the normal 2.1s05:29
wgrantSo one slave may just be almost there.05:29
lifelesscould be05:29
wgrantReliably <2s now05:30
wgrant(note that this is still GiST... will improve with GIN in a couple of weeks :))05:30
wallyworld_StevenK: you need node etc in your test js, not the html as yui as a whole is loaded but you still need to specify the dependencies in the js as like with all yui modules05:33
wgrantlifeless: Currently the tiebreak sort key is always bugtask ASC or bug ASC. I've currently got the indices set up to handle the default direction for each of the common sort orders, but they doesn't work for the reverse because the tiebreaker is the wrong direction.05:34
StevenKwallyworld_: http://pastebin.ubuntu.com/942063/05:34
wgrantlifeless: Do you think the tiebreak order is significant enough that I should double the number of indices, or can I reverse the tiebreak order in that case?05:34
wgrant(currently reversing the order doesn't reverse the tiebreaker)05:34
wallyworld_StevenK: so you've included overlay.js in your html test harness?05:35
StevenKwallyworld_: http://pastebin.ubuntu.com/942064/ is the current test05:36
wallyworld_StevenK: /build/js/yui/yui/overlay/overlay.js is bogus05:36
wallyworld_it's more like lp/app/javascript/overlay/overlay.js05:36
wallyworld_/build/lp/app/...even05:37
StevenKThat's there too05:37
StevenKA few lines down05:37
wallyworld_ah so it is. you don't need any yui stuff in there though05:37
wallyworld_looks like you're missing anim05:38
wallyworld_probs also extras05:38
wallyworld_you sort of need to look at the requires bits of the things you are importing and import those dependencies also05:38
StevenKwallyworld_: Right. I was unsure if I needed to care about the YUI bits included too, like oop and event.05:39
wallyworld_not in the html, just the requires of the test js05:39
wallyworld_if required05:40
StevenKwallyworld_: But overlay only has: "requires": ["oop", "overlay", "event", "widget", "widget-stack", "widget-position"] ?05:42
lifelesswgrant: so you have foo bar quux id all asc, and reversed foo desc bar desc quux desc id desc ?05:42
wallyworld_StevenK: from memory, choiceedit requires those other things i mentioned05:43
wallyworld_yes, lp.anim05:43
wgrantlifeless: Currently we default to (importance ASC, bugtask DESC). When you reverse that listing, you get (importance DESC, bugtask DESC) rather than (importance DESC, bugtask ASC)05:43
wallyworld_StevenK: and lp.anim requires lp.extras05:44
StevenKwallyworld_: But choiceedit isn't complaining, and it's imported above overlay?05:44
lifelesswgrant: ok, thats a bug05:44
wgrantlifeless: Arguably.05:44
wallyworld_StevenK: such are the mysteries of yui. i'm not sure why that would be the behaviour05:44
wgrantSo I can change it?05:44
lifelessyes05:44
wgrantThanks.05:45
lifelessreversing the sort should reverse all the rows, which it won't be atm05:45
StevenKwallyworld_: http://pastebin.ubuntu.com/942069/ is a new diff05:45
wallyworld_StevenK: looks reasonable, but only way to know is to try05:46
StevenKwallyworld_: It still gives the YUI not defined errors05:48
wallyworld_hmmm. still missing dependencies i would guess. what's the branch?05:48
StevenKwallyworld_: Let me commit this test and push05:49
wallyworld_ok05:49
StevenKwallyworld_: lp:~stevenk/launchpad/bugs-information_type-ui-portlet05:50
wallyworld_ok, let me save my current branch05:50
wallyworld_StevenK: you appear to have one to many ../ in your js paths05:55
wallyworld_s/to/too05:55
StevenKwallyworld_: All of them?05:56
wallyworld_all the ones referencing build/***05:56
wallyworld_my IDE told me that immediately :-P05:57
StevenKBah05:57
StevenKI copied from the lp/app/testing/testrunner.js line05:57
wallyworld_StevenK: our javascript dirs are entirely 100% consistent05:58
wallyworld_StevenK: also, you will need05:58
wallyworld_      <script type="text/javascript"05:58
wallyworld_          src="../../../../../../build/js/lp/app/effects/effects.js"></script>05:58
wallyworld_      <script type="text/javascript"05:58
wallyworld_          src="../../../../../../build/js/lp/app/lazr/lazr.js"></script>05:58
StevenKeffects I added05:58
wallyworld_definitely lazr.js then also05:58
wallyworld_and i think that solves the initial bootstrapping of the test05:59
wallyworld_so now you just have to make everything pass :-)05:59
StevenKwallyworld_: But I need to drop a '../' from every src line involving build/js ?06:00
wallyworld_yes06:00
wallyworld_StevenK: but not the lp/app ones06:00
wallyworld_the template is correct for those includes06:00
StevenKOr the YUI one?06:00
wallyworld_correct06:01
StevenKLooks like only bugtask_index.js then06:01
StevenKBah, I still get YUI is not defined06:02
wallyworld_ah, sorry i meant the top 2 lp/apps lines06:02
wallyworld_id didn't notice the ones near the bottom were lp/app06:02
wallyworld_should 5 5 changes06:03
wallyworld_be06:03
wallyworld_1 bugs one and 4 lp app ones06:03
StevenKYou mean the one under Dependencies?06:03
wallyworld_yes06:03
wallyworld_the top 2 lp/app lines under dependencies are ok06:04
wallyworld_the next 4 aren't06:04
wallyworld_the next 2 are ok06:04
wallyworld_and then bug task isn't06:04
StevenKwallyworld_: So extras, anim, effects, lazr.js, overlay and expander?06:05
wallyworld_StevenK: https://pastebin.canonical.com/64716/06:05
wallyworld_i should have pasted the diff in the first place06:05
StevenKHaha, yes.06:06
wallyworld_sorry06:06
wallyworld_got myself confused06:06
wallyworld_the only correct lines were overaly and expander06:07
StevenKOooh, it loads the test now.06:08
StevenKY.lp.testing.mockio is undefined.06:08
wallyworld_should be 5 ../06:09
wallyworld_you may also find you need to include the subscription_list from lp/app06:09
StevenKwallyworld_: It's actually calling the JS test now.06:12
wallyworld_yay :-)06:12
StevenKTesting completed at Mon Apr 23 2012 16:13:21 GMT+1000 (EST).06:13
StevenKPassed:1 Failed:0 Total:1 (0 ignored)06:13
wallyworld_progress!06:13
StevenKMind you, the test doesn't assert anything yet. :-P06:13
StevenKBut yes, progress!06:13
wgrantlifeless: Barring massive issues I was aiming to turn it on for ~launchpad-beta-testers tomorrow morning.06:16
wgrantBut maybe earlier is good.06:16
stublifeless, wgrant: But I don't think perfect behaviour justifies doubling the number of indexes. Do real searches actually use the tiebreaker, or is it only to keep things perfectly stable for the test suite?06:16
wgrantstub: Real searches use them.06:16
wgrantstub: eg. Launchpad's default bug listings06:16
wgrantstub: We only use three statuses, so the tiebreaker is used for the majority of the sorting.06:17
stubok06:17
wgrants/statuses/importances/06:17
lifelesswgrant: tomorrow is a bad day to make changes06:19
lifelesswgrant: as wed is a pub hol06:19
wgrantMmmm, I guess.06:21
wgrantWill run some more tests and do it this evening.06:21
StevenKwallyworld_: I can't see how to grab hold of the choicesource after I simulate the click on the edit link.06:24
wallyworld_StevenK: bit more info?06:24
StevenKwallyworld_: I've got an edit link in the test HTML, which has href='#'. That is passed into setup_information_type_choice() so it should bring up the choicesource overlay when it's clicked.06:25
StevenKwallyworld_: I want to click one of the options on the choicesource06:26
wallyworld_right, i see. look at test_shareetable.js where i test the choicesource popups for changing info type for a sharee06:26
=== mthaddon` is now known as mthaddon
wallyworld_StevenK: test_permission_update_click06:27
wallyworld_lines 277-28206:28
StevenKwallyworld_: Right. I'm doing the same thing06:28
wallyworld_figured as much :-)06:28
StevenKvar choicesource = Y.one('.yui3-ichoicelist-content');06:29
StevenKAnd choicesource is null06:29
wallyworld_the click to open the choice source popup must be wrong06:29
StevenKHmm, I think I'm missing a div06:32
wallyworld_have you looked behind the couch?06:33
StevenKHah06:33
wallyworld_one of my favourite t-shirts is "I found Jesus. He was hiding behind the sofa"06:34
StevenKRight, that gets the choicesource working06:35
centohi06:36
centois possible to browse launchpad project by program language?06:36
lifelessyou could use the API, but we don't have a built in thing for that AFAIK06:37
centolifeless, and through web page?06:39
centofor example, to list all Python project06:39
wallyworld_StevenK: the test passes?06:39
StevenKwallyworld_: I have it working, I just need to figure out how to get the content of the <strong> so I can assert against it.06:40
wallyworld_node.get('text')06:40
wgrantStevenK: btw, while you're there, care to fix the capitalisation in BugSecrecyEditView.label?06:41
StevenKWhich capitalisation?06:41
wgrant        if bool(getFeatureFlag(06:41
wgrant            'disclosure.show_information_type_in_ui.enabled')):06:41
wgrant            label += 'Information type'06:41
wgrant        else:06:41
wgrant            label += 'visibility and security'06:41
wgrantThe first has a capital, the second does not.06:41
wgrantNeither should.06:41
StevenKRight06:41
wgrantlifeless: Meh, shall I play dangerously and switch it on now?06:43
wgrantI can't see any issues, apart from the rare index usage due to tiebreaker, but it's still strictly better than the old code.06:43
lifelesswgrant: for all beta +106:44
wgrantThanks.06:44
lifelesswgrant: lets hit it fully on thurs.06:44
wgrantSounds good.06:45
wgrantWe may see particular issues with searching by structsub or tag, but it should be slightly better than the old one.06:45
StevenKwallyworld_: Y.lp.bugs.bugtask_index.portlets is undef :-(06:52
StevenKAnd adding "lp.bugs.bugtask_index.portlets" to requires didn't help06:52
wallyworld_StevenK: did you include subscription_list?06:59
StevenKwallyworld_: Yup06:59
wallyworld_hmmm. must be something else. i'll see if i can see what it is07:00
wallyworld_StevenK: lp.app.errors, lp.app.confirmationoverlay07:00
StevenKFor portlets?07:01
wallyworld_lp.bugs.javascript.subscription07:01
wallyworld_i think portlets renders subscriptions?07:01
wallyworld_haven't checked07:01
wallyworld_but if so, then subscriptions uses those things07:01
StevenKSo I should include lp/bugs/subscription.js too?07:01
wallyworld_yes07:02
wallyworld_subscribers_list is the other one I was thinking off07:03
wallyworld_of07:03
StevenKwallyworld_: Firebug showing NOT loaded is stuff I'm including and don't have to?07:05
wallyworld_can't remember. example?07:05
wgrantThere's one thing that always appears there07:06
wgrantwidget-position-ex or something07:06
StevenKyui: NOT loaded: lp07:06
StevenKyui.js (line 4009)07:06
wallyworld_that's spurious, ignore it07:06
wallyworld_the lp one shouldn't appear07:07
wallyworld_that means there is a missing dependency07:07
wallyworld_StevenK: i'm off to soccer, back later. i'll check in to see how you're going07:10
StevenKwallyworld_: I should stop working anyway07:10
wallyworld_you left off " and start drinking"07:11
StevenKHah07:11
wallyworld_i might have a few after soccer07:11
StevenKI've been dealing with JS all day, that's *implied*.07:11
wallyworld_yeah, i'm stuck in js as well atm07:11
StevenKwallyworld_: Besides s/drinking/drinking heavily/07:12
wallyworld_almost got it sorted07:12
wgrantDidn't you hear, we're rewriting LP in node.js.07:12
wallyworld_luckily i know you're joking07:12
adeuringgood morning07:30
wgrantstub, lifeless: https://code.launchpad.net/~wgrant/launchpad/bugtaskflat-more-indices/+merge/10306309:24
wgrantHeh09:24
wgrantThe backup completed 5s before I proposed the live index changes.09:24
wgrantHandy.09:24
lifelessstub: you've got that ?09:26
stubyer09:26
stubWill ordering by date_last_updated without a tie-breaker break tests? I guess fixing the tests would be better.09:28
stubWe could fake date_created by using id instead09:29
stubI wonder if this would be better accomplished by dropping the IS NOT NULL on the existing partial indexes? Not for this live update, but that might work.09:31
stubNo obvious errors. No point me questioning the utility of the indexes - I understand they are driven by the searches we allow09:40
wgrantstub: date_last_updated and date_created have always been regarded as unambiguous. We've never used a tiebreaker with them, despite the fact that they could technically be ambiguous.09:46
stubWe have, but only to keep tests happy and that was probably a mistake.09:47
wgrantThe existing bug search code did not use a tiebreaker.09:47
stubbetter data is a better option09:47
wgrantSo the new one does not.09:47
stubYer, that is all fine.09:48
wgrantstub: Dropping the IS NOT NULL doesn't work reliably.09:48
wgrantThese indices are just for sorting.09:48
wgrantAll the existing ones work marvellously :)09:49
stubk. Would be nice if they did. My guess would be they would often be used, but occasionally odd stats would cause them to be skipped. But empirical results override my assumptions :)09:50
wgrantstub: Dropping the NOT NULL doesn't help contextless sorts, as the index will still be primarily ordered by context.09:51
stuboh, right09:51
wgrantUnfortunately COUNT(*) over Ubuntu's open bugs still takes slightly over a second.09:53
wgrantBut better than the 4s it used to take.09:54
czajkowskijtv: ping10:01
stubwgrant: At least it isn't EOVERFLOW yet10:02
stubwgrant: Those indexes are live on staging & qastaging10:02
jtvhi czajkowski10:02
wgrantstub: Let me do a few checks.10:02
wgrantMarvellous, https://bugs.qastaging.launchpad.net/bugs/+bugs in 1.5s10:03
stubAnything stopping us taking this live?10:04
wgrantJust checking Ubuntu.10:04
wgrantBut it looks good.10:04
czajkowskijtv: sorry I'm ssure it's late over there but a bit unsure about https://bugs.launchpad.net/launchpad/+bug/987199 dpm reported on translations10:04
wgrantYeah, all more than fine. Enlivenate.10:04
_mup_Bug #987199: Automatic translation exports fail to commit latest translations  <Launchpad itself:Confirmed> < https://launchpad.net/bugs/987199 >10:04
jtvczajkowski: I don't think it's that changes weren't detected.  More likely is that the exports are breaking.  You'll want to watch the launchpad-error-reports mailing list for error output from the translations-to-branch export script.  Often it's a lingering branch lock.  Another possibility is that imports and exports cross, in which case the export script backs off, for fear of overwriting something.10:08
wgrantThere's only one perma-locked branch.10:08
wgrantThe export script hasn't complained about ubuntu-doc ever AFAICR10:08
wgrantnautilus-pastebin is the only regular victim.10:08
czajkowskihmm need to check folders for launcpad-error-reports ml10:09
wgrantczajkowski: You're not subscribed by default.10:10
czajkowskinope I'm not10:10
czajkowskijust checked all my folders10:10
jtvczajkowski, wgrant: I suppose it's not unthinkable that the translations branch was broken _for a while in the past_, and changes accumulated during that time.10:17
wgrantI'm embarrassingly ignorant of how this stuff works.10:18
jtvIf you know anything at all, that's far from embarrassing.  As I recall, the exporter—at least at the time I wrot eit—lacked a way to see whether a particular change had already been exported.10:19
jtvInstead, it just exported changes from some fixed time window — even if that meant occasionally exporting something twice.10:19
jtvThe branch is clearly getting translation updates committed to it.10:19
jtvAnother possibility is might be that the translations are activated in such a way that their timestamps aren't updated.10:20
jtvczajkowski, wgrant: errr… there _was_ a discrepancy, right?  Because I see 337 untranslated strings on both the Ubuntu side and the upstream side now.10:21
wgrantI trusted the bug, didn't check myself.10:22
jtvOh, need to check the branch contents apparently, not the translation stats.10:22
jtvShould've known; David's very thorough.10:22
wgrantThat's why I didn't bother checking :)10:22
jtvGood call, as it turns out.  Yes, the branch has a bunch of untranslated strings that are translated in Launchpad.10:26
jtvSo it sounds as if sharing is to blame.10:26
jtvNot certain, of course.10:27
czajkowskinods10:28
czajkowskithanks jtv10:28
jtvSo the suspicion is that a string can become current, without updating timestamps in such a way that the translations-to-branch exporter notices that anything new needs exporting.10:29
dpmjtv, yes, as mentioned on the bug, it seems the stats in LP are correct, it's just the branch that does not seem to get them committed.10:29
jtvHi dpm10:29
dpmjtv, hey, thanks for looking at this :)10:29
jtvI *think* it's the timestamp on the POFile that the translations-to-branch exporter looks at, and which most likely does not get updated.10:30
dpmjtv, could it be that translations have been done offline, uploaded, and LP hasn't noticed there have been some changes?10:30
jtvShouldn't make a difference in itself.10:30
dpmah, good to know10:30
jtvAny idea whether those changes were made in the Precise series itself, or made in e.g. Oneiric and shared with Precise?10:31
dpmin Precise, afaik, which is what the French team were translating10:31
jtvBecause I don't _think_ (mostly guesswork for me at this point) that we update the POFile timestamp when we share translations.  It's actually pretty costly to figure out which POFiles might be affected.10:31
jtvHmm… if the work was done in Precise itself, you'd expect that timestamp to update.10:31
dpmthey might have been doing the work on the upstream project too10:32
jtvAh10:34
jtvThat'd do it.10:34
dpmwhat do you mean exactly?10:35
jtvWell, sharing an individual string's translation (a TranslationMessage) between upstream and the Ubuntu package is a matter of setting an extra “this is the current translation for Ubuntu” flag on the TM.10:35
dpmwhat's a TM and who or what sets that flag?10:36
dpmah, TranslationMessage10:36
dpmbut the sharing seems to happen, just the commit doesn't seem to10:37
jtvTranslationMessage.  LP sets that flag to make it the current translation in a given context; each TM has one such flag for current-upstream and current-in-Ubuntu.  Setting that flag may affect Ubuntu-side POFiles for the same language in whose POTemplate the TM's POTMsgSet participates, _and_ which don't have another, newer translation to that language _and_ which don't have a diverged translation for that language.  With me so far?10:37
jtvNever mind, I'm pulling your leg.  :)10:37
jtvIt's a bit complicated, and that's exactly my point.10:37
jtvWhen LP sets that flag, it's doing something relatively simple in itself.  But once you start looking at all the POFiles that it might possibly affect, it gets a bit complicated.10:38
jtvAnd so I suspect we don't really seek out those affected POFiles to update their timestamps.10:38
jtvAnd if we don't update their timestamps, chances are the translations-to-branch exporter doesn't notice that anything's changed in its version of the POFile.10:39
dpmoh, I thought the point of having translation sharing between upstreams and Ubuntu was to be able to commit Ubuntu translations to the upstream projects. Do I understand it correctly that that actually does not happen by design? Or is it rather a bug?10:39
jtvIt goes both ways, with exceptions.10:40
jtv(The exceptions are: Ubuntu translations only go automatically into upstream if the reviewer happens to have review privileges upstream as well; diverged messages, which override a message's translation in just one upstream release series or in just one Ubuntu release, stay diverged; and I think changes on either side don't override those on the other side unless the translation on the other side is either missing or identical to the old translatio10:42
jtvAll that seems to work just fine, but it all clouds our view of which POFiles just received a new translation.  It's a lot of work to get the exact set.10:42
jtvAnd so, I don't think we actually do all that work.10:43
jtvA fresh translation upstream will probably automatically go into all Ubuntu releases of the package that have the same English string, but probably without updating POFile.date_changed for all those places it ends up.10:44
dpmOk, so the summary is that I should stop recommending maintainers to use sharing + automatic exports and tell them to request tarball exports instead. Does that sound reasonable?10:45
jtvIt'd be a sad step back.  :(10:45
dpmabsolutely, and I'll be the first to regret it :/, but now I understand that sharing + exports does not work as expected, which I didn't know before.10:46
jtvAnother equally sad way to cope would be to make some arbitrary change to the Ubuntu-side POFile.10:47
jtvI didn't know it either.  I'm not even entirely sure.  But I did just verify that POFile.date_changed is what drives the exports.10:47
dpmjtv, thanks a lot for the research and the help. I think it might make sense to add the relevant parts of the conversation to the bug. I'd volunteer for it, but you're the one who knows how everything fits together, would you mind adding your findings so far?10:57
jtvOK10:57
jtvdpm: I've updated the bug.  I hope it's clear enough; if not, please say.11:01
dpmthanks a lot jtv, it looks clear enough to me for the parts I can understand :)11:07
jtvOK.  Remember, if you really need one of these exports to happen, just make an arbitrary change and then re-active the original translation.11:08
jtvNot very nice, but it should do the trick for that POFile.11:08
=== jamestunnicliff_ is now known as jamestunnicliffe
dpmjtv, gotcha, that's good to know, as a workaround, thanks.11:12
czajkowskijtv: marked triage and low, should it be higher?11:13
jtvczajkowski: I would call David's problem a high priority; the one about the wrong credits a low priority.11:13
czajkowskinods11:14
czajkowskijtv: fixered cheers11:14
jtvok11:14
=== Pendulum_ is now known as Pendulum
czajkowskirick_h: are you alive over there yet? if not ping when you come on thanks.11:52
rick_hczajkowski: yep, I'm alive11:52
czajkowskirick_h: did you do stuff with OAuth mails last week ?https://answers.launchpad.net/launchpad/+question/19428111:53
rick_hczajkowski: yes I did11:53
rick_hczajkowski: I'll reply.11:53
czajkowskilovely jubbly11:54
czajkowskicheers and happy monday to you :)11:54
rick_hyou too!11:55
rick_hczajkowski: answered12:01
rick_hand doh, newline fail on my twitter reply :/12:01
czajkowskirick_h: :) I did say please after all12:03
=== benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 3.47*10^2
rick_hczajkowski: yea, supposed to be a nice bash response: command not found12:03
czajkowskisounds like monday12:03
czajkowskirick_h: good weekend ?12:03
rick_hbusy, but yea can't complain12:04
rick_hnow monday...she's really starting to get irritating12:05
deryckMorning, all.13:00
jmlhi13:01
jmlgary_poster: fwiw, I haven't had a chance to look at bug 986421 and bug 986434, and probably won't get a chance soon.13:01
_mup_Bug #986421: In parallel tests, global tags eventually become misapplied <Testrepository:New> < https://launchpad.net/bugs/986421 >13:01
_mup_Bug #986434: global tags unnecessarily inflate concurrent test results <testtools:New> < https://launchpad.net/bugs/986434 >13:01
gary_posterjml, hey.  np.  the inflation one is no biggie.  The global misapplication is actually not a showstopper for us either right now but we may want to investigate it and propose something if we make progress elsewhere.  In terms of official release of the packages, I think the global tag one should be addressed before a release.  The file inflation one might be invalid as far as you are concerned.13:03
jmlwell, I think a strong case can be made for it being a genuine bug.13:04
gary_postercool13:05
abentleywgrant: is bug 983420 really fix released?13:26
_mup_Bug #983420: Celery branch scan jobs intermittantly fail to show revisions for new branches <branch-scanner> <fast-slow-lane-jobs> <qa-needstesting> <Launchpad itself:Fix Committed by abentley> < https://launchpad.net/bugs/983420 >13:26
abentleywgrant: Oh, nm.13:26
wgrantabentley: Yeah, sorry, misclicked and reverted.13:28
abentleywgrant: np.13:28
czajkowskisinzui: did you just spring clean license review?14:20
sinzuiczajkowski, I send a license-conflict email to one that was in review14:21
czajkowskisinzui: ah that was the one I'd left to check with you14:21
czajkowskijust wondered refreshed page and was gone14:21
czajkowskigood day :)14:21
rick_hderyck: ping14:26
deryckrick_h, yo14:26
rick_hderyck: so I think the issue is that the distro source pkg isn't 'published' so using the factory.makeSourcePackgePublishingHistory14:26
rick_hderyck: do you know if I should make sure it's published for each series then?14:27
deryckrick_h, honestly, I don't know.  sorry.14:27
rick_hI guess does that click/make sense14:27
rick_hok, will test it out thanks14:27
deryckrick_h, yeah, I'd guess you need it in each series first. but don't know for sure.14:27
StevenKrick_h: I'm on my way to bed, but an SPPH is per series and has a status. I think the default is PENDING.14:30
StevenKThe enum is in soyuz or archivepublisher. Probably the former.14:31
rick_hStevenK: ok, thaks.14:31
abentleybenji: Could you please review https://code.launchpad.net/~abentley/launchpad/celery-everywhere-4/+merge/102918 ?14:38
benjiabentley: sure, how urgently do you need it?14:39
abentleybenji: not urgent.14:39
benjik14:39
gary_posterjelmer, if you have a minute, we need a build of jml's subunit filter branch in our PPA, and we are not having success with our recipe.  We thought we copied all the relevant bits from your recipe (https://code.launchpad.net/~testing-cabal/+recipe/subunit-daily) to ours (https://code.launchpad.net/~yellow/+recipe/subunit-daily) but debhelper is not installed for precise or lucid.  Can you take a glance and give us a15:16
gary_posterny hints as to what we're doing wrong?  If you don't have time, np, understood of course.  can try someone else15:16
jelmergary_poster: you're missing python-testtools15:16
jelmergary_poster: you'll need an updated version - you can either also add the testtools recipe, or at the testing-cabal PPA as a dependency for yours15:17
gary_posterjelmer, ah ok.  to add the testing-cabal PPA as a dependency, I need to fork your packaging branch, right?15:18
jelmergary_poster: no, that should be a setting in your PPA15:18
cjwatsonIf you're using debhelper, you should explicitly build-depend on it, anyway15:25
cjwatson(Or your packaging branch should15:25
gary_posterjelmer, I see.  We do have a recent python-testtools in our PPA already though (0.9.14-bzr267~ppa40~precise1 in https://code.launchpad.net/~yellow/+archive/ppa , from the April 19)15:25
cjwatson)15:25
gary_postercjwatson, yeah that's what I was wondering15:25
gary_posterso we could fork jelmer's packaging and add it explicitly and try that...15:26
gary_posterwill try15:26
cjwatsonThe only things you're allowed to assume are Essential: yes + build-essential15:26
cjwatsoni.e. libc6-dev, gcc, g++, make, dpkg-dev15:27
gary_posterack15:27
gary_posterthanks15:27
jelmercjwatson, gary_poster: there already is a dependency on debhelper >= 9 in the subunit package15:31
cjwatsonDependency or build-dependency?15:31
jelmersorry, build dependency15:32
jelmeras far as I can tell this is merely pbuilder-satisfy-depends giving a confusing error15:32
jelmergary_poster: does your testtools ship python3-testtools?15:32
gary_posterjelmer, no.  we found that part of the build to be broken15:33
cjwatsonAh, yeah15:33
gary_postermaybe we disabled it prematurely15:33
jmlwe really need better CA stuff.15:33
jelmergary_poster: I suspect that's the real cause of the error - the subunit python3 package depends on the testtools python3 package, and the source package depends on it too15:34
jmlCI15:34
gary_posterah :-/15:34
gary_posterjelmer, ok so we need to go back to why python-testtools was not building for us15:34
gary_posteron oython 315:34
jelmergary_poster: do you actually care about ubuntu releases all the way back to lucid?15:35
jelmerbecause if you do that might be the cause15:35
gary_posterjelmer, it is necessary to have this available in precise and it is convenient to have it available in lucid.  don't care about the rest.15:37
gary_posterjelmer, so for python-testtools your suggestion is to change that to only precise, and return to the original packaging branch, and see if that works, yeah?  trying that15:39
jelmergary_poster: that's what I was going to suggest, but looking at the testing-cabal recipe now..15:40
jelmerit looks like testtools is broken for python3 at the moment because it uses some python2-specific code in trunk15:40
jelmerjml: ^15:40
jelmerhttps://launchpadlibrarian.net/102600248/buildlog_ubuntu-natty-i386.python-testtools_0.9.14%2Bbzr254~ppa37~natty1_FAILEDTOBUILD.txt.gz15:40
jelmergary_poster: you can either add a dependency to the testing-cabal in your PPA, copy the testtools package or your own recipes (perhaps based on the ~testing-cabal ones) that disable the python3 support15:45
jelmeror of course contribute a fix for the python3 build issue in trunk :)15:45
gary_posterjelmer, :-) I'll plan to contribute fix but the other ideas help us *now* which is always nice.  But we already have a working python-testtools build that copies the cabal packaging branch and recipe and disable the python 3 support; I guess you mean we wouls also then need to disable the python3 support in subunit to go down that road?15:48
gary_posterit struck me that we could also be gross as a temporary fix15:48
gary_posterwell doesn't even have to be gross15:48
jelmergary_poster: ah, yes - if you have a working testtools (that's new enough) then just disabling the python3 support in the subunit package should be sufficient too15:48
gary_posterbut we coud have our own testtools branch for now with the fix.  OK, I see some roads out of this.  Thanks jelmer!15:49
=== jcsackett_ is now known as jcsackett
sinzuijcsackett, are you aware of BaseBreadcrumbTestCase16:43
sinzuiI think we want to extend it if it does not work to test your changes16:44
* jcsackett looks16:45
jcsackettyeah, i remember taking a look at this, and discarding it for some reason.16:46
jcsacketti'll double check why i discarded it, and update it.16:46
sinzuiI think other developers will be confused if they use this test, then spend hours to discover it does not support their needs.16:47
sinzuijcsackett, you have been frustrated by cases like this in the past. I think I should be able to add the view to the list of traversed objects to get breadscrumbs16:47
=== salgado is now known as salgado-lunch
sinzuijcsackett, I think getBreadcrumbsForUrl() is the method that needs improvement16:49
jcsackettsinzui: yeah, that seems right.16:49
sinzuijcsackett, I think getBreadcrumbsForObject() might work, but it is not obvious that the object can be a view.16:50
sinzuiI think I passed the view in lp.registry.browser.tests.test_breadcrumbs to verify some views were correct.16:52
=== salgado-lunch is now known as salgado
=== matsubara is now known as matsubara-lunch
abentleyjames_w: I'm trying to migrate all Jobs to be able to run under Celery, but it seems like CopyArchiveJob is unused.  Is that correct?17:37
abentleybenji: thanks for your review.17:53
benjiabentley: my pleasure17:53
=== matsubara-lunch is now known as matsubara
lifelessmorning19:11
abentleybenji: Could you please review https://code.launchpad.net/~abentley/launchpad/celery-everywhere-5/+merge/103161 ?19:38
benjiabentley: sure19:40
* sinzui has exceeded his writing about non-interesting problems again19:58
* sinzui considers the last beer as crutch19:58
lifelessflacoste: ping19:59
lifelessflacoste: my calendar claims now is our catchup19:59
benjiabentley: done with your latest review20:25
abentleybenji: Thanks.20:25
benjinp20:25
flacostelifeless: yeah, i apologize, i had a conflicting appointment with my osteopath, do you mind if we catch-up tomorrow instead?21:08
lifelesssure21:11
=== benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
wallyworld_StevenK: lib/lp/bugs/javascript/bug_subscription_portlet.js22:14
StevenKwallyworld_: Thanks.22:16
lifelesswallyworld_: sinzui: let me know when you want me22:25
wallyworld_lifeless: will do, soon :-)22:25
=== matsubara is now known as matsubara-afk
sinzuilifeless, we are on mumble22:37
lifelesswallyworld_: lib/lp/services/longpoll/adapters/storm.py22:47
lifelesswallyworld_: end of the file22:47
sinzuijcsackett,  r=me23:02
lifelesswallyworld_: http://www.ubuntu.com/devices/android23:22
StevenKwallyworld_: Branch updated, lp:~stevenk/launchpad/bugs-information_type-ui-portlet23:42
wallyworld_cool. will test23:42
StevenKwallyworld_: Ta. Just pastebin me the output of bin/test -vvt test_information_type_choice23:43
wallyworld_yup23:43
wallyworld_StevenK: https://pastebin.canonical.com/64795/23:52
wallyworld_StevenK: doing a make resulted in this error though23:53
wallyworld_ERROR: LP_BUGS_BUGTASK_INDEX_PORTLETS is not mentioned in the meta file23:53
wallyworld_make: *** [combobuild] Error 123:53
StevenKThat is probably me.23:55
StevenKwallyworld_: Do I need to worry about the NOT loaded messages?23:55
wallyworld_don't think so23:56
StevenKwallyworld_: I've fixed the issue that caused the LP_BUGS_BUGTASK_INDEX_PORTLETS error.23:56
wallyworld_cool23:57
wallyworld_yes, now make is clean23:58
StevenKYeah, if you reference stuff that doesn't exist in requires, convoy will scream.23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!