[00:05] <StevenK> wgrant: http://bazaar.qastaging.launchpad.net/~launchpad-pqm/launchpad/devel/files looks okay to my testing, but I'm not sure how to stress it
[00:09] <wgrant> StevenK: Have you tried some non-ASCII paths?
[00:11] <StevenK> That means finding a branch that happens to have some, or making my own...
[00:20] <wgrant> danilos: Are bug #869264 and bug #730610 duplicates?
[00:20] <_mup_> Bug #869264: Old upstream translations overwrite newer upstream translations in Ubuntu <lp-translations> <message-sharing> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/869264 >
[00:20] <_mup_> Bug #730610: translation uploads from old series replace the current series translations <message-sharing> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730610 >
[00:23] <StevenK> wgrant: http://bazaar.qastaging.launchpad.net/~stevenk/+junk/foo/files
[00:24] <wgrant> StevenK: Does that crash on prod?
[00:24] <StevenK> Nope
[00:25] <wgrant> Then it's not a dreadfully good testcas
[00:25] <wgrant> e
[00:26] <wgrant> It's in a dirname function
[00:26] <wgrant> So it will probably need to be a directory name
[00:27] <StevenK> Yes, I'm pushing r2 now
[00:28] <StevenK> http://bazaar.launchpad.net/~stevenk/+junk/foo/files/head:/foo%C7%80%C6%B6/foo%C6%86/ => OOPS
[00:28]  * StevenK reaches for qas
[00:28] <StevenK> wgrant: OOPS-3b6ba4d894000fb4d96e6e6e947ef2cd on prod, a lack of oops on qas
[00:28] <wgrant> StevenK: Marvellous
[00:29] <wgrant> Does it look sane on qas?
[00:29] <StevenK> Yes
[00:29] <wgrant> So it does
[00:29] <StevenK> wgrant: http://bazaar.qastaging.launchpad.net/~stevenk/+junk/foo/files
[00:29] <wgrant> qa-ok!
[00:29] <StevenK> wgrant: Does sir wish for a NDT?
[00:29] <wgrant> Indeedily.
[03:45] <wgrant> Ahhh
[03:45] <wgrant> Tracked down the translation consistency bug finally
[03:45] <wgrant> The export bug
[03:45] <wgrant> czajkowski: ^^
[03:57] <StevenK> wgrant: Oh?
[04:00] <wgrant> StevenK: Filtering a left join in the wrong place
[04:00] <wgrant> Similar to a mailing list bug we had a couple of years ago
[04:00] <StevenK> Handy
[04:00] <StevenK> And sort of subtle without tests
[04:01] <wgrant> Yeah
[04:02] <wgrant> It excludes messages when the potmsgset is shared, and the only translation is a diverged one for the other side.
[04:35] <wgrant> jtv: Around?
[04:35] <jtv> Hi wgrant
[04:35] <jtv> I mean: yes.
[04:35] <wgrant> jtv: Hi
[04:36] <wgrant> jtv: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/translations/doc/vpoexport.txt#L101
[04:36] <wgrant> The last section
[04:36] <wgrant> I think the test is insane
[04:36] <wgrant> Do you remember enough about this to sanity check it?
[04:36] <wgrant> The test asserts that the untranslated messages aren't returned by getTranslationRows
[04:37] <wgrant> But normal untranslated messages *are* returned
[04:37] <jtv> Im looking.
[04:37] <wgrant> The ones in this test are not returned due to a probably SQL bug which excludes messages if the only translation is a diverged one on the other side
[04:37] <jtv> Oh, by the way: it's a doctest, so safe assumption that it's insane.
[04:37] <wgrant> If I were to create a new template item, it would show up there
[04:37] <wgrant> Hhe
[04:37] <wgrant> Yes
[04:38] <jtv> Excludes messages from what?  From the list of suggestions?  Or from the translation?
[04:38] <jtv> Because if the only translation is a diverged one on the other side, that means there's no translation where you're looking.
[04:38] <wgrant> getTranslationRows is used for PO exports, so it's meant to return all of the template's messages
[04:38] <wgrant> Even if they're untranslated
[04:39] <wgrant> It achieves this, except that the left join condition is in the wrong place, so if the only translation for a potmsgset is a diverged translation for another template, the message doesn't show up at all, even with a null translation.
[04:39] <jtv> Oh, the potmsgset is being left out?
[04:39] <jtv> Gotcha.
[04:39] <wgrant> The test asserts that behaviour; both of those examples at the end should also return a None row AFAICT
[04:41] <jtv> Let me just look closer into what getTranslationRows was meant to do...
[04:41] <wgrant> Sure
[04:42] <wgrant> https://pastebin.canonical.com/87982/
[04:42] <wgrant> If I add a third message with no translations at all, it shows up, as I assume the correct behaviour is
[04:42] <jtv> #@$ 2fa
[04:43] <wgrant> But the messages that have only diverged translations don't show up at all.
[04:43] <wgrant> http://paste.ubuntu.com/5654235/
[04:43] <jtv> Trying to load that first pastebin first.
[04:43] <wgrant> The second one is the same
[04:43] <wgrant> But no 2FA :)
[04:43] <jtv> Ah
[04:43] <jtv> I'll have a look.
[04:46] <jtv> Why, why, why did nobody accept my proposal of using null instead of 0 as a sequence number for obsolete messages?
[04:46] <wgrant> I have been wondering that for a good portion of the last week
[04:46] <wgrant> I hoped you'd have an excuse :P
[04:47] <jtv> Well okay, I think I remember the reason: it was left for later.
[04:48] <wgrant> Heh
[04:48] <jtv> I guess the condition on TranslationMessage.potemplate is in the wrong place, yeah.
[04:48] <wgrant> Yeah
[04:48] <wgrant> If I move it into the left join where it should be, the bug goes away and that test fails
[04:49] <jtv> Watch out for performance changes.
[04:49] <jtv> (Even if it gets faster, I'd like to hear about it :)
[04:49] <wgrant> There's also the slightly confusing aspect that in its current position it's actually a three-way disjunction: no TM, undiverged TM, or diverged TM for this template
[04:49] <wgrant> Yeah
[04:49] <wgrant> The existing query is absolutely dreadful, so I'm not too concerned
[04:49] <wgrant> But I will check
[04:50] <wgrant> Oh
[04:50] <wgrant> The query itself is only a couple of seconds
[04:50] <wgrant> I guess the data transfer and object loading may be big
[04:50] <wgrant> 'cause it appears to take about a minute
[04:50] <wgrant> When run in a harness
[04:50] <jtv> FWIW VPOExport used to be a view in the database.  Meant as an optimization, had turned into maintenance *and* performance drag.
[04:51] <jtv> A minute in harness?  What kind of dataset are you  running against!?
[04:51] <wgrant> ddtp-ubuntu-universe
[04:51] <jtv> Ah OK then :)
[04:51] <wgrant> aka. the biggest template
[04:51] <wgrant> It's what the bug was reported against
[04:51] <wgrant> And I had no idea what was wrong
[04:51] <wgrant> So couldn't really minimise it
[04:51] <jtv> Any major speed changes for that query though?
[04:51] <wgrant> Checking
[04:51] <jtv> Congratulations on spotting this.  Good sleuthing.
[04:52] <StevenK> And wgrant has spent 1.5 weeks with his head in rosetta :-(
[04:52] <wgrant> (the export of just the French raring translation takes upwards of 5 minutes, so testing without a harness is a bit painful...)
[04:52] <jtv> Gah
[04:52] <wgrant> StevenK: This bug is only today, though
[04:52] <jtv> So many things I would still have loved to have time to clean up...
[04:52] <jtv> Import approvals in particular!
[04:52] <StevenK> jtv: Like lp.translations.model.* ? :-)
[04:52] <jtv> Pretty much.
[04:53] <jtv> The trick with these things is that you have to hate complexity, ambiguity, and tech debt.  Seek it out & destroy it.
[04:54] <jtv> Oh, and you want my guess as to why data transfer & object loading seem to take so long?
[04:54] <wgrant> Oh
[04:54] <wgrant> It's grabbing context etc.
[04:55] <jtv> If the query includes POTemplate or POFile, those have text fields containing the headers.
[04:55] <wgrant> So lots of reasonably sized strings, I suppose
[04:55] <jtv> Unreasonably sized header strings.
[04:55] <wgrant> And pomsgids
[04:55] <jtv> Unicode conversion, awkward data sizes.
[04:55] <wgrant> No POTemplates or POFiles here
[04:55] <jtv> Oh that's a relief.
[04:55] <wgrant> Just POTMsgSets and POMsgIDs
[04:55] <wgrant> But still not small, especially for that template
[04:55] <jtv> pomsgids are short and de-duped.
[04:55] <wgrant> Descriptions are fat
[04:55] <jtv> Ah true
[04:56] <jtv> But they should only come in at the end, basically.
[04:56] <wgrant> This template is about the worst possible case for everything, except that it has no plurals.
[04:56] <jtv> Quite.
[04:56] <wgrant> Anyway, thanks for your help
[04:56] <wgrant> :)
[04:56] <jtv> No worries.
[05:25] <wgrant> StevenK: https://code.launchpad.net/~wgrant/launchpad/bug-1158854/+merge/155892
[05:33] <StevenK> wgrant: r=me
[05:33] <wgrant> Thanks
[05:33] <StevenK> I'd like Stormification, but not now
[05:34] <wgrant> Translations is difficult to Stormify due to the dynamic column names
[06:45] <nigelb> Oh hey, jtv is still around! ;)
[06:46] <jtv> Yup.  Hi nigelb!
[06:46] <jtv> How's life?
[06:46] <nigelb> Pretty good! Surving. How about you?
[06:46] <nigelb> *Surviving
[06:47] <nigelb> Are you still out East?
[06:48] <jtv> Yup.
[06:48] <jtv> Hot  here.
[06:48] <nigelb> Here too. 35C is *not* pleasant.
[06:59] <StevenK> It hit 32 in Sydney today
[07:00] <nigelb> I'm on the top floor of this building. Sometimes I end up sitting on the floor because it's slightly less warmer.
[07:05] <jtv> I've got a fan blowing at me right now.
[07:06] <danilos> wgrant, yeah, they seem to be describing the same thing, though 869264 is a bit more specific
[07:07] <wgrant> danilos: Yeah, I was going to dupe them in that direction
[07:07] <wgrant> Since 869264 is clearer
[07:08] <wgrant> Thanks
[08:31] <czajkowski> wgrant: StevenK translation bug all fixed ?
[08:32] <wgrant> czajkowski: Not on production yet, but the missing string thing is fixed in trunk
[08:32] <wgrant> Will be deployed Tue or Wed
[08:33] <wgrant> That's the two bugs from the 23rd
[08:34] <czajkowski> excellent
[08:34] <czajkowski> thanks folks