[00:00] <bigjools> the text formatter is broken then, it linkifies bug\nNNN
[00:06] <wgrant> bigjools: It deliberately does that, to cope with wordwrapped emails
[00:06] <bigjools> see my text in bug 1081834
[00:06] <_mup_> Bug #1081834: When setting a milestone on a bug task, the edit icon ends up as the "broken graphic" <Launchpad itself:New> < https://launchpad.net/bugs/1081834 >
[00:07] <bigjools> I say tough titty if it's word wrapped
[00:07] <bigjools> coz now you have a broken link
[00:08] <wgrant> It's probably less bad to have that extra link than it is to miss some
[00:09] <bigjools> I disagree
[00:09] <wgrant> Why?
[00:09] <wgrant> You can just not click on that link :)
[00:09] <bigjools> tell that to oops reports
[00:09] <bigjools> you can just type a bug number too
[00:10] <wgrant> But I can just as easily generate a bad link malevolently by typing "bug 2" in a comment
[00:10] <wgrant> Those sorts of things should be (but aren't presently) excluded from OOPS reports
[00:10] <bigjools> why is it even linked if it doesn't exist?
[00:10] <wgrant> Because user error can cause them
[00:10] <bigjools> it's ab ug
[00:10] <bigjools> should not be linked
[00:11] <wgrant> It's turned grey by JS if it doesn't exist.
[00:11] <bigjools> a start, I suppose
[00:11] <wgrant> Users can type bad URLs
[00:11] <wgrant> They can type bad bug numbers too
[00:12] <bigjools> well it doesn't go anywhere, I thought it was a real link, so it's opk
[00:12] <wgrant> It is a real link until the JS detects that it doesn't actually exist
[00:12] <wgrant> Which isn't ideal, but I think it's a reasonable compromise
[00:12] <bigjools> so robots will still click it then
[00:12] <wgrant> Yes
[00:13] <bigjools> I am sure there was code in the formatter to not linkify non-existent bugs
[00:13] <wgrant> There was, but we removed it
[00:13] <wgrant> Because it was completely terrible
[00:13] <bigjools> let me guess ...
[00:13] <wgrant> Making a separate query for each bug reference
[00:13] <bigjools> yeah
[00:13] <wgrant> And it's difficult to do otherwise without a multi-pass template system, which TAL is not
[00:13] <bigjools> I remember it being a problem on changelogs
[00:14] <wgrant> Anyway
[00:14] <wgrant> "bug 2" is just a special type of URL
[00:14] <wgrant> Users can type bad URLs
[00:14] <bigjools> and when I wanted to remove it to fix performance, all of Ubuntu screamed like a thousand voices in the night
[00:14] <wgrant> We still linkify them
[00:14] <wgrant> Why not do the same with bugs?
[00:14] <wgrant> It's not harming anyone
[00:15] <bigjools> ok, you're on maintenance :)
[00:15] <wgrant> 404s are a discretionary sort of OOPS, since users will always be able to generate them simply by typing a bad URL somewhere on LP
[00:15]  * StevenK stabs gina.txt and twists the knife.
[00:15] <wgrant> They don't unconditionally fall under ZOP
[00:15] <bigjools> ok
[00:15] <wgrant> And they don't degrade user experience
[00:15] <wgrant> => meh
[00:16] <bigjools> well at least you thought about it
[00:17] <wgrant> The linkification of bullets isn't ideal, but there's not much we can do about it without breaking other cases
[00:17] <wgrant> Of course, if you used punctuation properly it wouldn't be a problem :P
[00:18] <bigjools> => meh
[00:18] <bigjools> DWIM etc etc
[00:18] <wgrant> Heh
[00:24] <StevenK> wgrant: Hmmm, what about bug 837563?
[00:24] <_mup_> Bug #837563: language-pack-exporter script raised a KeyError <lp-translations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/837563 >
[00:25] <StevenK> I thought OOPS prefixes were long dead
[00:26] <wgrant> StevenK: Sure, the REPORTIFSEEN part of that bug is no longer relevant
[00:26] <wgrant> The underlying OOPS is
[01:47] <bigjools> hmmm, why would ^\s not work in my sed expression
[01:48] <StevenK> \s as in 'any whitespace' ?
[01:48] <bigjools> yup
[01:48] <bigjools> ^\s should be leading whitespace
[01:49] <bigjools> yet it matcheth not
[01:49] <lifeless> \\s ?
[01:50] <bigjools> nope
[01:50] <bigjools> removing the ^ matches but I want to be more specific
[01:51] <cjwatson> [[:space:]]
[01:52] <cjwatson> \s is a perl-regex-ism
[01:52] <bigjools> it's in the sed man page :)
[01:52] <cjwatson> not in the one I have here it's not
[01:52] <cjwatson> anyway, [[:space:]] is the POSIX form
[01:53] <bigjools> ^[[:space:]] didn't work
[01:53] <cjwatson> full example please
[01:54] <bigjools> sed -ri "s|^[[:space:]]generator:\shttp://.*:5243/|  http://bar:5243/|" test.tmp
[01:54] <cjwatson> you still have a \s there
[01:55] <cjwatson> and I think perhaps you mean [[:space:]]* for zero or more whitespace characters?
[01:55] <bigjools> makes no difference if I change it
[01:55] <cjwatson> [[:space:]] (or \s in regex engines that support that) means exactly one whitespace character
[01:55] <bigjools> I want a + actually, I'll change it
[01:55] <bigjools> \o/
[01:55] <bigjools> thanks cjwatson
[01:56] <cjwatson> yw
[01:56] <bigjools> I really hate regex
[01:56] <StevenK> bigjools: You now have two problems.
[01:56] <bigjools> exactly
[01:56] <StevenK> Or three, if you're also dealing with mod_rewrite
[01:57] <bigjools> heh
[01:58] <StevenK> wgrant: Can't find OOPS-8e259d8c3b45707c6a37f986d3c8127d on carob's filesystem. :-(
[02:00] <StevenK> wgrant: I wonder if we can just destroy the branch puller
[02:04] <wgrant> mwhudson had plans :)
[02:05] <lifeless> and doesn't now ?
[02:09] <mwhudson> i just don't have time now
[02:09] <mwhudson> wgrant: actually i futzed a little on the flight to linaro connect about this and wanted to talk to someone (probably you!) about this
[02:11] <mwhudson> wgrant: my plan went something like this:
[02:11] <mwhudson> set up a launchpad user on each importd
[02:11] <mwhudson> have a fkey from codeimportmachine to this user
[02:11] <mwhudson> change _canWrite to something like this:
[02:11] <mwhudson> +        elif branch.branch_type == BranchType.IMPORTED:
[02:11] <mwhudson> +            job = branch.code_import.import_job
[02:11] <mwhudson> +            if job.machine:
[02:11] <mwhudson> +                return job.machine.user == requester
[02:11] <mwhudson> +            else:
[02:11] <mwhudson> +                return False
[02:11] <lifeless> mwhudson: so did you decide to stay w/Linaro or move back into Canonical ?
[02:13] <wgrant> mwhudson: That's probably the cheapest way to do it, but ideally we'd also solve stacking at the same time
[02:13] <mwhudson> change things around so that a lp:// url rather than a escudero url gets passed to the worker
[02:13] <mwhudson> wgrant: "solve stacking"?
[02:13] <wgrant> mwhudson: Stacking imports would sometimes be nice.
[02:13] <mwhudson> oh right
[02:14] <wgrant> Which means access to more than just that branch
[02:14] <wgrant> Which gets messy
[02:14] <mwhudson> well
[02:14] <mwhudson> imports would end up stacked on the project trunk in this case
[02:14] <mwhudson> but they would get imported from scratch first
[02:14] <mwhudson> which would not be so ideal
[02:14] <wgrant> Well
[02:15] <wgrant> They might end up using /+branch-id, which wouldn't stack
[02:15] <mwhudson> oh true
[02:15] <mwhudson> and indeed, that would be easiest
[02:15] <wgrant> And safest
[02:16] <mwhudson> so the fix is to make pushing to +branch-id stack?
[02:16] <wgrant> Maybe
[02:16] <wgrant> Or just ignore stacking
[02:16] <mwhudson> i don't think attempting to solve stacking at the same time seems especially sensible
[02:17] <wgrant> Indeed, perhaps not
[02:17] <wgrant> But it should be kept in mind as a future consideration, perhaps
[02:17] <mwhudson> once imports don't need the puller, converting mirrored to imported branches should be trivial
[02:18] <mwhudson> and then one can rm -rf lib/lp/codehosting/puller
[02:18] <mwhudson> which would get you lots of LOC credit :-)
[02:18] <wgrant> Yep
[02:18] <wgrant> That's why I haven't converted mirrors yet
[02:18] <wgrant> Because of the escudero mess
[02:18] <mwhudson> right
[02:19] <mwhudson> someone should delete all the codeimportevent mess too :(
[02:19] <mwhudson> that was mostly ddaa really but i feel bad for letting it happen
[02:19] <wgrant> What's the mess there?
[02:19] <wgrant> I admit I don't know those corners of the model well
[02:19] <mwhudson> well, the whole concept is stupid
[02:19] <mwhudson> not stupid
[02:20] <mwhudson> but should have been YAGNIed before implementation
[02:20] <wgrant> Oh
[02:20] <wgrant> That's distinct from codeimportresult... I... seee
[02:20] <mwhudson> yeah
[02:20] <lifeless> thats hardly unique in LP
[02:21] <wgrant> Ah
[02:21] <wgrant> So CodeImportEvent is an audit log, basically
[02:21] <mwhudson> yes
[02:21] <mwhudson> that you mostly can't read
[02:21] <lifeless> auditor being the now extant answer for that
[02:21] <wgrant> Well
[02:22] <wgrant> Except it isn't extant yet :)
[02:22] <lifeless> ENODEPLOY ?
[02:22] <mwhudson> and that means that standard methods are not used to mutate them, which means that e.g. field validators don't run
[02:22] <wgrant> Yeah
[02:22] <mwhudson> wgrant: https://code.launchpad.net/~mwhudson/launchpad has a few branches at the top if that's useful at all
[02:23] <mwhudson> i can't really remember how far i got, i was looking at changing the worker invocation code when i got discouraged i think
[02:24] <wgrant> Hmm, have you made a terrible mistake...
[02:24] <wgrant> I guess we'll see
[02:25] <wgrant> (pushing a series of branches which will each want to create the same new revisions, and they'll be scanned concurrently now...)
[02:25] <mwhudson> oh
[02:25] <mwhudson> heh
[02:25] <mwhudson> they certainly don't seem to be being scanned
[02:25] <wgrant> It takes a 80s at best to do a fresh scan of an LP branch nowadays
[02:26] <wgrant> Often a few minutes
[02:28] <wgrant> Yeah, they all OOPSed
[02:29] <mwhudson> sigh
[02:30] <wgrant> The branch scanner does not win any points for considering locks or using sensible transaction sizes :(
[02:34] <StevenK> lifeless: We don't have the staging instance deployed yet, let alone think about production.
[02:46] <StevenK> wgrant: http://pastebin.ubuntu.com/1376306/ but I'm not sure how to test it
[02:47] <wgrant> Um no
[02:47] <wgrant> Transaction management does not go in DB model classes :)
[02:48] <StevenK> Wrap the generateIncrementalDiff in BMPJ run() ?
[02:50] <StevenK> wgrant: http://pastebin.ubuntu.com/1376309/
[02:51] <StevenK> That I can test, if I can patch out generateIncrementalDiff()
[02:51] <wgrant> Well
[02:51] <wgrant> We in general need a strategy for handling conflicts in jobs
[02:52] <wgrant> eg. what mwhudson just ran into
[02:52] <wgrant> Ignoring all IntegrityErrors is probably not it :)
[03:18] <StevenK> wgrant: Hmmm, it looks like cronscripts/rosetta-export-queue.py already uses the slave DB and based on the logs it isn't getting reaped. bug 504821
[03:18] <_mup_> Bug #504821: poimport (export)_uses a single long transaction, gets reaped <lp-translations> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/504821 >
[03:20] <wgrant> It's still vile
[03:20] <wgrant> But indeed, I haven't seen reaping in quite some time
[03:21] <StevenK> wgrant: Downgrade or close?
[03:21] <wgrant> Invalid until we have reason to believe otherwise
[03:24] <wgrant> It's still something we need to adjust eventually, but it's not causing any trouble today
[03:24] <wgrant> And it's not actively a problem until we ratchet our paranoia up another notch
[03:26] <wgrant> StevenK: In what circumstances does one not have an id?
[03:31] <StevenK> I'm guessing when we get a error page or something like that
[03:31] <StevenK> There are a whole stack of oopses on neem
[03:31] <wgrant> Right, they certainly happen
[03:31] <wgrant> But we should work out why
[03:32] <wgrant> Blindly bypassing an error condition without understanding it is not the way to fix criticals, sadly
[03:32] <wgrant> (unless we have no other choice)
[03:34] <StevenK> I just noticed the OOPSes while looking if there was a langpack export OOPS
[03:35] <StevenK> Checking if they're all the same bugtracker
[03:35] <wgrant> Sure
[03:35] <wgrant> It's been happening for a while
[03:35] <wgrant> And it's likely to be a simple fix
[03:36] <wgrant> But we need to understand why
[03:36] <lifeless> using the slave db is now pointless
[03:36] <lifeless> as transaction bloat is propogated back to the master
[03:37] <wgrant> lifeless: Pointless for bloat
[03:37] <wgrant> Not pointless for load
[03:37] <lifeless> not terribly interesting either ;>
[03:37] <wgrant> (not that load is an issue on any system today)
[03:37] <wgrant> right
[04:00] <StevenK> Oh, hah
[04:00] <StevenK> mercurial has replaced their roundup instance with bugzilla and are redirecting
[04:00] <wgrant> Ah
[04:05] <StevenK> wgrant: Right, there's mercurial fixed, two link farm bugtrackers deleted and I yet to figure out what setuptools is doing
[04:06] <wgrant> :)
[08:30] <adeuring> good morning
[09:15] <czajkowski> aloha
[10:22] <czajkowski> cjwatson: ello how do you want this triaged, https://bugs.launchpad.net/launchpad/+bug/1081860
[10:22] <_mup_> Bug #1081860: uploads to oneiric/partner are redirected to oneiric-proposed/partner <Launchpad itself:New> < https://launchpad.net/bugs/1081860 >
[10:29] <cjwatson> czajkowski: I stole it for ubuntu-archive-tools
[10:29] <cjwatson> So you won't have to care :)
[10:29] <czajkowski> oki dokie :)
[10:29] <czajkowski> thanks colin
[17:31] <czajkowski> cjwatson: me again :) can you moderate a mail if you get a chance later on to devel
[17:32] <czajkowski> really find it hard to type devel without going devil :/
[17:33] <cjwatson> czajkowski: done
[17:34] <czajkowski> cjwatson: thank you
[18:18] <jml> is there a secret way of only retrieving certain keys when GETting Launchpad API resources?
[18:18] <jml> some code_review_comments are really, really big.
[18:20] <jml> further, the size of the download is 2x the size of the comment, because as_quoted_email is included in the dict
[18:21] <jml> also, they both get stored in memory as unicode objects, which take up 4x as many bytes as the string has characters
[18:21] <jml> bringing us up to 8x
[18:22] <jml> then, if you are getting it as part of a collection, the naive way of using it (looping through), will give you a copy of the data in the collection _and_ a copy in the entry object
[18:22] <jml> taking us up to 16x
[18:23] <jml> for this use case, I don't care about the comment body at all. All I want is the dates.
[18:35] <czajkowski> jml: most of the devs are off today as USA based
[18:42] <jml> czajkowski: so I should try to avoid having hard problems today and tomorrow?
[19:30] <czajkowski> jml: most are back tomorrow tbh, or leave a message for wgrant he'll be back online shortlyish
[19:31] <czajkowski> jml: adeuring will be back jtv bigjools, or gmb
[22:26] <wgrant> jml: Well, you can retrieve individual attributes directly, but you can't retrieve a subset of elements that isn't either the whole lot or a single one.
[22:26] <wgrant> lazr.restful unfortunately provides no more advanced facility
[22:26] <wgrant> jml: You should upgrade to Python 3.3 :)
[22:28] <abentley> jml: Are these really, really big code_review_comments full of test results?
[23:15] <james_w`> abentley, yes