=== Ursinha is now known as Ursinha-afk === jamesh_ is now known as jamesh [04:59] mwhudson: ping; how do I tell if a given loggerhead rev is is production? [05:01] thumper: https://bugs.edge.launchpad.net/loggerhead/+bug/518689 - does that need to be private? Seems like no to me. Web crawlers and users can hit that easily enough [05:01] lifeless: production only lags lp:~launchpad-pqm/loggerhead/devel by at most 24 hours [05:01] and if someone were to dos it we can filter [05:01] mwhudson: its auto rolled out ? [05:01] lifeless: if you need more precision than that, ask someone who can log into guava i guess (that includes me) [05:01] mwhudson: oh, I see, I need to do missing between the two [05:01] lifeless: yes [05:02] mwhudson: is lp-pqm's branch a series ? [05:02] lifeless: no, maybe it should be though [05:02] lifeless: Logging as a sec vuln as this is a rather easy way to currently DOS codebrowse. [05:02] lifeless: that was from the bug comment [05:02] yeah [05:02] I'm saying I disagree [05:02] using loggerhead is an easy way to DOS is [05:02] s/is/it [05:02] wasn't that the pygments problem? [05:02] lifeless: ok, fair enough [05:03] I'm trying to decide between [05:03] a) public or b) subscribe max [05:03] mwhudson: I think it may have been [05:03] mwhudson: yes, but root cause suggests thread pool limits are to blame too [05:03] mwhudson: do we have a max pygments limit yet? [05:04] huh [05:05] I can't reject https://code.edge.launchpad.net/~loggerhead-team/loggerhead/trunk-rich/+merge/26010 [05:05] lifeless: man, the thread pool stuff is so messed up, please fix that [05:05] which is silly, as I created it. [05:05] thumper: yes, i think so [05:05] is there a bug open about the permissions there? [05:05] lifeless: it is a known issue [05:05] shall I file a bug ? [05:05] lifeless: and somewhat releated to the proposal status [05:06] and that we don't have a withdrawn [05:06] yes, I can see the place in the code [05:06] why not delete it? [05:06] does it have any real value? [05:06] well, I wanted to be able to point ppl to it :P thats ok, deleted it [05:06] lifeless: lets subscribe max for now [05:06] what is his lp id? [05:07] mkanat i think [05:08] subscribed [05:09] thumper: already openned it based on your ok, fair enough able [05:09] *above* [05:13] mwhudson: the paste guts seem clear enough, it knows how many total threads, it should be able to have a hard cap. [05:14] lifeless: but it has this strange concept of 'hung' threads and if threads are 'hung' it spawns more [05:14] yes [05:14] the specific check is [05:15] spawn_if_under [05:15] + hung threads don't count as workers [05:15] lifeless: probably the limits are way too generous, we should kill threads more aggressively [05:15] that too, but killing threads is likely to show up locking bugs - mutex ownership issues, etc. [05:15] losa [05:16] how many requests/sec does loggerhead suffer? [05:16] the limits are set in scripts/start-loggerhead.py [05:16] in the launchpad tree [05:16] grah [05:16] lifeless: https://lpstats.canonical.com/graphs/CodebrowseHTTPResponses/ [05:16] thats backend? [05:17] man [05:17] I was going to book plane tickets First Thing today [05:17] lifeless: um, it's extracted from the apache logs on the frontend [05:17] theres no squid yet? [05:18] if there was, we'd need a separate graph. [05:18] no [05:18] well, at least if there is, noone's told me about it [05:20] lifeless, i believe the critical incident policy tells you who's meant to follow up etc [05:21] hmm, lpstats taking -forever- [05:22] mwhudson: I can't see that graph [05:22] mwhudson: can you just tell me ? [05:24] lifeless: "less than 1 req/s" would seem to be the summary [05:24] would measuring in minutes be better ? [05:25] 'd like a >1 figure, so I can do math. [05:25] lifeless: between 80 and 180 per 5 minutes [05:25] so that's, errrrrrrr, 16-36 per minute? [05:25] 1/4-1/2 per second [05:26] the lack of robots.txt pushed it to ~1 per second and everything fell over [05:26] I have a theory [05:27] actually, robots.txt hit a bad url on many branches [05:27] and that pushed it over [05:27] http://trac.pythonpaste.org/pythonpaste/ticket/416 [05:30] yeah, annotate of files with deep history would be a good one to pound if you want to take codebrowse out [05:31] lifeless: huh! [05:31] lifeless: the thread killing does work at least sometimes [05:31] so I'd rather say 'lh handled 4 times the load fine, except that we have a bug on some urls' [05:31] mwhudson: possibly with low-valued threadids, or something. [05:32] this is another thing we should do, of course: change the default view from the filelisting to be a simple listing, not annotate [05:33] mwhudson: is there a bug open on that ? [05:34] if now, can you -please- open [05:34] also, are you happy with all of johns work landing on LP ? [05:34] yes, but we should try to pre-seed the history.db for a few select large branches first [05:34] like launchpad, mysql, the kernel [05:35] I'm mainly checking we don't need a seperate 'for lp' branch for Max [05:35] lifeless: t [05:35] grr [05:35] https://bugs.edge.launchpad.net/loggerhead/+bug/568148 [05:35] Bug #568148: Default view for a file should be its content [06:31] thumper: ping === almaisan-away is now known as al-maisan [07:22] lifeless: pong (although leaving the office to eat) [07:22] thumper: hi [07:22] so [07:22] lifeless: leave a message and I'll get back to you as soon as possible :) [07:23] I had a bug asking about getting allist of broken-due-to-upgraded-trunk branches [07:23] you've closed the bug [07:23] should I file a fresh dedicated one ? [07:23] yes [07:24] ok will do [08:21] hmm [08:21] https://blueprints.edge.launchpad.net/ubuntu-arm/+spec/arm-m-validation-dashboard [08:21] (it was there a moment ago, now it's an OOPS) [08:21] OOPS-1607ED337 [08:22] can anyone help with this please? [08:22] bah, ok - the spec was renamed, sorry for causing noise [08:22] (still - the search did link to the wrong place) [08:57] Morning [10:51] Ursinha-afk: Have you considered running your bugbot as a separate LP person, so that it is obvious that its actions are automated? [11:09] Morning, all. [11:13] bigjools: AAAAAAA. https://edge.launchpad.net/ubuntu/+source/vowpal-wabbit/4.1+20100420-1/+build/1755930 is scary on several levels. [11:14] yes, we've seen it [11:15] "special" :-) [11:15] I thought we'd fixed this by stopping queue-builder :/ [11:15] We must have fixed at least 5 of this sort of bug in the last six months. [11:15] At least it will become impossible soon. [11:15] well "fixed" is a strong word here :) [11:16] For that particular one, yes. [11:54] wgrant: so [11:55] it's the result of a give-back [11:55] why the builder does that instead of failing the build is somewhat odd [11:55] is there any SQL query that can extract a small subset of a big text field in a more efficient way (preferably without having to copy the whole field form the db to the client) [11:56] bigjools: Yeah, I worked that out quite a while ago. [11:56] assuming I know the start/end offsets (in bytes/characters) [11:56] It's the 'Illegal instruction' in the log. [11:56] I forgot that we just fixed it so that it doesn't break b-m; the display is still all broken. [11:56] I need to kill that build anyway [11:56] I think that problem should also be a failed build, not a give-back [11:57] Maybe it expects that some builders might be able to build some stuff. [11:57] (like, say, armel) [11:57] But that's an utterly stupid way of arranging that. [11:57] So, yes, a quick deletion of that line will fix things. [11:57] I can't think of any practical legit uses. === mrevell is now known as mrevell-lunch === matsubara-afk is now known as matsubara === salgado-afk is now known as salgado === mrevell-lunch is now known as mrevell [13:54] sinzui, please feel free to remove the bugs test, if the same condition is tested in test_securitycontact.py [13:56] deryck, I did. I unconsciously knew it was a duplicate. It was a simple verification of the form label and that the form works. I wonder why it passed on ec2? [13:56] yeah, that seems odd that it would pass there. not sure why. === Ursinha-afk is now known as Ursinha === matsubara is now known as matsubara-lunch === salgado is now known as salgado-lunch === matsubara-lunch is now known as matsubara === beuno is now known as beuno-lunch [17:34] bigjools, hi [17:35] Ursinha: heyu [17:35] or something that might be spelled better [17:35] bigjools, :) [17:35] bigjools, I have an oops and think you can help: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1600E1649 [17:36] Ursinha: that's the same as the one you told me about a while ago [17:36] Ursinha: noodles775 is looking into it, feel free to remind him :) [17:37] hehe [17:37] hahahahaha [17:37] bigjools, he heard you [17:38] Ursinha: actually didn't you already file a bug about it? I think he commented on there [17:39] bigjools, I filed a bug that is a NotOneError [17:39] it's basically down to a previous bug where sun-java6 ended up in partner and main at the same time [17:39] just found it [17:39] bug 580181 [17:39] Bug #580181: DistributionSourcePackageRelease page still oopsing with NotOneError [17:40] bigjools, OOPS-1600E1649 is a LocationError [17:40] are they the same? [17:41] Ursinha: it's essentially the same underlying problem [17:41] if you could reference this oops on that bug it would be helpful [17:41] bigjools, surely [17:41] "Past week count: 686" [17:41] eek [17:41] bigjools, if noodles785 could fix it, it would be helpful as well :) [17:42] :) [17:42] bigjools, yesterday's count: 701 [17:42] hmm no referrer [17:42] bigjools, Ursinha: I haven't been looking into it... I just did some initial investigation, but can look into it tomorrow. [17:42] noodles785, that would be appreciated, thanks :) [17:43] noodles785: cheers === noodles785 is now known as noodles775 [17:52] bigjools, thanks :) [17:52] Ursinha: any time === salgado-lunch is now known as salgado [18:09] Righto peoples, I'm off. Ta ra. === beuno-lunch is now known as beuno === al-maisan is now known as almaisan-away [19:41] how active is the search for a Launchpad Web Engineer (http://webapps.ubuntu.com/employment/canonical_LPWE/)? [19:42] Alkini, what do you mean? [19:43] has the position just been sitting around for six months? or are a dozen people a day being considered? [19:45] Alkini, it's been opened up recently [19:45] and some people are being interviewed [19:45] not super sure what you're getting at :) [19:46] I just applied today and don't know much beyond the "you might not hear back from us for three weeks" email so I just thought I'd ask :-) [19:46] well, it takes a day or two for the CV to be passed on from HR [19:47] and then it depends on the availability of the team lead [19:47] sure, totally understandbale; I didn't mean to be impatient, just curious [19:47] I'm sure it's a position eager to be filled, so it'll probably be sooner than later [19:48] alright, cool [19:55] it's, presumably, the same story for the software engineer reporting to the launchpad code team lead? (http://webapps.ubuntu.com/employment/canonical_LP-SEC/) [19:55] well, different position, different people involved, but same HR process [19:56] right [20:00] you're on the ubuntu one team these days? [20:02] Alkini, yes I am, it's been a good 3 or 4 months now [20:02] still love Launchpad though :) [20:02] heh === matsubara is now known as matsubara-afk === salgado is now known as salgado-afk === Ursinha is now known as Ursinha-afk