[04:59] <lifeless> mwhudson: ping; how do I tell if a given loggerhead rev is is production?
[05:01] <lifeless> thumper: https://bugs.edge.launchpad.net/loggerhead/+bug/518689 - does that need to be private? Seems like no to me. Web crawlers and users can hit that easily enough
[05:01] <mwhudson> lifeless: production only lags lp:~launchpad-pqm/loggerhead/devel by at most 24 hours
[05:01] <lifeless> and if someone were to dos it we can filter
[05:01] <lifeless> mwhudson: its auto rolled out ?
[05:01] <mwhudson> lifeless: if you need more precision than that, ask someone who can log into guava i guess (that includes me)
[05:01] <lifeless> mwhudson: oh, I see, I need to do missing between the two
[05:01] <mwhudson> lifeless: yes
[05:02] <lifeless> mwhudson: is lp-pqm's branch a series ?
[05:02] <mwhudson> lifeless: no, maybe it should be though
[05:02] <thumper> lifeless: Logging as a sec vuln as this is a rather easy way to currently DOS codebrowse.
[05:02] <thumper> lifeless: that was from the bug comment
[05:02] <lifeless> yeah
[05:02] <lifeless> I'm saying I disagree
[05:02] <lifeless> using loggerhead is an easy way to DOS is
[05:02] <lifeless> s/is/it
[05:02] <mwhudson> wasn't that the pygments problem?
[05:02] <thumper> lifeless: ok, fair enough
[05:03] <lifeless> I'm trying to decide between
[05:03] <lifeless> a) public or b) subscribe max
[05:03] <thumper> mwhudson: I think it may have been
[05:03] <lifeless> mwhudson: yes, but root cause suggests thread pool limits are to blame too
[05:03] <thumper> mwhudson: do we have a max pygments limit yet?
[05:04] <lifeless> huh
[05:05] <lifeless> I can't reject https://code.edge.launchpad.net/~loggerhead-team/loggerhead/trunk-rich/+merge/26010
[05:05] <mwhudson> lifeless: man, the thread pool stuff is so messed up, please fix that
[05:05] <lifeless> which is silly, as I created it.
[05:05] <mwhudson> thumper: yes, i think so
[05:05] <lifeless> is there a bug open about the permissions there?
[05:05] <thumper> lifeless: it is a known issue
[05:05] <lifeless> shall I file a bug ?
[05:05] <thumper> lifeless: and somewhat releated to the proposal status
[05:06] <thumper> and that we don't have a withdrawn
[05:06] <lifeless> yes, I can see the place in the code
[05:06] <thumper> why not delete it?
[05:06] <thumper> does it have any real value?
[05:06] <lifeless> well, I wanted to be able to point ppl to it :P thats ok, deleted it
[05:06] <thumper> lifeless: lets subscribe max for now
[05:06] <thumper> what is his lp id?
[05:07] <mwhudson> mkanat i think
[05:08] <thumper> subscribed
[05:09] <lifeless> thumper: already openned it based on your ok, fair enough able
[05:09] <lifeless> *above*
[05:13] <lifeless> mwhudson: the paste guts seem clear enough, it knows how many total threads, it should be able to have a hard cap.
[05:14] <mwhudson> lifeless: but it has this strange concept of 'hung' threads and if threads are 'hung' it spawns more
[05:14] <lifeless> yes
[05:14] <lifeless> the specific check is
[05:15] <lifeless> spawn_if_under
[05:15] <lifeless> + hung threads don't count as workers
[05:15] <mwhudson> lifeless: probably the limits are way too generous, we should kill threads more aggressively
[05:15] <lifeless> that too, but killing threads is likely to show up locking bugs - mutex ownership issues, etc.
[05:15] <lifeless> losa
[05:16] <lifeless> how many requests/sec does loggerhead suffer?
[05:16] <mwhudson> the limits are set in scripts/start-loggerhead.py
[05:16] <mwhudson> in the launchpad tree
[05:16] <lifeless> grah
[05:16] <mwhudson> lifeless: https://lpstats.canonical.com/graphs/CodebrowseHTTPResponses/
[05:16] <lifeless> thats backend?
[05:17] <lifeless> man
[05:17] <lifeless> I was going to book plane tickets First Thing today
[05:17] <mwhudson> lifeless: um, it's extracted from the apache logs on the frontend
[05:17] <lifeless> theres no squid yet?
[05:18] <lifeless> if there was, we'd need a separate graph.
[05:18] <mwhudson> no
[05:18] <mwhudson> well, at least if there is, noone's told me about it
[05:20] <poolie> lifeless, i believe the critical incident policy tells you who's meant to follow up etc
[05:21] <lifeless> hmm, lpstats taking -forever-
[05:22] <lifeless> mwhudson: I can't see that graph
[05:22] <lifeless> mwhudson: can you just tell me ?
[05:24] <mwhudson> lifeless: "less than 1 req/s" would seem to be the summary
[05:24] <lifeless> would measuring in minutes be better ?
[05:25] <lifeless> 'd like a >1 figure, so I can do math.
[05:25] <mwhudson> lifeless: between 80 and 180 per 5 minutes
[05:25] <mwhudson> so that's, errrrrrrr, 16-36 per minute?
[05:25] <lifeless> 1/4-1/2 per second
[05:26] <mwhudson> the lack of robots.txt pushed it to ~1 per second and everything fell over
[05:26] <lifeless> I have a theory
[05:27] <lifeless> actually, robots.txt hit a bad url on many branches
[05:27] <lifeless> and that pushed it over
[05:27] <lifeless> http://trac.pythonpaste.org/pythonpaste/ticket/416
[05:30] <mwhudson> yeah, annotate of files with deep history would be a good one to pound if you want to take codebrowse out
[05:31] <mwhudson> lifeless: huh!
[05:31] <mwhudson> lifeless: the thread killing does work at least sometimes
[05:31] <lifeless> so I'd rather say 'lh handled 4 times the load fine, except that we have a bug on some urls'
[05:31] <lifeless> mwhudson: possibly with low-valued threadids, or something.
[05:32] <mwhudson> this is another thing we should do, of course: change the default view from the filelisting to be a simple listing, not annotate
[05:33] <lifeless> mwhudson: is there a bug open on that ?
[05:34] <lifeless> if now, can you -please- open
[05:34] <lifeless> also, are you happy with all of johns work landing on LP ?
[05:34] <mwhudson> yes, but we should try to pre-seed the history.db for a few select large branches first
[05:34] <mwhudson> like launchpad, mysql, the kernel
[05:35] <lifeless> I'm mainly checking we don't need a seperate 'for lp' branch for Max
[05:35] <mwhudson> lifeless: t
[05:35] <mwhudson> grr
[05:35] <mwhudson> https://bugs.edge.launchpad.net/loggerhead/+bug/568148
[05:35] <mup> Bug #568148: Default view for a file should be its content <performance> <loggerhead:Confirmed> <https://launchpad.net/bugs/568148>
[06:31] <lifeless> thumper: ping
[07:22] <thumper> lifeless: pong (although leaving the office to eat)
[07:22] <lifeless> thumper: hi
[07:22] <lifeless> so
[07:22] <thumper> lifeless: leave a message and I'll get back to you as soon as possible :)
[07:23] <lifeless> I had a bug asking about getting allist of broken-due-to-upgraded-trunk branches
[07:23] <lifeless> you've closed the bug
[07:23] <lifeless> should I file a fresh dedicated one ?
[07:23] <thumper> yes
[07:24] <lifeless> ok will do
[08:21] <zyga> hmm
[08:21] <zyga> https://blueprints.edge.launchpad.net/ubuntu-arm/+spec/arm-m-validation-dashboard
[08:21] <zyga> (it was there a moment ago, now it's an OOPS)
[08:21] <zyga> OOPS-1607ED337
[08:22] <zyga> can anyone help with this please?
[08:22] <zyga> bah, ok - the spec was renamed, sorry for causing noise
[08:22] <zyga> (still - the search did link to the wrong place)
[08:57] <mrevell> Morning
[10:51] <maxb> Ursinha-afk: Have you considered running your bugbot as a separate LP person, so that it is obvious that its actions are automated?
[11:09] <deryck> Morning, all.
[11:13] <wgrant> bigjools: AAAAAAA. https://edge.launchpad.net/ubuntu/+source/vowpal-wabbit/4.1+20100420-1/+build/1755930 is scary on several levels.
[11:14] <bigjools> yes, we've seen it
[11:15] <maxb> "special" :-)
[11:15] <bigjools> I thought we'd fixed this by stopping queue-builder :/
[11:15] <wgrant> We must have fixed at least 5 of this sort of bug in the last six months.
[11:15] <wgrant> At least it will become impossible soon.
[11:15] <bigjools> well "fixed" is a strong word here :)
[11:16] <wgrant> For that particular one, yes.
[11:54] <bigjools> wgrant: so
[11:55] <bigjools> it's the result of a give-back
[11:55] <bigjools> why the builder does that instead of failing the build is somewhat odd
[11:55] <zyga> is there any SQL query that can extract a small subset of a big text field in a more efficient way (preferably without having to copy the whole field form the db to the client)
[11:56] <wgrant> bigjools: Yeah, I worked that out quite a while ago.
[11:56] <zyga> assuming I know the start/end offsets (in bytes/characters)
[11:56] <wgrant> It's the 'Illegal instruction' in the log.
[11:56] <wgrant> I forgot that we just fixed it so that it doesn't break b-m; the display is still all broken.
[11:56] <bigjools> I need to kill that build anyway
[11:56] <bigjools> I think that problem should also be a failed build, not a give-back
[11:57] <wgrant> Maybe it expects that some builders might be able to build some stuff.
[11:57] <wgrant> (like, say, armel)
[11:57] <wgrant> But that's an utterly stupid way of arranging that.
[11:57] <wgrant> So, yes, a quick deletion of that line will fix things.
[11:57] <wgrant> I can't think of any practical legit uses.
[13:54] <deryck> sinzui, please feel free to remove the bugs test, if the same condition is tested in test_securitycontact.py
[13:56] <sinzui> deryck, I did. I unconsciously knew it was a duplicate. It was a simple verification of the form label and that the form works. I wonder why it passed on ec2?
[13:56] <deryck> yeah, that seems odd that it would pass there.  not sure why.
[17:34] <Ursinha> bigjools, hi
[17:35] <bigjools> Ursinha: heyu
[17:35] <bigjools> or something that might be spelled better
[17:35] <Ursinha> bigjools, :)
[17:35] <Ursinha> bigjools, I have an oops and think you can help: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1600E1649
[17:36] <bigjools> Ursinha: that's the same as the one you told me about a while ago
[17:36] <bigjools> Ursinha: noodles775 is looking into it, feel free to remind him :)
[17:37] <Ursinha> hehe
[17:37] <Ursinha> hahahahaha
[17:37] <Ursinha> bigjools, he heard you
[17:38] <bigjools> Ursinha: actually didn't you already file a bug about it?  I think he commented on there
[17:39] <Ursinha> bigjools, I filed a bug that is a NotOneError
[17:39] <bigjools> it's basically down to a previous bug where sun-java6 ended up in partner and main at the same time
[17:39] <Ursinha> just found it
[17:39] <Ursinha> bug 580181
[17:39] <mup> Bug #580181:  DistributionSourcePackageRelease page still oopsing with NotOneError <oops> <Soyuz:Triaged> <https://launchpad.net/bugs/580181>
[17:40] <Ursinha> bigjools, OOPS-1600E1649 is a LocationError
[17:40] <Ursinha> are they the same?
[17:41] <bigjools> Ursinha: it's essentially the same underlying problem
[17:41] <bigjools> if you could reference this oops on that bug it would be helpful
[17:41] <Ursinha> bigjools, surely
[17:41] <bigjools> "Past week count: 686"
[17:41] <bigjools> eek
[17:41] <Ursinha> bigjools, if noodles785 could fix it, it would be helpful as well :)
[17:42] <bigjools> :)
[17:42] <Ursinha> bigjools, yesterday's count: 701
[17:42] <bigjools> hmm no referrer
[17:42] <noodles785> bigjools, Ursinha: I haven't been looking into it... I just did some initial investigation, but can look into it tomorrow.
[17:42] <Ursinha> noodles785, that would be appreciated, thanks :)
[17:43] <bigjools> noodles785: cheers
[17:52] <Ursinha> bigjools, thanks :)
[17:52] <bigjools> Ursinha: any time
[18:09] <mrevell> Righto peoples, I'm off. Ta ra.
[19:41] <Alkini> how active is the search for a Launchpad Web Engineer (http://webapps.ubuntu.com/employment/canonical_LPWE/)?
[19:42] <beuno> Alkini, what do you mean?
[19:43] <Alkini> has the position just been sitting around for six months? or are a dozen people a day being considered?
[19:45] <beuno> Alkini, it's been opened up recently
[19:45] <beuno> and some people are being interviewed
[19:45] <beuno> not super sure what you're getting at :)
[19:46] <Alkini> I just applied today and don't know much beyond the "you might not hear back from us for three weeks" email so I just thought I'd ask :-)
[19:46] <beuno> well, it takes a day or two for the CV to be passed on from HR
[19:47] <beuno> and then it depends on the availability of the team lead
[19:47] <Alkini> sure, totally understandbale; I didn't mean to be impatient, just curious
[19:47] <beuno> I'm sure it's a position eager to be filled, so it'll probably be sooner than later
[19:48] <Alkini> alright, cool
[19:55] <Alkini> it's, presumably, the same story for the software engineer reporting to the launchpad code team lead? (http://webapps.ubuntu.com/employment/canonical_LP-SEC/)
[19:55] <beuno> well, different position, different people involved, but same HR process
[19:56] <Alkini> right
[20:00] <Alkini> you're on the ubuntu one team these days?
[20:02] <beuno> Alkini, yes I am, it's been a good 3 or 4 months now
[20:02] <beuno> still love Launchpad though  :)
[20:02] <Alkini> heh