[00:02] <huwshimi> I had an ec2 test failure on Friday, but didn't receive an email with the details. Any ideas why not?
[01:02] <wallyworld> huwshimi: sometimes that happens if the attachment is too big (too many errors). a while back, there was also a config change for bzr. is this your first landing in a while?
[01:02] <huwshimi> wallyworld: Yeah, in ages
[01:03] <wallyworld> huwshimi: so let me see if i can recall the config change
[01:05] <huwshimi> wallyworld: Thank mate!
[01:06] <wallyworld> huwshimi: back in march there was an issue where the email settings in your .bazaar config file needed to be under a section header, but i believe ec2 land was fixed
[01:08] <huwshimi> wallyworld: Hmmm
[01:08] <huwshimi> wallyworld: So my current config should still work then?
[01:08] <wallyworld> i believe so
[01:09] <wallyworld> mine has the smtp server settings at the top, not under a section header
[01:09] <huwshimi> wallyworld: Mine are under [DEFAULT]
[01:10] <wallyworld> my email address and lp user name is under [DEFAULT] but the smtp settings are above that
[01:10] <wallyworld> but i think either way works
[01:11] <wallyworld> so i'm not sure what else to suggest of the top of my head
[01:12] <huwshimi> wallyworld: All good, thanks anyway. I'm going to resubmit it and see if it was a once off
[01:13] <wallyworld> huwshimi: there's options you can pass to keep the session open after testing has finished
[01:13] <wallyworld> so you can ssh in and look around
[01:13] <huwshimi> wallyworld: Oh, I've already submitted this...
[01:14] <huwshimi> (22 minutes ago)
[01:14] <wallyworld> see how it goes then. you can ssh in or point your browser at the logged web address while the tests run
[01:14] <wallyworld> or you can run ec2 ls every so ofter
[01:15] <wallyworld> it will tell youif any tests have failed
[01:18] <huwshimi> wallyworld: Yeah, that's what I've been doing :)
[01:18] <huwshimi> wallyworld: Thanks mate, I'll see how I go
[01:19] <wallyworld> good luck!
[01:26] <cinerama> hey guys, i have a launchpadlib question
[01:27] <cinerama> is there a way i can get the openid identifier for a user?
[01:28] <wgrant> cinerama: No -- that doesn't make sense.
[01:28] <wgrant> cinerama: It makes sense to go from an OpenID identifier to a user
[01:28] <wgrant> But not vice-versa
[01:29] <cinerama> wgrant: for my case, it does :) but i can understand why it might not more generally
[01:29] <wgrant> What is your case?
[01:29] <wgrant> It's probably not actually what you think it is :)
[01:29] <cinerama> wgrant: we need to provide openid urls to HR
[01:29] <cinerama> wgrant: when the new users sign up
[01:30] <wgrant> Ah, $HIDEOUS_HR_TOOL, my old nemesis.
[01:30] <cinerama> i have a bit of shell i can use but i was hoping to do it in a tidier way. maybe that's still the go though
[01:31] <wgrant> Are you using the undocumented illegal interface on https://launchpad/~username?
[01:31] <wgrant> People can have more than one OpenID identifier, so that's not reliable.
[01:31] <cinerama> mm
[01:32] <cinerama> it might have to be good enough for the particular application to assume 1
[01:33] <wgrant> Lots of applications have thought that, and then they send users after us when it doesn't work.
[01:33] <wgrant> Despite the fact that we told them not to do that.
[01:40] <cinerama> i know it's awful, but given the fairly limited application of what i'm doing, i think it will be OK
[01:40] <wgrant> Potentially. Just don't send them after us when it breaks :)
[01:57] <lifeless> cinerama: what does HR want with open id urls ?
[02:02] <cinerama> lifeless: it's for one of the HR systems
[02:02] <cinerama> lifeless: doing a bit of work on gathering information for new hires
[02:15] <lifeless> sure
[02:15] <lifeless> but, to what purpose - I mean, can we define something to cross check to identify *which* openid they need?
[02:18] <wgrant> I suspect it actually has nothing to do with Launchpad.
[02:19] <wgrant> It's just that SSO doesn't provide a way for a user to see their identity.
[02:19] <wgrant> So they key off LP username instead for setup.
[02:19] <wgrant> So they ask for LP username when they should really ask for SSO ID
[02:29] <StevenK> wallyworld: Can haz QA for r15567?
[02:30] <wallyworld> StevenK: i did that first thing this morning, i must have messed up settign the tag
[02:30] <wallyworld> fixed now
[02:31] <StevenK> Except the QA tagger won't update for ten minutes
[02:31] <StevenK> Which gives me that long to figure how the heck I'm going to QA my IBranch.visibleByUser() change
[02:32] <wgrant> StevenK: We need to QA danilo's stuff too.
[02:32] <wgrant> I don't want to deploy the visibleByUser stuff without my two testfixes.
[02:33] <StevenK> That's a point
[02:33] <StevenK> At least r15577, 15576 can be ignored on prod
[02:34] <wgrant> Hm?
[02:34] <wgrant> <15577 is possibly a memory leak on prod
[02:34] <wgrant> It was Sunday so I didn't dig into it deeply.
[02:35] <StevenK> wgrant: That still doesn't answer my concern about how ot QA it :-)
[02:37] <wgrant> StevenK: Check that things don't time out too much, and that branches have approximately correct access rules.
[02:38] <wgrant> Nothing's going to be too broken, since Code's tests are pretty good.
[02:38] <StevenK> I've hit a few branch pages on qas already
[02:39] <wgrant> Listings etc. too
[02:41] <StevenK> https://code.qastaging.launchpad.net/launchpad seems pretty snappy, but almost all of the branches are public
[02:42] <wgrant> Look, a bug!
[02:42] <StevenK> :-(
[02:42] <wgrant> StevenK: Open up the page's query log
[02:42] <wgrant> Search for branchsubscription
[02:42] <wgrant> wtf a bit
[02:43] <StevenK> Bah
[02:43] <wgrant> Also, the page is hardly snappy, but the core query is like 30x faster than prod, which often times out.
[02:43] <wgrant> So it's qa-not-quite-awesome-but-close-enough-for-now
[02:44] <wgrant> Even better if you fix those queries or convince me to
[02:44] <StevenK> My plan for this afternoon was going to staple auditor to packageupload
[02:45] <wgrant> I shall fix them :)
[02:45] <wgrant> Need a full rewrite, anyway
[02:45] <wgrant> I'm not actually sure where they are...
[02:46] <StevenK> wgrant: You can link to bug 933938, I've slammed it back to In Progress
[02:46] <_mup_> Bug #933938: Migrate branch visibility rules to project sharing <disclosure> <qa-ok> <sharing> <tech-debt> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/933938 >
[02:46] <wgrant> Thanks.
[03:02] <wgrant> Hmmm
[03:02] <wgrant> BMP needs product denormed.
[03:02] <StevenK> It does?
[03:04] <wgrant> StevenK: Even without access checks the pending MP query is 220ms for launchpad.
[03:04] <wgrant> Because it has to do a nested loop
[03:04] <wgrant> To find the MPs for each branch.
[03:04] <wgrant> (it could potentially do a hash join, but that wouldn't be significantly better)
[03:09] <StevenK> wgrant: So, looks like r15572 is bad, it causes an OOPS.
[03:09] <wgrant> StevenK: Link?
[03:10] <StevenK> https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-12.04-beta-1 and https://oops.canonical.com/oops/?oopsid=OOPS-7ec83b58aab8ad14b1b194bf93116da6
[03:11] <wgrant> Oh
[03:11] <wgrant> I didn't think to look in the template
[03:12] <StevenK> Guess deploying today was too much to hope for. :-P
[03:12] <wgrant> StevenK: Please revert that rev.
[03:51] <StevenK> And sinzui is *still* landing stuff
[03:52] <StevenK> wgrant: r15572 has been rollbacked in r15583.
[04:56] <StevenK> wgrant: What does ComponentLookupError: (<InterfaceClass lp.services.webapp.interfaces.IPlacelessAuthUtility>, '') mean?
[05:45] <wgrant> StevenK: You're in the wrong layer.
[05:45] <wgrant> StevenK: DatabaseLayer rather than DatabaseFunctionalLayer, most probably.
[05:45] <wgrant> Or UnitTests rather than DFL
[05:47] <StevenK> wgrant: Yeah, AuditorLayer inherited from BaseLayer. I moved it up the stack.
[05:53] <StevenK> Sigh, I might need a commit() or to move it further up than LFL
[05:53] <wgrant> StevenK: Why?
[05:53] <wgrant> Why's it interacting with the auth system at all?
[05:55] <StevenK> wgrant: TestCaseWithFactory
[05:55] <wgrant> Ah
[05:55] <wgrant> StevenK: Er
[05:55] <wgrant> StevenK: What are you trying to do?
[05:55] <wgrant> A test of auditorlayer probably shouldn't be using the factory.
[05:56] <wgrant> And a test of something using the auditor shouldn't be in AuditorLayer.
[05:56] <wgrant> It should probably be in LaunchpadFunctionalLaye
[05:56] <wgrant> r
[05:56] <wgrant> With the minor issue that this screws everything up.
[05:56] <wgrant> Because a lot of actions are going to want to talk to the auditor.
[05:56] <wgrant> So this will make just about every test require LaunchpadFunctionalLayer.
[05:56] <wgrant> Which is slow and stupid.
[05:57] <lifeless> why does auditor laye rneed lp functional layer ?
[05:57] <wgrant> We don't run other services in DatabaseFunctionalLayer
[05:57] <wgrant> So anything needing the auditor probably has to use LaunchpadFunctionalLayer
[05:58] <lifeless> not what I meant
[05:58] <lifeless> the layer itself should inherit baselayer only, surely?
[05:59] <wgrant> Right
[05:59] <wgrant> And that's fine.
[05:59] <wgrant> Except StevenK apparently needs the factory.
[05:59] <wgrant> So the test probably shouldn't be in AuditorLayer
[06:13] <StevenK> wgrant: I'm trying to staple IPackageUpload.acceptFromQueue() to send a POST to a configured auditor instance if a feature flag is set.
[06:15] <wgrant> StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer
[06:15] <wgrant> Which probably means LaunchpadFunctionalLayer
[06:15] <StevenK> So I could certainly decouple this, and have a function that only requires auditor and test it with AuditorLayer, and then have a PU test that mocks out that function call
[06:16] <wgrant> We don't have an existing pattern for this.
[06:16] <wgrant> We probably should.
[06:17] <wgrant> stub: Hi
[06:17] <StevenK> Well, my thought is if everything is going to be throwing data at the auditor (and retrieving it, but meh, details), we should make it easy
[06:18] <stub> yo
[06:18] <wgrant> StevenK: Certainly
[06:18] <StevenK> wgrant: I can hold off if you, lifeless and I want to talk about how to best to do that.
[06:18] <wgrant> stub: I have three index patches up for review, if you have time today.
[06:19] <StevenK> I've spent a few hours on this, but I can toss it away for a better idea when we have a plan
[06:19] <lifeless> StevenK: nothing sounds alarming about what you are describing
[06:20] <StevenK> lifeless: Which bit? A function call in acceptFromQueue that is mocked out?
[06:20] <lifeless> StevenK: 18:13 < StevenK> wgrant: I'm trying to staple IPackageUpload.acceptFromQueue() to send a POST to a configured auditor instance if a feature flag is set.
[06:20] <lifeless> 18:15 < wgrant> StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer
[06:20] <lifeless> 18:15 < wgrant> Which probably means LaunchpadFunctionalLayer
[06:20] <lifeless> StevenK: I don't see much point in mocking internally as well
[06:21] <wgrant> lifeless: This means basically every Launchpad test will start up the librarian etc.
[06:21] <wgrant> Which is insane.
[06:21] <wgrant> Because that's slow and pointless.
[06:21] <lifeless> StevenK: we've made auditor's test environment lightweight precisely to avoid spurious mocking
[06:21] <lifeless> wgrant: why does it mean that ?
[06:21] <wgrant> Because LaunchpadFunctionalLayer means that
[06:21] <lifeless> and what means LFL here ?
[06:21] <wgrant> 16:20:45 < lifeless> 18:15 < wgrant> StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer
[06:22] <wgrant> 16:20:48 < lifeless> 18:15 < wgrant> Which probably means LaunchpadFunctionalLayer
[06:22] <lifeless> wgrant: for the test that StevenK is writing.
[06:22] <lifeless> wgrant: which is >< 'basically every Launchpad teset'.
[06:22] <lifeless> wgrant: you'll need to unpack what you see.
[06:22] <StevenK> wgrant is predicting the future
[06:22] <wgrant> lifeless: So, I have these methods
[06:22] <wgrant> On model objects
[06:22] <stub> I can never find my reviews-to-do page
[06:22] <wgrant> They talk to the auditor service
[06:22] <wgrant> The auditor service is not there, because we don't mock it out :(
[06:22] <wgrant> I die.
[06:23] <wgrant> stub: http://code.launchpad.net/~stub/launchpad/+activereviews
[06:23] <stub> ta
[06:23] <wgrant> (you could be forgiven for not finding it; there is no link)
[06:23] <lifeless> wgrant: same same for Librarian etc.
[06:24] <wgrant> lifeless: Except many fewer things talk to the librarian implicitly.
[06:25] <wgrant> lifeless: Basically everything will talk to auditor.
[06:25] <wgrant> Implicitly.
[06:25] <StevenK> lifeless: Yes, but you don't need the librarian for say, merging a person, deleting a team, copying a package ...
[06:25] <stub> wgrant: Didn't we already create a trigram index somewhere, and install the extension?
[06:25] <lifeless> ok, so
[06:25] <lifeless> this relates back to using storm objects vs POP model objects
[06:25] <lifeless> etc
[06:25] <lifeless> etc
[06:25] <wgrant> stub: That's this branch that we discussed but never got reviewed.
[06:26] <stub> ok
[06:26] <lifeless> I've -no- objection to tested fakes
[06:26] <wgrant> lifeless: Sure, but it's a big regression from what we have now.
[06:26] <lifeless> if you want to write one.
[06:26] <wgrant> This fake can indeed be so fake that it does nothing at all, I suspect.
[06:26] <lifeless> wgrant: define big; how many ms to start up the auditor test fixture, how many ms to send the reset POST to it, how many ms to audit an item
[06:26] <wgrant> lifeless: +11s to start the librarian and 4s for rabbitmq
[06:27] <wgrant> Since we don't have fixtures.
[06:27] <lifeless> however, if you write one, please make it a tested fake. Do not use a mock
[06:28] <lifeless> why is the librarian and rabbitmq +times here ?
[06:28] <lifeless> - why are they not already being used ?
[06:28] <lifeless> - if they aren't being used, why are they being dragged in?
[06:29] <wgrant> lifeless: We don't have fixtures, and we probably don't want len(resources)! test layers.
[06:30] <wgrant> librarian isn't used by much code, fortunately, so it is rarely required unless you're molesting Soyuz or using makeBugAttachment.
[06:30] <lifeless> I'm thoroughly lost.
[06:30] <lifeless> This seems simple to me.
[06:31] <lifeless> Add the layer needed with the minimal closure of dependencies, per the existing pattern.
[06:31] <lifeless> Which has problems yes, but works.
[06:31] <wgrant> It's also an excellent path to a 12-hour test suite :/
[06:31] <lifeless> You seem to be saying we should do something different, which we could, but it has the problems you articulate of bringing up unneeded daemons.
[06:31] <lifeless> doing what we currently do of making a tailored combination of layers, is much less of a latency problem for tests.
[06:32] <lifeless> -> dinner.
[06:32] <wgrant> stub: btw, did you see that qastaging and staging were failing to connect to their respective DBs for >24h over the weekend? staging should have been down, but qastaging should not.
[06:32] <wgrant> lifeless: Well
[06:32] <wgrant> lifeless: We're going to have one layer for services that are expensive.
[06:32] <stub> wgrant: yeah, that was me
[06:32] <wgrant> And another layer for things like auditor that aren't expensive yet but will probably be in 6 months...
[06:32] <wgrant> stub: Ah, good.
[06:32] <wgrant> stub: I was a bit worried, since code+logs said it should not have been down.
[06:32] <stub> wgrant: THe staging rebuild scripts work fine now apart from controlling pgbouncer, I need to fit that in somewhere
[06:33] <wgrant> stub: Yeah. I guess we need to divorce them.
[06:33] <stub> yeah
[06:34] <stub> alas standard debian startup scripts don't support that, so need to create those and get them blessed
[06:36] <wgrant> Yeah
[06:42] <stub> wgrant: I'm thinking we might want a test helper that confirms a query uses an index.
[06:43] <stub> but there might be false positives since we would need to disable sequential scans.
[06:59] <wgrant> stub: The lack of stats makes them pretty difficult to do, even with seqscans disabled.
[07:00] <stub> hmm...
[07:00] <stub> so maybe a smoke test to run on staging
[07:01] <wgrant> Possibly, yeah.
[07:02] <wgrant> It'd be great to have *something* :/
[07:06] <wgrant> stub: Yeah, the idx2s will get renamed in the patch that deletes the old ones.
[07:09] <lifeless> wgrant: so your conclusion is, that StevenK should add just the services he needs to this test, not all of launchpadfunctionaltestlayer
[07:11] <wgrant> lifeless: Well
[07:11] <wgrant> lifeless: That solution is neither practical nor ideal.
[07:11] <wgrant> So I would discourage it.
[07:14] <StevenK> I've decided to write an AuditorClient so we can abstract out all of the rubbish from say, packageupload.
[07:14] <StevenK> Otherwise wgrant *will* murder me.
[07:14] <wgrant> Consider it done.
[07:14] <wgrant> stub: So, feel like trying to apply the trigram stuff live?
[07:14] <wgrant> Oh, bah, backups.
[07:15] <stub> ya, dem backups
[07:16] <wgrant> stub: We almost need to move the backup, actually
[07:16] <stub> Might more to hot backups one day but they are harder to work with
[07:16] <wgrant> stub: It's been finishing dangerously close to fdt lately.
[07:16] <wgrant> nightly.sh completed Sun Jul 8 09:48:03 UTC 2012
[07:16] <wgrant> nightly.sh completed Sat Jul 7 09:56:39 UTC 2012
[07:17] <stub> webops can reschedule it. Sounds like moving it forward another few hours would be good.
[07:17] <stub> unless we have wrapped around :P
[07:17] <StevenK> Moving it forward would completely block FDT
[07:17] <wgrant> StevenK: Forward != backward
[07:17] <stub> forward as in earlier, the other forward
[07:18] <StevenK> Time iz hard, let's go shopping.
[07:28] <lifeless> wgrant: ideal involves fixtures, more or less; practical it is though.
[07:29] <wgrant> lifeless: It's not particularly practical, because we don't currently have a way to specify a subset of services without drawing arbitrary lines or introducing len(services)! layers
[07:32] <lifeless> there is little consequence of adding addtional layers.
[07:32] <lifeless> and we don't need len(services)! layers, thats the pathological case which we are not at.
[07:33] <wgrant> We'll see.
[07:33] <lifeless> wgrant: I appreciate you feel strongly about this, but the data I see don't support the conclusion you're pushing.
[07:33] <wgrant> lifeless: I want to avoid repeating the problem we have now where you can have this nice fast test.
[07:33] <wgrant> Then you realise you need a bugattachment
[07:33] <wgrant> And now your test takes 20s to start.
[07:34] <lifeless> I understand. The general solution is to finish the separation of store and logic layers
[07:34] <lifeless> bring in tested doubles across the board, have no ORM involved in logic tests, only in store and fetch tests.
[07:35] <lifeless> and treat the orm and other services homogeneously
[07:35] <wgrant> Right.
[07:35] <wgrant> Although s/finish/start/
[07:35] <lifeless> ian has started it :)
[07:35] <lifeless> though I'm still waiting on the report IIRC.
[07:35] <wgrant> In other news, I think I might try to deploy my BugSummary patch tonight.
[07:36] <lifeless> my point though, is that holding new work hostage to the general solution just slows down the new work, which is only incrementally worse - and locally at that, not globally - and doesn't improve the overall situation at all.
[07:36] <wgrant> lifeless: It will soon be pretty much global.
[07:36] <lifeless> when in fact, the new work is*part* of the general solution, though its quite circuitous to get there solely through new work.
[07:37] <lifeless> wgrant: how long does auditorfixture take to come up for you?
[07:37] <wgrant> I've not dared to try.
[07:37] <wgrant> But I imagine it's at least a few hundred milliseconds
[07:37] <lifeless> so, get data!
[07:44] <adeuring> good morning
[08:51] <jam> jml: did you get a chance to look into the httplib thing? it is weird, I'm seeing it trying to go to an "http://" URL (api.launchpad.dev/devel) trying to get the WADL. I wonder if setting disable_ssl_certificate_validation confuses things if the URL *isn't* http
[09:11] <jam> ah, ok, we get a redirect to the HTTPS side
[09:25] <jam> jml: ugh.... just ugh. wsgi_intercept is screwing us
[09:25] <jam> it is overriding httplib2.HTTPSConnectionWithTimeout the 'variable' with class wsgi_intercept.httplib2_intercept.HTTPS_WSGIInterceptorWithTimeout
[09:26] <jam> and then this check in httplib2: issublcass(connection_type, HTTPSConnectionWithTimeout) fails
[09:26] <jam> because we got the connection_type from SCHEME_TO_CONNECTION, which wsgilib did *not* override
[09:26]  * jam goes in the corner and cries...
[09:33] <jam> and with upgrading wsgi_intercept to 0.5.1, the tests now pass
[09:34] <jam> finally
[09:34] <jml> jam: no, sorry.
[09:35] <jml> jam: I'm glad they're passing now. I'm sorry the world is so horrible.
[10:15] <jam> jml: just so many layers of indirection
[10:15] <jam> and not expecting monkey patching changing the class definitions
[10:15] <jam> pdb + stepping helps a lot
[10:15] <jml> jam: there's a lot of action-at-a-distance in Launchpad's test suite.
[10:17] <jam> jml: yeah, and it is also encouraged by the libs. lazr.restful uses launchpadlib which wraps httplib2 which actually uses httplib, but wsgi_intercept comes in the middle, while ...
[10:26] <psychognite> sir i have a project to submit on ubuntu showdown
[10:27] <psychognite> i have created openpgp keys and also signed in code on coduct ...what to next
[10:27] <psychognite> sir please help me out .....
[10:27] <psychognite> !!
[10:28] <jml> psychognite: next up you need to create a PPA, I think.
[10:29] <jml> psychognite: oh right, you're on #ubuntu-app-devel, that's a much better place to get help.
[10:29] <psychognite> sir i have created ppa then how to upload files their
[10:55] <rick_h_> morning
[11:46] <jam> cjwatson: any chance to get an update about the maverick disk-space-cleanup? Even just point me in the right direction to run the script myself. I just didn't understand what files you were using to compare against
[11:47] <jam> cjwatson: also, I'd like to get a feel for what we could do in launchpad to make it easier for everyone to do this in the future
[11:57] <cjwatson> jam: I'll try to do it today.  I'm comparing against all the Packages files that are actually published in the archive - you know, real data ;-)
[11:57] <jam> cjwatson: sure, but what is the actual files? I think you mentioned not downloading the stuff from old-releases.ubuntu.com
[11:57] <cjwatson> jam: I think this should always require Ubuntu signoff, though - I don't think it's OK for Launchpad staff to delete Ubuntu files out of the librarian without asking us first
[11:57] <jam> cjwatson: basically, I'd like to at least file bugs that could be approached in the future to make things easier
[11:57] <cjwatson> jam: Packages files are well-known files published in the Ubuntu archive
[11:58] <cjwatson> You could get them from the dists tree on old-releases if you liked - I just used cocoplum (ftpmaster) since that's quicker for me
[11:58] <jam> cjwatson: you just grab the individual package listing for each arch?
[11:58] <jam> and cat them together?
[11:58] <cjwatson> It was a bit more than that.  I'll keep notes and mail you
[11:59] <jam> cjwatson: thanks, along with the script you run would be good to get it all together.
[12:00] <cjwatson> But this is a sanity check that ought to be performed on the Ubuntu side by somebody familiar with the archive layout
[12:00] <jam> I'm happy to have Ubuntu in the loop
[12:00] <cjwatson> I don't want to over-mechanise it since the point is to be a sanity check against database queries
[12:01] <cjwatson> Totally scripting it so that it can be run with no understanding of the archive layout defeats the purpose slightly, I think :)
[12:09] <adeuring> rick_h_: could you please review this MP: https://code.launchpad.net/~adeuring/lazr.jobrunner/bug1015667-2/+merge/113962 ?
[12:29] <adeuring> rick_h_: ping
[13:30] <jam> abentley: /wave . I'm not sure if you're the person to ask, but 'sendbranchmail' seems to be stuck running for 3.25hrs. Is it safe to just kill it?
[13:30] <abentley> jam: otp
[13:44] <abentley> jam: I think it's safe.  It runs a series of jobs, so at worst one job fails.
[14:08] <jam> dpm: just a quick check about where translations are at, are we still opening it up?
[14:43] <jam> abentley: so it looks like it took about 3.5hrs and 1.5GB of memory to process a mysql branch, but did, eventually, succeed. Is there any way to get visibility into what happened?
[14:44] <jam> 012-07-09 13:31:39 INFO    Ran 1 RevisionMailJobs.
[14:44] <jam> 012-07-09 13:31:39 DEBUG   sendbranchmail ran in 13111.023446s (excl. load & lock)
[14:45] <jam> after that finally finished, the other jobs look like "ran 60 jobs in 180s" etc.
[14:45] <abentley> jam: I guess you could look into the mail server logs to see whether it was actually sending lots of email.  That's all I can think of.
[14:45] <jam> abentley: well, ps at the time basically said it was using 100% CPU for those 3 hours
[14:45] <jam> we caught it just before it finished
[14:46] <jam> https://pastebin.canonical.com/69662/ (215min of CPU usage, if I'm reading that correctyl)
[14:47] <abentley> jam: It may have been walking the revision tree.  Let me refresh my memory.
[14:50] <jam> abentley: it does look like the recent commits could have been big: https://code.launchpad.net/~mysql/mysql-server/5.5
[14:50] <jam> a few 'empty weave merge of 5.1-security => 5.5'
[14:54] <abentley> jam: So I don't see a smoking gun.  If this was a RevisionAddedJob, it could be down to find_unique_ancestors, but if it's a RevisionMailJob, I don't think it can.
[14:55] <abentley> jam: Ah, it's not distinguishing between types in the log message, so it could have been a RevisionAddedJob.
[14:56] <jam> abentley: yeah, they seem to just get appeneded
[14:57] <jam> the log file doesn't have any "Ran ? RevisionAddedJob' so I think it is just len(queue) and queue has both types
[15:03] <jam> abentley: thanks for poking into this with me. I'm at EOD now, so no need to dig much further. Though I guess making sure the process doesn't get blocked in the future would be good.
[15:19] <benji> gary_poster: I'll get it
[15:32] <abentley> adeuring: We do late imports of Celery-related things precisely *because* importing from Celery has the side effect of configuring Celery.
[15:33] <adeuring> abentley: right. But doing an import twice -- in two tests -- does not work well: The second import is ignored, if I am not completely mistaken.
[15:33] <abentley> adeuring: Yes, of course.
[15:34] <adeuring> abentley: and we also need the late imports to set up the proper config module, in tests as well as in real life.
[15:34] <abentley> adeuring: This is a bug in celery that was going to be fixed  in the next release.
[15:34] <adeuring> abentley: ok, but for now we have to live with the problem, I'm afraid...
[15:35] <abentley> adeuring: The tests were written under the assumption that there was a specific configuration used for the tests.
[15:36] <abentley> adeuring: And only the celeryd configurations would vary.
[15:36] <adeuring> abentley: ok, that's fine for most tests, but the clear-aqueues script should be tested with two configs, I think
[15:36] <adeuring> ...like celeryd
[15:40] <abentley> adeuring: is there any advantage in using the running context manager instead of using subprocess.call or something?
[15:42] <adeuring> abentley: it just saves the wait() call
[15:47] <abentley> adeuring: But communicate already waits for the process to terminate.
[15:48] <adeuring> abentley: ouch, yes, youa are right. So I'll revert running() and use Popen  directly
[15:48] <abentley> adeuring: Cool.
[15:54] <adeuring> abentley: chnages pushed
[16:01] <abentley> adeuring: r=me
[16:01] <adeuring> abentley: thanks
[16:06] <abentley> adeuring: btw, Celery 3.0 is out now.
[17:14] <rick_h_> sinzui: ping, have a few min?
[17:15] <sinzui> I do
[17:21] <jam> benji: any chance you could look at: https://code.launchpad.net/~jameinel/launchpad/py27-introduction-1020667/+merge/113938 it is mostly just updating versions.cfg to newer packages, and then updating bin/test to pass an env var so that httplib2 doesn't try to check the ssl cert during the test suite.
[17:21] <jam> (Its the last patch for python-2.7 compatibility)
[17:21] <benji> jam: sure
[17:21] <benji> cool
[17:34] <jam> benji: thanks, what is the launchpad pqm email address again? launchpad@pqm.canonical.com?
[17:35] <jam> (I ran it through ec test already, since I wanted to be safe with package updating, and I don't think I have lp-land configured correctcly)
[17:37] <benji> jam: right, launchpad@pqm.canonical.com
[17:38] <jam> thanks
[21:52] <mwhudson> is there a js version of launchpadlib?
[22:04] <wallyworld_> sinzui: mumble hates me this morning. it refuses to connect
[22:04] <sinzui> Yet I see you in mumbe?
[22:05] <wallyworld_> sinzui: the mumble window becomes non responsive and takes 100% cpu
[22:05] <sinzui> ah
[22:05] <sinzui> I had to restart when that happened. I think access to audio was broken
[22:25] <StevenK> wallyworld_: I've seen that -- it seems to happen when you quit mumble and the process does not die.
[22:26] <wallyworld_> StevenK: it wasn't running this time
[22:26] <StevenK> wallyworld_: Not even in 'ps aux | grep mumble' ?
[22:26] <mwhudson> i think mumble just decided to hate people every now and again
[22:26] <wallyworld_> StevenK: nope, nothing there
[22:27] <StevenK> I'd suggest 'strace', but you'll hate me.
[22:38] <sinzui> wallyworld_, https://bugs.launchpad.net/launchpad/+bug/206811
[22:38] <_mup_> Bug #206811: Bug feeds for BugTargets need to ensure a sufficient number of bugs are fetched <feeds> <lp-bugs> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/206811 >
[22:48] <wgrant> wallyworld_: +        return getUtility(IBugSet).getDistinctBugsForBugTasks(
[22:48] <wgrant> +            self.context.searchTasks(params), self.user, limit=5)
[23:58] <StevenK>   Running:
[23:58] <StevenK>  lp.registry.tests.test_sharingjob.RemoveArtifactSubscriptionsJobTestCase.test_admins_retain_subscriptions^C^C^C^\zsh: quit (core dumped)  bin/test -vvt test_sharingjob
[23:58]  * StevenK stabs things.
[23:59] <sinzui> Finally. I can see an icon on the bug listing that does not lie or hurt my eyes