| huwshimi | I had an ec2 test failure on Friday, but didn't receive an email with the details. Any ideas why not? | 00:02 |
|---|---|---|
| wallyworld | huwshimi: sometimes that happens if the attachment is too big (too many errors). a while back, there was also a config change for bzr. is this your first landing in a while? | 01:02 |
| huwshimi | wallyworld: Yeah, in ages | 01:02 |
| wallyworld | huwshimi: so let me see if i can recall the config change | 01:03 |
| huwshimi | wallyworld: Thank mate! | 01:05 |
| wallyworld | huwshimi: back in march there was an issue where the email settings in your .bazaar config file needed to be under a section header, but i believe ec2 land was fixed | 01:06 |
| huwshimi | wallyworld: Hmmm | 01:08 |
| huwshimi | wallyworld: So my current config should still work then? | 01:08 |
| wallyworld | i believe so | 01:08 |
| wallyworld | mine has the smtp server settings at the top, not under a section header | 01:09 |
| huwshimi | wallyworld: Mine are under [DEFAULT] | 01:09 |
| wallyworld | my email address and lp user name is under [DEFAULT] but the smtp settings are above that | 01:10 |
| wallyworld | but i think either way works | 01:10 |
| wallyworld | so i'm not sure what else to suggest of the top of my head | 01:11 |
| huwshimi | wallyworld: All good, thanks anyway. I'm going to resubmit it and see if it was a once off | 01:12 |
| wallyworld | huwshimi: there's options you can pass to keep the session open after testing has finished | 01:13 |
| wallyworld | so you can ssh in and look around | 01:13 |
| huwshimi | wallyworld: Oh, I've already submitted this... | 01:13 |
| huwshimi | (22 minutes ago) | 01:14 |
| wallyworld | see how it goes then. you can ssh in or point your browser at the logged web address while the tests run | 01:14 |
| wallyworld | or you can run ec2 ls every so ofter | 01:14 |
| wallyworld | it will tell youif any tests have failed | 01:15 |
| huwshimi | wallyworld: Yeah, that's what I've been doing :) | 01:18 |
| huwshimi | wallyworld: Thanks mate, I'll see how I go | 01:18 |
| wallyworld | good luck! | 01:19 |
| cinerama | hey guys, i have a launchpadlib question | 01:26 |
| cinerama | is there a way i can get the openid identifier for a user? | 01:27 |
| wgrant | cinerama: No -- that doesn't make sense. | 01:28 |
| wgrant | cinerama: It makes sense to go from an OpenID identifier to a user | 01:28 |
| wgrant | But not vice-versa | 01:28 |
| cinerama | wgrant: for my case, it does :) but i can understand why it might not more generally | 01:29 |
| wgrant | What is your case? | 01:29 |
| wgrant | It's probably not actually what you think it is :) | 01:29 |
| cinerama | wgrant: we need to provide openid urls to HR | 01:29 |
| cinerama | wgrant: when the new users sign up | 01:29 |
| wgrant | Ah, $HIDEOUS_HR_TOOL, my old nemesis. | 01:30 |
| cinerama | i have a bit of shell i can use but i was hoping to do it in a tidier way. maybe that's still the go though | 01:30 |
| wgrant | Are you using the undocumented illegal interface on https://launchpad/~username? | 01:31 |
| wgrant | People can have more than one OpenID identifier, so that's not reliable. | 01:31 |
| cinerama | mm | 01:31 |
| cinerama | it might have to be good enough for the particular application to assume 1 | 01:32 |
| wgrant | Lots of applications have thought that, and then they send users after us when it doesn't work. | 01:33 |
| wgrant | Despite the fact that we told them not to do that. | 01:33 |
| cinerama | i know it's awful, but given the fairly limited application of what i'm doing, i think it will be OK | 01:40 |
| wgrant | Potentially. Just don't send them after us when it breaks :) | 01:40 |
| lifeless | cinerama: what does HR want with open id urls ? | 01:57 |
| cinerama | lifeless: it's for one of the HR systems | 02:02 |
| cinerama | lifeless: doing a bit of work on gathering information for new hires | 02:02 |
| lifeless | sure | 02:15 |
| lifeless | but, to what purpose - I mean, can we define something to cross check to identify *which* openid they need? | 02:15 |
| wgrant | I suspect it actually has nothing to do with Launchpad. | 02:18 |
| wgrant | It's just that SSO doesn't provide a way for a user to see their identity. | 02:19 |
| wgrant | So they key off LP username instead for setup. | 02:19 |
| wgrant | So they ask for LP username when they should really ask for SSO ID | 02:19 |
| StevenK | wallyworld: Can haz QA for r15567? | 02:29 |
| wallyworld | StevenK: i did that first thing this morning, i must have messed up settign the tag | 02:30 |
| wallyworld | fixed now | 02:30 |
| StevenK | Except the QA tagger won't update for ten minutes | 02:31 |
| StevenK | Which gives me that long to figure how the heck I'm going to QA my IBranch.visibleByUser() change | 02:31 |
| wgrant | StevenK: We need to QA danilo's stuff too. | 02:32 |
| wgrant | I don't want to deploy the visibleByUser stuff without my two testfixes. | 02:32 |
| StevenK | That's a point | 02:33 |
| StevenK | At least r15577, 15576 can be ignored on prod | 02:33 |
| wgrant | Hm? | 02:34 |
| wgrant | <15577 is possibly a memory leak on prod | 02:34 |
| wgrant | It was Sunday so I didn't dig into it deeply. | 02:34 |
| StevenK | wgrant: That still doesn't answer my concern about how ot QA it :-) | 02:35 |
| wgrant | StevenK: Check that things don't time out too much, and that branches have approximately correct access rules. | 02:37 |
| wgrant | Nothing's going to be too broken, since Code's tests are pretty good. | 02:38 |
| StevenK | I've hit a few branch pages on qas already | 02:38 |
| wgrant | Listings etc. too | 02:39 |
| StevenK | https://code.qastaging.launchpad.net/launchpad seems pretty snappy, but almost all of the branches are public | 02:41 |
| wgrant | Look, a bug! | 02:42 |
| StevenK | :-( | 02:42 |
| wgrant | StevenK: Open up the page's query log | 02:42 |
| wgrant | Search for branchsubscription | 02:42 |
| wgrant | wtf a bit | 02:42 |
| StevenK | Bah | 02:43 |
| wgrant | Also, the page is hardly snappy, but the core query is like 30x faster than prod, which often times out. | 02:43 |
| wgrant | So it's qa-not-quite-awesome-but-close-enough-for-now | 02:43 |
| wgrant | Even better if you fix those queries or convince me to | 02:44 |
| StevenK | My plan for this afternoon was going to staple auditor to packageupload | 02:44 |
| wgrant | I shall fix them :) | 02:45 |
| wgrant | Need a full rewrite, anyway | 02:45 |
| wgrant | I'm not actually sure where they are... | 02:45 |
| StevenK | wgrant: You can link to bug 933938, I've slammed it back to In Progress | 02:46 |
| _mup_ | Bug #933938: Migrate branch visibility rules to project sharing <disclosure> <qa-ok> <sharing> <tech-debt> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/933938 > | 02:46 |
| wgrant | Thanks. | 02:46 |
| wgrant | Hmmm | 03:02 |
| wgrant | BMP needs product denormed. | 03:02 |
| StevenK | It does? | 03:02 |
| wgrant | StevenK: Even without access checks the pending MP query is 220ms for launchpad. | 03:04 |
| wgrant | Because it has to do a nested loop | 03:04 |
| wgrant | To find the MPs for each branch. | 03:04 |
| wgrant | (it could potentially do a hash join, but that wouldn't be significantly better) | 03:04 |
| StevenK | wgrant: So, looks like r15572 is bad, it causes an OOPS. | 03:09 |
| wgrant | StevenK: Link? | 03:09 |
| StevenK | https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-12.04-beta-1 and https://oops.canonical.com/oops/?oopsid=OOPS-7ec83b58aab8ad14b1b194bf93116da6 | 03:10 |
| wgrant | Oh | 03:11 |
| wgrant | I didn't think to look in the template | 03:11 |
| StevenK | Guess deploying today was too much to hope for. :-P | 03:12 |
| wgrant | StevenK: Please revert that rev. | 03:12 |
| StevenK | And sinzui is *still* landing stuff | 03:51 |
| StevenK | wgrant: r15572 has been rollbacked in r15583. | 03:52 |
| StevenK | wgrant: What does ComponentLookupError: (<InterfaceClass lp.services.webapp.interfaces.IPlacelessAuthUtility>, '') mean? | 04:56 |
| wgrant | StevenK: You're in the wrong layer. | 05:45 |
| wgrant | StevenK: DatabaseLayer rather than DatabaseFunctionalLayer, most probably. | 05:45 |
| wgrant | Or UnitTests rather than DFL | 05:45 |
| StevenK | wgrant: Yeah, AuditorLayer inherited from BaseLayer. I moved it up the stack. | 05:47 |
| StevenK | Sigh, I might need a commit() or to move it further up than LFL | 05:53 |
| wgrant | StevenK: Why? | 05:53 |
| wgrant | Why's it interacting with the auth system at all? | 05:53 |
| StevenK | wgrant: TestCaseWithFactory | 05:55 |
| wgrant | Ah | 05:55 |
| wgrant | StevenK: Er | 05:55 |
| wgrant | StevenK: What are you trying to do? | 05:55 |
| wgrant | A test of auditorlayer probably shouldn't be using the factory. | 05:55 |
| wgrant | And a test of something using the auditor shouldn't be in AuditorLayer. | 05:56 |
| wgrant | It should probably be in LaunchpadFunctionalLaye | 05:56 |
| wgrant | r | 05:56 |
| wgrant | With the minor issue that this screws everything up. | 05:56 |
| wgrant | Because a lot of actions are going to want to talk to the auditor. | 05:56 |
| wgrant | So this will make just about every test require LaunchpadFunctionalLayer. | 05:56 |
| wgrant | Which is slow and stupid. | 05:56 |
| lifeless | why does auditor laye rneed lp functional layer ? | 05:57 |
| wgrant | We don't run other services in DatabaseFunctionalLayer | 05:57 |
| wgrant | So anything needing the auditor probably has to use LaunchpadFunctionalLayer | 05:57 |
| lifeless | not what I meant | 05:58 |
| lifeless | the layer itself should inherit baselayer only, surely? | 05:58 |
| wgrant | Right | 05:59 |
| wgrant | And that's fine. | 05:59 |
| wgrant | Except StevenK apparently needs the factory. | 05:59 |
| wgrant | So the test probably shouldn't be in AuditorLayer | 05:59 |
| StevenK | wgrant: I'm trying to staple IPackageUpload.acceptFromQueue() to send a POST to a configured auditor instance if a feature flag is set. | 06:13 |
| wgrant | StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer | 06:15 |
| wgrant | Which probably means LaunchpadFunctionalLayer | 06:15 |
| StevenK | So I could certainly decouple this, and have a function that only requires auditor and test it with AuditorLayer, and then have a PU test that mocks out that function call | 06:15 |
| wgrant | We don't have an existing pattern for this. | 06:16 |
| wgrant | We probably should. | 06:16 |
| wgrant | stub: Hi | 06:17 |
| StevenK | Well, my thought is if everything is going to be throwing data at the auditor (and retrieving it, but meh, details), we should make it easy | 06:17 |
| stub | yo | 06:18 |
| wgrant | StevenK: Certainly | 06:18 |
| StevenK | wgrant: I can hold off if you, lifeless and I want to talk about how to best to do that. | 06:18 |
| wgrant | stub: I have three index patches up for review, if you have time today. | 06:18 |
| StevenK | I've spent a few hours on this, but I can toss it away for a better idea when we have a plan | 06:19 |
| lifeless | StevenK: nothing sounds alarming about what you are describing | 06:19 |
| StevenK | lifeless: Which bit? A function call in acceptFromQueue that is mocked out? | 06:20 |
| lifeless | StevenK: 18:13 < StevenK> wgrant: I'm trying to staple IPackageUpload.acceptFromQueue() to send a POST to a configured auditor instance if a feature flag is set. | 06:20 |
| lifeless | 18:15 < wgrant> StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer | 06:20 |
| lifeless | 18:15 < wgrant> Which probably means LaunchpadFunctionalLayer | 06:20 |
| lifeless | StevenK: I don't see much point in mocking internally as well | 06:20 |
| wgrant | lifeless: This means basically every Launchpad test will start up the librarian etc. | 06:21 |
| wgrant | Which is insane. | 06:21 |
| wgrant | Because that's slow and pointless. | 06:21 |
| lifeless | StevenK: we've made auditor's test environment lightweight precisely to avoid spurious mocking | 06:21 |
| lifeless | wgrant: why does it mean that ? | 06:21 |
| wgrant | Because LaunchpadFunctionalLayer means that | 06:21 |
| lifeless | and what means LFL here ? | 06:21 |
| wgrant | 16:20:45 < lifeless> 18:15 < wgrant> StevenK: Right, that needs DatabaseLayer, FunctionalLayer and AuditorLayer | 06:21 |
| wgrant | 16:20:48 < lifeless> 18:15 < wgrant> Which probably means LaunchpadFunctionalLayer | 06:22 |
| lifeless | wgrant: for the test that StevenK is writing. | 06:22 |
| lifeless | wgrant: which is >< 'basically every Launchpad teset'. | 06:22 |
| lifeless | wgrant: you'll need to unpack what you see. | 06:22 |
| StevenK | wgrant is predicting the future | 06:22 |
| wgrant | lifeless: So, I have these methods | 06:22 |
| wgrant | On model objects | 06:22 |
| stub | I can never find my reviews-to-do page | 06:22 |
| wgrant | They talk to the auditor service | 06:22 |
| wgrant | The auditor service is not there, because we don't mock it out :( | 06:22 |
| wgrant | I die. | 06:22 |
| wgrant | stub: http://code.launchpad.net/~stub/launchpad/+activereviews | 06:23 |
| stub | ta | 06:23 |
| wgrant | (you could be forgiven for not finding it; there is no link) | 06:23 |
| lifeless | wgrant: same same for Librarian etc. | 06:23 |
| wgrant | lifeless: Except many fewer things talk to the librarian implicitly. | 06:24 |
| wgrant | lifeless: Basically everything will talk to auditor. | 06:25 |
| wgrant | Implicitly. | 06:25 |
| StevenK | lifeless: Yes, but you don't need the librarian for say, merging a person, deleting a team, copying a package ... | 06:25 |
| stub | wgrant: Didn't we already create a trigram index somewhere, and install the extension? | 06:25 |
| lifeless | ok, so | 06:25 |
| lifeless | this relates back to using storm objects vs POP model objects | 06:25 |
| lifeless | etc | 06:25 |
| lifeless | etc | 06:25 |
| wgrant | stub: That's this branch that we discussed but never got reviewed. | 06:25 |
| stub | ok | 06:26 |
| lifeless | I've -no- objection to tested fakes | 06:26 |
| wgrant | lifeless: Sure, but it's a big regression from what we have now. | 06:26 |
| lifeless | if you want to write one. | 06:26 |
| wgrant | This fake can indeed be so fake that it does nothing at all, I suspect. | 06:26 |
| lifeless | wgrant: define big; how many ms to start up the auditor test fixture, how many ms to send the reset POST to it, how many ms to audit an item | 06:26 |
| wgrant | lifeless: +11s to start the librarian and 4s for rabbitmq | 06:26 |
| wgrant | Since we don't have fixtures. | 06:27 |
| lifeless | however, if you write one, please make it a tested fake. Do not use a mock | 06:27 |
| lifeless | why is the librarian and rabbitmq +times here ? | 06:28 |
| lifeless | - why are they not already being used ? | 06:28 |
| lifeless | - if they aren't being used, why are they being dragged in? | 06:28 |
| wgrant | lifeless: We don't have fixtures, and we probably don't want len(resources)! test layers. | 06:29 |
| wgrant | librarian isn't used by much code, fortunately, so it is rarely required unless you're molesting Soyuz or using makeBugAttachment. | 06:30 |
| lifeless | I'm thoroughly lost. | 06:30 |
| lifeless | This seems simple to me. | 06:30 |
| lifeless | Add the layer needed with the minimal closure of dependencies, per the existing pattern. | 06:31 |
| lifeless | Which has problems yes, but works. | 06:31 |
| wgrant | It's also an excellent path to a 12-hour test suite :/ | 06:31 |
| lifeless | You seem to be saying we should do something different, which we could, but it has the problems you articulate of bringing up unneeded daemons. | 06:31 |
| lifeless | doing what we currently do of making a tailored combination of layers, is much less of a latency problem for tests. | 06:31 |
| lifeless | -> dinner. | 06:32 |
| wgrant | stub: btw, did you see that qastaging and staging were failing to connect to their respective DBs for >24h over the weekend? staging should have been down, but qastaging should not. | 06:32 |
| wgrant | lifeless: Well | 06:32 |
| wgrant | lifeless: We're going to have one layer for services that are expensive. | 06:32 |
| stub | wgrant: yeah, that was me | 06:32 |
| wgrant | And another layer for things like auditor that aren't expensive yet but will probably be in 6 months... | 06:32 |
| wgrant | stub: Ah, good. | 06:32 |
| wgrant | stub: I was a bit worried, since code+logs said it should not have been down. | 06:32 |
| stub | wgrant: THe staging rebuild scripts work fine now apart from controlling pgbouncer, I need to fit that in somewhere | 06:32 |
| wgrant | stub: Yeah. I guess we need to divorce them. | 06:33 |
| stub | yeah | 06:33 |
| stub | alas standard debian startup scripts don't support that, so need to create those and get them blessed | 06:34 |
| wgrant | Yeah | 06:36 |
| stub | wgrant: I'm thinking we might want a test helper that confirms a query uses an index. | 06:42 |
| stub | but there might be false positives since we would need to disable sequential scans. | 06:43 |
| wgrant | stub: The lack of stats makes them pretty difficult to do, even with seqscans disabled. | 06:59 |
| stub | hmm... | 07:00 |
| stub | so maybe a smoke test to run on staging | 07:00 |
| wgrant | Possibly, yeah. | 07:01 |
| wgrant | It'd be great to have *something* :/ | 07:02 |
| wgrant | stub: Yeah, the idx2s will get renamed in the patch that deletes the old ones. | 07:06 |
| lifeless | wgrant: so your conclusion is, that StevenK should add just the services he needs to this test, not all of launchpadfunctionaltestlayer | 07:09 |
| wgrant | lifeless: Well | 07:11 |
| wgrant | lifeless: That solution is neither practical nor ideal. | 07:11 |
| wgrant | So I would discourage it. | 07:11 |
| StevenK | I've decided to write an AuditorClient so we can abstract out all of the rubbish from say, packageupload. | 07:14 |
| StevenK | Otherwise wgrant *will* murder me. | 07:14 |
| wgrant | Consider it done. | 07:14 |
| wgrant | stub: So, feel like trying to apply the trigram stuff live? | 07:14 |
| wgrant | Oh, bah, backups. | 07:14 |
| stub | ya, dem backups | 07:15 |
| wgrant | stub: We almost need to move the backup, actually | 07:16 |
| stub | Might more to hot backups one day but they are harder to work with | 07:16 |
| wgrant | stub: It's been finishing dangerously close to fdt lately. | 07:16 |
| wgrant | nightly.sh completed Sun Jul 8 09:48:03 UTC 2012 | 07:16 |
| wgrant | nightly.sh completed Sat Jul 7 09:56:39 UTC 2012 | 07:16 |
| stub | webops can reschedule it. Sounds like moving it forward another few hours would be good. | 07:17 |
| stub | unless we have wrapped around :P | 07:17 |
| StevenK | Moving it forward would completely block FDT | 07:17 |
| wgrant | StevenK: Forward != backward | 07:17 |
| stub | forward as in earlier, the other forward | 07:17 |
| StevenK | Time iz hard, let's go shopping. | 07:18 |
| lifeless | wgrant: ideal involves fixtures, more or less; practical it is though. | 07:28 |
| wgrant | lifeless: It's not particularly practical, because we don't currently have a way to specify a subset of services without drawing arbitrary lines or introducing len(services)! layers | 07:29 |
| lifeless | there is little consequence of adding addtional layers. | 07:32 |
| lifeless | and we don't need len(services)! layers, thats the pathological case which we are not at. | 07:32 |
| wgrant | We'll see. | 07:33 |
| lifeless | wgrant: I appreciate you feel strongly about this, but the data I see don't support the conclusion you're pushing. | 07:33 |
| wgrant | lifeless: I want to avoid repeating the problem we have now where you can have this nice fast test. | 07:33 |
| wgrant | Then you realise you need a bugattachment | 07:33 |
| wgrant | And now your test takes 20s to start. | 07:33 |
| lifeless | I understand. The general solution is to finish the separation of store and logic layers | 07:34 |
| lifeless | bring in tested doubles across the board, have no ORM involved in logic tests, only in store and fetch tests. | 07:34 |
| lifeless | and treat the orm and other services homogeneously | 07:35 |
| wgrant | Right. | 07:35 |
| wgrant | Although s/finish/start/ | 07:35 |
| lifeless | ian has started it :) | 07:35 |
| lifeless | though I'm still waiting on the report IIRC. | 07:35 |
| wgrant | In other news, I think I might try to deploy my BugSummary patch tonight. | 07:35 |
| lifeless | my point though, is that holding new work hostage to the general solution just slows down the new work, which is only incrementally worse - and locally at that, not globally - and doesn't improve the overall situation at all. | 07:36 |
| wgrant | lifeless: It will soon be pretty much global. | 07:36 |
| lifeless | when in fact, the new work is*part* of the general solution, though its quite circuitous to get there solely through new work. | 07:36 |
| lifeless | wgrant: how long does auditorfixture take to come up for you? | 07:37 |
| wgrant | I've not dared to try. | 07:37 |
| wgrant | But I imagine it's at least a few hundred milliseconds | 07:37 |
| lifeless | so, get data! | 07:37 |
| adeuring | good morning | 07:44 |
| === adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2 | ||
| jam | jml: did you get a chance to look into the httplib thing? it is weird, I'm seeing it trying to go to an "http://" URL (api.launchpad.dev/devel) trying to get the WADL. I wonder if setting disable_ssl_certificate_validation confuses things if the URL *isn't* http | 08:51 |
| jam | ah, ok, we get a redirect to the HTTPS side | 09:11 |
| jam | jml: ugh.... just ugh. wsgi_intercept is screwing us | 09:25 |
| jam | it is overriding httplib2.HTTPSConnectionWithTimeout the 'variable' with class wsgi_intercept.httplib2_intercept.HTTPS_WSGIInterceptorWithTimeout | 09:25 |
| jam | and then this check in httplib2: issublcass(connection_type, HTTPSConnectionWithTimeout) fails | 09:26 |
| jam | because we got the connection_type from SCHEME_TO_CONNECTION, which wsgilib did *not* override | 09:26 |
| * jam goes in the corner and cries... | 09:26 | |
| jam | and with upgrading wsgi_intercept to 0.5.1, the tests now pass | 09:33 |
| jam | finally | 09:34 |
| jml | jam: no, sorry. | 09:34 |
| jml | jam: I'm glad they're passing now. I'm sorry the world is so horrible. | 09:35 |
| === dpm is now known as dpm-bbl | ||
| jam | jml: just so many layers of indirection | 10:15 |
| jam | and not expecting monkey patching changing the class definitions | 10:15 |
| jam | pdb + stepping helps a lot | 10:15 |
| jml | jam: there's a lot of action-at-a-distance in Launchpad's test suite. | 10:15 |
| jam | jml: yeah, and it is also encouraged by the libs. lazr.restful uses launchpadlib which wraps httplib2 which actually uses httplib, but wsgi_intercept comes in the middle, while ... | 10:17 |
| psychognite | sir i have a project to submit on ubuntu showdown | 10:26 |
| psychognite | i have created openpgp keys and also signed in code on coduct ...what to next | 10:27 |
| psychognite | sir please help me out ..... | 10:27 |
| psychognite | !! | 10:27 |
| jml | psychognite: next up you need to create a PPA, I think. | 10:28 |
| jml | psychognite: oh right, you're on #ubuntu-app-devel, that's a much better place to get help. | 10:29 |
| psychognite | sir i have created ppa then how to upload files their | 10:29 |
| === dpm-bbl is now known as dpm | ||
| rick_h_ | morning | 10:55 |
| === matsubara is now known as matsubara-afk | ||
| jam | cjwatson: any chance to get an update about the maverick disk-space-cleanup? Even just point me in the right direction to run the script myself. I just didn't understand what files you were using to compare against | 11:46 |
| jam | cjwatson: also, I'd like to get a feel for what we could do in launchpad to make it easier for everyone to do this in the future | 11:47 |
| cjwatson | jam: I'll try to do it today. I'm comparing against all the Packages files that are actually published in the archive - you know, real data ;-) | 11:57 |
| jam | cjwatson: sure, but what is the actual files? I think you mentioned not downloading the stuff from old-releases.ubuntu.com | 11:57 |
| cjwatson | jam: I think this should always require Ubuntu signoff, though - I don't think it's OK for Launchpad staff to delete Ubuntu files out of the librarian without asking us first | 11:57 |
| jam | cjwatson: basically, I'd like to at least file bugs that could be approached in the future to make things easier | 11:57 |
| cjwatson | jam: Packages files are well-known files published in the Ubuntu archive | 11:57 |
| cjwatson | You could get them from the dists tree on old-releases if you liked - I just used cocoplum (ftpmaster) since that's quicker for me | 11:58 |
| jam | cjwatson: you just grab the individual package listing for each arch? | 11:58 |
| jam | and cat them together? | 11:58 |
| === benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: benji | Firefighting: - | Critical bugs: 4.0*10^2 | ||
| cjwatson | It was a bit more than that. I'll keep notes and mail you | 11:58 |
| jam | cjwatson: thanks, along with the script you run would be good to get it all together. | 11:59 |
| cjwatson | But this is a sanity check that ought to be performed on the Ubuntu side by somebody familiar with the archive layout | 12:00 |
| jam | I'm happy to have Ubuntu in the loop | 12:00 |
| cjwatson | I don't want to over-mechanise it since the point is to be a sanity check against database queries | 12:00 |
| cjwatson | Totally scripting it so that it can be run with no understanding of the archive layout defeats the purpose slightly, I think :) | 12:01 |
| adeuring | rick_h_: could you please review this MP: https://code.launchpad.net/~adeuring/lazr.jobrunner/bug1015667-2/+merge/113962 ? | 12:09 |
| adeuring | rick_h_: ping | 12:29 |
| jam | abentley: /wave . I'm not sure if you're the person to ask, but 'sendbranchmail' seems to be stuck running for 3.25hrs. Is it safe to just kill it? | 13:30 |
| abentley | jam: otp | 13:30 |
| abentley | jam: I think it's safe. It runs a series of jobs, so at worst one job fails. | 13:44 |
| jam | dpm: just a quick check about where translations are at, are we still opening it up? | 14:08 |
| jam | abentley: so it looks like it took about 3.5hrs and 1.5GB of memory to process a mysql branch, but did, eventually, succeed. Is there any way to get visibility into what happened? | 14:43 |
| jam | 012-07-09 13:31:39 INFO Ran 1 RevisionMailJobs. | 14:44 |
| jam | 012-07-09 13:31:39 DEBUG sendbranchmail ran in 13111.023446s (excl. load & lock) | 14:44 |
| jam | after that finally finished, the other jobs look like "ran 60 jobs in 180s" etc. | 14:45 |
| abentley | jam: I guess you could look into the mail server logs to see whether it was actually sending lots of email. That's all I can think of. | 14:45 |
| jam | abentley: well, ps at the time basically said it was using 100% CPU for those 3 hours | 14:45 |
| jam | we caught it just before it finished | 14:45 |
| jam | https://pastebin.canonical.com/69662/ (215min of CPU usage, if I'm reading that correctyl) | 14:46 |
| abentley | jam: It may have been walking the revision tree. Let me refresh my memory. | 14:47 |
| jam | abentley: it does look like the recent commits could have been big: https://code.launchpad.net/~mysql/mysql-server/5.5 | 14:50 |
| jam | a few 'empty weave merge of 5.1-security => 5.5' | 14:50 |
| abentley | jam: So I don't see a smoking gun. If this was a RevisionAddedJob, it could be down to find_unique_ancestors, but if it's a RevisionMailJob, I don't think it can. | 14:54 |
| abentley | jam: Ah, it's not distinguishing between types in the log message, so it could have been a RevisionAddedJob. | 14:55 |
| jam | abentley: yeah, they seem to just get appeneded | 14:56 |
| jam | the log file doesn't have any "Ran ? RevisionAddedJob' so I think it is just len(queue) and queue has both types | 14:57 |
| jam | abentley: thanks for poking into this with me. I'm at EOD now, so no need to dig much further. Though I guess making sure the process doesn't get blocked in the future would be good. | 15:03 |
| benji | gary_poster: I'll get it | 15:19 |
| abentley | adeuring: We do late imports of Celery-related things precisely *because* importing from Celery has the side effect of configuring Celery. | 15:32 |
| adeuring | abentley: right. But doing an import twice -- in two tests -- does not work well: The second import is ignored, if I am not completely mistaken. | 15:33 |
| abentley | adeuring: Yes, of course. | 15:33 |
| adeuring | abentley: and we also need the late imports to set up the proper config module, in tests as well as in real life. | 15:34 |
| abentley | adeuring: This is a bug in celery that was going to be fixed in the next release. | 15:34 |
| adeuring | abentley: ok, but for now we have to live with the problem, I'm afraid... | 15:34 |
| abentley | adeuring: The tests were written under the assumption that there was a specific configuration used for the tests. | 15:35 |
| abentley | adeuring: And only the celeryd configurations would vary. | 15:36 |
| adeuring | abentley: ok, that's fine for most tests, but the clear-aqueues script should be tested with two configs, I think | 15:36 |
| adeuring | ...like celeryd | 15:36 |
| abentley | adeuring: is there any advantage in using the running context manager instead of using subprocess.call or something? | 15:40 |
| adeuring | abentley: it just saves the wait() call | 15:42 |
| abentley | adeuring: But communicate already waits for the process to terminate. | 15:47 |
| adeuring | abentley: ouch, yes, youa are right. So I'll revert running() and use Popen directly | 15:48 |
| abentley | adeuring: Cool. | 15:48 |
| adeuring | abentley: chnages pushed | 15:54 |
| abentley | adeuring: r=me | 16:01 |
| adeuring | abentley: thanks | 16:01 |
| abentley | adeuring: btw, Celery 3.0 is out now. | 16:06 |
| === salgado is now known as salgado-lunch | ||
| === salgado-lunch is now known as salgado | ||
| rick_h_ | sinzui: ping, have a few min? | 17:14 |
| sinzui | I do | 17:15 |
| jam | benji: any chance you could look at: https://code.launchpad.net/~jameinel/launchpad/py27-introduction-1020667/+merge/113938 it is mostly just updating versions.cfg to newer packages, and then updating bin/test to pass an env var so that httplib2 doesn't try to check the ssl cert during the test suite. | 17:21 |
| jam | (Its the last patch for python-2.7 compatibility) | 17:21 |
| benji | jam: sure | 17:21 |
| benji | cool | 17:21 |
| jam | benji: thanks, what is the launchpad pqm email address again? launchpad@pqm.canonical.com? | 17:34 |
| jam | (I ran it through ec test already, since I wanted to be safe with package updating, and I don't think I have lp-land configured correctcly) | 17:35 |
| benji | jam: right, launchpad@pqm.canonical.com | 17:37 |
| jam | thanks | 17:38 |
| === benji changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2 | ||
| === salgado is now known as salgado-afk | ||
| mwhudson | is there a js version of launchpadlib? | 21:52 |
| wallyworld_ | sinzui: mumble hates me this morning. it refuses to connect | 22:04 |
| sinzui | Yet I see you in mumbe? | 22:04 |
| wallyworld_ | sinzui: the mumble window becomes non responsive and takes 100% cpu | 22:05 |
| sinzui | ah | 22:05 |
| sinzui | I had to restart when that happened. I think access to audio was broken | 22:05 |
| StevenK | wallyworld_: I've seen that -- it seems to happen when you quit mumble and the process does not die. | 22:25 |
| wallyworld_ | StevenK: it wasn't running this time | 22:26 |
| StevenK | wallyworld_: Not even in 'ps aux | grep mumble' ? | 22:26 |
| mwhudson | i think mumble just decided to hate people every now and again | 22:26 |
| wallyworld_ | StevenK: nope, nothing there | 22:26 |
| StevenK | I'd suggest 'strace', but you'll hate me. | 22:27 |
| sinzui | wallyworld_, https://bugs.launchpad.net/launchpad/+bug/206811 | 22:38 |
| _mup_ | Bug #206811: Bug feeds for BugTargets need to ensure a sufficient number of bugs are fetched <feeds> <lp-bugs> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/206811 > | 22:38 |
| wgrant | wallyworld_: + return getUtility(IBugSet).getDistinctBugsForBugTasks( | 22:48 |
| wgrant | + self.context.searchTasks(params), self.user, limit=5) | 22:48 |
| StevenK | Running: | 23:58 |
| StevenK | lp.registry.tests.test_sharingjob.RemoveArtifactSubscriptionsJobTestCase.test_admins_retain_subscriptions^C^C^C^\zsh: quit (core dumped) bin/test -vvt test_sharingjob | 23:58 |
| * StevenK stabs things. | 23:58 | |
| sinzui | Finally. I can see an icon on the bug listing that does not lie or hurt my eyes | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!