| lifeless | wgrant: https://bugs.edge.launchpad.net/soyuz/+bug/669717 | 00:18 |
|---|---|---|
| _mup_ | Bug #669717: archive:+index timeout <dba> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/669717> | 00:18 |
| lifeless | wgrant: just to keep you on your toes ;) | 00:18 |
| lifeless | holy f*ck: 5156 OOPS-1765XMLP299 MailingListApplication:MailingListAPIView | 00:27 |
| lifeless | thats a query count | 00:27 |
| thumper | lifeless: yeah | 00:27 |
| thumper | lifeless: I think that is the one I fixed | 00:28 |
| * thumper is a sad bunny | 00:28 | |
| thumper | it seems that the storm insert query has non-deterministic column ordering | 00:28 |
| lifeless | s/insert // | 00:28 |
| thumper | so two subsequent test runs with LP_DEBUG_SQL_EXTRA clash all the time | 00:28 |
| thumper | lifeless: do you know a fix? | 00:29 |
| lifeless | do you mean column or row ? | 00:29 |
| lifeless | also, ECONTEXT | 00:29 |
| lifeless | thumper: https://bugs.edge.launchpad.net/launchpad-registry/+bug/666580 - I've put the pageid in there - it helps me a lot. | 00:30 |
| _mup_ | Bug #666580: MailingListApplication:MailingListAPIView (getMessageDispositions ) mailing list xmlrpc api call makes excessive queries <mailing-lists> <qa-ok> <timeout> <Launchpad Registry:Fix Committed by thumper> <https://launchpad.net/bugs/666580> | 00:30 |
| wgrant | lifeless: %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s to you too. | 00:32 |
| wgrant | But wow. | 00:32 |
| lifeless | thumper: did you mail matt? | 00:33 |
| mars | lifeless, for shortening the release timeline, I was wondering if we could lock monday, QA tuesday, unlock wednesday, assuming staging updated ok. Timeline goes from one week to three days. | 00:40 |
| mars | if people finish their QA on Friday, then you can lock and unlock on monday | 00:42 |
| lifeless | mars: I think there is not enough slack | 00:42 |
| lifeless | mars: the critical issue is 'what if the db conversion is a problem' - it takes many hours to do a respin of that test | 00:42 |
| mars | yeah, but if it went OK, and the QA is up-to-date, then you are looking at a much shorter release | 00:43 |
| mars | "if the stars are all aligned just so" - thankfully there are only three to worry about :) | 00:44 |
| lifeless | right | 00:44 |
| lifeless | but we plan for how to handle failure | 00:44 |
| lifeless | not how to handle success ;) | 00:44 |
| lifeless | mars: I think we still need to start friday | 00:44 |
| lifeless | but we could certainly unlock earlier | 00:44 |
| lifeless | mars: I thought my proposal permitted that | 00:44 |
| mars | absolutely, it does | 00:45 |
| wgrant | lifeless: Can't you just freeze db-devel on Friday? | 00:45 |
| mars | with that extra detail, "Must start Friday, can unlock ASAP", then you can rewrite the release docs | 00:45 |
| wgrant | lifeless: devel doesn't have to be frozen to do DB restore testing. | 00:45 |
| lifeless | wgrant: we don't need to freeze db-devel at all | 00:45 |
| lifeless | wgrant: its not db restore testing | 00:46 |
| lifeless | wgrant: its upgrade-script application | 00:46 |
| wgrant | That's what I meant, sorry. | 00:46 |
| lifeless | wgrant: which we can start doing on qastaging | 00:46 |
| lifeless | (and should) | 00:46 |
| wgrant | Hmm. | 00:46 |
| wgrant | What's the point? | 00:46 |
| lifeless | convergence | 00:46 |
| mars | simplicity | 00:47 |
| lifeless | ponies! | 00:47 |
| mars | I would say 'speed', but ponies, well... | 00:47 |
| lifeless | wgrant: qastaging is sitting there without any db patches, so we don't need to do a complete restore to test the upgrade | 00:47 |
| lifeless | wgrant: which is /much/ faster | 00:47 |
| wgrant | lifeless: True. | 00:47 |
| lifeless | wgrant: and it updates its code every 30 minutes | 00:48 |
| wgrant | But this still means we have a week where production doesn't update. | 00:48 |
| lifeless | so if we have to fine tune the release, we can do so more rapidly. | 00:48 |
| lifeless | wgrant: perhaps we should change buildbot to only merge QA-ok revisions from stable to db-devel. | 00:48 |
| lifeless | wgrant: the week with prod not updating saddens me, but not as much as a month without updates | 00:49 |
| wgrant | Right, but it doesn't mean we shouldn't try to think of a way to minimise that interval. | 00:50 |
| lifeless | agreed | 00:50 |
| wgrant | Particularly now that there are no CPs. | 00:50 |
| wgrant | I don't see a huge benefit in doing the DB upgrade testing on qastaging rather than staging. And doing it on qastaging means we have to make devel undeployable several days earlier than it would be otherwise. | 00:51 |
| lifeless | mmm | 00:55 |
| lifeless | wgrant: I disagree | 00:55 |
| wgrant | Oh? | 00:56 |
| lifeless | wgrant: friday avo, sat, sun are all undeployable anyway | 00:56 |
| wgrant | True. | 00:57 |
| lifeless | I think if we get 3 clean deploys in a row without needing the extra time, we could move it up to monday. | 00:58 |
| thumper | lifeless: yes I emailed matt | 00:58 |
| lifeless | also if we have a completely clean development-dbstable report leading up to the report | 00:58 |
| wgrant | I guess that's a good idea. | 00:59 |
| StevenK | wgrant: You still have misgivings? | 01:03 |
| lifeless | StevenK: I think he's looking for a larger win | 01:04 |
| lifeless | which is good | 01:04 |
| wgrant | StevenK: I'd like there to be as little freeze of landings and production deployments as possible. So I will have misgivings forever, unless we reduce both to zero :) | 01:06 |
| wgrant | But this is better than what we have now, so it'll do. | 01:08 |
| thumper | lifeless: I've got a problem with feature flags | 01:09 |
| thumper | lifeless: it is breaking tests | 01:09 |
| thumper | lifeless: and I think I know why | 01:09 |
| lifeless | ok | 01:10 |
| lifeless | 'sup? | 01:10 |
| thumper | lifeless: there is code that checks the timeout values and uses a feature flag | 01:10 |
| thumper | this causes a query, and prior to that a flush | 01:10 |
| thumper | which can cause partially constructed object to be written to the db | 01:10 |
| lifeless | say what? | 01:10 |
| thumper | causing integrity constraint violations | 01:10 |
| lifeless | that check is done before any domain code | 01:11 |
| lifeless | its in publication | 01:11 |
| thumper | File "/home/tim/src/launchpad/incremental-diff-job/lib/lp/services/features/rulesource.py", line 90, in getAllRulesAsTuples | 01:11 |
| thumper | .find(FeatureFlag) | 01:11 |
| thumper | File "/home/tim/src/launchpad/devel/eggs/storm-0.18-py2.6-linux-x86_64.egg/storm/store.py", line 210, in find | 01:11 |
| thumper | self.flush() | 01:11 |
| thumper | those are the bits causing test explosions | 01:11 |
| thumper | perhaps for general publication | 01:11 |
| thumper | but not for tests | 01:11 |
| lifeless | so, there are several things we could do | 01:12 |
| lifeless | firstly, outside of a publication stack or similar context, timeouts are meaningless | 01:12 |
| lifeless | so we could remove that | 01:12 |
| lifeless | secondly, the feature values are cached (including absence) | 01:12 |
| lifeless | so evaluating it (e.g. by get_request_timeout()) will cache that outside your code | 01:13 |
| thumper | how does that interact with the feature flag context manager for tests? | 01:14 |
| lifeless | thirdly test requests have a Null scope provider - they could sensible /also/ install a Null rules provider | 01:14 |
| lifeless | which wouldn't call into the db | 01:14 |
| thumper | I don't grok that last oen | 01:14 |
| thumper | one | 01:14 |
| lifeless | see LaunchpadTestRequest.__init__ | 01:14 |
| thumper | lifeless: it has a NullFeatureController | 01:17 |
| lifeless | thumper: ok | 01:18 |
| lifeless | thumper: so in requests using LTR, no db access should happen at all. | 01:18 |
| thumper | but it is | 01:18 |
| lifeless | thumper: are you using maris' context manager? | 01:18 |
| thumper | the code I'm looking at is yes | 01:18 |
| thumper | self.useContext(feature_flags()) | 01:18 |
| thumper | set_feature_flag(...) | 01:18 |
| thumper | although ... | 01:19 |
| thumper | it is the set_feature_flag code that triggers the flush and the bomb... | 01:20 |
| * thumper is now confused | 01:20 | |
| thumper | because the diff should be fully constructed there... | 01:20 |
| lifeless | hmm, that would be nicer as a Fixture I think. | 01:22 |
| lifeless | something for another time. | 01:22 |
| * thumper relocates | 01:30 | |
| MTecknology | I'm gonna miss edge | 01:46 |
| lifeless | why? | 01:46 |
| MTecknology | lifeless: the thing about recipes that I spammed in the other channel | 01:47 |
| * thumper is very confused | 02:31 | |
| thumper | self.useContext(feature_flags()) | 02:31 |
| thumper | set_feature_flag(u'code.incremental_diffs.enabled', u'enabled') | 02:31 |
| thumper | self.assertTrue(getFeatureFlag('code.incremental_diffs.enabled')) | 02:31 |
| thumper | fails | 02:31 |
| thumper | in a test | 02:31 |
| thumper | but only if run after a different test | 02:31 |
| MTecknology | thumper: probably has something to do with that "self." thing you have going on. Anytimg I rely on myself for something I crash. | 02:33 |
| StevenK | Oh dear | 02:33 |
| thumper | MTecknology: :-) | 02:33 |
| lifeless | so this may be related to the cross-test thing gmb/deryck mentioned | 02:33 |
| thumper | lifeless: almost certainly | 02:33 |
| wallyworld_ | lifeless: email to devel is on my todo list - i just wanted to get the mp set up first | 02:42 |
| lifeless | wallyworld_: sure | 02:43 |
| lifeless | wallyworld_: Belts and braces ;) | 02:43 |
| wallyworld_ | lifeless: save me looking it up, what's the losa mail list? i didn't know there was one | 02:44 |
| thumper | lifeless: I'm pretty sure it is a cache invalidation problem :) | 02:44 |
| lifeless | thumper: at what layer? we shouldn't have the same featurecontroller reference should we? | 02:44 |
| lifeless | thumper: I'd guess at the request<->thread-local lifetimes being different. | 02:45 |
| lifeless | thumper: perhaps a function to ensure when one is altered, the other is too | 02:45 |
| * thumper is poking some more | 02:47 | |
| wallyworld_ | lifeless: don't worry, found it | 02:49 |
| lifeless | wallyworld_: oh sorry, yes. | 02:50 |
| wallyworld_ | lifeless: np. i was just letting you know. it wasn't mean to be snarky :-) | 02:50 |
| lifeless | I know :) | 02:50 |
| lifeless | man | 02:51 |
| lifeless | I hope noone is pissed about all these bug closes :) | 02:51 |
| MTecknology | lifeless: closed as in fixed- or closed as in won't fix? | 02:52 |
| lifeless | both | 02:52 |
| thumper | heh | 02:57 |
| thumper | set_feature_flag has a store.flush | 02:57 |
| thumper | no... | 02:58 |
| thumper | set_feature_flag adds a feature to the db | 02:58 |
| thumper | when the feature is being flushed | 02:58 |
| thumper | it is doing a db query | 02:58 |
| thumper | which queries the feature flags | 02:58 |
| thumper | which sets the _rules cache | 02:58 |
| thumper | which means the newly set feature isn't in the cache | 02:58 |
| rockstar | thumper, yeah, deryck and I fought with that last week. | 02:59 |
| rockstar | It caused much head scratching. | 02:59 |
| thumper | I'm not yet sure why sometimes it hits this and other times it doesn't | 02:59 |
| * thumper thinks it is a flush ordering problem | 03:00 | |
| thumper | I think it is easily solved though | 03:00 |
| StevenK | transaction.commit() ? | 03:00 |
| lifeless | no | 03:00 |
| StevenK | thumper: No fair sending wallyworld_ my way with a 1,500 line MP | 03:01 |
| StevenK | I know where you live ... | 03:01 |
| thumper | StevenK: I didn't know it was that big | 03:01 |
| thumper | sorry | 03:01 |
| wallyworld_ | StevenK: sorry :-( most of it is unit tests :-) | 03:01 |
| wallyworld_ | thumper: looks like devel is still in test fix mode? my pqm-submit was rejected again for that reason | 03:02 |
| StevenK | wallyworld_: I'm working through it, but the current way to get MPs > 800 lines reviewed is to 1. Not, or 2. Bribe a reviewer | 03:02 |
| StevenK | wallyworld_: Consult Ye Olde Buildbot | 03:02 |
| StevenK | We shouldn't be | 03:03 |
| wallyworld_ | StevenK: what can thumper offer you by way of inducement? :-) | 03:03 |
| StevenK | It isn't thumper's MP ... *cough* *hint* | 03:03 |
| thumper | lifeless: http://pastebin.ubuntu.com/524222/ | 03:03 |
| wallyworld_ | StevenK: yeah, but he was the one who *made* me throw you the hospital pass :-) | 03:04 |
| lifeless | thumper: does that work for you? | 03:04 |
| StevenK | It's all about choice | 03:04 |
| wallyworld_ | said it would be good for your soul :-) | 03:04 |
| thumper | lifeless: yep | 03:04 |
| lifeless | thumper: something smells here | 03:05 |
| lifeless | thumper: I think features.per_thread.features is stale perhaps ? | 03:05 |
| lifeless | thumper: if so, thats critical to fix. | 03:05 |
| thumper | lifeless: yes... in that the features has cached the rules before we add one | 03:05 |
| StevenK | wallyworld_: If thumper said that with a straight face, then I take the comment about his poker face back | 03:05 |
| lifeless | thumper: if its cached the rules from another test, or even another *request*, then there is an isolation bug that this qualifies as a workaround for. | 03:06 |
| wallyworld_ | StevenK: i can offer you paul's eternal gratitutde as he *really* wants to get merge queues done before he takes his sabatical from the code team | 03:06 |
| thumper | lifeless: more of a problem with tests than real life | 03:06 |
| thumper | lifeless: let me plug in headphones | 03:06 |
| StevenK | wallyworld_: I thought I already had that | 03:06 |
| thumper | lifeless: then skype may help here | 03:06 |
| lifeless | ok | 03:06 |
| wallyworld_ | StevenK: hmmm. i'm running out of possible bribes. | 03:07 |
| StevenK | wallyworld_: Hah, serves you right :-P | 03:13 |
| lifeless | thumper: so - two lines | 03:16 |
| lifeless | da.set_permit_timeout_from_features(False) | 03:16 |
| lifeless | ... | 03:16 |
| lifeless | da.set_permit_timeout_from_features(True) | 03:17 |
| lifeless | webapp/testing/helpers.py | 03:23 |
| thumper | lifeless: http://pastebin.ubuntu.com/524230/ | 03:23 |
| lifeless | AdapterIsolator = FunctionFixture(set_request_started, lambda x:clear_request_started()) | 03:23 |
| lifeless | in setUp | 03:23 |
| thumper | http://pastebin.ubuntu.com/524231/ | 03:23 |
| lifeless | self.useFixture(AdapterIsolator) | 03:23 |
| lifeless | --- lib/lp/testing/__init__.py 2010-10-26 15:47:24 +0000 | 03:26 |
| lifeless | +++ lib/lp/testing/__init__.py 2010-11-02 03:26:01 +0000 | 03:26 |
| lifeless | @@ -495,6 +495,9 @@ | 03:26 |
| lifeless | self.oopses = [] | 03:26 |
| lifeless | self.useFixture(ZopeEventHandlerFixture(self._recordOops)) | 03:26 |
| lifeless | self.addCleanup(self.attachOopses) | 03:26 |
| lifeless | + from canonical.launchpad.webapp import adapter | 03:26 |
| lifeless | + self.useFixture(fixtures.FunctionFixture(adapter.set_request_started, | 03:26 |
| lifeless | + lambda _:clear_request_started()) | 03:26 |
| lifeless | thumper: set_permit_timeout_from_features(False) | 03:31 |
| lifeless | thats all | 03:31 |
| lifeless | 11:33 < lifeless> https://dev.launchpad.net/PolicyAndProcess/OptionalReviews | 03:35 |
| lifeless | 11:33 < lifeless> Activities | 03:35 |
| lifeless | 11:33 < lifeless> Submit the branch to create an MP (our toolchains can look at this and it provides a location for a post landing review if the branch has that done to it). Self review with review type 'unreviewed'. Land via the normal | 03:35 |
| lifeless | landing process. | 03:35 |
| thumper | https://code.launchpad.net/~thumper/launchpad/fix-features/+merge/39819 | 03:42 |
| === mwhudson_ is now known as mwhudson | ||
| thumper | lifeless: bzr lp-land checks to make sure you don't review your own code :) | 04:00 |
| thumper | lifeless: do we land it rs=? | 04:01 |
| lifeless | thumper: change lp-land ? | 04:01 |
| thumper | maybe it asked a slave | 04:01 |
| * thumper checks | 04:01 | |
| StevenK | lp-land doesn't deal with rs= | 04:02 |
| StevenK | Or MP-less lands | 04:02 |
| thumper | the code seems to indicate it should be fine :( | 04:07 |
| thumper | but something weird is happening | 04:08 |
| thumper | me gives up and pqm-submits | 04:09 |
| lifeless | thumper: please file a bug on foundations | 04:12 |
| thumper | lifeless: on the thread local leaking? | 04:12 |
| lifeless | on lp-land not accepting your MP | 04:12 |
| lifeless | thumper: I bet its the review type though | 04:12 |
| lifeless | thumper: try changing the type to nothing/code | 04:12 |
| thumper | ah | 04:12 |
| thumper | yeah, you are right | 04:12 |
| thumper | which is fucked | 04:13 |
| * thumper leaves to cook | 04:13 | |
| StevenK | At 5pm? | 04:18 |
| thumper | StevenK: I have kids | 04:18 |
| lifeless | StevenK: hey | 04:23 |
| lifeless | you know soyuz stuff | 04:23 |
| lifeless | help: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html | 04:23 |
| StevenK | I will look after shopping | 04:25 |
| thumper | heh | 04:31 |
| lifeless | thanks | 04:32 |
| LPCIBot | Project devel build (173): STILL FAILING in 3 hr 52 min: https://hudson.wedontsleep.org/job/devel/173/ | 04:33 |
| lifeless | wow Archive:EntryResource:getBuildSummariesForSourceIds is slow | 04:39 |
| lifeless | still timing out at 20 seconds | 04:39 |
| lifeless | 1/2 sec per SELECT * FROM ((SELECT BinaryPackageBuild.distro_arch_series, BinaryPackageBuild.id, BinaryPackageBuild.package_build, | 04:40 |
| wgrant | lifeless: Do we have graphs of page performance vs time? | 04:47 |
| wgrant | lifeless: ie. do we know if it's a pg 8.4 regression, or a BFJ refactor regression, or is it just generally crap? | 04:47 |
| lifeless | its crap now | 04:47 |
| lifeless | the api wasn't timing out before, so 8.4 regression | 04:48 |
| lifeless | I think | 04:48 |
| lifeless | no we don't have charts per-query | 04:48 |
| lifeless | wgrant: yeah - https://bugs.edge.launchpad.net/soyuz/+bug/662523 | 04:49 |
| _mup_ | Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/662523> | 04:49 |
| === almaisan-away is now known as al-maisan | ||
| jtv | Hi henninge | 07:18 |
| henninge | Hey jtv! ;) | 07:19 |
| henninge | Feeling better? | 07:19 |
| jtv | Grr why don't we have checksums on the Ubuntu iso download page? | 07:38 |
| thumper | interesting failure on launchpad prod branch in buildbot - http://pastebin.ubuntu.com/524300/ | 07:39 |
| wgrant | jtv: I suspect that you don't want to know. | 07:39 |
| wgrant | (the design team probably said they were user-hostile, or something) | 07:39 |
| wgrant | But they're easy enough to find on the mirrors. | 07:40 |
| lifeless | night all | 07:40 |
| wgrant | Night lifeless. | 07:40 |
| jtv | wgrant: well I do want to bloody know. I now have supposedly identical ISOs with different checksums and it should be easy to figure out which, if any, is right. | 07:40 |
| jtv | Night lifeless | 07:40 |
| wgrant | jtv: http://releases.ubuntu.com/10.10/MD5SUMS | 07:40 |
| jtv | Thanks | 07:40 |
| wgrant | Nice security.py speedup, btw. | 07:41 |
| wgrant | It's, er, quite effective. | 07:41 |
| jtv | Thanks. | 07:41 |
| jtv | I didn't bother with the remaining largest time waster inside the script, since it only took 4 seconds. | 07:41 |
| jtv | Script startup however does seem to take quite a while. | 07:41 |
| wgrant | Most of the remaining 'make schema' time seems to come from build. | 07:42 |
| jtv | So that's our next target. | 07:43 |
| StevenK | The WADL isn't very fast either | 07:44 |
| wgrant | Then lifeless can delete ZCML. | 07:44 |
| wgrant | And we can build a tree in seconds! | 07:44 |
| wgrant | StevenK: That's in build. | 07:44 |
| wgrant | compile has slow buildout. | 07:44 |
| wgrant | build has compile and WADL. | 07:44 |
| StevenK | wgrant: I spent the trip from ORD to LAX exporting blueprints, I got to know very well how long it takes to generate. | 07:45 |
| wgrant | StevenK: I tried to profile the WADL generator. | 07:45 |
| wgrant | But it took too long. | 07:45 |
| StevenK | Haha | 07:45 |
| * StevenK attempts to come up with a clean joke, fails | 07:45 | |
| StevenK | Hmm. Apparently a horse race happened today | 07:47 |
| wgrant | I've been deliberately avoiding finding out who won. | 07:48 |
| StevenK | I didn't even realise until Sarah loaded smh.com.au while I was walking past | 07:49 |
| wgrant | Hah. | 07:51 |
| wgrant | It's hard not to realise down here; it's a public holiday. | 07:51 |
| StevenK | Yeah, I knew that | 07:52 |
| wgrant | Why a sporting event gets a public holiday I will never know. | 07:53 |
| jpds | Is it the cricket? | 07:54 |
| wgrant | Less boring than that. | 07:55 |
| StevenK | Paint drying? | 07:55 |
| wgrant | Indeed. | 07:56 |
| nigelb | Something less boring than cricket? Interesting. | 07:57 |
| jtv | I wouldn't go _that_ far… | 07:59 |
| wgrant | jtv: Do you suggest that something is more boring than cricket? | 08:00 |
| LPCIBot | Project devel build (174): STILL FAILING in 3 hr 27 min: https://hudson.wedontsleep.org/job/devel/174/ | 08:00 |
| LPCIBot | Launchpad Patch Queue Manager: [r=jtv][ui=none][bug=667554] Don't send email for work in progress | 08:00 |
| LPCIBot | merge proposals when requesting reviews or modifying the proposal. | 08:00 |
| jtv | wgrant: read what nigelb said more closely! | 08:01 |
| wgrant | Ah ha.. | 08:01 |
| jtv | My dad was dragged into a cricket match once. Didn't even find it boring, to his surprise. | 08:01 |
| jtv | A decade later he happened to open a cricket mag (at the dentist's or something) and guess what? They were still talking about that famous exciting match. | 08:01 |
| nigelb | lol | 08:02 |
| nigelb | jtv: heh, you caught that :D | 08:02 |
| jtv | nigelb: red-handed | 08:02 |
| nigelb | heh | 08:02 |
| nigelb | I watched IPL to any extent only for one season. Lost interest. | 08:03 |
| jtv | IPL = Initial Program Load? Still talking about profiling? | 08:03 |
| nigelb | No, was talking about criket :) | 08:03 |
| jtv | What's IPL stand for? | 08:04 |
| nigelb | Indian Premier League. The 'big thing' in 20-20. meh. | 08:04 |
| jtv | nigelb: thank you for TLA #23662 | 08:04 |
| _mup_ | Bug #23662: [network-admin] 'connection settings' should not be greyed out when not active (in network settings) <network-admin> <gnome-system-tools (Ubuntu):Invalid by desktop-bugs> <https://launchpad.net/bugs/23662> | 08:04 |
| jtv | No mup, not that. | 08:05 |
| jtv | http://xs4all.nl/~jtv/gtf/ | 08:05 |
| nigelb | jtv: lol | 08:05 |
| jtv | nigelb: you can now carry the GTF Contributor Program (GCP) logo on your website or home page. | 08:05 |
| nigelb | \o/ | 08:06 |
| nigelb | Well, this day did bring one achievement. | 08:06 |
| jtv | Quite. | 08:06 |
| StevenK | jtv: steven@liquified:~$ host hugeurl.wiggy.net | 08:06 |
| StevenK | Host hugeurl.wiggy.net not found: 3(NXDOMAIN) | 08:06 |
| StevenK | :-( | 08:06 |
| nigelb | liquified, no wonder. | 08:07 |
| jtv | I guess wichert must have shut it down. It was about a decade ago. | 08:07 |
| jtv | The idea was that tinyurl etc. are nice, but small URLs don't look impressive. | 08:07 |
| jtv | So he created a URL stretcher. | 08:07 |
| StevenK | Indeed | 08:07 |
| StevenK | Yes, and Wichert is the kind of guy to do it. :-) | 08:07 |
| jtv | One of the encoding schemes available was: look for three-letter combinations that are in the GTF, and expand them. | 08:07 |
| jtv | Haven't seen him in ages. You? | 08:08 |
| StevenK | Neither | 08:08 |
| jtv | :( | 08:08 |
| StevenK | Not for 5 years or so | 08:08 |
| jtv | One wonders how he is. | 08:08 |
| StevenK | He fell into the black hole that previous DPLs get sucked into | 08:09 |
| jtv | Does that apply to former colleagues and university friends? | 08:09 |
| StevenK | jtv: Ah, you were at XS with him? | 08:09 |
| jtv | No, cistron | 08:10 |
| jtv | And the first FOSDEM—then still called OSDEM. | 08:10 |
| jtv | I arrived late, having escaped from a lady's bedroom window that morning in a different country. | 08:10 |
| StevenK | Now there's something that sounds like an interesting story | 08:11 |
| jtv | I think it was our mutual friend Ray who started the vicious and baseless rumour that I had escaped from a lady's bathroom window. | 08:11 |
| jtv | If you ever hear that version, don't believe it! | 08:11 |
| StevenK | Perhaps it was her ensuite window | 08:11 |
| jtv | Her wha? | 08:12 |
| StevenK | An ensuite is a small bathroom directly accessible from a bedroom | 08:12 |
| jtv | Ah. | 08:12 |
| jtv | No, it was definitely bedroom. | 08:12 |
| jtv | But thanks for teaching me that word. | 08:15 |
| StevenK | Heh | 08:16 |
| nigelb | jtv: Wait, do we get to hear more of that exploit? | 08:18 |
| jtv | nigelb: what do _you_ think? | 08:18 |
| wgrant | We must. | 08:18 |
| nigelb | We demand. | 08:19 |
| jtv | Get me drunk and we'll talk. | 08:19 |
| nigelb | dammit, if I had the money, I'd fly to Australia just to get you drunk ;) | 08:19 |
| jtv | If you don't have the money to fly to Australia, what makes you think you have the money to get me drunk? | 08:19 |
| nigelb | Good point. | 08:20 |
| nigelb | I was planning on getting you drunk enough for you to hand me you purse and hards :p | 08:20 |
| jtv | My secret is safe for now. | 08:20 |
| nigelb | Good plan? ;D | 08:20 |
| wgrant | Until January. | 08:20 |
| nigelb | What's in Jan? Epic? | 08:20 |
| wgrant | Ja. | 08:20 |
| nigelb | Too bad its a closed event :( | 08:21 |
| jtv | Face it. You're not looking for an open event. You're looking for an open bar. | 08:21 |
| jtv | (See "money" above) | 08:21 |
| nigelb | heh | 08:22 |
| nigelb | True, that. | 08:22 |
| nigelb | jtv: Love your homepage. "Nor do I speak for my employer; he is quite old enough to speak for himself." | 08:24 |
| jtv | That's served me well for a fair number of employers. :) | 08:25 |
| jpds | What if they're a she? | 08:26 |
| jtv | As now, I suppose, they are. | 08:27 |
| jtv | I didn't know about that particular bit of the English language at the time of writing. Will rectify. | 08:27 |
| adeuring | good morning | 08:42 |
| henninge | Everybody: land you branches! Testfix coming up again ... :-( | 09:13 |
| StevenK | I've been seeing the same failures in Hudson | 09:14 |
| mrevell | Hello | 09:16 |
| deryck | Morning, all. | 11:03 |
| LPCIBot | Yippie, build fixed! | 11:06 |
| LPCIBot | Project db-devel build (111): FIXED in 3 hr 55 min: https://hudson.wedontsleep.org/job/db-devel/111/ | 11:06 |
| deryck | gmb, hi. Did you see thumper's email about test isolation failure and his fix to set_feature_flag? | 11:13 |
| gmb | deryck: Yes; I've already merged my branch but I'll try splitting the tests up again and see if it works now. | 11:14 |
| deryck | gmb, ok, cool. I expect that will fix the problem we were seeing, too. If using the testing helpers. mars fixture would need to flush too. | 11:14 |
| gmb | Right. | 11:14 |
| * deryck feels like getting on to the kids.... "you need to flush every time!" | 11:15 | |
| LPCIBot | Project devel build (175): STILL FAILING in 3 hr 31 min: https://hudson.wedontsleep.org/job/devel/175/ | 11:32 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=bac][ui=none][bug=586461, 634326, | 11:32 |
| LPCIBot | 634646] Stop generating invalid memcache keys and allow more cache | 11:32 |
| LPCIBot | sharing. | 11:32 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=668194] Fix IHasTranslationImports in the | 11:32 |
| LPCIBot | API. | 11:32 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=668194] Split off some interfaces stuff | 11:32 |
| LPCIBot | into separate files. | 11:32 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa] Move specification enums into | 11:32 |
| LPCIBot | lp.blueprints.enums. | 11:32 |
| gmb | Hurrah for ambiguous messages from bots. | 11:39 |
| cjohnston | deryck: bug 483027 seems to be working now.. I was able to subscribe someone else and myself | 11:48 |
| _mup_ | Bug #483027: Display problem when adding subscribers to a bug report <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/483027> | 11:48 |
| deryck | cjohnston, awesome. Thanks for the confirmation. | 11:48 |
| cjohnston | np | 11:48 |
| * gmb just self-reviewed a branch, feels naughty. | 11:50 | |
| cjohnston | gotta do what you gotta do sometimes | 11:50 |
| jml | mrevell: hello | 12:41 |
| mrevell | hello juml | 12:41 |
| mrevell | oh, jml | 12:41 |
| jml | mrevell: :) | 12:41 |
| mrevell | :) | 12:42 |
| jml | mrevell: some guy at UDS came up to me and said, "How do you think of all those clever gadgets?" | 12:42 |
| mrevell | Seriously? Excellent :) | 12:42 |
| jml | yeah :) | 12:42 |
| jml | mrevell: anyway, I was wondering if I could do anything to help with the user testing process discussion? | 12:43 |
| mrevell | jml, Do you have time for a call this afternoon? | 12:43 |
| jml | mrevell: yeah, I could have a call either after or before the standup | 12:44 |
| mrevell | jml, After would be great. Thanks. I think we got pretty close to a general rule: if you need a LEP, you are likely to benefit from user testing. | 12:44 |
| deryck | allenap, hi. On my review yesterday, you suggested I use inTeam, but I don't follow why, since bug.owner can never be a team? | 13:16 |
| allenap | deryck: It was a general comment that it's not always safe to just compare IDs, and inTeam() does the right thing. (I assume bug.owner is the reporter, not the assignee; if the latter then it can be a team.) | 13:18 |
| deryck | allenap, right, it's the reporter. So I'd prefer to keep it on ID, even if inTeam covers that case, just so it's clear. Cool? | 13:21 |
| allenap | deryck: Okay. | 13:21 |
| deryck | allenap, ok, thanks. | 13:22 |
| allenap | deryck: I didn't mean - I rarely mean - for my review comments to be prescriptive. | 13:23 |
| deryck | allenap, oh, I didn't take it that way. Just wanted to chat more to make sure I wasn't misunderstanding you. :) | 13:23 |
| jml | I have a soyuz question | 13:50 |
| jml | I'd like to open up "o" and "p" series for Ubuntu. I'm told that this ought to work, but everyone I've asked has sounded unsure | 13:51 |
| jml | a) how could we test that this works? or | 13:51 |
| jml | b) if we go ahead and do it, are we capable of easily undoing the mistake? | 13:51 |
| jelmer | jml: I also think that it *should* work. You just want to have them in the database I assume, not allow uploads yet? | 13:53 |
| jml | jelmer: exactly | 13:53 |
| jelmer | jml: We should be able to test it on dogfood or staging I guess. It might be nice to check with bigjools or Colin that there isn't anything I'm overseeing, I haven't been involved in adding distroseries before so there might be some subtle bug I'm not aware of. | 13:55 |
| jml | jelmer: bigjools has said much the same: "it should work, let's test on dogfood" | 13:55 |
| jml | jelmer: and cjwatson, iirc, didn't have any reservations | 13:56 |
| jelmer | jml: Ah, great. Just checking. :-) | 13:56 |
| jml | jelmer: could you please test that on dogfood? | 13:56 |
| jelmer | jml: Sure. | 13:58 |
| jml | jelmer: thanks. it might be a good idea to also get someone platformish to try to break it. maybe cjwatson? | 13:59 |
| allenap | deryck: I just realised that the bug importer could create a new bug with a team as owner. | 14:00 |
| deryck | allenap, ah, good catch. I'll add a test then with team ownership and fix up the code. | 14:00 |
| allenap | deryck: The team would need to already exist in Launchpad and have an email address corresponding to an email address in the bug import XML. | 14:01 |
| jelmer | jml: I'll check with him once I've got it set up. | 14:01 |
| jml | jelmer: sweet. thanks. | 14:01 |
| allenap | So, if I have a branch that I don't think needs review, how do I indicate that? | 14:02 |
| allenap | Do I just approve the mp myself and land it? | 14:03 |
| jml | allenap: that seems sensible | 14:04 |
| allenap | jml: Okay, thanks :) | 14:04 |
| jml | which makes me think of a thing | 14:04 |
| jelmer | jml: btw, this would require knowing the distroseries codenames beforehand. would that possible? | 14:04 |
| jml | jelmer: does it really? can we not rename them later? | 14:04 |
| deryck | allenap, jml -- I thought there was something about [r=unreviewed] to track these. | 14:04 |
| deryck | not sure our tools support that yet, obviously. | 14:05 |
| jml | deryck: :9 | 14:05 |
| jml | (that was a frown fail) | 14:05 |
| deryck | heh | 14:05 |
| deryck | I thought it was a district 9 grimace. | 14:05 |
| * allenap tried to do that with his mouth. | 14:05 | |
| jml | deryck: I don't know. is there a wiki page for the 'speriment? | 14:05 |
| deryck | i think so.... | 14:06 |
| * deryck is looking | 14:06 | |
| jelmer | jml: I don't think we've ever tried that. I guess files in PPA's with the wrong names won't be an issue as we won't enable these distroseries yet, I wonder if there's other places where we have the distroseries name hardcoded. | 14:06 |
| jml | allenap: btw, a thing that we do with testtools reviews is we have a 24hr timeout on review requests | 14:06 |
| deryck | https://dev.launchpad.net/PolicyAndProcess/OptionalReviews | 14:06 |
| allenap | jml: So, after 24h you can land without a review? | 14:06 |
| jml | allenap: yeah | 14:06 |
| jml | allenap: one thing I'll be doing for my own landings under this process is asking for a review and having a 5-10m timeout | 14:07 |
| jml | deryck: ta | 14:07 |
| allenap | jml: If I want to land sucky code I should submit on 24th or 31st December? | 14:07 |
| jml | allenap: 24hr timeout on testtools for folk w/ commit access :) | 14:08 |
| jml | allenap: if you don't have commit access, sucks to be you | 14:08 |
| allenap | jml: Ah, and I've probably just made it harder to get commit access ;) | 14:09 |
| jml | heh heh | 14:09 |
| allenap | Ah, bin/ec2 land does not consider "unreviewed" reviews as reviews and won't land. | 14:12 |
| jml | patch patch patch :) | 14:13 |
| jelmer | jml: Can you think of any other things that have a distroseries name hardcoded at the moment? The packaging branches uses something based on the database id, right? | 14:14 |
| jml | jelmer: hmm, yeah, but the stacking URL is hardcoded in the .bzr dir | 14:14 |
| jml | jelmer: perhaps though we forbid setting the official branch for distroseries that are in FUTURE? | 14:15 |
| jml | iirc the check is tied to "can you upload"? | 14:15 |
| jelmer | jml: aren't they generally stacked on the project trunk though? | 14:15 |
| jml | jelmer: packaging branches are stacked on lp:ubuntu/foo | 14:15 |
| jelmer | jml: ah, ok | 14:15 |
| jelmer | jml: yeah, forbidding the setting of official branches for non-enabled distroseries makes sense. | 14:16 |
| jml | jelmer: but I'm not 100% sure that's the case. would need to test. | 14:16 |
| jml | (or read the code) | 14:16 |
| lifeless | garh | 14:26 |
| allenap | jml: We could say that if you review your own mp then it's unreviewed, then we don't need to patch the tools, and there's no loss of information. Right now both bzr-pqm and lp need to be patched :-/ | 14:26 |
| jml | lifeless: your sleep cycle is still off? | 14:27 |
| allenap | lifeless: ^ too. | 14:27 |
| lifeless | jml: I just woke up. | 14:27 |
| lifeless | jml: I feel a little tired but 'awake'. | 14:27 |
| jml | allenap: yeah, that makes sense to me. I guess it would be nice to patch ./utilities/ec2 to accept self-reviews tagged 'unreviewed' | 14:28 |
| lifeless | allenap: As long as I can reliably find the MP's that were self-reviewed, for the metrics angle. | 14:28 |
| allenap | jml: Okay. I'll file a bug for that (which I might fix myself anyway). | 14:28 |
| allenap | lifeless: Cool, I'm make sure we can do that. Is via the API enough? | 14:29 |
| allenap | Actually, can merge proposals be searched for via the web UI anyway? | 14:29 |
| === shadeslayer_ is now known as shadeslayer | ||
| allenap | Only by status it seems. | 14:31 |
| lifeless | allenap: API is fine. | 14:31 |
| allenap | Cool. | 14:31 |
| LPCIBot | Project db-devel build (112): SUCCESS in 3 hr 31 min: https://hudson.wedontsleep.org/job/db-devel/112/ | 14:38 |
| LPCIBot | Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11820, | 14:38 |
| LPCIBot | 11821, 11822, 11823, 11824 included. | 14:38 |
| LPCIBot | Project devel build (176): STILL FAILING in 3 hr 29 min: https://hudson.wedontsleep.org/job/devel/176/ | 15:01 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=656823] Various bits and pieces around | 15:01 |
| LPCIBot | PersonSubscriptionsView. | 15:01 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=jml, | 15:01 |
| LPCIBot | stevenk][ui=none][bug=666660] Lower log level for 'Translations ... | 15:01 |
| LPCIBot | match n existing translations.' to INFO to avoid it being | 15:01 |
| LPCIBot | turned into an OOPS. | 15:01 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=638920] Only try to show links to private | 15:01 |
| LPCIBot | branches to authorized users ond on a product series' | 15:01 |
| LPCIBot | translations page. | 15:01 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=664566, | 15:01 |
| LPCIBot | 664569] BugNotificationLevel now has a more readable set of | 15:01 |
| LPCIBot | descriptions for Bug:+subscribe. BugNotificationLevel.NOTHING | 15:01 |
| LPCIBot | is no longer accepted when one is attempting to subscribe to a | 15:01 |
| LPCIBot | bug through the web UI. | 15:01 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Explicitly flush the store when adding | 15:01 |
| LPCIBot | feature flags, and stop our tests from checking features for timeouts. | 15:01 |
| dobey | that is a noisy bot | 15:04 |
| flacoste | lifeless: if you are bored, can you review https://code.launchpad.net/~flacoste/launchpad/ppr-constant-memory/+merge/39666 ? | 15:07 |
| dobey | so with the API, it appears that branch.landing_targets doesn't contain all the proposals, if branch.status == 'Merged'; is that correct? | 15:17 |
| dobey | gary_poster: ^^ you wanted to talk to me about reviews API anyway, right? :) | 15:24 |
| lifeless | flacoste: bored. Hah! have you seen my job description :) - sit around bored ain't on it :P | 15:26 |
| gary_poster | dobey: I'm afraid I don't know the answer to that. :-/ | 15:26 |
| flacoste | lifeless: well, it might help you sleep :-) | 15:26 |
| dobey | gary_poster: hrmm, ok. i'm having some issues with landing branches with prerequisites becuase of it :( | 15:27 |
| === benji is now known as benji-lunch | ||
| gary_poster | dobey: I'd be reading the code along with you. Someone from the code team would probably be much more efficient help, if they are around. (That said, if you really think a particular bit of code would be valuable for me to stare at, go ahead and send me there) | 15:29 |
| lifeless | gary_poster: hi | 15:35 |
| lifeless | gary_poster: got a few minutes to catch up? I particularly want to talk edge with you | 15:35 |
| gary_poster | lifeless, sure | 15:39 |
| gary_poster | now on Skype and mumble both | 15:39 |
| dobey | gary_poster: i have no idea where in lp the code is. http://pastebin.ubuntu.com/524489/ is the code in tarmac that's checking landing_targets for prerequisite branches. and len(merges) seems to be 0 there, when the prerequisite branch is merged :( | 15:42 |
| flacoste | lol, "trendy vs free"! | 15:50 |
| flacoste | jml: should we list other internal lists tangentially related like canonical-tech or canonical-javascripters? | 15:50 |
| flacoste | they are not strictly launchpad | 15:51 |
| flacoste | otherwise, i don't see anything missing | 15:51 |
| jml | flacoste: I guess we can list those in a separate section. | 15:51 |
| === beuno is now known as beuno-lunch | ||
| === al-maisan is now known as almaisan-away | ||
| lifeless | jml: flacoste: want to continue our calls ? | 16:20 |
| flacoste | lifeless: going to lunch | 16:20 |
| lifeless | gary_poster: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/670013 | 16:24 |
| _mup_ | Bug #670013: preflight check for removing edge cluster <Launchpad Foundations:New> <https://launchpad.net/bugs/670013> | 16:24 |
| gary_poster | ack lifeless, thanks | 16:25 |
| jml | lifeless: I'm otp | 16:25 |
| gary_poster | dobey, swamped, will ping you when I'm up for air | 16:25 |
| lifeless | dobey: please file a question on launchpad-code | 16:25 |
| lifeless | dobey: that will get to the right people | 16:26 |
| dobey | ok | 16:28 |
| === Ursula is now known as Ursinha-afk | ||
| deryck | lifeless, I'd like to change the hard_timeout rule for BugTask:+create-question to 25 seconds. Any objections? | 16:40 |
| lifeless | yes | 16:45 |
| lifeless | past 20 seconds is likely to start permitting starvation of appservers via haproxy | 16:45 |
| lifeless | sadly. | 16:46 |
| lifeless | we have some data that mails are taking a long time to send. | 16:46 |
| lifeless | adding more datagathering there would be a --very good-- idea | 16:47 |
| lifeless | deryck: We know that that page goes past 30 seconds | 16:47 |
| lifeless | deryck: and that will just get cut off by haproxy - there's no benefit taking it up that high. | 16:47 |
| deryck | lifeless, no benefit past 30, for the case where we know we'll hit 30? | 16:48 |
| lifeless | deryck: if the total time before the response hits haproxy is 30 seconds, the user gets a 'could not contact lp' | 16:48 |
| lifeless | deryck: the hard_timeout only influences time from when the request servicing *starts* in LP | 16:49 |
| lifeless | deryck: haproxy feeds a backlog of 4 requests per appserver (active threads=4, depth=8) | 16:49 |
| lifeless | deryck: Taking the hard_timeout above 20 seconds means that < 10 seconds queue time (under load) is permitted, and the queue depth (under load) will be 4 on the server... | 16:50 |
| lifeless | deryck: so I wouldn't take the hard_timeout much above 20 seconds | 16:50 |
| lifeless | deryck: I set milestones to 22 seconds because the data suggest its capped there, more or less. | 16:51 |
| lifeless | deryck: For +create-question, I don't think you'll let many more pages complete by adding 5 seconds - I think there is a big gap | 16:51 |
| lifeless | between 'ok' and 'fucked' | 16:51 |
| deryck | lifeless, ah, ok. I follow you now. | 16:51 |
| deryck | lifeless, the OOPS from micahg had 15.XX seconds. But the graph seemed to indicate lots on the right side at 20, so was guessing really. | 16:52 |
| lifeless | deryck: we need to debug the mail thing | 16:52 |
| deryck | lifeless, right. I also think we should do 20 then to see if that helps micahg temporariliy and get this work scheduled ASAP. cool? | 16:53 |
| lifeless | deryck: I realise its strictly foundations, but I'd like to encourage you to just go ahead and add instrumentation to find out where the time is going :) | 16:53 |
| lifeless | deryck: cool | 16:53 |
| deryck | lifeless, oh, I really don't think like that short of trying to assign bugs correctly :-) I'm happy to add it. | 16:53 |
| deryck | I'm extremely overloaded right now, though. So it will be late this week or first of next before I can add it. And want to get micahg going again if I can. | 16:54 |
| sinzui | flacoste, mumble? | 17:34 |
| flacoste | sinzui: yes sir | 17:34 |
| jml | lifeless: would you be ok w/ me upgrading LP to use a build of testtools trunk? | 17:47 |
| lifeless | +10000000 | 17:47 |
| jml | cool. | 17:48 |
| jml | lifeless: I figure that upgrading fixtures & testtools separately will ease landing of my testtools-experiment branch | 17:48 |
| lifeless | jml: I'm always happy with snapshots [of upstream] that make things better. Snapshots [not of upstream] need a /little/ more thought. | 17:49 |
| jml | lifeless: makes sense. | 17:50 |
| jml | normally I would do a release first, but I'm reluctant to release all of this deferred stuff until I've proven that it works for at least one project's test suite. | 17:51 |
| lifeless | zigactly. | 17:51 |
| lifeless | jml: Can we chat? | 17:53 |
| lifeless | or rather. Speak. | 17:53 |
| === benji-lunch is now known as benji | ||
| jml | lifeless: yes. gimme a sec to finish dumping state. | 17:55 |
| mars | has anyone hit PQM with a testfix yet? | 17:55 |
| mars | I'll assume no - I have a one-line change I can hit it with | 17:57 |
| jml | lifeless: ok. ready. | 18:00 |
| lifeless | skype doesn'tthink so | 18:00 |
| lifeless | jml: you've dropped off? | 18:17 |
| === leonardr is now known as leonardr-voting | ||
| lifeless | flacoste: have discussed work queues with jml - EFUTURE, but broadly interesting | 18:32 |
| flacoste | lifeless: ok | 18:38 |
| flacoste | lifeless: btw, seems like gary and you see eye to eye on the value of regression tests :-) | 18:38 |
| lifeless | :) | 18:38 |
| gary_poster | :-) | 18:39 |
| lifeless | bah | 18:40 |
| lifeless | I'm going to have to do the mentoring patch on db-devel. | 18:40 |
| lifeless | the magic cross-check stuff bites | 18:40 |
| jml | g'night all. | 18:53 |
| lifeless | night | 18:53 |
| lifeless | jelmer: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html - we're blocked on QAing that | 18:53 |
| lifeless | deryck: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-pageids.html | 18:54 |
| * deryck looks | 18:54 | |
| lifeless | deryck: click on '99% under time' and observe that the top (or near top) row is BugTask:+create-question, with a 99% completion time of 102 seconds. | 18:54 |
| * deryck is waiting on the page | 18:55 | |
| lifeless | science bitches, it works! | 18:55 |
| === leonardr-voting is now known as leonardr | ||
| cr3 | leonardr: hi there, I've been looking at the EntryResource class under the lazr.restful._resource package to find out how I could potentially add the attribute http_link, similar to the self_link, across all my objects. | 19:05 |
| leonardr | cr3: are you trying to fix the bug about this? i thought rockstar already had something | 19:05 |
| cr3 | leonardr: I wasn't aware of any bug, might it be related to the EntryResource class is assumed in a few places, like EntryHTMLView for example? | 19:06 |
| lifeless | I'm going to - shock, horror - write some code | 19:08 |
| deryck | lifeless, so to simply state your assertion -- "changing the default_timeout will not help. We have to fix the page to help micahg." | 19:23 |
| lifeless | deryck: yeah :) | 19:24 |
| deryck | gotcha | 19:24 |
| lifeless | deryck: I thought the evidence would be useful, and scary. | 19:24 |
| deryck | lifeless, yes, it is useful. Nothing scares me about lp anymore. Scars me, yes. Scares me, no. | 19:25 |
| lifeless | \o/ | 19:25 |
| lifeless | you've passed the fear threshold | 19:25 |
| deryck | or else I'm too dumb to be afraid. | 19:25 |
| lifeless | rotfl | 19:25 |
| lifeless | EdwinGrubbs: hey | 19:28 |
| lifeless | EdwinGrubbs: did I answer your question sufficiently about that timeout bug the other day ? I put it in the bug discussion. | 19:29 |
| SpamapS | I heard at UDS that stats for PPA downloads are coming soon? Anybody know if there's an open bug/blueprint for it that I can point others at? | 19:30 |
| lifeless | SpamapS: http://www.google.com/search?sourceid=chrome&client=ubuntu&channel=cs&ie=UTF-8&q=ppa+stats | 19:31 |
| lifeless | SpamapS: learn to love the search | 19:31 |
| SpamapS | lifeless: i'd like to personally thank you for mot lmgtfy'ing me for that gross misappropriation of your time and brainpower. ;) | 19:33 |
| EdwinGrubbs | lifeless: thanks for adding that comment. I'll try the eager loading, although I have doubts that it will help, since it looks like most of the time is spent creating storm objects. However, I've only looked at ++profile++show so far, and I need to check if the ++profile++log got copied over to devpad. | 19:33 |
| lifeless | EdwinGrubbs: so the storm object creation concern is coupled to storm object volume | 19:34 |
| lifeless | EdwinGrubbs: less queries can reduce that volume | 19:34 |
| lifeless | EdwinGrubbs: how many objects are being created? 5K ? 10K ? | 19:34 |
| lifeless | SpamapS: I'd never do that to you :). | 19:35 |
| EdwinGrubbs | lifeless: I'm guessing 7,500 based on the number of _set_values calls. I just noticed that the ++profile++show says "Total(ms)". Isn't that in seconds? | 19:38 |
| lifeless | EdwinGrubbs: yes | 19:38 |
| lifeless | its bong from the 2.4 change to profiling | 19:38 |
| lifeless | EAttentionToDetail. | 19:38 |
| lifeless | EdwinGrubbs: for clarity, yes, its in seconds. | 19:43 |
| lifeless | the ms clause is from the older profiler which did report in ms. | 19:44 |
| james_w | jml, hi, I'm hearing rumours that a merge of the bug and blueprint code is planned, is that the case? | 19:47 |
| bryceh | james_w, wow | 19:49 |
| lifeless | james_w: jml has EOD'd. | 19:49 |
| lifeless | james_w: but yes. | 19:49 |
| lifeless | james_w: during 2011 | 19:50 |
| james_w | is there more information available anywhere? | 19:50 |
| lifeless | dev.launchpad.net/IssueTracker is probably the best source today | 19:50 |
| lifeless | I don't know if that captures jmls current thinking | 19:50 |
| lifeless | but its certainly not going to be hugely far off | 19:51 |
| cr3 | have there been requests to get the web url for objects retrieved through the api? | 19:58 |
| james_w | cr3, yes | 19:59 |
| cr3 | james_w: cool, I had a feeling there might be a reason why it wasn't currently available, either because there was no such request or because there might be a fundamental problem with providing this information | 20:00 |
| cr3 | and searching bugs for "api url" under the launchpad project, which returned nothing, only reinforced that feeling :) | 20:03 |
| james_w | https://bugs.edge.launchpad.net/launchpadlib/+bug/316694 | 20:04 |
| _mup_ | Bug #316694: Add web_link property to resources <launchpadlib :Triaged> <https://launchpad.net/bugs/316694> | 20:04 |
| cr3 | james_w: cool, now I know what to name it too! | 20:05 |
| leonardr | cr3: you should definitely talk to rockstar, i know he did some work on this | 20:09 |
| rockstar | leonardr, I did, but we got stuck on some weird test failures that made things more complicated. | 20:09 |
| cr3 | leonardr: has there been a consensus on whether this should be done client or server side? I found the thread in the bug fascinating and I have no position myself | 20:10 |
| leonardr | cr3: i think server side | 20:10 |
| cr3 | rockstar: might there be a branch I could peek at? | 20:11 |
| thumper | morning | 20:11 |
| lifeless | note that canonical_url is a known performance problem. | 20:12 |
| lifeless | it would be sad to make APIs substantially slower to do this; perhaps working on canonical_url would be a good first step. | 20:13 |
| rockstar | cr3, there is indeed. Lemme find it. | 20:14 |
| rockstar | thumper, morning. Might we have a chance to catch up? | 20:14 |
| thumper | rockstar: we may | 20:15 |
| thumper | rockstar: let me play with the mixer and mic | 20:15 |
| rockstar | thumper, let me find this branch for cr3 real quick and then I'll jump on skype. | 20:15 |
| rockstar | cr3, | 20:17 |
| rockstar | <beuno> so I preemptively blacked it out | 20:17 |
| rockstar | Argh... | 20:17 |
| rockstar | Stupid middle click grabbing randomness... | 20:17 |
| rockstar | cr3, https://code.launchpad.net/~rockstar/lazr.restful/web_link | 20:17 |
| cr3 | rockstar: cheers! | 20:18 |
| lifeless | man, its so nice being able to just use lp directly. | 20:34 |
| lifeless | \o/ | 20:34 |
| lifeless | james_w: did that help you ? | 20:35 |
| allenap | Does anyone know why I have had to re-authorize my launchpad-branch-lander API key 2 (3 maybe?) times in the last couple of days? | 20:35 |
| james_w | lifeless, the spec on blueprints and bugs? | 20:36 |
| lifeless | james_w: the wiki page :) | 20:36 |
| james_w | lifeless, as much as a spec from 2006 can, yes | 20:36 |
| lifeless | kk | 20:36 |
| lifeless | I'm happy to tell you more | 20:36 |
| lifeless | or voice chat | 20:36 |
| lifeless | it depends what you want to know | 20:37 |
| james_w | I'm mainly interested in where it fits with other blueprint changes, such as the accepted stakeholder proposal for a better workload page | 20:37 |
| lifeless | so the broad description is perhaps best said as 'combine the siloed 'blueprints' and 'bugs' into a single unified workflow with the ability to move from one focus to another as may make sense' | 20:38 |
| lifeless | we're not going to throw away things folk use | 20:38 |
| james_w | right, but at the same time, you probably don't want to be adding more features to blueprints until it is done | 20:39 |
| james_w | and I have several requests for blueprint changes that I am being asked to bring to the stakeholders meeting | 20:40 |
| lifeless | uhm | 20:42 |
| lifeless | so adding new things in blueprints doesn't really make merging easier or harder. | 20:42 |
| lifeless | the vast bulk of existing stuff already exists. | 20:43 |
| rockstar | thumper, I wish mumble was happier in New Zealand. | 20:43 |
| thumper | rockstar: actually | 20:43 |
| lifeless | so I would say that there's no particular change to worry about vis a vis requested work | 20:43 |
| thumper | rockstar: can we try that here? | 20:43 |
| rockstar | thumper, sure, one sec. | 20:43 |
| thumper | rockstar: I'm using a different provider | 20:43 |
| lifeless | thumper: whom | 20:43 |
| thumper | lifeless: WIC (I'm hotdesking at the centre for innovation) | 20:44 |
| thumper | rockstar: mumble seems confused | 20:45 |
| rockstar | thumper, looks like it. | 20:45 |
| * thumper tries again | 20:46 | |
| thumper | rockstar: trying to tell mumble to use pulse and it shits itself | 20:46 |
| rockstar | thumper, yeah, that's how it started to feel about my USB headset. | 20:47 |
| rockstar | thumper, you should try blowing away your mumble config and starting over. | 20:48 |
| thumper | rockstar: yeah, where is that kept? | 20:48 |
| rockstar | thumper, no idea. ~/.config/mumble? | 20:48 |
| * thumper blew away ~/.config/Mumble | 20:49 | |
| thumper | mumble is working well here | 20:55 |
| jelmer | lifeless: *nod* | 21:03 |
| wgrant | jelmer: Evening. | 21:08 |
| jelmer | wgrant: hey - your branch failed to merge. I've been meaning to look at resolving the conflict but other things have been interrupting me all day. | 21:11 |
| wgrant | jelmer: Ah. I'll fix that. | 21:13 |
| lifeless | jelmer: if you can simply assert that deploying bug 627608 to the nodowntime alias won't break anything, we can qa-ok the bug- it owuld be ok to deploy. | 21:19 |
| _mup_ | Bug #627608: Got a 401 on a fresh purchase <qa-needstesting> <Software Center Agent:Fix Released> <Soyuz:Fix Committed by michael.nelson> <software-center (Ubuntu):Fix Released> <https://launchpad.net/bugs/627608> | 21:19 |
| lifeless | jelmer: but I can't tell if thats true or false | 21:19 |
| wgrant | It's fine to go anywhere except germanium. | 21:20 |
| wgrant | But it's not that hard to QA... | 21:21 |
| lifeless | wgrant: pls help :) | 21:21 |
| wgrant | Perhaps I should have added "for those with DF access right now" | 21:22 |
| wallyworld | rockstar: abentley: thumper: standup? | 21:22 |
| thumper | mumble is working 100% for me here | 21:22 |
| thumper | wallyworld: we are doing it right now | 21:22 |
| rockstar | wallyworld, mumble! | 21:22 |
| thumper | wallyworld: on mumble | 21:22 |
| lifeless | wgrant: its important to get the necessary discipline to qa on qastaging. | 21:23 |
| lifeless | wgrant: how would one qa it | 21:23 |
| wallyworld | i haven't got mumble installed | 21:23 |
| rockstar | wallyworld, sudo apt-get install mumble | 21:24 |
| wallyworld | rockstar: yep, doing it right now | 21:24 |
| wgrant | lifeless: I see a few methods. 1) Add a sleep to the script to make the window for adding a new token long enough to actually do it. 2) Hack the finish time in the DB to make that window longer. 3) Hack the token creation time in the DB. or 4) Be really really quick and activate a subscription at just the right time. | 21:25 |
| lifeless | wgrant: so, disable the script. Add lots of PPAs. Say 20. Get up to the last screen in adding another ppa. run the script and add the ppa | 21:25 |
| wgrant | lifeless: A token is created when a user activates their subscription, not on PPA creation. | 21:26 |
| wgrant | But yes. | 21:26 |
| wallyworld | rockstar: so what server do i connect to? any? | 21:27 |
| lifeless | wgrant: ah | 21:27 |
| lifeless | wgrant: so those question are for verification | 21:27 |
| lifeless | wgrant: I want validation - is it safe to deploy, not is the bug fixed. | 21:28 |
| lifeless | wgrant: sounds like activating a private PPA token on qastaging will do. | 21:28 |
| wgrant | lifeless: Ah, I didn't know that that was also unknown. | 21:28 |
| rockstar | wallyworld, there's a wiki page. One sec. | 21:28 |
| wgrant | lifeless: You'd need to do that then run the token generation script, which qastaging probably doesn't have configs for. | 21:28 |
| lifeless | wgrant: ok, lets set it up | 21:29 |
| lifeless | I wonder if I can make a p3a | 21:29 |
| lifeless | actually, I can probably nuke and refresh a token | 21:29 |
| lifeless | the reset passwor dbutton, right ? | 21:30 |
| wgrant | That should do it. | 21:30 |
| * wgrant checks. | 21:30 | |
| wgrant | Yeah. | 21:31 |
| lifeless | ok | 21:31 |
| wgrant | That'll work. | 21:31 |
| lifeless | I've reset it | 21:31 |
| lifeless | now, we check by looking in the htaccess, yeah? | 21:31 |
| wgrant | Now we need to run cronscripts/generate-ppa-access.py, and watch what explodes. | 21:31 |
| lifeless | mbarnett: ^ | 21:31 |
| lifeless | on qastaging | 21:31 |
| wgrant | It has no interesting flags. | 21:31 |
| mbarnett | so, a vanilla run of that with a bunch of -vvvvvvv s | 21:32 |
| wgrant | But it will need config, I suspect. | 21:32 |
| mbarnett | ? | 21:32 |
| wgrant | Right. | 21:32 |
| wgrant | As many -v s as you can muster! | 21:32 |
| mbarnett | i can mimic how it runs on prod | 21:32 |
| wgrant | How does it run on prod? | 21:32 |
| lifeless | very slowly | 21:33 |
| lifeless | boom-tish | 21:33 |
| mbarnett | give me a sec to extract myself from this interview, will be literally one minute | 21:33 |
| lifeless | mbarnett: no panic | 21:33 |
| lifeless | mbarnett: thank you | 21:33 |
| wallyworld | rockstar: connected now. what channel? | 21:34 |
| rockstar | wallyworld, there's a code channel. | 21:34 |
| mbarnett | cronscripts/generate-ppa-htaccess.py -vv | 21:35 |
| mbarnett | is all we do on prod | 21:35 |
| wallyworld | rockstar: i've added myself to that already | 21:35 |
| lifeless | wgrant: so mbarnett runs that. | 21:38 |
| lifeless | wgrant: then what | 21:38 |
| wgrant | lifeless: Find out where the qastaging config puts private PPAs. | 21:40 |
| wgrant | (personalpackagearchive.private_root is the config key) | 21:40 |
| mbarnett | here is the run on qastaging: http://pastebin.ubuntu.com/524684/ | 21:41 |
| lifeless | whats the config key ? | 21:41 |
| lifeless | wgrant: e.g. 'where do I look' | 21:41 |
| wgrant | lifeless: I just said. | 21:42 |
| lifeless | wgrant: sorry, link futzing | 21:42 |
| wgrant | Yay, DB config fail. | 21:42 |
| wgrant | Also, that librarian looks wrong. | 21:43 |
| lifeless | mbarnett: did you run that on 'staging' or 'qastaging' | 21:43 |
| wgrant | Half is qastaging, half is staging. | 21:43 |
| wgrant | Huh. | 21:43 |
| mbarnett | lifeless: i ran it out of qastaging | 21:45 |
| lifeless | mbarnett: -odd- | 21:46 |
| lifeless | mbarnett: so, we're missing a user in the qastaging db - the generatehtppaccess (spelling?) user | 21:47 |
| lifeless | mbarnett: and it seemed to be speaking to the wrong librarian; could you confirm the production configs are @ rev 152? | 21:47 |
| wgrant | lifeless: But it's also using the staging librarian and half of staging's zcml. | 21:47 |
| lifeless | wgrant: possibly stale pyc files | 21:47 |
| lifeless | wgrant: before we -panic- | 21:47 |
| wgrant | Heh. | 21:48 |
| lifeless | wgrant: possibly not, of course. | 21:48 |
| mbarnett | lifeless: sorry, looking now | 21:50 |
| mbarnett | too many things at once! :) | 21:50 |
| mbarnett | nope, 151 | 21:51 |
| lifeless | mbarnett: please pull 152 in, it fixes qastagin to use the right librarian. And we need to setup and configure that user. | 21:52 |
| mbarnett | so, can't actually just pull there. have to figure out the best way to get 152 there | 21:53 |
| mbarnett | branched it locally, pushing it over now | 21:56 |
| wgrant | jelmer: I pushed the conflict resolution. Could you try to land it again, please? | 22:01 |
| mbarnett | haha, that was a lot of work for a 1 line change... i probably should have just taken a look at the diff between those two revnos | 22:01 |
| lifeless | gary_poster: so, I've looked at one of those bugs | 22:07 |
| lifeless | gary_poster: I'm going to stare a little harder and see if I can spot any potential bugs; I need to grab the python 2.6 code first though | 22:08 |
| jelmer | lifeless: sure, I'm looking into a different generate-ppa-htaccess.py bug at the moment | 22:11 |
| jelmer | wgrant: Sure | 22:11 |
| mbarnett | lifeless: the configs are updated (with that 1 line change). will just take a bit of figuring to get the database user configured properly | 22:11 |
| lifeless | mbarnett: kk | 22:12 |
| wgrant | jelmer: Thanks. | 22:14 |
| gary_poster | thanks lifeless | 22:41 |
| lifeless | gary_poster: no probs | 22:47 |
| lifeless | gary_poster: it looks like an interpreter shutdown issue to me, IMBW | 22:47 |
| gary_poster | lifeless: but as mwhudson said, that would happen after the LOSAs had run ``stop``--so I'd guess that what we are seeing is in phase two of the problem | 22:49 |
| gary_poster | and we don't have visibility on phase 1 | 22:49 |
| gary_poster | so one debugging step is to ask the losas to run the gdb thread script on hung processes *before* they try calling stop | 22:49 |
| gary_poster | will pass that to them on -ops... | 22:50 |
| lifeless | gary_poster: well | 22:50 |
| lifeless | that presumes that the problem exists before shutdown | 22:50 |
| lifeless | gary_poster: if its not /hung/, the shutdown may be the problem. | 22:50 |
| lifeless | remember that we're doing deploys more often | 22:50 |
| gary_poster | in the bug description it is after a nagios check | 22:51 |
| lifeless | right | 22:51 |
| lifeless | the sequence I'm thinking is this | 22:51 |
| gary_poster | we can verify that the nagios had been successful previously | 22:51 |
| lifeless | we deploy 11808 | 22:51 |
| lifeless | some servers don't die properly | 22:51 |
| lifeless | nagios goes 'hey' | 22:51 |
| lifeless | we find rev 11793 still running | 22:51 |
| lifeless | and wedged | 22:51 |
| gary_poster | ok reasonable hypothesis | 22:52 |
| lifeless | it was reported on the 1st | 22:52 |
| lifeless | the day we deployed 11811 | 22:52 |
| lifeless | it has rev 11793 itself | 22:53 |
| gary_poster | in your hypothesis, right? We don't have evidence of that | 22:53 |
| LPCIBot | Project devel build (177): STILL FAILING in 3 hr 59 min: https://hudson.wedontsleep.org/job/devel/177/ | 22:54 |
| gary_poster | if this hypothesis were correct, then I think the following would not be true: | 22:54 |
| gary_poster | nagios saw all servers alive after a roll out | 22:54 |
| lifeless | agreed | 22:54 |
| lifeless | gary_poster: we have the deploy log | 22:55 |
| lifeless | gary_poster: which isn't accruate to the minute sadly. Its on LPS. | 22:55 |
| gary_poster | right | 22:55 |
| gary_poster | do we have a nagios log? | 22:55 |
| lifeless | spm: ^ :) | 22:55 |
| mwhudson | interpreter teardown is a bucket of mess | 22:56 |
| gary_poster | heh | 22:56 |
| lifeless | mwhudson: its a shame we can't see thread 0 :) | 22:57 |
| gary_poster | if nagios did show all servers up before this, then we're back to asking losas to try to gather thread info before they issue init.d stop | 22:57 |
| lifeless | mwhudson: could it perhaps have exited already? whilst holding the HEAD_LOCK or something crazy like that ? | 22:57 |
| mwhudson | i've semi-seriously advocated running appservers in vms and just terminating the vm when you want it to go away before... | 22:57 |
| mwhudson | lifeless: there isn't a thread 0 usually? | 22:57 |
| gary_poster | heh | 22:57 |
| lifeless | mwhudson: I thought gdb was 0-indexed | 22:57 |
| mwhudson | lifeless: as i said in the bug report, the rest of the traceback from thread 1 would be interesting | 22:57 |
| * gary_poster thinks the question mark in mwhudson's statement was misleading :-P | 22:58 | |
| lifeless | mwhudson: I certainly can't see a thread with interpreter bootstrap present | 22:58 |
| mwhudson | lifeless: pygdb's backtrace walks though c until you hit python and then only displays python | 22:58 |
| lifeless | mwhudson: what is the 'stop script' you mean ? | 22:58 |
| lifeless | mwhudson: can you fix that ? | 22:58 |
| mwhudson | lifeless: i guess | 22:58 |
| lifeless | mwhudson: pretty please with a We Can Fix This Then on top ? | 22:59 |
| mwhudson | lp:pygdb is team-owned fwiw :-) | 22:59 |
| mwhudson | but it's a bit write only code | 22:59 |
| lifeless | mwhudson: I can context switch into it if needed, but I'd *prefer* to not bootstrap on that this week. | 22:59 |
| gary_poster | according to flacoste this enters drop-everything mode after the next strike | 23:00 |
| gary_poster | but not yet | 23:00 |
| mwhudson | lifeless: the hard part is the heuristics | 23:00 |
| gary_poster | I should run go do family things. lifeless, should I pass the pygdb-before-init.d-stop request to the losas, and the nagios question, via RT, or may I leave it with you? | 23:01 |
| * gary_poster needs to go. Younger son is playing the paper-tube-o-phone | 23:03 | |
| gary_poster | thank you and good night | 23:03 |
| mwhudson | ooh, i have one good heuristic actually | 23:04 |
| spm | lifeless: heyo. just gimme a sec to get up to speed wrt context/backlog/handover | 23:07 |
| mwhudson | huh, i want a multiset for about the first time ever | 23:13 |
| spiv | mwhudson: collections.defaultdict(lambda: 0) ? | 23:14 |
| mwhudson | spiv: well yeah + some sugar on that | 23:15 |
| mwhudson | lifeless: r52 of lp:pygdb should be more useful in this case | 23:16 |
| mwhudson | spiv: did you know that collections.defaultdict(int) is the same as that? :) | 23:16 |
| spiv | mwhudson: I choose not to know that ;) | 23:19 |
| spiv | I find it a bit weird for immutable, non-container objects to have constructors that make "empty" or "zero" instances of themselves. | 23:20 |
| spiv | so list() and dict() seem reasonable, but int() and str() make me uncomfortable. | 23:20 |
| spiv | And tuple()... I could go either way on that ;) | 23:20 |
| spiv | mwhudson: how does pygdb relate to the shinier python-gdb stuff that's builtin in maverick? | 23:22 |
| spiv | (I mean shinier relative to previous ubuntus, rather than implying it's shinier than than pygdb) | 23:23 |
| mwhudson | spiv: it works on lucid i guess :-) | 23:25 |
| mwhudson | i don't really know tbh, it's all a bit different | 23:25 |
| mwhudson | in particular it drives gdb from outside | 23:25 |
| lifeless | spm: we want to start using a newer pygdb to get backtraces from hung appserverfs. | 23:30 |
| lifeless | spm: and, we think deploys trigger this, so we want to deliberately look for it after deploy | 23:31 |
| spm | lifeless: why do you feel that deploys cause this? | 23:31 |
| spm | (sure on the pygdb, np there) | 23:31 |
| lifeless | spm: the most recent two show rev 11793 in their trace and come from the day we deployed 11811 | 23:32 |
| lifeless | spm: and the internals smell of interpreter shutdown. | 23:32 |
| lifeless | so we think its all a dupe of the 'apservers not shutting down' bug | 23:33 |
| spm | hmmm. one was actually dead for quite a long period... /me refreshes memory with times/dates | 23:33 |
| lifeless | spm: that would fit quite well actually. | 23:34 |
| lifeless | mwhudson: thank you | 23:34 |
| spm | yes lpnet11, that was a dead on a monday (here), but istr had been dead for ~ 12-16 hours. | 23:34 |
| lifeless | spm: hmmm | 23:34 |
| lifeless | well the other requested thing | 23:34 |
| spm | yeah - it doesn't quite fit. | 23:34 |
| spm | sure. rev 52? | 23:35 |
| lifeless | is to start getting a pygdb run *before* shutdown on deployments | 23:35 |
| mwhudson | lifeless: np, it was easy when i actually switched my brain on a bit | 23:35 |
| spm | Oooo kat | 23:35 |
| spm | kay even | 23:35 |
| lifeless | however | 23:35 |
| lifeless | I think thats a problem | 23:35 |
| lifeless | it will be a lot of data | 23:35 |
| mwhudson | pygdb is fairly slow | 23:35 |
| spm | mwhudson: it's overrated - brain switch on | 23:35 |
| lifeless | that we won't look at | 23:35 |
| lifeless | and its slow | 23:35 |
| mwhudson | generating a core and pygdb-ing that will be less invasive | 23:35 |
| lifeless | so I'd like to go with 'it looks like shutdown to us, and the new pygdb will tell us more' | 23:36 |
| mwhudson | but will obviously use a lot more disk, if only temporarily | 23:36 |
| spm | have you seen how big those core files are?? :-) | 23:36 |
| lifeless | spm: hey, do you have the core from lpnet11 ? | 23:36 |
| mwhudson | i guess dumping the core takes a while too | 23:36 |
| spm | lifeless: probably... lemme check. | 23:36 |
| lifeless | spm: if so please run rev52 against it | 23:36 |
| lifeless | we should get more data immediately. | 23:36 |
| spm | lifeless: I don't believe I did on yesterdays(??), but pretty sure I did on monday | 23:36 |
| spm | bugger no. I didn't. most recent is 2010-10-18 03:42 lpnet12-2010-10-18.core.24795 | 23:37 |
| spm | I'm 8sure* I did take one this week tho... poking... | 23:38 |
| lifeless | give it a shot | 23:38 |
| lifeless | its more likely that we have one problem than two causing hangs. | 23:38 |
| spm | hrm. might have been a codebounce hang that I dumped. | 23:39 |
| spm | bother | 23:39 |
| lifeless | lpnet12 should be useful to analyse | 23:39 |
| lifeless | rev52 will show us the entry point to the app and hopefully where its at outside of python frames | 23:39 |
| spm | nod | 23:39 |
| spm | err - how do I run vs a core? backtrace <corefile> ? | 23:40 |
| lifeless | mwhudson: ^ | 23:41 |
| mwhudson | spm: -c $core | 23:41 |
| * spm won't point to --help giving a smash. cause that'd be rude. :-P | 23:41 | |
| mwhudson | spm: right | 23:41 |
| spm | heh | 23:41 |
| mwhudson | this was hardly written using principles of user interaction design | 23:42 |
| spm | I can tell | 23:42 |
| mwhudson | (or software engineering, come to that) | 23:42 |
| lifeless | mwhudson: punching is a principle. | 23:42 |
| mwhudson | lifeless: only one that was followed by accident in this case | 23:42 |
| spm | lifeless: mwhudson: https://pastebin.canonical.com/39285/ | 23:42 |
| mwhudson | um | 23:43 |
| mwhudson | that's not very useful, is python2.6-dbg installed on that machine? | 23:43 |
| spm | ii python2.6-dbg 2.6.5-1ubuntu6 | 23:44 |
| spm | that dump is oldish - dunno if that messes things at all | 23:44 |
| spm | 2-3 weeks old. | 23:44 |
| mwhudson | shouldn't do | 23:44 |
| spm | I can always gcore something now, and we can verify? | 23:44 |
| * lifeless sings the edge is dead happy song | 23:44 | |
| spm | :-) | 23:44 |
| mwhudson | i'm not sure there's much point | 23:44 |
| spm | edge is dead, long live edge! | 23:44 |
| mwhudson | the modification i made is fairly single-purpose: give more info when something hangs in a __del__ method | 23:45 |
| lifeless | mwhudson: so I'd really quite like ot see the stack back to main() | 23:46 |
| lifeless | mwhudson: the python lines are ok, but every interpreter hang previously I've solved purely on the C frames | 23:47 |
| lifeless | mwhudson: if we have to *choose*, lets get the full C frame. | 23:47 |
| lifeless | mwhudson: "'NoneType' object has no attribute 'exception'",), <traceback at remote 0x2b44aceb5128>), kw=0x0) from ../Python/ceval.c | 23:47 |
| lifeless | mwhudson: \o/ | 23:48 |
| mwhudson | lifeless: if the c traceback is enough, thread apply all bt is all you need? | 23:48 |
| lifeless | mwhudson: maybe just doing that in paralle. | 23:48 |
| lifeless | spm: can you do thread apply all bt in that core too ? | 23:48 |
| lifeless | spm: also, we need to start gathering the appserver log for the time that it hung | 23:50 |
| lifeless | there may well be things like | 23:50 |
| lifeless | 'unhandled exception in thread xxx' in stderr | 23:50 |
| spm | lifeless: hrm. likely pebkac here. gdb <core> ; thread apply all bt<cr> ?? | 23:50 |
| lifeless | yes | 23:50 |
| spm | hrm. "/home/launchpad/lpnet12-2010-10-18.core.24795": not in executable format: File format not recognised | 23:51 |
| lifeless | gdb `which python` <core> | 23:51 |
| spm | ah | 23:51 |
| spm | https://pastebin.canonical.com/39286/ | 23:52 |
| spm | curious. it even looks the same, with the ??'s. | 23:52 |
| lifeless | garh | 23:57 |
| lifeless | thanks | 23:57 |
| lifeless | warning: core file may not match specified executable file. | 23:57 |
| mwhudson | lines like "#7 0x0000000000000010 in ?? ()" don't look very plausible to me | 23:58 |
| mwhudson | is it possible that this core file predates the update to lucid, or something? | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!