| lifeless | is there a wiki page listing the performance problem-and-resolutions folk have run into before ? | 00:20 |
|---|---|---|
| jelmer | lifeless, there should be, but I don't think there is. | 00:33 |
| lifeless | jelmer: https://dev.launchpad.net/Performance has one now | 00:53 |
| jelmer | lifeless: Yeah, I saw it. Thanks, also for that summary of improvements you had made and how/why earlier. | 00:54 |
| lifeless | which summary? | 00:55 |
| jelmer | the one you sent at the end of your day yesterday | 00:55 |
| jelmer | well, it wasn't really a summary I guess | 00:55 |
| jelmer | English fail. | 00:56 |
| lifeless | :) | 00:57 |
| lifeless | the the 'some thoughts on performance' ? | 00:57 |
| jelmer | lifeless: I meant "Performance tuesday - a few notes" | 00:59 |
| jelmer | which I pretty much read as the summary of your day | 00:59 |
| jelmer | and now I'm getting sleep before I do more gibbering :-) | 00:59 |
| lifeless | :) | 01:00 |
| lifeless | I'm glad its useful | 01:01 |
| lifeless | that was two days ago :> | 01:01 |
| jelmer | lifeless: Heh, timezone fail. Tuesday was yesterday. >-) | 01:02 |
| mwhudson | lies | 01:02 |
| lifeless | jelmer: 36 hours then :P | 01:02 |
| lifeless | wgrant: so | 01:15 |
| lifeless | wgrant: I'm thinking of writing an extension to cachedproperty | 01:16 |
| lifeless | to register the caches in __cachedproperties__ | 01:16 |
| lifeless | and a SQLBase base class storm invalidation hook to clear that out. | 01:16 |
| lifeless | wgrant: ignoring performance, does that raise any red flags for you? | 01:16 |
| lifeless | mwhudson: ^ also | 01:16 |
| thumper | lifeless: a bit of advice if you can... | 01:17 |
| thumper | I want my tests to all flush the store to make sure database constraints are triggered | 01:18 |
| thumper | should I just add a cleanup from the test setup? | 01:18 |
| thumper | is that a reasonable thing to do? | 01:18 |
| thumper | mwhudson_: comments on the above? | 01:19 |
| wgrant | lifeless: On what would it be invalidated? | 01:20 |
| thumper | is an error in cleanup execution still considered a failure? | 01:21 |
| wgrant | Isn't the entire object already invalidated on the transaction boundary? | 01:21 |
| thumper | wgrant: removed from the storm cache, but not invalidated | 01:22 |
| thumper | AFAIK | 01:22 |
| wgrant | thumper: So if I hold a reference to an object, it will stay around? | 01:23 |
| wgrant | That seems bad. | 01:23 |
| thumper | used in many places though | 01:23 |
| thumper | like scripts | 01:23 |
| wgrant | Hmm, true. | 01:23 |
| lifeless | thumper: an error in cleanups is an error in the test, yes. | 01:26 |
| lifeless | wgrant: it will be invalidated when storm invalidates the object | 01:26 |
| lifeless | wgrant: I need to check exactly what triggers that | 01:26 |
| lifeless | the storm cache is flushed on transactions but only in the webapps | 01:27 |
| lifeless | scripts and tests don't do that | 01:27 |
| lifeless | storm invalidates objects before reusing them | 01:27 |
| wgrant | lifeless: I'd really like the cache to not be populated by an attribute access. | 01:27 |
| wgrant | Because that makes it happen *everywhere*. | 01:27 |
| lifeless | the reason it has an object cache is so that scripts and the like with long running references don't end up with aliasing problems | 01:27 |
| lifeless | wgrant: I'd like attribute accesses to always be cheap. Because they happen everywhere. | 01:27 |
| lifeless | wgrant: we're /dying/ from this today. | 01:28 |
| lifeless | wgrant: Pages are doing thousands of queries | 01:28 |
| lifeless | and prejoining is effectively useless | 01:28 |
| lifeless | because: | 01:28 |
| lifeless | - its too greedy (pay the cost everywhere) | 01:28 |
| wgrant | lifeless: In most cases it won't help. | 01:28 |
| lifeless | - its too specific (fails on many non-collection lookups) | 01:28 |
| lifeless | wgrant: its a tool when used with tuple queries that works very well | 01:28 |
| wgrant | lifeless: @cachedproperty normally only significantly saves queries when you do a prejoin beforehand. | 01:29 |
| lifeless | thumper: yes, a store flush seems reasonable where you are doing updates/inserts | 01:29 |
| wgrant | If you're doing a prejoin and you want this behaviour, *then* populate the cache. | 01:30 |
| lifeless | wgrant: that requires the cache to exist | 01:30 |
| wgrant | Doing it implicitly without checking every callsite is a recipe for disaster. | 01:30 |
| lifeless | wgrant: Can you quantify that at all? | 01:30 |
| lifeless | I don't like the idea of a 'sometimes its cached' decorator - it will be confusing as hell when looking at code | 01:31 |
| lifeless | 'is this cached? I don't know? maybe?' bwah | 01:31 |
| lifeless | wgrant: let me add some data here: | 01:31 |
| lifeless | - we *already* have cachedproperty on model objects | 01:32 |
| wgrant | We do. | 01:32 |
| lifeless | - we *already* have adhoc hooks of that into storm | 01:32 |
| lifeless | I'm proposing to systematise it so that we don't have accidental failures to flush those caches when the backing data is recalculated | 01:33 |
| lifeless | you seem to be objecting to an improvement, which confuses me | 01:33 |
| * thumper afk for lunch | 01:33 | |
| wgrant | Well, you also a couple of days ago proposed to immediately start caching tonnes of attributes. | 01:34 |
| wgrant | I don't object to your invalidation improvement. | 01:34 |
| lifeless | yes, which is btw no worse that prejoins AFAICT | 01:34 |
| lifeless | we've been around that discussion | 01:35 |
| wgrant | Assuming that DB triggers/views don't play tricks, prejoining is perfectly safe, since it only works for references. | 01:36 |
| lifeless | if you're talking risk, thats a fairly big one :) | 01:36 |
| wgrant | Hm? | 01:37 |
| lifeless | we had a storm bug with triggers just this cycle | 01:37 |
| lifeless | which took a day to debug | 01:37 |
| lifeless | more-or-less | 01:37 |
| wgrant | Right. But triggers are well-known and few. | 01:38 |
| wgrant | They are part of the model. | 01:38 |
| lifeless | wgrant: here's another way to think about it | 01:38 |
| lifeless | on one hand we have death by sql | 01:38 |
| lifeless | on the other we have a) cachedproperty and b) some huge handwaving I've done with nothing concrete and c) nil, zip, nada, nothing | 01:39 |
| lifeless | I'm -very- interested in alternative approaches without the potential downside of cachedproperty. | 01:39 |
| wgrant | Or d) cachedproperty populated by prejoins only when we explicitly want it | 01:39 |
| lifeless | prejoins don't help. | 01:39 |
| wgrant | How are you going to reduce the query count without a prejoin? | 01:40 |
| lifeless | So please stop talking about them - I'm starting to think you don't understand the problem or something, which I *know* is wrong, because you do. | 01:40 |
| wgrant | An uncached cachedproperty will still cause a query. | 01:40 |
| lifeless | wgrant: lets define prejoin. I mean 'lp.services.database.prejoin' when I say prejoin. | 01:40 |
| lifeless | I don't mean the general tactic of prepopulating things. | 01:40 |
| wgrant | Ah. | 01:40 |
| wgrant | That's the problem, then. I mean what you did with /participations or whatever it was. | 01:41 |
| lifeless | ok | 01:41 |
| wgrant | Like prejoining (in that we only return the first element), but also populating caches behind the scenes. | 01:41 |
| lifeless | lets call that prepopulating | 01:41 |
| wgrant | OK. | 01:41 |
| wgrant | That technique is good. I need it for some publisher optimisations. | 01:41 |
| wgrant | But having to turn everything into a cachedproperty for that sucks. | 01:42 |
| wgrant | There are some parts of the publisher where I cannot safely do that. | 01:42 |
| wgrant | I need them to only be populated when I ask. Not when I just request the attribute. | 01:42 |
| lifeless | why | 01:42 |
| lifeless | I'm asking for details here | 01:43 |
| lifeless | not to be silly | 01:43 |
| wgrant | eg. SPR.files. The uploader needs to add things to that, then iterate over them later. | 01:43 |
| wgrant | But the publisher needs to get them all in one query. | 01:43 |
| wgrant | If I make it a cachedproperty, I can prepopulate it in the publisher. | 01:43 |
| lifeless | and you can update the cache when you add things to it | 01:43 |
| wgrant | But why? | 01:43 |
| wgrant | In the uploader case, I don't want it cached! | 01:44 |
| wgrant | I shouldn't have to invalidate a cache that serves no purpose. The cache shouldn't exist. | 01:44 |
| lifeless | no? You *want* to go back to the DB when you iterate? | 01:44 |
| wgrant | Possibly not. But imposing a cache invalidation problem on every object modification in the entire application is... not good. | 01:44 |
| lifeless | so there are a few broad approaches here | 01:45 |
| wgrant | if we were writing this from scratch, perhaps. | 01:45 |
| lifeless | *) different attributes - some cached, some not. Make them visually distinctive | 01:45 |
| wgrant | But so much code has been written that assumes that attributes behave sanely. | 01:45 |
| lifeless | *) single attributes which are sometimes cached and sometimes not, with a switch to toggle. | 01:45 |
| wgrant | We should not be immediately breaking assumptions made by a few hundred thousand lines of code. | 01:46 |
| lifeless | wgrant: I understand you feel strongly about this, but you're inflating your argument with hyperbole; its not helping us design a solution that addresses what you want and what I want. | 01:46 |
| lifeless | any one attribute can be looked at and assesed. | 01:46 |
| lifeless | the surface area for any one change is not 'the entire code base' | 01:47 |
| wgrant | But we want a solution that can be widely applied. | 01:47 |
| lifeless | and the entire code base is < 200K *anyway* | 01:47 |
| lifeless | wgrant: sure, so lets get back to talking about it. | 01:47 |
| lifeless | I'm just saying, going off the deep end shuts down conversation and brainstorming | 01:47 |
| wgrant | lifeless: There's like 400KLOC of Python. | 01:47 |
| lifeless | 270K including tests. | 01:48 |
| lifeless | and for assessing the risk tests are not relevant: they are a problem for whomever puts a specific branch together, its the headache when we silently break production that really worries me. | 01:49 |
| wgrant | My "find -name '*.py' | xargs wc -l" says 511K, but anyway. | 01:50 |
| wgrant | Right. | 01:50 |
| lifeless | I used sloccount | 01:51 |
| lifeless | on lib/lp | 01:51 |
| lifeless | and then lib/canonical | 01:51 |
| lifeless | because including all of subvertpy etc didn't make sense to me | 01:51 |
| lifeless | anyhow | 01:52 |
| wgrant | Well, this is a clean branch without sourcecode linked. | 01:52 |
| lifeless | theres code | 01:52 |
| wgrant | But yes, anwyay. | 01:52 |
| lifeless | it does stuff | 01:52 |
| lifeless | and many many things it does are terribly slow because we do incremental DB access | 01:52 |
| lifeless | So back to approaches | 01:52 |
| wgrant | Certainly. | 01:52 |
| lifeless | excluding radical ones - I want to do them, just *after* we are in a decent place vis-a-vis performance. | 01:53 |
| lifeless | *) different attributes - some cached, some not. Make them visually distinctive | 01:53 |
| lifeless | *) single attributes which are sometimes cached and sometimes not, with a switch to toggle. | 01:53 |
| lifeless | *) attributes are never cached, add cached methods | 01:53 |
| lifeless | *) attributes are always cached, add noncached methods | 01:53 |
| lifeless | *) Alternatives go here <...> | 01:54 |
| lifeless | what you appear to be arguing 8for* is the second one. | 01:54 |
| wgrant | None are pretty. | 01:54 |
| lifeless | right | 01:54 |
| lifeless | I *totally* agree. | 01:54 |
| wgrant | I'd prefer it if we could add a caching wrapper around objects. | 01:54 |
| wgrant | But that is impossible. | 01:55 |
| lifeless | well, its possible, but I think it would give us two problems :) | 01:55 |
| lifeless | (its python, with sufficient will most anything is possible) | 01:55 |
| wgrant | Heh. | 01:55 |
| wgrant | But anyway, that leaves us with mutating the global Storm objects. | 01:55 |
| lifeless | yes | 01:55 |
| lifeless | or a lookaside weak dict | 01:56 |
| lifeless | which is equivalent and slower. | 01:56 |
| lifeless | so lets exclude that | 01:56 |
| wgrant | Well, regardless we have to modify the global objects' behaviour. | 01:56 |
| lifeless | sure | 01:56 |
| lifeless | so here are some constraints I have | 01:57 |
| lifeless | I want the *default* to be fast. | 01:57 |
| lifeless | or if I can't get that, that when someone makes something fast, it behaves decently in a very broad scope. | 01:57 |
| lifeless | I want this because most of our pages are slow, and most of them do not mutate things so most of the time any of the implementations we've thought of should be ok. | 01:58 |
| lifeless | at *worst* I want it to be very easy to take something from slow to fast | 01:59 |
| lifeless | secondly, I do not want heisenbugs. | 01:59 |
| mwhudson_ | thumper: yeah, sounds basically ok | 01:59 |
| mwhudson_ | thumper: errors in tearDown or cleanups fail the test | 01:59 |
| lifeless | Folk debugging the code should not need to see object state to tell if something is or isn't caching. | 02:00 |
| lifeless | Its ok that they need to read the code (but not ideal) | 02:00 |
| lifeless | its not ok if they need a memory dump from the live server | 02:00 |
| wgrant | Thaat's true. | 02:01 |
| lifeless | A criticism I have of the solution you appear to favour is that its likely to need that memory dump, if - e.g. a kept storm object has the 'cache this attr now' turned on and not reset (due to a bug) | 02:01 |
| lifeless | its vulnerable to cascade failures | 02:01 |
| lifeless | always on | 02:01 |
| mwhudson_ | lifeless: are you sure storm doesn't flush it's per-thread cache on transacton boundaries btw? | 02:02 |
| lifeless | or always off | 02:02 |
| wgrant | It is. | 02:02 |
| lifeless | mwhudson_: check publication/webapp.py or whatever. | 02:02 |
| lifeless | mwhudson_: we do it | 02:02 |
| wgrant | But it means we need to audit *lots* of code. | 02:02 |
| lifeless | line 699 | 02:02 |
| mwhudson_ | lifeless: you're probably right, but just because we do something by hand doesn't mean it doesn't happen automatically too :-) | 02:02 |
| lifeless | mwhudson_: oh, so yes, I am sure. | 02:03 |
| lifeless | mwhudson_: or all our doctests would fail. | 02:03 |
| mwhudson_ | ok | 02:03 |
| mwhudson_ | um | 02:03 |
| === mwhudson_ is now known as mwhudson | ||
| lifeless | because the object outside userbrowser lookups would be different | 02:03 |
| lifeless | and hilarity would ensue with aliasing issues and the like | 02:03 |
| lifeless | if thread_name != 'MainThread' or name.endswith('-slave'): | 02:04 |
| lifeless | store.reset() | 02:04 |
| lifeless | is what we do, for exactly that reason. | 02:04 |
| mwhudson | aaah | 02:04 |
| lifeless | yes, yes you should run screaming. | 02:04 |
| mwhudson | hang on a sec | 02:05 |
| mwhudson | presumably we somehow don't call transaction.begin() (or moral equiv) at the start of testbrowser requests either | 02:05 |
| mwhudson | or we would have similar hilarity | 02:06 |
| lifeless | I haven't looked at exactly what testbrowser does | 02:06 |
| lifeless | it scares me that we avoid the publication machinery, given what is hooked into publication | 02:06 |
| * mwhudson confuzzles | 02:07 | |
| wgrant | lifeless: So, OK, your approach is OK if we are going to move to it throughout the application. | 02:09 |
| wgrant | But it's a fair bit of work. | 02:09 |
| lifeless | what do you think of having foo_uncached attributes where you need them ? | 02:10 |
| lifeless | or perhaps | 02:10 |
| lifeless | a dynamically generated getter | 02:10 |
| lifeless | @cachedproperty('bwah') | 02:10 |
| lifeless | def foo | 02:10 |
| lifeless | -> _bwah_cached cache attribute | 02:10 |
| lifeless | -> bwah_uncached lookup bypassing the cache | 02:11 |
| lifeless | it won't layer well | 02:11 |
| lifeless | if you have Foo referencing Bar and Bar has a thing you need uncached from Foo, you're ok. | 02:11 |
| lifeless | but if you have Quux->Foo->Bar, you're screwed if you try .foo.bar.thing | 02:12 |
| lifeless | OTOH you shouldn't do that anyhow :P | 02:12 |
| wgrant | Well. Let's think about the cases where we are going to need something uncached. | 02:12 |
| wgrant | 1) We start a new transaction. | 02:13 |
| wgrant | 2) We make a change. The method making the change should invalidate or update the cache. | 02:13 |
| wgrant | 3) DB triggers or views play tricks on us. | 02:14 |
| wgrant | Any others? | 02:14 |
| lifeless | 1) isn't strictly true, but its a damn sensible default | 02:15 |
| lifeless | 2) yes, agreed. | 02:15 |
| lifeless | 3) yes, agreed (and thats where storm broke this release too) | 02:15 |
| wgrant | 2) is difficult, particularly if we're doing mass-updates as it seems we might. | 02:16 |
| lifeless | if we're doing mass updates we shouldn't realise the objects in the first place | 02:16 |
| lifeless | the existing storm objects will get broken by that anyway, so its not a unique problem to cachedproperty decorators | 02:17 |
| wgrant | I guess that case is pretty much the same as 3) | 02:17 |
| wgrant | So 3) should be revised to changes in the DB, not going through the objects. | 02:17 |
| lifeless | functionally equivalent | 02:17 |
| lifeless | yes | 02:17 |
| lifeless | is DistroSeries:EntryResource:getBuildRecords fixed? | 02:24 |
| lifeless | wgrant: so, thats when we need uncached things | 02:24 |
| lifeless | wgrant: do we then say 'if you have a cache, and you invalidate it,the owning object invalidates the cache' | 02:25 |
| lifeless | e.g. Foo.newBar() -> Foo's Barcache is Foo's job to invalidate | 02:25 |
| wgrant | lifeless: It should be fixed, yes. | 02:25 |
| wgrant | lifeless: And I think that example is correct. | 02:26 |
| lifeless | I wonder if we can hook in to make a simple del attrname invalidate without wiping the cache logic | 02:28 |
| lifeless | we have about 40 uses of cachedproperty, in model code, in a simple check today | 02:32 |
| lifeless | including sourcepackagerelease | 02:32 |
| lifeless | wgrant: did you fix DistroSeries:EntryResource:getBuildRecords or is it my imagination ? | 02:34 |
| wgrant | lifeless: I'm pretty sure it's fixed. | 02:38 |
| wgrant | I mean, the query it executes now sucks 30x less. | 02:38 |
| jamesh | if you're doing mass updates via Storm's ResultSet.set() API, Storm should keep its own caches coherent | 02:39 |
| jamesh | if that helps. | 02:39 |
| lifeless | jamesh: does it trigger invalidate() ? | 02:44 |
| jamesh | lifeless: no, but it marks the columns you're updating as AutoReload (or the new values if the query is simple enough) | 02:45 |
| lifeless | ah | 02:45 |
| lifeless | simple would have to be darn simple ;) | 02:45 |
| jamesh | see the compile_python code in storm.expr for an idea of what counts as simple | 02:46 |
| lifeless | I just mean it has to either be explicit primary key values | 02:47 |
| lifeless | or all rows | 02:47 |
| lifeless | no? | 02:48 |
| jamesh | it can do more than that. | 02:48 |
| lifeless | I didn't think UPDATE returned the row ids | 02:48 |
| jamesh | if you do store.find(Foo, name="blah").set(x=42), it will update any cached Foo object with name=="blah" | 02:49 |
| lifeless | ok, thats nice | 02:49 |
| jamesh | it tries to create a python function based on the where conditions. If it can't, then it will invalidate the column on all cached Foo objects | 02:50 |
| lifeless | so | 02:51 |
| lifeless | we want to cache derived data | 02:51 |
| lifeless | like store.find(Foo, And(Foo.id==self.foo, Foo.status=="finished")) | 02:51 |
| lifeless | on Bar | 02:52 |
| lifeless | to avoid death-by-sql | 02:52 |
| jamesh | this sounds like you want a constrained version of Reference() | 02:52 |
| lifeless | + eager loading | 02:52 |
| lifeless | but only when we want it | 02:53 |
| jamesh | or are you after something more general? | 02:53 |
| lifeless | and very fine control | 02:53 |
| lifeless | jamesh: does storm have constrained references ? | 02:55 |
| jamesh | no | 02:55 |
| lifeless | heh | 02:55 |
| lifeless | https://dev.launchpad.net/Database/Performance | 02:55 |
| lifeless | is a first sketch starting to write up some of this | 02:55 |
| lifeless | also | 02:55 |
| lifeless | https://dev.launchpad.net/Performance may be interesting-though-I-bet-you-know-it | 02:56 |
| jamesh | IIRC, when I was optimising some soyuz views way back, I did one query to get all the pairs of objects and used lazr decorates(...) code to override the slow attribute in the view. | 02:59 |
| lifeless | yeah | 02:59 |
| lifeless | the problem with that is that the default is 'behave badly' :) | 03:00 |
| jamesh | well, the definition of good behaviour often depended on what the view was trying to do. | 03:01 |
| lifeless | thats true | 03:01 |
| lifeless | you don't want unnecessary columns included | 03:01 |
| lifeless | s/columns/tables/ | 03:01 |
| lifeless | what I meant is that when you have any model methods | 03:01 |
| lifeless | a delegated approach to fix things doesn't help the core object | 03:02 |
| jamesh | right. | 03:02 |
| lifeless | so you end up going through contortions to achieve that | 03:02 |
| jamesh | if you are working on this, don't treat storm as a black box though: improving Storm may be a better way of fixing your problem | 03:03 |
| jamesh | (we can always use more help) | 03:04 |
| lifeless | jamesh: oh, I'm not | 03:05 |
| lifeless | I spent some time talking with jkakar about this | 03:05 |
| jamesh | awesome | 03:05 |
| lifeless | my rough ideas there are: | 03:06 |
| lifeless | - stop subclassing to get objects - use a separate compile step, so we have clear separation of responsibilities | 03:06 |
| lifeless | - define a formal user accessible caching layer which can handle more complex algebras | 03:06 |
| jamesh | "subclassing to get objects"? | 03:07 |
| lifeless | we have classes that represent what storm gets from the DB *and* what we write code on | 03:07 |
| lifeless | thats confusing responsibilities | 03:07 |
| lifeless | its confusing because it becomes unclear whether foo.bar implies database access | 03:09 |
| lifeless | or local data only access | 03:09 |
| lifeless | so I want to structure things such that when you're in python land, there is no ambiguity | 03:09 |
| lifeless | put data access and updates alongside rather than mixed-in | 03:09 |
| lifeless | what we have is awesome for interactive use, ok for batch processing, and poor for web serving - IMNSHO | 03:10 |
| lifeless | I don't see value, from a performance perspective, in the attribute lookup facilities of Reference, ReferenceSet | 03:11 |
| lifeless | I may being overly cautious here. | 03:11 |
| lifeless | Its early days now | 03:12 |
| lifeless | but I'm thinking about how to move LP to a staged model where no DB access /can/ happen during template rendering | 03:12 |
| lifeless | and we have really really clear 'this is doing db access' now behaviour | 03:13 |
| lifeless | because as developers, its really hard to reason about performance when things are sometimes fast and sometimes slow. | 03:13 |
| jamesh | I know therve's twisted-integration branch tried to get things to a point where database access only happened at defined points. | 03:13 |
| lifeless | I think if you make all the objects readonly and don't use Reference or ReferenceSet, that it will be fine. | 03:14 |
| lifeless | push updates over to ResultSet | 03:14 |
| jamesh | that work has never really gotten to a point where it can be used for real work though. | 03:14 |
| lifeless | anyhow - this is long term stuff | 03:14 |
| lifeless | we can get a long way with cachedproperty and DecoratedResultSet | 03:14 |
| lifeless | but when we're down solidly under 5 seconds hard timeout, then I want to start looking at this. | 03:15 |
| lifeless | jamesh: https://lists.launchpad.net/launchpad-dev/msg04228.html shows where we're at - and most of these pages are driven by that sort of problem IME | 03:18 |
| lifeless | which is admitedly limited :) | 03:18 |
| lifeless | elmo: when you get up | 03:32 |
| lifeless | https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100712/20100811/ | 03:32 |
| lifeless | you can see a oops spike on a weekly basis | 03:32 |
| lifeless | elmo: can we determine whats going on there ? Perhaps its that backup issue we identified ? | 03:33 |
| lifeless | rockstar: hey | 03:38 |
| rockstar | lifeless, hi. | 03:38 |
| lifeless | rockstar: you might like to file a bug that our email stack *lets* us send emails without rationales | 03:38 |
| lifeless | rockstar: separately from the rosetta instance | 03:38 |
| spiv | +1 | 03:39 |
| rockstar | lifeless, hadn't thought about that. | 03:39 |
| lifeless | rockstar: we should at least have to say explicitly that we don't want a rationale | 03:39 |
| rockstar | lifeless, I can't imagine why we would ever not want a rationale. | 03:40 |
| lifeless | right | 03:40 |
| lifeless | but at the moment we don't have one in N places | 03:40 |
| lifeless | so that would provide a migration strategy | 03:40 |
| lifeless | the folk searching lp bugs for porn amuse me. | 03:41 |
| lifeless | anyhow, its just a thought | 03:43 |
| lifeless | we have a class of bugs that can be easily made explicit rather than accidental | 03:43 |
| rockstar | lifeless, people actually type "porn" in the bug search box? | 03:44 |
| * rockstar goes to do just that | 03:45 | |
| rockstar | Oh man, these bugs are hilarious. | 03:45 |
| lifeless | rockstar: check the oops report from lpnet today | 03:46 |
| lifeless | look at the exceptions | 03:46 |
| rockstar | lifeless, oh man, that's awesome! | 03:48 |
| lifeless | its possibly an attack by a spammer trying to inject that | 03:51 |
| lifeless | I haven't looked closely | 03:51 |
| lifeless | (pun intended) | 03:51 |
| ajmitch | someone doing a bit of security testing on LP? :) | 03:54 |
| lifeless | not with those queries ;) | 03:54 |
| ajmitch | now you've made me curious | 03:55 |
| lifeless | fairly scummy porn terms, in permutations, with some verging on illegality most places | 03:59 |
| ajmitch | heh | 04:00 |
| poolie | lifeless/whoever, my last run of flags-webapp had one failure in the twistedjobrunner, which i'm confident is spurious | 04:25 |
| poolie | and one in lp.registry.windmill.tests.test_team_index.TestTeamIndex.test_addmember | 04:26 |
| poolie | which is probably spurious - do you want to reassure me it is? | 04:26 |
| lifeless | poolie: hi | 04:35 |
| lifeless | its in windmill | 04:35 |
| lifeless | its probably spurious. | 04:35 |
| lifeless | is it repeatable ? | 04:35 |
| poolie | just finished, and it didn't fail the second time | 04:48 |
| poolie | \o/ | 04:48 |
| poolie | what a saga | 04:48 |
| poolie | so i guess i can't land this atm anyhow? not anywhere? | 04:48 |
| lifeless | thats right | 04:53 |
| lifeless | not the PQM is release-critical message above is cleared | 04:54 |
| poolie | ok | 04:55 |
| lifeless | poolie: I'm really glad you've gotten it solid | 05:02 |
| lifeless | poolie: If you want to keep momentum up, I'd start a new branch building on it | 05:02 |
| thumper | poolie: or you could fix the commit to a stacked branch bug ;-) | 05:07 |
| poolie | lifeless: yes i think i'll do that; thumper john is working on commit-to-stacked when he flushes his queue | 05:14 |
| thumper | ok | 05:15 |
| lifeless | morning stub | 05:37 |
| lifeless | stub: I has a pressie for you - more code to review :) | 05:37 |
| stub | yo | 05:37 |
| lifeless | Ursinha-bbl: hi | 05:40 |
| lifeless | Ursinha-bbl: your script has gone nuts ;) | 05:40 |
| lifeless | https://bugs.edge.launchpad.net/launchpad-registry/+bug/615237 | 05:42 |
| _mup_ | Bug #615237: /participants API timing out <timeout> <Launchpad Registry:In Progress by lifeless> <https://launchpad.net/bugs/615237> | 05:42 |
| StevenK | lifeless: You can't QA that one before it lands? | 05:46 |
| lifeless | StevenK: its not landed | 05:46 |
| lifeless | StevenK: the qa bot went nuts | 05:46 |
| lifeless | also bug 84838 | 05:47 |
| _mup_ | Bug #84838: code browser should use oops system <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by spiv> <https://launchpad.net/bugs/84838> | 05:47 |
| lifeless | although that one is merged, so ignore me on this point | 05:47 |
| === Ursinha-bbl is now known as Ursinha | ||
| Ursinha | wgrant, hi | 05:50 |
| Ursinha | hmm | 05:52 |
| Ursinha | thumper, hi | 05:52 |
| lifeless | hi Ursinha | 05:53 |
| Ursinha | hi lifeless | 05:55 |
| spiv | lifeless: co-incidentally I'm just about to hit "submit" on the merge proposal for the production configs change for codebrowse OOPSes. | 05:56 |
| lifeless | spiv: \o/ | 05:57 |
| lifeless | spiv: after you do that | 05:57 |
| poolie | nice | 05:57 |
| lifeless | spiv: you need to file a bug on oopstools to have matsubara update the config so the back end finds them | 05:57 |
| poolie | next meliaes? | 05:57 |
| spiv | lifeless: yes, and do an RT to make sure those OOPSes will be synced or whatever is supposed to happen to them, I think. | 05:59 |
| lifeless | spiv: yes please | 05:59 |
| spiv | doing now :P | 06:00 |
| lifeless | Ursinha: I'm curious why tagger script takked 615237 as fix committed etc | 06:01 |
| wgrant | Ursinha: Hi. | 06:01 |
| lifeless | Ursinha: I think there was some confusion | 06:01 |
| Ursinha | bug 615237 | 06:01 |
| _mup_ | Bug #615237: /participants API timing out <timeout> <Launchpad Registry:In Progress by lifeless> <https://launchpad.net/bugs/615237> | 06:01 |
| lifeless | Ursinha: you'll need to look at it to see the activity | 06:02 |
| Ursinha | lifeless, I'm looking, wanted the link :) | 06:02 |
| lifeless | :> | 06:02 |
| lifeless | Ursinha: pad.lv/615237 - nearly as easy | 06:02 |
| Ursinha | lifeless, actually I'm kinda smoke testing the refactored tagger | 06:03 |
| lifeless | Ursinha: cool! | 06:03 |
| Ursinha | so there's the confusion | 06:03 |
| poolie | lifeless: can you think of any good examples i should crib from for the flag editor ui? | 06:03 |
| poolie | either for code or tests or both | 06:03 |
| lifeless | of course, I hope I didn't add a bug to it ;) | 06:03 |
| Ursinha | lifeless, no, the refactoring was really cool :) | 06:03 |
| lifeless | poolie: hmm, you know, I'm not sure. | 06:03 |
| Ursinha | lifeless, it's working ok, only think that's a PITA is that devpad is hardy and runs old version of lplib | 06:03 |
| lifeless | thumper: hey ^ poolies question. | 06:04 |
| Ursinha | lifeless, is there anything you could do about it? | 06:04 |
| Ursinha | lifeless, so stuff works locally but not there | 06:04 |
| Ursinha | that sucks | 06:04 |
| lifeless | Ursinha: about hardy on devpad ? | 06:04 |
| Ursinha | lifeless, yes, or maybe upgrading lplib | 06:05 |
| lifeless | the losas don't want to break things while they are sprinting | 06:05 |
| Ursinha | wgrant, do you know if all of your soyuz changes have been tested? | 06:05 |
| Ursinha | lifeless, argh | 06:05 |
| lifeless | but we can request that devpad be near/at the front of the list | 06:05 |
| Ursinha | really? | 06:05 |
| Ursinha | yay | 06:05 |
| lifeless | for next week | 06:05 |
| Ursinha | cool! | 06:05 |
| lifeless | I don't see why not | 06:05 |
| lifeless | staging is lucid already | 06:05 |
| wgrant | Ursinha: I don't know how non-bug QA works. | 06:05 |
| wgrant | So I can't really tell. | 06:06 |
| lifeless | Ursinha: I will mail the losas and flacoste now | 06:06 |
| Ursinha | lifeless, thanks muchly | 06:06 |
| Ursinha | wgrant, for those that have bugs | 06:06 |
| wgrant | Most of them are untestable, anyway. | 06:06 |
| Ursinha | wgrant, I'll give you the list and you tell me | 06:06 |
| * wgrant checks. | 06:06 | |
| Ursinha | please | 06:06 |
| wgrant | Sure. | 06:06 |
| lifeless | Ursinha: is that soon enough ? | 06:06 |
| Ursinha | lifeless, for what? | 06:07 |
| lifeless | devpad to lucid | 06:07 |
| Ursinha | lifeless, I guess | 06:07 |
| lifeless | if you need something urgently there may be a backported lplib in CAT which we can use | 06:07 |
| lifeless | Ursinha: you tell me :) | 06:07 |
| Ursinha | lifeless, well, I need to port the script to run newer lplib | 06:08 |
| Ursinha | that doesn't have transitionToMilestone and the like | 06:08 |
| lifeless | I thought that was just the LP API version | 06:08 |
| lifeless | if you specify that you want 1.0, it will use .milestone + lp_save() | 06:08 |
| Ursinha | lifeless, iirc I tried and didn't work | 06:08 |
| Ursinha | don't remember why now, too late for my brain | 06:09 |
| poolie | should the ui code be in say registry, or is it ok in services? | 06:09 |
| lifeless | Ursinha: ok | 06:09 |
| lifeless | poolie: services - keep it with the component | 06:09 |
| lifeless | poolie: features/browser/* | 06:09 |
| lifeless | poolie: and features/templates/* | 06:09 |
| Ursinha | wgrant, Bug 590708, Bug 241129, Bug 491418, Bug 605002, Bug 612157 | 06:11 |
| _mup_ | Bug #590708: DistroSeries.getBuildRecords often timing out <api> <qa-ok> <soyuz-build> <timeout> <Launchpad Foundations:Triaged by benji> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/590708> | 06:11 |
| _mup_ | Bug #241129: 'queue override' command scales at O(n^2) with the number of packages on the commandline <qa-needstesting> <soyuz-ftpmaster-tools> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/241129> | 06:11 |
| _mup_ | Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418> | 06:11 |
| _mup_ | Bug #605002: Soyuz doesn't accept upload with "Architecture: linux-any" <qa-needstesting> <soyuz-build> <soyuz-upload> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/605002> | 06:11 |
| _mup_ | Bug #612157: PPA size calculation should no longer exclude DDEBs <qa-untestable> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/612157> | 06:11 |
| Ursinha | sorry the flood | 06:11 |
| poolie | maybe featuredproject is a good comparable simple example? | 06:11 |
| wgrant | Ursinha: The first and last are already OK and untestable respectively. 241129 needs DF console access, 491418 requires a buildd rollout (so isn't really relevant to the release anyway), and 605002 requires DF access too. | 06:12 |
| wgrant | So, 2 and 4 should really be QAd, but I can't do it. | 06:12 |
| Ursinha | wgrant, right. I'll talk to bigjools tomorrow | 06:13 |
| wgrant | Thanks. | 06:13 |
| Ursinha | wgrant, are those changes likely to explode everything else if qa-bad? | 06:13 |
| wgrant | Ursinha: 4 could be pretty terrible, but that code has been exercised reasonably well on DF lately. | 06:14 |
| wgrant | Just not the new case. | 06:14 |
| Ursinha | wgrant, ah, right, so almost QAed :) | 06:14 |
| lifeless | poolie: actually, I know an example that might be good | 06:14 |
| lifeless | poolie: 'jabber ids' on people | 06:14 |
| lifeless | thats something that is list-list | 06:15 |
| lifeless | list-like | 06:15 |
| Ursinha | wgrant, thanks for the info | 06:15 |
| lifeless | https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.tag=qa-needstesting | 06:20 |
| lifeless | hmm | 06:20 |
| lifeless | kindof want to see assignee there | 06:20 |
| Ursinha | lifeless, in the list? | 06:21 |
| lifeless | yeah | 06:21 |
| lifeless | not always | 06:21 |
| lifeless | just now :P | 06:21 |
| Ursinha | hehe | 06:21 |
| lifeless | Ursinha: so, how do we say 'xxx is not relevant to the release, even though its not qa tested yet' | 06:22 |
| lifeless | https://bugs.edge.launchpad.net/launchpad-code/+bug/84838 | 06:22 |
| _mup_ | Bug #84838: code browser should use oops system <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by spiv> <https://launchpad.net/bugs/84838> | 06:22 |
| lifeless | is fix committed | 06:22 |
| lifeless | and is not tested | 06:22 |
| lifeless | but its got a pending related branch for the prod configs | 06:22 |
| lifeless | and then rsync + backend oops change | 06:22 |
| lifeless | so it sshouldn't affect the release | 06:22 |
| lifeless | Ursinha: would changing the milestone be enough ? | 06:23 |
| Ursinha | lifeless, if that's going to be released next cycle, assign to the milestone in which that should be released | 06:23 |
| Ursinha | lifeless, yes | 06:23 |
| lifeless | done | 06:23 |
| Ursinha | thanks | 06:23 |
| lifeless | and to say 'does not need qa' ? | 06:23 |
| lifeless | qa-untestable, or is there something more precise ? | 06:24 |
| lifeless | bug https://bugs.edge.launchpad.net/launchpad-registry/+bug/611853 | 06:24 |
| _mup_ | Bug #611853: remove remove_security_proxy_and_shout_at_engineer calls <qa-needstesting> <tech-debt> <Launchpad Registry:Fix Committed by sinzui> <https://launchpad.net/bugs/611853> | 06:24 |
| * lifeless goes with untestable for now | 06:25 | |
| lifeless | wgrant: can you aid with the qa of the other three soyuz things? | 06:25 |
| Ursinha | lifeless, qa-untestable | 06:26 |
| lifeless | rockstar: don't support you're around ? https://bugs.edge.launchpad.net/launchpad-code/+bug/592937 | 06:26 |
| Ursinha | lifeless, I had this question on epic, but I couldn't convince myself that another kind of untestable tag was justified | 06:26 |
| _mup_ | Bug #592937: "View source package recipes" oddly located on Person index <confusing-ui> <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by rockstar> <https://launchpad.net/bugs/592937> | 06:26 |
| lifeless | Ursinha: its fine by me as it is; just wanting to do the right thing | 06:27 |
| wgrant | lifeless: The three Ursinha gave me above? | 06:27 |
| Ursinha | lifeless, ok :) | 06:27 |
| Ursinha | lifeless, I guess wgrant already gave his take on those three | 06:27 |
| Ursinha | he needs access he doesn't have to test that | 06:27 |
| lifeless | ah | 06:28 |
| wgrant | Two of them are untested, one of them won't be rolled out tonight anyway. Of the two that are untested, one has been mostly tested, and the other is in a rarely-used archive admin script. | 06:28 |
| wgrant | So, not terribly critical. | 06:28 |
| lifeless | wgrant: there are 4 ;> | 06:28 |
| lifeless | 3 soyuz one buildd | 06:28 |
| wgrant | What are the three Soyuz? | 06:28 |
| wgrant | There's one Soyuz/buildd. | 06:28 |
| * lifeless refreshes the page | 06:29 | |
| lifeless | https://bugs.edge.launchpad.net/soyuz/+bug/241129 | 06:29 |
| _mup_ | Bug #241129: 'queue override' command scales at O(n^2) with the number of packages on the commandline <qa-needstesting> <soyuz-ftpmaster-tools> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/241129> | 06:29 |
| lifeless | https://bugs.edge.launchpad.net/soyuz/+bug/491418 | 06:29 |
| Ursinha | lifeless, I need to poke bigjools tomorrow about those | 06:29 |
| _mup_ | Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418> | 06:29 |
| lifeless | https://bugs.edge.launchpad.net/soyuz/+bug/605002 | 06:29 |
| _mup_ | Bug #605002: Soyuz doesn't accept upload with "Architecture: linux-any" <qa-needstesting> <soyuz-build> <soyuz-upload> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/605002> | 06:29 |
| lifeless | and the buildd one | 06:29 |
| lifeless | https://bugs.edge.launchpad.net/launchpad-buildd/+bug/491418 | 06:29 |
| _mup_ | Bug #491418: build architecture should not be hard-coded at ppa install time <ppa> <qa-needstesting> <soyuz-build> <Launchpad Auto Build System:Fix Committed by wgrant> <Soyuz:Fix Committed by wgrant> <https://launchpad.net/bugs/491418> | 06:29 |
| wgrant | lifeless: 491418 is the buildd/Soyuz one. | 06:29 |
| lifeless | right | 06:29 |
| lifeless | so 491418 we can ignore - lamonts baby | 06:30 |
| wgrant | RIght. | 06:30 |
| lifeless | the queue override one is the console one | 06:30 |
| lifeless | I'm not sure why the other two are untestable | 06:30 |
| lifeless | can't you toss something at stagings upload queue and see what happens ? | 06:30 |
| wgrant | staging doesn't have an upload queue. | 06:30 |
| lifeless | ! | 06:30 |
| wgrant | DF does, but it needs someone to run everything. | 06:31 |
| lifeless | I'll RT that now. | 06:31 |
| wgrant | It doesn't have an archive setup. That's why we have DF. | 06:31 |
| lifeless | wgrant: staging /exists/ for QA | 06:31 |
| lifeless | if it can't QA everything, its a bug. | 06:31 |
| wgrant | And DF exists for Soyuz QA! | 06:32 |
| lifeless | DF is intended to be rather broader than that. | 06:32 |
| lifeless | wgrant: do you think we should /not/ be able to QA this on staging? | 06:32 |
| wgrant | lifeless: No, but I think it's a fair bit of work to get it set up there. | 06:33 |
| lifeless | wgrant: longest journey, smallest step | 06:33 |
| lifeless | done | 06:37 |
| lifeless | wgrant: the goal of the QA environment is to QA changes; dogfood is great for experimenting and playing with stuff, but its not generally accessible nor maintained in the same strict way staging is. | 06:38 |
| lifeless | wgrant: *if* staging ran an upload queue and had a virtualised builder, we could qa recipes, and both of your fixes. | 06:38 |
| lifeless | rt 40837 for reference | 06:39 |
| thumper | poolie: I'd suggest playing with balsamiq a bit to design the interface | 06:46 |
| poolie | thumper: how do i get an account or access? | 06:48 |
| Ursinha | thumper, do you know about QA of bug 614858 and bug 592937? | 06:49 |
| _mup_ | Bug #614858: Recipe builds break the world when requester does not have a preferred email address <qa-needstesting> <Launchpad Bazaar Integration:Fix Committed by thumper> <https://launchpad.net/bugs/614858> | 06:49 |
| _mup_ | Bug #592937: "View source package recipes" oddly located on Person index <confusing-ui> <qa-needstesting> <recipe> <Launchpad Bazaar Integration:Fix Committed by rockstar> <https://launchpad.net/bugs/592937> | 06:49 |
| thumper | poolie: canonical has a license to use it | 06:49 |
| thumper | poolie: and you can download and trial easily | 06:49 |
| thumper | Ursinha: 614858 requires a daily build to run on staging which requires a losa to run the script | 06:50 |
| thumper | 592937 not so much | 06:50 |
| Ursinha | thumper, anyone of those could destroy the world if qa-bad? | 06:51 |
| thumper | Ursinha: given my unittests, I'm pretty certain it is fine | 06:51 |
| thumper | Ursinha: but I can't qa it right now | 06:51 |
| thumper | 592937 isn't that important | 06:51 |
| thumper | it just moves a link around | 06:51 |
| wgrant | And it's on edge -- it's fine. | 06:52 |
| Ursinha | thumper, right | 06:53 |
| Ursinha | thumper, when could you QA it? | 06:53 |
| lifeless | stub: I tried to capture some common things - https://dev.launchpad.net/Database/Performance - would love it if you were to garden that page :> | 06:58 |
| lifeless | stub: I didn't get into db design issues at all, I think that with you reviewing every db patch we don't need to try to and pre-communicate about that at this stage | 06:59 |
| stub | I'll look after fooding | 07:04 |
| lifeless | yeah no rush | 07:05 |
| poolie | what's the tasteful way to define the page title? make it a property of the view object? | 07:15 |
| lifeless | theres a slot that is automatically filled | 07:16 |
| wgrant | There are page_title and label properties, IIRC. | 07:17 |
| wgrant | I forget which one is which. | 07:17 |
| wgrant | I think page_title is used in the <title> and breadcrumbs, while label is used in the h1. | 07:17 |
| === spm changed the topic of #launchpad-dev to: Launchpad down/read-only from 0800-0930 UTC for a code update | Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | ||
| === Ursinha is now known as Ursinha-zzz | ||
| lifeless | spm: we're releasing ? | 08:21 |
| spm | lifeless: yes; tho that you're questioning implies a level of surprise? | 08:22 |
| lifeless | excitement | 08:22 |
| lifeless | I plan to watch the hourly oops report like a hawk | 08:22 |
| maxb | buildbot/pqm has been switched to watching the *lucid* builders <--- exciting too | 08:22 |
| lifeless | stub: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-timeout-candidates.html is now updated | 08:22 |
| lifeless | stub: and yeah, LFA is terribad :) | 08:22 |
| maxb | That's a nice new word :-) | 08:23 |
| kb9vqf | Any idea what might suddenly cause this? | 08:23 |
| kb9vqf | http://pastebin.ubuntu.com/476804/ | 08:23 |
| lifeless | kb9vqf: looks like an API has changed | 08:24 |
| kb9vqf | This is on all buildd systems now | 08:24 |
| kb9vqf | Uh oh | 08:24 |
| lifeless | is this on the launchpad.net deployment ? | 08:24 |
| kb9vqf | A local copy of Launchpad | 08:24 |
| lifeless | ok | 08:24 |
| kb9vqf | I did just upgrade some packages on the master server | 08:25 |
| lifeless | so make sure you're running the latest launchpad-buildd and launchpad code bases | 08:25 |
| kb9vqf | lifeless: I don't think it's an API change--I just looked at the latest slave.py file in bazaar and the API for the failing function is identical | 08:36 |
| kb9vqf | (to my version, that is) | 08:36 |
| kb9vqf | This all started after I tried to add a new distribution series to the database | 08:36 |
| lifeless | wgrant: ^ ring any bells? | 08:37 |
| === henninge_ is now known as henninge | ||
| wgrant | kb9vqf: I haven't seen that in ages. You've restarted buildd-manager? | 08:39 |
| wgrant | It just seems to... happen... sometimes. | 08:40 |
| wgrant | I think I knew the reaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaspoint. | 08:40 |
| wgrant | Woah. | 08:40 |
| kb9vqf | ??? | 08:40 |
| wgrant | "reason at some point", that was. | 08:40 |
| kb9vqf | Yeah, me too | 08:40 |
| kb9vqf | But I can't pull it up in my notes | 08:40 |
| kb9vqf | Seems really familiar | 08:40 |
| * kb9vqf goes to restart buildd manager | 08:40 | |
| adeuring | good morning | 08:43 |
| kb9vqf | wgrant: That was it...a buildd-manager restart cured the problem | 08:44 |
| kb9vqf | I'm writing that down somewhere for reference ;-) | 08:45 |
| wgrant | Hmm. | 08:46 |
| kb9vqf | wgrant: I don't know if this will help, but I inserted a debug statement printing the RPC function call arguments | 08:49 |
| kb9vqf | The results are here: http://pastebin.ubuntu.com/476811/ | 08:49 |
| kb9vqf | It looks like something sends an invalid set of arguments beginning with "urlbase" when the failure occurs | 08:50 |
| === almaisan-away is now known as al-maisan | ||
| === henninge changed the topic of #launchpad-dev to: Launchpad is currently down/read-only until 0930 UTC for a code update | Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad | ||
| mrevell | Hey up | 09:39 |
| poolie | lifeless, if any, i have a form showing current rules | 09:39 |
| poolie | just trying to work out how to make it cope nicely with items with no single integer primary key | 09:40 |
| lifeless | poolie: nice | 10:06 |
| lifeless | mthaddon: hey, when you have a second: what time does tuolumne do the hourly oops graph updates? | 10:44 |
| === spm changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad | ||
| wgrant | buildd-manager seems to be doing reasonably well. | 11:01 |
| mwhudson_ | oh, we haz rollout? | 11:02 |
| lifeless | yep | 11:06 |
| bigjools | wgrant: it's not doing too bad is it, although there's a couple of problems | 11:08 |
| bigjools | not seen them again since it start though | 11:08 |
| wgrant | bigjools: It's still slow. | 11:15 |
| wgrant | bigjools: But it's at least staggered. | 11:15 |
| wgrant | So it should be less prone to getting itself into spirals of slow, slow death. | 11:15 |
| bigjools | it will be slightly better than before | 11:15 |
| bigjools | in terms of throughput | 11:15 |
| wgrant | Right. | 11:15 |
| bigjools | because we're interleaving more stuff now | 11:15 |
| wgrant | The main problem before was resumes, right? | 11:16 |
| wgrant | They'd block everything for a couple of minutes. | 11:16 |
| bigjools | although I am getting "error: (4, 'Interrupted system call')" a lot :/ | 11:16 |
| jml | is Launchpad writable yet? | 11:16 |
| bigjools | and it ends up disabling the builder | 11:16 |
| bigjools | jml: yes, since 30 mins ago | 11:16 |
| jml | yay | 11:16 |
| wgrant | bigjools: Hm, is that a slave exception? | 11:17 |
| bigjools | wgrant: no | 11:17 |
| wgrant | Also, what's with the db_lp failure? | 11:17 |
| jml | it normally means that a signal happened during a system call | 11:17 |
| jml | and *that* normally means that you should try again | 11:17 |
| bigjools | yes - I suspect it's getting sigchld since I am interleaving Popen with data comms to builders | 11:17 |
| jml | that would make sense | 11:18 |
| bigjools | jml: since it's used Twisted XMLRPC I would have hope that it would have taken care of that for me :/ | 11:18 |
| jml | bigjools, what? no. | 11:18 |
| jml | bigjools, Twisted XMLRPC doesn't do magic process handling | 11:18 |
| jml | bigjools, reactor.spawnProcess, otoh. | 11:18 |
| bigjools | surely it should catch EINTR and retry | 11:18 |
| bigjools | http://pastebin.ubuntu.com/476868/ | 11:20 |
| jml | the reactor does that. | 11:20 |
| poolie | bigjools: strangely enough i just fixed a bug like that in non-twisted lp code | 11:20 |
| jml | bigjools, you are not using Twisted XMLRPC there. | 11:20 |
| poolie | i wonder if something we import hooks a signal so we see it | 11:20 |
| bigjools | ah you know what, that bit's not twisted | 11:20 |
| jml | :) | 11:20 |
| poolie | see eintr when we normally won't | 11:20 |
| bigjools | poolie: ah hmmm | 11:21 |
| jml | poolie, aiui, what we're seeing is the normal behaviour of Python's libraries | 11:22 |
| lifeless | there's a regular sigchld bug | 11:22 |
| lifeless | hmm, actually. ESLEEP. | 11:22 |
| lifeless | I shall read backchat tomorrow :) | 11:23 |
| poolie | you shouldn't get eintry from sigchld unless you've registered for sigchld | 11:23 |
| jml | heh | 11:23 |
| poolie | i think | 11:23 |
| poolie | it's complicated | 11:23 |
| poolie | anyhow i should go too | 11:23 |
| poolie | good night | 11:23 |
| jml | testr branches will be reviewed mañana | 11:23 |
| bigjools | we have registered it I think | 11:23 |
| * jml stops talking about programming and instead reads resumes | 11:24 | |
| bigjools | that so needs an accented e | 11:25 |
| mwhudson_ | risumi | 11:25 |
| bigjools | rezumey | 11:25 |
| bigjools | I thought he was resuming reading | 11:26 |
| mwhudson_ | i is what you get if you send e-with-acute-in-latin-1 via an email system that strips the high bit | 11:27 |
| bigjools | it would be so much easier if everyone spoke the Queen's English | 11:27 |
| lifeless | mthaddon: https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100811/20100812/ seems to be missing about 10 hours of samples ? | 11:30 |
| lifeless | mthaddon: ah thanks, dunno what you did but it just updated | 11:33 |
| bigjools | losa: is everything looking ok from your PoV? | 11:38 |
| jml | bigjools, the compose key needs to be easier to use | 11:41 |
| * jml wishes again that Linux just used Mac-style keybindings | 11:42 | |
| bigjools | prob more of a desktop environment issue | 11:42 |
| jml | I'm a muppet user. It's all Linux to me. | 11:44 |
| bigjools | which muppet character would you be? | 11:45 |
| wgrant | jml: How could the compose key be easier? RAlt+' e | 11:45 |
| jml | wgrant, for a start, I can never remember if it's RAlt or the menu key (it's the menu key on my laptop) | 11:46 |
| jml | wgrant, secondly, discovering key combinations is hard | 11:47 |
| wgrant | jml: Well, it's configurable. | 11:47 |
| jml | wgrant, thirdly, some of the key combinations in system compose files just don't work | 11:47 |
| wgrant | Hah. | 11:47 |
| jml | wgrant, I know that. I don't want to configure it. | 11:47 |
| jml | fourthly, some of the compose combinations are silly, especially when compared to the Mac equivalent | 11:48 |
| jml | e.g. en dash is Compose+- - . on Linux, and Opt+- on Mac | 11:48 |
| jml | em dash is Compose + - - - vs Opt + _ | 11:49 |
| wgrant | Hmm. | 11:50 |
| stub | The first problem is having to enable that crap, which usually involves configuring it, which means everyone uses a different setup, so documentation becomes non-existent and people think it just doesn't work. | 11:52 |
| jelmer | AltGr seems to work pretty consistently across Linux desktops, I don't remember having to configure it | 11:53 |
| jelmer | jml: What keyboard layout are you using? | 11:53 |
| jml | jelmer, dvorak | 11:54 |
| wgrant | jelmer: Most of us don't have an AltGr. | 11:54 |
| jelmer | jml: What country ? | 11:54 |
| jml | US | 11:55 |
| jelmer | hmm, same here | 12:00 |
| deryck | Morning, all. | 12:06 |
| lifeless | deryck: we'll find out if the new search gets widespread +1 or -1 today ;) | 12:09 |
| deryck | lifeless, indeed! Fingers crossed. :-) | 12:09 |
| bigjools | wgrant: the buildd-manager suddenly decided to start dispatching a build for a disabled rebuild archive :/ | 12:18 |
| wgrant | bigjools: Not again... :/ | 12:27 |
| wgrant | bigjools: How fatal is that now? | 12:31 |
| wgrant | There's no cycle to abort every time. | 12:31 |
| wgrant | Although I guess that just means every builder tries to dispatch it. | 12:31 |
| bigjools | wgrant: so it's pretty fatal as yes, every build tries in turn to dispatch it :) | 12:36 |
| bigjools | builder* | 12:36 |
| bigjools | wgrant: what I am more confused about is why is the disabled flag getting ignored now | 12:36 |
| wgrant | bigjools: archive.disabled hasn't been honoured for a while. | 12:40 |
| wgrant | job.status == SUSPENDED is how it's done. | 12:40 |
| bigjools | wgrant: I beg to differ | 12:40 |
| bigjools | oh ok, well same shit | 12:40 |
| wgrant | Well, the latter could possibly be inconsistent with archive.disabled, so the distinction is important.... | 12:41 |
| bigjools | this means the jobs got re-enabled somehow | 12:41 |
| bigjools | yes | 12:41 |
| wgrant | Possibly a queue-builder run? | 12:41 |
| bigjools | I'm butchered them now anyway | 12:41 |
| bigjools | q-b hasn't run in months | 12:41 |
| bigjools | now I need to fix this eintr issue, we're getting regular disabled builders :( | 12:42 |
| === bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08 | 10.08 is released, PQM is OPEN | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad | ||
| * bigjools wonders how to test something that handles eintr | 12:50 | |
| === salgado-afk is now known as salgado | ||
| jelmer | Would make sense for ./utilities/ec2 land to CC the MP in question with the test results? I currently find myself going through failure emails that all have the same subject, it would be nicer if the results could just be found in the MP. | 13:51 |
| jml | jelmer, hmm. | 14:06 |
| jml | jelmer, first up, it would make sense to have the MP name in the subject. | 14:07 |
| jml | jelmer, secondly, you are reminding me about how poor the email is and how I ought to fix it. | 14:07 |
| === matsubara-afk is now known as matsubara | ||
| jml | jelmer, I don't think the MP should have test results in the case of failure (although I wouldn't be opposed to attaching the subunit log) | 14:08 |
| jml | jelmer, what actual data would you like to see on the MP? | 14:08 |
| jelmer | jml: Personally what I'm interested in is the list of test ids that is failing, so I can run just those tests on my local machine. | 14:09 |
| jelmer | Having the full log attached would also be useful in case of e.g. spurious test failures, but is less important imo | 14:09 |
| jelmer | It might be different for other people though | 14:10 |
| jelmer | jml: Why should the MP not have the test results in case of failure? | 14:10 |
| === Ursinha-zzz is now known as Ursinha | ||
| jml | jelmer, well, it depends on what you mean by test results. the current email is awful. | 14:17 |
| jml | jelmer, a list of failing tests would be fine by me. | 14:17 |
| jml | jelmer, and I don't think MPs have attachments, sadly. | 14:17 |
| jelmer | jml: Ah, right. I was confusing them with bugs. :-/ | 14:17 |
| jelmer | What would be totally awesome is if Launchpad could support attachments for MPs in the same places as comments, and then display summaries when it encounters subunit files. | 14:18 |
| * jelmer dreams on | 14:18 | |
| jml | jelmer, yeah, that would be good. :) | 14:19 |
| === al-maisan is now known as almaisan-away | ||
| sinzui | bigjools, ping | 14:31 |
| sinzui | bigjools, I can make all bugs fix released. all lp, or just my team | 14:35 |
| sinzui | flacoste, I can make all bugs fix released. all lp, or just my team | 14:35 |
| flacoste | sinzui: i think diogo has a script hat did the same, no? | 14:38 |
| sinzui | I have never seen it close all the fixcommitted bugs | 14:39 |
| sinzui | I have done all lp for the last two months and I have always done it for the projects I manag | 14:39 |
| matsubara | sinzui, flacoste: https://code.edge.launchpad.net/~launchpad/lp-qa-tools/bug-editor | 14:40 |
| === almaisan-away is now known as al-maisan | ||
| bigjools | sinzui, matsubara: ya, please release them | 15:01 |
| bigjools | the script I committed to do that ages ago got moved to launchpadlib contrib and now I can't find that either | 15:02 |
| jml | jcsackett, as a new developer, I'd like your opinion on the README that just landed. (no rush though) | 15:03 |
| bigjools | matsubara: your script doesn't work any more | 15:09 |
| bigjools | sinzui: where's yours? | 15:09 |
| matsubara | bigjools, it's been a long time since I used it. should I fix it or are there any other newer tools that does the same thing? | 15:10 |
| * bigjools has no idea | 15:10 | |
| sinzui | I just closed the registry bugs. I can start on the whole team | 15:10 |
| bigjools | what replaced transitionToStatus? | 15:10 |
| sinzui | bigjools, ``bug_task.status = u'Fix Released'`` it is exported as a mutator in the api | 15:12 |
| bigjools | ah | 15:12 |
| candrea | sinzui: on the registry bugs I'm subscribed to I've seen the comment "Fix released in...", but no status changes (e.g. bug 70613) | 15:22 |
| _mup_ | Bug #70613: Encourage projects to configure all services (eg. bug tracker) <bridging-the-gap> <qa-ok> <Launchpad Registry:Fix Committed by bac> <https://launchpad.net/bugs/70613> | 15:22 |
| sinzui | candrea, :( Yes. I see that to. I am missing an lp_save | 15:25 |
| jcsackett | jml: sure; i've got a suite running now so i can take a look. anything in particular you would like comments on? | 15:35 |
| jml | jcsackett, nothing in particular. does it help you understand Launchpad, do you wish you had it a week ago, is it confusing, etc | 15:36 |
| jcsackett | jml: dig. | 15:37 |
| jcsackett | jml: definitely helpful, it covers a fair bit i was dumped into in the first days. | 15:38 |
| jcsackett | jml: assuming you don't mind a bit of crossover with dev.launchpad, a teeny bit about the rocketfuel utils might be nice. | 15:39 |
| jml | jcsackett, good idea | 15:40 |
| jml | jcsackett, though I personally _never_ use the rocketfuel utils | 15:40 |
| jcsackett | jml: i'm still wavering between using them and straight bzr. i find rocketfuel a tad confusing, but the auto run of makebuild &c when you branch isn't half bad. | 15:41 |
| jcsackett | jml: i also agree with flacoste. you hit the mark on tone. | 15:42 |
| jml | jcsackett, thanks :) | 15:53 |
| jcsackett | jml: you're welcome. | 15:54 |
| deryck | gmb, hi. Is bug 424849 done and out in 10.08? The branches are listed as merged, but I thought this work was still in progress. | 15:55 |
| _mup_ | Bug #424849: Launchpad should batch attachment notification emails <story-better-bug-notification> <story-better-notification-sending> <Launchpad Bugs:In Progress by gmb> <apport (Ubuntu):Invalid> <https://launchpad.net/bugs/424849> | 15:55 |
| gmb | deryck, Argh, it's listed as done because I did part of the work, not all of it. It isn't finished, so I'll mark it as Triaged for now since we won't be coming back to it for a bit. | 15:58 |
| deryck | gmb, great, thanks. Remove the milestone, too, please. | 15:58 |
| gmb | Done | 15:59 |
| jml | james_w, are your html matchers available in Launchpad? | 16:09 |
| jml | as in, for Launchpad development | 16:09 |
| james_w | jml: rockstar might have landed them, I'm not sure. | 16:10 |
| jml | james_w, thanks. | 16:10 |
| james_w | jml: what would you like to use them for? (use case gathering) | 16:10 |
| jml | james_w, testing link formatters | 16:10 |
| rockstar | jml, I have a branch that lands them, but that was before james_w added his matcher stuff too. :) | 16:10 |
| rockstar | (I did it on the plane back from Prague) | 16:10 |
| jml | james_w, testing this code: http://paste.ubuntu.com/476985/ | 16:11 |
| jml | rockstar, cool. | 16:12 |
| james_w | jml: cool, probably well suited | 16:12 |
| === deryck is now known as deryck[lunch] | ||
| rockstar | jml, yeah, abentley and I have been trying hard to use unit tests instead of doctests for pagetesting. | 16:13 |
| === Ursinha is now known as Ursinha-lunch | ||
| sinzui | gmb, deryck: I would like a mid implementation review for https://bugs.edge.launchpad.net/launchpad-registry/+bug/613610 I have a paste of what I thought would fix the issue. Can either of you look at the bug and the paste in the comment and explain my incompetence. | 16:18 |
| _mup_ | Bug #613610: +needspackaging bug counts are wrong <bridging-the-gap> <Launchpad Registry:In Progress by sinzui> <https://launchpad.net/bugs/613610> | 16:18 |
| gmb | sinzui, I'll take a look. No promise that I'll actually be any help, though :) | 16:19 |
| sinzui | thanks | 16:19 |
| gmb | Hrm | 16:24 |
| matsubara | bigjools, I fixed the bugeditor script and release 18 foundations bugs | 16:30 |
| bigjools | matsubara: can you do launchpad-project ! | 16:31 |
| bigjools | or I can do it | 16:31 |
| * bigjools does it | 16:31 | |
| matsubara | bigjools, ok, rev 16 is the one you want with the fix | 16:33 |
| bigjools | matsubara: yeah I have it running | 16:33 |
| matsubara | cool | 16:34 |
| gmb | sinzui, Sodomy non sapiens, I'm afraid. | 16:38 |
| gmb | It *should* work; I don't understand why it doesn't. | 16:39 |
| sinzui | I suspect that the trigger fires while the bug is open | 16:39 |
| gmb | sinzui, Yes, that might be the case. Gah. Death to triggers. Might be worth roping stub in to see whether he can shed any light on proceedings. | 16:40 |
| sinzui | thanks gmb | 16:41 |
| gmb | np; sorry I couldn't be more helpful. | 16:42 |
| === deryck[lunch] is now known as deryck | ||
| === matsubara is now known as matsubara-lunch | ||
| === benji is now known as benji-lunch | ||
| === benji-lunch is now known as benji | ||
| === benji is now known as benji-lunch | ||
| === salgado is now known as salgado-lunch | ||
| === Ursinha-lunch is now known as Ursinha | ||
| === matsubara-lunch is now known as matsubara | ||
| === benji-lunch is now known as benji | ||
| === beuno is now known as beuno-lunch | ||
| === salgado-lunch is now known as salgado | ||
| * jml gone | 18:26 | |
| === beuno-lunch is now known as beuno | ||
| EdwinGrubbs | leonardr: ping | 19:16 |
| nigelb | Oh, I like the new format for "reported by $foo" :) | 19:18 |
| nigelb | for bugs that is. | 19:18 |
| leonardr | edwingrubbs, hi | 19:20 |
| EdwinGrubbs | leonardr: I just found operation_for_version(). I assume that will let me specify two different @operation_parameters lists, so I can add a parameter without getting the error: getTimeline" doesn't have the following exported parameters in version "(earliest version)": max_releases. | 19:23 |
| leonardr | Edwin: kind of. you will need to have _something_ happen with max_releases in the earliest version, or lazr.restful won't know what value to pass into it | 19:25 |
| leonardr | but you can give it a default value or None or something | 19:25 |
| EdwinGrubbs | leonardr: do I reference the earliest version as '1.0'? | 19:26 |
| leonardr | edwin: you shouldn't need to reference it explicitly, but it is 'beta' | 19:26 |
| leonardr | the code at the bottom of the pile of annotations applies to the earliest version | 19:27 |
| === al-maisan is now known as almaisan-away | ||
| lifeless | MOIN | 20:47 |
| lifeless | that ism good morning | 20:48 |
| lifeless | flacoste: morning | 21:00 |
| lifeless | flacoste: whats the upload address for staging ? | 21:00 |
| flacoste | lifeless: you mean for sftp / poppy? | 21:01 |
| lifeless | yeah | 21:01 |
| flacoste | li have no idea | 21:01 |
| lifeless | ahh :) | 21:01 |
| flacoste | not sure if such a service runs there | 21:01 |
| lifeless | that may be the missing services then | 21:01 |
| lifeless | losa ping | 21:01 |
| lifeless | ^ | 21:01 |
| flacoste | lifeless: they are EOD | 21:01 |
| lifeless | flacoste: I know | 21:01 |
| lifeless | flacoste: thanks : | 21:01 |
| lifeless | ) | 21:02 |
| flacoste | i see that you have a lot of faith in scrollback :-) | 21:02 |
| lifeless | for some things :) | 21:02 |
| lifeless | if they don't notice, its not a big deal :) | 21:02 |
| lifeless | so hows its looking, 10 hours after release? | 21:03 |
| lifeless | flacoste: do you know where the 10.08 release notes are? | 21:07 |
| flacoste | lifeless: looks good i think | 21:07 |
| flacoste | lifeless: do you mean the roll-out report? | 21:07 |
| lifeless | no | 21:07 |
| lifeless | I saw that | 21:07 |
| flacoste | that's all, we don't produce release notes anymore | 21:08 |
| lifeless | I mean mrevell's user facing explanation of whats new | 21:08 |
| flacoste | we don't do these anymore | 21:08 |
| flacoste | we only publish blog posts | 21:08 |
| lifeless | the losa template needs a change then :) | 21:08 |
| lifeless | There are separate messages with release notes, so not duplicated here. | 21:08 |
| flacoste | lol | 21:08 |
| flacoste | yeah | 21:08 |
| flacoste | indeed | 21:08 |
| flacoste | blog posts will be published tomorrow | 21:08 |
| lifeless | ok cool | 21:08 |
| lifeless | hmm | 21:09 |
| lifeless | it would be nice to do that a little closer to the change | 21:09 |
| lifeless | rfwtad will help | 21:09 |
| lifeless | danilos: still around ? | 21:10 |
| flacoste | lifeless: it usually is, mrevell decided to play it cautiaus this time around | 21:14 |
| flacoste | lifeless: i think it was because he was in London and travelling back | 21:14 |
| flacoste | wanted to make sure we didn't have to pull stuff off | 21:14 |
| lifeless | can other people help with it ? | 21:15 |
| lifeless | e.g. put it as a draft in wordpress, allow any editor to tweak-and-publish ? | 21:15 |
| flacoste | pushing the buttons to publish them? i guess | 21:15 |
| flacoste | i know kfogel used to have access, jml probably also has | 21:15 |
| lifeless | mrevell has given me access to blog on the blog | 21:16 |
| lifeless | and I would blog about search (yay!) but I'm sure his release blog includes because we were chatting about that | 21:16 |
| lifeless | anyhow, minor issue | 21:17 |
| lifeless | now, I'm landing webapp-flags | 21:17 |
| lifeless | martin fixed all the test fallout and we were waiting for pqm to be open ;) | 21:17 |
| lifeless | Launchpad Development Channel | Week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | 21:18 |
| === lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | ||
| thumper | lifeless: it isn't week 1 yet | 21:24 |
| lifeless | thumper: it isn't ? | 21:24 |
| thumper | lifeless: not until next week | 21:24 |
| lifeless | >< | 21:24 |
| thumper | :) | 21:25 |
| lifeless | if the release is the end of the process? :) | 21:25 |
| thumper | we do calendar weeks | 21:25 |
| thumper | don't confuse our poor wee brains | 21:25 |
| === lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08; I hate mondays -> week 1 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | ||
| ajmitch | so the 'I hate mondays' is just a general statement then? | 21:31 |
| lifeless | :P | 21:32 |
| mars | Is there some way to tell LP to re-scan a merge proposal diff? Mine has conflicts that have since been resolved. | 22:48 |
| wgrant | It should automatically update within a couple of minutes of the source branch push. | 22:48 |
| === cody-somerville_ is now known as cody-somerville | ||
| jelmer | mars: the diff generator was down earlier | 22:49 |
| jelmer | rockstar: is that the fire you're fighting perhaps? | 22:49 |
| mars | ok. Thanks guys | 22:49 |
| rockstar | jelmer, no, I'm fighting with maverick recipe builds failing. | 22:50 |
| === matsubara_ is now known as matsubara-afk | ||
| thumper | jelmer: no the diff generator is a twisted bug that should be impossible | 23:00 |
| thumper | jelmer: we know it exists, but haven't yet been able to work out where | 23:00 |
| jelmer | ah, ok | 23:01 |
| === jpds_ is now known as jpds | ||
| wgrant | Do imports stack yet? | 23:53 |
| mwhudson | wgrant: no | 23:55 |
| mwhudson | jelmer: hi, did you get a mail i sent you about importing non-master branches yet? | 23:56 |
| mwhudson | -yet | 23:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!