[00:19] * vila woke up one hour later than yesterday, one more week and the jet lag will be sorted out :-( [00:24] jam: no worries [00:24] vila: you should definitely not be up now :) [00:25] jam: indeed :-/ [00:25] jam: nice summary in bug #589521, thanks for that [00:25] Launchpad bug 589521 in Ubuntu Distributed Development "nagios monitoring of package imports needed" [Critical,Triaged] https://launchpad.net/bugs/589521 [00:38] vila: Sorted out? What's wrong with things like they are? It sounds like a perfectly sensible schedule to me :p [00:40] fullermd: the truth is, the arguments with wife and daughters tend to be easier to manage under this schedule... [00:40] See? Come to the dark side... [02:54] mkanat: hey, you said the other day that b.l.n had the wrong theme? [02:54] lifeless: Yeah. [02:54] mkanat: I meant to ask what you mean by that, but got local interrupts [02:54] lifeless: I'll see if I can show you. [02:54] lifeless: This page has the wrong CSS: http://bazaar.launchpad.net/~loggerhead-team/loggerhead/trunk-rich/view/head:/.bzrignore [02:55] lifeless: Because this file is in fact missing: http://bazaar.launchpad.net/static/css/view.css [02:55] what css should it have? [02:55] lifeless: That CSS exists in the actual branch. [02:56] lifeless: But it is a 404 on bln. [02:56] let me just nab someone [02:56] mkanat, i don't suppose you would have any time left to actually measure trunk (without historydb) vs 1.18 performance on a large branch? [02:57] poolie: I might. I have 2.15 hours left. [02:57] I mean, 2 hrs and 15 minutes. [02:57] facts might bring a merciful death to this thread [02:57] poolie: I don't think anybody ever said that history_db was slower. [02:57] robert thinks that john said it would be slower if the db was not enabled [02:58] poolie: I didn't see that on the list. [02:58] mkanat: jam said this: [02:58] Old New Action [02:58] 10s 30s First request of a branch in a project that has not been [02:58] previously seen. (slower) [02:58] (amongst other things) [02:58] note that none of our projects have been previously seen [02:59] lifeless: That's with history_db enabled or without? [02:59] and a 30s request will be killed [02:59] mkanat: with [02:59] lifeless: Okay, we're not talking about that situation, then. [02:59] lifeless: That situation would already be handled by what you'd do to enable history_db on codehosting. [02:59] mkanat: he also said that without the plugin, it uses plain bzr apis [02:59] lifeless: Which it does now. [02:59] so will be slower than before as well [03:00] lifeless: He said that there is some difference there from before history_db? Because the branch you're currently running uses plain bzr apis. [03:01] Oh, I may be seeing what he means. [03:04] ? [03:04] is it that it's removed caching code that would have helped in 1.18? [03:04] I'm still reading over the code. [03:04] poolie: I believe so, yes. [03:05] I'm not sure how much that actually matters, though. [03:05] What I don't know is how to turn off history_db to test. [03:05] remove the plugin [03:05] the historydb change is the following: [03:05] lifeless: It's a part of loggerhead. [03:05] - discard all the caching apis [03:05] - switch to raw bzrlib apis [03:05] ok, so istm it would be a worthwhile use of time to measure whether trunk-minus-hdb is in fact slow [03:05] lifeless, mkanat: 30s is slower for the first request on a project when it has to convert the full ancestry to the cached form. Note that it can incrementally build the cache, so even if the thread is killed it will eventually succeed. [03:05] - have a *bzrlib* plugin that makes those apis faster. [03:06] Also note, it is only for Emacs sized projects. bzr is a few seconds, IIRC [03:06] Yeah, launchpad also seems to be a few seconds, in my testing. [03:06] And yeah, even if history_db can't write to the disk, it will create its cache in memory. [03:06] history_db was pulled into the loggerhead codebase [03:06] jam: ah, k [03:07] So I suppose history_db is always enabled. [03:07] and 10s would become *every* request if you remove all the caching [03:07] so, this is really a bit of a distraction; unless its a no brainer: < a day of work, including getting up to speed, its not suitable for a maintenance squad at this time. [03:07] jam: That would happen basically if I just don't specify "--cache", right? [03:07] mkanat: I don't remember from the command line, tbh [03:07] the question is: is there anything you could do in ~1 day that will get it at least no worse than 1.18? [03:08] poolie: I suspect that for the actual average use case, it is already no worse than 1.18. [03:08] poolie: With the scale of codebrowse, it's likely that current stable will be rebuilding caches quite a bit. [03:08] ok [03:08] poolie: This is just my suspicion, though, not a proven fact. [03:09] poolie: I think the most immediate thing would be to teach Launchpad to shared caches between projects [03:09] at which point you'll hit a couple timeouts, but afterwards everything should be better [03:09] mkanat, i'm wondering if we can do some measurements to get more confidence in whether that is true or not [03:10] poolie: Yeah, I could do that, for sure. [03:10] jam, i know, but it seems like that is going to be more than a day of workL [03:10] poolie: I have a loggerhead load tester already. [03:10] poolie: it *should* just be changing the path to not include the branch name, just the project name [03:10] poolie: Let me do a test right now. [03:10] won't be shared between users, but probably pretty good [03:10] but I don't fully understand the codehosting remapping, etc. [03:10] jam: would that permit leakage from private projects [03:10] lifeless: revision id and revno stuff [03:11] e.g. if someone has the revid for a private project, makes that a ghost in their branch of same project [03:11] no revision information [03:11] lifeless: the cache is just the graph of revnos and revision_ids, it doesn't cache the actual content [03:11] jam: revids are sensitive too, given they include email addresses [03:11] so if you have the revision id, then you have all the information you're going to get [03:11] lifeless: but you just said you *have* it [03:11] right, thats one scenario [03:12] since you are making it a ghost [03:12] Okay, so I think that disabling the on-disk caching of historydb involves commenting out two lines. [03:12] the other one is it likely to expose a revid when it shouldn't. [03:12] Then it should cache in memory. [03:12] e.g. due to bugs [03:12] mkanat, let's try it! [03:12] poolie: Okay, going to try it now. [03:13] remember [03:13] to test it with a stacked branch and HTTP backend [03:13] poolie, lifeless, jam: Another option is to have codebrowse cache everything in one directory, in one file. [03:13] for apples to apples comparison [03:13] well [03:13] his hardware and load is undoubtedly not exactly the same [03:14] My load tester creates significantly more load than codebrowse experiences. [03:14] but perhaps we can get some data [03:14] Okay, I can already tell you that if you disable the on-disk cache, yes, trunk is slow. [03:14] I don't even have to run the tester. [03:14] Getting information for ViewUI: 15.945 secs [03:15] on emacs? [03:15] On launchpad. [03:16] ok, and you're confident that's representative? [03:16] if so, that means we can't just deploy trunk [03:16] jam: I imagine that if we used loggerhead's normal configuration where there's only one history_db.sql file for all branches combined, then it would become unmanageably large, on codehosting? [03:16] we would need to do whatever it is to get historydb in use [03:17] poolie: Yeah--what I didn't realize was that history_db was always being enabled, in my earlier tests. [03:17] poolie: Even if you don't specify a cache directory, loggerhead creates a temporary one when you start it. [03:17] and the question then is: can that task be made sufficiently small to do it ahead of doing other loggerhead work [03:18] poolie: It's all infrastructural work on codehosting, AIUI. [03:18] so it does cache to disk at the moment? but in a different form [03:18] poolie: No, your current stable loggerhead does not cache to disk. [03:18] poolie: But trunk always does. [03:19] uhm [03:19] My understanding is that it uses sqlite caches still [03:19] mwhudson: ^ [03:19] lifeless: It *can*. [03:19] lifeless: I don't think codebrowse has that enabled. [03:20] i'm pretty sure it does [03:20] so am I [03:20] it seems like changing it to write historydb caches into the same place should not be too traumatic [03:20] Yeah. [03:20] mkanat: I don't think it would become unmanageable, but probably bigger, and after that first attempt, probably faster. [03:20] Actually, that wouldn't be a problem at all, if you are already using the on-disk cache. [03:20] jam, so it would be better to use larger caching scopes, but doing it per branch would be ok? [03:20] it won't die on the first access? [03:20] Yeah, so you could actually just deploy history_db now. [03:20] mkanat: no, loggerhead always cached to disk, it always created a temp one, even before history_db [03:21] jam: Hmm, except that I could always make loggerhead re-generate its branch caches before history_db, with enough simultaneous access. [03:21] jam: It looked to me like it was only using the lru_cache. [03:21] (Unless you configured it to use the on-disk cache.) [03:21] mkanat: it just happened to regen the cache all the time [03:22] if you don't specify a cache, one is created that only lasts the lifetime of the process, and then is thrown out [03:22] jam: Oh indeed. [03:22] (separately, lifeless, is deletion of the /beta api near enough we should worry?) [03:23] poolie: I have to double check the code a bit. but I believe it will populate the cache file incrementaly, so even if it was killed in the first request, the second one will succeed for any given branch [03:23] or maybe the 18th? [03:23] poolie: in principle we're not adding anything to it [03:23] jam: In that case, history_db sounds like a net win with little infrastructural work required. [03:23] jam: well the second request may be to a different instance [03:23] lifeless: but the disk cache is in the same location [03:23] aiui [03:23] I find the hand wave on infrastructure work scary. [03:24] using the same cache that loggerhead is using today [03:24] lifeless: Why would you? [03:24] lifeless: It's just a new file in the same directory. [03:24] lifeless: That's the extent of the change that would happen. [03:24] jam: is historydb broken or unfinished or dangerous in any regard? [03:25] lifeless: Out of curiosity, do both processes currently use the same on-disk cache directory? [03:25] yes [03:25] they do [03:25] poolie: I don't know of anything specifically broken [03:25] caveat code has bugs [03:25] which is one thing jam highlighted as being a concern [03:25] sure [03:25] probably nothing worse than the existing code [03:25] and rollouts are hard [03:25] but, we still need to get through them [03:25] mkanat: I find it scary because handwaving around a bit of core plumbing is somewhat cavalier [03:26] mkanat: when one considers all the bits that can - and have - gone wrong in the past. [03:26] lifeless: Are you talking about infrastructure or about the internal architecture of loggerhead? [03:26] mkanat: infrastructure [03:26] and architecture [03:27] lifeless: Okay. Are you using (or do you have) sqlite 3.7? If you use a WAL, I bet both processes would be fine accessing the same cache. [03:27] I haven't looked at the risk of disclosure for private branches other than my brief queries here [03:27] lifeless: Risk of disclosure in what way? [03:27] lifeless: history_db, when we're just talking about loggerhead, is only a backend change. [03:27] bugs disclosing revision ids not in the branch being accessed [03:28] but, i think at the moment we're not talking about having a shared cache? [03:28] mkanat: I realise that its a backend change. [03:28] poolie: one of jams caveats was slower initial scans w/out a shared cache. [03:28] But we would have a shared cache. [03:28] sigh [03:28] Because we'd have only one cache. [03:29] I'll come back to this, I feel that the point is ....> over there [03:29] doing a bunch of work to assess whether its small enough, based on guesses - is at best a guess. Right now, I still wouldn't confidently ask curtis or julian to take this on within maintenance scope [03:30] lifeless: It doesn't sound like any code work is required. [03:30] if I'm over concerned about the risk, its because of a history of problems. [03:30] it's like a major version bump in a dependency [03:30] perhaps one that has caused trouble in the past [03:30] Yeah, sure. I mean, there's still the testing thing that I just talked about on the list. [03:30] I'm fully with you on that. [03:30] mkanat: I don't know that I've claimed code work; the same engineers do code and integration - everything except deploy & server ops. [03:30] mkanat: its an integrated team leading right up to the ops team [03:31] lifeless: Okay. What do you mean by "integration"? [03:31] for me to say to curtis or julian that its in scope for maintenance, I'd want to be quite confident that there is < 12 hours work asessing, measuring, qaing on staging, deploying & dealing with the fallout. [03:32] lifeless, how would you handle an upgrade to some other component we depend upon? [03:32] there may well be some that are over 12h [03:32] poolie: if its small a maintenance team just does it. If its not a feature squad gets assigned it - as will happen to python2.7 [03:33] ok [03:33] now, someone devs may scratch an itch and get python2.7 workable on dev machines, but actually upgrading prod etc requires considerable care. [03:33] sure [03:33] 12h is an arbitrary figure [03:33] ubuntu upgrades are another case that comes to mind [03:33] istm we have the option of trundling along 1.18.x until we get time to safely do a bigger deployment [03:34] right, they were traumatic, and we're doing them differently next time - separating OS and dependencies; we'll go to py 2.7 *before* the next OS upgrade. [03:34] so that we're in place. [03:34] but we'd probably prefer not to do major development that's duplicating what's been done in an upstream later release [03:34] poolie: I don't see that as an option; I see that as the /only/ option given the info I've got about whats up. [03:34] max has said, rightly or wrongly, that he doesn't think there's any more low hanging fruit for performanec [03:35] so, i don't think we can usefully schedule small blocks on this [03:35] Talking about actual development of loggerhead, that is. [03:35] Which would be a separate issue from rolling out this one. [03:35] and if we're going to schedule a big block, we might as well start it by getting historydb running [03:35] lifeless: Without tests, I can't say how much time deployment would take. But if you also start changing *any* branch significantly, my answer would be the same. [03:35] since we know that will probably help alot [03:36] lifeless: If everything goes fine, I suspect it will take no longer than a normal rollout, perhaps an hour or two more. [03:36] lifeless: The worst that could happen--that I can predict now--is that the two processes may need separate cache directories so that they don't lock each other out. [03:36] mkanat: so, I'm aiming at 10 minute downtime windows; thats a factor of 12 more than 'ok' [03:37] mkanat: or maybe you don't mean downtime window ? [03:37] lifeless: I don't see any downtime happening. [03:37] lifeless: Yeah, I mean total maintenance work time. [03:37] lifeless: To do the rollout. [03:37] it seems like we should easily be able to run it in parallel [03:37] since it's readonly etc [03:37] poolie: history_db gets written to quite a bit. [03:37] all this is fine [03:38] BUT ITS WORK. [03:38] logically readonly [03:38] who is going to do it? what if the guess is wrong? whats the impact on load? will the machine thrash? [03:38] poolie: Yeah. [03:38] indeed [03:38] lifeless: The machine should use considerably less memory with history_db than it does now. [03:38] I mean, I'm totally fine with whatever plan, my point has *never* been that its crap, its been that *we need to do work to make this usable in prod* [03:38] effort [03:38] time * energy. [03:38] lifeless, the point is, i don't see that there is any other sensible next step other than deploying this [03:38] lifeless: In an ideal situation, the rollout requires no attention from anybody--it would just be a standard merge and push. [03:38] which btw, this conversation is sapping for me. [03:38] i'm not saying you have to do it right now [03:39] but there's no point discussing how to make it possible to do other things off 1.18 [03:39] lifeless: I totally understand. :-) [03:40] poolie: But thats not what I argued *for*. I argued that other than oops fixing, which there are plenty of sensible things to do in the current lp branch, we're not going ot be undertaking major works for 6-9 months, based on my knowledge of jml's pipeline. [03:40] ok [03:40] so again we come back to assumptions [03:40] do you agree that nearly 20K unhandled exceptions a day is a little bad? [03:40] max thinks it is not easy to fix oopses in the current branch [03:40] Ah, I wouldn't exactly say that. [03:40] and do you agree that historydb is an *operational unknown*? [03:40] It was easy to fix the "no such revision" one. [03:41] lifeless: Actually, now that I understand it, I only agree minimally. [03:41] lifeless: Because codebrowse is already using on-disk caches. [03:41] lifeless, i'm only spending time on this because i think it needs to be improved and i want to see us take the best course on it [03:41] lifeless: This is just another on-disk cache. [03:41] mkanat: which we have several deployment options for [03:41] a) per branch [03:41] b) shared cross branch [03:41] c) memory only [03:41] d) huge one [03:41] e) ...? [03:42] lifeless: Right, and (d) requires no code work at all. [03:42] we've seen locking issues with sqlite already [03:42] so we have some undelivered work in trunk [03:42] we all agree that actually deploying it will take work [03:42] a) would be equivalent to our current cache story in terms of disk, but higher upfront scans and no benefit cross-branch [03:42] etc [03:42] lifeless: (d) is essentially what codebrowse is doing right now. [03:43] mkanat: I thought a) was [03:43] lifeless: Nope--one file in one directory. [03:43] mwhudson: ^ cache per branch, right ? [03:43] there would be little point in doing (a) [03:43] lifeless: currently? [03:43] yes [03:43] not sure actually [03:43] * mwhudson checks [03:43] lifeless: At least, AIUI. [03:43] mwh, for my education, where are you checking to find out? [03:44] poolie: lib/lp/launchpad_loggerhead/app.py in the launchpad tree [03:44] Oh right, launchpad-loggerhead. [03:45] mwhudson, i don't have that directory... [03:45] lifeless: it's one per branch currently [03:45] poolie: -lp/ sorry [03:45] poolie: It's a separate project, lp:launchpad-loggerhead [03:45] mkanat: not any more it's not [03:45] mwhudson: Oh! didn't know. [03:45] mwhudson: I don't have that directory, for whatever reason [03:45] poolie: lib/launchpad_loggerhead/app.py [03:46] right [03:46] so, back to facts, we're deployed with (a) today [03:46] shared between two instances [03:46] lifeless: Ahh. [03:46] with some threads each [03:46] Yes, in that case, history_db would be different. [03:46] ok, and are you saying that historydb is going to be slower in that setup? [03:46] poolie: No, just different. [03:47] lifeless: changing to one cache per would be very easy [03:47] jam says initial scan is thrice the time, subsequent operations are faster [03:47] mwhudson: I know, but its all time [03:47] mwhudson: for a maintenance squad without your context [03:47] mwhudson: who will be writing up what they figure out like crazy [03:48] lifeless: less time than this conversation has been going on for [03:48] lifeless: sure, they are welcome to talk to me :) [03:48] mwhudson: sure feels like it; or are you serious ? [03:49] lifeless: sorry, not quite sure i understand [03:49] mwhudson: 'less time than this conversation has been going on for' [03:49] lifeless: I'm pretty sure he was serious. [03:49] lifeless: yes, i was serious about that [03:49] ok [03:49] it's a few lines [03:50] + a test run :) [03:50] + qa to make sure it behaves [03:50] anyhow [03:50] yeah [03:50] sadly, a test run won't make any difference, but yes [03:51] poolie: so whats your proposal thats different to what I've suggested? [03:51] mwhudson: Where's the on-disk cache setup in app.py? I only see "self.graph_cache = lru_cache.LRUCache(10)". [03:51] lifeless: this diff will switch to using a single cache http://pastebin.ubuntu.com/558839/ [03:51] poolie: or are you trying to say I've got a bad axoim or assumption? [03:51] lifeless: keep both branches as they are [03:51] mkanat: cachepath = [03:51] lifeless: Ah, thanks. [03:51] if there's genuinely low fruit in 1.18, do that [03:51] file a bug saying "should upgrade to trunk" [03:51] do that in a safe way, but don't be scared of it [03:51] mkanat: the BranchWSGIApp constructor [03:51] mwhudson: Yeah, I found it. :-) [03:52] poolie: I'm not scared of it; honestly. [03:52] I'm assessing how long its taken mwhudson to do this in the past [03:52] the things that have gone wrong in the past [03:52] lifeless: Okay, I'm starting to agree with you more now that I see how codebrowse currently runs. [03:52] and our pipeline of stakeholder requests [03:52] lifeless: I was under the impression that it was doing (d). [03:52] it's fine if it stays in the pipeline until it's a priority [03:52] mkanat: god no, it would be flattened in an instant [03:53] lifeless: That's sort of what I thought, yeah. :-) [03:53] lifeless: i'm not so sure about that actually [03:53] mwhudson: that was humour [03:53] you might have some locking related fun [03:53] ok [03:53] I'm pretty sure we do benefit from the caches [03:53] lifeless: That's possibly resolvable by WAL, but that would indeed be infrastructural and integration work for sure. [03:53] but I'm also sure there is friction in there [03:54] which needs time and attention to detail [03:54] lifeless: Okay. Out of curiosity, are you sure because you have tracebacks, or just because you suspect? [03:54] poolie: I don't think putting it in the pipeline as a given makes sense; saying 'do something to make codebrowse better' is something we should put in the pipeline; one concrete thing we can do there is productionise and deploy history db [03:55] mkanat: have seen the odd thing go by [03:55] lifeless: Okay. [03:55] mkanat: such as sqlite lock contention errors [03:55] it's essentially already on the kanban towards the right hand side [03:55] poolie: I have to disagree. [03:55] lifeless: I think that it would probably be sensible and manageable to do oops fixing on trunk and 1.18 both simultaneously, and then schedule trunk rollout for a few months away. [03:55] Or on ~launchpad-pqm/devel and trunk. [03:56] poolie: theres a bunch of work been done on it, yes. But no team has been pushing it forward; its a new-situation for whomever comes along. [03:56] lifeless, because... it's not that far to the right if deployment hasn't been thoroughly considered? [03:56] lifeless: If you limit the changes to just oops-fixing until trunk is ready, I think you'd be fine. [03:56] https://lp-oops.canonical.com/oops.py/?oopsid=fixing [03:56] if jam's willing (not too scarred by lp-serve) we could ask him to do it with help from mwh and it would not be new [03:57] poolie: that, plus what I said, plus the work done hasn't been done with a view to the whole lp story around this; its been somewhat siloed off [03:57] poolie: if you want to do that, that would be awesome; don't underestimate it though - I mean, you know how gung ho I am about stuff generally, right ? [03:58] heh [03:58] poolie: this is *it can be done and there is a pie on the line collins* being concerned. [03:58] :) [03:58] lifeless: Given codebrowse's history, I don't blame you. [03:58] so in a way i would be ok with saying "this is really not a good fit, let's throw it out" [03:58] lifeless: Also given the fact that right now, it's far more stable than it's ever been. [03:58] ideally, if we had a specific reason for that [03:58] so if you want to do that, say so; if we leave trunk as-is, I'll setup a separate project to get code reviews and bugs visible to the lp team until we're converged. [03:59] but, if it's done and we don't think there's anything actually wrong, we just find it hard to deploy [03:59] that just seems like a bad reason [04:00] if true, it seems to mean some bugs can just never be fixed :( [04:00] poolie: we have a similar set of reliability and robustness concerns around soyuz [04:01] do we ultimately want to get historydb running or not? [04:01] poolie: I would think so. [04:01] i'd be fine with deciding that it was just a spike experiment and we want to take another go at it, if that's what we really think [04:01] compared to loggerhead, which has - I think - one person familiar with it still on the lp team, soyuz has 4 folk familiar with it, so we can actually react quickly to issues [04:02] from my perspective, its an experiment until either a) a feature squad is given codebrowse improvements as a card, and agrees that historydb is the way forward, or b) someone else does the work necessary to get it live [04:02] It's too bad I don't have enough time left to stick around and help with the rollout. [04:02] I can't prejudge what the feature squad engineers will decide makes sense [04:03] [well I can, but I'd hate it if someone did that to me, so I don't do it to them] [04:03] mkanat, it seems like organizing rollouts is a pretty hard thing to do unless you are actually full time on the lp team [04:03] lifeless: That's understandable. That might just be re-having a conversation that's already been had, though, from back when jam originally proposed history_db. [04:03] mkanat: it may be a nobrainer [04:03] lifeless: Sure. But I understand that the specific application to LP is still something to discuss. [04:03] I'm just being clear about what I can commit to confidently. [04:03] lifeless: Sure, I totally understand. [04:03] The primary thing that a designer has to deal with is the fact of uncertainty. [04:04] yup :) [04:04] are there any concerns about trunk other than that the deployment may be very bumpy (and i realize that may be a very large conrcern) [04:04] Yeah. It's something that a lot of people don't understand. They try to predict an unpredictable future. :-) [04:04] poolie: Unpredictable problems that I can't imagine now, based on the fact that there have been a lot of changes. [04:04] poolie: I haven't done a rev by revision review because I think thats a waste of time. [04:04] s/bumpy/& laborious whatever [04:04] poolie: once the card size is assessed as being > maintenance bucket, it goes into the queue. [04:05] poolie: history db alone is enough to do that for me [04:05] ok [04:05] so i propose to file a bug saying that historydb ought to be tried out and deployed [04:06] if in the course of trying it out, it seems that it is not a good solution, or too much work, or whatever [04:06] we can mark the bug wontfix and then pull it out of loggerhead's trunk [04:06] poolie: Where would it be tried out, though? There's no edge. [04:06] who will do the trying out ? [04:06] mkanat: we do have bazaar.qastaging.launchpad.net [04:06] lifeless: Oh, okay. [04:07] thats in the qa pipeline [04:07]