[01:20] OMG, actual sun. [01:22] heh [01:22] I thought you guys had plenty of sun. oh wait, because winter? [01:23] Because winter and because Sydney. [01:23] Ah! [01:25] nigelb: The suburb I live in got 90mm of rain yesterday. [01:25] Woah. [01:28] wgrant: We need at least two DB patches. We need to drop NOT NULL from private and transitively_private [01:32] StevenK: False. They both have defaults. [01:34] Ah [01:35] Right, so they get set to false irrespectively to information_type and they can just get dumped. [01:36] StevenK: transitively_private default to true, but yes. [01:36] StevenK: We can just stop setting them and they will default to relatively sensible things that nothing will see anyway. [01:36] Then we can drop them. [02:02] wgrant: You were talking about bug_build_access_cache in 16-0 that I should look at? [02:07] StevenK: That sort of thing, yes. [03:32] wgrant: ping (LEP, impl notes) [03:37] ugh [03:37] this post-merge-dif-nuking really is disruptive. [03:37] https://code.launchpad.net/~lifeless/launchpad/memcache/+merge/109551 [04:32] lifeless: It doesn't even look like that branch has been scanned. [04:36] indeed [04:36] wedged-is-U [04:36] S [04:39] lifeless: bzr push -r -2 ... ? [04:50] lifeless: I approve of +49/-1182 [05:00] wgrant: Am I on the right track? http://pastebin.ubuntu.com/1036633/ [05:14] StevenK: that thing is landed :> [05:14] garyposter: I have checkmarkified https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#preview [05:14] jml: I'd like to know if https://checkmarkable.com/team/sthederj/checklist/1263/view would have helped your confusion around db schema qa etc the other day. [06:03] lifeless: Hmmm, memcache for garbo is useful. If we had some other way to record stuff between runs it could die a horrible death [06:23] * StevenK stabs the branch scanner [06:23] StevenK: Do you mean, like, some sort of persistent database? [06:24] stub: Haha [06:24] Its a pretty bad fit for garbo. If that is the last use of it, we should replace it with something suitable. [06:24] Keep the IMemcache interface for lazyness [06:25] stub: I think it's the last use. [06:25] And only a few jobs use it [06:25] One job, in fact. [06:25] And it could be replaced with sqlite easily enough or even something less like a hack [06:26] wallyworld_: Do you have a sec to look over https://code.launchpad.net/~stevenk/launchpad/drop-set-spph-packageupload/+merge/109776 ? [06:27] woah, LP has a new intern? [06:27] You guys are going to break him, aren't you? ;-) [06:27] StevenK: a sec yes, i'm about to go see aus vs japan :-) [06:27] wallyworld_: It's +1/-133, so it should be very quick [06:27] wallyworld_: rugby? [06:27] Soccer [06:27] World Cup [06:27] wat. [06:28] Soccer World Cup? [06:28] nigelb: football [06:28] o_O [06:28] world cup qualifier [06:28] aha [06:28] that makes better sense [06:28] We are going to lose horribly [06:28] yep :-( [06:28] I was following Eurocup and couldn't believe there was world cup going on. [06:29] StevenK: r=me [06:29] wallyworld_: Thanks. Go and enjoy the soccer. [06:30] will do. hope we at least make a decent game of it [06:34] StevenK: two jobs when I looked in trunk [06:38] lifeless: Yup, I just removed one in the MP wallyworld_ reviewed. [06:46] StevenK: ah, kk. [06:46] we could do a persistent transactional k:v store in the DB [06:46] or we could grab redis or something with persistent backing and use that. [06:46] memcache is fine I think today. [06:47] stub: sqlite in production would be more, not less of a hack :) [06:48] lifeless: yer, I'm not suggesting it is a briliant idea. just that there are alternatives. [06:49] There is a new hstore datatype in PG 9.1 for k,v stuff. [06:49] And json to match the existing xml datatype === matsubara_ is now known as matsubara [07:36] lifeless: Hi, just added them. https://dev.launchpad.net/DisklessArchives and https://dev.launchpad.net/LEP/DisklessArchives [07:40] I'll do an edit pass and then let elmo etc know [07:44] wgrant: we may want two leps; a horizontal scaling one covering the uploader bug etc, and this one. *may* [07:44] lifeless: Possibly, yes. [07:45] But we also need *something* for the whole picture, I think. [07:47] Can't fault your logic [07:48] uhm, we'll see. I'm applying humanizing to this doc just now :) [07:48] thank you for putting them together. [07:49] wgrant: you've let a lot of implementation detail leak into the LEP (I mention this so you don't think I'm crazy for deleting it). I'll ensure it survives somewhere. [07:51] lifeless: Probably. [07:53] good mornin [07:57] wgrant: could you check (e.g. on dogfood, or give me a query) the number of unique people uploading to ppa.l.n ? [07:57] g'morning adeuring [07:57] hi jelmer [07:59] lifeless: SELECTing. === almaisan-away is now known as al-maisan [08:28] wgrant: https://dev.launchpad.net/LEP/DisklessArchives edited, please check I didn't trash your intent :) [08:31] lifeless: Looks reasonable. [08:31] Thanks. === jamestunnicliff_ is now known as jamestunnicliffe [09:18] wgrant: still around ? [09:18] lifeless: A little. [09:19] lifeless: 'sup? [09:20] so you claim external acls are cached on the full request path. [09:20] But [09:20] makeExternalAclKey [09:20] claims otherwise [09:20] I want to check why you say that before I delete it as wrong :) [09:21] wgrant: ^ [09:22] lifeless: makeExternalAclKey? Is that some Squid thing? [09:22] yes [09:23] lifeless: Squid can't know that we only care about the first two segments of the path. [09:23] its the thing that makes the helper cache key from the helper parameter format string [09:23] wgrant: thats what the third helper is for. [09:23] wgrant: it does that, trivially, and tags the request. [09:23] Well, Squid itself doesn't cache it. One of the helpers does, if there are two. [09:23] Does the acl helper get the tags? [09:23] The rewrite helper doesn't. [09:25] so, there are three helpers [09:25] mapper [09:25] it takes url, returns ppa string, does no DB lookup at all. Purely local. [09:25] it is cached by squid on full url. [09:26] authenticator [09:26] That's the ideal situation. [09:26] it takes the auth header + the ppa tag, does DB lookup (via webservice). [09:26] its cache by squid on the auth header value + the ppa tag value. [09:26] But I didn't see a way for the mapper to execute before the others and tag the request in a way they could see it. [09:27] you use it in an acl earlier. [09:27] we may need four helpers in fact, we need to detect ppa.l.n requests to private ppas. [09:28] Ahh. There's an %EXT_TAG new in 3.2 [09:28] the mapper always succeeds, and you say http_access deny !mapper [09:28] (which isn't in Ubuntu) [09:28] or some such. [09:28] just %TAG should be fine, no ? [09:28] Is that a thing? [09:29] ah no [09:29] misread the code. [09:29] So we need that ported, or to run 3.2. [09:29] noting. [09:29] "%EXT_TAG tag= value returned by previous external ACL calls. Tag may not be altered once set." [09:29] lifeless: So we can't stack tags. === gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | Welcome our new intern: ivorykr8 | On call reviewer: gmb | Firefighting: - | Critical bugs: 3.47*10^2 [09:30] lifeless: I was handling public-on-private by an arg on the the librarian mapper func, but something nicer might be better. [09:31] wgrant: oh, two rewriters? [09:32] lifeless: Or just pass the hostname in. [09:32] But two rewriters might be cleaner. [09:32] well, a public and private ppa may map to the same file [09:33] I propose just having one rewriter that doesn't care about public/private, and do the public/private check separately. [09:33] for public requests we check the ppa is public, for private ones we check its authed and their creds match [09:33] I guess the extra public roundtrip doesn't suck too much if we're using a separate API service that isn't LP. [09:34] lifeless: [09:34] lifeless: that's the checklist at the top of the process page, but with some detailed notes [09:35] lifeless: I made edits to the wiki page as I went, correcting sources of my confusion or uncertainty [09:35] lifeless: So we just have a separate verify_public external_acl_type? [09:35] Which uses a separate helper [09:36] jml: it is, but its also a stateful checklist so you can work through it as yor patch progresses. [09:36] jml: bayme thats not helpful. [09:37] wgrant: yes, and use squid acl rules to route to one or tother. [09:37] lifeless: Right, if you look at my sample config it's trivial to do. [09:37] lifeless: that might help. I couldn't say. I don't feel I lost track of where I was at any point, despite it being a multi-day process [09:38] wgrant: we're going to want a small patch to squid [09:38] wgrant: to let the helper turn off caching for failures. [09:38] incidentally, I made this: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AsE2-aW-RE8adG5vbW1nN3dJV0o0WUVKSElFRE9UUFE#gid=0 [09:38] lifeless: negative_ttl=0? [09:38] jml: we're running an experiment with checkmarkable, if you want to help that you could use it next time. [09:39] lifeless: ok. [09:39] wgrant: oh, that will do fine :) [09:39] wgrant: I forgots. [09:39] lifeless: external_acl_type has it, not sure if url_rewrite_helper does [09:39] But I assume so [09:39] lifeless: I can't say it's grabbed me. [09:39] * jml *sighs* [09:39] jml: its trying hard :) [09:39] I'm just a little sick of rattling my tin cup every time I want a review. [09:39] * jml drags it out again [09:39] Alms for the poor? https://code.launchpad.net/~jml/launchpad/remove-top-tests/+merge/109664 [09:40] url_rewrite_program doesn't have any such config. Probably somewhere else determines it. [09:40] gmb: ^ [09:40] Spare a penny for poor ex-leper? https://code.launchpad.net/~jml/lp-dev-utils/apply-top-tests-fixes/+merge/109665 [09:41] jml: remove-top-tests r=me [09:41] wgrant: rewriter gets user= passed to it, if we want it [09:41] jml, Looking now. [09:42] wgrant: src/redirect.cc redirectStart(ClientHttpRequest * http, RH * handler, void *data) [09:42] gmb: I grabbed the LP one already [09:42] StevenK, Ok. [09:43] wgrant: I'm not sure the rewriter has caching; we may need a aggressive cache layer just for it. [09:43] ... the worrying implication, carrying on from jml's script, is that StevenK is somehow the messiah. This has far too many unpleasant implications for me, so I'm just going to look at the lp-dev-utils branch and STFU. [09:43] lifeless: I don't think user= is useful.extuser= might be. [09:43] lifeless: Right, we'd basically need completely custom caching for the rewriter anyway. [09:43] wgrant: rephrase, I'm reasonably sure it doesn't have caching. [09:43] wgrant: custom, how so ? [09:43] lifeless: As I detail in my musings on the different cache coherency constraints. [09:43] lifeless: pool/ should cache forever and then some. [09:43] dists/ should not [09:43] wgrant: you forgot that the cache keys are based on the rewrite value [09:43] gmb: Unpleasant for you [09:43] wgrant: unless you disproved that. [09:43] ? What about ME!? [09:44] lifeless: I mean the lookups. [09:44] StevenK, You're the one that's going to end up nailed to a tree. [09:44] gmb: he's not the messiah. he's a very naughty boy. [09:44] lifeless: Pool file rewrites never have to hit the DB ever again. [09:44] jml: I was waiting for that. [09:44] lifeless: Because they never change./ [09:44] wgrant: ah! That was entirely unclear. [09:44] StevenK: thanks for the review. [09:44] * lifeless reedits. [09:44] jml, I'm so, so sad to say that I didn't actually see that coming :) [09:44] Hahaha [09:45] gmb: you need to get out more [09:45] jml, Anyroad up, your lp-dev-utils branch is approved. Want me to merge it for you? [09:45] lifeless: "Caching of archive path to librarian URL rewrites is a bit of an unresolved issue. While pool/ lookups should be cachable forever without requiring revalidation" [09:45] lifeless: Pretty explicit :) [09:45] gmb: yes please. I'll land the removal branch. [09:45] czajkowski, This is not, in fact, something with which I would disagree. [09:45] wgrant: and yes, entirely unclear :) [09:45] s/yes/yet/ [09:46] lifeless: I should have mentioned, though, that we'd confirmed Squid cached the content based on the post-rewrite URL [09:46] lifeless: Howso? :) [09:46] wgrant: I had no sleep sunday night. *anything* may fail to be unclear. [09:46] Heh [09:46] May fail to be unclear? [09:46] Heh [09:46] Anyway, I await your edits. [09:46] czajkowski, Still, only 1 week and I get to catch the bollocks o'clock train to our shiny new HQ. [09:46] gmb: whoo shall see you there with bells on [09:47] \0/ [09:47] wgrant: soon, I think. [09:48] jml, Merged and pushed. [09:49] wgrant: I think we want memcache in there. [09:49] wgrant: if you disagree, I'll step you through it. [09:50] wgrant: either that, or we make the mapping step so lightweight we don't need a cache at all. [09:50] lifeless: memcache is probably a good place to cache this, yes. [09:50] wgrant: I haven't gotten into the schema implications of that as yet. [09:50] Hm? [09:50] I deliberately left the caching mechanism unspecified. [09:50] memcached may well be a good option. [09:51] Not sure what you mean by schema implementations. [09:51] implications [09:53] well, what does it take to make url->librarian url mapping take 3ms [09:53] or, lets be generous, 5. [09:53] With a bit of work the DB can answer it is about 2-3ms. [09:53] s/is/in/ [09:54] But we should cache it to avoid hitting it too often. [09:54] gmb: thanks. [09:54] If we use something shared like memcache we can even have the publisher shootdown the index caches. [09:54] wgrant: well, at 3K rps, we're looking at 9 seconds worth of load/second. [09:54] Rather than revalidating every time. [09:54] Or having a TTL of like 5s. [09:54] so we'd need 9 cores at peak. [09:55] We can easily take that, but let's not. [09:55] and 3 at normal, which we can trivially do. [09:55] wgrant: lets also avoid writing something we don't need :). I am going to call it out, but also let it be a thing to test and validate, not blindly do. [09:56] That's true. [09:56] it looks like the database rollouts have been blocked for a few days on a non-QAd bug: http://launchpad.net/bugs/1007333 [09:56] <_mup_> Bug #1007333: update_database_disk_utilization fails with GIN indexes < https://launchpad.net/bugs/1007333 > [09:56] anything I can do to push that along? [09:56] * jml confesses to knowing nothing as yet about GIN [09:58] jml: Not blocked for a few days. [09:58] Just that we have other things to do tonight, and .au wasn't here yesterday. [09:58] wgrant: perhaps I'm misreading lpqateam and the bug report, but it says it was marked as needing QA on Jun 1 [09:59] THat's possible -- we've had a lot of backlog to get through. [09:59] ok. [10:02] wgrant: what do you mean by 'migration path for PPAs' ? [10:02] wgrant: do you mean 'we need to be able to bring up and validate the new environment before breaking what works on germanium? [10:02] It was also a two part landing which would have confused things. [10:02] Second landing didn't happen until Friday? [10:03] * lifeless assumes he does [10:03] lifeless: We probably want the ability to run both concurrently for a time, but also we need to be able to migrate to the new one. [10:03] Sorry, distracted by pgbouncer :) [10:03] wgrant: what does that mean though ? [10:03] wgrant: we have two sets of boxes [10:03] lifeless: eg. we need to load all the Packages and Sources and everything into the DB. [10:03] lifeless: And keep them up to date. [10:04] Without taking down PPA publishing for a day. [10:04] wgrant: I don't think we do :) [10:04] qa-ok since staging still works [10:04] Oh? [10:04] lifeless: What is your cunning plan? [10:05] wgrant: If we bring up the new system, and run a 'garbo' job across it to regenerate everything, filling in the missing data, what is the impact on germanium [10:05] lifeless: The garbo job will take more than 5 minutes. [10:05] wgrant: so... [10:05] lifeless: So there won't be holes any more, but stuff will be out of date. [10:05] wgrant: we make the existing publisher make holes. [10:06] lifeless: So the publisher knows about the new DB structures, but only to shoot them in the head? [10:06] its a thought [10:06] Perhaps. [10:06] I don't want to make publishing on germanium *slower* [10:06] that scares me, for hopefully obvious reasons. [10:06] That's a reasonable point. [10:06] so my proposal is to bring up a parallel structure [10:06] But the extra load should be minimal. [10:06] and teach it to cope with germanium doing its thing [10:07] but ignore that germanium did its thing, or detect stale, or something. [10:07] I think that's probably more work. [10:07] But it is indeed a reasonable thought. [10:07] anyhow, now I know what you mean, I can clarify. [10:08] Great. [10:17] any tips on how to use the LP web ui when it's running within an LXC container? [10:17] I'm guessing I should just copy /etc/hosts but set it to the lxc instance's ip [10:19] jml: https://dev.launchpad.net/Running/LXC [10:19] Step 10 is https://dev.launchpad.net/Running/RemoteAccess [10:19] wgrant: thanks. [10:20] buh wuh [10:21] apache2 isn't installed? [10:21] That sounds quite extraordinary. [10:21] Sure you ran rocketfuel-setup? [10:21] Is launchpad-developer-dependencies installed? [10:22] In order: Yes it does. I think I did but I am not certain. Yes. [10:23] observe: http://paste.ubuntu.com/1036945/ [10:23] Oh right, rocketfuel-setup installs apache separately later on. [10:23] Perhaps it died early and you failed to notice? [10:24] REQUIRED_PACKAGES="launchpad-developer-dependencies apache2 apache2-mpm-worker libapache2-mod-wsgi" [10:24] for pkg in $REQUIRED_PACKAGES; do do_install; [10:24] done [10:26] heh. [10:26] wow. [10:26] jml: Re. bug #1011611 I wasn't saying it was reasonable, just that it might be helpful for diagnosis. [10:26] <_mup_> Bug #1011611: Not possible to file embargoed security bug < https://launchpad.net/bugs/1011611 > [10:26] ? [10:26] wow? [10:27] it's like dependencies, but in shell. [10:27] Yeah [10:27] Pretty awesome. [10:27] We don't need no packaging system. [10:28] ... [10:28] wgrant: (have added this to the bug report) To clarify: clicking on "Public" brings up the chooser in my browser (Chrome) and from there I can change it and submit. I would never have thought to click on it, since there's absolutely no indication in Chrome that it's anything other than a declaration of fact (This bug will be Public). [10:29] jml: Yeah. [10:29] If you mouseover it it gets an underline, but that's it. [10:29] in Firefox it has a drunken exclamation mark. [10:30] :\ [10:32] I wonder if there's a more declarative way to say "LP needs these apache modules enabled in order to run" [10:33] I don't believe so. [10:35] yay running LP [10:35] But can you browse to it? [10:35] (maybe I'll need to do this again next time I lxc-start?) [10:35] Need to do what? [10:35] The Apache setup is one-off [10:36] Running make run or make start needs to be done every time. [10:36] wgrant: install apache, enable 5 modules etc. [10:36] these virtual machines confuse me a little. one of the ones I use resets to a template whenever I use it. [10:37] lxc-start shouldn't. [10:37] lxc-start-ephemeral is the ephemeral variant. [10:37] wgrant: so [10:37] * jml decides that the only sane thing to do is to eat the last of the Wensleydale. [10:37] wgrant: primary archive + diskless [10:38] lifeless: lalalalala [10:38] wgrant: I just wrote this: [10:38] split the work in these scripts between on-disk logic and non-disk logic, and migrate the non-disk logic component off of germanium early in the development process, reducing its current load, and giving a single source of logic for the new schema as it comes of age. [10:38] wgrant: how would that interact with primary archive - if we did this, would it *interfere* with it. [10:39] lifeless: It'd mean redesigning the primary archive publisher. [10:39] wgrant: e.g. I'm thinking the first stage would generate jobs to do the spooling to disk etc [10:39] Well, that could work. [10:39] wgrant: because its still apt-ftparchive ? [10:39] lifeless: Right, we can't generate primary archive indices outside the actual on-disk archive. [10:39] lifeless: And because we need to maintain the on-disk archive we need a queue of disk files to publish. [10:40] wgrant: so, we'd need a kludge for the primary archive, to not generate indices upfront, but the rest of the work could be partitioned sanely ? [10:40] lifeless: That's traditionally SPPH WHERE datepublished IS NULL [10:40] lifeless: But in your proposal that would be maintained separately. [10:40] wgrant: that queue could be explicit as job entries though, yes? [10:40] So we'd a set of events to publish the files. [10:40] Yes. [10:40] I think this is the best approach so far. [10:40] Yeah [10:40] It means rewriting a bit more. [10:40] It's by far the cleanest. [10:41] It's a significant amount of extra work. [10:42] I discounted it as being too much work, so I don't really have a good estimate of how much work it would be. [10:42] But it basically means rewriting the entire backend. [10:42] Rather than just extending it in a couple of places. [10:43] we can let james_w and jml decide. [10:43] I suspect that 'more work' here giving a more debuggable and understandable system is well worth it. [10:44] probably a smaller system too when I think about it. [10:44] bigger change from what we have today - sure. [10:44] It's also throwing away 7-year-old tried and battle-tested code, and forcing people who don't know Soyuz to rewrite Soyuz :) [10:44] wgrant: can you expand a little on the concurrency issue with parse-ppa-apache-access-logs ? [10:44] But they do have packaging experience, so I guess all is not lost :) [10:44] wgrant: the latter is a good thing [10:45] wgrant: fresh eyes, no stockholm syndrome. [10:45] lifeless: Redesigning Soyuz perhaps. [10:45] wgrant: precisely. [10:45] Rewriting Soyuz with the same data model and all the boobytraps it has? No. [10:45] now, -logs thingy. [10:45] lifeless: Well, we currently parse n lines at a time. [10:46] lifeless: And then update the counts all at once. [10:46] lifeless: If n is too large, we'll get lots of rollbacks. [10:46] As there'll be conflicting count updates. [10:46] so by problems you mean 'lots of waste' vs 'bad data' [10:46] There's also the ParsedApacheLog problem, but I may be too paranoid there. [10:46] We currently identify a log file by its first line. [10:46] antything else ? [10:47] If two logs happen to have the same first line, they'll be confused. [10:47] THat's all I know about. [10:49] wgrant: whats the cycle for log parsing [10:49] cron? /15 ? [10:49] lifeless: cron, disabled [10:49] But something around there. [10:50] * wgrant checks. [10:50] ok [10:50] wgrant: and how long does it take when it runs? 2-3m ? [10:50] It's */30 [10:50] I don't know how long it takes nowadays. [10:50] It was set up before I worked here, and I haven't had cause to look at it since. [10:51] So I have no clue how long it ever took on prod. [10:51] Ah, we still have logs. [10:51] * wgrant looks [10:52] lifeless: Seems to take 5-10 minutes at load of about 6. [10:54] man, it would be great if pagetests handled unexpected form submission errors by showing the unexpected form submission errors. [10:55] jml: The appserver just says "Unexpected form data". It doesn't give any extra data. [10:55] wgrant: I mean in cases where validate() shows errors but the test wasn't expected it to [10:56] Oh [10:56] Handy. === matsubara is now known as matsubara-lunch [10:58] jml: ...pagetests... [10:58] jml: is all I saw. [10:59] well, it'd also be great if Launchpad had an app server that wasn't also a web server. [10:59] i.e. if the web UI was an api client [10:59] jml: funny you should mention that. [11:55]