=== wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 244 - 0:[#######=]:256 [03:02] Project devel build #1,010: STILL FAILING in 4 hr 5 min: https://lpci.wedontsleep.org/job/devel/1010/ [03:32] Just when I thought I had great success in not breaking a truckload of tests with an MP, it fails buildbot. [03:39] jelmer: https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1313/steps/shell_6/logs/stdio <- your fix yesterday may be flaky. [03:39] Forcing. [03:58] * wgrant vomits [03:58] httplib2 has its own certificate store. [04:05] What is wrong with being sensible :( [05:21] wiki says wgrant or gmb is review bitch. Any chance of https://code.launchpad.net/~stub/launchpad/pgbouncer-fixture/+merge/73283 ? [05:21] heh, review bitch. [05:22] stub: Hm, changing to TCP connections? [05:23] Ah, I see, you don't want to overwrite the config. [05:23] We really need to fix lazr.config to not suck for this. [05:24] stub: Why are you overriding PGHOST? [05:24] wgrant: Because otherwise it will connect using UNIX sockets, and fail as the pgbouncer package does not (yet) set unix_socket_dir so they don't work. [05:25] stub: But you've set host=localhost in the connection strings. [05:25] stub: So setting PGHOST=localhost sounds redundant. [05:25] wgrant: I probably won't bother changing it either, as this better represents production [05:25] wgrant: It might be redundant now, yes. [05:25] stub: Also, you can hopefully use self.useFixture instead of self.addCleanup and setUp. [05:26] That sounds good. [05:26] I think the base Fixture from fixtures should support that. [05:26] Do you want me to pull the PGHOST bit if it is redundant, or leave it since it might be useful in the future if I do what I don't think I'm going to do? [05:27] stub: I think you should pull it out. [05:27] We have more than 650KLOC already. Let's not add more. [05:28] pypi has terrible google juice [05:28] py* has terrible lots of things. [05:29] stub: I don't think this is going to interact stunningly with PgTestSetup. [05:29] # Stash the name we use in the config if a writable config is [05:29] # available. [05:30] # Avoid circular imports [05:30] section = """[database] [05:30] rw_main_master: dbname=%s [05:30] rw_main_slave: dbname=%s [05:30] Why yes, that is revolting. [05:31] * stub wonders how it is working atm [05:31] Possibly PGHOST :) [05:31] ahh... yes :-) [05:32] 213 + # Database is still working. [05:32] 214 + assert self.is_connected() [05:32] s/still working/working again/ [05:40] Pushing with those changes. [06:14] * stub wonders why the ec2 image doesn't upgrade its own packages [06:18] Anyone around setup to update the ec2 test image? [06:18] Just need launchpad dependency packages updated [06:22] stub: I thought it did upgrade… anyway, I should be able to do that. [06:23] It doesn't :( [06:23] stub: What needs updating? It was last rebuilt a week ago. [06:23] Meanwhile, does anybody have any idea what the current crop of codehosting buildbot failures is all about? [06:23] Seems simple enough, but I don't want to run an hour long process right at the moment and might not have perms on ec2 either. [06:24] wgrant: Just pull in the newly build launchpad dependencies [06:24] jtv: I poked jelmer about them, as they might be related to the stuff he's landed recently. [06:24] c/build/built [06:24] launchpad-developer-dependencies [06:24] wgrant: he may have landed something unfinished… I also noticed a lot of recent lint in his name. [06:24] stub: pgbouncer needs a list of users to let through? [06:25] wgrant: yes [06:25] stub: What about *_ro and read? [06:25] stub: Or are we just going to hope no test that uses them ever goes near the fixture? [06:26] wgrant: I'll need to add them too. [06:26] StevenK: your AMI rev. 519 is the current, valid image? [06:26] jtv: He's on leave, but yes. [06:26] It was created last week. [06:26] thanks [06:27] read isn't a user though, so don't need that one added. [06:28] BTW, why do we use a 64-bit AMI? I can't imagine any of our test processes needing more than 3GB address space, and I thought x86 was likely to be faster than amd64 for python. [06:29] Because we run 64bit on production [06:30] Right, production is mostly amd64. [06:30] i386 tests are quite a bit faster, because RAM usage is roughly halved. [06:30] So you get tonnes of extra cache. [06:55] wgrant: looking into gina domination a bit more. Do I understand correctly that we want to mark SPPHs that have been superseded by newer publications of the same package as superseded, even if the package has subsequently "fallen out of" the Debian Sources lists? [07:00] jtv: Ideally. But that's probably only a problem for the first time, so we could do it as a one-off. [07:00] Well it happens to be what the dominator already does. It's just buried among heaps of other code. [07:01] It also looks like it could be made much faster if it had to. [07:04] wgrant: by the way, it looks like you're OCR today. In which case: https://code.launchpad.net/~jtv/launchpad/bug-836743/+merge/73267 plz? [07:04] The archivepublisher dominator should take <5s per context now. [07:04] Once we finish denorming SPN onto SPPH it will be trivial time. [07:05] jtv: However, we already know what's not published any more, so I think using the dominator in the normal case is pointless. [07:06] jtv: Looking at that review now. [07:06] Except we still need domination done—might as well. [07:06] Thanks. [07:06] jtv: Why? [07:07] Apart from the initial run, the algorithm is as I outlined in the bug. [07:07] Because we don't want to keep keeping Debian SPPHs in unjustified active statuses. [07:07] The dominator will not match what Debian does. [07:07] In what way? [07:08] Debian does not handle superseded packages quite as Ubuntu does. [07:08] Superseded sources are kept around in the indices until they are unreferenced. [07:08] By binaries, that is. [07:09] Yes, but doesn't that mean they're still superseded? [07:09] There is a difference between "superseded" and "deleted" after all. [07:09] That's something that is not clear. [07:09] Julian argues that anything in the indices is by definition not superseded. [07:10] Project devel build #1,011: STILL FAILING in 4 hr 7 min: https://lpci.wedontsleep.org/job/devel/1011/ [07:12] Soyuz currently works under the assumption that (in the indices == published). [07:12] I've argued that that should be changed, to allow us to gain Debian's enhanced supersededness stuff. But I don't think now is the time to be violating or changing that assumption. [07:13] Isn't that what we have pockets for? I imagine it's a valid assumption for ubuntu exactly because we don't do it that way. [07:13] We want to do it that way. [07:13] There's a critical bug for it. [07:13] It's unrelated to pockets. [07:15] jtv: Branch looks good. [07:15] Thanks. So in the future we want to keep multiple SPPHs "published" for the same source package name, distroseries, and pocket? [07:17] We want to keep multiple SPPHs with the same context (archive, distroseries, pocket) and name in the indices. It's not clear whether that means we should leave the status as Published. [07:18] I've traditionally argued that we shouldn't leave the status as published, but it has been a point of significant contention. [07:20] jtv: Does anybody use launchpad_langpack on staging these days? [07:21] # - Run Langpack ((re)-create a langpack DB that the [07:21] # translations team can use for testing. [07:21] That's very very old, and we used to use it even for production tarballs sometimes. Today, we may still want it for Q/A. [07:22] That sounds within tolerances of "no". [07:22] I don't know because I've been on Soyuz feature "rotation" since two interglacials ago. [07:22] Megafeature :) [07:23] Scope creep. It's the difference between your burndown rate and your burnout rate. [07:23] Well, it also should have been split into at least three separate features from the start :( [07:24] In a way, it was. They just kept being piled into the same feature rotation. [07:24] Well, yes, that's what I meant. [07:25] Anyway, about the superseding of source publications: what would determine for Ubuntu whether a package is still in the Sources list? Where would that information be kept? [07:25] datepublished IS NOT NULL AND dateremoved IS NOT NULL [07:27] Actually, probably datepublished IS NOT NULL AND datemadepending IS NOT NULL [07:27] (datemadepending is the date it was scheduled for deletion, not the date it was made pending, because 2005 hates us) [07:29] good morning [07:29] hi adeuring [07:29] hi jtv! [07:30] wgrant: it sounds to me then as if domination should be fixed to handle that uniformly for debian, ubuntu, and derivatives; and gina should merely be updated to set those date fields. [07:31] jtv: Possibly, but that is feature scale work. [07:32] Possibly, but then I don't see why I should solve that problem for just Debian right now in a short-term branch. [07:33] I don't think it makes sense to run a dominator over an imported archive, since it is archive publishing logic. The imported archive has the information we need right now. When we remove the (in indices <=> published) invariant, that may have to change. [08:25] http://flossmole.org/content/everything-you-ever-wanted-know-about-software-forges-code-forges-june-2011 [08:25] interesting link [08:25] that's a long URL. [08:25] (twss, yeah I know) [08:27] Ooh, interesting cloud. [08:28] LP apparently doesn't have announcements. [08:28] It does. Right? [08:28] The RSS thing that pops in the right corder of a project. [08:28] Yes. [08:29] Intriguingly, Trac is included as a feature in the feature matrix. [08:29] wgrant: hi [08:30] Hm, I thought we still provided DOAP. [08:30] Maybe we dropped that. [08:33] Hello [08:34] Wasn't someone talking about having javascript OOPSes? [08:34] I just saw this http://errorception.com [08:35] I don't know how it works, but apparently some kind of OOPS system. Paid for though :| [08:38] mrevell: hi === gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 244 - 0:[#######=]:256 [08:52] wgrant: it looks like the failing test was one that was flapping earlier, which supposedly should be fixed by the introduction of the newer Twisted [09:00] jelmer: Yeah, I thought it could have been that :( [09:01] But since you'd touched that area recently, I thought I might as well see if you had any thoughts or fault :) [09:01] wgrant: thanks for the utf8 fix [09:02] not sure wtf the extra blank lines are coming from but it's not exactly important [09:02] bigjools: The iterator probably stops at the start of the next entry. [09:03] bigjools: You can probably sensibly just call strip() on the changelog string. [09:03] as for the person/date line... that would need a change in python-debian methinks. [09:03] yeah, I was considering using strip but it might also be in the template [09:03] bigjools: python-debian doesn't give you access to the various components of the entry? [09:03] bigjools: gina does it somehow. [09:03] Although it has its own parser. [09:03] gina has its own parser [09:04] Heh. [09:04] it's do-able, but the right place is in python-debian [09:04] aha [09:04] I can do block.changes() I tihnk [09:05] bigjools: You could also set _no_trailer on the block. [09:05] Then __unicode__ won't include that line. [09:05] Hah. [09:05] unicode? [09:05] Rather than using the utf-8 stuff, use unicode() instead of str() [09:05] Its __str__ just calls __unicode__ [09:05] unicode... what unicode :) [09:05] which python-debian do you have? [09:06] Ah, I'm on oneiric. [09:06] Let's see lucid... [09:06] ;) [09:06] Package `python-debian' is not installed. [09:06] Huh. [09:06] Oh. [09:06] We still use the one in sourcecode/ [09:06] :( [09:07] We should update that and use proper unicode :) [09:07] Although that one does still support _no_trailer. [09:07] I love the way that the API expects you to poke private variables to change behaviour [09:08] You're not meant to poke them from the outside. [09:08] We're just doing unexpected things. [09:08] I'm not sure about your use of _raw_version either, but meh. [09:08] well, quite [09:08] the api is a bit rubbish [09:09] looks like I can just set _no_trailer then [09:19] wgrant: instead of decoding the utf8 in fetch_information it seems to make more sense to do it in aggregate_changelog [09:23] bigjools: Probably. [09:23] bigjools: I just did it there to get it unbroken. [09:23] np [09:38] AFICT wer are in a good position for a stable deploy, aren't we? [09:38] both rollbacks have landed. [09:39] henninge: Yes. [09:39] * henninge goes about starting that. [09:39] Once danilos' QA is done. [09:39] wgrant, was it not done? [09:39] Not a few hours ago when I last checked the deployment report. [09:39] danilos, wgrant: all green [09:39] So it is. [09:40] hmm, I should do QA! [09:40] doit [09:40] or should I? [09:40] * nigelb didn't didn't see a tag show up. [09:41] nigelb: It should be there in 20 minutes or so. [09:41] Actually, I'm behind. May have been there 10 minutes ago... [09:41] Ahh. [09:42] Revision 13818 can not be deployed: needstesting [09:42] Assignee: nigelbabu [09:42] *whee* [09:42] * wallyworld just got power back on after a 90 minute power outage :-( [09:42] * nigelb QAs [09:42] FUUUUU [09:42] I wasn't subscribed. [09:42] Why does assigning the bug to myself not auto-subscribe me :x [09:44] You won't be explicitly subscribed, but you'll still get notifications. [09:44] Ha, 13 minutes ago. [09:44] wgrant: can you think of any good reason why I should not make makeChangelog return a unicode changelog? [09:44] No wonder. [09:44] bigjools: No. [09:45] bigjools: There's little place for bytestrings in LP. [09:45] good answer :) [09:45] * bigjools does the file encoding trick [09:45] Hm? [09:46] # -*- coding: utf-8 -*- [09:46] Why? [09:47] That's an emacs hint [09:47] no it's not [09:47] isn't that a vim hint? [09:47] Python hint. [09:47] it's a PYthon hint [09:47] That's the critical bit. [09:47] However, it's probably overkill. [09:47] What do you think you are up to? [09:47] QA'd [09:47] I've had this conversation before [09:48] If I want to type human-readable unicode in Python source I need that [09:48] bigjools: Oh, this is for a test? [09:48] I guess that's OK then. [09:48] in factory.py [09:48] but yes [09:48] see the top of test_ppauploadprocessor.py for example [09:49] wgrant: A hint for you with fixtures: useFixture is the bomb. [09:49] lifeless: Didn't I just suggest that to stub? :) [09:49] I may have missed that [09:49] 15:25:55 < wgrant> stub: Also, you can hopefully use self.useFixture instead of self.addCleanup and setUp. [09:49] I was going off of the MP thread. [09:49] I am not guilty. [09:49] Ah. [09:50] I felt some of the issues needed discussion, so did it on IRC instead. [09:50] something I like doing after IRC discussion is to paste that into the MP [09:51] Probably should have, yeah. Will do so in future. [09:53] bigjools: You prefer using coding: utf-8 over \N{SNOWMAN}? [09:53] hell yes [09:53] Bah. [09:56] * jelmer wonders how you would type non-ascii characters on a UK keyboard [09:56] ♯ [09:56] buy a new keyboard! [09:56] Actually there's a character map thing somewhere you can use :) [09:56] jelmer: Compose keys solve everything. [09:56] Except for in oneiric, where I can't work out how to configure one... [09:57] GNOME 3 wins. [09:58] OTOH, the special keys handling dialog in GNOME2 excelled in being complex and confusing... [09:58] Yeah. [09:58] But I can't find anywhere to configure keyboard layouts at all any more. [09:59] it *was* originally an Emacs hint. Python just appropriated it. [09:59] In fact, the new gnome-control-center seems to take GNOME policy to the extreme and delete most of the options. [10:00] wgrant: yeah [10:01] Or perhaps Ubuntu's Language Support button has replaced the GNOME 3 regional settings one or something. === jtv is now known as jtv-eat [10:03] Meh. I should just stop. Not helping. [10:04] nigelb: hmm? [10:04] #launchpad, I presume. [10:04] YEah. [10:04] cheater is right, our UI is terrible, and attempting to argue otherwise is probably silly :) [10:04] LP was designed in a different era, and was never pulled out of it. [10:04] heh, I agree with that. BUt its not /that/ terrible. [10:05] It is to people who haven't used it before. [10:05] And even to those that have it is cluttered and awkward. [10:05] Also, I'm trying to say how the UX is different from other similar services. [10:05] And does not provide dashboards or anything else that contemporary social applications have. [10:05] I failed at that spectacularly.' [10:05] It is a good point, but difficult to argue successfully. [10:06] What would it take to build a dashboard like approach? [10:06] dashboards would be really nice, especially given we track not just stuff on launchpad but elsewhere as well. [10:06] Ahahah [10:06] Besides lots of crying. [10:09] btw, no StevenK these days? [10:09] StevenK is away this week. [10:09] Or is he using perl to delete large parts of launchpad in secret. [10:09] Well, not away, but not meant to be here. [10:09] wgrant: Like you were a few weeks back. [10:10] Shh. [10:22] Bad metaphors need to be shot down. [10:22] nigelb: shot down from where? [10:22] jml: WRT Facebook and Google+, congrats! [10:23] StevenK: thanks :D [10:23] jml: oh yeah! Congrats! [10:23] nigelb: cheers :) [10:23] jml: Pictures of the ring are *required* [10:23] jml: I'm adding my comments to the discussion in the other channel. [10:23] StevenK: haha, I just realized, I did pictures of the ring with which you proposed :D [10:23] nigelb: You did? [10:23] nigelb: was being witty, "shot down" is itself a metaphor [10:24] jml: haha. Damn. I fail at my own joke :) [10:24] or maybe that's what you meant [10:24] StevenK: Yeah, #ubuntu-women ^-^ [10:24] back then. [10:30] I need unicode help :( [10:30] whenever I have to do unicode in Python, I get frustrated [10:31] wgrant: do you know the appropriate runes to get makeChangelog working with a unicode changelog? the librarian addFile keeps blowing chunks on me [10:32] bigjools: Python makes it really hard. [10:32] jml: it does :( [10:32] bigjools: You'll need to encode the changelog before you pass it to makeLFA [10:32] bigjools: you should see the insane stuff mgz has done to get testtools have proper unicode handling. [10:32] wgrant: I am, it still bitches [10:32] bigjools: Bitches how? [10:32] UnicodeEncodeError: 'ascii' codec can't encode character .... blah [10:32] jml: Python 3's sort of much better, though. [10:32] bigjools: What is your .encode invocation? [10:32] jml: I can well imagine [10:33] wgrant: yeah. it's the cross-Python support that makes it hard. Would be better with actual real honest-to-goodness types, and banning the word "string" [10:33] content=changelog.decode("utf-8") [10:33] bigjools: encode. [10:33] ffs [10:34] jml: Yeah, I think they designed stuff like the bytes backports to 2.6 just to make everyone's lives hell. [10:34] I use "backports" loosely. [10:34] heh. yeah. [10:34] I end up mostly resorting to casting everything to bytearray instead. [10:34] Which actually has consistent behaviour... [10:35] this is just utterly confusing [10:35] bigjools: Oh? [10:35] bigjools: You were going completely the wrong way before. [10:36] encode/decode/unicode etc [10:36] That's not Python, it's just people being stupid and not using Unicode everywhere. [10:36] I'm confused about whenI need to use encode and decode [10:36] If your data is not binary or UTF-8, you are wrong. [10:37] bigjools: Librarian files are bytestreams. You need to encode before you put data into an LFA, and decode once you get it out. [10:37] Because Python 2 doesn't do text files. [10:39] jml: How's the world of app development, anyway? [10:40] Hopefully not too much Python unicode support. [10:40] wgrant: fun! I have a spec :) [10:40] wgrant: https://blueprints.launchpad.net/developer-portal/+spec/automagic-binary-packaging [10:41] unicode support only matters insofar as a testtools release would help me and many others at Canonical get things done faster [10:41] This reminds me... I must work out what is up with poppy-sftp's logging tomorrow. [10:41] bigjools: Have you touched that at all? [10:41] bigjools: It's completely screwed. [10:42] wgrant: no, not at all [10:42] So, codeswarm for Launchpad was fun watching. [10:42] bigjools: Normally doesn't log anything, except when you log something explicitly, then it starts logging everything 50-60 times. [10:42] and you can actually see exceptions that occur. [10:42] After three recursion depth exceeded errors in the log handler. [10:42] I remember that happening when I was adding ftp support, then it went away [10:43] never got to the bottom of it [10:43] jml: Hows your day job? :) [10:44] nigelb: Do you have it for Launchpad? [10:44] jml: So we're going for widespread package-all-the-things before we have sandboxing, or are we hoping sandboxing will be practical before this is mature? [10:44] StevenK: Yes. [10:44] nigelb: Can haz? [10:44] StevenK: http://www.youtube.com/watch?v=9Pe70fw83Zs [10:45] wgrant: more the latter [10:45] Lots of lifeless floating around ;) [10:45] wgrant: for commercial apps, which is my focus for now, we are already operating without sandboxing in much the same way that we do with partner or the things currently in the software center [10:46] jml: Yeah, but I don't think that can scale while maintaining responsiveness. [10:46] wgrant: agreed. [10:46] jml: I love your work, except for the bits where its limited to one arch :( [10:46] (well, for now) [10:46] nigelb: we're moving up to two! [10:46] oooh \o/ [10:46] * bigjools weeps at having 4 cores which doesn't help when everything bottlenecks on disk [10:46] wgrant: I think sandboxing is the next step. [10:47] jml: It should have been the next step for about 15 years, but yeah :) [10:47] heh [10:47] bigjools: tmpfs + LXC + 6 parallel test runs makes good use of cores. [10:47] sandboxing's already nearly practical - see arkose [10:48] (I'm sure jml knows about that already) [10:48] Yeah, it's getting there. [10:48] yeah, I do. [10:48] wgrant: I used to run parallel tests using kvm but really, everything bottlenecks on disk. [10:48] it's good enough for skype, which is pretty damn good [10:48] Although haven't had the play with Arkose that I'd like to. [10:48] (the above is not to be taken to mean "skype is pretty damn good") [10:48] I know tmpfs is supposed to help but I still struggled to speed it up [10:48] Might do that today, actually, since most of it's consumed by long weekend email recovery [10:49] cjwatson: I can see the tabloids "Ubuntu developer says skype is damn good!" [10:49] bigjools: Which is why I implemented the LXC-based system that lifeless and I have tested. Have a single base chroot, then start 6 COW guests using aufs on tmpfs. [10:49] bigjools: All writes go to tmpfs, and the chroot cache is shared between all of them. [10:49] It was entirely CPU bound when I tried it. [10:49] excellent [10:50] I need a new machine. [10:50] 2 cores is now old. [10:50] my 4 core is a Phenom, a bit old [10:50] Buddy of mine got an 8-core. Runs gentoo one it and says he has great build times. [10:50] the i5 feels quicker, even though it only has 2 (although has the pseudo extra 2) [10:51] (I repllied with "I use Ubuntu, no build time on my laptop at all!") [10:51] gentoo users are weird people :) [10:51] haha [10:52] jml: that spec looks scary - so.many.items! [10:53] nigelb: I like having lots of small items. [10:53] nigelb: gives me illusions of control and progress. [10:53] Ah, they're small. [10:53] Far easier to feel that you're making progress that way. [10:53] I have like one item from UDS that's going to take 2 cycles to do. [10:54] Hmm, https://myapps.developer.ubuntu.com/dev/ looks slick. Except I had to read about it on Rick's blog. [10:54] I wish ISD blogged on the planet. === matsubara-afk is now known as matsubara [10:55] anyone got a second to talk about a bug, and a possible bug? [10:56] err a possible patch that is :) [10:57] "patch" - the magic word! [10:57] G: Actually, you should just talk about it and someone will respond [10:57] nigelb: yeah, well my fingers were a timezone behind my brain :) [10:58] I know that feeling well :) [10:58] Anyway, I spotted https://bugs.launchpad.net/launchpad/+bug/837290 last night and thought "that looks wrong", I took a look/poke at the Javascript tonight, and think I have isolated the issue [10:58] <_mup_> Bug #837290: When there is no "Other bug subscribers" incorrect descriptive text is displayed < https://launchpad.net/bugs/837290 > [11:00]