[00:19] Project windmill-devel build #268: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/268/ [00:28] lifeless: Your LXC guide is vastly superior to the U1 one. [00:28] LXC looks really nice. === spm` is now known as spm [00:40] Project parallel-test build #62: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/62/ === cinerama_ is now known as cinerama [01:05] Yippie, build fixed! [01:05] Project db-devel build #657: FIXED in 5 hr 53 min: https://lpci.wedontsleep.org/job/db-devel/657/ [01:10] jelmer: You seem to have removed the custom KDE layout from bzr-svn on the basis that they've moved to git, but I see new svn revs yesterday... [01:13] wallyworld_, do you have time to mumble [01:14] sinzui: yes, starting mumble now [01:19] lifeless: Hm, did you run into trouble with needing to run buildout under linux32 when using an i386 container on an amd64 kernel? [01:26] Project windmill-db-devel build #418: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/418/ [01:28] wgrant: I'm just fiddling with that now [01:29] wgrant: yeah the u1 seemed to ignore issues ;) [01:29] * StevenK sighs at the topic. Wasn't the critical bug count only 205 yesterday? [01:32] wgrant: feel free to tweak the page! [01:32] wgrant: hallyn has a magic lxc build coming to his ppa that will automate the bind mount [01:32] lifeless: I'm not sharing my ~, but it all seems to work OK as long as make always runs under linux32. [01:33] Otherwise buildout uses x86_64 meliae :( [01:33] interesting [01:33] Surely it doesn't try to parse uname... [01:33] Still, it is buildout. [01:33] that could be because arch reports 64bit (which is uname based I wager) [01:33] wgrant: file a bug ? [01:33] Maybe/ [01:34] arch claims equivalent to uname -m [01:34] Now, automated COW-on-tmpfs LXC parallel testing... [01:35] LVM [01:35] Slow! [01:36] But maybe. [01:36] 'meh' [01:36] Well, I need a faster disk. [01:36] And/or SSD. [01:36] only deltas get written [01:36] Sure. [01:36] you may need more bandwidth available [01:36] how many cores? [01:36] But I don't care about persistence. [01:36] 6 [01:37] doit [01:37] testr --paralell [01:37] I've been thinking about adding a profile option [01:37] testr --parallel --profile=lxc [01:39] Project devel build #828: FAILURE in 5 hr 23 min: https://lpci.wedontsleep.org/job/devel/828/ [01:44] lifeless: How does --parallel divide the tests? [01:44] by time [01:44] Huh? [01:44] testr records the runtime of tests [01:44] How does it know about new tests? [01:45] they get round robined [01:45] Right, but how does it know to dispatch them? [01:45] it runs --list-tests [01:45] Ahh [01:45] I see. [01:46] if you supply a list it could skp that [01:46] wgrant: linux32 on lxc-start [01:46] Ahh, that could work. [01:46] s/could/does [01:47] its in the wiki page now [01:48] sudo lxc-start -n lucid-test-lp -d usr/bin/linux32 sbin/init [01:48] robertc@lifeless-64:~$ ssh 192.168.122.44 [01:48] $ arch [01:48] i686 [01:48] wgrant: so you're not bind mounting; nfs? [01:49] lifeless: Nothing at the moment. [01:49] Since I don't want to break my external environment. [01:49] Which rebuilding it for i386 will. [01:49] another reason to use packages for deps [01:55] wgrant: the eggs are arch specific [01:55] wgrant: so nothing should break [01:57] lifeless: It's not just eggs. [01:57] lifeless: Some of sourcecode/ has some binaries. [02:00] Oops [02:01] ? [02:02] lxc-destroy does not just shut down the container. [02:02] -stop is what you wanted [02:02] Yeah, I'm too used to Xen. [02:02] but its glitching with lucid + arch diff for some reason [02:03] Also, I ended up running two aufs clones of a container while still running the original container. That doesn't work very well. [02:10] heh [02:11] wgrant: the binariies in subverypy etc are also arch guarded paths. [02:12] https://lpstats.canonical.com/graphs/LPProjectCriticalBurndown/20100624/20110624/ is upsetting. [02:12] lifeless: IIRC it's pygettextpo or pygpgme [02:12] ah, yes [02:12] bad gettextpo [02:12] urgle, we symlink gettextpo.so ?!? into lib [02:13] Of course! [02:13] we should import gettextpo.gettextpo, then we could indirect sensibly [02:14] Yes, but that wouldn't be ancient and crufty enough. [02:15] and I think I'm finally setup [02:15] with a tonne more memory available. [02:15] and a 32 bit build for leaner tests [02:24] ugh, something is generating awful spew and breaking testr -- -t codehosting :( [02:36] :( [02:43] File "/home/robertc/source/launchpad/lp-branches/working/lib/canonical/launchpad/scripts/__init__.py", line 62, in execute_zcml_for_scripts [02:43] 'Setting up Zopeless CA when Zopefull CA is already running' [02:43] AssertionError: Setting up Zopeless CA when Zopefull CA is already running [02:44] Yay [03:13] that was unexpected === lifeless_ is now known as lifeless [03:14] Oh? [03:14] I'm guessing OOM kill got gdm [03:14] Nice! [03:14] Haha [03:15] Certainly fixed the memory pressure, though [03:15] given I was watching a railsconf video I can't really be sure [03:52] StevenK, wgrant: does either of you have time to show me how to create a sync upload on dogfood? [04:15] wgrant: 15:11 < hallyn> lifeless: do'h, that was easier than i thought, lxc already does the right thing, you just need 'lxc.arch = i686' in the config file. [04:40] lifeless: Ah! [04:40] lifeless: Which channel is that? [04:40] #ubuntu-virt? [04:42] Why is my pipe to the DC so terrible :( [04:42] wgrant: #ubuntu-server [04:50] also, https://bugs.staging.launchpad.net/ubuntu/+bugtarget-portlet-bugfilters-stats [05:03] what part of our code would be doing fork in twisted ? [05:03] Upon execvpe sleep ['sleep', '2'] in environment id 280573852 [05:03] :Traceback (most recent call last): [05:03] File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/process.py", line 412, in _fork [05:03] self._setupChild(**kwargs) [05:03] ... [05:03] File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/posixbase.py", line 171, in wakeUp [05:03] util.untilConcludes(os.write, self.o, 'x') [05:03] File "/home/robertc/source/launchpad/lp-sourcedeps/eggs/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/python/util.py", line 783, in untilConcludes [05:03] return f(*a, **kw) [05:03] OSError: [Errno 9] Bad file descriptor [05:21] Project parallel-test build #63: STILL FAILING in 1 hr 10 min: https://lpci.wedontsleep.org/job/parallel-test/63/ [05:23] anyone around that knows our mm integration ? [05:24] I know a liiiittle. [05:24] Enough to debug it mostly. [05:24] so [05:24] lib/lp/services/mailman/tests/test_mm_cfg.py [05:24] runs in FunctionalLayer [05:24] does FunctionalLayer *start* mailman ? I thought it didn't. === _mup__ is now known as _mup_ [05:24] thus, I am confused. [05:25] it wants a /var/tmp/mailman directory [05:25] this is unsafe for parallel testing [05:25] I want to make it dynamic [05:25] Doesn't buildmailman create /var/tmp/mailman? [05:25] (vomit) [05:26] Barry wants to kill buildmailman [05:26] Yeah, I think it does. [05:26] I fully support this [05:26] buildmailman configures mailman into /var/tmp/mailman [05:28] lifeless: So, I think you probably want to run them in a layer that is non-parallelisable. [05:28] Or just delete mailman. [05:29] why? [05:29] Why what? [05:30] why non-parallelisable (btw that does not exist) or delete (don't we use it?) [05:30] We use it, but it's crap. [05:30] Project windmill-db-devel build #419: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/419/ [05:30] And it's hard to efficiently parallelise because you'd probably need to rebuild mailman to change the path. [05:30] breaking lists.launchpad.net would probably be considered a regression [05:30] Quite possibly. [05:32] I can't quite tell if you're trolling or not [05:33] We should be able to setup a transient mailman per test runner fairly directly [05:33] Directly but slowly. [05:33] I have a fixture around that creates a python module [05:34] lib/mailman isn't even importable [05:34] it shouldn't be under lib [05:39] 23 seconds [05:45] anyone got attachments to /var/tmp/mailman ? [05:45] I'd like to shove it under the source tree [05:45] e.g. ./var/mailman [05:46] I'd like to kill it from the tree entirely and move to MM3 [05:52] we'll want a network fake that we can use to make sure we interact with it correctly [05:53] I'd love to see that done and it move out of tree [05:53] * wgrant throws rocks at aufs. [05:56] I think I need to tweak fixtures a little first [05:57] hey there wgrant! [05:57] Hi jtv. Did you work out what you needed? [05:57] Nope [05:57] StevenK possibly knows, or I can work it out. [05:57] Big J said it was easy though [05:58] Haven't heard from him either. [05:58] But it was a feature introduced by your squad, so you should know :P [05:58] did he say 'a small matter of programming' ? [05:58] Unlikely [05:58] Since he and I wrote the support [05:59] I suppose I can just run "make harness" on dogfood, but it'd be nice if I had a UI way to do it. [05:59] jtv: the DSD sync thing will create jobs over an FF-configurable threshold, won't it? [05:59] When the job runs, it will check if it's allowed. If not, it will put itself in the queue. [05:59] No, that's not the case anymore. It has to do that in all cases now. [06:00] Do we cron DSDJobs now? [06:00] On dogfood, I mean? [06:01] I'd check the crontab. [06:04] ah — distroseriesdifference_job.py, every 2 minutes [06:05] StevenK, wgrant: mind if I sync some packages on df for a while? [06:05] jtv: Go ahead. [06:05] It won't love you, but go ahead [06:06] StevenK: I can't live without love, but I probably can without yours. :) [06:06] Well, this seems to be working OK. [06:06] (Hmm the error reporting on +localpackagediffs actually works pretty well for me now… needs a bit more tweaking but it's clear there's an error and it takes up no new space) [06:07] I have three tmpfs aufs COW LXCs running the test suite. [06:08] want want want! [06:09] Now I just need to automatically provision them (mkdir, mount, mount, sed, lxc-start does it), and tie it into testr. [06:09] well that's the interesting part, isn't it? [06:10] Well, aufs is infamous for its flakiness. [06:10] I don't think I know it. [06:10] We used it at uni for a while and had to give up. [06:10] It's like unionfs, except less flaky. [06:10] But still flaky. [06:10] I believe it's still used for livecds. [06:10] wgrant: nice [06:11] wgrant: you'll want to bring one up for --list-tests [06:11] So you're giving each container its own union mount overlayed on the branch setup? [06:11] jtv: Right, I have the base container with postgres and LP installed. [06:11] Then I create a tmpfs for each instance. [06:12] Mount that as an RW overlay over the RO base container for each instance. [06:12] wgrant: hows the memory pressure ? [06:12] lifeless: It's i386, so it's nothing. [06:12] * jtv underlines "per-directory journaling QoS" on his filesystems wishlist [06:12] I only have 8GiB of RAM and killed Chromium recently. But I have 3GB free and a couple of GB cached. [06:12] wgrant: well, when you get to 6 instances, it may be less -nothing- :> [06:12] True. [06:12] Let's see what happens if I spin up a few more... [06:13] Wouldn't it be great if you could tell a regular filesystem, "feel free to forget the entire contents of this directory on next mount, if it's more convenient than recovery"? [06:14] tmpfs? [06:14] jtv: Indeed. [06:14] StevenK: no, that ties the policy to the mechanism. And you have to manage it, e.g. unmount it when you no longer need it. And it causes worries about how it interacts with swap. [06:15] Meh, they're just pages [06:15] What I want is a /tmp that's still a regular directory on my regular disk, but doesn't waste my time journalling. [06:15] Then create it as ext2 [06:16] *cough* [06:16] seriously [06:16] Or use tune2fs to remove the journal [06:16] *cough* *cough* [06:16] which will btw make it ext2 [06:16] lifeless: 6 running now, hovering at 64% RAM used, all 6 cores >95% [06:16] I do still want journaling _and_ per-file system on my FS by default, thank you. [06:17] wgrant: schweet [06:17] lifeless: Sort of. It can be ext3 with other ext3 features and no journal [06:17] And it's all tmpfs, so no iowait. [06:17] wgrant: and I don't suppose there's any CPU overhead to lxc? [06:17] Despite my system drive currently being a shitty WD Green. [06:17] jtv: bugger all [06:17] jtv: It's just namespacing, basically. [06:17] should i expect that we could add a new (null) column to an existing table "live"? and populate it with a script? and then mark the column as not null [06:17] jtv: All runs on the system kernel. [06:17] Sweet. [06:17] jtv: uses built in namespace facilities in the kernel, needs a new userspace library instance etc [06:18] and you have a bit of networking thunks going on [06:18] lifeless: The first two runs were on their own. DatabaseLayer took 1:21 to run. The sixth just passed DatabaseLayer, took 1:22 [06:18] Is it the trick where the kernel basically just keeps separate pointers for its data structures, so there are no cross-references? [06:18] lifeless: So it seems to scale rather well. [06:19] * jtv is so looking forward to this [06:19] jtv: https://dev.launchpad.net/Running/LXC [06:19] Thanks [06:19] It's just a bit lighter-weight than KVM. [06:19] jtv: thats now debugged to the point that it works; wgrants aufs and sed stuff will sit on top of that basic config [06:19] its a *tonne* lighter than kvm [06:19] Should be, yes. No qemu. [06:20] …Right? [06:20] i tried to get lxc working on my box but got errors with the network devices trying to start it up :-( [06:20] wallyworld_: what errors? [06:20] * wallyworld_ looks [06:20] wallyworld_: were you following my notes or the u1 notes or something else again ? [06:20] lifeless: your notes [06:20] ian@wallyworld:~/projects/lp-branches/devel-sandbox$ sudo lxc-start -n natty-lp [06:20] lxc-start: failed to attach 'vethgHMwnY' to the bridge 'virbr0' : No such device [06:20] lxc-start: failed to create netdev [06:20] lxc-start: failed to create the network [06:20] lxc-start: failed to spawn 'natty-lp' [06:20] lxc-start: No such file or directory - failed to remove cgroup '/sys/fs/cgroup/cpu/natty-lp' [06:21] wallyworld_: do you have libvirt installed ? [06:21] * wallyworld_ checks [06:21] wallyworld_: if you've used virt-manager you will [06:21] but if you've not, even if you have used virtualbox, you probably don't. [06:22] By the way, exactly when does test_on_merge.py run? (I'm assuming this shifted a bit since the script was named) [06:22] lifeless: no libvert installed [06:22] jtv: it doesn't [06:22] * wallyworld_ fires up apt-get [06:23] wallyworld_: you want libvirt-bin [06:23] lifeless: then we can get rid of it? [06:23] jtv: no, we may well want it when we do the parallel testing work [06:23] ISTRM getting the near-useless "one huge chunk twice an hour" output. [06:24] The reason I ask is, we have CommandSpawner now which is probably better at processing output from sub-processes. [06:24] lifeless: \o/ thanks! [06:25] I think test_on_merge just waits for a huge fixed-size chunk of output and then dumps the whole thing (even if the chunk ends in the middle of a line, hence the huge chunks) [06:25] Using MF would fix both. [06:26] jtv: IIRC test on merge did more than just that [06:26] It does do more. [06:26] jtv: I would be happy to see test on merge ported [06:26] lifeless: so on a 64 bit system (which i have), do i understand correctly that using a 32 bit lxc will use less memory running lp? or did i make that bit up? [06:26] I wouldn't be happy to have it deleted yet [06:26] wallyworld_: significantly less [06:26] so i would install postgres etc etc into the vm? [06:26] wallyworld_: we'll want to run the production tests on 64 bit to catch C bugs, but for devs it should be totally fine to run 32 bit [06:26] wallyworld_: yes [06:27] The MF would need some extensions, I'm sure, but it's basically a ready-made simple API to managing parallel sub-processes. [06:27] cool. i have 4GB and am constantly up at ~90% usage [06:27] wallyworld_: I suspect this will help [06:27] awesome. will try it. thanks [06:27] jtv: MF ? [06:28] may have to set up my ide to do "remote" debugging :-) [06:28] wgrant: I wonder if the unionfs overlay would also make a faster way to restore the test playground db between tests. :-) [06:28] lifeless: it stands for CommandSpawner. [06:28] jtv: no, because you'd need to bounce postgresql [06:28] bugger :) [06:28] Hmm. [06:29] aufs seems to sometimes interact badly with the hardlink security checks. [06:29] In particularly when rm -r'ing directories on the underlay. [06:29] lifeless: CommandSpawner was originally conceived as a Mother class that produces and manages child processes, i.e. a Forker. [06:29] Still, with /var/tmp/bazaar.launchpad.dev gone from the underlay it is happy again. [06:30] jtv: does that make it a Mother Forker? since it manages child processes? [06:31] wallyworld_: shhhh [06:31] Hmm. [06:32] I guess the 6 runs don't really compete for cache. [06:32] Since they are using the same underlying files. [06:32] yes [06:32] the writes are the only IO load you should have [06:33] At least this isn't shared aufs on top of Solaris NFS. [06:33] That is *fun*. [06:33] And kernel OOPSish. [06:33] * jtv runs in terror [06:34] Ah, bug #729338 [06:34] <_mup_> Bug #729338: yama hardlink restriction misbehaves under aufs < https://launchpad.net/bugs/729338 > [06:34] anyone want to do 2 small reviews? [06:34] https://code.launchpad.net/~wallyworld/launchpad/picker-textfield-focus/+merge/65608 [06:34] https://code.launchpad.net/~wallyworld/launchpad/admins-can-unsubscribe-bugs/+merge/65615 [06:35] I'll take the first. [06:35] thanks [06:35] the 2nd one is only 44 lines [06:35] Damn, picked the wrong one. [06:36] why? javascript? [06:36] No, not 44 lines. [06:36] Oh, wait, this one's 40 lines. I win! [06:36] :-) [06:36] …as soon as the page will load. [06:36] i did say small [06:41] Hmm. [06:41] Some tests are pretty RAM-hungry. [06:41] Almost OOMed there, because they were in sync. [06:41] yeah [06:41] you need the full run to guage the footprint [06:41] wgrant: you have what, 6G ? [06:42] test_process_job_sources_cronjob creates heaps of processes. [06:42] 8G [06:42] wgrant: on a hex cpu ? [06:42] Full LP processes :( [06:42] Yes. [06:42] wgrant: thats... unusual [06:42] Oh? [06:43] AIUI the general thing is to fill 3 banks with identical memory and you get twice the memory bandwidth [06:44] wgrant: http://www.intel.com/support/motherboards/desktop/sb/CS-011965.htm#triple [06:44] lifeless: This is an AMD chip. [06:44] ah [06:45] ignore me then :) [06:45] Does Intel have any hex-core chips out yet? [06:45] (I was just going to go with a cheap AMD quad-core, but the hex-core was only slightly more expensive so I thought why not...) [06:47] wallyworld_: first one's done. [06:47] jtv: thanks! [06:48] wgrant: http://ark.intel.com/Product.aspx?id=53568&processor=E7-2803&spec-codes=SLC3M [06:48] lifeless: OK, but Xeons released a month ago don't count. [06:48] wgrant: is yours 6 cores + ?threadpercor ? [06:48] Haha [06:48] Why not? [06:48] jtv: i had the same question about not using 'unseen'. i should have done it as a driveby perhaps [06:49] wgrant: they've had 6 core xeons for a while, though the 10 core ones look *shiny* [06:49] wallyworld_: I was wondering whether it might be deliberate, e.g. so as not to hinder inclusion in lazr-js. [06:49] maybe. i was going to ask tomorrow [06:51] wgrant: I actually had reason to go with a 2-core not so long ago because it had better single-thread performance… that wasn't for me though: what you're doing will probably make lots of cores more attractive. :) [07:02] Project db-devel build #658: FAILURE in 5 hr 57 min: https://lpci.wedontsleep.org/job/db-devel/658/ === jtv is now known as jtv-eat [07:10] jtv-eat: 2-core 4-threads? [07:10] lifeless: probably, though AMD has realized that SMT is no longer a very good idea. [07:11] * jtv-eat logs in halfway across the world to check [07:11] wgrant: so I was aksing, is yours 6(12) or 6(6) [07:11] lifeless: Oops, missed that. 6(6). [07:12] Bulldozer? [07:13] Is Bulldozer out yet? [07:13] dunno [07:13] Q3, IIRC. [07:15] * jtv-eat decides to live up to his name [07:32] Project windmill-devel build #269: STILL FAILING in 3 min 26 sec: https://lpci.wedontsleep.org/job/windmill-devel/269/ [07:34] StevenK: Could you please perma-kill the Windmill builders? [07:34] Yippie, build fixed! [07:34] Project devel build #829: FIXED in 5 hr 54 min: https://lpci.wedontsleep.org/job/devel/829/ [07:34] The jobs themselves or the slaves? [07:34] 06:59:26 < deryck> Can we please kill those windmill notices? Windmill is dead to us. [07:34] 07:08:09 < deryck> abentley, indeed. we'll move to selenium2/webdriver at the Thunderdome. [07:34] 07:08:33 < deryck> flacoste, sinzui -- ok, thanks. StevenK can we kill the Windmill runner? Not needed now. [07:35] I missed that [07:35] wgrant: Delete or disable? [07:35] StevenK: Disable pls. [07:36] I don't trust that we'll have anything good by next week. [07:36] Both done [07:36] Thankyou sir. [07:36] Ooooh, selinium instead of windmill. Nice. [07:40] * StevenK kills a slave while looking at Jenkins [07:41] I need to work out why Jenkins is now taking nearly six hours [07:46] http://arstechnica.com/business/news/2011/06/amd-launches-second-fusion-cpu-gives-glimpse-at-future-of-cpugpu.ars === jtv-eat is now known as jtv [07:50] StevenK: how could it possibly perform well without Oracle? [07:50] lifeless: el Reg had a nice article about HP's new APU-based laptops IIRC. [07:52] jtv: Your implant is acting up. [07:52] errrhuh? [07:53] be more Enterprise, Steve! [07:53] (i.e. massively slower at three times the price) [08:33] allenap: actually I'm redoing your implementation for those bugs - see https://bugs.launchpad.net/testtools/+bug/801031 [08:33] <_mup_> Bug #801031: gather_details evaluates details < https://launchpad.net/bugs/801031 > [08:46] good morning [08:48] bigjools, good morning [09:06] lifeless: Cool. I eagerly await the outcome :) [09:13] allenap: its pushed [09:14] allenap: along with a new stock fixture and gathering details from child fixtures [09:14] lifeless: Ah, I've just seen it. Neat :) [09:18] yah, its nearly the inner loop from gather_details [09:22] and 0.3.6 pushed === allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 211 - 0:[######=_]:256 [10:01] mpt: Hi. Did you want to talk about that UI lightning talk today? [10:03] huwshimi, yep, in an hour or so if that's good for you [10:03] mpt: Yeah that's fine with me [10:04] wgrant: still there? [10:05] jelmer: Indeed. [10:05] you're well today? [10:05] Or at least moreso? [10:05]