[09:11] gmb: good morning, I am currently working on my introductory blog post, but i'd be really happy to stop it to pair on something, like the buildbot bug Gary mentioned [09:12] frankban, Okay. I'm putting the finishing touches on fixing bug 974608, so how about we start pairing in about 30 minutes or so? [09:12] <_mup_> Bug #974608: bugtask-find-similar.txt fails intemittently on paralleltest runs < https://launchpad.net/bugs/974608 > [09:12] gmb: perfect [09:13] Cool. [09:54] frankban, I'll be ready to pair shortly; waiting for the upgrade / reboot dance to complete. [09:55] ok gmb [10:15] frankban, I'm in https://plus.google.com/hangouts/_/extras/canonical.com/builbot-teaching when you're ready. [12:08] bac benji frankban gmb call in 2 [12:25] gmb, you are gone in case you didn't notice [12:25] Arse [12:26] gary_poster: Complete lockup. Not sure how much you heard. [12:26] gmb really not so much :-/ [12:29] bac, gmb will be working late so if buildbot card is tsill going and you are looking for work by then it might be a good task [12:39] * gmb goes to get a cup of tea whilst the slave churns [12:56] * gary_poster takes kids to school [12:56] frankban, Okay, the sllave is up. I'm in https://plus.google.com/hangouts/_/extras/canonical.com/buildbot-teaching. [12:56] frankban, I've added your keys to the master and slave: [12:56] master: ec2-23-21-29-239.compute-1.amazonaws.com [12:57] slave: ec2-174-129-115-195.compute-1.amazonaws.com [12:57] great thanks gmb [13:08] gary_poster, benji : it looks like the problems i've been seeing with juju not properly parsing --constraints is being caused by our juju wrapper used for testing. it is stripping the quotes, i think. [13:10] gary_poster: jml has decided that the per-worker tagging is a good idea, so he implemented it too. However, his implementation is optional and he likes it that way. He suggests we get the result we want by modifying testr to use the components he's built (he made ConcurrentTestSuite pluggable and made a plugin for it) to get the effect we're after [13:11] bac, ah! [13:11] that makes sense [13:11] benji, eh, ok. Do you have any objections? [13:12] not really [13:12] here is what /usr/bin/juju receives are sys.argv [13:12] ['/usr/bin/juju', 'deploy', '--config=/home/bac/juju/oneiric/buildbot-slave/examples/lpbuildbot.yaml', '--repository=/home/bac/juju', 'local:buildbot-slave', '--constraints', 'cpu=8', 'mem=6G'] [13:12] kinda too bad he didn't coordinate with us better, but hard to complain too much, benji [13:12] bac, yeah, that's the same kind of problem that I was trying to explain about why xvfb-run sometimes barfs [13:13] gary_poster: I assume I should take it for granted that the code will land and attempt to set up a testr dev environment using these branches and work from there [13:15] benji, I'd say landing's a safe bet, given he's the creator and maintainer of testtools. Sounds like a good next step, yes [13:21] benji, do you think you or I should say anything to jml about improved coordination? [13:23] gary_poster: nah [13:23] (yes, it actually took me that long to compose that reply) [13:24] benji, lol, cool. obviously I thought about it long enough that I felt I wanted to ask you :-) [13:26] Yeah. It is a little irritating, but between the (low) amount of harm inflicted (on the company, I don't really mind that much) and the probability of it happening often (low) and the probability of bringing it up actually changing anything (low), it doesn't seem worth it [13:33] right [13:33] benji: any idea on how to get the juju wrapper to respect quoted params? [13:34] e.g. --constraint "cpu=8 mem=4G" [13:34] it is transforming to --constraints cpu=8 mem=4G, which causes juju to barf [13:34] bac: hmm, let me look at it; one second [13:35] bac: this is a problem with bash scripts that I don't understand; there must be a general solution, but I haven't been able to find it [13:36] a hack would be to double-quote, like so: juju --constraint "'cpu=8 mem=4G'" [13:38] bac: maybe changing $@ in the wrapper to "$@" would do the trick [13:41] benji, nope [13:42] darn [13:44] hmm, man bash reinforces my suspicion that "$@" should do the trick [13:46] bac: the script is a sh script, not a bash script, I wonder if that's an issue; try "$@" and change it to be run under bash [13:46] benji, i tried that too early. :( [13:46] darn [13:47] yeah, I'm reading man sh now and it says the same things about "$@" that man bash says [13:48] $@ will keep 'cpu=8 mem=4G' together as one argument, but when passed they are still white-space separated so the receiving script doesn't see them quoted [13:50] bac: I got it to work for me, I did have to change all three instances of $@ in the wrapper to get them all though [13:53] benji, so it does. my testing was flawed. thanks. [13:53] cool! [13:53] benji, pairing! [13:53] :) [13:54] and now I finally know the right way to handle argument lists in bash. I hope I never have to use that knowlege again. [13:59] * benji reboots for great justice. [14:17] benji, so juju_wrapper has been deleted from our charms? does it exist in any project? It is referenced in the READMEs for the master and slave. [14:18] bac: I didn't remember it being deleted. [14:19] bac: revno: 36, timestamp: Wed 2012-02-29 12:10:14 +0100, message:[r=gmb] Added revision file. Removed juju_wrapper. [14:20] I like it when commit messages explain why an action was taken. [14:25] frankban, test_command=xvfb-run ./bin/test --subunit $IDOPTION [14:29] * gmb -> out for a while; back later [14:54] gary_poster, i've manually started an lxc locally to look at the celery issue...but cannot log into it. the container is lucid but root/root nor ubuntu/ubuntu work as logins. what am i forgetting? [14:54] bac on call there soon [14:59] gary_poster: I'd like more info about the goal of the card I am working on, when you have a minute [14:59] ack [15:02] bac, did you create the lucid container binding your home directory, per https://dev.launchpad.net/Running/LXC? If so, you should be able to log in as yourself. If not, I suggest destroying that container and making a new one [15:03] frankban, https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde ? [15:03] There now [15:03] gary_poster: great [15:04] gary_poster, thx logging in as me worked. duh. [15:21] bac cool. frankban has an instance running too that he is about to shut down. Is there anything you'd like him to run on that instance before he shuts down? [15:21] bac, I mean an 8 core ec2 instance, I think [15:23] gary_poster, no, thanks [15:26] cool, welcome [16:02] gary_poster: our lpbuildbot is bzr+ssh://bazaar.launchpad.net/~launchpad/lpbuildbot/public/, right? [16:02] yes frankban [16:02] cool thanks [16:11] gmb: for when you are backL I am running a full test run on your ec2 instances, and later I will run another, I will take care of shutting down the instances later [16:27] I hope the Comcast tech across the street isn't a bad omen for my connectivity. [16:47] * gary_poster goes to bank [17:28] benji, did Comcast let you stay on-line? [17:28] bac: apparently [17:29] they're nice that way [18:01] * gary_poster has triumphed over Ubuntu's keyboard layout! He can enter euros € and accent graves è and accent agues é! He can write umlauts ü! HE HAS THE POWER! [18:28] gary_poster, +1 [18:28] :-) [18:29] gary_poster, have you noticed OSX now has IOS-like alternate character selection? [18:30] if you hold down 'a' it'll show you a bunch of options for a-umlaut, etc [18:30] bac, you press the key down? yeah, I think I had heard of it [18:30] I actually have not logged into the osx side in some time other than by accident [18:30] ipad and iphone are floating my apple boat these days [18:33] gary_poster, so celery seems very unhappy due to semaphores. :( [18:33] bac, you have to remove that file, remember? [18:33] gary_poster, http://pastebin.ubuntu.com/928292/ [18:33] or that doesn't fix it? [18:33] gary_poster, i did, i did [18:34] bac, when? those are the exact symptoms that the bug is about [18:35] when? i did it before running the test, as we discussed yesterday [18:35] bac, IOW, when I say when, I mean, did you remove the dir... [18:35] ok [18:35] perhaps, maybe the work-around is a no-work-around [18:35] or a broke-around, if you will [18:35] bac, it worked for me. but yeah. So maybe it is an ephemeral issue [18:36] bac, try this: [18:36] start up lxc instance [18:36] gary_poster, same same on my local lxc [18:36] oh [18:36] ok nm [18:37] bac, try this, if you are willing & able: [18:37] both. [18:37] well, one sec, lemme see if the newest lxc is around (the one that is supposed to have the actual fix...which is supposed to be equivalent to the workaround... [18:39] * bac brb [18:41] bac, I don't think the newest one is there yet. I'm going to try to dupe locally. [18:53] bac, ok, I got the skinny. The workaround I described at the bottom of comment #1 is correct for lucid: [18:53] Adding this line to the container's fstab will fix the problem: [18:53] none dev/shm tmpfs defaults 0 0 [18:55] * bac trying [18:55] my /etc/fstab was empty prior [18:55] what hallyn describes in comment 2 is apparently only for precise containers [18:55] bac, wait! [18:55] no [18:55] you add it to [18:56] /var/lib/lxc/lptests/fstab [18:56] bac ^^ [18:56] ok. kill container, add it via host, restart container [18:57] bac, yes [18:57] bac, you may need to add back rootfs/dev/shm [18:57] checking [18:58] yep [18:58] lxc-start fails [18:58] lxc-start: No such file or directory - failed to mount 'none' on '/usr/lib/lxc/root/dev/shm' [19:00] >>> from multiprocessing.synchronize import Lock [19:00] >>> Lock() [19:00] [19:00] works [19:00] hurrah [19:03] * bac running all celery tests now [19:04] all celery test now pass, gary_poster [19:05] bac, yay! [19:06] the /etc/mailname fix is still required (i think) [19:06] gary_poster or bac: anyone want to help me flail on testr? [19:06] bac, please verify [19:07] gary_poster, running now w/out /etc/mailname [19:07] cool bac. bac, does that mean we only have soyuz upload intermittent error and unclean reactor? [19:07] and bac, I don't think /etc/mailname is proper fix [19:08] gary_poster, tests fail without /etc/mailname [19:08] bac, could you please set a pdb in the username method on your local machine where it works, and determine where it is getting the name? [19:08] sure [19:09] Thanks [19:09] i'll mv .bazaar first [19:09] benji, I can in a few [19:09] gary_poster: k [19:09] * bac predicts it'll fail without .bazaar or /etc/mailname [19:09] from inspecting the code [19:09] bac, I think that won't matter: in my experiments, the .bazaar was actually set to be in a tmp dir when I ran the test [19:09] but maybe I'm wrong [19:10] and thus, an experiment :-) [19:10] ok [19:10] and gary_poster your local machine has no /etc/mailname? [19:10] bac, right, but I *only* Have tests on lxc here [19:11] gary_poster, have you taken your childrens to see the calder exhibit yet? i may go this weekend [19:13] i removed /etc/mailname from my local machine (leaving .bazaar in place) and the whoami error was seen [19:15] bac, calder: no had not heard it was here. looks cool, thank you for heads up [19:16] bac, ok, that's interesting. so /etc/mailname is the difference. I wonder if /etc/mailname is supposed to be added by postfix. I think it is but could be wrong [19:17] I'm pretty sure I've never added it explicitly [19:17] gary_poster, dpkg -L postfix does not show it [19:17] dpkg -S /etc/mailname also does not show any package claiming it [19:17] ah I was just trying to figure that sort of thing out [19:19] if the postinstall script of postfix write it, i wonder if dpkg would know about it [19:19] recall when you install postfix it asks you a bunch of questions...it may be written at that time [19:19] bac, um, ok. I guess we could change setuplxc to write localhost there...yeah, I think I've been asked. Well, we could do dpkg-reconfigure [19:20] um [19:20] that's not the command [19:20] what am I thinking of [19:20] oh it is the command [19:20] but it is not in my lucid [19:20] yes it is [19:20] sigh [19:21] bac, yes [19:21] dpkg-reconfigure lets you set up system locally [19:21] suggests localhost [19:23] gary_poster, but we're not installing postfix (or any MTA) right? we=setuplxc [19:23] bac, it is installed somehow [19:23] or is it a depend of lp-dev-depend [19:23] may well be [19:23] I suspect it is [19:23] but if you look in the container [19:23] I think you will find that postfix is installed [19:24] benji, um, my "few minutes" is starting to look like "sometime after 4" :-/ [19:24] gary_poster: heh, ok [19:24] * benji goes to get another Dr Pepper. [19:29] gary_poster, yes, it is installed in the container but i can't figure out why [19:29] 'apt-cache showpkg postfix' doesn't show anything interesting [19:30] hmm, i wonder if postfix is just part of main installation [19:31] bac, bsd-mailx depends on default-mta [19:31] which postfix provides [19:31] that's the only dependency I see [19:31] default-mta is not installed in my container [19:33] bac, I did the lame-o approach of starting aptitude and pretending I wanted to uninstall postfix and seeing what complained [19:33] for some reason my terminal won't let me select the complaint [19:33] so I could copynpaste [19:33] but you could dupe [19:33] if that is of interes [19:33] t [19:33] which I question [19:33] i tried 'sudo apt-get -s remove postfix' [19:34] which should be the same (-s == dryrun) [19:34] nothing was listed [19:35] so, it looks like a developer/test box is automatically configured with an MTA and a properly configured one sets up /etc/mailname [19:35] gary_poster, so why do you think it unwise to have our setup write that file? [19:37] bac, I am worried that others who do not use setuplxc (or lpsetup) will encounter this. I am worried that something else really should have added this for us. I guess it is fine. However, I suggest filing a bug so people can find out the solution in the future (and so that people like RObert can educate us if necessary) [19:38] So file a bug that this test fails in a lucid lxc because /etc/mailname is not set up [19:38] and then go ahead and make a card and fix it since it in setuplxc since it is easy [19:38] maybe make a card in slack for lpsetup too [19:38] I need to go get kids from school (K's foot is sprained) [19:39] back in a bit [19:41] ok [20:12] bac, everything cool? [20:13] gary_poster, yep, just finished the MP. want to review? [20:13] sure bac [20:13] https://code.launchpad.net/~bac/launchpad/bug-981114/+merge/101970 [20:13] such good timing [20:13] i'm nearing EOW so make it snappy! :) [20:14] bac :-P [20:14] I added a card for lpsetup btw; I saw your bug/card, thank you [20:16] gary_poster, thx i hadn't gotten to that yet [20:17] gary_poster, i haven't verified this works yet. i'm about to relocate but when i get home i'll start an ec2 run and only land the change after i've proven it works. [20:17] i did show that 'localhost' in /etc/mailname was sufficient [20:18] bac, you could also do this in pure Python from the host; might have preferred that but won't ask you to change it. As a fly-by, could you please add a comment to the line *before* yours such as "# XXX Work around for kernel bug 959352"? That is, before [20:18] root_sshcall('ln -s /usr/lib/graphviz /rootfs/usr/lib/graphviz') [20:18] <_mup_> Bug #959352: Ephemeral containers have "/rootfs" prefix in /proc/self/maps entries

< https://launchpad.net/bugs/959352 > [20:18] oh wait [20:18] I think that the comment is already there [20:18] nm [20:18] sorry [20:18] so anyway, will approve [20:19] gary_poster, oh, wait, doing it host-side makes sense. 6 vs 1/2 dozen? [20:19] bac, doing it without a subprocess call seems more elegant [20:19] but no real diff [20:19] gary_poster, Do you know how far frankban got with the work we were doing before I had to go out? [20:20] 6.1 vs half dozen [20:20] I meant ot ask him when I got back but forgot. [20:20] gmb, yes. He determined that the problem was buggy. [20:20] gary_poster, ok, i'll look at doing it that way [20:20] gmb, he fixed the code issue, but we still need testr bug 949950 (tracking) in order for this to work [20:21] <_mup_> Bug #949950: Feature request: an option to show full subunit stream of running tests < https://launchpad.net/bugs/949950 > [20:21] gary_poster, Ah, because the master doesn't actually get subunit output form the slave. [20:21] * bac -> goes home [20:21] right [20:21] have a good wknd bac [20:21] We'd kinda figured that out when I left, but I'm glad it wasn't just that we were doing something silly. [20:21] u2 [20:21] :-) [20:22] gmb, how is 974608? Can review something if that helps [20:24] benji, can talk testr for the last half hour of the day if that helps :-) [20:24] gary_poster: it can't hurt [20:24] I could make really unpleasant jokes. That would hurt. [20:25] heh [20:25] benji, I'm in https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde [21:06] benji, please send us tags method just to make sure we are on right track [21:06] this weekend sometime [21:06] gary_poster: if you can still see my terminal, it's there [21:07] (along with a whitespace fix) [21:07] got it thanks benji [21:07] cool [21:07] * benji goes afk. [21:10] * gary_poster heads out [21:11] bye [21:12] gmb, meh, do you want me t try and give 4 min review of MP? Must run soon [21:12] but I saw you just got it in :-) [21:12] gary_poster, That would be awesome if you could - it's fairly mechanical anyway. [21:12] ack, on ity [21:14] gmb, not my best review work :-P but I thought it looked very good [21:14] Thank you! [21:14] have a great weekend gmb [21:14] gary_poster, Thanks! Have a good weekend; I'll see you Tuesday. [21:14] ah yes, thanks for reminder [21:14] bye [21:14] Cya