[00:08] hazmat, yes --fancy shows them running [00:08] hazmat, yes pending status, which is all I have ever seen since 1.14.0 [00:08] hmm [00:09] that's a good start [00:09] sinzui, jorge and marco had this issue, thumper had to do surgery on them.. but tbd if its needed here [00:10] * hazmat waits for cloudinit output [00:10] hazmat, cloud-init: http://pastebin.ubuntu.com/6333073/ [00:10] cool.. the source of fail [00:10] which agrees with [00:10] curtis-local-machine-1 RUNNING - - YES [00:11] sinzui, so let's try a clean tear down.. [00:11] I wonder if I need to locate abentley's bug that explained his two step clear lxc cache [00:11] sinzui, issue there is no cloudinit data made it to the container [00:11] sinzui, juju destroy-environment [00:12] done [00:12] sinzui, anything in ls /etc/lxc/auto ? [00:12] no [00:13] sinzui, how about ls /var/lib/juju/containers/ [00:13] hazmat, empty [00:13] sinzui, hmm.. and ls /var/lib/juju/removed-containers/ [00:14] sinzui, this is trunk or 1.17 ? [00:14] er. 1.16 [00:14] hazmat, 1.16.1 release candidate...the one that is supposed to fix lxc [00:14] ah.. one of those [00:14] hazmat, Lots in removed containers [00:15] curtis-local-machine-1 curtis-me-machine-1 curtis-me-machine-1.2 curtis-me-machine-1.4 [00:15] curtis-local-machine-2 curtis-me-machine-1.1 curtis-me-machine-1.3 curtis-me-machine-2 [00:15] sinzui go ahead and kill them. [00:15] and move your $JUJU_HOME/$local_provider_name out of the way [00:15] hazmat, done [00:15] and that should be pristine for a new bootstrap [00:15] sinzui, if your bootstraping.. with 1.16.1 tools you may need --upload-tools [00:15] if there not published [00:16] I think that last step is key. I have not done that today [00:16] sinzui, yeah.. --debug on bootstrap and pastebin for posterity if needed [00:22] hazmat, nothing different from my previous tests...so far http://pastebin.ubuntu.com/6333122/ [00:23] sinzui, can you set juju set-env logging-config="=DEBUG;juju.provisioner=TRACE" [00:23] and then deploy something.. [00:25] sinzui, btw re logs $JUJU_HOME/$local-env-name/machine-0.log has the logs of interest [00:26] yeah, that is what I am tailing [00:26] oh.. cool [00:26] sinzui: no all machines log for local at this stage [00:26] sinzui, so .. i get roughly the same content you did [00:26] 2013-10-31 00:26:17 INFO juju.provisioner provisioner_task.go:367 started machine 1 as instance kapil-local-machine-1 with hardware [00:27] so, what's the summary of the current issues? [00:28] hazmat, 3 running machines without net [00:28] thumper, local provider seems to work on trunk.. sinzui has it failing for him 1.16.1 [00:28] thumper, somewhat more critical is that it doesn't seem to work on maas as a container at all atm [00:28] hazmat, all three services/machines are pending [00:29] sinzui, you can watch them do upstart and lxc-create via pstree [00:29] * thumper sighs... [00:30] thumper, the containers on maas are critical for some ODS stuff [00:30] * thumper nods [00:30] let's get that working first then [00:30] thumper, i'm going to give it a go with trunk on the maas i think.. [00:30] just to see if that helps at all. plus to have a source compile i can instrument.. since the logging here seems to be nil [00:31] hazmat: do you have any instructions for a virtual maas setup [00:31] sinzui, cool, is cloudinit done from pstree? [00:31] so we can test locally? [00:31] abentley reported this bug. I don't think it is related because I start my lts container every day [00:31] https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1236490 [00:31] <_mup_> Bug #1236490: Container has no networking

[00:31] thumper, you mean did i attend the maas training ;-) ? [00:31] no actually.. [00:31] * bigjools bans the maas training from existence [00:31] sinzui: hangout? I can talk through some things [00:32] sinzui, can you pastebin lxc-ls --fancy [00:32] not sure what you mean by no networking [00:32] hazmat, yes it is. I can tell when my fan stops actually [00:32] cloudinit is injected here via file [00:32] sinzui, wow.. is that spinning rust? [00:32] sinzui, or ssd? [00:32] ssd [00:32] * hazmat runs a dozen containers normally [00:33] only 2-4 for juju though [00:33] i don't even notice them [00:33] I have my fan tuned to come on often...its a mac and they run hot [00:33] gotcha [00:33] hazmat, no improvement http://pastebin.ubuntu.com/6333175/ [00:34] that's intriguing [00:34] sinzui, can you go to /var/lib/juju/containers [00:35] there should be console.log in there for each machine [00:35] and pastebin the console.log [00:35] that is the output of running the cloudinit [00:35] although in this case i think we want the container.log [00:35] sinzui: also pastebin ifconfig [00:35] the netns stuff should have happened pre lxc [00:36] er. pre cloudinit [00:36] hazmat, console.log http://pastebin.ubuntu.com/6333189/ [00:37] thumper, my ifconfig or one from a container? [00:37] sinzui: from the host [00:38] sinzui, can you pastebin the container.log post ifconfig .. [00:38] that should detail why the no netns [00:38] thumper, ifconfig http://pastebin.ubuntu.com/6333193/ [00:38] huh [00:38] you've got three veth devices there. [00:38] for your three containers [00:38] sinzui, is dnsmasq running ? [00:39] sinzui: how about the lxc.conf from the /var/lib/juju/containers// [00:39] you should see something like... 116 1190 0.0 0.0 28184 804 ? S Oct15 0:04 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative [00:39] er.. ignoring the preamble.. that's from ps aux | grep dnsmasq [00:39] hazmat, yes http://pastebin.ubuntu.com/6333198/ [00:40] sinzui, the container.log should have the netns come up.... [00:40] hazmat, the lxc.conf from -1 http://pastebin.ubuntu.com/6333199/ [00:41] * hazmat leaves to thumper's capable hands [00:41] ok, this needs help from an lxc person [00:41] as it is outside the remit of juju [00:42] the container didn't get networking [00:42] prior to all the juju shit [00:42] thumper, fab, as I think too [00:42] sinzui: if you bring up a machine manually, does it get an ip address? [00:42] while abentley's bug is not identical, I will try the work around [00:43] thumper, do you mean bring one down than bring it back up again? [00:43] sinzui: what do you get with "ls -l /var/cache/lxc" ? [00:43] actually [00:43] sinzui: ls -lh /var/cache/lxc/cloud-precise/ [00:43] thumper, [00:43] drwxr-xr-x 2 root root 4096 Jul 1 18:29 cloud-precise [00:43] drwxr-xr-x 3 root root 4096 Jan 9 2013 precise [00:44] thumper, -rw-r--r-- 1 root root 221M Jun 24 03:07 ubuntu-12.04.2-server-cloudimg-amd64-root.tar.gz [00:44] -rw-r--r-- 1 root root 206M Dec 17 2012 ubuntu-12.04-server-cloudimg-amd64-root.tar.gz [00:44] ok, that's it [00:44] ubuntu-12.04-server-cloudimg-amd64-root.tar.gz is too old [00:44] and the refresh mechanism isn't working [00:44] this is the problem marco had [00:44] and jorge [00:45] smoser is aware [00:45] sinzui: delete the file [00:45] sinzui, can you post the console.log [00:45] destroy the environment [00:45] er.. container.log [00:45] sinzui, ^ [00:45] and start again [00:45] before destroying [00:45] thats a known issue. [00:45] yeah, remove that file. [00:45] lxc will download a fresh copy of the cloud image [00:45] oh.. that one [00:45] and it "should" work [00:45] i forget about that.. i ran into that as well [00:46] smoser, so what causes that? [00:46] smoser, with the new lxc cloud template / cloud img download there's no dot versioning around point releases.. [00:47] but never really understood why the old container wouldn't netns [00:47] sinzui: still there? [00:47] "new lxc cloud template" ? [00:47] hazmat, I have a copy of the container log. anything else before I start purging [00:47] there is no such thing as "dot" [00:47] ever. [00:47] 12.04.3 is no different than 12.04 [00:48] the cloud images do not distinguish them in name. they did at one point and that was a bug. [00:48] smoser, it is though from a cached file sense [00:48] that is independent of .2 verus .3 [00:48] previously it did.. and it worked better for caching and invalidation purposes when the point release incremented [00:48] previously it was broken on the server side in naming [00:49] which broke people who expect to 'wget http://path/to/some/known/current/release/file.img' [00:49] https://bugs.launchpad.net/ubuntu/+bug/1220366 [00:49] <_mup_> Bug #1220366: cloud-images have inconsistent filenames in 12.04.3

[00:50] thumper, I am bootstrapping again [00:50] ic [00:50] that should fix sinzui's issue then [00:50] the cloud template needs to be smarter. [00:50] but we really would rather make a lxc wrapper that was smarter. [00:51] ie, hallyn doesn't think such knowledge should live in lxc itself. [00:52] smoser, ala the kvm front end.. or ala docker front end? [00:52] uvtool-lxc is the plan in my head. [00:52] which is also ideally where the "--use-the-fastest" cloning argument goes. [00:52] My fan has started [00:52] :) [00:53] smoser, the default behavior with -s works for me generally.. ie autodetect btrfs ;-).. but i'm curious about the docker lvm thin provisioning on sparse file dev pool [00:54] whoa! there it is. Installed [00:54] sidnei just got the that merged into lxc-clone (re lvm-thin) support [00:54] sinzui, do we have tools published for 1.16.1 or tarball? [00:54] hazmat, i think there were issues with that default behavior. [00:54] are you sure it even remains ? [00:55] and it wouldn't select unionfs [00:55] * hazmat checks [00:55] Thank you rocket scientists! hazmat thumper smoser. I really appreciate your patience [00:55] sinzui: awesome [00:55] that's one fixed [00:55] and that fault should definitely be logged on the local provider troubleshooting page <- jcastro, marcoceppi [00:55] now to look at maas [00:55] o/ [00:56] I just got pinged about this [00:56] hazmat: what versions were you playing with? [00:56] narrowed it down any yet? [00:56] bigjools: not started looking yet [00:56] thumper, 1.16.0 for maas [00:56] thumper: I guess I should fire up my maas server? :) [00:56] bigjools: gimmie time to shower (haven't since back from the gym yet), and we can have an initial call prior to working out debugging plan [00:56] bigjools: ack [00:56] hazmat, yes, my testing tools are in place in aws and hp and canonistack. I place them in the testing/ subdir and point the tools url to them: http://juju-dist.s3.amazonaws.com/ [00:56] thumper: roger mcdodger [00:57] hazmat, and I will officially make the tarball in about 10 minutes [00:57] smoser, it requires explicit lxc-clone -s but yeah.. it works without specing btrfs directly [00:57] du -hs /var/lib/lxc/container-name -> 7mb [00:58] hazmat, and you can see all the scripts I am using at https://code.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2 [00:58] you can make your own deb without any juju parts installed [01:10] bigjools: how long does an juju azure bootstrap instance normally take to spin up? [01:11] wallyworld_, ~ 5 minutes i think [01:11] wallyworld_: from memory not at all quick, I think it was like 5-10 m [01:11] ok, ta. just bootstrapped an and waiting for juju status [01:11] s/just/have been [01:12] thumper: can you summarize what should be logged? [01:13] marcoceppi: the symptom was that the machines were pending, and lxc-ls --fancy showed the machines started but with no ipv4 address [01:13] marcoceppi: ls -lh /var/cache/lxc/cloud-precise showed ubuntu-12.04-server-cloudimg-amd64-root.tar.gz to be over 4 months old [01:13] marcoceppi: delete that file, destroy the environment and try again [01:14] a new lxc cloud image will be downloaded and all should be good [01:17] bigjools: hangout? [01:18] marcoceppi: does that all make sense? [01:19] thumper, marcoceppi It makes sense to me. marcoceppi can interview me tomorrow or he/nick can invite me to try updating the docs [01:19] * thumper nods [01:19] thanks sinzui [01:19] thumper: calling you.... [01:20] which me? [01:20] the head [01:20] wrong one? [01:20] this is why I don';t have two g+ profiles [01:48] thumper: yeah, that's what happened to me [01:48] already recorded [01:50] bigjools: do you know the magic command to ssh into a bootstrap node on azure? i have the ip address of the machine [01:51] juju ssh 0 :) [01:51] bigjools: doesn't work. times out [01:51] bootstrap has failed [01:51] but i have the ip address of the bootstrap node [01:51] ummmm in that case you're probably screwed [01:52] it will be firewalled [01:52] well, when i say bootstrap failed, it completes but juju status fails [01:52] ok [01:52] and i want to find out why [01:52] can you telnet port 22? [01:52] i'll try [01:53] yes [01:53] when i try ssh directly, it fails with a permission denied error (publickey) [01:53] so it's up but hasn't added the key [01:53] yeah, seems so [01:53] wallyworld_, with azure, I wait 30 minutes for the machine to really be up [01:53] which means bootstrap was not as successful as you might think [01:54] sinzui: really? oh joy [01:54] ! [01:54] wtf takes so long [01:54] oh - yes the crazy provisioner [01:55] you can see in the MS dashboard [01:55] so before when juju status timed out and i destoyed my env, i should have just waited [01:55] wallyworld_, on canonistack, we get TLS timeouts while the bootstrap node is coming up, but azure is silent. I have seen the machine come up in 20 minutes, but 30 is realistic [01:55] dashboard? you have a link? [01:56] sinzui: ok. i think i've fixed the azure incompatibility but am trying to make sure [01:56] wallyworld_: well not exactly you would need my password. let me look at it [01:56] ok [01:56] bigjools: i bet i can guess it [01:56] aussieaussieaussieoioioi [01:56] wallyworld_: "ianisacocktrumpet" [01:57] that would have been my next guess [01:58] wallyworld_: in fact it was so secure I can't remember it :) [01:58] ha ha ha [01:59] wallyworld_: get your own credentials [01:59] bigjools: they're in the mail apparently [02:00] i only needed yours cause of this azure cockup and the time pressure to fix it [02:00] yeah np [02:00] I encrypted my creds and put them in the CI machine so that my team could build better testing [02:00] i wonder what else besides juju msft broke with their change? [02:01] wallyworld_: btw baldrick left a hidden present that the mower found [02:01] \o/ :-D [02:01] you can imagine the result [02:01] did it splat all over you /me asks hopefully? [02:01] >:( [02:01] Your dog is named baldrick? [02:01] yep :-D [02:01] he has a cunning plan [02:02] * sinzui was just telling his son about Ebenezer black adder 5 minutes ago. [02:02] wallyworld_: I see two VMs running [02:02] * wallyworld_ is finding it hard to type cause he is laughing so hard to bigjools [02:02] one of them from juju the other I think for a gwacl run [02:02] bigjools: ah the gwacl one needs to be shutdown. i thought i did [02:03] * bigjools considers putting baldrick's present in wallyworld_'s mailbox [02:03] sinzui: blackadder is one of my favourite tv shows [02:03] bigjools: how can you, it's all vapourised all over you, right? [02:03] there's some left [02:04] next time get jen to mow the lawn after baldrick has visited :-D [02:04] wallyworld_: so the cpu graph shows it finishing a heavy load at 11:45 [02:04] the juju one? [02:05] what's the ip address? [02:05] 137.135.11.14 [02:06] yep, that's the one juju status keeps polling mongo for trying to do a status [02:06] :-( [02:06] i really need to get the logs off that sucker [02:06] wallyworld_: 17070 and 37017 ports are open [02:06] that's it [02:07] 37017 is the mgo port [02:07] * bigjools wonders if it's the non-ssl mgo [02:07] and it says conection refused [02:07] ah [02:07] that could be it [02:07] * bigjools gets lunch while wallyworld_ fixes it [02:08] * wallyworld_ doesn't know how to fix that one [02:08] deploy with saucy? [02:08] ah ok [02:08] good idea [02:08] it might have failed to add the cloud archive [02:09] yeah, sounds likely [02:09] although doesn't explain why you can't ssh in [02:09] hopefully saucy will work [02:09] heard a lot of swearing coming from the living room earlier - jen found jake pulling keys off her laptop again... ROFL [02:10] * bigjools lunches [02:10] lol [02:11] sinzui: still around? [02:11] I am [02:11] sinzui: I have a critical bug [02:11] sinzui: for 1.16 [02:12] sinzui: it is the lxc and maas problem [02:12] that they are using for ods [02:12] demo [02:12] always last minute [02:12] it is a regression introduced when the provisioner moved to the api [02:12] I just cut 1.16.1, but plan a 1.16.2 for azure. [02:12] but no one noticed [02:12] thumper, Do I need to take down 1.16.1 [02:12] no [02:12] I don't think so [02:12] because they can use a custom release [02:12] I think [02:13] but we may have a 1.16.2 real soon now [02:13] if they do need a release [02:13] what's the timeframe for 1.16.2? [02:13] bigjools: I have a plan [02:13] I can make it tomorrow immediately after 1.16.1 is needed. I hope for an azure fix of couse [02:13] * thumper nods [02:13] ok [02:13] let me file this bug and get to work [02:15] * sinzui creates milestone [02:17] sinzui: azure fix will be commited to gwacl today i hope [02:17] wallyworld_, excellent. Testing will be quick since it is limited to two providers [02:18] i'm doing a local test now [02:18] Did I mention abentley got rudimentary CI running on HP. we can run about 200 upgrade and bootstrap tests a day [02:18] ran into the precise mongo thing i think [02:18] trying on saucy [02:23] wallyworld_: are you chasing the azure issue? [02:23] wallyworld_: is there a bug? [02:24] yes. the issue is with msft, not us [02:24] we need to change gwacl cause they broke the api [02:24] i'm testing now [02:24] :) [02:24] wallyworld_: can you target the bug to 1.16.2? [02:24] but azure is sloooooow [02:25] thumper: sure. there's 2 of them - a public and a private [02:25] * thumper nods [02:26] bigjools: lp:1246556 for the juju maas bug [02:26] which it seems mup can't understand, bug 1246556 [02:26] <_mup_> Bug #1246556: lxc containers broken with maas

[02:41] * thumper moved some code around, all tests still pass [02:42] I find this mildly disturbing [03:06] bigjools: not sure how many azure instances i have running - i think 2. a bootstrap using saucy still doesn't allow juju status to connect sadly [03:07] thumper: i have 2 small azure mp's for the 1.16.2 release if you have a moment sometime https://code.launchpad.net/~wallyworld/gwacl/azure-management-api-change https://code.launchpad.net/~wallyworld/gwacl/fix-request-eof-2/+merge/193346 [03:10] * thumper is in deep thought [03:10] sorry [03:11] np [03:11] bigjools: what is the price for 2 small gwacl reviews? [03:12] needed for juju 1.16.2 release tomorrow [03:28] bigjools: bugger... [03:28] bigjools: it seems like the quick fix won't work very well at al [03:28] may have to fix it the right way [03:29] wallyworld_: I can look for you, after all you came bearing gifts yesterday [03:29] indeed i did [03:29] they are only small [03:29] branches [03:29] thumper: great architecture FTW [03:30] 466 lines [03:30] is small in juju world? [03:30] bigjools: sadly i have a saucy node running now which also disallows connections to mongo [03:30] wallyworld_: is the firewall endpoint open? [03:30] not sure. what port is that do you know? [03:31] i thought you said before 37017 was open? [03:31] which is the port the client is sing to contact mongo [03:32] ok [03:32] wallyworld_: hang on I need to make call first [03:32] ok [03:49] wallyworld_: first comment - pease remove the factored version numbers [03:49] why? [03:50] I previously had someone stop doing that - each api request is separately versioned and if you factor it you will introduce subtle bugs [03:50] it kinda sucks having them all copied and pasted in the code [03:50] yes - but it needs to be done [03:50] ok then [03:50] I rooted out at least 3 bugs last time I unfactored it [03:50] sigh [03:51] I know [03:51] how the fook do we keep track of all the apis [03:51] sounds like a nightmare [03:52] wallyworld_: fwiw the version should be associated with the request struct ideally [03:52] they change versions when the format changes [03:52] so they are very closely tied [03:52] ok. for now though, i'll just go back to how it was [03:53] and do a search and replace of string [03:53] wallyworld_: for example if you update the refactored version you would change all the calls at once, and introduce potentially subtle sematic changes [03:53] i thought they would all change together [03:54] as a group [03:54] IME they dont [03:54] :-( [03:54] it might say they do.... [03:54] but I call BS [03:54] fair enough. it is msft after all [03:56] you might want to mention this in the code [04:00] ok [04:02] arg [04:02] stabby stabby [04:02] * thumper is trying to unpick something [04:02] but others are sewing things tighter together [04:04]