[13:45] <rharper> smoser: https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/+packages  is this the daily ppa?  last publish was 8/18 -- was that the git switch over?
[13:47] <smoser> :-(
[13:47] <smoser> it has definitely built since git. i'll look.
[13:47] <smoser> :-(
[13:48] <rharper> I was going to get an updated core snap that points to cloud-init daily PPA and have it build core snap daily against that so we always have a fresh core.snap
[13:49] <smoser> https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily
[13:50] <smoser> it sucks... that contents sometimes gets changed like randomly
[13:50] <smoser> 'bzr-builder' i think is what is killing it
[13:50] <smoser> there are no changes to bzr 'lp:cloud-init'
[13:50] <smoser> will look at what its supposed to have
[13:51] <rharper> huh
[13:54] <smoser> yeah, i'm not making it up. just now changed it to http://paste.ubuntu.com/23379003/
[13:54] <smoser> click the green thing and it goes to bzr-builder
[13:55] <smoser> i'm going to delete it and hope recreating makes better
[13:55] <rharper> whoa
[13:55] <smoser> thought i'd seen a bug at some point
[13:56] <rharper> we should get powersj to have a jenkins job to boot lxc container, add this ppa and install it
[13:56]  * powersj likes that idea
[13:56] <jgrimm> powersj, good morning!
[13:56] <powersj> Can do something similar to my daily ISO test and compare when a new one comes out too
[13:56] <powersj> morning :)
[13:57] <smoser> well, *not* boot an lxc container and install the thing
[13:57] <smoser> thats absically the first proof of integration test
[13:57] <smoser> wgrant in launchpad pointed me at https://bugs.launchpad.net/launchpad/+bug/1623924 with work around . so looking.
[13:58] <powersj> right, so we could have a daily integration test based on the daily ppa to run a subset of tests.
[14:02] <smoser> alright. building now.
[14:02] <smoser> rharper, thanks for noticing.
[14:06] <rharper> np
[15:20] <NerdyBiker> Hey all, does anyone know of any issues using Cloud-init chef function on a HVM type instance in AWS? As soon as we use a HVM instance, we get "Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_chef.pyc'>) failed" in the logs
[15:20] <smoser> NerdyBiker, more info ?
[15:20] <smoser> it works on non-hvm ?
[15:21] <NerdyBiker> yup works on PV types
[15:21] <smoser> distro ? cloud-init version ?
[15:21] <smoser> that is really odd
[15:21] <NerdyBiker> Oh sorry :D CentOS 6.5
[15:21] <smoser> can you paste the cloud-init.log ?
[15:21] <NerdyBiker> version 0.7.5 cloud-init
[15:21] <smoser> or at least the stack trace
[15:22] <NerdyBiker> Perferred method of pasting?
[15:23] <smoser> pastebin other than pastebin.com :)
[15:23] <NerdyBiker> http://pastebin.com/kGLPkXZf
[15:23] <smoser> that was just rude :)
[15:23] <NerdyBiker> damn
[15:24] <NerdyBiker> haha!
[15:24] <NerdyBiker> terrible timing
[15:24] <smoser> thats fine.
[15:24] <smoser> its just ad-everywhere
[15:24] <smoser> /var/lib/cloud/instance/scripts/runcmd: line 3: chef-client: command not found
[15:24] <smoser> well that'd be it.
[15:25] <smoser> i guess your ami that yul're running for vhm does not have that package installed and the pvm does
[15:25] <NerdyBiker> the user data is set to install chef
[15:25] <smoser> user-data runs after package install
[15:26] <smoser> but cc_chef.py might install it.
[15:26] <smoser> looking
[15:27] <smoser> hm..
[15:27] <NerdyBiker> https://gist.github.com/MattDevUK/064ece1a309616176695b9b67a866aa6
[15:27] <smoser> actually, its your runcmd that is failing.
[15:27] <NerdyBiker> thats the user data we pass in
[15:27] <smoser> you have /var/log/cloud-init-output.log ?
[15:27] <NerdyBiker> that was the first pass
[15:27] <NerdyBiker> paste*
[15:28] <smoser> oh. yeah.
[15:28] <smoser> do you ahve a /var/log/cloud-init.log ?
[15:28] <NerdyBiker> in every other image we've used it on, this installs chef automatically using the params set in the chef block, then runs chef-client.
[15:28] <smoser> that one will have the stack trace and generally more info
[15:28] <NerdyBiker> sure :D
[15:29] <NerdyBiker> it's empty :(
[15:34] <NerdyBiker> any log levels we can change to try and get any content?
[15:37] <smoser> NerdyBiker, its a bug in c loud-init that leaves it empty.
[15:37] <smoser> can you get into that instance ?
[15:37] <NerdyBiker> yup, I'm on it
[15:37] <smoser> vi /etc/cloud/cloud.cfg.d/05*
[15:37] <smoser> the logging file
[15:37] <smoser> comment out
[15:37] <smoser>  
[15:37] <smoser>  - [ *log_base, *log_syslog ]
[15:37] <smoser> then reboot
[15:38] <smoser> maybe rm -Rf /var/lib/cloud /var/log/cloud*
[15:38] <smoser> and reboot
[15:38] <smoser> that should make it think its a first instance
[15:38] <NerdyBiker> ok so just leave &log_file in there?
[15:39] <NerdyBiker> ooo
[15:39] <NerdyBiker> no i see it now :D
[15:40] <NerdyBiker> booting now
[15:43] <NerdyBiker> so... it apparently worked that time
[15:43] <smoser> hm..
[15:43] <smoser> maybe the omnibus installer is doing something int he background ?
[15:43] <smoser> and we're just racing it
[15:46] <NerdyBiker> the output after the reboot; https://gist.github.com/MattDevUK/9ad23f20e3570e82a1b2912e21d57ef0
[15:46] <NerdyBiker> you can see where the "running chef failed" line was previously, now says installing Chef
[15:48] <NerdyBiker> on a positive note, that's the first time we've managed to get that to happen on a HVM image, out of the 10 or so we've tried.
[15:49] <smoser> follow bill's advice.
[15:49] <smoser> reboot!
[15:49] <smoser> and reinstall
[15:54] <NerdyBiker> so uninstallec chef, remvoed cloud files, rebooted and success again. wonder if it's timing when starting a fresh instance
[16:01] <NerdyBiker> @smoser just launched 2 fresh instances, both of them failed the cc_chef part on first run.
[16:03] <smoser> first boot io is very slow
[16:03] <smoser> but if the omnibus installer is backgrounding... thats a pita
[16:03] <smoser> and it should not do that
[16:04] <NerdyBiker> hm, just tried a basic restart, and it failed again
[16:08] <NerdyBiker> seems to be somehting timing related. cleared the /var/lib/cloud, rebooted and it worked. :(
[16:08] <NerdyBiker> damn that sucks
[16:15] <NerdyBiker> I'll tets out some other scenarios and see what comes up
[16:16] <NerdyBiker> test*
[16:30] <NerdyBiker> @smoser so just got the logging change into a base AMI, started an instance and it failed, this is in the cloud-init.log; https://gist.github.com/MattDevUK/9f58b9716a6914406536839a77cadcf0
[16:31] <smoser> hm..
[16:32] <NerdyBiker> there is also a stack for the cc_locale run, but don't think that impacts the chef issue as that fails een when chef succeeds
[16:32] <NerdyBiker> even*
[16:33] <smoser> right.
[16:44] <NerdyBiker> is there any more info I can give? Is this a valid problem?
[18:27] <smoser> NerdyBiker, i think probably valid
[18:27] <smoser> it seems likely to me to be related to omnibus
[18:28] <smoser> woudl be nice to try with newer cloud-init.
[20:34] <smoser> magicalChicken, https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/306731 fails rebase for me
[22:19] <magicalChicken> smoser: I'll look into it, I hadn't noticed issues before