[13:45] smoser: https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/+packages is this the daily ppa? last publish was 8/18 -- was that the git switch over? [13:47] :-( [13:47] it has definitely built since git. i'll look. [13:47] :-( [13:48] I was going to get an updated core snap that points to cloud-init daily PPA and have it build core snap daily against that so we always have a fresh core.snap [13:49] https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily [13:50] it sucks... that contents sometimes gets changed like randomly [13:50] 'bzr-builder' i think is what is killing it [13:50] there are no changes to bzr 'lp:cloud-init' [13:50] will look at what its supposed to have [13:51] huh [13:54] yeah, i'm not making it up. just now changed it to http://paste.ubuntu.com/23379003/ [13:54] click the green thing and it goes to bzr-builder [13:55] i'm going to delete it and hope recreating makes better [13:55] whoa [13:55] thought i'd seen a bug at some point [13:56] we should get powersj to have a jenkins job to boot lxc container, add this ppa and install it [13:56] * powersj likes that idea [13:56] powersj, good morning! [13:56] Can do something similar to my daily ISO test and compare when a new one comes out too [13:56] morning :) [13:57] well, *not* boot an lxc container and install the thing [13:57] thats absically the first proof of integration test [13:57] wgrant in launchpad pointed me at https://bugs.launchpad.net/launchpad/+bug/1623924 with work around . so looking. [13:58] right, so we could have a daily integration test based on the daily ppa to run a subset of tests. [14:02] alright. building now. [14:02] rharper, thanks for noticing. [14:06] np [15:20] Hey all, does anyone know of any issues using Cloud-init chef function on a HVM type instance in AWS? As soon as we use a HVM instance, we get "Running chef () failed" in the logs [15:20] NerdyBiker, more info ? [15:20] it works on non-hvm ? [15:21] yup works on PV types [15:21] distro ? cloud-init version ? [15:21] that is really odd [15:21] Oh sorry :D CentOS 6.5 [15:21] can you paste the cloud-init.log ? [15:21] version 0.7.5 cloud-init [15:21] or at least the stack trace [15:22] Perferred method of pasting? [15:23] pastebin other than pastebin.com :) [15:23] http://pastebin.com/kGLPkXZf [15:23] that was just rude :) [15:23] damn [15:24] haha! [15:24] terrible timing [15:24] thats fine. [15:24] its just ad-everywhere [15:24] /var/lib/cloud/instance/scripts/runcmd: line 3: chef-client: command not found [15:24] well that'd be it. [15:25] i guess your ami that yul're running for vhm does not have that package installed and the pvm does [15:25] the user data is set to install chef [15:25] user-data runs after package install [15:26] but cc_chef.py might install it. [15:26] looking [15:27] hm.. [15:27] https://gist.github.com/MattDevUK/064ece1a309616176695b9b67a866aa6 [15:27] actually, its your runcmd that is failing. [15:27] thats the user data we pass in [15:27] you have /var/log/cloud-init-output.log ? [15:27] that was the first pass [15:27] paste* [15:28] oh. yeah. [15:28] do you ahve a /var/log/cloud-init.log ? [15:28] in every other image we've used it on, this installs chef automatically using the params set in the chef block, then runs chef-client. [15:28] that one will have the stack trace and generally more info [15:28] sure :D [15:29] it's empty :( [15:34] any log levels we can change to try and get any content? [15:37] NerdyBiker, its a bug in c loud-init that leaves it empty. [15:37] can you get into that instance ? [15:37] yup, I'm on it [15:37] vi /etc/cloud/cloud.cfg.d/05* [15:37] the logging file [15:37] comment out [15:37] [15:37] - [ *log_base, *log_syslog ] [15:37] then reboot [15:38] maybe rm -Rf /var/lib/cloud /var/log/cloud* [15:38] and reboot [15:38] that should make it think its a first instance [15:38] ok so just leave &log_file in there? [15:39] ooo [15:39] no i see it now :D [15:40] booting now [15:43] so... it apparently worked that time [15:43] hm.. [15:43] maybe the omnibus installer is doing something int he background ? [15:43] and we're just racing it [15:46] the output after the reboot; https://gist.github.com/MattDevUK/9ad23f20e3570e82a1b2912e21d57ef0 [15:46] you can see where the "running chef failed" line was previously, now says installing Chef [15:48] on a positive note, that's the first time we've managed to get that to happen on a HVM image, out of the 10 or so we've tried. [15:49] follow bill's advice. [15:49] reboot! [15:49] and reinstall [15:54] so uninstallec chef, remvoed cloud files, rebooted and success again. wonder if it's timing when starting a fresh instance [16:01] @smoser just launched 2 fresh instances, both of them failed the cc_chef part on first run. [16:03] first boot io is very slow [16:03] but if the omnibus installer is backgrounding... thats a pita [16:03] and it should not do that [16:04] hm, just tried a basic restart, and it failed again [16:08] seems to be somehting timing related. cleared the /var/lib/cloud, rebooted and it worked. :( [16:08] damn that sucks [16:15] I'll tets out some other scenarios and see what comes up [16:16] test* [16:30] @smoser so just got the logging change into a base AMI, started an instance and it failed, this is in the cloud-init.log; https://gist.github.com/MattDevUK/9f58b9716a6914406536839a77cadcf0 [16:31] hm.. [16:32] there is also a stack for the cc_locale run, but don't think that impacts the chef issue as that fails een when chef succeeds [16:32] even* [16:33] right. [16:44] is there any more info I can give? Is this a valid problem? [18:27] NerdyBiker, i think probably valid [18:27] it seems likely to me to be related to omnibus [18:28] woudl be nice to try with newer cloud-init. [20:34] magicalChicken, https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/306731 fails rebase for me [22:19] smoser: I'll look into it, I hadn't noticed issues before