minfrinA quick question on AWS and cloud-init, to see whether anyone has seen this behaviour before.15:16
minfrinI have an EC2 instance with an EBS volume attached to it, and I;ve asked cloud-init to format it like so:15:16
minfrin  - label: data15:16
minfrin    device: /dev/xvdh15:16
minfrin    filesystem: ext415:16
minfrinWhen cloud-init starts, mke2fs fails as follows:15:16
minfrin2016-02-10 15:03:42,593 - util.py[WARNING]: Failed during filesystem operation15:16
minfrinFailed to exec of '['/sbin/mkfs.ext4', '/dev/xvdh', '-L', 'data']':15:16
minfrinUnexpected error while running command.15:17
minfrinCommand: ['/sbin/mkfs.ext4', '/dev/xvdh', '-L', 'data']15:17
minfrinExit code: 115:17
minfrinReason: -15:17
minfrinStdout: ''15:17
minfrinStderr: 'mke2fs 1.42.9 (4-Feb-2014)\nCould not stat /dev/xvdh --- No such file or directory\n\nThe device apparently does not exist; did you specify it correctly?\n'15:17
minfrinLater on, after sshing into the machine, I am able to run mke2fs manually and the formatting of the block device works fine.15:17
minfrinAre they are delays that people are aware of when deploying AWS EBS drives that would cause cloud-init to fail?15:17
waldicheck the kernel log15:22
minfrinNot following - what would I check the kernel log for? (I assume you mean dmesg)15:23
rcjsmoser, I'm debugging a new DS where the config module fails during boot "Can not apply stage config, no datasource found!" but after boot I can run it 'service cloud-config start' just fine.  Any tips on debugging or ideas of what is happening?15:25
waldiminfrin: it should list when xvdh appeared. why do you have _eight_ disks?15:30
minfrinThe only mention in dmesg of the xvdh disk is the very last lines, which could be warnings triggered during the successful mkfs.ext4:15:38
minfrin[   40.226724] blkfront: xvdh: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;15:38
minfrin[   40.233951]  xvdh: unknown partition table15:38
minfrinWhat I was hoping to find was if anyone had any AWS experience, and whether there were any known issues with AWS running cloud-init before all resources were ready?15:38
smoserrcj, probably log says WARN somewhere15:40
smoserand also /run/cloud-init/*.json15:40
smoserminfrin, you're rpobably right. the disk probably was not there when cloud-init r un.15:41
smoseri'd not seen that, but the no such file or directory is pretty clear15:42
minfrinSome digging has found this patch to something called "rubber" from 2012 which referred to bugs with AWS volumes not being ready: https://github.com/rubber/rubber/pull/156/files15:42
minfrinThe patch seems to pause in a loop until the device exists, and then continues.15:42
smoserthats definitely a sane path for some situations.15:47
smoserbut in others its not sne.15:47
smoserits posible we could let you define the behavior to cloud-init.15:47
smoserthe issue is that it sits there and waits forever for that device to appear, and its not there....15:48
smoserthen its not going to get to the point where it gets your ssh keys or runs your user-data15:48
smoserand you're not going to ever be able to get in.15:48
rcjsmoser, no WARN.  This is a container environment where we're emitting some missing signals for network and mounting FSes.  Could be that I've inadvertently changed the order the cloud-init jobs are run.  is cloud-config run before cloud-init-local?15:51
smoserdo you have something in /run/cloud-init ?15:51
rcjsmoser, ^ there is results.json (prior run) and here is cloud-init-output.log from the latest run http://paste.ubuntu.com/15009195/15:55
smoserrcj, config definitely did start before init15:57
rcjsmoser, should it?16:07
minfrinsmoser: An option to work around the device-not-ready problem is a "wait" option, giving the longest time we're prepared to wait for the drive to become available.16:46
smoserminfrin, well, nto if you're trying to mkfs16:48
smoseroh. i see, yeah.16:48
smosersorry. i thought you were saying upstart or systemd wait for the filesystem16:48
smoserbut yeah16:48
minfrinsmoser: Still waiting for word from AWS support about this, need to work out how widespread the problem is and how it will affect us.18:00
rcjsmoser, thanks.  this is an issue with this particular container running things out of order due to upstart overrides that change emit timing for cloud-init prereqs.18:06
jmccannI was wondering if someone could help me finish getting a CCA signed so I could try contributing to cloud-init18:53
jmccannI'm on page http://www.ubuntu.com/legal/contributors/submit and not sure what to fill for "Please add the Canonical Project Manager or contact"18:53
harlowja_at_homesmoser, ^18:56
harlowja_at_homescott moser i think is said contact still18:56
jmccannthanks much!18:57
smoserjmccann, yeah. you can put me that is fine18:57
rcjsmoser, http://paste.ubuntu.com/15010669/ is what I need in this container environment (or perfectly replicating the FS-related emits) because right now I have a race.19:54
rcjsmoser, but I don't know that is acceptable for Trusty SRU19:54
smoserrcj, somethign else is getting broked20:04
rcjsmoser, how so?20:06
smoserso 'filesystem' should not be emitted until /  is mounted rw20:08
smoserand / should not be mounted rw until cluod-init is stopped20:08
rcjbut it's a container so when it's started all the filesystems are mounted and ready.  they have a mountall.override in this environment to emit / /run local-filesystems,... filesystems.20:08
rcjso they all land at the same time20:09
smoserlxc works fine . this works tehre.20:09
rcjsmoser, I understand that.20:09
smoserupstart will not pass 'mounted mountpoint=/'' until things that 'start on' that are done20:09
smoserie, tahts a blockign event.20:10
rcjsmoser, it's not happy on an lx-branded solaris zone20:10
smoserthat make sense?20:10
smosercan you boot init --verbose or --debug ?20:10
smoserto /sbin/init ?20:10
rcjno, I don't have a kernel cmdline to alter20:10
smoserbut how is /sbin/init invoked ?20:11
rcjsmoser, https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/brand/lx/lx_init/lxinit.c20:16
rcjLine 83120:16
smoserwell, fun20:16
smoseryou can wrap it20:16
tmartinshey guys... When using cloud-init with OpenStack Heat, is it possible to have 2 "user_data" sections at the same time?20:59
tmartinske this: http://pastebin.com/7HiDpjWr ?20:59
tmartinsOr something similar ?20:59
smosertmartins, cloud-config-archive is what is basically taht.21:00
smoseroh. wait... id otn know about heat and if you can actually add the 2 sections tehre.21:01
smoserbut cluod-config-archive baseically allows you to put m ultiple into one user-data21:01
tmartinsbasically, initially, I'm creating a user, then, I'll run a script, that will call Ansible...21:02
tmartinssimpler -> complex21:02
tmartinsMaybe a mix of: "#cloud-config", which is simpler, than, a shell / Ansible, which is more complex...21:04
tmartinsMmm... I think I know how to do it!21:05
tmartinsThe "#cloud-config" supports write_files and runcmd...21:06

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!