[14:52] <cornfeedhobo> hellloooo
[14:52] <cornfeedhobo> i am using AWS + centos 7. could someone tell me why this snippet, pretty much stolen straight from the docs, is not executing? https://gist.github.com/cornfeedhobo/35d60eca0a09ec4bbfb94a957d7520ff
[14:54] <smoser> cornfeedhobo, could you paste /var/log/cloud-init.log and /var/log/cloud-init-output.log if there is anythign interesting there.
[14:56] <utlemming> cornfeedhobo: for CentOS? likely a dependency problem. I don't think CentOS includes growpart and sfdisk by default (I reserve the right to be wrong here...)
[14:57] <cornfeedhobo> smoser: coming up
[14:57] <cornfeedhobo> utlemming: interesting point
[14:57] <cornfeedhobo> (i'll double check that too)
[15:00] <cornfeedhobo> smoser: gist updated
[15:00] <utlemming> cornfeedhobo: how about a dump of 'journalctl' too?
[15:01] <smoser> well thats pretty useless
[15:01] <utlemming> centos is a bit dense when it comes to logging on disk for Cloud-init, sadly
[15:01] <smoser> normally cloud-init.log shoudl have debug level output, which is quite noisy
[15:02] <utlemming> you'd think so...but on CentOS/RH/Fedora, there is a packaging issue where the logs are supposed to be owned by 'adm' (as in Debian/Ubuntu) but the user is not created
[15:02] <utlemming> the result is cloud-init.log is there, but empty
[15:04] <cornfeedhobo> oh yeah, journalctl output is really verbose. updated again.
[15:06] <cornfeedhobo> i'm going through it all now, hadn't thought to check journal.
[15:06] <utlemming> cornfeedhobo: how about /etc/cloud/cloud.cfg? and /etc/cloud/cloud.cfg.d/*
[15:06] <utlemming> looks like its not configured to setup the disks
[15:06] <cornfeedhobo> yeah, i can paste those, but fwiw, this is just the stock AMI for centos7 ... oh interesting. incoming.
[15:10] <cornfeedhobo> okay. updated. .d/* only has a logging cfg, but pasted
[15:11] <utlemming> cornfeedhobo: yup, that's the problem...the default config doesn't setup disks
[15:13] <smoser> cornfeedhobo, you should be able to run (and i'd be interested in the output of)
[15:13] <cornfeedhobo> ah, i saw growpart and mounts and assumed
[15:13] <smoser> sudo cloud-init single --debug --name=disk_setup --frequency=always
[15:13] <smoser> er... maybe --debug before single
[15:13] <cornfeedhobo> okay, let me make sure that module is on disk
[15:14] <utlemming> cornfeedhobo: I just confirmed that the default config doesn't include that
[15:14] <smoser> it'll be there. just disable.d
[15:14] <utlemming> I would argue that this should be a bug in CentOS
[15:15] <cornfeedhobo> hmmm, i cant find the modules folder on disk
[15:16] <cornfeedhobo> sorry, i'm super new to cloud init
[15:16] <cornfeedhobo> repoquery to the rescue, maybe
[15:17] <cornfeedhobo> okay. testing.
[15:18] <cornfeedhobo> https://gist.github.com/cornfeedhobo/64bcf6e2b760af27ddf16889b9849686
[15:19] <cornfeedhobo> it doesn't seem to be loading it, but i see the python module in site-packages
[15:22] <smoser> cornfeedhobo, look in /var/log/cloud-init.log see if you see
[15:22] <smoser>   Running module disk_setup
[15:25] <cornfeedhobo> okay. one sec. question first: is there a way to specify, in my aws userdata, that i want to add fs
[15:25] <cornfeedhobo> fs_setup module to cloud_config_modules  **
[15:26] <cornfeedhobo> from what i can tell, if i add a cloud_config_modules section, it overrides the one in cloud.cfg
[15:28] <utlemming> cornfeedhobo: my $0.02 ZWD....copy /etc/cloud/cloud.cfg to /etc/cloud/cloud.cfg.d/99-me.cfg and add "- disk-setup" to the modules section
[15:31] <cornfeedhobo> hmm, but that means i would have to create a custom AMI with that copied file in place, or kick off cloud-init again, after aws has executed it?
[15:38] <utlemming> cornfeedhobo: there are other ways to convince cloud-init....
[15:40] <cornfeedhobo> well, question: does cloud-init re-process the yaml files between init, config, and final?
[15:42] <smoser> cornfeedhobo, yes.
[15:42] <smoser> so you can feed that list in as user-data
[15:42] <smoser> but it really should have run when you ran it with single there.
[15:43] <cornfeedhobo> hmmm
[15:45] <smoser> i might have to resort to littering /usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py with LOG.info or LOG.warn messages
[15:47] <cornfeedhobo> maybe AWS invokes with more flags? something that pulls from http://169.254.169.254/latest/user-data ?
[15:51] <cornfeedhobo> no, i dont see anythnig special in the service files
[15:51] <cornfeedhobo> i'm going to retry with utlemming's $0.02, utilizing bootcmd
[15:59] <cornfeedhobo> muhahahahaha, it worked
[15:59] <cornfeedhobo> sed 's@cloud_config_modules:@cloud_config_modules:\n - disk-setup@' -i /etc/cloud/cloud.cfg
[16:00] <cornfeedhobo> is fs_setup wrapped up in disk-setup?
[16:08] <smoser> cornfeedhobo, the module name is 'disk_setup' (or disk-setup). that handles both top level config entries 'disk_setup' and 'fs_setup'
[16:09] <cornfeedhobo> that is what i thought, especially after looking at the modules in site-packages. hmmm, i must just have an error in the fs_setup i just tried. i'm going to look around some logs
[16:19] <cornfeedhobo> ah, interesting. "Unable to convert swap to a device"
[16:28] <smoser> powersj, https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/320994
[16:29] <powersj> smoser: sup
[16:29] <smoser> can you tell me why i dont want to just pull that ? and why is the bug referenced there (1669306) ?
[16:30] <powersj> 1) I need to update source and remove source-branch those are pointing at my branch (that was me testing it and forgetting to change it back
[16:30] <smoser> yeah. just saw that and pushed a coment
[16:30] <powersj> 2) the LP is a refrence to "command: usr/bin/python3 $SNAP/bin/cloud-init" which is not the normal line
[16:30] <powersj> it should have been as simple as command: cloud-init
[16:31] <powersj> but there are some python3 issues with classic confined snaps documented in that LP I refrence
[16:31] <powersj> reference*
[16:32] <powersj> smoser: oh and do you want the source to be the master git url or to be '.'
[16:33] <powersj> pointing at master i.e. https://git.launchpad.net/cloud-init made sense, but wanted to confirm
[16:33] <smoser> powersj, id ont know what do you think ?
[16:34] <smoser> what is the "right snap" way to do it.
[16:34] <smoser> i had this general question once looking at snaps
[16:35] <powersj> smoser: well if we use '.' we don't checkout cloud-init yet again, however if you use '.' you can try snapping local changes. As far as convention, I didn't feel there was overarching best practice from what I saw.
[16:35] <powersj> I do like using the git master URL though as it prevents you from doing something silly like making a local change and then having that make it somewhere
[16:37] <smoser> well, you can pick one. and we can figure out why it was wrong later ;)
[16:38] <powersj> hahaha
[16:40] <powersj> smoser: updated with the URL
[16:44] <cornfeedhobo> okay, got that sorted out. thanks for the help smoser & utlemming!!
[17:17] <smoser> powersj, https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/322137 how can i see the pylint warings ?
[17:18] <powersj> smoser: edit the .pylint rc
[17:18] <smoser> i jsut removed it.
[17:19] <smoser> and i dont see anyting
[17:19] <powersj> that works too ;) but you may get even more messages
[17:21] <smoser> sure. but i dont see ones, like if i run 'pylint tools/mock-meta' i see a buch of errors, but nothing about the log.warning
[17:21] <smoser> oh. now id o
[17:23] <powersj> did you change something?
[17:24] <smoser> just wasnt looking right
[18:11] <devicenull> hey, we've started to run into issues with this change https://lists.ubuntu.com/archives/ubuntu-devel/2017-February/039697.html
[18:11] <devicenull> basically, we used to use cloud-init, and dropped it ~4-6 months ago
[18:11] <devicenull> however, a bunch of our customers had created OS images back when we used it
[18:11] <devicenull> and now they're starting to run into error messages when they boot up from those images
[18:11] <devicenull> and I don't really see any way for us to address that, given that the images already exist
[18:11] <devicenull> backporting that to 16.04 seems like an issue
[18:14] <devicenull> I'm guessing we'll probably resort to pretending to be ec2 if this ends up becoming a bigger issue
[18:15] <nacc> if you stopped using cloud-init 4-6 months ago, how woudl a change in cloud-init from 2 months ago be affecting your images?
[18:15] <nacc> devicenull: --^
[18:15] <devicenull> customers still have images with cloud-init built in
[18:15] <rharper> I think they're old images which still have cloud-init inside them
[18:15] <devicenull> and that's getting updated (automatically?), which then triggers the message on next boot
[18:15] <devicenull> correct
[18:16] <rharper> sure, a dist-upgrade would update the cloud-init in them
[18:16] <nacc> ah i thought they weren't able to boot the image
[18:16] <nacc> so i wasn't sure how cloud-init was updating
[18:16] <rharper> well, they're getting the warning
[18:16] <rharper> it's bootable in xenial
[18:16] <nacc> rharper: oh i see
[18:17] <nacc> i read 'error message' as fatal
[18:17] <devicenull> yea, we've only had a couple reports of it so far, so I'm not sure how big of a problem it's going to turn into
[18:17] <rharper> the error message should include the cloud-config that the user can emit in to the image to disable the warning and the check
[18:17] <rharper> it can also be a boot parameter
[18:18] <devicenull> yea, it throws this https://gist.githubusercontent.com/devicenull/63f96ecf5a84c0c9163c09002726d204/raw/b4d0f508382fe4de9a33e5eaf19e7ac92fed8e23/gistfile1.txt
[18:18] <devicenull> which then they end up reporting to us
[18:18] <rharper> ack
[18:21] <devicenull> we've just been telling them to remove cloud-init, but I thought I'd bring this up here
[18:22] <rharper> thanks for reporting;   if possible filing a bug for the issue with the details requested w.r.t the provide; that might help users find that it's been reported
[18:23] <devicenull> yea, we'll probably add some documentation on it too
[18:23] <rharper> we could also in there point back to you're recommendation that best fits the cloud;  alternatively we could explore whether you could expose an identifier (ideall through SMBIOS settings )
[18:26] <devicenull> I'm not sure you really want to be adding code detecting us, when we really don't use cloud-init anymore?
[18:29] <rharper> possibly not; I suppose it may depend on the customer images