#cloud-init 2014-02-03
<harlowja> smoser if u get some time, check out that patch set
#cloud-init 2014-02-04
<yacoco> I'm creating custom datasource for cloud-init, but I'm a little confused of what is the metadata used for.
<yacoco> We could say that the userdata should be a cloud-init configuration written by the user. What should metadata contains? And in what format?
#cloud-init 2014-02-05
<longdays> I have a cloud-config yaml file. It failed during instance initialization. Is there any way I can test the validity of that file from a bash shell after boot time?
<natorious> something like http://yaml-online-parser.appspot.com/ ?
<harlowja> longdays u can just take http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/tools/validate-yaml.py to
<longdays> I was looking to see if I can run the file not during init. the yaml is valid
<longdays> hope that makes sense
<longdays> I just want to debug locally without the long feedback loop of a reboot or respinup of a new machine
<harlowja> longdays ya, just pass the file to validate-yaml
<longdays> so installing the salt minion
<longdays> CalledProcessError: Command '['apt-get', '--option', 'Dpkg::Options::=--force-confold', '--assume-yes', 'install', 'salt']' returned non-zero exit status 100
<longdays> http://pastebin.com/w6mP28Q4
<longdays> https://github.com/number5/cloud-init/blob/8a5e01d7484307a846430ecfe18873627e4b18d1/cloudinit/config/cc_salt_minion.py
<longdays> looking at that file
<longdays> and https://github.com/number5/cloud-init/blob/8a5e01d7484307a846430ecfe18873627e4b18d1/cloudinit/distros/debian.py
<longdays> looks like the -minion is getting stripped
<longdays> still looking through the code though
<longdays> looks like a dated version on the default ubuntu ami in ec2
<longdays> 12.04
#cloud-init 2014-02-07
<mfisch> smoser: would it be possible to pass-in a TENANT_ID (with some changes to cloud-init)?
<harlowja> mfisch what type of cloud are u using, if the cloud doesn't provide the tenant-id in a accessible way to cloud-init, then makes it impossible for cloud-init to know about it
<mfisch> openstack nova
<mfisch> and I'm not sure it is being provided
<mfisch> if it is, I'd like to help enable it, it will let me do some better tagging with landscape
<harlowja> i don't think it is :-/
<mfisch> I was hoping I could do tagging in landscape by tenant
<mfisch> makes it easier for me to separate stuff
<harlowja> agreed
<mfisch> well if not, I'll just work on getting landscpae registration functional at all
<mfisch> right now it's choking
<harlowja> i know i've made a patch that does provide it, but i haven't upstreamed it yet 
<mfisch> harlowja: I found smoser's sample config, you're saying that wont work?
<harlowja> mfisch so that would probably be via user-data, which would work
<harlowja> the other nicer way is to just have openstack write it (since it obviously knows the userid/tenantid)
<harlowja> mfisch http://paste.openstack.org/show/63128/
<harlowja> its nothing complicated
<harlowja> i should just upstream that sometime
<harlowja> then it shows up in metadata (accessible then via everywhere in cloud-init)
<harlowja> maybe someone has already even upstreamed that, i haven't checked
<mfisch> looking now
<mfisch> thats easier to fix than cloud-init in 12.04 actually
<harlowja> yup
<harlowja> cloud-init already reads the metadata and exposes it as a dict, these are just 2 new keys
<harlowja> *and harmless keys (so i wouldn't expect anyone to complain about it upstream)
<mfisch> is instance_id passed in there
<mfisch> probably this guy
<mfisch> instance_metadata.InstanceMetadata(instance,
<harlowja> ya, thats there
<harlowja> metadata has 'instance-id' key afaik
<harlowja> or uuid
<harlowja> one of those 
<mfisch> makes sense
<mfisch> you push it up, I'll +1 it 
<harlowja> :)
<harlowja> k, let me see
<mfisch> I have a good use case for it
<mfisch> harlowja: if you do it, ping me
<harlowja> k
<mfisch> now back to getting base landscape working
<harlowja> in progress, sh9ouldn't be long
<harlowja> just gotta figure out where that code moved to, lol
<mfisch> line 2512
<mfisch> one more dumb question, it looks like the INSTANCE_ID is lost for "runcmd", its only there for bootcmd
<harlowja> hmm
<mfisch> so I guess I'll dump it to a file there and pull it back out?
<harlowja> mfisch are u using config-drive or ec2 datasource in cloudinit?
<harlowja> that will actually change what metadata is avail afaik
<mfisch> so this is a dumb answer but I don't know
<harlowja> k
<mfisch> just starting playing with this a few hours ago
<mfisch> I believe the logs say ec2
<harlowja> ah, then this patch might not help 
<mfisch> found data source: DataSourceEc2
<mfisch> what decides the datasource?
<mfisch> this is on a RHEL system, not what I'm deploying on
<harlowja> cloud-init, but openstack has a config which actually decides it
<mfisch> if that matters
<mfisch> oh I can hack that
<harlowja> cloud-init will serach for one
<harlowja> let me check, the code paths are 2 different paths for the different pieces in openstack
<harlowja> ya, let me see, i think it might work
<med_> hey smoser.
 * med_ was asked to join
<mfisch> med_: I'm talking with harlowja actually
 * med_ sees harlowja but no utlemming
<med_> xlnt
<mfisch> med_: our poc is using ec2 datasource, what implications will that have for something like the landscape config working?
<med_> Hmmm. I think I've mentioned this at one point or another, I don't know jack about landscape.
<med_> lame excuse but haven't had time to dig into it.
<med_> Not sure why it would matter for cloud-init-y stuff.
<mfisch> we're also discussing the other question of how to add a TENANT_ID into the data passed from nova, harlowja has a patch for nova
<med_> cool beans
<harlowja> mfisch  https://review.openstack.org/72018
<harlowja> feel free to add comment with your use-case, incase the nova reviewrs say this is dumb
<harlowja> it does appear that without altering the ec2 format though i can't add this change into that part of the code :(
<harlowja> so its either https://code.launchpad.net/~harlowja/cloud-init/ds-openstack needs to merge or u might have to switch to using the config-drive option
<harlowja> http://docs.openstack.org/grizzly/openstack-compute/admin/content/config-drive.html
<harlowja> lets see how much the nova people complain :-p
<mfisch> harlowja: looking
<mfisch> harlowja: and I assume that cloud-init can parse from the config-drive in lieu of ec2
<harlowja> mfisch ya
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/ds-openstack  will use the more native openstack metadata endpoint
<harlowja> *unifying that code in cloud-init (since its the same data exposed via the metadata rest endpoint and the config drive)
<mfisch> that may not help us until 14.04 I'd imagine
<harlowja> correct
<harlowja> cfgdrive though does work
<harlowja> and should be in most modernish versions of cloud-init
 * harlowja not sure when ubuntu started shipping the version that had it
<mfisch> I can look
<harlowja> *canonical, not ubuntu
<harlowja> that company, haha
<mfisch> I suspect that I'll be happy to get partially there and then happier when 14.04 comes out
<harlowja> sure
<med_> config drive just works
<mfisch> med_: is it default enabled on ubuntu?
<med_> We've also got to to worry about cloudbase-init (for Win images)
<med_> 'default enabled on ubuntu' ? For config drive? It's an openstack thing
<med_> not sure what you mean by default enabled.
<med_> or do you mean does cloud-init grok config drive? Yes, it does.
<mfisch> force_config_drive=True
<mfisch> looks like we have it enabled
<harlowja> ah, cool
<harlowja> u should be able to check inside a vm
<harlowja> blkid 
<harlowja> will tell u 
<harlowja> something like 'LABEL="config-2" UUID="5B88-2F2F" TYPE="vfat" '
<mfisch> right
<mfisch> that system with it enabled is not fully working atm, I think I may have another one though
<med_> config drive has been in precise since at least Feb 2012
<harlowja> k
<harlowja> ya, i forget when it appeared
<harlowja> smoser might remember
<med_> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/857378
<med_> to make OVF work with cloud-init
<harlowja> k, so should be fine i think
#cloud-init 2014-02-08
<harlowja> smoser ok, https://code.launchpad.net/~harlowja/cloud-init/ds-openstack/ all good, tests all over and all that
<astol> hi all, how do I tell cloud-init to run first boot commands again on next reboot? (ie reset first boot attribute)
#cloud-init 2014-02-09
<veselin> Hi all, a question on 'cloud config'. What is the correct way to reference user-data values into the cloud-config yaml file? E.g. if I pass a Mime Multi Part User-Data message which includes a free text 'myvar=123' and 'text/cloud-config' yaml, how could I call retrieve the myvar value from within the yaml config?
#cloud-init 2015-02-02
<harmw> hm, wasn't thre someone from Ironic here?
<harlowja> ya, jayf although not sure where he is
<harmw> ah yes
<harmw> smoser: whats keeping you from pushing out a new release?
<smoser> python3
<harmw> beeing p3 compatible I presume?
<smoser> yeah. barry did most of the wor, but i want to get finished up on that.
<harmw> ok, cool
#cloud-init 2015-02-03
<Kodokuu> Hi
<Kodokuu> I have a issue with cloud-init : http://pastebin.com/4iwZvCxU
<Kodokuu> I use openstack and vcenter backend for hypervisor
<Kodokuu> How I force cloud-init to use EC2 cloud-type ?
<Kodokuu> Ping ?
<larsks> Kodokuu: I think you can modify the 'cloud.cfg' in your image to restrict the available data sources.  There may also be a boot parameter you can use.
<kodokuu_> larsks i have modify cloud.cfg and add datasource but always vsphere in my log
<kodokuu_> and always warning : Unhandled non-multipart (text/x-not-multipart) userdata:
<kodokuu_> I try to install package with user-data, so I add packages httpd in user-data. 
<kodokuu_> [WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'packages:\n - httpd\n - my...'
<kodokuu_> so cloud-init see my user-data but no execute package install ( I have package-update-upgrade-install)
<larsks> kodokuu_: does your user-data start with '#cloud-config'?
<larsks> That error sounds like cloud-init doesn't recognize what you're passing it.
<kodokuu_> Shame on me :/
<kodokuu_> i retry
<kodokuu_> larsks Ok I Have no warning  but cloud-init don't install packages
<kodokuu_> Always same error in log : http://pastebin.com/4iwZvCxU
<larsks> kodokuu_: What if you modify the cloud.cfg *on disk* to have: datasource_list: ['OpenStack']
<larsks> Or "Ec2" I guess, or whatever the EC2 data source is called.
<larsks> What if you try using a config drive? I think that takes precedence over network data sources.
<kodokuu_> larsks a config drive ?
<larsks> kodokuu_: yes, cloud-init can read config from a locally attached cd-rom device rather than a network data source.  http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#config-drive
<larsks> With openstack, you get a config drive if you specify --config-drive=true when booting an image.
<kodokuu_> i boot with horizon ^^
<larsks> There's probably a checkbox for that.
<kodokuu_> And I use vcenter hypervisor so hard to boot with config drive
<kodokuu_> I think is for KVM libvirt
<larsks> There is nothing kvm/libvirt specific about config drivers.
<larsks> In horizion, in the "advanced options" tab, there is a "configuration drive" checkbox.
<kodokuu_> I can reach 169.254.169.254, i have a route so why use config drive ?
<kodokuu_> I use Icehouse version, I dont have this checkbox
<larsks> kodokuu_: Huh, bummer.  Anyway, I was just suggesting ways of working around the error.
<kodokuu_> ok thx for you help :) I can execute bash script in user-data, cloud-init execute my script but cloud-init format never works
<larsks> this is sometimes caused if the default cloud.cfg does not enable the package-update-upgrade-install module. Check /etc/cloud/cloud.cfg.
<larsks> ...at least, if the only problem is that packages won't install.
<larsks> If cloud-config user-data doesn't work at all, I'm not sure.
<kodokuu> larsks I enabled this module in cloud.cfg  
<kodokuu> I retry with other module
<smoser> kodokuu, can you give example of the config ?
<smoser> it has to start with '#cloud-config' .
<kodokuu> smoser yes
<kodokuu> http://pastebin.com/e8QtYwrU
<smoser> what is cloud-init version and OS ?
<kodokuu> Maybe important: I use cloud-init 0.7.4
<smoser> and is there /var/log/cloud-init*
<kodokuu> and Rhel 6.4
<kodokuu> smoser http://pastebin.com/VD7LHr13
<kodokuu> Since rm -rf /var/lib/cloud/  i have  No instance datasource found!
<smoser> ah. yeah. 
<smoser> sudo cloud-init init --local
<kodokuu> ok
<kodokuu> and after ? 
<kodokuu> I will try tomorrow thxfor help :p^
#cloud-init 2015-02-04
<larsks> What if...what if cloud-init located data sources and config modules through something like stevedore
<larsks> ?
<larsks> Or is that just crazy talk?
<smoser> larsks, well, that is the plan. 
<smoser> something like that.
<larsks> Oh, awesome. So not crazy talk :)
<smoser> the explicit config provides a copule things though:
<smoser> a.) a order (hint, ec2 sucks cauase it has to poll and timeout)
<smoser> b.) ability to turn off bad behaving ones easily
<larsks> Well, I can see keeping the explicit config and still locating modules via the plugin mechanism.
<smoser> you could accomplish both with loading and osme mechanism, but whats there is easy.
<larsks> That is, a module is only used if (it is discovered) and (it is enabled explicilty)
<smoser> well, even that is a pain.
<smoser> because peo;le want to just drop their datasource in and not have to modify a config (that woul dnot explicitly list their datasource)
<larsks> Hmmm.
<larsks> Seems like you need the explicit config for (a) and (b).  Maybe those people are just picky :). 
#cloud-init 2016-02-08
<Creeture> How do I access the key/value pairs that are generated when cloud-init reads from DataSourceOVF? Is it a custom scripting effort or something that I can get with #cloud-config ?
#cloud-init 2016-02-10
<minfrin> A quick question on AWS and cloud-init, to see whether anyone has seen this behaviour before.
<minfrin> I have an EC2 instance with an EBS volume attached to it, and I;ve asked cloud-init to format it like so:
<minfrin> fs_setup:
<minfrin>   - label: data
<minfrin>     device: /dev/xvdh
<minfrin>     filesystem: ext4
<minfrin> When cloud-init starts, mke2fs fails as follows:
<minfrin> 2016-02-10 15:03:42,593 - util.py[WARNING]: Failed during filesystem operation
<minfrin> Failed to exec of '['/sbin/mkfs.ext4', '/dev/xvdh', '-L', 'data']':
<minfrin> Unexpected error while running command.
<minfrin> Command: ['/sbin/mkfs.ext4', '/dev/xvdh', '-L', 'data']
<minfrin> Exit code: 1
<minfrin> Reason: -
<minfrin> Stdout: ''
<minfrin> Stderr: 'mke2fs 1.42.9 (4-Feb-2014)\nCould not stat /dev/xvdh --- No such file or directory\n\nThe device apparently does not exist; did you specify it correctly?\n'
<minfrin> Later on, after sshing into the machine, I am able to run mke2fs manually and the formatting of the block device works fine.
<minfrin> Are they are delays that people are aware of when deploying AWS EBS drives that would cause cloud-init to fail?
<waldi> check the kernel log
<minfrin> Not following - what would I check the kernel log for? (I assume you mean dmesg)
<rcj> smoser, I'm debugging a new DS where the config module fails during boot "Can not apply stage config, no datasource found!" but after boot I can run it 'service cloud-config start' just fine.  Any tips on debugging or ideas of what is happening?
<waldi> minfrin: it should list when xvdh appeared. why do you have _eight_ disks?
<minfrin> The only mention in dmesg of the xvdh disk is the very last lines, which could be warnings triggered during the successful mkfs.ext4:
<minfrin> [   40.226724] blkfront: xvdh: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
<minfrin> [   40.233951]  xvdh: unknown partition table
<minfrin> What I was hoping to find was if anyone had any AWS experience, and whether there were any known issues with AWS running cloud-init before all resources were ready?
<smoser> rcj, probably log says WARN somewhere
<smoser> and also /run/cloud-init/*.json
<smoser> minfrin, you're rpobably right. the disk probably was not there when cloud-init r un.
<smoser> i'd not seen that, but the no such file or directory is pretty clear
<minfrin> Some digging has found this patch to something called "rubber" from 2012 which referred to bugs with AWS volumes not being ready: https://github.com/rubber/rubber/pull/156/files
<minfrin> The patch seems to pause in a loop until the device exists, and then continues.
<smoser> thats definitely a sane path for some situations.
<smoser> but in others its not sne.
<smoser> its posible we could let you define the behavior to cloud-init.
<smoser> the issue is that it sits there and waits forever for that device to appear, and its not there....
<smoser> then its not going to get to the point where it gets your ssh keys or runs your user-data
<smoser> and you're not going to ever be able to get in.
<rcj> smoser, no WARN.  This is a container environment where we're emitting some missing signals for network and mounting FSes.  Could be that I've inadvertently changed the order the cloud-init jobs are run.  is cloud-config run before cloud-init-local?
<smoser> do you have something in /run/cloud-init ?
<rcj> yes
<rcj> http://paste.ubuntu.com/15009184/
<rcj> smoser, ^ there is results.json (prior run) and here is cloud-init-output.log from the latest run http://paste.ubuntu.com/15009195/
<smoser> rcj, config definitely did start before init
<rcj> smoser, should it?
<smoser> no.
<minfrin> smoser: An option to work around the device-not-ready problem is a "wait" option, giving the longest time we're prepared to wait for the drive to become available.
<smoser> minfrin, well, nto if you're trying to mkfs
<smoser> oh. i see, yeah.
<smoser> sorry. i thought you were saying upstart or systemd wait for the filesystem
<smoser> but yeah
<minfrin> smoser: Still waiting for word from AWS support about this, need to work out how widespread the problem is and how it will affect us.
<rcj> smoser, thanks.  this is an issue with this particular container running things out of order due to upstart overrides that change emit timing for cloud-init prereqs.
<jmccann> I was wondering if someone could help me finish getting a CCA signed so I could try contributing to cloud-init
<jmccann> I'm on page http://www.ubuntu.com/legal/contributors/submit and not sure what to fill for "Please add the Canonical Project Manager or contact"
<harlowja_at_home> smoser, ^
<harlowja_at_home> scott moser i think is said contact still
<harlowja_at_home> mr.scott
<harlowja_at_home> lol
<jmccann> thanks much!
<smoser> jmccann, yeah. you can put me that is fine
<jmccann> thanks!
<rcj> smoser, http://paste.ubuntu.com/15010669/ is what I need in this container environment (or perfectly replicating the FS-related emits) because right now I have a race.
<rcj> smoser, but I don't know that is acceptable for Trusty SRU
<smoser> rcj, somethign else is getting broked
<rcj> smoser, how so?
<smoser> so 'filesystem' should not be emitted until /  is mounted rw
<smoser> and / should not be mounted rw until cluod-init is stopped
<rcj> but it's a container so when it's started all the filesystems are mounted and ready.  they have a mountall.override in this environment to emit / /run local-filesystems,... filesystems.
<rcj> so they all land at the same time
<smoser> lxc works fine . this works tehre.
<rcj> smoser, I understand that.
<smoser> upstart will not pass 'mounted mountpoint=/'' until things that 'start on' that are done
<smoser> ie, tahts a blockign event.
<rcj> smoser, it's not happy on an lx-branded solaris zone
<smoser> that make sense?
<smoser> can you boot init --verbose or --debug ?
<smoser> to /sbin/init ?
<rcj> no, I don't have a kernel cmdline to alter
<smoser> but how is /sbin/init invoked ?
<rcj> smoser, https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/brand/lx/lx_init/lxinit.c
<rcj> Line 831
<smoser> well, fun
<smoser> you can wrap it
<smoser> http://paste.ubuntu.com/15010796/
<rcj> yes
<tmartins> hey guys... When using cloud-init with OpenStack Heat, is it possible to have 2 "user_data" sections at the same time?
<tmartins> ke this: http://pastebin.com/7HiDpjWr ?
<tmartins> Or something similar ?
<smoser> tmartins, cloud-config-archive is what is basically taht.
<smoser> oh. wait... id otn know about heat and if you can actually add the 2 sections tehre.
<smoser> but cluod-config-archive baseically allows you to put m ultiple into one user-data
<tmartins> Mmm...
<tmartins> basically, initially, I'm creating a user, then, I'll run a script, that will call Ansible...
<tmartins> simpler -> complex
<tmartins> Maybe a mix of: "#cloud-config", which is simpler, than, a shell / Ansible, which is more complex...
<tmartins> Mmm... I think I know how to do it!
<tmartins> The "#cloud-config" supports write_files and runcmd...
#cloud-init 2016-02-11
<bswift> i'm trying to customize a CentOS image, and I've never used cloud-init before.   I prefer to configure it properly rather than nuke it.  default_user - is configured in /etc/cloud/cloud.cfg.   I'd like to remove that entry via this "user data" method. Expecting to drop a file that overrides default configs..  am I on the right path?
<bswift> ah, found the clouds.cfg.d folder.   that works for me..   must learn more about cloud-init..  love it :)
#cloud-init 2016-02-12
<drue> when making AMIs (with packer and from upstream ubuntu/trusty), do i need to clear our /var/lib/cloud? cloud-init seems to run fine without doing that, but i'm not sure why.
#cloud-init 2017-02-06
<Cust0sLimen> hi
<Cust0sLimen> I want to use rhel-guest-image-7.0-20140930.0.x86_64.qcow2 on libvirt directly
<Cust0sLimen> so I found some info regarding cloudinit http://blog.oddbit.com/2015/03/10/booting-cloud-images-with-libvirt/
<Cust0sLimen> it says you create a iso with some files in and then put it in vm cdrom
<Cust0sLimen> what I want to now though is - is there some other way to do this except for ios ?
<Cust0sLimen> what I want to now though is - is there some other way to do this except for iso ?
<Cust0sLimen> can't I run some webserver or something ?
<cpaelzer> Cust0sLimen: well, there are many datasources - the question is what is available at a usual qemu/libvirt level
<cpaelzer> Cust0sLimen: http://cloudinit.readthedocs.io/en/latest/topics/datasources.html
<cpaelzer> Cust0sLimen: the link you posted creats a config drive
<Cust0sLimen> yeah
<Cust0sLimen> so I basically want same but not a iso but rather like webserver or something
<cpaelzer> Cust0sLimen: in general there are e.g. http://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html
<cpaelzer> Cust0sLimen: also aws and google sources are network based
<cpaelzer> Cust0sLimen: yet - be aware
<Cust0sLimen> hmm ok
<cpaelzer> Cust0sLimen: these cause all sort of issues (slow boot by probing all), potential security risks of people insering configs, ...
<cpaelzer> Cust0sLimen: therefore most of these have extra checks to only access them if in the right env
<Cust0sLimen> but then I have to have openstack metadata service running ?
<cpaelzer> Well you could also mimic it, but I'd expect more guards to those network services to appear which makes it even harder to use
<cpaelzer> I can't find the right link at the moment, but we are working on guarding that more
<Cust0sLimen> ok let me rather just do ios
<Cust0sLimen> iso*
<Cust0sLimen> config disk
<Cust0sLimen> so where is the allowed values/schema for the meta-data and user-data files on config disk ?
<cpaelzer> Cust0sLimen: only see your question now
<cpaelzer> Cust0sLimen: you mean on-disk like the config-disk as iso?
<cpaelzer> Cust0sLimen: or if you modify the target image?
<Cust0sLimen> cpaelzer, no on config-disk ISO
<cpaelzer> Most is really reachable from http://cloudinit.readthedocs.io/en/latest/topics/datasources.html
<cpaelzer> let me check
<Cust0sLimen> but actually nvm, the right way to do it is virt-customize I think
<Cust0sLimen> I'm using that now
<cpaelzer> ok
<cpaelzer> smoser: if you read abcklog is there anything you might add?
<Cust0sLimen> it can set root password and add keys which is fine
<cpaelzer> I happen to miss the wider scope sometimes but smoser has it
#cloud-init 2017-02-08
<Raboo> hi
<Raboo> is the meta-data part needed for a working cloud-init configuration?
<larsks> Raboo: can you elaborated on your question? What are you trying to do?
<powersj> magicalChicken: the integration tests for proposed seem to have gotten stuck recently during upgrade system
<powersj> python3 -m tests.cloud_tests run -v -n xenial --repo 'deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main' -t tests/cloud_tests/configs/modules/write_files.yaml
<powersj> is an example run that just hangs
<magicalChicken> powersj: huh, that's strange. is this the current version of the tests or the one in master?
<magicalChicken> if its the version of tests currently in master then i bet something's broken in -proposed
<magicalChicken> since that version has the old upgrade behavior
<powersj> magicalChicken: master
<powersj> is it still the case that when we add the proposed repo we only install cloud-init, so other pkgs would not affect us?
<magicalChicken> no, that behavior is in the devel branch
<magicalChicken> it isn't in master
<magicalChicken> it didn't make it into the original merge
<magicalChicken> master is broken still
<powersj> magicalChicken: ah! ok
<powersj> thanks! that should explain it
<magicalChicken> that fix is included in the big merge proposal at https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218, but I can pull it out into a separate branch as well if you want to build from taht
<powersj> Nah, however, since you brought it up, what work are you doing to that merge?
<magicalChicken> after discussion with smoser, I pulled the bddeb stuff out into a different mp, since there's additional work to do there to streamline build process and to make it a more general purpose tool
<magicalChicken> I switched the ubuntu releases config back to using ubuntu daily, so we're testing official images, and went through some more testing on tha
<smoser> magicalChicken, wheres that ?
<magicalChicken> smoser: one sec, let me find it
<magicalChicken> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/314496
<magicalChicken> I haven't gotten to any of the additional cleanup stuff on that, but I may be able to this week
<magicalChicken> the original mp has been fixed to use ubuntu daily images
<powersj> magicalChicken: thanks
<magicalChicken> there's a bit more cleanup work to do on the system_ready_scripts that was discussed, and I want to test all of this a bit more, I can probably get that done today
<powersj> magicalChicken: let me know if I can help testing
<magicalChicken> powersj: sure, my goal is to just run through most of the common testing scenarios on the releases that are enabled and make sure they work, it may be worth testing if jenkins + tox + citests all work together nicely as well
<magicalChicken> its looking right now like the system ready scripts for ubuntu releases aren't happy, I forgot to fix that after the rebase
<magicalChicken> I'll get that sorted in a bit
#cloud-init 2017-02-09
<ffledgling> Hello, I was wondering if someone would happen to know how to delay installing packages until the last step/stage when cloud init runs?
<Raboo> larsks if i use the NoCloud provisioner does it need the <url>/meta-data or is it sufficient with the <url>/user-data? So if i reply with an empty meta-data..
<larsks> Raboo: the nocloud data source doesn't require any url; it gets metadata and userdata from files on disk.  See http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<Raboo> larsks you can use SeedFrom clause
<Raboo> larsks seedfrom: http://my.example.com/i-abcde
<larsks> Sure, but you didn't say you were using that, and I am not a mind reader.  Looking through the code it *looks* as if you should be able to get away with only providing the user-data. Are you seeing different behavior?
<Raboo> larsks did a fast try on a machine, it failed, but i don't know where since I couldn't login, so im going to add a static user to that image so I can login and debug
<Raboo> larsks but first i wanted to know if the meta-data was a requirement, since i don't have that implemented I thought I would start there if it was.
<larsks> Have you also tried providing some minimal meta-data just to see what happens?  That should be an easy test.
<Raboo> haven't yet.
<larsks> Just serve out the metadata from the nocloud examples as a static file or something.
<Raboo> ok
<Raboo> but it's a bit more than 'just' since it's a plugin for another tool
#cloud-init 2017-02-10
<bugbuster> Hi
<Guest7672> I have a problem with understanding the merging of config with cloud-init.
<Guest7672> We have a custom image with a custom /etc/cloud/cloud.cfg containing the defaults to bootstrap out instances. This config file also contains "bootcmd" and "runcmd".
<Guest7672> When a user specifies additional "bootcmd" via userdata, only the use provided bootcmd's are executed, not the once in /etc/cloud/cloud.cfg.
<Guest7672> I already added merge_how: 'dict(recurse_array,no_replace)+list(append)' to the userdata yaml, however same result.
<Guest7672> Any hints ?
<Guest7672> (p.s. I need the bootcmd as puppet4 is not supported)
<Guest7672> By the way: why not use "puppet config set NAME VALUE --section", which hides config files and puppet versions from cloud-init code ? (and thus making https://bugs.launchpad.net/cloud-init/+bug/1446804 easier to fix)
<Guest7672> Could the merge behaviour described be caused by https://bugs.launchpad.net/cloud-init/+bug/1532234
#cloud-init 2018-02-05
<jchevalier> Hello
<jchevalier> I struggle configuring cloud-init on azure with a SLES image, can anyone help me please?
<jchevalier> Anyone is user cloud-init on azure ? I'd need some help to make it work!
<jchevalier> thx
<Odd_Bloke> jchevalier: It's better to just describe the problem you're having, rather than asking for help.
<Odd_Bloke> People may come online later and be able to respond when you are idle.
<Odd_Bloke> Or people may not want to respond without knowing the scope of what they're offering to help with. :)
<mgerdts> I'm confused.  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1667735 says that a fix was released, but I don't see anything in the bug that suggests in what release was fixed and I don't see anything in git log or git grep that mentions this bug.
<ubot5> Ubuntu bug 1667735 in cloud-init (Ubuntu Trusty) "cloud-init doesn't retry metadata lookups and hangs forever if metadata is down" [Medium,Confirmed]
<mgerdts> I certainly don't see the flush+negotiate in there, but that seems to be a separate issue from a timeout.
<paulmey> jchevalier: I can help, hope you're still here... :-)
<paulmey> smoser: is there an ETA for an SRU to get some of the bugfixes from the last two months into trusty/xenial/artful?
<smoser> paulmey: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1747059
<ubot5> Ubuntu bug 1747059 in cloud-init (Ubuntu Artful) "sru cloud-init (17.1.46-g7acc9e86-0ubuntu1) update to 17.2.30-gf7deaf15-0ubuntu1" [Medium,Confirmed]
<smoser> paulmey: ^ that bug is the sru bug, so... its in progress.
<blackboxsw> paulmey:  I'd guess ~2 weeks as that's how long our last SRU took
<blackboxsw> and this is a pretty big SRU
<paulmey> smoser: blackboxsw: yeah, we saved up a lot... :-)
<paulmey> hence the question...
<paulmey> but thanks, easy to track now
<smoser> blackboxsw: i wrote a template for https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1581416
<ubot5> Ubuntu bug 1581416 in cloud-init (Ubuntu Xenial) "grub-legacy-ec2 installs files in /etc/kernel/kernel/post(inst|rm).d" [Low,Confirmed]
<smoser> i thikn we're good on bug statenow
<blackboxsw> thanks smoser
<mgerdts> @smoser - any comments on the bug I mentioned earlier?  It was closed by you.  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1667735
<ubot5> Ubuntu bug 1667735 in cloud-init (Ubuntu Trusty) "cloud-init doesn't retry metadata lookups and hangs forever if metadata is down" [Medium,Confirmed]
<mgerdts> I don't know if was accidentally closed or I am just showing how much of a newbie I am with launchpad.
<smoser> hm.
<smoser> mgerdts: it looks in the right state.
<mgerdts> The last comment says: Changed in cloud-init (Ubuntu):
<mgerdts> status:	Fix Released â Confirmed
<mgerdts> but above, it lists lots of releases where it's not fixed.  I'm having troubles decoding that.
<mgerdts> I think an experiment I did last week suggests it's not fixed.
<mgerdts> (not fixed in the 17.10 image for SmartOS, that is)
<smoser> ok. so i think this is in the right state, a bit of launchpad education needed (i agree, it is confusing)
<mgerdts> perhaps "Confirmed" is an earlier state than "fix released"
<mgerdts> was accidentally set to fix released, then updated?
<smoser> the 'cloud-init' task is the upstream project.  its not fixed in upstream, 'Confirmed' is the right state.
<smoser> yeah, it was accidently changed i think.
<smoser> the 'cloud-init (Ubuntu)' tracks if this bug affects "Ubuntu". which by default means "in the development release", which is currently bionic.
<mgerdts> got it.  I had read that as it was fixed, then some testing was done to confirm that it was really fixed.
<smoser> then there are 'Precise' and 'Trusty' tasks that are also Confirmed...
<smoser> we could "also affects distribution/package" to add 'Xenial' and 'Bionic' specifically, but we dont really have to
<smoser> and honestly i should just remove Precise becase we're never going to fix it there.
<smoser> and probably not in trusty
 * mgerdts needs to find the decoder ring for Ubuntu versions.
<smoser> for your info... it'd take some pushing to get this fixed in trusty.
<smoser> just due to age.
<smoser> https://wiki.ubuntu.com/Releases
<mgerdts> thanks... I didn't mean to suggest I was asking you to answer what google can answer quickly.
<smoser> mgerdts: fix commit message to
<smoser> a.) be Subject-Line\n<blank-line>\nmore-description
<smoser> b.) reference your bugs with:
<smoser>  LP: #BUG
<smoser> (see examples in 'git log')
<mgerdts> ok
<jchevalier> I am trying to install cloud-init on SLES image for azure, but whatever I try 1- I can't connect through ssh keys ans 2- can't run a script with the custom-data parameter
<jchevalier> What I tried :
<jchevalier> Create a vm with the image SUSE:SLES:12-SP2:latest
<jchevalier> Then install cloud-init following this: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/cloudinit-prepare-custom-image
<jchevalier> Set cloud-init enabled through systemctl
<jchevalier> Install packages to access ntfs partitions (I had errors in cloud-init log)
<jchevalier> Then I deprovision the user with waagent
<jchevalier> Finally, deallocate the vm, generalize it and create a image from it
<paulmey> jchevalier: sounds like you're doing the right things...
<jchevalier> Thx for answering
<paulmey> enable boot diagnostics when you deploy your new cloud-init VM and take a look at the console output to see if there is anything interesting in there
<jchevalier> I can do that from CLI?
<paulmey> for some reason they have not implemented that yet
<paulmey> you can enable it later, but that doesn't help you much
<jchevalier> I'll try to create my vm from the dashboard
<paulmey> or you can take a look at this example: https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-from-user-image
<paulmey> I think you can also deploy that directly from github, but ymmv...
<jchevalier> I get a look at the example
<jchevalier> Re, sorry my connection is not as stable as I want
<jchevalier> Here is the boot diagnostic result
<jchevalier> [    0.000000] Initializing cgroup subsys cpuset [    0.000000] Initializing cgroup subsys cpu [    0.000000] Initializing cgroup subsys cpuacct [    0.000000] Linux version 4.4.103-92.56-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Wed Dec 27 16:24:31 UTC 2017 (2fd2155) [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.4.103-92.56-default root=UUID=223f6e86-5f3c-46e4-84ec-fb8b064219af root=/dev/disk/by-
<jchevalier> Oh, cr*p
<jchevalier> too long
<rharper> jchevalier: you may want to use paste.ubuntu.com  or other simple paste services for sharing
<jchevalier> Here the log
<jchevalier> https://paste.ubuntu.com/26527141/
<jchevalier> I can see that cloud-init has issues to find waagent
#cloud-init 2018-02-06
<paulmey> jchevalier: yep, that's the issue
<paulmey> [   59.017405] cloud-init[1306]: 2018-02-05 23:05:12,405 - util.py[WARNING]: agent command '['service', 'walinuxagent', 'start']' failed.
<paulmey> I'm deploying a SUSE:SLES:12-SP2:latest vm to see if that service even exists (or maybe it's called something else)
<rharper> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1747554
<ubot5> Ubuntu bug 1747554 in cloud-init (Ubuntu) "cloud-init status in bionic stack traces" [Undecided,New]
<rharper> blackboxsw: ^^
<paulmey> jchevalier: it's called 'waagent'
<paulmey> in the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg file that you created, you can add this yaml snippet to change the command it tries to execute:
<paulmey> https://www.irccloud.com/pastebin/QwTiV5GT/
<paulmey> or you can also add a new file in /etc/cloud/cloud.cfg.d with this content
<blackboxsw> rharper: that's fixed and landed already :/ still waiting on cloud images to build :)
 * blackboxsw marks as duplicate. need to find the original
<blackboxsw> #1744796
<blackboxsw> bug #1744796
<ubot5> bug 1744796 in cloud-init "cloud-init status traceback on successful Ec2" [Medium,Fix committed] https://launchpad.net/bugs/1744796
<Arianrhod> hi to all
<Arianrhod> can i set on cloud init file a specific release for red hat SO?
<Arianrhod> i found how to set user and passwd to register a vm when i create it... buit can i force to use a specific release?
<smoser> Arianrhod: specific release ?
<Arianrhod> yes
<smoser> release of what?
<Arianrhod> sorry
<Arianrhod> for my new instance, that is red hat
<smoser> you're wanting to tell
<smoser> you're wanting to tell cloud-init what release of cloud-init ? or of redhat ?
<Arianrhod> uh.. ok sorry cloud-init 0.7.9 and red hat 7.3
<Arianrhod> when i create a new instance by horizon, i add a file with an additional conf to specify user and pÃ¨assword to register the instance... i tried to use the key "release" but it tell me that "release is not a valid key"
<PsyRabbit> Hi, I have a VMWare V-Cloud Director and want to use cloud init with it. What is the best way to get the meta data delivered to the VM? I'm starting from scratch and read all from the documentation... The configuration itself shoulnt be a problem I just dont understand how to get the metadata into the machine like instance id. Would be nice if someone can provide a useful link. Thanks
<smoser> PsyRabbit: interesting.
<smoser> we have two contributors from vmware that have focused on this, and contributed code, but i honeslyt do not know how it is all set up.
<smoser> they do use the OVF datasource (a cdrom attached with ovf-environment.xml on it)
<PsyRabbit> Thank you for the hint. Do you know where the code is contributed? Maybe I can figure out how they did their setup.
<ybaumy> https://github.com/number5/cloud-init/tree/master/doc/sources/ovf
<PsyRabbit> Thank you very much!
<ybaumy> after reading a little bit i still dont get it.. i must say
<ybaumy> but look at the code maybe you will figure it out
<PsyRabbit> I hope so... I think they created a new VM and some external stuff communicated with the API, get the data, built the image and then the first VM start shuld be triggered via API... This will be a long night in my employers basement
<smoser> PsyRabbit: if they were around, Sankar Tanguturi is stanguturi and Maitreyee Saikia is msaikia
<smoser> but neither is currently here.
<ybaumy> stanguturi: are you psychic?
<PsyRabbit> I will look out for them... Thank you, you helped me alot.
<stanguturi> ybaumy: Sorry. I didn't understand.
<ybaumy> stanguturi: we were just talking about you
<ybaumy> ;)
<ybaumy> and then you joined
<dpb1> heh
<stanguturi> ybaumy: Oh ok. Yeah about what?
<stanguturi> ybaumy: May I ask the topic?
<dpb1> that might have had something to do with the email scott sent
<ybaumy> dpb1: ahh ok
<ybaumy> :D
<dpb1> :)
<ybaumy> stanguturi: PsyRabbit has questions about ovf datasource
<smoser> stanguturi: thanks for joining.
<stanguturi> ybaumy: ok. will chat with PsyRabbit.
<PsyRabbit> We were talking about V-Cloud Director and how to get the metadata inside. I'm starting from scratch and read all from the documentation... The configuration itself shoulnt be a problem I just dont understand how to get the metadata into the machine like instance id. smoder and ybaumy told me you did something and provided me a link to your ovf datasource
<stanguturi> PsyRabbit: Have a VM with latest open-vm-tools installed and host it in V-Cloud Director. Apply the configuration spec, when the VM gets powered on, the open-vm-tools daemon picks up the necessary configuration from the hypervisor and hands over the content to cloud-init
<jchevalier> Hello, I'm back with my sles cloud-init image on Azure
<jchevalier> Still, I can only connect to the vm on the root user, and the script or config file given to custom-data is totally ignored
<paulmey> jchevalier: did you see my messages from yesterday?
<jchevalier> Hello paulmey, about the logs on azure?
<paulmey> jchevalier: Add this to a .cfg file in /etc/cloud/cloud.cfg.d : https://www.irccloud.com/pastebin/QwTiV5GT/
<jchevalier> I was off a few hours, and my PC was out of battery, so I didn't got the messages unfortunately
<jchevalier> Thx I'll try this now
<paulmey> on SUSE, the service is named differently, so the command to start the agent needs to be slightly modified
<paulmey> we should probably log a bug for the SUSE cloud-init package if that's how you installed it
<jchevalier> when you say "to a config file" it has to have specific name?
<jchevalier> Ok for the waagent service name
<jchevalier> (my VM is booting, I crashed the last one)
<paulmey> iirc, it needs to end in '.cfg' for cloud-init to pick it up
<paulmey> jchevalier: 1:25 PM <paulmey> iirc, it needs to end in '.cfg' for cloud-init to pick it up
<jchevalier> ok so just the .cfg is important thx
<paulmey> yep: https://git.launchpad.net/cloud-init/tree/cloudinit/util.py#n946
<jchevalier> good to know
<jchevalier> prep:/home/jchevalier # systemctl status waagent
<jchevalier> â waagent.service - Azure Linux Agent
<jchevalier>    Loaded: loaded (/usr/lib/systemd/system/waagent.service; enabled; vendor preset: disabled)
<jchevalier>    Active: active (running) since Tue 2018-02-06 21:32:39 UTC; 6min ago
<jchevalier> waagent seems to be an available service in suse
<jchevalier> is it this service you talk about paulmey?
<paulmey> jchevalier: correct, but your logs from yesterday showed that cloud-init was trying to start walinuxagent instead of waagent
<jchevalier> yes ok
<jchevalier> I'm looking for walinuxagent in config files just in case
#cloud-init 2018-02-07
<enleth> Hi there. I'm trying to write a simple network-config for a VM running on an OVH dedicated server, with a public IP for the VM. The problem is, OVH's IP assignment is kinda wonky and requires using a gateway from a completely different subnet than the machine's IP.
<enleth> This is a working network config I set manually: https://paste.q3k.org/paste/6VsfMrhe#DWJwJjp+6nlZCcd5aGvqWolB7VXzqr8FztVi2cJgHPQ
<enleth> yes, it works, yes, it uses a gateway address in a subnet different than the interface address, yes, the interface has no address in the gateway's subnet - and it must not have one
<enleth> so, how do I define a link-scope route in network-config?
<LaborejoGuest9> hi there, it there any module doesn't listed in /etc/cloud/cloud.cfg?
<LaborejoGuest9> since I didn't found document says every module belongs to which stage
<mgerdts> I submitted a merge proposal the other day and needed to make some updates.  After I made the updates, I deleted the remote branch and pushed my squashed changes to a branch of the same name.  1) was this the right way?  2) Do I need to do anything else to alert reviewers that it was updated?
<mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+ref/bug-1746605
<smoser> mgerdts: i'll review. i'm sorry, just beenbehind. always am.
<smoser> mgerdts: you can basically do 1 of 2 ways
<mgerdts> no worries.  I've made other changes aside from the commit message as well.  Those changes are not in the same files as were in the original commit.
<smoser> a.) push up original merge proposal, get feedback, add commits that say 'address feedback', rince and repeat
<smoser> rinse
<smoser> b.) push up original, get feedback, git commit --amend, git push --force
<smoser> i do both
<smoser> when we pull, we squash and use the commit message from the merge proposal
<smoser> so the "address feedback" comments get tossed.
<mgerdts> ok, thanks for clarifiying.
<smoser> mgerdts: so, in the 'Set commit message' there
<smoser> put a good git commit message
<smoser>  Summary:
<smoser>  blank
<smoser>  mer stuff
<smoser>  blank line
<smoser>  LP: #1746605
<ubot5> Launchpad bug 1746605 in cloud-init "DataSourceSmartOS needs locking and retries" [Undecided,New] https://launchpad.net/bugs/1746605
<smoser> make sense ?
<smoser> your 'Description of change' is reasonable, just format in summary and body, and < 74 chars wide.
<mgerdts> I think when I ran tox the first time around I must have done it as a non-root user because I wasn't seeing serial failures.  If it doesn't have write access to the serial port, it skips that test.  My updates this time around (aside from commit message) were to add pyserial and fix a bug with deb package creation.
<smoser> you sould not have to run tox as root
<mgerdts> I updated the commit message in the changeset.  Are you saying I was supposed to do that in LP?
<smoser> it should mock anything that needs.
<smoser> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/337106 right ?
<mgerdts> As the comments in the new test explain, that's not really possible.
<smoser> i just see one commit, with a one line message.
<mgerdts> that's stale
<mgerdts> I don't know how to tell it that I have replaced that changeset.
<mgerdts> that is, I deleted the branch then pushed again.
<mgerdts> perhaps this request needs to be closed and refiled?
<smoser> ok. hm.. lets just do that.
<smoser> http://paste.ubuntu.com/26536015/ <-- just to not lose that
<smoser> so now go https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+ref/bug-1746605 and propose for merging
<smoser> is 37ea241 your tip ?
<mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/337271
<mgerdts> 67cd602d
<mgerdts> oh wait...
<mgerdts> yeah, you got the right one
<mgerdts> I was in a different branch when I gave you the bogus response
<smoser> :)
<smoser> mgerdts: do you think you have fixed 1667735 also ?
<mgerdts> No, I have a separate fix for that just about ready.
<smoser> ok
<mgerdts> @smoser if the test isn't a good fit for unit tests, I can remove it.  Since this (and the fix for 1667735) depend on behavior that is specific to serial ports, I'm unsure of if/how any automated tests can be written.
<smoser> mgerdts: yeah... i know. you end up having to mock a lot of infrastructure
<smoser> write a "pretend mock server" sort of thing.
<mgerdts> If Linux had something like FreeBSD's nmdm (null modem) driver that could be used.  Unless I'm missing something, any attempt to mock a serial port with a pipe or socket will result in invalid tests.
<mgerdts> pipes and serial ports will create one stream (thinking of solaris, don't know the linux implementation details) that ensure there is no cross-talk.
<smoser> yeah.
<mgerdts> Also, there's no guarantee that the VFS ops related to locking with non-serial ports are representative of those used on serial ports.
<smoser> you'd just have to fabricate the crosstalk
<smoser> but yeah. i know . its a lot of pain.
<smoser> i'll ask blackboxsw to have a think also.
<mgerdts> ok.  I guess I need to come up to speed on mock so that I can maybe be a bit more imaginative in coming up with test strategies.
<enleth> I'm having a problem getting network-config to work with nocloud local ds
<enleth> https://paste.q3k.org/paste/tbFmVALJ#N1adKr6domaWXz-ayWaK7phn3aRzxoUNFkbc5I1Y0if - here's my configuration and the error I'm getting
<enleth> meta-data, user-data and network-config are packed into an iso file attached to a VM
<enleth> hostname and user are being set up correctly
<enleth> googling for the error message suggests that cloud-init detected the network-config file but for some reason parsed it as an empty (?) config
<smoser> enleth: i need a key to read that.
<enleth> smoser: the key is in the URL, it's the anchor
<enleth> needs JS to work, decrypts it in the browser
<enleth> I can re-post it to a regular pastebin
<enleth> https://pastebin.com/1jKntbhx
<enleth> smoser: ^
<enleth> if I run this without the network-config file, it just runs DHCP on the intherface as expected and everything else works too
<smoser> enleth: well, it says "could not decrypt data (Wrong key?)
<smoser> so... i dont know. i'm just using firefox and no js disabled or anything
<enleth> no matter, use the second link
<enleth> if that's of any importance, the iso file is generated like this: genisoimage  -output seed.iso -volid cidata -joliet -rock user-data meta-data network-config
<smoser> enleth: i'msorry.. pulled away. can you file a bug ?
<enleth> is there anything I can check now to actually get it over with and continue my work?
<smoser> enleth: well, i suggest trying the network config v1
<smoser> which is a bit more explicit
<enleth> smoser: how does matching by MAC address work there? what I hoped to achieve was a shared config ISO for several VMs, with interface entries for each of them matching by MAC - so that the correct one would be used to set a static IP on first boot and the rest would be ignored
<enleth> the docs for v2 explicitly say this is how it works
<enleth> but it's not so clear for v1, where the interface name is described as the primary piece of data
<enleth> maybe let me rephrase my question - if I define several interfaces in v1 config format, with names that don't match any default network adapter names assigned by kernel/udev (like, say, "ethpub1", "ethpub2", etc. instead of ethX or enoX or whatever) and define their MAC addresses, will it simply match one of those to an existing interface by MAC, rename it and ignore the rest?
<smoser> enleth: i'm sorry, not really all the way here righ tnow.  i can look some later.
<blackboxsw> per the cloud-init status bug rharper, you mentioned checking results.json presence, but that file could also exist already if we are running some non-local stage of cloud-init via cli too. so I think I'll have to check both, presence of result.json & no stage in status.json with a non-zero start and no finished time.
<rharper> blackboxsw: hrm, generally I don't think cloud-init status would react to anything after cloud-final as completed and result.json is written
<rharper> cloud-init cli invocations are not asynchronous , they're blocking
<rharper> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1747965
<ubot5> Ubuntu bug 1747965 in cloud-init (Ubuntu) "cloud-init status reports done before boot is finished" [Undecided,New]
<blackboxsw> http://paste.ubuntu.com/26536818
<blackboxsw> rharper: ^ that's the status example on ec2 where a stage that's never going to offically get run is leaked/written into status.json
<rharper> modules-init should be ignored AFAIK, smoser ?
<blackboxsw> maybe it's worth us ensuring modules-init isn't written
<blackboxsw> yeah
<rharper> that said, how is that stage related to reporting 'done' early ?
<blackboxsw> rharper: status logic used to say if any stage didn't have a start time report 'running'
<blackboxsw> now logic says if start != null and finished == null RUNNING
<blackboxsw> but we really could just simply rely on result.json existing and no incomplete (started but not finished) stages in status.json
<blackboxsw> or we could look at ps :) but this feels like overkill
<rharper> I think the result.json parsing would be best,
<rharper> err, status
<blackboxsw> not much to parse there, but ..
<blackboxsw> ahh
<blackboxsw> right status.json
<blackboxsw> also one things status didn't take into account is stage: != null means we are in flight on a given stage
<blackboxsw> but as a stage completes I believe that always gets set back to null before starting the next stage. so there are gaps where cloud-init status would mistakenly represent 'done'
<blackboxsw> so anyway, I'll work it and ping you with a branch on this
<rharper>  cool
<smoser> "ignored" ?
<smoser> yeah, we shouldnt write that to status
<smoser> only the stages that we run.
<enleth> smoser: FYI v1 config loads properly
<enleth> shame it can't do wildcard matches on the MAC
<blackboxsw> so how it looks as far as status.json. cloud-init modules --mode=init will write a start and finished time for that stage when running those 'modules-init'  modules if we run on the commandline
<enleth> or can it? the docs don't say
<blackboxsw> during normal system startup/config of cloud-init we don't explicitly run modules --mode=init
<blackboxsw> so that will always have null/null for start/finished times
<smoser> enleth: ok. let me loop at your issue again.
<blackboxsw> those modules-init modules are run during cloud-init's init(network) stage.  the cloud_config_modules are run during modules:config stage
<smoser> enleth: so https://pastebin.com/raw/1jKntbhx is actually waht you put in ?
<smoser> i dont understand "wildcard"
<blackboxsw> so in terms of status.json, the "init" section represents modules-init during typical system startup and cloud-init configuration
<enleth> smoser: not related to the issue, I just wanted to use "match: macaddress: 52:*" for another interface which is why I tried v2
<enleth> smoser: yes, the pastebin shows my entire cloud-init config
<enleth> didn't paste the user-data file but it only contains the users entry with my ssh keys and stuff
<smoser> enleth: this is fixed in trunk. i'm pretty sure.
<enleth> when I converted the config to v1, it loads and works
<enleth> smoser: you mean v2 not being loaded?
<smoser> so your v2 would work with 17.1 or 17.2 release.
<smoser> yeah.
<enleth> wonder when debian releasing publishing openstack images with cloud-init upgraded to that version
<enleth> *s/publishing//
<smoser> i'd be itnerested in knowing for sure
<smoser> you can build a deb gfrom trunk easily enough, or probably just ggrab it from
<smoser> https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/+packages
<smoser> and patch your image
<enleth> are those going to work on debian?
<smoser> i do not know that they dont.
<enleth> I was under impresion that some ubuntu packages do work with debian but it's more "by accident" than anything deliberate that can be depended upon
<smoser> enleth: i'm not suggesting you use a daily build for anything production.
<smoser> just to test if it is fixed.
<enleth> sure
<enleth> the network interface name changes need cloud-init to run on every boot, right?
<enleth> I tried creating /etc/cloud/cloud-init.disabled after the first boot so that existing VMs don't break when I do something stupid to the config in the datasource, but the interfaces are no longer getting renamed so whatever got created in /etc/network/interfaces.d/50-cloud-init.cfg no longer works at all
<smoser> enleth: hm..
<smoser> enleth: is that ifupdown ?
<smoser> cloud-init writes udev rules to make that work
<smoser> but *it* might not support the 'match' if you're using that.
<enleth> debian9, ifupdown
<enleth> nothing in /etc/udev/rules.d/
<enleth> using v1 config right now
<smoser> really.
<smoser> it really should write /etc/udev/rules.d/70-persistent-net.rules
<enleth> is there a "verbose" mode I can enable to see what it does and when?
<enleth> ah, there's quite a bit of logs in /var/log/cloud-init.log
<enleth> ok, I did a clean boot - it set the runtime interface names correctly
<enleth> /etc/udev/rules.d/ empty
<enleth> grep rules /var/log/cloud* comes up empty
<enleth> same for udev (save for a mountpoint list)
<enleth> smoser: https://pastebin.com/Y1m5PxRx this is the most interesting part I found
<enleth> other refernces to "network" in the log are accidental or just refer to the network-config file being found at the beginning
<smoser> enleth: so in newer cloud-init you'll get https://hastebin.com/kaxelapipo
<enleth> is that just a matter of an old version in debian?
<smoser> i think so yes.
<enleth> something's not right, I grepped /usr/lib/python3/dist-packages/cloudinit/ for "persistent-rules" and two hits come up
<enleth> the code is there
<enleth> yes, the code to use netrules_path and write to it is definitely there
<enleth> I'm trying to figure out why wouldn't it run
<enleth> ah, so
<enleth> distro/debian.py initializes the Rendered with 'netrules_path': None
<enleth> why would anyone do that?
<mgerdts> I thought that https://mail.python.org/pipermail/python-dev/2013-September/128287.html would mean that the development branch of cloud-init wouldn't need to support Python 2.6.  Sigh.
<enleth> eh, setting netrules_path correctly for debian works
<enleth> I'm at loss as to why it was set to None
<smoser> mgerdts: we're still supporting centos 6. which is python 2.6
<Odd_Bloke> Until November 2020, apparently.
<Odd_Bloke> Though RHEL 6 is also 2.6, and supported until 2024. :p
<enleth> smoser: OK, thanks for all the help, for now I've jury-rigged a patch to support udev rules on debian with the old version I have and switched to v1 network config, I'll upgrade whenever debian starts shipping the new version
<blackboxsw> rharper: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/337306 for cloud-init status cmdline fix
<rharper> blackboxsw: thx, will look at it
<jchevalier> paulmey: Hi, I'm sorry I'm that long to answer, I have out of sync work hours!
<jchevalier> So I tried to add the .cfg file to my vm before to make it a image
<jchevalier> but then the image never spawns
<jchevalier> the boot fails in Azure dashboard
#cloud-init 2018-02-08
<jchevalier> (I'll let my computer on non-stop to avoid any message lost)
<jchevalier> If anyone has more infos, I try to install cloud-init on SLES to create an cloud-init ready image on AZure
<amitprakash> Can someone please help me make sense of why cloud-init is failing? https://gist.github.com/291e61880cfdc6c9e1bc46b243d47760 I've verified that /bin/ip exists
<amitprakash> Modifying net/*.py with abspath to binaries allows it to find ip/resize2fs etc commands
<amitprakash> any ideas why the path is broken for cloud-init
<smoser> amitprakash: where is this ?
<smoser> what environment ? cloud-init just gets the PATH that init gives it. one would think that systemd would have a sane PATH (including /bin)
<amitprakash> smoser, custom ami for ec2
<amitprakash> gentoo
<amitprakash> brb
<jchevalier> Hello everyone
<jchevalier> I can't run a script at boot on my vm
<jchevalier> But here what I have
<jchevalier> test10:~ # cat /var/lib/cloud/instance/user-data.txt
<jchevalier> Content-Type: multipart/mixed; boundary="===============1572917255304427428=="
<jchevalier> MIME-Version: 1.0
<jchevalier> --===============1572917255304427428==
<jchevalier> Content-Type: text/x-shellscript; charset="us-ascii"
<jchevalier> MIME-Version: 1.0
<jchevalier> Content-Transfer-Encoding: 7bit
<jchevalier> Content-Disposition: attachment; filename="jchevalier.sh"
<jchevalier> #!/bin/bash
<jchevalier> touch /tmp/jchevalier
<jchevalier> --===============1572917255304427428==--
<jchevalier> test10:~ # cloud-init init
<jchevalier> Cloud-init v. 0.7.8 running 'init' at Thu, 08 Feb 2018 16:22:46 +0000. Up 335.54 seconds.
<jchevalier> ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
<jchevalier> ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
<jchevalier> ci-info: | Device |  Up  |           Address           |      Mask     | Scope |     Hw-Address    |
<jchevalier> ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
<jchevalier> ci-info: |   lo   | True |          127.0.0.1          |   255.0.0.0   |   .   |         .         |
<jchevalier> ci-info: |   lo   | True |           ::1/128           |       .       |  host |         .         |
<jchevalier> ci-info: |  eth0  | True |           10.0.0.8          | 255.255.255.0 |   .   | 00:0d:3a:1a:82:f1 |
<jchevalier> ci-info: |  eth0  | True | fe80::20d:3aff:fe1a:82f1/64 |       .       |  link | 00:0d:3a:1a:82:f1 |
<jchevalier> ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
<jchevalier> ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
<jchevalier> ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
<jchevalier> ci-info: | Route |   Destination   | Gateway  |     Genmask     | Interface | Flags |
<jchevalier> ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
<jchevalier> ci-info: |   0   |     0.0.0.0     | 10.0.0.1 |     0.0.0.0     |    eth0   |   UG  |
<jchevalier> ci-info: |   1   |     10.0.0.0    | 0.0.0.0  |  255.255.255.0  |    eth0   |   U   |
<jchevalier> ci-info: |   2   |  168.63.129.16  | 10.0.0.1 | 255.255.255.255 |    eth0   |  UGH  |
<jchevalier> ci-info: |   3   | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 |    eth0   |  UGH  |
<jchevalier> ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
<jchevalier> test10:~ # Le script seemed to be loaded, and cloud-init does not raise an error, but the file is not created...
<jchevalier> Any idea?
<dpb1> jchevalier: use a pastebin next time. :)
<jchevalier> Right my bad :)
<smoser> jchevalier: can you post /run/cloud-init/status.json ?
<smoser> i assume that one of the stages has not run.
<jchevalier> https://paste.ubuntu.com/26542228/
<jchevalier> hmm indeed
<blackboxsw> smoser: did you already write an sru test case for one of the SRU features we are releasing? maas oauth maybe of OVF related. I thought I recalled something
<blackboxsw> but I can't find the link
<smoser> blackboxsw: i did the grub-legacy-ec2 one
<smoser>  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1581416
<ubot5> Ubuntu bug 1581416 in cloud-init (Ubuntu Xenial) "grub-legacy-ec2 installs files in /etc/kernel/kernel/post(inst|rm).d" [Low,Fix committed]
<blackboxsw> ahh thank you
<smoser> i will test that now
 * blackboxsw will ping the authors on the SRU related bugs
<blackboxsw> and pushing up my sru bug verification docs ubuntu-sru.
<blackboxsw> think I'm missing a couple still
<blackboxsw> almost there
<jchevalier> Seems that the setup_disk module crashed the thing
<jchevalier> but now I have to manually run cloud-init modules -m final
<jchevalier> Otherwize the script in custom-data in not tun
<jchevalier> run*
<smoser> jchevalier: /var/log/cloud-init.log ?
<jchevalier> https://paste.ubuntu.com/26542654/
<smoser> jchevalier: what made you thnk setup_disk failed ?
<blackboxsw> smoser: rharper found an SRU-blocker the identity doc stuff we landed  in ec2 doesn't work on upgrade path. self.identity doesn't exist on the pickled datasource
#cloud-init 2018-02-09
 * blackboxsw needs to add an integration test for restarting cloudinit.service on upgrade path
<rharper> blackboxsw: bummer, we all missed it: https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/329657  in the review +1 on integration test
<blackboxsw> yeah I know, I'll add an upgrade cli invocation integration test with the fix for this rharper
<blackboxsw> so we don't get hit by any other upgrade obj.pkl isms on new class attrs
<blackboxsw> was thinking about the following http://paste.ubuntu.com/26544332/ will test it out
<smoser> blackboxsw: thanks for finding :-(
<smoser> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1735331
<ubot5> Ubuntu bug 1735331 in cloud-init "ec2: zesty tempfile sandbox dhclient.pid file can't be created" [High,Fix released]
<smoser> a comment there. i asked for logs.
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1748354
<ubot5> Ubuntu bug 1748354 in cloud-init (Ubuntu) "ec2: systemctl restart cloud-init after upgrade 17.1.46 -> 17.2.30" [High,Triaged]
<amitprakash> Can someone please help me make sense of why cloud-init is failing? https://gist.github.com/291e61880cfdc6c9e1bc46b243d47760 I've verified that /bin/ip exists. Modifying net/*.py with abspath to binaries allows it to find ip/resize2fs etc commands
<amitprakash> any ideas why the path is broken for cloud-init
<kutasek> hello, I am trying to set a hostname for an AWS Ami on reboot with the instance id in the name. Is it possible to substiture some variable in the hostname param?
<kutasek> e.g I wish to be able to do something like: "hostname: abc-dev-$INSTANCE_ID.domain.com"
<jchevalier> HEllo, modules-config and modules-final won't start, but works fine manually
<jchevalier> How can I debug this? Is there a configuration somewhere saying which steps must run?
<PsyRabbit> s as-admin-p2.central.infra.maz
<PsyRabbit> Sorry, wrong window :/
<blackboxsw> kutasek: watch for this branch to land https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335290   you can then provide  something like this https://pastebin.ubuntu.com/26546614/
<kutasek> blackboxsw: I can put that template directly in the user-data of my aws instance?
<blackboxsw> correct kutasek once I get that branch touched up and landed (it would make it into bionic images until we SRU it)
<blackboxsw> SRU (stable release update into xenial and artful)
<blackboxsw> yeah you'd just have to provied the ## template: jinja2 header above the #cloud-config you provide anyway
<blackboxsw> sorry ## template: jinja   I mean
<kutasek> blackboxsw: Thanks for the info, one more question will it support aws meta-data variable out of the box?
<blackboxsw> kutasek: it  pulls anything from /run/cloud-init/instance-data.json ... leeme get you an ec2 example
<blackboxsw> https://paste.ubuntu.com/26368359/
<kutasek> great, and about that v1. prefix, will there be more ? is it linked to the aws meta-data api date they use for versioning?
<blackboxsw> v1 is our stadardized/generalized fields across all clouds. we will be adding more
<blackboxsw> the rest of the top level keys are cloud specific
<kutasek> awesome thanks a lot, can't wait for it, gonna have to use bash in meantime :D
<blackboxsw> kutasek: yeah I bet with a runcmd section containing   - cat /run/cloud-init/instance-data.json | jq -r '.v1.instance-id'   you might get pretty far
<kutasek> blackboxsw: I am not able to change the hostname with runcmd for some reasons (need to update it on each boot)
<smoser> mgerdts: around ?
<mgerdts> yep
<smoser> http://paste.ubuntu.com/26547352/
<smoser> thatfixes your 2.6 issues
<mgerdts> great, thanks!
<blackboxsw> smoser: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/337475
<blackboxsw> i;m afk a while so if it needs desc commit message work to land  can't get to it for an hour pr so
<smoser> i still dont know what we do about python2
<smoser> blackboxsw: looking to fix. fixed the flake, but the test is failing for me.
<blackboxsw> just saw my latest change busted the tst
<blackboxsw> pushed
<blackboxsw> back in 1-
<blackboxsw> back in 10
<smoser> oh. ok.
<smoser> blackboxsw: still no :-(
<smoser> you needc http://paste.ubuntu.com/26547563/ for flake
<smoser> but i still get the test failure.
<smoser> http://paste.ubuntu.com/26547572/
<blackboxsw> ok looking for real
<smoser> its not getting registered correctly
<blackboxsw> was a complex dict registration on httppretty
<blackboxsw> fixing now to handle children
<blackboxsw> I was originally testing with just the specific instanceId mocked and tried changing the test to mock the whole nested instance-identity document
<blackboxsw> without all the nested children routes
<blackboxsw> fixing that now
<smoser> hm.
<blackboxsw> the unit tests right now fall over when trying to materialize dynamic/instance-identity/document url
<blackboxsw> which isn't registered via httpretty properly
<smoser> i'm missing nit.
<blackboxsw> ohh the problem is  in our registration_helper we can't actually return a dictionary if provided, we actually register separate URIs for each key-value
<blackboxsw> ok
<smoser> blackboxsw: http://paste.ubuntu.com/=nNgvNtJJCD/
<smoser> i'm guessing you have that now though
<smoser> blackboxsw: i gotta walk away for 30-45 minutes. i'll return though and will ack your MP assuming c-i does.
<smoser> my only comment is that the @property is tempting (for self.identity)
<smoser> blackboxsw: you there ?
<smoser> did you open a bug for that ?
<blackboxsw> smoser: yep, pushed changes and finding the bug
<blackboxsw> I'm sparsely here, need to mark a sick day today
<smoser> blackboxsw: i merged
<blackboxsw> ok thanks was just writing the commit message over with real content and the https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1748354
<ubot5> Ubuntu bug 1748354 in cloud-init (Ubuntu) "ec2: systemctl restart cloud-init after upgrade 17.1.46 -> 17.2.30" [High,In progress]
<blackboxsw> ok thanks
<blackboxsw> a lot
<smoser> and i'm going to push overwrite
<smoser> i can upload to bionic
<blackboxsw> thanks, pretty much absent today.
<smoser> on its way to bionic.
#cloud-init 2020-02-03
<blackboxsw> finally merged PR #162 sysconfig suse/rhel flavors Thanks robjo
<robjo> Thanks
<meena> now what?
<meena> how do we undo that hack? ð
<wyoung> git reset HEAD~1
<blackboxsw> heh
<wyoung> oops, I forgot --hard
#cloud-init 2020-02-04
<_NiC> If I have an existing user 'debian' in my image, and in /etc/cloud/cloud.cfg I have system_info: -> default_user: -> name: debian, will the settings from cloud.cfg not be applied because the user already exists? Because the settings there are not applied. :-\ (debian 9, cloud-init 0.7.9-2)
<_NiC> oh, perhaps it's skipped because I'm specifying another user in my user-data, and not also explicitly telling it to create the default user as well?
<meena> _NiC: perhaps! is the user from user-data created?
<_NiC> meena, yes. and I didn't have "- default" listed under users:
<_NiC> meena, so from what I could gather, that meant to *not* care about the default_user as defined in cloud.cfg. So when I added "- default" it worked as expected again. :)
<meena> ð
<_NiC> When running the openstack create command, you get sent back an 'adminPass' though, and I tried to figure out if that could be injected somehow, but not easily it seems, so I ignored that as well. I have my custom user, so it's all good. :)
<brandor5> hello everyone: we've got a pipeline that builds custom cloud images for openstack... I've added centos8 support and I'm having trouble with adduser for our custom user... on all the other builds stamping in an updated cloud.cfg that replaces the centos user with our user works ok, but with centos8 it doesnt... that user never gets added and our pipeline fails... any ideas on whats going on?
<hetii> Hello
<hetii> I have a vmware instance where I spawn by terraform rancheros. The issue is that in this environment DHCP is disable so I need to provide network configuration. The question is should I do this by metadata or by cloud-config? What happens if I have defined in both? with one is taken?
<onklmaps> Hi :) I want to deploy some cloud servers at Hetzner. I plan to run 3-4 PrestaShops on them. I think nginx+php-fpm is the best and then a separate mysql server. So my question: does anyone have a good cloud-init-file for nginx + php-fpm that would be suitable for running prestashop? PHP 7.2 is needed
<_NiC> onklmaps, my knowledge of cloud-init is pretty limited, but I suspect you don't want to try to do *all* of your config using cloud-init
<_NiC> onklmaps, rather use cloud-init to get a good base, and use something else to configure the rest. (such as ansible, puppet, chef, or similar)
<onklmaps> Ahh. That makes sense.. You know any good base for nginx + php-fpm?
<_NiC> I'm not familiar with prestashop either, but unless its requirements are very specific, any standard setup should work
<_NiC> if you need high performance etc etc you should probably look into various forms of caching, but that's a topic for another channel I think.. :-)
<onklmaps> It's pretty standard, yes..I have some custom things that needs to go into the server block, but other than that its pretty standard
<onklmaps> Yeah - caching... I know.. It's a big subject
<onklmaps> I just installed nginx as reverse proxy in front however. I might do static cache there?
<_NiC> that's possible
<onklmaps> PrestaShop is much easier to set up with apache.. I might just serve the prestashops using apache + php-fpm, then use nginx reverse proxy with static cache in front
<onklmaps> What you think?
<_NiC> Not sure I have an opinion about that :-)
<wyoung> onklmaps: I think php is a bad idea.
<blackboxsw> hey folks. let's do this....
<blackboxsw> #startmeeting Cloud-init bi-weekly status
<meetingology> Meeting started Tue Feb  4 17:23:28 2020 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> morning, afternoon and evening folks. Time for another cloud-init community status meeting
<blackboxsw> #chair rharper
<meetingology> Current chairs: blackboxsw rharper
<blackboxsw> #chair Odd_Bloke
<meetingology> Current chairs: Odd_Bloke blackboxsw rharper
<blackboxsw> Coud-init upstream uses this meeting as a platform for community updates, feature/bug discussions, and an opportunity to get some extra input on current development.
<blackboxsw> The next scheduled status meeting is always listed in the topic of this channel, so feel free to drop in on next session if you miss this one
<blackboxsw> while we're at it I'll update for next status meeting.
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting February 16 17:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> 2 weeks from today, same bat time, same bat channel
<blackboxsw> Our previous meeting minutes line here:
<blackboxsw> #link https://cloud-init.github.io/
<blackboxsw> *live here* rather
<blackboxsw> the topics we cover in this meeting are the following: Previous Actions, Recent Changes, In-progress Development, Community Charter, Upcoming Meetings, Office Hours (~30 mins).
<blackboxsw> new topics or intejections are always welcome
<blackboxsw> #topic Previous Actions
<blackboxsw> From last meeting we had no unresolved actions so we can jump to the next section
<blackboxsw> #topic Recent Changes
<blackboxsw> found from tip of master with    `git log --since 01/21/2020`
<blackboxsw>     - sysconfig: distro-specific config rendering for BOOTPROTO option (#162)
<blackboxsw>       [Robert Schweikert] (LP: #1800854)
<blackboxsw>     - cloudinit: replace "from six import X" imports (except in util.py) (#183)
<blackboxsw>     - run-container: use 'test -n' instead of 'test ! -z' (#202)
<blackboxsw>       [Paride Legovini]
<blackboxsw>     - net/cmdline: correctly handle static ip= config (#201)
<blackboxsw>       [Dimitri John Ledkov] (LP: #1861412)
<ubot5> Launchpad bug 1800854 in cloud-init "BOTOPROTO handling between RHEL/Centos/Fedora and SUSE distros is different" [Medium,Triaged] https://launchpad.net/bugs/1800854
<blackboxsw>     - Replace mock library with unittest.mock (#186)
<blackboxsw>     - HACKING.rst: update CLA link (#199)
<ubot5> Launchpad bug 1861412 in cloud-init (Ubuntu) "cloud-init crashes with static network configuration" [Undecided,Fix committed] https://launchpad.net/bugs/1861412
<blackboxsw>     - Scaleway: Fix DatasourceScaleway to avoid backtrace (#128)
<blackboxsw>       [Louis Bouchard]
<blackboxsw>     - cloudinit/cmd/devel/net_convert.py: add missing space (#191)
<blackboxsw>     - tools/run-container: drop support for python2 (#192) [Paride Legovini]
<blackboxsw>     - Print ssh key fingerprints using sha256 hash (#188) (LP: #1860789)
<blackboxsw>     - Make the RPM build use Python 3 (#190) [Paride Legovini]
<ubot5> Launchpad bug 1860789 in cloud-init (Ubuntu) "ssh_authkey_fingerprints must use sha256 not md5" [Undecided,Fix committed] https://launchpad.net/bugs/1860789
<powersj> thought we were going to use pastebin :P
<blackboxsw> heh, that is a good point (I wondered if anyone would call me on that)
<blackboxsw> #link https://paste.ubuntu.com/p/3jQdKZVPcM/
<blackboxsw> generally speaking, dropping use of six since our code based is not python3-only, tooling dropping py2,  sysconfig rendering flavors for opensuse, doc fixes and read the docs fixups
<blackboxsw> thanks all for the contributions over the last couple weeks
<blackboxsw> #topic  In-progress Development,
<blackboxsw> #topic  In-progress Development
<blackboxsw> Any existing PRs are up for review at the following url:
<blackboxsw> #link https://github.com/canonical/cloud-init/pulls
<blackboxsw> generally speaking we are in the 'long tail' part of a couple of feature-sets:
<blackboxsw> * we are trying to wrap up tooling for our automated CI,  publishing processes and documentation for the shift to github from launchpad
<blackboxsw> * we are in progress on cloud-init handling network hotplug  for a couple of datasources
<blackboxsw> * in progress on boot speed improvements for various platforms
<blackboxsw> We also recently validated and released cloud-init v 19.4.33 to Xenial, Bionic and Eoan  (1/9/2020)
<ahosmanMSFT> Hi @blackboxsw I'm no longer in the provisioning team, but there's an urgency for the cloud test to be resilient. Have you looked at those issues, I can dedicate as much time as needed to this. If you have time, can we tackle this today?
<blackboxsw> there are also a number of PRs in flight for FreeBSD,NetBSD, OpenSUSE and CentOS that need attention so we can better enable those distros
<ahosmanMSFT> azurecloudtest that is
<blackboxsw> hi ahosmanMSFT I can spend some time on office hours here to peek more at it. my individual runs didn't hit the timeouts again, so we might need a reproducer cmdline from you in a new bug maybe?
<ahosmanMSFT> So your able to run all tests successfully without timeout and image not building>
<blackboxsw> ahosmanMSFT: but yes I can spend a little time on this today. and I think ultimately we'll have to find the tox command line that exhibits this error. I'll go checkout my test run again and see. I don't think I saw the failure. but I might be invoking tests differently than you
<ahosmanMSFT> blackboxsw: hmm that's interesting, thanks let me know
<blackboxsw> same here ahosmanMSFT, can you file a bug with the traceback you see and the tox cmdline you are running?
<blackboxsw> then I know exactly what to look for
<ahosmanMSFT> Sure, will do now
<blackboxsw> cool.
<blackboxsw> ok next topic
<blackboxsw> #topic Community Charter
<blackboxsw> ok this section is reserved to raise general community work/goals.
<blackboxsw> At last cloud-init summit we raised a couple of general themes of improvements cloud-init would like to achieve
<blackboxsw> These themes fell into two categories for this year: datasource documentation updates and cloud-init json schema validation for the 50+ config modules in cloudinit/config/cc_*py  so that we can better raise user-config errors and remove some of cloud-init's "sharp edges"
<blackboxsw> we converted a number of these feature requests in into bugs which can be searched here:
<blackboxsw> #link https://bugs.launchpad.net/cloud-init/+bugs?field.tag=bitesize
<blackboxsw> tasks in this list should be fairly easy one-time bugs for folks with a little time available to help improve cloud-init.
<blackboxsw> we'll revisit this set of bugs/features and the community charter goals near the end of 2020 at the next cloud-init summit
<blackboxsw> #topic Office Hours (next ~30 mins)
<blackboxsw> this time is spent with cloud-init upstream dev eyes on this channel for any cloud-init feature, bug  or implementation discussions. In the absence of such discussions, we'll review the active PRs to try to tidy up the review queue and unblock developers
<blackboxsw> for the moment, I'll look over some Azure test timeouts ahosmanMSFT is seeing
<blackboxsw> any other topics, concerns, bugs, questions are welcome and someone should be around to field them
<blackboxsw> ahosmanMSFT: so timeouts running integration tests, you said you are getting them about half the time?
<ahosmanMSFT> blackboxsw: Yes, I tracked it down to platforms/instance._wait_for_system
<ahosmanMSFT> I invoke it after initializing vm in platform/azurecloudtest/instance.start
<ahosmanMSFT> when removed, everything works as expected
<ahosmanMSFT> looks like it's needed for cloud tests so thought I'd leave it to you, since I don't know how ec2/lxd/... rely on
<powersj> ahosmanMSFT, can you file a bug please with the cli example?
<powersj> that would help us triage and make a proper decision on what change to make
<ahosmanMSFT> powersj, yes, was in the middle of that side tracked by meeting. On it now
<powersj> thanks!
<blackboxsw> aaaand, I should probably wrap the meeting for the day.
<blackboxsw> Thanks all for the time and energy you put into improving cloud-init! See you next time, or anytime in between
<blackboxsw> #endmeeting
<meetingology> Meeting ended Tue Feb  4 19:08:14 2020 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-02-04-17.23.moin.txt
<ahosmanMSFT> blackboxsw, powersj: submitted the bug https://bugs.launchpad.net/cloud-init/+bug/1861921
<ubot5> Ubuntu bug 1861921 in cloud-init "cloud tests ssh time out" [Undecided,New]
<powersj> ahosmanMSFT, can you give us the full CLI you are using to launch?
<ahosmanMSFT> sure
<ahosmanMSFT> powersj: added full log to the bug
<powersj> ahosmanMSFT, perfect thanks
<ahosmanMSFT> powersj: I found it most reproducible when going Idle
<ahosmanMSFT> for some reason that repro's it for me, for example launching the tests then checking on something else outside the vm where I am running it
<powersj> ahosmanMSFT, "reproduce this is by going idle." what do you mean by that?
<meena> i missed another one!
<meena> this time my excuse is that I'm visiting family in Ireland
<rharper> meena: =)
<ahosmanMSFT> powersj, I run cloud tests on a vm, so when not on the vm it's self , ie use the browser. It typically hit's that time out error. This doesn't occur though when removing that wait function.
<rharper> hrm, do you think the VM itself is suspending ?
<rharper> the wait has a timeout which is likely what it's hitting
<rharper> if you remove the timeout, then when the VM resumes, it just continues executing
<rharper> when you resume, the VM time will catch up, that will trip timers , ie, if you've suspeneded for longer than the 300 seconds, then when it resumes the timer expires immediately
<rharper> ahosmanMSFT: can you confirm that if you stay active in the VM that it does not fail ?
<rharper> alternatively, launch an Ubuntu instance in Azure; and run cloud-tests from within the Azure instances (instead of a local VM)
<ahosmanMSFT> This was on a ubuntu VM in Azure, rharper
<ahosmanMSFT> but, yes, it's less likely to happen when staying on VM
<rharper> quite strange
<rharper> it still smells like some sort of suspend
<rharper> you could spawn some other cpu bound task  in the background  ?
<blackboxsw> meena: you have to disavow all family  and friends like the rest of us... that's the only way this is going to work ;)
<blackboxsw> yeah ahosmanMSFT my full ci tests against azure are still running and I'm not seeing any of those failures yet. but I'm running from an Ubuntu lxc on Ubuntu host OS. If memory serves you may be running on an Ubuntu lxc running on an OSX host system?   How the 'vms' behave on OSX when focus shifts to other services (like browsers etc) could likely be the culprit
<blackboxsw> and I'd follow rharper's suggestion of "alternatively, launch an Ubuntu instance in Azure; and run cloud-tests from within the Azure instances (instead of a local VM)"
<rharper> blackboxsw: I think ahosmanMSFT is doing that
<blackboxsw> oops just read that
<rharper> blackboxsw: I suppose we could try that
<rharper> but still not sure what "going idle" means from that perspective
<blackboxsw> sure I can kick off an instance to do that. I was just thrown by the "browser" comment
<rharper> running from within a screen/tmux or something else
<rharper> blackboxsw: as was I
<ahosmanMSFT> it's more random than anything
<ahosmanMSFT> ssh timeout
<rharper> so I suspect there's still something else tooling-wise were missing
<rharper> ahosmanMSFT: sure, but there are really only two causes here;  1) bad connection between the cloud_test client and target instance  2) the instance is taking longer than the timeout to boot up to ssh
<rharper> (1) and (2) can be intermittent
<ahosmanMSFT> I agree, but the second is usually not the case. It usually never goes pass the boot time out of 300s
<blackboxsw> just looking over those pastes more closely, I *am* running basically the same command as ahosmanMSFT : .tox/citest/bin/python -m tests.cloud_tests run --verbose   --os-name bionic  --data-dir results --preserve-data --platform azurecloud
<rharper> ahosmanMSFT: do we not have the console-log configured ?
<ahosmanMSFT> rharpher: can you elaborate
<rharper> serial_console_log_blob_uri
<ahosmanMSFT> that should be created
<rharper> looks like we enable the serial console, so cloud_tests should capture the serial-console log on failure
<rharper> I'd like to see the serial console log of the failing instance
<rharper> and comparing the failing serial-console log to the passing one likely will reveal some differences which may explain what's going on
<blackboxsw> I'm also a bit surprised to see  a 3:45 min gap between the following logs
<blackboxsw> 2020-02-04 19:11:03,628 - tests.cloud_tests - DEBUG - image not found, launching instance with base image
<blackboxsw> 2020-02-04 19:14:44,950 - tests.cloud_tests - DEBUG - executing "wait for instance start"
<ahosmanMSFT> That is stored in a storage account in your resource group when running, but the resource group is deleted after the cloud-tests
<blackboxsw> in my runs, typically that gap should only be around the time to launch and get a response back from Azure API. which is about 30 seconds on 'az vm create'
<rharper> ahosmanMSFT: ah you mentioned that bug;  we should dump the value before tearing down
<rharper> blackboxsw: the compute_client.images.get() may be taking a long time
<ahosmanMSFT> preserve should keep it, but it doesn't working on fixing that too
<blackboxsw> ahosmanMSFT: I think --preserve-instance and --preserve-data should keep that around post failure
<rharper> depending on the REST query; I've seen *long* REST calls to list of images
<blackboxsw> hrm :/
<rharper> though 3 minutes seems excessive
<ahosmanMSFT> Out of three of these bugs, the current biggest flaw isn't even the timeout, it's that images aren't created properly due to azurecloud/image._img_instance staying None despite re-assignment, but this is more on my side, but can use your help. As the ssh timeout isn't 100%, this issue is.
<ahosmanMSFT> the preserve call should be a simple fix
<ahosmanMSFT> ssh timeout I think is more machine/system dependent now that I'm seeing your not getting it
<rharper> ahasenack: the default timeout is 300, it would be a terrible thing to bump that on azure platform to say 600 and see if that runs more reliabling to completion
<ahasenack> ahosmanMSFT: ^
<ahasenack> :)
<rharper> ahasenack: sorry
<ahasenack> np :)
<rharper> lack of extra tab
<ahosmanMSFT> rharper: I agree, I don't think timeout is the issue
<ahosmanMSFT> default timeout that is
<ahosmanMSFT> rharper, blackboxsw: Are you still in today?
<rharper> yep
<ahosmanMSFT> Mind taking a look at azurecloud/image._img_instance
<ahosmanMSFT> it's not re-assigning correctly, which means images don't build correctly
<ahosmanMSFT> it works on ec2, right?
<ahosmanMSFT> I placed some print statements and it stays instantiated as None
<rharper> ahosmanMSFT: sure, do you have more details on "images aren't created properly due to azurecloud/image._img_instance staying None";
<ahosmanMSFT> rharper: yes...
<ahosmanMSFT> so when cloud-tests starts it hit's image.py checks if there is an image and if there isn't it creates one via streams parameters. Then in the next test it checks, is there a clean image, in our case the variable that specifies this never changes and always stays None for some reason
<ahosmanMSFT> because of that, it always launches a base image, not a clean image
<ahosmanMSFT> which negates the whole image process
<ahosmanMSFT> of creating a clean image for all future tests
<rharper> ahosmanMSFT: are you specifying a cloud-init.deb or is one getting built ?
<ahosmanMSFT> One get's built, looking at the code, seems to be abstracted
<ahosmanMSFT> The fault comes when _img_instance is used in snapshot, it never passes the initial conditional
<ahosmanMSFT> https://paste.ubuntu.com/p/nwgm8KjXJk/
<ahosmanMSFT> rharper
<rharper> ahosmanMSFT: just looking at some differences, in azurecloud/image.py:_instance(),   you call platform.create_images(); but you don't start it;
<ahosmanMSFT> rharper: hmm I changed it and added some logging and still get that it's None
<ahosmanMSFT> https://paste.ubuntu.com/p/VPgqXZXkkX/
<rharper> ahosmanMSFT: so, if it's not started, then when snapshot gets called there's not self._img_instance set and it returns the AzureCloudSnapshot() rather than doing the boot clean script and such
<ahosmanMSFT> rharper: running again, I'll show you the logs and all the logging I added when trying to debug
<ahosmanMSFT> I modified _instance(), as shown above
<rharper> right, so the call path goes from platform.get_image() -> AzureCloudImage() -> platform.get_snapshot() -> snapshot.launch();  where you create an instance, but don't start it either ()
<ahosmanMSFT> rharper: that might be it, I didn't spot that
<ahosmanMSFT> I'll make the change and run it again, keep you update
<rharper> yeah, I'm just walking through cloud_tests/platforms/__init__.py  cloud_tests/bddeb.py:setup_build() and comparing platform/ec2/* to platform/azurecloud/*
<ahosmanMSFT> are you running this in a container
<rharper> no, I'm just reading the code
<ahosmanMSFT> I mean where the tests are run
<rharper> no, just under tox on an jenkins node;
#cloud-init 2020-02-05
<rharper> ahosmanMSFT: ok, I'm making some progress on your issue; so should have an update to the bug later today;
<ahosmanMSFT> rharper: Thanks, is this the ssh issue, I'm still looking into _img_instance
<rharper> ahosmanMSFT: I think they are related;  in that, what I saw was that during "run" we don't make a custom image, but only launch the base image as the setup_image stage doesn't have anything to do unless you wan to upgrade/install-deb etc to the base image
<rharper> and the name of the VM if it's the same on subsequent runs, the image.py _instance() method was not starting the image; which lead to ssh timeouts
<rharper> I could observe the VM in the portal in Stopped state while the tests were attempting to ssh into it
<ahosmanMSFT> rharper: Let me know if you need anything from my side, I'll do some tests later today and let you know what I find.
<rharper> ahosmanMSFT: ok
#cloud-init 2020-02-06
<rharper> ahosmanMSFT: I've run out of time for today; but in general one issue that needs resolving is that the instance.py and image.py both use 'image_id' as a way to reference different things;  that is the instance uses image_id where 'vmName' really ought to be used, but certain places in image.py only have a image_id reference, so when snapshot is run, it needs the image id, but generaize and deallocate etc need the vm_name;  so I think some refactoring
<rharper> of this needs to happen.  I'll keep working on this tomorrow as well.
<ahosmanMSFT> rharper: I agree, at first my ideas was having them be interchangeable but seeing how itâs impeding in functionality it needs to be changed.
<Xat`> hello guys
<Xat`> I have a per-instance script installed on 2 servers which are part of the same AWS ASG
<Xat`> when instances first start, the script is run on each instance . But I strictly need to avoid the simultaneity of the execution . Any idea ? I could use a random sleep, but it seems crappy
<malasfar> Hello .. are there any documented steps on how to prepare vSphere templates with cloud-init . we are currently having a issues getting cloud-init to work ?  is this the right channel for such requests?
<rharper> malasfar: I'm not aware of any documentation for the templates w.r.t cloud-init upstream;  I suspect there are some vmware knowledge base articles
<malasfar> any help on this appreciated as there are a lot of misleading information out there including the VMware KBs, i have been trying to get this to work since early 2019 and the results has been always inconsistent depending on what type of IP assignment I use ( Static , DHCP ) . it would be great if someone can jump on a webex/zoom session with me .
<malasfar> we just need some insight from the cloud-init team.
<rharper> Xat`:  you need some coordination between two servers so they don't execute the script at the same time?  can the central resource queue/block while ?
<rharper> malasfar: to be frank, the vmware datasource with network config is a mess;  there are conflicting modes of operation
<rharper> certainly happy to help debug/diagnose; but we've been working with the vmware team to help sort out how to enable all of the various workflows that are available;
<malasfar> i don't disagree .. we are trying to get high visibility on this internally with the BU.. i m a cloud management Staff SE @ vmware
<malasfar> that's great to hear . anyone in particular i can reach out to internally on our side. we still dont have a solid way to get this to work and specially for our customers . the only way i personal got it to work if i only used DHCP. and its a hit and mess
<rharper> malasfar: on the technical side, if you look through cloud-init git commit history for @vmware.com those are the folks making changes
<malasfar> the issue here we provision a machine from vRealize Automation to vSphere . we dont have such issues we provision things to AWS, Azure or GCP
<malasfar> alright thanks , i will check the git commit history. just want to say that this a huge issue for us at the moment affecting al lot of customers world wide
<malasfar> it would be great if cloud-init actually worked on vSphere
<malasfar> thank you for your time
<rharper> malasfar: indeed;  sure, stay in touch
<amansi26> Hi.. I need to change the hostname to the global domain name, right now it is getting assigned to the name to which I deployed the VM. I am using RHEL distro. wherecan I modify that??
<amansi26> Hi.. I need to change the hostname to the global domain name, right now it is getting assigned to the name to which I deployed the VM. I am using RHEL distro. where can I modify that??
<minfrin> Hey hey. Quick question on cloud-init and swap. Can cloud-init perform the mkswap and swapon steps for a swap partition for us? Struggling to find an example, and my attempts see everything configured (/etc/fstab is correct) but no mkswap or swapon.
<blackboxsw> amansi26: you can provide #cloud-config  user-data to the vm at launch time to do this and set the hostname and/or fqdn fields https://cloudinit.readthedocs.io/en/latest/topics/modules.html#update-hostname
<blackboxsw> amansi26: if  your cloud may provide dynamic hostname and fqdn information from the metadata service, you can check `sudo cloud-init query` to see if any hostname or fqdn values are what you are looking for for the vm, then you could provide a jinja template to source those query parameters from your metadata if needed: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data
<blackboxsw> sorry about the run-on sentence
<amansi26> blackboxsw: I just observed, my resolv.conf is missing with dns server details, that is why failing to set the correct hostname. I just put the dns details and rebooted, hostname comes up. Where is this resolv.conf being handled?
<blackboxsw> amansi26: something like this maybe https://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-resolv-conf
<blackboxsw> or https://cloudinit.readthedocs.io/en/latest/topics/modules.html#resolv-conf
<blackboxsw> minfrin: is this what you are looking for https://cloudinit.readthedocs.io/en/latest/topics/examples.html#adjust-mount-points-mounted?
<blackboxsw> minfrin: from https://cloudinit.readthedocs.io/en/latest/topics/modules.html#mounts
<rharper> blackboxsw: https://github.com/canonical/cloud-init/pull/205   any idea why this shows CI not complete but if I click through, it shows passed ?
<rharper> ahosmanMSFT: https://github.com/canonical/cloud-init/pull/205
<Odd_Bloke> rharper: Travis was likely having a normal one.  I've tried restarting the fastest of the sub-builds to see if that causes it to attempt to report again.
<rharper> Odd_Bloke: hehe, thanks for checking
<Odd_Bloke> If that doesn't do it, then retrying the whole build is the next step, and that usually works.
<rharper> ok
<crimson_king> Can cloud-init be used to override the SSH port configured at /etc/ssh/sshd_config? Because I'm trying to change the SSH server port in my VPS and it's not working (it's always port 22). After talking about this on #ubuntu, they told me it could be cloud-init's fault, since it's being used by my hosting provider to setup the image.
<rharper> crimson_king: hrm, lemme check if we support that config option
<rharper> crimson_king: no, we don't have that in our ssh config namespace;  you could use bootcmd, which runs very early to make the change with some other tool ( sed, or awk, etc)
<crimson_king> rharper, I don't see bootcmd installed, so there's something else overriding the SSH port. Do you have any idea what could it be?
<crimson_king> If it's not cloud-init...
<crimson_king> Something is locking the SSH port to 22, and doesn't let me change it.
<crimson_king> I've talked to the hosting provider support, they couldn't help. I've talked to people at #ubuntu, and they told me to see if cloud-init is doing it
<rharper> crimson_king: sorry, bootcmd is a cloud-init config module,
<rharper> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#bootcmd
<rharper> would execute commands early in boot
<crimson_king> oh..
<rharper> the other possibility is provider firewalls
<rharper> sometimes they block ports by default and you need to enable 22 through their apis
<blackboxsw> rharper: I had to re-run an ubuntu-advantage-client CI job in travis to get it to re-report success output to the PR
<rharper> blackboxsw: ah, ok Odd_Bloke was saying the same thing
<rharper> looks like it's green now
<blackboxsw> yeah something hiccupped today on the reporting back, the restart of the full job was required on my end
<rharper> y
<blackboxsw> crimson_king: if you think something in your cloud provider was providing user-data or meta-data that ran bootcmds on the system, you could run "sudo cloud-init query --all" on the vm and see if anything looked like it was providing vendor data or user-data that specified runcmd or bootcmd
<blackboxsw> though as rharper mentioned you could provide your own overrides via #cloud-config user-data which could 'fix' your vms to the proper sshd_config
<crimson_king> blackboxsw, I get a warning and the following error: ERROR: Missing instance-data file: /run/cloud-init/instance-data.json
<rharper> maybe it didn't run
<blackboxsw> cloud-init --version?
<crimson_king> blackboxsw, /usr/bin/cloud-init 19.4-33-gbb4131a2-0ubuntu1~18.04.1
<blackboxsw> yeah cloud-init may not have run as rharper mentioned 'cloud-init status --long'
<crimson_king> blackboxsw, Status disabled, cloud-init disabled by cloud-init-generator
<blackboxsw> yeah something in the systemd dependency chain on that image may be broken and cloud-init was pulled out of the boot goals. right rharper?
<rharper> blackboxsw: yes, but it wouldn't be disabled
<rharper> crimson_king: that seems like there wasn't any data provided for cloud-init to use,
<rharper> in which case cloud-init won't run if it doesn't detect a datasource
<crimson_king> rharper, So cloud-init is not doing anything here?
<blackboxsw> and that status is due to seeing the file /etc/cloud/cloud-init.disabled on the system
<rharper> crimson_king: doesn't look like it,  /run/cloud-init/ds-identify.log will have details
<blackboxsw> cat /run/cloud-init/ds-identify.log can tell you if cloud-init didn't detect any valid datasource
<malasfar> Hi .. i m trying to trouble shoot an issue when provisioning an ubuntu 16.04 on sphere where i dont have cloud-init logs to check since they simply dont get generated. if I run cloud-init status --long i get the following -- > Can not apply stage config, no datasource found! Likely bad things to come!  - Can not apply stage final, no datasource
<malasfar> found! Likely bad things to come!     Cloud-init is set to use OVF as the only datasource.  via an iso connected to the vSphere machine which i verified its connected. what can i do next to trouble shoot this ?
<Odd_Bloke> malasfar: Where is your image from?
<malasfar> i built it my self for the official ubuntu image then installed cloud-init
<malasfar> from*
<malasfar> its always a 50 - 50 chance that it works and but it doesn't thats the error i get
<Odd_Bloke> malasfar: From the ISO, you mean?  Would any of the pre-built images at http://cloud-images.ubuntu.com/releases/xenial/release/ work for you?
<malasfar> Yes from an iso .. . your say to try the cloud images ?
<malasfar> saying* sorry about the typo
<malasfar> i have not tried the cloud images you referenced before. would it make any difference ?
<Odd_Bloke> malasfar: Those come with cloud-init preinstalled in a known-good way, so it reduces the moving parts involved in working out what the issue is. :)
<malasfar> okay will give it a try .. thank you Odd_Bloke
<Odd_Bloke> malasfar: Most of the cloud-init team will be finishing their day fairly soon, so if you continue to see issues then I suggest filing a bug using the link in the /topic so that someone definitely sees it. :)
<malasfar> Odd_Bloke i m assuming these are already pre-built VMs for vSphere which file i would use . i am looking to down this file ubuntu-16.04-server-cloudimg-amd64.ova
<Odd_Bloke> malasfar: I'm not familiar with vSphere I'm afraid, you'll have to determine which image (if any) will work for your platform.
<malasfar> okay thank you again
<malasfar> Odd_Bloke . i just provisioned one from the cloud image and i didn't even see cloud-init execute .. Whats the credentials for these images ? so i can access the VM
<malasfar> oh snap.,. that actually worked cloud-init did execute
<sarnold> malasfar: normally you supply the account name and login credentials in a datasource; where I get super-confused about cloud-init is I've got no idea how to provide a datasource. (I figured something out for running qemu by hand a few years ago..)
<malasfar> i just trying to see how these images were setup .. there is a default user called ubuntu for the ubuntu 16.04 i just imported to vSphere
<powersj> either the cloud will provide that detail in metadata or you can provide it via user data.
<powersj> and then cloud-init will setup what it was given
<powersj> Our images come with a default user, but expect an SSH key to be provided to avoid passwords in the first place
<blackboxsw> sweet malasfar, so in vSphere, you can pass in a #cloud-config text file as user data
<blackboxsw> look what I found:   https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/
<blackboxsw> might have some relevance to you
<blackboxsw> nice to see someone referencing validating cloud-config schema  `cloud-init devel schema --config-file my-user-data-file.txt`
<malasfar> yes i have seen this blog.. i m using vRealize Automation 8.0 and i can use infrast
<malasfar> yes i have seen this blog.. i m using vRealize Automation 8.0 and i can use infrastructure as in the form of a blueprint with Cloud Config Code. that gets passed to the VM via the CD-ROM device as a ISO
<blackboxsw> and https://vmsysadmin.wordpress.com/2019/09/20/cloning-ubuntu-18-04-lts-cloud-image-on-vmware-using-cloud-init/ is interesting
<minfrin> @blackboxsw I had found the links you posted, but they seem to describe the creation of a swap file rather than a dedicated swap disk. Our full question is here: kivtqz2OlyCalhu2fiud
<sarnold> minfrin: is that a password that needs to be changed?
<minfrin> Yes it is, and my keyboard wants to caps lock, too.
<minfrin> https://stackoverflow.com/questions/60095848/how-do-you-configure-a-swap-partition-using-cloud-init
 * minfrin changes password, has word with keyboard, glares at command C.
<malasfar> thats very weird if i deploy a VM using a DHCP IP assignment it works fine the minute i start using vSphere customization i get this error " Customization of the guest operating system 'ubuntu64Guest' is not supported in this configuration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and
<malasfar> VMware Tools versions. Refer to vCenter documentation for supported configurations."
<malasfar> but it doesn't say what configuration that its not supported
<malasfar> When i use DHCP  customization is not invoked . the machine gets an ip from the DHCP server and cloud init works. ( Executes the Cloud config Code )  . when i introduce customization so i can set the hostname with a static IP . i get the above error
<malasfar> this with using the cloud image
#cloud-init 2020-02-07
<ahosmanMSFT> Hey rharper and blackboxsw I just saw your message. Was at a team event all day yesterday.
<rharper> np
<ahosmanMSFT> @rharper: I see you got a PR up, looking at it now
<rharper> great
<ahosmanMSFT> How did testing go?
<rharper> all seems well on the branch I proposed
<rharper> i can run things the way you do, as well as using a --deb, etc
<ahosmanMSFT> oh so images launch?
<rharper> I think the two core issues were the missing start() in the image.py and using vm_name vs. image_id
<rharper> yes
<ahosmanMSFT> Nice, two birds with one stone. I suppose that permanently fixes the ssh issue?
<rharper> I believe so
<ahosmanMSFT> I was also looking into preserve instance, have you looked into that. Since I delete the resource group, the instance gets deleted when reserve instance gets invoked. I have a fix in mind to not delete the resource group, but the user would be in charge of deleting it. What do you think @rharper and @blackboxsw?
<rharper> ahosmanMSFT: w.r.t preserve-instance,  I do believe that's expected; it may be useful to emit the resource group name in the logging , something like "No removing resourse group %s due to --preserve-instance setting; requires manual removal"
<ahosmanMSFT> Great, that's what I was thinking. Is preserve instance passed as a boolean?
<ahosmanMSFT> Also @rharper, approved the PR is another reviewer needed
<rharper> ahosmanMSFT: yes, boolean, and yes blackboxsw or Odd_Bloke will need to review as well
<ahosmanMSFT> Ok @rharper looking into that now
<rharper> nice
<ahosmanMSFT> @rharper In stages.py preserve_instance is initialized and destroys VM if False, this doesn't account for the resource group.
<ahosmanMSFT> How would you like this done
<ahosmanMSFT> I can omit the destruction of the resource group but I would have to add to stages
<ahosmanMSFT> which should be generic by nature
<minfrin> Hey all, always being dragged off somewhere after answering the question.
<minfrin> I am trying to find out how cloud-init can be told to mkswap and swapon a swap disk - most specifically a swap disk, not a swap file.
<minfrin> This doesn't appear to be enough:
<minfrin> fs_setup:  - label: vidi    device: /dev/xvde    filesystem: ext4  - label: swap    device: /dev/xvdg    filesystem: swapmounts:- [ /dev/xvde, /var/lib/vidispine, ext4, defaults, 0, 0 ]- [ /dev/xvdg, none, swap, sw, 0, 0 ]
<blackboxsw> minfrin: can you paste somewhere with fomatting? can't quite grok that.
<blackboxsw> https://paste.ubuntu.com would work
<blackboxsw> I'm assuming that's part of #cloud-config
<blackboxsw> or paste.openstack.org maybe
<minfrin> Yep, formatting a bit stuffed. Here is a paste: https://paste.ubuntu.com/p/TpjfznyJZJ/
<blackboxsw> thx, ok valid yaml.
<blackboxsw> minfrin: missing  something like a top-level swap: key maybe?
<blackboxsw> minfrin: like this maybe https://paste.ubuntu.com/p/sg7Wky7tZ9/
<blackboxsw> I'm not totally sure, just looking through the code in cloudinit/config/cc_mounts.py https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_mounts.py#L458-L467
<blackboxsw> it looks like needsswap is set True if the overall cloud-config contains a top-level "swap" key with definitions
<blackboxsw> and then swapon gets called https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_mounts.py#L510L511
<rharper> ahosmanMSFT: I'll take a look
<ahosmanMSFT> @rharper: I actually have a fix
<ahosmanMSFT> https://paste.ubuntu.com/p/4KzMXmjCKS/
<ahosmanMSFT> update azurecloudtest/platform.destroy
<ahosmanMSFT> can you add that with your fix, so we can get it merged faster
<rharper> ahosmanMSFT: I suspect we want some sort of args passed down into the Instance or Image class from stages
<rharper> ahosmanMSFT: no, we'll do this in a separate PR
<ahosmanMSFT> ok
<ahosmanMSFT> rharper: Did that with the paste, what do you think? I can push the PR
<minfrin> @blackboxsw: If I'm reading this correctly from https://github.com/canonical/cloud-init/blob/06e324ff8edb3126e5a8060757a48ceab2b1a121/cloudinit/config/cc_mounts.py#L481 - if line[2] is swap we get swapon.
<minfrin> blackboxsw: In theory this line should be enough to trigger a swapon?
<minfrin> - [ /dev/xvdg, none, swap, sw, 0, 0 ]
 * minfrin tries https://paste.ubuntu.com/p/sg7Wky7tZ9/
<ahosmanMSFT> @rharper: https://github.com/canonical/cloud-init/compare/master...AOhassan:azurecitestsupdate?expand=1
<minfrin> blackboxsw: Configured as per the paste, the mkswap fails as per the paste: https://paste.ubuntu.com/p/6pBDvzsST8/
<minfrin> It looks like despite no maxsize being passed, cloud-init passes the empty string:
<minfrin> Command: ['/sbin/mkswap', '/dev/xvdg', '-L', 'swap', '']
<minfrin> That then fails, as expected:
<rharper> ahosmanMSFT: I don't think that's goign to work, as we need the PlatformComponent instance from collect_test_data() in collect;  I suspect we may want to pass the args.preserve_instance  setting into the partial so the platform instance retains this setting
<minfrin> Stderr: mkswap: invalid block count argument: ''
<minfrin> blaackboxsw: Looks like with maxsize passed, we're still passed the empty string.
<blackboxsw> interesting minfrin, could be a bug there. trying to see if we had an integration tests covering this
<blackboxsw> hrm nope
<blackboxsw> minfrin: have an updated #cloud-config snippet you could paste
<blackboxsw> ?
<rharper> blackboxsw: afaict, there's not way with swap: to create this against a block device, which is what it sounds like minfrin wants;  I would suggest to use bootcmd: ['mkswap', '/dev/xxxx']; and then a mount:  entry
<rharper> in general the swapfile is more portable which is why cloud-init prefers that;  if you capture an image from an instance, the configured swap goes with it; vs having to ensure you have a secondary disk always present at a specific location
<blackboxsw> rharper: minfrin while I keep making wild guesses at this :) I also noticed Odd_Bloke's cloud-config test for swap SRU test was referencing just the device name, not the full device path https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/b59870ca.txt#L16-L28
<minfrin> I've raised this as a bug here: https://bugs.launchpad.net/cloud-init/+bug/1862417
<ubot5> Ubuntu bug 1862417 in cloud-init "cloud-init: Attempt to mkswap on a partition fails: invalid block count argument: ''" [Undecided,New]
<blackboxsw> thanks for the bug minfrin, we can respond there.
<minfrin> Passing the empty string as the "size" parameter definitely looks odd.
<rharper> blackboxsw: well, the swap cloud-config specifies filename, but we don't specifically say it cannot be an existing block device; but the code as written isn't going to correctly mkswap on a disk or partition;  and I don't think it was intended to;
<rharper> as I mentioned, it's non-portable with image capture;  that doesn't mean cloudinit can't support mkswap on devices;
<minfrin> What confused me was examples that showed swap disks being created, but not examples where mkswap was included.
<rharper> minfrin: which example?
<blackboxsw> minfrin: also in your bug, I see you specified  mount_default_fields: [ None, None, "auto", "defaults", "0", "2" ]
<blackboxsw> I think proper yaml should be none instead of None. otherwise it is treated as the string "None"
<blackboxsw> instead of python's None
<blackboxsw> not sure if that affects things or not
<blackboxsw> sorry I mean null in yaml
<blackboxsw> example of what I mean https://paste.ubuntu.com/p/4M7nFWWBnQ/
<blackboxsw> though I realize that cloud-config is documented in cloud-init's rtd page. we must have a mapping that sets that appropriately
<minfrin> rharper: this is what I worked from: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#adjust-mount-points-mounted?
<blackboxsw> funny also that those docs there say: # complete.  This must be an array, and must have 7 fields.
<blackboxsw> yet the docs provide a list of 6 items
<rharper> the comments field
<rharper> minfrin: thanks
<blackboxsw> even in code     defvals = [None, None, "auto", def_mnt_opts, "0", "2"]   6 items. I'll propose a tiny doc fix for that aspect now
<rharper> hrm, somethings strange, as the fs_setup config for swap shouldn't have the trailing '' ;  I  see it
<rharper> it's a disk, so cc_disk_setup uses lookup_force_flag and passes in the filesystem type, and swap is not one of the types, so it returns ''
<rharper> which is appended to the cmd and fails
<blackboxsw> rharper: but it checks             if force_flag: before trying to append ''
<rharper> right
<blackboxsw> so I think it rejects that
<rharper> just saw that
<rharper> so... how did it get the extra '' ?
 * blackboxsw was wondering about fs_opts being non-zero
<rharper> I have a unittest that does not fail
<rharper> fs_cfg.get('extra_opts', [])
<blackboxsw> but I can't see where extra_opts comes from
<blackboxsw> yeah
<rharper> yeah, this is very strange
<blackboxsw> minfrin: do you see a log in /var/log/cloud-init.log like "Using cmd: " just after a log "Creating file system %s on %s"
<blackboxsw> minfrin: actually if possible. please attach the tarfile of 'cloud-init collect-logs' to your bug https://bugs.launchpad.net/cloud-init/+bug/1862417
<ubot5> Ubuntu bug 1862417 in cloud-init "cloud-init: Attempt to mkswap on a partition fails: invalid block count argument: ''" [Undecided,New]
<blackboxsw> then we can check it out
<blackboxsw> ohh well I guess that's reported anyway in Command: ['/sbin/mkswap', '/dev/xvdg', '-L', 'swap', '']
<rharper> blackboxsw: well, I'd like to confirm the while log and the user-data that was provided ...
<rharper> I'm just not seeing how we can get an extra element in the code path
<blackboxsw> yeah in either case the logs would help
<blackboxsw> yeah same
<minfrin> I looked in the source, and the only reference to mkswap is here: https://github.com/canonical/cloud-init/blob/06e324ff8edb3126e5a8060757a48ceab2b1a121/cloudinit/config/cc_mounts.py#L264
<minfrin> This is the version we are using:
<minfrin> ii  cloud-init                        19.4-33-gbb4131a2-0ubuntu1~18.04.1  all          Init scripts for cloud instances
<blackboxsw> minfrin: and here: https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_disk_setup.py#L971
<blackboxsw> that gets constructed becauase mkfs.swap utiltiy doesn't exist
<rharper> im testing the supplied config in openstack instanc enow
<blackboxsw> this is the module where  the problem lies
<blackboxsw> cc_disk_setup.py somehow
<rharper> cc_disk_setup
<blackboxsw> yeah somewhere in https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_disk_setup.py#L984-L996
<blackboxsw> where we are getting another '' injected
<rharper> 2020-02-07 21:25:08,540 - cc_disk_setup.py[WARNING]: Force flag for swap is unknown.
<rharper> 2020-02-07 21:25:08,541 - cc_disk_setup.py[DEBUG]: Creating file system swap on /dev/vdc
<rharper> 2020-02-07 21:25:08,541 - cc_disk_setup.py[DEBUG]:      Using cmd: ['/sbin/mkswap', '/dev/vdc', '-L', 'swap']
<rharper> 2020-02-07 21:25:08,541 - util.py[DEBUG]: Running command ['/sbin/mkswap', '/dev/vdc', '-L', 'swap'] with allowed return codes [0] (shell=False, capture=True)
<rharper> 2020-02-07 21:25:08,627 - util.py[DEBUG]: Creating fs for /dev/vdc took 0.103 seconds
<minfrin> I updated https://bugs.launchpad.net/cloud-init/+bug/1862417 with the full log.
<ubot5> Ubuntu bug 1862417 in cloud-init "cloud-init: Attempt to mkswap on a partition fails: invalid block count argument: ''" [Medium,Incomplete]
<blackboxsw> thanks minfrin
<blackboxsw> and success on rharper's side.
<blackboxsw> hrm
<rharper> minfrin: at a min, the swap: config needs to go
<rharper> it writes a file on top of the block device; which messes things up
<minfrin> Removing the swap section still shows the mkswap with the extra empty string at the end.
<rharper> yeah, that's the part I don't get
<rharper> so somethings disconnected between what you're running/user-data and cloud-init code
<minfrin> Github's search is not that great, I stumbled on this PR: https://github.com/canonical/cloud-init/pull/143
<minfrin> Looks like a fix about a month ago.
<rharper> but you show 19.4-33
<rharper> which has the fix
<rharper> but maybe, you don't have that version ?
<rharper> which would explain the failure
<minfrin> The machine I'm working on uses cloud-init to update itself, it might only have the fix after the updates.
<rharper> ah, interesting
<rharper> cat /etc/cloud/build.info
<rharper> that'll give us a point in time for which version you have
<rharper> and I suspect you're right, the top of your cloud-init.log will the original version
<minfrin> build_name: serverserial: 20190514
<minfrin> Definitely way older than the PR.
<rharper> yep
<minfrin> I suspect the fix in our case is to use the latest image of Ubuntu from end Jan 2020.
<rharper> yep
<rharper> and drop the swap: section;
<rharper> otherwise, things look good on my recent images with similar cloud-config
<rharper> https://paste.ubuntu.com/p/y8JBCrfK9K/
<Odd_Bloke> rharper: Is this the force flag issue that was fixed recently?
<Odd_Bloke> Oh, I was scrolled up.
<rharper> Odd_Bloke: indeed it was
<Odd_Bloke> < rharper> Yes.
<Odd_Bloke> :)
<rharper> minfrin: found the PR and we confirmed the image in use was out of date
<rharper> and we can have a PR to update the lookup_flags to add the -f
<rharper> for fstype swap
<minfrin> Thanks for the help getting to the bottom of this, I appreciate it.
<rharper> minfrin: yw
<minfrin> Managed to confirm - very latest Ubuntu Bionix image is fixed:
<minfrin> 2020-02-07 22:21:09,266 - cc_disk_setup.py[WARNING]: Force flag for swap is unknown.2020-02-07 22:21:09,266 - cc_disk_setup.py[DEBUG]: Creating file system swap on /dev/xvdg2020-02-07 22:21:09,266 - cc_disk_setup.py[DEBUG]:      Using cmd: ['/sbin/mkswap', '/dev/xvdg', '-L', 'swap']2020-02-07 22:21:09,266 - util.py[DEBUG]: Running command
<minfrin> ['/sbin/mkswap', '/dev/xvdg', '-L', 'swap'] with allowed return codes [0] (shell=False, capture=True)2020-02-07 22:21:09,290 - util.py[DEBUG]: Creating fs for /dev/xvdg took 0.058 seconds
<rharper> minfrin: nice!
<rharper> https://github.com/canonical/cloud-init/pull/207
<rharper> I put that up for the warning
#cloud-init 2020-02-08
<malasfar> Hello!..  I am using a provisioning tool to provision VMs on vSphere we pass user data via an ISO . cloud-init is configured to look for OVF datasource only. As it works sometimes I m noticing that most of the time it cannot find the datasource . if i do
<malasfar> cloud-init clean then cloud-init init commands it finds it..
<malasfar> looks to me its a race condition but i m not sure where or with which component. how can i guarantee that the datasource is always found. or what possibly could be happening for cloud-init to miss the datasource
<malasfar> Okay is there away to delay cloud-init at boot
#cloud-init 2020-02-09
<dsofeir> Hello, I am placing a file network-config in the CIDATA ISO and it will not apply the config to the Ubunutu 18.04 netplan
