#cloud-init 2014-03-24
<orion195> hi guys, quick question
<orion195> in a fc19 cloud image, the /etc/cloud/cloud.cfg file
<orion195> doesn't contain: preserve_hostname: false
<orion195> but contains: cloud_init_modules: set_hostname
<orion195> does this mean that cloud_init will update the hostname or not?
<orion195> anyone around?
<kwadronaut> orion195: just stick around.
<orion195> you mean I have to stick here around until someone will reply? sure why not
<kwadronaut> or use a mailinglist or something
<orion195> do you know where I can find any default values in the cloud.cfg file?
<kwadronaut> http://cloudinit.readthedocs.org/en/latest/ ?
<harlowja> suok i'm bck
<harlowja> *ok
<harmw> harlowja: what would be the odds on you hving an update from sean? :p
<harlowja> harmw doesn't seem like much, he's not on YIM
<harlowja> :(
<harmw> thought so :p
<harmw> smoser: how should mdev work for cirros, or would you prefer how it currently is with the addition of some specific if [ -d .. modprobe stuff?
<harmw> stuff like this: [ -d /sys/bus/.. ] && modprobe bla 
<smoser> harmw, i'm sorry, i'm not really here till like thursday.
<smoser> i'd like mdev to be running fully unless that isn't sane
<harmw> so mdev should take care of loading hyper-v drivers? 
<harmw> meaning hv_vmbus and then hv_storvsc (blockdev) to get to initrd which has the other drivers like hv_netvsc for networking
<smoser> it is supposedto, isnt it ?
<harlowja> u guys and your cirros
<harlowja> smoser should start cirros inc.
<harlowja> none of that ubuntu crap
<harlowja> lol
<harlowja> *canonical
<smoser> ipo coming soon.
<harmw> it should go like that smoser , yes, but I didn't get it to do just that :p
<smoser> harmw, hm..
#cloud-init 2014-03-25
<oobx> smoser: you told me to look at in the examples folder for documentation.  I've done that.  Should I consider the directives in those examples a complete, all-inclusive set of potential directives?
<smoser> oobx, its reasonably complete.
<oobx> :) I'll accept that answer.  thanks for all the work!
<oobx> i'll point my customers at those examples
<smoser> oobx, we're open to patches to improve documentation too :)
<oobx_> cool.  thanks!
#cloud-init 2014-03-26
<smoser> alexpilotti, around ?
<alexpilotti> smoser: hey!
<smoser> would you think it possible to install windows via :
<smoser>  mkfs.ntfs(some.disk)
<smoser>  mount some.disk
<smoser>  tar -C moutpoint -Sxvzf my-windows-tarfile.tar.gz
<smoser>  fixup_windows(mountpoint)
<smoser>  reboot 
<alexpilotti> if my-windows-tarfile.tar.gz is a preinstalled copy of WIndows, sure
<alexpilotti> it's no different then ghosting a machine
<smoser> and fixup_windows() is some amount of things that have to be done
<smoser> right.
<alexpilotti> not even
<alexpilotti> but
<alexpilotti> you'll have to consider that you need to expand the file system
<alexpilotti> I mean, if your initial image is say, 15 GB
<alexpilotti> and your target partition is 20GB
<smoser> right. but that could be done in fixup_windows
<smoser> right ?
<alexpilotti> yep, I though you meant to fix only the MBR / UEFI boot, etc
<smoser> well, whatever amount of "fix" required
<alexpilotti> if it's an image with cloudbase-init installed you don't even have to care, as partitions get resized automatically on boot
<smoser> but those are filesystem images
<smoser> could you do the same wtih tar image ?
<alexpilotti> can you give some context about where this could be needed?
<smoser> mainly just wondering if i could install windows that way.
<smoser> doesnt have to support a bunch of smart options, just get installed and boot.
<cloud-init-n00b> hello everyone
<cloud-init-n00b> i've been trying to get familiar with cloud-init for modifying image installs on openstack
<cloud-init-n00b> but i'm having a really hard time getting it to work
<cloud-init-n00b> and google is not being too helpful
<cloud-init-n00b> i'm trying to do something simple first
<cloud-init-n00b> basically create just a user and verify that it is actually being created
<cloud-init-n00b> #cloud-config  users:   - default   - name: stack     plain_text_passwd: 'testPass'     lock_passwd: False     sudo: ["ALL=(ALL) NOPASSWD:ALL\nDefaults:stack !requiretty"]     shell: /bin/bash
<cloud-init-n00b> whoa bad formatting
<smoser> alexpilotti, thanks for your input.
<cloud-init-n00b> http://pastebin.com/Nd7H6aHS
<cloud-init-n00b> that is what my simple config looks like
<cloud-init-n00b> any chance someone could give me a hand
<cloud-init-n00b> i'm not sure what it is i'm missing
<cloud-init-n00b> :/
<alexpilotti> smoser: sorry, got interrupted
<alexpilotti> smoser: to be sure, what do you mean with tar image?
<alexpilotti> smoser: an OS installer image, like an ISO?
<smoser> mount -t ntfs /dev/my-windows-partition /mnt
<smoser> tar -C /mnt -Scpzf my-windows-tarball.tar.gz
<smoser> umount /mnt
<alexpilotti> got it
<smoser> cloud-init-n00b, what you have there doesn't look completely unreasonable
<smoser> with a recent cloud-init, you can poke around quicker by running the user-groups module manuall
<alexpilotti> to be compliant with MSFT support, you need to sysprep the machine
<smoser> sudo cloud-init single --frequency=always --name=user-groups
<smoser> and then printf debugging that way (or looking at /var/log/cloud-init.log)
<alexpilotti> but for a brutal "let's get it running" it works
<cloud-init-n00b> smoser, if you have any links on how to run that module manually 
<alexpilotti> it's no different than ghosting a physical machine in the end
<cloud-init-n00b> i'd be very thankful
<cloud-init-n00b> don't mind reading at all :)
<smoser> sysprep would be ok. especially if it can be rebooted into sysprep and then rebooted automatically from there.
<smoser> cloud-init-n00b, look there ^
<smoser> sudo cloud-init single --frequency=always --name=user-groups
<alexpilotti> the tricky part is that the only way to do that would be to inhect in the registry a script that would run sysprep on the next boot
<alexpilotti> while normally you'd run the machine, sysprep, shutdown, copy the image
<alexpilotti> if you can control the machine this way, it's fine
<alexpilotti> s/inhect/inject/
<cloud-init-n00b> smoser i'm afraid i did not understand your reference, where is it that you advised me to look into (thanks)
<smoser> cloud-init-n00b: run: sudo cloud-init single --frequency=always --name=user-groups
<smoser> that will re-run the user-groups m odule
<smoser> that would read that data you gave
<smoser> and then it should log some things to /var/log/cloud-init.log
<cloud-init-n00b> got it. i'll give it a run then. Thanks!
<cloud-init-n00b> i'm getting an error saying :
<cloud-init-n00b> bad command single. use one of ('start', 'start-local')
#cloud-init 2014-03-27
<cjdc_> Hello all
<cjdc_> quick question: in cloud config's write_file userdata
<cjdc_> how do i put a new line in the content?
<smoser> cjdc_, there are a lot of ways to do it.
<smoser> its yaml formated.
<cjdc_> | should be enoughj right?
<cjdc_> maybe the issues I am having is with escape characters
<smoser> cjdc_, http://paste.ubuntu.com/7162529/
<smoser> oops. the thing at line 15 was wrong.... i actually hadn't updated /tmp/foo to what it shows.
<smoser> http://paste.ubuntu.com/7162554/
<smoser> so in short, the guaranteed-works answer is to just use yaml.dump
<smoser> you can also use json if you prefer
<smoser> yaml is pure superset of json
<sayalilunkad> Hello everyone, I am trying to write a cloud init pliugin to load the contents of a file. But I am not very familiar with the code base for cloud init. Could someone guide me a bit here
<smoser> cjdc_, theres an example of dumping json.
<smoser> http://paste.ubuntu.com/7162563/
<cjdc_> smoser: I am trying that right now. thanks
<smoser> you can also base64 encode stuff.
<smoser> or use the '!!binary'
<smoser> sayalilunkad, "loads the contents of a file" ?
<smoser> the two "plugin" ways to do something new in cloud-init are "config-modules" and "part-handler"
<sayalilunkad> smoser: Yes, loading data from a file 
<smoser> additionally, if what you need to do is just run stuff on boot, you can lay down scripts in certain directories and they'll run.
<smoser> thats less "plugin".
<sayalilunkad> smoser: Actually I am applying to an organization had given me this problem statement 
<sayalilunkad> Write a Cloud-Init plug-in to load data from a file was the exact exercise to do
<smoser> well, if you just said that, i'd think that "config module" would probably be the right thing.
<smoser> those are in cloudinit/config/cc_*
<smoser> and then you enable them by adding them to one of the module lists in config/cloud.cfg
<sayalilunkad> ok so configuring a module can help with doing this?
<sayalilunkad> smoser: Since I am fairly new to this can you give me some links that would help
<sayalilunkad> Currently using this http://cloudinit.readthedocs.org/en/latest/topics/hacking.html
<smoser> well, you can definitly run 'open(some_path_to_file).read()' from a config module, whic would qualify as "loading data from a file" in some sense.
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/790
<smoser> is an example of adding a config module.
<sayalilunkad> Alright, thanks. Will try it 
<smoser> in that commit you can ignore the noise in util.py and cc_resizefs.py, as that was just moving a block of code from one place to a common place
<sayalilunkad> smoser: okay
<sayalilunkad> smoser: Also in my cloud-init/config I have only these two file cloud.cfg  cloud.cfg.d
<sayalilunkad> is that ok?
<smoser> thats expected, yes. look at the changes done in that diff, and make similar changes for what you want to accomplish.
<smoser> i'm sorry, but I really dont' have time to help you much more than that.
<smoser> documentation improvements are welcome of course.
<sayalilunkad> smoser: no problem! thanks a lot :)
<cjdc_> smoser: it worked with dump!
<cjdc_> thanks for the help
<mdorman> i'd like to have ssh not start until after clout-init has run (to install the ssh key), basically so that the user cannot even connect to ssh at all until the keys are installed.  is there a way to tell cloud-init to start ssh after configuring the keys?
<smoser> mdorman, cloud-init doesnt start ssh
<smoser> on ubuntu ssh starts normally.
<smoser> your request makes sense though.
<smoser> to do it you'd have to modify the ssh job.
<pquerna> https://gist.github.com/pquerna/491a3487951131f35f8f <- example of an.. upcoming openstack ironic interfaces file that'll get dropped into configdrive (still needs some tweaks).  ubuntu/deb should be fine, but will try to make the patches to get it working on rhel/etc in the next few weeks
<mdorman> smoser:  yep, got it.  makes sense.  it does _restart_ ssh as part of the set-password module, but not totally what i want.  i'll figure something out.
<smoser> right. it only restarts if it changes the config
<smoser> because it has to.
<mdorman> yeah.
<smoser> mdorman, is this ubuntu ?
<mdorman> nope, centos
<smoser> mdorman, it migh tbe possible to run something as a boothook that stopped ssh from starting.
<smoser> i tihnk that'd be possible on ubuntu.
<smoser> but i dont know the centos jobs well enough
<mdorman> yeah i was able to figure out a different solution
<smoser> what did you do ?
<mdorman> figured out that sysvinit wise, sshd was set to start after cloud-init
<mdorman> so it should have just worked
<mdorman> however, we use power broker to do auth on our linux machines to AD, and the bootstrap for that (which starts earlier) was actually starting ssh on its own
<mdorman> so just needed to turn that piece off, so that it started in the correct order according to stock init scripts
<mdorman> thanks for the help.  gotta run
#cloud-init 2014-03-28
<smoser> harmw, are you going to atlanta in a few weeks?
<harmw> smoser: nope, not unless you can arrange some fancy hotel and get me a flight ticket :p
<harmw> you will be there?
<smoser> yeah. 
<smoser> what about paris in october?
<smoser> you're UK ?
<harmw> NL :)
<harmw> paris? o my..
<harmw> I can probably drive to Paris (not that I'd want to)
<smoser> openstack summit is paris in october
<harmw> now that is nice
<harmw> (but Paris sucks balls when it comes to driving, not to mention the language)
<smoser> :)
<smoser> i thought they had reasonable trains on that continent.
<harmw> ah yes, could be
<harmw> I prefer using cars though :p
<harmw> though in this case, perhaps
<smoser> i asked about atlanta to make sure i could get you a cirros shirt if you were.
<harmw> lol
<smoser> "I prefer using cars though :p"
<harmw> how nice :)
<smoser> you'd fit in well in america
<harmw> hehe
<harmw> wasn't your wife going to make some sort of cirros logo to put on such a shirt? :)
<harmw> isn't it listed on openstack.org?
<smoser> yeah, my wife showed me some mock up last night of cirros logo, so that is waht got me thinking.
<smoser> harmw, http://www.slideshare.net/openstack/openstack-march-2014-community-marketing-meeting
<smoser> they announced that at the last summit.
<smoser> see slid 3 there, but thats all i can find as to "official documentation" :)
<harmw> ah ok
<harmw> thanks
<smoser> also here http://www.slideshare.net/openstack/openstack-march-2014-community-marketing-meeting
<smoser> 5 minutes in
<harmw> thats the same link, right?
<smoser> bah
<smoser> https://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/openstack-keynote-featuring-concur-digitalfilm-tree-shutterstock
<harmw> ah
<harmw> smoser: I've got my Cubietruck arm board, just need to figure out how to enable HYP mode to do the virtualisation stuff
<smoser> nice.
<smoser> thanks for support, harlow!
<harlowja> smoser np!
<harlowja> i am here to serve :)
<harlowja> harmw i talked to sean a couple days ago, still suck in mail land
<smoser> i see right through you.
<harlowja> mail land not want to upgrade there freebsd (fear of upgrading?)
<harmw> omg, how hard can it be
<smoser> you just read up and saw i was offering cirros shirts :)
<harlowja> cirros shirts
<harmw> lol smoser 
<harlowja> where
<harlowja> reading
<harlowja> ohh, isn't that your wife making shirts, not u smoser  ;)
<harlowja> i see through you!
<harlowja> lol
<harlowja> harmw ya, he says its hard (not sure why, lol)
<harmw> hmk, how hard can postfix/dovecot be
<harmw> :p
<harmw> I'm curious how the y! setup is btw
<harmw> given all the apparent troubles
<harlowja> yea, sean would be the man to know all of that
<harlowja> i don't want to know such info, hahaha
<harlowja> :-P
<smoser> upgrade ?
<smoser> do people still do that?
<harlowja> lol
<smoser> they're not pets man, they're cattle.
<harlowja> ya, mail is old school still...
<harlowja> even there running of freebsd isn't desired...
<harlowja> harmw no offense intended :-P
<harmw> ? I'm only running freebsd on my tplink1043 routers :>
<harmw> eg. embedded stuff
<harmw> and a cloudy instance, every once in a while
<smoser> oh. yahoo mail. 
<smoser> wow.
<harlowja> *not all of y! mail
<smoser> i still use juno.
<harlowja> some part of it (not really sure what part)
<harmw> whats juno?
<smoser> http://webmail.juno.com/
<smoser> webmail popular at the same time that yahoo mail was popular.
<harmw> ah
<harmw> neither of which made it here in Holland :p
<harlowja> harmw whats popular there?
<harlowja> hollandmail
<harlowja> snailmail
<harmw> hotmail, live, gmail
<harlowja> ah, good ole hotmail
<harmw> most of us probably used hotmail
<harmw> before gmail
<harlowja> ya, my mom uses that :-P
<harlowja> once u get stuck i guess nobody really switches
<harlowja> as long as it works good enough
<harlowja> my dad stuck on y! mail, i'm more of gmail
<harlowja> *don't tell people at y!*
<harlowja> lol
<harmw> lol
<harmw> I've got my own, way better
<harlowja> of course
<harlowja> smoser u gonna be at atlanta openstack summit?
 * harlowja forgot if u already told so
<smoser> yeah
<harlowja> cool
<harlowja> do some talks?
<harlowja> could do the cloud-init one again
<harlowja> cause that was so fun, haha
#cloud-init 2014-03-30
<jclift> Hmmm, is /var/lib/cloud/instance/obj.pkl meant to be consumable via python on spun up VM instances?
 * jclift is trying to pass meta-data to an instance, for reading by the instance after cloud-init has finished
<jclift> While it's would be possible to mount the config drive somewhere (eg mount /dev/xvdd /foo) and get it that way
<jclift> It seems like it'd be simpler to be able to unpickle /var/lib/cloud/instance/obj.pkl and get the data that way
<jclift> Or am I think about this all wrong? (completely possible) :)
<jclift> Ahhh.  from cloudinit import util
<jclift> Got it working. :)
#cloud-init 2015-03-23
<harlowja> smoser yo yo
<smoser> harlowja, hey.
<smoser> horay for speaking.
<harlowja> yaaaa
<smoser> you have it all done already ?
<harlowja> now guess we gotta figure out what to talk about, lol
<harlowja> hahaha
<harlowja> i thought u had it done?
<harlowja> why u slacking man
 * smoser imagines this same conversation taking place in vancouver :)
<harlowja> :)
<harlowja> then presentation becomes as much BS as we can create
<harlowja> lol
<harlowja> ^ the best presentation ever, lol
<harlowja> and then we both get fired, lol
<smoser> luckily we'll be somewhere where we can both get re-hired
<harmw> lol
<harlowja> :)
<harlowja> +2
<harmw> if you guys go in like that,make sure to hire Statler and Waldorf
<harmw> (or whatever way you type that)
<harlowja> not penn and teller?
<harlowja> final act, disaeapper, lol
<harmw> :p
<harlowja> *right before they can fire us
<harmw> smoser should wear his cirros tshirt on stage :P
<harlowja> def
<harlowja> i'll just be naked
<harlowja> lol
<harmw> :>
<harlowja> i mean, errr
<harmw> is there a common way of sorting output tables from tools like nova or neutron?
<harlowja> i think there are various wayss, not sure of a common way :-P
<harmw> hmk, I'm not talking about using other cmdline tools btw :p
<harmw> piping along in the process
<harlowja> http://lists.openstack.org/pipermail/openstack-dev/2015-January/thread.html#53734 
<harlowja> seems relevant :-P
<harlowja> 'Note that this behavior is not consistent across the projects.', lol
<harmw> lol
<harmw> how very openstacky is that :P
<harlowja> :)
<zilberstein> why cloud-init do not set timezone ?
<zilberstein> It is specified in /etc/cloud/cloud.cfg
<zilberstein> but not set in /etc/sysconfig/clock
<zilberstein> I try to set UTC
<harlowja> how did u specifiy it?
<harlowja> and whats your contents of /etc/cloud/cloud.cfg
<nk121> could someone explain the merge syntax to me? i want to have a base config file and then adds addiitional things (runcmd, write_files mostly) for specific instance types
<nk121> the example in the docs when none is provided is
<nk121> list()+dict()+str()
<nk121> but when i submit two cloud-config files through mime-type mechanism, it seems to completely ignore the 2nd one
<nk121> and actually in my case runcmd and writefiles are additive, but i'd like to have separate bootcmd setions for each instance (as i'm writing the instance type into the hostname)
#cloud-init 2015-03-24
<nk121> anyone around? trying to merge to cloud-init files and not having any luck
<nk121> the file /var/lib/cloud/instances/INSTANCE_ID/cloud.cfg  only shows the first file, and i can see both files in user-data.txt
<nk121> n/m its working now
<nk121> well, working more, its combining the two files, but its replacing sections, not combining
<nk121> this doc http://cloudinit.readthedocs.org/en/latest/topics/merging.html  seems to imply the new way should combine 
<nk121> and it gives an exact example of what i'm looking for -- combinging run_cmds
<nk121> i tried the merge_how from comment #4 from https://bugs.launchpad.net/cloud-init/+bug/1316323
<nk121> but same thing
<nk121> my run_cmd section is not merging
<nk121> merge_how: "dict(allow_delete,recurse_array)+list(recurse_array,append)"
<nk121> shouldn't that list(recurse_array,append) do the trick?
<zilberstein> harlowja_away, I specified timezone as the following:
<zilberstein> timezone: UTC
<zilberstein> doesnt work
<zilberstein> I specified timezone in /etc/cloud/cloud.cfg
<zilberstein> but it was not set
<zilberstein> so why that bloody cloud-init do not set timezone ?!
<Odd_Bloke> zilberstein: Is there anything in /var/log/cloud-init.log to suggest the problem?
<zilberstein> Odd_Bloke, nothing related to timezone
<zilberstein> that's the thing !
<zilberstein> it silently ignores timezone !
<Odd_Bloke> zilberstein: How are you checking what timezone is set? (And what distro are you on?)
<zilberstein> cat /etc/sysconfig/clock, redhat
<zilberstein> 6
<zilberstein> also /etc/localtime 
<Odd_Bloke> zilberstein: Is timezone in one of the module lists in cloud.cfg?
<smoser> zilberstein, the timezone module only runs once per instance.
<smoser> so booting, then changing /etc/cloud/cloud.cfg to have it, will not result in a reboot causing any change.
<smoser> you'd need to specify user-data to affect it. or either
<smoser>  a.) rm /var/lib/cloud/instance/sem/config_timezone && reboot
<smoser> b.) sudo cloud-init single --frequency=always --name=timezone
<smoser> c.) give user-data (or a part of multipart user-data) including:
<smoser>  #cloud-config
<smoser>  timezone: UTC
<smoser> d.) capture the image, and then deploy it again (which changes the instance-id, which makes that "per-instance" module run again).
<smoser> also, as Odd_Bloke said, you have to make sure that the module is enabled in /etc/cloud/cloud.cfg
<smoser> i'm not 100% certain that cloud-init timezone works for rhel, but what it looks like it intends to do is
<smoser>  delete /etc/localtime
<smoser>  symlink /etc/localtime -> /usr/share/zoneinfo/UTC
<smoser> now, if /usr/share/zoneinfo/UTC did not exist, it log an exception 'Invalid timezone'
<smoser> zilberstein, ^
<zilberstein> smoser, I know. Timezone was configured before instance started
<smoser> zilberstein, cloud-init will configure it (i think correctly), but only does that once per instance. cloud-init is about initialization, not runtime sytem configuration.
<zilberstein> smoser, I thought cloud-init should remove /etc/localtime and create new symlink on its own
<smoser> well, it does.
<smoser> once per instance.
<smoser> launch an instance with the user-data (in 'c' above) and see if you get what you want. or do 'a'. i think it will.
<nk121> good morning friends
<nk121> i'm having trouble merginging to cloud init configurations
<nk121> i don't understand the merge syntax
<nk121> its not combining my runcmd and write_files sections, but instead overriding them in the second file
<harmw> smoser: whats that ubuntu version and branch for cirros-dev again?
<smoser> https://code.launchpad.net/cirros
<harmw> sry, u meant the updated buildroot for ubuntu 14.04 (?) :)
<harmw> *I meant
<smoser> ah
<smoser> https://code.launchpad.net/~smoser/cirros/buildroot-upgrade
<harmw> cool
<nk121> smoser: any tips on how to debug merges?
<harlowja> nk121 eck, damn merging :(
<nk121> :(
<harlowja> haha
<harlowja> i sorta created it, unleashed it on the world, lol
<nk121> thanks :)
<harlowja> ^ for better or worse, ha
<harlowja> ha
<nk121> i feel i'm trying to do something simple, just combine two files, no overwriting
<nk121> actually q1- is there a way to re-run the merge without launching a new instance? (i figured out how to re-run cloud init on a box, but thats after the merge step)
<nk121> q2- the docs seem to imply the defaults for the new merge style is what i'm looking for, but i don't see that being the case
<harlowja> hmmm, i don't think so; pretty sure the merging happens to early
<harlowja> let me see if i can make a way for u to try this merge stuff without having to boot instances all the time
<nk121> if i don't have any merge_how sections, should the default do what i'm expecting?
<nk121> which is, combine the two runcmd and write_files sections from two cloud init files
<nk121> not overwrite or replace 
<nk121> (i'm running on trusty using the ec2 default image)
<harlowja> one way to find out :-P
<nk121> well so far, does not seem to :P
<harlowja> :)
<harlowja> nk121 ok, so http://paste.ubuntu.com/10671425/ that should make it easier for u to mess around
<harlowja> if u just run that on a vm with cloud-init; u can adjust that and see what the final result is...
<harmw> smoser: I noticed some nice fbsd specific additions on LP
<nk121> harlowja: ok, i'll give it a shot
<harlowja> for example http://paste.ubuntu.com/10671436/ might be what u want
<harlowja> that comes up with
<harlowja> Merged result:
<harlowja> {'a': 'b', 'c': [1, 2, 3, 4]}
<harlowja> ^ list appends
<harlowja> if u feel like it, keep track of a little readme doc about this while u are learning, could be useful docs for others
<harlowja> since i know its not so obvious...
<nk121> harlowja: sure, if i figure this out, i'll write up what i did and how to debug it
<harlowja> thx; feel free to mutate the above script; probably useful for others to
<harlowja> mutate/own/whatever lol
<harlowja> maybe mutate not best word :-P
<nk121> i follow ya
<nk121> wow, its so much faster to develop when you can re-run locally =-]
<nk121> and your second paste lead me to the solution
<nk121> i only had the merge_how section in the first file
<nk121> that seems to not be whats described in Specifying multiple types and its effect  in http://cloudinit.readthedocs.org/en/latest/topics/merging.html
<nk121> or maybe i'm just not parsing that right
<nk121> (the doc text)
<harlowja> ya, u probably don't need it in certain ones, but how it works is that the current payload defines the merge_how ; which will then be used with the prior already merged/whatever yaml
<harlowja> it doesn't really retain what was the 'last'
<harlowja> although it could i suppose, just doesn't currently
<nk121> i think that doc page could use a working example, and an explanation of the merge options
<harlowja> sure
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/merging.rst
<harlowja> commits welcome ;)
<harlowja> or any adjustments u feel appropriate
<nk121> ok i've added it to my list, i'll give it a shot after i get this milestone completed
<harlowja> cools
<nk121> but regardless, you've got me unstuck, and i really appreciate that
<harlowja> np
<harlowja> here to serve!
#cloud-init 2015-03-25
<ndonegan> Anyone seen issues where cloud-init is setting the hostname to localhost.localdomain?
<ndonegan> The advertised metata-data/hostname seems to be ok!
<Odd_Bloke> ndonegan: That sounds familiar; what OS/cloud are you running?
<Odd_Bloke> Actually, some searching around suggests I might have been thinking of something else. :/
<ndonegan> Odd_Bloke: Openstack Icehouse (moving to Kilo in the next few months) with Centos 6.5 VMs.
<Odd_Bloke> ndonegan: Is there anything relevant in cloud-init.log?
<ndonegan> Nothing that I can see. I'll get me some caffine and have a look again.
<ndonegan> Caffine taking priority right now :D
<Odd_Bloke> :D
<ndonegan> Absolutely nothing related to update_hostname or friends in the log that I can see.
<ndonegan> Ah! PEBKAC.
<Odd_Bloke> ndonegan: My favourite sort of problem. ;)
<ndonegan> A hacky fix for an older problem has come back to bite.
<ndonegan> Confirmed. Very much PEBKAC.
<Odd_Bloke> smoser: Review and merge of https://code.launchpad.net/~daniel-thewatkins/cloud-init/smartos-v2-metadata/+merge/251775 would be appreciated; it'd be good to get it in to vivid if possible.
<Odd_Bloke> (harlowja_away has already given it a once over)
<smoser> Odd_Bloke, ok. will take a look. thanks.
<Odd_Bloke> smoser: Thanks! :)
<smoser> Odd_Bloke, is there information on v2 metadata  somwhere? or on its differences from v1 ?
<Odd_Bloke> smoser: http://eng.joyent.com/mdata/protocol.html is the specification.
<Odd_Bloke> It's a completely different format from v1.
<Odd_Bloke> The keys stay the same though.
<smoser> put a link in code 
<smoser> ?
<smoser> also, i probably dont like how many LOG.info you do.
<smoser> there should be well defined criteria for when debug/info/warn are used, but there are not. instead you have smoser saying "probably".
<Odd_Bloke> smoser: I'll switch the info calls to debug.
<Odd_Bloke> smoser: Pushed those fixes up.
<Odd_Bloke> (Including the link to the spec)
<harlowja> smoser maybe https://wiki.openstack.org/wiki/LoggingStandards#Log_level_definitions should be just what we recommend?
<smoser> harlowja, dont you think that'd be better if it said "smoser's judgement call"
<harlowja> +1
<harlowja> lol
<harlowja> all hail dear leader smoser 
<smoser> the big thing is console output
<harlowja> true dat dear leader
<smoser> now and in cloud-init v2 , the console should get "OH NO, BAD STUFF" and very few "Yep all went well".
<smoser> and then fingerprints and such
<harlowja> that reminds me, oh where art thou  PCManticore  person for https://github.com/cloud-init/cloud-init/pull/2 :-/
<claudiupopa> harlowja: that's me.
<harlowja> sweet
<harlowja> how goes it
<harlowja> 'license change GPLv3 to Apache 2.0'
<harlowja> intersting
<harlowja> smoser is your boss ok with this :-P
<harlowja> bossman
<claudiupopa> well, I was just waiting for the licensing issue to settle, I should update the pull request this week and hopefully it gets integrated.
<harlowja> kk
<harlowja> guess they are settling
<smoser> bossman ack'd it.
<smoser> harlowja, its actuall "apache 2.0 or GPLv3" dual license
<smoser> which , is actually inherent to all apache 2.0 projects
<harlowja> interesting, didn't know there was an 'OR'
<smoser> but it is explicitly stated here.
<harlowja> intersting
<smoser> ie, apache2 is "upstream compatible" with gplv3
 * harlowja this stuff confuses me, lol
<smoser> so you can grab whatever you want that is apache2 and relicense it to someone under gplv3
<harlowja> cools i guess, ha
<smoser> but if i cna get away with it, i'm going to not explicitly list the GPLv3 info in the file headers.
<smoser> but unfortunately, i suspect if a lawyer gets involved, then..
<harlowja> i promise not to involve my sister (who is a lawyer), lol
<smoser> thank you
<harlowja> np
<harlowja> let's keep this between me and u smoser 
<harlowja> no need to involve lawyers, jeez
<harlowja> jeezsh
<smoser> Odd_Bloke, the only other comment...
<smoser> you're not sniffing one or the other, just expect that v2 is there? or are you
<Odd_Bloke> smoser: We just expect v2 is there (which is a fair assumption in Joyent).
<smoser> is there code we can drop then ?
<Odd_Bloke> smoser: The code which did v1 stuff has already been dropped.
<Odd_Bloke> smoser: We should eventually be able to get rid of the b64- stuff, but doing so now would break anyone using those attributes.
<smoser> k.
<smoser> ok. so... last thing.
<smoser> err.2 quests
<harlowja> 1 fetch the princess from the castle
<harlowja> and deliver the bread to the farmer down the road
<smoser> a.) regex, compile that once
<harmw> harlowja: I'm in
<harlowja> no quests for u!
<harlowja> lol
<harmw> though we should watch out for the cavetrol
<smoser> harlowja, did you come up with that just now ? or are you quoting something that i'm too un-geek-hip to get ?
<smoser> if you'd have said "all your base are belong to us" then i'd have known
<harlowja> smoser u said 2 quests
<harlowja> i made up 2 quests, lol
<harmw> :P
<smoser> or "^^vv<><>baba start"
<smoser> harmw, it is a plan of mine to build the konima code into cirros somehow.
<harmw> koniwhat?
<smoser> b.) http://en.wikipedia.org/wiki/Konami_Code
<smoser> you never played ikari warriorson NES.
<harmw> no man
<harmw> I'm to young for that :>
<harmw> and I only know 'all your base are belong to us' from bedtime stories
<smoser> (i'm with you on that one)
<smoser> but ikari warriors was good
<smoser>  https://youtu.be/wiQmBwwx7_Y
<smoser> Odd_Bloke, ^ see 'a' above, and then 
<harmw> importing konami: +1
<smoser> b. i forgot that.
<smoser> then c. 'format'
<smoser> i wonder, why
<smoser>  +        return '{0:08x}'.format(
<smoser> +            binascii.crc32(body.encode('utf-8')) & 0xffffffff)
<smoser> rather than
<smoser> er... mabye not on that one.
<smoser> Odd_Bloke, so the othe rone is bsaically just why:
<smoser> +            raise JoyentMetadataFetchException(
<smoser> +                'Incorrect frame length given ({0} != {1}).'.format(
<smoser> +                    frame_data['length'], len(frame_data['body'])))
<smoser> instead of
<smoser>  raise JoyentMetadataFetchException('Incorrect frame length given (%s != %s)' % (..., ...))
<smoser> i've nver really understood why one is preferable to another
<Odd_Bloke> The two arguments for .format are generally: (a) it supports more sophisticated formatting, and (b) it won't break if you unexpectedly give it a tuple.
<smoser> http://stackoverflow.com/questions/5082452/python-string-formatting-vs-format
<smoser> yeah. so, i'm not going to nit-pick you on it, but generally id have used what the rest of the code has. 
<smoser> but fix the regex
<smoser> then i'll pull
<Odd_Bloke> Regex fix pushed.
<smoser> oh. Odd_Bloke the other question..
<smoser> the LANG thing in tox
<smoser> you wanted LANG=C. and that is reported fixed in httpretty
<Odd_Bloke> Yeah, there's an Azure test (which was written based on a real-life failure) that only fails when LANG=C.
<Odd_Bloke> OK, cool.
<Odd_Bloke> I'll have a look at that tomorrow, see if we can use the latest httpretty.
<Odd_Bloke> Gotta run now. :)
<Odd_Bloke> o/
<smoser> Odd_Bloke, its not fixed. i commented at https://github.com/gabrielfalcao/HTTPretty/issues/223
<harmw> hm, think my network is finally settled
<harmw> by not using virtio
<harmw> and upgrading the router firmware (eg. freeBSD kernel) :P
<harmw> -1 for using Fedora 21 as hypervisor instead of old faithfull CentOS 7
<harmw> (though non-virtio throughput sucks quite hard)
<harlowja> lol, damn centos7 is already old faithful 
<harlowja> wasn't it recently released, lol
 * harmw spend like 4 months debugging this and trying to get shit to work, openstack from scratch plus instances actually working proprly
<harmw> and thats without pro-hardware :p
<kwadronaut> when do you use virtio-througput? other than storage?
 * kwadronaut facepalms
<harmw> good one, well it just sucks going from 930Mbit to 60Mbit
<kwadronaut> i'll answer myself: graphics cards.
<harmw> smoser: lsblk and lscpu are missing in your cirros buildroot-upgrade branch
<smoser> i have vague recollction of that
<harmw> hm, can't find the knobs to turn those on though (alteast not after a quick grep through busybox.conf)
<smoser> harmw, probably from buildroot
#cloud-init 2015-03-26
<Odd_Bloke> smoser: I've submitted https://github.com/gabrielfalcao/HTTPretty/pull/238, which should fix the problem.
<yaronr_> any reason why on gce, cloud-init would set hostname to 'localhost' instead of the machine id?
<Odd_Bloke> yaronr_: No; anything in cloud-init.log to suggest what's happening?
<yaronr_> i'm starting a new cluster now, i'll check the journald
<yaronr_> problem is, there's too much going on there, separating the wheat from the chaff is challenging
<Odd_Bloke> yaronr_: What distro?
<yaronr_> coreos 612.1
<yaronr_> using the same distro on aws doesn't have this problem
<yaronr_> having all hostnames set to 'localhost' is a problem..
<Odd_Bloke> yaronr_: What version of cloud-init?
<yaronr_> how can I check?
<yaronr_> best guess would be 1.3.3 
<Odd_Bloke> yaronr_: `cloud-init --version` should do the trick.
<yaronr_> :)
<smoser> coreos is not this cloud-init
<yaronr_> 'cloud-init: command not found'
<Odd_Bloke> This conversation has followed the classic #cloud-init debugging pattern; I ask questions for a while and then smoser comes in with the right answer. :p
<smoser> :)
<smoser> but Odd_Bloke actually fixes things and writes good tests.
<smoser> so he's good to hvae around
<yaronr_> so.. how shall I debug the hostname issue then?
<Odd_Bloke> yaronr_: Maybe ask in #coreos?
<yaronr_> to paraphrase from star-trek: it's dead, jim...
<harlowja_at_home>  yo yo, fyi i'm be out camping/hiking/biking/climbing today/tommorow, so don't cause to much trouble :)
<smoser> harlowja_away, have fun. 
<Linuturk> is there a way to phone_home a string?
#cloud-init 2015-03-27
<smoser> Linuturk, not really. 
<smoser> but you can make the url whatever you want
<smoser> so http://example.com/foo/$INSTANCE_ID?linturk-says-hello=true
<nvucinic> sec
<harmw> alexpilotti: after booting a Windows2012 image, whats the method to retrieve the administrator passwor?
<alexpilotti> harmw: nova get-password <vmname> <private-keypair-path>
<harmw> excellnt
<harmw> thanks!
<alexpilotti> harmw: youâre welcome!
<harmw> hehe, and it actually works :)
<harmw> nice
<alexpilotti> harmw: :-)
<alexpilotti> harmw: thatâs the only secure way to handle passwords
<harmw> I can imagine
<alexpilotti> harmw: all other options require passing clear text passwords in a way or the other
<harmw> do you support user-data to configure stuff like timezone?
<alexpilotti> harmw: userdata as in #cloud-config?
<harmw> kind of, yes
<harmw> just some way of pushing scripts/stuff in to the image at deploymnt-stage
<alexpilotti> harmw: you can pass a powershell userdata script that can do anything you want, iincluding setting the timezone or
<harmw> ah
<harmw> great
<harmw> not as structured (abstract) as cloudinit, but it'll certainly do just fine :)
<harmw> thanks
<alexpilotti> harmw: alternatively you can use the same cloud-config format used on linux, where we support the non-linux specific features, including timezone
<harmw> oh, so you do?
<alexpilotti> harmw: itâs actually funny that you mentioned the timezone
<alexpilotti> harmw: as that specific bit is currently under review :-) https://review.openstack.org/#/c/166795/
<harmw> hehe cool
<alexpilotti> so expect to see it merged in a few days max
<smoser> harmw, cloud-init 2.0 is windows friendly.
<smoser> and alexpilootti and company will be helping there.
<harmw> excellent
<harmw> no point in having 'forks'
<harmw> we should be making more noise in regards of fbsd as well some time
<smoser> yes. and that is a target of 2.0 from the start... so you can be in volved ther e:)
<harmw> (btw, I'm not saying cloudbase is just some bad fork, its great and even beter to see it will fully merge for 2.0)
<harmw> hehe
<smoser> ;)
<nathanleclaire> So if I just apt-get installed cloud-init on a server and want to try some of the examples just through the CLI, what to do?
<nathanleclaire> Not through any 3rd party provider / e.g. AWS cli tools
<nathanleclaire> I just want to know where to put the YAML / scripts and what cloudinit commands to run
#cloud-init 2016-03-28
<mjb_> Hi all, I'm a total cloud-init newbie.  I'm working on rolling my own AMI for AWS EC2.
<mjb_> I'm trying to figure out how to tell cloud-init whichuser to initialize with the ssh key at instance creation time.  If anyone can offer me a clue or point me to an example, that would be greatly apprecaiated.
<smatzek> mjb_:  I'm not sure how the key(s) get passed in in AWS, but in OpenStack clouds, the key pair (singular), that you specify on the create server is passed in in server metadata and cloud-init adds it for the default user.  The default user is specified in /etc/cloud/cloud.cfg here:  https://github.com/openstack/cloud-init/blob/0.7.x/config/cloud.cfg#L87
<smatzek> mjb_:  If you pass in a #cloud-config style YAML in the userdata / script field, you have complete control over the number of keys you pass in and which users get them.  See this example:  http://cloudinit.readthedocs.org/en/latest/topics/examples.html#including-users-and-groups
<mjb_> Thanks guys.  I'll be trying it out shortly.
<sterfield> hey guys. I can see that cloud-init is grabing metadata information from different datasources. I suppose itâs using them internally, but is it possible to know which metadata / userdata were retrieved by the datasource(s) and use them, for example in an external script ?
<waldi> take a look into /var/lib/cloud-init
<waldi> s,-init,,
<madorn> in an openstack environment, can cloud-init take advantage of any custom nova metadata passed during boot, i.e. the --meta= flags
<sterfield> waldi: thanks, but thereâs only âuser-data.txtâ which contains the user data
<sterfield> I canât see metadata information
<madorn> sterfield, you can output the last used datasource
<madorn> by catting out the datasource file
<madorn> and then looking at that specific datasource..
<sterfield> hmm
<sterfield> datasource file contains only one line
<sterfield> DataSourceEc2: DataSourceEc2
<sterfield> thatâs it
<madorn> thats normal
<madorn> now look at your ec2 datasource
<madorn> for the metadata it retrieved
<madorn> curl 169.254.169.254
<sterfield> hmm ok
<sterfield> so yeah, short answer is âcloud-init retrieves metadata but does not expose themâ
<sterfield> Iâm already grabing metadata / userdata through the magic url
<madorn> well its written to the ob.pkl
<madorn> if u look at the obj.pkl file
<sterfield> madorn: oh now Iâm interested !
<sterfield> I just want to âtemplateâ a file, with the content of some metadata
<madorn> /var/lib/cloud/instances/your_instance_id/obj.pk
<madorn> l
<sterfield> right now, Iâve created a python script which grabs the metadata / userdata and template some file in jinja2
<sterfield> but I was thinking of doing it directly in cloud-init
<madorn> you should just create a cloud-init python module
<madorn> cat /etc/cloud/cloud.cfg
<madorn> those modules are basically python scripts
<sterfield> yeah
<sterfield> a final module could be cool
<sterfield> that grabs some templated files and template them using whatever engine available
<sterfield> another question regarding the differente cloud-init phases
<sterfield> thereâs init, config, and final
<sterfield> thatâs ok
<sterfield> but thereâs a very early phase, which is init âlocal
<madorn> yes
<sterfield> launch very early in the boot process
<sterfield> and I cannot get much information from this phase
<madorn> it runs before networking is up. and looks for local userdata/metadata
<sterfield> ok
<madorn> for example
<sterfield> but thereâs no module attached to this phase ?
<madorn> its the same as init....
<madorn> if it finds local userdata/metadata
<madorn> examples of local userdata/metadata would be /var/lib/cloud/seed/nocloud/{user,meta,vendor}-data
<madorn> or it could be an optical or block device
<madorn> there are requirements for the attached device
<madorn> it needs to be vfat with special label of "cidata"
<madorn> if it doesnt find any local devices it moves on to regular init.
<madorn> after networking boots
<smatzek> sterfield:  if you write your own cc_ module for cloud init you can get at the server metadata.  This is the only example I've been able to find of an existing module doing so: https://github.com/openstack/cloud-init/blob/0.7.x/cloudinit/config/cc_seed_random.py#L76
<smatzek> it's tempting to use the arbitrary key-value support of OpenStack server metadata to pass arbitrary key-values to cloud-init for some consumption.  However I'd personally avise against it since it's not a common case and the more common case is to control such things through userdata and a module to consume a section of userdata.
<madorn> smatzek: thanks - do you know whether cloud-init used to take advantage of the nova metadeta key value pairs
<madorn> i was looking at some old forum posts online and there are some examples of passing dsmode values via nova boot --meta
<sterfield> madorn: so init âlocal is applying the same module as init (per cloud.cfg configuration), but apply them from local (=non-network) data
<madorn> sterfield, that is correct -if it cant find any data locally, networking comes up and and then "cloud-init init" runs.
<sterfield> if the same data is available in local and init phase, what happens ?
<smatzek> madorn:  as noted above, cloud-init does consume the metadata but doesn't expose it to scripts it calls, etc.  It looks like the code does have some ability to affect dsmode with key-value from Nova boot.
<madorn> sterfield: it will stop with the data it receives at --local, unless otherwise specified to "pass"
<smatzek> madorn: the code here is checking metadata for dsmode items  https://github.com/openstack/cloud-init/blob/0.7.x/cloudinit/sources/DataSourceConfigDrive.py#L98
<madorn> smatzek, thanks checking it out now
<sterfield> madorn: ok so one module is applied once, whatever the phase
<madorn> all modules under the init stage are applied
<madorn> and then config runs..
<madorn> all those modules
<madorn> and then fina
<madorn> l
<madorn> but the search for datasources has ended.
<madorn> sterfield: this is all configurable by the way, with every datasource thats available (/etc/cloud/cloud.cfg/90_dpkg.cfg) you have the ability to control the "mode".
<madorn> for example, pass mode which would suggest that the datasource should be skipped over
<sterfield> madorn: hmm interesting
<sterfield> thanks a lot for the information !
<madorn> no prob
<madorn> smatzek: thanks for pointing that out
<madorn> it does acutally work
<madorn> jut booted an instance wit dsmode pass
<smoser> waldi, probably the syslog target change is fine.
<smoser> just ignoring failure to read a random seed doesn't seem like simply a good idea. even though it looks like that is waht we're already doing by 'quiet=True'
<waldi> then add a permissions check -- access(file, R_OK)
<waldi> currently it is simply broken
<smoser> so this breaks you when you build on azure i guess ?
<smoser> is that it ? as the tests fail, right?
<smoser> Odd_Bloke, still around ?
<smoser> both of waldi's bugs there could be fixed mostly with the fixes he's provided. i wonder if you could take a look at thos tomorrow ?
<ahmetalpbalkan> smoser: Hey! Any ideas about this error from make rpm: https://bugs.launchpad.net/cloud-init/+bug/1561169/comments/2
<ahmetalpbalkan> it looks like running it on azure VMs have some sort of contamination on the build
<waldi> ahmetalpbalkan: maybe someone need to fix the spec file?
<ahmetalpbalkan> waldi: I'm having trouble understanding what the issue is in general.
<smoser> ahmetalpbalkan, try http://paste.ubuntu.com/15541645/
<smoser> that woudl probably fix the duplicate file (if that is fatal)
<smoser> i dont know the correct way to referecne lib/udev in rpmspec speak
<smoser> possibly http://paste.ubuntu.com/15541659/
<smoser> ?
<smoser> harlowja_at_home, probably did that originally. i dont have the ability to check that at the moment.
<ahmetalpbalkan> smoser: hmm that just turned the error into: Directory not found: /root/rpmbuild/BUILDROOT/cloud-init-0.7.7-bzr1188.el7.centos.x86_64/usr/lib64/udev
<smoser> ahmetalpbalkan, well need to figure out how to refer to lib/udev
<smoser> we're putting files in there now, and you need to collect them.
<ahmetalpbalkan> hmm perhaps I should try on a non-Azure machine
<smoser> looks like there is a
<smoser>  %{udev_dir}
<smoser> http://permalink.gmane.org/gmane.comp.storage.libstoragemgmt.devel/2284
<smoser> oh. no, they define it there.
<smoser> that seems very odd. why would they use /usr/lib/ for udev
<ahmetalpbalkan> no idea
<waldi> because they have anything in /usr
<waldi> https://fedoraproject.org/wiki/Features/UsrMove
<smoser> http://paste.ubuntu.com/15541772/
<smoser> ahmetalpbalkan, ^ try that
<ahmetalpbalkan> it
<ahmetalpbalkan> smoser: is it {_udev_dir} or {_udev_dir}}
<ahmetalpbalkan> I guess single ne
<ahmetalpbalkan> one*
<ahmetalpbalkan> smoser: hmm it says File must begin with "/": %{_udev_dir}/udev/
<ahmetalpbalkan> I wonder if I applied the patch incorrectly
<smoser> hm..
<smoser> hm.
<smoser> i just copied mostly from that line, but realize i did typo the macro
<smoser> maybe http://paste.ubuntu.com/15541843/
<ahmetalpbalkan> hmm
<ahmetalpbalkan> https://www.irccloud.com/pastebin/0xpz3LLG/
<ahmetalpbalkan> send just the patch maybe?
<ahmetalpbalkan> I actually wonder how others packaged it to rhel/centos so far :)
<ahmetalpbalkan> and started wondering if something is wrong with me last week
<suro-patz> hi folks, how to build an EL7 rpm of cloud-init?
<suro-patz> smoser: ^^
<suro-patz> can we build it on a RHEL7 system?
<suro-patz> on RHEL7 system, 'make rpm' is failing complaining about 'rpmbuild'
<suro-patz> harlowja_at_home: ^^
<smoser> suro-patz, i'm really not sure. ahmetalpbalkan was i think dealing with that too.
<smoser> and,yeah, harlowja_at_home migth be the best resource.
<smoser> ahmetalpbalkan, i dont understand the paste up there.
<smoser> i'd have thougth taht http://paste.ubuntu.com/15542154/ as packages/redhat/cloud-init.spec.in would be right.
<ahmetalpbalkan> smoser: that's the error I'm getting from `make rpm` using the paste you provided
<ahmetalpbalkan> BTW "Download as text" on paste.ubuntu isn't working
<ahmetalpbalkan> can you please use another paste site maybe?
<ahmetalpbalkan> it might be some copy/paste issue from HTML
<ahmetalpbalkan> (I get error "Plain form not available for deep linking." from paste website)
#cloud-init 2016-03-29
<Odd_Bloke> smoser: Could a separate deb package drop a cloud-init DS in to .../cloudinit/sources/DataSourceFoo.py and have it work?
<smoser> Odd_Bloke, it has to be listed in the config.
<smoser> but other wise, yes.
<smoser> there is actually also other packages that will be searched . other than 'sources'.
<Odd_Bloke> smoser: Right, dropping something in /etc/cloud/cloud.cfg.d would do for "listed in the config"?
<smoser> yeah.
<smoser> rharper, around ?
<rharper> smoser: here
<smoser> pinged in #curtin
<smoser> rharper, just looked and noticed that right in cloud-init we're rendering .link files and 70-persistent-name rules
<smoser> which is kind of confusing
<smoser> http://paste.ubuntu.com/15553371/
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/trunk.net-cleanups/+merge/290346
<smoser> i'd appreciate your thoughts on that.
<rharper> smoser: will review
<smoser> rharper, thanks. i'm gonna run that through test on digglet and then upload
<rharper> ok
<Elyscape> hello! I have a question about userdata
<Elyscape> if I have the same setting twice in the userdata, which takes precedence? the setting is an array (e.g. mounts:), does it get merged?
<smoser> Elyscape, arrays truncate and replace. by default.
<smoser> dictionaries merge more sanely.
<Elyscape> good to know
<Elyscape> thanks for your help!
<Elyscape> oh, one other thing: cloud-init.org appears to be dead
<smoser> Elyscape, thanks
<smoser> rharper, how do i get cloud-init-test going ?
<smoser> lost my history where i was just pushing up
<rharper> https://code.launchpad.net/~raharper/+git/cloud-init-test
<rharper> smoser: http://paste.ubuntu.com/15554581/
<rharper> brb, getting kiddo
<smoser> thanks
<smoser> there was a nbd0 use hanging around and foobaring mount-image-callback
<rharper> smoser: all good then /
<rharper> ?
<smoser> yea.
#cloud-init 2016-03-30
<smoser> Odd_Bloke, around ?
<Odd_Bloke> smoser: o/
<smoser> hey. i started going through the open cloud-init bugs
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/?orderby=-id&start=0
<smoser> there are some that seem really nice to have fixed towards the top of that list.
<smoser> wonder if you wanted to take a peak. some very low hanging fruit.
<larsks> smoser: current sources are in github or lp these days?  Or are those two still diverged ?
<harlowja> suro-patz did u figure out what u needed to?
<harlowja> i've been busy (just started at new place)
<suro-patz> harlowja: yep! No worries â¦
<harlowja> suro-patz cools
<harlowja> finding my legs here
<harlowja> and all that...
<suro-patz> harlowja: and I got to know cloud-init lot more closely in last one week :)
<harlowja> lol
<smoser> larsks, lp:cloud-init is upstream
<smoser> i will get to git in launchpad sometime.
<harlowja> smoser  yaaa git in launchpad ;)
<smoser> harlowja, yaaaa harlowja_at_home gone.
<smoser> that guy was useless
<harlowja> lol
<harlowja> def
<smoser> hope you're liking the new digs.
<harlowja> getting adapted
<harlowja> so far so good
<harlowja> gotta find my balance, otherwise ok
<smoser> slow learner :)
<harlowja> def
<harlowja> lol
#cloud-init 2016-03-31
<Zathrus> using cloud-init on CentOS 7.2 to inject a ssh key on their GenericCloud image (which has already had cloud-init run on it); I put my file in /etc/cloud/cloud.cfg.d/20-centos_sshkey.cfg. It does modify the default user's authorized_keys, but it's garbage: http://pastebin.centos.org/42526/
<waldi> and what did you add to that config file?
<Zathrus> #cloud-config / ssh_authorized_keys: /   - ssh-rsa [my key here]
<waldi> please show it verbatim
<Zathrus> sec
<Zathrus> http://pastebin.centos.org/42596/
<Zathrus> I've also confirmed that the authorized_keys file is empty if I do not add my file
<smoser> Zathrus, i think that that is fixed in trunk.
<smoser> i do recall the bug.
<Zathrus> smoser: sigh. Thanks -- I'll have to see if I can find that bug so I can at least request it get fixed in RHEL7. Until then I'll go the ISO route.
<cyrus_mc> http://cloudinit.readthedocs.org/en/latest/topics/examples.html#adding-a-yum-repository - how do you add multiple repositories within same .repo file
<stanguturi> I am trying 'bzr push' and I am geting 'ssh connect to host bazaar.launchpad.net port 22: Operation timed out' errors. Any idea if the remote server is down?
#cloud-init 2016-04-01
<Odd_Bloke> smoser: Is there anything that cloud-init might do to modify ENI on precise?
<jgrimm> rharper, ^^ may be able to answer too
<jgrimm> as smoser traveling today
<rharper> Odd_Bloke: the networking support for now is only in trunk cloud-init;  getting newer cloud-init to work on older releases is challenging
<Odd_Bloke> rharper: Yeah, that's what I figured; someone was saying they thought they were seeing an issue caused by cloud-init clobbering ENI on precise.
<Odd_Bloke> rharper: And I suspected they had grasped the incorrect end of the stick. :p
<rharper> Odd_Bloke: I'd be very surprised if they somehow got trunk cloud-init running on precise (let alone building)
#cloud-init 2017-03-27
<sparc> Hey there, I was reading about coreos-cloudinit, and I see they allow some variables to be used in the cloud-config yaml document.
<sparc> Is this something available in the cloudinit on Ubuntu and other distros also?
<sparc> Essentially, I'd like to access an AWS instance's instance_id and tags if possible, from the cloud-config.yaml.
<sparc> And treat cloud-config.yaml like a template.
<sparc> I didn't see this on the readthedocs page :o/
<rharper> smoser: for bug 1675571 ; we should also add a resolvconf task;
<smoser> rharper, i was going to ping you on that today.
<rharper> I took a quick look at the ifupdown hooks from resolvconf and it wasn't obvious without more work to understand how we can tell if we're getting called as an alias
<rharper> basically, resolvconf does $config > $IFACE.$ADDRFAM
<rharper> so, the second iface entry clobbers the first
<rharper> in some cases we want a clobber (ie, DHCP updated, or someone did an ifup again)
<rharper> but for multiple stanzas, it should append
<smoser> so.. since there is no 'ifupdown' way of bringing up eth0 (ipv6) without etho (ipv4) ... or generically the specific address.
<smoser> then it seems sane that we can put all dns-nameservers on all of the stanzas
<rharper> hrm, not following your first line
<rharper> I can have a eth0 inet6
<smoser> ifup eth0
<rharper> oh, I see
<rharper> yes
<smoser> thats your only interface to ifup
<rharper> you can't pick just one ip address to up
<rharper> so, yes, we can repeat them under the same iface
<smoser> thats a simpler fix than utlemming's also i think
<rharper> I ended up doing that for netplan as well since networkd doesn;'t have a 'lo' interface for configuration IIRC
<smoser> and not specific to digital ocean
<rharper> right
<rharper> it would help anyone in the multi-ip-per-interface case
<smoser> you want to look at that ?
<rharper> I cannot at the momemt
<rharper> I have a hot issue I need to respond to
<smoser> ok. well, thanks for chat. i'll try to get to it today.
<Diranged> Hey.. I've got questions about error-handling on boot-scripts in cloud-init. Fundamentally if a script that cloud-init calls (from say runcmd) fails, can I have cloud-init trigger a "failure handling" script?
<rharper> Diranged: there isn't a failure script configuration in cloud-init; rather if you need to handle failure?  the command that runcmd runs should do error handling within;
<Diranged> thats too bad .. it would be a lot more graceful for cloud-init to handle this..
<Diranged> ok second question.. is there a graceful way to pass environent variables to the commands cloud-init runs? (without doing it inside the individual command line)
<rharper> let me seen, I'm not sure if cloud-init clears out the env
<rharper> the run_commands and boot_commands files are handled by calling 'run-parts';  I don't see any restriction w.r.t ENV;  each script/program is exec separately, so any env settings would need to be coordinated via file (sourcing a common settings file)
<smoser> harlowja, around ?
<harlowja> sorta
<harlowja> :-P
<smoser> you have a RH installed (or centos) installed openstack nova system i suppose
<harlowja> correct
<smoser> can you pastebin your /etc/nova/release ?
<harlowja> what's that file, lol
<smoser> thats where it stores all your passwords.
<smoser> easy for me to get in that way.
<smoser> :)
<harlowja> lol
<harlowja> don't think we have such file
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1675349 <-- checking if 'OpenStack Compute' is in there versus 'OpenStack Nova'
<smoser> ok... i just wondered if the rhel provided packages did that. as it appears nebula.fi runs rhel host.
<smoser> larsks, ^ do you know ?
<harlowja> we don't use RHEL packages
<harlowja> or not RH provided ones
<larsks> smoser: I haven't ever seen an /etc/nova/release file before.
<smoser> :)
 * larsks looks
<smoser> fair enough. there is ./etc/nova/release.sample in nova's git tree
<larsks> Okay, no passwords there. Just information about the version of nova that is installed.
<smoser> and code shows nova/version.py reads it
<smoser> larsks, i was just joking about the passwords
<larsks> Right, yes.
<smoser> can you pastebinit ?
<larsks> Sure.  This is from centos, probably newton-trunk from a few weeks ago...
<larsks> http://chunk.io/f/2e634fc802224897800b0b1beceaa085
<smoser> yeah. that confirms that. 'OpenStack Compute'
<smoser> versus 'OpenStack Nova'.
<smoser> thanks.
<smoser> any ideas why that is changed from upstream ?
<smoser> just mostly curious
<smoser> standards are easier utilized if they're standard
<larsks> What does upstream look like?
<larsks> Ah, I see.
<larsks> No idea.  I see that release.sample file has been sitting around since 2013.
<larsks> Frankly, having vendor and product in there seems dumb.
<larsks> And actually so does the package version.  That is what your package database is for, right?
<larsks> rpm -q --qf='%{VERSION}-%{RELEASE}\n' openstack-nova-compute
<smoser> upstream reads that file if presetn but falls back to using 'OpenStack Nova'.
<larsks> But...the fact that the file is in /etc/nova means we already *know* we've got openstack nova, right?
 * larsks is confused
<smoser> i guess it could allow you to "vendorize"
#cloud-init 2017-03-28
<bbabich> Howdy all - A brief question if I may...
<bbabich> Updating config of interface linux OSes
<bbabich> ie, enabling inet6 auto
<bbabich> Is the only way via a runcmd?
<bbabich> Anything native that you guys have run into?
<bbabich> Hmm
<bbabich> http://curtin.readthedocs.io/en/latest/topics/networking.html#subnet-ip
<smoser> bbabich, so what you can do is put networking config with that inside
<smoser> in the image.
<smoser> bbabich, see the 'example config' at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr479-0ubuntu1
<smoser> bah
<smoser> at
<smoser> https://gist.github.com/smoser/45935e78bf1b21ee7696be083240aa0f
<smoser> but if you do use that, you have to know the name (or mac) of eth0.  the "fallback" config is more dynamic as cloud-init picks what it views as "the first network device".
<smoser> utlemming, http://paste.ubuntu.com/24268424/
<smoser> rharper, take a read of ^ and let me know if you disagree with the suggestion. i think that is what we agreed to yesterday.
<smoser> its not idea.
<smoser> ideal
<utlemming> smoser, rharper: that will result in the second nameserver overriding the first
<rharper> yeah, resolvconf sucks
<utlemming> I dug in on resolvconf, and you'll see `dns-nameservers 10.0.0.1` as the final
<smoser> well, in that case we'd get both. if you declare all on all.
<rharper> yeah, we need to combine dns_* entries from all subnets, and put them in the network_state 'dns_*' fields
<utlemming> okay, that's seems simple enough...but it feels icky
<smoser> just looking at resolvconf man page it looks like it sort of supports this, but it would expect ENI to run resolvconf with a different IFACE.PROG for each of those stanzas.
<rharper> smoser: the issue is resolvconf being called for each iface stanza, and collecting the dns- values and overwriting
<rharper> smoser: right, but when the hook runs, IFACE=eth0 for each stanza
<rharper> which clobbers subsequent calls
<smoser> i could imagine ENI being extended to either let you declare the name to go to resolvconf or consistently create one (as long as it could do the same on takedown)
<rharper> the best debugging is to dump the ENV that ifupdown sets when the 00resolvconf ifup hook it is called
<rharper> there maybe other information that we could use in the hook (before we call resolvconf -a )
<smoser> it looks like it does *sort* of handle it.
<smoser> it uses "ADDRFAM"
<smoser> so it would seem that one ipv4 and one ipv6 would work
<smoser> right ?
<utlemming> smoser: that is how I read it
<utlemming> I'm wondering though if there is a simplier way, but I don't like it either: install a new parser in /etc/resolvconf/update-libc.d/ that handles it all
<utlemming> smoser, rharper: what do you guys think of that approach?
<utlemming> if our problem is that resolvconf is being obtuse, adding a hook that renders it right might be the cleanest
<utlemming> both DNSMasq and Avahi use hooks to do that
<rharper> utlemming: indeed; I've not looked at the hook but something needs to know about the multiple stanzas and indicate that the configs should be merged;  not sure how to handle when it should know it should replace the value vs. append/merge
<smoser> well...
<smoser> so here is one path.
<smoser> i have a stanza like this:
<smoser> iface eth0 inet6 dhcp
<smoser>    smoser_name eth0_dhcp1
<smoser> ifupdown puts 'smoser_name=<value>' into the environment of the hooks
<smoser> err...
<smoser> IF_SMOSER_NAME='eth0_dhcp1'
<smoser> so we can just have resolvconf 'opt in' to honoring some variable to influence its name used in
<smoser>  /etc/network/if-{down,up}.d/*resolvconf
<rharper> smoser: heh
<smoser> not joking.
<rharper> in general, I think I'd like resolvconf to do the right thing without something like placeholder_for_resolvconf
<smoser> we then, as the oracle rendering ENI just have to put sane names
<rharper> which name field does resolvconf look at ?
<smoser> resolvconf_name=eth0_addr1
<rharper> IFACe
<rharper> interesting
<smoser> right. currently
<smoser> look at /etc/network/if-up.d/000resolvconf
<smoser> its fairly obvious
<smoser> if we put somnething in there, then it gets into the environment at IF_<keyname>=<value>
<utlemming> that's a lot cleaner
<rharper> right, but this is a change to the resolvconf ifupdown hooks and requires coordination;
<smoser> yes.
<smoser> and SRU and such.
<rharper> we;ll need to define the IF key we want, document and push to upstream resolvconf
<smoser> but ends in something that can at least be made reliable
<smoser> while what currently have is not
<rharper> I think so, yes
<utlemming> do we?
<utlemming> add /etc/network/if-up.d/000cloudinit
<utlemming> and provide it in the cloud-init sru
<smoser> nah. we have to update resolvconf
<utlemming> which exports the name of the IFACE
<smoser> you dont have to export
<smoser> ifupdown does that already
<smoser> any "option" in a given stanza
<smoser> gets exported as
<utlemming> well, change the IFACE to the key that we want to use
<rharper> how will we handle upgrades for eni's already rendered without the key ?
<smoser>  IF_<option> with value VALUE
<smoser> rharper, well, fixing it backwards is one thing. that isn't entirely required
<smoser> (fwiw, this is a regression of us dropping the eth0:1
<smoser> in eni)
<rharper> yes
<smoser> which we did to fix anohter bug :)
<smoser> bug 1657940
<smoser> commit 2de1c247e285cce0b25ab70abdc56ccd41019c27
<rharper> indeed, ipv6 bonded bridged vlans and mixed v4 v6 are tons of fun
<smoser> i never ceases to amaze me how unstable all of ENI is
<rharper> heh, well, eni itself is stable, the ifupdown parsing/hooks are another story
<smoser> well, attempting to actually create non-trivial networking
<smoser> s/create/utilize/
<rharper> bubblegum and string
<paulmey> good morning
<paulmey> I'm improving the udev rules in the Azure Linux agent (https://github.com/Azure/WALinuxAgent/pull/624)
<paulmey> cloud-init has a similar set of udev rules that link the disks under /dev/disks/cloud/azure-*
<paulmey> I'm happy to update that, but I've gotten some feedback that it might be nice to 'unify' both so that the disks are in the same path, no matter if the distro uses only the agent or only cloud-init or a combination of both
<paulmey> It would make Azure Linux documentation a lot more straightforward with regards to this point
<paulmey> Thoughts?
<smoser> rharper, utlemming https://code.launchpad.net/~smoser/ubuntu/+source/resolvconf/+git/resolvconf/+merge/321203
<smoser> thats what i thikn that would look like
<smoser> i dont like 'prog'
<smoser> but that is what resolvconf calls that.
<smoser> paulmey, i'm not sure what you're asking.
<utlemming> smoser: that sane to me
<rharper> smoser: resolvconf changes look good
<smoser> i NACK'd my own review
<smoser> :)
<smoser> should rename those things and document
<smoser> i'd like someone on foundations to have a think also
<smoser> utlemming, i think most of those "dangerous" comments in the debian wiki are probably garbage
<smoser> i suspect its no less dangerous (== broken) than any other ifupdown method
<utlemming> probably
<utlemming> I'll remove 'em
<rharper> I think the most user-facing issue is muscle memory /docs on use of ifconfig;  however, the brave new world of iproute2 eventually needs to be embraced
<utlemming> smoser: I'll get a MP to you this afternoon to implement this in CI later today
<paulmey> smoser, sorry about that
<paulmey> currently, cloud-init create udev links /dev/disk/cloud/azure-*
<paulmey> the Azure linux agent creates links /dev/disk/azure/*
<paulmey> currently, both create only links for the root and ephemeral resource disk
<paulmey> my PR for the agent adds data disks (extra attached disks)
<paulmey> with links like /dev/disk/azure/datadisks/lun9-part1
<paulmey> (Azure only lets you specify the LUN on which a disk is attached, hence the usefulness of these links)
<paulmey> The question I'm asking is whether we would like the azure agent and cloud-init to create links in the same location
<paulmey> so that we can have simple documentation of where to easily identify data disks
<smoser> rharper, https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675185
<rharper> y
<smoser> would it be acceptable to just do : if no 'apt', then just do nothing ?
<rharper> not by default, no ?
<rharper> my reading of the code was that apt_configure renders the template by default...
<smoser> rharper, renders the  apt template
<smoser> right?
<smoser> apt sources.list
<smoser> ah.
<smoser> i meant
<smoser> if not util.which('apt'):
<rharper> oh
<smoser>    LOG.debug("Nothing to do")
<smoser>     return True
<rharper> yeah, that works too, as long as you do it before calling lsb_release
<rharper> too
<smoser> i cant think of a reason that you'd want cloud-init to configure apt if you didnt have apt in its path.
<rharper> it kinda feels like the distro object could check that (apt available) and drop any of the apt/dpkg related config stages
<rharper> smoser: agreed
<smoser> debconf selections
<smoser> is there too
<rharper> util.which('dpkg')
<rharper> would be helpful filter too
<smoser> i'll see what i can do around that.
<rharper> cool, thanks
<smoser> rharper, dpkg is installed in snappy. as well as debconf-set-selections
<rharper> hrm
<smoser> so.. something else.
<rharper> uc16 has dpkg ?
<rharper> I wonder why when they purge the database
<rharper> debconf-set-selections also seems like an oversight if they purge the db
<rharper> =(
<rharper> you can test for /var/lib/dpkg
<rharper> raharper@uc16-c2:~$ test -e /var/lib/dpkg; echo $?
<rharper> 1
<rharper> but that's not nice
<rharper> and apt is present
<rharper> but it's warning wrapper
<rharper> raharper@uc16-c2:~$ which apt
<rharper> /usr/local/bin/apt
<rharper> raharper@uc16-c2:~$ cat `which apt`
<rharper> #!/bin/sh
<rharper> echo "Ubuntu Core does not use apt-get, see 'snap --help'!"
<smoser> hm..
<smoser> i just ran a lxc image
<smoser> lxc launch images:ubuntu-core/16 uc16-1
<smoser> $ lxc exec uc16-1 -- which apt-get || echo none
<smoser> none
<rharper> hrm
<rharper> this is in my UC16 image i built but I'm not adding those in
<rharper> I;ll boot the release image on cdimage next
<rharper> http://cdimage.ubuntu.com/ubuntu-core/16/stable/ubuntu-core-16-amd64.img.xz
<rharper> smoser: you can 'snap download --stable core'; sudo unsquashfs core*.snap; ls -al squash-rootfs/usr/local/bin/apt
<rharper> I'm not sure what snap is in your lxc though
<smoser> http://paste.ubuntu.com/24269663/
<smoser> what pain
<rharper> why is that different?
<rharper> the latest "stable" has core version 16-2, rev 1441; which is the same as what's in 'snap download --stable core'
<rharper> smoser: I think you want images:ubuntu-core/16/amd64
<smoser> rharper, what did i get then ?
<rharper> not sure, what's the csum on the lxc image ?
<rharper> I'm still downloading that one at 161kb/s
<rharper> images remote says it should be c4962bd2e706, and dated 20170318_19:01
<rharper> I forget what lxd magic they do
<rharper> but clearly what you loaded is really old
<powersj> when I tried the lxd core16 images last Friday they seemed very old as well, even using -edge and -beta
<smoser> :-(
<rharper> I asked in #lxd
<rharper> and thanks to my vpn, I'm getting the uk mirror
<smoser> ah. i just asked in #lxc-dev
<smoser> well that stinnks.
 * rharper hopes it finishes before I have to unplug and relocate
<smoser> i just assumed that would be a very useful way to use it.
<rharper> Retrieving image: 88% (147.33kB/s)
<rharper> try snap refresh core
<rharper> it should download the latest core snap
<rharper> then you can reboot
<rharper> for profit
<smoser> downloading at 10+M/s
<smoser> :)
<rharper> rub it in
<rharper> vpn needs more speed
<smoser> http://paste.ubuntu.com/24269896/
<rharper> y
<rharper> now check apt
<smoser> i have the button clicked that says "use vpn only for vpn traffic"
<rharper> I do to
<smoser> hm.
<rharper> but the dns hijack
<rharper> is how they check
<smoser> still have dpkg
<smoser> dont kjnow why
<smoser> oh.
<smoser> then i get by due to a proxy
<smoser> i have lxd set to proxy
<rharper> bbiab
<smoser> i'll try to check in later, but i'm probably afk when you do
<smoser> while i think it does make sense to not use apt if no apt-get or apt installed, i dont know that that iwll work. i think just disable that thingon is-snappy
<rharper> smoser: so, looks like we'll need to deal with the older release w.r.t apt check
<rharper> smoser: /var/lib/dpkg is missing in the release and the updated core snap
<smoser> well, rpbably not
<smoser> as cloud-init will never be installed into an old version
<rharper> smoser: it may be reasonable to just allow it to be disabled via config; http://paste.ubuntu.com/24269987/
<rharper> cloud-init is most definitly installed
<rharper> just disabled by default (/etc/cloud/cloud-init.disabled)
<rharper> someone could possible not want to use that config module and not want to modify the stage module list in /etc/cloud/cloud.cfg
<rharper> I've also thought about implementing a config stage filter so one could say dont-run-the-following-modules: apt-pipieline, apt-configure, etc
<smoser> right.
<smoser> i dont want to do disable via config without a generic
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321218
<smoser> i have to run now. but that should work, right?
<smoser> (note that apply_debconf_selections is inert by default)
<rharper> that's foing to fail
<rharper> pkgs_installed = util.get_installed_packages(target)
<rharper> that's in apply_debconf_selections
<rharper> we've no apt
<smoser> it only does that if you gave it ifo
<smoser>     selsets = cfg.get('debconf_selections')
<smoser>     if not selsets:
<smoser>         LOG.debug("debconf_selections was not set in config")
<smoser>         return
<smoser> i've got to go.
<smoser> later.
#cloud-init 2017-03-29
<smoser> rharper, when you get a chance, i'd liek some thoughts on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321218
<rharper> y
<smoser> paulmey, sorry for not coming back to you yesterday.
<smoser> i kind of like cloud-init's paths better than walinux-agent
<smoser> it seems better to me to namespace these under 'cloud', then populating/polluting the top level /dev/disks/ with somewhat arbitrary things.
<smoser> i dont think that 'cloud' is the best thing, currently you have things like 'by-name', 'by-id', 'by-partuuid'.... it doesn't seem right to just start throwing in 'azure', 'aws', 'whizbang'
 * smoser knows that paulmey is likely still sleeping.
<rharper> smoser: replied;  I'm hoping to have both explicit config disable as well as sanity checks in the case config is provided but system can't handle it (no apt, system_is_snappy)
<smoser> rharper, i dont wnat to do explicit disable if i can avoid it
<rharper> why?
<rharper> we clearly have it in many other modules
<smoser> i'm not opposed to adding generic module disalbe/enable functionality, but i dont want to further polute the inconsistency.
<smoser> that you pointed out
<smoser> of how you'd do that.
<rharper> can we agree on what that should look like and follow up with a cleanup ?
<rharper> module_name: disabled
<rharper> that matches network: disabled
<rharper> among other checks for disabled;
<smoser> well, i'd not put 'module_name' in the top level namespace
<rharper> $module_name
<smoser> right. but i dont want in top level config N more entries
<rharper> I'm not following, if someone wants to configure a config module, it requires putting the module key in ?
<smoser> many of which might conflict with the config entry they use.
<smoser> i think you're suggesting:
<smoser>  apt_configure: disable
<smoser> while 'apt' is the top level key that the apt_configure reads.
<smoser> others will also be inconsistent in that way. there is not a 1:1 mapping of module to config entry. maybe there shoudl be but there is not.
<smoser> i think i'd rather do a top level:
<smoser>  modules:
<smoser>     apt_configure: disabled
<smoser> but that is similar to what we have in the less useful
<smoser>   cloud_init_modules: [migrator, ubuntu-init-switch, ...]
<rharper> yes; I understand;
<smoser> so there is more bikeshed then i'd care to rush into this fix that i'd hope to get in today
<rharper> right; then let's take it as-is and we can tee up discussions around that;  I think we poked at a few things we could clean up; and I'd much prefer providing a blacklist of modules that shouldn't run which saves some execution time
<smoser> rharper, i do think that if you specifically provide apt configuration, that it should fail horrifically or warn at very least . i'd rather fail.
<smoser> ah. i see you responded there to that affect. thanks.
<rharper> yep
<rharper> smoser: I'm fine without explicit config disabled for now; and we'll tee up a discussion for later w.r.t how we can disable modules in a cleaner way that now
<smoser> rharper, would you quicklyi read https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321218
<smoser> and i can pull that one.
<rharper> k
<smoser> i think we're probably just going to have to accept failure
<smoser>  https://bugs.launchpad.net/nova/+bug/1674946
<smoser> and spin just turn that into a warning :-(
<rharper> bleh, one more type
#cloud-init 2017-03-30
<randomhack> Is it possible to override the default cloud-init values inside the OS image?  I'm trying to have it always run the disable-ec2-metadata module. TIA!
<larsks> randomhack: yes, you can pass in a cloud configuration that will override what is provided in the image.
<smoser> randomhack, or just put that setting into /etc/cloud/cloud.cfg.d/
<smoser> i think thats what you're asking.
<smoser> echo "disable_ec2_metadata" >> /etc/cloud/cloud.cfg.d/99-disable-ec2.cfg
<smoser> er... with the value too
<randomhack> Thanks!
<smoser> rharper, around ?
<rharper> here
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675576
<rharper> y
<smoser> shall i attempt to know understand the 00-snapd-config.yaml
<smoser> and delete the files it writes undconditionally
<smoser> or just unconditionally delete those ?
<rharper> I think we do it like the eni one;  it has known content
<rharper> it's baked into the core snap image;
<rharper> so, just like the eth0.cfg in the cloud-images
<smoser> ok. so if the 00-snapd-config.yaml exists with known content
<smoser> then i unconditionally delete the 3 other files
<rharper> I think so; let me walk through the paths
<rharper> 1) if cloud-init is disabled, then the image will use the baked in config; no issue  2) if cloud-init is enabled and has no network-config; it should still generate a fallback config, like we do today; uc16 will enable the netplan renderer; the baked in config should be removed  3) cloud-init is enabled, and we have a network config;  uc16 is configured to use netplan renderer, so we should delete
<rharper> (2) and (3) I think cover the cases for if content is known, delete it as cloud-init will render a fallback (or specified) network config
<rharper> and yes to wiping the files that netplan would have generated as the netplan renderer will call 'netplan generate' which will create those files as needed (from the cloud-init config it rendered)
<jgrimm> powersj and magicalChicken, you are probably interested in reviewing this too -> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321397
<jgrimm> rharper, thanks for writing ^^ fantastic
<powersj> rharper: that doc is incredibly helpful. thank you!
<rharper> cool
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321465
<rharper> k
<rharper> smoser: I proposed that the other night
<smoser> oh. link ?
<smoser> i thought i'd seen it
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321222
<rharper> mines better because you need to import logging and create  LOG object
<smoser> :)
<smoser> yeah
<smoser> ok
<smoser> rharper, http://paste.ubuntu.com/24282363/
<smoser> are you opposed to that on top of yours ?
<rharper> looking
<rharper> smoser: that looks fine
<smoser> rharper, thats merged. if you want to read... https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471
<smoser> i need to add tests
<smoser> its very simplistic and targetted. i had more generic looking things, but the scaffolding is unnecessary at this time. if we need somethign smarter later, add it then.
<rharper> smoser: ok, looking
<smoser> *just* p ushed with teests
<rharper> smoser: also, is it reasonable for ds-identify to also look in /etc/cloud/cloud.cfg.d/ ? for maas datasource ?
<rharper> the debconf_setselections hook for maas IIRC, writes the maas datasource cfg to /etc/cloud/cloud.cfg.d
<rharper> but ds-identify detect_maas only searchs PATH_CLOUD_CONFD/*maas*cfg ;  which I believe will miss the .cfg.d dir
<smoser> rharper, it does look for maas datasource there, no?
<smoser> it should. let me check.
<rharper> would be nice if check_config mentioned the files it checked in the debug log so we could be certain
<smoser> bah. i thjin its busted.
<rharper> debugging a uc16 deploy with plars and it's not finding the datasource
<rharper> smoser:  it does check files in .cfg.d/*maas*;  it;'s looking for a top-level key 'MAAS' but I believe we specify it via datasource:\n\tMAAS: {}
<rharper> so it doesn't match
<smoser> no. its just not looking right. the file names input are wrong.
<smoser> i dont think it checks in .cfg.d
<smoser> at all
<rharper> hrm
<rharper> I added a debug log to check_config to print the fnames that match
<rharper> [up 659.59s] ds-identify
<rharper> policy loaded: mode=search report=false found=all maybe=all notfound=disabled
<rharper> checking config file: /etc/cloud/cloud.cfg
<rharper> checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg
<rharper> checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg
<rharper> checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg
<rharper> shows it reading it
<rharper> now checking why we don't match
<smoser>  http://paste.ubuntu.com/24282702/
<smoser> hm..
<smoser> oh. those are just other things reading it.
<smoser> it does read those for some sources
<smoser> for some calls to check_config
<smoser> but not for maas's
<smoser> try that paste ?
<rharper> ok;  currently it looks like key was getting clobbered from a value in the file
<rharper> ahh, I see
<rharper> other methods were looking there but the dscheck_MAAS wasn't
<rharper> lemme try yours
<rharper> smoser: ok, that works!
<rharper> checking config file: /etc/cloud/cloud.cfg for key: MAAS
<rharper> checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg for key: MAAS
<rharper> checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg for key: MAAS
<rharper> checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg for key: MAAS
<rharper> check for 'MAAS' returned found
<smoser> that PATH_CLOUD_CONFD
<smoser> was very confusing
<rharper> smoser: that said, I think we want to check for datasource: MAAS  though
<smoser> i need to get my unit tests for ds-identify
<rharper> not just 'MAAS' does it walk indented colons ?
<smoser> thers a comment about that
<smoser> "currently does not respect any hierarchy in searching for key."
<rharper> urg
<rharper> so, how do specify the datasource: MAAS  then ?
<rharper> we need to emit MAAS: foo and the datasource: MAAS ?
<smoser> i'm confused.
<smoser> datasource:
<smoser>  MAAS:
<smoser>    stuf here
<smoser> that will work
<smoser> the parser doesn't know that 'MAAS' there is under datasource.
<smoser> iti just finds a key named MAAS
<rharper> ok; the comment about hierarchy led me to believe we'd not look at anything but the top level keys
<smoser> its just broken in a different way then you imagined :)
<rharper> if it was written datasource: { MAAS: { } }
<smoser> basically it just flattens everything and looks at only key names.
<rharper> it'd break as well
<smoser> maybe, yes.
<rharper> ok, well, that's good enough until we do unittests and sort out edge cases
<smoser> yeah. it doesnt support reading that.
<smoser> wait. maybe it does.
 * rharper tries
<rharper> nope
<smoser> right. it doesnt. we need a tool that can merge cloud config in the same way cloud-init does and run fast.
<rharper> y
<smoser> rharper, mp https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321482
<rharper> k
<smoser> rharper, fwiw, stumbled upon free speedup in cloud-init at
<smoser>  http://rmcgibbo.github.io/blog/2013/05/23/faster-yaml-parsing-with-libyaml/
<smoser> seemingly free
<jgrimm> smoser, have you benchmarked?
<smoser> it probably wont be huge for cloud-init, but it is faster per
<smoser>  http://stackoverflow.com/questions/16864132/is-there-an-up-to-date-fast-yaml-parser-with-python-bindings
<smoser>  http://stackoverflow.com/questions/27743711/can-i-speedup-yaml
<jgrimm> ack, yeah that's what i was curious about, whether it actually makes a difference, tho  small, but free is something
<smoser> rharper, cloudinit/net/netplan.py
<smoser> bah
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471
<smoser> prints removed
<rharper> smoser: whoa , we have libyaml ?
<smoser> well, in a new container this works:
<smoser>  python3 -c 'import yaml; from yaml import CLoader'
<smoser> so i'd think that'd trow exception otherwisd
<jgrimm> i had a trusty container handy, seemed to work their too (and i can see libyaml in dpkg -l too)
<jgrimm> s/their/there
<smoser> # apt-cache show python3-yaml | grep Depe
<smoser> Depends: python3 (<< 3.6), python3 (>= 3.5~), python3:any (>= 3.3.2-2~), libc6 (>= 2.14), libyaml-0-2
<rharper> do we run py3 on trusty c-i ?
<smoser> probably python2 there.
<jgrimm> i ran python2 there
<smoser> and we should do a try: except ImportError anyway
<jgrimm> https://github.com/msmbuilder/msmbuilder-legacy/pull/199/commits/beb21bf10892dd566fccf567f907404957db1aed
<rharper> oh I suppose it should still work for py2 to call to libyaml though right ?
<jgrimm> looks like easy pattern to adopt
<smoser> yeah
<smoser> rharper, intentionally or not, your bond rename devices test case shows that vlans have the same mac as the nic they use also
<smoser> so have to ignore those
<smoser> so, good that you did that .
<rharper> huh
<rharper> yeah, I guess that makes  sense
<rharper> they just need to check for the vlan header but the mac is going to be the same
<rharper> w.r.t "is this packet for me"
<rharper> yay unittests!
<jgrimm> powersj, any strong reason to require --data-dir to the integration test 'collect'?  That is, why not allow for tmpname as it gets output to stdout with 'run' (so could be inspected if left alone and/or verify'd later)
<powersj> jgrimm: I don't think it is a very strong feeling, but it was to prevent writing it to /tmp so you didn't loose your collect after each reboot :)
<powersj> strong reason* ugh
<jgrimm> cool, i assumed so. just didn't work as i'd guessed it would (since i didn't read the docs, doh)
<powersj> :)
<powersj> yeah if I recall when I was writing all the initial tests it would go to tmp and then the next day I would spend the first 20mins running collect again :\
<jgrimm> fwiw, the docs could be a bit more explicit on 'use collect' if you want to actually inspect a run
<jgrimm> i can certain submit a pull request if agreed
<powersj> magicalChicken has a MR to keep the records around when doing a run
<jgrimm> powersj, cool, that's what i was wanting to do and figured collect was the intended way of doing that
<powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/320401
<jgrimm> powersj, do we run integration tests on trusty as part of automation?
<powersj> that however does still need a doc update :)
<powersj> not yet
<powersj> A few too many of the tests initially did not work with trusty versus xenial and newer due to changes in behavior
<jgrimm> heh, ok i want to play with the tests and better understand.. randomly chose a module.. blows up on trusty, not elsewhere. :)
<jgrimm> cool. known issue(s). :)
<powersj> haha yeah magicalChicken has yet another MR for distro flags, to be able to specify what tests can run on what distros
<powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/321029
<jgrimm> indeed, looking forward to that landing, really nice addition
 * jgrimm checks whether this one is fixed already. 
<jgrimm> powersj, but overall I think the integration test infrastructure is really nice.. easy to add to it
<powersj> jgrimm: thanks! magicalChicken: ^^^ :)
<jgrimm> \o/
<magicalChicken> powersj, jgrimm: nice thanks :)
<magicalChicken> jgrimm: you mentioned getting everything going on trusty, i still have a couple more feature flags to mess with there i think
<magicalChicken> jgrimm: part of the issue is out of date cloud-init, so i need to build a py2 deb with current master for trusty and see how many tests fail just because of the old version
<jgrimm> magicalChicken, very cool
<jgrimm> indeed, i'm guessing this is an issue with being out of date
#cloud-init 2017-03-31
<jgrimm> smoser, i'm gonna work on https://bugs.launchpad.net/cloud-init/+bug/1583841 if you don't mind
<smoser> that is odd that we explicitly install ruby1.8
<smoser> oh... because we just dont know i guess? as the gems installer is not from the archive/
<smoser> maybe. but thanks!
<jgrimm> i'm assuming at one point there were multiple rubys and it needed specific one
<jgrimm> also noticed that our chef example config is broken, i have that working locally.. i'll submit bug & MR
<jgrimm> the upstream apt repo changed
<smoser> jgrimm, fyi, cloud-init got let in this morning to zesty.
<jgrimm> i saw!
<jgrimm> \o/
<jgrimm> powersj, if i want to run the tox citest against code changes i've made in my local cloud-init repo.  i'm guessing i need to create a dep and supply it, otherwise it will test against archive cloud-init?
<jgrimm> s/dep/deb
<rharper> jgrimm: yes;
<rharper> I think we should add that workflow example to the tests.rst doc
<jgrimm> thanks, right..  i found it in practice, but the docs could be clearer
<rharper> I'm pretty sure you can: ./package/bddeb  ; and then feed the resulting deb to the test ... now;  magicalChicken did work on adding a 'build deb' stage
<jgrimm> sweet
<rharper> as not everyone has deps installed for binary build
<jgrimm> right
<rharper> I've just not tested it, so don't know the flags/steps etc
<rharper> if you figure them out, a MR for doc updates would be great
<jgrimm> :) i was mostly looking for confirmation of what i was seeing
<jgrimm> and yes, on docs updates
<rharper> =)
<powersj> or we could just accept wes' merge that does it for you :)
<jgrimm> heh, indeed i want that to happen soon
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321578
<rharper> looking
<rharper> neat
<rharper> reviewed, somewhat concerned by always filtering 'stolen' mac  entries;  I'd like to scope the filtering to just the rename case where we know that we don't want those interface mappings
<smoser> rharper, read my comment there please. and see what you think
<smoser> bah. adding the RuntimeError makes unit tests fail on my system
<smoser> RuntimeError: duplicate mac found! both 'virbr0-nic' and 'virbr0' have mac '52:54:00:4c:a1:4a'
<rharper> smoser: looking
<rharper> smoser: if all callers are expecting 'physical' nics then maybe the method ought it be called get_physical_interfaces_by_mac();  it certainly appears that all users of the method were interested in 'physical' only;
<rharper> smoser: shouldn't we mocking the response there?
<rharper> w.r.t your RunTimeError
<rharper> that looks like host data
<smoser> well, yeah, it *is* host data.
<smoser> which yeah should be mocked, but is a good find. thats a bridge, which i think most of the time will have the address of a nic also.
<rharper> y
<rharper> hehe
<smoser> i dont like the word 'physical'
<smoser> because explicitly im' including veth
<smoser> "base nics" ? i dont know.
<rharper> I don't know either
<rharper> in the docs I wrote
<smoser> but yeah. all callers were expecting "ethernet interfaces"
<rharper> I mentioned this was a mix of criteria
<rharper> s/ethernet/network interfaces;  and I suspect 'present without configuration/composition'
<smoser> :)
<smoser> get_interfaces_present_without_configuration_by_mac()
<rharper> heh
<smoser> rolls right off the tounge
<rharper> clearly
<rharper> get_renameable_interfaces
<smoser> so i think i'm going to just exclude bridges for now
<rharper> I think the property is certainly key
<rharper> we were before as well (excluding bridges) and (name=veth*)
<smoser> well, except i'm explicitly including in that list stuff that has a '0' in addr_assign_type
<smoser> well the get_interfaces_by_mac() didn't previously exlude bridges
<rharper> which is fine, no?  those devices were *created* in the kernel vs. added/renamed/modified during boot
<smoser> i dont follow.
<smoser> so i just changed get_interfaces_by_mac() to exclude bridges
<smoser> which i think is right. because bridges are not "present_without_configuration"
<rharper> I don't have a bridge around , what addr type was it?
<rharper> there's one; 1 for me
<rharper> so would have been allowed,; yeah I'm in agreement;  I was confused; in the fallback networing code, there is some filtering done there when selecting nics; in that case bridges, lo, and nics with 'veth*' in the name were dropped
<rharper> it might be worthwhile to consolidate this 'get_interfaces_by_mac_without_config_excluding_lo' into something common
<nacc> is 'lo' the literal value used? maybe get_interfaces_by_mac(with_config: bool, excludes:list) ?
<smoser> nacc, yes.
<smoser>  invalid_interfaces = set(['lo'])
<nacc> makes for at least a shorter function name :)
<nacc> i would possibly make flags: enum or something, if there are multiple types of withouts
<rharper> nacc: yeah, so ls -1 /sys/class/net/*  are all of the 'net' devices;  some of those don't support renaming, and also aren't useful for a default "first nic which we can likely expect DHCP to work on"
<nacc> rharper: yep, makes sense
<rharper> smoser: fyi, just tested the zesty cloud-init in a uc16 image; it cleans up the default netplan config perfectly and rendered a new one as expected;  nice!
<smoser> \o/
<rharper> testing in OpenStack next, and then I'll upload an image for plars to check out on maas
<smoser> rharper, nothing is ever easy
<rharper> =)
<jgrimm> powersj,  i haven't looked into it, but it would be nice to be able to override 'enabled: False' without editing a testcase.yaml..
<smoser> rharper, and for the resolvconf/ifupdown i need to file a debian bug
<powersj> jgrimm: like when you try running it by hand with -t <testcase> and have it still run?
<jgrimm> yep!
<powersj> ah! ok :)
<powersj> I'll jot that one down
<jgrimm> powersj, as there are testcases disabed due to 'taking too long' for part of normal ci
<jgrimm> but i want to easily run it
<powersj> jgrimm: none should be disabled for taking too long, only if 1) we know they don't work 2) they are not done 3) it is covered by another test case
<jgrimm> powersj, see tests/cloud_tests/configs/example/TODO.md
<jgrimm> powersj, but indeed, i did not find the chef one to actually take longer than anything else really to run
<nacc> jgrimm: ok, just verified the fix for the cloud-images does work
<nacc> Odd_Bloke: --^ fyi
<jgrimm> cool
<jgrimm> nacc, i'm assuming iscsi. :)
<powersj> jgrimm: ah yeah that list is things that have not even been written yet
<nacc> jgrimm: ack
<jgrimm> powersj, heh.. well someone wrote ' (takes > 60 seconds to run)' so I assumed they'd actually run it
<jgrimm> powersj,  indeed comments can lie. :)
<powersj> ah ok :)
<jgrimm> powersj, i'm using that install_run_chef_recipes.yaml to test https://bugs.launchpad.net/cloud-init/+bug/1678145 ...  I'm ambivalent whether I enable it for citest? thoughts?
<powersj> If you get it running, go for it
<jgrimm> right now i've only cobbled in a single test just to see that chef installed.. as good enough, with FIXME to add more comprehensively
<powersj> if you don't let me know and I can finish it up. I'd prefer the test while you are working on it :)
<jgrimm> my hestitations were 1) that someone explicitly said it should be disabled (per comment I showed you) 2) it depends on an upstream  repo  .. I'm not thrilled with external network dependencies in CI  3) only LTSes supported by external apt repo
<jgrimm> so #3 probably needs to wait for wes's featureflag to do skips??
<magicalChicken> jgrimm: that would be pretty easy to do using feature flags, just add a 'ubuntu_lts' flag and require it there
<jgrimm> yep! indeed quite handy
<powersj> hmm I agree that the 3rd party repo is undesirable :\
<jgrimm> heh, but that's what the feature is there for.. easy of installing the upstream
<jgrimm> but shouldn't a hard error if the resource is unavailable would be desired behavior
<powersj> I know having unreliable and false negatives is annoying, so do we have any idea how available  it is? If it is failing for you even during development then I think that is an easy no.
<jgrimm> powersj, i have no idea at all whether its _really_ unreliable.. just my overall concern with external resources
<jgrimm> heh, except that its a hard failure today because they changed the repo
<jgrimm> and we'd not noticed as we don't have any test. :)
<powersj> right so a test would have caught this :)
<powersj> haha
<jgrimm> exactly!
<magicalChicken> powersj, jgrimm: i could add in support for overriding base feature flags from cmdline. that way the feature could be unavailable everywhere by default, and we could just enable it when we know the resource is available
 * powersj wants to just add it and if smoser sees too many failures we can pull it
<jgrimm> i think that may be handy generally
<jgrimm> powersj, yep, i'm fine with enabling it. skip again if its not worth it
<jgrimm> but.. i'm going to wait for feature flags
<smoser> which is this ?
<smoser> talking about a test for chef ?
<jgrimm> yep, specifically install_run_chef_recipes.yaml
<jgrimm> which is our chef config example
<jgrimm> no test for it today, which is why this was never noticed:   https://bugs.launchpad.net/cloud-init/+bug/1678145
<rharper> smoser: plars tested the ds-identify change for maas cfg parsing, that's working as well
<Odd_Bloke> nacc: \o/
<smoser> jgrimm, yeah.. so we can enable / add one. but as we've seen external dependencies suck
<smoser> ie, curtin dependencies on launchpad or archive have a very non-zero failure rate.
<smoser> adding a 3rd party dependency has potential for the same.
<jgrimm> smoser, yep.  totally aware, that's why i flagged it as an issue
<nacc> Odd_Bloke: thanks for your help!
<jgrimm> smoser, for now, i was just looking to add a test that we can use on demand, i'm more wary of having it always enabled.
<smoser> right.
<jgrimm> smoser, with zesty cloud-init uploaded, i'm assuming you are working on the corresponding xenial SRU?
<smoser> i'm still working on this bond thing :)
<smoser> but yeah, that'd be the plan
<jgrimm> ok, yeah, we are gated on next steps with that SRU
<powersj> jgrimm: seems you got a harder review than I did ;)
<jgrimm> powersj, lol.  its the nature of reviews.  you can see something new every time you look in
<smoser> rharper, still there?
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1677846 . i think i just need to ggive up
<rharper> here
<rharper> smoser: I thought we already pointed back at the OpenStack change there
<rharper> just a dup, right?
<smoser> yeah. its just that i think i'm going to give up. at lest for 'dvs'
<smoser> and re-push upstream proposal
<rharper> why?
<smoser> the real rason for ping
<smoser> http://paste.ubuntu.com/24290015/
<smoser> ^ that is the diff i have  locally
<smoser> have to remove use of 'assert_called()'
<smoser> but wanted you to review that.
<rharper> looking
<smoser> taht is diff agastin what is in https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321578
<smoser> does that provide rasonable test ?
<smoser> the thing is kind of hard to test. wihtout mock-o-rama
 * rharper wishes for assert_called();  I use len(mock_obj.call_arg_list) != 0 IIRC 
<rharper> so, in the case that we see a duplicate, why is devs reset to empty list ?
<rharper> OSError is for something besides the RuntimeError for duplicates I suspect
<rharper> still not sure why the list would be reset in that case; documenting why we reset the list here would help (it's not obvious to me)
<rharper> that said, the unittests look good
<smoser> where do you see the list is reset ?
<smoser> rharper, ^ ?
<rharper> line 25- > 27
<rharper> on the OSError exception if errno.ENOENT
<rharper> devs = []
<smoser> that has always been there.
<rharper> code motion then
<rharper> ok
<smoser> if get_device_list() raises an exception
<smoser> then it sets the list to []
<rharper> still seems odd to wipe the list, rather than return what it already had
<smoser> ie, if you didn't have /sys/class/net
<smoser> then you get "no devices"
<smoser> there is no "already had"
<rharper> I see, if devs was none
<rharper> it just looks strange to set it to list, and then continue with a for name in devs which wont do anything, just return []
<rharper> oh, I see, we're returning a dict; well could return an empty dict
<rharper> it's fine
<rharper> it's harder to see with just the patch context;
<smoser> yeah.
<smoser> and you're right. return {}
<smoser> would be the same
<smoser> i dropped the 'devs=' param
<smoser> as it was wierd.
<smoser>     Also drop the 'devs' argument to get_interfaces_by_mac.  It was
<smoser>     non-obvious what the result should be if a device in the input
<smoser>     list was filtered out. ie should the following have an entry for
<smoser>     bond0 or not.  get_interfaces_by_mac(devs=['bond0'])
<rharper> yeah, not sure who we pre-loading it with a set of params
<smoser> nothing used it like that.
<rharper> s/we/would
<rharper> yeah
<rharper> maybe for easier unittesting =)
<smoser> right
<smoser> thats why i added it i'm sure
<rharper> =_
<rharper> err, =)
<jgrimm> smoser, thoughts on URL shortener that won't disappear (or desire to use/avoid in the project??) -> https://bugs.launchpad.net/cloud-init/+bug/1669727/comments/1
<jgrimm> smoser, fwiw... i'm trying to catch up to my long ago promise to sweep through the old bugs and do some housekeeping. long overdue, sorry
#cloud-init 2018-03-26
<resmo> hi
<resmo> I looked for a way to determine if cloud-init has finished to configure the instance. What is the right way to do so?
<resmo> ok, found the necessary infos here https://github.com/number5/cloud-init/blob/master/doc/status.txt
<resmo> Thanks ;)
<do3meli> smoser: thanks for the CI run. the error that got thrown does not seems to be related to my branch. am i right with that?
<do3meli> error given in https://jenkins.ubuntu.com/server/job/cloud-init-ci/928/console
<do3meli> cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()
<smoser> do3meli: rebase to master.
<smoser> (that is fixed in trunk already)
<smoser> make sense ?
<do3meli> allright, let me do that
<do3meli> smoser: rebase done and pushed
<smoser> do3meli: running https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci/929/console
<do3meli> thx
<smoser> do3meli: curious... have you tested this with linux ?
<smoser> or are you just adding support for zfs resize of freebsd
<do3meli> i am mainly focusing on freebsd
<do3meli> i have no zfs on linux to test
<do3meli> with root on zpool
<smoser> right.
<smoser> do3meli i'm reviewing. i'll just put comments tthere.
<smoser> but will submit hopefully in 5 minutes
<do3meli> perfect, thanks
<smoser> do3meli: i responded.
<smoser> i think that the end result is just "update the message"
<smoser> but i'd like for you to think through the "safe for linux" thought path also.
<do3meli> thanks for the feedback. will go over it later and let you know my thoughts
<do3meli> smoser: what you think about the adjusted commit message?
<smoser> do3meli: updated a bit
<do3meli> perfect, thx
<resmo> btw added a cloud_init_facts ansible module https://github.com/ansible/ansible/pull/37932
<do3meli> i think through the other stuff you mentioned now.
<do3meli> smoser: should i try add some tests with the outputs you mentioned from curtin'
<smoser> do3meli: if you are willing to do that, that might be nice.
<smoser> but actually..
<smoser> the files used for linux are /proc/1/mountinfo
<smoser> which differs in format from /proc/1/mounts
<smoser> er...
<smoser> /proc/mounts
<smoser> which is what we're using incloud-init
<smoser> so the info isn't really there.
<heatfanjohn> http://cloud-init.org is returning 502 Gateway timeout errors
<heatfanjohn> Sorry that 502 Bad gateway errors
<nacc> heatfanjohn: are you looking for docs? (agree on the 502)
<smoser> heatfanjohn: thanks
<blackboxsw> rharper: in your chrony branch, what's the purpose of the 'name' key in each ntp client config dictionary?
<blackboxsw> I don't think I see it used anywhere?
<rharper> blackboxsw: I thought it was used to extract the client name (which differs from the installed package)
<rharper> but, that may have been refactored out, looks droppable right now
<smoser> rharper: regarding ConditionFileExists evaluation time
<smoser>  http://paste.ubuntu.com/p/7vhSWgPKfc/
<smoser> that indicates that it is at unit start time.
<rharper> yeah
<blackboxsw> rharper: last volley of comments and schema pastebin @ https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/339438
<rharper> blackboxsw: thanks, fixing up your first round
<blackboxsw> rharper: I also tweaked the unittest to use temp_utils.mkstemp instead of NamedTemporaryFile
<blackboxsw> I'm going to get a couple of deployments going on this to kick the tires
<smoser> hey. i have to be headed out. i'll check back in for you all in probably 3.5 hours.
<rharper> k
<rharper> blackboxsw: interesting, ok
<smoser> i just pushed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342005
<smoser> i think i'm going to just upload that too.
<smoser> just to have that change into ubuntu now.
<blackboxsw> silly revert pused up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342154
<blackboxsw> pushed even
<blackboxsw> rharper: I'll checkout your updates thx. I think I already made the change to temp_utils and unit test mock in my pastebin
#cloud-init 2018-03-27
<blackboxsw> thx smoser, review-mps commit message linting. bah
<smoser> blackboxsw: good now. i fixed.
<blackboxsw> yeah, I think we'll have to make a cut of cc_ntp tomorrow for bionic. some things need wrap up there and I want a couple of live tests centos/sles which I'm going through now
<blackboxsw> and it's bedtime ;)
<smoser> indeed.
<smoser> nn
<smoser> rharper: https://code.launchpad.net/~mwhudson/livecd-rootfs/default-LANG-C.UTF-8/+merge/342144
<smoser> \o/
<smoser> i think that means we wont hit that generate any more.
<smoser> do3meli: https://jenkins.ubuntu.com/server/job/cloud-init-ci/933/
<smoser> and i am going to add you to the group that gets c-i run automatically for you.
<do3meli> @smoser: thanks
<do3meli> smoser: ci build was successfull. from my point of view good to merge now. if you feel the same you can set the status of the MR to approve ;-)
<rharper> smoser: yay LANG!
<smoser> do3meli: on its way.
<smoser> thanks!
<smoser> do3meli: you need more work now ?
<smoser> :)
<do3meli> haha :-) guess my company wants me to move on with other stuff now :-P
<smoser> boooo
<do3meli> well i will stay connected i guess ;-) maybe bit bug management or something like that
<do3meli> well see
<smoser> alright. then i wont set you up to get auto c-i just yet.
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342214
<blackboxsw> smoser: rharper filed the release 18.2 bug
<blackboxsw> including it in a cloud-init release bump branch now
<blackboxsw> smoser: let's include the dependency
<blackboxsw> sound good?
<blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1759318
<ubot5> Ubuntu bug 1759318 in cloud-init "Release 18.2" [Undecided,New]
<smoser> eyah. in trunk it doesnt hurt anythin. and we want it in packaging.
<smoser> so if you want to pull that, go ahead.
<blackboxsw> yeah, figured almost zero risk
<blackboxsw> landing your branch (after tox)
<smoser> blackboxsw: i'm landing ubuntu branch change now.
<blackboxsw> +1 smoser
<blackboxsw> not touching it
<smoser> done
<blackboxsw> thx pulling
<blackboxsw> smoser: are you landing https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342215 ?
<blackboxsw> wanted to pull that into 18.2
 * blackboxsw misunderstood the 'I'm landing ubuntu branch change" comment
<smoser> yeah, lets grab that.
<smoser> blackboxsw: i'm landing that one now
<blackboxsw> thanks ok will repropose 18.2 after that then
<blackboxsw> hrm just dropped tip of master on a cent6 container. it didn't dedect DataSourceNoCloud and it's currently trying to hit Hetzner cloud... checking out what's up
<smoser> hm.
 * blackboxsw is thinking minimally something is up with the exception_cb
<smoser> hm. hm.
<smoser> blackboxsw: how do i reproduce that?
<blackboxsw> 2018-03-27 17:00:29,960 - handlers.py[DEBUG]: finish: init-local/search-NoCloud: SUCCESS: no local data found from DataSourceNoCloud
<blackboxsw> hmm, umm I expected that to find yes in the cent6 container. /me thinks..
<blackboxsw> looking over ubuntu
<smoser> blackboxsw: you're sure you have /var/lib/cloud/ stuff there ?
<blackboxsw> [root@cloud-init-centos-2f4670b4 ~]# ls /var/lib/cloud/seed/
<blackboxsw> [root@cloud-init-centos-2f4670b4 ~]#
<smoser> the linuxcontainers.org provided images do not have metaata that says it should write that.
<blackboxsw> hrm why not, thought that was baked into the lxc images
<smoser> you would have to add the metadata to them to write it.
<blackboxsw> ahh, ok, oops. thought I hadn't had to do that in the past... but maybe I just forgot a step (as I've only recently just run unittests on cent*)
<blackboxsw> nevermind the crazy guy in the corner: PEBCAK
<smoser> well, to run unit tests.
<smoser>  ./tools/run-centos
<smoser> ./tools/run-centos --rpm --srpm --unittest 6
<blackboxsw> yeah that's what I kicked with -k -r
<blackboxsw> it left me with the container that I then attempted to install the rpm on. neglecting to make sure the image had a seed
<smoser> ah. right.
<blackboxsw> ok created the seed on centos container, everything happy again with ds detection
<blackboxsw> sorry for the noise
<blackboxsw> though /me wants to test hetzner cloud again to make sure get_data bails quickly if !hetzner cloud
<smoser> blackboxsw: it does not.
<blackboxsw> ahh right timeout=2, sec_between=2, retries=30):
<blackboxsw> ok
<blackboxsw> so 2 mins of read_metadata attempts
<blackboxsw> ok; as intended. I was uncertain if it was looping forever on my container. will check that it limits to 30 retries
<blackboxsw> I see you landed the dhcp client. ... putting up cloud-init 18.2 in a min here
<smoser> blackboxsw: i can fix hetzer
<smoser> and i think maybe we should
<blackboxsw> it's near last in our list of datasources, but I think we could throw up our hands a bit earlier
<blackboxsw> to avoid making IBM wait :)
<blackboxsw> smoser: ok on awaiting hetzner  fix for 18.2
<smoser> http://paste.ubuntu.com/p/YvkFJCFyXk/
<blackboxsw> checking ds-identify for comparison
<blackboxsw> +1 system-manafacturer -> sys_vendor in sysfs
<blackboxsw> ok
<blackboxsw> s/hetzer/hetzner
<blackboxsw> yeah 4 mins of Hetzner waiting for both read_metadata, read_userdata is too much waiting
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342226
 * smoser goes to fire up a hetzner cloud instance to just make sure i dont break it.
<blackboxsw> smoser: +1, I'll let you merge it once you've hitup hetzner
<blackboxsw> I
<blackboxsw> I've approved the branch and tested on lxc without seed to see a quick fail
<smoser> i put some info there that it was good.
<smoser> now actually installing the deb and cloud-init-clean
<blackboxsw> pylint and flake8 good on your branch
<smoser> you can merge ?
<blackboxsw> https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci/937/ waiting on ci, about 10 mins left looks like. but I think I can merge it when it's done.
<blackboxsw> or I can merge now if you want as I expect a clean CI bill of health
<smoser> oh. right.
<smoser> i thought you were saying it ran
<smoser> we wait on the all powerful jenkins
<blackboxsw> pay no attention to the man behind the curtain
<blackboxsw> yeah will wait and merge post-jenkins
 * blackboxsw relocates
<blackboxsw> smoser: rharper https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342217 is ready for review
<blackboxsw> I pushed a new 18.2 signed tag hopefully that's all we need.as git describe now tells me I'm on 18.2
<blackboxsw> smoser's dhcp and hetzner fixes are landed
<smoser> blackboxsw: woot.
<smoser> ok.
<rharper> citest noob;  so the ntp tests fail on my branch, if I ran citest via tox, where do I find the collected data, like cloud-init logs and the files the test collected?
<rharper> smoser: for the ntp updates I think I need to a) update the ntp test-case match trunk w.r.t which client is to be installed; note this test changes depending on the release image; xenial, for example is going to always use timesyncd, but in bionic it would be chrony;  b) do we have a way to construct yaml on a per-release basis?  c) does ci-test run only on xenial or what other releases ?
<blackboxsw> back
<blackboxsw> rharper: ../results generally
<rharper> that
<rharper> that's in toplevel ?
<rharper> when run via tox ?
<blackboxsw> it's above your source tree root
<blackboxsw> I *think* that's the default... checking
<rharper> I think tox ends up doing tmpdir thingys ... I dunno
<blackboxsw> might need --preserve-data
<rharper> ok, and I wonder if that works under tox
<rharper> well, I should just install the deps and run from tree getting it working and then test tox
<blackboxsw> tox -e citest -- --preserve-data --results ../results
<blackboxsw> or something like that... checking
<blackboxsw> rharper .tox/citest/bin/python3 -m tests.cloud_tests run --os-name=bionic --platform=lxd --preserve-data --data-dir=../results --verbose -t modules/ntp
<blackboxsw> and -t modules/ntp_servers   -t modules/ntp_pools
<smoser> rharper: http://paste.ubuntu.com/p/629gZx94qs/
<smoser> that is what i run
<smoser> like:
<smoser>   ../scripts/go-test lxc --test=tests/cloud_tests/testcases/modules/final_message.py
<smoser> and it puts results into ./results.lxc.d/
<smoser> you can change the 'xenial' around
<smoser> c-i i think only runs with xenial
<smoser> but we have other runners that run on test with different releases.
<rharper> k
<smoser> you should be able to check the release that is being used in the test case
<smoser> i dont think we have per-release config
<smoser> but you can make pre-release assertions
<blackboxsw> ci looks done on 18.2 for cloud-init https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342217
<rharper> smoser: ok, I'm going to update the config to work with xenial, and branch out into other variants after that
<rharper> tox-venv ?
<rharper> need to pip install that I guess
 * rharper relocates
<smoser> rharper: tox-venv is ./tools/
<smoser> tox-venv <name>
<smoser> enters <name> and executes a command
<smoser> the difference compared to 'tox -e' is that you can run any command and it doesnt do setup.py first (which is slow)
<blackboxsw> incredibly slow
<smoser> blackboxsw: where is the doc for release ?
<smoser> https://hackmd.io/zbeM1cG-REq2NE3BvesqOw
<blackboxsw> smoser: you mean rtd?
<blackboxsw> ahh
<smoser> we need to move that to somewhere i think
<smoser> blackboxsw: do you want to do the rest ?
<blackboxsw> +1, yeah, gist?
<smoser> i pushed your tag
<smoser> i can do curtin release then.
<blackboxsw> +1 I'll do the rest
<blackboxsw> on cloud-init
<blackboxsw> smoser: rharper  can you see Create release here? https://launchpad.net/cloud-init/+milestone/18.2
<rharper> looking
<blackboxsw> I think I'm missing perms, as I can see that option on landscape projects
<rharper> nope
<blackboxsw> I hoped to see it under "Expected: "
<blackboxsw> but that date is already set to today(maybe smoser already ticked it)
<smoser> https://launchpad.net/cloud-init/+milestone/18.2 <--
<blackboxsw> hrm nope:  cloud-init 18.2	2018-03-22	not yet released  from https://launchpad.net/cloud-init/trunk
<smoser> thta was teh "due date"
<smoser> now we have the release date
<blackboxsw> hrm, ok that view changed for me now from what it used to be. probably clicked something I didn't have perms for I guess.
<rharper> https://launchpad.net/cloud-init/+series   that has a reference to 2.0 and stackforge ...
<rharper> we probably can nuke that
<rharper> smoser: ^
<smoser> gonzo
<rharper> sweet
<blackboxsw> smoser: I'm closing bugs now for 18.2 ok ?
<smoser> yeah. i'm doing it now for curtin
<smoser> stepping away from machine.
<smoser> will check back in later tonight.
<smoser> bugger
<smoser> that stupid script really needs '--project'
<smoser> i missed a s/Cloud-init/curtin/
<smoser> oh well
<blackboxsw> ohh right smoser  lp-bugs-released needs a --project-name
<blackboxsw> will add it
<rharper> hehe
<blackboxsw> cloud-init bugs are marked fix-released
<blackboxsw> missing tasks for: 1298921 1757176 1759307.... checking those out now
<blackboxsw> I marked the cloud-init task fix released on https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1759307
<ubot5> Ubuntu bug 1759307 in initramfs-tools (Ubuntu) "missing dependency on isc-dhcp-client (dhclient)" [Undecided,New]
 * blackboxsw fires up a cloud-init MP for bionic
 * rharper is iterating with ci-test script from smoser;  excellent 
<rharper> having fun with system_config and user-data merging ...  now, I wonder what's going on; don't seem to be getting user-data ntp: dict merged when cc_ntp is running, so it's running in auto mode even when we specify a ntp_client in user-data
<blackboxsw> 18.2 release publish MP into bionic https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342239
 * blackboxsw creates an SRU bug for xenial for 18.2
<blackboxsw> xenial & artful rather
<rharper> can confirm that the distro object is created before we cloudify, which reads the user-data, so distro itself doesn't have user-data; =(
<smoser> blackboxsw: ok. that MP is on its way up... assuming it builds here.
<smoser> i uploaded source tarball to https://launchpad.net/cloud-init/trunk/18.2
<smoser> and now i'm out.
<blackboxsw> thanks smoser see ya
 * rharper refactors getting user-data config back int
<rharper> and
 * rharper relocates
<blackboxsw> Xenial SRU for 18.2: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342248
<blackboxsw> Artful SRU for 18.2: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342249
<blackboxsw> ^ includes the debian/control changes for isc-dhcp-client
<blackboxsw> ok just gen'd a trello card for cloud-init SRU https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> oops
<blackboxsw> https://trello.com/c/RZDKaWSv/719-cloud-init-sru-182-artful-xenial
<rharper> ok, unbricked unittests after re-introducing usercfg to the get_ntp_client_info() path as distro object does not have user-data config
<rharper> \o/ passing the updated ntp tests
<rharper> now to add new client tests
#cloud-init 2018-03-28
<rharper> ok, systemd_timesyncd and chrony done;
<rharper> now I'm done, tomorrow fixing up py26 unittests
<rharper> *sigh*
<smoser> blackboxsw: i would probably not add the new depends into x, a
<smoser> it probably *is* strictly required.
<smoser> but because isc-dhcp-client is part of ubuntu-minimal
<smoser> and you're really expected to create a ubuntu without 'ubuntu-minimal'
<smoser> it is not likely that cloud-init is to be installed in such a place.
<smoser> so.. i'd just leave it be.  additionally, adding depends or recommends like that to a stable can be problematic
<smoser> see someone complaining about a similar add
<smoser>  https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1633643
<ubot5> Ubuntu bug 1633643 in initramfs-tools (Ubuntu) "unnecessary dependency upon isc-dhcp-client" [Undecided,Invalid]
<smoser> and then also the squashfuse bug (bug 1628289)
<ubot5> bug 1628289 in Snappy "snapd should depend on squashfuse (for use in containers)" [Undecided,In progress] https://launchpad.net/bugs/1628289
<blackboxsw> +1 on dropping isc-dhcp-client depends
<blackboxsw> will repost the MPs without that
<blackboxsw> smoser: thanks for the reference to https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1756173
<ubot5> Ubuntu bug 1756173 in snapd (Ubuntu Artful) "[SRU] 2.32" [Undecided,New]
<blackboxsw> I've added a card https://trello.com/c/OTYKbFEu/720-drop-squashfuse-install-logic-from-ccsnap-when-https-bugslaunchpadnet-ubuntu-source-snapd-bug-1756173-lands
<blackboxsw> pushed ubuntu/xenial and ubuntu/artful without isc-dhcp-client explicit dependency
<rharper> smoser: ./tools/run-centos is super handy =)
<smoser> yeah. we need to make it work for suse too
<smoser> and actually.. i've even wnated it to work for ubuntu
<smoser> when there was some bug trhat didnt reproduce on my desktoip, but c-i saw.
<rharper> interesting
<smoser> blackboxsw: ok... can you give qa-scripts/new-upstream-snapshot a try ?
<smoser> you can see the last commit message there.
<smoser> i think i have addressed most of the pain in it.
<blackboxsw> checking/updating now
<blackboxsw> and thanks!
<blackboxsw> smoser: we'll drop new-upstream-snapshot script after the SRU or as part of the SRU?
<blackboxsw> in the ubuntu/xenial|artful branches
<smoser> i think no hurry to drop it.
<smoser> so i'd say after.
<smoser> but if you want to drop it we can.
 * blackboxsw checks out the refresh patches logic. that was a surprise
<smoser> thats why your xenial didnt work
<smoser> because the patch would only apply with fuzz
<smoser> as we'd changed ds-identify aroudn the xenial patches
<blackboxsw> gotcha, yeah I just didn't expect to see a separate commit message for it, and changelog entry. Both make sense as separate items, I just wanted to understand the quilt magic
<blackboxsw> smoser: pushed just ubuntu/xenial branch again
<blackboxsw> shall I push artful again?
<blackboxsw> new-upstream-snapshot works like a charm
<smoser> blackboxsw: i uploaded artful all ready.
<blackboxsw> ahh hadn't checked, thx
<smoser> i don tthink any reason to re-do it.
<blackboxsw> changelog didn't end up changing anything, so good there on artful anyway
<ihre> I'd like to test some changes in user-data after applying some changes to user-data. So far, I've mostly used cloud-init w/ qemu and genisoimage for testing, but I think there should be an easier way. I've tested w/ manual changes to cloud-config.txt/user-data.txt/user-data.txt.1 in /var/lib/cloud/instances/$instance-id/, and by reattaching the cdrom through the qemu monitor, yet changes aren't picked up.
<ihre> Which file on disk should I modify to be able to rerun cloud-init without having to go through the qemu/genisoimage steps?
<blackboxsw> ihre: there are several semaphores that block re-running specific cloud-init modules across reboots
<blackboxsw> you can clear out those semaphores by running "sudo cloud-init clean --logs --reboot"
<blackboxsw> it'll remove all cloud-init former run logs and reboot the machine so cloud-init can re-run "fresh" with any user data you've provided
<blackboxsw> I generally iterate using lxc on ubuntu
<blackboxsw> via lxc launch ubuntu-daily:bionic myb1
<blackboxsw> lxc exec myb1 bash   # then manipulate user-data in /var/lib/cloud/seed/nocloud-net/user-data
<blackboxsw> in lxcs, the  no-cloud/seed directory is kept across reboots.    I'll check how the qemu image detected  re-run to better answer your question
<blackboxsw> hrm, I actually have to run on an errand, will get back to that in ~45 mins
<ihre> blackboxsw: Thanks, didnt think of using a container to test it, that seems a nice alternative as well.
<ihre> Previously, I just removed a/the semaphore(s) by hand and reran cloud-init modules --mode final.
<blackboxsw> it's at least the fastest iterative method I've found for dev-test cycles.
 * blackboxsw heads out 
<ihre> No rush on my end, the current way works fine though, it just feels a little cumbersome at times, anyways, thanks for taking the time to look into it :)
<smoser> blackboxsw: your quilt refrsh differs in behavior from mine.
<nacc> blackboxsw: smoser: it's sensitive to ~/.quiltrc
<nacc> use --quiltrc -
<nacc> (just a guess)
<smoser> yeah.
<nacc> we have to do that in git-ubuntu
<nacc> smoser: i'm guessing this is about reproducibility?
<smoser> blackboxsw: grab tip of qa-scripts and try again, would you ?
<smoser> nacc: well, yeah.
<smoser>  
<smoser> http://paste.ubuntu.com/p/3BwC5n3v4P/
<nacc> smoser: yeah
<smoser> we run 'new-upstream-snapshot' to create a new upstream snapshot of cloud-init.
<smoser> and that is a run here versus on chad's system.
<nacc> yeah, you want your snapshot to be the same as chad
<nacc> similar idea for us, we want our same-argument builds to be contentfully the same
<blackboxsw> smoser: just pushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/ubuntu/xenial
<smoser> blackboxsw: woot.
<smoser> mine to yours : http://paste.ubuntu.com/p/spBd2CCrKg/
<smoser> new feature there to strip bug numbers on sru
<smoser> and make you give an SRU bug
<blackboxsw> ahh right I totally forgot to strip bug#'s on artful
<blackboxsw> smoser: what should I do w.r.t. devel/arful
<blackboxsw> smoser: what should I do w.r.t. devel/artful. should we strip bug#'s
<smoser> blackboxsw: oh. shoot. yeah i guess we should. right ?
<smoser> we are supposed to.
<smoser> we can just re-upload
<blackboxsw> yeah, I think so, yeah I'll repropose and do that right.
<blackboxsw> forgot that part of the process obviously
<smoser> well now the script does the right thing
<smoser> you forgot, i forgot too.
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342329 just --forced a new-upstream-release for artful
<blackboxsw> merge conflicts against existing ubuntu/artful.
<blackboxsw> for all the bug comment drops
<smoser> blackboxsw: how bout this..
<smoser> git checkout ubuntu/artful
<smoser> git reset --hard ubuntu/17.2-35-gf576b2a2-0ubuntu1_17.10.2
<smoser> new-upstream-snapshot
<smoser> dch --release --distribution=artful
<smoser> git push chad.smith HEAD --force
<smoser> basically,  lets just re-do it.
<blackboxsw> yeah I did basically that.
<blackboxsw>  git checkout -b ubuntu/artful a2ce41af52c619c13da66f84ee2a789b9df8f3a2
<blackboxsw> and re-ran everything
<blackboxsw> I'll use your git-fu here and try again
<smoser> yours is probably ok now.
<smoser> let me see. i thought you were saying it wasnt
<blackboxsw> In the UI I see merge conflicts
<blackboxsw> in debian/changelog
<blackboxsw> but all the other content is good
<smoser> yours is good.
<smoser> k. i'll just build and push and push over upstream
<blackboxsw> excellent. and the moral if this story is: Chad don't push branches late in the evening
<smoser> i'm quite happy with those changes to that script.
<smoser> i think we have covered most of the pain poitns of it.
<blackboxsw> smoser: I love it. though I throw eggs at any shell script.
 * blackboxsw will become better versed in shell someday.
<smoser> blackboxsw: fair enough. it is lots of variables and such in there. definitely "shell script"
<blackboxsw> smoser: right so although you pushed the upload, nothing showed in artful-proposed  yet so we're safe to re-push right?
<blackboxsw> I'm looking for the proposed  build queue in launchpad again
<blackboxsw> https://launchpad.net/ubuntu/xenial/+queue?queue_state=1&queue_text=cloud-init
<blackboxsw> or https://launchpad.net/ubuntu/artful/+queue?queue_state=1&queue_text=cloud-init rather
<smoser> i just uploaded artful
<blackboxsw> ok great so that list tells me you just re-submitted same unapproved upload 30 seconds ago
<smoser> yep
<blackboxsw> now do we ping in ubuntu-release for attention on artful/xenial?
<smoser> we can, yeah.
<blackboxsw> will do.
<blackboxsw> ok smoser I'm going to get to work on updating out manual cloud test scripts for this SRU https://hackmd.io/jlq3C4qbSgurZ_DZ5GTiuw?both
<blackboxsw> smoser: I hadn't tested review-mps yet on non-master branches. I didn't want to automatically botch the merges
<blackboxsw> all 3 branches are approved
<blackboxsw> I can use review-mps to land them later if you don't land them now
<ihre> thanks for the tip re: lxc/lxd blackboxsw, that is definitely an easier workflow than using qemu :)
<blackboxsw> good to hear ihre, I'll probably write up a blog post about it (and use of other cloud-init CLI commands=)
<ihre> i'll keep an eye out for it!
<blackboxsw> ok smoser landed your ubuntu/(artful|xenial) mps and fixed up qa-scripts:review-mps tool to properly handle --upstream-branch origin/ubuntu/*
<blackboxsw> thanks for the cloud-init release email  BTW https://lists.launchpad.net/cloud-init/msg00145.html
* blackboxsw changed the topic of #cloud-init to:  Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 4/2 16:00 UTC | cloud-init 18.2 released (03/28/2018)
#cloud-init 2018-03-29
<do3meli> anyone else expiriencing problems running tox on local branches after 18.2 release?
<do3meli> RuntimeError: Failed running ['/usr/bin/python3', 'tools/read-version'] [rc=1] (, git describe version (18.1-42-g2fb6008f) differs from cloudinit.version (18.2)
<smoser> do3meli: you need to pull the tag from upstream
<do3meli> allright, thx
<smoser> let me know if that doesnt work
<do3meli> it worked ;)
<smoser> k.  that can be annoying for sure.  its just there to make sure we dont mess up.
<blackboxsw> rharper: smoser, do we need to talk about what we want to land in bionic my EOD/tomorrow?
<smoser> blackboxsw: ok.
<blackboxsw> maybe just chrony?   our SRU process feels like it is getting easier and easier, so maybe it's no big deal if something misses freeze
<blackboxsw> forgot I have a 1:1
 * blackboxsw is working on SRU validation scripts in https://github.com/cloud-init/ubuntu-sru/ for the next ubuntu sru. specifically I'm changing our manual tests to include user-data test & validate some of the exception_cb changes that we know affect this release. 
<smoser> blackboxsw: around?
<dojordan> hey folks- does anyone know how to identify the commit hash of cloud init in the daily canonical images? i see https://cloud-images.ubuntu.com/locator/daily/ but that doesn't have release notes or packages
<smoser> dojordan: well, i would guess that mostly the serial in the Release column will line up with
<smoser> one in http://cloud-images.ubuntu.com/daily/server/xenial/
<smoser> i guess in an ideal world, the stream data for azure (http://cloud-images.ubuntu.com/daily/streams/v1/com.ubuntu.cloud:daily:azure.sjson)
<smoser> would have manifest files
<smoser> dojordan: which specifically were you after ?
<Odd_Bloke> dojordan: The serial in Azure will be aligned with the serial on cloud-images.u.c, so you can use that as a reference.
<dojordan> sorry, i guess what i was after was how do I specifically identify if com.ubuntu.cloud.daily:server:18.04 version 20180327 has the 18.2 release of cloud-init?
<dojordan> i.e. this manifest file
<dojordan> got it, makes sense
<dojordan> thanks all
<blackboxsw> smoser: back
 * blackboxsw jumps into cloud-init hangout for the day folks want to chat about SRU or bionic release stuff
<blackboxsw> merging do3meli's doc branches
<blackboxsw> smoser: I know you are out of here, hogarth's branch + review comment fixes are at https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342428
<blackboxsw> I'm going to rebase and merge my commit into his after a bit more testing on azure/ec2. so my separate commit will disappear.
<blackboxsw> I think I need to address rharper's comments on hogarth's branch too.
<smoser> blackboxsw: awesome.
<blackboxsw> smoser: I'm wrapping up rharper's and my missed review comments on it now.
<blackboxsw> had one minor fix for lxc.
<blackboxsw> ok rebased and pushed, testing ec2 and azure now
<blackboxsw> smoser: net-tools drop won't make it tonight, will get it in shape for next week. I see some discrepancies on ec2 ipv6 route print info (missing route link details when using ip vs route cmd output).
<blackboxsw> need to sort the absence of info on ec2 first so there'll be another change to the logic
<blackboxsw> in netinfo.py
<blackboxsw> got it, netdev_info_netstat calls netstat -A inet6 -rn  too. but netdev_info_iproute doesn't compile v6 routes
<blackboxsw> ok will have to work it out
<blackboxsw> I'm off til Monday. have a good easter folks
#cloud-init 2018-03-30
<blackboxsw> rharper: know why route -A inet6 has more ip6-localhost/multicast routes listed than 'ip 6 route show'? https://pastebin.ubuntu.com/p/vJD798wDJV/
<blackboxsw> https://pastebin.ubuntu.com/p/FfzWrJ6nzB/
<dj_drops> Hello all, I am newbie to clout-init but I started to experimenting. Basically everything working as I want - but I have problem with runcmd module. Script is correctly creater under /var/lib/cloud/instance/scripts/runcmd (I have in module simple: touch /tmp/it_works). Even when I run cloud-init single --frequency=always --name=cc_runcmd nothing happens. OS Debian9.4 and cloud-init from debian repo 0.7.9
<rharper> blackboxsw: you want ip -6 route list table all
<blkadder> Morning all. Is there a way to resize an AWS ec2 volume on launch with cloud-init? I know how to do it via aws cli but was hoping to avoid that.
<blkadder> EBS...
<blkadder> Sorry
<rharper> yeah, it should automatically
<rharper> sorry, it resizes root
<rharper> lemme look at config, I think it may also find attached EBS as well
<blkadder> My use case is that I have a AMI built with a 35GB root partition that I want to change to 250GB on launch for a specific role type if that helps.
<rharper> yeah, in cc_growpart
<rharper>  http://cloudinit.readthedocs.io/en/latest/topics/modules.html#growpart
<blkadder> Cool thx.
<rharper> it does root by default, but you can list other devices as well
<blkadder> I just want to do root so that's fine.
<rharper> cool
<rharper> there are restrictions w.r.t how you have your device partition,  is it a single partition ?
<rharper> that is the rootfs has to be on the "last" partition before the free space;
<blkadder> Yes, but looking at this, it looks like it will only grow the partition to fill space.
<blkadder> I need to change the size on launch.
<blkadder> Perhaps I need to use the aws cli for that when launching?
<rharper> oh, I see, you mean the actual device, not just the file system
<blkadder> Yep
<rharper> ah, yeah cloud-init doesn't call into to aws cli/api
<rharper> I think you can specify that at launch though
<blkadder> Ok, let me go figure that part out, thx.
<blkadder> Then I can just use growpart for the last bit.
<rharper> y
<blkadder> ty
<powersj> rharper: if I add an invalid key to cloud-config is there suppose to be an error of some sort or is it ignored?
 * powersj can't recall the behavior 
<rharper> powersj: in some cases we have schema
<rharper> so in bionic in the modules where it's enabled, like ntp, it'll throw an exception
<powersj> ah that's right!
<powersj> ok thanks :)
<rharper> in general, if you pass in an unknown key at the top level, you won't see any error
<rharper> if you specify an unknown value in sub dicts, also not likely to trip anything up save schema enabled modules
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/339438
<rharper> powersj: that's ready for test
<rharper> if you feel so inclined
<powersj> rharper: thx will be early next week most likly
<rharper> sure, no worries
<rascal999> I've got a cloud-config script I'd like to use with a VPS but the provider no longer supports cloud-init. Is it possible to install cloud-init and put the user data somewhere and reboot? Distro is Ubuntu
<rharper> rascal999: yes, you want to use the NoCloud datatsource
<rharper> http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<powersj> rascal999: are you the one who sent mail to the mailing list?
<powersj> if I recall he added the limitation of being unable to add a device
<rharper> that was VMware stuff
<powersj> oh look at that.. my bad
<rharper> well, it's somewhat related, the difference was wanting to genernate network config outside of the cloud-init boot process, which we don't currently support
<rharper> I've not yet replied to that on list
<rharper> it's something that could be added to the cli; just not there today
<rharper> somewhat complicated by *when* can you run scripts in your image;
<rascal999> rharper: thanks for this I'll have a look
<rascal999> powersj: I haven't raised this in mailing list
#cloud-init 2020-03-23
<andras-kovacs> Hi! Do you have any idea how can I disable cloud-init's resolv-conf module? I've added the manage_resolv_conf: false to my cloud.cfg but seems like cloud-init doesn't care about it. â¹ï¸ Thank you! (I'm using cloud-init-18.5)
<pmjdebruijn> .wc
<tribaal> account ran out of funds
<tribaal> bouncer died
<tribaal> oops, wrong chan. Apologies :)
<meena> tribaal: you don't have to apologize to anyone for *that*
<tribaal> hehe
<tribaal> Well, you'd think that working for a public cloud you would somehow be more disciplined with regards to paying your cloud bills... :)
<meena> imagine if you ran out of money, and the utilities company shut off your water and electricity tho, that would suck.
<meena> not saying an irc bouncer is as important as electricity, but we've been selling people on the idea that cloud is a commodity, but we treat it â¦ differently.
<tribaal> meena: that's a good point :) In this case however the blame sits squarely on myself and my email filtering rules: one of my account is "free" and antoher account is "paid" to test things as a normal user would, and my email rules dismissed the many  warning emails we usually send our users :)
<AnhVoMSFT> rharper blackboxsw is there anyway to ask cloud-init to apply network config again through command line post boot?
<AnhVoMSFT> (as in query metadata source, get networking information, render network config, apply it)
<Odd_Bloke> andras-kovacs: What behaviour are you trying to disable?
<Odd_Bloke> meena: The one time I've paid for electricity up-front (by topping up a card which plugs into the meter), that's exactly what happened when you ran out.
<meena> Odd_Bloke: i thought you lived in a more civilised countryâ¦
<Odd_Bloke> meena: Well, the advantage of the system is that the utility company doesn't know who you are (you literally go to a corner shop with the card and can pay cash to top it up).
<Odd_Bloke> The flip side is that they don't have anyone to chase for unpaid bills, so it does turn off when you don't have money on it.
<Odd_Bloke> Goneri: Travis failed to report results on #268 for some reason, looks like you have CI failures though: https://travis-ci.org/github/canonical/cloud-init/builds/665860962
<Goneri> Odd_Bloke, thanks.
<Odd_Bloke> meena: (But yes, I do believe that utilities should be publicly-owned and ~free. ^_^)
<Goneri> meena, https://github.com/canonical/cloud-init/pull/269
<meena> looking
<rharper> AnhVoMSFT: on the cli for re-rendering the network config to the system; that's not there at the moment. it wouldn't be too hard to do though; we have a sub-command 'cloud-init devel net-convert'  which could get extended to take a datasource name, and optional flag to attempt to read from object cache
<Smithx10> Hello fellow cloud-init'rs.    I was thinking of extending cloud-init to support void-linux
<Smithx10> it uses rc.d,  what is a good way to get started.... just look at distros/ ?
<cloaked1> powersj: still waiting for an answer on https://discourse.maas.io/t/how-to-mount-nfs-volume-via-preseed/1404 :)  Hopefully, the person who can answer the question becomes available today
<blackboxsw> Smithx10:  I think cloudinit.distros would be a good place to start.  NetBSD was just added a couple weeks ago. https://github.com/canonical/cloud-init/pull/62
<Smithx10> Yeah,  it would be nice to see how non systemd systems interface
<Smithx10> Thanks
<rharper> cloaked1: I just replied, let me know if you have any more questions in the discourse thread, I'll reply there
<cloaked1> Thank you rharper. Taking a look.
<Goneri> Odd_Bloke, could you drop the wip label? https://github.com/canonical/cloud-init/pull/147
<Goneri> Odd_Bloke, a review is also welcome :-)
<blackboxsw> done Goneri
<Odd_Bloke> Minor refactoring I found when digging into the archive mirror bug: https://github.com/canonical/cloud-init/pull/271
<blackboxsw> Odd_Bloke: rharper thoughts on this approach for Oracle non-initramfs distros? https://github.com/canonical/cloud-init/pull/174#pullrequestreview-379733515
<blackboxsw> I think we can avoid going doing EphemeralDHCP duplicate route setup in iscsi environments also by checking connectivity_url on Oracle only.
<blackboxsw> landed #271
<rharper> blackboxsw: did you see my email about daily ppa build failure ?
<blackboxsw> Odd_Bloke: shepherding question, are we waiting another week on robjo's feedback for https://github.com/canonical/cloud-init/pull/160 ?
<blackboxsw> rharper: no, looking now
<rharper> blackboxsw: thanks, it looks like the ec2 static patches aren't building correctly; they don't apply
<robjo> I was just about to look at that when something else popped up. I'll get to it tomorrow
<rharper> this is breaking all daily builds since it landed
<blackboxsw> robjo: excellent, thanks sir
<Odd_Bloke> Goneri: Label removed.  Apologies, thought Chad had done that eariler. :)
<Goneri> thanks. I just saw a couple of remaining comments from rharper. Otherwise the PR works fine.
<johnsonshi> Hey there, do you know what the syntax for a multiline runcmd is? Because of our specific use case, we cannot rely on the write_files module to write the files.
<johnsonshi> What I am trying to do is use the runcmd module to do a here-doc cat
<johnsonshi> cat <<EOF >> /some/file
<johnsonshi> But it wouldn't be proper YAML syntax if a runcmd command spans multiple lines. I couldn't find examples of runcmd with a multiline command. Thanks as always guys :)
<rharper> johnsonshi: have you tried write_files ?
<rharper> and you can template that cloud-config;  lemme see if I can work out a multi-line runcmd for you... I know we have a few of them; prepare for some yaml anchors
<johnsonshi> For our purpose, we need to use runcmd because we are need to run `sed -i`, and right after that, run `cat <<EOF`
<rharper> k
<Odd_Bloke> johnsonshi: YAML does support multi-line strings.
<Odd_Bloke> johnsonshi: https://yaml-multiline.info/ <-- you want the literal block style, I believe
<johnsonshi> Odd_Bloke: Thanks! Yeah I was trying out the other anchor (>) and it was wrong. Turns out (|) is what I needed. Thanks :)
<rharper> johnsonshi: https://paste.ubuntu.com/p/jvRWBkHxbN/    we make use of this  a lot in another project where we write quite a bit of shell in yaml;  this lets me write literal shell under my anchor and then just reference the anchor in a runcmd exec of sh or bash -c
<johnsonshi> rharper: Thanks. That syntax is neat and fits our purpose exactly. :)
<rharper> cool
<Odd_Bloke> johnsonshi: Yep, it's one of those things that once you know you know, but if you don't know what you're looking for then it's super hard to find. :)
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/272 is the tests that I'll be modifying as I make my changes for the archive mirror selection bug we've been working
<Odd_Bloke> Figured I may as well submit them separately, as they're useful regardless of the rest of the work (and I'm almost EOD).
<Goneri> meena, could you take a look at this one? https://github.com/canonical/cloud-init/pull/273
#cloud-init 2020-03-24
<eggbean> I usually have a bash script for my aws ec2 user script. I have just confugured an instance with EFS file storage, so it has added a load of #cloud-config yaml.  How do I include my bash script to this?
<eggbean> I suppose I should just rewrite everything in the bash script to be done in  cloud-init instead?
<eggbean> But is that the only way?
<andras-kovacs> <Odd_Bloke "andras-kovacs: What behaviour ar"> Hi Odd_Bloke ! If I give an immutable flag for the resolv.conf ( I know it's ugly as hell but I had to :'( ) cloud-init-local will fail whatever I set in the cloud.cfg. I want cloud-init to completely leave alone the /etc/resolv.conf. I can make it only happen by disable cloud-init completely after the first run (with a runcmd).
<andras-kovacs> cloaked1: are you sure in the suid flag on the nfs mount?
<robjo> blackboxsw: Odd_Bloke #160 reviewed, just a couple of comments, overall LGTM. Thanks
<Odd_Bloke> eggbean: You could try moving your shell script to a runcmd stanza: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#runcmd
<eggbean> Odd_Bloke: Thanks.  I'll have a read.
<Odd_Bloke> Alternatively, you could investigate passing multipart user-data to cloud-init, which would allow you to leave the generated YAML completely alone.
<Odd_Bloke> But runcmd is definitely easier if it works for you.
<eggbean> okay, cheers
<Odd_Bloke> andras-kovacs: Could you pastebin the traceback you (presumably) get in /var/log/cloud-init.log?  That'd hopefully give me a hint as to what is poking at resolv.conf.
<Odd_Bloke> robjo: Thanks!
<fholmes> Does anyone know how I can make the hostname equal the vm name in vsphere using cloud-init?  I have a template I have setup with govc guestinfo.metadata and guestinfo.userdata strings.
<fholmes> Not sure if I need to customize those either manually or a custom script or if I can use jinja2 templates in cloud-init to pull this off.  I don't really see any queriable instance-data atttributes I can use for vsphere (esxi or whatever).
<Odd_Bloke> fholmes: I'm not familiar with VMWare products, but you can see the available attributes by running `cloud-init query -a`.
<fholmes> Thanks Odd_Bloke!  I will check that out and see what i can come up with then.
<Odd_Bloke> :)
<Odd_Bloke> Heads-up: Travis are having issues: https://www.traviscistatus.com/incidents/ntxbkxt8lrx9
<fholmes> Well ultimately the information is not there that I need, so I guess it is a custom script then.  Thanks again Odd_Bloke, I really appreciate the assistance.
<Odd_Bloke> fholmes: Are you aware of the runcmd: stanza, to execute the script for you as part of cloud-init's run?
<fholmes> Yes, I definitely am
<Odd_Bloke> OK, good. :)
<andras-kovacs> fholmes: it sounds interesting. Which name do you want to make static?  (place in /etc/hostname)? The "local" short one or the FQDN? For me, cloud-init prints the FQDN there.
<fholmes> @andras-kovacs I want to use the name of the vm that shows up in the vcenter client to be the local-hostname of the deployed VM.
<fholmes> For instance govc vm.clone -vm ubuntu-1804-kube-v1.17.3 kube-master-1.   I want the hostname to be the kube-master-1 basically.
<Odd_Bloke> meena: Last call for comments on https://github.com/canonical/cloud-init/pull/160
<Odd_Bloke> (I'm going to land it in the next couple of hours.)
<Odd_Bloke> (Of course if you do have any comments after that, I will still listen to you. ;)
<andras-kovacs> <Odd_Bloke "andras-kovacs: Could you pastebi"> so I have manage_resolv_conf: 0 in my cloud.cfg, there is an immutable flag on the resolv.conf and cloud-init-local.service gets failed on every reboot. Here are the logs from /var/log/cloud-init.log https://pastebin.com/szz2QaB9
<andras-kovacs> I'm wondering... which is better? the FQDN or the local_hostname to store in /etc/hostname? Cloud-init puts the FQDN there in my VMs.
<Odd_Bloke> rharper: ^ sounds very familiar, but I can't find a bug report about it.
<fholmes> Well this seems to do the trick.  https://pastebin.com/aHVJanmW
<Odd_Bloke> Nice!
<fholmes> Would be nice to see better support for vsphere, but beggars cant be choosers!  lol.  I know there most likely has to be a better way honestly.  This is my first real foray into cloud-init though so I still have a lot to learn here.
<meena> somebody please explain to me how y'all are able to focus on work
<fholmes> Very easy.  I have worked from home for the past three years at least and this is a normal day for me.  If you look at the facts instead of watching the news there is an overall death rate of ~1%.  China has stopped the spread of the virus in their country so if we all just follow some basic common sense we are going to get through this in one
<fholmes> piece.
<rick_h_> heh, little harder than that but someone once said something famous about accepting things I cannot control, taking care of business when I can, and such.
<Odd_Bloke> rharper: Travis had failed to report status on #274, so I've just kicked off one of the sub-jobs again to try and get Travis to report correctly without a full re-run, FYI.
<andras-kovacs> meena: :D I've chosen a job which is kind of my hobby also.
<andras-kovacs> fholmes: my colleague from China told me it's 1-2 weeks from now and they will reopen their office
<fholmes> @andras-kovac That is definitely good news!  I am truly hoping their lives can get back to normal as quickly as possible.  Along with the rest of the world.
<fholmes> And yes, I was lucky enough to find my passion in computers at 8 years old, about 34 years ago. lol
<andras-kovacs> but my other colleague from India told me... it's getting harder to get groceries
<andras-kovacs> fholmes: you can be still that kid with 34 years of experience
<fholmes> lol, I am definitely that kid in a candy store when it comes to my daily life.  :) . Learning new technologies is what I live for each day and I am blessed to be in a position where I get paid to do it and share it with others.
<andras-kovacs> I wish my colleagues would be the same as you with the attitude
<fholmes> Yes, there are no beans and rice anywhere here in Arizona I am sure of it.  Food is still available but the store shelves empty very quickly.  I do have three kids and one of them just finished chemo for cancer and they are concerned it might have spread.  So ultimately I have more pressing things to worry about honestly.  However, he is
<fholmes> immunocompromised but I refuse to let the fear of him getting sick enter my mind.  Luckily the county where we live have 0 reports so hopefully everything works out.
<fholmes> Yes, I feel you @andras-kovacs it is painful working with many people in this industry.  I was very lucky to have gotten my current job.  There are no egos to content with on my team and everyone loves what they do.  It is definitely a very refreshing change and extremely rare in this industry.
<fholmes> s/content/contend
<andras-kovacs> I wish the best for your kid. I hope he will get better soon! My mother had cancer also. It took time but luckily she is well now.  And I wish great power for you!
<andras-kovacs> yes, the IT industry nowadays like... bunch of people work for the money only while they are hating it. If you are in a different environment... that can be good.
<fholmes> @andras-kovacs Thank you very much.  I am positive he will be fine.
<smoser> someone on my local user group (http://www.mug.org) just linked to https://canonical.com/careers/2075752
<rick_h_> smoser:  yea, I shared it in the old ubuntu-us-mi channel
<rick_h_> howdy smoser
<smoser> o/ rick_h_
<smoser> just make sure people *here* are aware. its a good company to work for and a good team to be on.
<rick_h_> smoser:  cool yea that's awesome. Hopefully can find some awesome folks for it. Thanks for the plug in here.
<Odd_Bloke> rharper: https://github.com/canonical/cloud-init/pull/275#pullrequestreview-380626877 <-- reviewed
<cloaked1> I'm curious. I'm trying to set bonds up for interfaces on a couple of machines and for some reason I keep getting the error "All parents must belong to the same VLAN" which doesn't make sense. What does "parents" mean in this context?
<cloaked1> I found the code: https://github.com/maas/maas/blob/master/src/maasserver/forms/interface.py#L574
<cloaked1> but that still doesn't make sense. This error is senseless to me.
<cloaked1> maybe one must put the individual interfaces in the right VLAN before creating the bond in the target VLAN?
<cloaked1> that's it. That error is horribly ambiguous.
<rick_h_> cloaked1:  sounds about right. Sanity checking the interfaces can both participate in the bond I'd guess
<cloaked1> yeah, at the least the error should be better worded.
<cloaked1> not even a google check was very helpful per se.
<rharper> cloaked1: generally you want to vlan over a bond.  so the bond will have a list of interfaces, and then the vlan would specify the bondX as it's vlan_link,
<cloaked1> indeed. I'm aware of that.
<cloaked1> the issue was that I needed to preset the interfaces to the desired VLAN first and then set the bond
<rharper> is this a MAAS ui issue then ?
<cloaked1> that workflow was not understood.
<rharper> as that confused me coming from the bottom  up at least
<cloaked1> I tried setting the desired VLAN during the time of bonding; the VLAN of which was different than wat the interfaces were on in the first place.
<powersj> hmm blackboxsw were we suppose to do a status meeting today :\ are off a week because of the sprint
<rharper> powersj: IIRC blackboxsw set it for every day this week
<cloaked1> rharper: not sure but it doesn't seem right.
<rharper> so maybe it's a mystery as to which day ?
 * blackboxsw checkes oooh right :) cloud-init status roulette now
<rharper> cloaked1: you point to the form, so I'm guessing UI;  versus some sort of network config error coming from cloud-init or netplan, etc
<blackboxsw> I 'fixed' the invite for status and set it today, and forgot to host it.
<blackboxsw> sooo, shall we hold the cloud-init status meeting now?
<rharper> I suspect there's some database relation going on, in that a bond may select any of the member interfaces at any time, so guess they want the physical interfaces to be included in the same vlan (maas internal I suspect)
<cloaked1> yes, correct. It was UI. Should I create a bug?
<rharper> cloaked1: yeah, confusion is confusion =/
<cloaked1> ok. I'll create a bug report.
<eggbean> I am putting bash script commands into #cloud-config,runcmd, but maybe some things can be done in pure cloud-init?  adding/modifying users, setting timezone, locale, hostname associating elastic IP, modifying hosts file, rebooting... which of these can be done in cloud-init?
<powersj> blackboxsw, I think we get more folks in the US morning
<powersj> eggbean, have you looked through some of the cloud-config examples? https://cloudinit.readthedocs.io/en/latest/topics/examples.html
<eggbean> powersj: thanks
<powersj> eggbean, here are the docs for every module: https://cloudinit.readthedocs.io/en/latest/topics/modules.html
<eggbean> cheers
<rharper> eggbean: almost all of those have explicit cloud-config
<eggbean> cool
<rharper> the "hostname associating with elastic IP" is new to me, so maybe not that one
<eggbean> rharper: missed a comma after hostname
<rharper> ah;  I don't think we have a module for the elastic ip association;  but I've not done that via command line so not sure what command to use
<blackboxsw> powersj: rharper Odd_Bloke shall I then shift status meeting to tomorrow (my holiday), Friday or Tuesday next week?
<eggbean> aws cli.  Looks like I'll still be using runcmd for that then
<rharper> yeah
<powersj> blackboxsw, let's bump to next tuesday
<powersj> thanks
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting March 31 16:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> thx for the uss-tableflip cherry-pick branch review Odd_Bloke/rharper. Wrapping those comments up now.
<cloaked1> @here https://bugs.launchpad.net/maas/+bug/1868868
<ubot5> Ubuntu bug 1868868 in MAAS "UI: bonding NICs on different VLAN causes ambiguous error" [Undecided,New]
<cloaked1> hopefully that is worded well enough for your understanding. If it is not, let me know and I can work through it.
<andras-kovacs> is there a way to use the meta data in the hosts.*.tmpl files?
<blackboxsw> interesting questions andras-kovacs, generally with the user-data, we allow any keys from 'cloud-init query --all' using a #template: jinja header before #cloud-config and referencing variables like  {{ v1.cloud }} in your template files. Example here https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data
<blackboxsw> andras-kovacs: what tmpl file are you looking to use substitution on. I can double check quickly
<andras-kovacs> I've added to this file: /etc/cloud/templates/hosts.redhat.tmpl
<andras-kovacs> this line: {{ds.meta_data.local_ipv4}} {{fqdn}} {{hostname}}
<andras-kovacs> But it's not working.
<andras-kovacs> tl;dr app team asked for a line with the ip address of eth0 with the FQDN name. cloud-init makes it like: 127.0.0.1 {{fqdn}} {{hostname}}
<blackboxsw> andras-kovacs: it's worth a bug,  I think I may have only turned on that jinja-template handling of instance-data.json variables for cloud-config user-data. It would be worth extending that to other template rendering I think.
<andras-kovacs> blackboxsw: ok, thank you! I can go with runcmd of course but IMHO it would look more sophisticated with a template
<andras-kovacs> where should I raise the bug? in GitHub?
<blackboxsw> +1 andras-kovacs right. And I assume you've grok'd that runcmd can be a jinja-template too right ?
<blackboxsw> ## template: jinja
<blackboxsw> #cloud-config
<blackboxsw> runcmd:
<blackboxsw> andras-kovacs: we track bugs in Launchpad https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> we don't use issue tracking in github yet
<andras-kovacs> yes i saw that but to be honest..  I was using it like: - cloud-init query region
<andras-kovacs> so I should add the #cloud-config and ## template: jinja lines in the /etc/cloud/cloud.cfg also?
<rharper> cloaked1: thanks
<blackboxsw> andras-kovacs: nope on that file, it is sources essntially cloud-config yaml anyway. But another interesting application would be to allow jinja templating /etc/cloud/cloud.cfg too on a filesystem.  I don't think that is currently handled either
<rharper> andras-kovacs: for bugs we track them in launchpad,  https://bugs.launchpad.net/cloud-init/+filebug
<rharper> ah, blackboxsw already mentioned that, sorry;
<blackboxsw> I think we 'could' extend /etc/cloud/cloud.cfg and cloud.cfg.d to support jinja-template files, but I'm not sure the machinery applies there yet
<blackboxsw> no worries rharper thx. And I think it's worth a separate bug/feature request to support jinja-templates in /etc/cloud/cloud.cfg* files
<andras-kovacs> ok "and that's the story how I sold my soul to Canonical" :D
<andras-kovacs> yes, probably I gave it a try but it printed my variable names so that's why I'm using the query command in cloud.cfg. Thanks, I remember now.
<rharper> blackboxsw: hrm, well;  it's not clear to me that we can
<rharper> we read and source system config (/etc/cloud/*) which sets up things like datasource lists and other things that are unrelated to the instance
<rharper> so I worry that folks would jinja things that cannot be expanded
<rharper> existing template files /etc/cloud/templates/*  I think make sense as they're consumed after we're on a ds and have an instance (including metadata)
 * rharper gets some coffee , brb 
<andras-kovacs> so how people generally use it? I didn't wanted to make a special use-case here
<andras-kovacs> I mean... I know here we are using chef and ansible most of the time but I like make things easier and if cloud-init can make these things happen from the beginning, I don't want to make it more complicated
<andras-kovacs> If I could include the basics in my VM templates than everybody would be happy and I could evangelize cloud-init :D
<blackboxsw> ahh right rharper, trying to render jinja template before we have instance data (like setting the original datasource_list which ds-identify consumes) would be putting the cart before the horse.
<andras-kovacs> I got your point but when I run the runcmd commands I have everything.
<blackboxsw> though it's possible we could support some level of templating in /etc/cloud/cloud.cfg* for certain use-cases I think, especially if the templates were not depending on specific instance-data from 'cloud-init query'.
<blackboxsw> And right andras-kovacs at runcmd time, you have everything.
<andras-kovacs> oh I see sg. else too!
<blackboxsw> at any cloud-config stage after the datasource is detected you'll have instance-data (the metadata variables), (which is either cloud-init-local or cloud-init-netwok stage depending on your cloud)  https://cloudinit.readthedocs.io/en/latest/topics/boot.html
<blackboxsw> I mean "at any cloud-init stage*"   not cloud-config stage
<andras-kovacs> so should I need to file a bug? I don't want to waste your / anyone else's time
<blackboxsw> andras-kovacs: I think it's worth one bug on trying to use instance-data templates for your hosts.tmpl file
<andras-kovacs> yes, I got that
<blackboxsw> it's not a waste if we respond and decide it's not worth it for upstream, because it will document that decision for others. but, specifically at host rendering time based on distro, I *think* we are already past accumulating and persisting instance-data
<blackboxsw> so it might be low hanging fruit that we could provide a feature for
<blackboxsw> or something a "cloud-init evangelist in the community" could help with ;)
<andras-kovacs> I really like the simplicity and the way how cloud-init works I don't want to mess it up with my requests
<blackboxsw> no worries. never a mess up to have feature requests
<blackboxsw> gives us a stream of suggestions to prioritize
<andras-kovacs> like... it's great you haven't forget to hit a systemctl daemon-reload before mounting with the mount module
<rharper> andras-kovacs: I'm +1 on allowing instance metadata use in /etc/cloud/templates/*  files ;  we need to think a bit more on the general /etc/cloud/ system config files
<rick_h_> lol blackboxsw way to slide that in there
<blackboxsw> heh
<blackboxsw> rick_h_: /me likes the instance-data and json schema drum :)
<blackboxsw> it's my favorite song :)
<andras-kovacs> I would buy that CD than warez download the mp3 version and listen to it only on spotify(on-demand).
<powersj> lol
<blackboxsw> Odd_Bloke: comment on https://github.com/canonical/cloud-init/pull/277
<blackboxsw> hah!
<rharper> andras-kovacs: haha, awesome!
 * cloaked1 loves IRC.
<cloaked1> slack is good and all but nothing beats IRC imo.
<blackboxsw> Odd_Bloke: https://github.com/canonical/cloud-init/pull/272#pullrequestreview-380713598 approved, just needs rebase
<blackboxsw> and merge
<Odd_Bloke> Thanks!
<blackboxsw> https://github.com/canonical/cloud-init/pull/277 approved for real too Odd_Bloke
<Odd_Bloke> Nice, thank you.
<blackboxsw> can circle back if needed to point folks at your existing pytest example
<rharper> https://github.com/canonical/cloud-init/pull/275  now has another hefty example of pytest.mark.parametrize
<blackboxsw> rharper: +1 excellent, hadn't seen it yet.
<blackboxsw> rharper: I didn't fully grok your review comment.https://github.com/canonical/cloud-init/pull/267#discussion_r397373585 isn
<blackboxsw> per your comment "We want to allow a local cloud-config file to set the the policy. "   isn't my branch already sourcing the merged config from the 'local cloud-config' on disk and overriding distro defaults?
<blackboxsw> or are you suggesting we shouldn't have distros even define a preferred_renderer_priority and we only do it in distro image default /etc/cloud/config.cfg
<andras-kovacs> (I like this word now: grok. Thanks!)
<blackboxsw> grok: "understand (something) intuitively or by empathy"    I'm all about empathy... :)
<blackboxsw> rharper: I'm not seeing where you mention we have a default policy set for network renderers in ubuntu. I know we set it for *bsd but the typical Ubuntu cloud-images aren't adding network: renderers to the /etc/cloud/cloud.cfg that we write in cloud-images.
<blackboxsw> we *can* and it would make the patch simpler if we dropped the Distro.preferred_renderer_priority() if you don't think we should add it (since we prioritize merged_cfg anyway if present
<andras-kovacs> oh before I forget here is what I've added to my runcmd:
<andras-kovacs>  - cloud-init query -f "{{ ds.meta_data.local_ipv4 }} {{ ds.meta_data.public_hostname }} {{ v1.local_hostname }}"
<andras-kovacs>      | tee -a /etc/hosts /etc/cloud/templates/hosts.redhat.tmpl
<andras-kovacs> and it works just great! thank you guys!
<blackboxsw> nice andras-kovacs, excellent to hear
<andras-kovacs> this way I could eliminate some custom shell scripts and systemd units
<andras-kovacs> Very many thanks! I've added you on linkedin, I hope you don't mind.
<blackboxsw> good, good andras-kovacs. my linked in is generally stale except when we are hiring :)  per here https://www.linkedin.com/posts/chad-t-smith-b8a5b54_ubuntu-server-software-engineer-activity-6639574095736426496-u5ja
<andras-kovacs> blackboxsw: thanks!
<andras-kovacs> Could you tell me what "resize_rootfs_tmp: /dev" does in the cloud.cfg? I see it belongs to the resiztefs module but I can't figure out the meaning of it.
#cloud-init 2020-03-25
<andras-kovacs> Is that a good assumption the CA Certs module can work on ubuntu and debian only?
<Odd_Bloke> andras-kovacs: I believe it depends on tooling that is only present in Debian derivatives, yes.
 * andras-kovacs sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/AuwhMuZRuZyPzlqOuLWFhGUG >
 * andras-kovacs sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/YqvjIFpIhuWgYdZbGgNSEieo >
<andras-kovacs> hmm maybe it's time for me to try to contribute to this project
<Odd_Bloke> andras-kovacs: Contributions would certainly be welcome!
<apollo13> hi there, can anyone tell me what the diff between nocloud, genericcloud and generic in https://cloud.debian.org/images/cloud/buster/20200210-166/ is?
<apollo13> ah found it here: https://salsa.debian.org/noahm/debian-cloud-images/blob/master/README.md
<ananke> does the proxy directive in apt module support yum?
<ananke> I guess I could try and see
<ananke> actually, it appears it doesn't. dang
<Odd_Bloke> ananke: It would be surprising if anything in the apt module supported yum. :)
<ananke> Odd_Bloke: well, I was going by the notion that apt_update and apt_upgrade are aliases
<ananke> and there is no other module that deals with package manager options. hmm, not sure if i'll be able to use 'packages' module then, without being able to set a proxy first
<ananke> runcmd runs afterwards
<powersj> ananke, there is a yum module
<Odd_Bloke> Yep, they're aliases, but in cc_package_update_upgrade_install which is generic.
<Odd_Bloke> Looks like the yum module only adds extra repos, rather than allowing config like proxy to be applied.
<ananke> powersj: there appears only yum_repos, is that what you're referring to?
<Odd_Bloke> ananke: Would a bootcmd work for you?
<Odd_Bloke> (It's runcmd but earlier.)
<ananke> Odd_Bloke: potentially, that's a good idea
<ananke> would you happen to know if write_files is before packages or afterwards?
 * ananke should just look at a sample failed host
<Odd_Bloke> Check /etc/cloud/cloud.cfg to see the configured order.
<Odd_Bloke> (Alternatively, run https://github.com/canonical/cloud-init/blob/master/config/cloud.cfg.tmpl through your brain's Jinja2 parser. :)
<ananke> our AMI build environment is behind a squid proxy, which makes a lot of things like that challenging
<ananke> ahh, good, write-files is much earlier. it's at the beginning of cloud_init_modules, while package-update-upgrade-install is towards the end of cloud_config_modules
<ananke> yay, that worked. sadly yum doesn't appear to support dropping in a file, so I have to change existing /etc/yum.conf
<ananke> on that subject, does write_files 'path' directive create missing dirs?
<rharper> Odd_Bloke: the cla check says:
<rharper> 3s
<rharper> We are currently unable to download the log. Please try again later.    -- what does that mean ?
<rharper> ananke: yes, it effictively runs "mkdir -p $(basename file)"
<rharper> ananke: it also has an append mode
<ananke> rharper: thanks! appears to be working like a charm
<ananke> had to create a systemd environment file for docker, so it can use proxy
<Odd_Bloke> rharper: No idea!
<rharper> ananke: nice trick!
<rharper> Odd_Bloke: did we already land something from garlof ?
<rharper> I was looking at the mime-data type bug and updated that with a fix, but noticed the cla check had a weird result
<rharper> I "reran" the checks bit it stayed the same
<ananke> rharper: that appears to be the sanctioned method. here's how the net result looks like with cloud-init: http://dpaste.com/3D3B7XC
<rharper> cool
<Odd_Bloke> rharper: I see a couple of commits from Kurt Garloff.
<rharper> Odd_Bloke: ok, I thought I remembered that.    So I wonder what to do with the check failing though;  we have time, waiting for garlof to respond
<Odd_Bloke> rharper: So the process is (correctly) failing on one of the PRs because we don't have garloff in our in-repo lists of CLA signers.
<Odd_Bloke> (I don't know why it's failing to check out on the other.)
<Odd_Bloke> But regardless, I think you'll need to email them on their CLA-signing email address asking them to confirm that they control the GH account in question.
<rharper> Odd_Bloke: ok
<Odd_Bloke> (And then update the in-repo metadata, of course.)
#cloud-init 2020-03-26
<faa> hello, maybe someone has an example debian double network interface?
<johnsonshi> rharper: As discussed previously, cloud-init's swap file module runs before the mount module. Cloud-init swap create also does not ensure that it has a path to the file, thereby throwing an exception. https://bugs.launchpad.net/cloud-init/+bug/1869114
<ubot5> Ubuntu bug 1869114 in cloud-init "swap module runs before mount module" [Undecided,New]
<faa> help required, debian, may cloud-init write file (/etc/network/interfaces.d/enp0s0) previous network stage? datasource NoCloud iso
<andras-kovacs> imho by default it makes a config for your interface but what is the problem you are facing right now?
<faa> debian if interface not status link up, cloud-init ignor this interface (only write config) require restart network
<faa> log https://pastebin.com/gv6HgpjY first interface link up in image template
<andras-kovacs>  enp0s1 |  True |         127.0.0.1 that's strange, doesn't it?
<andras-kovacs> are you using NetworkManager?
<faa> ip changed, not standart debian 10 service, with wery old cloud-init
<andras-kovacs> sorry, I can't follow you
<andras-kovacs> if you have problems with the old network.service try to disble it and use NetworkManager instead (but I'm not 100% surre how it looks like in Debian nowadays)
<faa> it's minimal server install without external packages
<nrajasekhar> Hello
<nrajasekhar> I need some help with cloud-init usage on suse
<nrajasekhar> I want to change the password on first login after creating the aws instance
<nrajasekhar> How can I achieve it
<nrajasekhar> any help here is much appreciated
<andras-kovacs> do you want to change only or store it in a meta service somewhere?
<andras-kovacs> I would change it with a runcmd command maybe
<nrajasekhar> change and save it
<andras-kovacs> I think you can find a whole solution with google for that
<nrajasekhar> yes, I have tried all the available options. none of them worked for me
<nrajasekhar> I have referred the link that I am pasting here : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/installation_and_configuration_guide/setting_up_cloud_init
<andras-kovacs> I would call a simple shell script which would set up a new password ,encrypt it with my ssh pubkey and upload it to the meta data server
<andras-kovacs> first things first, are you trying to make it work with Atomic?
<nrajasekhar> sorry but I am not sure what that option mean from the link.
<andras-kovacs> which option?
<nrajasekhar> password: atomic
<andras-kovacs> this how-to while looks perfectly good for me I think it's a general one and not for AWS exactly
<nrajasekhar> oh ok
<nrajasekhar> from open build service, I have taken sample template and modifying to achieve it
<nrajasekhar> but none of the options worked
<andras-kovacs> I don't know that one
<nrajasekhar> could you please suggest any method or example from a link?
<andras-kovacs> but we should know that what do you want to achieve
<nrajasekhar> https://build.opensuse.org/
<nrajasekhar> ok let me explain
<nrajasekhar> we generate Appliance in the OVA format
<andras-kovacs> why?
<andras-kovacs> I mean why in ova? do you use virtualbox or what?
<nrajasekhar> that's the way we deliver it to customers
<nrajasekhar> recently my team has decided to deploy our product on to AWS
<andras-kovacs> idk but ova ususally means virtualbox which also means IDE attached disks
<nrajasekhar> upon googling I found that OVA can be used to create AWS ami and an instance
<nrajasekhar> https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html
<nrajasekhar> I could able to successfully deploy OVA and create an instance out it and access my application
<andras-kovacs> than what's the problem?
<nrajasekhar> but in the OVA, we have hardcoded the root password
<andras-kovacs> do you want to randomize some passwords?
<nrajasekhar> now we want to make sure on first login, the user changes the password and save it
<andras-kovacs> finally! :D
<andras-kovacs> so this is what you want here
<nrajasekhar> yes :-)
<andras-kovacs> you just need to expire the password of that account I think and that's all
<andras-kovacs> https://www.tecmint.com/force-user-to-change-password-next-login-in-linux/
<andras-kovacs> you can put it in a runcmd command also but I would set it in the "ova" before I upload it to AWS.
<andras-kovacs> I mean you can do it without cloud-init also
<nrajasekhar> ok
<nrajasekhar> out of curiosity, Is there any option in clou-init?
<nrajasekhar> *cloud-int
<nrajasekhar> *cloud-init
<andras-kovacs> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#users-and-groups
<andras-kovacs> anything else you want there is runcmd for that
<nrajasekhar> sure I will give a try.
<nrajasekhar> @andras-kovacs: thanks for all the help and info.
<andras-kovacs> ywc!
<Goneri> the CI is broken because of https://github.com/gabrielfalcao/HTTPretty/issues/397
<Goneri> a 1.0.2 release has just been pushed.
<Odd_Bloke> nrajasekhar: You should consider whether having a password in a cloud environment is appropriate at all, but https://cloudinit.readthedocs.io/en/latest/topics/modules.html#set-passwords will allow you to set passwords and by default they should require resetting on first login.
<Odd_Bloke> Goneri: I've just restarted the CI for your PR.  Assuming that new version has been released, we should pick it up automatically.
<Odd_Bloke> andras-kovacs: (Thanks for helping out!)
<Goneri> thanks
<Odd_Bloke> And yep, it's got past the point it failed before.
<Goneri> Odd_Bloke, once this patch is merged, I will clean up cloudinit/sources/DataSourceNoCloud.py to remove the OS specific code we use to build devlist.
<Goneri> it should be in cloudinit/util.py
<Odd_Bloke> OK, nice!
<Goneri> and the find_devs() method should be in cloudinit/distros/
<Goneri> since it's OS (and distro) specific
<eggbean> Is there a vs code extension for #cloud-config?
<eggbean> Can't find anything
<andras-kovacs> it's not so complex IMHO
<andras-kovacs> # vim:syntax=yaml
<andras-kovacs> so just use yaml syntax and that's all
<eggbean> andras-kovacs: yeh okay.  It's just that as I am new to it, autocomplete would have been useful to remind me of the exact keys
<andras-kovacs> I see :D
<amansi26> Hi, I have a doubt . I am using ConfigDrive as datasource and deployig a DHCP network. netcfg.get('config') in stages.py is coming to None. Distro is rhel and cloud init version is 19.1.
<amansi26> can someone lease guide me
<amansi26> please*
<andras-kovacs> wait a minute
<andras-kovacs> so it's rhel 8, right?
<andras-kovacs> in 7 the latest is 18.5 as I remember
<amansi26> it is rhel 7.7
<andras-kovacs> oh wow
<amansi26> It is just failing for DHCP network. Static is working fine
<andras-kovacs> I'm not sure I understand why do you need to "pull" the DHCP config
<andras-kovacs> the DHCP server doesn't supply the necessary data?
<andras-kovacs> I mean I don't get the deploy part. Cloud-init makes a dhcp config for your interface and that should work.
<amansi26> Didn't get your last message
<Odd_Bloke> eggbean: I'm not aware of any such extension.  There is JSON Schema for some of the cloud config modules, you might be able to do something with that?
<Odd_Bloke> rharper: You still have requested changes on https://github.com/canonical/cloud-init/pull/147, which I believe are all addressed.  Could you either dismiss your review or do enough to give it an Approve?
<rharper> lemme look
<robjo> rharper: what file contains the more or less complete jsonschema for network and routing configuration?
<rharper> we've no schema written for network config v1;  netplan (v2) I think has schema in source;
<robjo> OK, I guess I have to keep winging it :(
<rharper> Odd_Bloke: maybe I'm missing something, but on conversation page, I see my requested changes/comments, there's a 'view changes' button, which almost always ends up on a 'whoops can't find your changes' page ...  what is that supposed to do?
<blackboxsw> Odd_Bloke: approved (but needs rebase) https://github.com/canonical/cloud-init/pull/278/files
<Goneri> rharper, is there anything else I need to adjust? https://github.com/canonical/cloud-init/pull/147
<rharper> Goneri: thanks, I'm re-reviewing
<bwatson> Anyone have any luck bootstrapping a RHEL 8.1 or CentOS 8 generic cloud image (kvm) via cloud-init on VMWare?  I add the *.iso as a virtual CD-ROM to the machine and power it on, but the only thing I see on screen is: Probing EDD (edd=off to disable)... ok
<bwatson> or is EL8 just not ready for this yet?  I've been using this technique with EL 6/7 and Ubuntu 16/18 for some time now
<Goneri> rharper, thanks. I'm testing a fix.
<Goneri> rharper, https://github.com/canonical/cloud-init/pull/147/commits/02694900dd656b94ed056a03952eeab276aa2694
<Goneri> rharper, I don't default on DHCP anymore.
<rharper> Goneri: ok, dhcp on just one interface then ?
<rharper> dhcp_interfaces() does this  ever return more than one ?
<Goneri> it depends on the metadata
<Goneri> so yes you can get more than one, in this case the defaul route will be indeed important
<Goneri> but it's not different to what we do we FreeBSD or NetBSD.
<Goneri> I would say, we cannot fix the network environment for the user.
<rharper> well,  they don't have control over it in a cloud
<rharper> in Azure, for example, you have to DHCP on primary nic, and secondary nics can also DHCP but the route can break the primary nic configuration;  so we ensure the network-config we write to the OS has a metric value that's lower priority for non-primary nics
<Odd_Bloke> rharper: My guess is that a force-push means that the commits references are no longer present.
<rharper> Odd_Bloke: yeah; I was thinking that;  seems like if a force push happens the message should go away since its not a great experience;
<Goneri> rharper, if you've got two DHCP network without a default route, bad things will happen.
<rharper> no, they * both* have a default route
<Goneri> but it's already the case with the other BSD. I'm not sure how I can fix that.
<rharper> they may even point to the same router; but the question out of which interface does the packet egress
<rharper> on Azure for example, packets destined for the internet may *only* come from eth0;
<rharper> they are dropped otherwise
<rharper> so the routing table entries matter
<rharper> multi-nic DHCP needs help since the platform network metadata is imprecise (they just say DHCP on all of the nics)
<Odd_Bloke> blackboxsw: Thanks for the review!
<rharper> cloud-init helps out here by ensuring that we don't clobber the default route for the primary interface
<rharper> on AWS, this can happen as well and on OpenStack
<rharper> I would prefer to *skip* multi-nic DHCP until we know that we won't break primary nic networking configuration
<Goneri> rharper, so basically,: if NIC is not primary and default_route_exists, then ignore default route
<rharper> yes; the preference is to assign a route metric of lower priority for all nics but primary
<rharper> or you can use route-tables
<Goneri> ok FreeBSD and NetBSD need to be adjusted too then.
<Goneri> Could we put that aside for now and merge the current patch. I will prepare a set-up to test the setup that you describe.
<Goneri> and come with a new patch that address this problem.
<rharper> Goneri: I think that's reasonable since it's a general fix for all BSD networking
<Goneri> I would also like to push a fix to clean up the devlist mess.
<rharper> https://github.com/aws/ec2-net-utils/blob/master/ec2net-functions
<rharper> this is linux specific, but it may be of help to understand how it's solved on Ec2
<Goneri> Yes, understood.
<rharper> Odd_Bloke: I'm +1 on 147 now, with the note that Goneri will follow up to handle multi-nic DHCP (and other cleanups)
<Odd_Bloke> Nice, thanks!
<Odd_Bloke> Goneri: #147 landed! \o/
<Goneri> ahah!
<Goneri> I will refresh https://bsd-cloud-image.org/ later today.
<Goneri> thanks meena Odd_Bloke and rharper for the review.
<blackboxsw> Odd_Bloke: sorry about missing the build recipe failures a few days ago, I had mistakenly thought it was just focal. but as you pointed out, xenial has been failing for a wihle.
<blackboxsw> so I stopped at fixing focal as I saw bionic eoan were fine. I forgot to look through xenial
<blackboxsw> working that now
<blackboxsw> should be minor
<rharper> blackboxsw: that was me poking you on that ... I'm subscribed to the recipe failures
<Odd_Bloke> rharper: I poked him in privmsg because I was going to take a look if he wasn't already.
<rharper> ah
<rharper> Odd_Bloke: thanks! =)
<blackboxsw> rharper/Odd_Bloke right. I fixed focal a few days ago, didn't realize that xenial patch was broken too
<blackboxsw> so that did fall through the cracks until Odd_Bloke pinging me pvt
<Odd_Bloke> I'm not sure why I'm not getting those emails TBH.
<blackboxsw> privately
<Odd_Bloke> Maybe I'm filtering them, let me check.
<blackboxsw> Odd_Bloke: rharper so, we use SRU_BLOCKER/RELEASE_BLOCKER comment in cloud-init to give us a heads up about something that needs attention during the next SRU or RELEASE. I have to quilt refresh a number of patches at the moment on xenial, and I was wondering if either of you have a suggestion on a common comment prefix we can also use for resolved SRU/RELEASE blocker differences.
<blackboxsw> a common prefix would allow us to easily see in an ubuntu series if there are things patches that will remind us of differing behavior on other series
<blackboxsw> since I have to refesh multiple debian/patches now it'd be nice to start instrumenting that "SRU_RESOLVED" maybe?
<blackboxsw> SRU_FIX?
 * blackboxsw goes with SRU_FIX unless there are firm objections
<blackboxsw> put up https://github.com/canonical/cloud-init/pull/284
<blackboxsw> for review
<blackboxsw> I'm thinking we probably should also queue bionic and eoan with new-upstream-snapshot --skip-release just so we'll have a common changeset to look at once we actually do perform an SRU to those series in the future
<blackboxsw> what do you folks think? do this for bionic and eoan as well so daily recipes are building the same 'snapshots'
<blackboxsw> even though build recipes aren't failing there
 * blackboxsw queues those for review pending discussion
<Odd_Bloke> blackboxsw: The daily recipes merge master in, so they'd be building the same snapshot regardless, I think?
<blackboxsw> Odd_Bloke:
 * Odd_Bloke nods sagely in response.
<blackboxsw> I think so right. so functionally may not make a difference on finished deb.
<blackboxsw> hah
<blackboxsw> though it'll make for vastly different xenial vs bionic/eoan on our next SRU
<Odd_Bloke> Well, we'll still merge in master at that point to each branch.
<blackboxsw> because current fixed xenial  will have snapshotted up until today
<blackboxsw> yes bionic/eoan won't do that until we actually try to perform the next SRU
<Odd_Bloke> We aren't releasing these branches anywhere, we're essentially just fixing merge conflicts.
<blackboxsw> not releasing, but at SRU time, we will review a PR like 284 for xenial and it'll be way different from the PR for bionic/eoan
<blackboxsw> bionic/eoan will be a lot bigger and will include what we will be pushing the upstream/ubuntu/xenial today
<Odd_Bloke> We don't really review those PR diffs, though, we basically just confirm that the uploader hasn't fat-fingered using the tooling.
<rharper> blackboxsw: I think that's fine, we run the same tools on each branch
<Odd_Bloke> (i.e. I'm not reading through that xenial diff, I'm reviewing it by performing the actions locally.)
<rharper> so they can vary, but my output should match the PR
<rharper> Odd_Bloke: exactly
<rharper> I end up diffing my local branch against the PR
<rharper> and the delta should be timestamps and names
<blackboxsw> ahh roger
<blackboxsw> in that case it doesn't matter, for future of bionic/eoan
<blackboxsw> for this xenial branch you'll see my diffs as well for the manual quilt refresh changes as I changed the patches a bit with the SRU_FIX prefix
<blackboxsw> I went through https://github.com/CanonicalLtd/uss-tableflip/blob/master/doc/ubuntu_release_process.md#when-the-daily-recipe-build-fails
<blackboxsw> rharper: are we ok with this response? https://github.com/canonical/cloud-init/pull/284#issuecomment-604692176
<blackboxsw> also should I put up a doc PR  against uss-tableflip suggesting the use of SRU_FIX in patch comments ?
<blackboxsw> for when we add new patches (like netplan eni priority in the near future)
<rharper> blackboxsw: it wasn;t clear to me if you added any strings or just ran the steps from the docs ?
<rharper> blackboxsw: I don't want to add any markers in the patches or code;  I'm just not going to remember to look
<rharper> I just want to run the tools and compare branches
<blackboxsw> right rharper, but once https://github.com/canonical/cloud-init/pull/267 lands we need another patch and it'd be nice if that patch represented that it omitted content from ubuntu/bionic branch to retain original behavior
<blackboxsw> kindof like the existing requirements.txt patch comments that we aren't adding jsonschema deps to avoid changing behavior
<blackboxsw> except that comment and text currently is unstructured, so breadcrumbs are hard to find.
<blackboxsw> Also we need to have some structure/procedure to track in cloud-init source certain features that we don't want to accidentally release into stable releases. And that convention currently is loosely SRU_BLOCKER in comments and no uss-tableflip documentation that says, hey make sure you double check during next sru that you aren't leaking unintented behavior
<blackboxsw> *unintended behavior*
<Odd_Bloke> blackboxsw: I can find that requirements.txt comment by looking through d/patches, I don't need to grep for it.
<Odd_Bloke> Is it true of all the other modifications we make that they're in d/patches?
<blackboxsw> Odd_Bloke: good point. so maybe no prefix required on patched files
<Odd_Bloke> If so, then I think that's a better catalogue of changes than any manually managed prefix is going to give us.
<rharper> blackboxsw: I suspect before adding the strings, we need a complete use-case and walk through what it will actually improve.   I can see a use-case for documenting behavioral differences between releases ...
<rharper> but let's set out with that in-mind and design a tool/process with the goal in mind;
<blackboxsw> Odd_Bloke: it's not a whole set of patches as we directly have adapted cloudinit/settings.py and debian/cloud-init.templates per release to enable datasources after the release goes stable
<blackboxsw> and there may be others.  but mostly debian/patches captures the majority of the functional differences in cloud-init source. not packaging diffs
<blackboxsw> rharper/Odd_Bloke: good points. so shall I leave the debian/patch comments untouched then. and just manually merge as best I can to avoid any other diff introduced by SRU_FIX prefix?
<blackboxsw> I have one approve at the moment
<blackboxsw> but can alter that approach to keep the debian/patch diff smaller
<Odd_Bloke> rharper: blackboxsw: If either of you are looking for some small reviews to do, I've opened 4 small PRs that cleanup some more Py2 support code I found.
<blackboxsw> Im all about the smalls
<blackboxsw> :)
<rharper> blackboxsw: I would prefer to leave them untouched;  when you say "merge as best you can" what do you mean?
<blackboxsw> ok rharper sorry force pushed 284. without comment changing liberties
<blackboxsw> diff from yours should be smaller now
<blackboxsw> rharper: I meant manually merging the existing patch because quilt failed to update
<blackboxsw> nothing generally should be dropped, but it presented me with an opportunity I thought to standardize comments. But, I'm good not doing that. it's of limited use anyway
<blackboxsw> in manually merging a patch conflict, there really shouldn't be any changes unless upstream changed some subset of the exact lines of the patch file and there could be a possibility that the patch refresh author needs to make the appropriate manual decision on what functionally should remain after the patch (like if we have a variable renamed or something). So "merge as best you can" meant being smart about your
<blackboxsw> manual choice when resolving that quilt patch update
<blackboxsw> rharper: I forgot to ping you again earlier on the netplan proiritization branch https://github.com/canonical/cloud-init/pull/267   do you think it needs a cloud_test to install ifupdown on focal to confirm behavior?
<blackboxsw> or shall we just chalk that into a manual one-off SRU/release test
<blackboxsw> I thought at standup I had missed review comments from you, but I don't see anything new there.
#cloud-init 2020-03-27
<faa> hello cloud-init now support python 2.7?
<faa> search in issues "Thanks for the pull request! cloud-init 19.4 was the last version of  cloud-init to support Python 2"
<AnhVoMSFT> rharper powersj I am seeing ec2 prefix on ssh public key generation on Azure instance https://paste.ubuntu.com/p/QSkhKyPS2B/ - Is this expected? Looks like this is output of ssh-keygen? Is it possible to change that to platform name?
<Odd_Bloke> faa: That's correct, we dropped Python 2 support between 19.4 and 20.1.
<Odd_Bloke> AnhVoMSFT: I'm not sure where that string comes from, TBH; could you file a bug?
<Odd_Bloke> AnhVoMSFT: Oh, looks like we had a bug filed overnight: https://bugs.launchpad.net/bugs/1869277
<ubot5> Ubuntu bug 1869277 in cloud-init "azure vms with cloud-init display ec2 info in output" [Undecided,New]
<Odd_Bloke> Oh, hmph, we're seeing the integration test hang (and therefore timeout) issues we've seen intermittently before: https://travis-ci.org/github/canonical/cloud-init/jobs/667711759
<Odd_Bloke> rharper: I remember we discussed ^ some, but did we reach a conclusion as to what we thought the problem was?
<rharper> Odd_Bloke: the most recent issues were blocked on snapd.seeded.service taking more than the 300 seconds we wait
<Odd_Bloke> rharper: These are xenial instances though, so I think we concluded that there aren't any snaps being seeded?
<Odd_Bloke> (I think we had this exact exchange last time too. :p)
<rharper> AnhVoMSFT: ec2[1850]   ,  in systemd journal, things get logged with pid arg0 and pid number,  ...  cloud-init execs are clearly marked cloudinit[PID] ...
<rharper> AnhVoMSFT: can you reproduce on stock RHEL7.7 or Centos7.7 images versus that OpenLogic image ?
<Odd_Bloke> rharper: You can reproduce it in an Ubuntu lxd.
<Odd_Bloke> (The "ec2" thing, I mean.)
<rharper>  ?
<Odd_Bloke> In the xenial container I just launched to check snapd.seeded, I have: Mar 27 14:27:03 renewed-corgi ec2[653]: 1024 SHA256:x/P+1raytPVp9l5tcVB28yV48aTs6jG5OhR6Fw09ciM root@renewed-corgi (DSA)
<rharper> oh, interesting, focal does not
<Odd_Bloke> I'm currently getting ~100kB/s downloading an image locally.
<Odd_Bloke> So I bet that's what's causing our CI failures.
<rharper> Odd_Bloke: what's your build.info on that xenial image ?
<Odd_Bloke> 20191107
<rharper> I have a vm launched yesterday serial: 20200320
<rharper> dailys are fine
<rharper> huh
<rharper> I can see it a container now as well
<rharper> not quite sure why it's not showing up in the VM
<Odd_Bloke> So the code uses util.multi_log, which has a comment about behaviour being different in containers.
<Odd_Bloke> https://github.com/canonical/cloud-init/blob/master/cloudinit/util.py#L515-L522
<rharper>  This file is part of cloud-init. See LICENSE file for license information.
<rharper> logger_opts="-p user.info -t ec2"
<rharper>  /usr/lib/cloud-init/write-ssh-key-fingerprints
<rharper> it's been there since 2012
<rharper> so that's nothing new
<AnhVoMSFT> yes I think it has been there for a long time. We just noticed it yesterday :-)
<rharper> hehe
<rharper> we can use the bug to adjust that to something else;  I suspect we want to log as cloud-init instead of ec2
<AnhVoMSFT> looking through google I think someone using OpenStack was also seeing it. It's basically there for all cloud-init users
<AnhVoMSFT> I think ec2 will need their own prefix. I saw some bug against it back then
<AnhVoMSFT> https://bugs.launchpad.net/ubuntu/+source/ec2-init/+bug/458576
<ubot5> Ubuntu bug 458576 in ec2-init (Ubuntu Karmic) "ec2: ssh public key fingerprint in console output does not match EC2 standards" [Low,Fix released]
<rharper> I guess someone was extracting the public keys from the serial console log ?
<rharper> and yes, ec2-init  was cloud-init before it was cloud-init
<AnhVoMSFT> i think we let datasource provide their own prefix if they would like to (ec2 in this case). Otherwise it should just say cloud-init
<Odd_Bloke> It should just say cloud-init, I think.
<rharper> yeah
<Odd_Bloke> rharper: https://github.com/canonical/cloud-init/pull/287 <-- this will extend the time it takes before CI times out, which should work around the problem for now
<Odd_Bloke> It sounds like we're just getting caught in transatlantic link congestion that's out of Canonical's direct control.
<Odd_Bloke> I'd like us to investigate a couple of other things: (a) caching images between Travis runs, and (b) emitting progress information during the step that currently times out, so we don't need to use travis_wait.
<rharper> Yeah, wondering if we want to bump the timeouts in CI after checkout
<rharper> Odd_Bloke: I know lxd can be configured to point to other image servers ...   I suspect if we were to cache them and then we'd need an image alias to ensure the image names we use are found in the configured image repo
<rharper> Odd_Bloke: we may want to chat with stgraber on travis lxd image caching
<Odd_Bloke> Perhaps; doesn't the test framework itself perform the download?
<rharper> yeah, I'll look at the implementation here in a second
<rharper> for bumping timeout, there's this setting: tests/cloud_tests/platforms.yaml:get_image_timeout: 300   ; we can sed that into whatever large value we want ...
<rharper> image caching is harder, we point to  DEFAULT_SSTREAMS_SERVER = "https://images.linuxcontainers.org:8443"   ; , I see that's also set in tests/cloud_tests/releases.yaml, lxd: sstreams_server ... so if we stood up a cache, we could point it there
<Odd_Bloke> rharper: That isn't the timeout we're hitting, I commented on the PR.
<Odd_Bloke> Ideally, I think we'd be able to identify the path that contains the cache that lxd/cloud-tests uses, and then ask Travis to just retain that between runs.
<Odd_Bloke> Using https://docs.travis-ci.com/user/caching/#arbitrary-directories
<blackboxsw> rharper: if you get a chance today, netplan prioritization branch needs your response. Should we write up a cloud_test for this? https://github.com/canonical/cloud-init/pull/267
<rharper> blackboxsw: I saw your note, while I would like a cloud-test, we don't yet have a way to reboot between runs ...
<blackboxsw> ok will probably write up that multi-stage cloud_test
<powersj> blackboxsw, multi-stage?
<blackboxsw> powersj: I may peek at it more in a bit. the ifupdown test sort of requires us to cloud-init clean and re-render networking . so we may need an additional collect stage for that type of test. especially if we want to reboot and recollect after that boot
<powersj> blackboxsw, make a card for it and move on
<blackboxsw> rharper: https://github.com/canonical/cloud-init/pull/267 ok, I think since we know Focal needs netplan prioritization, let's land this, and we can revist cloud_tests later for this once we've discussed with cpc/foundations what the plan is with handling ifupdown gaps
<rharper> blackboxsw: reviewed, looks good, I've suggested a unittest to add, then we can land.
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/288 <-- small precursor PR for the mirror URL sanitisation work
<rharper> You need to leave a comment indicating the requested changes.
<rharper> I swear
<drag0nius> anyone tried running cloud-init on pre-baked raspbian in rpi?
<drag0nius> i want to know how to get started and where to put my cloud-configs into baked image
<blackboxsw> drag0nius: might that be in the #raspberrypi  channel on freenode?
<blackboxsw> I know waveform in Canonical has been looking over raspi development using cloud-init.
<Odd_Bloke> Anyone else just getting a loading screen on https://travis-ci.org/ ?
<Odd_Bloke> rharper: I've moved that function, net makes more sense for sure.
<rharper> cool!
<Odd_Bloke> (And this is an example of why smaller code reviews are better; who knows if we'd have picked up on that if I proposed all my code at once.)
<rharper> yeah, thanks
<ananke> I'm finally having a few mins to look back at kali again. Tried their earlier AMI, and I'm stumped: the same issue. cloud-init status claims cloud-init is still running
<ananke> What's more baffling is the fact that 'blame analyze' shows close to a minute being spent on this:
<ananke> -- Boot Record 01 -- 51.20500s (init-local/search-Ec2Local)
<rharper> do you have a the full cloud-init collect-logs tarball ?
<ananke> I'm preparing them right now
<rharper> that looks like maybe networking isn't setup up properly and it's timing out on the URL lookup
<rharper> but we'll know more for sure with logs
<rharper> is kali debian based or something else?
<ananke> I believe so
<ananke> we're feeding it an identical cloud-init config via user-data as we are ubuntu & centos targets, which makes it so baffling
<rharper> I doubt it's your user-data
<rharper> more likely issues with the OS and cloud-init integration ...
<rharper> if it's debian based, they may have a much older cloud-init
<ananke> 18.3, but we saw the same issue with cloud-init 19 on their 2020 AMI. thank you for your interest in this issue. what would be a good way to share the log tarball?
<ananke> with the covid19 pandemic and push to go online as much as possible, our workload has increased, as we provide an educational platform via AWS. fun, but busy
<rharper> https://bugs.launchpad.net/cloud-init/+filebug
<rharper> and attach the tarball
<blackboxsw> xenial daily build recipe fixed & rebuilt working on a parameterized  unit test on netplan over eni priority pr
<rharper> blackboxsw: \o/
<rharper> blackboxsw: heh, or you could paste the one I had ... I know it's begging for a pytest.mark.paramtrized ...
<rharper> but .. landing focal features > pytest refactor IMO
<blackboxsw> rharper: fair point, I was only going to spend ~20 on it shouldn't be long. how long til EOW??
<rharper> heh, ok
<Odd_Bloke> Travis seems to be back now, but heads-up that I had to logout/login to get the UI to behave properly (it wasn't showing me canonical projects in the sidebar, and I didn't have the restart button on jobs).
<ananke> rharper: done, thank you! https://bugs.launchpad.net/cloud-init/+bug/1869430
<ubot5> Ubuntu bug 1869430 in cloud-init "cloud-init persists in running state on Kali in AWS" [Undecided,New]
<rharper> ananke: I'll take a look
<ananke> thanks!
<ananke> right now that's a bit of a problem for us, as the build stage runs cloud-init status --wait, so it never finishes
<rharper> yeah, that's the right thing to do; so let's see what's going on
<rharper> ananke: it appears that cloud-config.service has not yet started  (nor final) so it's hanging ... cloud-config.service waits for systemd's 'network-online.target'  --- in the journal file, I don't see the 'Reached target Network is Online' which is emitted when this happens.
<rharper> I'll update the bug ... do you have an ami id to reference?  I suspect there are issues with the network service files
<ananke> sure thing, I can add the AMIs we've tried in the bug report
<ananke> â network-online.target - Network is Online
<ananke>    Loaded: loaded (/lib/systemd/system/network-online.target; static; vendor preset: disabled)
<ananke>    Active: inactive (dead)
<ananke> interesting
<rharper> typically means nothing "asked" for it; in ubuntu we have a networking.service scripts which depends on it and should bring in the target
<rharper> I would think debian networking.service would as well
<rharper> however, kali has both ifupdown and network-manager  installed
<ananke> yes, network-manager appears to be running, and it has ifup/ifdown
<rharper> cloud-init also has had some edge cases with  ifupdown and nm installed at the same time ;
<ananke> great :)
<ananke> rharper: are systemd logs uniform across distros? as in, would 'Reached target Network is Online.' be different from 'Reached target Network.'?
<ananke> hmm, it appears it is. I'll have to dig into this deeper
<rharper> there are 3 targets
<rharper> network-pre, network, and network-online
<rharper> cloud-init runs before network.target as it writes out networking config, and then the networking.service nm.service run to bring the network online;  once they are active (and the -wait-online.services complete) then the network-online.target is reached
<rharper> ananke: I'm not having luck with those images ...  it's asking me to subscribe to the kali marketplace and I can't do that ...
<ananke> yeah, I was afraid of that. Eventhough they're free, they are still on the marketplace. I am looking at your last notes, those are very astute observations
<ananke> though one might wonder why is it taking 40 seconds to resolve IP based URL
<rharper> I think dns is not working likely related how resolv.conf is configured
<ananke> thank you, we'll dig into it and update the bug report with findings. that's very valuable information though, we were stumped why cloud-init was not finishing, despite the host being up and no major issues on the surface
<rharper> I'll update the bug with the files I'm looking for
<ananke> I have to tend to kids for a bit, since there is no school. thank you, and I'll be back in a bit
<rharper> sure
<blackboxsw> rharper: wrapped up a parametrized pytest that allows dropping a few other renderer unit tests
<blackboxsw> there obviously is more refactor that could be had, but net.renderer.select doesn't really rely on the distro type, just available/not-available renderers
<blackboxsw> so adding unit tests there doesn't really exercise the merged config rendered case here
<rharper> blackboxsw: you mean the system-config policy  ?
<rharper> it doesn't really matter where it comes from; we already know that system config is merged over the default policy;
<rharper> blackboxsw: right, I didn;t look hard enough; but I *think* in the ntp test-case I've got one that loads the config/cloud-config.cfg.tmpl , renders it and verifies it's default behavior
<rharper> we want something like that for renderer
#cloud-init 2020-03-28
<blackboxsw> rharper: for monday cloud.cfg.tmpl rendering test added to  https://github.com/canonical/cloud-init/pull/267   also I had forgotten to click submit on review response to https://github.com/canonical/cloud-init/pull/152  (jsonschema for write_files)
<andras-kovacs> <rharper " /usr/lib/cloud-init/write-ssh-k"> this folder is /usr/libexec/cloud-init/ in RHEL and thanks for the info!
<andras-kovacs> and the NetworkManager + ifup/down topic
<andras-kovacs> IF NetworkManager is running then network.service's ifup/down will call nmcli connection up :)
<andras-kovacs> <rharper "however, kali has both ifupdown "> so it's pretty much the same IMHO
<andras-kovacs> <ananke "yes, network-manager appears to "> so NetworkManager doesn't have ifup/down I think but the ifup/ifdown shell scripts calling the NetworkManager
<andras-kovacs> To be honest, I'm building RHEL templates these days with the network.service disabled.
<andras-kovacs> I believe that NetworkManager is the way even if Facebook, etc. thinks differently.
#cloud-init 2020-03-29
<Gaffel> I'm unable to get cloud-init on Centos 8.1.1911 to parse my user-data file. It parses meta-data and changes the hostname accordingly but I'm unable to log in because none of the things in user-data is applied.
<Gaffel> This is what my user-data and meta-data looks like: https://paste.centos.org/view/f126b91b
<Gaffel> Is there anything weird with user-data? I've followed both cloud-init's docs and guides from Red Hat.
<apollo13> Gaffel: what do the logs say? the usually provide some kind of hint
<apollo13> I had the problems just yesterday that it wouldn't create my user because admin as a user won't work if the system already has an admin groupâ¦
<apollo13> but the logs were rather clear about that
<Gaffel> apollo13, how do I access logs if I can't log in?
<Gaffel> The username is "centos" and the password is blank. But that doesn't work. Nor does it accept my SSH key and it doesn't enable password login on SSH even if I specify that. I've recreated dozens of VMs so far and all I can get is meta-data.
<apollo13> Gaffel: rescue cd usually
<Goneri> meena, https://www.reddit.com/r/BSD/comments/frb8zd/cloud_images_for_bsd_based_on_cloudinit/
<meena> Goneri: nice!
<Goneri> meena, thanks :-)
<faa> freebsd now actual port ;) https://www.freshports.org/net/cloud-init
