#cloud-init 2013-12-16
<smoser> harmw, yeah, i wanted to get one before i leave for holiday.
<smoser> there is an 0.3.2~pre2 (which is not really released at http://download.cirros-cloud.net/local/0.3.2~pre2/)
<smoser> i've not done any testing on it though, so i didn't even release it as a "pre2" yet.
<smoser> so any feedback/test you could give it would be good.
<harmw> ok, well i'll see what I can do tonight
<harmw> there are some good fixes, iirc not that hard to verify
<harmw> smoser: just me or is that ~ in the url causing bigtime 404 errors
<harmw> ah, no
<harmw> forgot the /local
<smoser> yeah.
<harmw> shouldn't it instant-reboot after resizing /?
<harmw> i'm seeing hostroutes, excellent (so yes, metadata was fetched aswell since my hostname is correct)
<harmw> hm, after reboot it hasn't grown to 3GB (the volume size)
<harmw> or am I completely wrong and is growing root a step I specifically had to set/configure?
<med_> harmw, you may need to reinvoke smoser if you wanted some feedback on that 3G growth question
<med_> I suspect after two hours he looked  away
<smoser> harlowja_away, it should grow the roto filesystem . i think.
<smoser> i can't imagine that that was broken.
<smoser> i thikn you're talking about cirros, right?
<smoser> it should have done it.
<med_> is harmw the same as harlowja_away ?
<smoser> shoot. no.
<smoser> harmw, ^
<harlowja> haha
<smoser> hm.. harmw cirros *should* grow the root partition and resize2fs it also. you should probably see something on console output if it didnt. (console output as in 'nova console-log')
<harmw> good one med_ , though i wasn't particularely after a quick-reply :)
<med_> heh.
<harmw> and yes smoser, it should've done that though I didn't see it happen
<harmw> let me try again
<harmw> (on a different cloud)
<smoser> harmw, you're booting the disk.img file ?
<harmw> yea, converted to raw and then it goes to glance
<harmw> i remember having seen a syslog line telling it resized, just cant remember when that was
<smoser> you don thave to convert to raw, harmw 
<harmw> without that I can't copy-on-write, and glance doesn't (yet) do it for me
<smoser> well glance shouldn't convert for you
<smoser> glance image-create --disk-format=qcow2 --container-format=bare --name=smoser/cirros-0.3.2~pre2-amd64-disk.img --file ~smoser/src/cirros/mirror/0.3.2~pre2/cirros-0.3.2~pre2-x86_64-disk.img
<smoser> nova should convert for you.
<smoser> nova should see a qcow and convert it to raw and then create qcow backed by the raw.
<harmw> isn't nova supposed to just use whatever cinder throws at it? in my case, a glance image that originates in rbd (ceph)
<harmw> growpart: mode: auto devices: ['/']
<harmw> i have that in my userdata script, amongst some other stuff
<harmw> though that would probably not be needed to have cloudinit autogrow, right?
<smoser> oh. i see.
<smoser> cinder. you did say that. boot from volume.
<smoser> yeah. you're right.
<harmw> yes, sorry
<smoser> so get the cuos yeah, i would have expected that to work, but have not tested that.
<smoser> so what steps do you do to acocmplish that ?
<smoser>  * download image, convert to raw.  
<smoser> how did you then upload / register a boot-from-volume image?
<smoser> i dont know if i hcan do this with the cloud i have here.
<smoser> harmw, http://paste.ubuntu.com/6584812/ 
<smoser> thats patebin of non-volume boot. and you see line 266 there.
<harmw> yes, the GROWROOT:bla line is what i've seen before
<smoser> there could be race conditions there.
<smoser> :-(
<harmw> ah
<harmw> well, i didn't debug it 
<harmw> perhaps i can do that later on 
<harmw> let me paste the commands i'm using first
<harmw> http://paste.ubuntu.com/6584832/
<smoser> oh. maybe i can do this. i'll try
<smoser> harmw, bah. i can't really test. 
<smoser> the cinder create for me causes error
<harmw> i'm back in a few minutes
<smoser> harmw, i did this: http://paste.ubuntu.com/6585009/
<smoser> taking 'path 2' there. (attach a volume and populate it myself)
<smoser> then console-log http://paste.ubuntu.com/6585013/
<smoser> so it *could* be racy still, but it seemed to work here.
<smoser> harmw, hm.. so i see it has failed in my instance here.
<harlowja> openstack, racy, not possible, lol
<harmw> GROWROOT: CHANGED: partition=1 start=16065 old: size=64260 end=80325 new: size=6265350,end=6281415
<harmw> weird
<med_> harlowja, openstack is racy. Danica Patrick is our new sportscar driver.
<harlowja> :)
<smoser> harmw, yea, so it does do the partition resize. that works.
<harmw> indeed, but that wasn't on my console this afternoon
<smoser> but then the call to 'resize2fs' which happens from 
<smoser> /etc/init.d/resizefs
<harmw> console also tells me the hostname is cirros
<smoser> seems to return failure with no error to stderr.
<harmw> hm, it didn't set the hostname 
<harmw> whre does cloud-init keep its log in cirros?
<harmw> hm, looks like it received the public-hostname, though hostname is empty (in /var/run/cirros/datasource/data
<smoser> harmw, thats odd. mine get their hostnames.
<harmw> mine normally do to :)
<smoser> what does 'hostname' say ?
<harmw> hostname said the right thing
<harmw> i've just booted a new instance, all is ok with this one :|
<harlowja> smoser is metswin:) still the login for cirros?
<harlowja> or was that cubwin
<harmw> cubs
<harlowja> nice, ha
<harmw> this new instance has a proper hostname, GROWROOT did something aswell but resizefs didn't do squat and fdisk -l does not show a fully used disk either
<smoser> harmw, fdisk -l doesn't ?
<smoser> that is odd.
<smoser> what does cat /proc/partitions say ?
<smoser> harmw, so it works in 2 portions
<smoser> growroot runs in iinitramfs
<smoser> and changes the parttion table
<smoser> then unmounts the target
<smoser> err..
<smoser> unmounts
<smoser> grows
<smoser> remounts
<smoser> then pvirot root
<smoser> then /etc/rc3.d/S55-resizefs
<smoser> *should* resize the filesystem "per-once"
<smoser> and i see this functional on "ephemeral store" images.
<harmw> /dev/vda1   *       16065     6281414     3132675   83  Linux
<harmw> $ cat /proc/partitions 
<harmw> major minor  #blocks  name
<harmw> 253        0    3145728 vda 253        1    3132675 vda1
<harmw> i've now manually run /etc/rc3.d/S55-resizefs start, and after thats done df doesn't show any change
<harmw> though after another second or so it does show the changed fs, as beeing bigger
<smoser> harmw, right. it backgrounds.
<smoser> that is by design. (possibly bad design)
<harmw> hmk
<smoser> but it does so because resize2fs in cirros is pretty slow.
<harmw> aha
<smoser> at least compared to 12.10 ubuntu or later.
<harmw> ok
<smoser> and the thoguht was that people want to run scirors on very un-performant systems
<smoser> such as non-kvm qemu
<harmw> what bugs me is it didn't resize on initial boot, only just now when I did it manually
<smoser> yeah. that was confusing to me too.
<smoser> i even put debug into that /etc/rc3.d/S55-resizefs
<smoser> to run 'cat /proc/partitions'
<smoser> and output was correct.
<smoser> hm.. wait a minute. i think i just saw it fail on ephemeral root
<smoser> when it fails, you'll see a file /tmp/resize.out
<smoser> that should have stderr and stdout of the 'resize2fs /dev/root'
<smoser> but it doesnt
<harmw> yea
<harmw> $ cat /tmp/resize.out 
<harmw> resize2fs 1.42.2 (27-Mar-2012)
<harmw> and thats it
<harmw> well, new instance (again): growroot did its trick according to consolelog but resizefs only put the above in the logfile
<smoser> harmw, its really odd. 
<smoser> cirros-per doesn't even put its marker down
<smoser> which it should do even on failure.
<harmw> smoser: its not putting down the marker file, indeed, it only shows this:
<harmw> using state_dir=/var/lib/cirros/sem, marker=once.once.resizefs
<harmw> [once] sh -c o=/tmp/resize.out;
<smoser> yeah, its just wierd.
<harmw> followed by resize2fs [..]
<smoser> i dont get it.
<harmw> me neither
<smoser> it must be something stupid
<harmw> yea, well even after a reboot it didn't resize - so growfs isn't causing the pain
<harmw> ill see again, tomorrow
#cloud-init 2013-12-17
<smoser> harmw, well, some news. seems busted in my 0.3.2~pre1 also.
<smoser> hm.. and interesting. it seems busted in 0.3.1 also. the partition table does get grown.
<harmw> smoser: I just filed https://bugs.launchpad.net/cloud-init/+bug/1261710
<harmw> I'll see if I can work on that tonight
<harmw> annoying
<harmw> hm, and why didn't I file that against cirros :|
<harmw> there, changed it
<smoser> harmw, thanks.  i can't figure it out. i think something is killing the resize2fs process after it starts.
<smoser> i changed 'resize2fs' to 'sleep 5' and it seems to die also.  also tried putting the content of the 'sh -c' in its own program. that didnt work either.
<smoser> the one thing that does work is removing the '&'
<smoser> ie, not backgrounding.
<smoser> i'll put this in the bug.
<harmw> great, nice
<harmw> (-1 for that silly typo btw)
<smoser> harmw, it seems that using start-stop-daemon does work. but i couldn't get start-stop-daemon to sanely call a 'sh' -c program
<smoser> so had to put it in a different program
<harmw> ah great
<smoser> harmw, want to take the fix for a spin ?
<smoser> get cirros trunk (bzr branch lp:cirros)
<smoser> then
<smoser> tar -C src -cvf - sbin/resize-filesystem etc/init.d/resizefs lib/cirros/shlib | ssh cirros@$IP 'sudo tar -C / -xvf - && sudo reboot'
<smoser> seems to work for me, and when its done you get a nice message like:
<smoser> /dev/root resized successfully [took 43.12s]
<harmw> smoser: ill give it a go in a few minutes
<smoser> harmw, k. good. i dont really know how i feel about "fixing" it. given the 45 seconds it took to run.
<smoser> i'm apt to put this in and then turn it off by default.
<harmw> it kinda goes straight into the whole cirros philosophy
<harmw> any thoughts on how to enable it if the default where to have it disabled?
<harmw> === cirros: current=0.3.2~pre1 latest=0.3.1 uptime=100.09 ===
<harmw> +1 for a lightspeed cloud :>
<harmw> === cirros: current=0.3.2~pre1 latest=0.3.1 uptime=63.64 ===
<harmw> and that was after the reboot + resizefs stuff
<harmw> # grep resize e1ba8f11-097f-4bc1-bf9b-475710fee5ef/console.log 
<harmw> wheeljack login: /dev/root resized successfully [took 92.04s]
<smoser> one thought on how to enable it if it were disabled is via file injection (through config drive or openstack api)
<smoser> i think it would work.
<smoser> if /etc/init.d/resizefs read /etc/default/resizefs.conf
<harmw> ./sem/instance.i-00000052.userdata
<harmw> ./sem/instance.i-00000052.cirros-apply-net
<harmw> ./sem/once.once.resize-rootfs
<harmw> ./sem/instance.i-00000052.check-version
<harmw> isn't that missing the tiny touch of consistency? :)
<harmw> can't it be specified in user-data? a believe there already is a module to allow for this to happen
<harmw> smoser: perhaps have the resizefs script read /var/run/cirros/datasource/data/user-data to make the decision to run/start
<smoser> consistency ?
<smoser> well, 'instance' is that this is 'per-instance', and then the second portion is the instance id. and then the key.
<smoser> for 'once', there is no useful second portion
<smoser> so i just reused the sring 'once'
<smoser> ie, the resize will only run once, ever.
<smoser> it could be 'per-instance', and probably woudl work. but i really dont think people are "rebundling" cirros :)
<smoser> we could have it specified as user-data, yes. but at the moment cirros doesn't support any sort of "cloud-config" user data.
<smoser> it only supports '#!'
<smoser> if i had some cloud-config analog then it'd be perfect. but thats not there yet.
<smoser> the inject-files in the openstack api "should work" as it is right now i think. 
<smoser> with no other features needed from cirros except having the rc.d script source (if it exists) that file.
<harmw> true
<smoser> i agree that cloud-config like function is better long term goal
<harmw> indeed
<smoser> but i'd wan to have this enable-able in 0.3.2 and want that "real soon now"
<harmw> though having it work based on the presence of a certain file is cool enough for now :)
<smoser> well, i'd have it source the file
<smoser> if present
<smoser> ie, so the file could have:
<smoser> enabled=0
<smoser> background=0
<smoser> or something to that affect.
<smoser> or mabye just 
<harmw> isn't it easier to just check if a (0byte) file exists?
<smoser> mode=(foreground|disabled|background)
<smoser> existence doesn't allow  me to do foreground and background
<smoser> and i really do thikn the backgrounding is weird
<harmw> right on that
<smoser> cause for race conditions
<smoser> harmw, feel free to give this a try
<smoser> http://paste.ubuntu.com/6590740/
<smoser> bah. bad syntax.
<harmw> darn, I was just busy on something myself :)
<smoser> http://paste.ubuntu.com/6590744/
<smoser> harmw, sorry to get in your way, i'd love for you to particpate in cirros developement :)
<harmw> and I should have openstack create the /etc/default/resizefs file
<harmw> haha np :)
<smoser> yeah, you should be able to do somethin glike:
<smoser> nova boot --file /etc/default/resizefs=my.resizefs.txt
<harmw> yea
<harmw> hm, seems my cloud is not very fond of inserting files
<harmw> like this: /data/openstack/nova//instances/b0471e49-094c-4eed-b8f0-7a979ed35957/disk: No such file or directory
<harmw> NovaException: Error mounting /data/openstack/nova//instances/b0471e49-094c-4eed-b8f0-7a979ed35957/disk with libguestfs
<smoser> harmw, i think you can turn that off. 
<smoser> we dont need the host to insert it
<smoser> (i would never find that an acceptable solution)
<smoser> there is a way where the "inserted files" just appear in the config drive
<smoser> and then cirros reads them and *it* inserts them.
<smoser> harlowja, do you know how to set that ?
<harlowja> hmmm
 * smoser goes to try to remember if cloud-init supports that in config drive. i thin it does
<harmw> well, i don't use config drive atm and enabling it causes the other vm's to fail on boot iirc
<smoser> harmw, well, you dont need config drive. cirros should read it from the MD also
<smoser> (the files can be inserted there too)
<harmw> yea, i thought so
<harmw> was expecting that as well
<smoser> but clearly if you're trying to insert config for ifconfig eth0' then its probably not going to work :)
<harmw> hehe
<smoser> harlowja needs to write a proper cloud-init DataSourceOpenstack
<harlowja> :)
<smoser> that reads from the openstck MD. we dont really ahve that now. only could reaad those files from the config drive.
<harlowja> one sec, let me see what the option is
<smoser> harlowja, that really is something i'd like for 14.04
<harlowja> hmmmm
<smoser> proper oepnstack metadata service datasource (rather than just riding on ec2)
<harlowja> agreed
<harmw> smoser: does cirros read metadata? since then we could use a metadata key+value instead of inserting a file
<smoser> harmw, i think it does read it in some sense.
<smoser> in that it would write keys to files in the datadir
<smoser> so it could pretty easiliy use that
<smoser> the thing i dont like about that is that i dont know that i want to support it going forward
<smoser> i can trivially keep support for /etc/default/.... injection
<harlowja> harlowja i think if u make sure all 'inject_partition=0', 'inject_key=0' nova.conf values are off that will be step 1
<harlowja> apparently this is on by default
<harlowja> inject_key=true
<harlowja> https://github.com/openstack/nova/blob/master/etc/nova/nova.conf.sample#L2510
<harlowja> but u are hitting this code
<harlowja> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2570
<harlowja> "If we're not using config_drive, inject into root fs"
<harlowja> :(
<harlowja> harmw i think u can also use the following
<harlowja> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2544
<harlowja> inject_files and CONF.libvirt.inject_partition != -2
<harlowja> so make inject_partition=-2
<harlowja> weirddd
<harlowja> ha
<harlowja> # The partition to inject to : -2 => disable, -1 => inspect
<harlowja> # (libguestfs only), 0 => not partitioned, >0 => partition
<harlowja> # number (integer value)
<harlowja> #inject_partition=1
<harlowja> by default 1
<harlowja> crazy
<harmw> hehe
<smoser> some day the default will be -2
<harlowja> ya, i hope so
<harmw> so change to -2 and then nova boot --file should work just fine
<harlowja> harmw using config drive or nothing at all?
<harmw> nope, just metadata
<harmw> no configdrive here
<harlowja> so sure, then if smoser is correct and cirros understands special openstack metadata then u should be good to go
<harlowja> never know with that smoser  guy, ha
<harmw> time to find out :)
<harlowja> :)
<harlowja> but ya, i should (or should get someone from y!) to write a best-impl of an openstack metadata/config-drive thing
<harlowja> *datasource thing
<harlowja> as smoser  suggested
<smoser> uh oh. 
<smoser> it imight only do that from config-drive (not MD). hmm.
<smoser> suck. i don tthink cirros has proper openstack metadata source either.
<smoser> so harmw that isn't going to work :-(
<harlowja> hmmm
<harmw> indeed, it didn't inject the file
<harmw> uhm, why doesn't it read the stuff from user-data again?
<smoser> it should execute user-data
<smoser> if it starts with '#!'
<smoser> but thats its only consumption of user-data at the moment.
<smoser> and that runs after resize would have run
<smoser> but if you want, from that '#!' you could just *run* '/sbin/resize-filesystem /dev/root'
<smoser> at the beginning.
<harmw> nah, thats far to easy :)
<harlowja> smoser a start at this, http://paste.openstack.org/show/55206/
<smoser> harmw, you think thats a acceptable work around for now ?
<smoser> and then when we get either a true file insertion or "cirros-config" format in user-data we'll allow enabling it that way.
<smoser> (in the mean time the user can enable it by actually modifying the image, or by config-drive still should work)
<smoser> harlowja, woohoo
<harmw> smoser: yes, probably good enough for now
<harmw> though ultimately reading from user-data would be the best choice 
<smoser> yeah. i agree.
<smoser> https://bugs.launchpad.net/cirros/+bug/1232239
<smoser> is a start at that.
<smoser> ie, there is code there to read  mime-multipart
<smoser> and since i have a json reader, we can pretty easily support a "cloud-config-archive" like thing in a similar fashion
<harmw> nice
<harmw> how does this behave when I've put yaml in user-data?
<smoser> probably barf.
<smoser> i only have a json parser
<smoser> and i dont plan on a proper yaml one.
<smoser> its easy enough to convert yaml to json before passing to cirros
<smoser> and hard (without C programming) in cirros to write a yaml parser.
<harmw> you mean I should feed json user-data to nova on instance boot, or have cirros convert user-data to json upon loading/booting?
<smoser> 2 things here (kind of unrelated)
<smoser> a.) want to have cirros support "multi-part" user-data as input.
<smoser>  a.1) mime-multipart (as seen in that bug with the awk parser)
<smoser>  a.2) cloud-config-archive like multipart (cloud-config-archive is a cloud-init thing that basically just allows you to make an "archive" of parts in yaml rather than mime)
<smoser> for a.2, cirros would not support that being yaml, but instead just json. but same basic format.
<smoser> b.) some of those parts could be "cirros-config" which would be a json config format similar to "cloud-config"
<smoser> with the same caveat of not supporting yaml, but only json
<smoser> in the future if somehow i got a small enough yaml parser, we could just support yaml in both of those places.
<harmw> ok
<harmw> would've been easier to just put cloud-init in cirros :p
<smoser> $ apt-cache show python2.7-minimal | grep -i size
<smoser> Installed-Size: 3399
<smoser> Size: 1187530
<harmw> ye yea, bit to large
<smoser> so 3.XMB for python-minimal.
<smoser> i'm sure you could get stuff smaller
<smoser> but all of cirros is ~ 8M (the disk images are bigger because of filesystem overhead and the fact that the disk1.img file actually has *2* copies of cirros in it)
<smoser> oh. wait. no it doesnt.
<harlowja> smoser i'll try to work on improving that
<smoser> python ?
<harlowja> *openstackdatasource, ha
<harlowja> not python, ha
<harlowja> smoser u fix python, lol
<smoser> i have to admit, i kind of like programming in awk and busybox sh
<smoser> having python in cirros would just be no fun.
<harmw> hehe
<harmw> smoser: is multipart realy required straight away? or would a simpler implementation be fine as well, for starters. To just have some json user-data beeing applied. Allowing more input in a second phase, where you implement multipart.
<harmw> having it read user-data and apply any json found inside would be a nice start I'd guess
<harmw> however, given there already is a multipart parser...
<harmw> shoot, I'll just go back to do some work on the fbsd module - tomorrow
<smoser> harmw, i dont understand what yoiure asking really.
<smoser> i dont wanto to just overload userdata. like grepping through it to see if there is a key in it.
<harmw> agreed
<smoser> harmw, thanks for all your input. i will try to get a re-build of cirros and a proper "0.3.2~pre2" out.
<smoser> (proper meaning actually available in simplestreams data and without 'local
<smoser>  and tagged and such)
<smoser> before i leave for holiday (friday).
<harmw> ah excellent
<smoser> i just pushed the configuration change.
<harmw> ill take a look tomorrow
#cloud-init 2013-12-18
<slagle> i'm using the ubuntu cloud image (saucy) with cloud-init.  for some reason, the cloud-init final module is not getting run on boot
<slagle> an OpenStack Config Drive is my only data source, with a user-data file supplied that runs some commands
<slagle> after boot, if i run "cloud-init modules --mode=final" from the command line, i see that those commands from user-data are executed
<slagle> but for some reason, that's not happen automatically
<slagle> it's like the upstart job /etc/init/cloud-final.conf never gets executed
<smoser> slagle, you have 'upgrade' on ?
<smoser> becausae it sounds like you're hitting
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384
<slagle> smoser: can you elaborate? not sure what you mean?
 * slagle looks
<smoser> but that *should* be fixed.
<smoser> ie, 'apt_upgrade' in config or something.
<smoser> 2 things.
<smoser> it sounds like you dont.
<smoser> so i'm kind of confused as to what would be the issue here.
 * smoser really wishes 'cloud-localds' supported configdrive output so i could easily test this locally.
<slagle> i don't have apt_upgrade specified in /etc/cloud/cloud.cfg
<smoser> slagle, woudl you be able to give me the config drive ? just to  make this easier for me?
<smoser> to try to reproduce ?
<smoser> or just a tarball of the contents filtered for whatever you dont want to share is fine.
<slagle> smoser: sure, let me upload it somewhere
<slagle> smoser: https://s3.amazonaws.com/slagle-tripleo/openstack-config-drive.iso
<slagle> i have cloud-init version 0.7.3-0ubuntu2
<slagle> i have this as well on the image, not sure if it's needed or not:
<slagle> root@undercloud:/etc/cloud/cloud.cfg.d# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg 
<slagle> # to update this file, run dpkg-reconfigure cloud-init
<slagle> datasource_list: [ ConfigDrive ]
<smoser> thanks. i'll take a look.
<smoser> slagle. using the latest 'release' image http://cloud-images.ubuntu.com/releases/saucy/release-20131215/
<smoser> i got your iso and booted in kvm with that attached.
<smoser> user-data script does get executed, but fails due to lack of some state i'm guessing is on the machine
<smoser>  (ie, no /var/lib/heat-cfntools/cfn-init-data file)
<smoser> fwiw, i always recommend
<smoser> output: {all: '| tee -a /var/log/cloud-init-output.log'}
<smoser> so that output of that runcommand would go into that file.
<smoser> (in addition to the consoel)
<slagle> smoser: hmm, ok, let me try that
<smoser> slagle, i also patched the image before booting it like this
<smoser> sudo mount-image-callback -v disk1.img -- sh -c './backdoor-image/backdoor-image --password=passw0rd $MOUNTPOINT && echo "output: {all: \"| tee -a /var/log/cloud-init-output.log\"}" >> $MOUNTPOINT/etc/cloud/cloud.cfg.d/output.cfg'
<smoser> mount-image-callback is from cloud-utils, its simliar to guestfs-mount or something.
<smoser> backdoor-image is lp:~smoser/+junk/backdoor-image (just a tool for inserting a user in)
<smoser> heres cloud-init-output.log
<smoser> http://paste.ubuntu.com/6595411/
<slagle> smoser: ok, it's working for me now too. i believe it was user error on my part
<slagle> smoser: sorry for wasting time
<slagle> i had an old /var/lib/cloud/seed directory that i think was throwing it off
<smoser> slagle, yeah. it might hvae read that. you should see in /var/lib/cloud-init.log a message to that affect. 
<smoser> something like:
<smoser> Dec 17 21:35:32 test1 [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.5 finished at Tue, 1
<smoser> 7 Dec 2013 21:35:32 +0000. Datasource DataSourceNoCloudNet [seed=/var/lib/cloud/seed/noc
<smoser> loud-net][dsmode=net].  Up 345444.06 seconds
<smoser> without the bad line breakage
<smoser> harmw, around ?
<smoser> harmw, i had 2 things for you.
<smoser> a.) i just kicked off a cirros build that i hope to push as 0.3.2~pre2.
<smoser> b.) questions about bsd cloud-init work
<kwadronaut> awwww i got a fan here 'grmbl i'm gonna purge cloud-init everywhere'
<smoser> ?
<harlowja> kwadronaut don't do it!
<harlowja> ha
<harlowja> not much of a fan if said person wants to purge cloud-init
<harlowja> *anti-fan?
<kwadronaut> well, not my problem ;-) 
<kwadronaut> it's more of a config problem of him, i presumed.
<harmw> smoser: shoot
<smoser> ah. good.
<smoser> were you planning on doing any work on tha t?
<smoser> i told harlow he could look at pulling it
<harmw> fbsd? ofcourse
<smoser> do you think its in a reasonable state to be at least somewhat useful at the moment ?
<harmw> 'at least somewhat useful' pretty decent description :)
<harmw> i pushed some changes some days ago
<harmw> how nice, the mergerequest also shows all additional commits from the latest push
<harmw> (im still fairly new to launchpad/bzr)
<smoser> harlowja, so if you want to give that a review and ideally a quick spin, i'm happy to take it. especially with the promise of more maintenance there (loose promise)
<harlowja> ah, quick spin, thats gonna be harder :-P
 * harlowja doesn't have freebsd on hand, haha
<harmw> harlowja: you could use oz to create an image
<harlowja> harmw ya, there is a freebsd guy here that has an image
<harlowja> let me see if i can locate that
<harmw> https://github.com/clalancette/oz/blob/master/oz/FreeBSD.py
<harmw> hm, I should probably give that some love aswell sometime soon
<harlowja> ya, let me see if i can get the image, then i'll bug u about how to package this and get it into the image :-P
<harmw> hehe
<harlowja> k, email sent, i downloaded the thing he produced once, just can't remember where it is now
<harmw> ok
<harmw> fbsd comes with all required dependencies, except jsonpatch
<harlowja> even boto?
 * harlowja not sure how many dependencies he has in this image, will see
<harlowja> hmmm, then the question becomes at y! is do we want to allow people to use more freebsd, ha
<harlowja> certain projects are addicted to it
<harmw> pkg install python27 py27-yaml py27-requests py27-prettytable py27-cheetah py27-boto dmidecode e2fsprogs gpart sudo
<harmw> that does it, plus manually setup.py-ing jsonpatch
<harmw> (hm, why did i put sudo in that list again..)
<harlowja> k
 * harlowja so apparently his is close to stable/10
<harmw> yea
<harmw> im still running my stuff on rc3, and just now noticed it's at rc2 already
<harlowja> hehe, i wonder if i can get the freebsd guy to do this work for me, lol
<harlowja> http://people.freebsd.org/~sbruno/ :-P
<harlowja> although i tried to get him to help with cloud-init before, don't think he ever got time
<harmw> thats 'the other guy'? 
<harlowja> ya
<harmw> haha 
<harmw> nice
<harlowja> he's like a clone of bill murray (not kidding)
<harmw> :)
<harmw> reminds me of ghostbusters :)
<harlowja> ha
<smoser> harlowja, we should ditch the boto i think.
<smoser> and go back to crawling ourselves.
<smoser> i like being fickle.
<harlowja> sweet, harmw smoser  he is going to help me do this so 'Want me to do that in
<harlowja> a test image for you so it doesn't make you want to kill kittens?'
<harlowja> hahahaha
<harlowja> bruno is pretty funny and nice guy, ha
<harmw> hehe
<smoser> except for the slaughtering of cute little kittens.
<harmw> harlowja: can he also give some pointers on where to go to have jsonpatch added in the repo?
<harmw> and ofcourse cloud-init
<harlowja> harmw sure
<harlowja> the repo i am assuming being upstream freebsd ?
<harmw> yes
<harlowja> k
<harmw> my /usr/local/etc/pkg.conf:
<harmw> PACKAGESITE         : pkg+http://pkgbeta.freebsd.org/${ABI}/latest
<harlowja> k, gonna see if i can get him to jump on here
<harlowja> instead of josh-proxy, lol
<smoser> thanks harlowja and harmw 
<smoser> any doc on this that can/should go into cloud-init would be great.
<harlowja> agreed
<harmw> definately
<harmw> speaking of which, some more docs inside cloud-init's code would be nice aswell :p
<harlowja> :)
<smoser> booo
<harlowja> tears of happiness from smoser 
<harlowja> lol
<harlowja> harmw lets see if bruno gets in here, then we can all bug him, haha
<harmw> :)
<harlowja> *save the kittens!
<smoser> harmw, ok. 0.3.2~pre2 is "officially" released.
<harmw> nice job
<smoser> yeah. pita.
<smoser> i dont have much automation around the publishing. i had to re-figure out how i did a mirror to cloud files.
<harlowja> hmmm, where's bruno, arg
<harmw> no tiny little deploy.sh?
<smoser> i'd only done the push to clodu files once.
<smoser> and the tools for signing and producing the simplestreams i'd not documented for myself either.
<smoser> :)
#cloud-init 2013-12-19
<harmw> harlowja: you've had a chance to test fbsd yet?
<harlowja> harmw have not, ran into sean at lunch, pulled him by ear to get into here, lol
<harlowja> will see if i can do it today, if he doesn't show :-P
<harlowja> *sean bruno
<harmw> today? what's your tz anyway :p
<harmw> since its 22:44 overhere
<harlowja> us/PST
<harmw> ah, thats -8
<harlowja> :)
<harlowja> move to PST, ha
<harmw> ha
<harlowja> i think the image sean gave me is gonna require changes anyway
<harlowja> he setup it up with paritions afaik
<harlowja> do u do that?
<harlowja> in our RHEL images its split into 3 disks (ramdisk, kernel,root)
<harlowja> which matches more of whats typical in openstack
<harmw> root@ravage:~/dev/cloud-init # df -h
<harmw> Filesystem      Size    Used   Avail Capacity  Mounted on
<harmw> /dev/vtbd0p2    1.8G    1.1G    562M    68%    /
<harmw> devfs           1.0K    1.0K      0B   100%    /dev
<harmw> thats my fbsd instance
<harlowja> k, ya, thats what i thought it woudl be
<harlowja> he's done
<harlowja> swap is gpt partition 4
<harlowja> arg, irc
<harlowja> "/ is gpt partition 3"
<harlowja> "/home is gpt partition 5"
<harlowja> "swap is gpt partition 4"
<harmw> haha, whats wrong with /say :p
<harlowja> ah
<harlowja> should use that
<harmw> ye
 * harlowja never can remember all the irc commands
<harmw> anyway, your layout makes it impossible to resize the / fs
<harlowja> right
<harlowja> thats what i've told him
<harmw> which is why i've put it al in 1
<harlowja> right
<harmw> plus its easies to test :p
<harlowja> correct
<harlowja> so i gotta get him to change that anyway, since the image i got is with those 3
<harlowja> and have to remove stuff in /etc/rc.d/yahooinit (that i think cloud-init is replacing with better goodness)
<harmw> probably, yes
<harlowja> :)
<harmw> /bin/rm that :>
<harlowja> def
<harlowja> i'll poke him again today if he doesn't jump on here
<harlowja> told him about these partition changes yesterday
<harmw> sure
<harlowja> so i'll let u know, hopefully soon but hard to nail sean down, lol
<harmw> sure thing :)
#cloud-init 2013-12-20
<ikkeT> hi, anyone succesfully using write_files ?
<ikkeT> I tried but i always fail with it
<ikkeT> is there any documentation about it other than: http://cloudinit.readthedocs.org/en/latest/topics/examples.html
<ikkeT> cant get this to work:
<ikkeT> write_files:
<ikkeT>  - content: |
<ikkeT>         <html> <body> <img src=kortti.jpg></img></body> </html>
<ikkeT>     path: /var/www/index.html
<ikkeT>     permissions: '0755'
<smoser> ikkeT, it will only work on something recent
<smoser> probably like 0.7.2+
<smoser> wont work on 12.04
<smoser> (whihc is 0.6.3)
<mdorman> i have an image with some cloud-init config stuff baked in to /etc/cloud/cloud.cfg.d  It works great when instances are launched from it.  however, if there is any user-data provided to the vm, then only the user-data stuff gets run by cloud-init, not the baked in stuff.  is there some config i need to make sure they both run?  and ideall, the baked in stuff runs first?
<harlowja> harmw yt
<harmw> omg
<harmw> just, like, now
<harmw> lol
<harlowja> trying to advice sean more on the partitioning
<harlowja> and wanted your current advice
<harlowja> his setup
<harmw> lol, well, I'm by no means a fbsd addept :) but sure, shoot
<harlowja> his layout  /dev/vda1: unknown, /dev/vda2: vfat, /dev/vda3: ufs, /dev/vda4: unknown, /dev/vda5: ufs
<harlowja> 1 --> boot
<harlowja> 10:40 2 --> silly cfg things
<harlowja> 10:40 3 --> /
<harlowja> 10:40 4 --> swap
<harlowja> 10:40 5 --> /home
<harlowja> so wondering what your layout is since i'm pretty sure yours works and his likely needs changes
<harlowja> *ignore 10:40 timestamp
<harlowja> so i think to make it work in openstack 1 -> ramdisk like file (not sure here)
<harlowja> 2,3,4,5 collapsed (swap removed really)
<harlowja> harmw so i think if sean figures this out he might know who to talk to to get ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/11.0-CURRENT/amd64/Latest/ adjusted for cloud usage
<harlowja> which would be cool
 * harmw reads
<harmw> ok, i've got only vda1
<harmw> i think i've pasted that the other day
<harmw> /dev/vtbd0p2    1.8G    1.1G    562M    68%    /
<harmw> and thats it
<harmw> no swap, obviously
<harmw> vm-images, didn't know that existed :)
<mdorman> how can i configure something to run during the cloud-init "local" stage.  i have some items baked in to my image that i want to run before anything that comes from user-data.  the cloud-init phase runs the user-data stuff before the locallyconfigred items
<harlowja> harmw thx, i'll let sean know about that
<harlowja> mdorman ubuntu right?
<mdorman> harlowja: no, centos 6 actually
<harlowja> ah, hmmm, so if u want something to run before cloud-init local, then u need to jump into rc.d i think and make a corresponding script there
<mdorman> well i'm fine with my stuff running in clout-init local, i'm just having a hard time figuring out how to actually configre it to run in the local phase
<mdorman> vs. the later cloud-init stage
<harlowja> so i guess a question is what is your stuff, additional scripts?
<harlowja> scripts provided via userdata?
<harlowja> *also i guess important what cloud-init version
<harlowja> or since u baked it into the image its probably different scripts
<harlowja> if its the baked in stuff, then u just need to make sure those scripts get ran before cloud-inits rc.d order does
<harlowja> which i think is done by putting a dependency on your rc.d scripts to go before cloudinit
<harlowja> or u can adjust http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/sysvinit/redhat/cloud-init-local#L27 (and make cloud-init-local depend on your scripts running)
<harlowja> 'Required-Start:    $local_fs $remote_fs' 
<mdorman> ok. yeah.  i mean my baked in stuff are just a couple things to set up some custom AD auth stuff, and register the vm into spacewalk.  both things we want to have happen before anything from user-data runs
<mdorman> i really wanted to avoid having a custom rc script to do that, but sounds like that might be the only wya to make sure it goes before user-data
<mdorman> cloud-init 0.6.3, btw
<harlowja> mdorman sorry, just got back
<harlowja> mdorman ya, i think thats really the only way
<harlowja> if you are baking the images then u can do this
<mdorman> yep, everything i'm finding is pointing toward that
<mdorman> right.
<mdorman> just was hoping to do everything with cloud-init, so i wouldn't have to manage a service.  but, not a big deal.
<harlowja> cloud-init will run user-scripts, but it doesn't have the concept (afaik) of running image-based scripts, excepts u afaik to just use the system service
<mdorman> yep
<mdorman> thanks for the advice
<harlowja> *which varies between operating systems
#cloud-init 2014-12-15
<yfried> hi, I'm looking for a way define ifcfg files via cloud-init. can anyone point me to an example?
<harlowja> fyi smoser  + others ; http://blog.oddbit.com/2014/12/10/cloudinit-and-the-case-of-the-changing-hostname/
<harlowja> perhaps time to adjust that?
#cloud-init 2014-12-16
<tennis> smoser: u there??? Got a min for a question?
#cloud-init 2014-12-18
<harmw> harlowja_: ping!
<harlowja_> harmw sup
<harmw> you've got some python code I can use to test computing power?
<harmw> eg. I'm eager to find out if my ARM system is capable of running keystone :)
<harmw> instead of my old Atom x86_64 rig
<harmw> and instead of googling, I'd figure you would know this one by heart :p
<harlowja_> hmmmm
<harlowja_> python test computing power, ha
<harlowja_> u so funny
<harlowja_> lol
<harmw> hehe
<harlowja_> u can try running http://docs.openstack.org/developer/taskflow/examples.html#distributed-mandelbrot-complex a bunch of times, ha
<harlowja_> but meh, i'm not sure i'd use python, lol
<harmw> maybe I should take some to explain my case :p
<harmw> my Atom can't realy keep up with keystone, generating tokens/keys etc.
<harlowja_> ya, somewhat expected i guess ;)
<harmw> and I'd like to see/test if an ARM platform would be up for the task
<harmw> true, true
<harlowja_> run keystone on it? see what happens
<harmw> but instead of diving in with 'just' running keystone on it, I'd like to run some test code first :) 
<harmw> lol
<harlowja_> hmmm, use python-cryptography to generate a crap ton of keys?
<harlowja_> see how that goes?
<harlowja_> i think thats what keystone uses anyway
<harlowja_> https://cryptography.io/en/latest/ 
<harmw> hmk
<harmw> thanks, I guess :p
<harmw> running time openssl rand -hex 100 is interesting as well
<harlowja_> or https://github.com/pyca/cryptography/tree/master/tests/hazmat/primitives (might have something interesting?)
<harlowja_> run there test suite a bunch of times, ha
<harmw> :) thanks
<harlowja_> suree, let me know how it goes :-P
<anotheral> would anyone be able to help me figure out why my new AMI based off the amazon ubuntu 14.04, isn't updating the 'ubuntu' account authorized_keys?
<anotheral> ok i think i *finally* figured it out
<anotheral> I override the AuthorizedKeysFile in /etc/ssh/sshd_config, and I think the cloud-init script is horking on that
<anotheral> yes!  here's my sshd_config
<anotheral> AuthorizedKeysFile%h/.ssh/authorized_keys /etc/ssh/authorized_keys/%u
<anotheral> sorry
<anotheral> AuthorizedKeysFile %h/.ssh/authorized_keys /etc/ssh/authorized_keys/%u
<anotheral> (this allows for a second directory of authorized keyfiles named for the user
<anotheral> )
<anotheral> but cloud-init is actually creating the path "/home/ubuntu/.ssh/authorized_keys /etc/ssh/authorized_keys/ubuntu" (space is part of the dir name), with the right key in it
#cloud-init 2014-12-19
<anotheral> there, opened https://bugs.launchpad.net/cloud-init/+bug/1404060
<smoser> harlowja_, around ?
<harlowja_> smoser sup
#cloud-init 2015-12-14
<cornfeedhobo> hello, i am in desperate need of help; i am tired of uploading failed images
<cornfeedhobo> i am building a lvm+xfs centos7 machine, and cloud-init won't grow the root filesystem no matter what i try
<cornfeedhobo> i have make sure growpart is available and lvm + xfs tools
<cornfeedhobo> made*
<smoser> growpart does not work on lvm devices.
<smoser> natorious, thought he might add taht support, or might have another solution for you.
<smoser> cornfeedhobo, ^
<cornfeedhobo> that explains alot
<cornfeedhobo> you would think lvm support would be more demanded. it makes dealing with a few things much better
<cornfeedhobo> natorious: i would be interested in hearing what you would suggest
<smoser> cornfeedhobo, i am becomming more interested in it.
<smoser> but largely lvm is not that improtant in "cloud" . stricktly because yoru OS disk is just easilyi small and ptu other things on other disks.
<cornfeedhobo> well, for example, resizing
<cornfeedhobo> without lvm on aws, you must stop, detach. snapshot. relauch as larger. resize.
<cornfeedhobo> with, you just create a new volume, attach, add to lvm, resize
<cornfeedhobo> its a bit like a jbod then, but no downtime
<smoser> ok. the response woudl be that you should not do that on your OS disk
<smoser> that make sense ? i do see the value, and woudl accept patches to cloud-init or growpart.
<smoser> but just keep your data on an ebs volume that you make an lvm on
<smoser> and keep your os disk small and OS only.
<cornfeedhobo> hmmm
<cornfeedhobo> ya there is some logic to that, isn't there
<cornfeedhobo> smoser: i'll brb. i also had a plan ext4 image that wasn't expanding also, but i think i know what that problem was. should be back around in about an hour and a half
#cloud-init 2015-12-15
<cornfeedhobo> smoser: i forgot to come back yesterday and say i solved my issue, and i think you are right. seperate data volumes is probably a better convention
<cornfeedhobo> in case someone is interested in solving lvm support, the "docker-storage-setup" project seems to have solved it
#cloud-init 2015-12-16
<smoser> cornfeedhobo, thanks
<cornfeedhobo> thank you as well :)
<jlebon> hi all, hitting a strange issue on f23 with 0.7.6
<jlebon> it seems like commands in the bootcmd section are run twice?
<jlebon> once during init-local and once more during 'init'
<jlebon> is this the expected behaviour?
<smoser> jlebon, they should only run at one of those
<jlebon> smoser: hmm strange. i'll check if fedora is holding any patches that might affect this
<jlebon> which of the two should it run at, init-local?
<smoser> well, it should run when the datasource is found.
<smoser> i'm guessing you're config drive ?
<jlebon> no, i'm using NoCloud
<jlebon> giving seedfrom on the commandline
<smoser> hm.
<smoser> just thinking, it coudl be a bug.
<jlebon> hmm, yeah I don't see anything related to this in the fedora patches
<smoser> jlebon, can you post /var/log/cloud-init* ?
<smoser> somethign is goign wrong in that if init-local finds a datasource and runs the init modules, then init should not
<jlebon> smoser: sure, let me get it from a fresh instance
<smoser> and generally speaking if you're seeding you want 'dsmode=net' . but it shoudl actually do the right thing if it is seeded from http
<jlebon> smoser: ahhh, i'm using dsmode=nocloud;s=/path/to/cloud-init-data
<jlebon> smoser: http://ur1.ca/ocj8n
<jlebon> the ln msg is how I know it's runnign twice
<jlebon> i didn't use ln -f so it fails on the second time
<smoser> yeah, so you want:
<smoser>  ds=nocloud,dsmode=nocloud-net
<jlebon> even though i'm seeding locally?
<smoser> well, if you want networking to be guaranteed when the bootcmd commands run
<smoser> then you need dsmode=net
<smoser> maybe you need journalctl or systemctl info
<smoser> the pasete there isnt very much other than to show like you said that its probably running twice. although the order there is still odd.
<smoser> (that they come before the 'running init')
<smoser> maybe i just misunderstood your problem
<smoser> bootcmds do run on everyboot
<smoser> not "per-instance"
<jlebon> i can paste the full journal if needed
<jlebon> right, it does run which is good
<jlebon> my issue is that it runs twice
<jlebon> it's not a big deal in this case because i can make my bootcmd idempotent
<jlebon> by using ln -f
<smoser> it shoudl not run twice per boot though.
<smoser> it shoudl run once on every boot
<jlebon> right, so there's a mismatch somewhere -- did you mean earlier that using the ds=nocloud,dsmode=nocloud-net cmdline would work around this issue?
<jlebon> the files themselves are located locally, so i don't really need net access for sourcing
<smoser> well, it shoudlnt happen. can you get the full log ?
<smoser> i'm nto sure why you're running twice.
<smoser> the order of the log you pasted is really confusing
<smoser> either the ln ran the second time *before* the 'running "init"' message
<smoser> or the order is just flipflopped
<smoser> (stdout/stderr possibly)
<jlebon> smoser: journalctl output http://ur1.ca/ocj9o
<jlebon> notice the bootcmd module running on line 1143 and on line 1427
<smoser> well notice line 996 and 1252
<smoser> init-local is running twice
<jlebon> hmm, i thought the 1252 line was just to "close off" the logs for that instance
<jlebon> they're the same PID
<jlebon> the first bootcmd is run by pid 742 and the second by 856
<smoser> jlebon, so, what is suppose to happen is that cloud-init local should write a file that said that it ran . and the cloud-init-net shoudl basically do nothing
<smoser> i really can'tdebug this much more right now, i'm sorry.
<smoser> i do think you're right about the same pid though (i was wrong about local running twice)
<jlebon> smoser: no worries, this is not high priority for me as well, just thought i'd investigate on here
<jlebon> thanks for looking into it
<jlebon> if it does become a bigger issue for me, i'll see if i can look into it
#cloud-init 2015-12-17
<jlebon> smoser: quick q, is there an easy way for third-party pkgs to add their own cloud-init modules? do they just drop them in e.g. /usr/lib/python3.4/site-packages/cloudinit/config/ ?
<jlebon> or is there a more friendly location?
<smoser> jlebon, there may be.
<smoser> but you still have to get them into the list of modules to call
<jlebon> smoser: right, of course. i'd have to override e.g. cloud_config_modules
<smoser> it doesnt look like it.
<smoser> but i'd be open to patches to support it.
<jlebon> awww, that's too bad
<jlebon> hehe, will think about it
<jlebon> my use case is adding a module that autogenerates the account passwords -- it didn't seem like anything did that already
<smoser> jlebon, you can put a random somewhere. to get a random password.
<smoser> let me see if that is generic
<smoser> yeah, see: doc/examples/cloud-config.txt
<smoser> but it just sets them to a random value (and writes it to console)
<jlebon> wow, nice. I missed those
<jlebon> hmmm -- yeah, that could work. though I would prefer some way to fetch them so I can print them on the console myself in another boot-up script
<smoser> jlebon, yeah, that pretty clear missing necessary feature.
#cloud-init 2015-12-19
<spaok> hello
#cloud-init 2016-12-19
<smoser> rharper, able to quickly look at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/313458 ?
#cloud-init 2016-12-20
<smoser> magicalChicken, are you around today ?
<magicalChicken> smoser: I am out of town, but I can do some work
<magicalChicken> I may just be slow to respond
<smoser> do you want to ? i got somethign i want to have done today, but i can do it or let you do it :)
<magicalChicken> Sure
<smoser> ok. give me 10 minutes. and i'll point you at it
<magicalChicken> smoser: ok
<smoser> bah. almost there,
<smoser> magicalChicken, you still there?
<magicalChicken> smoser: yeah
<smoser> for bug https://bugs.launchpad.net/maas/+bug/1582323 . i regressed that when i "fixed" it.  i want to put the old behavior back in
<smoser> git commit is 63501f44 that regressed
<magicalChicken> smoser: so it was the fix for LP:1595302 that brought the regression in
<smoser> yeah
<smoser> so just rewvert the commit
<smoser> but need to add some unit tests to make sure i dont stupidly break it again
<smoser> :)
<smoser> i just read code wrong
<smoser> and didn't test
<smoser> util.mergemanydict has a wierd function signature
<smoser> it kindo f works backwards ...
<magicalChicken> smoser: sure, I'll get some unittess written
<magicalChicken> smoser: oh, yeah it makes more sense with reverse=True
<smoser> magicalChicken, http://paste.ubuntu.com/23659465/
<smoser> that is what i had just been playing with
<smoser> lots of times i find that decorators get long winded, so this was kind of a way to try to make them less, but still wrap the 3 functions inside that fetch_base_config that basically need tob e wrapped
<smoser> magicalChicken, ^ you see that ? if you think you make reasonable tests without my fancy / overengineered wrap_and_call, then feel free :)
<magicalChicken> smoser: Yeah, just saw that
<magicalChicken> it looks pretty good to me, the decorator is pretty simple
<smoser> see test_ntp_handler
<smoser> that is what i wanted to kind of avoid
<smoser> the large indentation that you get with something liek that
<smoser> or the lots-of-decorators if you got that route instead.
<magicalChicken> yeah, the with chain gets hard to read
<magicalChicken> it may actually be nice to move the multi-wrap function into helpers, the 'cloudinit.stages.util' could be set with a kwarg to make it more generic
<smoser> magicalChicken, yeah.
<smoser> i was thinking that too
<magicalChicken> smoser: I moved the wrap_and_call function into helpers and copied the unittest you had written in
<magicalChicken> I've gotta head out for a little bit, but I can the rest of the unittests for fetch_base_config() written this afternoon
<smoser> magicalChicken, great.
<smoser> magicalChicken, note, i have *kwargs wrong
<smoser> should be **kwargs
<smoser>         try:
<smoser>             return func(*args, *kwargs)
 * powersj really needs to get the unit tests on merge requests going :(
<aixtools> hi, working on my aix-port. Quick question on /var/lib/cloud/data/ssl (specific case), but also on function and usage of /var/lib/cloud by deployed images
<aixtools> i am guessing - but it seems /var/lib/cloud is where the first data-source gets "copied" to, so that on subsequent reboots - the "networked" datasource is no accessed - everything is coming from /var/lib/cloud.
<aixtools> I would like a simple world :)
<aixtools> re: /var/lib/cloud/data/ssl - is cloud-init (going to copy) that data from there to where "it should be" aka dist default location(s)?
<aixtools> new question: keeping it to two sources: what is the difference between source NoCloud and ConfigDrive - both seem to be referencing (virtual) CD/DVD, but how are they different?
<aixtools> bbl, have to change wifi networks...
<powersj> smoser: Are you going to have a chance this week to look at the integration test merge proposal? I believe all the changes you requested are complete.
<magicalChicken> smoser: I got that fixed in the version in helpers. I'm running a bit late, but I will get a pull request submitted for the unittests tonight
#cloud-init 2016-12-21
<smoser> powersj, yeah... i will try. i really hope to
<magicalChicken> smoser: I got the pull request in for the unittests for stages.fetch_base_config() at: https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/313665
<smoser> magicalChicken, around ?
<magicalChicken> smoser: I'm around late today and tonight, but I'm not going to have internet for most of the day
<jgrimm> magicalChicken, I'm guessing you just missed smoser; he's taking half day vacation today, won't be back on till afternoon
<magicalChicken> jgrimm: oh, okay. I'm mostly offline this week anyway, since I'm picking up days next week
<jgrimm> magicalChicken, yep ,all good.
<smoser> magicalChicken, ii merged and pushed. thanks. a few small changes.
<magicalChicken> smoser: awesome, thanks
#cloud-init 2016-12-23
<powersj> smoser: I think I got the heading levels wrong on the readthedocs stuff, the sidebar has a few extra line items now :(
<smoser> oh. that sok
<smoser> they are really wierd
<smoser> see doc/rtd/index.rst
<smoser> theres a comment there that says what i was trying to keep
<smoser> but its really , really odd... you kind of pick whatever your headers are dynamically and it just tries to sort it all out
<smoser> powersj, i filed doc/rtd/index.rst
<smoser> https://bugs.launchpad.net/curtin/+bug/1652329
<smoser> as i'm tired of being bit by distro package failed to build when tox was fine :)
<smoser> i'm not going to fix that today though, just going to fix the one pycodestyle complaint
<powersj> ok - so effectively run tox with latest pip packages and run tox with pinned versions of zesty? what are you pinning to?
<powersj> do we need pinned versions for every release?
<smoser> so there are 2 kinds of tox targets.
<smoser> a.) unit test
<smoser> b.) code style
<smoser> for 'a', it makes sense to have pinned versions everywhere that we need to run
<smoser> for 'b', it doesnt really matter. we pick one set and stick with it.
<smoser> currently, we've got this cocktail mix where we run some code-style checkers during the package build
<powersj> ok
<smoser> and thus we're forced to have code style pass with  multiple different versions of checkers
<smoser> which, kind of is insane
<powersj> yeah
<smoser> but i never really thoguht clearly on this before. i think i've convinced myself that running coding style checks during package build is not helpful
<smoser> as long as they're run on trunk (and ideally gate)
<powersj> yeah
<smoser> do you want to fix your formating ?
<smoser> you can if you want, and have it in a 0.7.9 that is due... inthe next 2 hours
<powersj> reading your comments now to see what I need to do
<smoser> powersj, http://paste.ubuntu.com/23673609/
<smoser> that is my interpretation
<smoser> of your sections and such
<smoser> the whole dynamic "pick whatever you want" makes it really hard for a human to even understand what should be what
<powersj> smoser: that looks like what I have now. I didn't realize having === on top and bottom made a difference
<smoser> really odd
<smoser> :)
<powersj> do you want to just merge that? or want an mp?
<smoser> if you're happy with that, i'll do it.
<powersj> go for it
<smoser> note i dropped the toc stuff at the top too
<powersj> same :)
<powersj> I saw how it looked at didn't like it
<nacc> smoser: time for a /topic update? :)
* smoser changed the topic of #cloud-init to:  cloud-init 0.7.9 released 2016-12-23. 0.7.10 open. reviews: https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews | fly the W
 * smoser leaves the fly the W on
<smoser> powersj, i'm almost out the door
<smoser> gonna upload to zesty 0.7.9
<powersj> smoser: sweet!
<powersj> thanks for the merge this week
<smoser> http://cloudinit.readthedocs.io/en/latest/ works
<smoser> and 0.7.9 magically showed up
<powersj> ah yes that looks good :)
<powersj> smoser: so on deck for cloud-init then is the tox changes, updating the jenkins runs to use master and not my branch, and I have another merge from magicalChicken I still need to do.
<smoser> powersj, curtin has similar nonsense in its tox
<smoser> wrt pycodestyle stuff
<powersj> ah yes
<smoser> we shoudl fix that in the same way both places
<powersj> agreed, when you get back I'll try to have merge requests for both
<prometheanfire> hmm, for some reason cloud-init isn't passing gw or hwaddr info to the old netconfig stuff
<prometheanfire> 0.7.9 has the bug too
<prometheanfire> seems like it's being removed before even being passed to the gentoo portions
<nacc> prometheanfire: you're sort of hitting the worst time calendar wise :) EOD/EOW/EOY
<prometheanfire> I know
<prometheanfire> I've been needing to rewrite the gentoo pieces
<prometheanfire> guess this gives me a reason
<prometheanfire> I thought it was working, wonder why it's not now
<prometheanfire> maybe newton changed the format or something
#cloud-init 2017-12-18
<robjo> smoser: rharper blackboxsw with 17.2 out the door, any chance we can make some progress on the following?
<robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334241
<robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904
<robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904
<robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
<robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334338
<robjo> The firs two are related in that the second is the requested test, at least from my perspective, for the first MP
<robjo> I know there is a "grand plan" and that keeps you guys rather busy.
<robjo> I just get the impression that the "grand plan" does not consider review time until shortly before the next release, which means stuff from people that are no directly working on the "grand plan" is kind of lacking attention
<robjo> Also I need some guidance on the "ip" vs "ifconfig" issue, see my mailing list message
<smoser> robjo: thank you for pestering
<smoser> :)
<smoser> yes, we do have more things and we really do not mean to ignore you
<smoser> robjo to be fair, nobody's "grand plan" should involve "throw stuff in right before you mark a release".
<smoser> you were owed review cycles well before that.
<robjo> I'd rather not be the squeaky wheel, but will be if needed, would rather figure out a way where things move somewhat more steadily
<robjo> was not trying to imply the "stuff gets thrown in before release" it just appears that that happens to be the time when you guys have some extra cycles to look at the MPs
<robjo> the combination of that is probably less than desirable ;)
<rharper> we generally walk through all of the active MPs to see if there are any that are ready to take, vs. things that need more discussion/fixing/etc;  so yeah, there is a sweep; but it's not mean to ignore proper review, but rather picking up any "low hanging fruit" before the release
<robjo> Well this one was really low hanging ;) https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334241
<blackboxsw> speaking of clearing out the review queue. I'm +1 on landing https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<blackboxsw> Robert said that azure's backend infrastructure will not activate the preprovisioning until March timeframe I think (and likely targetted at xenial only to begin w/)
<blackboxsw> s/Robert/Douglas/
<blackboxsw> ...  <dojordan>  let me try again. As long as my changes land in an azure image it will help unblock our testing. But when we go GA (middle of next year hopefully), we will start only preprovisioning (using the ovf setting) 16.04 LTS. The purpose of the marker file is incase we occur a VM reboot in the middle of the process
<blackboxsw> no time to review Robert's branches
<blackboxsw> now time to review*
<blackboxsw> robjo: was there a specific use-case or intent you had behind supporting multiple post-up clauses in a single interface stanza per  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904
<blackboxsw> I had a couple inline questions there in that review
<blackboxsw> hrm rharper smoser I honestly don't know where to go with https://code.launchpad.net/~james-hogarth/cloud-init/+git/cloud-init/+merge/333657 per the discussion we had today about dropping ifconfig (iptools) logic. Most of this is logic around printing that table or network interfaces in cloud-init-output.log. Did we agree we don't care about that human-readable printed table across the switch from net-tools ->
<blackboxsw> iproute2 ?
<smoser> blackboxsw: i'm still lost time :-(
<blackboxsw> most of his branch is allowing for printing the same tabular format of network information and if we are diverging from that in our switch iproute2 support, do we take his branch and adapt it or just minimally adapt ip cli output to print network config info in cloud-init-output.log
<smoser> i've stuck on getting history into curtin that will allow us to merge into distro branches.
<smoser> :-(
<smoser> wrt that merge.
<smoser> i think that we *do* want identical table output.
<smoser> there are two  orthoganal things here.
<smoser> a.) getting off of 'net-tools'
<smoser> b.) possibly dropping human friendly table output for networking information
<smoser> i really dn't think that 'b' is necessary, and definitely in my head is not somethinng that just tags along with other changes.
<smoser> we should not tag along external-facing changes with internal implementation changes.
<blackboxsw> but 'b' was created via calls to 'a' tools
<blackboxsw> ok, though we do more work to support 'b' by parsing and restructuring iproute2 tools output to continue using that nework information table in cloud-init-output.log
<blackboxsw> if we are okay with the cost of external consistency, then no problem.
<rharper> blackboxsw: IIRC, we're happy that the output matches as a check, longer term we may look to using 'ipaddress' instead to dump the output in table form rather than translation, but that's not a requirement for 'a' as smoser  says
<rharper> blackboxsw: right, I recall we're ok with the extra work for now
<blackboxsw> ok was just re-reviewing your comments on that branch rharper and wanted to make sure.   So. since hogarth did initial work in this space > 1 month ago, do I attempt a branch using his as prerequisite and we'll approach that MP as co-authored maybe ?
<blackboxsw> or try to work his branch into something we like as is and have a follow up to drop other uses of net-tools
<rharper> blackboxsw: wouldn't a rebase of the branch against master apply cleanly as-is ?
<blackboxsw> rharper: yes, though it doesn't tackle any of the Azure specific uses of ifup/down for that network bounce (and I think his branch fails integration tests)
<rharper> those are in freebsd path, right ?
<blackboxsw> rharper: we have one in ubuntu path that currently fails on artful bionic https://bugs.launchpad.net/cloud-init/+bug/1722668
<ubot5`> Launchpad bug 1722668 in cloud-init (Ubuntu) "Azure: bouncing of network device/publishing of hostname fails on artful" [Critical,Confirmed]
<rharper> blackboxsw: ah, ok
<blackboxsw> so I'll add a couple review comments toJames' branch and create a separate branch for the azure fix.
<rharper> I wonder if we really need an interface bounce or something else
<blackboxsw> and make it dependent on his
<blackboxsw> I thought we were going to look at the potential of avoiding the bounce on systemd/networkd
<rharper> if we do need to link down/link up;  we may want some sort of cloud-init/net/utils space where we could abstract the needs (and implement them via which tools are present)
<rharper> right, well, networkd can't bounce anything
<rharper> one needs to poke interfaces via ip link set instead
<blackboxsw> as the hostname update seems to already be handled in the systemd space. But, that statement requires a little validation.
<rharper> yeah
<blackboxsw> also around this space, dojordan's approved preprovisioning branch is relying on this bounce too to notify Azure's new IMDS service of preprovisioning 'complete'
<blackboxsw> though, again, that's only Xenial
<blackboxsw> which has net-tools still
<rharper> yeah; maybe we can get dojordan or paulmey to confirm what's required w.r.t just a link down/up or some other action (re-issue dhcp request) etc
<rharper> and get that documented
#cloud-init 2017-12-19
<catmando> hey all
<rharper> howdy
<catmando> does anyone have experience with cloud-init on power?
<rharper> catmando: yeah;  though cloud-init's python, it's generally the same everywhere; is there a specific environment ?
<catmando> yes, ubuntu 16.04 on ppc. the issue is likely in documentation
<catmando> i am trying to work out how to set the hostname of a new instance to be the name of the instance if the dns reverse lookup does not work
<rharper> catmando: interesting problem
<catmando> powervc has a custom module that allows set_hostname_from_dns
<rharper> cloud-init isnt going to do any hostname reverse lookup validation; none that I'm aware of
<catmando> (and also set_hostname_from_interface // set_dns_shortname)
<catmando> https://www.ibm.com/support/knowledgecenter/SSXK2N_1.4.0/com.ibm.powervc.standard.help.doc/powervc_custom_modules.html
<catmando> but I am wondering how to deal with the case when that module failes
<catmando>  s/failes/fails
<smoser> blackboxsw: did you want to talk about ssh keys ?
<blackboxsw> yeah will rejoin. thought powersj needed to be there but let's chat
<blackboxsw> hrm, was just deploying azure bionic and I expected to see the traceback about ifdown/ifup not working.... but the bionic images in azure have net-tools now. hmm
<smoser> blackboxsw: are you sure ?
<smoser> i have absolutely seen the warn before
<smoser> oh. yeah, we talked about htis yesterday
<smoser> byobu
<smoser> :)
 * blackboxsw checks what brought that in https://pastebin.ubuntu.com/26216372/
<blackboxsw> ohh right
<blackboxsw> but I'm wondering why this failed before... didn't we always have byobu on artful when you filed  https://bugs.launchpad.net/cloud-init/+bug/1722668
<ubot5`> Launchpad bug 1722668 in cloud-init (Ubuntu) "Azure: bouncing of network device/publishing of hostname fails on artful" [Critical,Confirmed]
<blackboxsw> nevermind
<blackboxsw> I crossed my stream
<blackboxsw> I crossed my streams
<blackboxsw> that's from ifupdown which byobu doesn't depend on
<blackboxsw> and still isn't in the bionic images
 * blackboxsw will  find another reason why bionic azure images didn't traceback like in the bug above.
<smoser> bionic images (lxc) definitely do have net-tools
<blackboxsw> maybe I just need to provide a hostname in cloud-config to reproduce the issue with bounce
<smoser> http://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.squashfs.manifest
<smoser> but id' think you'd still see a warning or something
<smoser> as even if it weret there the bounce would use it
<blackboxsw> https://pastebin.ubuntu.com/26216469/
<blackboxsw> ok able to reproduce if my cloud-config contains a hostname different than the hostname I created the vm with on the cmdline
<blackboxsw> paulmey: have a couple mins for some Azure questions? The datasource currently bounces the network interface across hostname changes (from user-data) to update DDNS with the new hostname record on the interface. It relies on ifdown ifup utils that are now abset(deprecated) in artful and bionic because ifupdown package is no longer in cloud images in factor of moving to the iproute2 package in a systemd world.
<blackboxsw> The question is, it seems in artful and and bionic a simple call to hostname (without the ifdown ifup) has properly updated ddns as I can nslookup <mynewhostname> and get the authoritative answer.
<blackboxsw> I mean the question is can we drop the bounce in the systemd world because my only understanding about the hostname bounce was that it was used to inform DDNS, which seems to work.
<blackboxsw> I'll run through xenial test without the bounce versus artful test without the bounce and see if anything differs. What I have found is that even if we get a Traceback on missing ifdown/ifup cmdline utils, hostname of the machine is still set properly, and nslookup <myhostname> is being properly reported by ddns
 * powersj is in vpc hell
<rharper> powersj: nuke it
<powersj> heh
<blackboxsw> powers nukes hell.... story at 11
<blackboxsw> just noted on xenial if we set hostname via #cloud-config, DDNS is not updated properly. artful and bionic do handle it (despite the ifdown/ifup Traceback)
<blackboxsw> might open a bug against xenial azure for this
<blackboxsw> hrm de-opped because of my nick registration maybe?
<blackboxsw> powersj: rharper can you op me again?
<rharper> yeah
<blackboxsw> I think I've sorted IRC nick registration
<powersj> how does on OP someone
<blackboxsw> like that ;)
<powersj> so if my nick is registered does that mean I keep op?
<blackboxsw> for us powersj click their nick on the right and click 'op'
<rharper>  /OP <nick>
<rharper> powersj: the ChanServ handles that
<blackboxsw> or that ;)
<blackboxsw> :)
<powersj> ah
<rharper> once you have a registered nic via NickServ, then the channel owner says the following registered nics have these ops, etc
 * rharper regrets giving too much power to blackboxsw 
<powersj> ah so if smoser has it setup to give us OP it will auto-give us op?
<rharper> yeah
<rharper> prolly needs to poke some config in the ChanServ bot
<blackboxsw> my children regret the same.... and their fighting against the "dad power" ever since
<powersj> he should do that for #curtin as well
<rharper> yeah
<blackboxsw> rharper: are yahoo employees covered under our CLA?
<blackboxsw> we just got a branch from someone at yahoo
<rharper> blackboxsw: not sure
#cloud-init 2017-12-20
<powersj> blackboxsw: if you are still around... in your review of the ec2 merge you asked me to add pyyaml to the tox ci env?
<smoser> powersj: i think i did auto-op you.
<smoser> very confusing
<smoser> http://boto3.readthedocs.io/en/latest/reference/services/ec2.html
<smoser> look for 'UserData' there
<smoser> should you give that as base64 or not?
<powersj> smoser: I use https://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances that function
<powersj> which says UserData='string',
<powersj> If you are using a command line tool, base64-encoding is performed for you, and you can load the text from a file. Otherwise, you must provide base64-encoded text.
<smoser> and then in bold
<powersj> and then right below..
<powersj> yeah :)
<smoser> we should figure that out and pass None (or not pass the kwarg)
<smoser> rather than b''
<blackboxsw> hrm not understanding why the diff isn't being created here https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335468
<blackboxsw> might resubmit a third time and avoid prerequisite branch on hogarth's branch as they don't really have to be dependent branches
<smoser> no ifrs
<smoser> no idea. maybe you have an oops in your inbox
<powersj> smoser: I'll look at the b'' shortly
<blackboxsw> ok resubmitted the simple fix (not dependent on any branch)
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335470
<powersj> smoser: https://paste.ubuntu.com/26223182/
<powersj> wiht **args of course
<smoser> powersj: yeah
<smoser> i just hit 'submit' on a review there.
<powersj> heh I took the t2.micro comment from your last review
<smoser> blackboxsw: responded
<blackboxsw> smoser: sure on not even bouncing if ifupdown not present, though FreeBSD uses ifconfig down|up. Shall we also check for that ?
<blackboxsw> or alternately, if util.is_FreeBSD()?
<blackboxsw> it makes things simple to avoid combing through datasource.cfg['hostname_bounce']['command'] for ifdown references
<blackboxsw> the only problem I see with not calling bounce_network_with_azure_hostname is that the bounce method actually might be the only thing calling set_hostname in the first place. Will check
<smoser> blackboxsw: hm...
<blackboxsw> yep, the bounce_network_with_azure_hostname won't actually allow cloud-init to observer #cloud-config\nhostname: mynewname   declarations
<blackboxsw> on artful/bionic the vm is then left with the hostname it was originally created with in azure's UI/CLI
<blackboxsw> because no set-hostname is called
<smoser> ?
<smoser> confused.
<smoser> and that code is confusing
<blackboxsw> could show you in hangout. bionic vs xenial behavior, that bounce method is wrapped up in all the get/set_hostname from metadata/user-data calls in the temp_hostname context manager, if we skip it altogether we don't actualy 'see/observe' the user-data hostname provided
<blackboxsw> probably best if we don't hangout if you value the rest of your work day ;)
<smoser> blackboxsw: ok. hanoug
<blackboxsw> yeah, might be worth refactoring that code a bit since I'm touching it. Those two functions are tightly coupled and they probably shouldn't be.
<powersj> hmmm console log output isn't always there
<blackboxsw> smoser: Xmas list for me:
<blackboxsw>  - https://code.launchpad.net/~sw37th/cloud-init/+git/cloud-init/+merge/335217
<powersj> python3 only?
<blackboxsw>  - https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333772  (land as-is and followup with separate refactor branch of yours)
<blackboxsw>  - and once I've updated this:    https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335470
#cloud-init 2017-12-21
<smoser> powersj: oh. humm. that sucks.
<smoser> 2 possible things
<smoser> a.) it could be broken, i surely hope that is not the case, and in my experience it isnt
<smoser> b.) it updates only like every 4 minutes. but i think after a terminate its usually there (and is for as long as an instance sticks around)
<smoser> blackboxsw: i got 2 and 3 on that list (i hope your mp for azure was ready)
<smoser> note your review tool complained about it. not sure why.
<blackboxsw> ohh curious, as that commit message was empty on line 2 hrm
<blackboxsw> thanks for trying to use the review tool
#cloud-init 2019-12-16
<parallel21> Runnin' `Cloud-init v. 19.2-36-g059d049c-0ubuntu2~18.04.1`
<smoser> did anyone push on pstart issue ?
<smoser> lxd and lxc-pstart issue.
<Odd_Bloke> Not that I'm aware of yet, no.
<powersj> I believe rharper had the action item out of the retrospective to at least file a bug
<rharper> smoser: I'll file an issue day
<smoser> ok. great. thank you.  I think the simplist thing that shows error is just the 'init', pstart /bin/true (and automatic stop), then init again.
<smoser> as i showed, it gets wrong ownership of lots of files in /
<rharper> y
<smoser> so either  abug in shifting, or a race
<meena> Good morning happy people
<meena> https://github.com/canonical/cloud-init/pull/61#pullrequestreview-332699520 approved Goneri's pr
<smoser> rharper: https://github.com/smoser/qa-scripts/tree/mini-issue if you're interested.
<smoser> that simplifies lxc-pstart for the recreate
<smoser> to
<smoser> a.) use profile name pstart1 (to avoid messing up an existing system)
<smoser> b.) not create the network at all
<smoser> c.) in-line psuedo-init ... so one python program contains all the required stuff to recreate
<rharper> smoser: thanks
<meena> has anyone had a chance to look at https://hackmd.io/3-YBj1t9TAeKhmfLBQUjXQ ?
<Odd_Bloke> meena: So I had a look, of course.  It's not 100% clear to me what our concrete next steps are.  Is that something you have an idea of (and are looking for buy-in for), or are next steps what you're hoping we can define together?
<meena> Odd_Bloke: define buy in?
<meena> and, no, i think i'd like to define next steps together, and then present it to the mailing list (again)
<meena> so, the buy in or opposite of that would come from there. i'm not looking for anyone to commit to implementing thisâ¦ just maybe help with testing xD
<Odd_Bloke> meena: By "looking for buy-in" I meant "reach consensus on a proposed approach".  But it sounds like we're more in the latter mode, coming up with next steps.
<meena> *nod *nod
<meena> Odd_Bloke:  anyway, i like your suggestion, and i'd like to try that out.
<Odd_Bloke> meena: IMO, I think that's the next step, to inform our next decision.
<blackboxsw> meena: I'm +1 on that (spike) suggestion  approach. It'll give is a feel for how to iterate through that work for most network-related functions.
<meena> We should def name all cloudnet classes Cloudnet.
<blackboxsw> powersj: Odd_Bloke so I'm finally getting around to reviewing your PR https://github.com/canonical/cloud-init/pull/109 through bug https://bugs.launchpad.net/cloud-init/+bug/1854236.    I think this is both a documentation issue and also something that we need to limit when set-hostname is being called only when init.is_new_instance().
<ubot5> Ubuntu bug 1854236 in cloud-init "documentation for cc_set_hostname module doesn't mention runs during "init" phase" [Medium,Triaged]
<blackboxsw> I think we'll need a little extra logic added there to cloudinit/cmd/main to limit such early(init-local) hostname setting to allow for folks how manually reset hostname to something different after initial provisioning
<blackboxsw> I'm adding comments and suggestions to the PR
<Odd_Bloke> blackboxsw: That sounds like a separate bug, not something that a docs PR should be expanded to include.
<Odd_Bloke> (And we'd obviously have to think carefully about changing that behaviour.)
<blackboxsw> the initial bug that was filed related to that feature was due to initial boot on dhcp servers hosted in vSphere land which used DDNS to set dns entries to whatever the initial DHCP request asked for. (which in the case of a large automated deployment came up all 'ubuntu').
<blackboxsw> sure Odd_Bloke we can/should separate that to another branch. I'll file the doc comment suggestions and a pointer to something that we'd need to do to avoid resetting hostname on every boot to what our potential stale metadata has
<blackboxsw> I don't think a doc-only branch fixes #1854236 in entirety though
<powersj> The bug is literally titled "documentation for..."
<powersj> :P
<blackboxsw> true. but it's documentation of a behavior that probably shouldn't be as broad as it is
<blackboxsw> we can correct the docs to current behavior "per always", but I'd like to get it back to "per instance"
<blackboxsw> and it *could* have impact on vsphere, though I *think* not.
<blackboxsw> and it will affect anyone that has come to rely on resetting the hostname across every boot
<powersj> agreed, I think that is worth a new bug
<powersj> blackboxsw, why not file one while it is fresh
<blackboxsw> yeah will do, will wrap up your review and file a new bug.
<meena> > @blackboxsw> and it will affect anyone that has come to rely on resetting the hostname across every boot < oh no!
<meena> Goneri: """This branch is out-of-date with the base branch""" â 61 needs another rebase.
<blackboxsw> meena: I think I missed a joke. didn't you file the original  bug with concerns about losing the manually set hostname ? :)
<meena> reported by https://launchpad.net/~nathanst
<meena> https://bugs.launchpad.net/cloud-init/+bug/1855170 this is the one i reported.
<ubot5> Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,Incomplete]
 * blackboxsw mixed up context. meena I saw your "oh no" and interpreted that as a concern about some user-story that I might have missed \
<Goneri> meena, done. Can someone review the freebsd_net_renderer please?
<blackboxsw> to me, why would someone need to attempt reset hostname to the metadata provided hostname every boot (unless they were working with some sort of cfgmanagement solution that required stable local hostnames for ongoing maintenance of a node).
<blackboxsw> and *if* someone needed that stable hostname, wouldn't they just use a cfg management/maintenance solution like chef/puppet/ansible etc to maintain that hostname over time (instead of relying on cloud-init's initial system config features)
<meena> blackboxsw: usually when you have a hostname that your CfgMgmt depends on; you don't want that to change ever, not even on reboot.
<meena> some people draw a fact from the hostnameâ¦ soâ¦ yeah, changing it might blow up the systemâ¦ or rather, if the hostname changes, your CfgMgmt system might blow up your server.
<blackboxsw> right. so if root  manually came in and changed the hostname some days/months after initial boot, I don't think I'd want cloud-init to re-set to the original metadata next time I rebooted the vm/machine
<blackboxsw> because root probably had a good reason to change such a hostname
<blackboxsw> or you got hacked ;)
<meena> accidents happen.
<meena> ð¤·ââï¸
<blackboxsw> heh true.
<blackboxsw> community notice: cloud-init 19.3.41 is officially released via Ubuntu SRU (stable release update) Xenial, Bionic Disco and Eoan have access to this version and cloud-images will be updated nightly to get these new cloud-init bits. Thanks for all the help folks!
<meena> \o/
<meena> congratulations
<Odd_Bloke> blackboxsw: I believe the usecase (as described in that doc PR :) is for platforms that require hostname to be set correctly for DHCP to complete successfully.
<Odd_Bloke> In such cases, you would want it to happen every boot, because otherwise you aren't getting networking. :p
<blackboxsw> ... specifically during initial deployment I had thought.
<blackboxsw> but I suppose that need would hold true across reboot so you don't have DDNS hostname colllisions
<meena> i thought once the hostname is setâ¦ it'sâ¦ set.?!
<Odd_Bloke> Perhaps it is limited to initial deployment, I don't know for sure.
<meena> we should find out, before we try too hard ;)
<Odd_Bloke> Not sure that DDNS plays into it here, platforms that require you to conform to their expected hostname aren't going to use the presented hostname dynamically.
 * meena is composing a mail to  the list about REFACTORING 2.0
<meena> message sent.
<meena> and here's today's dead code: https://gist.github.com/10d305f65194470cb6dbad58690f123c
<powersj> nice
<powersj> I think we have that check turned off in pylint currently
<blackboxsw> Odd_Bloke: I think in the past via juju deployed models on vSphere, each deplyed node ended up requesting ddns-hostname ddns-domain-name params or something to that end. Our intial dhcp setup in init-local was passing these values to whatever platform dhcp server was available, and in some vsphere/juju cases DDNS got updated to set/reset any dhcp IP allocated as hostname 'ubuntu'.   This caused problems for juju
<blackboxsw> deployments trying to install software and react to hooks as juju would automatically start trying to talk to a new vm that 'became' ubuntu as far as DNS was concerned.
<blackboxsw> I think what we decided to do, is say that, if the platform provides cloud-init with a desired hostname, we should assume that's what we need to be always, unless a user overrides that with user-data (#cloud-config) etc.
<blackboxsw> though I probably misunderstood your comment about "conform to their expected hostname aren't going to use the presented hostname dynamically."
<blackboxsw> I guess you meant that cloud-init wouldn't expect the metadata hostname to change on platforms that expect hostname to conform. (So don't expect to support people coming in after the fact to change the hostname if it was launched with a certain meta-data hostname set)
<Odd_Bloke> blackboxsw: I was talking about the cases where we _need_ this, not the cases where it has caused problems.
<Odd_Bloke> So I think we were just talking past one another a bit. :)
<blackboxsw> roger. /me works on reading comrehension
<meena> if i had known this would be formatted like ass, i would have run it thru fold(1) before sending https://lists.launchpad.net/cloud-init/msg00237.html
<meena> welcome to the mailing list, we hope you like the 1970s!
<meena> remember when i did code reviews via email? those were the times
<meena> Goneri: did you some how lose my find_fallback_nic function from your code?
<rharper> smoser: https://github.com/lxc/lxd/issues/6632
#cloud-init 2019-12-17
<Goneri> meena, hum, let's me check
<Goneri> meena, my bad, I forgot it during the last rebase. sorry. I just pushed a fix.
<rharper> smoser: http://paste.ubuntu.com/p/CQRgW9HQRw/
<smoser> rharper: yeah, i verified that works too.  was ripping the ctn_cfg out from other places.
<meena> powersj: vendor-data is a bit underdocumented.
<meena> I fucked up.
<Odd_Bloke> meena: Should we revert https://github.com/canonical/cloud-init/commit/e2840f1771158748780a768f6bfbb117cd7610c6 ?
<meena> Odd_Bloke: i guess we an leave the test
<meena> I'll blame it on mental overload
<Odd_Bloke> These things happen!
<Odd_Bloke> Particularly in complex code bases. :p
<Odd_Bloke> meena: Do you want to propose the revert, or shall I?
<powersj> meena, is that a kind hint that you want me to go write more docs :)
<smoser> rharper: https://github.com/cloud-init/qa-scripts/pull/15
<meena> Odd_Bloke: im on the go, rn, so if you're in the code, go ahead
<meena> powersj: if i knew how it works, i'd do it myself ð
<Odd_Bloke> meena: Cool, will do.
<Odd_Bloke> meena: https://github.com/canonical/cloud-init/pull/116
<Odd_Bloke> powersj: I don't appear to be able to add non-Canonical folks as reviewers on PRs. :/
<Odd_Bloke> meena: (I would have added you as a reviewer if not for ^. :p)
<powersj> Odd_Bloke, nor am I
<rharper> smoser: thanks!
<Odd_Bloke> powersj: I've tried adding meena as a Read permitted collaborator.
<Odd_Bloke> To see if that adds them to the list.
<Odd_Bloke> smoser: rharper: So is that profile PR a fix for the pstart issue we're seeing, or a simplification so we can reproduce it with less moving parts?
<smoser> that shoudl fix it, Odd_Bloke
<smoser> the issue was the "cached" (memory) config that we restored on exit was stale.
<rharper> Odd_Bloke: I used smoser simplified reproducier to file the issue with lxd yesterday
<smoser> and then we got the response expected
<rharper> and stgraber pointed out the error and the solution as well
<smoser> and used --dump-commands
<rharper> ah, yes
<smoser> which was helpful.
<rharper> and I followed up with a, and what _should_ we do
<rharper> so, team effort to extract a solution
<smoser> the one thing i'im not sure about is how old 'lxc profile add'
<smoser> is
<rharper> right, I've not verified it on Xenial
<rharper> but we're running bionic + snap-based lxd in CI
<smoser> seems old enough.
<rharper> ooo, it's 19.4 day
<smoser> https://github.com/cloud-init/qa-scripts/pull/16 gets rid of:
<smoser>   rcontainer = remote + ":" + name if remote else name
<meena> Odd_Bloke: did i approve too early (before accepting your invitation?) or do i still have no write access, like my github message says?
<Odd_Bloke> meena: I didn't grant you write access, I just granted you explicit read access, in the hopes that would mean you showed up in the drop-down listing potential reviewers.
<Odd_Bloke> And I _think_ that worked.
<blackboxsw> Odd_Bloke: that looks good on my side and I see someone was able to set meena as reviewer
<blackboxsw> on the PR
<Odd_Bloke> I believe meena just reviewed it of their own accord.
<Odd_Bloke> Rather than it being requested.
<Odd_Bloke> (Obviously it _was_ requested, just not through the GH UI. :p)
<blackboxsw> followup suggestion on powersj's set hostname branch https://github.com/canonical/cloud-init/pull/109#discussion_r358543988
<Odd_Bloke> blackboxsw: Which platforms base dynamic DNS on the initial DHCP requests?  (I thought those were the platforms that the current behaviour doesn't work well for?)
<blackboxsw> and per our conversation yesterday Odd_Bloke I'm in agreement with you I think. If metadata or user-data set a hostname for a platform, it seems correct for cloud-init to assert that the hostname remains set per the opinionated deployment directives.
<blackboxsw> Odd_Bloke: juju driving some version of vSphere (so OVF + juju). I believe the interactions of both substrates rely on DDNS that falls over during our init-local sandbox DHCP attempts caused the DDNS in that scenario to keep transfering DNS entries for the 'ubuntu' hostname to different vms as they are brought up
<blackboxsw> Odd_Bloke: more context on that is here https://bugs.launchpad.net/cloud-init/+bug/1746455/comments/5  and here https://bugs.launchpad.net/cloud-init/+bug/1746455/comments/23
<ubot5> Ubuntu bug 1746455 in cloud-init "cloud-init vSphere cloud provider DHCP unique hostname issue" [High,Fix released]
<blackboxsw> Odd_Bloke: so juju deploys (or deployed) some magic that adds DHCP hostname request logic to the deployed target (and a lot of cloud-init user-data). It is possible that there is an issue in juju machine deployment logic that needs a bit more investment to make sure to avoid triggering this case
<Odd_Bloke> blackboxsw: I'm a little confused.  That's a case where this behaviour causes problems, but your proposed paragraph suggests that that's the motivation for having this behaviour.
<blackboxsw> Odd_Bloke: sorry I wasn't clear here. That is the case where cloud-init's behavior of setting hostname from metadata before running sandboxed dhcp client fixes that issue.
<blackboxsw> if we didn't set hostname before our sandbox dhclient run, we would break juju deploying on vsphere
<Odd_Bloke> Oh, you know what, I'm thinking of Azure-specific behaviour.
<Odd_Bloke> Which is handled by the datasource, not through this route.
<blackboxsw> I don't really know about platforms on which us setting hostname early and repeatedly resetting is causing problems
<blackboxsw> ahh right Or
<blackboxsw> ahh right Odd_Bloke
<Odd_Bloke> Which explains why I couldn't understand what was going on. :p
<blackboxsw> yet, doesn't explain why I didn't. But, some mysteries are better left unsolved
 * powersj looks for 19.4
<blackboxsw> Odd_Bloke: powersj rharper I know we are in discussion about best process for reviewing PRs quickly
<blackboxsw> can we in the nearterm label a PR with incomplete if it is waiting on Proposer feedback/resolution?
<Odd_Bloke> blackboxsw: I have reservations about introducing a stopgap when we plan on having a full process in the next few days.
<blackboxsw> thanks Odd_Bloke +1
<blackboxsw> will leave PRs undecorated at the moment
 * blackboxsw was going through initial upstream 19.4 board tasks 
<Odd_Bloke> blackboxsw: Did you cut 19.4 yet?
<blackboxsw> Odd_Bloke: nope, doing that in a matter of a few minutes
<Odd_Bloke> blackboxsw: Because I think https://github.com/canonical/cloud-init/pull/116 is worth including.
<blackboxsw> just reviewing CI right now
<blackboxsw> ok Odd_Bloke will review that
<blackboxsw> Odd_Bloke: was there a bug associated with that issue?
<blackboxsw> that PR I mean
<blackboxsw> as in, how did we/you discover the issue?
<blackboxsw> just an errant extra chaneset that snuck  in in https://github.com/canonical/cloud-init/commit/e2840f1771158748780a768f6bfbb117cd7610c6 or something else triggered discovery of the issue?
<blackboxsw> *changeset*
<blackboxsw> maybe here is the context I was missing https://bugs.launchpad.net/cloud-init/+bug/1854594/comments/8
<ubot5> Ubuntu bug 1854594 in cloud-init "lock passwd implemented wrong on FreeBSD" [Medium,Incomplete]
<blackboxsw> the last comment being, oops shouldn't have done that
<meena> blackboxsw: basically, the thing that hid the issue was system_info/distro being overwritten â me misinterpreting pw(8) wrong.
<blackboxsw> my question around that PR is that https://bugs.launchpad.net/cloud-init/+bug/1854594 isn't really fixed then right?
<ubot5> Ubuntu bug 1854594 in cloud-init "lock passwd implemented wrong on FreeBSD" [Medium,Incomplete]
<blackboxsw> and wasn't really broken either right
<meena> blackboxsw: you said something about reading comprehension too this morning, i believe.
<meena> blackboxsw: yes, wasn't broken â¹ shouldn't have fixed it.
<meena> i'm very sorry about this mishap.
<blackboxsw> roger doger. ok I'm marking that bug as invalid then. no worries
<blackboxsw> ok merged
<meena> my perfect score is gone
<Goneri> you're still perfect <3 :-)
<Odd_Bloke> meena: Honestly, don't worry about it, these things happen.
<Odd_Bloke> And we caught it!
<blackboxsw> heh.
<blackboxsw> +1
<meena> Odd_Bloke: i just wish we caught it before SRU release xD
<Odd_Bloke> meena: The SRU doesn't affect FreeBSD, so no harm there.
<Odd_Bloke> blackboxsw: So I've got someone to open up the CLA migration MP and PR.  Do I merge the PR and close the MP out?
<blackboxsw> Odd_Bloke: since we can't use review-mps from git@github.com:CanonicalLtd/uss-tableflip.git  anymore due to our security config, we'll have to do that manually. Right, merge PR locally once you confirm the matching LP Branch proposal on associated LP account. And Manually close the LP-side branch proposal with a comment pointing to the upstream github commitish
<blackboxsw> and you've probably seen ad-m's next PR that can followup after this
<Odd_Bloke> Yep, I'm working through his things ATM.
<blackboxsw> tkx
<Odd_Bloke> blackboxsw: Why do I need to merge the PR locally?
<Odd_Bloke> (As opposed to using the GH UI.)
<blackboxsw> Odd_Bloke: sorry I meant using the GH UI. (I meant 'local' in the sense of in github, not over in launchpad)
<Odd_Bloke> Cool
<blackboxsw> Odd_Bloke: rharper what do we think? https://github.com/canonical/cloud-init/pull/77
<ahosmanMSFT> Hi all, I'm working on this cloud-init (Azure) bug that might be connected to cloud config. Here is the behavior: start a vm, ssh via password, cloud-init clean, then restart. Now the user can't ssh via password. Looking into DataSourceAzure, the functionality is there, but the behavior doesn't match. Am I missing something? https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n1206
<blackboxsw> static6 eni support for 19.4?
<blackboxsw> I'm thinking hold as it needs a bit more review and we will have a 20.1 early next year.
<blackboxsw> hi ahosmanMSFT.
<blackboxsw> ahosmanMSFT: could it be related to the byteswapping branch that just landed?
<blackboxsw> I didn't see this in SRU testing
<blackboxsw> which didn't have your fix
<ahosmanMSFT> blackboxsw: This bug has apparently existed for a while
<blackboxsw> ahosmanMSFT: a good attempt would be to try reproducing on current xenial vms
<blackboxsw> to see if it has the same broken behavior
<ahosmanMSFT> I've reproduced the behavior on both xenial and bionic
<blackboxsw> generally after a cloud-init clean all cloud-init semaphores should be removed and all modules should get re-run
<blackboxsw> ok is there a bug filed ?
<ahosmanMSFT> blackboxsw: Not sure, saw this in our internal backlog and picked it up
<ahosmanMSFT> It's because the password get's redacted in ovf-env.xml file
<blackboxsw> best bet is to check to see if a bug is filed @ https://bugs.launchpad.net/cloud-init/+bugs. If not, then try filing a bug at https://bugs.launchpad.net/cloud-init/+filebug file a bug  with the cloud-config required to show the problem and attach logs tarfile from cloud-init collect-logs .  Generally we'd expect to see the ssh, password and user_groups cofig modules running after a clean.  But, due to sourcing
<blackboxsw> content from ovf-env.xml this information might not be present anymore after a clean.
<blackboxsw> as you said, if it was redacted and only run once
<blackboxsw> then new cloud-init boots after clean may not have that metadata available anymore
<blackboxsw> TBH, cloud-init clean is a developer tool, and a blunt hammer.  It's not intended for typical use-cases for ongoing system maintenance
<blackboxsw> so it is not a typical consumer use pattern that I'd expect you to jump through hoops to support if the initial provisioning of a vm does properly setup that configuration
<ahosmanMSFT> blackboxsw: Is there a way to persist metadata, or mitigating the issue cleaning+restarting causing password lock. You are right, the typical consumer wouldn't run in to this. But I'd like to find some resolution
<rharper> ahosmanMSFT: sounds similar to this, https://bugs.launchpad.net/cloud-init/+bug/1849677
<ubot5> Ubuntu bug 1849677 in cloud-init "azure locks existing user if instance id changes" [Medium,Fix committed]
<rharper> ahosmanMSFT: w.r.t persistent metadata, it's not clear to me what you mean;   for debugging, I suggest creating an additional user on the system after the initial launch with known password and sudo privs;  then after a clean + reboot; login as the secondary user you created, so you can collect cloud-init logs
<ahosmanMSFT> rharper: That is the bug I am dealing with. Creating a new user/pass was how I was getting on the vm to collect logs.
<rharper> ok, I didn't see any logs from the secondary boot to contrast with the first one;
<rharper> ahosmanMSFT: interesting, so it seems to me that it' may be a side-effect of redacting the password value in ovf-xml file;  that's not kept in original state in your boot, clean, reboot scenario, right ?
<rharper> in which case, on the secondary boot (after clean) the xml contents are still redacted instead of the original password supplied ?
<rharper> does that sound plausible ?
<rharper> ahosmanMSFT: on comment #2, https://bugs.launchpad.net/cloud-init/+bug/1849677/comments/2  ; there is mention that the iso only exists on first boot
<ubot5> Ubuntu bug 1849677 in cloud-init "azure locks existing user if instance id changes" [Medium,Fix committed]
<ahosmanMSFT> rharper: exactly, the ovf is cached via waagent
<ahosmanMSFT> Trying the committed fix now...
<rharper> ah, ok
<rharper> sounds like you've reproduced the issue that we now have a fix for
<meena> anyone considered adding CircleCI to Travis?
<meena> â¦ instead of just having travis.
<ahosmanMSFT> blackboxsw rharper: Has this fix worked for you? https://bugs.launchpad.net/cloud-init/+bug/1849677/comments/5
<ubot5> Ubuntu bug 1849677 in cloud-init "azure locks existing user if instance id changes" [Medium,Fix committed]
<ahosmanMSFT> I just tried it and it didn't work, still get locked out
<rharper> ahosmanMSFT: we did not test it directly
<rharper> do you have logs from first and second boot ?
<Odd_Bloke> meena: Why would we use both?
<meena> Odd_Bloke: for freebsd.
<Odd_Bloke> meena: Oh, I didn't know they supported it.
<blackboxsw> ok upstream release bug complete for 19.4
<blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1856761
<ubot5> Ubuntu bug 1856761 in cloud-init "Release 19.4" [Undecided,New]
<blackboxsw> adding a simple script update PR to make this a bit more automated
<meena> Odd_Bloke: at least last time i looked at it
<powersj> blackboxsw, thank you!
<meena> Odd_Bloke: cirrus ci
<meena> https://cirrus-ci.org/guide/FreeBSD/
<Odd_Bloke> Aha, OK.
<Odd_Bloke> That explains why I couldn't find it in the Circle docs. :p
<meena> they're all ci and c and stuff
<blackboxsw> https://github.com/CanonicalLtd/uss-tableflip/pull/27
<blackboxsw> upstream release process update and simple script
<blackboxsw> upstream release commit https://github.com/canonical/cloud-init/pull/121
<meena> so, anyone got anything against merging Goneri's pr, so we can go on with life and refactor cloudinit.net? https://md.hecke.rs/z17JGX4HT4emH5jTEhMuTA
<blackboxsw> thanks Odd_Bloke . scubbed changelog (we probably need that change for log2dch too now to clean out merge markers)
<blackboxsw> srubbed even
<Odd_Bloke> blackboxsw: Well, we shouldn't need to any longer, as we've disabled merge commits.
<meena> who allowed hetzner to be merged without documentation???
<blackboxsw> hrm I think I may have run out of daylight and reviewers on the upstream 19.4 cut for today, rharper or powersj or smoser if around I've queued  https://github.com/canonical/cloud-init/pull/121 from tip for cloud-init 19.4 release
<blackboxsw> if that lands today, I'll publish 19.4 to focal, and send a release email & discourse post out
#cloud-init 2019-12-18
<smoser> blackboxsw: sorry. kind of confused by Odd_Bloke comments there.
<smoser> oh. i see. you fixed.
<bugbuilder> Hi everyone! I'll be very glad if someone could give me an advise, about a weird issue that I having with a cloud-init and a packer template. When I boot the image the first time, using terraform provider openstack and libvirt for local, sometime the cloud-init doesnt start. I check the systemd and said tha the cloud-init is disable but this services was enabled in the image creation.
<bugbuilder> This happends randomly. Sometime the same temple provision the VM without problems.
<bugbuilder> *template
<smoser> bugbuilder: its probably related to ds-identify.
<smoser> but you should file a bug and attach cloud-init collect-logs output
<smoser> ideally inboth "good" and "bad" cases.
<bugbuilder> yep I checked them and in the case when the cloud.init service is disable I dont have any log.
<bugbuilder> I'm thinking to put the datasource for openstack to see if the identifier could fix it
<smoser> run cloud-init collect-logs
<smoser> anad attach the output
<bugbuilder> k
<smoser> it gets mroe than /var/log/cloud-init.log
<bugbuilder> I looking on them.. Let me read them first and I coming back in a bit thank yoy smoser
<smoser> sure.then jsut file a bug. attach and you can ping me with a link here.
<bugbuilder> I red dmesh and journal without luck.
<bugbuilder> *read
<bugbuilder> but I found a problem about the DS as you said
<bugbuilder> di_report:
<bugbuilder>   datasource_list: [  ]
<bugbuilder>   # reporting not found result. notfound=disabled.
<bugbuilder> Do you think if I create thee datsource for openstack this problem could be disapper?
<smoser> iits possible that if you put 'datasource_list: [OpenStack, None]' into /etc/cloud/some.cfg that it will fix the issue.
<bugbuilder> also which would be for local develoment
<smoser> but you can potentially fix the issue for other people "the right way" (tm)
<smoser> er... help fix the issue
<smoser> by filing a bug
<bugbuilder> sure, how I could do that.
<smoser> by filing a bug with that info ;)
<bugbuilder> lol
<bugbuilder> yeah sure but I dindt know where
<smoser> is it ubuntu ?
<bugbuilder> Suse
<smoser> https://bugs.launchpad.net/cloud-init/+filebug
<bugbuilder> let me try the fix, I'll come with news and then I'll fill the report
 * smoser has to go afk
<smoser> night night
<blackboxsw> thanks smoser. yeah I had force pushed the changes. I need to stop doing that so the PR more easily represents the fixes.
<smoser> i think force-push to a pr is fine
<mkrai_> Hi I am trying to boot a baremetal server in OpenStack with two network, the provisioning is succesful but the node is not reachable. I saw that the instance folder in /var/lib/cloud was not created.
<mkrai_> When i try to boot the same server with only network, provisioning is succesful and the node is reachable too. cloud-init does the network configuration properly.
<mkrai_> Can someone please help me identify the problem with cloud-init or how to debug this issue?
<mkrai_> It seems that the DataSourceConfigDrive is not known to cloud init
<mkrai_> smoser, Hi
<mkrai_> blackboxsw, portdirect Odd_Bloke rharper Hi, Can you please help?
<tribaal> mkrai: most people on this chan are in US timezones. If you can prepare the following it would help them be efficient when they wake up:
<tribaal> mkrai: run "cloud-init collect-logs" and upload the result of that command somewhere (this collects the logs as the name implies)
<mkrai> tribaal, Ok thanks I will do it now
<mkrai> tribaal, I get error when i run this command "IOError: [Errno 2] No such file or directory: '/var/log/cloud-init-output.log'"
<mkrai> I have /var/log/cloud-init.log
<tribaal> that is interesting - perhaps cloud init is not actually run at boot for some reason
<mkrai> tribaal, http://paste.openstack.org/show/787707/
<tribaal> looks like it works fine as far as I can tell
<tribaal> I'm afraid you'll have to wait for somebody more knowledgeable than myself to come online
<mkrai_> tribaal, np thanks for the help though :)
<mkrai_> blackboxsw, portdirect Odd_Bloke rharper smoser Here are the logs http://paste.openstack.org/show/787707/
<smoser> mkrai__: i think you're hitting
<smoser>  https://bugs.launchpad.net/cloud-init/+bug/1801364
<ubot5> Ubuntu bug 1801364 in cloud-init "persisting OpenStack metadata fails" [Undecided,Fix committed]
<smoser> so i would suggest trying https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/
<meena> Goneri: your branch is again out of synch?
<meena> you gotta rebase against tip
<Goneri> I'm rebasing it
<Goneri> thanks meena for the notice
<Goneri> For the record. I've tested the branch against FreeBSD-10, 11, and 12 yesterday on OpenStack. And it works just fine. I'm in the process to upload all the qcow2, if anyone wants to give them a try.
<Goneri> it would be great if the branch could be merged. It has been here for a while, and it resolved several long standing problems.
<Goneri> the PR is here, https://github.com/canonical/cloud-init/pull/61
<meena> i have tested this on hetzner, and i'm quite  happy with what it does, that is: exactly what it promises to do: it configures the network and renames the device. IPv4 only so far, but it's a start
<wolfiepresley> Hello everyone, I have a few questions about a thing I am trying to do but due to my lack of knowledge I am struggling quite a bit. If you have 5-10 minutes it would help a lot maybe. Thank you
<smoser> wolfiepresley: well, try to write your question
<smoser> and if someone can answer,t hey will
<wolfiepresley> Sure :)
<wolfiepresley> Been using Amazon Linux inside an AWS Workspaces.
<wolfiepresley> create a bundle and assign that bundle to a new workspace.
<wolfiepresley> But the whole idea is that the user should not be prompted to enter the password when he executes the command. So I had the idea to edit /etc/sudoers but when the image/workspace is being created it cleans it.
<wolfiepresley> Wanted to add something like --  testuser ALL=NOPASSWD: /sbin/openvpn
<smoser> well, i dont knwo what is doing cleaning. but that is not somethign that cloud-init control.
<smoser> cloud-init's general goal is to not require cleaning
<smoser> you can absolutely use user-data to make cloud-init re-add the rules you want.
<wolfiepresley> I was thinking if  I can edit the cloud.cfg or cloud.cfg.d add a new file here where at the boot it adds that line to sudoers.d
<smoser> or add systemd jobs that did it, or some other way.
<smoser>  /var/lib/cloud/per-instance/your-script.sh
<smoser> chmod 755 that and it should get run once per instance.
<smoser> but if your cleaner deletes it, then there is nothing i can do.
<smoser> basically... you can't fight a cleaner that applies its own arbitrary rules.
<smoser> it could just as well decide to delete things in /etc/cloud/
<wolfiepresley> Here is some info from AWS Workspaces custom bundle documentation
<wolfiepresley> https://docs.aws.amazon.com/workspaces/latest/adminguide/create-custom-bundle.html
<wolfiepresley> Seems contents from /var/lib/cloud are removed :(
<smoser> :q
<smoser> well, file a bug. they should not delete /var/lib/cloud/
<wolfiepresley> Thing is this happens only when you create the image, if for e.g you have a workspace already built on top of that image/bundle it will not remove its contents.
<smoser> all the files in that list have valid reasons to keep them.
<smoser> sure.
<smoser>  /var/log/amazon/ssm
<smoser> funny my initials show up there.
<wolfiepresley> :D
<smoser> anyway... you can absolutely work around their custom bundle stuff. and do whatever you want.
<smoser> but they can later decide to update their custom bundler and screw you
<smoser> but i do consider it a bug to delete
<smoser> /var/lib/cloud
<smoser> because
<smoser> a.) cloud-init should manage that correctly
<smoser>  (and does)
<smoser> b.) there are interfaces with behaviors based on files in that path... so it shoudlnt get cleaned as that breaks cloud-init's promised interfaces.
<wolfiepresley> I am not experienced with this at all.. actually I started working on this last week and i've been reading through a lot of stuff but it's a whole mess in my head as you may have realised already. Appreciate you taking the time to write this.
<wolfiepresley> To be honest i thought if I edit the /etc/cloud/cloud.cfg
<wolfiepresley> openvpn without the password.
<Odd_Bloke> Looks like Travis is broken in master ATM because of tag issues.  I'm looking into it.
<Odd_Bloke> OK, I think I've addressed it.
<Odd_Bloke> I believe the 19.4 tag was pointing at a commit in the proposed branch, which changed when it was squashed.
<Odd_Bloke> And the tag needs to be signed, AFAICT, for read-version to be happy with it, so that took a couple of iterations too.
<powersj> Odd_Bloke, good to know - is that something we should doc?
<chillysurfer> why would cloud-init.service have only an After directive on systemd-networkd-wait-online.service? shouldn't that be a requirement?
<smoser> well, systemd-networkd-wait-online.service is obnoxious
<smoser> what cloud-init wants to wait for is "all configured networking to be up".
<chillysurfer> smoser: but we have no hard requirement for an online network then right?
<smoser> but what systemd-networkd-wait-online.service provides is
<smoser>  "you have network connectivity".
<chillysurfer> otherwise what is the mechanism that says "dont do cloud-init things until we are connected"
<chillysurfer> ?
<smoser> the difference is that if you intentionally have no network connectivity, or you have no network devices
<smoser> then you will not be online, but that is sufficient for cloud-init. it does not have a requirement that there be network.
<chillysurfer> smoser: ok so this is basically by design then, not to have a hard requirement on network connectivity?
<chillysurfer> ok
<chillysurfer> that answers my question
<chillysurfer> is it a weird thing and un-cloud-init-like to make that a requirement?
<chillysurfer> i don't see _why_ that would be a bad thing to modify, but curious on second opinions
<smoser> the use case that I think is completely valid is
<smoser>  - take ubuntu cloud image
<smoser>  - add a data disk
<smoser>  - add a NoCloud datasource
<smoser>  - userdata says to do a bunch of stuff (compile maybe) and put output on datadisk
<smoser> no network needed, nor desired.
<chillysurfer> yeah makes sense
<smoser> if you change that to be "you  must have network connectivity", then it doesnt work.
<chillysurfer> because at the moment, for instance, cloud-init does *not* require systemd-networkd.service to be up and running
<smoser> i really thikn that what cloud-init wants is what systemd-networkd-wait-online.service should provide
<chillysurfer> and i think i have a use-case where we absolutely need networking to be up and running before running cloud-init
<smoser> what does "online" even mean?  (AOL sound-bite has played ? )
<chillysurfer> smoser: able to make network requests..? i think
<smoser> on one network interface ? on all of them ?
<smoser> ipv4 ipv6 ?
<smoser> its not really straight forward
<chillysurfer> right i see where you're going with that
<chillysurfer> basically i'm working on an issue right now where systemd-networkd.service isn't up and running, and therefore we're not able to contact the cloud fabric. that's an issue (i don't know when that wouldn't be an issue for azure at least..?)
<smoser> i am remembering things kind of fuzzy....
<smoser> but what cloud-init *wants* is "All configured network interfaces are up."
<chillysurfer> smoser: where do you see that *wants* directive?
<smoser> oh. thats more of a goal.
<smoser> not a reall 'Wants'
<chillysurfer> but cloud-init will continue even if network interfaces aren't up, correct?
<smoser> you mean if they failed ?
<chillysurfer> no not even a failure. just never started
<chillysurfer> for instance, cloud-init will continue on it's way even if systemd-networkd.service isn't started
<smoser> cloud-init.service should not run until networking is up.
<chillysurfer> smoser: i don't see a directive in cloud-init.service that enforces that?
<smoser> well,. in ubuntu, cloud-init gets that to happen via netplan apply
<smoser> at the local time frame i think
<smoser> this is kind of fuzzy, and probasbly isnt perfect
<smoser> and i think that is why we have After=systemd-networkd-wait-online.service
<chillysurfer> smoser: what runs netplan apply?
<chillysurfer> smoser: yeah but After isn't a requirement, only timing
<smoser> cloudinit/net/netplan.py: will call netplan generate
<Orion]> hi I have a question about cloud-init
<smoser> sorry... i have to go. i think i've probably not been terribly helpful
<chillysurfer> smoser: you've definitely been very helpful! thanks for the assistance
<Orion]> I am using a VPS from a cloud provider on debian 10, and I found that the network configuration by default is /etc/network/interfaces.d/50-cloud-init.cfg
<Orion]> cloud init confuses me, I want to set up my networking to support multiple ips
<Orion]> is it correct to think cloud init is not relevant to me, and I can remove it's configuration file and create my own
<meena> Odd_Bloke: i approved, but that still does nothing in github's eyes: https://github.com/canonical/cloud-init/pull/120 also, your branch needs a rebase
<blackboxsw> meena: that UI shows a nice grey â
<blackboxsw> thanks for the review there. I've updated the PR and waiting on CI to land
<wolfiepresley> If I try to edit the cloud.cfg file and add under section -- users -- the following
<wolfiepresley>   - name: myuser
<wolfiepresley> system: false
<wolfiepresley> if some of those field are empty do they still need to be present?
<Odd_Bloke> powersj: Yes, we should either doc the tagging requirement and/or adjust tooling.
<meena> blackboxsw: it does, now at your review has landed
<Odd_Bloke> chillysurfer: AIUI, the only effect that A declaring a Requires on B has is that if A is included in boot, then B will be included as well.
<Odd_Bloke> If systemd-networkd-wait-online.service is relevant to cloud-init, then something else should already "Requires" it, so it's included in boot.
<Odd_Bloke> If nothing else in boot Requires systemd-networkd-wait-online.service then that _probably_ means that networkd isn't in use on the system, so cloud-init "Requires"ing it would be incorrect.
<Odd_Bloke> The Wants=systemd-networkd-wait-online.service is effectively saying "if systemd-networkd-wait-online.service is included in boot, then cloud-init should run after it", which I believe is correct.
<chillysurfer> Odd_Bloke: right exactly, but there is no Wants directive, it's `After=systemd-networkd-wait-online.service`
<chillysurfer> i would think adding in Wants=systemd-networkd-wait-online.service would be a possible fix to ensuring network is up and running for cloud-init
<Odd_Bloke> chillysurfer: Are you seeing that systemd-networkd-wait-online.service isn't running during boot?
<chillysurfer> Odd_Bloke: correct
<chillysurfer> Odd_Bloke: and i think making it a requirement might alleviate this issue
<chillysurfer> like adding it as a Wants
<Odd_Bloke> OK, right.
<Odd_Bloke> I'm surprised that it isn't already included.
<Odd_Bloke> chillysurfer: What release are you looking at?
<chillysurfer> 19.2-36-g059d049c-0ubuntu2~18.04.1
<chillysurfer> but smoser did bring up a good point. there are cases where a network requirement is a bad thing
<blackboxsw> Orion]: depending on the cloud platform, cloud-init does support configuring multiple ip addresses and/or  multiple nics. It all depends on the metadata datasource used. Orion] if you are adding more nics on a platform that doesn't support configuring multiple IPs you could add additional nic configuration in /etc/network/interfaces.d/50-your-additional.cfg  or you can either describe your own network config in
<blackboxsw> entirty to cloud-init (and cloud-init will write it out) or you can disable cloud-init network configuration in general by adding a network: {config:disabled} in a file somewhere in /etc/cloud/cloud.cfg.d/*.cfg per https://cloudinit.readthedocs.io/en/latest/topics/network-config.html
<blackboxsw> https://github.com/canonical/cloud-init/pull/120 landed thx meena
<chillysurfer> Odd_Bloke: i guess i could just add that Wants directive when i think it's necessary
<chillysurfer> maybe not a bad idea
<Odd_Bloke> chillysurfer: Can you pastebin `systemctl status systemd-networkd-wait-online`, please?  I'm seeing it run in a lxd container locally, so I'm confused as to why it isn't running.
<blackboxsw> minor branch to validate CLA signing and fail CI with a message about reading hacking doc if the PR author hasn't already signed the CLA.
<blackboxsw> https://github.com/canonical/cloud-init/pull/124
<chillysurfer> Odd_Bloke: i'll save the pastebin as it's easy to describe. it runs sometimes but takes a reboot for it to run
<chillysurfer> Odd_Bloke: here's what i'm seeing.... 1) first boot, cloud-init tries to provision with no networking as systemd-networkd is not running 2) something reboots the machine 3) networking comes up, but provisioning is already completed with errors
<chillysurfer> Odd_Bloke: so during the 1 phase above, networking just isn't up, including systemd-networkd-wait-online. and it's causing issues
<Odd_Bloke> systemd-networkd not running sounds like something is very broken, TBH.  I'm not sure that's a cloud-init issue.
<chillysurfer> Odd_Bloke: my thoughts to remedy this include making cloud-init have a hard requirement
<chillysurfer> Odd_Bloke: no doubt, i completely agree with you. this was simply a workaround to tell cloud-init that it shouldn't run if systemd-networkd.service isn't running
<chillysurfer> it's a workaround for a much larger issue, i agree
<Odd_Bloke> chillysurfer: I guess what I'm confused about is how you've ended up with a system with this behaviour.  Because I assume this isn't an issue in the default behaviour of bionic on Azure?
<chillysurfer> Odd_Bloke: correct. custom image
<rharper> chillysurfer: at generator time, we can't know if systemd-networkd is present; and it won't be running;  cloud-init local runs *before* networking on purpose;  I don't understand the exact scenario;  why is there no networking (missing nics?, not connected?) and how would cloud-init know that there isn't any networking available?
<chillysurfer> rharper: why is there no networking? great question. that's the root of the problem, i wanted to add a unit directive as a workaround to the problem in the interim
<ananke> I was wondering if any of you seed cloud-init configs with packer. the goal is to customize existing AMIs and automate their build process. I can't seem to find much in terms of examples of such workflow
<chillysurfer> rharper: for some reason, networking isn't up for the 2-3 minutes on first boot so provisioning has issues. and then _something_ reboots the vm (don't know what). and then when it comes back up, networking works but provisioning is already complete (with issues)
<rharper> chillysurfer: having a journal would be helpful
<rharper> chillysurfer: do you want cloud-init to run
<rharper> and without knowing why networking isn't available, it's not easy to drop-in some logic to check for such a state and have cloud-init skip running this boot
<chillysurfer> rharper: totally understand. i've combed through it pretty thoroughly. nothing that really jumps out as to why networking doesn't come up for some time
<rharper> chillysurfer: I'm happy to look at the journal if you can share
<chillysurfer> rharper: yep totally understand
<chillysurfer> rharper: i'll have to check with a few people before i could share it, thanks for the offer!
<rharper> sure, understood
<chillysurfer> rharper: thanks for your willingness to help! super appreciated
<rharper> https://github.com/coreos/docs/issues/519
<rharper> chillysurfer: I suggest adding this to the image; it dumps way more debugging for networkd
<chillysurfer> rharper: oh wow that's super helpful actually
<chillysurfer> thank you
<blackboxsw> rharper: https://github.com/canonical/cloud-init/pull/123 for ubuntu/devel so I can upload to focal today if possible
<blackboxsw> the 19.4 release
<blackboxsw> also Travis validation of CLA is passing CI in https://github.com/canonical/cloud-init/pull/124/files
<blackboxsw> thanks for the review on ubuntu/devel smoser. True on adding RbxCloud to templates
<blackboxsw> done
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/124 reviewed
<Odd_Bloke> 28 contributors from 29 domains?
<Odd_Bloke> powersj: rharper: blackboxsw: https://github.com/canonical/cloud-init/pull/125 is the first workflow automation piece; it will auto-close PRs after 21 days of inactivity (it first marks a PR as stale after 14 days).
<blackboxsw> nice!
<blackboxsw> addressed 124
<Odd_Bloke> powersj: blackboxsw: meena: I've responded to the comment that you either made or reacted on: https://github.com/canonical/cloud-init/pull/125#issuecomment-567190235
<meena> Odd_Bloke: okay, fair.
<blackboxsw> Odd_Bloke: responded with an inline question/comment. I'm +1 regardless of your stance
<Odd_Bloke> blackboxsw: Yes, I think we do, let me update.
<meena> rharper: would you like to take a look at this again? https://github.com/canonical/cloud-init/pull/53
<Odd_Bloke> blackboxsw: Updated.
<meena> blackboxsw / Odd_Bloke: would you like to take a look at thisâ¦
<meena> https://github.com/canonical/cloud-init/pull/69
<blackboxsw> approved Odd_Bloke
<blackboxsw> looking meena
 * blackboxsw will run through openstack once on that changeset meena, I'm +1 otherwise, but I don't think it'll actually trigger due to our current code structure in DatasourceOpenStack
<blackboxsw> it could possibly in Hetzner.
<meena> blackboxsw: it does; that's how i test Goneri's changeset :P
<blackboxsw> hrm, then I'll have to get through that live test to to re-educate myself about how that'd work across reboots (without cloud-init clean)
<blackboxsw> community notice: Upload to Ubuntu Focal (20.04) of cloud-init 19.4 complete  [ubuntu/focal-proposed] cloud-init 19.4-1-g8c96cbc1-0ubuntu1 (Accepted)
<Odd_Bloke> powersj: This PR brought to you by having to read the cc_ssh docs repeatedly over the past few days: https://github.com/canonical/cloud-init/pull/126 :p
<powersj> hahahaha!
<powersj> Odd_Bloke, was this from a grep on 'ssh'?
<Odd_Bloke> powersj: `ait -s '[^-_/.a-zA-Z]ssh[^-/_a-zA-Z]' cloudinit doc/rtd`
<Odd_Bloke> (Where `ait` is `ag --ignore tests`.)
<meena> i keep forgetting that hetzner is overriding my system_info/distro
<Odd_Bloke> meena: Have you attempted to escalate that with them?
<Goneri> I've rebuilt my FreeBSD images, they go from 10 to 12: e.g: https://virt-lightning.org/images/freebsd-11/disk.qcow2
<Odd_Bloke> https://www.youtube.com/watch?v=KOO5S4vxi0o
<Goneri> Odd_Bloke, you made me reconsider a lot of things in my life.
<meena> Odd_Bloke: nah, i have considered overriding vendor-data, however ;)
<meena> for one: FreeBSD doesn't need a rng seed; cuz it's got an implementation that doesn't suck
<meena> (sorry Linux folk)
<meena> However, i don't know how to do that!
 * meena blames powersj 
<powersj> lol
<meena> https://bugs.launchpad.net/cloud-init/+bug/1837530 not sure if my reading comprehension is enough to get anything out of this bug report
<ubot5> Ubuntu bug 1837530 in cloud-init "cloud-config in vendordata overriden by user-data" [Undecided,Expired]
<powersj> the merging doc needs to be updated, but I'll be honestly I don't understand it all enough to do the updates
<powersj> generally user data always wins versus vendor data is about all I can somewhat intelligently say
<meena> so, given my vendor-data: https://bugs.launchpad.net/cloud-init/+bug/1855170 maybe it's enough that I say: system_info: distro: freebsd ; and be done with it
<ubot5> Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,Incomplete]
<Odd_Bloke> meena: That's certainly worth a try.
<Odd_Bloke> If it doesn't work, then disabling all vendor data should do it.
<meena> Odd_Bloke: that's not documented EITHER!
<meena> oh my gosh, it's like a dream
<meena> runcmd: section seems to have been ignored tho :(
<Odd_Bloke> meena: `alias man=grep -R` <-- fixed the docs problem
<meena> Odd_Bloke: lookit, my personal wisdom is, read the source, but STILL.
<meena> so, almost midnight, time to give up.
<meena> Goneri: found a "bug"
<meena> Goneri: https://gist.github.com/8fb0a17238b4264cfa8a37093017c7d1 can you find it too? ;)
<meena> so, bed time.
<meena> I've commented on https://github.com/canonical/cloud-init/pull/61 with my experience report ;)
<meena> next up, finding out why runcmd doesn't workâ¦
<meena> or, sleep???
<meena> anyway, i'm now paying hetzner double the money for the privilege of developing cloud-init
<meena> (almost 6â¬)
<meena> whoa, y'all have useful reviews comments! (it's much easier to give those useful reviews when you know the codebase better i guess;)
<Goneri> meena, I suspect the ifconfig_vtnet0 line to come from the ephemeral dhcp call
<Goneri> meena, could you send me the cloud metadata?
#cloud-init 2019-12-19
<meena> Goneri: ifconfig_vtnet0=dhcp was already there before cloud-init touched it. I'm talking about the defaultrouter line!
<Goneri> meena, there is probably a name=eth0 in the metadata that force the renaming, so line 12 and 13 are ok
<Goneri> and so far, the provider does not clean up the previous lines from the /etc/rc.conf, should it?
<Goneri> ah nice defaultrouter
<meena> re cleanup: nah
 * meena sleep now
<Goneri> thanks you rharper and Odd_Bloke for the reviews, I think I've addressed all of them. I'm regenerating new images.
<meena> https://github.com/canonical/cloud-init/pull/61#discussion_r359656257 hell no.
<wyoung> o/
<meena> wyoung: Heya
<wyoung> Hey meena !  How's stuff and / or junk?
<mkrai__> Hi cloud-init team, is there any centos cloud image with cloud-init-18.5?
<mkrai__> meena, Hi
<meena> mkrai__: why so specific?
<mkrai__> meena, Hi, I am facing issue on centos 7.6 with cloud-init
<mkrai__> network is not configured
<mkrai__> the cloud-init configures the wrong interface
<mkrai__> meena, ^
<meena> mkrai__: which cloud? how exactly is it configured wrong?
<mkrai__> meena,
<mkrai__> meena OpenStack
<meena> mkrai__: what do the logs say?
<mkrai__> meena, I have 3 interfaces on the node, 2 ethernet(enps0f0 and enps0f1) and 1 HFI(ib0). Openstack tries to boot interface on enps0f1 but cloud-init does on enps0f0
<mkrai__> meena, http://paste.openstack.org/show/787707/
<meena> mkrai__: and paste cloud-init query --all
<meena> that was supposed to be expressed less urgent and more friendly, but, phone swipe
<mkrai__> meena, http://paste.openstack.org/show/787784/
<meena> mkrai__: as root or with sudo ;)
<meena> wait that is
<meena> weird
<mkrai__> meena, with sudo
<meena> mkrai__: yeah, sorry, what i mean is, the result is weird
<mkrai__> meena, yeah, that seems weird to me to ;)
<meena> mkrai__: can you curl the metadata URL?
<mkrai__> meena network is not configured so I can't
<meena> https://docs.openstack.org/nova/latest/admin/metadata-service.html
<meena> well, yeah, that makes sense
<meena> i just admit I've never setup a three interface computer with cloud-init, we'd usually only setup one interface, and do the rest with puppet
<ananke> I'm trying to wrap my head around how to leverage packer and cloud-init to automate creation of golden AMIs. there are some actions that need to take place while the given AMI is being baked, and there are some that will happen on each boot of an instance using that AMI
<ananke> with that in mind for the actions that need to take place on each boot, should I be placing cloud-init stuff in /var/lib/cloud/scripts/per-boot, or should I tack onto /etc/cloud/cloud.cfg.d/ ?
<ananke> my source will be various linux distros sanctioned by their vendors, such as ubuntu/centos/etc, so they already have cloud-init in their AMIs
<meena> ananke: unless Packer provides a metadata service, https://cloudinit.readthedocs.io/en/latest/topics/datasources.html you'd wanna stick with a script
<meena> ananke: cloud-init should only run on the first boot on your AMI, to fix the partition and fs sizes and setup network and hostname
<ananke> meena: packer utilizes ec2 metadata server to seed it the initial user data. these will be golden AMIs that will receive additional setup when they become instances
<meena> ananke: cool cool cool
<ananke> seeing as cloud-init is supposed to provide all of the above functionality, per-instance, per-boot, per-once, etc - I'm wondering what is the appropriate location I should be leveraging. it seems that these days cloud-init is baked into most major AMIs
<meena> I'm just a firm believer in idempotency, and puppet
<meena> what with being Co-founder of voxpupuli and all
<Odd_Bloke> ananke: The /var/lib/cloud/scripts/per-* paths sound like what you want, I think.
<Odd_Bloke> meena: cloud-init runs on every boot, but a lot of what it does only happens on first boot.
<otubo> Odd_Bloke: meena thanks for the reviews. I updated my branch with the fixes. Also, I force pushed a commit to remove an automatic non-ff merge from master (ugh, sorry), shouldn't be a problem, though.
<meena> Odd_Bloke: that does not change my preference towards systems that are made to be idempotent.
<Odd_Bloke> meena: Sure, but "cloud-init should only run on the first boot on your AMI" does not describe cloud-init's default behaviour, nor its behaviour in any distros I'm aware of. :p
<Odd_Bloke> (If you want to install Ruby on your systems, that's your business. ;)
<meena> Ruby is one of the best debuggable systems i know, so yes, i love working with it
<meena> hit me up, as soon as python has something close in comfort and power to pry and pry-byebug
<Odd_Bloke> meena: ipdb?
<meena> Odd_Bloke: how do i show the source code of a thing in ipdb?
<powersj> meena, if you are at a break point type 'l'
<Odd_Bloke> meena: `l` will show you your current source code context, is that what you meant?
<meena> Odd_Bloke: nah, more like show me the implementation of x
<Odd_Bloke> So in IPython you can do foo?? and it will display that.  `ipdb` gives me a SyntaxError if I try to do that, so maybe it isn't possible.
<Odd_Bloke> meena: Aha, `psource`.
<mkrai__> Odd_Bloke, portdirect smoser meena https://bugs.launchpad.net/cloud-init/+bug/1857031 whenever you've time. Thanks!
<ubot5> Ubuntu bug 1857031 in OpenStack Compute (nova) "cloud-init configures wrong interface when trying to configure two interfaces with OpenStack cloud" [Undecided,New]
<blackboxsw> mkrai__: can you please run "cloud-init query --all" on that machine and attach the content to the bug?  I'd particularly like to see the contenst of network_json that is reported there
<blackboxsw> bah missed it, I'll add it to the bug
<smoser> he's gone
<blackboxsw> yeah followup up on the bug :/
<smoser> look for WARN
<smoser> thats the issue
<smoser> centos 7 .
<blackboxsw> ahh thanks, just did a quick driveby and hadn't caught up on backlog
<blackboxsw> and looks like the submitter also attached a paste of network data .json already to the bug anyway too
<smoser> its a dupe of https://bugs.launchpad.net/cloud-init/+bug/1801364
<ubot5> Ubuntu bug 1801364 in cloud-init "persisting OpenStack metadata fails" [Undecided,Fix committed]
<blackboxsw> ahh right, yeah
<blackboxsw> you nailed that symptom/fix, but that shouldn't have affected which network device got chosen I would think. but looking a bit more into it
<smoser> well,  it fails during local
<smoser> so it just tries the fallback/dhcp
<blackboxsw> and interesting is that the 'fix' was released in 18.5 I think and logs are from 19.2 cloud-init
<blackboxsw> ohhh 18.2
<blackboxsw> not 19.2
<blackboxsw> my mind was playing tricks on me
<smoser> i'm not certain ther are not other issues at play
<smoser> there may well be.
<smoser> but if it doesn't persist the data then we're down a sour path
<ananke> Odd_Bloke: thanks!
<blackboxsw> thx for the triage on that bug smoser. yes on the instance-data.json being a sour path. Yet, the perist_instance_data happens after _get_data was already run successfully or otherwise. So, I *think* we just don't get the instance-data.json in that case without any other fallout.   In either event, 18.2 is missing a bunch of fixes over the last year and a half for both network, so I just pointed them at
<blackboxsw> copr/el-testing and asked for a reproducer
<ananke> oops, sorry
<smoser> blackboxsw: yeah, for a lot of reasons copr repo is a good iea there.
<meena> i approve: https://github.com/canonical/cloud-init/pull/132
<meena> rharper: added your test: https://github.com/canonical/cloud-init/pull/53
<Odd_Bloke> meena: rharper is done for the year, FYI.
<Odd_Bloke> powersj: blackboxsw: https://github.com/canonical/cloud-init/pull/130 <-- would be good to land this clarification to HACKING
<meena> Odd_Bloke: right, you mentioned that yesterday.
<meena> Odd_Bloke: you should also add how to do stuff at lp, cuz lotsa people find it confusing.
<Odd_Bloke> meena: "do stuff at lp"?
<meena> Odd_Bloke: how to fork stuff @ launchpad
<meena> and how to push there, cuz i've seen at least 2 found that challenging.
<Odd_Bloke> meena: Oh, for the CLA stuff?
<Odd_Bloke> Yeah, that would be good.
<Odd_Bloke> I was on vacation when the migration happened, so I don't really know how much the script handles etc.
<meena> i haven't looked at the script lately, tbf. so maybe it's magically fixed by now
<Odd_Bloke> meena: So regarding Cirrus CI, we'd be happy to introduce it as a non-voting status check provided it wouldn't cost us anything (which, looking at the docs, it shouldn't).  So if you want to go take a look at making that happen, please feel free!
<blackboxsw> #130 is approved. I'll run through the migration tool again now in light of upstream now being in github to see if it works
<Odd_Bloke> meena: (For reference, when I was iterating on our Travis config, I just configured it against my fork, so hopefully you shouldn't need anything from us to get something working.)
<meena> Odd_Bloke: i've configured cirrus before: https://github.com/bsdci/libioc/blob/master/.cirrus.yml
<Odd_Bloke> Nice!
<Goneri> rharper, Odd_Bloke: I've addressed all your comments https://github.com/canonical/cloud-init/pull/61 and re-tested with FreeBSD 10, 11 and 12. Bonus point, I've pushed two extra fixes for minor issues.
<Goneri> I'm totally confident the branch can be merged.
<meena> I'll be testing it in a minute.
<Odd_Bloke> Goneri: Nice!
<Goneri> I'm uploading the refresh qcow2 images
<Odd_Bloke> Goneri: Looks like a pyflakes failure on #61
<Goneri> naahh, it's impossible!
 * Goneri takes a look
<Odd_Bloke> Goneri: Also it looks like you haven't migrated your CLA signature over to GitHub yet.
<Odd_Bloke> Goneri: https://cloudinit.readthedocs.io/en/latest/topics/hacking.html tells you how to do that (TL;DR: use tools/migrate-lp-user-to-github)
<Goneri> Odd_Bloke, https://github.com/canonical/cloud-init/pull/133
<Odd_Bloke> Thanks!
<meena> Goneri: i got a stacktrace.
<meena> Goneri: ah, yeah, makes sense. https://gist.github.com/c4201008d52bc1bab7bd9cc81d217956
<Goneri> meena, lets me guess IPV6 with CIDR notation?
<meena> absolutely.
<meena> lemme show you ;)
<Odd_Bloke> If we know IPv6 isn't supported yet, is that a blocker?
<Goneri> not really, I will just add a skip if netmask is null
<Odd_Bloke> OK, cool.
<meena> https://gist.github.com/92db11324aecb2ee1eb376745339ed65
<meena> Goneri: you should skip everything that's None ;)
<Goneri> https://imgflip.com/i/3jzpiv
<meena> "review" https://github.com/canonical/cloud-init/pull/61#pullrequestreview-334933456
<Goneri> meena, https://github.com/goneri/cloud-init/commit/3e6a1199031d9477ddd8d33234023f942a2ba79f
<blackboxsw> Goneri: per https://github.com/canonical/cloud-init/pull/133  I'm missing a correlating launchpad branch for you. and I expected the PR message was going to be something other than goneri as the username
<Odd_Bloke> blackboxsw: I closed out the MP.
<Odd_Bloke> blackboxsw: And what do you mean by "something other than goneri as the username"?
<Goneri> I'm confused too :-X
<blackboxsw> https://github.com/canonical/cloud-init/pull/133/files
<blackboxsw> that mp is open
<Odd_Bloke> That's a PR.
<Odd_Bloke> I closed the MP.
<blackboxsw> oops
<blackboxsw> ahhh
<blackboxsw> ok nevermind, you were on it Odd_Bloke
<Odd_Bloke> :)
<blackboxsw> I was expecting to see a branch in LP authored by
<blackboxsw> GonÃ©ri Le Bouder
<blackboxsw> which I didn't realize mapped to goneri :)
<Odd_Bloke> blackboxsw: https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/377008 :)
<blackboxsw> and since I saw no branch over there I thought maybe it was an accidental wrong username supplied
<meena> Goneri: IPv4 could also be in CIDR notation.
<blackboxsw> +1 Odd_Bloke Goneri ... don't mind me
<meena> or? do we need to back-calculate it then?
<Goneri> meena, the existing code base does not support IPv4 CIDR notation, so it's not a regression.
<Goneri> and it's a corner case anyway, I doubt it happens often :-)
<meena> does any of the other renderers handle it?
<Odd_Bloke> If it's not supported on FreeBSD in the current codebase, I think we can land this PR without it.
<Odd_Bloke> meena: Do you disagree?
<meena> Odd_Bloke: aye, makes sense given that nothing much is supported in the current code base.
<Odd_Bloke> Cool. :)
<Odd_Bloke> meena: Goneri: So are we good to merge #61 (after an up-to-date CI run, of course)?  Or do you want to do another round of testing?
<meena> i'll do another reboot and see what it does :O
<meena> also, can someone tell me why cloud-init would refuse to run my runcmd section?
<Goneri> I'm confident we won't see a horde of FreeBSD users coming on this channel in the coming days
<Goneri> I've to go, I will be back later today.
<blackboxsw> either, already ran on that instance (not per boot),  bad yaml, or disabled runcmd in /etc/cloud/cloud.cfg(.d/*cfg) ?
<meena> blackboxsw: could be bad yamlâ¦
<meena> blackboxsw: https://gist.github.com/90f3ea89f4339d2297a9f5a356e04b8d
<blackboxsw> meena: try this:     sudo cloud-init query userdata > myud.yaml; cloud-init devel schema -c myud.yaml --annotate
<blackboxsw> it'll quickly vet your yaml at least
<meena> aye
<blackboxsw> runcmd.   <- the dot
<blackboxsw> hrm nope ... sorry : it was offscreen for me on the laptop
<meena> blackboxsw: i'm thinking it's - # these comments probably don't work
<meena> yupp. that's what it is.
<meena> Cloud config schema errors: runcmd.2: None is not valid under any of the given schemas, runcmd.5: None is not valid under any of the given schemas, runcmd.8: None is not valid under any of the given schemas
<meena> lines 2, 5, and 8 are comments
<blackboxsw> ooh interesting. hrm. that schema is defined so validation was attempted.
<Odd_Bloke> Lines 2, 5 and 8 are empty list items followed by comments.
<Odd_Bloke> If the comment marker was before the -, everything would be fine.
<blackboxsw> I see, word wrap was making that hard to read for me
<meena> so the damage is self-inflicted.
<blackboxsw> ahh and meena I'd also expect to see something like 2019-12-19 21:43:05,262 - util.py[WARNING]: Failed to shellify .... into /var/lib/cloud/instances/loved-seal/scripts/runcmd
<meena> let's see.
<blackboxsw> in your /var/log/cloud-init.log
<meena> i'm pretty, and sure, i've posted the logs; but let's try again.
<blackboxsw> it would be nice if we bubbled up warnings to cloud-init status --long
<meena> blackboxsw: also, i'd expect that to be a slightly higher error level, so you know you fucked.
<blackboxsw> yeah fairly. though cloud-init walks a line and tries not to be fatal if some components are not run. runcmd may not be critical to the operation of the node. but it certainly is a case where the user should be alerted
<meena> blackboxsw: maybe it'd be cool if *i* could decide if runcmd is critical or notâ¦
<blackboxsw> meena: heh truly
<meena> yeah, so cloud-init status only gives us errors: https://gist.github.com/126df102d581a75572fff2c74eb69b0f
<blackboxsw> yeah meena, it only bubbles up errors, it doesn't queue and raise warnings
<blackboxsw> and I think it may only bubble up one error (not all of them)
<blackboxsw> so there is improvement to be had there
<blackboxsw> it's worth a bug on runcmd not being rendered, at least we could raise the warning to error
<blackboxsw> and it's also a place that is a source of many many human-introduced errors
<meena> i don't have a WARNING failed to shellify blah, cuz it's failing earlier. (i reckon)
<meena> I'll post the logs maybe you can read them easier.
<blackboxsw> thanks
<meena> https://gist.github.com/6cfa360eb13d2c2396f6a4cea63c912d
<meena> https://gist.github.com/c6c5eb792918e252b5b21cda3b5c9c25
<Odd_Bloke> -ENOGONERI
<Odd_Bloke> meena: I'm heading out now, but I'll check back in later because I'd love to get #61 landed.  If Goneri returns, could the two of you decide if we're good to go and ping me to let me know?
<meena> Odd_Bloke: ok, cool â thanks for all the help
<smoser> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1801364 is fix-released ?
<ubot5> Ubuntu bug 1801364 in cloud-init "persisting OpenStack metadata fails" [Undecided,Fix committed]
<blackboxsw> smoser: should be as of 19.2-53-g067516d7b
<blackboxsw> which is Xenial ++
<smoser> so that bug should be Fix Relased
<smoser> though
<smoser> right?
<smoser>  not Committed ?
<smoser> somehow it got missed
<blackboxsw> smoser: ahh right, I think this could be due to the shift to github. yes. I'll scrub the bug list that's been released and confir,
<blackboxsw> confirm even
<meena> i'm happy with Goneri's patch. but I'll add another patch to beautify the code a bit.
<blackboxsw> ok found 44 bugs released in 19.2 or later. only 17 or which were fix released.
<blackboxsw> I'm running a script now to close them out as fix released with the appropriate comment about the expected cloud-init version that released them
<blackboxsw> ok script complete, if anyone notices bugs in fix committed state that should be fix released please let us know. I'm adapting our release and publishing scripts to try to account for this gap.
<blackboxsw> thanks for the pretty red and green label colors OddBloke. I'm getting into the Christmas Spirit
<blackboxsw> otubo: if you get a chance in the next few days please run:     ./tools/migrate-lp-user-to-github otubo otubo       # to correlate your launchpad user to github user accounts
<meena> opened a fresh PR for Goneri: https://github.com/goneri/cloud-init/pull/2
<meena> Goneri: https://github.com/goneri/cloud-init/pull/2 some Ã¦sthetics.
 * meena â bed.
#cloud-init 2019-12-20
<blackboxsw> and drop the super deprecated modules part 2 https://github.com/canonical/cloud-init/pull/134
<blackboxsw> ok EOD for me
<mkrai> blackboxsw, Hi I get error running clour-init query all command http://paste.openstack.org/show/787808/
<otubo> I have to work in such a different timezone, any of the maintainers here can help with the migrate-lp-user-to-github script?
<otubo> smoser: blackboxsw perhaps are already up :
<otubo> *I hate to work, I can't even blame the autocorrect, it's my mind that want's to leave on vacation already :-D
<otubo> Ha! No worries smoser blackboxsw, I just figured out the problem I was having. That's my launchpad-github mapping https://github.com/canonical/cloud-init/pull/135
<otubo> Also, I run the script with --dryrun enabled but it ended up doing the whole proccess. Perhaps there's a bug there. :-)
<Odd_Bloke> blackboxsw: I'm seeing a bunch of automated "This bug is believed to be fixed in cloud-init in version 19.2-67" bug mail from you, with the last number changing in (what looks like) each one.  Is that intentional?
<Odd_Bloke> Oh, I see the PR now.
<smoser> Odd_Bloke: i just did https://github.com/canonical/cloud-init/pull/135
<smoser> approved it, but if you can land it.. i'm not sure (i need to figure out) what the landing process is.
<otubo> smoser: Odd_Bloke rharper blackboxsw guys, I'm leaving for the shutdown now. Thanks for the good work this year. See ya in the next decade. :-)
<Odd_Bloke> otubo: Thanks, have a good break! :)
<Odd_Bloke> smoser: I can't see o'tubo in our CLA spreadsheet, so I'm checking internally if we have a non-individual signature that covers them.
<danmux> Hi, is it ok to ask for help on using cloud-init in here, it's not just reserved for development?
<Odd_Bloke> danmux: Yep, please do!
<danmux> ace! specifically multipart user-data and jinja2
<danmux> Should I be using `Content-Type: text/jinja2` in place of `text/cloud-config` ?
<danmux> (also can i confirm `cloud-init devel render` does not handle muilti-part files, just the raw template bit)
<danmux> in which case how best can I test my multipart `user-data.txt`
<Odd_Bloke> Hmm, good questions.
<Odd_Bloke> I haven't really dealt with multipart user-data much myself.
<Odd_Bloke> smoser: blackboxsw: Perhaps one of you have context to answer (some of) the above without digging into the code too much?
<danmux> I've been digging around the code a bit, and it looks to me like it is either or thing but I'm not 100%
<danmux> I can't tell if it would go on and honour the `template` directive - for example...
<danmux> ```Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"MIME-Version: 1.0--MIMEBOUNDARYContent-Transfer-Encoding: 7bitContent-Type: text/cloud-configMime-Version: 1.0## template: jinja#cloud-configwrite_files:  - path: /usr/local/etc/nomad/client.hcl```
<danmux> (oops)
<danmux> for info `/usr/bin/cloud-init 19.2-36-g059d049c-0ubuntu2~18.04.1`
<blackboxsw> danmux: it's been a while since I've touched that part of the code. will run a test. I think you might have had to pass text/jinja2 as content type. But, will validate and get you a working example
<danmux> blackboxsw hang fire a sec - im able to test now
<blackboxsw> sweet
<danmux> hmm ok it failed
<danmux> oh should i be editing the `.i` file ?
<danmux> blackboxsw - ok I was running a single module again, but i think the templating is done in `init`?
<danmux> ah - im doing something dumb
<blackboxsw> danmux, here's what works for me
<blackboxsw> ... sorry getting a paste together
<danmux> (y)
<blackboxsw> danmux: at long last https://pastebin.ubuntu.com/p/GzbR23WjQg/
<danmux> tyvm - you couldn't paste the `mime-msg` as well blackboxsw :pray:
<blackboxsw> to retest on your system that had already booted cloud-init. you can run the following to clean you system, force a reboot and re-run cloud-init with :     sudo cloud-init clean --logs --reboot
<blackboxsw> ahh will do now danmux
<blackboxsw> danmux: https://pastebin.ubuntu.com/p/XKwSf7x4bX/
<danmux> great ty for the test advice - otherwise I should just setup a container as per your example.
<blackboxsw> the tool in cloud-init ./tools/make-mime.py -a <your_yaml>:<mime-type>
<blackboxsw> will generate that content for you
<blackboxsw> if you have cloud-init source tree
<blackboxsw> https://github.com/canonical/cloud-init/blob/master/tools/make-mime.py
<danmux> yup - got it - ours is being generated by terraform. But with your info I'm sure i can make some progress!
<blackboxsw> good luck :)
<blackboxsw> hope it helps
<danmux> Woohooo! blackboxsw that nailed it! tyvm - no I can sign off for Christmas (UK timezone) - Merry Christmas to you and to Odd_Bloke for the referral (y)
<blackboxsw> Schweet! danmux
<danmux> s/no/now
<blackboxsw> good to hear
<meena> ah, the joy of freelancing. no holidays, no vacations, no time off, no nothing
<meena> (i think i'm mostly joking, but i can't really tell)
<meena> https://github.com/canonical/cloud-init/pull/61#pullrequestreview-335429532 merge?
<blackboxsw> hah!
<blackboxsw> powersj: minor suggestion on your doc branch and I think we can land it https://github.com/canonical/cloud-init/pull/109#pullrequestreview-335437127
<powersj> blackboxsw, fixed
<blackboxsw> meena: I'll let Odd_Bloke close out on that review
<blackboxsw> approved waiting on CI
<Odd_Bloke> meena: Goneri: It's landed!
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/136
<meena> cool, cool, cool
<meena> now, it's time for some refactoring
<blackboxsw> Odd_Bloke: on it for 136. and powers I replied to your doc https://github.com/canonical/cloud-init/pull/104/files
<blackboxsw> ooh meena, putting the fun back into refactoring :)
<meena> "can't spell refactoring without fun" â blackboxsw
<blackboxsw> :)   -> lunch -> coffee
<blackboxsw> or -> coffee -> lunch .... can't decide
<meena> i find lunch - > coffee to be slightly more conducive to having a proper post lunch sleep and then wake up refreshed
<Odd_Bloke> "refunctoring" means "rewriting in Haskell", right?
<meena> I'm partial to any functional language i can write, haskell i can only read
<Odd_Bloke> powersj: blackboxsw: Let's kick off the holidays (and the decade) the right way: https://github.com/canonical/cloud-init/pull/137
<blackboxsw> woot!
<blackboxsw> Odd_Bloke: in the theme of dropping all the dead code:  https://github.com/canonical/cloud-init/pull/134
<blackboxsw> your above is +2'd just waiting on CI
<Odd_Bloke> blackboxsw: The PR I've already Approved, you mean? ;)
<blackboxsw> hah
<blackboxsw> ahh I'll wait for your py.27 to land and I'll close out 134
<Odd_Bloke> Thanks for the speedy reviews!
<blackboxsw> unfractured attention makes for light work :)
<Odd_Bloke> Landed. \o/
<blackboxsw> Odd_Bloke: can you confirm breakage of make config/cloud.cfg
<blackboxsw> or powersj
<blackboxsw> on focal it doesn't work for me as it tries to interpret $user from config/cloud.cfg.tmpl line 17
<blackboxsw> as a required substitution param
<Odd_Bloke> blackboxsw: Breakage of what exactly?
<blackboxsw>     return str(selected_params[key])
<blackboxsw> KeyError: 'user'
<blackboxsw> make: *** [Makefile:79: config/cloud.cfg] Error 1
<blackboxsw> no emitted 'rendered' cloud-config
<blackboxsw> I'm unable to make that target (to generate cloud.cfg from template)
<blackboxsw> and if I do provide a bogus user: ME to the templater.render_string(...., tpl_params)  then I get a rendered output file that subtitutes ME for $user on line 17 of  config/cloud.cfg.tmpl
<blackboxsw> it's tools/render-cloudcfg is called by setup.py during package creation. I'm wondering why that hasn't broken our package builds too
<blackboxsw> interesting. this works on xenial: ./tools/render-cloudcfg --variant ubuntu config/cloud.cfg.tmpl
<blackboxsw> not on focal
<Odd_Bloke> I have never seen these Make targets before. :p
<blackboxsw> yeah I hadn't until I started fgreping.   so something else in the template engine is getting hung up on the $user in the config template file
<blackboxsw> I think it may be focal only
<Odd_Bloke> blackboxsw: What do you want me to run to reproduce?
<blackboxsw> checking eoan /bionic I might have a trivial patch
<blackboxsw> Odd_Bloke   ./tools/render-cloudcfg --variant ubuntu config/cloud.cfg.tmpl
<Odd_Bloke> And what are these used for?  (Should we remove them?)
<blackboxsw> should emit output to stdout with the rendered template
<Odd_Bloke> It does.
<blackboxsw> Odd_Bloke: they are used in setup.py to build the package
<blackboxsw> but we probably can drop the makefile target
<powersj> blackboxsw, if these were broken, how did you upload to focal?
<Odd_Bloke> OK, so we probably shouldn't remove them. :p
<blackboxsw> powersj: I think something in the python environment used though dh_python is succeeding there whereas direct script running through python3 does not.
<powersj> ah
<Odd_Bloke> blackboxsw: I can't reproduce a problem on focal.
<blackboxsw> ok so something from where I sit
<blackboxsw> hrm
<blackboxsw> ahh I'm on eoan :/
<blackboxsw> checking eoan and focal containers
<blackboxsw> and upgrading my laptop
<blackboxsw> ok success on focal
<blackboxsw> ok; dirty local dev environment. Fresh eoan no problems. something I've inevitably pip installed that is colliding w/ the renders
<blackboxsw> Odd_Bloke: powersj n/m thanks for validation
<powersj> :D
 * blackboxsw should work in a clean dev container to avoid thrashing 
<meena> someone post an image link or a github thingy pls
<blackboxsw> https://imgflip.com/i/3k3kft
<meena> excellent, previews are working.
<meena> now, to get back to refunctoring.
<meena> whoa
<meena> Python2.7 is dead?!
<Goneri> <Odd_Bloke> meena: Goneri: It's landed! <3
<Odd_Bloke> meena: Yep! \o/
<meena> Odd_Bloke: can i get some starting help with the refactor?
<blackboxsw> long live python 3
<meena> just to get the class(es) started, i can do the rest myself.
<meena> blackboxsw: ETA for Python 4? :P
<Odd_Bloke> meena: This is wrong point in the decade to ask for help, I'm afraid. ;)
<Goneri> Odd_Bloke, thank you, now I will refresh the netbsd branch :-)
<meena> Odd_Bloke: in that case, i'll just repeat all of Distros mistake, and you get to fix them next decade.
<Odd_Bloke> meena: I would experiment with moving one of the two "*_on_{freebsd,linux}" functions onto either the Distro or Renderer classes.
<Odd_Bloke> Which one they go on will probably depend on what makes sense in the structure of the code.
<Goneri> I think we need a distros/linux.py and some inheritance
<Goneri> and a lot of linux specific content should be move from net/__init__.py to this new distros/linux.py
<meena> Goneri: that's what distros/__init__.py is.
<Goneri> distros/__init__.py is linux specific, it should be agnostic
<Odd_Bloke> On the Distro would seem more natural to me, as the functions aren't to do with the rendering of network stuff.
<Odd_Bloke> I would suggest that refactoring them onto an existing class would make sense.
<Odd_Bloke> Refactoring to have a Linux Distro and an agnostic, shared super-class is something I would support, but we would need to discuss that as a project.
<Goneri> a large part of net/__init__.py is actually just some Linux specific code.
<Goneri> Odd_Bloke, yes maybe, it's indeed already a good step in the right direction.
<Odd_Bloke> As a reminder, the intent of the exercise is to gather data about what refactoring looks like it would make sense.
<Odd_Bloke> So we don't need it to be Done (TM).
<Odd_Bloke> We just need to be able to see roughly what it would look like, and modifying the existing distros.Distro to add functionality and overriding it in freebsd.Distro would do that, I think.
<meena> Good point.
<meena> points.
<Goneri> IMO, a good metric is the number of is_FreeBSD in the code base
<Goneri> the less, the better
<Odd_Bloke> Agreed.
<meena> Odd_Bloke: how do we access the distro from cloudinit/net?
<Odd_Bloke> meena: That will be one thing that will need to be worked out.
<meena> Odd_Bloke: >_>
<meena> okay.
<Odd_Bloke> (=
<meena> i wonder if we can do that with decoratorsâ¦ maybe. probably not.
<meena> maybe the easiest way would be to
<meena> hmmâ¦ a class?
 * meena gives up for tonight o/~
#cloud-init 2019-12-21
<Nick_A> I have "package-update-upgrade-install" under cloud_final_modules as the default in our Ubuntu 18 cloud image. What would be the best way to prevent that module from running with cloud config user data rather than modifying the image itself?
#cloud-init 2019-12-22
<joshualyle> I'm trying to find a way to write an openbsd module for cloud-init and I started looking at util.py in get_linux_distro since that's where it seems to recognize FreeBSD. However, regardless of what get_linux_distro() returns, it seems to default to use the ubuntu Distro class regardless. I can only get it to change the Distro class if it's directly set in cloud.cfg. What is the purpose of get_linux_dist
<joshualyle> ro then?
<meena> joshualyle: is your cloud.cfg actually saying system_info: distro: Ubuntu?
<meena> if not, it could also be the fallthru, default
<meena> which makes me wonder, did Goneri adapt that for his NetBSD branch?
<meena> ah, yes, he simplified it: https://github.com/canonical/cloud-init/pull/62/files
<Goneri> my experimental NetBSD works fine, I now need to clean up the patch, and refresh the unit-test
<Goneri> :-)
<joshualyle> meena: It attempts to use ubuntu regardless of if I include system_info: distro: Ubuntu in cloud.cfg or not. I was originally using a cloud.cfg that only had information for users without system_info: defined anywhere but it always identified (or fell back to) ubuntu
<joshualyle> explicitly setting system_info: distro: openbsd was the only way I could get it to try to use an openbsd Distro class
<joshualyle> I guess what I'm not getting is, how does cloud-init determine the class to use if it's not explicitly defined in cloud.cfg
<meena> joshualyle: iirc, Ubuntu is default
<joshualyle> meena: but then do you have any idea why there is so much code to discover the OS if it essentially just relies on the config file?
<meena> the config file can override the detection, in some cases. the point is that your os doesn't have a working get_linux_distro() implemented
<meena> Although, Goneri's implementation should work for all BSDs
<joshualyle> I hope I'm not being annoying at this point but I guess what I still don't understand is when the detection takes effect. If the distro is defined in cloud.cfg, it is used to determine the distro class. If it is not, it seems to use settings.py to fallback to ubuntu
<meena> joshualyle: nah, you're not annoying, it's just late here, and most of the maintainers are off until next year
<meena> aaaaand, while i have dabbled in that code, i haven't added a whole new distro / os
<joshualyle> alrighty I gotcha. Well I appreciate your help. I'll keep digging!
<meena> i would just take Goneri's implementation of get_linux_distro() and start from there
<meena> Also, his NetBSD branch might give you other ideas!
<meena> also, https://hackmd.io/3-YBj1t9TAeKhmfLBQUjXQ
<meena> (i'm @igalic on github, btw)
<Goneri> joshualyle, I would like to focus on OpenBSD once NetBSD will be merged. But first, I need to figure out how to build a disk image.
<Goneri> something like https://github.com/virt-lightning/netbsd-cloud-images
<Goneri> this is the first step
<Goneri> my last NetBSD qcow2 snapshot: https://virt-lightning.org/images/netbsd-8/disk.qcow2
<Goneri> works fine with VirtLightning and OpenStack
<joshualyle> meena: thank you! Once I figure out the structure of cloud-init in master, I'll probably branch off of Goneri's netbsd branch to see if I can figure it out for openbsd
<joshualyle> Goneri: what exactly does that build script do? Install netbsd, add cloudinit, and reboot?
<Goneri> it prepares an image disk, and indeed, install cloud-init after.
<Goneri> joshualyle, to be honest, I've never used OpenBSD, so you will probably be faster than me :-)
<Goneri> so if you've something functional, I will be happy to help.
<Goneri> in general, the hardest part to do right, is the filesystem/partition resizing.
<joshualyle> I appreciate it! I haven't contributed to OSS before but I've done a lot in cloud work and was sad to see the state of adoption of BSD in cloud environments so I wanted to see if I could help
<meena> 22:09 <Goneri> in general, the hardest part to do right, is the filesystem/partition resizing.  â¬ï¸ cuz you're not using zfs
<meena> I'm starting to think that a decorator is our best chance
<Goneri> meena, there is no ZFS on OpenBSD and NetBSD, even if things are change with NetBSD-9
