#cloud-init 2014-03-10
<smoser> harmw, i really apprciate your help on cirros.
<harmw> but... :>
<smoser> no but. i'll look to get your cirros-status update in.
<smoser> and after 0.3.2 we can branch for 0.4.0
<smoser> and that can have *some* more packages (i didn't look at your list entirely, but remember size matters)
<harmw> cool
<smoser> and i want a 3.13 kernel and 'mdev' or udev (some solution to that better than what is there now)
<harmw> yea totally, I was just highly annoyed by a missing tcpdump/tshark
<harmw> I got some trouble while adding lsblk/lscpu back in (?) so I committed those package changes perhaps a bit to careless
<harmw> could you look into some of the bugs as well? I've replied on 2 earlier today, perhaps you can share your opinion as well
<harmw> harlowja: 
<harmw> 15:16:42 < smoser> harmw, i really apprciate your help on cirros.
<harmw> bet he never said that to you :>
<harlowja> lol
<harlowja> ya, i haven't messed around with cirros ;)
<harmw> ;)
<harmw> you've talked with sean about ci in ports?
<harlowja> yup
<harlowja> where'd he go again
<harlowja> lol
<harlowja> doesn't seem online in YIM
<harlowja> hmmm
<harlowja> will see where's he's at once i find him, lol
<harmw> lol
<harmw> ok
<smoser> i make sure to only say bad things about harlowja 
<harlowja> lol
<smoser> if you complement him, he lets it go to his head.
<harlowja> def
<harmw> hhe
<harmw> smoser: http://packages.ubuntu.com/trusty/kernel/linux-image-3.13.0-16-generic 
<harmw> should I go after that kernel, or is there a magic url somewhere else?
<smoser> harmw, the README shows how to download the kernels an dsuch.
<smoser> but yeah, basically i grab the latest '-updates' kernel at that point.
<harmw> ah, README
<harmw> the one file I always manage to purge at once
<smoser> well, sort of. 
<smoser>  https://launchpad.net/ubuntu/+source/linux
<smoser> it will probably ahve to change some for trusty naming
<harmw> ah now I did chck that url in README 
<harmw> though I was kinda clueless on what it should be for an updated kernel :)
<harmw> https://launchpad.net/ubuntu/+source/linux/3.13.0-16.36/+build/5659610/+files/linux-image-3.13.0-16-generic_3.13.0-16.36_amd64.deb
<harmw> something like that?
<harmw> (I'm a true ubuntu adept, hurray)
<smoser> harmw, heres the translations for trusty anmees
<smoser> http://paste.ubuntu.com/7069858/
<harmw> wicked
<harmw> well, that seemed easy
<harmw> im running 3.13 now, without additional rebuilding of the buildroot
#cloud-init 2014-03-11
<harlowja> smoser yt, are u ok with https://code.launchpad.net/~vlastimil-holer/cloud-init/net-reconfigure/+merge/186352 ?
<smoser> looking
<smoser> i shouldn't be here.
<harlowja> lol
<harlowja> np
<smoser> what do i get wit this ?
 * harlowja not sure whose been testing snapshots at y!
<harlowja> lol
<harlowja> but the ips haven't been right, lol
<harlowja> so smoser  i think what u'll get is the ifdown part before networking modification
<harlowja> another idea, i think http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/sysvinit/redhat/cloud-init-local#L30 would probably like to have Should-Stop: $network
<harlowja> although not sure
<harlowja> i think the issue we have just noticed (don't ask why nobody noticed or told me before) is that snapshot saves image, with old netconfig, starts up off of snapshot, but init-local happens after networking starts (so config drive network writing is to late at this point)
<harlowja> anyways
<harlowja> let me know what u think when u get some time
<harlowja> the /etc/rc.d/rc5.d/ that this is part of http://paste.openstack.org/show/73103/
<smoser> harlowja, yeah, i knew it was broken in that way.
<smoser> well, i knew it was broken for inserting statick eth0
<smoser> i hadn't thought about other nics that were previosly configured.
<smoser> it seems a bit broken to bring dwn configured network interfaces though.
<harlowja> agreed
<harlowja> what to do i wonder :-P
<harlowja> b b
<harmw> and another day goes by without new cirros release :p
<kwadronaut> be nice ;-)
<harmw> I'm always nice :>
<kwadronaut> (-:
<smoser> harmw, http://paste.ubuntu.com/7075003/
<smoser> can you test that ?
<smoser> adds and uses 'asroot' and adds a container output.
<harmw> ahyea, lets see
<smoser> then i'll pull your patch if that works.
<smoser> and then we'll be closer 
<harmw> awk: /proc/1/environ: Permission denied
<harmw> I'm running without sudo now, manually invoking cirros-status
<smoser> bah. right.
<harmw> it does print ssh keys nicely now
<smoser> k. i shoudl have just used 'lxc-is-container', but i was hoping to avoid the fork :)
<smoser> http://paste.ubuntu.com/7075099/
<harlowja> smoser u around
<smoser> here
<harlowja> so about that stupid ifconfig crap
<harlowja> lol
<smoser> k
<harlowja> any thoughts on a good longer term solution, not that hard to adjust the sysvinit file
<harlowja> or other way is to say users should delete there ifconfig before snapshotting as part of cleanup
<harlowja> bb
<smoser> harmw, i really dont want to tell pepole they have to clean up
<smoser> i'd love to have a way around that.
<smoser> err... harlowja 
<harlowja> ya, smoser, hmmm
<harmw> smoser: iamroot() doesn't work
<harmw> iamroot() { return $(id -u)
<harmw> }
<harmw> anything wrong with doing it that way?
<smoser> harmw, bah.
<smoser> add 
<smoser>  [ "$UID" = "0" ]
<smoser> after it sets uid.
<smoser> ie
<smoser> iamroot() {
<smoser>     [ -z "$UID" ] || UID=$(id -u)
<smoser>     [ "$UID" = "0" ]
<smoser> }
<smoser> sorry. 
<smoser> yeah, the way i had it it would always say it was root
<harmw> uhm, how's this any different :P
<harmw> ie. still doesn't work
<harmw> it's now always using sudo
<smoser> bah. harmw i'ms orry.
#cloud-init 2014-03-12
<penguinRaider> hi I am new to cloudinit. I am trying to boot ubuntu server image in my local kvm installation. When the image boots up which cloudinit script does it run the one in /etc/cloud/cloud.cfg or /var/lib/cloud or some place else?
<smoser> penguinRaider, http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html
<penguinRaider> smoser, thanks sorry but I have one more question, instead of creating and mounting the another image can I put it somewhere in the ubuntuserver image to bypass the authetication?
<penguinRaider> I mean the seed part
<smoser> penguinRaider, you can, yes. you can put that same data into /var/lib/cloud/seed/nocloud-net inside the image.
<smoser> there a program 'ci-tool' https://code.launchpad.net/~smoser/cloud-init/ci-tool/
<smoser> that can help do that.
<smoser> and it might make sense to use '
<smoser>  mount-image-callback in conjunction with it
<smoser> described at http://bazaar.launchpad.net/~smoser/+junk/ppc64el-cloudimage/view/head:/README-usage.txt
<smoser> (ignoring the ppc64 arch part of it, its all the same for amd64)
#cloud-init 2014-03-14
<smoser> harmw_, around ?
<smoser> hm.. harmw_ someone is asking for respecting mtu returned in dhcp.
<smoser> i wish i'd have fixed it better with your staticroutes fix
<smoser> so that we weren't hard coding that in a binary
<smoser> but instead wrapping a udch call or something
<harmw_> smoser: hr
<smoser> hey.
<smoser> i wish i'd just wrapped udchpc
<smoser> so that i didn't have re-build in order to change the options that it is called with on ifup
<smoser> that just seems silly
<harmw> ah
<harmw> rebuilding a cirros image to chang a udhcpclint thingy, right
<harmw> would be silly inded
<harmw> is tht someone asking on lp?
 * harmw likes some place to read the whole story :)
<harmw> btw, Ive been trying to get me a new cirros image with updated kernel+mdev+hypervdrivers but after booting in hyperv it apparently forgot the initrd - while booting in qemu-kvm just goes without problems
<smoser> yeah, ubuntu kernel will work fine in qemu-kvm with built in drivers.
<smoser> not asking in launchpad, no.
<smoser> they were just setting the mtu on a network and sending that in dhcp
<smoser> and cirros wasn't oblighing
<smoser> obliging
<harmw> so cirros should just apply whatever DHCP options are inside the dhcp<whatsitcalled>
<smoser> yeah.
<harmw> sounds like a cool feature
<harmw> for 0.4 or somthing :p
<smoser> i'm not really sure how udchp does it.
<smoser> ie, udhcp is calling that ifupdown script
<harmw> nah, its calling the default.script
<smoser> and must be putting into environment varaibles names of the reponses.
<harmw> from /usr/something
<smoser> yeah. thats what i meant.
<smoser> ifupdown -> udchp -> default.script
<harmw> yep
<harmw> http://git.busybox.net/busybox/tree/networking/udhcp/README.udhcpc?h=1_1_stable&id=179f41778880d0a233435f5c9ed04e17316f479a#n68
<harmw> we can hack default.script to just apply whatever is set, in a reasonable way
<smoser> hm..
<smoser> so why the change we made for '-O' 
<harmw> well, not just everything ofc
<harmw> -O made it get these extras in the first place
<smoser> http://paste.ubuntu.com/7091665/
<smoser> so we have to one by one ask for things ?
<smoser> [annoying]
<harmw> -O,--request-option OPTRequest option OPT from server (cumulative)
<harmw> thats annoying, yes
<harmw> ill create a bug on LP first
<harmw> for future reference
<harmw> since I take it you're not diving in on this one right now :)
<smoser> right.
<smoser> did you try my last suggestion on that iamroot thing?
<harmw> uhm
<smoser> http://paste.ubuntu.com/7091714/
<harmw> hm, guess I missed that :|
<harmw> sry, let me check
<harmw> great, that works smoser 
<smoser> ok. i'm going to commit it then.
<harmw> I've did a simple test with just those 2 functions
<harmw> iamroot and asroot
<harmw> its using sudo whn it should
<harmw> so, ok
<harmw> nice
<harmw> smoser: please hold on the cirros-status merge
<smoser> ?
<harmw> grep DMI /var/log/messages gives no luck on HyperV
<harmw> while dmesg|grep DMI does
<smoser> really.
<harmw> yup
<smoser> the benefit of /var/log/messages would be that its permenant
<smoser> dmesg is a circular buffer
<harmw> indeed
<harmw> though I wonder on how much is changed for dmesg, and how mch that would matter
<smoser> i honestly dont even know how /var/log/messages is written :)
<smoser> pretty much anything that "just worked" with buildroot i did not bother figuring out.
<harmw> isn't syslog just doing that?
<harmw> hm, now my qemu vm has no /etc/mdev.conf while the hyperv vm does
<harmw> shouldnt I apply http://paste.ubuntu.com/7091714/ on my branch btw? then change this dmesg thingy and then you can just merge, right?
<smoser> is that right? does your branch have nothing else?
<smoser> if so, yeah, go for it.
<smoser> your hyperv you re-built cirros
<smoser> i thought
<harmw> uhu, new kernel as well
<smoser> right. well that'd be where mdev.conf came from silly
<smoser> oh. odd. its built in.
<harmw> i have the same disk.img loaded in qemu
<smoser> odd
<harmw> http://bazaar.launchpad.net/~harmw/cirros/cirros-sysinfo/changes
<harmw> doesn't look like unwanted changes in there :)
<harmw> https://code.launchpad.net/~harmw/cirros/cirros-sysinfo/+merge/209322
<smoser> k. 
<smoser> gracias
<smoser> harmw, merged. and pushed. thankm you.
<harmw> nice, thanks
<harmw> hm, LP didn't mail me about any merge
<harmw> smoser: any reason why console logs are disabled on tty0 from end of kernel boot until login prompt?
<harmw> (its logged to file though)
<smoser> yes.
<smoser> :)
<smoser> because there is no easy way to dot hat.
<smoser> kernel messages go to each target of 'console=' on the cmdline.
<harmw> indeed
<smoser> but init's output (and any process it spawns) can only go to one place
<harmw> hm yes
<smoser> outside of using 'tee' or something.
<smoser> so the code tries to figure out what the right place is.
<smoser> and i prefer serial console to vga
<harmw> +1 on that
<smoser> i'm not opposed to some way to make it go to both places.
<smoser> as long as it works.
<harmw> ok
<smoser> we have the same problem on ubntu too.
<smoser> its a real PITA
<smoser> the biggest pain, is that you can't actually tell if /dev/console is working
<smoser> echo "foo" > /dev/console
<smoser> works for a while
<smoser> (kernel buffers)
<smoser> and then it realizes that there is no /dev/ttyS0 (or wherever you had 'console=' on the cmdline)
<smoser> and it then starts to fail
<harmw> so, useless
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1123220
<harmw> ah ys
<harmw> smoser: any bugs/issues still open before 0.3.2 final?
<smoser> no. 
<smoser> jsut time.
<smoser> i jsut have to do it.
<harmw> ok, thn take time :)
<smoser> alright. so lets say i'm just going to go cowboy and do it.
<smoser> and call it 0.3.2 with minimal-ish testing
<smoser> but then we'll test it fairly well before we announce that anywhere.
<harmw> you could go rc1.. :)
<harmw> but 0.3.2 is better
<smoser> yeah. the problem is then people like you just want to put more stuff in
<smoser> (or people like me too, as i'd like the MTU thing fixed)
<harmw> so 0.3.2 it is :)
<smoser> you know, a young go getter like yourself could set up a nightly build service
<smoser> :)
<smoser> and automate all this. and get ifrasturcture infor gated c-i
<harmw> doesn't LP supply thta?
<smoser> it will build debs
<smoser> i dont think i could shoe-horn this in there.
<smoser> it might be interesting to try though :)
<harmw> hmk, well hosting something Jenkins-like for cirros or c-i sounds like a nice thing :)
<harmw> or just somthing manually, without Jenkins
<harmw> i like the idea smoser 
<harmw> smoser: whre shhould I put nightly disk images 
#cloud-init 2014-03-15
<tomoiaga> I was wondering about something. I used cloud init to setup a user and a password. However at leas in CentOS 6.5 the password has !! in front of it. It seems it was inserted without removing the previousely present exclamation marks. Is this a normal behaviour ?
<harmw_> smoser: http://buildservice.weites.com/cirros/
<harmw> it'll get updated nightly, if there are changes in trunk
<harmw> currently only x86_64 is beeing built, though crossing over to i386/arm should be possible aswell
<harmw> just a custom script btw, nothing fancy yet really
#cloud-init 2014-03-16
<smoser> harmw, i was really only kidding :)
<smoser> thats awesome
<harmw> smoser: if the scripts work there should be a new auto build in about 12hrs
<harmw> hmk, bin/build-release something to look at :p (I've just scripted all steps from the README)
<smoser> harmw, yeah, build-release should have everything together.
<harmw> looks like I'm doing most stuff in there aswell, there's some specific stuff like uploading to LP that I skip ofc
<harmw> (I've seen your mail btw)
<harmw> you didn't change the releaseversion in the cirros libs, so I'm still reading it as 0.3.2~pre1 (which sucks, so I've regexed it to read -pre1)
<harmw> cool: http://buildservice.weites.com/cirros/
<philips> Is there a style guide on whether dashes or underscores are preferred in cloud-init? 
<philips> write_file vs ssh-authorized-keys for example
<philips> filed a bug here for discussion https://bugs.launchpad.net/cloud-init/+bug/1293254
#cloud-init 2015-03-11
<jetole> Hey guys. I have a app that I am using to provision instances and it's passing a user password to the instances via http://169.254.169.254/openstack/latest/meta_data.json
<jetole> Can I use cloud-init to set the user password based on this data? Also I forgot to mention it has other variables all in json 
<smoser> jetole, use ssh keys , or use cloud-config to set password. 
<jetole> smoser: neither are a option in this case. It's a closed source app and it passes the data via a json tag in http://169.254.169.254/openstack/latest/meta_data.json
<smoser> i dont follow.
<smoser> some entity is running an instance for you?
<smoser> jetole, i think in the past i've said that patches are welcome.
<smoser> but if this is a close source app, and you're trying to get into it in some way, its likely the'd have such functionality disabled in their app even if cloud-init did support it.
<jetole> smoser: yes a software that the firm I work for has bought is creating instances and it's passing the user password via a json tag in http://169.254.169.254/openstack/latest/meta_data.json. Are you saying that cloud-init cannot access this data?
<smoser> it can access it sure.
<smoser> but it does not.
<jetole> smoser: and I cannot configure cloud-init to set a password based on a tag inserted into that json data?
<jetole> I mean aside from writing a script?
<jetole> ...which I've already begin doing anyways
#cloud-init 2015-03-12
<steveporter92> hi im having trouble executing the awscli tool from my cloud-init config on ec2
<steveporter92> in runcmd:
<steveporter92> im pip installing the awscli
<steveporter92> and running - 'aws s3 cp --region eu-west-1 s3://repo/puppet-latest.tar.gz /tmp/puppet-latest.tar.gz'
<steveporter92> but it throws the error /var/lib/cloud/instance/scripts/runcmd: 5: /var/lib/cloud/instance/scripts/runcmd: aws: not found
<steveporter92> in my log
<steveporter92> any ideas?
<Odd_Bloke> steveporter92: Is awscli installed?
<Odd_Bloke> (Check the log to see if it's actually being installed before your script is run; I can't remember what order things happen in)
<smoser> steveporter92, its either ordering or path
<smoser> runcmd may not have /usr/local/bin in its path, i think it should, but may not.
<smoser>  /var/log/cloud-init-output.log should show you some more info. you can also just try running that command yourself.
<steveporter92> yeah it was the path
<steveporter92> thanks guys :)
#cloud-init 2015-03-13
<devicenull> is there any real support in cloud-init for ipv6 only?
<devicenull> I guess i'd need to update the metadata source
#cloud-init 2015-03-15
<nk121> hi, how can i pull down apt package like linux-image-extra-`uname -r` in cloud-init?  specifically, i need uname -r to be resolved at run time
#cloud-init 2016-03-14
<smoser> Odd_Bloke, do you think my returning "......" is silly ?
<smoser> for dmi data.
<Odd_Bloke> smoser: In what case?
<smoser> recent commit to trunk, i made it return ..... (what dmi decode would return) when /sys is full of \xff
<smoser> but i relally think "" seems better, as otherwise the user has to specifically know about this "..."
<smoser> (and worse, i return the length of the value in sys, basically .replace('\xff', ".")
<smatzek> smoser:  I've been seeing the recent activity around here on dev of new cloud-init function around networking, which is where the majority of my interest lies.  What dev is going on?  Is this on a 0.7.x branch?  It seems the v2.0 work died off.
<smoser> 0.7.x
<smoser> my 'net1' branch is where i'm working on it, and i want to get at least part of that into ubuntu today.
<smatzek> and then you'll merge that back to master?
<smoser> oh. well, yes. it will all land in trunk
<smoser> before ubuntu
<smatzek> thanks
<Odd_Bloke> smoser: Yeah, I definitely think returning '' or None would be better.
<smoser> '' seems better than None to me.
<smoser> in order to differenciate between "NO VALUE THERE" and "value not set"
<Odd_Bloke> smoser: Oh, sorry, this is for all Fs, not all 0s?
<Odd_Bloke> Then, yeah, agreed on empty-string.
<Odd_Bloke> Of course, the question then is if we modify dmidecode's output in that case? :p
<smoser> Odd_Bloke, i'm modifying dmidecode's output
<smoser> Odd_Bloke, review?
<smoser> https://code.launchpad.net/~smoser/cloud-init/trunk.dmidecode-null/+merge/288925
<Odd_Bloke> smoser: I'm heading in to a (hopefully) half-hour meeting; will review after. :)
<smoser> meh. you can review while you wait for everyone else to show up
<smoser> really quick
<smoser> but your suggestion is also ok
<Odd_Bloke> smoser: Cool; +1'd.
#cloud-init 2016-03-17
<madorn> Hi, currently running cloud-init 0.7.4 - i can't seem to get a cloud-boothook to run.  the handler places the boothook script into the proper directory under /var/lib/cloud, but never runs it..
<madorn> actually i believe its working..my script was bad :(
<smoser> magicalChicken, around ?
<magicalChicken> smoser: hey
<smoser> hey.
<smoser> have you gotten a chance to look at ip= stuff  at all ?
<magicalChicken> I have a branch that's loading the config, I'm gonna get unit tests done today
<magicalChicken> I'm parsing from the files and merging in after ns is loaded, that made the most sense to me
<magicalChicken> lp:~wesley-wiedenmeier/cloud-init/trunk.net1-cmdline-ip
#cloud-init 2016-03-18
<harlowja_at_home> smoser, woot, i'm an openstack PTL now :-P
<harlowja_at_home> take that bossman
<harlowja_at_home> lol
<harlowja_at_home> crap i never wrote that parser
<harlowja_at_home> lol
<smoser> oh my.
<smoser> yeah, crap to you
<smoser> but congrats too
<Odd_Bloke> All hail harlowja_at_home
<harlowja_at_home> lol
<smoser> harlowja_at_home, because i'm too lazy to google
<smoser> what PTL ?
<harlowja_at_home> project technical lead smoser
<harlowja_at_home> whatever that means, ha
<harlowja_at_home> :-P
<smoser> harlowja_at_home, i know what it meant.
<smoser> which ptl are you
<harlowja_at_home> oh
<harlowja_at_home> lol
<smoser> 'what PTL', not whats ptl ?
<harlowja_at_home> oslo, the common library stuff/glue
<smoser> wow. you actually are important.
<harlowja_at_home> ha
<harlowja_at_home> idk
<harlowja_at_home> i'm just a poor boy from a poor family
<harlowja_at_home> easy come easy go
<harlowja_at_home> lol
<magicalChicken> smoser: Hey, I've been working on finishing up merging ip= configuration into net config and I think everything is working correctly,
<magicalChicken> but I'm not sure if some of the values I'm writing out should be omitted or not
<magicalChicken> here's a demo run: http://paste.ubuntu.com/15418034/
<suro-patz> hello folks !
<suro-patz> Is there a way to re-run network config by cloud init, after the node got booted up
<suro-patz> I think one alternative is removing /var/lib/cloud/instances/* and then reboot
<suro-patz> I am trying to save the reboot each time
<suro-patz> harlowja_at_home:^^
<smoser> magicalChicken, ok. great. i have some other stuff for you...
<smoser> suro-patz, you can probably re-run cloud-init local (or cloud-init) which ever wrote the data for you.
<smoser> magicalChicken, ok. so it looks there like 'gateway' is missing from eth0, right ?
<smoser> oh. i see the comment.
<smoser> so you're saying it only added the new interface
<smoser> i think that is fine.
<smoser> suro-patz, rm -Rf /var/lib/cloud /var/log/cloud-init*; cloud-init init --local'
<smoser> magicalChicken, http://paste.ubuntu.com/15419211/
<smoser> rharper, ^
<smoser> thoughts on that if you see it. i hope to put some more work into that.
<smoser> i actually think its fairly well throught through and solid at this point.
<suro-patz> smoser: will try the same, thanks!
#cloud-init 2016-03-19
<rharper> smoser: s/ENI/etc\/network\/interfaces (ENI)
<smoser> rharper, ?
<rharper> smoser: re your network config writeup
<smoser> just shorten you mean ?
<rharper> I mean expand it first
<rharper> and indicate in parens you are shortening
<rharper> cloud-init writes out /etc/network/interfaces (ENI) ...
<rharper> I assume this is for documentation or for our consumption only (review) ?
<smoser> http://paste.ubuntu.com/15420393/
<smoser> i might throw it itno a google doc for easier comment
<rharper> yeah
<rharper> please
<smoser> much of it is description that was in my head and as i worked through things.
<rharper> yep
<smoser> i think its actually reasonably stable at this point.
<smoser> rharper, i'll do that before end of weekend.
<smoser> and will schedule  something early monday to hangout
<smoser> magicalChicken, leave your availability here.
<smoser> others here playing along, i'll go ahead and make it public google doc so that you could read and comment if interested.
<smoser> as i know harlowja_at_home in his PTL billiance almost has my converter done
<rharper> smoser: I'll miss monday morning, I've a 9AM dr. appt, but after lunch Monday would be fine
<rharper> smoser: PTL, what's PTL? =)
<smoser> project technical lead.
<smoser> rharper, ok. tell me when you can. i want as early as i can
<rharper> smoser: I _know_ what PTL is, just like you did  earlier with harlowja_at_home
<rharper> smoser: ok, I'll ping you  as soon as I'm back, I could probably make 10 AM central, 10:15 if I'm late.
<smoser> k. gracias
<smoser> magicalChicken, send me email if you dont want to leave that at home.
<smoser> rharper, ah. saw your joke
 * smoser turns to pumpkin. good night all
<harlowja_at_home> lol
<magicalChicken> smoser: Hey, sorry I saw this pretty late, I'm available anytime monday morning
#cloud-init 2016-03-20
<smoser> magicalChicken, ok.
<smoser> thanks
#cloud-init 2017-03-13
<powersj> hehe "AssertionError: 'HST' not found in 'HDT\n'" silly daylight savings messing with my integration tests
<rharper> lol
<smoser> powersj, i put  a comment on your mp... i may be over thinking things.
<powersj> smoser: hmm but won't that change between -0900 and -0800 depending on day light savings?
<smoser> it should not.
<powersj> and that command depends on what timezone you are in
<powersj> which is ok, since we are setting
<smoser> no... we specify the full time with offset -0400
<powersj> yes, but the output changes... here is mine: "Wed, 02 Nov 2016 22:47:00 -0600"
<smoser>  Thu, 03 Nov 2016 00:47:00 -0400 == Wed, 02 Nov 2016 19:47:00 -0900
<powersj> oh... maybe that's what I'm not getting
<smoser> the output changes based on the default time zone of the system.
<smoser> (your default time zone is 0600)
<smoser> but the test is testing that we can set the default time zone to US/wierd-timezone
<smoser> that must be hawaii ?
<powersj> it is Hawaii
<smoser> so .. if we correctly set the timezone
<smoser> date will always convert the '--date' to the default timezone of the system.
<smoser> and so we're checking that dates output was converted to the time zone that we expect to have set the system to
<smoser> right?
<powersj> yeah, let me change it and play with it a bit. Never used date like that before, so still grawking it.
<smoser> :)
<rharper> alternatively can't we check that /etc/localtime points to the right tz we specified?
<powersj> ah that might be good too, check that symlink
<smoser> http://paste.ubuntu.com/24171815/
<smoser> maybe that helps ?
<smoser> well, the symlink could work, but that is somewhat of an implementation detail.
<powersj> smoser: it does, thanks :)
<smoser> where as the other way depends on 'date' behaviing correclty.
<smoser> powersj, did you open a bug ?
<smoser> for the timezone?
<powersj> smoser: no
<smoser> k
<powersj> I did wrt the new password change (a 2nd one)
<powersj> I'll get the timezone merge fixed up here in a bit
<smoser> http://paste.ubuntu.com/24172068/
<powersj> lol
<powersj> thank you :)
<smoser> i'll just take that now. i think the comnments there make it more clear than my explanation before.
<powersj> smoser: yes they do and wfm thanks
<smoser> early november is probably a really bad time to pick
<powersj> why?
<smoser> as someone might think that is on the cusp of a time change.
<powersj> ah
<smoser> but i'm using it anyway
<smoser> :)
<smoser> powersj, you mentioned something about cloud-init tags and buildilng ?
<smoser> in your qa report
<smoser> more info ?
<powersj> I was trying to understand why we needed the tags when setup.py runs. I had not looked at that file enough to understand how it gets packaged up
<powersj> So looked into how you specify an init system and the check that occurs when building cloud-init via tag + version in __init__
<smoser> yeah, the check can be annoying for sure, but you should always have some tag to base it off of.
<powersj> I've been played with snap'ing cloud-init as well and the python plugin doesn't take an option so I couldn't specify an init system
<powersj> Even if it doesn't quite make sense too, it was an interesting lesson :)
<smoser> powersj, python plugin doesnt take an option
<smoser> ?
<smoser> you mean tryin to get the init system into setup.py ?
<powersj> yes
<powersj> so your part would look like this: https://paste.ubuntu.com/24172666/
<powersj> and we would want a "options: ['--systemd']" or something like that
<powersj> This all started because I was looking at https://build.snapcraft.io/
#cloud-init 2017-03-14
<dtp> i've worked around https://bugs.launchpad.net/cloud-init/+bug/1671927, but now the gateway is not being rendered to ifcfg-eth0.  gateway is defined in network_data.json, but not content/0000, so more evidence i guess that it's preferring 0000 in this case
<smoser> dtp, can you give the config drive that you ave?
<dtp> smoser some of the files are available at http://imgur.com/a/qEElh
<dtp> and others are attached to the bug.  or is there another you need?
<smoser> dtp, you couldnt use pastebin ? for text :)
<dtp> i don't have net access to the VM / only have spice console
<smoser> dtp, so the issue... is that cloud-init did not write that /etc/network/interfaces
<smoser> as it says, nova "injected" it.
<smoser> essentially, your cloud platform broke your image by thinking it knows what it is doing.
<smoser> and cloud-init does not attempt to un-break it for that scenario.
<smoser> that make sense ?
<dtp> which /etc/network/interfaces?
<smoser> i'm 95% certain that nova mounted the image and write /etc/network/interfaces
<smoser> oh wait.
<smoser> no.
<smoser> ok. let me re-think
<dtp> this is fedora 25 / no /etc/network/interfaces, only network-scripts/... files
<smoser> right
<dtp> so where i'm coming from is this all worked on fedora 23 w/ older cloud-init
<dtp> trying to get fedora 25 up
<smoser> you're right. it sure would appear that cloud-init is preferring to parse the 0000 file
<smoser> theres no way you can get the disk there?
<dtp> which file do you need?
<smoser> you shoudl be able to bring up networking by hand
<smoser> if you just get me the config drive (dd if=/dev/vdb of=my.img)
<smoser> or something to that affect, thats the easiest.
<dtp> ok, i'm in a meeting but will after
<smoser> dtp, just reading code, it sure seems like it should be looking at the network json in 0.7.8
<dtp> some of the other guys in here on Friday seemed to think it was skipping that stuff and using meta data or something?
<smoser> well if you attached a config drive then it should be reading that.
<smoser> there is a config drive, right ?
<smoser> thats what you showed with 'cat' ?
<smoser> yeah
<dtp> yeah
<dtp> they were saying something else in the config drive caused it to ignore network_data.json
<dtp> meta_data i think
<dtp> but i didn't see much in there
<smoser> hm.
<smoser> can you show me ls /sr0/openstack/ ?
<dtp> smoser - http://imgur.com/a/YXfc5 - still working on bringing network up
<smoser> hm.
<smoser> that wasnt what i expected :)
<dtp> oh?
<smoser> that should work. i expected that you were going to honly have 2016-10-06 and 2017-02-22
<dtp> one thing we noticed was that the debug logs in ConfigDrive where it says (if network_data use it, else use 0000) weren't being logged
<dtp> so it wasn't making it there before it went to write net conf
<dtp> i guess the trace is in that original imgur
<dtp> would explain how it skipped that part
<smoser> well, if you can get me a copy of the cdrom i can get further. without that, ... i'm not sure whats going on. and /var/log/cloud-init.log woudl be good too
<dtp> yeah, that's in the imgur already - http://imgur.com/a/qEElh
<dtp> at bottom
<smoser> well, the full thing..
<dtp> ok
<dtp> i'll keep working to get networking up on it
<rharper> smoser: re: network passthrough/netplan support;  for zesty it's too late to switch to using netplan/systemd by default; so I'd like your opinion on how best to set a policy in the Distro object (or elsewhere that the Distro can use to select the right renderer)
<smoser> well, rharper for ubuntu we have (i think) 3 cases:
<smoser> a.) ifupdown without netplan: use ifupdown
<smoser> b.) netplan without ifupdown: use netplan
<smoser> c.) ifupdown and netplan: this is where we need a choice
<rharper> correct; it's the (c) I was interested in
<smoser> and i guess
<smoser> d.) no ifupdown, no netplan: scream
<rharper> yakkety and zesty; netplan (nplan) is seeded in the image (it's in main); so we're always in (c) for yak and zesty images
<smoser> so i guess generically, we have  a list of available "backends" that cloud-init will come up with.
<smoser> in 'c', that'd be: available = ['ifupdown', 'netplan']
<rharper> renders
<smoser> or something
<rharper> eni and netplan
<rharper> and sysconfig
<smoser> ok. yeah.
<smoser> i think that the distro should coome up with an ordered list of which are available.
<smoser> available = ['eni', 'netplan']
<smoser> for 'c' above
<smoser> and for 'a', that'd just be: available = ['eni']
<smoser> then, i guess we just have config (config/cloud.cfg i guess) that has a order of preference
<smoser> system_info/network/render_priority = ['eni', 'sysconfig', 'netplan']
<smoser> and pick the first entry in render_priority that is also in available
<smoser> ?
<rharper> ok, so order of the available renders is the policy
<rharper> so, for say UC16 (or to be named 17.10 release), we 'd update the list to indicate the preference
<rharper> smoser: that seems reasonable; is config only reasonable?  I know we do things like enumerate the DataSource classes to build a list; do we need to do that here for net/render.py to probe for possible renderers?
<smoser> well, i think something has to do that. we can put a generic implementation in the distro class.
<rharper> sure, but I'm questioning whether we're dynamically building that list from probing cloud-init .py files (or class loading); versus returning a list of known renderer names;
<smoser> it has to be dynamic
<smoser> such that in 'a', netplan is not available
<smoser> we can allow a force of netplan even if the discovery says netplan isnt there.
<rharper> well, there's whether cloud-init has a netplan renderer vs. whether the system has the tool to consume the rendered file
<rharper> so, older cloud-inits won't have a config list with 'netplan' in it without a netplan class;  but they system may have nplan package and netplan binary available
<smoser> well, older cloud-inits wont read that list
<smoser> they're just goign to pick eni
<rharper> smoser: how's this look http://paste.ubuntu.com/24178612/
<smoser> rharper, hangout ?
<rharper> y
<smoser> http://c.brickies.net/hangout
<smoser> rharper, r class loading); versus returning a list of known renderer names;
<smoser> oos
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1671927
<smoser> http://paste.ubuntu.com/24178813/
<dtp> smoser i'm giving up on getting networking up on this thing
<dtp> the command i need to run to add the default route is not working
<dtp> i'll add more of the cloud-init log to the imgur
<dtp> and let me know if there's anything else i can give
<dtp> smoser nevermind, i got the sr0 image
<dtp> how do you want it?
#cloud-init 2017-03-15
<jsheeren> hi all, is there a way to detect if a cloud init has failed, besides polling the log?
<rharper> jsheeren: yeah, /var/lib/cloud/data/{result,status}.json
<smoser> jsheeren, and those are in /run/cloud-init/ also. i think thats probably preferred place as /var/lib/ could be stale
<jsheeren> aha! thanks!
<jsheeren> is there a config option somewhere where i can run a program when cloudinit fails?  or is it best to write some sort of daemon who polls the /var/lib/cloud/data/{result,status}.jso files?
<rharper> I don't think there's any sort of run this on failure in cloud-init directly;
<rharper> I suggest looking at using an inotify watch on those files to get a trigger when they're written
<jsheeren> rharper: ok, thanks, i'm going to look into that
<smoser> result will be written only once, and atomicly
<smoser> but only if cloud-init-final ran
<smoser> status is written with each stage.
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/319943 woudl apprecate a review. it wont fix the root cause of bug 1671927 but it will fix something.
<rharper> smoser: commented
<powersj> magicalChicken: if I start working on the KVM code without some of your new branches, how much hurt will that be?
<magicalChicken> powersj: the signature for some of the platform methods changed
<magicalChicken> it would probably be best to just base it off the new integration-testing branch
<powersj> magicalChicken: this one right? https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+ref/integration-testing
<magicalChicken> powersj: also the image config format changed and the config handling changed, which would be an issue
<magicalChicken> powersj: yeah
<magicalChicken> if you want to start off of that, it should work out ok, i wont rebase again
<powersj> so in terms of branches of yours we want to land there are 3? integration-testing, integration-testing-distor-features, and integration-testing-invocation-cleanup?
<magicalChicken> integration-testing and integration-testing-invocation-cleanup are the two ready right now
<magicalChicken> although i still need to rebase invocation-cleanup
<magicalChicken> the distro-features one i'm still debugging some test failures on centos
<powersj> ok
<powersj> oh what about integration-testing-pylxd?
<magicalChicken> oh, that one isn't needed, the pylxd fix is in the main integration-testing
<powersj> ok
<magicalChicken> i should probably delete that one
<powersj> ok, so for now we need smoser to look at integration-testing
<magicalChicken> yeah, that's the main thing right now
<smoser>  /o\
<powersj> is that the reverse of being excited? hands over ears?
<magicalChicken> lol
<smoser> is your commit message still all correct ?
<smoser>    - when using 2.2 the test suite cannot complete large
<smoser>      runs without hitting the pylxd too many open files
<smoser>      issue, so 2.2 should only be used for development
<magicalChicken> smoser: no, thanks for catching that, i hadn't updated the commit message at the top yet
<smoser> magicalChicken, but you think that the commit series is sane as they are ?
<smoser> ie, i'm ok to do a pull of mlutiple commits in a series here.
<magicalChicken> yes, i think so
<magicalChicken> i've gone through and rewritteen all the commit messages to make sense from outside the integration-testing dev branch
<magicalChicken> and the commits in that series are all pretty well separated from each other
<powersj> magicalChicken: why multiple tox entries? Do we really need the old one?
<magicalChicken> powersj: i can pull it out
<smoser> yeah, i'd drop that.
<powersj> +1
<magicalChicken> sure, one minute
<smoser> and i'd drop support for 2.1 as it seems generally broken
<smoser> right?
<magicalChicken> smoser: yeah, it doesn't return command exit code which is pretty bad
<smoser> if execute('/bin/false') blissfully goes on
<smoser> do you have an reason for 'ignore_errors' rather than explicit values like subp ?
<magicalChicken> smoser: i could do something more along the lines of subp
<smoser> ignore_errors == subp(['/bin/false'], rcs=range(0, 256))
<magicalChicken> yeah, that gives us some more control when we need to ignore errors
<smoser> ie, if i want to just ignore anything, thats pretty easy to do with an explicit list of rcs
<smoser> and  less surprises when somethign returned 14 and we only really expected 0 or 1
<magicalChicken> currently the ignore errors is pretty much only used for the actual collect scripts. and i want to remove that eventually, but i haven't gone through all the collect scripts to see what the impact would be
<smoser> dont worry about re-writing the commit message at the moment.
<magicalChicken> ok
<magicalChicken> i was going to rebase to pull out the tox env, do you want me to just add a new commit for it instead?
<magicalChicken> smoser: i pushed commits to remove the tox env and switch to rcs=(0,1) for execute(), should be ready to go now
<powersj> magicalChicken: thanks!
<smoser> ok. i'll oook a bit mroe
<magicalChicken> aah, it looks like i may have messed up a bit in the rcs 1, i'm going to amend that real quick
<magicalChicken> should be good to go now, i messed up the change in collect and didn't catch it until one of the tests failed a collect script
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/feature/net-render-priority
<smoser> that is the branch i have.
<rharper> smoser: cool
<smoser> but from cloudinit.net import render_factory
<smoser> render_factory.select_renderer()
<rharper> smoser: and in my branch, I'll insert the netplan stuff
<smoser> ok. i have to run for a while, probaly check back in later tonight.
<rharper> ok
<rharper> I should have an update pushed with unittests
<Odd_Bloke> smoser: Can I change the default username from {vendor,user}-data?  I still want my renamed user to get the default SSH keys etc.
<Odd_Bloke> Aha, using "user:" rather than "users:" works.
<dtp> hello again.  i'm having a problem similar to https://bugs.launchpad.net/cloud-init/+bug/1645597, but i'm on Openstack and ConfigDrive; should i log a new bug or just add on to this one?
<rharper> smoser: ok, pushed v2 netconfig branch again, reasonable unittests for v2 rendering and other places;
#cloud-init 2017-03-16
<rharper> smoser: I'm going to merge the policy branch into my net-passthrough and then see if this all still works =)
<smoser> rharper, ok... i just did something, tried to test yiour branch and lxc and didn't seem to do what i expected.
<smoser> putting together something to show you what i tried.
<rharper> ok
<rharper> smoser: in the policy feature; you've not yet wired it into the distro yet, right?  also not poked at what the systemconfig key/val would be yet?
<rharper> smoser: so, we don't have a common config dictionary for renders;  that makes the distro part where it fetches the first renderer via policy, it still needs to know which renderer it is can construct possible different configs;  unless we come up with a common config structure
<smoser> rharper, well, at first i dont think we *need* any configuration on nit.
<rharper> renderers today all that a config dict; and they're all different; that's just the code today without any new code;
<rharper> some things are common, for example, eni and sysconfig take a 'netrules_path' key for udev rule hooks
<rharper> but eni has a eni_header (which we could truncate to header, and reuse for netplan as well, we use the same one anyhow);
<smoser> rharper, right. so we do have to know how to call each of the renderers, which kind of suck
<rharper> sysconfig uses dns_path  and sysconf_dir,
<rharper> yeah
<smoser> but we do not need to necessarily have that configured
<smoser> the datasource can just raise a RunTimeError if it doens't know how to call the selected_renderer
<smoser> but even then, i think the renderers all take config={}
<rharper> they all take the config; I'm saying it's annoying in the distro that supports multiple renders
<smoser> and probably do something sane-ish... they can  just be improved to better dtrt
<rharper> the logic should be 1) render_cls = load_net_render_by_policy();  2) render  = render_cls(config)
<rharper> but instead, we have to ask well, if render_cls == 'eni' then X; elif render_cls == 'netplan' Y
<rharper> where the config dict looks different
<rharper> that's what I don't like;
<smoser> i understand.
<smoser> but 2 things
<smoser> a.) maybe you dont' have to do that.... you just call renderer(config={})
<smoser>   (why can't that do the right thing)
<smoser> i dont remember 'b'
<rharper> yeah, looking at the constructors; they all have sane defaults;  save the HEADER isn;'t enabled by default;
<rharper> and the header is common, so that could work
<rharper> alternatively, the distro could pack both sets of config into a single dict; they do not overlap
 * rharper plays with some code an unittests
<smoser> rharper, right. i considered that too
<smoser> (putting both configs in)
<smoser> but honestly that isn't any more difficult than this:
<smoser> configs = {'eni': {'eni_data_for_config_param': 1}, 'netplan': {'netplan stuff'}}
<smoser> name, renderer = load_net_render_by_policy()
<smoser> renderer(config=configs[name])
<smoser> or config=configs.get(name, {})
<rharper> well, load_renderer instantiates the class, you pass the config in
<rharper> but I'll play around with something
<rharper> we could pass the config dict as you show with the class name configs and have the loader pull out the config
<rharper> that seems sane
<smoser> rharper, http://paste.ubuntu.com/24189094/
<smoser> that doesnt seem to get networking. it ends up rendering yaml, but with only 'lo' in the 50-cloud-init
<smoser> trying with:
<smoser>  /tmp/try-lxc-v2 ubuntu-daily:zesty /tmp/cloud-init_all.deb
<rharper> k
<rharper> unrelated, I merged your feature branch in and I'm getting unittest errors with 'is_exe' is not defined
<smoser> rharper, i was i might not have committeed that... let me check. i did see that.
<smoser> thought i had fixed it.
<smoser> will fix now
<rharper> k
<smoser> so that paste above... i verify in logs that you're rendering netplan
<smoser> and there is a netplan config, and there is no 50-cloud-* (i was wrong)
<smoser> but it does not get networking
<rharper> ok, it;s likely related to other systemd unit changes
<rharper> you can check: systemctl status systemd-networkd (it should be active) and systemctl status sytemd-network-wait-online.service (should be active too)
<rharper> it's likely that one or the other didn't trigger which means we write the file but nothing runs
<rharper> also /run/systemd/network/* should have 10-netplan-* files if cloud-init called netplan generate (which it should if it rendered the 50-cloud-init.yaml )
<rharper> this is the "fun" of systemd unit magic
<smoser> rharper,  i pushed my renderes branch... i had the is_exe change locally but not pushed. sorry
<smoser> and confirmed that lxd works after reboot
<smoser> with your branch network comes up after reboot
<rharper> smoser: ok; I hacked in my own is_exe, but will switch
<rharper> I need to update my unittests now that the default policy is to use eni; my netplan paths which I mocked out a which('netplan') require a bit more work since it finds 'eni' first
<rharper> smoser: do you have an idea on what we'd put in sysconfig dict to override the default policy and pass it ?  I think the Distro object would config.get('network_render_policy') or something like that
<smoser> rharper, looking
<szb> hi all, I'm getting errors from apt 'Could not get lock /var/lib/apt/lists/lock' when trying to add a source. It looks like the initial apt-get update/upgrade is still running? Why doesn't this block the rest of the script?
<paulmey> I'm thinking about the scenario where we make the Azure datasource a local datasource...
<paulmey> do local datasource get to execute any code at the end of cloud-init provisioning?
<smoser> \o/
<smoser> paulmey, yes.
<smoser> well, as much as network datasources do
<smoser> szb, what is "the rest of the script" ?
<paulmey> ok, so even a local data source gets to signal successful provisioning over the network in the end...?
<smoser> everything is serial currently in cloud-init. perhaps you had an old lock file there that had got captured?
<smoser> paulmey, where does that happen for azure now ?
<paulmey> let me look...
<smoser> so the answer is... we'll make it work and fix what we have to to do so.
<szb> @smoser: I have "package_upgrade: true" at the top, and then farther down I have an "apt: sources:" section and it breaks there because apt is still locked upgrading.
<smoser> szb, can you show me what you have for apt: sources: ?
<smoser> but fwiw, this is not run "top down".
<szb> @smoser: http://pastebin.com/BURKWmZs
<smoser> szb, can you paste a /var/log/cloud-init.log ?
<smoser> (and fwiw, xenial ubuntu images have 'pastebinit') so you can just run 'pastebinit /var/log/cloud-init.log'
<smoser> szb, i dropped the 'fs_setup' section, and it worked for me here.
<szb> @smoser: http://paste.ubuntu.com/24190140/
<szb> @smoser: Hmm, interesting. Is this not a good use for cloud-init, setting up mongo? I'm not sure but I like settings things up this way rather than coding a shell script.
<smoser> unrelated suggestion, i suggest even though its much longer that you specify the key, not the keyid.  that way you remove the dependency on a gpg server.
<smoser> szb, it seems sane to me.
<smoser> can you paste /var/log/cloud-init-output.log ?
<smoser> i suspect that is what showed you the lock complaint
<smoser> but i dont know what would cause apt to be running that early.
<paulmey> @smoser, it's currently done at the end of the func that gets the metadata
<paulmey> that's not the right place anyway
<paulmey> :-/
<smoser> well, it doesnt seem terrible at the end of that func. as its deciding that its "done" at taht point.
<smoser> i think it might fit in 'activate'
<smoser> activate will be called when networking is configured.
<szb> @smoser: http://paste.ubuntu.com/24190179/
<smoser> szb, i'm not sure what has that lock
<szb> @smoser: Ah, well ty for looking
<smoser> so, i'm pretty sure that cloud-init is not doing it.
<smoser> i've never seen this error before.
<smoser> is this a stock ubuntu image ?
<smoser> szb, ^ ?
<szb> base 16.04 LTS AMI with some customizations, built from packer.
<szb> @smoer, could this in our APT config be it? APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1";
<rharper> harlowja: on the sysconfig path, in rhel.py if we use the 'apply_network_config' path which uses the renderer instead of the _write_network_config() path I noticed that the section that writes out 'etc/sysconfig/network' file isn't rendered;  I suspect that should be common ?
<harlowja> i would agree
<rharper> https://git.launchpad.net/cloud-init/tree/cloudinit/distros/rhel.py#n100
<rharper> in practice, is that already configured?
<harlowja> no afaik
<harlowja> *not
<rharper> I would have thought that fedora/rhel images would have failed if they were using the apply_network_config path if those values aren't set
<harlowja> ya perhaps cut off the 'if dev_names'
<harlowja> and just let it happen
<rharper> y
<rharper> it needs a bit more to make it common between the two methods; but I'll see what I can do
<rharper> something like a _enable_networking() method
<rharper> which takes the v6 booloean
<rharper> then we can call it from both methods
<harlowja> wfm
<rangerpb> heya Odd_Bloke , you around possible to talk about https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n77 ?
<rangerpb> looking for a little background in it
<Odd_Bloke> rangerpb: I'm in a meeting ATM, but should be out soon; if you ask me some questions I'll answer once I have a minute. :)
<rangerpb> Odd_Bloke, well , lets start with what is the purpose of that method?  can you remember why it is needed?
<rangerpb> then the next question is that I have had to recently patch the non-agent provisioning path in DataSourceAzure to perform the whole set hostname and bounce of the network to propogate things to DDNS.  it duplicates a lot of code like https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n115 and lines following (including the call to the bounce method).
<rangerpb> smoser has suggested I see if I can refactor some of the code between both paths to reduce duplication ... but that contextlib method seems like it is going to be problematic
<smoser> rangerpb, hey.
<smoser> so on azure, i thought you sue NetworkManager rahter than sysconfig ?
<smoser> did i make that up?
<smoser> ah. never mind. i understand what i was going to ask. i was wondering how we render networking. but i guess on your images there, fallback networking gets rendered, but nothing uses it.
<rangerpb> not sure what you mean
<smoser> you use network manager in some images right?
<rangerpb> in some yes
<rangerpb> but i guess that decision is made by who makes the image
<smoser> well, as it is right now, cloud-initdoes not support reading network configuration from a datasource and rendering it to NetworkManager.
<smoser> so it works for you just because cloud-init will render the sysconfig networkign on fedora, but your image doesnt pay any attention to it.
<rangerpb> I believe the only thing that is read is the hostname
<rharper> smoser: maybe time for netplan
<rharper> it writes NetworkManager configs
<smoser> :)
<smoser> rangerpb, on azure that is true.
<smoser> there, cloud-init selects a "fallback network configuration" that equates to "run dhcp on the first network interface".
<smoser> but on other clouds (digital ocean, openstack, lxd, smartos) the cloud provider provides information on how the network should be configured
<szb> @smoser: Fixed, I think the APT unattended upgrades in the base AMI were the issue. I removed those and rebuilt the AMI and now it's working. ty for your hlep!
<smoser> szb, yeah... you want to file a bug ?
<smoser> that is quite obnoxious...
<Odd_Bloke> rangerpb: Hmm, let me see if I can remember.
<smoser> generally speaking, i think that apt unattended upgrades thing is a pita. as it can cause *anything* to fail
<Odd_Bloke> rangerpb: https://bugs.launchpad.net/ubuntu/+source/walinuxagent/+bug/1375252 was the bug I was fixing.
<Odd_Bloke> rangerpb: IIRC, the Azure fabric expects you to use the hostname that it has recorded for your instance.
<Odd_Bloke> rangerpb: So cloud-init was always resetting to that hostname on boot.
<Odd_Bloke> rangerpb: Whereas temporary_hostname just sets back to that hostname while the agent is talking to the fabric, and then returns it to the previously-set one.
<Odd_Bloke> (So that users can modify hostnames.)
<smoser> powersj, around ?
<smoser> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/319878
<powersj> smoser: yes
<smoser> jane's entry in /etc/shadow should have $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
<smoser> right?
<powersj> yes
<powersj> want a simple test of that one?
<smoser> yea
<smoser> do you know what that passowrd is ?
<powersj> I don't I think I grabbed a string and messed it up more
<smoser>  ?
<smoser>  !$1$ssisyfpf$YqvuJLfrrW6Cg/l53Pi1n1
<smoser> https://git.launchpad.net/cirros/commit/?id=95f4ffa2f5339aa04226718895a780f2994817b9
<powersj> oh well then maybe I didn't lol
<powersj> want me to change it?
<smoser> yeah, lets use that.
<smoser> just becauase its good to use something i think that we knwo.
<smoser> and [put a comment in there.
<smoser> or we can just put their names for each one
<smoser> powersj, i'll do this an give you a suggestion
<powersj> smoser: ok, are you fine with the Random -> RANDOM changes as well? I kind of lumped those in and was not sure if that should be another merge or not
<smoser> its fine. its supposed to be RANDOM to indicate a random password, right?
<powersj> yes
<rangerpb> Odd_Bloke, but it appears to me in the code it sets the hostname to what azure expects by getting the hostname from the metadata...
<Odd_Bloke> rangerpb: Right, because walinuxagent does a DHCP bounce which includes the hostname.
<rangerpb> so ... why is the method needed then ?
 * rangerpb must be misunderstanding something
<Odd_Bloke> rangerpb: I think it's this: (1) User boots instance, (2) user changes hostname of instance, (3) user reboots.  At that second boot, we still need to report the original hostname.
<Odd_Bloke> These assumptions may no longer hold; I was modifying a strategy that was already in place.
<rangerpb> so this covers an instance when the user changes the hostname for some ungodly reason ?
<rangerpb> interesting ...
<rangerpb> not something I had considered
<Odd_Bloke> rangerpb: I believe so, yes.
<rangerpb> paulmey, do you know if this still holds water?
<rangerpb> thanks Odd_Bloke !
<Odd_Bloke> :)
<rangerpb> Odd_Bloke, one reason I am questioning this method is that if the hostname is not set up correctly and the network bounces, then things like DNS may not work correctly
<rangerpb> specifically resolving the VMs DNS name <> IP
<Odd_Bloke> rangerpb: What do you mean by "the hostname is not set up correctly"?  When might that happen?
<rangerpb> meaning, the hostname is not what azure wants it to be
<Odd_Bloke> Ah, you mean if the network bounces outside of cloud-init's control?
<Odd_Bloke> (Rather than the intentional bounce on boot?)
<Odd_Bloke> I think it only needs to be configured as Azure expects when walinuxagent starts.
<smoser> Odd_Bloke, well, in the "builtin"  path, cloud-init is not currently bouncing the interface.
<smoser> the goal for rangerpb was to get the hostname from the environment file and then "publish" it to azure (where "publish" means "dhcp with set-hostname")
<smoser> which is why we were ever bouncing the hostname
<rangerpb> Odd_Bloke, sure, but frankly everyone wants off of the agent for provisioning
<rangerpb> and what smoser said
 * rangerpb runs out of gass
<smoser> powersj, how do i save the collect stuff ?
<smoser> so that i can verify --data-dir= ?
<smoser> magicalChicken, ?^
<powersj> smoser: instead of a run you want to do a collect
<powersj> collect -n xenial -d /tmp/collection for example
<magicalChicken> smoser: yeah, if you run collect you can then use verify afterwards on the collected data
<smoser> so just instead of 'run' i 'collect' ?
<magicalChicken> yeah
<smoser> i think i'd like an option to 'run' to take --data-dir
<magicalChicken> the rest of the args are the same, but you have to add --data-dir
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/tests.html#collect ;)
<magicalChicken> smoser: i have plans for adding a --preserve=always for run and a --data-dir option there too
<magicalChicken> just haven't gotten to that yet, too much other stuff first
<magicalChicken> and that would be based on the tmpdir stuff from the bddeb branch which needs to be merged as well
<smoser> powersj, i'm about to leave
<smoser> but
<smoser> http://paste.ubuntu.com/24191229/
<smoser> is what ihave
<powersj> smoser: thx I'll take a look at it
<smoser> hm.m
<smoser> test_shadow_passwords (tests.cloud_tests.testcases.get_suite.<locals>.tmp) ... ok
<smoser> '<locals>'
<smoser> powersj, that just passed a 'run' for me. so i think its good.
<smoser> the password -> passwd  change i found when one of my asserts failed.
<powersj> interesting is that worth a doc change?
<smoser> doc is right :)
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#set-passwords says password
<smoser> its correct there.
<powersj> I made all these test based off of the read the docs
<powersj> oh
<smoser> in a users: it is 'passwd'
<smoser> so.. based on that quite reasonable misundstanding of yours...
<smoser> maybe we should make the code take 'password' or 'passwd'
<powersj> ah right
<smoser> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#users-and-groups
<smoser> thats what you were writing to. 'passwd'
<smoser> i got to run.
<smoser> later
<smoser> .
<powersj> thx o/
#cloud-init 2017-03-17
<askb> Are there any known issues reading shell env variables before executing a script from the user-data ?
<askb> the script is unable to read the updated $PATH from /etc/environment ?
<amarao> Hello. Did anyone knows how to deal with random interfaces names? I'm trying to use cloud-init on baremetal instances, and there are many possible interface names (ens3, p4p1, e4s5, etc), and sometimes them just not 'up' for DHCP.
<smoser> amarao, hey.
<smoser> so there are a couple ways to do this.
<amarao> Hello.
<smoser> one way is to change the random interface names off
<smoser>  https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
<smoser> that gets you the old 'eth0' and the like.
<amarao> Yes, that's one way I though. Is any cloud-init-like way to say 'use all interfaces available'?
<smoser> the bad thing about that is that .... it was changed for a reason.  nothing was ever really guaranteeing that nics would come up named correctly, and even the udev renaming code can fail
<smoser> what do you want it to do ?
<amarao> To have such image, which would be able to boot (get metadata) regardless of the name of interfaces.
<smoser> some more background.. if cloud-init can find networking configuration from the "datasource" (the cloud provider ... whatever you want to call that), then it can use that and configure the system as described.
<smoser> this happens on openstack with config drive and in smartOS and in maas (there, more indirectly).
<amarao> Yes, but if that datasource is openstack metadata, it needs dhcp...
<smoser> so you are on openstack ?
<smoser> ironic ?
<amarao> Ironic, to be precise.
<amarao> Yep.
<smoser> and not (ubuntu 12.04 precise) to be precise.
<amarao> And I found that 4 my lab servers each has different names for their interfaces...
<smoser> i thought that at one point ironic deployment would provide a config drive or put the config drive data somehwere in the deployed sysstem.
<amarao> Nah. I want to run Xenial, Jessie, Centos7, and, maybe, some suse.
<smoser> is that not the case now?
<smoser> how is this "supposed" to work on ironic?
<amarao> smoser: Not exactly. There was a bug in ironic (until I report it) which prevented to usage of config drive with cloud init at all.
<smoser> so, with a config drive, it should work.
<amarao> https://bugs.launchpad.net/ironic/+bug/1656854
<smoser> 'unbound' means "not connected" ?
<amarao> I tried to use ConfigDrive, but it very unrelable. Older versions of cloud-init does not want to configure network from config drive. I was able to make it works with Xenial, but jessie is not good (cloud-init 0.7.6)
<smoser> yeah, you're going to need newer cloud-init
<smoser> sorry, but that just is what it is.
<amarao> And centos7 has no newver version at all... (+ 30Mb of python dependencies which come with 0.7.6->0.7.7). So I'm falling back to dhcp...
<smoser> amarao, thats a well written bug, thank you!
<smoser> where do the 30M of python dependencies come from ?
<smoser> probably in a python2 -> python3 change i suspect
<smoser> cloud-init can still run in python2
<smoser> but the packaging would have to be addressed to support that.
<smoser> i realize thats a lot, and you just want it to work
<smoser> amarao, so have you tried xenial with that unbound fix in place ?
<amarao> Yep. For jessie I was able to use backports, (with 0.7.7), but with centos7 I feel very exhausted...
<smoser> amarao, ^
<amarao> smoser: That bug (with 'unbound') is fixed now, and we has our fix in place (just replaces 'unbound' with 'physical' at metadata creation). I show it as illustration that 'ConfigDrive' is not an expected way for Ironic. When I asked guys at #ironic they said that it's unusual (to use CloudDrive).
<amarao> And there is one more problem. In ironic cloud drive is /dev/sda2 (partition at the end of the physical disk or raid array). And cloud-init docs says that CloudDrive should be whole disk, not partition.
<amarao> "Must be a un-partitioned block device (/dev/vdb, not /dev/vdb1)" http://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html#version-2
<smoser> amarao, if it has a correct label (config-v2) then it probably will work
<amarao> I can't say this is Ironic bug, because there is no way to create 'whole disk' out of nothing in hardware environment. (And it worked as /dev/sda2 for Xenial, anyway). So it looks like bug in docs for cloud-init.
<smoser> let me check, and i'd fix that in cloud-init.
<amarao> Should I report it as a bug for cloud-init?
<smoser> i'm pretty sure the doc is just wrong
<smoser> amarao, have you tried?
<smoser> amarao, i'm also interested in thoughts on how this realistically could work for bare metal without any locally provided information..
<smoser> what would cloud-init be expected to do?
<smoser> dhcp on all nics and attempt to load http://169.254.169.254/openstack on each one is going to be unreliable / timeout / racey
<smoser> i'm pretty sure reading current code that if the filesystem has a label of 'config-2', then it will work
<smoser> ie, does not have to be a full dis.
<smoser> disk
<amarao>  smoser It's a nice point about multiple DCHP....
<amarao> I'll report it as bug for cloud-init docs, I think.
<amarao> it == whole disk thing
<smoser> well,. please test and make sure i'm right or wrong :)
<smoser> amarao, sorry to pester you... where do you get the ubuntu image / how do you make it ?
<amarao> smoser: I build it myself.
<smoser> how?
<amarao> Here the main part:       DIB_CLOUD_INIT_DATASOURCES: "ConfigDrive, OpenStack"
<amarao> + cloud-init-datasources element.
<smoser> there is some tool that you use i guess ?
<amarao> diskimage-builder mostly.
<amarao> Plus dibctl to manage it.
<amarao> (The last one is self-made and under development at https://github.com/serverscom/dibctl)
<smoser> what pops out of diskimage-builder ?
<smoser> ie, what is the file format of the image ?
<amarao> qcow image. baremetal-ubuntu-xenial.img.qcow2
<smoser> and its a full disk image ? partitioned with bootloader? ?
<amarao> Ironic (with agent_ipmitool) write it to the baremetal node, and then writes a small ConfigDrive partition at the end (if nova boot used with --config-drive True).
<amarao> Yes, it's a whole image with grub inside.
<amarao> Elements: cache-url cloud-init-datasources dib-run-parts dpkg install-static package-installs ubuntu-minimal vm
<smoser> so agent_ipmitool does a dd of that qcow image (converted to raw) to hopefully the bios boot disk.
<amarao> ubuntu-minimal => debootstrap
<smoser> and then it edits the partition table to add a partition at the end of the disk ?
<amarao> Yes. It uses ipmitool to switch into PXE, and server is configured to use 1st disk to boot.
<amarao> I believe so, but I never looked into IPA (agent) code. But there is a /dev/sda2 inside instance with all metadata (including networking-data.json)
<amarao> Unfortunately networking-data.json is ignored by older cloud-inits :(
<amarao> <0.7.7
<smoser> amarao, yes it is.
<amarao> smoser: thanks, I got confirmation. I'll try to use ConfigDrive before giving up. Jessie probably would work eventually, centos is harder to do. If not, I'll dance again around dhcp-for-everyone...
<smoser> amarao, the cloud provider (ironic in this case) really needs to some how tell the system what the networking configuration is expected to be.
<smoser> amarao, i'd be pretty interested in knowing how this "builder" script works
<amarao> smoser: ... and it's a real pain... Ironic now even does not support multiple switches (physical networks).
<smoser> http://paste.ubuntu.com/24195576/ <--
<smoser> i'd suspect htat has a fair chance of being something that could be used.
<amarao> cloud-image-utils: /usr/bin/mount-image-callback
<smoser> ?
<amarao> I've just searched what is 'mount-image-callback'
<smoser> ah. yeah. its nice.  https://ubuntu-smoser.blogspot.com/2014/08/mount-image-callback-easily-modify.html?showComment=1453265690071
<amarao> smoser, I have other question about cloud-init. If I boot instance with ConfigDrive and dhcp, it writes into interfaces.d/ that interface should be dhcp, not the ip from networking-data.json. Is this a bug, feature or  just coincidence?
<smoser> "with configdrive and dhcp" ?
<smoser> if cloud-init does not find networking information (such as networking-data.json) then it will render "fallback" networking config that would be the equivalent of "dhcp on eth0"
<smoser> so.. .you could be hitting that
<smoser>  /var/log/cloud-init.log will have info
<amarao> I mean I boot instance with dhcp-enabled network and --config-drive=True. And it created file /etc/network/interfaces.d/50-cloud-init.cfg with content 'auto eno2  iface eno2 inet dhcp'
<amarao> pplying network configuration from ds bringup=False: {'version': 1, 'config': [{'name': 'eno2', 'mac_address': '18:66:da:5f:07:f5', 'subnets': [{'type': 'dhcp4'}], 'type': 'physical', 'mtu': 1500}, {'address': '8.8.8.8', 'type': 'nameserver'}]}
<amarao> 2017-03-17 12:14:39,022 - util.py[DEBUG]: Writing to /etc/network/interfaces.d/50-cloud-init.cfg - wb: [420] 408 bytes
<amarao> 2017-03-17 12:14:39,022 - main.py[DEBUG]: [local] Exiting. datasource DataSourceConfigDrive [net,ver=2][source=/dev/sda2] not in local mode.
<amarao> It sounds like ConfigDrive which created dhcp interface...
<amarao> *dhcp settings for interface.
<smoser> i'm confused.
<smoser> it soulds like you cloud-init did what it should have.
<smoser> no?
<smoser> that ll looks like working to me
<smoser> unless you're saying 'auto eno2  iface' is on one line
<smoser> amarao, ^ ?
<amarao> I just shorten it up.
<amarao> I thought about it, it looks reasonable to me. If network is 'dhcp' type interfaces should be dhcp.
<amarao> (My initial expectation was that if ConfigDrive is enabled, it always would be a static ip, but I was wrong).
<smoser> ah. right. as you showed there, the network config provided (if you snow network-data.json it'd be good). but it certainly looks like network-data.json told cloud-init to dhcp on that nic.
<amarao> network was '--enable-dhcp' type, it had worked fine (until I deleted it just now). But I'll try now to use ConfigDrive with static IPs (no dhcp). It's my actual goal and the best how it may work for us.
<amarao>  smoser, btw, where is a bugtracker for cloud-init? I tried to find one and I failed.
<rharper> https://bugs.launchpad.net/cloud-init
<rharper> amarao: ^
<amarao> Oh, thanks.
<powersj> magicalChicken: around?
<powersj> magicalChicken: nevermind, figured it out :)
<paulmey> rangerpb, Odd_Bloke: I'm going to figure out if the hostname reported matters in any way
<paulmey> I don't think it should, since it's all keyed off of the container ID
<paulmey> the wire server knows which VM is talking to it and should attribute the status update accordingly
<paulmey> (otherwise, it's time to file some bugs here...)
<Odd_Bloke> paulmey: Thanks!
<smoser> powersj, i dont understand
<smoser> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/319878
<smoser> when i ran with the .py files input, it ran my tests.. ithought
<powersj> smoser: double check because when I ran it wan exactly 0 tests for those two
<smoser> PasswordListTest inherits from CloudTestCase
<powersj> https://paste.ubuntu.com/24195907/
<smoser> so why would you need
<smoser>  TestPasswordList(base.PasswordListTest, base.CloudTestCase):
<smoser> explicitly ?
<powersj> I believe it has to do with where we are running nose from and how nose discovers tests
<smoser> agreed on the 'base' filling up, if we do have that problem, we just move that to 'passwordtests.py' or something.
<smoser> i dont understand.
<smoser> why would it not think this needs testing:
<smoser> http://paste.ubuntu.com/24196107/
<powersj> I didn't try a pass hmm
<smoser> that has 'Test' in its name, and extends (indirectly) from UnitTest.TestCase
<smoser> so... i dont know why it would not liek
<magicalChicken> smoser: discovery isn't done via the unittest loader directly
<magicalChicken> smoser: because we have to be able to get the data and cloud-config into the testcase's attributes
<magicalChicken> see tests/cloud_tests/testcases/__init__.py
<magicalChicken> i agree that it would be better for just base.PasswordListTest to work, i can see if i can extend the way i detect inheritance from base.CloudTestCase to handle a test class indirectly inheriting from there
<magicalChicken> i can't create instances there though, since that has to be done via the unittest loader, and it is a bit more difficult to do that on types
<smoser> magicalChicken, i dont understand though.
<smoser> i dont understand why powersj's' change made any difference
<magicalChicken> because CloudTestCase needs to be a base for the individual test class
<magicalChicken> the way it is currently written it doesn't recurse through base classes to check, so it doesn't realize that PasswordListTest inherits from CloudTestCase
<magicalChicken> i can modify it to be able to detect that though, once i finish up debugging centos tests
<smoser> hm.. where is that check ?
<smoser> you didnt' just check isinstance ?
<magicalChicken> it isn't possible to use isinstance there because its just the class type, not an instance
<magicalChicken> its in tests/cloud_tests/testcases/__init__.py on line 23
<smoser> ah
<smoser> magicalChicken, http://stackoverflow.com/questions/1401661/list-all-base-classes-in-a-hierarchy-of-given-class
<smoser> getmro ?
<magicalChicken> smoser: nice, i didn't know inspect had that
<magicalChicken> should be easy to switch over
<magicalChicken> i can throw together a separate mp for that real quick
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320233
<smoser> now has tests
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/319943 i'd like your thoguths as if its ok to take that ... perhaps we should just open a bug to handle the _write_network path... i will do that
<smoser> and then i hitnk that one is good
<smoser> and  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320212 too
<magicalChicken> smoser: i got the mp for switching to inspect.getmro() in if you want that in for the password list test
<magicalChicken> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/320234
<smoser> you commit message is odd
<smoser> in english sometimes we use a '.' to separate sentences. not only the '- '
<smoser> :)
<smoser> ie, your list there could just be a paragraph
<magicalChicken> smoser: right :)
<magicalChicken> i think almost all of the integration-testing commits are formatted like that
<smoser> well, when you do a lot of things in a single commit or MP, it makes sometimes sense to list what you did
<smoser> but when you do one thing , a list isnt necessary
<magicalChicken> yeah, this is a 1 liner though
<smoser> i'll pull ;)
<magicalChicken> thanks
<smoser> powersj, ^ now you should be able to simpllify your test like i suggested
<powersj> smoser: ok will take a look. The getmro find is cool, never heard of that
<powersj> smoser: updated
<magicalChicken> powersj: have you looked into building rpms for integration-testing on centos?
<powersj> magicalChicken: no I have not
<magicalChicken> i think the only remaining test failures in teh distro-features branch are just because i only have a 0.7.5 rpm
<magicalChicken> powersj: oh, ok
<magicalChicken> i've tried to get rpmbuild working but there's something wrong with how i did the spec file
<powersj> magicalChicken: did you see how brpm works?
<magicalChicken> powersj: yeah, i'm basing my spec file off the template it uses, and the call to rpmbuild looks good
<magicalChicken> its failing the install stage though becasue the default config files aren't where it expects
<rharper> smoser: ok, reviewed both, approved second
<smoser> rharper, ok. thanks. responded to second, will make some changes.
<rharper> cool
<smoser> er... to the one that needed fixing
<rharper> right
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320212 <--
<smoser> that is pretty simple.
<smoser> other than being arguably not necessary, it seems safe
<rharper> smoser: +1
<smoser> rharper, updated https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320233
<rharper> k
<rharper> smoser: so does lp not update the MR with the new diff?  I'm not seeing the new changes yet
 * smoser adds --force
<smoser> rharper, reload
<rharper> y
<rharper> smoser: missing the execption for none found
<rharper> and the unittest of that
<rharper> smoser: one other comment in MR;  we should use /etc/network/interfaces.d as it's owned by ifupdown package (where eni is not since it's volatile )
<rharper> w.r.t verifying paths in eni.available
<smoser> but we dont require /etc/network/interfaces.d
<smoser> if its not there and the renderer shoudl be used, then it will work fine
<rharper> true; but it's going to be present if ifupdown is installed
<rharper> no
<rharper> we will recreate the path
<smoser> right.
<rharper> but we expect ti to be there
<smoser> which is fine.
<rharper> the facthat we recreate it is someone bogus
<rharper> somewhat
<rharper> it's part of the package; we're using dir paths to avoid doing package lookups
<smoser> i dont know. we dont need it to be there, and the fact that it is there does not give us any more information as to whether or not it should be used.
<smoser> where as lack of /etc/network/interfaces indicates that at least currently, it will not work.
<rharper> well it *should* be there if you have ifupdown installed
<rharper> which image would we encounter if it's not there?;  its the same for the sysconfig paths; they might not be there (and maybe sysconfig render also does extra ensure dirs which we should have to)
<rharper> in many cases we "fix up stuff" and others we "raise exceptions"; I can see this either way;  so I won't push any more;
<smoser> http://paste.ubuntu.com/24196962/
<smoser> directories owned by packages are wierd.
<smoser> especially empty ones.
<rharper> yeah
<smoser> dpkg itself doesnt even care that i deleted it
<rharper> well, dpkg likely has a bug
<smoser> no. it really doesnt track directories
<rharper> because if you query what owns the dir it tells you
<smoser> its only there for cleanup
<rharper> dpkg -S clearly does
<smoser> when you removve it if the directory is empty it will clean it up
<rharper> so, inconsistency/bug ..
<smoser> multiple things can own a directory too
<smoser> the same one and no conflicts
<rharper> which path does dpkg -S return a list of multiple packages?
<rharper> oh
<rharper>  /etc
<rharper> awsome
<smoser> :)
<smoser> yeah.
<smoser> it just means "create on install, remove if empty on removal"
<smoser> at least that is my experience.
<rharper> well, then in general I'm not so sure we should be checking paths at all then
<rharper> maybe in the sysconfig available we can check other binaries that present but not elsewhere
<smoser> well, we're trying to check "is this the used network tool"
<rharper> right
<rharper> I don't think we have a good way to do that reliably
<rharper> these are just guesses
<smoser> the path i check in sysconfig *is* a binary
<rharper> oh, yes we have some
<smoser> etc/sysconfig/network-scripts/ifdown-eth
<smoser> is effectively a binary.
<rharper> I just meant we dont have a way to ask "OS, which networking configuration system are you using"
<smoser> so... the goal here is to determine "is this thing the network tool for this system".
<rharper> centos/fedora likely have both sysconfig and network-manager (much like we have in UC16)
<smoser> right
<rharper> or even sysconfig/networkd/network-manager and we won't really know
<smoser> so i think over time we'll just improve this.
<rharper> y
<smoser> as it is right now, it does what it did before
<smoser> unless you try to trick it
<smoser> hm...
<smoser> thinking of my desktop
<smoser> it uses both ifupdown and network manager
<smoser> (or at least it could)
<smoser> wheeee!
<rharper> yeah
<smoser> rharper, so we dont write /etc/network/interfaces
<smoser> right ?
<smoser> currently, we expect it is there.
<smoser> and that it has 'source /etc/network/interfaces.d/' in it
<smoser> and bah. i had the other test case you asked fro locally
<smoser> smoser git fail today
<smoser> rharper, you're still working on https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/319259
<smoser> ?
<rharper> smoser: yes, waiting for your two changes to land in master and then I'll rebase
<rharper> stepping out for 30 mins, have to go pickup the kids; then I'll finish up the rebase
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320233
<smoser> rharper, ^ quick ...
<rharper> y
<smoser> can you click agree if you do ?
<smoser> then i'll pull
<smoser> if you dont thats fine too, but i thoguht we were in agreement at this point
<rharper> yes
<smoser> suck.
<rharper> bbiab
<smoser> go now read when you return on that link... i'll put some text there
<smoser> rangerpb, around ?
<smoser> in the case where you use NetworkManager to bring up networking
<smoser> rharper, just pushed another commit on top of that branch... woudl like your feedback when you see it.
<rharper> smoser: ok
<rharper> smoser: so, I'm for (a) since it means we don't touch stages.py ;  that said, this is effectively the same as just having a hard-coded default in the distro
<rharper> I had suggested that the distro class itself could set a default policy is none is found;
<smoser> rharper, well i implemented something like a'
<smoser> in  my commit
<smoser> but with  more information
<rharper> ok
<rharper> I'm find with that;  we can drop the exception for hardcoded default in the distro if we wanted to later
<paulmey> rangerpb, Odd_Bloke: it seems that on Azure, the intent is that the instance DNS name is equal to the hostname set on the VM
<paulmey> the waagent publish_hostname functionality is proof of that
<paulmey> so the "with temporary_hostname(...)" code is not necessary if the hostname has been set correctly already
<paulmey> at least, that's how I interpret the intent of https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n77
<paulmey> and its usage here: https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n120
<Odd_Bloke> paulmey: When you say "correctly", you mean "set to the same as the DNS name", right?  This code was introduced because people sometimes want a hostname on an instance that is different to the hostname that Azure expects an instance to have.
<paulmey> if it was ever the case that Azure cared about the hostname, that is no longer so
<paulmey> Azure provides the hostname that people put in at VM creation time through the ovf file
<paulmey> but people are free to change that
<paulmey> the agent has code to detect a hostname change and bounce the interface to update DNS
<paulmey> but that is just a convenience
#cloud-init 2018-03-12
<do3meli> good morning - can i get a CI run against https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340757 ?
<ybaumy> smoser: online?
<dojordan> @blackboxsw, sorry to bother but gentle bump on my MP: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
#cloud-init 2018-03-13
<smoser> ybaumy: here now
<ybaumy> smoser: you talked to psyrabbit lately about the vmware ovf dataprovider problem right?
<ybaumy> the ovf just doenst work
<ybaumy> cause we looked at the code and found that you commented out few lines in datasource provider.py that were responsible for vmguest tools
<ybaumy> those just dont exist anymore
<ybaumy> but we found a way to make vmware + cloud-init happen maybe
<ybaumy> just a sec i will get the links
<ybaumy> have to logon to work
<ybaumy> https://code.vmware.com/web/dp/tool/ovf/4.1.0
<ybaumy> https://blogs.vmware.com/vsphere/2012/06/leveraging-vapp-vm-custom-properties-in-vcloud-director.html
<ybaumy> https://www.virtuallyghetto.com/2011/01/how-to-extract-host-information-from.html
<ybaumy> if you have time next week in the meeting we would like to participate because of that matter
<ybaumy> psyrabbit maybe interrested in extending or writing a datasource provider for vcloud
<ybaumy> we need that for work anyways
<ybaumy> https://github.com/xing/cloudinit-vmware-guestinfo oh here another link
<dpb1> ybaumy: is this applicable to vsphere/esx at all, or just vcloud?
<ybaumy> dpb1: we need vcloud .. so thats what we looked at yet
<dpb1> ok
<dpb1> it's fine, was just curious
<ybaumy> but it would be nice to get some support from you guys if we do that ..
<ybaumy> also the question of maintaining is open for us
<ybaumy> initial development would be ok i think
#cloud-init 2018-03-14
<bemis> I asked a few days back but didn't get any response; trying again - are there any centos/rhel/oel/sl users in here?
<setner> Hi everyone. I am trying to understand the differences between ansible and cloud-init. Why do people tend to use cloud-init in conjunction with ansible? Why not just ansible or just cloud-init? Can ansible and cloud-init perform the same actions?
<setner> Can anyone help me with this question: what can cloud-init do that ansible (for example) can't do?
<smoser> ybaumy: if you could open a bug with details of what is not working we can take a look.
<smoser> setner, cloud-init is often times the first thing that is used, and its in the image already.  It helps bootstrapping.  it will set up a user and import ssh keys that may then be used by ansible.
<setner> smoser: ok, it helps on the bootstrap process. But isn't that also achievable with ansible?
<Odd_Bloke> bemis: There are people around; if you have a specific question, it's probably best to ask it so they can respond when they are around.
<smoser> setner: i dont know how you would do that with ansible
<smoser> take a stock image and get your ssh keys into it
<bemis> Odd_Bloke: :) fair enough, thanks
<bemis> specifically i'm looking for help getting 18.1 packaged for rhel6/rhel7
<bemis> and hoping that perhaps others have walked this path before me and can guide me around the jagged edges
<ch007m> Does one of you know how we can disable biosdevname when we use cloud-init / user-data config file ?
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341349
<smoser> could you take a quick look at that ? simple-ish removal of duplicate code.
<smoser> bemis: we have daily builds of trunk in copr
<smoser>  at https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<smoser> ch007m: biosdevname kind of happens too late.
<smoser> you can build your imgaes with changes to disable it.
<ch007m> So we can't define bootloader as we can do with kickstart then - https://goo.gl/PUjERh ?
<ch007m> As `packages:` will install yum packages the first time the vm will start, how can we know that all the packages defined are installed ?
<smoser> ch007m: they'll be installed when cloud-init finishes.  you can look at /run/cloud-init/result.json to see the result.
<smoser> is that what you're asking ?
<smoser> you can't change the bootloader in an image for the first boot, cloud-init runs after that.
<ybaumy> smoser: its not a but .. it wont work the way it was working years ago anymore since those features you were using dont exist in the commands anymore. things changed
<ybaumy> s/but/bug
<smoser> ybaumy: i need more information. I'm really not sure what you're talking about. Please file a bug and explain clearly what you were expecting to work and what is not working.
<ybaumy> smoser: doesnt really matter what the current status is. its not what we need. psyrabbit is going to try to implement a new datasource provider for vcloud
<ybaumy> this is what we should focus on
<smoser> blackboxsw: so on your snap branch
<blackboxsw> yes all ears
<blackboxsw> just got through your jsonschema branch. landing it now
<blackboxsw> post a tox run
<bemis> smoser: whoa, thanks!
<smoser> blackboxsw: i just hit 'submit' there.
<ybaumy> smoser: we hope .. we can get a little support from you guys when he is implementing the datasource for vcloud. we think we found a way to handle data transfers
<blackboxsw> smoser: checking
<blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341349 merged
<bemis> smoser: did the project drop EL6 support at some point between 0.75 and now?
<dpb1> ybaumy: support time will be limited over the next couple weeks
<dpb1> ybaumy: we have a release we are pushing for end of month
<dpb1> ybaumy: but we will do what we can
<ybaumy> dpb1: ok this is not a "no" for an answer and thats what we need
<dpb1> ybaumy: ok
<dpb1> bemis: it likely still works, are you seeing something broken?
<bemis> dpb1: i wasn't finding el6 builds in the copr link pasted - i've since found them and am doing some testing now
<dpb1> ah ok
<dpb1> bemis: it's not something we stay on top of that much, but would be an interesting data point
<dpb1> (tip on el6)
<bemis> what led to the join initially was the conditional logic around systemd was removed at some point and the service file required components that aren't in el7
<bemis> so i thought maybe the el landscape was an afterthought
<ybaumy> dpb1: we have no timeline set on the project so its no problem if help is 1 day apart
<dpb1> bemis: not an afterthought, but I'm not of how widely used it is at this point in the lifecycle of 6. :)
<dpb1> so there are likely gaps
<bemis> dpb1: that's fair - i'll make sure to update here as my project progresses then :)
<bemis> thank you
 * dpb1 nods
<blackboxsw> smoser: thanks for the snap config review comments. I'm inclined to add the error exception message as you suggested instead of just listing the failed command.  I'm also hesitant to move the instance-data.json tests into base because we run them on every test case, increasing the cost of each of our tests by another ssh roundtrip to pull down that file & validate the same content
<dojordan> @blackboxsw, sorry to bother but can you take a look at my MP? https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
<blackboxsw> dojordan: hiya, sorry about my silence earlier, I was on vacation monday/tuesday
<dojordan> ah no worries!
<blackboxsw> I'm digging out of my backlog and hopefully can get to that branch soon. It'll be reviewed this week for sure
<blackboxsw> I had one exception path I wanted to test dojordan, connection not present versus, path not present on existing service(404)
<blackboxsw> we thought maybe there was a better way to handle things since you are dropping the timeout
<blackboxsw> dojordan: like maybe using readurl instead of wait for url
<blackboxsw> but I wanted to test that before giving you the suggestion
<blackboxsw> might as well say that now so you could glance at that as a potential solution
<dojordan> sure, thats a good idea
<blackboxsw> yeah it just seemed clunky that we were using wait_for_url there (which calls readurl and actually has the content of the url but ignores it only to return the endpoint's URI). So your branch would effectively be calling readurl twice on success path, once in wait_for_url to see that the service is up and once to actually recapture the content
<blackboxsw> that wait made sense when the timeout was larger, but not so much anymore :/. Ok thanks for glancing at the potential
<dojordan> right- we were actually discussing this recently. the other option is make wait for url return both success and optionally content but read_url seems cleaner
<dojordan> if the connection is not present we would re-DHCP anyway as it would still bubble up to a url error
<blackboxsw> yeah I thought about the same too (augmenting wair_for_url), but it looked like we were just using the wrong tool for the job.  wait_for_url was really only meant to iterate through a list of potential metadata URLs to return the first one that responds. smoser and I had talked about the potential of dropping the wait_for_url function at some point in the future since there are only a couple of callsites that
<blackboxsw> could be consolidated into a simpler helper. so something that uses readurl directly would make things simpler if we drop wait_for_url.
<dojordan> cool, ill make the change
<blackboxsw> good deal thanks dojordan
<ch007m> when I log on the first time to the vm, some packages are still installed as the yum process is locked -> https://www.dropbox.com/s/dh0zewvgpy0o80y/Screenshot%202018-03-14%2018.31.24.png?dl=0
<ch007m> Is there a command that I could pass or use to know what are the packages currently downloaded for installation in order to know the status of the progressions ?
<smoser> ch007m: probalby some 'ps' or look at /var/log/cloud-init-output.log
<ch007m> this line is recorded -> "2018-03-14 17:30:54,672 - util.py[DEBUG]: Running command ['yum', '-t', '-y', 'install', 'git', 'docker', 'openssl', 'net-tools', 'NetworkManager'] with allowed return codes [0] (shell=False, capture=False)". When that will happen, I suppose then that all the packages will be installed then ?
<smoser> when it finishes yes. its possible that it is locked/blocked for some reason
<ch007m> When is cloud-init executed exactly during vm bootstrapping ?
<smoser> ch007m: fairly late.
<smoser>  cloud-init-final.service
<smoser> which is After=multi-user.target
<ch007m> +1. Thanks.
<ch007m> I suppose that OOTB, it is packaged within this image (CentOS-7-x86_64-GenericCloud-1802.qcow2c) ?
<ch007m> My centos qcow2 includes this version `cloud-init 0.7.9` which isn't the latest. Have you discussed with Centos team to do a an upgrade ?
<dpb1> ch007m: no, not really
<dpb1> ch007m: you'd need to get in touch with one of the rhel disto guys to see what their plans were
<ch007m> ok
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341417
<okie> good morning everyone
<okie> actually afternoon
<okie> i got a question if anyone can assist. I am running a centos 7 aws image and having a problem with cloud-init it looks like. I have consistant network device naming turned off but cloud init is not respecting that. I can advise it is cloud init as there are ifcfg-ens* entries with a comment in each of them advising this was automatically generated by cloud-init
<okie> how do i get cloud init to respect that i have disabled consistent network device naming.
<smoser> okie: cloud-init doesnt really do anything with that. what is the problem you're trying to solve ?
<smoser> then also, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341420
<blackboxsw> heh, smoser I'm circling the unit test drain on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/339720
<okie> so right now device names are using predictable network device naming or otherwise known as consistent network device naming... So instead of eth0 i have ens{insert number}. I want to go back to the "traditional" way which is eth0. I have followed the redhat docs on how to do this but ens entries are being generated by cloud-init
<okie> grep ^ /etc/sysconfig/network-scripts/ifcfg-*
<okie> #/etc/sysconfig/network-scripts/ifcfg-ens3:# Created by cloud-init on instance boot automatically, do not edit.
<okie> #/etc/sysconfig/network-scripts/ifcfg-ens5:# Created by cloud-init on instance boot automatically, do not edit.
<smoser> okie: you may possibly need to clean up a bit.
<smoser> your image.. if its not pristin.
<okie> its a base centos7 image from aws market place
<smoser> in most cases, c loud-init reads the devices rthat it sees asa they are. and then configures networking, and ensures that the names it writes are right.
<smoser> can you post a /var/log/cloud-init.log ?
<okie> ya
<smoser> https://hastebin.com/
<smoser> is wonderful (before you went to pastebin.com)
<okie> https://hastebin.com/onekedolod.sql
<okie> file was too big to save in hastebin so you got part of the cloud init log
<smoser> oh. really ? it said too big ?
<okie> yuppers
<smoser> well then, feel free to paste the whole thing somewehere else
<smoser> paste.ubuntu.com is good
<okie> way to go ubuntu pastebin.. that works
<okie> https://paste.ubuntu.com/p/YbT9RWQ8zm/
<smoser> yeah, but in ubuntu pastebin you have to log in to get raw
<smoser> okie: when cloud-init ran it saw devices named 'ens3'
<smoser> so i thii if you were expecting to get 'eth0' device, then you must have done something wrong.
<okie> where is it looking sysconfig/network-scripts/ or where?
<okie> that it is determining this.. as the base centos7 image is disabling the udev rule to use consistent network device names..
<okie> i dont get where cloud-init is pulling ens3 from?
<smoser> okie: it looks in /sys for the names.
<smoser> and then writes config making sure that they stay with those names.
<okie> thanks
<dojordan> @blackboxsw, pushed the changes: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
<blackboxsw> smoser: pushed the unit test coverage for set-hostname-early branch https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/339720. I'm done now with that and hitting your reviews
<blackboxsw> thx dojordan\
<okie> @smoser thanks for the help... It was helped me in the right direction
<smoser> great!
<blackboxsw> smoser: minor nit on your doc branch https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341420 and approved
<blackboxsw> smoser: I think 46cb6716c27d4496ce3d2bea7684803f522f277d broke util.subp
<blackboxsw> it treats args as a list always, which isn't true, so any pure string passed to subp will fall apart
<blackboxsw> as it gets treated as an iterable character by character
<dojordan> (sorry to but in) but that's what broke get_hostname on azure
<blackboxsw> dojordan: yeah, I recall your comment on how that branch shouldn't have passed CI.  yeah, just hit it on my branch too (where I provide strings to subp instead of lists too). I figured your fix was still safe/good practice etc.
<dojordan> One possibility is to add an assertion on the type being passed into util.subp?
<blackboxsw> dojordan: yes, you are correct. I just looked through that subp encoding changeset in more detail to see what the intent was (instead of blazing past it).  And sure enough
<dojordan> Also, this is me just rambling, but at some point type hints could help prevent things like this...
<blackboxsw> true, I think thanks probably a nice feature, to catch other potential programming errors on what gets passed into subp. though subp should support both string and list
<dojordan> true, it could just construct a 1 element tuple if need be
<blackboxsw> rambling welcome. and preaching to the choir ;)    Yes I want generally to have strict schema validation on all user-provided content from cloudinit's config modules which would both document expected types and explicitly warn/error on invalid types.
<dojordan> *dojordan swoons*
<blackboxsw> it's a long road on that front, but handling that on common utility functions like subp is a good start to prevent misuse of core functions/methods
<blackboxsw> "cloud-init devel schema --doc"
<blackboxsw> only currently about 7 examples of schema types
<blackboxsw> or locally in your repo directory: python3 -m cloudinit.cmd.main devel schema --doc
<dojordan> nice, its a good start
<blackboxsw> gotta start somewhere :)
 * blackboxsw heads out for a bit. checking in later
#cloud-init 2018-03-15
<blackboxsw> smoser: for tomorrow https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341437 a cut at the subp fix to allow for string as a command
 * smoser thinks we should format code with isort
<smoser> https://pypi.python.org/pypi/isort
<ch007m> Hi
<ch007m> Can we define the version of the package to be installed under `packages:`?
<smoser> ch007m: possibly.
<smoser> packages are installed from the distro
<smoser> so.. for example in apt/ubuntu, you could do:
<smoser>  packages: ['mypackage=1.2']
<smoser> but that assumes taht there is a 'mypackage' at version 1.2 in your repos
<smoser> if you had ppas or somethign that would work, but generally speaking ubuntu doesnt have 2 versions of the same package in its archives.
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341465
<blackboxsw> smoser: you can try landing that
<blackboxsw> since review-mps runs tox, I'm not concerned about waiting for ci
<smoser> k
<blackboxsw> hrm something failed integration testing of my subp branch twice in a row last night. I can't reproduce the failure locally with:   tox -e citest -- run --verbose --os-name xenial --test  modules/set_password_list.  I'm kicking the builder again since we know some services were under maintenance last night
<do3meli> smoser: you may have some spare mins to look at https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340757 ? ci run completed successfully.
<smoser> do3meli: that looks fine. I dont like the long line though.
<smoser> not sure what to do with it.
<smoser> maybe just: See rc.conf(5) in FreeBSD 11.1
<do3meli> smoser: hmm that would be possible. it's the same whay in rhel.py distro class...
<do3meli> *way
<smoser> see comment in MP
<smoser> just change the comment i think to make shorter.
<do3meli> smoser: fine for me. will do right now.
<do3meli> as it is not version specific i will leave that away. ok?
<smoser> sure. you can drop the version
<smoser> you had that inherently in the link
<smoser> so i left it
<do3meli> smoser: perfect, i have commited it. ready for setting it to approve ;-) thank you
<do3meli> (local tox run was good)
<do3meli> thanks smoser. wish you guys a restful evening. cheers
<ch007m> @smoser. thanks for the answer
<smoser> blackboxsw: commented on subp branch. just a suggestion. feel free to pull either way.
<smoser> blackboxsw: i'd like a review of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/340140
<blackboxsw> smoser: just left comments on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341417 to see what you think
<blackboxsw> grabbing your other branch
<blackboxsw> I know apt retries is a bit of a hot button issue :)
<smoser> blackboxsw: hm.. that should be fixed
<smoser> the apt thing.
<smoser> there it is
<smoser>  6f2aaf7b0fa9fd9376561e9b6233b4d36de51da1
<blackboxsw> you mean apt should fix it, or cloud-init should work around it
<blackboxsw> checking
<blackboxsw> git push :) I'm still on d7e5385
<smoser> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1693361
<ubot5`> Ubuntu bug 1693361 in APT "cloud-init sometimes fails on dpkg lock due to concurrent apt-daily.service execution" [Unknown,Confirmed]
<blackboxsw> ohh I see, then this is a symptom of another problem
<smoser> or it wasnt correclty fixed.
<blackboxsw> each cloud-init clean --reboot --logs  traces on "apt update" on a xenial container when providing #cloud-config\ndrivers:\nnvidia: {license-accepted: true}
<blackboxsw> without some sort of retry on CalledProcessError. I could try emitting ps output to see what else might have apt locks
<blackboxsw> took your suggestion on fix-subp branch
<blackboxsw> awaiting my local tox run before merging
<blackboxsw> simpoir: I just rebased the feature/snap -module branch to pull in the util-supb fixes so unit tests will now pass there
<blackboxsw> I didn't have to resolve any conflicts in the rebase so no changes to what you are reviewing, but thought you should know
<blackboxsw> as unit tests locally wouldn't run until the rebase fixed it
<simpoir> blackboxsw: ok
<simpoir> blackboxsw: I was wondering about those
<smoser> blackboxsw: your mrege tool should comment on the bug (possibly in addition to the merge proposal)
<smoser> or instead of. dont know.
<blackboxsw> smoser: you mean on merge, the bug comment should point to the commitish as committed in trunk?
<blackboxsw> I think you mentioned that last week, and I failed to add it
 * blackboxsw has to grab lunch back in a bit
<smoser> hm..
<smoser> no. you added it so it adds a comment.
<smoser> blackboxsw: well, it makes a comment on the merge proposal
<smoser> but does not comment on a linked bug
<smoser>  https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341437
<smoser> is an example.
<smoser> i'd like for the bug to also get a comment. possibly instead of the merge proposal, as the merge proposal does get a state change event.
<smoser> blackboxsw: ping when around
<blackboxsw> smoser: hangout?
<blackboxsw> simpoir: I think we are going to try wrapping up the snap module for inclusion in cloud-init if you have a chance to wrap up a review there https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/338366
<blackboxsw> smoser: bionic MP https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341482
<blackboxsw> I was awaiting snap module, but I don't think it'll make it by EOD
<smoser> This commit fixes get_hostname on the AzureDataSource.
<smoser> we need to be better when reading messages.
<smoser> self referencing in the subject :)
<smoser> instead just
<smoser>  Fix get_hostname in Azure
<blackboxsw> ahh I see s/This commit fixes/Fix/   gotcha. makes sense too noisy +1
 * blackboxsw and "too noisy" are synonymous
<simpoir> blackboxsw: added comments on MP; mostly typos and small things that may be symptomatic of my lack of knowing the codebase
<blackboxsw> +1 simpoir I see review comments on snap stuff. fielding them now thank you
<blackboxsw> simpoir:.... "lack of knowing the codebase"  welcome to my life.  :)
 * blackboxsw might have a fixup to handle for containers on this branch as a result of my testing.  simpoir and I left you without a leg to stand on because I didn't give testing instructions
<blackboxsw> sorry about that. I'll add them for reference now in the description
<smoser> blackboxsw: ok. uploaded
<smoser> and i fixed my branch for drivers i think
<smoser> pylint really being a pain.
 * blackboxsw sits on artful & xenial dev environments. I need to move to a bionic container/base system for development so I can feel the pain too
<blackboxsw> thx smoser
<blackboxsw> ok so we can queue drivers and snap modules for tomorrows bionic build. I might have time for review of some of the other outstanding branches  too.
<blackboxsw> my goal is to get the ubuntu-advantage module up for your review tomorrow
<blackboxsw> simpoir: thanks again, I finally added testing instructions that would have helped a bit. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/338366 sorry 'bout that
<simpoir> blackboxsw: were your testing instructions to run tox && make deb && throw in a container and call manually with a random snap like lolcat?
<blackboxsw> yeah just that. minor sneakiness with user-data seeding
 * simpoir feels proud for figuring it out
<blackboxsw> I'm sure you did all that. but, it would have helped me, and I just lacked the consideration making you work hard for it
<blackboxsw> :)
<blackboxsw> lolcat.... good choice ;)
<simpoir> did not know know about the clean command. I'll keep that in mind
#cloud-init 2018-03-16
<AscII> smoser: regarding https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init/+merge/338439
<AscII> you asked if the local-hostname is always present
<AscII> we do generate or enforce a valid hostname in the API/frontend
<AscII> so it should be present
<do3meli> smoser: re your comment in https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340220 - am i seeing that right that for resizefs module no unit tests at all exists so far?
<smoser> do3meli: :-(
<smoser> probably
<do3meli> i found some in the meanwhile. but not a lot...
<smoser> AscII: could you put a comment in coede to that affect ?
<smoser> thanks.
<AscII> sure
<AscII> smoser: done and pushed
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341530
<smoser> and want to chat on ntp ?
<blackboxsw> smoser: probably 10 mins for me. wrapping up unit tests on ubuntu-advantage
<smoser> k.
<blackboxsw> I'm sitting in hangout, but will be distracted until then
<dojordan> hey @blackboxsw, any chance to take a look at the azuretimeouts branch? https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
<blackboxsw> dojordan: nope, but it's 2nd in my queue today so it'll definitely get reviewed today
<dojordan> :) thanks for the update
<bemis> hey, google is failing me here - is this an expected error? https://paste.fedoraproject.org/paste/8KirXN0pTDwsFOlzzlaZOA
<bemis> (AttributeError: 'Namespace' object has no attribute 'action') for those that don't wish to click
<bemis> output from cloud-init-18.1 on centos6 built from copr url linked in here earlier this week (i went back to 18.1 release day so no git-tag)
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341542
<blackboxsw> smoser: simpoir ubuntu-advantage module is  up for review
<blackboxsw> bemis: +1 on that, it is a valid bug.... I can sort that next week
<blackboxsw> bemis: if you get a chance can you file a bug for it? https://bugs.launchpad.net/cloud-init/+filebug
<simpoir> blackboxsw: oh, good! I was wondering where this was
<blackboxsw> simpoir: yeah just had to sort unit tests
<blackboxsw> always takes me a while
<blackboxsw> I'll add a testing description now (but basically the same thing as the snap module
<blackboxsw> simpoir: oops and I need to set prerequisite branch one min
<bemis> blackboxsw: thanks - another seems to be that 18.1 can't detect OracleLinux's distro (there's inconsistency between whether it returns "none" or "ubuntu")
<bemis> i'll file a bug once i have some real data on that
<blackboxsw> simpoir: smoser resubmitted the branch https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341543
<blackboxsw> thanks bemis for the heads up there. yeah a bug will give us something actionable.
<blackboxsw> smoser: per lint warnings that you recently fixed:  did you see this one? cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()
 * blackboxsw needs to rebase I think
<smoser> blackboxsw: i have not seen that.
 * blackboxsw lands snap-module branch
<blackboxsw> and smoser 's pylint branch
<smoser> AscII: hostname may be fqdn and also may not be ?
<smoser> blackboxsw: ugh. deprecated method
<smoser> are you fixing that ?
<smoser> what started that.
<blackboxsw> smoser: pylint picked it up today afaict. I landed a workaround in snap module
<blackboxsw> to ignore the warning on readfd
<blackboxsw> but I think we need a helpers fix
<blackboxsw> smoser: that's in cc_puppet right?
<blackboxsw> or somewhere else?
<smoser> why did pylint start seeing it ?
 * blackboxsw thinks it could have been seeing it for a while, I just today ran tox -r -e pylint to rebuild dependencies before linting 
<blackboxsw> I'll check on the origins
<smoser> hm.
<smoser> http://paste.ubuntu.com/p/3j6QNh8Q9j/
<smoser> bugger. you ruined my tox output
<blackboxsw> gah fixing unmocked stderr
<smoser> blackboxsw: i thin the warnings are a new pip version.
<smoser> as the hetzner is starting to show it, and it didnt' move with trunk.
<smoser> and the c-i reviewer will always (i hope) do a clean tox
<smoser> we froze the version of pylint, but not of any dependencies
<blackboxsw> smoser: want an MP for that stderr leak or a trivial fix?
<blackboxsw> http://paste.ubuntu.com/p/n4vQSjZXf8/
<blackboxsw> this stops the leaked msgs on tox -e xenial  cloudinit/config/tests/test_snap.py
<smoser> you can quick-fix
<blackboxsw> ok landing
<blackboxsw> smoser: minor nit on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/340140
<blackboxsw> smoser: is it time for me to make a bionic release cut for today?
<smoser> yeah.... if you want i just pushed to fix-iscsi
<smoser> we can grab that.
<smoser> assuming tox pass.
<smoser> blackboxsw: ping
<smoser> bugger.
<smoser>  cloudinit/config/tests/test_snap.py
<blackboxsw> pong
<smoser> cloudinit/config/tests/test_snap.py:448:62: F841 local variable 'm_stderr' is assigned to but never used
<blackboxsw> bah you are out of time. lost track of it.
<blackboxsw> can hangout to see what you want to do
<smoser> well,. fix that...  or i can.
<blackboxsw> fixing that
<blackboxsw> pushing and making and MP for bionic
<smoser> k
<smoser> i can come back in in a while... so why not try grab the iscsi
<smoser> get your fix in, then i can rebase and push
<blackboxsw> ok
<smoser> then wait on c-i for that and approve.
<blackboxsw> waitingon flake
<smoser> and then bioinc
<blackboxsw> smoser: 7ce839f846de705980839f9c7851bd0fd7353aad pushed
<blackboxsw> will watch your iscsi branch and land it prior to bionic MP
<smoser> just pushed.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/340140 should get marked c-i happy "real soon now"
 * blackboxsw watches
<bemis> blackboxsw: it's like a pot waiting to boil dude - stop watching or it'll take forever
<blackboxsw> hahaha
<blackboxsw> I looked away and wham, tea is ready  ;)
<blackboxsw> merging now and proposing a bionic merge of tip :)
<bemis> just the tip
<bemis> (sorry - it's like a macro-trigger or something)
<blackboxsw> heh, ok Bionic update proposed smoser https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341552
 * blackboxsw needs to wrap up the azuretimeouts review. nearly there 
<bemis> i don't recall who all i've spoken with on this, but it looks like 18.1 actually does drop EL6 support (via using python27+ methods)
<blackboxsw> bemis: I recall seeing your message exchanges earlier, generally we try not to break compatibility with older versions of python. We even monkey patch functionality in some cases to bridge the gap (even as far back as py2.6). It might be worth bringing up the specific discussion on this on Monday's status meeting
<blackboxsw> as RHEL folks may be in attendance
<blackboxsw> per the topic: Next status meeting: Monday 3/19 16:00 UTC
<blackboxsw> in this channel
<blackboxsw> if nothing else. more eyes will be on this channel for discussion
<bemis> blackboxsw: awesome, i'll try very hard to attend with useful data -- for right now the only confirmed thing i have is running analyze show https://paste.fedoraproject.org/paste/S5xXDUkIuW-Vl5pplr5uaw
<blackboxsw> bemis: out of my own curiousity, what prompted you to play with some of cloud-init's  commandline utilities/scripts? I'm asking as I merged a bunch of them into cloud-init proper recently as they were helpful for me, but I wasn't sure about the general population.
<blackboxsw> did you find out about those features via cloudinit.readthedocs.io? or just playing around with "cloud-init -h"
<blackboxsw> analyze specifcally was our own development tool written by rharper, but I hadn't known if others outside of our team were interested in using it
<bemis> blackboxsw: 100% tinkering - i think i might have seen it in readthedocs when trying to figure out why my userdata that worked in 079 didn't work in 18.1?
<bemis> so it kinda turned into "what throws errors?"
<bemis> (i still haven't finished reading the docs at readthedocs - so i'm in the weekend reading phase)
<smoser> blackboxsw 'build-and-push' is in progress. should get uploaded in 3 minutes or so.
<smoser> later.
<blackboxsw> couple minor comments posted dojordan. I need a bit of discussion on this with folks on Monday to see if we want to tweak UrlError handling a bit more
<dojordan> awesome- will address asap
#cloud-init 2018-03-17
<satsuma> Hi what is the syntax for creating a empty directory with #cloud-config?
<satsuma> I have tested to create files with write_files: but now I need a empty directory should I go for a ugly runcmd: with mkdir?
#cloud-init 2020-03-09
<Odd_Bloke> blackboxsw: rharper: https://github.com/isaacs/github/issues/1303#issuecomment-595231303 looks relevant
<rharper> Odd_Bloke: oi!
<Odd_Bloke> So it sounds like GitHub changed behaviour, and has now reverted that change due to it causing exactly the sort of issue that we saw.
<rharper> "We made a change yesterday that will be causing that."
<rharper> dang
<Odd_Bloke> And Fred was just unlucky when it came to the timing of the merge.
<rharper> so, in the face of that, do we still want to leave it as it is? or force push to fix ?
<Odd_Bloke> I think we should make very sure we attribute Fred in the release notes/changelog appropriately, but I don't think a force push is worth it for this, frustrating as that is for Fred.
<rharper> "By doing so, you have the last word in "authoring" the resulting squashed commit."   -- so if the sqaush/merger has the last word;   it's not clear to me how we make sure it's the PR author in there; do we include Author: <...> in the commit message ?
<Odd_Bloke> They've reverted the change, so I don't think we need to do anything.
<Odd_Bloke> https://github.com/isaacs/github/issues/1303#issuecomment-595595284 is the latest info.
<rharper> heh, until it is; I guess the fact that in the UI it's not clear who is getting the Authorship is my concern
<Odd_Bloke> "the PR author becomes the squashed commit's author"
<rharper> right; I wonder how we can see that in the UI
<Odd_Bloke> I don't know that we can.
<rharper> =(
<rharper> someone gets to be unlucky again
<Odd_Bloke> Yeah. :/
<Odd_Bloke> Did one of us cause https://github.com/canonical/cloud-init/pull/236 to be submitted?
<blackboxsw> Odd_Bloke: I was wondering that too.
<rharper> not me
<Odd_Bloke> powersj: ?
<blackboxsw> figured we'd talk in github meeting today
<rharper> yeah
<blackboxsw> not I said the fly
<rharper> lol
<blackboxsw> and did this rennovate plugin also cause/trigger the 'wip' action to run on https://github.com/canonical/cloud-init/pull/214
<blackboxsw> or is the wip action another magic 'thing' that just showed up too
<blackboxsw> separately
<powersj> I think someone installed it for the canonical org and enabled it for us as well
<Odd_Bloke> Aha, that makes sense.
<blackboxsw> thanks for the review rharper on https://github.com/canonical/cloud-init/pull/214   addressed comments (and dropped a bunch of duplicated doc content I hadn't meant to preserve
 * blackboxsw is only ec2 secondary ipv4/ipv6 support branch https://github.com/canonical/cloud-init/pull/214
<blackboxsw> then will hit the review queue for cloud-init
<blackboxsw> I mean ec2 PR is https://github.com/canonical/cloud-init/pull/114
<blackboxsw> https://getyarn.io/yarn-clip/1c229570-cf51-4374-8dd2-55648e940164
<Odd_Bloke> smoser: Thanks for the review on that pytest branch, we definitely ended up in a better place as a result. :)
<Odd_Bloke> blackboxsw: Your request changes (on https://github.com/canonical/cloud-init/pull/211) is blocking it from landing, could you take a look when you have a minute?
<blackboxsw> Odd_Bloke: definitely. rharper I see you approved https://github.com/canonical/cloud-init/pull/214
<rharper> blackboxsw: yes, looks good now
<blackboxsw> so my question really is,  (given ubuntu feature freeze) what we should do with 214 and 211
<rharper> we landed and uploaded our bug fixes, right ?
<rharper> and 214 has an FFE< right ?
<blackboxsw> rharper: bug fixes are landed, and 214 is FFe
<blackboxsw> but 211 should probably land too, but it's not an FFe or bug-fix really
<rharper> we have two ffes right ?
<rharper> are they both done yet ?
<blackboxsw> rharper: 214 is 'done' and 114 is in progress and I probably can finish it today
<blackboxsw> ec2 secondary nics
<rharper> I have a feeling that 211 ?
<blackboxsw> ec2 secondary ips I mean
<rharper> oh right, 211 (pytest) 214 (sys_info in instance-data), and secondary ips (114)
<rharper> heh, "old"
<blackboxsw> yeah should've done the secondary ips a looong time ago ... ahh well
<rharper> lacking a clear answer here, I'd ask in #ubuntu-devel on , or read more on FFE w.r.t whether we can do a new-upstream-snapshot into focal or if we have to cherry pick  and then cleanup when we do first SRU into focal after release
#cloud-init 2020-03-10
<Odd_Bloke> rharper: blackboxsw: I'd really like us to resolve the FFE/landing new things logjam today.  AIUI, we have two changes that we would like to land in focal before release, both of which have PRs open.  https://github.com/canonical/cloud-init/pull/114 isn't ready to land, but https://github.com/canonical/cloud-init/pull/214/ is.
<Odd_Bloke> blackboxsw: What are we waiting on for #114?  CI is failing, is fixing that all we need?
<Odd_Bloke> Actually, #214 is Approved, CI is passing and the FFE has been granted, so I've just landed that.
<Odd_Bloke> Better to ask forgiveness etc.
<rharper> Odd_Bloke: +1 on landing
<rharper> Odd_Bloke: AIUI,  I was just looking for confirmation that we'll land whatever in master, and cherry pick into focal, and next ubuntu/devel new-upstream-snapshot will just need to drop the cherry picks (as patches) as the snashot will contain those;    I'm being lazy hoping someone else confirms;  but I could step through that process on local branches and make sure the tools DTRT
<Odd_Bloke> blackboxsw: Given our earlier conversation, I think https://github.com/canonical/cloud-init/pull/211 can now get a +1 from you?
<blackboxsw> Odd_Bloke: yes I think we can, wrapping up a meeting. and we are starting the next meeting
<blackboxsw>  #startmeeting Cloud-init bi-weekly status
<blackboxsw> Hello and welcome to another cloud-init community status meeting.
<blackboxsw> our IRC channel topic carries the next planned status meeting for those that wish to participate.
<blackboxsw> All are welcome and interruptions encouraged
<blackboxsw> #chair rharper Odd_Bloke smoser
<blackboxsw> Let's try that again
<blackboxsw> #startmeeting Cloud-init bi-weekly status
<meetingology> Meeting started Tue Mar 10 16:22:58 2020 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> Hello and welcome to another cloud-init community status meeting.
<blackboxsw> our IRC channel topic carries the next planned status meeting for those that wish to participate.
<blackboxsw> All are welcome and interruptions encouraged
<blackboxsw> #chair rharper Odd_Bloke smoser
<meetingology> Current chairs: Odd_Bloke blackboxsw rharper smoser
<blackboxsw> Previous meeting notes are here
<blackboxsw> #link https://cloud-init.github.io/status-2020-02-18.html#status-2020-02-18
<blackboxsw> he topics we cover in this meeting are the following: Previous Actions, Recent Changes, In-progress Development, Community Charter, Upcoming Meetings, Office Hours (~30 mins).
<blackboxsw> *the even
<Odd_Bloke> o/
<blackboxsw> #topic Previous Actions
<blackboxsw> \O
<blackboxsw> sorry, have my big head on
<tribaal> o/
<blackboxsw> :).    Last meeting had no actions carried over. So I think this topic is a noop this week
<blackboxsw> #topic Recent Changes
<blackboxsw> recent changes landed in tip of master via git log --since 2020-02-18  https://paste.ubuntu.com/p/sJVpvjFbPj/
<blackboxsw> we've added some tooling/actions for github, ec2 IMDSv2 token redacting from logs, alloowing kernel cmdline to tell cloud-init network-config=disabled  and not falling back to IMDSv1 on Ec2-proper platform
<blackboxsw> thanks fred-lefebvre for the ec2 IMDS fallback branch and others for some additional driveby doc updates
<blackboxsw> Also, we performed an upload to Ubuntu Focal (20.04) series with latest tip of cloud-init to make sure the development release is up to date with recent features
<blackboxsw> Ubuntu Focal release is currently in feature freeze, so that will affect what patches we pull into Ubuntu Focal at this time as 'new features' would need a Feature Freeze Exception during the stage of Ubuntu development.
<blackboxsw> #link https://wiki.ubuntu.com/FocalFossa/ReleaseSchedule
<blackboxsw> As Odd_Bloke alluded to just before this meeting, we are trying to keep tip of master
<blackboxsw> open for commits.
<blackboxsw> On the ubuntu-side of the house we will sort cherry picking bug fixes into ubuntu focal during this short feature freeze period of time
<blackboxsw> If there are significant features that your cloud platform really would like to see on the first public release of Ubuntu Focal, then please get ahold of us in channel of on the mailing list to suggest that we shepherd those features in during this freeze.
<blackboxsw> But, generally cloud-init team will continue to follow the SRU process to get updates into Focal after feature freeze is lifted.
<blackboxsw> and again, our SRU test/verification process for Ubuntu will continue to target Xenial, Bionic, Eoan and Focal series for the updates we plan to make in the near future
<blackboxsw> one thing to note in recent changes as well is that we've now added the ability to query distro, kernel, cpu arch, python runtime version and and merged cloud-config to cloud-config jinja templates. So #cloud-config  userdata can be opinionated based on your target distribution or runtime environment.
<blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#format-of-instance-data-json
<blackboxsw> I think that about wraps recent changes
<blackboxsw> #topic In-progress Development
<blackboxsw> #link https://github.com/canonical/cloud-init/pulls
<blackboxsw> our active pulls above is probably the best source of info on features/bugfixes in flight.
<blackboxsw> Though behind the scenes we have held a couple of meetings to determine how much more automation/tooling  we need to clean up to improve our github developer process
<blackboxsw> I think Odd_Bloke and I have around 4 PRs that we are hoping to clean up to get a couple of things in place:
<blackboxsw> foremost I believe Odd_Bloke is scrubbing the github review process PR so that we have a good starting point for expectations for every developer, author or committer.
<blackboxsw> I think ultimately the goal there is to make sure committers can provide a set of expectations on active PR reviews to PR authors, so that active developers get better prioritized reviews.
<blackboxsw> we are going to add and enable a number of github actions and workflows that should do the following:
<blackboxsw>   - age PRs and add labelling to indicate to reviewers and authors that a review needs attention or it will be automatically closed (after around 4 weeks of languishing)
<blackboxsw> -  label/notify steps to signed the contributor license agreement if unsigned
<blackboxsw> - run addtional integration tests on active PRs in CI
<blackboxsw> not sure if I am I missing anything else there.
<blackboxsw> but ideally we'd like to find a process that helps upstream unblock PRs and get review comments faster
<blackboxsw> so, it'll be an iterative process
<blackboxsw> and thanks for the reviews and suggestions so far on https://github.com/canonical/cloud-init/pull/160 as it is the first cut at trying to document the process
<tribaal> Nice!
<blackboxsw> woot. the hope I believe is to land that this week as well as branches like https://github.com/canonical/cloud-init/pull/164 https://github.com/canonical/cloud-init/pull/236 and  https://github.com/canonical/cloud-init/pull/125
<blackboxsw> additionally a gap that we still have vs when we hosted in Launchpad, is our auto-merge Launchpad bug commenting/maintenance:
<blackboxsw>  We still have a need for the following:
<blackboxsw>   - comment on LP bugs linking to an open github PR
<blackboxsw>  - comment on merged commitish in github and Fix Commited state when a PR lands in upstream
<blackboxsw> so we'll be tackling that too in order to make upstream maintainers happier and better advertise fixes to bug filers
<blackboxsw> right now that's all a manual process since we haven't retooled our bug-related tooling
<blackboxsw> #topic Community Charter  and upcoming meeting
<blackboxsw> let's set the status meeting for next session
<blackboxsw> oops and I realize only now that I blew it this week due to daylight savings... meeting wasn't 'supposed' to start until 25 mins from now : /
<blackboxsw> #topic cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting March 1 16:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting March 1 16:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> that's better.
<AnhVoMSFT> it says March 1, is that correct?
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting March 24 16:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<tribaal> there :)
<blackboxsw> hah, good I got another participant
<blackboxsw> :)
<AnhVoMSFT> UK will also have their own daylight savings March 29th I believe
<blackboxsw> sorry AnhVoMSFT tribaal :)
<blackboxsw> yeah this time of year always messes with timing. We try to set things in terms of UTC to avoid thrashing
<blackboxsw> but even that fails due to human error (my bad)
<blackboxsw> Other community charter tasks are generally categorized in bugs labelled bitesize
<blackboxsw> #link  https://bugs.launchpad.net/cloud-init/+bugs?field.tag=bitesize
<blackboxsw> general topics for this year were tasks that are easily done in parallel, such as json schema addtions and datasource readthe docs updates/corrections and fleshing out.
<blackboxsw> jsonschema example is here for review if anyone wants to take a stab at testing it out. or extending schema for other config modules.https://github.com/canonical/cloud-init/pull/152
<blackboxsw> and again all cloud-init contributors are encouraged to review/comment any active cloud-init PRs  @ https://github.com/canonical/cloud-init/pulls  the more voices, the better the quality
<blackboxsw> #topic Office Hours (next 30 mins)
<blackboxsw> During this topic, please bring up any questions, discussions, bugs or features or paper cuts that need attention. there should be a couple of cloud-init developers with eyes on the channel to actively respond.
<blackboxsw> just before the start of this meeting Odd_Bloke was asking about getting the pytest branch landed for cloud-init. (moving off of nosetests as the project is EOL/unmaintained)
<tribaal> that's nice. pytest is becoming the de-facto standard these days anyway
<blackboxsw> yeah, didn't want to get stuck using something that becomes unsupported or unsupportable. we don't have the bandwidth in this project to maintain stacks that aren't being looked at by the collective internet ;)
<Odd_Bloke> blackboxsw: I believe that branch is now only blocked on you removing your "Request changes" review now that we've established that we don't need to hold off on landing things for Feature Freeze.
<tribaal> makes total sense :)
<blackboxsw> Odd_Bloke: do you know if rharper did the new-upstream-snapshot into focal already
<blackboxsw> to have a 'clean slate' for the pytest branch landing
<Odd_Bloke> blackboxsw: We can new-upstream-snapshot from an older commit, I don't believe that's a blocker.
<blackboxsw> Odd_Bloke: not a blocker, but I could do that now as it's 5 mins
<blackboxsw> then we can land right aftr
<blackboxsw> sound good? I see nothing queued https://launchpad.net/ubuntu/focal/+queue?queue_state=3&queue_text=cloud-init
<Odd_Bloke> Sure, if you're happier doing that. :)
<blackboxsw> I aam :)
<blackboxsw> ok doing that right now
<blackboxsw> then we can start the cherry picking just after
<blackboxsw> Odd_Bloke: I'm adding this as the debian/changelog section title
<blackboxsw>   * New upstream snapshot: bug-fix-only feature-freeze-exception
<blackboxsw> instead of   * New upstream snapshot:
<blackboxsw> sound reasonable?
<Odd_Bloke> It isn't bug-fix-only because we have the FFe for a non-bugfix.
<Odd_Bloke> I was just reading the wiki page that suggested wording, I think, let me take a look.
<blackboxsw> https://github.com/canonical/cloud-init/pull/241
<blackboxsw> ahh right, reviewing that now
<blackboxsw> yeah not quite sure how to handle our FFe uploads
<Odd_Bloke> Oh, if it's only that change, then I don't think we need anything specific in the changelog.  We have an FFE bug that we're closing with that upload.
 * blackboxsw re-reads https://wiki.ubuntu.com/FreezeExceptionProcess
<Odd_Bloke> And that means it's _definitely_ not bug-fix-only, there isn't a single bugfix in there. ;)
<blackboxsw> ok that sounds good, will just keep the New upstream snapshot
<blackboxsw> ok thanks for the review. redoing.
<Odd_Bloke> Hopefully I'm right and you don't get yelled at. :p
<powersj> better to upload and ask for forgiveness...
<blackboxsw> ehh, unlikely :) it seems like steve and others have been pretty lax about changelog text (or manipulating it after the fact) :)
<Odd_Bloke> Cool, I'll do the same thing locally to review.
<blackboxsw> Odd_Bloke: force pushed
<blackboxsw> https://github.com/canonical/cloud-init/pull/241/files
<Odd_Bloke> blackboxsw: Approved.
<blackboxsw> thanks Odd_Bloke
<blackboxsw> build-and-pushing it up
<blackboxsw> ok onto your pytest branch now
<blackboxsw> while I await the ubuntu "accepted" response email
<blackboxsw>   Uploading cloud-init_20.1-10-g71af48df-0ubuntu1.dsc: done.
<blackboxsw> just waiting on review/acceptance
<blackboxsw> community-notice: tip of cloud-init published into Ubuntu Focal (20.04) [ubuntu/focal-proposed] cloud-init 20.1-10-g71af48df-0ubuntu1 (Accepted)
<blackboxsw> ok Odd_Bloke merged at long last https://github.com/canonical/cloud-init/pull/211
<blackboxsw> nosetest is dead, long live pytest
<Odd_Bloke> \o/
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/239 <-- another small one for you
<blackboxsw> approve Odd_Bloke and thanks. it is waiting on you for merge (and CI completion)
<blackboxsw> works on my focal box
<blackboxsw> which was broken before
<blackboxsw> Odd_Bloke: https://github.com/canonical/cloud-init/pull/164 is ready for you I think (labeling  in cron)
<blackboxsw> for CLA ! CLA.
<blackboxsw> or do we want that PR to actually ignore certain paths (like doc changes) as not-requiring CLA
<Goneri> blackboxsw, could you take a look at https://github.com/canonical/cloud-init/pull/62 Pleaaaase :-)
<blackboxsw> Goneri: yes, and so sorry about that.
<Goneri> np, and Yeah! for pytest :-D
<blackboxsw> ... and that about wraps up on our cloud-init status meeting. :) I'll close it out and post the minutes for next time
<blackboxsw> #endmeeting
<meetingology> Meeting ended Tue Mar 10 17:45:01 2020 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-03-10-16.22.moin.txt
<blackboxsw> ok minutes published to cloud-init.github.io should be up sson
<blackboxsw> soon
<meena> sweet mother of glob, it's Tuesday already
<Odd_Bloke> blackboxsw: I've approved https://github.com/canonical/cloud-init/pull/164 and have just rebased it. :)
<blackboxsw> thanks Odd_Bloke I'll watch CI and land when complete
<tribaal> meena: scary isn't it?
<meena> what a week / month
 * meena is now doing the mom at home thing, and my partner is back to work
<blackboxsw> meena it is crazy how much the the being at home thing affects us all :). if I didn't have a calendar invite telling me it's tuesday, I'd never know what day it was.
<meena> i had an appointment today (well, the baby did) so i was out, and i should've known which day it is, but since then it feels like another two days could've gone by
<blackboxsw> heh it's tough on disrupted sleep too I bet
<blackboxsw> Odd_Bloke: rharper: just confirmed the ec2 redacted token fix that landed is good on latest ec2 cloud-images for cloud-init 20.1.9 and Fred's not falling back to IMDSv1 is in on ec2-proper good shape too.
<blackboxsw> just double checked since that landed in our previous new-upstream-snapshot before the FFe today and cloud-images looked updated
<rharper> blackboxsw: nice
<sarnold> blackboxsw: do you have a link handy for that fix?
<Odd_Bloke> We're seeing intermittent CI failures during cloud tests.  For example, https://travis-ci.org/canonical/cloud-init/jobs/660756966.  An educated guess is that we're taking too long to download lxd images due to bandwidth constraints, but I don't know that for sure.
<blackboxsw> sarnold: you mean the fix to avoid falling back to IMDSv1 on true ec2 clouds?
<blackboxsw> https://github.com/canonical/cloud-init/commit/1f860e5ac7ebb5b809c72d8703a0b7cb3e84ccd0 for the above
<sarnold> blackboxsw: hmm, I meant the redacted tokens in the json
<blackboxsw> ahh one sec
<blackboxsw> sarnold: https://github.com/canonical/cloud-init/commit/fa639704f67539d9c1d8668383f755cb0213fd4a
<sarnold> blackboxsw: cool, thanks
<blackboxsw> no problem.
<rharper> Odd_Bloke: so, paride reported iso tests failing due to snapd.seeded issues once again
<rharper> Odd_Bloke: I would suspect it's the same for those images with lxd as snap
<blackboxsw> sarnold: also FWIW, I just launched a focal minimal VM via an IAM profile and re-enabled security-credentials crawling to validate that that sensitive key is properly redacted for non-root user by runnning cloud-init query --all | grep redact | pastebinit   https://paste.ubuntu.com/p/874pcvn326/
<rharper> ahosmanMSFT: I tested your https://github.com/canonical/cloud-init/pull/206 today; and it needed just a bit more to keep the instance around;  without the extra changes when the snapshot and image get torn down in my testing
<Odd_Bloke> rharper: Aha, good point.
<Odd_Bloke> Another small tox PR: https://github.com/canonical/cloud-init/pull/242
<Odd_Bloke> blackboxsw: I noticed a slight issue with the CLA workflow which is fixed in https://github.com/canonical/cloud-init/pull/243
<blackboxsw> cool Odd_Bloke can you add a trello card + PR link there for tracking please?
<blackboxsw> so if I don't get to it we can make sure to wrap it up tomorrow early
 * blackboxsw is in omg need more trello cards mood at the moment
<Odd_Bloke> blackboxsw: Why don't you assign yourself? :p
<blackboxsw> heh
<blackboxsw> can do
<Odd_Bloke> rharper: Were those snapd.seeded issues happening on xenial, do you know?
<rharper> no
<rharper> bionic and newer
<rharper> lxd is still a deb in Xenial AFAIK
<Odd_Bloke> OK, so probably not the source of our CI issues, as that should be running xenial.
<rharper> ah, ok
<blackboxsw> created my needed trello card... and the world is right again
<Nick_A> what would be the best way to get cloud-init to add "critical: true" to every interface it sticks in netplan on ubuntu?
<sarnold> ohai Nick_A :)
<Nick_A> hey
<Nick_A> what channels are you not in?
<sarnold> I ask myself that every day ..
<powersj> Nick_A, a bug would be a good start
<powersj> I don't think our network config handles that, but rharper can confirm
<rharper> Nick_A: in the case that you provide netplan as  network-config to cloud-init we pass-through this config to the OS;  we do not tag any interface with critical: true as we don't know that it is (except in iscsi-root scenario, where we do mark the iscsi interfaces a critical-connection)
<blackboxsw> rharper: just confirmed ec2 doesn't provide classless static routes via dhcp responses.  so  I *think* incrementing route-metric is unneeded
<blackboxsw> will pull it from my branch
<rharper> blackboxsw: ok
<blackboxsw> rharper: pastebin coming to confirm with you
<blackboxsw> https://paste.ubuntu.com/p/zrFtx7S7Ct/
<rharper> blackboxsw: yes, agreed
<blackboxsw> I think the ec2 networkd dhcp internal response representation lacking "ROUTES" means no classless routes are used for multiple  nics
<blackboxsw> ok goo
<blackboxsw> good even. and goo
<blackboxsw> ok will have ec2 ready for review tomorrow. touching up unit test now
<blackboxsw> and will file the related FFe bug
<blackboxsw> for ubuntu inclusion in focal
#cloud-init 2020-03-11
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1866930 created and https://github.com/canonical/cloud-init/pull/114 pushed for tomorrow rharper
<ubot5> Ubuntu bug 1866930 in cloud-init (Ubuntu) "[FFe] ec2 add support for configuring secondary ipv4 and ipv6 addresses" [Undecided,New]
<slaweq> hi
<slaweq> I'm working on OpenStack Neutron and I want to talk with You about Metadata over IPv6
<slaweq> since some time we have spec https://review.opendev.org/#/c/315604/ to add support for IPv6 in Neutron
<slaweq> *for Metadata over IPv6 in Neutron
<slaweq> the proposal in this spec is to use fe80::a9fe:a9fe as IPv6 address which is equivalent to 169.254.169.254
<slaweq> but as cloud-init is one of the main users of metadata service, I would like to ask also You about that, what do You think about it and if this will work for You
<slaweq> any feedback here or in the spec's review will be appreciated :)
<tribaal> slaweq: just a note - most of the cloud-init community lives in US timezones, so replies can take a few hours to come.
<slaweq> tribaal: thx for that info
<slaweq> tribaal: is there any active ML for cloud-init where I can maybe send such question?
<tribaal> slaweq: yes, I think https://launchpad.net/~cloud-init (notice the mailing list link) would be the relevant contact point. Note that you need to be part of the "team" to post, but to my knowledge that's just a matter of clicking "join" :)
<slaweq> tribaal: thx a lot
<slaweq> I will try that
<tribaal> slaweq: from a scan of the openstack-specific DS code in the tree it seems like using ipv6 (or an ipv6 fallback) for the datasource is possible
<tribaal> I'm not sure how that is implemented in other datasources however (currently the openstack datasource will check on http://169.254.169.254 unconditionally)
<tribaal> ah, my bad. it will check there unless the metadata ur is supplied through the config as metadata_urls
<slaweq> tribaal: can You point me to the code with that?
<tribaal> sure, sec
<slaweq> tribaal: thx a lot
<tribaal> slaweq: https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/DataSourceOpenStack.py#L59
<slaweq> tribaal: so in fact if we will add som IPv6 address, it can be configured in cloud-init's config to be used
<slaweq> so in fact no changes in cloud-init code are required in fact
<tribaal> slaweq: yes, it looks like that would eb the case.
<slaweq> tribaal: maybe we can propose to add such IPv6 address to the default list later
<slaweq> but it's not required to make it working
<tribaal> slaweq: if the ipv6 option becomes an openstack standard perhaps you could consider adding it to the default URLs (making it a list). But some guidance from the real cloud-init developers would be better than mine here :)
<tribaal> yes, it looks like.
<slaweq> tribaal: ok, thx a lot for help
<tribaal> from my uneducated perspective at least :)
<tribaal> slaweq: anytime :)
<slaweq> I will send email to the ML to get some more feedback about that but You helped me a lot with that
<tribaal> good to hear!
<otubo> meena: I saw you left a comment on the pull request https://github.com/canonical/cloud-init/pull/204
<otubo> Odd_Bloke: ^
<otubo> I also saw a comment about considering the CVE-2020-8631.
<otubo> meena: Odd_Bloke I'm just a little confused if this PR actually addressed the CVE or not, because it still uses random.choice
<Odd_Bloke> otubo: It doesn't use random.choice, it uses random.SystemRandom.choice.
<Odd_Bloke> Which uses the system's random pool instead of the pseudorandom Mersenne Twister that random.choice uses.
<otubo> Odd_Bloke: I noticed that. And that was the reason I was confused if it was really fixed or not :) Now it's clear. Thanks!
<Odd_Bloke> :)
<Odd_Bloke> rharper: blackboxsw: https://github.com/canonical/cloud-init/pull/244/ <-- this adds the capability to store just GitHub usernames as CLA signers (and adds dhensby as the first such person)
<Odd_Bloke> rharper: Thanks for the review, I've addressed your comment. :)
<rharper> cool, I'll re-review
<Odd_Bloke> Thanks!
<Odd_Bloke> (I happened to look at the tab as your review showed up, I wasn't just sitting there waiting for a review to come in. ;)
<rharper> hehe
<AnhVoMSFT> checking out master and running tox gives me error tox.ConfigError: ConfigError: No support for the posargs substitution type
<powersj> AnhVoMSFT, can you paste the full output? and `tox --version` and `python -V`
<powersj> err python3 -V :D
<AnhVoMSFT> tox --version
<AnhVoMSFT> 2.3.1 imported from /usr/lib/python3/dist-packages/tox/__init__.py
<AnhVoMSFT> python3 -V
<AnhVoMSFT> Python 3.5.2
<powersj> that looks like Ubuntu 16.04?
<powersj> xenial?
<AnhVoMSFT> yes powersj
<powersj> Odd_Bloke, ^ possible regression from yesterday? If I checkout master without yesterday's commits all is well
<AnhVoMSFT> powersj full output https://paste.ubuntu.com/p/b4ncHRJhxc/
<AnhVoMSFT> (sorry for the delay, multitasking)
<Odd_Bloke> Oh, hmm.
<Odd_Bloke> Let me take a look.
<Odd_Bloke> Yeah, OK, looks like it was introduced by my pytest branch.
<Odd_Bloke> AnhVoMSFT: A short-term workaround is to install tox from PyPI into a virtualenv.  That tox version should be fine.
<Odd_Bloke> In the meantime, I'll dig into fixing it properly.
<Odd_Bloke> blackboxsw: rharper: https://github.com/canonical/cloud-init/pull/245 <-- this fixes running tox on xenial
 * blackboxsw tries to reproduce that problem in the first place
<blackboxsw> approved Odd_Bloke just waiting on CI
<blackboxsw> .... again
<powersj> Odd_Bloke, thanks!
<rharper> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1867029  ; didn't we look at an issue like this?  IIRC, the bionic ifupdown package does not include the source /etc/network/interfaces.d/*.cfg line in /etc/network/interfaces, right ?
<ubot5> Ubuntu bug 1867029 in cloud-init "Package ifupdown breaks network configuration by cloud-init" [Undecided,New]
<Odd_Bloke> Landed.
<Odd_Bloke> AnhVoMSFT: master should now be back to working, thanks for letting us know it was broken!
<Odd_Bloke> blackboxsw: Looking at timestamps, we weren't waiting for Travis on that one, just on our actual CI jobs.
<blackboxsw> rharper: checking
<Odd_Bloke> Our worst queue times are likely in the mornings, because I think most of the multipass team are in Europe. :p
<blackboxsw> rharper: we did have an issue just like this (iternally) someone was installing ubuntu-desktop on cloud-images which pulled in ifupdown
<rharper> yes
<rharper> and on bionic the package does not include the source line
<rharper> so we render eni
<rharper> but nothing sources it
<blackboxsw> right, presence of ifupdown on bionic is non-standard/recommended and cloud-init checks ifupdown first, but stock cloud images have ifupdown includes disabled
<blackboxsw> because we should be using netplan
<blackboxsw> right
<blackboxsw> still feels a bit like it should be a bug in ifupdown package for bionic. if installing it, it should probably emit the proper include, or cloud-init can emit a warning saying "hey there I see ifupdown and will be rendering that but I see no includes for the interfaces.d file I'm about to write, so I'm going to include it kthxbye"
<blackboxsw> I think it's probably worth considering it as a cloud-init bug, because we should be smart enough to look at the environment config files for the rendering we are emitting to make sure it is valid
<blackboxsw> what do you think?
<blackboxsw> and it seems like a low hanging fruit kind of bug too
<blackboxsw> if we handle it that way in net renderer.
<Odd_Bloke> I would like to see a clearer description of the problem; what is the gap when ifupdown is installed on bionic?
<blackboxsw> when ifupdown and netplan are both installed cloudinit network renders prioritize writing /etc/network/interfaces.d/*.cfg instead of /etc/netplan/50-cloud-init.yaml
<blackboxsw> problem on certified cloud-images is that the magic /etc/network/interfaces config file is in a disabled state
<blackboxsw> http://paste.ubuntu.com/p/9yhHVZjqGB/
<rharper> blackboxsw: yes
<rharper> I updated the bug
<Odd_Bloke> Right.  This feels like it could be an ifupdown bug, we should pursue that route first, I think.
<blackboxsw> and ifupdown package doesn't actually write or update that file if it already exists in images
<rharper> the workaround is to just put the source line in there;  we need to sort out if ifupdown packaging or cloud-init (or both) should ensure that's present
<Odd_Bloke> Because that comment is currently a lie, I think. :p
<rharper> blackboxsw: even if it were missing, it doesn't put the right source line IIRC
<blackboxsw> yeah I think ifupdown package postinst script should be a bit smarter instead of just a presence check (because we know cpc images intentially disable that file with comments etc)_
<blackboxsw> rharper: if /etc/network/interfaces file is missing  it writes the appropriate include
<blackboxsw> "it" being ifupdown deb postinst script
<blackboxsw> yeah and that comment is a lie, because just installing ifupdown leaves the same /etc/network/interfaces file in place (disabled)
<blackboxsw> just tested that
<blackboxsw> same ENI content
<rharper> blackboxsw: it's functional, but ISTR some subtly between source-directory and source *.cfg
<rharper> I forget what at the moment
<rharper> I think there's a built-in regex that ifupdown uses
<rharper> with source_directory vs. source *.cfg
<blackboxsw> and on bionic the ifupdown.postinst which source-directories the right dir http://paste.ubuntu.com/p/TKtfmN7Zyt/
<rharper> I think I saw a debian bug related to this since their images used source_directory
<rharper> blackboxsw: yes, it's the correct directory
<rharper> but it restricts what files it reads within it
<rharper> cloud-init's cfg is read
<rharper> but there are subtle regex differences for what files are named
<rharper> and what ifupdown actually reads
<rharper> so, ubuntu cloud-image with ifupdown and source *.cfg will not always see the same files as an image with source_directory
<blackboxsw> rharper: are you saying the "source-directory /etc/network/interfaces.d   is more strict than "source /etc/network/interfaces.d/*.cfg"   ?
<rharper> possibly the opposite
<blackboxsw> ahh less strict
<rharper> but I need to find the bug
<rharper> https://github.com/canonical/cloud-init/commit/a6faf3acef02bd8cd4d46ac9efeebf24b3f21d81
<rharper> The current filename, 50-cloud-init.cfg, does not match against the RE
<rharper> that is used to scan the directory for configurations (ASCII upper- and
<rharper> lower-case letters, ASCII digits, ASCII underscores, and ASCII
<rharper> minus-hyphens):
<blackboxsw> ahh interesting. ok
<Saviq> Odd_Bloke: just hired someone in Argentina ;)
<blackboxsw> so do we want cloud-init behavior to prescribe that anyone wanting network config in ENI need have *.cfg files?
<rharper> blackboxsw: well, the distro class has a say in it
<Saviq> Also, one of our devs is in Tennessee, soâ¦ ;P
<blackboxsw> Saviq: so annual meetings in Columbia to split the difference?
<blackboxsw> so TN and Agentina have to travel the same distance :)
<Saviq> With how things are going, no f2f meetings for a year ;P
<blackboxsw> true story
<Odd_Bloke> Saviq: Please go to bed (and stop kicking off CI runs). ;)
<Odd_Bloke> I'm looking at the daily build failures next.
<Odd_Bloke> (I believe it should just be a case of updating the Build-Depends in each packaging branch.)
<rharper> Odd_Bloke: awesome!
<Odd_Bloke> Hmm, actually, this is a little more complicated than that.  If I update the Build-Depends without pulling in the upstream commit, then builds will fail when we just cherry-pick changes (because nose won't be installed).
<Odd_Bloke> s/the upstream commit/\0 that introduced pytest/
<blackboxsw> rharper: https://github.com/canonical/cloud-init/pull/114 is up for re-review (ec2 secondary ips). one question I have is how do we want cloud-init
<blackboxsw> finally got my unittests fixes. FWIW assertItemsEqual for dict type items instead of assertEqual because python dict key ordering is different on xenial version on python as opposed to > xenial
<blackboxsw> I keep forgetting that
<rharper> blackboxsw: yeah,I saw that change in your tests; that was new to me w.r.t having a non-ordered content comparision
<blackboxsw> yeah it bit us on landscape tests all the time, and sometimes I've seen it in cloud-init too
<blackboxsw> most of the time i remember when writing the test... ahh well
<tribaal> I get bit by that all the time :)
<blackboxsw> silly former landscapers :)
<tribaal> hehe you'd think we learned byt the time :)
<blackboxsw> hah
<rharper> tribaal: btw, thanks for point slaweq to the mailing list and the datasource bits; much appreciated
<tribaal> anytime :)
<tribaal> seems like a pretty straightforward patch to write (to me), but the multicast point brought up on the list as a followup is an interesting approach
<Odd_Bloke> assertEqual should work on dicts.
<rharper> yeah, and I'm worried about the cost of either hitting ipv4 when there is no, or ipv6 when there is none
<tribaal> even though I'm not directly involved with an openstack cloud anymore, I could see the multicast thing transfer well to other clouds as well, indeed
<rharper> tribaal: I
<tribaal> you?
<tribaal> :)
<rharper> hehe, I've not responded to that yet; the idea of having a common "IMDS" url is interesting, but I'm not sure it makes anything more common other than the location;  the content is still per-platform;
<tribaal> Yeah, that's true
<Odd_Bloke> blackboxsw: Just added a comment about assertItemsEqual; you're actually papering over a _list_ being out-of-order, this isn't to do with dict key ordering.
<Odd_Bloke> (Unless, I suppose, that list is being generated from dict keys.)
<blackboxsw> Odd_Bloke: ahh right, so we could/should sort that list to avoid thrashing on different versions of python
<blackboxsw> right the source comes ultimately from keys
<blackboxsw> we can sort it in code proper or assertItemsEqual
<blackboxsw> I'd prefer to handle in unit tests because we don't truly need to do the sorting work for what cloud-init renders
<Odd_Bloke> I think better to have the code produce something that's stable across Python versions.
<blackboxsw> but I can be swayed either way
<blackboxsw> rharper: I need a swing vote on that :)
<Odd_Bloke> Well, I give an example where the order does make a difference.
<blackboxsw> surely
<Odd_Bloke> I don't know if it makes a _practical_ difference, but there would be different internal state at least.
<rharper> is the a *value* list or  the order of the keys() output ?
<rharper> we end up running json.dumps, with sort_keys=True, so what we write out is in sorted *key* order, but the value indexed by the key is not sorted AFAICT in our code;
<rharper> at some unittime cost; it may be better to dump the output via util.json_dumps() and read it back and compare
<blackboxsw> rharper: the list value comes out sorted differently from macs_to_nics.items()
<rharper> the value, right
<blackboxsw> so the value of the 'addresses' sorts differently on xenial
<blackboxsw> based on what mac we originally process from IMDS
<blackboxsw> I can make the change to sort the macs_to_nics dict. and then we should not have this problem.
<blackboxsw> easy enough. and minor cpu cost\
<rharper> that seems better and then assertEqual should work as well, IIUC
<blackboxsw> right then assertEqual will work
<blackboxsw> and we may have need for ordered internal state in the future anyway.
<rharper> blackboxsw: thoughts on whether we can generically walk the dict, and on value types apply sorted ?
<rharper> that is, if we add new keys in the future, how do we prevent having more unsorted lists
<Odd_Bloke> blackboxsw: rharper: What do you think to this approach: https://github.com/canonical/cloud-init/pull/246
<rharper> looking
<Odd_Bloke> It means our daily builds will needlessly install python3-nose and our archive builds will needlessly install python3-pytest.
<Odd_Bloke> The alternative would be to cherry-pick the pytest commit onto each release branch, so that we can consistently just test with pytest.
<rharper> hrm
<Odd_Bloke> (And all of this inconsistency will go away in the next SRU regardless.)
<rharper> leading with that last comment there; I;m in favor of some short term "needless" package installs to avoid git cherry pick on each release branch pain
<blackboxsw> Odd_Bloke: I think we ultimately have to cherry-pick into each ubuntu/<series> anyway right
<rharper> blackboxsw: on sru, we new-upstream-snapshot *and* update build deps
<rharper> so, no ?
<blackboxsw> our new-upstream-snapshot doesn't pull int debian/* file changes
<blackboxsw> rharper: ahh I see, right deps should get corrected via new-upstream-snapshot
<blackboxsw> ok
<rharper> right, but issue is that tools/read-deps picks up the new change, but debian/ is stale
<Odd_Bloke> The deps won't be corrected by new-upstream-snapshot.
<blackboxsw> just ubuntu/xenial/debian doesn't see updates from ubuntu/devel/debian due to new-upstream-snapshot
<Saviq> Odd_Bloke: btw, are we actually causing you guys delays? this is Travis, I assume? is it b/c we're under the same canonical/ GH org?
<Odd_Bloke> We will need to manually drop the python3-nose Build-Depends once it is actually unused.
<Odd_Bloke> Saviq: Yeah, we've had the canonical org put onto a paid Travis plan so that will hopefully help.
<rharper> Odd_Bloke: right, we move release branch to master; *and* update debian Build-Deps
<Odd_Bloke> Saviq: Part of the problem is that we aren't (yet) using a landing bot like bors, so we really notice delays.
<blackboxsw> ok Odd_Bloke is it worth an SRU_BLOCKER comment added to ubuntu/(xenial|bionic|eoan)/debian/control   or a trello card on our SRU template board?
<Odd_Bloke> blackboxsw: This isn't a blocker for SRUs IMO.
<blackboxsw> I guess what I'm wondering is how we avoid losing that work item
<Saviq> Odd_Bloke: ah so that's why travis-ci.com got enabled on the org ;)
<rharper> blackboxsw: I was just thinking we might want to to release branch merging in our CI builds; so we could have see this fail in CI if we merged release branch debian on top of the PR branch ,
<Odd_Bloke> blackboxsw: Perhaps on the SRU template board?
<Odd_Bloke> (If we do drop it it's not the end of the world, we'll just be installing nose in our build environments unnecessarily.)
<Odd_Bloke> s/drop it/forget to make the change/
<Odd_Bloke> rharper: Chad is +1 on the PR, I take it from your above comments that you are too?
<rharper> yes, I'll mark approve as well
<Odd_Bloke> OK, I'm going to make sure that focal now builds, then will propose a similar change for each other release.
<Odd_Bloke> Saviq: Yep, that's why. :)
<Saviq> I suppose that was inevitable, and makes sense anywayâ¦
<blackboxsw> ok, Im reviewing Goner's netbsd(*bsd) support PR https://github.com/canonical/cloud-init/pull/62 and the network config emitted from the distro is v1 in both cases. I'm thinking since we are touching/reviewing that support, maybe we should make that generate_fallback config could be simple v2 https://github.com/canonical/cloud-init/pull/62/files#diff-1708fc6fbf7cc4ca958a7adab7ad615eR35-R40
<blackboxsw> rharper: Odd_Bloke any opinions on generally whether we should be moving distro network rendering to v2 or leave them all v1?
<rharper> blackboxsw: do you mean datasource ?
<blackboxsw> I knew datasource-wise  we'd like to make that migration to v2 in the datasource
<blackboxsw> rharper: I do mean distro in some cases constuct v1 network config for fallback
<blackboxsw> we could move them to v2 as they are being touched like the link above
<blackboxsw> so that eventually we can move our internal state off of v1
<blackboxsw> and drop our network config v1 -> v2 translation layer logic
<rharper> what generates v1 in distro ?
<blackboxsw> distro.generate_fallback_config is the method I'm talking about
<rharper> just bsd no ?
<Goneri> could this be done in a 2nd iteration. I understand the idea, but I would like to land the PR too :-)
<rharper> it calls net.generate_fallback_config()
<blackboxsw> well our base Distro class does
<rharper> which emits v2
<rharper> blackboxsw: huh ?
<blackboxsw> right rharper in the existing PR, it did v1.
<rharper>    def generate_fallback_config(self):
<rharper>         return net.generate_fallback_config()
<blackboxsw> for netbsd only. and I wanted to steer that to v2
<blackboxsw> rharper: https://github.com/canonical/cloud-init/pull/62/files#diff-1708fc6fbf7cc4ca958a7adab7ad615eR35-R40
<blackboxsw> I want that v2
<blackboxsw> Goneri: it could be done on a followup
<rharper> blackboxsw: I suppose, I'm not too worried about it;  the biggest win to moving to v2 in fallback is that if target distro supports v2, then we can pass it through as it is
<rharper> it's unlikely that v2 will ever be around on bsd, so it's not, IMO, a big issue to emit v1 to convert to network_state()
<blackboxsw> right, which isn't going to happen on *bsd. so its probably more paper work to move to v2
<blackboxsw> right
<blackboxsw> ok makes sense, thanks for being the sounding board. Goneri we won't worry about that then
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/247 <-- eoan
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/248 <-- bionic
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/249 <-- xenial
<Odd_Bloke> blackboxsw: rharper: Review of ^ would be appreciated!
<rharper> ok
<blackboxsw> 247 merged
<blackboxsw> rharper: you on 248 or 249?
<blackboxsw> I object to https://github.com/canonical/cloud-init/pull/248/files formatting as cloud-init attribution in previous changelog entries generally carries a different 'style'
<rharper> blackboxsw: neither as you hopped on one already; was thinking it was easier to review all of them together for comparision purposes
<blackboxsw> rharper: +1 though Id like you and Odd_Bloke weigh in on 248
<blackboxsw> specifically my unrealistic https://github.com/canonical/cloud-init/pull/248#discussion_r391278819
<blackboxsw> or pedantic
<blackboxsw> it just represents something 'different' and I get all prickly with change ;)
<blackboxsw> same nit/request w.r.t https://github.com/canonical/cloud-init/pull/249/files
<blackboxsw> I think generally we redacted authorship attribution in debian/changelogs for the upstream team. in ubuntu deb changelog entries, but should that be different w.r.t. specific packaging changes to the debian/* directory
<rharper> blackboxsw: generally I think it might be useful to keep authorship for the debian dir change sinces that's downstream changes rather than upstream code ; but I'm fine redacting as well;
<blackboxsw> ok /me backs down
<blackboxsw> works for me
<blackboxsw> though when new-upstream-snapshot combines these items I wonder if it's worth us redacting that to make the changelog a bit more readable
<blackboxsw> since it's likely to include a lot of fixes
<blackboxsw> redacting the authorship sections
<blackboxsw> heh I see Odd_Bloke redacted :) thanks man
<blackboxsw> I mean thanks Dan
<Odd_Bloke> Done for both.
<blackboxsw> approved waiting on ci
<Odd_Bloke> FWIW, those names are categorically different from the "New upstream snapshot" lines where we redact the names, and are a standard feature of Debian changelogs.
<blackboxsw> though Odd_Bloke do you think that not bumping debian/changelog version will actually not allow our builds to succeed
<blackboxsw> https://github.com/canonical/cloud-init/pull/249 and https://github.com/canonical/cloud-init/pull/248
<Odd_Bloke> Why?
<blackboxsw> I think we actually need a new dch -i right?
<blackboxsw> ohh wait recipe builds failed
<blackboxsw> so there isn't a build of that binary already in repos
<Odd_Bloke> That version isn't really used anyway, the recipe build produces its own I think?
<Odd_Bloke> I think the changelog is only used for proper uploads, and we haven't released these versions yet.
<blackboxsw> Odd_Bloke: I was thinking only in terms of daily repos
<blackboxsw> daily ppa
<blackboxsw> which should already have a successful build pre-pytest
<blackboxsw> maybe??
<Odd_Bloke> The recipe builds don't use that version string.
<Odd_Bloke> They look like  20.1-2346-g65a1b90-0ubuntu1+1792~trunk~ubuntu20.04.1
<blackboxsw> ahh right it's based on something else ahh right  https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily
<blackboxsw> ok safe!
<blackboxsw> thanks for that
<blackboxsw> which makes sense as we might fix a ubuntu/<release> branch with more commits and not alter the debian/changelog so we'd need a different unique counter
<blackboxsw> to allow unreleased uploads to get tested.
<blackboxsw> ok I'm back on waiting for CI again :)
<blackboxsw> actually on a netbsd review
<Odd_Bloke> blackboxsw: I'm watching CI for those packaging branches, you can focus on something else. :)
<Odd_Bloke> OK, all daily builds are now back to working. \o/
<powersj> rharper, https://github.com/canonical/cloud-init/pull/238 is that ready?
<blackboxsw> powersj: I think I need a 2nd pass there.
<blackboxsw> my irc bouncer died again
<blackboxsw> will be making the switch to thelounge.chat tonight
<rharper> powersj: I would like to have some runtime verification on it before landing
#cloud-init 2020-03-12
<shaykeren> hi
<shaykeren> I'm trying to write to root home directory ~/ from userdata (/bin/sh) in ec2 instance(AWS), but I'm getting error because directory not exists, I found that HOME env variable is empty
<shaykeren> any idea?
<Odd_Bloke> shaykeren: Instead of using ~, you could just use /root/?
<shaykeren> sure, but I would like to know why it is working when my userdata runs with #!/bin/bash and also the HOME env var is not empty in that case,
<shaykeren> why is it behave differently?
<ananke> sounds like the issue with shell, not cloud init. is it /bin/sh from busybox or some other trimmed shell?
<shaykeren> im using ubuntu ami
<Odd_Bloke> shaykeren: Can you pastebin your userdata, please?
<ananke> they may be using dash, who knows how that behaves
<shaykeren> wait..
<shaykeren> userdata:
<shaykeren> #!/bin/shecho test>~/testecho $HOMEecho "[cs18][$(date +"%T.%3N")] start of chronos user-data";echo "[cs18][$(date +"%T.%3N")] end of chronos user-data";
<shaykeren> cloud-init-output:
<shaykeren> Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~16.04.2 running 'modules:config' at Thu, 12 Mar 2020 15:34:10 +0000. Up 25.08 seconds./var/lib/cloud/instance/scripts/part-001: 2: /var/lib/cloud/instance/scripts/part-001: cannot create ~/test: Directory nonexistent[cs18][15:34:12.364] start of chronos user-data[cs18][15:34:12.366] end of chronos user-data
<shaykeren> I also saw that in case the userdata has #!/bin/bash it is executed as /bin/bash [part-001 path] and incase of #!/bin/sh it is executed as /bin/bash[part-001 path]
<shaykeren> I also figured that if Ill run this script manually (inside the ec2 instance) /bin/sh[part-001 path] it is running ok without any issue
<Odd_Bloke> shaykeren: Sorry, when I say pastebin I mean something like https://paste.ubuntu.com/.  Could you paste that all there and then post the link in here?
<Odd_Bloke> (It's much easier to figure out what's going on when newlines are intact. :)
<shaykeren> sure my fault
<shaykeren> https://paste.ubuntu.com/p/Sxrf9727n7/
<shaykeren> cloud-init-output logs: https://paste.ubuntu.com/p/37sK5ggYTb/
<Odd_Bloke> shaykeren: OK, so that's the failing case, right?  What does the passing case look like?
<shaykeren> changing the #!/bin/sh to #!/bin/bash in the userdata
<Odd_Bloke> shaykeren: And what does the output look like, if you could paste that too?
<shaykeren> ok let me run it with /bin/bash
<shaykeren> cloud-init-output - https://paste.ubuntu.com/p/TbdhvB7gkb/ , file was created under home directory of the root user
<shaykeren> userdata - https://paste.ubuntu.com/p/jV9WBTGBGM/
<shaykeren> in both cases the HOME env variable was empty but in case of #!/bin/bash the file created
<Odd_Bloke> shaykeren: So I'm not sure why there's a difference in behaviour there, but I don't believe that cloud-init treats the two files differently.  So you're probably seeing differences in behaviour between bash and dash with the environment that cloud-init executes them in.
<Odd_Bloke> shaykeren: If you want to dig into this more, then please file a bug at https://bugs.launchpad.net/cloud-init/+filebug after reproducing on a more recent version of cloud-init (using the latest image in EC2 would do the trick).
<Odd_Bloke> And attach the output of `cloud-init collect-logs` on the /bin/sh instance, too.
<shaykeren> it is weird  because if I run it manually it is run with no issue
<Odd_Bloke> Yeah, it'll be something to do with the execution environment that cloud-init uses.
<Odd_Bloke> If you run it from a logged-in shell then there'll be a lot more environment variables around, for example.
<Odd_Bloke> And cloud-init also runs early enough in boot that some stuff may not yet be on disk, which can sometimes affect behaviour.  (I doubt it in this case, but you never know.)
<shaykeren> ok ill collect the logs and upload it
<shaykeren> Thanks!
<Odd_Bloke> :)
<blackboxsw> Odd_Bloke: this is a very thoughtful concern you have about ordering https://github.com/canonical/cloud-init/pull/114#discussion_r391222054 . What I think this means is that cloud-init in network_config only (always) adds route information to the first static IP address listed on an interface. Which means that we could currently be adding a normalized route to an IPv6 address (internal net_config subnet) xenial
<blackboxsw> and on an IPV4 interface on Bionic+ due to python key dict ordering difference right?
<blackboxsw> sorry that's a weighty question out of left field I realize
<blackboxsw> I'm trying to wrap my head around where/if this is a bug/shortcoming in existing cloud-init net_config translation for interfaces with multiple static addresses on a single interface
<blackboxsw> ahh I see now. that for loop on routes is actually adding all routes ipv4 and ipv6 to the first additional static subnet that we configure on an interface.
<blackboxsw> testsimple_render_bond_v2_input_netplan seems to be the only test that exercises this route adding
<blackboxsw> ahh and ok I get it. we are ok on ordering changes because cloud-init renders netplan with routes on the interface, not the specific address to which we are attaching all those routes.
<blackboxsw> here, https://github.com/canonical/cloud-init/blob/master/tests/unittests/test_net.py#L2061-L2099
<blackboxsw> ok routes in initial yaml-v2, get assigned to first IPv4 or ipv6 in internal network_confing addresses  list, and then gets bubbled up to netplan config output under network.bonds.bond0.routes instead of hanging off under network.bonds.bond0.addresses[0] as our internal code may have implied. I'll add a comment about this in the code so we don't have to dig into this again next time
<Goneri> blackboxsw, https://github.com/canonical/cloud-init/pull/62 you can merge the patch. It's all good here.
<blackboxsw> excellent Goneri, will you take a followup work item PR for *BSD to sort the package_command('upgrade') call?
<Goneri> Yes, I think, but first, I will rebase OpenBSD branch.
<blackboxsw> +1 no rush on that, just checking whether you agree that is something that should eventually be tackled
<Goneri> I would also like to take a look at the Ephemeral DHCP thing, it's a bit of a pain in the neck
<Goneri> unlike Linux, a BSD image should come with (mostly) zero packages. So it's less critical.
<Goneri> the base system is not part of a packaging system
<blackboxsw> Goneri: does the description of your PR now look acceptable? I'll be using that for the squash merge commit message https://github.com/canonical/cloud-init/pull/62
<blackboxsw> if you have any changes/corrections to the PR description text. please update and let me know when you are done reviewing it
<Goneri> yeah, it's fine. I've updated the list of the OS used during the tests
<Goneri> could you avoid a squash merge? I tried to isolate each patch as much as I could.
<Goneri> (well, ignore what I just said, the result is not that great. /me goes hide)
<blackboxsw> heh Goneri project-wise we squash merge the world in cloud-init
<Goneri> Roger that.
<blackboxsw> which is why generally we want to keep PRs  smaller and more concise
<blackboxsw> ... where possible
<Goneri> Yeah, I know the feeling :-)
<blackboxsw> merged Goneri  thanks again for that work
<Goneri> \o/ I've started the branch 1 year ago + 4 days :-) \o/
<Goneri> thanks all, I feel emotional now!
<blackboxsw> heh holy moly
<blackboxsw> should've landed that on the anniversary 4 days ago
<blackboxsw> !
<blackboxsw> it may be worth an update to https://cloudinit.readthedocs.io/en/latest/topics/availability.html#distributions
<blackboxsw> to add NetBSD in there too
<blackboxsw> rharper: Odd_Bloke https://github.com/canonical/cloud-init/pull/114/files#r391823711      reality check for me. I think dropping public-ipv4 check in ec2 still allows us to properly setup dhcp on primary/fallback nic always.
<rharper> blackboxsw: without looking, what's the benefit to dropping it? what sort of LOC or execution time are we saving ?
<blackboxsw> rharper: I don't think there is a really a benefit, probably just a risk
<blackboxsw> yeah one key lookup isn't going to break anything. so we can leave that logic in place
<blackboxsw> or reproduce it in the new v2 config
<blackboxsw> rharper: ec2 also doesn't actually add public IP configuration details to the instance at all. a work item that we should discuss
<blackboxsw> at some point.
<rharper> blackboxsw: please file a bug for that with details, that way it gets on the backlog
<blackboxsw> currently we rely on dhcp and any secondary local-ipv4s or ipv6s. no additional config added for reading ec2's public_ipv4s
<blackboxsw> rharper: will do
<rharper> AFAIK, they only publish public DNS names
<blackboxsw> rharper: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#example-output shows an example
<blackboxsw> and this PR 114 shows the updates as well with the new EC2 API version
<blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1867197
<ubot5> Ubuntu bug 1867197 in cloud-init "ec2: add support for rendering public IP addresses in network config" [Undecided,New]
<Goneri> blackboxsw, https://github.com/canonical/cloud-init/pull/250
<Odd_Bloke> Goneri: Congrats on the branch landing! \o/
<powersj> blackboxsw, what's left on your ec2 secondary nics branch?
<blackboxsw> powers working it actively right now. ahh and rharper my question about public-ipv4s handling is moot in this branch as that config in the network_config  v1 version of Ec2 only setup dhcp: true if we were fallback_nic, had local-ipv4s or had public-ipv4s metadata values.. in the new v2 network config for Ec2 per PR #114 we are setting dhcp4: true on all nics... I'm looking at trying to reconstitute only adding
<blackboxsw> dhcp4: true if fallback_nic public-ipv4s or local-ipv4s
<blackboxsw> powersj: I think the only thing left it resolving what I dropped regarding public-ipv4s reading from the metadata.
<blackboxsw> as least so we have no risk of regression, even though I believe it doesn't regress anything in it's current state, best to be sure
<rharper> blackboxsw: I don't understand the previous logic;  we always want to dhcp4 on primary nic;  and optionally add static ips if present
<rharper> IIUC, that logic came before we parsed network config from IMDS, no ?
<blackboxsw> rharper: the release  ec2 network_config would have added network-config information that didn't enable dhcp4 at all on a device if nic is not primary and doesn't have public or local ipv4 addrs
<rharper> yes, correct
<blackboxsw> the *released* ec2 network_config
<rharper> yes, I see that
<blackboxsw> the network_config in PR 114, was setting up dhcp4 on all nics
<rharper> ah, and you're fixing that
<blackboxsw> regardless of secondary nic with no local-ipv4 addrs
<blackboxsw> right
<blackboxsw> I think that's a gap in the switch that I left
<rharper> yes
<blackboxsw> so logic should be this for ec2:
<rharper> we dhcp *only* on primary; and for all nics (primary included) if there are secondary ips, add them (v4 or v6)
<blackboxsw> if fallback_nic(primary nic): dhcp4: true
<blackboxsw> if scondary nic: dhcp4 true only if local-ipv4s or public-ipv4s is present
<rharper> wait
<blackboxsw> yep stating case 3 then will wait
<rharper> I've never seen ec2 say you can dhcp on secondary nics
<blackboxsw> 3: if any nic and len(local-ipv4s) > 1  or len(ipv6s) > 1 then add those secondary IPs to nic config
<blackboxsw> rharper: I just tested that on an ec2 instance with 2 nics
<blackboxsw> dhcp on both gets you the proper matching ipv4 addrs and routes that secondary config would have setup statically
<blackboxsw> but maybe that's unsupported?
<rharper> but it is not required
<rharper> don't we already have the private ips ?
<rharper> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html
<rharper> says, yes you can dhcp on interfaces with private IP
<rharper> I'm not seeing them say dhcp on secondary interfaces should work; though I believe it did work for you
<blackboxsw> right "We allocate private IPv4 addresses to instances using DHCP"
<blackboxsw> but, what about secondary nics with only public ip addresses (is that even a thing?)
<rharper> public ips are 'elastic ips'
<rharper> I doubt those are allocated via DHCP
<rharper> and I'm not sure you're going to get additional private ips on the same interface
<blackboxsw> if we had nic2, only public-ipv4s, no local-ipv4s, would dhcp work. I think not as there is no private IP allocated to that nic
<rharper> can you confirm that your secondary nic dhcp response included more than one IP ?
<rharper> you always get a private ip (local-ipv4) no ?
<blackboxsw> rharper: yeah will setup one now
<rharper> thats how you communicate nic to nic internally from instance to instance
<blackboxsw> I think we always get a private IP on any attached interface
<blackboxsw> right
<rharper> yes
<blackboxsw> so I think my comment about "public-ipv4s" existing and no local-ipv4s is *not* a thing (not a viable vm network config)
<rharper> so, yes you can DHCP on all nics if we wanted;  I'm just wondering if that's useful vs. just assiging static ips
<blackboxsw> I think a prerequisite of having the attached nic is that it *must* have a local-ipv4s addr
<blackboxsw> rharper: right, probably not as useful. we could avoid doing that. though if ec2 instruments custom dhcp options we'd miss out due to our static ip config on secondary nic
<rharper> well, let's see what dhcp response shows up on secondary nics; I suspect it's the same as the primary
<blackboxsw> but if we add dhcp, on all nics ec2 vms also pay the cost of that dhcp roundtrip right
<blackboxsw> sounds good. will setup the instance now
<rharper> we can check what ec2net-utils does as well
<rharper> and check with Fred
<blackboxsw> so powersj the branch is close, I think we are circling the drain on final implementation. it doesn't involve a whole lot of work either way and I'd like to see it landed if we can today or tomorrow so we can get it into the CPC image pipeline
<blackboxsw> for focal
<powersj> blackboxsw, how do you and rharper close on the remaining work?
<blackboxsw> for this current ec2 branch?
<powersj> yes
<rharper> powersj: it's not going to happen in the short term
<rharper> we need to step back and confirm how we want to do it
<blackboxsw> I think I have to spin up an instance, we need to check dhcp config output and confirm we aren't missing something interesting by using static addresses?
<rharper> let's look at the AmazonLinux package; even see what a multi-nic multi-ip AmazonLinux instance looks like
<rharper> and then, I'd check with Fred to see if that's optimal, or if there are better things to do; and then we can make a decision
<rharper> now, if we wanted to "land it today"  I'd keep it with dhcp on primary, and then *static ips only*  if present on all other interfaces present
<blackboxsw> ok fair. so sample configs on multi-nic ubuntu and multi-nic amazonlinux and suggest what's best in email to fred/ahnvo ?
<rharper> blackboxsw: once we enable DHCP on all interfaces, we certainly need to do route-metric again
<rharper> just fred
<blackboxsw> rharper: why do we need route-metric if ec2 isn't using classless routes in dhcp?
<rharper> Ahn doesn't care about Ec2 networking I bet =)
<rharper> if it has a gateway
<rharper> you don't want it to clobber primary route
<blackboxsw> ahh I thought it was only gateway, plus classless static routes in dhcp that caused this concern
<rharper> gateway is a route
<rharper> the default
<rharper> classless are additional routes (which may include a default route as well) in which you ignore the gateway value,
<rharper> we know they don't currently put in a classless static route set; but there may be a GATEWAY= value  in which case we still need route metrics to ensure that we don't route packets meant for the internet out of the secondary interfaces
<blackboxsw> ok so short term potential of only enabling dhcp on primary interface and static for all the other  nics, would that get us into an upgrade pickle if we went dhcp after discussion with Fred?
<rharper> it may not have a GATEWAY value, but it could show up (accidental or on purpose) so it's best to put a metric on secondary interfaes
<rharper> yeah, unless we render network on every boot
<rharper> which we should discuss, with Fred;  I blieve we already do each boot but on Ec2 classic only
<blackboxsw> rharper: I think we also have CPC image magic that invalidates the cache on cloud images. but can confirm
<blackboxsw> so that'd be rendering network every boot, everywhere but that may also be limited to a specific ubuntu series
<rharper> I don't think so
<rharper> on Azure we added a dropin to rm the obj.pkl
<rharper> only on ec2-classic do we render every boot; due to MAC address on nic changes between stops/stars
<rharper> on vpc, all is fine
<rharper> this reminds me of wanting a table on datasource capabilities (check_instance_id, network_config, update_event tpes)
<blackboxsw> rharper: https://paste.ubuntu.com/p/ZyFKkxPDCB/
<blackboxsw> so I'm on a vpc instance (non-classic) and I see cache invalid for Ec2Local ds detection across simple reboots
<rharper>         # Non-VPC (aka Classic) Ec2 instances need to rewrite the
<rharper>             # network config file every boot due to MAC address change.
<rharper>             if self.is_classic_instance():
<rharper>                 self.update_events['network'].add(EventType.BOOT)
<rharper> so I don't know what's going on in the image but I do know what code we wrote
<blackboxsw> right agreed on what that code does. there is just some drop in magic at play here I think in ec2 images
<rharper> and we don't persist object.pkl on ec2
<rharper> as it doesn't implement check_instance_id()
<blackboxsw> I've wrapped myself around the axle at the moment on this. first I'll get that multi-nic instance up with PR 114 so we can dissect the dhcp response from networkd
<rharper> blackboxsw: Odd_Bloke: interested in your thoughts on this: https://github.com/canonical/cloud-init/pull/238#issuecomment-598408582
<rharper> when you have time to look
<blackboxsw> rharper: sorry here's dhcp info on dual-nic ec2 vm https://pastebin.ubuntu.com/p/tg2ZhZ3V6Z/
<rharper> ROUTER=172.31.32.1
<rharper> that will put in a default route
<rharper> so, we defintely want a dhcp-route-metric
<rharper> if we go with dhcp on secondary interfaces
<blackboxsw> so metrics required in this case. ok
<blackboxsw> route-metric rather.
<rharper> dhcpX-override: {'route-metric': NNN}'
<blackboxsw> and /me just removed it. sorry a more concerted discuss was in order yesterday or the day before to make sure I was gong down the right path.
<rharper> hehe
<rharper> blackboxsw: also, on your unittests there was a bunch of mac.lower() after you had capitalized on of the MAC values;  what was that about ?
<blackboxsw> rharper: I think that was earlier me making sure we exercised some of the internal logic in cloud-init which I know lower()'s  the mac addr we get from IMDS. I should have instead just added a specific unit test that validated uppercase and lowercase macs result in same rendered net config
<rharper> blackboxsw: ah, ok, yeah; less splash damage to other  tests
<blackboxsw> yeah, and clear documentation of the intent
<blackboxsw> rharper: ok, so what path do we want to go on for focal for ec2 secondary nics do you think?
<blackboxsw> static addr setup on secondary nics?
<blackboxsw> as it stands currently, it looks like published cloud-init on bionic only configs primary nic even on dual-nic boxes
<rharper> let's look at ec2utils and see what AmazonLinux does; if they dhcp + additional private ips; then I think we do the same
<blackboxsw> hrm, I see a bunch of primary actions (on eth0 only) https://github.com/aws/ec2-net-utils/blob/master/ec2net-functions
<blackboxsw> checking around for stuff handling 2nd nic
<blackboxsw> which calls plug_interface for all interfaces and only activate_primary on each hotplug add
<blackboxsw> https://github.com/aws/ec2-net-utils/blob/master/ec2ifup
<blackboxsw> yeah seems in all cases rewrite_primary gets called which noops on !eth0
<rharper> no, it ignores eht0
<rharper> that's code for all other interfaces
<rharper> no ?
<rharper> so they do dhcp on non-eth0
<rharper> and then ensure the rules for non-eth0 don't clobber eth0 (which Im sure they already dhcp on eth0 )
<rharper> blackboxsw: so, to me, that's equivalent to what we're suggesting now;  dhcp on all interfaces, add secondary ips on all interfaces that have them, and ensure non-primary interfaces have a route-metric (which is what they do with the route table bits)
<blackboxsw> hahah right complete misread
<blackboxsw> ok will reconstitute the route-metric bits.
<blackboxsw> RTABLE=${INTERFACE#eth}
<blackboxsw> let RTABLE+=10000
<blackboxsw> right
<blackboxsw> 10000 offset per nic
<blackboxsw> o k
<blackboxsw> so they add route-metric on nic >= 1
<blackboxsw> is there a base route metric I wonder
<blackboxsw> on eth0
<blackboxsw> which is what we do on Azure
<rharper> ah, its not a metric
<rharper> its a different routing table altogether
<rharper> but it has the same effect in that the primary route table is consulted *first* before looking at higher value tables
<blackboxsw> rharper: I was curious if different routing table name/id is equivalent to different metric
<blackboxsw> right
<blackboxsw> ij
<blackboxsw> ok
<blackboxsw> rharper: so plan of attack for cloud-init ec2 multi-nic, multi-ip
<blackboxsw> https://hackmd.io/rBjW9rjPRg6LYydxOgW8cQ?sync=&type=
<rharper> right, we dhcp6 as well if interface has ipv6 right?  so the logic is the same for static ipv6 secondary ips as v4
<blackboxsw> right. yes rharper
<rharper> cool
<blackboxsw> dhcp6 only active ipv6s values
<blackboxsw> ok I can correct this branch as I think all it needs to route-metrics at the moment
<rharper> excellent
<rharper> Do we need a doc update to the Ec2 Datasource docs w.r.t network configuration ?
<rharper> if not, I think it would be a good add to describe what we plan to configure in the multi-nic, multi-ip scenarios (v4, v6 and mixed)
<blackboxsw> rharper: can add that to the datasource since we are touching it w.r.t. secondary_nic config option
<blackboxsw> makes sense
<blackboxsw> rharper: so route-metric: 0 for eth0?
<blackboxsw> or route-metric: 100
<rharper> no, we did 100
<blackboxsw> for eth0
<blackboxsw> right ok
<blackboxsw> just wanted to confirm
<rharper> (index  + 1) * 100
<rharper> same as azure IIRC
<blackboxsw> agreed
#cloud-init 2020-03-13
<blackboxsw> rharper: just pushed followup on https://github.com/canonical/cloud-init/pull/114 with doc updates and renamed ds config option to ``apply_network_config`` which aligns with both openstack and azure config option for getting stuff from imds
<blackboxsw> s/stuff/network config/
<blackboxsw> https://paste.ubuntu.com/p/DyvY7vjFyN/ is the successful network config for multi-nics + secondary-ips
<blackboxsw> sorry multi-nic only, will add secondary ips to that too (and some ipv6) for a more full success route test
<rharper> blackboxsw: ok
<Odd_Bloke> rharper: blackboxsw: powersj: Just pushed up my revisions to the code review doc: https://github.com/canonical/cloud-init/pull/160/
<rharper> k
<blackboxsw> sweet Odd_Bloke will grab that next. wrapping up two other cloud-init pr reviews
<blackboxsw> rharper: anything else in progress you are working related to initramfs ? https://github.com/canonical/cloud-init/pull/238
<blackboxsw> or is that branch ready for review
<rharper> what's 238 ?
<rharper> sorry
<rharper> heads down in reviewing your 114
<rharper> the initramfs.netplan one needs review from xnox/vorlon before we move forward
<blackboxsw> np just wondering if you are expecting more work on your #238 (I'll not review it until I get through OddBloke's 160)
<blackboxsw> ok thanks rharper
<rharper> blackboxsw: ok, finished review 114
<powersj> Odd_Bloke, approved
<Odd_Bloke> Thanks!
<blackboxsw> onto 160 now. closed out https://github.com/canonical/cloud-init/pull/75
<blackboxsw> updated FFe bug with secondary nic & secondary ip address context https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1866930
<ubot5> Ubuntu bug 1866930 in cloud-init (Ubuntu) "[FFe] ec2 add support for configuring secondary NICs and secondary ipv4 and ipv6 addresses" [High,In progress]
<blackboxsw> Odd_Bloke: approved with nits https://github.com/canonical/cloud-init/pull/160#pullrequestreview-374601125
<blackboxsw> if you want the nits take 'em otherwise merge at will
<Odd_Bloke> blackboxsw: Well, I'm going to want to ping community folks for feedback, just giving core folks a head start. :)
<powersj> blackboxsw, did you get another review on ec2 branch?
<blackboxsw> powersj: rharper: said he's heads down on it
<blackboxsw> right now
<rharper> blackboxsw: I finished up 30 minutes ago .. .did you push up again ?
<rharper> I left like 20 comments
<blackboxsw> powersj: rharper I also validated upgrade path + reboot doesn't rewrite network config with multiple nics if former net config only had eth0 rendered.
<blackboxsw> rharper: no didn't see what reviewing OddBloke's but done there.
<blackboxsw> will address comments
<powersj> rharper, thanks
<rharper> np
<blackboxsw> rharper: so primary interface discovery on ec2 is really related to find_fallback_nic_on_linux() not truly what ec2 mentioned as device-number: 0 https://github.com/canonical/cloud-init/pull/114/files#r392430307
<blackboxsw> as in, there is nothing currently in ec2 net config that truly checks device-number. but I guess you were saying we should document this correlation
<blackboxsw> ultimately rharper do you think the route-metric should be set to 100 * nic_metadata['device-number'] +1 and we could drop the nic_idx counter variable?
<rharper> blackboxsw: I would be shocked if those don't align
<blackboxsw> rharper: I'm almost positive they align, but the code itself doesn't *do* that is what I was raising
<blackboxsw> but maybe it should :)
<rharper> I feel like it should too, but I'm somewhat worried...
<rharper> let me reread the ec2 docs
<rharper> blackboxsw: it says, device-index number corresponds to eth<number>   ;   I wonder if that matches 'ifindex' value
<blackboxsw> rharper: ssh ubuntu@18.191.148.254
<blackboxsw> has 2 nics
<rharper> hrm, I wonder why they're called ethX instead of en (predictable names)
<rharper> no, ifindex does not relate
<rharper> that's xen, I wonder what nitro ones look like
<rharper> blackboxsw: well, maybe we can do work on device-id later then
<rharper> I'd rather be sure than change it now
<blackboxsw> rharper: +1
<blackboxsw> waiting on that.
<blackboxsw> rharper: clarification for you  https://github.com/canonical/cloud-init/pull/114/files#r392493895
<blackboxsw> if we are doing primary nic, it can have secondary addrs so we still need the apply_network_config check before adding secondary ips
<blackboxsw> sound ok? (that's current behavior of cloud-init everywhere anyway) primary nic, no secondary ips
<blackboxsw> ahh interesting. and I think I have something to fix then
<blackboxsw> https://github.com/canonical/cloud-init/pull/114/files#r392419963
<blackboxsw> ok I saw our unittest which showed rendered v2 routes that were not under the specific address.
<rharper> routes are per interface in v2
<rharper> there is no top-level routes in v2
<rharper> that will not validate via netplan generate
<rharper> blackboxsw: on apply_network_config;  if False, primary gets dhcp4 only, no secondary ; and dhcp6 instead of dhcp4 if ipv6s is non-empty, Yes?  do we do dual-stack v4 + v6 (dhcp4/dhcp6) as well (and all of this is with apply_network_config: False)
<rharper> if apply_network_config=True ;; then all of the things everywhere
<blackboxsw> 1.  on apply_network_config;  if False, primary gets dhcp4 only, no secondary ; YES
<blackboxsw> 2.  and dhcp6 instead of dhcp4 if ipv6s is non-empty, Yes?       No. it gets both dhcp4 and dhcp6
<blackboxsw> we bring up dhcp4 always on primary because it is needed anyway to talk to the imds
<blackboxsw> so we know it works because we couldn't have getting IMDS networkconfig otherwise
<blackboxsw> 3. do we do dual stack ipv4/v6 when apply_network_config == false : YES
<blackboxsw> just no secondary ips in that case (or secondary nics)
<blackboxsw> rharper: ^ sorry forgot to address me response
<blackboxsw> *my8
<rharper> ok, on False, agreed, forgot about v4 for imds
<blackboxsw> just pushed all I have. scrubbing rharper's review comments to make sure I addressed them all
<blackboxsw> ahh right subnet route ordering is wonky https://github.com/canonical/cloud-init/pull/114/files#r392419963
<rharper> blackboxsw: one more suggestion
<rharper> in convert, I  think if we separate out the logic into two parts,  apply_network_config==False (this is the primary nic dhcp4 + optionally v6, no additional ips) ; and then apply_network_config==True, and this just configures *all* the nics with all features per metadata
<rharper> it's easier to see the path that older relases will take, and only the apply_network_config==True path will loop over all interfaces and enable all of the things.
<blackboxsw> ahh rharper no prob. that makes sense and simplifies the for loop for all
<blackboxsw> adding now
<blackboxsw> rharper: just pushed that separation of logic
<blackboxsw> 1f35018da2abd253561da82bdc7f577eb1058cb2
<blackboxsw> I know we're running out of daylight
<blackboxsw> :/
